2024-03-27T21:28:20.691 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-27T21:28:20.691 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_yuriw@teuthology --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} --archive /home/teuthworker/archive/yuriw-2024-03-27_14:58:57-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626549 --name yuriw-2024-03-27_14:58:57-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-03-27_14:58:57-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626549/orig.config.yaml 2024-03-27T21:28:20.730 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-27T21:28:20.816 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-27_14:58:57-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626549 branch: wip-yuri4-testing-2024-03-26-1132-reef description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} email: yweinste@redhat.com first_in_suite: false job_id: '7626549' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-03-27_14:58:57-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri4-testing-2024-03-26-1132-reef ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 3 ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request sha1: 7c37d80f7a88f4eb561c4fe02558660a650b10ac ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 7c37d80f7a88f4eb561c4fe02558660a650b10ac thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2024-03-26-1132-reef sha1: 7c37d80f7a88f4eb561c4fe02558660a650b10ac owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6904 sha1: 7c37d80f7a88f4eb561c4fe02558660a650b10ac sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2024-03-26-1132-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7c37d80f7a88f4eb561c4fe02558660a650b10ac/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7c37d80f7a88f4eb561c4fe02558660a650b10ac targets: smithi060.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFseykjbMcoy37jwDf5haCtBthQp6+lf1FiaXkGRWge5FX0PR1O4cJ2V30pH6x72tmNxBTPzCx9KT7u+fUIv6h8= smithi191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFEoU/N98mvWvDGXsH9+Sn5SsdMdzH4v6vuk/Nq61Sr8dUL3puAnexOkO+5lZaf8xGqQoaGc/SGAsyBoAbeo/t0= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-27_14:58:57 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-27T21:28:20.816 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7c37d80f7a88f4eb561c4fe02558660a650b10ac/qa; will attempt to use it 2024-03-27T21:28:20.817 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7c37d80f7a88f4eb561c4fe02558660a650b10ac/qa/tasks 2024-03-27T21:28:20.817 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-27T21:28:20.819 INFO:teuthology.task.internal:Checking packages... 2024-03-27T21:28:20.844 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '7c37d80f7a88f4eb561c4fe02558660a650b10ac' 2024-03-27T21:28:20.844 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-27T21:28:20.844 INFO:teuthology.packaging:ref: None 2024-03-27T21:28:20.844 INFO:teuthology.packaging:tag: None 2024-03-27T21:28:20.844 INFO:teuthology.packaging:branch: wip-yuri4-testing-2024-03-26-1132-reef 2024-03-27T21:28:20.844 INFO:teuthology.packaging:sha1: 7c37d80f7a88f4eb561c4fe02558660a650b10ac 2024-03-27T21:28:20.844 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri4-testing-2024-03-26-1132-reef 2024-03-27T21:28:21.065 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1053.ge58b9979 2024-03-27T21:28:21.067 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-27T21:28:21.073 INFO:teuthology.task.internal:no buildpackages task found 2024-03-27T21:28:21.073 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-27T21:28:21.198 INFO:teuthology.task.internal:Saving configuration 2024-03-27T21:28:21.213 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-27T21:28:21.234 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-27T21:28:21.255 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-27_14:58:57-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626549', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-27 21:23:34.404231', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFseykjbMcoy37jwDf5haCtBthQp6+lf1FiaXkGRWge5FX0PR1O4cJ2V30pH6x72tmNxBTPzCx9KT7u+fUIv6h8='} 2024-03-27T21:28:21.275 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-27_14:58:57-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626549', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-27 21:23:34.405724', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFEoU/N98mvWvDGXsH9+Sn5SsdMdzH4v6vuk/Nq61Sr8dUL3puAnexOkO+5lZaf8xGqQoaGc/SGAsyBoAbeo/t0='} 2024-03-27T21:28:21.275 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-27T21:28:21.299 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-27T21:28:21.300 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-27T21:28:21.300 INFO:teuthology.run_tasks:Running task console_log... 2024-03-27T21:28:21.444 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0723eb7f70>, signals=[15]) 2024-03-27T21:28:21.445 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-27T21:28:21.452 INFO:teuthology.task.internal:Opening connections... 2024-03-27T21:28:21.452 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi060.front.sepia.ceph.com 2024-03-27T21:28:21.454 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T21:28:21.563 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2024-03-27T21:28:21.564 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T21:28:21.667 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-27T21:28:21.673 DEBUG:teuthology.orchestra.run.smithi060:> uname -m 2024-03-27T21:28:21.692 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2024-03-27T21:28:21.693 DEBUG:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2024-03-27T21:28:21.751 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="CentOS Stream" 2024-03-27T21:28:21.753 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="9" 2024-03-27T21:28:21.753 INFO:teuthology.orchestra.run.smithi060.stdout:ID="centos" 2024-03-27T21:28:21.753 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE="rhel fedora" 2024-03-27T21:28:21.753 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="9" 2024-03-27T21:28:21.753 INFO:teuthology.orchestra.run.smithi060.stdout:PLATFORM_ID="platform:el9" 2024-03-27T21:28:21.753 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-27T21:28:21.753 INFO:teuthology.orchestra.run.smithi060.stdout:ANSI_COLOR="0;31" 2024-03-27T21:28:21.753 INFO:teuthology.orchestra.run.smithi060.stdout:LOGO="fedora-logo-icon" 2024-03-27T21:28:21.753 INFO:teuthology.orchestra.run.smithi060.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-27T21:28:21.753 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://centos.org/" 2024-03-27T21:28:21.753 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-27T21:28:21.754 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-27T21:28:21.754 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-27T21:28:21.754 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2024-03-27T21:28:21.783 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2024-03-27T21:28:21.802 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2024-03-27T21:28:21.802 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2024-03-27T21:28:21.859 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="CentOS Stream" 2024-03-27T21:28:21.859 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="9" 2024-03-27T21:28:21.860 INFO:teuthology.orchestra.run.smithi191.stdout:ID="centos" 2024-03-27T21:28:21.860 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="rhel fedora" 2024-03-27T21:28:21.860 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="9" 2024-03-27T21:28:21.860 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el9" 2024-03-27T21:28:21.860 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-27T21:28:21.860 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2024-03-27T21:28:21.860 INFO:teuthology.orchestra.run.smithi191.stdout:LOGO="fedora-logo-icon" 2024-03-27T21:28:21.860 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-27T21:28:21.860 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://centos.org/" 2024-03-27T21:28:21.860 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-27T21:28:21.860 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-27T21:28:21.860 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-27T21:28:21.861 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2024-03-27T21:28:21.882 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-27T21:28:21.893 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-27T21:28:21.910 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-27T21:28:21.911 DEBUG:teuthology.orchestra.run.smithi060:> test '!' -e /home/ubuntu/cephtest 2024-03-27T21:28:21.915 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2024-03-27T21:28:21.931 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-27T21:28:21.937 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-27T21:28:21.937 DEBUG:teuthology.orchestra.run.smithi060:> test -z $(ls -A /var/lib/ceph) 2024-03-27T21:28:21.973 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2024-03-27T21:28:21.998 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-27T21:28:22.057 INFO:teuthology.run_tasks:Running task kernel... 2024-03-27T21:28:22.072 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-27T21:28:22.072 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-27T21:28:22.072 DEBUG:teuthology.orchestra.run.smithi060:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-27T21:28:22.073 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-27T21:28:22.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T21:28:22.090 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-03-27T21:28:22.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T21:28:22.091 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-03-27T21:28:22.147 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-361.el9.x86_64 2024-03-27T21:28:22.147 INFO:teuthology.task.kernel:Running kernel on smithi191: 5.14.0-361.el9.x86_64 2024-03-27T21:28:22.147 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum install -y kernel 2024-03-27T21:28:22.149 INFO:teuthology.orchestra.run.smithi060.stdout:5.14.0-361.el9.x86_64 2024-03-27T21:28:22.149 INFO:teuthology.task.kernel:Running kernel on smithi060: 5.14.0-361.el9.x86_64 2024-03-27T21:28:22.149 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum install -y kernel 2024-03-27T21:28:24.281 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-27T21:28:24.437 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-27T21:28:25.639 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2024-03-27T21:28:25.990 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-03-27T21:28:27.364 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 7.8 MB/s | 8.0 MB 00:01 2024-03-27T21:28:29.165 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-03-27T21:28:29.888 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-03-27T21:28:31.839 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-27T21:28:36.182 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 1.8 MB/s | 6.1 MB 00:03 2024-03-27T21:28:36.248 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 6.4 MB/s | 6.1 MB 00:00 2024-03-27T21:28:37.565 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 85 kB/s | 16 kB 00:00 2024-03-27T21:28:38.708 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-27T21:28:39.300 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-27T21:28:40.920 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 9.9 MB/s | 21 MB 00:02 2024-03-27T21:28:44.291 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-27T21:28:45.993 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-27T21:28:46.448 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-27T21:28:46.448 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-27T21:28:46.534 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:28:46.538 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:28:46.538 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-03-27T21:28:46.539 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:28:46.539 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-27T21:28:46.539 INFO:teuthology.orchestra.run.smithi191.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-27T21:28:46.539 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-27T21:28:46.539 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-27T21:28:46.539 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-27T21:28:46.539 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-27T21:28:46.539 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:28:46.539 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-27T21:28:46.539 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:28:46.539 INFO:teuthology.orchestra.run.smithi191.stdout:Install 4 Packages 2024-03-27T21:28:46.539 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:28:46.540 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 76 M 2024-03-27T21:28:46.540 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 123 M 2024-03-27T21:28:46.540 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-27T21:28:46.776 INFO:teuthology.orchestra.run.smithi191.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 540 kB/s | 66 kB 00:00 2024-03-27T21:28:48.291 INFO:teuthology.orchestra.run.smithi060.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-27T21:28:48.291 INFO:teuthology.orchestra.run.smithi060.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-27T21:28:48.385 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:28:48.397 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:28:48.397 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-03-27T21:28:48.398 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:28:48.398 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-27T21:28:48.398 INFO:teuthology.orchestra.run.smithi060.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-27T21:28:48.398 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-27T21:28:48.398 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-27T21:28:48.398 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-27T21:28:48.398 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-27T21:28:48.398 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:28:48.398 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-27T21:28:48.398 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:28:48.398 INFO:teuthology.orchestra.run.smithi060.stdout:Install 4 Packages 2024-03-27T21:28:48.398 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:28:48.399 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 76 M 2024-03-27T21:28:48.399 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 123 M 2024-03-27T21:28:48.399 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-27T21:28:48.577 INFO:teuthology.orchestra.run.smithi060.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 1.0 MB/s | 66 kB 00:00 2024-03-27T21:28:49.136 INFO:teuthology.orchestra.run.smithi060.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 24 MB/s | 15 MB 00:00 2024-03-27T21:28:49.736 INFO:teuthology.orchestra.run.smithi060.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 27 MB/s | 33 MB 00:01 2024-03-27T21:28:49.809 INFO:teuthology.orchestra.run.smithi191.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 4.7 MB/s | 15 MB 00:03 2024-03-27T21:28:50.003 INFO:teuthology.orchestra.run.smithi060.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 19 MB/s | 27 MB 00:01 2024-03-27T21:28:50.003 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:28:50.003 INFO:teuthology.orchestra.run.smithi060.stdout:Total 47 MB/s | 76 MB 00:01 2024-03-27T21:28:50.700 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-27T21:28:50.842 INFO:teuthology.orchestra.run.smithi191.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 8.0 MB/s | 33 MB 00:04 2024-03-27T21:28:51.133 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-27T21:28:51.133 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-27T21:28:52.059 INFO:teuthology.orchestra.run.smithi191.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 5.2 MB/s | 27 MB 00:05 2024-03-27T21:28:52.060 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:28:52.060 INFO:teuthology.orchestra.run.smithi191.stdout:Total 14 MB/s | 76 MB 00:05 2024-03-27T21:28:52.146 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-27T21:28:52.147 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-27T21:28:52.729 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-27T21:28:53.071 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-27T21:28:53.071 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-27T21:28:54.079 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-27T21:28:54.080 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-27T21:28:54.737 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-27T21:28:55.396 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-27T21:28:55.487 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T21:28:56.530 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-27T21:28:57.132 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-27T21:28:57.227 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T21:28:58.363 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T21:28:58.500 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T21:28:59.980 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T21:29:00.114 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T21:29:03.582 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T21:29:03.626 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-27T21:29:04.841 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T21:29:04.873 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-27T21:29:08.725 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T21:29:09.611 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T21:29:38.847 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T21:29:38.847 INFO:teuthology.orchestra.run.smithi191.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-27T21:29:38.847 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:29:38.892 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-27T21:29:39.265 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-27T21:29:39.265 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-27T21:29:39.265 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T21:29:39.265 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T21:29:39.673 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T21:29:39.673 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:29:39.673 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-27T21:29:39.673 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-27T21:29:39.673 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-27T21:29:39.673 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-27T21:29:39.673 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-27T21:29:39.673 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:29:39.673 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:29:39.960 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T21:29:39.960 INFO:teuthology.orchestra.run.smithi060.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-27T21:29:39.960 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:29:40.006 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-27T21:29:40.011 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2024-03-27T21:29:40.123 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-27T21:29:40.123 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-27T21:29:40.123 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-27T21:29:40.123 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-27T21:29:40.123 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-27T21:29:40.123 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-27T21:29:40.124 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi191.front.sepia.ceph.com, path=None, version=distro) 2024-03-27T21:29:40.124 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2024-03-27T21:29:40.203 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-27T21:29:40.204 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-27T21:29:40.204 DEBUG:teuthology.orchestra.run.smithi191:> sudo rpm -qi grub2-tools 2024-03-27T21:29:40.238 INFO:teuthology.orchestra.run.smithi191.stdout:Name : grub2-tools 2024-03-27T21:29:40.238 INFO:teuthology.orchestra.run.smithi191.stdout:Epoch : 1 2024-03-27T21:29:40.238 INFO:teuthology.orchestra.run.smithi191.stdout:Version : 2.06 2024-03-27T21:29:40.238 INFO:teuthology.orchestra.run.smithi191.stdout:Release : 68.el9 2024-03-27T21:29:40.238 INFO:teuthology.orchestra.run.smithi191.stdout:Architecture: x86_64 2024-03-27T21:29:40.238 INFO:teuthology.orchestra.run.smithi191.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-27T21:29:40.239 INFO:teuthology.orchestra.run.smithi191.stdout:Group : Unspecified 2024-03-27T21:29:40.239 INFO:teuthology.orchestra.run.smithi191.stdout:Size : 8264254 2024-03-27T21:29:40.239 INFO:teuthology.orchestra.run.smithi191.stdout:License : GPLv3+ 2024-03-27T21:29:40.239 INFO:teuthology.orchestra.run.smithi191.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-27T21:29:40.239 INFO:teuthology.orchestra.run.smithi191.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-27T21:29:40.239 INFO:teuthology.orchestra.run.smithi191.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-27T21:29:40.239 INFO:teuthology.orchestra.run.smithi191.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-27T21:29:40.239 INFO:teuthology.orchestra.run.smithi191.stdout:Packager : builder@centos.org 2024-03-27T21:29:40.239 INFO:teuthology.orchestra.run.smithi191.stdout:Vendor : CentOS 2024-03-27T21:29:40.239 INFO:teuthology.orchestra.run.smithi191.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-27T21:29:40.239 INFO:teuthology.orchestra.run.smithi191.stdout:Summary : Support tools for GRUB. 2024-03-27T21:29:40.239 INFO:teuthology.orchestra.run.smithi191.stdout:Description : 2024-03-27T21:29:40.239 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:29:40.239 INFO:teuthology.orchestra.run.smithi191.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-27T21:29:40.239 INFO:teuthology.orchestra.run.smithi191.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-27T21:29:40.240 INFO:teuthology.orchestra.run.smithi191.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-27T21:29:40.240 INFO:teuthology.orchestra.run.smithi191.stdout:hardware devices. 2024-03-27T21:29:40.240 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:29:40.240 INFO:teuthology.orchestra.run.smithi191.stdout:This subpackage provides tools for support of all platforms. 2024-03-27T21:29:40.241 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-27T21:29:40.241 INFO:teuthology.task.kernel:Updating grub on smithi191 to boot 5.14.0-432.el9.x86_64 2024-03-27T21:29:40.241 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-27T21:29:40.406 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-27T21:29:40.406 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-27T21:29:40.406 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T21:29:40.406 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T21:29:40.797 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T21:29:40.797 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:29:40.797 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-27T21:29:40.798 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-27T21:29:40.798 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-27T21:29:40.798 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-27T21:29:40.798 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-27T21:29:40.798 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:29:40.798 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:29:40.875 INFO:teuthology.orchestra.run.smithi191.stderr:Generating grub configuration file ... 2024-03-27T21:29:41.155 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q kernel | sort -rV | head -n 1 2024-03-27T21:29:41.256 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-27T21:29:41.256 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-27T21:29:41.256 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-27T21:29:41.256 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-27T21:29:41.256 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-27T21:29:41.256 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-27T21:29:41.256 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi060.front.sepia.ceph.com, path=None, version=distro) 2024-03-27T21:29:41.256 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q kernel | sort -rV | head -n 1 2024-03-27T21:29:41.340 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-27T21:29:41.340 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-27T21:29:41.340 DEBUG:teuthology.orchestra.run.smithi060:> sudo rpm -qi grub2-tools 2024-03-27T21:29:41.382 INFO:teuthology.orchestra.run.smithi060.stdout:Name : grub2-tools 2024-03-27T21:29:41.382 INFO:teuthology.orchestra.run.smithi060.stdout:Epoch : 1 2024-03-27T21:29:41.382 INFO:teuthology.orchestra.run.smithi060.stdout:Version : 2.06 2024-03-27T21:29:41.382 INFO:teuthology.orchestra.run.smithi060.stdout:Release : 68.el9 2024-03-27T21:29:41.382 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: x86_64 2024-03-27T21:29:41.382 INFO:teuthology.orchestra.run.smithi060.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-27T21:29:41.382 INFO:teuthology.orchestra.run.smithi060.stdout:Group : Unspecified 2024-03-27T21:29:41.382 INFO:teuthology.orchestra.run.smithi060.stdout:Size : 8264254 2024-03-27T21:29:41.383 INFO:teuthology.orchestra.run.smithi060.stdout:License : GPLv3+ 2024-03-27T21:29:41.383 INFO:teuthology.orchestra.run.smithi060.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-27T21:29:41.383 INFO:teuthology.orchestra.run.smithi060.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-27T21:29:41.383 INFO:teuthology.orchestra.run.smithi060.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-27T21:29:41.383 INFO:teuthology.orchestra.run.smithi060.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-27T21:29:41.383 INFO:teuthology.orchestra.run.smithi060.stdout:Packager : builder@centos.org 2024-03-27T21:29:41.383 INFO:teuthology.orchestra.run.smithi060.stdout:Vendor : CentOS 2024-03-27T21:29:41.383 INFO:teuthology.orchestra.run.smithi060.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-27T21:29:41.383 INFO:teuthology.orchestra.run.smithi060.stdout:Summary : Support tools for GRUB. 2024-03-27T21:29:41.383 INFO:teuthology.orchestra.run.smithi060.stdout:Description : 2024-03-27T21:29:41.383 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:29:41.383 INFO:teuthology.orchestra.run.smithi060.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-27T21:29:41.383 INFO:teuthology.orchestra.run.smithi060.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-27T21:29:41.383 INFO:teuthology.orchestra.run.smithi060.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-27T21:29:41.383 INFO:teuthology.orchestra.run.smithi060.stdout:hardware devices. 2024-03-27T21:29:41.383 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:29:41.384 INFO:teuthology.orchestra.run.smithi060.stdout:This subpackage provides tools for support of all platforms. 2024-03-27T21:29:41.384 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-27T21:29:41.385 INFO:teuthology.task.kernel:Updating grub on smithi060 to boot 5.14.0-432.el9.x86_64 2024-03-27T21:29:41.385 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-27T21:29:41.987 INFO:teuthology.orchestra.run.smithi060.stderr:Generating grub configuration file ... 2024-03-27T21:29:42.321 INFO:teuthology.orchestra.run.smithi191.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-27T21:29:42.352 INFO:teuthology.orchestra.run.smithi191.stderr:done 2024-03-27T21:29:42.354 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2024-03-27T21:29:42.370 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.qSc0FtCihP 2024-03-27T21:29:42.371 DEBUG:teuthology.orchestra.run.smithi191:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.qSc0FtCihP 2024-03-27T21:29:42.439 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.qSc0FtCihP 2024-03-27T21:29:42.586 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.qSc0FtCihP is 6KB 2024-03-27T21:29:42.635 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.qSc0FtCihP 2024-03-27T21:29:42.650 DEBUG:teuthology.orchestra.run.smithi191:> sudo /bin/ls /boot/loader/entries || true 2024-03-27T21:29:42.714 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-27T21:29:42.714 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-27T21:29:42.714 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-27T21:29:42.714 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-27T21:29:42.715 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-27T21:29:42.846 DEBUG:teuthology.orchestra.run.smithi191:> sudo shutdown -r now 2024-03-27T21:29:43.428 INFO:teuthology.orchestra.run.smithi060.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-27T21:29:43.450 INFO:teuthology.orchestra.run.smithi060.stderr:done 2024-03-27T21:29:43.450 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-03-27T21:29:43.466 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.zcO0ZEwIm8 2024-03-27T21:29:43.466 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zcO0ZEwIm8 2024-03-27T21:29:43.538 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.zcO0ZEwIm8 2024-03-27T21:29:43.687 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.zcO0ZEwIm8 is 6KB 2024-03-27T21:29:43.736 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.zcO0ZEwIm8 2024-03-27T21:29:43.751 DEBUG:teuthology.orchestra.run.smithi060:> sudo /bin/ls /boot/loader/entries || true 2024-03-27T21:29:43.819 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-27T21:29:43.820 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-27T21:29:43.820 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-27T21:29:43.820 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-27T21:29:43.821 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-27T21:29:43.975 DEBUG:teuthology.orchestra.run.smithi060:> sudo shutdown -r now 2024-03-27T21:30:12.878 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-27T21:30:12.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-27T21:30:12.880 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T21:30:13.978 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-27T21:30:13.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-03-27T21:30:13.979 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T21:30:31.327 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.191 2024-03-27T21:30:32.350 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.60 2024-03-27T21:30:40.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-27T21:30:40.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T21:30:41.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-03-27T21:30:41.353 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T21:30:43.395 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.191 2024-03-27T21:30:55.578 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-27T21:30:55.579 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T21:30:58.658 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.191 2024-03-27T21:31:13.672 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-27T21:31:13.673 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T21:31:31.998 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.191 2024-03-27T21:31:41.362 DEBUG:teuthology.orchestra.remote:timed out 2024-03-27T21:31:50.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-27T21:31:50.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T21:31:50.369 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-03-27T21:31:50.889 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-27T21:31:50.889 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-27T21:31:50.889 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-03-27T21:31:50.903 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-432.el9.x86_64 2024-03-27T21:31:50.903 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-27T21:31:50.903 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-27T21:31:50.903 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-27T21:31:51.904 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-27T21:31:51.904 DEBUG:teuthology.orchestra.run.smithi191:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-27T21:31:51.982 INFO:teuthology.orchestra.run.smithi191.stdout:ttyS1 2024-03-27T21:31:52.013 DEBUG:teuthology.parallel:result is None 2024-03-27T21:31:53.363 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-03-27T21:31:53.364 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T21:31:53.701 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-03-27T21:31:54.103 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-03-27T21:31:54.103 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-27T21:31:54.103 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-03-27T21:31:54.120 INFO:teuthology.orchestra.run.smithi060.stdout:5.14.0-432.el9.x86_64 2024-03-27T21:31:54.121 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-27T21:31:54.121 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-27T21:31:54.121 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-27T21:31:55.122 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-27T21:31:55.122 DEBUG:teuthology.orchestra.run.smithi060:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-27T21:31:55.192 INFO:teuthology.orchestra.run.smithi060.stdout:ttyS1 2024-03-27T21:31:55.240 DEBUG:teuthology.parallel:result is None 2024-03-27T21:31:55.240 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-27T21:31:55.248 INFO:teuthology.task.internal:Creating test directory... 2024-03-27T21:31:55.248 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-27T21:31:55.250 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-27T21:31:55.269 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-27T21:31:55.308 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-27T21:31:55.385 INFO:teuthology.task.internal:Creating archive directory... 2024-03-27T21:31:55.385 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-27T21:31:55.387 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-27T21:31:55.685 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-27T21:31:55.692 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-27T21:31:55.692 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-27T21:31:55.695 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-27T21:31:55.732 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-27T21:31:55.733 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-27T21:31:55.745 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-27T21:31:55.747 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-27T21:31:55.749 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-27T21:31:55.756 INFO:teuthology.task.internal:Configuring sudo... 2024-03-27T21:31:55.756 DEBUG:teuthology.orchestra.run.smithi060:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-27T21:31:55.790 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-27T21:31:55.819 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-27T21:31:55.828 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-27T21:31:55.828 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-27T21:31:55.860 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-27T21:31:55.875 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-27T21:31:55.958 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-27T21:31:56.004 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-27T21:31:56.091 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-27T21:31:56.137 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:31:56.138 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-27T21:31:56.214 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-27T21:31:56.263 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-27T21:31:56.318 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-27T21:31:56.404 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-27T21:31:56.448 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-27T21:31:56.448 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-27T21:31:56.519 DEBUG:teuthology.orchestra.run.smithi060:> sudo service rsyslog restart 2024-03-27T21:31:56.521 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2024-03-27T21:31:56.568 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-27T21:31:56.601 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-27T21:31:56.944 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-27T21:31:56.952 INFO:teuthology.task.internal:Starting timer... 2024-03-27T21:31:56.952 INFO:teuthology.run_tasks:Running task pcp... 2024-03-27T21:31:57.025 INFO:teuthology.run_tasks:Running task selinux... 2024-03-27T21:31:57.068 DEBUG:teuthology.orchestra.run.smithi060:> sudo service auditd rotate 2024-03-27T21:31:57.145 INFO:teuthology.orchestra.run.smithi060.stdout:Rotating logs: 2024-03-27T21:31:57.147 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2024-03-27T21:31:57.213 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: 2024-03-27T21:31:57.215 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-27T21:31:57.215 DEBUG:teuthology.orchestra.run.smithi060:> /usr/sbin/getenforce 2024-03-27T21:31:57.239 INFO:teuthology.orchestra.run.smithi060.stdout:Permissive 2024-03-27T21:31:57.240 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2024-03-27T21:31:57.272 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2024-03-27T21:31:57.273 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi060.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi191.front.sepia.ceph.com': 'permissive'} 2024-03-27T21:31:57.273 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-27T21:31:57.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T21:31:57.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-27T21:31:57.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T21:31:57.342 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-27T21:31:57.342 DEBUG:teuthology.orchestra.run.smithi060:> sudo /usr/sbin/setenforce permissive 2024-03-27T21:31:57.383 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2024-03-27T21:31:57.415 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-27T21:31:57.426 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-27T21:31:57.444 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-27T21:31:57.445 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi060.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-27T21:36:34.545 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi060.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2024-03-27T21:36:34.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-03-27T21:36:34.548 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T21:36:34.643 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-03-27T21:36:34.726 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-03-27T21:36:34.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-27T21:36:34.727 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T21:36:34.816 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-03-27T21:36:34.897 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-27T21:36:34.897 INFO:teuthology.run_tasks:Running task clock... 2024-03-27T21:36:34.908 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-27T21:36:34.909 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-27T21:36:34.909 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-27T21:36:34.912 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-27T21:36:34.912 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-27T21:36:34.945 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-27T21:36:34.963 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-27T21:36:34.968 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-27T21:36:34.985 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-27T21:36:34.994 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: ntpd: command not found 2024-03-27T21:36:35.009 INFO:teuthology.orchestra.run.smithi060.stdout:506 Cannot talk to daemon 2024-03-27T21:36:35.014 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2024-03-27T21:36:35.026 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-27T21:36:35.027 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2024-03-27T21:36:35.043 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-27T21:36:35.044 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-27T21:36:35.060 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-27T21:36:35.094 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-03-27T21:36:35.097 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-27T21:36:35.097 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-03-27T21:36:35.097 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T21:36:35.097 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T21:36:35.097 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T21:36:35.098 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T21:36:35.111 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-03-27T21:36:35.114 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-27T21:36:35.115 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-03-27T21:36:35.115 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T21:36:35.115 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T21:36:35.115 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T21:36:35.115 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T21:36:35.116 INFO:teuthology.run_tasks:Running task install... 2024-03-27T21:36:35.125 DEBUG:teuthology.task.install:project ceph 2024-03-27T21:36:35.125 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7c37d80f7a88f4eb561c4fe02558660a650b10ac'}} 2024-03-27T21:36:35.125 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '7c37d80f7a88f4eb561c4fe02558660a650b10ac'} 2024-03-27T21:36:35.126 INFO:teuthology.task.install:Using flavor: default 2024-03-27T21:36:35.131 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-27T21:36:35.131 INFO:teuthology.task.install:extra packages: [] 2024-03-27T21:36:35.132 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7c37d80f7a88f4eb561c4fe02558660a650b10ac', 'tag': None, 'wait_for_package': False} 2024-03-27T21:36:35.132 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7c37d80f7a88f4eb561c4fe02558660a650b10ac 2024-03-27T21:36:35.135 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7c37d80f7a88f4eb561c4fe02558660a650b10ac', 'tag': None, 'wait_for_package': False} 2024-03-27T21:36:35.135 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7c37d80f7a88f4eb561c4fe02558660a650b10ac 2024-03-27T21:36:35.279 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-03-26-1132-reef/7c37d80f7a88f4eb561c4fe02558660a650b10ac/centos/9/flavors/default/ 2024-03-27T21:36:35.280 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1021.g7c37d80f 2024-03-27T21:36:35.364 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-03-26-1132-reef/7c37d80f7a88f4eb561c4fe02558660a650b10ac/centos/9/flavors/default/ 2024-03-27T21:36:35.364 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1021.g7c37d80f 2024-03-27T21:36:35.461 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-03-26-1132-reef/7c37d80f7a88f4eb561c4fe02558660a650b10ac/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-03-26-1132-reef/7c37d80f7a88f4eb561c4fe02558660a650b10ac/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-03-26-1132-reef/7c37d80f7a88f4eb561c4fe02558660a650b10ac/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-27T21:36:35.461 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:36:35.461 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-27T21:36:35.490 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-27T21:36:35.491 DEBUG:teuthology.orchestra.run.smithi060:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7c37d80f7a88f4eb561c4fe02558660a650b10ac/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-27T21:36:35.531 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-03-26-1132-reef/7c37d80f7a88f4eb561c4fe02558660a650b10ac/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-03-26-1132-reef/7c37d80f7a88f4eb561c4fe02558660a650b10ac/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-03-26-1132-reef/7c37d80f7a88f4eb561c4fe02558660a650b10ac/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-27T21:36:35.532 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-27T21:36:35.532 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-27T21:36:35.560 DEBUG:teuthology.orchestra.run.smithi060:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-27T21:36:35.561 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-27T21:36:35.561 DEBUG:teuthology.orchestra.run.smithi191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7c37d80f7a88f4eb561c4fe02558660a650b10ac/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-27T21:36:35.631 DEBUG:teuthology.orchestra.run.smithi191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-27T21:36:35.641 DEBUG:teuthology.orchestra.run.smithi060:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-27T21:36:35.708 INFO:teuthology.orchestra.run.smithi060.stdout:check_obsoletes = 1 2024-03-27T21:36:35.709 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum clean all 2024-03-27T21:36:35.710 DEBUG:teuthology.orchestra.run.smithi191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-27T21:36:35.777 INFO:teuthology.orchestra.run.smithi191.stdout:check_obsoletes = 1 2024-03-27T21:36:35.778 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum clean all 2024-03-27T21:36:35.959 INFO:teuthology.orchestra.run.smithi060.stdout:74 files removed 2024-03-27T21:36:35.990 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-radosgw 2024-03-27T21:36:36.022 INFO:teuthology.orchestra.run.smithi191.stdout:74 files removed 2024-03-27T21:36:36.049 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-radosgw 2024-03-27T21:36:36.693 INFO:teuthology.orchestra.run.smithi191.stdout:ceph packages for x86_64 224 kB/s | 77 kB 00:00 2024-03-27T21:36:36.714 INFO:teuthology.orchestra.run.smithi060.stdout:ceph packages for x86_64 185 kB/s | 77 kB 00:00 2024-03-27T21:36:37.061 INFO:teuthology.orchestra.run.smithi191.stdout:ceph noarch packages 31 kB/s | 9.9 kB 00:00 2024-03-27T21:36:37.081 INFO:teuthology.orchestra.run.smithi060.stdout:ceph noarch packages 31 kB/s | 9.9 kB 00:00 2024-03-27T21:36:37.415 INFO:teuthology.orchestra.run.smithi060.stdout:ceph source packages 6.3 kB/s | 1.8 kB 00:00 2024-03-27T21:36:37.437 INFO:teuthology.orchestra.run.smithi191.stdout:ceph source packages 5.5 kB/s | 1.8 kB 00:00 2024-03-27T21:36:37.885 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-27T21:36:37.998 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-27T21:36:39.493 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2024-03-27T21:36:39.624 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-27T21:36:40.459 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-27T21:36:42.040 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.0 MB 00:02 2024-03-27T21:36:42.993 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-03-27T21:36:43.691 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 31 MB/s | 19 MB 00:00 2024-03-27T21:36:47.819 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 5.2 MB/s | 6.1 MB 00:01 2024-03-27T21:36:49.262 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 16 kB 00:00 2024-03-27T21:36:50.980 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-27T21:36:55.042 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 803 kB/s | 6.1 MB 00:07 2024-03-27T21:36:56.047 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-27T21:36:56.651 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-27T21:36:58.322 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-27T21:36:58.641 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:36:58.645 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:36:58.645 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-03-27T21:36:58.645 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:36:58.645 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-27T21:36:58.645 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 7.7 M 2024-03-27T21:36:58.645 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrading: 2024-03-27T21:36:58.645 INFO:teuthology.orchestra.run.smithi060.stdout: librados2 x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 3.3 M 2024-03-27T21:36:58.645 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1 x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 3.0 M 2024-03-27T21:36:58.645 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-27T21:36:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options 2024-03-27T21:36:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-27T21:36:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 5.1 M 2024-03-27T21:36:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 18 M 2024-03-27T21:36:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 25 k 2024-03-27T21:36:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-27T21:36:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-27T21:36:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-27T21:36:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-27T21:36:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2 x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 702 k 2024-03-27T21:36:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-27T21:36:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-27T21:36:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-27T21:36:58.647 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1 x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 474 k 2024-03-27T21:36:58.647 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-27T21:36:58.647 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2 x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 4.5 M 2024-03-27T21:36:58.647 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-27T21:36:58.647 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-27T21:36:58.647 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-27T21:36:58.647 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse 2024-03-27T21:36:58.647 INFO:teuthology.orchestra.run.smithi060.stdout: x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 45 k 2024-03-27T21:36:58.647 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 129 k 2024-03-27T21:36:58.647 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 162 k 2024-03-27T21:36:58.647 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-27T21:36:58.647 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 321 k 2024-03-27T21:36:58.647 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 298 k 2024-03-27T21:36:58.648 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 99 k 2024-03-27T21:36:58.648 INFO:teuthology.orchestra.run.smithi060.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-27T21:36:58.648 INFO:teuthology.orchestra.run.smithi060.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-27T21:36:58.648 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:36:58.648 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-27T21:36:58.648 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:36:58.648 INFO:teuthology.orchestra.run.smithi060.stdout:Install 28 Packages 2024-03-27T21:36:58.648 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrade 2 Packages 2024-03-27T21:36:58.648 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:36:58.649 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 53 M 2024-03-27T21:36:58.649 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-27T21:36:59.498 INFO:teuthology.orchestra.run.smithi060.stdout:(1/30): ceph-radosgw-18.2.2-1021.g7c37d80f.el9. 11 MB/s | 7.7 MB 00:00 2024-03-27T21:36:59.557 INFO:teuthology.orchestra.run.smithi060.stdout:(2/30): ceph-selinux-18.2.2-1021.g7c37d80f.el9. 432 kB/s | 25 kB 00:00 2024-03-27T21:36:59.632 INFO:teuthology.orchestra.run.smithi060.stdout:(3/30): libcephfs2-18.2.2-1021.g7c37d80f.el9.x8 9.2 MB/s | 702 kB 00:00 2024-03-27T21:36:59.690 INFO:teuthology.orchestra.run.smithi060.stdout:(4/30): libradosstriper1-18.2.2-1021.g7c37d80f. 7.9 MB/s | 474 kB 00:00 2024-03-27T21:36:59.874 INFO:teuthology.orchestra.run.smithi060.stdout:(5/30): ceph-base-18.2.2-1021.g7c37d80f.el9.x86 4.7 MB/s | 5.1 MB 00:01 2024-03-27T21:36:59.966 INFO:teuthology.orchestra.run.smithi060.stdout:(6/30): librgw2-18.2.2-1021.g7c37d80f.el9.x86_6 16 MB/s | 4.5 MB 00:00 2024-03-27T21:36:59.991 INFO:teuthology.orchestra.run.smithi060.stdout:(7/30): python3-ceph-argparse-18.2.2-1021.g7c37 385 kB/s | 45 kB 00:00 2024-03-27T21:37:00.017 INFO:teuthology.orchestra.run.smithi060.stdout:(8/30): python3-ceph-common-18.2.2-1021.g7c37d8 2.5 MB/s | 129 kB 00:00 2024-03-27T21:37:00.050 INFO:teuthology.orchestra.run.smithi060.stdout:(9/30): python3-cephfs-18.2.2-1021.g7c37d80f.el 2.7 MB/s | 162 kB 00:00 2024-03-27T21:37:00.084 INFO:teuthology.orchestra.run.smithi060.stdout:(10/30): python3-rados-18.2.2-1021.g7c37d80f.el 4.7 MB/s | 321 kB 00:00 2024-03-27T21:37:00.117 INFO:teuthology.orchestra.run.smithi060.stdout:(11/30): python3-rbd-18.2.2-1021.g7c37d80f.el9. 4.4 MB/s | 298 kB 00:00 2024-03-27T21:37:00.142 INFO:teuthology.orchestra.run.smithi060.stdout:(12/30): python3-rgw-18.2.2-1021.g7c37d80f.el9. 1.7 MB/s | 99 kB 00:00 2024-03-27T21:37:00.276 INFO:teuthology.orchestra.run.smithi060.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 667 kB/s | 106 kB 00:00 2024-03-27T21:37:00.318 INFO:teuthology.orchestra.run.smithi060.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 907 kB/s | 159 kB 00:00 2024-03-27T21:37:00.343 INFO:teuthology.orchestra.run.smithi060.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 660 kB/s | 44 kB 00:00 2024-03-27T21:37:00.410 INFO:teuthology.orchestra.run.smithi060.stdout:(16/30): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-27T21:37:00.436 INFO:teuthology.orchestra.run.smithi060.stdout:(17/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-27T21:37:00.586 INFO:teuthology.orchestra.run.smithi060.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 235 kB/s | 41 kB 00:00 2024-03-27T21:37:00.994 INFO:teuthology.orchestra.run.smithi060.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 198 kB/s | 111 kB 00:00 2024-03-27T21:37:01.187 INFO:teuthology.orchestra.run.smithi060.stdout:(20/30): ceph-common-18.2.2-1021.g7c37d80f.el9. 7.7 MB/s | 18 MB 00:02 2024-03-27T21:37:01.212 INFO:teuthology.orchestra.run.smithi060.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 494 kB/s | 309 kB 00:00 2024-03-27T21:37:01.321 INFO:teuthology.orchestra.run.smithi060.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-03-27T21:37:01.446 INFO:teuthology.orchestra.run.smithi060.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 540 kB/s | 67 kB 00:00 2024-03-27T21:37:01.463 INFO:teuthology.orchestra.run.smithi060.stdout:(24/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 94 kB/s | 26 kB 00:00 2024-03-27T21:37:01.621 INFO:teuthology.orchestra.run.smithi060.stdout:(25/30): libarrow-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 4.4 MB 00:00 2024-03-27T21:37:01.722 INFO:teuthology.orchestra.run.smithi060.stdout:(26/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.0 MB/s | 838 kB 00:00 2024-03-27T21:37:01.780 INFO:teuthology.orchestra.run.smithi060.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-03-27T21:37:01.814 INFO:teuthology.orchestra.run.smithi060.stdout:(28/30): re2-20211101-3.el9.x86_64.rpm 546 kB/s | 192 kB 00:00 2024-03-27T21:37:02.072 INFO:teuthology.orchestra.run.smithi060.stdout:(29/30): librbd1-18.2.2-1021.g7c37d80f.el9.x86_ 10 MB/s | 3.0 MB 00:00 2024-03-27T21:37:02.264 INFO:teuthology.orchestra.run.smithi060.stdout:(30/30): librados2-18.2.2-1021.g7c37d80f.el9.x8 6.0 MB/s | 3.3 MB 00:00 2024-03-27T21:37:02.269 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:37:02.269 INFO:teuthology.orchestra.run.smithi060.stdout:Total 15 MB/s | 53 MB 00:03 2024-03-27T21:37:02.269 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-27T21:37:02.309 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-27T21:37:02.309 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-27T21:37:02.780 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-27T21:37:02.780 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-27T21:37:03.220 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-27T21:37:04.305 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-27T21:37:04.383 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-03-27T21:37:04.410 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-03-27T21:37:04.461 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-03-27T21:37:04.899 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-03-27T21:37:04.985 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librados2-2:18.2.2-1021.g7c37d80f.el9.x86_64 5/32 2024-03-27T21:37:05.076 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:18.2.2-1021.g7c37d80f.el9.x86_64 5/32 2024-03-27T21:37:05.163 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rados-2:18.2.2-1021.g7c37d80f.el9.x86_64 6/32 2024-03-27T21:37:05.195 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs2-2:18.2.2-1021.g7c37d80f.el9.x86_64 7/32 2024-03-27T21:37:05.257 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephfs2-2:18.2.2-1021.g7c37d80f.el9.x86_64 7/32 2024-03-27T21:37:05.286 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-03-27T21:37:05.330 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-argparse-2:18.2.2-1021.g7c37d80f.el9. 9/32 2024-03-27T21:37:05.397 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cephfs-2:18.2.2-1021.g7c37d80f.el9.x86_64 10/32 2024-03-27T21:37:05.422 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libradosstriper1-2:18.2.2-1021.g7c37d80f.el9.x86_6 11/32 2024-03-27T21:37:05.470 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1021.g7c37d80f.el9.x86_6 11/32 2024-03-27T21:37:05.581 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-03-27T21:37:05.613 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-03-27T21:37:05.653 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-03-27T21:37:05.686 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-03-27T21:37:05.709 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:37:05.713 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-03-27T21:37:05.721 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:37:05.721 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-27T21:37:05.721 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:37:05.721 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-27T21:37:05.721 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 7.7 M 2024-03-27T21:37:05.721 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrading: 2024-03-27T21:37:05.721 INFO:teuthology.orchestra.run.smithi191.stdout: librados2 x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 3.3 M 2024-03-27T21:37:05.721 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1 x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 3.0 M 2024-03-27T21:37:05.721 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-27T21:37:05.721 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options 2024-03-27T21:37:05.721 INFO:teuthology.orchestra.run.smithi191.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-27T21:37:05.722 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 5.1 M 2024-03-27T21:37:05.722 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 18 M 2024-03-27T21:37:05.722 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 25 k 2024-03-27T21:37:05.722 INFO:teuthology.orchestra.run.smithi191.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-27T21:37:05.722 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-27T21:37:05.722 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-27T21:37:05.722 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-27T21:37:05.722 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2 x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 702 k 2024-03-27T21:37:05.722 INFO:teuthology.orchestra.run.smithi191.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-27T21:37:05.722 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-27T21:37:05.722 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-27T21:37:05.722 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1 x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 474 k 2024-03-27T21:37:05.722 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-27T21:37:05.723 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2 x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 4.5 M 2024-03-27T21:37:05.723 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-27T21:37:05.723 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-27T21:37:05.723 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-27T21:37:05.723 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse 2024-03-27T21:37:05.723 INFO:teuthology.orchestra.run.smithi191.stdout: x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 45 k 2024-03-27T21:37:05.723 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 129 k 2024-03-27T21:37:05.723 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 162 k 2024-03-27T21:37:05.723 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-27T21:37:05.723 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 321 k 2024-03-27T21:37:05.723 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 298 k 2024-03-27T21:37:05.723 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 99 k 2024-03-27T21:37:05.723 INFO:teuthology.orchestra.run.smithi191.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-27T21:37:05.723 INFO:teuthology.orchestra.run.smithi191.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-27T21:37:05.724 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:05.724 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-27T21:37:05.724 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:37:05.724 INFO:teuthology.orchestra.run.smithi191.stdout:Install 28 Packages 2024-03-27T21:37:05.724 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrade 2 Packages 2024-03-27T21:37:05.724 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:05.724 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 53 M 2024-03-27T21:37:05.724 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-27T21:37:05.749 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-03-27T21:37:05.781 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-03-27T21:37:06.096 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-03-27T21:37:06.128 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librbd1-2:18.2.2-1021.g7c37d80f.el9.x86_64 20/32 2024-03-27T21:37:06.221 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:18.2.2-1021.g7c37d80f.el9.x86_64 20/32 2024-03-27T21:37:06.249 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rbd-2:18.2.2-1021.g7c37d80f.el9.x86_64 21/32 2024-03-27T21:37:06.294 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-03-27T21:37:06.717 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-03-27T21:37:06.746 INFO:teuthology.orchestra.run.smithi191.stdout:(1/30): ceph-base-18.2.2-1021.g7c37d80f.el9.x86 7.8 MB/s | 5.1 MB 00:00 2024-03-27T21:37:06.747 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librgw2-2:18.2.2-1021.g7c37d80f.el9.x86_64 24/32 2024-03-27T21:37:06.818 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librgw2-2:18.2.2-1021.g7c37d80f.el9.x86_64 24/32 2024-03-27T21:37:06.872 INFO:teuthology.orchestra.run.smithi191.stdout:(2/30): ceph-radosgw-18.2.2-1021.g7c37d80f.el9. 9.9 MB/s | 7.7 MB 00:00 2024-03-27T21:37:06.889 INFO:teuthology.orchestra.run.smithi191.stdout:(3/30): ceph-selinux-18.2.2-1021.g7c37d80f.el9. 177 kB/s | 25 kB 00:00 2024-03-27T21:37:06.889 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rgw-2:18.2.2-1021.g7c37d80f.el9.x86_64 25/32 2024-03-27T21:37:06.916 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-common-2:18.2.2-1021.g7c37d80f.el9.x8 26/32 2024-03-27T21:37:06.939 INFO:teuthology.orchestra.run.smithi191.stdout:(4/30): libcephfs2-18.2.2-1021.g7c37d80f.el9.x8 10 MB/s | 702 kB 00:00 2024-03-27T21:37:06.964 INFO:teuthology.orchestra.run.smithi191.stdout:(5/30): libradosstriper1-18.2.2-1021.g7c37d80f. 6.1 MB/s | 474 kB 00:00 2024-03-27T21:37:07.032 INFO:teuthology.orchestra.run.smithi191.stdout:(6/30): python3-ceph-argparse-18.2.2-1021.g7c37 674 kB/s | 45 kB 00:00 2024-03-27T21:37:07.107 INFO:teuthology.orchestra.run.smithi191.stdout:(7/30): python3-ceph-common-18.2.2-1021.g7c37d8 1.7 MB/s | 129 kB 00:00 2024-03-27T21:37:07.191 INFO:teuthology.orchestra.run.smithi191.stdout:(8/30): librgw2-18.2.2-1021.g7c37d80f.el9.x86_6 18 MB/s | 4.5 MB 00:00 2024-03-27T21:37:07.225 INFO:teuthology.orchestra.run.smithi191.stdout:(9/30): python3-cephfs-18.2.2-1021.g7c37d80f.el 1.3 MB/s | 162 kB 00:00 2024-03-27T21:37:07.250 INFO:teuthology.orchestra.run.smithi191.stdout:(10/30): python3-rados-18.2.2-1021.g7c37d80f.el 5.3 MB/s | 321 kB 00:00 2024-03-27T21:37:07.275 INFO:teuthology.orchestra.run.smithi191.stdout:(11/30): python3-rbd-18.2.2-1021.g7c37d80f.el9. 5.8 MB/s | 298 kB 00:00 2024-03-27T21:37:07.293 INFO:teuthology.orchestra.run.smithi191.stdout:(12/30): python3-rgw-18.2.2-1021.g7c37d80f.el9. 2.3 MB/s | 99 kB 00:00 2024-03-27T21:37:07.585 INFO:teuthology.orchestra.run.smithi191.stdout:(13/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 545 kB/s | 159 kB 00:00 2024-03-27T21:37:07.626 INFO:teuthology.orchestra.run.smithi191.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 301 kB/s | 106 kB 00:00 2024-03-27T21:37:07.653 INFO:teuthology.orchestra.run.smithi191.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 656 kB/s | 44 kB 00:00 2024-03-27T21:37:07.811 INFO:teuthology.orchestra.run.smithi191.stdout:(16/30): librdkafka-1.6.1-102.el9.x86_64.rpm 3.5 MB/s | 661 kB 00:00 2024-03-27T21:37:07.837 INFO:teuthology.orchestra.run.smithi191.stdout:(17/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.5 MB/s | 282 kB 00:00 2024-03-27T21:37:07.863 INFO:teuthology.orchestra.run.smithi191.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 821 kB/s | 41 kB 00:00 2024-03-27T21:37:07.970 INFO:teuthology.orchestra.run.smithi191.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 828 kB/s | 111 kB 00:00 2024-03-27T21:37:08.020 INFO:teuthology.orchestra.run.smithi191.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.9 MB/s | 309 kB 00:00 2024-03-27T21:37:08.065 INFO:teuthology.orchestra.run.smithi191.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 622 kB/s | 26 kB 00:00 2024-03-27T21:37:08.121 INFO:teuthology.orchestra.run.smithi191.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 829 kB/s | 49 kB 00:00 2024-03-27T21:37:08.180 INFO:teuthology.orchestra.run.smithi191.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-27T21:37:08.247 INFO:teuthology.orchestra.run.smithi191.stdout:(24/30): libarrow-9.0.0-10.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-03-27T21:37:08.281 INFO:teuthology.orchestra.run.smithi191.stdout:(25/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.1 MB/s | 838 kB 00:00 2024-03-27T21:37:08.306 INFO:teuthology.orchestra.run.smithi191.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-27T21:37:08.365 INFO:teuthology.orchestra.run.smithi191.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-27T21:37:08.682 INFO:teuthology.orchestra.run.smithi191.stdout:(28/30): librados2-18.2.2-1021.g7c37d80f.el9.x8 8.7 MB/s | 3.3 MB 00:00 2024-03-27T21:37:08.774 INFO:teuthology.orchestra.run.smithi191.stdout:(29/30): librbd1-18.2.2-1021.g7c37d80f.el9.x86_ 7.3 MB/s | 3.0 MB 00:00 2024-03-27T21:37:09.029 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:18.2.2-1021.g7c37d80f.el9.x86_64 27/32 2024-03-27T21:37:09.085 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-common-2:18.2.2-1021.g7c37d80f.el9.x86_64 27/32 2024-03-27T21:37:09.100 INFO:teuthology.orchestra.run.smithi191.stdout:(30/30): ceph-common-18.2.2-1021.g7c37d80f.el9. 6.1 MB/s | 18 MB 00:03 2024-03-27T21:37:09.104 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:37:09.104 INFO:teuthology.orchestra.run.smithi191.stdout:Total 16 MB/s | 53 MB 00:03 2024-03-27T21:37:09.104 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-27T21:37:09.137 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-27T21:37:09.138 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-27T21:37:09.585 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-27T21:37:09.586 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-27T21:37:09.638 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:18.2.2-1021.g7c37d80f.el9.x86_64 27/32 2024-03-27T21:37:09.683 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-base-2:18.2.2-1021.g7c37d80f.el9.x86_64 28/32 2024-03-27T21:37:09.776 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-base-2:18.2.2-1021.g7c37d80f.el9.x86_64 28/32 2024-03-27T21:37:09.776 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-27T21:37:09.776 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-27T21:37:09.776 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:09.809 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-selinux-2:18.2.2-1021.g7c37d80f.el9.x86_64 29/32 2024-03-27T21:37:10.959 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-27T21:37:11.052 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-03-27T21:37:11.080 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-03-27T21:37:11.130 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-03-27T21:37:11.448 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-03-27T21:37:11.513 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librados2-2:18.2.2-1021.g7c37d80f.el9.x86_64 5/32 2024-03-27T21:37:11.587 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:18.2.2-1021.g7c37d80f.el9.x86_64 5/32 2024-03-27T21:37:11.672 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rados-2:18.2.2-1021.g7c37d80f.el9.x86_64 6/32 2024-03-27T21:37:11.707 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs2-2:18.2.2-1021.g7c37d80f.el9.x86_64 7/32 2024-03-27T21:37:11.794 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs2-2:18.2.2-1021.g7c37d80f.el9.x86_64 7/32 2024-03-27T21:37:11.882 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-03-27T21:37:12.002 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-argparse-2:18.2.2-1021.g7c37d80f.el9. 9/32 2024-03-27T21:37:12.067 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cephfs-2:18.2.2-1021.g7c37d80f.el9.x86_64 10/32 2024-03-27T21:37:12.102 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libradosstriper1-2:18.2.2-1021.g7c37d80f.el9.x86_6 11/32 2024-03-27T21:37:12.168 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1021.g7c37d80f.el9.x86_6 11/32 2024-03-27T21:37:12.282 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-03-27T21:37:12.319 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-03-27T21:37:12.359 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-03-27T21:37:12.401 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-03-27T21:37:12.428 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-03-27T21:37:12.464 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-03-27T21:37:12.497 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-03-27T21:37:12.814 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-03-27T21:37:12.844 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librbd1-2:18.2.2-1021.g7c37d80f.el9.x86_64 20/32 2024-03-27T21:37:12.918 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:18.2.2-1021.g7c37d80f.el9.x86_64 20/32 2024-03-27T21:37:12.949 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rbd-2:18.2.2-1021.g7c37d80f.el9.x86_64 21/32 2024-03-27T21:37:12.990 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-03-27T21:37:13.393 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-03-27T21:37:13.422 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librgw2-2:18.2.2-1021.g7c37d80f.el9.x86_64 24/32 2024-03-27T21:37:13.477 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librgw2-2:18.2.2-1021.g7c37d80f.el9.x86_64 24/32 2024-03-27T21:37:13.554 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rgw-2:18.2.2-1021.g7c37d80f.el9.x86_64 25/32 2024-03-27T21:37:13.584 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-common-2:18.2.2-1021.g7c37d80f.el9.x8 26/32 2024-03-27T21:37:15.663 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:18.2.2-1021.g7c37d80f.el9.x86_64 27/32 2024-03-27T21:37:15.691 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-common-2:18.2.2-1021.g7c37d80f.el9.x86_64 27/32 2024-03-27T21:37:16.231 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:18.2.2-1021.g7c37d80f.el9.x86_64 27/32 2024-03-27T21:37:16.271 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-base-2:18.2.2-1021.g7c37d80f.el9.x86_64 28/32 2024-03-27T21:37:16.355 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-base-2:18.2.2-1021.g7c37d80f.el9.x86_64 28/32 2024-03-27T21:37:16.355 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-27T21:37:16.355 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-27T21:37:16.355 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:16.398 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-selinux-2:18.2.2-1021.g7c37d80f.el9.x86_64 29/32 2024-03-27T21:37:19.878 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1021.g7c37d80f.el9.x86_64 29/32 2024-03-27T21:37:19.878 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /sys 2024-03-27T21:37:19.878 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /proc 2024-03-27T21:37:19.878 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /mnt 2024-03-27T21:37:19.879 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /var/tmp 2024-03-27T21:37:19.879 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /home 2024-03-27T21:37:19.879 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /root 2024-03-27T21:37:19.879 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /tmp 2024-03-27T21:37:19.879 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:20.641 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-radosgw-2:18.2.2-1021.g7c37d80f.el9.x86_64 30/32 2024-03-27T21:37:20.664 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1021.g7c37d80f.el9.x86_64 30/32 2024-03-27T21:37:20.664 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T21:37:20.665 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-27T21:37:20.665 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-27T21:37:20.665 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-27T21:37:20.665 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:20.666 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-27T21:37:20.752 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-27T21:37:20.752 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-27T21:37:21.872 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-27T21:37:21.873 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-base-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/32 2024-03-27T21:37:21.873 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-common-2:18.2.2-1021.g7c37d80f.el9.x86_64 2/32 2024-03-27T21:37:21.873 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-radosgw-2:18.2.2-1021.g7c37d80f.el9.x86_64 3/32 2024-03-27T21:37:21.873 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-selinux-2:18.2.2-1021.g7c37d80f.el9.x86_64 4/32 2024-03-27T21:37:21.873 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs2-2:18.2.2-1021.g7c37d80f.el9.x86_64 5/32 2024-03-27T21:37:21.873 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libradosstriper1-2:18.2.2-1021.g7c37d80f.el9.x86_6 6/32 2024-03-27T21:37:21.873 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librgw2-2:18.2.2-1021.g7c37d80f.el9.x86_64 7/32 2024-03-27T21:37:21.873 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1021.g7c37d80f.el9. 8/32 2024-03-27T21:37:21.873 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-common-2:18.2.2-1021.g7c37d80f.el9.x8 9/32 2024-03-27T21:37:21.873 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cephfs-2:18.2.2-1021.g7c37d80f.el9.x86_64 10/32 2024-03-27T21:37:21.873 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rados-2:18.2.2-1021.g7c37d80f.el9.x86_64 11/32 2024-03-27T21:37:21.873 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rbd-2:18.2.2-1021.g7c37d80f.el9.x86_64 12/32 2024-03-27T21:37:21.873 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rgw-2:18.2.2-1021.g7c37d80f.el9.x86_64 13/32 2024-03-27T21:37:21.874 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-03-27T21:37:21.874 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-03-27T21:37:21.874 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-03-27T21:37:21.874 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-03-27T21:37:21.874 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-03-27T21:37:21.874 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-03-27T21:37:21.874 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-03-27T21:37:21.874 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-03-27T21:37:21.874 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-03-27T21:37:21.874 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-03-27T21:37:21.874 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-03-27T21:37:21.874 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-03-27T21:37:21.874 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-03-27T21:37:21.874 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-03-27T21:37:21.875 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-03-27T21:37:21.875 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:18.2.2-1021.g7c37d80f.el9.x86_64 29/32 2024-03-27T21:37:21.875 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-03-27T21:37:21.876 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:18.2.2-1021.g7c37d80f.el9.x86_64 31/32 2024-03-27T21:37:22.844 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-03-27T21:37:22.844 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:22.844 INFO:teuthology.orchestra.run.smithi060.stdout:Upgraded: 2024-03-27T21:37:22.844 INFO:teuthology.orchestra.run.smithi060.stdout: librados2-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:22.844 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:22.844 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-27T21:37:22.844 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-27T21:37:22.844 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:22.844 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:22.844 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:22.844 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:22.845 INFO:teuthology.orchestra.run.smithi060.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-27T21:37:22.845 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-27T21:37:22.845 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-27T21:37:22.845 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-27T21:37:22.845 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:22.845 INFO:teuthology.orchestra.run.smithi060.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-27T21:37:22.845 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-27T21:37:22.845 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-27T21:37:22.845 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:22.845 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-27T21:37:22.845 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:22.845 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-27T21:37:22.845 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-27T21:37:22.845 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-27T21:37:22.845 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:22.846 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:22.846 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:22.846 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-27T21:37:22.846 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:22.846 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:22.846 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:22.846 INFO:teuthology.orchestra.run.smithi060.stdout: re2-1:20211101-3.el9.x86_64 2024-03-27T21:37:22.846 INFO:teuthology.orchestra.run.smithi060.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-27T21:37:22.846 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:22.846 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:37:23.166 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-test 2024-03-27T21:37:23.737 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:27 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:37:23.906 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:37:23.906 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:37:23.906 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-03-27T21:37:23.906 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:37:23.907 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-27T21:37:23.907 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 40 M 2024-03-27T21:37:23.907 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-27T21:37:23.907 INFO:teuthology.orchestra.run.smithi060.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-27T21:37:23.907 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 166 k 2024-03-27T21:37:23.907 INFO:teuthology.orchestra.run.smithi060.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-27T21:37:23.907 INFO:teuthology.orchestra.run.smithi060.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-27T21:37:23.907 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-27T21:37:23.907 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:23.907 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-27T21:37:23.907 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:37:23.907 INFO:teuthology.orchestra.run.smithi060.stdout:Install 6 Packages 2024-03-27T21:37:23.907 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:23.908 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 41 M 2024-03-27T21:37:23.908 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 165 M 2024-03-27T21:37:23.908 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-27T21:37:24.344 INFO:teuthology.orchestra.run.smithi060.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-27T21:37:24.386 INFO:teuthology.orchestra.run.smithi060.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-03-27T21:37:24.420 INFO:teuthology.orchestra.run.smithi060.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-27T21:37:24.553 INFO:teuthology.orchestra.run.smithi060.stdout:(4/6): libcephsqlite-18.2.2-1021.g7c37d80f.el9. 445 kB/s | 166 kB 00:00 2024-03-27T21:37:24.578 INFO:teuthology.orchestra.run.smithi060.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-27T21:37:25.751 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1021.g7c37d80f.el9.x86_64 29/32 2024-03-27T21:37:25.752 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /sys 2024-03-27T21:37:25.752 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /proc 2024-03-27T21:37:25.752 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /mnt 2024-03-27T21:37:25.752 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /var/tmp 2024-03-27T21:37:25.752 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /home 2024-03-27T21:37:25.752 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /root 2024-03-27T21:37:25.752 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /tmp 2024-03-27T21:37:25.752 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:26.395 INFO:teuthology.orchestra.run.smithi060.stdout:(6/6): ceph-test-18.2.2-1021.g7c37d80f.el9.x86_ 18 MB/s | 40 MB 00:02 2024-03-27T21:37:26.398 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:37:26.398 INFO:teuthology.orchestra.run.smithi060.stdout:Total 16 MB/s | 41 MB 00:02 2024-03-27T21:37:26.413 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-27T21:37:26.435 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-27T21:37:26.435 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-27T21:37:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-radosgw-2:18.2.2-1021.g7c37d80f.el9.x86_64 30/32 2024-03-27T21:37:26.540 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1021.g7c37d80f.el9.x86_64 30/32 2024-03-27T21:37:26.540 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T21:37:26.540 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-27T21:37:26.540 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-27T21:37:26.540 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-27T21:37:26.540 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:26.542 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-27T21:37:26.625 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-27T21:37:26.625 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-27T21:37:26.795 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-27T21:37:26.795 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-27T21:37:27.346 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-27T21:37:27.422 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-27T21:37:27.455 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-27T21:37:27.479 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-27T21:37:27.521 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-27T21:37:27.548 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephsqlite-2:18.2.2-1021.g7c37d80f.el9.x86_64 5/6 2024-03-27T21:37:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-27T21:37:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-base-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/32 2024-03-27T21:37:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-common-2:18.2.2-1021.g7c37d80f.el9.x86_64 2/32 2024-03-27T21:37:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-radosgw-2:18.2.2-1021.g7c37d80f.el9.x86_64 3/32 2024-03-27T21:37:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-selinux-2:18.2.2-1021.g7c37d80f.el9.x86_64 4/32 2024-03-27T21:37:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs2-2:18.2.2-1021.g7c37d80f.el9.x86_64 5/32 2024-03-27T21:37:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libradosstriper1-2:18.2.2-1021.g7c37d80f.el9.x86_6 6/32 2024-03-27T21:37:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librgw2-2:18.2.2-1021.g7c37d80f.el9.x86_64 7/32 2024-03-27T21:37:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1021.g7c37d80f.el9. 8/32 2024-03-27T21:37:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-common-2:18.2.2-1021.g7c37d80f.el9.x8 9/32 2024-03-27T21:37:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cephfs-2:18.2.2-1021.g7c37d80f.el9.x86_64 10/32 2024-03-27T21:37:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rados-2:18.2.2-1021.g7c37d80f.el9.x86_64 11/32 2024-03-27T21:37:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rbd-2:18.2.2-1021.g7c37d80f.el9.x86_64 12/32 2024-03-27T21:37:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rgw-2:18.2.2-1021.g7c37d80f.el9.x86_64 13/32 2024-03-27T21:37:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-03-27T21:37:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-03-27T21:37:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-03-27T21:37:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-03-27T21:37:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-03-27T21:37:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-03-27T21:37:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-03-27T21:37:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-03-27T21:37:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-03-27T21:37:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-03-27T21:37:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-03-27T21:37:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-03-27T21:37:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-03-27T21:37:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-03-27T21:37:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-03-27T21:37:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:18.2.2-1021.g7c37d80f.el9.x86_64 29/32 2024-03-27T21:37:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-03-27T21:37:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:18.2.2-1021.g7c37d80f.el9.x86_64 31/32 2024-03-27T21:37:28.982 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-03-27T21:37:28.983 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:28.983 INFO:teuthology.orchestra.run.smithi191.stdout:Upgraded: 2024-03-27T21:37:28.983 INFO:teuthology.orchestra.run.smithi191.stdout: librados2-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:28.983 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:28.983 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-27T21:37:28.983 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-27T21:37:28.983 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:28.983 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:28.983 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:28.983 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:28.983 INFO:teuthology.orchestra.run.smithi191.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-27T21:37:28.983 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-27T21:37:28.983 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-27T21:37:28.984 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-27T21:37:28.984 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:28.984 INFO:teuthology.orchestra.run.smithi191.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-27T21:37:28.984 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-27T21:37:28.984 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-27T21:37:28.984 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:28.984 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-27T21:37:28.984 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:28.984 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-27T21:37:28.984 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-27T21:37:28.984 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-27T21:37:28.984 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:28.984 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:28.984 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:28.985 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-27T21:37:28.985 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:28.985 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:28.985 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:28.985 INFO:teuthology.orchestra.run.smithi191.stdout: re2-1:20211101-3.el9.x86_64 2024-03-27T21:37:28.985 INFO:teuthology.orchestra.run.smithi191.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-27T21:37:28.985 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:28.985 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:37:29.327 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-test 2024-03-27T21:37:29.886 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:26 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:37:30.035 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:37:30.036 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:37:30.036 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-27T21:37:30.036 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:37:30.036 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-27T21:37:30.037 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 40 M 2024-03-27T21:37:30.037 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-27T21:37:30.037 INFO:teuthology.orchestra.run.smithi191.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-27T21:37:30.037 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 166 k 2024-03-27T21:37:30.037 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-27T21:37:30.037 INFO:teuthology.orchestra.run.smithi191.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-27T21:37:30.037 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-27T21:37:30.037 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:30.037 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-27T21:37:30.037 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:37:30.037 INFO:teuthology.orchestra.run.smithi191.stdout:Install 6 Packages 2024-03-27T21:37:30.037 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:30.038 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 41 M 2024-03-27T21:37:30.038 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 165 M 2024-03-27T21:37:30.038 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-27T21:37:30.320 INFO:teuthology.orchestra.run.smithi191.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-27T21:37:30.370 INFO:teuthology.orchestra.run.smithi191.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-27T21:37:30.404 INFO:teuthology.orchestra.run.smithi191.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-27T21:37:30.554 INFO:teuthology.orchestra.run.smithi191.stdout:(4/6): libcephsqlite-18.2.2-1021.g7c37d80f.el9. 411 kB/s | 166 kB 00:00 2024-03-27T21:37:30.679 INFO:teuthology.orchestra.run.smithi191.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 680 kB/s | 187 kB 00:00 2024-03-27T21:37:31.496 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1021.g7c37d80f.el9.x86_64 5/6 2024-03-27T21:37:31.576 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-test-2:18.2.2-1021.g7c37d80f.el9.x86_64 6/6 2024-03-27T21:37:33.330 INFO:teuthology.orchestra.run.smithi191.stdout:(6/6): ceph-test-18.2.2-1021.g7c37d80f.el9.x86_ 12 MB/s | 40 MB 00:03 2024-03-27T21:37:33.333 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:37:33.333 INFO:teuthology.orchestra.run.smithi191.stdout:Total 12 MB/s | 41 MB 00:03 2024-03-27T21:37:33.348 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-27T21:37:33.369 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-27T21:37:33.369 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-27T21:37:33.582 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-test-2:18.2.2-1021.g7c37d80f.el9.x86_64 6/6 2024-03-27T21:37:33.583 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-test-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/6 2024-03-27T21:37:33.583 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephsqlite-2:18.2.2-1021.g7c37d80f.el9.x86_64 2/6 2024-03-27T21:37:33.583 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-27T21:37:33.583 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-27T21:37:33.583 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-27T21:37:33.743 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-27T21:37:33.743 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-27T21:37:33.945 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-27T21:37:33.945 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:33.945 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-27T21:37:33.945 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:33.945 INFO:teuthology.orchestra.run.smithi060.stdout: jq-1.6-16.el9.x86_64 2024-03-27T21:37:33.945 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:33.946 INFO:teuthology.orchestra.run.smithi060.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-27T21:37:33.946 INFO:teuthology.orchestra.run.smithi060.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-27T21:37:33.946 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-27T21:37:33.946 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:33.946 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:37:34.231 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph 2024-03-27T21:37:34.272 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-27T21:37:34.347 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-27T21:37:34.381 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-27T21:37:34.413 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-27T21:37:34.455 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-27T21:37:34.492 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephsqlite-2:18.2.2-1021.g7c37d80f.el9.x86_64 5/6 2024-03-27T21:37:34.803 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:38 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:37:34.961 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:37:34.963 INFO:teuthology.orchestra.run.smithi060.stdout:========================================================================================= 2024-03-27T21:37:34.963 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-03-27T21:37:34.964 INFO:teuthology.orchestra.run.smithi060.stdout:========================================================================================= 2024-03-27T21:37:34.964 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-27T21:37:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: ceph x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 6.5 k 2024-03-27T21:37:34.964 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-27T21:37:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 2.1 M 2024-03-27T21:37:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 1.4 M 2024-03-27T21:37:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1021.g7c37d80f.el9 ceph-noarch 245 k 2024-03-27T21:37:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 4.7 M 2024-03-27T21:37:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 16 M 2024-03-27T21:37:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-27T21:37:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-27T21:37:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-27T21:37:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-27T21:37:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-27T21:37:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-27T21:37:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-27T21:37:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-27T21:37:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-27T21:37:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-27T21:37:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-27T21:37:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-27T21:37:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-27T21:37:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-27T21:37:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-27T21:37:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-27T21:37:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-27T21:37:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-27T21:37:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-27T21:37:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-27T21:37:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-27T21:37:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-27T21:37:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-27T21:37:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-27T21:37:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-27T21:37:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-27T21:37:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-27T21:37:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:34.966 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-27T21:37:34.966 INFO:teuthology.orchestra.run.smithi060.stdout:========================================================================================= 2024-03-27T21:37:34.966 INFO:teuthology.orchestra.run.smithi060.stdout:Install 33 Packages 2024-03-27T21:37:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:34.967 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 29 M 2024-03-27T21:37:34.967 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 104 M 2024-03-27T21:37:34.967 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-27T21:37:35.605 INFO:teuthology.orchestra.run.smithi060.stdout:(1/33): ceph-18.2.2-1021.g7c37d80f.el9.x86_64.r 41 kB/s | 6.5 kB 00:00 2024-03-27T21:37:35.856 INFO:teuthology.orchestra.run.smithi060.stdout:(2/33): ceph-mgr-18.2.2-1021.g7c37d80f.el9.x86_ 3.5 MB/s | 1.4 MB 00:00 2024-03-27T21:37:35.906 INFO:teuthology.orchestra.run.smithi060.stdout:(3/33): ceph-mds-18.2.2-1021.g7c37d80f.el9.x86_ 4.6 MB/s | 2.1 MB 00:00 2024-03-27T21:37:35.989 INFO:teuthology.orchestra.run.smithi060.stdout:(4/33): ceph-mgr-modules-core-18.2.2-1021.g7c37 2.9 MB/s | 245 kB 00:00 2024-03-27T21:37:36.106 INFO:teuthology.orchestra.run.smithi060.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 144 kB 00:00 2024-03-27T21:37:36.140 INFO:teuthology.orchestra.run.smithi060.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 954 kB/s | 32 kB 00:00 2024-03-27T21:37:36.182 INFO:teuthology.orchestra.run.smithi060.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-27T21:37:36.224 INFO:teuthology.orchestra.run.smithi060.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 925 kB/s | 39 kB 00:00 2024-03-27T21:37:36.282 INFO:teuthology.orchestra.run.smithi060.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-27T21:37:36.358 INFO:teuthology.orchestra.run.smithi060.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-27T21:37:36.450 INFO:teuthology.orchestra.run.smithi060.stdout:(11/33): ceph-mon-18.2.2-1021.g7c37d80f.el9.x86 5.5 MB/s | 4.7 MB 00:00 2024-03-27T21:37:36.475 INFO:teuthology.orchestra.run.smithi060.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 879 kB/s | 103 kB 00:00 2024-03-27T21:37:36.500 INFO:teuthology.orchestra.run.smithi060.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-27T21:37:36.525 INFO:teuthology.orchestra.run.smithi060.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-27T21:37:36.651 INFO:teuthology.orchestra.run.smithi060.stdout:(15/33): python3-requests-2.25.1-8.el9.noarch.r 832 kB/s | 125 kB 00:00 2024-03-27T21:37:36.818 INFO:teuthology.orchestra.run.smithi060.stdout:(16/33): ceph-osd-18.2.2-1021.g7c37d80f.el9.x86 17 MB/s | 16 MB 00:00 2024-03-27T21:37:36.843 INFO:teuthology.orchestra.run.smithi060.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 137 kB/s | 43 kB 00:00 2024-03-27T21:37:36.927 INFO:teuthology.orchestra.run.smithi060.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-03-27T21:37:36.986 INFO:teuthology.orchestra.run.smithi060.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 304 kB/s | 18 kB 00:00 2024-03-27T21:37:37.011 INFO:teuthology.orchestra.run.smithi060.stdout:(20/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-03-27T21:37:37.036 INFO:teuthology.orchestra.run.smithi060.stdout:(21/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 446 kB/s | 172 kB 00:00 2024-03-27T21:37:37.053 INFO:teuthology.orchestra.run.smithi060.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 345 kB/s | 23 kB 00:00 2024-03-27T21:37:37.070 INFO:teuthology.orchestra.run.smithi060.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 330 kB/s | 19 kB 00:00 2024-03-27T21:37:37.087 INFO:teuthology.orchestra.run.smithi060.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 389 kB/s | 20 kB 00:00 2024-03-27T21:37:37.137 INFO:teuthology.orchestra.run.smithi060.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2024-03-27T21:37:37.162 INFO:teuthology.orchestra.run.smithi060.stdout:(26/33): python3-logutils-0.3.5-21.el9.noarch.r 424 kB/s | 46 kB 00:00 2024-03-27T21:37:37.188 INFO:teuthology.orchestra.run.smithi060.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-03-27T21:37:37.213 INFO:teuthology.orchestra.run.smithi060.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-03-27T21:37:37.246 INFO:teuthology.orchestra.run.smithi060.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 610 kB/s | 36 kB 00:00 2024-03-27T21:37:37.272 INFO:teuthology.orchestra.run.smithi060.stdout:(30/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 824 kB/s | 90 kB 00:00 2024-03-27T21:37:37.297 INFO:teuthology.orchestra.run.smithi060.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-27T21:37:37.331 INFO:teuthology.orchestra.run.smithi060.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-03-27T21:37:37.356 INFO:teuthology.orchestra.run.smithi060.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-03-27T21:37:37.359 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:37:37.359 INFO:teuthology.orchestra.run.smithi060.stdout:Total 12 MB/s | 29 MB 00:02 2024-03-27T21:37:37.373 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-27T21:37:37.416 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-27T21:37:37.417 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-27T21:37:37.771 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-27T21:37:37.772 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-27T21:37:38.224 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1021.g7c37d80f.el9.x86_64 5/6 2024-03-27T21:37:38.299 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-test-2:18.2.2-1021.g7c37d80f.el9.x86_64 6/6 2024-03-27T21:37:39.070 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-27T21:37:39.133 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-27T21:37:39.166 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-27T21:37:39.199 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-27T21:37:39.303 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-27T21:37:39.379 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-27T21:37:39.436 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-27T21:37:39.510 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-27T21:37:39.571 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-27T21:37:39.608 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-27T21:37:39.666 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-27T21:37:39.739 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-27T21:37:39.941 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-27T21:37:39.995 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-27T21:37:40.037 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-27T21:37:40.076 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-27T21:37:40.106 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-27T21:37:40.185 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-27T21:37:40.216 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-27T21:37:40.258 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-27T21:37:40.320 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-27T21:37:40.358 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-27T21:37:40.393 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-27T21:37:40.485 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-27T21:37:40.496 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-test-2:18.2.2-1021.g7c37d80f.el9.x86_64 6/6 2024-03-27T21:37:40.496 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-test-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/6 2024-03-27T21:37:40.496 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephsqlite-2:18.2.2-1021.g7c37d80f.el9.x86_64 2/6 2024-03-27T21:37:40.496 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-27T21:37:40.496 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-27T21:37:40.497 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-27T21:37:40.524 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-27T21:37:40.599 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-27T21:37:40.740 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-27T21:37:40.872 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-27T21:37:40.875 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-27T21:37:40.875 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:40.875 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-27T21:37:40.875 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:40.876 INFO:teuthology.orchestra.run.smithi191.stdout: jq-1.6-16.el9.x86_64 2024-03-27T21:37:40.876 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:40.876 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-27T21:37:40.876 INFO:teuthology.orchestra.run.smithi191.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-27T21:37:40.876 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-27T21:37:40.876 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:40.876 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:37:41.032 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1021.g7c37d80f.el9. 28/33 2024-03-27T21:37:41.062 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-2:18.2.2-1021.g7c37d80f.el9.x86_64 29/33 2024-03-27T21:37:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1021.g7c37d80f.el9.x86_64 29/33 2024-03-27T21:37:41.086 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T21:37:41.086 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-27T21:37:41.086 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-27T21:37:41.086 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-27T21:37:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:41.173 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph 2024-03-27T21:37:41.737 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:38 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:37:41.888 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:37:41.890 INFO:teuthology.orchestra.run.smithi191.stdout:========================================================================================= 2024-03-27T21:37:41.890 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-27T21:37:41.890 INFO:teuthology.orchestra.run.smithi191.stdout:========================================================================================= 2024-03-27T21:37:41.890 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-27T21:37:41.890 INFO:teuthology.orchestra.run.smithi191.stdout: ceph x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 6.5 k 2024-03-27T21:37:41.890 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-27T21:37:41.891 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 2.1 M 2024-03-27T21:37:41.891 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 1.4 M 2024-03-27T21:37:41.891 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1021.g7c37d80f.el9 ceph-noarch 245 k 2024-03-27T21:37:41.891 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 4.7 M 2024-03-27T21:37:41.891 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 16 M 2024-03-27T21:37:41.891 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-27T21:37:41.891 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-27T21:37:41.891 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-27T21:37:41.891 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-27T21:37:41.891 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-27T21:37:41.891 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-27T21:37:41.891 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-27T21:37:41.891 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-27T21:37:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-27T21:37:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-27T21:37:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-27T21:37:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-27T21:37:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-27T21:37:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-27T21:37:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-27T21:37:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-27T21:37:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-27T21:37:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-27T21:37:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-27T21:37:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-27T21:37:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-27T21:37:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-27T21:37:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-27T21:37:41.893 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-27T21:37:41.893 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-27T21:37:41.893 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-27T21:37:41.893 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-27T21:37:41.893 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:41.893 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-27T21:37:41.893 INFO:teuthology.orchestra.run.smithi191.stdout:========================================================================================= 2024-03-27T21:37:41.893 INFO:teuthology.orchestra.run.smithi191.stdout:Install 33 Packages 2024-03-27T21:37:41.893 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:41.894 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 29 M 2024-03-27T21:37:41.894 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 104 M 2024-03-27T21:37:41.894 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-27T21:37:42.507 INFO:teuthology.orchestra.run.smithi191.stdout:(1/33): ceph-18.2.2-1021.g7c37d80f.el9.x86_64.r 42 kB/s | 6.5 kB 00:00 2024-03-27T21:37:42.596 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-osd-2:18.2.2-1021.g7c37d80f.el9.x86_64 30/33 2024-03-27T21:37:42.624 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-osd-2:18.2.2-1021.g7c37d80f.el9.x86_64 30/33 2024-03-27T21:37:42.624 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T21:37:42.624 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-27T21:37:42.624 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-27T21:37:42.624 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-27T21:37:42.624 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:42.749 INFO:teuthology.orchestra.run.smithi191.stdout:(2/33): ceph-mds-18.2.2-1021.g7c37d80f.el9.x86_ 5.3 MB/s | 2.1 MB 00:00 2024-03-27T21:37:42.813 INFO:teuthology.orchestra.run.smithi191.stdout:(3/33): ceph-mgr-18.2.2-1021.g7c37d80f.el9.x86_ 3.1 MB/s | 1.4 MB 00:00 2024-03-27T21:37:42.891 INFO:teuthology.orchestra.run.smithi191.stdout:(4/33): ceph-mgr-modules-core-18.2.2-1021.g7c37 3.1 MB/s | 245 kB 00:00 2024-03-27T21:37:42.999 INFO:teuthology.orchestra.run.smithi191.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 144 kB 00:00 2024-03-27T21:37:43.033 INFO:teuthology.orchestra.run.smithi191.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 950 kB/s | 32 kB 00:00 2024-03-27T21:37:43.066 INFO:teuthology.orchestra.run.smithi191.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-27T21:37:43.089 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mon-2:18.2.2-1021.g7c37d80f.el9.x86_64 31/33 2024-03-27T21:37:43.100 INFO:teuthology.orchestra.run.smithi191.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-27T21:37:43.111 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mon-2:18.2.2-1021.g7c37d80f.el9.x86_64 31/33 2024-03-27T21:37:43.111 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T21:37:43.111 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-27T21:37:43.111 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-27T21:37:43.111 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-27T21:37:43.112 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:43.167 INFO:teuthology.orchestra.run.smithi191.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-27T21:37:43.250 INFO:teuthology.orchestra.run.smithi191.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-27T21:37:43.284 INFO:teuthology.orchestra.run.smithi191.stdout:(11/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-27T21:37:43.318 INFO:teuthology.orchestra.run.smithi191.stdout:(12/33): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-27T21:37:43.331 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mds-2:18.2.2-1021.g7c37d80f.el9.x86_64 32/33 2024-03-27T21:37:43.354 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mds-2:18.2.2-1021.g7c37d80f.el9.x86_64 32/33 2024-03-27T21:37:43.354 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T21:37:43.354 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-27T21:37:43.354 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-27T21:37:43.354 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-27T21:37:43.354 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:43.360 INFO:teuthology.orchestra.run.smithi191.stdout:(13/33): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-27T21:37:43.414 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-2:18.2.2-1021.g7c37d80f.el9.x86_64 33/33 2024-03-27T21:37:43.593 INFO:teuthology.orchestra.run.smithi191.stdout:(14/33): ceph-osd-18.2.2-1021.g7c37d80f.el9.x86 20 MB/s | 16 MB 00:00 2024-03-27T21:37:43.669 INFO:teuthology.orchestra.run.smithi191.stdout:(15/33): ceph-mon-18.2.2-1021.g7c37d80f.el9.x86 4.0 MB/s | 4.7 MB 00:01 2024-03-27T21:37:43.794 INFO:teuthology.orchestra.run.smithi191.stdout:(16/33): python3-requests-2.25.1-8.el9.noarch.r 288 kB/s | 125 kB 00:00 2024-03-27T21:37:43.911 INFO:teuthology.orchestra.run.smithi191.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 137 kB/s | 43 kB 00:00 2024-03-27T21:37:43.986 INFO:teuthology.orchestra.run.smithi191.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 143 kB/s | 11 kB 00:00 2024-03-27T21:37:44.061 INFO:teuthology.orchestra.run.smithi191.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-03-27T21:37:44.087 INFO:teuthology.orchestra.run.smithi191.stdout:(20/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 436 kB/s | 172 kB 00:00 2024-03-27T21:37:44.145 INFO:teuthology.orchestra.run.smithi191.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 277 kB/s | 23 kB 00:00 2024-03-27T21:37:44.162 INFO:teuthology.orchestra.run.smithi191.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-03-27T21:37:44.221 INFO:teuthology.orchestra.run.smithi191.stdout:(23/33): python3-cherrypy-18.6.1-2.el9.noarch.r 840 kB/s | 358 kB 00:00 2024-03-27T21:37:44.238 INFO:teuthology.orchestra.run.smithi191.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 213 kB/s | 20 kB 00:00 2024-03-27T21:37:44.255 INFO:teuthology.orchestra.run.smithi191.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 502 kB/s | 46 kB 00:00 2024-03-27T21:37:44.288 INFO:teuthology.orchestra.run.smithi191.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2024-03-27T21:37:44.322 INFO:teuthology.orchestra.run.smithi191.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 245 kB/s | 16 kB 00:00 2024-03-27T21:37:44.364 INFO:teuthology.orchestra.run.smithi191.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-27T21:37:44.389 INFO:teuthology.orchestra.run.smithi191.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 533 kB/s | 36 kB 00:00 2024-03-27T21:37:44.456 INFO:teuthology.orchestra.run.smithi191.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.2 MB/s | 272 kB 00:00 2024-03-27T21:37:44.481 INFO:teuthology.orchestra.run.smithi191.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-03-27T21:37:44.531 INFO:teuthology.orchestra.run.smithi191.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2024-03-27T21:37:44.573 INFO:teuthology.orchestra.run.smithi191.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.3 MB/s | 427 kB 00:00 2024-03-27T21:37:44.578 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:37:44.579 INFO:teuthology.orchestra.run.smithi191.stdout:Total 11 MB/s | 29 MB 00:02 2024-03-27T21:37:44.591 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-27T21:37:44.629 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-27T21:37:44.629 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-27T21:37:44.982 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-27T21:37:44.982 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-27T21:37:45.060 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-2:18.2.2-1021.g7c37d80f.el9.x86_64 33/33 2024-03-27T21:37:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/33 2024-03-27T21:37:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mds-2:18.2.2-1021.g7c37d80f.el9.x86_64 2/33 2024-03-27T21:37:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-2:18.2.2-1021.g7c37d80f.el9.x86_64 3/33 2024-03-27T21:37:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mon-2:18.2.2-1021.g7c37d80f.el9.x86_64 4/33 2024-03-27T21:37:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-osd-2:18.2.2-1021.g7c37d80f.el9.x86_64 5/33 2024-03-27T21:37:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1021.g7c37d80f.el9. 6/33 2024-03-27T21:37:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-27T21:37:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-27T21:37:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-27T21:37:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-27T21:37:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-27T21:37:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-27T21:37:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-27T21:37:45.062 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-27T21:37:45.062 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-27T21:37:45.062 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-27T21:37:45.062 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-27T21:37:45.062 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-27T21:37:45.062 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-27T21:37:45.062 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-27T21:37:45.062 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-27T21:37:45.062 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-27T21:37:45.062 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-27T21:37:45.063 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-27T21:37:45.063 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-27T21:37:45.063 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-27T21:37:45.063 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-27T21:37:45.063 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-27T21:37:45.063 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-27T21:37:45.063 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-27T21:37:45.064 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-27T21:37:45.064 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-27T21:37:46.043 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-27T21:37:46.043 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:46.044 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-27T21:37:46.044 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:46.044 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:46.044 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:46.044 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core-2:18.2.2-1021.g7c37d80f.el9.noarch 2024-03-27T21:37:46.044 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:46.044 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:46.044 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-27T21:37:46.044 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-27T21:37:46.044 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-27T21:37:46.044 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-27T21:37:46.045 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-27T21:37:46.045 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-27T21:37:46.045 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-27T21:37:46.045 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-27T21:37:46.045 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-27T21:37:46.045 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-27T21:37:46.045 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-27T21:37:46.045 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-27T21:37:46.045 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-27T21:37:46.045 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-27T21:37:46.045 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-27T21:37:46.045 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply-3.11-14.el9.noarch 2024-03-27T21:37:46.045 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-27T21:37:46.046 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-27T21:37:46.046 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-27T21:37:46.046 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-27T21:37:46.046 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-27T21:37:46.046 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-27T21:37:46.046 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-27T21:37:46.046 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-27T21:37:46.046 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-27T21:37:46.046 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-27T21:37:46.047 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-27T21:37:46.047 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:46.047 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:37:46.239 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-27T21:37:46.301 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-27T21:37:46.323 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-base 2024-03-27T21:37:46.333 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-27T21:37:46.366 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-27T21:37:46.457 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-27T21:37:46.527 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-27T21:37:46.568 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-27T21:37:46.648 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-27T21:37:46.702 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-27T21:37:46.741 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-27T21:37:46.790 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-27T21:37:46.846 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-27T21:37:46.891 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:50 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:37:46.990 INFO:teuthology.orchestra.run.smithi060.stdout:Package ceph-base-2:18.2.2-1021.g7c37d80f.el9.x86_64 is already installed. 2024-03-27T21:37:47.040 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-27T21:37:47.045 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:37:47.047 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-03-27T21:37:47.047 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:37:47.095 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-27T21:37:47.105 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install cephadm 2024-03-27T21:37:47.136 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-27T21:37:47.177 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-27T21:37:47.207 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-27T21:37:47.282 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-27T21:37:47.317 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-27T21:37:47.350 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-27T21:37:47.391 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-27T21:37:47.426 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-27T21:37:47.461 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-27T21:37:47.550 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-27T21:37:47.591 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-27T21:37:47.664 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-27T21:37:47.675 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:51 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:37:47.800 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-27T21:37:47.829 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:37:47.830 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:37:47.830 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-03-27T21:37:47.830 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:37:47.830 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-27T21:37:47.830 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm noarch 2:18.2.2-1021.g7c37d80f.el9 ceph-noarch 224 k 2024-03-27T21:37:47.830 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:47.830 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-27T21:37:47.830 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:37:47.830 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-27T21:37:47.830 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:47.830 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 224 k 2024-03-27T21:37:47.831 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 216 k 2024-03-27T21:37:47.831 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-27T21:37:47.934 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-27T21:37:48.062 INFO:teuthology.orchestra.run.smithi060.stdout:cephadm-18.2.2-1021.g7c37d80f.el9.noarch.rpm 969 kB/s | 224 kB 00:00 2024-03-27T21:37:48.063 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:37:48.063 INFO:teuthology.orchestra.run.smithi060.stdout:Total 963 kB/s | 224 kB 00:00 2024-03-27T21:37:48.063 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-27T21:37:48.069 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-27T21:37:48.069 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-27T21:37:48.071 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-27T21:37:48.072 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-27T21:37:48.084 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1021.g7c37d80f.el9. 28/33 2024-03-27T21:37:48.113 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-2:18.2.2-1021.g7c37d80f.el9.x86_64 29/33 2024-03-27T21:37:48.137 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1021.g7c37d80f.el9.x86_64 29/33 2024-03-27T21:37:48.138 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T21:37:48.138 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-27T21:37:48.138 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-27T21:37:48.138 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-27T21:37:48.138 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:48.144 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-27T21:37:48.591 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephadm-2:18.2.2-1021.g7c37d80f.el9.noarch 1/1 2024-03-27T21:37:48.678 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephadm-2:18.2.2-1021.g7c37d80f.el9.noarch 1/1 2024-03-27T21:37:49.011 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephadm-2:18.2.2-1021.g7c37d80f.el9.noarch 1/1 2024-03-27T21:37:49.298 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephadm-2:18.2.2-1021.g7c37d80f.el9.noarch 1/1 2024-03-27T21:37:49.298 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:49.298 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-27T21:37:49.298 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm-2:18.2.2-1021.g7c37d80f.el9.noarch 2024-03-27T21:37:49.298 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:49.298 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:37:49.500 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-immutable-object-cache 2024-03-27T21:37:49.598 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-osd-2:18.2.2-1021.g7c37d80f.el9.x86_64 30/33 2024-03-27T21:37:49.626 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-osd-2:18.2.2-1021.g7c37d80f.el9.x86_64 30/33 2024-03-27T21:37:49.626 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T21:37:49.627 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-27T21:37:49.627 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-27T21:37:49.627 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-27T21:37:49.627 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:50.066 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mon-2:18.2.2-1021.g7c37d80f.el9.x86_64 31/33 2024-03-27T21:37:50.070 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:54 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:37:50.089 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mon-2:18.2.2-1021.g7c37d80f.el9.x86_64 31/33 2024-03-27T21:37:50.089 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T21:37:50.089 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-27T21:37:50.089 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-27T21:37:50.089 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-27T21:37:50.090 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:50.228 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:37:50.228 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 142 k 2024-03-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 142 k 2024-03-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 395 k 2024-03-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-27T21:37:50.291 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mds-2:18.2.2-1021.g7c37d80f.el9.x86_64 32/33 2024-03-27T21:37:50.314 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mds-2:18.2.2-1021.g7c37d80f.el9.x86_64 32/33 2024-03-27T21:37:50.315 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T21:37:50.315 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-27T21:37:50.315 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-27T21:37:50.315 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-27T21:37:50.315 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:50.381 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-2:18.2.2-1021.g7c37d80f.el9.x86_64 33/33 2024-03-27T21:37:50.568 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-immutable-object-cache-18.2.2-1021.g7c37d8 421 kB/s | 142 kB 00:00 2024-03-27T21:37:50.569 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:37:50.569 INFO:teuthology.orchestra.run.smithi060.stdout:Total 419 kB/s | 142 kB 00:00 2024-03-27T21:37:50.569 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-27T21:37:50.577 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-27T21:37:50.577 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-27T21:37:50.632 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-27T21:37:50.632 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-27T21:37:50.774 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-27T21:37:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1021.g7c37d80f. 1/1 2024-03-27T21:37:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1021.g7c37d80f. 1/1 2024-03-27T21:37:50.860 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T21:37:50.860 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-27T21:37:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:51.825 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1021.g7c37d80f. 1/1 2024-03-27T21:37:51.826 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:51.826 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-27T21:37:51.826 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:51.826 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:51.826 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:37:52.089 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr 2024-03-27T21:37:52.269 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-2:18.2.2-1021.g7c37d80f.el9.x86_64 33/33 2024-03-27T21:37:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/33 2024-03-27T21:37:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mds-2:18.2.2-1021.g7c37d80f.el9.x86_64 2/33 2024-03-27T21:37:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-2:18.2.2-1021.g7c37d80f.el9.x86_64 3/33 2024-03-27T21:37:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mon-2:18.2.2-1021.g7c37d80f.el9.x86_64 4/33 2024-03-27T21:37:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-osd-2:18.2.2-1021.g7c37d80f.el9.x86_64 5/33 2024-03-27T21:37:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1021.g7c37d80f.el9. 6/33 2024-03-27T21:37:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-27T21:37:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-27T21:37:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-27T21:37:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-27T21:37:52.270 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-27T21:37:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-27T21:37:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-27T21:37:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-27T21:37:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-27T21:37:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-27T21:37:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-27T21:37:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-27T21:37:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-27T21:37:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-27T21:37:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-27T21:37:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-27T21:37:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-27T21:37:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-27T21:37:52.271 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-27T21:37:52.272 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-27T21:37:52.272 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-27T21:37:52.272 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-27T21:37:52.273 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-27T21:37:52.273 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-27T21:37:52.273 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-27T21:37:52.273 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-27T21:37:52.672 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:56 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:37:52.772 INFO:teuthology.orchestra.run.smithi060.stdout:Package ceph-mgr-2:18.2.2-1021.g7c37d80f.el9.x86_64 is already installed. 2024-03-27T21:37:52.827 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:37:52.828 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-03-27T21:37:52.828 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:37:52.891 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-dashboard 2024-03-27T21:37:53.253 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-27T21:37:53.253 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:53.253 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-27T21:37:53.253 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:53.254 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:53.254 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:53.254 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core-2:18.2.2-1021.g7c37d80f.el9.noarch 2024-03-27T21:37:53.254 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:53.254 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:53.254 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-27T21:37:53.254 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-27T21:37:53.254 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-27T21:37:53.254 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-27T21:37:53.254 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-27T21:37:53.254 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-27T21:37:53.254 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-27T21:37:53.254 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-27T21:37:53.254 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-27T21:37:53.255 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-27T21:37:53.255 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-27T21:37:53.255 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-27T21:37:53.255 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-27T21:37:53.255 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-27T21:37:53.255 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-27T21:37:53.255 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply-3.11-14.el9.noarch 2024-03-27T21:37:53.255 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-27T21:37:53.255 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-27T21:37:53.255 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-27T21:37:53.255 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-27T21:37:53.255 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-27T21:37:53.255 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-27T21:37:53.255 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-27T21:37:53.256 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-27T21:37:53.256 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-27T21:37:53.256 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-27T21:37:53.256 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-27T21:37:53.256 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:53.256 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:37:53.458 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:57 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:37:53.613 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:37:53.613 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:37:53.613 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-03-27T21:37:53.613 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:37:53.614 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-27T21:37:53.614 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1021.g7c37d80f.el9 ceph-noarch 1.8 M 2024-03-27T21:37:53.614 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-27T21:37:53.614 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1021.g7c37d80f.el9 ceph-noarch 23 k 2024-03-27T21:37:53.614 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1021.g7c37d80f.el9 ceph-noarch 15 k 2024-03-27T21:37:53.614 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-27T21:37:53.614 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-27T21:37:53.614 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:53.614 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-27T21:37:53.614 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:37:53.614 INFO:teuthology.orchestra.run.smithi060.stdout:Install 5 Packages 2024-03-27T21:37:53.614 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:53.615 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 2.0 M 2024-03-27T21:37:53.616 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 8.6 M 2024-03-27T21:37:53.616 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-27T21:37:53.616 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-base 2024-03-27T21:37:53.922 INFO:teuthology.orchestra.run.smithi060.stdout:(1/5): ceph-prometheus-alerts-18.2.2-1021.g7c37 58 kB/s | 15 kB 00:00 2024-03-27T21:37:53.940 INFO:teuthology.orchestra.run.smithi060.stdout:(2/5): ceph-grafana-dashboards-18.2.2-1021.g7c3 85 kB/s | 23 kB 00:00 2024-03-27T21:37:53.990 INFO:teuthology.orchestra.run.smithi060.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 460 kB/s | 31 kB 00:00 2024-03-27T21:37:54.032 INFO:teuthology.orchestra.run.smithi060.stdout:(4/5): ceph-mgr-dashboard-18.2.2-1021.g7c37d80f 4.9 MB/s | 1.8 MB 00:00 2024-03-27T21:37:54.057 INFO:teuthology.orchestra.run.smithi060.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-03-27T21:37:54.058 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:37:54.059 INFO:teuthology.orchestra.run.smithi060.stdout:Total 4.6 MB/s | 2.0 MB 00:00 2024-03-27T21:37:54.059 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-27T21:37:54.066 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-27T21:37:54.066 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-27T21:37:54.126 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-27T21:37:54.126 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-27T21:37:54.178 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:51 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:37:54.274 INFO:teuthology.orchestra.run.smithi191.stdout:Package ceph-base-2:18.2.2-1021.g7c37d80f.el9.x86_64 is already installed. 2024-03-27T21:37:54.328 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:37:54.329 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-27T21:37:54.329 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:37:54.357 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-27T21:37:54.389 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install cephadm 2024-03-27T21:37:54.422 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-03-27T21:37:54.462 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-03-27T21:37:54.490 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1021.g7c37d80f.el9.n 3/5 2024-03-27T21:37:54.741 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1021.g7c37d80f.el9. 4/5 2024-03-27T21:37:54.768 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1021.g7c37d80f.el9.noarc 5/5 2024-03-27T21:37:54.945 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:51 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:37:55.093 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:37:55.094 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:37:55.094 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-27T21:37:55.094 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:37:55.094 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-27T21:37:55.094 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm noarch 2:18.2.2-1021.g7c37d80f.el9 ceph-noarch 224 k 2024-03-27T21:37:55.094 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:55.094 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-27T21:37:55.094 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:37:55.095 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-27T21:37:55.095 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:55.095 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 224 k 2024-03-27T21:37:55.095 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 216 k 2024-03-27T21:37:55.095 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-27T21:37:55.257 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1021.g7c37d80f.el9.noarc 5/5 2024-03-27T21:37:55.257 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1021.g7c37d80f.el9. 1/5 2024-03-27T21:37:55.257 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1021.g7c37d80f.el9.noarc 2/5 2024-03-27T21:37:55.258 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1021.g7c37d80f.el9.n 3/5 2024-03-27T21:37:55.258 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-03-27T21:37:55.456 INFO:teuthology.orchestra.run.smithi191.stdout:cephadm-18.2.2-1021.g7c37d80f.el9.noarch.rpm 621 kB/s | 224 kB 00:00 2024-03-27T21:37:55.457 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:37:55.457 INFO:teuthology.orchestra.run.smithi191.stdout:Total 618 kB/s | 224 kB 00:00 2024-03-27T21:37:55.457 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-27T21:37:55.462 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-27T21:37:55.462 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-27T21:37:55.465 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-27T21:37:55.465 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-27T21:37:55.534 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-27T21:37:55.537 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-03-27T21:37:55.537 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:55.538 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-27T21:37:55.538 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards-2:18.2.2-1021.g7c37d80f.el9.noarch 2024-03-27T21:37:55.538 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard-2:18.2.2-1021.g7c37d80f.el9.noarch 2024-03-27T21:37:55.538 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts-2:18.2.2-1021.g7c37d80f.el9.noarch 2024-03-27T21:37:55.538 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-27T21:37:55.538 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-27T21:37:55.538 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:55.538 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:37:55.724 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-27T21:37:55.991 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:18.2.2-1021.g7c37d80f.el9.noarch 1/1 2024-03-27T21:37:56.085 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephadm-2:18.2.2-1021.g7c37d80f.el9.noarch 1/1 2024-03-27T21:37:56.287 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:00 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:37:56.442 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:37:56.443 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================================ 2024-03-27T21:37:56.443 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-03-27T21:37:56.443 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================================ 2024-03-27T21:37:56.443 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-27T21:37:56.443 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1021.g7c37d80f.el9 ceph-noarch 7.4 M 2024-03-27T21:37:56.444 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-27T21:37:56.444 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-27T21:37:56.444 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-27T21:37:56.444 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-27T21:37:56.444 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-27T21:37:56.444 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-27T21:37:56.444 INFO:teuthology.orchestra.run.smithi060.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-27T21:37:56.444 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-27T21:37:56.444 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-27T21:37:56.444 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-27T21:37:56.444 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-27T21:37:56.444 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-27T21:37:56.444 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:56.444 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-27T21:37:56.445 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================================ 2024-03-27T21:37:56.445 INFO:teuthology.orchestra.run.smithi060.stdout:Install 12 Packages 2024-03-27T21:37:56.445 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:37:56.445 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 37 M 2024-03-27T21:37:56.445 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 209 M 2024-03-27T21:37:56.445 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-27T21:37:56.484 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:18.2.2-1021.g7c37d80f.el9.noarch 1/1 2024-03-27T21:37:56.737 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephadm-2:18.2.2-1021.g7c37d80f.el9.noarch 1/1 2024-03-27T21:37:56.737 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:56.737 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-27T21:37:56.738 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm-2:18.2.2-1021.g7c37d80f.el9.noarch 2024-03-27T21:37:56.738 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:56.738 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:37:56.896 INFO:teuthology.orchestra.run.smithi060.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 222 kB/s | 32 kB 00:00 2024-03-27T21:37:56.930 INFO:teuthology.orchestra.run.smithi060.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 517 kB/s | 17 kB 00:00 2024-03-27T21:37:56.943 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-immutable-object-cache 2024-03-27T21:37:56.963 INFO:teuthology.orchestra.run.smithi060.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-27T21:37:57.147 INFO:teuthology.orchestra.run.smithi060.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2024-03-27T21:37:57.281 INFO:teuthology.orchestra.run.smithi060.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-27T21:37:57.322 INFO:teuthology.orchestra.run.smithi060.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.3 MB/s | 217 kB 00:00 2024-03-27T21:37:57.431 INFO:teuthology.orchestra.run.smithi060.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.4 MB/s | 3.0 MB 00:00 2024-03-27T21:37:57.515 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:54 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:37:57.669 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:37:57.670 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:37:57.670 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-27T21:37:57.670 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:37:57.670 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-27T21:37:57.670 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 142 k 2024-03-27T21:37:57.670 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:57.670 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-27T21:37:57.670 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:37:57.670 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-27T21:37:57.670 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:57.670 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 142 k 2024-03-27T21:37:57.671 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 395 k 2024-03-27T21:37:57.671 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-27T21:37:57.731 INFO:teuthology.orchestra.run.smithi060.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-27T21:37:57.890 INFO:teuthology.orchestra.run.smithi060.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-03-27T21:37:57.965 INFO:teuthology.orchestra.run.smithi060.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 841 kB/s | 195 kB 00:00 2024-03-27T21:37:58.029 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-immutable-object-cache-18.2.2-1021.g7c37d8 398 kB/s | 142 kB 00:00 2024-03-27T21:37:58.029 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:37:58.029 INFO:teuthology.orchestra.run.smithi191.stdout:Total 396 kB/s | 142 kB 00:00 2024-03-27T21:37:58.030 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-27T21:37:58.037 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-27T21:37:58.037 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-27T21:37:58.089 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-27T21:37:58.089 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-27T21:37:58.090 INFO:teuthology.orchestra.run.smithi060.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-03-27T21:37:58.243 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-27T21:37:58.296 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1021.g7c37d80f. 1/1 2024-03-27T21:37:58.321 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1021.g7c37d80f. 1/1 2024-03-27T21:37:58.321 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T21:37:58.321 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-27T21:37:58.321 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:58.557 INFO:teuthology.orchestra.run.smithi060.stdout:(12/12): ceph-mgr-diskprediction-local-18.2.2-1 4.1 MB/s | 7.4 MB 00:01 2024-03-27T21:37:58.561 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:37:58.561 INFO:teuthology.orchestra.run.smithi060.stdout:Total 18 MB/s | 37 MB 00:02 2024-03-27T21:37:58.601 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-27T21:37:58.626 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-27T21:37:58.626 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-27T21:37:59.106 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-27T21:37:59.106 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-27T21:37:59.321 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1021.g7c37d80f. 1/1 2024-03-27T21:37:59.322 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:59.322 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-27T21:37:59.322 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:37:59.322 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:37:59.322 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:37:59.582 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr 2024-03-27T21:37:59.834 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-27T21:37:59.922 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-27T21:37:59.953 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-27T21:38:00.059 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-27T21:38:00.098 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-27T21:38:00.149 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:57 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:38:00.245 INFO:teuthology.orchestra.run.smithi191.stdout:Package ceph-mgr-2:18.2.2-1021.g7c37d80f.el9.x86_64 is already installed. 2024-03-27T21:38:00.292 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-27T21:38:00.297 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:38:00.298 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-27T21:38:00.298 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:38:00.322 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-27T21:38:00.359 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-dashboard 2024-03-27T21:38:00.415 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-27T21:38:00.921 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:57 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:38:01.071 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:38:01.072 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:01.072 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-27T21:38:01.072 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:01.072 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-27T21:38:01.072 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1021.g7c37d80f.el9 ceph-noarch 1.8 M 2024-03-27T21:38:01.072 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-27T21:38:01.072 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1021.g7c37d80f.el9 ceph-noarch 23 k 2024-03-27T21:38:01.072 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1021.g7c37d80f.el9 ceph-noarch 15 k 2024-03-27T21:38:01.072 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-27T21:38:01.072 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-27T21:38:01.072 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:01.072 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-27T21:38:01.073 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:01.073 INFO:teuthology.orchestra.run.smithi191.stdout:Install 5 Packages 2024-03-27T21:38:01.073 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:01.073 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 2.0 M 2024-03-27T21:38:01.073 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 8.6 M 2024-03-27T21:38:01.073 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-27T21:38:01.109 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-27T21:38:01.214 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-27T21:38:01.316 INFO:teuthology.orchestra.run.smithi191.stdout:(1/5): ceph-prometheus-alerts-18.2.2-1021.g7c37 76 kB/s | 15 kB 00:00 2024-03-27T21:38:01.417 INFO:teuthology.orchestra.run.smithi191.stdout:(2/5): ceph-grafana-dashboards-18.2.2-1021.g7c3 79 kB/s | 23 kB 00:00 2024-03-27T21:38:01.525 INFO:teuthology.orchestra.run.smithi191.stdout:(3/5): ceph-mgr-dashboard-18.2.2-1021.g7c37d80f 4.4 MB/s | 1.8 MB 00:00 2024-03-27T21:38:01.609 INFO:teuthology.orchestra.run.smithi191.stdout:(4/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 105 kB/s | 31 kB 00:00 2024-03-27T21:38:01.942 INFO:teuthology.orchestra.run.smithi191.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 359 kB/s | 188 kB 00:00 2024-03-27T21:38:01.944 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:38:01.944 INFO:teuthology.orchestra.run.smithi191.stdout:Total 2.3 MB/s | 2.0 MB 00:00 2024-03-27T21:38:01.945 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-27T21:38:01.952 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-27T21:38:01.952 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-27T21:38:02.018 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-27T21:38:02.018 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-27T21:38:02.253 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-27T21:38:02.332 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-03-27T21:38:02.363 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-03-27T21:38:02.392 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1021.g7c37d80f.el9.n 3/5 2024-03-27T21:38:02.527 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-27T21:38:02.632 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1021.g7c37d80f.el9. 4/5 2024-03-27T21:38:02.662 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1021.g7c37d80f.el9.noarc 5/5 2024-03-27T21:38:03.214 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1021.g7c37d80f.el9.noarc 5/5 2024-03-27T21:38:03.214 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1021.g7c37d80f.el9. 1/5 2024-03-27T21:38:03.215 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1021.g7c37d80f.el9.noarc 2/5 2024-03-27T21:38:03.215 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1021.g7c37d80f.el9.n 3/5 2024-03-27T21:38:03.215 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-03-27T21:38:03.495 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-03-27T21:38:03.495 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:03.495 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-27T21:38:03.495 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards-2:18.2.2-1021.g7c37d80f.el9.noarch 2024-03-27T21:38:03.495 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard-2:18.2.2-1021.g7c37d80f.el9.noarch 2024-03-27T21:38:03.495 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts-2:18.2.2-1021.g7c37d80f.el9.noarch 2024-03-27T21:38:03.496 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-27T21:38:03.496 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-27T21:38:03.496 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:03.496 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:38:03.599 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-27T21:38:03.621 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1021.g7c37d 12/12 2024-03-27T21:38:03.719 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-27T21:38:04.278 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:01 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:38:04.429 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:38:04.430 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2024-03-27T21:38:04.430 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-27T21:38:04.430 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2024-03-27T21:38:04.430 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-27T21:38:04.430 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1021.g7c37d80f.el9 ceph-noarch 7.4 M 2024-03-27T21:38:04.430 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-27T21:38:04.430 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-27T21:38:04.430 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-27T21:38:04.430 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-27T21:38:04.430 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-27T21:38:04.430 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-27T21:38:04.431 INFO:teuthology.orchestra.run.smithi191.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-27T21:38:04.431 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-27T21:38:04.431 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-27T21:38:04.431 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-27T21:38:04.431 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-27T21:38:04.431 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-27T21:38:04.431 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:04.431 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-27T21:38:04.431 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2024-03-27T21:38:04.431 INFO:teuthology.orchestra.run.smithi191.stdout:Install 12 Packages 2024-03-27T21:38:04.431 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:04.432 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 37 M 2024-03-27T21:38:04.432 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 209 M 2024-03-27T21:38:04.432 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-27T21:38:04.732 INFO:teuthology.orchestra.run.smithi191.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 211 kB/s | 32 kB 00:00 2024-03-27T21:38:04.765 INFO:teuthology.orchestra.run.smithi191.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 534 kB/s | 17 kB 00:00 2024-03-27T21:38:04.807 INFO:teuthology.orchestra.run.smithi191.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 831 kB/s | 34 kB 00:00 2024-03-27T21:38:04.866 INFO:teuthology.orchestra.run.smithi191.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-27T21:38:05.075 INFO:teuthology.orchestra.run.smithi191.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-03-27T21:38:05.141 INFO:teuthology.orchestra.run.smithi191.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-03-27T21:38:05.158 INFO:teuthology.orchestra.run.smithi191.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-27T21:38:05.525 INFO:teuthology.orchestra.run.smithi191.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-27T21:38:05.650 INFO:teuthology.orchestra.run.smithi191.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-03-27T21:38:05.742 INFO:teuthology.orchestra.run.smithi191.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 902 kB/s | 195 kB 00:00 2024-03-27T21:38:05.984 INFO:teuthology.orchestra.run.smithi191.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 736 kB/s | 245 kB 00:00 2024-03-27T21:38:06.179 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1021.g7c37d 12/12 2024-03-27T21:38:06.180 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1021.g7c37d 1/12 2024-03-27T21:38:06.180 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-27T21:38:06.180 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-27T21:38:06.180 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-27T21:38:06.180 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-27T21:38:06.180 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-27T21:38:06.180 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-27T21:38:06.180 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-27T21:38:06.180 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-27T21:38:06.180 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-27T21:38:06.180 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-27T21:38:06.451 INFO:teuthology.orchestra.run.smithi191.stdout:(12/12): ceph-mgr-diskprediction-local-18.2.2-1 4.0 MB/s | 7.4 MB 00:01 2024-03-27T21:38:06.453 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:38:06.454 INFO:teuthology.orchestra.run.smithi191.stdout:Total 18 MB/s | 37 MB 00:02 2024-03-27T21:38:06.494 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-27T21:38:06.518 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-27T21:38:06.518 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-27T21:38:06.580 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-27T21:38:06.581 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:06.581 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-27T21:38:06.581 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1021.g7c37d80f.el9.noarch 2024-03-27T21:38:06.581 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-27T21:38:06.581 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-27T21:38:06.581 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-27T21:38:06.581 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-27T21:38:06.581 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-27T21:38:06.581 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-27T21:38:06.581 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-27T21:38:06.581 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-27T21:38:06.581 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-27T21:38:06.581 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-27T21:38:06.582 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-27T21:38:06.582 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:06.582 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:38:06.933 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-rook 2024-03-27T21:38:06.949 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-27T21:38:06.949 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-27T21:38:07.503 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:11 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:38:07.658 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:38:07.659 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2024-03-27T21:38:07.659 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-03-27T21:38:07.659 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2024-03-27T21:38:07.659 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-27T21:38:07.659 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook noarch 2:18.2.2-1021.g7c37d80f.el9 ceph-noarch 49 k 2024-03-27T21:38:07.659 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-27T21:38:07.659 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-27T21:38:07.659 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-27T21:38:07.659 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-27T21:38:07.660 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-27T21:38:07.660 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-27T21:38:07.660 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-27T21:38:07.660 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-27T21:38:07.660 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-27T21:38:07.660 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-27T21:38:07.660 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-27T21:38:07.660 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-27T21:38:07.660 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-27T21:38:07.660 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:07.660 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-27T21:38:07.660 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2024-03-27T21:38:07.660 INFO:teuthology.orchestra.run.smithi060.stdout:Install 13 Packages 2024-03-27T21:38:07.661 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:07.661 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 2.0 M 2024-03-27T21:38:07.661 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 26 M 2024-03-27T21:38:07.661 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-27T21:38:07.666 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-27T21:38:07.749 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-27T21:38:07.780 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-27T21:38:07.866 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-27T21:38:07.900 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-27T21:38:08.086 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-27T21:38:08.116 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-27T21:38:08.179 INFO:teuthology.orchestra.run.smithi060.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 178 kB/s | 24 kB 00:00 2024-03-27T21:38:08.196 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-27T21:38:08.204 INFO:teuthology.orchestra.run.smithi060.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 111 kB/s | 18 kB 00:00 2024-03-27T21:38:08.305 INFO:teuthology.orchestra.run.smithi060.stdout:(3/13): ceph-mgr-rook-18.2.2-1021.g7c37d80f.el9 188 kB/s | 49 kB 00:00 2024-03-27T21:38:08.330 INFO:teuthology.orchestra.run.smithi060.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.1 MB/s | 174 kB 00:00 2024-03-27T21:38:08.347 INFO:teuthology.orchestra.run.smithi060.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 931 kB/s | 132 kB 00:00 2024-03-27T21:38:08.372 INFO:teuthology.orchestra.run.smithi060.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.1 MB/s | 211 kB 00:00 2024-03-27T21:38:08.398 INFO:teuthology.orchestra.run.smithi060.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2024-03-27T21:38:08.498 INFO:teuthology.orchestra.run.smithi060.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 120 kB/s | 15 kB 00:00 2024-03-27T21:38:08.523 INFO:teuthology.orchestra.run.smithi060.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 184 kB/s | 32 kB 00:00 2024-03-27T21:38:08.732 INFO:teuthology.orchestra.run.smithi060.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 607 kB/s | 203 kB 00:00 2024-03-27T21:38:08.757 INFO:teuthology.orchestra.run.smithi060.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 253 kB/s | 59 kB 00:00 2024-03-27T21:38:08.807 INFO:teuthology.orchestra.run.smithi060.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-03-27T21:38:08.835 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-27T21:38:08.943 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-27T21:38:09.199 INFO:teuthology.orchestra.run.smithi060.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2024-03-27T21:38:09.201 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:38:09.201 INFO:teuthology.orchestra.run.smithi060.stdout:Total 1.3 MB/s | 2.0 MB 00:01 2024-03-27T21:38:09.202 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-27T21:38:09.217 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-27T21:38:09.218 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-27T21:38:09.320 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-27T21:38:09.321 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-27T21:38:09.839 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-27T21:38:10.011 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-27T21:38:10.077 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-27T21:38:10.133 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-27T21:38:10.168 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-27T21:38:10.182 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-27T21:38:10.211 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-27T21:38:10.307 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-27T21:38:10.426 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-27T21:38:10.479 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-27T21:38:11.217 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-27T21:38:11.239 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-27T21:38:11.250 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1021.g7c37d 12/12 2024-03-27T21:38:11.292 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-27T21:38:11.325 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-27T21:38:11.365 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-27T21:38:11.403 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-rook-2:18.2.2-1021.g7c37d80f.el9.noarch 13/13 2024-03-27T21:38:12.153 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1021.g7c37d80f.el9.noarch 13/13 2024-03-27T21:38:12.153 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1021.g7c37d80f.el9.noarch 1/13 2024-03-27T21:38:12.153 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-27T21:38:12.153 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-27T21:38:12.153 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-27T21:38:12.154 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-27T21:38:12.154 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-27T21:38:12.154 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-27T21:38:12.154 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-27T21:38:12.154 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-27T21:38:12.154 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-27T21:38:12.154 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-27T21:38:12.154 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-27T21:38:12.504 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-27T21:38:12.504 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:12.504 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-27T21:38:12.504 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook-2:18.2.2-1021.g7c37d80f.el9.noarch 2024-03-27T21:38:12.504 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-27T21:38:12.504 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-27T21:38:12.505 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-27T21:38:12.505 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-27T21:38:12.505 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-27T21:38:12.505 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-27T21:38:12.505 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-27T21:38:12.505 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-27T21:38:12.505 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-27T21:38:12.505 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-27T21:38:12.505 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-27T21:38:12.505 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-27T21:38:12.505 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:12.505 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:38:12.745 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-cephadm 2024-03-27T21:38:13.335 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:17 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:38:13.491 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:38:13.492 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2024-03-27T21:38:13.492 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-03-27T21:38:13.492 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2024-03-27T21:38:13.492 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-27T21:38:13.492 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1021.g7c37d80f.el9 ceph-noarch 138 k 2024-03-27T21:38:13.492 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-27T21:38:13.492 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-27T21:38:13.493 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-27T21:38:13.493 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-27T21:38:13.493 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-27T21:38:13.493 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-27T21:38:13.493 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:13.493 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-27T21:38:13.493 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2024-03-27T21:38:13.493 INFO:teuthology.orchestra.run.smithi060.stdout:Install 6 Packages 2024-03-27T21:38:13.493 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:13.494 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 6.8 M 2024-03-27T21:38:13.494 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 33 M 2024-03-27T21:38:13.494 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-27T21:38:13.820 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1021.g7c37d 12/12 2024-03-27T21:38:13.820 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1021.g7c37d 1/12 2024-03-27T21:38:13.820 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-27T21:38:13.820 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-27T21:38:13.820 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-27T21:38:13.821 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-27T21:38:13.821 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-27T21:38:13.821 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-27T21:38:13.821 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-27T21:38:13.821 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-27T21:38:13.821 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-27T21:38:13.821 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-27T21:38:13.959 INFO:teuthology.orchestra.run.smithi060.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-27T21:38:14.093 INFO:teuthology.orchestra.run.smithi060.stdout:(2/6): ceph-mgr-cephadm-18.2.2-1021.g7c37d80f.e 337 kB/s | 138 kB 00:00 2024-03-27T21:38:14.272 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-27T21:38:14.272 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:14.272 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-27T21:38:14.272 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1021.g7c37d80f.el9.noarch 2024-03-27T21:38:14.273 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-27T21:38:14.273 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-27T21:38:14.273 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-27T21:38:14.273 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-27T21:38:14.273 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-27T21:38:14.273 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-27T21:38:14.273 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-27T21:38:14.273 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-27T21:38:14.273 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-27T21:38:14.273 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-27T21:38:14.273 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-27T21:38:14.273 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:14.273 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:38:14.301 INFO:teuthology.orchestra.run.smithi060.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 401 kB/s | 248 kB 00:00 2024-03-27T21:38:14.393 INFO:teuthology.orchestra.run.smithi060.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 193 kB/s | 58 kB 00:00 2024-03-27T21:38:14.451 INFO:teuthology.orchestra.run.smithi060.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-27T21:38:14.585 INFO:teuthology.orchestra.run.smithi060.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 180 kB/s | 51 kB 00:00 2024-03-27T21:38:14.588 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:38:14.588 INFO:teuthology.orchestra.run.smithi060.stdout:Total 6.2 MB/s | 6.8 MB 00:01 2024-03-27T21:38:14.602 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-27T21:38:14.612 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-27T21:38:14.612 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-27T21:38:14.645 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-rook 2024-03-27T21:38:14.704 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-27T21:38:14.705 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-27T21:38:15.046 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-27T21:38:15.208 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-27T21:38:15.213 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:12 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:38:15.263 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-27T21:38:15.363 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:38:15.364 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================== 2024-03-27T21:38:15.364 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-27T21:38:15.365 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================== 2024-03-27T21:38:15.365 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-27T21:38:15.365 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook noarch 2:18.2.2-1021.g7c37d80f.el9 ceph-noarch 49 k 2024-03-27T21:38:15.365 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-27T21:38:15.365 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-27T21:38:15.365 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-27T21:38:15.365 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-27T21:38:15.365 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-27T21:38:15.365 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-27T21:38:15.365 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-27T21:38:15.365 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-27T21:38:15.365 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-27T21:38:15.365 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-27T21:38:15.366 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-27T21:38:15.366 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-27T21:38:15.366 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-27T21:38:15.366 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:15.366 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-27T21:38:15.366 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================== 2024-03-27T21:38:15.366 INFO:teuthology.orchestra.run.smithi191.stdout:Install 13 Packages 2024-03-27T21:38:15.366 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:15.367 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 2.0 M 2024-03-27T21:38:15.367 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 26 M 2024-03-27T21:38:15.367 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-27T21:38:15.532 INFO:teuthology.orchestra.run.smithi191.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 154 kB/s | 18 kB 00:00 2024-03-27T21:38:15.557 INFO:teuthology.orchestra.run.smithi191.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 171 kB/s | 24 kB 00:00 2024-03-27T21:38:15.600 INFO:teuthology.orchestra.run.smithi191.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-27T21:38:15.625 INFO:teuthology.orchestra.run.smithi191.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-27T21:38:15.642 INFO:teuthology.orchestra.run.smithi191.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.9 MB/s | 211 kB 00:00 2024-03-27T21:38:15.667 INFO:teuthology.orchestra.run.smithi191.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-27T21:38:15.701 INFO:teuthology.orchestra.run.smithi191.stdout:(7/13): ceph-mgr-rook-18.2.2-1021.g7c37d80f.el9 172 kB/s | 49 kB 00:00 2024-03-27T21:38:15.784 INFO:teuthology.orchestra.run.smithi191.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-03-27T21:38:15.809 INFO:teuthology.orchestra.run.smithi191.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 192 kB/s | 32 kB 00:00 2024-03-27T21:38:15.813 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-27T21:38:15.901 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-27T21:38:15.918 INFO:teuthology.orchestra.run.smithi191.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2024-03-27T21:38:15.935 INFO:teuthology.orchestra.run.smithi191.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 865 kB/s | 203 kB 00:00 2024-03-27T21:38:15.970 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-27T21:38:15.994 INFO:teuthology.orchestra.run.smithi191.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-03-27T21:38:15.999 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1021.g7c37d80f.el9.noarch 6/6 2024-03-27T21:38:16.035 INFO:teuthology.orchestra.run.smithi191.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.1 MB/s | 1.0 MB 00:00 2024-03-27T21:38:16.037 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:38:16.037 INFO:teuthology.orchestra.run.smithi191.stdout:Total 3.0 MB/s | 2.0 MB 00:00 2024-03-27T21:38:16.037 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-27T21:38:16.050 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-27T21:38:16.050 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-27T21:38:16.138 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-27T21:38:16.139 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-27T21:38:16.673 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-27T21:38:16.771 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1021.g7c37d80f.el9.noarch 6/6 2024-03-27T21:38:16.771 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1021.g7c37d80f.el9.noarch 1/6 2024-03-27T21:38:16.771 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-27T21:38:16.771 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-27T21:38:16.771 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-27T21:38:16.771 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-27T21:38:16.838 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-27T21:38:16.886 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-27T21:38:16.941 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-27T21:38:16.979 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-27T21:38:17.020 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-27T21:38:17.061 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-27T21:38:17.061 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:17.061 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-27T21:38:17.061 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm-2:18.2.2-1021.g7c37d80f.el9.noarch 2024-03-27T21:38:17.061 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-27T21:38:17.061 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-27T21:38:17.061 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-27T21:38:17.061 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-27T21:38:17.061 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-27T21:38:17.061 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:17.061 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:38:17.113 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-27T21:38:17.221 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-27T21:38:17.271 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-27T21:38:17.308 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-fuse 2024-03-27T21:38:17.881 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:21 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:38:17.985 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-27T21:38:18.035 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-27T21:38:18.036 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:38:18.036 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:38:18.036 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-03-27T21:38:18.036 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:38:18.036 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-27T21:38:18.037 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 844 k 2024-03-27T21:38:18.037 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-27T21:38:18.037 INFO:teuthology.orchestra.run.smithi060.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-27T21:38:18.037 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:18.037 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-27T21:38:18.037 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:38:18.037 INFO:teuthology.orchestra.run.smithi060.stdout:Install 2 Packages 2024-03-27T21:38:18.037 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:18.038 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 922 k 2024-03-27T21:38:18.038 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 2.7 M 2024-03-27T21:38:18.038 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-27T21:38:18.069 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-27T21:38:18.117 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-27T21:38:18.131 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-rook-2:18.2.2-1021.g7c37d80f.el9.noarch 13/13 2024-03-27T21:38:18.170 INFO:teuthology.orchestra.run.smithi060.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 595 kB/s | 79 kB 00:00 2024-03-27T21:38:18.496 INFO:teuthology.orchestra.run.smithi060.stdout:(2/2): ceph-fuse-18.2.2-1021.g7c37d80f.el9.x86_ 1.8 MB/s | 844 kB 00:00 2024-03-27T21:38:18.497 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:38:18.497 INFO:teuthology.orchestra.run.smithi060.stdout:Total 2.0 MB/s | 922 kB 00:00 2024-03-27T21:38:18.497 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-27T21:38:18.519 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-27T21:38:18.520 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-27T21:38:18.595 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-27T21:38:18.595 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-27T21:38:18.788 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-27T21:38:18.922 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-27T21:38:18.935 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1021.g7c37d80f.el9.noarch 13/13 2024-03-27T21:38:18.935 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1021.g7c37d80f.el9.noarch 1/13 2024-03-27T21:38:18.935 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-27T21:38:18.935 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-27T21:38:18.935 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-27T21:38:18.935 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-27T21:38:18.935 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-27T21:38:18.935 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-27T21:38:18.936 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-27T21:38:18.936 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-27T21:38:18.936 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-27T21:38:18.936 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-27T21:38:18.936 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-27T21:38:18.998 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-fuse-2:18.2.2-1021.g7c37d80f.el9.x86_64 2/2 2024-03-27T21:38:19.297 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-27T21:38:19.298 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:19.298 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-27T21:38:19.298 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook-2:18.2.2-1021.g7c37d80f.el9.noarch 2024-03-27T21:38:19.298 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-27T21:38:19.298 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-27T21:38:19.298 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-27T21:38:19.298 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-27T21:38:19.298 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-27T21:38:19.298 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-27T21:38:19.298 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-27T21:38:19.299 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-27T21:38:19.299 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-27T21:38:19.299 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-27T21:38:19.299 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-27T21:38:19.299 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-27T21:38:19.299 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:19.299 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:38:19.590 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-cephadm 2024-03-27T21:38:19.736 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1021.g7c37d80f.el9.x86_64 2/2 2024-03-27T21:38:19.736 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-fuse-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/2 2024-03-27T21:38:20.030 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-27T21:38:20.030 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:20.030 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-27T21:38:20.030 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse-2:18.2.2-1021.g7c37d80f.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-27T21:38:20.031 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:20.031 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:38:20.151 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:17 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:38:20.290 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-volume 2024-03-27T21:38:20.298 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:38:20.299 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================== 2024-03-27T21:38:20.299 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-27T21:38:20.299 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================== 2024-03-27T21:38:20.299 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-27T21:38:20.299 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1021.g7c37d80f.el9 ceph-noarch 138 k 2024-03-27T21:38:20.299 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-27T21:38:20.299 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-27T21:38:20.300 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-27T21:38:20.300 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-27T21:38:20.300 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-27T21:38:20.300 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-27T21:38:20.300 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:20.300 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-27T21:38:20.300 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================== 2024-03-27T21:38:20.300 INFO:teuthology.orchestra.run.smithi191.stdout:Install 6 Packages 2024-03-27T21:38:20.300 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:20.301 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 6.8 M 2024-03-27T21:38:20.301 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 33 M 2024-03-27T21:38:20.301 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-27T21:38:20.662 INFO:teuthology.orchestra.run.smithi191.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-03-27T21:38:20.746 INFO:teuthology.orchestra.run.smithi191.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-27T21:38:20.771 INFO:teuthology.orchestra.run.smithi191.stdout:(3/6): ceph-mgr-cephadm-18.2.2-1021.g7c37d80f.e 465 kB/s | 138 kB 00:00 2024-03-27T21:38:20.863 INFO:teuthology.orchestra.run.smithi191.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 495 kB/s | 58 kB 00:00 2024-03-27T21:38:20.867 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:24 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:38:20.897 INFO:teuthology.orchestra.run.smithi191.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 406 kB/s | 51 kB 00:00 2024-03-27T21:38:20.964 INFO:teuthology.orchestra.run.smithi191.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-03-27T21:38:20.965 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:38:20.965 INFO:teuthology.orchestra.run.smithi191.stdout:Total 10 MB/s | 6.8 MB 00:00 2024-03-27T21:38:20.978 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-27T21:38:20.988 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-27T21:38:20.988 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-27T21:38:21.024 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:38:21.024 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:38:21.024 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-03-27T21:38:21.024 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:38:21.024 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-27T21:38:21.024 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume noarch 2:18.2.2-1021.g7c37d80f.el9 ceph-noarch 264 k 2024-03-27T21:38:21.025 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-27T21:38:21.025 INFO:teuthology.orchestra.run.smithi060.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-27T21:38:21.025 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:21.025 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-27T21:38:21.025 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:38:21.025 INFO:teuthology.orchestra.run.smithi060.stdout:Install 2 Packages 2024-03-27T21:38:21.025 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:21.026 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 332 k 2024-03-27T21:38:21.026 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 1.5 M 2024-03-27T21:38:21.026 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-27T21:38:21.077 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-27T21:38:21.077 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-27T21:38:21.158 INFO:teuthology.orchestra.run.smithi060.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 519 kB/s | 69 kB 00:00 2024-03-27T21:38:21.292 INFO:teuthology.orchestra.run.smithi060.stdout:(2/2): ceph-volume-18.2.2-1021.g7c37d80f.el9.no 992 kB/s | 264 kB 00:00 2024-03-27T21:38:21.293 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:38:21.293 INFO:teuthology.orchestra.run.smithi060.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-27T21:38:21.294 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-27T21:38:21.302 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-27T21:38:21.303 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-27T21:38:21.340 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-27T21:38:21.340 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-27T21:38:21.378 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-27T21:38:21.494 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-27T21:38:21.556 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-27T21:38:21.607 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-27T21:38:21.672 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-27T21:38:21.711 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-volume-2:18.2.2-1021.g7c37d80f.el9.noarch 2/2 2024-03-27T21:38:21.727 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-volume-2:18.2.2-1021.g7c37d80f.el9.noarch 2/2 2024-03-27T21:38:21.727 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T21:38:21.727 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-27T21:38:21.727 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:22.127 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-27T21:38:22.210 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-27T21:38:22.279 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-27T21:38:22.302 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1021.g7c37d80f.el9.noarch 6/6 2024-03-27T21:38:22.402 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-volume-2:18.2.2-1021.g7c37d80f.el9.noarch 1/2 2024-03-27T21:38:22.725 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-27T21:38:22.725 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:22.725 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-27T21:38:22.725 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume-2:18.2.2-1021.g7c37d80f.el9.noarch 2024-03-27T21:38:22.725 INFO:teuthology.orchestra.run.smithi060.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-27T21:38:22.725 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:22.725 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:38:23.085 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librados-devel 2024-03-27T21:38:23.124 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1021.g7c37d80f.el9.noarch 6/6 2024-03-27T21:38:23.124 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1021.g7c37d80f.el9.noarch 1/6 2024-03-27T21:38:23.124 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-27T21:38:23.124 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-27T21:38:23.124 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-27T21:38:23.124 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-27T21:38:23.435 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-27T21:38:23.435 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:23.435 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-27T21:38:23.435 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm-2:18.2.2-1021.g7c37d80f.el9.noarch 2024-03-27T21:38:23.436 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-27T21:38:23.436 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-27T21:38:23.436 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-27T21:38:23.436 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-27T21:38:23.436 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-27T21:38:23.436 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:23.436 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:38:23.682 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:27 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:38:23.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-fuse 2024-03-27T21:38:23.839 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:38:23.841 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:38:23.841 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-27T21:38:23.841 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:38:23.841 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-27T21:38:23.841 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 127 k 2024-03-27T21:38:23.841 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:23.842 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-27T21:38:23.842 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:38:23.842 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-27T21:38:23.842 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:23.842 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 127 k 2024-03-27T21:38:23.842 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 456 k 2024-03-27T21:38:23.842 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-27T21:38:24.041 INFO:teuthology.orchestra.run.smithi060.stdout:librados-devel-18.2.2-1021.g7c37d80f.el9.x86_64 634 kB/s | 127 kB 00:00 2024-03-27T21:38:24.042 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:38:24.042 INFO:teuthology.orchestra.run.smithi060.stdout:Total 628 kB/s | 127 kB 00:00 2024-03-27T21:38:24.042 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-27T21:38:24.048 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-27T21:38:24.048 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-27T21:38:24.105 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-27T21:38:24.106 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-27T21:38:24.258 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-27T21:38:24.291 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:21 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:38:24.359 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librados-devel-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:24.438 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:38:24.439 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:24.439 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-27T21:38:24.439 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:24.439 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-27T21:38:24.439 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 844 k 2024-03-27T21:38:24.439 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-27T21:38:24.439 INFO:teuthology.orchestra.run.smithi191.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-27T21:38:24.439 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:24.440 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-27T21:38:24.440 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:24.440 INFO:teuthology.orchestra.run.smithi191.stdout:Install 2 Packages 2024-03-27T21:38:24.440 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:24.440 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 922 k 2024-03-27T21:38:24.440 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 2.7 M 2024-03-27T21:38:24.440 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-27T21:38:24.574 INFO:teuthology.orchestra.run.smithi191.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 591 kB/s | 79 kB 00:00 2024-03-27T21:38:24.757 INFO:teuthology.orchestra.run.smithi191.stdout:(2/2): ceph-fuse-18.2.2-1021.g7c37d80f.el9.x86_ 2.6 MB/s | 844 kB 00:00 2024-03-27T21:38:24.759 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:38:24.759 INFO:teuthology.orchestra.run.smithi191.stdout:Total 2.8 MB/s | 922 kB 00:00 2024-03-27T21:38:24.759 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-27T21:38:24.774 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados-devel-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:24.779 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-27T21:38:24.779 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-27T21:38:24.851 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-27T21:38:24.851 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-27T21:38:25.036 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-27T21:38:25.071 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados-devel-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:25.071 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:25.071 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-27T21:38:25.071 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:38:25.071 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:25.071 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:38:25.179 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-27T21:38:25.247 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-fuse-2:18.2.2-1021.g7c37d80f.el9.x86_64 2/2 2024-03-27T21:38:25.312 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install libcephfs2 2024-03-27T21:38:25.888 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:29 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:38:25.983 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1021.g7c37d80f.el9.x86_64 2/2 2024-03-27T21:38:25.983 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-fuse-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/2 2024-03-27T21:38:25.987 INFO:teuthology.orchestra.run.smithi060.stdout:Package libcephfs2-2:18.2.2-1021.g7c37d80f.el9.x86_64 is already installed. 2024-03-27T21:38:26.044 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:38:26.045 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-03-27T21:38:26.045 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:38:26.119 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install libcephfs-devel 2024-03-27T21:38:26.313 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-27T21:38:26.313 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:26.314 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-27T21:38:26.314 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse-2:18.2.2-1021.g7c37d80f.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-27T21:38:26.314 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:26.314 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:38:26.577 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-volume 2024-03-27T21:38:26.692 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:30 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:38:26.847 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:38:26.848 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:38:26.848 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-27T21:38:26.848 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:38:26.848 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-27T21:38:26.848 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 31 k 2024-03-27T21:38:26.848 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:26.848 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-27T21:38:26.848 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:38:26.849 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-27T21:38:26.849 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:26.849 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 31 k 2024-03-27T21:38:26.849 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 139 k 2024-03-27T21:38:26.849 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-27T21:38:27.086 INFO:teuthology.orchestra.run.smithi060.stdout:libcephfs-devel-18.2.2-1021.g7c37d80f.el9.x86_6 131 kB/s | 31 kB 00:00 2024-03-27T21:38:27.087 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:38:27.087 INFO:teuthology.orchestra.run.smithi060.stdout:Total 130 kB/s | 31 kB 00:00 2024-03-27T21:38:27.087 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-27T21:38:27.091 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-27T21:38:27.091 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-27T21:38:27.123 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-27T21:38:27.123 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-27T21:38:27.155 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:24 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:38:27.214 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-27T21:38:27.305 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:38:27.305 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:27.305 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-27T21:38:27.305 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:27.305 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-27T21:38:27.305 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume noarch 2:18.2.2-1021.g7c37d80f.el9 ceph-noarch 264 k 2024-03-27T21:38:27.306 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-27T21:38:27.306 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-27T21:38:27.306 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:27.306 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-27T21:38:27.306 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:27.306 INFO:teuthology.orchestra.run.smithi191.stdout:Install 2 Packages 2024-03-27T21:38:27.306 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:27.307 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 332 k 2024-03-27T21:38:27.307 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 1.5 M 2024-03-27T21:38:27.307 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-27T21:38:27.329 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs-devel-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:27.428 INFO:teuthology.orchestra.run.smithi191.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 567 kB/s | 69 kB 00:00 2024-03-27T21:38:27.578 INFO:teuthology.orchestra.run.smithi191.stdout:(2/2): ceph-volume-18.2.2-1021.g7c37d80f.el9.no 972 kB/s | 264 kB 00:00 2024-03-27T21:38:27.579 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:38:27.580 INFO:teuthology.orchestra.run.smithi191.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-27T21:38:27.580 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-27T21:38:27.588 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-27T21:38:27.588 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-27T21:38:27.629 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-27T21:38:27.630 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-27T21:38:27.686 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-27T21:38:27.941 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs-devel-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:27.941 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:27.941 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-27T21:38:27.941 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:38:27.941 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:27.941 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:38:27.960 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-27T21:38:28.006 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-volume-2:18.2.2-1021.g7c37d80f.el9.noarch 2/2 2024-03-27T21:38:28.021 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-volume-2:18.2.2-1021.g7c37d80f.el9.noarch 2/2 2024-03-27T21:38:28.022 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T21:38:28.022 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-27T21:38:28.022 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:28.126 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librados2 2024-03-27T21:38:28.700 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:32 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:38:28.799 INFO:teuthology.orchestra.run.smithi060.stdout:Package librados2-2:18.2.2-1021.g7c37d80f.el9.x86_64 is already installed. 2024-03-27T21:38:28.802 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-volume-2:18.2.2-1021.g7c37d80f.el9.noarch 1/2 2024-03-27T21:38:28.856 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:38:28.857 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-03-27T21:38:28.857 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:38:28.935 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librbd1 2024-03-27T21:38:29.112 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-27T21:38:29.112 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:29.112 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-27T21:38:29.112 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume-2:18.2.2-1021.g7c37d80f.el9.noarch 2024-03-27T21:38:29.112 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-27T21:38:29.112 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:29.112 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:38:29.475 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librados-devel 2024-03-27T21:38:29.510 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:33 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:38:29.610 INFO:teuthology.orchestra.run.smithi060.stdout:Package librbd1-2:18.2.2-1021.g7c37d80f.el9.x86_64 is already installed. 2024-03-27T21:38:29.666 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:38:29.667 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-03-27T21:38:29.667 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:38:29.738 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rados 2024-03-27T21:38:30.051 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:27 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:38:30.199 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:38:30.200 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:30.200 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-27T21:38:30.200 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:30.200 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-27T21:38:30.200 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 127 k 2024-03-27T21:38:30.200 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:30.200 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-27T21:38:30.200 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:30.200 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-27T21:38:30.200 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:30.200 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 127 k 2024-03-27T21:38:30.200 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 456 k 2024-03-27T21:38:30.201 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-27T21:38:30.310 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:34 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:38:30.409 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-rados-2:18.2.2-1021.g7c37d80f.el9.x86_64 is already installed. 2024-03-27T21:38:30.426 INFO:teuthology.orchestra.run.smithi191.stdout:librados-devel-18.2.2-1021.g7c37d80f.el9.x86_64 562 kB/s | 127 kB 00:00 2024-03-27T21:38:30.426 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:38:30.427 INFO:teuthology.orchestra.run.smithi191.stdout:Total 559 kB/s | 127 kB 00:00 2024-03-27T21:38:30.427 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-27T21:38:30.432 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-27T21:38:30.432 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-27T21:38:30.465 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:38:30.466 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-03-27T21:38:30.467 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:38:30.486 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-27T21:38:30.487 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-27T21:38:30.535 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rgw 2024-03-27T21:38:30.624 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-27T21:38:30.715 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librados-devel-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:31.105 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:35 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:38:31.186 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados-devel-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:31.204 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-rgw-2:18.2.2-1021.g7c37d80f.el9.x86_64 is already installed. 2024-03-27T21:38:31.260 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:38:31.262 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-03-27T21:38:31.262 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:38:31.342 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-cephfs 2024-03-27T21:38:31.484 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados-devel-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:31.484 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:31.484 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-27T21:38:31.484 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:38:31.484 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:31.485 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:38:31.787 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install libcephfs2 2024-03-27T21:38:31.913 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:35 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:38:32.012 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-cephfs-2:18.2.2-1021.g7c37d80f.el9.x86_64 is already installed. 2024-03-27T21:38:32.067 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:38:32.068 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-03-27T21:38:32.068 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:38:32.135 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rbd 2024-03-27T21:38:32.366 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:29 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:38:32.469 INFO:teuthology.orchestra.run.smithi191.stdout:Package libcephfs2-2:18.2.2-1021.g7c37d80f.el9.x86_64 is already installed. 2024-03-27T21:38:32.525 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:38:32.526 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-27T21:38:32.526 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:38:32.595 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install libcephfs-devel 2024-03-27T21:38:32.708 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:36 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:38:32.808 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-rbd-2:18.2.2-1021.g7c37d80f.el9.x86_64 is already installed. 2024-03-27T21:38:32.864 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:38:32.865 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-03-27T21:38:32.865 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:38:32.934 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-fuse 2024-03-27T21:38:33.160 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:30 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:38:33.309 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:38:33.310 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:33.310 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-27T21:38:33.310 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:33.310 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-27T21:38:33.310 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 31 k 2024-03-27T21:38:33.310 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:33.310 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-27T21:38:33.310 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:33.311 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-27T21:38:33.311 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:33.311 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 31 k 2024-03-27T21:38:33.311 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 139 k 2024-03-27T21:38:33.311 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-27T21:38:33.513 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:37 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:38:33.554 INFO:teuthology.orchestra.run.smithi191.stdout:libcephfs-devel-18.2.2-1021.g7c37d80f.el9.x86_6 128 kB/s | 31 kB 00:00 2024-03-27T21:38:33.554 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:38:33.555 INFO:teuthology.orchestra.run.smithi191.stdout:Total 127 kB/s | 31 kB 00:00 2024-03-27T21:38:33.555 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-27T21:38:33.558 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-27T21:38:33.558 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-27T21:38:33.582 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-27T21:38:33.582 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-27T21:38:33.671 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:38:33.671 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:38:33.671 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-03-27T21:38:33.671 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:38:33.671 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-27T21:38:33.671 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 87 k 2024-03-27T21:38:33.672 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:33.672 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-27T21:38:33.672 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:38:33.672 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-27T21:38:33.672 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:33.672 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 87 k 2024-03-27T21:38:33.672 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 227 k 2024-03-27T21:38:33.673 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-27T21:38:33.674 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-27T21:38:33.777 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs-devel-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:33.981 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-fuse-18.2.2-1021.g7c37d80f.el9.x86_64.rpm 281 kB/s | 87 kB 00:00 2024-03-27T21:38:33.981 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:38:33.982 INFO:teuthology.orchestra.run.smithi060.stdout:Total 280 kB/s | 87 kB 00:00 2024-03-27T21:38:33.982 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-27T21:38:33.989 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-27T21:38:33.990 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-27T21:38:34.046 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-27T21:38:34.046 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-27T21:38:34.176 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:34.193 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-27T21:38:34.307 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-fuse-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:34.422 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs-devel-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:34.423 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:34.423 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-27T21:38:34.423 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:38:34.423 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:34.423 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:38:34.609 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librados2 2024-03-27T21:38:34.671 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:34.943 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-fuse-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:34.943 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:34.943 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-27T21:38:34.944 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:38:34.944 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:34.944 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:38:35.175 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-mirror 2024-03-27T21:38:35.176 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:32 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:38:35.273 INFO:teuthology.orchestra.run.smithi191.stdout:Package librados2-2:18.2.2-1021.g7c37d80f.el9.x86_64 is already installed. 2024-03-27T21:38:35.325 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:38:35.326 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-27T21:38:35.326 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:38:35.395 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librbd1 2024-03-27T21:38:35.750 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:39 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:38:35.906 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:38:35.907 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:38:35.907 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-27T21:38:35.907 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:38:35.907 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-27T21:38:35.907 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 3.0 M 2024-03-27T21:38:35.907 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:35.907 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-27T21:38:35.907 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:38:35.907 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-27T21:38:35.907 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:35.908 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 3.0 M 2024-03-27T21:38:35.908 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 12 M 2024-03-27T21:38:35.908 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-27T21:38:35.959 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:32 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:38:36.056 INFO:teuthology.orchestra.run.smithi191.stdout:Package librbd1-2:18.2.2-1021.g7c37d80f.el9.x86_64 is already installed. 2024-03-27T21:38:36.107 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:38:36.109 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-27T21:38:36.109 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:38:36.182 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rados 2024-03-27T21:38:36.451 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-mirror-18.2.2-1021.g7c37d80f.el9.x86_64.rpm 5.5 MB/s | 3.0 MB 00:00 2024-03-27T21:38:36.452 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:38:36.452 INFO:teuthology.orchestra.run.smithi060.stdout:Total 5.5 MB/s | 3.0 MB 00:00 2024-03-27T21:38:36.452 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-27T21:38:36.462 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-27T21:38:36.462 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-27T21:38:36.538 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-27T21:38:36.538 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-27T21:38:36.741 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:33 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:38:36.838 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rados-2:18.2.2-1021.g7c37d80f.el9.x86_64 is already installed. 2024-03-27T21:38:36.891 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:38:36.892 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-27T21:38:36.892 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:38:36.972 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rgw 2024-03-27T21:38:36.976 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-27T21:38:37.027 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-mirror-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:37.051 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:37.051 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T21:38:37.051 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-27T21:38:37.051 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-27T21:38:37.051 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-27T21:38:37.051 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:37.536 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:34 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:38:37.633 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rgw-2:18.2.2-1021.g7c37d80f.el9.x86_64 is already installed. 2024-03-27T21:38:37.686 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:38:37.687 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-27T21:38:37.687 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:38:37.762 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-cephfs 2024-03-27T21:38:38.147 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-mirror-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:38.147 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:38.147 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-27T21:38:38.147 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:38:38.147 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:38.147 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:38:38.323 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:35 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:38:38.390 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-nbd 2024-03-27T21:38:38.420 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-cephfs-2:18.2.2-1021.g7c37d80f.el9.x86_64 is already installed. 2024-03-27T21:38:38.473 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:38:38.475 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-27T21:38:38.475 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:38:38.544 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rbd 2024-03-27T21:38:38.965 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:42 ago on Wed 27 Mar 2024 09:36:56 PM UTC. 2024-03-27T21:38:39.106 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:36 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:38:39.121 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-27T21:38:39.122 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:38:39.122 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-03-27T21:38:39.122 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:38:39.122 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-27T21:38:39.123 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 172 k 2024-03-27T21:38:39.123 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:39.123 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-27T21:38:39.123 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-27T21:38:39.123 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-27T21:38:39.123 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:39.123 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 172 k 2024-03-27T21:38:39.123 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 494 k 2024-03-27T21:38:39.123 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-27T21:38:39.204 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rbd-2:18.2.2-1021.g7c37d80f.el9.x86_64 is already installed. 2024-03-27T21:38:39.257 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:38:39.258 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-27T21:38:39.258 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:38:39.330 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-fuse 2024-03-27T21:38:39.479 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-nbd-18.2.2-1021.g7c37d80f.el9.x86_64.rpm 484 kB/s | 172 kB 00:00 2024-03-27T21:38:39.480 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:38:39.480 INFO:teuthology.orchestra.run.smithi060.stdout:Total 482 kB/s | 172 kB 00:00 2024-03-27T21:38:39.480 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-27T21:38:39.489 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-27T21:38:39.489 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-27T21:38:39.548 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-27T21:38:39.548 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-27T21:38:39.703 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-27T21:38:39.806 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-nbd-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:39.891 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:36 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:38:40.040 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:38:40.041 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:40.041 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-03-27T21:38:40.041 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:40.041 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-27T21:38:40.041 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 87 k 2024-03-27T21:38:40.041 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:40.041 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-27T21:38:40.041 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:40.041 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-27T21:38:40.041 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:40.041 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 87 k 2024-03-27T21:38:40.042 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 227 k 2024-03-27T21:38:40.042 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-27T21:38:40.273 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:40.332 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-fuse-18.2.2-1021.g7c37d80f.el9.x86_64.rpm 298 kB/s | 87 kB 00:00 2024-03-27T21:38:40.332 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:38:40.333 INFO:teuthology.orchestra.run.smithi191.stdout:Total 297 kB/s | 87 kB 00:00 2024-03-27T21:38:40.333 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-27T21:38:40.340 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-27T21:38:40.340 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-27T21:38:40.394 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-27T21:38:40.395 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-27T21:38:40.529 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-27T21:38:40.565 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-nbd-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:40.566 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:40.566 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-27T21:38:40.566 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:38:40.566 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:40.566 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-27T21:38:40.622 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-fuse-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:40.812 DEBUG:teuthology.parallel:result is None 2024-03-27T21:38:41.020 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:41.305 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-fuse-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:41.305 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:41.305 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-27T21:38:41.306 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:38:41.306 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:41.306 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:38:41.545 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-mirror 2024-03-27T21:38:42.117 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:39 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:38:42.273 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:38:42.273 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:42.274 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-27T21:38:42.274 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:42.274 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-27T21:38:42.274 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 3.0 M 2024-03-27T21:38:42.274 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:42.274 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-27T21:38:42.274 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:42.274 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-27T21:38:42.274 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:42.274 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 3.0 M 2024-03-27T21:38:42.274 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 12 M 2024-03-27T21:38:42.274 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-27T21:38:42.660 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-mirror-18.2.2-1021.g7c37d80f.el9.x86_64.rpm 7.8 MB/s | 3.0 MB 00:00 2024-03-27T21:38:42.660 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:38:42.661 INFO:teuthology.orchestra.run.smithi191.stdout:Total 7.7 MB/s | 3.0 MB 00:00 2024-03-27T21:38:42.661 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-27T21:38:42.670 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-27T21:38:42.670 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-27T21:38:42.742 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-27T21:38:42.742 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-27T21:38:43.157 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-27T21:38:43.227 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-mirror-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:43.251 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:43.251 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T21:38:43.251 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-27T21:38:43.251 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-27T21:38:43.251 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-27T21:38:43.252 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:44.355 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-mirror-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:44.355 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:44.355 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-27T21:38:44.355 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:38:44.355 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:44.355 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:38:44.578 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-nbd 2024-03-27T21:38:45.151 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:42 ago on Wed 27 Mar 2024 09:37:03 PM UTC. 2024-03-27T21:38:45.300 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-27T21:38:45.300 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:45.301 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-03-27T21:38:45.301 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:45.301 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-27T21:38:45.301 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd x86_64 2:18.2.2-1021.g7c37d80f.el9 ceph 172 k 2024-03-27T21:38:45.301 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:45.301 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-27T21:38:45.301 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-27T21:38:45.301 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-27T21:38:45.301 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:45.302 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 172 k 2024-03-27T21:38:45.302 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 494 k 2024-03-27T21:38:45.302 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-27T21:38:45.529 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-nbd-18.2.2-1021.g7c37d80f.el9.x86_64.rpm 757 kB/s | 172 kB 00:00 2024-03-27T21:38:45.530 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-27T21:38:45.530 INFO:teuthology.orchestra.run.smithi191.stdout:Total 752 kB/s | 172 kB 00:00 2024-03-27T21:38:45.530 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-27T21:38:45.538 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-27T21:38:45.538 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-27T21:38:45.594 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-27T21:38:45.594 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-27T21:38:45.736 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-27T21:38:45.828 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-nbd-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:46.262 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:46.554 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-nbd-2:18.2.2-1021.g7c37d80f.el9.x86_64 1/1 2024-03-27T21:38:46.554 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:46.554 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-27T21:38:46.554 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd-2:18.2.2-1021.g7c37d80f.el9.x86_64 2024-03-27T21:38:46.554 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:46.554 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-27T21:38:46.806 DEBUG:teuthology.parallel:result is None 2024-03-27T21:38:46.807 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7c37d80f7a88f4eb561c4fe02558660a650b10ac 2024-03-27T21:38:46.949 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-27T21:38:46.974 INFO:teuthology.orchestra.run.smithi060.stdout:18.2.2-1021.g7c37d80f.el9 2024-03-27T21:38:46.975 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1021.g7c37d80f.el9 2024-03-27T21:38:46.975 INFO:teuthology.task.install:The correct ceph version 18.2.2-1021.g7c37d80f is installed. 2024-03-27T21:38:46.977 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7c37d80f7a88f4eb561c4fe02558660a650b10ac 2024-03-27T21:38:47.120 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-27T21:38:47.142 INFO:teuthology.orchestra.run.smithi191.stdout:18.2.2-1021.g7c37d80f.el9 2024-03-27T21:38:47.142 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1021.g7c37d80f.el9 2024-03-27T21:38:47.142 INFO:teuthology.task.install:The correct ceph version 18.2.2-1021.g7c37d80f is installed. 2024-03-27T21:38:47.144 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-27T21:38:47.145 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:38:47.145 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-27T21:38:47.178 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-27T21:38:47.178 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-27T21:38:47.211 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-27T21:38:47.211 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:38:47.211 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/daemon-helper 2024-03-27T21:38:47.247 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-27T21:38:47.315 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-27T21:38:47.315 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/daemon-helper 2024-03-27T21:38:47.343 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-27T21:38:47.411 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-27T21:38:47.411 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:38:47.411 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-27T21:38:47.441 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-27T21:38:47.508 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-27T21:38:47.508 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-27T21:38:47.536 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-27T21:38:47.603 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-27T21:38:47.604 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:38:47.604 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/stdin-killer 2024-03-27T21:38:47.632 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-27T21:38:47.701 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-27T21:38:47.701 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/stdin-killer 2024-03-27T21:38:47.728 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-27T21:38:47.795 INFO:teuthology.run_tasks:Running task ceph... 2024-03-27T21:38:47.860 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-27T21:38:47.860 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 777 /var/log/ceph 2024-03-27T21:38:47.863 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /var/log/ceph 2024-03-27T21:38:47.890 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-27T21:38:47.890 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-27T21:38:47.932 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-27T21:38:47.959 INFO:tasks.ceph:Creating extra log directories... 2024-03-27T21:38:47.959 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-27T21:38:48.000 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-27T21:38:48.038 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-27T21:38:48.038 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-27T21:38:48.038 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-03-27_14:58:57-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626549', 'branch': 'wip-yuri4-testing-2024-03-26-1132-reef', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7626549', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-03-27_14:58:57-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2024-03-26-1132-reef'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'sha1': '7c37d80f7a88f4eb561c4fe02558660a650b10ac'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7c37d80f7a88f4eb561c4fe02558660a650b10ac'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2024-03-26-1132-reef', 'sha1': '7c37d80f7a88f4eb561c4fe02558660a650b10ac'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6904, 'sha1': '7c37d80f7a88f4eb561c4fe02558660a650b10ac', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri4-testing-2024-03-26-1132-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_7c37d80f7a88f4eb561c4fe02558660a650b10ac/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '7c37d80f7a88f4eb561c4fe02558660a650b10ac', 'targets': {'smithi060.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFseykjbMcoy37jwDf5haCtBthQp6+lf1FiaXkGRWge5FX0PR1O4cJ2V30pH6x72tmNxBTPzCx9KT7u+fUIv6h8=', 'smithi191.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFEoU/N98mvWvDGXsH9+Sn5SsdMdzH4v6vuk/Nq61Sr8dUL3puAnexOkO+5lZaf8xGqQoaGc/SGAsyBoAbeo/t0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-27_14:58:57', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-27T21:38:48.039 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-27T21:38:48.068 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-27T21:38:48.082 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/run/ceph 2024-03-27T21:38:48.123 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/run/ceph 2024-03-27T21:38:48.151 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:38:48.151 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/scratch_devs of=/dev/stdout 2024-03-27T21:38:48.203 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-27T21:38:48.203 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_1 2024-03-27T21:38:48.258 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-27T21:38:48.258 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T21:38:48.258 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-27T21:38:48.258 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T21:38:48.258 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T21:38:48.258 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-03-27 21:38:37.329643491 +0000 2024-03-27T21:38:48.258 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-03-27 21:35:51.468919558 +0000 2024-03-27T21:38:48.258 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-03-27 21:35:51.468919558 +0000 2024-03-27T21:38:48.258 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-03-27 21:35:51.468919558 +0000 2024-03-27T21:38:48.259 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-27T21:38:48.322 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-03-27T21:38:48.322 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-03-27T21:38:48.322 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 9.5743e-05 s, 5.3 MB/s 2024-03-27T21:38:48.323 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-27T21:38:48.378 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_2 2024-03-27T21:38:48.432 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-27T21:38:48.432 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T21:38:48.433 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-27T21:38:48.433 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T21:38:48.433 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T21:38:48.433 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-03-27 21:38:37.330643482 +0000 2024-03-27T21:38:48.433 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-03-27 21:35:51.788916640 +0000 2024-03-27T21:38:48.433 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-03-27 21:35:51.788916640 +0000 2024-03-27T21:38:48.433 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-03-27 21:35:51.788916640 +0000 2024-03-27T21:38:48.433 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-27T21:38:48.497 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-03-27T21:38:48.497 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-03-27T21:38:48.497 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 9.5649e-05 s, 5.4 MB/s 2024-03-27T21:38:48.498 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-27T21:38:48.553 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_3 2024-03-27T21:38:48.608 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-27T21:38:48.608 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T21:38:48.608 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-27T21:38:48.608 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T21:38:48.608 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T21:38:48.608 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-03-27 21:38:37.330643482 +0000 2024-03-27T21:38:48.608 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-03-27 21:35:52.136913465 +0000 2024-03-27T21:38:48.609 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-03-27 21:35:52.136913465 +0000 2024-03-27T21:38:48.609 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-03-27 21:35:52.136913465 +0000 2024-03-27T21:38:48.609 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-27T21:38:48.673 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-03-27T21:38:48.673 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-03-27T21:38:48.673 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 9.2434e-05 s, 5.5 MB/s 2024-03-27T21:38:48.674 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-27T21:38:48.729 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_4 2024-03-27T21:38:48.784 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-27T21:38:48.784 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T21:38:48.784 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-27T21:38:48.784 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T21:38:48.784 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T21:38:48.784 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-03-27 21:38:37.331643472 +0000 2024-03-27T21:38:48.784 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-03-27 21:35:52.470910419 +0000 2024-03-27T21:38:48.784 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-03-27 21:35:52.470910419 +0000 2024-03-27T21:38:48.784 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-03-27 21:35:52.470910419 +0000 2024-03-27T21:38:48.784 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-27T21:38:48.848 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-03-27T21:38:48.848 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-03-27T21:38:48.848 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 9.0439e-05 s, 5.7 MB/s 2024-03-27T21:38:48.851 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-27T21:38:48.905 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-27T21:38:48.906 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-27T21:38:48.906 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2024-03-27T21:38:48.921 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-27T21:38:48.921 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2024-03-27T21:38:48.975 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-27T21:38:48.975 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T21:38:48.975 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-27T21:38:48.975 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T21:38:48.975 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T21:38:48.975 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-27 21:38:43.535642700 +0000 2024-03-27T21:38:48.975 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-27 21:35:49.045000093 +0000 2024-03-27T21:38:48.975 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-27 21:35:49.045000093 +0000 2024-03-27T21:38:48.976 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-27 21:35:49.045000093 +0000 2024-03-27T21:38:48.976 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-27T21:38:49.040 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-27T21:38:49.041 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-27T21:38:49.041 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000183931 s, 2.8 MB/s 2024-03-27T21:38:49.042 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-27T21:38:49.097 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2024-03-27T21:38:49.153 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-27T21:38:49.153 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T21:38:49.153 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-27T21:38:49.153 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T21:38:49.153 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T21:38:49.153 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-27 21:38:43.536642682 +0000 2024-03-27T21:38:49.153 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-27 21:35:49.391994053 +0000 2024-03-27T21:38:49.153 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-27 21:35:49.391994053 +0000 2024-03-27T21:38:49.153 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-27 21:35:49.391994053 +0000 2024-03-27T21:38:49.153 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-27T21:38:49.219 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-27T21:38:49.219 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-27T21:38:49.219 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000172711 s, 3.0 MB/s 2024-03-27T21:38:49.220 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-27T21:38:49.277 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2024-03-27T21:38:49.333 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-27T21:38:49.333 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T21:38:49.333 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-27T21:38:49.333 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T21:38:49.333 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T21:38:49.333 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-27 21:38:43.536642682 +0000 2024-03-27T21:38:49.333 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-27 21:35:49.735988066 +0000 2024-03-27T21:38:49.333 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-27 21:35:49.735988066 +0000 2024-03-27T21:38:49.333 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-27 21:35:49.735988066 +0000 2024-03-27T21:38:49.333 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-27T21:38:49.398 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-27T21:38:49.398 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-27T21:38:49.399 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000209859 s, 2.4 MB/s 2024-03-27T21:38:49.400 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-27T21:38:49.455 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2024-03-27T21:38:49.509 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-27T21:38:49.509 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T21:38:49.509 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-27T21:38:49.509 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T21:38:49.510 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T21:38:49.510 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-27 21:38:43.536642682 +0000 2024-03-27T21:38:49.510 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-27 21:35:50.112981505 +0000 2024-03-27T21:38:49.510 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-27 21:35:50.112981505 +0000 2024-03-27T21:38:49.510 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-27 21:35:50.112981505 +0000 2024-03-27T21:38:49.510 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-27T21:38:49.575 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-27T21:38:49.575 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-27T21:38:49.575 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000220734 s, 2.3 MB/s 2024-03-27T21:38:49.576 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-27T21:38:49.631 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-27T21:38:49.631 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-27T21:38:49.631 INFO:tasks.ceph:Generating config... 2024-03-27T21:38:49.632 INFO:tasks.ceph:[client] debug monc = 20 2024-03-27T21:38:49.633 INFO:tasks.ceph:[client] debug ms = 1 2024-03-27T21:38:49.633 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-27T21:38:49.633 INFO:tasks.ceph:[client] debug rados = 20 2024-03-27T21:38:49.633 INFO:tasks.ceph:[global] lockdep = True 2024-03-27T21:38:49.633 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-27T21:38:49.633 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-27T21:38:49.633 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-27T21:38:49.633 INFO:tasks.ceph:[global] ms type = async 2024-03-27T21:38:49.633 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-27T21:38:49.633 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-27T21:38:49.633 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-27T21:38:49.633 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-27T21:38:49.633 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-27T21:38:49.634 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-27T21:38:49.634 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-27T21:38:49.634 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-27T21:38:49.634 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-27T21:38:49.634 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-27T21:38:49.634 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-27T21:38:49.634 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-27T21:38:49.634 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-27T21:38:49.634 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2024-03-27T21:38:49.634 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-27T21:38:49.634 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-27T21:38:49.634 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-27T21:38:49.634 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-27T21:38:49.635 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-27T21:38:49.635 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-27T21:38:49.635 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-27T21:38:49.635 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-27T21:38:49.635 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-27T21:38:49.635 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-27T21:38:49.635 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-27T21:38:49.635 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-27T21:38:49.635 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-27T21:38:49.635 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-27T21:38:49.635 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-27T21:38:49.635 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-27T21:38:49.635 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-27T21:38:49.635 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-27T21:38:49.635 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-27T21:38:49.636 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-27T21:38:49.636 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-27T21:38:49.636 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-27T21:38:49.636 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-27T21:38:49.636 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-27T21:38:49.636 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-27T21:38:49.636 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-27T21:38:49.636 INFO:tasks.ceph:Setting up mon.a... 2024-03-27T21:38:49.636 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-27T21:38:49.692 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/ceph.keyring 2024-03-27T21:38:49.695 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-27T21:38:49.742 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-27T21:38:49.769 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.60'), ('mon.c', '[v2:172.21.15.60:3301,v1:172.21.15.60:6790]'), ('mon.b', '172.21.15.191')] 2024-03-27T21:38:49.769 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.60,[v2:172.21.15.60:3301,v1:172.21.15.60:6790],172.21.15.191', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-27T21:38:49.770 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:38:49.770 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-27T21:38:49.825 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.60 --addv c '[v2:172.21.15.60:3301,v1:172.21.15.60:6790]' --add b 172.21.15.191 --print /home/ubuntu/cephtest/ceph.monmap 2024-03-27T21:38:49.908 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-27T21:38:49.908 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: generated fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:38:49.908 INFO:teuthology.orchestra.run.smithi060.stdout:setting min_mon_release = pacific 2024-03-27T21:38:49.909 INFO:teuthology.orchestra.run.smithi060.stdout:epoch 0 2024-03-27T21:38:49.909 INFO:teuthology.orchestra.run.smithi060.stdout:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:38:49.909 INFO:teuthology.orchestra.run.smithi060.stdout:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:38:49.909 INFO:teuthology.orchestra.run.smithi060.stdout:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:38:49.909 INFO:teuthology.orchestra.run.smithi060.stdout:min_mon_release 16 (pacific) 2024-03-27T21:38:49.909 INFO:teuthology.orchestra.run.smithi060.stdout:election_strategy: 3 2024-03-27T21:38:49.909 INFO:teuthology.orchestra.run.smithi060.stdout:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:38:49.909 INFO:teuthology.orchestra.run.smithi060.stdout:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:38:49.909 INFO:teuthology.orchestra.run.smithi060.stdout:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:38:49.910 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-27T21:38:49.911 DEBUG:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-27T21:38:49.925 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID dcd19d9a-535c-4b41-9095-d588268e2860... 2024-03-27T21:38:49.925 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-27T21:38:49.967 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-27T21:38:50.014 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2024-03-27T21:38:50.014 INFO:teuthology.orchestra.run.smithi060.stdout: chdir = "" 2024-03-27T21:38:50.014 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-27T21:38:50.015 INFO:teuthology.orchestra.run.smithi060.stdout: auth supported = cephx 2024-03-27T21:38:50.015 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.015 INFO:teuthology.orchestra.run.smithi060.stdout: filestore xattr use omap = true 2024-03-27T21:38:50.015 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.015 INFO:teuthology.orchestra.run.smithi060.stdout: mon clock drift allowed = 1.000 2024-03-27T21:38:50.015 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.015 INFO:teuthology.orchestra.run.smithi060.stdout: osd crush chooseleaf type = 0 2024-03-27T21:38:50.015 INFO:teuthology.orchestra.run.smithi060.stdout: auth debug = true 2024-03-27T21:38:50.015 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.015 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on old message = true 2024-03-27T21:38:50.015 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on bug = true 2024-03-27T21:38:50.016 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.016 INFO:teuthology.orchestra.run.smithi060.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-27T21:38:50.016 INFO:teuthology.orchestra.run.smithi060.stdout: mon pg warn max object skew = 0 2024-03-27T21:38:50.016 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.016 INFO:teuthology.orchestra.run.smithi060.stdout: # disable pg_autoscaler by default for new pools 2024-03-27T21:38:50.016 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-27T21:38:50.016 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.016 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default size = 2 2024-03-27T21:38:50.016 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.016 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow primary affinity = true 2024-03-27T21:38:50.016 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow pg remap = true 2024-03-27T21:38:50.017 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on legacy crush tunables = false 2024-03-27T21:38:50.017 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on crush straw calc version zero = false 2024-03-27T21:38:50.017 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on no sortbitwise = false 2024-03-27T21:38:50.017 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on osd down out interval zero = false 2024-03-27T21:38:50.017 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on too few osds = false 2024-03-27T21:38:50.017 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-27T21:38:50.017 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-27T21:38:50.017 INFO:teuthology.orchestra.run.smithi060.stdout: mon_allow_pool_size_one = true 2024-03-27T21:38:50.017 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.017 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-27T21:38:50.017 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.017 INFO:teuthology.orchestra.run.smithi060.stdout: osd default data pool replay window = 5 2024-03-27T21:38:50.017 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.017 INFO:teuthology.orchestra.run.smithi060.stdout: mon allow pool delete = true 2024-03-27T21:38:50.018 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.018 INFO:teuthology.orchestra.run.smithi060.stdout: mon cluster log file level = debug 2024-03-27T21:38:50.018 INFO:teuthology.orchestra.run.smithi060.stdout: debug asserts on shutdown = true 2024-03-27T21:38:50.018 INFO:teuthology.orchestra.run.smithi060.stdout: mon health detail to clog = false 2024-03-27T21:38:50.018 INFO:teuthology.orchestra.run.smithi060.stdout: mon host = "172.21.15.60,[v2:172.21.15.60:3301,v1:172.21.15.60:6790],172.21.15.191" 2024-03-27T21:38:50.018 INFO:teuthology.orchestra.run.smithi060.stdout: lockdep = True 2024-03-27T21:38:50.018 INFO:teuthology.orchestra.run.smithi060.stdout: mon client directed command retry = 5 2024-03-27T21:38:50.018 INFO:teuthology.orchestra.run.smithi060.stdout: mon election default strategy = 3 2024-03-27T21:38:50.018 INFO:teuthology.orchestra.run.smithi060.stdout: ms inject socket failures = 5000 2024-03-27T21:38:50.018 INFO:teuthology.orchestra.run.smithi060.stdout: ms type = async 2024-03-27T21:38:50.018 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:38:50.018 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.018 INFO:teuthology.orchestra.run.smithi060.stdout:[osd] 2024-03-27T21:38:50.018 INFO:teuthology.orchestra.run.smithi060.stdout: osd journal size = 100 2024-03-27T21:38:50.019 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.019 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub load threshold = 5.0 2024-03-27T21:38:50.019 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub max interval = 600 2024-03-27T21:38:50.019 INFO:teuthology.orchestra.run.smithi060.stdout: osd mclock profile = high_recovery_ops 2024-03-27T21:38:50.019 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.019 INFO:teuthology.orchestra.run.smithi060.stdout: osd recover clone overlap = true 2024-03-27T21:38:50.019 INFO:teuthology.orchestra.run.smithi060.stdout: osd recovery max chunk = 1048576 2024-03-27T21:38:50.019 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.019 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug shutdown = true 2024-03-27T21:38:50.019 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug op order = true 2024-03-27T21:38:50.019 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify stray on activate = true 2024-03-27T21:38:50.019 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.019 INFO:teuthology.orchestra.run.smithi060.stdout: osd open classes on start = true 2024-03-27T21:38:50.019 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug pg log writeout = true 2024-03-27T21:38:50.019 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.020 INFO:teuthology.orchestra.run.smithi060.stdout: osd deep scrub update digest min age = 30 2024-03-27T21:38:50.020 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.020 INFO:teuthology.orchestra.run.smithi060.stdout: osd map max advance = 10 2024-03-27T21:38:50.020 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.020 INFO:teuthology.orchestra.run.smithi060.stdout: journal zero on create = true 2024-03-27T21:38:50.020 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.020 INFO:teuthology.orchestra.run.smithi060.stdout: filestore ondisk finisher threads = 3 2024-03-27T21:38:50.020 INFO:teuthology.orchestra.run.smithi060.stdout: filestore apply finisher threads = 3 2024-03-27T21:38:50.020 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.020 INFO:teuthology.orchestra.run.smithi060.stdout: bdev debug aio = true 2024-03-27T21:38:50.020 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug misdirected ops = true 2024-03-27T21:38:50.020 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore block size = 96636764160 2024-03-27T21:38:50.020 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore compression algorithm = zlib 2024-03-27T21:38:50.020 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore compression mode = aggressive 2024-03-27T21:38:50.021 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore fsck on mount = True 2024-03-27T21:38:50.021 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore zero block detection = True 2024-03-27T21:38:50.021 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluefs = 20 2024-03-27T21:38:50.021 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluestore = 20 2024-03-27T21:38:50.021 INFO:teuthology.orchestra.run.smithi060.stdout: debug monc = 20 2024-03-27T21:38:50.021 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-03-27T21:38:50.021 INFO:teuthology.orchestra.run.smithi060.stdout: debug osd = 20 2024-03-27T21:38:50.021 INFO:teuthology.orchestra.run.smithi060.stdout: debug rocksdb = 10 2024-03-27T21:38:50.021 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd backfillfull_ratio = 0.85 2024-03-27T21:38:50.021 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd full ratio = 0.9 2024-03-27T21:38:50.021 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd nearfull ratio = 0.8 2024-03-27T21:38:50.021 INFO:teuthology.orchestra.run.smithi060.stdout: osd blocked scrub grace period = 3600 2024-03-27T21:38:50.021 INFO:teuthology.orchestra.run.smithi060.stdout: osd class default list = * 2024-03-27T21:38:50.022 INFO:teuthology.orchestra.run.smithi060.stdout: osd class load list = * 2024-03-27T21:38:50.022 INFO:teuthology.orchestra.run.smithi060.stdout: osd client watch timeout = 120 2024-03-27T21:38:50.022 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify cached snaps = True 2024-03-27T21:38:50.022 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify missing on start = True 2024-03-27T21:38:50.022 INFO:teuthology.orchestra.run.smithi060.stdout: osd failsafe full ratio = 0.95 2024-03-27T21:38:50.022 INFO:teuthology.orchestra.run.smithi060.stdout: osd max markdown count = 1000 2024-03-27T21:38:50.022 INFO:teuthology.orchestra.run.smithi060.stdout: osd mclock override recovery settings = True 2024-03-27T21:38:50.022 INFO:teuthology.orchestra.run.smithi060.stdout: osd objectstore = bluestore 2024-03-27T21:38:50.022 INFO:teuthology.orchestra.run.smithi060.stdout: osd op queue = debug_random 2024-03-27T21:38:50.022 INFO:teuthology.orchestra.run.smithi060.stdout: osd op queue cut off = debug_random 2024-03-27T21:38:50.022 INFO:teuthology.orchestra.run.smithi060.stdout: osd shutdown pgref assert = True 2024-03-27T21:38:50.022 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.022 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr] 2024-03-27T21:38:50.022 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-03-27T21:38:50.023 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgr = 20 2024-03-27T21:38:50.023 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2024-03-27T21:38:50.023 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-03-27T21:38:50.023 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-03-27T21:38:50.023 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-03-27T21:38:50.023 INFO:teuthology.orchestra.run.smithi060.stdout: mgr/telemetry/nag = false 2024-03-27T21:38:50.023 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.023 INFO:teuthology.orchestra.run.smithi060.stdout:[mon] 2024-03-27T21:38:50.023 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-03-27T21:38:50.023 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2024-03-27T21:38:50.023 INFO:teuthology.orchestra.run.smithi060.stdout: debug paxos = 20 2024-03-27T21:38:50.023 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-03-27T21:38:50.023 INFO:teuthology.orchestra.run.smithi060.stdout: mon data avail warn = 5 2024-03-27T21:38:50.023 INFO:teuthology.orchestra.run.smithi060.stdout: mon mgr mkfs grace = 240 2024-03-27T21:38:50.024 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-03-27T21:38:50.024 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd reporter subtree level = osd 2024-03-27T21:38:50.024 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd prime pg temp = true 2024-03-27T21:38:50.024 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-03-27T21:38:50.024 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.024 INFO:teuthology.orchestra.run.smithi060.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-27T21:38:50.024 INFO:teuthology.orchestra.run.smithi060.stdout: auth mon ticket ttl = 660 # 11m 2024-03-27T21:38:50.024 INFO:teuthology.orchestra.run.smithi060.stdout: auth service ticket ttl = 240 # 4m 2024-03-27T21:38:50.024 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.024 INFO:teuthology.orchestra.run.smithi060.stdout: # don't complain about insecure global_id in the test suite 2024-03-27T21:38:50.024 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-27T21:38:50.024 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-27T21:38:50.024 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.024 INFO:teuthology.orchestra.run.smithi060.stdout: # 1m isn't quite enough 2024-03-27T21:38:50.024 INFO:teuthology.orchestra.run.smithi060.stdout: mon_down_mkfs_grace = 2m 2024-03-27T21:38:50.025 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.025 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_filestore_osds = false 2024-03-27T21:38:50.025 INFO:teuthology.orchestra.run.smithi060.stdout: mon min osdmap epochs = 50 2024-03-27T21:38:50.025 INFO:teuthology.orchestra.run.smithi060.stdout: mon osdmap full prune interval = 2 2024-03-27T21:38:50.025 INFO:teuthology.orchestra.run.smithi060.stdout: mon osdmap full prune min = 15 2024-03-27T21:38:50.025 INFO:teuthology.orchestra.run.smithi060.stdout: mon osdmap full prune txsize = 2 2024-03-27T21:38:50.025 INFO:teuthology.orchestra.run.smithi060.stdout: mon scrub interval = 300 2024-03-27T21:38:50.025 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on pool no app = False 2024-03-27T21:38:50.025 INFO:teuthology.orchestra.run.smithi060.stdout: paxos service trim min = 10 2024-03-27T21:38:50.025 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:38:50.025 INFO:teuthology.orchestra.run.smithi060.stdout:[client] 2024-03-27T21:38:50.025 INFO:teuthology.orchestra.run.smithi060.stdout: rgw cache enabled = true 2024-03-27T21:38:50.025 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable ops log = true 2024-03-27T21:38:50.025 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable usage log = true 2024-03-27T21:38:50.025 INFO:teuthology.orchestra.run.smithi060.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-27T21:38:50.026 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-27T21:38:50.026 INFO:teuthology.orchestra.run.smithi060.stdout: debug monc = 20 2024-03-27T21:38:50.026 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-03-27T21:38:50.026 INFO:teuthology.orchestra.run.smithi060.stdout: debug objecter = 20 2024-03-27T21:38:50.026 INFO:teuthology.orchestra.run.smithi060.stdout: debug rados = 20 2024-03-27T21:38:50.026 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.a] 2024-03-27T21:38:50.026 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.c] 2024-03-27T21:38:50.026 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.b] 2024-03-27T21:38:50.026 INFO:teuthology.orchestra.run.smithi191.stdout:[global] 2024-03-27T21:38:50.027 INFO:teuthology.orchestra.run.smithi191.stdout: chdir = "" 2024-03-27T21:38:50.027 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-27T21:38:50.027 INFO:teuthology.orchestra.run.smithi191.stdout: auth supported = cephx 2024-03-27T21:38:50.027 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.027 INFO:teuthology.orchestra.run.smithi191.stdout: filestore xattr use omap = true 2024-03-27T21:38:50.027 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.027 INFO:teuthology.orchestra.run.smithi191.stdout: mon clock drift allowed = 1.000 2024-03-27T21:38:50.027 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.027 INFO:teuthology.orchestra.run.smithi191.stdout: osd crush chooseleaf type = 0 2024-03-27T21:38:50.027 INFO:teuthology.orchestra.run.smithi191.stdout: auth debug = true 2024-03-27T21:38:50.027 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.027 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on old message = true 2024-03-27T21:38:50.027 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on bug = true 2024-03-27T21:38:50.027 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.028 INFO:teuthology.orchestra.run.smithi191.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-27T21:38:50.028 INFO:teuthology.orchestra.run.smithi191.stdout: mon pg warn max object skew = 0 2024-03-27T21:38:50.028 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.028 INFO:teuthology.orchestra.run.smithi191.stdout: # disable pg_autoscaler by default for new pools 2024-03-27T21:38:50.028 INFO:teuthology.orchestra.run.smithi191.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-27T21:38:50.028 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.028 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default size = 2 2024-03-27T21:38:50.028 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.028 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow primary affinity = true 2024-03-27T21:38:50.028 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow pg remap = true 2024-03-27T21:38:50.028 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on legacy crush tunables = false 2024-03-27T21:38:50.028 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on crush straw calc version zero = false 2024-03-27T21:38:50.028 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on no sortbitwise = false 2024-03-27T21:38:50.029 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on osd down out interval zero = false 2024-03-27T21:38:50.029 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on too few osds = false 2024-03-27T21:38:50.029 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-27T21:38:50.029 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-27T21:38:50.029 INFO:teuthology.orchestra.run.smithi191.stdout: mon_allow_pool_size_one = true 2024-03-27T21:38:50.029 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.029 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-27T21:38:50.029 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.029 INFO:teuthology.orchestra.run.smithi191.stdout: osd default data pool replay window = 5 2024-03-27T21:38:50.029 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.029 INFO:teuthology.orchestra.run.smithi191.stdout: mon allow pool delete = true 2024-03-27T21:38:50.029 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.029 INFO:teuthology.orchestra.run.smithi191.stdout: mon cluster log file level = debug 2024-03-27T21:38:50.029 INFO:teuthology.orchestra.run.smithi191.stdout: debug asserts on shutdown = true 2024-03-27T21:38:50.029 INFO:teuthology.orchestra.run.smithi191.stdout: mon health detail to clog = false 2024-03-27T21:38:50.030 INFO:teuthology.orchestra.run.smithi191.stdout: mon host = "172.21.15.60,[v2:172.21.15.60:3301,v1:172.21.15.60:6790],172.21.15.191" 2024-03-27T21:38:50.030 INFO:teuthology.orchestra.run.smithi191.stdout: lockdep = True 2024-03-27T21:38:50.030 INFO:teuthology.orchestra.run.smithi191.stdout: mon client directed command retry = 5 2024-03-27T21:38:50.030 INFO:teuthology.orchestra.run.smithi191.stdout: mon election default strategy = 3 2024-03-27T21:38:50.030 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject socket failures = 5000 2024-03-27T21:38:50.030 INFO:teuthology.orchestra.run.smithi191.stdout: ms type = async 2024-03-27T21:38:50.030 INFO:teuthology.orchestra.run.smithi191.stdout: fsid = dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:38:50.030 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.030 INFO:teuthology.orchestra.run.smithi191.stdout:[osd] 2024-03-27T21:38:50.030 INFO:teuthology.orchestra.run.smithi191.stdout: osd journal size = 100 2024-03-27T21:38:50.030 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.030 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub load threshold = 5.0 2024-03-27T21:38:50.030 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub max interval = 600 2024-03-27T21:38:50.030 INFO:teuthology.orchestra.run.smithi191.stdout: osd mclock profile = high_recovery_ops 2024-03-27T21:38:50.030 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.031 INFO:teuthology.orchestra.run.smithi191.stdout: osd recover clone overlap = true 2024-03-27T21:38:50.031 INFO:teuthology.orchestra.run.smithi191.stdout: osd recovery max chunk = 1048576 2024-03-27T21:38:50.031 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.031 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug shutdown = true 2024-03-27T21:38:50.031 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug op order = true 2024-03-27T21:38:50.031 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify stray on activate = true 2024-03-27T21:38:50.031 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.031 INFO:teuthology.orchestra.run.smithi191.stdout: osd open classes on start = true 2024-03-27T21:38:50.031 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug pg log writeout = true 2024-03-27T21:38:50.031 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.031 INFO:teuthology.orchestra.run.smithi191.stdout: osd deep scrub update digest min age = 30 2024-03-27T21:38:50.031 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.031 INFO:teuthology.orchestra.run.smithi191.stdout: osd map max advance = 10 2024-03-27T21:38:50.031 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.032 INFO:teuthology.orchestra.run.smithi191.stdout: journal zero on create = true 2024-03-27T21:38:50.032 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.032 INFO:teuthology.orchestra.run.smithi191.stdout: filestore ondisk finisher threads = 3 2024-03-27T21:38:50.032 INFO:teuthology.orchestra.run.smithi191.stdout: filestore apply finisher threads = 3 2024-03-27T21:38:50.032 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.032 INFO:teuthology.orchestra.run.smithi191.stdout: bdev debug aio = true 2024-03-27T21:38:50.032 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug misdirected ops = true 2024-03-27T21:38:50.032 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore block size = 96636764160 2024-03-27T21:38:50.032 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore compression algorithm = zlib 2024-03-27T21:38:50.032 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore compression mode = aggressive 2024-03-27T21:38:50.032 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore fsck on mount = True 2024-03-27T21:38:50.032 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore zero block detection = True 2024-03-27T21:38:50.032 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluefs = 20 2024-03-27T21:38:50.033 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluestore = 20 2024-03-27T21:38:50.033 INFO:teuthology.orchestra.run.smithi191.stdout: debug monc = 20 2024-03-27T21:38:50.033 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-03-27T21:38:50.033 INFO:teuthology.orchestra.run.smithi191.stdout: debug osd = 20 2024-03-27T21:38:50.033 INFO:teuthology.orchestra.run.smithi191.stdout: debug rocksdb = 10 2024-03-27T21:38:50.033 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd backfillfull_ratio = 0.85 2024-03-27T21:38:50.033 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd full ratio = 0.9 2024-03-27T21:38:50.033 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd nearfull ratio = 0.8 2024-03-27T21:38:50.033 INFO:teuthology.orchestra.run.smithi191.stdout: osd blocked scrub grace period = 3600 2024-03-27T21:38:50.033 INFO:teuthology.orchestra.run.smithi191.stdout: osd class default list = * 2024-03-27T21:38:50.033 INFO:teuthology.orchestra.run.smithi191.stdout: osd class load list = * 2024-03-27T21:38:50.033 INFO:teuthology.orchestra.run.smithi191.stdout: osd client watch timeout = 120 2024-03-27T21:38:50.033 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify cached snaps = True 2024-03-27T21:38:50.033 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify missing on start = True 2024-03-27T21:38:50.033 INFO:teuthology.orchestra.run.smithi191.stdout: osd failsafe full ratio = 0.95 2024-03-27T21:38:50.034 INFO:teuthology.orchestra.run.smithi191.stdout: osd max markdown count = 1000 2024-03-27T21:38:50.034 INFO:teuthology.orchestra.run.smithi191.stdout: osd mclock override recovery settings = True 2024-03-27T21:38:50.034 INFO:teuthology.orchestra.run.smithi191.stdout: osd objectstore = bluestore 2024-03-27T21:38:50.034 INFO:teuthology.orchestra.run.smithi191.stdout: osd op queue = debug_random 2024-03-27T21:38:50.034 INFO:teuthology.orchestra.run.smithi191.stdout: osd op queue cut off = debug_random 2024-03-27T21:38:50.034 INFO:teuthology.orchestra.run.smithi191.stdout: osd shutdown pgref assert = True 2024-03-27T21:38:50.034 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.034 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr] 2024-03-27T21:38:50.034 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-03-27T21:38:50.034 INFO:teuthology.orchestra.run.smithi191.stdout: debug mgr = 20 2024-03-27T21:38:50.034 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2024-03-27T21:38:50.034 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2024-03-27T21:38:50.035 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2024-03-27T21:38:50.035 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2024-03-27T21:38:50.035 INFO:teuthology.orchestra.run.smithi191.stdout: mgr/telemetry/nag = false 2024-03-27T21:38:50.035 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.035 INFO:teuthology.orchestra.run.smithi191.stdout:[mon] 2024-03-27T21:38:50.035 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-03-27T21:38:50.035 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2024-03-27T21:38:50.035 INFO:teuthology.orchestra.run.smithi191.stdout: debug paxos = 20 2024-03-27T21:38:50.035 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2024-03-27T21:38:50.035 INFO:teuthology.orchestra.run.smithi191.stdout: mon data avail warn = 5 2024-03-27T21:38:50.035 INFO:teuthology.orchestra.run.smithi191.stdout: mon mgr mkfs grace = 240 2024-03-27T21:38:50.035 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2024-03-27T21:38:50.035 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd reporter subtree level = osd 2024-03-27T21:38:50.035 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd prime pg temp = true 2024-03-27T21:38:50.036 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2024-03-27T21:38:50.036 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.036 INFO:teuthology.orchestra.run.smithi191.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-27T21:38:50.036 INFO:teuthology.orchestra.run.smithi191.stdout: auth mon ticket ttl = 660 # 11m 2024-03-27T21:38:50.036 INFO:teuthology.orchestra.run.smithi191.stdout: auth service ticket ttl = 240 # 4m 2024-03-27T21:38:50.036 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.036 INFO:teuthology.orchestra.run.smithi191.stdout: # don't complain about insecure global_id in the test suite 2024-03-27T21:38:50.036 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-27T21:38:50.036 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-27T21:38:50.036 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.036 INFO:teuthology.orchestra.run.smithi191.stdout: # 1m isn't quite enough 2024-03-27T21:38:50.036 INFO:teuthology.orchestra.run.smithi191.stdout: mon_down_mkfs_grace = 2m 2024-03-27T21:38:50.036 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.036 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_filestore_osds = false 2024-03-27T21:38:50.036 INFO:teuthology.orchestra.run.smithi191.stdout: mon min osdmap epochs = 50 2024-03-27T21:38:50.037 INFO:teuthology.orchestra.run.smithi191.stdout: mon osdmap full prune interval = 2 2024-03-27T21:38:50.037 INFO:teuthology.orchestra.run.smithi191.stdout: mon osdmap full prune min = 15 2024-03-27T21:38:50.037 INFO:teuthology.orchestra.run.smithi191.stdout: mon osdmap full prune txsize = 2 2024-03-27T21:38:50.037 INFO:teuthology.orchestra.run.smithi191.stdout: mon scrub interval = 300 2024-03-27T21:38:50.037 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on pool no app = False 2024-03-27T21:38:50.037 INFO:teuthology.orchestra.run.smithi191.stdout: paxos service trim min = 10 2024-03-27T21:38:50.037 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-27T21:38:50.037 INFO:teuthology.orchestra.run.smithi191.stdout:[client] 2024-03-27T21:38:50.037 INFO:teuthology.orchestra.run.smithi191.stdout: rgw cache enabled = true 2024-03-27T21:38:50.037 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable ops log = true 2024-03-27T21:38:50.037 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable usage log = true 2024-03-27T21:38:50.037 INFO:teuthology.orchestra.run.smithi191.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-27T21:38:50.037 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-27T21:38:50.037 INFO:teuthology.orchestra.run.smithi191.stdout: debug monc = 20 2024-03-27T21:38:50.038 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-03-27T21:38:50.038 INFO:teuthology.orchestra.run.smithi191.stdout: debug objecter = 20 2024-03-27T21:38:50.038 INFO:teuthology.orchestra.run.smithi191.stdout: debug rados = 20 2024-03-27T21:38:50.038 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.a] 2024-03-27T21:38:50.038 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.c] 2024-03-27T21:38:50.038 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.b] 2024-03-27T21:38:50.039 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-27T21:38:50.039 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-27T21:38:50.094 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-27T21:38:50.095 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:38:50.095 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-27T21:38:50.110 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:38:50.110 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-27T21:38:50.163 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi060.front.sepia.ceph.com 2024-03-27T21:38:50.164 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:38:50.164 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-27T21:38:50.164 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-27T21:38:50.242 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:38:50.242 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-27T21:38:50.297 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi191.front.sepia.ceph.com 2024-03-27T21:38:50.297 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-27T21:38:50.297 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-27T21:38:50.297 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-27T21:38:50.337 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-27T21:38:50.337 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-27T21:38:50.392 INFO:tasks.ceph:Setting up mon nodes... 2024-03-27T21:38:50.392 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-27T21:38:50.392 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-27T21:38:50.446 INFO:teuthology.orchestra.run.smithi060.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-27T21:38:50.448 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-27T21:38:50.507 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-27T21:38:50.509 INFO:tasks.ceph:Setting up mds nodes... 2024-03-27T21:38:50.510 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-27T21:38:50.510 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-27T21:38:50.552 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-27T21:38:50.567 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-27T21:38:50.609 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-27T21:38:50.624 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-27T21:38:50.624 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-27T21:38:50.624 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-27T21:38:50.651 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-27T21:38:50.651 INFO:tasks.ceph:role: osd.0 2024-03-27T21:38:50.651 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com 2024-03-27T21:38:50.651 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-27T21:38:50.717 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T21:38:50.717 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T21:38:50.717 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T21:38:50.717 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T21:38:50.717 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T21:38:50.717 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-03-27T21:38:50.718 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T21:38:50.718 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T21:38:50.718 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T21:38:50.718 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T21:38:50.754 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-03-27T21:38:50.756 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-03-27T21:38:50.756 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-27T21:38:50.826 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-27T21:38:50.896 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-27T21:38:50.963 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-27T21:38:50.963 INFO:tasks.ceph:role: osd.1 2024-03-27T21:38:50.963 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com 2024-03-27T21:38:50.963 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-27T21:38:51.028 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T21:38:51.028 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T21:38:51.028 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T21:38:51.028 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T21:38:51.029 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T21:38:51.029 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-03-27T21:38:51.029 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T21:38:51.029 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T21:38:51.029 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T21:38:51.029 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T21:38:51.065 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-03-27T21:38:51.067 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-03-27T21:38:51.067 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-27T21:38:51.137 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-27T21:38:51.208 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-27T21:38:51.274 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-27T21:38:51.274 INFO:tasks.ceph:role: osd.2 2024-03-27T21:38:51.274 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com 2024-03-27T21:38:51.274 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-27T21:38:51.340 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T21:38:51.340 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T21:38:51.340 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T21:38:51.340 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T21:38:51.340 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T21:38:51.340 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-03-27T21:38:51.340 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T21:38:51.340 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T21:38:51.341 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T21:38:51.341 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T21:38:51.376 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-03-27T21:38:51.377 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-03-27T21:38:51.377 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-27T21:38:51.447 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-27T21:38:51.521 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-27T21:38:51.587 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-27T21:38:51.587 INFO:tasks.ceph:role: osd.3 2024-03-27T21:38:51.587 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi060.front.sepia.ceph.com 2024-03-27T21:38:51.587 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-27T21:38:51.652 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T21:38:51.653 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T21:38:51.653 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T21:38:51.653 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T21:38:51.653 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T21:38:51.653 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-03-27T21:38:51.653 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T21:38:51.653 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T21:38:51.653 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T21:38:51.653 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T21:38:51.691 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-03-27T21:38:51.693 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-03-27T21:38:51.693 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-27T21:38:51.768 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-27T21:38:51.840 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T21:38:51.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:38:51.947+0000 7feddf138640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-27T21:38:51.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:38:51.947+0000 7feddf138640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-27T21:38:51.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:38:51.947+0000 7feddf138640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-27T21:38:51.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:38:51.947+0000 7feddf138640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-27T21:38:51.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:38:51.947+0000 7feddf138640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-27T21:38:51.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:38:51.947+0000 7feddf138640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-27T21:38:54.776 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-27T21:38:54.808 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T21:38:54.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:38:54.913+0000 7f42892d9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-27T21:38:54.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:38:54.913+0000 7f42892d9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-27T21:38:54.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:38:54.913+0000 7f42892d9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-27T21:38:54.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:38:54.913+0000 7f42892d9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-27T21:38:54.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:38:54.913+0000 7f42892d9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-27T21:38:54.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:38:54.913+0000 7f42892d9640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-27T21:38:57.756 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-27T21:38:57.787 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T21:38:57.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:38:57.892+0000 7f08d18d3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-27T21:38:57.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:38:57.892+0000 7f08d18d3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-27T21:38:57.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:38:57.892+0000 7f08d18d3640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-27T21:38:57.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:38:57.892+0000 7f08d18d3640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-27T21:38:57.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:38:57.892+0000 7f08d18d3640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-27T21:38:57.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:38:57.892+0000 7f08d18d3640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-27T21:39:00.727 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-27T21:39:00.757 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T21:39:00.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:00.861+0000 7f621e213640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-27T21:39:00.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:00.861+0000 7f621e213640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-27T21:39:00.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:00.861+0000 7f621e213640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-27T21:39:00.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:00.861+0000 7f621e213640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-27T21:39:00.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:00.861+0000 7f621e213640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-27T21:39:00.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:00.862+0000 7f621e213640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-27T21:39:03.705 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-27T21:39:03.737 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-27T21:39:03.770 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-27T21:39:03.770 INFO:tasks.ceph:role: osd.4 2024-03-27T21:39:03.770 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-27T21:39:03.770 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-27T21:39:03.847 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T21:39:03.847 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T21:39:03.847 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T21:39:03.847 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T21:39:03.848 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T21:39:03.848 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-03-27T21:39:03.848 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T21:39:03.848 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T21:39:03.848 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T21:39:03.848 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T21:39:04.322 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-03-27T21:39:04.324 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-03-27T21:39:04.324 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-27T21:39:04.463 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-27T21:39:04.500 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-27T21:39:04.569 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-27T21:39:04.569 INFO:tasks.ceph:role: osd.5 2024-03-27T21:39:04.569 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-27T21:39:04.570 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-27T21:39:04.645 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T21:39:04.646 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T21:39:04.646 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T21:39:04.646 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T21:39:04.646 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T21:39:04.646 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-03-27T21:39:04.646 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T21:39:04.646 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T21:39:04.646 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T21:39:04.646 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T21:39:05.109 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-03-27T21:39:05.111 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-03-27T21:39:05.112 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-27T21:39:05.206 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-27T21:39:05.239 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-27T21:39:05.308 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-27T21:39:05.309 INFO:tasks.ceph:role: osd.6 2024-03-27T21:39:05.309 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-27T21:39:05.309 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-27T21:39:05.384 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T21:39:05.385 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T21:39:05.385 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T21:39:05.385 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T21:39:05.385 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T21:39:05.385 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-03-27T21:39:05.385 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T21:39:05.385 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T21:39:05.385 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T21:39:05.385 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T21:39:05.871 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-03-27T21:39:05.873 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-03-27T21:39:05.873 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-27T21:39:06.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-27T21:39:06.044 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-27T21:39:06.116 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-27T21:39:06.116 INFO:tasks.ceph:role: osd.7 2024-03-27T21:39:06.116 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-27T21:39:06.116 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-27T21:39:06.192 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T21:39:06.193 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T21:39:06.193 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T21:39:06.193 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T21:39:06.193 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T21:39:06.193 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-03-27T21:39:06.193 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T21:39:06.193 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T21:39:06.193 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T21:39:06.193 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T21:39:06.682 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-03-27T21:39:06.684 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-03-27T21:39:06.685 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-27T21:39:06.827 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-27T21:39:06.862 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T21:39:06.960 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:06.973+0000 7f1c624b5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-27T21:39:06.961 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:06.974+0000 7f1c624b5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-27T21:39:06.961 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:06.974+0000 7f1c624b5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-27T21:39:06.961 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:06.974+0000 7f1c624b5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-27T21:39:06.961 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:06.974+0000 7f1c624b5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-27T21:39:06.961 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:06.974+0000 7f1c624b5640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-27T21:39:09.817 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-27T21:39:09.848 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T21:39:09.939 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:09.953+0000 7fd315622640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-27T21:39:09.939 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:09.953+0000 7fd315622640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-27T21:39:09.939 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:09.953+0000 7fd315622640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-27T21:39:09.939 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:09.953+0000 7fd315622640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-27T21:39:09.939 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:09.953+0000 7fd315622640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-27T21:39:09.940 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:09.953+0000 7fd315622640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-27T21:39:12.778 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-27T21:39:12.807 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T21:39:12.895 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:12.908+0000 7f2c16c20640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-27T21:39:12.895 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:12.909+0000 7f2c16c20640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-27T21:39:12.895 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:12.909+0000 7f2c16c20640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-27T21:39:12.895 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:12.909+0000 7f2c16c20640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-27T21:39:12.895 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:12.909+0000 7f2c16c20640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-27T21:39:12.896 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:12.909+0000 7f2c16c20640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-27T21:39:15.746 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-27T21:39:15.776 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T21:39:15.865 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:15.878+0000 7fde2ed1d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-27T21:39:15.865 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:15.879+0000 7fde2ed1d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-27T21:39:15.865 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:15.879+0000 7fde2ed1d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-27T21:39:15.865 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:15.879+0000 7fde2ed1d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-27T21:39:15.865 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:15.879+0000 7fde2ed1d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-27T21:39:15.866 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:15.879+0000 7fde2ed1d640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-27T21:39:18.726 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-27T21:39:18.757 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-27T21:39:18.758 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:39:18.758 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-27T21:39:18.789 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:39:18.789 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-27T21:39:18.863 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:39:18.863 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-27T21:39:18.933 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:39:18.933 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-27T21:39:19.002 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:39:19.002 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-27T21:39:19.071 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-27T21:39:19.071 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-27T21:39:19.099 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-27T21:39:19.099 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-27T21:39:19.166 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-27T21:39:19.166 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-27T21:39:19.233 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-27T21:39:19.234 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-27T21:39:19.307 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-27T21:39:19.308 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-27T21:39:19.375 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:39:19.375 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-27T21:39:19.392 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-27T21:39:19.392 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-27T21:39:19.429 INFO:tasks.ceph:Adding keys to all mons... 2024-03-27T21:39:19.429 DEBUG:teuthology.orchestra.run.smithi060:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-27T21:39:19.435 DEBUG:teuthology.orchestra.run.smithi191:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-27T21:39:19.473 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr.y] 2024-03-27T21:39:19.473 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDqkQRmYih1GxAApv8QB/lxV3d2z45qFLv2Yw== 2024-03-27T21:39:19.473 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.0] 2024-03-27T21:39:19.473 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDrkQRmemqHOBAAou6Q6SeURKbutUA7Fqb+Xw== 2024-03-27T21:39:19.473 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.1] 2024-03-27T21:39:19.473 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDukQRmZoF+NhAAtHpTDlhC8trIBuunilbMGQ== 2024-03-27T21:39:19.473 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.2] 2024-03-27T21:39:19.473 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDxkQRme/46NRAAz7Gm6K8fuP57QGUF/r4+pA== 2024-03-27T21:39:19.473 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.3] 2024-03-27T21:39:19.473 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQD0kQRmkZpoMxAAdN1QJYsLcXrCKocMROlsNA== 2024-03-27T21:39:19.474 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr.x] 2024-03-27T21:39:19.474 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDqkQRm/xQZHxAA2+fNL6muQu9DFL1HhrHgSw== 2024-03-27T21:39:19.474 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.4] 2024-03-27T21:39:19.474 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQD6kQRmC/8OOhAAV6tvnqY94EonZ5bKUbEY9Q== 2024-03-27T21:39:19.474 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.5] 2024-03-27T21:39:19.474 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQD9kQRmTHzTOBAAd1HdOW6+YppaAHpciNAtMg== 2024-03-27T21:39:19.474 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.6] 2024-03-27T21:39:19.474 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAAkgRmq+wvNhAAdK9M3jt8Zb71POsicijuXQ== 2024-03-27T21:39:19.474 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.7] 2024-03-27T21:39:19.474 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQADkgRm9xZnNBAA2vOD+6aUz8+xWu0XM96E7Q== 2024-03-27T21:39:19.474 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2024-03-27T21:39:19.474 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDqkQRmBGzHIRAAq+xukIrECFJdowXpA276Fw== 2024-03-27T21:39:19.474 INFO:teuthology.orchestra.run.smithi060.stdout:[client.1] 2024-03-27T21:39:19.475 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQDqkQRmascuJRAAr0lVlYem6UTqCt517x+4OA== 2024-03-27T21:39:19.495 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr.y] 2024-03-27T21:39:19.495 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDqkQRmYih1GxAApv8QB/lxV3d2z45qFLv2Yw== 2024-03-27T21:39:19.495 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.0] 2024-03-27T21:39:19.495 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDrkQRmemqHOBAAou6Q6SeURKbutUA7Fqb+Xw== 2024-03-27T21:39:19.495 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.1] 2024-03-27T21:39:19.495 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDukQRmZoF+NhAAtHpTDlhC8trIBuunilbMGQ== 2024-03-27T21:39:19.496 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.2] 2024-03-27T21:39:19.496 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDxkQRme/46NRAAz7Gm6K8fuP57QGUF/r4+pA== 2024-03-27T21:39:19.496 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.3] 2024-03-27T21:39:19.496 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQD0kQRmkZpoMxAAdN1QJYsLcXrCKocMROlsNA== 2024-03-27T21:39:19.496 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr.x] 2024-03-27T21:39:19.496 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDqkQRm/xQZHxAA2+fNL6muQu9DFL1HhrHgSw== 2024-03-27T21:39:19.496 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.4] 2024-03-27T21:39:19.496 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQD6kQRmC/8OOhAAV6tvnqY94EonZ5bKUbEY9Q== 2024-03-27T21:39:19.496 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.5] 2024-03-27T21:39:19.496 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQD9kQRmTHzTOBAAd1HdOW6+YppaAHpciNAtMg== 2024-03-27T21:39:19.496 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.6] 2024-03-27T21:39:19.496 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAAkgRmq+wvNhAAdK9M3jt8Zb71POsicijuXQ== 2024-03-27T21:39:19.496 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.7] 2024-03-27T21:39:19.496 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQADkgRm9xZnNBAA2vOD+6aUz8+xWu0XM96E7Q== 2024-03-27T21:39:19.497 INFO:teuthology.orchestra.run.smithi191.stdout:[client.0] 2024-03-27T21:39:19.497 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDqkQRmBGzHIRAAq+xukIrECFJdowXpA276Fw== 2024-03-27T21:39:19.497 INFO:teuthology.orchestra.run.smithi191.stdout:[client.1] 2024-03-27T21:39:19.497 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDqkQRmascuJRAAr0lVlYem6UTqCt517x+4OA== 2024-03-27T21:39:19.498 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-27T21:39:19.518 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-27T21:39:19.590 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T21:39:19.593 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T21:39:19.645 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T21:39:19.647 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T21:39:19.698 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T21:39:19.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T21:39:19.752 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T21:39:19.754 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T21:39:19.805 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-27T21:39:19.808 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-27T21:39:19.862 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T21:39:19.865 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T21:39:19.916 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T21:39:19.918 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T21:39:19.969 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T21:39:19.972 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T21:39:20.024 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T21:39:20.026 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T21:39:20.078 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-27T21:39:20.080 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-27T21:39:20.136 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-27T21:39:20.138 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-27T21:39:20.193 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-27T21:39:20.194 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-27T21:39:20.222 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-27T21:39:20.318 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-27T21:39:20.344 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-27T21:39:20.411 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-27T21:39:20.506 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-27T21:39:20.533 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-27T21:39:20.560 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-27T21:39:20.658 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-27T21:39:20.687 DEBUG:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-27T21:39:20.689 DEBUG:teuthology.orchestra.run.smithi191:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-27T21:39:20.742 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-27T21:39:20.742 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-27T21:39:20.742 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-27T21:39:20.744 INFO:tasks.ceph.mon.a:Started 2024-03-27T21:39:20.745 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-27T21:39:20.745 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-27T21:39:20.746 INFO:tasks.ceph.mon.c:Started 2024-03-27T21:39:20.747 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-27T21:39:20.747 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-27T21:39:20.784 INFO:tasks.ceph.mon.b:Started 2024-03-27T21:39:20.784 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-27T21:39:20.785 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-27T21:39:20.785 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-27T21:39:20.787 INFO:tasks.ceph.mgr.y:Started 2024-03-27T21:39:20.788 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-27T21:39:20.788 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-27T21:39:20.790 INFO:tasks.ceph.mgr.x:Started 2024-03-27T21:39:20.790 INFO:tasks.ceph:Setting crush tunables to default 2024-03-27T21:39:20.790 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd crush tunables default 2024-03-27T21:39:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.906+0000 7f4d0e42f640 10 monclient: get_monmap_and_config 2024-03-27T21:39:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.906+0000 7f4d0e42f640 10 monclient: build_initial_monmap 2024-03-27T21:39:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.906+0000 7f4d0e42f640 10 monclient: monmap: 2024-03-27T21:39:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:20.907268+0000 2024-03-27T21:39:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:20.907268+0000 2024-03-27T21:39:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:20.894 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:20.894 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:20.894 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:20.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.906+0000 7f4d0e42f640 1 Processor -- start 2024-03-27T21:39:20.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0e42f640 1 -- start start 2024-03-27T21:39:20.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0e42f640 10 monclient: init 2024-03-27T21:39:20.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0e42f640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:20.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0e42f640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-03-27T21:39:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0e42f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d080b40b0 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0e42f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 0x7f4d0813b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0e42f640 10 monclient(hunting): picked mon.noname-c con 0x7f4d0813b020 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0e42f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 0x7f4d08141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0e42f640 10 monclient(hunting): picked mon.noname-b con 0x7f4d081416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0e42f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d080b3830 con 0x7f4d080b40b0 2024-03-27T21:39:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0e42f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0e42f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d080b4490 con 0x7f4d0813b020 2024-03-27T21:39:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0e42f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0e42f640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d080aa520 con 0x7f4d081416f0 2024-03-27T21:39:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0e42f640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0e42f640 10 monclient(hunting): authenticate will time out at 802.128523s 2024-03-27T21:39:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0c9cc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 0x7f4d08141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0c9cc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 0x7f4d08141b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:37642/0 (socket says 172.21.15.60:37642) 2024-03-27T21:39:20.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0c9cc640 1 -- 172.21.15.60:0/3236694290 learned_addr learned my addr 172.21.15.60:0/3236694290 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:20.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0c9cc640 10 monclient(hunting): get_auth_request con 0x7f4d081416f0 auth_method 0 2024-03-27T21:39:20.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0c9cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:20.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0c9cc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:20.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0c9cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:20.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0c9cc640 1 -- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 msgr2=0x7f4d08141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-27T21:39:20.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0c9cc640 1 -- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 msgr2=0x7f4d08141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-27T21:39:20.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0c9cc640 1 --2- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 0x7f4d08141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-27T21:39:20.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d0c9cc640 1 --2- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 0x7f4d08141b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-27T21:39:20.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d077fe640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 0x7f4d0813b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:20.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d077fe640 10 monclient(hunting): get_auth_request con 0x7f4d0813b020 auth_method 0 2024-03-27T21:39:20.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:20.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d077fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:20.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.907+0000 7f4d077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:20.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.908+0000 7f4d077fe640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 msgr2=0x7f4d0813b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-27T21:39:20.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.908+0000 7f4d077fe640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 msgr2=0x7f4d0813b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-27T21:39:20.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.908+0000 7f4d077fe640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 0x7f4d0813b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-27T21:39:20.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:20.908+0000 7f4d077fe640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 0x7f4d0813b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-27T21:39:21.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.108+0000 7f4d0c9cc640 1 --2- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 0x7f4d08141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:21.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.108+0000 7f4d0c9cc640 10 monclient(hunting): get_auth_request con 0x7f4d081416f0 auth_method 0 2024-03-27T21:39:21.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.108+0000 7f4d0c9cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:21.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.108+0000 7f4d0c9cc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:21.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.108+0000 7f4d0c9cc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:39:21.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.108+0000 7f4d0c9cc640 1 -- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 msgr2=0x7f4d08141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-27T21:39:21.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.108+0000 7f4d0c9cc640 1 -- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 msgr2=0x7f4d08141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-27T21:39:21.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.108+0000 7f4d0c9cc640 1 --2- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 0x7f4d08141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-27T21:39:21.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.108+0000 7f4d0c9cc640 1 --2- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 0x7f4d08141b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-27T21:39:21.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.109+0000 7f4d077fe640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 0x7f4d0813b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:21.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.109+0000 7f4d077fe640 10 monclient(hunting): get_auth_request con 0x7f4d0813b020 auth_method 0 2024-03-27T21:39:21.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.109+0000 7f4d077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:21.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.109+0000 7f4d077fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:21.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.109+0000 7f4d077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:39:21.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.109+0000 7f4d077fe640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 msgr2=0x7f4d0813b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-27T21:39:21.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.109+0000 7f4d077fe640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 msgr2=0x7f4d0813b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-27T21:39:21.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.109+0000 7f4d077fe640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 0x7f4d0813b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-27T21:39:21.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.109+0000 7f4d077fe640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 0x7f4d0813b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-27T21:39:21.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.509+0000 7f4d0c9cc640 1 --2- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 0x7f4d08141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:21.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.509+0000 7f4d0c9cc640 10 monclient(hunting): get_auth_request con 0x7f4d081416f0 auth_method 0 2024-03-27T21:39:21.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.509+0000 7f4d0c9cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:21.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.509+0000 7f4d0c9cc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:21.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.509+0000 7f4d0c9cc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:39:21.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.509+0000 7f4d0c9cc640 1 -- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 msgr2=0x7f4d08141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-27T21:39:21.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.509+0000 7f4d0c9cc640 1 -- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 msgr2=0x7f4d08141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-27T21:39:21.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.509+0000 7f4d0c9cc640 1 --2- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 0x7f4d08141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-27T21:39:21.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.509+0000 7f4d0c9cc640 1 --2- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 0x7f4d08141b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-27T21:39:21.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.510+0000 7f4d077fe640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 0x7f4d0813b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:21.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.510+0000 7f4d077fe640 10 monclient(hunting): get_auth_request con 0x7f4d0813b020 auth_method 0 2024-03-27T21:39:21.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.510+0000 7f4d077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:21.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.510+0000 7f4d077fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:21.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.510+0000 7f4d077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:39:21.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.510+0000 7f4d077fe640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 msgr2=0x7f4d0813b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-27T21:39:21.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.510+0000 7f4d077fe640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 msgr2=0x7f4d0813b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-27T21:39:21.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.510+0000 7f4d077fe640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 0x7f4d0813b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-27T21:39:21.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:21.510+0000 7f4d077fe640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 0x7f4d0813b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-27T21:39:22.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:22.310+0000 7f4d0c9cc640 1 --2- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 0x7f4d08141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:22.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:22.310+0000 7f4d0c9cc640 10 monclient(hunting): get_auth_request con 0x7f4d081416f0 auth_method 0 2024-03-27T21:39:22.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:22.310+0000 7f4d0c9cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:22.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:22.310+0000 7f4d0c9cc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:22.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:22.310+0000 7f4d0c9cc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:39:22.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:22.310+0000 7f4d0c9cc640 1 -- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 msgr2=0x7f4d08141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-27T21:39:22.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:22.310+0000 7f4d0c9cc640 1 -- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 msgr2=0x7f4d08141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-27T21:39:22.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:22.310+0000 7f4d0c9cc640 1 --2- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 0x7f4d08141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-27T21:39:22.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:22.310+0000 7f4d0c9cc640 1 --2- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 0x7f4d08141b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-27T21:39:22.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:22.311+0000 7f4d077fe640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 0x7f4d0813b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:22.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:22.311+0000 7f4d077fe640 10 monclient(hunting): get_auth_request con 0x7f4d0813b020 auth_method 0 2024-03-27T21:39:22.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:22.311+0000 7f4d077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:22.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:22.311+0000 7f4d077fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:22.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:22.311+0000 7f4d077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:39:22.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:22.311+0000 7f4d077fe640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 msgr2=0x7f4d0813b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-27T21:39:22.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:22.311+0000 7f4d077fe640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 msgr2=0x7f4d0813b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-27T21:39:22.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:22.311+0000 7f4d077fe640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 0x7f4d0813b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-27T21:39:22.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:22.311+0000 7f4d077fe640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 0x7f4d0813b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-27T21:39:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d05ffb640 10 monclient(hunting): tick 2024-03-27T21:39:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d05ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-27T21:39:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d05ffb640 1 monclient(hunting): continuing hunt 2024-03-27T21:39:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d05ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-27T21:39:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d05ffb640 1 -- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 msgr2=0x7f4d08141b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T21:39:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d05ffb640 1 --2- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d081416f0 0x7f4d08141b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d05ffb640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 msgr2=0x7f4d0813b480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T21:39:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d05ffb640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 0x7f4d0813b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d05ffb640 1 -- 172.21.15.60:0/3236694290 >> v1:172.21.15.191:6789/0 conn(0x7f4d080b40b0 legacy=0x7f4d080a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d05ffb640 10 monclient: _add_conns ranks=[3,0] 2024-03-27T21:39:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d05ffb640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 0x7f4cf4005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d05ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f4cf40036e0 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d05ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4cf4006170 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d05ffb640 1 -- 172.21.15.60:0/3236694290 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d080a9470 con 0x7f4cf4006170 2024-03-27T21:39:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d05ffb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d05ffb640 1 -- 172.21.15.60:0/3236694290 --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d080b4490 con 0x7f4cf40036e0 2024-03-27T21:39:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d05ffb640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d07fff640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 0x7f4cf4005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d07fff640 10 monclient(hunting): get_auth_request con 0x7f4cf40036e0 auth_method 0 2024-03-27T21:39:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d07fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d07fff640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 msgr2=0x7f4cf4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-27T21:39:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d07fff640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 msgr2=0x7f4cf4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-27T21:39:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d07fff640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 0x7f4cf4005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-27T21:39:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:23.908+0000 7f4d07fff640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 0x7f4cf4005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-27T21:39:24.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:24.109+0000 7f4d07fff640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 0x7f4cf4005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:24.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:24.109+0000 7f4d07fff640 10 monclient(hunting): get_auth_request con 0x7f4cf40036e0 auth_method 0 2024-03-27T21:39:24.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:24.109+0000 7f4d07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:24.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:24.109+0000 7f4d07fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:24.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:24.109+0000 7f4d07fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:39:24.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:24.109+0000 7f4d07fff640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 msgr2=0x7f4cf4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-27T21:39:24.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:24.109+0000 7f4d07fff640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 msgr2=0x7f4cf4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-27T21:39:24.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:24.109+0000 7f4d07fff640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 0x7f4cf4005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-27T21:39:24.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:24.109+0000 7f4d07fff640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 0x7f4cf4005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-27T21:39:24.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:24.510+0000 7f4d07fff640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 0x7f4cf4005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:24.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:24.510+0000 7f4d07fff640 10 monclient(hunting): get_auth_request con 0x7f4cf40036e0 auth_method 0 2024-03-27T21:39:24.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:24.510+0000 7f4d07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:24.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:24.510+0000 7f4d07fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:24.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:24.510+0000 7f4d07fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:39:24.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:24.510+0000 7f4d07fff640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 msgr2=0x7f4cf4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-27T21:39:24.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:24.510+0000 7f4d07fff640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 msgr2=0x7f4cf4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-27T21:39:24.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:24.510+0000 7f4d07fff640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 0x7f4cf4005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-27T21:39:24.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:24.510+0000 7f4d07fff640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 0x7f4cf4005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-27T21:39:25.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:25.311+0000 7f4d07fff640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 0x7f4cf4005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:25.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:25.311+0000 7f4d07fff640 10 monclient(hunting): get_auth_request con 0x7f4cf40036e0 auth_method 0 2024-03-27T21:39:25.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:25.311+0000 7f4d07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:25.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:25.311+0000 7f4d07fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:25.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:25.311+0000 7f4d07fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:39:25.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:25.311+0000 7f4d07fff640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 msgr2=0x7f4cf4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-27T21:39:25.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:25.311+0000 7f4d07fff640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 msgr2=0x7f4cf4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-27T21:39:25.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:25.311+0000 7f4d07fff640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 0x7f4cf4005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-27T21:39:25.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:25.311+0000 7f4d07fff640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 0x7f4cf4005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-27T21:39:26.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.907+0000 7f4d05ffb640 10 monclient(hunting): tick 2024-03-27T21:39:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.907+0000 7f4d05ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-27T21:39:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.907+0000 7f4d05ffb640 1 monclient(hunting): continuing hunt 2024-03-27T21:39:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.907+0000 7f4d05ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-27T21:39:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.907+0000 7f4d05ffb640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 msgr2=0x7f4cf4005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T21:39:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.907+0000 7f4d05ffb640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf40036e0 0x7f4cf4005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d05ffb640 1 -- 172.21.15.60:0/3236694290 >> v1:172.21.15.60:6789/0 conn(0x7f4cf4006170 legacy=0x7f4cf4008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d05ffb640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-03-27T21:39:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d05ffb640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 0x7f4cf4009ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d0c9cc640 1 -- 172.21.15.60:0/3236694290 reap_dead start 2024-03-27T21:39:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d05ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f4d0813b020 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d05ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4cf400a530 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d05ffb640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf4005bc0 0x7f4cf400a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d05ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f4cf4005bc0 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d05ffb640 1 -- 172.21.15.60:0/3236694290 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cf400af10 con 0x7f4cf400a530 2024-03-27T21:39:26.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d05ffb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:26.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d05ffb640 1 -- 172.21.15.60:0/3236694290 --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d080b4490 con 0x7f4cf4005bc0 2024-03-27T21:39:26.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d05ffb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:26.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d05ffb640 1 -- 172.21.15.60:0/3236694290 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d080aa520 con 0x7f4d0813b020 2024-03-27T21:39:26.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d05ffb640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:26.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d0c9cc640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf4005bc0 0x7f4cf400a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:26.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d0c9cc640 10 monclient(hunting): get_auth_request con 0x7f4cf4005bc0 auth_method 0 2024-03-27T21:39:26.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d0c9cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:26.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d0c9cc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:26.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d0c9cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:26.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d0c9cc640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf4005bc0 msgr2=0x7f4cf400a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-27T21:39:26.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d0c9cc640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf4005bc0 msgr2=0x7f4cf400a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-27T21:39:26.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d0c9cc640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf4005bc0 0x7f4cf400a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-27T21:39:26.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d07fff640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 0x7f4cf4009ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:26.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d0c9cc640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf4005bc0 0x7f4cf400a910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-27T21:39:26.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d07fff640 10 monclient(hunting): get_auth_request con 0x7f4d0813b020 auth_method 0 2024-03-27T21:39:26.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:26.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d07fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:26.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:26.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d07fff640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 msgr2=0x7f4cf4009ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-27T21:39:26.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d07fff640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 msgr2=0x7f4cf4009ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-27T21:39:26.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d07fff640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 0x7f4cf4009ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-27T21:39:26.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:26.908+0000 7f4d07fff640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 0x7f4cf4009ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-27T21:39:27.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d0c9cc640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.60:3300/0 conn(0x7f4cf4005bc0 0x7f4cf400a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:27.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d0c9cc640 10 monclient(hunting): get_auth_request con 0x7f4cf4005bc0 auth_method 0 2024-03-27T21:39:27.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d0c9cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:27.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d0c9cc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:27.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d0c9cc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:39:27.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d07fff640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 0x7f4cf4009ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:27.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d07fff640 10 monclient(hunting): get_auth_request con 0x7f4d0813b020 auth_method 0 2024-03-27T21:39:27.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:27.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d07fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:27.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d07fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:39:27.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d0c9cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:27.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d0c9cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:27.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d0c9cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:27.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:27.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:27.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:27.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d0c9cc640 10 monclient(hunting): handle_auth_done global_id 4098 payload 274 2024-03-27T21:39:27.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d0c9cc640 1 -- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 msgr2=0x7f4cf4009ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:27.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d0c9cc640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 0x7f4cf4009ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:27.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d0c9cc640 1 -- 172.21.15.60:0/3236694290 >> v1:172.21.15.191:6789/0 conn(0x7f4cf400a530 legacy=0x7f4cf4008d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:27.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d0c9cc640 10 monclient: _finish_hunting 0 2024-03-27T21:39:27.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d0c9cc640 1 monclient: found mon.noname-a 2024-03-27T21:39:27.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d0c9cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:27.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d0c9cc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:27.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.109+0000 7f4d0c9cc640 1 -- 172.21.15.60:0/3236694290 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf4008640 con 0x7f4cf4005bc0 2024-03-27T21:39:27.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.110+0000 7f4d0c9cc640 1 --2- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf4005bc0 0x7f4cf400a910 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4d08141370 tx=0x7f4d0002cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83de8c83803e3b52 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:27.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.110+0000 7f4d06ffd640 1 -- 172.21.15.60:0/3236694290 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0002d840 con 0x7f4cf4005bc0 2024-03-27T21:39:27.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.110+0000 7f4d06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:27.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.110+0000 7f4d06ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:27.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.110+0000 7f4d06ffd640 10 monclient: dump: 2024-03-27T21:39:27.100 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:27.100 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:27.100 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:27.100 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:27.100 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:27.100 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:27.101 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:27.101 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:27.101 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:27.101 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:27.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.110+0000 7f4d06ffd640 10 monclient: _finish_auth 0 2024-03-27T21:39:27.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.110+0000 7f4d06ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:39:27.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.110+0000 7f4d06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:27.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.110+0000 7f4d06ffd640 1 -- 172.21.15.60:0/3236694290 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d00031070 con 0x7f4cf4005bc0 2024-03-27T21:39:27.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.110+0000 7f4d0e42f640 5 monclient: authenticate success, global_id 4098 2024-03-27T21:39:27.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.110+0000 7f4d0e42f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:27.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.110+0000 7f4d06ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:27.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.110+0000 7f4d0e42f640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:27.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.110+0000 7f4d0e42f640 4 set_mon_vals no callback set 2024-03-27T21:39:27.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.110+0000 7f4d06ffd640 1 -- 172.21.15.60:0/3236694290 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0002dbe0 con 0x7f4cf4005bc0 2024-03-27T21:39:27.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.110+0000 7f4d06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:27.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.110+0000 7f4d06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:27.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.110+0000 7f4d06ffd640 10 monclient: dump: 2024-03-27T21:39:27.102 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:27.102 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:27.102 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:27.102 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:27.102 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:27.102 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:27.102 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:27.102 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:27.102 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:27.102 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:27.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.111+0000 7f4d0e42f640 10 monclient: shutdown 2024-03-27T21:39:27.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.111+0000 7f4d0e42f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:27.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.111+0000 7f4d0e42f640 1 -- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf4005bc0 msgr2=0x7f4cf400a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:27.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.111+0000 7f4d0e42f640 1 --2- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf4005bc0 0x7f4cf400a910 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4d08141370 tx=0x7f4d0002cc30 comp rx=0 tx=0).stop 2024-03-27T21:39:27.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.111+0000 7f4d0e42f640 1 -- 172.21.15.60:0/3236694290 shutdown_connections 2024-03-27T21:39:27.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.111+0000 7f4d0e42f640 1 --2- 172.21.15.60:0/3236694290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf4005bc0 0x7f4cf400a910 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:27.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.111+0000 7f4d0e42f640 1 --2- 172.21.15.60:0/3236694290 >> v2:172.21.15.191:3300/0 conn(0x7f4d0813b020 0x7f4cf4009ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:27.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.111+0000 7f4d0e42f640 1 -- 172.21.15.60:0/3236694290 >> 172.21.15.60:0/3236694290 conn(0x7f4d080608e0 msgr2=0x7f4d08060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:27.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.111+0000 7f4d0e42f640 1 -- 172.21.15.60:0/3236694290 shutdown_connections 2024-03-27T21:39:27.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.111+0000 7f4d0e42f640 1 -- 172.21.15.60:0/3236694290 wait complete. 2024-03-27T21:39:27.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.112+0000 7f4d0e42f640 10 monclient: build_initial_monmap 2024-03-27T21:39:27.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.112+0000 7f4d0e42f640 10 monclient: monmap: 2024-03-27T21:39:27.109 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:27.109 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:27.109 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:27.109 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:27.109 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:27.109 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:27.109 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:27.110 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:27.110 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:27.110 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:27.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.112+0000 7f4d0e42f640 1 librados: starting msgr at 2024-03-27T21:39:27.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.112+0000 7f4d0e42f640 1 librados: starting objecter 2024-03-27T21:39:27.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.112+0000 7f4d0e42f640 1 Processor -- start 2024-03-27T21:39:27.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.113+0000 7f4d0e42f640 1 -- start start 2024-03-27T21:39:27.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.113+0000 7f4d0e42f640 1 librados: setting wanted keys 2024-03-27T21:39:27.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.113+0000 7f4d0e42f640 1 librados: calling monclient init 2024-03-27T21:39:27.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.113+0000 7f4d0e42f640 10 monclient: init 2024-03-27T21:39:27.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.113+0000 7f4d0e42f640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:27.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.113+0000 7f4d0e42f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:39:27.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.113+0000 7f4d0e42f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cf4005bc0 0x7f4d080af990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:27.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.113+0000 7f4d0e42f640 10 monclient(hunting): picked mon.noname-b con 0x7f4cf4005bc0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:27.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.113+0000 7f4d0e42f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf400a530 0x7f4d08138ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:27.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.113+0000 7f4d0e42f640 10 monclient(hunting): picked mon.noname-a con 0x7f4cf400a530 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:27.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.113+0000 7f4d0e42f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d0813b020 0x7f4d081d9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:27.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.113+0000 7f4d0e42f640 10 monclient(hunting): picked mon.noname-c con 0x7f4d0813b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:27.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.113+0000 7f4d0e42f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:27.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.113+0000 7f4d0e42f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d080aeb50 con 0x7f4cf400a530 2024-03-27T21:39:27.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.113+0000 7f4d0e42f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:27.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.113+0000 7f4d0e42f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d080b0470 con 0x7f4cf4005bc0 2024-03-27T21:39:27.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.113+0000 7f4d0e42f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:27.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.113+0000 7f4d0e42f640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d08139000 con 0x7f4d0813b020 2024-03-27T21:39:27.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.113+0000 7f4d0e42f640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:27.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.113+0000 7f4d0e42f640 10 monclient(hunting): authenticate will time out at 808.334834s 2024-03-27T21:39:27.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d077fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf400a530 0x7f4d08138ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:27.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d0c9cc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d0813b020 0x7f4d081d9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:27.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d077fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf400a530 0x7f4d08138ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56164/0 (socket says 172.21.15.60:56164) 2024-03-27T21:39:27.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d0c9cc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d0813b020 0x7f4d081d9f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47740/0 (socket says 172.21.15.60:47740) 2024-03-27T21:39:27.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d077fe640 1 -- 172.21.15.60:0/1616172912 learned_addr learned my addr 172.21.15.60:0/1616172912 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:27.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d077fe640 10 monclient(hunting): get_auth_request con 0x7f4cf400a530 auth_method 0 2024-03-27T21:39:27.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:27.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d077fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:27.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:27.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d0c9cc640 10 monclient(hunting): get_auth_request con 0x7f4d0813b020 auth_method 0 2024-03-27T21:39:27.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d0c9cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:27.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d0c9cc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:27.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d0c9cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:27.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d07fff640 1 --2- 172.21.15.60:0/1616172912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cf4005bc0 0x7f4d080af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:27.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:27.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:27.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:27.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d07fff640 10 monclient(hunting): get_auth_request con 0x7f4cf4005bc0 auth_method 0 2024-03-27T21:39:27.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:27.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d07fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:27.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:27.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d0c9cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:27.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d0c9cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:27.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d0c9cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:27.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:27.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:27.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:27.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d077fe640 10 monclient(hunting): handle_auth_done global_id 4101 payload 824 2024-03-27T21:39:27.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d077fe640 1 -- 172.21.15.60:0/1616172912 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d0813b020 msgr2=0x7f4d081d9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:27.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d077fe640 1 --2- 172.21.15.60:0/1616172912 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d0813b020 0x7f4d081d9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:27.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d077fe640 1 -- 172.21.15.60:0/1616172912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cf4005bc0 msgr2=0x7f4d080af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:27.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d077fe640 1 --2- 172.21.15.60:0/1616172912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cf4005bc0 0x7f4d080af990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:27.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d077fe640 10 monclient: _finish_hunting 0 2024-03-27T21:39:27.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d077fe640 1 monclient: found mon.noname-a 2024-03-27T21:39:27.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:27.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d077fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:27.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d077fe640 1 -- 172.21.15.60:0/1616172912 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d000097a0 con 0x7f4cf400a530 2024-03-27T21:39:27.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d077fe640 1 --2- 172.21.15.60:0/1616172912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf400a530 0x7f4d08138ac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8011fd0 tx=0x7f4cf80139d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:27.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d04ff9640 10 client.?.objecter ms_handle_connect 0x7f4cf400a530 2024-03-27T21:39:27.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d04ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:27.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d04ff9640 1 -- 172.21.15.60:0/1616172912 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf8015690 con 0x7f4cf400a530 2024-03-27T21:39:27.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d04ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:27.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d04ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:27.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d04ff9640 10 monclient: dump: 2024-03-27T21:39:27.115 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:27.115 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:27.115 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:27.115 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:27.115 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:27.115 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:27.115 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:27.115 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:27.115 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:27.115 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:27.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d04ff9640 10 monclient: _finish_auth 0 2024-03-27T21:39:27.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d04ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:39:27.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d04ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:27.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d04ff9640 1 -- 172.21.15.60:0/1616172912 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cf80157f0 con 0x7f4cf400a530 2024-03-27T21:39:27.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d0e42f640 5 monclient: authenticate success, global_id 4101 2024-03-27T21:39:27.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d0e42f640 10 monclient: _renew_subs 2024-03-27T21:39:27.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d0e42f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:27.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d0e42f640 1 -- 172.21.15.60:0/1616172912 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d08139220 con 0x7f4cf400a530 2024-03-27T21:39:27.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d04ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:27.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d04ff9640 1 -- 172.21.15.60:0/1616172912 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf8015b80 con 0x7f4cf400a530 2024-03-27T21:39:27.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d04ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:27.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d04ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:27.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d04ff9640 10 monclient: dump: 2024-03-27T21:39:27.116 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:27.116 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:27.116 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:27.116 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:27.116 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:27.117 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:27.117 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:27.117 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:27.117 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:27.117 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:27.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.115+0000 7f4d0e42f640 10 client.4101.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:27.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.115+0000 7f4d0e42f640 10 monclient: _renew_subs 2024-03-27T21:39:27.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.114+0000 7f4d067fc640 4 set_mon_vals no callback set 2024-03-27T21:39:27.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.115+0000 7f4d0e42f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:27.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.115+0000 7f4d0e42f640 1 -- 172.21.15.60:0/1616172912 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d080b0130 con 0x7f4cf400a530 2024-03-27T21:39:27.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.115+0000 7f4d0e42f640 1 librados: init done 2024-03-27T21:39:27.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.115+0000 7f4d04ff9640 1 -- 172.21.15.60:0/1616172912 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4cf8016460 con 0x7f4cf400a530 2024-03-27T21:39:27.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.115+0000 7f4d04ff9640 1 -- 172.21.15.60:0/1616172912 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4cf80219f0 con 0x7f4cf400a530 2024-03-27T21:39:27.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.115+0000 7f4d04ff9640 10 client.4101.objecter ms_dispatch 0x7f4d08000b90 osd_map(1..1 src has 1..1) v4 2024-03-27T21:39:27.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.115+0000 7f4d04ff9640 3 client.4101.objecter handle_osd_map got epochs [1,1] > 0 2024-03-27T21:39:27.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.115+0000 7f4d04ff9640 3 client.4101.objecter handle_osd_map decoding full epoch 1 2024-03-27T21:39:27.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.115+0000 7f4d04ff9640 20 client.4101.objecter dump_active .. 0 homeless 2024-03-27T21:39:27.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.116+0000 7f4d0e42f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:27.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.116+0000 7f4d0e42f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:27.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.116+0000 7f4d0e42f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:27.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.116+0000 7f4d0e42f640 1 -- 172.21.15.60:0/1616172912 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d08069470 con 0x7f4cf400a530 2024-03-27T21:39:27.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.118+0000 7f4d04ff9640 1 -- 172.21.15.60:0/1616172912 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f4cf8026060 con 0x7f4cf400a530 2024-03-27T21:39:27.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.118+0000 7f4d04ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:27.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.118+0000 7f4d04ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:27.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.188+0000 7f4d0e42f640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-27T21:39:27.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.188+0000 7f4d0e42f640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-27T21:39:27.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.188+0000 7f4d0e42f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:27.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.188+0000 7f4d0e42f640 1 -- 172.21.15.60:0/1616172912 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f4d080b59b0 con 0x7f4cf400a530 2024-03-27T21:39:27.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.191+0000 7f4d04ff9640 1 -- 172.21.15.60:0/1616172912 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f4cf801c070 con 0x7f4cf400a530 2024-03-27T21:39:27.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.191+0000 7f4d04ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-27T21:39:27.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.191+0000 7f4d04ff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-27T21:39:27.178 INFO:teuthology.orchestra.run.smithi060.stderr:adjusted tunables profile to default 2024-03-27T21:39:27.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.192+0000 7f4d0e42f640 10 librados: watch_flush enter 2024-03-27T21:39:27.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.192+0000 7f4d0e42f640 10 librados: watch_flush exit 2024-03-27T21:39:27.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.192+0000 7f4d0e42f640 20 client.4101.objecter shutdown clearing up homeless session... 2024-03-27T21:39:27.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.192+0000 7f4d0e42f640 10 client.4101.objecter successfully canceled tick 2024-03-27T21:39:27.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.192+0000 7f4d0e42f640 10 monclient: shutdown 2024-03-27T21:39:27.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.192+0000 7f4d0e42f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:27.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.192+0000 7f4d0e42f640 1 -- 172.21.15.60:0/1616172912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf400a530 msgr2=0x7f4d08138ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:27.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.192+0000 7f4d0e42f640 1 --2- 172.21.15.60:0/1616172912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf400a530 0x7f4d08138ac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8011fd0 tx=0x7f4cf80139d0 comp rx=0 tx=0).stop 2024-03-27T21:39:27.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.192+0000 7f4d0e42f640 1 -- 172.21.15.60:0/1616172912 shutdown_connections 2024-03-27T21:39:27.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.192+0000 7f4d0e42f640 1 --2- 172.21.15.60:0/1616172912 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d0813b020 0x7f4d081d9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:27.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.192+0000 7f4d0e42f640 1 --2- 172.21.15.60:0/1616172912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf400a530 0x7f4d08138ac0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:27.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.192+0000 7f4d0e42f640 1 --2- 172.21.15.60:0/1616172912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cf4005bc0 0x7f4d080af990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:27.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.192+0000 7f4d0e42f640 1 -- 172.21.15.60:0/1616172912 >> 172.21.15.60:0/1616172912 conn(0x7f4d080608e0 msgr2=0x7f4d08141b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:27.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.192+0000 7f4d0e42f640 1 -- 172.21.15.60:0/1616172912 shutdown_connections 2024-03-27T21:39:27.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.192+0000 7f4d0e42f640 1 -- 172.21.15.60:0/1616172912 wait complete. 2024-03-27T21:39:27.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:27.192+0000 7f4d0e42f640 1 librados: shutdown 2024-03-27T21:39:27.190 INFO:tasks.ceph:check_enable_crimson: False 2024-03-27T21:39:27.191 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-27T21:39:27.191 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:39:27.191 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-27T21:39:27.218 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:39:27.218 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-27T21:39:27.286 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:39:27.286 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-27T21:39:27.353 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-27T21:39:27.353 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-27T21:39:27.419 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-27T21:39:27.419 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-27T21:39:27.447 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-27T21:39:27.447 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-27T21:39:27.515 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-27T21:39:27.515 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-27T21:39:27.582 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-27T21:39:27.582 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-27T21:39:27.613 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:27.627+0000 7f6438632040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-27T21:39:27.649 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 778aaa64-6720-495b-bcdc-86fbcece9d58 0 2024-03-27T21:39:27.669 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:27.682+0000 7f6438632040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-27T21:39:27.787 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.801+0000 7fa89f84a640 10 monclient: get_monmap_and_config 2024-03-27T21:39:27.788 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.801+0000 7fa89f84a640 10 monclient: build_initial_monmap 2024-03-27T21:39:27.788 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.801+0000 7fa89f84a640 10 monclient: monmap: 2024-03-27T21:39:27.788 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-27T21:39:27.788 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:27.788 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:39:27.802346+0000 2024-03-27T21:39:27.788 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:39:27.802346+0000 2024-03-27T21:39:27.788 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:27.788 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:27.788 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:27.788 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:27.789 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:27.789 INFO:teuthology.orchestra.run.smithi191.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:27.789 INFO:teuthology.orchestra.run.smithi191.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:27.789 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:27.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.801+0000 7fa89f84a640 1 Processor -- start 2024-03-27T21:39:27.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.801+0000 7fa89f84a640 1 -- start start 2024-03-27T21:39:27.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.801+0000 7fa89f84a640 10 monclient: init 2024-03-27T21:39:27.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89f84a640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:27.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89f84a640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-03-27T21:39:27.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89f84a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa8980b40b0 0x7fa8980a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:27.797 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89f84a640 10 monclient(hunting): picked mon.noname-c con 0x7fa8980b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:27.798 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89f84a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa89813b020 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:27.798 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89f84a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa89813b480 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:27.798 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89f84a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa89813b8d0 con 0x7fa89813b480 2024-03-27T21:39:27.798 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89f84a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8980b3830 con 0x7fa89813b020 2024-03-27T21:39:27.798 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89f84a640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:27.798 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89f84a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8980b4490 con 0x7fa8980b40b0 2024-03-27T21:39:27.798 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89f84a640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:27.798 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89cde5640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fa89813b020 0x7fa8981416f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:37422/0 (socket says 172.21.15.191:37422) 2024-03-27T21:39:27.798 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89d5e6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa8980b40b0 0x7fa8980a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:27.798 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89cde5640 1 -- 172.21.15.191:0/1633330249 learned_addr learned my addr 172.21.15.191:0/1633330249 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-27T21:39:27.798 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89f84a640 10 monclient(hunting): authenticate will time out at 785.086295s 2024-03-27T21:39:27.798 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89d5e6640 10 monclient(hunting): get_auth_request con 0x7fa8980b40b0 auth_method 0 2024-03-27T21:39:27.798 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89d5e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:27.798 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89d5e6640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:27.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89d5e6640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:27.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89d5e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:27.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89d5e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:27.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89d5e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:27.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89d5e6640 10 monclient(hunting): handle_auth_done global_id 4117 payload 274 2024-03-27T21:39:27.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89d5e6640 1 -- 172.21.15.191:0/1633330249 >> v1:172.21.15.191:6789/0 conn(0x7fa89813b020 legacy=0x7fa8981416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:27.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89d5e6640 1 -- 172.21.15.191:0/1633330249 >> v1:172.21.15.60:6789/0 conn(0x7fa89813b480 legacy=0x7fa89814bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:27.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89d5e6640 10 monclient: _finish_hunting 0 2024-03-27T21:39:27.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89d5e6640 1 monclient: found mon.noname-c 2024-03-27T21:39:27.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa88ffff640 1 -- 172.21.15.191:0/1633330249 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870307140 0 0) 0x7fa89813b8d0 con 0x7fa89813b480 2024-03-27T21:39:27.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89d5e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:27.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89d5e6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:39:27.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa89d5e6640 1 -- 172.21.15.191:0/1633330249 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8980aa520 con 0x7fa8980b40b0 2024-03-27T21:39:27.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.802+0000 7fa88ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:39:27.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.803+0000 7fa89d5e6640 1 --2- 172.21.15.191:0/1633330249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa8980b40b0 0x7fa8980a8b90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa8880097a0 tx=0x7fa88802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e12ae46c0972336 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:27.800 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.803+0000 7fa88ffff640 1 -- 172.21.15.191:0/1633330249 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa888031830 con 0x7fa8980b40b0 2024-03-27T21:39:27.800 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.803+0000 7fa88ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:27.800 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.803+0000 7fa88ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:27.800 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.803+0000 7fa88ffff640 10 monclient: dump: 2024-03-27T21:39:27.800 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:27.800 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:27.800 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:27.800 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:27.800 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:27.800 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:27.800 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:27.800 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:27.800 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:27.800 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:27.801 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.803+0000 7fa88ffff640 10 monclient: _finish_auth 0 2024-03-27T21:39:27.801 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.803+0000 7fa88ffff640 10 monclient: _check_auth_tickets 2024-03-27T21:39:27.801 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.803+0000 7fa88ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:27.801 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.803+0000 7fa89f84a640 5 monclient: authenticate success, global_id 4117 2024-03-27T21:39:27.801 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.803+0000 7fa89f84a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:27.801 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.803+0000 7fa88ffff640 1 -- 172.21.15.191:0/1633330249 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa888031990 con 0x7fa8980b40b0 2024-03-27T21:39:27.801 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.803+0000 7fa88ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:27.801 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.803+0000 7fa89f84a640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:27.801 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.803+0000 7fa89f84a640 4 set_mon_vals no callback set 2024-03-27T21:39:27.801 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.803+0000 7fa88ffff640 1 -- 172.21.15.191:0/1633330249 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa888031d20 con 0x7fa8980b40b0 2024-03-27T21:39:27.801 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.803+0000 7fa88ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:27.801 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.803+0000 7fa88ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:27.802 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.803+0000 7fa88ffff640 10 monclient: dump: 2024-03-27T21:39:27.802 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:27.802 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:27.802 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:27.802 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:27.802 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:27.802 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:27.802 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:27.802 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:27.802 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:27.802 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:27.802 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.804+0000 7fa89f84a640 10 monclient: shutdown 2024-03-27T21:39:27.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.804+0000 7fa89f84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:27.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.804+0000 7fa89f84a640 1 -- 172.21.15.191:0/1633330249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa8980b40b0 msgr2=0x7fa8980a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:27.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.804+0000 7fa89f84a640 1 --2- 172.21.15.191:0/1633330249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa8980b40b0 0x7fa8980a8b90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa8880097a0 tx=0x7fa88802b8d0 comp rx=0 tx=0).stop 2024-03-27T21:39:27.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.804+0000 7fa89f84a640 1 -- 172.21.15.191:0/1633330249 shutdown_connections 2024-03-27T21:39:27.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.804+0000 7fa89f84a640 1 --2- 172.21.15.191:0/1633330249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa8980b40b0 0x7fa8980a8b90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:27.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.804+0000 7fa89f84a640 1 -- 172.21.15.191:0/1633330249 >> 172.21.15.191:0/1633330249 conn(0x7fa8980608e0 msgr2=0x7fa898060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:27.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.804+0000 7fa89f84a640 1 -- 172.21.15.191:0/1633330249 shutdown_connections 2024-03-27T21:39:27.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.804+0000 7fa89f84a640 1 -- 172.21.15.191:0/1633330249 wait complete. 2024-03-27T21:39:27.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.805+0000 7fa89f84a640 10 monclient: build_initial_monmap 2024-03-27T21:39:27.804 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.805+0000 7fa89f84a640 10 monclient: monmap: 2024-03-27T21:39:27.804 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-27T21:39:27.804 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:27.804 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-27T21:39:27.804 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-27T21:39:27.804 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:27.804 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-27T21:39:27.804 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:27.804 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:27.804 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:27.804 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:27.804 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.805+0000 7fa89f84a640 1 librados: starting msgr at 2024-03-27T21:39:27.804 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.805+0000 7fa89f84a640 1 librados: starting objecter 2024-03-27T21:39:27.804 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.805+0000 7fa89f84a640 1 Processor -- start 2024-03-27T21:39:27.804 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.805+0000 7fa89f84a640 1 -- start start 2024-03-27T21:39:27.805 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.805+0000 7fa89f84a640 1 librados: setting wanted keys 2024-03-27T21:39:27.805 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.805+0000 7fa89f84a640 1 librados: calling monclient init 2024-03-27T21:39:27.805 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.805+0000 7fa89f84a640 10 monclient: init 2024-03-27T21:39:27.805 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89f84a640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:27.805 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89f84a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:39:27.805 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89f84a640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa8980b40b0 0x7fa8980af070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:27.805 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89f84a640 10 monclient(hunting): picked mon.noname-c con 0x7fa8980b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:27.805 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89f84a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa89813b020 0x7fa8980afce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:27.805 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89f84a640 10 monclient(hunting): picked mon.noname-b con 0x7fa89813b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:27.805 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89f84a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa89813b480 0x7fa8981d9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:27.805 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89f84a640 10 monclient(hunting): picked mon.noname-a con 0x7fa89813b480 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:27.805 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89f84a640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:27.805 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89f84a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8981d9910 con 0x7fa89813b480 2024-03-27T21:39:27.805 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89f84a640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:27.805 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89f84a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8980ae230 con 0x7fa89813b020 2024-03-27T21:39:27.806 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89f84a640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:27.806 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89f84a640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8980b0220 con 0x7fa8980b40b0 2024-03-27T21:39:27.806 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89f84a640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:27.806 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89f84a640 10 monclient(hunting): authenticate will time out at 785.089779s 2024-03-27T21:39:27.806 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89cde5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa89813b020 0x7fa8980afce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:27.806 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89cde5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa89813b020 0x7fa8980afce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:52488/0 (socket says 172.21.15.191:52488) 2024-03-27T21:39:27.806 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89cde5640 1 -- 172.21.15.191:0/1053309869 learned_addr learned my addr 172.21.15.191:0/1053309869 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-27T21:39:27.806 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89cde5640 10 monclient(hunting): get_auth_request con 0x7fa89813b020 auth_method 0 2024-03-27T21:39:27.806 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89cde5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:27.806 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89cde5640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:27.806 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89cde5640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:27.806 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89cde5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:27.806 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89d5e6640 1 --2- 172.21.15.191:0/1053309869 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa8980b40b0 0x7fa8980af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:27.806 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89cde5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:27.806 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89dde7640 1 --2- 172.21.15.191:0/1053309869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa89813b480 0x7fa8981d9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:27.807 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89cde5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:27.807 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89dde7640 10 monclient(hunting): get_auth_request con 0x7fa89813b480 auth_method 0 2024-03-27T21:39:27.807 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89dde7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:27.807 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89dde7640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:27.807 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89dde7640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:27.807 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89d5e6640 10 monclient(hunting): get_auth_request con 0x7fa8980b40b0 auth_method 0 2024-03-27T21:39:27.807 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89d5e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:27.807 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89d5e6640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:27.807 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89d5e6640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:27.807 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89dde7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:27.807 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89dde7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:27.807 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89dde7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:27.807 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89d5e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:27.807 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89d5e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:27.808 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89d5e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:27.808 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89cde5640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2024-03-27T21:39:27.808 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89d5e6640 1 -- 172.21.15.191:0/1053309869 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa8980b40b0 msgr2=0x7fa8980af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-27T21:39:27.808 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89d5e6640 1 -- 172.21.15.191:0/1053309869 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa8980b40b0 msgr2=0x7fa8980af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-27T21:39:27.808 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89d5e6640 1 --2- 172.21.15.191:0/1053309869 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa8980b40b0 0x7fa8980af070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-27T21:39:27.808 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89cde5640 1 -- 172.21.15.191:0/1053309869 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa8980b40b0 msgr2=0x7fa8980af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:27.808 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89d5e6640 1 --2- 172.21.15.191:0/1053309869 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa8980b40b0 0x7fa8980af070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-27T21:39:27.808 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89cde5640 1 --2- 172.21.15.191:0/1053309869 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa8980b40b0 0x7fa8980af070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:27.808 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89cde5640 1 -- 172.21.15.191:0/1053309869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa89813b480 msgr2=0x7fa8981d9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:27.808 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89cde5640 1 --2- 172.21.15.191:0/1053309869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa89813b480 0x7fa8981d9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:27.808 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89cde5640 10 monclient: _finish_hunting 0 2024-03-27T21:39:27.808 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89cde5640 1 monclient: found mon.noname-b 2024-03-27T21:39:27.808 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89cde5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:27.809 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89cde5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:39:27.809 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89cde5640 1 -- 172.21.15.191:0/1053309869 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa888009400 con 0x7fa89813b020 2024-03-27T21:39:27.809 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89cde5640 1 --2- 172.21.15.191:0/1053309869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa89813b020 0x7fa8980afce0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa89400c400 tx=0x7fa89400c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:27.809 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa88dffb640 10 client.?.objecter ms_handle_connect 0x7fa89813b020 2024-03-27T21:39:27.809 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa88dffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:27.809 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa88dffb640 1 -- 172.21.15.191:0/1053309869 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa894005ce0 con 0x7fa89813b020 2024-03-27T21:39:27.809 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa88dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:27.809 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa88dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:27.809 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa88dffb640 10 monclient: dump: 2024-03-27T21:39:27.809 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:27.809 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:27.809 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:27.809 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:27.810 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:27.810 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:27.810 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:27.810 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:27.810 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:27.811 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:27.811 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa88dffb640 10 monclient: _finish_auth 0 2024-03-27T21:39:27.811 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa88dffb640 10 monclient: _check_auth_tickets 2024-03-27T21:39:27.811 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa88dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:27.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89f84a640 5 monclient: authenticate success, global_id 4123 2024-03-27T21:39:27.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa88dffb640 1 -- 172.21.15.191:0/1053309869 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa894005e40 con 0x7fa89813b020 2024-03-27T21:39:27.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89f84a640 10 monclient: _renew_subs 2024-03-27T21:39:27.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89f84a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:27.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa89f84a640 1 -- 172.21.15.191:0/1053309869 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8980ac560 con 0x7fa89813b020 2024-03-27T21:39:27.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa88dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:27.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa88dffb640 1 -- 172.21.15.191:0/1053309869 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8940072d0 con 0x7fa89813b020 2024-03-27T21:39:27.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa88dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:27.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa88dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:27.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa88dffb640 10 monclient: dump: 2024-03-27T21:39:27.812 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:27.812 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:27.812 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:27.812 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:27.812 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:27.813 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:27.813 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:27.813 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:27.813 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:27.813 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:27.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.806+0000 7fa88f7fe640 4 set_mon_vals no callback set 2024-03-27T21:39:27.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.807+0000 7fa89f84a640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:27.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.807+0000 7fa89f84a640 10 monclient: _renew_subs 2024-03-27T21:39:27.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.807+0000 7fa89f84a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:27.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.807+0000 7fa89f84a640 1 -- 172.21.15.191:0/1053309869 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8981da550 con 0x7fa89813b020 2024-03-27T21:39:27.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.807+0000 7fa89f84a640 1 librados: init done 2024-03-27T21:39:27.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.807+0000 7fa88dffb640 1 -- 172.21.15.191:0/1053309869 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa89401c440 con 0x7fa89813b020 2024-03-27T21:39:27.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.807+0000 7fa88dffb640 1 -- 172.21.15.191:0/1053309869 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa894012810 con 0x7fa89813b020 2024-03-27T21:39:27.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.807+0000 7fa88dffb640 10 client.4123.objecter ms_dispatch 0x7fa898000b90 osd_map(2..2 src has 1..2) v4 2024-03-27T21:39:27.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.807+0000 7fa88dffb640 3 client.4123.objecter handle_osd_map got epochs [2,2] > 0 2024-03-27T21:39:27.814 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.807+0000 7fa88dffb640 3 client.4123.objecter handle_osd_map decoding full epoch 2 2024-03-27T21:39:27.814 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.807+0000 7fa88dffb640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-27T21:39:27.814 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.808+0000 7fa89f84a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:27.814 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.808+0000 7fa89f84a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:27.814 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.808+0000 7fa89f84a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:27.814 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.808+0000 7fa89f84a640 1 -- 172.21.15.191:0/1053309869 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8981da770 con 0x7fa89813b020 2024-03-27T21:39:27.814 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.810+0000 7fa88dffb640 1 -- 172.21.15.191:0/1053309869 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fa8981da770 con 0x7fa89813b020 2024-03-27T21:39:27.814 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.810+0000 7fa88dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:27.814 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.810+0000 7fa88dffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:27.821 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:27.835+0000 7f4298a0e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-27T21:39:27.867 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.879+0000 7fa89f84a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "778aaa64-6720-495b-bcdc-86fbcece9d58", "id": 0}] 2024-03-27T21:39:27.867 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.879+0000 7fa89f84a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "778aaa64-6720-495b-bcdc-86fbcece9d58", "id": 0}] 2024-03-27T21:39:27.867 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.879+0000 7fa89f84a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:27.867 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.879+0000 7fa89f84a640 1 -- 172.21.15.191:0/1053309869 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "778aaa64-6720-495b-bcdc-86fbcece9d58", "id": 0} v 0) v1 -- 0x7fa898069470 con 0x7fa89813b020 2024-03-27T21:39:27.872 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.885+0000 7fa88dffb640 1 -- 172.21.15.191:0/1053309869 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "778aaa64-6720-495b-bcdc-86fbcece9d58", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa894017070 con 0x7fa89813b020 2024-03-27T21:39:27.872 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.885+0000 7fa88dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "778aaa64-6720-495b-bcdc-86fbcece9d58", "id": 0}] 2024-03-27T21:39:27.872 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.885+0000 7fa88dffb640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:27.872 INFO:teuthology.orchestra.run.smithi191.stdout:0 2024-03-27T21:39:27.873 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.886+0000 7fa89f84a640 10 librados: watch_flush enter 2024-03-27T21:39:27.873 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.886+0000 7fa89f84a640 10 librados: watch_flush exit 2024-03-27T21:39:27.873 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.886+0000 7fa89f84a640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-27T21:39:27.873 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.886+0000 7fa89f84a640 10 client.4123.objecter successfully canceled tick 2024-03-27T21:39:27.873 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.886+0000 7fa89f84a640 10 monclient: shutdown 2024-03-27T21:39:27.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.886+0000 7fa89f84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:27.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.886+0000 7fa89f84a640 1 -- 172.21.15.191:0/1053309869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa89813b020 msgr2=0x7fa8980afce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:27.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.886+0000 7fa89f84a640 1 --2- 172.21.15.191:0/1053309869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa89813b020 0x7fa8980afce0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa89400c400 tx=0x7fa89400c8c0 comp rx=0 tx=0).stop 2024-03-27T21:39:27.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.886+0000 7fa89f84a640 1 -- 172.21.15.191:0/1053309869 shutdown_connections 2024-03-27T21:39:27.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.886+0000 7fa89f84a640 1 --2- 172.21.15.191:0/1053309869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa89813b480 0x7fa8981d9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:27.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.886+0000 7fa89f84a640 1 --2- 172.21.15.191:0/1053309869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa89813b020 0x7fa8980afce0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:27.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.886+0000 7fa89f84a640 1 --2- 172.21.15.191:0/1053309869 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa8980b40b0 0x7fa8980af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:27.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.886+0000 7fa89f84a640 1 -- 172.21.15.191:0/1053309869 >> 172.21.15.191:0/1053309869 conn(0x7fa8980608e0 msgr2=0x7fa8980b65d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:27.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.886+0000 7fa89f84a640 1 -- 172.21.15.191:0/1053309869 shutdown_connections 2024-03-27T21:39:27.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.886+0000 7fa89f84a640 1 -- 172.21.15.191:0/1053309869 wait complete. 2024-03-27T21:39:27.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.886+0000 7fa89f84a640 1 librados: shutdown 2024-03-27T21:39:27.875 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:27.888+0000 7f4298a0e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-27T21:39:27.883 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new d960adf1-3d83-4e24-a612-8df53fa43d0c 1 2024-03-27T21:39:27.983 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.995+0000 7fba96c10640 10 monclient: get_monmap_and_config 2024-03-27T21:39:27.983 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.995+0000 7fba96c10640 10 monclient: build_initial_monmap 2024-03-27T21:39:27.983 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.995+0000 7fba96c10640 10 monclient: monmap: 2024-03-27T21:39:27.983 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-27T21:39:27.983 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:27.983 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:39:27.997223+0000 2024-03-27T21:39:27.983 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:39:27.997223+0000 2024-03-27T21:39:27.983 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:27.983 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:27.983 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:27.984 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:27.984 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:27.984 INFO:teuthology.orchestra.run.smithi191.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:27.984 INFO:teuthology.orchestra.run.smithi191.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:27.984 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:27.998 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.996+0000 7fba96c10640 1 Processor -- start 2024-03-27T21:39:27.998 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.996+0000 7fba96c10640 1 -- start start 2024-03-27T21:39:27.998 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.996+0000 7fba96c10640 10 monclient: init 2024-03-27T21:39:27.998 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba96c10640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:27.998 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba96c10640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-03-27T21:39:27.998 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba96c10640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fba900b40b0 0x7fba900a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:27.998 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba96c10640 10 monclient(hunting): picked mon.noname-b con 0x7fba900b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:27.999 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba96c10640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fba9013b020 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:27.999 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba96c10640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fba9013b480 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:27.999 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba96c10640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba900b3830 con 0x7fba9013b480 2024-03-27T21:39:27.999 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba96c10640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba900aa520 con 0x7fba9013b020 2024-03-27T21:39:27.999 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba96c10640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:27.999 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba96c10640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba9013b910 con 0x7fba900b40b0 2024-03-27T21:39:27.999 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba96c10640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:27.999 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba96c10640 10 monclient(hunting): authenticate will time out at 785.280915s 2024-03-27T21:39:27.999 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8ffff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fba9013b020 0x7fba901416f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:37424/0 (socket says 172.21.15.191:37424) 2024-03-27T21:39:27.999 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8ffff640 1 -- 172.21.15.191:0/4244583511 learned_addr learned my addr 172.21.15.191:0/4244583511 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-27T21:39:27.999 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba949ac640 1 --2- 172.21.15.191:0/4244583511 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fba900b40b0 0x7fba900a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:27.999 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba949ac640 10 monclient(hunting): get_auth_request con 0x7fba900b40b0 auth_method 0 2024-03-27T21:39:27.999 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba949ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:27.999 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba949ac640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.000 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba949ac640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.000 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 1 -- 172.21.15.191:0/4244583511 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897086800 0 0) 0x7fba900aa520 con 0x7fba9013b020 2024-03-27T21:39:28.000 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 10 monclient(hunting): _negotiate 2024-03-27T21:39:28.000 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.000 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.000 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 10 monclient(hunting): my global_id is 4126 2024-03-27T21:39:28.000 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 1 -- 172.21.15.191:0/4244583511 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba80003520 con 0x7fba9013b020 2024-03-27T21:39:28.000 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba949ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.000 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba949ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.000 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba949ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:28.000 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 1 -- 172.21.15.191:0/4244583511 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733144528 0 0) 0x7fba900b3830 con 0x7fba9013b480 2024-03-27T21:39:28.000 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 10 monclient(hunting): _negotiate 2024-03-27T21:39:28.000 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.000 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.000 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 10 monclient(hunting): my global_id is 4116 2024-03-27T21:39:28.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 1 -- 172.21.15.191:0/4244583511 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba900aa520 con 0x7fba9013b480 2024-03-27T21:39:28.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 1 -- 172.21.15.191:0/4244583511 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3127977287 0 0) 0x7fba80003520 con 0x7fba9013b020 2024-03-27T21:39:28.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 1 -- 172.21.15.191:0/4244583511 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fba900b40b0 msgr2=0x7fba900a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:28.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 1 --2- 172.21.15.191:0/4244583511 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fba900b40b0 0x7fba900a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 1 -- 172.21.15.191:0/4244583511 >> v1:172.21.15.60:6789/0 conn(0x7fba9013b480 legacy=0x7fba9014bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 10 monclient: _finish_hunting 0 2024-03-27T21:39:28.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 1 monclient: found mon.noname-c-legacy 2024-03-27T21:39:28.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:28.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.191:6789/0 2024-03-27T21:39:28.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 1 -- 172.21.15.191:0/4244583511 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba900a9470 con 0x7fba9013b020 2024-03-27T21:39:28.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:28.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:28.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:28.002 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba96c10640 5 monclient: authenticate success, global_id 4126 2024-03-27T21:39:28.002 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 1 -- 172.21.15.191:0/4244583511 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2933500084 0 0) 0x7fba7c001670 con 0x7fba9013b020 2024-03-27T21:39:28.002 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:28.002 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-03-27T21:39:28.002 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 10 monclient: dump: 2024-03-27T21:39:28.002 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:28.002 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.002 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.002 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.002 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:28.002 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.002 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:28.002 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:28.003 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:28.003 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.003 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] but i'm connected to v1:172.21.15.191:6789/0, reconnecting 2024-03-27T21:39:28.003 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:28.003 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.997+0000 7fba8f7fe640 1 -- 172.21.15.191:0/4244583511 >> v1:172.21.15.191:6789/0 conn(0x7fba9013b020 legacy=0x7fba901416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.003 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8f7fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:39:28.003 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8f7fe640 1 --2- 172.21.15.191:0/4244583511 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fba80003b00 0x7fba80003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.003 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8f7fe640 10 monclient(hunting): picked mon.c con 0x7fba80003b00 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:28.003 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8f7fe640 1 --2- 172.21.15.191:0/4244583511 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba800047c0 0x7fba80004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.003 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8f7fe640 10 monclient(hunting): picked mon.a con 0x7fba800047c0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:28.003 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8f7fe640 1 --2- 172.21.15.191:0/4244583511 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba80005180 0x7fba8000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.003 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8f7fe640 10 monclient(hunting): picked mon.b con 0x7fba80005180 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:28.003 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8f7fe640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8f7fe640 1 -- 172.21.15.191:0/4244583511 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8000bfa0 con 0x7fba800047c0 2024-03-27T21:39:28.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8f7fe640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8f7fe640 1 -- 172.21.15.191:0/4244583511 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8000c110 con 0x7fba80005180 2024-03-27T21:39:28.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8f7fe640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8f7fe640 1 -- 172.21.15.191:0/4244583511 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8000c2d0 con 0x7fba80003b00 2024-03-27T21:39:28.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8f7fe640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:28.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-27T21:39:28.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba96c10640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:28.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba949ac640 1 --2- 172.21.15.191:0/4244583511 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fba900b40b0 0x7fba900a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:28.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba951ad640 1 --2- 172.21.15.191:0/4244583511 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba80005180 0x7fba8000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba951ad640 10 monclient(hunting): get_auth_request con 0x7fba80005180 auth_method 0 2024-03-27T21:39:28.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba951ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba951ad640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8ffff640 1 --2- 172.21.15.191:0/4244583511 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba800047c0 0x7fba80004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba951ad640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:39:28.004 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8ffff640 10 monclient(hunting): get_auth_request con 0x7fba800047c0 auth_method 0 2024-03-27T21:39:28.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:39:28.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba951ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba951ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba949ac640 1 --2- 172.21.15.191:0/4244583511 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fba80003b00 0x7fba80003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba951ad640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:39:28.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba949ac640 10 monclient(hunting): get_auth_request con 0x7fba80003b00 auth_method 0 2024-03-27T21:39:28.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba949ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba949ac640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba949ac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:39:28.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:39:28.005 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba949ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba949ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba949ac640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:39:28.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba951ad640 10 monclient(hunting): handle_auth_done global_id 4126 payload 293 2024-03-27T21:39:28.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba951ad640 1 -- 172.21.15.191:0/4244583511 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fba80003b00 msgr2=0x7fba80003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:28.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba951ad640 1 --2- 172.21.15.191:0/4244583511 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fba80003b00 0x7fba80003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba951ad640 1 -- 172.21.15.191:0/4244583511 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba800047c0 msgr2=0x7fba80004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:28.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba951ad640 1 --2- 172.21.15.191:0/4244583511 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba800047c0 0x7fba80004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba951ad640 10 monclient: _finish_hunting 0 2024-03-27T21:39:28.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba951ad640 1 monclient: found mon.b 2024-03-27T21:39:28.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba951ad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:28.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba951ad640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:28.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba951ad640 1 -- 172.21.15.191:0/4244583511 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba80003520 con 0x7fba80005180 2024-03-27T21:39:28.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba951ad640 10 monclient: _finish_auth 0 2024-03-27T21:39:28.006 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba951ad640 10 monclient: _check_auth_tickets 2024-03-27T21:39:28.007 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.998+0000 7fba951ad640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:28.007 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.999+0000 7fba951ad640 1 --2- 172.21.15.191:0/4244583511 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba80005180 0x7fba8000b980 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fba88001fd0 tx=0x7fba800057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a95edad27f842a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:28.007 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.999+0000 7fba8f7fe640 1 -- 172.21.15.191:0/4244583511 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba88003b70 con 0x7fba80005180 2024-03-27T21:39:28.007 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.999+0000 7fba8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:28.007 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.999+0000 7fba8f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:28.007 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.999+0000 7fba8f7fe640 10 monclient: dump: 2024-03-27T21:39:28.007 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:28.007 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.007 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.007 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.007 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:28.007 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.007 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:28.007 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:28.007 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:28.007 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.008 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.999+0000 7fba8f7fe640 1 -- 172.21.15.191:0/4244583511 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba88003cd0 con 0x7fba80005180 2024-03-27T21:39:28.008 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.999+0000 7fba96c10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:28.008 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.999+0000 7fba8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:28.008 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.999+0000 7fba8f7fe640 1 -- 172.21.15.191:0/4244583511 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba880052f0 con 0x7fba80005180 2024-03-27T21:39:28.008 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.999+0000 7fba96c10640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:28.008 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.999+0000 7fba96c10640 4 set_mon_vals no callback set 2024-03-27T21:39:28.008 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.999+0000 7fba8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:28.008 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.999+0000 7fba8f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:28.008 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:27.999+0000 7fba8f7fe640 10 monclient: dump: 2024-03-27T21:39:28.008 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:28.008 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.008 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.008 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.008 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:28.008 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.009 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:28.009 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:28.009 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:28.009 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.009 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.000+0000 7fba96c10640 10 monclient: shutdown 2024-03-27T21:39:28.009 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.000+0000 7fba96c10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:28.009 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.000+0000 7fba96c10640 1 -- 172.21.15.191:0/4244583511 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba80005180 msgr2=0x7fba8000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.009 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.000+0000 7fba96c10640 1 --2- 172.21.15.191:0/4244583511 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba80005180 0x7fba8000b980 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fba88001fd0 tx=0x7fba800057f0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.009 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.000+0000 7fba951ad640 1 -- 172.21.15.191:0/4244583511 reap_dead start 2024-03-27T21:39:28.009 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.000+0000 7fba96c10640 1 -- 172.21.15.191:0/4244583511 shutdown_connections 2024-03-27T21:39:28.009 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.000+0000 7fba96c10640 1 -- 172.21.15.191:0/4244583511 >> 172.21.15.191:0/4244583511 conn(0x7fba900608e0 msgr2=0x7fba90060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:28.009 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.000+0000 7fba96c10640 1 -- 172.21.15.191:0/4244583511 shutdown_connections 2024-03-27T21:39:28.009 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.000+0000 7fba96c10640 1 -- 172.21.15.191:0/4244583511 wait complete. 2024-03-27T21:39:28.009 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.001+0000 7fba96c10640 10 monclient: build_initial_monmap 2024-03-27T21:39:28.009 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.001+0000 7fba96c10640 10 monclient: monmap: 2024-03-27T21:39:28.009 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-27T21:39:28.010 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:28.010 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-27T21:39:28.010 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-27T21:39:28.010 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:28.010 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-27T21:39:28.010 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:28.010 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:28.010 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:28.010 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.010 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.001+0000 7fba96c10640 1 librados: starting msgr at 2024-03-27T21:39:28.010 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.001+0000 7fba96c10640 1 librados: starting objecter 2024-03-27T21:39:28.010 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.001+0000 7fba96c10640 1 Processor -- start 2024-03-27T21:39:28.010 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.001+0000 7fba96c10640 1 -- start start 2024-03-27T21:39:28.010 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.001+0000 7fba96c10640 1 librados: setting wanted keys 2024-03-27T21:39:28.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.001+0000 7fba96c10640 1 librados: calling monclient init 2024-03-27T21:39:28.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.001+0000 7fba96c10640 10 monclient: init 2024-03-27T21:39:28.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.002+0000 7fba96c10640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:28.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.002+0000 7fba96c10640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:39:28.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.002+0000 7fba96c10640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba80005180 0x7fba901cc140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.002+0000 7fba96c10640 10 monclient(hunting): picked mon.noname-b con 0x7fba80005180 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:28.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.002+0000 7fba96c10640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba9013da90 0x7fba9013a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.002+0000 7fba96c10640 10 monclient(hunting): picked mon.noname-a con 0x7fba9013da90 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:28.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.002+0000 7fba96c10640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fba9013a670 0x7fba901d9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.002+0000 7fba96c10640 10 monclient(hunting): picked mon.noname-c con 0x7fba9013a670 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:28.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.002+0000 7fba96c10640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.002+0000 7fba96c10640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9013e0b0 con 0x7fba9013da90 2024-03-27T21:39:28.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.002+0000 7fba96c10640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.002+0000 7fba96c10640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba901ccc80 con 0x7fba80005180 2024-03-27T21:39:28.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.002+0000 7fba96c10640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.002+0000 7fba96c10640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba9013d220 con 0x7fba9013a670 2024-03-27T21:39:28.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.002+0000 7fba96c10640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:28.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.002+0000 7fba96c10640 10 monclient(hunting): authenticate will time out at 785.286545s 2024-03-27T21:39:28.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.002+0000 7fba949ac640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba80005180 0x7fba901cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba949ac640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba80005180 0x7fba901cc140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:52512/0 (socket says 172.21.15.191:52512) 2024-03-27T21:39:28.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba949ac640 1 -- 172.21.15.191:0/694056425 learned_addr learned my addr 172.21.15.191:0/694056425 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-27T21:39:28.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba949ac640 10 monclient(hunting): get_auth_request con 0x7fba80005180 auth_method 0 2024-03-27T21:39:28.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba949ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba949ac640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba949ac640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba951ad640 1 --2- 172.21.15.191:0/694056425 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fba9013a670 0x7fba901d9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8ffff640 1 --2- 172.21.15.191:0/694056425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba9013da90 0x7fba9013a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba951ad640 10 monclient(hunting): get_auth_request con 0x7fba9013a670 auth_method 0 2024-03-27T21:39:28.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba951ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba951ad640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba951ad640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba949ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba949ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba949ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:28.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8ffff640 10 monclient(hunting): get_auth_request con 0x7fba9013da90 auth_method 0 2024-03-27T21:39:28.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba951ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba951ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba951ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:28.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:28.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba949ac640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2024-03-27T21:39:28.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba949ac640 1 -- 172.21.15.191:0/694056425 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fba9013a670 msgr2=0x7fba901d9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba949ac640 1 --2- 172.21.15.191:0/694056425 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fba9013a670 0x7fba901d9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba949ac640 1 -- 172.21.15.191:0/694056425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba9013da90 msgr2=0x7fba9013a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba949ac640 1 --2- 172.21.15.191:0/694056425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba9013da90 0x7fba9013a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba949ac640 10 monclient: _finish_hunting 0 2024-03-27T21:39:28.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba949ac640 1 monclient: found mon.noname-b 2024-03-27T21:39:28.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba949ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:28.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba949ac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:39:28.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba949ac640 1 -- 172.21.15.191:0/694056425 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9013e630 con 0x7fba80005180 2024-03-27T21:39:28.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba949ac640 1 --2- 172.21.15.191:0/694056425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba80005180 0x7fba901cc140 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fba84011c00 tx=0x7fba8400f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:28.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8d7fa640 10 client.?.objecter ms_handle_connect 0x7fba80005180 2024-03-27T21:39:28.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8d7fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:28.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8d7fa640 1 -- 172.21.15.191:0/694056425 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba84011cf0 con 0x7fba80005180 2024-03-27T21:39:28.015 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:28.015 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:28.015 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8d7fa640 10 monclient: dump: 2024-03-27T21:39:28.015 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:28.015 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.015 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.015 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.015 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:28.015 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.015 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:28.015 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:28.015 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:28.015 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.015 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8d7fa640 10 monclient: _finish_auth 0 2024-03-27T21:39:28.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8d7fa640 10 monclient: _check_auth_tickets 2024-03-27T21:39:28.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:28.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba96c10640 5 monclient: authenticate success, global_id 4129 2024-03-27T21:39:28.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba96c10640 10 monclient: _renew_subs 2024-03-27T21:39:28.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8d7fa640 1 -- 172.21.15.191:0/694056425 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba84011e50 con 0x7fba80005180 2024-03-27T21:39:28.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba96c10640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:28.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba96c10640 1 -- 172.21.15.191:0/694056425 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba9013aaf0 con 0x7fba80005180 2024-03-27T21:39:28.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:28.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8d7fa640 1 -- 172.21.15.191:0/694056425 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba84014640 con 0x7fba80005180 2024-03-27T21:39:28.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:28.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:28.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8d7fa640 10 monclient: dump: 2024-03-27T21:39:28.016 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:28.016 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.016 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.017 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.017 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:28.017 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.017 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:28.017 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:28.017 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:28.017 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8effd640 4 set_mon_vals no callback set 2024-03-27T21:39:28.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba96c10640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:28.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba96c10640 10 monclient: _renew_subs 2024-03-27T21:39:28.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba96c10640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:28.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba96c10640 1 -- 172.21.15.191:0/694056425 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba901da060 con 0x7fba80005180 2024-03-27T21:39:28.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba96c10640 1 librados: init done 2024-03-27T21:39:28.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8d7fa640 1 -- 172.21.15.191:0/694056425 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fba84014aa0 con 0x7fba80005180 2024-03-27T21:39:28.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8d7fa640 1 -- 172.21.15.191:0/694056425 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fba840263f0 con 0x7fba80005180 2024-03-27T21:39:28.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8d7fa640 10 client.4129.objecter ms_dispatch 0x7fba90000b90 osd_map(3..3 src has 1..3) v4 2024-03-27T21:39:28.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8d7fa640 3 client.4129.objecter handle_osd_map got epochs [3,3] > 0 2024-03-27T21:39:28.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8d7fa640 3 client.4129.objecter handle_osd_map decoding full epoch 3 2024-03-27T21:39:28.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.003+0000 7fba8d7fa640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-27T21:39:28.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.004+0000 7fba96c10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:28.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.004+0000 7fba96c10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:28.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.004+0000 7fba96c10640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:28.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.004+0000 7fba96c10640 1 -- 172.21.15.191:0/694056425 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba9013e320 con 0x7fba80005180 2024-03-27T21:39:28.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.006+0000 7fba8d7fa640 1 -- 172.21.15.191:0/694056425 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fba8401d400 con 0x7fba80005180 2024-03-27T21:39:28.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.006+0000 7fba8d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:28.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.006+0000 7fba8d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:28.065 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.078+0000 7fba96c10640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d960adf1-3d83-4e24-a612-8df53fa43d0c", "id": 1}] 2024-03-27T21:39:28.065 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.078+0000 7fba96c10640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d960adf1-3d83-4e24-a612-8df53fa43d0c", "id": 1}] 2024-03-27T21:39:28.065 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.078+0000 7fba96c10640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:28.065 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.078+0000 7fba96c10640 1 -- 172.21.15.191:0/694056425 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d960adf1-3d83-4e24-a612-8df53fa43d0c", "id": 1} v 0) v1 -- 0x7fba9013de70 con 0x7fba80005180 2024-03-27T21:39:28.072 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.085+0000 7fba8d7fa640 1 -- 172.21.15.191:0/694056425 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d960adf1-3d83-4e24-a612-8df53fa43d0c", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fba8401d5a0 con 0x7fba80005180 2024-03-27T21:39:28.072 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.085+0000 7fba8d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d960adf1-3d83-4e24-a612-8df53fa43d0c", "id": 1}] 2024-03-27T21:39:28.073 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.085+0000 7fba8d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:28.073 INFO:teuthology.orchestra.run.smithi191.stdout:1 2024-03-27T21:39:28.074 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.086+0000 7fba96c10640 10 librados: watch_flush enter 2024-03-27T21:39:28.074 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.086+0000 7fba96c10640 10 librados: watch_flush exit 2024-03-27T21:39:28.074 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.086+0000 7fba96c10640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-27T21:39:28.074 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.086+0000 7fba96c10640 10 client.4129.objecter successfully canceled tick 2024-03-27T21:39:28.074 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.086+0000 7fba96c10640 10 monclient: shutdown 2024-03-27T21:39:28.074 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.086+0000 7fba96c10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:28.074 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.086+0000 7fba96c10640 1 -- 172.21.15.191:0/694056425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba80005180 msgr2=0x7fba901cc140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.075 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.086+0000 7fba96c10640 1 --2- 172.21.15.191:0/694056425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba80005180 0x7fba901cc140 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fba84011c00 tx=0x7fba8400f610 comp rx=0 tx=0).stop 2024-03-27T21:39:28.075 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.086+0000 7fba96c10640 1 -- 172.21.15.191:0/694056425 shutdown_connections 2024-03-27T21:39:28.075 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.086+0000 7fba96c10640 1 --2- 172.21.15.191:0/694056425 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fba9013a670 0x7fba901d9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.075 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.086+0000 7fba96c10640 1 --2- 172.21.15.191:0/694056425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba9013da90 0x7fba9013a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.075 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.086+0000 7fba96c10640 1 --2- 172.21.15.191:0/694056425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba80005180 0x7fba901cc140 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.075 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.086+0000 7fba96c10640 1 -- 172.21.15.191:0/694056425 >> 172.21.15.191:0/694056425 conn(0x7fba900608e0 msgr2=0x7fba90068be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:28.075 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.086+0000 7fba96c10640 1 -- 172.21.15.191:0/694056425 shutdown_connections 2024-03-27T21:39:28.075 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.087+0000 7fba96c10640 1 -- 172.21.15.191:0/694056425 wait complete. 2024-03-27T21:39:28.075 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.087+0000 7fba96c10640 1 librados: shutdown 2024-03-27T21:39:28.084 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new b8ffc48b-cf6f-4701-a6aa-c25fdb0f7977 2 2024-03-27T21:39:28.183 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.196+0000 7f726cb1b640 10 monclient: get_monmap_and_config 2024-03-27T21:39:28.183 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.196+0000 7f726cb1b640 10 monclient: build_initial_monmap 2024-03-27T21:39:28.183 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.196+0000 7f726cb1b640 10 monclient: monmap: 2024-03-27T21:39:28.183 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-27T21:39:28.183 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.184 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:39:28.197904+0000 2024-03-27T21:39:28.184 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:39:28.197904+0000 2024-03-27T21:39:28.184 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:28.184 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.184 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:28.184 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:28.184 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:28.184 INFO:teuthology.orchestra.run.smithi191.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:28.184 INFO:teuthology.orchestra.run.smithi191.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:28.184 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.194 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.197+0000 7f726cb1b640 1 Processor -- start 2024-03-27T21:39:28.194 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.197+0000 7f726cb1b640 1 -- start start 2024-03-27T21:39:28.194 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.197+0000 7f726cb1b640 10 monclient: init 2024-03-27T21:39:28.194 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.197+0000 7f726cb1b640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:28.194 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.197+0000 7f726cb1b640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-03-27T21:39:28.194 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.197+0000 7f726cb1b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f72680b40b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:28.195 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.197+0000 7f726cb1b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f726813b020 0x7f726813b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.195 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f726cb1b640 10 monclient(hunting): picked mon.noname-a con 0x7f726813b020 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:28.195 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f726cb1b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f72681416f0 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:28.195 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f726cb1b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72680b3830 con 0x7f72680b40b0 2024-03-27T21:39:28.195 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f726cb1b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72680aa520 con 0x7f72681416f0 2024-03-27T21:39:28.195 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f726cb1b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.195 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f726cb1b640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72680b4490 con 0x7f726813b020 2024-03-27T21:39:28.195 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f726cb1b640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:28.195 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f726cb1b640 10 monclient(hunting): authenticate will time out at 785.481690s 2024-03-27T21:39:28.195 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f7266d9d640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f72681416f0 0x7f7268141b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:37436/0 (socket says 172.21.15.191:37436) 2024-03-27T21:39:28.195 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f7266d9d640 1 -- 172.21.15.191:0/678768369 learned_addr learned my addr 172.21.15.191:0/678768369 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-27T21:39:28.195 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f725ffff640 1 --2- 172.21.15.191:0/678768369 >> v2:172.21.15.60:3300/0 conn(0x7f726813b020 0x7f726813b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.195 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f725ffff640 10 monclient(hunting): get_auth_request con 0x7f726813b020 auth_method 0 2024-03-27T21:39:28.195 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f725ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f725ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f725ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f725ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f725ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f725ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:28.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f7265d9b640 1 -- 172.21.15.191:0/678768369 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812551581 0 0) 0x7f72680aa520 con 0x7f72681416f0 2024-03-27T21:39:28.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f7265d9b640 10 monclient(hunting): _negotiate 2024-03-27T21:39:28.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f7265d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f7265d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f7265d9b640 10 monclient(hunting): my global_id is 4132 2024-03-27T21:39:28.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f7265d9b640 1 -- 172.21.15.191:0/678768369 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7254003520 con 0x7f72681416f0 2024-03-27T21:39:28.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f7265d9b640 1 -- 172.21.15.191:0/678768369 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077885933 0 0) 0x7f72680b3830 con 0x7f72680b40b0 2024-03-27T21:39:28.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f7265d9b640 10 monclient(hunting): _negotiate 2024-03-27T21:39:28.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f7265d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f7265d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f7265d9b640 10 monclient(hunting): my global_id is 4125 2024-03-27T21:39:28.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f7265d9b640 1 -- 172.21.15.191:0/678768369 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72680aa520 con 0x7f72680b40b0 2024-03-27T21:39:28.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f725ffff640 10 monclient(hunting): handle_auth_done global_id 4122 payload 274 2024-03-27T21:39:28.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f725ffff640 1 -- 172.21.15.191:0/678768369 >> v1:172.21.15.191:6789/0 conn(0x7f72681416f0 legacy=0x7f7268141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f725ffff640 1 -- 172.21.15.191:0/678768369 >> v1:172.21.15.60:6789/0 conn(0x7f72680b40b0 legacy=0x7f72680a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f725ffff640 10 monclient: _finish_hunting 0 2024-03-27T21:39:28.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f725ffff640 1 monclient: found mon.noname-a 2024-03-27T21:39:28.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f725ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:28.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f725ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:28.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.198+0000 7f725ffff640 1 -- 172.21.15.191:0/678768369 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72680a9470 con 0x7f726813b020 2024-03-27T21:39:28.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.199+0000 7f725ffff640 1 --2- 172.21.15.191:0/678768369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f726813b020 0x7f726813b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f72500097a0 tx=0x7f725002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab07f34601cbe0dc server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:28.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.199+0000 7f7265d9b640 1 -- 172.21.15.191:0/678768369 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7250030610 con 0x7f726813b020 2024-03-27T21:39:28.198 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.199+0000 7f7265d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:28.198 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.199+0000 7f7265d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:28.198 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.199+0000 7f7265d9b640 10 monclient: dump: 2024-03-27T21:39:28.198 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:28.198 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.198 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.198 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.198 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:28.198 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.198 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:28.198 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:28.198 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:28.198 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.198 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.199+0000 7f7265d9b640 10 monclient: _finish_auth 0 2024-03-27T21:39:28.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.199+0000 7f7265d9b640 10 monclient: _check_auth_tickets 2024-03-27T21:39:28.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.199+0000 7f7265d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:28.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.199+0000 7f7265d9b640 1 -- 172.21.15.191:0/678768369 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7250030770 con 0x7f726813b020 2024-03-27T21:39:28.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.199+0000 7f726cb1b640 5 monclient: authenticate success, global_id 4122 2024-03-27T21:39:28.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.199+0000 7f726cb1b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:28.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.199+0000 7f7265d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:28.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.199+0000 7f726cb1b640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:28.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.199+0000 7f7265d9b640 1 -- 172.21.15.191:0/678768369 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7250030b00 con 0x7f726813b020 2024-03-27T21:39:28.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.199+0000 7f7265d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:28.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.199+0000 7f726cb1b640 4 set_mon_vals no callback set 2024-03-27T21:39:28.200 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.199+0000 7f7265d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:28.200 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.199+0000 7f7265d9b640 10 monclient: dump: 2024-03-27T21:39:28.200 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:28.200 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.200 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.200 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.200 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:28.200 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.200 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:28.200 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:28.200 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:28.201 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.201 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.199+0000 7f726cb1b640 10 monclient: shutdown 2024-03-27T21:39:28.201 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.200+0000 7f726cb1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:28.201 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.200+0000 7f726cb1b640 1 -- 172.21.15.191:0/678768369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f726813b020 msgr2=0x7f726813b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.201 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.200+0000 7f726cb1b640 1 --2- 172.21.15.191:0/678768369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f726813b020 0x7f726813b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f72500097a0 tx=0x7f725002b960 comp rx=0 tx=0).stop 2024-03-27T21:39:28.201 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.200+0000 7f726cb1b640 1 -- 172.21.15.191:0/678768369 shutdown_connections 2024-03-27T21:39:28.201 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.200+0000 7f726cb1b640 1 --2- 172.21.15.191:0/678768369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f726813b020 0x7f726813b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.201 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.200+0000 7f726cb1b640 1 -- 172.21.15.191:0/678768369 >> 172.21.15.191:0/678768369 conn(0x7f72680608e0 msgr2=0x7f7268060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:28.201 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.200+0000 7f726cb1b640 1 -- 172.21.15.191:0/678768369 shutdown_connections 2024-03-27T21:39:28.201 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.200+0000 7f726cb1b640 1 -- 172.21.15.191:0/678768369 wait complete. 2024-03-27T21:39:28.201 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.200+0000 7f726cb1b640 10 monclient: build_initial_monmap 2024-03-27T21:39:28.201 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.200+0000 7f726cb1b640 10 monclient: monmap: 2024-03-27T21:39:28.201 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-27T21:39:28.202 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:28.202 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-27T21:39:28.202 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-27T21:39:28.202 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:28.202 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-27T21:39:28.202 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:28.202 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:28.202 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:28.202 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.202 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 1 librados: starting msgr at 2024-03-27T21:39:28.202 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 1 librados: starting objecter 2024-03-27T21:39:28.202 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 1 Processor -- start 2024-03-27T21:39:28.202 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 1 -- start start 2024-03-27T21:39:28.203 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 1 librados: setting wanted keys 2024-03-27T21:39:28.203 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 1 librados: calling monclient init 2024-03-27T21:39:28.203 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 10 monclient: init 2024-03-27T21:39:28.203 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:28.203 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:39:28.203 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f72680b40b0 0x7f7268139d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.203 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 10 monclient(hunting): picked mon.noname-c con 0x7f72680b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:28.203 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f726813b020 0x7f726813e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.203 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 10 monclient(hunting): picked mon.noname-b con 0x7f726813b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:28.203 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72681416f0 0x7f72681396c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.203 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 10 monclient(hunting): picked mon.noname-a con 0x7f72681416f0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:28.203 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.203 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726813a490 con 0x7f72681416f0 2024-03-27T21:39:28.203 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.204 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7268133130 con 0x7f726813b020 2024-03-27T21:39:28.204 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.204 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72681332f0 con 0x7f72680b40b0 2024-03-27T21:39:28.204 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:28.204 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f726cb1b640 10 monclient(hunting): authenticate will time out at 785.485470s 2024-03-27T21:39:28.204 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f725ffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f726813b020 0x7f726813e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.204 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f725ffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f726813b020 0x7f726813e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:52524/0 (socket says 172.21.15.191:52524) 2024-03-27T21:39:28.204 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f725ffff640 1 -- 172.21.15.191:0/1236022222 learned_addr learned my addr 172.21.15.191:0/1236022222 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-27T21:39:28.204 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f725ffff640 10 monclient(hunting): get_auth_request con 0x7f726813b020 auth_method 0 2024-03-27T21:39:28.204 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f725ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.204 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f725ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.204 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f725ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.201+0000 7f7266d9d640 1 --2- 172.21.15.191:0/1236022222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72681416f0 0x7f72681396c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f7266d9d640 10 monclient(hunting): get_auth_request con 0x7f72681416f0 auth_method 0 2024-03-27T21:39:28.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f7266d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f7266d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f7266d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f726659c640 1 --2- 172.21.15.191:0/1236022222 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f72680b40b0 0x7f7268139d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f726659c640 10 monclient(hunting): get_auth_request con 0x7f72680b40b0 auth_method 0 2024-03-27T21:39:28.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f726659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f726659c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f726659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:28.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f7266d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.206 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f7266d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.206 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f7266d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:28.206 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f726659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.206 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f726659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.206 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f726659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:28.206 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f7266d9d640 10 monclient(hunting): handle_auth_done global_id 4128 payload 824 2024-03-27T21:39:28.206 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f7266d9d640 1 -- 172.21.15.191:0/1236022222 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f72680b40b0 msgr2=0x7f7268139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.206 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f7266d9d640 1 --2- 172.21.15.191:0/1236022222 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f72680b40b0 0x7f7268139d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.206 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f7266d9d640 1 -- 172.21.15.191:0/1236022222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f726813b020 msgr2=0x7f726813e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.206 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f7266d9d640 1 --2- 172.21.15.191:0/1236022222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f726813b020 0x7f726813e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.206 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f7266d9d640 10 monclient: _finish_hunting 0 2024-03-27T21:39:28.206 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f7266d9d640 1 monclient: found mon.noname-a 2024-03-27T21:39:28.206 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f7266d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:28.207 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f7266d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:28.207 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f7266d9d640 1 -- 172.21.15.191:0/1236022222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7250009400 con 0x7f72681416f0 2024-03-27T21:39:28.207 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725ffff640 1 --2- 172.21.15.191:0/1236022222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f726813b020 0x7f726813e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:28.207 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f7266d9d640 1 --2- 172.21.15.191:0/1236022222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72681416f0 0x7f72681396c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f726000ca70 tx=0x7f726000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:28.207 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725effd640 10 client.?.objecter ms_handle_connect 0x7f72681416f0 2024-03-27T21:39:28.207 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725effd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:28.207 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725effd640 1 -- 172.21.15.191:0/1236022222 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726000ab70 con 0x7f72681416f0 2024-03-27T21:39:28.207 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:28.207 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:28.207 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725effd640 10 monclient: dump: 2024-03-27T21:39:28.207 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:28.207 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.207 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.207 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.207 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:28.208 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.208 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:28.208 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:28.208 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:28.208 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.208 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725effd640 10 monclient: _finish_auth 0 2024-03-27T21:39:28.208 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725effd640 10 monclient: _check_auth_tickets 2024-03-27T21:39:28.208 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:28.208 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f726cb1b640 5 monclient: authenticate success, global_id 4128 2024-03-27T21:39:28.208 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725effd640 1 -- 172.21.15.191:0/1236022222 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f726000acd0 con 0x7f72681416f0 2024-03-27T21:39:28.208 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f726cb1b640 10 monclient: _renew_subs 2024-03-27T21:39:28.208 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f726cb1b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:28.208 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f726cb1b640 1 -- 172.21.15.191:0/1236022222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7268133570 con 0x7f72681416f0 2024-03-27T21:39:28.208 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725effd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:28.208 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725effd640 1 -- 172.21.15.191:0/1236022222 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7260012640 con 0x7f72681416f0 2024-03-27T21:39:28.209 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:28.209 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:28.209 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725effd640 10 monclient: dump: 2024-03-27T21:39:28.209 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:28.209 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.209 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.209 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.209 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:28.209 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.209 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:28.210 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:28.210 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:28.210 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.210 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f726559a640 4 set_mon_vals no callback set 2024-03-27T21:39:28.210 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f726cb1b640 10 client.4128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:28.210 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f726cb1b640 10 monclient: _renew_subs 2024-03-27T21:39:28.210 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f726cb1b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:28.210 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f726cb1b640 1 -- 172.21.15.191:0/1236022222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7268133a50 con 0x7f72681416f0 2024-03-27T21:39:28.210 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f726cb1b640 1 librados: init done 2024-03-27T21:39:28.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725effd640 1 -- 172.21.15.191:0/1236022222 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7260012aa0 con 0x7f72681416f0 2024-03-27T21:39:28.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725effd640 1 -- 172.21.15.191:0/1236022222 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f7260007490 con 0x7f72681416f0 2024-03-27T21:39:28.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725effd640 10 client.4128.objecter ms_dispatch 0x7f7268000b90 osd_map(4..4 src has 1..4) v4 2024-03-27T21:39:28.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725effd640 3 client.4128.objecter handle_osd_map got epochs [4,4] > 0 2024-03-27T21:39:28.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725effd640 3 client.4128.objecter handle_osd_map decoding full epoch 4 2024-03-27T21:39:28.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.202+0000 7f725effd640 20 client.4128.objecter dump_active .. 0 homeless 2024-03-27T21:39:28.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.203+0000 7f726cb1b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:28.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.203+0000 7f726cb1b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:28.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.203+0000 7f726cb1b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:28.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.203+0000 7f726cb1b640 1 -- 172.21.15.191:0/1236022222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f726813a5d0 con 0x7f72681416f0 2024-03-27T21:39:28.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.207+0000 7f725effd640 1 -- 172.21.15.191:0/1236022222 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f726813a5d0 con 0x7f72681416f0 2024-03-27T21:39:28.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.207+0000 7f725effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:28.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.207+0000 7f725effd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:28.212 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:28.220+0000 7f6438632040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-27T21:39:28.261 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:28.274+0000 7f6438632040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-27T21:39:28.263 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.276+0000 7f726cb1b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b8ffc48b-cf6f-4701-a6aa-c25fdb0f7977", "id": 2}] 2024-03-27T21:39:28.263 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.276+0000 7f726cb1b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b8ffc48b-cf6f-4701-a6aa-c25fdb0f7977", "id": 2}] 2024-03-27T21:39:28.263 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.276+0000 7f726cb1b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:28.263 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.276+0000 7f726cb1b640 1 -- 172.21.15.191:0/1236022222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b8ffc48b-cf6f-4701-a6aa-c25fdb0f7977", "id": 2} v 0) v1 -- 0x7f7268069470 con 0x7f72681416f0 2024-03-27T21:39:28.266 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.279+0000 7f725effd640 1 -- 172.21.15.191:0/1236022222 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b8ffc48b-cf6f-4701-a6aa-c25fdb0f7977", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7260017070 con 0x7f72681416f0 2024-03-27T21:39:28.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.279+0000 7f725effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b8ffc48b-cf6f-4701-a6aa-c25fdb0f7977", "id": 2}] 2024-03-27T21:39:28.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.279+0000 7f725effd640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:28.267 INFO:teuthology.orchestra.run.smithi191.stdout:2 2024-03-27T21:39:28.268 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.280+0000 7f726cb1b640 10 librados: watch_flush enter 2024-03-27T21:39:28.268 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.280+0000 7f726cb1b640 10 librados: watch_flush exit 2024-03-27T21:39:28.268 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.280+0000 7f726cb1b640 20 client.4128.objecter shutdown clearing up homeless session... 2024-03-27T21:39:28.268 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.280+0000 7f726cb1b640 10 client.4128.objecter successfully canceled tick 2024-03-27T21:39:28.269 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.280+0000 7f726cb1b640 10 monclient: shutdown 2024-03-27T21:39:28.269 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.280+0000 7f726cb1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:28.269 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.280+0000 7f726cb1b640 1 -- 172.21.15.191:0/1236022222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72681416f0 msgr2=0x7f72681396c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.269 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.280+0000 7f726cb1b640 1 --2- 172.21.15.191:0/1236022222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72681416f0 0x7f72681396c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f726000ca70 tx=0x7f726000cf30 comp rx=0 tx=0).stop 2024-03-27T21:39:28.269 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.280+0000 7f726cb1b640 1 -- 172.21.15.191:0/1236022222 shutdown_connections 2024-03-27T21:39:28.269 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.280+0000 7f726cb1b640 1 --2- 172.21.15.191:0/1236022222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72681416f0 0x7f72681396c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.269 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.280+0000 7f726cb1b640 1 --2- 172.21.15.191:0/1236022222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f726813b020 0x7f726813e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.269 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.280+0000 7f726cb1b640 1 --2- 172.21.15.191:0/1236022222 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f72680b40b0 0x7f7268139d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.269 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.280+0000 7f726cb1b640 1 -- 172.21.15.191:0/1236022222 >> 172.21.15.191:0/1236022222 conn(0x7f72680608e0 msgr2=0x7f72680b5f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:28.269 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.280+0000 7f726cb1b640 1 -- 172.21.15.191:0/1236022222 shutdown_connections 2024-03-27T21:39:28.269 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.280+0000 7f726cb1b640 1 -- 172.21.15.191:0/1236022222 wait complete. 2024-03-27T21:39:28.269 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.280+0000 7f726cb1b640 1 librados: shutdown 2024-03-27T21:39:28.278 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 0f14f2d8-b271-48e4-a198-aa5279a01c06 3 2024-03-27T21:39:28.308 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:28.321+0000 7f6438632040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-27T21:39:28.363 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:28.376+0000 7f6438632040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-27T21:39:28.375 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.388+0000 7f9c28192640 10 monclient: get_monmap_and_config 2024-03-27T21:39:28.375 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.388+0000 7f9c28192640 10 monclient: build_initial_monmap 2024-03-27T21:39:28.375 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.388+0000 7f9c28192640 10 monclient: monmap: 2024-03-27T21:39:28.375 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-27T21:39:28.376 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.376 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:39:28.390089+0000 2024-03-27T21:39:28.376 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:39:28.390089+0000 2024-03-27T21:39:28.376 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:28.376 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.376 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:28.376 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:28.376 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:28.376 INFO:teuthology.orchestra.run.smithi191.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:28.376 INFO:teuthology.orchestra.run.smithi191.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:28.376 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.384 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.389+0000 7f9c28192640 1 Processor -- start 2024-03-27T21:39:28.384 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.389+0000 7f9c28192640 1 -- start start 2024-03-27T21:39:28.384 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.389+0000 7f9c28192640 10 monclient: init 2024-03-27T21:39:28.384 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c28192640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:28.384 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c28192640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-03-27T21:39:28.384 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c28192640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9c200b40b0 0x7f9c200a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.385 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c28192640 10 monclient(hunting): picked mon.noname-c con 0x7f9c200b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:28.385 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c28192640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9c2013b020 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:28.385 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c28192640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9c2013b480 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:28.385 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c28192640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c2013b8d0 con 0x7f9c2013b020 2024-03-27T21:39:28.385 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c28192640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c200b3830 con 0x7f9c2013b480 2024-03-27T21:39:28.385 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c28192640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.385 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c28192640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c200b4490 con 0x7f9c200b40b0 2024-03-27T21:39:28.385 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c28192640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:28.385 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c28192640 10 monclient(hunting): authenticate will time out at 785.673840s 2024-03-27T21:39:28.385 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c25f2e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9c200b40b0 0x7f9c200a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.385 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c25f2e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9c200b40b0 0x7f9c200a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:52526/0 (socket says 172.21.15.191:52526) 2024-03-27T21:39:28.385 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c25f2e640 1 -- 172.21.15.191:0/2487912677 learned_addr learned my addr 172.21.15.191:0/2487912677 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-27T21:39:28.385 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c25f2e640 10 monclient(hunting): get_auth_request con 0x7f9c200b40b0 auth_method 0 2024-03-27T21:39:28.385 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c25f2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.385 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c25f2e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.386 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c25f2e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.386 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c25f2e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.386 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c25f2e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.386 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c25f2e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:28.386 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c25f2e640 10 monclient(hunting): handle_auth_done global_id 4138 payload 274 2024-03-27T21:39:28.386 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c25f2e640 1 -- 172.21.15.191:0/2487912677 >> v1:172.21.15.191:6789/0 conn(0x7f9c2013b480 legacy=0x7f9c2014bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.386 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c25f2e640 1 -- 172.21.15.191:0/2487912677 >> v1:172.21.15.60:6789/0 conn(0x7f9c2013b020 legacy=0x7f9c201416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.386 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c25f2e640 10 monclient: _finish_hunting 0 2024-03-27T21:39:28.386 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c25f2e640 1 monclient: found mon.noname-c 2024-03-27T21:39:28.386 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c25f2e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:28.386 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c25f2e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:39:28.386 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.390+0000 7f9c25f2e640 1 -- 172.21.15.191:0/2487912677 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c200aa520 con 0x7f9c200b40b0 2024-03-27T21:39:28.386 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.391+0000 7f9c25f2e640 1 --2- 172.21.15.191:0/2487912677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c200b40b0 0x7f9c200a8b90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c009870 tx=0x7f9c1c02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a99f170c70caca4b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:28.387 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.391+0000 7f9c24f2c640 1 -- 172.21.15.191:0/2487912677 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c030860 con 0x7f9c200b40b0 2024-03-27T21:39:28.387 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.391+0000 7f9c24f2c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:28.387 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.391+0000 7f9c24f2c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:28.387 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.391+0000 7f9c24f2c640 10 monclient: dump: 2024-03-27T21:39:28.387 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:28.387 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.387 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.387 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.387 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:28.387 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.387 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:28.387 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:28.387 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:28.387 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.388 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.391+0000 7f9c24f2c640 10 monclient: _finish_auth 0 2024-03-27T21:39:28.388 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.391+0000 7f9c24f2c640 10 monclient: _check_auth_tickets 2024-03-27T21:39:28.388 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.391+0000 7f9c24f2c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:28.388 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.391+0000 7f9c24f2c640 1 -- 172.21.15.191:0/2487912677 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1c0309c0 con 0x7f9c200b40b0 2024-03-27T21:39:28.388 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.391+0000 7f9c24f2c640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:28.388 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.391+0000 7f9c24f2c640 1 -- 172.21.15.191:0/2487912677 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c030d50 con 0x7f9c200b40b0 2024-03-27T21:39:28.388 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.391+0000 7f9c28192640 5 monclient: authenticate success, global_id 4138 2024-03-27T21:39:28.388 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.391+0000 7f9c28192640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:28.388 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.391+0000 7f9c28192640 4 set_mon_vals no callback set 2024-03-27T21:39:28.388 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.391+0000 7f9c24f2c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:28.388 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.391+0000 7f9c24f2c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:28.388 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.391+0000 7f9c24f2c640 10 monclient: dump: 2024-03-27T21:39:28.388 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:28.388 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.389 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.389 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.389 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:28.389 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.389 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:28.389 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:28.389 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:28.389 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.389 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.392+0000 7f9c28192640 10 monclient: shutdown 2024-03-27T21:39:28.389 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.392+0000 7f9c28192640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:28.389 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.392+0000 7f9c28192640 1 -- 172.21.15.191:0/2487912677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c200b40b0 msgr2=0x7f9c200a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.389 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.392+0000 7f9c28192640 1 --2- 172.21.15.191:0/2487912677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c200b40b0 0x7f9c200a8b90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c009870 tx=0x7f9c1c02b9a0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.389 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.392+0000 7f9c28192640 1 -- 172.21.15.191:0/2487912677 shutdown_connections 2024-03-27T21:39:28.389 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.392+0000 7f9c28192640 1 --2- 172.21.15.191:0/2487912677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c200b40b0 0x7f9c200a8b90 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.390 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.392+0000 7f9c28192640 1 -- 172.21.15.191:0/2487912677 >> 172.21.15.191:0/2487912677 conn(0x7f9c200608e0 msgr2=0x7f9c20060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:28.390 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.392+0000 7f9c28192640 1 -- 172.21.15.191:0/2487912677 shutdown_connections 2024-03-27T21:39:28.390 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.392+0000 7f9c28192640 1 -- 172.21.15.191:0/2487912677 wait complete. 2024-03-27T21:39:28.390 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.392+0000 7f9c28192640 10 monclient: build_initial_monmap 2024-03-27T21:39:28.390 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.392+0000 7f9c28192640 10 monclient: monmap: 2024-03-27T21:39:28.390 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-27T21:39:28.390 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:28.390 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-27T21:39:28.390 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-27T21:39:28.390 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:28.390 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-27T21:39:28.390 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:28.390 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:28.390 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:28.391 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 1 librados: starting msgr at 2024-03-27T21:39:28.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 1 librados: starting objecter 2024-03-27T21:39:28.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 1 Processor -- start 2024-03-27T21:39:28.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 1 -- start start 2024-03-27T21:39:28.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 1 librados: setting wanted keys 2024-03-27T21:39:28.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 1 librados: calling monclient init 2024-03-27T21:39:28.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 10 monclient: init 2024-03-27T21:39:28.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:28.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:39:28.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c200b40b0 0x7f9c200affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 10 monclient(hunting): picked mon.noname-c con 0x7f9c200b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:28.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c2013b020 0x7f9c201d6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 10 monclient(hunting): picked mon.noname-b con 0x7f9c2013b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:28.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c2013b480 0x7f9c201d71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 10 monclient(hunting): picked mon.noname-a con 0x7f9c2013b480 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:28.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c200ae960 con 0x7f9c2013b480 2024-03-27T21:39:28.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c200af530 con 0x7f9c2013b020 2024-03-27T21:39:28.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c200b0990 con 0x7f9c200b40b0 2024-03-27T21:39:28.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:28.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c28192640 10 monclient(hunting): authenticate will time out at 785.677512s 2024-03-27T21:39:28.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.393+0000 7f9c2572d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c2013b020 0x7f9c201d6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2572d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c2013b020 0x7f9c201d6ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:52542/0 (socket says 172.21.15.191:52542) 2024-03-27T21:39:28.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2572d640 1 -- 172.21.15.191:0/2832999345 learned_addr learned my addr 172.21.15.191:0/2832999345 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-27T21:39:28.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2572d640 10 monclient(hunting): get_auth_request con 0x7f9c2013b020 auth_method 0 2024-03-27T21:39:28.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2672f640 1 --2- 172.21.15.191:0/2832999345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c2013b480 0x7f9c201d71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2572d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2572d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2572d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c25f2e640 1 --2- 172.21.15.191:0/2832999345 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c200b40b0 0x7f9c200affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2672f640 10 monclient(hunting): get_auth_request con 0x7f9c2013b480 auth_method 0 2024-03-27T21:39:28.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2672f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2672f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2672f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c25f2e640 10 monclient(hunting): get_auth_request con 0x7f9c200b40b0 auth_method 0 2024-03-27T21:39:28.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c25f2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c25f2e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c25f2e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2572d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2572d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2572d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:28.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2672f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2672f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2672f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:28.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c25f2e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c25f2e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c25f2e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:28.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2572d640 10 monclient(hunting): handle_auth_done global_id 4141 payload 824 2024-03-27T21:39:28.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2572d640 1 -- 172.21.15.191:0/2832999345 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c200b40b0 msgr2=0x7f9c200affa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2572d640 1 --2- 172.21.15.191:0/2832999345 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c200b40b0 0x7f9c200affa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2572d640 1 -- 172.21.15.191:0/2832999345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c2013b480 msgr2=0x7f9c201d71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2572d640 1 --2- 172.21.15.191:0/2832999345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c2013b480 0x7f9c201d71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2572d640 10 monclient: _finish_hunting 0 2024-03-27T21:39:28.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2572d640 1 monclient: found mon.noname-b 2024-03-27T21:39:28.395 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2572d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:28.395 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2572d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:39:28.395 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2572d640 1 -- 172.21.15.191:0/2832999345 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1c0094d0 con 0x7f9c2013b020 2024-03-27T21:39:28.395 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c2572d640 1 --2- 172.21.15.191:0/2832999345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c2013b020 0x7f9c201d6ca0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9c1400c450 tx=0x7f9c1400c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:28.395 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c127fc640 10 client.?.objecter ms_handle_connect 0x7f9c2013b020 2024-03-27T21:39:28.395 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c127fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:28.395 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c127fc640 1 -- 172.21.15.191:0/2832999345 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c140052d0 con 0x7f9c2013b020 2024-03-27T21:39:28.395 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:28.395 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:28.395 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c127fc640 10 monclient: dump: 2024-03-27T21:39:28.395 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:28.395 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.395 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.395 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.396 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:28.396 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.396 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:28.396 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:28.396 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:28.396 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.396 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c127fc640 10 monclient: _finish_auth 0 2024-03-27T21:39:28.396 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c127fc640 10 monclient: _check_auth_tickets 2024-03-27T21:39:28.396 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:28.396 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c127fc640 1 -- 172.21.15.191:0/2832999345 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c14005430 con 0x7f9c2013b020 2024-03-27T21:39:28.396 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c28192640 5 monclient: authenticate success, global_id 4141 2024-03-27T21:39:28.396 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:28.396 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c127fc640 1 -- 172.21.15.191:0/2832999345 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c14012870 con 0x7f9c2013b020 2024-03-27T21:39:28.396 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c28192640 10 monclient: _renew_subs 2024-03-27T21:39:28.397 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c28192640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:28.397 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c28192640 1 -- 172.21.15.191:0/2832999345 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c201d6780 con 0x7f9c2013b020 2024-03-27T21:39:28.397 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:28.397 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:28.397 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c13fff640 4 set_mon_vals no callback set 2024-03-27T21:39:28.397 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c127fc640 10 monclient: dump: 2024-03-27T21:39:28.397 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:28.397 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.397 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.398 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.398 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:28.398 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.398 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:28.398 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:28.398 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:28.398 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.398 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c28192640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:28.398 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c28192640 10 monclient: _renew_subs 2024-03-27T21:39:28.398 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c28192640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:28.398 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c28192640 1 -- 172.21.15.191:0/2832999345 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c200b06c0 con 0x7f9c2013b020 2024-03-27T21:39:28.398 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c28192640 1 librados: init done 2024-03-27T21:39:28.398 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c127fc640 1 -- 172.21.15.191:0/2832999345 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9c14012cd0 con 0x7f9c2013b020 2024-03-27T21:39:28.399 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c127fc640 1 -- 172.21.15.191:0/2832999345 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f9c1401b890 con 0x7f9c2013b020 2024-03-27T21:39:28.399 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c127fc640 10 client.4141.objecter ms_dispatch 0x7f9c20000b90 osd_map(5..5 src has 1..5) v4 2024-03-27T21:39:28.399 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c127fc640 3 client.4141.objecter handle_osd_map got epochs [5,5] > 0 2024-03-27T21:39:28.399 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c127fc640 3 client.4141.objecter handle_osd_map decoding full epoch 5 2024-03-27T21:39:28.399 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.394+0000 7f9c127fc640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-27T21:39:28.399 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.395+0000 7f9c28192640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:28.399 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.395+0000 7f9c28192640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:28.399 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.395+0000 7f9c28192640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:28.399 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.395+0000 7f9c28192640 1 -- 172.21.15.191:0/2832999345 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c20069470 con 0x7f9c2013b020 2024-03-27T21:39:28.399 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.397+0000 7f9c127fc640 1 -- 172.21.15.191:0/2832999345 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f9c1401b400 con 0x7f9c2013b020 2024-03-27T21:39:28.399 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.397+0000 7f9c127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:28.399 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.397+0000 7f9c127fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:28.407 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:28.421+0000 7f6438632040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-27T21:39:28.407 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:28.421+0000 7f4298a0e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-27T21:39:28.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.467+0000 7f9c28192640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0f14f2d8-b271-48e4-a198-aa5279a01c06", "id": 3}] 2024-03-27T21:39:28.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.467+0000 7f9c28192640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0f14f2d8-b271-48e4-a198-aa5279a01c06", "id": 3}] 2024-03-27T21:39:28.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.467+0000 7f9c28192640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:28.455 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.467+0000 7f9c28192640 1 -- 172.21.15.191:0/2832999345 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0f14f2d8-b271-48e4-a198-aa5279a01c06", "id": 3} v 0) v1 -- 0x7f9c20141130 con 0x7f9c2013b020 2024-03-27T21:39:28.458 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.471+0000 7f9c127fc640 1 -- 172.21.15.191:0/2832999345 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0f14f2d8-b271-48e4-a198-aa5279a01c06", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9c1401b6f0 con 0x7f9c2013b020 2024-03-27T21:39:28.458 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.471+0000 7f9c127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0f14f2d8-b271-48e4-a198-aa5279a01c06", "id": 3}] 2024-03-27T21:39:28.458 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.471+0000 7f9c127fc640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:28.458 INFO:teuthology.orchestra.run.smithi191.stdout:3 2024-03-27T21:39:28.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.472+0000 7f9c28192640 10 librados: watch_flush enter 2024-03-27T21:39:28.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.472+0000 7f9c28192640 10 librados: watch_flush exit 2024-03-27T21:39:28.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.472+0000 7f9c28192640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-27T21:39:28.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.472+0000 7f9c28192640 10 client.4141.objecter successfully canceled tick 2024-03-27T21:39:28.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.472+0000 7f9c28192640 10 monclient: shutdown 2024-03-27T21:39:28.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.472+0000 7f9c28192640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:28.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.472+0000 7f9c28192640 1 -- 172.21.15.191:0/2832999345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c2013b020 msgr2=0x7f9c201d6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.472+0000 7f9c28192640 1 --2- 172.21.15.191:0/2832999345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c2013b020 0x7f9c201d6ca0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9c1400c450 tx=0x7f9c1400c910 comp rx=0 tx=0).stop 2024-03-27T21:39:28.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.472+0000 7f9c28192640 1 -- 172.21.15.191:0/2832999345 shutdown_connections 2024-03-27T21:39:28.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.472+0000 7f9c28192640 1 --2- 172.21.15.191:0/2832999345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c2013b480 0x7f9c201d71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.472+0000 7f9c28192640 1 --2- 172.21.15.191:0/2832999345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c2013b020 0x7f9c201d6ca0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.472+0000 7f9c28192640 1 --2- 172.21.15.191:0/2832999345 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c200b40b0 0x7f9c200affa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.461 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.472+0000 7f9c28192640 1 -- 172.21.15.191:0/2832999345 >> 172.21.15.191:0/2832999345 conn(0x7f9c200608e0 msgr2=0x7f9c20140900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:28.461 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.472+0000 7f9c28192640 1 -- 172.21.15.191:0/2832999345 shutdown_connections 2024-03-27T21:39:28.461 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.472+0000 7f9c28192640 1 -- 172.21.15.191:0/2832999345 wait complete. 2024-03-27T21:39:28.461 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.472+0000 7f9c28192640 1 librados: shutdown 2024-03-27T21:39:28.461 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:28.474+0000 7f4298a0e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-27T21:39:28.469 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 8cdeb7ae-1c6e-4d0d-ba44-80248eff2fb6 4 2024-03-27T21:39:28.509 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:28.522+0000 7f4298a0e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-27T21:39:28.527 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:28.541+0000 7f6438632040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-27T21:39:28.564 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:28.577+0000 7f4298a0e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-27T21:39:28.567 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.580+0000 7f34a5eb4640 10 monclient: get_monmap_and_config 2024-03-27T21:39:28.567 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.580+0000 7f34a5eb4640 10 monclient: build_initial_monmap 2024-03-27T21:39:28.567 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.580+0000 7f34a5eb4640 10 monclient: monmap: 2024-03-27T21:39:28.567 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-27T21:39:28.567 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.567 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:39:28.582061+0000 2024-03-27T21:39:28.568 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:39:28.582061+0000 2024-03-27T21:39:28.568 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:28.568 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.568 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:28.568 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:28.568 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:28.568 INFO:teuthology.orchestra.run.smithi191.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:28.568 INFO:teuthology.orchestra.run.smithi191.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:28.568 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.577 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.581+0000 7f34a5eb4640 1 Processor -- start 2024-03-27T21:39:28.577 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.581+0000 7f34a5eb4640 1 -- start start 2024-03-27T21:39:28.577 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.581+0000 7f34a5eb4640 10 monclient: init 2024-03-27T21:39:28.577 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f34a5eb4640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:28.577 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f34a5eb4640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-03-27T21:39:28.577 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f34a5eb4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f34a0060970 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:28.577 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f34a5eb4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f34a00af2c0 0x7f34a00ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.578 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f34a5eb4640 10 monclient(hunting): picked mon.noname-a con 0x7f34a00af2c0 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:28.578 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f34a5eb4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f34a0144930 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:28.578 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f34a5eb4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a0060d50 con 0x7f34a0060970 2024-03-27T21:39:28.578 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f34a5eb4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a00aeda0 con 0x7f34a0144930 2024-03-27T21:39:28.578 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f34a5eb4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.578 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f34a5eb4640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34a00b0570 con 0x7f34a00af2c0 2024-03-27T21:39:28.578 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f34a5eb4640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:28.578 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f34a5eb4640 10 monclient(hunting): authenticate will time out at 785.865761s 2024-03-27T21:39:28.578 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349ffff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f34a0144930 0x7f34a0144dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:37446/0 (socket says 172.21.15.191:37446) 2024-03-27T21:39:28.578 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349ffff640 1 -- 172.21.15.191:0/3580281144 learned_addr learned my addr 172.21.15.191:0/3580281144 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-27T21:39:28.578 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349effd640 1 --2- 172.21.15.191:0/3580281144 >> v2:172.21.15.60:3300/0 conn(0x7f34a00af2c0 0x7f34a00ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.578 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349effd640 10 monclient(hunting): get_auth_request con 0x7f34a00af2c0 auth_method 0 2024-03-27T21:39:28.578 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.578 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349e7fc640 1 -- 172.21.15.191:0/3580281144 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800832191 0 0) 0x7f34a00aeda0 con 0x7f34a0144930 2024-03-27T21:39:28.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349e7fc640 10 monclient(hunting): _negotiate 2024-03-27T21:39:28.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349e7fc640 10 monclient(hunting): my global_id is 4144 2024-03-27T21:39:28.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349e7fc640 1 -- 172.21.15.191:0/3580281144 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3488003520 con 0x7f34a0144930 2024-03-27T21:39:28.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:28.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349e7fc640 1 -- 172.21.15.191:0/3580281144 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522974308 0 0) 0x7f3494001110 con 0x7f34a0060970 2024-03-27T21:39:28.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349e7fc640 10 monclient(hunting): _negotiate 2024-03-27T21:39:28.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349e7fc640 10 monclient(hunting): my global_id is 4140 2024-03-27T21:39:28.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349e7fc640 1 -- 172.21.15.191:0/3580281144 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34a00aeda0 con 0x7f34a0060970 2024-03-27T21:39:28.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349effd640 10 monclient(hunting): handle_auth_done global_id 4137 payload 274 2024-03-27T21:39:28.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349effd640 1 -- 172.21.15.191:0/3580281144 >> v1:172.21.15.191:6789/0 conn(0x7f34a0144930 legacy=0x7f34a0144dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349e7fc640 1 -- 172.21.15.191:0/3580281144 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2605648876 0 0) 0x7f3488003520 con 0x7f34a0144930 2024-03-27T21:39:28.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349effd640 1 -- 172.21.15.191:0/3580281144 >> v1:172.21.15.60:6789/0 conn(0x7f34a0060970 legacy=0x7f34a013e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349effd640 10 monclient: _finish_hunting 0 2024-03-27T21:39:28.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349effd640 1 monclient: found mon.noname-a 2024-03-27T21:39:28.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:28.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:28.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349effd640 1 -- 172.21.15.191:0/3580281144 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a00ae2b0 con 0x7f34a00af2c0 2024-03-27T21:39:28.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.582+0000 7f349e7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:39:28.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.583+0000 7f349effd640 1 --2- 172.21.15.191:0/3580281144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34a00af2c0 0x7f34a00ae6d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3490009720 tx=0x7f349002b6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5da52eba98e387b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:28.581 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.583+0000 7f349e7fc640 1 -- 172.21.15.191:0/3580281144 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34980019f0 con 0x7f34a00af2c0 2024-03-27T21:39:28.581 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.583+0000 7f349e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:28.581 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.583+0000 7f349e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:28.581 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.583+0000 7f349e7fc640 10 monclient: dump: 2024-03-27T21:39:28.581 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:28.581 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.581 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.581 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.581 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:28.581 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.581 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:28.581 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:28.581 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:28.581 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.583+0000 7f349e7fc640 10 monclient: _finish_auth 0 2024-03-27T21:39:28.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.583+0000 7f349e7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:39:28.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.583+0000 7f349e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:28.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.583+0000 7f349e7fc640 1 -- 172.21.15.191:0/3580281144 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349002e050 con 0x7f34a00af2c0 2024-03-27T21:39:28.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.583+0000 7f349e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:28.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.583+0000 7f34a5eb4640 5 monclient: authenticate success, global_id 4137 2024-03-27T21:39:28.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.583+0000 7f34a5eb4640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:28.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.583+0000 7f34a5eb4640 4 set_mon_vals no callback set 2024-03-27T21:39:28.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.583+0000 7f349e7fc640 1 -- 172.21.15.191:0/3580281144 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3490030710 con 0x7f34a00af2c0 2024-03-27T21:39:28.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.583+0000 7f349e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:28.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.583+0000 7f349e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:28.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.583+0000 7f349e7fc640 10 monclient: dump: 2024-03-27T21:39:28.583 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:28.583 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.583 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.583 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.583 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:28.583 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.583 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:28.583 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:28.583 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:28.583 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.583 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.584+0000 7f34a5eb4640 10 monclient: shutdown 2024-03-27T21:39:28.583 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.584+0000 7f34a5eb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:28.583 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.584+0000 7f34a5eb4640 1 -- 172.21.15.191:0/3580281144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34a00af2c0 msgr2=0x7f34a00ae6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.583 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.584+0000 7f34a5eb4640 1 --2- 172.21.15.191:0/3580281144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34a00af2c0 0x7f34a00ae6d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3490009720 tx=0x7f349002b6a0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.584+0000 7f34a5eb4640 1 -- 172.21.15.191:0/3580281144 shutdown_connections 2024-03-27T21:39:28.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.584+0000 7f34a5eb4640 1 --2- 172.21.15.191:0/3580281144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34a00af2c0 0x7f34a00ae6d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.584+0000 7f34a5eb4640 1 -- 172.21.15.191:0/3580281144 >> 172.21.15.191:0/3580281144 conn(0x7f34a00b3cd0 msgr2=0x7f34a00b40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:28.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.584+0000 7f34a5eb4640 1 -- 172.21.15.191:0/3580281144 shutdown_connections 2024-03-27T21:39:28.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.584+0000 7f34a5eb4640 1 -- 172.21.15.191:0/3580281144 wait complete. 2024-03-27T21:39:28.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.584+0000 7f34a5eb4640 10 monclient: build_initial_monmap 2024-03-27T21:39:28.584 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.584+0000 7f34a5eb4640 10 monclient: monmap: 2024-03-27T21:39:28.584 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-27T21:39:28.584 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:28.584 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-27T21:39:28.584 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-27T21:39:28.584 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:28.584 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-27T21:39:28.584 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:28.584 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:28.585 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:28.585 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 1 librados: starting msgr at 2024-03-27T21:39:28.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 1 librados: starting objecter 2024-03-27T21:39:28.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 1 Processor -- start 2024-03-27T21:39:28.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 1 -- start start 2024-03-27T21:39:28.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 1 librados: setting wanted keys 2024-03-27T21:39:28.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 1 librados: calling monclient init 2024-03-27T21:39:28.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 10 monclient: init 2024-03-27T21:39:28.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:28.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:39:28.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f34a0060970 0x7f34a0070d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 10 monclient(hunting): picked mon.noname-c con 0x7f34a0060970 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:28.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34a00af2c0 0x7f34a006e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 10 monclient(hunting): picked mon.noname-b con 0x7f34a00af2c0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:28.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34a0144930 0x7f34a01d6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 10 monclient(hunting): picked mon.noname-a con 0x7f34a0144930 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:28.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a01d68d0 con 0x7f34a0144930 2024-03-27T21:39:28.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a01d65a0 con 0x7f34a00af2c0 2024-03-27T21:39:28.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a0139da0 con 0x7f34a0060970 2024-03-27T21:39:28.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:28.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f34a5eb4640 10 monclient(hunting): authenticate will time out at 785.869512s 2024-03-27T21:39:28.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.585+0000 7f349effd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34a00af2c0 0x7f34a006e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349effd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34a00af2c0 0x7f34a006e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:52546/0 (socket says 172.21.15.191:52546) 2024-03-27T21:39:28.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349effd640 1 -- 172.21.15.191:0/678775731 learned_addr learned my addr 172.21.15.191:0/678775731 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-27T21:39:28.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349effd640 10 monclient(hunting): get_auth_request con 0x7f34a00af2c0 auth_method 0 2024-03-27T21:39:28.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349ffff640 1 --2- 172.21.15.191:0/678775731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34a0144930 0x7f34a01d6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349ffff640 10 monclient(hunting): get_auth_request con 0x7f34a0144930 auth_method 0 2024-03-27T21:39:28.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349f7fe640 1 --2- 172.21.15.191:0/678775731 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f34a0060970 0x7f34a0070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349f7fe640 10 monclient(hunting): get_auth_request con 0x7f34a0060970 auth_method 0 2024-03-27T21:39:28.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.587 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.588 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.588 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.588 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:28.588 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.588 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.588 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:28.588 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.588 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.588 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:28.588 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349ffff640 10 monclient(hunting): handle_auth_done global_id 4143 payload 824 2024-03-27T21:39:28.588 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349ffff640 1 -- 172.21.15.191:0/678775731 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f34a0060970 msgr2=0x7f34a0070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.588 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349ffff640 1 --2- 172.21.15.191:0/678775731 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f34a0060970 0x7f34a0070d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.588 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349ffff640 1 -- 172.21.15.191:0/678775731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34a00af2c0 msgr2=0x7f34a006e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.589 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349ffff640 1 --2- 172.21.15.191:0/678775731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34a00af2c0 0x7f34a006e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.589 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349ffff640 10 monclient: _finish_hunting 0 2024-03-27T21:39:28.589 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349ffff640 1 monclient: found mon.noname-a 2024-03-27T21:39:28.589 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:28.589 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:28.589 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349ffff640 1 -- 172.21.15.191:0/678775731 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3490009400 con 0x7f34a0144930 2024-03-27T21:39:28.589 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349effd640 1 --2- 172.21.15.191:0/678775731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34a00af2c0 0x7f34a006e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:28.589 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349ffff640 1 --2- 172.21.15.191:0/678775731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34a0144930 0x7f34a01d6d60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f349800c890 tx=0x7f349800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:28.589 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f3487fff640 10 client.?.objecter ms_handle_connect 0x7f34a0144930 2024-03-27T21:39:28.589 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f3487fff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:28.589 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f3487fff640 1 -- 172.21.15.191:0/678775731 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349800a5e0 con 0x7f34a0144930 2024-03-27T21:39:28.589 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f3487fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:28.589 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f3487fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:28.589 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f3487fff640 10 monclient: dump: 2024-03-27T21:39:28.590 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:28.590 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.590 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.590 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.590 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:28.590 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.590 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:28.590 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:28.590 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:28.590 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.590 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f3487fff640 10 monclient: _finish_auth 0 2024-03-27T21:39:28.590 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f3487fff640 10 monclient: _check_auth_tickets 2024-03-27T21:39:28.591 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f3487fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:28.591 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f3487fff640 1 -- 172.21.15.191:0/678775731 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349800a760 con 0x7f34a0144930 2024-03-27T21:39:28.591 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f3487fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:28.591 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f34a5eb4640 5 monclient: authenticate success, global_id 4143 2024-03-27T21:39:28.591 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f3487fff640 1 -- 172.21.15.191:0/678775731 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349800ab50 con 0x7f34a0144930 2024-03-27T21:39:28.591 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f34a5eb4640 10 monclient: _renew_subs 2024-03-27T21:39:28.591 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f34a5eb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:28.591 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f34a5eb4640 1 -- 172.21.15.191:0/678775731 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a006fef0 con 0x7f34a0144930 2024-03-27T21:39:28.591 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f3487fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:28.591 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f3487fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:28.591 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f3487fff640 10 monclient: dump: 2024-03-27T21:39:28.591 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:28.591 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.591 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.591 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.592 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:28.592 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.592 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:28.592 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:28.592 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:28.592 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.592 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f349dffb640 4 set_mon_vals no callback set 2024-03-27T21:39:28.592 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f34a5eb4640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:28.592 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f34a5eb4640 10 monclient: _renew_subs 2024-03-27T21:39:28.592 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f34a5eb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:28.592 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f34a5eb4640 1 -- 172.21.15.191:0/678775731 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a01d7480 con 0x7f34a0144930 2024-03-27T21:39:28.592 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f34a5eb4640 1 librados: init done 2024-03-27T21:39:28.592 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.586+0000 7f3487fff640 1 -- 172.21.15.191:0/678775731 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f34980070d0 con 0x7f34a0144930 2024-03-27T21:39:28.592 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.587+0000 7f3487fff640 1 -- 172.21.15.191:0/678775731 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f3498007890 con 0x7f34a0144930 2024-03-27T21:39:28.593 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.587+0000 7f3487fff640 10 client.4143.objecter ms_dispatch 0x7f34a0000b90 osd_map(6..6 src has 1..6) v4 2024-03-27T21:39:28.593 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.587+0000 7f3487fff640 3 client.4143.objecter handle_osd_map got epochs [6,6] > 0 2024-03-27T21:39:28.593 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.587+0000 7f3487fff640 3 client.4143.objecter handle_osd_map decoding full epoch 6 2024-03-27T21:39:28.593 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.587+0000 7f3487fff640 20 client.4143.objecter dump_active .. 0 homeless 2024-03-27T21:39:28.593 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.587+0000 7f34a5eb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:28.593 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.587+0000 7f34a5eb4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:28.593 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.587+0000 7f34a5eb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:28.593 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.587+0000 7f34a5eb4640 1 -- 172.21.15.191:0/678775731 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a01d6a40 con 0x7f34a0144930 2024-03-27T21:39:28.593 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.589+0000 7f3487fff640 1 -- 172.21.15.191:0/678775731 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f3498018070 con 0x7f34a0144930 2024-03-27T21:39:28.593 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.589+0000 7f3487fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:28.593 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.589+0000 7f3487fff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:28.600 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:28.614+0000 7f6438632040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-27T21:39:28.608 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:28.621+0000 7f4298a0e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-27T21:39:28.646 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.659+0000 7f34a5eb4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8cdeb7ae-1c6e-4d0d-ba44-80248eff2fb6", "id": 4}] 2024-03-27T21:39:28.646 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.659+0000 7f34a5eb4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8cdeb7ae-1c6e-4d0d-ba44-80248eff2fb6", "id": 4}] 2024-03-27T21:39:28.647 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.659+0000 7f34a5eb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:28.647 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.659+0000 7f34a5eb4640 1 -- 172.21.15.191:0/678775731 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8cdeb7ae-1c6e-4d0d-ba44-80248eff2fb6", "id": 4} v 0) v1 -- 0x7f34a0069470 con 0x7f34a0144930 2024-03-27T21:39:28.648 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:28.661+0000 7f6438632040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-27T21:39:28.649 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.662+0000 7f3487fff640 1 -- 172.21.15.191:0/678775731 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8cdeb7ae-1c6e-4d0d-ba44-80248eff2fb6", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3498007450 con 0x7f34a0144930 2024-03-27T21:39:28.649 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.662+0000 7f3487fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8cdeb7ae-1c6e-4d0d-ba44-80248eff2fb6", "id": 4}] 2024-03-27T21:39:28.649 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.662+0000 7f3487fff640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:28.649 INFO:teuthology.orchestra.run.smithi191.stdout:4 2024-03-27T21:39:28.651 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.663+0000 7f34a5eb4640 10 librados: watch_flush enter 2024-03-27T21:39:28.651 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.663+0000 7f34a5eb4640 10 librados: watch_flush exit 2024-03-27T21:39:28.651 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.663+0000 7f34a5eb4640 20 client.4143.objecter shutdown clearing up homeless session... 2024-03-27T21:39:28.651 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.663+0000 7f34a5eb4640 10 client.4143.objecter successfully canceled tick 2024-03-27T21:39:28.651 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.663+0000 7f34a5eb4640 10 monclient: shutdown 2024-03-27T21:39:28.651 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.663+0000 7f34a5eb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:28.651 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.663+0000 7f34a5eb4640 1 -- 172.21.15.191:0/678775731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34a0144930 msgr2=0x7f34a01d6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.651 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.663+0000 7f34a5eb4640 1 --2- 172.21.15.191:0/678775731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34a0144930 0x7f34a01d6d60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f349800c890 tx=0x7f349800cd50 comp rx=0 tx=0).stop 2024-03-27T21:39:28.651 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.663+0000 7f34a5eb4640 1 -- 172.21.15.191:0/678775731 shutdown_connections 2024-03-27T21:39:28.651 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.663+0000 7f34a5eb4640 1 --2- 172.21.15.191:0/678775731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34a0144930 0x7f34a01d6d60 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.651 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.663+0000 7f34a5eb4640 1 --2- 172.21.15.191:0/678775731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34a00af2c0 0x7f34a006e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.651 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.663+0000 7f34a5eb4640 1 --2- 172.21.15.191:0/678775731 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f34a0060970 0x7f34a0070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.652 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.663+0000 7f34a5eb4640 1 -- 172.21.15.191:0/678775731 >> 172.21.15.191:0/678775731 conn(0x7f34a00b3cd0 msgr2=0x7f34a0144d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:28.652 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.663+0000 7f34a5eb4640 1 -- 172.21.15.191:0/678775731 shutdown_connections 2024-03-27T21:39:28.652 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.663+0000 7f34a5eb4640 1 -- 172.21.15.191:0/678775731 wait complete. 2024-03-27T21:39:28.652 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.663+0000 7f34a5eb4640 1 librados: shutdown 2024-03-27T21:39:28.660 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 93748cf8-d3fe-46b8-950f-e01e28a5e02c 5 2024-03-27T21:39:28.702 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:28.716+0000 7f6438632040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-27T21:39:28.727 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:28.740+0000 7f4298a0e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-27T21:39:28.751 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:28.764+0000 7f6438632040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-27T21:39:28.799 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:28.812+0000 7f4298a0e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-27T21:39:28.800 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.812+0000 7fa87e674640 10 monclient: get_monmap_and_config 2024-03-27T21:39:28.800 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.812+0000 7fa87e674640 10 monclient: build_initial_monmap 2024-03-27T21:39:28.800 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.812+0000 7fa87e674640 10 monclient: monmap: 2024-03-27T21:39:28.800 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-27T21:39:28.800 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.800 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:39:28.813927+0000 2024-03-27T21:39:28.800 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:39:28.813927+0000 2024-03-27T21:39:28.800 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:28.800 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.800 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:28.800 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:28.801 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:28.801 INFO:teuthology.orchestra.run.smithi191.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:28.801 INFO:teuthology.orchestra.run.smithi191.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:28.801 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.801 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.813+0000 7fa87e674640 1 Processor -- start 2024-03-27T21:39:28.801 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.813+0000 7fa87e674640 1 -- start start 2024-03-27T21:39:28.801 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.813+0000 7fa87e674640 10 monclient: init 2024-03-27T21:39:28.810 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.813+0000 7fa87e674640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:28.810 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.813+0000 7fa87e674640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-03-27T21:39:28.811 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.813+0000 7fa87e674640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa87806e760 0x7fa87806eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.811 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.813+0000 7fa87e674640 10 monclient(hunting): picked mon.noname-b con 0x7fa87806e760 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:28.811 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.813+0000 7fa87e674640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa878137d30 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:28.811 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.813+0000 7fa87e674640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa87813e840 0x7fa878138170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.811 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.813+0000 7fa87e674640 10 monclient(hunting): picked mon.noname-c con 0x7fa87813e840 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:28.811 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.813+0000 7fa87e674640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa878071690 con 0x7fa878137d30 2024-03-27T21:39:28.811 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.813+0000 7fa87e674640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.811 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.813+0000 7fa87e674640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8780afde0 con 0x7fa87813e840 2024-03-27T21:39:28.811 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.813+0000 7fa87e674640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.811 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.813+0000 7fa87e674640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8780709b0 con 0x7fa87806e760 2024-03-27T21:39:28.811 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.813+0000 7fa87e674640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:28.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.813+0000 7fa87e674640 10 monclient(hunting): authenticate will time out at 786.097608s 2024-03-27T21:39:28.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa87cc11640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa87813e840 0x7fa878138170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa87cc11640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa87813e840 0x7fa878138170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:52552/0 (socket says 172.21.15.191:52552) 2024-03-27T21:39:28.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa87cc11640 1 -- 172.21.15.191:0/1314550359 learned_addr learned my addr 172.21.15.191:0/1314550359 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-27T21:39:28.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa87cc11640 10 monclient(hunting): get_auth_request con 0x7fa87813e840 auth_method 0 2024-03-27T21:39:28.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa87cc11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa87cc11640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa87cc11640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa87cc11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa87cc11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa87cc11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:28.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa877fff640 1 --2- 172.21.15.191:0/1314550359 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa87806e760 0x7fa87806eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa877fff640 10 monclient(hunting): get_auth_request con 0x7fa87806e760 auth_method 0 2024-03-27T21:39:28.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa877fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa877fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa877fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa87cc11640 10 monclient(hunting): handle_auth_done global_id 4150 payload 274 2024-03-27T21:39:28.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa87cc11640 1 -- 172.21.15.191:0/1314550359 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa87806e760 msgr2=0x7fa87806eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:28.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa87cc11640 1 --2- 172.21.15.191:0/1314550359 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa87806e760 0x7fa87806eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa87cc11640 1 -- 172.21.15.191:0/1314550359 >> v1:172.21.15.191:6789/0 conn(0x7fa878137d30 legacy=0x7fa87813e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa87cc11640 10 monclient: _finish_hunting 0 2024-03-27T21:39:28.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa87cc11640 1 monclient: found mon.noname-c 2024-03-27T21:39:28.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa87cc11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:28.814 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa87cc11640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:39:28.814 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.814+0000 7fa87cc11640 1 -- 172.21.15.191:0/1314550359 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa878070c90 con 0x7fa87813e840 2024-03-27T21:39:28.814 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa87cc11640 1 --2- 172.21.15.191:0/1314550359 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa87813e840 0x7fa878138170 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa870009e50 tx=0x7fa87002bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a404415316ab106c server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:28.814 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa876ffd640 1 -- 172.21.15.191:0/1314550359 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87002ca70 con 0x7fa87813e840 2024-03-27T21:39:28.814 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa876ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:28.814 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa876ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:28.814 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa876ffd640 10 monclient: dump: 2024-03-27T21:39:28.814 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:28.814 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.814 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.814 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.814 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:28.815 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.815 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:28.815 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:28.815 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:28.815 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.815 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa876ffd640 10 monclient: _finish_auth 0 2024-03-27T21:39:28.815 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa876ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:39:28.815 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa876ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:28.816 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa876ffd640 1 -- 172.21.15.191:0/1314550359 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa87002cbd0 con 0x7fa87813e840 2024-03-27T21:39:28.816 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa87e674640 5 monclient: authenticate success, global_id 4150 2024-03-27T21:39:28.816 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa87e674640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:28.816 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa876ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:28.816 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa876ffd640 1 -- 172.21.15.191:0/1314550359 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8700343f0 con 0x7fa87813e840 2024-03-27T21:39:28.816 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa876ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:28.816 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa876ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:28.816 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa876ffd640 10 monclient: dump: 2024-03-27T21:39:28.816 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:28.816 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.816 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.817 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.817 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:28.817 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.817 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:28.817 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:28.817 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:28.817 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.817 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa87e674640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:28.817 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa87e674640 4 set_mon_vals no callback set 2024-03-27T21:39:28.817 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa87e674640 10 monclient: shutdown 2024-03-27T21:39:28.817 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa87e674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:28.817 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa87e674640 1 -- 172.21.15.191:0/1314550359 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa87813e840 msgr2=0x7fa878138170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa87e674640 1 --2- 172.21.15.191:0/1314550359 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa87813e840 0x7fa878138170 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa870009e50 tx=0x7fa87002bed0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa87e674640 1 -- 172.21.15.191:0/1314550359 shutdown_connections 2024-03-27T21:39:28.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa87e674640 1 --2- 172.21.15.191:0/1314550359 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa87813e840 0x7fa878138170 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa87e674640 1 --2- 172.21.15.191:0/1314550359 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa87806e760 0x7fa87806eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa87e674640 1 -- 172.21.15.191:0/1314550359 >> 172.21.15.191:0/1314550359 conn(0x7fa8780b3d60 msgr2=0x7fa8780b4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:28.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa87e674640 1 -- 172.21.15.191:0/1314550359 shutdown_connections 2024-03-27T21:39:28.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.815+0000 7fa87e674640 1 -- 172.21.15.191:0/1314550359 wait complete. 2024-03-27T21:39:28.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.816+0000 7fa87e674640 10 monclient: build_initial_monmap 2024-03-27T21:39:28.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.816+0000 7fa87e674640 10 monclient: monmap: 2024-03-27T21:39:28.818 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-27T21:39:28.819 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:28.819 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-27T21:39:28.819 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-27T21:39:28.819 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:28.819 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-27T21:39:28.819 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:28.819 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:28.819 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:28.819 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.820 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.816+0000 7fa87e674640 1 librados: starting msgr at 2024-03-27T21:39:28.820 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.816+0000 7fa87e674640 1 librados: starting objecter 2024-03-27T21:39:28.820 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.816+0000 7fa87e674640 1 Processor -- start 2024-03-27T21:39:28.820 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87e674640 1 -- start start 2024-03-27T21:39:28.820 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87e674640 1 librados: setting wanted keys 2024-03-27T21:39:28.820 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87e674640 1 librados: calling monclient init 2024-03-27T21:39:28.820 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87e674640 10 monclient: init 2024-03-27T21:39:28.820 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87e674640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:28.820 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87e674640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:39:28.820 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87e674640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa87806e760 0x7fa87812f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.820 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87e674640 10 monclient(hunting): picked mon.noname-b con 0x7fa87806e760 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:28.820 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87e674640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa878137d30 0x7fa87813b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.821 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87e674640 10 monclient(hunting): picked mon.noname-a con 0x7fa878137d30 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:28.821 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87e674640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa878134b10 0x7fa878138740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:28.821 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87e674640 10 monclient(hunting): picked mon.noname-c con 0x7fa878134b10 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:28.821 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87e674640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.821 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87e674640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa878137bd0 con 0x7fa878137d30 2024-03-27T21:39:28.821 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87e674640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.821 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87e674640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87813a6e0 con 0x7fa87806e760 2024-03-27T21:39:28.821 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87e674640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:28.821 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87e674640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa878134160 con 0x7fa878134b10 2024-03-27T21:39:28.821 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87e674640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:28.821 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87e674640 10 monclient(hunting): authenticate will time out at 786.101253s 2024-03-27T21:39:28.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa877fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa87806e760 0x7fa87812f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa877fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa87806e760 0x7fa87812f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:52568/0 (socket says 172.21.15.191:52568) 2024-03-27T21:39:28.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa877fff640 1 -- 172.21.15.191:0/43292550 learned_addr learned my addr 172.21.15.191:0/43292550 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-27T21:39:28.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa877fff640 10 monclient(hunting): get_auth_request con 0x7fa87806e760 auth_method 0 2024-03-27T21:39:28.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa877fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa877fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa877fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87cc11640 1 --2- 172.21.15.191:0/43292550 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa878134b10 0x7fa878138740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa877fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa877fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa8777fe640 1 --2- 172.21.15.191:0/43292550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa878137d30 0x7fa87813b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:28.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa877fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:28.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87cc11640 10 monclient(hunting): get_auth_request con 0x7fa878134b10 auth_method 0 2024-03-27T21:39:28.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87cc11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87cc11640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87cc11640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa8777fe640 10 monclient(hunting): get_auth_request con 0x7fa878137d30 auth_method 0 2024-03-27T21:39:28.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa8777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:28.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa8777fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:28.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa8777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:28.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87cc11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87cc11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa87cc11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:28.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa8777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:28.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa8777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:28.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.817+0000 7fa8777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:28.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa877fff640 10 monclient(hunting): handle_auth_done global_id 4153 payload 824 2024-03-27T21:39:28.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa877fff640 1 -- 172.21.15.191:0/43292550 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa878134b10 msgr2=0x7fa878138740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa877fff640 1 --2- 172.21.15.191:0/43292550 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa878134b10 0x7fa878138740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa877fff640 1 -- 172.21.15.191:0/43292550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa878137d30 msgr2=0x7fa87813b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa877fff640 1 --2- 172.21.15.191:0/43292550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa878137d30 0x7fa87813b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa877fff640 10 monclient: _finish_hunting 0 2024-03-27T21:39:28.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa877fff640 1 monclient: found mon.noname-b 2024-03-27T21:39:28.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa877fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:28.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa877fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:39:28.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa877fff640 1 -- 172.21.15.191:0/43292550 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa870009b30 con 0x7fa87806e760 2024-03-27T21:39:28.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa877fff640 1 --2- 172.21.15.191:0/43292550 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa87806e760 0x7fa87812f440 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa86c00d800 tx=0x7fa86c00dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:28.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa874ff9640 10 client.?.objecter ms_handle_connect 0x7fa87806e760 2024-03-27T21:39:28.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa874ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:28.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa874ff9640 1 -- 172.21.15.191:0/43292550 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c00b6c0 con 0x7fa87806e760 2024-03-27T21:39:28.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa874ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:28.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa874ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:28.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa874ff9640 10 monclient: dump: 2024-03-27T21:39:28.825 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:28.825 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.825 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.825 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.825 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:28.826 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.826 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:28.826 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:28.826 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:28.826 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa874ff9640 10 monclient: _finish_auth 0 2024-03-27T21:39:28.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa874ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:39:28.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa874ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:28.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa87e674640 5 monclient: authenticate success, global_id 4153 2024-03-27T21:39:28.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa874ff9640 1 -- 172.21.15.191:0/43292550 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa86c00b820 con 0x7fa87806e760 2024-03-27T21:39:28.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa874ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:28.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa87e674640 10 monclient: _renew_subs 2024-03-27T21:39:28.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa874ff9640 1 -- 172.21.15.191:0/43292550 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c00bbb0 con 0x7fa87806e760 2024-03-27T21:39:28.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa87e674640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:28.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa87e674640 1 -- 172.21.15.191:0/43292550 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa878071cd0 con 0x7fa87806e760 2024-03-27T21:39:28.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa874ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:28.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa874ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:28.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa874ff9640 10 monclient: dump: 2024-03-27T21:39:28.827 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:28.827 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:28.827 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.827 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:28.827 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:28.828 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:28.828 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:28.828 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:28.828 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:28.828 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:28.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa8767fc640 4 set_mon_vals no callback set 2024-03-27T21:39:28.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa87e674640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:28.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa87e674640 10 monclient: _renew_subs 2024-03-27T21:39:28.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa87e674640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:28.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa87e674640 1 -- 172.21.15.191:0/43292550 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa878139790 con 0x7fa87806e760 2024-03-27T21:39:28.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa87e674640 1 librados: init done 2024-03-27T21:39:28.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa874ff9640 1 -- 172.21.15.191:0/43292550 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa86c01e760 con 0x7fa87806e760 2024-03-27T21:39:28.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa874ff9640 1 -- 172.21.15.191:0/43292550 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fa86c014af0 con 0x7fa87806e760 2024-03-27T21:39:28.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa874ff9640 10 client.4153.objecter ms_dispatch 0x7fa878000b90 osd_map(7..7 src has 1..7) v4 2024-03-27T21:39:28.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa874ff9640 3 client.4153.objecter handle_osd_map got epochs [7,7] > 0 2024-03-27T21:39:28.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa874ff9640 3 client.4153.objecter handle_osd_map decoding full epoch 7 2024-03-27T21:39:28.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.818+0000 7fa874ff9640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-27T21:39:28.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.819+0000 7fa87e674640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:28.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.819+0000 7fa87e674640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:28.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.819+0000 7fa87e674640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:28.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.819+0000 7fa87e674640 1 -- 172.21.15.191:0/43292550 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa878073f90 con 0x7fa87806e760 2024-03-27T21:39:28.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.821+0000 7fa874ff9640 1 -- 172.21.15.191:0/43292550 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fa86c00bd10 con 0x7fa87806e760 2024-03-27T21:39:28.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.821+0000 7fa874ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:28.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.821+0000 7fa874ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:28.846 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:28.860+0000 7f4298a0e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-27T21:39:28.878 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.891+0000 7fa87e674640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "93748cf8-d3fe-46b8-950f-e01e28a5e02c", "id": 5}] 2024-03-27T21:39:28.878 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.891+0000 7fa87e674640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "93748cf8-d3fe-46b8-950f-e01e28a5e02c", "id": 5}] 2024-03-27T21:39:28.878 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.891+0000 7fa87e674640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:28.878 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.891+0000 7fa87e674640 1 -- 172.21.15.191:0/43292550 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "93748cf8-d3fe-46b8-950f-e01e28a5e02c", "id": 5} v 0) v1 -- 0x7fa878069470 con 0x7fa87806e760 2024-03-27T21:39:28.882 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.895+0000 7fa874ff9640 1 -- 172.21.15.191:0/43292550 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "93748cf8-d3fe-46b8-950f-e01e28a5e02c", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa86c01eb50 con 0x7fa87806e760 2024-03-27T21:39:28.882 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.895+0000 7fa874ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "93748cf8-d3fe-46b8-950f-e01e28a5e02c", "id": 5}] 2024-03-27T21:39:28.882 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.895+0000 7fa874ff9640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:28.882 INFO:teuthology.orchestra.run.smithi191.stdout:5 2024-03-27T21:39:28.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.896+0000 7fa87e674640 10 librados: watch_flush enter 2024-03-27T21:39:28.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.896+0000 7fa87e674640 10 librados: watch_flush exit 2024-03-27T21:39:28.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.896+0000 7fa87e674640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-27T21:39:28.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.896+0000 7fa87e674640 10 client.4153.objecter successfully canceled tick 2024-03-27T21:39:28.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.896+0000 7fa87e674640 10 monclient: shutdown 2024-03-27T21:39:28.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.896+0000 7fa87e674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:28.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.896+0000 7fa87e674640 1 -- 172.21.15.191:0/43292550 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa87806e760 msgr2=0x7fa87812f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:28.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.896+0000 7fa87e674640 1 --2- 172.21.15.191:0/43292550 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa87806e760 0x7fa87812f440 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa86c00d800 tx=0x7fa86c00dcc0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.896+0000 7fa87e674640 1 -- 172.21.15.191:0/43292550 shutdown_connections 2024-03-27T21:39:28.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.896+0000 7fa87e674640 1 --2- 172.21.15.191:0/43292550 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa878134b10 0x7fa878138740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.896+0000 7fa87e674640 1 --2- 172.21.15.191:0/43292550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa878137d30 0x7fa87813b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.896+0000 7fa87e674640 1 --2- 172.21.15.191:0/43292550 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa87806e760 0x7fa87812f440 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:28.885 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.896+0000 7fa87e674640 1 -- 172.21.15.191:0/43292550 >> 172.21.15.191:0/43292550 conn(0x7fa8780b3d60 msgr2=0x7fa8780b4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:28.885 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.896+0000 7fa87e674640 1 -- 172.21.15.191:0/43292550 shutdown_connections 2024-03-27T21:39:28.885 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.896+0000 7fa87e674640 1 -- 172.21.15.191:0/43292550 wait complete. 2024-03-27T21:39:28.885 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:28.896+0000 7fa87e674640 1 librados: shutdown 2024-03-27T21:39:28.894 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 1ad21434-cbc5-4de9-8d63-76f894f7e736 6 2024-03-27T21:39:28.902 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:28.915+0000 7f4298a0e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-27T21:39:28.954 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:28.967+0000 7f4298a0e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-27T21:39:29.000 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.013+0000 7fe571f8b640 10 monclient: get_monmap_and_config 2024-03-27T21:39:29.000 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.013+0000 7fe571f8b640 10 monclient: build_initial_monmap 2024-03-27T21:39:29.000 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.013+0000 7fe571f8b640 10 monclient: monmap: 2024-03-27T21:39:29.000 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-27T21:39:29.000 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:29.000 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:39:29.014357+0000 2024-03-27T21:39:29.000 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:39:29.014357+0000 2024-03-27T21:39:29.000 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:29.001 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:29.001 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:29.001 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:29.001 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:29.001 INFO:teuthology.orchestra.run.smithi191.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:29.001 INFO:teuthology.orchestra.run.smithi191.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:29.001 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:29.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.013+0000 7fe571f8b640 1 Processor -- start 2024-03-27T21:39:29.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe571f8b640 1 -- start start 2024-03-27T21:39:29.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe571f8b640 10 monclient: init 2024-03-27T21:39:29.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe571f8b640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:29.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe571f8b640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-03-27T21:39:29.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe571f8b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe56c073640 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:29.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe571f8b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe56c070560 0x7fe56c1427c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:29.011 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe571f8b640 10 monclient(hunting): picked mon.noname-b con 0x7fe56c070560 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:29.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe571f8b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe56c13cbd0 0x7fe56c142d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:29.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe571f8b640 10 monclient(hunting): picked mon.noname-c con 0x7fe56c13cbd0 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:29.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe571f8b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe56c071730 con 0x7fe56c073640 2024-03-27T21:39:29.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe571f8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:29.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe571f8b640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe56c06fea0 con 0x7fe56c13cbd0 2024-03-27T21:39:29.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe571f8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:29.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe571f8b640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c0709f0 con 0x7fe56c070560 2024-03-27T21:39:29.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe571f8b640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:29.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe571f8b640 10 monclient(hunting): authenticate will time out at 786.298274s 2024-03-27T21:39:29.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe56bfff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe56c13cbd0 0x7fe56c142d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:29.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe56bfff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe56c13cbd0 0x7fe56c142d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:52584/0 (socket says 172.21.15.191:52584) 2024-03-27T21:39:29.012 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe56bfff640 1 -- 172.21.15.191:0/2367660668 learned_addr learned my addr 172.21.15.191:0/2367660668 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-27T21:39:29.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe56bfff640 10 monclient(hunting): get_auth_request con 0x7fe56c13cbd0 auth_method 0 2024-03-27T21:39:29.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe56bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:29.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe56bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:29.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe56bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:29.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe563fff640 1 --2- 172.21.15.191:0/2367660668 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe56c070560 0x7fe56c1427c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:29.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe56bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:29.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe56bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:29.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe56bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:29.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe563fff640 10 monclient(hunting): get_auth_request con 0x7fe56c070560 auth_method 0 2024-03-27T21:39:29.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe563fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:29.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe563fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:29.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.014+0000 7fe563fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:29.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.015+0000 7fe563fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:29.013 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.015+0000 7fe563fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:29.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.015+0000 7fe563fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:29.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.015+0000 7fe56bfff640 10 monclient(hunting): handle_auth_done global_id 4156 payload 274 2024-03-27T21:39:29.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.015+0000 7fe56bfff640 1 -- 172.21.15.191:0/2367660668 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe56c070560 msgr2=0x7fe56c1427c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:29.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.015+0000 7fe56bfff640 1 --2- 172.21.15.191:0/2367660668 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe56c070560 0x7fe56c1427c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.015+0000 7fe56bfff640 1 -- 172.21.15.191:0/2367660668 >> v1:172.21.15.60:6789/0 conn(0x7fe56c073640 legacy=0x7fe56c13c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:29.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.015+0000 7fe56bfff640 10 monclient: _finish_hunting 0 2024-03-27T21:39:29.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.015+0000 7fe56bfff640 1 monclient: found mon.noname-c 2024-03-27T21:39:29.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.015+0000 7fe56bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:29.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.015+0000 7fe56bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:39:29.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.015+0000 7fe56bfff640 1 -- 172.21.15.191:0/2367660668 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56c143240 con 0x7fe56c13cbd0 2024-03-27T21:39:29.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.016+0000 7fe56bfff640 1 --2- 172.21.15.191:0/2367660668 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe56c13cbd0 0x7fe56c142d00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe564009e50 tx=0x7fe56402bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4679ccaf32f40679 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:29.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.016+0000 7fe56affd640 1 -- 172.21.15.191:0/2367660668 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56402ca70 con 0x7fe56c13cbd0 2024-03-27T21:39:29.014 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.016+0000 7fe56affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:29.015 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.016+0000 7fe56affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:29.015 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.016+0000 7fe56affd640 10 monclient: dump: 2024-03-27T21:39:29.015 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:29.015 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:29.015 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.015 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.015 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:29.015 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:29.015 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:29.015 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:29.015 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:29.015 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:29.015 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.016+0000 7fe56affd640 10 monclient: _finish_auth 0 2024-03-27T21:39:29.015 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.016+0000 7fe56affd640 10 monclient: _check_auth_tickets 2024-03-27T21:39:29.015 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.016+0000 7fe56affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:29.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.016+0000 7fe571f8b640 5 monclient: authenticate success, global_id 4156 2024-03-27T21:39:29.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.016+0000 7fe571f8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:29.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.016+0000 7fe56affd640 1 -- 172.21.15.191:0/2367660668 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56402cbd0 con 0x7fe56c13cbd0 2024-03-27T21:39:29.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.016+0000 7fe56affd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:29.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.016+0000 7fe571f8b640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:29.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.016+0000 7fe571f8b640 4 set_mon_vals no callback set 2024-03-27T21:39:29.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.016+0000 7fe56affd640 1 -- 172.21.15.191:0/2367660668 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5640343f0 con 0x7fe56c13cbd0 2024-03-27T21:39:29.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.016+0000 7fe56affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:29.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.016+0000 7fe56affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:29.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.016+0000 7fe56affd640 10 monclient: dump: 2024-03-27T21:39:29.016 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:29.016 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:29.016 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.016 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.016 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:29.016 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:29.016 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:29.017 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:29.017 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:29.017 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:29.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.017+0000 7fe571f8b640 10 monclient: shutdown 2024-03-27T21:39:29.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.017+0000 7fe571f8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:29.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.017+0000 7fe571f8b640 1 -- 172.21.15.191:0/2367660668 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe56c13cbd0 msgr2=0x7fe56c142d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:29.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.017+0000 7fe571f8b640 1 --2- 172.21.15.191:0/2367660668 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe56c13cbd0 0x7fe56c142d00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe564009e50 tx=0x7fe56402bed0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.017+0000 7fe571f8b640 1 -- 172.21.15.191:0/2367660668 shutdown_connections 2024-03-27T21:39:29.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.017+0000 7fe571f8b640 1 --2- 172.21.15.191:0/2367660668 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe56c13cbd0 0x7fe56c142d00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.017+0000 7fe571f8b640 1 --2- 172.21.15.191:0/2367660668 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe56c070560 0x7fe56c1427c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.017+0000 7fe571f8b640 1 -- 172.21.15.191:0/2367660668 >> 172.21.15.191:0/2367660668 conn(0x7fe56c0b3e70 msgr2=0x7fe56c069ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:29.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.017+0000 7fe571f8b640 1 -- 172.21.15.191:0/2367660668 shutdown_connections 2024-03-27T21:39:29.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.017+0000 7fe571f8b640 1 -- 172.21.15.191:0/2367660668 wait complete. 2024-03-27T21:39:29.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.017+0000 7fe571f8b640 10 monclient: build_initial_monmap 2024-03-27T21:39:29.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.017+0000 7fe571f8b640 10 monclient: monmap: 2024-03-27T21:39:29.018 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-27T21:39:29.018 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:29.018 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-27T21:39:29.018 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-27T21:39:29.018 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:29.018 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-27T21:39:29.018 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:29.018 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:29.018 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:29.018 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:29.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 1 librados: starting msgr at 2024-03-27T21:39:29.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 1 librados: starting objecter 2024-03-27T21:39:29.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 1 Processor -- start 2024-03-27T21:39:29.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 1 -- start start 2024-03-27T21:39:29.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 1 librados: setting wanted keys 2024-03-27T21:39:29.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 1 librados: calling monclient init 2024-03-27T21:39:29.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 10 monclient: init 2024-03-27T21:39:29.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:29.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:39:29.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe56c073640 0x7fe56c142c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:29.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 10 monclient(hunting): picked mon.noname-a con 0x7fe56c073640 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:29.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe56c13cbd0 0x7fe56c142190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:29.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 10 monclient(hunting): picked mon.noname-c con 0x7fe56c13cbd0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:29.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe56c13ef50 0x7fe56c1d3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:29.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 10 monclient(hunting): picked mon.noname-b con 0x7fe56c13ef50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:29.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:29.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c13fbf0 con 0x7fe56c073640 2024-03-27T21:39:29.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:29.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c13f400 con 0x7fe56c13ef50 2024-03-27T21:39:29.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:29.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c13f6d0 con 0x7fe56c13cbd0 2024-03-27T21:39:29.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:29.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe571f8b640 10 monclient(hunting): authenticate will time out at 786.302369s 2024-03-27T21:39:29.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe56bfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe56c13ef50 0x7fe56c1d3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:29.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe56bfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe56c13ef50 0x7fe56c1d3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:52592/0 (socket says 172.21.15.191:52592) 2024-03-27T21:39:29.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe56bfff640 1 -- 172.21.15.191:0/674681575 learned_addr learned my addr 172.21.15.191:0/674681575 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-27T21:39:29.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe56bfff640 10 monclient(hunting): get_auth_request con 0x7fe56c13ef50 auth_method 0 2024-03-27T21:39:29.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe56bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:29.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe56bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:29.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe56bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:29.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe56b7fe640 1 --2- 172.21.15.191:0/674681575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe56c073640 0x7fe56c142c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:29.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe56b7fe640 10 monclient(hunting): get_auth_request con 0x7fe56c073640 auth_method 0 2024-03-27T21:39:29.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe56b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:29.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe56b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:29.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe56b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:29.021 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe56bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:29.021 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe56bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:29.021 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.018+0000 7fe563fff640 1 --2- 172.21.15.191:0/674681575 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe56c13cbd0 0x7fe56c142190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:29.021 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe56bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:29.021 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe563fff640 10 monclient(hunting): get_auth_request con 0x7fe56c13cbd0 auth_method 0 2024-03-27T21:39:29.021 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe563fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:29.021 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe563fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:29.021 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe563fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:29.021 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe56b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:29.021 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe56b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:29.021 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe56b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:29.021 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe563fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:29.021 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe563fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:29.021 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe563fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:29.021 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe56bfff640 10 monclient(hunting): handle_auth_done global_id 4159 payload 824 2024-03-27T21:39:29.021 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe56bfff640 1 -- 172.21.15.191:0/674681575 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe56c13cbd0 msgr2=0x7fe56c142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:29.022 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe56bfff640 1 --2- 172.21.15.191:0/674681575 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe56c13cbd0 0x7fe56c142190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.022 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe56bfff640 1 -- 172.21.15.191:0/674681575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe56c073640 msgr2=0x7fe56c142c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:29.022 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe56bfff640 1 --2- 172.21.15.191:0/674681575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe56c073640 0x7fe56c142c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.022 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe56bfff640 10 monclient: _finish_hunting 0 2024-03-27T21:39:29.022 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe56bfff640 1 monclient: found mon.noname-b 2024-03-27T21:39:29.022 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe56bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:29.022 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe56bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:39:29.022 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe56bfff640 1 -- 172.21.15.191:0/674681575 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe564009b30 con 0x7fe56c13ef50 2024-03-27T21:39:29.022 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe56bfff640 1 --2- 172.21.15.191:0/674681575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe56c13ef50 0x7fe56c1d3e00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe564009e50 tx=0x7fe564004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:29.022 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe568ff9640 10 client.?.objecter ms_handle_connect 0x7fe56c13ef50 2024-03-27T21:39:29.022 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe568ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:29.022 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe568ff9640 1 -- 172.21.15.191:0/674681575 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe564009510 con 0x7fe56c13ef50 2024-03-27T21:39:29.022 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe568ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:29.022 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe568ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:29.023 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe568ff9640 10 monclient: dump: 2024-03-27T21:39:29.023 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:29.023 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:29.023 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.023 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.023 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:29.023 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:29.023 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:29.023 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:29.024 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:29.024 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:29.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe568ff9640 10 monclient: _finish_auth 0 2024-03-27T21:39:29.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe568ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:39:29.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe568ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:29.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe571f8b640 5 monclient: authenticate success, global_id 4159 2024-03-27T21:39:29.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe568ff9640 1 -- 172.21.15.191:0/674681575 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56403e450 con 0x7fe56c13ef50 2024-03-27T21:39:29.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe568ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:29.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe568ff9640 1 -- 172.21.15.191:0/674681575 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56403e800 con 0x7fe56c13ef50 2024-03-27T21:39:29.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe571f8b640 10 monclient: _renew_subs 2024-03-27T21:39:29.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe571f8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:29.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe56a7fc640 4 set_mon_vals no callback set 2024-03-27T21:39:29.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe571f8b640 1 -- 172.21.15.191:0/674681575 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe56c1d4340 con 0x7fe56c13ef50 2024-03-27T21:39:29.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe568ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:29.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe568ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:29.025 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe568ff9640 10 monclient: dump: 2024-03-27T21:39:29.025 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:29.025 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:29.025 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.025 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.025 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:29.025 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:29.025 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:29.025 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:29.025 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:29.025 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:29.025 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe571f8b640 10 client.4159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:29.025 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe571f8b640 10 monclient: _renew_subs 2024-03-27T21:39:29.025 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe571f8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:29.025 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe571f8b640 1 -- 172.21.15.191:0/674681575 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe56c1d4850 con 0x7fe56c13ef50 2024-03-27T21:39:29.025 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe571f8b640 1 librados: init done 2024-03-27T21:39:29.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe568ff9640 1 -- 172.21.15.191:0/674681575 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe56403ec60 con 0x7fe56c13ef50 2024-03-27T21:39:29.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe568ff9640 1 -- 172.21.15.191:0/674681575 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fe564048a40 con 0x7fe56c13ef50 2024-03-27T21:39:29.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe568ff9640 10 client.4159.objecter ms_dispatch 0x7fe56c000b90 osd_map(8..8 src has 1..8) v4 2024-03-27T21:39:29.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe568ff9640 3 client.4159.objecter handle_osd_map got epochs [8,8] > 0 2024-03-27T21:39:29.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe568ff9640 3 client.4159.objecter handle_osd_map decoding full epoch 8 2024-03-27T21:39:29.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.019+0000 7fe568ff9640 20 client.4159.objecter dump_active .. 0 homeless 2024-03-27T21:39:29.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.020+0000 7fe571f8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:29.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.020+0000 7fe571f8b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:29.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.020+0000 7fe571f8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:29.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.020+0000 7fe571f8b640 1 -- 172.21.15.191:0/674681575 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe56c069470 con 0x7fe56c13ef50 2024-03-27T21:39:29.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.022+0000 7fe568ff9640 1 -- 172.21.15.191:0/674681575 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fe564043020 con 0x7fe56c13ef50 2024-03-27T21:39:29.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.022+0000 7fe568ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:29.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.022+0000 7fe568ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:29.079 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.092+0000 7fe571f8b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1ad21434-cbc5-4de9-8d63-76f894f7e736", "id": 6}] 2024-03-27T21:39:29.079 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.092+0000 7fe571f8b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1ad21434-cbc5-4de9-8d63-76f894f7e736", "id": 6}] 2024-03-27T21:39:29.079 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.092+0000 7fe571f8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:29.079 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.092+0000 7fe571f8b640 1 -- 172.21.15.191:0/674681575 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1ad21434-cbc5-4de9-8d63-76f894f7e736", "id": 6} v 0) v1 -- 0x7fe56c13f890 con 0x7fe56c13ef50 2024-03-27T21:39:29.083 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.096+0000 7fe568ff9640 1 -- 172.21.15.191:0/674681575 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1ad21434-cbc5-4de9-8d63-76f894f7e736", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe564048400 con 0x7fe56c13ef50 2024-03-27T21:39:29.083 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.096+0000 7fe568ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1ad21434-cbc5-4de9-8d63-76f894f7e736", "id": 6}] 2024-03-27T21:39:29.083 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.096+0000 7fe568ff9640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:29.083 INFO:teuthology.orchestra.run.smithi191.stdout:6 2024-03-27T21:39:29.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.097+0000 7fe571f8b640 10 librados: watch_flush enter 2024-03-27T21:39:29.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.097+0000 7fe571f8b640 10 librados: watch_flush exit 2024-03-27T21:39:29.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.097+0000 7fe571f8b640 20 client.4159.objecter shutdown clearing up homeless session... 2024-03-27T21:39:29.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.097+0000 7fe571f8b640 10 client.4159.objecter successfully canceled tick 2024-03-27T21:39:29.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.097+0000 7fe571f8b640 10 monclient: shutdown 2024-03-27T21:39:29.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.097+0000 7fe571f8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:29.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.097+0000 7fe571f8b640 1 -- 172.21.15.191:0/674681575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe56c13ef50 msgr2=0x7fe56c1d3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:29.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.097+0000 7fe571f8b640 1 --2- 172.21.15.191:0/674681575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe56c13ef50 0x7fe56c1d3e00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe564009e50 tx=0x7fe564004a40 comp rx=0 tx=0).stop 2024-03-27T21:39:29.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.097+0000 7fe571f8b640 1 -- 172.21.15.191:0/674681575 shutdown_connections 2024-03-27T21:39:29.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.097+0000 7fe571f8b640 1 --2- 172.21.15.191:0/674681575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe56c13ef50 0x7fe56c1d3e00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.097+0000 7fe571f8b640 1 --2- 172.21.15.191:0/674681575 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe56c13cbd0 0x7fe56c142190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.086 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.097+0000 7fe571f8b640 1 --2- 172.21.15.191:0/674681575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe56c073640 0x7fe56c142c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.086 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.097+0000 7fe571f8b640 1 -- 172.21.15.191:0/674681575 >> 172.21.15.191:0/674681575 conn(0x7fe56c0b3e70 msgr2=0x7fe56c069ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:29.086 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.097+0000 7fe571f8b640 1 -- 172.21.15.191:0/674681575 shutdown_connections 2024-03-27T21:39:29.086 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.097+0000 7fe571f8b640 1 -- 172.21.15.191:0/674681575 wait complete. 2024-03-27T21:39:29.086 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.097+0000 7fe571f8b640 1 librados: shutdown 2024-03-27T21:39:29.098 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 9fbd0927-ee2f-471b-8df8-c4e0fc40b9b0 7 2024-03-27T21:39:29.148 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:29.162+0000 7f6438632040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-27T21:39:29.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.209+0000 7f9c34e5c640 10 monclient: get_monmap_and_config 2024-03-27T21:39:29.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.209+0000 7f9c34e5c640 10 monclient: build_initial_monmap 2024-03-27T21:39:29.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.209+0000 7f9c34e5c640 10 monclient: monmap: 2024-03-27T21:39:29.197 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-27T21:39:29.197 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:29.197 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:39:29.211114+0000 2024-03-27T21:39:29.197 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:39:29.211114+0000 2024-03-27T21:39:29.197 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:29.197 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:29.197 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:29.197 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:29.198 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:29.198 INFO:teuthology.orchestra.run.smithi191.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:29.198 INFO:teuthology.orchestra.run.smithi191.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:29.198 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:29.210 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.210+0000 7f9c34e5c640 1 Processor -- start 2024-03-27T21:39:29.210 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.210+0000 7f9c34e5c640 1 -- start start 2024-03-27T21:39:29.210 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.210+0000 7f9c34e5c640 10 monclient: init 2024-03-27T21:39:29.210 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c34e5c640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:29.210 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c34e5c640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-03-27T21:39:29.210 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c34e5c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9c300b40b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:29.210 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c34e5c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9c3013b020 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:29.210 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c34e5c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c3013b480 0x7f9c30141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:29.210 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c34e5c640 10 monclient(hunting): picked mon.noname-b con 0x7f9c3013b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:29.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c34e5c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c300b3830 con 0x7f9c300b40b0 2024-03-27T21:39:29.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c34e5c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c300aa520 con 0x7f9c3013b020 2024-03-27T21:39:29.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c34e5c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:29.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c34e5c640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3013b910 con 0x7f9c3013b480 2024-03-27T21:39:29.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c34e5c640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:29.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c34e5c640 10 monclient(hunting): authenticate will time out at 786.494888s 2024-03-27T21:39:29.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2dd9b640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f9c3013b020 0x7f9c301416f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:37464/0 (socket says 172.21.15.191:37464) 2024-03-27T21:39:29.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2dd9b640 1 -- 172.21.15.191:0/2648873088 learned_addr learned my addr 172.21.15.191:0/2648873088 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-27T21:39:29.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2ed9d640 1 --2- 172.21.15.191:0/2648873088 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c3013b480 0x7f9c30141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:29.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2ed9d640 10 monclient(hunting): get_auth_request con 0x7f9c3013b480 auth_method 0 2024-03-27T21:39:29.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:29.211 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2ed9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:29.212 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:29.212 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:29.212 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:29.212 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:29.212 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2d59a640 1 -- 172.21.15.191:0/2648873088 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872292798 0 0) 0x7f9c300aa520 con 0x7f9c3013b020 2024-03-27T21:39:29.212 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2d59a640 10 monclient(hunting): _negotiate 2024-03-27T21:39:29.212 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2d59a640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:29.212 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:29.212 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2d59a640 10 monclient(hunting): my global_id is 4162 2024-03-27T21:39:29.212 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2d59a640 1 -- 172.21.15.191:0/2648873088 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c18003520 con 0x7f9c3013b020 2024-03-27T21:39:29.212 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2d59a640 1 -- 172.21.15.191:0/2648873088 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660643357 0 0) 0x7f9c300b3830 con 0x7f9c300b40b0 2024-03-27T21:39:29.212 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2d59a640 10 monclient(hunting): _negotiate 2024-03-27T21:39:29.212 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2d59a640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:29.212 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:29.212 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2d59a640 10 monclient(hunting): my global_id is 4155 2024-03-27T21:39:29.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2d59a640 1 -- 172.21.15.191:0/2648873088 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c300aa520 con 0x7f9c300b40b0 2024-03-27T21:39:29.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2ed9d640 10 monclient(hunting): handle_auth_done global_id 4139 payload 274 2024-03-27T21:39:29.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2ed9d640 1 -- 172.21.15.191:0/2648873088 >> v1:172.21.15.191:6789/0 conn(0x7f9c3013b020 legacy=0x7f9c301416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:29.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2ed9d640 1 -- 172.21.15.191:0/2648873088 >> v1:172.21.15.60:6789/0 conn(0x7f9c300b40b0 legacy=0x7f9c300a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:29.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2ed9d640 10 monclient: _finish_hunting 0 2024-03-27T21:39:29.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2ed9d640 1 monclient: found mon.noname-b 2024-03-27T21:39:29.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:29.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:29.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.211+0000 7f9c2ed9d640 1 -- 172.21.15.191:0/2648873088 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c300a9470 con 0x7f9c3013b480 2024-03-27T21:39:29.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.212+0000 7f9c2ed9d640 1 --2- 172.21.15.191:0/2648873088 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c3013b480 0x7f9c30141e00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9c28009dc0 tx=0x7f9c2802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7094d1beb25ad84a server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:29.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.212+0000 7f9c2d59a640 1 -- 172.21.15.191:0/2648873088 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2802ccd0 con 0x7f9c3013b480 2024-03-27T21:39:29.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.212+0000 7f9c2d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:29.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.212+0000 7f9c2d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:29.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.212+0000 7f9c2d59a640 10 monclient: dump: 2024-03-27T21:39:29.213 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:29.214 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:29.214 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.214 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.214 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:29.214 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:29.214 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:29.214 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:29.214 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:29.214 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:29.214 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.212+0000 7f9c2d59a640 10 monclient: _finish_auth 0 2024-03-27T21:39:29.214 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.212+0000 7f9c2d59a640 10 monclient: _check_auth_tickets 2024-03-27T21:39:29.214 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.212+0000 7f9c2d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:29.214 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.212+0000 7f9c34e5c640 5 monclient: authenticate success, global_id 4139 2024-03-27T21:39:29.214 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.212+0000 7f9c2d59a640 1 -- 172.21.15.191:0/2648873088 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2802ce30 con 0x7f9c3013b480 2024-03-27T21:39:29.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.212+0000 7f9c34e5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:29.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.212+0000 7f9c2d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:29.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.212+0000 7f9c34e5c640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:29.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.212+0000 7f9c2d59a640 1 -- 172.21.15.191:0/2648873088 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c28034660 con 0x7f9c3013b480 2024-03-27T21:39:29.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.212+0000 7f9c34e5c640 4 set_mon_vals no callback set 2024-03-27T21:39:29.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.212+0000 7f9c2d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:29.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.212+0000 7f9c2d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:29.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.212+0000 7f9c2d59a640 10 monclient: dump: 2024-03-27T21:39:29.215 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:29.215 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:29.215 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.215 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.215 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:29.215 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:29.216 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:29.216 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:29.216 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:29.216 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:29.216 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.213+0000 7f9c34e5c640 10 monclient: shutdown 2024-03-27T21:39:29.216 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.213+0000 7f9c34e5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:29.216 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.213+0000 7f9c34e5c640 1 -- 172.21.15.191:0/2648873088 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c3013b480 msgr2=0x7f9c30141e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:29.216 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.213+0000 7f9c34e5c640 1 --2- 172.21.15.191:0/2648873088 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c3013b480 0x7f9c30141e00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9c28009dc0 tx=0x7f9c2802bf20 comp rx=0 tx=0).stop 2024-03-27T21:39:29.216 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.213+0000 7f9c34e5c640 1 -- 172.21.15.191:0/2648873088 shutdown_connections 2024-03-27T21:39:29.216 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.213+0000 7f9c34e5c640 1 --2- 172.21.15.191:0/2648873088 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c3013b480 0x7f9c30141e00 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.216 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.213+0000 7f9c34e5c640 1 -- 172.21.15.191:0/2648873088 >> 172.21.15.191:0/2648873088 conn(0x7f9c300608e0 msgr2=0x7f9c30060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:29.216 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.213+0000 7f9c34e5c640 1 -- 172.21.15.191:0/2648873088 shutdown_connections 2024-03-27T21:39:29.216 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.213+0000 7f9c34e5c640 1 -- 172.21.15.191:0/2648873088 wait complete. 2024-03-27T21:39:29.216 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.214+0000 7f9c34e5c640 10 monclient: build_initial_monmap 2024-03-27T21:39:29.216 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.214+0000 7f9c34e5c640 10 monclient: monmap: 2024-03-27T21:39:29.216 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-27T21:39:29.217 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:29.217 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-27T21:39:29.217 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-27T21:39:29.217 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:29.217 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-27T21:39:29.217 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:29.217 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:29.217 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:29.217 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:29.217 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.214+0000 7f9c34e5c640 1 librados: starting msgr at 2024-03-27T21:39:29.217 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.214+0000 7f9c34e5c640 1 librados: starting objecter 2024-03-27T21:39:29.217 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.214+0000 7f9c34e5c640 1 Processor -- start 2024-03-27T21:39:29.217 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.214+0000 7f9c34e5c640 1 -- start start 2024-03-27T21:39:29.217 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.214+0000 7f9c34e5c640 1 librados: setting wanted keys 2024-03-27T21:39:29.217 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.214+0000 7f9c34e5c640 1 librados: calling monclient init 2024-03-27T21:39:29.217 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.214+0000 7f9c34e5c640 10 monclient: init 2024-03-27T21:39:29.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c34e5c640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:29.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c34e5c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:39:29.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c34e5c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c300b40b0 0x7f9c3013e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:29.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c34e5c640 10 monclient(hunting): picked mon.noname-a con 0x7f9c300b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:29.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c34e5c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c3013b020 0x7f9c301331c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:29.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c34e5c640 10 monclient(hunting): picked mon.noname-b con 0x7f9c3013b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:29.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c34e5c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c3013b480 0x7f9c30133700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:29.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c34e5c640 10 monclient(hunting): picked mon.noname-c con 0x7f9c3013b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:29.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c34e5c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:29.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c34e5c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c30133c40 con 0x7f9c300b40b0 2024-03-27T21:39:29.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c34e5c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:29.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c34e5c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3013a640 con 0x7f9c3013b020 2024-03-27T21:39:29.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c34e5c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:29.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c34e5c640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c301391c0 con 0x7f9c3013b480 2024-03-27T21:39:29.219 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c34e5c640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:29.219 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c34e5c640 10 monclient(hunting): authenticate will time out at 786.498940s 2024-03-27T21:39:29.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2dd9b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c3013b020 0x7f9c301331c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:29.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2dd9b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c3013b020 0x7f9c301331c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:52606/0 (socket says 172.21.15.191:52606) 2024-03-27T21:39:29.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2dd9b640 1 -- 172.21.15.191:0/3341354382 learned_addr learned my addr 172.21.15.191:0/3341354382 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-27T21:39:29.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2dd9b640 10 monclient(hunting): get_auth_request con 0x7f9c3013b020 auth_method 0 2024-03-27T21:39:29.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:29.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:29.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:29.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2e59c640 1 --2- 172.21.15.191:0/3341354382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c300b40b0 0x7f9c3013e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:29.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2ed9d640 1 --2- 172.21.15.191:0/3341354382 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c3013b480 0x7f9c30133700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:29.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2e59c640 10 monclient(hunting): get_auth_request con 0x7f9c300b40b0 auth_method 0 2024-03-27T21:39:29.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:29.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2e59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:29.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:29.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2ed9d640 10 monclient(hunting): get_auth_request con 0x7f9c3013b480 auth_method 0 2024-03-27T21:39:29.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:29.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2ed9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:29.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:29.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:29.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:29.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:29.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:29.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:29.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:29.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:29.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:29.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:29.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2dd9b640 10 monclient(hunting): handle_auth_done global_id 4165 payload 824 2024-03-27T21:39:29.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2dd9b640 1 -- 172.21.15.191:0/3341354382 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c3013b480 msgr2=0x7f9c30133700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:29.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2dd9b640 1 --2- 172.21.15.191:0/3341354382 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c3013b480 0x7f9c30133700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2dd9b640 1 -- 172.21.15.191:0/3341354382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c300b40b0 msgr2=0x7f9c3013e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:29.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2dd9b640 1 --2- 172.21.15.191:0/3341354382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c300b40b0 0x7f9c3013e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2dd9b640 10 monclient: _finish_hunting 0 2024-03-27T21:39:29.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2dd9b640 1 monclient: found mon.noname-b 2024-03-27T21:39:29.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:29.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:39:29.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2dd9b640 1 -- 172.21.15.191:0/3341354382 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c28009a70 con 0x7f9c3013b020 2024-03-27T21:39:29.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2e59c640 1 --2- 172.21.15.191:0/3341354382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c300b40b0 0x7f9c3013e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:29.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.215+0000 7f9c2ed9d640 1 --2- 172.21.15.191:0/3341354382 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c3013b480 0x7f9c30133700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:29.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c2dd9b640 1 --2- 172.21.15.191:0/3341354382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c3013b020 0x7f9c301331c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9c2000c530 tx=0x7f9c2000c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:29.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c1effd640 10 client.?.objecter ms_handle_connect 0x7f9c3013b020 2024-03-27T21:39:29.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c1effd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:29.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c1effd640 1 -- 172.21.15.191:0/3341354382 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c20005ce0 con 0x7f9c3013b020 2024-03-27T21:39:29.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:29.225 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c1effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:29.225 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c1effd640 10 monclient: dump: 2024-03-27T21:39:29.225 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:29.225 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:29.225 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.225 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.225 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:29.225 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:29.225 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:29.225 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:29.225 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:29.225 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:29.225 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c1effd640 10 monclient: _finish_auth 0 2024-03-27T21:39:29.225 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c1effd640 10 monclient: _check_auth_tickets 2024-03-27T21:39:29.226 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:29.226 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c34e5c640 5 monclient: authenticate success, global_id 4165 2024-03-27T21:39:29.226 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c1effd640 1 -- 172.21.15.191:0/3341354382 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c20005e60 con 0x7f9c3013b020 2024-03-27T21:39:29.226 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c34e5c640 10 monclient: _renew_subs 2024-03-27T21:39:29.226 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c34e5c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:29.226 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c34e5c640 1 -- 172.21.15.191:0/3341354382 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c301385f0 con 0x7f9c3013b020 2024-03-27T21:39:29.226 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:29.226 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c1effd640 1 -- 172.21.15.191:0/3341354382 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c200072b0 con 0x7f9c3013b020 2024-03-27T21:39:29.226 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:29.226 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c1effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:29.226 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c1effd640 10 monclient: dump: 2024-03-27T21:39:29.226 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-27T21:39:29.226 INFO:teuthology.orchestra.run.smithi191.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:29.226 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.227 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.227 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:29.227 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-27T21:39:29.227 INFO:teuthology.orchestra.run.smithi191.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:29.227 INFO:teuthology.orchestra.run.smithi191.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:29.227 INFO:teuthology.orchestra.run.smithi191.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:29.227 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-27T21:39:29.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c34e5c640 10 client.4165.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:29.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c34e5c640 10 monclient: _renew_subs 2024-03-27T21:39:29.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c2cd99640 4 set_mon_vals no callback set 2024-03-27T21:39:29.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c34e5c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:29.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c34e5c640 1 -- 172.21.15.191:0/3341354382 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c300b0dc0 con 0x7f9c3013b020 2024-03-27T21:39:29.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c34e5c640 1 librados: init done 2024-03-27T21:39:29.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c1effd640 1 -- 172.21.15.191:0/3341354382 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9c2001c440 con 0x7f9c3013b020 2024-03-27T21:39:29.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c1effd640 1 -- 172.21.15.191:0/3341354382 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f9c20012af0 con 0x7f9c3013b020 2024-03-27T21:39:29.228 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c1effd640 10 client.4165.objecter ms_dispatch 0x7f9c30000b90 osd_map(9..9 src has 1..9) v4 2024-03-27T21:39:29.228 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c1effd640 3 client.4165.objecter handle_osd_map got epochs [9,9] > 0 2024-03-27T21:39:29.228 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c1effd640 3 client.4165.objecter handle_osd_map decoding full epoch 9 2024-03-27T21:39:29.228 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.216+0000 7f9c1effd640 20 client.4165.objecter dump_active .. 0 homeless 2024-03-27T21:39:29.228 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.217+0000 7f9c34e5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:29.228 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.217+0000 7f9c34e5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:29.228 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.217+0000 7f9c34e5c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:29.228 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.217+0000 7f9c34e5c640 1 -- 172.21.15.191:0/3341354382 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c3013a360 con 0x7f9c3013b020 2024-03-27T21:39:29.228 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.219+0000 7f9c1effd640 1 -- 172.21.15.191:0/3341354382 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f9c3013a360 con 0x7f9c3013b020 2024-03-27T21:39:29.228 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.219+0000 7f9c1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:29.228 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.219+0000 7f9c1effd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:29.276 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.289+0000 7f9c34e5c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9fbd0927-ee2f-471b-8df8-c4e0fc40b9b0", "id": 7}] 2024-03-27T21:39:29.276 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.289+0000 7f9c34e5c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9fbd0927-ee2f-471b-8df8-c4e0fc40b9b0", "id": 7}] 2024-03-27T21:39:29.276 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.289+0000 7f9c34e5c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:29.276 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.289+0000 7f9c34e5c640 1 -- 172.21.15.191:0/3341354382 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9fbd0927-ee2f-471b-8df8-c4e0fc40b9b0", "id": 7} v 0) v1 -- 0x7f9c30069470 con 0x7f9c3013b020 2024-03-27T21:39:29.280 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.293+0000 7f9c1effd640 1 -- 172.21.15.191:0/3341354382 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9fbd0927-ee2f-471b-8df8-c4e0fc40b9b0", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9c20017070 con 0x7f9c3013b020 2024-03-27T21:39:29.280 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.293+0000 7f9c1effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9fbd0927-ee2f-471b-8df8-c4e0fc40b9b0", "id": 7}] 2024-03-27T21:39:29.280 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.293+0000 7f9c1effd640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:29.280 INFO:teuthology.orchestra.run.smithi191.stdout:7 2024-03-27T21:39:29.282 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.294+0000 7f9c34e5c640 10 librados: watch_flush enter 2024-03-27T21:39:29.282 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.294+0000 7f9c34e5c640 10 librados: watch_flush exit 2024-03-27T21:39:29.282 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.294+0000 7f9c34e5c640 20 client.4165.objecter shutdown clearing up homeless session... 2024-03-27T21:39:29.282 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.294+0000 7f9c34e5c640 10 client.4165.objecter successfully canceled tick 2024-03-27T21:39:29.282 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.294+0000 7f9c34e5c640 10 monclient: shutdown 2024-03-27T21:39:29.282 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.294+0000 7f9c34e5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:29.282 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.294+0000 7f9c34e5c640 1 -- 172.21.15.191:0/3341354382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c3013b020 msgr2=0x7f9c301331c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:29.282 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.294+0000 7f9c34e5c640 1 --2- 172.21.15.191:0/3341354382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c3013b020 0x7f9c301331c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9c2000c530 tx=0x7f9c2000c9f0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.282 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.294+0000 7f9c34e5c640 1 -- 172.21.15.191:0/3341354382 shutdown_connections 2024-03-27T21:39:29.282 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.294+0000 7f9c34e5c640 1 --2- 172.21.15.191:0/3341354382 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c3013b480 0x7f9c30133700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.282 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.294+0000 7f9c34e5c640 1 --2- 172.21.15.191:0/3341354382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c3013b020 0x7f9c301331c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.282 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.294+0000 7f9c34e5c640 1 --2- 172.21.15.191:0/3341354382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c300b40b0 0x7f9c3013e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.283 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.294+0000 7f9c34e5c640 1 -- 172.21.15.191:0/3341354382 >> 172.21.15.191:0/3341354382 conn(0x7f9c300608e0 msgr2=0x7f9c300b5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:29.283 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.294+0000 7f9c34e5c640 1 -- 172.21.15.191:0/3341354382 shutdown_connections 2024-03-27T21:39:29.283 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.294+0000 7f9c34e5c640 1 -- 172.21.15.191:0/3341354382 wait complete. 2024-03-27T21:39:29.283 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-27T21:39:29.294+0000 7f9c34e5c640 1 librados: shutdown 2024-03-27T21:39:29.289 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:29.303+0000 7f6438632040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-27T21:39:29.291 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-27T21:39:29.292 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-27T21:39:29.294 INFO:tasks.ceph.osd.0:Started 2024-03-27T21:39:29.294 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-27T21:39:29.295 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-27T21:39:29.297 INFO:tasks.ceph.osd.1:Started 2024-03-27T21:39:29.298 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-27T21:39:29.298 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-27T21:39:29.300 INFO:tasks.ceph.osd.2:Started 2024-03-27T21:39:29.300 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-27T21:39:29.300 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-27T21:39:29.302 INFO:tasks.ceph.osd.3:Started 2024-03-27T21:39:29.302 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-27T21:39:29.303 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-27T21:39:29.304 INFO:tasks.ceph.osd.4:Started 2024-03-27T21:39:29.305 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-27T21:39:29.305 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-27T21:39:29.307 INFO:tasks.ceph.osd.5:Started 2024-03-27T21:39:29.307 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-27T21:39:29.307 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-27T21:39:29.309 INFO:tasks.ceph.osd.6:Started 2024-03-27T21:39:29.309 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-27T21:39:29.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-27T21:39:29.311 INFO:tasks.ceph.osd.7:Started 2024-03-27T21:39:29.312 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T21:39:29.376 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:29.389+0000 7f4298a0e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-27T21:39:29.431 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:29.444+0000 7f6438632040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-27T21:39:29.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.456+0000 7f5605511640 10 monclient: get_monmap_and_config 2024-03-27T21:39:29.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.456+0000 7f5605511640 10 monclient: build_initial_monmap 2024-03-27T21:39:29.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.456+0000 7f5605511640 10 monclient: monmap: 2024-03-27T21:39:29.444 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:29.444 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:29.444 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:29.457362+0000 2024-03-27T21:39:29.444 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:29.457362+0000 2024-03-27T21:39:29.444 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:29.444 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:29.444 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:29.444 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:29.444 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:29.444 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:29.444 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:29.444 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:29.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.456+0000 7f5605511640 1 Processor -- start 2024-03-27T21:39:29.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.456+0000 7f5605511640 1 -- start start 2024-03-27T21:39:29.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.456+0000 7f5605511640 10 monclient: init 2024-03-27T21:39:29.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f5605511640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:29.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f5605511640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-03-27T21:39:29.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f5605511640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f56000b40b0 0x7f56000a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:29.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f5605511640 10 monclient(hunting): picked mon.noname-b con 0x7f56000b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:29.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f5605511640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f560013b020 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:29.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f5605511640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f560013b480 0x7f5600141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:29.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f5605511640 10 monclient(hunting): picked mon.noname-c con 0x7f560013b480 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:29.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f5605511640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56000b3830 con 0x7f560013b020 2024-03-27T21:39:29.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f5605511640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:29.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f5605511640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f560013b910 con 0x7f560013b480 2024-03-27T21:39:29.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f5605511640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:29.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f5605511640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56000b4490 con 0x7f56000b40b0 2024-03-27T21:39:29.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f5605511640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:29.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f5605511640 10 monclient(hunting): authenticate will time out at 810.678524s 2024-03-27T21:39:29.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f55feffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f56000b40b0 0x7f56000a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:29.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f55feffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f56000b40b0 0x7f56000a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47772/0 (socket says 172.21.15.60:47772) 2024-03-27T21:39:29.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f55feffd640 1 -- 172.21.15.60:0/1578471651 learned_addr learned my addr 172.21.15.60:0/1578471651 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:29.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f55feffd640 10 monclient(hunting): get_auth_request con 0x7f56000b40b0 auth_method 0 2024-03-27T21:39:29.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f55feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:29.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f55feffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:29.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f55feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:29.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f55feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:29.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f55feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:29.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f55feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:29.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f55ff7fe640 1 --2- 172.21.15.60:0/1578471651 >> v2:172.21.15.191:3300/0 conn(0x7f560013b480 0x7f5600141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:29.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f55ff7fe640 10 monclient(hunting): get_auth_request con 0x7f560013b480 auth_method 0 2024-03-27T21:39:29.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f55ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:29.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f55ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:29.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.457+0000 7f55ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:29.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55feffd640 10 monclient(hunting): handle_auth_done global_id 4160 payload 274 2024-03-27T21:39:29.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55feffd640 1 -- 172.21.15.60:0/1578471651 >> v2:172.21.15.191:3300/0 conn(0x7f560013b480 msgr2=0x7f5600141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:29.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55feffd640 1 --2- 172.21.15.60:0/1578471651 >> v2:172.21.15.191:3300/0 conn(0x7f560013b480 0x7f5600141e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55feffd640 1 -- 172.21.15.60:0/1578471651 >> v1:172.21.15.191:6789/0 conn(0x7f560013b020 legacy=0x7f56001416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:29.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55feffd640 10 monclient: _finish_hunting 0 2024-03-27T21:39:29.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55feffd640 1 monclient: found mon.noname-b 2024-03-27T21:39:29.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:29.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55feffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:29.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55feffd640 1 -- 172.21.15.60:0/1578471651 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56000aa520 con 0x7f56000b40b0 2024-03-27T21:39:29.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55feffd640 1 --2- 172.21.15.60:0/1578471651 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f56000b40b0 0x7f56000a8b90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f55f4009870 tx=0x7f55f402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95ddcf80534f21ea server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:29.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55fdffb640 1 -- 172.21.15.60:0/1578471651 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f4031640 con 0x7f56000b40b0 2024-03-27T21:39:29.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:29.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55fdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:29.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55fdffb640 10 monclient: dump: 2024-03-27T21:39:29.456 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:29.456 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:29.456 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.456 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.456 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:29.456 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:29.456 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:29.456 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:29.456 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:29.456 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:29.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55fdffb640 10 monclient: _finish_auth 0 2024-03-27T21:39:29.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55fdffb640 10 monclient: _check_auth_tickets 2024-03-27T21:39:29.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:29.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f5605511640 5 monclient: authenticate success, global_id 4160 2024-03-27T21:39:29.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55fdffb640 1 -- 172.21.15.60:0/1578471651 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f40317a0 con 0x7f56000b40b0 2024-03-27T21:39:29.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f5605511640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:29.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55fdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:29.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f5605511640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:29.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55fdffb640 1 -- 172.21.15.60:0/1578471651 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f4031b30 con 0x7f56000b40b0 2024-03-27T21:39:29.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:29.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f5605511640 4 set_mon_vals no callback set 2024-03-27T21:39:29.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:29.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.458+0000 7f55fdffb640 10 monclient: dump: 2024-03-27T21:39:29.457 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:29.457 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:29.458 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.458 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.458 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:29.458 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:29.458 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:29.458 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:29.458 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:29.458 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:29.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.459+0000 7f5605511640 10 monclient: shutdown 2024-03-27T21:39:29.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.459+0000 7f5605511640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:29.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.459+0000 7f5605511640 1 -- 172.21.15.60:0/1578471651 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f56000b40b0 msgr2=0x7f56000a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:29.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.459+0000 7f5605511640 1 --2- 172.21.15.60:0/1578471651 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f56000b40b0 0x7f56000a8b90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f55f4009870 tx=0x7f55f402b8f0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.459+0000 7f5605511640 1 -- 172.21.15.60:0/1578471651 shutdown_connections 2024-03-27T21:39:29.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.459+0000 7f5605511640 1 --2- 172.21.15.60:0/1578471651 >> v2:172.21.15.191:3300/0 conn(0x7f560013b480 0x7f5600141e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.459+0000 7f5605511640 1 --2- 172.21.15.60:0/1578471651 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f56000b40b0 0x7f56000a8b90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.459+0000 7f5605511640 1 -- 172.21.15.60:0/1578471651 >> 172.21.15.60:0/1578471651 conn(0x7f56000608e0 msgr2=0x7f5600060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:29.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.459+0000 7f5605511640 1 -- 172.21.15.60:0/1578471651 shutdown_connections 2024-03-27T21:39:29.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.459+0000 7f5605511640 1 -- 172.21.15.60:0/1578471651 wait complete. 2024-03-27T21:39:29.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.460+0000 7f5605511640 10 monclient: build_initial_monmap 2024-03-27T21:39:29.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.460+0000 7f5605511640 10 monclient: monmap: 2024-03-27T21:39:29.459 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:29.459 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:29.459 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:29.459 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:29.459 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:29.459 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:29.460 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.460+0000 7f5605511640 1 librados: starting msgr at 2024-03-27T21:39:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.460+0000 7f5605511640 1 librados: starting objecter 2024-03-27T21:39:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.460+0000 7f5605511640 1 Processor -- start 2024-03-27T21:39:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.460+0000 7f5605511640 1 -- start start 2024-03-27T21:39:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.460+0000 7f5605511640 1 librados: setting wanted keys 2024-03-27T21:39:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.460+0000 7f5605511640 1 librados: calling monclient init 2024-03-27T21:39:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.460+0000 7f5605511640 10 monclient: init 2024-03-27T21:39:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f5605511640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f5605511640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:39:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f5605511640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56000b40b0 0x7f560013e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f5605511640 10 monclient(hunting): picked mon.noname-a con 0x7f56000b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f5605511640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f560013b020 0x7f560013da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f5605511640 10 monclient(hunting): picked mon.noname-c con 0x7f560013b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f5605511640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f560013b480 0x7f56000abdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f5605511640 10 monclient(hunting): picked mon.noname-b con 0x7f560013b480 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f5605511640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f5605511640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56000ab6f0 con 0x7f56000b40b0 2024-03-27T21:39:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f5605511640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f5605511640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560013cbd0 con 0x7f560013b480 2024-03-27T21:39:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f5605511640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f5605511640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f560013a8c0 con 0x7f560013b020 2024-03-27T21:39:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f5605511640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f5605511640 10 monclient(hunting): authenticate will time out at 810.682257s 2024-03-27T21:39:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55fe7fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f560013b020 0x7f560013da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:29.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55feffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56000b40b0 0x7f560013e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:29.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55fe7fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f560013b020 0x7f560013da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47774/0 (socket says 172.21.15.60:47774) 2024-03-27T21:39:29.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55fe7fc640 1 -- 172.21.15.60:0/4119669062 learned_addr learned my addr 172.21.15.60:0/4119669062 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:29.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55fe7fc640 10 monclient(hunting): get_auth_request con 0x7f560013b020 auth_method 0 2024-03-27T21:39:29.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:29.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:29.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:29.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55feffd640 10 monclient(hunting): get_auth_request con 0x7f56000b40b0 auth_method 0 2024-03-27T21:39:29.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:29.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55feffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:29.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:29.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55ff7fe640 1 --2- 172.21.15.60:0/4119669062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f560013b480 0x7f56000abdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:29.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55ff7fe640 10 monclient(hunting): get_auth_request con 0x7f560013b480 auth_method 0 2024-03-27T21:39:29.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:29.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:29.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:29.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55fe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:29.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55fe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:29.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55fe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:29.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:29.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:29.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:29.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:29.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:29.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:29.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55fe7fc640 10 monclient(hunting): handle_auth_done global_id 4163 payload 824 2024-03-27T21:39:29.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55fe7fc640 1 -- 172.21.15.60:0/4119669062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f560013b480 msgr2=0x7f56000abdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:29.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55fe7fc640 1 --2- 172.21.15.60:0/4119669062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f560013b480 0x7f56000abdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55fe7fc640 1 -- 172.21.15.60:0/4119669062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56000b40b0 msgr2=0x7f560013e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:29.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55fe7fc640 1 --2- 172.21.15.60:0/4119669062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56000b40b0 0x7f560013e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55fe7fc640 10 monclient: _finish_hunting 0 2024-03-27T21:39:29.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55fe7fc640 1 monclient: found mon.noname-c 2024-03-27T21:39:29.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:29.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55fe7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:39:29.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55fe7fc640 1 -- 172.21.15.60:0/4119669062 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f40094d0 con 0x7f560013b020 2024-03-27T21:39:29.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55feffd640 1 --2- 172.21.15.60:0/4119669062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56000b40b0 0x7f560013e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:29.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.461+0000 7f55fe7fc640 1 --2- 172.21.15.60:0/4119669062 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f560013b020 0x7f560013da10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f55f000c450 tx=0x7f55f000c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:29.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55e77fe640 10 client.?.objecter ms_handle_connect 0x7f560013b020 2024-03-27T21:39:29.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55e77fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:29.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55e77fe640 1 -- 172.21.15.60:0/4119669062 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f00052d0 con 0x7f560013b020 2024-03-27T21:39:29.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:29.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55e77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:29.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55e77fe640 10 monclient: dump: 2024-03-27T21:39:29.465 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:29.465 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:29.465 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.465 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.465 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:29.465 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:29.465 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:29.465 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:29.465 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:29.465 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:29.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55e77fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:29.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55e77fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:29.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:29.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55e77fe640 1 -- 172.21.15.60:0/4119669062 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f0005430 con 0x7f560013b020 2024-03-27T21:39:29.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f5605511640 5 monclient: authenticate success, global_id 4163 2024-03-27T21:39:29.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f5605511640 10 monclient: _renew_subs 2024-03-27T21:39:29.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f5605511640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:29.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f5605511640 1 -- 172.21.15.60:0/4119669062 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f560013aae0 con 0x7f560013b020 2024-03-27T21:39:29.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:29.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55e77fe640 1 -- 172.21.15.60:0/4119669062 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f0012870 con 0x7f560013b020 2024-03-27T21:39:29.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:29.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:29.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55e77fe640 10 monclient: dump: 2024-03-27T21:39:29.467 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:29.467 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:29.467 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.467 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:29.467 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:29.467 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:29.467 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:29.467 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:29.467 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:29.467 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:29.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55fd7fa640 4 set_mon_vals no callback set 2024-03-27T21:39:29.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f5605511640 10 client.4163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:29.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f5605511640 10 monclient: _renew_subs 2024-03-27T21:39:29.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f5605511640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:29.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f5605511640 1 -- 172.21.15.60:0/4119669062 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f560013e1f0 con 0x7f560013b020 2024-03-27T21:39:29.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f5605511640 1 librados: init done 2024-03-27T21:39:29.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55e77fe640 1 -- 172.21.15.60:0/4119669062 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f55f0012cd0 con 0x7f560013b020 2024-03-27T21:39:29.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55e77fe640 1 -- 172.21.15.60:0/4119669062 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f55f0011380 con 0x7f560013b020 2024-03-27T21:39:29.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55e77fe640 10 client.4163.objecter ms_dispatch 0x7f5600000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T21:39:29.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55e77fe640 3 client.4163.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T21:39:29.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55e77fe640 3 client.4163.objecter handle_osd_map decoding full epoch 10 2024-03-27T21:39:29.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.462+0000 7f55e77fe640 20 client.4163.objecter dump_active .. 0 homeless 2024-03-27T21:39:29.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.463+0000 7f5605511640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:29.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.463+0000 7f5605511640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:29.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.463+0000 7f5605511640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:29.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.463+0000 7f5605511640 1 -- 172.21.15.60:0/4119669062 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f560014bce0 con 0x7f560013b020 2024-03-27T21:39:29.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.465+0000 7f55e77fe640 1 -- 172.21.15.60:0/4119669062 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f55f00035f0 con 0x7f560013b020 2024-03-27T21:39:29.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.465+0000 7f55e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:29.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.465+0000 7f55e77fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:29.479 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:29.493+0000 7f6438632040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-27T21:39:29.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.526+0000 7f5605511640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:39:29.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.526+0000 7f5605511640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:39:29.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.526+0000 7f5605511640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:29.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.526+0000 7f5605511640 1 -- 172.21.15.60:0/4119669062 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f560013e6e0 con 0x7f560013b020 2024-03-27T21:39:29.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.526+0000 7f55e77fe640 1 -- 172.21.15.60:0/4119669062 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f55f0003790 con 0x7f560013b020 2024-03-27T21:39:29.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.526+0000 7f55e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:39:29.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.526+0000 7f55e77fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:29.514 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:39:29.514 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":10,"fsid":"dcd19d9a-535c-4b41-9095-d588268e2860","created":"2024-03-27T21:39:27.092191+0000","modified":"2024-03-27T21:39:29.291620+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T21:39:29.291620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"778aaa64-6720-495b-bcdc-86fbcece9d58","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d960adf1-3d83-4e24-a612-8df53fa43d0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b8ffc48b-cf6f-4701-a6aa-c25fdb0f7977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0f14f2d8-b271-48e4-a198-aa5279a01c06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8cdeb7ae-1c6e-4d0d-ba44-80248eff2fb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93748cf8-d3fe-46b8-950f-e01e28a5e02c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1ad21434-cbc5-4de9-8d63-76f894f7e736","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9fbd0927-ee2f-471b-8df8-c4e0fc40b9b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T21:39:29.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.527+0000 7f5605511640 10 librados: watch_flush enter 2024-03-27T21:39:29.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.527+0000 7f5605511640 10 librados: watch_flush exit 2024-03-27T21:39:29.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.527+0000 7f5605511640 20 client.4163.objecter shutdown clearing up homeless session... 2024-03-27T21:39:29.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.527+0000 7f5605511640 10 client.4163.objecter successfully canceled tick 2024-03-27T21:39:29.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.527+0000 7f5605511640 10 monclient: shutdown 2024-03-27T21:39:29.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.527+0000 7f5605511640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:29.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.527+0000 7f5605511640 1 -- 172.21.15.60:0/4119669062 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f560013b020 msgr2=0x7f560013da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:29.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.527+0000 7f5605511640 1 --2- 172.21.15.60:0/4119669062 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f560013b020 0x7f560013da10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f55f000c450 tx=0x7f55f000c910 comp rx=0 tx=0).stop 2024-03-27T21:39:29.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.527+0000 7f5605511640 1 -- 172.21.15.60:0/4119669062 shutdown_connections 2024-03-27T21:39:29.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.527+0000 7f5605511640 1 --2- 172.21.15.60:0/4119669062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f560013b480 0x7f56000abdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.527+0000 7f5605511640 1 --2- 172.21.15.60:0/4119669062 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f560013b020 0x7f560013da10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.527+0000 7f5605511640 1 --2- 172.21.15.60:0/4119669062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56000b40b0 0x7f560013e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:29.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.527+0000 7f5605511640 1 -- 172.21.15.60:0/4119669062 >> 172.21.15.60:0/4119669062 conn(0x7f56000608e0 msgr2=0x7f56000b4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:29.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.527+0000 7f5605511640 1 -- 172.21.15.60:0/4119669062 shutdown_connections 2024-03-27T21:39:29.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.527+0000 7f5605511640 1 -- 172.21.15.60:0/4119669062 wait complete. 2024-03-27T21:39:29.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:29.527+0000 7f5605511640 1 librados: shutdown 2024-03-27T21:39:29.517 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:29.529+0000 7f4298a0e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-27T21:39:29.524 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-27T21:39:29.524 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-27T21:39:29.597 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:29.610+0000 7f6438632040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-27T21:39:29.626 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:29.639+0000 7f4298a0e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-27T21:39:29.672 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:29.685+0000 7f4298a0e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-27T21:39:29.695 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:29.709+0000 7f6438632040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-27T21:39:29.781 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:29.794+0000 7f4298a0e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-27T21:39:29.855 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:29.869+0000 7f6438632040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-27T21:39:29.874 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:29.887+0000 7f4298a0e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-27T21:39:29.899 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:29.913+0000 7f6438632040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-27T21:39:29.907 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:39:29.920+0000 7f439fc0d640 -1 Falling back to public interface 2024-03-27T21:39:29.917 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:39:29.930+0000 7f912600f640 -1 Falling back to public interface 2024-03-27T21:39:29.922 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:39:29.932+0000 7f33e679e640 -1 Falling back to public interface 2024-03-27T21:39:29.922 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:39:29.934+0000 7fa178d24640 -1 Falling back to public interface 2024-03-27T21:39:29.923 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:39:29.932+0000 7f4920de1640 -1 Falling back to public interface 2024-03-27T21:39:29.923 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:39:29.932+0000 7f1294c1e640 -1 Falling back to public interface 2024-03-27T21:39:29.923 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:39:29.933+0000 7f129350a640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6815/0: (98) Address already in use 2024-03-27T21:39:29.933 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:39:29.947+0000 7f84b3e08640 -1 Falling back to public interface 2024-03-27T21:39:29.935 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:39:29.948+0000 7fd003723640 -1 Falling back to public interface 2024-03-27T21:39:30.031 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:30.044+0000 7f4298a0e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-27T21:39:30.060 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:30.074+0000 7f6438632040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-27T21:39:30.072 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:30.085+0000 7f4298a0e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-27T21:39:30.231 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:30.244+0000 7f4298a0e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-27T21:39:30.330 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T21:39:30.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.432+0000 7f8d70cbd640 10 monclient: get_monmap_and_config 2024-03-27T21:39:30.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.432+0000 7f8d70cbd640 10 monclient: build_initial_monmap 2024-03-27T21:39:30.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.432+0000 7f8d70cbd640 10 monclient: monmap: 2024-03-27T21:39:30.419 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-27T21:39:30.419 INFO:teuthology.misc.health.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:30.419 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-27T21:39:30.433708+0000 2024-03-27T21:39:30.419 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-27T21:39:30.433708+0000 2024-03-27T21:39:30.420 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:30.420 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:30.420 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:30.420 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:30.420 INFO:teuthology.misc.health.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:30.420 INFO:teuthology.misc.health.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:30.420 INFO:teuthology.misc.health.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:30.420 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-27T21:39:30.427 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.433+0000 7f8d70cbd640 1 Processor -- start 2024-03-27T21:39:30.427 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.433+0000 7f8d70cbd640 1 -- start start 2024-03-27T21:39:30.427 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.433+0000 7f8d70cbd640 10 monclient: init 2024-03-27T21:39:30.427 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.433+0000 7f8d70cbd640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:30.427 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.433+0000 7f8d70cbd640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-03-27T21:39:30.427 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d70cbd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8d6c0716e0 0x7f8d6c06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:30.427 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d70cbd640 10 monclient(hunting): picked mon.noname-b con 0x7f8d6c0716e0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:30.427 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d70cbd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8d6c071ac0 0x7f8d6c141880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:30.428 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d70cbd640 10 monclient(hunting): picked mon.noname-c con 0x7f8d6c071ac0 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:30.428 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d70cbd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d6c0ad490 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:30.428 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d70cbd640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d6c06e840 con 0x7f8d6c0ad490 2024-03-27T21:39:30.428 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d70cbd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:30.428 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d70cbd640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d6c070990 con 0x7f8d6c071ac0 2024-03-27T21:39:30.428 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d70cbd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:30.428 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d70cbd640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6c070c70 con 0x7f8d6c0716e0 2024-03-27T21:39:30.428 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d70cbd640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:30.428 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d70cbd640 10 monclient(hunting): authenticate will time out at 811.655013s 2024-03-27T21:39:30.428 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d6a59c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8d6c0716e0 0x7f8d6c06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:30.428 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d6a59c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8d6c0716e0 0x7f8d6c06ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47788/0 (socket says 172.21.15.60:47788) 2024-03-27T21:39:30.428 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d6a59c640 1 -- 172.21.15.60:0/528945706 learned_addr learned my addr 172.21.15.60:0/528945706 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:30.428 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d6a59c640 10 monclient(hunting): get_auth_request con 0x7f8d6c0716e0 auth_method 0 2024-03-27T21:39:30.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d6a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:30.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d6a59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:30.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d6a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:30.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d6a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:30.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d6a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:30.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d6a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:30.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d69d9b640 1 --2- 172.21.15.60:0/528945706 >> v2:172.21.15.191:3300/0 conn(0x7f8d6c071ac0 0x7f8d6c141880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:30.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d69d9b640 10 monclient(hunting): get_auth_request con 0x7f8d6c071ac0 auth_method 0 2024-03-27T21:39:30.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d69d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:30.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d69d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:30.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d69d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:30.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d69d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:30.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d69d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:30.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d69d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:30.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d6a59c640 10 monclient(hunting): handle_auth_done global_id 4166 payload 274 2024-03-27T21:39:30.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d6a59c640 1 -- 172.21.15.60:0/528945706 >> v2:172.21.15.191:3300/0 conn(0x7f8d6c071ac0 msgr2=0x7f8d6c141880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:30.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d6a59c640 1 --2- 172.21.15.60:0/528945706 >> v2:172.21.15.191:3300/0 conn(0x7f8d6c071ac0 0x7f8d6c141880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:30.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d6a59c640 1 -- 172.21.15.60:0/528945706 >> v1:172.21.15.191:6789/0 conn(0x7f8d6c0ad490 legacy=0x7f8d6c13e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:30.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d6a59c640 10 monclient: _finish_hunting 0 2024-03-27T21:39:30.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d6a59c640 1 monclient: found mon.noname-b 2024-03-27T21:39:30.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d6a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:30.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d6a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:30.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.434+0000 7f8d6a59c640 1 -- 172.21.15.60:0/528945706 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6c070f50 con 0x7f8d6c0716e0 2024-03-27T21:39:30.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.435+0000 7f8d6a59c640 1 --2- 172.21.15.60:0/528945706 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8d6c0716e0 0x7f8d6c06ecd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c009870 tx=0x7f8d5c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5681117b0d425de2 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:30.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.435+0000 7f8d6959a640 1 -- 172.21.15.60:0/528945706 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c0317c0 con 0x7f8d6c0716e0 2024-03-27T21:39:30.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.435+0000 7f8d6959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:30.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.435+0000 7f8d6959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:30.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.435+0000 7f8d6959a640 10 monclient: dump: 2024-03-27T21:39:30.431 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-27T21:39:30.431 INFO:teuthology.misc.health.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:30.431 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:30.431 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:30.431 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:30.431 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:30.431 INFO:teuthology.misc.health.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:30.431 INFO:teuthology.misc.health.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:30.431 INFO:teuthology.misc.health.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:30.431 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-27T21:39:30.431 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.435+0000 7f8d6959a640 10 monclient: _finish_auth 0 2024-03-27T21:39:30.431 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.435+0000 7f8d6959a640 10 monclient: _check_auth_tickets 2024-03-27T21:39:30.431 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.435+0000 7f8d6959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:30.431 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.435+0000 7f8d6959a640 1 -- 172.21.15.60:0/528945706 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5c031920 con 0x7f8d6c0716e0 2024-03-27T21:39:30.432 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.435+0000 7f8d70cbd640 5 monclient: authenticate success, global_id 4166 2024-03-27T21:39:30.432 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.435+0000 7f8d70cbd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:30.432 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.435+0000 7f8d6959a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:30.432 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.435+0000 7f8d70cbd640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:30.432 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.435+0000 7f8d6959a640 1 -- 172.21.15.60:0/528945706 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c031cb0 con 0x7f8d6c0716e0 2024-03-27T21:39:30.432 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.435+0000 7f8d70cbd640 4 set_mon_vals no callback set 2024-03-27T21:39:30.432 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.435+0000 7f8d6959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:30.432 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.435+0000 7f8d6959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:30.432 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.435+0000 7f8d6959a640 10 monclient: dump: 2024-03-27T21:39:30.432 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-27T21:39:30.432 INFO:teuthology.misc.health.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:30.432 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:30.432 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:30.432 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:30.433 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:30.433 INFO:teuthology.misc.health.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:30.433 INFO:teuthology.misc.health.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:30.433 INFO:teuthology.misc.health.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:30.433 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-27T21:39:30.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.435+0000 7f8d70cbd640 10 monclient: shutdown 2024-03-27T21:39:30.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.436+0000 7f8d70cbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:30.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.436+0000 7f8d70cbd640 1 -- 172.21.15.60:0/528945706 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8d6c0716e0 msgr2=0x7f8d6c06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:30.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.436+0000 7f8d70cbd640 1 --2- 172.21.15.60:0/528945706 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8d6c0716e0 0x7f8d6c06ecd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c009870 tx=0x7f8d5c02b860 comp rx=0 tx=0).stop 2024-03-27T21:39:30.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.436+0000 7f8d70cbd640 1 -- 172.21.15.60:0/528945706 shutdown_connections 2024-03-27T21:39:30.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.436+0000 7f8d70cbd640 1 --2- 172.21.15.60:0/528945706 >> v2:172.21.15.191:3300/0 conn(0x7f8d6c071ac0 0x7f8d6c141880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:30.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.436+0000 7f8d70cbd640 1 --2- 172.21.15.60:0/528945706 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8d6c0716e0 0x7f8d6c06ecd0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:30.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.436+0000 7f8d70cbd640 1 -- 172.21.15.60:0/528945706 >> 172.21.15.60:0/528945706 conn(0x7f8d6c0b4020 msgr2=0x7f8d6c0b4430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:30.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.436+0000 7f8d70cbd640 1 -- 172.21.15.60:0/528945706 shutdown_connections 2024-03-27T21:39:30.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.436+0000 7f8d70cbd640 1 -- 172.21.15.60:0/528945706 wait complete. 2024-03-27T21:39:30.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.436+0000 7f8d70cbd640 10 monclient: build_initial_monmap 2024-03-27T21:39:30.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.436+0000 7f8d70cbd640 10 monclient: monmap: 2024-03-27T21:39:30.434 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-27T21:39:30.434 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:30.434 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:30.434 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-27T21:39:30.434 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:30.434 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:30.434 INFO:teuthology.misc.health.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:30.434 INFO:teuthology.misc.health.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:30.434 INFO:teuthology.misc.health.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:30.434 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-27T21:39:30.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 1 librados: starting msgr at 2024-03-27T21:39:30.435 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 1 librados: starting objecter 2024-03-27T21:39:30.435 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 1 Processor -- start 2024-03-27T21:39:30.435 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 1 -- start start 2024-03-27T21:39:30.435 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 1 librados: setting wanted keys 2024-03-27T21:39:30.435 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 1 librados: calling monclient init 2024-03-27T21:39:30.435 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 10 monclient: init 2024-03-27T21:39:30.435 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:30.435 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:39:30.435 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d6c0716e0 0x7f8d6c13d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:30.435 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 10 monclient(hunting): picked mon.noname-b con 0x7f8d6c0716e0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:30.435 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8d6c071ac0 0x7f8d6c13f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:30.435 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 10 monclient(hunting): picked mon.noname-c con 0x7f8d6c071ac0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:30.435 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d6c0ad490 0x7f8d6c0ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:30.435 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 10 monclient(hunting): picked mon.noname-a con 0x7f8d6c0ad490 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:30.435 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:30.436 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6c13ef90 con 0x7f8d6c0ad490 2024-03-27T21:39:30.436 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:30.436 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6c13db60 con 0x7f8d6c0716e0 2024-03-27T21:39:30.436 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:30.436 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6c13f880 con 0x7f8d6c071ac0 2024-03-27T21:39:30.436 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:30.436 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d70cbd640 10 monclient(hunting): authenticate will time out at 811.658755s 2024-03-27T21:39:30.436 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d69d9b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8d6c071ac0 0x7f8d6c13f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:30.436 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d6ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d6c0ad490 0x7f8d6c0ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:30.436 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d6ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d6c0ad490 0x7f8d6c0ac160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56220/0 (socket says 172.21.15.60:56220) 2024-03-27T21:39:30.436 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d69d9b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8d6c071ac0 0x7f8d6c13f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47804/0 (socket says 172.21.15.60:47804) 2024-03-27T21:39:30.436 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d6ad9d640 1 -- 172.21.15.60:0/3876984138 learned_addr learned my addr 172.21.15.60:0/3876984138 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:30.436 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d6ad9d640 10 monclient(hunting): get_auth_request con 0x7f8d6c0ad490 auth_method 0 2024-03-27T21:39:30.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d6ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:30.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d6ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:30.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.437+0000 7f8d6ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:30.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d69d9b640 10 monclient(hunting): get_auth_request con 0x7f8d6c071ac0 auth_method 0 2024-03-27T21:39:30.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d69d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:30.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d69d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:30.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d69d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:30.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d6a59c640 1 --2- 172.21.15.60:0/3876984138 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d6c0716e0 0x7f8d6c13d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:30.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d6a59c640 10 monclient(hunting): get_auth_request con 0x7f8d6c0716e0 auth_method 0 2024-03-27T21:39:30.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d6a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:30.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d6a59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:30.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d6a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:30.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d69d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:30.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d69d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:30.438 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d69d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:30.438 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d6ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:30.438 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d6ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:30.438 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d6ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:30.438 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d6a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:30.438 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d6a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:30.438 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d6a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:30.438 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d69d9b640 10 monclient(hunting): handle_auth_done global_id 4169 payload 824 2024-03-27T21:39:30.438 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d69d9b640 1 -- 172.21.15.60:0/3876984138 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d6c0716e0 msgr2=0x7f8d6c13d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:30.438 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d69d9b640 1 --2- 172.21.15.60:0/3876984138 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d6c0716e0 0x7f8d6c13d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:30.438 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d69d9b640 1 -- 172.21.15.60:0/3876984138 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d6c0ad490 msgr2=0x7f8d6c0ac160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:30.438 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d69d9b640 1 --2- 172.21.15.60:0/3876984138 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d6c0ad490 0x7f8d6c0ac160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:30.438 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d69d9b640 10 monclient: _finish_hunting 0 2024-03-27T21:39:30.439 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d69d9b640 1 monclient: found mon.noname-c 2024-03-27T21:39:30.439 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d69d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:30.439 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d69d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:39:30.439 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d69d9b640 1 -- 172.21.15.60:0/3876984138 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5c0094d0 con 0x7f8d6c071ac0 2024-03-27T21:39:30.439 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d6ad9d640 1 --2- 172.21.15.60:0/3876984138 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d6c0ad490 0x7f8d6c0ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:30.439 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d69d9b640 1 --2- 172.21.15.60:0/3876984138 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8d6c071ac0 0x7f8d6c13f340 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000e840 tx=0x7f8d6000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:30.439 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d5affd640 10 client.?.objecter ms_handle_connect 0x7f8d6c071ac0 2024-03-27T21:39:30.439 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d5affd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:30.439 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d5affd640 1 -- 172.21.15.60:0/3876984138 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d60009590 con 0x7f8d6c071ac0 2024-03-27T21:39:30.439 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:30.439 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d5affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:30.439 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d5affd640 10 monclient: dump: 2024-03-27T21:39:30.439 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-27T21:39:30.439 INFO:teuthology.misc.health.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:30.440 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:30.440 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:30.440 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:30.440 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:30.440 INFO:teuthology.misc.health.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:30.440 INFO:teuthology.misc.health.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:30.440 INFO:teuthology.misc.health.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:30.440 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-27T21:39:30.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d5affd640 10 monclient: _finish_auth 0 2024-03-27T21:39:30.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d5affd640 10 monclient: _check_auth_tickets 2024-03-27T21:39:30.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:30.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d70cbd640 5 monclient: authenticate success, global_id 4169 2024-03-27T21:39:30.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d5affd640 1 -- 172.21.15.60:0/3876984138 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6000c940 con 0x7f8d6c071ac0 2024-03-27T21:39:30.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d70cbd640 10 monclient: _renew_subs 2024-03-27T21:39:30.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d70cbd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:30.441 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d70cbd640 1 -- 172.21.15.60:0/3876984138 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d6c0acf60 con 0x7f8d6c071ac0 2024-03-27T21:39:30.441 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d5affd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:30.441 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d5affd640 1 -- 172.21.15.60:0/3876984138 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6000cd10 con 0x7f8d6c071ac0 2024-03-27T21:39:30.441 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:30.441 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d5affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:30.441 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d5affd640 10 monclient: dump: 2024-03-27T21:39:30.441 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-27T21:39:30.441 INFO:teuthology.misc.health.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:30.441 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:30.441 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:30.441 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:30.441 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:30.441 INFO:teuthology.misc.health.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:30.441 INFO:teuthology.misc.health.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:30.441 INFO:teuthology.misc.health.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:30.442 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-27T21:39:30.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d68d99640 4 set_mon_vals no callback set 2024-03-27T21:39:30.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d70cbd640 10 client.4169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:30.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d70cbd640 10 monclient: _renew_subs 2024-03-27T21:39:30.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d70cbd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:30.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d70cbd640 1 -- 172.21.15.60:0/3876984138 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d6c0ac6a0 con 0x7f8d6c071ac0 2024-03-27T21:39:30.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d70cbd640 1 librados: init done 2024-03-27T21:39:30.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d5affd640 1 -- 172.21.15.60:0/3876984138 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8d60004320 con 0x7f8d6c071ac0 2024-03-27T21:39:30.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d5affd640 1 -- 172.21.15.60:0/3876984138 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f8d60003ce0 con 0x7f8d6c071ac0 2024-03-27T21:39:30.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d5affd640 10 client.4169.objecter ms_dispatch 0x7f8d6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-27T21:39:30.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d5affd640 3 client.4169.objecter handle_osd_map got epochs [10,10] > 0 2024-03-27T21:39:30.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d5affd640 3 client.4169.objecter handle_osd_map decoding full epoch 10 2024-03-27T21:39:30.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.438+0000 7f8d5affd640 20 client.4169.objecter dump_active .. 0 homeless 2024-03-27T21:39:30.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.439+0000 7f8d70cbd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:30.443 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.439+0000 7f8d70cbd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:30.443 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.439+0000 7f8d70cbd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:30.443 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.439+0000 7f8d70cbd640 1 -- 172.21.15.60:0/3876984138 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d6c069470 con 0x7f8d6c071ac0 2024-03-27T21:39:30.443 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.441+0000 7f8d5affd640 1 -- 172.21.15.60:0/3876984138 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f8d60021060 con 0x7f8d6c071ac0 2024-03-27T21:39:30.443 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.441+0000 7f8d5affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:30.443 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.441+0000 7f8d5affd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:30.490 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.503+0000 7f8d70cbd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:39:30.490 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.503+0000 7f8d70cbd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:39:30.490 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.503+0000 7f8d70cbd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:30.490 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.503+0000 7f8d70cbd640 1 -- 172.21.15.60:0/3876984138 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d6c0b20b0 con 0x7f8d6c071ac0 2024-03-27T21:39:30.490 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.504+0000 7f8d5affd640 1 -- 172.21.15.60:0/3876984138 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f8d60017070 con 0x7f8d6c071ac0 2024-03-27T21:39:30.490 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.504+0000 7f8d5affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:39:30.491 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.504+0000 7f8d5affd640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:30.491 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-27T21:39:30.491 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"dcd19d9a-535c-4b41-9095-d588268e2860","created":"2024-03-27T21:39:27.092191+0000","modified":"2024-03-27T21:39:29.291620+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T21:39:29.291620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"778aaa64-6720-495b-bcdc-86fbcece9d58","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d960adf1-3d83-4e24-a612-8df53fa43d0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b8ffc48b-cf6f-4701-a6aa-c25fdb0f7977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0f14f2d8-b271-48e4-a198-aa5279a01c06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8cdeb7ae-1c6e-4d0d-ba44-80248eff2fb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93748cf8-d3fe-46b8-950f-e01e28a5e02c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1ad21434-cbc5-4de9-8d63-76f894f7e736","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9fbd0927-ee2f-471b-8df8-c4e0fc40b9b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T21:39:30.492 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.504+0000 7f8d70cbd640 10 librados: watch_flush enter 2024-03-27T21:39:30.492 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.504+0000 7f8d70cbd640 10 librados: watch_flush exit 2024-03-27T21:39:30.492 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.504+0000 7f8d70cbd640 20 client.4169.objecter shutdown clearing up homeless session... 2024-03-27T21:39:30.492 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.504+0000 7f8d70cbd640 10 client.4169.objecter successfully canceled tick 2024-03-27T21:39:30.493 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.504+0000 7f8d70cbd640 10 monclient: shutdown 2024-03-27T21:39:30.493 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.504+0000 7f8d70cbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:30.493 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.504+0000 7f8d70cbd640 1 -- 172.21.15.60:0/3876984138 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8d6c071ac0 msgr2=0x7f8d6c13f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:30.493 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.504+0000 7f8d70cbd640 1 --2- 172.21.15.60:0/3876984138 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8d6c071ac0 0x7f8d6c13f340 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000e840 tx=0x7f8d6000ed00 comp rx=0 tx=0).stop 2024-03-27T21:39:30.493 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.505+0000 7f8d70cbd640 1 -- 172.21.15.60:0/3876984138 shutdown_connections 2024-03-27T21:39:30.493 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.505+0000 7f8d70cbd640 1 --2- 172.21.15.60:0/3876984138 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d6c0ad490 0x7f8d6c0ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:30.493 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.505+0000 7f8d70cbd640 1 --2- 172.21.15.60:0/3876984138 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8d6c071ac0 0x7f8d6c13f340 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:30.493 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.505+0000 7f8d70cbd640 1 --2- 172.21.15.60:0/3876984138 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d6c0716e0 0x7f8d6c13d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:30.493 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.505+0000 7f8d70cbd640 1 -- 172.21.15.60:0/3876984138 >> 172.21.15.60:0/3876984138 conn(0x7f8d6c0b4020 msgr2=0x7f8d6c0b1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:30.493 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.505+0000 7f8d70cbd640 1 -- 172.21.15.60:0/3876984138 shutdown_connections 2024-03-27T21:39:30.493 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.505+0000 7f8d70cbd640 1 -- 172.21.15.60:0/3876984138 wait complete. 2024-03-27T21:39:30.493 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:30.505+0000 7f8d70cbd640 1 librados: shutdown 2024-03-27T21:39:30.500 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-27T21:39:30.898 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:30.912+0000 7f6438632040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-27T21:39:31.065 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:31.078+0000 7f4298a0e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-27T21:39:31.181 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:31.195+0000 7f6438632040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-27T21:39:31.234 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:31.248+0000 7f6438632040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-27T21:39:31.333 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:31.346+0000 7f4298a0e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-27T21:39:31.354 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:31.367+0000 7f6438632040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-27T21:39:31.385 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:31.398+0000 7f4298a0e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-27T21:39:31.500 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-27T21:39:31.513+0000 7f4298a0e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-27T21:39:32.380 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:32.393+0000 7f63d8d11640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-27T21:39:32.382 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:39:32.395+0000 7f63d8d11640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-27T21:39:33.359 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:39:33.373+0000 7f439fc0d640 -1 osd.1 0 log_to_monitors true 2024-03-27T21:39:33.399 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:39:33.412+0000 7f33e679e640 -1 osd.3 0 log_to_monitors true 2024-03-27T21:39:33.422 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:39:33.435+0000 7f912600f640 -1 osd.4 0 log_to_monitors true 2024-03-27T21:39:33.438 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:39:33.451+0000 7fa178d24640 -1 osd.0 0 log_to_monitors true 2024-03-27T21:39:33.438 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:39:33.451+0000 7f1294c1e640 -1 osd.5 0 log_to_monitors true 2024-03-27T21:39:33.443 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:39:33.457+0000 7f84b3e08640 -1 osd.6 0 log_to_monitors true 2024-03-27T21:39:33.446 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:39:33.459+0000 7f4920de1640 -1 osd.7 0 log_to_monitors true 2024-03-27T21:39:33.453 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:39:33.467+0000 7fd003723640 -1 osd.2 0 log_to_monitors true 2024-03-27T21:39:34.396 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:39:34.410+0000 7f439b531640 -1 osd.1 0 waiting for initial osdmap 2024-03-27T21:39:34.400 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:39:34.414+0000 7f439735a640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-27T21:39:35.392 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:39:35.405+0000 7fcfff6b0640 -1 osd.2 0 waiting for initial osdmap 2024-03-27T21:39:35.392 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:39:35.406+0000 7f84afd95640 -1 osd.6 0 waiting for initial osdmap 2024-03-27T21:39:35.395 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:39:35.409+0000 7fa174cb1640 -1 osd.0 0 waiting for initial osdmap 2024-03-27T21:39:35.396 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:39:35.409+0000 7f84abbbe640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-27T21:39:35.398 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:39:35.412+0000 7f9121a1e640 -1 osd.4 0 waiting for initial osdmap 2024-03-27T21:39:35.399 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:39:35.412+0000 7fa170ada640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-27T21:39:35.399 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:39:35.413+0000 7f491cd6e640 -1 osd.7 0 waiting for initial osdmap 2024-03-27T21:39:35.402 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:39:35.416+0000 7fcffb4d9640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-27T21:39:35.403 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:39:35.415+0000 7f911d847640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-27T21:39:35.403 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:39:35.417+0000 7f4918b97640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-27T21:39:37.307 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T21:39:37.400 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.413+0000 7f6aa9749640 10 monclient: get_monmap_and_config 2024-03-27T21:39:37.400 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.413+0000 7f6aa9749640 10 monclient: build_initial_monmap 2024-03-27T21:39:37.400 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.413+0000 7f6aa9749640 10 monclient: monmap: 2024-03-27T21:39:37.400 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-27T21:39:37.400 INFO:teuthology.misc.health.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:37.400 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-27T21:39:37.414520+0000 2024-03-27T21:39:37.400 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-27T21:39:37.414520+0000 2024-03-27T21:39:37.400 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:37.400 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:37.401 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:37.401 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:37.401 INFO:teuthology.misc.health.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:37.401 INFO:teuthology.misc.health.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:37.401 INFO:teuthology.misc.health.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:37.401 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-27T21:39:37.404 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.414+0000 7f6aa9749640 1 Processor -- start 2024-03-27T21:39:37.405 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.414+0000 7f6aa9749640 1 -- start start 2024-03-27T21:39:37.405 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.414+0000 7f6aa9749640 10 monclient: init 2024-03-27T21:39:37.405 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa9749640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:37.405 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa9749640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-03-27T21:39:37.405 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa9749640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6aa406b740 0x7f6aa406c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:37.405 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa9749640 10 monclient(hunting): picked mon.noname-c con 0x7f6aa406b740 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:37.405 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa9749640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6aa4070290 0x7f6aa413e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:37.405 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa9749640 10 monclient(hunting): picked mon.noname-b con 0x7f6aa4070290 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:37.405 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa9749640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6aa413ee10 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:37.405 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa9749640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aa406cee0 con 0x7f6aa413ee10 2024-03-27T21:39:37.405 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa9749640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:37.405 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa9749640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aa40b1810 con 0x7f6aa406b740 2024-03-27T21:39:37.405 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa9749640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:37.406 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa9749640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa4144a30 con 0x7f6aa4070290 2024-03-27T21:39:37.406 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa9749640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:37.406 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa9749640 10 monclient(hunting): authenticate will time out at 818.636312s 2024-03-27T21:39:37.406 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6a9a7fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6aa4070290 0x7f6aa413e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:37.406 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa2ffd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6aa406b740 0x7f6aa406c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:37.406 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6a9a7fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6aa4070290 0x7f6aa413e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:38726/0 (socket says 172.21.15.60:38726) 2024-03-27T21:39:37.406 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6a9a7fc640 1 -- 172.21.15.60:0/2870798599 learned_addr learned my addr 172.21.15.60:0/2870798599 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:37.406 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6a9a7fc640 10 monclient(hunting): get_auth_request con 0x7f6aa4070290 auth_method 0 2024-03-27T21:39:37.406 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6a9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:37.406 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6a9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:37.406 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6a9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:37.406 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa2ffd640 10 monclient(hunting): get_auth_request con 0x7f6aa406b740 auth_method 0 2024-03-27T21:39:37.406 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:37.406 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:37.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:37.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6a9a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:37.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6a9a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:37.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6a9a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:37.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:37.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:37.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:37.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa2ffd640 10 monclient(hunting): handle_auth_done global_id 4246 payload 274 2024-03-27T21:39:37.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa2ffd640 1 -- 172.21.15.60:0/2870798599 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6aa4070290 msgr2=0x7f6aa413e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:37.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa2ffd640 1 --2- 172.21.15.60:0/2870798599 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6aa4070290 0x7f6aa413e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:37.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa2ffd640 1 -- 172.21.15.60:0/2870798599 >> v1:172.21.15.191:6789/0 conn(0x7f6aa413ee10 legacy=0x7f6aa4144f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:37.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa2ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:39:37.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa2ffd640 1 monclient: found mon.noname-c 2024-03-27T21:39:37.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:37.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:39:37.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa2ffd640 1 -- 172.21.15.60:0/2870798599 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa4145660 con 0x7f6aa406b740 2024-03-27T21:39:37.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa27fc640 1 -- 172.21.15.60:0/2870798599 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5250 0 0) 0x7f6aa406cee0 con 0x7f6aa413ee10 2024-03-27T21:39:37.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6aa27fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:39:37.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.415+0000 7f6a9a7fc640 1 --2- 172.21.15.60:0/2870798599 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6aa4070290 0x7f6aa413e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:37.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.416+0000 7f6aa2ffd640 1 --2- 172.21.15.60:0/2870798599 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6aa406b740 0x7f6aa406c9a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6a940097a0 tx=0x7f6a9402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d6ff8db2eb75655 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:37.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.416+0000 7f6aa27fc640 1 -- 172.21.15.60:0/2870798599 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9402e070 con 0x7f6aa406b740 2024-03-27T21:39:37.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.416+0000 7f6aa27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:37.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.416+0000 7f6aa27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:37.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.416+0000 7f6aa27fc640 10 monclient: dump: 2024-03-27T21:39:37.408 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-27T21:39:37.408 INFO:teuthology.misc.health.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:37.408 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:37.408 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:37.408 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:37.409 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:37.409 INFO:teuthology.misc.health.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:37.409 INFO:teuthology.misc.health.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:37.409 INFO:teuthology.misc.health.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:37.409 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-27T21:39:37.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.416+0000 7f6aa27fc640 10 monclient: _finish_auth 0 2024-03-27T21:39:37.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.416+0000 7f6aa27fc640 10 monclient: _check_auth_tickets 2024-03-27T21:39:37.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.416+0000 7f6aa27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:37.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.416+0000 7f6aa27fc640 1 -- 172.21.15.60:0/2870798599 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a940315a0 con 0x7f6aa406b740 2024-03-27T21:39:37.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.416+0000 7f6aa9749640 5 monclient: authenticate success, global_id 4246 2024-03-27T21:39:37.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.416+0000 7f6aa9749640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:37.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.416+0000 7f6aa27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:37.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.416+0000 7f6aa9749640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:37.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.416+0000 7f6aa9749640 4 set_mon_vals no callback set 2024-03-27T21:39:37.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.416+0000 7f6aa27fc640 1 -- 172.21.15.60:0/2870798599 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a94031930 con 0x7f6aa406b740 2024-03-27T21:39:37.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.416+0000 7f6aa27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:37.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.416+0000 7f6aa27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:37.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.416+0000 7f6aa27fc640 10 monclient: dump: 2024-03-27T21:39:37.410 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-27T21:39:37.410 INFO:teuthology.misc.health.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:37.410 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:37.410 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:37.410 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:37.410 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:37.410 INFO:teuthology.misc.health.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:37.410 INFO:teuthology.misc.health.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:37.410 INFO:teuthology.misc.health.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:37.410 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-27T21:39:37.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.417+0000 7f6aa9749640 10 monclient: shutdown 2024-03-27T21:39:37.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.417+0000 7f6aa9749640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:37.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.417+0000 7f6aa9749640 1 -- 172.21.15.60:0/2870798599 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6aa406b740 msgr2=0x7f6aa406c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:37.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.417+0000 7f6aa9749640 1 --2- 172.21.15.60:0/2870798599 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6aa406b740 0x7f6aa406c9a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6a940097a0 tx=0x7f6a9402b840 comp rx=0 tx=0).stop 2024-03-27T21:39:37.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.417+0000 7f6aa9749640 1 -- 172.21.15.60:0/2870798599 shutdown_connections 2024-03-27T21:39:37.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.417+0000 7f6aa9749640 1 --2- 172.21.15.60:0/2870798599 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6aa4070290 0x7f6aa413e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:37.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.417+0000 7f6aa9749640 1 --2- 172.21.15.60:0/2870798599 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6aa406b740 0x7f6aa406c9a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:37.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.417+0000 7f6aa9749640 1 -- 172.21.15.60:0/2870798599 >> 172.21.15.60:0/2870798599 conn(0x7f6aa40b5180 msgr2=0x7f6aa40b5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:37.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.417+0000 7f6aa9749640 1 -- 172.21.15.60:0/2870798599 shutdown_connections 2024-03-27T21:39:37.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.417+0000 7f6aa9749640 1 -- 172.21.15.60:0/2870798599 wait complete. 2024-03-27T21:39:37.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.418+0000 7f6aa9749640 10 monclient: build_initial_monmap 2024-03-27T21:39:37.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.418+0000 7f6aa9749640 10 monclient: monmap: 2024-03-27T21:39:37.411 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-27T21:39:37.411 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:37.411 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:37.412 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-27T21:39:37.412 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:37.412 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:37.412 INFO:teuthology.misc.health.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:37.412 INFO:teuthology.misc.health.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:37.412 INFO:teuthology.misc.health.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:37.412 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-27T21:39:37.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.418+0000 7f6aa9749640 1 librados: starting msgr at 2024-03-27T21:39:37.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.418+0000 7f6aa9749640 1 librados: starting objecter 2024-03-27T21:39:37.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.418+0000 7f6aa9749640 1 Processor -- start 2024-03-27T21:39:37.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.418+0000 7f6aa9749640 1 -- start start 2024-03-27T21:39:37.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.418+0000 7f6aa9749640 1 librados: setting wanted keys 2024-03-27T21:39:37.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.418+0000 7f6aa9749640 1 librados: calling monclient init 2024-03-27T21:39:37.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.418+0000 7f6aa9749640 10 monclient: init 2024-03-27T21:39:37.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa9749640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:37.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa9749640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:39:37.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa9749640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aa406b740 0x7f6aa4143a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:37.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa9749640 10 monclient(hunting): picked mon.noname-a con 0x7f6aa406b740 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:37.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa9749640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6aa4070290 0x7f6aa40ae410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:37.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa9749640 10 monclient(hunting): picked mon.noname-c con 0x7f6aa4070290 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:37.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa9749640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6aa413ee10 0x7f6aa41330a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:37.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa9749640 10 monclient(hunting): picked mon.noname-b con 0x7f6aa413ee10 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:37.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa9749640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:37.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa9749640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa40add80 con 0x7f6aa406b740 2024-03-27T21:39:37.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa9749640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:37.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa9749640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa40aee20 con 0x7f6aa413ee10 2024-03-27T21:39:37.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa9749640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:37.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa9749640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa4144150 con 0x7f6aa4070290 2024-03-27T21:39:37.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa9749640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:37.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa9749640 10 monclient(hunting): authenticate will time out at 818.640412s 2024-03-27T21:39:37.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6a9a7fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6aa4070290 0x7f6aa40ae410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:37.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6a9a7fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6aa4070290 0x7f6aa40ae410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:38742/0 (socket says 172.21.15.60:38742) 2024-03-27T21:39:37.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6a9a7fc640 1 -- 172.21.15.60:0/568567424 learned_addr learned my addr 172.21.15.60:0/568567424 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:37.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6a9a7fc640 10 monclient(hunting): get_auth_request con 0x7f6aa4070290 auth_method 0 2024-03-27T21:39:37.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6a9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:37.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6a9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:37.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6a9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:37.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa2ffd640 1 --2- 172.21.15.60:0/568567424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aa406b740 0x7f6aa4143a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:37.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa37fe640 1 --2- 172.21.15.60:0/568567424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6aa413ee10 0x7f6aa41330a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:37.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa37fe640 10 monclient(hunting): get_auth_request con 0x7f6aa413ee10 auth_method 0 2024-03-27T21:39:37.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:37.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa37fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:37.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:37.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6a9a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:37.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6a9a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:37.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6a9a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:37.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa2ffd640 10 monclient(hunting): get_auth_request con 0x7f6aa406b740 auth_method 0 2024-03-27T21:39:37.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:37.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:37.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:37.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:37.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:37.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:37.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:37.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:37.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6aa2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:37.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6a9a7fc640 10 monclient(hunting): handle_auth_done global_id 4217 payload 824 2024-03-27T21:39:37.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.419+0000 7f6a9a7fc640 1 -- 172.21.15.60:0/568567424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6aa413ee10 msgr2=0x7f6aa41330a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:37.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9a7fc640 1 --2- 172.21.15.60:0/568567424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6aa413ee10 0x7f6aa41330a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:37.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9a7fc640 1 -- 172.21.15.60:0/568567424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aa406b740 msgr2=0x7f6aa4143a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:37.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9a7fc640 1 --2- 172.21.15.60:0/568567424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aa406b740 0x7f6aa4143a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:37.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9a7fc640 10 monclient: _finish_hunting 0 2024-03-27T21:39:37.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9a7fc640 1 monclient: found mon.noname-c 2024-03-27T21:39:37.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:37.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:39:37.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9a7fc640 1 -- 172.21.15.60:0/568567424 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8c0094d0 con 0x7f6aa4070290 2024-03-27T21:39:37.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6aa37fe640 1 --2- 172.21.15.60:0/568567424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6aa413ee10 0x7f6aa41330a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:37.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9a7fc640 1 --2- 172.21.15.60:0/568567424 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6aa4070290 0x7f6aa40ae410 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00ea00 tx=0x7f6a8c00eec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:37.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9bfff640 10 client.?.objecter ms_handle_connect 0x7f6aa4070290 2024-03-27T21:39:37.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9bfff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:37.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9bfff640 1 -- 172.21.15.60:0/568567424 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c00ca60 con 0x7f6aa4070290 2024-03-27T21:39:37.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:37.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:37.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9bfff640 10 monclient: dump: 2024-03-27T21:39:37.422 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-27T21:39:37.422 INFO:teuthology.misc.health.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:37.422 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:37.422 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:37.422 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:37.422 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:37.422 INFO:teuthology.misc.health.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:37.422 INFO:teuthology.misc.health.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:37.422 INFO:teuthology.misc.health.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:37.422 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-27T21:39:37.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9bfff640 10 monclient: _finish_auth 0 2024-03-27T21:39:37.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9bfff640 10 monclient: _check_auth_tickets 2024-03-27T21:39:37.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:37.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6aa9749640 5 monclient: authenticate success, global_id 4217 2024-03-27T21:39:37.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6aa9749640 10 monclient: _renew_subs 2024-03-27T21:39:37.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6aa9749640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:37.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9bfff640 1 -- 172.21.15.60:0/568567424 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a8c00cbc0 con 0x7f6aa4070290 2024-03-27T21:39:37.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6aa9749640 1 -- 172.21.15.60:0/568567424 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a94009400 con 0x7f6aa4070290 2024-03-27T21:39:37.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:37.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9bfff640 1 -- 172.21.15.60:0/568567424 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c013470 con 0x7f6aa4070290 2024-03-27T21:39:37.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:37.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:37.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6a9bfff640 10 monclient: dump: 2024-03-27T21:39:37.423 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-27T21:39:37.423 INFO:teuthology.misc.health.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:37.423 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:37.423 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:37.423 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:37.423 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:37.424 INFO:teuthology.misc.health.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:37.424 INFO:teuthology.misc.health.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:37.424 INFO:teuthology.misc.health.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:37.424 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-27T21:39:37.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6aa1ffb640 4 set_mon_vals no callback set 2024-03-27T21:39:37.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6aa9749640 10 client.4217.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:37.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6aa9749640 10 monclient: _renew_subs 2024-03-27T21:39:37.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6aa9749640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:37.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6aa9749640 1 -- 172.21.15.60:0/568567424 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aa41335e0 con 0x7f6aa4070290 2024-03-27T21:39:37.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.420+0000 7f6aa9749640 1 librados: init done 2024-03-27T21:39:37.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.421+0000 7f6aa9749640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:37.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.421+0000 7f6aa9749640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:37.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.421+0000 7f6aa9749640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:37.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.421+0000 7f6aa9749640 1 -- 172.21.15.60:0/568567424 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aa40aef90 con 0x7f6aa4070290 2024-03-27T21:39:37.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.421+0000 7f6a9bfff640 1 -- 172.21.15.60:0/568567424 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 4) v1 ==== 89081+0+0 (secure 0 0 0) 0x7f6a8c0135d0 con 0x7f6aa4070290 2024-03-27T21:39:37.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.422+0000 7f6a9bfff640 1 --2- 172.21.15.60:0/568567424 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6a8406c500 0x7f6a8406e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:37.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.422+0000 7f6a9bfff640 1 -- 172.21.15.60:0/568567424 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3911+0+0 (secure 0 0 0) 0x7f6a8c088df0 con 0x7f6aa4070290 2024-03-27T21:39:37.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.422+0000 7f6a9bfff640 10 client.4217.objecter ms_dispatch 0x7f6aa4000b90 osd_map(15..15 src has 1..15) v4 2024-03-27T21:39:37.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.422+0000 7f6aa2ffd640 1 --2- 172.21.15.60:0/568567424 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6a8406c500 0x7f6a8406e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:37.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.422+0000 7f6a9bfff640 3 client.4217.objecter handle_osd_map got epochs [15,15] > 0 2024-03-27T21:39:37.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.422+0000 7f6a9bfff640 3 client.4217.objecter handle_osd_map decoding full epoch 15 2024-03-27T21:39:37.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.422+0000 7f6aa2ffd640 10 monclient: get_auth_request con 0x7f6a8406c500 auth_method 0 2024-03-27T21:39:37.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.422+0000 7f6a9bfff640 20 client.4217.objecter dump_active .. 0 homeless 2024-03-27T21:39:37.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.422+0000 7f6aa2ffd640 1 --2- 172.21.15.60:0/568567424 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6a8406c500 0x7f6a8406e9c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6a94036000 tx=0x7f6a9403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:37.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.422+0000 7f6a9bfff640 10 client.4217.objecter ms_handle_connect 0x7f6a8406c500 2024-03-27T21:39:37.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.426+0000 7f6a9bfff640 1 -- 172.21.15.60:0/568567424 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6a8c054870 con 0x7f6aa4070290 2024-03-27T21:39:37.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.426+0000 7f6a9bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:37.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.426+0000 7f6a9bfff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:37.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.606+0000 7f6aa9749640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:39:37.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.606+0000 7f6aa9749640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:39:37.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.606+0000 7f6aa9749640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:37.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.606+0000 7f6aa9749640 1 -- 172.21.15.60:0/568567424 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6aa4069470 con 0x7f6aa4070290 2024-03-27T21:39:37.594 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.607+0000 7f6a9bfff640 1 -- 172.21.15.60:0/568567424 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+9715 (secure 0 0 0) 0x7f6a8c057e90 con 0x7f6aa4070290 2024-03-27T21:39:37.594 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.607+0000 7f6a9bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:39:37.594 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.607+0000 7f6a9bfff640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:37.594 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-27T21:39:37.594 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":15,"fsid":"dcd19d9a-535c-4b41-9095-d588268e2860","created":"2024-03-27T21:39:27.092191+0000","modified":"2024-03-27T21:39:37.403177+0000","last_up_change":"2024-03-27T21:39:36.401726+0000","last_in_change":"2024-03-27T21:39:29.291620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"778aaa64-6720-495b-bcdc-86fbcece9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":34603},{"type":"v1","addr":"172.21.15.60:6817","nonce":34603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":34603},{"type":"v1","addr":"172.21.15.60:6819","nonce":34603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":34603},{"type":"v1","addr":"172.21.15.60:6823","nonce":34603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":34603},{"type":"v1","addr":"172.21.15.60:6821","nonce":34603}]},"public_addr":"172.21.15.60:6817/34603","cluster_addr":"172.21.15.60:6819/34603","heartbeat_back_addr":"172.21.15.60:6823/34603","heartbeat_front_addr":"172.21.15.60:6821/34603","state":["exists","up"]},{"osd":1,"uuid":"d960adf1-3d83-4e24-a612-8df53fa43d0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":34602},{"type":"v1","addr":"172.21.15.60:6801","nonce":34602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":34602},{"type":"v1","addr":"172.21.15.60:6803","nonce":34602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":34602},{"type":"v1","addr":"172.21.15.60:6807","nonce":34602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":34602},{"type":"v1","addr":"172.21.15.60:6805","nonce":34602}]},"public_addr":"172.21.15.60:6801/34602","cluster_addr":"172.21.15.60:6803/34602","heartbeat_back_addr":"172.21.15.60:6807/34602","heartbeat_front_addr":"172.21.15.60:6805/34602","state":["exists","up"]},{"osd":2,"uuid":"b8ffc48b-cf6f-4701-a6aa-c25fdb0f7977","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":34604},{"type":"v1","addr":"172.21.15.60:6825","nonce":34604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":34604},{"type":"v1","addr":"172.21.15.60:6827","nonce":34604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":34604},{"type":"v1","addr":"172.21.15.60:6831","nonce":34604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":34604},{"type":"v1","addr":"172.21.15.60:6829","nonce":34604}]},"public_addr":"172.21.15.60:6825/34604","cluster_addr":"172.21.15.60:6827/34604","heartbeat_back_addr":"172.21.15.60:6831/34604","heartbeat_front_addr":"172.21.15.60:6829/34604","state":["exists","up"]},{"osd":3,"uuid":"0f14f2d8-b271-48e4-a198-aa5279a01c06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8cdeb7ae-1c6e-4d0d-ba44-80248eff2fb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":34332},{"type":"v1","addr":"172.21.15.191:6801","nonce":34332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":34332},{"type":"v1","addr":"172.21.15.191:6803","nonce":34332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":34332},{"type":"v1","addr":"172.21.15.191:6807","nonce":34332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":34332},{"type":"v1","addr":"172.21.15.191:6805","nonce":34332}]},"public_addr":"172.21.15.191:6801/34332","cluster_addr":"172.21.15.191:6803/34332","heartbeat_back_addr":"172.21.15.191:6807/34332","heartbeat_front_addr":"172.21.15.191:6805/34332","state":["exists","up"]},{"osd":5,"uuid":"93748cf8-d3fe-46b8-950f-e01e28a5e02c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1ad21434-cbc5-4de9-8d63-76f894f7e736","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":34361},{"type":"v1","addr":"172.21.15.191:6825","nonce":34361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":34361},{"type":"v1","addr":"172.21.15.191:6827","nonce":34361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":34361},{"type":"v1","addr":"172.21.15.191:6831","nonce":34361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":34361},{"type":"v1","addr":"172.21.15.191:6829","nonce":34361}]},"public_addr":"172.21.15.191:6825/34361","cluster_addr":"172.21.15.191:6827/34361","heartbeat_back_addr":"172.21.15.191:6831/34361","heartbeat_front_addr":"172.21.15.191:6829/34361","state":["exists","up"]},{"osd":7,"uuid":"9fbd0927-ee2f-471b-8df8-c4e0fc40b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":34334},{"type":"v1","addr":"172.21.15.191:6810","nonce":34334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":34334},{"type":"v1","addr":"172.21.15.191:6814","nonce":34334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6819","nonce":34334},{"type":"v1","addr":"172.21.15.191:6821","nonce":34334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6815","nonce":34334},{"type":"v1","addr":"172.21.15.191:6817","nonce":34334}]},"public_addr":"172.21.15.191:6810/34334","cluster_addr":"172.21.15.191:6814/34334","heartbeat_back_addr":"172.21.15.191:6821/34334","heartbeat_front_addr":"172.21.15.191:6817/34334","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.476097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.411243+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.458154+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.431201+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.414076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T21:39:37.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.609+0000 7f6aa9749640 10 librados: watch_flush enter 2024-03-27T21:39:37.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.609+0000 7f6aa9749640 10 librados: watch_flush exit 2024-03-27T21:39:37.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.609+0000 7f6aa9749640 20 client.4217.objecter shutdown clearing up homeless session... 2024-03-27T21:39:37.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.609+0000 7f6aa9749640 10 client.4217.objecter successfully canceled tick 2024-03-27T21:39:37.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.609+0000 7f6aa9749640 1 -- 172.21.15.60:0/568567424 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6a8406c500 msgr2=0x7f6a8406e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:37.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.609+0000 7f6aa9749640 1 --2- 172.21.15.60:0/568567424 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6a8406c500 0x7f6a8406e9c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6a94036000 tx=0x7f6a9403d040 comp rx=0 tx=0).stop 2024-03-27T21:39:37.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.609+0000 7f6aa9749640 10 monclient: shutdown 2024-03-27T21:39:37.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.609+0000 7f6aa9749640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:37.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.609+0000 7f6aa9749640 1 -- 172.21.15.60:0/568567424 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6aa4070290 msgr2=0x7f6aa40ae410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:37.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.609+0000 7f6aa9749640 1 --2- 172.21.15.60:0/568567424 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6aa4070290 0x7f6aa40ae410 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00ea00 tx=0x7f6a8c00eec0 comp rx=0 tx=0).stop 2024-03-27T21:39:37.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.609+0000 7f6aa9749640 1 -- 172.21.15.60:0/568567424 shutdown_connections 2024-03-27T21:39:37.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.609+0000 7f6aa9749640 1 --2- 172.21.15.60:0/568567424 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6a8406c500 0x7f6a8406e9c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:37.598 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.609+0000 7f6aa9749640 1 --2- 172.21.15.60:0/568567424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6aa413ee10 0x7f6aa41330a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:37.598 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.609+0000 7f6aa9749640 1 --2- 172.21.15.60:0/568567424 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6aa4070290 0x7f6aa40ae410 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:37.598 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.609+0000 7f6aa9749640 1 --2- 172.21.15.60:0/568567424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aa406b740 0x7f6aa4143a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:37.598 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.609+0000 7f6aa9749640 1 -- 172.21.15.60:0/568567424 >> 172.21.15.60:0/568567424 conn(0x7f6aa40b5180 msgr2=0x7f6aa41369d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:37.598 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.610+0000 7f6aa9749640 1 -- 172.21.15.60:0/568567424 shutdown_connections 2024-03-27T21:39:37.598 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.610+0000 7f6aa9749640 1 -- 172.21.15.60:0/568567424 wait complete. 2024-03-27T21:39:37.598 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:37.610+0000 7f6aa9749640 1 librados: shutdown 2024-03-27T21:39:37.606 DEBUG:teuthology.misc:6 of 8 OSDs are up 2024-03-27T21:39:38.040 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:39:38.053+0000 7f12904ee640 -1 osd.5 0 waiting for initial osdmap 2024-03-27T21:39:38.054 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:39:38.067+0000 7f128830f640 -1 osd.5 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-27T21:39:38.222 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:39:38.235+0000 7f33e272b640 -1 osd.3 0 waiting for initial osdmap 2024-03-27T21:39:38.236 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:39:38.250+0000 7f33da54c640 -1 osd.3 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-27T21:39:44.414 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T21:39:44.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.527+0000 7ff3f5639640 10 monclient: get_monmap_and_config 2024-03-27T21:39:44.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.527+0000 7ff3f5639640 10 monclient: build_initial_monmap 2024-03-27T21:39:44.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.528+0000 7ff3f5639640 10 monclient: monmap: 2024-03-27T21:39:44.515 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-27T21:39:44.515 INFO:teuthology.misc.health.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:44.515 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-27T21:39:44.528977+0000 2024-03-27T21:39:44.515 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-27T21:39:44.528977+0000 2024-03-27T21:39:44.515 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:44.515 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:44.515 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:44.515 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:44.515 INFO:teuthology.misc.health.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:44.515 INFO:teuthology.misc.health.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:44.516 INFO:teuthology.misc.health.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:44.516 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-27T21:39:44.523 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.528+0000 7ff3f5639640 1 Processor -- start 2024-03-27T21:39:44.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.528+0000 7ff3f5639640 1 -- start start 2024-03-27T21:39:44.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.528+0000 7ff3f5639640 10 monclient: init 2024-03-27T21:39:44.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3f5639640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:44.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3f5639640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-03-27T21:39:44.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3f5639640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff3f00a8a70 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:44.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3f5639640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff3f0142290 0x7ff3f0141610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:44.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3f5639640 10 monclient(hunting): picked mon.noname-b con 0x7ff3f0142290 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:44.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3f5639640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff3f00a94a0 0x7ff3f00b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:44.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3f5639640 10 monclient(hunting): picked mon.noname-c con 0x7ff3f00a94a0 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:44.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3f5639640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f0141e00 con 0x7ff3f00a8a70 2024-03-27T21:39:44.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3f5639640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:44.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3f5639640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3f00a8e50 con 0x7ff3f00a94a0 2024-03-27T21:39:44.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3f5639640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:44.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3f5639640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f0141b50 con 0x7ff3f0142290 2024-03-27T21:39:44.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3f5639640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:44.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3f5639640 10 monclient(hunting): authenticate will time out at 825.750171s 2024-03-27T21:39:44.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ee7fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff3f0142290 0x7ff3f0141610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:44.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ee7fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff3f0142290 0x7ff3f0141610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47408/0 (socket says 172.21.15.60:47408) 2024-03-27T21:39:44.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ee7fc640 1 -- 172.21.15.60:0/101008069 learned_addr learned my addr 172.21.15.60:0/101008069 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:44.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ee7fc640 10 monclient(hunting): get_auth_request con 0x7ff3f0142290 auth_method 0 2024-03-27T21:39:44.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:44.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:44.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:44.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:44.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:44.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:44.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ef7fe640 1 --2- 172.21.15.60:0/101008069 >> v2:172.21.15.191:3300/0 conn(0x7ff3f00a94a0 0x7ff3f00b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:44.526 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ef7fe640 10 monclient(hunting): get_auth_request con 0x7ff3f00a94a0 auth_method 0 2024-03-27T21:39:44.526 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:44.526 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:44.526 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:44.526 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ee7fc640 10 monclient(hunting): handle_auth_done global_id 4229 payload 274 2024-03-27T21:39:44.526 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ee7fc640 1 -- 172.21.15.60:0/101008069 >> v2:172.21.15.191:3300/0 conn(0x7ff3f00a94a0 msgr2=0x7ff3f00b29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:44.526 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ee7fc640 1 --2- 172.21.15.60:0/101008069 >> v2:172.21.15.191:3300/0 conn(0x7ff3f00a94a0 0x7ff3f00b29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:44.526 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ee7fc640 1 -- 172.21.15.60:0/101008069 >> v1:172.21.15.191:6789/0 conn(0x7ff3f00a8a70 legacy=0x7ff3f00b1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:44.526 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ee7fc640 10 monclient: _finish_hunting 0 2024-03-27T21:39:44.526 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ee7fc640 1 monclient: found mon.noname-b 2024-03-27T21:39:44.526 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:44.526 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ee7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:44.527 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.529+0000 7ff3ee7fc640 1 -- 172.21.15.60:0/101008069 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f00b4b50 con 0x7ff3f0142290 2024-03-27T21:39:44.527 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.530+0000 7ff3ee7fc640 1 --2- 172.21.15.60:0/101008069 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff3f0142290 0x7ff3f0141610 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8009870 tx=0x7ff3d802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c3021a2c7228ddf server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:44.527 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.530+0000 7ff3edffb640 1 -- 172.21.15.60:0/101008069 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d80317c0 con 0x7ff3f0142290 2024-03-27T21:39:44.527 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.530+0000 7ff3edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:44.527 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.530+0000 7ff3edffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:44.527 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.530+0000 7ff3edffb640 10 monclient: dump: 2024-03-27T21:39:44.527 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-27T21:39:44.527 INFO:teuthology.misc.health.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:44.527 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:44.527 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:44.527 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:44.527 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:44.528 INFO:teuthology.misc.health.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:44.528 INFO:teuthology.misc.health.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:44.528 INFO:teuthology.misc.health.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:44.528 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-27T21:39:44.528 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.530+0000 7ff3edffb640 10 monclient: _finish_auth 0 2024-03-27T21:39:44.528 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.530+0000 7ff3edffb640 10 monclient: _check_auth_tickets 2024-03-27T21:39:44.528 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.530+0000 7ff3edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:44.528 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.530+0000 7ff3edffb640 1 -- 172.21.15.60:0/101008069 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d8031920 con 0x7ff3f0142290 2024-03-27T21:39:44.528 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.530+0000 7ff3f5639640 5 monclient: authenticate success, global_id 4229 2024-03-27T21:39:44.528 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.530+0000 7ff3f5639640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:44.528 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.530+0000 7ff3edffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:44.528 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.530+0000 7ff3edffb640 1 -- 172.21.15.60:0/101008069 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8031cb0 con 0x7ff3f0142290 2024-03-27T21:39:44.528 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.530+0000 7ff3f5639640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:44.528 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.530+0000 7ff3f5639640 4 set_mon_vals no callback set 2024-03-27T21:39:44.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.530+0000 7ff3edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:44.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.530+0000 7ff3edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:44.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.530+0000 7ff3edffb640 10 monclient: dump: 2024-03-27T21:39:44.529 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-27T21:39:44.529 INFO:teuthology.misc.health.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:44.529 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:44.529 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:44.529 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:44.529 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:44.529 INFO:teuthology.misc.health.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:44.529 INFO:teuthology.misc.health.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:44.530 INFO:teuthology.misc.health.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:44.530 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-27T21:39:44.530 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.531+0000 7ff3f5639640 10 monclient: shutdown 2024-03-27T21:39:44.530 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.531+0000 7ff3f5639640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:44.530 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.531+0000 7ff3f5639640 1 -- 172.21.15.60:0/101008069 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff3f0142290 msgr2=0x7ff3f0141610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:44.530 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.531+0000 7ff3f5639640 1 --2- 172.21.15.60:0/101008069 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff3f0142290 0x7ff3f0141610 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8009870 tx=0x7ff3d802b860 comp rx=0 tx=0).stop 2024-03-27T21:39:44.530 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.531+0000 7ff3f5639640 1 -- 172.21.15.60:0/101008069 shutdown_connections 2024-03-27T21:39:44.530 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.531+0000 7ff3f5639640 1 --2- 172.21.15.60:0/101008069 >> v2:172.21.15.191:3300/0 conn(0x7ff3f00a94a0 0x7ff3f00b29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:44.530 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.531+0000 7ff3f5639640 1 --2- 172.21.15.60:0/101008069 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff3f0142290 0x7ff3f0141610 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:44.530 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.531+0000 7ff3f5639640 1 -- 172.21.15.60:0/101008069 >> 172.21.15.60:0/101008069 conn(0x7ff3f00b3c60 msgr2=0x7ff3f00b4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:44.531 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.531+0000 7ff3f5639640 1 -- 172.21.15.60:0/101008069 shutdown_connections 2024-03-27T21:39:44.531 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.531+0000 7ff3f5639640 1 -- 172.21.15.60:0/101008069 wait complete. 2024-03-27T21:39:44.531 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.531+0000 7ff3f5639640 10 monclient: build_initial_monmap 2024-03-27T21:39:44.531 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.531+0000 7ff3f5639640 10 monclient: monmap: 2024-03-27T21:39:44.531 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-27T21:39:44.531 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:44.531 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:44.531 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-27T21:39:44.531 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:44.531 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:44.531 INFO:teuthology.misc.health.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:44.531 INFO:teuthology.misc.health.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:44.532 INFO:teuthology.misc.health.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:44.532 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-27T21:39:44.532 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.532+0000 7ff3f5639640 1 librados: starting msgr at 2024-03-27T21:39:44.532 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.532+0000 7ff3f5639640 1 librados: starting objecter 2024-03-27T21:39:44.532 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.532+0000 7ff3f5639640 1 Processor -- start 2024-03-27T21:39:44.532 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.532+0000 7ff3f5639640 1 -- start start 2024-03-27T21:39:44.532 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.532+0000 7ff3f5639640 1 librados: setting wanted keys 2024-03-27T21:39:44.532 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.532+0000 7ff3f5639640 1 librados: calling monclient init 2024-03-27T21:39:44.532 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.532+0000 7ff3f5639640 10 monclient: init 2024-03-27T21:39:44.533 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3f5639640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:44.533 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3f5639640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:39:44.533 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3f5639640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3f00a8a70 0x7ff3f00af9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:44.533 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3f5639640 10 monclient(hunting): picked mon.noname-b con 0x7ff3f00a8a70 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:44.533 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3f5639640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3f00a94a0 0x7ff3f01d6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:44.533 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3f5639640 10 monclient(hunting): picked mon.noname-a con 0x7ff3f00a94a0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:44.533 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3f5639640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff3f0142290 0x7ff3f00add20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:44.533 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3f5639640 10 monclient(hunting): picked mon.noname-c con 0x7ff3f0142290 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:44.533 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3f5639640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:44.533 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3f5639640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f01d6580 con 0x7ff3f00a94a0 2024-03-27T21:39:44.533 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3f5639640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:44.533 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3f5639640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f00af2f0 con 0x7ff3f00a8a70 2024-03-27T21:39:44.533 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3f5639640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:44.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3f5639640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f00acf00 con 0x7ff3f0142290 2024-03-27T21:39:44.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3f5639640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:44.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3f5639640 10 monclient(hunting): authenticate will time out at 825.754079s 2024-03-27T21:39:44.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ee7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3f00a94a0 0x7ff3f01d6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:44.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ef7fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff3f0142290 0x7ff3f00add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:44.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ee7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3f00a94a0 0x7ff3f01d6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47614/0 (socket says 172.21.15.60:47614) 2024-03-27T21:39:44.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ee7fc640 1 -- 172.21.15.60:0/3694950172 learned_addr learned my addr 172.21.15.60:0/3694950172 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:44.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ee7fc640 10 monclient(hunting): get_auth_request con 0x7ff3f00a94a0 auth_method 0 2024-03-27T21:39:44.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:44.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:44.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:44.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ef7fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff3f0142290 0x7ff3f00add20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47414/0 (socket says 172.21.15.60:47414) 2024-03-27T21:39:44.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ef7fe640 10 monclient(hunting): get_auth_request con 0x7ff3f0142290 auth_method 0 2024-03-27T21:39:44.535 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:44.535 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:44.535 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:44.535 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:44.535 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:44.535 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3eeffd640 1 --2- 172.21.15.60:0/3694950172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3f00a8a70 0x7ff3f00af9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:44.535 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:44.535 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:44.535 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:44.535 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:44.535 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3eeffd640 10 monclient(hunting): get_auth_request con 0x7ff3f00a8a70 auth_method 0 2024-03-27T21:39:44.535 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:44.535 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:44.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:44.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:44.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:44.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:44.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ef7fe640 10 monclient(hunting): handle_auth_done global_id 4232 payload 824 2024-03-27T21:39:44.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ef7fe640 1 -- 172.21.15.60:0/3694950172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3f00a8a70 msgr2=0x7ff3f00af9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:44.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ef7fe640 1 --2- 172.21.15.60:0/3694950172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3f00a8a70 0x7ff3f00af9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:44.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ef7fe640 1 -- 172.21.15.60:0/3694950172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3f00a94a0 msgr2=0x7ff3f01d6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:44.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ef7fe640 1 --2- 172.21.15.60:0/3694950172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3f00a94a0 0x7ff3f01d6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:44.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ef7fe640 10 monclient: _finish_hunting 0 2024-03-27T21:39:44.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ef7fe640 1 monclient: found mon.noname-c 2024-03-27T21:39:44.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:44.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ef7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:39:44.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ef7fe640 1 -- 172.21.15.60:0/3694950172 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d80094d0 con 0x7ff3f0142290 2024-03-27T21:39:44.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ee7fc640 1 --2- 172.21.15.60:0/3694950172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3f00a94a0 0x7ff3f01d6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:44.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3ef7fe640 1 --2- 172.21.15.60:0/3694950172 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff3f0142290 0x7ff3f00add20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0007d90 tx=0x7ff3e000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:44.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3e77fe640 10 client.?.objecter ms_handle_connect 0x7ff3f0142290 2024-03-27T21:39:44.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3e77fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:44.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.533+0000 7ff3e77fe640 1 -- 172.21.15.60:0/3694950172 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0018070 con 0x7ff3f0142290 2024-03-27T21:39:44.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:44.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3e77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:44.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3e77fe640 10 monclient: dump: 2024-03-27T21:39:44.537 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-27T21:39:44.537 INFO:teuthology.misc.health.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:44.537 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:44.537 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:44.537 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:44.538 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:44.538 INFO:teuthology.misc.health.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:44.538 INFO:teuthology.misc.health.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:44.538 INFO:teuthology.misc.health.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:44.538 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-27T21:39:44.538 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3e77fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:44.538 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3e77fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:44.538 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:44.538 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3e77fe640 1 -- 172.21.15.60:0/3694950172 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e000edd0 con 0x7ff3f0142290 2024-03-27T21:39:44.538 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3f5639640 5 monclient: authenticate success, global_id 4232 2024-03-27T21:39:44.538 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3f5639640 10 monclient: _renew_subs 2024-03-27T21:39:44.538 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3f5639640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:44.538 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3f5639640 1 -- 172.21.15.60:0/3694950172 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f00ae260 con 0x7ff3f0142290 2024-03-27T21:39:44.539 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:44.539 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3e77fe640 1 -- 172.21.15.60:0/3694950172 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0003bb0 con 0x7ff3f0142290 2024-03-27T21:39:44.539 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:44.539 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:44.539 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3e77fe640 10 monclient: dump: 2024-03-27T21:39:44.539 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-27T21:39:44.539 INFO:teuthology.misc.health.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:44.539 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:44.539 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:44.539 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:44.539 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:44.539 INFO:teuthology.misc.health.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:44.539 INFO:teuthology.misc.health.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:44.540 INFO:teuthology.misc.health.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:44.540 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-27T21:39:44.540 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3ed7fa640 4 set_mon_vals no callback set 2024-03-27T21:39:44.540 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3f5639640 10 client.4232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:44.540 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3f5639640 10 monclient: _renew_subs 2024-03-27T21:39:44.540 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3f5639640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:44.540 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3f5639640 1 -- 172.21.15.60:0/3694950172 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f01d6f50 con 0x7ff3f0142290 2024-03-27T21:39:44.540 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.534+0000 7ff3f5639640 1 librados: init done 2024-03-27T21:39:44.540 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.535+0000 7ff3f5639640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:44.540 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.535+0000 7ff3f5639640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:44.540 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.535+0000 7ff3f5639640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:44.540 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.535+0000 7ff3f5639640 1 -- 172.21.15.60:0/3694950172 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f00ad180 con 0x7ff3f0142290 2024-03-27T21:39:44.540 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.535+0000 7ff3e77fe640 1 -- 172.21.15.60:0/3694950172 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff3e0014420 con 0x7ff3f0142290 2024-03-27T21:39:44.540 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.535+0000 7ff3e77fe640 1 --2- 172.21.15.60:0/3694950172 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff3d006c5c0 0x7ff3d006ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:44.540 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.535+0000 7ff3e77fe640 1 -- 172.21.15.60:0/3694950172 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff3e0089110 con 0x7ff3f0142290 2024-03-27T21:39:44.541 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.535+0000 7ff3e77fe640 10 client.4232.objecter ms_dispatch 0x7ff3f0000b90 osd_map(18..18 src has 1..18) v4 2024-03-27T21:39:44.541 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.535+0000 7ff3e77fe640 3 client.4232.objecter handle_osd_map got epochs [18,18] > 0 2024-03-27T21:39:44.541 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.535+0000 7ff3e77fe640 3 client.4232.objecter handle_osd_map decoding full epoch 18 2024-03-27T21:39:44.541 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.535+0000 7ff3eeffd640 1 --2- 172.21.15.60:0/3694950172 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff3d006c5c0 0x7ff3d006ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:44.541 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.536+0000 7ff3eeffd640 10 monclient: get_auth_request con 0x7ff3d006c5c0 auth_method 0 2024-03-27T21:39:44.541 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.536+0000 7ff3e77fe640 20 client.4232.objecter dump_active .. 0 homeless 2024-03-27T21:39:44.541 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.536+0000 7ff3e77fe640 10 client.4232.objecter ms_handle_connect 0x7ff3d006c5c0 2024-03-27T21:39:44.541 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.536+0000 7ff3eeffd640 1 --2- 172.21.15.60:0/3694950172 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff3d006c5c0 0x7ff3d006ea80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8000f60 tx=0x7ff3e8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:44.542 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.540+0000 7ff3e77fe640 1 -- 172.21.15.60:0/3694950172 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff3e00546f0 con 0x7ff3f0142290 2024-03-27T21:39:44.542 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.540+0000 7ff3e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:44.542 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.540+0000 7ff3e77fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:44.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.719+0000 7ff3f5639640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:39:44.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.719+0000 7ff3f5639640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:39:44.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.719+0000 7ff3f5639640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:44.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.719+0000 7ff3f5639640 1 -- 172.21.15.60:0/3694950172 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3f01d66f0 con 0x7ff3f0142290 2024-03-27T21:39:44.707 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.720+0000 7ff3e77fe640 1 -- 172.21.15.60:0/3694950172 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12382 (secure 0 0 0) 0x7ff3e0057d10 con 0x7ff3f0142290 2024-03-27T21:39:44.707 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.720+0000 7ff3e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:39:44.707 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.720+0000 7ff3e77fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:44.707 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-27T21:39:44.707 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":18,"fsid":"dcd19d9a-535c-4b41-9095-d588268e2860","created":"2024-03-27T21:39:27.092191+0000","modified":"2024-03-27T21:39:40.419630+0000","last_up_change":"2024-03-27T21:39:38.411432+0000","last_in_change":"2024-03-27T21:39:29.291620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T21:39:37.438493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"778aaa64-6720-495b-bcdc-86fbcece9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":34603},{"type":"v1","addr":"172.21.15.60:6817","nonce":34603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":34603},{"type":"v1","addr":"172.21.15.60:6819","nonce":34603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":34603},{"type":"v1","addr":"172.21.15.60:6823","nonce":34603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":34603},{"type":"v1","addr":"172.21.15.60:6821","nonce":34603}]},"public_addr":"172.21.15.60:6817/34603","cluster_addr":"172.21.15.60:6819/34603","heartbeat_back_addr":"172.21.15.60:6823/34603","heartbeat_front_addr":"172.21.15.60:6821/34603","state":["exists","up"]},{"osd":1,"uuid":"d960adf1-3d83-4e24-a612-8df53fa43d0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":34602},{"type":"v1","addr":"172.21.15.60:6801","nonce":34602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":34602},{"type":"v1","addr":"172.21.15.60:6803","nonce":34602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":34602},{"type":"v1","addr":"172.21.15.60:6807","nonce":34602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":34602},{"type":"v1","addr":"172.21.15.60:6805","nonce":34602}]},"public_addr":"172.21.15.60:6801/34602","cluster_addr":"172.21.15.60:6803/34602","heartbeat_back_addr":"172.21.15.60:6807/34602","heartbeat_front_addr":"172.21.15.60:6805/34602","state":["exists","up"]},{"osd":2,"uuid":"b8ffc48b-cf6f-4701-a6aa-c25fdb0f7977","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":34604},{"type":"v1","addr":"172.21.15.60:6825","nonce":34604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":34604},{"type":"v1","addr":"172.21.15.60:6827","nonce":34604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":34604},{"type":"v1","addr":"172.21.15.60:6831","nonce":34604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":34604},{"type":"v1","addr":"172.21.15.60:6829","nonce":34604}]},"public_addr":"172.21.15.60:6825/34604","cluster_addr":"172.21.15.60:6827/34604","heartbeat_back_addr":"172.21.15.60:6831/34604","heartbeat_front_addr":"172.21.15.60:6829/34604","state":["exists","up"]},{"osd":3,"uuid":"0f14f2d8-b271-48e4-a198-aa5279a01c06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":34613},{"type":"v1","addr":"172.21.15.60:6809","nonce":34613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":34613},{"type":"v1","addr":"172.21.15.60:6811","nonce":34613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":34613},{"type":"v1","addr":"172.21.15.60:6815","nonce":34613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":34613},{"type":"v1","addr":"172.21.15.60:6813","nonce":34613}]},"public_addr":"172.21.15.60:6809/34613","cluster_addr":"172.21.15.60:6811/34613","heartbeat_back_addr":"172.21.15.60:6815/34613","heartbeat_front_addr":"172.21.15.60:6813/34613","state":["exists","up"]},{"osd":4,"uuid":"8cdeb7ae-1c6e-4d0d-ba44-80248eff2fb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":34332},{"type":"v1","addr":"172.21.15.191:6801","nonce":34332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":34332},{"type":"v1","addr":"172.21.15.191:6803","nonce":34332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":34332},{"type":"v1","addr":"172.21.15.191:6807","nonce":34332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":34332},{"type":"v1","addr":"172.21.15.191:6805","nonce":34332}]},"public_addr":"172.21.15.191:6801/34332","cluster_addr":"172.21.15.191:6803/34332","heartbeat_back_addr":"172.21.15.191:6807/34332","heartbeat_front_addr":"172.21.15.191:6805/34332","state":["exists","up"]},{"osd":5,"uuid":"93748cf8-d3fe-46b8-950f-e01e28a5e02c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6809","nonce":34333},{"type":"v1","addr":"172.21.15.191:6811","nonce":34333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6813","nonce":34333},{"type":"v1","addr":"172.21.15.191:6816","nonce":34333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":34333},{"type":"v1","addr":"172.21.15.191:6823","nonce":34333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":34333},{"type":"v1","addr":"172.21.15.191:6820","nonce":34333}]},"public_addr":"172.21.15.191:6811/34333","cluster_addr":"172.21.15.191:6816/34333","heartbeat_back_addr":"172.21.15.191:6823/34333","heartbeat_front_addr":"172.21.15.191:6820/34333","state":["exists","up"]},{"osd":6,"uuid":"1ad21434-cbc5-4de9-8d63-76f894f7e736","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":34361},{"type":"v1","addr":"172.21.15.191:6825","nonce":34361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":34361},{"type":"v1","addr":"172.21.15.191:6827","nonce":34361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":34361},{"type":"v1","addr":"172.21.15.191:6831","nonce":34361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":34361},{"type":"v1","addr":"172.21.15.191:6829","nonce":34361}]},"public_addr":"172.21.15.191:6825/34361","cluster_addr":"172.21.15.191:6827/34361","heartbeat_back_addr":"172.21.15.191:6831/34361","heartbeat_front_addr":"172.21.15.191:6829/34361","state":["exists","up"]},{"osd":7,"uuid":"9fbd0927-ee2f-471b-8df8-c4e0fc40b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":34334},{"type":"v1","addr":"172.21.15.191:6810","nonce":34334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":34334},{"type":"v1","addr":"172.21.15.191:6814","nonce":34334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6819","nonce":34334},{"type":"v1","addr":"172.21.15.191:6821","nonce":34334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6815","nonce":34334},{"type":"v1","addr":"172.21.15.191:6817","nonce":34334}]},"public_addr":"172.21.15.191:6810/34334","cluster_addr":"172.21.15.191:6814/34334","heartbeat_back_addr":"172.21.15.191:6821/34334","heartbeat_front_addr":"172.21.15.191:6817/34334","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.476097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.411243+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.458154+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.415285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.431201+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.456013+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.474448+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.414076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T21:39:44.709 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.722+0000 7ff3f5639640 10 librados: watch_flush enter 2024-03-27T21:39:44.709 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.722+0000 7ff3f5639640 10 librados: watch_flush exit 2024-03-27T21:39:44.709 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.722+0000 7ff3f5639640 20 client.4232.objecter shutdown clearing up homeless session... 2024-03-27T21:39:44.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.722+0000 7ff3f5639640 10 client.4232.objecter successfully canceled tick 2024-03-27T21:39:44.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.722+0000 7ff3f5639640 1 -- 172.21.15.60:0/3694950172 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff3d006c5c0 msgr2=0x7ff3d006ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:44.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.722+0000 7ff3f5639640 1 --2- 172.21.15.60:0/3694950172 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff3d006c5c0 0x7ff3d006ea80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8000f60 tx=0x7ff3e8009040 comp rx=0 tx=0).stop 2024-03-27T21:39:44.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.722+0000 7ff3f5639640 10 monclient: shutdown 2024-03-27T21:39:44.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.722+0000 7ff3f5639640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:44.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.722+0000 7ff3f5639640 1 -- 172.21.15.60:0/3694950172 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff3f0142290 msgr2=0x7ff3f00add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:44.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.722+0000 7ff3f5639640 1 --2- 172.21.15.60:0/3694950172 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff3f0142290 0x7ff3f00add20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0007d90 tx=0x7ff3e000e4b0 comp rx=0 tx=0).stop 2024-03-27T21:39:44.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.722+0000 7ff3f5639640 1 -- 172.21.15.60:0/3694950172 shutdown_connections 2024-03-27T21:39:44.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.722+0000 7ff3f5639640 1 --2- 172.21.15.60:0/3694950172 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff3f0142290 0x7ff3f00add20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:44.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.722+0000 7ff3f5639640 1 --2- 172.21.15.60:0/3694950172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3f00a94a0 0x7ff3f01d6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:44.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.722+0000 7ff3f5639640 1 --2- 172.21.15.60:0/3694950172 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff3d006c5c0 0x7ff3d006ea80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:44.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.722+0000 7ff3f5639640 1 --2- 172.21.15.60:0/3694950172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3f00a8a70 0x7ff3f00af9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:44.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.722+0000 7ff3f5639640 1 -- 172.21.15.60:0/3694950172 >> 172.21.15.60:0/3694950172 conn(0x7ff3f00b3c60 msgr2=0x7ff3f013aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:44.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.722+0000 7ff3f5639640 1 -- 172.21.15.60:0/3694950172 shutdown_connections 2024-03-27T21:39:44.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.722+0000 7ff3f5639640 1 -- 172.21.15.60:0/3694950172 wait complete. 2024-03-27T21:39:44.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-27T21:39:44.722+0000 7ff3f5639640 1 librados: shutdown 2024-03-27T21:39:44.718 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-27T21:39:44.718 INFO:tasks.ceph:Creating RBD pool 2024-03-27T21:39:44.719 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd pool create rbd 8 2024-03-27T21:39:44.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.832+0000 7f6741284640 10 monclient: get_monmap_and_config 2024-03-27T21:39:44.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.832+0000 7f6741284640 10 monclient: build_initial_monmap 2024-03-27T21:39:44.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.832+0000 7f6741284640 10 monclient: monmap: 2024-03-27T21:39:44.819 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:44.819 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:44.819 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:44.833158+0000 2024-03-27T21:39:44.819 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:44.833158+0000 2024-03-27T21:39:44.819 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:44.819 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:44.819 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:44.819 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:44.819 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:44.819 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:44.819 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:44.819 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.832+0000 7f6741284640 1 Processor -- start 2024-03-27T21:39:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.832+0000 7f6741284640 1 -- start start 2024-03-27T21:39:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.832+0000 7f6741284640 10 monclient: init 2024-03-27T21:39:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f6741284640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f6741284640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-03-27T21:39:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f6741284640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f673c0b40b0 0x7f673c0a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f6741284640 10 monclient(hunting): picked mon.noname-b con 0x7f673c0b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f6741284640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f673c13b020 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f6741284640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f673c13b480 0x7f673c141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f6741284640 10 monclient(hunting): picked mon.noname-a con 0x7f673c13b480 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f6741284640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673c0b3830 con 0x7f673c13b020 2024-03-27T21:39:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f6741284640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f6741284640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f673c13b910 con 0x7f673c13b480 2024-03-27T21:39:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f6741284640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f6741284640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673c0b4490 con 0x7f673c0b40b0 2024-03-27T21:39:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f6741284640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f6741284640 10 monclient(hunting): authenticate will time out at 826.054345s 2024-03-27T21:39:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673affd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f673c0b40b0 0x7f673c0a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673a7fc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f673c13b020 0x7f673c1416f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43480/0 (socket says 172.21.15.60:43480) 2024-03-27T21:39:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673a7fc640 1 -- 172.21.15.60:0/3393893858 learned_addr learned my addr 172.21.15.60:0/3393893858 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:39:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673b7fe640 1 --2- 172.21.15.60:0/3393893858 >> v2:172.21.15.60:3300/0 conn(0x7f673c13b480 0x7f673c141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673affd640 10 monclient(hunting): get_auth_request con 0x7f673c0b40b0 auth_method 0 2024-03-27T21:39:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673affd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673b7fe640 10 monclient(hunting): get_auth_request con 0x7f673c13b480 auth_method 0 2024-03-27T21:39:44.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:44.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:44.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:44.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:44.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:44.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:44.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:44.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:44.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:44.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673b7fe640 10 monclient(hunting): handle_auth_done global_id 4248 payload 274 2024-03-27T21:39:44.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673b7fe640 1 -- 172.21.15.60:0/3393893858 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f673c0b40b0 msgr2=0x7f673c0a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:44.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673b7fe640 1 --2- 172.21.15.60:0/3393893858 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f673c0b40b0 0x7f673c0a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:44.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673b7fe640 1 -- 172.21.15.60:0/3393893858 >> v1:172.21.15.60:6789/0 conn(0x7f673c13b020 legacy=0x7f673c1416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:44.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673b7fe640 10 monclient: _finish_hunting 0 2024-03-27T21:39:44.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673b7fe640 1 monclient: found mon.noname-a 2024-03-27T21:39:44.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:44.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:44.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673b7fe640 1 -- 172.21.15.60:0/3393893858 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673c0aa520 con 0x7f673c13b480 2024-03-27T21:39:44.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f6739ffb640 1 -- 172.21.15.60:0/3393893858 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584591990 0 0) 0x7f673c0b3830 con 0x7f673c13b020 2024-03-27T21:39:44.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f6739ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:39:44.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.833+0000 7f673affd640 1 --2- 172.21.15.60:0/3393893858 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f673c0b40b0 0x7f673c0a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:44.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.834+0000 7f673b7fe640 1 --2- 172.21.15.60:0/3393893858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f673c13b480 0x7f673c141e00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f672c009dc0 tx=0x7f672c02be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18489f92aea6b40e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:44.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.834+0000 7f6739ffb640 1 -- 172.21.15.60:0/3393893858 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c02cbb0 con 0x7f673c13b480 2024-03-27T21:39:44.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.834+0000 7f6739ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:44.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.834+0000 7f6739ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:44.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.834+0000 7f6739ffb640 10 monclient: dump: 2024-03-27T21:39:44.830 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:44.830 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:44.830 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:44.831 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:44.831 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:44.831 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:44.831 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:44.831 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:44.831 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:44.831 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:44.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.834+0000 7f6739ffb640 10 monclient: _finish_auth 0 2024-03-27T21:39:44.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.834+0000 7f6739ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:39:44.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.834+0000 7f6739ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:44.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.834+0000 7f6739ffb640 1 -- 172.21.15.60:0/3393893858 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672c02cd10 con 0x7f673c13b480 2024-03-27T21:39:44.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.834+0000 7f6739ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:44.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.834+0000 7f6739ffb640 1 -- 172.21.15.60:0/3393893858 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c034660 con 0x7f673c13b480 2024-03-27T21:39:44.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.834+0000 7f6741284640 5 monclient: authenticate success, global_id 4248 2024-03-27T21:39:44.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.834+0000 7f6741284640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.834+0000 7f6741284640 4 set_mon_vals no callback set 2024-03-27T21:39:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.834+0000 7f6739ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.834+0000 7f6739ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.834+0000 7f6739ffb640 10 monclient: dump: 2024-03-27T21:39:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:44.832 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.835+0000 7f6741284640 10 monclient: shutdown 2024-03-27T21:39:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.835+0000 7f6741284640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.835+0000 7f6741284640 1 -- 172.21.15.60:0/3393893858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f673c13b480 msgr2=0x7f673c141e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.835+0000 7f6741284640 1 --2- 172.21.15.60:0/3393893858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f673c13b480 0x7f673c141e00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f672c009dc0 tx=0x7f672c02be00 comp rx=0 tx=0).stop 2024-03-27T21:39:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.835+0000 7f6741284640 1 -- 172.21.15.60:0/3393893858 shutdown_connections 2024-03-27T21:39:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.835+0000 7f6741284640 1 --2- 172.21.15.60:0/3393893858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f673c13b480 0x7f673c141e00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.835+0000 7f6741284640 1 --2- 172.21.15.60:0/3393893858 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f673c0b40b0 0x7f673c0a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.835+0000 7f6741284640 1 -- 172.21.15.60:0/3393893858 >> 172.21.15.60:0/3393893858 conn(0x7f673c0608e0 msgr2=0x7f673c060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.835+0000 7f6741284640 1 -- 172.21.15.60:0/3393893858 shutdown_connections 2024-03-27T21:39:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.835+0000 7f6741284640 1 -- 172.21.15.60:0/3393893858 wait complete. 2024-03-27T21:39:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.835+0000 7f6741284640 10 monclient: build_initial_monmap 2024-03-27T21:39:44.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.835+0000 7f6741284640 10 monclient: monmap: 2024-03-27T21:39:44.834 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:44.834 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:44.834 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:44.834 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:44.834 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:44.834 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:44.834 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:44.834 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:44.834 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:44.834 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:44.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.836+0000 7f6741284640 1 librados: starting msgr at 2024-03-27T21:39:44.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.836+0000 7f6741284640 1 librados: starting objecter 2024-03-27T21:39:44.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.836+0000 7f6741284640 1 Processor -- start 2024-03-27T21:39:44.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.836+0000 7f6741284640 1 -- start start 2024-03-27T21:39:44.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.836+0000 7f6741284640 1 librados: setting wanted keys 2024-03-27T21:39:44.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.836+0000 7f6741284640 1 librados: calling monclient init 2024-03-27T21:39:44.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.836+0000 7f6741284640 10 monclient: init 2024-03-27T21:39:44.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:44.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:39:44.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f673c0b40b0 0x7f673c139b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:44.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 10 monclient(hunting): picked mon.noname-a con 0x7f673c0b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:44.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f673c13b020 0x7f673c13d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:44.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 10 monclient(hunting): picked mon.noname-b con 0x7f673c13b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:44.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f673c13b480 0x7f673c1d6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:44.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 10 monclient(hunting): picked mon.noname-c con 0x7f673c13b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:44.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:44.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c13de30 con 0x7f673c0b40b0 2024-03-27T21:39:44.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:44.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c13a640 con 0x7f673c13b020 2024-03-27T21:39:44.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:44.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673c13d840 con 0x7f673c13b480 2024-03-27T21:39:44.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:44.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 10 monclient(hunting): authenticate will time out at 826.058077s 2024-03-27T21:39:44.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f673c0b40b0 0x7f673c139b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:44.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f673c0b40b0 0x7f673c139b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47630/0 (socket says 172.21.15.60:47630) 2024-03-27T21:39:44.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673affd640 1 -- 172.21.15.60:0/1729326442 learned_addr learned my addr 172.21.15.60:0/1729326442 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:44.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673affd640 10 monclient(hunting): get_auth_request con 0x7f673c0b40b0 auth_method 0 2024-03-27T21:39:44.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:44.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673affd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:44.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:44.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673b7fe640 1 --2- 172.21.15.60:0/1729326442 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f673c13b480 0x7f673c1d6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:44.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673b7fe640 10 monclient(hunting): get_auth_request con 0x7f673c13b480 auth_method 0 2024-03-27T21:39:44.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:44.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:44.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:44.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:44.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:44.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:44.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:44.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:44.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:44.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673a7fc640 1 --2- 172.21.15.60:0/1729326442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f673c13b020 0x7f673c13d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:44.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673a7fc640 10 monclient(hunting): get_auth_request con 0x7f673c13b020 auth_method 0 2024-03-27T21:39:44.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:44.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:44.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:44.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673affd640 10 monclient(hunting): handle_auth_done global_id 4254 payload 824 2024-03-27T21:39:44.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673affd640 1 -- 172.21.15.60:0/1729326442 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f673c13b480 msgr2=0x7f673c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:44.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673affd640 1 --2- 172.21.15.60:0/1729326442 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f673c13b480 0x7f673c1d6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:44.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673affd640 1 -- 172.21.15.60:0/1729326442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f673c13b020 msgr2=0x7f673c13d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:44.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673affd640 1 --2- 172.21.15.60:0/1729326442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f673c13b020 0x7f673c13d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:44.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673affd640 10 monclient: _finish_hunting 0 2024-03-27T21:39:44.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673affd640 1 monclient: found mon.noname-a 2024-03-27T21:39:44.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:44.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:44.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673affd640 1 -- 172.21.15.60:0/1729326442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6734009400 con 0x7f673c0b40b0 2024-03-27T21:39:44.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673b7fe640 1 --2- 172.21.15.60:0/1729326442 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f673c13b480 0x7f673c1d6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:44.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f673affd640 1 --2- 172.21.15.60:0/1729326442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f673c0b40b0 0x7f673c139b30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f673400ebe0 tx=0x7f673400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:44.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f67337fe640 10 client.?.objecter ms_handle_connect 0x7f673c0b40b0 2024-03-27T21:39:44.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f67337fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:44.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f67337fe640 1 -- 172.21.15.60:0/1729326442 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6734009760 con 0x7f673c0b40b0 2024-03-27T21:39:44.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f67337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:44.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f67337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:44.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f67337fe640 10 monclient: dump: 2024-03-27T21:39:44.839 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:44.839 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:44.839 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:44.839 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:44.839 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:44.839 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:44.839 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:44.839 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:44.839 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:44.840 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:44.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f67337fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:44.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f67337fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:44.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f67337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:44.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f67337fe640 1 -- 172.21.15.60:0/1729326442 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f673400cc30 con 0x7f673c0b40b0 2024-03-27T21:39:44.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 5 monclient: authenticate success, global_id 4254 2024-03-27T21:39:44.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 10 monclient: _renew_subs 2024-03-27T21:39:44.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:44.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 1 -- 172.21.15.60:0/1729326442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f672c009a70 con 0x7f673c0b40b0 2024-03-27T21:39:44.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f67337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:44.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f67337fe640 1 -- 172.21.15.60:0/1729326442 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673400eee0 con 0x7f673c0b40b0 2024-03-27T21:39:44.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f67337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:44.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f67337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:44.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f67337fe640 10 monclient: dump: 2024-03-27T21:39:44.841 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:44.841 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:44.841 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:44.841 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:44.841 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:44.841 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:44.841 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:44.841 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:44.841 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:44.841 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:44.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f67397fa640 4 set_mon_vals no callback set 2024-03-27T21:39:44.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:44.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 10 monclient: _renew_subs 2024-03-27T21:39:44.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:44.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 1 -- 172.21.15.60:0/1729326442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f673c1d69b0 con 0x7f673c0b40b0 2024-03-27T21:39:44.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.837+0000 7f6741284640 1 librados: init done 2024-03-27T21:39:44.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.838+0000 7f6741284640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:44.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.838+0000 7f6741284640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:44.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.838+0000 7f6741284640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:44.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.838+0000 7f6741284640 1 -- 172.21.15.60:0/1729326442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673c13dc00 con 0x7f673c0b40b0 2024-03-27T21:39:44.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.839+0000 7f67337fe640 1 -- 172.21.15.60:0/1729326442 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6734013470 con 0x7f673c0b40b0 2024-03-27T21:39:44.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.839+0000 7f67337fe640 1 --2- 172.21.15.60:0/1729326442 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f671c06c590 0x7f671c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:44.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.839+0000 7f67337fe640 1 -- 172.21.15.60:0/1729326442 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6734089190 con 0x7f673c0b40b0 2024-03-27T21:39:44.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.839+0000 7f67337fe640 10 client.4254.objecter ms_dispatch 0x7f673c000b90 osd_map(18..18 src has 1..18) v4 2024-03-27T21:39:44.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.839+0000 7f67337fe640 3 client.4254.objecter handle_osd_map got epochs [18,18] > 0 2024-03-27T21:39:44.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.840+0000 7f67337fe640 3 client.4254.objecter handle_osd_map decoding full epoch 18 2024-03-27T21:39:44.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.839+0000 7f673a7fc640 1 --2- 172.21.15.60:0/1729326442 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f671c06c590 0x7f671c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:44.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.840+0000 7f673a7fc640 10 monclient: get_auth_request con 0x7f671c06c590 auth_method 0 2024-03-27T21:39:44.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.840+0000 7f67337fe640 20 client.4254.objecter dump_active .. 0 homeless 2024-03-27T21:39:44.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.840+0000 7f67337fe640 10 client.4254.objecter ms_handle_connect 0x7f671c06c590 2024-03-27T21:39:44.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.840+0000 7f673a7fc640 1 --2- 172.21.15.60:0/1729326442 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f671c06c590 0x7f671c06ea50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f673c13cc00 tx=0x7f6724007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:44.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.844+0000 7f67337fe640 1 -- 172.21.15.60:0/1729326442 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f673c13dc00 con 0x7f673c0b40b0 2024-03-27T21:39:44.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.844+0000 7f67337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:44.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:44.844+0000 7f67337fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:45.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.022+0000 7f6741284640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-03-27T21:39:45.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.022+0000 7f6741284640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-03-27T21:39:45.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.022+0000 7f6741284640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:45.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.022+0000 7f6741284640 1 -- 172.21.15.60:0/1729326442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f673c13dbd0 con 0x7f673c0b40b0 2024-03-27T21:39:45.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.452+0000 7f67337fe640 1 -- 172.21.15.60:0/1729326442 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v19) v1 ==== 109+0+0 (secure 0 0 0) 0x7f6734058290 con 0x7f673c0b40b0 2024-03-27T21:39:45.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.452+0000 7f67337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-03-27T21:39:45.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.452+0000 7f67337fe640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-03-27T21:39:45.439 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'rbd' created 2024-03-27T21:39:45.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.454+0000 7f6741284640 10 librados: watch_flush enter 2024-03-27T21:39:45.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.454+0000 7f6741284640 10 librados: watch_flush exit 2024-03-27T21:39:45.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.454+0000 7f6741284640 20 client.4254.objecter shutdown clearing up homeless session... 2024-03-27T21:39:45.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.454+0000 7f6741284640 10 client.4254.objecter successfully canceled tick 2024-03-27T21:39:45.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.454+0000 7f6741284640 1 -- 172.21.15.60:0/1729326442 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f671c06c590 msgr2=0x7f671c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:45.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.454+0000 7f6741284640 1 --2- 172.21.15.60:0/1729326442 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f671c06c590 0x7f671c06ea50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f673c13cc00 tx=0x7f6724007b50 comp rx=0 tx=0).stop 2024-03-27T21:39:45.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.455+0000 7f6741284640 10 monclient: shutdown 2024-03-27T21:39:45.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.455+0000 7f6741284640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:45.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.455+0000 7f6741284640 1 -- 172.21.15.60:0/1729326442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f673c0b40b0 msgr2=0x7f673c139b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:45.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.455+0000 7f6741284640 1 --2- 172.21.15.60:0/1729326442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f673c0b40b0 0x7f673c139b30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f673400ebe0 tx=0x7f673400c520 comp rx=0 tx=0).stop 2024-03-27T21:39:45.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.455+0000 7f6741284640 1 -- 172.21.15.60:0/1729326442 shutdown_connections 2024-03-27T21:39:45.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.455+0000 7f6741284640 1 --2- 172.21.15.60:0/1729326442 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f673c13b480 0x7f673c1d6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:45.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.455+0000 7f6741284640 1 --2- 172.21.15.60:0/1729326442 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f671c06c590 0x7f671c06ea50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:45.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.455+0000 7f6741284640 1 --2- 172.21.15.60:0/1729326442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f673c13b020 0x7f673c13d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:45.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.455+0000 7f6741284640 1 --2- 172.21.15.60:0/1729326442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f673c0b40b0 0x7f673c139b30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:45.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.455+0000 7f6741284640 1 -- 172.21.15.60:0/1729326442 >> 172.21.15.60:0/1729326442 conn(0x7f673c0608e0 msgr2=0x7f673c13fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:45.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.455+0000 7f6741284640 1 -- 172.21.15.60:0/1729326442 shutdown_connections 2024-03-27T21:39:45.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.455+0000 7f6741284640 1 -- 172.21.15.60:0/1729326442 wait complete. 2024-03-27T21:39:45.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.455+0000 7f6741284640 1 librados: shutdown 2024-03-27T21:39:45.454 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-27T21:39:45.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.570+0000 7fc95a8b6640 10 monclient: get_monmap_and_config 2024-03-27T21:39:45.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.570+0000 7fc95a8b6640 10 monclient: build_initial_monmap 2024-03-27T21:39:45.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.570+0000 7fc95a8b6640 10 monclient: monmap: 2024-03-27T21:39:45.557 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:45.558 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:45.558 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:45.571841+0000 2024-03-27T21:39:45.558 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:45.571841+0000 2024-03-27T21:39:45.558 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:45.558 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:45.558 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:45.558 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:45.558 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:45.558 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:45.558 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:45.558 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:45.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.571+0000 7fc95a8b6640 1 Processor -- start 2024-03-27T21:39:45.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.571+0000 7fc95a8b6640 1 -- start start 2024-03-27T21:39:45.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.571+0000 7fc95a8b6640 10 monclient: init 2024-03-27T21:39:45.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc95a8b6640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:45.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc95a8b6640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-03-27T21:39:45.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc95a8b6640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc9540b3980 0x7fc9541449c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:45.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc95a8b6640 10 monclient(hunting): picked mon.noname-a con 0x7fc9540b3980 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:45.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc95a8b6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc9540b3d60 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:45.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc95a8b6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc9540a86e0 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:45.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc95a8b6640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc954144f00 con 0x7fc9540b3d60 2024-03-27T21:39:45.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc95a8b6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9540b07f0 con 0x7fc9540a86e0 2024-03-27T21:39:45.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc95a8b6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:45.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc95a8b6640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9540aa1d0 con 0x7fc9540b3980 2024-03-27T21:39:45.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc95a8b6640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:45.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc95a8b6640 10 monclient(hunting): authenticate will time out at 826.793071s 2024-03-27T21:39:45.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc953fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc9540b3980 0x7fc9541449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:45.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc9537fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc9540b3d60 0x7fc9540b0c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43492/0 (socket says 172.21.15.60:43492) 2024-03-27T21:39:45.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc9537fe640 1 -- 172.21.15.60:0/4009890028 learned_addr learned my addr 172.21.15.60:0/4009890028 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:39:45.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc953fff640 10 monclient(hunting): get_auth_request con 0x7fc9540b3980 auth_method 0 2024-03-27T21:39:45.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc953fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:45.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc953fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:45.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc953fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:45.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc953fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:45.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc953fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:45.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc953fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:45.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc953fff640 10 monclient(hunting): handle_auth_done global_id 4257 payload 274 2024-03-27T21:39:45.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc953fff640 1 -- 172.21.15.60:0/4009890028 >> v1:172.21.15.191:6789/0 conn(0x7fc9540a86e0 legacy=0x7fc9540b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:45.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc953fff640 1 -- 172.21.15.60:0/4009890028 >> v1:172.21.15.60:6789/0 conn(0x7fc9540b3d60 legacy=0x7fc9540b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:45.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc953fff640 10 monclient: _finish_hunting 0 2024-03-27T21:39:45.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc952ffd640 1 -- 172.21.15.60:0/4009890028 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763725144 0 0) 0x7fc954144f00 con 0x7fc9540b3d60 2024-03-27T21:39:45.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc953fff640 1 monclient: found mon.noname-a 2024-03-27T21:39:45.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc953fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:45.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc953fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:45.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc953fff640 1 -- 172.21.15.60:0/4009890028 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9540aa4b0 con 0x7fc9540b3980 2024-03-27T21:39:45.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.572+0000 7fc952ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:39:45.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.573+0000 7fc953fff640 1 --2- 172.21.15.60:0/4009890028 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9540b3980 0x7fc9541449c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc9440097a0 tx=0x7fc94402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=420f02a4c6b23b68 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:45.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.573+0000 7fc952ffd640 1 -- 172.21.15.60:0/4009890028 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc944030610 con 0x7fc9540b3980 2024-03-27T21:39:45.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.573+0000 7fc952ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:45.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.573+0000 7fc952ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:45.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.573+0000 7fc952ffd640 10 monclient: dump: 2024-03-27T21:39:45.571 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:45.571 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:45.571 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:45.571 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:45.571 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:45.571 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:45.571 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:45.571 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:45.571 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:45.571 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:45.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.573+0000 7fc952ffd640 10 monclient: _finish_auth 0 2024-03-27T21:39:45.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.573+0000 7fc952ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:39:45.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.573+0000 7fc952ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:45.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.573+0000 7fc952ffd640 1 -- 172.21.15.60:0/4009890028 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc944030770 con 0x7fc9540b3980 2024-03-27T21:39:45.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.573+0000 7fc95a8b6640 5 monclient: authenticate success, global_id 4257 2024-03-27T21:39:45.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.573+0000 7fc95a8b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:45.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.573+0000 7fc952ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:45.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.573+0000 7fc95a8b6640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:45.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.573+0000 7fc952ffd640 1 -- 172.21.15.60:0/4009890028 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc944030b00 con 0x7fc9540b3980 2024-03-27T21:39:45.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.573+0000 7fc952ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:45.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.573+0000 7fc95a8b6640 4 set_mon_vals no callback set 2024-03-27T21:39:45.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.573+0000 7fc952ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:45.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.573+0000 7fc952ffd640 10 monclient: dump: 2024-03-27T21:39:45.572 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:45.572 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:45.572 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:45.572 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:45.572 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:45.572 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:45.572 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:45.573 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:45.573 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:45.573 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:45.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.574+0000 7fc95a8b6640 10 monclient: shutdown 2024-03-27T21:39:45.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.574+0000 7fc95a8b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:45.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.574+0000 7fc95a8b6640 1 -- 172.21.15.60:0/4009890028 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9540b3980 msgr2=0x7fc9541449c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:45.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.574+0000 7fc95a8b6640 1 --2- 172.21.15.60:0/4009890028 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9540b3980 0x7fc9541449c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc9440097a0 tx=0x7fc94402b960 comp rx=0 tx=0).stop 2024-03-27T21:39:45.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.574+0000 7fc95a8b6640 1 -- 172.21.15.60:0/4009890028 shutdown_connections 2024-03-27T21:39:45.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.574+0000 7fc95a8b6640 1 --2- 172.21.15.60:0/4009890028 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9540b3980 0x7fc9541449c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:45.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.574+0000 7fc95a8b6640 1 -- 172.21.15.60:0/4009890028 >> 172.21.15.60:0/4009890028 conn(0x7fc954071410 msgr2=0x7fc954071820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:45.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.574+0000 7fc95a8b6640 1 -- 172.21.15.60:0/4009890028 shutdown_connections 2024-03-27T21:39:45.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.574+0000 7fc95a8b6640 1 -- 172.21.15.60:0/4009890028 wait complete. 2024-03-27T21:39:45.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.574+0000 7fc95a8b6640 10 monclient: build_initial_monmap 2024-03-27T21:39:45.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.574+0000 7fc95a8b6640 10 monclient: monmap: 2024-03-27T21:39:45.574 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:45.574 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:45.574 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:45.574 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:45.574 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:45.574 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:45.574 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:45.574 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:45.574 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:45.574 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:45.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.574+0000 7fc95a8b6640 1 librados: starting msgr at 2024-03-27T21:39:45.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.574+0000 7fc95a8b6640 1 librados: starting objecter 2024-03-27T21:39:45.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc95a8b6640 1 Processor -- start 2024-03-27T21:39:45.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc95a8b6640 1 -- start start 2024-03-27T21:39:45.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc95a8b6640 1 librados: setting wanted keys 2024-03-27T21:39:45.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc95a8b6640 1 librados: calling monclient init 2024-03-27T21:39:45.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc95a8b6640 10 monclient: init 2024-03-27T21:39:45.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc95a8b6640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:45.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc95a8b6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:39:45.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc95a8b6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9540a86e0 0x7fc9541d64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:45.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc95a8b6640 10 monclient(hunting): picked mon.noname-b con 0x7fc9540a86e0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:45.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc95a8b6640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc9540b3980 0x7fc9540ab280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:45.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc95a8b6640 10 monclient(hunting): picked mon.noname-c con 0x7fc9540b3980 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:45.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc95a8b6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9541d6f00 0x7fc9540ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:45.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc95a8b6640 10 monclient(hunting): picked mon.noname-a con 0x7fc9541d6f00 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:45.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc95a8b6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:45.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc95a8b6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9540abd00 con 0x7fc9541d6f00 2024-03-27T21:39:45.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc95a8b6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:45.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc95a8b6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9540aca40 con 0x7fc9540a86e0 2024-03-27T21:39:45.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc95a8b6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:45.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc95a8b6640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9540aad60 con 0x7fc9540b3980 2024-03-27T21:39:45.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc95a8b6640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:45.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc95a8b6640 10 monclient(hunting): authenticate will time out at 826.796662s 2024-03-27T21:39:45.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc9537fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc9540b3980 0x7fc9540ab280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:45.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc958e53640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9541d6f00 0x7fc9540ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:45.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc9537fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc9540b3980 0x7fc9540ab280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47446/0 (socket says 172.21.15.60:47446) 2024-03-27T21:39:45.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc958e53640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9541d6f00 0x7fc9540ab7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47646/0 (socket says 172.21.15.60:47646) 2024-03-27T21:39:45.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc9537fe640 1 -- 172.21.15.60:0/658793967 learned_addr learned my addr 172.21.15.60:0/658793967 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:45.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc9537fe640 10 monclient(hunting): get_auth_request con 0x7fc9540b3980 auth_method 0 2024-03-27T21:39:45.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc9537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:45.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc9537fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:45.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc9537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:45.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc958e53640 10 monclient(hunting): get_auth_request con 0x7fc9541d6f00 auth_method 0 2024-03-27T21:39:45.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc958e53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:45.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc958e53640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:45.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc958e53640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:45.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc9537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:45.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc9537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:45.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc9537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:45.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc958e53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:45.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc958e53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:45.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.575+0000 7fc953fff640 1 --2- 172.21.15.60:0/658793967 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9540a86e0 0x7fc9541d64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:45.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc958e53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:45.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc953fff640 10 monclient(hunting): get_auth_request con 0x7fc9540a86e0 auth_method 0 2024-03-27T21:39:45.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc953fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:45.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc953fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:45.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc953fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:45.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc953fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:45.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc953fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:45.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc953fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:45.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc9537fe640 10 monclient(hunting): handle_auth_done global_id 4241 payload 824 2024-03-27T21:39:45.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc9537fe640 1 -- 172.21.15.60:0/658793967 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9540a86e0 msgr2=0x7fc9541d64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:45.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc9537fe640 1 --2- 172.21.15.60:0/658793967 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9540a86e0 0x7fc9541d64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:45.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc9537fe640 1 -- 172.21.15.60:0/658793967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9541d6f00 msgr2=0x7fc9540ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:45.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc9537fe640 1 --2- 172.21.15.60:0/658793967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9541d6f00 0x7fc9540ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:45.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc9537fe640 10 monclient: _finish_hunting 0 2024-03-27T21:39:45.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc9537fe640 1 monclient: found mon.noname-c 2024-03-27T21:39:45.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc9537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:45.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc9537fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:39:45.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc9537fe640 1 -- 172.21.15.60:0/658793967 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc944009400 con 0x7fc9540b3980 2024-03-27T21:39:45.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc958e53640 1 --2- 172.21.15.60:0/658793967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9541d6f00 0x7fc9540ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:45.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc9537fe640 1 --2- 172.21.15.60:0/658793967 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc9540b3980 0x7fc9540ab280 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc94800c3c0 tx=0x7fc94800c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:45.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc950ff9640 10 client.?.objecter ms_handle_connect 0x7fc9540b3980 2024-03-27T21:39:45.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc950ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:45.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc950ff9640 1 -- 172.21.15.60:0/658793967 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9480074e0 con 0x7fc9540b3980 2024-03-27T21:39:45.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc950ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:45.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc950ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:45.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc950ff9640 10 monclient: dump: 2024-03-27T21:39:45.579 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:45.579 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:45.579 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:45.579 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:45.579 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:45.579 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:45.579 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:45.580 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:45.580 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:45.580 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:45.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc950ff9640 10 monclient: _finish_auth 0 2024-03-27T21:39:45.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc950ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:39:45.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc950ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:45.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc95a8b6640 5 monclient: authenticate success, global_id 4241 2024-03-27T21:39:45.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc95a8b6640 10 monclient: _renew_subs 2024-03-27T21:39:45.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc95a8b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:45.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc950ff9640 1 -- 172.21.15.60:0/658793967 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc948007640 con 0x7fc9540b3980 2024-03-27T21:39:45.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc95a8b6640 1 -- 172.21.15.60:0/658793967 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95413d0b0 con 0x7fc9540b3980 2024-03-27T21:39:45.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc950ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:45.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc950ff9640 1 -- 172.21.15.60:0/658793967 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc948004a20 con 0x7fc9540b3980 2024-03-27T21:39:45.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc950ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:45.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc950ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:45.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc950ff9640 10 monclient: dump: 2024-03-27T21:39:45.581 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:45.581 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:45.581 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:45.581 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:45.581 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:45.581 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:45.581 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:45.581 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:45.581 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:45.581 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:45.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc9527fc640 4 set_mon_vals no callback set 2024-03-27T21:39:45.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc95a8b6640 10 client.4241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:45.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc95a8b6640 10 monclient: _renew_subs 2024-03-27T21:39:45.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc95a8b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:45.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc95a8b6640 1 -- 172.21.15.60:0/658793967 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9540aa9b0 con 0x7fc9540b3980 2024-03-27T21:39:45.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.576+0000 7fc95a8b6640 1 librados: init done 2024-03-27T21:39:45.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.577+0000 7fc95a8b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:45.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.577+0000 7fc95a8b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:45.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.577+0000 7fc95a8b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:45.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.577+0000 7fc95a8b6640 1 -- 172.21.15.60:0/658793967 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9540670d0 con 0x7fc9540b3980 2024-03-27T21:39:45.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.577+0000 7fc950ff9640 1 -- 172.21.15.60:0/658793967 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc948004b80 con 0x7fc9540b3980 2024-03-27T21:39:45.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.578+0000 7fc950ff9640 1 --2- 172.21.15.60:0/658793967 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc94006c590 0x7fc94006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:45.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.578+0000 7fc950ff9640 1 -- 172.21.15.60:0/658793967 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5199+0+0 (secure 0 0 0) 0x7fc948088e40 con 0x7fc9540b3980 2024-03-27T21:39:45.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.578+0000 7fc950ff9640 10 client.4241.objecter ms_dispatch 0x7fc954000b90 osd_map(19..19 src has 1..19) v4 2024-03-27T21:39:45.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.578+0000 7fc950ff9640 3 client.4241.objecter handle_osd_map got epochs [19,19] > 0 2024-03-27T21:39:45.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.578+0000 7fc950ff9640 3 client.4241.objecter handle_osd_map decoding full epoch 19 2024-03-27T21:39:45.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.578+0000 7fc953fff640 1 --2- 172.21.15.60:0/658793967 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc94006c590 0x7fc94006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:45.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.578+0000 7fc953fff640 10 monclient: get_auth_request con 0x7fc94006c590 auth_method 0 2024-03-27T21:39:45.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.578+0000 7fc950ff9640 20 client.4241.objecter dump_active .. 0 homeless 2024-03-27T21:39:45.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.578+0000 7fc950ff9640 10 client.4241.objecter ms_handle_connect 0x7fc94006c590 2024-03-27T21:39:45.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.578+0000 7fc953fff640 1 --2- 172.21.15.60:0/658793967 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc94006c590 0x7fc94006ea50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc944036000 tx=0x7fc944008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:45.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.582+0000 7fc950ff9640 1 -- 172.21.15.60:0/658793967 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc9480542e0 con 0x7fc9540b3980 2024-03-27T21:39:45.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.582+0000 7fc950ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:45.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.582+0000 7fc950ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:45.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.763+0000 7fc95a8b6640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-03-27T21:39:45.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.763+0000 7fc95a8b6640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-03-27T21:39:45.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.763+0000 7fc95a8b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:45.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:45.763+0000 7fc95a8b6640 1 -- 172.21.15.60:0/658793967 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc954069470 con 0x7fc9540b3980 2024-03-27T21:39:46.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.454+0000 7fc950ff9640 1 -- 172.21.15.60:0/658793967 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v20) v1 ==== 173+0+0 (secure 0 0 0) 0x7fc948057900 con 0x7fc9540b3980 2024-03-27T21:39:46.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.454+0000 7fc950ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-03-27T21:39:46.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.454+0000 7fc950ff9640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-03-27T21:39:46.441 INFO:teuthology.orchestra.run.smithi060.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-27T21:39:46.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.458+0000 7fc95a8b6640 10 librados: watch_flush enter 2024-03-27T21:39:46.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.458+0000 7fc95a8b6640 10 librados: watch_flush exit 2024-03-27T21:39:46.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.458+0000 7fc95a8b6640 20 client.4241.objecter shutdown clearing up homeless session... 2024-03-27T21:39:46.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.458+0000 7fc95a8b6640 10 client.4241.objecter successfully canceled tick 2024-03-27T21:39:46.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.458+0000 7fc95a8b6640 1 -- 172.21.15.60:0/658793967 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc94006c590 msgr2=0x7fc94006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:46.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.458+0000 7fc95a8b6640 1 --2- 172.21.15.60:0/658793967 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc94006c590 0x7fc94006ea50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc944036000 tx=0x7fc944008030 comp rx=0 tx=0).stop 2024-03-27T21:39:46.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.458+0000 7fc95a8b6640 10 monclient: shutdown 2024-03-27T21:39:46.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.458+0000 7fc95a8b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:46.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.458+0000 7fc95a8b6640 1 -- 172.21.15.60:0/658793967 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc9540b3980 msgr2=0x7fc9540ab280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:46.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.458+0000 7fc95a8b6640 1 --2- 172.21.15.60:0/658793967 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc9540b3980 0x7fc9540ab280 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc94800c3c0 tx=0x7fc94800c880 comp rx=0 tx=0).stop 2024-03-27T21:39:46.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.459+0000 7fc95a8b6640 1 -- 172.21.15.60:0/658793967 shutdown_connections 2024-03-27T21:39:46.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.459+0000 7fc95a8b6640 1 --2- 172.21.15.60:0/658793967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9541d6f00 0x7fc9540ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:46.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.459+0000 7fc95a8b6640 1 --2- 172.21.15.60:0/658793967 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc9540b3980 0x7fc9540ab280 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:46.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.459+0000 7fc95a8b6640 1 --2- 172.21.15.60:0/658793967 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc94006c590 0x7fc94006ea50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:46.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.459+0000 7fc95a8b6640 1 --2- 172.21.15.60:0/658793967 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9540a86e0 0x7fc9541d64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:46.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.459+0000 7fc95a8b6640 1 -- 172.21.15.60:0/658793967 >> 172.21.15.60:0/658793967 conn(0x7fc954071410 msgr2=0x7fc95413f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:46.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.459+0000 7fc95a8b6640 1 -- 172.21.15.60:0/658793967 shutdown_connections 2024-03-27T21:39:46.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.459+0000 7fc95a8b6640 1 -- 172.21.15.60:0/658793967 wait complete. 2024-03-27T21:39:46.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.459+0000 7fc95a8b6640 1 librados: shutdown 2024-03-27T21:39:46.459 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-27T21:39:46.459 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-27T21:39:46.459 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-27T21:39:46.459 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-27T21:39:46.460 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-27T21:39:46.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.579+0000 7f059e4bd640 10 monclient: get_monmap_and_config 2024-03-27T21:39:46.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.579+0000 7f059e4bd640 10 monclient: build_initial_monmap 2024-03-27T21:39:46.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.579+0000 7f059e4bd640 10 monclient: monmap: 2024-03-27T21:39:46.566 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:46.566 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:46.566 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:46.580721+0000 2024-03-27T21:39:46.566 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:46.580721+0000 2024-03-27T21:39:46.566 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:46.567 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:46.567 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:46.567 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:46.567 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:46.567 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:46.567 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:46.567 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:46.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.580+0000 7f059e4bd640 1 Processor -- start 2024-03-27T21:39:46.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.580+0000 7f059e4bd640 1 -- start start 2024-03-27T21:39:46.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.580+0000 7f059e4bd640 10 monclient: init 2024-03-27T21:39:46.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f059e4bd640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:46.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f059e4bd640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-03-27T21:39:46.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f059e4bd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0598071a50 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:46.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f059e4bd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f059813ef20 0x7f05980a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:46.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f059e4bd640 10 monclient(hunting): picked mon.noname-b con 0x7f059813ef20 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:46.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f059e4bd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0598141630 0x7f0598141ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:46.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f059e4bd640 10 monclient(hunting): picked mon.noname-c con 0x7f0598141630 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:46.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f059e4bd640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05980a8e90 con 0x7f0598071a50 2024-03-27T21:39:46.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f059e4bd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:46.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f059e4bd640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0598071e30 con 0x7f0598141630 2024-03-27T21:39:46.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f059e4bd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:46.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f059e4bd640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05980aa040 con 0x7f059813ef20 2024-03-27T21:39:46.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f059e4bd640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:46.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f059e4bd640 10 monclient(hunting): authenticate will time out at 827.802097s 2024-03-27T21:39:46.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f05977fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f059813ef20 0x7f05980a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:46.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f0597fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0598071a50 0x7f059813e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43494/0 (socket says 172.21.15.60:43494) 2024-03-27T21:39:46.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f0597fff640 1 -- 172.21.15.60:0/541762763 learned_addr learned my addr 172.21.15.60:0/541762763 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:39:46.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f05977fe640 10 monclient(hunting): get_auth_request con 0x7f059813ef20 auth_method 0 2024-03-27T21:39:46.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f05977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:46.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f05977fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:46.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f05977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:46.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f05977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:46.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f05977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:46.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f05977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:46.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f059ca5a640 1 --2- 172.21.15.60:0/541762763 >> v2:172.21.15.191:3300/0 conn(0x7f0598141630 0x7f0598141ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:46.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f059ca5a640 10 monclient(hunting): get_auth_request con 0x7f0598141630 auth_method 0 2024-03-27T21:39:46.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f059ca5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:46.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f059ca5a640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:46.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f059ca5a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:46.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f05977fe640 10 monclient(hunting): handle_auth_done global_id 4244 payload 274 2024-03-27T21:39:46.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f0596ffd640 1 -- 172.21.15.60:0/541762763 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394455457 0 0) 0x7f05980a8e90 con 0x7f0598071a50 2024-03-27T21:39:46.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f05977fe640 1 -- 172.21.15.60:0/541762763 >> v2:172.21.15.191:3300/0 conn(0x7f0598141630 msgr2=0x7f0598141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:46.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f05977fe640 1 --2- 172.21.15.60:0/541762763 >> v2:172.21.15.191:3300/0 conn(0x7f0598141630 0x7f0598141ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:46.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f05977fe640 1 -- 172.21.15.60:0/541762763 >> v1:172.21.15.60:6789/0 conn(0x7f0598071a50 legacy=0x7f059813e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:46.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f05977fe640 10 monclient: _finish_hunting 0 2024-03-27T21:39:46.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f05977fe640 1 monclient: found mon.noname-b 2024-03-27T21:39:46.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f05977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:46.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f05977fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:46.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f05977fe640 1 -- 172.21.15.60:0/541762763 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05980aa2b0 con 0x7f059813ef20 2024-03-27T21:39:46.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.581+0000 7f0596ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:39:46.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.582+0000 7f05977fe640 1 --2- 172.21.15.60:0/541762763 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f059813ef20 0x7f05980a87e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f05880097a0 tx=0x7f058802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e96c664f71b4030d server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:46.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.582+0000 7f0596ffd640 1 -- 172.21.15.60:0/541762763 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0588031830 con 0x7f059813ef20 2024-03-27T21:39:46.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.582+0000 7f0596ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:46.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.582+0000 7f0596ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:46.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.582+0000 7f0596ffd640 10 monclient: dump: 2024-03-27T21:39:46.579 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:46.579 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:46.579 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:46.579 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:46.579 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:46.579 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:46.580 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:46.580 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:46.580 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:46.580 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:46.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.582+0000 7f0596ffd640 10 monclient: _finish_auth 0 2024-03-27T21:39:46.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.582+0000 7f0596ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:39:46.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.582+0000 7f0596ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:46.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.582+0000 7f0596ffd640 1 -- 172.21.15.60:0/541762763 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0588031990 con 0x7f059813ef20 2024-03-27T21:39:46.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.582+0000 7f0596ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:46.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.582+0000 7f0596ffd640 1 -- 172.21.15.60:0/541762763 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0588031d20 con 0x7f059813ef20 2024-03-27T21:39:46.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.582+0000 7f059e4bd640 5 monclient: authenticate success, global_id 4244 2024-03-27T21:39:46.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.582+0000 7f059e4bd640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:46.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.582+0000 7f059e4bd640 4 set_mon_vals no callback set 2024-03-27T21:39:46.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.582+0000 7f0596ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:46.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.582+0000 7f0596ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:46.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.582+0000 7f0596ffd640 10 monclient: dump: 2024-03-27T21:39:46.581 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:46.581 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:46.581 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:46.581 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:46.581 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:46.581 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:46.581 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:46.581 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:46.581 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:46.581 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:46.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.582+0000 7f059e4bd640 10 monclient: shutdown 2024-03-27T21:39:46.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.582+0000 7f059e4bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:46.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.582+0000 7f059e4bd640 1 -- 172.21.15.60:0/541762763 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f059813ef20 msgr2=0x7f05980a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:46.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.582+0000 7f059e4bd640 1 --2- 172.21.15.60:0/541762763 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f059813ef20 0x7f05980a87e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f05880097a0 tx=0x7f058802b8d0 comp rx=0 tx=0).stop 2024-03-27T21:39:46.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.583+0000 7f059e4bd640 1 -- 172.21.15.60:0/541762763 shutdown_connections 2024-03-27T21:39:46.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.583+0000 7f059e4bd640 1 --2- 172.21.15.60:0/541762763 >> v2:172.21.15.191:3300/0 conn(0x7f0598141630 0x7f0598141ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:46.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.583+0000 7f059e4bd640 1 --2- 172.21.15.60:0/541762763 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f059813ef20 0x7f05980a87e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:46.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.583+0000 7f059e4bd640 1 -- 172.21.15.60:0/541762763 >> 172.21.15.60:0/541762763 conn(0x7f05980b7300 msgr2=0x7f05980b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:46.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.583+0000 7f059e4bd640 1 -- 172.21.15.60:0/541762763 shutdown_connections 2024-03-27T21:39:46.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.583+0000 7f059e4bd640 1 -- 172.21.15.60:0/541762763 wait complete. 2024-03-27T21:39:46.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.583+0000 7f059e4bd640 10 monclient: build_initial_monmap 2024-03-27T21:39:46.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.583+0000 7f059e4bd640 10 monclient: monmap: 2024-03-27T21:39:46.582 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:46.582 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:46.583 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:46.583 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:46.583 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:46.583 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:46.583 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:46.583 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:46.583 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:46.583 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:46.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.583+0000 7f059e4bd640 1 librados: starting msgr at 2024-03-27T21:39:46.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.583+0000 7f059e4bd640 1 librados: starting objecter 2024-03-27T21:39:46.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059e4bd640 1 Processor -- start 2024-03-27T21:39:46.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059e4bd640 1 -- start start 2024-03-27T21:39:46.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059e4bd640 1 librados: setting wanted keys 2024-03-27T21:39:46.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059e4bd640 1 librados: calling monclient init 2024-03-27T21:39:46.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059e4bd640 10 monclient: init 2024-03-27T21:39:46.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059e4bd640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:46.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059e4bd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:39:46.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059e4bd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0598071a50 0x7f0598139a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:46.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059e4bd640 10 monclient(hunting): picked mon.noname-b con 0x7f0598071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:46.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059e4bd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f059813ef20 0x7f059813a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:46.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059e4bd640 10 monclient(hunting): picked mon.noname-c con 0x7f059813ef20 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:46.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059e4bd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0598141630 0x7f059813a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:46.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059e4bd640 10 monclient(hunting): picked mon.noname-a con 0x7f0598141630 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:46.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059e4bd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:46.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059e4bd640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05980ab810 con 0x7f0598141630 2024-03-27T21:39:46.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059e4bd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:46.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059e4bd640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0598138ec0 con 0x7f0598071a50 2024-03-27T21:39:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059e4bd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059e4bd640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059813ae90 con 0x7f059813ef20 2024-03-27T21:39:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059e4bd640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059e4bd640 10 monclient(hunting): authenticate will time out at 827.805722s 2024-03-27T21:39:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f05977fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f059813ef20 0x7f059813a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059ca5a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0598141630 0x7f059813a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f05977fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f059813ef20 0x7f059813a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47474/0 (socket says 172.21.15.60:47474) 2024-03-27T21:39:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f05977fe640 1 -- 172.21.15.60:0/1096847233 learned_addr learned my addr 172.21.15.60:0/1096847233 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f05977fe640 10 monclient(hunting): get_auth_request con 0x7f059813ef20 auth_method 0 2024-03-27T21:39:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f05977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f05977fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f05977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059ca5a640 10 monclient(hunting): get_auth_request con 0x7f0598141630 auth_method 0 2024-03-27T21:39:46.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059ca5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:46.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059ca5a640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:46.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.584+0000 7f059ca5a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:46.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f05977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:46.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f05977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:46.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f05977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:46.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f059ca5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:46.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f059ca5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:46.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f059ca5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:46.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f05977fe640 10 monclient(hunting): handle_auth_done global_id 4247 payload 824 2024-03-27T21:39:46.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f05977fe640 1 -- 172.21.15.60:0/1096847233 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0598071a50 msgr2=0x7f0598139a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:46.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f0597fff640 1 --2- 172.21.15.60:0/1096847233 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0598071a50 0x7f0598139a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:46.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f05977fe640 1 --2- 172.21.15.60:0/1096847233 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0598071a50 0x7f0598139a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:46.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f05977fe640 1 -- 172.21.15.60:0/1096847233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0598141630 msgr2=0x7f059813a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:46.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f05977fe640 1 --2- 172.21.15.60:0/1096847233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0598141630 0x7f059813a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:46.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f05977fe640 10 monclient: _finish_hunting 0 2024-03-27T21:39:46.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f05977fe640 1 monclient: found mon.noname-c 2024-03-27T21:39:46.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f05977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:46.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f05977fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:39:46.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f05977fe640 1 -- 172.21.15.60:0/1096847233 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0588009400 con 0x7f059813ef20 2024-03-27T21:39:46.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f059ca5a640 1 --2- 172.21.15.60:0/1096847233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0598141630 0x7f059813a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:46.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f05977fe640 1 --2- 172.21.15.60:0/1096847233 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f059813ef20 0x7f059813a410 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f058802bee0 tx=0x7f05880046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:46.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f0594ff9640 10 client.?.objecter ms_handle_connect 0x7f059813ef20 2024-03-27T21:39:46.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f0594ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:46.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f0594ff9640 1 -- 172.21.15.60:0/1096847233 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058803a6f0 con 0x7f059813ef20 2024-03-27T21:39:46.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f0594ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:46.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f0594ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:46.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f0594ff9640 10 monclient: dump: 2024-03-27T21:39:46.588 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:46.588 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:46.588 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:46.588 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:46.588 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:46.588 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:46.588 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:46.588 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:46.588 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:46.588 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:46.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f0594ff9640 10 monclient: _finish_auth 0 2024-03-27T21:39:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f0594ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:39:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f0594ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f059e4bd640 5 monclient: authenticate success, global_id 4247 2024-03-27T21:39:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f059e4bd640 10 monclient: _renew_subs 2024-03-27T21:39:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f0594ff9640 1 -- 172.21.15.60:0/1096847233 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058803a870 con 0x7f059813ef20 2024-03-27T21:39:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f059e4bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f059e4bd640 1 -- 172.21.15.60:0/1096847233 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05980abc30 con 0x7f059813ef20 2024-03-27T21:39:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f0594ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f0594ff9640 1 -- 172.21.15.60:0/1096847233 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058803ac20 con 0x7f059813ef20 2024-03-27T21:39:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f0594ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f0594ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f0594ff9640 10 monclient: dump: 2024-03-27T21:39:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:46.590 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f059e4bd640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f059e4bd640 10 monclient: _renew_subs 2024-03-27T21:39:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f0595ffb640 4 set_mon_vals no callback set 2024-03-27T21:39:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f059e4bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f059e4bd640 1 -- 172.21.15.60:0/1096847233 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05981388e0 con 0x7f059813ef20 2024-03-27T21:39:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.585+0000 7f059e4bd640 1 librados: init done 2024-03-27T21:39:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.586+0000 7f059e4bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.586+0000 7f059e4bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.586+0000 7f059e4bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.586+0000 7f059e4bd640 1 -- 172.21.15.60:0/1096847233 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0598067900 con 0x7f059813ef20 2024-03-27T21:39:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.586+0000 7f0594ff9640 1 -- 172.21.15.60:0/1096847233 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f058804c3b0 con 0x7f059813ef20 2024-03-27T21:39:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.587+0000 7f0594ff9640 1 --2- 172.21.15.60:0/1096847233 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f058406c590 0x7f058406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.587+0000 7f0594ff9640 1 -- 172.21.15.60:0/1096847233 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f05880aae50 con 0x7f059813ef20 2024-03-27T21:39:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.587+0000 7f0597fff640 1 --2- 172.21.15.60:0/1096847233 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f058406c590 0x7f058406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.587+0000 7f0594ff9640 10 client.4247.objecter ms_dispatch 0x7f0598000b90 osd_map(20..20 src has 1..20) v4 2024-03-27T21:39:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.587+0000 7f0594ff9640 3 client.4247.objecter handle_osd_map got epochs [20,20] > 0 2024-03-27T21:39:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.587+0000 7f0594ff9640 3 client.4247.objecter handle_osd_map decoding full epoch 20 2024-03-27T21:39:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.587+0000 7f0597fff640 10 monclient: get_auth_request con 0x7f058406c590 auth_method 0 2024-03-27T21:39:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.587+0000 7f0594ff9640 20 client.4247.objecter dump_active .. 0 homeless 2024-03-27T21:39:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.587+0000 7f0594ff9640 10 client.4247.objecter ms_handle_connect 0x7f058406c590 2024-03-27T21:39:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.587+0000 7f0597fff640 1 --2- 172.21.15.60:0/1096847233 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f058406c590 0x7f058406ea50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f058c007fd0 tx=0x7f058c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.591+0000 7f0594ff9640 1 -- 172.21.15.60:0/1096847233 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0588076480 con 0x7f059813ef20 2024-03-27T21:39:46.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.591+0000 7f0594ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:46.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.591+0000 7f0594ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:46.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.821+0000 7f059e4bd640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-03-27T21:39:46.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.821+0000 7f059e4bd640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-03-27T21:39:46.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.821+0000 7f059e4bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:46.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.821+0000 7f059e4bd640 1 -- 172.21.15.60:0/1096847233 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f05980ab660 con 0x7f059813ef20 2024-03-27T21:39:46.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.824+0000 7f0594ff9640 1 -- 172.21.15.60:0/1096847233 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+170923 (secure 0 0 0) 0x7f0588079aa0 con 0x7f059813ef20 2024-03-27T21:39:46.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.824+0000 7f0594ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-03-27T21:39:46.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.824+0000 7f0594ff9640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:46.813 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:39:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.827+0000 7f059e4bd640 10 librados: watch_flush enter 2024-03-27T21:39:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.827+0000 7f059e4bd640 10 librados: watch_flush exit 2024-03-27T21:39:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.827+0000 7f059e4bd640 20 client.4247.objecter shutdown clearing up homeless session... 2024-03-27T21:39:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.827+0000 7f059e4bd640 10 client.4247.objecter successfully canceled tick 2024-03-27T21:39:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.827+0000 7f059e4bd640 1 -- 172.21.15.60:0/1096847233 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f058406c590 msgr2=0x7f058406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.827+0000 7f059e4bd640 1 --2- 172.21.15.60:0/1096847233 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f058406c590 0x7f058406ea50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f058c007fd0 tx=0x7f058c009040 comp rx=0 tx=0).stop 2024-03-27T21:39:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.827+0000 7f059e4bd640 10 monclient: shutdown 2024-03-27T21:39:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.827+0000 7f059e4bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.827+0000 7f059e4bd640 1 -- 172.21.15.60:0/1096847233 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f059813ef20 msgr2=0x7f059813a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.827+0000 7f059e4bd640 1 --2- 172.21.15.60:0/1096847233 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f059813ef20 0x7f059813a410 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f058802bee0 tx=0x7f05880046c0 comp rx=0 tx=0).stop 2024-03-27T21:39:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.827+0000 7f059e4bd640 1 -- 172.21.15.60:0/1096847233 shutdown_connections 2024-03-27T21:39:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.827+0000 7f059e4bd640 1 --2- 172.21.15.60:0/1096847233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0598141630 0x7f059813a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.827+0000 7f059e4bd640 1 --2- 172.21.15.60:0/1096847233 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f059813ef20 0x7f059813a410 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.827+0000 7f059e4bd640 1 --2- 172.21.15.60:0/1096847233 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f058406c590 0x7f058406ea50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.827+0000 7f059e4bd640 1 --2- 172.21.15.60:0/1096847233 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0598071a50 0x7f0598139a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.827+0000 7f059e4bd640 1 -- 172.21.15.60:0/1096847233 >> 172.21.15.60:0/1096847233 conn(0x7f05980b7300 msgr2=0x7f059813b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.827+0000 7f059e4bd640 1 -- 172.21.15.60:0/1096847233 shutdown_connections 2024-03-27T21:39:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.827+0000 7f059e4bd640 1 -- 172.21.15.60:0/1096847233 wait complete. 2024-03-27T21:39:46.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.827+0000 7f059e4bd640 1 librados: shutdown 2024-03-27T21:39:46.824 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":5,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":34387},{"type":"v1","addr":"172.21.15.60:6833","nonce":34387}]},"active_addr":"172.21.15.60:6833/34387","active_change":"2024-03-27T21:39:31.381865+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4106,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":3656808405}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":42544838}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":1240895904}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":59937682}]}]} 2024-03-27T21:39:46.827 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-27T21:39:46.828 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-27T21:39:46.828 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T21:39:46.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.941+0000 7f6eb03f6640 10 monclient: get_monmap_and_config 2024-03-27T21:39:46.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.941+0000 7f6eb03f6640 10 monclient: build_initial_monmap 2024-03-27T21:39:46.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.941+0000 7f6eb03f6640 10 monclient: monmap: 2024-03-27T21:39:46.928 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:46.928 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:46.942456+0000 2024-03-27T21:39:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:46.942456+0000 2024-03-27T21:39:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:46.929 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6eb03f6640 1 Processor -- start 2024-03-27T21:39:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6eb03f6640 1 -- start start 2024-03-27T21:39:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6eb03f6640 10 monclient: init 2024-03-27T21:39:46.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6eb03f6640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:46.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6eb03f6640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-03-27T21:39:46.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6eb03f6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6ea8071a50 0x7f6ea80a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:46.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6eb03f6640 10 monclient(hunting): picked mon.noname-c con 0x7f6ea8071a50 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:46.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6eb03f6640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6ea813e810 0x7f6ea813ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:46.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6eb03f6640 10 monclient(hunting): picked mon.noname-b con 0x7f6ea813e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:46.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6eb03f6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ea8141630 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:46.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6eb03f6640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea813f190 con 0x7f6ea8141630 2024-03-27T21:39:46.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6eb03f6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:46.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6eb03f6640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ea8071e30 con 0x7f6ea8071a50 2024-03-27T21:39:46.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6eb03f6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:46.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6eb03f6640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea80a8e90 con 0x7f6ea813e810 2024-03-27T21:39:46.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6eb03f6640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:46.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6eb03f6640 10 monclient(hunting): authenticate will time out at 828.163704s 2024-03-27T21:39:46.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6ead991640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6ea813e810 0x7f6ea813ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:46.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6ead991640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6ea813e810 0x7f6ea813ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47476/0 (socket says 172.21.15.60:47476) 2024-03-27T21:39:46.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6ead991640 1 -- 172.21.15.60:0/1814632165 learned_addr learned my addr 172.21.15.60:0/1814632165 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:46.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6ead991640 10 monclient(hunting): get_auth_request con 0x7f6ea813e810 auth_method 0 2024-03-27T21:39:46.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6ead991640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:46.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6ead991640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:46.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.942+0000 7f6ead991640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:46.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.943+0000 7f6ead991640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:46.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.943+0000 7f6ead991640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:46.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.943+0000 7f6ead991640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:46.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.943+0000 7f6eae192640 1 --2- 172.21.15.60:0/1814632165 >> v2:172.21.15.191:3300/0 conn(0x7f6ea8071a50 0x7f6ea80a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:46.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.943+0000 7f6eae192640 10 monclient(hunting): get_auth_request con 0x7f6ea8071a50 auth_method 0 2024-03-27T21:39:46.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.943+0000 7f6eae192640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:46.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.943+0000 7f6eae192640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:46.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.943+0000 7f6eae192640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:46.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.943+0000 7f6ead991640 10 monclient(hunting): handle_auth_done global_id 4250 payload 274 2024-03-27T21:39:46.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.943+0000 7f6ead190640 1 -- 172.21.15.60:0/1814632165 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1997958900 0 0) 0x7f6ea0001490 con 0x7f6ea8141630 2024-03-27T21:39:46.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.943+0000 7f6ead991640 1 -- 172.21.15.60:0/1814632165 >> v2:172.21.15.191:3300/0 conn(0x7f6ea8071a50 msgr2=0x7f6ea80a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:46.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.943+0000 7f6ead991640 1 --2- 172.21.15.60:0/1814632165 >> v2:172.21.15.191:3300/0 conn(0x7f6ea8071a50 0x7f6ea80a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:46.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.943+0000 7f6ead991640 1 -- 172.21.15.60:0/1814632165 >> v1:172.21.15.60:6789/0 conn(0x7f6ea8141630 legacy=0x7f6ea8141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:46.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.943+0000 7f6ead991640 10 monclient: _finish_hunting 0 2024-03-27T21:39:46.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.943+0000 7f6ead991640 1 monclient: found mon.noname-b 2024-03-27T21:39:46.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.943+0000 7f6ead991640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:46.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.943+0000 7f6ead991640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:46.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.943+0000 7f6ead991640 1 -- 172.21.15.60:0/1814632165 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea80aa040 con 0x7f6ea813e810 2024-03-27T21:39:46.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.943+0000 7f6ead190640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:39:46.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.943+0000 7f6ead991640 1 --2- 172.21.15.60:0/1814632165 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6ea813e810 0x7f6ea813ec50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4009870 tx=0x7f6ea402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bf1ebafdf8b9c61 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:46.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6ead190640 1 -- 172.21.15.60:0/1814632165 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea4030840 con 0x7f6ea813e810 2024-03-27T21:39:46.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6ead190640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:46.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6ead190640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:46.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6ead190640 10 monclient: dump: 2024-03-27T21:39:46.941 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:46.941 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:46.941 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:46.941 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:46.941 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:46.941 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:46.941 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:46.941 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:46.941 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:46.941 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:46.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6ead190640 10 monclient: _finish_auth 0 2024-03-27T21:39:46.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6ead190640 10 monclient: _check_auth_tickets 2024-03-27T21:39:46.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6ead190640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:46.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6eb03f6640 5 monclient: authenticate success, global_id 4250 2024-03-27T21:39:46.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6eb03f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:46.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6ead190640 1 -- 172.21.15.60:0/1814632165 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea40309a0 con 0x7f6ea813e810 2024-03-27T21:39:46.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6ead190640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:46.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6eb03f6640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:46.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6eb03f6640 4 set_mon_vals no callback set 2024-03-27T21:39:46.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6ead190640 1 -- 172.21.15.60:0/1814632165 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea4030d30 con 0x7f6ea813e810 2024-03-27T21:39:46.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6ead190640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:46.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6ead190640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:46.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6ead190640 10 monclient: dump: 2024-03-27T21:39:46.942 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:46.942 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:46.943 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:46.943 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:46.943 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:46.943 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:46.943 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:46.943 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:46.943 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:46.943 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:46.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6eb03f6640 10 monclient: shutdown 2024-03-27T21:39:46.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6eb03f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:46.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6eb03f6640 1 -- 172.21.15.60:0/1814632165 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6ea813e810 msgr2=0x7f6ea813ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:46.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6eb03f6640 1 --2- 172.21.15.60:0/1814632165 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6ea813e810 0x7f6ea813ec50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4009870 tx=0x7f6ea402b980 comp rx=0 tx=0).stop 2024-03-27T21:39:46.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6eb03f6640 1 -- 172.21.15.60:0/1814632165 shutdown_connections 2024-03-27T21:39:46.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6eb03f6640 1 --2- 172.21.15.60:0/1814632165 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6ea813e810 0x7f6ea813ec50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:46.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6eb03f6640 1 --2- 172.21.15.60:0/1814632165 >> v2:172.21.15.191:3300/0 conn(0x7f6ea8071a50 0x7f6ea80a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:46.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6eb03f6640 1 -- 172.21.15.60:0/1814632165 >> 172.21.15.60:0/1814632165 conn(0x7f6ea80b7300 msgr2=0x7f6ea80b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:46.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6eb03f6640 1 -- 172.21.15.60:0/1814632165 shutdown_connections 2024-03-27T21:39:46.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.944+0000 7f6eb03f6640 1 -- 172.21.15.60:0/1814632165 wait complete. 2024-03-27T21:39:46.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.945+0000 7f6eb03f6640 10 monclient: build_initial_monmap 2024-03-27T21:39:46.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.945+0000 7f6eb03f6640 10 monclient: monmap: 2024-03-27T21:39:46.944 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:46.944 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:46.944 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:46.944 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:46.944 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:46.944 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:46.944 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:46.945 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:46.945 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:46.945 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:46.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.945+0000 7f6eb03f6640 1 librados: starting msgr at 2024-03-27T21:39:46.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.945+0000 7f6eb03f6640 1 librados: starting objecter 2024-03-27T21:39:46.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.945+0000 7f6eb03f6640 1 Processor -- start 2024-03-27T21:39:46.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.945+0000 7f6eb03f6640 1 -- start start 2024-03-27T21:39:46.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.945+0000 7f6eb03f6640 1 librados: setting wanted keys 2024-03-27T21:39:46.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.945+0000 7f6eb03f6640 1 librados: calling monclient init 2024-03-27T21:39:46.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.945+0000 7f6eb03f6640 10 monclient: init 2024-03-27T21:39:46.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eb03f6640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:46.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eb03f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:39:46.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eb03f6640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6ea8071a50 0x7f6ea81d6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:46.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eb03f6640 10 monclient(hunting): picked mon.noname-c con 0x7f6ea8071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:46.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eb03f6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ea813e810 0x7f6ea81d7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:46.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eb03f6640 10 monclient(hunting): picked mon.noname-b con 0x7f6ea813e810 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:46.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eb03f6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea8141630 0x7f6ea8132f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:46.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eb03f6640 10 monclient(hunting): picked mon.noname-a con 0x7f6ea8141630 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:46.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eb03f6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:46.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eb03f6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea8138ed0 con 0x7f6ea8141630 2024-03-27T21:39:46.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eb03f6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:46.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eb03f6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea8137380 con 0x7f6ea813e810 2024-03-27T21:39:46.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eb03f6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:46.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eb03f6640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea8137540 con 0x7f6ea8071a50 2024-03-27T21:39:46.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eb03f6640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:46.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eb03f6640 10 monclient(hunting): authenticate will time out at 828.167206s 2024-03-27T21:39:46.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae192640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6ea8071a50 0x7f6ea81d6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:46.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae192640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6ea8071a50 0x7f6ea81d6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47490/0 (socket says 172.21.15.60:47490) 2024-03-27T21:39:46.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae192640 1 -- 172.21.15.60:0/2553540721 learned_addr learned my addr 172.21.15.60:0/2553540721 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:46.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae192640 10 monclient(hunting): get_auth_request con 0x7f6ea8071a50 auth_method 0 2024-03-27T21:39:46.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae192640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:46.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae192640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:46.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae192640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:46.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae993640 1 --2- 172.21.15.60:0/2553540721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea8141630 0x7f6ea8132f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:46.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae993640 10 monclient(hunting): get_auth_request con 0x7f6ea8141630 auth_method 0 2024-03-27T21:39:46.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae993640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:46.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae993640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:46.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae993640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:46.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae993640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:46.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae993640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:46.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae993640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:46.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6ead991640 1 --2- 172.21.15.60:0/2553540721 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ea813e810 0x7f6ea81d7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:46.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae192640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:46.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae192640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:46.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae192640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:46.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6ead991640 10 monclient(hunting): get_auth_request con 0x7f6ea813e810 auth_method 0 2024-03-27T21:39:46.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6ead991640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:46.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6ead991640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:46.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6ead991640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:46.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae993640 10 monclient(hunting): handle_auth_done global_id 4275 payload 824 2024-03-27T21:39:46.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae993640 1 -- 172.21.15.60:0/2553540721 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6ea8071a50 msgr2=0x7f6ea81d6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:46.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae993640 1 --2- 172.21.15.60:0/2553540721 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6ea8071a50 0x7f6ea81d6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:46.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae993640 1 -- 172.21.15.60:0/2553540721 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ea813e810 msgr2=0x7f6ea81d7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:46.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae993640 1 --2- 172.21.15.60:0/2553540721 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ea813e810 0x7f6ea81d7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:46.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae993640 10 monclient: _finish_hunting 0 2024-03-27T21:39:46.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae993640 1 monclient: found mon.noname-a 2024-03-27T21:39:46.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae993640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:46.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae993640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:46.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae993640 1 -- 172.21.15.60:0/2553540721 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea40094d0 con 0x7f6ea8141630 2024-03-27T21:39:46.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6ead991640 1 --2- 172.21.15.60:0/2553540721 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ea813e810 0x7f6ea81d7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:39:46.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6eae993640 1 --2- 172.21.15.60:0/2553540721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea8141630 0x7f6ea8132f30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6ea000c780 tx=0x7f6ea000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:46.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6e9affd640 10 client.?.objecter ms_handle_connect 0x7f6ea8141630 2024-03-27T21:39:46.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6e9affd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:46.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.946+0000 7f6e9affd640 1 -- 172.21.15.60:0/2553540721 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0003b70 con 0x7f6ea8141630 2024-03-27T21:39:46.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6e9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:46.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6e9affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:46.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6e9affd640 10 monclient: dump: 2024-03-27T21:39:46.949 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:46.949 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:46.949 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:46.949 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:46.950 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:46.950 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:46.950 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:46.950 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:46.950 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:46.950 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:46.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6e9affd640 10 monclient: _finish_auth 0 2024-03-27T21:39:46.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6e9affd640 10 monclient: _check_auth_tickets 2024-03-27T21:39:46.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6e9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:46.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6eb03f6640 5 monclient: authenticate success, global_id 4275 2024-03-27T21:39:46.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6e9affd640 1 -- 172.21.15.60:0/2553540721 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea0003cd0 con 0x7f6ea8141630 2024-03-27T21:39:46.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6eb03f6640 10 monclient: _renew_subs 2024-03-27T21:39:46.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6eb03f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:46.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6eb03f6640 1 -- 172.21.15.60:0/2553540721 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea8133470 con 0x7f6ea8141630 2024-03-27T21:39:46.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6e9affd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:46.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6e9affd640 1 -- 172.21.15.60:0/2553540721 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea00052b0 con 0x7f6ea8141630 2024-03-27T21:39:46.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6e9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:46.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6e9affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:46.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6e9affd640 10 monclient: dump: 2024-03-27T21:39:46.951 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:46.951 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:46.951 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:46.951 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:46.951 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:46.951 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:46.951 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:46.951 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:46.951 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:46.952 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:46.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6eb03f6640 10 client.4275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:46.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6eac98f640 4 set_mon_vals no callback set 2024-03-27T21:39:46.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6eb03f6640 10 monclient: _renew_subs 2024-03-27T21:39:46.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6eb03f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:46.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6eb03f6640 1 -- 172.21.15.60:0/2553540721 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea81339a0 con 0x7f6ea8141630 2024-03-27T21:39:46.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.947+0000 7f6eb03f6640 1 librados: init done 2024-03-27T21:39:46.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.948+0000 7f6eb03f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:46.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.948+0000 7f6eb03f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:46.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.948+0000 7f6eb03f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:46.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.948+0000 7f6eb03f6640 1 -- 172.21.15.60:0/2553540721 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea8139040 con 0x7f6ea8141630 2024-03-27T21:39:46.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.948+0000 7f6e9affd640 1 -- 172.21.15.60:0/2553540721 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6ea0005410 con 0x7f6ea8141630 2024-03-27T21:39:46.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.948+0000 7f6e9affd640 1 --2- 172.21.15.60:0/2553540721 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6e8c06c590 0x7f6e8c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:46.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.948+0000 7f6e9affd640 1 -- 172.21.15.60:0/2553540721 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6ea0088720 con 0x7f6ea8141630 2024-03-27T21:39:46.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.948+0000 7f6e9affd640 10 client.4275.objecter ms_dispatch 0x7f6ea8000b90 osd_map(20..20 src has 1..20) v4 2024-03-27T21:39:46.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.948+0000 7f6e9affd640 3 client.4275.objecter handle_osd_map got epochs [20,20] > 0 2024-03-27T21:39:46.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.948+0000 7f6e9affd640 3 client.4275.objecter handle_osd_map decoding full epoch 20 2024-03-27T21:39:46.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.948+0000 7f6eae192640 1 --2- 172.21.15.60:0/2553540721 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6e8c06c590 0x7f6e8c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:46.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.948+0000 7f6eae192640 10 monclient: get_auth_request con 0x7f6e8c06c590 auth_method 0 2024-03-27T21:39:46.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.948+0000 7f6e9affd640 20 client.4275.objecter dump_active .. 0 homeless 2024-03-27T21:39:46.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.949+0000 7f6eae192640 1 --2- 172.21.15.60:0/2553540721 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6e8c06c590 0x7f6e8c06ea50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c006fd0 tx=0x7f6e9c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:46.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.949+0000 7f6e9affd640 10 client.4275.objecter ms_handle_connect 0x7f6e8c06c590 2024-03-27T21:39:46.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.953+0000 7f6e9affd640 1 -- 172.21.15.60:0/2553540721 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6ea0053b90 con 0x7f6ea8141630 2024-03-27T21:39:46.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.953+0000 7f6e9affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:46.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:46.953+0000 7f6e9affd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:47.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.133+0000 7f6eb03f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:39:47.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.133+0000 7f6eb03f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:39:47.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.133+0000 7f6eb03f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:47.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.133+0000 7f6eb03f6640 1 -- 172.21.15.60:0/2553540721 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ea8067900 con 0x7f6ea8141630 2024-03-27T21:39:47.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.134+0000 7f6e9affd640 1 -- 172.21.15.60:0/2553540721 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+14181 (secure 0 0 0) 0x7f6ea00571b0 con 0x7f6ea8141630 2024-03-27T21:39:47.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.134+0000 7f6e9affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:39:47.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.134+0000 7f6e9affd640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:47.121 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:39:47.121 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"fsid":"dcd19d9a-535c-4b41-9095-d588268e2860","created":"2024-03-27T21:39:27.092191+0000","modified":"2024-03-27T21:39:46.450733+0000","last_up_change":"2024-03-27T21:39:38.411432+0000","last_in_change":"2024-03-27T21:39:29.291620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T21:39:37.438493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-27T21:39:45.042103+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"778aaa64-6720-495b-bcdc-86fbcece9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":34603},{"type":"v1","addr":"172.21.15.60:6817","nonce":34603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":34603},{"type":"v1","addr":"172.21.15.60:6819","nonce":34603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":34603},{"type":"v1","addr":"172.21.15.60:6823","nonce":34603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":34603},{"type":"v1","addr":"172.21.15.60:6821","nonce":34603}]},"public_addr":"172.21.15.60:6817/34603","cluster_addr":"172.21.15.60:6819/34603","heartbeat_back_addr":"172.21.15.60:6823/34603","heartbeat_front_addr":"172.21.15.60:6821/34603","state":["exists","up"]},{"osd":1,"uuid":"d960adf1-3d83-4e24-a612-8df53fa43d0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":34602},{"type":"v1","addr":"172.21.15.60:6801","nonce":34602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":34602},{"type":"v1","addr":"172.21.15.60:6803","nonce":34602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":34602},{"type":"v1","addr":"172.21.15.60:6807","nonce":34602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":34602},{"type":"v1","addr":"172.21.15.60:6805","nonce":34602}]},"public_addr":"172.21.15.60:6801/34602","cluster_addr":"172.21.15.60:6803/34602","heartbeat_back_addr":"172.21.15.60:6807/34602","heartbeat_front_addr":"172.21.15.60:6805/34602","state":["exists","up"]},{"osd":2,"uuid":"b8ffc48b-cf6f-4701-a6aa-c25fdb0f7977","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":34604},{"type":"v1","addr":"172.21.15.60:6825","nonce":34604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":34604},{"type":"v1","addr":"172.21.15.60:6827","nonce":34604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":34604},{"type":"v1","addr":"172.21.15.60:6831","nonce":34604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":34604},{"type":"v1","addr":"172.21.15.60:6829","nonce":34604}]},"public_addr":"172.21.15.60:6825/34604","cluster_addr":"172.21.15.60:6827/34604","heartbeat_back_addr":"172.21.15.60:6831/34604","heartbeat_front_addr":"172.21.15.60:6829/34604","state":["exists","up"]},{"osd":3,"uuid":"0f14f2d8-b271-48e4-a198-aa5279a01c06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":34613},{"type":"v1","addr":"172.21.15.60:6809","nonce":34613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":34613},{"type":"v1","addr":"172.21.15.60:6811","nonce":34613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":34613},{"type":"v1","addr":"172.21.15.60:6815","nonce":34613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":34613},{"type":"v1","addr":"172.21.15.60:6813","nonce":34613}]},"public_addr":"172.21.15.60:6809/34613","cluster_addr":"172.21.15.60:6811/34613","heartbeat_back_addr":"172.21.15.60:6815/34613","heartbeat_front_addr":"172.21.15.60:6813/34613","state":["exists","up"]},{"osd":4,"uuid":"8cdeb7ae-1c6e-4d0d-ba44-80248eff2fb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":34332},{"type":"v1","addr":"172.21.15.191:6801","nonce":34332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":34332},{"type":"v1","addr":"172.21.15.191:6803","nonce":34332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":34332},{"type":"v1","addr":"172.21.15.191:6807","nonce":34332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":34332},{"type":"v1","addr":"172.21.15.191:6805","nonce":34332}]},"public_addr":"172.21.15.191:6801/34332","cluster_addr":"172.21.15.191:6803/34332","heartbeat_back_addr":"172.21.15.191:6807/34332","heartbeat_front_addr":"172.21.15.191:6805/34332","state":["exists","up"]},{"osd":5,"uuid":"93748cf8-d3fe-46b8-950f-e01e28a5e02c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6809","nonce":34333},{"type":"v1","addr":"172.21.15.191:6811","nonce":34333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6813","nonce":34333},{"type":"v1","addr":"172.21.15.191:6816","nonce":34333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":34333},{"type":"v1","addr":"172.21.15.191:6823","nonce":34333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":34333},{"type":"v1","addr":"172.21.15.191:6820","nonce":34333}]},"public_addr":"172.21.15.191:6811/34333","cluster_addr":"172.21.15.191:6816/34333","heartbeat_back_addr":"172.21.15.191:6823/34333","heartbeat_front_addr":"172.21.15.191:6820/34333","state":["exists","up"]},{"osd":6,"uuid":"1ad21434-cbc5-4de9-8d63-76f894f7e736","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":34361},{"type":"v1","addr":"172.21.15.191:6825","nonce":34361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":34361},{"type":"v1","addr":"172.21.15.191:6827","nonce":34361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":34361},{"type":"v1","addr":"172.21.15.191:6831","nonce":34361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":34361},{"type":"v1","addr":"172.21.15.191:6829","nonce":34361}]},"public_addr":"172.21.15.191:6825/34361","cluster_addr":"172.21.15.191:6827/34361","heartbeat_back_addr":"172.21.15.191:6831/34361","heartbeat_front_addr":"172.21.15.191:6829/34361","state":["exists","up"]},{"osd":7,"uuid":"9fbd0927-ee2f-471b-8df8-c4e0fc40b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":34334},{"type":"v1","addr":"172.21.15.191:6810","nonce":34334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":34334},{"type":"v1","addr":"172.21.15.191:6814","nonce":34334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6819","nonce":34334},{"type":"v1","addr":"172.21.15.191:6821","nonce":34334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6815","nonce":34334},{"type":"v1","addr":"172.21.15.191:6817","nonce":34334}]},"public_addr":"172.21.15.191:6810/34334","cluster_addr":"172.21.15.191:6814/34334","heartbeat_back_addr":"172.21.15.191:6821/34334","heartbeat_front_addr":"172.21.15.191:6817/34334","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.476097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.411243+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.458154+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.415285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.431201+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.456013+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.474448+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.414076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T21:39:47.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.135+0000 7f6eb03f6640 10 librados: watch_flush enter 2024-03-27T21:39:47.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.135+0000 7f6eb03f6640 10 librados: watch_flush exit 2024-03-27T21:39:47.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.135+0000 7f6eb03f6640 20 client.4275.objecter shutdown clearing up homeless session... 2024-03-27T21:39:47.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.135+0000 7f6eb03f6640 10 client.4275.objecter successfully canceled tick 2024-03-27T21:39:47.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.135+0000 7f6eb03f6640 1 -- 172.21.15.60:0/2553540721 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6e8c06c590 msgr2=0x7f6e8c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.135+0000 7f6eb03f6640 1 --2- 172.21.15.60:0/2553540721 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6e8c06c590 0x7f6e8c06ea50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c006fd0 tx=0x7f6e9c008040 comp rx=0 tx=0).stop 2024-03-27T21:39:47.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.135+0000 7f6eb03f6640 10 monclient: shutdown 2024-03-27T21:39:47.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.135+0000 7f6eb03f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:47.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.135+0000 7f6eb03f6640 1 -- 172.21.15.60:0/2553540721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea8141630 msgr2=0x7f6ea8132f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.135+0000 7f6eb03f6640 1 --2- 172.21.15.60:0/2553540721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea8141630 0x7f6ea8132f30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6ea000c780 tx=0x7f6ea000cc40 comp rx=0 tx=0).stop 2024-03-27T21:39:47.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.135+0000 7f6eb03f6640 1 -- 172.21.15.60:0/2553540721 shutdown_connections 2024-03-27T21:39:47.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.135+0000 7f6eb03f6640 1 --2- 172.21.15.60:0/2553540721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea8141630 0x7f6ea8132f30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.135+0000 7f6eb03f6640 1 --2- 172.21.15.60:0/2553540721 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6e8c06c590 0x7f6e8c06ea50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.135+0000 7f6eb03f6640 1 --2- 172.21.15.60:0/2553540721 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6ea813e810 0x7f6ea81d7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.135+0000 7f6eb03f6640 1 --2- 172.21.15.60:0/2553540721 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6ea8071a50 0x7f6ea81d6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.135+0000 7f6eb03f6640 1 -- 172.21.15.60:0/2553540721 >> 172.21.15.60:0/2553540721 conn(0x7f6ea80b7300 msgr2=0x7f6ea81368f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:47.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.136+0000 7f6eb03f6640 1 -- 172.21.15.60:0/2553540721 shutdown_connections 2024-03-27T21:39:47.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.136+0000 7f6eb03f6640 1 -- 172.21.15.60:0/2553540721 wait complete. 2024-03-27T21:39:47.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.136+0000 7f6eb03f6640 1 librados: shutdown 2024-03-27T21:39:47.133 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-27T21:39:47.133 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T21:39:47.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.245+0000 7faf72d8e640 10 monclient: get_monmap_and_config 2024-03-27T21:39:47.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.245+0000 7faf72d8e640 10 monclient: build_initial_monmap 2024-03-27T21:39:47.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.245+0000 7faf72d8e640 10 monclient: monmap: 2024-03-27T21:39:47.232 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:47.232 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.232 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:47.247035+0000 2024-03-27T21:39:47.232 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:47.247035+0000 2024-03-27T21:39:47.233 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:47.233 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.233 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:47.233 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:47.233 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:47.233 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:47.233 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:47.233 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.245+0000 7faf72d8e640 1 Processor -- start 2024-03-27T21:39:47.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.245+0000 7faf72d8e640 1 -- start start 2024-03-27T21:39:47.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.245+0000 7faf72d8e640 10 monclient: init 2024-03-27T21:39:47.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf72d8e640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:47.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf72d8e640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-03-27T21:39:47.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf72d8e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faf6c0b40b0 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:47.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf72d8e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7faf6c13b020 0x7faf6c13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf72d8e640 10 monclient(hunting): picked mon.noname-c con 0x7faf6c13b020 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:47.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf72d8e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faf6c1416f0 0x7faf6c141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf72d8e640 10 monclient(hunting): picked mon.noname-b con 0x7faf6c1416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf72d8e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf6c0b3830 con 0x7faf6c0b40b0 2024-03-27T21:39:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf72d8e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf72d8e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf6c0b4490 con 0x7faf6c13b020 2024-03-27T21:39:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf72d8e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf72d8e640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c0aa520 con 0x7faf6c1416f0 2024-03-27T21:39:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf72d8e640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf72d8e640 10 monclient(hunting): authenticate will time out at 828.468272s 2024-03-27T21:39:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf7132b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faf6c1416f0 0x7faf6c141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf7132b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faf6c1416f0 0x7faf6c141b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47502/0 (socket says 172.21.15.60:47502) 2024-03-27T21:39:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf7132b640 1 -- 172.21.15.60:0/615004532 learned_addr learned my addr 172.21.15.60:0/615004532 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf7132b640 10 monclient(hunting): get_auth_request con 0x7faf6c1416f0 auth_method 0 2024-03-27T21:39:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf7132b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf7132b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf7132b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf7132b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf7132b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf7132b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf6bfff640 1 --2- 172.21.15.60:0/615004532 >> v2:172.21.15.191:3300/0 conn(0x7faf6c13b020 0x7faf6c13b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf6bfff640 10 monclient(hunting): get_auth_request con 0x7faf6c13b020 auth_method 0 2024-03-27T21:39:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf7132b640 10 monclient(hunting): handle_auth_done global_id 4256 payload 274 2024-03-27T21:39:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf7132b640 1 -- 172.21.15.60:0/615004532 >> v2:172.21.15.191:3300/0 conn(0x7faf6c13b020 msgr2=0x7faf6c13b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf7132b640 1 --2- 172.21.15.60:0/615004532 >> v2:172.21.15.191:3300/0 conn(0x7faf6c13b020 0x7faf6c13b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf7132b640 1 -- 172.21.15.60:0/615004532 >> v1:172.21.15.191:6789/0 conn(0x7faf6c0b40b0 legacy=0x7faf6c0a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf7132b640 10 monclient: _finish_hunting 0 2024-03-27T21:39:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf7132b640 1 monclient: found mon.noname-b 2024-03-27T21:39:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf7132b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf7132b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.246+0000 7faf7132b640 1 -- 172.21.15.60:0/615004532 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6c0a9470 con 0x7faf6c1416f0 2024-03-27T21:39:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.247+0000 7faf7132b640 1 --2- 172.21.15.60:0/615004532 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faf6c1416f0 0x7faf6c141b80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faf64009e50 tx=0x7faf6402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42f6605e6e894074 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.247+0000 7faf6b7fe640 1 -- 172.21.15.60:0/615004532 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6402ca70 con 0x7faf6c1416f0 2024-03-27T21:39:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.247+0000 7faf6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.247+0000 7faf6b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.247+0000 7faf6b7fe640 10 monclient: dump: 2024-03-27T21:39:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.246 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.247+0000 7faf6b7fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.247+0000 7faf6b7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.247+0000 7faf6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.247+0000 7faf6b7fe640 1 -- 172.21.15.60:0/615004532 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6402cbd0 con 0x7faf6c1416f0 2024-03-27T21:39:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.247+0000 7faf72d8e640 5 monclient: authenticate success, global_id 4256 2024-03-27T21:39:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.247+0000 7faf72d8e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.247+0000 7faf6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.247+0000 7faf72d8e640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.247+0000 7faf6b7fe640 1 -- 172.21.15.60:0/615004532 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf640343f0 con 0x7faf6c1416f0 2024-03-27T21:39:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.247+0000 7faf6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.247+0000 7faf72d8e640 4 set_mon_vals no callback set 2024-03-27T21:39:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.247+0000 7faf6b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.247+0000 7faf6b7fe640 10 monclient: dump: 2024-03-27T21:39:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.247 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.248+0000 7faf72d8e640 10 monclient: shutdown 2024-03-27T21:39:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.248+0000 7faf72d8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.248+0000 7faf72d8e640 1 -- 172.21.15.60:0/615004532 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faf6c1416f0 msgr2=0x7faf6c141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.248+0000 7faf72d8e640 1 --2- 172.21.15.60:0/615004532 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faf6c1416f0 0x7faf6c141b80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faf64009e50 tx=0x7faf6402bed0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.248+0000 7faf72d8e640 1 -- 172.21.15.60:0/615004532 shutdown_connections 2024-03-27T21:39:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.248+0000 7faf72d8e640 1 --2- 172.21.15.60:0/615004532 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faf6c1416f0 0x7faf6c141b80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.248+0000 7faf72d8e640 1 --2- 172.21.15.60:0/615004532 >> v2:172.21.15.191:3300/0 conn(0x7faf6c13b020 0x7faf6c13b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.248+0000 7faf72d8e640 1 -- 172.21.15.60:0/615004532 >> 172.21.15.60:0/615004532 conn(0x7faf6c0608e0 msgr2=0x7faf6c060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.248+0000 7faf72d8e640 1 -- 172.21.15.60:0/615004532 shutdown_connections 2024-03-27T21:39:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.248+0000 7faf72d8e640 1 -- 172.21.15.60:0/615004532 wait complete. 2024-03-27T21:39:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.248+0000 7faf72d8e640 10 monclient: build_initial_monmap 2024-03-27T21:39:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.248+0000 7faf72d8e640 10 monclient: monmap: 2024-03-27T21:39:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:47.249 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.249+0000 7faf72d8e640 1 librados: starting msgr at 2024-03-27T21:39:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.249+0000 7faf72d8e640 1 librados: starting objecter 2024-03-27T21:39:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.249+0000 7faf72d8e640 1 Processor -- start 2024-03-27T21:39:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.249+0000 7faf72d8e640 1 -- start start 2024-03-27T21:39:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.249+0000 7faf72d8e640 1 librados: setting wanted keys 2024-03-27T21:39:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.249+0000 7faf72d8e640 1 librados: calling monclient init 2024-03-27T21:39:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.249+0000 7faf72d8e640 10 monclient: init 2024-03-27T21:39:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.249+0000 7faf72d8e640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.249+0000 7faf72d8e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:39:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.249+0000 7faf72d8e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faf6c0b40b0 0x7faf6c139fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.249+0000 7faf72d8e640 10 monclient(hunting): picked mon.noname-c con 0x7faf6c0b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.249+0000 7faf72d8e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf6c13b020 0x7faf6c1d61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.249+0000 7faf72d8e640 10 monclient(hunting): picked mon.noname-b con 0x7faf6c13b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.249+0000 7faf72d8e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf6c1416f0 0x7faf6c1d6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.249+0000 7faf72d8e640 10 monclient(hunting): picked mon.noname-a con 0x7faf6c1416f0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf72d8e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf72d8e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c13aa90 con 0x7faf6c1416f0 2024-03-27T21:39:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf72d8e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf72d8e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c133e10 con 0x7faf6c13b020 2024-03-27T21:39:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf72d8e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf72d8e640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c13e090 con 0x7faf6c0b40b0 2024-03-27T21:39:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf72d8e640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf72d8e640 10 monclient(hunting): authenticate will time out at 828.471937s 2024-03-27T21:39:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf70b2a640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faf6c0b40b0 0x7faf6c139fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf70b2a640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faf6c0b40b0 0x7faf6c139fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47512/0 (socket says 172.21.15.60:47512) 2024-03-27T21:39:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf7132b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf6c1416f0 0x7faf6c1d6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf70b2a640 1 -- 172.21.15.60:0/90312388 learned_addr learned my addr 172.21.15.60:0/90312388 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf70b2a640 10 monclient(hunting): get_auth_request con 0x7faf6c0b40b0 auth_method 0 2024-03-27T21:39:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf70b2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf70b2a640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf70b2a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf7132b640 10 monclient(hunting): get_auth_request con 0x7faf6c1416f0 auth_method 0 2024-03-27T21:39:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf7132b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf7132b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf7132b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf70b2a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf70b2a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf70b2a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf7132b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf7132b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf7132b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf70b2a640 10 monclient(hunting): handle_auth_done global_id 4259 payload 824 2024-03-27T21:39:47.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf6bfff640 1 --2- 172.21.15.60:0/90312388 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf6c13b020 0x7faf6c1d61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf70b2a640 1 -- 172.21.15.60:0/90312388 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf6c13b020 msgr2=0x7faf6c1d61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf70b2a640 1 --2- 172.21.15.60:0/90312388 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf6c13b020 0x7faf6c1d61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf70b2a640 1 -- 172.21.15.60:0/90312388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf6c1416f0 msgr2=0x7faf6c1d6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf70b2a640 1 --2- 172.21.15.60:0/90312388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf6c1416f0 0x7faf6c1d6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf70b2a640 10 monclient: _finish_hunting 0 2024-03-27T21:39:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf70b2a640 1 monclient: found mon.noname-c 2024-03-27T21:39:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf70b2a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf70b2a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:39:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf70b2a640 1 -- 172.21.15.60:0/90312388 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf64009b30 con 0x7faf6c0b40b0 2024-03-27T21:39:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf7132b640 1 --2- 172.21.15.60:0/90312388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf6c1416f0 0x7faf6c1d6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf70b2a640 1 --2- 172.21.15.60:0/90312388 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faf6c0b40b0 0x7faf6c139fc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faf6000b3f0 tx=0x7faf6000b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf697fa640 10 client.?.objecter ms_handle_connect 0x7faf6c0b40b0 2024-03-27T21:39:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf697fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf697fa640 1 -- 172.21.15.60:0/90312388 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf60011830 con 0x7faf6c0b40b0 2024-03-27T21:39:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf697fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf697fa640 10 monclient: dump: 2024-03-27T21:39:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.254 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.254 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.254 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.254 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.254 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.254 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.254 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf697fa640 10 monclient: _finish_auth 0 2024-03-27T21:39:47.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf697fa640 10 monclient: _check_auth_tickets 2024-03-27T21:39:47.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:47.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf72d8e640 5 monclient: authenticate success, global_id 4259 2024-03-27T21:39:47.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf697fa640 1 -- 172.21.15.60:0/90312388 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf60011990 con 0x7faf6c0b40b0 2024-03-27T21:39:47.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf697fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:47.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf697fa640 1 -- 172.21.15.60:0/90312388 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf60011d40 con 0x7faf6c0b40b0 2024-03-27T21:39:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf72d8e640 10 monclient: _renew_subs 2024-03-27T21:39:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf72d8e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf72d8e640 1 -- 172.21.15.60:0/90312388 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf6c13d7f0 con 0x7faf6c0b40b0 2024-03-27T21:39:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf6affd640 4 set_mon_vals no callback set 2024-03-27T21:39:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf697fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf697fa640 10 monclient: dump: 2024-03-27T21:39:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.256 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.256 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.256 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf72d8e640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:47.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf72d8e640 10 monclient: _renew_subs 2024-03-27T21:39:47.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf72d8e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:47.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf72d8e640 1 -- 172.21.15.60:0/90312388 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf6c13dc00 con 0x7faf6c0b40b0 2024-03-27T21:39:47.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.250+0000 7faf72d8e640 1 librados: init done 2024-03-27T21:39:47.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.251+0000 7faf72d8e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:47.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.251+0000 7faf72d8e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:47.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.251+0000 7faf72d8e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:47.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.251+0000 7faf72d8e640 1 -- 172.21.15.60:0/90312388 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf6c13de80 con 0x7faf6c0b40b0 2024-03-27T21:39:47.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.252+0000 7faf697fa640 1 -- 172.21.15.60:0/90312388 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7faf6001a400 con 0x7faf6c0b40b0 2024-03-27T21:39:47.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.252+0000 7faf697fa640 1 --2- 172.21.15.60:0/90312388 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7faf5006c590 0x7faf5006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.252+0000 7faf697fa640 1 -- 172.21.15.60:0/90312388 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7faf60088e80 con 0x7faf6c0b40b0 2024-03-27T21:39:47.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.252+0000 7faf697fa640 10 client.4259.objecter ms_dispatch 0x7faf6c000b90 osd_map(20..20 src has 1..20) v4 2024-03-27T21:39:47.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.252+0000 7faf697fa640 3 client.4259.objecter handle_osd_map got epochs [20,20] > 0 2024-03-27T21:39:47.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.252+0000 7faf697fa640 3 client.4259.objecter handle_osd_map decoding full epoch 20 2024-03-27T21:39:47.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.252+0000 7faf6bfff640 1 --2- 172.21.15.60:0/90312388 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7faf5006c590 0x7faf5006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.252+0000 7faf6bfff640 10 monclient: get_auth_request con 0x7faf5006c590 auth_method 0 2024-03-27T21:39:47.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.252+0000 7faf697fa640 20 client.4259.objecter dump_active .. 0 homeless 2024-03-27T21:39:47.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.253+0000 7faf697fa640 10 client.4259.objecter ms_handle_connect 0x7faf5006c590 2024-03-27T21:39:47.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.253+0000 7faf6bfff640 1 --2- 172.21.15.60:0/90312388 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7faf5006c590 0x7faf5006ea50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7faf6c0b5d20 tx=0x7faf58008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.256+0000 7faf697fa640 1 -- 172.21.15.60:0/90312388 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faf600542e0 con 0x7faf6c0b40b0 2024-03-27T21:39:47.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.256+0000 7faf697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:47.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.256+0000 7faf697fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:47.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.439+0000 7faf72d8e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:39:47.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.439+0000 7faf72d8e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:39:47.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.439+0000 7faf72d8e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:47.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.439+0000 7faf72d8e640 1 -- 172.21.15.60:0/90312388 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf6c069470 con 0x7faf6c0b40b0 2024-03-27T21:39:47.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.440+0000 7faf697fa640 1 -- 172.21.15.60:0/90312388 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+14181 (secure 0 0 0) 0x7faf60057900 con 0x7faf6c0b40b0 2024-03-27T21:39:47.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.440+0000 7faf697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:39:47.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.440+0000 7faf697fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:47.428 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:39:47.428 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"fsid":"dcd19d9a-535c-4b41-9095-d588268e2860","created":"2024-03-27T21:39:27.092191+0000","modified":"2024-03-27T21:39:46.450733+0000","last_up_change":"2024-03-27T21:39:38.411432+0000","last_in_change":"2024-03-27T21:39:29.291620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T21:39:37.438493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-27T21:39:45.042103+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"778aaa64-6720-495b-bcdc-86fbcece9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":34603},{"type":"v1","addr":"172.21.15.60:6817","nonce":34603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":34603},{"type":"v1","addr":"172.21.15.60:6819","nonce":34603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":34603},{"type":"v1","addr":"172.21.15.60:6823","nonce":34603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":34603},{"type":"v1","addr":"172.21.15.60:6821","nonce":34603}]},"public_addr":"172.21.15.60:6817/34603","cluster_addr":"172.21.15.60:6819/34603","heartbeat_back_addr":"172.21.15.60:6823/34603","heartbeat_front_addr":"172.21.15.60:6821/34603","state":["exists","up"]},{"osd":1,"uuid":"d960adf1-3d83-4e24-a612-8df53fa43d0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":34602},{"type":"v1","addr":"172.21.15.60:6801","nonce":34602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":34602},{"type":"v1","addr":"172.21.15.60:6803","nonce":34602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":34602},{"type":"v1","addr":"172.21.15.60:6807","nonce":34602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":34602},{"type":"v1","addr":"172.21.15.60:6805","nonce":34602}]},"public_addr":"172.21.15.60:6801/34602","cluster_addr":"172.21.15.60:6803/34602","heartbeat_back_addr":"172.21.15.60:6807/34602","heartbeat_front_addr":"172.21.15.60:6805/34602","state":["exists","up"]},{"osd":2,"uuid":"b8ffc48b-cf6f-4701-a6aa-c25fdb0f7977","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":34604},{"type":"v1","addr":"172.21.15.60:6825","nonce":34604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":34604},{"type":"v1","addr":"172.21.15.60:6827","nonce":34604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":34604},{"type":"v1","addr":"172.21.15.60:6831","nonce":34604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":34604},{"type":"v1","addr":"172.21.15.60:6829","nonce":34604}]},"public_addr":"172.21.15.60:6825/34604","cluster_addr":"172.21.15.60:6827/34604","heartbeat_back_addr":"172.21.15.60:6831/34604","heartbeat_front_addr":"172.21.15.60:6829/34604","state":["exists","up"]},{"osd":3,"uuid":"0f14f2d8-b271-48e4-a198-aa5279a01c06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":34613},{"type":"v1","addr":"172.21.15.60:6809","nonce":34613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":34613},{"type":"v1","addr":"172.21.15.60:6811","nonce":34613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":34613},{"type":"v1","addr":"172.21.15.60:6815","nonce":34613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":34613},{"type":"v1","addr":"172.21.15.60:6813","nonce":34613}]},"public_addr":"172.21.15.60:6809/34613","cluster_addr":"172.21.15.60:6811/34613","heartbeat_back_addr":"172.21.15.60:6815/34613","heartbeat_front_addr":"172.21.15.60:6813/34613","state":["exists","up"]},{"osd":4,"uuid":"8cdeb7ae-1c6e-4d0d-ba44-80248eff2fb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":34332},{"type":"v1","addr":"172.21.15.191:6801","nonce":34332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":34332},{"type":"v1","addr":"172.21.15.191:6803","nonce":34332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":34332},{"type":"v1","addr":"172.21.15.191:6807","nonce":34332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":34332},{"type":"v1","addr":"172.21.15.191:6805","nonce":34332}]},"public_addr":"172.21.15.191:6801/34332","cluster_addr":"172.21.15.191:6803/34332","heartbeat_back_addr":"172.21.15.191:6807/34332","heartbeat_front_addr":"172.21.15.191:6805/34332","state":["exists","up"]},{"osd":5,"uuid":"93748cf8-d3fe-46b8-950f-e01e28a5e02c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6809","nonce":34333},{"type":"v1","addr":"172.21.15.191:6811","nonce":34333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6813","nonce":34333},{"type":"v1","addr":"172.21.15.191:6816","nonce":34333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":34333},{"type":"v1","addr":"172.21.15.191:6823","nonce":34333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":34333},{"type":"v1","addr":"172.21.15.191:6820","nonce":34333}]},"public_addr":"172.21.15.191:6811/34333","cluster_addr":"172.21.15.191:6816/34333","heartbeat_back_addr":"172.21.15.191:6823/34333","heartbeat_front_addr":"172.21.15.191:6820/34333","state":["exists","up"]},{"osd":6,"uuid":"1ad21434-cbc5-4de9-8d63-76f894f7e736","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":34361},{"type":"v1","addr":"172.21.15.191:6825","nonce":34361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":34361},{"type":"v1","addr":"172.21.15.191:6827","nonce":34361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":34361},{"type":"v1","addr":"172.21.15.191:6831","nonce":34361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":34361},{"type":"v1","addr":"172.21.15.191:6829","nonce":34361}]},"public_addr":"172.21.15.191:6825/34361","cluster_addr":"172.21.15.191:6827/34361","heartbeat_back_addr":"172.21.15.191:6831/34361","heartbeat_front_addr":"172.21.15.191:6829/34361","state":["exists","up"]},{"osd":7,"uuid":"9fbd0927-ee2f-471b-8df8-c4e0fc40b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":34334},{"type":"v1","addr":"172.21.15.191:6810","nonce":34334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":34334},{"type":"v1","addr":"172.21.15.191:6814","nonce":34334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6819","nonce":34334},{"type":"v1","addr":"172.21.15.191:6821","nonce":34334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6815","nonce":34334},{"type":"v1","addr":"172.21.15.191:6817","nonce":34334}]},"public_addr":"172.21.15.191:6810/34334","cluster_addr":"172.21.15.191:6814/34334","heartbeat_back_addr":"172.21.15.191:6821/34334","heartbeat_front_addr":"172.21.15.191:6817/34334","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.476097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.411243+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.458154+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.415285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.431201+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.456013+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.474448+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.414076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T21:39:47.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.442+0000 7faf72d8e640 10 librados: watch_flush enter 2024-03-27T21:39:47.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.442+0000 7faf72d8e640 10 librados: watch_flush exit 2024-03-27T21:39:47.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.442+0000 7faf72d8e640 20 client.4259.objecter shutdown clearing up homeless session... 2024-03-27T21:39:47.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.442+0000 7faf72d8e640 10 client.4259.objecter successfully canceled tick 2024-03-27T21:39:47.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.442+0000 7faf72d8e640 1 -- 172.21.15.60:0/90312388 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7faf5006c590 msgr2=0x7faf5006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.442+0000 7faf72d8e640 1 --2- 172.21.15.60:0/90312388 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7faf5006c590 0x7faf5006ea50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7faf6c0b5d20 tx=0x7faf58008040 comp rx=0 tx=0).stop 2024-03-27T21:39:47.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.442+0000 7faf72d8e640 10 monclient: shutdown 2024-03-27T21:39:47.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.442+0000 7faf72d8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:47.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.442+0000 7faf72d8e640 1 -- 172.21.15.60:0/90312388 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faf6c0b40b0 msgr2=0x7faf6c139fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.442+0000 7faf72d8e640 1 --2- 172.21.15.60:0/90312388 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faf6c0b40b0 0x7faf6c139fc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faf6000b3f0 tx=0x7faf6000b8b0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.442+0000 7faf72d8e640 1 -- 172.21.15.60:0/90312388 shutdown_connections 2024-03-27T21:39:47.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.442+0000 7faf72d8e640 1 --2- 172.21.15.60:0/90312388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf6c1416f0 0x7faf6c1d6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.442+0000 7faf72d8e640 1 --2- 172.21.15.60:0/90312388 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7faf5006c590 0x7faf5006ea50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.442+0000 7faf72d8e640 1 --2- 172.21.15.60:0/90312388 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf6c13b020 0x7faf6c1d61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.442+0000 7faf72d8e640 1 --2- 172.21.15.60:0/90312388 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faf6c0b40b0 0x7faf6c139fc0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.442+0000 7faf72d8e640 1 -- 172.21.15.60:0/90312388 >> 172.21.15.60:0/90312388 conn(0x7faf6c0608e0 msgr2=0x7faf6c0b53e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:47.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.442+0000 7faf72d8e640 1 -- 172.21.15.60:0/90312388 shutdown_connections 2024-03-27T21:39:47.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.442+0000 7faf72d8e640 1 -- 172.21.15.60:0/90312388 wait complete. 2024-03-27T21:39:47.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.442+0000 7faf72d8e640 1 librados: shutdown 2024-03-27T21:39:47.441 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-27T21:39:47.442 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-27T21:39:47.442 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-27T21:39:47.443 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-27T21:39:47.443 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-27T21:39:47.444 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-27T21:39:47.444 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-27T21:39:47.445 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-27T21:39:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.625+0000 7f9678e3b640 10 monclient: get_monmap_and_config 2024-03-27T21:39:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.625+0000 7f9678e3b640 10 monclient: build_initial_monmap 2024-03-27T21:39:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.626+0000 7f9678e3b640 10 monclient: monmap: 2024-03-27T21:39:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:47.627989+0000 2024-03-27T21:39:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:47.627989+0000 2024-03-27T21:39:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:47.617 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.627+0000 7f9678e3b640 1 Processor -- start 2024-03-27T21:39:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.627+0000 7f9678e3b640 1 -- start start 2024-03-27T21:39:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.627+0000 7f9678e3b640 10 monclient: init 2024-03-27T21:39:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9678e3b640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9678e3b640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-03-27T21:39:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9678e3b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f96740a8ba0 0x7f96740ad4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9678e3b640 10 monclient(hunting): picked mon.noname-a con 0x7f96740a8ba0 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9678e3b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f96740ad9f0 0x7f9674151860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9678e3b640 10 monclient(hunting): picked mon.noname-b con 0x7f96740ad9f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9678e3b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9674151da0 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9678e3b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96740a9330 con 0x7f9674151da0 2024-03-27T21:39:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9678e3b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9678e3b640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96740a9f50 con 0x7f96740a8ba0 2024-03-27T21:39:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9678e3b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9678e3b640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96740aab10 con 0x7f96740ad9f0 2024-03-27T21:39:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9678e3b640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9678e3b640 10 monclient(hunting): authenticate will time out at 828.850038s 2024-03-27T21:39:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f7296d42640 10 monclient: get_monmap_and_config 2024-03-27T21:39:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f7296d42640 10 monclient: build_initial_monmap 2024-03-27T21:39:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f7296d42640 10 monclient: monmap: 2024-03-27T21:39:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:47.630128+0000 2024-03-27T21:39:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:47.630128+0000 2024-03-27T21:39:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:47.620 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7f07f1d30640 10 monclient: get_monmap_and_config 2024-03-27T21:39:47.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7f07f1d30640 10 monclient: build_initial_monmap 2024-03-27T21:39:47.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7f07f1d30640 10 monclient: monmap: 2024-03-27T21:39:47.621 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:47.621 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.621 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:47.631148+0000 2024-03-27T21:39:47.621 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:47.631148+0000 2024-03-27T21:39:47.621 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:47.622 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f40cf071640 10 monclient: get_monmap_and_config 2024-03-27T21:39:47.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7f7296d42640 1 Processor -- start 2024-03-27T21:39:47.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7f7296d42640 1 -- start start 2024-03-27T21:39:47.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7f7296d42640 10 monclient: init 2024-03-27T21:39:47.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f40cf071640 10 monclient: build_initial_monmap 2024-03-27T21:39:47.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f40cf071640 10 monclient: monmap: 2024-03-27T21:39:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:47.630797+0000 2024-03-27T21:39:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:47.630797+0000 2024-03-27T21:39:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:47.625 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9672ffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f96740ad9f0 0x7f9674151860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9672ffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f96740ad9f0 0x7f9674151860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47516/0 (socket says 172.21.15.60:47516) 2024-03-27T21:39:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9672ffd640 1 -- 172.21.15.60:0/3627026996 learned_addr learned my addr 172.21.15.60:0/3627026996 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9672ffd640 10 monclient(hunting): get_auth_request con 0x7f96740ad9f0 auth_method 0 2024-03-27T21:39:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9672ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9672ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9672ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f96737fe640 1 --2- 172.21.15.60:0/3627026996 >> v2:172.21.15.60:3300/0 conn(0x7f96740a8ba0 0x7f96740ad4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f96737fe640 10 monclient(hunting): get_auth_request con 0x7f96740a8ba0 auth_method 0 2024-03-27T21:39:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f96737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f96737fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f96737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9672ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9672ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.628+0000 7f9672ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7f96737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7f96737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7f96737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7f9672ffd640 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2024-03-27T21:39:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7f9672ffd640 1 -- 172.21.15.60:0/3627026996 >> v2:172.21.15.60:3300/0 conn(0x7f96740a8ba0 msgr2=0x7f96740ad4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7f9672ffd640 1 --2- 172.21.15.60:0/3627026996 >> v2:172.21.15.60:3300/0 conn(0x7f96740a8ba0 0x7f96740ad4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7f9672ffd640 1 -- 172.21.15.60:0/3627026996 >> v1:172.21.15.60:6789/0 conn(0x7f9674151da0 legacy=0x7f96740ab4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7fed1b563640 10 monclient: get_monmap_and_config 2024-03-27T21:39:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7fed1b563640 10 monclient: build_initial_monmap 2024-03-27T21:39:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7fed1b563640 10 monclient: monmap: 2024-03-27T21:39:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:47.628 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.628 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:47.631983+0000 2024-03-27T21:39:47.628 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:47.631983+0000 2024-03-27T21:39:47.628 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:47.628 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.628 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:47.628 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:47.628 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:47.628 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:47.628 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:47.629 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7f2662fca640 10 monclient: get_monmap_and_config 2024-03-27T21:39:47.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7f2662fca640 10 monclient: build_initial_monmap 2024-03-27T21:39:47.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7f2662fca640 10 monclient: monmap: 2024-03-27T21:39:47.629 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:47.629 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.630 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:47.631462+0000 2024-03-27T21:39:47.630 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:47.631462+0000 2024-03-27T21:39:47.630 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:47.630 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.630 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:47.630 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:47.630 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:47.630 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:47.630 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:47.630 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7ff39b0e0640 10 monclient: get_monmap_and_config 2024-03-27T21:39:47.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7ff39b0e0640 10 monclient: build_initial_monmap 2024-03-27T21:39:47.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7ff39b0e0640 10 monclient: monmap: 2024-03-27T21:39:47.631 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:47.631 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.631 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:47.631726+0000 2024-03-27T21:39:47.631 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:47.631726+0000 2024-03-27T21:39:47.631 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:47.631 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.631 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:47.631 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:47.632 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:47.632 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:47.632 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:47.632 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7f9672ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:39:47.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7f9672ffd640 1 monclient: found mon.noname-b 2024-03-27T21:39:47.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7f9672ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:47.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7f9672ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:47.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.629+0000 7f9672ffd640 1 -- 172.21.15.60:0/3627026996 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96740aadf0 con 0x7f96740ad9f0 2024-03-27T21:39:47.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f3ddbfff640 10 monclient: get_monmap_and_config 2024-03-27T21:39:47.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f3ddbfff640 10 monclient: build_initial_monmap 2024-03-27T21:39:47.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f3ddbfff640 10 monclient: monmap: 2024-03-27T21:39:47.633 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:47.633 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.633 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:47.632272+0000 2024-03-27T21:39:47.634 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:47.632272+0000 2024-03-27T21:39:47.634 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:47.634 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.634 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:47.634 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:47.634 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:47.634 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:47.634 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:47.634 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f40cf071640 1 Processor -- start 2024-03-27T21:39:47.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f7296d42640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:47.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f7296d42640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-03-27T21:39:47.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f7296d42640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f72900b0ec0 0x7f7290142970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f7296d42640 10 monclient(hunting): picked mon.noname-c con 0x7f72900b0ec0 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:47.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f7296d42640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7290142eb0 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:47.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f7296d42640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f72901493e0 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:47.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f7296d42640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72900b2760 con 0x7f72901493e0 2024-03-27T21:39:47.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f7296d42640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7290143320 con 0x7f7290142eb0 2024-03-27T21:39:47.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f7296d42640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f7296d42640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f729006cef0 con 0x7f72900b0ec0 2024-03-27T21:39:47.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f7296d42640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:47.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f7296d42640 10 monclient(hunting): authenticate will time out at 828.852206s 2024-03-27T21:39:47.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f07f1d30640 1 Processor -- start 2024-03-27T21:39:47.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f9672ffd640 1 --2- 172.21.15.60:0/3627026996 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f96740ad9f0 0x7f9674151860 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f966400a900 tx=0x7f966400a3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a24c313fe7554e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7fed1b563640 1 Processor -- start 2024-03-27T21:39:47.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7fed1b563640 1 -- start start 2024-03-27T21:39:47.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7fed1b563640 10 monclient: init 2024-03-27T21:39:47.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f07f1d30640 1 -- start start 2024-03-27T21:39:47.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f07f1d30640 10 monclient: init 2024-03-27T21:39:47.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f72952df640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f72901493e0 0x7f7290150060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43530/0 (socket says 172.21.15.60:43530) 2024-03-27T21:39:47.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f72952df640 1 -- 172.21.15.60:0/837238915 learned_addr learned my addr 172.21.15.60:0/837238915 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:39:47.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f96727fc640 1 -- 172.21.15.60:0/3627026996 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966402d8f0 con 0x7f96740ad9f0 2024-03-27T21:39:47.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f96727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f96727fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:47.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f96727fc640 10 monclient: dump: 2024-03-27T21:39:47.639 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.639 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.640 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f96727fc640 10 monclient: _finish_auth 0 2024-03-27T21:39:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f96727fc640 10 monclient: _check_auth_tickets 2024-03-27T21:39:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f96727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f9678e3b640 5 monclient: authenticate success, global_id 4262 2024-03-27T21:39:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f9678e3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f96727fc640 1 -- 172.21.15.60:0/3627026996 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f966402da50 con 0x7f96740ad9f0 2024-03-27T21:39:47.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f96727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:47.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f9678e3b640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:47.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f9678e3b640 4 set_mon_vals no callback set 2024-03-27T21:39:47.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f96727fc640 1 -- 172.21.15.60:0/3627026996 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966402de00 con 0x7f96740ad9f0 2024-03-27T21:39:47.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f96727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f96727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:47.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f96727fc640 10 monclient: dump: 2024-03-27T21:39:47.641 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.641 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.641 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.641 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.641 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.641 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.641 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.642 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.642 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.642 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f40cf071640 1 -- start start 2024-03-27T21:39:47.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f40cf071640 10 monclient: init 2024-03-27T21:39:47.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f40cf071640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:47.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f40cf071640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-03-27T21:39:47.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f40cf071640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f40c8140bd0 0x7f40c81414d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f40cf071640 10 monclient(hunting): picked mon.noname-b con 0x7f40c8140bd0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:47.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f40cf071640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f40c806ec20 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:47.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f40cf071640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f40c813a6f0 0x7f40c813ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f40cf071640 10 monclient(hunting): picked mon.noname-c con 0x7f40c813a6f0 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:47.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f40cf071640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c813b080 con 0x7f40c806ec20 2024-03-27T21:39:47.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f40cf071640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f40cf071640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c806e230 con 0x7f40c813a6f0 2024-03-27T21:39:47.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f40cf071640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f40cf071640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c806ff80 con 0x7f40c8140bd0 2024-03-27T21:39:47.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f40cf071640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:47.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f40cf071640 10 monclient(hunting): authenticate will time out at 828.853690s 2024-03-27T21:39:47.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f3ddbfff640 1 Processor -- start 2024-03-27T21:39:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f3ddbfff640 1 -- start start 2024-03-27T21:39:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f3ddbfff640 10 monclient: init 2024-03-27T21:39:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7ff39b0e0640 1 Processor -- start 2024-03-27T21:39:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7ff39b0e0640 1 -- start start 2024-03-27T21:39:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7ff39b0e0640 10 monclient: init 2024-03-27T21:39:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7ff39b0e0640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7ff39b0e0640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-03-27T21:39:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f2662fca640 1 Processor -- start 2024-03-27T21:39:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f2662fca640 1 -- start start 2024-03-27T21:39:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.630+0000 7f2662fca640 10 monclient: init 2024-03-27T21:39:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f2662fca640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f2662fca640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-03-27T21:39:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f2662fca640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f265c06d700 0x7f265c06eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f2662fca640 10 monclient(hunting): picked mon.noname-c con 0x7f265c06d700 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f2662fca640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f265c13e000 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f2662fca640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f265c0ae6a0 0x7f265c0ad6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f2662fca640 10 monclient(hunting): picked mon.noname-b con 0x7f265c0ae6a0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f2662fca640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f265c0af590 con 0x7f265c13e000 2024-03-27T21:39:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f2662fca640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f2662fca640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f265c13d840 con 0x7f265c06d700 2024-03-27T21:39:47.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f2662fca640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f2662fca640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f265c13dae0 con 0x7f265c0ae6a0 2024-03-27T21:39:47.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f2662fca640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:47.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f2662fca640 10 monclient(hunting): authenticate will time out at 828.853699s 2024-03-27T21:39:47.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f2661567640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f265c0ae6a0 0x7f265c0ad6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f72877fe640 1 -- 172.21.15.60:0/837238915 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2625625075 0 0) 0x7f72900b2760 con 0x7f72901493e0 2024-03-27T21:39:47.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f72877fe640 10 monclient(hunting): _negotiate 2024-03-27T21:39:47.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f72877fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f72877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f72877fe640 10 monclient(hunting): my global_id is 4284 2024-03-27T21:39:47.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7f72877fe640 1 -- 172.21.15.60:0/837238915 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7278003520 con 0x7f72901493e0 2024-03-27T21:39:47.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f7294ade640 1 --2- 172.21.15.60:0/837238915 >> v2:172.21.15.191:3300/0 conn(0x7f72900b0ec0 0x7f7290142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f2661567640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f265c0ae6a0 0x7f265c0ad6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47526/0 (socket says 172.21.15.60:47526) 2024-03-27T21:39:47.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f2661567640 1 -- 172.21.15.60:0/997409329 learned_addr learned my addr 172.21.15.60:0/997409329 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:47.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f2661567640 10 monclient(hunting): get_auth_request con 0x7f265c0ae6a0 auth_method 0 2024-03-27T21:39:47.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f2661567640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f2661567640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f2661567640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f7294ade640 10 monclient(hunting): get_auth_request con 0x7f72900b0ec0 auth_method 0 2024-03-27T21:39:47.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f7294ade640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f7294ade640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f7294ade640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f2661567640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f2661567640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddbfff640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:47.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddbfff640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-03-27T21:39:47.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddbfff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3ddc060970 0x7f3ddc0b0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddbfff640 10 monclient(hunting): picked mon.noname-c con 0x7f3ddc060970 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:47.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddbfff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3ddc0adfe0 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:47.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3ddc06e710 0x7f3ddc071670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddbfff640 10 monclient(hunting): picked mon.noname-a con 0x7f3ddc06e710 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:47.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddbfff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ddc0afbc0 con 0x7f3ddc0adfe0 2024-03-27T21:39:47.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddbfff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddbfff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ddc060d50 con 0x7f3ddc06e710 2024-03-27T21:39:47.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddbfff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddbfff640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ddc1432c0 con 0x7f3ddc060970 2024-03-27T21:39:47.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddbfff640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:47.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddbfff640 10 monclient(hunting): authenticate will time out at 828.854303s 2024-03-27T21:39:47.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f7294ade640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f7294ade640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f7294ade640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f40cd60e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f40c813a6f0 0x7f40c813ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7ff39b0e0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff3940b0ec0 0x7ff3941498b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7ff39b0e0640 10 monclient(hunting): picked mon.noname-a con 0x7ff3940b0ec0 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:47.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7ff39b0e0640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff3940b18e0 0x7ff394142ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7ff39b0e0640 10 monclient(hunting): picked mon.noname-c con 0x7ff3940b18e0 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:47.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.631+0000 7ff39b0e0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff3940b1d30 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:47.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7ff39b0e0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff39406cef0 con 0x7ff3940b1d30 2024-03-27T21:39:47.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7ff39b0e0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7ff39b0e0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3940b12a0 con 0x7ff3940b0ec0 2024-03-27T21:39:47.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7ff39b0e0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7ff39b0e0640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff39406d140 con 0x7ff3940b18e0 2024-03-27T21:39:47.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7ff39b0e0640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:47.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7ff39b0e0640 10 monclient(hunting): authenticate will time out at 828.853938s 2024-03-27T21:39:47.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f40c7fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f40c806ec20 0x7f40c80adc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.60:49028/0 (socket says 172.21.15.60:49028) 2024-03-27T21:39:47.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f40c7fff640 1 -- 172.21.15.60:0/2324141607 learned_addr learned my addr 172.21.15.60:0/2324141607 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:39:47.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f9678e3b640 10 monclient: shutdown 2024-03-27T21:39:47.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f9678e3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:47.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f9678e3b640 1 -- 172.21.15.60:0/3627026996 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f96740ad9f0 msgr2=0x7f9674151860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f9678e3b640 1 --2- 172.21.15.60:0/3627026996 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f96740ad9f0 0x7f9674151860 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f966400a900 tx=0x7f966400a3f0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f9678e3b640 1 -- 172.21.15.60:0/3627026996 shutdown_connections 2024-03-27T21:39:47.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f9678e3b640 1 --2- 172.21.15.60:0/3627026996 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f96740ad9f0 0x7f9674151860 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f966400a900 tx=0x7f966400a3f0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f9678e3b640 1 --2- 172.21.15.60:0/3627026996 >> v2:172.21.15.60:3300/0 conn(0x7f96740a8ba0 0x7f96740ad4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f9678e3b640 1 -- 172.21.15.60:0/3627026996 >> 172.21.15.60:0/3627026996 conn(0x7f9674060700 msgr2=0x7f9674060b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:47.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddb7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3ddc06e710 0x7f3ddc071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddb7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3ddc06e710 0x7f3ddc071670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47710/0 (socket says 172.21.15.60:47710) 2024-03-27T21:39:47.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddb7fe640 1 -- 172.21.15.60:0/3150557403 learned_addr learned my addr 172.21.15.60:0/3150557403 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:47.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddb7fe640 10 monclient(hunting): get_auth_request con 0x7f3ddc06e710 auth_method 0 2024-03-27T21:39:47.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f2660d66640 1 --2- 172.21.15.60:0/997409329 >> v2:172.21.15.191:3300/0 conn(0x7f265c06d700 0x7f265c06eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f2661567640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f2660d66640 10 monclient(hunting): get_auth_request con 0x7f265c06d700 auth_method 0 2024-03-27T21:39:47.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f2660d66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f2660d66640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f2660d66640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed1b563640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:47.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed1b563640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-03-27T21:39:47.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed1b563640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fed0c035d10 0x7fed0c0c4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed1b563640 10 monclient(hunting): picked mon.noname-c con 0x7fed0c035d10 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:47.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed1b563640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fed0c02b4a0 0x7fed0c0c5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed1b563640 10 monclient(hunting): picked mon.noname-b con 0x7fed0c02b4a0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:47.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed1b563640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed0c0cb330 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed1b563640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0c030c00 con 0x7fed0c0cb330 2024-03-27T21:39:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed1b563640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed1b563640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed0c02b220 con 0x7fed0c035d10 2024-03-27T21:39:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed1b563640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed1b563640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c0360f0 con 0x7fed0c02b4a0 2024-03-27T21:39:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed1b563640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed1b563640 10 monclient(hunting): authenticate will time out at 828.854192s 2024-03-27T21:39:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed19d60640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fed0c02b4a0 0x7fed0c0c5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed19d60640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fed0c02b4a0 0x7fed0c0c5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47530/0 (socket says 172.21.15.60:47530) 2024-03-27T21:39:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed19d60640 1 -- 172.21.15.60:0/234445324 learned_addr learned my addr 172.21.15.60:0/234445324 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed19d60640 10 monclient(hunting): get_auth_request con 0x7fed0c02b4a0 auth_method 0 2024-03-27T21:39:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed19d60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed19d60640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed19d60640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7fed1a561640 1 --2- 172.21.15.60:0/234445324 >> v2:172.21.15.191:3300/0 conn(0x7fed0c035d10 0x7fed0c0c4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f40cce0d640 1 --2- 172.21.15.60:0/2324141607 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f40c8140bd0 0x7f40c81414d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f40cce0d640 10 monclient(hunting): get_auth_request con 0x7f40c8140bd0 auth_method 0 2024-03-27T21:39:47.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f40cce0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f40cce0d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f40cce0d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f265b7fe640 1 -- 172.21.15.60:0/997409329 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078652570 0 0) 0x7f265c0af590 con 0x7f265c13e000 2024-03-27T21:39:47.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f265b7fe640 10 monclient(hunting): _negotiate 2024-03-27T21:39:47.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f40cd60e640 10 monclient(hunting): get_auth_request con 0x7f40c813a6f0 auth_method 0 2024-03-27T21:39:47.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f40cd60e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f40cd60e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f40cd60e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f265b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f265b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f265b7fe640 10 monclient(hunting): my global_id is 4294 2024-03-27T21:39:47.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f265b7fe640 1 -- 172.21.15.60:0/997409329 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2644003520 con 0x7f265c13e000 2024-03-27T21:39:47.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7fed19d60640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f2660d66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f7294ade640 10 monclient(hunting): handle_auth_done global_id 4288 payload 274 2024-03-27T21:39:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f7294ade640 1 -- 172.21.15.60:0/837238915 >> v1:172.21.15.191:6789/0 conn(0x7f7290142eb0 legacy=0x7f7290148cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f7294ade640 1 -- 172.21.15.60:0/837238915 >> v1:172.21.15.60:6789/0 conn(0x7f72901493e0 legacy=0x7f7290150060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f7294ade640 10 monclient: _finish_hunting 0 2024-03-27T21:39:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f7294ade640 1 monclient: found mon.noname-c 2024-03-27T21:39:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f7294ade640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f7294ade640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:39:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f7294ade640 1 -- 172.21.15.60:0/837238915 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f729006d0d0 con 0x7f72900b0ec0 2024-03-27T21:39:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7fed19d60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7fed19d60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f2660d66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7fed1a561640 10 monclient(hunting): get_auth_request con 0x7fed0c035d10 auth_method 0 2024-03-27T21:39:47.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f2660d66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7fed1a561640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7fed1a561640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7fed1a561640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40cce0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40cce0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40cce0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40c77fe640 1 -- 172.21.15.60:0/2324141607 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236118990 0 0) 0x7f40c813b080 con 0x7f40c806ec20 2024-03-27T21:39:47.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40c77fe640 10 monclient(hunting): _negotiate 2024-03-27T21:39:47.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40c77fe640 10 monclient(hunting): my global_id is 4303 2024-03-27T21:39:47.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40c77fe640 1 -- 172.21.15.60:0/2324141607 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40b8003520 con 0x7f40c806ec20 2024-03-27T21:39:47.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7ff393fff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff3940b18e0 0x7ff394142ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7ff393fff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff3940b18e0 0x7ff394142ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:56572/0 (socket says 172.21.15.60:56572) 2024-03-27T21:39:47.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7ff393fff640 1 -- 172.21.15.60:0/1438981634 learned_addr learned my addr 172.21.15.60:0/1438981634 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:47.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40cd60e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40cd60e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40cd60e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f07f1d30640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:47.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f07f1d30640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-03-27T21:39:47.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f07f1d30640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f07ec0a8b10 0x7f07ec142930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f07f1d30640 10 monclient(hunting): picked mon.noname-b con 0x7f07ec0a8b10 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:47.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f07f1d30640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07ec142e70 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:47.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f07f1d30640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f07ec0a8ef0 0x7f07ec150010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f07f1d30640 10 monclient(hunting): picked mon.noname-c con 0x7f07ec0a8ef0 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:47.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7ff393fff640 10 monclient(hunting): get_auth_request con 0x7ff3940b18e0 auth_method 0 2024-03-27T21:39:47.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7ff393fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7ff393fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7ff393fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f07f1d30640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07ec0aa7a0 con 0x7f07ec142e70 2024-03-27T21:39:47.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f07f1d30640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f07f1d30640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07ec145700 con 0x7f07ec0a8ef0 2024-03-27T21:39:47.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f07f1d30640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f07f1d30640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec0aaa80 con 0x7f07ec0a8b10 2024-03-27T21:39:47.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f07f1d30640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:47.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f07f1d30640 10 monclient(hunting): authenticate will time out at 828.854966s 2024-03-27T21:39:47.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f2661567640 10 monclient(hunting): handle_auth_done global_id 4265 payload 274 2024-03-27T21:39:47.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f2661567640 1 -- 172.21.15.60:0/997409329 >> v2:172.21.15.191:3300/0 conn(0x7f265c06d700 msgr2=0x7f265c06eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:47.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f2661567640 1 --2- 172.21.15.60:0/997409329 >> v2:172.21.15.191:3300/0 conn(0x7f265c06d700 0x7f265c06eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f2661567640 1 -- 172.21.15.60:0/997409329 >> v1:172.21.15.191:6789/0 conn(0x7f265c13e000 legacy=0x7f265c06dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f2661567640 10 monclient: _finish_hunting 0 2024-03-27T21:39:47.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f2661567640 1 monclient: found mon.noname-b 2024-03-27T21:39:47.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f2661567640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:47.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f2661567640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:47.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f2661567640 1 -- 172.21.15.60:0/997409329 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265c0aeaf0 con 0x7f265c0ae6a0 2024-03-27T21:39:47.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40cce0d640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-03-27T21:39:47.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40cce0d640 1 -- 172.21.15.60:0/2324141607 >> v2:172.21.15.191:3300/0 conn(0x7f40c813a6f0 msgr2=0x7f40c813ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:47.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40cce0d640 1 --2- 172.21.15.60:0/2324141607 >> v2:172.21.15.191:3300/0 conn(0x7f40c813a6f0 0x7f40c813ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40c77fe640 1 -- 172.21.15.60:0/2324141607 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1156433848 0 0) 0x7f40b8003520 con 0x7f40c806ec20 2024-03-27T21:39:47.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40cce0d640 1 -- 172.21.15.60:0/2324141607 >> v1:172.21.15.191:6789/0 conn(0x7f40c806ec20 legacy=0x7f40c80adc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7ff393fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7ff393fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7ff393fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40cce0d640 10 monclient: _finish_hunting 0 2024-03-27T21:39:47.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40cce0d640 1 monclient: found mon.noname-b 2024-03-27T21:39:47.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40cce0d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:47.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40cce0d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:47.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40cce0d640 1 -- 172.21.15.60:0/2324141607 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c80ae330 con 0x7f40c8140bd0 2024-03-27T21:39:47.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f40c77fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:39:47.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f9678e3b640 1 -- 172.21.15.60:0/3627026996 shutdown_connections 2024-03-27T21:39:47.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7ff398e7c640 1 --2- 172.21.15.60:0/1438981634 >> v2:172.21.15.60:3300/0 conn(0x7ff3940b0ec0 0x7ff3941498b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.632+0000 7f3ddaffd640 1 --2- 172.21.15.60:0/3150557403 >> v2:172.21.15.191:3300/0 conn(0x7f3ddc060970 0x7f3ddc0b0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f3ddaffd640 10 monclient(hunting): get_auth_request con 0x7f3ddc060970 auth_method 0 2024-03-27T21:39:47.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f3ddaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f07f0d2e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f07ec0a8b10 0x7f07ec142930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f3ddaffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f3ddaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f3ddaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f3ddaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f3ddaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f3ddaffd640 10 monclient(hunting): handle_auth_done global_id 4306 payload 274 2024-03-27T21:39:47.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f3ddaffd640 1 -- 172.21.15.60:0/3150557403 >> v2:172.21.15.60:3300/0 conn(0x7f3ddc06e710 msgr2=0x7f3ddc071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:47.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f3ddaffd640 1 --2- 172.21.15.60:0/3150557403 >> v2:172.21.15.60:3300/0 conn(0x7f3ddc06e710 0x7f3ddc071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f3ddaffd640 1 -- 172.21.15.60:0/3150557403 >> v1:172.21.15.60:6789/0 conn(0x7f3ddc0adfe0 legacy=0x7f3ddc145690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:47.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f07ebfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f07ec142e70 0x7f07ec145c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43554/0 (socket says 172.21.15.60:43554) 2024-03-27T21:39:47.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f07ebfff640 1 -- 172.21.15.60:0/4011084398 learned_addr learned my addr 172.21.15.60:0/4011084398 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:39:47.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f07f152f640 1 --2- 172.21.15.60:0/4011084398 >> v2:172.21.15.191:3300/0 conn(0x7f07ec0a8ef0 0x7f07ec150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f07f152f640 10 monclient(hunting): get_auth_request con 0x7f07ec0a8ef0 auth_method 0 2024-03-27T21:39:47.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f07f152f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f07f152f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f07f152f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f7294ade640 1 --2- 172.21.15.60:0/837238915 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72900b0ec0 0x7f7290142970 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f72800097a0 tx=0x7f728002b600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e350706bdecab9a server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7fed19d60640 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2024-03-27T21:39:47.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7fed19d60640 1 -- 172.21.15.60:0/234445324 >> v2:172.21.15.191:3300/0 conn(0x7fed0c035d10 msgr2=0x7fed0c0c4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:47.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7fed19d60640 1 --2- 172.21.15.60:0/234445324 >> v2:172.21.15.191:3300/0 conn(0x7fed0c035d10 0x7fed0c0c4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7fed19d60640 1 -- 172.21.15.60:0/234445324 >> v1:172.21.15.60:6789/0 conn(0x7fed0c0cb330 legacy=0x7fed0c0cb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:47.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7fed19d60640 10 monclient: _finish_hunting 0 2024-03-27T21:39:47.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7fed19d60640 1 monclient: found mon.noname-b 2024-03-27T21:39:47.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7fed19d60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:47.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7fed19d60640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:47.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7fed19d60640 1 -- 172.21.15.60:0/234445324 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c02c1a0 con 0x7fed0c02b4a0 2024-03-27T21:39:47.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7fed19d60640 1 --2- 172.21.15.60:0/234445324 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fed0c02b4a0 0x7fed0c0c5180 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fed10009850 tx=0x7fed1002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f735312f2d8f55f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7f9678e3b640 1 -- 172.21.15.60:0/3627026996 wait complete. 2024-03-27T21:39:47.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f9678e3b640 10 monclient: build_initial_monmap 2024-03-27T21:39:47.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f9678e3b640 10 monclient: monmap: 2024-03-27T21:39:47.677 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:47.677 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:47.677 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:47.677 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:47.677 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:47.677 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:47.678 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:47.678 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:47.678 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:47.678 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f2661567640 1 --2- 172.21.15.60:0/997409329 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f265c0ae6a0 0x7f265c0ad6c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2650009870 tx=0x7f265002fd00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10514daad23f46d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f40cce0d640 1 --2- 172.21.15.60:0/2324141607 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f40c8140bd0 0x7f40c81414d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f40bc0097a0 tx=0x7f40bc02b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc6e485566e777b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f72877fe640 1 -- 172.21.15.60:0/837238915 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7288001970 con 0x7f72900b0ec0 2024-03-27T21:39:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f72877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f72877fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f72877fe640 10 monclient: dump: 2024-03-27T21:39:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.680 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f72877fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:47.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f72877fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:47.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f72877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:47.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f7296d42640 5 monclient: authenticate success, global_id 4288 2024-03-27T21:39:47.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f7296d42640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:47.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7ff398e7c640 10 monclient(hunting): get_auth_request con 0x7ff3940b0ec0 auth_method 0 2024-03-27T21:39:47.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7ff398e7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7ff398e7c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.633+0000 7ff398e7c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7ff393fff640 10 monclient(hunting): handle_auth_done global_id 4312 payload 274 2024-03-27T21:39:47.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7ff393fff640 1 -- 172.21.15.60:0/1438981634 >> v2:172.21.15.60:3300/0 conn(0x7ff3940b0ec0 msgr2=0x7ff3941498b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:47.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7ff393fff640 1 --2- 172.21.15.60:0/1438981634 >> v2:172.21.15.60:3300/0 conn(0x7ff3940b0ec0 0x7ff3941498b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7ff393fff640 1 -- 172.21.15.60:0/1438981634 >> v1:172.21.15.191:6789/0 conn(0x7ff3940b1d30 legacy=0x7ff3941458a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7ff393fff640 10 monclient: _finish_hunting 0 2024-03-27T21:39:47.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7ff393fff640 1 monclient: found mon.noname-c 2024-03-27T21:39:47.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7ff393fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:47.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7ff393fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:39:47.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7ff393fff640 1 -- 172.21.15.60:0/1438981634 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff39406d420 con 0x7ff3940b18e0 2024-03-27T21:39:47.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f9678e3b640 1 librados: starting msgr at 2024-03-27T21:39:47.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f9678e3b640 1 librados: starting objecter 2024-03-27T21:39:47.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f72877fe640 1 -- 172.21.15.60:0/837238915 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f728002e050 con 0x7f72900b0ec0 2024-03-27T21:39:47.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f72877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:47.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f7296d42640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:47.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f7296d42640 4 set_mon_vals no callback set 2024-03-27T21:39:47.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f9678e3b640 1 Processor -- start 2024-03-27T21:39:47.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f72877fe640 1 -- 172.21.15.60:0/837238915 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7280030510 con 0x7f72900b0ec0 2024-03-27T21:39:47.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f72877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f72877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:47.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f72877fe640 10 monclient: dump: 2024-03-27T21:39:47.684 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.684 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.684 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.684 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.685 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.685 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.685 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.685 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.685 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.685 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f07f0d2e640 10 monclient(hunting): get_auth_request con 0x7f07ec0a8b10 auth_method 0 2024-03-27T21:39:47.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f07f0d2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f07f0d2e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f07f0d2e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f07f0d2e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f07f0d2e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f07f0d2e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f07f152f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f07f152f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f07f152f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f07f0d2e640 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2024-03-27T21:39:47.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f07f0d2e640 1 -- 172.21.15.60:0/4011084398 >> v2:172.21.15.191:3300/0 conn(0x7f07ec0a8ef0 msgr2=0x7f07ec150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:47.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f07f0d2e640 1 --2- 172.21.15.60:0/4011084398 >> v2:172.21.15.191:3300/0 conn(0x7f07ec0a8ef0 0x7f07ec150010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f07f0d2e640 1 -- 172.21.15.60:0/4011084398 >> v1:172.21.15.60:6789/0 conn(0x7f07ec142e70 legacy=0x7f07ec145c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f07f0d2e640 10 monclient: _finish_hunting 0 2024-03-27T21:39:47.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f07f0d2e640 1 monclient: found mon.noname-b 2024-03-27T21:39:47.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f07f0d2e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:47.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f07f0d2e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:47.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f07f0d2e640 1 -- 172.21.15.60:0/4011084398 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ec0aad60 con 0x7f07ec0a8b10 2024-03-27T21:39:47.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f3ddaffd640 10 monclient: _finish_hunting 0 2024-03-27T21:39:47.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f3ddaffd640 1 monclient: found mon.noname-c 2024-03-27T21:39:47.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f3ddaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:47.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f3ddaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:39:47.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.634+0000 7f3ddaffd640 1 -- 172.21.15.60:0/3150557403 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ddc06fb50 con 0x7f3ddc060970 2024-03-27T21:39:47.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f40c77fe640 1 -- 172.21.15.60:0/2324141607 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b4001670 con 0x7f40c8140bd0 2024-03-27T21:39:47.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f40c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f40c77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:47.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f40c77fe640 10 monclient: dump: 2024-03-27T21:39:47.688 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.688 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.688 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.688 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.688 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.688 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.688 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.688 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.688 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.688 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f40c77fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:47.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f40c77fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:47.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f40c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:47.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f7296d42640 10 monclient: shutdown 2024-03-27T21:39:47.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f7296d42640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:47.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f7296d42640 1 -- 172.21.15.60:0/837238915 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72900b0ec0 msgr2=0x7f7290142970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f7296d42640 1 --2- 172.21.15.60:0/837238915 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72900b0ec0 0x7f7290142970 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f72800097a0 tx=0x7f728002b600 comp rx=0 tx=0).stop 2024-03-27T21:39:47.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f265b7fe640 1 -- 172.21.15.60:0/997409329 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2650030ab0 con 0x7f265c0ae6a0 2024-03-27T21:39:47.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f265b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f265b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:47.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f265b7fe640 10 monclient: dump: 2024-03-27T21:39:47.689 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.690 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.690 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.690 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.690 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.690 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.690 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.690 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.690 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.690 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f265b7fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:47.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f265b7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:47.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f265b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:47.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f7296d42640 1 -- 172.21.15.60:0/837238915 shutdown_connections 2024-03-27T21:39:47.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f7296d42640 1 --2- 172.21.15.60:0/837238915 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72900b0ec0 0x7f7290142970 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7fed1955f640 1 -- 172.21.15.60:0/234445324 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1002e070 con 0x7fed0c02b4a0 2024-03-27T21:39:47.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f40c77fe640 1 -- 172.21.15.60:0/2324141607 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40bc02e050 con 0x7f40c8140bd0 2024-03-27T21:39:47.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f40c77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:47.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f265b7fe640 1 -- 172.21.15.60:0/997409329 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2650030c10 con 0x7f265c0ae6a0 2024-03-27T21:39:47.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f265b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:47.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f40c77fe640 1 -- 172.21.15.60:0/2324141607 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b4001670 con 0x7f40c8140bd0 2024-03-27T21:39:47.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f40c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f40c77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:47.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f40c77fe640 10 monclient: dump: 2024-03-27T21:39:47.692 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.692 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.693 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.693 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.693 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.693 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.693 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.693 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.693 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.693 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f265b7fe640 1 -- 172.21.15.60:0/997409329 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2650030ab0 con 0x7f265c0ae6a0 2024-03-27T21:39:47.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f40cf071640 5 monclient: authenticate success, global_id 4271 2024-03-27T21:39:47.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f40cf071640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:47.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f40cf071640 4 set_mon_vals no callback set 2024-03-27T21:39:47.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f265b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f265b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:47.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f265b7fe640 10 monclient: dump: 2024-03-27T21:39:47.694 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.694 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.694 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.694 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.695 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7ff398e7c640 1 --2- 172.21.15.60:0/1438981634 >> v2:172.21.15.60:3300/0 conn(0x7ff3940b0ec0 0x7ff3941498b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:39:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7fed1955f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f9678e3b640 1 -- start start 2024-03-27T21:39:47.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f9678e3b640 1 librados: setting wanted keys 2024-03-27T21:39:47.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f9678e3b640 1 librados: calling monclient init 2024-03-27T21:39:47.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f9678e3b640 10 monclient: init 2024-03-27T21:39:47.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f7296d42640 1 -- 172.21.15.60:0/837238915 >> 172.21.15.60:0/837238915 conn(0x7f72900b3ae0 msgr2=0x7f72900b3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:47.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f7296d42640 1 -- 172.21.15.60:0/837238915 shutdown_connections 2024-03-27T21:39:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7fed1955f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7fed1955f640 10 monclient: dump: 2024-03-27T21:39:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.697 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7ff393fff640 1 --2- 172.21.15.60:0/1438981634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3940b18e0 0x7ff394142ae0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff3940b3320 tx=0x7ff38802b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a91d81ec7c064e86 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7fed1955f640 10 monclient: _finish_auth 0 2024-03-27T21:39:47.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7fed1955f640 10 monclient: _check_auth_tickets 2024-03-27T21:39:47.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7fed1955f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:47.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7fed1955f640 1 -- 172.21.15.60:0/234445324 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed10004030 con 0x7fed0c02b4a0 2024-03-27T21:39:47.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7fed1955f640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:47.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7fed1955f640 1 -- 172.21.15.60:0/234445324 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed100043e0 con 0x7fed0c02b4a0 2024-03-27T21:39:47.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7fed1955f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7fed1955f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:47.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7fed1955f640 10 monclient: dump: 2024-03-27T21:39:47.699 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.699 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.699 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.699 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.699 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.699 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.700 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.700 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.700 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.700 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7fed1b563640 5 monclient: authenticate success, global_id 4268 2024-03-27T21:39:47.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7fed1b563640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:47.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7fed1b563640 4 set_mon_vals no callback set 2024-03-27T21:39:47.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f07f0d2e640 1 --2- 172.21.15.60:0/4011084398 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f07ec0a8b10 0x7f07ec142930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f07dc0097a0 tx=0x7f07dc02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad86711e74c20d80 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f3ddaffd640 1 --2- 172.21.15.60:0/3150557403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ddc060970 0x7f3ddc0b0050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc06abf0 tx=0x7f3dcc02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=deceaaf8a93e4c03 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7ff3937fe640 1 -- 172.21.15.60:0/1438981634 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff388030800 con 0x7ff3940b18e0 2024-03-27T21:39:47.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7ff3937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7ff3937fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:47.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7ff3937fe640 10 monclient: dump: 2024-03-27T21:39:47.701 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.702 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.702 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.702 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.702 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.702 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.702 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.702 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.702 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.702 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7ff3937fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:47.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7ff3937fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:47.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7ff3937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:47.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7ff3937fe640 1 -- 172.21.15.60:0/1438981634 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff388030960 con 0x7ff3940b18e0 2024-03-27T21:39:47.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7ff3937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7ff3937fe640 1 -- 172.21.15.60:0/1438981634 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff388030cd0 con 0x7ff3940b18e0 2024-03-27T21:39:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7ff3937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7ff3937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7ff3937fe640 10 monclient: dump: 2024-03-27T21:39:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.703 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7ff39b0e0640 5 monclient: authenticate success, global_id 4312 2024-03-27T21:39:47.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7ff39b0e0640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:47.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7ff39b0e0640 4 set_mon_vals no callback set 2024-03-27T21:39:47.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f2662fca640 5 monclient: authenticate success, global_id 4265 2024-03-27T21:39:47.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f2662fca640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:47.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.635+0000 7f2662fca640 4 set_mon_vals no callback set 2024-03-27T21:39:47.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7fed1b563640 10 monclient: shutdown 2024-03-27T21:39:47.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7fed1b563640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:47.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7fed1b563640 1 -- 172.21.15.60:0/234445324 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fed0c02b4a0 msgr2=0x7fed0c0c5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7fed1b563640 1 --2- 172.21.15.60:0/234445324 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fed0c02b4a0 0x7fed0c0c5180 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fed10009850 tx=0x7fed1002b840 comp rx=0 tx=0).stop 2024-03-27T21:39:47.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9678e3b640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:47.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9678e3b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:39:47.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9678e3b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f96740a8ba0 0x7f96741e70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9678e3b640 10 monclient(hunting): picked mon.noname-c con 0x7f96740a8ba0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:47.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9678e3b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9674151da0 0x7f96741f7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9678e3b640 10 monclient(hunting): picked mon.noname-a con 0x7f9674151da0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:47.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9678e3b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96741f7aa0 0x7f967414dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9678e3b640 10 monclient(hunting): picked mon.noname-b con 0x7f96741f7aa0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:47.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9678e3b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9678e3b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967406edd0 con 0x7f9674151da0 2024-03-27T21:39:47.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9678e3b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9678e3b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967406f0b0 con 0x7f96741f7aa0 2024-03-27T21:39:47.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9678e3b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9678e3b640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967406f390 con 0x7f96740a8ba0 2024-03-27T21:39:47.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9678e3b640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:47.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9678e3b640 10 monclient(hunting): authenticate will time out at 828.858248s 2024-03-27T21:39:47.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9672ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9674151da0 0x7f96741f7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9672ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9674151da0 0x7f96741f7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47714/0 (socket says 172.21.15.60:47714) 2024-03-27T21:39:47.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9672ffd640 1 -- 172.21.15.60:0/2982922564 learned_addr learned my addr 172.21.15.60:0/2982922564 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:47.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9672ffd640 10 monclient(hunting): get_auth_request con 0x7f9674151da0 auth_method 0 2024-03-27T21:39:47.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9672ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9672ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9672ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f96737fe640 1 --2- 172.21.15.60:0/2982922564 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f96740a8ba0 0x7f96741e70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f96737fe640 10 monclient(hunting): get_auth_request con 0x7f96740a8ba0 auth_method 0 2024-03-27T21:39:47.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f96737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f96737fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f96737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9672ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9672ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9672ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f96737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f96737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f96737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9672ffd640 10 monclient(hunting): handle_auth_done global_id 4296 payload 824 2024-03-27T21:39:47.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9672ffd640 1 -- 172.21.15.60:0/2982922564 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f96740a8ba0 msgr2=0x7f96741e70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f2662fca640 10 monclient: shutdown 2024-03-27T21:39:47.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f2662fca640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:47.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f2662fca640 1 -- 172.21.15.60:0/997409329 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f265c0ae6a0 msgr2=0x7f265c0ad6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f2662fca640 1 --2- 172.21.15.60:0/997409329 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f265c0ae6a0 0x7f265c0ad6c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2650009870 tx=0x7f265002fd00 comp rx=0 tx=0).stop 2024-03-27T21:39:47.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f3dd9ffb640 1 -- 172.21.15.60:0/3150557403 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc031640 con 0x7f3ddc060970 2024-03-27T21:39:47.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f3dd9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f3dd9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f3dd9ffb640 10 monclient: dump: 2024-03-27T21:39:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.710 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f3dd9ffb640 10 monclient: _finish_auth 0 2024-03-27T21:39:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f3dd9ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:39:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f3dd9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f3dd9ffb640 1 -- 172.21.15.60:0/3150557403 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dcc0317a0 con 0x7f3ddc060970 2024-03-27T21:39:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f3ddbfff640 5 monclient: authenticate success, global_id 4306 2024-03-27T21:39:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f3ddbfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f3dd9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f3dd9ffb640 1 -- 172.21.15.60:0/3150557403 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc031b70 con 0x7f3ddc060970 2024-03-27T21:39:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f40cf071640 10 monclient: shutdown 2024-03-27T21:39:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f40cf071640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f40cf071640 1 -- 172.21.15.60:0/2324141607 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f40c8140bd0 msgr2=0x7f40c81414d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f40cf071640 1 --2- 172.21.15.60:0/2324141607 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f40c8140bd0 0x7f40c81414d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f40bc0097a0 tx=0x7f40bc02b710 comp rx=0 tx=0).stop 2024-03-27T21:39:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f07eb7fe640 1 -- 172.21.15.60:0/4011084398 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07dc02e070 con 0x7f07ec0a8b10 2024-03-27T21:39:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f07eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f07eb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f07eb7fe640 10 monclient: dump: 2024-03-27T21:39:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.712 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f07eb7fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f07eb7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f07eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f07f1d30640 5 monclient: authenticate success, global_id 4274 2024-03-27T21:39:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f07f1d30640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f07eb7fe640 1 -- 172.21.15.60:0/4011084398 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07dc0315a0 con 0x7f07ec0a8b10 2024-03-27T21:39:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f07eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f07f1d30640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f07f1d30640 4 set_mon_vals no callback set 2024-03-27T21:39:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7ff39b0e0640 10 monclient: shutdown 2024-03-27T21:39:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7ff39b0e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7ff39b0e0640 1 -- 172.21.15.60:0/1438981634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3940b18e0 msgr2=0x7ff394142ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7ff39b0e0640 1 --2- 172.21.15.60:0/1438981634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3940b18e0 0x7ff394142ae0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff3940b3320 tx=0x7ff38802b940 comp rx=0 tx=0).stop 2024-03-27T21:39:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f40cf071640 1 -- 172.21.15.60:0/2324141607 shutdown_connections 2024-03-27T21:39:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f40cf071640 1 --2- 172.21.15.60:0/2324141607 >> v2:172.21.15.191:3300/0 conn(0x7f40c813a6f0 0x7f40c813ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f40cf071640 1 --2- 172.21.15.60:0/2324141607 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f40c8140bd0 0x7f40c81414d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f40cf071640 1 -- 172.21.15.60:0/2324141607 >> 172.21.15.60:0/2324141607 conn(0x7f40c80b2cc0 msgr2=0x7f40c80b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f07eb7fe640 1 -- 172.21.15.60:0/4011084398 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07dc031930 con 0x7f07ec0a8b10 2024-03-27T21:39:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f07eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f07eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f07eb7fe640 10 monclient: dump: 2024-03-27T21:39:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.715 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9672ffd640 1 --2- 172.21.15.60:0/2982922564 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f96740a8ba0 0x7f96741e70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9672ffd640 1 -- 172.21.15.60:0/2982922564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96741f7aa0 msgr2=0x7f967414dd70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-27T21:39:47.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9672ffd640 1 --2- 172.21.15.60:0/2982922564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96741f7aa0 0x7f967414dd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9672ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:39:47.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9672ffd640 1 monclient: found mon.noname-a 2024-03-27T21:39:47.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9672ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:47.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9672ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:47.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f9672ffd640 1 -- 172.21.15.60:0/2982922564 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9664006e70 con 0x7f9674151da0 2024-03-27T21:39:47.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f7296d42640 1 -- 172.21.15.60:0/837238915 wait complete. 2024-03-27T21:39:47.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7ff39b0e0640 1 -- 172.21.15.60:0/1438981634 shutdown_connections 2024-03-27T21:39:47.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7ff39b0e0640 1 --2- 172.21.15.60:0/1438981634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3940b18e0 0x7ff394142ae0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7ff39b0e0640 1 --2- 172.21.15.60:0/1438981634 >> v2:172.21.15.60:3300/0 conn(0x7ff3940b0ec0 0x7ff3941498b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7ff39b0e0640 1 -- 172.21.15.60:0/1438981634 >> 172.21.15.60:0/1438981634 conn(0x7ff3940b7300 msgr2=0x7ff3940b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:47.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7fed1b563640 1 -- 172.21.15.60:0/234445324 shutdown_connections 2024-03-27T21:39:47.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7fed1b563640 1 --2- 172.21.15.60:0/234445324 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fed0c02b4a0 0x7fed0c0c5180 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7fed1b563640 1 --2- 172.21.15.60:0/234445324 >> v2:172.21.15.191:3300/0 conn(0x7fed0c035d10 0x7fed0c0c4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7fed1b563640 1 -- 172.21.15.60:0/234445324 >> 172.21.15.60:0/234445324 conn(0x7fed0c007930 msgr2=0x7fed0c007d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:47.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f2662fca640 1 -- 172.21.15.60:0/997409329 shutdown_connections 2024-03-27T21:39:47.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f2662fca640 1 --2- 172.21.15.60:0/997409329 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f265c0ae6a0 0x7f265c0ad6c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f2662fca640 1 --2- 172.21.15.60:0/997409329 >> v2:172.21.15.191:3300/0 conn(0x7f265c06d700 0x7f265c06eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f2662fca640 1 -- 172.21.15.60:0/997409329 >> 172.21.15.60:0/997409329 conn(0x7f265c0b2ea0 msgr2=0x7f265c0b32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:47.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f3ddbfff640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:47.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.636+0000 7f3ddbfff640 4 set_mon_vals no callback set 2024-03-27T21:39:47.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f3dd9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f3dd9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:47.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f3dd9ffb640 10 monclient: dump: 2024-03-27T21:39:47.719 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.719 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.719 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.719 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.719 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.719 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.719 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.719 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.719 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.719 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f40cf071640 1 -- 172.21.15.60:0/2324141607 shutdown_connections 2024-03-27T21:39:47.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f40cf071640 1 -- 172.21.15.60:0/2324141607 wait complete. 2024-03-27T21:39:47.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f2662fca640 1 -- 172.21.15.60:0/997409329 shutdown_connections 2024-03-27T21:39:47.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f3ddbfff640 10 monclient: shutdown 2024-03-27T21:39:47.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f3ddbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:47.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f3ddbfff640 1 -- 172.21.15.60:0/3150557403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ddc060970 msgr2=0x7f3ddc0b0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f3ddbfff640 1 --2- 172.21.15.60:0/3150557403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ddc060970 0x7f3ddc0b0050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc06abf0 tx=0x7f3dcc02b8f0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7fed1b563640 1 -- 172.21.15.60:0/234445324 shutdown_connections 2024-03-27T21:39:47.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f3ddbfff640 1 -- 172.21.15.60:0/3150557403 shutdown_connections 2024-03-27T21:39:47.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f3ddbfff640 1 --2- 172.21.15.60:0/3150557403 >> v2:172.21.15.60:3300/0 conn(0x7f3ddc06e710 0x7f3ddc071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f3ddbfff640 1 --2- 172.21.15.60:0/3150557403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ddc060970 0x7f3ddc0b0050 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f3ddbfff640 1 -- 172.21.15.60:0/3150557403 >> 172.21.15.60:0/3150557403 conn(0x7f3ddc069ce0 msgr2=0x7f3ddc06a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:47.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f2662fca640 1 -- 172.21.15.60:0/997409329 wait complete. 2024-03-27T21:39:47.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f3ddbfff640 1 -- 172.21.15.60:0/3150557403 shutdown_connections 2024-03-27T21:39:47.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f3ddbfff640 1 -- 172.21.15.60:0/3150557403 wait complete. 2024-03-27T21:39:47.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f07f1d30640 10 monclient: shutdown 2024-03-27T21:39:47.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f07f1d30640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:47.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7fed1b563640 1 -- 172.21.15.60:0/234445324 wait complete. 2024-03-27T21:39:47.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f07f1d30640 1 -- 172.21.15.60:0/4011084398 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f07ec0a8b10 msgr2=0x7f07ec142930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f07f1d30640 1 --2- 172.21.15.60:0/4011084398 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f07ec0a8b10 0x7f07ec142930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f07dc0097a0 tx=0x7f07dc02b840 comp rx=0 tx=0).stop 2024-03-27T21:39:47.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7ff39b0e0640 1 -- 172.21.15.60:0/1438981634 shutdown_connections 2024-03-27T21:39:47.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7ff39b0e0640 1 -- 172.21.15.60:0/1438981634 wait complete. 2024-03-27T21:39:47.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f07f1d30640 1 -- 172.21.15.60:0/4011084398 shutdown_connections 2024-03-27T21:39:47.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f07f1d30640 1 --2- 172.21.15.60:0/4011084398 >> v2:172.21.15.191:3300/0 conn(0x7f07ec0a8ef0 0x7f07ec150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f07f1d30640 1 --2- 172.21.15.60:0/4011084398 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f07ec0a8b10 0x7f07ec142930 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f07f1d30640 1 -- 172.21.15.60:0/4011084398 >> 172.21.15.60:0/4011084398 conn(0x7f07ec0b2c20 msgr2=0x7f07ec0b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f07f1d30640 1 -- 172.21.15.60:0/4011084398 shutdown_connections 2024-03-27T21:39:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f07f1d30640 1 -- 172.21.15.60:0/4011084398 wait complete. 2024-03-27T21:39:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f9672ffd640 1 --2- 172.21.15.60:0/2982922564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9674151da0 0x7f96741f7560 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9664000c00 tx=0x7f96640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f965bfff640 10 client.?.objecter ms_handle_connect 0x7f9674151da0 2024-03-27T21:39:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f965bfff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f40cf071640 10 monclient: build_initial_monmap 2024-03-27T21:39:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f40cf071640 10 monclient: monmap: 2024-03-27T21:39:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:47.726 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7fed1b563640 10 monclient: build_initial_monmap 2024-03-27T21:39:47.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7fed1b563640 10 monclient: monmap: 2024-03-27T21:39:47.727 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:47.727 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:47.727 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:47.727 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:47.727 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:47.727 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:47.727 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:47.727 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:47.727 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:47.727 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7fed1b563640 1 librados: starting msgr at 2024-03-27T21:39:47.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7fed1b563640 1 librados: starting objecter 2024-03-27T21:39:47.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7fed1b563640 1 Processor -- start 2024-03-27T21:39:47.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f07f1d30640 10 monclient: build_initial_monmap 2024-03-27T21:39:47.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f07f1d30640 10 monclient: monmap: 2024-03-27T21:39:47.728 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:47.728 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:47.728 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:47.728 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:47.728 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:47.728 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:47.728 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:47.729 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:47.729 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:47.729 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f965bfff640 1 -- 172.21.15.60:0/2982922564 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9664003610 con 0x7f9674151da0 2024-03-27T21:39:47.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f965bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f965bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:47.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f965bfff640 10 monclient: dump: 2024-03-27T21:39:47.729 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.729 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.729 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.729 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.730 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f965bfff640 10 monclient: _finish_auth 0 2024-03-27T21:39:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f965bfff640 10 monclient: _check_auth_tickets 2024-03-27T21:39:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f965bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f965bfff640 1 -- 172.21.15.60:0/2982922564 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9664003790 con 0x7f9674151da0 2024-03-27T21:39:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f965bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f3ddbfff640 10 monclient: build_initial_monmap 2024-03-27T21:39:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f3ddbfff640 10 monclient: monmap: 2024-03-27T21:39:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:47.731 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f40cf071640 1 librados: starting msgr at 2024-03-27T21:39:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f40cf071640 1 librados: starting objecter 2024-03-27T21:39:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f40cf071640 1 Processor -- start 2024-03-27T21:39:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f07f1d30640 1 librados: starting msgr at 2024-03-27T21:39:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f3ddbfff640 1 librados: starting msgr at 2024-03-27T21:39:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f3ddbfff640 1 librados: starting objecter 2024-03-27T21:39:47.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f07f1d30640 1 librados: starting objecter 2024-03-27T21:39:47.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f07f1d30640 1 Processor -- start 2024-03-27T21:39:47.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f96717fa640 4 set_mon_vals no callback set 2024-03-27T21:39:47.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f965bfff640 1 -- 172.21.15.60:0/2982922564 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9664003b40 con 0x7f9674151da0 2024-03-27T21:39:47.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f965bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f965bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:47.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f965bfff640 10 monclient: dump: 2024-03-27T21:39:47.733 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.733 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.734 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.734 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.734 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.734 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.734 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.734 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.734 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.734 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7fed1b563640 1 -- start start 2024-03-27T21:39:47.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7fed1b563640 1 librados: setting wanted keys 2024-03-27T21:39:47.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7fed1b563640 1 librados: calling monclient init 2024-03-27T21:39:47.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7fed1b563640 10 monclient: init 2024-03-27T21:39:47.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7fed1b563640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:47.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7fed1b563640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:39:47.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7fed1b563640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fed0c02b4a0 0x7fed0c0c6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1b563640 10 monclient(hunting): picked mon.noname-c con 0x7fed0c02b4a0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:47.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1b563640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed0c035d10 0x7fed0c0c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1b563640 10 monclient(hunting): picked mon.noname-b con 0x7fed0c035d10 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:47.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1b563640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed0c0cb330 0x7fed0c0c89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1b563640 10 monclient(hunting): picked mon.noname-a con 0x7fed0c0cb330 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:47.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1b563640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1b563640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c0c7b40 con 0x7fed0c0cb330 2024-03-27T21:39:47.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1b563640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1b563640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c0c61e0 con 0x7fed0c035d10 2024-03-27T21:39:47.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1b563640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1b563640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c0c64c0 con 0x7fed0c02b4a0 2024-03-27T21:39:47.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1b563640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:47.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1b563640 10 monclient(hunting): authenticate will time out at 828.862158s 2024-03-27T21:39:47.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1a561640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fed0c02b4a0 0x7fed0c0c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1a561640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fed0c02b4a0 0x7fed0c0c6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47576/0 (socket says 172.21.15.60:47576) 2024-03-27T21:39:47.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1a561640 1 -- 172.21.15.60:0/2146763059 learned_addr learned my addr 172.21.15.60:0/2146763059 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:47.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1a561640 10 monclient(hunting): get_auth_request con 0x7fed0c02b4a0 auth_method 0 2024-03-27T21:39:47.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1a561640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1a561640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1a561640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f07f1d30640 1 -- start start 2024-03-27T21:39:47.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f07f1d30640 1 librados: setting wanted keys 2024-03-27T21:39:47.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f07f1d30640 1 librados: calling monclient init 2024-03-27T21:39:47.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f07f1d30640 10 monclient: init 2024-03-27T21:39:47.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f2662fca640 10 monclient: build_initial_monmap 2024-03-27T21:39:47.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f2662fca640 10 monclient: monmap: 2024-03-27T21:39:47.737 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:47.737 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:47.737 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:47.737 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:47.737 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:47.738 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:47.738 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:47.738 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:47.738 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:47.738 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f2662fca640 1 librados: starting msgr at 2024-03-27T21:39:47.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f2662fca640 1 librados: starting objecter 2024-03-27T21:39:47.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f2662fca640 1 Processor -- start 2024-03-27T21:39:47.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f3ddbfff640 1 Processor -- start 2024-03-27T21:39:47.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f3ddbfff640 1 -- start start 2024-03-27T21:39:47.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f3ddbfff640 1 librados: setting wanted keys 2024-03-27T21:39:47.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f3ddbfff640 1 librados: calling monclient init 2024-03-27T21:39:47.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f3ddbfff640 10 monclient: init 2024-03-27T21:39:47.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7ff39b0e0640 10 monclient: build_initial_monmap 2024-03-27T21:39:47.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7ff39b0e0640 10 monclient: monmap: 2024-03-27T21:39:47.739 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:47.739 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:47.739 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:47.739 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:47.740 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:47.740 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:47.740 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:47.740 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:47.740 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:47.740 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7ff39b0e0640 1 librados: starting msgr at 2024-03-27T21:39:47.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7ff39b0e0640 1 librados: starting objecter 2024-03-27T21:39:47.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7ff39b0e0640 1 Processor -- start 2024-03-27T21:39:47.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7ff39b0e0640 1 -- start start 2024-03-27T21:39:47.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7ff39b0e0640 1 librados: setting wanted keys 2024-03-27T21:39:47.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7ff39b0e0640 1 librados: calling monclient init 2024-03-27T21:39:47.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7ff39b0e0640 10 monclient: init 2024-03-27T21:39:47.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39b0e0640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:47.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39b0e0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:39:47.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39b0e0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3940b0ec0 0x7ff394145cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39b0e0640 10 monclient(hunting): picked mon.noname-a con 0x7ff3940b0ec0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:47.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39b0e0640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff3940b18e0 0x7ff394147e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39b0e0640 10 monclient(hunting): picked mon.noname-c con 0x7ff3940b18e0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:47.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39b0e0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3940b1d30 0x7ff3940b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39b0e0640 10 monclient(hunting): picked mon.noname-b con 0x7ff3940b1d30 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:47.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39b0e0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39b0e0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3940b6580 con 0x7ff3940b0ec0 2024-03-27T21:39:47.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39b0e0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39b0e0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3940b6a60 con 0x7ff3940b1d30 2024-03-27T21:39:47.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39b0e0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39b0e0640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3940b6c20 con 0x7ff3940b18e0 2024-03-27T21:39:47.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39b0e0640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:47.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39b0e0640 10 monclient(hunting): authenticate will time out at 828.862434s 2024-03-27T21:39:47.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff398e7c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3940b0ec0 0x7ff394145cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f7296d42640 10 monclient: build_initial_monmap 2024-03-27T21:39:47.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.637+0000 7f7296d42640 10 monclient: monmap: 2024-03-27T21:39:47.742 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:47.742 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:47.742 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:47.742 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:47.742 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:47.742 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:47.743 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:47.743 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:47.743 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:47.743 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f7296d42640 1 librados: starting msgr at 2024-03-27T21:39:47.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f7296d42640 1 librados: starting objecter 2024-03-27T21:39:47.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f7296d42640 1 Processor -- start 2024-03-27T21:39:47.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f7296d42640 1 -- start start 2024-03-27T21:39:47.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f7296d42640 1 librados: setting wanted keys 2024-03-27T21:39:47.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f7296d42640 1 librados: calling monclient init 2024-03-27T21:39:47.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f7296d42640 10 monclient: init 2024-03-27T21:39:47.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f7296d42640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:47.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f7296d42640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:39:47.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f7296d42640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72900b0ec0 0x7f7290144710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f7296d42640 10 monclient(hunting): picked mon.noname-a con 0x7f72900b0ec0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:47.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f7296d42640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7290142eb0 0x7f72901dab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7294ade640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72900b0ec0 0x7f7290144710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7294ade640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72900b0ec0 0x7f7290144710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47720/0 (socket says 172.21.15.60:47720) 2024-03-27T21:39:47.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7296d42640 10 monclient(hunting): picked mon.noname-c con 0x7f7290142eb0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:47.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7296d42640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72901493e0 0x7f7290138fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7287fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7290142eb0 0x7f72901dab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7287fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7290142eb0 0x7f72901dab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47570/0 (socket says 172.21.15.60:47570) 2024-03-27T21:39:47.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7296d42640 10 monclient(hunting): picked mon.noname-b con 0x7f72901493e0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:47.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7296d42640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7296d42640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729013fc50 con 0x7f72900b0ec0 2024-03-27T21:39:47.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7296d42640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7296d42640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72901da650 con 0x7f72901493e0 2024-03-27T21:39:47.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7296d42640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7296d42640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7290144c50 con 0x7f7290142eb0 2024-03-27T21:39:47.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7296d42640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:47.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7296d42640 10 monclient(hunting): authenticate will time out at 828.862254s 2024-03-27T21:39:47.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7294ade640 1 -- 172.21.15.60:0/1265218146 learned_addr learned my addr 172.21.15.60:0/1265218146 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:47.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7287fff640 10 monclient(hunting): get_auth_request con 0x7f7290142eb0 auth_method 0 2024-03-27T21:39:47.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7287fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7287fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7287fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7294ade640 10 monclient(hunting): get_auth_request con 0x7f72900b0ec0 auth_method 0 2024-03-27T21:39:47.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7294ade640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7294ade640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7294ade640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7287fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7287fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7287fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f72952df640 1 --2- 172.21.15.60:0/1265218146 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72901493e0 0x7f7290138fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f72952df640 10 monclient(hunting): get_auth_request con 0x7f72901493e0 auth_method 0 2024-03-27T21:39:47.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f72952df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f72952df640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f72952df640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7294ade640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7294ade640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f7294ade640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f72952df640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f72952df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f72952df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f72952df640 10 monclient(hunting): handle_auth_done global_id 4321 payload 824 2024-03-27T21:39:47.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f72952df640 1 -- 172.21.15.60:0/1265218146 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7290142eb0 msgr2=0x7f72901dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f72952df640 1 --2- 172.21.15.60:0/1265218146 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7290142eb0 0x7f72901dab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f07f1d30640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:47.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f07f1d30640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:39:47.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f07f1d30640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f07ec0a8b10 0x7f07ec144b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f07f1d30640 10 monclient(hunting): picked mon.noname-c con 0x7f07ec0a8b10 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:47.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f07f1d30640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f07ec0a8ef0 0x7f07ec1da820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f07f1d30640 10 monclient(hunting): picked mon.noname-b con 0x7f07ec0a8ef0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:47.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f07f1d30640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07ec142e70 0x7f07ec1dad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f07f1d30640 10 monclient(hunting): picked mon.noname-a con 0x7f07ec142e70 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:47.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f07f1d30640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f07f1d30640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec0b0be0 con 0x7f07ec142e70 2024-03-27T21:39:47.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f07f1d30640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f07f1d30640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec1db2a0 con 0x7f07ec0a8ef0 2024-03-27T21:39:47.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f07f1d30640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f07f1d30640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec0b2080 con 0x7f07ec0a8b10 2024-03-27T21:39:47.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f07f1d30640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:47.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f07f1d30640 10 monclient(hunting): authenticate will time out at 828.862842s 2024-03-27T21:39:47.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07ebfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f07ec0a8ef0 0x7f07ec1da820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07ebfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f07ec0a8ef0 0x7f07ec1da820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:56676/0 (socket says 172.21.15.60:56676) 2024-03-27T21:39:47.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07ebfff640 1 -- 172.21.15.60:0/3415792531 learned_addr learned my addr 172.21.15.60:0/3415792531 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:47.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07ebfff640 10 monclient(hunting): get_auth_request con 0x7f07ec0a8ef0 auth_method 0 2024-03-27T21:39:47.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07f152f640 1 --2- 172.21.15.60:0/3415792531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07ec142e70 0x7f07ec1dad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f3ddbfff640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:47.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddbfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:39:47.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ddc060970 0x7f3ddc140cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff398e7c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3940b0ec0 0x7ff394145cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47730/0 (socket says 172.21.15.60:47730) 2024-03-27T21:39:47.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff398e7c640 1 -- 172.21.15.60:0/1985325899 learned_addr learned my addr 172.21.15.60:0/1985325899 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:47.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff398e7c640 10 monclient(hunting): get_auth_request con 0x7ff3940b0ec0 auth_method 0 2024-03-27T21:39:47.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff398e7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff398e7c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff398e7c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39967d640 1 --2- 172.21.15.60:0/1985325899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3940b1d30 0x7ff3940b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39967d640 10 monclient(hunting): get_auth_request con 0x7ff3940b1d30 auth_method 0 2024-03-27T21:39:47.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39967d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39967d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39967d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39967d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39967d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7ff39967d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f40cf071640 1 -- start start 2024-03-27T21:39:47.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f40cf071640 1 librados: setting wanted keys 2024-03-27T21:39:47.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f40cf071640 1 librados: calling monclient init 2024-03-27T21:39:47.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.638+0000 7f40cf071640 10 monclient: init 2024-03-27T21:39:47.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f40cf071640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:47.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f40cf071640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:39:47.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f40cf071640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40c806ec20 0x7f40c80b13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f40cf071640 10 monclient(hunting): picked mon.noname-a con 0x7f40c806ec20 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:47.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f40cf071640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f40c813a6f0 0x7f40c80b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f40cf071640 10 monclient(hunting): picked mon.noname-c con 0x7f40c813a6f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:47.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f40cf071640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40c8140bd0 0x7f40c81d59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f40cf071640 10 monclient(hunting): picked mon.noname-b con 0x7f40c8140bd0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:47.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f40cf071640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f40cf071640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c8140780 con 0x7f40c806ec20 2024-03-27T21:39:47.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f40cf071640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f40cf071640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c81d55a0 con 0x7f40c8140bd0 2024-03-27T21:39:47.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f40cf071640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f40cf071640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c80b2480 con 0x7f40c813a6f0 2024-03-27T21:39:47.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f40cf071640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:47.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f40cf071640 10 monclient(hunting): authenticate will time out at 828.861894s 2024-03-27T21:39:47.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f72952df640 1 -- 172.21.15.60:0/1265218146 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72900b0ec0 msgr2=0x7f7290144710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f72952df640 1 --2- 172.21.15.60:0/1265218146 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72900b0ec0 0x7f7290144710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f72952df640 10 monclient: _finish_hunting 0 2024-03-27T21:39:47.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f72952df640 1 monclient: found mon.noname-b 2024-03-27T21:39:47.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f72952df640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:47.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f72952df640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:39:47.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f72952df640 1 -- 172.21.15.60:0/1265218146 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7280009400 con 0x7f72901493e0 2024-03-27T21:39:47.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07f152f640 10 monclient(hunting): get_auth_request con 0x7f07ec142e70 auth_method 0 2024-03-27T21:39:47.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07f152f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07f152f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07f152f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddbfff640 10 monclient(hunting): picked mon.noname-a con 0x7f3ddc060970 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:47.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddbfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ddc06e710 0x7f3ddc1daae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddbfff640 10 monclient(hunting): picked mon.noname-b con 0x7f3ddc06e710 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:47.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddbfff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3ddc0adfe0 0x7f3ddc13b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f2662fca640 1 -- start start 2024-03-27T21:39:47.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddbfff640 10 monclient(hunting): picked mon.noname-c con 0x7f3ddc0adfe0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:47.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddbfff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddbfff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc1da3b0 con 0x7f3ddc060970 2024-03-27T21:39:47.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff39967d640 10 monclient(hunting): handle_auth_done global_id 4324 payload 824 2024-03-27T21:39:47.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff39967d640 1 -- 172.21.15.60:0/1985325899 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff3940b18e0 msgr2=0x7ff394147e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff39967d640 1 --2- 172.21.15.60:0/1985325899 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff3940b18e0 0x7ff394147e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff39967d640 1 -- 172.21.15.60:0/1985325899 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3940b0ec0 msgr2=0x7ff394145cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff39967d640 1 --2- 172.21.15.60:0/1985325899 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3940b0ec0 0x7ff394145cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff39967d640 10 monclient: _finish_hunting 0 2024-03-27T21:39:47.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff39967d640 1 monclient: found mon.noname-b 2024-03-27T21:39:47.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff39967d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:47.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff39967d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:39:47.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff39967d640 1 -- 172.21.15.60:0/1985325899 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3880094d0 con 0x7ff3940b1d30 2024-03-27T21:39:47.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07f0d2e640 1 --2- 172.21.15.60:0/3415792531 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f07ec0a8b10 0x7f07ec144b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f2662fca640 1 librados: setting wanted keys 2024-03-27T21:39:47.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f2662fca640 1 librados: calling monclient init 2024-03-27T21:39:47.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.639+0000 7f2662fca640 10 monclient: init 2024-03-27T21:39:47.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1ad62640 1 --2- 172.21.15.60:0/2146763059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed0c0cb330 0x7fed0c0c89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1ad62640 10 monclient(hunting): get_auth_request con 0x7fed0c0cb330 auth_method 0 2024-03-27T21:39:47.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f9678e3b640 5 monclient: authenticate success, global_id 4296 2024-03-27T21:39:47.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f9678e3b640 10 monclient: _renew_subs 2024-03-27T21:39:47.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f9678e3b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:47.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f9678e3b640 1 -- 172.21.15.60:0/2982922564 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967414f100 con 0x7f9674151da0 2024-03-27T21:39:47.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f2662fca640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:47.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f2662fca640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:39:47.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f2662fca640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f265c06d700 0x7f265c0a98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f2662fca640 10 monclient(hunting): picked mon.noname-c con 0x7f265c06d700 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:47.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f2662fca640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f265c0ae6a0 0x7f265c1c5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f2662fca640 10 monclient(hunting): picked mon.noname-b con 0x7f265c0ae6a0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:47.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f2662fca640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f265c13e000 0x7f265c1c5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f2662fca640 10 monclient(hunting): picked mon.noname-a con 0x7f265c13e000 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:47.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f2662fca640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f2662fca640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c0ac8c0 con 0x7f265c13e000 2024-03-27T21:39:47.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f2662fca640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f2662fca640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c0a9e10 con 0x7f265c0ae6a0 2024-03-27T21:39:47.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f2662fca640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f2662fca640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f265c1c71e0 con 0x7f265c06d700 2024-03-27T21:39:47.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f2662fca640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:47.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddaffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ddc060970 0x7f3ddc140cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddb7fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3ddc0adfe0 0x7f3ddc13b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddaffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ddc060970 0x7f3ddc140cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47744/0 (socket says 172.21.15.60:47744) 2024-03-27T21:39:47.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddaffd640 1 -- 172.21.15.60:0/689226680 learned_addr learned my addr 172.21.15.60:0/689226680 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:47.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddbfff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddbfff640 1 -- 172.21.15.60:0/689226680 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc1429a0 con 0x7f3ddc06e710 2024-03-27T21:39:47.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff39967d640 1 --2- 172.21.15.60:0/1985325899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3940b1d30 0x7ff3940b4ad0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00c730 tx=0x7ff38c00cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1ad62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1ad62640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7fed1ad62640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7fed1ad62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7fed1ad62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7fed1ad62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7fed19d60640 1 --2- 172.21.15.60:0/2146763059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed0c035d10 0x7fed0c0c84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7fed19d60640 10 monclient(hunting): get_auth_request con 0x7fed0c035d10 auth_method 0 2024-03-27T21:39:47.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7fed19d60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7fed19d60640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7fed19d60640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7fed1ad62640 10 monclient(hunting): handle_auth_done global_id 4302 payload 824 2024-03-27T21:39:47.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7fed1ad62640 1 -- 172.21.15.60:0/2146763059 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fed0c02b4a0 msgr2=0x7fed0c0c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7fed1ad62640 1 --2- 172.21.15.60:0/2146763059 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fed0c02b4a0 0x7fed0c0c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7fed1ad62640 1 -- 172.21.15.60:0/2146763059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed0c035d10 msgr2=0x7fed0c0c84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.640+0000 7f2662fca640 10 monclient(hunting): authenticate will time out at 828.862810s 2024-03-27T21:39:47.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f265bfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f265c0ae6a0 0x7f265c1c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f265bfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f265c0ae6a0 0x7f265c1c5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:56684/0 (socket says 172.21.15.60:56684) 2024-03-27T21:39:47.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f265bfff640 1 -- 172.21.15.60:0/3529972586 learned_addr learned my addr 172.21.15.60:0/3529972586 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:47.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f265bfff640 10 monclient(hunting): get_auth_request con 0x7f265c0ae6a0 auth_method 0 2024-03-27T21:39:47.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f265bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f265bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f265bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f265bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f265bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f265bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddbfff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:47.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddbfff640 1 -- 172.21.15.60:0/689226680 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc1da5c0 con 0x7f3ddc0adfe0 2024-03-27T21:39:47.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddbfff640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:47.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff3917fa640 10 client.?.objecter ms_handle_connect 0x7ff3940b1d30 2024-03-27T21:39:47.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff3917fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:47.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddbfff640 10 monclient(hunting): authenticate will time out at 828.863574s 2024-03-27T21:39:47.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f72952df640 1 --2- 172.21.15.60:0/1265218146 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72901493e0 0x7f7290138fd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f728800cac0 tx=0x7f728800cf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f72857fa640 10 client.?.objecter ms_handle_connect 0x7f72901493e0 2024-03-27T21:39:47.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f72857fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:47.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddaffd640 10 monclient(hunting): get_auth_request con 0x7f3ddc060970 auth_method 0 2024-03-27T21:39:47.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddaffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f9678e3b640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:47.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f9678e3b640 10 monclient: _renew_subs 2024-03-27T21:39:47.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f9678e3b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:47.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f265bfff640 10 monclient(hunting): handle_auth_done global_id 4333 payload 824 2024-03-27T21:39:47.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f265bfff640 1 -- 172.21.15.60:0/3529972586 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f265c06d700 msgr2=0x7f265c0a98d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T21:39:47.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f9678e3b640 1 -- 172.21.15.60:0/2982922564 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96741f7f30 con 0x7f9674151da0 2024-03-27T21:39:47.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f9678e3b640 1 librados: init done 2024-03-27T21:39:47.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddb7fe640 10 monclient(hunting): get_auth_request con 0x7f3ddc0adfe0 auth_method 0 2024-03-27T21:39:47.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff3917fa640 1 -- 172.21.15.60:0/1985325899 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c006d50 con 0x7ff3940b1d30 2024-03-27T21:39:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff3917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff3917fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff3917fa640 10 monclient: dump: 2024-03-27T21:39:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.768 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff3917fa640 10 monclient: _finish_auth 0 2024-03-27T21:39:47.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff3917fa640 10 monclient: _check_auth_tickets 2024-03-27T21:39:47.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff3917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:47.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f3ddb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7fed1ad62640 1 --2- 172.21.15.60:0/2146763059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed0c035d10 0x7fed0c0c84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7fed1ad62640 10 monclient: _finish_hunting 0 2024-03-27T21:39:47.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7fed1ad62640 1 monclient: found mon.noname-a 2024-03-27T21:39:47.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7fed1ad62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:47.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7fed1ad62640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:47.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7fed1ad62640 1 -- 172.21.15.60:0/2146763059 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed10009460 con 0x7fed0c0cb330 2024-03-27T21:39:47.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7fed1ad62640 1 --2- 172.21.15.60:0/2146763059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed0c0cb330 0x7fed0c0c89f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fed14072db0 tx=0x7fed14067040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff39b0e0640 5 monclient: authenticate success, global_id 4324 2024-03-27T21:39:47.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff39b0e0640 10 monclient: _renew_subs 2024-03-27T21:39:47.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff39b0e0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:47.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7ff39b0e0640 1 -- 172.21.15.60:0/1985325899 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3941f52e0 con 0x7ff3940b1d30 2024-03-27T21:39:47.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7ff39b0e0640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:47.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7ff39b0e0640 10 monclient: _renew_subs 2024-03-27T21:39:47.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f40cce0d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40c806ec20 0x7f40c80b13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7ff39b0e0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:47.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7ff39b0e0640 1 -- 172.21.15.60:0/1985325899 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3941f57c0 con 0x7ff3940b1d30 2024-03-27T21:39:47.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7ff39b0e0640 1 librados: init done 2024-03-27T21:39:47.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7fed0affd640 10 client.?.objecter ms_handle_connect 0x7fed0c0cb330 2024-03-27T21:39:47.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7fed0affd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:47.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f3ddb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f3ddb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f3ddaffd640 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2024-03-27T21:39:47.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f3dda7fc640 1 --2- 172.21.15.60:0/689226680 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ddc06e710 0x7f3ddc1daae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07f0d2e640 10 monclient(hunting): get_auth_request con 0x7f07ec0a8b10 auth_method 0 2024-03-27T21:39:47.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07f0d2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07f0d2e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07f0d2e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07ebfff640 10 monclient(hunting): handle_auth_done global_id 4327 payload 824 2024-03-27T21:39:47.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07ebfff640 1 -- 172.21.15.60:0/3415792531 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f07ec0a8b10 msgr2=0x7f07ec144b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07ebfff640 1 --2- 172.21.15.60:0/3415792531 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f07ec0a8b10 0x7f07ec144b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07ebfff640 1 -- 172.21.15.60:0/3415792531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07ec142e70 msgr2=0x7f07ec1dad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f3ddaffd640 1 -- 172.21.15.60:0/689226680 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3ddc0adfe0 msgr2=0x7f3ddc13b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f3ddaffd640 1 --2- 172.21.15.60:0/689226680 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3ddc0adfe0 0x7f3ddc13b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f3ddaffd640 1 -- 172.21.15.60:0/689226680 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ddc06e710 msgr2=0x7f3ddc1daae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f3ddaffd640 1 --2- 172.21.15.60:0/689226680 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ddc06e710 0x7f3ddc1daae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f40cce0d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40c806ec20 0x7f40c80b13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47724/0 (socket says 172.21.15.60:47724) 2024-03-27T21:39:47.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f40cce0d640 1 -- 172.21.15.60:0/2404035079 learned_addr learned my addr 172.21.15.60:0/2404035079 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:47.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f40cce0d640 10 monclient(hunting): get_auth_request con 0x7f40c806ec20 auth_method 0 2024-03-27T21:39:47.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f40cce0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:47.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f40cce0d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:47.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f40cce0d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:47.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07ebfff640 1 --2- 172.21.15.60:0/3415792531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07ec142e70 0x7f07ec1dad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07ebfff640 10 monclient: _finish_hunting 0 2024-03-27T21:39:47.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07ebfff640 1 monclient: found mon.noname-b 2024-03-27T21:39:47.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:47.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07ebfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:39:47.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f07ebfff640 1 -- 172.21.15.60:0/3415792531 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07dc009400 con 0x7f07ec0a8ef0 2024-03-27T21:39:47.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f3ddaffd640 10 monclient: _finish_hunting 0 2024-03-27T21:39:47.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f3ddaffd640 1 monclient: found mon.noname-a 2024-03-27T21:39:47.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f3ddaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:47.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f3ddaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:47.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f3ddaffd640 1 -- 172.21.15.60:0/689226680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dcc009400 con 0x7f3ddc060970 2024-03-27T21:39:47.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f3dda7fc640 10 monclient: get_auth_request con 0x7f3ddc06e710 auth_method 0 2024-03-27T21:39:47.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f3dda7fc640 1 --2- 172.21.15.60:0/689226680 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ddc06e710 0x7f3ddc1daae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:39:47.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f40cce0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:47.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f40cce0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:47.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f40cce0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:47.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f07ebfff640 1 --2- 172.21.15.60:0/3415792531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f07ec0a8ef0 0x7f07ec1da820 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f07e000c480 tx=0x7f07e000c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f07e97fa640 10 client.?.objecter ms_handle_connect 0x7f07ec0a8ef0 2024-03-27T21:39:47.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f07e97fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:47.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f2660d66640 1 --2- 172.21.15.60:0/3529972586 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f265c06d700 0x7f265c0a98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f40cce0d640 10 monclient(hunting): handle_auth_done global_id 4314 payload 824 2024-03-27T21:39:47.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f72857fa640 1 -- 172.21.15.60:0/1265218146 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728800a640 con 0x7f72901493e0 2024-03-27T21:39:47.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f40cce0d640 1 -- 172.21.15.60:0/2404035079 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f40c813a6f0 msgr2=0x7f40c80b1f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T21:39:47.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f40cce0d640 1 --2- 172.21.15.60:0/2404035079 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f40c813a6f0 0x7f40c80b1f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f40cce0d640 1 -- 172.21.15.60:0/2404035079 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40c8140bd0 msgr2=0x7f40c81d59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f40cce0d640 1 --2- 172.21.15.60:0/2404035079 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40c8140bd0 0x7f40c81d59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f40cce0d640 10 monclient: _finish_hunting 0 2024-03-27T21:39:47.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f40cce0d640 1 monclient: found mon.noname-a 2024-03-27T21:39:47.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f40cce0d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:47.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f265bfff640 1 --2- 172.21.15.60:0/3529972586 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f265c06d700 0x7f265c0a98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f265bfff640 1 -- 172.21.15.60:0/3529972586 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f265c13e000 msgr2=0x7f265c1c5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f265bfff640 1 --2- 172.21.15.60:0/3529972586 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f265c13e000 0x7f265c1c5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f40cce0d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:47.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f40cce0d640 1 -- 172.21.15.60:0/2404035079 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40bc009400 con 0x7f40c806ec20 2024-03-27T21:39:47.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f265bfff640 10 monclient: _finish_hunting 0 2024-03-27T21:39:47.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f265bfff640 1 monclient: found mon.noname-b 2024-03-27T21:39:47.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f265bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:47.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f265bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:39:47.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f265bfff640 1 -- 172.21.15.60:0/3529972586 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26500094d0 con 0x7f265c0ae6a0 2024-03-27T21:39:47.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7fed0affd640 1 -- 172.21.15.60:0/2146763059 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed140723c0 con 0x7fed0c0cb330 2024-03-27T21:39:47.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7fed19d60640 1 --2- 172.21.15.60:0/2146763059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed0c035d10 0x7fed0c0c84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:39:47.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7fed0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7fed0affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:47.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7fed0affd640 10 monclient: dump: 2024-03-27T21:39:47.780 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.781 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.781 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.781 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.781 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.781 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.781 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.781 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.781 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.781 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7fed0affd640 10 monclient: _finish_auth 0 2024-03-27T21:39:47.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7fed0affd640 10 monclient: _check_auth_tickets 2024-03-27T21:39:47.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7fed0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:47.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f265bfff640 1 --2- 172.21.15.60:0/3529972586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f265c0ae6a0 0x7f265c1c5500 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f264c00c430 tx=0x7f264c00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f26597fa640 10 client.?.objecter ms_handle_connect 0x7f265c0ae6a0 2024-03-27T21:39:47.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f26597fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:47.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f3ddaffd640 1 --2- 172.21.15.60:0/689226680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ddc060970 0x7f3ddc140cb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc009890 tx=0x7f3dcc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f3dc37fe640 10 client.?.objecter ms_handle_connect 0x7f3ddc060970 2024-03-27T21:39:47.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f3dc37fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:47.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f72857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f72857fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:47.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f72857fa640 10 monclient: dump: 2024-03-27T21:39:47.783 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.783 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.783 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.783 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.783 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.783 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.783 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.783 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.784 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.784 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f72857fa640 10 monclient: _finish_auth 0 2024-03-27T21:39:47.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f72857fa640 10 monclient: _check_auth_tickets 2024-03-27T21:39:47.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f72857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:47.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f72857fa640 1 -- 172.21.15.60:0/1265218146 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f728800a7a0 con 0x7f72901493e0 2024-03-27T21:39:47.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f72857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:47.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.641+0000 7f7286ffd640 4 set_mon_vals no callback set 2024-03-27T21:39:47.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f7296d42640 5 monclient: authenticate success, global_id 4321 2024-03-27T21:39:47.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f7296d42640 10 monclient: _renew_subs 2024-03-27T21:39:47.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f7296d42640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:47.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f7296d42640 1 -- 172.21.15.60:0/1265218146 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7290144ed0 con 0x7f72901493e0 2024-03-27T21:39:47.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f7296d42640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:47.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f7296d42640 10 monclient: _renew_subs 2024-03-27T21:39:47.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f7296d42640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:47.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f7296d42640 1 -- 172.21.15.60:0/1265218146 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72901452e0 con 0x7f72901493e0 2024-03-27T21:39:47.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.642+0000 7f7296d42640 1 librados: init done 2024-03-27T21:39:47.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f72857fa640 1 -- 172.21.15.60:0/1265218146 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728800abd0 con 0x7f72901493e0 2024-03-27T21:39:47.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f72857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f72857fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:47.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f72857fa640 10 monclient: dump: 2024-03-27T21:39:47.785 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.785 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.785 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.785 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.785 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.785 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.786 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.786 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.786 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.786 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f3dc37fe640 1 -- 172.21.15.60:0/689226680 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc03bbf0 con 0x7f3ddc060970 2024-03-27T21:39:47.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f3dc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f3dc37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:47.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f3dc37fe640 10 monclient: dump: 2024-03-27T21:39:47.787 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.787 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.787 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.787 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.787 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.787 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.787 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.787 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.787 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.787 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f3dc37fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:47.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f3dc37fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:47.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f3dc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:47.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f3dc37fe640 1 -- 172.21.15.60:0/689226680 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dcc03bd70 con 0x7f3ddc060970 2024-03-27T21:39:47.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f3dc37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:47.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f3dc37fe640 1 -- 172.21.15.60:0/689226680 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc045640 con 0x7f3ddc060970 2024-03-27T21:39:47.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f3dc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f3dc37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:47.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f3dc37fe640 10 monclient: dump: 2024-03-27T21:39:47.788 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.788 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.788 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.788 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.788 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.789 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.789 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.789 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.789 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.789 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f07e97fa640 1 -- 172.21.15.60:0/3415792531 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e0005ce0 con 0x7f07ec0a8ef0 2024-03-27T21:39:47.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f07e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f07e97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:47.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f07e97fa640 10 monclient: dump: 2024-03-27T21:39:47.789 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.790 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.790 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.790 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.790 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.790 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.790 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.790 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.790 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.790 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f40cce0d640 1 --2- 172.21.15.60:0/2404035079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40c806ec20 0x7f40c80b13c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f40bc0098c0 tx=0x7f40bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f07e97fa640 10 monclient: _finish_auth 0 2024-03-27T21:39:47.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f07e97fa640 10 monclient: _check_auth_tickets 2024-03-27T21:39:47.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f07e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:47.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f07f1d30640 5 monclient: authenticate success, global_id 4327 2024-03-27T21:39:47.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f07f1d30640 10 monclient: _renew_subs 2024-03-27T21:39:47.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f07f1d30640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:47.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f07f1d30640 1 -- 172.21.15.60:0/3415792531 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07ec0b2360 con 0x7f07ec0a8ef0 2024-03-27T21:39:47.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f40c57fa640 10 client.?.objecter ms_handle_connect 0x7f40c806ec20 2024-03-27T21:39:47.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f07f1d30640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:47.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f07f1d30640 10 monclient: _renew_subs 2024-03-27T21:39:47.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f07f1d30640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:47.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f07f1d30640 1 -- 172.21.15.60:0/3415792531 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07ec0b1130 con 0x7f07ec0a8ef0 2024-03-27T21:39:47.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7fed1b563640 5 monclient: authenticate success, global_id 4302 2024-03-27T21:39:47.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7fed1b563640 10 monclient: _renew_subs 2024-03-27T21:39:47.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7fed1b563640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:47.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7fed1b563640 1 -- 172.21.15.60:0/2146763059 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0c0bf6f0 con 0x7fed0c0cb330 2024-03-27T21:39:47.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7fed1b563640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:47.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7fed1b563640 10 monclient: _renew_subs 2024-03-27T21:39:47.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7fed1b563640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:47.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7fed1b563640 1 -- 172.21.15.60:0/2146763059 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0c0c8f30 con 0x7fed0c0cb330 2024-03-27T21:39:47.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7fed1b563640 1 librados: init done 2024-03-27T21:39:47.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f07f1d30640 1 librados: init done 2024-03-27T21:39:47.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f40c57fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:47.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f7296d42640 10 client.4321.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:47.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f7296d42640 15 client.4321.objecter _session_command_op_assign -1 1 2024-03-27T21:39:47.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f7296d42640 20 client.4321.objecter _get_session osd=-1 returning homeless 2024-03-27T21:39:47.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f7296d42640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:47.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f7296d42640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:47.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f7296d42640 1 -- 172.21.15.60:0/1265218146 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f72900b12a0 con 0x7f72901493e0 2024-03-27T21:39:47.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7fed0affd640 1 -- 172.21.15.60:0/2146763059 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed140690b0 con 0x7fed0c0cb330 2024-03-27T21:39:47.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7fed0affd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:47.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f3dd8ff9640 4 set_mon_vals no callback set 2024-03-27T21:39:47.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f3ddbfff640 5 monclient: authenticate success, global_id 4311 2024-03-27T21:39:47.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f3ddbfff640 10 monclient: _renew_subs 2024-03-27T21:39:47.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f3ddbfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:47.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f3ddbfff640 1 -- 172.21.15.60:0/689226680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ddc1db020 con 0x7f3ddc060970 2024-03-27T21:39:47.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f3ddbfff640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:47.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f3ddbfff640 10 monclient: _renew_subs 2024-03-27T21:39:47.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f3ddbfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:47.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f3ddbfff640 1 -- 172.21.15.60:0/689226680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ddc13bbb0 con 0x7f3ddc060970 2024-03-27T21:39:47.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f3ddbfff640 1 librados: init done 2024-03-27T21:39:47.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f07e97fa640 1 -- 172.21.15.60:0/3415792531 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e0005e40 con 0x7f07ec0a8ef0 2024-03-27T21:39:47.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.643+0000 7f965bfff640 1 -- 172.21.15.60:0/2982922564 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9664003ca0 con 0x7f9674151da0 2024-03-27T21:39:47.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f965bfff640 1 --2- 172.21.15.60:0/2982922564 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f966006c660 0x7f966006eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f965bfff640 1 -- 172.21.15.60:0/2982922564 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f96640ab9e0 con 0x7f9674151da0 2024-03-27T21:39:47.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f965bfff640 10 client.4296.objecter ms_dispatch 0x7f9674000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:47.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f965bfff640 3 client.4296.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:47.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f965bfff640 3 client.4296.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:47.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f965bfff640 20 client.4296.objecter dump_active .. 0 homeless 2024-03-27T21:39:47.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f26597fa640 1 -- 172.21.15.60:0/3529972586 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c0052d0 con 0x7f265c0ae6a0 2024-03-27T21:39:47.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f26597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f26597fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:47.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f26597fa640 10 monclient: dump: 2024-03-27T21:39:47.798 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.798 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.798 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.798 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.798 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.798 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.798 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.798 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.798 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.798 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f40c57fa640 1 -- 172.21.15.60:0/2404035079 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b4001670 con 0x7f40c806ec20 2024-03-27T21:39:47.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f26597fa640 10 monclient: _finish_auth 0 2024-03-27T21:39:47.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f26597fa640 10 monclient: _check_auth_tickets 2024-03-27T21:39:47.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f26597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:47.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f26597fa640 1 -- 172.21.15.60:0/3529972586 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f264c005430 con 0x7f265c0ae6a0 2024-03-27T21:39:47.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f26597fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:47.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f26597fa640 1 -- 172.21.15.60:0/3529972586 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c011870 con 0x7f265c0ae6a0 2024-03-27T21:39:47.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f26597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f26597fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:47.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f26597fa640 10 monclient: dump: 2024-03-27T21:39:47.800 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.800 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.800 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.800 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.800 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.800 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.800 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.800 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.800 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.800 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f40c57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f2662fca640 5 monclient: authenticate success, global_id 4333 2024-03-27T21:39:47.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f40c57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:47.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f2662fca640 10 monclient: _renew_subs 2024-03-27T21:39:47.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f40c57fa640 10 monclient: dump: 2024-03-27T21:39:47.802 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.802 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.802 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.802 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.802 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.802 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.802 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.802 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.802 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.802 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f40c57fa640 10 monclient: _finish_auth 0 2024-03-27T21:39:47.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f2662fca640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:47.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f40c57fa640 10 monclient: _check_auth_tickets 2024-03-27T21:39:47.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f2662fca640 1 -- 172.21.15.60:0/3529972586 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f265c0a8440 con 0x7f265c0ae6a0 2024-03-27T21:39:47.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f40c57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:47.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f2662fca640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:47.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f2662fca640 10 monclient: _renew_subs 2024-03-27T21:39:47.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f2662fca640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:47.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f2662fca640 1 -- 172.21.15.60:0/3529972586 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f265c1c6160 con 0x7f265c0ae6a0 2024-03-27T21:39:47.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f2662fca640 1 librados: init done 2024-03-27T21:39:47.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f3dc17fa640 10 client.4311.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:47.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f3dc17fa640 15 client.4311.objecter _session_command_op_assign -1 1 2024-03-27T21:39:47.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f3dc17fa640 20 client.4311.objecter _get_session osd=-1 returning homeless 2024-03-27T21:39:47.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f3dc17fa640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:47.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f3dc17fa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:47.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f3dc17fa640 1 -- 172.21.15.60:0/689226680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3ddc1db240 con 0x7f3ddc060970 2024-03-27T21:39:47.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7ff3917fa640 1 -- 172.21.15.60:0/1985325899 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff38c003970 con 0x7ff3940b1d30 2024-03-27T21:39:47.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7ff3917fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:47.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7ff392ffd640 4 set_mon_vals no callback set 2024-03-27T21:39:47.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7ff39b0e0640 10 client.4324.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:47.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7ff39b0e0640 15 client.4324.objecter _session_command_op_assign -1 1 2024-03-27T21:39:47.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7ff39b0e0640 20 client.4324.objecter _get_session osd=-1 returning homeless 2024-03-27T21:39:47.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7ff39b0e0640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:47.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f72857fa640 1 -- 172.21.15.60:0/1265218146 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f728800ad30 con 0x7f72901493e0 2024-03-27T21:39:47.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7fed0bfff640 4 set_mon_vals no callback set 2024-03-27T21:39:47.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7fed1b563640 10 client.4302.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:47.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7fed1b563640 15 client.4302.objecter _session_command_op_assign -1 1 2024-03-27T21:39:47.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7fed1b563640 20 client.4302.objecter _get_session osd=-1 returning homeless 2024-03-27T21:39:47.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7fed1b563640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:47.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7fed0affd640 1 -- 172.21.15.60:0/2146763059 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed14069420 con 0x7fed0c0cb330 2024-03-27T21:39:47.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f72857fa640 1 --2- 172.21.15.60:0/1265218146 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f727c06c660 0x7f727c06eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7fed1b563640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:47.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7fed1b563640 1 -- 172.21.15.60:0/2146763059 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fecfc0028a0 con 0x7fed0c0cb330 2024-03-27T21:39:47.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7fed0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7fed0affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:47.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f72857fa640 1 -- 172.21.15.60:0/1265218146 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7288088a70 con 0x7f72901493e0 2024-03-27T21:39:47.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f72857fa640 10 client.4321.objecter ms_dispatch 0x7f7290000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:47.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f72857fa640 3 client.4321.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:47.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f72857fa640 3 client.4321.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:47.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f72857fa640 10 client.4321.objecter checking command 1 2024-03-27T21:39:47.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f72857fa640 1 --2- 172.21.15.60:0/1265218146 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f727c0728d0 0x7f727c074cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7fed0affd640 10 monclient: dump: 2024-03-27T21:39:47.810 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.810 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.810 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.810 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.810 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.810 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.810 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.810 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.810 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.810 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f72857fa640 20 client.4321.objecter _get_session s=0x7f728800a7a0 osd=7 3 2024-03-27T21:39:47.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f72857fa640 20 client.4321.objecter put_session s=0x7f728800a7a0 osd=7 3 2024-03-27T21:39:47.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f72857fa640 15 client.4321.objecter _session_command_op_remove -1 1 2024-03-27T21:39:47.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f72857fa640 20 client.4321.objecter _get_session s=0x7f728800a7a0 osd=7 3 2024-03-27T21:39:47.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f72857fa640 20 client.4321.objecter get_session s=0x7f728800a7a0 osd=7 3 2024-03-27T21:39:47.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f72857fa640 15 client.4321.objecter _session_command_op_assign 7 1 2024-03-27T21:39:47.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f72857fa640 20 client.4321.objecter put_session s=0x7f728800a7a0 osd=7 4 2024-03-27T21:39:47.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f72857fa640 10 client.4321.objecter _send_command 1 2024-03-27T21:39:47.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f72857fa640 1 -- 172.21.15.60:0/1265218146 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f727c0753a0 con 0x7f727c0728d0 2024-03-27T21:39:47.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f72857fa640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-27T21:39:47.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7ff3917fa640 1 -- 172.21.15.60:0/1985325899 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c003d60 con 0x7ff3940b1d30 2024-03-27T21:39:47.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7ff3917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7ff3917fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:47.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7ff3917fa640 10 monclient: dump: 2024-03-27T21:39:47.812 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.812 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.812 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.812 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.812 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.812 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.813 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.813 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.813 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.813 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7ff3917fa640 1 -- 172.21.15.60:0/1985325899 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff38c0050d0 con 0x7ff3940b1d30 2024-03-27T21:39:47.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7ff3917fa640 1 --2- 172.21.15.60:0/1985325899 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff37806c6e0 0x7ff37806eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f40cf071640 5 monclient: authenticate success, global_id 4314 2024-03-27T21:39:47.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f40cf071640 10 monclient: _renew_subs 2024-03-27T21:39:47.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f40cf071640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:47.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f40cf071640 1 -- 172.21.15.60:0/2404035079 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c813e9c0 con 0x7f40c806ec20 2024-03-27T21:39:47.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f40cf071640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:47.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f40cf071640 10 monclient: _renew_subs 2024-03-27T21:39:47.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f40cf071640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:47.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f40cf071640 1 -- 172.21.15.60:0/2404035079 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c81d5f30 con 0x7f40c806ec20 2024-03-27T21:39:47.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.645+0000 7f40cf071640 1 librados: init done 2024-03-27T21:39:47.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f7294ade640 1 --2- 172.21.15.60:0/1265218146 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f727c06c660 0x7f727c06eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f7287fff640 1 --2- 172.21.15.60:0/1265218146 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f727c0728d0 0x7f727c074cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f07e97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:47.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f07e97fa640 1 -- 172.21.15.60:0/3415792531 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e00072f0 con 0x7f07ec0a8ef0 2024-03-27T21:39:47.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f07e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f7287fff640 10 monclient: get_auth_request con 0x7f727c0728d0 auth_method 0 2024-03-27T21:39:47.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f40c57fa640 1 -- 172.21.15.60:0/2404035079 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40bc032540 con 0x7f40c806ec20 2024-03-27T21:39:47.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f265a7fc640 4 set_mon_vals no callback set 2024-03-27T21:39:47.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f40c57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:47.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7ff3917fa640 1 -- 172.21.15.60:0/1985325899 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff38c088720 con 0x7ff3940b1d30 2024-03-27T21:39:47.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f40c57fa640 1 -- 172.21.15.60:0/2404035079 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc0328b0 con 0x7f40c806ec20 2024-03-27T21:39:47.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7ff3917fa640 10 client.4324.objecter ms_dispatch 0x7ff394000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:47.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f40c57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f40c57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f40c57fa640 10 monclient: dump: 2024-03-27T21:39:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.819 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f9678e3b640 10 client.4296.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:47.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f9678e3b640 15 client.4296.objecter _session_command_op_assign -1 1 2024-03-27T21:39:47.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f9678e3b640 1 --2- 172.21.15.60:0/2982922564 >> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] conn(0x7f967414e7c0 0x7f9674089140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f9678e3b640 20 client.4296.objecter _get_session s=0x7f96740ade50 osd=2 3 2024-03-27T21:39:47.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f9678e3b640 20 client.4296.objecter put_session s=0x7f96740ade50 osd=2 3 2024-03-27T21:39:47.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f9678e3b640 20 client.4296.objecter _get_session s=0x7f96740ade50 osd=2 3 2024-03-27T21:39:47.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f9678e3b640 15 client.4296.objecter _session_command_op_remove -1 1 2024-03-27T21:39:47.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f9678e3b640 20 client.4296.objecter get_session s=0x7f96740ade50 osd=2 3 2024-03-27T21:39:47.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f9678e3b640 15 client.4296.objecter _session_command_op_assign 2 1 2024-03-27T21:39:47.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f9678e3b640 20 client.4296.objecter put_session s=0x7f96740ade50 osd=2 4 2024-03-27T21:39:47.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f9678e3b640 10 client.4296.objecter _send_command 1 2024-03-27T21:39:47.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f9678e3b640 1 -- 172.21.15.60:0/2982922564 --> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96740a8f80 con 0x7f967414e7c0 2024-03-27T21:39:47.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7fed0affd640 1 -- 172.21.15.60:0/2146763059 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fed14092400 con 0x7fed0c0cb330 2024-03-27T21:39:47.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7ff39b0e0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:47.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7ff39b0e0640 1 -- 172.21.15.60:0/1985325899 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff3740028a0 con 0x7ff3940b1d30 2024-03-27T21:39:47.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7ff3917fa640 3 client.4324.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:47.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7ff3917fa640 3 client.4324.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:47.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f40c6ffd640 4 set_mon_vals no callback set 2024-03-27T21:39:47.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f07e97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:47.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f07e97fa640 10 monclient: dump: 2024-03-27T21:39:47.821 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:47.821 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:47.822 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.822 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:47.822 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:47.822 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:47.822 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:47.822 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:47.822 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:47.822 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:47.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7ff3917fa640 10 client.4324.objecter checking command 1 2024-03-27T21:39:47.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.644+0000 7f07eaffd640 4 set_mon_vals no callback set 2024-03-27T21:39:47.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f07e97fa640 1 -- 172.21.15.60:0/3415792531 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f07e0007510 con 0x7f07ec0a8ef0 2024-03-27T21:39:47.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7ff3917fa640 1 --2- 172.21.15.60:0/1985325899 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7ff378072950 0x7ff378074d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7ff3917fa640 20 client.4324.objecter _get_session s=0x7ff38c003970 osd=5 3 2024-03-27T21:39:47.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7ff3917fa640 20 client.4324.objecter put_session s=0x7ff38c003970 osd=5 3 2024-03-27T21:39:47.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7ff3917fa640 15 client.4324.objecter _session_command_op_remove -1 1 2024-03-27T21:39:47.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7ff3917fa640 20 client.4324.objecter _get_session s=0x7ff38c003970 osd=5 3 2024-03-27T21:39:47.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7ff3917fa640 20 client.4324.objecter get_session s=0x7ff38c003970 osd=5 3 2024-03-27T21:39:47.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f07d2ffd640 10 client.4327.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:47.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f07d2ffd640 15 client.4327.objecter _session_command_op_assign -1 1 2024-03-27T21:39:47.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7ff3917fa640 15 client.4324.objecter _session_command_op_assign 5 1 2024-03-27T21:39:47.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7ff3917fa640 20 client.4324.objecter put_session s=0x7ff38c003970 osd=5 4 2024-03-27T21:39:47.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f07d2ffd640 20 client.4327.objecter _get_session osd=-1 returning homeless 2024-03-27T21:39:47.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f07d2ffd640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:47.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7ff3917fa640 10 client.4324.objecter _send_command 1 2024-03-27T21:39:47.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f07d2ffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:47.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f07d2ffd640 1 -- 172.21.15.60:0/3415792531 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f07cc0028a0 con 0x7f07ec0a8ef0 2024-03-27T21:39:47.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7ff3917fa640 1 -- 172.21.15.60:0/1985325899 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff378075420 con 0x7ff378072950 2024-03-27T21:39:47.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7ff3917fa640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-27T21:39:47.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f72857fa640 1 -- 172.21.15.60:0/1265218146 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7288007d80 con 0x7f72901493e0 2024-03-27T21:39:47.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.646+0000 7f72857fa640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-03-27T21:39:47.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7fed0affd640 1 --2- 172.21.15.60:0/2146763059 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fecf806c660 0x7fecf806eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed0affd640 1 -- 172.21.15.60:0/2146763059 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fed14102240 con 0x7fed0c0cb330 2024-03-27T21:39:47.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed0affd640 10 client.4302.objecter ms_dispatch 0x7fed14000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:47.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed0affd640 3 client.4302.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:47.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed0affd640 3 client.4302.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:47.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f96737fe640 1 --2- 172.21.15.60:0/2982922564 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f966006c660 0x7f966006eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f96737fe640 10 monclient: get_auth_request con 0x7f966006c660 auth_method 0 2024-03-27T21:39:47.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f9673fff640 1 --2- 172.21.15.60:0/2982922564 >> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] conn(0x7f967414e7c0 0x7f9674089140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7f9673fff640 10 monclient: get_auth_request con 0x7f967414e7c0 auth_method 0 2024-03-27T21:39:47.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f2642ffd640 10 client.4333.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:47.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f2642ffd640 15 client.4333.objecter _session_command_op_assign -1 1 2024-03-27T21:39:47.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f2642ffd640 20 client.4333.objecter _get_session osd=-1 returning homeless 2024-03-27T21:39:47.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f2642ffd640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:47.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f2642ffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:47.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f2642ffd640 1 -- 172.21.15.60:0/3529972586 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f265c0adb80 con 0x7f265c0ae6a0 2024-03-27T21:39:47.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f26597fa640 1 -- 172.21.15.60:0/3529972586 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f264c0119d0 con 0x7f265c0ae6a0 2024-03-27T21:39:47.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f40cf071640 10 client.4314.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:47.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f40cf071640 15 client.4314.objecter _session_command_op_assign -1 1 2024-03-27T21:39:47.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f40cf071640 20 client.4314.objecter _get_session osd=-1 returning homeless 2024-03-27T21:39:47.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f40cf071640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:47.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f40cf071640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:47.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f40cf071640 1 -- 172.21.15.60:0/2404035079 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f40a80028a0 con 0x7f40c806ec20 2024-03-27T21:39:47.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.647+0000 7f7294ade640 10 monclient: get_auth_request con 0x7f727c06c660 auth_method 0 2024-03-27T21:39:47.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed0affd640 10 client.4302.objecter checking command 1 2024-03-27T21:39:47.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7f7294ade640 1 --2- 172.21.15.60:0/1265218146 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f727c06c660 0x7f727c06eb20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f728002b640 tx=0x7f7280008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed0affd640 1 --2- 172.21.15.60:0/2146763059 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fecf80728d0 0x7fecf8074cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7f07e97fa640 1 --2- 172.21.15.60:0/3415792531 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f07d806c660 0x7f07d806eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed0affd640 20 client.4302.objecter _get_session s=0x7fed140690b0 osd=0 3 2024-03-27T21:39:47.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed0affd640 20 client.4302.objecter put_session s=0x7fed140690b0 osd=0 3 2024-03-27T21:39:47.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed0affd640 15 client.4302.objecter _session_command_op_remove -1 1 2024-03-27T21:39:47.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed0affd640 20 client.4302.objecter _get_session s=0x7fed140690b0 osd=0 3 2024-03-27T21:39:47.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed1a561640 1 --2- 172.21.15.60:0/2146763059 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fecf806c660 0x7fecf806eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed0affd640 20 client.4302.objecter get_session s=0x7fed140690b0 osd=0 3 2024-03-27T21:39:47.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed0affd640 15 client.4302.objecter _session_command_op_assign 0 1 2024-03-27T21:39:47.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed19d60640 1 --2- 172.21.15.60:0/2146763059 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fecf80728d0 0x7fecf8074cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed1a561640 10 monclient: get_auth_request con 0x7fecf806c660 auth_method 0 2024-03-27T21:39:47.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed0affd640 20 client.4302.objecter put_session s=0x7fed140690b0 osd=0 4 2024-03-27T21:39:47.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed19d60640 10 monclient: get_auth_request con 0x7fecf80728d0 auth_method 0 2024-03-27T21:39:47.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed0affd640 10 client.4302.objecter _send_command 1 2024-03-27T21:39:47.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed0affd640 1 -- 172.21.15.60:0/2146763059 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecf80753a0 con 0x7fecf80728d0 2024-03-27T21:39:47.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed0affd640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-27T21:39:47.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed0affd640 1 -- 172.21.15.60:0/2146763059 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed140cd4b0 con 0x7fed0c0cb330 2024-03-27T21:39:47.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7fed0affd640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-03-27T21:39:47.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7f07f0d2e640 1 --2- 172.21.15.60:0/3415792531 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f07d806c660 0x7f07d806eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7f9673fff640 1 --2- 172.21.15.60:0/2982922564 >> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] conn(0x7f967414e7c0 0x7f9674089140 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7f965bfff640 10 client.4296.objecter ms_handle_connect 0x7f967414e7c0 2024-03-27T21:39:47.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7f7287fff640 1 --2- 172.21.15.60:0/1265218146 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f727c0728d0 0x7f727c074cf0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7f07f0d2e640 10 monclient: get_auth_request con 0x7f07d806c660 auth_method 0 2024-03-27T21:39:47.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7f96737fe640 1 --2- 172.21.15.60:0/2982922564 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f966006c660 0x7f966006eb20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f966800dd40 tx=0x7f966800f040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7f965bfff640 10 client.4296.objecter ms_handle_connect 0x7f966006c660 2024-03-27T21:39:47.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7f72857fa640 10 client.4321.objecter ms_handle_connect 0x7f727c06c660 2024-03-27T21:39:47.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7f72857fa640 10 client.4321.objecter ms_handle_connect 0x7f727c0728d0 2024-03-27T21:39:47.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7ff398e7c640 1 --2- 172.21.15.60:0/1985325899 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff37806c6e0 0x7ff37806eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7ff398e7c640 10 monclient: get_auth_request con 0x7ff37806c6e0 auth_method 0 2024-03-27T21:39:47.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7ff393fff640 1 --2- 172.21.15.60:0/1985325899 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7ff378072950 0x7ff378074d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7ff3917fa640 1 -- 172.21.15.60:0/1985325899 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff38c001690 con 0x7ff3940b1d30 2024-03-27T21:39:47.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7f07e97fa640 1 -- 172.21.15.60:0/3415792531 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f07e0088600 con 0x7f07ec0a8ef0 2024-03-27T21:39:47.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7f07e97fa640 10 client.4327.objecter ms_dispatch 0x7f07ec000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:47.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7f07e97fa640 3 client.4327.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:47.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7ff3917fa640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-03-27T21:39:47.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.648+0000 7f07e97fa640 3 client.4327.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:47.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7ff393fff640 10 monclient: get_auth_request con 0x7ff378072950 auth_method 0 2024-03-27T21:39:47.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f40c57fa640 1 -- 172.21.15.60:0/2404035079 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f40bc039050 con 0x7f40c806ec20 2024-03-27T21:39:47.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7fed0affd640 10 client.4302.objecter ms_handle_connect 0x7fecf80728d0 2024-03-27T21:39:47.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7fed19d60640 1 --2- 172.21.15.60:0/2146763059 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fecf80728d0 0x7fecf8074cf0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f07e97fa640 10 client.4327.objecter checking command 1 2024-03-27T21:39:47.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f07e97fa640 1 --2- 172.21.15.60:0/3415792531 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f07d8072a40 0x7f07d8074e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f07e97fa640 20 client.4327.objecter _get_session s=0x7f07d8072370 osd=1 3 2024-03-27T21:39:47.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f07e97fa640 20 client.4327.objecter put_session s=0x7f07d8072370 osd=1 3 2024-03-27T21:39:47.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f07e97fa640 15 client.4327.objecter _session_command_op_remove -1 1 2024-03-27T21:39:47.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f07e97fa640 20 client.4327.objecter _get_session s=0x7f07d8072370 osd=1 3 2024-03-27T21:39:47.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f07e97fa640 20 client.4327.objecter get_session s=0x7f07d8072370 osd=1 3 2024-03-27T21:39:47.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f07e97fa640 15 client.4327.objecter _session_command_op_assign 1 1 2024-03-27T21:39:47.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f07e97fa640 20 client.4327.objecter put_session s=0x7f07d8072370 osd=1 4 2024-03-27T21:39:47.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f07e97fa640 10 client.4327.objecter _send_command 1 2024-03-27T21:39:47.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f07e97fa640 1 -- 172.21.15.60:0/3415792531 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f07d8075510 con 0x7f07d8072a40 2024-03-27T21:39:47.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f07e97fa640 20 client.4327.objecter dump_active .. 0 homeless 2024-03-27T21:39:47.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f07e97fa640 1 -- 172.21.15.60:0/3415792531 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f07e0053af0 con 0x7f07ec0a8ef0 2024-03-27T21:39:47.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f07e97fa640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-03-27T21:39:47.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f3dc37fe640 1 -- 172.21.15.60:0/689226680 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3dcc02f050 con 0x7f3ddc060970 2024-03-27T21:39:47.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7fed1a561640 1 --2- 172.21.15.60:0/2146763059 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fecf806c660 0x7fecf806eb20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fed0400ba70 tx=0x7fed04008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f72857fa640 1 -- 172.21.15.60:0/1265218146 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f727c0753a0 con 0x7f727c0728d0 2024-03-27T21:39:47.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f72857fa640 10 client.4321.objecter ms_dispatch 0x7f7290000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:39:47.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f72857fa640 10 client.4321.objecter _finish_command 1 = system:0 2024-03-27T21:39:47.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7fed0affd640 10 client.4302.objecter ms_handle_connect 0x7fecf806c660 2024-03-27T21:39:47.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f72857fa640 20 client.4321.objecter put_session s=0x7f728800a7a0 osd=7 4 2024-03-27T21:39:47.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f72857fa640 15 client.4321.objecter _session_command_op_remove 7 1 2024-03-27T21:39:47.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f965bfff640 1 -- 172.21.15.60:0/2982922564 <== osd.2 v2:172.21.15.60:6824/34604 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f966c00a700 con 0x7f967414e7c0 2024-03-27T21:39:47.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f965bfff640 10 client.4296.objecter ms_dispatch 0x7f9674000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:39:47.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f965bfff640 10 client.4296.objecter _finish_command 1 = system:0 2024-03-27T21:39:47.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f965bfff640 20 client.4296.objecter put_session s=0x7f96740ade50 osd=2 4 2024-03-27T21:39:47.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f965bfff640 15 client.4296.objecter _session_command_op_remove 2 1 2024-03-27T21:39:47.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f07f152f640 1 --2- 172.21.15.60:0/3415792531 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f07d8072a40 0x7f07d8074e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f07f152f640 10 monclient: get_auth_request con 0x7f07d8072a40 auth_method 0 2024-03-27T21:39:47.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7ff398e7c640 1 --2- 172.21.15.60:0/1985325899 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff37806c6e0 0x7ff37806eba0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff39413ac50 tx=0x7ff380008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7ff3917fa640 10 client.4324.objecter ms_handle_connect 0x7ff37806c6e0 2024-03-27T21:39:47.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f26597fa640 1 --2- 172.21.15.60:0/3529972586 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f264806c660 0x7f264806eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f26597fa640 1 -- 172.21.15.60:0/3529972586 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f264c088670 con 0x7f265c0ae6a0 2024-03-27T21:39:47.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f07f0d2e640 1 --2- 172.21.15.60:0/3415792531 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f07d806c660 0x7f07d806eb20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f07dc036000 tx=0x7f07dc03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f07e97fa640 10 client.4327.objecter ms_handle_connect 0x7f07d806c660 2024-03-27T21:39:47.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f26597fa640 10 client.4333.objecter ms_dispatch 0x7f265c000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:47.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f26597fa640 3 client.4333.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:47.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f26597fa640 3 client.4333.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:47.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f26597fa640 10 client.4333.objecter checking command 1 2024-03-27T21:39:47.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f26597fa640 1 --2- 172.21.15.60:0/3529972586 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f2648072a40 0x7f2648074e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f26597fa640 20 client.4333.objecter _get_session s=0x7f2648072370 osd=3 3 2024-03-27T21:39:47.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f26597fa640 20 client.4333.objecter put_session s=0x7f2648072370 osd=3 3 2024-03-27T21:39:47.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f26597fa640 15 client.4333.objecter _session_command_op_remove -1 1 2024-03-27T21:39:47.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f26597fa640 20 client.4333.objecter _get_session s=0x7f2648072370 osd=3 3 2024-03-27T21:39:47.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f26597fa640 20 client.4333.objecter get_session s=0x7f2648072370 osd=3 3 2024-03-27T21:39:47.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f26597fa640 15 client.4333.objecter _session_command_op_assign 3 1 2024-03-27T21:39:47.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f26597fa640 20 client.4333.objecter put_session s=0x7f2648072370 osd=3 4 2024-03-27T21:39:47.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f26597fa640 10 client.4333.objecter _send_command 1 2024-03-27T21:39:47.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f26597fa640 1 -- 172.21.15.60:0/3529972586 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2648075510 con 0x7f2648072a40 2024-03-27T21:39:47.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f26597fa640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-27T21:39:47.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f26597fa640 1 -- 172.21.15.60:0/3529972586 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f264c08ac90 con 0x7f265c0ae6a0 2024-03-27T21:39:47.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f26597fa640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-03-27T21:39:47.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f07f152f640 1 --2- 172.21.15.60:0/3415792531 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f07d8072a40 0x7f07d8074e60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f40c57fa640 1 --2- 172.21.15.60:0/2404035079 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f40ac06c590 0x7f40ac06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f40c57fa640 1 -- 172.21.15.60:0/2404035079 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f40bc0aa9a0 con 0x7f40c806ec20 2024-03-27T21:39:47.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f40c57fa640 10 client.4314.objecter ms_dispatch 0x7f40c8000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:47.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f40c57fa640 3 client.4314.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:47.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f40c57fa640 3 client.4314.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:47.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f40c57fa640 10 client.4314.objecter checking command 1 2024-03-27T21:39:47.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f40c57fa640 1 --2- 172.21.15.60:0/2404035079 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f40ac072970 0x7f40ac074d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f40c57fa640 20 client.4314.objecter _get_session s=0x7f40ac0722a0 osd=6 3 2024-03-27T21:39:47.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f40c57fa640 20 client.4314.objecter put_session s=0x7f40ac0722a0 osd=6 3 2024-03-27T21:39:47.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f40c57fa640 15 client.4314.objecter _session_command_op_remove -1 1 2024-03-27T21:39:47.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f40c57fa640 20 client.4314.objecter _get_session s=0x7f40ac0722a0 osd=6 3 2024-03-27T21:39:47.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f40c57fa640 20 client.4314.objecter get_session s=0x7f40ac0722a0 osd=6 3 2024-03-27T21:39:47.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f40c57fa640 15 client.4314.objecter _session_command_op_assign 6 1 2024-03-27T21:39:47.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f40c57fa640 20 client.4314.objecter put_session s=0x7f40ac0722a0 osd=6 4 2024-03-27T21:39:47.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f40c57fa640 10 client.4314.objecter _send_command 1 2024-03-27T21:39:47.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f40c57fa640 1 -- 172.21.15.60:0/2404035079 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f40ac075440 con 0x7f40ac072970 2024-03-27T21:39:47.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f40c57fa640 20 client.4314.objecter dump_active .. 0 homeless 2024-03-27T21:39:47.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f40c57fa640 1 -- 172.21.15.60:0/2404035079 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40bc0af2a0 con 0x7f40c806ec20 2024-03-27T21:39:47.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.649+0000 7f40c57fa640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-03-27T21:39:47.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f40cd60e640 1 --2- 172.21.15.60:0/2404035079 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f40ac072970 0x7f40ac074d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f40c7fff640 1 --2- 172.21.15.60:0/2404035079 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f40ac06c590 0x7f40ac06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f07e97fa640 10 client.4327.objecter ms_handle_connect 0x7f07d8072a40 2024-03-27T21:39:47.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f40cd60e640 10 monclient: get_auth_request con 0x7f40ac072970 auth_method 0 2024-03-27T21:39:47.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f40c7fff640 10 monclient: get_auth_request con 0x7f40ac06c590 auth_method 0 2024-03-27T21:39:47.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f2660d66640 1 --2- 172.21.15.60:0/3529972586 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f264806c660 0x7f264806eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7ff393fff640 1 --2- 172.21.15.60:0/1985325899 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7ff378072950 0x7ff378074d70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7ff3917fa640 10 client.4324.objecter ms_handle_connect 0x7ff378072950 2024-03-27T21:39:47.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f2660d66640 10 monclient: get_auth_request con 0x7f264806c660 auth_method 0 2024-03-27T21:39:47.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7fed0affd640 1 -- 172.21.15.60:0/2146763059 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fecf80753a0 con 0x7fecf80728d0 2024-03-27T21:39:47.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7fed0affd640 10 client.4302.objecter ms_dispatch 0x7fed14000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:39:47.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7fed0affd640 10 client.4302.objecter _finish_command 1 = system:0 2024-03-27T21:39:47.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7fed0affd640 20 client.4302.objecter put_session s=0x7fed140690b0 osd=0 4 2024-03-27T21:39:47.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7fed0affd640 15 client.4302.objecter _session_command_op_remove 0 1 2024-03-27T21:39:47.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f2661567640 1 --2- 172.21.15.60:0/3529972586 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f2648072a40 0x7f2648074e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f2661567640 10 monclient: get_auth_request con 0x7f2648072a40 auth_method 0 2024-03-27T21:39:47.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f3dc37fe640 1 --2- 172.21.15.60:0/689226680 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3dc806c660 0x7f3dc806eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f3dc37fe640 1 -- 172.21.15.60:0/689226680 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3dcc0ab6c0 con 0x7f3ddc060970 2024-03-27T21:39:47.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f3dc37fe640 10 client.4311.objecter ms_dispatch 0x7f3ddc000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:47.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f3dc37fe640 3 client.4311.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:47.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f3dc37fe640 3 client.4311.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:47.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f3dc37fe640 10 client.4311.objecter checking command 1 2024-03-27T21:39:47.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f3dc37fe640 1 --2- 172.21.15.60:0/689226680 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f3dc8072a40 0x7f3dc8074e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:47.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f3dc37fe640 20 client.4311.objecter _get_session s=0x7f3dc8072370 osd=4 3 2024-03-27T21:39:47.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f3dc37fe640 20 client.4311.objecter put_session s=0x7f3dc8072370 osd=4 3 2024-03-27T21:39:47.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f3dc37fe640 15 client.4311.objecter _session_command_op_remove -1 1 2024-03-27T21:39:47.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f3dc37fe640 20 client.4311.objecter _get_session s=0x7f3dc8072370 osd=4 3 2024-03-27T21:39:47.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f3dc37fe640 20 client.4311.objecter get_session s=0x7f3dc8072370 osd=4 3 2024-03-27T21:39:47.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f3dc37fe640 15 client.4311.objecter _session_command_op_assign 4 1 2024-03-27T21:39:47.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f3dc37fe640 20 client.4311.objecter put_session s=0x7f3dc8072370 osd=4 4 2024-03-27T21:39:47.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f3dc37fe640 10 client.4311.objecter _send_command 1 2024-03-27T21:39:47.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f3dc37fe640 1 -- 172.21.15.60:0/689226680 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3dc8075510 con 0x7f3dc8072a40 2024-03-27T21:39:47.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f3dc37fe640 20 client.4311.objecter dump_active .. 0 homeless 2024-03-27T21:39:47.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f3dc37fe640 1 -- 172.21.15.60:0/689226680 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3dcc0aba70 con 0x7f3ddc060970 2024-03-27T21:39:47.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.650+0000 7f3dc37fe640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-03-27T21:39:47.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7f3ddb7fe640 1 --2- 172.21.15.60:0/689226680 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f3dc8072a40 0x7f3dc8074e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7f3ddb7fe640 10 monclient: get_auth_request con 0x7f3dc8072a40 auth_method 0 2024-03-27T21:39:47.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7f2660d66640 1 --2- 172.21.15.60:0/3529972586 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f264806c660 0x7f264806eb20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2654009fd0 tx=0x7f265400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7f26597fa640 10 client.4333.objecter ms_handle_connect 0x7f264806c660 2024-03-27T21:39:47.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7f3dda7fc640 1 --2- 172.21.15.60:0/689226680 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3dc806c660 0x7f3dc806eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:47.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7f3dda7fc640 10 monclient: get_auth_request con 0x7f3dc806c660 auth_method 0 2024-03-27T21:39:47.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7f07e97fa640 1 -- 172.21.15.60:0/3415792531 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f07d8075510 con 0x7f07d8072a40 2024-03-27T21:39:47.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7f07e97fa640 10 client.4327.objecter ms_dispatch 0x7f07ec000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:39:47.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7f07e97fa640 10 client.4327.objecter _finish_command 1 = system:0 2024-03-27T21:39:47.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7f07e97fa640 20 client.4327.objecter put_session s=0x7f07d8072370 osd=1 4 2024-03-27T21:39:47.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7f07e97fa640 15 client.4327.objecter _session_command_op_remove 1 1 2024-03-27T21:39:47.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7f2661567640 1 --2- 172.21.15.60:0/3529972586 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f2648072a40 0x7f2648074e60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7f26597fa640 10 client.4333.objecter ms_handle_connect 0x7f2648072a40 2024-03-27T21:39:47.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7f3dc37fe640 10 client.4311.objecter ms_handle_connect 0x7f3dc8072a40 2024-03-27T21:39:47.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7f3ddb7fe640 1 --2- 172.21.15.60:0/689226680 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f3dc8072a40 0x7f3dc8074e60 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7ff3917fa640 1 -- 172.21.15.60:0/1985325899 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7ff378075420 con 0x7ff378072950 2024-03-27T21:39:47.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7ff3917fa640 10 client.4324.objecter ms_dispatch 0x7ff394000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:39:47.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7f40c57fa640 10 client.4314.objecter ms_handle_connect 0x7f40ac06c590 2024-03-27T21:39:47.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7f40c7fff640 1 --2- 172.21.15.60:0/2404035079 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f40ac06c590 0x7f40ac06ea50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f40b4005880 tx=0x7f40b4001000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7f40cd60e640 1 --2- 172.21.15.60:0/2404035079 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f40ac072970 0x7f40ac074d90 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.652+0000 7f3dc37fe640 10 client.4311.objecter ms_handle_connect 0x7f3dc806c660 2024-03-27T21:39:47.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7ff3917fa640 10 client.4324.objecter _finish_command 1 = system:0 2024-03-27T21:39:47.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7ff3917fa640 20 client.4324.objecter put_session s=0x7ff38c003970 osd=5 4 2024-03-27T21:39:47.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.651+0000 7ff3917fa640 15 client.4324.objecter _session_command_op_remove 5 1 2024-03-27T21:39:47.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.652+0000 7f3dda7fc640 1 --2- 172.21.15.60:0/689226680 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3dc806c660 0x7f3dc806eb20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc142550 tx=0x7f3dd0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:47.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.652+0000 7f40c57fa640 10 client.4314.objecter ms_handle_connect 0x7f40ac072970 2024-03-27T21:39:47.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.652+0000 7f26597fa640 1 -- 172.21.15.60:0/3529972586 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f2648075510 con 0x7f2648072a40 2024-03-27T21:39:47.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.652+0000 7f26597fa640 10 client.4333.objecter ms_dispatch 0x7f265c000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:39:47.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.652+0000 7f26597fa640 10 client.4333.objecter _finish_command 1 = system:0 2024-03-27T21:39:47.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.652+0000 7f26597fa640 20 client.4333.objecter put_session s=0x7f2648072370 osd=3 4 2024-03-27T21:39:47.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.652+0000 7f26597fa640 15 client.4333.objecter _session_command_op_remove 3 1 2024-03-27T21:39:47.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.652+0000 7f3dc37fe640 1 -- 172.21.15.60:0/689226680 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f3dc8075510 con 0x7f3dc8072a40 2024-03-27T21:39:47.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.652+0000 7f3dc37fe640 10 client.4311.objecter ms_dispatch 0x7f3ddc000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:39:47.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.652+0000 7f3dc37fe640 10 client.4311.objecter _finish_command 1 = system:0 2024-03-27T21:39:47.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.652+0000 7f3dc37fe640 20 client.4311.objecter put_session s=0x7f3dc8072370 osd=4 4 2024-03-27T21:39:47.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.652+0000 7f3dc37fe640 15 client.4311.objecter _session_command_op_remove 4 1 2024-03-27T21:39:47.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.652+0000 7f40c57fa640 1 -- 172.21.15.60:0/2404035079 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f40ac075440 con 0x7f40ac072970 2024-03-27T21:39:47.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.652+0000 7f40c57fa640 10 client.4314.objecter ms_dispatch 0x7f40c8000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:39:47.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.652+0000 7f40c57fa640 10 client.4314.objecter _finish_command 1 = system:0 2024-03-27T21:39:47.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.653+0000 7f40c57fa640 20 client.4314.objecter put_session s=0x7f40ac0722a0 osd=6 4 2024-03-27T21:39:47.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.653+0000 7f40c57fa640 15 client.4314.objecter _session_command_op_remove 6 1 2024-03-27T21:39:47.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.680+0000 7f9678e3b640 10 client.4296.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-27T21:39:47.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.680+0000 7f9678e3b640 15 client.4296.objecter _session_command_op_assign -1 2 2024-03-27T21:39:47.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f9678e3b640 20 client.4296.objecter _get_session s=0x7f96740ade50 osd=2 3 2024-03-27T21:39:47.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f9678e3b640 20 client.4296.objecter put_session s=0x7f96740ade50 osd=2 3 2024-03-27T21:39:47.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f7296d42640 10 client.4321.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-27T21:39:47.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f7296d42640 15 client.4321.objecter _session_command_op_assign -1 2 2024-03-27T21:39:47.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f7296d42640 20 client.4321.objecter _get_session s=0x7f728800a7a0 osd=7 3 2024-03-27T21:39:47.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f7296d42640 20 client.4321.objecter put_session s=0x7f728800a7a0 osd=7 3 2024-03-27T21:39:47.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f7296d42640 20 client.4321.objecter _get_session s=0x7f728800a7a0 osd=7 3 2024-03-27T21:39:47.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f7296d42640 15 client.4321.objecter _session_command_op_remove -1 2 2024-03-27T21:39:47.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f7296d42640 20 client.4321.objecter get_session s=0x7f728800a7a0 osd=7 3 2024-03-27T21:39:47.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f9678e3b640 20 client.4296.objecter _get_session s=0x7f96740ade50 osd=2 3 2024-03-27T21:39:47.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f9678e3b640 15 client.4296.objecter _session_command_op_remove -1 2 2024-03-27T21:39:47.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f9678e3b640 20 client.4296.objecter get_session s=0x7f96740ade50 osd=2 3 2024-03-27T21:39:47.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f9678e3b640 15 client.4296.objecter _session_command_op_assign 2 2 2024-03-27T21:39:47.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f9678e3b640 20 client.4296.objecter put_session s=0x7f96740ade50 osd=2 4 2024-03-27T21:39:47.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f9678e3b640 10 client.4296.objecter _send_command 2 2024-03-27T21:39:47.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f9678e3b640 1 -- 172.21.15.60:0/2982922564 --> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f967408aba0 con 0x7f967414e7c0 2024-03-27T21:39:47.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f7296d42640 15 client.4321.objecter _session_command_op_assign 7 2 2024-03-27T21:39:47.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f7296d42640 20 client.4321.objecter put_session s=0x7f728800a7a0 osd=7 4 2024-03-27T21:39:47.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f7296d42640 10 client.4321.objecter _send_command 2 2024-03-27T21:39:47.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f7296d42640 1 -- 172.21.15.60:0/1265218146 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7290070030 con 0x7f727c0728d0 2024-03-27T21:39:47.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f965bfff640 1 -- 172.21.15.60:0/2982922564 <== osd.2 v2:172.21.15.60:6824/34604 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f966c00ac70 con 0x7f967414e7c0 2024-03-27T21:39:47.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f965bfff640 10 client.4296.objecter ms_dispatch 0x7f9674000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:39:47.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f965bfff640 10 client.4296.objecter _finish_command 2 = system:0 2024-03-27T21:39:47.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f965bfff640 20 client.4296.objecter put_session s=0x7f96740ade50 osd=2 4 2024-03-27T21:39:47.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.681+0000 7f965bfff640 15 client.4296.objecter _session_command_op_remove 2 2 2024-03-27T21:39:47.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.682+0000 7f72857fa640 1 -- 172.21.15.60:0/1265218146 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f728c001690 con 0x7f727c0728d0 2024-03-27T21:39:47.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.682+0000 7f72857fa640 10 client.4321.objecter ms_dispatch 0x7f7290000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:39:47.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.682+0000 7f72857fa640 10 client.4321.objecter _finish_command 2 = system:0 2024-03-27T21:39:47.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.682+0000 7f72857fa640 20 client.4321.objecter put_session s=0x7f728800a7a0 osd=7 4 2024-03-27T21:39:47.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.682+0000 7f72857fa640 15 client.4321.objecter _session_command_op_remove 7 2 2024-03-27T21:39:47.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.682+0000 7f9678e3b640 10 librados: watch_flush enter 2024-03-27T21:39:47.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.682+0000 7f9678e3b640 10 librados: watch_flush exit 2024-03-27T21:39:47.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.682+0000 7f9678e3b640 10 client.4296.objecter close_session for osd.2 2024-03-27T21:39:47.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.682+0000 7f9678e3b640 1 -- 172.21.15.60:0/2982922564 >> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] conn(0x7f967414e7c0 msgr2=0x7f9674089140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.682+0000 7f9678e3b640 1 --2- 172.21.15.60:0/2982922564 >> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] conn(0x7f967414e7c0 0x7f9674089140 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.682+0000 7f9678e3b640 20 client.4296.objecter put_session s=0x7f96740ade50 osd=2 1 2024-03-27T21:39:47.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.682+0000 7f7276ffd640 10 librados: watch_flush enter 2024-03-27T21:39:47.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.682+0000 7f9678e3b640 20 client.4296.objecter shutdown clearing up homeless session... 2024-03-27T21:39:47.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.682+0000 7f9678e3b640 10 client.4296.objecter successfully canceled tick 2024-03-27T21:39:47.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f9678e3b640 1 -- 172.21.15.60:0/2982922564 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f966006c660 msgr2=0x7f966006eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f9678e3b640 1 --2- 172.21.15.60:0/2982922564 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f966006c660 0x7f966006eb20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f966800dd40 tx=0x7f966800f040 comp rx=0 tx=0).stop 2024-03-27T21:39:47.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f7276ffd640 10 librados: watch_flush exit 2024-03-27T21:39:47.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f7276ffd640 10 client.4321.objecter close_session for osd.7 2024-03-27T21:39:47.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f9678e3b640 10 monclient: shutdown 2024-03-27T21:39:47.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f7276ffd640 1 -- 172.21.15.60:0/1265218146 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f727c0728d0 msgr2=0x7f727c074cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f7276ffd640 1 --2- 172.21.15.60:0/1265218146 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f727c0728d0 0x7f727c074cf0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f9678e3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:47.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f7276ffd640 20 client.4321.objecter put_session s=0x7f728800a7a0 osd=7 1 2024-03-27T21:39:47.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f9678e3b640 1 -- 172.21.15.60:0/2982922564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9674151da0 msgr2=0x7f96741f7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f7276ffd640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-27T21:39:47.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f7276ffd640 10 client.4321.objecter successfully canceled tick 2024-03-27T21:39:47.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f9678e3b640 1 --2- 172.21.15.60:0/2982922564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9674151da0 0x7f96741f7560 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9664000c00 tx=0x7f96640046c0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f9673fff640 1 -- 172.21.15.60:0/2982922564 reap_dead start 2024-03-27T21:39:47.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f7276ffd640 1 -- 172.21.15.60:0/1265218146 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f727c06c660 msgr2=0x7f727c06eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f7276ffd640 1 --2- 172.21.15.60:0/1265218146 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f727c06c660 0x7f727c06eb20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f728002b640 tx=0x7f7280008030 comp rx=0 tx=0).stop 2024-03-27T21:39:47.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f7276ffd640 10 monclient: shutdown 2024-03-27T21:39:47.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f7276ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:47.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f7276ffd640 1 -- 172.21.15.60:0/1265218146 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72901493e0 msgr2=0x7f7290138fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f7276ffd640 1 --2- 172.21.15.60:0/1265218146 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72901493e0 0x7f7290138fd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f728800cac0 tx=0x7f728800cf80 comp rx=0 tx=0).stop 2024-03-27T21:39:47.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f2642ffd640 10 client.4333.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-27T21:39:47.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f2642ffd640 15 client.4333.objecter _session_command_op_assign -1 2 2024-03-27T21:39:47.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f2642ffd640 20 client.4333.objecter _get_session s=0x7f2648072370 osd=3 3 2024-03-27T21:39:47.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f2642ffd640 20 client.4333.objecter put_session s=0x7f2648072370 osd=3 3 2024-03-27T21:39:47.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f2642ffd640 20 client.4333.objecter _get_session s=0x7f2648072370 osd=3 3 2024-03-27T21:39:47.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f2642ffd640 15 client.4333.objecter _session_command_op_remove -1 2 2024-03-27T21:39:47.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f2642ffd640 20 client.4333.objecter get_session s=0x7f2648072370 osd=3 3 2024-03-27T21:39:47.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f2642ffd640 15 client.4333.objecter _session_command_op_assign 3 2 2024-03-27T21:39:47.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f2642ffd640 20 client.4333.objecter put_session s=0x7f2648072370 osd=3 4 2024-03-27T21:39:47.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f2642ffd640 10 client.4333.objecter _send_command 2 2024-03-27T21:39:47.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f2642ffd640 1 -- 172.21.15.60:0/3529972586 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f265c069470 con 0x7f2648072a40 2024-03-27T21:39:47.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f07d2ffd640 10 client.4327.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-27T21:39:47.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f07d2ffd640 15 client.4327.objecter _session_command_op_assign -1 2 2024-03-27T21:39:47.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f07d2ffd640 20 client.4327.objecter _get_session s=0x7f07d8072370 osd=1 3 2024-03-27T21:39:47.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f07d2ffd640 20 client.4327.objecter put_session s=0x7f07d8072370 osd=1 3 2024-03-27T21:39:47.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f07d2ffd640 20 client.4327.objecter _get_session s=0x7f07d8072370 osd=1 3 2024-03-27T21:39:47.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f07d2ffd640 15 client.4327.objecter _session_command_op_remove -1 2 2024-03-27T21:39:47.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f07d2ffd640 20 client.4327.objecter get_session s=0x7f07d8072370 osd=1 3 2024-03-27T21:39:47.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f07d2ffd640 15 client.4327.objecter _session_command_op_assign 1 2 2024-03-27T21:39:47.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f07d2ffd640 20 client.4327.objecter put_session s=0x7f07d8072370 osd=1 4 2024-03-27T21:39:47.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f07d2ffd640 10 client.4327.objecter _send_command 2 2024-03-27T21:39:47.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f07d2ffd640 1 -- 172.21.15.60:0/3415792531 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f07cc0011a0 con 0x7f07d8072a40 2024-03-27T21:39:47.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f72952df640 1 -- 172.21.15.60:0/1265218146 reap_dead start 2024-03-27T21:39:47.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f9678e3b640 1 -- 172.21.15.60:0/2982922564 shutdown_connections 2024-03-27T21:39:47.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f9678e3b640 1 -- 172.21.15.60:0/2982922564 >> 172.21.15.60:0/2982922564 conn(0x7f9674060700 msgr2=0x7f96740afc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:47.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f7276ffd640 1 -- 172.21.15.60:0/1265218146 shutdown_connections 2024-03-27T21:39:47.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7fed1b563640 10 client.4302.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-27T21:39:47.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7fed1b563640 15 client.4302.objecter _session_command_op_assign -1 2 2024-03-27T21:39:47.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7fed1b563640 20 client.4302.objecter _get_session s=0x7fed140690b0 osd=0 3 2024-03-27T21:39:47.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7fed1b563640 20 client.4302.objecter put_session s=0x7fed140690b0 osd=0 3 2024-03-27T21:39:47.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f7276ffd640 1 -- 172.21.15.60:0/1265218146 >> 172.21.15.60:0/1265218146 conn(0x7f72900b3ae0 msgr2=0x7f72900b62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:47.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7fed1b563640 20 client.4302.objecter _get_session s=0x7fed140690b0 osd=0 3 2024-03-27T21:39:47.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7fed1b563640 15 client.4302.objecter _session_command_op_remove -1 2 2024-03-27T21:39:47.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7fed1b563640 20 client.4302.objecter get_session s=0x7fed140690b0 osd=0 3 2024-03-27T21:39:47.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7fed1b563640 15 client.4302.objecter _session_command_op_assign 0 2 2024-03-27T21:39:47.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7fed1b563640 20 client.4302.objecter put_session s=0x7fed140690b0 osd=0 4 2024-03-27T21:39:47.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7fed1b563640 10 client.4302.objecter _send_command 2 2024-03-27T21:39:47.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7fed1b563640 1 -- 172.21.15.60:0/2146763059 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fecfc0011a0 con 0x7fecf80728d0 2024-03-27T21:39:47.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f26597fa640 1 -- 172.21.15.60:0/3529972586 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f265c069470 con 0x7f2648072a40 2024-03-27T21:39:47.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f26597fa640 10 client.4333.objecter ms_dispatch 0x7f265c000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:39:47.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f26597fa640 10 client.4333.objecter _finish_command 2 = system:0 2024-03-27T21:39:47.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f26597fa640 20 client.4333.objecter put_session s=0x7f2648072370 osd=3 4 2024-03-27T21:39:47.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f26597fa640 15 client.4333.objecter _session_command_op_remove 3 2 2024-03-27T21:39:47.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f3dc17fa640 10 client.4311.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-27T21:39:47.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f3dc17fa640 15 client.4311.objecter _session_command_op_assign -1 2 2024-03-27T21:39:47.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f3dc17fa640 20 client.4311.objecter _get_session s=0x7f3dc8072370 osd=4 3 2024-03-27T21:39:47.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f3dc17fa640 20 client.4311.objecter put_session s=0x7f3dc8072370 osd=4 3 2024-03-27T21:39:47.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f3dc17fa640 20 client.4311.objecter _get_session s=0x7f3dc8072370 osd=4 3 2024-03-27T21:39:47.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f3dc17fa640 15 client.4311.objecter _session_command_op_remove -1 2 2024-03-27T21:39:47.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f3dc17fa640 20 client.4311.objecter get_session s=0x7f3dc8072370 osd=4 3 2024-03-27T21:39:47.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f3dc17fa640 15 client.4311.objecter _session_command_op_assign 4 2 2024-03-27T21:39:47.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f3dc17fa640 20 client.4311.objecter put_session s=0x7f3dc8072370 osd=4 4 2024-03-27T21:39:47.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f3dc17fa640 10 client.4311.objecter _send_command 2 2024-03-27T21:39:47.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f3dc17fa640 1 -- 172.21.15.60:0/689226680 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3ddc1db240 con 0x7f3dc8072a40 2024-03-27T21:39:47.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f7276ffd640 1 -- 172.21.15.60:0/1265218146 shutdown_connections 2024-03-27T21:39:47.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f7276ffd640 1 -- 172.21.15.60:0/1265218146 wait complete. 2024-03-27T21:39:47.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.683+0000 7f7276ffd640 1 librados: shutdown 2024-03-27T21:39:47.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7fed0affd640 1 -- 172.21.15.60:0/2146763059 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fecfc0011a0 con 0x7fecf80728d0 2024-03-27T21:39:47.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7fed0affd640 10 client.4302.objecter ms_dispatch 0x7fed14000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:39:47.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f9678e3b640 1 -- 172.21.15.60:0/2982922564 shutdown_connections 2024-03-27T21:39:47.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7fed0affd640 10 client.4302.objecter _finish_command 2 = system:0 2024-03-27T21:39:47.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7fed0affd640 20 client.4302.objecter put_session s=0x7fed140690b0 osd=0 4 2024-03-27T21:39:47.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7fed0affd640 15 client.4302.objecter _session_command_op_remove 0 2 2024-03-27T21:39:47.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f07e97fa640 1 -- 172.21.15.60:0/3415792531 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f07cc0011a0 con 0x7f07d8072a40 2024-03-27T21:39:47.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f07e97fa640 10 client.4327.objecter ms_dispatch 0x7f07ec000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:39:47.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f07e97fa640 10 client.4327.objecter _finish_command 2 = system:0 2024-03-27T21:39:47.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f07e97fa640 20 client.4327.objecter put_session s=0x7f07d8072370 osd=1 4 2024-03-27T21:39:47.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f07e97fa640 15 client.4327.objecter _session_command_op_remove 1 2 2024-03-27T21:39:47.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f9678e3b640 1 -- 172.21.15.60:0/2982922564 wait complete. 2024-03-27T21:39:47.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f9678e3b640 1 librados: shutdown 2024-03-27T21:39:47.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7ff39b0e0640 10 client.4324.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-27T21:39:47.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7ff39b0e0640 15 client.4324.objecter _session_command_op_assign -1 2 2024-03-27T21:39:47.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7ff39b0e0640 20 client.4324.objecter _get_session s=0x7ff38c003970 osd=5 3 2024-03-27T21:39:47.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7ff39b0e0640 20 client.4324.objecter put_session s=0x7ff38c003970 osd=5 3 2024-03-27T21:39:47.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7ff39b0e0640 20 client.4324.objecter _get_session s=0x7ff38c003970 osd=5 3 2024-03-27T21:39:47.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7ff39b0e0640 15 client.4324.objecter _session_command_op_remove -1 2 2024-03-27T21:39:47.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7ff39b0e0640 20 client.4324.objecter get_session s=0x7ff38c003970 osd=5 3 2024-03-27T21:39:47.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7ff39b0e0640 15 client.4324.objecter _session_command_op_assign 5 2 2024-03-27T21:39:47.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7ff39b0e0640 20 client.4324.objecter put_session s=0x7ff38c003970 osd=5 4 2024-03-27T21:39:47.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7ff39b0e0640 10 client.4324.objecter _send_command 2 2024-03-27T21:39:47.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7ff39b0e0640 1 -- 172.21.15.60:0/1985325899 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff3740011d0 con 0x7ff378072950 2024-03-27T21:39:47.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7fed1b563640 10 librados: watch_flush enter 2024-03-27T21:39:47.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f2642ffd640 10 librados: watch_flush enter 2024-03-27T21:39:47.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f2642ffd640 10 librados: watch_flush exit 2024-03-27T21:39:47.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f2642ffd640 10 client.4333.objecter close_session for osd.3 2024-03-27T21:39:47.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f2642ffd640 1 -- 172.21.15.60:0/3529972586 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f2648072a40 msgr2=0x7f2648074e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f2642ffd640 1 --2- 172.21.15.60:0/3529972586 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f2648072a40 0x7f2648074e60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f2642ffd640 20 client.4333.objecter put_session s=0x7f2648072370 osd=3 1 2024-03-27T21:39:47.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f2642ffd640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-27T21:39:47.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f2642ffd640 10 client.4333.objecter successfully canceled tick 2024-03-27T21:39:47.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7fed1b563640 10 librados: watch_flush exit 2024-03-27T21:39:47.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7fed1b563640 10 client.4302.objecter close_session for osd.0 2024-03-27T21:39:47.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7fed1b563640 1 -- 172.21.15.60:0/2146763059 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fecf80728d0 msgr2=0x7fecf8074cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7fed1b563640 1 --2- 172.21.15.60:0/2146763059 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fecf80728d0 0x7fecf8074cf0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7fed1b563640 20 client.4302.objecter put_session s=0x7fed140690b0 osd=0 1 2024-03-27T21:39:47.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7fed1b563640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-27T21:39:47.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f2642ffd640 1 -- 172.21.15.60:0/3529972586 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f264806c660 msgr2=0x7f264806eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f2642ffd640 1 --2- 172.21.15.60:0/3529972586 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f264806c660 0x7f264806eb20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2654009fd0 tx=0x7f265400b040 comp rx=0 tx=0).stop 2024-03-27T21:39:47.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f2642ffd640 10 monclient: shutdown 2024-03-27T21:39:47.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7fed1b563640 10 client.4302.objecter successfully canceled tick 2024-03-27T21:39:47.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f2642ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:47.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f2642ffd640 1 -- 172.21.15.60:0/3529972586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f265c0ae6a0 msgr2=0x7f265c1c5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f2642ffd640 1 --2- 172.21.15.60:0/3529972586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f265c0ae6a0 0x7f265c1c5500 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f264c00c430 tx=0x7f264c00c8f0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f3dc37fe640 1 -- 172.21.15.60:0/689226680 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3ddc1db240 con 0x7f3dc8072a40 2024-03-27T21:39:47.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f3dc37fe640 10 client.4311.objecter ms_dispatch 0x7f3ddc000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:39:47.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f3dc37fe640 10 client.4311.objecter _finish_command 2 = system:0 2024-03-27T21:39:47.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f3dc37fe640 20 client.4311.objecter put_session s=0x7f3dc8072370 osd=4 4 2024-03-27T21:39:47.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f3dc37fe640 15 client.4311.objecter _session_command_op_remove 4 2 2024-03-27T21:39:47.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f2661567640 1 -- 172.21.15.60:0/3529972586 reap_dead start 2024-03-27T21:39:47.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f2642ffd640 1 -- 172.21.15.60:0/3529972586 shutdown_connections 2024-03-27T21:39:47.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f2642ffd640 1 -- 172.21.15.60:0/3529972586 >> 172.21.15.60:0/3529972586 conn(0x7f265c0b2ea0 msgr2=0x7f265c0b24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:47.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7fed1b563640 1 -- 172.21.15.60:0/2146763059 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fecf806c660 msgr2=0x7fecf806eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7fed1b563640 1 --2- 172.21.15.60:0/2146763059 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fecf806c660 0x7fecf806eb20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fed0400ba70 tx=0x7fed04008040 comp rx=0 tx=0).stop 2024-03-27T21:39:47.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.684+0000 7f07d2ffd640 10 librados: watch_flush enter 2024-03-27T21:39:47.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f07d2ffd640 10 librados: watch_flush exit 2024-03-27T21:39:47.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f07d2ffd640 10 client.4327.objecter close_session for osd.1 2024-03-27T21:39:47.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f07d2ffd640 1 -- 172.21.15.60:0/3415792531 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f07d8072a40 msgr2=0x7f07d8074e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f2642ffd640 1 -- 172.21.15.60:0/3529972586 shutdown_connections 2024-03-27T21:39:47.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f07d2ffd640 1 --2- 172.21.15.60:0/3415792531 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f07d8072a40 0x7f07d8074e60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7ff3917fa640 1 -- 172.21.15.60:0/1985325899 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff3740011d0 con 0x7ff378072950 2024-03-27T21:39:47.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f2642ffd640 1 -- 172.21.15.60:0/3529972586 wait complete. 2024-03-27T21:39:47.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7ff3917fa640 10 client.4324.objecter ms_dispatch 0x7ff394000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:39:47.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7ff3917fa640 10 client.4324.objecter _finish_command 2 = system:0 2024-03-27T21:39:47.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7ff3917fa640 20 client.4324.objecter put_session s=0x7ff38c003970 osd=5 4 2024-03-27T21:39:47.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7ff3917fa640 15 client.4324.objecter _session_command_op_remove 5 2 2024-03-27T21:39:47.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f07d2ffd640 20 client.4327.objecter put_session s=0x7f07d8072370 osd=1 1 2024-03-27T21:39:47.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f07d2ffd640 20 client.4327.objecter shutdown clearing up homeless session... 2024-03-27T21:39:47.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7fed1b563640 10 monclient: shutdown 2024-03-27T21:39:47.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f07d2ffd640 10 client.4327.objecter successfully canceled tick 2024-03-27T21:39:47.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f2642ffd640 1 librados: shutdown 2024-03-27T21:39:47.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7fed1b563640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:47.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f3dc17fa640 10 librados: watch_flush enter 2024-03-27T21:39:47.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7fed1b563640 1 -- 172.21.15.60:0/2146763059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed0c0cb330 msgr2=0x7fed0c0c89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7fed1b563640 1 --2- 172.21.15.60:0/2146763059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed0c0cb330 0x7fed0c0c89f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fed14072db0 tx=0x7fed14067040 comp rx=0 tx=0).stop 2024-03-27T21:39:47.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f3dc17fa640 10 librados: watch_flush exit 2024-03-27T21:39:47.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f3dc17fa640 10 client.4311.objecter close_session for osd.4 2024-03-27T21:39:47.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7fed1ad62640 1 -- 172.21.15.60:0/2146763059 reap_dead start 2024-03-27T21:39:47.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f07d2ffd640 1 -- 172.21.15.60:0/3415792531 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f07d806c660 msgr2=0x7f07d806eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f3dc17fa640 1 -- 172.21.15.60:0/689226680 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f3dc8072a40 msgr2=0x7f3dc8074e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f07d2ffd640 1 --2- 172.21.15.60:0/3415792531 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f07d806c660 0x7f07d806eb20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f07dc036000 tx=0x7f07dc03d040 comp rx=0 tx=0).stop 2024-03-27T21:39:47.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f3dc17fa640 1 --2- 172.21.15.60:0/689226680 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f3dc8072a40 0x7f3dc8074e60 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7fed1b563640 1 -- 172.21.15.60:0/2146763059 shutdown_connections 2024-03-27T21:39:47.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7fed1b563640 1 -- 172.21.15.60:0/2146763059 >> 172.21.15.60:0/2146763059 conn(0x7fed0c007930 msgr2=0x7fed0c0c4a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:47.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f3dc17fa640 20 client.4311.objecter put_session s=0x7f3dc8072370 osd=4 1 2024-03-27T21:39:47.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f07d2ffd640 10 monclient: shutdown 2024-03-27T21:39:47.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f3dc17fa640 20 client.4311.objecter shutdown clearing up homeless session... 2024-03-27T21:39:47.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f07d2ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:47.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f3dc17fa640 10 client.4311.objecter successfully canceled tick 2024-03-27T21:39:47.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f07d2ffd640 1 -- 172.21.15.60:0/3415792531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f07ec0a8ef0 msgr2=0x7f07ec1da820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f07d2ffd640 1 --2- 172.21.15.60:0/3415792531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f07ec0a8ef0 0x7f07ec1da820 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f07e000c480 tx=0x7f07e000c940 comp rx=0 tx=0).stop 2024-03-27T21:39:47.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f07f152f640 1 -- 172.21.15.60:0/3415792531 reap_dead start 2024-03-27T21:39:47.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f07d2ffd640 1 -- 172.21.15.60:0/3415792531 shutdown_connections 2024-03-27T21:39:47.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7f07d2ffd640 1 -- 172.21.15.60:0/3415792531 >> 172.21.15.60:0/3415792531 conn(0x7f07ec0b2c20 msgr2=0x7f07ec0ad300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:47.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7fed1b563640 1 -- 172.21.15.60:0/2146763059 shutdown_connections 2024-03-27T21:39:47.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7fed1b563640 1 -- 172.21.15.60:0/2146763059 wait complete. 2024-03-27T21:39:47.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7fed1b563640 1 librados: shutdown 2024-03-27T21:39:47.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7ff39b0e0640 10 librados: watch_flush enter 2024-03-27T21:39:47.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7ff39b0e0640 10 librados: watch_flush exit 2024-03-27T21:39:47.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7ff39b0e0640 10 client.4324.objecter close_session for osd.5 2024-03-27T21:39:47.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7ff39b0e0640 1 -- 172.21.15.60:0/1985325899 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7ff378072950 msgr2=0x7ff378074d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7ff39b0e0640 1 --2- 172.21.15.60:0/1985325899 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7ff378072950 0x7ff378074d70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7ff39b0e0640 20 client.4324.objecter put_session s=0x7ff38c003970 osd=5 1 2024-03-27T21:39:47.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7ff39b0e0640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-27T21:39:47.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.685+0000 7ff39b0e0640 10 client.4324.objecter successfully canceled tick 2024-03-27T21:39:47.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.686+0000 7ff39b0e0640 1 -- 172.21.15.60:0/1985325899 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff37806c6e0 msgr2=0x7ff37806eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.686+0000 7ff39b0e0640 1 --2- 172.21.15.60:0/1985325899 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff37806c6e0 0x7ff37806eba0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff39413ac50 tx=0x7ff380008040 comp rx=0 tx=0).stop 2024-03-27T21:39:47.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.686+0000 7ff39b0e0640 10 monclient: shutdown 2024-03-27T21:39:47.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.686+0000 7ff39b0e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:47.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.686+0000 7ff39b0e0640 1 -- 172.21.15.60:0/1985325899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3940b1d30 msgr2=0x7ff3940b4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.686+0000 7ff39b0e0640 1 --2- 172.21.15.60:0/1985325899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3940b1d30 0x7ff3940b4ad0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00c730 tx=0x7ff38c00cbf0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.686+0000 7ff39967d640 1 -- 172.21.15.60:0/1985325899 reap_dead start 2024-03-27T21:39:47.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.686+0000 7ff39b0e0640 1 -- 172.21.15.60:0/1985325899 shutdown_connections 2024-03-27T21:39:47.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.686+0000 7ff39b0e0640 1 -- 172.21.15.60:0/1985325899 >> 172.21.15.60:0/1985325899 conn(0x7ff3940b7300 msgr2=0x7ff3940b3970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:47.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.686+0000 7f3dc17fa640 1 -- 172.21.15.60:0/689226680 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3dc806c660 msgr2=0x7f3dc806eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.686+0000 7f3dc17fa640 1 --2- 172.21.15.60:0/689226680 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3dc806c660 0x7f3dc806eb20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc142550 tx=0x7f3dd0009040 comp rx=0 tx=0).stop 2024-03-27T21:39:47.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.686+0000 7f3dc17fa640 10 monclient: shutdown 2024-03-27T21:39:47.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.686+0000 7f3dc17fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:47.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.686+0000 7f3dc17fa640 1 -- 172.21.15.60:0/689226680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ddc060970 msgr2=0x7f3ddc140cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.686+0000 7f3dc17fa640 1 --2- 172.21.15.60:0/689226680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ddc060970 0x7f3ddc140cb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc009890 tx=0x7f3dcc0046c0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.686+0000 7f3ddb7fe640 1 -- 172.21.15.60:0/689226680 reap_dead start 2024-03-27T21:39:47.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.686+0000 7f3dc17fa640 1 -- 172.21.15.60:0/689226680 shutdown_connections 2024-03-27T21:39:47.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.686+0000 7f3dc17fa640 1 -- 172.21.15.60:0/689226680 >> 172.21.15.60:0/689226680 conn(0x7f3ddc069ce0 msgr2=0x7f3ddc0aeca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:47.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.686+0000 7f40cf071640 10 client.4314.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-27T21:39:47.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.686+0000 7f40cf071640 15 client.4314.objecter _session_command_op_assign -1 2 2024-03-27T21:39:47.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7f40cf071640 20 client.4314.objecter _get_session s=0x7f40ac0722a0 osd=6 3 2024-03-27T21:39:47.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7f40cf071640 20 client.4314.objecter put_session s=0x7f40ac0722a0 osd=6 3 2024-03-27T21:39:47.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7f40cf071640 20 client.4314.objecter _get_session s=0x7f40ac0722a0 osd=6 3 2024-03-27T21:39:47.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7f40cf071640 15 client.4314.objecter _session_command_op_remove -1 2 2024-03-27T21:39:47.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7ff39b0e0640 1 -- 172.21.15.60:0/1985325899 shutdown_connections 2024-03-27T21:39:47.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7f40cf071640 20 client.4314.objecter get_session s=0x7f40ac0722a0 osd=6 3 2024-03-27T21:39:47.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7ff39b0e0640 1 -- 172.21.15.60:0/1985325899 wait complete. 2024-03-27T21:39:47.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7f40cf071640 15 client.4314.objecter _session_command_op_assign 6 2 2024-03-27T21:39:47.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7f40cf071640 20 client.4314.objecter put_session s=0x7f40ac0722a0 osd=6 4 2024-03-27T21:39:47.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7f40cf071640 10 client.4314.objecter _send_command 2 2024-03-27T21:39:47.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7f40cf071640 1 -- 172.21.15.60:0/2404035079 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f40a80011a0 con 0x7f40ac072970 2024-03-27T21:39:47.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.686+0000 7f07d2ffd640 1 -- 172.21.15.60:0/3415792531 shutdown_connections 2024-03-27T21:39:47.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7ff39b0e0640 1 librados: shutdown 2024-03-27T21:39:47.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7f07d2ffd640 1 -- 172.21.15.60:0/3415792531 wait complete. 2024-03-27T21:39:47.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7f07d2ffd640 1 librados: shutdown 2024-03-27T21:39:47.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7f3dc17fa640 1 -- 172.21.15.60:0/689226680 shutdown_connections 2024-03-27T21:39:47.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7f3dc17fa640 1 -- 172.21.15.60:0/689226680 wait complete. 2024-03-27T21:39:47.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7f3dc17fa640 1 librados: shutdown 2024-03-27T21:39:47.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7f40c57fa640 1 -- 172.21.15.60:0/2404035079 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f40a80011a0 con 0x7f40ac072970 2024-03-27T21:39:47.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7f40c57fa640 10 client.4314.objecter ms_dispatch 0x7f40c8000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:39:47.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7f40c57fa640 10 client.4314.objecter _finish_command 2 = system:0 2024-03-27T21:39:47.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7f40c57fa640 20 client.4314.objecter put_session s=0x7f40ac0722a0 osd=6 4 2024-03-27T21:39:47.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.687+0000 7f40c57fa640 15 client.4314.objecter _session_command_op_remove 6 2 2024-03-27T21:39:47.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.688+0000 7f40cf071640 10 librados: watch_flush enter 2024-03-27T21:39:47.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.688+0000 7f40cf071640 10 librados: watch_flush exit 2024-03-27T21:39:47.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.689+0000 7f40cf071640 10 client.4314.objecter close_session for osd.6 2024-03-27T21:39:47.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.689+0000 7f40cf071640 1 -- 172.21.15.60:0/2404035079 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f40ac072970 msgr2=0x7f40ac074d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.689+0000 7f40cf071640 1 --2- 172.21.15.60:0/2404035079 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f40ac072970 0x7f40ac074d90 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.689+0000 7f40cf071640 20 client.4314.objecter put_session s=0x7f40ac0722a0 osd=6 1 2024-03-27T21:39:47.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.689+0000 7f40cf071640 20 client.4314.objecter shutdown clearing up homeless session... 2024-03-27T21:39:47.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.689+0000 7f40cf071640 10 client.4314.objecter successfully canceled tick 2024-03-27T21:39:47.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.689+0000 7f40cf071640 1 -- 172.21.15.60:0/2404035079 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f40ac06c590 msgr2=0x7f40ac06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.689+0000 7f40cf071640 1 --2- 172.21.15.60:0/2404035079 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f40ac06c590 0x7f40ac06ea50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f40b4005880 tx=0x7f40b4001000 comp rx=0 tx=0).stop 2024-03-27T21:39:47.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.689+0000 7f40cf071640 10 monclient: shutdown 2024-03-27T21:39:47.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.689+0000 7f40cf071640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:47.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.689+0000 7f40cf071640 1 -- 172.21.15.60:0/2404035079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40c806ec20 msgr2=0x7f40c80b13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:47.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.689+0000 7f40cf071640 1 --2- 172.21.15.60:0/2404035079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40c806ec20 0x7f40c80b13c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f40bc0098c0 tx=0x7f40bc0046c0 comp rx=0 tx=0).stop 2024-03-27T21:39:47.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.689+0000 7f40cd60e640 1 -- 172.21.15.60:0/2404035079 reap_dead start 2024-03-27T21:39:47.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.689+0000 7f40cf071640 1 -- 172.21.15.60:0/2404035079 shutdown_connections 2024-03-27T21:39:47.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.689+0000 7f40cf071640 1 -- 172.21.15.60:0/2404035079 >> 172.21.15.60:0/2404035079 conn(0x7f40c80b2cc0 msgr2=0x7f40c8134560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:47.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.690+0000 7f40cf071640 1 -- 172.21.15.60:0/2404035079 shutdown_connections 2024-03-27T21:39:47.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.690+0000 7f40cf071640 1 -- 172.21.15.60:0/2404035079 wait complete. 2024-03-27T21:39:47.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:47.690+0000 7f40cf071640 1 librados: shutdown 2024-03-27T21:39:47.907 INFO:teuthology.orchestra.run.smithi060.stdout:60129542149 2024-03-27T21:39:47.907 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-27T21:39:47.908 INFO:teuthology.orchestra.run.smithi060.stdout:68719476741 2024-03-27T21:39:47.908 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-27T21:39:47.909 INFO:teuthology.orchestra.run.smithi060.stdout:60129542149 2024-03-27T21:39:47.909 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-27T21:39:47.910 INFO:teuthology.orchestra.run.smithi060.stdout:60129542149 2024-03-27T21:39:47.910 INFO:teuthology.orchestra.run.smithi060.stdout:68719476742 2024-03-27T21:39:47.911 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-27T21:39:47.911 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-27T21:39:47.912 INFO:teuthology.orchestra.run.smithi060.stdout:60129542149 2024-03-27T21:39:47.912 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-27T21:39:47.913 INFO:teuthology.orchestra.run.smithi060.stdout:55834574853 2024-03-27T21:39:47.913 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-27T21:39:47.914 INFO:teuthology.orchestra.run.smithi060.stdout:60129542149 2024-03-27T21:39:47.915 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-27T21:39:48.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.131+0000 7fefae170640 10 monclient: get_monmap_and_config 2024-03-27T21:39:48.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.131+0000 7fefae170640 10 monclient: build_initial_monmap 2024-03-27T21:39:48.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.131+0000 7fefae170640 10 monclient: monmap: 2024-03-27T21:39:48.119 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:48.119 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.119 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:48.133521+0000 2024-03-27T21:39:48.119 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:48.133521+0000 2024-03-27T21:39:48.119 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:48.120 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.120 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:48.120 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:48.120 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:48.120 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:48.120 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:48.120 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.131+0000 7fdba80fb640 10 monclient: get_monmap_and_config 2024-03-27T21:39:48.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.131+0000 7fdba80fb640 10 monclient: build_initial_monmap 2024-03-27T21:39:48.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.131+0000 7fdba80fb640 10 monclient: monmap: 2024-03-27T21:39:48.121 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:48.121 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.121 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:48.133838+0000 2024-03-27T21:39:48.121 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:48.133838+0000 2024-03-27T21:39:48.121 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:48.121 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.121 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:48.121 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:48.121 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:48.122 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:48.122 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:48.122 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.131+0000 7fbd91c8b640 10 monclient: get_monmap_and_config 2024-03-27T21:39:48.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.132+0000 7fbd91c8b640 10 monclient: build_initial_monmap 2024-03-27T21:39:48.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.132+0000 7fbd91c8b640 10 monclient: monmap: 2024-03-27T21:39:48.123 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:48.123 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.123 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:48.134020+0000 2024-03-27T21:39:48.123 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:48.134020+0000 2024-03-27T21:39:48.123 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:48.123 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.123 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:48.123 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:48.123 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:48.123 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:48.124 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:48.124 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.132+0000 7fefae170640 1 Processor -- start 2024-03-27T21:39:48.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.132+0000 7fefae170640 1 -- start start 2024-03-27T21:39:48.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.132+0000 7fefae170640 10 monclient: init 2024-03-27T21:39:48.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.133+0000 7fdba80fb640 1 Processor -- start 2024-03-27T21:39:48.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.133+0000 7fdba80fb640 1 -- start start 2024-03-27T21:39:48.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.133+0000 7fdba80fb640 10 monclient: init 2024-03-27T21:39:48.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.133+0000 7fefae170640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:48.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.133+0000 7fefae170640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-03-27T21:39:48.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.133+0000 7fefae170640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fefa8137c50 0x7fefa80b3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.133+0000 7fefae170640 10 monclient(hunting): picked mon.noname-a con 0x7fefa8137c50 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:48.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.133+0000 7fefae170640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fefa80a8b10 0x7fefa80a8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.133+0000 7fefae170640 10 monclient(hunting): picked mon.noname-c con 0x7fefa80a8b10 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:48.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.133+0000 7fefae170640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fefa81417e0 0x7fefa8141c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.133+0000 7fefae170640 10 monclient(hunting): picked mon.noname-b con 0x7fefa81417e0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:48.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.133+0000 7fefae170640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.133+0000 7fefae170640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefa8142190 con 0x7fefa8137c50 2024-03-27T21:39:48.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.133+0000 7fefae170640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.133+0000 7fefae170640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefa81412e0 con 0x7fefa80a8b10 2024-03-27T21:39:48.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.133+0000 7fefae170640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.133+0000 7fefae170640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefa80b3eb0 con 0x7fefa81417e0 2024-03-27T21:39:48.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.133+0000 7fefae170640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:48.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.133+0000 7fefae170640 10 monclient(hunting): authenticate will time out at 829.355628s 2024-03-27T21:39:48.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fefad96f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fefa81417e0 0x7fefa8141c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba80fb640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:48.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba80fb640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-03-27T21:39:48.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba80fb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdb980375b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:48.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba80fb640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdb98034520 0x7fdb980c4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba80fb640 10 monclient(hunting): picked mon.noname-b con 0x7fdb98034520 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:48.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba80fb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdb980c5060 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:48.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba80fb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb9802d620 con 0x7fdb980375b0 2024-03-27T21:39:48.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba80fb640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb9802d900 con 0x7fdb980c5060 2024-03-27T21:39:48.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba80fb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba80fb640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb98037990 con 0x7fdb98034520 2024-03-27T21:39:48.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba80fb640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:48.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba80fb640 10 monclient(hunting): authenticate will time out at 829.356108s 2024-03-27T21:39:48.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fefad96f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fefa81417e0 0x7fefa8141c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47616/0 (socket says 172.21.15.60:47616) 2024-03-27T21:39:48.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fefad96f640 1 -- 172.21.15.60:0/2459201580 learned_addr learned my addr 172.21.15.60:0/2459201580 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:48.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fefad96f640 10 monclient(hunting): get_auth_request con 0x7fefa81417e0 auth_method 0 2024-03-27T21:39:48.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba5696640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdb98034520 0x7fdb980c4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba5696640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdb98034520 0x7fdb980c4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47624/0 (socket says 172.21.15.60:47624) 2024-03-27T21:39:48.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba5696640 1 -- 172.21.15.60:0/3018867299 learned_addr learned my addr 172.21.15.60:0/3018867299 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:48.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba5696640 10 monclient(hunting): get_auth_request con 0x7fdb98034520 auth_method 0 2024-03-27T21:39:48.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba5696640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba5696640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba5696640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fefad96f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fefad96f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fbd91c8b640 1 Processor -- start 2024-03-27T21:39:48.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fb578cc4640 10 monclient: get_monmap_and_config 2024-03-27T21:39:48.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fb578cc4640 10 monclient: build_initial_monmap 2024-03-27T21:39:48.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fb578cc4640 10 monclient: monmap: 2024-03-27T21:39:48.134 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:48.134 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.134 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:48.136775+0000 2024-03-27T21:39:48.134 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:48.136775+0000 2024-03-27T21:39:48.134 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:48.134 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.134 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:48.134 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:48.134 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:48.135 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:48.135 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:48.135 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7f4dfb895640 10 monclient: get_monmap_and_config 2024-03-27T21:39:48.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7f4dfb895640 10 monclient: build_initial_monmap 2024-03-27T21:39:48.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7f4dfb895640 10 monclient: monmap: 2024-03-27T21:39:48.135 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:48.136 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.136 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:48.136700+0000 2024-03-27T21:39:48.136 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:48.136700+0000 2024-03-27T21:39:48.136 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:48.136 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.136 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:48.136 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:48.136 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:48.136 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:48.136 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:48.136 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fefad96f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fefad96f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fefad96f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fefad96f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fefad96f640 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2024-03-27T21:39:48.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fefad96f640 1 -- 172.21.15.60:0/2459201580 >> v2:172.21.15.191:3300/0 conn(0x7fefa80a8b10 msgr2=0x7fefa80a8f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T21:39:48.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fefad96f640 1 --2- 172.21.15.60:0/2459201580 >> v2:172.21.15.191:3300/0 conn(0x7fefa80a8b10 0x7fefa80a8f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fefad96f640 1 -- 172.21.15.60:0/2459201580 >> v2:172.21.15.60:3300/0 conn(0x7fefa8137c50 msgr2=0x7fefa80b3970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T21:39:48.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fefad96f640 1 --2- 172.21.15.60:0/2459201580 >> v2:172.21.15.60:3300/0 conn(0x7fefa8137c50 0x7fefa80b3970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fefad96f640 10 monclient: _finish_hunting 0 2024-03-27T21:39:48.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fefad96f640 1 monclient: found mon.noname-b 2024-03-27T21:39:48.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fefad96f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:48.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fefad96f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:48.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fefad96f640 1 -- 172.21.15.60:0/2459201580 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa8138030 con 0x7fefa81417e0 2024-03-27T21:39:48.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba5696640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba5696640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba5696640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba5696640 10 monclient(hunting): handle_auth_done global_id 4295 payload 274 2024-03-27T21:39:48.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba5696640 1 -- 172.21.15.60:0/3018867299 >> v1:172.21.15.191:6789/0 conn(0x7fdb980c5060 legacy=0x7fdb980cf280 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-27T21:39:48.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba5696640 1 -- 172.21.15.60:0/3018867299 >> v1:172.21.15.60:6789/0 conn(0x7fdb980375b0 legacy=0x7fdb980c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:48.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba5696640 10 monclient: _finish_hunting 0 2024-03-27T21:39:48.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba5696640 1 monclient: found mon.noname-b 2024-03-27T21:39:48.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba5696640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:48.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba5696640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:48.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fdba5696640 1 -- 172.21.15.60:0/3018867299 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9802b600 con 0x7fdb98034520 2024-03-27T21:39:48.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fbd91c8b640 1 -- start start 2024-03-27T21:39:48.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.134+0000 7fbd91c8b640 10 monclient: init 2024-03-27T21:39:48.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.135+0000 7fbd91c8b640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:48.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.135+0000 7fbd91c8b640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-03-27T21:39:48.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.135+0000 7fbd91c8b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbd84038d70 0x7fbd840c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.135+0000 7fbd91c8b640 10 monclient(hunting): picked mon.noname-a con 0x7fbd84038d70 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:48.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.135+0000 7fbd91c8b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbd84039150 0x7fbd840c8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.135+0000 7fbd91c8b640 10 monclient(hunting): picked mon.noname-c con 0x7fbd84039150 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:48.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.135+0000 7fefad96f640 1 --2- 172.21.15.60:0/2459201580 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fefa81417e0 0x7fefa8141c50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fefa80b4620 tx=0x7fefa002be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb58a9e8e6971be5 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.135+0000 7f4dfb895640 1 Processor -- start 2024-03-27T21:39:48.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.135+0000 7fbd91c8b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbd840c5760 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:48.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.135+0000 7fbd91c8b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd840c7e60 con 0x7fbd840c5760 2024-03-27T21:39:48.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.135+0000 7fbd91c8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.135+0000 7fbd91c8b640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd84037600 con 0x7fbd84038d70 2024-03-27T21:39:48.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.135+0000 7fbd91c8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.135+0000 7fbd91c8b640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd84035430 con 0x7fbd84039150 2024-03-27T21:39:48.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.135+0000 7fbd91c8b640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:48.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.135+0000 7fbd91c8b640 10 monclient(hunting): authenticate will time out at 829.357821s 2024-03-27T21:39:48.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fb578cc4640 1 Processor -- start 2024-03-27T21:39:48.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fef9bfff640 1 -- 172.21.15.60:0/2459201580 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa002ca30 con 0x7fefa81417e0 2024-03-27T21:39:48.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fef9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fef9bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:48.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fef9bfff640 10 monclient: dump: 2024-03-27T21:39:48.144 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.144 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.144 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.144 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.144 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.144 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.144 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.144 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.144 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.144 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fef9bfff640 10 monclient: _finish_auth 0 2024-03-27T21:39:48.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fef9bfff640 10 monclient: _check_auth_tickets 2024-03-27T21:39:48.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fef9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:48.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fef9bfff640 1 -- 172.21.15.60:0/2459201580 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa002cb90 con 0x7fefa81417e0 2024-03-27T21:39:48.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fef9bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:48.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fef9bfff640 1 -- 172.21.15.60:0/2459201580 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa00343f0 con 0x7fefa81417e0 2024-03-27T21:39:48.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fef9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fef9bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:48.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fef9bfff640 10 monclient: dump: 2024-03-27T21:39:48.145 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.145 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.145 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.145 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.146 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.146 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.146 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.146 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.146 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.146 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fefae170640 5 monclient: authenticate success, global_id 4292 2024-03-27T21:39:48.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fefae170640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:48.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fefae170640 4 set_mon_vals no callback set 2024-03-27T21:39:48.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.135+0000 7fbd90c89640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbd84038d70 0x7fbd840c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd90c89640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbd84038d70 0x7fbd840c5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47772/0 (socket says 172.21.15.60:47772) 2024-03-27T21:39:48.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd90c89640 1 -- 172.21.15.60:0/1281431702 learned_addr learned my addr 172.21.15.60:0/1281431702 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:48.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd90c89640 10 monclient(hunting): get_auth_request con 0x7fbd84038d70 auth_method 0 2024-03-27T21:39:48.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd90c89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd90c89640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd90c89640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd8bfff640 1 --2- 172.21.15.60:0/1281431702 >> v2:172.21.15.191:3300/0 conn(0x7fbd84039150 0x7fbd840c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd8bfff640 10 monclient(hunting): get_auth_request con 0x7fbd84039150 auth_method 0 2024-03-27T21:39:48.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fb578cc4640 1 -- start start 2024-03-27T21:39:48.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fb578cc4640 10 monclient: init 2024-03-27T21:39:48.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd90c89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd90c89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd90c89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7f4d7d3cb640 10 monclient: get_monmap_and_config 2024-03-27T21:39:48.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7f4d7d3cb640 10 monclient: build_initial_monmap 2024-03-27T21:39:48.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7f4d7d3cb640 10 monclient: monmap: 2024-03-27T21:39:48.149 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:48.149 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.150 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:48.138218+0000 2024-03-27T21:39:48.150 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:48.138218+0000 2024-03-27T21:39:48.150 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:48.150 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.150 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:48.150 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:48.150 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:48.150 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:48.150 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:48.150 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fd3e2c2f640 10 monclient: get_monmap_and_config 2024-03-27T21:39:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fd3e2c2f640 10 monclient: build_initial_monmap 2024-03-27T21:39:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fd3e2c2f640 10 monclient: monmap: 2024-03-27T21:39:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:48.138797+0000 2024-03-27T21:39:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:48.138797+0000 2024-03-27T21:39:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:48.152 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fefae170640 10 monclient: shutdown 2024-03-27T21:39:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fefae170640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fefae170640 1 -- 172.21.15.60:0/2459201580 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fefa81417e0 msgr2=0x7fefa8141c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fefae170640 1 --2- 172.21.15.60:0/2459201580 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fefa81417e0 0x7fefa8141c50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fefa80b4620 tx=0x7fefa002be90 comp rx=0 tx=0).stop 2024-03-27T21:39:48.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd8bfff640 10 monclient(hunting): handle_auth_done global_id 4336 payload 274 2024-03-27T21:39:48.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd8bfff640 1 -- 172.21.15.60:0/1281431702 >> v2:172.21.15.60:3300/0 conn(0x7fbd84038d70 msgr2=0x7fbd840c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:48.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd8bfff640 1 --2- 172.21.15.60:0/1281431702 >> v2:172.21.15.60:3300/0 conn(0x7fbd84038d70 0x7fbd840c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd8bfff640 1 -- 172.21.15.60:0/1281431702 >> v1:172.21.15.191:6789/0 conn(0x7fbd840c5760 legacy=0x7fbd840cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:48.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd8bfff640 10 monclient: _finish_hunting 0 2024-03-27T21:39:48.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd8bfff640 1 monclient: found mon.noname-c 2024-03-27T21:39:48.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:48.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd8bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:39:48.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.136+0000 7fbd8bfff640 1 -- 172.21.15.60:0/1281431702 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd84035860 con 0x7fbd84039150 2024-03-27T21:39:48.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7f4d7d3cb640 1 Processor -- start 2024-03-27T21:39:48.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7f4d7d3cb640 1 -- start start 2024-03-27T21:39:48.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7f4d7d3cb640 10 monclient: init 2024-03-27T21:39:48.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fefae170640 1 -- 172.21.15.60:0/2459201580 shutdown_connections 2024-03-27T21:39:48.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fefae170640 1 --2- 172.21.15.60:0/2459201580 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fefa81417e0 0x7fefa8141c50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fefae170640 1 --2- 172.21.15.60:0/2459201580 >> v2:172.21.15.191:3300/0 conn(0x7fefa80a8b10 0x7fefa80a8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fefae170640 1 --2- 172.21.15.60:0/2459201580 >> v2:172.21.15.60:3300/0 conn(0x7fefa8137c50 0x7fefa80b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fefae170640 1 -- 172.21.15.60:0/2459201580 >> 172.21.15.60:0/2459201580 conn(0x7fefa8060700 msgr2=0x7fefa8060b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fb578cc4640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fb578cc4640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-03-27T21:39:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fb578cc4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb574073640 0x7fb574146030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fb578cc4640 10 monclient(hunting): picked mon.noname-c con 0x7fb574073640 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:48.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7f4dfb895640 1 -- start start 2024-03-27T21:39:48.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7f4dfb895640 10 monclient: init 2024-03-27T21:39:48.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fbd8bfff640 1 --2- 172.21.15.60:0/1281431702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd84039150 0x7fbd840c8310 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbd780097a0 tx=0x7fbd7802b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7af5d85358fd73b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fd3e2c2f640 1 Processor -- start 2024-03-27T21:39:48.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fb578cc4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5740afb20 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:48.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fd3e2c2f640 1 -- start start 2024-03-27T21:39:48.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fd3e2c2f640 10 monclient: init 2024-03-27T21:39:48.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fb57259c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb574073640 0x7fb574146030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fb57259c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb574073640 0x7fb574146030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:56720/0 (socket says 172.21.15.60:56720) 2024-03-27T21:39:48.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fb57259c640 1 -- 172.21.15.60:0/1057087092 learned_addr learned my addr 172.21.15.60:0/1057087092 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:48.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fb571d9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb5740afb20 0x7fb57406da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43570/0 (socket says 172.21.15.60:43570) 2024-03-27T21:39:48.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d7d3cb640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:48.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d7d3cb640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-03-27T21:39:48.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d7d3cb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d7806f770 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fbd8b7fe640 1 -- 172.21.15.60:0/1281431702 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7802f050 con 0x7fbd84039150 2024-03-27T21:39:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d7d3cb640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4d7806fb50 0x7f4d78142b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d7d3cb640 10 monclient(hunting): picked mon.noname-c con 0x7f4d7806fb50 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d7d3cb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4d78143060 0x7f4d78149a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d7d3cb640 10 monclient(hunting): picked mon.noname-a con 0x7f4d78143060 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d7d3cb640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d78142690 con 0x7f4d7806f770 2024-03-27T21:39:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d7d3cb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d7d3cb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d78143510 con 0x7f4d78143060 2024-03-27T21:39:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d7d3cb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d7d3cb640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d780ad6c0 con 0x7f4d7806fb50 2024-03-27T21:39:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d7d3cb640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d7d3cb640 10 monclient(hunting): authenticate will time out at 829.360092s 2024-03-27T21:39:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fb578cc4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5740ade30 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fbd8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fb578cc4640 1 -- 172.21.15.60:0/1057087092 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57406f470 con 0x7fb5740afb20 2024-03-27T21:39:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fbd8b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fbd8b7fe640 10 monclient: dump: 2024-03-27T21:39:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.162 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fbd8b7fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fbd8b7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d777fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4d78143060 0x7f4d78149a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d777fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4d78143060 0x7f4d78149a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47788/0 (socket says 172.21.15.60:47788) 2024-03-27T21:39:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d777fe640 1 -- 172.21.15.60:0/2719490311 learned_addr learned my addr 172.21.15.60:0/2719490311 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.137+0000 7fdba5696640 1 --2- 172.21.15.60:0/3018867299 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdb98034520 0x7fdb980c4b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdba004f930 tx=0x7fdba00a77d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5fbd8f4dad48b9f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fdba4e95640 1 -- 172.21.15.60:0/3018867299 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba00af620 con 0x7fdb98034520 2024-03-27T21:39:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fdba4e95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fdba4e95640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fdba4e95640 10 monclient: dump: 2024-03-27T21:39:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.164 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fdba4e95640 10 monclient: _finish_auth 0 2024-03-27T21:39:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fb578cc4640 1 -- 172.21.15.60:0/1057087092 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57406f6e0 con 0x7fb5740ade30 2024-03-27T21:39:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fb57159a640 1 -- 172.21.15.60:0/1057087092 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2127265493 0 0) 0x7fb57406f470 con 0x7fb5740afb20 2024-03-27T21:39:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fb578cc4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d777fe640 10 monclient(hunting): get_auth_request con 0x7f4d78143060 auth_method 0 2024-03-27T21:39:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d777fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fdba4e95640 10 monclient: _check_auth_tickets 2024-03-27T21:39:48.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fdba4e95640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:48.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fdba4e95640 1 -- 172.21.15.60:0/3018867299 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba00af780 con 0x7fdb98034520 2024-03-27T21:39:48.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fdba80fb640 5 monclient: authenticate success, global_id 4295 2024-03-27T21:39:48.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fdba80fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:48.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fb578cc4640 1 -- 172.21.15.60:0/1057087092 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb574073a20 con 0x7fb574073640 2024-03-27T21:39:48.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fb578cc4640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:48.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d767fc640 1 --2- 172.21.15.60:0/2719490311 >> v2:172.21.15.191:3300/0 conn(0x7f4d7806fb50 0x7f4d78142b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fb578cc4640 10 monclient(hunting): authenticate will time out at 829.360562s 2024-03-27T21:39:48.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fb57259c640 10 monclient(hunting): get_auth_request con 0x7fb574073640 auth_method 0 2024-03-27T21:39:48.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fb57259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fb57259c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fb57259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fb57159a640 10 monclient(hunting): _negotiate 2024-03-27T21:39:48.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fb57159a640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fb57159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fb57159a640 10 monclient(hunting): my global_id is 4320 2024-03-27T21:39:48.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d767fc640 10 monclient(hunting): get_auth_request con 0x7f4d7806fb50 auth_method 0 2024-03-27T21:39:48.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fb57159a640 1 -- 172.21.15.60:0/1057087092 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb55c003520 con 0x7fb5740afb20 2024-03-27T21:39:48.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d767fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fbd8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:48.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fa804251640 10 monclient: get_monmap_and_config 2024-03-27T21:39:48.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fa804251640 10 monclient: build_initial_monmap 2024-03-27T21:39:48.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fa804251640 10 monclient: monmap: 2024-03-27T21:39:48.170 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:48.170 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.170 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:48.140761+0000 2024-03-27T21:39:48.170 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:48.140761+0000 2024-03-27T21:39:48.170 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:48.170 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.170 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:48.170 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:48.170 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:48.170 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:48.170 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:48.170 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fbd91c8b640 5 monclient: authenticate success, global_id 4336 2024-03-27T21:39:48.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fbd91c8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:48.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fbd8b7fe640 1 -- 172.21.15.60:0/1281431702 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd78031700 con 0x7fbd84039150 2024-03-27T21:39:48.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fbd8b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:48.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fbd8b7fe640 1 -- 172.21.15.60:0/1281431702 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd78031a90 con 0x7fbd84039150 2024-03-27T21:39:48.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fbd8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fbd8b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:48.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fbd8b7fe640 10 monclient: dump: 2024-03-27T21:39:48.171 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.172 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.172 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.172 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.172 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.172 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.172 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.172 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.172 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.172 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fb57259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fb57259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fb57259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fefae170640 1 -- 172.21.15.60:0/2459201580 shutdown_connections 2024-03-27T21:39:48.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7f4d777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4d767fc640 10 monclient(hunting): handle_auth_done global_id 4345 payload 274 2024-03-27T21:39:48.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4d767fc640 1 -- 172.21.15.60:0/2719490311 >> v2:172.21.15.60:3300/0 conn(0x7f4d78143060 msgr2=0x7f4d78149a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:48.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fd3e2c2f640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:48.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fd3e2c2f640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-03-27T21:39:48.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fd3e2c2f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd3dc060970 0x7fd3dc0ae150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fd3e2c2f640 10 monclient(hunting): picked mon.noname-c con 0x7fd3dc060970 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:48.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fd3e2c2f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd3dc0af1f0 0x7fd3dc0ad020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fbd91c8b640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:48.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fbd91c8b640 4 set_mon_vals no callback set 2024-03-27T21:39:48.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4d767fc640 1 --2- 172.21.15.60:0/2719490311 >> v2:172.21.15.60:3300/0 conn(0x7f4d78143060 0x7f4d78149a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4d767fc640 1 -- 172.21.15.60:0/2719490311 >> v1:172.21.15.191:6789/0 conn(0x7f4d7806f770 legacy=0x7f4d7813c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4d767fc640 10 monclient: _finish_hunting 0 2024-03-27T21:39:48.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4d767fc640 1 monclient: found mon.noname-c 2024-03-27T21:39:48.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4d767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:48.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4d767fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:39:48.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4d767fc640 1 -- 172.21.15.60:0/2719490311 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d780719c0 con 0x7f4d7806fb50 2024-03-27T21:39:48.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fb57259c640 10 monclient(hunting): handle_auth_done global_id 4342 payload 274 2024-03-27T21:39:48.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fb57259c640 1 -- 172.21.15.60:0/1057087092 >> v1:172.21.15.191:6789/0 conn(0x7fb5740ade30 legacy=0x7fb5741500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fb57259c640 1 -- 172.21.15.60:0/1057087092 >> v1:172.21.15.60:6789/0 conn(0x7fb5740afb20 legacy=0x7fb57406da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fb57259c640 10 monclient: _finish_hunting 0 2024-03-27T21:39:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fb57259c640 1 monclient: found mon.noname-c 2024-03-27T21:39:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fb57259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fb57259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:39:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fb57259c640 1 -- 172.21.15.60:0/1057087092 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5740adaf0 con 0x7fb574073640 2024-03-27T21:39:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4d777fe640 1 --2- 172.21.15.60:0/2719490311 >> v2:172.21.15.60:3300/0 conn(0x7f4d78143060 0x7f4d78149a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fdba4e95640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fdba80fb640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.138+0000 7fdba80fb640 4 set_mon_vals no callback set 2024-03-27T21:39:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fdba80fb640 10 monclient: shutdown 2024-03-27T21:39:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fdba80fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fdba80fb640 1 -- 172.21.15.60:0/3018867299 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdb98034520 msgr2=0x7fdb980c4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fdba80fb640 1 --2- 172.21.15.60:0/3018867299 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdb98034520 0x7fdb980c4b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdba004f930 tx=0x7fdba00a77d0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fbd91c8b640 10 monclient: shutdown 2024-03-27T21:39:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fd3dbfff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd3dc0af1f0 0x7fd3dc0ad020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fd3dbfff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd3dc0af1f0 0x7fd3dc0ad020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47636/0 (socket says 172.21.15.60:47636) 2024-03-27T21:39:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fd3dbfff640 1 -- 172.21.15.60:0/2131985940 learned_addr learned my addr 172.21.15.60:0/2131985940 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fd3e2c2f640 10 monclient(hunting): picked mon.noname-b con 0x7fd3dc0af1f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fd3e2c2f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd3dc0ad560 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fd3e2c2f640 1 -- 172.21.15.60:0/2131985940 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3dc13f080 con 0x7fd3dc0ad560 2024-03-27T21:39:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fd3e2c2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fd3e2c2f640 1 -- 172.21.15.60:0/2131985940 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3dc060d50 con 0x7fd3dc060970 2024-03-27T21:39:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fd3e2c2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fd3e2c2f640 1 -- 172.21.15.60:0/2131985940 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc1414c0 con 0x7fd3dc0af1f0 2024-03-27T21:39:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fa804251640 1 Processor -- start 2024-03-27T21:39:48.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fbd91c8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:48.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fbd91c8b640 1 -- 172.21.15.60:0/1281431702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd84039150 msgr2=0x7fbd840c8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fbd91c8b640 1 --2- 172.21.15.60:0/1281431702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd84039150 0x7fbd840c8310 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbd780097a0 tx=0x7fbd7802b7a0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fd3e2c2f640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:48.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fd3e2c2f640 10 monclient(hunting): authenticate will time out at 829.361884s 2024-03-27T21:39:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4dfb895640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4dfb895640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-03-27T21:39:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4dfb895640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4dec0394c0 0x7f4dec0c5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4dfb895640 10 monclient(hunting): picked mon.noname-a con 0x7f4dec0394c0 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4dfb895640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4dec0398a0 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4dfb895640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4dec035eb0 0x7f4dec0cbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4dfb895640 10 monclient(hunting): picked mon.noname-c con 0x7f4dec035eb0 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4dfb895640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dec0c7d00 con 0x7f4dec0398a0 2024-03-27T21:39:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4dfb895640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4dfb895640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dec036400 con 0x7f4dec0394c0 2024-03-27T21:39:48.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4dfb895640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4dfb895640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dec0c8c40 con 0x7f4dec035eb0 2024-03-27T21:39:48.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4dfb895640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:48.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4dfb895640 10 monclient(hunting): authenticate will time out at 829.361172s 2024-03-27T21:39:48.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4df9631640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4dec0394c0 0x7f4dec0c5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4df9631640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4dec0394c0 0x7f4dec0c5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47804/0 (socket says 172.21.15.60:47804) 2024-03-27T21:39:48.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fbd91c8b640 1 -- 172.21.15.60:0/1281431702 shutdown_connections 2024-03-27T21:39:48.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fbd91c8b640 1 --2- 172.21.15.60:0/1281431702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd84039150 0x7fbd840c8310 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fbd91c8b640 1 --2- 172.21.15.60:0/1281431702 >> v2:172.21.15.60:3300/0 conn(0x7fbd84038d70 0x7fbd840c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fbd91c8b640 1 -- 172.21.15.60:0/1281431702 >> 172.21.15.60:0/1281431702 conn(0x7fbd840065d0 msgr2=0x7fbd840069e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4df9631640 1 -- 172.21.15.60:0/2815889185 learned_addr learned my addr 172.21.15.60:0/2815889185 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4df9631640 10 monclient(hunting): get_auth_request con 0x7f4dec0394c0 auth_method 0 2024-03-27T21:39:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4df9631640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4df9631640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7f4df9631640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fdba80fb640 1 -- 172.21.15.60:0/3018867299 shutdown_connections 2024-03-27T21:39:48.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fdba80fb640 1 --2- 172.21.15.60:0/3018867299 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdb98034520 0x7fdb980c4b20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fdba80fb640 1 -- 172.21.15.60:0/3018867299 >> 172.21.15.60:0/3018867299 conn(0x7fdb9803a920 msgr2=0x7fdb98036bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:48.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fa804251640 1 -- start start 2024-03-27T21:39:48.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fa804251640 10 monclient: init 2024-03-27T21:39:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fd3dbfff640 10 monclient(hunting): get_auth_request con 0x7fd3dc0af1f0 auth_method 0 2024-03-27T21:39:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fd3dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fd3dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fd3dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fd3e09cb640 1 --2- 172.21.15.60:0/2131985940 >> v2:172.21.15.191:3300/0 conn(0x7fd3dc060970 0x7fd3dc0ae150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fd3e09cb640 10 monclient(hunting): get_auth_request con 0x7fd3dc060970 auth_method 0 2024-03-27T21:39:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fd3e09cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fd3e09cb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fd3e09cb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.139+0000 7fefae170640 1 -- 172.21.15.60:0/2459201580 wait complete. 2024-03-27T21:39:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fefae170640 10 monclient: build_initial_monmap 2024-03-27T21:39:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fefae170640 10 monclient: monmap: 2024-03-27T21:39:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:48.187 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fd3dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fefae170640 1 librados: starting msgr at 2024-03-27T21:39:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fefae170640 1 librados: starting objecter 2024-03-27T21:39:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fb57259c640 1 --2- 172.21.15.60:0/1057087092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb574073640 0x7fb574146030 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb564009e50 tx=0x7fb56402bcd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de573178e388ac88 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7f4df9631640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7f4df9631640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7f4df9631640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fbd91c8b640 1 -- 172.21.15.60:0/1281431702 shutdown_connections 2024-03-27T21:39:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7f4df9e32640 1 --2- 172.21.15.60:0/2815889185 >> v2:172.21.15.191:3300/0 conn(0x7f4dec035eb0 0x7f4dec0cbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7f4df9e32640 10 monclient(hunting): get_auth_request con 0x7f4dec035eb0 auth_method 0 2024-03-27T21:39:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7f4df9e32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7f4df9e32640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7f4df9e32640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fefae170640 1 Processor -- start 2024-03-27T21:39:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7f4d767fc640 1 --2- 172.21.15.60:0/2719490311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d7806fb50 0x7f4d78142b20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4d780700a0 tx=0x7f4d6c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70f8f3e68cc73041 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fa804251640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fa804251640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-03-27T21:39:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fb57159a640 1 -- 172.21.15.60:0/1057087092 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb568001670 con 0x7fb574073640 2024-03-27T21:39:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fb57159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fb57159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fb57159a640 10 monclient: dump: 2024-03-27T21:39:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.191 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fb57159a640 10 monclient: _finish_auth 0 2024-03-27T21:39:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fb57159a640 10 monclient: _check_auth_tickets 2024-03-27T21:39:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fb57159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fb57159a640 1 -- 172.21.15.60:0/1057087092 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56402c5f0 con 0x7fb574073640 2024-03-27T21:39:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fb57159a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fb57159a640 1 -- 172.21.15.60:0/1057087092 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56402c980 con 0x7fb574073640 2024-03-27T21:39:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fb57159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fb57159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fb57159a640 10 monclient: dump: 2024-03-27T21:39:48.192 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.192 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.192 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.192 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.192 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.192 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.192 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.192 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.192 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.192 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fb578cc4640 5 monclient: authenticate success, global_id 4342 2024-03-27T21:39:48.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fb578cc4640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:48.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fb578cc4640 4 set_mon_vals no callback set 2024-03-27T21:39:48.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fbd91c8b640 1 -- 172.21.15.60:0/1281431702 wait complete. 2024-03-27T21:39:48.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fa804251640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7fc071a50 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:48.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fa804251640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa7fc13ef20 0x7fa7fc0a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fa804251640 10 monclient(hunting): picked mon.noname-a con 0x7fa7fc13ef20 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:48.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fa804251640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7fc141630 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:48.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fa804251640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7fc0a8e90 con 0x7fa7fc071a50 2024-03-27T21:39:48.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fa804251640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7fc0aa040 con 0x7fa7fc141630 2024-03-27T21:39:48.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fa804251640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fa804251640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7fc071e30 con 0x7fa7fc13ef20 2024-03-27T21:39:48.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fa804251640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:48.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fa804251640 10 monclient(hunting): authenticate will time out at 829.362896s 2024-03-27T21:39:48.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7f4df9e32640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7f4df9e32640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7f4df9e32640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7f4df9631640 10 monclient(hunting): handle_auth_done global_id 4326 payload 274 2024-03-27T21:39:48.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7f4df9631640 1 -- 172.21.15.60:0/2815889185 >> v2:172.21.15.191:3300/0 conn(0x7f4dec035eb0 msgr2=0x7f4dec0cbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:48.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7f4df9631640 1 --2- 172.21.15.60:0/2815889185 >> v2:172.21.15.191:3300/0 conn(0x7f4dec035eb0 0x7f4dec0cbcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7f4df9631640 1 -- 172.21.15.60:0/2815889185 >> v1:172.21.15.60:6789/0 conn(0x7f4dec0398a0 legacy=0x7f4dec0cb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7f4df9631640 10 monclient: _finish_hunting 0 2024-03-27T21:39:48.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7f4df9631640 1 monclient: found mon.noname-a 2024-03-27T21:39:48.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7f4df9631640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:48.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7f4df9631640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:48.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7f4df9631640 1 -- 172.21.15.60:0/2815889185 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dec0374d0 con 0x7f4dec0394c0 2024-03-27T21:39:48.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fd3dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fd3dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fd3e09cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fd3e09cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fd3e09cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fd3e09cb640 10 monclient(hunting): handle_auth_done global_id 4351 payload 274 2024-03-27T21:39:48.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fd3e09cb640 1 -- 172.21.15.60:0/2131985940 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd3dc0af1f0 msgr2=0x7fd3dc0ad020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:48.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fd3e09cb640 1 --2- 172.21.15.60:0/2131985940 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd3dc0af1f0 0x7fd3dc0ad020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fd3e09cb640 1 -- 172.21.15.60:0/2131985940 >> v1:172.21.15.60:6789/0 conn(0x7fd3dc0ad560 legacy=0x7fd3dc141950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fd3e09cb640 10 monclient: _finish_hunting 0 2024-03-27T21:39:48.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fd3e09cb640 1 monclient: found mon.noname-c 2024-03-27T21:39:48.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fd3e09cb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:48.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fd3e09cb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:39:48.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fd3e09cb640 1 -- 172.21.15.60:0/2131985940 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3dc0b02c0 con 0x7fd3dc060970 2024-03-27T21:39:48.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fd3dbfff640 1 --2- 172.21.15.60:0/2131985940 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd3dc0af1f0 0x7fd3dc0ad020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:48.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fa8017ec640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa7fc13ef20 0x7fa7fc0a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fa8017ec640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa7fc13ef20 0x7fa7fc0a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47808/0 (socket says 172.21.15.60:47808) 2024-03-27T21:39:48.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fa8017ec640 1 -- 172.21.15.60:0/3301199706 learned_addr learned my addr 172.21.15.60:0/3301199706 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:48.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fa8017ec640 10 monclient(hunting): get_auth_request con 0x7fa7fc13ef20 auth_method 0 2024-03-27T21:39:48.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fa8017ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fa8017ec640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fa8017ec640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fefae170640 1 -- start start 2024-03-27T21:39:48.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fb578cc4640 10 monclient: shutdown 2024-03-27T21:39:48.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fb578cc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:48.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fefae170640 1 librados: setting wanted keys 2024-03-27T21:39:48.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fefae170640 1 librados: calling monclient init 2024-03-27T21:39:48.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fefae170640 10 monclient: init 2024-03-27T21:39:48.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fefae170640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:48.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fb578cc4640 1 -- 172.21.15.60:0/1057087092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb574073640 msgr2=0x7fb574146030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fb578cc4640 1 --2- 172.21.15.60:0/1057087092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb574073640 0x7fb574146030 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb564009e50 tx=0x7fb56402bcd0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fefae170640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:39:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fefae170640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefa80a8b10 0x7fefa813a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fefae170640 10 monclient(hunting): picked mon.noname-b con 0x7fefa80a8b10 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fefae170640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa8137c50 0x7fefa8151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fefae170640 10 monclient(hunting): picked mon.noname-a con 0x7fefa8137c50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fefae170640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fefa81417e0 0x7fefa81524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fefae170640 10 monclient(hunting): picked mon.noname-c con 0x7fefa81417e0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fefae170640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fefae170640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa813e5e0 con 0x7fefa8137c50 2024-03-27T21:39:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fefae170640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fefae170640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa8152a20 con 0x7fefa80a8b10 2024-03-27T21:39:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fefae170640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fefae170640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefa806bf90 con 0x7fefa81417e0 2024-03-27T21:39:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fefae170640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fefae170640 10 monclient(hunting): authenticate will time out at 829.363679s 2024-03-27T21:39:48.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fb578cc4640 1 -- 172.21.15.60:0/1057087092 shutdown_connections 2024-03-27T21:39:48.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefac96d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa8137c50 0x7fefa8151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefac96d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa8137c50 0x7fefa8151fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47816/0 (socket says 172.21.15.60:47816) 2024-03-27T21:39:48.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefac96d640 1 -- 172.21.15.60:0/4003671765 learned_addr learned my addr 172.21.15.60:0/4003671765 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:48.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fbd91c8b640 10 monclient: build_initial_monmap 2024-03-27T21:39:48.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fbd91c8b640 10 monclient: monmap: 2024-03-27T21:39:48.202 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:48.202 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:48.202 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:48.202 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:48.202 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:48.202 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:48.202 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:48.202 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:48.202 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:48.202 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fbd91c8b640 1 librados: starting msgr at 2024-03-27T21:39:48.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fbd91c8b640 1 librados: starting objecter 2024-03-27T21:39:48.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fb578cc4640 1 --2- 172.21.15.60:0/1057087092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb574073640 0x7fb574146030 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefac96d640 10 monclient(hunting): get_auth_request con 0x7fefa8137c50 auth_method 0 2024-03-27T21:39:48.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefac96d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefac96d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefac96d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fa8017ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fa8017ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fa8017ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fb578cc4640 1 -- 172.21.15.60:0/1057087092 >> 172.21.15.60:0/1057087092 conn(0x7fb5740b3de0 msgr2=0x7fb574069ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:48.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fbd91c8b640 1 Processor -- start 2024-03-27T21:39:48.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fb578cc4640 1 -- 172.21.15.60:0/1057087092 shutdown_connections 2024-03-27T21:39:48.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fb578cc4640 1 -- 172.21.15.60:0/1057087092 wait complete. 2024-03-27T21:39:48.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefad16e640 1 --2- 172.21.15.60:0/4003671765 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefa80a8b10 0x7fefa813a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fbd91c8b640 1 -- start start 2024-03-27T21:39:48.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fbd91c8b640 1 librados: setting wanted keys 2024-03-27T21:39:48.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fbd91c8b640 1 librados: calling monclient init 2024-03-27T21:39:48.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fbd91c8b640 10 monclient: init 2024-03-27T21:39:48.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefad16e640 10 monclient(hunting): get_auth_request con 0x7fefa80a8b10 auth_method 0 2024-03-27T21:39:48.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefad16e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefad16e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefad16e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7f4df9631640 1 --2- 172.21.15.60:0/2815889185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dec0394c0 0x7f4dec0c5260 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4dec03e4b0 tx=0x7f4df002b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7f0b3ff37b344c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefac96d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefac96d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefac96d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fdba80fb640 1 -- 172.21.15.60:0/3018867299 shutdown_connections 2024-03-27T21:39:48.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.140+0000 7fdba80fb640 1 -- 172.21.15.60:0/3018867299 wait complete. 2024-03-27T21:39:48.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fdba80fb640 10 monclient: build_initial_monmap 2024-03-27T21:39:48.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.141+0000 7fdba80fb640 10 monclient: monmap: 2024-03-27T21:39:48.208 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:48.208 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:48.208 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:48.208 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:48.208 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:48.208 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:48.208 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:48.208 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:48.208 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:48.208 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fdba80fb640 1 librados: starting msgr at 2024-03-27T21:39:48.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fdba80fb640 1 librados: starting objecter 2024-03-27T21:39:48.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fdba80fb640 1 Processor -- start 2024-03-27T21:39:48.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fa800feb640 1 -- 172.21.15.60:0/3301199706 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276921689 0 0) 0x7fa7fc0aa040 con 0x7fa7fc141630 2024-03-27T21:39:48.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fa800feb640 10 monclient(hunting): _negotiate 2024-03-27T21:39:48.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fa800feb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fa800feb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fa800feb640 10 monclient(hunting): my global_id is 4357 2024-03-27T21:39:48.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fa800feb640 1 -- 172.21.15.60:0/3301199706 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7e8003520 con 0x7fa7fc141630 2024-03-27T21:39:48.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fa8017ec640 10 monclient(hunting): handle_auth_done global_id 4335 payload 274 2024-03-27T21:39:48.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fa8017ec640 1 -- 172.21.15.60:0/3301199706 >> v1:172.21.15.191:6789/0 conn(0x7fa7fc141630 legacy=0x7fa7fc141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fa8017ec640 1 -- 172.21.15.60:0/3301199706 >> v1:172.21.15.60:6789/0 conn(0x7fa7fc071a50 legacy=0x7fa7fc13e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fa8017ec640 10 monclient: _finish_hunting 0 2024-03-27T21:39:48.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fa8017ec640 1 monclient: found mon.noname-a 2024-03-27T21:39:48.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fa8017ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:48.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fa8017ec640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:48.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fa8017ec640 1 -- 172.21.15.60:0/3301199706 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7fc0aa2b0 con 0x7fa7fc13ef20 2024-03-27T21:39:48.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefad16e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefad16e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefad16e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fd3e09cb640 1 --2- 172.21.15.60:0/2131985940 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3dc060970 0x7fd3dc0ae150 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd3d00097a0 tx=0x7fd3d002b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39238deb35e7467c server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fdba80fb640 1 -- start start 2024-03-27T21:39:48.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fdba80fb640 1 librados: setting wanted keys 2024-03-27T21:39:48.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fdba80fb640 1 librados: calling monclient init 2024-03-27T21:39:48.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fdba80fb640 10 monclient: init 2024-03-27T21:39:48.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fd3db7fe640 1 -- 172.21.15.60:0/2131985940 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0030710 con 0x7fd3dc060970 2024-03-27T21:39:48.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fd3db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fd3db7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:48.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fd3db7fe640 10 monclient: dump: 2024-03-27T21:39:48.213 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.213 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.213 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.213 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.213 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.213 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.213 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.214 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.214 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.214 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fd3db7fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:48.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fd3db7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:48.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fd3db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:48.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fd3db7fe640 1 -- 172.21.15.60:0/2131985940 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d002e050 con 0x7fd3dc060970 2024-03-27T21:39:48.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fd3db7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:48.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fd3db7fe640 1 -- 172.21.15.60:0/2131985940 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0030a90 con 0x7fd3dc060970 2024-03-27T21:39:48.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fd3db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fd3db7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:48.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fd3db7fe640 10 monclient: dump: 2024-03-27T21:39:48.214 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.215 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.215 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.215 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.215 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.215 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.215 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.215 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.215 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.215 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fd3e2c2f640 5 monclient: authenticate success, global_id 4351 2024-03-27T21:39:48.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefad16e640 10 monclient(hunting): handle_auth_done global_id 4360 payload 824 2024-03-27T21:39:48.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefad16e640 1 -- 172.21.15.60:0/4003671765 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fefa81417e0 msgr2=0x7fefa81524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefad16e640 1 --2- 172.21.15.60:0/4003671765 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fefa81417e0 0x7fefa81524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fd3e2c2f640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:48.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fb578cc4640 10 monclient: build_initial_monmap 2024-03-27T21:39:48.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefad16e640 1 -- 172.21.15.60:0/4003671765 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa8137c50 msgr2=0x7fefa8151fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefad16e640 1 --2- 172.21.15.60:0/4003671765 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa8137c50 0x7fefa8151fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefad16e640 10 monclient: _finish_hunting 0 2024-03-27T21:39:48.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefad16e640 1 monclient: found mon.noname-b 2024-03-27T21:39:48.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefad16e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:48.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefad16e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:39:48.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7fefad16e640 1 -- 172.21.15.60:0/4003671765 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa0009b30 con 0x7fefa80a8b10 2024-03-27T21:39:48.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fbd91c8b640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:48.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fd3e2c2f640 4 set_mon_vals no callback set 2024-03-27T21:39:48.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fb578cc4640 10 monclient: monmap: 2024-03-27T21:39:48.218 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:48.218 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:48.218 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:48.218 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:48.218 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:48.218 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:48.218 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:48.218 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:48.218 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:48.219 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fbd91c8b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:39:48.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fbd91c8b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd84038d70 0x7fbd840cb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7f4d75ffb640 1 -- 172.21.15.60:0/2719490311 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c02e070 con 0x7f4d7806fb50 2024-03-27T21:39:48.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba80fb640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:48.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba80fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:39:48.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fbd91c8b640 10 monclient(hunting): picked mon.noname-a con 0x7fbd84038d70 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:48.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba80fb640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdb980375b0 0x7fdb980c86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fbd91c8b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd840c5760 0x7fbd8400a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba80fb640 10 monclient(hunting): picked mon.noname-c con 0x7fdb980375b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:48.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba80fb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb980c5060 0x7fdb98161780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba80fb640 10 monclient(hunting): picked mon.noname-a con 0x7fdb980c5060 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:48.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba80fb640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb9802dc80 0x7fdb981645c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba80fb640 10 monclient(hunting): picked mon.noname-b con 0x7fdb9802dc80 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:48.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba80fb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba80fb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9802f560 con 0x7fdb980c5060 2024-03-27T21:39:48.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba80fb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba80fb640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9802f840 con 0x7fdb9802dc80 2024-03-27T21:39:48.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba80fb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba80fb640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb980c78b0 con 0x7fdb980375b0 2024-03-27T21:39:48.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba80fb640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:48.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba80fb640 10 monclient(hunting): authenticate will time out at 829.365654s 2024-03-27T21:39:48.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fbd91c8b640 10 monclient(hunting): picked mon.noname-b con 0x7fbd840c5760 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:48.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fbd90c89640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd84038d70 0x7fbd840cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fbd90c89640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd84038d70 0x7fbd840cb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47828/0 (socket says 172.21.15.60:47828) 2024-03-27T21:39:48.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba5696640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb980c5060 0x7fdb98161780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba5696640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb980c5060 0x7fdb98161780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47842/0 (socket says 172.21.15.60:47842) 2024-03-27T21:39:48.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba5696640 1 -- 172.21.15.60:0/75719323 learned_addr learned my addr 172.21.15.60:0/75719323 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:48.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba5696640 10 monclient(hunting): get_auth_request con 0x7fdb980c5060 auth_method 0 2024-03-27T21:39:48.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba5696640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba5696640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba5696640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fb578cc4640 1 librados: starting msgr at 2024-03-27T21:39:48.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fb578cc4640 1 librados: starting objecter 2024-03-27T21:39:48.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fb578cc4640 1 Processor -- start 2024-03-27T21:39:48.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba5696640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fdba5696640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba6698640 1 --2- 172.21.15.60:0/75719323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb9802dc80 0x7fdb981645c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba5696640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba6698640 10 monclient(hunting): get_auth_request con 0x7fdb9802dc80 auth_method 0 2024-03-27T21:39:48.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba6698640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba6698640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba6698640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd90c89640 1 -- 172.21.15.60:0/588725866 learned_addr learned my addr 172.21.15.60:0/588725866 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:48.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd8bfff640 1 --2- 172.21.15.60:0/588725866 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd840c5760 0x7fbd8400a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7f4d75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7f4d75ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:48.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7f4d75ffb640 10 monclient: dump: 2024-03-27T21:39:48.226 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.226 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.226 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.226 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.226 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.226 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.226 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.226 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.226 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.226 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7f4d75ffb640 10 monclient: _finish_auth 0 2024-03-27T21:39:48.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7f4d75ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:39:48.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7f4d75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:48.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7f4d75ffb640 1 -- 172.21.15.60:0/2719490311 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d6c0315a0 con 0x7f4d7806fb50 2024-03-27T21:39:48.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7f4d75ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:48.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7f4d7d3cb640 5 monclient: authenticate success, global_id 4345 2024-03-27T21:39:48.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7f4d75ffb640 1 -- 172.21.15.60:0/2719490311 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c02e070 con 0x7f4d7806fb50 2024-03-27T21:39:48.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7f4d7d3cb640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:48.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7f4d7d3cb640 4 set_mon_vals no callback set 2024-03-27T21:39:48.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7f4d75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7f4d75ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:48.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.142+0000 7f4d75ffb640 10 monclient: dump: 2024-03-27T21:39:48.227 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.227 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.227 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.227 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.228 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.228 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.228 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.228 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.228 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.228 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7f4d7d3cb640 10 monclient: shutdown 2024-03-27T21:39:48.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba5e97640 1 --2- 172.21.15.60:0/75719323 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdb980375b0 0x7fdb980c86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd91c8b640 1 --2- 172.21.15.60:0/588725866 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbd84161300 0x7fbd841617b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7f4d7d3cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:48.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7f4d7d3cb640 1 -- 172.21.15.60:0/2719490311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d7806fb50 msgr2=0x7f4d78142b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7f4d7d3cb640 1 --2- 172.21.15.60:0/2719490311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d7806fb50 0x7f4d78142b20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4d780700a0 tx=0x7f4d6c02b840 comp rx=0 tx=0).stop 2024-03-27T21:39:48.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4d7d3cb640 1 -- 172.21.15.60:0/2719490311 shutdown_connections 2024-03-27T21:39:48.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4d7d3cb640 1 --2- 172.21.15.60:0/2719490311 >> v2:172.21.15.60:3300/0 conn(0x7f4d78143060 0x7f4d78149a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4d7d3cb640 1 --2- 172.21.15.60:0/2719490311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d7806fb50 0x7f4d78142b20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4d7d3cb640 1 -- 172.21.15.60:0/2719490311 >> 172.21.15.60:0/2719490311 conn(0x7f4d780b2ea0 msgr2=0x7f4d780b32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:48.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd91c8b640 10 monclient(hunting): picked mon.noname-c con 0x7fbd84161300 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:48.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd9148a640 1 --2- 172.21.15.60:0/588725866 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbd84161300 0x7fbd841617b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba6698640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba6698640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba6698640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba5e97640 10 monclient(hunting): get_auth_request con 0x7fdb980375b0 auth_method 0 2024-03-27T21:39:48.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba5e97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba5e97640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba5e97640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fd3e2c2f640 10 monclient: shutdown 2024-03-27T21:39:48.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4debfff640 1 -- 172.21.15.60:0/2815889185 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df0030890 con 0x7f4dec0394c0 2024-03-27T21:39:48.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4debfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4debfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:48.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4debfff640 10 monclient: dump: 2024-03-27T21:39:48.232 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.232 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.232 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.232 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.232 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.232 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.232 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.232 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.232 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.232 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4debfff640 10 monclient: _finish_auth 0 2024-03-27T21:39:48.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4debfff640 10 monclient: _check_auth_tickets 2024-03-27T21:39:48.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4debfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:48.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4debfff640 1 -- 172.21.15.60:0/2815889185 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df00309f0 con 0x7f4dec0394c0 2024-03-27T21:39:48.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4debfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:48.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fd3e2c2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:48.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4debfff640 1 -- 172.21.15.60:0/2815889185 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df0030890 con 0x7f4dec0394c0 2024-03-27T21:39:48.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4debfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4debfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:48.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4debfff640 10 monclient: dump: 2024-03-27T21:39:48.233 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.233 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.234 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.234 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.234 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.234 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.234 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.234 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.234 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.234 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba5696640 10 monclient(hunting): handle_auth_done global_id 4344 payload 824 2024-03-27T21:39:48.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fd3e2c2f640 1 -- 172.21.15.60:0/2131985940 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3dc060970 msgr2=0x7fd3dc0ae150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fd3e2c2f640 1 --2- 172.21.15.60:0/2131985940 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3dc060970 0x7fd3dc0ae150 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd3d00097a0 tx=0x7fd3d002b710 comp rx=0 tx=0).stop 2024-03-27T21:39:48.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fefad16e640 1 --2- 172.21.15.60:0/4003671765 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefa80a8b10 0x7fefa813a520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fef9c009fd0 tx=0x7fef9c00b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fef99ffb640 10 client.?.objecter ms_handle_connect 0x7fefa80a8b10 2024-03-27T21:39:48.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fef99ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:48.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.143+0000 7fef99ffb640 1 -- 172.21.15.60:0/4003671765 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9c011830 con 0x7fefa80a8b10 2024-03-27T21:39:48.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fef99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fef99ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:48.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fef99ffb640 10 monclient: dump: 2024-03-27T21:39:48.236 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.236 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.236 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.236 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.236 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.236 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.236 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.236 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.236 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.236 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fef99ffb640 10 monclient: _finish_auth 0 2024-03-27T21:39:48.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fef99ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:39:48.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fef99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:48.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fef99ffb640 1 -- 172.21.15.60:0/4003671765 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef9c011990 con 0x7fefa80a8b10 2024-03-27T21:39:48.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fef99ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:48.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fef9b7fe640 4 set_mon_vals no callback set 2024-03-27T21:39:48.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4dfb895640 5 monclient: authenticate success, global_id 4326 2024-03-27T21:39:48.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4dfb895640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:48.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4dfb895640 4 set_mon_vals no callback set 2024-03-27T21:39:48.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fd3e2c2f640 1 -- 172.21.15.60:0/2131985940 shutdown_connections 2024-03-27T21:39:48.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd91c8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fd3e2c2f640 1 --2- 172.21.15.60:0/2131985940 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd3dc0af1f0 0x7fd3dc0ad020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd91c8b640 1 -- 172.21.15.60:0/588725866 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8400a760 con 0x7fbd84038d70 2024-03-27T21:39:48.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd91c8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fd3e2c2f640 1 --2- 172.21.15.60:0/2131985940 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3dc060970 0x7fd3dc0ae150 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd91c8b640 1 -- 172.21.15.60:0/588725866 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8400b6d0 con 0x7fbd840c5760 2024-03-27T21:39:48.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fd3e2c2f640 1 -- 172.21.15.60:0/2131985940 >> 172.21.15.60:0/2131985940 conn(0x7fd3dc069ce0 msgr2=0x7fd3dc06a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:48.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd91c8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd91c8b640 1 -- 172.21.15.60:0/588725866 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd84160e70 con 0x7fbd84161300 2024-03-27T21:39:48.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd91c8b640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:48.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd91c8b640 10 monclient(hunting): authenticate will time out at 829.366606s 2024-03-27T21:39:48.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fb578cc4640 1 -- start start 2024-03-27T21:39:48.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fb578cc4640 1 librados: setting wanted keys 2024-03-27T21:39:48.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fb578cc4640 1 librados: calling monclient init 2024-03-27T21:39:48.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fb578cc4640 10 monclient: init 2024-03-27T21:39:48.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fb578cc4640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:48.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fb578cc4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:39:48.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fb578cc4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb574073640 0x7fb57413da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fb578cc4640 10 monclient(hunting): picked mon.noname-a con 0x7fb574073640 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:48.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fb578cc4640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb5740ade30 0x7fb5741ce960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fb578cc4640 10 monclient(hunting): picked mon.noname-c con 0x7fb5740ade30 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:48.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fb578cc4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5740afb20 0x7fb57413cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fb578cc4640 10 monclient(hunting): picked mon.noname-b con 0x7fb5740afb20 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:48.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fb578cc4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fb578cc4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5741ceea0 con 0x7fb574073640 2024-03-27T21:39:48.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fb578cc4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fb578cc4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5741cf160 con 0x7fb5740afb20 2024-03-27T21:39:48.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fb578cc4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fb578cc4640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5741ce1d0 con 0x7fb5740ade30 2024-03-27T21:39:48.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fb578cc4640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:48.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fb578cc4640 10 monclient(hunting): authenticate will time out at 829.366896s 2024-03-27T21:39:48.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fef99ffb640 1 -- 172.21.15.60:0/4003671765 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9c011d40 con 0x7fefa80a8b10 2024-03-27T21:39:48.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fef99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fef99ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:48.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fef99ffb640 10 monclient: dump: 2024-03-27T21:39:48.242 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.242 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.242 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.242 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.242 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.242 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.243 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.243 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.243 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.243 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4d7d3cb640 1 -- 172.21.15.60:0/2719490311 shutdown_connections 2024-03-27T21:39:48.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba5696640 1 -- 172.21.15.60:0/75719323 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdb980375b0 msgr2=0x7fdb980c86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba5696640 1 --2- 172.21.15.60:0/75719323 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdb980375b0 0x7fdb980c86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba5696640 1 -- 172.21.15.60:0/75719323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb9802dc80 msgr2=0x7fdb981645c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba5696640 1 --2- 172.21.15.60:0/75719323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb9802dc80 0x7fdb981645c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba5696640 10 monclient: _finish_hunting 0 2024-03-27T21:39:48.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba5696640 1 monclient: found mon.noname-a 2024-03-27T21:39:48.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba5696640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:48.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba5696640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:48.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba5696640 1 -- 172.21.15.60:0/75719323 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba006be70 con 0x7fdb980c5060 2024-03-27T21:39:48.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fdba5e97640 1 --2- 172.21.15.60:0/75719323 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdb980375b0 0x7fdb980c86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:39:48.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd90c89640 10 monclient(hunting): get_auth_request con 0x7fbd84038d70 auth_method 0 2024-03-27T21:39:48.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd90c89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd90c89640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd90c89640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd8bfff640 10 monclient(hunting): get_auth_request con 0x7fbd840c5760 auth_method 0 2024-03-27T21:39:48.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fd3e2c2f640 1 -- 172.21.15.60:0/2131985940 shutdown_connections 2024-03-27T21:39:48.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fd3e2c2f640 1 -- 172.21.15.60:0/2131985940 wait complete. 2024-03-27T21:39:48.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fa8017ec640 1 --2- 172.21.15.60:0/3301199706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7fc13ef20 0x7fa7fc0a87e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4009e50 tx=0x7fa7f402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=158c1f366e7a4102 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb571d9b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb5740ade30 0x7fb5741ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb571d9b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb5740ade30 0x7fb5741ce960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47676/0 (socket says 172.21.15.60:47676) 2024-03-27T21:39:48.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb572d9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5740afb20 0x7fb57413cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb571d9b640 1 -- 172.21.15.60:0/3016765488 learned_addr learned my addr 172.21.15.60:0/3016765488 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:48.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb571d9b640 10 monclient(hunting): get_auth_request con 0x7fb5740ade30 auth_method 0 2024-03-27T21:39:48.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb571d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb571d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb571d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb572d9d640 10 monclient(hunting): get_auth_request con 0x7fb5740afb20 auth_method 0 2024-03-27T21:39:48.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb572d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb572d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb572d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb571d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb571d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb571d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb572d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb572d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb572d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb57259c640 1 --2- 172.21.15.60:0/3016765488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb574073640 0x7fb57413da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb57259c640 10 monclient(hunting): get_auth_request con 0x7fb574073640 auth_method 0 2024-03-27T21:39:48.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb57259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb57259c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb57259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb57259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb57259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb57259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb572d9d640 10 monclient(hunting): handle_auth_done global_id 4366 payload 824 2024-03-27T21:39:48.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb572d9d640 1 -- 172.21.15.60:0/3016765488 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb5740ade30 msgr2=0x7fb5741ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb572d9d640 1 --2- 172.21.15.60:0/3016765488 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb5740ade30 0x7fb5741ce960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb572d9d640 1 -- 172.21.15.60:0/3016765488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb574073640 msgr2=0x7fb57413da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb572d9d640 1 --2- 172.21.15.60:0/3016765488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb574073640 0x7fb57413da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb572d9d640 10 monclient: _finish_hunting 0 2024-03-27T21:39:48.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb572d9d640 1 monclient: found mon.noname-b 2024-03-27T21:39:48.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb572d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:48.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb572d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:39:48.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb572d9d640 1 -- 172.21.15.60:0/3016765488 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb564009b30 con 0x7fb5740afb20 2024-03-27T21:39:48.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb572d9d640 1 --2- 172.21.15.60:0/3016765488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5740afb20 0x7fb57413cd00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb57406eb30 tx=0x7fb56c00c7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb562ffd640 10 client.?.objecter ms_handle_connect 0x7fb5740afb20 2024-03-27T21:39:48.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fb562ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:48.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fb562ffd640 1 -- 172.21.15.60:0/3016765488 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c0052d0 con 0x7fb5740afb20 2024-03-27T21:39:48.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fb562ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fb562ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:48.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fb562ffd640 10 monclient: dump: 2024-03-27T21:39:48.249 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.250 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.250 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.250 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.250 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.250 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.250 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.250 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.250 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.250 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fb562ffd640 10 monclient: _finish_auth 0 2024-03-27T21:39:48.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fb562ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:39:48.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fb562ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:48.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fb562ffd640 1 -- 172.21.15.60:0/3016765488 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56c005430 con 0x7fb5740afb20 2024-03-27T21:39:48.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7f4dfb895640 10 monclient: shutdown 2024-03-27T21:39:48.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7f4dfb895640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:48.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7f4dfb895640 1 -- 172.21.15.60:0/2815889185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dec0394c0 msgr2=0x7f4dec0c5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7f4dfb895640 1 --2- 172.21.15.60:0/2815889185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dec0394c0 0x7f4dec0c5260 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4dec03e4b0 tx=0x7f4df002b9d0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7f4dfb895640 1 -- 172.21.15.60:0/2815889185 shutdown_connections 2024-03-27T21:39:48.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7f4dfb895640 1 --2- 172.21.15.60:0/2815889185 >> v2:172.21.15.191:3300/0 conn(0x7f4dec035eb0 0x7f4dec0cbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7f4dfb895640 1 --2- 172.21.15.60:0/2815889185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dec0394c0 0x7f4dec0c5260 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4dec03e4b0 tx=0x7f4df002b9d0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7f4dfb895640 1 -- 172.21.15.60:0/2815889185 >> 172.21.15.60:0/2815889185 conn(0x7f4dec0070f0 msgr2=0x7f4dec037da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:48.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7f4dfb895640 1 -- 172.21.15.60:0/2815889185 shutdown_connections 2024-03-27T21:39:48.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7f4dfb895640 1 -- 172.21.15.60:0/2815889185 wait complete. 2024-03-27T21:39:48.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7f4d7d3cb640 1 -- 172.21.15.60:0/2719490311 wait complete. 2024-03-27T21:39:48.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7f4d7d3cb640 10 monclient: build_initial_monmap 2024-03-27T21:39:48.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7f4d7d3cb640 10 monclient: monmap: 2024-03-27T21:39:48.252 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:48.252 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:48.252 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:48.252 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:48.252 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:48.252 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:48.252 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:48.252 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:48.252 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:48.253 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7f4d7d3cb640 1 librados: starting msgr at 2024-03-27T21:39:48.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7f4d7d3cb640 1 librados: starting objecter 2024-03-27T21:39:48.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7f4d7d3cb640 1 Processor -- start 2024-03-27T21:39:48.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fdba5696640 1 --2- 172.21.15.60:0/75719323 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb980c5060 0x7fdb98161780 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdba004f2a0 tx=0x7fdba00aa000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fdb967fc640 10 client.?.objecter ms_handle_connect 0x7fdb980c5060 2024-03-27T21:39:48.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fdb967fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:48.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdb967fc640 1 -- 172.21.15.60:0/75719323 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba00b9640 con 0x7fdb980c5060 2024-03-27T21:39:48.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdb967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdb967fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:48.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdb967fc640 10 monclient: dump: 2024-03-27T21:39:48.254 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.254 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.254 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.254 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.254 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.254 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.254 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.254 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.254 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.254 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdb967fc640 10 monclient: _finish_auth 0 2024-03-27T21:39:48.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdb967fc640 10 monclient: _check_auth_tickets 2024-03-27T21:39:48.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdb967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:48.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdb967fc640 1 -- 172.21.15.60:0/75719323 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba00b97a0 con 0x7fdb980c5060 2024-03-27T21:39:48.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdb967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:48.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdb967fc640 1 -- 172.21.15.60:0/75719323 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba00b9b50 con 0x7fdb980c5060 2024-03-27T21:39:48.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdb967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdb967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:48.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdb967fc640 10 monclient: dump: 2024-03-27T21:39:48.255 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.255 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.255 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.255 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.255 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.255 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.255 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.255 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.256 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.256 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdb977fe640 4 set_mon_vals no callback set 2024-03-27T21:39:48.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd9148a640 10 monclient(hunting): get_auth_request con 0x7fbd84161300 auth_method 0 2024-03-27T21:39:48.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd9148a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd9148a640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.144+0000 7fbd9148a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fbd9148a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fbd9148a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fbd9148a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fbd9148a640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-03-27T21:39:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fbd9148a640 1 -- 172.21.15.60:0/588725866 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd840c5760 msgr2=0x7fbd8400a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fbd9148a640 1 --2- 172.21.15.60:0/588725866 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd840c5760 0x7fbd8400a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fbd8bfff640 1 --2- 172.21.15.60:0/588725866 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd840c5760 0x7fbd8400a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:39:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fbd9148a640 1 -- 172.21.15.60:0/588725866 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd84038d70 msgr2=0x7fbd840cb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fbd9148a640 1 --2- 172.21.15.60:0/588725866 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd84038d70 0x7fbd840cb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fbd9148a640 10 monclient: _finish_hunting 0 2024-03-27T21:39:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fbd9148a640 1 monclient: found mon.noname-c 2024-03-27T21:39:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fbd9148a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fbd9148a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:39:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fbd9148a640 1 -- 172.21.15.60:0/588725866 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd78009400 con 0x7fbd84161300 2024-03-27T21:39:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fbd9148a640 1 --2- 172.21.15.60:0/588725866 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbd84161300 0x7fbd841617b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0639b0 tx=0x7fbd8c067040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fbd90c89640 1 --2- 172.21.15.60:0/588725866 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd84038d70 0x7fbd840cb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:39:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fbd897fa640 10 client.?.objecter ms_handle_connect 0x7fbd84161300 2024-03-27T21:39:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fbd897fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fbd897fa640 1 -- 172.21.15.60:0/588725866 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c0723c0 con 0x7fbd84161300 2024-03-27T21:39:48.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fbd897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fbd897fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:48.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fbd897fa640 10 monclient: dump: 2024-03-27T21:39:48.258 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.258 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.258 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.258 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.258 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.258 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.258 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.258 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.258 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.258 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fbd897fa640 10 monclient: _finish_auth 0 2024-03-27T21:39:48.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fbd897fa640 10 monclient: _check_auth_tickets 2024-03-27T21:39:48.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fbd897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:48.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fbd897fa640 1 -- 172.21.15.60:0/588725866 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8c0690b0 con 0x7fbd84161300 2024-03-27T21:39:48.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fd3e2c2f640 10 monclient: build_initial_monmap 2024-03-27T21:39:48.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fd3e2c2f640 10 monclient: monmap: 2024-03-27T21:39:48.259 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:48.259 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:48.259 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:48.259 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:48.259 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:48.259 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:48.260 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fd3e2c2f640 1 librados: starting msgr at 2024-03-27T21:39:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fd3e2c2f640 1 librados: starting objecter 2024-03-27T21:39:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fa800feb640 1 -- 172.21.15.60:0/3301199706 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f402cbb0 con 0x7fa7fc13ef20 2024-03-27T21:39:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fa800feb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fa800feb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fa800feb640 10 monclient: dump: 2024-03-27T21:39:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.261 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.261 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.261 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.261 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.261 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.261 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.261 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.261 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.261 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fa800feb640 10 monclient: _finish_auth 0 2024-03-27T21:39:48.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fa800feb640 10 monclient: _check_auth_tickets 2024-03-27T21:39:48.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fa800feb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:48.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fa804251640 5 monclient: authenticate success, global_id 4335 2024-03-27T21:39:48.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fa804251640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:48.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fa800feb640 1 -- 172.21.15.60:0/3301199706 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f402cd10 con 0x7fa7fc13ef20 2024-03-27T21:39:48.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fa800feb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:48.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fa804251640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:48.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.145+0000 7fa804251640 4 set_mon_vals no callback set 2024-03-27T21:39:48.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fa804251640 10 monclient: shutdown 2024-03-27T21:39:48.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fa804251640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:48.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fa804251640 1 -- 172.21.15.60:0/3301199706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7fc13ef20 msgr2=0x7fa7fc0a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fa800feb640 1 -- 172.21.15.60:0/3301199706 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f402cbb0 con 0x7fa7fc13ef20 2024-03-27T21:39:48.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fa804251640 1 --2- 172.21.15.60:0/3301199706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7fc13ef20 0x7fa7fc0a87e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4009e50 tx=0x7fa7f402c010 comp rx=0 tx=0).stop 2024-03-27T21:39:48.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fa800feb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-27T21:39:48.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fbd91c8b640 5 monclient: authenticate success, global_id 4304 2024-03-27T21:39:48.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fbd91c8b640 10 monclient: _renew_subs 2024-03-27T21:39:48.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fbd91c8b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:48.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fbd91c8b640 1 -- 172.21.15.60:0/588725866 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd8400b8d0 con 0x7fbd84161300 2024-03-27T21:39:48.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fd3e2c2f640 1 Processor -- start 2024-03-27T21:39:48.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fd3e2c2f640 1 -- start start 2024-03-27T21:39:48.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fd3e2c2f640 1 librados: setting wanted keys 2024-03-27T21:39:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fa804251640 1 -- 172.21.15.60:0/3301199706 shutdown_connections 2024-03-27T21:39:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fa804251640 1 --2- 172.21.15.60:0/3301199706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7fc13ef20 0x7fa7fc0a87e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fa804251640 1 -- 172.21.15.60:0/3301199706 >> 172.21.15.60:0/3301199706 conn(0x7fa7fc0b7300 msgr2=0x7fa7fc0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fd3e2c2f640 1 librados: calling monclient init 2024-03-27T21:39:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fd3e2c2f640 10 monclient: init 2024-03-27T21:39:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdba80fb640 5 monclient: authenticate success, global_id 4344 2024-03-27T21:39:48.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdba80fb640 10 monclient: _renew_subs 2024-03-27T21:39:48.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdba80fb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:48.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdba80fb640 1 -- 172.21.15.60:0/75719323 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb980c25e0 con 0x7fdb980c5060 2024-03-27T21:39:48.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdba80fb640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:48.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdba80fb640 10 monclient: _renew_subs 2024-03-27T21:39:48.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdba80fb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:48.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdba80fb640 1 -- 172.21.15.60:0/75719323 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb98161cc0 con 0x7fdb980c5060 2024-03-27T21:39:48.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fdba80fb640 1 librados: init done 2024-03-27T21:39:48.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fb578cc4640 5 monclient: authenticate success, global_id 4366 2024-03-27T21:39:48.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fb578cc4640 10 monclient: _renew_subs 2024-03-27T21:39:48.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fb578cc4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:48.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fb578cc4640 1 -- 172.21.15.60:0/3016765488 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb574141910 con 0x7fb5740afb20 2024-03-27T21:39:48.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fb578cc4640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:48.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fb578cc4640 10 monclient: _renew_subs 2024-03-27T21:39:48.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fb578cc4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:48.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fb578cc4640 1 -- 172.21.15.60:0/3016765488 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb57413d420 con 0x7fb5740afb20 2024-03-27T21:39:48.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fb578cc4640 1 librados: init done 2024-03-27T21:39:48.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7fb562ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:48.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7fb562ffd640 1 -- 172.21.15.60:0/3016765488 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c003830 con 0x7fb5740afb20 2024-03-27T21:39:48.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7fb562ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7fb562ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:48.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7fb562ffd640 10 monclient: dump: 2024-03-27T21:39:48.266 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.266 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.267 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.267 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.267 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.267 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.267 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.267 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.267 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.267 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7f4dfb895640 10 monclient: build_initial_monmap 2024-03-27T21:39:48.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7f4dfb895640 10 monclient: monmap: 2024-03-27T21:39:48.267 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:48.268 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:48.268 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:48.268 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:48.268 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:48.268 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:48.268 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:48.268 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:48.268 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:48.268 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7f4dfb895640 1 librados: starting msgr at 2024-03-27T21:39:48.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7f4dfb895640 1 librados: starting objecter 2024-03-27T21:39:48.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7f4dfb895640 1 Processor -- start 2024-03-27T21:39:48.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7f4dfb895640 1 -- start start 2024-03-27T21:39:48.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7f4dfb895640 1 librados: setting wanted keys 2024-03-27T21:39:48.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7f4dfb895640 1 librados: calling monclient init 2024-03-27T21:39:48.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7f4dfb895640 10 monclient: init 2024-03-27T21:39:48.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7f4dfb895640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:48.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7f4dfb895640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:39:48.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7f4dfb895640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dec035eb0 0x7f4dec02d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7f4dfb895640 10 monclient(hunting): picked mon.noname-a con 0x7f4dec035eb0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:48.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7f4dfb895640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4dec031450 0x7f4dec0318b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7f4dfb895640 10 monclient(hunting): picked mon.noname-b con 0x7f4dec031450 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:48.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7f4dfb895640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4dec0bc720 0x7f4dec0bcba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7f4dfb895640 10 monclient(hunting): picked mon.noname-c con 0x7f4dec0bc720 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:48.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7f4dfb895640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7f4dfb895640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec02ed30 con 0x7f4dec035eb0 2024-03-27T21:39:48.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7f4dfb895640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7f4dfb895640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec031df0 con 0x7f4dec031450 2024-03-27T21:39:48.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7f4dfb895640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7f4dfb895640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec033b90 con 0x7f4dec0bc720 2024-03-27T21:39:48.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7f4dfb895640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:48.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7f4dfb895640 10 monclient(hunting): authenticate will time out at 829.370294s 2024-03-27T21:39:48.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fefae170640 5 monclient: authenticate success, global_id 4360 2024-03-27T21:39:48.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fefae170640 10 monclient: _renew_subs 2024-03-27T21:39:48.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fefae170640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:48.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fefae170640 1 -- 172.21.15.60:0/4003671765 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa81e44a0 con 0x7fefa80a8b10 2024-03-27T21:39:48.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fefae170640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:48.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fefae170640 10 monclient: _renew_subs 2024-03-27T21:39:48.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fefae170640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:48.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fefae170640 1 -- 172.21.15.60:0/4003671765 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa813e850 con 0x7fefa80a8b10 2024-03-27T21:39:48.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.146+0000 7fefae170640 1 librados: init done 2024-03-27T21:39:48.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fef8b7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fef8b7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fef8b7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:48.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fef8b7fe640 1 -- 172.21.15.60:0/4003671765 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefa81e49c0 con 0x7fefa80a8b10 2024-03-27T21:39:48.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7fd3e2c2f640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:48.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7fd3e2c2f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:39:48.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7fd3e2c2f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd3dc060970 0x7fd3dc06e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7fd3e2c2f640 10 monclient(hunting): picked mon.noname-c con 0x7fd3dc060970 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:48.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fd3e09cb640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd3dc060970 0x7fd3dc06e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fd3e09cb640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd3dc060970 0x7fd3dc06e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47684/0 (socket says 172.21.15.60:47684) 2024-03-27T21:39:48.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fd3e09cb640 1 -- 172.21.15.60:0/1664087114 learned_addr learned my addr 172.21.15.60:0/1664087114 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:48.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fd3e2c2f640 1 --2- 172.21.15.60:0/1664087114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3dc0ad560 0x7fd3dc1da110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fd3e2c2f640 10 monclient(hunting): picked mon.noname-a con 0x7fd3dc0ad560 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:48.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fd3e2c2f640 1 --2- 172.21.15.60:0/1664087114 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3dc0af1f0 0x7fd3dc1da650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7fa804251640 1 -- 172.21.15.60:0/3301199706 shutdown_connections 2024-03-27T21:39:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7fbd91c8b640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7fbd91c8b640 10 monclient: _renew_subs 2024-03-27T21:39:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7fbd91c8b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7fbd91c8b640 1 -- 172.21.15.60:0/588725866 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd840c8e50 con 0x7fbd84161300 2024-03-27T21:39:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7fbd91c8b640 1 librados: init done 2024-03-27T21:39:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7fbd897fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7fbd8affd640 4 set_mon_vals no callback set 2024-03-27T21:39:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fbd897fa640 1 -- 172.21.15.60:0/588725866 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c069420 con 0x7fbd84161300 2024-03-27T21:39:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fbd897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fbd897fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fbd897fa640 10 monclient: dump: 2024-03-27T21:39:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.274 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.274 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.274 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.274 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.274 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.274 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.274 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.274 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fd3e2c2f640 10 monclient(hunting): picked mon.noname-b con 0x7fd3dc0af1f0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:48.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fd3e2c2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fd3e2c2f640 1 -- 172.21.15.60:0/1664087114 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc1d9cc0 con 0x7fd3dc0ad560 2024-03-27T21:39:48.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fd3e2c2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7fa804251640 1 -- 172.21.15.60:0/3301199706 wait complete. 2024-03-27T21:39:48.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7fa804251640 10 monclient: build_initial_monmap 2024-03-27T21:39:48.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fd3e2c2f640 1 -- 172.21.15.60:0/1664087114 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc131690 con 0x7fd3dc0af1f0 2024-03-27T21:39:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.147+0000 7fa804251640 10 monclient: monmap: 2024-03-27T21:39:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:48.276 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fa804251640 1 librados: starting msgr at 2024-03-27T21:39:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fa804251640 1 librados: starting objecter 2024-03-27T21:39:48.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fa804251640 1 Processor -- start 2024-03-27T21:39:48.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fd3e2c2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fd3e11cc640 1 --2- 172.21.15.60:0/1664087114 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3dc0af1f0 0x7fd3dc1da650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fd3e2c2f640 1 -- 172.21.15.60:0/1664087114 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc1369f0 con 0x7fd3dc060970 2024-03-27T21:39:48.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fd3e2c2f640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:48.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fd3e2c2f640 10 monclient(hunting): authenticate will time out at 829.370782s 2024-03-27T21:39:48.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fd3e09cb640 10 monclient(hunting): get_auth_request con 0x7fd3dc060970 auth_method 0 2024-03-27T21:39:48.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fd3e09cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fd3e09cb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fdba80fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fdba80fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fdba80fb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:48.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fdba80fb640 1 -- 172.21.15.60:0/75719323 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb9802f6a0 con 0x7fdb980c5060 2024-03-27T21:39:48.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fd3e09cb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df8e30640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4dec031450 0x7f4dec0318b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df8e30640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4dec031450 0x7f4dec0318b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:56792/0 (socket says 172.21.15.60:56792) 2024-03-27T21:39:48.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df8e30640 1 -- 172.21.15.60:0/2928743767 learned_addr learned my addr 172.21.15.60:0/2928743767 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:48.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df8e30640 10 monclient(hunting): get_auth_request con 0x7f4dec031450 auth_method 0 2024-03-27T21:39:48.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df8e30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df8e30640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df8e30640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df9631640 1 --2- 172.21.15.60:0/2928743767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dec035eb0 0x7f4dec02d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df9631640 10 monclient(hunting): get_auth_request con 0x7f4dec035eb0 auth_method 0 2024-03-27T21:39:48.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df9631640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df9631640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df9631640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fd3e11cc640 10 monclient(hunting): get_auth_request con 0x7fd3dc0af1f0 auth_method 0 2024-03-27T21:39:48.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7fd3e11cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7fd3e11cc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7fd3e11cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fb563fff640 4 set_mon_vals no callback set 2024-03-27T21:39:48.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.148+0000 7fb578cc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7fb578cc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7fb578cc4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:48.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df8e30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df8e30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df8e30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7fb578cc4640 1 -- 172.21.15.60:0/3016765488 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb558005040 con 0x7fb5740afb20 2024-03-27T21:39:48.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df9631640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df9631640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df9631640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7fa804251640 1 -- start start 2024-03-27T21:39:48.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df9631640 10 monclient(hunting): handle_auth_done global_id 4353 payload 824 2024-03-27T21:39:48.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df9631640 1 -- 172.21.15.60:0/2928743767 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4dec0bc720 msgr2=0x7f4dec0bcba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df9631640 1 --2- 172.21.15.60:0/2928743767 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4dec0bc720 0x7f4dec0bcba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df9631640 1 -- 172.21.15.60:0/2928743767 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4dec031450 msgr2=0x7f4dec0318b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df9631640 1 --2- 172.21.15.60:0/2928743767 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4dec031450 0x7f4dec0318b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df9631640 10 monclient: _finish_hunting 0 2024-03-27T21:39:48.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df9631640 1 monclient: found mon.noname-a 2024-03-27T21:39:48.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df9631640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:48.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df9631640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:48.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df9631640 1 -- 172.21.15.60:0/2928743767 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df00094d0 con 0x7f4dec035eb0 2024-03-27T21:39:48.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4df8e30640 1 --2- 172.21.15.60:0/2928743767 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4dec031450 0x7f4dec0318b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:48.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7fbd897fa640 1 -- 172.21.15.60:0/588725866 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fbd8c092400 con 0x7fbd84161300 2024-03-27T21:39:48.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7fa804251640 1 librados: setting wanted keys 2024-03-27T21:39:48.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4d7d3cb640 1 -- start start 2024-03-27T21:39:48.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4d7d3cb640 1 librados: setting wanted keys 2024-03-27T21:39:48.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4d7d3cb640 1 librados: calling monclient init 2024-03-27T21:39:48.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7f4d7d3cb640 10 monclient: init 2024-03-27T21:39:48.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7fa804251640 1 librados: calling monclient init 2024-03-27T21:39:48.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fa804251640 10 monclient: init 2024-03-27T21:39:48.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4df9631640 1 --2- 172.21.15.60:0/2928743767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dec035eb0 0x7f4dec02d7d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4df0033000 tx=0x7f4df00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d7d3cb640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:48.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d7d3cb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:39:48.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d7d3cb640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d7806f770 0x7f4d781e3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d7d3cb640 10 monclient(hunting): picked mon.noname-c con 0x7f4d7806f770 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:48.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d7d3cb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d7806fb50 0x7f4d781e3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d7d3cb640 10 monclient(hunting): picked mon.noname-a con 0x7f4d7806fb50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:48.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d7d3cb640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d78143060 0x7f4d780a95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fd3dbfff640 1 --2- 172.21.15.60:0/1664087114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3dc0ad560 0x7fd3dc1da110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fd3dbfff640 10 monclient(hunting): get_auth_request con 0x7fd3dc0ad560 auth_method 0 2024-03-27T21:39:48.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fd3dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fd3dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fd3dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d7d3cb640 10 monclient(hunting): picked mon.noname-b con 0x7f4d78143060 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:48.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d7d3cb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d7d3cb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d781e3580 con 0x7f4d7806fb50 2024-03-27T21:39:48.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d7d3cb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d7d3cb640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7806ea80 con 0x7f4d78143060 2024-03-27T21:39:48.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d7d3cb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d7d3cb640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d780a8a40 con 0x7f4d7806f770 2024-03-27T21:39:48.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d7d3cb640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:48.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d7d3cb640 10 monclient(hunting): authenticate will time out at 829.372081s 2024-03-27T21:39:48.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fd3e11cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fd3e11cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4de9ffb640 10 client.?.objecter ms_handle_connect 0x7f4dec035eb0 2024-03-27T21:39:48.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4de9ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:48.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.149+0000 7fb562ffd640 1 -- 172.21.15.60:0/3016765488 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb56c003990 con 0x7fb5740afb20 2024-03-27T21:39:48.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fb562ffd640 1 --2- 172.21.15.60:0/3016765488 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb55406c690 0x7fb55406eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fb57259c640 1 --2- 172.21.15.60:0/3016765488 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb55406c690 0x7fb55406eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fb562ffd640 1 -- 172.21.15.60:0/3016765488 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb56c0884f0 con 0x7fb5740afb20 2024-03-27T21:39:48.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fb562ffd640 10 client.4366.objecter ms_dispatch 0x7fb574000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:48.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fbd897fa640 1 --2- 172.21.15.60:0/588725866 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fbd7006c590 0x7fbd7006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fb562ffd640 3 client.4366.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:48.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fb562ffd640 3 client.4366.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:48.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fb57259c640 10 monclient: get_auth_request con 0x7fb55406c690 auth_method 0 2024-03-27T21:39:48.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d76ffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d7806f770 0x7f4d781e3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fb562ffd640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-27T21:39:48.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d76ffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d7806f770 0x7f4d781e3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47704/0 (socket says 172.21.15.60:47704) 2024-03-27T21:39:48.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d76ffd640 1 -- 172.21.15.60:0/993249291 learned_addr learned my addr 172.21.15.60:0/993249291 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:48.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d76ffd640 10 monclient(hunting): get_auth_request con 0x7f4d7806f770 auth_method 0 2024-03-27T21:39:48.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fbd91c8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fbd91c8b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4de9ffb640 1 -- 172.21.15.60:0/2928743767 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df003e070 con 0x7f4dec035eb0 2024-03-27T21:39:48.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4de9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4de9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:48.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4de9ffb640 10 monclient: dump: 2024-03-27T21:39:48.292 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.292 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.293 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.293 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.293 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.293 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.293 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.293 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.293 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.293 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4de9ffb640 10 monclient: _finish_auth 0 2024-03-27T21:39:48.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4de9ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:39:48.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4de9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:48.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4dfb895640 5 monclient: authenticate success, global_id 4353 2024-03-27T21:39:48.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4dfb895640 10 monclient: _renew_subs 2024-03-27T21:39:48.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4dfb895640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:48.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4dfb895640 1 -- 172.21.15.60:0/2928743767 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dec02ef10 con 0x7f4dec035eb0 2024-03-27T21:39:48.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d76ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d76ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d76ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d76ffd640 10 monclient(hunting): handle_auth_done global_id 4313 payload 824 2024-03-27T21:39:48.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d76ffd640 1 -- 172.21.15.60:0/993249291 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d78143060 msgr2=0x7f4d780a95e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-27T21:39:48.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d76ffd640 1 --2- 172.21.15.60:0/993249291 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d78143060 0x7f4d780a95e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d76ffd640 1 -- 172.21.15.60:0/993249291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d7806fb50 msgr2=0x7f4d781e3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fbd91c8b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:48.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fbd90c89640 1 --2- 172.21.15.60:0/588725866 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fbd7006c590 0x7fbd7006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fbd91c8b640 1 -- 172.21.15.60:0/588725866 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd84161000 con 0x7fbd84161300 2024-03-27T21:39:48.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d76ffd640 1 --2- 172.21.15.60:0/993249291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d7806fb50 0x7f4d781e3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4d76ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:39:48.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fbd90c89640 10 monclient: get_auth_request con 0x7fbd7006c590 auth_method 0 2024-03-27T21:39:48.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fbd897fa640 1 -- 172.21.15.60:0/588725866 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fbd8c0d1b80 con 0x7fbd84161300 2024-03-27T21:39:48.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fbd897fa640 10 client.4304.objecter ms_dispatch 0x7fbd8c000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:48.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fbd897fa640 3 client.4304.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:48.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fbd897fa640 3 client.4304.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:48.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fbd897fa640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-27T21:39:48.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7f4d76ffd640 1 monclient: found mon.noname-c 2024-03-27T21:39:48.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7f4d76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:48.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7f4d76ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:39:48.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7f4d76ffd640 1 -- 172.21.15.60:0/993249291 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d70009b30 con 0x7f4d7806f770 2024-03-27T21:39:48.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7fb57259c640 1 --2- 172.21.15.60:0/3016765488 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb55406c690 0x7fb55406eb50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb564035000 tx=0x7fb564030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4dfb895640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:48.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4dfb895640 10 monclient: _renew_subs 2024-03-27T21:39:48.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4dfb895640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:48.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4dfb895640 1 -- 172.21.15.60:0/2928743767 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dec032e90 con 0x7f4dec035eb0 2024-03-27T21:39:48.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7f4dfb895640 1 librados: init done 2024-03-27T21:39:48.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7fb562ffd640 10 client.4366.objecter ms_handle_connect 0x7fb55406c690 2024-03-27T21:39:48.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7fbd90c89640 1 --2- 172.21.15.60:0/588725866 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fbd7006c590 0x7fbd7006ea50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbd800094a0 tx=0x7fbd80008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7fbd897fa640 10 client.4304.objecter ms_handle_connect 0x7fbd7006c590 2024-03-27T21:39:48.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7fdb967fc640 1 -- 172.21.15.60:0/75719323 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fdba00ad070 con 0x7fdb980c5060 2024-03-27T21:39:48.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7f4d76ffd640 1 --2- 172.21.15.60:0/993249291 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d7806f770 0x7f4d781e3a10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4d6800c450 tx=0x7f4d6800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7f4d5f7fe640 10 client.?.objecter ms_handle_connect 0x7f4d7806f770 2024-03-27T21:39:48.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7f4d5f7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:48.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7fef99ffb640 1 -- 172.21.15.60:0/4003671765 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fef9c01a400 con 0x7fefa80a8b10 2024-03-27T21:39:48.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.150+0000 7fd3e11cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7f4de9ffb640 1 -- 172.21.15.60:0/2928743767 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df00039a0 con 0x7f4dec035eb0 2024-03-27T21:39:48.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7f4de9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:48.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7f4de9ffb640 1 -- 172.21.15.60:0/2928743767 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df0030890 con 0x7f4dec035eb0 2024-03-27T21:39:48.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7f4de9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7f4de9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:48.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7f4de9ffb640 10 monclient: dump: 2024-03-27T21:39:48.300 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.300 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.300 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.301 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.301 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.301 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.301 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.301 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.301 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.301 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7f4deb7fe640 4 set_mon_vals no callback set 2024-03-27T21:39:48.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7fd3dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7fd3dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7fd3dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7fd3e09cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7fd3e09cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7fd3e09cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7fa804251640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:48.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7fa804251640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:39:48.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7fa804251640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa7fc071a50 0x7fa7fc1e69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.151+0000 7fa804251640 10 monclient(hunting): picked mon.noname-c con 0x7fa7fc071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:48.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa804251640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7fc13ef20 0x7fa7fc14bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa804251640 10 monclient(hunting): picked mon.noname-b con 0x7fa7fc13ef20 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:48.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa804251640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7fc141630 0x7fa7fc14c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fd3e11cc640 10 monclient(hunting): handle_auth_done global_id 4369 payload 824 2024-03-27T21:39:48.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7f4dfb895640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa8027ee640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7fc141630 0x7fa7fc14c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7f4dfb895640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7f4dfb895640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:48.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa8027ee640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7fc141630 0x7fa7fc14c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47898/0 (socket says 172.21.15.60:47898) 2024-03-27T21:39:48.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fef99ffb640 1 --2- 172.21.15.60:0/4003671765 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fef8c06c5e0 0x7fef8c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fef99ffb640 1 -- 172.21.15.60:0/4003671765 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fef9c089220 con 0x7fefa80a8b10 2024-03-27T21:39:48.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fef99ffb640 10 client.4360.objecter ms_dispatch 0x7fefa8000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:48.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fef99ffb640 3 client.4360.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:48.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fef99ffb640 3 client.4360.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:48.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fef99ffb640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-27T21:39:48.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa8027ee640 1 -- 172.21.15.60:0/2465738873 learned_addr learned my addr 172.21.15.60:0/2465738873 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:48.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7f4dfb895640 1 -- 172.21.15.60:0/2928743767 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd4005040 con 0x7f4dec035eb0 2024-03-27T21:39:48.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa804251640 10 monclient(hunting): picked mon.noname-a con 0x7fa7fc141630 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:48.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fdb967fc640 1 --2- 172.21.15.60:0/75719323 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdb8406c5e0 0x7fdb8406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa804251640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa804251640 1 -- 172.21.15.60:0/2465738873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc1e73a0 con 0x7fa7fc141630 2024-03-27T21:39:48.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa804251640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa801fed640 1 --2- 172.21.15.60:0/2465738873 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa7fc071a50 0x7fa7fc1e69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa804251640 1 -- 172.21.15.60:0/2465738873 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc1329c0 con 0x7fa7fc13ef20 2024-03-27T21:39:48.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa804251640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:48.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa804251640 1 -- 172.21.15.60:0/2465738873 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc132ca0 con 0x7fa7fc071a50 2024-03-27T21:39:48.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa804251640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:48.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa8017ec640 1 --2- 172.21.15.60:0/2465738873 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7fc13ef20 0x7fa7fc14bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fefac96d640 1 --2- 172.21.15.60:0/4003671765 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fef8c06c5e0 0x7fef8c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fefac96d640 10 monclient: get_auth_request con 0x7fef8c06c5e0 auth_method 0 2024-03-27T21:39:48.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fd3e11cc640 1 -- 172.21.15.60:0/1664087114 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd3dc060970 msgr2=0x7fd3dc06e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fd3e11cc640 1 --2- 172.21.15.60:0/1664087114 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd3dc060970 0x7fd3dc06e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa804251640 10 monclient(hunting): authenticate will time out at 829.374700s 2024-03-27T21:39:48.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fdb967fc640 1 -- 172.21.15.60:0/75719323 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fdba0124c10 con 0x7fdb980c5060 2024-03-27T21:39:48.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fdb967fc640 10 client.4344.objecter ms_dispatch 0x7fdba0000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:48.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fd3e11cc640 1 -- 172.21.15.60:0/1664087114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3dc0ad560 msgr2=0x7fd3dc1da110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa8027ee640 10 monclient(hunting): get_auth_request con 0x7fa7fc141630 auth_method 0 2024-03-27T21:39:48.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa8027ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa8027ee640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fdb967fc640 3 client.4344.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:48.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fdb967fc640 3 client.4344.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:48.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa8027ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa801fed640 10 monclient(hunting): get_auth_request con 0x7fa7fc071a50 auth_method 0 2024-03-27T21:39:48.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa801fed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa801fed640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa801fed640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fd3e11cc640 1 --2- 172.21.15.60:0/1664087114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3dc0ad560 0x7fd3dc1da110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fd3e11cc640 10 monclient: _finish_hunting 0 2024-03-27T21:39:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fd3e11cc640 1 monclient: found mon.noname-b 2024-03-27T21:39:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fd3e11cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fd3e11cc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:39:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fd3e11cc640 1 -- 172.21.15.60:0/1664087114 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c80094d0 con 0x7fd3dc0af1f0 2024-03-27T21:39:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fd3dbfff640 1 --2- 172.21.15.60:0/1664087114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3dc0ad560 0x7fd3dc1da110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:48.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fdb967fc640 20 client.4344.objecter dump_active .. 0 homeless 2024-03-27T21:39:48.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fd3e09cb640 1 --2- 172.21.15.60:0/1664087114 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd3dc060970 0x7fd3dc06e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:48.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fdba5e97640 1 --2- 172.21.15.60:0/75719323 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdb8406c5e0 0x7fdb8406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fdba5e97640 10 monclient: get_auth_request con 0x7fdb8406c5e0 auth_method 0 2024-03-27T21:39:48.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fd3d97fa640 10 client.?.objecter ms_handle_connect 0x7fd3dc0af1f0 2024-03-27T21:39:48.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fd3d97fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:48.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fd3e11cc640 1 --2- 172.21.15.60:0/1664087114 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3dc0af1f0 0x7fd3dc1da650 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd3d400c8d0 tx=0x7fd3d400cd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa8027ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa8027ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.152+0000 7fa8027ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fa801fed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:48.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fa801fed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:48.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fa801fed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:48.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fa8017ec640 10 monclient(hunting): get_auth_request con 0x7fa7fc13ef20 auth_method 0 2024-03-27T21:39:48.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fa8017ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:48.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fa8017ec640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:48.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fa8017ec640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:48.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fa8027ee640 10 monclient(hunting): handle_auth_done global_id 4359 payload 824 2024-03-27T21:39:48.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fefac96d640 1 --2- 172.21.15.60:0/4003671765 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fef8c06c5e0 0x7fef8c06eaa0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fefa813db30 tx=0x7fefa4009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fef99ffb640 10 client.4360.objecter ms_handle_connect 0x7fef8c06c5e0 2024-03-27T21:39:48.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fa8027ee640 1 -- 172.21.15.60:0/2465738873 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa7fc071a50 msgr2=0x7fa7fc1e69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fa8027ee640 1 --2- 172.21.15.60:0/2465738873 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa7fc071a50 0x7fa7fc1e69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fa8027ee640 1 -- 172.21.15.60:0/2465738873 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7fc13ef20 msgr2=0x7fa7fc14bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fa8027ee640 1 --2- 172.21.15.60:0/2465738873 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7fc13ef20 0x7fa7fc14bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fa8027ee640 10 monclient: _finish_hunting 0 2024-03-27T21:39:48.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fa8027ee640 1 monclient: found mon.noname-a 2024-03-27T21:39:48.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fa8027ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:48.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fdba5e97640 1 --2- 172.21.15.60:0/75719323 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdb8406c5e0 0x7fdb8406eaa0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdb90001750 tx=0x7fdb90009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fdb967fc640 10 client.4344.objecter ms_handle_connect 0x7fdb8406c5e0 2024-03-27T21:39:48.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fa8027ee640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:48.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fa8027ee640 1 -- 172.21.15.60:0/2465738873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f4009b30 con 0x7fa7fc141630 2024-03-27T21:39:48.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.153+0000 7fa8017ec640 1 --2- 172.21.15.60:0/2465738873 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7fc13ef20 0x7fa7fc14bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:39:48.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.154+0000 7fa7f27fc640 10 client.?.objecter ms_handle_connect 0x7fa7fc141630 2024-03-27T21:39:48.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.154+0000 7fa7f27fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:48.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.154+0000 7fa8027ee640 1 --2- 172.21.15.60:0/2465738873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7fc141630 0x7fa7fc14c170 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec00c750 tx=0x7fa7ec00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.154+0000 7fef99ffb640 1 -- 172.21.15.60:0/4003671765 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fef9c054820 con 0x7fefa80a8b10 2024-03-27T21:39:48.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.154+0000 7fef99ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.154+0000 7fef99ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:48.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3d97fa640 1 -- 172.21.15.60:0/1664087114 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d4003da0 con 0x7fd3dc0af1f0 2024-03-27T21:39:48.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3d97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:48.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3d97fa640 10 monclient: dump: 2024-03-27T21:39:48.319 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.319 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.319 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.319 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.319 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.319 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.319 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.319 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.319 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.319 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3d97fa640 10 monclient: _finish_auth 0 2024-03-27T21:39:48.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3d97fa640 10 monclient: _check_auth_tickets 2024-03-27T21:39:48.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:48.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3e2c2f640 5 monclient: authenticate success, global_id 4369 2024-03-27T21:39:48.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3e2c2f640 10 monclient: _renew_subs 2024-03-27T21:39:48.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3e2c2f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:48.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3e2c2f640 1 -- 172.21.15.60:0/1664087114 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d0009400 con 0x7fd3dc0af1f0 2024-03-27T21:39:48.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3d97fa640 1 -- 172.21.15.60:0/1664087114 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d4005070 con 0x7fd3dc0af1f0 2024-03-27T21:39:48.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3e2c2f640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:48.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3d97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:48.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3e2c2f640 10 monclient: _renew_subs 2024-03-27T21:39:48.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3e2c2f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:48.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3e2c2f640 1 -- 172.21.15.60:0/1664087114 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3dc1379d0 con 0x7fd3dc0af1f0 2024-03-27T21:39:48.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3e2c2f640 1 librados: init done 2024-03-27T21:39:48.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3daffd640 4 set_mon_vals no callback set 2024-03-27T21:39:48.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3d97fa640 1 -- 172.21.15.60:0/1664087114 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d4003da0 con 0x7fd3dc0af1f0 2024-03-27T21:39:48.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3d97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:48.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fd3d97fa640 10 monclient: dump: 2024-03-27T21:39:48.321 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.321 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.321 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.321 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.321 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.321 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.321 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.321 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.321 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.321 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.158+0000 7fb562ffd640 1 -- 172.21.15.60:0/3016765488 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb56c053ae0 con 0x7fb5740afb20 2024-03-27T21:39:48.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.159+0000 7fb562ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.159+0000 7fb562ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:48.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.159+0000 7fd3e2c2f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.159+0000 7fd3e2c2f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.159+0000 7fd3e2c2f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:48.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.159+0000 7fd3e2c2f640 1 -- 172.21.15.60:0/1664087114 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3dc0af7f0 con 0x7fd3dc0af1f0 2024-03-27T21:39:48.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7fd3d97fa640 1 -- 172.21.15.60:0/1664087114 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd3d40051e0 con 0x7fd3dc0af1f0 2024-03-27T21:39:48.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7fbd897fa640 1 -- 172.21.15.60:0/588725866 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbd8c0cd7c0 con 0x7fbd84161300 2024-03-27T21:39:48.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7fbd897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7fbd897fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:48.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7f4d5f7fe640 1 -- 172.21.15.60:0/993249291 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d680052d0 con 0x7f4d7806f770 2024-03-27T21:39:48.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7f4d5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7f4d5f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:48.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7f4d5f7fe640 10 monclient: dump: 2024-03-27T21:39:48.324 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.324 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.325 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.325 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.325 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.325 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.325 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.325 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.325 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.325 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7f4d5f7fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:48.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7f4d5f7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:48.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7f4d5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:48.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7fd3d97fa640 1 --2- 172.21.15.60:0/1664087114 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd3c006c650 0x7fd3c006eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7f4d5f7fe640 1 -- 172.21.15.60:0/993249291 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d68005430 con 0x7f4d7806f770 2024-03-27T21:39:48.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7f4d7d3cb640 5 monclient: authenticate success, global_id 4313 2024-03-27T21:39:48.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7f4d7d3cb640 10 monclient: _renew_subs 2024-03-27T21:39:48.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7f4d7d3cb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:48.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7f4d7d3cb640 1 -- 172.21.15.60:0/993249291 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d6c009400 con 0x7f4d7806f770 2024-03-27T21:39:48.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7f4d7d3cb640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:48.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7f4d7d3cb640 10 monclient: _renew_subs 2024-03-27T21:39:48.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7f4d7d3cb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:48.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7f4d7d3cb640 1 -- 172.21.15.60:0/993249291 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d780ab6a0 con 0x7f4d7806f770 2024-03-27T21:39:48.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.160+0000 7f4d7d3cb640 1 librados: init done 2024-03-27T21:39:48.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.161+0000 7f4d5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:48.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.161+0000 7fdb967fc640 1 -- 172.21.15.60:0/75719323 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdb9802f6a0 con 0x7fdb980c5060 2024-03-27T21:39:48.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.161+0000 7fdb967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.161+0000 7fdb967fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:48.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.161+0000 7fd3d97fa640 1 -- 172.21.15.60:0/1664087114 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd3d40885d0 con 0x7fd3dc0af1f0 2024-03-27T21:39:48.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.161+0000 7fd3d97fa640 10 client.4369.objecter ms_dispatch 0x7fd3dc000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:48.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.161+0000 7fd3d97fa640 3 client.4369.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:48.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.161+0000 7fd3d97fa640 3 client.4369.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:48.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.161+0000 7fd3d97fa640 20 client.4369.objecter dump_active .. 0 homeless 2024-03-27T21:39:48.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.161+0000 7fd3e09cb640 1 --2- 172.21.15.60:0/1664087114 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd3c006c650 0x7fd3c006eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.161+0000 7fd3e09cb640 10 monclient: get_auth_request con 0x7fd3c006c650 auth_method 0 2024-03-27T21:39:48.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.161+0000 7fd3d97fa640 10 client.4369.objecter ms_handle_connect 0x7fd3c006c650 2024-03-27T21:39:48.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.161+0000 7fd3e09cb640 1 --2- 172.21.15.60:0/1664087114 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd3c006c650 0x7fd3c006eb10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd3d002b750 tx=0x7fd3d0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.161+0000 7f4d757fa640 4 set_mon_vals no callback set 2024-03-27T21:39:48.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.162+0000 7f4d5f7fe640 1 -- 172.21.15.60:0/993249291 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d68012850 con 0x7f4d7806f770 2024-03-27T21:39:48.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.162+0000 7f4d5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.162+0000 7f4d5f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:48.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.162+0000 7f4d5f7fe640 10 monclient: dump: 2024-03-27T21:39:48.329 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.329 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.329 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.329 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.329 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.329 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.329 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.329 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.329 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.329 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.162+0000 7fa7f27fc640 1 -- 172.21.15.60:0/2465738873 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec005070 con 0x7fa7fc141630 2024-03-27T21:39:48.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.162+0000 7fa7f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.162+0000 7fa7f27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:48.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.162+0000 7fa7f27fc640 10 monclient: dump: 2024-03-27T21:39:48.331 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.331 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.331 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.331 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.331 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.331 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.331 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.331 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.331 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.331 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.162+0000 7fa7f27fc640 10 monclient: _finish_auth 0 2024-03-27T21:39:48.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.162+0000 7fa7f27fc640 10 monclient: _check_auth_tickets 2024-03-27T21:39:48.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.162+0000 7fa7f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:48.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.162+0000 7fa804251640 5 monclient: authenticate success, global_id 4359 2024-03-27T21:39:48.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.162+0000 7fa804251640 10 monclient: _renew_subs 2024-03-27T21:39:48.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.162+0000 7fa804251640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:48.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.162+0000 7fa804251640 1 -- 172.21.15.60:0/2465738873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7fc14c6b0 con 0x7fa7fc141630 2024-03-27T21:39:48.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.162+0000 7fa804251640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:48.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.162+0000 7fa804251640 10 monclient: _renew_subs 2024-03-27T21:39:48.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.162+0000 7fa804251640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:48.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.162+0000 7fa804251640 1 -- 172.21.15.60:0/2465738873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7fc1e6f10 con 0x7fa7fc141630 2024-03-27T21:39:48.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.163+0000 7f4d7d3cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.163+0000 7f4d7d3cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.163+0000 7f4d7d3cb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:48.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.163+0000 7f4d7d3cb640 1 -- 172.21.15.60:0/993249291 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d58005040 con 0x7f4d7806f770 2024-03-27T21:39:48.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.162+0000 7fa804251640 1 librados: init done 2024-03-27T21:39:48.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.164+0000 7f4de9ffb640 1 -- 172.21.15.60:0/2928743767 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4df0003b50 con 0x7f4dec035eb0 2024-03-27T21:39:48.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.164+0000 7f4de9ffb640 1 --2- 172.21.15.60:0/2928743767 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4dd806c660 0x7f4dd806eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.164+0000 7f4d5f7fe640 1 -- 172.21.15.60:0/993249291 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4d680129b0 con 0x7f4d7806f770 2024-03-27T21:39:48.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.163+0000 7fa7f27fc640 1 -- 172.21.15.60:0/2465738873 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7ec0051d0 con 0x7fa7fc141630 2024-03-27T21:39:48.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.163+0000 7fa7f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:48.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.163+0000 7fa7f3fff640 4 set_mon_vals no callback set 2024-03-27T21:39:48.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.164+0000 7fa7f27fc640 1 -- 172.21.15.60:0/2465738873 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec005070 con 0x7fa7fc141630 2024-03-27T21:39:48.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.164+0000 7fa7f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:48.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.164+0000 7fa7f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:48.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.164+0000 7fa7f27fc640 10 monclient: dump: 2024-03-27T21:39:48.335 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:48.335 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:48.335 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.335 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:48.335 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:48.335 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:48.335 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:48.335 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:48.337 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:48.337 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:48.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.164+0000 7f4df8e30640 1 --2- 172.21.15.60:0/2928743767 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4dd806c660 0x7f4dd806eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.164+0000 7f4df8e30640 10 monclient: get_auth_request con 0x7f4dd806c660 auth_method 0 2024-03-27T21:39:48.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.165+0000 7f4d5f7fe640 1 --2- 172.21.15.60:0/993249291 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4d6406c660 0x7f4d6406eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.165+0000 7f4de9ffb640 1 -- 172.21.15.60:0/2928743767 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4df00ac260 con 0x7f4dec035eb0 2024-03-27T21:39:48.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.165+0000 7f4de9ffb640 10 client.4353.objecter ms_dispatch 0x7f4df4000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:48.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.165+0000 7f4de9ffb640 3 client.4353.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:48.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.165+0000 7f4de9ffb640 3 client.4353.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:48.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.165+0000 7f4d5f7fe640 1 -- 172.21.15.60:0/993249291 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4d68088c50 con 0x7f4d7806f770 2024-03-27T21:39:48.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.165+0000 7f4d5f7fe640 10 client.4313.objecter ms_dispatch 0x7f4d78000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:48.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.165+0000 7f4d5f7fe640 3 client.4313.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:48.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.165+0000 7f4d5f7fe640 3 client.4313.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:48.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.165+0000 7f4de9ffb640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-27T21:39:48.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.165+0000 7f4df8e30640 1 --2- 172.21.15.60:0/2928743767 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4dd806c660 0x7f4dd806eb20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4dec0bdae0 tx=0x7f4de4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.165+0000 7f4de9ffb640 10 client.4353.objecter ms_handle_connect 0x7f4dd806c660 2024-03-27T21:39:48.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.165+0000 7f4d5f7fe640 20 client.4313.objecter dump_active .. 0 homeless 2024-03-27T21:39:48.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.165+0000 7f4d767fc640 1 --2- 172.21.15.60:0/993249291 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4d6406c660 0x7f4d6406eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.165+0000 7f4d767fc640 10 monclient: get_auth_request con 0x7f4d6406c660 auth_method 0 2024-03-27T21:39:48.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.166+0000 7f4d767fc640 1 --2- 172.21.15.60:0/993249291 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4d6406c660 0x7f4d6406eb20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c036000 tx=0x7f4d6c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.166+0000 7f4d5f7fe640 10 client.4313.objecter ms_handle_connect 0x7f4d6406c660 2024-03-27T21:39:48.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.167+0000 7fa804251640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.167+0000 7fa804251640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.167+0000 7fa804251640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:48.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.167+0000 7fa804251640 1 -- 172.21.15.60:0/2465738873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e4005040 con 0x7fa7fc141630 2024-03-27T21:39:48.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.167+0000 7fd3d97fa640 1 -- 172.21.15.60:0/1664087114 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd3d4053a60 con 0x7fd3dc0af1f0 2024-03-27T21:39:48.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.167+0000 7fd3d97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.167+0000 7fd3d97fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:48.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.174+0000 7f4de9ffb640 1 -- 172.21.15.60:0/2928743767 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4df0077610 con 0x7f4dec035eb0 2024-03-27T21:39:48.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.174+0000 7f4de9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.174+0000 7f4de9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:48.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.174+0000 7f4d5f7fe640 1 -- 172.21.15.60:0/993249291 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4d68054000 con 0x7f4d7806f770 2024-03-27T21:39:48.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.174+0000 7f4d5f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.174+0000 7f4d5f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:48.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.174+0000 7fa7f27fc640 1 -- 172.21.15.60:0/2465738873 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa7ec007020 con 0x7fa7fc141630 2024-03-27T21:39:48.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.175+0000 7fa7f27fc640 1 --2- 172.21.15.60:0/2465738873 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa7e006c650 0x7fa7e006eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:48.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.175+0000 7fa7f27fc640 1 -- 172.21.15.60:0/2465738873 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa7ec057740 con 0x7fa7fc141630 2024-03-27T21:39:48.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.175+0000 7fa7f27fc640 10 client.4359.objecter ms_dispatch 0x7fa7fc000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:48.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.175+0000 7fa7f27fc640 3 client.4359.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:48.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.175+0000 7fa7f27fc640 3 client.4359.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:48.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.175+0000 7fa7f27fc640 20 client.4359.objecter dump_active .. 0 homeless 2024-03-27T21:39:48.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.178+0000 7fa801fed640 1 --2- 172.21.15.60:0/2465738873 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa7e006c650 0x7fa7e006eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:48.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.178+0000 7fa801fed640 10 monclient: get_auth_request con 0x7fa7e006c650 auth_method 0 2024-03-27T21:39:48.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.179+0000 7fa7f27fc640 10 client.4359.objecter ms_handle_connect 0x7fa7e006c650 2024-03-27T21:39:48.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.179+0000 7fa801fed640 1 --2- 172.21.15.60:0/2465738873 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa7e006c650 0x7fa7e006eb10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8000f60 tx=0x7fa7f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:48.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.182+0000 7fa7f27fc640 1 -- 172.21.15.60:0/2465738873 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa7ec054210 con 0x7fa7fc141630 2024-03-27T21:39:48.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.182+0000 7fa7f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:48.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.182+0000 7fa7f27fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:48.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.572+0000 7fef8b7fe640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-27T21:39:48.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.572+0000 7fef8b7fe640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-27T21:39:48.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.572+0000 7fef8b7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:48.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.573+0000 7fef8b7fe640 1 -- 172.21.15.60:0/4003671765 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fefa81e49c0 con 0x7fefa80a8b10 2024-03-27T21:39:48.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.573+0000 7fef99ffb640 1 -- 172.21.15.60:0/4003671765 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fef9c015090 con 0x7fefa80a8b10 2024-03-27T21:39:48.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.573+0000 7fef99ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-27T21:39:48.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.573+0000 7fef99ffb640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:48.562 INFO:teuthology.orchestra.run.smithi060.stdout:60129542147 2024-03-27T21:39:48.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.576+0000 7fefae170640 10 librados: watch_flush enter 2024-03-27T21:39:48.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.577+0000 7fefae170640 10 librados: watch_flush exit 2024-03-27T21:39:48.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.577+0000 7fefae170640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-27T21:39:48.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.577+0000 7fefae170640 10 client.4360.objecter successfully canceled tick 2024-03-27T21:39:48.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.577+0000 7fefae170640 1 -- 172.21.15.60:0/4003671765 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fef8c06c5e0 msgr2=0x7fef8c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.577+0000 7fefae170640 1 --2- 172.21.15.60:0/4003671765 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fef8c06c5e0 0x7fef8c06eaa0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fefa813db30 tx=0x7fefa4009190 comp rx=0 tx=0).stop 2024-03-27T21:39:48.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.577+0000 7fefae170640 10 monclient: shutdown 2024-03-27T21:39:48.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.577+0000 7fefae170640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:48.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.577+0000 7fefae170640 1 -- 172.21.15.60:0/4003671765 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefa80a8b10 msgr2=0x7fefa813a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.577+0000 7fefae170640 1 --2- 172.21.15.60:0/4003671765 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefa80a8b10 0x7fefa813a520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fef9c009fd0 tx=0x7fef9c00b870 comp rx=0 tx=0).stop 2024-03-27T21:39:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.577+0000 7fefae170640 1 -- 172.21.15.60:0/4003671765 shutdown_connections 2024-03-27T21:39:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.578+0000 7fefae170640 1 --2- 172.21.15.60:0/4003671765 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fefa81417e0 0x7fefa81524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.578+0000 7fefae170640 1 --2- 172.21.15.60:0/4003671765 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa8137c50 0x7fefa8151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.578+0000 7fefae170640 1 --2- 172.21.15.60:0/4003671765 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fef8c06c5e0 0x7fef8c06eaa0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.578+0000 7fefae170640 1 --2- 172.21.15.60:0/4003671765 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fefa80a8b10 0x7fefa813a520 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.578+0000 7fefae170640 1 -- 172.21.15.60:0/4003671765 >> 172.21.15.60:0/4003671765 conn(0x7fefa8060700 msgr2=0x7fefa81409e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.578+0000 7fefae170640 1 -- 172.21.15.60:0/4003671765 shutdown_connections 2024-03-27T21:39:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.578+0000 7fefae170640 1 -- 172.21.15.60:0/4003671765 wait complete. 2024-03-27T21:39:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.578+0000 7fefae170640 1 librados: shutdown 2024-03-27T21:39:48.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.582+0000 7fb578cc4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-27T21:39:48.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.582+0000 7fb578cc4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-27T21:39:48.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.582+0000 7fb578cc4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:48.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.583+0000 7fb578cc4640 1 -- 172.21.15.60:0/3016765488 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fb5580052f0 con 0x7fb5740afb20 2024-03-27T21:39:48.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.583+0000 7fb562ffd640 1 -- 172.21.15.60:0/3016765488 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb56c057100 con 0x7fb5740afb20 2024-03-27T21:39:48.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.583+0000 7fb562ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-27T21:39:48.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.583+0000 7fb562ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:48.571 INFO:teuthology.orchestra.run.smithi060.stdout:68719476740 2024-03-27T21:39:48.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.584+0000 7fbd91c8b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-27T21:39:48.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.584+0000 7fbd91c8b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-27T21:39:48.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.585+0000 7fbd91c8b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:48.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.585+0000 7fbd91c8b640 1 -- 172.21.15.60:0/588725866 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fbd840056d0 con 0x7fbd84161300 2024-03-27T21:39:48.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.586+0000 7fbd897fa640 1 -- 172.21.15.60:0/588725866 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbd8c0d0de0 con 0x7fbd84161300 2024-03-27T21:39:48.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.586+0000 7fbd897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-27T21:39:48.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.586+0000 7fbd897fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:48.574 INFO:teuthology.orchestra.run.smithi060.stdout:60129542147 2024-03-27T21:39:48.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.587+0000 7fb560ff9640 10 librados: watch_flush enter 2024-03-27T21:39:48.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.587+0000 7fb560ff9640 10 librados: watch_flush exit 2024-03-27T21:39:48.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.587+0000 7fb560ff9640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-27T21:39:48.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.587+0000 7fb560ff9640 10 client.4366.objecter successfully canceled tick 2024-03-27T21:39:48.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.587+0000 7fb560ff9640 1 -- 172.21.15.60:0/3016765488 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb55406c690 msgr2=0x7fb55406eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.587+0000 7fb560ff9640 1 --2- 172.21.15.60:0/3016765488 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb55406c690 0x7fb55406eb50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb564035000 tx=0x7fb564030000 comp rx=0 tx=0).stop 2024-03-27T21:39:48.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.587+0000 7fb560ff9640 10 monclient: shutdown 2024-03-27T21:39:48.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.587+0000 7fb560ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:48.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.587+0000 7fb560ff9640 1 -- 172.21.15.60:0/3016765488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5740afb20 msgr2=0x7fb57413cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.587+0000 7fb560ff9640 1 --2- 172.21.15.60:0/3016765488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5740afb20 0x7fb57413cd00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb57406eb30 tx=0x7fb56c00c7a0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.587+0000 7fb560ff9640 1 -- 172.21.15.60:0/3016765488 shutdown_connections 2024-03-27T21:39:48.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.587+0000 7fb560ff9640 1 --2- 172.21.15.60:0/3016765488 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb55406c690 0x7fb55406eb50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.587+0000 7fb560ff9640 1 --2- 172.21.15.60:0/3016765488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5740afb20 0x7fb57413cd00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.587+0000 7fb560ff9640 1 --2- 172.21.15.60:0/3016765488 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb5740ade30 0x7fb5741ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.587+0000 7fb560ff9640 1 --2- 172.21.15.60:0/3016765488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb574073640 0x7fb57413da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.587+0000 7fdba80fb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-27T21:39:48.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.587+0000 7fdba80fb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-27T21:39:48.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.587+0000 7fdba80fb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:48.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.587+0000 7fdba80fb640 1 -- 172.21.15.60:0/75719323 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fdb981610c0 con 0x7fdb980c5060 2024-03-27T21:39:48.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.588+0000 7fb560ff9640 1 -- 172.21.15.60:0/3016765488 >> 172.21.15.60:0/3016765488 conn(0x7fb5740b3de0 msgr2=0x7fb574069ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:48.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.588+0000 7fb560ff9640 1 -- 172.21.15.60:0/3016765488 shutdown_connections 2024-03-27T21:39:48.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.588+0000 7fb560ff9640 1 -- 172.21.15.60:0/3016765488 wait complete. 2024-03-27T21:39:48.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.588+0000 7fd3e2c2f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-27T21:39:48.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.588+0000 7fb560ff9640 1 librados: shutdown 2024-03-27T21:39:48.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.588+0000 7fd3e2c2f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-27T21:39:48.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.588+0000 7fd3e2c2f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:48.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.588+0000 7fd3e2c2f640 1 -- 172.21.15.60:0/1664087114 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd3dc069470 con 0x7fd3dc0af1f0 2024-03-27T21:39:48.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.588+0000 7fdb967fc640 1 -- 172.21.15.60:0/75719323 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdb981610c0 con 0x7fdb980c5060 2024-03-27T21:39:48.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.588+0000 7fdb967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-27T21:39:48.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.588+0000 7fdb967fc640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:48.579 INFO:teuthology.orchestra.run.smithi060.stdout:68719476741 2024-03-27T21:39:48.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.589+0000 7fbd91c8b640 10 librados: watch_flush enter 2024-03-27T21:39:48.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.589+0000 7fbd91c8b640 10 librados: watch_flush exit 2024-03-27T21:39:48.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.589+0000 7fbd91c8b640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-27T21:39:48.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.589+0000 7fbd91c8b640 10 client.4304.objecter successfully canceled tick 2024-03-27T21:39:48.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.589+0000 7fbd91c8b640 1 -- 172.21.15.60:0/588725866 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fbd7006c590 msgr2=0x7fbd7006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.581 INFO:teuthology.orchestra.run.smithi060.stdout:55834574852 2024-03-27T21:39:48.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.589+0000 7fd3d97fa640 1 -- 172.21.15.60:0/1664087114 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd3d4053a60 con 0x7fd3dc0af1f0 2024-03-27T21:39:48.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.589+0000 7fd3d97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-27T21:39:48.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.589+0000 7fd3d97fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:48.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.589+0000 7fbd91c8b640 1 --2- 172.21.15.60:0/588725866 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fbd7006c590 0x7fbd7006ea50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbd800094a0 tx=0x7fbd80008040 comp rx=0 tx=0).stop 2024-03-27T21:39:48.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.589+0000 7fbd91c8b640 10 monclient: shutdown 2024-03-27T21:39:48.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.589+0000 7fbd91c8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:48.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.589+0000 7fbd91c8b640 1 -- 172.21.15.60:0/588725866 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbd84161300 msgr2=0x7fbd841617b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.589+0000 7fbd91c8b640 1 --2- 172.21.15.60:0/588725866 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbd84161300 0x7fbd841617b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0639b0 tx=0x7fbd8c067040 comp rx=0 tx=0).stop 2024-03-27T21:39:48.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.590+0000 7fbd91c8b640 1 -- 172.21.15.60:0/588725866 shutdown_connections 2024-03-27T21:39:48.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.590+0000 7fbd91c8b640 1 --2- 172.21.15.60:0/588725866 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbd84161300 0x7fbd841617b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.590+0000 7fbd91c8b640 1 --2- 172.21.15.60:0/588725866 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fbd7006c590 0x7fbd7006ea50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.590+0000 7fbd91c8b640 1 --2- 172.21.15.60:0/588725866 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd840c5760 0x7fbd8400a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.590+0000 7fbd91c8b640 1 --2- 172.21.15.60:0/588725866 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd84038d70 0x7fbd840cb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.590+0000 7fbd91c8b640 1 -- 172.21.15.60:0/588725866 >> 172.21.15.60:0/588725866 conn(0x7fbd840065d0 msgr2=0x7fbd840069b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:48.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.590+0000 7fbd91c8b640 1 -- 172.21.15.60:0/588725866 shutdown_connections 2024-03-27T21:39:48.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.590+0000 7fbd91c8b640 1 -- 172.21.15.60:0/588725866 wait complete. 2024-03-27T21:39:48.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.591+0000 7fbd91c8b640 1 librados: shutdown 2024-03-27T21:39:48.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.592+0000 7fd3e2c2f640 10 librados: watch_flush enter 2024-03-27T21:39:48.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.592+0000 7fd3e2c2f640 10 librados: watch_flush exit 2024-03-27T21:39:48.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.592+0000 7fd3e2c2f640 20 client.4369.objecter shutdown clearing up homeless session... 2024-03-27T21:39:48.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.592+0000 7fd3e2c2f640 10 client.4369.objecter successfully canceled tick 2024-03-27T21:39:48.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.592+0000 7fd3e2c2f640 1 -- 172.21.15.60:0/1664087114 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd3c006c650 msgr2=0x7fd3c006eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.592+0000 7fd3e2c2f640 1 --2- 172.21.15.60:0/1664087114 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd3c006c650 0x7fd3c006eb10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd3d002b750 tx=0x7fd3d0008030 comp rx=0 tx=0).stop 2024-03-27T21:39:48.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.592+0000 7fd3e2c2f640 10 monclient: shutdown 2024-03-27T21:39:48.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.592+0000 7fd3e2c2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:48.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fdba80fb640 10 librados: watch_flush enter 2024-03-27T21:39:48.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fdba80fb640 10 librados: watch_flush exit 2024-03-27T21:39:48.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fdba80fb640 20 client.4344.objecter shutdown clearing up homeless session... 2024-03-27T21:39:48.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fdba80fb640 10 client.4344.objecter successfully canceled tick 2024-03-27T21:39:48.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.592+0000 7fd3e2c2f640 1 -- 172.21.15.60:0/1664087114 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3dc0af1f0 msgr2=0x7fd3dc1da650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.592+0000 7fd3e2c2f640 1 --2- 172.21.15.60:0/1664087114 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3dc0af1f0 0x7fd3dc1da650 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd3d400c8d0 tx=0x7fd3d400cd90 comp rx=0 tx=0).stop 2024-03-27T21:39:48.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fd3e2c2f640 1 -- 172.21.15.60:0/1664087114 shutdown_connections 2024-03-27T21:39:48.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fdba80fb640 1 -- 172.21.15.60:0/75719323 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdb8406c5e0 msgr2=0x7fdb8406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fdba80fb640 1 --2- 172.21.15.60:0/75719323 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdb8406c5e0 0x7fdb8406eaa0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdb90001750 tx=0x7fdb90009040 comp rx=0 tx=0).stop 2024-03-27T21:39:48.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fd3e2c2f640 1 --2- 172.21.15.60:0/1664087114 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd3c006c650 0x7fd3c006eb10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fdba80fb640 10 monclient: shutdown 2024-03-27T21:39:48.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fdba80fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:48.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fdba80fb640 1 -- 172.21.15.60:0/75719323 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb980c5060 msgr2=0x7fdb98161780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fdba80fb640 1 --2- 172.21.15.60:0/75719323 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb980c5060 0x7fdb98161780 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdba004f2a0 tx=0x7fdba00aa000 comp rx=0 tx=0).stop 2024-03-27T21:39:48.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fd3e2c2f640 1 --2- 172.21.15.60:0/1664087114 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3dc0af1f0 0x7fd3dc1da650 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fd3e2c2f640 1 --2- 172.21.15.60:0/1664087114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3dc0ad560 0x7fd3dc1da110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fdba80fb640 1 -- 172.21.15.60:0/75719323 shutdown_connections 2024-03-27T21:39:48.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fdba80fb640 1 --2- 172.21.15.60:0/75719323 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdb8406c5e0 0x7fdb8406eaa0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fdba80fb640 1 --2- 172.21.15.60:0/75719323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb9802dc80 0x7fdb981645c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fdba80fb640 1 --2- 172.21.15.60:0/75719323 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb980c5060 0x7fdb98161780 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fdba80fb640 1 --2- 172.21.15.60:0/75719323 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdb980375b0 0x7fdb980c86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fdba80fb640 1 -- 172.21.15.60:0/75719323 >> 172.21.15.60:0/75719323 conn(0x7fdb9803a920 msgr2=0x7fdb98037f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:48.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fd3e2c2f640 1 --2- 172.21.15.60:0/1664087114 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd3dc060970 0x7fd3dc06e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.593+0000 7fd3e2c2f640 1 -- 172.21.15.60:0/1664087114 >> 172.21.15.60:0/1664087114 conn(0x7fd3dc069ce0 msgr2=0x7fd3dc06a0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:48.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.594+0000 7fd3e2c2f640 1 -- 172.21.15.60:0/1664087114 shutdown_connections 2024-03-27T21:39:48.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.594+0000 7fd3e2c2f640 1 -- 172.21.15.60:0/1664087114 wait complete. 2024-03-27T21:39:48.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.594+0000 7fdba80fb640 1 -- 172.21.15.60:0/75719323 shutdown_connections 2024-03-27T21:39:48.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.594+0000 7fdba80fb640 1 -- 172.21.15.60:0/75719323 wait complete. 2024-03-27T21:39:48.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.594+0000 7fd3e2c2f640 1 librados: shutdown 2024-03-27T21:39:48.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.594+0000 7fdba80fb640 1 librados: shutdown 2024-03-27T21:39:48.593 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542147 for osd.2 2024-03-27T21:39:48.593 INFO:teuthology.orchestra.run.smithi060.stdout:60129542147 2024-03-27T21:39:48.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.596+0000 7f4dfb895640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-27T21:39:48.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.596+0000 7f4dfb895640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-27T21:39:48.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.596+0000 7f4dfb895640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:48.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.596+0000 7f4dfb895640 1 -- 172.21.15.60:0/2928743767 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f4dd40055c0 con 0x7f4dec035eb0 2024-03-27T21:39:48.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.597+0000 7f4de9ffb640 1 -- 172.21.15.60:0/2928743767 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4df0034020 con 0x7f4dec035eb0 2024-03-27T21:39:48.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.597+0000 7f4de9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-27T21:39:48.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.597+0000 7f4de9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:48.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.597+0000 7f4d7d3cb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-27T21:39:48.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.597+0000 7f4d7d3cb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-27T21:39:48.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.598+0000 7f4d7d3cb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:48.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.598+0000 7f4d7d3cb640 1 -- 172.21.15.60:0/993249291 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4d580052f0 con 0x7f4d7806f770 2024-03-27T21:39:48.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.599+0000 7f4d5f7fe640 1 -- 172.21.15.60:0/993249291 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4d68057620 con 0x7f4d7806f770 2024-03-27T21:39:48.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.599+0000 7f4d5f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-27T21:39:48.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.599+0000 7f4d5f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:48.595 INFO:teuthology.orchestra.run.smithi060.stdout:60129542147 2024-03-27T21:39:48.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.600+0000 7f4dfb895640 10 librados: watch_flush enter 2024-03-27T21:39:48.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.600+0000 7f4dfb895640 10 librados: watch_flush exit 2024-03-27T21:39:48.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.600+0000 7f4dfb895640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-27T21:39:48.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.600+0000 7f4dfb895640 10 client.4353.objecter successfully canceled tick 2024-03-27T21:39:48.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.600+0000 7f4dfb895640 1 -- 172.21.15.60:0/2928743767 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4dd806c660 msgr2=0x7f4dd806eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.600+0000 7f4dfb895640 1 --2- 172.21.15.60:0/2928743767 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4dd806c660 0x7f4dd806eb20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4dec0bdae0 tx=0x7f4de4009040 comp rx=0 tx=0).stop 2024-03-27T21:39:48.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.600+0000 7f4dfb895640 10 monclient: shutdown 2024-03-27T21:39:48.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.600+0000 7f4dfb895640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:48.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.600+0000 7f4dfb895640 1 -- 172.21.15.60:0/2928743767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dec035eb0 msgr2=0x7f4dec02d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.600+0000 7f4dfb895640 1 --2- 172.21.15.60:0/2928743767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dec035eb0 0x7f4dec02d7d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4df0033000 tx=0x7f4df00046c0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.601+0000 7f4dfb895640 1 -- 172.21.15.60:0/2928743767 shutdown_connections 2024-03-27T21:39:48.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.601+0000 7f4dfb895640 1 --2- 172.21.15.60:0/2928743767 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4dec0bc720 0x7f4dec0bcba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.601+0000 7f4dfb895640 1 --2- 172.21.15.60:0/2928743767 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4dd806c660 0x7f4dd806eb20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.601+0000 7f4dfb895640 1 --2- 172.21.15.60:0/2928743767 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4dec031450 0x7f4dec0318b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.601+0000 7f4dfb895640 1 --2- 172.21.15.60:0/2928743767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dec035eb0 0x7f4dec02d7d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.601+0000 7f4dfb895640 1 -- 172.21.15.60:0/2928743767 >> 172.21.15.60:0/2928743767 conn(0x7f4dec0070f0 msgr2=0x7f4dec03bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:48.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.601+0000 7f4dfb895640 1 -- 172.21.15.60:0/2928743767 shutdown_connections 2024-03-27T21:39:48.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.601+0000 7f4dfb895640 1 -- 172.21.15.60:0/2928743767 wait complete. 2024-03-27T21:39:48.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.601+0000 7f4dfb895640 1 librados: shutdown 2024-03-27T21:39:48.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.602+0000 7f4d7d3cb640 10 librados: watch_flush enter 2024-03-27T21:39:48.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.602+0000 7f4d7d3cb640 10 librados: watch_flush exit 2024-03-27T21:39:48.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.602+0000 7f4d7d3cb640 20 client.4313.objecter shutdown clearing up homeless session... 2024-03-27T21:39:48.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.602+0000 7f4d7d3cb640 10 client.4313.objecter successfully canceled tick 2024-03-27T21:39:48.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.602+0000 7f4d7d3cb640 1 -- 172.21.15.60:0/993249291 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4d6406c660 msgr2=0x7f4d6406eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.602+0000 7f4d7d3cb640 1 --2- 172.21.15.60:0/993249291 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4d6406c660 0x7f4d6406eb20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c036000 tx=0x7f4d6c030000 comp rx=0 tx=0).stop 2024-03-27T21:39:48.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.602+0000 7f4d7d3cb640 10 monclient: shutdown 2024-03-27T21:39:48.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.602+0000 7f4d7d3cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:48.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.602+0000 7f4d7d3cb640 1 -- 172.21.15.60:0/993249291 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d7806f770 msgr2=0x7f4d781e3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.602+0000 7f4d7d3cb640 1 --2- 172.21.15.60:0/993249291 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d7806f770 0x7f4d781e3a10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4d6800c450 tx=0x7f4d6800c910 comp rx=0 tx=0).stop 2024-03-27T21:39:48.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.602+0000 7f4d7d3cb640 1 -- 172.21.15.60:0/993249291 shutdown_connections 2024-03-27T21:39:48.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.602+0000 7f4d7d3cb640 1 --2- 172.21.15.60:0/993249291 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4d6406c660 0x7f4d6406eb20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.602+0000 7f4d7d3cb640 1 --2- 172.21.15.60:0/993249291 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d78143060 0x7f4d780a95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.602+0000 7f4d7d3cb640 1 --2- 172.21.15.60:0/993249291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d7806fb50 0x7f4d781e3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.602+0000 7f4d7d3cb640 1 --2- 172.21.15.60:0/993249291 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d7806f770 0x7f4d781e3a10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.602+0000 7f4d7d3cb640 1 -- 172.21.15.60:0/993249291 >> 172.21.15.60:0/993249291 conn(0x7f4d780b2ea0 msgr2=0x7f4d780ae090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:48.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.605+0000 7f4d7d3cb640 1 -- 172.21.15.60:0/993249291 shutdown_connections 2024-03-27T21:39:48.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.605+0000 7fa804251640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-27T21:39:48.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.605+0000 7fa804251640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-27T21:39:48.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.605+0000 7fa804251640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:48.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.605+0000 7fa804251640 1 -- 172.21.15.60:0/2465738873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa7e40055c0 con 0x7fa7fc141630 2024-03-27T21:39:48.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.605+0000 7f4d7d3cb640 1 -- 172.21.15.60:0/993249291 wait complete. 2024-03-27T21:39:48.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.605+0000 7f4d7d3cb640 1 librados: shutdown 2024-03-27T21:39:48.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.605+0000 7fa7f27fc640 1 -- 172.21.15.60:0/2465738873 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa7ec054210 con 0x7fa7fc141630 2024-03-27T21:39:48.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.605+0000 7fa7f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-27T21:39:48.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.605+0000 7fa7f27fc640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:48.602 INFO:teuthology.orchestra.run.smithi060.stdout:60129542147 2024-03-27T21:39:48.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.608+0000 7fa804251640 10 librados: watch_flush enter 2024-03-27T21:39:48.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.608+0000 7fa804251640 10 librados: watch_flush exit 2024-03-27T21:39:48.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.608+0000 7fa804251640 20 client.4359.objecter shutdown clearing up homeless session... 2024-03-27T21:39:48.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.608+0000 7fa804251640 10 client.4359.objecter successfully canceled tick 2024-03-27T21:39:48.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.608+0000 7fa804251640 1 -- 172.21.15.60:0/2465738873 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa7e006c650 msgr2=0x7fa7e006eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.608+0000 7fa804251640 1 --2- 172.21.15.60:0/2465738873 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa7e006c650 0x7fa7e006eb10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8000f60 tx=0x7fa7f8009040 comp rx=0 tx=0).stop 2024-03-27T21:39:48.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.608+0000 7fa804251640 10 monclient: shutdown 2024-03-27T21:39:48.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.608+0000 7fa804251640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:48.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.608+0000 7fa804251640 1 -- 172.21.15.60:0/2465738873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7fc141630 msgr2=0x7fa7fc14c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:48.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.608+0000 7fa804251640 1 --2- 172.21.15.60:0/2465738873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7fc141630 0x7fa7fc14c170 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec00c750 tx=0x7fa7ec00cc10 comp rx=0 tx=0).stop 2024-03-27T21:39:48.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.608+0000 7fa804251640 1 -- 172.21.15.60:0/2465738873 shutdown_connections 2024-03-27T21:39:48.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.608+0000 7fa804251640 1 --2- 172.21.15.60:0/2465738873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7fc141630 0x7fa7fc14c170 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.608+0000 7fa804251640 1 --2- 172.21.15.60:0/2465738873 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa7e006c650 0x7fa7e006eb10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.608+0000 7fa804251640 1 --2- 172.21.15.60:0/2465738873 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7fc13ef20 0x7fa7fc14bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.608+0000 7fa804251640 1 --2- 172.21.15.60:0/2465738873 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa7fc071a50 0x7fa7fc1e69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:48.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.608+0000 7fa804251640 1 -- 172.21.15.60:0/2465738873 >> 172.21.15.60:0/2465738873 conn(0x7fa7fc0b7300 msgr2=0x7fa7fc13e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:48.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.608+0000 7fa804251640 1 -- 172.21.15.60:0/2465738873 shutdown_connections 2024-03-27T21:39:48.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.609+0000 7fa804251640 1 -- 172.21.15.60:0/2465738873 wait complete. 2024-03-27T21:39:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:48.609+0000 7fa804251640 1 librados: shutdown 2024-03-27T21:39:48.605 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476740 for osd.3 2024-03-27T21:39:48.606 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542147 for osd.4 2024-03-27T21:39:48.606 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.1 2024-03-27T21:39:48.607 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476741 for osd.5 2024-03-27T21:39:48.608 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542147 for osd.7 2024-03-27T21:39:48.609 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542147 for osd.0 2024-03-27T21:39:48.611 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542147 for osd.6 2024-03-27T21:39:49.594 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-27T21:39:49.606 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-27T21:39:49.607 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-27T21:39:49.607 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-27T21:39:49.608 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-27T21:39:49.609 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-27T21:39:49.609 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-27T21:39:49.612 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-27T21:39:49.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.768+0000 7fda1ffff640 10 monclient: get_monmap_and_config 2024-03-27T21:39:49.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.768+0000 7fda1ffff640 10 monclient: build_initial_monmap 2024-03-27T21:39:49.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.768+0000 7fda1ffff640 10 monclient: monmap: 2024-03-27T21:39:49.756 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:49.756 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.756 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:49.770821+0000 2024-03-27T21:39:49.756 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:49.770821+0000 2024-03-27T21:39:49.757 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:49.757 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.757 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:49.757 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:49.757 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:49.757 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:49.757 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:49.757 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.769+0000 7fda1ffff640 1 Processor -- start 2024-03-27T21:39:49.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.769+0000 7fda1ffff640 1 -- start start 2024-03-27T21:39:49.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.769+0000 7fda1ffff640 10 monclient: init 2024-03-27T21:39:49.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.770+0000 7fda1ffff640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:49.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.770+0000 7fda1ffff640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-03-27T21:39:49.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.770+0000 7fda1ffff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fda200b0ec0 0x7fda20142830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.770+0000 7fda1ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fda200b0ec0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:49.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.770+0000 7fda1ffff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fda20142d70 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:49.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.770+0000 7fda1ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fda200b40e0 0x7fda2014ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.770+0000 7fda1ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fda200b40e0 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:49.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.770+0000 7fda1ffff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda2006d290 con 0x7fda20142d70 2024-03-27T21:39:49.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.770+0000 7fda1ffff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.770+0000 7fda1ffff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda200b12a0 con 0x7fda200b40e0 2024-03-27T21:39:49.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.770+0000 7fda1ffff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.770+0000 7fda1ffff640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda200b3390 con 0x7fda200b0ec0 2024-03-27T21:39:49.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.770+0000 7fda1ffff640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:49.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.770+0000 7fda1ffff640 10 monclient(hunting): authenticate will time out at 830.992875s 2024-03-27T21:39:49.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.771+0000 7fda1f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fda200b40e0 0x7fda2014ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.771+0000 7fda1f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fda200b40e0 0x7fda2014ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47910/0 (socket says 172.21.15.60:47910) 2024-03-27T21:39:49.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.771+0000 7fda1f7fe640 1 -- 172.21.15.60:0/154132184 learned_addr learned my addr 172.21.15.60:0/154132184 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:49.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.771+0000 7fda1f7fe640 10 monclient(hunting): get_auth_request con 0x7fda200b40e0 auth_method 0 2024-03-27T21:39:49.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.771+0000 7fda1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.771+0000 7fda1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.771+0000 7fda1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.771+0000 7fda1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.771+0000 7fda1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.771+0000 7fda1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:49.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.771+0000 7fda1effd640 1 --2- 172.21.15.60:0/154132184 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fda200b0ec0 0x7fda20142830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.771+0000 7fda1effd640 10 monclient(hunting): get_auth_request con 0x7fda200b0ec0 auth_method 0 2024-03-27T21:39:49.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.771+0000 7fda1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.771+0000 7fda1effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.771+0000 7fda1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.771+0000 7fda1f7fe640 10 monclient(hunting): handle_auth_done global_id 4362 payload 274 2024-03-27T21:39:49.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.771+0000 7fda1f7fe640 1 -- 172.21.15.60:0/154132184 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fda200b0ec0 msgr2=0x7fda20142830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:49.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.771+0000 7fda1f7fe640 1 --2- 172.21.15.60:0/154132184 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fda200b0ec0 0x7fda20142830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.771+0000 7fda1f7fe640 1 -- 172.21.15.60:0/154132184 >> v1:172.21.15.60:6789/0 conn(0x7fda20142d70 legacy=0x7fda200b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:49.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.772+0000 7fda1f7fe640 10 monclient: _finish_hunting 0 2024-03-27T21:39:49.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.772+0000 7fda1f7fe640 1 monclient: found mon.noname-a 2024-03-27T21:39:49.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.772+0000 7fda1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:49.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.772+0000 7fda1f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:49.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.772+0000 7fda1f7fe640 1 -- 172.21.15.60:0/154132184 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda200b2b60 con 0x7fda200b40e0 2024-03-27T21:39:49.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.772+0000 7fda1effd640 1 --2- 172.21.15.60:0/154132184 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fda200b0ec0 0x7fda20142830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:39:49.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.773+0000 7fda1f7fe640 1 --2- 172.21.15.60:0/154132184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda200b40e0 0x7fda2014ffa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fda18009e80 tx=0x7fda1802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cce5b85cc7e2110 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.774+0000 7fda1dffb640 1 -- 172.21.15.60:0/154132184 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1802ca60 con 0x7fda200b40e0 2024-03-27T21:39:49.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.774+0000 7fda1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.774+0000 7fda1dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:49.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.774+0000 7fda1dffb640 10 monclient: dump: 2024-03-27T21:39:49.766 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.766 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.766 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.766 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.766 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.766 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.766 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.767 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.767 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.767 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.774+0000 7fda1dffb640 10 monclient: _finish_auth 0 2024-03-27T21:39:49.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.774+0000 7fda1dffb640 10 monclient: _check_auth_tickets 2024-03-27T21:39:49.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.774+0000 7fda1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:49.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.774+0000 7fda1ffff640 5 monclient: authenticate success, global_id 4362 2024-03-27T21:39:49.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.774+0000 7fda1ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:49.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.774+0000 7fda1dffb640 1 -- 172.21.15.60:0/154132184 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1802cbc0 con 0x7fda200b40e0 2024-03-27T21:39:49.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.774+0000 7fda1dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:49.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.775+0000 7fda1ffff640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:49.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.775+0000 7fda1ffff640 4 set_mon_vals no callback set 2024-03-27T21:39:49.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.776+0000 7fda1ffff640 10 monclient: shutdown 2024-03-27T21:39:49.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.776+0000 7fda1ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:49.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.776+0000 7fda1ffff640 1 -- 172.21.15.60:0/154132184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda200b40e0 msgr2=0x7fda2014ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.776+0000 7fda1ffff640 1 --2- 172.21.15.60:0/154132184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda200b40e0 0x7fda2014ffa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fda18009e80 tx=0x7fda1802bec0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.776+0000 7fda1ffff640 1 -- 172.21.15.60:0/154132184 shutdown_connections 2024-03-27T21:39:49.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.776+0000 7fda1ffff640 1 --2- 172.21.15.60:0/154132184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda200b40e0 0x7fda2014ffa0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.776+0000 7fda1ffff640 1 --2- 172.21.15.60:0/154132184 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fda200b0ec0 0x7fda20142830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.776+0000 7fda1ffff640 1 -- 172.21.15.60:0/154132184 >> 172.21.15.60:0/154132184 conn(0x7fda20060700 msgr2=0x7fda20060b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:49.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.777+0000 7fda1ffff640 1 -- 172.21.15.60:0/154132184 shutdown_connections 2024-03-27T21:39:49.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.777+0000 7fda1ffff640 1 -- 172.21.15.60:0/154132184 wait complete. 2024-03-27T21:39:49.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.778+0000 7fda1ffff640 10 monclient: build_initial_monmap 2024-03-27T21:39:49.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.778+0000 7fda1ffff640 10 monclient: monmap: 2024-03-27T21:39:49.775 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:49.775 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:49.775 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:49.775 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:49.775 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:49.775 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:49.776 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:49.776 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:49.776 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:49.776 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.778+0000 7fda1ffff640 1 librados: starting msgr at 2024-03-27T21:39:49.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.778+0000 7fda1ffff640 1 librados: starting objecter 2024-03-27T21:39:49.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.779+0000 7fda1ffff640 1 Processor -- start 2024-03-27T21:39:49.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.779+0000 7fda1ffff640 1 -- start start 2024-03-27T21:39:49.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.779+0000 7fda1ffff640 1 librados: setting wanted keys 2024-03-27T21:39:49.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.779+0000 7fda1ffff640 1 librados: calling monclient init 2024-03-27T21:39:49.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.779+0000 7fda1ffff640 10 monclient: init 2024-03-27T21:39:49.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.779+0000 7fda1ffff640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:49.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.780+0000 7fda1ffff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:39:49.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.780+0000 7fda1ffff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fda200b0ec0 0x7fda2008a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.780+0000 7fda1ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fda200b0ec0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:49.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.780+0000 7fda1ffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda20142d70 0x7fda200ac9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.780+0000 7fda1ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fda20142d70 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:49.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.780+0000 7fda1ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda200acee0 0x7fda2012de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.780+0000 7fda1effd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fda200b0ec0 0x7fda2008a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.780+0000 7fda1effd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fda200b0ec0 0x7fda2008a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47738/0 (socket says 172.21.15.60:47738) 2024-03-27T21:39:49.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.781+0000 7fda1f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda200acee0 0x7fda2012de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.781+0000 7fda1f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda200acee0 0x7fda2012de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47924/0 (socket says 172.21.15.60:47924) 2024-03-27T21:39:49.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fda200acee0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:49.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1ffff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1ffff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda200ac2b0 con 0x7fda200acee0 2024-03-27T21:39:49.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1ffff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1ffff640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda200abe20 con 0x7fda20142d70 2024-03-27T21:39:49.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1ffff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1ffff640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda200ae4f0 con 0x7fda200b0ec0 2024-03-27T21:39:49.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1ffff640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:49.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1ffff640 10 monclient(hunting): authenticate will time out at 831.003981s 2024-03-27T21:39:49.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1effd640 1 -- 172.21.15.60:0/2138659475 learned_addr learned my addr 172.21.15.60:0/2138659475 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:49.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1effd640 10 monclient(hunting): get_auth_request con 0x7fda200b0ec0 auth_method 0 2024-03-27T21:39:49.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1f7fe640 10 monclient(hunting): get_auth_request con 0x7fda200acee0 auth_method 0 2024-03-27T21:39:49.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:49.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1f7fe640 10 monclient(hunting): handle_auth_done global_id 4368 payload 824 2024-03-27T21:39:49.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1f7fe640 1 -- 172.21.15.60:0/2138659475 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fda200b0ec0 msgr2=0x7fda2008a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1f7fe640 1 --2- 172.21.15.60:0/2138659475 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fda200b0ec0 0x7fda2008a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1f7fe640 1 -- 172.21.15.60:0/2138659475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda20142d70 msgr2=0x7fda200ac9a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T21:39:49.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1f7fe640 1 --2- 172.21.15.60:0/2138659475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda20142d70 0x7fda200ac9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1f7fe640 10 monclient: _finish_hunting 0 2024-03-27T21:39:49.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1f7fe640 1 monclient: found mon.noname-a 2024-03-27T21:39:49.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:49.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:49.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.782+0000 7fda1f7fe640 1 -- 172.21.15.60:0/2138659475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda18009b30 con 0x7fda200acee0 2024-03-27T21:39:49.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.783+0000 7fda1f7fe640 1 --2- 172.21.15.60:0/2138659475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda200acee0 0x7fda2012de40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fda18032000 tx=0x7fda18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.783+0000 7fda1effd640 1 --2- 172.21.15.60:0/2138659475 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fda200b0ec0 0x7fda2008a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:39:49.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.785+0000 7fda077fe640 10 client.?.objecter ms_handle_connect 0x7fda200acee0 2024-03-27T21:39:49.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.785+0000 7fda077fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:49.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.785+0000 7fda077fe640 1 -- 172.21.15.60:0/2138659475 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1803e450 con 0x7fda200acee0 2024-03-27T21:39:49.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.785+0000 7fda077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.785+0000 7fda077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:49.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.785+0000 7fda077fe640 10 monclient: dump: 2024-03-27T21:39:49.780 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.780 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.780 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.781 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.781 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.781 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.781 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.781 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.781 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.781 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.785+0000 7fda077fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:49.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.785+0000 7fda077fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:49.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.785+0000 7fda077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:49.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.786+0000 7fda1ffff640 5 monclient: authenticate success, global_id 4368 2024-03-27T21:39:49.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.786+0000 7fda1ffff640 10 monclient: _renew_subs 2024-03-27T21:39:49.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.786+0000 7fda1ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:49.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.786+0000 7fda1ffff640 1 -- 172.21.15.60:0/2138659475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda200ab250 con 0x7fda200acee0 2024-03-27T21:39:49.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.786+0000 7fda1ffff640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:49.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.786+0000 7fda1ffff640 10 monclient: _renew_subs 2024-03-27T21:39:49.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.786+0000 7fda1ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:49.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.786+0000 7fda1ffff640 1 -- 172.21.15.60:0/2138659475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda2008abc0 con 0x7fda200acee0 2024-03-27T21:39:49.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.786+0000 7fda1ffff640 1 librados: init done 2024-03-27T21:39:49.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.789+0000 7fda077fe640 1 -- 172.21.15.60:0/2138659475 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1803e5b0 con 0x7fda200acee0 2024-03-27T21:39:49.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.789+0000 7fda077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:49.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.789+0000 7fda077fe640 1 -- 172.21.15.60:0/2138659475 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1803e720 con 0x7fda200acee0 2024-03-27T21:39:49.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.789+0000 7fda077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.789+0000 7fda077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:49.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.789+0000 7fda1cff9640 4 set_mon_vals no callback set 2024-03-27T21:39:49.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.790+0000 7fda077fe640 10 monclient: dump: 2024-03-27T21:39:49.785 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.785 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.785 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.785 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.785 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.785 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.786 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.786 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.786 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.786 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.790+0000 7fda077fe640 1 -- 172.21.15.60:0/2138659475 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fda1803e900 con 0x7fda200acee0 2024-03-27T21:39:49.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.791+0000 7fda077fe640 1 --2- 172.21.15.60:0/2138659475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fda0806c890 0x7fda0806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.791+0000 7fda1effd640 1 --2- 172.21.15.60:0/2138659475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fda0806c890 0x7fda0806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.792+0000 7fda1effd640 10 monclient: get_auth_request con 0x7fda0806c890 auth_method 0 2024-03-27T21:39:49.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.792+0000 7fda1effd640 1 --2- 172.21.15.60:0/2138659475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fda0806c890 0x7fda0806ed50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fda10006d40 tx=0x7fda10008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.793+0000 7fda077fe640 10 client.4368.objecter ms_handle_connect 0x7fda0806c890 2024-03-27T21:39:49.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.793+0000 7fda077fe640 1 -- 172.21.15.60:0/2138659475 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fda180abab0 con 0x7fda200acee0 2024-03-27T21:39:49.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.793+0000 7fda077fe640 10 client.4368.objecter ms_dispatch 0x7fda20000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:49.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.793+0000 7fda077fe640 3 client.4368.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:49.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.793+0000 7fda077fe640 3 client.4368.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:49.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.793+0000 7fda077fe640 20 client.4368.objecter dump_active .. 0 homeless 2024-03-27T21:39:49.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.795+0000 7fda1ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:49.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.795+0000 7fda1ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:49.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.795+0000 7fda1ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:49.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.795+0000 7fda1ffff640 1 -- 172.21.15.60:0/2138659475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda00005040 con 0x7fda200acee0 2024-03-27T21:39:49.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.804+0000 7fda077fe640 1 -- 172.21.15.60:0/2138659475 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fda18077010 con 0x7fda200acee0 2024-03-27T21:39:49.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.804+0000 7fda077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:49.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.804+0000 7fda077fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:49.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.824+0000 7fdddcba9640 10 monclient: get_monmap_and_config 2024-03-27T21:39:49.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.824+0000 7fdddcba9640 10 monclient: build_initial_monmap 2024-03-27T21:39:49.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.824+0000 7fdddcba9640 10 monclient: monmap: 2024-03-27T21:39:49.812 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:49.812 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.812 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:49.826837+0000 2024-03-27T21:39:49.813 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:49.826837+0000 2024-03-27T21:39:49.813 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:49.813 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.813 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:49.813 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:49.813 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:49.813 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:49.813 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:49.813 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.825+0000 7f75c1c18640 10 monclient: get_monmap_and_config 2024-03-27T21:39:49.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.825+0000 7f75c1c18640 10 monclient: build_initial_monmap 2024-03-27T21:39:49.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.825+0000 7f75c1c18640 10 monclient: monmap: 2024-03-27T21:39:49.814 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:49.814 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.814 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:49.827873+0000 2024-03-27T21:39:49.814 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:49.827873+0000 2024-03-27T21:39:49.814 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:49.814 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.814 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:49.814 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:49.815 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:49.815 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:49.815 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:49.815 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.825+0000 7f1e792b6640 10 monclient: get_monmap_and_config 2024-03-27T21:39:49.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.825+0000 7f1e792b6640 10 monclient: build_initial_monmap 2024-03-27T21:39:49.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.826+0000 7f1e792b6640 10 monclient: monmap: 2024-03-27T21:39:49.816 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:49.816 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.816 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:49.827990+0000 2024-03-27T21:39:49.816 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:49.827990+0000 2024-03-27T21:39:49.816 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:49.816 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.816 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:49.816 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:49.816 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:49.816 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:49.816 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:49.816 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.826+0000 7fdddcba9640 1 Processor -- start 2024-03-27T21:39:49.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.826+0000 7efe1359e640 10 monclient: get_monmap_and_config 2024-03-27T21:39:49.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.826+0000 7efe1359e640 10 monclient: build_initial_monmap 2024-03-27T21:39:49.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.826+0000 7efe1359e640 10 monclient: monmap: 2024-03-27T21:39:49.817 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:49.817 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.818 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:49.828612+0000 2024-03-27T21:39:49.818 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:49.828612+0000 2024-03-27T21:39:49.818 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:49.818 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.818 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:49.818 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:49.818 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:49.818 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:49.818 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:49.818 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.826+0000 7f1e792b6640 1 Processor -- start 2024-03-27T21:39:49.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7efe1359e640 1 Processor -- start 2024-03-27T21:39:49.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.826+0000 7fdddcba9640 1 -- start start 2024-03-27T21:39:49.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.826+0000 7fdddcba9640 10 monclient: init 2024-03-27T21:39:49.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7fdddcba9640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:49.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7fdddcba9640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-03-27T21:39:49.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7fdddcba9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fddd80b40b0 0x7fddd80a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7fdddcba9640 10 monclient(hunting): picked mon.noname-c con 0x7fddd80b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:49.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7fdddcba9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fddd813b020 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:49.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7fdddcba9640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fddd813b480 0x7fddd8141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7fb3f2cd2640 10 monclient: get_monmap_and_config 2024-03-27T21:39:49.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7fb3f2cd2640 10 monclient: build_initial_monmap 2024-03-27T21:39:49.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.826+0000 7f1e792b6640 1 -- start start 2024-03-27T21:39:49.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.826+0000 7f1e792b6640 10 monclient: init 2024-03-27T21:39:49.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7f1e792b6640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:49.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7f1e792b6640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-03-27T21:39:49.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7f1e792b6640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e74060970 0x7f1e74142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7f1e792b6640 10 monclient(hunting): picked mon.noname-b con 0x7f1e74060970 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:49.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7f1e792b6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1e74071b30 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:49.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7f1e792b6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1e7406e750 0x7f1e74142f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7f1e792b6640 10 monclient(hunting): picked mon.noname-c con 0x7f1e7406e750 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:49.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7f1e792b6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e741434a0 con 0x7f1e74071b30 2024-03-27T21:39:49.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7f1e792b6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7f1e792b6640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e74060d50 con 0x7f1e7406e750 2024-03-27T21:39:49.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7f1e792b6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7f1e792b6640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e740b03f0 con 0x7f1e74060970 2024-03-27T21:39:49.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7f1e792b6640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:49.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7f1e792b6640 10 monclient(hunting): authenticate will time out at 831.049886s 2024-03-27T21:39:49.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7fb3f2cd2640 10 monclient: monmap: 2024-03-27T21:39:49.823 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:49.823 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.823 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:49.829875+0000 2024-03-27T21:39:49.823 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:49.829875+0000 2024-03-27T21:39:49.823 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:49.824 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.824 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:49.824 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:49.824 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:49.824 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:49.824 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:49.824 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f1e72ffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e74060970 0x7f1e74142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f1e72ffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e74060970 0x7f1e74142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47754/0 (socket says 172.21.15.60:47754) 2024-03-27T21:39:49.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f1e72ffd640 1 -- 172.21.15.60:0/911357422 learned_addr learned my addr 172.21.15.60:0/911357422 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:49.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7efe1359e640 1 -- start start 2024-03-27T21:39:49.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7efe1359e640 10 monclient: init 2024-03-27T21:39:49.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7efe1359e640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:49.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7efe1359e640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-03-27T21:39:49.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f1e72ffd640 10 monclient(hunting): get_auth_request con 0x7f1e74060970 auth_method 0 2024-03-27T21:39:49.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7fa55c4db640 10 monclient: get_monmap_and_config 2024-03-27T21:39:49.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7fa55c4db640 10 monclient: build_initial_monmap 2024-03-27T21:39:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7efe1359e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efe14148f10 0x7efe141492f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7efe1359e640 10 monclient(hunting): picked mon.noname-a con 0x7efe14148f10 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7efe1359e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7efe1406ea80 0x7efe14149830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7efe1359e640 10 monclient(hunting): picked mon.noname-c con 0x7efe1406ea80 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7efe1359e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efe140ad910 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7efe1359e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1406d520 con 0x7efe140ad910 2024-03-27T21:39:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7efe1359e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7efe1359e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe1406d770 con 0x7efe14148f10 2024-03-27T21:39:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7efe1359e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7efe1359e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe1406f420 con 0x7efe1406ea80 2024-03-27T21:39:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7efe1359e640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7efe1359e640 10 monclient(hunting): authenticate will time out at 831.050595s 2024-03-27T21:39:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7f75c1c18640 1 Processor -- start 2024-03-27T21:39:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7f75c1c18640 1 -- start start 2024-03-27T21:39:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.827+0000 7f75c1c18640 10 monclient: init 2024-03-27T21:39:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7fa55c4db640 10 monclient: monmap: 2024-03-27T21:39:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:49.830600+0000 2024-03-27T21:39:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:49.830600+0000 2024-03-27T21:39:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:49.829 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f1e72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f1e72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f1e72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f75c1c18640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:49.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7fb3f2cd2640 1 Processor -- start 2024-03-27T21:39:49.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f1e72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f1e72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f1e72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:49.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f1e72ffd640 10 monclient(hunting): handle_auth_done global_id 4325 payload 274 2024-03-27T21:39:49.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7fdddcba9640 10 monclient(hunting): picked mon.noname-b con 0x7fddd813b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:49.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f1e72ffd640 1 -- 172.21.15.60:0/911357422 >> v2:172.21.15.191:3300/0 conn(0x7f1e7406e750 msgr2=0x7f1e74142f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T21:39:49.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f1e72ffd640 1 --2- 172.21.15.60:0/911357422 >> v2:172.21.15.191:3300/0 conn(0x7f1e7406e750 0x7f1e74142f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7fdddcba9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd80b3830 con 0x7fddd813b020 2024-03-27T21:39:49.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f1e72ffd640 1 -- 172.21.15.60:0/911357422 >> v1:172.21.15.191:6789/0 conn(0x7f1e74071b30 legacy=0x7f1e74148e30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-27T21:39:49.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7efe12d9d640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7efe140ad910 0x7efe14142840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.60:49100/0 (socket says 172.21.15.60:49100) 2024-03-27T21:39:49.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7efe12d9d640 1 -- 172.21.15.60:0/3601352137 learned_addr learned my addr 172.21.15.60:0/3601352137 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:39:49.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7efe11d9b640 1 --2- 172.21.15.60:0/3601352137 >> v2:172.21.15.191:3300/0 conn(0x7efe1406ea80 0x7efe14149830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7efe11d9b640 10 monclient(hunting): get_auth_request con 0x7efe1406ea80 auth_method 0 2024-03-27T21:39:49.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7efe11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7efe11d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7efe11d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7efe1259c640 1 --2- 172.21.15.60:0/3601352137 >> v2:172.21.15.60:3300/0 conn(0x7efe14148f10 0x7efe141492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7efe1259c640 10 monclient(hunting): get_auth_request con 0x7efe14148f10 auth_method 0 2024-03-27T21:39:49.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7efe1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7efe1259c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7efe1259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7efe11d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7efe11d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7efe11d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:49.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7efe1259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7efe1259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7efe1259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:49.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7efe11d9b640 10 monclient(hunting): handle_auth_done global_id 4381 payload 274 2024-03-27T21:39:49.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7efe11d9b640 1 -- 172.21.15.60:0/3601352137 >> v2:172.21.15.60:3300/0 conn(0x7efe14148f10 msgr2=0x7efe141492f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:49.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f1e72ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:39:49.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f1e72ffd640 1 monclient: found mon.noname-b 2024-03-27T21:39:49.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f1e72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:49.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f1e72ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:49.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f1e72ffd640 1 -- 172.21.15.60:0/911357422 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e740affe0 con 0x7f1e74060970 2024-03-27T21:39:49.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7efe1159a640 1 -- 172.21.15.60:0/3601352137 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186274158 0 0) 0x7efe1406d520 con 0x7efe140ad910 2024-03-27T21:39:49.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fb3f2cd2640 1 -- start start 2024-03-27T21:39:49.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fb3f2cd2640 10 monclient: init 2024-03-27T21:39:49.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f2cd2640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:49.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f2cd2640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-03-27T21:39:49.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f2cd2640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb3ec0b40b0 0x7fb3ec0a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f2cd2640 10 monclient(hunting): picked mon.noname-b con 0x7fb3ec0b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:49.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f2cd2640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb3ec13b020 0x7fb3ec13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f2cd2640 10 monclient(hunting): picked mon.noname-c con 0x7fb3ec13b020 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:49.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f2cd2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb3ec1416f0 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:49.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f2cd2640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ec0b3830 con 0x7fb3ec1416f0 2024-03-27T21:39:49.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f2cd2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f2cd2640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3ec0b4490 con 0x7fb3ec13b020 2024-03-27T21:39:49.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fa55c4db640 1 Processor -- start 2024-03-27T21:39:49.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f2cd2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f2cd2640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec142290 con 0x7fb3ec0b40b0 2024-03-27T21:39:49.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f2cd2640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:49.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f2cd2640 10 monclient(hunting): authenticate will time out at 831.052110s 2024-03-27T21:39:49.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7efe11d9b640 1 --2- 172.21.15.60:0/3601352137 >> v2:172.21.15.60:3300/0 conn(0x7efe14148f10 0x7efe141492f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7efe11d9b640 1 -- 172.21.15.60:0/3601352137 >> v1:172.21.15.191:6789/0 conn(0x7efe140ad910 legacy=0x7efe14142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f75c1c18640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-03-27T21:39:49.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7fdddcba9640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7fdddcba9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd813b910 con 0x7fddd80b40b0 2024-03-27T21:39:49.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7fdddcba9640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7fdddcba9640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd80b4490 con 0x7fddd813b480 2024-03-27T21:39:49.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7fdddcba9640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:49.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7fdddcba9640 10 monclient(hunting): authenticate will time out at 831.050386s 2024-03-27T21:39:49.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7fddd5d9b640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fddd813b020 0x7fddd81416f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.60:49084/0 (socket says 172.21.15.60:49084) 2024-03-27T21:39:49.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7fddd5d9b640 1 -- 172.21.15.60:0/1027132564 learned_addr learned my addr 172.21.15.60:0/1027132564 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:39:49.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 1 -- 172.21.15.60:0/1027132564 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018915448 0 0) 0x7fddd80b3830 con 0x7fddd813b020 2024-03-27T21:39:49.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 10 monclient(hunting): _negotiate 2024-03-27T21:39:49.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 10 monclient(hunting): my global_id is 4378 2024-03-27T21:39:49.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 1 -- 172.21.15.60:0/1027132564 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc0003520 con 0x7fddd813b020 2024-03-27T21:39:49.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd6d9d640 1 --2- 172.21.15.60:0/1027132564 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fddd813b480 0x7fddd8141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd6d9d640 10 monclient(hunting): get_auth_request con 0x7fddd813b480 auth_method 0 2024-03-27T21:39:49.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd6d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f0a6e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb3ec0b40b0 0x7fb3ec0a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f0a6e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb3ec0b40b0 0x7fb3ec0a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47780/0 (socket says 172.21.15.60:47780) 2024-03-27T21:39:49.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f0a6e640 1 -- 172.21.15.60:0/3841215020 learned_addr learned my addr 172.21.15.60:0/3841215020 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:49.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f0a6e640 10 monclient(hunting): get_auth_request con 0x7fb3ec0b40b0 auth_method 0 2024-03-27T21:39:49.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f0a6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f0a6e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f0a6e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f0a6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f0a6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f0a6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:49.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:49.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd659c640 1 --2- 172.21.15.60:0/1027132564 >> v2:172.21.15.191:3300/0 conn(0x7fddd80b40b0 0x7fddd80a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 1 -- 172.21.15.60:0/1027132564 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2844326765 0 0) 0x7fddc0003520 con 0x7fddd813b020 2024-03-27T21:39:49.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 1 -- 172.21.15.60:0/1027132564 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fddd813b480 msgr2=0x7fddd8141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:49.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 1 --2- 172.21.15.60:0/1027132564 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fddd813b480 0x7fddd8141e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 1 -- 172.21.15.60:0/1027132564 >> v2:172.21.15.191:3300/0 conn(0x7fddd80b40b0 msgr2=0x7fddd80a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:49.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 1 --2- 172.21.15.60:0/1027132564 >> v2:172.21.15.191:3300/0 conn(0x7fddd80b40b0 0x7fddd80a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 10 monclient: _finish_hunting 0 2024-03-27T21:39:49.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 1 monclient: found mon.noname-c-legacy 2024-03-27T21:39:49.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:49.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.191:6789/0 2024-03-27T21:39:49.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 1 -- 172.21.15.60:0/1027132564 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd80aa520 con 0x7fddd813b020 2024-03-27T21:39:49.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 10 monclient: _finish_auth 0 2024-03-27T21:39:49.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 10 monclient: _check_auth_tickets 2024-03-27T21:39:49.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:49.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 1 -- 172.21.15.60:0/1027132564 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2933500084 0 0) 0x7fddd00019f0 con 0x7fddd813b020 2024-03-27T21:39:49.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-03-27T21:39:49.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 10 monclient: dump: 2024-03-27T21:39:49.844 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.844 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.844 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.844 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.845 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.845 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.845 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.845 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.845 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.845 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] but i'm connected to v1:172.21.15.191:6789/0, reconnecting 2024-03-27T21:39:49.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:49.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 1 -- 172.21.15.60:0/1027132564 >> v1:172.21.15.191:6789/0 conn(0x7fddd813b020 legacy=0x7fddd81416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:39:49.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 1 --2- 172.21.15.60:0/1027132564 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fddd813b480 0x7fddc00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 10 monclient(hunting): picked mon.c con 0x7fddd813b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:49.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 1 --2- 172.21.15.60:0/1027132564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddc0003f20 0x7fddc00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 10 monclient(hunting): picked mon.a con 0x7fddc0003f20 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:49.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 1 --2- 172.21.15.60:0/1027132564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fddc0004d00 0x7fddc0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 10 monclient(hunting): picked mon.b con 0x7fddc0004d00 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:49.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 1 -- 172.21.15.60:0/1027132564 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd80b4490 con 0x7fddc0003f20 2024-03-27T21:39:49.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 1 -- 172.21.15.60:0/1027132564 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc000bb10 con 0x7fddc0004d00 2024-03-27T21:39:49.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 1 -- 172.21.15.60:0/1027132564 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc000bcd0 con 0x7fddd813b480 2024-03-27T21:39:49.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:49.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd559a640 10 monclient(hunting): _finish_auth 0 2024-03-27T21:39:49.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7fddd659c640 1 --2- 172.21.15.60:0/1027132564 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fddd813b480 0x7fddc00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd659c640 10 monclient(hunting): get_auth_request con 0x7fddd813b480 auth_method 0 2024-03-27T21:39:49.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd659c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:39:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd6d9d640 1 --2- 172.21.15.60:0/1027132564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fddc0004d00 0x7fddc0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd6d9d640 10 monclient(hunting): get_auth_request con 0x7fddc0004d00 auth_method 0 2024-03-27T21:39:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd6d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:39:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:39:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:39:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd5d9b640 1 --2- 172.21.15.60:0/1027132564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddc0003f20 0x7fddc00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd5d9b640 10 monclient(hunting): get_auth_request con 0x7fddc0003f20 auth_method 0 2024-03-27T21:39:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:39:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd6d9d640 10 monclient(hunting): handle_auth_done global_id 4378 payload 293 2024-03-27T21:39:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd6d9d640 1 -- 172.21.15.60:0/1027132564 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fddd813b480 msgr2=0x7fddc00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd6d9d640 1 --2- 172.21.15.60:0/1027132564 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fddd813b480 0x7fddc00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd6d9d640 1 -- 172.21.15.60:0/1027132564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddc0003f20 msgr2=0x7fddc00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd6d9d640 1 --2- 172.21.15.60:0/1027132564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddc0003f20 0x7fddc00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd6d9d640 10 monclient: _finish_hunting 0 2024-03-27T21:39:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd6d9d640 1 monclient: found mon.b 2024-03-27T21:39:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:49.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd6d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:49.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd6d9d640 1 -- 172.21.15.60:0/1027132564 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd80aa520 con 0x7fddc0004d00 2024-03-27T21:39:49.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd6d9d640 10 monclient: _finish_auth 0 2024-03-27T21:39:49.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd6d9d640 10 monclient: _check_auth_tickets 2024-03-27T21:39:49.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fddd6d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:49.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fdddcba9640 5 monclient: authenticate success, global_id 4378 2024-03-27T21:39:49.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fdddcba9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:49.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fddd6d9d640 1 --2- 172.21.15.60:0/1027132564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fddc0004d00 0x7fddc0005180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fddcc00fe90 tx=0x7fddc00057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=519d4c9f15aedd46 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7efe11d9b640 10 monclient: _finish_hunting 0 2024-03-27T21:39:49.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7efe11d9b640 1 monclient: found mon.noname-c 2024-03-27T21:39:49.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7efe11d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:49.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7efe11d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:39:49.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7efe11d9b640 1 -- 172.21.15.60:0/3601352137 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1406f700 con 0x7efe1406ea80 2024-03-27T21:39:49.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7efe1159a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:39:49.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7efe1259c640 1 --2- 172.21.15.60:0/3601352137 >> v2:172.21.15.60:3300/0 conn(0x7efe14148f10 0x7efe141492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:49.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7efe11d9b640 1 --2- 172.21.15.60:0/3601352137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe1406ea80 0x7efe14149830 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efe140b0ad0 tx=0x7efe0802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3646954bcecdc6f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fa55c4db640 1 -- start start 2024-03-27T21:39:49.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fa55c4db640 10 monclient: init 2024-03-27T21:39:49.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fa55c4db640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:49.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fa55c4db640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-03-27T21:39:49.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fa55c4db640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa54c038fe0 0x7fa54c0c9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fa55c4db640 10 monclient(hunting): picked mon.noname-c con 0x7fa54c038fe0 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:49.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fa55c4db640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa54c035640 0x7fa54c0cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fa55c4db640 10 monclient(hunting): picked mon.noname-a con 0x7fa54c035640 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:49.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fa55c4db640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa54c0cc600 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:49.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fa55c4db640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54c035bf0 con 0x7fa54c0cc600 2024-03-27T21:39:49.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fa55c4db640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fa55c4db640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa54c0393c0 con 0x7fa54c035640 2024-03-27T21:39:49.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fa55c4db640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fa55c4db640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa54c0373e0 con 0x7fa54c038fe0 2024-03-27T21:39:49.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fa55c4db640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:49.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fa55c4db640 10 monclient(hunting): authenticate will time out at 831.053410s 2024-03-27T21:39:49.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fa55aa78640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fa54c0cc600 0x7fa54c034420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.60:49122/0 (socket says 172.21.15.60:49122) 2024-03-27T21:39:49.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fa55aa78640 1 -- 172.21.15.60:0/1892362006 learned_addr learned my addr 172.21.15.60:0/1892362006 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:39:49.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fa55a277640 1 --2- 172.21.15.60:0/1892362006 >> v2:172.21.15.191:3300/0 conn(0x7fa54c038fe0 0x7fa54c0c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fa55a277640 10 monclient(hunting): get_auth_request con 0x7fa54c038fe0 auth_method 0 2024-03-27T21:39:49.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fa55a277640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fa55a277640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7fa55a277640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f75c1c18640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f75bc06f6c0 0x7f75bc13fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f75c1c18640 10 monclient(hunting): picked mon.noname-c con 0x7f75bc06f6c0 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:49.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f75c1c18640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75bc06faa0 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f75c1c18640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f75bc0af6e0 0x7f75bc0afb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f75c1c18640 10 monclient(hunting): picked mon.noname-a con 0x7f75bc0af6e0 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f75c1c18640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc0af430 con 0x7f75bc06faa0 2024-03-27T21:39:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f75c1c18640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f75c1c18640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75bc140040 con 0x7f75bc0af6e0 2024-03-27T21:39:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f75c1c18640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.828+0000 7f75c1c18640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75bc071350 con 0x7f75bc06f6c0 2024-03-27T21:39:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f75c1c18640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f75c1c18640 10 monclient(hunting): authenticate will time out at 831.050927s 2024-03-27T21:39:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f75c0c16640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f75bc06f6c0 0x7f75bc13fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f75c0c16640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f75bc06f6c0 0x7f75bc13fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:56854/0 (socket says 172.21.15.60:56854) 2024-03-27T21:39:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f75c0c16640 1 -- 172.21.15.60:0/4168057092 learned_addr learned my addr 172.21.15.60:0/4168057092 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f75c0c16640 10 monclient(hunting): get_auth_request con 0x7f75bc06f6c0 auth_method 0 2024-03-27T21:39:49.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f75c0c16640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f75c0c16640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f75c0c16640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f75c1417640 1 --2- 172.21.15.60:0/4168057092 >> v2:172.21.15.60:3300/0 conn(0x7f75bc0af6e0 0x7f75bc0afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f75c1417640 10 monclient(hunting): get_auth_request con 0x7f75bc0af6e0 auth_method 0 2024-03-27T21:39:49.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f75c1417640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fa559a76640 1 --2- 172.21.15.60:0/1892362006 >> v2:172.21.15.60:3300/0 conn(0x7fa54c035640 0x7fa54c0cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fa55a277640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f75c1417640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f75c1417640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f75c0c16640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.829+0000 7f75c0c16640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7f75c0c16640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:49.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7f75c0c16640 10 monclient(hunting): handle_auth_done global_id 4387 payload 274 2024-03-27T21:39:49.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7f75c0c16640 1 -- 172.21.15.60:0/4168057092 >> v2:172.21.15.60:3300/0 conn(0x7f75bc0af6e0 msgr2=0x7f75bc0afb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:49.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7f75c0c16640 1 --2- 172.21.15.60:0/4168057092 >> v2:172.21.15.60:3300/0 conn(0x7f75bc0af6e0 0x7f75bc0afb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7f75c0c16640 1 -- 172.21.15.60:0/4168057092 >> v1:172.21.15.60:6789/0 conn(0x7f75bc06faa0 legacy=0x7f75bc138cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7f75bb7fe640 1 -- 172.21.15.60:0/4168057092 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293904171 0 0) 0x7f75bc0af430 con 0x7f75bc06faa0 2024-03-27T21:39:49.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7f75c0c16640 10 monclient: _finish_hunting 0 2024-03-27T21:39:49.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7f75c0c16640 1 monclient: found mon.noname-c 2024-03-27T21:39:49.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7f75c0c16640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:49.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7f75c0c16640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:39:49.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7f75c0c16640 1 -- 172.21.15.60:0/4168057092 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75bc071630 con 0x7f75bc06f6c0 2024-03-27T21:39:49.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7f75c1417640 1 --2- 172.21.15.60:0/4168057092 >> v2:172.21.15.60:3300/0 conn(0x7f75bc0af6e0 0x7f75bc0afb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:39:49.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.831+0000 7f75bb7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:39:49.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fa55a277640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f0a6e640 10 monclient(hunting): handle_auth_done global_id 4331 payload 274 2024-03-27T21:39:49.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f0a6e640 1 -- 172.21.15.60:0/3841215020 >> v2:172.21.15.191:3300/0 conn(0x7fb3ec13b020 msgr2=0x7fb3ec13b480 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-27T21:39:49.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f0a6e640 1 --2- 172.21.15.60:0/3841215020 >> v2:172.21.15.191:3300/0 conn(0x7fb3ec13b020 0x7fb3ec13b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f0a6e640 1 -- 172.21.15.60:0/3841215020 >> v1:172.21.15.191:6789/0 conn(0x7fb3ec1416f0 legacy=0x7fb3ec141b80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-27T21:39:49.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f0a6e640 10 monclient: _finish_hunting 0 2024-03-27T21:39:49.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f0a6e640 1 monclient: found mon.noname-b 2024-03-27T21:39:49.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fa55a277640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:49.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f0a6e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:49.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f0a6e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:49.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.830+0000 7fb3f0a6e640 1 -- 172.21.15.60:0/3841215020 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ec0aa520 con 0x7fb3ec0b40b0 2024-03-27T21:39:49.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7efe1159a640 1 -- 172.21.15.60:0/3601352137 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe08031640 con 0x7efe1406ea80 2024-03-27T21:39:49.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7efe1159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7efe1159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:49.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7efe1159a640 10 monclient: dump: 2024-03-27T21:39:49.859 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.859 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.859 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.859 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.859 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.859 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.859 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.859 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.859 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.859 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7efe1159a640 10 monclient: _finish_auth 0 2024-03-27T21:39:49.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7efe1159a640 10 monclient: _check_auth_tickets 2024-03-27T21:39:49.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7efe1159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:49.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7efe1359e640 5 monclient: authenticate success, global_id 4381 2024-03-27T21:39:49.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7efe1359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:49.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fa559275640 1 -- 172.21.15.60:0/1892362006 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785867844 0 0) 0x7fa54c035bf0 con 0x7fa54c0cc600 2024-03-27T21:39:49.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fa559275640 10 monclient(hunting): _negotiate 2024-03-27T21:39:49.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7efe1159a640 1 -- 172.21.15.60:0/3601352137 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe080317a0 con 0x7efe1406ea80 2024-03-27T21:39:49.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7efe1159a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:49.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fa559275640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fa559275640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fa559275640 10 monclient(hunting): my global_id is 4393 2024-03-27T21:39:49.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fa559275640 1 -- 172.21.15.60:0/1892362006 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa540003520 con 0x7fa54c0cc600 2024-03-27T21:39:49.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fb3f0a6e640 1 --2- 172.21.15.60:0/3841215020 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb3ec0b40b0 0x7fb3ec0a8b90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0009870 tx=0x7fb3e002b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f836468f05cf6313 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7f75c0c16640 1 --2- 172.21.15.60:0/4168057092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75bc06f6c0 0x7f75bc13fb00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f75bc0724b0 tx=0x7f75b003c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e51070c42078a1c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fa55a277640 10 monclient(hunting): handle_auth_done global_id 4390 payload 274 2024-03-27T21:39:49.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fa55a277640 1 -- 172.21.15.60:0/1892362006 >> v2:172.21.15.60:3300/0 conn(0x7fa54c035640 msgr2=0x7fa54c0cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:49.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fa55a277640 1 --2- 172.21.15.60:0/1892362006 >> v2:172.21.15.60:3300/0 conn(0x7fa54c035640 0x7fa54c0cc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fa55a277640 1 -- 172.21.15.60:0/1892362006 >> v1:172.21.15.191:6789/0 conn(0x7fa54c0cc600 legacy=0x7fa54c034420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fa55a277640 10 monclient: _finish_hunting 0 2024-03-27T21:39:49.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fa55a277640 1 monclient: found mon.noname-c 2024-03-27T21:39:49.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fa55a277640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:49.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fa55a277640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:39:49.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fa55a277640 1 -- 172.21.15.60:0/1892362006 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54c037650 con 0x7fa54c038fe0 2024-03-27T21:39:49.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fddd559a640 1 -- 172.21.15.60:0/1027132564 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc057ca0 con 0x7fddc0004d00 2024-03-27T21:39:49.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fddd559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fddd559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:49.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fddd559a640 10 monclient: dump: 2024-03-27T21:39:49.864 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.864 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.864 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.864 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.865 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.865 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.865 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.865 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.865 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.865 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fddd559a640 1 -- 172.21.15.60:0/1027132564 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc057e00 con 0x7fddc0004d00 2024-03-27T21:39:49.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fddd559a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:49.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fddd559a640 1 -- 172.21.15.60:0/1027132564 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc05f680 con 0x7fddc0004d00 2024-03-27T21:39:49.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fddd559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fddd559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:49.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fddd559a640 10 monclient: dump: 2024-03-27T21:39:49.865 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.866 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.866 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.866 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.866 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.866 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.866 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.866 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.866 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.866 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fdddcba9640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:49.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fdddcba9640 4 set_mon_vals no callback set 2024-03-27T21:39:49.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7f1e72ffd640 1 --2- 172.21.15.60:0/911357422 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e74060970 0x7f1e74142a20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1e64009e50 tx=0x7f1e6402bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c32ac0fdcb20da75 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fa559a76640 10 monclient: get_auth_request con 0x7fa54c035640 auth_method 0 2024-03-27T21:39:49.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7f1e71ffb640 1 -- 172.21.15.60:0/911357422 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6402c980 con 0x7f1e74060970 2024-03-27T21:39:49.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f1e71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7efe1359e640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:49.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7efe1359e640 4 set_mon_vals no callback set 2024-03-27T21:39:49.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7efe1159a640 1 -- 172.21.15.60:0/3601352137 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe08031b70 con 0x7efe1406ea80 2024-03-27T21:39:49.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7efe1159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7efe1159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:49.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f1e71ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:49.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7efe1159a640 10 monclient: dump: 2024-03-27T21:39:49.868 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.869 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.869 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.869 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.869 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.869 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.869 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.869 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.869 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.869 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f1e71ffb640 10 monclient: dump: 2024-03-27T21:39:49.870 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.870 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.870 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.870 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.870 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.870 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.870 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.871 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.871 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.871 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f1e71ffb640 10 monclient: _finish_auth 0 2024-03-27T21:39:49.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f1e71ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:39:49.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f1e71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:49.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f1e71ffb640 1 -- 172.21.15.60:0/911357422 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e6402cae0 con 0x7f1e74060970 2024-03-27T21:39:49.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f1e792b6640 5 monclient: authenticate success, global_id 4325 2024-03-27T21:39:49.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f1e792b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:49.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f1e71ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:49.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f1e71ffb640 1 -- 172.21.15.60:0/911357422 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6402ce70 con 0x7f1e74060970 2024-03-27T21:39:49.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7fa559a76640 1 --2- 172.21.15.60:0/1892362006 >> v2:172.21.15.60:3300/0 conn(0x7fa54c035640 0x7fa54c0cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:39:49.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7efe1359e640 10 monclient: shutdown 2024-03-27T21:39:49.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7efe1359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:49.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7efe1359e640 1 -- 172.21.15.60:0/3601352137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe1406ea80 msgr2=0x7efe14149830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7efe1359e640 1 --2- 172.21.15.60:0/3601352137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe1406ea80 0x7efe14149830 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efe140b0ad0 tx=0x7efe0802b8f0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7efe1359e640 1 -- 172.21.15.60:0/3601352137 shutdown_connections 2024-03-27T21:39:49.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7efe1359e640 1 --2- 172.21.15.60:0/3601352137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe1406ea80 0x7efe14149830 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7efe1359e640 1 --2- 172.21.15.60:0/3601352137 >> v2:172.21.15.60:3300/0 conn(0x7efe14148f10 0x7efe141492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7efe1359e640 1 -- 172.21.15.60:0/3601352137 >> 172.21.15.60:0/3601352137 conn(0x7efe140b30f0 msgr2=0x7efe140b3500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fb3eb7fe640 1 -- 172.21.15.60:0/3841215020 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e0031870 con 0x7fb3ec0b40b0 2024-03-27T21:39:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fb3eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fb3eb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fb3eb7fe640 10 monclient: dump: 2024-03-27T21:39:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.874 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.874 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.874 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.874 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.874 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.874 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.874 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7efe1359e640 1 -- 172.21.15.60:0/3601352137 shutdown_connections 2024-03-27T21:39:49.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fb3eb7fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:49.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fb3eb7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:49.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fb3eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:49.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fb3eb7fe640 1 -- 172.21.15.60:0/3841215020 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e00319d0 con 0x7fb3ec0b40b0 2024-03-27T21:39:49.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fb3eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:49.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fb3eb7fe640 1 -- 172.21.15.60:0/3841215020 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e0031d60 con 0x7fb3ec0b40b0 2024-03-27T21:39:49.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fb3eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fb3eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7f75bb7fe640 1 -- 172.21.15.60:0/4168057092 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b00112d0 con 0x7f75bc06f6c0 2024-03-27T21:39:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7f75bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7f75bb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7f75bb7fe640 10 monclient: dump: 2024-03-27T21:39:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.877 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7f75bb7fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7f75bb7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fb3eb7fe640 10 monclient: dump: 2024-03-27T21:39:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.878 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7f75bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7f75c1c18640 5 monclient: authenticate success, global_id 4387 2024-03-27T21:39:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7f75c1c18640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7f75bb7fe640 1 -- 172.21.15.60:0/4168057092 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b0011430 con 0x7f75bc06f6c0 2024-03-27T21:39:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7f75bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7f75c1c18640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.832+0000 7f75c1c18640 4 set_mon_vals no callback set 2024-03-27T21:39:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f75c1c18640 10 monclient: shutdown 2024-03-27T21:39:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f75c1c18640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f75c1c18640 1 -- 172.21.15.60:0/4168057092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75bc06f6c0 msgr2=0x7f75bc13fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f75c1c18640 1 --2- 172.21.15.60:0/4168057092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75bc06f6c0 0x7f75bc13fb00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f75bc0724b0 tx=0x7f75b003c960 comp rx=0 tx=0).stop 2024-03-27T21:39:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f2f78fd5640 10 monclient: get_monmap_and_config 2024-03-27T21:39:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f2f78fd5640 10 monclient: build_initial_monmap 2024-03-27T21:39:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f2f78fd5640 10 monclient: monmap: 2024-03-27T21:39:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.880 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:49.835787+0000 2024-03-27T21:39:49.880 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:49.835787+0000 2024-03-27T21:39:49.880 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:49.880 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.880 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:49.880 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:49.880 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:49.880 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:49.880 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:49.880 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fdddcba9640 10 monclient: shutdown 2024-03-27T21:39:49.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fdddcba9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:49.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fdddcba9640 1 -- 172.21.15.60:0/1027132564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fddc0004d00 msgr2=0x7fddc0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fdddcba9640 1 --2- 172.21.15.60:0/1027132564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fddc0004d00 0x7fddc0005180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fddcc00fe90 tx=0x7fddc00057f0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fb3f2cd2640 5 monclient: authenticate success, global_id 4331 2024-03-27T21:39:49.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fb3f2cd2640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:49.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fb3f2cd2640 4 set_mon_vals no callback set 2024-03-27T21:39:49.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fddd6d9d640 1 -- 172.21.15.60:0/1027132564 reap_dead start 2024-03-27T21:39:49.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7fa55a277640 1 --2- 172.21.15.60:0/1892362006 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa54c038fe0 0x7fa54c0c9640 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa54c037950 tx=0x7fa55002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d47d1a3c7099f570 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7efe1359e640 1 -- 172.21.15.60:0/3601352137 wait complete. 2024-03-27T21:39:49.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7f75c1c18640 1 -- 172.21.15.60:0/4168057092 shutdown_connections 2024-03-27T21:39:49.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7f75c1c18640 1 --2- 172.21.15.60:0/4168057092 >> v2:172.21.15.60:3300/0 conn(0x7f75bc0af6e0 0x7f75bc0afb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7f75c1c18640 1 --2- 172.21.15.60:0/4168057092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75bc06f6c0 0x7f75bc13fb00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7f75c1c18640 1 -- 172.21.15.60:0/4168057092 >> 172.21.15.60:0/4168057092 conn(0x7f75bc0b4ce0 msgr2=0x7f75bc0b50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:49.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f1e792b6640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:49.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f1e792b6640 4 set_mon_vals no callback set 2024-03-27T21:39:49.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f1e71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f1e71ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:49.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.833+0000 7f1e71ffb640 10 monclient: dump: 2024-03-27T21:39:49.883 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.883 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.883 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.883 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.883 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.883 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.884 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.884 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.884 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.884 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7f1e792b6640 10 monclient: shutdown 2024-03-27T21:39:49.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7f1e792b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:49.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7f1e792b6640 1 -- 172.21.15.60:0/911357422 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e74060970 msgr2=0x7f1e74142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7f1e792b6640 1 --2- 172.21.15.60:0/911357422 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e74060970 0x7f1e74142a20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1e64009e50 tx=0x7f1e6402bde0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fa559275640 1 -- 172.21.15.60:0/1892362006 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55002e070 con 0x7fa54c038fe0 2024-03-27T21:39:49.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fa559275640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fa559275640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:49.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fa559275640 10 monclient: dump: 2024-03-27T21:39:49.885 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.885 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.885 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.885 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.885 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.885 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.886 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.886 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.886 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.886 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fa559275640 10 monclient: _finish_auth 0 2024-03-27T21:39:49.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7f1e792b6640 1 -- 172.21.15.60:0/911357422 shutdown_connections 2024-03-27T21:39:49.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7f1e792b6640 1 --2- 172.21.15.60:0/911357422 >> v2:172.21.15.191:3300/0 conn(0x7f1e7406e750 0x7f1e74142f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7f1e792b6640 1 --2- 172.21.15.60:0/911357422 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e74060970 0x7f1e74142a20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7f1e792b6640 1 -- 172.21.15.60:0/911357422 >> 172.21.15.60:0/911357422 conn(0x7f1e740b3d60 msgr2=0x7f1e740b4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fa559275640 10 monclient: _check_auth_tickets 2024-03-27T21:39:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fa559275640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fa55c4db640 5 monclient: authenticate success, global_id 4390 2024-03-27T21:39:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fa55c4db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fa559275640 1 -- 172.21.15.60:0/1892362006 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5500315a0 con 0x7fa54c038fe0 2024-03-27T21:39:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fa559275640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fa55c4db640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fa55c4db640 4 set_mon_vals no callback set 2024-03-27T21:39:49.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fdddcba9640 1 -- 172.21.15.60:0/1027132564 shutdown_connections 2024-03-27T21:39:49.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fdddcba9640 1 -- 172.21.15.60:0/1027132564 >> 172.21.15.60:0/1027132564 conn(0x7fddd80608e0 msgr2=0x7fddd8060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:49.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fb3f2cd2640 10 monclient: shutdown 2024-03-27T21:39:49.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fb3f2cd2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:49.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fb3f2cd2640 1 -- 172.21.15.60:0/3841215020 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb3ec0b40b0 msgr2=0x7fb3ec0a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fb3f2cd2640 1 --2- 172.21.15.60:0/3841215020 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb3ec0b40b0 0x7fb3ec0a8b90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0009870 tx=0x7fb3e002b910 comp rx=0 tx=0).stop 2024-03-27T21:39:49.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7f1e792b6640 1 -- 172.21.15.60:0/911357422 shutdown_connections 2024-03-27T21:39:49.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7f2f78fd5640 1 Processor -- start 2024-03-27T21:39:49.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fb3f2cd2640 1 -- 172.21.15.60:0/3841215020 shutdown_connections 2024-03-27T21:39:49.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fb3f2cd2640 1 --2- 172.21.15.60:0/3841215020 >> v2:172.21.15.191:3300/0 conn(0x7fb3ec13b020 0x7fb3ec13b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fb3f2cd2640 1 --2- 172.21.15.60:0/3841215020 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb3ec0b40b0 0x7fb3ec0a8b90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fb3f2cd2640 1 -- 172.21.15.60:0/3841215020 >> 172.21.15.60:0/3841215020 conn(0x7fb3ec0608e0 msgr2=0x7fb3ec060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:49.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fa559275640 1 -- 172.21.15.60:0/1892362006 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa550031930 con 0x7fa54c038fe0 2024-03-27T21:39:49.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7fa559275640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7fa559275640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:49.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7fa559275640 10 monclient: dump: 2024-03-27T21:39:49.890 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.890 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.890 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.891 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7f75c1c18640 1 -- 172.21.15.60:0/4168057092 shutdown_connections 2024-03-27T21:39:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7f75c1c18640 1 -- 172.21.15.60:0/4168057092 wait complete. 2024-03-27T21:39:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7fdddcba9640 1 -- 172.21.15.60:0/1027132564 shutdown_connections 2024-03-27T21:39:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7efe1359e640 10 monclient: build_initial_monmap 2024-03-27T21:39:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7fa55c4db640 10 monclient: shutdown 2024-03-27T21:39:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7fa55c4db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7fa55c4db640 1 -- 172.21.15.60:0/1892362006 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa54c038fe0 msgr2=0x7fa54c0c9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7fa55c4db640 1 --2- 172.21.15.60:0/1892362006 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa54c038fe0 0x7fa54c0c9640 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa54c037950 tx=0x7fa55002b840 comp rx=0 tx=0).stop 2024-03-27T21:39:49.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7f2f78fd5640 1 -- start start 2024-03-27T21:39:49.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7f2f78fd5640 10 monclient: init 2024-03-27T21:39:49.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7efe1359e640 10 monclient: monmap: 2024-03-27T21:39:49.893 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:49.893 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:49.893 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:49.893 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:49.893 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:49.893 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:49.894 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:49.894 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:49.894 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:49.894 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7fa55c4db640 1 -- 172.21.15.60:0/1892362006 shutdown_connections 2024-03-27T21:39:49.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7fa55c4db640 1 --2- 172.21.15.60:0/1892362006 >> v2:172.21.15.60:3300/0 conn(0x7fa54c035640 0x7fa54c0cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7fa55c4db640 1 --2- 172.21.15.60:0/1892362006 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa54c038fe0 0x7fa54c0c9640 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7fa55c4db640 1 -- 172.21.15.60:0/1892362006 >> 172.21.15.60:0/1892362006 conn(0x7fa54c0065d0 msgr2=0x7fa54c0069e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:49.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7fa55c4db640 1 -- 172.21.15.60:0/1892362006 shutdown_connections 2024-03-27T21:39:49.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7fa55c4db640 1 -- 172.21.15.60:0/1892362006 wait complete. 2024-03-27T21:39:49.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7efe1359e640 1 librados: starting msgr at 2024-03-27T21:39:49.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7efe1359e640 1 librados: starting objecter 2024-03-27T21:39:49.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.834+0000 7f1e792b6640 1 -- 172.21.15.60:0/911357422 wait complete. 2024-03-27T21:39:49.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7efe1359e640 1 Processor -- start 2024-03-27T21:39:49.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7efe1359e640 1 -- start start 2024-03-27T21:39:49.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7efe1359e640 1 librados: setting wanted keys 2024-03-27T21:39:49.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7efe1359e640 1 librados: calling monclient init 2024-03-27T21:39:49.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7f1e792b6640 10 monclient: build_initial_monmap 2024-03-27T21:39:49.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7f1e792b6640 10 monclient: monmap: 2024-03-27T21:39:49.897 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:49.897 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:49.897 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:49.897 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:49.897 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:49.897 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:49.897 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:49.897 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:49.897 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:49.897 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7efe1359e640 10 monclient: init 2024-03-27T21:39:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7f75c1c18640 10 monclient: build_initial_monmap 2024-03-27T21:39:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7f75c1c18640 10 monclient: monmap: 2024-03-27T21:39:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:49.899 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:49.899 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:49.899 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:49.899 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:49.899 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.835+0000 7fdddcba9640 1 -- 172.21.15.60:0/1027132564 wait complete. 2024-03-27T21:39:49.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f2f78fd5640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:49.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f2f78fd5640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-03-27T21:39:49.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f2f78fd5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2f7406a4d0 0x7f2f740716b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f2f78fd5640 10 monclient(hunting): picked mon.noname-a con 0x7f2f7406a4d0 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:49.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f75c1c18640 1 librados: starting msgr at 2024-03-27T21:39:49.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f75c1c18640 1 librados: starting objecter 2024-03-27T21:39:49.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f75c1c18640 1 Processor -- start 2024-03-27T21:39:49.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7fa55c4db640 10 monclient: build_initial_monmap 2024-03-27T21:39:49.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7fa55c4db640 10 monclient: monmap: 2024-03-27T21:39:49.901 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:49.901 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:49.901 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:49.901 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:49.901 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:49.901 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:49.901 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:49.901 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:49.901 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:49.901 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f2f78fd5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2f74071bf0 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:49.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f2f78fd5640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2f7406a8b0 0x7f2f74141cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f1e792b6640 1 librados: starting msgr at 2024-03-27T21:39:49.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f1e792b6640 1 librados: starting objecter 2024-03-27T21:39:49.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f2f78fd5640 10 monclient(hunting): picked mon.noname-b con 0x7f2f7406a8b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:49.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7efe1359e640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:49.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f2f78fd5640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f7406f660 con 0x7f2f74071bf0 2024-03-27T21:39:49.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7efe1359e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:39:49.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f2f78fd5640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7efe1359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe1406ea80 0x7efe14145e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f2f78fd5640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f7413ede0 con 0x7f2f7406a4d0 2024-03-27T21:39:49.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f2f78fd5640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f2f78fd5640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7406f940 con 0x7f2f7406a8b0 2024-03-27T21:39:49.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f2f78fd5640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:49.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f2f78fd5640 10 monclient(hunting): authenticate will time out at 831.058538s 2024-03-27T21:39:49.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7fdddcba9640 10 monclient: build_initial_monmap 2024-03-27T21:39:49.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7fdddcba9640 10 monclient: monmap: 2024-03-27T21:39:49.905 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:49.905 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:49.905 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:49.905 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:49.905 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:49.905 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:49.905 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:49.906 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:49.906 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:49.906 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f1e792b6640 1 Processor -- start 2024-03-27T21:39:49.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f2f72d9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2f7406a8b0 0x7f2f74141cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f2f72d9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2f7406a8b0 0x7f2f74141cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47784/0 (socket says 172.21.15.60:47784) 2024-03-27T21:39:49.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f2f72d9d640 1 -- 172.21.15.60:0/2143680276 learned_addr learned my addr 172.21.15.60:0/2143680276 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:49.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f2f72d9d640 10 monclient(hunting): get_auth_request con 0x7f2f7406a8b0 auth_method 0 2024-03-27T21:39:49.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f2f72d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f2f72d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f2f72d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f1e792b6640 1 -- start start 2024-03-27T21:39:49.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f1e792b6640 1 librados: setting wanted keys 2024-03-27T21:39:49.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f1e792b6640 1 librados: calling monclient init 2024-03-27T21:39:49.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f75c1c18640 1 -- start start 2024-03-27T21:39:49.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f75c1c18640 1 librados: setting wanted keys 2024-03-27T21:39:49.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f75c1c18640 1 librados: calling monclient init 2024-03-27T21:39:49.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f75c1c18640 10 monclient: init 2024-03-27T21:39:49.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7f1e792b6640 10 monclient: init 2024-03-27T21:39:49.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7fa55c4db640 1 librados: starting msgr at 2024-03-27T21:39:49.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7fa55c4db640 1 librados: starting objecter 2024-03-27T21:39:49.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7efe1359e640 10 monclient(hunting): picked mon.noname-a con 0x7efe1406ea80 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:49.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7efe1359e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe140ad910 0x7efe14143520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7efe1359e640 10 monclient(hunting): picked mon.noname-b con 0x7efe140ad910 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7efe1359e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efe14148f10 0x7efe14143a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7efe1359e640 10 monclient(hunting): picked mon.noname-c con 0x7efe14148f10 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7efe1359e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7efe1359e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe14143fa0 con 0x7efe1406ea80 2024-03-27T21:39:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7efe1359e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7efe1359e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe141417b0 con 0x7efe140ad910 2024-03-27T21:39:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7efe1359e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7efe1359e640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe140acf30 con 0x7efe14148f10 2024-03-27T21:39:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7efe1359e640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7efe1359e640 10 monclient(hunting): authenticate will time out at 831.058711s 2024-03-27T21:39:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7efe1259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe1406ea80 0x7efe14145e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7fa55c4db640 1 Processor -- start 2024-03-27T21:39:49.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7fa55c4db640 1 -- start start 2024-03-27T21:39:49.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7fa55c4db640 1 librados: setting wanted keys 2024-03-27T21:39:49.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7fa55c4db640 1 librados: calling monclient init 2024-03-27T21:39:49.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.836+0000 7fa55c4db640 10 monclient: init 2024-03-27T21:39:49.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f2f7259c640 1 --2- 172.21.15.60:0/2143680276 >> v2:172.21.15.60:3300/0 conn(0x7f2f7406a4d0 0x7f2f740716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f2f7259c640 10 monclient(hunting): get_auth_request con 0x7f2f7406a4d0 auth_method 0 2024-03-27T21:39:49.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f2f7259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f2f7259c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f2f7259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f2f72d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f2f72d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f2f72d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:49.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f2f7259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f2f7259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f2f7259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:49.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7efe12d9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efe14148f10 0x7efe14143a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7efe1259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe1406ea80 0x7efe14145e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47978/0 (socket says 172.21.15.60:47978) 2024-03-27T21:39:49.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7efe1259c640 1 -- 172.21.15.60:0/1559011480 learned_addr learned my addr 172.21.15.60:0/1559011480 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:49.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7efe1259c640 10 monclient(hunting): get_auth_request con 0x7efe1406ea80 auth_method 0 2024-03-27T21:39:49.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7efe1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7efe1259c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7efe1259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f75c1c18640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f75c1c18640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:39:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f75c1c18640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75bc06f6c0 0x7f75bc0ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f75c1c18640 10 monclient(hunting): picked mon.noname-b con 0x7f75bc06f6c0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f75c1c18640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f75bc06faa0 0x7f75bc1d1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f75c1c18640 10 monclient(hunting): picked mon.noname-c con 0x7f75bc06faa0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f75c1c18640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75bc0af6e0 0x7f75bc1d2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f75c1c18640 10 monclient(hunting): picked mon.noname-a con 0x7f75bc0af6e0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f75c1c18640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f75c1c18640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc1d2a80 con 0x7f75bc0af6e0 2024-03-27T21:39:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f75c1c18640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f75c1c18640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc0ae9e0 con 0x7f75bc06f6c0 2024-03-27T21:39:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f2f72d9d640 10 monclient(hunting): handle_auth_done global_id 4334 payload 274 2024-03-27T21:39:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f2f72d9d640 1 -- 172.21.15.60:0/2143680276 >> v2:172.21.15.60:3300/0 conn(0x7f2f7406a4d0 msgr2=0x7f2f740716b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f2f72d9d640 1 --2- 172.21.15.60:0/2143680276 >> v2:172.21.15.60:3300/0 conn(0x7f2f7406a4d0 0x7f2f740716b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f2f72d9d640 1 -- 172.21.15.60:0/2143680276 >> v1:172.21.15.191:6789/0 conn(0x7f2f74071bf0 legacy=0x7f2f741415e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T21:39:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f75c1c18640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f75c1c18640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc0abc40 con 0x7f75bc06faa0 2024-03-27T21:39:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f75c1c18640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f75c1c18640 10 monclient(hunting): authenticate will time out at 831.059485s 2024-03-27T21:39:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7efe12d9d640 10 monclient(hunting): get_auth_request con 0x7efe14148f10 auth_method 0 2024-03-27T21:39:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7efe12d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7efe12d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7efe12d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f2f72d9d640 10 monclient: _finish_hunting 0 2024-03-27T21:39:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f2f72d9d640 1 monclient: found mon.noname-b 2024-03-27T21:39:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f2f72d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:49.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f2f72d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:49.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f2f72d9d640 1 -- 172.21.15.60:0/2143680276 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7406fc20 con 0x7f2f7406a8b0 2024-03-27T21:39:49.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7fb3f2cd2640 1 -- 172.21.15.60:0/3841215020 shutdown_connections 2024-03-27T21:39:49.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f1e792b6640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:49.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7f1e792b6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:39:49.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e792b6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e74060970 0x7f1e741456e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e792b6640 10 monclient(hunting): picked mon.noname-b con 0x7f1e74060970 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:49.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e792b6640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e7406e750 0x7f1e740a8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e72ffd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e74060970 0x7f1e741456e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e72ffd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e74060970 0x7f1e741456e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:56894/0 (socket says 172.21.15.60:56894) 2024-03-27T21:39:49.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e792b6640 10 monclient(hunting): picked mon.noname-c con 0x7f1e7406e750 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:49.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e72ffd640 1 -- 172.21.15.60:0/3714157888 learned_addr learned my addr 172.21.15.60:0/3714157888 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:49.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e792b6640 1 --2- 172.21.15.60:0/3714157888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e74071b30 0x7f1e740a8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e792b6640 10 monclient(hunting): picked mon.noname-a con 0x7f1e74071b30 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:49.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f75bbfff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f75bc06faa0 0x7f75bc1d1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e727fc640 1 --2- 172.21.15.60:0/3714157888 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e7406e750 0x7f1e740a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e792b6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e792b6640 1 -- 172.21.15.60:0/3714157888 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e741462c0 con 0x7f1e74071b30 2024-03-27T21:39:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e792b6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e792b6640 1 -- 172.21.15.60:0/3714157888 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e740aa3e0 con 0x7f1e74060970 2024-03-27T21:39:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e792b6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e792b6640 1 -- 172.21.15.60:0/3714157888 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e740aa6c0 con 0x7f1e7406e750 2024-03-27T21:39:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e792b6640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e792b6640 10 monclient(hunting): authenticate will time out at 831.060654s 2024-03-27T21:39:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7fb3f2cd2640 1 -- 172.21.15.60:0/3841215020 wait complete. 2024-03-27T21:39:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fb3f2cd2640 10 monclient: build_initial_monmap 2024-03-27T21:39:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fb3f2cd2640 10 monclient: monmap: 2024-03-27T21:39:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:49.921 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7fa55c4db640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:49.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7fa55c4db640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:39:49.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7fa55c4db640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa54c035640 0x7fa54c0c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7fa55c4db640 10 monclient(hunting): picked mon.noname-b con 0x7fa54c035640 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:49.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7fa55c4db640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa54c038fe0 0x7fa54c0beef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa559a76640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa54c038fe0 0x7fa54c0beef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa559a76640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa54c038fe0 0x7fa54c0beef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47796/0 (socket says 172.21.15.60:47796) 2024-03-27T21:39:49.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa559a76640 1 -- 172.21.15.60:0/3119910424 learned_addr learned my addr 172.21.15.60:0/3119910424 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:49.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa55c4db640 10 monclient(hunting): picked mon.noname-c con 0x7fa54c038fe0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:49.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa55c4db640 1 --2- 172.21.15.60:0/3119910424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa54c0cc600 0x7fa54c02dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa55c4db640 10 monclient(hunting): picked mon.noname-a con 0x7fa54c0cc600 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:49.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa55c4db640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa55c4db640 1 -- 172.21.15.60:0/3119910424 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c0bc630 con 0x7fa54c0cc600 2024-03-27T21:39:49.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa55c4db640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa55c4db640 1 -- 172.21.15.60:0/3119910424 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c02fca0 con 0x7fa54c035640 2024-03-27T21:39:49.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa55c4db640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa55c4db640 1 -- 172.21.15.60:0/3119910424 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c0c3830 con 0x7fa54c038fe0 2024-03-27T21:39:49.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa55c4db640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:49.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa55c4db640 10 monclient(hunting): authenticate will time out at 831.060159s 2024-03-27T21:39:49.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa559a76640 10 monclient(hunting): get_auth_request con 0x7fa54c038fe0 auth_method 0 2024-03-27T21:39:49.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa559a76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa559a76640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa559a76640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa559a76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa559a76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa559a76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:49.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa55aa78640 1 --2- 172.21.15.60:0/3119910424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa54c0cc600 0x7fa54c02dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa55aa78640 10 monclient(hunting): get_auth_request con 0x7fa54c0cc600 auth_method 0 2024-03-27T21:39:49.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa55aa78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa55aa78640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa55aa78640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa559a76640 10 monclient(hunting): handle_auth_done global_id 4340 payload 824 2024-03-27T21:39:49.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa559a76640 1 -- 172.21.15.60:0/3119910424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa54c035640 msgr2=0x7fa54c0c32f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T21:39:49.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f2f7259c640 1 --2- 172.21.15.60:0/2143680276 >> v2:172.21.15.60:3300/0 conn(0x7f2f7406a4d0 0x7f2f740716b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:49.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa559a76640 1 --2- 172.21.15.60:0/3119910424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa54c035640 0x7fa54c0c32f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa559a76640 1 -- 172.21.15.60:0/3119910424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa54c0cc600 msgr2=0x7fa54c02dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa559a76640 1 --2- 172.21.15.60:0/3119910424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa54c0cc600 0x7fa54c02dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa559a76640 10 monclient: _finish_hunting 0 2024-03-27T21:39:49.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa559a76640 1 monclient: found mon.noname-c 2024-03-27T21:39:49.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa559a76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:49.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa559a76640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:39:49.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fa559a76640 1 -- 172.21.15.60:0/3119910424 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa550009400 con 0x7fa54c038fe0 2024-03-27T21:39:49.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fb3f2cd2640 1 librados: starting msgr at 2024-03-27T21:39:49.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fb3f2cd2640 1 librados: starting objecter 2024-03-27T21:39:49.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fb3f2cd2640 1 Processor -- start 2024-03-27T21:39:49.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f75bbfff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f75bc06faa0 0x7f75bc1d1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47806/0 (socket says 172.21.15.60:47806) 2024-03-27T21:39:49.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f75bbfff640 1 -- 172.21.15.60:0/3470966457 learned_addr learned my addr 172.21.15.60:0/3470966457 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:49.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f75bbfff640 10 monclient(hunting): get_auth_request con 0x7f75bc06faa0 auth_method 0 2024-03-27T21:39:49.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f75bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f75bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f75bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f75c0c16640 1 --2- 172.21.15.60:0/3470966457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75bc06f6c0 0x7f75bc0ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f75c0c16640 10 monclient(hunting): get_auth_request con 0x7f75bc06f6c0 auth_method 0 2024-03-27T21:39:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f75c0c16640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f75c0c16640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f75c0c16640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75c0c16640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75c0c16640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75c0c16640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7efe12d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7efe12d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7efe12d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7efe1259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7efe1259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7efe1259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7efe12d9d640 10 monclient(hunting): handle_auth_done global_id 4337 payload 824 2024-03-27T21:39:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7efe12d9d640 1 -- 172.21.15.60:0/1559011480 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe140ad910 msgr2=0x7efe14143520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7efe12d9d640 1 --2- 172.21.15.60:0/1559011480 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe140ad910 0x7efe14143520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7efe12d9d640 1 -- 172.21.15.60:0/1559011480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe1406ea80 msgr2=0x7efe14145e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7efe12d9d640 1 --2- 172.21.15.60:0/1559011480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe1406ea80 0x7efe14145e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7efe12d9d640 10 monclient: _finish_hunting 0 2024-03-27T21:39:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7efe12d9d640 1 monclient: found mon.noname-c 2024-03-27T21:39:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7efe12d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:49.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7efe12d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:39:49.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7efe12d9d640 1 -- 172.21.15.60:0/1559011480 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe04009400 con 0x7efe14148f10 2024-03-27T21:39:49.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f2f72d9d640 1 --2- 172.21.15.60:0/2143680276 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2f7406a8b0 0x7f2f74141cf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009e50 tx=0x7f2f6c02be70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69bcbe3725e6d59a server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7fdddcba9640 1 librados: starting msgr at 2024-03-27T21:39:49.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7fdddcba9640 1 librados: starting objecter 2024-03-27T21:39:49.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7fdddcba9640 1 Processor -- start 2024-03-27T21:39:49.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fb3f2cd2640 1 -- start start 2024-03-27T21:39:49.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fb3f2cd2640 1 librados: setting wanted keys 2024-03-27T21:39:49.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fb3f2cd2640 1 librados: calling monclient init 2024-03-27T21:39:49.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fb3f2cd2640 10 monclient: init 2024-03-27T21:39:49.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7fdddcba9640 1 -- start start 2024-03-27T21:39:49.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7fdddcba9640 1 librados: setting wanted keys 2024-03-27T21:39:49.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7fdddcba9640 1 librados: calling monclient init 2024-03-27T21:39:49.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.837+0000 7fdddcba9640 10 monclient: init 2024-03-27T21:39:49.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fdddcba9640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:49.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fdddcba9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:39:49.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fdddcba9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fddc0004d00 0x7fddd80ab850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fdddcba9640 10 monclient(hunting): picked mon.noname-b con 0x7fddc0004d00 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:49.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fdddcba9640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fddd80abd90 0x7fddd806db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fdddcba9640 10 monclient(hunting): picked mon.noname-c con 0x7fddd80abd90 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:49.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e72ffd640 10 monclient(hunting): get_auth_request con 0x7f1e74060970 auth_method 0 2024-03-27T21:39:49.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fdddcba9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddd806e040 0x7fddd8137c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fdddcba9640 10 monclient(hunting): picked mon.noname-a con 0x7fddd806e040 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:49.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fdddcba9640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fdddcba9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd80aae90 con 0x7fddd806e040 2024-03-27T21:39:49.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fdddcba9640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fdddcba9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd806f530 con 0x7fddc0004d00 2024-03-27T21:39:49.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fdddcba9640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fdddcba9640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd80aaa40 con 0x7fddd80abd90 2024-03-27T21:39:49.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fdddcba9640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:49.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7fdddcba9640 10 monclient(hunting): authenticate will time out at 831.060565s 2024-03-27T21:39:49.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.838+0000 7f1e72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e737fe640 1 --2- 172.21.15.60:0/3714157888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e74071b30 0x7f1e740a8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e737fe640 10 monclient(hunting): get_auth_request con 0x7f1e74071b30 auth_method 0 2024-03-27T21:39:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e737fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e727fc640 10 monclient(hunting): get_auth_request con 0x7f1e7406e750 auth_method 0 2024-03-27T21:39:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e727fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75c1417640 1 --2- 172.21.15.60:0/3470966457 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75bc0af6e0 0x7f75bc1d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75c1417640 10 monclient(hunting): get_auth_request con 0x7f75bc0af6e0 auth_method 0 2024-03-27T21:39:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75c1417640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75c1417640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75c1417640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75c0c16640 10 monclient(hunting): handle_auth_done global_id 4396 payload 824 2024-03-27T21:39:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75c0c16640 1 -- 172.21.15.60:0/3470966457 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f75bc06faa0 msgr2=0x7f75bc1d1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75c0c16640 1 --2- 172.21.15.60:0/3470966457 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f75bc06faa0 0x7f75bc1d1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75c0c16640 1 -- 172.21.15.60:0/3470966457 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75bc0af6e0 msgr2=0x7f75bc1d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75c0c16640 1 --2- 172.21.15.60:0/3470966457 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75bc0af6e0 0x7f75bc1d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75c0c16640 10 monclient: _finish_hunting 0 2024-03-27T21:39:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75c0c16640 1 monclient: found mon.noname-b 2024-03-27T21:39:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75c0c16640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:49.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75c0c16640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:39:49.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75c0c16640 1 -- 172.21.15.60:0/3470966457 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b0011d70 con 0x7f75bc06f6c0 2024-03-27T21:39:49.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e72ffd640 10 monclient(hunting): handle_auth_done global_id 4399 payload 824 2024-03-27T21:39:49.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fa559a76640 1 --2- 172.21.15.60:0/3119910424 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa54c038fe0 0x7fa54c0beef0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa54400cb60 tx=0x7fa544007410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fa54affd640 10 client.?.objecter ms_handle_connect 0x7fa54c038fe0 2024-03-27T21:39:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fa54affd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7efe12d9d640 1 --2- 172.21.15.60:0/1559011480 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efe14148f10 0x7efe14143a60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efe0c001560 tx=0x7efe0c00cbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7efe02ffd640 10 client.?.objecter ms_handle_connect 0x7efe14148f10 2024-03-27T21:39:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7efe02ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e72ffd640 1 -- 172.21.15.60:0/3714157888 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e7406e750 msgr2=0x7f1e740a8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75c0c16640 1 --2- 172.21.15.60:0/3470966457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75bc06f6c0 0x7f75bc0ae4a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f75b0043000 tx=0x7f75b00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f75b97fa640 10 client.?.objecter ms_handle_connect 0x7f75bc06f6c0 2024-03-27T21:39:49.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e72ffd640 1 --2- 172.21.15.60:0/3714157888 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e7406e750 0x7f1e740a8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75b97fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:49.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e72ffd640 1 -- 172.21.15.60:0/3714157888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e74071b30 msgr2=0x7f1e740a8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e72ffd640 1 --2- 172.21.15.60:0/3714157888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e74071b30 0x7f1e740a8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e72ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:39:49.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3f2cd2640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:49.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3f2cd2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:39:49.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75b97fa640 1 -- 172.21.15.60:0/3470966457 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b004cc80 con 0x7f75bc06f6c0 2024-03-27T21:39:49.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75b97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:49.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75b97fa640 10 monclient: dump: 2024-03-27T21:39:49.939 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.939 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.939 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.939 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.939 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.939 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.939 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.939 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.939 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.940 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75b97fa640 10 monclient: _finish_auth 0 2024-03-27T21:39:49.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75b97fa640 10 monclient: _check_auth_tickets 2024-03-27T21:39:49.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:49.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75b97fa640 1 -- 172.21.15.60:0/3470966457 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b004cde0 con 0x7f75bc06f6c0 2024-03-27T21:39:49.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75b97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:49.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75b97fa640 1 -- 172.21.15.60:0/3470966457 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b004a670 con 0x7f75bc06f6c0 2024-03-27T21:39:49.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75b97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:49.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75b97fa640 10 monclient: dump: 2024-03-27T21:39:49.940 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.940 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.940 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.940 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.940 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.941 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.941 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.941 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.941 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.941 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e72ffd640 1 monclient: found mon.noname-b 2024-03-27T21:39:49.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:49.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e72ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:39:49.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f1e72ffd640 1 -- 172.21.15.60:0/3714157888 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e64009b30 con 0x7f1e74060970 2024-03-27T21:39:49.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f2f71d9b640 1 -- 172.21.15.60:0/2143680276 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c02ca10 con 0x7f2f7406a8b0 2024-03-27T21:39:49.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f2f71d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f2f71d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:49.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f2f71d9b640 10 monclient: dump: 2024-03-27T21:39:49.942 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.942 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.942 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.942 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.942 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.942 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.942 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.942 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.942 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.942 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f2f71d9b640 10 monclient: _finish_auth 0 2024-03-27T21:39:49.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f2f71d9b640 10 monclient: _check_auth_tickets 2024-03-27T21:39:49.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f2f71d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:49.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f2f71d9b640 1 -- 172.21.15.60:0/2143680276 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6c02cb70 con 0x7f2f7406a8b0 2024-03-27T21:39:49.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f2f71d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:49.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f2f71d9b640 1 -- 172.21.15.60:0/2143680276 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c0343f0 con 0x7f2f7406a8b0 2024-03-27T21:39:49.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f2f71d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f2f71d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:49.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3f2cd2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3ec13b020 0x7fb3ec134bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3f2cd2640 10 monclient(hunting): picked mon.noname-a con 0x7fb3ec13b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:49.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f2f71d9b640 10 monclient: dump: 2024-03-27T21:39:49.944 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.944 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.944 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.944 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.944 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.944 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.944 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.944 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.944 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.944 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f2f78fd5640 5 monclient: authenticate success, global_id 4334 2024-03-27T21:39:49.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f2f78fd5640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:49.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7f2f78fd5640 4 set_mon_vals no callback set 2024-03-27T21:39:49.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fddd6d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddd806e040 0x7fddd8137c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fddd5d9b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fddd80abd90 0x7fddd806db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fddd6d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddd806e040 0x7fddd8137c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48020/0 (socket says 172.21.15.60:48020) 2024-03-27T21:39:49.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fddd6d9d640 1 -- 172.21.15.60:0/256875073 learned_addr learned my addr 172.21.15.60:0/256875073 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:49.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fddd6d9d640 10 monclient(hunting): get_auth_request con 0x7fddd806e040 auth_method 0 2024-03-27T21:39:49.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fddd6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fddd6d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fddd6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fddd659c640 1 --2- 172.21.15.60:0/256875073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fddc0004d00 0x7fddd80ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fddd659c640 10 monclient(hunting): get_auth_request con 0x7fddc0004d00 auth_method 0 2024-03-27T21:39:49.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fddd659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fddd659c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fddd659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fddd6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fddd6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3f2cd2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3ec1416f0 0x7fb3ec13aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3f2cd2640 10 monclient(hunting): picked mon.noname-b con 0x7fb3ec1416f0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:49.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3f2cd2640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb3ec13ba60 0x7fb3ec1312a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3f2cd2640 10 monclient(hunting): picked mon.noname-c con 0x7fb3ec13ba60 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:49.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3f2cd2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3f2cd2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec139fd0 con 0x7fb3ec13b020 2024-03-27T21:39:49.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3f2cd2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3f2cd2640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec13c530 con 0x7fb3ec1416f0 2024-03-27T21:39:49.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.839+0000 7fddd6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:49.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fddd6d9d640 10 monclient(hunting): handle_auth_done global_id 4395 payload 824 2024-03-27T21:39:49.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fddd6d9d640 1 -- 172.21.15.60:0/256875073 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fddd80abd90 msgr2=0x7fddd806db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fddd6d9d640 1 --2- 172.21.15.60:0/256875073 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fddd80abd90 0x7fddd806db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fddd6d9d640 1 -- 172.21.15.60:0/256875073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fddc0004d00 msgr2=0x7fddd80ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fddd6d9d640 1 --2- 172.21.15.60:0/256875073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fddc0004d00 0x7fddd80ab850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fddd6d9d640 10 monclient: _finish_hunting 0 2024-03-27T21:39:49.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fddd6d9d640 1 monclient: found mon.noname-a 2024-03-27T21:39:49.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fddd6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:49.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3f2cd2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3f2cd2640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec13c010 con 0x7fb3ec13ba60 2024-03-27T21:39:49.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3f2cd2640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:49.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3f2cd2640 10 monclient(hunting): authenticate will time out at 831.062276s 2024-03-27T21:39:49.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fddd6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:49.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fddd6d9d640 1 -- 172.21.15.60:0/256875073 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd80aac00 con 0x7fddd806e040 2024-03-27T21:39:49.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fddd5d9b640 10 monclient: get_auth_request con 0x7fddd80abd90 auth_method 0 2024-03-27T21:39:49.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fddd5d9b640 1 --2- 172.21.15.60:0/256875073 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fddd80abd90 0x7fddd806db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:39:49.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fddd6d9d640 1 --2- 172.21.15.60:0/256875073 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddd806e040 0x7fddd8137c80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fddcc062040 tx=0x7fddcc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fddc6ffd640 10 client.?.objecter ms_handle_connect 0x7fddd806e040 2024-03-27T21:39:49.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fddc6ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:49.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3ebfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3ec1416f0 0x7fb3ec13aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3ebfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3ec1416f0 0x7fb3ec13aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:56910/0 (socket says 172.21.15.60:56910) 2024-03-27T21:39:49.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3ebfff640 1 -- 172.21.15.60:0/491830654 learned_addr learned my addr 172.21.15.60:0/491830654 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:49.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3ebfff640 10 monclient(hunting): get_auth_request con 0x7fb3ec1416f0 auth_method 0 2024-03-27T21:39:49.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:49.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3f0a6e640 1 --2- 172.21.15.60:0/491830654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3ec13b020 0x7fb3ec134bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3f0a6e640 10 monclient(hunting): get_auth_request con 0x7fb3ec13b020 auth_method 0 2024-03-27T21:39:49.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3f0a6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3f0a6e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3f0a6e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3ebfff640 10 monclient(hunting): handle_auth_done global_id 4405 payload 824 2024-03-27T21:39:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3ebfff640 1 -- 172.21.15.60:0/491830654 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb3ec13ba60 msgr2=0x7fb3ec1312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3ebfff640 1 --2- 172.21.15.60:0/491830654 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb3ec13ba60 0x7fb3ec1312a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3ebfff640 1 -- 172.21.15.60:0/491830654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3ec13b020 msgr2=0x7fb3ec134bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3ebfff640 1 --2- 172.21.15.60:0/491830654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3ec13b020 0x7fb3ec134bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3ebfff640 10 monclient: _finish_hunting 0 2024-03-27T21:39:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fb3ebfff640 1 monclient: found mon.noname-b 2024-03-27T21:39:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fb3ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fb3ebfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:39:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fb3ebfff640 1 -- 172.21.15.60:0/491830654 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e00094d0 con 0x7fb3ec1416f0 2024-03-27T21:39:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75baffd640 4 set_mon_vals no callback set 2024-03-27T21:39:49.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75c1c18640 5 monclient: authenticate success, global_id 4396 2024-03-27T21:39:49.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75c1c18640 10 monclient: _renew_subs 2024-03-27T21:39:49.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75c1c18640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:49.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75c1c18640 1 -- 172.21.15.60:0/3470966457 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75bc0abec0 con 0x7f75bc06f6c0 2024-03-27T21:39:49.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75c1c18640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:49.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75c1c18640 10 monclient: _renew_subs 2024-03-27T21:39:49.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75c1c18640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:49.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75c1c18640 1 -- 172.21.15.60:0/3470966457 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75bc0ac2f0 con 0x7f75bc06f6c0 2024-03-27T21:39:49.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f75c1c18640 1 librados: init done 2024-03-27T21:39:49.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f1e72ffd640 1 --2- 172.21.15.60:0/3714157888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e74060970 0x7f1e741456e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1e640093e0 tx=0x7f1e64004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f1e6b7fe640 10 client.?.objecter ms_handle_connect 0x7f1e74060970 2024-03-27T21:39:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f1e6b7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f1e6b7fe640 1 -- 172.21.15.60:0/3714157888 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e64007cb0 con 0x7f1e74060970 2024-03-27T21:39:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f1e6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f1e6b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f1e6b7fe640 10 monclient: dump: 2024-03-27T21:39:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.954 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f1e6b7fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f1e6b7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f1e6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f1e6b7fe640 1 -- 172.21.15.60:0/3714157888 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e64007e10 con 0x7f1e74060970 2024-03-27T21:39:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f1e6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f1e6b7fe640 1 -- 172.21.15.60:0/3714157888 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6403e690 con 0x7f1e74060970 2024-03-27T21:39:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7f1e6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7f1e6b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7f1e6b7fe640 10 monclient: dump: 2024-03-27T21:39:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.955 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7f1e717fa640 4 set_mon_vals no callback set 2024-03-27T21:39:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7f1e792b6640 5 monclient: authenticate success, global_id 4399 2024-03-27T21:39:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7f1e792b6640 10 monclient: _renew_subs 2024-03-27T21:39:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7f1e792b6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7f1e792b6640 1 -- 172.21.15.60:0/3714157888 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e74134770 con 0x7f1e74060970 2024-03-27T21:39:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7f1e792b6640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7f1e792b6640 10 monclient: _renew_subs 2024-03-27T21:39:49.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7f1e792b6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:49.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7f1e792b6640 1 -- 172.21.15.60:0/3714157888 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e740aadc0 con 0x7f1e74060970 2024-03-27T21:39:49.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7f1e792b6640 1 librados: init done 2024-03-27T21:39:49.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7efe02ffd640 1 -- 172.21.15.60:0/1559011480 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0c00ce30 con 0x7efe14148f10 2024-03-27T21:39:49.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7efe02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7efe02ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:49.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7efe02ffd640 10 monclient: dump: 2024-03-27T21:39:49.956 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.956 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.957 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.957 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.957 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.957 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.957 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.957 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.957 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.957 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7efe02ffd640 10 monclient: _finish_auth 0 2024-03-27T21:39:49.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7efe02ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:39:49.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7efe02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:49.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7efe1359e640 5 monclient: authenticate success, global_id 4337 2024-03-27T21:39:49.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7efe1359e640 10 monclient: _renew_subs 2024-03-27T21:39:49.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7efe1359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:49.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7efe1359e640 1 -- 172.21.15.60:0/1559011480 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe08009400 con 0x7efe14148f10 2024-03-27T21:39:49.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7efe1359e640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:49.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7efe1359e640 10 monclient: _renew_subs 2024-03-27T21:39:49.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7efe1359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:49.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7efe1359e640 1 -- 172.21.15.60:0/1559011480 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe14146520 con 0x7efe14148f10 2024-03-27T21:39:49.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7efe1359e640 1 librados: init done 2024-03-27T21:39:49.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7efe02ffd640 1 -- 172.21.15.60:0/1559011480 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe0c00a640 con 0x7efe14148f10 2024-03-27T21:39:49.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7efe02ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:49.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7efe10d99640 4 set_mon_vals no callback set 2024-03-27T21:39:49.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.842+0000 7efe02ffd640 1 -- 172.21.15.60:0/1559011480 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0c00aa10 con 0x7efe14148f10 2024-03-27T21:39:49.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fa54affd640 1 -- 172.21.15.60:0/3119910424 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54400ce50 con 0x7fa54c038fe0 2024-03-27T21:39:49.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7fa54affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fa54affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:49.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fa54affd640 10 monclient: dump: 2024-03-27T21:39:49.959 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.959 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.959 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.959 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.959 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.959 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.959 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.959 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.959 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.959 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fa54affd640 10 monclient: _finish_auth 0 2024-03-27T21:39:49.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fa54affd640 10 monclient: _check_auth_tickets 2024-03-27T21:39:49.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fa54affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:49.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fa55c4db640 5 monclient: authenticate success, global_id 4340 2024-03-27T21:39:49.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fa55c4db640 10 monclient: _renew_subs 2024-03-27T21:39:49.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fa55c4db640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:49.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fa55c4db640 1 -- 172.21.15.60:0/3119910424 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa54c0bf430 con 0x7fa54c038fe0 2024-03-27T21:39:49.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fa55c4db640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:49.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fa55c4db640 10 monclient: _renew_subs 2024-03-27T21:39:49.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fa55c4db640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:49.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fa55c4db640 1 -- 172.21.15.60:0/3119910424 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa54c02e340 con 0x7fa54c038fe0 2024-03-27T21:39:49.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fa55c4db640 1 librados: init done 2024-03-27T21:39:49.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fa54affd640 1 -- 172.21.15.60:0/3119910424 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa544007b20 con 0x7fa54c038fe0 2024-03-27T21:39:49.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fa54affd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:49.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fa54affd640 1 -- 172.21.15.60:0/3119910424 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5440113f0 con 0x7fa54c038fe0 2024-03-27T21:39:49.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fa54affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fa54affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:49.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fa54affd640 10 monclient: dump: 2024-03-27T21:39:49.961 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.961 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.961 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.961 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.961 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.961 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.961 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.961 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.961 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.961 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.841+0000 7fa558a74640 4 set_mon_vals no callback set 2024-03-27T21:39:49.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f2f78fd5640 10 monclient: shutdown 2024-03-27T21:39:49.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f2f78fd5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:49.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f2f78fd5640 1 -- 172.21.15.60:0/2143680276 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2f7406a8b0 msgr2=0x7f2f74141cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.840+0000 7f2f78fd5640 1 --2- 172.21.15.60:0/2143680276 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2f7406a8b0 0x7f2f74141cf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009e50 tx=0x7f2f6c02be70 comp rx=0 tx=0).stop 2024-03-27T21:39:49.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.842+0000 7fddc6ffd640 1 -- 172.21.15.60:0/256875073 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc069600 con 0x7fddd806e040 2024-03-27T21:39:49.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.842+0000 7fddc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.842+0000 7fddc6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:49.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.842+0000 7fddc6ffd640 10 monclient: dump: 2024-03-27T21:39:49.963 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.963 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.963 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.963 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.963 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.963 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.963 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.964 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.842+0000 7fddc6ffd640 10 monclient: _finish_auth 0 2024-03-27T21:39:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.842+0000 7fddc6ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:39:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.842+0000 7fddc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.842+0000 7fddc6ffd640 1 -- 172.21.15.60:0/256875073 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc069760 con 0x7fddd806e040 2024-03-27T21:39:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.842+0000 7fddc6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.842+0000 7fddc6ffd640 1 -- 172.21.15.60:0/256875073 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc069ad0 con 0x7fddd806e040 2024-03-27T21:39:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.842+0000 7fddc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.842+0000 7fddc6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.842+0000 7fddc6ffd640 10 monclient: dump: 2024-03-27T21:39:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.965 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.965 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.965 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.965 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.965 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.965 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.965 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.965 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.842+0000 7fa55c4db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:49.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.842+0000 7fddd4d99640 4 set_mon_vals no callback set 2024-03-27T21:39:49.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.842+0000 7fa55c4db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:49.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.842+0000 7fa55c4db640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:49.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.842+0000 7fa55c4db640 1 -- 172.21.15.60:0/3119910424 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa53c005040 con 0x7fa54c038fe0 2024-03-27T21:39:49.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.842+0000 7fb3ebfff640 1 --2- 172.21.15.60:0/491830654 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3ec1416f0 0x7fb3ec13aa50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb3d800b480 tx=0x7fb3d800b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7fb3e97fa640 10 client.?.objecter ms_handle_connect 0x7fb3ec1416f0 2024-03-27T21:39:49.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7fb3e97fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:49.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f1e697fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:49.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7fdddcba9640 5 monclient: authenticate success, global_id 4395 2024-03-27T21:39:49.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f1e697fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:49.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7fdddcba9640 10 monclient: _renew_subs 2024-03-27T21:39:49.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7fdddcba9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:49.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7fdddcba9640 1 -- 172.21.15.60:0/256875073 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd806e4c0 con 0x7fddd806e040 2024-03-27T21:39:49.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7fdddcba9640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:49.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7fdddcba9640 10 monclient: _renew_subs 2024-03-27T21:39:49.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7fdddcba9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:49.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7fdddcba9640 1 -- 172.21.15.60:0/256875073 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd81381c0 con 0x7fddd806e040 2024-03-27T21:39:49.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7fdddcba9640 1 librados: init done 2024-03-27T21:39:49.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f1e697fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:49.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f1e697fa640 1 -- 172.21.15.60:0/3714157888 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e4c005040 con 0x7f1e74060970 2024-03-27T21:39:49.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f75c1c18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:49.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f75c1c18640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:49.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f75c1c18640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:49.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f75c1c18640 1 -- 172.21.15.60:0/3470966457 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75bc069470 con 0x7f75bc06f6c0 2024-03-27T21:39:49.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7fa54affd640 1 -- 172.21.15.60:0/3119910424 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa544011550 con 0x7fa54c038fe0 2024-03-27T21:39:49.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f2f78fd5640 1 -- 172.21.15.60:0/2143680276 shutdown_connections 2024-03-27T21:39:49.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f2f78fd5640 1 --2- 172.21.15.60:0/2143680276 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2f7406a8b0 0x7f2f74141cf0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f2f78fd5640 1 --2- 172.21.15.60:0/2143680276 >> v2:172.21.15.60:3300/0 conn(0x7f2f7406a4d0 0x7f2f740716b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f75b97fa640 1 -- 172.21.15.60:0/3470966457 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f75b004a7d0 con 0x7f75bc06f6c0 2024-03-27T21:39:49.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f75b97fa640 1 --2- 172.21.15.60:0/3470966457 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f75a806c590 0x7f75a806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f2f78fd5640 1 -- 172.21.15.60:0/2143680276 >> 172.21.15.60:0/2143680276 conn(0x7f2f740b4e50 msgr2=0x7f2f740b5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:49.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f75b97fa640 1 -- 172.21.15.60:0/3470966457 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f75b00bbdb0 con 0x7f75bc06f6c0 2024-03-27T21:39:49.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f75b97fa640 10 client.4396.objecter ms_dispatch 0x7f75bc000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:49.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f75b97fa640 3 client.4396.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:49.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f75b97fa640 3 client.4396.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:49.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f75b97fa640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-27T21:39:49.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7fa54affd640 1 --2- 172.21.15.60:0/3119910424 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa53806c5e0 0x7fa53806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f1e6b7fe640 1 -- 172.21.15.60:0/3714157888 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1e6403e7f0 con 0x7f1e74060970 2024-03-27T21:39:49.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7fa55a277640 1 --2- 172.21.15.60:0/3119910424 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa53806c5e0 0x7fa53806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7fdddcba9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:49.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7fdddcba9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:49.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7fa55a277640 10 monclient: get_auth_request con 0x7fa53806c5e0 auth_method 0 2024-03-27T21:39:49.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7fdddcba9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:49.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f2f78fd5640 1 -- 172.21.15.60:0/2143680276 shutdown_connections 2024-03-27T21:39:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f2f78fd5640 1 -- 172.21.15.60:0/2143680276 wait complete. 2024-03-27T21:39:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f75bbfff640 1 --2- 172.21.15.60:0/3470966457 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f75a806c590 0x7f75a806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7f75bbfff640 10 monclient: get_auth_request con 0x7f75a806c590 auth_method 0 2024-03-27T21:39:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7fdddcba9640 1 -- 172.21.15.60:0/256875073 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd80acf80 con 0x7fddd806e040 2024-03-27T21:39:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.844+0000 7fa54affd640 1 -- 172.21.15.60:0/3119910424 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa544089750 con 0x7fa54c038fe0 2024-03-27T21:39:49.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.842+0000 7efe02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fa54affd640 10 client.4340.objecter ms_dispatch 0x7fa554000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:49.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fa54affd640 3 client.4340.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:49.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fa54affd640 3 client.4340.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:49.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7efe02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:49.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7efe02ffd640 10 monclient: dump: 2024-03-27T21:39:49.976 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.976 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.976 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.976 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.976 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.976 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.976 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.976 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.976 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.977 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fa54affd640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-27T21:39:49.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7f1e6b7fe640 1 --2- 172.21.15.60:0/3714157888 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1e5406c660 0x7f1e5406eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3e97fa640 1 -- 172.21.15.60:0/491830654 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8010870 con 0x7fb3ec1416f0 2024-03-27T21:39:49.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3e97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:49.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3e97fa640 10 monclient: dump: 2024-03-27T21:39:49.978 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.978 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.978 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.978 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.978 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.978 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.978 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.978 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.978 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.978 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3e97fa640 10 monclient: _finish_auth 0 2024-03-27T21:39:49.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3e97fa640 10 monclient: _check_auth_tickets 2024-03-27T21:39:49.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7f1e727fc640 1 --2- 172.21.15.60:0/3714157888 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1e5406c660 0x7f1e5406eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7f1e727fc640 10 monclient: get_auth_request con 0x7f1e5406c660 auth_method 0 2024-03-27T21:39:49.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7efe1359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:49.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7efe1359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:49.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7efe1359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:49.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7efe1359e640 1 -- 172.21.15.60:0/1559011480 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf8005040 con 0x7efe14148f10 2024-03-27T21:39:49.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:49.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3e97fa640 1 -- 172.21.15.60:0/491830654 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d80109f0 con 0x7fb3ec1416f0 2024-03-27T21:39:49.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7f75b97fa640 10 client.4396.objecter ms_handle_connect 0x7f75a806c590 2024-03-27T21:39:49.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7f1e6b7fe640 1 -- 172.21.15.60:0/3714157888 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1e640abef0 con 0x7f1e74060970 2024-03-27T21:39:49.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7f1e6b7fe640 10 client.4399.objecter ms_dispatch 0x7f1e74000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:49.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3e97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:49.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3e97fa640 1 -- 172.21.15.60:0/491830654 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8010da0 con 0x7fb3ec1416f0 2024-03-27T21:39:49.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7f75bbfff640 1 --2- 172.21.15.60:0/3470966457 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f75a806c590 0x7f75a806ea50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f75bc0b3ed0 tx=0x7f75ac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7f1e6b7fe640 3 client.4399.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:49.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7f1e6b7fe640 3 client.4399.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:49.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:49.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3e97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:49.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3e97fa640 10 monclient: dump: 2024-03-27T21:39:49.982 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:49.982 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:49.982 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.982 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:49.982 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:49.982 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:49.982 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:49.982 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:49.982 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:49.983 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7f1e6b7fe640 20 client.4399.objecter dump_active .. 0 homeless 2024-03-27T21:39:49.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.846+0000 7fa55a277640 1 --2- 172.21.15.60:0/3119910424 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa53806c5e0 0x7fa53806eaa0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa550036000 tx=0x7fa550008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.846+0000 7fa54affd640 10 client.4340.objecter ms_handle_connect 0x7fa53806c5e0 2024-03-27T21:39:49.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.846+0000 7f1e727fc640 1 --2- 172.21.15.60:0/3714157888 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1e5406c660 0x7f1e5406eb20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c006fd0 tx=0x7f1e6c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.846+0000 7f1e6b7fe640 10 client.4399.objecter ms_handle_connect 0x7f1e5406c660 2024-03-27T21:39:49.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.846+0000 7f2f78fd5640 10 monclient: build_initial_monmap 2024-03-27T21:39:49.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.846+0000 7f2f78fd5640 10 monclient: monmap: 2024-03-27T21:39:49.985 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:49.985 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:49.985 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:49.985 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:49.985 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:49.985 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:49.985 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:49.985 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:49.985 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:49.985 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:49.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.846+0000 7f2f78fd5640 1 librados: starting msgr at 2024-03-27T21:39:49.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.846+0000 7f2f78fd5640 1 librados: starting objecter 2024-03-27T21:39:49.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3f2cd2640 5 monclient: authenticate success, global_id 4405 2024-03-27T21:39:49.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3f2cd2640 10 monclient: _renew_subs 2024-03-27T21:39:49.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3f2cd2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:49.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3f2cd2640 1 -- 172.21.15.60:0/491830654 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3ec1317e0 con 0x7fb3ec1416f0 2024-03-27T21:39:49.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3f2cd2640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:49.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3f2cd2640 10 monclient: _renew_subs 2024-03-27T21:39:49.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3f2cd2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:49.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3f2cd2640 1 -- 172.21.15.60:0/491830654 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3ec131ca0 con 0x7fb3ec1416f0 2024-03-27T21:39:49.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fb3f2cd2640 1 librados: init done 2024-03-27T21:39:49.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.847+0000 7f2f78fd5640 1 Processor -- start 2024-03-27T21:39:49.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.847+0000 7f2f78fd5640 1 -- start start 2024-03-27T21:39:49.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.847+0000 7fb3f2cd2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:49.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.847+0000 7fb3f2cd2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:49.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.847+0000 7fb3f2cd2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:49.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.847+0000 7fb3f2cd2640 1 -- 172.21.15.60:0/491830654 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3ec069470 con 0x7fb3ec1416f0 2024-03-27T21:39:49.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.847+0000 7f2f78fd5640 1 librados: setting wanted keys 2024-03-27T21:39:49.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.847+0000 7f2f78fd5640 1 librados: calling monclient init 2024-03-27T21:39:49.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.847+0000 7f2f78fd5640 10 monclient: init 2024-03-27T21:39:49.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.845+0000 7fddc6ffd640 1 -- 172.21.15.60:0/256875073 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fddcc069c30 con 0x7fddd806e040 2024-03-27T21:39:49.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.846+0000 7fddc6ffd640 1 --2- 172.21.15.60:0/256875073 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fddb806c660 0x7fddb806eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.846+0000 7fddc6ffd640 1 -- 172.21.15.60:0/256875073 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fddcc0d6ea0 con 0x7fddd806e040 2024-03-27T21:39:49.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.846+0000 7fddc6ffd640 10 client.4395.objecter ms_dispatch 0x7fddd8000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:49.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.846+0000 7fddc6ffd640 3 client.4395.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:49.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.846+0000 7fddc6ffd640 3 client.4395.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:49.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.846+0000 7fddd659c640 1 --2- 172.21.15.60:0/256875073 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fddb806c660 0x7fddb806eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.846+0000 7fddd659c640 10 monclient: get_auth_request con 0x7fddb806c660 auth_method 0 2024-03-27T21:39:49.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.847+0000 7fddc6ffd640 20 client.4395.objecter dump_active .. 0 homeless 2024-03-27T21:39:49.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.847+0000 7fddd659c640 1 --2- 172.21.15.60:0/256875073 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fddb806c660 0x7fddb806eb20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fddc800a9a0 tx=0x7fddc8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.847+0000 7fddc6ffd640 10 client.4395.objecter ms_handle_connect 0x7fddb806c660 2024-03-27T21:39:49.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.847+0000 7fb3eaffd640 4 set_mon_vals no callback set 2024-03-27T21:39:49.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.846+0000 7efe02ffd640 1 -- 172.21.15.60:0/1559011480 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7efe0c00ab70 con 0x7efe14148f10 2024-03-27T21:39:49.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.847+0000 7efe02ffd640 1 --2- 172.21.15.60:0/1559011480 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7efdf406c660 0x7efdf406eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.847+0000 7efe02ffd640 1 -- 172.21.15.60:0/1559011480 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7efe0c088950 con 0x7efe14148f10 2024-03-27T21:39:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.847+0000 7efe02ffd640 10 client.4337.objecter ms_dispatch 0x7efe14000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.847+0000 7efe02ffd640 3 client.4337.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.847+0000 7efe02ffd640 3 client.4337.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.847+0000 7efe02ffd640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-27T21:39:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.847+0000 7efe1259c640 1 --2- 172.21.15.60:0/1559011480 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7efdf406c660 0x7efdf406eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.847+0000 7efe1259c640 10 monclient: get_auth_request con 0x7efdf406c660 auth_method 0 2024-03-27T21:39:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.848+0000 7f1e6b7fe640 1 -- 172.21.15.60:0/3714157888 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1e64077450 con 0x7f1e74060970 2024-03-27T21:39:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.848+0000 7f1e6b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.848+0000 7f1e6b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:49.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.848+0000 7f2f78fd5640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:49.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.848+0000 7f2f78fd5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:39:49.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.848+0000 7f2f78fd5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f7406a4d0 0x7f2f74139140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.848+0000 7f2f78fd5640 10 monclient(hunting): picked mon.noname-b con 0x7f2f7406a4d0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:49.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.848+0000 7f2f78fd5640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2f7406a8b0 0x7f2f74143cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.848+0000 7f2f78fd5640 10 monclient(hunting): picked mon.noname-c con 0x7f2f7406a8b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:49.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.848+0000 7f2f6bfff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2f7406a8b0 0x7f2f74143cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.848+0000 7f2f6bfff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2f7406a8b0 0x7f2f74143cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47854/0 (socket says 172.21.15.60:47854) 2024-03-27T21:39:49.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.848+0000 7f2f6bfff640 1 -- 172.21.15.60:0/2831369609 learned_addr learned my addr 172.21.15.60:0/2831369609 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:49.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.848+0000 7efe1259c640 1 --2- 172.21.15.60:0/1559011480 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7efdf406c660 0x7efdf406eb20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efe04009d30 tx=0x7efe04009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.848+0000 7f2f78fd5640 1 --2- 172.21.15.60:0/2831369609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f74071bf0 0x7f2f74142ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:49.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.848+0000 7efe02ffd640 10 client.4337.objecter ms_handle_connect 0x7efdf406c660 2024-03-27T21:39:49.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f72d9d640 1 --2- 172.21.15.60:0/2831369609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f74071bf0 0x7f2f74142ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f78fd5640 10 monclient(hunting): picked mon.noname-a con 0x7f2f74071bf0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:49.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f78fd5640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f78fd5640 1 -- 172.21.15.60:0/2831369609 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f74144210 con 0x7f2f74071bf0 2024-03-27T21:39:49.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f78fd5640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f78fd5640 1 -- 172.21.15.60:0/2831369609 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7413bb10 con 0x7f2f7406a4d0 2024-03-27T21:39:49.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f78fd5640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:49.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f78fd5640 1 -- 172.21.15.60:0/2831369609 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f741427c0 con 0x7f2f7406a8b0 2024-03-27T21:39:49.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f78fd5640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:49.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f78fd5640 10 monclient(hunting): authenticate will time out at 831.071131s 2024-03-27T21:39:49.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f7259c640 1 --2- 172.21.15.60:0/2831369609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f7406a4d0 0x7f2f74139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:49.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f7259c640 10 monclient(hunting): get_auth_request con 0x7f2f7406a4d0 auth_method 0 2024-03-27T21:39:49.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f7259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f7259c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f7259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f72d9d640 10 monclient(hunting): get_auth_request con 0x7f2f74071bf0 auth_method 0 2024-03-27T21:39:49.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f72d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:49.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f72d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:49.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f72d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:49.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f7259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f7259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f7259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:49.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f72d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:49.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f72d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:49.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f72d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:49.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f7259c640 10 monclient(hunting): handle_auth_done global_id 4408 payload 824 2024-03-27T21:39:49.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f7259c640 1 -- 172.21.15.60:0/2831369609 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2f7406a8b0 msgr2=0x7f2f74143cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f7259c640 1 --2- 172.21.15.60:0/2831369609 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2f7406a8b0 0x7f2f74143cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f7259c640 1 -- 172.21.15.60:0/2831369609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f74071bf0 msgr2=0x7f2f74142ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:49.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f7259c640 1 --2- 172.21.15.60:0/2831369609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f74071bf0 0x7f2f74142ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:49.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f7259c640 10 monclient: _finish_hunting 0 2024-03-27T21:39:49.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f7259c640 1 monclient: found mon.noname-b 2024-03-27T21:39:49.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f7259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:49.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f7259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:39:49.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.849+0000 7f2f7259c640 1 -- 172.21.15.60:0/2831369609 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f64009400 con 0x7f2f7406a4d0 2024-03-27T21:39:49.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.850+0000 7f2f72d9d640 1 --2- 172.21.15.60:0/2831369609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f74071bf0 0x7f2f74142ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:49.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.850+0000 7f2f6bfff640 10 monclient: get_auth_request con 0x7f2f7406a8b0 auth_method 0 2024-03-27T21:39:49.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.850+0000 7f2f6bfff640 1 --2- 172.21.15.60:0/2831369609 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2f7406a8b0 0x7f2f74143cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:39:49.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.850+0000 7f2f7259c640 1 --2- 172.21.15.60:0/2831369609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f7406a4d0 0x7f2f74139140 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2f6400e930 tx=0x7f2f6400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:49.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.850+0000 7f2f6affd640 10 client.?.objecter ms_handle_connect 0x7f2f7406a4d0 2024-03-27T21:39:49.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.850+0000 7f2f6affd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:49.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.851+0000 7fa54affd640 1 -- 172.21.15.60:0/3119910424 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa544054b90 con 0x7fa54c038fe0 2024-03-27T21:39:49.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.851+0000 7fa54affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:49.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.851+0000 7fa54affd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:49.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7fddc6ffd640 1 -- 172.21.15.60:0/256875073 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fddcc0a24b0 con 0x7fddd806e040 2024-03-27T21:39:50.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7fddc6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:50.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7fddc6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:50.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7f75b97fa640 1 -- 172.21.15.60:0/3470966457 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f75b0087170 con 0x7f75bc06f6c0 2024-03-27T21:39:50.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7f75b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:50.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7f75b97fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:50.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7f2f6affd640 1 -- 172.21.15.60:0/2831369609 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6400ca00 con 0x7f2f7406a4d0 2024-03-27T21:39:50.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7f2f6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:50.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7f2f6affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:50.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7f2f6affd640 10 monclient: dump: 2024-03-27T21:39:50.001 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:50.001 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:50.001 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:50.001 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:50.001 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:50.001 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:50.001 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:50.001 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:50.002 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:50.002 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:50.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7f2f6affd640 10 monclient: _finish_auth 0 2024-03-27T21:39:50.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7f2f6affd640 10 monclient: _check_auth_tickets 2024-03-27T21:39:50.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7f2f6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:50.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7f2f6affd640 1 -- 172.21.15.60:0/2831369609 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6400cb60 con 0x7f2f7406a4d0 2024-03-27T21:39:50.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7f2f6affd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:50.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7f2f6affd640 1 -- 172.21.15.60:0/2831369609 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f64013470 con 0x7f2f7406a4d0 2024-03-27T21:39:50.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7f2f6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:50.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7f2f6affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:50.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7f2f6affd640 10 monclient: dump: 2024-03-27T21:39:50.003 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:50.003 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:50.003 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:50.003 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:50.003 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:50.003 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:50.003 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:50.003 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:50.003 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:50.003 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:50.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7f2f78fd5640 5 monclient: authenticate success, global_id 4408 2024-03-27T21:39:50.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7f2f78fd5640 10 monclient: _renew_subs 2024-03-27T21:39:50.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7f2f78fd5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:50.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.852+0000 7f2f78fd5640 1 -- 172.21.15.60:0/2831369609 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6c009b30 con 0x7f2f7406a4d0 2024-03-27T21:39:50.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.853+0000 7f2f78fd5640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:50.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.853+0000 7f2f78fd5640 10 monclient: _renew_subs 2024-03-27T21:39:50.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.853+0000 7f2f78fd5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:50.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.853+0000 7f2f78fd5640 1 -- 172.21.15.60:0/2831369609 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f7413c110 con 0x7f2f7406a4d0 2024-03-27T21:39:50.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.853+0000 7f2f78fd5640 1 librados: init done 2024-03-27T21:39:50.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.854+0000 7f2f7159a640 4 set_mon_vals no callback set 2024-03-27T21:39:50.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.855+0000 7fb3e97fa640 1 -- 172.21.15.60:0/491830654 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb3d8023480 con 0x7fb3ec1416f0 2024-03-27T21:39:50.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.855+0000 7f2f78fd5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:50.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.855+0000 7f2f78fd5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:50.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.855+0000 7f2f78fd5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:50.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.855+0000 7f2f78fd5640 1 -- 172.21.15.60:0/2831369609 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f58005040 con 0x7f2f7406a4d0 2024-03-27T21:39:50.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.856+0000 7fb3e97fa640 1 --2- 172.21.15.60:0/491830654 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb3d006c690 0x7fb3d006eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:50.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.856+0000 7fb3e97fa640 1 -- 172.21.15.60:0/491830654 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb3d8089460 con 0x7fb3ec1416f0 2024-03-27T21:39:50.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.856+0000 7fb3e97fa640 10 client.4405.objecter ms_dispatch 0x7fb3ec000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:50.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.856+0000 7fb3f0a6e640 1 --2- 172.21.15.60:0/491830654 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb3d006c690 0x7fb3d006eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:50.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.856+0000 7fb3f0a6e640 10 monclient: get_auth_request con 0x7fb3d006c690 auth_method 0 2024-03-27T21:39:50.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.856+0000 7fb3e97fa640 3 client.4405.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:50.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.856+0000 7fb3e97fa640 3 client.4405.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:50.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.856+0000 7fb3e97fa640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-27T21:39:50.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.857+0000 7fb3e97fa640 10 client.4405.objecter ms_handle_connect 0x7fb3d006c690 2024-03-27T21:39:50.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.857+0000 7fb3f0a6e640 1 --2- 172.21.15.60:0/491830654 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb3d006c690 0x7fb3d006eb50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0036000 tx=0x7fb3e003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:50.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.857+0000 7fb3e97fa640 1 -- 172.21.15.60:0/491830654 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb3d80548a0 con 0x7fb3ec1416f0 2024-03-27T21:39:50.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.857+0000 7fb3e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:50.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.857+0000 7fb3e97fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:50.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.858+0000 7efe02ffd640 1 -- 172.21.15.60:0/1559011480 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efe0c053e40 con 0x7efe14148f10 2024-03-27T21:39:50.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.858+0000 7efe02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:50.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.858+0000 7efe02ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:50.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.859+0000 7f2f6affd640 1 -- 172.21.15.60:0/2831369609 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2f64004020 con 0x7f2f7406a4d0 2024-03-27T21:39:50.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.859+0000 7f2f6affd640 1 --2- 172.21.15.60:0/2831369609 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2f5406c610 0x7f2f5406ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:50.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.859+0000 7f2f6affd640 1 -- 172.21.15.60:0/2831369609 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2f64089270 con 0x7f2f7406a4d0 2024-03-27T21:39:50.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.859+0000 7f2f6affd640 10 client.4408.objecter ms_dispatch 0x7f2f74000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:50.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.859+0000 7f2f6affd640 3 client.4408.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:50.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.859+0000 7f2f6affd640 3 client.4408.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:50.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.859+0000 7f2f6bfff640 1 --2- 172.21.15.60:0/2831369609 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2f5406c610 0x7f2f5406ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:50.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.859+0000 7f2f6bfff640 10 monclient: get_auth_request con 0x7f2f5406c610 auth_method 0 2024-03-27T21:39:50.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.859+0000 7f2f6affd640 20 client.4408.objecter dump_active .. 0 homeless 2024-03-27T21:39:50.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.860+0000 7f2f6affd640 10 client.4408.objecter ms_handle_connect 0x7f2f5406c610 2024-03-27T21:39:50.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.860+0000 7f2f6bfff640 1 --2- 172.21.15.60:0/2831369609 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2f5406c610 0x7f2f5406ead0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2f7413a2b0 tx=0x7f2f5c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:50.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.862+0000 7f2f6affd640 1 -- 172.21.15.60:0/2831369609 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2f64054750 con 0x7f2f7406a4d0 2024-03-27T21:39:50.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.862+0000 7f2f6affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:50.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:49.862+0000 7f2f6affd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:50.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.221+0000 7fda1ffff640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-27T21:39:50.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.221+0000 7fda1ffff640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-27T21:39:50.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.221+0000 7fda1ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:50.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.221+0000 7fda1ffff640 1 -- 172.21.15.60:0/2138659475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fda000052f0 con 0x7fda200acee0 2024-03-27T21:39:50.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.223+0000 7fda077fe640 1 -- 172.21.15.60:0/2138659475 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fda18043070 con 0x7fda200acee0 2024-03-27T21:39:50.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.223+0000 7fda077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-27T21:39:50.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.223+0000 7fda077fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:50.210 INFO:teuthology.orchestra.run.smithi060.stdout:60129542149 2024-03-27T21:39:50.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.226+0000 7fda057fa640 10 librados: watch_flush enter 2024-03-27T21:39:50.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.226+0000 7fda057fa640 10 librados: watch_flush exit 2024-03-27T21:39:50.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.226+0000 7fda057fa640 20 client.4368.objecter shutdown clearing up homeless session... 2024-03-27T21:39:50.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.226+0000 7fda057fa640 10 client.4368.objecter successfully canceled tick 2024-03-27T21:39:50.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.226+0000 7fda057fa640 1 -- 172.21.15.60:0/2138659475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fda0806c890 msgr2=0x7fda0806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.226+0000 7fda057fa640 1 --2- 172.21.15.60:0/2138659475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fda0806c890 0x7fda0806ed50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fda10006d40 tx=0x7fda10008040 comp rx=0 tx=0).stop 2024-03-27T21:39:50.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.226+0000 7fda057fa640 10 monclient: shutdown 2024-03-27T21:39:50.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.226+0000 7fda057fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:50.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.226+0000 7fda057fa640 1 -- 172.21.15.60:0/2138659475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda200acee0 msgr2=0x7fda2012de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.226+0000 7fda057fa640 1 --2- 172.21.15.60:0/2138659475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda200acee0 0x7fda2012de40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fda18032000 tx=0x7fda18004a40 comp rx=0 tx=0).stop 2024-03-27T21:39:50.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.226+0000 7fda057fa640 1 -- 172.21.15.60:0/2138659475 shutdown_connections 2024-03-27T21:39:50.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.226+0000 7fda057fa640 1 --2- 172.21.15.60:0/2138659475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda200acee0 0x7fda2012de40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.226+0000 7fda057fa640 1 --2- 172.21.15.60:0/2138659475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fda0806c890 0x7fda0806ed50 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fda10006d40 tx=0x7fda10008040 comp rx=0 tx=0).stop 2024-03-27T21:39:50.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.226+0000 7fda057fa640 1 --2- 172.21.15.60:0/2138659475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda20142d70 0x7fda200ac9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.227+0000 7fda057fa640 1 --2- 172.21.15.60:0/2138659475 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fda200b0ec0 0x7fda2008a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.227+0000 7fda057fa640 1 -- 172.21.15.60:0/2138659475 >> 172.21.15.60:0/2138659475 conn(0x7fda20060700 msgr2=0x7fda20068be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:50.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.227+0000 7fda057fa640 1 -- 172.21.15.60:0/2138659475 shutdown_connections 2024-03-27T21:39:50.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.229+0000 7fda057fa640 1 -- 172.21.15.60:0/2138659475 wait complete. 2024-03-27T21:39:50.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.229+0000 7fda057fa640 1 librados: shutdown 2024-03-27T21:39:50.232 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.2 2024-03-27T21:39:50.232 DEBUG:teuthology.parallel:result is None 2024-03-27T21:39:50.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.253+0000 7fa55c4db640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-27T21:39:50.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.253+0000 7fa55c4db640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-27T21:39:50.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.253+0000 7fa55c4db640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:50.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.253+0000 7fa55c4db640 1 -- 172.21.15.60:0/3119910424 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa53c0055c0 con 0x7fa54c038fe0 2024-03-27T21:39:50.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.254+0000 7fa54affd640 1 -- 172.21.15.60:0/3119910424 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa54401f020 con 0x7fa54c038fe0 2024-03-27T21:39:50.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.254+0000 7fa54affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-27T21:39:50.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.254+0000 7fa54affd640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:50.241 INFO:teuthology.orchestra.run.smithi060.stdout:60129542149 2024-03-27T21:39:50.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.256+0000 7fa55c4db640 10 librados: watch_flush enter 2024-03-27T21:39:50.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.256+0000 7fa55c4db640 10 librados: watch_flush exit 2024-03-27T21:39:50.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.256+0000 7fa55c4db640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-27T21:39:50.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.256+0000 7fa55c4db640 10 client.4340.objecter successfully canceled tick 2024-03-27T21:39:50.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.256+0000 7fa55c4db640 1 -- 172.21.15.60:0/3119910424 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa53806c5e0 msgr2=0x7fa53806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.256+0000 7fa55c4db640 1 --2- 172.21.15.60:0/3119910424 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa53806c5e0 0x7fa53806eaa0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa550036000 tx=0x7fa550008030 comp rx=0 tx=0).stop 2024-03-27T21:39:50.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.256+0000 7fa55c4db640 10 monclient: shutdown 2024-03-27T21:39:50.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.256+0000 7fa55c4db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:50.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.256+0000 7fa55c4db640 1 -- 172.21.15.60:0/3119910424 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa54c038fe0 msgr2=0x7fa54c0beef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.256+0000 7fa55c4db640 1 --2- 172.21.15.60:0/3119910424 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa54c038fe0 0x7fa54c0beef0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa54400cb60 tx=0x7fa544007410 comp rx=0 tx=0).stop 2024-03-27T21:39:50.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.257+0000 7fa55c4db640 1 -- 172.21.15.60:0/3119910424 shutdown_connections 2024-03-27T21:39:50.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.257+0000 7fa55c4db640 1 --2- 172.21.15.60:0/3119910424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa54c0cc600 0x7fa54c02dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.257+0000 7fa55c4db640 1 --2- 172.21.15.60:0/3119910424 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa54c038fe0 0x7fa54c0beef0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.257+0000 7fa55c4db640 1 --2- 172.21.15.60:0/3119910424 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa53806c5e0 0x7fa53806eaa0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.257+0000 7fa55c4db640 1 --2- 172.21.15.60:0/3119910424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa54c035640 0x7fa54c0c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.257+0000 7fa55c4db640 1 -- 172.21.15.60:0/3119910424 >> 172.21.15.60:0/3119910424 conn(0x7fa54c0065d0 msgr2=0x7fa54c0c3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:50.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.257+0000 7fa55c4db640 1 -- 172.21.15.60:0/3119910424 shutdown_connections 2024-03-27T21:39:50.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.257+0000 7fa55c4db640 1 -- 172.21.15.60:0/3119910424 wait complete. 2024-03-27T21:39:50.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.257+0000 7fa55c4db640 1 librados: shutdown 2024-03-27T21:39:50.254 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.0 2024-03-27T21:39:50.254 DEBUG:teuthology.parallel:result is None 2024-03-27T21:39:50.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.271+0000 7fdddcba9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-27T21:39:50.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.272+0000 7fdddcba9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-27T21:39:50.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.272+0000 7fdddcba9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:50.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.272+0000 7fdddcba9640 1 -- 172.21.15.60:0/256875073 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fddd80ad330 con 0x7fddd806e040 2024-03-27T21:39:50.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.272+0000 7fddc6ffd640 1 -- 172.21.15.60:0/256875073 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fddcc0a5ad0 con 0x7fddd806e040 2024-03-27T21:39:50.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.272+0000 7fddc6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-27T21:39:50.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.272+0000 7fddc6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:50.259 INFO:teuthology.orchestra.run.smithi060.stdout:68719476741 2024-03-27T21:39:50.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.274+0000 7f75c1c18640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-27T21:39:50.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.274+0000 7f75c1c18640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-27T21:39:50.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.274+0000 7f1e697fa640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-27T21:39:50.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.274+0000 7f75c1c18640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:50.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.274+0000 7f75c1c18640 1 -- 172.21.15.60:0/3470966457 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f75bc06a290 con 0x7f75bc06f6c0 2024-03-27T21:39:50.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.274+0000 7f1e697fa640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-27T21:39:50.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.274+0000 7f1e697fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:50.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.274+0000 7f1e697fa640 1 -- 172.21.15.60:0/3714157888 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f1e4c0055c0 con 0x7f1e74060970 2024-03-27T21:39:50.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.274+0000 7f75b97fa640 1 -- 172.21.15.60:0/3470966457 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f75b008a790 con 0x7f75bc06f6c0 2024-03-27T21:39:50.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.274+0000 7f75b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-27T21:39:50.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.274+0000 7f75b97fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:50.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.275+0000 7fdddcba9640 10 librados: watch_flush enter 2024-03-27T21:39:50.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.275+0000 7fdddcba9640 10 librados: watch_flush exit 2024-03-27T21:39:50.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.275+0000 7fdddcba9640 20 client.4395.objecter shutdown clearing up homeless session... 2024-03-27T21:39:50.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.275+0000 7fdddcba9640 10 client.4395.objecter successfully canceled tick 2024-03-27T21:39:50.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.275+0000 7fdddcba9640 1 -- 172.21.15.60:0/256875073 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fddb806c660 msgr2=0x7fddb806eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.275+0000 7fdddcba9640 1 --2- 172.21.15.60:0/256875073 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fddb806c660 0x7fddb806eb20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fddc800a9a0 tx=0x7fddc8008040 comp rx=0 tx=0).stop 2024-03-27T21:39:50.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.275+0000 7fdddcba9640 10 monclient: shutdown 2024-03-27T21:39:50.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.275+0000 7fdddcba9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:50.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.275+0000 7fdddcba9640 1 -- 172.21.15.60:0/256875073 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddd806e040 msgr2=0x7fddd8137c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.275+0000 7fdddcba9640 1 --2- 172.21.15.60:0/256875073 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddd806e040 0x7fddd8137c80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fddcc062040 tx=0x7fddcc004730 comp rx=0 tx=0).stop 2024-03-27T21:39:50.264 INFO:teuthology.orchestra.run.smithi060.stdout:55834574853 2024-03-27T21:39:50.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.274+0000 7f1e6b7fe640 1 -- 172.21.15.60:0/3714157888 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1e6407fe60 con 0x7f1e74060970 2024-03-27T21:39:50.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.275+0000 7f1e6b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-27T21:39:50.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.275+0000 7f1e6b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:50.265 INFO:teuthology.orchestra.run.smithi060.stdout:68719476742 2024-03-27T21:39:50.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.275+0000 7fdddcba9640 1 -- 172.21.15.60:0/256875073 shutdown_connections 2024-03-27T21:39:50.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.275+0000 7fdddcba9640 1 --2- 172.21.15.60:0/256875073 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddd806e040 0x7fddd8137c80 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.275+0000 7fdddcba9640 1 --2- 172.21.15.60:0/256875073 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fddd80abd90 0x7fddd806db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.275+0000 7fdddcba9640 1 --2- 172.21.15.60:0/256875073 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fddb806c660 0x7fddb806eb20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.275+0000 7fdddcba9640 1 --2- 172.21.15.60:0/256875073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fddc0004d00 0x7fddd80ab850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.275+0000 7fdddcba9640 1 -- 172.21.15.60:0/256875073 >> 172.21.15.60:0/256875073 conn(0x7fddd80608e0 msgr2=0x7fddd808a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:50.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.276+0000 7fdddcba9640 1 -- 172.21.15.60:0/256875073 shutdown_connections 2024-03-27T21:39:50.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.276+0000 7fdddcba9640 1 -- 172.21.15.60:0/256875073 wait complete. 2024-03-27T21:39:50.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.276+0000 7fdddcba9640 1 librados: shutdown 2024-03-27T21:39:50.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.277+0000 7f75c1c18640 10 librados: watch_flush enter 2024-03-27T21:39:50.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.277+0000 7f75c1c18640 10 librados: watch_flush exit 2024-03-27T21:39:50.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.277+0000 7f75c1c18640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-27T21:39:50.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.277+0000 7f75c1c18640 10 client.4396.objecter successfully canceled tick 2024-03-27T21:39:50.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.277+0000 7f75c1c18640 1 -- 172.21.15.60:0/3470966457 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f75a806c590 msgr2=0x7f75a806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.277+0000 7f75c1c18640 1 --2- 172.21.15.60:0/3470966457 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f75a806c590 0x7f75a806ea50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f75bc0b3ed0 tx=0x7f75ac009040 comp rx=0 tx=0).stop 2024-03-27T21:39:50.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.277+0000 7f75c1c18640 10 monclient: shutdown 2024-03-27T21:39:50.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.277+0000 7f75c1c18640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:50.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.277+0000 7f75c1c18640 1 -- 172.21.15.60:0/3470966457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75bc06f6c0 msgr2=0x7f75bc0ae4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.277+0000 7f75c1c18640 1 --2- 172.21.15.60:0/3470966457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75bc06f6c0 0x7f75bc0ae4a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f75b0043000 tx=0x7f75b00046c0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.277+0000 7f75c1c18640 1 -- 172.21.15.60:0/3470966457 shutdown_connections 2024-03-27T21:39:50.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.277+0000 7f75c1c18640 1 --2- 172.21.15.60:0/3470966457 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75bc0af6e0 0x7f75bc1d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.277+0000 7f75c1c18640 1 --2- 172.21.15.60:0/3470966457 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f75bc06faa0 0x7f75bc1d1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.277+0000 7f75c1c18640 1 --2- 172.21.15.60:0/3470966457 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f75a806c590 0x7f75a806ea50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.277+0000 7f75c1c18640 1 --2- 172.21.15.60:0/3470966457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75bc06f6c0 0x7f75bc0ae4a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.277+0000 7f75c1c18640 1 -- 172.21.15.60:0/3470966457 >> 172.21.15.60:0/3470966457 conn(0x7f75bc0b4ce0 msgr2=0x7f75bc0b02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:50.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.277+0000 7f75c1c18640 1 -- 172.21.15.60:0/3470966457 shutdown_connections 2024-03-27T21:39:50.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.277+0000 7f75c1c18640 1 -- 172.21.15.60:0/3470966457 wait complete. 2024-03-27T21:39:50.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.277+0000 7f75c1c18640 1 librados: shutdown 2024-03-27T21:39:50.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.278+0000 7f1e697fa640 10 librados: watch_flush enter 2024-03-27T21:39:50.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.278+0000 7f1e697fa640 10 librados: watch_flush exit 2024-03-27T21:39:50.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.278+0000 7f1e697fa640 20 client.4399.objecter shutdown clearing up homeless session... 2024-03-27T21:39:50.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.278+0000 7f1e697fa640 10 client.4399.objecter successfully canceled tick 2024-03-27T21:39:50.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.278+0000 7f1e697fa640 1 -- 172.21.15.60:0/3714157888 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1e5406c660 msgr2=0x7f1e5406eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.278+0000 7f1e697fa640 1 --2- 172.21.15.60:0/3714157888 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1e5406c660 0x7f1e5406eb20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c006fd0 tx=0x7f1e6c008040 comp rx=0 tx=0).stop 2024-03-27T21:39:50.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.278+0000 7f1e697fa640 10 monclient: shutdown 2024-03-27T21:39:50.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.278+0000 7f1e697fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:50.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.278+0000 7f1e697fa640 1 -- 172.21.15.60:0/3714157888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e74060970 msgr2=0x7f1e741456e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.278+0000 7f1e697fa640 1 --2- 172.21.15.60:0/3714157888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e74060970 0x7f1e741456e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1e640093e0 tx=0x7f1e64004a40 comp rx=0 tx=0).stop 2024-03-27T21:39:50.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.278+0000 7f1e697fa640 1 -- 172.21.15.60:0/3714157888 shutdown_connections 2024-03-27T21:39:50.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.278+0000 7f1e697fa640 1 --2- 172.21.15.60:0/3714157888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e74071b30 0x7f1e740a8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.278+0000 7f1e697fa640 1 --2- 172.21.15.60:0/3714157888 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e7406e750 0x7f1e740a8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.278+0000 7f1e697fa640 1 --2- 172.21.15.60:0/3714157888 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1e5406c660 0x7f1e5406eb20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.278+0000 7f1e697fa640 1 --2- 172.21.15.60:0/3714157888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e74060970 0x7f1e741456e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.278+0000 7f1e697fa640 1 -- 172.21.15.60:0/3714157888 >> 172.21.15.60:0/3714157888 conn(0x7f1e740b3d60 msgr2=0x7f1e74148e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:50.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.278+0000 7f1e697fa640 1 -- 172.21.15.60:0/3714157888 shutdown_connections 2024-03-27T21:39:50.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.278+0000 7f1e697fa640 1 -- 172.21.15.60:0/3714157888 wait complete. 2024-03-27T21:39:50.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.278+0000 7f1e697fa640 1 librados: shutdown 2024-03-27T21:39:50.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.280+0000 7efe1359e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-27T21:39:50.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.280+0000 7efe1359e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-27T21:39:50.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.280+0000 7efe1359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:50.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.280+0000 7efe1359e640 1 -- 172.21.15.60:0/1559011480 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7efdf80052f0 con 0x7efe14148f10 2024-03-27T21:39:50.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.281+0000 7f2f78fd5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-27T21:39:50.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.281+0000 7f2f78fd5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-27T21:39:50.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.281+0000 7f2f78fd5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:50.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.281+0000 7f2f78fd5640 1 -- 172.21.15.60:0/2831369609 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f2f580055c0 con 0x7f2f7406a4d0 2024-03-27T21:39:50.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.281+0000 7efe02ffd640 1 -- 172.21.15.60:0/1559011480 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efe0c018070 con 0x7efe14148f10 2024-03-27T21:39:50.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.281+0000 7efe02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-27T21:39:50.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.281+0000 7efe02ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:50.275 INFO:teuthology.orchestra.run.smithi060.stdout:60129542149 2024-03-27T21:39:50.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.281+0000 7f2f6affd640 1 -- 172.21.15.60:0/2831369609 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2f64021090 con 0x7f2f7406a4d0 2024-03-27T21:39:50.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.281+0000 7f2f6affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-27T21:39:50.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.281+0000 7f2f6affd640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:50.275 INFO:teuthology.orchestra.run.smithi060.stdout:60129542149 2024-03-27T21:39:50.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.282+0000 7fb3f2cd2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-27T21:39:50.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.282+0000 7fb3f2cd2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-27T21:39:50.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.282+0000 7fb3f2cd2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:39:50.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.282+0000 7fb3f2cd2640 1 -- 172.21.15.60:0/491830654 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fb3ec131ca0 con 0x7fb3ec1416f0 2024-03-27T21:39:50.276 INFO:teuthology.orchestra.run.smithi060.stdout:60129542149 2024-03-27T21:39:50.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.282+0000 7fb3e97fa640 1 -- 172.21.15.60:0/491830654 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb3d8014020 con 0x7fb3ec1416f0 2024-03-27T21:39:50.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.282+0000 7fb3e97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-27T21:39:50.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.282+0000 7fb3e97fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:50.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7f2f78fd5640 10 librados: watch_flush enter 2024-03-27T21:39:50.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7efe1359e640 10 librados: watch_flush enter 2024-03-27T21:39:50.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7efe1359e640 10 librados: watch_flush exit 2024-03-27T21:39:50.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7efe1359e640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-27T21:39:50.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7efe1359e640 10 client.4337.objecter successfully canceled tick 2024-03-27T21:39:50.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7f2f78fd5640 10 librados: watch_flush exit 2024-03-27T21:39:50.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7efe1359e640 1 -- 172.21.15.60:0/1559011480 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7efdf406c660 msgr2=0x7efdf406eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7efe1359e640 1 --2- 172.21.15.60:0/1559011480 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7efdf406c660 0x7efdf406eb20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efe04009d30 tx=0x7efe04009190 comp rx=0 tx=0).stop 2024-03-27T21:39:50.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7f2f78fd5640 20 client.4408.objecter shutdown clearing up homeless session... 2024-03-27T21:39:50.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7efe1359e640 10 monclient: shutdown 2024-03-27T21:39:50.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7efe1359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:50.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7efe1359e640 1 -- 172.21.15.60:0/1559011480 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efe14148f10 msgr2=0x7efe14143a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7efe1359e640 1 --2- 172.21.15.60:0/1559011480 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efe14148f10 0x7efe14143a60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efe0c001560 tx=0x7efe0c00cbe0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7f2f78fd5640 10 client.4408.objecter successfully canceled tick 2024-03-27T21:39:50.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7f2f78fd5640 1 -- 172.21.15.60:0/2831369609 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2f5406c610 msgr2=0x7f2f5406ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7f2f78fd5640 1 --2- 172.21.15.60:0/2831369609 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2f5406c610 0x7f2f5406ead0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2f7413a2b0 tx=0x7f2f5c008040 comp rx=0 tx=0).stop 2024-03-27T21:39:50.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7f2f78fd5640 10 monclient: shutdown 2024-03-27T21:39:50.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7efe1359e640 1 -- 172.21.15.60:0/1559011480 shutdown_connections 2024-03-27T21:39:50.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7efe1359e640 1 --2- 172.21.15.60:0/1559011480 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efe14148f10 0x7efe14143a60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7efe1359e640 1 --2- 172.21.15.60:0/1559011480 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7efdf406c660 0x7efdf406eb20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7efe1359e640 1 --2- 172.21.15.60:0/1559011480 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe140ad910 0x7efe14143520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7efe1359e640 1 --2- 172.21.15.60:0/1559011480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe1406ea80 0x7efe14145e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7efe1359e640 1 -- 172.21.15.60:0/1559011480 >> 172.21.15.60:0/1559011480 conn(0x7efe140b30f0 msgr2=0x7efe1413e050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:50.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7efe1359e640 1 -- 172.21.15.60:0/1559011480 shutdown_connections 2024-03-27T21:39:50.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7efe1359e640 1 -- 172.21.15.60:0/1559011480 wait complete. 2024-03-27T21:39:50.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7f2f78fd5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:50.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7efe1359e640 1 librados: shutdown 2024-03-27T21:39:50.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7f2f78fd5640 1 -- 172.21.15.60:0/2831369609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f7406a4d0 msgr2=0x7f2f74139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7f2f78fd5640 1 --2- 172.21.15.60:0/2831369609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f7406a4d0 0x7f2f74139140 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2f6400e930 tx=0x7f2f6400edf0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7f2f78fd5640 1 -- 172.21.15.60:0/2831369609 shutdown_connections 2024-03-27T21:39:50.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7f2f78fd5640 1 --2- 172.21.15.60:0/2831369609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f74071bf0 0x7f2f74142ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7f2f78fd5640 1 --2- 172.21.15.60:0/2831369609 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2f7406a8b0 0x7f2f74143cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7f2f78fd5640 1 --2- 172.21.15.60:0/2831369609 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2f5406c610 0x7f2f5406ead0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7f2f78fd5640 1 --2- 172.21.15.60:0/2831369609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f7406a4d0 0x7f2f74139140 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7f2f78fd5640 1 -- 172.21.15.60:0/2831369609 >> 172.21.15.60:0/2831369609 conn(0x7f2f740b4e50 msgr2=0x7f2f740b44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:50.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7f2f78fd5640 1 -- 172.21.15.60:0/2831369609 shutdown_connections 2024-03-27T21:39:50.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7f2f78fd5640 1 -- 172.21.15.60:0/2831369609 wait complete. 2024-03-27T21:39:50.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.284+0000 7f2f78fd5640 1 librados: shutdown 2024-03-27T21:39:50.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.286+0000 7fb3f2cd2640 10 librados: watch_flush enter 2024-03-27T21:39:50.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.286+0000 7fb3f2cd2640 10 librados: watch_flush exit 2024-03-27T21:39:50.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.286+0000 7fb3f2cd2640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-27T21:39:50.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.286+0000 7fb3f2cd2640 10 client.4405.objecter successfully canceled tick 2024-03-27T21:39:50.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.286+0000 7fb3f2cd2640 1 -- 172.21.15.60:0/491830654 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb3d006c690 msgr2=0x7fb3d006eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.286+0000 7fb3f2cd2640 1 --2- 172.21.15.60:0/491830654 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb3d006c690 0x7fb3d006eb50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0036000 tx=0x7fb3e003c040 comp rx=0 tx=0).stop 2024-03-27T21:39:50.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.286+0000 7fb3f2cd2640 10 monclient: shutdown 2024-03-27T21:39:50.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.286+0000 7fb3f2cd2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:50.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.286+0000 7fb3f2cd2640 1 -- 172.21.15.60:0/491830654 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3ec1416f0 msgr2=0x7fb3ec13aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.286+0000 7fb3f2cd2640 1 --2- 172.21.15.60:0/491830654 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3ec1416f0 0x7fb3ec13aa50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb3d800b480 tx=0x7fb3d800b940 comp rx=0 tx=0).stop 2024-03-27T21:39:50.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.287+0000 7fb3f2cd2640 1 -- 172.21.15.60:0/491830654 shutdown_connections 2024-03-27T21:39:50.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.287+0000 7fb3f2cd2640 1 --2- 172.21.15.60:0/491830654 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb3ec13ba60 0x7fb3ec1312a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.287+0000 7fb3f2cd2640 1 --2- 172.21.15.60:0/491830654 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb3d006c690 0x7fb3d006eb50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.287+0000 7fb3f2cd2640 1 --2- 172.21.15.60:0/491830654 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3ec1416f0 0x7fb3ec13aa50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.287+0000 7fb3f2cd2640 1 --2- 172.21.15.60:0/491830654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3ec13b020 0x7fb3ec134bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.287+0000 7fb3f2cd2640 1 -- 172.21.15.60:0/491830654 >> 172.21.15.60:0/491830654 conn(0x7fb3ec0608e0 msgr2=0x7fb3ec140310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:50.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.287+0000 7fb3f2cd2640 1 -- 172.21.15.60:0/491830654 shutdown_connections 2024-03-27T21:39:50.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.287+0000 7fb3f2cd2640 1 -- 172.21.15.60:0/491830654 wait complete. 2024-03-27T21:39:50.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.287+0000 7fb3f2cd2640 1 librados: shutdown 2024-03-27T21:39:50.287 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476741 for osd.3 2024-03-27T21:39:50.288 DEBUG:teuthology.parallel:result is None 2024-03-27T21:39:50.288 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476742 for osd.5 2024-03-27T21:39:50.289 DEBUG:teuthology.parallel:result is None 2024-03-27T21:39:50.289 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.7 2024-03-27T21:39:50.289 DEBUG:teuthology.parallel:result is None 2024-03-27T21:39:50.290 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.1 2024-03-27T21:39:50.290 DEBUG:teuthology.parallel:result is None 2024-03-27T21:39:50.291 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.6 2024-03-27T21:39:50.291 DEBUG:teuthology.parallel:result is None 2024-03-27T21:39:50.292 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.4 2024-03-27T21:39:50.292 DEBUG:teuthology.parallel:result is None 2024-03-27T21:39:50.292 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-27T21:39:50.292 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T21:39:50.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.444+0000 7f686935c640 10 monclient: get_monmap_and_config 2024-03-27T21:39:50.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.444+0000 7f686935c640 10 monclient: build_initial_monmap 2024-03-27T21:39:50.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.444+0000 7f686935c640 10 monclient: monmap: 2024-03-27T21:39:50.431 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:50.431 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:50.431 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:50.445089+0000 2024-03-27T21:39:50.431 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:50.445089+0000 2024-03-27T21:39:50.431 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:50.431 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:50.431 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:50.431 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:50.431 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:50.431 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:50.431 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:50.431 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:50.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.444+0000 7f686935c640 1 Processor -- start 2024-03-27T21:39:50.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.444+0000 7f686935c640 1 -- start start 2024-03-27T21:39:50.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.444+0000 7f686935c640 10 monclient: init 2024-03-27T21:39:50.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f686935c640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:50.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f686935c640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-03-27T21:39:50.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f686935c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f68640b40b0 0x7f68640a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:50.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f686935c640 10 monclient(hunting): picked mon.noname-a con 0x7f68640b40b0 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:50.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f686935c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f686413b020 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:50.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f686935c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f686413b480 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:50.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f686935c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686413b8d0 con 0x7f686413b020 2024-03-27T21:39:50.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f686935c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68640b3830 con 0x7f686413b480 2024-03-27T21:39:50.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f686935c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:50.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f686935c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68640b4490 con 0x7f68640b40b0 2024-03-27T21:39:50.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f686935c640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:50.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f686935c640 10 monclient(hunting): authenticate will time out at 831.666284s 2024-03-27T21:39:50.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f6862ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f68640b40b0 0x7f68640a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:50.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f6862ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f68640b40b0 0x7f68640a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48044/0 (socket says 172.21.15.60:48044) 2024-03-27T21:39:50.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f6862ffd640 1 -- 172.21.15.60:0/3677085466 learned_addr learned my addr 172.21.15.60:0/3677085466 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:50.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f6862ffd640 10 monclient(hunting): get_auth_request con 0x7f68640b40b0 auth_method 0 2024-03-27T21:39:50.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f6862ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:50.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f6862ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:50.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f6862ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:50.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f68627fc640 1 --1- 172.21.15.60:0/3677085466 >> v1:172.21.15.60:6789/0 conn(0x7f686413b020 0x7f68641416f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43630/0 (socket says 172.21.15.60:43630) 2024-03-27T21:39:50.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f6862ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:50.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f6862ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:50.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f6862ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:50.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f6862ffd640 10 monclient(hunting): handle_auth_done global_id 4404 payload 274 2024-03-27T21:39:50.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f6862ffd640 1 -- 172.21.15.60:0/3677085466 >> v1:172.21.15.191:6789/0 conn(0x7f686413b480 legacy=0x7f686414bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:50.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f6862ffd640 1 -- 172.21.15.60:0/3677085466 >> v1:172.21.15.60:6789/0 conn(0x7f686413b020 legacy=0x7f68641416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f6862ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:39:50.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f6862ffd640 1 monclient: found mon.noname-a 2024-03-27T21:39:50.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f6862ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:50.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f6862ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:50.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f6861ffb640 1 -- 172.21.15.60:0/3677085466 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9656567 0 0) 0x7f686413b8d0 con 0x7f686413b020 2024-03-27T21:39:50.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f6862ffd640 1 -- 172.21.15.60:0/3677085466 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68640aa520 con 0x7f68640b40b0 2024-03-27T21:39:50.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.445+0000 7f6861ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:39:50.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.446+0000 7f6862ffd640 1 --2- 172.21.15.60:0/3677085466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68640b40b0 0x7f68640a8b90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6854009e50 tx=0x7f685402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a749598ce41ec9c server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:50.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.446+0000 7f6861ffb640 1 -- 172.21.15.60:0/3677085466 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685402ca80 con 0x7f68640b40b0 2024-03-27T21:39:50.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.446+0000 7f6861ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:50.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.446+0000 7f6861ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:50.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.446+0000 7f6861ffb640 10 monclient: dump: 2024-03-27T21:39:50.444 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:50.444 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:50.444 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:50.444 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:50.444 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:50.445 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:50.445 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:50.445 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:50.445 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:50.445 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:50.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.446+0000 7f6861ffb640 10 monclient: _finish_auth 0 2024-03-27T21:39:50.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.446+0000 7f6861ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:39:50.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.446+0000 7f6861ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:50.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.446+0000 7f686935c640 5 monclient: authenticate success, global_id 4404 2024-03-27T21:39:50.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.446+0000 7f686935c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:50.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.446+0000 7f6861ffb640 1 -- 172.21.15.60:0/3677085466 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685402cbe0 con 0x7f68640b40b0 2024-03-27T21:39:50.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.446+0000 7f6861ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:50.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.446+0000 7f686935c640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:50.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.446+0000 7f686935c640 4 set_mon_vals no callback set 2024-03-27T21:39:50.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.446+0000 7f6861ffb640 1 -- 172.21.15.60:0/3677085466 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68540343f0 con 0x7f68640b40b0 2024-03-27T21:39:50.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.446+0000 7f6861ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:50.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.446+0000 7f6861ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:50.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.446+0000 7f6861ffb640 10 monclient: dump: 2024-03-27T21:39:50.446 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:50.446 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:50.446 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:50.446 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:50.446 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:50.446 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:50.446 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:50.446 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:50.446 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:50.447 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:50.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.447+0000 7f686935c640 10 monclient: shutdown 2024-03-27T21:39:50.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.447+0000 7f686935c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:50.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.447+0000 7f686935c640 1 -- 172.21.15.60:0/3677085466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68640b40b0 msgr2=0x7f68640a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.447+0000 7f686935c640 1 --2- 172.21.15.60:0/3677085466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68640b40b0 0x7f68640a8b90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6854009e50 tx=0x7f685402bee0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.447+0000 7f686935c640 1 -- 172.21.15.60:0/3677085466 shutdown_connections 2024-03-27T21:39:50.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.447+0000 7f686935c640 1 --2- 172.21.15.60:0/3677085466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68640b40b0 0x7f68640a8b90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.447+0000 7f686935c640 1 -- 172.21.15.60:0/3677085466 >> 172.21.15.60:0/3677085466 conn(0x7f68640608e0 msgr2=0x7f6864060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:50.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.447+0000 7f686935c640 1 -- 172.21.15.60:0/3677085466 shutdown_connections 2024-03-27T21:39:50.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.447+0000 7f686935c640 1 -- 172.21.15.60:0/3677085466 wait complete. 2024-03-27T21:39:50.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.447+0000 7f686935c640 10 monclient: build_initial_monmap 2024-03-27T21:39:50.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.447+0000 7f686935c640 10 monclient: monmap: 2024-03-27T21:39:50.447 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:50.447 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:50.448 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:50.448 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:50.448 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:50.448 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:50.448 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:50.448 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:50.448 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:50.448 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:50.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 1 librados: starting msgr at 2024-03-27T21:39:50.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 1 librados: starting objecter 2024-03-27T21:39:50.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 1 Processor -- start 2024-03-27T21:39:50.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 1 -- start start 2024-03-27T21:39:50.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 1 librados: setting wanted keys 2024-03-27T21:39:50.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 1 librados: calling monclient init 2024-03-27T21:39:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 10 monclient: init 2024-03-27T21:39:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:39:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f68640b40b0 0x7f68641d9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 10 monclient(hunting): picked mon.noname-c con 0x7f68640b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f686413b020 0x7f68641d9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 10 monclient(hunting): picked mon.noname-a con 0x7f686413b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f686413b480 0x7f6864138d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 10 monclient(hunting): picked mon.noname-b con 0x7f686413b480 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68640ae2d0 con 0x7f686413b020 2024-03-27T21:39:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68640b0300 con 0x7f686413b480 2024-03-27T21:39:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:50.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68640b0470 con 0x7f68640b40b0 2024-03-27T21:39:50.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:50.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.448+0000 7f686935c640 10 monclient(hunting): authenticate will time out at 831.669861s 2024-03-27T21:39:50.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68627fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f686413b020 0x7f68641d9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:50.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f6862ffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f68640b40b0 0x7f68641d9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:50.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f6862ffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f68640b40b0 0x7f68641d9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47864/0 (socket says 172.21.15.60:47864) 2024-03-27T21:39:50.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68627fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f686413b020 0x7f68641d9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48060/0 (socket says 172.21.15.60:48060) 2024-03-27T21:39:50.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f6862ffd640 1 -- 172.21.15.60:0/2691189169 learned_addr learned my addr 172.21.15.60:0/2691189169 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:50.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f6862ffd640 10 monclient(hunting): get_auth_request con 0x7f68640b40b0 auth_method 0 2024-03-27T21:39:50.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f6862ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:50.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f6862ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:50.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f6862ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:50.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68627fc640 10 monclient(hunting): get_auth_request con 0x7f686413b020 auth_method 0 2024-03-27T21:39:50.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:50.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68627fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:50.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:50.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f6862ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:50.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f6862ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:50.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f6862ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:50.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:50.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:50.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:50.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68637fe640 1 --2- 172.21.15.60:0/2691189169 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f686413b480 0x7f6864138d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:50.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68627fc640 10 monclient(hunting): handle_auth_done global_id 4410 payload 824 2024-03-27T21:39:50.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68627fc640 1 -- 172.21.15.60:0/2691189169 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f68640b40b0 msgr2=0x7f68641d9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68627fc640 1 --2- 172.21.15.60:0/2691189169 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f68640b40b0 0x7f68641d9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68627fc640 1 -- 172.21.15.60:0/2691189169 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f686413b480 msgr2=0x7f6864138d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68627fc640 1 --2- 172.21.15.60:0/2691189169 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f686413b480 0x7f6864138d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68627fc640 10 monclient: _finish_hunting 0 2024-03-27T21:39:50.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68627fc640 1 monclient: found mon.noname-a 2024-03-27T21:39:50.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:50.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68627fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:50.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68627fc640 1 -- 172.21.15.60:0/2691189169 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6854009b30 con 0x7f686413b020 2024-03-27T21:39:50.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68637fe640 10 monclient: get_auth_request con 0x7f686413b480 auth_method 0 2024-03-27T21:39:50.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68637fe640 1 --2- 172.21.15.60:0/2691189169 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f686413b480 0x7f6864138d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:39:50.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f6862ffd640 1 --2- 172.21.15.60:0/2691189169 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f68640b40b0 0x7f68641d9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:50.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68627fc640 1 --2- 172.21.15.60:0/2691189169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f686413b020 0x7f68641d9d50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f684c00c480 tx=0x7f684c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:50.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f685b7fe640 10 client.?.objecter ms_handle_connect 0x7f686413b020 2024-03-27T21:39:50.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f685b7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:50.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f685b7fe640 1 -- 172.21.15.60:0/2691189169 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684c0074e0 con 0x7f686413b020 2024-03-27T21:39:50.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f685b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:50.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f685b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:50.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f685b7fe640 10 monclient: dump: 2024-03-27T21:39:50.453 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:50.453 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:50.453 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:50.453 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:50.453 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:50.453 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:50.453 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:50.453 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:50.453 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:50.453 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:50.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f685b7fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:50.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f685b7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:50.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f685b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:50.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f685b7fe640 1 -- 172.21.15.60:0/2691189169 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f684c007640 con 0x7f686413b020 2024-03-27T21:39:50.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f686935c640 5 monclient: authenticate success, global_id 4410 2024-03-27T21:39:50.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f686935c640 10 monclient: _renew_subs 2024-03-27T21:39:50.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f686935c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:50.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f686935c640 1 -- 172.21.15.60:0/2691189169 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68641392b0 con 0x7f686413b020 2024-03-27T21:39:50.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f685b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:50.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f685b7fe640 1 -- 172.21.15.60:0/2691189169 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684c004a20 con 0x7f686413b020 2024-03-27T21:39:50.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f685b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:50.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f685b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:50.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f685b7fe640 10 monclient: dump: 2024-03-27T21:39:50.454 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:50.454 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:50.454 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:50.454 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:50.455 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:50.455 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:50.455 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:50.455 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:50.455 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:50.455 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:50.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f68617fa640 4 set_mon_vals no callback set 2024-03-27T21:39:50.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f686935c640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:50.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f686935c640 10 monclient: _renew_subs 2024-03-27T21:39:50.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f686935c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:50.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f686935c640 1 -- 172.21.15.60:0/2691189169 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6864139810 con 0x7f686413b020 2024-03-27T21:39:50.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.449+0000 7f686935c640 1 librados: init done 2024-03-27T21:39:50.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.450+0000 7f686935c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:50.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.450+0000 7f686935c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:50.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.450+0000 7f686935c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:50.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.450+0000 7f686935c640 1 -- 172.21.15.60:0/2691189169 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6864069470 con 0x7f686413b020 2024-03-27T21:39:50.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.451+0000 7f685b7fe640 1 -- 172.21.15.60:0/2691189169 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f684c004b80 con 0x7f686413b020 2024-03-27T21:39:50.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.451+0000 7f685b7fe640 1 --2- 172.21.15.60:0/2691189169 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f684406c590 0x7f684406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:50.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.451+0000 7f685b7fe640 1 -- 172.21.15.60:0/2691189169 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f684c088540 con 0x7f686413b020 2024-03-27T21:39:50.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.451+0000 7f685b7fe640 10 client.4410.objecter ms_dispatch 0x7f6864000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:50.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.451+0000 7f685b7fe640 3 client.4410.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:50.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.451+0000 7f685b7fe640 3 client.4410.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:50.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.451+0000 7f6862ffd640 1 --2- 172.21.15.60:0/2691189169 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f684406c590 0x7f684406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:50.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.451+0000 7f6862ffd640 10 monclient: get_auth_request con 0x7f684406c590 auth_method 0 2024-03-27T21:39:50.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.451+0000 7f685b7fe640 20 client.4410.objecter dump_active .. 0 homeless 2024-03-27T21:39:50.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.451+0000 7f685b7fe640 10 client.4410.objecter ms_handle_connect 0x7f684406c590 2024-03-27T21:39:50.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.451+0000 7f6862ffd640 1 --2- 172.21.15.60:0/2691189169 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f684406c590 0x7f684406ea50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f685402bf20 tx=0x7f6854030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:50.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.455+0000 7f685b7fe640 1 -- 172.21.15.60:0/2691189169 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f684c053a50 con 0x7f686413b020 2024-03-27T21:39:50.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.455+0000 7f685b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:50.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.455+0000 7f685b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:50.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.632+0000 7f686935c640 1 -- 172.21.15.60:0/2691189169 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f68640a90d0 con 0x7f684406c590 2024-03-27T21:39:50.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.634+0000 7f685b7fe640 1 -- 172.21.15.60:0/2691189169 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48318 (secure 0 0 0) 0x7f6854003990 con 0x7f684406c590 2024-03-27T21:39:50.621 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:39:50.622 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-03-27T21:39:50.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.636+0000 7f686935c640 10 librados: watch_flush enter 2024-03-27T21:39:50.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.636+0000 7f686935c640 10 librados: watch_flush exit 2024-03-27T21:39:50.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.636+0000 7f686935c640 20 client.4410.objecter shutdown clearing up homeless session... 2024-03-27T21:39:50.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.636+0000 7f686935c640 10 client.4410.objecter successfully canceled tick 2024-03-27T21:39:50.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.636+0000 7f686935c640 1 -- 172.21.15.60:0/2691189169 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f684406c590 msgr2=0x7f684406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.636+0000 7f686935c640 1 --2- 172.21.15.60:0/2691189169 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f684406c590 0x7f684406ea50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f685402bf20 tx=0x7f6854030000 comp rx=0 tx=0).stop 2024-03-27T21:39:50.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.636+0000 7f686935c640 10 monclient: shutdown 2024-03-27T21:39:50.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.636+0000 7f686935c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:50.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.636+0000 7f686935c640 1 -- 172.21.15.60:0/2691189169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f686413b020 msgr2=0x7f68641d9d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.636+0000 7f686935c640 1 --2- 172.21.15.60:0/2691189169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f686413b020 0x7f68641d9d50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f684c00c480 tx=0x7f684c00c940 comp rx=0 tx=0).stop 2024-03-27T21:39:50.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.636+0000 7f686935c640 1 -- 172.21.15.60:0/2691189169 shutdown_connections 2024-03-27T21:39:50.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.636+0000 7f686935c640 1 --2- 172.21.15.60:0/2691189169 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f684406c590 0x7f684406ea50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.636+0000 7f686935c640 1 --2- 172.21.15.60:0/2691189169 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f686413b480 0x7f6864138d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.636+0000 7f686935c640 1 --2- 172.21.15.60:0/2691189169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f686413b020 0x7f68641d9d50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.636+0000 7f686935c640 1 --2- 172.21.15.60:0/2691189169 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f68640b40b0 0x7f68641d9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.636+0000 7f686935c640 1 -- 172.21.15.60:0/2691189169 >> 172.21.15.60:0/2691189169 conn(0x7f68640608e0 msgr2=0x7f68640b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:50.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.637+0000 7f686935c640 1 -- 172.21.15.60:0/2691189169 shutdown_connections 2024-03-27T21:39:50.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.637+0000 7f686935c640 1 -- 172.21.15.60:0/2691189169 wait complete. 2024-03-27T21:39:50.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.637+0000 7f686935c640 1 librados: shutdown 2024-03-27T21:39:50.634 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-03-27T21:39:49.390687+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":221896,"kb_used_data":1620,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":754752824,"statfs":{"total":773094113280,"available":772866891776,"internally_reserved":0,"allocated":1658880,"data_stored":1573540,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":12742,"internal_metadata":225431098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001235"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:46.474962+0000","last_change":"2024-03-27T21:39:46.470433+0000","last_active":"2024-03-27T21:39:46.474962+0000","last_peered":"2024-03-27T21:39:46.474962+0000","last_clean":"2024-03-27T21:39:46.474962+0000","last_became_active":"2024-03-27T21:39:46.470153+0000","last_became_peered":"2024-03-27T21:39:46.470153+0000","last_unstale":"2024-03-27T21:39:46.474962+0000","last_undegraded":"2024-03-27T21:39:46.474962+0000","last_fullsized":"2024-03-27T21:39:46.474962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T21:39:45.811197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:46.468505+0000","last_change":"2024-03-27T21:39:46.467893+0000","last_active":"2024-03-27T21:39:46.468505+0000","last_peered":"2024-03-27T21:39:46.468505+0000","last_clean":"2024-03-27T21:39:46.468505+0000","last_became_active":"2024-03-27T21:39:46.467576+0000","last_became_peered":"2024-03-27T21:39:46.467576+0000","last_unstale":"2024-03-27T21:39:46.468505+0000","last_undegraded":"2024-03-27T21:39:46.468505+0000","last_fullsized":"2024-03-27T21:39:46.468505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:29:50.190006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:47.405006+0000","last_change":"2024-03-27T21:39:46.679788+0000","last_active":"2024-03-27T21:39:47.405006+0000","last_peered":"2024-03-27T21:39:47.405006+0000","last_clean":"2024-03-27T21:39:47.405006+0000","last_became_active":"2024-03-27T21:39:46.679561+0000","last_became_peered":"2024-03-27T21:39:46.679561+0000","last_unstale":"2024-03-27T21:39:47.405006+0000","last_undegraded":"2024-03-27T21:39:47.405006+0000","last_fullsized":"2024-03-27T21:39:47.405006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T23:53:50.373923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:46.679751+0000","last_change":"2024-03-27T21:39:46.679751+0000","last_active":"2024-03-27T21:39:46.679751+0000","last_peered":"2024-03-27T21:39:46.679751+0000","last_clean":"2024-03-27T21:39:46.679751+0000","last_became_active":"2024-03-27T21:39:46.679509+0000","last_became_peered":"2024-03-27T21:39:46.679509+0000","last_unstale":"2024-03-27T21:39:46.679751+0000","last_undegraded":"2024-03-27T21:39:46.679751+0000","last_fullsized":"2024-03-27T21:39:46.679751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:43:02.145158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T21:39:47.458020+0000","last_change":"2024-03-27T21:39:46.459503+0000","last_active":"2024-03-27T21:39:47.458020+0000","last_peered":"2024-03-27T21:39:47.458020+0000","last_clean":"2024-03-27T21:39:47.458020+0000","last_became_active":"2024-03-27T21:39:46.459225+0000","last_became_peered":"2024-03-27T21:39:46.459225+0000","last_unstale":"2024-03-27T21:39:47.458020+0000","last_undegraded":"2024-03-27T21:39:47.458020+0000","last_fullsized":"2024-03-27T21:39:47.458020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:17:43.512127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:46.680826+0000","last_change":"2024-03-27T21:39:46.679791+0000","last_active":"2024-03-27T21:39:46.680826+0000","last_peered":"2024-03-27T21:39:46.680826+0000","last_clean":"2024-03-27T21:39:46.680826+0000","last_became_active":"2024-03-27T21:39:46.679503+0000","last_became_peered":"2024-03-27T21:39:46.679503+0000","last_unstale":"2024-03-27T21:39:46.680826+0000","last_undegraded":"2024-03-27T21:39:46.680826+0000","last_fullsized":"2024-03-27T21:39:46.680826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:07:25.151346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:47.404983+0000","last_change":"2024-03-27T21:39:46.458326+0000","last_active":"2024-03-27T21:39:47.404983+0000","last_peered":"2024-03-27T21:39:47.404983+0000","last_clean":"2024-03-27T21:39:47.404983+0000","last_became_active":"2024-03-27T21:39:46.458074+0000","last_became_peered":"2024-03-27T21:39:46.458074+0000","last_unstale":"2024-03-27T21:39:47.404983+0000","last_undegraded":"2024-03-27T21:39:47.404983+0000","last_fullsized":"2024-03-27T21:39:47.404983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:06:29.798229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T21:39:47.458062+0000","last_change":"2024-03-27T21:39:46.459166+0000","last_active":"2024-03-27T21:39:47.458062+0000","last_peered":"2024-03-27T21:39:47.458062+0000","last_clean":"2024-03-27T21:39:47.458062+0000","last_became_active":"2024-03-27T21:39:46.458958+0000","last_became_peered":"2024-03-27T21:39:46.458958+0000","last_unstale":"2024-03-27T21:39:47.458062+0000","last_undegraded":"2024-03-27T21:39:47.458062+0000","last_fullsized":"2024-03-27T21:39:47.458062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T22:18:37.511546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":163,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:46.455960+0000","last_change":"2024-03-27T21:39:39.866806+0000","last_active":"2024-03-27T21:39:46.455960+0000","last_peered":"2024-03-27T21:39:46.455960+0000","last_clean":"2024-03-27T21:39:46.455960+0000","last_became_active":"2024-03-27T21:39:39.865948+0000","last_became_peered":"2024-03-27T21:39:39.865948+0000","last_unstale":"2024-03-27T21:39:46.455960+0000","last_undegraded":"2024-03-27T21:39:46.455960+0000","last_fullsized":"2024-03-27T21:39:46.455960+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_clean_scrub_stamp":"2024-03-27T21:39:38.411432+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:36:49.375869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":16,"seq":68719476742,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]}]},{"osd":3,"up_from":16,"seq":68719476741,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31848905},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]}]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]}]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":598418,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]}]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]}]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":592670,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T21:39:50.635 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T21:39:50.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.748+0000 7fa46b3c0640 10 monclient: get_monmap_and_config 2024-03-27T21:39:50.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.748+0000 7fa46b3c0640 10 monclient: build_initial_monmap 2024-03-27T21:39:50.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.748+0000 7fa46b3c0640 10 monclient: monmap: 2024-03-27T21:39:50.735 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:50.735 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:50.735 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:50.749056+0000 2024-03-27T21:39:50.735 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:50.749056+0000 2024-03-27T21:39:50.735 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:50.735 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:50.735 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:50.735 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:50.736 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:50.736 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:50.736 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:50.736 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:50.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.748+0000 7fa46b3c0640 1 Processor -- start 2024-03-27T21:39:50.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.748+0000 7fa46b3c0640 1 -- start start 2024-03-27T21:39:50.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.748+0000 7fa46b3c0640 10 monclient: init 2024-03-27T21:39:50.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46b3c0640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:50.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46b3c0640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-03-27T21:39:50.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46b3c0640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa4640b40b0 0x7fa4640a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:50.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46b3c0640 10 monclient(hunting): picked mon.noname-b con 0x7fa4640b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:50.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46b3c0640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa46413b020 0x7fa46413b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:50.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46b3c0640 10 monclient(hunting): picked mon.noname-c con 0x7fa46413b020 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:50.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46b3c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa4641416f0 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:50.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46b3c0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4640b3830 con 0x7fa4641416f0 2024-03-27T21:39:50.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46b3c0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:50.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46b3c0640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4640b4490 con 0x7fa46413b020 2024-03-27T21:39:50.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46b3c0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:50.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46b3c0640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa464142290 con 0x7fa4640b40b0 2024-03-27T21:39:50.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46b3c0640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:50.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46b3c0640 10 monclient(hunting): authenticate will time out at 831.970326s 2024-03-27T21:39:50.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46915c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa4640b40b0 0x7fa4640a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:50.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46915c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa4640b40b0 0x7fa4640a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47878/0 (socket says 172.21.15.60:47878) 2024-03-27T21:39:50.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46915c640 1 -- 172.21.15.60:0/1817788082 learned_addr learned my addr 172.21.15.60:0/1817788082 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:50.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46915c640 10 monclient(hunting): get_auth_request con 0x7fa4640b40b0 auth_method 0 2024-03-27T21:39:50.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46915c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:50.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46915c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:50.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46915c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:50.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46915c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:50.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46915c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:50.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46895b640 1 --2- 172.21.15.60:0/1817788082 >> v2:172.21.15.191:3300/0 conn(0x7fa46413b020 0x7fa46413b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:50.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46915c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:50.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46895b640 10 monclient(hunting): get_auth_request con 0x7fa46413b020 auth_method 0 2024-03-27T21:39:50.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46895b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:50.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46895b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:50.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46895b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:50.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46915c640 10 monclient(hunting): handle_auth_done global_id 4352 payload 274 2024-03-27T21:39:50.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46915c640 1 -- 172.21.15.60:0/1817788082 >> v2:172.21.15.191:3300/0 conn(0x7fa46413b020 msgr2=0x7fa46413b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:50.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46915c640 1 --2- 172.21.15.60:0/1817788082 >> v2:172.21.15.191:3300/0 conn(0x7fa46413b020 0x7fa46413b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46915c640 1 -- 172.21.15.60:0/1817788082 >> v1:172.21.15.191:6789/0 conn(0x7fa4641416f0 legacy=0x7fa464141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46915c640 10 monclient: _finish_hunting 0 2024-03-27T21:39:50.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46915c640 1 monclient: found mon.noname-b 2024-03-27T21:39:50.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46915c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:50.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46915c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:50.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46915c640 1 -- 172.21.15.60:0/1817788082 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4640aa520 con 0x7fa4640b40b0 2024-03-27T21:39:50.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.749+0000 7fa46895b640 1 --2- 172.21.15.60:0/1817788082 >> v2:172.21.15.191:3300/0 conn(0x7fa46413b020 0x7fa46413b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:39:50.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.750+0000 7fa46915c640 1 --2- 172.21.15.60:0/1817788082 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa4640b40b0 0x7fa4640a8b90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa45c009870 tx=0x7fa45c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f36f959a77d31a00 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:50.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.750+0000 7fa463fff640 1 -- 172.21.15.60:0/1817788082 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45c030840 con 0x7fa4640b40b0 2024-03-27T21:39:50.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.750+0000 7fa463fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:50.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.750+0000 7fa463fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:50.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.750+0000 7fa463fff640 10 monclient: dump: 2024-03-27T21:39:50.747 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:50.747 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:50.747 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:50.747 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:50.747 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:50.747 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:50.747 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:50.748 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:50.748 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:50.748 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:50.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.750+0000 7fa463fff640 10 monclient: _finish_auth 0 2024-03-27T21:39:50.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.750+0000 7fa463fff640 10 monclient: _check_auth_tickets 2024-03-27T21:39:50.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.750+0000 7fa463fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:50.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.750+0000 7fa46b3c0640 5 monclient: authenticate success, global_id 4352 2024-03-27T21:39:50.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.750+0000 7fa463fff640 1 -- 172.21.15.60:0/1817788082 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa45c0309a0 con 0x7fa4640b40b0 2024-03-27T21:39:50.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.750+0000 7fa463fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:50.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.750+0000 7fa46b3c0640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:50.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.750+0000 7fa46b3c0640 4 set_mon_vals no callback set 2024-03-27T21:39:50.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.750+0000 7fa463fff640 1 -- 172.21.15.60:0/1817788082 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45c030d30 con 0x7fa4640b40b0 2024-03-27T21:39:50.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.750+0000 7fa463fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:50.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.750+0000 7fa463fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:50.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.750+0000 7fa463fff640 10 monclient: dump: 2024-03-27T21:39:50.749 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:50.749 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:50.749 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:50.749 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:50.749 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:50.749 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:50.749 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:50.749 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:50.749 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:50.749 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:50.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.751+0000 7fa46b3c0640 10 monclient: shutdown 2024-03-27T21:39:50.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.751+0000 7fa46b3c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:50.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.751+0000 7fa46b3c0640 1 -- 172.21.15.60:0/1817788082 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa4640b40b0 msgr2=0x7fa4640a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.751+0000 7fa46b3c0640 1 --2- 172.21.15.60:0/1817788082 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa4640b40b0 0x7fa4640a8b90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa45c009870 tx=0x7fa45c02b980 comp rx=0 tx=0).stop 2024-03-27T21:39:50.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.751+0000 7fa46b3c0640 1 -- 172.21.15.60:0/1817788082 shutdown_connections 2024-03-27T21:39:50.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.751+0000 7fa46b3c0640 1 --2- 172.21.15.60:0/1817788082 >> v2:172.21.15.191:3300/0 conn(0x7fa46413b020 0x7fa46413b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.751+0000 7fa46b3c0640 1 --2- 172.21.15.60:0/1817788082 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa4640b40b0 0x7fa4640a8b90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.751+0000 7fa46b3c0640 1 -- 172.21.15.60:0/1817788082 >> 172.21.15.60:0/1817788082 conn(0x7fa4640608e0 msgr2=0x7fa464060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:50.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.751+0000 7fa46b3c0640 1 -- 172.21.15.60:0/1817788082 shutdown_connections 2024-03-27T21:39:50.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.751+0000 7fa46b3c0640 1 -- 172.21.15.60:0/1817788082 wait complete. 2024-03-27T21:39:50.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.751+0000 7fa46b3c0640 10 monclient: build_initial_monmap 2024-03-27T21:39:50.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.751+0000 7fa46b3c0640 10 monclient: monmap: 2024-03-27T21:39:50.751 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:50.751 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:50.751 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:50.751 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:50.752 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:50.752 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:50.752 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:50.752 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:50.752 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:50.752 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:50.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 1 librados: starting msgr at 2024-03-27T21:39:50.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 1 librados: starting objecter 2024-03-27T21:39:50.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 1 Processor -- start 2024-03-27T21:39:50.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 1 -- start start 2024-03-27T21:39:50.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 1 librados: setting wanted keys 2024-03-27T21:39:50.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 1 librados: calling monclient init 2024-03-27T21:39:50.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 10 monclient: init 2024-03-27T21:39:50.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:50.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:39:50.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4640b40b0 0x7fa464136ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:50.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 10 monclient(hunting): picked mon.noname-a con 0x7fa4640b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:50.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa46413b020 0x7fa46413ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:50.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 10 monclient(hunting): picked mon.noname-b con 0x7fa46413b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:50.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa4641416f0 0x7fa4641310a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:50.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 10 monclient(hunting): picked mon.noname-c con 0x7fa4641416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:50.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:50.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa46413c1c0 con 0x7fa4640b40b0 2024-03-27T21:39:50.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:50.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa46413a440 con 0x7fa46413b020 2024-03-27T21:39:50.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:50.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa46413abb0 con 0x7fa4641416f0 2024-03-27T21:39:50.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:50.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.752+0000 7fa46b3c0640 10 monclient(hunting): authenticate will time out at 831.973878s 2024-03-27T21:39:50.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46915c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4640b40b0 0x7fa464136ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:50.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46915c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4640b40b0 0x7fa464136ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48066/0 (socket says 172.21.15.60:48066) 2024-03-27T21:39:50.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46915c640 1 -- 172.21.15.60:0/4017277994 learned_addr learned my addr 172.21.15.60:0/4017277994 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:50.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46915c640 10 monclient(hunting): get_auth_request con 0x7fa4640b40b0 auth_method 0 2024-03-27T21:39:50.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46915c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:50.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46915c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:50.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46915c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:50.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46995d640 1 --2- 172.21.15.60:0/4017277994 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa4641416f0 0x7fa4641310a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:50.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46995d640 10 monclient(hunting): get_auth_request con 0x7fa4641416f0 auth_method 0 2024-03-27T21:39:50.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46995d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:50.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46995d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:50.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46995d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:50.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46915c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:50.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46915c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:50.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46915c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:50.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46995d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:50.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46995d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:50.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46995d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:50.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46895b640 1 --2- 172.21.15.60:0/4017277994 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa46413b020 0x7fa46413ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:50.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46915c640 10 monclient(hunting): handle_auth_done global_id 4413 payload 824 2024-03-27T21:39:50.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46915c640 1 -- 172.21.15.60:0/4017277994 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa4641416f0 msgr2=0x7fa4641310a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46915c640 1 --2- 172.21.15.60:0/4017277994 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa4641416f0 0x7fa4641310a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46915c640 1 -- 172.21.15.60:0/4017277994 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa46413b020 msgr2=0x7fa46413ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46915c640 1 --2- 172.21.15.60:0/4017277994 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa46413b020 0x7fa46413ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46915c640 10 monclient: _finish_hunting 0 2024-03-27T21:39:50.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46915c640 1 monclient: found mon.noname-a 2024-03-27T21:39:50.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46915c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46915c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46915c640 1 -- 172.21.15.60:0/4017277994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa45c0094d0 con 0x7fa4640b40b0 2024-03-27T21:39:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46895b640 10 monclient: get_auth_request con 0x7fa46413b020 auth_method 0 2024-03-27T21:39:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46895b640 1 --2- 172.21.15.60:0/4017277994 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa46413b020 0x7fa46413ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:39:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46915c640 1 --2- 172.21.15.60:0/4017277994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4640b40b0 0x7fa464136ab0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa45c000c00 tx=0x7fa45c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa461ffb640 10 client.?.objecter ms_handle_connect 0x7fa4640b40b0 2024-03-27T21:39:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa461ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa461ffb640 1 -- 172.21.15.60:0/4017277994 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45c039c90 con 0x7fa4640b40b0 2024-03-27T21:39:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa461ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa461ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa461ffb640 10 monclient: dump: 2024-03-27T21:39:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:50.758 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa461ffb640 10 monclient: _finish_auth 0 2024-03-27T21:39:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa461ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:39:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa461ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:50.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa461ffb640 1 -- 172.21.15.60:0/4017277994 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa45c039df0 con 0x7fa4640b40b0 2024-03-27T21:39:50.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46b3c0640 5 monclient: authenticate success, global_id 4413 2024-03-27T21:39:50.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46b3c0640 10 monclient: _renew_subs 2024-03-27T21:39:50.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46b3c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:50.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46b3c0640 1 -- 172.21.15.60:0/4017277994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa46413ad70 con 0x7fa4640b40b0 2024-03-27T21:39:50.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa461ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:50.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa461ffb640 1 -- 172.21.15.60:0/4017277994 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45c043640 con 0x7fa4640b40b0 2024-03-27T21:39:50.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa461ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:50.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa461ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:50.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa461ffb640 10 monclient: dump: 2024-03-27T21:39:50.760 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:50.760 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:50.760 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:50.760 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:50.760 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:50.760 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:50.760 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:50.760 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:50.760 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:50.760 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:50.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46b3c0640 10 client.4413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:50.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46b3c0640 10 monclient: _renew_subs 2024-03-27T21:39:50.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46b3c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:50.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa4637fe640 4 set_mon_vals no callback set 2024-03-27T21:39:50.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46b3c0640 1 -- 172.21.15.60:0/4017277994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4641317c0 con 0x7fa4640b40b0 2024-03-27T21:39:50.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.753+0000 7fa46b3c0640 1 librados: init done 2024-03-27T21:39:50.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.754+0000 7fa46b3c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:50.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.754+0000 7fa46b3c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:50.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.754+0000 7fa46b3c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:50.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.754+0000 7fa46b3c0640 1 -- 172.21.15.60:0/4017277994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa464069470 con 0x7fa4640b40b0 2024-03-27T21:39:50.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.755+0000 7fa461ffb640 1 -- 172.21.15.60:0/4017277994 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa45c0437a0 con 0x7fa4640b40b0 2024-03-27T21:39:50.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.755+0000 7fa461ffb640 1 --2- 172.21.15.60:0/4017277994 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa45006c590 0x7fa45006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:50.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.755+0000 7fa461ffb640 1 -- 172.21.15.60:0/4017277994 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa45c03e070 con 0x7fa4640b40b0 2024-03-27T21:39:50.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.755+0000 7fa461ffb640 10 client.4413.objecter ms_dispatch 0x7fa464000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:50.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.755+0000 7fa461ffb640 3 client.4413.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:50.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.755+0000 7fa461ffb640 3 client.4413.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:50.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.755+0000 7fa46895b640 1 --2- 172.21.15.60:0/4017277994 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa45006c590 0x7fa45006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:50.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.755+0000 7fa46895b640 10 monclient: get_auth_request con 0x7fa45006c590 auth_method 0 2024-03-27T21:39:50.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.755+0000 7fa461ffb640 20 client.4413.objecter dump_active .. 0 homeless 2024-03-27T21:39:50.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.756+0000 7fa461ffb640 10 client.4413.objecter ms_handle_connect 0x7fa45006c590 2024-03-27T21:39:50.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.756+0000 7fa46895b640 1 --2- 172.21.15.60:0/4017277994 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa45006c590 0x7fa45006ea50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa458006fd0 tx=0x7fa458008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:50.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.759+0000 7fa461ffb640 1 -- 172.21.15.60:0/4017277994 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa45c077540 con 0x7fa4640b40b0 2024-03-27T21:39:50.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.759+0000 7fa461ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:50.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.759+0000 7fa461ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:50.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.937+0000 7fa46b3c0640 1 -- 172.21.15.60:0/4017277994 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa4640a90d0 con 0x7fa45006c590 2024-03-27T21:39:50.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.939+0000 7fa461ffb640 1 -- 172.21.15.60:0/4017277994 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48318 (secure 0 0 0) 0x7fa458009650 con 0x7fa45006c590 2024-03-27T21:39:50.926 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:39:50.926 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-03-27T21:39:50.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.941+0000 7fa46b3c0640 10 librados: watch_flush enter 2024-03-27T21:39:50.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.941+0000 7fa46b3c0640 10 librados: watch_flush exit 2024-03-27T21:39:50.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.941+0000 7fa46b3c0640 20 client.4413.objecter shutdown clearing up homeless session... 2024-03-27T21:39:50.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.941+0000 7fa46b3c0640 10 client.4413.objecter successfully canceled tick 2024-03-27T21:39:50.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.941+0000 7fa46b3c0640 1 -- 172.21.15.60:0/4017277994 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa45006c590 msgr2=0x7fa45006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.941+0000 7fa46b3c0640 1 --2- 172.21.15.60:0/4017277994 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa45006c590 0x7fa45006ea50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa458006fd0 tx=0x7fa458008040 comp rx=0 tx=0).stop 2024-03-27T21:39:50.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.941+0000 7fa46b3c0640 10 monclient: shutdown 2024-03-27T21:39:50.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.941+0000 7fa46b3c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:50.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.941+0000 7fa46b3c0640 1 -- 172.21.15.60:0/4017277994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4640b40b0 msgr2=0x7fa464136ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:50.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.941+0000 7fa46b3c0640 1 --2- 172.21.15.60:0/4017277994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4640b40b0 0x7fa464136ab0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa45c000c00 tx=0x7fa45c004730 comp rx=0 tx=0).stop 2024-03-27T21:39:50.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.941+0000 7fa46b3c0640 1 -- 172.21.15.60:0/4017277994 shutdown_connections 2024-03-27T21:39:50.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.941+0000 7fa46b3c0640 1 --2- 172.21.15.60:0/4017277994 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa4641416f0 0x7fa4641310a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.941+0000 7fa46b3c0640 1 --2- 172.21.15.60:0/4017277994 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa45006c590 0x7fa45006ea50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.941+0000 7fa46b3c0640 1 --2- 172.21.15.60:0/4017277994 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa46413b020 0x7fa46413ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.941+0000 7fa46b3c0640 1 --2- 172.21.15.60:0/4017277994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4640b40b0 0x7fa464136ab0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:50.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.941+0000 7fa46b3c0640 1 -- 172.21.15.60:0/4017277994 >> 172.21.15.60:0/4017277994 conn(0x7fa4640608e0 msgr2=0x7fa464140c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:50.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.941+0000 7fa46b3c0640 1 -- 172.21.15.60:0/4017277994 shutdown_connections 2024-03-27T21:39:50.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.941+0000 7fa46b3c0640 1 -- 172.21.15.60:0/4017277994 wait complete. 2024-03-27T21:39:50.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:50.941+0000 7fa46b3c0640 1 librados: shutdown 2024-03-27T21:39:50.938 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-03-27T21:39:49.390687+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":221896,"kb_used_data":1620,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":754752824,"statfs":{"total":773094113280,"available":772866891776,"internally_reserved":0,"allocated":1658880,"data_stored":1573540,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":12742,"internal_metadata":225431098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001235"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:46.474962+0000","last_change":"2024-03-27T21:39:46.470433+0000","last_active":"2024-03-27T21:39:46.474962+0000","last_peered":"2024-03-27T21:39:46.474962+0000","last_clean":"2024-03-27T21:39:46.474962+0000","last_became_active":"2024-03-27T21:39:46.470153+0000","last_became_peered":"2024-03-27T21:39:46.470153+0000","last_unstale":"2024-03-27T21:39:46.474962+0000","last_undegraded":"2024-03-27T21:39:46.474962+0000","last_fullsized":"2024-03-27T21:39:46.474962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T21:39:45.811197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:46.468505+0000","last_change":"2024-03-27T21:39:46.467893+0000","last_active":"2024-03-27T21:39:46.468505+0000","last_peered":"2024-03-27T21:39:46.468505+0000","last_clean":"2024-03-27T21:39:46.468505+0000","last_became_active":"2024-03-27T21:39:46.467576+0000","last_became_peered":"2024-03-27T21:39:46.467576+0000","last_unstale":"2024-03-27T21:39:46.468505+0000","last_undegraded":"2024-03-27T21:39:46.468505+0000","last_fullsized":"2024-03-27T21:39:46.468505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:29:50.190006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:47.405006+0000","last_change":"2024-03-27T21:39:46.679788+0000","last_active":"2024-03-27T21:39:47.405006+0000","last_peered":"2024-03-27T21:39:47.405006+0000","last_clean":"2024-03-27T21:39:47.405006+0000","last_became_active":"2024-03-27T21:39:46.679561+0000","last_became_peered":"2024-03-27T21:39:46.679561+0000","last_unstale":"2024-03-27T21:39:47.405006+0000","last_undegraded":"2024-03-27T21:39:47.405006+0000","last_fullsized":"2024-03-27T21:39:47.405006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T23:53:50.373923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:46.679751+0000","last_change":"2024-03-27T21:39:46.679751+0000","last_active":"2024-03-27T21:39:46.679751+0000","last_peered":"2024-03-27T21:39:46.679751+0000","last_clean":"2024-03-27T21:39:46.679751+0000","last_became_active":"2024-03-27T21:39:46.679509+0000","last_became_peered":"2024-03-27T21:39:46.679509+0000","last_unstale":"2024-03-27T21:39:46.679751+0000","last_undegraded":"2024-03-27T21:39:46.679751+0000","last_fullsized":"2024-03-27T21:39:46.679751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:43:02.145158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T21:39:47.458020+0000","last_change":"2024-03-27T21:39:46.459503+0000","last_active":"2024-03-27T21:39:47.458020+0000","last_peered":"2024-03-27T21:39:47.458020+0000","last_clean":"2024-03-27T21:39:47.458020+0000","last_became_active":"2024-03-27T21:39:46.459225+0000","last_became_peered":"2024-03-27T21:39:46.459225+0000","last_unstale":"2024-03-27T21:39:47.458020+0000","last_undegraded":"2024-03-27T21:39:47.458020+0000","last_fullsized":"2024-03-27T21:39:47.458020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:17:43.512127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:46.680826+0000","last_change":"2024-03-27T21:39:46.679791+0000","last_active":"2024-03-27T21:39:46.680826+0000","last_peered":"2024-03-27T21:39:46.680826+0000","last_clean":"2024-03-27T21:39:46.680826+0000","last_became_active":"2024-03-27T21:39:46.679503+0000","last_became_peered":"2024-03-27T21:39:46.679503+0000","last_unstale":"2024-03-27T21:39:46.680826+0000","last_undegraded":"2024-03-27T21:39:46.680826+0000","last_fullsized":"2024-03-27T21:39:46.680826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:07:25.151346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:47.404983+0000","last_change":"2024-03-27T21:39:46.458326+0000","last_active":"2024-03-27T21:39:47.404983+0000","last_peered":"2024-03-27T21:39:47.404983+0000","last_clean":"2024-03-27T21:39:47.404983+0000","last_became_active":"2024-03-27T21:39:46.458074+0000","last_became_peered":"2024-03-27T21:39:46.458074+0000","last_unstale":"2024-03-27T21:39:47.404983+0000","last_undegraded":"2024-03-27T21:39:47.404983+0000","last_fullsized":"2024-03-27T21:39:47.404983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:06:29.798229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T21:39:47.458062+0000","last_change":"2024-03-27T21:39:46.459166+0000","last_active":"2024-03-27T21:39:47.458062+0000","last_peered":"2024-03-27T21:39:47.458062+0000","last_clean":"2024-03-27T21:39:47.458062+0000","last_became_active":"2024-03-27T21:39:46.458958+0000","last_became_peered":"2024-03-27T21:39:46.458958+0000","last_unstale":"2024-03-27T21:39:47.458062+0000","last_undegraded":"2024-03-27T21:39:47.458062+0000","last_fullsized":"2024-03-27T21:39:47.458062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T22:18:37.511546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":163,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:46.455960+0000","last_change":"2024-03-27T21:39:39.866806+0000","last_active":"2024-03-27T21:39:46.455960+0000","last_peered":"2024-03-27T21:39:46.455960+0000","last_clean":"2024-03-27T21:39:46.455960+0000","last_became_active":"2024-03-27T21:39:39.865948+0000","last_became_peered":"2024-03-27T21:39:39.865948+0000","last_unstale":"2024-03-27T21:39:46.455960+0000","last_undegraded":"2024-03-27T21:39:46.455960+0000","last_fullsized":"2024-03-27T21:39:46.455960+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_clean_scrub_stamp":"2024-03-27T21:39:38.411432+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:36:49.375869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":16,"seq":68719476742,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]}]},{"osd":3,"up_from":16,"seq":68719476741,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31848905},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]}]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]}]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":598418,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]}]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]}]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":592670,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T21:39:50.939 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-27T21:39:50.940 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-27T21:39:50.940 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-27T21:39:50.940 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-27T21:39:51.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.052+0000 7f1e4c082640 10 monclient: get_monmap_and_config 2024-03-27T21:39:51.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.052+0000 7f1e4c082640 10 monclient: build_initial_monmap 2024-03-27T21:39:51.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.052+0000 7f1e4c082640 10 monclient: monmap: 2024-03-27T21:39:51.039 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:51.039 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:51.039 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:51.053067+0000 2024-03-27T21:39:51.039 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:51.053067+0000 2024-03-27T21:39:51.039 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:51.039 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:51.039 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:51.039 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:51.039 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:51.039 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:51.039 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:51.039 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:51.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.052+0000 7f1e4c082640 1 Processor -- start 2024-03-27T21:39:51.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.052+0000 7f1e4c082640 1 -- start start 2024-03-27T21:39:51.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.052+0000 7f1e4c082640 10 monclient: init 2024-03-27T21:39:51.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4c082640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:51.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4c082640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-03-27T21:39:51.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4c082640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1e440b40b0 0x7f1e440a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:51.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4c082640 10 monclient(hunting): picked mon.noname-a con 0x7f1e440b40b0 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:51.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4c082640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1e4413b020 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:51.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4c082640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e4413b480 0x7f1e44141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:51.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4c082640 10 monclient(hunting): picked mon.noname-b con 0x7f1e4413b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:51.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4c082640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e440b3830 con 0x7f1e4413b020 2024-03-27T21:39:51.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4c082640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:51.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4c082640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e4413b910 con 0x7f1e440b40b0 2024-03-27T21:39:51.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4c082640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:51.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4c082640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e440b4490 con 0x7f1e4413b480 2024-03-27T21:39:51.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4c082640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:51.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4c082640 10 monclient(hunting): authenticate will time out at 832.274418s 2024-03-27T21:39:51.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4a61f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e4413b480 0x7f1e44141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:51.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e49e1e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1e440b40b0 0x7f1e440a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:51.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e49e1e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1e440b40b0 0x7f1e440a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48082/0 (socket says 172.21.15.60:48082) 2024-03-27T21:39:51.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4a61f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e4413b480 0x7f1e44141e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47902/0 (socket says 172.21.15.60:47902) 2024-03-27T21:39:51.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e49e1e640 1 -- 172.21.15.60:0/2820253904 learned_addr learned my addr 172.21.15.60:0/2820253904 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:51.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e49e1e640 10 monclient(hunting): get_auth_request con 0x7f1e440b40b0 auth_method 0 2024-03-27T21:39:51.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e49e1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:51.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e49e1e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:51.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e49e1e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:51.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4a61f640 10 monclient(hunting): get_auth_request con 0x7f1e4413b480 auth_method 0 2024-03-27T21:39:51.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4a61f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:51.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4a61f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:51.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4a61f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:51.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e49e1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:51.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e49e1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:51.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e49e1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:51.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4a61f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:51.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4a61f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:51.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4a61f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:51.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4a61f640 10 monclient(hunting): handle_auth_done global_id 4358 payload 274 2024-03-27T21:39:51.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4a61f640 1 -- 172.21.15.60:0/2820253904 >> v2:172.21.15.60:3300/0 conn(0x7f1e440b40b0 msgr2=0x7f1e440a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:51.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4a61f640 1 --2- 172.21.15.60:0/2820253904 >> v2:172.21.15.60:3300/0 conn(0x7f1e440b40b0 0x7f1e440a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4a61f640 1 -- 172.21.15.60:0/2820253904 >> v1:172.21.15.191:6789/0 conn(0x7f1e4413b020 legacy=0x7f1e441416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:51.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4a61f640 10 monclient: _finish_hunting 0 2024-03-27T21:39:51.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4a61f640 1 monclient: found mon.noname-b 2024-03-27T21:39:51.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4a61f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:51.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4a61f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:51.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e4a61f640 1 -- 172.21.15.60:0/2820253904 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e440aa520 con 0x7f1e4413b480 2024-03-27T21:39:51.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.053+0000 7f1e49e1e640 1 --2- 172.21.15.60:0/2820253904 >> v2:172.21.15.60:3300/0 conn(0x7f1e440b40b0 0x7f1e440a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:51.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.054+0000 7f1e4a61f640 1 --2- 172.21.15.60:0/2820253904 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e4413b480 0x7f1e44141e00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c0097a0 tx=0x7f1e3c02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bdaced66518e1e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:51.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.054+0000 7f1e48e1c640 1 -- 172.21.15.60:0/2820253904 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c0316f0 con 0x7f1e4413b480 2024-03-27T21:39:51.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.054+0000 7f1e48e1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:51.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.054+0000 7f1e48e1c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:51.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.054+0000 7f1e48e1c640 10 monclient: dump: 2024-03-27T21:39:51.051 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:51.051 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:51.051 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.051 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.051 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:51.051 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:51.051 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:51.051 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:51.052 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:51.052 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:51.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.054+0000 7f1e48e1c640 10 monclient: _finish_auth 0 2024-03-27T21:39:51.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.054+0000 7f1e48e1c640 10 monclient: _check_auth_tickets 2024-03-27T21:39:51.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.054+0000 7f1e48e1c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:51.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.054+0000 7f1e48e1c640 1 -- 172.21.15.60:0/2820253904 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3c031850 con 0x7f1e4413b480 2024-03-27T21:39:51.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.054+0000 7f1e4c082640 5 monclient: authenticate success, global_id 4358 2024-03-27T21:39:51.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.054+0000 7f1e4c082640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:51.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.054+0000 7f1e48e1c640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:51.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.054+0000 7f1e4c082640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:51.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.054+0000 7f1e48e1c640 1 -- 172.21.15.60:0/2820253904 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c031be0 con 0x7f1e4413b480 2024-03-27T21:39:51.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.054+0000 7f1e48e1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:51.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.054+0000 7f1e4c082640 4 set_mon_vals no callback set 2024-03-27T21:39:51.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.054+0000 7f1e48e1c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:51.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.054+0000 7f1e48e1c640 10 monclient: dump: 2024-03-27T21:39:51.053 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:51.053 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:51.053 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.053 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.053 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:51.053 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:51.053 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:51.053 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:51.053 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:51.053 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:51.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.055+0000 7f1e4c082640 10 monclient: shutdown 2024-03-27T21:39:51.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.055+0000 7f1e4c082640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:51.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.055+0000 7f1e4c082640 1 -- 172.21.15.60:0/2820253904 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e4413b480 msgr2=0x7f1e44141e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:51.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.055+0000 7f1e4c082640 1 --2- 172.21.15.60:0/2820253904 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e4413b480 0x7f1e44141e00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c0097a0 tx=0x7f1e3c02b790 comp rx=0 tx=0).stop 2024-03-27T21:39:51.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.055+0000 7f1e4c082640 1 -- 172.21.15.60:0/2820253904 shutdown_connections 2024-03-27T21:39:51.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.055+0000 7f1e4c082640 1 --2- 172.21.15.60:0/2820253904 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e4413b480 0x7f1e44141e00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.055+0000 7f1e4c082640 1 --2- 172.21.15.60:0/2820253904 >> v2:172.21.15.60:3300/0 conn(0x7f1e440b40b0 0x7f1e440a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.055+0000 7f1e4c082640 1 -- 172.21.15.60:0/2820253904 >> 172.21.15.60:0/2820253904 conn(0x7f1e440608e0 msgr2=0x7f1e44060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:51.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.055+0000 7f1e4c082640 1 -- 172.21.15.60:0/2820253904 shutdown_connections 2024-03-27T21:39:51.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.055+0000 7f1e4c082640 1 -- 172.21.15.60:0/2820253904 wait complete. 2024-03-27T21:39:51.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 10 monclient: build_initial_monmap 2024-03-27T21:39:51.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 10 monclient: monmap: 2024-03-27T21:39:51.054 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:51.054 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:51.054 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:51.054 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:51.054 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:51.054 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:51.055 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:51.055 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:51.055 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:51.055 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:51.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 1 librados: starting msgr at 2024-03-27T21:39:51.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 1 librados: starting objecter 2024-03-27T21:39:51.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 1 Processor -- start 2024-03-27T21:39:51.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 1 -- start start 2024-03-27T21:39:51.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 1 librados: setting wanted keys 2024-03-27T21:39:51.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 1 librados: calling monclient init 2024-03-27T21:39:51.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 10 monclient: init 2024-03-27T21:39:51.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:51.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:39:51.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e440b40b0 0x7f1e4413e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:51.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 10 monclient(hunting): picked mon.noname-b con 0x7f1e440b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:51.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e4413b020 0x7f1e441d68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:51.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 10 monclient(hunting): picked mon.noname-a con 0x7f1e4413b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:51.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e4413b480 0x7f1e441d6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:51.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 10 monclient(hunting): picked mon.noname-c con 0x7f1e4413b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:51.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:51.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e441d6460 con 0x7f1e4413b020 2024-03-27T21:39:51.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:51.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4413d3e0 con 0x7f1e440b40b0 2024-03-27T21:39:51.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:51.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4413ba60 con 0x7f1e4413b480 2024-03-27T21:39:51.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:51.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.056+0000 7f1e4c082640 10 monclient(hunting): authenticate will time out at 832.277911s 2024-03-27T21:39:51.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4a61f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e4413b480 0x7f1e441d6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:51.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4961d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e4413b020 0x7f1e441d68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:51.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4a61f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e4413b480 0x7f1e441d6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47908/0 (socket says 172.21.15.60:47908) 2024-03-27T21:39:51.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4a61f640 1 -- 172.21.15.60:0/804287367 learned_addr learned my addr 172.21.15.60:0/804287367 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:51.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4961d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e4413b020 0x7f1e441d68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48094/0 (socket says 172.21.15.60:48094) 2024-03-27T21:39:51.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4a61f640 10 monclient(hunting): get_auth_request con 0x7f1e4413b480 auth_method 0 2024-03-27T21:39:51.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4a61f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:51.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4a61f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:51.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4a61f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:51.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4961d640 10 monclient(hunting): get_auth_request con 0x7f1e4413b020 auth_method 0 2024-03-27T21:39:51.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4961d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:51.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4961d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:51.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4961d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:51.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4961d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:51.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4961d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:51.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4961d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:51.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4a61f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:51.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4a61f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:51.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4a61f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:51.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e49e1e640 1 --2- 172.21.15.60:0/804287367 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e440b40b0 0x7f1e4413e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:51.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4961d640 10 monclient(hunting): handle_auth_done global_id 4419 payload 824 2024-03-27T21:39:51.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4961d640 1 -- 172.21.15.60:0/804287367 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e4413b480 msgr2=0x7f1e441d6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:51.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4961d640 1 --2- 172.21.15.60:0/804287367 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e4413b480 0x7f1e441d6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4961d640 1 -- 172.21.15.60:0/804287367 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e440b40b0 msgr2=0x7f1e4413e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:51.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4961d640 1 --2- 172.21.15.60:0/804287367 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e440b40b0 0x7f1e4413e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4961d640 10 monclient: _finish_hunting 0 2024-03-27T21:39:51.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4961d640 1 monclient: found mon.noname-a 2024-03-27T21:39:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4961d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4961d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4961d640 1 -- 172.21.15.60:0/804287367 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e40009b30 con 0x7f1e4413b020 2024-03-27T21:39:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e49e1e640 10 monclient: get_auth_request con 0x7f1e440b40b0 auth_method 0 2024-03-27T21:39:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e49e1e640 1 --2- 172.21.15.60:0/804287367 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e440b40b0 0x7f1e4413e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:39:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4961d640 1 --2- 172.21.15.60:0/804287367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e4413b020 0x7f1e441d68f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1e3400c410 tx=0x7f1e3400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e3a7fc640 10 client.?.objecter ms_handle_connect 0x7f1e4413b020 2024-03-27T21:39:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e3a7fc640 1 -- 172.21.15.60:0/804287367 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e34005230 con 0x7f1e4413b020 2024-03-27T21:39:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e3a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e3a7fc640 10 monclient: dump: 2024-03-27T21:39:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:51.060 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:51.060 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.060 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.060 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:51.060 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:51.060 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:51.060 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:51.060 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:51.060 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:51.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e3a7fc640 10 monclient: _finish_auth 0 2024-03-27T21:39:51.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e3a7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:39:51.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:51.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e3a7fc640 1 -- 172.21.15.60:0/804287367 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e34005390 con 0x7f1e4413b020 2024-03-27T21:39:51.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4c082640 5 monclient: authenticate success, global_id 4419 2024-03-27T21:39:51.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4c082640 10 monclient: _renew_subs 2024-03-27T21:39:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4c082640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4c082640 1 -- 172.21.15.60:0/804287367 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e3c009400 con 0x7f1e4413b020 2024-03-27T21:39:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e3a7fc640 1 -- 172.21.15.60:0/804287367 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e34004820 con 0x7f1e4413b020 2024-03-27T21:39:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e3a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e3a7fc640 10 monclient: dump: 2024-03-27T21:39:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:51.062 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:51.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e3bfff640 4 set_mon_vals no callback set 2024-03-27T21:39:51.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4c082640 10 client.4419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:51.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4c082640 10 monclient: _renew_subs 2024-03-27T21:39:51.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4c082640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:51.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4c082640 1 -- 172.21.15.60:0/804287367 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e440ab100 con 0x7f1e4413b020 2024-03-27T21:39:51.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.057+0000 7f1e4c082640 1 librados: init done 2024-03-27T21:39:51.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.058+0000 7f1e4c082640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:51.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.058+0000 7f1e4c082640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:51.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.058+0000 7f1e4c082640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:51.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.058+0000 7f1e4c082640 1 -- 172.21.15.60:0/804287367 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e44069470 con 0x7f1e4413b020 2024-03-27T21:39:51.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.059+0000 7f1e3a7fc640 1 -- 172.21.15.60:0/804287367 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1e34004980 con 0x7f1e4413b020 2024-03-27T21:39:51.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.059+0000 7f1e3a7fc640 1 --2- 172.21.15.60:0/804287367 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1e2806c590 0x7f1e2806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:51.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.059+0000 7f1e3a7fc640 1 -- 172.21.15.60:0/804287367 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1e34088510 con 0x7f1e4413b020 2024-03-27T21:39:51.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.059+0000 7f1e3a7fc640 10 client.4419.objecter ms_dispatch 0x7f1e44000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:51.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.059+0000 7f1e3a7fc640 3 client.4419.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:51.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.059+0000 7f1e3a7fc640 3 client.4419.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:51.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.059+0000 7f1e49e1e640 1 --2- 172.21.15.60:0/804287367 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1e2806c590 0x7f1e2806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:51.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.059+0000 7f1e49e1e640 10 monclient: get_auth_request con 0x7f1e2806c590 auth_method 0 2024-03-27T21:39:51.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.059+0000 7f1e3a7fc640 20 client.4419.objecter dump_active .. 0 homeless 2024-03-27T21:39:51.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.060+0000 7f1e3a7fc640 10 client.4419.objecter ms_handle_connect 0x7f1e2806c590 2024-03-27T21:39:51.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.060+0000 7f1e49e1e640 1 --2- 172.21.15.60:0/804287367 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1e2806c590 0x7f1e2806ea50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1e40009fd0 tx=0x7f1e4000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:51.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.064+0000 7f1e3a7fc640 1 -- 172.21.15.60:0/804287367 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1e34053990 con 0x7f1e4413b020 2024-03-27T21:39:51.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.064+0000 7f1e3a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:51.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.064+0000 7f1e3a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:51.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.285+0000 7f1e4c082640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-03-27T21:39:51.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.285+0000 7f1e4c082640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-03-27T21:39:51.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.285+0000 7f1e4c082640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:51.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.285+0000 7f1e4c082640 1 -- 172.21.15.60:0/804287367 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1e440b2860 con 0x7f1e4413b020 2024-03-27T21:39:51.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.285+0000 7f1e3a7fc640 1 -- 172.21.15.60:0/804287367 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f1e34056fb0 con 0x7f1e4413b020 2024-03-27T21:39:51.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.285+0000 7f1e3a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-03-27T21:39:51.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.285+0000 7f1e3a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:51.273 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:39:51.273 INFO:teuthology.orchestra.run.smithi060.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-27T21:39:51.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.287+0000 7f1e4c082640 10 librados: watch_flush enter 2024-03-27T21:39:51.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.287+0000 7f1e4c082640 10 librados: watch_flush exit 2024-03-27T21:39:51.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.287+0000 7f1e4c082640 20 client.4419.objecter shutdown clearing up homeless session... 2024-03-27T21:39:51.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.287+0000 7f1e4c082640 10 client.4419.objecter successfully canceled tick 2024-03-27T21:39:51.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.288+0000 7f1e4c082640 1 -- 172.21.15.60:0/804287367 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1e2806c590 msgr2=0x7f1e2806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:51.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.288+0000 7f1e4c082640 1 --2- 172.21.15.60:0/804287367 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1e2806c590 0x7f1e2806ea50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1e40009fd0 tx=0x7f1e4000b040 comp rx=0 tx=0).stop 2024-03-27T21:39:51.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.288+0000 7f1e4c082640 10 monclient: shutdown 2024-03-27T21:39:51.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.288+0000 7f1e4c082640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:51.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.288+0000 7f1e4c082640 1 -- 172.21.15.60:0/804287367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e4413b020 msgr2=0x7f1e441d68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:51.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.288+0000 7f1e4c082640 1 --2- 172.21.15.60:0/804287367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e4413b020 0x7f1e441d68f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1e3400c410 tx=0x7f1e3400c8d0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.288+0000 7f1e4c082640 1 -- 172.21.15.60:0/804287367 shutdown_connections 2024-03-27T21:39:51.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.288+0000 7f1e4c082640 1 --2- 172.21.15.60:0/804287367 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1e4413b480 0x7f1e441d6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.288+0000 7f1e4c082640 1 --2- 172.21.15.60:0/804287367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e4413b020 0x7f1e441d68f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.288+0000 7f1e4c082640 1 --2- 172.21.15.60:0/804287367 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1e2806c590 0x7f1e2806ea50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.288+0000 7f1e4c082640 1 --2- 172.21.15.60:0/804287367 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e440b40b0 0x7f1e4413e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.288+0000 7f1e4c082640 1 -- 172.21.15.60:0/804287367 >> 172.21.15.60:0/804287367 conn(0x7f1e440608e0 msgr2=0x7f1e440a88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:51.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.288+0000 7f1e4c082640 1 -- 172.21.15.60:0/804287367 shutdown_connections 2024-03-27T21:39:51.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.288+0000 7f1e4c082640 1 -- 172.21.15.60:0/804287367 wait complete. 2024-03-27T21:39:51.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.288+0000 7f1e4c082640 1 librados: shutdown 2024-03-27T21:39:51.285 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-27T21:39:51.285 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-27T21:39:51.296 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-27T21:39:51.296 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-yuri4-testing-2024-03-26-1132-reef'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'sha1': '7c37d80f7a88f4eb561c4fe02558660a650b10ac'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7c37d80f7a88f4eb561c4fe02558660a650b10ac'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2024-03-26-1132-reef', 'sha1': '7c37d80f7a88f4eb561c4fe02558660a650b10ac'}} 2024-03-27T21:39:51.296 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-03-27T21:39:51.296 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-27T21:39:51.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.412+0000 7fec1fbca640 10 monclient: get_monmap_and_config 2024-03-27T21:39:51.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.412+0000 7fec1fbca640 10 monclient: build_initial_monmap 2024-03-27T21:39:51.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.412+0000 7fec1fbca640 10 monclient: monmap: 2024-03-27T21:39:51.399 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:51.399 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:51.399 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:51.413504+0000 2024-03-27T21:39:51.399 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:51.413504+0000 2024-03-27T21:39:51.399 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:51.399 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:51.399 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:51.399 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:51.400 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:51.400 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:51.400 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:51.400 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:51.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1fbca640 1 Processor -- start 2024-03-27T21:39:51.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1fbca640 1 -- start start 2024-03-27T21:39:51.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1fbca640 10 monclient: init 2024-03-27T21:39:51.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1fbca640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:51.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1fbca640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-03-27T21:39:51.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1fbca640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fec18071a50 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:51.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1fbca640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fec1813ef20 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:51.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1fbca640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fec180a87e0 0x7fec18141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:51.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1fbca640 10 monclient(hunting): picked mon.noname-a con 0x7fec180a87e0 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:51.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1fbca640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec180a8e90 con 0x7fec18071a50 2024-03-27T21:39:51.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1fbca640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec180aa040 con 0x7fec1813ef20 2024-03-27T21:39:51.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1fbca640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:51.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1fbca640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec18071e30 con 0x7fec180a87e0 2024-03-27T21:39:51.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1fbca640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:51.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1fbca640 10 monclient(hunting): authenticate will time out at 832.634754s 2024-03-27T21:39:51.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1e167640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fec180a87e0 0x7fec18141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:51.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1d966640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fec18071a50 0x7fec1813e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43640/0 (socket says 172.21.15.60:43640) 2024-03-27T21:39:51.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1d966640 1 -- 172.21.15.60:0/2052158031 learned_addr learned my addr 172.21.15.60:0/2052158031 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:39:51.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1e167640 10 monclient(hunting): get_auth_request con 0x7fec180a87e0 auth_method 0 2024-03-27T21:39:51.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1e167640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:51.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1e167640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:51.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.413+0000 7fec1e167640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:51.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.414+0000 7fec1e167640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:51.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.414+0000 7fec1e167640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:51.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.414+0000 7fec1e167640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:51.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.414+0000 7fec1e167640 10 monclient(hunting): handle_auth_done global_id 4422 payload 274 2024-03-27T21:39:51.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.414+0000 7fec1e167640 1 -- 172.21.15.60:0/2052158031 >> v1:172.21.15.191:6789/0 conn(0x7fec1813ef20 legacy=0x7fec18141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:51.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.414+0000 7fec1e167640 1 -- 172.21.15.60:0/2052158031 >> v1:172.21.15.60:6789/0 conn(0x7fec18071a50 legacy=0x7fec1813e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:51.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.414+0000 7fec1e167640 10 monclient: _finish_hunting 0 2024-03-27T21:39:51.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.414+0000 7fec1e167640 1 monclient: found mon.noname-a 2024-03-27T21:39:51.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.414+0000 7fec1c964640 1 -- 172.21.15.60:0/2052158031 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2535022966 0 0) 0x7fec180a8e90 con 0x7fec18071a50 2024-03-27T21:39:51.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.414+0000 7fec1e167640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:51.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.414+0000 7fec1e167640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:51.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.414+0000 7fec1e167640 1 -- 172.21.15.60:0/2052158031 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec18142280 con 0x7fec180a87e0 2024-03-27T21:39:51.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.414+0000 7fec1c964640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:39:51.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.414+0000 7fec1e167640 1 --2- 172.21.15.60:0/2052158031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec180a87e0 0x7fec18141d40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fec10009d90 tx=0x7fec1002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5502c47efd97294 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:51.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1c964640 1 -- 172.21.15.60:0/2052158031 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1002cd00 con 0x7fec180a87e0 2024-03-27T21:39:51.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1c964640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:51.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1c964640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:51.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1c964640 10 monclient: dump: 2024-03-27T21:39:51.411 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:51.411 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:51.411 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.411 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.411 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:51.411 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:51.411 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:51.411 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:51.411 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:51.412 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:51.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1c964640 10 monclient: _finish_auth 0 2024-03-27T21:39:51.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1c964640 10 monclient: _check_auth_tickets 2024-03-27T21:39:51.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1c964640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:51.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1c964640 1 -- 172.21.15.60:0/2052158031 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1002ce60 con 0x7fec180a87e0 2024-03-27T21:39:51.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1fbca640 5 monclient: authenticate success, global_id 4422 2024-03-27T21:39:51.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1fbca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:51.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1c964640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:51.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1fbca640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:51.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1c964640 1 -- 172.21.15.60:0/2052158031 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec10034660 con 0x7fec180a87e0 2024-03-27T21:39:51.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1c964640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:51.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1fbca640 4 set_mon_vals no callback set 2024-03-27T21:39:51.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1c964640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:51.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1c964640 10 monclient: dump: 2024-03-27T21:39:51.413 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:51.413 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:51.413 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.413 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.413 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:51.413 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:51.413 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:51.413 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:51.413 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:51.413 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:51.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1fbca640 10 monclient: shutdown 2024-03-27T21:39:51.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1fbca640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:51.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1fbca640 1 -- 172.21.15.60:0/2052158031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec180a87e0 msgr2=0x7fec18141d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:51.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1fbca640 1 --2- 172.21.15.60:0/2052158031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec180a87e0 0x7fec18141d40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fec10009d90 tx=0x7fec1002bf50 comp rx=0 tx=0).stop 2024-03-27T21:39:51.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1fbca640 1 -- 172.21.15.60:0/2052158031 shutdown_connections 2024-03-27T21:39:51.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1fbca640 1 --2- 172.21.15.60:0/2052158031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec180a87e0 0x7fec18141d40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1fbca640 1 -- 172.21.15.60:0/2052158031 >> 172.21.15.60:0/2052158031 conn(0x7fec180b7300 msgr2=0x7fec180b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:51.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1fbca640 1 -- 172.21.15.60:0/2052158031 shutdown_connections 2024-03-27T21:39:51.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.415+0000 7fec1fbca640 1 -- 172.21.15.60:0/2052158031 wait complete. 2024-03-27T21:39:51.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.416+0000 7fec1fbca640 10 monclient: build_initial_monmap 2024-03-27T21:39:51.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.416+0000 7fec1fbca640 10 monclient: monmap: 2024-03-27T21:39:51.414 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:51.414 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:51.414 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:51.414 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:51.414 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:51.414 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:51.414 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:51.415 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:51.415 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:51.415 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:51.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.416+0000 7fec1fbca640 1 librados: starting msgr at 2024-03-27T21:39:51.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.416+0000 7fec1fbca640 1 librados: starting objecter 2024-03-27T21:39:51.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.416+0000 7fec1fbca640 1 Processor -- start 2024-03-27T21:39:51.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.416+0000 7fec1fbca640 1 -- start start 2024-03-27T21:39:51.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.416+0000 7fec1fbca640 1 librados: setting wanted keys 2024-03-27T21:39:51.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.416+0000 7fec1fbca640 1 librados: calling monclient init 2024-03-27T21:39:51.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.416+0000 7fec1fbca640 10 monclient: init 2024-03-27T21:39:51.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1fbca640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:51.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1fbca640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:39:51.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1fbca640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec18071a50 0x7fec180b6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:51.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1fbca640 10 monclient(hunting): picked mon.noname-b con 0x7fec18071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:51.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1fbca640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fec180a87e0 0x7fec181d9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:51.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1fbca640 10 monclient(hunting): picked mon.noname-c con 0x7fec180a87e0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:51.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1fbca640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec1813ef20 0x7fec1813ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:51.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1fbca640 10 monclient(hunting): picked mon.noname-a con 0x7fec1813ef20 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:51.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1fbca640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:51.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1fbca640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec18138c60 con 0x7fec1813ef20 2024-03-27T21:39:51.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1fbca640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:51.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1fbca640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec181d9d90 con 0x7fec18071a50 2024-03-27T21:39:51.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1fbca640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:51.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1fbca640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec1813b140 con 0x7fec180a87e0 2024-03-27T21:39:51.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1fbca640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:51.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1fbca640 10 monclient(hunting): authenticate will time out at 832.638170s 2024-03-27T21:39:51.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1e167640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec1813ef20 0x7fec1813ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:51.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1d165640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fec180a87e0 0x7fec181d9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:51.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1e167640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec1813ef20 0x7fec1813ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48120/0 (socket says 172.21.15.60:48120) 2024-03-27T21:39:51.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1e167640 1 -- 172.21.15.60:0/2023067077 learned_addr learned my addr 172.21.15.60:0/2023067077 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:51.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1d165640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fec180a87e0 0x7fec181d9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47924/0 (socket says 172.21.15.60:47924) 2024-03-27T21:39:51.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1e167640 10 monclient(hunting): get_auth_request con 0x7fec1813ef20 auth_method 0 2024-03-27T21:39:51.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1e167640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:51.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1e167640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:51.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1e167640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:51.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1d165640 10 monclient(hunting): get_auth_request con 0x7fec180a87e0 auth_method 0 2024-03-27T21:39:51.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1d165640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:51.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1d165640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:51.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1d165640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:51.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1d966640 1 --2- 172.21.15.60:0/2023067077 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec18071a50 0x7fec180b6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:51.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1e167640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:51.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1e167640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:51.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1e167640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:51.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1d966640 10 monclient(hunting): get_auth_request con 0x7fec18071a50 auth_method 0 2024-03-27T21:39:51.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1d966640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:51.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1d966640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:51.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1d966640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:51.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1d165640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:51.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1d165640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:51.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1d165640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:51.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1d966640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:51.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1d966640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:51.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1d966640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:51.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1e167640 10 monclient(hunting): handle_auth_done global_id 4428 payload 824 2024-03-27T21:39:51.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1e167640 1 -- 172.21.15.60:0/2023067077 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fec180a87e0 msgr2=0x7fec181d9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:51.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1e167640 1 --2- 172.21.15.60:0/2023067077 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fec180a87e0 0x7fec181d9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1e167640 1 -- 172.21.15.60:0/2023067077 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec18071a50 msgr2=0x7fec180b6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:51.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1e167640 1 --2- 172.21.15.60:0/2023067077 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec18071a50 0x7fec180b6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1e167640 10 monclient: _finish_hunting 0 2024-03-27T21:39:51.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1e167640 1 monclient: found mon.noname-a 2024-03-27T21:39:51.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1e167640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:51.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1e167640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:51.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1e167640 1 -- 172.21.15.60:0/2023067077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec10009a70 con 0x7fec1813ef20 2024-03-27T21:39:51.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec1e167640 1 --2- 172.21.15.60:0/2023067077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec1813ef20 0x7fec1813ac00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fec10009d90 tx=0x7fec10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:51.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec0e7fc640 10 client.?.objecter ms_handle_connect 0x7fec1813ef20 2024-03-27T21:39:51.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec0e7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:51.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.417+0000 7fec0e7fc640 1 -- 172.21.15.60:0/2023067077 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1003d600 con 0x7fec1813ef20 2024-03-27T21:39:51.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:51.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec0e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:51.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec0e7fc640 10 monclient: dump: 2024-03-27T21:39:51.420 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:51.420 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:51.420 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.420 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.420 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:51.420 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:51.420 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:51.420 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:51.420 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:51.420 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:51.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec0e7fc640 10 monclient: _finish_auth 0 2024-03-27T21:39:51.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec0e7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:39:51.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:51.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec0e7fc640 1 -- 172.21.15.60:0/2023067077 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1003d780 con 0x7fec1813ef20 2024-03-27T21:39:51.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec1fbca640 5 monclient: authenticate success, global_id 4428 2024-03-27T21:39:51.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec1fbca640 10 monclient: _renew_subs 2024-03-27T21:39:51.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec1fbca640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:51.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec1fbca640 1 -- 172.21.15.60:0/2023067077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec181375b0 con 0x7fec1813ef20 2024-03-27T21:39:51.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:51.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec0e7fc640 1 -- 172.21.15.60:0/2023067077 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1003db30 con 0x7fec1813ef20 2024-03-27T21:39:51.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:51.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec0e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:51.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec0e7fc640 10 monclient: dump: 2024-03-27T21:39:51.421 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:51.421 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:51.421 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.422 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.422 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:51.422 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:51.422 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:51.422 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:51.422 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:51.422 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:51.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec1fbca640 10 client.4428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:51.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec1fbca640 10 monclient: _renew_subs 2024-03-27T21:39:51.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec0ffff640 4 set_mon_vals no callback set 2024-03-27T21:39:51.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec1fbca640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:51.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec1fbca640 1 -- 172.21.15.60:0/2023067077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec18139080 con 0x7fec1813ef20 2024-03-27T21:39:51.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec1fbca640 1 librados: init done 2024-03-27T21:39:51.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec1fbca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:51.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec1fbca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:51.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec1fbca640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:51.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.418+0000 7fec1fbca640 1 -- 172.21.15.60:0/2023067077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec18138da0 con 0x7fec1813ef20 2024-03-27T21:39:51.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.419+0000 7fec0e7fc640 1 -- 172.21.15.60:0/2023067077 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fec1003b040 con 0x7fec1813ef20 2024-03-27T21:39:51.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.419+0000 7fec0e7fc640 1 --2- 172.21.15.60:0/2023067077 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7febfc06c590 0x7febfc06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:51.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.420+0000 7fec1d966640 1 --2- 172.21.15.60:0/2023067077 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7febfc06c590 0x7febfc06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:51.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.420+0000 7fec1d966640 10 monclient: get_auth_request con 0x7febfc06c590 auth_method 0 2024-03-27T21:39:51.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.420+0000 7fec0e7fc640 1 -- 172.21.15.60:0/2023067077 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fec100ab060 con 0x7fec1813ef20 2024-03-27T21:39:51.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.420+0000 7fec0e7fc640 10 client.4428.objecter ms_dispatch 0x7fec18000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:51.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.420+0000 7fec0e7fc640 3 client.4428.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:51.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.420+0000 7fec0e7fc640 3 client.4428.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:51.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.420+0000 7fec0e7fc640 20 client.4428.objecter dump_active .. 0 homeless 2024-03-27T21:39:51.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.420+0000 7fec0e7fc640 10 client.4428.objecter ms_handle_connect 0x7febfc06c590 2024-03-27T21:39:51.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.420+0000 7fec1d966640 1 --2- 172.21.15.60:0/2023067077 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7febfc06c590 0x7febfc06ea50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fec14007f70 tx=0x7fec14007f00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:51.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.424+0000 7fec0e7fc640 1 -- 172.21.15.60:0/2023067077 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fec18138da0 con 0x7fec1813ef20 2024-03-27T21:39:51.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.424+0000 7fec0e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:51.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.424+0000 7fec0e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:51.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.604+0000 7fec1fbca640 10 start_mon_command cmd=[{"prefix": "config set", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes", "value": "true"}] 2024-03-27T21:39:51.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.604+0000 7fec1fbca640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes", "value": "true"}] 2024-03-27T21:39:51.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.604+0000 7fec1fbca640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:51.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.604+0000 7fec1fbca640 1 -- 172.21.15.60:0/2023067077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}] v 0) v1 -- 0x7fec18138da0 con 0x7fec1813ef20 2024-03-27T21:39:51.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.608+0000 7fec0e7fc640 1 -- 172.21.15.60:0/2023067077 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}]=0 v4)=0 v4) v1 ==== 136+0+0 (secure 0 0 0) 0x7fec18138da0 con 0x7fec1813ef20 2024-03-27T21:39:51.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.608+0000 7fec0e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes", "value": "true"}] 2024-03-27T21:39:51.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.608+0000 7fec0e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:51.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.610+0000 7fec1fbca640 10 librados: watch_flush enter 2024-03-27T21:39:51.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.610+0000 7fec1fbca640 10 librados: watch_flush exit 2024-03-27T21:39:51.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.610+0000 7fec1fbca640 20 client.4428.objecter shutdown clearing up homeless session... 2024-03-27T21:39:51.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.610+0000 7fec1fbca640 10 client.4428.objecter successfully canceled tick 2024-03-27T21:39:51.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.610+0000 7fec1fbca640 1 -- 172.21.15.60:0/2023067077 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7febfc06c590 msgr2=0x7febfc06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:51.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.610+0000 7fec1fbca640 1 --2- 172.21.15.60:0/2023067077 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7febfc06c590 0x7febfc06ea50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fec14007f70 tx=0x7fec14007f00 comp rx=0 tx=0).stop 2024-03-27T21:39:51.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.610+0000 7fec1fbca640 10 monclient: shutdown 2024-03-27T21:39:51.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.610+0000 7fec1fbca640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:51.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.610+0000 7fec1fbca640 1 -- 172.21.15.60:0/2023067077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec1813ef20 msgr2=0x7fec1813ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:51.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.610+0000 7fec1fbca640 1 --2- 172.21.15.60:0/2023067077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec1813ef20 0x7fec1813ac00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fec10009d90 tx=0x7fec10004a40 comp rx=0 tx=0).stop 2024-03-27T21:39:51.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.610+0000 7fec1fbca640 1 -- 172.21.15.60:0/2023067077 shutdown_connections 2024-03-27T21:39:51.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.610+0000 7fec1fbca640 1 --2- 172.21.15.60:0/2023067077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec1813ef20 0x7fec1813ac00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.610+0000 7fec1fbca640 1 --2- 172.21.15.60:0/2023067077 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fec180a87e0 0x7fec181d9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.610+0000 7fec1fbca640 1 --2- 172.21.15.60:0/2023067077 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7febfc06c590 0x7febfc06ea50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.610+0000 7fec1fbca640 1 --2- 172.21.15.60:0/2023067077 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec18071a50 0x7fec180b6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.610+0000 7fec1fbca640 1 -- 172.21.15.60:0/2023067077 >> 172.21.15.60:0/2023067077 conn(0x7fec180b7300 msgr2=0x7fec1813b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.610+0000 7fec1fbca640 1 -- 172.21.15.60:0/2023067077 shutdown_connections 2024-03-27T21:39:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.610+0000 7fec1fbca640 1 -- 172.21.15.60:0/2023067077 wait complete. 2024-03-27T21:39:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.610+0000 7fec1fbca640 1 librados: shutdown 2024-03-27T21:39:51.608 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-27T21:39:51.608 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-27T21:39:51.608 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T21:39:51.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.721+0000 7f6dcd9c2640 10 monclient: get_monmap_and_config 2024-03-27T21:39:51.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.721+0000 7f6dcd9c2640 10 monclient: build_initial_monmap 2024-03-27T21:39:51.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.721+0000 7f6dcd9c2640 10 monclient: monmap: 2024-03-27T21:39:51.709 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:51.709 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:51.709 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:51.722910+0000 2024-03-27T21:39:51.709 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:51.722910+0000 2024-03-27T21:39:51.709 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:51.709 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:51.710 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:51.710 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:51.710 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:51.710 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:51.710 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:51.710 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:51.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.722+0000 7f6dcd9c2640 1 Processor -- start 2024-03-27T21:39:51.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.722+0000 7f6dcd9c2640 1 -- start start 2024-03-27T21:39:51.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.722+0000 7f6dcd9c2640 10 monclient: init 2024-03-27T21:39:51.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dcd9c2640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:51.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dcd9c2640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-03-27T21:39:51.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dcd9c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6dc8071a50 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:51.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dcd9c2640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6dc813ef20 0x7f6dc80a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:51.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dcd9c2640 10 monclient(hunting): picked mon.noname-c con 0x7f6dc813ef20 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:51.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dcd9c2640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6dc8141630 0x7f6dc8141ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:51.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dcd9c2640 10 monclient(hunting): picked mon.noname-b con 0x7f6dc8141630 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:51.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dcd9c2640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc80a8e90 con 0x7f6dc8071a50 2024-03-27T21:39:51.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dcd9c2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:51.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dcd9c2640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dc8071e30 con 0x7f6dc813ef20 2024-03-27T21:39:51.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dcd9c2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:51.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dcd9c2640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc80aa040 con 0x7f6dc8141630 2024-03-27T21:39:51.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dcd9c2640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:51.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dcd9c2640 10 monclient(hunting): authenticate will time out at 832.944157s 2024-03-27T21:39:51.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dc77fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6dc8141630 0x7f6dc8141ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:51.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dc77fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6dc8141630 0x7f6dc8141ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47740/0 (socket says 172.21.15.60:47740) 2024-03-27T21:39:51.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dc77fe640 1 -- 172.21.15.60:0/1020583148 learned_addr learned my addr 172.21.15.60:0/1020583148 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:51.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dc77fe640 10 monclient(hunting): get_auth_request con 0x7f6dc8141630 auth_method 0 2024-03-27T21:39:51.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:51.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:51.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:51.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:51.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:51.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:51.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dc77fe640 10 monclient(hunting): handle_auth_done global_id 4367 payload 274 2024-03-27T21:39:51.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dbffff640 1 --2- 172.21.15.60:0/1020583148 >> v2:172.21.15.191:3300/0 conn(0x7f6dc813ef20 0x7f6dc80a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:51.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dc77fe640 1 -- 172.21.15.60:0/1020583148 >> v2:172.21.15.191:3300/0 conn(0x7f6dc813ef20 msgr2=0x7f6dc80a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:51.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dc77fe640 1 --2- 172.21.15.60:0/1020583148 >> v2:172.21.15.191:3300/0 conn(0x7f6dc813ef20 0x7f6dc80a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dc77fe640 1 -- 172.21.15.60:0/1020583148 >> v1:172.21.15.191:6789/0 conn(0x7f6dc8071a50 legacy=0x7f6dc813e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:51.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dc77fe640 10 monclient: _finish_hunting 0 2024-03-27T21:39:51.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dc77fe640 1 monclient: found mon.noname-b 2024-03-27T21:39:51.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:51.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dc77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:51.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dc77fe640 1 -- 172.21.15.60:0/1020583148 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc80aa2b0 con 0x7f6dc8141630 2024-03-27T21:39:51.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dbffff640 10 monclient: get_auth_request con 0x7f6dc813ef20 auth_method 0 2024-03-27T21:39:51.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.723+0000 7f6dbffff640 1 --2- 172.21.15.60:0/1020583148 >> v2:172.21.15.191:3300/0 conn(0x7f6dc813ef20 0x7f6dc80a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:39:51.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.724+0000 7f6dc77fe640 1 --2- 172.21.15.60:0/1020583148 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6dc8141630 0x7f6dc8141ac0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0009e50 tx=0x7f6dc002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10c6cb7c139798b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:51.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.724+0000 7f6dc67fc640 1 -- 172.21.15.60:0/1020583148 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc002ca70 con 0x7f6dc8141630 2024-03-27T21:39:51.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.724+0000 7f6dc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:51.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.724+0000 7f6dc67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:51.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.724+0000 7f6dc67fc640 10 monclient: dump: 2024-03-27T21:39:51.729 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:51.729 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:51.729 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.729 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.729 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:51.730 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:51.730 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:51.730 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:51.730 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:51.730 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:51.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.724+0000 7f6dc67fc640 10 monclient: _finish_auth 0 2024-03-27T21:39:51.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.724+0000 7f6dc67fc640 10 monclient: _check_auth_tickets 2024-03-27T21:39:51.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.724+0000 7f6dc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:51.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.724+0000 7f6dc67fc640 1 -- 172.21.15.60:0/1020583148 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dc002cbd0 con 0x7f6dc8141630 2024-03-27T21:39:51.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.724+0000 7f6dcd9c2640 5 monclient: authenticate success, global_id 4367 2024-03-27T21:39:51.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.724+0000 7f6dcd9c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:51.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.724+0000 7f6dc67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:51.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.724+0000 7f6dcd9c2640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:51.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.724+0000 7f6dc67fc640 1 -- 172.21.15.60:0/1020583148 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc00343f0 con 0x7f6dc8141630 2024-03-27T21:39:51.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.724+0000 7f6dc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:51.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.724+0000 7f6dcd9c2640 4 set_mon_vals no callback set 2024-03-27T21:39:51.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.724+0000 7f6dc67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:51.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.724+0000 7f6dc67fc640 10 monclient: dump: 2024-03-27T21:39:51.732 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:51.732 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:51.732 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.732 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.732 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:51.733 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:51.733 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:51.733 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:51.733 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:51.733 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:51.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.725+0000 7f6dcd9c2640 10 monclient: shutdown 2024-03-27T21:39:51.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.725+0000 7f6dcd9c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:51.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.725+0000 7f6dcd9c2640 1 -- 172.21.15.60:0/1020583148 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6dc8141630 msgr2=0x7f6dc8141ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:51.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.725+0000 7f6dcd9c2640 1 --2- 172.21.15.60:0/1020583148 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6dc8141630 0x7f6dc8141ac0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0009e50 tx=0x7f6dc002bed0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.725+0000 7f6dcd9c2640 1 -- 172.21.15.60:0/1020583148 shutdown_connections 2024-03-27T21:39:51.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.725+0000 7f6dcd9c2640 1 --2- 172.21.15.60:0/1020583148 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6dc8141630 0x7f6dc8141ac0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.725+0000 7f6dcd9c2640 1 --2- 172.21.15.60:0/1020583148 >> v2:172.21.15.191:3300/0 conn(0x7f6dc813ef20 0x7f6dc80a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.725+0000 7f6dcd9c2640 1 -- 172.21.15.60:0/1020583148 >> 172.21.15.60:0/1020583148 conn(0x7f6dc80b7300 msgr2=0x7f6dc80b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:51.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.725+0000 7f6dcd9c2640 1 -- 172.21.15.60:0/1020583148 shutdown_connections 2024-03-27T21:39:51.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.725+0000 7f6dcd9c2640 1 -- 172.21.15.60:0/1020583148 wait complete. 2024-03-27T21:39:51.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.725+0000 7f6dcd9c2640 10 monclient: build_initial_monmap 2024-03-27T21:39:51.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.725+0000 7f6dcd9c2640 10 monclient: monmap: 2024-03-27T21:39:51.734 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:51.734 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:51.734 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:51.734 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:51.734 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:51.734 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:51.734 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:51.735 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:51.735 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:51.735 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:51.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.725+0000 7f6dcd9c2640 1 librados: starting msgr at 2024-03-27T21:39:51.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.725+0000 7f6dcd9c2640 1 librados: starting objecter 2024-03-27T21:39:51.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dcd9c2640 1 Processor -- start 2024-03-27T21:39:51.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dcd9c2640 1 -- start start 2024-03-27T21:39:51.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dcd9c2640 1 librados: setting wanted keys 2024-03-27T21:39:51.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dcd9c2640 1 librados: calling monclient init 2024-03-27T21:39:51.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dcd9c2640 10 monclient: init 2024-03-27T21:39:51.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dcd9c2640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:51.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dcd9c2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:39:51.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dcd9c2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dc8071a50 0x7f6dc8135ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:51.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dcd9c2640 10 monclient(hunting): picked mon.noname-b con 0x7f6dc8071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:51.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dcd9c2640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6dc813ef20 0x7f6dc81d6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:51.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dcd9c2640 10 monclient(hunting): picked mon.noname-c con 0x7f6dc813ef20 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:51.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dcd9c2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dc8141630 0x7f6dc81d6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:51.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dcd9c2640 10 monclient(hunting): picked mon.noname-a con 0x7f6dc8141630 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:51.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dcd9c2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:51.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dcd9c2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc8136b10 con 0x7f6dc8141630 2024-03-27T21:39:51.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dcd9c2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:51.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dcd9c2640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc81399b0 con 0x7f6dc8071a50 2024-03-27T21:39:51.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dcd9c2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:51.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dcd9c2640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc81d6bb0 con 0x7f6dc813ef20 2024-03-27T21:39:51.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dcd9c2640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:51.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dcd9c2640 10 monclient(hunting): authenticate will time out at 832.947582s 2024-03-27T21:39:51.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dbffff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6dc813ef20 0x7f6dc81d6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:51.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dbffff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6dc813ef20 0x7f6dc81d6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47752/0 (socket says 172.21.15.60:47752) 2024-03-27T21:39:51.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dbffff640 1 -- 172.21.15.60:0/3469809510 learned_addr learned my addr 172.21.15.60:0/3469809510 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:51.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dbffff640 10 monclient(hunting): get_auth_request con 0x7f6dc813ef20 auth_method 0 2024-03-27T21:39:51.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:51.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dbffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:51.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:51.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dc77fe640 1 --2- 172.21.15.60:0/3469809510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dc8141630 0x7f6dc81d6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:51.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dc77fe640 10 monclient(hunting): get_auth_request con 0x7f6dc8141630 auth_method 0 2024-03-27T21:39:51.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:51.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:51.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:51.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:51.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:51.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:51.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:51.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:51.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:51.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dc6ffd640 1 --2- 172.21.15.60:0/3469809510 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dc8071a50 0x7f6dc8135ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:51.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.726+0000 7f6dbffff640 10 monclient(hunting): handle_auth_done global_id 4370 payload 824 2024-03-27T21:39:51.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dbffff640 1 -- 172.21.15.60:0/3469809510 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dc8071a50 msgr2=0x7f6dc8135ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:51.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dbffff640 1 --2- 172.21.15.60:0/3469809510 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dc8071a50 0x7f6dc8135ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dbffff640 1 -- 172.21.15.60:0/3469809510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dc8141630 msgr2=0x7f6dc81d6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:51.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dbffff640 1 --2- 172.21.15.60:0/3469809510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dc8141630 0x7f6dc81d6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dbffff640 10 monclient: _finish_hunting 0 2024-03-27T21:39:51.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dbffff640 1 monclient: found mon.noname-c 2024-03-27T21:39:51.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:51.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dbffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:39:51.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dbffff640 1 -- 172.21.15.60:0/3469809510 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc0009b30 con 0x7f6dc813ef20 2024-03-27T21:39:51.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dc6ffd640 10 monclient: get_auth_request con 0x7f6dc8071a50 auth_method 0 2024-03-27T21:39:51.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dc6ffd640 1 --2- 172.21.15.60:0/3469809510 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dc8071a50 0x7f6dc8135ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:39:51.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dbffff640 1 --2- 172.21.15.60:0/3469809510 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6dc813ef20 0x7f6dc81d6130 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6db000c7f0 tx=0x7f6db000ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:51.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dc4ff9640 10 client.?.objecter ms_handle_connect 0x7f6dc813ef20 2024-03-27T21:39:51.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dc4ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:51.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dc4ff9640 1 -- 172.21.15.60:0/3469809510 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db0007820 con 0x7f6dc813ef20 2024-03-27T21:39:51.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dc4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:51.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dc4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:51.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dc4ff9640 10 monclient: dump: 2024-03-27T21:39:51.740 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:51.740 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:51.741 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.741 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.741 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:51.741 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:51.741 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:51.741 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:51.741 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:51.741 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:51.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dc4ff9640 10 monclient: _finish_auth 0 2024-03-27T21:39:51.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dc4ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:39:51.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dc4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:51.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dcd9c2640 5 monclient: authenticate success, global_id 4370 2024-03-27T21:39:51.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dc4ff9640 1 -- 172.21.15.60:0/3469809510 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db0007980 con 0x7f6dc813ef20 2024-03-27T21:39:51.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dcd9c2640 10 monclient: _renew_subs 2024-03-27T21:39:51.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dcd9c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:51.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dcd9c2640 1 -- 172.21.15.60:0/3469809510 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc8139440 con 0x7f6dc813ef20 2024-03-27T21:39:51.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dc4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:51.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dc4ff9640 1 -- 172.21.15.60:0/3469809510 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db0007d10 con 0x7f6dc813ef20 2024-03-27T21:39:51.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dc4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:51.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dc4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:51.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dc4ff9640 10 monclient: dump: 2024-03-27T21:39:51.743 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:51.743 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:51.743 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.743 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:51.743 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:51.744 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:51.744 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:51.744 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:51.744 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:51.744 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:51.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dcd9c2640 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:51.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dc5ffb640 4 set_mon_vals no callback set 2024-03-27T21:39:51.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dcd9c2640 10 monclient: _renew_subs 2024-03-27T21:39:51.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dcd9c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:51.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dcd9c2640 1 -- 172.21.15.60:0/3469809510 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc81390d0 con 0x7f6dc813ef20 2024-03-27T21:39:51.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.727+0000 7f6dcd9c2640 1 librados: init done 2024-03-27T21:39:51.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.728+0000 7f6dcd9c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:51.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.728+0000 7f6dcd9c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:51.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.728+0000 7f6dcd9c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:51.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.728+0000 7f6dcd9c2640 1 -- 172.21.15.60:0/3469809510 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dc8139b70 con 0x7f6dc813ef20 2024-03-27T21:39:51.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.728+0000 7f6dc4ff9640 1 -- 172.21.15.60:0/3469809510 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6db00113f0 con 0x7f6dc813ef20 2024-03-27T21:39:51.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.729+0000 7f6dc4ff9640 1 --2- 172.21.15.60:0/3469809510 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6da806c590 0x7f6da806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:51.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.729+0000 7f6dc4ff9640 1 -- 172.21.15.60:0/3469809510 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6db0089150 con 0x7f6dc813ef20 2024-03-27T21:39:51.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.729+0000 7f6dc4ff9640 10 client.4370.objecter ms_dispatch 0x7f6dc8000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:51.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.729+0000 7f6dc4ff9640 3 client.4370.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:51.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.729+0000 7f6dc4ff9640 3 client.4370.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:51.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.729+0000 7f6dc6ffd640 1 --2- 172.21.15.60:0/3469809510 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6da806c590 0x7f6da806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:51.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.729+0000 7f6dc6ffd640 10 monclient: get_auth_request con 0x7f6da806c590 auth_method 0 2024-03-27T21:39:51.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.729+0000 7f6dc4ff9640 20 client.4370.objecter dump_active .. 0 homeless 2024-03-27T21:39:51.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.729+0000 7f6dc4ff9640 10 client.4370.objecter ms_handle_connect 0x7f6da806c590 2024-03-27T21:39:51.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.729+0000 7f6dc6ffd640 1 --2- 172.21.15.60:0/3469809510 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6da806c590 0x7f6da806ea50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6db8003eb0 tx=0x7f6db8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:51.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.733+0000 7f6dc4ff9640 1 -- 172.21.15.60:0/3469809510 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6db0054660 con 0x7f6dc813ef20 2024-03-27T21:39:51.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.733+0000 7f6dc4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:51.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.733+0000 7f6dc4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:51.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.908+0000 7f6dcd9c2640 1 -- 172.21.15.60:0/3469809510 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6dc81395d0 con 0x7f6da806c590 2024-03-27T21:39:51.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.910+0000 7f6dc4ff9640 1 -- 172.21.15.60:0/3469809510 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48297 (secure 0 0 0) 0x7f6dc81395d0 con 0x7f6da806c590 2024-03-27T21:39:51.897 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:39:51.897 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-03-27T21:39:51.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.912+0000 7f6dcd9c2640 10 librados: watch_flush enter 2024-03-27T21:39:51.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.912+0000 7f6dcd9c2640 10 librados: watch_flush exit 2024-03-27T21:39:51.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.912+0000 7f6dcd9c2640 20 client.4370.objecter shutdown clearing up homeless session... 2024-03-27T21:39:51.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.912+0000 7f6dcd9c2640 10 client.4370.objecter successfully canceled tick 2024-03-27T21:39:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.912+0000 7f6dcd9c2640 1 -- 172.21.15.60:0/3469809510 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6da806c590 msgr2=0x7f6da806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.912+0000 7f6dcd9c2640 1 --2- 172.21.15.60:0/3469809510 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6da806c590 0x7f6da806ea50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6db8003eb0 tx=0x7f6db8009040 comp rx=0 tx=0).stop 2024-03-27T21:39:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.912+0000 7f6dcd9c2640 10 monclient: shutdown 2024-03-27T21:39:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.912+0000 7f6dcd9c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.912+0000 7f6dcd9c2640 1 -- 172.21.15.60:0/3469809510 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6dc813ef20 msgr2=0x7f6dc81d6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.912+0000 7f6dcd9c2640 1 --2- 172.21.15.60:0/3469809510 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6dc813ef20 0x7f6dc81d6130 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6db000c7f0 tx=0x7f6db000ccb0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.912+0000 7f6dcd9c2640 1 -- 172.21.15.60:0/3469809510 shutdown_connections 2024-03-27T21:39:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.912+0000 7f6dcd9c2640 1 --2- 172.21.15.60:0/3469809510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dc8141630 0x7f6dc81d6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.912+0000 7f6dcd9c2640 1 --2- 172.21.15.60:0/3469809510 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6dc813ef20 0x7f6dc81d6130 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.912+0000 7f6dcd9c2640 1 --2- 172.21.15.60:0/3469809510 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6da806c590 0x7f6da806ea50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.912+0000 7f6dcd9c2640 1 --2- 172.21.15.60:0/3469809510 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dc8071a50 0x7f6dc8135ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.912+0000 7f6dcd9c2640 1 -- 172.21.15.60:0/3469809510 >> 172.21.15.60:0/3469809510 conn(0x7f6dc80b7300 msgr2=0x7f6dc813dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.912+0000 7f6dcd9c2640 1 -- 172.21.15.60:0/3469809510 shutdown_connections 2024-03-27T21:39:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.912+0000 7f6dcd9c2640 1 -- 172.21.15.60:0/3469809510 wait complete. 2024-03-27T21:39:51.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:51.912+0000 7f6dcd9c2640 1 librados: shutdown 2024-03-27T21:39:51.909 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-27T21:39:51.391200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":221912,"kb_used_data":1632,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":754752808,"statfs":{"total":773094113280,"available":772866875392,"internally_reserved":0,"allocated":1671168,"data_stored":1579288,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":12742,"internal_metadata":225431098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001315"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:46.474962+0000","last_change":"2024-03-27T21:39:46.470433+0000","last_active":"2024-03-27T21:39:46.474962+0000","last_peered":"2024-03-27T21:39:46.474962+0000","last_clean":"2024-03-27T21:39:46.474962+0000","last_became_active":"2024-03-27T21:39:46.470153+0000","last_became_peered":"2024-03-27T21:39:46.470153+0000","last_unstale":"2024-03-27T21:39:46.474962+0000","last_undegraded":"2024-03-27T21:39:46.474962+0000","last_fullsized":"2024-03-27T21:39:46.474962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T21:39:45.811197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:46.468505+0000","last_change":"2024-03-27T21:39:46.467893+0000","last_active":"2024-03-27T21:39:46.468505+0000","last_peered":"2024-03-27T21:39:46.468505+0000","last_clean":"2024-03-27T21:39:46.468505+0000","last_became_active":"2024-03-27T21:39:46.467576+0000","last_became_peered":"2024-03-27T21:39:46.467576+0000","last_unstale":"2024-03-27T21:39:46.468505+0000","last_undegraded":"2024-03-27T21:39:46.468505+0000","last_fullsized":"2024-03-27T21:39:46.468505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:29:50.190006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:47.405006+0000","last_change":"2024-03-27T21:39:46.679788+0000","last_active":"2024-03-27T21:39:47.405006+0000","last_peered":"2024-03-27T21:39:47.405006+0000","last_clean":"2024-03-27T21:39:47.405006+0000","last_became_active":"2024-03-27T21:39:46.679561+0000","last_became_peered":"2024-03-27T21:39:46.679561+0000","last_unstale":"2024-03-27T21:39:47.405006+0000","last_undegraded":"2024-03-27T21:39:47.405006+0000","last_fullsized":"2024-03-27T21:39:47.405006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T23:53:50.373923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:46.679751+0000","last_change":"2024-03-27T21:39:46.679751+0000","last_active":"2024-03-27T21:39:46.679751+0000","last_peered":"2024-03-27T21:39:46.679751+0000","last_clean":"2024-03-27T21:39:46.679751+0000","last_became_active":"2024-03-27T21:39:46.679509+0000","last_became_peered":"2024-03-27T21:39:46.679509+0000","last_unstale":"2024-03-27T21:39:46.679751+0000","last_undegraded":"2024-03-27T21:39:46.679751+0000","last_fullsized":"2024-03-27T21:39:46.679751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:43:02.145158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T21:39:47.458020+0000","last_change":"2024-03-27T21:39:46.459503+0000","last_active":"2024-03-27T21:39:47.458020+0000","last_peered":"2024-03-27T21:39:47.458020+0000","last_clean":"2024-03-27T21:39:47.458020+0000","last_became_active":"2024-03-27T21:39:46.459225+0000","last_became_peered":"2024-03-27T21:39:46.459225+0000","last_unstale":"2024-03-27T21:39:47.458020+0000","last_undegraded":"2024-03-27T21:39:47.458020+0000","last_fullsized":"2024-03-27T21:39:47.458020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:17:43.512127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:46.680826+0000","last_change":"2024-03-27T21:39:46.679791+0000","last_active":"2024-03-27T21:39:46.680826+0000","last_peered":"2024-03-27T21:39:46.680826+0000","last_clean":"2024-03-27T21:39:46.680826+0000","last_became_active":"2024-03-27T21:39:46.679503+0000","last_became_peered":"2024-03-27T21:39:46.679503+0000","last_unstale":"2024-03-27T21:39:46.680826+0000","last_undegraded":"2024-03-27T21:39:46.680826+0000","last_fullsized":"2024-03-27T21:39:46.680826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:07:25.151346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:47.404983+0000","last_change":"2024-03-27T21:39:46.458326+0000","last_active":"2024-03-27T21:39:47.404983+0000","last_peered":"2024-03-27T21:39:47.404983+0000","last_clean":"2024-03-27T21:39:47.404983+0000","last_became_active":"2024-03-27T21:39:46.458074+0000","last_became_peered":"2024-03-27T21:39:46.458074+0000","last_unstale":"2024-03-27T21:39:47.404983+0000","last_undegraded":"2024-03-27T21:39:47.404983+0000","last_fullsized":"2024-03-27T21:39:47.404983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:06:29.798229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T21:39:47.458062+0000","last_change":"2024-03-27T21:39:46.459166+0000","last_active":"2024-03-27T21:39:47.458062+0000","last_peered":"2024-03-27T21:39:47.458062+0000","last_clean":"2024-03-27T21:39:47.458062+0000","last_became_active":"2024-03-27T21:39:46.458958+0000","last_became_peered":"2024-03-27T21:39:46.458958+0000","last_unstale":"2024-03-27T21:39:47.458062+0000","last_undegraded":"2024-03-27T21:39:47.458062+0000","last_fullsized":"2024-03-27T21:39:47.458062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T22:18:37.511546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":163,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:46.455960+0000","last_change":"2024-03-27T21:39:39.866806+0000","last_active":"2024-03-27T21:39:46.455960+0000","last_peered":"2024-03-27T21:39:46.455960+0000","last_clean":"2024-03-27T21:39:46.455960+0000","last_became_active":"2024-03-27T21:39:39.865948+0000","last_became_peered":"2024-03-27T21:39:39.865948+0000","last_unstale":"2024-03-27T21:39:46.455960+0000","last_undegraded":"2024-03-27T21:39:46.455960+0000","last_fullsized":"2024-03-27T21:39:46.455960+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_clean_scrub_stamp":"2024-03-27T21:39:38.411432+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:36:49.375869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":16,"seq":68719476743,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]}]},{"osd":3,"up_from":16,"seq":68719476742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31848905},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5620000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90700000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4790000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2849999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]}]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":598418,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]}]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]}]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":592670,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T21:39:51.910 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T21:39:52.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.023+0000 7f3d40c5c640 10 monclient: get_monmap_and_config 2024-03-27T21:39:52.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.023+0000 7f3d40c5c640 10 monclient: build_initial_monmap 2024-03-27T21:39:52.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.023+0000 7f3d40c5c640 10 monclient: monmap: 2024-03-27T21:39:52.010 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:52.010 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.010 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:52.024724+0000 2024-03-27T21:39:52.010 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:52.024724+0000 2024-03-27T21:39:52.010 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:52.011 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:52.011 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:52.011 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:52.011 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:52.011 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:52.011 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:52.011 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.024+0000 7f3d40c5c640 1 Processor -- start 2024-03-27T21:39:52.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.024+0000 7f3d40c5c640 1 -- start start 2024-03-27T21:39:52.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.024+0000 7f3d40c5c640 10 monclient: init 2024-03-27T21:39:52.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.024+0000 7f3d40c5c640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:52.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.024+0000 7f3d40c5c640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-03-27T21:39:52.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.024+0000 7f3d40c5c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3d3c071a50 0x7f3d3c0a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.024+0000 7f3d40c5c640 10 monclient(hunting): picked mon.noname-c con 0x7f3d3c071a50 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:52.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.024+0000 7f3d40c5c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3d3c13e810 0x7f3d3c13ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.024+0000 7f3d40c5c640 10 monclient(hunting): picked mon.noname-b con 0x7f3d3c13e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:52.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d40c5c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3d3c141630 0x7f3d3c141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d40c5c640 10 monclient(hunting): picked mon.noname-a con 0x7f3d3c141630 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:52.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d40c5c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d40c5c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d3c071e30 con 0x7f3d3c141630 2024-03-27T21:39:52.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d40c5c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d40c5c640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d3c13f190 con 0x7f3d3c071a50 2024-03-27T21:39:52.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d40c5c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d40c5c640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c0a8e90 con 0x7f3d3c13e810 2024-03-27T21:39:52.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d40c5c640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:52.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d40c5c640 10 monclient(hunting): authenticate will time out at 833.245955s 2024-03-27T21:39:52.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d3ad9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3d3c141630 0x7f3d3c141aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d31d9b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3d3c13e810 0x7f3d3c13ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d3ad9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3d3c141630 0x7f3d3c141aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44326/0 (socket says 172.21.15.60:44326) 2024-03-27T21:39:52.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d31d9b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3d3c13e810 0x7f3d3c13ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47762/0 (socket says 172.21.15.60:47762) 2024-03-27T21:39:52.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d3ad9d640 1 -- 172.21.15.60:0/588294370 learned_addr learned my addr 172.21.15.60:0/588294370 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:52.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d3ad9d640 10 monclient(hunting): get_auth_request con 0x7f3d3c141630 auth_method 0 2024-03-27T21:39:52.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d3ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d3ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d3ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d31d9b640 10 monclient(hunting): get_auth_request con 0x7f3d3c13e810 auth_method 0 2024-03-27T21:39:52.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d31d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d31d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d3ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:52.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d3ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:52.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d3ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:52.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d31d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:52.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d31d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:52.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d31d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d3a59c640 1 --2- 172.21.15.60:0/588294370 >> v2:172.21.15.191:3300/0 conn(0x7f3d3c071a50 0x7f3d3c0a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d3a59c640 10 monclient(hunting): get_auth_request con 0x7f3d3c071a50 auth_method 0 2024-03-27T21:39:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d3a59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d31d9b640 10 monclient(hunting): handle_auth_done global_id 4373 payload 274 2024-03-27T21:39:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d31d9b640 1 -- 172.21.15.60:0/588294370 >> v2:172.21.15.191:3300/0 conn(0x7f3d3c071a50 msgr2=0x7f3d3c0a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d31d9b640 1 --2- 172.21.15.60:0/588294370 >> v2:172.21.15.191:3300/0 conn(0x7f3d3c071a50 0x7f3d3c0a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d31d9b640 1 -- 172.21.15.60:0/588294370 >> v2:172.21.15.60:3300/0 conn(0x7f3d3c141630 msgr2=0x7f3d3c141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d31d9b640 1 --2- 172.21.15.60:0/588294370 >> v2:172.21.15.60:3300/0 conn(0x7f3d3c141630 0x7f3d3c141aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d31d9b640 10 monclient: _finish_hunting 0 2024-03-27T21:39:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d31d9b640 1 monclient: found mon.noname-b 2024-03-27T21:39:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d31d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d31d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:52.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d31d9b640 1 -- 172.21.15.60:0/588294370 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3c0aa040 con 0x7f3d3c13e810 2024-03-27T21:39:52.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.025+0000 7f3d3ad9d640 1 --2- 172.21.15.60:0/588294370 >> v2:172.21.15.60:3300/0 conn(0x7f3d3c141630 0x7f3d3c141aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:52.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d31d9b640 1 --2- 172.21.15.60:0/588294370 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3d3c13e810 0x7f3d3c13ec50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c0b4a80 tx=0x7f3d2402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dea0322c56f93 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:52.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d39d9b640 1 -- 172.21.15.60:0/588294370 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2402e070 con 0x7f3d3c13e810 2024-03-27T21:39:52.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d39d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:52.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d39d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:52.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d39d9b640 10 monclient: dump: 2024-03-27T21:39:52.024 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:52.024 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.024 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.024 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.025 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:52.025 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:52.025 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:52.025 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:52.025 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:52.025 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d39d9b640 10 monclient: _finish_auth 0 2024-03-27T21:39:52.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d39d9b640 10 monclient: _check_auth_tickets 2024-03-27T21:39:52.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d39d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:52.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d40c5c640 5 monclient: authenticate success, global_id 4373 2024-03-27T21:39:52.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d40c5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:52.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d39d9b640 1 -- 172.21.15.60:0/588294370 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d240315a0 con 0x7f3d3c13e810 2024-03-27T21:39:52.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d39d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:52.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d39d9b640 1 -- 172.21.15.60:0/588294370 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24031930 con 0x7f3d3c13e810 2024-03-27T21:39:52.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d39d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d39d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d39d9b640 10 monclient: dump: 2024-03-27T21:39:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:52.026 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d40c5c640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d40c5c640 4 set_mon_vals no callback set 2024-03-27T21:39:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d40c5c640 10 monclient: shutdown 2024-03-27T21:39:52.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d40c5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:52.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d40c5c640 1 -- 172.21.15.60:0/588294370 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3d3c13e810 msgr2=0x7f3d3c13ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d40c5c640 1 --2- 172.21.15.60:0/588294370 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3d3c13e810 0x7f3d3c13ec50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c0b4a80 tx=0x7f3d2402b840 comp rx=0 tx=0).stop 2024-03-27T21:39:52.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d40c5c640 1 -- 172.21.15.60:0/588294370 shutdown_connections 2024-03-27T21:39:52.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d40c5c640 1 --2- 172.21.15.60:0/588294370 >> v2:172.21.15.60:3300/0 conn(0x7f3d3c141630 0x7f3d3c141aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d40c5c640 1 --2- 172.21.15.60:0/588294370 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3d3c13e810 0x7f3d3c13ec50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d40c5c640 1 --2- 172.21.15.60:0/588294370 >> v2:172.21.15.191:3300/0 conn(0x7f3d3c071a50 0x7f3d3c0a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d40c5c640 1 -- 172.21.15.60:0/588294370 >> 172.21.15.60:0/588294370 conn(0x7f3d3c0b7300 msgr2=0x7f3d3c0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:52.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d40c5c640 1 -- 172.21.15.60:0/588294370 shutdown_connections 2024-03-27T21:39:52.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.026+0000 7f3d40c5c640 1 -- 172.21.15.60:0/588294370 wait complete. 2024-03-27T21:39:52.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.027+0000 7f3d40c5c640 10 monclient: build_initial_monmap 2024-03-27T21:39:52.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.027+0000 7f3d40c5c640 10 monclient: monmap: 2024-03-27T21:39:52.027 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:52.027 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:52.028 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:52.028 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:52.028 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:52.028 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:52.028 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:52.028 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:52.028 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:52.028 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.027+0000 7f3d40c5c640 1 librados: starting msgr at 2024-03-27T21:39:52.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 1 librados: starting objecter 2024-03-27T21:39:52.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 1 Processor -- start 2024-03-27T21:39:52.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 1 -- start start 2024-03-27T21:39:52.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 1 librados: setting wanted keys 2024-03-27T21:39:52.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 1 librados: calling monclient init 2024-03-27T21:39:52.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 10 monclient: init 2024-03-27T21:39:52.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:52.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:39:52.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3d3c071a50 0x7f3d3c138220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 10 monclient(hunting): picked mon.noname-c con 0x7f3d3c071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:52.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d3c13e810 0x7f3d3c1d9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 10 monclient(hunting): picked mon.noname-b con 0x7f3d3c13e810 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:52.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d3c141630 0x7f3d3c0ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 10 monclient(hunting): picked mon.noname-a con 0x7f3d3c141630 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:52.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c1da9e0 con 0x7f3d3c141630 2024-03-27T21:39:52.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c0adc60 con 0x7f3d3c13e810 2024-03-27T21:39:52.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c1da4e0 con 0x7f3d3c071a50 2024-03-27T21:39:52.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:52.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d40c5c640 10 monclient(hunting): authenticate will time out at 833.249817s 2024-03-27T21:39:52.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d3a59c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3d3c071a50 0x7f3d3c138220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.028+0000 7f3d3ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d3c141630 0x7f3d3c0ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3a59c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3d3c071a50 0x7f3d3c138220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47764/0 (socket says 172.21.15.60:47764) 2024-03-27T21:39:52.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3a59c640 1 -- 172.21.15.60:0/405715916 learned_addr learned my addr 172.21.15.60:0/405715916 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:52.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3a59c640 10 monclient(hunting): get_auth_request con 0x7f3d3c071a50 auth_method 0 2024-03-27T21:39:52.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3a59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3ad9d640 10 monclient(hunting): get_auth_request con 0x7f3d3c141630 auth_method 0 2024-03-27T21:39:52.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:52.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:52.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:52.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:52.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:52.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:52.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d31d9b640 1 --2- 172.21.15.60:0/405715916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d3c13e810 0x7f3d3c1d9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d31d9b640 10 monclient(hunting): get_auth_request con 0x7f3d3c13e810 auth_method 0 2024-03-27T21:39:52.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d31d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d31d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3a59c640 10 monclient(hunting): handle_auth_done global_id 4376 payload 824 2024-03-27T21:39:52.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3a59c640 1 -- 172.21.15.60:0/405715916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d3c13e810 msgr2=0x7f3d3c1d9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3a59c640 1 --2- 172.21.15.60:0/405715916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d3c13e810 0x7f3d3c1d9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3a59c640 1 -- 172.21.15.60:0/405715916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d3c141630 msgr2=0x7f3d3c0ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3a59c640 1 --2- 172.21.15.60:0/405715916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d3c141630 0x7f3d3c0ae8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3a59c640 10 monclient: _finish_hunting 0 2024-03-27T21:39:52.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3a59c640 1 monclient: found mon.noname-c 2024-03-27T21:39:52.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:52.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:39:52.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3a59c640 1 -- 172.21.15.60:0/405715916 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d34009b30 con 0x7f3d3c071a50 2024-03-27T21:39:52.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3ad9d640 1 --2- 172.21.15.60:0/405715916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d3c141630 0x7f3d3c0ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:52.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d31d9b640 1 --2- 172.21.15.60:0/405715916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d3c13e810 0x7f3d3c1d9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:39:52.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3a59c640 1 --2- 172.21.15.60:0/405715916 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3d3c071a50 0x7f3d3c138220 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00d5f0 tx=0x7f3d2c00dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:52.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d337fe640 10 client.?.objecter ms_handle_connect 0x7f3d3c071a50 2024-03-27T21:39:52.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d337fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:52.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d337fe640 1 -- 172.21.15.60:0/405715916 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c0125f0 con 0x7f3d3c071a50 2024-03-27T21:39:52.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:52.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:52.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d337fe640 10 monclient: dump: 2024-03-27T21:39:52.033 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:52.033 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.033 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.033 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.033 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:52.033 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:52.033 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:52.033 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:52.033 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:52.033 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d337fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:52.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d337fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:52.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:52.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d40c5c640 5 monclient: authenticate success, global_id 4376 2024-03-27T21:39:52.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d337fe640 1 -- 172.21.15.60:0/405715916 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2c012750 con 0x7f3d3c071a50 2024-03-27T21:39:52.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:52.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d40c5c640 10 monclient: _renew_subs 2024-03-27T21:39:52.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d337fe640 1 -- 172.21.15.60:0/405715916 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c012b00 con 0x7f3d3c071a50 2024-03-27T21:39:52.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d40c5c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:52.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d40c5c640 1 -- 172.21.15.60:0/405715916 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d24009400 con 0x7f3d3c071a50 2024-03-27T21:39:52.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:52.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:52.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d337fe640 10 monclient: dump: 2024-03-27T21:39:52.034 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:52.035 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.035 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.035 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.035 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:52.035 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:52.035 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:52.035 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:52.035 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:52.035 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d3959a640 4 set_mon_vals no callback set 2024-03-27T21:39:52.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d40c5c640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:52.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d40c5c640 10 monclient: _renew_subs 2024-03-27T21:39:52.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d40c5c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:52.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d40c5c640 1 -- 172.21.15.60:0/405715916 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d3c0aefe0 con 0x7f3d3c071a50 2024-03-27T21:39:52.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.029+0000 7f3d40c5c640 1 librados: init done 2024-03-27T21:39:52.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.030+0000 7f3d40c5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:52.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.030+0000 7f3d40c5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:52.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.030+0000 7f3d40c5c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:52.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.030+0000 7f3d40c5c640 1 -- 172.21.15.60:0/405715916 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d3c13cb60 con 0x7f3d3c071a50 2024-03-27T21:39:52.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.031+0000 7f3d337fe640 1 -- 172.21.15.60:0/405715916 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3d2c012c60 con 0x7f3d3c071a50 2024-03-27T21:39:52.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.031+0000 7f3d337fe640 1 --2- 172.21.15.60:0/405715916 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3d1c06c590 0x7f3d1c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.031+0000 7f3d337fe640 1 -- 172.21.15.60:0/405715916 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3d2c089520 con 0x7f3d3c071a50 2024-03-27T21:39:52.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.031+0000 7f3d337fe640 10 client.4376.objecter ms_dispatch 0x7f3d3c000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:52.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.031+0000 7f3d31d9b640 1 --2- 172.21.15.60:0/405715916 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3d1c06c590 0x7f3d1c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.031+0000 7f3d337fe640 3 client.4376.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:52.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.031+0000 7f3d337fe640 3 client.4376.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:52.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.031+0000 7f3d31d9b640 10 monclient: get_auth_request con 0x7f3d1c06c590 auth_method 0 2024-03-27T21:39:52.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.031+0000 7f3d337fe640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-27T21:39:52.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.031+0000 7f3d337fe640 10 client.4376.objecter ms_handle_connect 0x7f3d1c06c590 2024-03-27T21:39:52.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.031+0000 7f3d31d9b640 1 --2- 172.21.15.60:0/405715916 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3d1c06c590 0x7f3d1c06ea50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3d24036000 tx=0x7f3d2403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:52.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.035+0000 7f3d337fe640 1 -- 172.21.15.60:0/405715916 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3d2c054980 con 0x7f3d3c071a50 2024-03-27T21:39:52.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.035+0000 7f3d337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:52.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.035+0000 7f3d337fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:52.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.212+0000 7f3d40c5c640 1 -- 172.21.15.60:0/405715916 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3d3c0aa590 con 0x7f3d1c06c590 2024-03-27T21:39:52.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.214+0000 7f3d337fe640 1 -- 172.21.15.60:0/405715916 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48297 (secure 0 0 0) 0x7f3d24031b30 con 0x7f3d1c06c590 2024-03-27T21:39:52.201 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:39:52.201 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-03-27T21:39:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.216+0000 7f3d40c5c640 10 librados: watch_flush enter 2024-03-27T21:39:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.216+0000 7f3d40c5c640 10 librados: watch_flush exit 2024-03-27T21:39:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.216+0000 7f3d40c5c640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-27T21:39:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.216+0000 7f3d40c5c640 10 client.4376.objecter successfully canceled tick 2024-03-27T21:39:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.216+0000 7f3d40c5c640 1 -- 172.21.15.60:0/405715916 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3d1c06c590 msgr2=0x7f3d1c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.216+0000 7f3d40c5c640 1 --2- 172.21.15.60:0/405715916 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3d1c06c590 0x7f3d1c06ea50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3d24036000 tx=0x7f3d2403d040 comp rx=0 tx=0).stop 2024-03-27T21:39:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.216+0000 7f3d40c5c640 10 monclient: shutdown 2024-03-27T21:39:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.216+0000 7f3d40c5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.216+0000 7f3d40c5c640 1 -- 172.21.15.60:0/405715916 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3d3c071a50 msgr2=0x7f3d3c138220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.216+0000 7f3d40c5c640 1 --2- 172.21.15.60:0/405715916 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3d3c071a50 0x7f3d3c138220 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00d5f0 tx=0x7f3d2c00dab0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.216+0000 7f3d40c5c640 1 -- 172.21.15.60:0/405715916 shutdown_connections 2024-03-27T21:39:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.216+0000 7f3d40c5c640 1 --2- 172.21.15.60:0/405715916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d3c141630 0x7f3d3c0ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.216+0000 7f3d40c5c640 1 --2- 172.21.15.60:0/405715916 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3d1c06c590 0x7f3d1c06ea50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.216+0000 7f3d40c5c640 1 --2- 172.21.15.60:0/405715916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d3c13e810 0x7f3d3c1d9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.216+0000 7f3d40c5c640 1 --2- 172.21.15.60:0/405715916 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3d3c071a50 0x7f3d3c138220 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.216+0000 7f3d40c5c640 1 -- 172.21.15.60:0/405715916 >> 172.21.15.60:0/405715916 conn(0x7f3d3c0b7300 msgr2=0x7f3d3c0a8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.216+0000 7f3d40c5c640 1 -- 172.21.15.60:0/405715916 shutdown_connections 2024-03-27T21:39:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.216+0000 7f3d40c5c640 1 -- 172.21.15.60:0/405715916 wait complete. 2024-03-27T21:39:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.216+0000 7f3d40c5c640 1 librados: shutdown 2024-03-27T21:39:52.213 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-27T21:39:51.391200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":221912,"kb_used_data":1632,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":754752808,"statfs":{"total":773094113280,"available":772866875392,"internally_reserved":0,"allocated":1671168,"data_stored":1579288,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":12742,"internal_metadata":225431098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001315"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:46.474962+0000","last_change":"2024-03-27T21:39:46.470433+0000","last_active":"2024-03-27T21:39:46.474962+0000","last_peered":"2024-03-27T21:39:46.474962+0000","last_clean":"2024-03-27T21:39:46.474962+0000","last_became_active":"2024-03-27T21:39:46.470153+0000","last_became_peered":"2024-03-27T21:39:46.470153+0000","last_unstale":"2024-03-27T21:39:46.474962+0000","last_undegraded":"2024-03-27T21:39:46.474962+0000","last_fullsized":"2024-03-27T21:39:46.474962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T21:39:45.811197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:46.468505+0000","last_change":"2024-03-27T21:39:46.467893+0000","last_active":"2024-03-27T21:39:46.468505+0000","last_peered":"2024-03-27T21:39:46.468505+0000","last_clean":"2024-03-27T21:39:46.468505+0000","last_became_active":"2024-03-27T21:39:46.467576+0000","last_became_peered":"2024-03-27T21:39:46.467576+0000","last_unstale":"2024-03-27T21:39:46.468505+0000","last_undegraded":"2024-03-27T21:39:46.468505+0000","last_fullsized":"2024-03-27T21:39:46.468505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:29:50.190006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:47.405006+0000","last_change":"2024-03-27T21:39:46.679788+0000","last_active":"2024-03-27T21:39:47.405006+0000","last_peered":"2024-03-27T21:39:47.405006+0000","last_clean":"2024-03-27T21:39:47.405006+0000","last_became_active":"2024-03-27T21:39:46.679561+0000","last_became_peered":"2024-03-27T21:39:46.679561+0000","last_unstale":"2024-03-27T21:39:47.405006+0000","last_undegraded":"2024-03-27T21:39:47.405006+0000","last_fullsized":"2024-03-27T21:39:47.405006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T23:53:50.373923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:46.679751+0000","last_change":"2024-03-27T21:39:46.679751+0000","last_active":"2024-03-27T21:39:46.679751+0000","last_peered":"2024-03-27T21:39:46.679751+0000","last_clean":"2024-03-27T21:39:46.679751+0000","last_became_active":"2024-03-27T21:39:46.679509+0000","last_became_peered":"2024-03-27T21:39:46.679509+0000","last_unstale":"2024-03-27T21:39:46.679751+0000","last_undegraded":"2024-03-27T21:39:46.679751+0000","last_fullsized":"2024-03-27T21:39:46.679751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:43:02.145158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T21:39:47.458020+0000","last_change":"2024-03-27T21:39:46.459503+0000","last_active":"2024-03-27T21:39:47.458020+0000","last_peered":"2024-03-27T21:39:47.458020+0000","last_clean":"2024-03-27T21:39:47.458020+0000","last_became_active":"2024-03-27T21:39:46.459225+0000","last_became_peered":"2024-03-27T21:39:46.459225+0000","last_unstale":"2024-03-27T21:39:47.458020+0000","last_undegraded":"2024-03-27T21:39:47.458020+0000","last_fullsized":"2024-03-27T21:39:47.458020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:17:43.512127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:46.680826+0000","last_change":"2024-03-27T21:39:46.679791+0000","last_active":"2024-03-27T21:39:46.680826+0000","last_peered":"2024-03-27T21:39:46.680826+0000","last_clean":"2024-03-27T21:39:46.680826+0000","last_became_active":"2024-03-27T21:39:46.679503+0000","last_became_peered":"2024-03-27T21:39:46.679503+0000","last_unstale":"2024-03-27T21:39:46.680826+0000","last_undegraded":"2024-03-27T21:39:46.680826+0000","last_fullsized":"2024-03-27T21:39:46.680826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:07:25.151346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:47.404983+0000","last_change":"2024-03-27T21:39:46.458326+0000","last_active":"2024-03-27T21:39:47.404983+0000","last_peered":"2024-03-27T21:39:47.404983+0000","last_clean":"2024-03-27T21:39:47.404983+0000","last_became_active":"2024-03-27T21:39:46.458074+0000","last_became_peered":"2024-03-27T21:39:46.458074+0000","last_unstale":"2024-03-27T21:39:47.404983+0000","last_undegraded":"2024-03-27T21:39:47.404983+0000","last_fullsized":"2024-03-27T21:39:47.404983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:06:29.798229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-27T21:39:47.458062+0000","last_change":"2024-03-27T21:39:46.459166+0000","last_active":"2024-03-27T21:39:47.458062+0000","last_peered":"2024-03-27T21:39:47.458062+0000","last_clean":"2024-03-27T21:39:47.458062+0000","last_became_active":"2024-03-27T21:39:46.458958+0000","last_became_peered":"2024-03-27T21:39:46.458958+0000","last_unstale":"2024-03-27T21:39:47.458062+0000","last_undegraded":"2024-03-27T21:39:47.458062+0000","last_fullsized":"2024-03-27T21:39:47.458062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T22:18:37.511546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":163,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-27T21:39:46.455960+0000","last_change":"2024-03-27T21:39:39.866806+0000","last_active":"2024-03-27T21:39:46.455960+0000","last_peered":"2024-03-27T21:39:46.455960+0000","last_clean":"2024-03-27T21:39:46.455960+0000","last_became_active":"2024-03-27T21:39:39.865948+0000","last_became_peered":"2024-03-27T21:39:39.865948+0000","last_unstale":"2024-03-27T21:39:46.455960+0000","last_undegraded":"2024-03-27T21:39:46.455960+0000","last_fullsized":"2024-03-27T21:39:46.455960+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_clean_scrub_stamp":"2024-03-27T21:39:38.411432+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:36:49.375869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":16,"seq":68719476743,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]}]},{"osd":3,"up_from":16,"seq":68719476742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31848905},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5620000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90700000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4790000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2849999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]}]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":598418,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]}]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":63742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]}]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":592670,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T21:39:52.214 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-27T21:39:52.215 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-03-27T21:39:52.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.327+0000 7fe409266640 10 monclient: get_monmap_and_config 2024-03-27T21:39:52.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.327+0000 7fe409266640 10 monclient: build_initial_monmap 2024-03-27T21:39:52.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.327+0000 7fe409266640 10 monclient: monmap: 2024-03-27T21:39:52.314 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:52.314 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.314 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:52.328873+0000 2024-03-27T21:39:52.315 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:52.328873+0000 2024-03-27T21:39:52.315 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:52.315 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:52.315 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:52.315 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:52.315 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:52.315 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:52.315 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:52.315 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.328+0000 7fe409266640 1 Processor -- start 2024-03-27T21:39:52.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.328+0000 7fe409266640 1 -- start start 2024-03-27T21:39:52.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.328+0000 7fe409266640 10 monclient: init 2024-03-27T21:39:52.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe409266640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:52.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe409266640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-03-27T21:39:52.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe409266640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe404071a50 0x7fe4040a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe409266640 10 monclient(hunting): picked mon.noname-c con 0x7fe404071a50 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:52.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe409266640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe40413e810 0x7fe40413ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe409266640 10 monclient(hunting): picked mon.noname-a con 0x7fe40413e810 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:52.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe409266640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe404141630 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:52.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe409266640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe40413f190 con 0x7fe404141630 2024-03-27T21:39:52.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe409266640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe409266640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe404071e30 con 0x7fe40413e810 2024-03-27T21:39:52.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe409266640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe409266640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4040a8e90 con 0x7fe404071a50 2024-03-27T21:39:52.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe409266640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:52.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe409266640 10 monclient(hunting): authenticate will time out at 833.550145s 2024-03-27T21:39:52.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe3fbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe40413e810 0x7fe40413ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe3fbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe40413e810 0x7fe40413ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44350/0 (socket says 172.21.15.60:44350) 2024-03-27T21:39:52.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe3fbfff640 1 -- 172.21.15.60:0/2619123431 learned_addr learned my addr 172.21.15.60:0/2619123431 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:52.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe3fbfff640 10 monclient(hunting): get_auth_request con 0x7fe40413e810 auth_method 0 2024-03-27T21:39:52.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe4037fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe404141630 0x7fe404141aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55822/0 (socket says 172.21.15.60:55822) 2024-03-27T21:39:52.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe3fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe3fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe3fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe3fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:52.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe3fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:52.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe3fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:52.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe402ffd640 1 --2- 172.21.15.60:0/2619123431 >> v2:172.21.15.191:3300/0 conn(0x7fe404071a50 0x7fe4040a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe402ffd640 10 monclient(hunting): get_auth_request con 0x7fe404071a50 auth_method 0 2024-03-27T21:39:52.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe402ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe402ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe402ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe402ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:52.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe402ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:52.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe402ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:52.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe3fbfff640 10 monclient(hunting): handle_auth_done global_id 4440 payload 274 2024-03-27T21:39:52.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe3fbfff640 1 -- 172.21.15.60:0/2619123431 >> v2:172.21.15.191:3300/0 conn(0x7fe404071a50 msgr2=0x7fe4040a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:52.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe3fbfff640 1 --2- 172.21.15.60:0/2619123431 >> v2:172.21.15.191:3300/0 conn(0x7fe404071a50 0x7fe4040a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe3fbfff640 1 -- 172.21.15.60:0/2619123431 >> v1:172.21.15.60:6789/0 conn(0x7fe404141630 legacy=0x7fe404141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe3fbfff640 10 monclient: _finish_hunting 0 2024-03-27T21:39:52.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe3fbfff640 1 monclient: found mon.noname-a 2024-03-27T21:39:52.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe3fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:52.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe3fbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:52.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe3fbfff640 1 -- 172.21.15.60:0/2619123431 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4040aa040 con 0x7fe40413e810 2024-03-27T21:39:52.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe4027fc640 1 -- 172.21.15.60:0/2619123431 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2731969573 0 0) 0x7fe3fc001490 con 0x7fe404141630 2024-03-27T21:39:52.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.329+0000 7fe4027fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:39:52.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.330+0000 7fe3fbfff640 1 --2- 172.21.15.60:0/2619123431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe40413e810 0x7fe40413ec50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe4040b4a80 tx=0x7fe3ec02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4843c06f8f1354f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:52.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.330+0000 7fe4027fc640 1 -- 172.21.15.60:0/2619123431 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec030860 con 0x7fe40413e810 2024-03-27T21:39:52.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.330+0000 7fe4027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:52.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.330+0000 7fe4027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:52.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.330+0000 7fe4027fc640 10 monclient: dump: 2024-03-27T21:39:52.329 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:52.329 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.330 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.330 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.330 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:52.330 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:52.330 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:52.330 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:52.330 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:52.330 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.330+0000 7fe4027fc640 10 monclient: _finish_auth 0 2024-03-27T21:39:52.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.330+0000 7fe4027fc640 10 monclient: _check_auth_tickets 2024-03-27T21:39:52.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.330+0000 7fe4027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:52.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.330+0000 7fe4027fc640 1 -- 172.21.15.60:0/2619123431 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3ec0309c0 con 0x7fe40413e810 2024-03-27T21:39:52.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.330+0000 7fe409266640 5 monclient: authenticate success, global_id 4440 2024-03-27T21:39:52.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.330+0000 7fe409266640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:52.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.330+0000 7fe4027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:52.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.330+0000 7fe4027fc640 1 -- 172.21.15.60:0/2619123431 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec030d50 con 0x7fe40413e810 2024-03-27T21:39:52.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.330+0000 7fe4027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:52.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.330+0000 7fe4027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:52.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.330+0000 7fe4027fc640 10 monclient: dump: 2024-03-27T21:39:52.331 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:52.331 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.331 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.331 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.331 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:52.331 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:52.331 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:52.332 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:52.332 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:52.332 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.330+0000 7fe409266640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:52.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.330+0000 7fe409266640 4 set_mon_vals no callback set 2024-03-27T21:39:52.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.331+0000 7fe409266640 10 monclient: shutdown 2024-03-27T21:39:52.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.331+0000 7fe409266640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:52.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.331+0000 7fe409266640 1 -- 172.21.15.60:0/2619123431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe40413e810 msgr2=0x7fe40413ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.331+0000 7fe409266640 1 --2- 172.21.15.60:0/2619123431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe40413e810 0x7fe40413ec50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe4040b4a80 tx=0x7fe3ec02b9a0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.331+0000 7fe409266640 1 -- 172.21.15.60:0/2619123431 shutdown_connections 2024-03-27T21:39:52.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.331+0000 7fe409266640 1 --2- 172.21.15.60:0/2619123431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe40413e810 0x7fe40413ec50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.331+0000 7fe409266640 1 --2- 172.21.15.60:0/2619123431 >> v2:172.21.15.191:3300/0 conn(0x7fe404071a50 0x7fe4040a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.331+0000 7fe409266640 1 -- 172.21.15.60:0/2619123431 >> 172.21.15.60:0/2619123431 conn(0x7fe4040b7300 msgr2=0x7fe4040b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:52.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.331+0000 7fe409266640 1 -- 172.21.15.60:0/2619123431 shutdown_connections 2024-03-27T21:39:52.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.331+0000 7fe409266640 1 -- 172.21.15.60:0/2619123431 wait complete. 2024-03-27T21:39:52.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.331+0000 7fe409266640 10 monclient: build_initial_monmap 2024-03-27T21:39:52.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.331+0000 7fe409266640 10 monclient: monmap: 2024-03-27T21:39:52.333 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:52.333 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:52.333 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:52.333 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:52.333 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:52.333 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:52.333 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:52.333 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:52.333 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:52.333 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 1 librados: starting msgr at 2024-03-27T21:39:52.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 1 librados: starting objecter 2024-03-27T21:39:52.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 1 Processor -- start 2024-03-27T21:39:52.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 1 -- start start 2024-03-27T21:39:52.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 1 librados: setting wanted keys 2024-03-27T21:39:52.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 1 librados: calling monclient init 2024-03-27T21:39:52.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 10 monclient: init 2024-03-27T21:39:52.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:52.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:39:52.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe404071a50 0x7fe40413aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 10 monclient(hunting): picked mon.noname-b con 0x7fe404071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:52.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe40413e810 0x7fe40413af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 10 monclient(hunting): picked mon.noname-a con 0x7fe40413e810 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:52.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe404141630 0x7fe4041d6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 10 monclient(hunting): picked mon.noname-c con 0x7fe404141630 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:52.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40413a5a0 con 0x7fe40413e810 2024-03-27T21:39:52.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40413a270 con 0x7fe404071a50 2024-03-27T21:39:52.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe404137690 con 0x7fe404141630 2024-03-27T21:39:52.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:52.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe409266640 10 monclient(hunting): authenticate will time out at 833.553713s 2024-03-27T21:39:52.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe3fbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe40413e810 0x7fe40413af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe3fbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe40413e810 0x7fe40413af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44366/0 (socket says 172.21.15.60:44366) 2024-03-27T21:39:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe3fbfff640 1 -- 172.21.15.60:0/2527696702 learned_addr learned my addr 172.21.15.60:0/2527696702 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe3fbfff640 10 monclient(hunting): get_auth_request con 0x7fe40413e810 auth_method 0 2024-03-27T21:39:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe3fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe3fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe3fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe4037fe640 1 --2- 172.21.15.60:0/2527696702 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe404141630 0x7fe4041d6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe4037fe640 10 monclient(hunting): get_auth_request con 0x7fe404141630 auth_method 0 2024-03-27T21:39:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe4037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe4037fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe4037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe3fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe3fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.332+0000 7fe3fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe4037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe4037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe4037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe402ffd640 1 --2- 172.21.15.60:0/2527696702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe404071a50 0x7fe40413aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe402ffd640 10 monclient(hunting): get_auth_request con 0x7fe404071a50 auth_method 0 2024-03-27T21:39:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe402ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe402ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe402ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fbfff640 10 monclient(hunting): handle_auth_done global_id 4446 payload 824 2024-03-27T21:39:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fbfff640 1 -- 172.21.15.60:0/2527696702 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe404141630 msgr2=0x7fe4041d6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fbfff640 1 --2- 172.21.15.60:0/2527696702 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe404141630 0x7fe4041d6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fbfff640 1 -- 172.21.15.60:0/2527696702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe404071a50 msgr2=0x7fe40413aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fbfff640 1 --2- 172.21.15.60:0/2527696702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe404071a50 0x7fe40413aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fbfff640 10 monclient: _finish_hunting 0 2024-03-27T21:39:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fbfff640 1 monclient: found mon.noname-a 2024-03-27T21:39:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fbfff640 1 -- 172.21.15.60:0/2527696702 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ec0094d0 con 0x7fe40413e810 2024-03-27T21:39:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fbfff640 1 --2- 172.21.15.60:0/2527696702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe40413e810 0x7fe40413af70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec02b9e0 tx=0x7fe3ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fb7fe640 10 client.?.objecter ms_handle_connect 0x7fe40413e810 2024-03-27T21:39:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fb7fe640 1 -- 172.21.15.60:0/2527696702 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec039c90 con 0x7fe40413e810 2024-03-27T21:39:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fb7fe640 10 monclient: dump: 2024-03-27T21:39:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:52.339 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fb7fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fb7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fb7fe640 1 -- 172.21.15.60:0/2527696702 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3ec039e10 con 0x7fe40413e810 2024-03-27T21:39:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe409266640 5 monclient: authenticate success, global_id 4446 2024-03-27T21:39:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe409266640 10 monclient: _renew_subs 2024-03-27T21:39:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fb7fe640 1 -- 172.21.15.60:0/2527696702 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec043640 con 0x7fe40413e810 2024-03-27T21:39:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe409266640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe409266640 1 -- 172.21.15.60:0/2527696702 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4041d6cd0 con 0x7fe40413e810 2024-03-27T21:39:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe4017fa640 4 set_mon_vals no callback set 2024-03-27T21:39:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe3fb7fe640 10 monclient: dump: 2024-03-27T21:39:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:52.340 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe409266640 10 client.4446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe409266640 10 monclient: _renew_subs 2024-03-27T21:39:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe409266640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe409266640 1 -- 172.21.15.60:0/2527696702 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4041d7180 con 0x7fe40413e810 2024-03-27T21:39:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.333+0000 7fe409266640 1 librados: init done 2024-03-27T21:39:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.334+0000 7fe409266640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.334+0000 7fe409266640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.334+0000 7fe409266640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.334+0000 7fe409266640 1 -- 172.21.15.60:0/2527696702 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4041d7430 con 0x7fe40413e810 2024-03-27T21:39:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.335+0000 7fe3fb7fe640 1 -- 172.21.15.60:0/2527696702 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe3ec0437a0 con 0x7fe40413e810 2024-03-27T21:39:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.335+0000 7fe3fb7fe640 1 --2- 172.21.15.60:0/2527696702 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe3e406c590 0x7fe3e406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.335+0000 7fe3fb7fe640 1 -- 172.21.15.60:0/2527696702 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe3ec03e070 con 0x7fe40413e810 2024-03-27T21:39:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.335+0000 7fe3fb7fe640 10 client.4446.objecter ms_dispatch 0x7fe404000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.335+0000 7fe3fb7fe640 3 client.4446.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.335+0000 7fe3fb7fe640 3 client.4446.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.335+0000 7fe402ffd640 1 --2- 172.21.15.60:0/2527696702 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe3e406c590 0x7fe3e406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.335+0000 7fe402ffd640 10 monclient: get_auth_request con 0x7fe3e406c590 auth_method 0 2024-03-27T21:39:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.335+0000 7fe3fb7fe640 20 client.4446.objecter dump_active .. 0 homeless 2024-03-27T21:39:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.335+0000 7fe3fb7fe640 10 client.4446.objecter ms_handle_connect 0x7fe3e406c590 2024-03-27T21:39:52.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.335+0000 7fe402ffd640 1 --2- 172.21.15.60:0/2527696702 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe3e406c590 0x7fe3e406ea50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009ad0 tx=0x7fe3f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:52.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.339+0000 7fe3fb7fe640 1 -- 172.21.15.60:0/2527696702 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe3ec0776a0 con 0x7fe40413e810 2024-03-27T21:39:52.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.339+0000 7fe3fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:52.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.339+0000 7fe3fb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:52.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.519+0000 7fe409266640 10 start_mon_command cmd=[{"prefix": "osd dump"}] 2024-03-27T21:39:52.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.519+0000 7fe409266640 10 monclient: _send_command 2 [{"prefix": "osd dump"}] 2024-03-27T21:39:52.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.519+0000 7fe409266640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:52.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.519+0000 7fe409266640 1 -- 172.21.15.60:0/2527696702 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7fe404137850 con 0x7fe40413e810 2024-03-27T21:39:52.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.519+0000 7fe3fb7fe640 1 -- 172.21.15.60:0/2527696702 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v21) v1 ==== 56+0+2766 (secure 0 0 0) 0x7fe3ec034ae0 con 0x7fe40413e810 2024-03-27T21:39:52.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.519+0000 7fe3fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump"}] 2024-03-27T21:39:52.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.519+0000 7fe3fb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:52.506 INFO:teuthology.orchestra.run.smithi060.stdout:epoch 21 2024-03-27T21:39:52.506 INFO:teuthology.orchestra.run.smithi060.stdout:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.506 INFO:teuthology.orchestra.run.smithi060.stdout:created 2024-03-27T21:39:27.092191+0000 2024-03-27T21:39:52.506 INFO:teuthology.orchestra.run.smithi060.stdout:modified 2024-03-27T21:39:47.452288+0000 2024-03-27T21:39:52.507 INFO:teuthology.orchestra.run.smithi060.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-27T21:39:52.507 INFO:teuthology.orchestra.run.smithi060.stdout:crush_version 5 2024-03-27T21:39:52.507 INFO:teuthology.orchestra.run.smithi060.stdout:full_ratio 0.95 2024-03-27T21:39:52.507 INFO:teuthology.orchestra.run.smithi060.stdout:backfillfull_ratio 0.9 2024-03-27T21:39:52.507 INFO:teuthology.orchestra.run.smithi060.stdout:nearfull_ratio 0.85 2024-03-27T21:39:52.507 INFO:teuthology.orchestra.run.smithi060.stdout:require_min_compat_client luminous 2024-03-27T21:39:52.507 INFO:teuthology.orchestra.run.smithi060.stdout:min_compat_client jewel 2024-03-27T21:39:52.507 INFO:teuthology.orchestra.run.smithi060.stdout:require_osd_release reef 2024-03-27T21:39:52.507 INFO:teuthology.orchestra.run.smithi060.stdout:stretch_mode_enabled false 2024-03-27T21:39:52.507 INFO:teuthology.orchestra.run.smithi060.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 18 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 8.00 2024-03-27T21:39:52.507 INFO:teuthology.orchestra.run.smithi060.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 application rbd read_balance_score 1.99 2024-03-27T21:39:52.507 INFO:teuthology.orchestra.run.smithi060.stdout:max_osd 8 2024-03-27T21:39:52.507 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] [v2:172.21.15.60:6818/34603,v1:172.21.15.60:6819/34603] exists,up 778aaa64-6720-495b-bcdc-86fbcece9d58 2024-03-27T21:39:52.507 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] [v2:172.21.15.60:6802/34602,v1:172.21.15.60:6803/34602] exists,up d960adf1-3d83-4e24-a612-8df53fa43d0c 2024-03-27T21:39:52.507 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] [v2:172.21.15.60:6826/34604,v1:172.21.15.60:6827/34604] exists,up b8ffc48b-cf6f-4701-a6aa-c25fdb0f7977 2024-03-27T21:39:52.508 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] [v2:172.21.15.60:6810/34613,v1:172.21.15.60:6811/34613] exists,up 0f14f2d8-b271-48e4-a198-aa5279a01c06 2024-03-27T21:39:52.508 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] [v2:172.21.15.191:6802/34332,v1:172.21.15.191:6803/34332] exists,up 8cdeb7ae-1c6e-4d0d-ba44-80248eff2fb6 2024-03-27T21:39:52.508 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 up in weight 1 up_from 16 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] [v2:172.21.15.191:6813/34333,v1:172.21.15.191:6816/34333] exists,up 93748cf8-d3fe-46b8-950f-e01e28a5e02c 2024-03-27T21:39:52.508 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] [v2:172.21.15.191:6826/34361,v1:172.21.15.191:6827/34361] exists,up 1ad21434-cbc5-4de9-8d63-76f894f7e736 2024-03-27T21:39:52.508 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] [v2:172.21.15.191:6812/34334,v1:172.21.15.191:6814/34334] exists,up 9fbd0927-ee2f-471b-8df8-c4e0fc40b9b0 2024-03-27T21:39:52.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.521+0000 7fe409266640 10 librados: watch_flush enter 2024-03-27T21:39:52.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.521+0000 7fe409266640 10 librados: watch_flush exit 2024-03-27T21:39:52.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.521+0000 7fe409266640 20 client.4446.objecter shutdown clearing up homeless session... 2024-03-27T21:39:52.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.521+0000 7fe409266640 10 client.4446.objecter successfully canceled tick 2024-03-27T21:39:52.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.521+0000 7fe409266640 1 -- 172.21.15.60:0/2527696702 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe3e406c590 msgr2=0x7fe3e406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.521+0000 7fe409266640 1 --2- 172.21.15.60:0/2527696702 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe3e406c590 0x7fe3e406ea50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009ad0 tx=0x7fe3f4008040 comp rx=0 tx=0).stop 2024-03-27T21:39:52.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.521+0000 7fe409266640 10 monclient: shutdown 2024-03-27T21:39:52.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.521+0000 7fe409266640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:52.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.521+0000 7fe409266640 1 -- 172.21.15.60:0/2527696702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe40413e810 msgr2=0x7fe40413af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.521+0000 7fe409266640 1 --2- 172.21.15.60:0/2527696702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe40413e810 0x7fe40413af70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec02b9e0 tx=0x7fe3ec004730 comp rx=0 tx=0).stop 2024-03-27T21:39:52.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.521+0000 7fe409266640 1 -- 172.21.15.60:0/2527696702 shutdown_connections 2024-03-27T21:39:52.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.521+0000 7fe409266640 1 --2- 172.21.15.60:0/2527696702 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe404141630 0x7fe4041d6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.521+0000 7fe409266640 1 --2- 172.21.15.60:0/2527696702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe40413e810 0x7fe40413af70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.521+0000 7fe409266640 1 --2- 172.21.15.60:0/2527696702 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe3e406c590 0x7fe3e406ea50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.521+0000 7fe409266640 1 --2- 172.21.15.60:0/2527696702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe404071a50 0x7fe40413aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.521+0000 7fe409266640 1 -- 172.21.15.60:0/2527696702 >> 172.21.15.60:0/2527696702 conn(0x7fe4040b7300 msgr2=0x7fe404141aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:52.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.521+0000 7fe409266640 1 -- 172.21.15.60:0/2527696702 shutdown_connections 2024-03-27T21:39:52.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.521+0000 7fe409266640 1 -- 172.21.15.60:0/2527696702 wait complete. 2024-03-27T21:39:52.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.521+0000 7fe409266640 1 librados: shutdown 2024-03-27T21:39:52.518 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] [v2:172.21.15.60:6818/34603,v1:172.21.15.60:6819/34603] exists,up 778aaa64-6720-495b-bcdc-86fbcece9d58', 'osd.1 up in weight 1 up_from 13 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] [v2:172.21.15.60:6802/34602,v1:172.21.15.60:6803/34602] exists,up d960adf1-3d83-4e24-a612-8df53fa43d0c', 'osd.2 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] [v2:172.21.15.60:6826/34604,v1:172.21.15.60:6827/34604] exists,up b8ffc48b-cf6f-4701-a6aa-c25fdb0f7977', 'osd.3 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] [v2:172.21.15.60:6810/34613,v1:172.21.15.60:6811/34613] exists,up 0f14f2d8-b271-48e4-a198-aa5279a01c06', 'osd.4 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] [v2:172.21.15.191:6802/34332,v1:172.21.15.191:6803/34332] exists,up 8cdeb7ae-1c6e-4d0d-ba44-80248eff2fb6', 'osd.5 up in weight 1 up_from 16 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] [v2:172.21.15.191:6813/34333,v1:172.21.15.191:6816/34333] exists,up 93748cf8-d3fe-46b8-950f-e01e28a5e02c', 'osd.6 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] [v2:172.21.15.191:6826/34361,v1:172.21.15.191:6827/34361] exists,up 1ad21434-cbc5-4de9-8d63-76f894f7e736', 'osd.7 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] [v2:172.21.15.191:6812/34334,v1:172.21.15.191:6814/34334] exists,up 9fbd0927-ee2f-471b-8df8-c4e0fc40b9b0'] 2024-03-27T21:39:52.519 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-03-27T21:39:52.621 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:39:52.621 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "mon.a", 2024-03-27T21:39:52.621 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster": "ceph", 2024-03-27T21:39:52.621 INFO:teuthology.orchestra.run.smithi060.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-27T21:39:52.621 INFO:teuthology.orchestra.run.smithi060.stdout: "admin_socket_mode": "", 2024-03-27T21:39:52.622 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-03-27T21:39:52.622 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_client_required": "cephx, none", 2024-03-27T21:39:52.622 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_cluster_required": "cephx", 2024-03-27T21:39:52.622 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_debug": "true", 2024-03-27T21:39:52.622 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-27T21:39:52.622 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-27T21:39:52.622 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_service_required": "cephx", 2024-03-27T21:39:52.622 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-27T21:39:52.622 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_supported": "cephx", 2024-03-27T21:39:52.622 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_aio": "true", 2024-03-27T21:39:52.622 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-27T21:39:52.622 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_aio_poll_ms": "250", 2024-03-27T21:39:52.622 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_aio_reap_max": "16", 2024-03-27T21:39:52.622 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_async_discard": "false", 2024-03-27T21:39:52.623 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_block_size": "4096", 2024-03-27T21:39:52.623 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_debug_aio": "false", 2024-03-27T21:39:52.623 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-27T21:39:52.623 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-27T21:39:52.623 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_debug_inflight_ios": "false", 2024-03-27T21:39:52.623 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_enable_discard": "false", 2024-03-27T21:39:52.623 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_flock_retry": "3", 2024-03-27T21:39:52.623 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-27T21:39:52.623 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_inject_crash": "0", 2024-03-27T21:39:52.623 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-27T21:39:52.623 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_ioring": "false", 2024-03-27T21:39:52.623 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_ioring_hipri": "false", 2024-03-27T21:39:52.623 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-27T21:39:52.623 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-27T21:39:52.624 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-27T21:39:52.624 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-27T21:39:52.624 INFO:teuthology.orchestra.run.smithi060.stdout: "bdev_type": "", 2024-03-27T21:39:52.624 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_alloc_size": "1048576", 2024-03-27T21:39:52.624 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_allocator": "hybrid", 2024-03-27T21:39:52.624 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_buffered_io": "true", 2024-03-27T21:39:52.624 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_check_for_zeros": "false", 2024-03-27T21:39:52.624 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-27T21:39:52.624 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-27T21:39:52.624 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_compact_log_sync": "false", 2024-03-27T21:39:52.624 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-27T21:39:52.624 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-27T21:39:52.624 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-27T21:39:52.624 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-27T21:39:52.624 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_max_log_runway": "4194304", 2024-03-27T21:39:52.625 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_max_prefetch": "1048576", 2024-03-27T21:39:52.625 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_min_flush_size": "524288", 2024-03-27T21:39:52.625 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_min_log_runway": "1048576", 2024-03-27T21:39:52.625 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_replay_recovery": "false", 2024-03-27T21:39:52.625 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-27T21:39:52.625 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-27T21:39:52.625 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_sync_write": "false", 2024-03-27T21:39:52.625 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-27T21:39:52.625 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-27T21:39:52.625 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-27T21:39:52.625 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_allocation_from_file": "true", 2024-03-27T21:39:52.625 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_allocator": "hybrid", 2024-03-27T21:39:52.625 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-27T21:39:52.625 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-27T21:39:52.626 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-27T21:39:52.626 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-27T21:39:52.626 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-27T21:39:52.626 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-27T21:39:52.626 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-27T21:39:52.626 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_block_create": "true", 2024-03-27T21:39:52.626 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_block_db_create": "false", 2024-03-27T21:39:52.626 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_block_db_path": "", 2024-03-27T21:39:52.626 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_block_db_size": "0", 2024-03-27T21:39:52.626 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_block_path": "", 2024-03-27T21:39:52.626 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_block_preallocate_file": "false", 2024-03-27T21:39:52.626 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_block_size": "107374182400", 2024-03-27T21:39:52.626 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_block_wal_create": "false", 2024-03-27T21:39:52.626 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_block_wal_path": "", 2024-03-27T21:39:52.627 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_block_wal_size": "100663296", 2024-03-27T21:39:52.627 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_bluefs": "true", 2024-03-27T21:39:52.627 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-27T21:39:52.627 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-27T21:39:52.627 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-27T21:39:52.627 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-27T21:39:52.627 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-27T21:39:52.627 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-27T21:39:52.627 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-27T21:39:52.627 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-27T21:39:52.627 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_autotune": "true", 2024-03-27T21:39:52.627 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-27T21:39:52.628 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-27T21:39:52.628 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-27T21:39:52.628 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-27T21:39:52.628 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_size": "0", 2024-03-27T21:39:52.628 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-27T21:39:52.628 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-27T21:39:52.628 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-27T21:39:52.628 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-27T21:39:52.628 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_type": "2q", 2024-03-27T21:39:52.628 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-27T21:39:52.628 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_clone_cow": "true", 2024-03-27T21:39:52.628 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-27T21:39:52.628 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-27T21:39:52.628 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-27T21:39:52.629 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-27T21:39:52.629 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-27T21:39:52.629 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-27T21:39:52.629 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-27T21:39:52.629 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_compression_mode": "none", 2024-03-27T21:39:52.629 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-27T21:39:52.629 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_csum_type": "crc32c", 2024-03-27T21:39:52.629 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-27T21:39:52.629 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_debug_freelist": "false", 2024-03-27T21:39:52.629 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-27T21:39:52.629 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-27T21:39:52.629 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_debug_inject_bug21040": "false", 2024-03-27T21:39:52.629 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-27T21:39:52.629 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-27T21:39:52.629 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-27T21:39:52.630 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-27T21:39:52.630 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-27T21:39:52.630 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-27T21:39:52.630 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-27T21:39:52.630 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-27T21:39:52.630 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-27T21:39:52.630 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-27T21:39:52.630 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-27T21:39:52.630 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_debug_small_allocations": "0", 2024-03-27T21:39:52.630 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-27T21:39:52.630 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_default_buffered_read": "true", 2024-03-27T21:39:52.630 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_default_buffered_write": "false", 2024-03-27T21:39:52.630 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-27T21:39:52.631 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-27T21:39:52.631 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-27T21:39:52.631 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-27T21:39:52.631 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-27T21:39:52.631 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-27T21:39:52.631 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-27T21:39:52.631 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-27T21:39:52.631 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fail_eio": "false", 2024-03-27T21:39:52.631 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-27T21:39:52.631 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-27T21:39:52.631 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-27T21:39:52.631 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-27T21:39:52.631 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-27T21:39:52.631 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-27T21:39:52.631 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_on_mount": "false", 2024-03-27T21:39:52.632 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-27T21:39:52.632 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_on_umount": "false", 2024-03-27T21:39:52.632 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-27T21:39:52.632 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-27T21:39:52.632 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-27T21:39:52.632 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-27T21:39:52.632 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-27T21:39:52.632 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-27T21:39:52.632 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-27T21:39:52.632 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-27T21:39:52.632 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_ignore_data_csum": "false", 2024-03-27T21:39:52.632 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-27T21:39:52.632 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-27T21:39:52.632 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-27T21:39:52.633 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-27T21:39:52.633 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_log_op_age": "5.000000", 2024-03-27T21:39:52.633 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_max_alloc_size": "0", 2024-03-27T21:39:52.633 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_max_blob_size": "0", 2024-03-27T21:39:52.633 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-27T21:39:52.633 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-27T21:39:52.633 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-27T21:39:52.633 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_max_deferred_txc": "32", 2024-03-27T21:39:52.633 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_min_alloc_size": "0", 2024-03-27T21:39:52.633 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-27T21:39:52.633 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-27T21:39:52.633 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_nid_prealloc": "1024", 2024-03-27T21:39:52.633 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-27T21:39:52.633 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-27T21:39:52.633 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-27T21:39:52.634 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-27T21:39:52.634 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_retry_disk_reads": "3", 2024-03-27T21:39:52.634 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_rocksdb_cf": "true", 2024-03-27T21:39:52.634 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-27T21:39:52.634 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-27T21:39:52.634 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-27T21:39:52.634 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-27T21:39:52.634 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-27T21:39:52.634 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-27T21:39:52.634 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_spdk_mem": "512", 2024-03-27T21:39:52.635 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-27T21:39:52.635 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-27T21:39:52.635 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-27T21:39:52.635 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-27T21:39:52.635 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-27T21:39:52.635 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-27T21:39:52.635 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-27T21:39:52.635 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_tracing": "false", 2024-03-27T21:39:52.635 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-27T21:39:52.635 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-27T21:39:52.635 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-27T21:39:52.635 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-27T21:39:52.635 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-27T21:39:52.635 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-27T21:39:52.636 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-27T21:39:52.636 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-27T21:39:52.636 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-27T21:39:52.636 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_zero_block_detection": "false", 2024-03-27T21:39:52.636 INFO:teuthology.orchestra.run.smithi060.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-27T21:39:52.636 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-27T21:39:52.636 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-27T21:39:52.636 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-27T21:39:52.636 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-27T21:39:52.636 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-27T21:39:52.636 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-27T21:39:52.636 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-03-27T21:39:52.636 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-27T21:39:52.636 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-27T21:39:52.637 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-27T21:39:52.637 INFO:teuthology.orchestra.run.smithi060.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-27T21:39:52.637 INFO:teuthology.orchestra.run.smithi060.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-27T21:39:52.637 INFO:teuthology.orchestra.run.smithi060.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-27T21:39:52.637 INFO:teuthology.orchestra.run.smithi060.stdout: "cephx_cluster_require_signatures": "false", 2024-03-27T21:39:52.637 INFO:teuthology.orchestra.run.smithi060.stdout: "cephx_cluster_require_version": "2", 2024-03-27T21:39:52.637 INFO:teuthology.orchestra.run.smithi060.stdout: "cephx_require_signatures": "false", 2024-03-27T21:39:52.637 INFO:teuthology.orchestra.run.smithi060.stdout: "cephx_require_version": "2", 2024-03-27T21:39:52.637 INFO:teuthology.orchestra.run.smithi060.stdout: "cephx_service_require_signatures": "false", 2024-03-27T21:39:52.637 INFO:teuthology.orchestra.run.smithi060.stdout: "cephx_service_require_version": "2", 2024-03-27T21:39:52.637 INFO:teuthology.orchestra.run.smithi060.stdout: "cephx_sign_messages": "true", 2024-03-27T21:39:52.637 INFO:teuthology.orchestra.run.smithi060.stdout: "chdir": "", 2024-03-27T21:39:52.637 INFO:teuthology.orchestra.run.smithi060.stdout: "client_acl_type": "", 2024-03-27T21:39:52.637 INFO:teuthology.orchestra.run.smithi060.stdout: "client_asio_thread_count": "2", 2024-03-27T21:39:52.638 INFO:teuthology.orchestra.run.smithi060.stdout: "client_cache_mid": "0.750000", 2024-03-27T21:39:52.638 INFO:teuthology.orchestra.run.smithi060.stdout: "client_cache_size": "16384", 2024-03-27T21:39:52.638 INFO:teuthology.orchestra.run.smithi060.stdout: "client_caps_release_delay": "5", 2024-03-27T21:39:52.638 INFO:teuthology.orchestra.run.smithi060.stdout: "client_check_pool_perm": "true", 2024-03-27T21:39:52.638 INFO:teuthology.orchestra.run.smithi060.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-27T21:39:52.638 INFO:teuthology.orchestra.run.smithi060.stdout: "client_debug_force_sync_read": "false", 2024-03-27T21:39:52.638 INFO:teuthology.orchestra.run.smithi060.stdout: "client_debug_getattr_caps": "false", 2024-03-27T21:39:52.638 INFO:teuthology.orchestra.run.smithi060.stdout: "client_debug_inject_tick_delay": "0", 2024-03-27T21:39:52.638 INFO:teuthology.orchestra.run.smithi060.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-27T21:39:52.638 INFO:teuthology.orchestra.run.smithi060.stdout: "client_die_on_failed_remount": "false", 2024-03-27T21:39:52.638 INFO:teuthology.orchestra.run.smithi060.stdout: "client_dirsize_rbytes": "true", 2024-03-27T21:39:52.639 INFO:teuthology.orchestra.run.smithi060.stdout: "client_force_lazyio": "false", 2024-03-27T21:39:52.639 INFO:teuthology.orchestra.run.smithi060.stdout: "client_fs": "", 2024-03-27T21:39:52.639 INFO:teuthology.orchestra.run.smithi060.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-27T21:39:52.639 INFO:teuthology.orchestra.run.smithi060.stdout: "client_inject_release_failure": "false", 2024-03-27T21:39:52.639 INFO:teuthology.orchestra.run.smithi060.stdout: "client_max_inline_size": "4096", 2024-03-27T21:39:52.639 INFO:teuthology.orchestra.run.smithi060.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-27T21:39:52.639 INFO:teuthology.orchestra.run.smithi060.stdout: "client_mds_namespace": "", 2024-03-27T21:39:52.639 INFO:teuthology.orchestra.run.smithi060.stdout: "client_metadata": "", 2024-03-27T21:39:52.639 INFO:teuthology.orchestra.run.smithi060.stdout: "client_mount_gid": "-1", 2024-03-27T21:39:52.639 INFO:teuthology.orchestra.run.smithi060.stdout: "client_mount_timeout": "300", 2024-03-27T21:39:52.639 INFO:teuthology.orchestra.run.smithi060.stdout: "client_mount_uid": "-1", 2024-03-27T21:39:52.639 INFO:teuthology.orchestra.run.smithi060.stdout: "client_mountpoint": "/", 2024-03-27T21:39:52.639 INFO:teuthology.orchestra.run.smithi060.stdout: "client_notify_timeout": "10", 2024-03-27T21:39:52.639 INFO:teuthology.orchestra.run.smithi060.stdout: "client_oc": "true", 2024-03-27T21:39:52.639 INFO:teuthology.orchestra.run.smithi060.stdout: "client_oc_max_dirty": "104857600", 2024-03-27T21:39:52.640 INFO:teuthology.orchestra.run.smithi060.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-27T21:39:52.640 INFO:teuthology.orchestra.run.smithi060.stdout: "client_oc_max_objects": "1000", 2024-03-27T21:39:52.640 INFO:teuthology.orchestra.run.smithi060.stdout: "client_oc_size": "209715200", 2024-03-27T21:39:52.640 INFO:teuthology.orchestra.run.smithi060.stdout: "client_oc_target_dirty": "8388608", 2024-03-27T21:39:52.640 INFO:teuthology.orchestra.run.smithi060.stdout: "client_permissions": "true", 2024-03-27T21:39:52.640 INFO:teuthology.orchestra.run.smithi060.stdout: "client_quota": "true", 2024-03-27T21:39:52.641 INFO:teuthology.orchestra.run.smithi060.stdout: "client_quota_df": "true", 2024-03-27T21:39:52.641 INFO:teuthology.orchestra.run.smithi060.stdout: "client_readahead_max_bytes": "0", 2024-03-27T21:39:52.641 INFO:teuthology.orchestra.run.smithi060.stdout: "client_readahead_max_periods": "4", 2024-03-27T21:39:52.641 INFO:teuthology.orchestra.run.smithi060.stdout: "client_readahead_min": "131072", 2024-03-27T21:39:52.641 INFO:teuthology.orchestra.run.smithi060.stdout: "client_reconnect_stale": "false", 2024-03-27T21:39:52.641 INFO:teuthology.orchestra.run.smithi060.stdout: "client_shutdown_timeout": "30", 2024-03-27T21:39:52.641 INFO:teuthology.orchestra.run.smithi060.stdout: "client_snapdir": ".snap", 2024-03-27T21:39:52.642 INFO:teuthology.orchestra.run.smithi060.stdout: "client_tick_interval": "1", 2024-03-27T21:39:52.642 INFO:teuthology.orchestra.run.smithi060.stdout: "client_trace": "", 2024-03-27T21:39:52.642 INFO:teuthology.orchestra.run.smithi060.stdout: "client_try_dentry_invalidate": "false", 2024-03-27T21:39:52.642 INFO:teuthology.orchestra.run.smithi060.stdout: "client_use_faked_inos": "false", 2024-03-27T21:39:52.642 INFO:teuthology.orchestra.run.smithi060.stdout: "client_use_random_mds": "false", 2024-03-27T21:39:52.642 INFO:teuthology.orchestra.run.smithi060.stdout: "clog_to_graylog": "false", 2024-03-27T21:39:52.642 INFO:teuthology.orchestra.run.smithi060.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-27T21:39:52.642 INFO:teuthology.orchestra.run.smithi060.stdout: "clog_to_graylog_port": "12201", 2024-03-27T21:39:52.642 INFO:teuthology.orchestra.run.smithi060.stdout: "clog_to_monitors": "default=true", 2024-03-27T21:39:52.642 INFO:teuthology.orchestra.run.smithi060.stdout: "clog_to_syslog": "false", 2024-03-27T21:39:52.642 INFO:teuthology.orchestra.run.smithi060.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-27T21:39:52.642 INFO:teuthology.orchestra.run.smithi060.stdout: "clog_to_syslog_level": "info", 2024-03-27T21:39:52.642 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "-", 2024-03-27T21:39:52.642 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_network": "", 2024-03-27T21:39:52.643 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_network_interface": "", 2024-03-27T21:39:52.643 INFO:teuthology.orchestra.run.smithi060.stdout: "compressor_zlib_isal": "false", 2024-03-27T21:39:52.643 INFO:teuthology.orchestra.run.smithi060.stdout: "compressor_zlib_level": "5", 2024-03-27T21:39:52.643 INFO:teuthology.orchestra.run.smithi060.stdout: "compressor_zlib_winsize": "-15", 2024-03-27T21:39:52.643 INFO:teuthology.orchestra.run.smithi060.stdout: "compressor_zstd_level": "1", 2024-03-27T21:39:52.643 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-03-27T21:39:52.643 INFO:teuthology.orchestra.run.smithi060.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-27T21:39:52.643 INFO:teuthology.orchestra.run.smithi060.stdout: "crimson_alien_op_num_threads": "6", 2024-03-27T21:39:52.643 INFO:teuthology.orchestra.run.smithi060.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-27T21:39:52.643 INFO:teuthology.orchestra.run.smithi060.stdout: "crimson_osd_obc_lru_size": "10", 2024-03-27T21:39:52.643 INFO:teuthology.orchestra.run.smithi060.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-27T21:39:52.643 INFO:teuthology.orchestra.run.smithi060.stdout: "crimson_seastar_smp": "1", 2024-03-27T21:39:52.643 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_location": "", 2024-03-27T21:39:52.643 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_location_hook": "", 2024-03-27T21:39:52.643 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_location_hook_timeout": "10", 2024-03-27T21:39:52.644 INFO:teuthology.orchestra.run.smithi060.stdout: "daemonize": "false", 2024-03-27T21:39:52.644 INFO:teuthology.orchestra.run.smithi060.stdout: "daos_pool": "tank", 2024-03-27T21:39:52.644 INFO:teuthology.orchestra.run.smithi060.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-27T21:39:52.644 INFO:teuthology.orchestra.run.smithi060.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-27T21:39:52.644 INFO:teuthology.orchestra.run.smithi060.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-27T21:39:52.644 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_alienstore": "0/5", 2024-03-27T21:39:52.644 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_allow_any_pool_priority": "false", 2024-03-27T21:39:52.644 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_asok": "1/5", 2024-03-27T21:39:52.644 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_asok_assert_abort": "false", 2024-03-27T21:39:52.644 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_asserts_on_shutdown": "true", 2024-03-27T21:39:52.644 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_auth": "20/20", 2024-03-27T21:39:52.644 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_bdev": "1/3", 2024-03-27T21:39:52.644 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_bluefs": "1/5", 2024-03-27T21:39:52.644 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_bluestore": "1/5", 2024-03-27T21:39:52.644 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_buffer": "0/1", 2024-03-27T21:39:52.645 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_ceph_exporter": "1/5", 2024-03-27T21:39:52.645 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_cephfs_mirror": "0/5", 2024-03-27T21:39:52.645 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_cephsqlite": "0/5", 2024-03-27T21:39:52.645 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_client": "0/5", 2024-03-27T21:39:52.645 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_compressor": "1/5", 2024-03-27T21:39:52.645 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_context": "0/1", 2024-03-27T21:39:52.645 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_crush": "1/1", 2024-03-27T21:39:52.645 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_crypto": "1/5", 2024-03-27T21:39:52.645 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_cyanstore": "0/5", 2024-03-27T21:39:52.645 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_deliberately_leak_memory": "false", 2024-03-27T21:39:52.645 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_disable_randomized_ping": "false", 2024-03-27T21:39:52.645 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_dpdk": "1/5", 2024-03-27T21:39:52.645 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_eventtrace": "1/5", 2024-03-27T21:39:52.645 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_filer": "0/1", 2024-03-27T21:39:52.645 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_filestore": "1/3", 2024-03-27T21:39:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_finisher": "1/1", 2024-03-27T21:39:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_fuse": "1/5", 2024-03-27T21:39:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_heartbeat_testing_span": "0", 2024-03-27T21:39:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_heartbeatmap": "1/5", 2024-03-27T21:39:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-27T21:39:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_javaclient": "1/5", 2024-03-27T21:39:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_journal": "1/3", 2024-03-27T21:39:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_journaler": "0/5", 2024-03-27T21:39:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_kstore": "1/5", 2024-03-27T21:39:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_leveldb": "4/5", 2024-03-27T21:39:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_lockdep": "0/1", 2024-03-27T21:39:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_mclock": "1/5", 2024-03-27T21:39:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_mds": "1/5", 2024-03-27T21:39:52.646 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_mds_balancer": "1/5", 2024-03-27T21:39:52.647 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_mds_locker": "1/5", 2024-03-27T21:39:52.647 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_mds_log": "1/5", 2024-03-27T21:39:52.647 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_mds_log_expire": "1/5", 2024-03-27T21:39:52.647 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_mds_migrator": "1/5", 2024-03-27T21:39:52.647 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_memstore": "1/5", 2024-03-27T21:39:52.647 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_mgr": "2/5", 2024-03-27T21:39:52.647 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_mgrc": "1/5", 2024-03-27T21:39:52.647 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_mon": "20/20", 2024-03-27T21:39:52.647 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_monc": "0/10", 2024-03-27T21:39:52.647 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_ms": "1/1", 2024-03-27T21:39:52.647 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_none": "0/5", 2024-03-27T21:39:52.647 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_objclass": "0/5", 2024-03-27T21:39:52.647 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_objectcacher": "0/5", 2024-03-27T21:39:52.647 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_objecter": "0/1", 2024-03-27T21:39:52.647 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_optracker": "0/5", 2024-03-27T21:39:52.648 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_osd": "1/5", 2024-03-27T21:39:52.648 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_paxos": "20/20", 2024-03-27T21:39:52.648 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_perfcounter": "1/5", 2024-03-27T21:39:52.648 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_prioritycache": "1/5", 2024-03-27T21:39:52.648 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_rados": "0/5", 2024-03-27T21:39:52.648 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_rbd": "0/5", 2024-03-27T21:39:52.648 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_rbd_mirror": "0/5", 2024-03-27T21:39:52.648 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_rbd_pwl": "0/5", 2024-03-27T21:39:52.648 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_rbd_replay": "0/5", 2024-03-27T21:39:52.648 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_refs": "0/0", 2024-03-27T21:39:52.648 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_reserver": "1/1", 2024-03-27T21:39:52.648 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_rgw": "1/5", 2024-03-27T21:39:52.648 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_rgw_access": "1/5", 2024-03-27T21:39:52.648 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_rgw_datacache": "1/5", 2024-03-27T21:39:52.648 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_rgw_dbstore": "1/5", 2024-03-27T21:39:52.649 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_rgw_flight": "1/5", 2024-03-27T21:39:52.649 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_rgw_sync": "1/5", 2024-03-27T21:39:52.649 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_rocksdb": "4/5", 2024-03-27T21:39:52.649 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_seastore": "0/5", 2024-03-27T21:39:52.649 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_seastore_backref": "0/5", 2024-03-27T21:39:52.649 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_seastore_cache": "0/5", 2024-03-27T21:39:52.649 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_seastore_cleaner": "0/5", 2024-03-27T21:39:52.649 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_seastore_device": "0/5", 2024-03-27T21:39:52.649 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_seastore_epm": "0/5", 2024-03-27T21:39:52.649 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-27T21:39:52.649 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_seastore_journal": "0/5", 2024-03-27T21:39:52.649 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_seastore_lba": "0/5", 2024-03-27T21:39:52.649 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_seastore_odata": "0/5", 2024-03-27T21:39:52.649 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_seastore_omap": "0/5", 2024-03-27T21:39:52.649 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_seastore_onode": "0/5", 2024-03-27T21:39:52.650 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_seastore_t": "0/5", 2024-03-27T21:39:52.650 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_seastore_tm": "0/5", 2024-03-27T21:39:52.650 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_striper": "0/1", 2024-03-27T21:39:52.650 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_test": "0/5", 2024-03-27T21:39:52.650 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_throttle": "1/1", 2024-03-27T21:39:52.650 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_timer": "0/1", 2024-03-27T21:39:52.650 INFO:teuthology.orchestra.run.smithi060.stdout: "debug_tp": "0/5", 2024-03-27T21:39:52.650 INFO:teuthology.orchestra.run.smithi060.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-03-27T21:39:52.650 INFO:teuthology.orchestra.run.smithi060.stdout: "device_failure_prediction_mode": "none", 2024-03-27T21:39:52.650 INFO:teuthology.orchestra.run.smithi060.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-03-27T21:39:52.650 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-27T21:39:52.650 INFO:teuthology.orchestra.run.smithi060.stdout: "err_to_graylog": "false", 2024-03-27T21:39:52.650 INFO:teuthology.orchestra.run.smithi060.stdout: "err_to_journald": "false", 2024-03-27T21:39:52.650 INFO:teuthology.orchestra.run.smithi060.stdout: "err_to_stderr": "true", 2024-03-27T21:39:52.651 INFO:teuthology.orchestra.run.smithi060.stdout: "err_to_syslog": "false", 2024-03-27T21:39:52.651 INFO:teuthology.orchestra.run.smithi060.stdout: "event_tracing": "false", 2024-03-27T21:39:52.651 INFO:teuthology.orchestra.run.smithi060.stdout: "exporter_addr": "0.0.0.0", 2024-03-27T21:39:52.651 INFO:teuthology.orchestra.run.smithi060.stdout: "exporter_http_port": "9926", 2024-03-27T21:39:52.651 INFO:teuthology.orchestra.run.smithi060.stdout: "exporter_prio_limit": "5", 2024-03-27T21:39:52.651 INFO:teuthology.orchestra.run.smithi060.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-27T21:39:52.651 INFO:teuthology.orchestra.run.smithi060.stdout: "exporter_sort_metrics": "true", 2024-03-27T21:39:52.651 INFO:teuthology.orchestra.run.smithi060.stdout: "exporter_stats_period": "5", 2024-03-27T21:39:52.651 INFO:teuthology.orchestra.run.smithi060.stdout: "fake_statfs_for_testing": "0", 2024-03-27T21:39:52.651 INFO:teuthology.orchestra.run.smithi060.stdout: "fatal_signal_handlers": "true", 2024-03-27T21:39:52.651 INFO:teuthology.orchestra.run.smithi060.stdout: "filer_max_purge_ops": "10", 2024-03-27T21:39:52.651 INFO:teuthology.orchestra.run.smithi060.stdout: "filer_max_truncate_ops": "128", 2024-03-27T21:39:52.651 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_apply_finisher_threads": "1", 2024-03-27T21:39:52.651 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_blackhole": "false", 2024-03-27T21:39:52.652 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_btrfs_clone_range": "true", 2024-03-27T21:39:52.652 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_btrfs_snap": "true", 2024-03-27T21:39:52.652 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_caller_concurrency": "10", 2024-03-27T21:39:52.652 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_collect_device_partition_information": "true", 2024-03-27T21:39:52.652 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_commit_timeout": "600.000000", 2024-03-27T21:39:52.652 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_debug_inject_read_err": "false", 2024-03-27T21:39:52.652 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_debug_omap_check": "false", 2024-03-27T21:39:52.652 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_debug_random_read_err": "0.000000", 2024-03-27T21:39:52.652 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_debug_verify_split": "false", 2024-03-27T21:39:52.652 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_dump_file": "", 2024-03-27T21:39:52.652 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-27T21:39:52.652 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-27T21:39:52.652 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_fadvise": "true", 2024-03-27T21:39:52.652 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_fail_eio": "true", 2024-03-27T21:39:52.652 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_fd_cache_shards": "16", 2024-03-27T21:39:52.653 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_fd_cache_size": "128", 2024-03-27T21:39:52.653 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_fiemap": "false", 2024-03-27T21:39:52.653 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_fiemap_threshold": "4096", 2024-03-27T21:39:52.653 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-27T21:39:52.653 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-27T21:39:52.653 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_inject_stall": "0", 2024-03-27T21:39:52.653 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_journal_parallel": "false", 2024-03-27T21:39:52.653 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_journal_trailing": "false", 2024-03-27T21:39:52.653 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_journal_writeahead": "false", 2024-03-27T21:39:52.653 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_kill_at": "0", 2024-03-27T21:39:52.653 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-27T21:39:52.653 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-27T21:39:52.653 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-27T21:39:52.653 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-27T21:39:52.653 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-27T21:39:52.654 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_max_inline_xattrs": "0", 2024-03-27T21:39:52.654 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-27T21:39:52.654 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-27T21:39:52.654 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-27T21:39:52.654 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-27T21:39:52.654 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_max_xattr_value_size": "0", 2024-03-27T21:39:52.654 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-27T21:39:52.654 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-27T21:39:52.654 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-27T21:39:52.654 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_merge_threshold": "-10", 2024-03-27T21:39:52.654 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-27T21:39:52.654 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_odsync_write": "false", 2024-03-27T21:39:52.654 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_omap_backend": "rocksdb", 2024-03-27T21:39:52.655 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_omap_backend_path": "", 2024-03-27T21:39:52.655 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-27T21:39:52.655 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-27T21:39:52.655 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-27T21:39:52.655 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_op_thread_timeout": "60", 2024-03-27T21:39:52.655 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_op_threads": "2", 2024-03-27T21:39:52.655 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_punch_hole": "false", 2024-03-27T21:39:52.655 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-27T21:39:52.655 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-27T21:39:52.655 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-27T21:39:52.655 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-27T21:39:52.655 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-27T21:39:52.655 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-27T21:39:52.655 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-27T21:39:52.655 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-27T21:39:52.656 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-27T21:39:52.656 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_queue_max_ops": "50", 2024-03-27T21:39:52.656 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-27T21:39:52.656 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_seek_data_hole": "false", 2024-03-27T21:39:52.656 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_sloppy_crc": "false", 2024-03-27T21:39:52.656 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-27T21:39:52.656 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_splice": "false", 2024-03-27T21:39:52.656 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_split_multiple": "2", 2024-03-27T21:39:52.656 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_split_rand_factor": "20", 2024-03-27T21:39:52.656 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_update_to": "1000", 2024-03-27T21:39:52.656 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-27T21:39:52.656 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-27T21:39:52.656 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-27T21:39:52.656 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-27T21:39:52.656 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-27T21:39:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-27T21:39:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_wbthrottle_enable": "true", 2024-03-27T21:39:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-27T21:39:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-27T21:39:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-27T21:39:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-27T21:39:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-27T21:39:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-27T21:39:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_xfs_extsize": "false", 2024-03-27T21:39:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: "filestore_zfs_snap": "false", 2024-03-27T21:39:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: "fio_dir": "/tmp/fio", 2024-03-27T21:39:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "dcd19d9a-535c-4b41-9095-d588268e2860", 2024-03-27T21:39:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: "fuse_allow_other": "true", 2024-03-27T21:39:52.657 INFO:teuthology.orchestra.run.smithi060.stdout: "fuse_atomic_o_trunc": "true", 2024-03-27T21:39:52.658 INFO:teuthology.orchestra.run.smithi060.stdout: "fuse_big_writes": "true", 2024-03-27T21:39:52.658 INFO:teuthology.orchestra.run.smithi060.stdout: "fuse_debug": "false", 2024-03-27T21:39:52.658 INFO:teuthology.orchestra.run.smithi060.stdout: "fuse_default_permissions": "false", 2024-03-27T21:39:52.658 INFO:teuthology.orchestra.run.smithi060.stdout: "fuse_disable_pagecache": "false", 2024-03-27T21:39:52.658 INFO:teuthology.orchestra.run.smithi060.stdout: "fuse_max_write": "0", 2024-03-27T21:39:52.658 INFO:teuthology.orchestra.run.smithi060.stdout: "fuse_multithreaded": "true", 2024-03-27T21:39:52.658 INFO:teuthology.orchestra.run.smithi060.stdout: "fuse_require_active_mds": "true", 2024-03-27T21:39:52.658 INFO:teuthology.orchestra.run.smithi060.stdout: "fuse_set_user_groups": "true", 2024-03-27T21:39:52.658 INFO:teuthology.orchestra.run.smithi060.stdout: "fuse_splice_move": "true", 2024-03-27T21:39:52.658 INFO:teuthology.orchestra.run.smithi060.stdout: "fuse_splice_read": "true", 2024-03-27T21:39:52.658 INFO:teuthology.orchestra.run.smithi060.stdout: "fuse_splice_write": "true", 2024-03-27T21:39:52.658 INFO:teuthology.orchestra.run.smithi060.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-27T21:39:52.658 INFO:teuthology.orchestra.run.smithi060.stdout: "fuse_use_invalidate_cb": "true", 2024-03-27T21:39:52.658 INFO:teuthology.orchestra.run.smithi060.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-27T21:39:52.659 INFO:teuthology.orchestra.run.smithi060.stdout: "gss_target_name": "ceph", 2024-03-27T21:39:52.659 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_file": "", 2024-03-27T21:39:52.659 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_inject_failure": "0", 2024-03-27T21:39:52.659 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_interval": "5", 2024-03-27T21:39:52.659 INFO:teuthology.orchestra.run.smithi060.stdout: "host": "smithi060", 2024-03-27T21:39:52.659 INFO:teuthology.orchestra.run.smithi060.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-27T21:39:52.659 INFO:teuthology.orchestra.run.smithi060.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-27T21:39:52.659 INFO:teuthology.orchestra.run.smithi060.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-27T21:39:52.659 INFO:teuthology.orchestra.run.smithi060.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-27T21:39:52.659 INFO:teuthology.orchestra.run.smithi060.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-27T21:39:52.659 INFO:teuthology.orchestra.run.smithi060.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-27T21:39:52.659 INFO:teuthology.orchestra.run.smithi060.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-27T21:39:52.659 INFO:teuthology.orchestra.run.smithi060.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-27T21:39:52.659 INFO:teuthology.orchestra.run.smithi060.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-27T21:39:52.659 INFO:teuthology.orchestra.run.smithi060.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-27T21:39:52.660 INFO:teuthology.orchestra.run.smithi060.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-27T21:39:52.660 INFO:teuthology.orchestra.run.smithi060.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-27T21:39:52.660 INFO:teuthology.orchestra.run.smithi060.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-27T21:39:52.660 INFO:teuthology.orchestra.run.smithi060.stdout: "inject_early_sigterm": "false", 2024-03-27T21:39:52.660 INFO:teuthology.orchestra.run.smithi060.stdout: "jaeger_agent_port": "6799", 2024-03-27T21:39:52.660 INFO:teuthology.orchestra.run.smithi060.stdout: "jaeger_tracing_enable": "false", 2024-03-27T21:39:52.660 INFO:teuthology.orchestra.run.smithi060.stdout: "journal_aio": "true", 2024-03-27T21:39:52.660 INFO:teuthology.orchestra.run.smithi060.stdout: "journal_align_min_size": "65536", 2024-03-27T21:39:52.660 INFO:teuthology.orchestra.run.smithi060.stdout: "journal_block_align": "true", 2024-03-27T21:39:52.660 INFO:teuthology.orchestra.run.smithi060.stdout: "journal_block_size": "4096", 2024-03-27T21:39:52.660 INFO:teuthology.orchestra.run.smithi060.stdout: "journal_dio": "true", 2024-03-27T21:39:52.660 INFO:teuthology.orchestra.run.smithi060.stdout: "journal_discard": "false", 2024-03-27T21:39:52.660 INFO:teuthology.orchestra.run.smithi060.stdout: "journal_force_aio": "false", 2024-03-27T21:39:52.660 INFO:teuthology.orchestra.run.smithi060.stdout: "journal_ignore_corruption": "false", 2024-03-27T21:39:52.661 INFO:teuthology.orchestra.run.smithi060.stdout: "journal_max_write_bytes": "10485760", 2024-03-27T21:39:52.661 INFO:teuthology.orchestra.run.smithi060.stdout: "journal_max_write_entries": "100", 2024-03-27T21:39:52.661 INFO:teuthology.orchestra.run.smithi060.stdout: "journal_replay_from": "0", 2024-03-27T21:39:52.661 INFO:teuthology.orchestra.run.smithi060.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-27T21:39:52.661 INFO:teuthology.orchestra.run.smithi060.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-27T21:39:52.661 INFO:teuthology.orchestra.run.smithi060.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-27T21:39:52.661 INFO:teuthology.orchestra.run.smithi060.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-27T21:39:52.661 INFO:teuthology.orchestra.run.smithi060.stdout: "journal_write_header_frequency": "0", 2024-03-27T21:39:52.661 INFO:teuthology.orchestra.run.smithi060.stdout: "journal_zero_on_create": "false", 2024-03-27T21:39:52.661 INFO:teuthology.orchestra.run.smithi060.stdout: "journaler_prefetch_periods": "10", 2024-03-27T21:39:52.661 INFO:teuthology.orchestra.run.smithi060.stdout: "journaler_prezero_periods": "5", 2024-03-27T21:39:52.661 INFO:teuthology.orchestra.run.smithi060.stdout: "journaler_write_head_interval": "15", 2024-03-27T21:39:52.661 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-03-27T21:39:52.661 INFO:teuthology.orchestra.run.smithi060.stdout: "keyfile": "", 2024-03-27T21:39:52.661 INFO:teuthology.orchestra.run.smithi060.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-27T21:39:52.662 INFO:teuthology.orchestra.run.smithi060.stdout: "kstore_backend": "rocksdb", 2024-03-27T21:39:52.662 INFO:teuthology.orchestra.run.smithi060.stdout: "kstore_default_stripe_size": "65536", 2024-03-27T21:39:52.662 INFO:teuthology.orchestra.run.smithi060.stdout: "kstore_fsck_on_mount": "false", 2024-03-27T21:39:52.662 INFO:teuthology.orchestra.run.smithi060.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-27T21:39:52.662 INFO:teuthology.orchestra.run.smithi060.stdout: "kstore_max_bytes": "67108864", 2024-03-27T21:39:52.662 INFO:teuthology.orchestra.run.smithi060.stdout: "kstore_max_ops": "512", 2024-03-27T21:39:52.662 INFO:teuthology.orchestra.run.smithi060.stdout: "kstore_nid_prealloc": "1024", 2024-03-27T21:39:52.662 INFO:teuthology.orchestra.run.smithi060.stdout: "kstore_onode_map_size": "1024", 2024-03-27T21:39:52.662 INFO:teuthology.orchestra.run.smithi060.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-27T21:39:52.662 INFO:teuthology.orchestra.run.smithi060.stdout: "kstore_sync_submit_transaction": "false", 2024-03-27T21:39:52.662 INFO:teuthology.orchestra.run.smithi060.stdout: "kstore_sync_transaction": "false", 2024-03-27T21:39:52.662 INFO:teuthology.orchestra.run.smithi060.stdout: "leveldb_block_size": "65536", 2024-03-27T21:39:52.662 INFO:teuthology.orchestra.run.smithi060.stdout: "leveldb_bloom_size": "0", 2024-03-27T21:39:52.662 INFO:teuthology.orchestra.run.smithi060.stdout: "leveldb_cache_size": "536870912", 2024-03-27T21:39:52.662 INFO:teuthology.orchestra.run.smithi060.stdout: "leveldb_compact_on_mount": "false", 2024-03-27T21:39:52.663 INFO:teuthology.orchestra.run.smithi060.stdout: "leveldb_compression": "false", 2024-03-27T21:39:52.663 INFO:teuthology.orchestra.run.smithi060.stdout: "leveldb_log": "", 2024-03-27T21:39:52.663 INFO:teuthology.orchestra.run.smithi060.stdout: "leveldb_log_to_ceph_log": "true", 2024-03-27T21:39:52.663 INFO:teuthology.orchestra.run.smithi060.stdout: "leveldb_max_open_files": "0", 2024-03-27T21:39:52.663 INFO:teuthology.orchestra.run.smithi060.stdout: "leveldb_paranoid": "false", 2024-03-27T21:39:52.663 INFO:teuthology.orchestra.run.smithi060.stdout: "leveldb_write_buffer_size": "33554432", 2024-03-27T21:39:52.663 INFO:teuthology.orchestra.run.smithi060.stdout: "librados_thread_count": "2", 2024-03-27T21:39:52.663 INFO:teuthology.orchestra.run.smithi060.stdout: "lockdep": "true", 2024-03-27T21:39:52.663 INFO:teuthology.orchestra.run.smithi060.stdout: "lockdep_force_backtrace": "false", 2024-03-27T21:39:52.663 INFO:teuthology.orchestra.run.smithi060.stdout: "log_coarse_timestamps": "true", 2024-03-27T21:39:52.663 INFO:teuthology.orchestra.run.smithi060.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-27T21:39:52.663 INFO:teuthology.orchestra.run.smithi060.stdout: "log_flush_on_exit": "false", 2024-03-27T21:39:52.663 INFO:teuthology.orchestra.run.smithi060.stdout: "log_graylog_host": "127.0.0.1", 2024-03-27T21:39:52.663 INFO:teuthology.orchestra.run.smithi060.stdout: "log_graylog_port": "12201", 2024-03-27T21:39:52.664 INFO:teuthology.orchestra.run.smithi060.stdout: "log_max_new": "1000", 2024-03-27T21:39:52.664 INFO:teuthology.orchestra.run.smithi060.stdout: "log_max_recent": "10000", 2024-03-27T21:39:52.664 INFO:teuthology.orchestra.run.smithi060.stdout: "log_stderr_prefix": "", 2024-03-27T21:39:52.664 INFO:teuthology.orchestra.run.smithi060.stdout: "log_stop_at_utilization": "0.970000", 2024-03-27T21:39:52.664 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": "true", 2024-03-27T21:39:52.664 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_graylog": "false", 2024-03-27T21:39:52.664 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_journald": "false", 2024-03-27T21:39:52.664 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_stderr": "false", 2024-03-27T21:39:52.664 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_syslog": "false", 2024-03-27T21:39:52.664 INFO:teuthology.orchestra.run.smithi060.stdout: "max_rotating_auth_attempts": "10", 2024-03-27T21:39:52.664 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-27T21:39:52.664 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_action_on_write_error": "1", 2024-03-27T21:39:52.664 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_alternate_name_max": "8192", 2024-03-27T21:39:52.664 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_asio_thread_count": "2", 2024-03-27T21:39:52.664 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_export_pin": "true", 2024-03-27T21:39:52.665 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_fragment_dirs": "true", 2024-03-27T21:39:52.665 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-27T21:39:52.665 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_fragment_interval": "5", 2024-03-27T21:39:52.665 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-27T21:39:52.665 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-27T21:39:52.665 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_interval": "10", 2024-03-27T21:39:52.665 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_max": "-1", 2024-03-27T21:39:52.665 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_max_until": "-1", 2024-03-27T21:39:52.665 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_merge_size": "50", 2024-03-27T21:39:52.665 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_midchunk": "0.300000", 2024-03-27T21:39:52.665 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-27T21:39:52.665 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_min_start": "0.200000", 2024-03-27T21:39:52.665 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_minchunk": "0.001000", 2024-03-27T21:39:52.665 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_mode": "0", 2024-03-27T21:39:52.666 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_need_max": "1.200000", 2024-03-27T21:39:52.666 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_need_min": "0.800000", 2024-03-27T21:39:52.666 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-27T21:39:52.666 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-27T21:39:52.666 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_split_bits": "3", 2024-03-27T21:39:52.666 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-27T21:39:52.666 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_split_size": "10000", 2024-03-27T21:39:52.666 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-27T21:39:52.666 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_target_decay": "10.000000", 2024-03-27T21:39:52.666 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-27T21:39:52.666 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_beacon_grace": "15.000000", 2024-03-27T21:39:52.666 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_beacon_interval": "4.000000", 2024-03-27T21:39:52.666 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-27T21:39:52.666 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-27T21:39:52.667 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_cache_mid": "0.700000", 2024-03-27T21:39:52.667 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_cache_release_free_interval": "10", 2024-03-27T21:39:52.667 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_cache_reservation": "0.050000", 2024-03-27T21:39:52.667 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-27T21:39:52.667 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_cache_trim_interval": "1", 2024-03-27T21:39:52.667 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_cache_trim_threshold": "262144", 2024-03-27T21:39:52.667 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-27T21:39:52.667 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-27T21:39:52.667 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-27T21:39:52.667 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_client_prealloc_inos": "1000", 2024-03-27T21:39:52.667 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-27T21:39:52.667 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_connect_bootstrapping": "false", 2024-03-27T21:39:52.667 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_damage_table_max_entries": "10000", 2024-03-27T21:39:52.667 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-27T21:39:52.668 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_debug_auth_pins": "false", 2024-03-27T21:39:52.668 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_debug_frag": "false", 2024-03-27T21:39:52.668 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_debug_scatterstat": "false", 2024-03-27T21:39:52.668 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_debug_subtrees": "false", 2024-03-27T21:39:52.668 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_decay_halflife": "5.000000", 2024-03-27T21:39:52.668 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_default_dir_hash": "2", 2024-03-27T21:39:52.668 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_defer_session_stale": "true", 2024-03-27T21:39:52.668 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_deny_all_reconnect": "false", 2024-03-27T21:39:52.668 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_dir_keys_per_op": "16384", 2024-03-27T21:39:52.668 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_dir_max_commit_size": "10", 2024-03-27T21:39:52.668 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_dir_max_entries": "0", 2024-03-27T21:39:52.668 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_dir_prefetch": "true", 2024-03-27T21:39:52.668 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-27T21:39:52.668 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-27T21:39:52.668 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_dump_cache_on_map": "false", 2024-03-27T21:39:52.669 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-27T21:39:52.669 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-27T21:39:52.669 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_early_reply": "true", 2024-03-27T21:39:52.669 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_enable_op_tracker": "true", 2024-03-27T21:39:52.669 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_enforce_unique_name": "true", 2024-03-27T21:39:52.669 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-27T21:39:52.669 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-27T21:39:52.669 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_export_ephemeral_random": "true", 2024-03-27T21:39:52.669 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-27T21:39:52.669 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-27T21:39:52.669 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-27T21:39:52.669 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-27T21:39:52.669 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-27T21:39:52.669 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-27T21:39:52.669 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-27T21:39:52.670 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-27T21:39:52.670 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_health_summarize_threshold": "10", 2024-03-27T21:39:52.670 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-27T21:39:52.670 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-27T21:39:52.670 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_inject_health_dummy": "false", 2024-03-27T21:39:52.670 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-27T21:39:52.670 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_inject_migrator_session_race": "false", 2024-03-27T21:39:52.670 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-27T21:39:52.670 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-27T21:39:52.670 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-27T21:39:52.670 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_join_fs": "", 2024-03-27T21:39:52.670 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_journal_format": "1", 2024-03-27T21:39:52.670 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_kill_create_at": "0", 2024-03-27T21:39:52.670 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_kill_export_at": "0", 2024-03-27T21:39:52.670 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_kill_import_at": "0", 2024-03-27T21:39:52.671 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_kill_journal_at": "0", 2024-03-27T21:39:52.671 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_kill_journal_expire_at": "0", 2024-03-27T21:39:52.671 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_kill_journal_replay_at": "0", 2024-03-27T21:39:52.671 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_kill_link_at": "0", 2024-03-27T21:39:52.671 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_kill_mdstable_at": "0", 2024-03-27T21:39:52.671 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_kill_openc_at": "0", 2024-03-27T21:39:52.671 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_kill_rename_at": "0", 2024-03-27T21:39:52.671 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-27T21:39:52.671 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_log_events_per_segment": "1024", 2024-03-27T21:39:52.671 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_log_max_events": "-1", 2024-03-27T21:39:52.671 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_log_max_segments": "128", 2024-03-27T21:39:52.671 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_log_pause": "false", 2024-03-27T21:39:52.671 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_log_segment_size": "0", 2024-03-27T21:39:52.671 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-27T21:39:52.671 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_log_warn_factor": "2.000000", 2024-03-27T21:39:52.672 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_max_caps_per_client": "1048576", 2024-03-27T21:39:52.672 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_max_completed_flushes": "100000", 2024-03-27T21:39:52.672 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_max_completed_requests": "100000", 2024-03-27T21:39:52.672 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_max_export_size": "20971520", 2024-03-27T21:39:52.672 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_max_file_recover": "32", 2024-03-27T21:39:52.672 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_max_purge_files": "64", 2024-03-27T21:39:52.672 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_max_purge_ops": "8192", 2024-03-27T21:39:52.672 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-27T21:39:52.672 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-27T21:39:52.672 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_max_snaps_per_dir": "100", 2024-03-27T21:39:52.672 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_metrics_update_interval": "2", 2024-03-27T21:39:52.672 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_min_caps_per_client": "100", 2024-03-27T21:39:52.672 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_min_caps_working_set": "10000", 2024-03-27T21:39:52.672 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-27T21:39:52.673 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_numa_node": "-1", 2024-03-27T21:39:52.673 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-27T21:39:52.673 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_op_complaint_time": "30.000000", 2024-03-27T21:39:52.673 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_op_history_duration": "600", 2024-03-27T21:39:52.673 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_op_history_size": "20", 2024-03-27T21:39:52.673 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_op_log_threshold": "5", 2024-03-27T21:39:52.673 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_ping_grace": "15", 2024-03-27T21:39:52.673 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_ping_interval": "5", 2024-03-27T21:39:52.673 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-27T21:39:52.673 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-27T21:39:52.673 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_recall_max_caps": "30000", 2024-03-27T21:39:52.673 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-27T21:39:52.673 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-27T21:39:52.673 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-27T21:39:52.674 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_recall_warning_threshold": "262144", 2024-03-27T21:39:52.674 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-27T21:39:52.674 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_replay_interval": "1.000000", 2024-03-27T21:39:52.674 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-27T21:39:52.674 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-27T21:39:52.674 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_root_ino_gid": "0", 2024-03-27T21:39:52.674 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_root_ino_uid": "0", 2024-03-27T21:39:52.674 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-27T21:39:52.674 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-27T21:39:52.674 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-27T21:39:52.674 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-27T21:39:52.674 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-27T21:39:52.674 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-27T21:39:52.674 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-27T21:39:52.675 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-27T21:39:52.675 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-27T21:39:52.675 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-27T21:39:52.675 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_shutdown_check": "0", 2024-03-27T21:39:52.675 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_skip_ino": "0", 2024-03-27T21:39:52.675 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-27T21:39:52.675 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_snap_max_uid": "4294967294", 2024-03-27T21:39:52.675 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_snap_min_uid": "0", 2024-03-27T21:39:52.675 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_snap_rstat": "false", 2024-03-27T21:39:52.675 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_standby_replay_damaged": "false", 2024-03-27T21:39:52.675 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_symlink_recovery": "true", 2024-03-27T21:39:52.675 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-27T21:39:52.675 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_thrash_exports": "0", 2024-03-27T21:39:52.675 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_thrash_fragments": "0", 2024-03-27T21:39:52.675 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_tick_interval": "5.000000", 2024-03-27T21:39:52.676 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_valgrind_exit": "false", 2024-03-27T21:39:52.676 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_verify_backtrace": "1", 2024-03-27T21:39:52.676 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_verify_scatter": "false", 2024-03-27T21:39:52.676 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-27T21:39:52.676 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_wipe_sessions": "false", 2024-03-27T21:39:52.676 INFO:teuthology.orchestra.run.smithi060.stdout: "mempool_debug": "false", 2024-03-27T21:39:52.676 INFO:teuthology.orchestra.run.smithi060.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-27T21:39:52.676 INFO:teuthology.orchestra.run.smithi060.stdout: "memstore_device_bytes": "1073741824", 2024-03-27T21:39:52.676 INFO:teuthology.orchestra.run.smithi060.stdout: "memstore_page_set": "false", 2024-03-27T21:39:52.676 INFO:teuthology.orchestra.run.smithi060.stdout: "memstore_page_size": "65536", 2024-03-27T21:39:52.676 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_client_bytes": "134217728", 2024-03-27T21:39:52.676 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_client_messages": "512", 2024-03-27T21:39:52.676 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-27T21:39:52.676 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-27T21:39:52.676 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-27T21:39:52.677 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-27T21:39:52.677 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_disabled_modules": "", 2024-03-27T21:39:52.677 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-27T21:39:52.677 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_max_pg_num_change": "128", 2024-03-27T21:39:52.677 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_mds_bytes": "134217728", 2024-03-27T21:39:52.677 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_mds_messages": "128", 2024-03-27T21:39:52.677 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-27T21:39:52.677 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_mon_bytes": "134217728", 2024-03-27T21:39:52.677 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_mon_messages": "128", 2024-03-27T21:39:52.678 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_osd_bytes": "536870912", 2024-03-27T21:39:52.678 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_osd_messages": "8192", 2024-03-27T21:39:52.678 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_pool": "true", 2024-03-27T21:39:52.678 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-27T21:39:52.678 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_standby_modules": "true", 2024-03-27T21:39:52.678 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_stats_period": "5", 2024-03-27T21:39:52.678 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_stats_threshold": "5", 2024-03-27T21:39:52.678 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_tick_period": "2", 2024-03-27T21:39:52.678 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-27T21:39:52.678 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-27T21:39:52.678 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_allow_pool_delete": "true", 2024-03-27T21:39:52.678 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_allow_pool_size_one": "true", 2024-03-27T21:39:52.678 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_auth_validate_all_caps": "true", 2024-03-27T21:39:52.679 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-27T21:39:52.679 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-27T21:39:52.679 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_client_bytes": "104857600", 2024-03-27T21:39:52.679 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_client_directed_command_retry": "5", 2024-03-27T21:39:52.679 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-27T21:39:52.679 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-27T21:39:52.679 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-27T21:39:52.679 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-27T21:39:52.679 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_client_hunt_parallel": "3", 2024-03-27T21:39:52.679 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_client_log_interval": "1.000000", 2024-03-27T21:39:52.679 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-27T21:39:52.679 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_client_ping_interval": "10.000000", 2024-03-27T21:39:52.679 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-27T21:39:52.679 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-27T21:39:52.680 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-27T21:39:52.680 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-27T21:39:52.680 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_cluster_log_file_level": "debug", 2024-03-27T21:39:52.680 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_cluster_log_to_file": "true", 2024-03-27T21:39:52.680 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-27T21:39:52.680 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-27T21:39:52.680 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-27T21:39:52.680 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_cluster_log_to_journald": "false", 2024-03-27T21:39:52.680 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-27T21:39:52.680 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-27T21:39:52.680 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-27T21:39:52.680 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-27T21:39:52.680 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_compact_on_bootstrap": "false", 2024-03-27T21:39:52.680 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_compact_on_start": "false", 2024-03-27T21:39:52.681 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_compact_on_trim": "true", 2024-03-27T21:39:52.681 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-27T21:39:52.681 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-27T21:39:52.681 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-27T21:39:52.681 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_cpu_threads": "4", 2024-03-27T21:39:52.681 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_crush_min_required_version": "hammer", 2024-03-27T21:39:52.681 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_daemon_bytes": "419430400", 2024-03-27T21:39:52.681 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-27T21:39:52.681 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_data_avail_crit": "5", 2024-03-27T21:39:52.681 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_data_avail_warn": "5", 2024-03-27T21:39:52.681 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_data_size_warn": "16106127360", 2024-03-27T21:39:52.681 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-27T21:39:52.681 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-27T21:39:52.681 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_debug_dump_json": "false", 2024-03-27T21:39:52.681 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-27T21:39:52.682 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_debug_dump_transactions": "false", 2024-03-27T21:39:52.682 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_debug_extra_checks": "false", 2024-03-27T21:39:52.682 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-27T21:39:52.682 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-27T21:39:52.682 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_debug_no_require_quincy": "false", 2024-03-27T21:39:52.682 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_debug_no_require_reef": "false", 2024-03-27T21:39:52.682 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-27T21:39:52.682 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-27T21:39:52.682 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-27T21:39:52.682 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_down_mkfs_grace": "120", 2024-03-27T21:39:52.682 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_election_default_strategy": "3", 2024-03-27T21:39:52.682 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_election_timeout": "5.000000", 2024-03-27T21:39:52.682 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-27T21:39:52.682 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_elector_ping_divisor": "2", 2024-03-27T21:39:52.683 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-27T21:39:52.683 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_enable_op_tracker": "true", 2024-03-27T21:39:52.683 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_fake_pool_delete": "false", 2024-03-27T21:39:52.683 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_force_quorum_join": "false", 2024-03-27T21:39:52.683 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_globalid_prealloc": "10000", 2024-03-27T21:39:52.683 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_health_detail_to_clog": "false", 2024-03-27T21:39:52.683 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_health_log_update_period": "5", 2024-03-27T21:39:52.683 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_health_max_detail": "50", 2024-03-27T21:39:52.683 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_health_to_clog": "true", 2024-03-27T21:39:52.683 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_health_to_clog_interval": "600", 2024-03-27T21:39:52.683 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-27T21:39:52.683 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_host": "172.21.15.60,[v2:172.21.15.60:3301,v1:172.21.15.60:6790],172.21.15.191", 2024-03-27T21:39:52.683 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_host_override": "", 2024-03-27T21:39:52.683 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_initial_members": "", 2024-03-27T21:39:52.683 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-27T21:39:52.684 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-27T21:39:52.684 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-27T21:39:52.684 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-27T21:39:52.684 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-27T21:39:52.684 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_lease": "5.000000", 2024-03-27T21:39:52.684 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-27T21:39:52.684 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-27T21:39:52.684 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_log_full_interval": "50", 2024-03-27T21:39:52.684 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_log_max": "10000", 2024-03-27T21:39:52.684 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_log_max_summary": "50", 2024-03-27T21:39:52.684 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-27T21:39:52.684 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_max_log_epochs": "500", 2024-03-27T21:39:52.684 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-27T21:39:52.684 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-27T21:39:52.684 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_max_osd": "10000", 2024-03-27T21:39:52.685 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_max_pg_per_osd": "10000", 2024-03-27T21:39:52.685 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_max_pool_pg_num": "65536", 2024-03-27T21:39:52.685 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-27T21:39:52.685 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-27T21:39:52.685 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_mds_force_trim_to": "0", 2024-03-27T21:39:52.685 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_mds_skip_sanity": "false", 2024-03-27T21:39:52.685 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_memory_autotune": "true", 2024-03-27T21:39:52.685 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_memory_target": "2147483648", 2024-03-27T21:39:52.685 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_mgr_beacon_grace": "30", 2024-03-27T21:39:52.685 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-27T21:39:52.685 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_mgr_digest_period": "5", 2024-03-27T21:39:52.685 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_mgr_inactive_grace": "60", 2024-03-27T21:39:52.685 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-27T21:39:52.685 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-27T21:39:52.685 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_min_osdmap_epochs": "50", 2024-03-27T21:39:52.686 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_op_complaint_time": "30", 2024-03-27T21:39:52.686 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_op_history_duration": "600", 2024-03-27T21:39:52.686 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_op_history_size": "20", 2024-03-27T21:39:52.686 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_op_history_slow_op_size": "20", 2024-03-27T21:39:52.686 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-27T21:39:52.686 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_op_log_threshold": "5", 2024-03-27T21:39:52.686 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-27T21:39:52.686 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-27T21:39:52.686 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-27T21:39:52.686 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_auto_mark_in": "false", 2024-03-27T21:39:52.686 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-27T21:39:52.686 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-27T21:39:52.686 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-27T21:39:52.686 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_cache_size": "500", 2024-03-27T21:39:52.687 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-27T21:39:52.687 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-27T21:39:52.687 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-27T21:39:52.687 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_down_out_interval": "600", 2024-03-27T21:39:52.687 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-27T21:39:52.687 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-27T21:39:52.687 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_force_trim_to": "0", 2024-03-27T21:39:52.687 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-27T21:39:52.687 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-27T21:39:52.687 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-27T21:39:52.687 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-27T21:39:52.688 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-27T21:39:52.688 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-27T21:39:52.688 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-27T21:39:52.688 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-27T21:39:52.688 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_min_down_reporters": "2", 2024-03-27T21:39:52.688 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-27T21:39:52.688 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-27T21:39:52.688 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-27T21:39:52.688 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-27T21:39:52.688 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-27T21:39:52.688 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-27T21:39:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_report_timeout": "900", 2024-03-27T21:39:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-27T21:39:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-27T21:39:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-27T21:39:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-27T21:39:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-27T21:39:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-27T21:39:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-27T21:39:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-27T21:39:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-27T21:39:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_pg_stuck_threshold": "60", 2024-03-27T21:39:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-27T21:39:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-27T21:39:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-27T21:39:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-27T21:39:52.690 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-27T21:39:52.690 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-27T21:39:52.690 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_probe_timeout": "2.000000", 2024-03-27T21:39:52.690 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_reweight_max_change": "0.050000", 2024-03-27T21:39:52.690 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_reweight_max_osds": "4", 2024-03-27T21:39:52.690 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-27T21:39:52.690 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-27T21:39:52.690 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-27T21:39:52.690 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-27T21:39:52.690 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-27T21:39:52.690 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_scrub_interval": "300", 2024-03-27T21:39:52.690 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_scrub_max_keys": "100", 2024-03-27T21:39:52.690 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_scrub_timeout": "300", 2024-03-27T21:39:52.690 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_session_timeout": "300", 2024-03-27T21:39:52.691 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_smart_report_timeout": "5", 2024-03-27T21:39:52.691 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_stat_smooth_intervals": "6", 2024-03-27T21:39:52.691 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-27T21:39:52.691 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_stretch_pool_min_size": "2", 2024-03-27T21:39:52.691 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_stretch_pool_size": "4", 2024-03-27T21:39:52.691 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-27T21:39:52.691 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-27T21:39:52.691 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_sync_debug": "false", 2024-03-27T21:39:52.691 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-27T21:39:52.691 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-27T21:39:52.691 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_sync_provider_kill_at": "0", 2024-03-27T21:39:52.691 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_sync_requester_kill_at": "0", 2024-03-27T21:39:52.691 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_sync_timeout": "60.000000", 2024-03-27T21:39:52.691 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_target_pg_per_osd": "100", 2024-03-27T21:39:52.691 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_tick_interval": "5", 2024-03-27T21:39:52.692 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_timecheck_interval": "300.000000", 2024-03-27T21:39:52.692 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-27T21:39:52.692 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_use_min_delay_socket": "false", 2024-03-27T21:39:52.692 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_warn_older_version_delay": "604800", 2024-03-27T21:39:52.692 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-27T21:39:52.692 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-27T21:39:52.692 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-27T21:39:52.692 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_warn_on_filestore_osds": "false", 2024-03-27T21:39:52.692 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-27T21:39:52.692 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-27T21:39:52.692 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-27T21:39:52.692 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_warn_on_misplaced": "false", 2024-03-27T21:39:52.692 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-27T21:39:52.692 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_warn_on_older_version": "true", 2024-03-27T21:39:52.692 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-27T21:39:52.693 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_warn_on_pool_no_app": "false", 2024-03-27T21:39:52.693 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-03-27T21:39:52.693 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-27T21:39:52.693 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-27T21:39:52.693 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-27T21:39:52.693 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-27T21:39:52.693 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-27T21:39:52.693 INFO:teuthology.orchestra.run.smithi060.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-27T21:39:52.693 INFO:teuthology.orchestra.run.smithi060.stdout: "monmap": "", 2024-03-27T21:39:52.693 INFO:teuthology.orchestra.run.smithi060.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-27T21:39:52.693 INFO:teuthology.orchestra.run.smithi060.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-27T21:39:52.693 INFO:teuthology.orchestra.run.smithi060.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-27T21:39:52.693 INFO:teuthology.orchestra.run.smithi060.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-27T21:39:52.693 INFO:teuthology.orchestra.run.smithi060.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-27T21:39:52.694 INFO:teuthology.orchestra.run.smithi060.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-27T21:39:52.694 INFO:teuthology.orchestra.run.smithi060.stdout: "motr_tracing_enabled": "false", 2024-03-27T21:39:52.694 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_async_op_threads": "3", 2024-03-27T21:39:52.694 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-27T21:39:52.694 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_async_rdma_cm": "false", 2024-03-27T21:39:52.694 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_async_rdma_device_name": "", 2024-03-27T21:39:52.694 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_async_rdma_dscp": "96", 2024-03-27T21:39:52.694 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-27T21:39:52.694 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-27T21:39:52.694 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_async_rdma_local_gid": "", 2024-03-27T21:39:52.694 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-27T21:39:52.694 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_async_rdma_port_num": "1", 2024-03-27T21:39:52.694 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-27T21:39:52.694 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-27T21:39:52.695 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-27T21:39:52.695 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-27T21:39:52.695 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_async_rdma_sl": "3", 2024-03-27T21:39:52.695 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_async_rdma_support_srq": "true", 2024-03-27T21:39:52.695 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_async_rdma_type": "ib", 2024-03-27T21:39:52.695 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_async_reap_threshold": "5", 2024-03-27T21:39:52.695 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_bind_before_connect": "false", 2024-03-27T21:39:52.695 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_bind_ipv4": "true", 2024-03-27T21:39:52.695 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_bind_ipv6": "false", 2024-03-27T21:39:52.695 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_bind_msgr1": "true", 2024-03-27T21:39:52.695 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_bind_msgr2": "true", 2024-03-27T21:39:52.695 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_bind_port_max": "7568", 2024-03-27T21:39:52.695 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_bind_port_min": "6800", 2024-03-27T21:39:52.695 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-27T21:39:52.695 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_bind_retry_count": "3", 2024-03-27T21:39:52.696 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_bind_retry_delay": "5", 2024-03-27T21:39:52.696 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_blackhole_client": "false", 2024-03-27T21:39:52.696 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_blackhole_mds": "false", 2024-03-27T21:39:52.696 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_blackhole_mgr": "false", 2024-03-27T21:39:52.696 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_blackhole_mon": "false", 2024-03-27T21:39:52.696 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_blackhole_osd": "false", 2024-03-27T21:39:52.696 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_client_mode": "crc secure", 2024-03-27T21:39:52.696 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_cluster_mode": "crc secure", 2024-03-27T21:39:52.696 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_cluster_type": "", 2024-03-27T21:39:52.696 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_compress_secure": "false", 2024-03-27T21:39:52.696 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_connection_idle_timeout": "900", 2024-03-27T21:39:52.696 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_connection_ready_timeout": "10", 2024-03-27T21:39:52.696 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_crc_data": "true", 2024-03-27T21:39:52.696 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_crc_header": "true", 2024-03-27T21:39:52.696 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_die_on_bad_msg": "false", 2024-03-27T21:39:52.696 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_die_on_bug": "true", 2024-03-27T21:39:52.697 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_die_on_old_message": "true", 2024-03-27T21:39:52.697 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_die_on_skipped_message": "false", 2024-03-27T21:39:52.697 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-27T21:39:52.697 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-27T21:39:52.697 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_dpdk_coremask": "0xF", 2024-03-27T21:39:52.697 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-27T21:39:52.697 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-27T21:39:52.697 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_dpdk_enable_tso": "true", 2024-03-27T21:39:52.697 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-27T21:39:52.697 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-27T21:39:52.697 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_dpdk_hugepages": "", 2024-03-27T21:39:52.697 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-27T21:39:52.697 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-27T21:39:52.697 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_dpdk_lro": "true", 2024-03-27T21:39:52.698 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_dpdk_memory_channel": "4", 2024-03-27T21:39:52.698 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-27T21:39:52.698 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_dpdk_pmd": "", 2024-03-27T21:39:52.698 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_dpdk_port_id": "0", 2024-03-27T21:39:52.698 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-27T21:39:52.698 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-27T21:39:52.698 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_dump_on_send": "false", 2024-03-27T21:39:52.698 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_initial_backoff": "0.200000", 2024-03-27T21:39:52.698 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_inject_delay_max": "1.000000", 2024-03-27T21:39:52.698 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-27T21:39:52.698 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_inject_delay_type": "", 2024-03-27T21:39:52.698 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-27T21:39:52.698 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_inject_network_congestion": "0", 2024-03-27T21:39:52.698 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_inject_socket_failures": "5000", 2024-03-27T21:39:52.699 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_learn_addr_from_peer": "true", 2024-03-27T21:39:52.699 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_max_accept_failures": "4", 2024-03-27T21:39:52.699 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_max_backoff": "15.000000", 2024-03-27T21:39:52.699 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_mon_client_mode": "secure crc", 2024-03-27T21:39:52.699 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-27T21:39:52.699 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_mon_service_mode": "secure crc", 2024-03-27T21:39:52.699 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_osd_compress_min_size": "1024", 2024-03-27T21:39:52.699 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_osd_compress_mode": "none", 2024-03-27T21:39:52.699 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-27T21:39:52.699 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-27T21:39:52.699 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_pq_min_cost": "65536", 2024-03-27T21:39:52.699 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_public_type": "", 2024-03-27T21:39:52.699 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_service_mode": "crc secure", 2024-03-27T21:39:52.699 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_tcp_listen_backlog": "512", 2024-03-27T21:39:52.699 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_tcp_nodelay": "true", 2024-03-27T21:39:52.700 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-27T21:39:52.700 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_tcp_rcvbuf": "0", 2024-03-27T21:39:52.700 INFO:teuthology.orchestra.run.smithi060.stdout: "ms_type": "async", 2024-03-27T21:39:52.700 INFO:teuthology.orchestra.run.smithi060.stdout: "no_config_file": "false", 2024-03-27T21:39:52.700 INFO:teuthology.orchestra.run.smithi060.stdout: "objecter_completion_locks_per_session": "32", 2024-03-27T21:39:52.700 INFO:teuthology.orchestra.run.smithi060.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-27T21:39:52.700 INFO:teuthology.orchestra.run.smithi060.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-27T21:39:52.700 INFO:teuthology.orchestra.run.smithi060.stdout: "objecter_inflight_ops": "1024", 2024-03-27T21:39:52.700 INFO:teuthology.orchestra.run.smithi060.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-27T21:39:52.700 INFO:teuthology.orchestra.run.smithi060.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-27T21:39:52.700 INFO:teuthology.orchestra.run.smithi060.stdout: "objecter_tick_interval": "5.000000", 2024-03-27T21:39:52.700 INFO:teuthology.orchestra.run.smithi060.stdout: "objecter_timeout": "10.000000", 2024-03-27T21:39:52.700 INFO:teuthology.orchestra.run.smithi060.stdout: "objectstore_blackhole": "false", 2024-03-27T21:39:52.700 INFO:teuthology.orchestra.run.smithi060.stdout: "openssl_engine_opts": "", 2024-03-27T21:39:52.700 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_agent_delay_time": "5.000000", 2024-03-27T21:39:52.701 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_agent_hist_halflife": "1000", 2024-03-27T21:39:52.701 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_agent_max_low_ops": "2", 2024-03-27T21:39:52.701 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_agent_max_ops": "4", 2024-03-27T21:39:52.701 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-27T21:39:52.701 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-27T21:39:52.701 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_agent_slop": "0.020000", 2024-03-27T21:39:52.701 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-27T21:39:52.701 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-27T21:39:52.701 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_asio_thread_count": "2", 2024-03-27T21:39:52.701 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_async_recovery_min_cost": "100", 2024-03-27T21:39:52.701 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-27T21:39:52.701 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-27T21:39:52.701 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_backfill_scan_max": "512", 2024-03-27T21:39:52.702 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_backfill_scan_min": "64", 2024-03-27T21:39:52.702 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_backoff_on_degraded": "false", 2024-03-27T21:39:52.702 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_backoff_on_peering": "false", 2024-03-27T21:39:52.702 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_backoff_on_unfound": "true", 2024-03-27T21:39:52.702 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_beacon_report_interval": "300", 2024-03-27T21:39:52.702 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_bench_duration": "30", 2024-03-27T21:39:52.702 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-27T21:39:52.702 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_bench_max_block_size": "67108864", 2024-03-27T21:39:52.702 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-27T21:39:52.702 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_blkin_trace_all": "false", 2024-03-27T21:39:52.703 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-27T21:39:52.703 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-27T21:39:52.703 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-27T21:39:52.703 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-27T21:39:52.703 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_check_for_log_corruption": "false", 2024-03-27T21:39:52.703 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-27T21:39:52.703 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-27T21:39:52.703 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-27T21:39:52.703 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-27T21:39:52.703 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_class_update_on_start": "true", 2024-03-27T21:39:52.703 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_client_message_cap": "256", 2024-03-27T21:39:52.703 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_client_message_size_cap": "524288000", 2024-03-27T21:39:52.703 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_client_op_priority": "63", 2024-03-27T21:39:52.703 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_client_watch_timeout": "30", 2024-03-27T21:39:52.704 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_command_max_records": "256", 2024-03-27T21:39:52.704 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_compact_on_start": "false", 2024-03-27T21:39:52.704 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-27T21:39:52.704 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_crush_chooseleaf_type": "0", 2024-03-27T21:39:52.704 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-27T21:39:52.704 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_crush_update_on_start": "true", 2024-03-27T21:39:52.704 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_crush_update_weight_set": "true", 2024-03-27T21:39:52.704 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-27T21:39:52.704 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-27T21:39:52.704 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-27T21:39:52.704 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-27T21:39:52.704 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-27T21:39:52.704 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_feed_pullee": "-1", 2024-03-27T21:39:52.704 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-27T21:39:52.704 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-27T21:39:52.705 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-27T21:39:52.705 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_misdirected_ops": "false", 2024-03-27T21:39:52.705 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_no_acting_change": "false", 2024-03-27T21:39:52.705 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_no_purge_strays": "false", 2024-03-27T21:39:52.705 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_op_order": "false", 2024-03-27T21:39:52.705 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-27T21:39:52.705 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-27T21:39:52.705 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-27T21:39:52.705 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-27T21:39:52.705 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_shutdown": "false", 2024-03-27T21:39:52.705 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-27T21:39:52.705 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-27T21:39:52.705 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-27T21:39:52.705 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-27T21:39:52.706 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_verify_snaps": "false", 2024-03-27T21:39:52.706 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-27T21:39:52.706 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-27T21:39:52.706 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_deep_scrub_keys": "1024", 2024-03-27T21:39:52.706 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-27T21:39:52.706 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-27T21:39:52.706 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-27T21:39:52.706 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_deep_scrub_stride": "524288", 2024-03-27T21:39:52.706 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-27T21:39:52.706 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_default_data_pool_replay_window": "5", 2024-03-27T21:39:52.706 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_default_notify_timeout": "30", 2024-03-27T21:39:52.706 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_delete_sleep": "0.000000", 2024-03-27T21:39:52.706 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-27T21:39:52.706 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-27T21:39:52.707 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-27T21:39:52.707 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_discard_disconnected_ops": "true", 2024-03-27T21:39:52.707 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_enable_op_tracker": "true", 2024-03-27T21:39:52.707 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-27T21:39:52.707 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-27T21:39:52.707 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-27T21:39:52.707 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-27T21:39:52.707 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_fast_info": "true", 2024-03-27T21:39:52.707 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_fast_shutdown": "true", 2024-03-27T21:39:52.707 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-27T21:39:52.707 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-27T21:39:52.707 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-27T21:39:52.707 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-27T21:39:52.707 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-27T21:39:52.707 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_function_tracing": "false", 2024-03-27T21:39:52.708 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_heartbeat_grace": "20", 2024-03-27T21:39:52.708 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_heartbeat_interval": "6", 2024-03-27T21:39:52.708 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-27T21:39:52.708 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_heartbeat_min_peers": "10", 2024-03-27T21:39:52.708 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_heartbeat_min_size": "2000", 2024-03-27T21:39:52.708 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_heartbeat_stale": "600", 2024-03-27T21:39:52.708 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-27T21:39:52.708 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_hit_set_max_size": "100000", 2024-03-27T21:39:52.708 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_hit_set_min_size": "1000", 2024-03-27T21:39:52.708 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-27T21:39:52.708 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-27T21:39:52.708 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-27T21:39:52.708 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-27T21:39:52.708 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-27T21:39:52.708 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-27T21:39:52.709 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_journal_size": "5120", 2024-03-27T21:39:52.709 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_kill_backfill_at": "0", 2024-03-27T21:39:52.709 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-27T21:39:52.709 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_map_cache_size": "50", 2024-03-27T21:39:52.709 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_map_dedup": "true", 2024-03-27T21:39:52.709 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_map_message_max": "40", 2024-03-27T21:39:52.709 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-27T21:39:52.709 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_map_share_max_epochs": "40", 2024-03-27T21:39:52.709 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_max_attr_name_len": "100", 2024-03-27T21:39:52.709 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_max_attr_size": "0", 2024-03-27T21:39:52.709 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_max_backfills": "1", 2024-03-27T21:39:52.709 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_max_markdown_count": "5", 2024-03-27T21:39:52.709 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_max_markdown_period": "600", 2024-03-27T21:39:52.709 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_max_object_name_len": "2048", 2024-03-27T21:39:52.710 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_max_object_namespace_len": "256", 2024-03-27T21:39:52.710 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_max_object_size": "134217728", 2024-03-27T21:39:52.710 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-27T21:39:52.710 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-27T21:39:52.710 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_max_pg_blocked_by": "16", 2024-03-27T21:39:52.710 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_max_pg_log_entries": "10000", 2024-03-27T21:39:52.710 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-27T21:39:52.710 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_max_pgls": "1024", 2024-03-27T21:39:52.710 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_max_push_cost": "8388608", 2024-03-27T21:39:52.710 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_max_push_objects": "10", 2024-03-27T21:39:52.710 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_max_scrubs": "3", 2024-03-27T21:39:52.710 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-27T21:39:52.711 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_max_trimming_pgs": "2", 2024-03-27T21:39:52.711 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_max_write_op_reply_len": "64", 2024-03-27T21:39:52.711 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_max_write_size": "90", 2024-03-27T21:39:52.711 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-27T21:39:52.711 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-27T21:39:52.711 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-27T21:39:52.711 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-27T21:39:52.711 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-27T21:39:52.711 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-27T21:39:52.711 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-27T21:39:52.711 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-27T21:39:52.711 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mclock_profile": "balanced", 2024-03-27T21:39:52.711 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-27T21:39:52.712 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-27T21:39:52.712 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-27T21:39:52.712 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-27T21:39:52.712 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-27T21:39:52.712 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-27T21:39:52.712 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-27T21:39:52.712 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-27T21:39:52.712 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-27T21:39:52.712 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-27T21:39:52.712 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-27T21:39:52.713 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_memory_base": "805306368", 2024-03-27T21:39:52.713 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_memory_cache_min": "134217728", 2024-03-27T21:39:52.713 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-27T21:39:52.713 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-27T21:39:52.713 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_memory_target": "4294967296", 2024-03-27T21:39:52.713 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_memory_target_autotune": "false", 2024-03-27T21:39:52.713 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-27T21:39:52.713 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_min_pg_log_entries": "250", 2024-03-27T21:39:52.713 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_min_recovery_priority": "0", 2024-03-27T21:39:52.713 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-27T21:39:52.713 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-27T21:39:52.713 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mon_report_interval": "5", 2024-03-27T21:39:52.713 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-27T21:39:52.713 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-27T21:39:52.714 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_num_cache_shards": "32", 2024-03-27T21:39:52.714 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_num_op_tracker_shard": "32", 2024-03-27T21:39:52.714 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_numa_auto_affinity": "true", 2024-03-27T21:39:52.714 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_numa_node": "-1", 2024-03-27T21:39:52.714 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_numa_prefer_iface": "true", 2024-03-27T21:39:52.714 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-27T21:39:52.714 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_objecter_finishers": "1", 2024-03-27T21:39:52.714 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_objectstore": "bluestore", 2024-03-27T21:39:52.714 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_objectstore_fuse": "false", 2024-03-27T21:39:52.714 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_objectstore_tracing": "false", 2024-03-27T21:39:52.715 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_op_complaint_time": "30.000000", 2024-03-27T21:39:52.715 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_op_history_duration": "600", 2024-03-27T21:39:52.715 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_op_history_size": "20", 2024-03-27T21:39:52.715 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_op_history_slow_op_size": "20", 2024-03-27T21:39:52.715 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-27T21:39:52.715 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_op_log_threshold": "5", 2024-03-27T21:39:52.715 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_op_num_shards": "0", 2024-03-27T21:39:52.715 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_op_num_shards_hdd": "5", 2024-03-27T21:39:52.715 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_op_num_shards_ssd": "8", 2024-03-27T21:39:52.715 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-27T21:39:52.715 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-27T21:39:52.715 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-27T21:39:52.715 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-27T21:39:52.715 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_op_pq_min_cost": "65536", 2024-03-27T21:39:52.715 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-27T21:39:52.716 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_op_queue_cut_off": "high", 2024-03-27T21:39:52.716 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-27T21:39:52.716 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_op_thread_timeout": "15", 2024-03-27T21:39:52.716 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_open_classes_on_start": "true", 2024-03-27T21:39:52.716 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_os_flags": "0", 2024-03-27T21:39:52.716 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_peering_op_priority": "255", 2024-03-27T21:39:52.716 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pg_delete_cost": "1048576", 2024-03-27T21:39:52.716 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pg_delete_priority": "5", 2024-03-27T21:39:52.716 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-27T21:39:52.716 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-27T21:39:52.716 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-27T21:39:52.716 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pg_log_trim_max": "10000", 2024-03-27T21:39:52.716 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pg_log_trim_min": "100", 2024-03-27T21:39:52.716 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-27T21:39:52.716 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-27T21:39:52.717 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pg_stat_report_interval_max": "500", 2024-03-27T21:39:52.717 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-27T21:39:52.717 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-27T21:39:52.717 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-27T21:39:52.717 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-27T21:39:52.717 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-27T21:39:52.717 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-27T21:39:52.717 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_crimson": "false", 2024-03-27T21:39:52.717 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-27T21:39:52.717 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-27T21:39:52.717 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-27T21:39:52.717 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-27T21:39:52.717 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-27T21:39:52.717 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-27T21:39:52.718 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-27T21:39:52.718 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-27T21:39:52.718 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_flags": "0", 2024-03-27T21:39:52.718 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-27T21:39:52.718 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_min_size": "0", 2024-03-27T21:39:52.718 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-27T21:39:52.718 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_pg_num": "32", 2024-03-27T21:39:52.718 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_pgp_num": "0", 2024-03-27T21:39:52.718 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-27T21:39:52.718 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_size": "2", 2024-03-27T21:39:52.718 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_default_type": "replicated", 2024-03-27T21:39:52.718 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-27T21:39:52.718 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-27T21:39:52.718 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_push_per_object_cost": "1000", 2024-03-27T21:39:52.719 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-27T21:39:52.719 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_recover_clone_overlap": "true", 2024-03-27T21:39:52.719 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-27T21:39:52.719 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_recovery_cost": "20971520", 2024-03-27T21:39:52.719 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-27T21:39:52.719 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_recovery_max_active": "0", 2024-03-27T21:39:52.719 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-27T21:39:52.719 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-27T21:39:52.719 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-27T21:39:52.719 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-27T21:39:52.719 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_recovery_max_single_start": "1", 2024-03-27T21:39:52.719 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_recovery_op_priority": "3", 2024-03-27T21:39:52.719 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-27T21:39:52.719 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_recovery_priority": "5", 2024-03-27T21:39:52.720 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-27T21:39:52.720 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_recovery_sleep": "0.000000", 2024-03-27T21:39:52.720 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-27T21:39:52.720 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-27T21:39:52.720 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-27T21:39:52.720 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_repair_during_recovery": "false", 2024-03-27T21:39:52.720 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_requested_scrub_priority": "120", 2024-03-27T21:39:52.720 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-27T21:39:52.720 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-27T21:39:52.720 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_auto_repair": "false", 2024-03-27T21:39:52.720 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-27T21:39:52.720 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-27T21:39:52.720 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_begin_hour": "0", 2024-03-27T21:39:52.720 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_begin_week_day": "0", 2024-03-27T21:39:52.720 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_chunk_max": "25", 2024-03-27T21:39:52.721 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_chunk_min": "5", 2024-03-27T21:39:52.721 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_cost": "52428800", 2024-03-27T21:39:52.721 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_during_recovery": "false", 2024-03-27T21:39:52.721 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_end_hour": "0", 2024-03-27T21:39:52.721 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_end_week_day": "0", 2024-03-27T21:39:52.721 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_event_cost": "4096", 2024-03-27T21:39:52.721 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-27T21:39:52.721 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-27T21:39:52.721 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_invalid_stats": "true", 2024-03-27T21:39:52.721 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-27T21:39:52.721 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-27T21:39:52.721 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_max_preemptions": "5", 2024-03-27T21:39:52.721 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-27T21:39:52.721 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_priority": "5", 2024-03-27T21:39:52.721 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_reservation_timeout": "5000", 2024-03-27T21:39:52.722 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_sleep": "0.000000", 2024-03-27T21:39:52.722 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_scrub_slow_reservation_response": "2200", 2024-03-27T21:39:52.722 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-27T21:39:52.722 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-27T21:39:52.722 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-27T21:39:52.722 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_skip_data_digest": "false", 2024-03-27T21:39:52.722 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_smart_report_timeout": "5", 2024-03-27T21:39:52.722 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_snap_trim_cost": "1048576", 2024-03-27T21:39:52.722 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_snap_trim_priority": "5", 2024-03-27T21:39:52.722 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-27T21:39:52.722 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-27T21:39:52.722 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-27T21:39:52.722 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-27T21:39:52.722 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-27T21:39:52.723 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-27T21:39:52.723 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-27T21:39:52.723 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_target_transaction_size": "30", 2024-03-27T21:39:52.723 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-27T21:39:52.723 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-27T21:39:52.723 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-27T21:39:52.723 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-27T21:39:52.723 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-27T21:39:52.723 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-27T21:39:52.723 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-27T21:39:52.723 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-27T21:39:52.723 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-27T21:39:52.723 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-27T21:39:52.723 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_tracing": "false", 2024-03-27T21:39:52.723 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_use_stale_snap": "false", 2024-03-27T21:39:52.724 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-27T21:39:52.724 INFO:teuthology.orchestra.run.smithi060.stdout: "osdc_blkin_trace_all": "false", 2024-03-27T21:39:52.724 INFO:teuthology.orchestra.run.smithi060.stdout: "paxos_kill_at": "0", 2024-03-27T21:39:52.724 INFO:teuthology.orchestra.run.smithi060.stdout: "paxos_max_join_drift": "10", 2024-03-27T21:39:52.724 INFO:teuthology.orchestra.run.smithi060.stdout: "paxos_min": "500", 2024-03-27T21:39:52.724 INFO:teuthology.orchestra.run.smithi060.stdout: "paxos_min_wait": "0.050000", 2024-03-27T21:39:52.724 INFO:teuthology.orchestra.run.smithi060.stdout: "paxos_propose_interval": "1.000000", 2024-03-27T21:39:52.724 INFO:teuthology.orchestra.run.smithi060.stdout: "paxos_service_trim_max": "500", 2024-03-27T21:39:52.724 INFO:teuthology.orchestra.run.smithi060.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-27T21:39:52.724 INFO:teuthology.orchestra.run.smithi060.stdout: "paxos_service_trim_min": "10", 2024-03-27T21:39:52.724 INFO:teuthology.orchestra.run.smithi060.stdout: "paxos_stash_full_interval": "25", 2024-03-27T21:39:52.724 INFO:teuthology.orchestra.run.smithi060.stdout: "paxos_trim_max": "500", 2024-03-27T21:39:52.724 INFO:teuthology.orchestra.run.smithi060.stdout: "paxos_trim_min": "250", 2024-03-27T21:39:52.724 INFO:teuthology.orchestra.run.smithi060.stdout: "perf": "true", 2024-03-27T21:39:52.725 INFO:teuthology.orchestra.run.smithi060.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-03-27T21:39:52.725 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-27T21:39:52.725 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-27T21:39:52.725 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "-", 2024-03-27T21:39:52.725 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrv": "", 2024-03-27T21:39:52.725 INFO:teuthology.orchestra.run.smithi060.stdout: "public_bind_addr": "-", 2024-03-27T21:39:52.725 INFO:teuthology.orchestra.run.smithi060.stdout: "public_network": "", 2024-03-27T21:39:52.725 INFO:teuthology.orchestra.run.smithi060.stdout: "public_network_interface": "", 2024-03-27T21:39:52.725 INFO:teuthology.orchestra.run.smithi060.stdout: "qat_compressor_enabled": "false", 2024-03-27T21:39:52.725 INFO:teuthology.orchestra.run.smithi060.stdout: "qat_compressor_session_max_number": "256", 2024-03-27T21:39:52.725 INFO:teuthology.orchestra.run.smithi060.stdout: "rados_mon_op_timeout": "0", 2024-03-27T21:39:52.725 INFO:teuthology.orchestra.run.smithi060.stdout: "rados_osd_op_timeout": "0", 2024-03-27T21:39:52.725 INFO:teuthology.orchestra.run.smithi060.stdout: "rados_tracing": "false", 2024-03-27T21:39:52.725 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_atime_update_interval": "60", 2024-03-27T21:39:52.725 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-27T21:39:52.726 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_balance_parent_reads": "false", 2024-03-27T21:39:52.726 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_balance_snap_reads": "false", 2024-03-27T21:39:52.726 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_blkin_trace_all": "false", 2024-03-27T21:39:52.726 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-27T21:39:52.726 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-27T21:39:52.726 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_cache": "true", 2024-03-27T21:39:52.726 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-27T21:39:52.726 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-27T21:39:52.726 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-27T21:39:52.726 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-27T21:39:52.726 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_cache_policy": "writearound", 2024-03-27T21:39:52.726 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_cache_size": "33554432", 2024-03-27T21:39:52.726 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-27T21:39:52.726 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-27T21:39:52.727 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_clone_copy_on_read": "false", 2024-03-27T21:39:52.727 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_compression_hint": "none", 2024-03-27T21:39:52.727 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_concurrent_management_ops": "10", 2024-03-27T21:39:52.727 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-27T21:39:52.727 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_default_clone_format": "auto", 2024-03-27T21:39:52.727 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_default_data_pool": "", 2024-03-27T21:39:52.727 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_default_features": "61", 2024-03-27T21:39:52.727 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_default_format": "2", 2024-03-27T21:39:52.727 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_default_map_options": "", 2024-03-27T21:39:52.727 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_default_order": "22", 2024-03-27T21:39:52.727 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_default_pool": "rbd", 2024-03-27T21:39:52.727 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-27T21:39:52.727 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_default_stripe_count": "0", 2024-03-27T21:39:52.727 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_default_stripe_unit": "0", 2024-03-27T21:39:52.728 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-27T21:39:52.728 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-27T21:39:52.728 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-27T21:39:52.728 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_enable_alloc_hint": "true", 2024-03-27T21:39:52.728 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-27T21:39:52.728 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_io_scheduler": "simple", 2024-03-27T21:39:52.728 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-27T21:39:52.728 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-27T21:39:52.728 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-27T21:39:52.728 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-27T21:39:52.728 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-27T21:39:52.728 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-27T21:39:52.728 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-27T21:39:52.728 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-27T21:39:52.728 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-27T21:39:52.729 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_journal_order": "24", 2024-03-27T21:39:52.729 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_journal_pool": "", 2024-03-27T21:39:52.729 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_journal_splay_width": "4", 2024-03-27T21:39:52.729 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_localize_parent_reads": "false", 2024-03-27T21:39:52.729 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_localize_snap_reads": "false", 2024-03-27T21:39:52.729 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-27T21:39:52.729 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-27T21:39:52.729 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-27T21:39:52.729 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-27T21:39:52.729 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-27T21:39:52.729 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-27T21:39:52.729 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-27T21:39:52.729 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-27T21:39:52.729 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-27T21:39:52.729 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-27T21:39:52.730 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-27T21:39:52.730 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-27T21:39:52.730 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-27T21:39:52.730 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-27T21:39:52.730 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-27T21:39:52.730 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-27T21:39:52.730 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-27T21:39:52.730 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-27T21:39:52.730 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-27T21:39:52.730 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-27T21:39:52.730 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-27T21:39:52.730 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-27T21:39:52.730 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-27T21:39:52.730 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-27T21:39:52.731 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-27T21:39:52.731 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-27T21:39:52.731 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-27T21:39:52.731 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-27T21:39:52.731 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-27T21:39:52.731 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-27T21:39:52.731 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-27T21:39:52.731 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_mtime_update_interval": "60", 2024-03-27T21:39:52.731 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_non_blocking_aio": "true", 2024-03-27T21:39:52.731 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_op_thread_timeout": "60", 2024-03-27T21:39:52.731 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_op_threads": "1", 2024-03-27T21:39:52.731 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_parent_cache_enabled": "false", 2024-03-27T21:39:52.731 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-27T21:39:52.731 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-27T21:39:52.731 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-27T21:39:52.732 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_plugins": "", 2024-03-27T21:39:52.732 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_qos_bps_burst": "0", 2024-03-27T21:39:52.732 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-27T21:39:52.732 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_qos_bps_limit": "0", 2024-03-27T21:39:52.732 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_qos_exclude_ops": "0", 2024-03-27T21:39:52.732 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_qos_iops_burst": "0", 2024-03-27T21:39:52.732 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-27T21:39:52.732 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_qos_iops_limit": "0", 2024-03-27T21:39:52.732 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-27T21:39:52.732 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-27T21:39:52.732 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-27T21:39:52.732 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-27T21:39:52.732 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-27T21:39:52.732 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-27T21:39:52.732 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-27T21:39:52.733 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-27T21:39:52.733 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-27T21:39:52.733 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-27T21:39:52.733 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-27T21:39:52.733 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-27T21:39:52.733 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-27T21:39:52.733 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-27T21:39:52.733 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_read_from_replica_policy": "default", 2024-03-27T21:39:52.733 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-27T21:39:52.733 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-27T21:39:52.733 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-27T21:39:52.733 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-27T21:39:52.733 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_skip_partial_discard": "true", 2024-03-27T21:39:52.733 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-27T21:39:52.733 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_tracing": "false", 2024-03-27T21:39:52.734 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_validate_names": "true", 2024-03-27T21:39:52.734 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_validate_pool": "true", 2024-03-27T21:39:52.734 INFO:teuthology.orchestra.run.smithi060.stdout: "restapi_base_url": "", 2024-03-27T21:39:52.734 INFO:teuthology.orchestra.run.smithi060.stdout: "restapi_log_level": "", 2024-03-27T21:39:52.734 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_acl_grants_max_num": "100", 2024-03-27T21:39:52.734 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_admin_entry": "admin", 2024-03-27T21:39:52.734 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-27T21:39:52.734 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_backend_store": "rados", 2024-03-27T21:39:52.734 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_barbican_url": "", 2024-03-27T21:39:52.734 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_beast_enable_async": "true", 2024-03-27T21:39:52.734 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-27T21:39:52.734 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-27T21:39:52.734 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-27T21:39:52.734 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-27T21:39:52.735 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-27T21:39:52.735 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-27T21:39:52.735 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-27T21:39:52.735 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_cache_enabled": "true", 2024-03-27T21:39:52.735 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_cache_expiry_interval": "900", 2024-03-27T21:39:52.735 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_cache_lru_size": "10000", 2024-03-27T21:39:52.735 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_config_store": "rados", 2024-03-27T21:39:52.735 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_content_length_compat": "false", 2024-03-27T21:39:52.735 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_copy_obj_progress": "true", 2024-03-27T21:39:52.735 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-27T21:39:52.735 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_cors_rules_max_num": "100", 2024-03-27T21:39:52.735 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_cross_domain_policy": "", 2024-03-27T21:39:52.735 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-27T21:39:52.735 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_kmip_addr": "", 2024-03-27T21:39:52.735 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-27T21:39:52.736 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-27T21:39:52.736 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-27T21:39:52.736 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-27T21:39:52.736 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_kmip_password": "", 2024-03-27T21:39:52.736 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-27T21:39:52.736 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_kmip_username": "", 2024-03-27T21:39:52.736 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_require_ssl": "true", 2024-03-27T21:39:52.736 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-27T21:39:52.736 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-27T21:39:52.736 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-27T21:39:52.736 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-27T21:39:52.736 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-27T21:39:52.736 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-27T21:39:52.736 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-27T21:39:52.737 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-27T21:39:52.737 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-27T21:39:52.737 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-27T21:39:52.737 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-27T21:39:52.737 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-27T21:39:52.737 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-27T21:39:52.737 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-27T21:39:52.737 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-27T21:39:52.737 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_vault_addr": "", 2024-03-27T21:39:52.737 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_vault_auth": "token", 2024-03-27T21:39:52.737 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_vault_namespace": "", 2024-03-27T21:39:52.737 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_vault_prefix": "", 2024-03-27T21:39:52.737 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-27T21:39:52.737 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-27T21:39:52.737 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-27T21:39:52.738 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-27T21:39:52.738 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_vault_token_file": "", 2024-03-27T21:39:52.738 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-27T21:39:52.738 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_curl_buffersize": "524288", 2024-03-27T21:39:52.738 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-27T21:39:52.738 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_curl_low_speed_time": "300", 2024-03-27T21:39:52.738 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-27T21:39:52.738 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-27T21:39:52.738 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-27T21:39:52.738 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-27T21:39:52.738 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-27T21:39:52.738 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-27T21:39:52.738 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-27T21:39:52.738 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-27T21:39:52.739 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-27T21:39:52.739 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-27T21:39:52.739 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-27T21:39:52.739 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_data_log_changes_size": "1000", 2024-03-27T21:39:52.739 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_data_log_num_shards": "128", 2024-03-27T21:39:52.739 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-27T21:39:52.739 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_data_log_window": "30", 2024-03-27T21:39:52.739 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-27T21:39:52.739 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-27T21:39:52.739 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-27T21:39:52.739 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-27T21:39:52.739 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-27T21:39:52.739 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-27T21:39:52.739 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-27T21:39:52.739 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-27T21:39:52.740 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-27T21:39:52.740 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-27T21:39:52.740 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-27T21:39:52.740 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-27T21:39:52.740 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-27T21:39:52.740 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-27T21:39:52.740 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-27T21:39:52.740 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-27T21:39:52.740 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-27T21:39:52.740 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-27T21:39:52.740 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-27T21:39:52.740 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-27T21:39:52.740 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-27T21:39:52.740 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-27T21:39:52.741 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-27T21:39:52.741 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-27T21:39:52.741 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_dns_name": "", 2024-03-27T21:39:52.741 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_dns_s3website_name": "", 2024-03-27T21:39:52.741 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_dynamic_resharding": "true", 2024-03-27T21:39:52.741 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-27T21:39:52.741 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_enable_gc_threads": "true", 2024-03-27T21:39:52.741 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_enable_lc_threads": "true", 2024-03-27T21:39:52.741 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_enable_ops_log": "false", 2024-03-27T21:39:52.741 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_enable_quota_threads": "true", 2024-03-27T21:39:52.741 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_enable_static_website": "false", 2024-03-27T21:39:52.741 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_enable_usage_log": "false", 2024-03-27T21:39:52.741 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_enforce_swift_acls": "true", 2024-03-27T21:39:52.741 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_exit_timeout_secs": "120", 2024-03-27T21:39:52.741 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_expose_bucket": "false", 2024-03-27T21:39:52.742 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_extended_http_attrs": "", 2024-03-27T21:39:52.742 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_filter": "none", 2024-03-27T21:39:52.742 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-27T21:39:52.742 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_frontends": "beast port=7480", 2024-03-27T21:39:52.742 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-27T21:39:52.742 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_gc_max_deferred": "50", 2024-03-27T21:39:52.742 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-27T21:39:52.742 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_gc_max_objs": "32", 2024-03-27T21:39:52.742 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-27T21:39:52.742 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-27T21:39:52.742 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-27T21:39:52.742 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-27T21:39:52.742 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_gc_processor_period": "3600", 2024-03-27T21:39:52.742 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-27T21:39:52.743 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-27T21:39:52.743 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-27T21:39:52.743 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-27T21:39:52.743 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_init_timeout": "300", 2024-03-27T21:39:52.743 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-27T21:39:52.743 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-27T21:39:52.743 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-27T21:39:52.743 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-27T21:39:52.743 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_admin_domain": "", 2024-03-27T21:39:52.743 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_admin_password": "", 2024-03-27T21:39:52.743 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_admin_password_path": "", 2024-03-27T21:39:52.743 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_admin_project": "", 2024-03-27T21:39:52.743 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_admin_tenant": "", 2024-03-27T21:39:52.743 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_admin_token": "", 2024-03-27T21:39:52.743 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_admin_token_path": "", 2024-03-27T21:39:52.744 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_admin_user": "", 2024-03-27T21:39:52.744 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_api_version": "2", 2024-03-27T21:39:52.744 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_barbican_domain": "", 2024-03-27T21:39:52.744 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_barbican_password": "", 2024-03-27T21:39:52.744 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_barbican_project": "", 2024-03-27T21:39:52.744 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-27T21:39:52.744 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_barbican_user": "", 2024-03-27T21:39:52.744 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-27T21:39:52.744 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-27T21:39:52.744 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-27T21:39:52.744 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-27T21:39:52.744 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-27T21:39:52.744 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_url": "", 2024-03-27T21:39:52.744 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-27T21:39:52.744 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_lc_debug_interval": "-1", 2024-03-27T21:39:52.745 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_lc_lock_max_time": "90", 2024-03-27T21:39:52.745 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_lc_max_objs": "32", 2024-03-27T21:39:52.745 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_lc_max_rules": "1000", 2024-03-27T21:39:52.745 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_lc_max_worker": "3", 2024-03-27T21:39:52.745 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-27T21:39:52.745 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_lc_thread_delay": "0", 2024-03-27T21:39:52.745 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-27T21:39:52.745 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_ldap_dnattr": "uid", 2024-03-27T21:39:52.745 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-27T21:39:52.745 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_ldap_searchfilter": "", 2024-03-27T21:39:52.745 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-27T21:39:52.745 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-27T21:39:52.745 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-27T21:39:52.745 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-27T21:39:52.745 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-27T21:39:52.746 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_log_http_headers": "", 2024-03-27T21:39:52.746 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-27T21:39:52.746 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-27T21:39:52.746 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_log_object_name_utc": "false", 2024-03-27T21:39:52.746 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-03-27T21:39:52.746 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_max_attr_name_len": "0", 2024-03-27T21:39:52.746 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_max_attr_size": "0", 2024-03-27T21:39:52.746 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-27T21:39:52.746 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_max_chunk_size": "4194304", 2024-03-27T21:39:52.746 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-27T21:39:52.746 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-27T21:39:52.746 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-27T21:39:52.746 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_max_listing_results": "1000", 2024-03-27T21:39:52.746 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_max_notify_retries": "10", 2024-03-27T21:39:52.747 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-27T21:39:52.747 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_max_put_param_size": "1048576", 2024-03-27T21:39:52.747 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_max_put_size": "5368709120", 2024-03-27T21:39:52.747 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_max_slo_entries": "1000", 2024-03-27T21:39:52.747 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_md_log_max_shards": "64", 2024-03-27T21:39:52.747 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-27T21:39:52.747 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-27T21:39:52.747 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-27T21:39:52.747 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-27T21:39:52.747 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_mp_lock_max_time": "600", 2024-03-27T21:39:52.747 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-27T21:39:52.747 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-27T21:39:52.747 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-27T21:39:52.747 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-27T21:39:52.747 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-27T21:39:52.748 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-27T21:39:52.748 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-27T21:39:52.748 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-27T21:39:52.748 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_nfs_max_gc": "300", 2024-03-27T21:39:52.748 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-27T21:39:52.748 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-27T21:39:52.748 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-27T21:39:52.748 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-27T21:39:52.748 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-27T21:39:52.748 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-27T21:39:52.748 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-27T21:39:52.748 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_num_async_rados_threads": "32", 2024-03-27T21:39:52.748 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_num_control_oids": "8", 2024-03-27T21:39:52.748 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_numa_node": "-1", 2024-03-27T21:39:52.748 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-27T21:39:52.749 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-27T21:39:52.749 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_objexp_chunk_size": "100", 2024-03-27T21:39:52.749 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_objexp_gc_interval": "600", 2024-03-27T21:39:52.749 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-27T21:39:52.749 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-27T21:39:52.749 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-27T21:39:52.749 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_op_thread_timeout": "600", 2024-03-27T21:39:52.749 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_op_tracing": "false", 2024-03-27T21:39:52.749 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_opa_token": "", 2024-03-27T21:39:52.749 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_opa_url": "", 2024-03-27T21:39:52.749 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_opa_verify_ssl": "true", 2024-03-27T21:39:52.749 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-27T21:39:52.749 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-27T21:39:52.749 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_ops_log_rados": "false", 2024-03-27T21:39:52.750 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_ops_log_socket_path": "", 2024-03-27T21:39:52.750 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-27T21:39:52.750 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-27T21:39:52.750 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-27T21:39:52.750 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_period_push_interval": "2.000000", 2024-03-27T21:39:52.750 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-27T21:39:52.750 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-27T21:39:52.750 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-27T21:39:52.750 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_print_continue": "true", 2024-03-27T21:39:52.750 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-27T21:39:52.750 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-27T21:39:52.750 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-27T21:39:52.750 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-27T21:39:52.750 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-27T21:39:52.751 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_rados_tracing": "false", 2024-03-27T21:39:52.751 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_realm": "", 2024-03-27T21:39:52.751 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_realm_id": "", 2024-03-27T21:39:52.751 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-27T21:39:52.751 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_region": "", 2024-03-27T21:39:52.751 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-27T21:39:52.751 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-27T21:39:52.751 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-27T21:39:52.751 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-27T21:39:52.751 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_request_uri": "", 2024-03-27T21:39:52.751 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_reshard_batch_size": "64", 2024-03-27T21:39:52.751 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-27T21:39:52.751 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_reshard_max_aio": "128", 2024-03-27T21:39:52.751 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_reshard_num_logs": "16", 2024-03-27T21:39:52.751 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_reshard_thread_interval": "600", 2024-03-27T21:39:52.752 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_resolve_cname": "false", 2024-03-27T21:39:52.752 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-27T21:39:52.752 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_run_sync_thread": "true", 2024-03-27T21:39:52.752 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-27T21:39:52.752 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-27T21:39:52.752 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-27T21:39:52.752 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-27T21:39:52.752 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-27T21:39:52.752 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-27T21:39:52.752 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-27T21:39:52.752 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-27T21:39:52.752 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_scheduler_type": "throttler", 2024-03-27T21:39:52.752 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_script_uri": "", 2024-03-27T21:39:52.752 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_service_provider_name": "", 2024-03-27T21:39:52.752 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-27T21:39:52.753 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_sts_client_id": "", 2024-03-27T21:39:52.753 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_sts_client_secret": "", 2024-03-27T21:39:52.753 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_sts_entry": "sts", 2024-03-27T21:39:52.753 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_sts_key": "sts", 2024-03-27T21:39:52.753 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-27T21:39:52.753 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_sts_min_session_duration": "900", 2024-03-27T21:39:52.753 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_sts_token_introspection_url": "", 2024-03-27T21:39:52.753 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_swift_account_in_url": "false", 2024-03-27T21:39:52.753 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_swift_auth_entry": "auth", 2024-03-27T21:39:52.753 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_swift_auth_url": "", 2024-03-27T21:39:52.753 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_swift_custom_header": "", 2024-03-27T21:39:52.753 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-27T21:39:52.753 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_swift_need_stats": "true", 2024-03-27T21:39:52.753 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_swift_tenant_name": "", 2024-03-27T21:39:52.753 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_swift_token_expiration": "86400", 2024-03-27T21:39:52.754 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_swift_url": "", 2024-03-27T21:39:52.754 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_swift_url_prefix": "swift", 2024-03-27T21:39:52.754 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-27T21:39:52.754 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-27T21:39:52.754 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-27T21:39:52.754 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_sync_lease_period": "120", 2024-03-27T21:39:52.754 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-27T21:39:52.754 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-27T21:39:52.754 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-27T21:39:52.754 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-27T21:39:52.754 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-27T21:39:52.754 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-27T21:39:52.754 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-27T21:39:52.754 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-27T21:39:52.755 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-27T21:39:52.755 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_thread_pool_size": "512", 2024-03-27T21:39:52.755 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_torrent_comment": "", 2024-03-27T21:39:52.755 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_torrent_createby": "", 2024-03-27T21:39:52.755 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_torrent_encoding": "", 2024-03-27T21:39:52.755 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_torrent_flag": "false", 2024-03-27T21:39:52.755 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_torrent_origin": "", 2024-03-27T21:39:52.755 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-27T21:39:52.755 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_torrent_tracker": "", 2024-03-27T21:39:52.755 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_trust_forwarded_https": "false", 2024-03-27T21:39:52.755 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-27T21:39:52.755 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-27T21:39:52.755 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_usage_max_shards": "32", 2024-03-27T21:39:52.755 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_usage_max_user_shards": "1", 2024-03-27T21:39:52.755 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_use_opa_authz": "false", 2024-03-27T21:39:52.756 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-27T21:39:52.756 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-27T21:39:52.756 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_user_max_buckets": "1000", 2024-03-27T21:39:52.756 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_user_policies_max_num": "100", 2024-03-27T21:39:52.756 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-27T21:39:52.756 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-27T21:39:52.756 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-27T21:39:52.756 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-27T21:39:52.756 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_user_unique_email": "true", 2024-03-27T21:39:52.756 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_verify_ssl": "true", 2024-03-27T21:39:52.756 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-27T21:39:52.756 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_zone": "", 2024-03-27T21:39:52.756 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_zone_id": "", 2024-03-27T21:39:52.756 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-27T21:39:52.757 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_zonegroup": "", 2024-03-27T21:39:52.757 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_zonegroup_id": "", 2024-03-27T21:39:52.757 INFO:teuthology.orchestra.run.smithi060.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-27T21:39:52.757 INFO:teuthology.orchestra.run.smithi060.stdout: "rgwlc_auto_session_clear": "true", 2024-03-27T21:39:52.757 INFO:teuthology.orchestra.run.smithi060.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-27T21:39:52.757 INFO:teuthology.orchestra.run.smithi060.stdout: "rocksdb_block_size": "4096", 2024-03-27T21:39:52.757 INFO:teuthology.orchestra.run.smithi060.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-27T21:39:52.757 INFO:teuthology.orchestra.run.smithi060.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-27T21:39:52.757 INFO:teuthology.orchestra.run.smithi060.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-27T21:39:52.757 INFO:teuthology.orchestra.run.smithi060.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-27T21:39:52.757 INFO:teuthology.orchestra.run.smithi060.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-27T21:39:52.757 INFO:teuthology.orchestra.run.smithi060.stdout: "rocksdb_cache_size": "536870912", 2024-03-27T21:39:52.757 INFO:teuthology.orchestra.run.smithi060.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-27T21:39:52.757 INFO:teuthology.orchestra.run.smithi060.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-27T21:39:52.757 INFO:teuthology.orchestra.run.smithi060.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-27T21:39:52.757 INFO:teuthology.orchestra.run.smithi060.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-27T21:39:52.758 INFO:teuthology.orchestra.run.smithi060.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-27T21:39:52.758 INFO:teuthology.orchestra.run.smithi060.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-27T21:39:52.758 INFO:teuthology.orchestra.run.smithi060.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-27T21:39:52.758 INFO:teuthology.orchestra.run.smithi060.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-27T21:39:52.758 INFO:teuthology.orchestra.run.smithi060.stdout: "rocksdb_index_type": "binary_search", 2024-03-27T21:39:52.758 INFO:teuthology.orchestra.run.smithi060.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-27T21:39:52.758 INFO:teuthology.orchestra.run.smithi060.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-27T21:39:52.758 INFO:teuthology.orchestra.run.smithi060.stdout: "rocksdb_partition_filters": "false", 2024-03-27T21:39:52.758 INFO:teuthology.orchestra.run.smithi060.stdout: "rocksdb_perf": "false", 2024-03-27T21:39:52.758 INFO:teuthology.orchestra.run.smithi060.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-27T21:39:52.758 INFO:teuthology.orchestra.run.smithi060.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-27T21:39:52.758 INFO:teuthology.orchestra.run.smithi060.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-27T21:39:52.758 INFO:teuthology.orchestra.run.smithi060.stdout: "run_dir": "/var/run/ceph", 2024-03-27T21:39:52.758 INFO:teuthology.orchestra.run.smithi060.stdout: "seastore_block_create": "true", 2024-03-27T21:39:52.759 INFO:teuthology.orchestra.run.smithi060.stdout: "seastore_cache_lru_size": "67108864", 2024-03-27T21:39:52.759 INFO:teuthology.orchestra.run.smithi060.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-27T21:39:52.759 INFO:teuthology.orchestra.run.smithi060.stdout: "seastore_default_max_object_size": "16777216", 2024-03-27T21:39:52.759 INFO:teuthology.orchestra.run.smithi060.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-27T21:39:52.759 INFO:teuthology.orchestra.run.smithi060.stdout: "seastore_device_size": "53687091200", 2024-03-27T21:39:52.759 INFO:teuthology.orchestra.run.smithi060.stdout: "seastore_journal_batch_capacity": "16", 2024-03-27T21:39:52.759 INFO:teuthology.orchestra.run.smithi060.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-27T21:39:52.759 INFO:teuthology.orchestra.run.smithi060.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-27T21:39:52.759 INFO:teuthology.orchestra.run.smithi060.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-27T21:39:52.759 INFO:teuthology.orchestra.run.smithi060.stdout: "seastore_main_device_type": "SSD", 2024-03-27T21:39:52.759 INFO:teuthology.orchestra.run.smithi060.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-27T21:39:52.759 INFO:teuthology.orchestra.run.smithi060.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-27T21:39:52.759 INFO:teuthology.orchestra.run.smithi060.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-27T21:39:52.759 INFO:teuthology.orchestra.run.smithi060.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-27T21:39:52.759 INFO:teuthology.orchestra.run.smithi060.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-27T21:39:52.760 INFO:teuthology.orchestra.run.smithi060.stdout: "seastore_segment_size": "67108864", 2024-03-27T21:39:52.760 INFO:teuthology.orchestra.run.smithi060.stdout: "set_keepcaps": "false", 2024-03-27T21:39:52.760 INFO:teuthology.orchestra.run.smithi060.stdout: "setgroup": "", 2024-03-27T21:39:52.760 INFO:teuthology.orchestra.run.smithi060.stdout: "setuser": "", 2024-03-27T21:39:52.760 INFO:teuthology.orchestra.run.smithi060.stdout: "setuser_match_path": "", 2024-03-27T21:39:52.760 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-27T21:39:52.760 INFO:teuthology.orchestra.run.smithi060.stdout: "thp": "false", 2024-03-27T21:39:52.760 INFO:teuthology.orchestra.run.smithi060.stdout: "threadpool_default_timeout": "60", 2024-03-27T21:39:52.760 INFO:teuthology.orchestra.run.smithi060.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-27T21:39:52.760 INFO:teuthology.orchestra.run.smithi060.stdout: "throttler_perf_counter": "true" 2024-03-27T21:39:52.760 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:39:52.762 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-27T21:39:52.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.873+0000 7f17c495c640 10 monclient: get_monmap_and_config 2024-03-27T21:39:52.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.873+0000 7f17c495c640 10 monclient: build_initial_monmap 2024-03-27T21:39:52.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.873+0000 7f17c495c640 10 monclient: monmap: 2024-03-27T21:39:52.860 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:52.860 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.860 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:52.874719+0000 2024-03-27T21:39:52.860 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:52.874719+0000 2024-03-27T21:39:52.860 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:52.860 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:52.860 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:52.861 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:52.861 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:52.861 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:52.861 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:52.861 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.874+0000 7f17c495c640 1 Processor -- start 2024-03-27T21:39:52.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.874+0000 7f17c495c640 1 -- start start 2024-03-27T21:39:52.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.874+0000 7f17c495c640 10 monclient: init 2024-03-27T21:39:52.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17c495c640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:52.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17c495c640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-03-27T21:39:52.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17c495c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f17c0071a50 0x7f17c00a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17c495c640 10 monclient(hunting): picked mon.noname-a con 0x7f17c0071a50 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:52.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17c495c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f17c013e810 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:52.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17c495c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c013ec50 0x7f17c0141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17c495c640 10 monclient(hunting): picked mon.noname-b con 0x7f17c013ec50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:52.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17c495c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c00a8e90 con 0x7f17c013e810 2024-03-27T21:39:52.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17c495c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17c495c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c0071e30 con 0x7f17c0071a50 2024-03-27T21:39:52.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17c495c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17c495c640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c00aa040 con 0x7f17c013ec50 2024-03-27T21:39:52.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17c495c640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:52.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17c495c640 10 monclient(hunting): authenticate will time out at 834.096070s 2024-03-27T21:39:52.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17bed9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c013ec50 0x7f17c0141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17be59c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f17c0071a50 0x7f17c00a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17be59c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f17c0071a50 0x7f17c00a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44372/0 (socket says 172.21.15.60:44372) 2024-03-27T21:39:52.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17bed9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c013ec50 0x7f17c0141d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47786/0 (socket says 172.21.15.60:47786) 2024-03-27T21:39:52.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17be59c640 1 -- 172.21.15.60:0/3379282727 learned_addr learned my addr 172.21.15.60:0/3379282727 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:52.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17be59c640 10 monclient(hunting): get_auth_request con 0x7f17c0071a50 auth_method 0 2024-03-27T21:39:52.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17be59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17be59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17bed9d640 10 monclient(hunting): get_auth_request con 0x7f17c013ec50 auth_method 0 2024-03-27T21:39:52.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17bed9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:52.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:52.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:52.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17be59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:52.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17be59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:52.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17be59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:52.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17be59c640 10 monclient(hunting): handle_auth_done global_id 4449 payload 274 2024-03-27T21:39:52.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17be59c640 1 -- 172.21.15.60:0/3379282727 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c013ec50 msgr2=0x7f17c0141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:52.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17be59c640 1 --2- 172.21.15.60:0/3379282727 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c013ec50 0x7f17c0141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17be59c640 1 -- 172.21.15.60:0/3379282727 >> v1:172.21.15.60:6789/0 conn(0x7f17c013e810 legacy=0x7f17c0141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17be59c640 10 monclient: _finish_hunting 0 2024-03-27T21:39:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17be59c640 1 monclient: found mon.noname-a 2024-03-27T21:39:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17be59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17be59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17be59c640 1 -- 172.21.15.60:0/3379282727 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c0142280 con 0x7f17c0071a50 2024-03-27T21:39:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17bd59a640 1 -- 172.21.15.60:0/3379282727 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1126550707 0 0) 0x7f17c00a8e90 con 0x7f17c013e810 2024-03-27T21:39:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17bd59a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:39:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.875+0000 7f17bed9d640 1 --2- 172.21.15.60:0/3379282727 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c013ec50 0x7f17c0141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17be59c640 1 --2- 172.21.15.60:0/3379282727 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c0071a50 0x7f17c00a87e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f17b0009870 tx=0x7f17b002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c37a0c63f5b827b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17bd59a640 1 -- 172.21.15.60:0/3379282727 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b0030840 con 0x7f17c0071a50 2024-03-27T21:39:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17bd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17bd59a640 10 monclient: dump: 2024-03-27T21:39:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:52.875 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17bd59a640 10 monclient: _finish_auth 0 2024-03-27T21:39:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17bd59a640 10 monclient: _check_auth_tickets 2024-03-27T21:39:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17bd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17bd59a640 1 -- 172.21.15.60:0/3379282727 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b00309a0 con 0x7f17c0071a50 2024-03-27T21:39:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17c495c640 5 monclient: authenticate success, global_id 4449 2024-03-27T21:39:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17c495c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17bd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17bd59a640 1 -- 172.21.15.60:0/3379282727 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b0030d30 con 0x7f17c0071a50 2024-03-27T21:39:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17bd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17bd59a640 10 monclient: dump: 2024-03-27T21:39:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:52.876 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17c495c640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17c495c640 4 set_mon_vals no callback set 2024-03-27T21:39:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17c495c640 10 monclient: shutdown 2024-03-27T21:39:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17c495c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17c495c640 1 -- 172.21.15.60:0/3379282727 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c0071a50 msgr2=0x7f17c00a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.876+0000 7f17c495c640 1 --2- 172.21.15.60:0/3379282727 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c0071a50 0x7f17c00a87e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f17b0009870 tx=0x7f17b002b980 comp rx=0 tx=0).stop 2024-03-27T21:39:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.877+0000 7f17c495c640 1 -- 172.21.15.60:0/3379282727 shutdown_connections 2024-03-27T21:39:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.877+0000 7f17c495c640 1 --2- 172.21.15.60:0/3379282727 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c013ec50 0x7f17c0141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.877+0000 7f17c495c640 1 --2- 172.21.15.60:0/3379282727 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c0071a50 0x7f17c00a87e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.877+0000 7f17c495c640 1 -- 172.21.15.60:0/3379282727 >> 172.21.15.60:0/3379282727 conn(0x7f17c00b7300 msgr2=0x7f17c00b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.877+0000 7f17c495c640 1 -- 172.21.15.60:0/3379282727 shutdown_connections 2024-03-27T21:39:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.877+0000 7f17c495c640 1 -- 172.21.15.60:0/3379282727 wait complete. 2024-03-27T21:39:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.877+0000 7f17c495c640 10 monclient: build_initial_monmap 2024-03-27T21:39:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.877+0000 7f17c495c640 10 monclient: monmap: 2024-03-27T21:39:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:52.878 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 1 librados: starting msgr at 2024-03-27T21:39:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 1 librados: starting objecter 2024-03-27T21:39:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 1 Processor -- start 2024-03-27T21:39:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 1 -- start start 2024-03-27T21:39:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 1 librados: setting wanted keys 2024-03-27T21:39:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 1 librados: calling monclient init 2024-03-27T21:39:52.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 10 monclient: init 2024-03-27T21:39:52.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:52.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:39:52.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c0071a50 0x7f17c01373a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 10 monclient(hunting): picked mon.noname-c con 0x7f17c0071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:52.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17c013e810 0x7f17c01d69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 10 monclient(hunting): picked mon.noname-b con 0x7f17c013e810 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:52.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c013ec50 0x7f17c013a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 10 monclient(hunting): picked mon.noname-a con 0x7f17c013ec50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:52.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c01d6270 con 0x7f17c013ec50 2024-03-27T21:39:52.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c00b6cc0 con 0x7f17c013e810 2024-03-27T21:39:52.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c01d64c0 con 0x7f17c0071a50 2024-03-27T21:39:52.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:52.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17c495c640 10 monclient(hunting): authenticate will time out at 834.099813s 2024-03-27T21:39:52.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17be59c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c0071a50 0x7f17c01373a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17bed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c013ec50 0x7f17c013a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.878+0000 7f17be59c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c0071a50 0x7f17c01373a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47798/0 (socket says 172.21.15.60:47798) 2024-03-27T21:39:52.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17be59c640 1 -- 172.21.15.60:0/835082692 learned_addr learned my addr 172.21.15.60:0/835082692 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:52.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17be59c640 10 monclient(hunting): get_auth_request con 0x7f17c0071a50 auth_method 0 2024-03-27T21:39:52.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17be59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17be59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bed9d640 10 monclient(hunting): get_auth_request con 0x7f17c013ec50 auth_method 0 2024-03-27T21:39:52.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bed9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:52.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:52.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:52.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17be59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:52.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17be59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:52.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17be59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:52.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bdd9b640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17c013e810 0x7f17c01d69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bdd9b640 10 monclient(hunting): get_auth_request con 0x7f17c013e810 auth_method 0 2024-03-27T21:39:52.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bed9d640 10 monclient(hunting): handle_auth_done global_id 4455 payload 824 2024-03-27T21:39:52.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bed9d640 1 -- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c0071a50 msgr2=0x7f17c01373a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bed9d640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c0071a50 0x7f17c01373a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bed9d640 1 -- 172.21.15.60:0/835082692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17c013e810 msgr2=0x7f17c01d69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bed9d640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17c013e810 0x7f17c01d69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bed9d640 10 monclient: _finish_hunting 0 2024-03-27T21:39:52.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bed9d640 1 monclient: found mon.noname-a 2024-03-27T21:39:52.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:52.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:52.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bed9d640 1 -- 172.21.15.60:0/835082692 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b8009a70 con 0x7f17c013ec50 2024-03-27T21:39:52.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bed9d640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c013ec50 0x7f17c013a2d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f17b8009db0 tx=0x7f17b800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:52.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17aeffd640 10 client.?.objecter ms_handle_connect 0x7f17c013ec50 2024-03-27T21:39:52.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17aeffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:52.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b800cd30 con 0x7f17c013ec50 2024-03-27T21:39:52.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:52.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17aeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:52.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17aeffd640 10 monclient: dump: 2024-03-27T21:39:52.882 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:52.883 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.883 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.883 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.883 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:52.883 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:52.883 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:52.883 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:52.883 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:52.883 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17aeffd640 10 monclient: _finish_auth 0 2024-03-27T21:39:52.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17aeffd640 10 monclient: _check_auth_tickets 2024-03-27T21:39:52.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:52.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b800f040 con 0x7f17c013ec50 2024-03-27T21:39:52.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:52.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b8003bd0 con 0x7f17c013ec50 2024-03-27T21:39:52.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17c495c640 5 monclient: authenticate success, global_id 4455 2024-03-27T21:39:52.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17c495c640 10 monclient: _renew_subs 2024-03-27T21:39:52.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17c495c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:52.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17c495c640 1 -- 172.21.15.60:0/835082692 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17b00094d0 con 0x7f17c013ec50 2024-03-27T21:39:52.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17bcd99640 4 set_mon_vals no callback set 2024-03-27T21:39:52.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:52.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:52.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17aeffd640 10 monclient: dump: 2024-03-27T21:39:52.884 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:52.884 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.884 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.884 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.884 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:52.884 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:52.884 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:52.885 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:52.885 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:52.885 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17c495c640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:52.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17c495c640 10 monclient: _renew_subs 2024-03-27T21:39:52.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17c495c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:52.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17c495c640 1 -- 172.21.15.60:0/835082692 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c01d7100 con 0x7f17c013ec50 2024-03-27T21:39:52.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.879+0000 7f17c495c640 1 librados: init done 2024-03-27T21:39:52.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.880+0000 7f17c495c640 10 start_mon_command cmd=[{"prefix": "mon dump", "format": "json"}] 2024-03-27T21:39:52.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.880+0000 7f17c495c640 10 monclient: _send_command 1 [{"prefix": "mon dump", "format": "json"}] 2024-03-27T21:39:52.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.880+0000 7f17c495c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:52.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.880+0000 7f17c495c640 1 -- 172.21.15.60:0/835082692 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f17c01d66e0 con 0x7f17c013ec50 2024-03-27T21:39:52.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.881+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f17b8014480 con 0x7f17c013ec50 2024-03-27T21:39:52.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.881+0000 7f17aeffd640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f17a006c590 0x7f17a006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.881+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f17b8089af0 con 0x7f17c013ec50 2024-03-27T21:39:52.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.881+0000 7f17be59c640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f17a006c590 0x7f17a006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.881+0000 7f17aeffd640 10 client.4455.objecter ms_dispatch 0x7f17c0000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:52.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.881+0000 7f17aeffd640 3 client.4455.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:52.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.881+0000 7f17aeffd640 3 client.4455.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:52.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.881+0000 7f17be59c640 10 monclient: get_auth_request con 0x7f17a006c590 auth_method 0 2024-03-27T21:39:52.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.881+0000 7f17aeffd640 20 client.4455.objecter dump_active .. 0 homeless 2024-03-27T21:39:52.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.881+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+1272 (secure 0 0 0) 0x7f17b808e300 con 0x7f17c013ec50 2024-03-27T21:39:52.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.881+0000 7f17aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "mon dump", "format": "json"}] 2024-03-27T21:39:52.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.881+0000 7f17aeffd640 10 monclient: _finish_command 1 = system:0 dumped monmap epoch 1 2024-03-27T21:39:52.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.881+0000 7f17c495c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:52.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.881+0000 7f17c495c640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c013a810 0x7f17c01330c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.881+0000 7f17c495c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.881+0000 7f17c495c640 1 -- 172.21.15.60:0/835082692 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c013abf0 con 0x7f17c013a810 2024-03-27T21:39:52.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.881+0000 7f17aeffd640 10 client.4455.objecter ms_handle_connect 0x7f17a006c590 2024-03-27T21:39:52.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.881+0000 7f17be59c640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f17a006c590 0x7f17a006ea50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f17b0009870 tx=0x7f17b0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:52.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.881+0000 7f17bdd9b640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c013a810 0x7f17c01330c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.881+0000 7f17bdd9b640 10 monclient: get_auth_request con 0x7f17c013a810 auth_method 0 2024-03-27T21:39:52.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17bdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:52.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:52.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:52.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_done global_id 4458 payload 824 2024-03-27T21:39:52.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17bdd9b640 1 -- 172.21.15.60:0/835082692 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f17c013f0c0 con 0x7f17c013a810 2024-03-27T21:39:52.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17aeffd640 10 client.4455.objecter ms_handle_connect 0x7f17c013a810 2024-03-27T21:39:52.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17bdd9b640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c013a810 0x7f17c01330c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f17c0139570 tx=0x7f17b400e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:52.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17aeffd640 10 client.4455.objecter resend_mon_ops 2024-03-27T21:39:52.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b40052e0 con 0x7f17c013a810 2024-03-27T21:39:52.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:52.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:52.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17aeffd640 10 monclient: dump: 2024-03-27T21:39:52.888 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:52.888 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.888 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.888 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.888 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:52.888 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:52.888 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:52.888 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:52.888 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:52.888 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 <== mon.0 v2:172.21.15.60:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f17b4005440 con 0x7f17c013a810 2024-03-27T21:39:52.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17aeffd640 10 monclient: handle_command_reply 2 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:52.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17aeffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:39:52.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c013a810 msgr2=0x7f17c01330c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17aeffd640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c013a810 0x7f17c01330c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f17c0139570 tx=0x7f17b400e960 comp rx=0 tx=0).stop 2024-03-27T21:39:52.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c013a810 msgr2=0x7f17c01330c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-27T21:39:52.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.882+0000 7f17aeffd640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c013a810 0x7f17c01330c0 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f17c0139570 tx=0x7f17b400e960 comp rx=0 tx=0).stop 2024-03-27T21:39:52.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.888+0000 7f17c495c640 10 start_mon_command cmd=[{"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}] 2024-03-27T21:39:52.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.888+0000 7f17c495c640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c0133600 0x7f17c01d9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.888+0000 7f17c495c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.888+0000 7f17c495c640 1 -- 172.21.15.60:0/835082692 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c0073e90 con 0x7f17c0133600 2024-03-27T21:39:52.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.888+0000 7f17bdd9b640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c0133600 0x7f17c01d9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.888+0000 7f17bdd9b640 10 monclient: get_auth_request con 0x7f17c0133600 auth_method 0 2024-03-27T21:39:52.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.888+0000 7f17bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.888+0000 7f17bdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.888+0000 7f17bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.889+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:52.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.889+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:52.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.889+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:52.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.889+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_done global_id 4461 payload 824 2024-03-27T21:39:52.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.889+0000 7f17bdd9b640 1 -- 172.21.15.60:0/835082692 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7f17c00aa590 con 0x7f17c0133600 2024-03-27T21:39:52.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.889+0000 7f17aeffd640 10 client.4455.objecter ms_handle_connect 0x7f17c0133600 2024-03-27T21:39:52.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.889+0000 7f17bdd9b640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c0133600 0x7f17c01d9770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f17c01d6870 tx=0x7f17b4001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.889+0000 7f17aeffd640 10 client.4455.objecter resend_mon_ops 2024-03-27T21:39:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.889+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b401d880 con 0x7f17c0133600 2024-03-27T21:39:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.889+0000 7f17aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.889+0000 7f17aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.889+0000 7f17aeffd640 10 monclient: dump: 2024-03-27T21:39:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:52.892 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.890+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 <== mon.0 v2:172.21.15.60:3300/0 2 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (secure 0 0 0) 0x7f17b401d9e0 con 0x7f17c0133600 2024-03-27T21:39:52.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.890+0000 7f17aeffd640 10 monclient: handle_command_reply 3 [{"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}] 2024-03-27T21:39:52.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.890+0000 7f17aeffd640 10 monclient: _finish_command 3 = system:0 mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-27T21:39:52.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.890+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c0133600 msgr2=0x7f17c01d9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.890+0000 7f17aeffd640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c0133600 0x7f17c01d9770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f17c01d6870 tx=0x7f17b4001080 comp rx=0 tx=0).stop 2024-03-27T21:39:52.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.890+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c0133600 msgr2=0x7f17c01d9770 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-27T21:39:52.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.890+0000 7f17aeffd640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c0133600 0x7f17c01d9770 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f17c01d6870 tx=0x7f17b4001080 comp rx=0 tx=0).stop 2024-03-27T21:39:52.893 INFO:teuthology.orchestra.run.smithi060.stdout:mon.a: {} 2024-03-27T21:39:52.896 INFO:teuthology.orchestra.run.smithi060.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-27T21:39:52.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.890+0000 7f17c495c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:52.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.890+0000 7f17c495c640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17c01d9cb0 0x7f17c01da0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.890+0000 7f17c495c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.890+0000 7f17c495c640 1 -- 172.21.15.60:0/835082692 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c014bde0 con 0x7f17c01d9cb0 2024-03-27T21:39:52.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.891+0000 7f17bdd9b640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17c01d9cb0 0x7f17c01da0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.891+0000 7f17bdd9b640 10 monclient: get_auth_request con 0x7f17c01d9cb0 auth_method 0 2024-03-27T21:39:52.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.891+0000 7f17bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.891+0000 7f17bdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.891+0000 7f17bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.891+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:52.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.891+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:52.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.891+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:52.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.891+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_done global_id 4435 payload 824 2024-03-27T21:39:52.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.891+0000 7f17bdd9b640 1 -- 172.21.15.60:0/835082692 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f17c01da600 con 0x7f17c01d9cb0 2024-03-27T21:39:52.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.891+0000 7f17bdd9b640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17c01d9cb0 0x7f17c01da0c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f17b400f010 tx=0x7f17b4009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:52.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.891+0000 7f17aeffd640 10 client.4455.objecter ms_handle_connect 0x7f17c01d9cb0 2024-03-27T21:39:52.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.891+0000 7f17aeffd640 10 client.4455.objecter resend_mon_ops 2024-03-27T21:39:52.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.891+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b4028a00 con 0x7f17c01d9cb0 2024-03-27T21:39:52.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.891+0000 7f17aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:52.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.891+0000 7f17aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:52.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.891+0000 7f17aeffd640 10 monclient: dump: 2024-03-27T21:39:52.897 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:52.897 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.898 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.898 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.898 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:52.898 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:52.898 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:52.898 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:52.898 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:52.898 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.892+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f17b4028b60 con 0x7f17c01d9cb0 2024-03-27T21:39:52.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.892+0000 7f17aeffd640 10 monclient: handle_command_reply 4 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:52.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.892+0000 7f17aeffd640 10 monclient: _finish_command 4 = system:0 2024-03-27T21:39:52.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.892+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17c01d9cb0 msgr2=0x7f17c01da0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.892+0000 7f17aeffd640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17c01d9cb0 0x7f17c01da0c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f17b400f010 tx=0x7f17b4009040 comp rx=0 tx=0).stop 2024-03-27T21:39:52.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.892+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17c01d9cb0 msgr2=0x7f17c01da0c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-27T21:39:52.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.892+0000 7f17aeffd640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17c01d9cb0 0x7f17c01da0c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:52.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.892+0000 7f17bed9d640 1 -- 172.21.15.60:0/835082692 reap_dead start 2024-03-27T21:39:52.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.898+0000 7f17c495c640 10 start_mon_command cmd=[{"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}] 2024-03-27T21:39:52.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.898+0000 7f17c495c640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17c01373a0 0x7f17c01d69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.898+0000 7f17c495c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.898+0000 7f17c495c640 1 -- 172.21.15.60:0/835082692 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c0137780 con 0x7f17c01373a0 2024-03-27T21:39:52.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.898+0000 7f17bdd9b640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17c01373a0 0x7f17c01d69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.898+0000 7f17bdd9b640 10 monclient: get_auth_request con 0x7f17c01373a0 auth_method 0 2024-03-27T21:39:52.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.898+0000 7f17bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.898+0000 7f17bdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.898+0000 7f17bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.898+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:52.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.898+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:52.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.898+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:52.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.898+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_done global_id 4438 payload 824 2024-03-27T21:39:52.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.899+0000 7f17bdd9b640 1 -- 172.21.15.60:0/835082692 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7f17c0069470 con 0x7f17c01373a0 2024-03-27T21:39:52.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.899+0000 7f17bdd9b640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17c01373a0 0x7f17c01d69e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f17b4001eb0 tx=0x7f17b4023000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:52.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.899+0000 7f17aeffd640 10 client.4455.objecter ms_handle_connect 0x7f17c01373a0 2024-03-27T21:39:52.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.899+0000 7f17aeffd640 10 client.4455.objecter resend_mon_ops 2024-03-27T21:39:52.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.899+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b4033d70 con 0x7f17c01373a0 2024-03-27T21:39:52.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.899+0000 7f17aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:52.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.899+0000 7f17aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:52.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.899+0000 7f17aeffd640 10 monclient: dump: 2024-03-27T21:39:52.900 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:52.900 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.900 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.900 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.900 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:52.901 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:52.901 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:52.901 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:52.901 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:52.901 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.900+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (secure 0 0 0) 0x7f17b4041070 con 0x7f17c01373a0 2024-03-27T21:39:52.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.900+0000 7f17aeffd640 10 monclient: handle_command_reply 5 [{"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}] 2024-03-27T21:39:52.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.900+0000 7f17aeffd640 10 monclient: _finish_command 5 = system:0 mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-27T21:39:52.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.900+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17c01373a0 msgr2=0x7f17c01d69e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.900+0000 7f17aeffd640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17c01373a0 0x7f17c01d69e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f17b4001eb0 tx=0x7f17b4023000 comp rx=0 tx=0).stop 2024-03-27T21:39:52.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.900+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17c01373a0 msgr2=0x7f17c01d69e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-27T21:39:52.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.900+0000 7f17aeffd640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17c01373a0 0x7f17c01d69e0 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f17b4001eb0 tx=0x7f17b4023000 comp rx=0 tx=0).stop 2024-03-27T21:39:52.902 INFO:teuthology.orchestra.run.smithi060.stdout:mon.b: {} 2024-03-27T21:39:52.904 INFO:teuthology.orchestra.run.smithi060.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-27T21:39:52.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.900+0000 7f17c495c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:52.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.900+0000 7f17c495c640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c0065a90 0x7f17c0065ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.900+0000 7f17c495c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.900+0000 7f17c495c640 1 -- 172.21.15.60:0/835082692 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c01339e0 con 0x7f17c0065a90 2024-03-27T21:39:52.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17bdd9b640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c0065a90 0x7f17c0065ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17bdd9b640 10 monclient: get_auth_request con 0x7f17c0065a90 auth_method 0 2024-03-27T21:39:52.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17bdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:52.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:52.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:52.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_done global_id 4388 payload 824 2024-03-27T21:39:52.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17bdd9b640 1 -- 172.21.15.60:0/835082692 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f17c0069dd0 con 0x7f17c0065a90 2024-03-27T21:39:52.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17bdd9b640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c0065a90 0x7f17c0065ea0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f17b400e270 tx=0x7f17b4034000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:52.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17aeffd640 10 client.4455.objecter ms_handle_connect 0x7f17c0065a90 2024-03-27T21:39:52.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17aeffd640 10 client.4455.objecter resend_mon_ops 2024-03-27T21:39:52.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b404f470 con 0x7f17c0065a90 2024-03-27T21:39:52.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:52.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:52.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17aeffd640 10 monclient: dump: 2024-03-27T21:39:52.906 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:52.906 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.906 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.906 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.906 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:52.906 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:52.906 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:52.906 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:52.907 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:52.907 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 <== mon.2 v2:172.21.15.60:3301/0 2 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f17b404f5d0 con 0x7f17c0065a90 2024-03-27T21:39:52.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17aeffd640 10 monclient: handle_command_reply 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:52.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17aeffd640 10 monclient: _finish_command 6 = system:0 2024-03-27T21:39:52.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c0065a90 msgr2=0x7f17c0065ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17aeffd640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c0065a90 0x7f17c0065ea0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f17b400e270 tx=0x7f17b4034000 comp rx=0 tx=0).stop 2024-03-27T21:39:52.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c0065a90 msgr2=0x7f17c0065ea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-27T21:39:52.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.901+0000 7f17aeffd640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c0065a90 0x7f17c0065ea0 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f17b400e270 tx=0x7f17b4034000 comp rx=0 tx=0).stop 2024-03-27T21:39:52.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.907+0000 7f17c495c640 10 start_mon_command cmd=[{"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}] 2024-03-27T21:39:52.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.907+0000 7f17c495c640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c00663e0 0x7f17c01e8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:52.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.907+0000 7f17c495c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:52.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.907+0000 7f17c495c640 1 -- 172.21.15.60:0/835082692 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c0139cb0 con 0x7f17c00663e0 2024-03-27T21:39:52.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.907+0000 7f17bdd9b640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c00663e0 0x7f17c01e8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:52.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.907+0000 7f17bdd9b640 10 monclient: get_auth_request con 0x7f17c00663e0 auth_method 0 2024-03-27T21:39:52.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.907+0000 7f17bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:52.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.907+0000 7f17bdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:52.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.907+0000 7f17bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:52.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.907+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:52.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.907+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:52.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.907+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:52.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.908+0000 7f17bdd9b640 10 monclient(hunting): handle_auth_done global_id 4391 payload 824 2024-03-27T21:39:52.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.908+0000 7f17bdd9b640 1 -- 172.21.15.60:0/835082692 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7f17c0139e70 con 0x7f17c00663e0 2024-03-27T21:39:52.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.908+0000 7f17bdd9b640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c00663e0 0x7f17c01e8ae0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f17b4003d50 tx=0x7f17b4009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:52.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.908+0000 7f17aeffd640 10 client.4455.objecter ms_handle_connect 0x7f17c00663e0 2024-03-27T21:39:52.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.908+0000 7f17aeffd640 10 client.4455.objecter resend_mon_ops 2024-03-27T21:39:52.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.908+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b404fc10 con 0x7f17c00663e0 2024-03-27T21:39:52.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.908+0000 7f17aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:52.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.908+0000 7f17aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:52.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.908+0000 7f17aeffd640 10 monclient: dump: 2024-03-27T21:39:52.909 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:52.909 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:52.909 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.909 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:52.909 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:52.909 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:52.909 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:52.909 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:52.909 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:52.909 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:52.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.909+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 <== mon.2 v2:172.21.15.60:3301/0 2 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (secure 0 0 0) 0x7f17b404fad0 con 0x7f17c00663e0 2024-03-27T21:39:52.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.909+0000 7f17aeffd640 10 monclient: handle_command_reply 7 [{"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}] 2024-03-27T21:39:52.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.909+0000 7f17aeffd640 10 monclient: _finish_command 7 = system:0 mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-27T21:39:52.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.909+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c00663e0 msgr2=0x7f17c01e8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.909+0000 7f17aeffd640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c00663e0 0x7f17c01e8ae0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f17b4003d50 tx=0x7f17b4009040 comp rx=0 tx=0).stop 2024-03-27T21:39:52.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.909+0000 7f17aeffd640 1 -- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c00663e0 msgr2=0x7f17c01e8ae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-27T21:39:52.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.909+0000 7f17aeffd640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f17c00663e0 0x7f17c01e8ae0 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f17b4003d50 tx=0x7f17b4009040 comp rx=0 tx=0).stop 2024-03-27T21:39:52.910 INFO:teuthology.orchestra.run.smithi060.stdout:mon.c: {} 2024-03-27T21:39:52.912 INFO:teuthology.orchestra.run.smithi060.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-27T21:39:52.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.909+0000 7f17c495c640 10 librados: watch_flush enter 2024-03-27T21:39:52.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.909+0000 7f17c495c640 10 librados: watch_flush exit 2024-03-27T21:39:52.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.909+0000 7f17c495c640 20 client.4455.objecter shutdown clearing up homeless session... 2024-03-27T21:39:52.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.909+0000 7f17c495c640 10 client.4455.objecter successfully canceled tick 2024-03-27T21:39:52.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.909+0000 7f17c495c640 1 -- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f17a006c590 msgr2=0x7f17a006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.909+0000 7f17c495c640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f17a006c590 0x7f17a006ea50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f17b0009870 tx=0x7f17b0008030 comp rx=0 tx=0).stop 2024-03-27T21:39:52.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.909+0000 7f17c495c640 10 monclient: shutdown 2024-03-27T21:39:52.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.909+0000 7f17c495c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:52.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.909+0000 7f17c495c640 1 -- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c013ec50 msgr2=0x7f17c013a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:52.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.909+0000 7f17c495c640 1 --2- 172.21.15.60:0/835082692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c013ec50 0x7f17c013a2d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f17b8009db0 tx=0x7f17b800c490 comp rx=0 tx=0).stop 2024-03-27T21:39:52.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.909+0000 7f17bed9d640 1 -- 172.21.15.60:0/835082692 reap_dead start 2024-03-27T21:39:52.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.909+0000 7f17c495c640 1 -- 172.21.15.60:0/835082692 shutdown_connections 2024-03-27T21:39:52.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.909+0000 7f17c495c640 1 -- 172.21.15.60:0/835082692 >> 172.21.15.60:0/835082692 conn(0x7f17c00b7300 msgr2=0x7f17c013b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:52.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.910+0000 7f17c495c640 1 -- 172.21.15.60:0/835082692 shutdown_connections 2024-03-27T21:39:52.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.910+0000 7f17c495c640 1 -- 172.21.15.60:0/835082692 wait complete. 2024-03-27T21:39:52.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:52.910+0000 7f17c495c640 1 librados: shutdown 2024-03-27T21:39:52.913 DEBUG:teuthology.orchestra.run.smithi191:> type ceph-objectstore-tool 2024-03-27T21:39:52.931 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-03-27T21:39:52.931 DEBUG:teuthology.orchestra.run.smithi060:> type ceph-objectstore-tool 2024-03-27T21:39:52.945 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-03-27T21:39:52.945 INFO:teuthology.run_tasks:Running task workunit... 2024-03-27T21:39:52.957 INFO:tasks.workunit:Pulling workunits from ref 7c37d80f7a88f4eb561c4fe02558660a650b10ac 2024-03-27T21:39:52.958 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-27T21:39:52.958 DEBUG:teuthology.orchestra.run.smithi060:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-27T21:39:52.958 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-27T21:39:52.958 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-27T21:39:52.958 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T21:39:52.959 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-27T21:39:52.959 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-27T21:39:52.959 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T21:39:52.959 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-27T21:39:52.959 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-27T21:39:52.959 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:39:52.960 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-27T21:39:52.960 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:39:52.960 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-27T21:39:52.960 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-27T21:39:52.987 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:39:53.000+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:39:53.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T21:39:53.000 INFO:teuthology.orchestra.run.smithi060.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-27T21:39:53.001 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-27T21:39:53.010 INFO:tasks.ceph.osd.2:Stopped 2024-03-27T21:39:53.010 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-03-27T21:39:53.010 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi060.front.sepia.ceph.com 2024-03-27T21:39:53.010 DEBUG:teuthology.orchestra.run.smithi060:> type ceph-objectstore-tool 2024-03-27T21:39:53.066 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-27T21:39:53.066 DEBUG:teuthology.orchestra.run.smithi060:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-27T21:39:53.067 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:39:53.080+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:39:53.068 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-03-27T21:39:53.069 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-03-27T21:39:53.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.103+0000 7f182f750640 10 monclient: get_monmap_and_config 2024-03-27T21:39:53.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.103+0000 7f182f750640 10 monclient: build_initial_monmap 2024-03-27T21:39:53.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.103+0000 7f182f750640 10 monclient: monmap: 2024-03-27T21:39:53.089 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:53.089 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:53.089 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:53.104118+0000 2024-03-27T21:39:53.089 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:53.104118+0000 2024-03-27T21:39:53.090 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:53.090 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:53.090 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:53.090 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:53.090 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:53.090 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:53.090 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:53.090 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:53.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.103+0000 7f182f750640 1 Processor -- start 2024-03-27T21:39:53.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182f750640 1 -- start start 2024-03-27T21:39:53.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182f750640 10 monclient: init 2024-03-27T21:39:53.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182f750640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:53.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182f750640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-03-27T21:39:53.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182f750640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f182806f900 0x7f182813fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:53.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182f750640 10 monclient(hunting): picked mon.noname-a con 0x7f182806f900 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:53.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182f750640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f182806fce0 0x7f18280af850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:53.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182f750640 10 monclient(hunting): picked mon.noname-c con 0x7f182806fce0 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:53.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182f750640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1828138bf0 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:53.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182f750640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1828071b50 con 0x7f1828138bf0 2024-03-27T21:39:53.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182f750640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:53.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182f750640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1828140100 con 0x7f182806f900 2024-03-27T21:39:53.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182f750640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:53.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182f750640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f182813f690 con 0x7f182806fce0 2024-03-27T21:39:53.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182f750640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:53.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182f750640 10 monclient(hunting): authenticate will time out at 834.325646s 2024-03-27T21:39:53.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182d4ec640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f182806f900 0x7f182813fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:53.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182d4ec640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f182806f900 0x7f182813fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44404/0 (socket says 172.21.15.60:44404) 2024-03-27T21:39:53.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182d4ec640 1 -- 172.21.15.60:0/1150148765 learned_addr learned my addr 172.21.15.60:0/1150148765 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:53.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182d4ec640 10 monclient(hunting): get_auth_request con 0x7f182806f900 auth_method 0 2024-03-27T21:39:53.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182d4ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:53.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182d4ec640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:53.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182d4ec640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:53.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182d4ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:53.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182d4ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:53.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182d4ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:53.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.104+0000 7f182cceb640 1 --2- 172.21.15.60:0/1150148765 >> v2:172.21.15.191:3300/0 conn(0x7f182806fce0 0x7f18280af850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:53.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.105+0000 7f182cceb640 10 monclient(hunting): get_auth_request con 0x7f182806fce0 auth_method 0 2024-03-27T21:39:53.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.105+0000 7f182cceb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:53.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.105+0000 7f182cceb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:53.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.105+0000 7f182cceb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:53.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.105+0000 7f182d4ec640 10 monclient(hunting): handle_auth_done global_id 4464 payload 274 2024-03-27T21:39:53.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.105+0000 7f182d4ec640 1 -- 172.21.15.60:0/1150148765 >> v2:172.21.15.191:3300/0 conn(0x7f182806fce0 msgr2=0x7f18280af850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:53.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.105+0000 7f182d4ec640 1 --2- 172.21.15.60:0/1150148765 >> v2:172.21.15.191:3300/0 conn(0x7f182806fce0 0x7f18280af850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:53.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.105+0000 7f182d4ec640 1 -- 172.21.15.60:0/1150148765 >> v1:172.21.15.191:6789/0 conn(0x7f1828138bf0 legacy=0x7f1828139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:53.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.105+0000 7f182d4ec640 10 monclient: _finish_hunting 0 2024-03-27T21:39:53.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.105+0000 7f182d4ec640 1 monclient: found mon.noname-a 2024-03-27T21:39:53.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.105+0000 7f182d4ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:53.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.105+0000 7f182d4ec640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:53.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.105+0000 7f182d4ec640 1 -- 172.21.15.60:0/1150148765 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f182806f130 con 0x7f182806f900 2024-03-27T21:39:53.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.105+0000 7f182d4ec640 1 --2- 172.21.15.60:0/1150148765 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f182806f900 0x7f182813fbc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f182806ce10 tx=0x7f182402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=539c3392198bfa90 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:53.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.106+0000 7f181ffff640 1 -- 172.21.15.60:0/1150148765 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1824031640 con 0x7f182806f900 2024-03-27T21:39:53.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.106+0000 7f181ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:53.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.106+0000 7f181ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:53.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.106+0000 7f181ffff640 10 monclient: dump: 2024-03-27T21:39:53.097 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:53.097 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:53.097 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:53.097 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:53.097 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:53.097 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:53.097 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:53.097 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:53.097 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:53.097 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:53.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.106+0000 7f181ffff640 10 monclient: _finish_auth 0 2024-03-27T21:39:53.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.106+0000 7f181ffff640 10 monclient: _check_auth_tickets 2024-03-27T21:39:53.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.106+0000 7f181ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:53.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.106+0000 7f181ffff640 1 -- 172.21.15.60:0/1150148765 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18240317a0 con 0x7f182806f900 2024-03-27T21:39:53.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.106+0000 7f182f750640 5 monclient: authenticate success, global_id 4464 2024-03-27T21:39:53.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.106+0000 7f182f750640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:53.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.106+0000 7f181ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:53.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.106+0000 7f182f750640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:53.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.106+0000 7f181ffff640 1 -- 172.21.15.60:0/1150148765 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1824031b30 con 0x7f182806f900 2024-03-27T21:39:53.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.106+0000 7f182f750640 4 set_mon_vals no callback set 2024-03-27T21:39:53.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.106+0000 7f181ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:53.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.106+0000 7f181ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:53.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.106+0000 7f181ffff640 10 monclient: dump: 2024-03-27T21:39:53.098 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:53.098 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:53.098 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:53.098 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:53.098 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:53.099 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:53.099 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:53.099 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:53.099 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:53.099 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:53.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.106+0000 7fd21d09d640 10 monclient: get_monmap_and_config 2024-03-27T21:39:53.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.106+0000 7fd21d09d640 10 monclient: build_initial_monmap 2024-03-27T21:39:53.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.106+0000 7fd21d09d640 10 monclient: monmap: 2024-03-27T21:39:53.099 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:53.099 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:53.100 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:53.107838+0000 2024-03-27T21:39:53.100 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:53.107838+0000 2024-03-27T21:39:53.100 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:53.100 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:53.100 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:53.100 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:53.100 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:53.100 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:53.100 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:53.100 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:53.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.107+0000 7f182f750640 10 monclient: shutdown 2024-03-27T21:39:53.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.107+0000 7f182f750640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:53.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.107+0000 7f182f750640 1 -- 172.21.15.60:0/1150148765 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f182806f900 msgr2=0x7f182813fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:53.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.107+0000 7f182f750640 1 --2- 172.21.15.60:0/1150148765 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f182806f900 0x7f182813fbc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f182806ce10 tx=0x7f182402b8f0 comp rx=0 tx=0).stop 2024-03-27T21:39:53.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.107+0000 7f182f750640 1 -- 172.21.15.60:0/1150148765 shutdown_connections 2024-03-27T21:39:53.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.107+0000 7f182f750640 1 --2- 172.21.15.60:0/1150148765 >> v2:172.21.15.191:3300/0 conn(0x7f182806fce0 0x7f18280af850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:53.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.107+0000 7f182f750640 1 --2- 172.21.15.60:0/1150148765 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f182806f900 0x7f182813fbc0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:53.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.107+0000 7f182f750640 1 -- 172.21.15.60:0/1150148765 >> 172.21.15.60:0/1150148765 conn(0x7f18280b4e50 msgr2=0x7f18280b5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:53.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.107+0000 7f182f750640 1 -- 172.21.15.60:0/1150148765 shutdown_connections 2024-03-27T21:39:53.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.107+0000 7f182f750640 1 -- 172.21.15.60:0/1150148765 wait complete. 2024-03-27T21:39:53.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.107+0000 7fd21d09d640 1 Processor -- start 2024-03-27T21:39:53.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.107+0000 7fd21d09d640 1 -- start start 2024-03-27T21:39:53.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.107+0000 7fd21d09d640 10 monclient: init 2024-03-27T21:39:53.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7f182f750640 10 monclient: build_initial_monmap 2024-03-27T21:39:53.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7f182f750640 10 monclient: monmap: 2024-03-27T21:39:53.103 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:53.103 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:53.103 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:53.103 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:53.103 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:53.103 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:53.103 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:53.103 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:53.103 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:53.103 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:53.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7f182f750640 1 librados: starting msgr at 2024-03-27T21:39:53.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7f182f750640 1 librados: starting objecter 2024-03-27T21:39:53.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21d09d640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:53.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21d09d640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-03-27T21:39:53.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21d09d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd21806f900 addr v1:172.21.15.191:6789/0 2024-03-27T21:39:53.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21d09d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd21806fce0 0x7fd21813fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:53.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21d09d640 10 monclient(hunting): picked mon.noname-c con 0x7fd21806fce0 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:53.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21d09d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd218138bf0 0x7fd218139060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:53.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21d09d640 10 monclient(hunting): picked mon.noname-a con 0x7fd218138bf0 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:53.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21d09d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd218071b50 con 0x7fd21806f900 2024-03-27T21:39:53.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21d09d640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:53.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21d09d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd218140100 con 0x7fd218138bf0 2024-03-27T21:39:53.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21d09d640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:53.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21d09d640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd21813f690 con 0x7fd21806fce0 2024-03-27T21:39:53.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21d09d640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:53.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21d09d640 10 monclient(hunting): authenticate will time out at 834.329493s 2024-03-27T21:39:53.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7f182f750640 1 Processor -- start 2024-03-27T21:39:53.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7f182f750640 1 -- start start 2024-03-27T21:39:53.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7f182f750640 1 librados: setting wanted keys 2024-03-27T21:39:53.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7f182f750640 1 librados: calling monclient init 2024-03-27T21:39:53.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21759e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd218138bf0 0x7fd218139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:53.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21759e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd218138bf0 0x7fd218139060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44408/0 (socket says 172.21.15.60:44408) 2024-03-27T21:39:53.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21759e640 1 -- 172.21.15.60:0/3528385330 learned_addr learned my addr 172.21.15.60:0/3528385330 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:53.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21759e640 10 monclient(hunting): get_auth_request con 0x7fd218138bf0 auth_method 0 2024-03-27T21:39:53.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:53.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21759e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:53.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21759e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:53.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7f182f750640 10 monclient: init 2024-03-27T21:39:53.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21659c640 1 --2- 172.21.15.60:0/3528385330 >> v2:172.21.15.191:3300/0 conn(0x7fd21806fce0 0x7fd21813fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:53.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21659c640 10 monclient(hunting): get_auth_request con 0x7fd21806fce0 auth_method 0 2024-03-27T21:39:53.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:53.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21659c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:53.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.108+0000 7fd21659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7fd21659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7fd215d9b640 1 -- 172.21.15.60:0/3528385330 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560539952 0 0) 0x7fd218071b50 con 0x7fd21806f900 2024-03-27T21:39:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7fd215d9b640 10 monclient(hunting): _negotiate 2024-03-27T21:39:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7fd215d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7fd215d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7fd215d9b640 10 monclient(hunting): my global_id is 4447 2024-03-27T21:39:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7fd215d9b640 1 -- 172.21.15.60:0/3528385330 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd200003520 con 0x7fd21806f900 2024-03-27T21:39:53.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182f750640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182f750640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:39:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182f750640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f182806fce0 0x7f182813d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182f750640 10 monclient(hunting): picked mon.noname-c con 0x7f182806fce0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182f750640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1828138bf0 0x7f182813c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182f750640 10 monclient(hunting): picked mon.noname-a con 0x7f1828138bf0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182cceb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1828138bf0 0x7f182813c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182cceb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1828138bf0 0x7f182813c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44424/0 (socket says 172.21.15.60:44424) 2024-03-27T21:39:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182f750640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1828133970 0x7f1828136720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182d4ec640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f182806fce0 0x7f182813d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182cceb640 1 -- 172.21.15.60:0/702603751 learned_addr learned my addr 172.21.15.60:0/702603751 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182f750640 10 monclient(hunting): picked mon.noname-b con 0x7f1828133970 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182f750640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:53.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7fd21659c640 10 monclient(hunting): handle_auth_done global_id 4444 payload 274 2024-03-27T21:39:53.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7fd21659c640 1 -- 172.21.15.60:0/3528385330 >> v2:172.21.15.60:3300/0 conn(0x7fd218138bf0 msgr2=0x7fd218139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:53.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7fd21659c640 1 --2- 172.21.15.60:0/3528385330 >> v2:172.21.15.60:3300/0 conn(0x7fd218138bf0 0x7fd218139060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:53.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7fd21659c640 1 -- 172.21.15.60:0/3528385330 >> v1:172.21.15.191:6789/0 conn(0x7fd21806f900 legacy=0x7fd2180af850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:53.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7fd21659c640 10 monclient: _finish_hunting 0 2024-03-27T21:39:53.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7fd21659c640 1 monclient: found mon.noname-c 2024-03-27T21:39:53.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182f750640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182813d790 con 0x7f1828138bf0 2024-03-27T21:39:53.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7fd21659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:53.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7fd21659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:39:53.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7fd21659c640 1 -- 172.21.15.60:0/3528385330 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2181395a0 con 0x7fd21806fce0 2024-03-27T21:39:53.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7fd21759e640 1 --2- 172.21.15.60:0/3528385330 >> v2:172.21.15.60:3300/0 conn(0x7fd218138bf0 0x7fd218139060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:53.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182f750640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:53.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182f750640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182813eb40 con 0x7f1828133970 2024-03-27T21:39:53.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182f750640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:53.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182f750640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182813ee20 con 0x7f182806fce0 2024-03-27T21:39:53.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182f750640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:53.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182f750640 10 monclient(hunting): authenticate will time out at 834.330857s 2024-03-27T21:39:53.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182d4ec640 10 monclient(hunting): get_auth_request con 0x7f182806fce0 auth_method 0 2024-03-27T21:39:53.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182d4ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:53.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182d4ec640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:53.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.109+0000 7f182d4ec640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:53.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182cceb640 10 monclient(hunting): get_auth_request con 0x7f1828138bf0 auth_method 0 2024-03-27T21:39:53.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182cceb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:53.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182cceb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:53.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182cceb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:53.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182dced640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1828133970 0x7f1828136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:53.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182dced640 10 monclient(hunting): get_auth_request con 0x7f1828133970 auth_method 0 2024-03-27T21:39:53.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182dced640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:53.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182dced640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:53.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182dced640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:53.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182d4ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:53.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182d4ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:53.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182d4ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:53.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182cceb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182cceb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182cceb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182dced640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182dced640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182dced640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182d4ec640 10 monclient(hunting): handle_auth_done global_id 4394 payload 824 2024-03-27T21:39:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182d4ec640 1 -- 172.21.15.60:0/702603751 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1828133970 msgr2=0x7f1828136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182d4ec640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1828133970 0x7f1828136720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182d4ec640 1 -- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1828138bf0 msgr2=0x7f182813c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182d4ec640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1828138bf0 0x7f182813c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182d4ec640 10 monclient: _finish_hunting 0 2024-03-27T21:39:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182d4ec640 1 monclient: found mon.noname-c 2024-03-27T21:39:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182d4ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182d4ec640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:39:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182d4ec640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18240094d0 con 0x7f182806fce0 2024-03-27T21:39:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182cceb640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1828138bf0 0x7f182813c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:53.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7fd21659c640 1 --2- 172.21.15.60:0/3528385330 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd21806fce0 0x7fd21813fbc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd21813cf80 tx=0x7fd20c02fce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbe9e14d1fe8e8e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:53.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f181dffb640 10 client.?.objecter ms_handle_connect 0x7f182806fce0 2024-03-27T21:39:53.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f181dffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:53.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f182d4ec640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f182806fce0 0x7f182813d250 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1824033000 tx=0x7f18240046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7fd215d9b640 1 -- 172.21.15.60:0/3528385330 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20c030a90 con 0x7fd21806fce0 2024-03-27T21:39:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7fd215d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7fd215d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7fd215d9b640 10 monclient: dump: 2024-03-27T21:39:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:53.120 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7fd215d9b640 10 monclient: _finish_auth 0 2024-03-27T21:39:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7fd215d9b640 10 monclient: _check_auth_tickets 2024-03-27T21:39:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7fd215d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7fd215d9b640 1 -- 172.21.15.60:0/3528385330 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20c030bf0 con 0x7fd21806fce0 2024-03-27T21:39:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7fd215d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7fd215d9b640 1 -- 172.21.15.60:0/3528385330 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20c038440 con 0x7fd21806fce0 2024-03-27T21:39:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7fd215d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7fd215d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:39:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7fd215d9b640 10 monclient: dump: 2024-03-27T21:39:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:53.121 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7fd21d09d640 5 monclient: authenticate success, global_id 4444 2024-03-27T21:39:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7fd21d09d640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7fd21d09d640 4 set_mon_vals no callback set 2024-03-27T21:39:53.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182403ac60 con 0x7f182806fce0 2024-03-27T21:39:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.110+0000 7f181dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7f181dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7f181dffb640 10 monclient: dump: 2024-03-27T21:39:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:53.123 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7f181dffb640 10 monclient: _finish_auth 0 2024-03-27T21:39:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7f181dffb640 10 monclient: _check_auth_tickets 2024-03-27T21:39:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7f181dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182403adc0 con 0x7f182806fce0 2024-03-27T21:39:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7f181dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1824044630 con 0x7f182806fce0 2024-03-27T21:39:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7f181dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7f181dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7f181dffb640 10 monclient: dump: 2024-03-27T21:39:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:53.125 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7f182f750640 5 monclient: authenticate success, global_id 4394 2024-03-27T21:39:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7f182f750640 10 monclient: _renew_subs 2024-03-27T21:39:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7f182f750640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7f182f750640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1828133e00 con 0x7f182806fce0 2024-03-27T21:39:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7f181f7fe640 4 set_mon_vals no callback set 2024-03-27T21:39:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7f182f750640 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7f182f750640 10 monclient: _renew_subs 2024-03-27T21:39:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7fd21d09d640 10 monclient: shutdown 2024-03-27T21:39:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7fd21d09d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7fd21d09d640 1 -- 172.21.15.60:0/3528385330 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd21806fce0 msgr2=0x7fd21813fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7fd21d09d640 1 --2- 172.21.15.60:0/3528385330 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd21806fce0 0x7fd21813fbc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd21813cf80 tx=0x7fd20c02fce0 comp rx=0 tx=0).stop 2024-03-27T21:39:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7f182f750640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.112+0000 7fd21d09d640 1 -- 172.21.15.60:0/3528385330 shutdown_connections 2024-03-27T21:39:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.112+0000 7fd21d09d640 1 --2- 172.21.15.60:0/3528385330 >> v2:172.21.15.60:3300/0 conn(0x7fd218138bf0 0x7fd218139060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.112+0000 7fd21d09d640 1 --2- 172.21.15.60:0/3528385330 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd21806fce0 0x7fd21813fbc0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.112+0000 7fd21d09d640 1 -- 172.21.15.60:0/3528385330 >> 172.21.15.60:0/3528385330 conn(0x7fd2180b4e50 msgr2=0x7fd2180b5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7f182f750640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f182813c540 con 0x7f182806fce0 2024-03-27T21:39:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.111+0000 7f182f750640 1 librados: init done 2024-03-27T21:39:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.112+0000 7f182f750640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:39:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.112+0000 7f182f750640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:39:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.112+0000 7f182f750640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.112+0000 7f182f750640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f1828069450 con 0x7f182806fce0 2024-03-27T21:39:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.112+0000 7fd21d09d640 1 -- 172.21.15.60:0/3528385330 shutdown_connections 2024-03-27T21:39:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.112+0000 7fd21d09d640 1 -- 172.21.15.60:0/3528385330 wait complete. 2024-03-27T21:39:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.113+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f182402f050 con 0x7f182806fce0 2024-03-27T21:39:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.113+0000 7fd21d09d640 10 monclient: build_initial_monmap 2024-03-27T21:39:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.113+0000 7fd21d09d640 10 monclient: monmap: 2024-03-27T21:39:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:53.130 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:53.130 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:53.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.113+0000 7fd21d09d640 1 librados: starting msgr at 2024-03-27T21:39:53.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.113+0000 7fd21d09d640 1 librados: starting objecter 2024-03-27T21:39:53.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.114+0000 7f181dffb640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f180c06c590 0x7f180c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:53.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.114+0000 7fd21d09d640 1 Processor -- start 2024-03-27T21:39:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.114+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f18240aaf30 con 0x7f182806fce0 2024-03-27T21:39:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.114+0000 7f181dffb640 10 client.4394.objecter ms_dispatch 0x7f1828000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.114+0000 7f181dffb640 3 client.4394.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.114+0000 7f181dffb640 3 client.4394.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.114+0000 7f181dffb640 20 client.4394.objecter dump_active .. 0 homeless 2024-03-27T21:39:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.114+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v21) v1 ==== 54+0+16 (secure 0 0 0) 0x7f1824076560 con 0x7f182806fce0 2024-03-27T21:39:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.114+0000 7f181dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:39:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.114+0000 7f181dffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:53.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.114+0000 7fd21d09d640 1 -- start start 2024-03-27T21:39:53.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.114+0000 7fd21d09d640 1 librados: setting wanted keys 2024-03-27T21:39:53.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.114+0000 7fd21d09d640 1 librados: calling monclient init 2024-03-27T21:39:53.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.114+0000 7fd21d09d640 10 monclient: init 2024-03-27T21:39:53.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.114+0000 7f182cceb640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f180c06c590 0x7f180c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:53.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.114+0000 7f182cceb640 10 monclient: get_auth_request con 0x7f180c06c590 auth_method 0 2024-03-27T21:39:53.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.114+0000 7f182f750640 10 client.4394.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:53.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.114+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign -1 1 2024-03-27T21:39:53.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd21d09d640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:53.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd21d09d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:39:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.114+0000 7f182f750640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f182813b6a0 0x7f1828132450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f1828149ac0 osd=0 3 2024-03-27T21:39:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f1828149ac0 osd=0 3 2024-03-27T21:39:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f1828149ac0 osd=0 3 2024-03-27T21:39:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7f182f750640 15 client.4394.objecter _session_command_op_remove -1 1 2024-03-27T21:39:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7f182f750640 20 client.4394.objecter get_session s=0x7f1828149ac0 osd=0 3 2024-03-27T21:39:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign 0 1 2024-03-27T21:39:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f1828149ac0 osd=0 4 2024-03-27T21:39:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7f182f750640 10 client.4394.objecter _send_command 1 2024-03-27T21:39:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7f182f750640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1828136c60 con 0x7f182813b6a0 2024-03-27T21:39:53.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd21d09d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd21806f900 0x7fd21813a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:53.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd216d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd21806f900 0x7fd21813a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:53.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd216d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd21806f900 0x7fd21813a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44434/0 (socket says 172.21.15.60:44434) 2024-03-27T21:39:53.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7f182dced640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f182813b6a0 0x7f1828132450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:53.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7f182dced640 10 monclient: get_auth_request con 0x7f182813b6a0 auth_method 0 2024-03-27T21:39:53.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd216d9d640 1 -- 172.21.15.60:0/1650982868 learned_addr learned my addr 172.21.15.60:0/1650982868 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:53.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd21d09d640 10 monclient(hunting): picked mon.noname-a con 0x7fd21806f900 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:53.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd21d09d640 1 --2- 172.21.15.60:0/1650982868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd21806fce0 0x7fd2181daa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:53.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd21d09d640 10 monclient(hunting): picked mon.noname-b con 0x7fd21806fce0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:53.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd21d09d640 1 --2- 172.21.15.60:0/1650982868 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd218138bf0 0x7fd2181daf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:53.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd21759e640 1 --2- 172.21.15.60:0/1650982868 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd218138bf0 0x7fd2181daf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:53.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7f182dced640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f182813b6a0 0x7f1828132450 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:53.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd21d09d640 10 monclient(hunting): picked mon.noname-c con 0x7fd218138bf0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:53.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd21d09d640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:53.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd21d09d640 1 -- 172.21.15.60:0/1650982868 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd21813c3f0 con 0x7fd21806f900 2024-03-27T21:39:53.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd21d09d640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:53.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd21d09d640 1 -- 172.21.15.60:0/1650982868 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd21813b980 con 0x7fd21806fce0 2024-03-27T21:39:53.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd21d09d640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:53.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd21d09d640 1 -- 172.21.15.60:0/1650982868 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd21813bc60 con 0x7fd218138bf0 2024-03-27T21:39:53.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd21d09d640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:53.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd21d09d640 10 monclient(hunting): authenticate will time out at 834.336895s 2024-03-27T21:39:53.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd216d9d640 10 monclient(hunting): get_auth_request con 0x7fd21806f900 auth_method 0 2024-03-27T21:39:53.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd216d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:53.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd216d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:53.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7fd216d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:53.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.115+0000 7f181dffb640 10 client.4394.objecter ms_handle_connect 0x7f182813b6a0 2024-03-27T21:39:53.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7f182cceb640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f180c06c590 0x7f180c06ea50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1828133900 tx=0x7f1818008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:53.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7f181dffb640 10 client.4394.objecter ms_handle_connect 0x7f180c06c590 2024-03-27T21:39:53.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd216d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:53.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd216d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:53.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd21659c640 1 --2- 172.21.15.60:0/1650982868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd21806fce0 0x7fd2181daa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:53.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd216d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:53.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd21659c640 10 monclient(hunting): get_auth_request con 0x7fd21806fce0 auth_method 0 2024-03-27T21:39:53.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd21659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:53.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd21659c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:53.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd21659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:53.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd21659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:53.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd21659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:53.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd21659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:53.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd216d9d640 10 monclient(hunting): handle_auth_done global_id 4473 payload 824 2024-03-27T21:39:53.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd216d9d640 1 -- 172.21.15.60:0/1650982868 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd218138bf0 msgr2=0x7fd2181daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:53.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd216d9d640 1 --2- 172.21.15.60:0/1650982868 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd218138bf0 0x7fd2181daf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:53.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd216d9d640 1 -- 172.21.15.60:0/1650982868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd21806fce0 msgr2=0x7fd2181daa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:53.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd216d9d640 1 --2- 172.21.15.60:0/1650982868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd21806fce0 0x7fd2181daa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:53.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd216d9d640 10 monclient: _finish_hunting 0 2024-03-27T21:39:53.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd216d9d640 1 monclient: found mon.noname-a 2024-03-27T21:39:53.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f1828136c60 con 0x7f182813b6a0 2024-03-27T21:39:53.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7f181dffb640 10 client.4394.objecter ms_dispatch 0x7f1828000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:39:53.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd216d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:53.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7f181dffb640 10 client.4394.objecter _finish_command 1 = system:0 2024-03-27T21:39:53.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7f181dffb640 20 client.4394.objecter put_session s=0x7f1828149ac0 osd=0 4 2024-03-27T21:39:53.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7f181dffb640 15 client.4394.objecter _session_command_op_remove 0 1 2024-03-27T21:39:53.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd216d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:53.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd216d9d640 1 -- 172.21.15.60:0/1650982868 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd210009b30 con 0x7fd21806f900 2024-03-27T21:39:53.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd21759e640 10 monclient: get_auth_request con 0x7fd218138bf0 auth_method 0 2024-03-27T21:39:53.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.116+0000 7fd21759e640 1 --2- 172.21.15.60:0/1650982868 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd218138bf0 0x7fd2181daf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:39:53.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd216d9d640 1 --2- 172.21.15.60:0/1650982868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd21806f900 0x7fd21813a5f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd20800c500 tx=0x7fd20800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:53.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd1ff7fe640 10 client.?.objecter ms_handle_connect 0x7fd21806f900 2024-03-27T21:39:53.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd1ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:53.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd1ff7fe640 1 -- 172.21.15.60:0/1650982868 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2080052d0 con 0x7fd21806f900 2024-03-27T21:39:53.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd1ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:53.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd1ff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:53.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd1ff7fe640 10 monclient: dump: 2024-03-27T21:39:53.145 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:53.145 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:53.146 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:53.146 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:53.146 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:53.146 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:53.146 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:53.146 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:53.146 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:53.146 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:53.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd1ff7fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:53.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd1ff7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:53.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd1ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:53.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd21d09d640 5 monclient: authenticate success, global_id 4473 2024-03-27T21:39:53.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd1ff7fe640 1 -- 172.21.15.60:0/1650982868 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd208005470 con 0x7fd21806f900 2024-03-27T21:39:53.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd1ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:53.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd1ff7fe640 1 -- 172.21.15.60:0/1650982868 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd208003830 con 0x7fd21806f900 2024-03-27T21:39:53.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd1ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:53.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd1ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:53.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd1ff7fe640 10 monclient: dump: 2024-03-27T21:39:53.147 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:53.147 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:53.147 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:53.147 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:53.147 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:53.147 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:53.147 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:53.147 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:53.147 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:53.147 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:53.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd21d09d640 10 monclient: _renew_subs 2024-03-27T21:39:53.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd21d09d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:53.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd21d09d640 1 -- 172.21.15.60:0/1650982868 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd20c009400 con 0x7fd21806f900 2024-03-27T21:39:53.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd21559a640 4 set_mon_vals no callback set 2024-03-27T21:39:53.148 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:39:53.148 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:39:53.148 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:39:53.148 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:39:53.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd21d09d640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:53.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd21d09d640 10 monclient: _renew_subs 2024-03-27T21:39:53.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd21d09d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:53.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd21d09d640 1 -- 172.21.15.60:0/1650982868 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2181db4d0 con 0x7fd21806f900 2024-03-27T21:39:53.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.117+0000 7fd21d09d640 1 librados: init done 2024-03-27T21:39:53.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.118+0000 7fd21d09d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:53.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.118+0000 7fd21d09d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:53.150 INFO:tasks.workunit:timeout=6h 2024-03-27T21:39:53.150 INFO:tasks.workunit:cleanup=True 2024-03-27T21:39:53.151 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7c37d80f7a88f4eb561c4fe02558660a650b10ac 2024-03-27T21:39:53.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.118+0000 7fd21d09d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:53.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.119+0000 7fd21d09d640 1 -- 172.21.15.60:0/1650982868 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd218142c30 con 0x7fd21806f900 2024-03-27T21:39:53.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.119+0000 7fd1ff7fe640 1 -- 172.21.15.60:0/1650982868 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd208003990 con 0x7fd21806f900 2024-03-27T21:39:53.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.119+0000 7fd1ff7fe640 1 --2- 172.21.15.60:0/1650982868 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd20406c590 0x7fd20406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:53.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.120+0000 7fd1ff7fe640 1 -- 172.21.15.60:0/1650982868 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd208088700 con 0x7fd21806f900 2024-03-27T21:39:53.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.120+0000 7fd1ff7fe640 10 client.4473.objecter ms_dispatch 0x7fd218000b90 osd_map(21..21 src has 1..21) v4 2024-03-27T21:39:53.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.120+0000 7fd1ff7fe640 3 client.4473.objecter handle_osd_map got epochs [21,21] > 0 2024-03-27T21:39:53.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.120+0000 7fd1ff7fe640 3 client.4473.objecter handle_osd_map decoding full epoch 21 2024-03-27T21:39:53.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.120+0000 7fd1ff7fe640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-27T21:39:53.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.120+0000 7fd21659c640 1 --2- 172.21.15.60:0/1650982868 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd20406c590 0x7fd20406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:53.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.120+0000 7fd21659c640 10 monclient: get_auth_request con 0x7fd20406c590 auth_method 0 2024-03-27T21:39:53.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.121+0000 7fd1ff7fe640 10 client.4473.objecter ms_handle_connect 0x7fd20406c590 2024-03-27T21:39:53.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.121+0000 7fd21659c640 1 --2- 172.21.15.60:0/1650982868 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd20406c590 0x7fd20406ea50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd20c03a000 tx=0x7fd20c034000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:53.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.124+0000 7fd1ff7fe640 1 -- 172.21.15.60:0/1650982868 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd208053af0 con 0x7fd21806f900 2024-03-27T21:39:53.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.124+0000 7fd1ff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:53.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.124+0000 7fd1ff7fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:53.154 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:39:53.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.133+0000 7f182f750640 10 client.4394.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:39:53.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.133+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign -1 2 2024-03-27T21:39:53.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.133+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f1828149ac0 osd=0 3 2024-03-27T21:39:53.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.133+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f1828149ac0 osd=0 3 2024-03-27T21:39:53.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.133+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f1828149ac0 osd=0 3 2024-03-27T21:39:53.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.133+0000 7f182f750640 15 client.4394.objecter _session_command_op_remove -1 2 2024-03-27T21:39:53.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.133+0000 7f182f750640 20 client.4394.objecter get_session s=0x7f1828149ac0 osd=0 3 2024-03-27T21:39:53.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.133+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign 0 2 2024-03-27T21:39:53.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.133+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f1828149ac0 osd=0 4 2024-03-27T21:39:53.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.133+0000 7f182f750640 10 client.4394.objecter _send_command 2 2024-03-27T21:39:53.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.133+0000 7f182f750640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1828132990 con 0x7f182813b6a0 2024-03-27T21:39:53.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.135+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f1828132990 con 0x7f182813b6a0 2024-03-27T21:39:53.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.135+0000 7f181dffb640 10 client.4394.objecter ms_dispatch 0x7f1828000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:39:53.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.135+0000 7f181dffb640 10 client.4394.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:39:53.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.135+0000 7f181dffb640 20 client.4394.objecter put_session s=0x7f1828149ac0 osd=0 4 2024-03-27T21:39:53.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.135+0000 7f181dffb640 15 client.4394.objecter _session_command_op_remove 0 2 2024-03-27T21:39:53.156 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:39:53.158 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:39:53.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.135+0000 7f182f750640 10 client.4394.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:53.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.135+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign -1 3 2024-03-27T21:39:53.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.135+0000 7f182f750640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f1828065fa0 0x7f1828066380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:53.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.135+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f1828132da0 osd=1 3 2024-03-27T21:39:53.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.135+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f1828132da0 osd=1 3 2024-03-27T21:39:53.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.135+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f1828132da0 osd=1 3 2024-03-27T21:39:53.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.135+0000 7f182f750640 15 client.4394.objecter _session_command_op_remove -1 3 2024-03-27T21:39:53.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.135+0000 7f182f750640 20 client.4394.objecter get_session s=0x7f1828132da0 osd=1 3 2024-03-27T21:39:53.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.135+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign 1 3 2024-03-27T21:39:53.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.135+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f1828132da0 osd=1 4 2024-03-27T21:39:53.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.135+0000 7f182f750640 10 client.4394.objecter _send_command 3 2024-03-27T21:39:53.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.135+0000 7f182f750640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18280668c0 con 0x7f1828065fa0 2024-03-27T21:39:53.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.135+0000 7f182d4ec640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f1828065fa0 0x7f1828066380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:53.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.135+0000 7f182d4ec640 10 monclient: get_auth_request con 0x7f1828065fa0 auth_method 0 2024-03-27T21:39:53.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.135+0000 7f181dffb640 10 client.4394.objecter ms_handle_connect 0x7f1828065fa0 2024-03-27T21:39:53.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.135+0000 7f182d4ec640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f1828065fa0 0x7f1828066380 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:53.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.136+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f18280668c0 con 0x7f1828065fa0 2024-03-27T21:39:53.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.136+0000 7f181dffb640 10 client.4394.objecter ms_dispatch 0x7f1828000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:39:53.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.136+0000 7f181dffb640 10 client.4394.objecter _finish_command 3 = system:0 2024-03-27T21:39:53.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.136+0000 7f181dffb640 20 client.4394.objecter put_session s=0x7f1828132da0 osd=1 4 2024-03-27T21:39:53.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.136+0000 7f181dffb640 15 client.4394.objecter _session_command_op_remove 1 3 2024-03-27T21:39:53.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.151+0000 7f182f750640 10 client.4394.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:39:53.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.151+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign -1 4 2024-03-27T21:39:53.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.151+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f1828132da0 osd=1 3 2024-03-27T21:39:53.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.151+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f1828132da0 osd=1 3 2024-03-27T21:39:53.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.151+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f1828132da0 osd=1 3 2024-03-27T21:39:53.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.151+0000 7f182f750640 15 client.4394.objecter _session_command_op_remove -1 4 2024-03-27T21:39:53.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.151+0000 7f182f750640 20 client.4394.objecter get_session s=0x7f1828132da0 osd=1 3 2024-03-27T21:39:53.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.151+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign 1 4 2024-03-27T21:39:53.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.151+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f1828132da0 osd=1 4 2024-03-27T21:39:53.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.151+0000 7f182f750640 10 client.4394.objecter _send_command 4 2024-03-27T21:39:53.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.151+0000 7f182f750640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1828066a60 con 0x7f1828065fa0 2024-03-27T21:39:53.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.152+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f1828066a60 con 0x7f1828065fa0 2024-03-27T21:39:53.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.152+0000 7f181dffb640 10 client.4394.objecter ms_dispatch 0x7f1828000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:39:53.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.152+0000 7f181dffb640 10 client.4394.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:39:53.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.152+0000 7f181dffb640 20 client.4394.objecter put_session s=0x7f1828132da0 osd=1 4 2024-03-27T21:39:53.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.152+0000 7f181dffb640 15 client.4394.objecter _session_command_op_remove 1 4 2024-03-27T21:39:53.161 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:39:53.162 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:39:53.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.152+0000 7f182f750640 10 client.4394.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:53.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.152+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign -1 5 2024-03-27T21:39:53.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.153+0000 7f182f750640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] conn(0x7f1828148430 0x7f1828148850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:53.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.153+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f18280b36f0 osd=2 3 2024-03-27T21:39:53.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.153+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f18280b36f0 osd=2 3 2024-03-27T21:39:53.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.153+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f18280b36f0 osd=2 3 2024-03-27T21:39:53.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.153+0000 7f182f750640 15 client.4394.objecter _session_command_op_remove -1 5 2024-03-27T21:39:53.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.153+0000 7f182f750640 20 client.4394.objecter get_session s=0x7f18280b36f0 osd=2 3 2024-03-27T21:39:53.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.153+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign 2 5 2024-03-27T21:39:53.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.153+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f18280b36f0 osd=2 4 2024-03-27T21:39:53.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.153+0000 7f182f750640 10 client.4394.objecter _send_command 5 2024-03-27T21:39:53.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.153+0000 7f182f750640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18280b3860 con 0x7f1828148430 2024-03-27T21:39:53.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.153+0000 7f182cceb640 1 -- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] conn(0x7f1828148430 msgr2=0x7f1828148850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6824/34604 2024-03-27T21:39:53.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.153+0000 7f181dffb640 1 client.4394.objecter ms_handle_refused on osd.2 2024-03-27T21:39:53.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.153+0000 7f182cceb640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] conn(0x7f1828148430 0x7f1828148850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-27T21:39:53.192 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:39:53.205+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:39:53.207 INFO:tasks.workunit.client.0.smithi060.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-27T21:39:53.266 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:39:53.279+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:39:53.299 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:53.299 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:53.308 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:39:53.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.322+0000 7fd21d09d640 10 start_mon_command cmd=[{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:39:53.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.322+0000 7fd21d09d640 10 monclient: _send_command 2 [{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:39:53.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.322+0000 7fd21d09d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:39:53.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.322+0000 7fd21d09d640 1 -- 172.21.15.60:0/1650982868 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fd218069470 con 0x7fd21806f900 2024-03-27T21:39:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.354+0000 7f182cceb640 1 -- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] conn(0x7f1828148430 msgr2=0x7f1828148850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6824/34604 2024-03-27T21:39:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.354+0000 7f182cceb640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] conn(0x7f1828148430 0x7f1828148850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-27T21:39:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.354+0000 7f181dffb640 1 client.4394.objecter ms_handle_refused on osd.2 2024-03-27T21:39:53.366 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:39:53.380+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:39:53.424 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:53.424 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:53.437 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:39:53.478 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:39:53.492+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:39:53.568 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:39:53.582+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:39:53.587 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:53.588 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:53.596 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:39:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.614+0000 7fd1ff7fe640 1 -- 172.21.15.60:0/1650982868 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v22) v1 ==== 87+0+0 (secure 0 0 0) 0x7fd208057110 con 0x7fd21806f900 2024-03-27T21:39:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.614+0000 7fd1ff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:39:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.614+0000 7fd1ff7fe640 10 monclient: _finish_command 2 = system:0 noscrub is set 2024-03-27T21:39:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:noscrub is set 2024-03-27T21:39:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.616+0000 7fd21d09d640 10 librados: watch_flush enter 2024-03-27T21:39:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.616+0000 7fd21d09d640 10 librados: watch_flush exit 2024-03-27T21:39:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.616+0000 7fd21d09d640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-27T21:39:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.616+0000 7fd21d09d640 10 client.4473.objecter successfully canceled tick 2024-03-27T21:39:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.616+0000 7fd21d09d640 1 -- 172.21.15.60:0/1650982868 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd20406c590 msgr2=0x7fd20406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.616+0000 7fd21d09d640 1 --2- 172.21.15.60:0/1650982868 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd20406c590 0x7fd20406ea50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd20c03a000 tx=0x7fd20c034000 comp rx=0 tx=0).stop 2024-03-27T21:39:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.616+0000 7fd21d09d640 10 monclient: shutdown 2024-03-27T21:39:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.616+0000 7fd21d09d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.616+0000 7fd21d09d640 1 -- 172.21.15.60:0/1650982868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd21806f900 msgr2=0x7fd21813a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.616+0000 7fd21d09d640 1 --2- 172.21.15.60:0/1650982868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd21806f900 0x7fd21813a5f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd20800c500 tx=0x7fd20800c9c0 comp rx=0 tx=0).stop 2024-03-27T21:39:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.616+0000 7fd21d09d640 1 -- 172.21.15.60:0/1650982868 shutdown_connections 2024-03-27T21:39:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.616+0000 7fd21d09d640 1 --2- 172.21.15.60:0/1650982868 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd218138bf0 0x7fd2181daf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.616+0000 7fd21d09d640 1 --2- 172.21.15.60:0/1650982868 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd20406c590 0x7fd20406ea50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.616+0000 7fd21d09d640 1 --2- 172.21.15.60:0/1650982868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd21806fce0 0x7fd2181daa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.616+0000 7fd21d09d640 1 --2- 172.21.15.60:0/1650982868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd21806f900 0x7fd21813a5f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.616+0000 7fd21d09d640 1 -- 172.21.15.60:0/1650982868 >> 172.21.15.60:0/1650982868 conn(0x7fd2180b4e50 msgr2=0x7fd2180b06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.617+0000 7fd21d09d640 1 -- 172.21.15.60:0/1650982868 shutdown_connections 2024-03-27T21:39:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.617+0000 7fd21d09d640 1 -- 172.21.15.60:0/1650982868 wait complete. 2024-03-27T21:39:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.617+0000 7fd21d09d640 1 librados: shutdown 2024-03-27T21:39:53.669 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:39:53.683+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:39:53.707 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:53.707 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:53.716 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:39:53.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.755+0000 7f182cceb640 1 -- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] conn(0x7f1828148430 msgr2=0x7f1828148850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6824/34604 2024-03-27T21:39:53.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.755+0000 7f182cceb640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] conn(0x7f1828148430 0x7f1828148850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-27T21:39:53.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:53.755+0000 7f181dffb640 1 client.4394.objecter ms_handle_refused on osd.2 2024-03-27T21:39:53.770 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:39:53.784+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:39:53.847 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:53.847 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:53.856 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:39:53.873 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:39:53.885+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:39:53.963 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:39:53.963 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:39:53.963 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:39:53.963 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:39:53.972 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:39:53.972 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:39:53.985+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:39:54.024 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-03-27T21:39:54.072 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:39:54.086+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:39:54.119 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:39:54.119 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:39:54.119 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:39:54.119 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:39:54.119 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:39:54.127 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:39:54.173 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:39:54.187+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:39:54.222 INFO:teuthology.orchestra.run.smithi060.stdout:2.3 2024-03-27T21:39:54.222 INFO:teuthology.orchestra.run.smithi060.stdout:2.1 2024-03-27T21:39:54.247 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:39:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:39:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:39:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:39:54.247 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:39:54.256 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:39:54.297 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:39:54.311+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:39:54.372 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:39:54.372 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:39:54.372 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:39:54.372 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:39:54.375 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:39:54.388+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:39:54.381 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:39:54.476 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:39:54.489+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:39:54.527 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:39:54.527 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:39:54.527 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:39:54.528 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:39:54.528 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:39:54.536 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:39:54.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:54.556+0000 7f182cceb640 1 -- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] conn(0x7f1828148430 msgr2=0x7f1828148850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6824/34604 2024-03-27T21:39:54.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:54.556+0000 7f182cceb640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] conn(0x7f1828148430 0x7f1828148850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-27T21:39:54.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:54.556+0000 7f181dffb640 1 client.4394.objecter ms_handle_refused on osd.2 2024-03-27T21:39:54.579 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:39:54.593+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:39:54.641 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:39:54.642 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:39:54.642 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:39:54.642 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:39:54.642 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:39:54.650 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:39:54.691 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:39:54.704+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:39:54.739 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 2.1 --file /var/log/ceph/exp.2.1.2 2024-03-27T21:39:54.778 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:39:54.792+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:39:54.795 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:39:54.795 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:39:54.795 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:39:54.796 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:39:54.804 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:39:54.879 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:39:54.893+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:39:54.947 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:39:54.947 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:39:54.947 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:39:54.947 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:39:54.948 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:39:54.956 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:39:55.000 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:39:55.013+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:39:55.061 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:39:55.061 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:39:55.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:39:55.061 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:39:55.061 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:39:55.070 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:39:55.112 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:39:55.125+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:39:55.181 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:39:55.195+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:39:55.216 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:39:55.216 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:39:55.216 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:39:55.217 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:39:55.225 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:39:55.308 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:39:55.322+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:39:55.370 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:39:55.371 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:39:55.371 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:39:55.371 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:39:55.371 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:39:55.379 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:39:55.384 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:39:55.397+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:39:55.485 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:39:55.498+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:39:55.526 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:39:55.526 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:39:55.526 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:39:55.526 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:39:55.526 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:39:55.534 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:39:55.585 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:39:55.599+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:39:55.616 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-27T21:39:55.681 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:39:55.681 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:39:55.682 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:39:55.682 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:39:55.686 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:39:55.700+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:39:55.690 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:39:55.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.737+0000 7f6be4e2b640 10 monclient: get_monmap_and_config 2024-03-27T21:39:55.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.737+0000 7f6be4e2b640 10 monclient: build_initial_monmap 2024-03-27T21:39:55.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.737+0000 7f6be4e2b640 10 monclient: monmap: 2024-03-27T21:39:55.723 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:55.724 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:55.724 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:55.738160+0000 2024-03-27T21:39:55.724 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:55.738160+0000 2024-03-27T21:39:55.724 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:55.724 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:55.724 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:55.724 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:55.724 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:55.724 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:55.724 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:55.724 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.737+0000 7f6be4e2b640 1 Processor -- start 2024-03-27T21:39:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.737+0000 7f6be4e2b640 1 -- start start 2024-03-27T21:39:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.737+0000 7f6be4e2b640 10 monclient: init 2024-03-27T21:39:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6be4e2b640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6be4e2b640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-03-27T21:39:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6be4e2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6be00b40b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6be4e2b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6be013b020 0x7f6be013b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6be4e2b640 10 monclient(hunting): picked mon.noname-b con 0x7f6be013b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6be4e2b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6be01416f0 0x7f6be0141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6be4e2b640 10 monclient(hunting): picked mon.noname-c con 0x7f6be01416f0 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6be4e2b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be00b3830 con 0x7f6be00b40b0 2024-03-27T21:39:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6be4e2b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6be4e2b640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be00b4490 con 0x7f6be01416f0 2024-03-27T21:39:55.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6be4e2b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:55.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6be4e2b640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be00aa520 con 0x7f6be013b020 2024-03-27T21:39:55.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6be4e2b640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:55.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6be4e2b640 10 monclient(hunting): authenticate will time out at 836.959448s 2024-03-27T21:39:55.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bd7fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6be013b020 0x7f6be013b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:55.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bde59c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6be00b40b0 0x7f6be00a8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55838/0 (socket says 172.21.15.60:55838) 2024-03-27T21:39:55.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bde59c640 1 -- 172.21.15.60:0/2432242351 learned_addr learned my addr 172.21.15.60:0/2432242351 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:39:55.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bd7fff640 10 monclient(hunting): get_auth_request con 0x7f6be013b020 auth_method 0 2024-03-27T21:39:55.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bd7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:55.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bd7fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:55.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:55.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bd7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:55.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bd7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bd7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bded9d640 1 --2- 172.21.15.60:0/2432242351 >> v2:172.21.15.191:3300/0 conn(0x7f6be01416f0 0x7f6be0141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bded9d640 10 monclient(hunting): get_auth_request con 0x7f6be01416f0 auth_method 0 2024-03-27T21:39:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bded9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bded9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bded9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bded9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bddd9b640 1 -- 172.21.15.60:0/2432242351 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185567792 0 0) 0x7f6be00b3830 con 0x7f6be00b40b0 2024-03-27T21:39:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bded9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bd7fff640 10 monclient(hunting): handle_auth_done global_id 4397 payload 274 2024-03-27T21:39:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bd7fff640 1 -- 172.21.15.60:0/2432242351 >> v2:172.21.15.191:3300/0 conn(0x7f6be01416f0 msgr2=0x7f6be0141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.738+0000 7f6bd7fff640 1 --2- 172.21.15.60:0/2432242351 >> v2:172.21.15.191:3300/0 conn(0x7f6be01416f0 0x7f6be0141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6bd7fff640 1 -- 172.21.15.60:0/2432242351 >> v1:172.21.15.60:6789/0 conn(0x7f6be00b40b0 legacy=0x7f6be00a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6bd7fff640 10 monclient: _finish_hunting 0 2024-03-27T21:39:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6bd7fff640 1 monclient: found mon.noname-b 2024-03-27T21:39:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6bd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6bd7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:39:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6bd7fff640 1 -- 172.21.15.60:0/2432242351 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be00a9470 con 0x7f6be013b020 2024-03-27T21:39:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6bddd9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:39:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6bd7fff640 1 --2- 172.21.15.60:0/2432242351 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6be013b020 0x7f6be013b480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6bc80097a0 tx=0x7f6bc802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9173cf564ed111b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6bddd9b640 1 -- 172.21.15.60:0/2432242351 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8031830 con 0x7f6be013b020 2024-03-27T21:39:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6bddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6bddd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:39:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6bddd9b640 10 monclient: dump: 2024-03-27T21:39:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:55.737 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6bddd9b640 10 monclient: _finish_auth 0 2024-03-27T21:39:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6bddd9b640 10 monclient: _check_auth_tickets 2024-03-27T21:39:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6bddd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6bddd9b640 1 -- 172.21.15.60:0/2432242351 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc8031990 con 0x7f6be013b020 2024-03-27T21:39:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6be4e2b640 5 monclient: authenticate success, global_id 4397 2024-03-27T21:39:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6be4e2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6bddd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6bddd9b640 1 -- 172.21.15.60:0/2432242351 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8031d20 con 0x7f6be013b020 2024-03-27T21:39:55.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6be4e2b640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:55.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6be4e2b640 4 set_mon_vals no callback set 2024-03-27T21:39:55.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6bddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:55.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6bddd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:55.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.739+0000 7f6bddd9b640 10 monclient: dump: 2024-03-27T21:39:55.738 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:55.738 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:55.738 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:55.738 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:55.738 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:55.738 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:55.738 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:55.738 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:55.738 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:55.739 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:55.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.740+0000 7f6be4e2b640 10 monclient: shutdown 2024-03-27T21:39:55.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.740+0000 7f6be4e2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:55.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.740+0000 7f6be4e2b640 1 -- 172.21.15.60:0/2432242351 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6be013b020 msgr2=0x7f6be013b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:55.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.740+0000 7f6be4e2b640 1 --2- 172.21.15.60:0/2432242351 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6be013b020 0x7f6be013b480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6bc80097a0 tx=0x7f6bc802b8d0 comp rx=0 tx=0).stop 2024-03-27T21:39:55.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.740+0000 7f6be4e2b640 1 -- 172.21.15.60:0/2432242351 shutdown_connections 2024-03-27T21:39:55.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.740+0000 7f6be4e2b640 1 --2- 172.21.15.60:0/2432242351 >> v2:172.21.15.191:3300/0 conn(0x7f6be01416f0 0x7f6be0141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:55.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.740+0000 7f6be4e2b640 1 --2- 172.21.15.60:0/2432242351 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6be013b020 0x7f6be013b480 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:55.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.740+0000 7f6be4e2b640 1 -- 172.21.15.60:0/2432242351 >> 172.21.15.60:0/2432242351 conn(0x7f6be00608e0 msgr2=0x7f6be0060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:55.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.740+0000 7f6be4e2b640 1 -- 172.21.15.60:0/2432242351 shutdown_connections 2024-03-27T21:39:55.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.740+0000 7f6be4e2b640 1 -- 172.21.15.60:0/2432242351 wait complete. 2024-03-27T21:39:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.741+0000 7f6be4e2b640 10 monclient: build_initial_monmap 2024-03-27T21:39:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.741+0000 7f6be4e2b640 10 monclient: monmap: 2024-03-27T21:39:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:55.740 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.741+0000 7f6be4e2b640 1 librados: starting msgr at 2024-03-27T21:39:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.741+0000 7f6be4e2b640 1 librados: starting objecter 2024-03-27T21:39:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.741+0000 7f6be4e2b640 1 Processor -- start 2024-03-27T21:39:55.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.741+0000 7f6be4e2b640 1 -- start start 2024-03-27T21:39:55.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.741+0000 7f6be4e2b640 1 librados: setting wanted keys 2024-03-27T21:39:55.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.741+0000 7f6be4e2b640 1 librados: calling monclient init 2024-03-27T21:39:55.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.741+0000 7f6be4e2b640 10 monclient: init 2024-03-27T21:39:55.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6be4e2b640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:55.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6be4e2b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:39:55.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6be4e2b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6be00b40b0 0x7f6be0139f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:55.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6be4e2b640 10 monclient(hunting): picked mon.noname-b con 0x7f6be00b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:55.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6be4e2b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6be013b020 0x7f6be01d6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:55.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6be4e2b640 10 monclient(hunting): picked mon.noname-c con 0x7f6be013b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:55.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6be4e2b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6be01416f0 0x7f6be01d66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:55.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6be4e2b640 10 monclient(hunting): picked mon.noname-a con 0x7f6be01416f0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:55.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6be4e2b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:55.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6be4e2b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be013a940 con 0x7f6be01416f0 2024-03-27T21:39:55.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6be4e2b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:55.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6be4e2b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be013d920 con 0x7f6be00b40b0 2024-03-27T21:39:55.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6be4e2b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:55.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6be4e2b640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be013dbc0 con 0x7f6be013b020 2024-03-27T21:39:55.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6be4e2b640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:55.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6be4e2b640 10 monclient(hunting): authenticate will time out at 836.963218s 2024-03-27T21:39:55.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd7fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6be013b020 0x7f6be01d6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:55.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bded9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6be01416f0 0x7f6be01d66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:55.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd7fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6be013b020 0x7f6be01d6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47844/0 (socket says 172.21.15.60:47844) 2024-03-27T21:39:55.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd7fff640 1 -- 172.21.15.60:0/3806520975 learned_addr learned my addr 172.21.15.60:0/3806520975 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:55.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd7fff640 10 monclient(hunting): get_auth_request con 0x7f6be013b020 auth_method 0 2024-03-27T21:39:55.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:55.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd7fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:55.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:55.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bded9d640 10 monclient(hunting): get_auth_request con 0x7f6be01416f0 auth_method 0 2024-03-27T21:39:55.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:55.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bded9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:55.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bded9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:55.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bde59c640 1 --2- 172.21.15.60:0/3806520975 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6be00b40b0 0x7f6be0139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:55.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bde59c640 10 monclient(hunting): get_auth_request con 0x7f6be00b40b0 auth_method 0 2024-03-27T21:39:55.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bde59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:55.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bde59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:55.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bde59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:55.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:55.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:55.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:55.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bded9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:55.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bded9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:55.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bded9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:55.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bde59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:55.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bde59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:55.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bde59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:55.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd7fff640 10 monclient(hunting): handle_auth_done global_id 4400 payload 824 2024-03-27T21:39:55.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd7fff640 1 -- 172.21.15.60:0/3806520975 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6be00b40b0 msgr2=0x7f6be0139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:55.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd7fff640 1 --2- 172.21.15.60:0/3806520975 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6be00b40b0 0x7f6be0139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:55.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd7fff640 1 -- 172.21.15.60:0/3806520975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6be01416f0 msgr2=0x7f6be01d66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:55.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd7fff640 1 --2- 172.21.15.60:0/3806520975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6be01416f0 0x7f6be01d66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:55.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd7fff640 10 monclient: _finish_hunting 0 2024-03-27T21:39:55.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd7fff640 1 monclient: found mon.noname-c 2024-03-27T21:39:55.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:55.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:39:55.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd7fff640 1 -- 172.21.15.60:0/3806520975 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc8009400 con 0x7f6be013b020 2024-03-27T21:39:55.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd7fff640 1 --2- 172.21.15.60:0/3806520975 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6be013b020 0x7f6be01d6180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6bc802bee0 tx=0x7f6bc80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:55.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd6ffd640 10 client.?.objecter ms_handle_connect 0x7f6be013b020 2024-03-27T21:39:55.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.742+0000 7f6bd6ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:55.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6bd6ffd640 1 -- 172.21.15.60:0/3806520975 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc803a6f0 con 0x7f6be013b020 2024-03-27T21:39:55.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6bd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:55.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6bd6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:55.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6bd6ffd640 10 monclient: dump: 2024-03-27T21:39:55.745 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:55.745 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:55.745 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:55.745 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:55.745 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:55.746 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:55.746 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:55.746 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:55.746 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:55.746 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:55.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6bd6ffd640 10 monclient: _finish_auth 0 2024-03-27T21:39:55.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6bd6ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:39:55.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6bd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:55.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6bd6ffd640 1 -- 172.21.15.60:0/3806520975 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc803a870 con 0x7f6be013b020 2024-03-27T21:39:55.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6be4e2b640 5 monclient: authenticate success, global_id 4400 2024-03-27T21:39:55.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6bd6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:55.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6bd6ffd640 1 -- 172.21.15.60:0/3806520975 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc803ac20 con 0x7f6be013b020 2024-03-27T21:39:55.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6be4e2b640 10 monclient: _renew_subs 2024-03-27T21:39:55.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6be4e2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6be4e2b640 1 -- 172.21.15.60:0/3806520975 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be013e050 con 0x7f6be013b020 2024-03-27T21:39:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6bd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6bdd59a640 4 set_mon_vals no callback set 2024-03-27T21:39:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6bd6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6bd6ffd640 10 monclient: dump: 2024-03-27T21:39:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:55.747 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6be4e2b640 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6be4e2b640 10 monclient: _renew_subs 2024-03-27T21:39:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6be4e2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6be4e2b640 1 -- 172.21.15.60:0/3806520975 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be013cff0 con 0x7f6be013b020 2024-03-27T21:39:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.743+0000 7f6be4e2b640 1 librados: init done 2024-03-27T21:39:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.744+0000 7f6be4e2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.744+0000 7f6be4e2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.744+0000 7f6be4e2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.744+0000 7f6be4e2b640 1 -- 172.21.15.60:0/3806520975 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be0069470 con 0x7f6be013b020 2024-03-27T21:39:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.744+0000 7f6bd6ffd640 1 -- 172.21.15.60:0/3806520975 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6bc8030380 con 0x7f6be013b020 2024-03-27T21:39:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.744+0000 7f6bd6ffd640 1 --2- 172.21.15.60:0/3806520975 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6bc006c590 0x7f6bc006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.745+0000 7f6bde59c640 1 --2- 172.21.15.60:0/3806520975 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6bc006c590 0x7f6bc006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.745+0000 7f6bde59c640 10 monclient: get_auth_request con 0x7f6bc006c590 auth_method 0 2024-03-27T21:39:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.745+0000 7f6bd6ffd640 1 -- 172.21.15.60:0/3806520975 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6bc80aa970 con 0x7f6be013b020 2024-03-27T21:39:55.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.745+0000 7f6bd6ffd640 10 client.4400.objecter ms_dispatch 0x7f6be0000b90 osd_map(23..23 src has 1..23) v4 2024-03-27T21:39:55.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.745+0000 7f6bd6ffd640 3 client.4400.objecter handle_osd_map got epochs [23,23] > 0 2024-03-27T21:39:55.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.745+0000 7f6bd6ffd640 3 client.4400.objecter handle_osd_map decoding full epoch 23 2024-03-27T21:39:55.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.745+0000 7f6bd6ffd640 20 client.4400.objecter dump_active .. 0 homeless 2024-03-27T21:39:55.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.745+0000 7f6bde59c640 1 --2- 172.21.15.60:0/3806520975 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6bc006c590 0x7f6bc006ea50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0007fd0 tx=0x7f6bd0007de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:55.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.745+0000 7f6bd6ffd640 10 client.4400.objecter ms_handle_connect 0x7f6bc006c590 2024-03-27T21:39:55.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.749+0000 7f6bd6ffd640 1 -- 172.21.15.60:0/3806520975 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6bc8075fa0 con 0x7f6be013b020 2024-03-27T21:39:55.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.749+0000 7f6bd6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:55.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.749+0000 7f6bd6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:55.787 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:39:55.800+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:39:55.793 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:39:55.793 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:39:55.794 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:39:55.794 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:39:55.794 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:39:55.802 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:39:55.870 INFO:teuthology.orchestra.run.smithi060.stderr:Exporting 2.1 info 2.1( empty local-lis/les=19/20 n=0 ec=19/19 lis/c=19/19 les/c/f=20/20/0 sis=19) 2024-03-27T21:39:55.870 INFO:teuthology.orchestra.run.smithi060.stderr:Export successful 2024-03-27T21:39:55.888 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:39:55.901+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:39:55.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.938+0000 7f6be4e2b640 10 start_mon_command cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:39:55.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.938+0000 7f6be4e2b640 10 monclient: _send_command 2 [{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:39:55.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.938+0000 7f6be4e2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:55.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:55.938+0000 7f6be4e2b640 1 -- 172.21.15.60:0/3806520975 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f6be0069e40 con 0x7f6be013b020 2024-03-27T21:39:55.950 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:39:55.951 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:39:55.951 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:39:55.951 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-03-27T21:39:55.951 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.951 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:39:55.951 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:39:55.951 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.530897961000001, 2024-03-27T21:39:55.951 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43537978700000002, 2024-03-27T21:39:55.951 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:39:55.951 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:39:55.951 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:39:55.952 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:39:55.952 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:39:55.952 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-03-27T21:39:55.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.952 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:39:55.952 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.952 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:39:55.952 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:39:55.952 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:55.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.952 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.952 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:39:55.952 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:39:55.952 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:55.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:39:55.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432019+0000", 2024-03-27T21:39:55.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-03-27T21:39:55.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:39:55.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432027+0000", 2024-03-27T21:39:55.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1200000000000002e-06 2024-03-27T21:39:55.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:39:55.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432039+0000", 2024-03-27T21:39:55.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1123e-05 2024-03-27T21:39:55.954 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.954 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.954 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:39:55.954 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432051+0000", 2024-03-27T21:39:55.954 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-03-27T21:39:55.954 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.954 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.954 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:39:55.954 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432157+0000", 2024-03-27T21:39:55.954 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106846 2024-03-27T21:39:55.954 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.954 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.955 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-03-27T21:39:55.955 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432176+0000", 2024-03-27T21:39:55.955 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8490000000000001e-05 2024-03-27T21:39:55.956 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.956 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.956 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:39:55.956 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866170+0000", 2024-03-27T21:39:55.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43399370100000001 2024-03-27T21:39:55.956 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.956 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.956 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:39:55.956 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866335+0000", 2024-03-27T21:39:55.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016559699999999999 2024-03-27T21:39:55.957 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.957 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.957 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:39:55.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866495+0000", 2024-03-27T21:39:55.957 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015944200000000001 2024-03-27T21:39:55.957 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.957 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.957 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:39:55.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867012+0000", 2024-03-27T21:39:55.957 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0005176 2024-03-27T21:39:55.957 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.957 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.957 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:39:55.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867334+0000", 2024-03-27T21:39:55.957 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032158600000000001 2024-03-27T21:39:55.958 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.958 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.958 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:39:55.958 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867374+0000", 2024-03-27T21:39:55.958 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0547000000000003e-05 2024-03-27T21:39:55.958 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.958 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.958 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:39:55.958 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867400+0000", 2024-03-27T21:39:55.958 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5958000000000001e-05 2024-03-27T21:39:55.958 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:55.958 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:39:55.958 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:55.958 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.959 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.959 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:39:55.959 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:39:55.959 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.033515510000001, 2024-03-27T21:39:55.959 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064560310000000001, 2024-03-27T21:39:55.959 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:39:55.959 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:39:55.959 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:39:55.959 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:39:55.959 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:39:55.959 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 70 2024-03-27T21:39:55.959 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.959 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:39:55.959 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.960 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:39:55.960 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:39:55.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:55.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.960 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:39:55.960 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:39:55.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:55.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.960 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:39:55.960 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929402+0000", 2024-03-27T21:39:55.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:39:55.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:39:55.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929407+0000", 2024-03-27T21:39:55.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-06 2024-03-27T21:39:55.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:39:55.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929414+0000", 2024-03-27T21:39:55.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5689999999999999e-06 2024-03-27T21:39:55.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:39:55.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929419+0000", 2024-03-27T21:39:55.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-03-27T21:39:55.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:39:55.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930245+0000", 2024-03-27T21:39:55.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082521500000000002 2024-03-27T21:39:55.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:39:55.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930287+0000", 2024-03-27T21:39:55.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2305000000000002e-05 2024-03-27T21:39:55.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:39:55.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930367+0000", 2024-03-27T21:39:55.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9589e-05 2024-03-27T21:39:55.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:39:55.963 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.931214+0000", 2024-03-27T21:39:55.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000847827 2024-03-27T21:39:55.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:39:55.963 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935807+0000", 2024-03-27T21:39:55.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045923780000000003 2024-03-27T21:39:55.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:39:55.963 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935840+0000", 2024-03-27T21:39:55.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3501999999999999e-05 2024-03-27T21:39:55.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:39:55.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935859+0000", 2024-03-27T21:39:55.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8845e-05 2024-03-27T21:39:55.964 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:55.964 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:39:55.964 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:55.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.965 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:39:55.965 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:39:55.965 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.026255919, 2024-03-27T21:39:55.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071695429999999996, 2024-03-27T21:39:55.965 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:39:55.965 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:39:55.965 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:39:55.965 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:39:55.965 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:39:55.965 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-03-27T21:39:55.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.965 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:39:55.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:39:55.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:39:55.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:55.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:39:55.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:39:55.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:55.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:39:55.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936662+0000", 2024-03-27T21:39:55.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:39:55.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.967 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:39:55.967 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936737+0000", 2024-03-27T21:39:55.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5303999999999996e-05 2024-03-27T21:39:55.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.967 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:39:55.967 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936746+0000", 2024-03-27T21:39:55.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6340000000000007e-06 2024-03-27T21:39:55.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.967 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:39:55.967 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936753+0000", 2024-03-27T21:39:55.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-03-27T21:39:55.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:39:55.968 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937926+0000", 2024-03-27T21:39:55.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011731280000000001 2024-03-27T21:39:55.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:39:55.968 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937969+0000", 2024-03-27T21:39:55.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3285999999999999e-05 2024-03-27T21:39:55.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:39:55.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938040+0000", 2024-03-27T21:39:55.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0358000000000006e-05 2024-03-27T21:39:55.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:39:55.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939324+0000", 2024-03-27T21:39:55.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012844899999999999 2024-03-27T21:39:55.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:39:55.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943780+0000", 2024-03-27T21:39:55.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044555630000000001 2024-03-27T21:39:55.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:39:55.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943811+0000", 2024-03-27T21:39:55.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1415999999999999e-05 2024-03-27T21:39:55.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:39:55.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943832+0000", 2024-03-27T21:39:55.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0981000000000001e-05 2024-03-27T21:39:55.970 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:55.970 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:39:55.970 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:55.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.970 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:39:55.971 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:39:55.971 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 16.026157903000001, 2024-03-27T21:39:55.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071139740000000003, 2024-03-27T21:39:55.971 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:39:55.971 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:39:55.971 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:39:55.971 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:39:55.971 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:39:55.971 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-03-27T21:39:55.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.971 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:39:55.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:39:55.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:39:55.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:55.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:39:55.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:39:55.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:55.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:39:55.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936760+0000", 2024-03-27T21:39:55.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:39:55.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:39:55.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936765+0000", 2024-03-27T21:39:55.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6330000000000004e-06 2024-03-27T21:39:55.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:39:55.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936771+0000", 2024-03-27T21:39:55.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0560000000000001e-06 2024-03-27T21:39:55.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:39:55.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936776+0000", 2024-03-27T21:39:55.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1510000000000003e-06 2024-03-27T21:39:55.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:39:55.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938605+0000", 2024-03-27T21:39:55.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001829589 2024-03-27T21:39:55.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:39:55.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938657+0000", 2024-03-27T21:39:55.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1860000000000002e-05 2024-03-27T21:39:55.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:39:55.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938734+0000", 2024-03-27T21:39:55.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6639999999999998e-05 2024-03-27T21:39:55.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.975 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:39:55.975 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939562+0000", 2024-03-27T21:39:55.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082803100000000001 2024-03-27T21:39:55.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.975 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:39:55.975 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943838+0000", 2024-03-27T21:39:55.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042757050000000003 2024-03-27T21:39:55.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.975 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:39:55.975 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943859+0000", 2024-03-27T21:39:55.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1175e-05 2024-03-27T21:39:55.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:39:55.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943875+0000", 2024-03-27T21:39:55.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5744e-05 2024-03-27T21:39:55.976 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:55.976 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:39:55.976 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:55.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.976 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:39:55.976 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:39:55.976 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 15.882643851999999, 2024-03-27T21:39:55.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059712230000000003, 2024-03-27T21:39:55.977 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:39:55.977 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:39:55.977 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:39:55.977 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:39:55.977 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:39:55.977 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 112 2024-03-27T21:39:55.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.977 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:39:55.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.977 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:39:55.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:39:55.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:55.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:39:55.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:39:55.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:55.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:39:55.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080274+0000", 2024-03-27T21:39:55.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:39:55.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:39:55.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080318+0000", 2024-03-27T21:39:55.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3634999999999998e-05 2024-03-27T21:39:55.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:39:55.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080329+0000", 2024-03-27T21:39:55.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1232e-05 2024-03-27T21:39:55.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:39:55.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080333+0000", 2024-03-27T21:39:55.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6009999999999999e-06 2024-03-27T21:39:55.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:39:55.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083786+0000", 2024-03-27T21:39:55.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034528940000000002 2024-03-27T21:39:55.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:39:55.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083861+0000", 2024-03-27T21:39:55.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4893000000000003e-05 2024-03-27T21:39:55.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:39:55.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083981+0000", 2024-03-27T21:39:55.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012058299999999999 2024-03-27T21:39:55.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:39:55.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086148+0000", 2024-03-27T21:39:55.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021671469999999999 2024-03-27T21:39:55.981 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.981 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:39:55.981 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086195+0000", 2024-03-27T21:39:55.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6768999999999999e-05 2024-03-27T21:39:55.981 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.981 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:39:55.981 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086224+0000", 2024-03-27T21:39:55.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8758000000000001e-05 2024-03-27T21:39:55.981 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.981 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:39:55.981 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086246+0000", 2024-03-27T21:39:55.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2064000000000001e-05 2024-03-27T21:39:55.982 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:55.982 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:39:55.982 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:55.982 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.982 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.982 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:39:55.982 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:39:55.982 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 15.882580904999999, 2024-03-27T21:39:55.982 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066196320000000003, 2024-03-27T21:39:55.982 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:39:55.982 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:39:55.982 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:39:55.983 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:39:55.983 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:39:55.983 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113 2024-03-27T21:39:55.983 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.983 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:39:55.983 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.983 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:39:55.983 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:39:55.983 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:55.983 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.983 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.983 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:39:55.984 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:39:55.984 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:55.984 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.984 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.984 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:39:55.984 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080337+0000", 2024-03-27T21:39:55.984 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:39:55.984 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.984 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.984 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:39:55.984 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080340+0000", 2024-03-27T21:39:55.984 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6690000000000002e-06 2024-03-27T21:39:55.984 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.984 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.985 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:39:55.985 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080344+0000", 2024-03-27T21:39:55.985 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6550000000000002e-06 2024-03-27T21:39:55.985 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.985 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.985 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:39:55.985 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080348+0000", 2024-03-27T21:39:55.985 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9169999999999999e-06 2024-03-27T21:39:55.985 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.985 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.985 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:39:55.985 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084595+0000", 2024-03-27T21:39:55.985 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042471719999999996 2024-03-27T21:39:55.985 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.986 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.986 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:39:55.986 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084676+0000", 2024-03-27T21:39:55.986 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1147000000000001e-05 2024-03-27T21:39:55.986 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.986 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.986 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:39:55.986 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084796+0000", 2024-03-27T21:39:55.986 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012032900000000001 2024-03-27T21:39:55.986 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.986 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.986 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:39:55.986 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086190+0000", 2024-03-27T21:39:55.986 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013934399999999999 2024-03-27T21:39:55.987 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.987 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.987 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:39:55.987 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086905+0000", 2024-03-27T21:39:55.987 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071536199999999999 2024-03-27T21:39:55.987 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.987 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.987 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:39:55.987 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086937+0000", 2024-03-27T21:39:55.987 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2125000000000002e-05 2024-03-27T21:39:55.987 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.987 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.987 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:39:55.987 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086957+0000", 2024-03-27T21:39:55.987 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0157999999999999e-05 2024-03-27T21:39:55.988 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:55.988 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:39:55.988 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:55.988 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.988 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.988 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:39:55.988 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:39:55.988 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 15.882568189000001, 2024-03-27T21:39:55.988 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007405661, 2024-03-27T21:39:55.988 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:39:55.988 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:39:55.988 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:39:55.988 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:39:55.988 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:39:55.989 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 114 2024-03-27T21:39:55.989 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.989 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:39:55.989 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.989 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:39:55.989 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:39:55.989 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:55.989 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.989 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.989 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:39:55.989 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:39:55.989 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:55.989 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.989 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.989 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:39:55.990 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:39:55.990 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-03-27T21:39:55.990 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.990 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.990 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:39:55.990 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080353+0000", 2024-03-27T21:39:55.990 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.441e-06 2024-03-27T21:39:55.990 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.990 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.990 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:39:55.990 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080356+0000", 2024-03-27T21:39:55.990 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0769999999999999e-06 2024-03-27T21:39:55.990 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.990 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.991 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:39:55.991 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080358+0000", 2024-03-27T21:39:55.991 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3369999999999998e-06 2024-03-27T21:39:55.991 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.991 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.991 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:39:55.991 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085118+0000", 2024-03-27T21:39:55.991 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047603949999999997 2024-03-27T21:39:55.991 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.991 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.991 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:39:55.991 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085210+0000", 2024-03-27T21:39:55.991 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1983000000000003e-05 2024-03-27T21:39:55.991 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.991 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.992 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:39:55.992 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085332+0000", 2024-03-27T21:39:55.992 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121803 2024-03-27T21:39:55.992 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.992 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.992 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:39:55.992 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086292+0000", 2024-03-27T21:39:55.992 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095961400000000002 2024-03-27T21:39:55.992 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.992 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.992 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:39:55.992 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087710+0000", 2024-03-27T21:39:55.992 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014181459999999999 2024-03-27T21:39:55.992 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.993 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.993 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:39:55.993 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087736+0000", 2024-03-27T21:39:55.993 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6475e-05 2024-03-27T21:39:55.993 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.993 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.993 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:39:55.993 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087756+0000", 2024-03-27T21:39:55.993 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9626000000000001e-05 2024-03-27T21:39:55.993 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:55.993 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:39:55.993 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:55.993 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.993 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.994 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:39:55.994 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:39:55.994 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 15.882545754000001, 2024-03-27T21:39:55.994 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075011720000000004, 2024-03-27T21:39:55.994 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:39:55.994 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:39:55.994 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:39:55.994 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:39:55.994 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:39:55.994 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115 2024-03-27T21:39:55.994 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.994 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:39:55.994 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.994 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:39:55.995 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:39:55.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:55.995 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.995 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:39:55.995 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:39:55.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:55.995 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.995 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:39:55.995 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:39:55.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:39:55.995 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.996 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:39:55.996 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080458+0000", 2024-03-27T21:39:55.996 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5228000000000005e-05 2024-03-27T21:39:55.996 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.996 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.996 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:39:55.996 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080465+0000", 2024-03-27T21:39:55.996 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.238e-06 2024-03-27T21:39:55.996 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.996 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.996 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:39:55.996 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080468+0000", 2024-03-27T21:39:55.996 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5109999999999999e-06 2024-03-27T21:39:55.996 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.997 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.997 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:39:55.997 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086326+0000", 2024-03-27T21:39:55.997 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058579169999999998 2024-03-27T21:39:55.997 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.997 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.997 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:39:55.997 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086382+0000", 2024-03-27T21:39:55.997 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5309999999999997e-05 2024-03-27T21:39:55.997 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.997 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.997 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:39:55.997 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086457+0000", 2024-03-27T21:39:55.997 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5413000000000005e-05 2024-03-27T21:39:55.998 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.998 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.998 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:39:55.998 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087762+0000", 2024-03-27T21:39:55.998 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013052300000000001 2024-03-27T21:39:55.998 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.998 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.998 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:39:55.998 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087798+0000", 2024-03-27T21:39:55.998 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5614999999999997e-05 2024-03-27T21:39:55.998 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.998 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.998 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:39:55.998 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087840+0000", 2024-03-27T21:39:55.998 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2196000000000001e-05 2024-03-27T21:39:55.999 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.999 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.999 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:39:55.999 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087874+0000", 2024-03-27T21:39:55.999 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3785999999999998e-05 2024-03-27T21:39:55.999 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:55.999 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:39:55.999 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:55.999 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:55.999 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:55.999 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:39:55.999 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:39:55.999 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 15.882445670999999, 2024-03-27T21:39:56.000 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075050680000000002, 2024-03-27T21:39:56.000 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:39:56.000 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:39:56.000 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:39:56.000 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:39:56.000 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:39:56.000 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 116 2024-03-27T21:39:56.000 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.000 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:39:56.000 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.000 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:39:56.000 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:39:56.000 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.000 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.000 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.001 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:39:56.001 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:39:56.001 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.001 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.001 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.001 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:39:56.001 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080472+0000", 2024-03-27T21:39:56.001 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:39:56.001 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.001 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.001 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:39:56.001 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080475+0000", 2024-03-27T21:39:56.001 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.712e-06 2024-03-27T21:39:56.001 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.001 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.002 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:39:56.002 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080478+0000", 2024-03-27T21:39:56.002 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1070000000000001e-06 2024-03-27T21:39:56.002 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.002 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.002 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:39:56.002 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080481+0000", 2024-03-27T21:39:56.002 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4269999999999998e-06 2024-03-27T21:39:56.002 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.002 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.002 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:39:56.002 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086992+0000", 2024-03-27T21:39:56.002 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065114099999999996 2024-03-27T21:39:56.002 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.003 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.003 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:39:56.003 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087064+0000", 2024-03-27T21:39:56.003 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1927000000000004e-05 2024-03-27T21:39:56.003 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.003 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.003 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:39:56.003 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087188+0000", 2024-03-27T21:39:56.003 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123585 2024-03-27T21:39:56.003 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.003 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.003 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:39:56.003 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087766+0000", 2024-03-27T21:39:56.003 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057814300000000004 2024-03-27T21:39:56.004 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.004 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.004 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:39:56.004 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087934+0000", 2024-03-27T21:39:56.004 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000167894 2024-03-27T21:39:56.004 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.004 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.004 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:39:56.004 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087958+0000", 2024-03-27T21:39:56.004 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4663000000000001e-05 2024-03-27T21:39:56.004 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.004 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.004 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:39:56.004 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087978+0000", 2024-03-27T21:39:56.005 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9636e-05 2024-03-27T21:39:56.005 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.005 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:39:56.005 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.005 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.005 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.005 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:39:56.005 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:39:56.005 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 15.882413549000001, 2024-03-27T21:39:56.005 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076850679999999998, 2024-03-27T21:39:56.005 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:39:56.005 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:39:56.005 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:39:56.006 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:39:56.006 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:39:56.006 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 117 2024-03-27T21:39:56.006 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.006 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:39:56.006 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.006 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:39:56.006 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:39:56.006 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.006 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.006 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.006 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:39:56.006 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:39:56.006 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.007 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.007 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:39:56.007 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:39:56.007 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:39:56.007 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.007 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:39:56.007 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080508+0000", 2024-03-27T21:39:56.007 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.298e-06 2024-03-27T21:39:56.007 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.007 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:39:56.007 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080515+0000", 2024-03-27T21:39:56.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0790000000000003e-06 2024-03-27T21:39:56.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.008 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:39:56.008 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080518+0000", 2024-03-27T21:39:56.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8509999999999999e-06 2024-03-27T21:39:56.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.008 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:39:56.008 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087444+0000", 2024-03-27T21:39:56.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0069259120000000002 2024-03-27T21:39:56.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.008 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:39:56.009 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087491+0000", 2024-03-27T21:39:56.009 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7043999999999999e-05 2024-03-27T21:39:56.009 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.009 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.009 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:39:56.009 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087564+0000", 2024-03-27T21:39:56.009 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3603999999999995e-05 2024-03-27T21:39:56.009 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.009 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.009 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:39:56.009 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087902+0000", 2024-03-27T21:39:56.009 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033734100000000001 2024-03-27T21:39:56.009 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.009 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.010 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:39:56.010 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088117+0000", 2024-03-27T21:39:56.010 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000215164 2024-03-27T21:39:56.010 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.010 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.010 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:39:56.010 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088160+0000", 2024-03-27T21:39:56.010 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2933999999999999e-05 2024-03-27T21:39:56.010 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.010 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.010 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:39:56.010 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088190+0000", 2024-03-27T21:39:56.010 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0139999999999999e-05 2024-03-27T21:39:56.011 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.011 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:39:56.011 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.011 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.011 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.011 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:39:56.011 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:39:56.011 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 15.46613316, 2024-03-27T21:39:56.011 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052465439999999997, 2024-03-27T21:39:56.011 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:39:56.011 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:39:56.011 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:39:56.011 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:39:56.011 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:39:56.011 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 133 2024-03-27T21:39:56.012 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.012 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:39:56.012 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.012 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:39:56.012 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:39:56.012 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.012 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.012 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.012 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:39:56.012 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:39:56.012 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.012 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.012 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.012 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:39:56.012 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:39:56.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:39:56.013 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:39:56.013 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496788+0000", 2024-03-27T21:39:56.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-03-27T21:39:56.013 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:39:56.013 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496793+0000", 2024-03-27T21:39:56.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4790000000000004e-06 2024-03-27T21:39:56.013 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:39:56.014 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496796+0000", 2024-03-27T21:39:56.014 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1e-06 2024-03-27T21:39:56.014 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.014 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.014 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:39:56.014 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500447+0000", 2024-03-27T21:39:56.014 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036504459999999999 2024-03-27T21:39:56.014 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.014 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.014 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:39:56.014 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500492+0000", 2024-03-27T21:39:56.014 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5765e-05 2024-03-27T21:39:56.014 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.015 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:39:56.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500562+0000", 2024-03-27T21:39:56.015 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9206000000000001e-05 2024-03-27T21:39:56.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.015 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:39:56.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501959+0000", 2024-03-27T21:39:56.015 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001397187 2024-03-27T21:39:56.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.015 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:39:56.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501984+0000", 2024-03-27T21:39:56.015 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.552e-05 2024-03-27T21:39:56.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:39:56.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502010+0000", 2024-03-27T21:39:56.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5232000000000002e-05 2024-03-27T21:39:56.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.016 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:39:56.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502032+0000", 2024-03-27T21:39:56.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2337999999999999e-05 2024-03-27T21:39:56.016 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.016 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:39:56.016 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.016 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.016 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:39:56.017 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:39:56.017 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 15.466074502, 2024-03-27T21:39:56.017 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052892119999999997, 2024-03-27T21:39:56.017 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:39:56.017 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:39:56.017 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:39:56.017 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:39:56.017 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:39:56.017 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 134 2024-03-27T21:39:56.017 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.017 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:39:56.017 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.017 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:39:56.017 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:39:56.017 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.018 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.018 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.018 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:39:56.018 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:39:56.018 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.018 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.018 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.018 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:39:56.018 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:39:56.018 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:39:56.018 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.018 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.018 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:39:56.018 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496929+0000", 2024-03-27T21:39:56.019 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5098000000000004e-05 2024-03-27T21:39:56.019 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.019 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.019 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:39:56.019 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496937+0000", 2024-03-27T21:39:56.019 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9559999999999997e-06 2024-03-27T21:39:56.019 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.019 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.019 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:39:56.019 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496942+0000", 2024-03-27T21:39:56.019 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6350000000000003e-06 2024-03-27T21:39:56.019 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.019 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.020 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:39:56.020 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500749+0000", 2024-03-27T21:39:56.020 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038068529999999998 2024-03-27T21:39:56.020 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.020 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:39:56.020 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500792+0000", 2024-03-27T21:39:56.020 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2823e-05 2024-03-27T21:39:56.020 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.020 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:39:56.020 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500864+0000", 2024-03-27T21:39:56.020 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1715e-05 2024-03-27T21:39:56.020 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.020 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:39:56.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502068+0000", 2024-03-27T21:39:56.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012044860000000001 2024-03-27T21:39:56.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:39:56.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502073+0000", 2024-03-27T21:39:56.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5700000000000003e-06 2024-03-27T21:39:56.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:39:56.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502098+0000", 2024-03-27T21:39:56.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5120999999999999e-05 2024-03-27T21:39:56.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:39:56.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502133+0000", 2024-03-27T21:39:56.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5306999999999999e-05 2024-03-27T21:39:56.022 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.022 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:39:56.022 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.022 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.022 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.022 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:39:56.022 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:39:56.022 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 15.465969195, 2024-03-27T21:39:56.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062529040000000001, 2024-03-27T21:39:56.022 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:39:56.022 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:39:56.022 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:39:56.022 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:39:56.023 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:39:56.023 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135 2024-03-27T21:39:56.023 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.023 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:39:56.023 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.023 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:39:56.023 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:39:56.023 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.023 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.023 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.023 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:39:56.023 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:39:56.023 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.023 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.024 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.024 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:39:56.024 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:39:56.024 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:39:56.024 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.024 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.024 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:39:56.024 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496953+0000", 2024-03-27T21:39:56.024 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0709999999999996e-06 2024-03-27T21:39:56.024 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.024 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.024 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:39:56.024 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496961+0000", 2024-03-27T21:39:56.024 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1300000000000001e-06 2024-03-27T21:39:56.025 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.025 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.025 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:39:56.025 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496968+0000", 2024-03-27T21:39:56.025 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0999999999999998e-06 2024-03-27T21:39:56.025 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.025 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.025 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:39:56.025 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501244+0000", 2024-03-27T21:39:56.025 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042756310000000002 2024-03-27T21:39:56.025 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.025 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.025 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:39:56.025 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501377+0000", 2024-03-27T21:39:56.026 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013348799999999999 2024-03-27T21:39:56.026 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.026 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.026 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:39:56.026 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501455+0000", 2024-03-27T21:39:56.026 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7410000000000006e-05 2024-03-27T21:39:56.026 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.026 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.026 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:39:56.026 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502138+0000", 2024-03-27T21:39:56.026 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068386599999999999 2024-03-27T21:39:56.026 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.026 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.026 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:39:56.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503152+0000", 2024-03-27T21:39:56.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010139680000000001 2024-03-27T21:39:56.027 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.027 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:39:56.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503180+0000", 2024-03-27T21:39:56.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7426000000000001e-05 2024-03-27T21:39:56.027 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.027 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:39:56.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503202+0000", 2024-03-27T21:39:56.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2463000000000002e-05 2024-03-27T21:39:56.027 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.027 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:39:56.028 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.028 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.028 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.028 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:39:56.028 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:39:56.028 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 15.465945057000001, 2024-03-27T21:39:56.028 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066253680000000004, 2024-03-27T21:39:56.028 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:39:56.028 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:39:56.028 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:39:56.028 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:39:56.028 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:39:56.028 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 136 2024-03-27T21:39:56.028 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.028 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:39:56.028 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:39:56.029 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:39:56.029 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.029 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.029 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:39:56.029 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:39:56.029 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.029 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.029 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:39:56.029 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:39:56.029 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:39:56.029 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.029 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:39:56.030 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496977+0000", 2024-03-27T21:39:56.030 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5049999999999999e-06 2024-03-27T21:39:56.030 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.030 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:39:56.030 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496984+0000", 2024-03-27T21:39:56.030 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1520000000000003e-06 2024-03-27T21:39:56.030 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.030 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:39:56.030 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496990+0000", 2024-03-27T21:39:56.030 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-03-27T21:39:56.030 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.031 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:39:56.031 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501669+0000", 2024-03-27T21:39:56.031 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046793859999999998 2024-03-27T21:39:56.031 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.031 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.031 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:39:56.031 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501714+0000", 2024-03-27T21:39:56.031 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5154999999999997e-05 2024-03-27T21:39:56.031 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.031 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.031 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:39:56.031 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501783+0000", 2024-03-27T21:39:56.031 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8454999999999994e-05 2024-03-27T21:39:56.031 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.032 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.032 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:39:56.032 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503226+0000", 2024-03-27T21:39:56.032 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001443441 2024-03-27T21:39:56.032 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.032 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.032 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:39:56.032 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503540+0000", 2024-03-27T21:39:56.032 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031347100000000001 2024-03-27T21:39:56.032 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.032 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.032 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:39:56.032 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503563+0000", 2024-03-27T21:39:56.032 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3223000000000001e-05 2024-03-27T21:39:56.033 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.033 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.033 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:39:56.033 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503599+0000", 2024-03-27T21:39:56.033 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6105000000000001e-05 2024-03-27T21:39:56.033 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.033 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:39:56.033 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.033 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.033 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.033 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:39:56.033 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:39:56.033 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 15.465922247, 2024-03-27T21:39:56.033 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074261320000000002, 2024-03-27T21:39:56.034 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:39:56.034 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:39:56.034 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:39:56.034 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:39:56.034 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:39:56.034 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 137 2024-03-27T21:39:56.034 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.034 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:39:56.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.034 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:39:56.034 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:39:56.034 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.034 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.035 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:39:56.035 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:39:56.035 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.035 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:39:56.035 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:39:56.035 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:39:56.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.035 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:39:56.035 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497077+0000", 2024-03-27T21:39:56.035 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1599000000000003e-05 2024-03-27T21:39:56.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.036 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.036 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:39:56.036 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497085+0000", 2024-03-27T21:39:56.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6109999999999996e-06 2024-03-27T21:39:56.036 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.036 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.036 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:39:56.036 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497092+0000", 2024-03-27T21:39:56.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.742e-06 2024-03-27T21:39:56.036 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.036 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.036 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:39:56.036 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502159+0000", 2024-03-27T21:39:56.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050666870000000003 2024-03-27T21:39:56.036 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.036 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.037 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:39:56.037 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502200+0000", 2024-03-27T21:39:56.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1919999999999998e-05 2024-03-27T21:39:56.037 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.037 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.037 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:39:56.037 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502280+0000", 2024-03-27T21:39:56.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9417000000000002e-05 2024-03-27T21:39:56.037 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.037 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.037 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:39:56.037 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503605+0000", 2024-03-27T21:39:56.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001325641 2024-03-27T21:39:56.037 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.037 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.038 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:39:56.038 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504361+0000", 2024-03-27T21:39:56.038 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075555399999999997 2024-03-27T21:39:56.038 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.038 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.038 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:39:56.038 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504386+0000", 2024-03-27T21:39:56.038 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5369999999999999e-05 2024-03-27T21:39:56.038 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.038 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.038 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:39:56.038 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504422+0000", 2024-03-27T21:39:56.038 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6019000000000002e-05 2024-03-27T21:39:56.039 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.039 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:39:56.039 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.039 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.039 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.039 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:39:56.039 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:39:56.039 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 15.46581845, 2024-03-27T21:39:56.039 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074059750000000004, 2024-03-27T21:39:56.039 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:39:56.039 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:39:56.039 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:39:56.039 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:39:56.039 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:39:56.040 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-03-27T21:39:56.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.040 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:39:56.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:39:56.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:39:56.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:39:56.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:39:56.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:39:56.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:39:56.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:39:56.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:39:56.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497103+0000", 2024-03-27T21:39:56.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.856e-06 2024-03-27T21:39:56.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:39:56.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497121+0000", 2024-03-27T21:39:56.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7190999999999999e-05 2024-03-27T21:39:56.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:39:56.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497126+0000", 2024-03-27T21:39:56.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3770000000000004e-06 2024-03-27T21:39:56.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:39:56.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502833+0000", 2024-03-27T21:39:56.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057073510000000003 2024-03-27T21:39:56.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:39:56.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502881+0000", 2024-03-27T21:39:56.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7961999999999999e-05 2024-03-27T21:39:56.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:39:56.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502949+0000", 2024-03-27T21:39:56.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7957e-05 2024-03-27T21:39:56.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:39:56.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504430+0000", 2024-03-27T21:39:56.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001480763 2024-03-27T21:39:56.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:39:56.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504459+0000", 2024-03-27T21:39:56.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8983999999999999e-05 2024-03-27T21:39:56.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.044 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:39:56.044 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504486+0000", 2024-03-27T21:39:56.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6883999999999999e-05 2024-03-27T21:39:56.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.044 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:39:56.044 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504506+0000", 2024-03-27T21:39:56.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0214e-05 2024-03-27T21:39:56.044 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.044 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:39:56.044 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.044 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:39:56.045 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:39:56.045 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 15.465786616000001, 2024-03-27T21:39:56.045 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074754859999999999, 2024-03-27T21:39:56.045 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:39:56.045 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:39:56.045 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:39:56.045 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:39:56.045 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:39:56.045 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-03-27T21:39:56.045 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.045 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:39:56.045 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.045 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:39:56.045 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:39:56.045 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:39:56.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:39:56.046 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:39:56.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:39:56.046 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:39:56.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:39:56.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497137+0000", 2024-03-27T21:39:56.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0730000000000004e-06 2024-03-27T21:39:56.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:39:56.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497143+0000", 2024-03-27T21:39:56.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3759999999999996e-06 2024-03-27T21:39:56.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:39:56.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497150+0000", 2024-03-27T21:39:56.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.002e-06 2024-03-27T21:39:56.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:39:56.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503249+0000", 2024-03-27T21:39:56.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060989750000000004 2024-03-27T21:39:56.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:39:56.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503291+0000", 2024-03-27T21:39:56.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1563999999999997e-05 2024-03-27T21:39:56.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:39:56.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503376+0000", 2024-03-27T21:39:56.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5752000000000002e-05 2024-03-27T21:39:56.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:39:56.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504545+0000", 2024-03-27T21:39:56.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169172 2024-03-27T21:39:56.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:39:56.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504550+0000", 2024-03-27T21:39:56.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7419999999999997e-06 2024-03-27T21:39:56.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:39:56.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504574+0000", 2024-03-27T21:39:56.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4124000000000001e-05 2024-03-27T21:39:56.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:39:56.050 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504607+0000", 2024-03-27T21:39:56.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3118999999999999e-05 2024-03-27T21:39:56.050 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.050 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:39:56.050 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.050 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:39:56.050 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:39:56.050 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 15.46576295, 2024-03-27T21:39:56.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078150589999999992, 2024-03-27T21:39:56.050 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:39:56.051 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:39:56.051 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:39:56.051 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:39:56.051 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:39:56.051 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-03-27T21:39:56.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.051 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:39:56.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:39:56.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:39:56.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:39:56.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:39:56.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:39:56.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497155+0000", 2024-03-27T21:39:56.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:39:56.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:39:56.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497230+0000", 2024-03-27T21:39:56.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4789000000000005e-05 2024-03-27T21:39:56.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:39:56.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497241+0000", 2024-03-27T21:39:56.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1442e-05 2024-03-27T21:39:56.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:39:56.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497255+0000", 2024-03-27T21:39:56.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3087e-05 2024-03-27T21:39:56.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:39:56.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503630+0000", 2024-03-27T21:39:56.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063755269999999998 2024-03-27T21:39:56.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:39:56.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503673+0000", 2024-03-27T21:39:56.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2616e-05 2024-03-27T21:39:56.054 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.054 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:39:56.054 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503738+0000", 2024-03-27T21:39:56.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5114e-05 2024-03-27T21:39:56.054 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.054 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:39:56.054 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504907+0000", 2024-03-27T21:39:56.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169432 2024-03-27T21:39:56.054 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.054 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:39:56.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504926+0000", 2024-03-27T21:39:56.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9086999999999999e-05 2024-03-27T21:39:56.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:39:56.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504950+0000", 2024-03-27T21:39:56.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3598999999999999e-05 2024-03-27T21:39:56.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:39:56.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504971+0000", 2024-03-27T21:39:56.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0772999999999999e-05 2024-03-27T21:39:56.055 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.055 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:39:56.055 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.056 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:39:56.056 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:39:56.056 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 15.465655910000001, 2024-03-27T21:39:56.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082962590000000003, 2024-03-27T21:39:56.056 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:39:56.056 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:39:56.056 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:39:56.056 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:39:56.056 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:39:56.056 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-03-27T21:39:56.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.056 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:39:56.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.057 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:39:56.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:39:56.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.057 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:39:56.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:39:56.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.057 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:39:56.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497262+0000", 2024-03-27T21:39:56.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:39:56.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:39:56.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497267+0000", 2024-03-27T21:39:56.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.955e-06 2024-03-27T21:39:56.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:39:56.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497275+0000", 2024-03-27T21:39:56.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.824e-06 2024-03-27T21:39:56.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:39:56.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497280+0000", 2024-03-27T21:39:56.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8230000000000002e-06 2024-03-27T21:39:56.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:39:56.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504628+0000", 2024-03-27T21:39:56.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007348561 2024-03-27T21:39:56.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:39:56.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504673+0000", 2024-03-27T21:39:56.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4313000000000001e-05 2024-03-27T21:39:56.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:39:56.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504745+0000", 2024-03-27T21:39:56.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2303000000000002e-05 2024-03-27T21:39:56.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:39:56.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505358+0000", 2024-03-27T21:39:56.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061281300000000003 2024-03-27T21:39:56.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:39:56.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505503+0000", 2024-03-27T21:39:56.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000145134 2024-03-27T21:39:56.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:39:56.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505526+0000", 2024-03-27T21:39:56.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3312e-05 2024-03-27T21:39:56.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.061 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:39:56.061 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505559+0000", 2024-03-27T21:39:56.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2821999999999998e-05 2024-03-27T21:39:56.061 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.061 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:39:56.061 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.061 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:39:56.061 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:39:56.061 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.856433671, 2024-03-27T21:39:56.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053753339999999998, 2024-03-27T21:39:56.062 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:39:56.062 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:39:56.062 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:39:56.062 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:39:56.062 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:39:56.062 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159 2024-03-27T21:39:56.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.062 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:39:56.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:39:56.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:39:56.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:39:56.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:39:56.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:39:56.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:39:56.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:39:56.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:39:56.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:39:56.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106488+0000", 2024-03-27T21:39:56.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9830000000000001e-06 2024-03-27T21:39:56.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:39:56.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106491+0000", 2024-03-27T21:39:56.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3459999999999998e-06 2024-03-27T21:39:56.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:39:56.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106493+0000", 2024-03-27T21:39:56.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.509e-06 2024-03-27T21:39:56.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:39:56.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110636+0000", 2024-03-27T21:39:56.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041427260000000002 2024-03-27T21:39:56.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:39:56.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110683+0000", 2024-03-27T21:39:56.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6557000000000002e-05 2024-03-27T21:39:56.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:39:56.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110755+0000", 2024-03-27T21:39:56.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2119000000000005e-05 2024-03-27T21:39:56.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:39:56.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111415+0000", 2024-03-27T21:39:56.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066065199999999996 2024-03-27T21:39:56.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:39:56.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111793+0000", 2024-03-27T21:39:56.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037732100000000001 2024-03-27T21:39:56.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:39:56.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111829+0000", 2024-03-27T21:39:56.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418999999999998e-05 2024-03-27T21:39:56.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:39:56.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:39:56.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:39:56.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111860+0000", 2024-03-27T21:39:56.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1003000000000001e-05 2024-03-27T21:39:56.067 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.067 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:39:56.067 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.067 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:39:56.067 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:39:56.067 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:39:56.068 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:39:56.069 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:39:56.082+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:39:56.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.109+0000 7f181cff9640 10 monclient: tick 2024-03-27T21:39:56.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.109+0000 7f181cff9640 10 monclient: _check_auth_tickets 2024-03-27T21:39:56.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.109+0000 7f181cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:56.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.109+0000 7f181cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:56.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.157+0000 7f182cceb640 1 -- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] conn(0x7f1828148430 msgr2=0x7f1828148850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6824/34604 2024-03-27T21:39:56.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.157+0000 7f182cceb640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] conn(0x7f1828148430 0x7f1828148850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-27T21:39:56.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.157+0000 7f181dffb640 1 client.4394.objecter ms_handle_refused on osd.2 2024-03-27T21:39:56.169 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:39:56.183+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:39:56.176 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:56.176 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:56.184 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:39:56.269 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:39:56.283+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:39:56.331 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:56.331 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:56.341 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:39:56.370 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:39:56.383+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:39:56.396 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 2.1 2024-03-27T21:39:56.450 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:56.450 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:56.460 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:39:56.471 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:39:56.485+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:39:56.566 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:56.567 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:56.572 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:39:56.586+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:39:56.575 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:39:56.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.626+0000 7f6bd6ffd640 1 -- 172.21.15.60:0/3806520975 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v24) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6bc8048090 con 0x7f6be013b020 2024-03-27T21:39:56.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.626+0000 7f6bd6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:39:56.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.626+0000 7f6bd6ffd640 10 monclient: _finish_command 2 = system:0 nodeep-scrub is set 2024-03-27T21:39:56.613 INFO:teuthology.orchestra.run.smithi060.stderr:nodeep-scrub is set 2024-03-27T21:39:56.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.628+0000 7f6be4e2b640 10 librados: watch_flush enter 2024-03-27T21:39:56.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.628+0000 7f6be4e2b640 10 librados: watch_flush exit 2024-03-27T21:39:56.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.628+0000 7f6be4e2b640 20 client.4400.objecter shutdown clearing up homeless session... 2024-03-27T21:39:56.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.628+0000 7f6be4e2b640 10 client.4400.objecter successfully canceled tick 2024-03-27T21:39:56.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.628+0000 7f6be4e2b640 1 -- 172.21.15.60:0/3806520975 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6bc006c590 msgr2=0x7f6bc006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:56.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.628+0000 7f6be4e2b640 1 --2- 172.21.15.60:0/3806520975 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6bc006c590 0x7f6bc006ea50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0007fd0 tx=0x7f6bd0007de0 comp rx=0 tx=0).stop 2024-03-27T21:39:56.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.628+0000 7f6be4e2b640 10 monclient: shutdown 2024-03-27T21:39:56.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.628+0000 7f6be4e2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:56.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.628+0000 7f6be4e2b640 1 -- 172.21.15.60:0/3806520975 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6be013b020 msgr2=0x7f6be01d6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:56.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.628+0000 7f6be4e2b640 1 --2- 172.21.15.60:0/3806520975 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6be013b020 0x7f6be01d6180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6bc802bee0 tx=0x7f6bc80046c0 comp rx=0 tx=0).stop 2024-03-27T21:39:56.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.628+0000 7f6be4e2b640 1 -- 172.21.15.60:0/3806520975 shutdown_connections 2024-03-27T21:39:56.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.628+0000 7f6be4e2b640 1 --2- 172.21.15.60:0/3806520975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6be01416f0 0x7f6be01d66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:56.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.628+0000 7f6be4e2b640 1 --2- 172.21.15.60:0/3806520975 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6be013b020 0x7f6be01d6180 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:56.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.628+0000 7f6be4e2b640 1 --2- 172.21.15.60:0/3806520975 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6bc006c590 0x7f6bc006ea50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:56.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.628+0000 7f6be4e2b640 1 --2- 172.21.15.60:0/3806520975 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6be00b40b0 0x7f6be0139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:56.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.628+0000 7f6be4e2b640 1 -- 172.21.15.60:0/3806520975 >> 172.21.15.60:0/3806520975 conn(0x7f6be00608e0 msgr2=0x7f6be0060cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:56.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.628+0000 7f6be4e2b640 1 -- 172.21.15.60:0/3806520975 shutdown_connections 2024-03-27T21:39:56.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.628+0000 7f6be4e2b640 1 -- 172.21.15.60:0/3806520975 wait complete. 2024-03-27T21:39:56.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:56.628+0000 7f6be4e2b640 1 librados: shutdown 2024-03-27T21:39:56.673 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:39:56.686+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:39:56.719 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:56.719 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:56.728 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:39:56.774 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:39:56.787+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:39:56.842 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:56.842 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:56.850 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:39:56.891 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:39:56.905+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:39:56.976 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:39:56.990+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:39:56.997 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:39:56.997 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:39:56.997 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:39:56.997 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:39:57.007 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:39:57.077 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:39:57.091+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:39:57.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:57.109+0000 7f181cff9640 10 monclient: tick 2024-03-27T21:39:57.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:57.109+0000 7f181cff9640 10 monclient: _check_auth_tickets 2024-03-27T21:39:57.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:57.109+0000 7f181cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:57.120 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:39:57.120 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:39:57.121 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:39:57.121 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:39:57.121 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:39:57.130 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:39:57.178 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:39:57.192+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:39:57.279 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:39:57.292+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:39:57.287 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:39:57.287 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:39:57.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:39:57.287 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:39:57.287 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:39:57.296 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:39:57.380 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:39:57.393+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:39:57.413 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:39:57.413 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:39:57.414 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:39:57.414 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:39:57.422 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:39:57.491 INFO:teuthology.orchestra.run.smithi060.stdout: marking collection for removal 2024-03-27T21:39:57.492 INFO:teuthology.orchestra.run.smithi060.stdout:setting '_remove' omap key 2024-03-27T21:39:57.496 INFO:teuthology.orchestra.run.smithi060.stdout:finish_remove_pgs 2.1_head removing 2.1 2024-03-27T21:39:57.497 INFO:teuthology.orchestra.run.smithi060.stdout:Remove successful 2024-03-27T21:39:57.506 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:39:57.520+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:39:57.568 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:39:57.568 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:39:57.568 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:39:57.568 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:39:57.568 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:39:57.577 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:39:57.618 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:39:57.631+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:39:57.683 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:39:57.697+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:39:57.722 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:39:57.722 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:39:57.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:39:57.722 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:39:57.722 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:39:57.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:39:57.784 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:39:57.797+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:39:57.836 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:39:57.836 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:39:57.836 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:39:57.836 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:39:57.844 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:39:57.885 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:39:57.899+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:39:57.986 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:39:57.999+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:39:57.990 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:39:57.990 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:39:57.990 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:39:57.990 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:39:57.990 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:39:57.999 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:39:58.015 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.1.2 2024-03-27T21:39:58.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.109+0000 7f181cff9640 10 monclient: tick 2024-03-27T21:39:58.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.109+0000 7f181cff9640 10 monclient: _check_auth_tickets 2024-03-27T21:39:58.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.109+0000 7f181cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:58.096 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:39:58.110+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:39:58.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181e7fc640 10 client.4394.objecter tick 2024-03-27T21:39:58.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181e7fc640 10 client.4394.objecter pinging osd that serves command tid 5 (osd.2) 2024-03-27T21:39:58.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181e7fc640 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:58.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181e7fc640 10 monclient: _renew_subs 2024-03-27T21:39:58.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181e7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:58.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181e7fc640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=22}) v3 -- 0x7f17f8004e60 con 0x7f182806fce0 2024-03-27T21:39:58.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181e7fc640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] -- ping magic: 0 v1 -- 0x7f17f80051f0 con 0x7f1828148430 2024-03-27T21:39:58.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== mon.2 v2:172.21.15.60:3301/0 7 ==== osd_map(22..24 src has 1..24) v4 ==== 908+0+0 (secure 0 0 0) 0x7f182407dfa0 con 0x7f182806fce0 2024-03-27T21:39:58.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 10 client.4394.objecter ms_dispatch 0x7f1828000b90 osd_map(22..24 src has 1..24) v4 2024-03-27T21:39:58.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 3 client.4394.objecter handle_osd_map got epochs [22,24] > 21 2024-03-27T21:39:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 3 client.4394.objecter handle_osd_map decoding incremental epoch 22 2024-03-27T21:39:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 10 client.4394.objecter checking command 5 2024-03-27T21:39:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 10 client.4394.objecter _check_command_map_dne tid 5 current 22 map_dne_bound 0 2024-03-27T21:39:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1828066a60 con 0x7f182806fce0 2024-03-27T21:39:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 10 client.4394.objecter close_session for osd.2 2024-03-27T21:39:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] conn(0x7f1828148430 msgr2=0x7f1828148850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T21:39:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6824/34604,v1:172.21.15.60:6825/34604] conn(0x7f1828148430 0x7f1828148850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 10 client.4394.objecter command_op 5 2024-03-27T21:39:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 20 client.4394.objecter put_session s=0x7f18280b36f0 osd=2 2 2024-03-27T21:39:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 15 client.4394.objecter _session_command_op_remove 2 5 2024-03-27T21:39:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 20 client.4394.objecter put_session s=0x7f18280b36f0 osd=2 1 2024-03-27T21:39:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 15 client.4394.objecter _session_command_op_assign -1 5 2024-03-27T21:39:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 3 client.4394.objecter handle_osd_map decoding incremental epoch 23 2024-03-27T21:39:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 10 client.4394.objecter checking command 5 2024-03-27T21:39:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 10 client.4394.objecter _check_command_map_dne tid 5 current 23 map_dne_bound 0 2024-03-27T21:39:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 3 client.4394.objecter handle_osd_map decoding incremental epoch 24 2024-03-27T21:39:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 10 client.4394.objecter checking command 5 2024-03-27T21:39:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 10 client.4394.objecter _check_command_map_dne tid 5 current 24 map_dne_bound 0 2024-03-27T21:39:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 20 client.4394.objecter dump_active .. 1 homeless 2024-03-27T21:39:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== mon.2 v2:172.21.15.60:3301/0 8 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1824082560 con 0x7f182806fce0 2024-03-27T21:39:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181dffb640 10 monclient: handle_get_version_reply finishing 1 version 24 2024-03-27T21:39:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181f7fe640 10 client.4394.objecter _check_command_map_dne tid 5 current 24 map_dne_bound 24 2024-03-27T21:39:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181f7fe640 10 client.4394.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:39:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f181f7fe640 15 client.4394.objecter _session_command_op_remove -1 5 2024-03-27T21:39:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:39:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f182f750640 10 client.4394.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:58.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign -1 6 2024-03-27T21:39:58.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f182f750640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f1828148020 0x7f182802be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:58.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f1828147eb0 osd=3 3 2024-03-27T21:39:58.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f1828147eb0 osd=3 3 2024-03-27T21:39:58.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f1828147eb0 osd=3 3 2024-03-27T21:39:58.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f182f750640 15 client.4394.objecter _session_command_op_remove -1 6 2024-03-27T21:39:58.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f182f750640 20 client.4394.objecter get_session s=0x7f1828147eb0 osd=3 3 2024-03-27T21:39:58.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign 3 6 2024-03-27T21:39:58.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f1828147eb0 osd=3 4 2024-03-27T21:39:58.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f182f750640 10 client.4394.objecter _send_command 6 2024-03-27T21:39:58.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.112+0000 7f182f750640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f182802c380 con 0x7f1828148020 2024-03-27T21:39:58.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.113+0000 7f182cceb640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f1828148020 0x7f182802be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:58.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.113+0000 7f182cceb640 10 monclient: get_auth_request con 0x7f1828148020 auth_method 0 2024-03-27T21:39:58.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.113+0000 7f181dffb640 10 client.4394.objecter ms_handle_connect 0x7f1828148020 2024-03-27T21:39:58.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.113+0000 7f182cceb640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f1828148020 0x7f182802be40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.114+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f182802c380 con 0x7f1828148020 2024-03-27T21:39:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.114+0000 7f181dffb640 10 client.4394.objecter ms_dispatch 0x7f1828000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:39:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.114+0000 7f181dffb640 10 client.4394.objecter _finish_command 6 = system:0 2024-03-27T21:39:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.114+0000 7f181dffb640 20 client.4394.objecter put_session s=0x7f1828147eb0 osd=3 4 2024-03-27T21:39:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.114+0000 7f181dffb640 15 client.4394.objecter _session_command_op_remove 3 6 2024-03-27T21:39:58.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.129+0000 7f182f750640 10 client.4394.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:39:58.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.129+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign -1 7 2024-03-27T21:39:58.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.129+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f1828147eb0 osd=3 3 2024-03-27T21:39:58.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.129+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f1828147eb0 osd=3 3 2024-03-27T21:39:58.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.129+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f1828147eb0 osd=3 3 2024-03-27T21:39:58.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.129+0000 7f182f750640 15 client.4394.objecter _session_command_op_remove -1 7 2024-03-27T21:39:58.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.129+0000 7f182f750640 20 client.4394.objecter get_session s=0x7f1828147eb0 osd=3 3 2024-03-27T21:39:58.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.129+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign 3 7 2024-03-27T21:39:58.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.129+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f1828147eb0 osd=3 4 2024-03-27T21:39:58.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.129+0000 7f182f750640 10 client.4394.objecter _send_command 7 2024-03-27T21:39:58.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.129+0000 7f182f750640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f18281edc40 con 0x7f1828148020 2024-03-27T21:39:58.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.132+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f18281edc40 con 0x7f1828148020 2024-03-27T21:39:58.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.132+0000 7f181dffb640 10 client.4394.objecter ms_dispatch 0x7f1828000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:39:58.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.132+0000 7f181dffb640 10 client.4394.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:39:58.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.132+0000 7f181dffb640 20 client.4394.objecter put_session s=0x7f1828147eb0 osd=3 4 2024-03-27T21:39:58.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.132+0000 7f181dffb640 15 client.4394.objecter _session_command_op_remove 3 7 2024-03-27T21:39:58.119 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:39:58.120 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:39:58.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.132+0000 7f182f750640 10 client.4394.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:58.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.132+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign -1 8 2024-03-27T21:39:58.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.132+0000 7f182f750640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f182802b520 0x7f18281f0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:58.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.132+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f182802b3b0 osd=4 3 2024-03-27T21:39:58.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.132+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f182802b3b0 osd=4 3 2024-03-27T21:39:58.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.132+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f182802b3b0 osd=4 3 2024-03-27T21:39:58.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.132+0000 7f182f750640 15 client.4394.objecter _session_command_op_remove -1 8 2024-03-27T21:39:58.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.132+0000 7f182f750640 20 client.4394.objecter get_session s=0x7f182802b3b0 osd=4 3 2024-03-27T21:39:58.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.132+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign 4 8 2024-03-27T21:39:58.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.132+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f182802b3b0 osd=4 4 2024-03-27T21:39:58.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.132+0000 7f182f750640 10 client.4394.objecter _send_command 8 2024-03-27T21:39:58.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.132+0000 7f182f750640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18281eb780 con 0x7f182802b520 2024-03-27T21:39:58.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.133+0000 7f182dced640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f182802b520 0x7f18281f0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:58.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.133+0000 7f182dced640 10 monclient: get_auth_request con 0x7f182802b520 auth_method 0 2024-03-27T21:39:58.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.133+0000 7f181dffb640 10 client.4394.objecter ms_handle_connect 0x7f182802b520 2024-03-27T21:39:58.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.133+0000 7f182dced640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f182802b520 0x7f18281f0fd0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:58.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.134+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f18281eb780 con 0x7f182802b520 2024-03-27T21:39:58.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.134+0000 7f181dffb640 10 client.4394.objecter ms_dispatch 0x7f1828000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:39:58.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.134+0000 7f181dffb640 10 client.4394.objecter _finish_command 8 = system:0 2024-03-27T21:39:58.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.134+0000 7f181dffb640 20 client.4394.objecter put_session s=0x7f182802b3b0 osd=4 4 2024-03-27T21:39:58.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.134+0000 7f181dffb640 15 client.4394.objecter _session_command_op_remove 4 8 2024-03-27T21:39:58.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.149+0000 7f182f750640 10 client.4394.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:39:58.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.149+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign -1 9 2024-03-27T21:39:58.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.149+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f182802b3b0 osd=4 3 2024-03-27T21:39:58.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.149+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f182802b3b0 osd=4 3 2024-03-27T21:39:58.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.149+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f182802b3b0 osd=4 3 2024-03-27T21:39:58.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.149+0000 7f182f750640 15 client.4394.objecter _session_command_op_remove -1 9 2024-03-27T21:39:58.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.149+0000 7f182f750640 20 client.4394.objecter get_session s=0x7f182802b3b0 osd=4 3 2024-03-27T21:39:58.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.149+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign 4 9 2024-03-27T21:39:58.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.149+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f182802b3b0 osd=4 4 2024-03-27T21:39:58.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.149+0000 7f182f750640 10 client.4394.objecter _send_command 9 2024-03-27T21:39:58.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.149+0000 7f182f750640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f18281eb780 con 0x7f182802b520 2024-03-27T21:39:58.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.151+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f18281eb780 con 0x7f182802b520 2024-03-27T21:39:58.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.151+0000 7f181dffb640 10 client.4394.objecter ms_dispatch 0x7f1828000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:39:58.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.151+0000 7f181dffb640 10 client.4394.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:39:58.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.151+0000 7f181dffb640 20 client.4394.objecter put_session s=0x7f182802b3b0 osd=4 4 2024-03-27T21:39:58.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.151+0000 7f181dffb640 15 client.4394.objecter _session_command_op_remove 4 9 2024-03-27T21:39:58.138 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:39:58.140 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:39:58.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.151+0000 7f182f750640 10 client.4394.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:58.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.151+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign -1 10 2024-03-27T21:39:58.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.151+0000 7f182f750640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f18281f1a20 0x7f18281f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:58.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.151+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f18281eb780 osd=5 3 2024-03-27T21:39:58.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.151+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f18281eb780 osd=5 3 2024-03-27T21:39:58.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.151+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f18281eb780 osd=5 3 2024-03-27T21:39:58.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.151+0000 7f182f750640 15 client.4394.objecter _session_command_op_remove -1 10 2024-03-27T21:39:58.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.151+0000 7f182f750640 20 client.4394.objecter get_session s=0x7f18281eb780 osd=5 3 2024-03-27T21:39:58.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.151+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign 5 10 2024-03-27T21:39:58.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.151+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f18281eb780 osd=5 4 2024-03-27T21:39:58.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.151+0000 7f182f750640 10 client.4394.objecter _send_command 10 2024-03-27T21:39:58.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.151+0000 7f182f750640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18281f6be0 con 0x7f18281f1a20 2024-03-27T21:39:58.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.151+0000 7f182d4ec640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f18281f1a20 0x7f18281f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:58.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.151+0000 7f182d4ec640 10 monclient: get_auth_request con 0x7f18281f1a20 auth_method 0 2024-03-27T21:39:58.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.152+0000 7f181dffb640 10 client.4394.objecter ms_handle_connect 0x7f18281f1a20 2024-03-27T21:39:58.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.152+0000 7f182d4ec640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f18281f1a20 0x7f18281f3e10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:58.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.152+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f18281f6be0 con 0x7f18281f1a20 2024-03-27T21:39:58.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.152+0000 7f181dffb640 10 client.4394.objecter ms_dispatch 0x7f1828000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:39:58.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.152+0000 7f181dffb640 10 client.4394.objecter _finish_command 10 = system:0 2024-03-27T21:39:58.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.152+0000 7f181dffb640 20 client.4394.objecter put_session s=0x7f18281eb780 osd=5 4 2024-03-27T21:39:58.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.152+0000 7f181dffb640 15 client.4394.objecter _session_command_op_remove 5 10 2024-03-27T21:39:58.145 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:39:58.145 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:39:58.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:39:58.145 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:39:58.145 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:39:58.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.166+0000 7f182f750640 10 client.4394.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:39:58.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.166+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign -1 11 2024-03-27T21:39:58.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.166+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f18281eb780 osd=5 3 2024-03-27T21:39:58.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.166+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f18281eb780 osd=5 3 2024-03-27T21:39:58.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.166+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f18281eb780 osd=5 3 2024-03-27T21:39:58.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.166+0000 7f182f750640 15 client.4394.objecter _session_command_op_remove -1 11 2024-03-27T21:39:58.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.166+0000 7f182f750640 20 client.4394.objecter get_session s=0x7f18281eb780 osd=5 3 2024-03-27T21:39:58.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.166+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign 5 11 2024-03-27T21:39:58.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.166+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f18281eb780 osd=5 4 2024-03-27T21:39:58.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.166+0000 7f182f750640 10 client.4394.objecter _send_command 11 2024-03-27T21:39:58.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.166+0000 7f182f750640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f18281f4350 con 0x7f18281f1a20 2024-03-27T21:39:58.155 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:39:58.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.168+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f18281f4350 con 0x7f18281f1a20 2024-03-27T21:39:58.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.168+0000 7f181dffb640 10 client.4394.objecter ms_dispatch 0x7f1828000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:39:58.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.168+0000 7f181dffb640 10 client.4394.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:39:58.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.168+0000 7f181dffb640 20 client.4394.objecter put_session s=0x7f18281eb780 osd=5 4 2024-03-27T21:39:58.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.168+0000 7f181dffb640 15 client.4394.objecter _session_command_op_remove 5 11 2024-03-27T21:39:58.156 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:39:58.157 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:39:58.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.168+0000 7f182f750640 10 client.4394.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:58.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.168+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign -1 12 2024-03-27T21:39:58.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.168+0000 7f182f750640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f18281f4a00 0x7f18281f6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:58.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.168+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f18281f4350 osd=6 3 2024-03-27T21:39:58.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.168+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f18281f4350 osd=6 3 2024-03-27T21:39:58.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.168+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f18281f4350 osd=6 3 2024-03-27T21:39:58.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.168+0000 7f182f750640 15 client.4394.objecter _session_command_op_remove -1 12 2024-03-27T21:39:58.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.168+0000 7f182f750640 20 client.4394.objecter get_session s=0x7f18281f4350 osd=6 3 2024-03-27T21:39:58.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.168+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign 6 12 2024-03-27T21:39:58.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.168+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f18281f4350 osd=6 4 2024-03-27T21:39:58.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.168+0000 7f182f750640 10 client.4394.objecter _send_command 12 2024-03-27T21:39:58.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.168+0000 7f182f750640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18281f9bc0 con 0x7f18281f4a00 2024-03-27T21:39:58.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.168+0000 7f182cceb640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f18281f4a00 0x7f18281f6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:58.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.168+0000 7f182cceb640 10 monclient: get_auth_request con 0x7f18281f4a00 auth_method 0 2024-03-27T21:39:58.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.169+0000 7f181dffb640 10 client.4394.objecter ms_handle_connect 0x7f18281f4a00 2024-03-27T21:39:58.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.169+0000 7f182cceb640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f18281f4a00 0x7f18281f6df0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:58.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.170+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f18281f9bc0 con 0x7f18281f4a00 2024-03-27T21:39:58.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.170+0000 7f181dffb640 10 client.4394.objecter ms_dispatch 0x7f1828000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:39:58.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.170+0000 7f181dffb640 10 client.4394.objecter _finish_command 12 = system:0 2024-03-27T21:39:58.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.170+0000 7f181dffb640 20 client.4394.objecter put_session s=0x7f18281f4350 osd=6 4 2024-03-27T21:39:58.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.170+0000 7f181dffb640 15 client.4394.objecter _session_command_op_remove 6 12 2024-03-27T21:39:58.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.186+0000 7f182f750640 10 client.4394.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:39:58.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.186+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign -1 13 2024-03-27T21:39:58.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.186+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f18281f4350 osd=6 3 2024-03-27T21:39:58.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.186+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f18281f4350 osd=6 3 2024-03-27T21:39:58.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.186+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f18281f4350 osd=6 3 2024-03-27T21:39:58.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.186+0000 7f182f750640 15 client.4394.objecter _session_command_op_remove -1 13 2024-03-27T21:39:58.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.186+0000 7f182f750640 20 client.4394.objecter get_session s=0x7f18281f4350 osd=6 3 2024-03-27T21:39:58.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.186+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign 6 13 2024-03-27T21:39:58.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.186+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f18281f4350 osd=6 4 2024-03-27T21:39:58.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.186+0000 7f182f750640 10 client.4394.objecter _send_command 13 2024-03-27T21:39:58.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.186+0000 7f182f750640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f18281f7330 con 0x7f18281f4a00 2024-03-27T21:39:58.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.188+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f18281f7330 con 0x7f18281f4a00 2024-03-27T21:39:58.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.188+0000 7f181dffb640 10 client.4394.objecter ms_dispatch 0x7f1828000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:39:58.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.188+0000 7f181dffb640 10 client.4394.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:39:58.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.188+0000 7f181dffb640 20 client.4394.objecter put_session s=0x7f18281f4350 osd=6 4 2024-03-27T21:39:58.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.188+0000 7f181dffb640 15 client.4394.objecter _session_command_op_remove 6 13 2024-03-27T21:39:58.175 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:39:58.178 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:39:58.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.188+0000 7f182f750640 10 client.4394.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:58.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.188+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign -1 14 2024-03-27T21:39:58.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.188+0000 7f182f750640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f18281f79b0 0x7f18281f9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:58.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.188+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f18281f7330 osd=7 3 2024-03-27T21:39:58.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.188+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f18281f7330 osd=7 3 2024-03-27T21:39:58.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.188+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f18281f7330 osd=7 3 2024-03-27T21:39:58.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.188+0000 7f182f750640 15 client.4394.objecter _session_command_op_remove -1 14 2024-03-27T21:39:58.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.188+0000 7f182f750640 20 client.4394.objecter get_session s=0x7f18281f7330 osd=7 3 2024-03-27T21:39:58.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.188+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign 7 14 2024-03-27T21:39:58.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.188+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f18281f7330 osd=7 4 2024-03-27T21:39:58.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.188+0000 7f182f750640 10 client.4394.objecter _send_command 14 2024-03-27T21:39:58.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.188+0000 7f182f750640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18281fcb70 con 0x7f18281f79b0 2024-03-27T21:39:58.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.188+0000 7f182dced640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f18281f79b0 0x7f18281f9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:58.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.188+0000 7f182dced640 10 monclient: get_auth_request con 0x7f18281f79b0 auth_method 0 2024-03-27T21:39:58.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.189+0000 7f181dffb640 10 client.4394.objecter ms_handle_connect 0x7f18281f79b0 2024-03-27T21:39:58.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.189+0000 7f182dced640 1 --2- 172.21.15.60:0/702603751 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f18281f79b0 0x7f18281f9da0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:58.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.189+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f18281fcb70 con 0x7f18281f79b0 2024-03-27T21:39:58.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.189+0000 7f181dffb640 10 client.4394.objecter ms_dispatch 0x7f1828000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:39:58.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.189+0000 7f181dffb640 10 client.4394.objecter _finish_command 14 = system:0 2024-03-27T21:39:58.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.189+0000 7f181dffb640 20 client.4394.objecter put_session s=0x7f18281f7330 osd=7 4 2024-03-27T21:39:58.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.189+0000 7f181dffb640 15 client.4394.objecter _session_command_op_remove 7 14 2024-03-27T21:39:58.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.206+0000 7f182f750640 10 client.4394.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:39:58.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.206+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign -1 15 2024-03-27T21:39:58.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.206+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f18281f7330 osd=7 3 2024-03-27T21:39:58.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.206+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f18281f7330 osd=7 3 2024-03-27T21:39:58.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.206+0000 7f182f750640 20 client.4394.objecter _get_session s=0x7f18281f7330 osd=7 3 2024-03-27T21:39:58.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.206+0000 7f182f750640 15 client.4394.objecter _session_command_op_remove -1 15 2024-03-27T21:39:58.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.206+0000 7f182f750640 20 client.4394.objecter get_session s=0x7f18281f7330 osd=7 3 2024-03-27T21:39:58.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.206+0000 7f182f750640 15 client.4394.objecter _session_command_op_assign 7 15 2024-03-27T21:39:58.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.206+0000 7f182f750640 20 client.4394.objecter put_session s=0x7f18281f7330 osd=7 4 2024-03-27T21:39:58.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.206+0000 7f182f750640 10 client.4394.objecter _send_command 15 2024-03-27T21:39:58.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.206+0000 7f182f750640 1 -- 172.21.15.60:0/702603751 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f18281fa2e0 con 0x7f18281f79b0 2024-03-27T21:39:58.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.208+0000 7f181dffb640 1 -- 172.21.15.60:0/702603751 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f18281fa2e0 con 0x7f18281f79b0 2024-03-27T21:39:58.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.208+0000 7f181dffb640 10 client.4394.objecter ms_dispatch 0x7f1828000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:39:58.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.208+0000 7f181dffb640 10 client.4394.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:39:58.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.208+0000 7f181dffb640 20 client.4394.objecter put_session s=0x7f18281f7330 osd=7 4 2024-03-27T21:39:58.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.208+0000 7f181dffb640 15 client.4394.objecter _session_command_op_remove 7 15 2024-03-27T21:39:58.195 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:39:58.195 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:39:58.198 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:39:58.199 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:39:58.202 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:39:58.216+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:39:58.265 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:58.265 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:58.274 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:39:58.315 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:39:58.328+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:39:58.389 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:39:58.403+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:39:58.419 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:58.419 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:58.428 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:39:58.490 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:39:58.503+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:39:58.533 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:58.533 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:58.542 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:39:58.590 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:39:58.604+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:39:58.626 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-27T21:39:58.647 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:58.647 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:58.656 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:39:58.691 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:39:58.705+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:39:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.740+0000 7ff927095640 10 monclient: get_monmap_and_config 2024-03-27T21:39:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.740+0000 7ff927095640 10 monclient: build_initial_monmap 2024-03-27T21:39:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.740+0000 7ff927095640 10 monclient: monmap: 2024-03-27T21:39:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:39:58.741688+0000 2024-03-27T21:39:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:39:58.741688+0000 2024-03-27T21:39:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:58.728 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:39:58.728 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:39:58.728 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:39:58.728 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:39:58.728 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:39:58.728 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.741+0000 7ff927095640 1 Processor -- start 2024-03-27T21:39:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.741+0000 7ff927095640 1 -- start start 2024-03-27T21:39:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.741+0000 7ff927095640 10 monclient: init 2024-03-27T21:39:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff927095640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff927095640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-03-27T21:39:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff927095640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff9200b40b0 0x7ff9200a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff927095640 10 monclient(hunting): picked mon.noname-a con 0x7ff9200b40b0 addr v2:172.21.15.60:3300/0 2024-03-27T21:39:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff927095640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff92013b020 0x7ff92013b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff927095640 10 monclient(hunting): picked mon.noname-c con 0x7ff92013b020 addr v2:172.21.15.191:3300/0 2024-03-27T21:39:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff927095640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9201416f0 addr v1:172.21.15.60:6789/0 2024-03-27T21:39:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff927095640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9200b3830 con 0x7ff9201416f0 2024-03-27T21:39:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff927095640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff927095640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9200b4490 con 0x7ff9200b40b0 2024-03-27T21:39:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff927095640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff927095640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9200aa520 con 0x7ff92013b020 2024-03-27T21:39:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff927095640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff927095640 10 monclient(hunting): authenticate will time out at 839.963040s 2024-03-27T21:39:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff924e31640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff9200b40b0 0x7ff9200a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff924e31640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff9200b40b0 0x7ff9200a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44452/0 (socket says 172.21.15.60:44452) 2024-03-27T21:39:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff924e31640 1 -- 172.21.15.60:0/2780220493 learned_addr learned my addr 172.21.15.60:0/2780220493 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff924e31640 10 monclient(hunting): get_auth_request con 0x7ff9200b40b0 auth_method 0 2024-03-27T21:39:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff924e31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff924e31640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:58.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff924e31640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:58.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff925632640 1 --1- 172.21.15.60:0/2780220493 >> v1:172.21.15.60:6789/0 conn(0x7ff9201416f0 0x7ff920141b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55856/0 (socket says 172.21.15.60:55856) 2024-03-27T21:39:58.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff924e31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:58.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff924e31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:58.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff924e31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:58.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff91ffff640 1 --2- 172.21.15.60:0/2780220493 >> v2:172.21.15.191:3300/0 conn(0x7ff92013b020 0x7ff92013b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:58.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff91ffff640 10 monclient(hunting): get_auth_request con 0x7ff92013b020 auth_method 0 2024-03-27T21:39:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff91ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff91ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff91ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff91ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff91ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff91ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff924e31640 10 monclient(hunting): handle_auth_done global_id 4485 payload 274 2024-03-27T21:39:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff924e31640 1 -- 172.21.15.60:0/2780220493 >> v2:172.21.15.191:3300/0 conn(0x7ff92013b020 msgr2=0x7ff92013b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:39:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff924e31640 1 --2- 172.21.15.60:0/2780220493 >> v2:172.21.15.191:3300/0 conn(0x7ff92013b020 0x7ff92013b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff924e31640 1 -- 172.21.15.60:0/2780220493 >> v1:172.21.15.60:6789/0 conn(0x7ff9201416f0 legacy=0x7ff920141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff924e31640 10 monclient: _finish_hunting 0 2024-03-27T21:39:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff924e31640 1 monclient: found mon.noname-a 2024-03-27T21:39:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff924e31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff924e31640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:39:58.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.742+0000 7ff924e31640 1 -- 172.21.15.60:0/2780220493 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9200a9470 con 0x7ff9200b40b0 2024-03-27T21:39:58.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.743+0000 7ff924e31640 1 --2- 172.21.15.60:0/2780220493 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9200b40b0 0x7ff9200a8b90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff9200b47e0 tx=0x7ff91402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe0e304d32d575a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:58.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.743+0000 7ff91f7fe640 1 -- 172.21.15.60:0/2780220493 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff914030860 con 0x7ff9200b40b0 2024-03-27T21:39:58.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.743+0000 7ff91f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:58.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.743+0000 7ff91f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:39:58.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.743+0000 7ff91f7fe640 10 monclient: dump: 2024-03-27T21:39:58.740 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:58.740 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:58.740 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:58.740 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:58.740 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:58.740 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:58.740 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:58.740 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:58.740 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:58.740 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:58.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.743+0000 7ff91f7fe640 10 monclient: _finish_auth 0 2024-03-27T21:39:58.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.743+0000 7ff91f7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:39:58.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.743+0000 7ff91f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:58.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.743+0000 7ff927095640 5 monclient: authenticate success, global_id 4485 2024-03-27T21:39:58.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.743+0000 7ff927095640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:39:58.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.743+0000 7ff91f7fe640 1 -- 172.21.15.60:0/2780220493 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9140309c0 con 0x7ff9200b40b0 2024-03-27T21:39:58.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.743+0000 7ff91f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:58.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.743+0000 7ff91f7fe640 1 -- 172.21.15.60:0/2780220493 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff914030d50 con 0x7ff9200b40b0 2024-03-27T21:39:58.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.743+0000 7ff927095640 10 monclient: get_monmap_and_config success 2024-03-27T21:39:58.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.743+0000 7ff927095640 4 set_mon_vals no callback set 2024-03-27T21:39:58.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.743+0000 7ff91f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:58.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.743+0000 7ff91f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:39:58.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.743+0000 7ff91f7fe640 10 monclient: dump: 2024-03-27T21:39:58.741 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:58.741 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:58.742 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:58.742 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:58.742 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:58.742 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:58.742 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:58.742 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:58.742 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:58.742 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:58.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.744+0000 7ff927095640 10 monclient: shutdown 2024-03-27T21:39:58.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.744+0000 7ff927095640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:58.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.744+0000 7ff927095640 1 -- 172.21.15.60:0/2780220493 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9200b40b0 msgr2=0x7ff9200a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:58.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.744+0000 7ff927095640 1 --2- 172.21.15.60:0/2780220493 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9200b40b0 0x7ff9200a8b90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff9200b47e0 tx=0x7ff91402b9a0 comp rx=0 tx=0).stop 2024-03-27T21:39:58.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.744+0000 7ff927095640 1 -- 172.21.15.60:0/2780220493 shutdown_connections 2024-03-27T21:39:58.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.744+0000 7ff927095640 1 --2- 172.21.15.60:0/2780220493 >> v2:172.21.15.191:3300/0 conn(0x7ff92013b020 0x7ff92013b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:58.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.744+0000 7ff927095640 1 --2- 172.21.15.60:0/2780220493 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9200b40b0 0x7ff9200a8b90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:58.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.744+0000 7ff927095640 1 -- 172.21.15.60:0/2780220493 >> 172.21.15.60:0/2780220493 conn(0x7ff9200608e0 msgr2=0x7ff920060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:58.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.744+0000 7ff927095640 1 -- 172.21.15.60:0/2780220493 shutdown_connections 2024-03-27T21:39:58.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.744+0000 7ff927095640 1 -- 172.21.15.60:0/2780220493 wait complete. 2024-03-27T21:39:58.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.744+0000 7ff927095640 10 monclient: build_initial_monmap 2024-03-27T21:39:58.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.744+0000 7ff927095640 10 monclient: monmap: 2024-03-27T21:39:58.743 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:39:58.743 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:39:58.743 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:39:58.743 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:39:58.743 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:39:58.743 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:39:58.743 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:39:58.743 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:39:58.743 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:39:58.744 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:58.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 1 librados: starting msgr at 2024-03-27T21:39:58.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 1 librados: starting objecter 2024-03-27T21:39:58.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 1 Processor -- start 2024-03-27T21:39:58.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 1 -- start start 2024-03-27T21:39:58.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 1 librados: setting wanted keys 2024-03-27T21:39:58.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 1 librados: calling monclient init 2024-03-27T21:39:58.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 10 monclient: init 2024-03-27T21:39:58.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 10 monclient: _reopen_session rank -1 2024-03-27T21:39:58.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:39:58.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9200b40b0 0x7ff9200aac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:58.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 10 monclient(hunting): picked mon.noname-b con 0x7ff9200b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:39:58.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff92013b020 0x7ff9201d67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:58.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 10 monclient(hunting): picked mon.noname-a con 0x7ff92013b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:39:58.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff9201416f0 0x7ff9201d6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 10 monclient(hunting): picked mon.noname-c con 0x7ff9201416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:39:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9200ac4b0 con 0x7ff92013b020 2024-03-27T21:39:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9201d62b0 con 0x7ff9200b40b0 2024-03-27T21:39:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 10 monclient(hunting): start opening mon connection 2024-03-27T21:39:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92013ba60 con 0x7ff9201416f0 2024-03-27T21:39:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 10 monclient(hunting): _renew_subs 2024-03-27T21:39:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.745+0000 7ff927095640 10 monclient(hunting): authenticate will time out at 839.966887s 2024-03-27T21:39:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff925632640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff9201416f0 0x7ff9201d6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff92013b020 0x7ff9201d67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff92013b020 0x7ff9201d67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44458/0 (socket says 172.21.15.60:44458) 2024-03-27T21:39:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff925632640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff9201416f0 0x7ff9201d6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47850/0 (socket says 172.21.15.60:47850) 2024-03-27T21:39:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91ffff640 1 -- 172.21.15.60:0/1879667701 learned_addr learned my addr 172.21.15.60:0/1879667701 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:39:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91ffff640 10 monclient(hunting): get_auth_request con 0x7ff92013b020 auth_method 0 2024-03-27T21:39:58.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:58.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:58.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:58.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff925632640 10 monclient(hunting): get_auth_request con 0x7ff9201416f0 auth_method 0 2024-03-27T21:39:58.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff925632640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:58.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff925632640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:58.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff925632640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:58.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff924e31640 1 --2- 172.21.15.60:0/1879667701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9200b40b0 0x7ff9200aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:58.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff924e31640 10 monclient(hunting): get_auth_request con 0x7ff9200b40b0 auth_method 0 2024-03-27T21:39:58.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff924e31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:39:58.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff924e31640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:39:58.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff924e31640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:39:58.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff925632640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:58.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff925632640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:58.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff925632640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:58.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:58.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:58.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:58.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff924e31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:39:58.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff924e31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:39:58.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff924e31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:39:58.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff925632640 10 monclient(hunting): handle_auth_done global_id 4406 payload 824 2024-03-27T21:39:58.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff925632640 1 -- 172.21.15.60:0/1879667701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9200b40b0 msgr2=0x7ff9200aac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:58.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff925632640 1 --2- 172.21.15.60:0/1879667701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9200b40b0 0x7ff9200aac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:58.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff925632640 1 -- 172.21.15.60:0/1879667701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff92013b020 msgr2=0x7ff9201d67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:58.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff925632640 1 --2- 172.21.15.60:0/1879667701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff92013b020 0x7ff9201d67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:58.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff925632640 10 monclient: _finish_hunting 0 2024-03-27T21:39:58.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff925632640 1 monclient: found mon.noname-c 2024-03-27T21:39:58.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff925632640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:39:58.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff925632640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:39:58.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff925632640 1 -- 172.21.15.60:0/1879667701 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9140094d0 con 0x7ff9201416f0 2024-03-27T21:39:58.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff924e31640 1 --2- 172.21.15.60:0/1879667701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9200b40b0 0x7ff9200aac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:58.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91ffff640 1 --2- 172.21.15.60:0/1879667701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff92013b020 0x7ff9201d67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:39:58.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff925632640 1 --2- 172.21.15.60:0/1879667701 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff9201416f0 0x7ff9201d6d10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff91800c730 tx=0x7ff91800cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:58.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91d7fa640 10 client.?.objecter ms_handle_connect 0x7ff9201416f0 2024-03-27T21:39:58.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91d7fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:39:58.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91d7fa640 1 -- 172.21.15.60:0/1879667701 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff918004d60 con 0x7ff9201416f0 2024-03-27T21:39:58.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:58.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:39:58.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91d7fa640 10 monclient: dump: 2024-03-27T21:39:58.748 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:58.749 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:58.749 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:58.749 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:58.749 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:58.749 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:58.749 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:58.749 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:58.749 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:58.749 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:58.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91d7fa640 10 monclient: _finish_auth 0 2024-03-27T21:39:58.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91d7fa640 10 monclient: _check_auth_tickets 2024-03-27T21:39:58.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:39:58.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91d7fa640 1 -- 172.21.15.60:0/1879667701 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91800a640 con 0x7ff9201416f0 2024-03-27T21:39:58.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff927095640 5 monclient: authenticate success, global_id 4406 2024-03-27T21:39:58.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:39:58.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91d7fa640 1 -- 172.21.15.60:0/1879667701 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91800aa10 con 0x7ff9201416f0 2024-03-27T21:39:58.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff927095640 10 monclient: _renew_subs 2024-03-27T21:39:58.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff927095640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:58.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff927095640 1 -- 172.21.15.60:0/1879667701 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff92013df10 con 0x7ff9201416f0 2024-03-27T21:39:58.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91effd640 4 set_mon_vals no callback set 2024-03-27T21:39:58.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:39:58.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:39:58.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff91d7fa640 10 monclient: dump: 2024-03-27T21:39:58.750 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:39:58.750 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:39:58.750 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:58.750 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:39:58.750 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:39:58.750 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:39:58.751 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:39:58.751 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:39:58.751 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:39:58.751 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:39:58.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff927095640 10 client.4406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:39:58.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff927095640 10 monclient: _renew_subs 2024-03-27T21:39:58.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff927095640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:58.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff927095640 1 -- 172.21.15.60:0/1879667701 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92013d5d0 con 0x7ff9201416f0 2024-03-27T21:39:58.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.746+0000 7ff927095640 1 librados: init done 2024-03-27T21:39:58.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.747+0000 7ff927095640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:58.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.747+0000 7ff927095640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:58.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.747+0000 7ff927095640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:58.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.747+0000 7ff927095640 1 -- 172.21.15.60:0/1879667701 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff920069470 con 0x7ff9201416f0 2024-03-27T21:39:58.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.748+0000 7ff91d7fa640 1 -- 172.21.15.60:0/1879667701 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff918007070 con 0x7ff9201416f0 2024-03-27T21:39:58.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.749+0000 7ff91d7fa640 1 --2- 172.21.15.60:0/1879667701 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff90406c590 0x7ff90406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:39:58.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.749+0000 7ff924e31640 1 --2- 172.21.15.60:0/1879667701 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff90406c590 0x7ff90406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:39:58.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.749+0000 7ff91d7fa640 1 -- 172.21.15.60:0/1879667701 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff9180885f0 con 0x7ff9201416f0 2024-03-27T21:39:58.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.749+0000 7ff91d7fa640 10 client.4406.objecter ms_dispatch 0x7ff920000b90 osd_map(24..24 src has 1..24) v4 2024-03-27T21:39:58.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.749+0000 7ff924e31640 10 monclient: get_auth_request con 0x7ff90406c590 auth_method 0 2024-03-27T21:39:58.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.749+0000 7ff91d7fa640 3 client.4406.objecter handle_osd_map got epochs [24,24] > 0 2024-03-27T21:39:58.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.749+0000 7ff91d7fa640 3 client.4406.objecter handle_osd_map decoding full epoch 24 2024-03-27T21:39:58.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.749+0000 7ff91d7fa640 20 client.4406.objecter dump_active .. 0 homeless 2024-03-27T21:39:58.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.749+0000 7ff91d7fa640 10 client.4406.objecter ms_handle_connect 0x7ff90406c590 2024-03-27T21:39:58.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.749+0000 7ff924e31640 1 --2- 172.21.15.60:0/1879667701 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff90406c590 0x7ff90406ea50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff914036000 tx=0x7ff914008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:39:58.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.753+0000 7ff91d7fa640 1 -- 172.21.15.60:0/1879667701 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff918053b00 con 0x7ff9201416f0 2024-03-27T21:39:58.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.753+0000 7ff91d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:39:58.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.753+0000 7ff91d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:39:58.792 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:39:58.805+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:39:58.800 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:58.800 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:58.808 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:39:58.893 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:39:58.906+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:39:58.912 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:58.912 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:58.921 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:39:58.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.954+0000 7ff927095640 10 start_mon_command cmd=[{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:39:58.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.954+0000 7ff927095640 10 monclient: _send_command 2 [{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:39:58.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.954+0000 7ff927095640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:39:58.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:58.954+0000 7ff927095640 1 -- 172.21.15.60:0/1879667701 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7ff92013bba0 con 0x7ff9201416f0 2024-03-27T21:39:58.994 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:39:59.008+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:39:59.030 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:59.030 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:59.038 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:39:59.122 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:39:59.136+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:39:59.164 INFO:teuthology.orchestra.run.smithi060.stdout:get_pg_num_history pg_num_history pg_num_history(e21 pg_nums {1={16=1},2={19=8}} deleted_pools ) 2024-03-27T21:39:59.164 INFO:teuthology.orchestra.run.smithi060.stdout:Importing pgid 2.1 2024-03-27T21:39:59.164 INFO:teuthology.orchestra.run.smithi060.stdout:write_pg epoch 20 info 2.1( empty local-lis/les=19/20 n=0 ec=19/19 lis/c=19/19 les/c/f=20/20/0 sis=19) 2024-03-27T21:39:59.169 INFO:teuthology.orchestra.run.smithi060.stdout:Import successful 2024-03-27T21:39:59.195 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:59.195 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:59.197 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:39:59.209+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:39:59.204 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:39:59.296 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:39:59.310+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:39:59.354 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:59.354 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:59.363 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:39:59.408 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:39:59.422+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:39:59.476 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:59.476 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:59.485 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:39:59.526 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:39:59.540+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:39:59.598 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:39:59.612+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:39:59.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.635+0000 7ff91d7fa640 1 -- 172.21.15.60:0/1879667701 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v25) v1 ==== 91+0+0 (secure 0 0 0) 0x7ff918018090 con 0x7ff9201416f0 2024-03-27T21:39:59.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.635+0000 7ff91d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:39:59.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.635+0000 7ff91d7fa640 10 monclient: _finish_command 2 = system:0 noscrub is unset 2024-03-27T21:39:59.622 INFO:teuthology.orchestra.run.smithi060.stderr:noscrub is unset 2024-03-27T21:39:59.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.637+0000 7ff927095640 10 librados: watch_flush enter 2024-03-27T21:39:59.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.637+0000 7ff927095640 10 librados: watch_flush exit 2024-03-27T21:39:59.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.637+0000 7ff927095640 20 client.4406.objecter shutdown clearing up homeless session... 2024-03-27T21:39:59.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.637+0000 7ff927095640 10 client.4406.objecter successfully canceled tick 2024-03-27T21:39:59.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.637+0000 7ff927095640 1 -- 172.21.15.60:0/1879667701 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff90406c590 msgr2=0x7ff90406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:59.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.637+0000 7ff927095640 1 --2- 172.21.15.60:0/1879667701 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff90406c590 0x7ff90406ea50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff914036000 tx=0x7ff914008030 comp rx=0 tx=0).stop 2024-03-27T21:39:59.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.637+0000 7ff927095640 10 monclient: shutdown 2024-03-27T21:39:59.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.637+0000 7ff927095640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:39:59.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.637+0000 7ff927095640 1 -- 172.21.15.60:0/1879667701 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff9201416f0 msgr2=0x7ff9201d6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:39:59.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.637+0000 7ff927095640 1 --2- 172.21.15.60:0/1879667701 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff9201416f0 0x7ff9201d6d10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff91800c730 tx=0x7ff91800cbf0 comp rx=0 tx=0).stop 2024-03-27T21:39:59.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.637+0000 7ff927095640 1 -- 172.21.15.60:0/1879667701 shutdown_connections 2024-03-27T21:39:59.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.637+0000 7ff927095640 1 --2- 172.21.15.60:0/1879667701 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff9201416f0 0x7ff9201d6d10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:59.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.637+0000 7ff927095640 1 --2- 172.21.15.60:0/1879667701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff92013b020 0x7ff9201d67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:59.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.637+0000 7ff927095640 1 --2- 172.21.15.60:0/1879667701 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff90406c590 0x7ff90406ea50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:59.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.637+0000 7ff927095640 1 --2- 172.21.15.60:0/1879667701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9200b40b0 0x7ff9200aac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:39:59.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.637+0000 7ff927095640 1 -- 172.21.15.60:0/1879667701 >> 172.21.15.60:0/1879667701 conn(0x7ff9200608e0 msgr2=0x7ff92013f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:39:59.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.637+0000 7ff927095640 1 -- 172.21.15.60:0/1879667701 shutdown_connections 2024-03-27T21:39:59.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.637+0000 7ff927095640 1 -- 172.21.15.60:0/1879667701 wait complete. 2024-03-27T21:39:59.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:39:59.637+0000 7ff927095640 1 librados: shutdown 2024-03-27T21:39:59.635 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:59.636 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:59.646 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:39:59.682 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f /var/log/ceph/exp.2.1.2 2024-03-27T21:39:59.699 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:39:59.712+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:39:59.799 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:39:59.813+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:39:59.830 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:59.830 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:59.840 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:39:59.899 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:39:59.913+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:39:59.955 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:39:59.955 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:39:59.965 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:40:00.006 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:00.020+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:00.101 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:00.114+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:00.124 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:00.124 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:00.133 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:40:00.200 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:40:00.217 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:00.231+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:00.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-03-27T21:40:00.286 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:00.286 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:00.296 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:40:00.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.337+0000 7f6206134640 10 monclient: get_monmap_and_config 2024-03-27T21:40:00.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.337+0000 7f6206134640 10 monclient: build_initial_monmap 2024-03-27T21:40:00.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.337+0000 7f6206134640 10 monclient: monmap: 2024-03-27T21:40:00.324 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:00.324 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:00.324 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:00.338903+0000 2024-03-27T21:40:00.324 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:00.338903+0000 2024-03-27T21:40:00.324 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:00.324 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:00.325 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:00.325 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:00.325 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:00.325 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:00.325 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:00.325 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:00.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.338+0000 7f6206134640 1 Processor -- start 2024-03-27T21:40:00.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.338+0000 7f6206134640 1 -- start start 2024-03-27T21:40:00.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.338+0000 7f6206134640 10 monclient: init 2024-03-27T21:40:00.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f6206134640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:00.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f6206134640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-03-27T21:40:00.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f6206134640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6200071a50 0x7f62000a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:00.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f6206134640 10 monclient(hunting): picked mon.noname-b con 0x7f6200071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:00.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f6206134640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f620013e810 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:00.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f6206134640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f620013ec50 0x7f6200141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:00.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f6206134640 10 monclient(hunting): picked mon.noname-c con 0x7f620013ec50 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:00.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f6206134640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62000a8e90 con 0x7f620013e810 2024-03-27T21:40:00.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f6206134640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:00.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f6206134640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6200071e30 con 0x7f620013ec50 2024-03-27T21:40:00.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f6206134640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:00.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f6206134640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62000aa040 con 0x7f6200071a50 2024-03-27T21:40:00.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f6206134640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:00.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f6206134640 10 monclient(hunting): authenticate will time out at 841.560415s 2024-03-27T21:40:00.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61feffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f620013e810 0x7f6200141630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55872/0 (socket says 172.21.15.60:55872) 2024-03-27T21:40:00.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61ff7fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6200071a50 0x7f62000a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:00.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61feffd640 1 -- 172.21.15.60:0/2652854 learned_addr learned my addr 172.21.15.60:0/2652854 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:40:00.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61ff7fe640 10 monclient(hunting): get_auth_request con 0x7f6200071a50 auth_method 0 2024-03-27T21:40:00.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:00.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:00.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:00.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:00.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:00.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61fffff640 1 --2- 172.21.15.60:0/2652854 >> v2:172.21.15.191:3300/0 conn(0x7f620013ec50 0x7f6200141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:00.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:00.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61fffff640 10 monclient(hunting): get_auth_request con 0x7f620013ec50 auth_method 0 2024-03-27T21:40:00.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:00.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61fffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:00.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:00.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61ff7fe640 10 monclient(hunting): handle_auth_done global_id 4409 payload 274 2024-03-27T21:40:00.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61ff7fe640 1 -- 172.21.15.60:0/2652854 >> v2:172.21.15.191:3300/0 conn(0x7f620013ec50 msgr2=0x7f6200141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:00.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61fe7fc640 1 -- 172.21.15.60:0/2652854 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801204366 0 0) 0x7f62000a8e90 con 0x7f620013e810 2024-03-27T21:40:00.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61ff7fe640 1 --2- 172.21.15.60:0/2652854 >> v2:172.21.15.191:3300/0 conn(0x7f620013ec50 0x7f6200141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:00.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61ff7fe640 1 -- 172.21.15.60:0/2652854 >> v1:172.21.15.60:6789/0 conn(0x7f620013e810 legacy=0x7f6200141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:00.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61ff7fe640 10 monclient: _finish_hunting 0 2024-03-27T21:40:00.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61ff7fe640 1 monclient: found mon.noname-b 2024-03-27T21:40:00.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:00.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61ff7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:00.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61ff7fe640 1 -- 172.21.15.60:0/2652854 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6200142280 con 0x7f6200071a50 2024-03-27T21:40:00.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61fffff640 1 --2- 172.21.15.60:0/2652854 >> v2:172.21.15.191:3300/0 conn(0x7f620013ec50 0x7f6200141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:40:00.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.339+0000 7f61fe7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:40:00.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.340+0000 7f61ff7fe640 1 --2- 172.21.15.60:0/2652854 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6200071a50 0x7f62000a87e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f61f00097a0 tx=0x7f61f002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c6a0607d7bd2b4b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:00.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.340+0000 7f61fe7fc640 1 -- 172.21.15.60:0/2652854 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0031830 con 0x7f6200071a50 2024-03-27T21:40:00.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.340+0000 7f61fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:00.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.340+0000 7f61fe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:00.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.340+0000 7f61fe7fc640 10 monclient: dump: 2024-03-27T21:40:00.336 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:00.336 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:00.336 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:00.336 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:00.336 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:00.336 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:00.336 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:00.336 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:00.336 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:00.336 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:00.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.340+0000 7f61fe7fc640 10 monclient: _finish_auth 0 2024-03-27T21:40:00.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.340+0000 7f61fe7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:40:00.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.340+0000 7f61fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:00.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.340+0000 7f6206134640 5 monclient: authenticate success, global_id 4409 2024-03-27T21:40:00.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.340+0000 7f6206134640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:00.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.340+0000 7f61fe7fc640 1 -- 172.21.15.60:0/2652854 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f0031990 con 0x7f6200071a50 2024-03-27T21:40:00.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.340+0000 7f61fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:00.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.340+0000 7f61fe7fc640 1 -- 172.21.15.60:0/2652854 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0031d20 con 0x7f6200071a50 2024-03-27T21:40:00.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.340+0000 7f6206134640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:00.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.340+0000 7f6206134640 4 set_mon_vals no callback set 2024-03-27T21:40:00.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.340+0000 7f61fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:00.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.340+0000 7f61fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:00.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.340+0000 7f61fe7fc640 10 monclient: dump: 2024-03-27T21:40:00.337 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:00.337 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:00.337 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:00.338 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:00.338 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:00.338 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:00.338 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:00.338 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:00.338 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:00.338 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:00.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.341+0000 7f6206134640 10 monclient: shutdown 2024-03-27T21:40:00.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.341+0000 7f6206134640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:00.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.341+0000 7f6206134640 1 -- 172.21.15.60:0/2652854 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6200071a50 msgr2=0x7f62000a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:00.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.341+0000 7f6206134640 1 --2- 172.21.15.60:0/2652854 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6200071a50 0x7f62000a87e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f61f00097a0 tx=0x7f61f002b8d0 comp rx=0 tx=0).stop 2024-03-27T21:40:00.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.341+0000 7f6206134640 1 -- 172.21.15.60:0/2652854 shutdown_connections 2024-03-27T21:40:00.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.341+0000 7f6206134640 1 --2- 172.21.15.60:0/2652854 >> v2:172.21.15.191:3300/0 conn(0x7f620013ec50 0x7f6200141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:00.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.341+0000 7f6206134640 1 --2- 172.21.15.60:0/2652854 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6200071a50 0x7f62000a87e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:00.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.341+0000 7f6206134640 1 -- 172.21.15.60:0/2652854 >> 172.21.15.60:0/2652854 conn(0x7f62000b7300 msgr2=0x7f62000b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:00.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.341+0000 7f6206134640 1 -- 172.21.15.60:0/2652854 shutdown_connections 2024-03-27T21:40:00.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.341+0000 7f6206134640 1 -- 172.21.15.60:0/2652854 wait complete. 2024-03-27T21:40:00.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.342+0000 7f6206134640 10 monclient: build_initial_monmap 2024-03-27T21:40:00.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.342+0000 7f6206134640 10 monclient: monmap: 2024-03-27T21:40:00.339 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:00.339 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:00.339 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:00.339 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:00.339 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:00.339 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:00.339 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:00.339 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:00.339 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:00.339 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:00.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.342+0000 7f6206134640 1 librados: starting msgr at 2024-03-27T21:40:00.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.342+0000 7f6206134640 1 librados: starting objecter 2024-03-27T21:40:00.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.342+0000 7f6206134640 1 Processor -- start 2024-03-27T21:40:00.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.342+0000 7f6206134640 1 -- start start 2024-03-27T21:40:00.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.342+0000 7f6206134640 1 librados: setting wanted keys 2024-03-27T21:40:00.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.342+0000 7f6206134640 1 librados: calling monclient init 2024-03-27T21:40:00.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.342+0000 7f6206134640 10 monclient: init 2024-03-27T21:40:00.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f6206134640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:00.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f6206134640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:40:00.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f6206134640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6200071a50 0x7f62000b6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:00.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f6206134640 10 monclient(hunting): picked mon.noname-a con 0x7f6200071a50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:00.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f6206134640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f620013e810 0x7f62001d9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:00.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f6206134640 10 monclient(hunting): picked mon.noname-b con 0x7f620013e810 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:00.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f6206134640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f620013ec50 0x7f62001d9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:00.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f6206134640 10 monclient(hunting): picked mon.noname-c con 0x7f620013ec50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:00.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f6206134640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:00.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f6206134640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62000b0c90 con 0x7f6200071a50 2024-03-27T21:40:00.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f6206134640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:00.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f6206134640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6200138750 con 0x7f620013e810 2024-03-27T21:40:00.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f6206134640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:00.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f6206134640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f620013a090 con 0x7f620013ec50 2024-03-27T21:40:00.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f6206134640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:00.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f6206134640 10 monclient(hunting): authenticate will time out at 841.564374s 2024-03-27T21:40:00.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61ff7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6200071a50 0x7f62000b6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:00.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61ff7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6200071a50 0x7f62000b6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44468/0 (socket says 172.21.15.60:44468) 2024-03-27T21:40:00.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61ff7fe640 1 -- 172.21.15.60:0/2499822549 learned_addr learned my addr 172.21.15.60:0/2499822549 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:00.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61ff7fe640 10 monclient(hunting): get_auth_request con 0x7f6200071a50 auth_method 0 2024-03-27T21:40:00.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:00.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:00.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:00.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61fffff640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f620013ec50 0x7f62001d9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:00.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61fffff640 10 monclient(hunting): get_auth_request con 0x7f620013ec50 auth_method 0 2024-03-27T21:40:00.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:00.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61fffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:00.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:00.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61feffd640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f620013e810 0x7f62001d9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:00.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:00.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:00.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:00.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61feffd640 10 monclient(hunting): get_auth_request con 0x7f620013e810 auth_method 0 2024-03-27T21:40:00.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:00.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61feffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:00.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:00.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:00.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:00.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:00.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:00.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:00.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:00.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61ff7fe640 10 monclient(hunting): handle_auth_done global_id 4497 payload 824 2024-03-27T21:40:00.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61ff7fe640 1 -- 172.21.15.60:0/2499822549 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f620013ec50 msgr2=0x7f62001d9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:00.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61ff7fe640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f620013ec50 0x7f62001d9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:00.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61ff7fe640 1 -- 172.21.15.60:0/2499822549 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f620013e810 msgr2=0x7f62001d9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:00.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61ff7fe640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f620013e810 0x7f62001d9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:00.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61ff7fe640 10 monclient: _finish_hunting 0 2024-03-27T21:40:00.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61ff7fe640 1 monclient: found mon.noname-a 2024-03-27T21:40:00.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:00.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61ff7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:00.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61ff7fe640 1 -- 172.21.15.60:0/2499822549 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f0009400 con 0x7f6200071a50 2024-03-27T21:40:00.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.343+0000 7f61fffff640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f620013ec50 0x7f62001d9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:00.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f61ff7fe640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6200071a50 0x7f62000b6780 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f61f0000c00 tx=0x7f61f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:00.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f61e7fff640 10 client.?.objecter ms_handle_connect 0x7f6200071a50 2024-03-27T21:40:00.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f61e7fff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:00.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f61e7fff640 1 -- 172.21.15.60:0/2499822549 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0004230 con 0x7f6200071a50 2024-03-27T21:40:00.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f61e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:00.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f61e7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:00.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f61e7fff640 10 monclient: dump: 2024-03-27T21:40:00.345 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:00.345 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:00.345 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:00.345 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:00.345 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:00.345 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:00.345 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:00.345 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:00.345 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:00.345 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:00.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f61e7fff640 10 monclient: _finish_auth 0 2024-03-27T21:40:00.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f61e7fff640 10 monclient: _check_auth_tickets 2024-03-27T21:40:00.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f61e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:00.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f61e7fff640 1 -- 172.21.15.60:0/2499822549 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f0004390 con 0x7f6200071a50 2024-03-27T21:40:00.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f6206134640 5 monclient: authenticate success, global_id 4497 2024-03-27T21:40:00.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f6206134640 10 monclient: _renew_subs 2024-03-27T21:40:00.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f6206134640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:00.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f6206134640 1 -- 172.21.15.60:0/2499822549 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62001da310 con 0x7f6200071a50 2024-03-27T21:40:00.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f61e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:00.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f61e7fff640 1 -- 172.21.15.60:0/2499822549 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f003a880 con 0x7f6200071a50 2024-03-27T21:40:00.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f61e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:00.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f61e7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:00.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f61e7fff640 10 monclient: dump: 2024-03-27T21:40:00.346 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:00.346 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:00.346 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:00.346 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:00.346 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:00.347 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:00.347 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:00.347 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:00.347 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:00.347 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:00.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f6206134640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:00.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f61fdffb640 4 set_mon_vals no callback set 2024-03-27T21:40:00.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f6206134640 10 monclient: _renew_subs 2024-03-27T21:40:00.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f6206134640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:00.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f6206134640 1 -- 172.21.15.60:0/2499822549 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f620013a8b0 con 0x7f6200071a50 2024-03-27T21:40:00.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.344+0000 7f6206134640 1 librados: init done 2024-03-27T21:40:00.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.345+0000 7f6206134640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:00.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.345+0000 7f6206134640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:00.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.345+0000 7f6206134640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:00.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.345+0000 7f6206134640 1 -- 172.21.15.60:0/2499822549 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f620013a250 con 0x7f6200071a50 2024-03-27T21:40:00.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.346+0000 7f61e7fff640 1 -- 172.21.15.60:0/2499822549 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f61f003a9e0 con 0x7f6200071a50 2024-03-27T21:40:00.348 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:00.354+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:00.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.346+0000 7f61e7fff640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f61ec06c590 0x7f61ec06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:00.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.346+0000 7f61e7fff640 1 -- 172.21.15.60:0/2499822549 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f61f00ab710 con 0x7f6200071a50 2024-03-27T21:40:00.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.346+0000 7f61feffd640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f61ec06c590 0x7f61ec06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:00.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.346+0000 7f61e7fff640 10 client.4497.objecter ms_dispatch 0x7f6200000b90 osd_map(25..25 src has 1..25) v4 2024-03-27T21:40:00.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.346+0000 7f61e7fff640 3 client.4497.objecter handle_osd_map got epochs [25,25] > 0 2024-03-27T21:40:00.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.346+0000 7f61e7fff640 3 client.4497.objecter handle_osd_map decoding full epoch 25 2024-03-27T21:40:00.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.346+0000 7f61feffd640 10 monclient: get_auth_request con 0x7f61ec06c590 auth_method 0 2024-03-27T21:40:00.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.346+0000 7f61e7fff640 20 client.4497.objecter dump_active .. 0 homeless 2024-03-27T21:40:00.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.346+0000 7f61e7fff640 1 -- 172.21.15.60:0/2499822549 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v25) v1 ==== 54+0+16 (secure 0 0 0) 0x7f61f00b2070 con 0x7f6200071a50 2024-03-27T21:40:00.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.346+0000 7f61e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:00.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.346+0000 7f61e7fff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:00.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.347+0000 7f6206134640 10 client.4497.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:00.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.347+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign -1 1 2024-03-27T21:40:00.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.347+0000 7f6206134640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f62001373c0 0x7f62001377a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:00.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.347+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f620013f0c0 osd=0 3 2024-03-27T21:40:00.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.347+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f620013f0c0 osd=0 3 2024-03-27T21:40:00.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.347+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f620013f0c0 osd=0 3 2024-03-27T21:40:00.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.347+0000 7f6206134640 15 client.4497.objecter _session_command_op_remove -1 1 2024-03-27T21:40:00.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.347+0000 7f6206134640 20 client.4497.objecter get_session s=0x7f620013f0c0 osd=0 3 2024-03-27T21:40:00.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.347+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign 0 1 2024-03-27T21:40:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.347+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f620013f0c0 osd=0 4 2024-03-27T21:40:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.347+0000 7f6206134640 10 client.4497.objecter _send_command 1 2024-03-27T21:40:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.347+0000 7f6206134640 1 -- 172.21.15.60:0/2499822549 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62000aa590 con 0x7f62001373c0 2024-03-27T21:40:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.347+0000 7f61e7fff640 10 client.4497.objecter ms_handle_connect 0x7f61ec06c590 2024-03-27T21:40:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.347+0000 7f61feffd640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f61ec06c590 0x7f61ec06ea50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f620013c210 tx=0x7f61f4007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.347+0000 7f61fffff640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f62001373c0 0x7f62001377a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.347+0000 7f61fffff640 10 monclient: get_auth_request con 0x7f62001373c0 auth_method 0 2024-03-27T21:40:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.347+0000 7f61e7fff640 10 client.4497.objecter ms_handle_connect 0x7f62001373c0 2024-03-27T21:40:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.347+0000 7f61fffff640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f62001373c0 0x7f62001377a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.348+0000 7f61e7fff640 1 -- 172.21.15.60:0/2499822549 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f61f800f040 con 0x7f62001373c0 2024-03-27T21:40:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.348+0000 7f61e7fff640 10 client.4497.objecter ms_dispatch 0x7f6200000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.348+0000 7f61e7fff640 10 client.4497.objecter _finish_command 1 = system:0 2024-03-27T21:40:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.348+0000 7f61e7fff640 20 client.4497.objecter put_session s=0x7f620013f0c0 osd=0 4 2024-03-27T21:40:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.348+0000 7f61e7fff640 15 client.4497.objecter _session_command_op_remove 0 1 2024-03-27T21:40:00.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.365+0000 7f6206134640 10 client.4497.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:00.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.365+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign -1 2 2024-03-27T21:40:00.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.365+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f620013f0c0 osd=0 3 2024-03-27T21:40:00.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.365+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f620013f0c0 osd=0 3 2024-03-27T21:40:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.365+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f620013f0c0 osd=0 3 2024-03-27T21:40:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.365+0000 7f6206134640 15 client.4497.objecter _session_command_op_remove -1 2 2024-03-27T21:40:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.365+0000 7f6206134640 20 client.4497.objecter get_session s=0x7f620013f0c0 osd=0 3 2024-03-27T21:40:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.365+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign 0 2 2024-03-27T21:40:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.365+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f620013f0c0 osd=0 4 2024-03-27T21:40:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.365+0000 7f6206134640 10 client.4497.objecter _send_command 2 2024-03-27T21:40:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.365+0000 7f6206134640 1 -- 172.21.15.60:0/2499822549 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f620014bde0 con 0x7f62001373c0 2024-03-27T21:40:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.367+0000 7f61e7fff640 1 -- 172.21.15.60:0/2499822549 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f61f8003f40 con 0x7f62001373c0 2024-03-27T21:40:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.367+0000 7f61e7fff640 10 client.4497.objecter ms_dispatch 0x7f6200000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.367+0000 7f61e7fff640 10 client.4497.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.367+0000 7f61e7fff640 20 client.4497.objecter put_session s=0x7f620013f0c0 osd=0 4 2024-03-27T21:40:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.367+0000 7f61e7fff640 15 client.4497.objecter _session_command_op_remove 0 2 2024-03-27T21:40:00.355 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:00.356 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:00.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.367+0000 7f6206134640 10 client.4497.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:00.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.367+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign -1 3 2024-03-27T21:40:00.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.367+0000 7f6206134640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f6200133110 0x7f62001334f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:00.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.367+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f6200069470 osd=1 3 2024-03-27T21:40:00.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.367+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f6200069470 osd=1 3 2024-03-27T21:40:00.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.367+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f6200069470 osd=1 3 2024-03-27T21:40:00.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.367+0000 7f6206134640 15 client.4497.objecter _session_command_op_remove -1 3 2024-03-27T21:40:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.367+0000 7f6206134640 20 client.4497.objecter get_session s=0x7f6200069470 osd=1 3 2024-03-27T21:40:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.367+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign 1 3 2024-03-27T21:40:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.367+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f6200069470 osd=1 4 2024-03-27T21:40:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.367+0000 7f6206134640 10 client.4497.objecter _send_command 3 2024-03-27T21:40:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.367+0000 7f6206134640 1 -- 172.21.15.60:0/2499822549 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6200133a30 con 0x7f6200133110 2024-03-27T21:40:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.367+0000 7f61ff7fe640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f6200133110 0x7f62001334f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.367+0000 7f61ff7fe640 10 monclient: get_auth_request con 0x7f6200133110 auth_method 0 2024-03-27T21:40:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.368+0000 7f61e7fff640 10 client.4497.objecter ms_handle_connect 0x7f6200133110 2024-03-27T21:40:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.368+0000 7f61ff7fe640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f6200133110 0x7f62001334f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.368+0000 7f61e7fff640 1 -- 172.21.15.60:0/2499822549 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6200133a30 con 0x7f6200133110 2024-03-27T21:40:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.368+0000 7f61e7fff640 10 client.4497.objecter ms_dispatch 0x7f6200000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.368+0000 7f61e7fff640 10 client.4497.objecter _finish_command 3 = system:0 2024-03-27T21:40:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.368+0000 7f61e7fff640 20 client.4497.objecter put_session s=0x7f6200069470 osd=1 4 2024-03-27T21:40:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.368+0000 7f61e7fff640 15 client.4497.objecter _session_command_op_remove 1 3 2024-03-27T21:40:00.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.383+0000 7f6206134640 10 client.4497.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:00.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.383+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign -1 4 2024-03-27T21:40:00.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.383+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f6200069470 osd=1 3 2024-03-27T21:40:00.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.383+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f6200069470 osd=1 3 2024-03-27T21:40:00.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.383+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f6200069470 osd=1 3 2024-03-27T21:40:00.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.383+0000 7f6206134640 15 client.4497.objecter _session_command_op_remove -1 4 2024-03-27T21:40:00.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.383+0000 7f6206134640 20 client.4497.objecter get_session s=0x7f6200069470 osd=1 3 2024-03-27T21:40:00.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.383+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign 1 4 2024-03-27T21:40:00.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.383+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f6200069470 osd=1 4 2024-03-27T21:40:00.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.383+0000 7f6206134640 10 client.4497.objecter _send_command 4 2024-03-27T21:40:00.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.383+0000 7f6206134640 1 -- 172.21.15.60:0/2499822549 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f62000a9260 con 0x7f6200133110 2024-03-27T21:40:00.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.384+0000 7f61e7fff640 1 -- 172.21.15.60:0/2499822549 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f62000a9260 con 0x7f6200133110 2024-03-27T21:40:00.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.384+0000 7f61e7fff640 10 client.4497.objecter ms_dispatch 0x7f6200000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:00.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.384+0000 7f61e7fff640 10 client.4497.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:00.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.384+0000 7f61e7fff640 20 client.4497.objecter put_session s=0x7f6200069470 osd=1 4 2024-03-27T21:40:00.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.384+0000 7f61e7fff640 15 client.4497.objecter _session_command_op_remove 1 4 2024-03-27T21:40:00.371 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:00.373 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:00.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.384+0000 7f6206134640 10 client.4497.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:00.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign -1 5 2024-03-27T21:40:00.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f6206134640 20 client.4497.objecter _get_session osd=-1 returning homeless 2024-03-27T21:40:00.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f6206134640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:00.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f6206134640 10 monclient: _renew_subs 2024-03-27T21:40:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f6206134640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f6206134640 1 -- 172.21.15.60:0/2499822549 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=26}) v3 -- 0x7f62000b1a20 con 0x7f6200071a50 2024-03-27T21:40:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f6206134640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f6206134640 1 -- 172.21.15.60:0/2499822549 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f62000b1d80 con 0x7f6200071a50 2024-03-27T21:40:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f61e7fff640 1 -- 172.21.15.60:0/2499822549 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61f007a540 con 0x7f6200071a50 2024-03-27T21:40:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f61e7fff640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-03-27T21:40:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f61fdffb640 10 client.4497.objecter _check_command_map_dne tid 5 current 25 map_dne_bound 25 2024-03-27T21:40:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f61fdffb640 10 client.4497.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:40:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f61fdffb640 15 client.4497.objecter _session_command_op_remove -1 5 2024-03-27T21:40:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:40:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f6206134640 10 client.4497.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign -1 6 2024-03-27T21:40:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f6206134640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f620002b170 0x7f620002b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f6200134940 osd=3 3 2024-03-27T21:40:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f6200134940 osd=3 3 2024-03-27T21:40:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f6200134940 osd=3 3 2024-03-27T21:40:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f6206134640 15 client.4497.objecter _session_command_op_remove -1 6 2024-03-27T21:40:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f6206134640 20 client.4497.objecter get_session s=0x7f6200134940 osd=3 3 2024-03-27T21:40:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign 3 6 2024-03-27T21:40:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f6200134940 osd=3 4 2024-03-27T21:40:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f6206134640 10 client.4497.objecter _send_command 6 2024-03-27T21:40:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f6206134640 1 -- 172.21.15.60:0/2499822549 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f620002ba90 con 0x7f620002b170 2024-03-27T21:40:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f61feffd640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f620002b170 0x7f620002b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f61feffd640 10 monclient: get_auth_request con 0x7f620002b170 auth_method 0 2024-03-27T21:40:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f61e7fff640 10 client.4497.objecter ms_handle_connect 0x7f620002b170 2024-03-27T21:40:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.385+0000 7f61feffd640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f620002b170 0x7f620002b550 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.386+0000 7f61e7fff640 1 -- 172.21.15.60:0/2499822549 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f620002ba90 con 0x7f620002b170 2024-03-27T21:40:00.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.386+0000 7f61e7fff640 10 client.4497.objecter ms_dispatch 0x7f6200000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:40:00.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.386+0000 7f61e7fff640 10 client.4497.objecter _finish_command 6 = system:0 2024-03-27T21:40:00.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.386+0000 7f61e7fff640 20 client.4497.objecter put_session s=0x7f6200134940 osd=3 4 2024-03-27T21:40:00.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.386+0000 7f61e7fff640 15 client.4497.objecter _session_command_op_remove 3 6 2024-03-27T21:40:00.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.400+0000 7f6206134640 10 client.4497.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:00.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.400+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign -1 7 2024-03-27T21:40:00.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.400+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f6200134940 osd=3 3 2024-03-27T21:40:00.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.400+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f6200134940 osd=3 3 2024-03-27T21:40:00.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.400+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f6200134940 osd=3 3 2024-03-27T21:40:00.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.400+0000 7f6206134640 15 client.4497.objecter _session_command_op_remove -1 7 2024-03-27T21:40:00.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.400+0000 7f6206134640 20 client.4497.objecter get_session s=0x7f6200134940 osd=3 3 2024-03-27T21:40:00.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.400+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign 3 7 2024-03-27T21:40:00.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.400+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f6200134940 osd=3 4 2024-03-27T21:40:00.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.400+0000 7f6206134640 10 client.4497.objecter _send_command 7 2024-03-27T21:40:00.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.400+0000 7f6206134640 1 -- 172.21.15.60:0/2499822549 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f620002c3f0 con 0x7f620002b170 2024-03-27T21:40:00.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.401+0000 7f61e7fff640 1 -- 172.21.15.60:0/2499822549 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f620002c3f0 con 0x7f620002b170 2024-03-27T21:40:00.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.401+0000 7f61e7fff640 10 client.4497.objecter ms_dispatch 0x7f6200000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:00.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.401+0000 7f61e7fff640 10 client.4497.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:00.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.401+0000 7f61e7fff640 20 client.4497.objecter put_session s=0x7f6200134940 osd=3 4 2024-03-27T21:40:00.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.401+0000 7f61e7fff640 15 client.4497.objecter _session_command_op_remove 3 7 2024-03-27T21:40:00.388 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:00.389 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:00.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.402+0000 7f6206134640 10 client.4497.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:00.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.402+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign -1 8 2024-03-27T21:40:00.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.402+0000 7f6206134640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f6200134530 0x7f62001e9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:00.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.402+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f62001343c0 osd=4 3 2024-03-27T21:40:00.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.402+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f62001343c0 osd=4 3 2024-03-27T21:40:00.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.402+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f62001343c0 osd=4 3 2024-03-27T21:40:00.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.402+0000 7f6206134640 15 client.4497.objecter _session_command_op_remove -1 8 2024-03-27T21:40:00.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.402+0000 7f6206134640 20 client.4497.objecter get_session s=0x7f62001343c0 osd=4 3 2024-03-27T21:40:00.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.402+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign 4 8 2024-03-27T21:40:00.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.402+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f62001343c0 osd=4 4 2024-03-27T21:40:00.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.402+0000 7f6206134640 10 client.4497.objecter _send_command 8 2024-03-27T21:40:00.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.402+0000 7f6206134640 1 -- 172.21.15.60:0/2499822549 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62001ec0f0 con 0x7f6200134530 2024-03-27T21:40:00.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.402+0000 7f61fffff640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f6200134530 0x7f62001e9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:00.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.402+0000 7f61fffff640 10 monclient: get_auth_request con 0x7f6200134530 auth_method 0 2024-03-27T21:40:00.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.402+0000 7f61e7fff640 10 client.4497.objecter ms_handle_connect 0x7f6200134530 2024-03-27T21:40:00.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.402+0000 7f61fffff640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f6200134530 0x7f62001e9320 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:00.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.403+0000 7f61e7fff640 1 -- 172.21.15.60:0/2499822549 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f62001ec0f0 con 0x7f6200134530 2024-03-27T21:40:00.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.403+0000 7f61e7fff640 10 client.4497.objecter ms_dispatch 0x7f6200000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:40:00.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.403+0000 7f61e7fff640 10 client.4497.objecter _finish_command 8 = system:0 2024-03-27T21:40:00.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.403+0000 7f61e7fff640 20 client.4497.objecter put_session s=0x7f62001343c0 osd=4 4 2024-03-27T21:40:00.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.403+0000 7f61e7fff640 15 client.4497.objecter _session_command_op_remove 4 8 2024-03-27T21:40:00.403 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:00.417+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:00.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.417+0000 7f6206134640 10 client.4497.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:00.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.417+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign -1 9 2024-03-27T21:40:00.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.417+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f62001343c0 osd=4 3 2024-03-27T21:40:00.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.417+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f62001343c0 osd=4 3 2024-03-27T21:40:00.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.417+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f62001343c0 osd=4 3 2024-03-27T21:40:00.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.417+0000 7f6206134640 15 client.4497.objecter _session_command_op_remove -1 9 2024-03-27T21:40:00.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.417+0000 7f6206134640 20 client.4497.objecter get_session s=0x7f62001343c0 osd=4 3 2024-03-27T21:40:00.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.417+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign 4 9 2024-03-27T21:40:00.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.417+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f62001343c0 osd=4 4 2024-03-27T21:40:00.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.417+0000 7f6206134640 10 client.4497.objecter _send_command 9 2024-03-27T21:40:00.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.417+0000 7f6206134640 1 -- 172.21.15.60:0/2499822549 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f620002be70 con 0x7f6200134530 2024-03-27T21:40:00.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.419+0000 7f61e7fff640 1 -- 172.21.15.60:0/2499822549 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f620002be70 con 0x7f6200134530 2024-03-27T21:40:00.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.419+0000 7f61e7fff640 10 client.4497.objecter ms_dispatch 0x7f6200000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:00.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.419+0000 7f61e7fff640 10 client.4497.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:00.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.419+0000 7f61e7fff640 20 client.4497.objecter put_session s=0x7f62001343c0 osd=4 4 2024-03-27T21:40:00.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.419+0000 7f61e7fff640 15 client.4497.objecter _session_command_op_remove 4 9 2024-03-27T21:40:00.406 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:00.407 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:00.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.419+0000 7f6206134640 10 client.4497.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:00.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.419+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign -1 10 2024-03-27T21:40:00.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.419+0000 7f6206134640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f620002bfe0 0x7f62001ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:00.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.419+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f620002be70 osd=5 3 2024-03-27T21:40:00.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.419+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f620002be70 osd=5 3 2024-03-27T21:40:00.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.419+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f620002be70 osd=5 3 2024-03-27T21:40:00.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.419+0000 7f6206134640 15 client.4497.objecter _session_command_op_remove -1 10 2024-03-27T21:40:00.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.419+0000 7f6206134640 20 client.4497.objecter get_session s=0x7f620002be70 osd=5 3 2024-03-27T21:40:00.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.419+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign 5 10 2024-03-27T21:40:00.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.419+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f620002be70 osd=5 4 2024-03-27T21:40:00.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.419+0000 7f6206134640 10 client.4497.objecter _send_command 10 2024-03-27T21:40:00.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.419+0000 7f6206134640 1 -- 172.21.15.60:0/2499822549 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6200066760 con 0x7f620002bfe0 2024-03-27T21:40:00.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.420+0000 7f61ff7fe640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f620002bfe0 0x7f62001ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:00.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.420+0000 7f61ff7fe640 10 monclient: get_auth_request con 0x7f620002bfe0 auth_method 0 2024-03-27T21:40:00.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.420+0000 7f61e7fff640 10 client.4497.objecter ms_handle_connect 0x7f620002bfe0 2024-03-27T21:40:00.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.420+0000 7f61ff7fe640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f620002bfe0 0x7f62001ea9e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:00.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.421+0000 7f61e7fff640 1 -- 172.21.15.60:0/2499822549 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6200066760 con 0x7f620002bfe0 2024-03-27T21:40:00.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.421+0000 7f61e7fff640 10 client.4497.objecter ms_dispatch 0x7f6200000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:40:00.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.421+0000 7f61e7fff640 10 client.4497.objecter _finish_command 10 = system:0 2024-03-27T21:40:00.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.421+0000 7f61e7fff640 20 client.4497.objecter put_session s=0x7f620002be70 osd=5 4 2024-03-27T21:40:00.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.421+0000 7f61e7fff640 15 client.4497.objecter _session_command_op_remove 5 10 2024-03-27T21:40:00.416 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:00.416 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:00.416 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:00.416 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:00.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.435+0000 7f6206134640 10 client.4497.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:00.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.435+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign -1 11 2024-03-27T21:40:00.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.435+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f620002be70 osd=5 3 2024-03-27T21:40:00.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.435+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f620002be70 osd=5 3 2024-03-27T21:40:00.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.435+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f620002be70 osd=5 3 2024-03-27T21:40:00.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.435+0000 7f6206134640 15 client.4497.objecter _session_command_op_remove -1 11 2024-03-27T21:40:00.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.435+0000 7f6206134640 20 client.4497.objecter get_session s=0x7f620002be70 osd=5 3 2024-03-27T21:40:00.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.435+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign 5 11 2024-03-27T21:40:00.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.435+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f620002be70 osd=5 4 2024-03-27T21:40:00.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.435+0000 7f6206134640 10 client.4497.objecter _send_command 11 2024-03-27T21:40:00.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.435+0000 7f6206134640 1 -- 172.21.15.60:0/2499822549 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6200066760 con 0x7f620002bfe0 2024-03-27T21:40:00.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.437+0000 7f61e7fff640 1 -- 172.21.15.60:0/2499822549 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6200066760 con 0x7f620002bfe0 2024-03-27T21:40:00.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.437+0000 7f61e7fff640 10 client.4497.objecter ms_dispatch 0x7f6200000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:00.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.437+0000 7f61e7fff640 10 client.4497.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:00.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.437+0000 7f61e7fff640 20 client.4497.objecter put_session s=0x7f620002be70 osd=5 4 2024-03-27T21:40:00.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.437+0000 7f61e7fff640 15 client.4497.objecter _session_command_op_remove 5 11 2024-03-27T21:40:00.424 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:00.425 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:00.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.437+0000 7f6206134640 10 client.4497.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:00.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.437+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign -1 12 2024-03-27T21:40:00.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.437+0000 7f6206134640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f62001eb430 0x7f62001eb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:00.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.437+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f6200066760 osd=6 3 2024-03-27T21:40:00.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.437+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f6200066760 osd=6 3 2024-03-27T21:40:00.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.437+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f6200066760 osd=6 3 2024-03-27T21:40:00.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.437+0000 7f6206134640 15 client.4497.objecter _session_command_op_remove -1 12 2024-03-27T21:40:00.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.437+0000 7f6206134640 20 client.4497.objecter get_session s=0x7f6200066760 osd=6 3 2024-03-27T21:40:00.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.437+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign 6 12 2024-03-27T21:40:00.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.437+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f6200066760 osd=6 4 2024-03-27T21:40:00.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.437+0000 7f6206134640 10 client.4497.objecter _send_command 12 2024-03-27T21:40:00.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.437+0000 7f6206134640 1 -- 172.21.15.60:0/2499822549 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6200066a10 con 0x7f62001eb430 2024-03-27T21:40:00.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.437+0000 7f61feffd640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f62001eb430 0x7f62001eb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:00.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.438+0000 7f61feffd640 10 monclient: get_auth_request con 0x7f62001eb430 auth_method 0 2024-03-27T21:40:00.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.438+0000 7f61feffd640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f62001eb430 0x7f62001eb890 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:00.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.438+0000 7f61e7fff640 10 client.4497.objecter ms_handle_connect 0x7f62001eb430 2024-03-27T21:40:00.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.439+0000 7f61e7fff640 1 -- 172.21.15.60:0/2499822549 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6200066a10 con 0x7f62001eb430 2024-03-27T21:40:00.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.439+0000 7f61e7fff640 10 client.4497.objecter ms_dispatch 0x7f6200000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:40:00.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.439+0000 7f61e7fff640 10 client.4497.objecter _finish_command 12 = system:0 2024-03-27T21:40:00.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.439+0000 7f61e7fff640 20 client.4497.objecter put_session s=0x7f6200066760 osd=6 4 2024-03-27T21:40:00.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.439+0000 7f61e7fff640 15 client.4497.objecter _session_command_op_remove 6 12 2024-03-27T21:40:00.431 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:40:00.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.452+0000 7f6206134640 10 client.4497.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:00.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.452+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign -1 13 2024-03-27T21:40:00.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.452+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f6200066760 osd=6 3 2024-03-27T21:40:00.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.452+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f6200066760 osd=6 3 2024-03-27T21:40:00.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.452+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f6200066760 osd=6 3 2024-03-27T21:40:00.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.452+0000 7f6206134640 15 client.4497.objecter _session_command_op_remove -1 13 2024-03-27T21:40:00.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.452+0000 7f6206134640 20 client.4497.objecter get_session s=0x7f6200066760 osd=6 3 2024-03-27T21:40:00.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.452+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign 6 13 2024-03-27T21:40:00.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.452+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f6200066760 osd=6 4 2024-03-27T21:40:00.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.452+0000 7f6206134640 10 client.4497.objecter _send_command 13 2024-03-27T21:40:00.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.452+0000 7f6206134640 1 -- 172.21.15.60:0/2499822549 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6200066a10 con 0x7f62001eb430 2024-03-27T21:40:00.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.454+0000 7f61e7fff640 1 -- 172.21.15.60:0/2499822549 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6200066a10 con 0x7f62001eb430 2024-03-27T21:40:00.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.454+0000 7f61e7fff640 10 client.4497.objecter ms_dispatch 0x7f6200000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:00.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.454+0000 7f61e7fff640 10 client.4497.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:00.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.454+0000 7f61e7fff640 20 client.4497.objecter put_session s=0x7f6200066760 osd=6 4 2024-03-27T21:40:00.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.454+0000 7f61e7fff640 15 client.4497.objecter _session_command_op_remove 6 13 2024-03-27T21:40:00.441 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:00.442 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:00.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.454+0000 7f6206134640 10 client.4497.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:00.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.454+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign -1 14 2024-03-27T21:40:00.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.454+0000 7f6206134640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f62001f0790 0x7f62001f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:00.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.454+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f6200066a10 osd=7 3 2024-03-27T21:40:00.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.454+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f6200066a10 osd=7 3 2024-03-27T21:40:00.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.454+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f6200066a10 osd=7 3 2024-03-27T21:40:00.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.454+0000 7f6206134640 15 client.4497.objecter _session_command_op_remove -1 14 2024-03-27T21:40:00.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.454+0000 7f6206134640 20 client.4497.objecter get_session s=0x7f6200066a10 osd=7 3 2024-03-27T21:40:00.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.454+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign 7 14 2024-03-27T21:40:00.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.454+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f6200066a10 osd=7 4 2024-03-27T21:40:00.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.454+0000 7f6206134640 10 client.4497.objecter _send_command 14 2024-03-27T21:40:00.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.454+0000 7f6206134640 1 -- 172.21.15.60:0/2499822549 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62001f5950 con 0x7f62001f0790 2024-03-27T21:40:00.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.454+0000 7f61fffff640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f62001f0790 0x7f62001f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:00.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.455+0000 7f61fffff640 10 monclient: get_auth_request con 0x7f62001f0790 auth_method 0 2024-03-27T21:40:00.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.455+0000 7f61e7fff640 10 client.4497.objecter ms_handle_connect 0x7f62001f0790 2024-03-27T21:40:00.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.455+0000 7f61fffff640 1 --2- 172.21.15.60:0/2499822549 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f62001f0790 0x7f62001f2b80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:00.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.456+0000 7f61e7fff640 1 -- 172.21.15.60:0/2499822549 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f62001f5950 con 0x7f62001f0790 2024-03-27T21:40:00.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.456+0000 7f61e7fff640 10 client.4497.objecter ms_dispatch 0x7f6200000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:40:00.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.456+0000 7f61e7fff640 10 client.4497.objecter _finish_command 14 = system:0 2024-03-27T21:40:00.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.456+0000 7f61e7fff640 20 client.4497.objecter put_session s=0x7f6200066a10 osd=7 4 2024-03-27T21:40:00.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.456+0000 7f61e7fff640 15 client.4497.objecter _session_command_op_remove 7 14 2024-03-27T21:40:00.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.469+0000 7f6206134640 10 client.4497.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:00.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.469+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign -1 15 2024-03-27T21:40:00.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.469+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f6200066a10 osd=7 3 2024-03-27T21:40:00.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.469+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f6200066a10 osd=7 3 2024-03-27T21:40:00.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.469+0000 7f6206134640 20 client.4497.objecter _get_session s=0x7f6200066a10 osd=7 3 2024-03-27T21:40:00.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.469+0000 7f6206134640 15 client.4497.objecter _session_command_op_remove -1 15 2024-03-27T21:40:00.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.469+0000 7f6206134640 20 client.4497.objecter get_session s=0x7f6200066a10 osd=7 3 2024-03-27T21:40:00.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.469+0000 7f6206134640 15 client.4497.objecter _session_command_op_assign 7 15 2024-03-27T21:40:00.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.469+0000 7f6206134640 20 client.4497.objecter put_session s=0x7f6200066a10 osd=7 4 2024-03-27T21:40:00.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.469+0000 7f6206134640 10 client.4497.objecter _send_command 15 2024-03-27T21:40:00.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.469+0000 7f6206134640 1 -- 172.21.15.60:0/2499822549 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f62001f30c0 con 0x7f62001f0790 2024-03-27T21:40:00.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.471+0000 7f61e7fff640 1 -- 172.21.15.60:0/2499822549 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f62001f30c0 con 0x7f62001f0790 2024-03-27T21:40:00.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.471+0000 7f61e7fff640 10 client.4497.objecter ms_dispatch 0x7f6200000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:00.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.471+0000 7f61e7fff640 10 client.4497.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:00.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.471+0000 7f61e7fff640 20 client.4497.objecter put_session s=0x7f6200066a10 osd=7 4 2024-03-27T21:40:00.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:00.471+0000 7f61e7fff640 15 client.4497.objecter _session_command_op_remove 7 15 2024-03-27T21:40:00.458 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:00.458 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:00.461 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:40:00.461 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:40:00.515 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:00.529+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:00.577 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:00.577 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:00.577 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:00.577 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:00.577 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:00.586 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:40:00.605 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:00.619+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:00.706 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:00.720+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:00.731 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:00.731 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:00.731 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:00.732 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:00.732 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:00.740 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:40:00.807 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:00.821+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:00.885 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:00.886 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:00.886 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:00.886 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:00.894 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:40:00.939 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:00.953+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:01.000 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:01.000 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:01.000 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:01.000 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:01.000 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:01.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:40:01.009 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:01.022+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:01.109 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:01.123+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:01.154 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:01.154 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:01.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:01.154 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:01.154 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:01.162 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:40:01.246 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:01.260+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:01.309 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:01.309 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:01.309 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:01.309 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:01.311 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:01.325+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:01.317 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:40:01.412 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:01.426+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:01.462 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:01.462 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:01.462 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:01.463 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:01.463 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:01.471 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:40:01.514 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:01.528+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:01.576 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:01.576 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:01.576 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:01.576 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:01.576 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:01.584 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:40:01.625 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:01.640+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:01.636 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-27T21:40:01.714 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:01.728+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:01.730 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:01.730 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:01.730 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:01.730 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:01.739 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:40:01.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.762+0000 7f61d0f67640 10 monclient: get_monmap_and_config 2024-03-27T21:40:01.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.762+0000 7f61d0f67640 10 monclient: build_initial_monmap 2024-03-27T21:40:01.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.762+0000 7f61d0f67640 10 monclient: monmap: 2024-03-27T21:40:01.750 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:01.750 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:01.750 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:01.763934+0000 2024-03-27T21:40:01.750 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:01.763934+0000 2024-03-27T21:40:01.750 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:01.750 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:01.751 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:01.751 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:01.751 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:01.751 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:01.751 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:01.751 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:01.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.763+0000 7f61d0f67640 1 Processor -- start 2024-03-27T21:40:01.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.763+0000 7f61d0f67640 1 -- start start 2024-03-27T21:40:01.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.763+0000 7f61d0f67640 10 monclient: init 2024-03-27T21:40:01.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61d0f67640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:01.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61d0f67640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-03-27T21:40:01.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61d0f67640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f61cc071a50 0x7f61cc0a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:01.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61d0f67640 10 monclient(hunting): picked mon.noname-b con 0x7f61cc071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:01.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61d0f67640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f61cc13e810 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:01.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61d0f67640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f61cc13ec50 0x7f61cc141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:01.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61d0f67640 10 monclient(hunting): picked mon.noname-c con 0x7f61cc13ec50 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:01.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61d0f67640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61cc0a8e90 con 0x7f61cc13e810 2024-03-27T21:40:01.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61d0f67640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:01.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61d0f67640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61cc071e30 con 0x7f61cc13ec50 2024-03-27T21:40:01.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61d0f67640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:01.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61d0f67640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61cc0aa040 con 0x7f61cc071a50 2024-03-27T21:40:01.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61d0f67640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:01.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61d0f67640 10 monclient(hunting): authenticate will time out at 842.985358s 2024-03-27T21:40:01.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61ca59c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f61cc071a50 0x7f61cc0a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:01.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61ca59c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f61cc071a50 0x7f61cc0a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:39402/0 (socket says 172.21.15.60:39402) 2024-03-27T21:40:01.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61ca59c640 1 -- 172.21.15.60:0/1759861467 learned_addr learned my addr 172.21.15.60:0/1759861467 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:01.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61ca59c640 10 monclient(hunting): get_auth_request con 0x7f61cc071a50 auth_method 0 2024-03-27T21:40:01.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:01.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61ca59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:01.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:01.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61cad9d640 1 --2- 172.21.15.60:0/1759861467 >> v2:172.21.15.191:3300/0 conn(0x7f61cc13ec50 0x7f61cc141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:01.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61cad9d640 10 monclient(hunting): get_auth_request con 0x7f61cc13ec50 auth_method 0 2024-03-27T21:40:01.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:01.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61cad9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:01.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:01.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:01.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:01.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:01.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:01.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:01.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:01.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61ca59c640 10 monclient(hunting): handle_auth_done global_id 4415 payload 274 2024-03-27T21:40:01.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61ca59c640 1 -- 172.21.15.60:0/1759861467 >> v2:172.21.15.191:3300/0 conn(0x7f61cc13ec50 msgr2=0x7f61cc141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:01.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61ca59c640 1 --2- 172.21.15.60:0/1759861467 >> v2:172.21.15.191:3300/0 conn(0x7f61cc13ec50 0x7f61cc141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:01.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61ca59c640 1 -- 172.21.15.60:0/1759861467 >> v1:172.21.15.191:6789/0 conn(0x7f61cc13e810 legacy=0x7f61cc141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:01.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61ca59c640 10 monclient: _finish_hunting 0 2024-03-27T21:40:01.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61ca59c640 1 monclient: found mon.noname-b 2024-03-27T21:40:01.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61ca59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:01.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61ca59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:01.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.764+0000 7f61ca59c640 1 -- 172.21.15.60:0/1759861467 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61cc142280 con 0x7f61cc071a50 2024-03-27T21:40:01.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.765+0000 7f61ca59c640 1 --2- 172.21.15.60:0/1759861467 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f61cc071a50 0x7f61cc0a87e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f61bc009870 tx=0x7f61bc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36d34ed47a6c4276 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:01.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.765+0000 7f61c9d9b640 1 -- 172.21.15.60:0/1759861467 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc030840 con 0x7f61cc071a50 2024-03-27T21:40:01.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.765+0000 7f61c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:01.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.765+0000 7f61c9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:01.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.765+0000 7f61c9d9b640 10 monclient: dump: 2024-03-27T21:40:01.764 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:01.765 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:01.765 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:01.765 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:01.765 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:01.765 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:01.765 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:01.765 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:01.765 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:01.765 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:01.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.765+0000 7f61c9d9b640 10 monclient: _finish_auth 0 2024-03-27T21:40:01.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.765+0000 7f61c9d9b640 10 monclient: _check_auth_tickets 2024-03-27T21:40:01.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.765+0000 7f61c9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:01.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.765+0000 7f61c9d9b640 1 -- 172.21.15.60:0/1759861467 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61bc0309a0 con 0x7f61cc071a50 2024-03-27T21:40:01.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.765+0000 7f61d0f67640 5 monclient: authenticate success, global_id 4415 2024-03-27T21:40:01.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.765+0000 7f61d0f67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:01.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.765+0000 7f61c9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:01.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.765+0000 7f61d0f67640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:01.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.765+0000 7f61c9d9b640 1 -- 172.21.15.60:0/1759861467 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc030d30 con 0x7f61cc071a50 2024-03-27T21:40:01.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.765+0000 7f61c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:01.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.765+0000 7f61d0f67640 4 set_mon_vals no callback set 2024-03-27T21:40:01.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.765+0000 7f61c9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:01.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.765+0000 7f61c9d9b640 10 monclient: dump: 2024-03-27T21:40:01.766 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:01.766 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:01.766 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:01.766 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:01.766 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:01.766 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:01.767 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:01.767 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:01.767 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:01.767 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:01.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.766+0000 7f61d0f67640 10 monclient: shutdown 2024-03-27T21:40:01.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.766+0000 7f61d0f67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:01.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.766+0000 7f61d0f67640 1 -- 172.21.15.60:0/1759861467 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f61cc071a50 msgr2=0x7f61cc0a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:01.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.766+0000 7f61d0f67640 1 --2- 172.21.15.60:0/1759861467 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f61cc071a50 0x7f61cc0a87e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f61bc009870 tx=0x7f61bc02b980 comp rx=0 tx=0).stop 2024-03-27T21:40:01.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.766+0000 7f61d0f67640 1 -- 172.21.15.60:0/1759861467 shutdown_connections 2024-03-27T21:40:01.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.766+0000 7f61d0f67640 1 --2- 172.21.15.60:0/1759861467 >> v2:172.21.15.191:3300/0 conn(0x7f61cc13ec50 0x7f61cc141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:01.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.766+0000 7f61d0f67640 1 --2- 172.21.15.60:0/1759861467 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f61cc071a50 0x7f61cc0a87e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:01.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.766+0000 7f61d0f67640 1 -- 172.21.15.60:0/1759861467 >> 172.21.15.60:0/1759861467 conn(0x7f61cc0b7300 msgr2=0x7f61cc0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:01.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.766+0000 7f61d0f67640 1 -- 172.21.15.60:0/1759861467 shutdown_connections 2024-03-27T21:40:01.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.766+0000 7f61d0f67640 1 -- 172.21.15.60:0/1759861467 wait complete. 2024-03-27T21:40:01.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.767+0000 7f61d0f67640 10 monclient: build_initial_monmap 2024-03-27T21:40:01.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.767+0000 7f61d0f67640 10 monclient: monmap: 2024-03-27T21:40:01.768 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:01.768 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:01.768 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:01.768 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:01.768 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:01.768 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:01.768 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:01.768 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:01.768 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:01.768 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:01.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.767+0000 7f61d0f67640 1 librados: starting msgr at 2024-03-27T21:40:01.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.767+0000 7f61d0f67640 1 librados: starting objecter 2024-03-27T21:40:01.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.767+0000 7f61d0f67640 1 Processor -- start 2024-03-27T21:40:01.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.767+0000 7f61d0f67640 1 -- start start 2024-03-27T21:40:01.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.767+0000 7f61d0f67640 1 librados: setting wanted keys 2024-03-27T21:40:01.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.767+0000 7f61d0f67640 1 librados: calling monclient init 2024-03-27T21:40:01.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.767+0000 7f61d0f67640 10 monclient: init 2024-03-27T21:40:01.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61d0f67640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:01.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61d0f67640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:40:01.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61d0f67640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f61cc071a50 0x7f61cc1d9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:01.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61d0f67640 10 monclient(hunting): picked mon.noname-c con 0x7f61cc071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:01.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61d0f67640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61cc13e810 0x7f61cc13a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:01.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61d0f67640 10 monclient(hunting): picked mon.noname-a con 0x7f61cc13e810 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:01.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61d0f67640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61cc13ec50 0x7f61cc13aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:01.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61d0f67640 10 monclient(hunting): picked mon.noname-b con 0x7f61cc13ec50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:01.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61d0f67640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:01.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61d0f67640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61cc13b010 con 0x7f61cc13e810 2024-03-27T21:40:01.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61d0f67640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:01.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61d0f67640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61cc138e00 con 0x7f61cc13ec50 2024-03-27T21:40:01.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61d0f67640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:01.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61d0f67640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61cc139d20 con 0x7f61cc071a50 2024-03-27T21:40:01.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61d0f67640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:01.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61d0f67640 10 monclient(hunting): authenticate will time out at 842.989053s 2024-03-27T21:40:01.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61ca59c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f61cc071a50 0x7f61cc1d9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:01.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61ca59c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f61cc071a50 0x7f61cc1d9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:39410/0 (socket says 172.21.15.60:39410) 2024-03-27T21:40:01.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61ca59c640 1 -- 172.21.15.60:0/553034812 learned_addr learned my addr 172.21.15.60:0/553034812 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:01.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61ca59c640 10 monclient(hunting): get_auth_request con 0x7f61cc071a50 auth_method 0 2024-03-27T21:40:01.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:01.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61ca59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:01.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:01.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c3fff640 1 --2- 172.21.15.60:0/553034812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61cc13e810 0x7f61cc13a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:01.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c3fff640 10 monclient(hunting): get_auth_request con 0x7f61cc13e810 auth_method 0 2024-03-27T21:40:01.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:01.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:01.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:01.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61cad9d640 1 --2- 172.21.15.60:0/553034812 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61cc13ec50 0x7f61cc13aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:01.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:01.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:01.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:01.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61cad9d640 10 monclient(hunting): get_auth_request con 0x7f61cc13ec50 auth_method 0 2024-03-27T21:40:01.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:01.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61cad9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:01.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:01.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:01.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:01.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:01.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:01.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:01.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:01.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61ca59c640 10 monclient(hunting): handle_auth_done global_id 4418 payload 824 2024-03-27T21:40:01.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61ca59c640 1 -- 172.21.15.60:0/553034812 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61cc13ec50 msgr2=0x7f61cc13aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:01.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61ca59c640 1 --2- 172.21.15.60:0/553034812 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61cc13ec50 0x7f61cc13aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:01.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61ca59c640 1 -- 172.21.15.60:0/553034812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61cc13e810 msgr2=0x7f61cc13a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:01.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61ca59c640 1 --2- 172.21.15.60:0/553034812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61cc13e810 0x7f61cc13a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:01.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61ca59c640 10 monclient: _finish_hunting 0 2024-03-27T21:40:01.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61ca59c640 1 monclient: found mon.noname-c 2024-03-27T21:40:01.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61ca59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:01.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61ca59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:40:01.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61ca59c640 1 -- 172.21.15.60:0/553034812 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61bc0094d0 con 0x7f61cc071a50 2024-03-27T21:40:01.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61ca59c640 1 --2- 172.21.15.60:0/553034812 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f61cc071a50 0x7f61cc1d9a90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f61bc000c00 tx=0x7f61bc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:01.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c2ffd640 10 client.?.objecter ms_handle_connect 0x7f61cc071a50 2024-03-27T21:40:01.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:01.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c2ffd640 1 -- 172.21.15.60:0/553034812 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc039a90 con 0x7f61cc071a50 2024-03-27T21:40:01.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:01.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:01.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c2ffd640 10 monclient: dump: 2024-03-27T21:40:01.773 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:01.773 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:01.773 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:01.773 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:01.773 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:01.773 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:01.773 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:01.773 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:01.773 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:01.773 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:01.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c2ffd640 10 monclient: _finish_auth 0 2024-03-27T21:40:01.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c2ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:40:01.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:01.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c2ffd640 1 -- 172.21.15.60:0/553034812 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61bc039bf0 con 0x7f61cc071a50 2024-03-27T21:40:01.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61d0f67640 5 monclient: authenticate success, global_id 4418 2024-03-27T21:40:01.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:01.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c2ffd640 1 -- 172.21.15.60:0/553034812 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc043410 con 0x7f61cc071a50 2024-03-27T21:40:01.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:01.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:01.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c2ffd640 10 monclient: dump: 2024-03-27T21:40:01.774 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:01.774 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:01.774 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:01.774 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:01.774 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:01.775 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:01.775 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:01.775 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:01.775 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:01.775 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:01.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.768+0000 7f61c959a640 4 set_mon_vals no callback set 2024-03-27T21:40:01.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.769+0000 7f61d0f67640 10 monclient: _renew_subs 2024-03-27T21:40:01.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.769+0000 7f61d0f67640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:01.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.769+0000 7f61d0f67640 1 -- 172.21.15.60:0/553034812 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61cc1374c0 con 0x7f61cc071a50 2024-03-27T21:40:01.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.769+0000 7f61d0f67640 10 client.4418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:01.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.769+0000 7f61d0f67640 10 monclient: _renew_subs 2024-03-27T21:40:01.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.769+0000 7f61d0f67640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:01.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.769+0000 7f61d0f67640 1 -- 172.21.15.60:0/553034812 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61cc1da4d0 con 0x7f61cc071a50 2024-03-27T21:40:01.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.769+0000 7f61d0f67640 1 librados: init done 2024-03-27T21:40:01.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.770+0000 7f61d0f67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:01.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.770+0000 7f61d0f67640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:01.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.770+0000 7f61d0f67640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:01.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.770+0000 7f61d0f67640 1 -- 172.21.15.60:0/553034812 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61cc067900 con 0x7f61cc071a50 2024-03-27T21:40:01.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.770+0000 7f61c2ffd640 1 -- 172.21.15.60:0/553034812 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f61bc02f390 con 0x7f61cc071a50 2024-03-27T21:40:01.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.770+0000 7f61c2ffd640 1 --2- 172.21.15.60:0/553034812 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f61ac06c590 0x7f61ac06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:01.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.770+0000 7f61c2ffd640 1 -- 172.21.15.60:0/553034812 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f61bc0ab4d0 con 0x7f61cc071a50 2024-03-27T21:40:01.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.770+0000 7f61c2ffd640 10 client.4418.objecter ms_dispatch 0x7f61cc000b90 osd_map(25..25 src has 1..25) v4 2024-03-27T21:40:01.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.770+0000 7f61c2ffd640 3 client.4418.objecter handle_osd_map got epochs [25,25] > 0 2024-03-27T21:40:01.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.770+0000 7f61c2ffd640 3 client.4418.objecter handle_osd_map decoding full epoch 25 2024-03-27T21:40:01.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.771+0000 7f61c3fff640 1 --2- 172.21.15.60:0/553034812 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f61ac06c590 0x7f61ac06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:01.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.771+0000 7f61c3fff640 10 monclient: get_auth_request con 0x7f61ac06c590 auth_method 0 2024-03-27T21:40:01.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.771+0000 7f61c2ffd640 20 client.4418.objecter dump_active .. 0 homeless 2024-03-27T21:40:01.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.771+0000 7f61c2ffd640 10 client.4418.objecter ms_handle_connect 0x7f61ac06c590 2024-03-27T21:40:01.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.771+0000 7f61c3fff640 1 --2- 172.21.15.60:0/553034812 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f61ac06c590 0x7f61ac06ea50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f61cc1d73e0 tx=0x7f61b4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:01.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.777+0000 7f61c2ffd640 1 -- 172.21.15.60:0/553034812 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f61bc076b00 con 0x7f61cc071a50 2024-03-27T21:40:01.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.777+0000 7f61c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:01.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.777+0000 7f61c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:01.816 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:01.830+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:01.844 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:01.844 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:01.844 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:01.844 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:01.844 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:01.853 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:40:01.917 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:01.931+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.967+0000 7f61d0f67640 10 start_mon_command cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:40:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.967+0000 7f61d0f67640 10 monclient: _send_command 2 [{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:40:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.967+0000 7f61d0f67640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:01.967+0000 7f61d0f67640 1 -- 172.21.15.60:0/553034812 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f61cc13a1c0 con 0x7f61cc071a50 2024-03-27T21:40:01.960 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:01.960 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:01.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:01.960 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-03-27T21:40:01.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.960 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:01.960 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:01.960 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 22.540427731000001, 2024-03-27T21:40:01.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43537978700000002, 2024-03-27T21:40:01.960 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:01.960 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:01.961 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:01.961 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:01.961 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:01.961 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-03-27T21:40:01.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.961 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:01.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:01.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:01.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:01.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:01.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:01.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:01.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:01.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432019+0000", 2024-03-27T21:40:01.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-03-27T21:40:01.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:01.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432027+0000", 2024-03-27T21:40:01.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1200000000000002e-06 2024-03-27T21:40:01.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:01.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432039+0000", 2024-03-27T21:40:01.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1123e-05 2024-03-27T21:40:01.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:01.963 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432051+0000", 2024-03-27T21:40:01.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-03-27T21:40:01.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:01.963 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432157+0000", 2024-03-27T21:40:01.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106846 2024-03-27T21:40:01.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-03-27T21:40:01.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432176+0000", 2024-03-27T21:40:01.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8490000000000001e-05 2024-03-27T21:40:01.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:01.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866170+0000", 2024-03-27T21:40:01.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43399370100000001 2024-03-27T21:40:01.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:01.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866335+0000", 2024-03-27T21:40:01.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016559699999999999 2024-03-27T21:40:01.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.965 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:01.965 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866495+0000", 2024-03-27T21:40:01.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015944200000000001 2024-03-27T21:40:01.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.965 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:01.965 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867012+0000", 2024-03-27T21:40:01.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0005176 2024-03-27T21:40:01.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.965 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:01.965 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867334+0000", 2024-03-27T21:40:01.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032158600000000001 2024-03-27T21:40:01.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:01.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867374+0000", 2024-03-27T21:40:01.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0547000000000003e-05 2024-03-27T21:40:01.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:01.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867400+0000", 2024-03-27T21:40:01.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5958000000000001e-05 2024-03-27T21:40:01.966 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:01.966 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:01.966 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:01.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.966 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:01.967 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:01.967 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 22.043045280000001, 2024-03-27T21:40:01.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064560310000000001, 2024-03-27T21:40:01.967 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:01.967 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:01.967 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:01.968 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:01.968 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:01.968 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 70 2024-03-27T21:40:01.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.968 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:01.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:01.968 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:01.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:01.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:01.968 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:01.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:01.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:01.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929402+0000", 2024-03-27T21:40:01.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:01.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:01.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929407+0000", 2024-03-27T21:40:01.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-06 2024-03-27T21:40:01.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:01.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929414+0000", 2024-03-27T21:40:01.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5689999999999999e-06 2024-03-27T21:40:01.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:01.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929419+0000", 2024-03-27T21:40:01.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-03-27T21:40:01.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:01.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930245+0000", 2024-03-27T21:40:01.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082521500000000002 2024-03-27T21:40:01.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:01.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930287+0000", 2024-03-27T21:40:01.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2305000000000002e-05 2024-03-27T21:40:01.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:01.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930367+0000", 2024-03-27T21:40:01.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9589e-05 2024-03-27T21:40:01.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:01.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.931214+0000", 2024-03-27T21:40:01.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000847827 2024-03-27T21:40:01.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:01.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935807+0000", 2024-03-27T21:40:01.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045923780000000003 2024-03-27T21:40:01.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:01.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935840+0000", 2024-03-27T21:40:01.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3501999999999999e-05 2024-03-27T21:40:01.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:01.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935859+0000", 2024-03-27T21:40:01.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8845e-05 2024-03-27T21:40:01.972 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:01.972 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:01.972 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:01.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.973 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:01.973 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:01.973 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 22.035785689000001, 2024-03-27T21:40:01.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071695429999999996, 2024-03-27T21:40:01.973 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:01.973 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:01.973 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:01.973 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:01.973 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:01.973 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-03-27T21:40:01.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.973 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:01.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:01.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:01.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:01.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:01.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:01.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:01.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:01.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936662+0000", 2024-03-27T21:40:01.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:01.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.975 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:01.975 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936737+0000", 2024-03-27T21:40:01.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5303999999999996e-05 2024-03-27T21:40:01.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.975 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:01.975 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936746+0000", 2024-03-27T21:40:01.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6340000000000007e-06 2024-03-27T21:40:01.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.975 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:01.975 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936753+0000", 2024-03-27T21:40:01.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-03-27T21:40:01.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:01.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937926+0000", 2024-03-27T21:40:01.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011731280000000001 2024-03-27T21:40:01.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:01.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937969+0000", 2024-03-27T21:40:01.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3285999999999999e-05 2024-03-27T21:40:01.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:01.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938040+0000", 2024-03-27T21:40:01.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0358000000000006e-05 2024-03-27T21:40:01.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.977 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:01.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939324+0000", 2024-03-27T21:40:01.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012844899999999999 2024-03-27T21:40:01.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.977 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:01.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943780+0000", 2024-03-27T21:40:01.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044555630000000001 2024-03-27T21:40:01.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:01.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943811+0000", 2024-03-27T21:40:01.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1415999999999999e-05 2024-03-27T21:40:01.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:01.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943832+0000", 2024-03-27T21:40:01.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0981000000000001e-05 2024-03-27T21:40:01.978 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:01.978 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:01.978 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:01.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.979 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:01.979 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:01.979 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 22.035687673000002, 2024-03-27T21:40:01.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071139740000000003, 2024-03-27T21:40:01.979 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:01.979 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:01.979 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:01.979 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:01.979 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:01.979 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-03-27T21:40:01.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.979 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:01.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:01.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:01.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:01.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:01.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:01.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:01.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:01.981 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936760+0000", 2024-03-27T21:40:01.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:01.981 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.981 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:01.981 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936765+0000", 2024-03-27T21:40:01.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6330000000000004e-06 2024-03-27T21:40:01.981 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.981 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:01.981 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936771+0000", 2024-03-27T21:40:01.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0560000000000001e-06 2024-03-27T21:40:01.981 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.982 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:01.982 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936776+0000", 2024-03-27T21:40:01.982 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1510000000000003e-06 2024-03-27T21:40:01.982 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.982 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.982 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:01.982 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938605+0000", 2024-03-27T21:40:01.982 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001829589 2024-03-27T21:40:01.982 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.982 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.982 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:01.982 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938657+0000", 2024-03-27T21:40:01.982 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1860000000000002e-05 2024-03-27T21:40:01.982 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.983 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.983 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:01.983 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938734+0000", 2024-03-27T21:40:01.983 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6639999999999998e-05 2024-03-27T21:40:01.983 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.983 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.983 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:01.983 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939562+0000", 2024-03-27T21:40:01.983 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082803100000000001 2024-03-27T21:40:01.983 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.983 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.983 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:01.983 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943838+0000", 2024-03-27T21:40:01.983 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042757050000000003 2024-03-27T21:40:01.983 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.984 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.984 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:01.984 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943859+0000", 2024-03-27T21:40:01.984 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1175e-05 2024-03-27T21:40:01.984 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.984 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.984 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:01.984 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943875+0000", 2024-03-27T21:40:01.984 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5744e-05 2024-03-27T21:40:01.984 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:01.984 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:01.984 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:01.984 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.984 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.984 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:01.985 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:01.985 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 21.892173622000001, 2024-03-27T21:40:01.985 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059712230000000003, 2024-03-27T21:40:01.985 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:01.985 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:01.985 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:01.985 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:01.985 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:01.985 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 112 2024-03-27T21:40:01.985 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.985 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:01.985 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.985 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:01.985 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:01.985 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:01.986 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.986 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.986 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:01.986 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:01.986 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:01.986 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.986 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.986 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:01.986 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080274+0000", 2024-03-27T21:40:01.986 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:01.986 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.986 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.986 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:01.986 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080318+0000", 2024-03-27T21:40:01.986 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3634999999999998e-05 2024-03-27T21:40:01.987 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.987 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.987 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:01.987 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080329+0000", 2024-03-27T21:40:01.987 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1232e-05 2024-03-27T21:40:01.987 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.987 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.987 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:01.987 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080333+0000", 2024-03-27T21:40:01.987 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6009999999999999e-06 2024-03-27T21:40:01.987 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.987 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.987 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:01.987 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083786+0000", 2024-03-27T21:40:01.987 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034528940000000002 2024-03-27T21:40:01.988 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.988 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.988 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:01.988 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083861+0000", 2024-03-27T21:40:01.988 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4893000000000003e-05 2024-03-27T21:40:01.988 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.988 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.988 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:01.988 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083981+0000", 2024-03-27T21:40:01.988 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012058299999999999 2024-03-27T21:40:01.988 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.988 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.988 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:01.988 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086148+0000", 2024-03-27T21:40:01.988 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021671469999999999 2024-03-27T21:40:01.989 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.989 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.989 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:01.989 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086195+0000", 2024-03-27T21:40:01.989 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6768999999999999e-05 2024-03-27T21:40:01.989 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.989 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.989 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:01.989 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086224+0000", 2024-03-27T21:40:01.989 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8758000000000001e-05 2024-03-27T21:40:01.989 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.989 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.989 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:01.989 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086246+0000", 2024-03-27T21:40:01.990 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2064000000000001e-05 2024-03-27T21:40:01.990 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:01.990 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:01.990 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:01.990 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.990 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.990 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:01.990 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:01.990 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 21.892110675000001, 2024-03-27T21:40:01.990 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066196320000000003, 2024-03-27T21:40:01.990 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:01.990 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113 2024-03-27T21:40:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.991 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.992 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:01.992 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:01.992 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:01.992 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.992 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.992 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:01.992 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080337+0000", 2024-03-27T21:40:01.992 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:01.992 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.992 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.992 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:01.992 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080340+0000", 2024-03-27T21:40:01.993 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6690000000000002e-06 2024-03-27T21:40:01.993 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.993 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.993 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:01.993 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080344+0000", 2024-03-27T21:40:01.993 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6550000000000002e-06 2024-03-27T21:40:01.993 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.993 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.993 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:01.993 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080348+0000", 2024-03-27T21:40:01.993 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9169999999999999e-06 2024-03-27T21:40:01.993 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.994 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.994 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:01.994 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084595+0000", 2024-03-27T21:40:01.994 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042471719999999996 2024-03-27T21:40:01.994 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.994 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.994 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:01.994 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084676+0000", 2024-03-27T21:40:01.994 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1147000000000001e-05 2024-03-27T21:40:01.994 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.994 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.994 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:01.994 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084796+0000", 2024-03-27T21:40:01.994 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012032900000000001 2024-03-27T21:40:01.994 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.995 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:01.995 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086190+0000", 2024-03-27T21:40:01.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013934399999999999 2024-03-27T21:40:01.995 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.995 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:01.995 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086905+0000", 2024-03-27T21:40:01.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071536199999999999 2024-03-27T21:40:01.995 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.995 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:01.995 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086937+0000", 2024-03-27T21:40:01.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2125000000000002e-05 2024-03-27T21:40:01.996 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.996 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.996 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:01.996 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086957+0000", 2024-03-27T21:40:01.996 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0157999999999999e-05 2024-03-27T21:40:01.996 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:01.996 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:01.996 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:01.996 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.996 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.996 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:01.996 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:01.996 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 21.892097959000001, 2024-03-27T21:40:01.996 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007405661, 2024-03-27T21:40:01.996 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:01.997 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:01.997 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:01.997 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:01.997 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:01.997 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 114 2024-03-27T21:40:01.997 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.997 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:01.997 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.997 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:01.997 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:01.997 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:01.997 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.997 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.997 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:01.997 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:01.998 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:01.998 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.998 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.998 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:01.998 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:01.998 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-03-27T21:40:01.998 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.998 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.998 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:01.998 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080353+0000", 2024-03-27T21:40:01.998 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.441e-06 2024-03-27T21:40:01.999 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.999 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.999 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:01.999 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080356+0000", 2024-03-27T21:40:01.999 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0769999999999999e-06 2024-03-27T21:40:01.999 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.999 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.999 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:01.999 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080358+0000", 2024-03-27T21:40:01.999 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3369999999999998e-06 2024-03-27T21:40:01.999 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:01.999 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:01.999 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:01.999 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085118+0000", 2024-03-27T21:40:01.999 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047603949999999997 2024-03-27T21:40:02.000 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.000 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.000 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:02.000 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085210+0000", 2024-03-27T21:40:02.000 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1983000000000003e-05 2024-03-27T21:40:02.000 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.000 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.000 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:02.000 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085332+0000", 2024-03-27T21:40:02.000 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121803 2024-03-27T21:40:02.000 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.001 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.001 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:02.001 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086292+0000", 2024-03-27T21:40:02.001 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095961400000000002 2024-03-27T21:40:02.001 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.001 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.001 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:02.001 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087710+0000", 2024-03-27T21:40:02.001 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014181459999999999 2024-03-27T21:40:02.001 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.001 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.001 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:02.001 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087736+0000", 2024-03-27T21:40:02.001 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6475e-05 2024-03-27T21:40:02.001 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.002 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.002 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:02.002 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087756+0000", 2024-03-27T21:40:02.002 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9626000000000001e-05 2024-03-27T21:40:02.002 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.002 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:02.002 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.002 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.002 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.002 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:02.002 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:02.002 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 21.892075523999999, 2024-03-27T21:40:02.002 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075011720000000004, 2024-03-27T21:40:02.003 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:02.003 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.003 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:02.003 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:02.003 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:02.003 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115 2024-03-27T21:40:02.003 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.003 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:02.003 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.003 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:02.003 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:02.003 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.003 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.003 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.003 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:02.004 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:02.004 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.004 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.004 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.004 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:02.004 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:02.004 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.004 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.004 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.004 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:02.005 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080458+0000", 2024-03-27T21:40:02.005 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5228000000000005e-05 2024-03-27T21:40:02.005 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.005 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.005 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:02.005 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080465+0000", 2024-03-27T21:40:02.005 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.238e-06 2024-03-27T21:40:02.006 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.006 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.006 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.006 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080468+0000", 2024-03-27T21:40:02.006 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5109999999999999e-06 2024-03-27T21:40:02.006 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.006 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.006 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:02.006 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086326+0000", 2024-03-27T21:40:02.006 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058579169999999998 2024-03-27T21:40:02.006 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.006 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.006 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:02.006 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086382+0000", 2024-03-27T21:40:02.007 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5309999999999997e-05 2024-03-27T21:40:02.007 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.007 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:02.007 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086457+0000", 2024-03-27T21:40:02.007 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5413000000000005e-05 2024-03-27T21:40:02.007 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.007 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:02.007 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087762+0000", 2024-03-27T21:40:02.007 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013052300000000001 2024-03-27T21:40:02.007 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.007 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:02.007 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087798+0000", 2024-03-27T21:40:02.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5614999999999997e-05 2024-03-27T21:40:02.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.008 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:02.008 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087840+0000", 2024-03-27T21:40:02.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2196000000000001e-05 2024-03-27T21:40:02.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.008 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:02.008 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087874+0000", 2024-03-27T21:40:02.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3785999999999998e-05 2024-03-27T21:40:02.008 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.008 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:02.008 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.009 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.009 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:02.009 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:02.009 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 21.891975441, 2024-03-27T21:40:02.009 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075050680000000002, 2024-03-27T21:40:02.009 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:02.009 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.009 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:02.009 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:02.009 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:02.009 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 116 2024-03-27T21:40:02.009 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.009 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:02.009 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.010 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:02.010 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:02.010 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.010 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.010 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.010 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:02.010 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:02.010 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.010 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.010 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.010 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:02.010 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080472+0000", 2024-03-27T21:40:02.010 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.011 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.011 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.011 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:02.011 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080475+0000", 2024-03-27T21:40:02.011 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.712e-06 2024-03-27T21:40:02.011 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.011 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.011 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:02.011 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080478+0000", 2024-03-27T21:40:02.011 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1070000000000001e-06 2024-03-27T21:40:02.011 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.011 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.011 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.011 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080481+0000", 2024-03-27T21:40:02.012 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4269999999999998e-06 2024-03-27T21:40:02.012 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.012 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.012 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:02.012 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086992+0000", 2024-03-27T21:40:02.012 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065114099999999996 2024-03-27T21:40:02.012 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.012 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.012 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:02.012 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087064+0000", 2024-03-27T21:40:02.012 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1927000000000004e-05 2024-03-27T21:40:02.012 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.012 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.012 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:02.012 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087188+0000", 2024-03-27T21:40:02.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123585 2024-03-27T21:40:02.013 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:02.013 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087766+0000", 2024-03-27T21:40:02.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057814300000000004 2024-03-27T21:40:02.013 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:02.013 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087934+0000", 2024-03-27T21:40:02.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000167894 2024-03-27T21:40:02.013 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:02.013 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087958+0000", 2024-03-27T21:40:02.014 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4663000000000001e-05 2024-03-27T21:40:02.014 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.014 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.014 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:02.014 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087978+0000", 2024-03-27T21:40:02.014 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9636e-05 2024-03-27T21:40:02.014 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.014 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:02.014 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.014 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.014 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.014 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:02.014 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:02.014 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 21.891943318999999, 2024-03-27T21:40:02.015 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076850679999999998, 2024-03-27T21:40:02.015 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:02.015 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.015 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:02.015 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:02.015 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:02.015 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 117 2024-03-27T21:40:02.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.015 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:02.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.015 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:02.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:02.015 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:02.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:02.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.016 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:02.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:02.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.016 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:02.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080508+0000", 2024-03-27T21:40:02.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.298e-06 2024-03-27T21:40:02.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.017 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.017 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:02.017 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080515+0000", 2024-03-27T21:40:02.017 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0790000000000003e-06 2024-03-27T21:40:02.017 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.017 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.017 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.017 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080518+0000", 2024-03-27T21:40:02.017 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8509999999999999e-06 2024-03-27T21:40:02.017 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.017 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.017 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:02.017 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087444+0000", 2024-03-27T21:40:02.017 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0069259120000000002 2024-03-27T21:40:02.017 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.017 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.018 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:02.018 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087491+0000", 2024-03-27T21:40:02.018 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7043999999999999e-05 2024-03-27T21:40:02.018 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.018 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.018 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:02.018 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087564+0000", 2024-03-27T21:40:02.018 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3603999999999995e-05 2024-03-27T21:40:02.018 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.018 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.018 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:02.018 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087902+0000", 2024-03-27T21:40:02.018 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033734100000000001 2024-03-27T21:40:02.018 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.019 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.019 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:02.019 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088117+0000", 2024-03-27T21:40:02.019 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000215164 2024-03-27T21:40:02.019 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.019 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.019 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:02.019 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088160+0000", 2024-03-27T21:40:02.019 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2933999999999999e-05 2024-03-27T21:40:02.019 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.019 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.019 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:02.019 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088190+0000", 2024-03-27T21:40:02.019 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0139999999999999e-05 2024-03-27T21:40:02.020 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.020 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:02.020 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.020 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.020 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:02.020 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:02.020 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 21.475662929999999, 2024-03-27T21:40:02.020 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052465439999999997, 2024-03-27T21:40:02.020 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:02.020 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.020 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:02.020 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:02.020 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:02.020 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 133 2024-03-27T21:40:02.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.021 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:02.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:02.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:02.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:02.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:02.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:02.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:02.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.022 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.022 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.022 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:02.022 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496788+0000", 2024-03-27T21:40:02.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-03-27T21:40:02.022 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.022 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.022 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:02.022 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496793+0000", 2024-03-27T21:40:02.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4790000000000004e-06 2024-03-27T21:40:02.022 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.022 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.022 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.022 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496796+0000", 2024-03-27T21:40:02.023 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1e-06 2024-03-27T21:40:02.023 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.023 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.023 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:02.023 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500447+0000", 2024-03-27T21:40:02.023 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036504459999999999 2024-03-27T21:40:02.023 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.023 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.023 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:02.023 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500492+0000", 2024-03-27T21:40:02.023 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5765e-05 2024-03-27T21:40:02.023 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.023 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.023 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:02.024 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500562+0000", 2024-03-27T21:40:02.024 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9206000000000001e-05 2024-03-27T21:40:02.024 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.024 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.024 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:02.024 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501959+0000", 2024-03-27T21:40:02.024 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001397187 2024-03-27T21:40:02.024 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.024 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.024 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:02.024 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501984+0000", 2024-03-27T21:40:02.024 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.552e-05 2024-03-27T21:40:02.024 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.024 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.024 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:02.025 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502010+0000", 2024-03-27T21:40:02.025 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5232000000000002e-05 2024-03-27T21:40:02.025 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.025 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.025 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:02.025 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502032+0000", 2024-03-27T21:40:02.025 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2337999999999999e-05 2024-03-27T21:40:02.025 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.025 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:02.025 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.025 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.025 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.025 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:02.025 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:02.025 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 21.475604271999998, 2024-03-27T21:40:02.026 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052892119999999997, 2024-03-27T21:40:02.026 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:02.026 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.026 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:02.026 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:02.026 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:02.026 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 134 2024-03-27T21:40:02.026 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.026 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:02.026 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.026 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:02.026 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:02.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.027 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.027 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:02.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:02.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.027 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.027 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:02.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:02.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.027 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.027 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:02.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496929+0000", 2024-03-27T21:40:02.028 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5098000000000004e-05 2024-03-27T21:40:02.028 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.028 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.028 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:02.028 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496937+0000", 2024-03-27T21:40:02.028 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9559999999999997e-06 2024-03-27T21:40:02.028 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.028 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.028 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.028 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496942+0000", 2024-03-27T21:40:02.028 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6350000000000003e-06 2024-03-27T21:40:02.028 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.028 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.028 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:02.029 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500749+0000", 2024-03-27T21:40:02.029 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038068529999999998 2024-03-27T21:40:02.029 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.029 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:02.029 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500792+0000", 2024-03-27T21:40:02.029 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2823e-05 2024-03-27T21:40:02.029 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.029 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:02.029 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500864+0000", 2024-03-27T21:40:02.029 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1715e-05 2024-03-27T21:40:02.029 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.029 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:02.030 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502068+0000", 2024-03-27T21:40:02.030 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012044860000000001 2024-03-27T21:40:02.030 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.030 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:02.030 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502073+0000", 2024-03-27T21:40:02.030 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5700000000000003e-06 2024-03-27T21:40:02.030 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.030 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:02.030 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502098+0000", 2024-03-27T21:40:02.030 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5120999999999999e-05 2024-03-27T21:40:02.030 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.031 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:02.031 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502133+0000", 2024-03-27T21:40:02.031 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5306999999999999e-05 2024-03-27T21:40:02.031 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.031 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:02.031 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.031 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.031 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.031 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:02.031 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:02.031 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 21.475498965, 2024-03-27T21:40:02.031 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062529040000000001, 2024-03-27T21:40:02.031 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:02.031 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.031 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:02.032 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:02.032 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:02.032 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135 2024-03-27T21:40:02.032 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.032 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:02.032 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.032 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:02.032 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:02.032 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.032 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.032 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.032 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:02.032 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:02.032 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.032 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.033 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.033 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:02.033 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:02.033 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.033 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.033 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.033 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:02.033 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496953+0000", 2024-03-27T21:40:02.033 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0709999999999996e-06 2024-03-27T21:40:02.033 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.033 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.033 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:02.033 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496961+0000", 2024-03-27T21:40:02.033 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1300000000000001e-06 2024-03-27T21:40:02.034 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.034 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.034 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496968+0000", 2024-03-27T21:40:02.034 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0999999999999998e-06 2024-03-27T21:40:02.034 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.034 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:02.034 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501244+0000", 2024-03-27T21:40:02.034 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042756310000000002 2024-03-27T21:40:02.034 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.034 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:02.034 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501377+0000", 2024-03-27T21:40:02.034 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013348799999999999 2024-03-27T21:40:02.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.035 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:02.035 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501455+0000", 2024-03-27T21:40:02.035 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7410000000000006e-05 2024-03-27T21:40:02.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.035 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:02.035 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502138+0000", 2024-03-27T21:40:02.035 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068386599999999999 2024-03-27T21:40:02.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.036 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:02.036 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503152+0000", 2024-03-27T21:40:02.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010139680000000001 2024-03-27T21:40:02.036 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.036 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.036 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:02.036 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503180+0000", 2024-03-27T21:40:02.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7426000000000001e-05 2024-03-27T21:40:02.036 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.036 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.036 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:02.036 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503202+0000", 2024-03-27T21:40:02.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2463000000000002e-05 2024-03-27T21:40:02.036 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.036 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:02.037 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.037 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.037 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.037 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:02.037 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:02.037 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 21.475474826999999, 2024-03-27T21:40:02.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066253680000000004, 2024-03-27T21:40:02.037 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:02.037 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.037 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:02.038 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:02.038 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:02.038 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 136 2024-03-27T21:40:02.038 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.038 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:02.038 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.038 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:02.038 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:02.039 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.039 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.039 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.039 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:02.039 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:02.039 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.039 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:02.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:02.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:02.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496977+0000", 2024-03-27T21:40:02.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5049999999999999e-06 2024-03-27T21:40:02.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:02.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496984+0000", 2024-03-27T21:40:02.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1520000000000003e-06 2024-03-27T21:40:02.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496990+0000", 2024-03-27T21:40:02.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-03-27T21:40:02.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:02.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501669+0000", 2024-03-27T21:40:02.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046793859999999998 2024-03-27T21:40:02.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:02.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501714+0000", 2024-03-27T21:40:02.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5154999999999997e-05 2024-03-27T21:40:02.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.044 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:02.044 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501783+0000", 2024-03-27T21:40:02.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8454999999999994e-05 2024-03-27T21:40:02.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.045 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:02.045 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503226+0000", 2024-03-27T21:40:02.045 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001443441 2024-03-27T21:40:02.045 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.045 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.045 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:02.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503540+0000", 2024-03-27T21:40:02.046 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031347100000000001 2024-03-27T21:40:02.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:02.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503563+0000", 2024-03-27T21:40:02.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3223000000000001e-05 2024-03-27T21:40:02.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:02.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503599+0000", 2024-03-27T21:40:02.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6105000000000001e-05 2024-03-27T21:40:02.047 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.048 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:02.048 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.048 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:02.048 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:02.048 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 21.475452016999999, 2024-03-27T21:40:02.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074261320000000002, 2024-03-27T21:40:02.048 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:02.048 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.048 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:02.048 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:02.049 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:02.049 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 137 2024-03-27T21:40:02.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.049 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:02.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:02.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:02.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:02.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:02.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:02.050 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:02.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:02.050 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497077+0000", 2024-03-27T21:40:02.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1599000000000003e-05 2024-03-27T21:40:02.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:02.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497085+0000", 2024-03-27T21:40:02.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6109999999999996e-06 2024-03-27T21:40:02.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497092+0000", 2024-03-27T21:40:02.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.742e-06 2024-03-27T21:40:02.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:02.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502159+0000", 2024-03-27T21:40:02.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050666870000000003 2024-03-27T21:40:02.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:02.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502200+0000", 2024-03-27T21:40:02.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1919999999999998e-05 2024-03-27T21:40:02.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:02.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502280+0000", 2024-03-27T21:40:02.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9417000000000002e-05 2024-03-27T21:40:02.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:02.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503605+0000", 2024-03-27T21:40:02.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001325641 2024-03-27T21:40:02.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:02.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504361+0000", 2024-03-27T21:40:02.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075555399999999997 2024-03-27T21:40:02.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:02.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504386+0000", 2024-03-27T21:40:02.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5369999999999999e-05 2024-03-27T21:40:02.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.054 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:02.054 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504422+0000", 2024-03-27T21:40:02.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6019000000000002e-05 2024-03-27T21:40:02.054 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.054 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:02.054 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.054 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.054 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:02.054 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:02.054 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 21.475348220000001, 2024-03-27T21:40:02.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074059750000000004, 2024-03-27T21:40:02.055 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:02.055 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.055 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:02.055 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:02.055 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:02.055 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-03-27T21:40:02.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.055 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:02.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:02.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:02.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:02.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:02.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:02.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:02.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:02.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497103+0000", 2024-03-27T21:40:02.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.856e-06 2024-03-27T21:40:02.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:02.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497121+0000", 2024-03-27T21:40:02.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7190999999999999e-05 2024-03-27T21:40:02.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.057 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497126+0000", 2024-03-27T21:40:02.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3770000000000004e-06 2024-03-27T21:40:02.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:02.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502833+0000", 2024-03-27T21:40:02.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057073510000000003 2024-03-27T21:40:02.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:02.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502881+0000", 2024-03-27T21:40:02.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7961999999999999e-05 2024-03-27T21:40:02.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:02.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502949+0000", 2024-03-27T21:40:02.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7957e-05 2024-03-27T21:40:02.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:02.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504430+0000", 2024-03-27T21:40:02.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001480763 2024-03-27T21:40:02.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:02.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504459+0000", 2024-03-27T21:40:02.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8983999999999999e-05 2024-03-27T21:40:02.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:02.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504486+0000", 2024-03-27T21:40:02.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6883999999999999e-05 2024-03-27T21:40:02.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:02.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504506+0000", 2024-03-27T21:40:02.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0214e-05 2024-03-27T21:40:02.060 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.061 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:02.061 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.061 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:02.061 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:02.061 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 21.475316385999999, 2024-03-27T21:40:02.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074754859999999999, 2024-03-27T21:40:02.061 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:02.062 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.062 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:02.062 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:02.062 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:02.062 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-03-27T21:40:02.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.062 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:02.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:02.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:02.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:02.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:02.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:02.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:02.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:02.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497137+0000", 2024-03-27T21:40:02.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0730000000000004e-06 2024-03-27T21:40:02.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:02.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497143+0000", 2024-03-27T21:40:02.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3759999999999996e-06 2024-03-27T21:40:02.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497150+0000", 2024-03-27T21:40:02.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.002e-06 2024-03-27T21:40:02.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:02.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503249+0000", 2024-03-27T21:40:02.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060989750000000004 2024-03-27T21:40:02.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:02.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503291+0000", 2024-03-27T21:40:02.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1563999999999997e-05 2024-03-27T21:40:02.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:02.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503376+0000", 2024-03-27T21:40:02.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5752000000000002e-05 2024-03-27T21:40:02.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:02.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504545+0000", 2024-03-27T21:40:02.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169172 2024-03-27T21:40:02.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:02.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504550+0000", 2024-03-27T21:40:02.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7419999999999997e-06 2024-03-27T21:40:02.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:02.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504574+0000", 2024-03-27T21:40:02.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4124000000000001e-05 2024-03-27T21:40:02.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:02.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504607+0000", 2024-03-27T21:40:02.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3118999999999999e-05 2024-03-27T21:40:02.067 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.067 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:02.067 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.067 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:02.067 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:02.067 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 21.475292719999999, 2024-03-27T21:40:02.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078150589999999992, 2024-03-27T21:40:02.067 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:02.067 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.067 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:02.067 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:02.067 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:02.068 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-03-27T21:40:02.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.069 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:02.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:02.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:02.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:02.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:02.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:02.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497155+0000", 2024-03-27T21:40:02.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:02.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497230+0000", 2024-03-27T21:40:02.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4789000000000005e-05 2024-03-27T21:40:02.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:02.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497241+0000", 2024-03-27T21:40:02.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1442e-05 2024-03-27T21:40:02.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497255+0000", 2024-03-27T21:40:02.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3087e-05 2024-03-27T21:40:02.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:02.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503630+0000", 2024-03-27T21:40:02.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063755269999999998 2024-03-27T21:40:02.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:02.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503673+0000", 2024-03-27T21:40:02.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2616e-05 2024-03-27T21:40:02.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:02.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503738+0000", 2024-03-27T21:40:02.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5114e-05 2024-03-27T21:40:02.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:02.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504907+0000", 2024-03-27T21:40:02.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169432 2024-03-27T21:40:02.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:02.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504926+0000", 2024-03-27T21:40:02.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9086999999999999e-05 2024-03-27T21:40:02.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.074 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:02.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504950+0000", 2024-03-27T21:40:02.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3598999999999999e-05 2024-03-27T21:40:02.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.074 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:02.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504971+0000", 2024-03-27T21:40:02.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0772999999999999e-05 2024-03-27T21:40:02.074 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.074 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:02.074 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.075 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:02.075 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:02.075 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 21.475185679999999, 2024-03-27T21:40:02.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082962590000000003, 2024-03-27T21:40:02.075 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:02.075 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.075 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:02.075 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:02.075 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:02.075 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-03-27T21:40:02.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.075 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:02.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:02.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:02.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:02.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:02.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:02.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497262+0000", 2024-03-27T21:40:02.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:02.077 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497267+0000", 2024-03-27T21:40:02.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.955e-06 2024-03-27T21:40:02.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:02.077 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497275+0000", 2024-03-27T21:40:02.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.824e-06 2024-03-27T21:40:02.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.077 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497280+0000", 2024-03-27T21:40:02.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8230000000000002e-06 2024-03-27T21:40:02.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:02.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504628+0000", 2024-03-27T21:40:02.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007348561 2024-03-27T21:40:02.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:02.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504673+0000", 2024-03-27T21:40:02.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4313000000000001e-05 2024-03-27T21:40:02.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:02.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504745+0000", 2024-03-27T21:40:02.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2303000000000002e-05 2024-03-27T21:40:02.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:02.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505358+0000", 2024-03-27T21:40:02.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061281300000000003 2024-03-27T21:40:02.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:02.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505503+0000", 2024-03-27T21:40:02.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000145134 2024-03-27T21:40:02.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.080 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:02.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505526+0000", 2024-03-27T21:40:02.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3312e-05 2024-03-27T21:40:02.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:02.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505559+0000", 2024-03-27T21:40:02.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2821999999999998e-05 2024-03-27T21:40:02.081 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.081 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:02.081 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.081 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:02.082 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:02.082 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 20.865963441000002, 2024-03-27T21:40:02.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053753339999999998, 2024-03-27T21:40:02.082 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:02.082 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.082 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:02.082 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:02.082 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:02.082 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159 2024-03-27T21:40:02.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.083 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:02.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:02.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:02.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:02.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:02.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:02.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:02.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:02.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:02.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106488+0000", 2024-03-27T21:40:02.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9830000000000001e-06 2024-03-27T21:40:02.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:02.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106491+0000", 2024-03-27T21:40:02.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3459999999999998e-06 2024-03-27T21:40:02.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106493+0000", 2024-03-27T21:40:02.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.509e-06 2024-03-27T21:40:02.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:02.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110636+0000", 2024-03-27T21:40:02.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041427260000000002 2024-03-27T21:40:02.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:02.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110683+0000", 2024-03-27T21:40:02.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6557000000000002e-05 2024-03-27T21:40:02.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:02.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110755+0000", 2024-03-27T21:40:02.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2119000000000005e-05 2024-03-27T21:40:02.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:02.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111415+0000", 2024-03-27T21:40:02.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066065199999999996 2024-03-27T21:40:02.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:02.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111793+0000", 2024-03-27T21:40:02.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037732100000000001 2024-03-27T21:40:02.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:02.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111829+0000", 2024-03-27T21:40:02.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418999999999998e-05 2024-03-27T21:40:02.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:02.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:02.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:02.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111860+0000", 2024-03-27T21:40:02.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1003000000000001e-05 2024-03-27T21:40:02.087 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.087 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:02.087 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.087 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:02.087 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:02.087 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:02.089 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:02.103+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:02.089 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:40:02.190 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:02.204+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:02.199 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:02.199 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:02.199 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:02.199 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:02.209 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:40:02.291 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:02.305+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:02.318 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:02.318 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:02.318 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:02.318 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:02.318 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:02.327 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:40:02.392 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:02.406+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:02.440 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:02.440 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:02.440 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:02.440 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-03-27T21:40:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:20 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2024-03-27T21:40:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 22.564623660999999, 2024-03-27T21:40:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015345630000000001, 2024-03-27T21:40:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888562+0000", 2024-03-27T21:40:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888735+0000", 2024-03-27T21:40:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017254799999999999 2024-03-27T21:40:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888747+0000", 2024-03-27T21:40:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2469999999999999e-05 2024-03-27T21:40:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888761+0000", 2024-03-27T21:40:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3254e-05 2024-03-27T21:40:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888804+0000", 2024-03-27T21:40:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3129e-05 2024-03-27T21:40:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888820+0000", 2024-03-27T21:40:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6426000000000002e-05 2024-03-27T21:40:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890060+0000", 2024-03-27T21:40:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012398050000000001 2024-03-27T21:40:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890098+0000", 2024-03-27T21:40:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7795e-05 2024-03-27T21:40:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2024-03-27T21:40:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 22.563799261, 2024-03-27T21:40:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001069555, 2024-03-27T21:40:02.446 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:02.446 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.446 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:02.446 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.446 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:02.446 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:02.446 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.446 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.446 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.446 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:02.446 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:02.446 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.446 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.447 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.447 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:02.447 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889387+0000", 2024-03-27T21:40:02.447 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:02.447 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.447 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.447 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:02.447 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889455+0000", 2024-03-27T21:40:02.447 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7930999999999997e-05 2024-03-27T21:40:02.447 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.447 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.447 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:02.447 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889470+0000", 2024-03-27T21:40:02.447 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5554e-05 2024-03-27T21:40:02.447 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.448 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.448 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.448 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889476+0000", 2024-03-27T21:40:02.448 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0619999999999997e-06 2024-03-27T21:40:02.448 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.448 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.448 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:02.448 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889513+0000", 2024-03-27T21:40:02.448 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6606999999999997e-05 2024-03-27T21:40:02.448 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.448 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.448 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:02.448 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889528+0000", 2024-03-27T21:40:02.448 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4982e-05 2024-03-27T21:40:02.449 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.449 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.449 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:02.449 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890421+0000", 2024-03-27T21:40:02.449 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00089293700000000001 2024-03-27T21:40:02.449 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.449 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.449 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:02.449 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890457+0000", 2024-03-27T21:40:02.449 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6260000000000002e-05 2024-03-27T21:40:02.449 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.449 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:02.449 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.450 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.450 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.450 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:38 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'24, mlcod=17'22)", 2024-03-27T21:40:02.450 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:02.450 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 22.546563294999999, 2024-03-27T21:40:02.450 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016182080000000001, 2024-03-27T21:40:02.450 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:02.450 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.450 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:02.450 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.450 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:02.450 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:02.450 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.451 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.451 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.451 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:02.451 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:02.451 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.451 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.451 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.451 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:02.451 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906623+0000", 2024-03-27T21:40:02.451 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:02.451 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.451 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.451 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:02.451 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906687+0000", 2024-03-27T21:40:02.451 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3706000000000003e-05 2024-03-27T21:40:02.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.452 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.452 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:02.452 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906698+0000", 2024-03-27T21:40:02.452 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1752e-05 2024-03-27T21:40:02.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.452 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.452 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.452 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906704+0000", 2024-03-27T21:40:02.452 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3940000000000004e-06 2024-03-27T21:40:02.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.452 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.452 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:02.452 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906741+0000", 2024-03-27T21:40:02.452 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6944999999999999e-05 2024-03-27T21:40:02.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.453 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.453 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:02.453 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906760+0000", 2024-03-27T21:40:02.453 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9293e-05 2024-03-27T21:40:02.453 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.453 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.453 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:02.453 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908211+0000", 2024-03-27T21:40:02.453 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014511350000000001 2024-03-27T21:40:02.453 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.453 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.453 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:02.453 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908242+0000", 2024-03-27T21:40:02.453 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0712e-05 2024-03-27T21:40:02.454 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.454 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:02.454 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.454 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.454 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.454 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:39 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'25, mlcod=17'22)", 2024-03-27T21:40:02.454 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:02.454 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 22.54571838, 2024-03-27T21:40:02.454 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012060199999999999, 2024-03-27T21:40:02.454 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:02.454 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.454 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:02.454 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.455 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:02.455 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:02.455 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.455 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:02.455 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:02.455 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.455 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:02.455 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:02.455 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:02.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.456 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:02.456 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907533+0000", 2024-03-27T21:40:02.456 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5675999999999999e-05 2024-03-27T21:40:02.456 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.456 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.456 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:02.456 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907545+0000", 2024-03-27T21:40:02.456 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.115e-05 2024-03-27T21:40:02.456 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.456 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.456 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.456 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907550+0000", 2024-03-27T21:40:02.456 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8320000000000002e-06 2024-03-27T21:40:02.456 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.457 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.457 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:02.457 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907592+0000", 2024-03-27T21:40:02.457 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1309e-05 2024-03-27T21:40:02.457 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.457 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.457 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:02.457 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907611+0000", 2024-03-27T21:40:02.457 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9009e-05 2024-03-27T21:40:02.457 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.457 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.457 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:02.457 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908638+0000", 2024-03-27T21:40:02.457 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001027533 2024-03-27T21:40:02.458 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.458 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.458 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:02.458 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908675+0000", 2024-03-27T21:40:02.458 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6329999999999999e-05 2024-03-27T21:40:02.458 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.458 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:02.458 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.458 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.458 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.458 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:57 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'35, mlcod=17'33)", 2024-03-27T21:40:02.458 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:02.458 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 22.531633948, 2024-03-27T21:40:02.459 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001346578, 2024-03-27T21:40:02.459 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:02.459 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.459 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:02.459 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.459 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:02.459 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:02.459 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.459 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.459 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.459 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:02.459 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:02.459 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.460 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.460 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:02.460 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921552+0000", 2024-03-27T21:40:02.460 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.460 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.460 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:02.460 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921615+0000", 2024-03-27T21:40:02.460 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2787999999999996e-05 2024-03-27T21:40:02.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.460 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.460 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:02.460 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921626+0000", 2024-03-27T21:40:02.460 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0888e-05 2024-03-27T21:40:02.461 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.461 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.461 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.461 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921635+0000", 2024-03-27T21:40:02.461 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4639999999999995e-06 2024-03-27T21:40:02.461 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.461 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.461 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:02.461 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921673+0000", 2024-03-27T21:40:02.461 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7891e-05 2024-03-27T21:40:02.461 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.461 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.461 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:02.461 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921694+0000", 2024-03-27T21:40:02.462 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0426000000000001e-05 2024-03-27T21:40:02.462 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.462 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.462 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:02.462 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922865+0000", 2024-03-27T21:40:02.462 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011709509999999999 2024-03-27T21:40:02.462 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.462 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.462 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:02.462 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922900+0000", 2024-03-27T21:40:02.462 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4819e-05 2024-03-27T21:40:02.462 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.462 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:02.463 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.463 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.463 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.463 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:71 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'45, mlcod=17'43)", 2024-03-27T21:40:02.463 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:02.463 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 22.516335236, 2024-03-27T21:40:02.463 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001460907, 2024-03-27T21:40:02.463 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:02.463 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.463 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:02.463 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.463 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:02.463 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:02.463 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.464 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.464 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.464 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:02.464 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:02.464 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.464 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.464 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.464 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:02.464 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936851+0000", 2024-03-27T21:40:02.464 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:02.464 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.464 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.465 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:02.465 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936903+0000", 2024-03-27T21:40:02.465 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2302999999999997e-05 2024-03-27T21:40:02.465 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.465 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.465 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:02.465 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936915+0000", 2024-03-27T21:40:02.465 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1967000000000001e-05 2024-03-27T21:40:02.465 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.465 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.465 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.465 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936923+0000", 2024-03-27T21:40:02.465 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9370000000000004e-06 2024-03-27T21:40:02.465 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.466 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.466 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:02.466 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936970+0000", 2024-03-27T21:40:02.466 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6987000000000003e-05 2024-03-27T21:40:02.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.466 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.466 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:02.466 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937002+0000", 2024-03-27T21:40:02.466 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2410999999999999e-05 2024-03-27T21:40:02.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.466 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.466 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:02.466 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938279+0000", 2024-03-27T21:40:02.466 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001276552 2024-03-27T21:40:02.467 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.467 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.467 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:02.467 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938313+0000", 2024-03-27T21:40:02.467 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3526999999999999e-05 2024-03-27T21:40:02.467 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.467 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:02.467 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.467 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.467 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.467 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:72 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'46, mlcod=17'43)", 2024-03-27T21:40:02.467 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:02.467 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 22.515697673999998, 2024-03-27T21:40:02.467 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014239490000000001, 2024-03-27T21:40:02.467 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:02.468 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.468 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:02.468 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.468 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:02.468 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:02.468 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.468 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.468 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.468 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:02.468 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:02.468 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.468 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.468 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.468 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:02.469 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937488+0000", 2024-03-27T21:40:02.469 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:02.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.469 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.469 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:02.469 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937547+0000", 2024-03-27T21:40:02.469 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8196999999999999e-05 2024-03-27T21:40:02.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.469 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.469 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:02.469 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937559+0000", 2024-03-27T21:40:02.469 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1928e-05 2024-03-27T21:40:02.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.469 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.470 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.470 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937564+0000", 2024-03-27T21:40:02.470 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4759999999999997e-06 2024-03-27T21:40:02.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.470 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.470 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:02.470 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937621+0000", 2024-03-27T21:40:02.470 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6712000000000002e-05 2024-03-27T21:40:02.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.470 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.470 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:02.470 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937640+0000", 2024-03-27T21:40:02.470 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9517e-05 2024-03-27T21:40:02.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.471 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:02.471 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938883+0000", 2024-03-27T21:40:02.471 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001242567 2024-03-27T21:40:02.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.471 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:02.471 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938913+0000", 2024-03-27T21:40:02.471 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0287e-05 2024-03-27T21:40:02.471 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.471 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:02.471 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.471 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:73 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'47, mlcod=17'43)", 2024-03-27T21:40:02.472 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:02.472 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 22.515021579999999, 2024-03-27T21:40:02.472 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010746239999999999, 2024-03-27T21:40:02.472 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:02.472 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.472 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:02.472 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.472 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:02.472 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:02.472 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.472 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.472 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.472 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:02.472 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:02.473 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.473 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.473 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.473 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:02.473 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:02.473 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:02.473 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.473 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.473 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:02.473 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938217+0000", 2024-03-27T21:40:02.473 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2743e-05 2024-03-27T21:40:02.473 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.473 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.473 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:02.474 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938229+0000", 2024-03-27T21:40:02.474 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1956999999999999e-05 2024-03-27T21:40:02.474 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.474 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.474 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.474 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938235+0000", 2024-03-27T21:40:02.474 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4589999999999997e-06 2024-03-27T21:40:02.474 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.474 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.474 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:02.474 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938341+0000", 2024-03-27T21:40:02.474 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000106054 2024-03-27T21:40:02.474 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.474 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.475 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:02.475 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938369+0000", 2024-03-27T21:40:02.475 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8291000000000001e-05 2024-03-27T21:40:02.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.475 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.475 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:02.475 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939209+0000", 2024-03-27T21:40:02.475 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084045400000000003 2024-03-27T21:40:02.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.475 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.475 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:02.475 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939240+0000", 2024-03-27T21:40:02.475 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0429999999999998e-05 2024-03-27T21:40:02.475 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.476 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:02.476 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.476 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.476 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.476 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:75 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'49, mlcod=17'47)", 2024-03-27T21:40:02.476 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:02.476 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 22.508583933000001, 2024-03-27T21:40:02.476 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001017001, 2024-03-27T21:40:02.476 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:02.476 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.476 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:02.476 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.476 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:02.476 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:02.477 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.477 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:02.477 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:02.477 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.477 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:02.477 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944601+0000", 2024-03-27T21:40:02.477 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999986 2024-03-27T21:40:02.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.477 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:02.478 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944614+0000", 2024-03-27T21:40:02.478 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2496e-05 2024-03-27T21:40:02.478 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.478 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.478 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:02.478 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944640+0000", 2024-03-27T21:40:02.478 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5865999999999999e-05 2024-03-27T21:40:02.478 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.478 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.478 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.478 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944651+0000", 2024-03-27T21:40:02.478 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1797000000000001e-05 2024-03-27T21:40:02.478 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.478 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.479 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:02.479 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944762+0000", 2024-03-27T21:40:02.479 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110412 2024-03-27T21:40:02.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.479 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.479 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:02.479 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944820+0000", 2024-03-27T21:40:02.479 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7649999999999999e-05 2024-03-27T21:40:02.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.479 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.479 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:02.479 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945585+0000", 2024-03-27T21:40:02.479 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076562600000000005 2024-03-27T21:40:02.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.479 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.480 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:02.481 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945620+0000", 2024-03-27T21:40:02.481 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4764000000000001e-05 2024-03-27T21:40:02.481 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.481 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:02.481 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.481 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.481 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.481 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:86 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'50)", 2024-03-27T21:40:02.482 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:02.482 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 22.500822446000001, 2024-03-27T21:40:02.482 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011392380000000001, 2024-03-27T21:40:02.482 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:02.482 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.482 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:02.482 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.482 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:02.482 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:02.482 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.482 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.482 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.482 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:02.482 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:02.483 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.483 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.483 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.483 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:02.483 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:02.483 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.483 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.483 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.483 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:02.483 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952373+0000", 2024-03-27T21:40:02.483 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9220000000000003e-06 2024-03-27T21:40:02.483 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.483 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.483 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:02.483 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952381+0000", 2024-03-27T21:40:02.484 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3070000000000003e-06 2024-03-27T21:40:02.484 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.484 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.484 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.484 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952385+0000", 2024-03-27T21:40:02.484 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.121e-06 2024-03-27T21:40:02.484 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.484 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.484 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:02.484 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952419+0000", 2024-03-27T21:40:02.484 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4082999999999999e-05 2024-03-27T21:40:02.484 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.484 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.484 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:02.484 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952440+0000", 2024-03-27T21:40:02.485 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0837999999999999e-05 2024-03-27T21:40:02.485 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.485 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.485 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:02.485 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953472+0000", 2024-03-27T21:40:02.485 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001032119 2024-03-27T21:40:02.485 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.485 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.485 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:02.485 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953504+0000", 2024-03-27T21:40:02.485 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1250000000000001e-05 2024-03-27T21:40:02.485 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.485 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:02.485 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.486 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.486 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.486 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:87 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-27T21:40:02.486 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:02.486 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 22.500455189, 2024-03-27T21:40:02.486 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00135519, 2024-03-27T21:40:02.486 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:02.486 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.486 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:02.486 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.486 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:02.486 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:02.486 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.486 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.487 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.487 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:02.487 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:02.487 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.487 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.487 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.487 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:02.487 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952731+0000", 2024-03-27T21:40:02.487 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.487 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.487 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.487 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:02.487 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952781+0000", 2024-03-27T21:40:02.487 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0161000000000003e-05 2024-03-27T21:40:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952794+0000", 2024-03-27T21:40:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2853e-05 2024-03-27T21:40:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952799+0000", 2024-03-27T21:40:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5560000000000001e-06 2024-03-27T21:40:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:02.488 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952832+0000", 2024-03-27T21:40:02.489 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3834999999999997e-05 2024-03-27T21:40:02.489 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.489 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.489 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:02.489 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952851+0000", 2024-03-27T21:40:02.489 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8289999999999999e-05 2024-03-27T21:40:02.489 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.489 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.489 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:02.489 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954055+0000", 2024-03-27T21:40:02.489 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012037949999999999 2024-03-27T21:40:02.489 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.489 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.490 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:02.490 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954087+0000", 2024-03-27T21:40:02.490 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2292999999999997e-05 2024-03-27T21:40:02.490 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.490 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:02.490 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.490 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.490 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.490 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:89 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'59, mlcod=17'57)", 2024-03-27T21:40:02.490 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:02.490 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 22.498180968, 2024-03-27T21:40:02.490 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00138893, 2024-03-27T21:40:02.490 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:02.491 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.491 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:02.491 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.491 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:02.491 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:02.491 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.491 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.491 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.491 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:02.491 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:02.491 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.491 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.491 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.491 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:02.491 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955005+0000", 2024-03-27T21:40:02.492 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.492 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:02.492 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955071+0000", 2024-03-27T21:40:02.492 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5290999999999995e-05 2024-03-27T21:40:02.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.492 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:02.492 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955081+0000", 2024-03-27T21:40:02.492 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0572e-05 2024-03-27T21:40:02.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.493 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.493 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955086+0000", 2024-03-27T21:40:02.493 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8380000000000001e-06 2024-03-27T21:40:02.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.493 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.493 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:02.493 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955122+0000", 2024-03-27T21:40:02.493 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5698000000000003e-05 2024-03-27T21:40:02.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.493 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.493 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:02.493 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955140+0000", 2024-03-27T21:40:02.493 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7900000000000001e-05 2024-03-27T21:40:02.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.494 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.494 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:02.494 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956366+0000", 2024-03-27T21:40:02.494 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001226471 2024-03-27T21:40:02.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.494 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.494 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:02.494 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956395+0000", 2024-03-27T21:40:02.494 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8782e-05 2024-03-27T21:40:02.494 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.494 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:02.494 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.494 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.495 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:90 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'60, mlcod=17'57)", 2024-03-27T21:40:02.495 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:02.495 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 22.497496390999999, 2024-03-27T21:40:02.495 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010778859999999999, 2024-03-27T21:40:02.495 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:02.495 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.495 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:02.495 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.495 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:02.495 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:02.495 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.495 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.495 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.495 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:02.496 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:02.496 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.496 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.496 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.496 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:02.496 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:02.496 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.496 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.496 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.496 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:02.496 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955747+0000", 2024-03-27T21:40:02.496 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7130999999999999e-05 2024-03-27T21:40:02.496 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.496 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.497 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:02.497 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955756+0000", 2024-03-27T21:40:02.497 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2750000000000001e-06 2024-03-27T21:40:02.497 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.497 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.497 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955761+0000", 2024-03-27T21:40:02.497 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3900000000000003e-06 2024-03-27T21:40:02.497 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.497 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:02.497 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955797+0000", 2024-03-27T21:40:02.497 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6650000000000003e-05 2024-03-27T21:40:02.497 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.498 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.498 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:02.498 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955816+0000", 2024-03-27T21:40:02.498 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8195000000000001e-05 2024-03-27T21:40:02.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.498 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.498 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:02.498 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956739+0000", 2024-03-27T21:40:02.498 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00092369700000000002 2024-03-27T21:40:02.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.498 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.498 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:02.499 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956768+0000", 2024-03-27T21:40:02.499 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9048e-05 2024-03-27T21:40:02.499 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.499 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:02.499 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.499 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.499 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:118 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'80, mlcod=17'78)", 2024-03-27T21:40:02.499 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:02.499 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 22.364505567999998, 2024-03-27T21:40:02.499 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014200669999999999, 2024-03-27T21:40:02.499 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:02.500 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.500 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:02.500 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.500 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:02.500 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:02.500 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.500 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.500 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.500 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:02.500 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:02.500 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.500 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.500 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.500 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:02.500 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:02.500 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.501 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.501 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.501 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:02.501 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088756+0000", 2024-03-27T21:40:02.501 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4902999999999998e-05 2024-03-27T21:40:02.501 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.501 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.501 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:02.501 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088764+0000", 2024-03-27T21:40:02.501 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8169999999999999e-06 2024-03-27T21:40:02.501 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.501 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.501 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.501 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088767+0000", 2024-03-27T21:40:02.502 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8310000000000003e-06 2024-03-27T21:40:02.502 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.502 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.502 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:02.502 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088798+0000", 2024-03-27T21:40:02.502 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0800999999999999e-05 2024-03-27T21:40:02.502 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.502 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.502 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:02.502 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088817+0000", 2024-03-27T21:40:02.502 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8467000000000001e-05 2024-03-27T21:40:02.502 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.502 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.502 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:02.503 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090074+0000", 2024-03-27T21:40:02.503 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012574089999999999 2024-03-27T21:40:02.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.503 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.503 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:02.503 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090101+0000", 2024-03-27T21:40:02.503 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7223e-05 2024-03-27T21:40:02.503 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.503 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:02.503 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.503 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.503 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:119 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'81, mlcod=17'78)", 2024-03-27T21:40:02.503 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:02.504 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 22.363770772999999, 2024-03-27T21:40:02.504 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014990870000000001, 2024-03-27T21:40:02.504 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:02.504 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.504 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:02.504 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.504 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:02.504 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:02.504 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.504 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.504 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.504 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:02.504 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:02.504 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.504 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.505 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:02.505 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:02.505 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.505 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:02.505 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089491+0000", 2024-03-27T21:40:02.505 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4805e-05 2024-03-27T21:40:02.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.505 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:02.505 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089498+0000", 2024-03-27T21:40:02.505 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0650000000000001e-06 2024-03-27T21:40:02.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.506 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.506 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.506 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089501+0000", 2024-03-27T21:40:02.506 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.54e-06 2024-03-27T21:40:02.506 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.506 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.506 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:02.506 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089533+0000", 2024-03-27T21:40:02.506 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1838999999999998e-05 2024-03-27T21:40:02.506 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.506 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.506 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:02.506 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089550+0000", 2024-03-27T21:40:02.506 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7465999999999998e-05 2024-03-27T21:40:02.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.507 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.507 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:02.507 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090890+0000", 2024-03-27T21:40:02.507 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013397979999999999 2024-03-27T21:40:02.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.507 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.507 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:02.507 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090915+0000", 2024-03-27T21:40:02.507 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4905e-05 2024-03-27T21:40:02.507 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.507 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:02.507 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.508 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.508 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:120 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'82, mlcod=17'78)", 2024-03-27T21:40:02.508 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:02.508 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 22.362970893, 2024-03-27T21:40:02.508 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001527093, 2024-03-27T21:40:02.508 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:02.508 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.508 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:02.508 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.508 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:02.508 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:02.508 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.508 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.508 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.509 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:02.509 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:02.509 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.509 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:02.509 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:02.509 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.509 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:02.509 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090267+0000", 2024-03-27T21:40:02.509 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1477000000000002e-05 2024-03-27T21:40:02.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.510 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:02.510 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090273+0000", 2024-03-27T21:40:02.510 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9519999999999999e-06 2024-03-27T21:40:02.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.510 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.510 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090276+0000", 2024-03-27T21:40:02.510 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.219e-06 2024-03-27T21:40:02.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.510 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:02.510 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090304+0000", 2024-03-27T21:40:02.510 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8073999999999999e-05 2024-03-27T21:40:02.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.511 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:02.511 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090316+0000", 2024-03-27T21:40:02.511 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2088e-05 2024-03-27T21:40:02.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.511 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:02.511 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091717+0000", 2024-03-27T21:40:02.511 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014003889999999999 2024-03-27T21:40:02.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.511 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:02.511 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091743+0000", 2024-03-27T21:40:02.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6234e-05 2024-03-27T21:40:02.512 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.512 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:02.512 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.512 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.512 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:121 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'83, mlcod=17'78)", 2024-03-27T21:40:02.512 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:02.512 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 22.362160528, 2024-03-27T21:40:02.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010226899999999999, 2024-03-27T21:40:02.512 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:02.512 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.512 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:02.512 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.513 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:02.513 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:02.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.513 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:02.513 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:02.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.513 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:02.513 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:02.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:02.514 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091142+0000", 2024-03-27T21:40:02.514 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011608300000000001 2024-03-27T21:40:02.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.514 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:02.514 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091151+0000", 2024-03-27T21:40:02.514 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2350000000000005e-06 2024-03-27T21:40:02.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.514 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.514 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091156+0000", 2024-03-27T21:40:02.514 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5419999999999999e-06 2024-03-27T21:40:02.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.515 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.515 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:02.515 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091191+0000", 2024-03-27T21:40:02.515 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5453000000000001e-05 2024-03-27T21:40:02.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.515 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.515 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:02.515 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091208+0000", 2024-03-27T21:40:02.515 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7119999999999999e-05 2024-03-27T21:40:02.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.515 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.515 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:02.515 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092024+0000", 2024-03-27T21:40:02.515 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000815539 2024-03-27T21:40:02.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.516 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:02.516 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092049+0000", 2024-03-27T21:40:02.516 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5171e-05 2024-03-27T21:40:02.516 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.516 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:02.516 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.516 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:142 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'100, mlcod=18'98)", 2024-03-27T21:40:02.516 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:02.516 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 21.947153574000001, 2024-03-27T21:40:02.516 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001198778, 2024-03-27T21:40:02.517 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:02.517 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.517 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:02.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.517 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:02.517 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:02.517 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.517 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:02.517 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:02.517 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.518 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:02.518 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:02.518 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.518 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:02.518 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506101+0000", 2024-03-27T21:40:02.518 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8202999999999997e-05 2024-03-27T21:40:02.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.518 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:02.518 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506109+0000", 2024-03-27T21:40:02.518 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3059999999999998e-06 2024-03-27T21:40:02.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.519 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.519 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506113+0000", 2024-03-27T21:40:02.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0119999999999999e-06 2024-03-27T21:40:02.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.519 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:02.519 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506137+0000", 2024-03-27T21:40:02.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4043999999999999e-05 2024-03-27T21:40:02.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.519 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:02.519 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506148+0000", 2024-03-27T21:40:02.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0446e-05 2024-03-27T21:40:02.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.520 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.520 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:02.520 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507213+0000", 2024-03-27T21:40:02.520 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001065215 2024-03-27T21:40:02.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.520 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.520 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:02.520 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507232+0000", 2024-03-27T21:40:02.520 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9038e-05 2024-03-27T21:40:02.520 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.520 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:02.520 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.521 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:143 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'101, mlcod=18'98)", 2024-03-27T21:40:02.521 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:02.521 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 21.946540955, 2024-03-27T21:40:02.521 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001357488, 2024-03-27T21:40:02.521 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:02.521 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.521 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:02.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.521 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:02.521 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:02.521 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.521 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:02.522 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:02.522 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.522 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.522 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.522 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:02.522 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:02.522 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.522 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.522 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:02.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506723+0000", 2024-03-27T21:40:02.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7404000000000006e-05 2024-03-27T21:40:02.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:02.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506730+0000", 2024-03-27T21:40:02.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5749999999999997e-06 2024-03-27T21:40:02.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506733+0000", 2024-03-27T21:40:02.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7979999999999999e-06 2024-03-27T21:40:02.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:02.524 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506758+0000", 2024-03-27T21:40:02.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4179e-05 2024-03-27T21:40:02.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:02.524 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506768+0000", 2024-03-27T21:40:02.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0740000000000001e-05 2024-03-27T21:40:02.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:02.524 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507978+0000", 2024-03-27T21:40:02.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012093900000000001 2024-03-27T21:40:02.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:02.525 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508003+0000", 2024-03-27T21:40:02.525 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5667e-05 2024-03-27T21:40:02.525 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.525 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:02.525 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.525 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.525 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:144 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'102, mlcod=18'98)", 2024-03-27T21:40:02.525 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:02.525 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 21.945882057999999, 2024-03-27T21:40:02.525 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013453390000000001, 2024-03-27T21:40:02.525 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:02.525 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:02.525 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:02.525 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.525 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:02.526 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:02.526 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.526 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:02.526 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:02.526 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:02.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.526 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:02.526 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507304+0000", 2024-03-27T21:40:02.526 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:02.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.527 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:02.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507397+0000", 2024-03-27T21:40:02.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2995e-05 2024-03-27T21:40:02.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.527 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:02.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507403+0000", 2024-03-27T21:40:02.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8640000000000001e-06 2024-03-27T21:40:02.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.527 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:02.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507407+0000", 2024-03-27T21:40:02.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5710000000000002e-06 2024-03-27T21:40:02.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.528 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.528 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:02.528 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507428+0000", 2024-03-27T21:40:02.528 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1192e-05 2024-03-27T21:40:02.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.528 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.528 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:02.528 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507438+0000", 2024-03-27T21:40:02.528 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8309999999999996e-06 2024-03-27T21:40:02.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.528 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.528 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:02.528 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508624+0000", 2024-03-27T21:40:02.528 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011861210000000001 2024-03-27T21:40:02.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:02.529 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:02.529 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:02.529 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508650+0000", 2024-03-27T21:40:02.529 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6118e-05 2024-03-27T21:40:02.529 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.529 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:02.529 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.529 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:02.529 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:02.529 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:02.529 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:40:02.531 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:40:02.532 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:02.545+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:02.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.643+0000 7f61c2ffd640 1 -- 172.21.15.60:0/553034812 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v26) v1 ==== 101+0+0 (secure 0 0 0) 0x7f61bc07a120 con 0x7f61cc071a50 2024-03-27T21:40:02.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.643+0000 7f61c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:40:02.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.643+0000 7f61c2ffd640 10 monclient: _finish_command 2 = system:0 nodeep-scrub is unset 2024-03-27T21:40:02.630 INFO:teuthology.orchestra.run.smithi060.stderr:nodeep-scrub is unset 2024-03-27T21:40:02.631 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:02.645+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:02.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.645+0000 7f61d0f67640 10 librados: watch_flush enter 2024-03-27T21:40:02.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.645+0000 7f61d0f67640 10 librados: watch_flush exit 2024-03-27T21:40:02.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.645+0000 7f61d0f67640 20 client.4418.objecter shutdown clearing up homeless session... 2024-03-27T21:40:02.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.645+0000 7f61d0f67640 10 client.4418.objecter successfully canceled tick 2024-03-27T21:40:02.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.645+0000 7f61d0f67640 1 -- 172.21.15.60:0/553034812 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f61ac06c590 msgr2=0x7f61ac06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:02.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.645+0000 7f61d0f67640 1 --2- 172.21.15.60:0/553034812 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f61ac06c590 0x7f61ac06ea50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f61cc1d73e0 tx=0x7f61b4009040 comp rx=0 tx=0).stop 2024-03-27T21:40:02.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.645+0000 7f61d0f67640 10 monclient: shutdown 2024-03-27T21:40:02.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.645+0000 7f61d0f67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:02.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.645+0000 7f61d0f67640 1 -- 172.21.15.60:0/553034812 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f61cc071a50 msgr2=0x7f61cc1d9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:02.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.645+0000 7f61d0f67640 1 --2- 172.21.15.60:0/553034812 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f61cc071a50 0x7f61cc1d9a90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f61bc000c00 tx=0x7f61bc004730 comp rx=0 tx=0).stop 2024-03-27T21:40:02.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.645+0000 7f61d0f67640 1 -- 172.21.15.60:0/553034812 shutdown_connections 2024-03-27T21:40:02.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.645+0000 7f61d0f67640 1 --2- 172.21.15.60:0/553034812 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f61ac06c590 0x7f61ac06ea50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:02.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.645+0000 7f61d0f67640 1 --2- 172.21.15.60:0/553034812 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61cc13ec50 0x7f61cc13aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:02.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.645+0000 7f61d0f67640 1 --2- 172.21.15.60:0/553034812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61cc13e810 0x7f61cc13a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:02.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.645+0000 7f61d0f67640 1 --2- 172.21.15.60:0/553034812 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f61cc071a50 0x7f61cc1d9a90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:02.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.645+0000 7f61d0f67640 1 -- 172.21.15.60:0/553034812 >> 172.21.15.60:0/553034812 conn(0x7f61cc0b7300 msgr2=0x7f61cc13d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:02.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.646+0000 7f61d0f67640 1 -- 172.21.15.60:0/553034812 shutdown_connections 2024-03-27T21:40:02.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.646+0000 7f61d0f67640 1 -- 172.21.15.60:0/553034812 wait complete. 2024-03-27T21:40:02.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.646+0000 7f61d0f67640 1 librados: shutdown 2024-03-27T21:40:02.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.697+0000 7f4234111640 10 monclient: get_monmap_and_config 2024-03-27T21:40:02.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.697+0000 7f4234111640 10 monclient: build_initial_monmap 2024-03-27T21:40:02.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.698+0000 7f4234111640 10 monclient: monmap: 2024-03-27T21:40:02.692 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:02.692 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:02.692 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:02.699002+0000 2024-03-27T21:40:02.692 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:02.699002+0000 2024-03-27T21:40:02.692 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:02.692 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:02.692 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:02.693 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:02.693 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:02.693 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:02.693 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:02.693 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:02.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.698+0000 7f4234111640 1 Processor -- start 2024-03-27T21:40:02.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.698+0000 7f4234111640 1 -- start start 2024-03-27T21:40:02.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.698+0000 7f4234111640 10 monclient: init 2024-03-27T21:40:02.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4234111640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:02.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4234111640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-03-27T21:40:02.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4234111640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f422c0b40b0 0x7f422c0a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:02.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4234111640 10 monclient(hunting): picked mon.noname-a con 0x7f422c0b40b0 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:02.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4234111640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f422c13b020 0x7f422c13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:02.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4234111640 10 monclient(hunting): picked mon.noname-b con 0x7f422c13b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:02.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4234111640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f422c1416f0 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:02.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4234111640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f422c0b3830 con 0x7f422c1416f0 2024-03-27T21:40:02.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4234111640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:02.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4234111640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422c0b4490 con 0x7f422c0b40b0 2024-03-27T21:40:02.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4234111640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:02.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4234111640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c142290 con 0x7f422c13b020 2024-03-27T21:40:02.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4234111640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:02.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4234111640 10 monclient(hunting): authenticate will time out at 843.920545s 2024-03-27T21:40:02.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f42316ac640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f422c13b020 0x7f422c13b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:02.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4231ead640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f422c0b40b0 0x7f422c0a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:02.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f42316ac640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f422c13b020 0x7f422c13b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:39420/0 (socket says 172.21.15.60:39420) 2024-03-27T21:40:02.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f42316ac640 1 -- 172.21.15.60:0/594308332 learned_addr learned my addr 172.21.15.60:0/594308332 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:02.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4231ead640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f422c0b40b0 0x7f422c0a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52648/0 (socket says 172.21.15.60:52648) 2024-03-27T21:40:02.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f42316ac640 10 monclient(hunting): get_auth_request con 0x7f422c13b020 auth_method 0 2024-03-27T21:40:02.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f42316ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:02.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f42316ac640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:02.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f42316ac640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:02.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4231ead640 10 monclient(hunting): get_auth_request con 0x7f422c0b40b0 auth_method 0 2024-03-27T21:40:02.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4231ead640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:02.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4231ead640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:02.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4231ead640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:02.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f42316ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:02.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f42316ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:02.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f42316ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:02.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4231ead640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:02.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4231ead640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:02.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.699+0000 7f4231ead640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:02.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.700+0000 7f42316ac640 10 monclient(hunting): handle_auth_done global_id 4421 payload 274 2024-03-27T21:40:02.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.700+0000 7f42316ac640 1 -- 172.21.15.60:0/594308332 >> v2:172.21.15.60:3300/0 conn(0x7f422c0b40b0 msgr2=0x7f422c0a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:02.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.700+0000 7f42316ac640 1 --2- 172.21.15.60:0/594308332 >> v2:172.21.15.60:3300/0 conn(0x7f422c0b40b0 0x7f422c0a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:02.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.700+0000 7f42316ac640 1 -- 172.21.15.60:0/594308332 >> v1:172.21.15.191:6789/0 conn(0x7f422c1416f0 legacy=0x7f422c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:02.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.700+0000 7f42316ac640 10 monclient: _finish_hunting 0 2024-03-27T21:40:02.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.700+0000 7f42316ac640 1 monclient: found mon.noname-b 2024-03-27T21:40:02.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.700+0000 7f42316ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:02.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.700+0000 7f42316ac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:02.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.700+0000 7f42316ac640 1 -- 172.21.15.60:0/594308332 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422c0aa520 con 0x7f422c13b020 2024-03-27T21:40:02.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.700+0000 7f42316ac640 1 --2- 172.21.15.60:0/594308332 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f422c13b020 0x7f422c13b480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f421c009870 tx=0x7f421c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a84f6e00cc42c77a server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:02.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.700+0000 7f4230eab640 1 -- 172.21.15.60:0/594308332 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c031640 con 0x7f422c13b020 2024-03-27T21:40:02.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.700+0000 7f4230eab640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:02.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.700+0000 7f4230eab640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:02.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.700+0000 7f4230eab640 10 monclient: dump: 2024-03-27T21:40:02.696 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:02.697 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:02.697 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:02.697 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:02.697 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:02.697 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:02.697 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:02.697 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:02.697 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:02.697 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:02.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4230eab640 10 monclient: _finish_auth 0 2024-03-27T21:40:02.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4230eab640 10 monclient: _check_auth_tickets 2024-03-27T21:40:02.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4230eab640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:02.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4234111640 5 monclient: authenticate success, global_id 4421 2024-03-27T21:40:02.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4230eab640 1 -- 172.21.15.60:0/594308332 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421c0317a0 con 0x7f422c13b020 2024-03-27T21:40:02.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4234111640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:02.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4230eab640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:02.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4234111640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:02.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4234111640 4 set_mon_vals no callback set 2024-03-27T21:40:02.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4230eab640 1 -- 172.21.15.60:0/594308332 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c031b30 con 0x7f422c13b020 2024-03-27T21:40:02.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4230eab640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:02.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4230eab640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:02.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4230eab640 10 monclient: dump: 2024-03-27T21:40:02.698 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:02.698 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:02.698 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:02.698 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:02.698 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:02.698 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:02.698 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:02.699 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:02.699 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:02.699 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:02.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4234111640 10 monclient: shutdown 2024-03-27T21:40:02.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4234111640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:02.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4234111640 1 -- 172.21.15.60:0/594308332 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f422c13b020 msgr2=0x7f422c13b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:02.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4234111640 1 --2- 172.21.15.60:0/594308332 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f422c13b020 0x7f422c13b480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f421c009870 tx=0x7f421c02b8f0 comp rx=0 tx=0).stop 2024-03-27T21:40:02.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4234111640 1 -- 172.21.15.60:0/594308332 shutdown_connections 2024-03-27T21:40:02.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4234111640 1 --2- 172.21.15.60:0/594308332 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f422c13b020 0x7f422c13b480 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:02.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4234111640 1 --2- 172.21.15.60:0/594308332 >> v2:172.21.15.60:3300/0 conn(0x7f422c0b40b0 0x7f422c0a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:02.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4234111640 1 -- 172.21.15.60:0/594308332 >> 172.21.15.60:0/594308332 conn(0x7f422c0608e0 msgr2=0x7f422c060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:02.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4234111640 1 -- 172.21.15.60:0/594308332 shutdown_connections 2024-03-27T21:40:02.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.701+0000 7f4234111640 1 -- 172.21.15.60:0/594308332 wait complete. 2024-03-27T21:40:02.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.702+0000 7f4234111640 10 monclient: build_initial_monmap 2024-03-27T21:40:02.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.702+0000 7f4234111640 10 monclient: monmap: 2024-03-27T21:40:02.700 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:02.700 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:02.700 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:02.700 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:02.700 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:02.700 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:02.700 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:02.700 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:02.700 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:02.700 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:02.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.702+0000 7f4234111640 1 librados: starting msgr at 2024-03-27T21:40:02.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.702+0000 7f4234111640 1 librados: starting objecter 2024-03-27T21:40:02.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.702+0000 7f4234111640 1 Processor -- start 2024-03-27T21:40:02.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.702+0000 7f4234111640 1 -- start start 2024-03-27T21:40:02.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.702+0000 7f4234111640 1 librados: setting wanted keys 2024-03-27T21:40:02.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.702+0000 7f4234111640 1 librados: calling monclient init 2024-03-27T21:40:02.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.702+0000 7f4234111640 10 monclient: init 2024-03-27T21:40:02.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4234111640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:02.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4234111640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:40:02.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4234111640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f422c0b40b0 0x7f422c13bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:02.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4234111640 10 monclient(hunting): picked mon.noname-c con 0x7f422c0b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:02.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4234111640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f422c13b020 0x7f422c1d9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:02.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4234111640 10 monclient(hunting): picked mon.noname-b con 0x7f422c13b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:02.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4234111640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f422c1416f0 0x7f422c1da180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:02.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4234111640 10 monclient(hunting): picked mon.noname-a con 0x7f422c1416f0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:02.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4234111640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:02.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4234111640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c13aaf0 con 0x7f422c1416f0 2024-03-27T21:40:02.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4234111640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:02.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4234111640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c13a680 con 0x7f422c13b020 2024-03-27T21:40:02.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4234111640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:02.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4234111640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c13a840 con 0x7f422c0b40b0 2024-03-27T21:40:02.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4234111640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:02.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4234111640 10 monclient(hunting): authenticate will time out at 843.924238s 2024-03-27T21:40:02.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4231ead640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f422c0b40b0 0x7f422c13bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:02.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4231ead640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f422c0b40b0 0x7f422c13bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:39422/0 (socket says 172.21.15.60:39422) 2024-03-27T21:40:02.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4231ead640 1 -- 172.21.15.60:0/2374211247 learned_addr learned my addr 172.21.15.60:0/2374211247 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:02.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4231ead640 10 monclient(hunting): get_auth_request con 0x7f422c0b40b0 auth_method 0 2024-03-27T21:40:02.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4231ead640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:02.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4231ead640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:02.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4231ead640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:02.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f42326ae640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f422c1416f0 0x7f422c1da180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:02.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f42326ae640 10 monclient(hunting): get_auth_request con 0x7f422c1416f0 auth_method 0 2024-03-27T21:40:02.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f42326ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:02.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f42326ae640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:02.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f42326ae640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:02.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4231ead640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:02.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4231ead640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:02.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4231ead640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:02.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f42326ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:02.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f42326ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:02.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f42326ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:02.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4231ead640 10 monclient(hunting): handle_auth_done global_id 4424 payload 824 2024-03-27T21:40:02.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4231ead640 1 -- 172.21.15.60:0/2374211247 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f422c13b020 msgr2=0x7f422c1d9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:02.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4231ead640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f422c13b020 0x7f422c1d9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:02.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4231ead640 1 -- 172.21.15.60:0/2374211247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f422c1416f0 msgr2=0x7f422c1da180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:02.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4231ead640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f422c1416f0 0x7f422c1da180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:02.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4231ead640 10 monclient: _finish_hunting 0 2024-03-27T21:40:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4231ead640 1 monclient: found mon.noname-c 2024-03-27T21:40:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4231ead640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4231ead640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:40:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4231ead640 1 -- 172.21.15.60:0/2374211247 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421c0094d0 con 0x7f422c0b40b0 2024-03-27T21:40:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f42326ae640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f422c1416f0 0x7f422c1da180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.703+0000 7f4231ead640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f422c0b40b0 0x7f422c13bcd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f422800e9b0 tx=0x7f422800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f42227fc640 10 client.?.objecter ms_handle_connect 0x7f422c0b40b0 2024-03-27T21:40:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f42227fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f42227fc640 1 -- 172.21.15.60:0/2374211247 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228009670 con 0x7f422c0b40b0 2024-03-27T21:40:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f42227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f42227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f42227fc640 10 monclient: dump: 2024-03-27T21:40:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:02.705 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:02.705 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:02.705 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:02.705 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:02.705 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:02.705 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:02.705 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:02.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f42227fc640 10 monclient: _finish_auth 0 2024-03-27T21:40:02.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f42227fc640 10 monclient: _check_auth_tickets 2024-03-27T21:40:02.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f42227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:02.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f4234111640 5 monclient: authenticate success, global_id 4424 2024-03-27T21:40:02.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f42227fc640 1 -- 172.21.15.60:0/2374211247 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422800c9a0 con 0x7f422c0b40b0 2024-03-27T21:40:02.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f42227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:02.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f42227fc640 1 -- 172.21.15.60:0/2374211247 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422800cd50 con 0x7f422c0b40b0 2024-03-27T21:40:02.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f4234111640 10 monclient: _renew_subs 2024-03-27T21:40:02.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f4234111640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:02.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f4234111640 1 -- 172.21.15.60:0/2374211247 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422c1da6c0 con 0x7f422c0b40b0 2024-03-27T21:40:02.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f4223fff640 4 set_mon_vals no callback set 2024-03-27T21:40:02.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f42227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:02.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f42227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:02.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f42227fc640 10 monclient: dump: 2024-03-27T21:40:02.706 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:02.706 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:02.706 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:02.706 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:02.706 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:02.706 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:02.706 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:02.706 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:02.706 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:02.707 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:02.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f4234111640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:02.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f4234111640 10 monclient: _renew_subs 2024-03-27T21:40:02.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f4234111640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:02.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f4234111640 1 -- 172.21.15.60:0/2374211247 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f422c137030 con 0x7f422c0b40b0 2024-03-27T21:40:02.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.704+0000 7f4234111640 1 librados: init done 2024-03-27T21:40:02.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.705+0000 7f4234111640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:02.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.705+0000 7f4234111640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:02.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.705+0000 7f4234111640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:02.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.705+0000 7f4234111640 1 -- 172.21.15.60:0/2374211247 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f422c069470 con 0x7f422c0b40b0 2024-03-27T21:40:02.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.706+0000 7f42227fc640 1 -- 172.21.15.60:0/2374211247 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4228013470 con 0x7f422c0b40b0 2024-03-27T21:40:02.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.706+0000 7f42227fc640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f421006c590 0x7f421006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:02.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.706+0000 7f42227fc640 1 -- 172.21.15.60:0/2374211247 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4228089340 con 0x7f422c0b40b0 2024-03-27T21:40:02.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.706+0000 7f42227fc640 10 client.4424.objecter ms_dispatch 0x7f422c000b90 osd_map(26..26 src has 1..26) v4 2024-03-27T21:40:02.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.706+0000 7f42227fc640 3 client.4424.objecter handle_osd_map got epochs [26,26] > 0 2024-03-27T21:40:02.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.706+0000 7f42227fc640 3 client.4424.objecter handle_osd_map decoding full epoch 26 2024-03-27T21:40:02.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.706+0000 7f42316ac640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f421006c590 0x7f421006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:02.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f42316ac640 10 monclient: get_auth_request con 0x7f421006c590 auth_method 0 2024-03-27T21:40:02.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f42227fc640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-27T21:40:02.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f42227fc640 1 -- 172.21.15.60:0/2374211247 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v26) v1 ==== 54+0+16 (secure 0 0 0) 0x7f4228090070 con 0x7f422c0b40b0 2024-03-27T21:40:02.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f42227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:02.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f42227fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:02.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f4234111640 10 client.4424.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:02.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign -1 1 2024-03-27T21:40:02.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f4234111640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f422c13c2b0 0x7f422c138ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:02.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c0a90d0 osd=0 3 2024-03-27T21:40:02.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c0a90d0 osd=0 3 2024-03-27T21:40:02.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c0a90d0 osd=0 3 2024-03-27T21:40:02.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f4234111640 15 client.4424.objecter _session_command_op_remove -1 1 2024-03-27T21:40:02.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f4234111640 20 client.4424.objecter get_session s=0x7f422c0a90d0 osd=0 3 2024-03-27T21:40:02.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign 0 1 2024-03-27T21:40:02.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c0a90d0 osd=0 4 2024-03-27T21:40:02.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f4234111640 10 client.4424.objecter _send_command 1 2024-03-27T21:40:02.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f4234111640 1 -- 172.21.15.60:0/2374211247 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f422c13ac30 con 0x7f422c13c2b0 2024-03-27T21:40:02.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f42326ae640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f422c13c2b0 0x7f422c138ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:02.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f42326ae640 10 monclient: get_auth_request con 0x7f422c13c2b0 auth_method 0 2024-03-27T21:40:02.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f42316ac640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f421006c590 0x7f421006ea50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f421c036000 tx=0x7f421c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:02.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f42227fc640 10 client.4424.objecter ms_handle_connect 0x7f421006c590 2024-03-27T21:40:02.711 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:02.711 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:02.711 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:02.711 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:02.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f42227fc640 10 client.4424.objecter ms_handle_connect 0x7f422c13c2b0 2024-03-27T21:40:02.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.707+0000 7f42326ae640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f422c13c2b0 0x7f422c138ce0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:02.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.708+0000 7f42227fc640 1 -- 172.21.15.60:0/2374211247 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f422400b900 con 0x7f422c13c2b0 2024-03-27T21:40:02.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.708+0000 7f42227fc640 10 client.4424.objecter ms_dispatch 0x7f422c000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:02.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.708+0000 7f42227fc640 10 client.4424.objecter _finish_command 1 = system:0 2024-03-27T21:40:02.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.708+0000 7f42227fc640 20 client.4424.objecter put_session s=0x7f422c0a90d0 osd=0 4 2024-03-27T21:40:02.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.708+0000 7f42227fc640 15 client.4424.objecter _session_command_op_remove 0 1 2024-03-27T21:40:02.713 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:40:02.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.723+0000 7f4234111640 10 client.4424.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:02.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.723+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign -1 2 2024-03-27T21:40:02.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.723+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c0a90d0 osd=0 3 2024-03-27T21:40:02.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.723+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c0a90d0 osd=0 3 2024-03-27T21:40:02.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.723+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c0a90d0 osd=0 3 2024-03-27T21:40:02.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.723+0000 7f4234111640 15 client.4424.objecter _session_command_op_remove -1 2 2024-03-27T21:40:02.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.723+0000 7f4234111640 20 client.4424.objecter get_session s=0x7f422c0a90d0 osd=0 3 2024-03-27T21:40:02.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.723+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign 0 2 2024-03-27T21:40:02.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.723+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c0a90d0 osd=0 4 2024-03-27T21:40:02.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.723+0000 7f4234111640 10 client.4424.objecter _send_command 2 2024-03-27T21:40:02.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.723+0000 7f4234111640 1 -- 172.21.15.60:0/2374211247 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f422c142540 con 0x7f422c13c2b0 2024-03-27T21:40:02.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.725+0000 7f42227fc640 1 -- 172.21.15.60:0/2374211247 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f422c142540 con 0x7f422c13c2b0 2024-03-27T21:40:02.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.725+0000 7f42227fc640 10 client.4424.objecter ms_dispatch 0x7f422c000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:02.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.725+0000 7f42227fc640 10 client.4424.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:02.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.725+0000 7f42227fc640 20 client.4424.objecter put_session s=0x7f422c0a90d0 osd=0 4 2024-03-27T21:40:02.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.725+0000 7f42227fc640 15 client.4424.objecter _session_command_op_remove 0 2 2024-03-27T21:40:02.715 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:02.717 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:02.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.725+0000 7f4234111640 10 client.4424.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:02.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.725+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign -1 3 2024-03-27T21:40:02.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.725+0000 7f4234111640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f422c139390 0x7f422c131260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:02.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.725+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c139220 osd=1 3 2024-03-27T21:40:02.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.725+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c139220 osd=1 3 2024-03-27T21:40:02.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.725+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c139220 osd=1 3 2024-03-27T21:40:02.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.725+0000 7f4234111640 15 client.4424.objecter _session_command_op_remove -1 3 2024-03-27T21:40:02.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.725+0000 7f4234111640 20 client.4424.objecter get_session s=0x7f422c139220 osd=1 3 2024-03-27T21:40:02.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.725+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign 1 3 2024-03-27T21:40:02.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.725+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c139220 osd=1 4 2024-03-27T21:40:02.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.725+0000 7f4234111640 10 client.4424.objecter _send_command 3 2024-03-27T21:40:02.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.725+0000 7f4234111640 1 -- 172.21.15.60:0/2374211247 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f422c1317a0 con 0x7f422c139390 2024-03-27T21:40:02.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.725+0000 7f4231ead640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f422c139390 0x7f422c131260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:02.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.725+0000 7f4231ead640 10 monclient: get_auth_request con 0x7f422c139390 auth_method 0 2024-03-27T21:40:02.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.726+0000 7f42227fc640 10 client.4424.objecter ms_handle_connect 0x7f422c139390 2024-03-27T21:40:02.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.726+0000 7f4231ead640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f422c139390 0x7f422c131260 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:02.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.726+0000 7f42227fc640 1 -- 172.21.15.60:0/2374211247 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f422c1317a0 con 0x7f422c139390 2024-03-27T21:40:02.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.726+0000 7f42227fc640 10 client.4424.objecter ms_dispatch 0x7f422c000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:02.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.726+0000 7f42227fc640 10 client.4424.objecter _finish_command 3 = system:0 2024-03-27T21:40:02.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.726+0000 7f42227fc640 20 client.4424.objecter put_session s=0x7f422c139220 osd=1 4 2024-03-27T21:40:02.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.726+0000 7f42227fc640 15 client.4424.objecter _session_command_op_remove 1 3 2024-03-27T21:40:02.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.741+0000 7f4234111640 10 client.4424.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:02.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.742+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign -1 4 2024-03-27T21:40:02.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.742+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c139220 osd=1 3 2024-03-27T21:40:02.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.742+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c139220 osd=1 3 2024-03-27T21:40:02.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.742+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c139220 osd=1 3 2024-03-27T21:40:02.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.742+0000 7f4234111640 15 client.4424.objecter _session_command_op_remove -1 4 2024-03-27T21:40:02.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.742+0000 7f4234111640 20 client.4424.objecter get_session s=0x7f422c139220 osd=1 3 2024-03-27T21:40:02.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.742+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign 1 4 2024-03-27T21:40:02.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.742+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c139220 osd=1 4 2024-03-27T21:40:02.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.742+0000 7f4234111640 10 client.4424.objecter _send_command 4 2024-03-27T21:40:02.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.742+0000 7f4234111640 1 -- 172.21.15.60:0/2374211247 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f422c131b50 con 0x7f422c139390 2024-03-27T21:40:02.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.743+0000 7f42227fc640 1 -- 172.21.15.60:0/2374211247 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f422c131b50 con 0x7f422c139390 2024-03-27T21:40:02.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.743+0000 7f42227fc640 10 client.4424.objecter ms_dispatch 0x7f422c000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:02.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.743+0000 7f42227fc640 10 client.4424.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:02.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.743+0000 7f42227fc640 20 client.4424.objecter put_session s=0x7f422c139220 osd=1 4 2024-03-27T21:40:02.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.743+0000 7f42227fc640 15 client.4424.objecter _session_command_op_remove 1 4 2024-03-27T21:40:02.730 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:02.730 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:02.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.743+0000 7f4234111640 10 client.4424.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:02.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.743+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign -1 5 2024-03-27T21:40:02.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.743+0000 7f4234111640 20 client.4424.objecter _get_session osd=-1 returning homeless 2024-03-27T21:40:02.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.743+0000 7f4234111640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:02.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.743+0000 7f4234111640 10 monclient: _renew_subs 2024-03-27T21:40:02.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.743+0000 7f4234111640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:02.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.743+0000 7f4234111640 1 -- 172.21.15.60:0/2374211247 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=27}) v3 -- 0x7f422c13fba0 con 0x7f422c0b40b0 2024-03-27T21:40:02.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.743+0000 7f4234111640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:02.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.743+0000 7f4234111640 1 -- 172.21.15.60:0/2374211247 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f422c13feb0 con 0x7f422c0b40b0 2024-03-27T21:40:02.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.744+0000 7f42227fc640 1 -- 172.21.15.60:0/2374211247 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4228058250 con 0x7f422c0b40b0 2024-03-27T21:40:02.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.744+0000 7f42227fc640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-03-27T21:40:02.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.744+0000 7f4223fff640 10 client.4424.objecter _check_command_map_dne tid 5 current 26 map_dne_bound 26 2024-03-27T21:40:02.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.744+0000 7f4223fff640 10 client.4424.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:40:02.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.744+0000 7f4223fff640 15 client.4424.objecter _session_command_op_remove -1 5 2024-03-27T21:40:02.733 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:40:02.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.744+0000 7f4234111640 10 client.4424.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:02.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.744+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign -1 6 2024-03-27T21:40:02.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.744+0000 7f4234111640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f422c02b170 0x7f422c02b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:02.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.744+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c066490 osd=3 3 2024-03-27T21:40:02.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.744+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c066490 osd=3 3 2024-03-27T21:40:02.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.744+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c066490 osd=3 3 2024-03-27T21:40:02.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.744+0000 7f4234111640 15 client.4424.objecter _session_command_op_remove -1 6 2024-03-27T21:40:02.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.744+0000 7f4234111640 20 client.4424.objecter get_session s=0x7f422c066490 osd=3 3 2024-03-27T21:40:02.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.744+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign 3 6 2024-03-27T21:40:02.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.744+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c066490 osd=3 4 2024-03-27T21:40:02.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.744+0000 7f4234111640 10 client.4424.objecter _send_command 6 2024-03-27T21:40:02.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.744+0000 7f4234111640 1 -- 172.21.15.60:0/2374211247 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f422c02bad0 con 0x7f422c02b170 2024-03-27T21:40:02.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.744+0000 7f42316ac640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f422c02b170 0x7f422c02b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:02.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.744+0000 7f42316ac640 10 monclient: get_auth_request con 0x7f422c02b170 auth_method 0 2024-03-27T21:40:02.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.744+0000 7f42227fc640 10 client.4424.objecter ms_handle_connect 0x7f422c02b170 2024-03-27T21:40:02.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.744+0000 7f42316ac640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f422c02b170 0x7f422c02b590 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:02.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.745+0000 7f42227fc640 1 -- 172.21.15.60:0/2374211247 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f422c02bad0 con 0x7f422c02b170 2024-03-27T21:40:02.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.745+0000 7f42227fc640 10 client.4424.objecter ms_dispatch 0x7f422c000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:40:02.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.745+0000 7f42227fc640 10 client.4424.objecter _finish_command 6 = system:0 2024-03-27T21:40:02.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.745+0000 7f42227fc640 20 client.4424.objecter put_session s=0x7f422c066490 osd=3 4 2024-03-27T21:40:02.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.745+0000 7f42227fc640 15 client.4424.objecter _session_command_op_remove 3 6 2024-03-27T21:40:02.735 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:02.749+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:02.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.760+0000 7f4234111640 10 client.4424.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:02.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.760+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign -1 7 2024-03-27T21:40:02.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.760+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c066490 osd=3 3 2024-03-27T21:40:02.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.760+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c066490 osd=3 3 2024-03-27T21:40:02.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.760+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c066490 osd=3 3 2024-03-27T21:40:02.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.760+0000 7f4234111640 15 client.4424.objecter _session_command_op_remove -1 7 2024-03-27T21:40:02.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.760+0000 7f4234111640 20 client.4424.objecter get_session s=0x7f422c066490 osd=3 3 2024-03-27T21:40:02.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.760+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign 3 7 2024-03-27T21:40:02.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.760+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c066490 osd=3 4 2024-03-27T21:40:02.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.760+0000 7f4234111640 10 client.4424.objecter _send_command 7 2024-03-27T21:40:02.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.760+0000 7f4234111640 1 -- 172.21.15.60:0/2374211247 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f422c02c430 con 0x7f422c02b170 2024-03-27T21:40:02.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.762+0000 7f42227fc640 1 -- 172.21.15.60:0/2374211247 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f422c02c430 con 0x7f422c02b170 2024-03-27T21:40:02.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.762+0000 7f42227fc640 10 client.4424.objecter ms_dispatch 0x7f422c000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:02.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.762+0000 7f42227fc640 10 client.4424.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:02.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.762+0000 7f42227fc640 20 client.4424.objecter put_session s=0x7f422c066490 osd=3 4 2024-03-27T21:40:02.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.762+0000 7f42227fc640 15 client.4424.objecter _session_command_op_remove 3 7 2024-03-27T21:40:02.749 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:02.751 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:02.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.762+0000 7f4234111640 10 client.4424.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:02.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.762+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign -1 8 2024-03-27T21:40:02.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.762+0000 7f4234111640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f422c066080 0x7f422c1e96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:02.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.762+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c065f10 osd=4 3 2024-03-27T21:40:02.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.762+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c065f10 osd=4 3 2024-03-27T21:40:02.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.762+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c065f10 osd=4 3 2024-03-27T21:40:02.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.762+0000 7f4234111640 15 client.4424.objecter _session_command_op_remove -1 8 2024-03-27T21:40:02.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.762+0000 7f4234111640 20 client.4424.objecter get_session s=0x7f422c065f10 osd=4 3 2024-03-27T21:40:02.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.762+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign 4 8 2024-03-27T21:40:02.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.762+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c065f10 osd=4 4 2024-03-27T21:40:02.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.762+0000 7f4234111640 10 client.4424.objecter _send_command 8 2024-03-27T21:40:02.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.762+0000 7f4234111640 1 -- 172.21.15.60:0/2374211247 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f422c1ec4a0 con 0x7f422c066080 2024-03-27T21:40:02.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.763+0000 7f42326ae640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f422c066080 0x7f422c1e96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:02.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.763+0000 7f42326ae640 10 monclient: get_auth_request con 0x7f422c066080 auth_method 0 2024-03-27T21:40:02.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.763+0000 7f42227fc640 10 client.4424.objecter ms_handle_connect 0x7f422c066080 2024-03-27T21:40:02.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.763+0000 7f42326ae640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f422c066080 0x7f422c1e96d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:02.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.764+0000 7f42227fc640 1 -- 172.21.15.60:0/2374211247 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f422c1ec4a0 con 0x7f422c066080 2024-03-27T21:40:02.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.764+0000 7f42227fc640 10 client.4424.objecter ms_dispatch 0x7f422c000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:40:02.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.764+0000 7f42227fc640 10 client.4424.objecter _finish_command 8 = system:0 2024-03-27T21:40:02.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.764+0000 7f42227fc640 20 client.4424.objecter put_session s=0x7f422c065f10 osd=4 4 2024-03-27T21:40:02.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.764+0000 7f42227fc640 15 client.4424.objecter _session_command_op_remove 4 8 2024-03-27T21:40:02.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.778+0000 7f4234111640 10 client.4424.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:02.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.778+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign -1 9 2024-03-27T21:40:02.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.778+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c065f10 osd=4 3 2024-03-27T21:40:02.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.778+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c065f10 osd=4 3 2024-03-27T21:40:02.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.778+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c065f10 osd=4 3 2024-03-27T21:40:02.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.778+0000 7f4234111640 15 client.4424.objecter _session_command_op_remove -1 9 2024-03-27T21:40:02.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.778+0000 7f4234111640 20 client.4424.objecter get_session s=0x7f422c065f10 osd=4 3 2024-03-27T21:40:02.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.778+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign 4 9 2024-03-27T21:40:02.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.778+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c065f10 osd=4 4 2024-03-27T21:40:02.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.778+0000 7f4234111640 10 client.4424.objecter _send_command 9 2024-03-27T21:40:02.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.778+0000 7f4234111640 1 -- 172.21.15.60:0/2374211247 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f422c02beb0 con 0x7f422c066080 2024-03-27T21:40:02.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.780+0000 7f42227fc640 1 -- 172.21.15.60:0/2374211247 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f422c02beb0 con 0x7f422c066080 2024-03-27T21:40:02.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.780+0000 7f42227fc640 10 client.4424.objecter ms_dispatch 0x7f422c000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:02.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.780+0000 7f42227fc640 10 client.4424.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:02.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.780+0000 7f42227fc640 20 client.4424.objecter put_session s=0x7f422c065f10 osd=4 4 2024-03-27T21:40:02.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.780+0000 7f42227fc640 15 client.4424.objecter _session_command_op_remove 4 9 2024-03-27T21:40:02.767 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:02.768 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:02.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.780+0000 7f4234111640 10 client.4424.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:02.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.780+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign -1 10 2024-03-27T21:40:02.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.780+0000 7f4234111640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f422c02c020 0x7f422c1ead90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:02.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.780+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c02beb0 osd=5 3 2024-03-27T21:40:02.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.780+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c02beb0 osd=5 3 2024-03-27T21:40:02.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.780+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c02beb0 osd=5 3 2024-03-27T21:40:02.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.780+0000 7f4234111640 15 client.4424.objecter _session_command_op_remove -1 10 2024-03-27T21:40:02.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.780+0000 7f4234111640 20 client.4424.objecter get_session s=0x7f422c02beb0 osd=5 3 2024-03-27T21:40:02.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.780+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign 5 10 2024-03-27T21:40:02.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.780+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c02beb0 osd=5 4 2024-03-27T21:40:02.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.780+0000 7f4234111640 10 client.4424.objecter _send_command 10 2024-03-27T21:40:02.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.780+0000 7f4234111640 1 -- 172.21.15.60:0/2374211247 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f422c0b5890 con 0x7f422c02c020 2024-03-27T21:40:02.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.781+0000 7f4231ead640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f422c02c020 0x7f422c1ead90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:02.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.781+0000 7f4231ead640 10 monclient: get_auth_request con 0x7f422c02c020 auth_method 0 2024-03-27T21:40:02.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.781+0000 7f42227fc640 10 client.4424.objecter ms_handle_connect 0x7f422c02c020 2024-03-27T21:40:02.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.781+0000 7f4231ead640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f422c02c020 0x7f422c1ead90 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:02.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.782+0000 7f42227fc640 1 -- 172.21.15.60:0/2374211247 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f422c0b5890 con 0x7f422c02c020 2024-03-27T21:40:02.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.782+0000 7f42227fc640 10 client.4424.objecter ms_dispatch 0x7f422c000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:40:02.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.782+0000 7f42227fc640 10 client.4424.objecter _finish_command 10 = system:0 2024-03-27T21:40:02.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.782+0000 7f42227fc640 20 client.4424.objecter put_session s=0x7f422c02beb0 osd=5 4 2024-03-27T21:40:02.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.782+0000 7f42227fc640 15 client.4424.objecter _session_command_op_remove 5 10 2024-03-27T21:40:02.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.796+0000 7f4234111640 10 client.4424.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:02.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.796+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign -1 11 2024-03-27T21:40:02.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.796+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c02beb0 osd=5 3 2024-03-27T21:40:02.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.796+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c02beb0 osd=5 3 2024-03-27T21:40:02.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.796+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c02beb0 osd=5 3 2024-03-27T21:40:02.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.796+0000 7f4234111640 15 client.4424.objecter _session_command_op_remove -1 11 2024-03-27T21:40:02.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.796+0000 7f4234111640 20 client.4424.objecter get_session s=0x7f422c02beb0 osd=5 3 2024-03-27T21:40:02.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.796+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign 5 11 2024-03-27T21:40:02.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.796+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c02beb0 osd=5 4 2024-03-27T21:40:02.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.796+0000 7f4234111640 10 client.4424.objecter _send_command 11 2024-03-27T21:40:02.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.796+0000 7f4234111640 1 -- 172.21.15.60:0/2374211247 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f422c0b5890 con 0x7f422c02c020 2024-03-27T21:40:02.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.798+0000 7f42227fc640 1 -- 172.21.15.60:0/2374211247 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f422c0b5890 con 0x7f422c02c020 2024-03-27T21:40:02.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.798+0000 7f42227fc640 10 client.4424.objecter ms_dispatch 0x7f422c000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:02.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.798+0000 7f42227fc640 10 client.4424.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:02.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.798+0000 7f42227fc640 20 client.4424.objecter put_session s=0x7f422c02beb0 osd=5 4 2024-03-27T21:40:02.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.798+0000 7f42227fc640 15 client.4424.objecter _session_command_op_remove 5 11 2024-03-27T21:40:02.785 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:02.786 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:02.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.798+0000 7f4234111640 10 client.4424.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:02.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.798+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign -1 12 2024-03-27T21:40:02.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.798+0000 7f4234111640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f422c1eb7e0 0x7f422c1ebc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:02.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.798+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c0b5890 osd=6 3 2024-03-27T21:40:02.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.798+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c0b5890 osd=6 3 2024-03-27T21:40:02.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.798+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c0b5890 osd=6 3 2024-03-27T21:40:02.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.798+0000 7f4234111640 15 client.4424.objecter _session_command_op_remove -1 12 2024-03-27T21:40:02.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.798+0000 7f4234111640 20 client.4424.objecter get_session s=0x7f422c0b5890 osd=6 3 2024-03-27T21:40:02.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.798+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign 6 12 2024-03-27T21:40:02.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.798+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c0b5890 osd=6 4 2024-03-27T21:40:02.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.798+0000 7f4234111640 10 client.4424.objecter _send_command 12 2024-03-27T21:40:02.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.798+0000 7f4234111640 1 -- 172.21.15.60:0/2374211247 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f422c1ec1a0 con 0x7f422c1eb7e0 2024-03-27T21:40:02.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.798+0000 7f42316ac640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f422c1eb7e0 0x7f422c1ebc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:02.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.798+0000 7f42316ac640 10 monclient: get_auth_request con 0x7f422c1eb7e0 auth_method 0 2024-03-27T21:40:02.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.799+0000 7f42316ac640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f422c1eb7e0 0x7f422c1ebc60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:02.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.799+0000 7f42227fc640 10 client.4424.objecter ms_handle_connect 0x7f422c1eb7e0 2024-03-27T21:40:02.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.799+0000 7f42227fc640 1 -- 172.21.15.60:0/2374211247 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f422c1ec1a0 con 0x7f422c1eb7e0 2024-03-27T21:40:02.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.799+0000 7f42227fc640 10 client.4424.objecter ms_dispatch 0x7f422c000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:40:02.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.799+0000 7f42227fc640 10 client.4424.objecter _finish_command 12 = system:0 2024-03-27T21:40:02.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.799+0000 7f42227fc640 20 client.4424.objecter put_session s=0x7f422c0b5890 osd=6 4 2024-03-27T21:40:02.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.799+0000 7f42227fc640 15 client.4424.objecter _session_command_op_remove 6 12 2024-03-27T21:40:02.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.815+0000 7f4234111640 10 client.4424.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:02.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.815+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign -1 13 2024-03-27T21:40:02.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.815+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c0b5890 osd=6 3 2024-03-27T21:40:02.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.815+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c0b5890 osd=6 3 2024-03-27T21:40:02.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.815+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c0b5890 osd=6 3 2024-03-27T21:40:02.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.815+0000 7f4234111640 15 client.4424.objecter _session_command_op_remove -1 13 2024-03-27T21:40:02.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.815+0000 7f4234111640 20 client.4424.objecter get_session s=0x7f422c0b5890 osd=6 3 2024-03-27T21:40:02.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.815+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign 6 13 2024-03-27T21:40:02.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.815+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c0b5890 osd=6 4 2024-03-27T21:40:02.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.815+0000 7f4234111640 10 client.4424.objecter _send_command 13 2024-03-27T21:40:02.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.815+0000 7f4234111640 1 -- 172.21.15.60:0/2374211247 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f422c1ec1a0 con 0x7f422c1eb7e0 2024-03-27T21:40:02.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.817+0000 7f42227fc640 1 -- 172.21.15.60:0/2374211247 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f422c1ec1a0 con 0x7f422c1eb7e0 2024-03-27T21:40:02.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.817+0000 7f42227fc640 10 client.4424.objecter ms_dispatch 0x7f422c000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:02.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.817+0000 7f42227fc640 10 client.4424.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:02.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.817+0000 7f42227fc640 20 client.4424.objecter put_session s=0x7f422c0b5890 osd=6 4 2024-03-27T21:40:02.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.817+0000 7f42227fc640 15 client.4424.objecter _session_command_op_remove 6 13 2024-03-27T21:40:02.804 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:02.805 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:02.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.817+0000 7f4234111640 10 client.4424.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:02.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.817+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign -1 14 2024-03-27T21:40:02.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.817+0000 7f4234111640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f422c1f0b40 0x7f422c1f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:02.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.817+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c1ec1a0 osd=7 3 2024-03-27T21:40:02.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.817+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c1ec1a0 osd=7 3 2024-03-27T21:40:02.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.817+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c1ec1a0 osd=7 3 2024-03-27T21:40:02.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.817+0000 7f4234111640 15 client.4424.objecter _session_command_op_remove -1 14 2024-03-27T21:40:02.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.817+0000 7f4234111640 20 client.4424.objecter get_session s=0x7f422c1ec1a0 osd=7 3 2024-03-27T21:40:02.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.817+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign 7 14 2024-03-27T21:40:02.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.817+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c1ec1a0 osd=7 4 2024-03-27T21:40:02.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.817+0000 7f4234111640 10 client.4424.objecter _send_command 14 2024-03-27T21:40:02.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.817+0000 7f4234111640 1 -- 172.21.15.60:0/2374211247 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f422c0b5ab0 con 0x7f422c1f0b40 2024-03-27T21:40:02.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.817+0000 7f42326ae640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f422c1f0b40 0x7f422c1f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:02.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.817+0000 7f42326ae640 10 monclient: get_auth_request con 0x7f422c1f0b40 auth_method 0 2024-03-27T21:40:02.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.818+0000 7f42227fc640 10 client.4424.objecter ms_handle_connect 0x7f422c1f0b40 2024-03-27T21:40:02.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.818+0000 7f42326ae640 1 --2- 172.21.15.60:0/2374211247 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f422c1f0b40 0x7f422c1f2f30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:02.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.818+0000 7f42227fc640 1 -- 172.21.15.60:0/2374211247 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f422c0b5ab0 con 0x7f422c1f0b40 2024-03-27T21:40:02.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.818+0000 7f42227fc640 10 client.4424.objecter ms_dispatch 0x7f422c000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:40:02.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.818+0000 7f42227fc640 10 client.4424.objecter _finish_command 14 = system:0 2024-03-27T21:40:02.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.818+0000 7f42227fc640 20 client.4424.objecter put_session s=0x7f422c1ec1a0 osd=7 4 2024-03-27T21:40:02.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.818+0000 7f42227fc640 15 client.4424.objecter _session_command_op_remove 7 14 2024-03-27T21:40:02.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.833+0000 7f4234111640 10 client.4424.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:02.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.833+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign -1 15 2024-03-27T21:40:02.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.833+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c1ec1a0 osd=7 3 2024-03-27T21:40:02.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.833+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c1ec1a0 osd=7 3 2024-03-27T21:40:02.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.833+0000 7f4234111640 20 client.4424.objecter _get_session s=0x7f422c1ec1a0 osd=7 3 2024-03-27T21:40:02.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.833+0000 7f4234111640 15 client.4424.objecter _session_command_op_remove -1 15 2024-03-27T21:40:02.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.833+0000 7f4234111640 20 client.4424.objecter get_session s=0x7f422c1ec1a0 osd=7 3 2024-03-27T21:40:02.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.833+0000 7f4234111640 15 client.4424.objecter _session_command_op_assign 7 15 2024-03-27T21:40:02.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.833+0000 7f4234111640 20 client.4424.objecter put_session s=0x7f422c1ec1a0 osd=7 4 2024-03-27T21:40:02.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.833+0000 7f4234111640 10 client.4424.objecter _send_command 15 2024-03-27T21:40:02.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.833+0000 7f4234111640 1 -- 172.21.15.60:0/2374211247 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f422c0b5ab0 con 0x7f422c1f0b40 2024-03-27T21:40:02.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.836+0000 7f42227fc640 1 -- 172.21.15.60:0/2374211247 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f422c0b5ab0 con 0x7f422c1f0b40 2024-03-27T21:40:02.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.836+0000 7f42227fc640 10 client.4424.objecter ms_dispatch 0x7f422c000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:02.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.836+0000 7f42227fc640 10 client.4424.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:02.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.836+0000 7f42227fc640 20 client.4424.objecter put_session s=0x7f422c1ec1a0 osd=7 4 2024-03-27T21:40:02.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:02.836+0000 7f42227fc640 15 client.4424.objecter _session_command_op_remove 7 15 2024-03-27T21:40:02.823 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:02.823 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:02.825 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:40:02.826 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:40:02.846 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:02.846 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:02.847 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:02.860+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:02.857 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:40:02.937 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:02.951+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:03.010 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:03.011 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:03.019 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:40:03.038 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:03.052+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:03.128 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:03.128 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:03.137 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:40:03.138 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:03.152+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:03.239 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:03.253+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:03.243 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:03.243 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:03.252 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:40:03.341 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:03.355+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:03.400 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:03.400 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:03.409 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:40:03.454 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:03.468+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:03.525 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:03.526 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:03.536 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:40:03.577 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:03.591+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:03.644 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:03.658+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:03.684 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:03.685 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:03.693 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:40:03.745 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:03.759+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:03.838 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:03.838 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:03.847 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:40:03.848 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:03.860+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:03.948 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:03.962+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:03.951 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:03.952 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:03.960 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:40:04.049 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:04.064+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:04.106 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:04.106 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:04.114 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:40:04.156 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:04.170+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:04.251 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:04.265+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:04.260 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:04.260 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:04.269 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:40:04.354 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:04.369+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:04.415 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:04.416 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:04.423 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:40:04.466 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:04.479+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:04.554 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:04.568+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:04.569 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:04.569 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:04.579 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:40:04.648 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-27T21:40:04.655 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:04.669+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:04.685 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:04.685 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:04.693 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:40:04.748 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-27T21:40:04.750 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T21:40:04.751 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-27T21:40:04.751 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 1.000000 2024-03-27T21:40:04.751 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-27T21:40:04.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.764+0000 7fdedf599640 10 monclient: get_monmap_and_config 2024-03-27T21:40:04.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.764+0000 7fdedf599640 10 monclient: build_initial_monmap 2024-03-27T21:40:04.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.764+0000 7fdedf599640 10 monclient: monmap: 2024-03-27T21:40:04.755 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:04.755 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:04.755 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:04.765295+0000 2024-03-27T21:40:04.755 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:04.765295+0000 2024-03-27T21:40:04.756 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:04.756 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:04.756 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:04.756 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:04.756 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:04.756 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:04.756 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:04.756 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:04.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.764+0000 7fdedf599640 1 Processor -- start 2024-03-27T21:40:04.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedf599640 1 -- start start 2024-03-27T21:40:04.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedf599640 10 monclient: init 2024-03-27T21:40:04.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedf599640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:04.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedf599640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-03-27T21:40:04.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedf599640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fded80b40b0 0x7fded80a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:04.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedf599640 10 monclient(hunting): picked mon.noname-b con 0x7fded80b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:04.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedf599640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fded813b020 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:04.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedf599640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fded813b480 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:04.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedf599640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded80b3830 con 0x7fded813b020 2024-03-27T21:40:04.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedf599640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded80aa520 con 0x7fded813b480 2024-03-27T21:40:04.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedf599640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:04.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedf599640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded813b910 con 0x7fded80b40b0 2024-03-27T21:40:04.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedf599640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:04.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedf599640 10 monclient(hunting): authenticate will time out at 845.986584s 2024-03-27T21:40:04.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedd335640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fded80b40b0 0x7fded80a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:04.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedcb34640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fded813b020 0x7fded81416f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42024/0 (socket says 172.21.15.60:42024) 2024-03-27T21:40:04.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedcb34640 1 -- 172.21.15.60:0/1525706230 learned_addr learned my addr 172.21.15.60:0/1525706230 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:40:04.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedd335640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fded80b40b0 0x7fded80a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:39424/0 (socket says 172.21.15.60:39424) 2024-03-27T21:40:04.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedd335640 10 monclient(hunting): get_auth_request con 0x7fded80b40b0 auth_method 0 2024-03-27T21:40:04.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedd335640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:04.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedd335640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:04.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedd335640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:04.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedd335640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:04.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedd335640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:04.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.765+0000 7fdedd335640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:04.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdedd335640 10 monclient(hunting): handle_auth_done global_id 4427 payload 274 2024-03-27T21:40:04.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdedd335640 1 -- 172.21.15.60:0/1525706230 >> v1:172.21.15.191:6789/0 conn(0x7fded813b480 legacy=0x7fded814bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:04.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdedd335640 1 -- 172.21.15.60:0/1525706230 >> v1:172.21.15.60:6789/0 conn(0x7fded813b020 legacy=0x7fded81416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:04.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdedd335640 10 monclient: _finish_hunting 0 2024-03-27T21:40:04.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdedd335640 1 monclient: found mon.noname-b 2024-03-27T21:40:04.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdedd335640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:04.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdedd335640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:04.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdedd335640 1 -- 172.21.15.60:0/1525706230 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded80a9470 con 0x7fded80b40b0 2024-03-27T21:40:04.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdedd335640 1 --2- 172.21.15.60:0/1525706230 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fded80b40b0 0x7fded80a8b90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fded40097d0 tx=0x7fded402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2046ff25a948974 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:04.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdecffff640 1 -- 172.21.15.60:0/1525706230 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded4031800 con 0x7fded80b40b0 2024-03-27T21:40:04.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdecffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:04.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdecffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:04.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdecffff640 10 monclient: dump: 2024-03-27T21:40:04.759 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:04.759 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:04.759 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.759 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.759 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:04.759 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:04.759 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:04.759 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:04.759 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:04.759 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:04.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdecffff640 10 monclient: _finish_auth 0 2024-03-27T21:40:04.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdecffff640 10 monclient: _check_auth_tickets 2024-03-27T21:40:04.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdecffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:04.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdecffff640 1 -- 172.21.15.60:0/1525706230 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded4031960 con 0x7fded80b40b0 2024-03-27T21:40:04.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdedf599640 5 monclient: authenticate success, global_id 4427 2024-03-27T21:40:04.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdedf599640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:04.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdecffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:04.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdedf599640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:04.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdedf599640 4 set_mon_vals no callback set 2024-03-27T21:40:04.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdecffff640 1 -- 172.21.15.60:0/1525706230 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded4031cf0 con 0x7fded80b40b0 2024-03-27T21:40:04.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdecffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:04.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdecffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:04.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.766+0000 7fdecffff640 10 monclient: dump: 2024-03-27T21:40:04.760 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:04.760 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:04.761 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.761 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.761 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:04.761 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:04.761 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:04.761 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:04.761 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:04.761 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:04.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.767+0000 7fdedf599640 10 monclient: shutdown 2024-03-27T21:40:04.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.767+0000 7fdedf599640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:04.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.767+0000 7fdedf599640 1 -- 172.21.15.60:0/1525706230 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fded80b40b0 msgr2=0x7fded80a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:04.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.767+0000 7fdedf599640 1 --2- 172.21.15.60:0/1525706230 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fded80b40b0 0x7fded80a8b90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fded40097d0 tx=0x7fded402b8a0 comp rx=0 tx=0).stop 2024-03-27T21:40:04.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.767+0000 7fdedf599640 1 -- 172.21.15.60:0/1525706230 shutdown_connections 2024-03-27T21:40:04.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.767+0000 7fdedf599640 1 --2- 172.21.15.60:0/1525706230 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fded80b40b0 0x7fded80a8b90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:04.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.767+0000 7fdedf599640 1 -- 172.21.15.60:0/1525706230 >> 172.21.15.60:0/1525706230 conn(0x7fded80608e0 msgr2=0x7fded8060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:04.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.767+0000 7fdedf599640 1 -- 172.21.15.60:0/1525706230 shutdown_connections 2024-03-27T21:40:04.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.767+0000 7fdedf599640 1 -- 172.21.15.60:0/1525706230 wait complete. 2024-03-27T21:40:04.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.768+0000 7fdedf599640 10 monclient: build_initial_monmap 2024-03-27T21:40:04.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.768+0000 7fdedf599640 10 monclient: monmap: 2024-03-27T21:40:04.767 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:04.767 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:04.767 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:04.767 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:04.767 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:04.767 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:04.767 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:04.767 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:04.767 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:04.767 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:04.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.768+0000 7fdedf599640 1 librados: starting msgr at 2024-03-27T21:40:04.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.768+0000 7fdedf599640 1 librados: starting objecter 2024-03-27T21:40:04.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.768+0000 7fdedf599640 1 Processor -- start 2024-03-27T21:40:04.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.768+0000 7fdedf599640 1 -- start start 2024-03-27T21:40:04.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.768+0000 7fdedf599640 1 librados: setting wanted keys 2024-03-27T21:40:04.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.768+0000 7fdedf599640 1 librados: calling monclient init 2024-03-27T21:40:04.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.768+0000 7fdedf599640 10 monclient: init 2024-03-27T21:40:04.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedf599640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:04.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedf599640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:40:04.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedf599640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fded80b40b0 0x7fded80afe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:04.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedf599640 10 monclient(hunting): picked mon.noname-c con 0x7fded80b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:04.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedf599640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded813b020 0x7fded8139000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:04.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedf599640 10 monclient(hunting): picked mon.noname-a con 0x7fded813b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:04.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedf599640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fded813b480 0x7fded8139540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:04.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedf599640 10 monclient(hunting): picked mon.noname-b con 0x7fded813b480 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:04.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedf599640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:04.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedf599640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded80ae7a0 con 0x7fded813b020 2024-03-27T21:40:04.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedf599640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:04.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedf599640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded80b06d0 con 0x7fded813b480 2024-03-27T21:40:04.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedf599640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:04.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedf599640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded80b03d0 con 0x7fded80b40b0 2024-03-27T21:40:04.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedf599640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:04.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedf599640 10 monclient(hunting): authenticate will time out at 845.989996s 2024-03-27T21:40:04.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedcb34640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded813b020 0x7fded8139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:04.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedd335640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fded80b40b0 0x7fded80afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:04.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedd335640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fded80b40b0 0x7fded80afe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:39434/0 (socket says 172.21.15.60:39434) 2024-03-27T21:40:04.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedcb34640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded813b020 0x7fded8139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52658/0 (socket says 172.21.15.60:52658) 2024-03-27T21:40:04.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedd335640 1 -- 172.21.15.60:0/2001375921 learned_addr learned my addr 172.21.15.60:0/2001375921 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:04.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedd335640 10 monclient(hunting): get_auth_request con 0x7fded80b40b0 auth_method 0 2024-03-27T21:40:04.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedd335640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:04.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedd335640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:04.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedd335640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:04.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedcb34640 10 monclient(hunting): get_auth_request con 0x7fded813b020 auth_method 0 2024-03-27T21:40:04.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedcb34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:04.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedcb34640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:04.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedcb34640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:04.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdeddb36640 1 --2- 172.21.15.60:0/2001375921 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fded813b480 0x7fded8139540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:04.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedcb34640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:04.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedcb34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:04.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedcb34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:04.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdeddb36640 10 monclient(hunting): get_auth_request con 0x7fded813b480 auth_method 0 2024-03-27T21:40:04.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdeddb36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:04.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdeddb36640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:04.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdeddb36640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:04.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedd335640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:04.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedd335640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:04.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedd335640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:04.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdeddb36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:04.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdeddb36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:04.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdeddb36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:04.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedcb34640 10 monclient(hunting): handle_auth_done global_id 4512 payload 824 2024-03-27T21:40:04.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedcb34640 1 -- 172.21.15.60:0/2001375921 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fded80b40b0 msgr2=0x7fded80afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:04.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedcb34640 1 --2- 172.21.15.60:0/2001375921 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fded80b40b0 0x7fded80afe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:04.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedcb34640 1 -- 172.21.15.60:0/2001375921 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fded813b480 msgr2=0x7fded8139540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:04.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedcb34640 1 --2- 172.21.15.60:0/2001375921 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fded813b480 0x7fded8139540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:04.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedcb34640 10 monclient: _finish_hunting 0 2024-03-27T21:40:04.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedcb34640 1 monclient: found mon.noname-a 2024-03-27T21:40:04.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedcb34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:04.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedcb34640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:04.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedcb34640 1 -- 172.21.15.60:0/2001375921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded4009400 con 0x7fded813b020 2024-03-27T21:40:04.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedd335640 1 --2- 172.21.15.60:0/2001375921 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fded80b40b0 0x7fded80afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:04.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedcb34640 1 --2- 172.21.15.60:0/2001375921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded813b020 0x7fded8139000 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdec800c440 tx=0x7fdec8001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:04.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdecdffb640 10 client.?.objecter ms_handle_connect 0x7fded813b020 2024-03-27T21:40:04.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdecdffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:04.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdecdffb640 1 -- 172.21.15.60:0/2001375921 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec8005ce0 con 0x7fded813b020 2024-03-27T21:40:04.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdecdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:04.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdecdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:04.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdecdffb640 10 monclient: dump: 2024-03-27T21:40:04.772 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:04.772 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:04.772 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.772 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.772 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:04.772 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:04.772 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:04.772 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:04.773 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:04.773 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:04.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdecdffb640 10 monclient: _finish_auth 0 2024-03-27T21:40:04.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdecdffb640 10 monclient: _check_auth_tickets 2024-03-27T21:40:04.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdecdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:04.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedf599640 5 monclient: authenticate success, global_id 4512 2024-03-27T21:40:04.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdecdffb640 1 -- 172.21.15.60:0/2001375921 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec8005e40 con 0x7fded813b020 2024-03-27T21:40:04.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdecdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:04.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdecdffb640 1 -- 172.21.15.60:0/2001375921 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec80072d0 con 0x7fded813b020 2024-03-27T21:40:04.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedf599640 10 monclient: _renew_subs 2024-03-27T21:40:04.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedf599640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:04.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdedf599640 1 -- 172.21.15.60:0/2001375921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded8139a80 con 0x7fded813b020 2024-03-27T21:40:04.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdecdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:04.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdecf7fe640 4 set_mon_vals no callback set 2024-03-27T21:40:04.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdecdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:04.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.769+0000 7fdecdffb640 10 monclient: dump: 2024-03-27T21:40:04.774 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:04.774 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:04.774 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.774 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.774 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:04.774 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:04.774 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:04.774 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:04.774 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:04.774 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:04.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.770+0000 7fdedf599640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:04.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.770+0000 7fdedf599640 10 monclient: _renew_subs 2024-03-27T21:40:04.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.770+0000 7fdedf599640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:04.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.770+0000 7fdedf599640 1 -- 172.21.15.60:0/2001375921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fded80af4e0 con 0x7fded813b020 2024-03-27T21:40:04.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.770+0000 7fdedf599640 1 librados: init done 2024-03-27T21:40:04.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.771+0000 7fdedf599640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:04.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.771+0000 7fdedf599640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:04.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.771+0000 7fdedf599640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:04.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.771+0000 7fdedf599640 1 -- 172.21.15.60:0/2001375921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fded80b0890 con 0x7fded813b020 2024-03-27T21:40:04.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.771+0000 7fdecdffb640 1 -- 172.21.15.60:0/2001375921 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fdec80074f0 con 0x7fded813b020 2024-03-27T21:40:04.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.771+0000 7fdecdffb640 1 --2- 172.21.15.60:0/2001375921 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdebc06c590 0x7fdebc06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:04.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.771+0000 7fdecdffb640 1 -- 172.21.15.60:0/2001375921 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fdec80885a0 con 0x7fded813b020 2024-03-27T21:40:04.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.771+0000 7fdecdffb640 10 client.4512.objecter ms_dispatch 0x7fded8000b90 osd_map(26..26 src has 1..26) v4 2024-03-27T21:40:04.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.771+0000 7fdecdffb640 3 client.4512.objecter handle_osd_map got epochs [26,26] > 0 2024-03-27T21:40:04.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.771+0000 7fdecdffb640 3 client.4512.objecter handle_osd_map decoding full epoch 26 2024-03-27T21:40:04.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.771+0000 7fdedd335640 1 --2- 172.21.15.60:0/2001375921 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdebc06c590 0x7fdebc06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:04.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.771+0000 7fdedd335640 10 monclient: get_auth_request con 0x7fdebc06c590 auth_method 0 2024-03-27T21:40:04.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.772+0000 7fdecdffb640 20 client.4512.objecter dump_active .. 0 homeless 2024-03-27T21:40:04.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.772+0000 7fdecdffb640 10 client.4512.objecter ms_handle_connect 0x7fdebc06c590 2024-03-27T21:40:04.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.772+0000 7fdedd335640 1 --2- 172.21.15.60:0/2001375921 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdebc06c590 0x7fdebc06ea50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fded402b8e0 tx=0x7fded403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:04.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.776+0000 7fdecdffb640 1 -- 172.21.15.60:0/2001375921 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdec8053a20 con 0x7fded813b020 2024-03-27T21:40:04.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.776+0000 7fdecdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:04.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.776+0000 7fdecdffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:04.778 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:04.793+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:04.827 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:40:04.839 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:04.839 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:04.847 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:40:04.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.882+0000 7fcc385e7640 10 monclient: get_monmap_and_config 2024-03-27T21:40:04.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.882+0000 7fcc385e7640 10 monclient: build_initial_monmap 2024-03-27T21:40:04.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.882+0000 7fcc385e7640 10 monclient: monmap: 2024-03-27T21:40:04.868 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:04.868 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:04.868 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:04.883141+0000 2024-03-27T21:40:04.869 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:04.883141+0000 2024-03-27T21:40:04.869 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:04.869 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:04.869 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:04.869 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:04.869 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:04.869 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:04.869 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:04.869 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:04.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.882+0000 7fcc385e7640 1 Processor -- start 2024-03-27T21:40:04.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.882+0000 7fcc385e7640 1 -- start start 2024-03-27T21:40:04.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.882+0000 7fcc385e7640 10 monclient: init 2024-03-27T21:40:04.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.883+0000 7fcc385e7640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:04.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.883+0000 7fcc385e7640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-03-27T21:40:04.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.883+0000 7fcc385e7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc3006f900 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:04.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.883+0000 7fcc385e7640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcc3006fce0 0x7fcc3013fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:04.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.883+0000 7fcc385e7640 10 monclient(hunting): picked mon.noname-b con 0x7fcc3006fce0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:04.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.883+0000 7fcc385e7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcc30138bf0 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:04.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.883+0000 7fcc385e7640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc30071b50 con 0x7fcc30138bf0 2024-03-27T21:40:04.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.883+0000 7fcc385e7640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc3013f690 con 0x7fcc3006f900 2024-03-27T21:40:04.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.883+0000 7fcc385e7640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:04.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.883+0000 7fcc385e7640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc30140100 con 0x7fcc3006fce0 2024-03-27T21:40:04.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.883+0000 7fcc385e7640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:04.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.883+0000 7fcc385e7640 10 monclient(hunting): authenticate will time out at 846.104585s 2024-03-27T21:40:04.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.883+0000 7fcc36b84640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcc30138bf0 0x7fcc30139080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42028/0 (socket says 172.21.15.60:42028) 2024-03-27T21:40:04.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.883+0000 7fcc36b84640 1 -- 172.21.15.60:0/204050860 learned_addr learned my addr 172.21.15.60:0/204050860 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:40:04.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.883+0000 7fcc35b82640 1 --2- 172.21.15.60:0/204050860 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcc3006fce0 0x7fcc3013fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:04.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.883+0000 7fcc35b82640 10 monclient(hunting): get_auth_request con 0x7fcc3006fce0 auth_method 0 2024-03-27T21:40:04.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.883+0000 7fcc35b82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:04.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.883+0000 7fcc35b82640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:04.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.883+0000 7fcc35b82640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:04.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.884+0000 7fcc35b82640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:04.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.884+0000 7fcc35b82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:04.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.884+0000 7fcc35b82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:04.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.884+0000 7fcc35b82640 10 monclient(hunting): handle_auth_done global_id 4433 payload 274 2024-03-27T21:40:04.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.884+0000 7fcc35b82640 1 -- 172.21.15.60:0/204050860 >> v1:172.21.15.191:6789/0 conn(0x7fcc3006f900 legacy=0x7fcc300af850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:04.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.884+0000 7fcc35b82640 1 -- 172.21.15.60:0/204050860 >> v1:172.21.15.60:6789/0 conn(0x7fcc30138bf0 legacy=0x7fcc30139080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:04.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.884+0000 7fcc35b82640 10 monclient: _finish_hunting 0 2024-03-27T21:40:04.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.884+0000 7fcc35b82640 1 monclient: found mon.noname-b 2024-03-27T21:40:04.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.884+0000 7fcc35b82640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:04.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.884+0000 7fcc35b82640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:04.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.884+0000 7fcc35b82640 1 -- 172.21.15.60:0/204050860 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3006f130 con 0x7fcc3006fce0 2024-03-27T21:40:04.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.884+0000 7fcc35381640 1 -- 172.21.15.60:0/204050860 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101540129 0 0) 0x7fcc3013f690 con 0x7fcc3006f900 2024-03-27T21:40:04.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.884+0000 7fcc35381640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:40:04.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.885+0000 7fcc35b82640 1 --2- 172.21.15.60:0/204050860 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcc3006fce0 0x7fcc3013fbc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcc200097d0 tx=0x7fcc2002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f4d57d2a04dc459 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:04.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.885+0000 7fcc35381640 1 -- 172.21.15.60:0/204050860 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc20031890 con 0x7fcc3006fce0 2024-03-27T21:40:04.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.885+0000 7fcc35381640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:04.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.885+0000 7fcc35381640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:04.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.885+0000 7fcc35381640 10 monclient: dump: 2024-03-27T21:40:04.880 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:04.880 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:04.880 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.880 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.880 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:04.880 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:04.880 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:04.881 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:04.881 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:04.881 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:04.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.885+0000 7fcc35381640 10 monclient: _finish_auth 0 2024-03-27T21:40:04.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.885+0000 7fcc35381640 10 monclient: _check_auth_tickets 2024-03-27T21:40:04.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.885+0000 7fcc35381640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:04.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.885+0000 7fcc35381640 1 -- 172.21.15.60:0/204050860 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc200319f0 con 0x7fcc3006fce0 2024-03-27T21:40:04.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.885+0000 7fcc385e7640 5 monclient: authenticate success, global_id 4433 2024-03-27T21:40:04.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.885+0000 7fcc385e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:04.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.885+0000 7fcc35381640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:04.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.885+0000 7fcc385e7640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:04.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.885+0000 7fcc35381640 1 -- 172.21.15.60:0/204050860 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc20031d80 con 0x7fcc3006fce0 2024-03-27T21:40:04.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.885+0000 7fcc35381640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:04.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.885+0000 7fcc385e7640 4 set_mon_vals no callback set 2024-03-27T21:40:04.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.885+0000 7fcc35381640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:04.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.885+0000 7fcc35381640 10 monclient: dump: 2024-03-27T21:40:04.882 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:04.882 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:04.882 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.882 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.882 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:04.882 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:04.882 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:04.882 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:04.882 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:04.882 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:04.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.886+0000 7fcc385e7640 10 monclient: shutdown 2024-03-27T21:40:04.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.886+0000 7fcc385e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:04.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.886+0000 7fcc385e7640 1 -- 172.21.15.60:0/204050860 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcc3006fce0 msgr2=0x7fcc3013fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:04.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.886+0000 7fcc385e7640 1 --2- 172.21.15.60:0/204050860 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcc3006fce0 0x7fcc3013fbc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcc200097d0 tx=0x7fcc2002b930 comp rx=0 tx=0).stop 2024-03-27T21:40:04.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.886+0000 7fcc385e7640 1 -- 172.21.15.60:0/204050860 shutdown_connections 2024-03-27T21:40:04.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.886+0000 7fcc385e7640 1 --2- 172.21.15.60:0/204050860 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcc3006fce0 0x7fcc3013fbc0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:04.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.886+0000 7fcc385e7640 1 -- 172.21.15.60:0/204050860 >> 172.21.15.60:0/204050860 conn(0x7fcc300b4e50 msgr2=0x7fcc300b5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:04.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.886+0000 7fcc385e7640 1 -- 172.21.15.60:0/204050860 shutdown_connections 2024-03-27T21:40:04.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.886+0000 7fcc385e7640 1 -- 172.21.15.60:0/204050860 wait complete. 2024-03-27T21:40:04.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.887+0000 7fcc385e7640 10 monclient: build_initial_monmap 2024-03-27T21:40:04.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.887+0000 7fcc385e7640 10 monclient: monmap: 2024-03-27T21:40:04.883 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:04.883 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:04.883 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:04.883 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:04.883 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:04.884 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:04.884 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:04.884 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:04.884 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:04.884 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:04.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.887+0000 7fcc385e7640 1 librados: starting msgr at 2024-03-27T21:40:04.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.887+0000 7fcc385e7640 1 librados: starting objecter 2024-03-27T21:40:04.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.887+0000 7fcc385e7640 1 Processor -- start 2024-03-27T21:40:04.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc385e7640 1 -- start start 2024-03-27T21:40:04.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc385e7640 1 librados: setting wanted keys 2024-03-27T21:40:04.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc385e7640 1 librados: calling monclient init 2024-03-27T21:40:04.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc385e7640 10 monclient: init 2024-03-27T21:40:04.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc385e7640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:04.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc385e7640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:40:04.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc385e7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc3006f900 0x7fcc300a9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:04.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc385e7640 10 monclient(hunting): picked mon.noname-b con 0x7fcc3006f900 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:04.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc385e7640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcc3006fce0 0x7fcc300ab720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:04.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc385e7640 10 monclient(hunting): picked mon.noname-c con 0x7fcc3006fce0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:04.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc385e7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc30138bf0 0x7fcc30131c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:04.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc385e7640 10 monclient(hunting): picked mon.noname-a con 0x7fcc30138bf0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:04.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc385e7640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:04.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc385e7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc300a9fd0 con 0x7fcc30138bf0 2024-03-27T21:40:04.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc385e7640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:04.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc385e7640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc300abc60 con 0x7fcc3006f900 2024-03-27T21:40:04.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc385e7640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:04.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc385e7640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc300ab020 con 0x7fcc3006fce0 2024-03-27T21:40:04.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc385e7640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:04.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc385e7640 10 monclient(hunting): authenticate will time out at 846.109483s 2024-03-27T21:40:04.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc35b82640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcc3006fce0 0x7fcc300ab720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:04.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc35b82640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcc3006fce0 0x7fcc300ab720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:39442/0 (socket says 172.21.15.60:39442) 2024-03-27T21:40:04.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc35b82640 1 -- 172.21.15.60:0/2177603658 learned_addr learned my addr 172.21.15.60:0/2177603658 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:04.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc35b82640 10 monclient(hunting): get_auth_request con 0x7fcc3006fce0 auth_method 0 2024-03-27T21:40:04.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc35b82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:04.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc35b82640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:04.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc35b82640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:04.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc36b84640 1 --2- 172.21.15.60:0/2177603658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc30138bf0 0x7fcc30131c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:04.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc36383640 1 --2- 172.21.15.60:0/2177603658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc3006f900 0x7fcc300a9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:04.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc36b84640 10 monclient(hunting): get_auth_request con 0x7fcc30138bf0 auth_method 0 2024-03-27T21:40:04.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc36b84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:04.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc36b84640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:04.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc36b84640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:04.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc36383640 10 monclient(hunting): get_auth_request con 0x7fcc3006f900 auth_method 0 2024-03-27T21:40:04.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc36383640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:04.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc36383640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:04.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc36383640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:04.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc35b82640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:04.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc35b82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:04.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc35b82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:04.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc36b84640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:04.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc36b84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:04.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc36b84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:04.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc36383640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:04.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc36383640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:04.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.888+0000 7fcc36383640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:04.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc36383640 10 monclient(hunting): handle_auth_done global_id 4495 payload 824 2024-03-27T21:40:04.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc36383640 1 -- 172.21.15.60:0/2177603658 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcc3006fce0 msgr2=0x7fcc300ab720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:04.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc36383640 1 --2- 172.21.15.60:0/2177603658 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcc3006fce0 0x7fcc300ab720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:04.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc36383640 1 -- 172.21.15.60:0/2177603658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc30138bf0 msgr2=0x7fcc30131c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:04.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc36383640 1 --2- 172.21.15.60:0/2177603658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc30138bf0 0x7fcc30131c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:04.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc36383640 10 monclient: _finish_hunting 0 2024-03-27T21:40:04.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc36383640 1 monclient: found mon.noname-b 2024-03-27T21:40:04.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc36383640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:04.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc36383640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:40:04.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc36383640 1 -- 172.21.15.60:0/2177603658 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc20009400 con 0x7fcc3006f900 2024-03-27T21:40:04.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc36b84640 1 --2- 172.21.15.60:0/2177603658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc30138bf0 0x7fcc30131c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:04.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc35b82640 1 --2- 172.21.15.60:0/2177603658 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcc3006fce0 0x7fcc300ab720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:04.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc36383640 1 --2- 172.21.15.60:0/2177603658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc3006f900 0x7fcc300a9940 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c00c450 tx=0x7fcc2c00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:04.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc26ffd640 10 client.?.objecter ms_handle_connect 0x7fcc3006f900 2024-03-27T21:40:04.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc26ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:04.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc26ffd640 1 -- 172.21.15.60:0/2177603658 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c0035f0 con 0x7fcc3006f900 2024-03-27T21:40:04.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:04.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc26ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:04.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc26ffd640 10 monclient: dump: 2024-03-27T21:40:04.889 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:04.889 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:04.889 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.889 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.889 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:04.889 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:04.889 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:04.889 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:04.889 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:04.889 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:04.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc26ffd640 10 monclient: _finish_auth 0 2024-03-27T21:40:04.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc26ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:40:04.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:04.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc385e7640 5 monclient: authenticate success, global_id 4495 2024-03-27T21:40:04.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc26ffd640 1 -- 172.21.15.60:0/2177603658 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2c003750 con 0x7fcc3006f900 2024-03-27T21:40:04.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:04.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc26ffd640 1 -- 172.21.15.60:0/2177603658 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c003b00 con 0x7fcc3006f900 2024-03-27T21:40:04.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc385e7640 10 monclient: _renew_subs 2024-03-27T21:40:04.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc385e7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:04.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc385e7640 1 -- 172.21.15.60:0/2177603658 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc300ab300 con 0x7fcc3006f900 2024-03-27T21:40:04.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc34b80640 4 set_mon_vals no callback set 2024-03-27T21:40:04.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:04.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc26ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:40:04.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc26ffd640 10 monclient: dump: 2024-03-27T21:40:04.891 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:04.891 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:04.891 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.891 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.891 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:04.891 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:04.891 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:04.891 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:04.891 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:04.891 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:04.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc385e7640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:04.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc385e7640 10 monclient: _renew_subs 2024-03-27T21:40:04.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc385e7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:04.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc385e7640 1 -- 172.21.15.60:0/2177603658 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc300a8ca0 con 0x7fcc3006f900 2024-03-27T21:40:04.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.889+0000 7fcc385e7640 1 librados: init done 2024-03-27T21:40:04.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.890+0000 7fcc385e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:04.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.890+0000 7fcc385e7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:04.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.890+0000 7fcc385e7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:04.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.890+0000 7fcc385e7640 1 -- 172.21.15.60:0/2177603658 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc300aa140 con 0x7fcc3006f900 2024-03-27T21:40:04.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.891+0000 7fcc26ffd640 1 -- 172.21.15.60:0/2177603658 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcc2c003c60 con 0x7fcc3006f900 2024-03-27T21:40:04.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.891+0000 7fcc26ffd640 1 --2- 172.21.15.60:0/2177603658 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcc1406c5c0 0x7fcc1406ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:04.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.892+0000 7fcc26ffd640 1 -- 172.21.15.60:0/2177603658 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fcc2c0886a0 con 0x7fcc3006f900 2024-03-27T21:40:04.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.892+0000 7fcc26ffd640 10 client.4495.objecter ms_dispatch 0x7fcc30000b90 osd_map(26..26 src has 1..26) v4 2024-03-27T21:40:04.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.892+0000 7fcc26ffd640 3 client.4495.objecter handle_osd_map got epochs [26,26] > 0 2024-03-27T21:40:04.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.892+0000 7fcc26ffd640 3 client.4495.objecter handle_osd_map decoding full epoch 26 2024-03-27T21:40:04.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.892+0000 7fcc35b82640 1 --2- 172.21.15.60:0/2177603658 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcc1406c5c0 0x7fcc1406ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:04.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.892+0000 7fcc35b82640 10 monclient: get_auth_request con 0x7fcc1406c5c0 auth_method 0 2024-03-27T21:40:04.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.892+0000 7fcc26ffd640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-27T21:40:04.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.892+0000 7fcc26ffd640 10 client.4495.objecter ms_handle_connect 0x7fcc1406c5c0 2024-03-27T21:40:04.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.892+0000 7fcc35b82640 1 --2- 172.21.15.60:0/2177603658 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcc1406c5c0 0x7fcc1406ea80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcc20000c00 tx=0x7fcc2003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:04.894 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:04.903+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:04.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.896+0000 7fcc26ffd640 1 -- 172.21.15.60:0/2177603658 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcc2c053b20 con 0x7fcc3006f900 2024-03-27T21:40:04.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.896+0000 7fcc26ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:04.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.896+0000 7fcc26ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:04.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.944+0000 7fc799527640 10 monclient: get_monmap_and_config 2024-03-27T21:40:04.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.944+0000 7fc799527640 10 monclient: build_initial_monmap 2024-03-27T21:40:04.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.945+0000 7fc799527640 10 monclient: monmap: 2024-03-27T21:40:04.941 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:04.941 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:04.941 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:04.945997+0000 2024-03-27T21:40:04.941 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:04.945997+0000 2024-03-27T21:40:04.941 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:04.941 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:04.941 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:04.941 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:04.941 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:04.941 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:04.941 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:04.941 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:04.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.945+0000 7fc799527640 1 Processor -- start 2024-03-27T21:40:04.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.945+0000 7fc799527640 1 -- start start 2024-03-27T21:40:04.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.945+0000 7fc799527640 10 monclient: init 2024-03-27T21:40:04.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc799527640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:04.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc799527640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-03-27T21:40:04.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc799527640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc7940b40b0 0x7fc7940a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:04.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc799527640 10 monclient(hunting): picked mon.noname-b con 0x7fc7940b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:04.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc799527640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc79413b020 0x7fc79413b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:04.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc799527640 10 monclient(hunting): picked mon.noname-a con 0x7fc79413b020 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:04.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc799527640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc7941416f0 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:04.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc799527640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7940b3830 con 0x7fc7941416f0 2024-03-27T21:40:04.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc799527640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:04.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc799527640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7940b4490 con 0x7fc79413b020 2024-03-27T21:40:04.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc799527640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:04.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc799527640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc794142290 con 0x7fc7940b40b0 2024-03-27T21:40:04.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc799527640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:04.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc799527640 10 monclient(hunting): authenticate will time out at 846.167179s 2024-03-27T21:40:04.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc7927fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc79413b020 0x7fc79413b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:04.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc792ffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc7940b40b0 0x7fc7940a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:04.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc792ffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc7940b40b0 0x7fc7940a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:39446/0 (socket says 172.21.15.60:39446) 2024-03-27T21:40:04.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc7937fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc7941416f0 0x7fc794141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42034/0 (socket says 172.21.15.60:42034) 2024-03-27T21:40:04.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc792ffd640 1 -- 172.21.15.60:0/2314127884 learned_addr learned my addr 172.21.15.60:0/2314127884 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:04.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc792ffd640 10 monclient(hunting): get_auth_request con 0x7fc7940b40b0 auth_method 0 2024-03-27T21:40:04.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc792ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:04.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc792ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:04.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc792ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:04.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc7927fc640 10 monclient(hunting): get_auth_request con 0x7fc79413b020 auth_method 0 2024-03-27T21:40:04.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc7927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:04.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc7927fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:04.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc7927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:04.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc7927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:04.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc7927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:04.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc7927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:04.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc792ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:04.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc792ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:04.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc792ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:04.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc7927fc640 10 monclient(hunting): handle_auth_done global_id 4518 payload 274 2024-03-27T21:40:04.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc7927fc640 1 -- 172.21.15.60:0/2314127884 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc7940b40b0 msgr2=0x7fc7940a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:04.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc7927fc640 1 --2- 172.21.15.60:0/2314127884 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc7940b40b0 0x7fc7940a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:04.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc7927fc640 1 -- 172.21.15.60:0/2314127884 >> v1:172.21.15.60:6789/0 conn(0x7fc7941416f0 legacy=0x7fc794141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:04.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc7927fc640 10 monclient: _finish_hunting 0 2024-03-27T21:40:04.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc7927fc640 1 monclient: found mon.noname-a 2024-03-27T21:40:04.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc7927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:04.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc7927fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:04.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc7927fc640 1 -- 172.21.15.60:0/2314127884 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7940aa520 con 0x7fc79413b020 2024-03-27T21:40:04.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.946+0000 7fc792ffd640 1 --2- 172.21.15.60:0/2314127884 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc7940b40b0 0x7fc7940a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:04.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.947+0000 7fc7927fc640 1 --2- 172.21.15.60:0/2314127884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc79413b020 0x7fc79413b480 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc7880097a0 tx=0x7fc78802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=252fd8a4bbb1767e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:04.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.947+0000 7fc791ffb640 1 -- 172.21.15.60:0/2314127884 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78802d040 con 0x7fc79413b020 2024-03-27T21:40:04.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.947+0000 7fc791ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:04.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.947+0000 7fc791ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:04.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.947+0000 7fc791ffb640 10 monclient: dump: 2024-03-27T21:40:04.945 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:04.945 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:04.945 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.946 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.946 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:04.946 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:04.946 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:04.946 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:04.946 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:04.946 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:04.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.947+0000 7fc791ffb640 10 monclient: _finish_auth 0 2024-03-27T21:40:04.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.947+0000 7fc791ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:40:04.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.947+0000 7fc791ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:04.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.947+0000 7fc791ffb640 1 -- 172.21.15.60:0/2314127884 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc788031810 con 0x7fc79413b020 2024-03-27T21:40:04.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.947+0000 7fc799527640 5 monclient: authenticate success, global_id 4518 2024-03-27T21:40:04.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.947+0000 7fc799527640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:04.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.947+0000 7fc791ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:04.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.947+0000 7fc791ffb640 1 -- 172.21.15.60:0/2314127884 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc788031ba0 con 0x7fc79413b020 2024-03-27T21:40:04.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.947+0000 7fc791ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:04.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.947+0000 7fc791ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:04.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.947+0000 7fc791ffb640 10 monclient: dump: 2024-03-27T21:40:04.947 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:04.947 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:04.947 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.947 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.947 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:04.947 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:04.947 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:04.947 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:04.947 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:04.948 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.947+0000 7fc799527640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.947+0000 7fc799527640 4 set_mon_vals no callback set 2024-03-27T21:40:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.948+0000 7fc799527640 10 monclient: shutdown 2024-03-27T21:40:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.948+0000 7fc799527640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.948+0000 7fc799527640 1 -- 172.21.15.60:0/2314127884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc79413b020 msgr2=0x7fc79413b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.948+0000 7fc799527640 1 --2- 172.21.15.60:0/2314127884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc79413b020 0x7fc79413b480 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc7880097a0 tx=0x7fc78802b8b0 comp rx=0 tx=0).stop 2024-03-27T21:40:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.948+0000 7fc799527640 1 -- 172.21.15.60:0/2314127884 shutdown_connections 2024-03-27T21:40:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.948+0000 7fc799527640 1 --2- 172.21.15.60:0/2314127884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc79413b020 0x7fc79413b480 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.948+0000 7fc799527640 1 --2- 172.21.15.60:0/2314127884 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc7940b40b0 0x7fc7940a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.948+0000 7fc799527640 1 -- 172.21.15.60:0/2314127884 >> 172.21.15.60:0/2314127884 conn(0x7fc7940608e0 msgr2=0x7fc794060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.948+0000 7fc799527640 1 -- 172.21.15.60:0/2314127884 shutdown_connections 2024-03-27T21:40:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.948+0000 7fc799527640 1 -- 172.21.15.60:0/2314127884 wait complete. 2024-03-27T21:40:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.949+0000 7fc799527640 10 monclient: build_initial_monmap 2024-03-27T21:40:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.949+0000 7fc799527640 10 monclient: monmap: 2024-03-27T21:40:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:04.949 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.950+0000 7fc799527640 1 librados: starting msgr at 2024-03-27T21:40:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.950+0000 7fc799527640 1 librados: starting objecter 2024-03-27T21:40:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.950+0000 7fc799527640 1 Processor -- start 2024-03-27T21:40:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.950+0000 7fc799527640 1 -- start start 2024-03-27T21:40:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.950+0000 7fc799527640 1 librados: setting wanted keys 2024-03-27T21:40:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.950+0000 7fc799527640 1 librados: calling monclient init 2024-03-27T21:40:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.950+0000 7fc799527640 10 monclient: init 2024-03-27T21:40:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc799527640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc799527640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:40:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc799527640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7940b40b0 0x7fc7941337c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc799527640 10 monclient(hunting): picked mon.noname-b con 0x7fc7940b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc799527640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc79413b020 0x7fc7941310a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc799527640 10 monclient(hunting): picked mon.noname-c con 0x7fc79413b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc799527640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7941416f0 0x7fc7941315e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc799527640 10 monclient(hunting): picked mon.noname-a con 0x7fc7941416f0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc799527640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc799527640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79413be60 con 0x7fc7941416f0 2024-03-27T21:40:04.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc799527640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:04.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc799527640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79413c2f0 con 0x7fc7940b40b0 2024-03-27T21:40:04.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc799527640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:04.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc799527640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc794131b20 con 0x7fc79413b020 2024-03-27T21:40:04.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc799527640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:04.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc7937fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7941416f0 0x7fc7941315e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:04.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc7927fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc79413b020 0x7fc7941310a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:04.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc7937fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7941416f0 0x7fc7941315e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52684/0 (socket says 172.21.15.60:52684) 2024-03-27T21:40:04.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc7937fe640 1 -- 172.21.15.60:0/4035379115 learned_addr learned my addr 172.21.15.60:0/4035379115 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:04.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc792ffd640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7940b40b0 0x7fc7941337c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:04.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc799527640 10 monclient(hunting): authenticate will time out at 846.172598s 2024-03-27T21:40:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc7927fc640 10 monclient(hunting): get_auth_request con 0x7fc79413b020 auth_method 0 2024-03-27T21:40:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc7927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc7927fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc7927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc7937fe640 10 monclient(hunting): get_auth_request con 0x7fc7941416f0 auth_method 0 2024-03-27T21:40:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc7937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc7937fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc7937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc792ffd640 10 monclient(hunting): get_auth_request con 0x7fc7940b40b0 auth_method 0 2024-03-27T21:40:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc792ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc792ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc792ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc7927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc7927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc7927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc7937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc7937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc7937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc792ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc792ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.951+0000 7fc792ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc792ffd640 10 monclient(hunting): handle_auth_done global_id 4498 payload 824 2024-03-27T21:40:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc792ffd640 1 -- 172.21.15.60:0/4035379115 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc79413b020 msgr2=0x7fc7941310a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc792ffd640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc79413b020 0x7fc7941310a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc792ffd640 1 -- 172.21.15.60:0/4035379115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7941416f0 msgr2=0x7fc7941315e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc792ffd640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7941416f0 0x7fc7941315e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc792ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:40:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc792ffd640 1 monclient: found mon.noname-b 2024-03-27T21:40:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc792ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:04.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc792ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:40:04.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc792ffd640 1 -- 172.21.15.60:0/4035379115 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7840094d0 con 0x7fc7940b40b0 2024-03-27T21:40:04.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc7927fc640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc79413b020 0x7fc7941310a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:04.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc7937fe640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7941416f0 0x7fc7941315e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:04.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc792ffd640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7940b40b0 0x7fc7941337c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc78400e900 tx=0x7fc78400edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:04.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc77b7fe640 10 client.?.objecter ms_handle_connect 0x7fc7940b40b0 2024-03-27T21:40:04.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc77b7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:04.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc77b7fe640 1 -- 172.21.15.60:0/4035379115 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78400ca00 con 0x7fc7940b40b0 2024-03-27T21:40:04.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc77b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:04.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc77b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:04.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc77b7fe640 10 monclient: dump: 2024-03-27T21:40:04.954 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:04.954 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:04.955 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.955 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.955 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:04.955 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:04.955 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:04.955 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:04.955 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:04.955 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:04.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc77b7fe640 10 monclient: _finish_auth 0 2024-03-27T21:40:04.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc77b7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:40:04.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc77b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:04.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc77b7fe640 1 -- 172.21.15.60:0/4035379115 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc78400cb60 con 0x7fc7940b40b0 2024-03-27T21:40:04.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc799527640 5 monclient: authenticate success, global_id 4498 2024-03-27T21:40:04.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc77b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:04.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc77b7fe640 1 -- 172.21.15.60:0/4035379115 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc784004030 con 0x7fc7940b40b0 2024-03-27T21:40:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc799527640 10 monclient: _renew_subs 2024-03-27T21:40:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc799527640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc799527640 1 -- 172.21.15.60:0/4035379115 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc788009400 con 0x7fc7940b40b0 2024-03-27T21:40:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc7917fa640 4 set_mon_vals no callback set 2024-03-27T21:40:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc77b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc77b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:40:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc77b7fe640 10 monclient: dump: 2024-03-27T21:40:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:04.957 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc799527640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc799527640 10 monclient: _renew_subs 2024-03-27T21:40:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc799527640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc799527640 1 -- 172.21.15.60:0/4035379115 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc79413adc0 con 0x7fc7940b40b0 2024-03-27T21:40:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.952+0000 7fc799527640 1 librados: init done 2024-03-27T21:40:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.953+0000 7fc799527640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.953+0000 7fc799527640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.953+0000 7fc799527640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.953+0000 7fc799527640 1 -- 172.21.15.60:0/4035379115 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fc794069470 con 0x7fc7940b40b0 2024-03-27T21:40:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.954+0000 7fc77b7fe640 1 -- 172.21.15.60:0/4035379115 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc784004190 con 0x7fc7940b40b0 2024-03-27T21:40:04.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc77b7fe640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc78006c590 0x7fc78006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:04.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc7927fc640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc78006c590 0x7fc78006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:04.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc77b7fe640 1 -- 172.21.15.60:0/4035379115 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc784089140 con 0x7fc7940b40b0 2024-03-27T21:40:04.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc77b7fe640 10 client.4498.objecter ms_dispatch 0x7fc794000b90 osd_map(26..26 src has 1..26) v4 2024-03-27T21:40:04.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc77b7fe640 3 client.4498.objecter handle_osd_map got epochs [26,26] > 0 2024-03-27T21:40:04.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc77b7fe640 3 client.4498.objecter handle_osd_map decoding full epoch 26 2024-03-27T21:40:04.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc7927fc640 10 monclient: get_auth_request con 0x7fc78006c590 auth_method 0 2024-03-27T21:40:04.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc77b7fe640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-27T21:40:04.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc77b7fe640 1 -- 172.21.15.60:0/4035379115 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v26) v1 ==== 54+0+16 (secure 0 0 0) 0x7fc784090070 con 0x7fc7940b40b0 2024-03-27T21:40:04.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc77b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:04.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc77b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:04.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc7797fa640 10 client.4498.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:04.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign -1 1 2024-03-27T21:40:04.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc7797fa640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fc79413c550 0x7fc794138840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:04.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc77b7fe640 10 client.4498.objecter ms_handle_connect 0x7fc78006c590 2024-03-27T21:40:04.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc7927fc640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc78006c590 0x7fc78006ea50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc788036000 tx=0x7fc78803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:04.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc7940a90d0 osd=0 3 2024-03-27T21:40:04.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc7940a90d0 osd=0 3 2024-03-27T21:40:04.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc7940a90d0 osd=0 3 2024-03-27T21:40:04.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_remove -1 1 2024-03-27T21:40:04.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc7797fa640 20 client.4498.objecter get_session s=0x7fc7940a90d0 osd=0 3 2024-03-27T21:40:04.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign 0 1 2024-03-27T21:40:04.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc7940a90d0 osd=0 4 2024-03-27T21:40:04.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc7797fa640 10 client.4498.objecter _send_command 1 2024-03-27T21:40:04.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc7797fa640 1 -- 172.21.15.60:0/4035379115 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc79413bb70 con 0x7fc79413c550 2024-03-27T21:40:04.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc7937fe640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fc79413c550 0x7fc794138840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:04.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.955+0000 7fc7937fe640 10 monclient: get_auth_request con 0x7fc79413c550 auth_method 0 2024-03-27T21:40:04.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.956+0000 7fc77b7fe640 10 client.4498.objecter ms_handle_connect 0x7fc79413c550 2024-03-27T21:40:04.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.956+0000 7fc7937fe640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fc79413c550 0x7fc794138840 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:04.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.957+0000 7fc77b7fe640 1 -- 172.21.15.60:0/4035379115 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fc79413bb70 con 0x7fc79413c550 2024-03-27T21:40:04.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.957+0000 7fc77b7fe640 10 client.4498.objecter ms_dispatch 0x7fc794000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:04.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.957+0000 7fc77b7fe640 10 client.4498.objecter _finish_command 1 = system:0 2024-03-27T21:40:04.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.957+0000 7fc77b7fe640 20 client.4498.objecter put_session s=0x7fc7940a90d0 osd=0 4 2024-03-27T21:40:04.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.957+0000 7fc77b7fe640 15 client.4498.objecter _session_command_op_remove 0 1 2024-03-27T21:40:04.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.972+0000 7fc7797fa640 10 client.4498.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:04.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.972+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign -1 2 2024-03-27T21:40:04.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.972+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc7940a90d0 osd=0 3 2024-03-27T21:40:04.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.972+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc7940a90d0 osd=0 3 2024-03-27T21:40:04.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.972+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc7940a90d0 osd=0 3 2024-03-27T21:40:04.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.972+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_remove -1 2 2024-03-27T21:40:04.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.972+0000 7fc7797fa640 20 client.4498.objecter get_session s=0x7fc7940a90d0 osd=0 3 2024-03-27T21:40:04.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.972+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign 0 2 2024-03-27T21:40:04.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.972+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc7940a90d0 osd=0 4 2024-03-27T21:40:04.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.972+0000 7fc7797fa640 10 client.4498.objecter _send_command 2 2024-03-27T21:40:04.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.972+0000 7fc7797fa640 1 -- 172.21.15.60:0/4035379115 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc79414bd20 con 0x7fc79413c550 2024-03-27T21:40:04.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.973+0000 7fc77b7fe640 1 -- 172.21.15.60:0/4035379115 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc79414bd20 con 0x7fc79413c550 2024-03-27T21:40:04.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.973+0000 7fc77b7fe640 10 client.4498.objecter ms_dispatch 0x7fc794000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:04.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.973+0000 7fc77b7fe640 10 client.4498.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:04.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.973+0000 7fc77b7fe640 20 client.4498.objecter put_session s=0x7fc7940a90d0 osd=0 4 2024-03-27T21:40:04.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.973+0000 7fc77b7fe640 15 client.4498.objecter _session_command_op_remove 0 2 2024-03-27T21:40:04.963 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:04.965 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:04.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.974+0000 7fc7797fa640 10 client.4498.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:04.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.974+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign -1 3 2024-03-27T21:40:04.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.974+0000 7fc7797fa640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fc7940b4bc0 0x7fc7940b4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:04.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.974+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc794139540 osd=1 3 2024-03-27T21:40:04.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.974+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc794139540 osd=1 3 2024-03-27T21:40:04.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.974+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc794139540 osd=1 3 2024-03-27T21:40:04.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.974+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_remove -1 3 2024-03-27T21:40:04.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.974+0000 7fc7797fa640 20 client.4498.objecter get_session s=0x7fc794139540 osd=1 3 2024-03-27T21:40:04.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.974+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign 1 3 2024-03-27T21:40:04.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.974+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc794139540 osd=1 4 2024-03-27T21:40:04.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.974+0000 7fc7797fa640 10 client.4498.objecter _send_command 3 2024-03-27T21:40:04.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.974+0000 7fc7797fa640 1 -- 172.21.15.60:0/4035379115 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc7941396b0 con 0x7fc7940b4bc0 2024-03-27T21:40:04.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.974+0000 7fc792ffd640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fc7940b4bc0 0x7fc7940b4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:04.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.974+0000 7fc792ffd640 10 monclient: get_auth_request con 0x7fc7940b4bc0 auth_method 0 2024-03-27T21:40:04.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.974+0000 7fc792ffd640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fc7940b4bc0 0x7fc7940b4fa0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:04.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.974+0000 7fc77b7fe640 10 client.4498.objecter ms_handle_connect 0x7fc7940b4bc0 2024-03-27T21:40:04.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.975+0000 7fc77b7fe640 1 -- 172.21.15.60:0/4035379115 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fc7941396b0 con 0x7fc7940b4bc0 2024-03-27T21:40:04.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.975+0000 7fc77b7fe640 10 client.4498.objecter ms_dispatch 0x7fc794000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:04.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.975+0000 7fc77b7fe640 10 client.4498.objecter _finish_command 3 = system:0 2024-03-27T21:40:04.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.975+0000 7fc77b7fe640 20 client.4498.objecter put_session s=0x7fc794139540 osd=1 4 2024-03-27T21:40:04.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:04.975+0000 7fc77b7fe640 15 client.4498.objecter _session_command_op_remove 1 3 2024-03-27T21:40:04.967 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:04.979+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:04.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.002+0000 7fdedf599640 10 start_mon_command cmd=[{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:40:04.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.002+0000 7fdedf599640 10 monclient: _send_command 2 [{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:40:04.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.002+0000 7fdedf599640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:04.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.002+0000 7fdedf599640 1 -- 172.21.15.60:0/2001375921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fded8069470 con 0x7fded813b020 2024-03-27T21:40:04.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.006+0000 7fc7797fa640 10 client.4498.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:04.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.006+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign -1 4 2024-03-27T21:40:04.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.006+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc794139540 osd=1 3 2024-03-27T21:40:04.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.006+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc794139540 osd=1 3 2024-03-27T21:40:04.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.006+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc794139540 osd=1 3 2024-03-27T21:40:04.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.006+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_remove -1 4 2024-03-27T21:40:04.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.006+0000 7fc7797fa640 20 client.4498.objecter get_session s=0x7fc794139540 osd=1 3 2024-03-27T21:40:04.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.006+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign 1 4 2024-03-27T21:40:04.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.006+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc794139540 osd=1 4 2024-03-27T21:40:04.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.006+0000 7fc7797fa640 10 client.4498.objecter _send_command 4 2024-03-27T21:40:04.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.006+0000 7fc7797fa640 1 -- 172.21.15.60:0/4035379115 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc794138fc0 con 0x7fc7940b4bc0 2024-03-27T21:40:04.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.007+0000 7fc77b7fe640 1 -- 172.21.15.60:0/4035379115 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc794138fc0 con 0x7fc7940b4bc0 2024-03-27T21:40:04.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.007+0000 7fc77b7fe640 10 client.4498.objecter ms_dispatch 0x7fc794000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:04.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.007+0000 7fc77b7fe640 10 client.4498.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:04.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.007+0000 7fc77b7fe640 20 client.4498.objecter put_session s=0x7fc794139540 osd=1 4 2024-03-27T21:40:04.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.007+0000 7fc77b7fe640 15 client.4498.objecter _session_command_op_remove 1 4 2024-03-27T21:40:04.994 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:04.996 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:04.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.007+0000 7fc7797fa640 10 client.4498.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:04.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.007+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign -1 5 2024-03-27T21:40:04.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.007+0000 7fc7797fa640 20 client.4498.objecter _get_session osd=-1 returning homeless 2024-03-27T21:40:04.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.007+0000 7fc7797fa640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:04.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.007+0000 7fc7797fa640 10 monclient: _renew_subs 2024-03-27T21:40:04.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.007+0000 7fc7797fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:04.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.007+0000 7fc7797fa640 1 -- 172.21.15.60:0/4035379115 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=27}) v3 -- 0x7fc7941e7550 con 0x7fc7940b40b0 2024-03-27T21:40:04.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.007+0000 7fc7797fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:04.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.007+0000 7fc7797fa640 1 -- 172.21.15.60:0/4035379115 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc7941393d0 con 0x7fc7940b40b0 2024-03-27T21:40:04.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.008+0000 7fc77b7fe640 1 -- 172.21.15.60:0/4035379115 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc784057bd0 con 0x7fc7940b40b0 2024-03-27T21:40:04.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.008+0000 7fc77b7fe640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-03-27T21:40:04.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.008+0000 7fc7917fa640 10 client.4498.objecter _check_command_map_dne tid 5 current 26 map_dne_bound 26 2024-03-27T21:40:04.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.008+0000 7fc7917fa640 10 client.4498.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:40:04.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.008+0000 7fc7917fa640 15 client.4498.objecter _session_command_op_remove -1 5 2024-03-27T21:40:04.997 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:40:04.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.008+0000 7fc7797fa640 10 client.4498.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:04.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.008+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign -1 6 2024-03-27T21:40:04.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.008+0000 7fc7797fa640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fc794066250 0x7fc794066630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:04.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.008+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc7940b5720 osd=3 3 2024-03-27T21:40:04.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.008+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc7940b5720 osd=3 3 2024-03-27T21:40:04.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.008+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc7940b5720 osd=3 3 2024-03-27T21:40:04.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.008+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_remove -1 6 2024-03-27T21:40:04.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.008+0000 7fc7797fa640 20 client.4498.objecter get_session s=0x7fc7940b5720 osd=3 3 2024-03-27T21:40:04.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.008+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign 3 6 2024-03-27T21:40:04.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.008+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc7940b5720 osd=3 4 2024-03-27T21:40:04.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.008+0000 7fc7797fa640 10 client.4498.objecter _send_command 6 2024-03-27T21:40:04.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.008+0000 7fc7797fa640 1 -- 172.21.15.60:0/4035379115 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc7941ea870 con 0x7fc794066250 2024-03-27T21:40:04.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.008+0000 7fc7927fc640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fc794066250 0x7fc794066630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:04.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.008+0000 7fc7927fc640 10 monclient: get_auth_request con 0x7fc794066250 auth_method 0 2024-03-27T21:40:04.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.008+0000 7fc77b7fe640 10 client.4498.objecter ms_handle_connect 0x7fc794066250 2024-03-27T21:40:04.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.008+0000 7fc7927fc640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fc794066250 0x7fc794066630 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:04.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.009+0000 7fc77b7fe640 1 -- 172.21.15.60:0/4035379115 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fc7941ea870 con 0x7fc794066250 2024-03-27T21:40:04.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.009+0000 7fc77b7fe640 10 client.4498.objecter ms_dispatch 0x7fc794000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:40:04.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.009+0000 7fc77b7fe640 10 client.4498.objecter _finish_command 6 = system:0 2024-03-27T21:40:04.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.009+0000 7fc77b7fe640 20 client.4498.objecter put_session s=0x7fc7940b5720 osd=3 4 2024-03-27T21:40:04.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.009+0000 7fc77b7fe640 15 client.4498.objecter _session_command_op_remove 3 6 2024-03-27T21:40:04.999 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:04.999 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:05.007 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:40:05.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.033+0000 7fc7797fa640 10 client.4498.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:05.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.033+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign -1 7 2024-03-27T21:40:05.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.033+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc7940b5720 osd=3 3 2024-03-27T21:40:05.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.033+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc7940b5720 osd=3 3 2024-03-27T21:40:05.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.033+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc7940b5720 osd=3 3 2024-03-27T21:40:05.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.033+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_remove -1 7 2024-03-27T21:40:05.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.033+0000 7fc7797fa640 20 client.4498.objecter get_session s=0x7fc7940b5720 osd=3 3 2024-03-27T21:40:05.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.033+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign 3 7 2024-03-27T21:40:05.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.033+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc7940b5720 osd=3 4 2024-03-27T21:40:05.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.033+0000 7fc7797fa640 10 client.4498.objecter _send_command 7 2024-03-27T21:40:05.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.033+0000 7fc7797fa640 1 -- 172.21.15.60:0/4035379115 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc794065cd0 con 0x7fc794066250 2024-03-27T21:40:05.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.034+0000 7fc77b7fe640 1 -- 172.21.15.60:0/4035379115 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc794065cd0 con 0x7fc794066250 2024-03-27T21:40:05.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.034+0000 7fc77b7fe640 10 client.4498.objecter ms_dispatch 0x7fc794000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:05.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.034+0000 7fc77b7fe640 10 client.4498.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:05.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.034+0000 7fc77b7fe640 20 client.4498.objecter put_session s=0x7fc7940b5720 osd=3 4 2024-03-27T21:40:05.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.034+0000 7fc77b7fe640 15 client.4498.objecter _session_command_op_remove 3 7 2024-03-27T21:40:05.021 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:05.023 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:05.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.035+0000 7fc7797fa640 10 client.4498.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:05.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.035+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign -1 8 2024-03-27T21:40:05.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.035+0000 7fc7797fa640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fc7941ea3c0 0x7fc7941e5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:05.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.035+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc794066080 osd=4 3 2024-03-27T21:40:05.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.035+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc794066080 osd=4 3 2024-03-27T21:40:05.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.035+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc794066080 osd=4 3 2024-03-27T21:40:05.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.035+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_remove -1 8 2024-03-27T21:40:05.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.035+0000 7fc7797fa640 20 client.4498.objecter get_session s=0x7fc794066080 osd=4 3 2024-03-27T21:40:05.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.035+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign 4 8 2024-03-27T21:40:05.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.035+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc794066080 osd=4 4 2024-03-27T21:40:05.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.035+0000 7fc7797fa640 10 client.4498.objecter _send_command 8 2024-03-27T21:40:05.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.035+0000 7fc7797fa640 1 -- 172.21.15.60:0/4035379115 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc7941ee100 con 0x7fc7941ea3c0 2024-03-27T21:40:05.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.035+0000 7fc7937fe640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fc7941ea3c0 0x7fc7941e5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:05.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.035+0000 7fc7937fe640 10 monclient: get_auth_request con 0x7fc7941ea3c0 auth_method 0 2024-03-27T21:40:05.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.035+0000 7fc77b7fe640 10 client.4498.objecter ms_handle_connect 0x7fc7941ea3c0 2024-03-27T21:40:05.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.035+0000 7fc7937fe640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fc7941ea3c0 0x7fc7941e5e40 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:05.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.036+0000 7fc77b7fe640 1 -- 172.21.15.60:0/4035379115 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fc7941ee100 con 0x7fc7941ea3c0 2024-03-27T21:40:05.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.036+0000 7fc77b7fe640 10 client.4498.objecter ms_dispatch 0x7fc794000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:40:05.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.036+0000 7fc77b7fe640 10 client.4498.objecter _finish_command 8 = system:0 2024-03-27T21:40:05.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.036+0000 7fc77b7fe640 20 client.4498.objecter put_session s=0x7fc794066080 osd=4 4 2024-03-27T21:40:05.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.036+0000 7fc77b7fe640 15 client.4498.objecter _session_command_op_remove 4 8 2024-03-27T21:40:05.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.063+0000 7fc7797fa640 10 client.4498.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:05.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.063+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign -1 9 2024-03-27T21:40:05.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.063+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc794066080 osd=4 3 2024-03-27T21:40:05.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.063+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc794066080 osd=4 3 2024-03-27T21:40:05.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.063+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc794066080 osd=4 3 2024-03-27T21:40:05.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.063+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_remove -1 9 2024-03-27T21:40:05.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.063+0000 7fc7797fa640 20 client.4498.objecter get_session s=0x7fc794066080 osd=4 3 2024-03-27T21:40:05.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.063+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign 4 9 2024-03-27T21:40:05.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.063+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc794066080 osd=4 4 2024-03-27T21:40:05.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.063+0000 7fc7797fa640 10 client.4498.objecter _send_command 9 2024-03-27T21:40:05.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.063+0000 7fc7797fa640 1 -- 172.21.15.60:0/4035379115 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc7941e9930 con 0x7fc7941ea3c0 2024-03-27T21:40:05.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.065+0000 7fc77b7fe640 1 -- 172.21.15.60:0/4035379115 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc7941e9930 con 0x7fc7941ea3c0 2024-03-27T21:40:05.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.065+0000 7fc77b7fe640 10 client.4498.objecter ms_dispatch 0x7fc794000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:05.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.065+0000 7fc77b7fe640 10 client.4498.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:05.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.065+0000 7fc77b7fe640 20 client.4498.objecter put_session s=0x7fc794066080 osd=4 4 2024-03-27T21:40:05.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.065+0000 7fc77b7fe640 15 client.4498.objecter _session_command_op_remove 4 9 2024-03-27T21:40:05.052 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:05.053 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:05.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.065+0000 7fc7797fa640 10 client.4498.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:05.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.065+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign -1 10 2024-03-27T21:40:05.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.065+0000 7fc7797fa640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fc7941e9aa0 0x7fc7941ec9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:05.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.065+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc7941e9930 osd=5 3 2024-03-27T21:40:05.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.065+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc7941e9930 osd=5 3 2024-03-27T21:40:05.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.065+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc7941e9930 osd=5 3 2024-03-27T21:40:05.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.065+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_remove -1 10 2024-03-27T21:40:05.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.065+0000 7fc7797fa640 20 client.4498.objecter get_session s=0x7fc7941e9930 osd=5 3 2024-03-27T21:40:05.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.065+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign 5 10 2024-03-27T21:40:05.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.065+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc7941e9930 osd=5 4 2024-03-27T21:40:05.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.065+0000 7fc7797fa640 10 client.4498.objecter _send_command 10 2024-03-27T21:40:05.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.065+0000 7fc7797fa640 1 -- 172.21.15.60:0/4035379115 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc7941e5990 con 0x7fc7941e9aa0 2024-03-27T21:40:05.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.065+0000 7fc792ffd640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fc7941e9aa0 0x7fc7941ec9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:05.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.065+0000 7fc792ffd640 10 monclient: get_auth_request con 0x7fc7941e9aa0 auth_method 0 2024-03-27T21:40:05.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.066+0000 7fc77b7fe640 10 client.4498.objecter ms_handle_connect 0x7fc7941e9aa0 2024-03-27T21:40:05.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.066+0000 7fc792ffd640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fc7941e9aa0 0x7fc7941ec9f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:05.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.066+0000 7fc77b7fe640 1 -- 172.21.15.60:0/4035379115 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fc7941e5990 con 0x7fc7941e9aa0 2024-03-27T21:40:05.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.066+0000 7fc77b7fe640 10 client.4498.objecter ms_dispatch 0x7fc794000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:40:05.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.066+0000 7fc77b7fe640 10 client.4498.objecter _finish_command 10 = system:0 2024-03-27T21:40:05.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.066+0000 7fc77b7fe640 20 client.4498.objecter put_session s=0x7fc7941e9930 osd=5 4 2024-03-27T21:40:05.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.066+0000 7fc77b7fe640 15 client.4498.objecter _session_command_op_remove 5 10 2024-03-27T21:40:05.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.085+0000 7fc7797fa640 10 client.4498.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:05.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.085+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign -1 11 2024-03-27T21:40:05.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.085+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc7941e9930 osd=5 3 2024-03-27T21:40:05.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.085+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc7941e9930 osd=5 3 2024-03-27T21:40:05.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.085+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc7941e9930 osd=5 3 2024-03-27T21:40:05.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.085+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_remove -1 11 2024-03-27T21:40:05.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.085+0000 7fc7797fa640 20 client.4498.objecter get_session s=0x7fc7941e9930 osd=5 3 2024-03-27T21:40:05.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.085+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign 5 11 2024-03-27T21:40:05.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.085+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc7941e9930 osd=5 4 2024-03-27T21:40:05.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.085+0000 7fc7797fa640 10 client.4498.objecter _send_command 11 2024-03-27T21:40:05.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.085+0000 7fc7797fa640 1 -- 172.21.15.60:0/4035379115 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc7941e5990 con 0x7fc7941e9aa0 2024-03-27T21:40:05.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.086+0000 7fc77b7fe640 1 -- 172.21.15.60:0/4035379115 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc7941e5990 con 0x7fc7941e9aa0 2024-03-27T21:40:05.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.086+0000 7fc77b7fe640 10 client.4498.objecter ms_dispatch 0x7fc794000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:05.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.086+0000 7fc77b7fe640 10 client.4498.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:05.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.086+0000 7fc77b7fe640 20 client.4498.objecter put_session s=0x7fc7941e9930 osd=5 4 2024-03-27T21:40:05.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.086+0000 7fc77b7fe640 15 client.4498.objecter _session_command_op_remove 5 11 2024-03-27T21:40:05.073 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:05.074 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:05.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.086+0000 7fc7797fa640 10 client.4498.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.086+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign -1 12 2024-03-27T21:40:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.086+0000 7fc7797fa640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fc7941ed440 0x7fc7941ed8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.086+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc7941e5990 osd=6 3 2024-03-27T21:40:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.086+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc7941e5990 osd=6 3 2024-03-27T21:40:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.086+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc7941e5990 osd=6 3 2024-03-27T21:40:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.086+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_remove -1 12 2024-03-27T21:40:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.086+0000 7fc7797fa640 20 client.4498.objecter get_session s=0x7fc7941e5990 osd=6 3 2024-03-27T21:40:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.086+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign 6 12 2024-03-27T21:40:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.086+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc7941e5990 osd=6 4 2024-03-27T21:40:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.086+0000 7fc7797fa640 10 client.4498.objecter _send_command 12 2024-03-27T21:40:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.086+0000 7fc7797fa640 1 -- 172.21.15.60:0/4035379115 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc7941edde0 con 0x7fc7941ed440 2024-03-27T21:40:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.087+0000 7fc7927fc640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fc7941ed440 0x7fc7941ed8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.087+0000 7fc7927fc640 10 monclient: get_auth_request con 0x7fc7941ed440 auth_method 0 2024-03-27T21:40:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.087+0000 7fc77b7fe640 10 client.4498.objecter ms_handle_connect 0x7fc7941ed440 2024-03-27T21:40:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.087+0000 7fc7927fc640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fc7941ed440 0x7fc7941ed8a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.088+0000 7fc77b7fe640 1 -- 172.21.15.60:0/4035379115 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fc7941edde0 con 0x7fc7941ed440 2024-03-27T21:40:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.088+0000 7fc77b7fe640 10 client.4498.objecter ms_dispatch 0x7fc794000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:40:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.088+0000 7fc77b7fe640 10 client.4498.objecter _finish_command 12 = system:0 2024-03-27T21:40:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.088+0000 7fc77b7fe640 20 client.4498.objecter put_session s=0x7fc7941e5990 osd=6 4 2024-03-27T21:40:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.088+0000 7fc77b7fe640 15 client.4498.objecter _session_command_op_remove 6 12 2024-03-27T21:40:05.090 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:05.105+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:05.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.107+0000 7fcc385e7640 10 start_mon_command cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}] 2024-03-27T21:40:05.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.108+0000 7fcc385e7640 10 monclient: _send_command 2 [{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}] 2024-03-27T21:40:05.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.108+0000 7fcc385e7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:05.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.108+0000 7fcc385e7640 1 -- 172.21.15.60:0/2177603658 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} v 0) v1 -- 0x7fcc30069470 con 0x7fcc3006f900 2024-03-27T21:40:05.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.110+0000 7fc7797fa640 10 client.4498.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.110+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign -1 13 2024-03-27T21:40:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.110+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc7941e5990 osd=6 3 2024-03-27T21:40:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.110+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc7941e5990 osd=6 3 2024-03-27T21:40:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.110+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc7941e5990 osd=6 3 2024-03-27T21:40:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.110+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_remove -1 13 2024-03-27T21:40:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.110+0000 7fc7797fa640 20 client.4498.objecter get_session s=0x7fc7941e5990 osd=6 3 2024-03-27T21:40:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.110+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign 6 13 2024-03-27T21:40:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.110+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc7941e5990 osd=6 4 2024-03-27T21:40:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.110+0000 7fc7797fa640 10 client.4498.objecter _send_command 13 2024-03-27T21:40:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.110+0000 7fc7797fa640 1 -- 172.21.15.60:0/4035379115 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc7941edde0 con 0x7fc7941ed440 2024-03-27T21:40:05.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.112+0000 7fc77b7fe640 1 -- 172.21.15.60:0/4035379115 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc7941edde0 con 0x7fc7941ed440 2024-03-27T21:40:05.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.112+0000 7fc77b7fe640 10 client.4498.objecter ms_dispatch 0x7fc794000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:05.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.112+0000 7fc77b7fe640 10 client.4498.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:05.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.112+0000 7fc77b7fe640 20 client.4498.objecter put_session s=0x7fc7941e5990 osd=6 4 2024-03-27T21:40:05.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.112+0000 7fc77b7fe640 15 client.4498.objecter _session_command_op_remove 6 13 2024-03-27T21:40:05.099 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.112+0000 7fc7797fa640 10 client.4498.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.112+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign -1 14 2024-03-27T21:40:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.112+0000 7fc7797fa640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fc7941f27a0 0x7fc7941f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.112+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc7941edde0 osd=7 3 2024-03-27T21:40:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.112+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc7941edde0 osd=7 3 2024-03-27T21:40:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.112+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc7941edde0 osd=7 3 2024-03-27T21:40:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.112+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_remove -1 14 2024-03-27T21:40:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.112+0000 7fc7797fa640 20 client.4498.objecter get_session s=0x7fc7941edde0 osd=7 3 2024-03-27T21:40:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.112+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign 7 14 2024-03-27T21:40:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.112+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc7941edde0 osd=7 4 2024-03-27T21:40:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.112+0000 7fc7797fa640 10 client.4498.objecter _send_command 14 2024-03-27T21:40:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.112+0000 7fc7797fa640 1 -- 172.21.15.60:0/4035379115 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc7941e5b00 con 0x7fc7941f27a0 2024-03-27T21:40:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.112+0000 7fc7937fe640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fc7941f27a0 0x7fc7941f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.112+0000 7fc7937fe640 10 monclient: get_auth_request con 0x7fc7941f27a0 auth_method 0 2024-03-27T21:40:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.112+0000 7fc77b7fe640 10 client.4498.objecter ms_handle_connect 0x7fc7941f27a0 2024-03-27T21:40:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.112+0000 7fc7937fe640 1 --2- 172.21.15.60:0/4035379115 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fc7941f27a0 0x7fc7941f4b90 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.113+0000 7fc77b7fe640 1 -- 172.21.15.60:0/4035379115 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fc7941e5b00 con 0x7fc7941f27a0 2024-03-27T21:40:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.113+0000 7fc77b7fe640 10 client.4498.objecter ms_dispatch 0x7fc794000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:40:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.113+0000 7fc77b7fe640 10 client.4498.objecter _finish_command 14 = system:0 2024-03-27T21:40:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.113+0000 7fc77b7fe640 20 client.4498.objecter put_session s=0x7fc7941edde0 osd=7 4 2024-03-27T21:40:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.113+0000 7fc77b7fe640 15 client.4498.objecter _session_command_op_remove 7 14 2024-03-27T21:40:05.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.127+0000 7fc7797fa640 10 client.4498.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:05.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.127+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign -1 15 2024-03-27T21:40:05.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.127+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc7941edde0 osd=7 3 2024-03-27T21:40:05.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.127+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc7941edde0 osd=7 3 2024-03-27T21:40:05.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.127+0000 7fc7797fa640 20 client.4498.objecter _get_session s=0x7fc7941edde0 osd=7 3 2024-03-27T21:40:05.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.127+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_remove -1 15 2024-03-27T21:40:05.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.127+0000 7fc7797fa640 20 client.4498.objecter get_session s=0x7fc7941edde0 osd=7 3 2024-03-27T21:40:05.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.127+0000 7fc7797fa640 15 client.4498.objecter _session_command_op_assign 7 15 2024-03-27T21:40:05.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.127+0000 7fc7797fa640 20 client.4498.objecter put_session s=0x7fc7941edde0 osd=7 4 2024-03-27T21:40:05.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.127+0000 7fc7797fa640 10 client.4498.objecter _send_command 15 2024-03-27T21:40:05.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.127+0000 7fc7797fa640 1 -- 172.21.15.60:0/4035379115 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc7941e5b00 con 0x7fc7941f27a0 2024-03-27T21:40:05.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.128+0000 7fc77b7fe640 1 -- 172.21.15.60:0/4035379115 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc7941e5b00 con 0x7fc7941f27a0 2024-03-27T21:40:05.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.128+0000 7fc77b7fe640 10 client.4498.objecter ms_dispatch 0x7fc794000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:05.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.128+0000 7fc77b7fe640 10 client.4498.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:05.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.128+0000 7fc77b7fe640 20 client.4498.objecter put_session s=0x7fc7941edde0 osd=7 4 2024-03-27T21:40:05.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.128+0000 7fc77b7fe640 15 client.4498.objecter _session_command_op_remove 7 15 2024-03-27T21:40:05.116 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:05.116 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:40:05.118 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:40:05.155 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:05.155 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:05.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-03-27T21:40:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 25.735882448000002, 2024-03-27T21:40:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43537978700000002, 2024-03-27T21:40:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:05.156 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:05.157 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-03-27T21:40:05.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.158 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:05.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:05.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:05.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:05.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432019+0000", 2024-03-27T21:40:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-03-27T21:40:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432027+0000", 2024-03-27T21:40:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1200000000000002e-06 2024-03-27T21:40:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432039+0000", 2024-03-27T21:40:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1123e-05 2024-03-27T21:40:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432051+0000", 2024-03-27T21:40:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-03-27T21:40:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432157+0000", 2024-03-27T21:40:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106846 2024-03-27T21:40:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.160 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-03-27T21:40:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432176+0000", 2024-03-27T21:40:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8490000000000001e-05 2024-03-27T21:40:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866170+0000", 2024-03-27T21:40:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43399370100000001 2024-03-27T21:40:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866335+0000", 2024-03-27T21:40:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016559699999999999 2024-03-27T21:40:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866495+0000", 2024-03-27T21:40:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015944200000000001 2024-03-27T21:40:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867012+0000", 2024-03-27T21:40:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0005176 2024-03-27T21:40:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867334+0000", 2024-03-27T21:40:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032158600000000001 2024-03-27T21:40:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867374+0000", 2024-03-27T21:40:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0547000000000003e-05 2024-03-27T21:40:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867400+0000", 2024-03-27T21:40:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5958000000000001e-05 2024-03-27T21:40:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.163 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 25.238499997000002, 2024-03-27T21:40:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064560310000000001, 2024-03-27T21:40:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 70 2024-03-27T21:40:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:05.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:05.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:05.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:05.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929402+0000", 2024-03-27T21:40:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929407+0000", 2024-03-27T21:40:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-06 2024-03-27T21:40:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929414+0000", 2024-03-27T21:40:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5689999999999999e-06 2024-03-27T21:40:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929419+0000", 2024-03-27T21:40:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-03-27T21:40:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930245+0000", 2024-03-27T21:40:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082521500000000002 2024-03-27T21:40:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930287+0000", 2024-03-27T21:40:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2305000000000002e-05 2024-03-27T21:40:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930367+0000", 2024-03-27T21:40:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9589e-05 2024-03-27T21:40:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.931214+0000", 2024-03-27T21:40:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000847827 2024-03-27T21:40:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935807+0000", 2024-03-27T21:40:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045923780000000003 2024-03-27T21:40:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935840+0000", 2024-03-27T21:40:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3501999999999999e-05 2024-03-27T21:40:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935859+0000", 2024-03-27T21:40:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8845e-05 2024-03-27T21:40:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.169 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 25.231240406000001, 2024-03-27T21:40:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071695429999999996, 2024-03-27T21:40:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-03-27T21:40:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:05.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936662+0000", 2024-03-27T21:40:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936737+0000", 2024-03-27T21:40:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5303999999999996e-05 2024-03-27T21:40:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936746+0000", 2024-03-27T21:40:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6340000000000007e-06 2024-03-27T21:40:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936753+0000", 2024-03-27T21:40:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-03-27T21:40:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937926+0000", 2024-03-27T21:40:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011731280000000001 2024-03-27T21:40:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937969+0000", 2024-03-27T21:40:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3285999999999999e-05 2024-03-27T21:40:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:05.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938040+0000", 2024-03-27T21:40:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0358000000000006e-05 2024-03-27T21:40:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939324+0000", 2024-03-27T21:40:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012844899999999999 2024-03-27T21:40:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943780+0000", 2024-03-27T21:40:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044555630000000001 2024-03-27T21:40:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943811+0000", 2024-03-27T21:40:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1415999999999999e-05 2024-03-27T21:40:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943832+0000", 2024-03-27T21:40:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0981000000000001e-05 2024-03-27T21:40:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:05.175 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 25.231142389999999, 2024-03-27T21:40:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071139740000000003, 2024-03-27T21:40:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-03-27T21:40:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936760+0000", 2024-03-27T21:40:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936765+0000", 2024-03-27T21:40:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6330000000000004e-06 2024-03-27T21:40:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936771+0000", 2024-03-27T21:40:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0560000000000001e-06 2024-03-27T21:40:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936776+0000", 2024-03-27T21:40:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1510000000000003e-06 2024-03-27T21:40:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938605+0000", 2024-03-27T21:40:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001829589 2024-03-27T21:40:05.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938657+0000", 2024-03-27T21:40:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1860000000000002e-05 2024-03-27T21:40:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938734+0000", 2024-03-27T21:40:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6639999999999998e-05 2024-03-27T21:40:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939562+0000", 2024-03-27T21:40:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082803100000000001 2024-03-27T21:40:05.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943838+0000", 2024-03-27T21:40:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042757050000000003 2024-03-27T21:40:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943859+0000", 2024-03-27T21:40:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1175e-05 2024-03-27T21:40:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943875+0000", 2024-03-27T21:40:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5744e-05 2024-03-27T21:40:05.180 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 25.087628338999998, 2024-03-27T21:40:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059712230000000003, 2024-03-27T21:40:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 112 2024-03-27T21:40:05.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080274+0000", 2024-03-27T21:40:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080318+0000", 2024-03-27T21:40:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3634999999999998e-05 2024-03-27T21:40:05.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:05.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080329+0000", 2024-03-27T21:40:05.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1232e-05 2024-03-27T21:40:05.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080333+0000", 2024-03-27T21:40:05.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6009999999999999e-06 2024-03-27T21:40:05.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:05.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083786+0000", 2024-03-27T21:40:05.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034528940000000002 2024-03-27T21:40:05.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:05.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083861+0000", 2024-03-27T21:40:05.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4893000000000003e-05 2024-03-27T21:40:05.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:05.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083981+0000", 2024-03-27T21:40:05.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012058299999999999 2024-03-27T21:40:05.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:05.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086148+0000", 2024-03-27T21:40:05.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021671469999999999 2024-03-27T21:40:05.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:05.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086195+0000", 2024-03-27T21:40:05.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6768999999999999e-05 2024-03-27T21:40:05.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:05.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086224+0000", 2024-03-27T21:40:05.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8758000000000001e-05 2024-03-27T21:40:05.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:05.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086246+0000", 2024-03-27T21:40:05.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2064000000000001e-05 2024-03-27T21:40:05.187 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.187 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:05.187 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.187 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:05.187 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:05.187 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 25.087565391999998, 2024-03-27T21:40:05.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066196320000000003, 2024-03-27T21:40:05.187 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:05.187 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.187 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:05.187 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:05.187 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:05.188 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113 2024-03-27T21:40:05.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.188 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:05.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:05.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:05.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:05.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:05.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:05.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080337+0000", 2024-03-27T21:40:05.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:05.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080340+0000", 2024-03-27T21:40:05.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6690000000000002e-06 2024-03-27T21:40:05.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:05.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080344+0000", 2024-03-27T21:40:05.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6550000000000002e-06 2024-03-27T21:40:05.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.190 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.190 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080348+0000", 2024-03-27T21:40:05.190 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9169999999999999e-06 2024-03-27T21:40:05.190 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.190 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:05.190 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084595+0000", 2024-03-27T21:40:05.190 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042471719999999996 2024-03-27T21:40:05.190 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.190 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:05.190 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084676+0000", 2024-03-27T21:40:05.190 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1147000000000001e-05 2024-03-27T21:40:05.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:05.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084796+0000", 2024-03-27T21:40:05.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012032900000000001 2024-03-27T21:40:05.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:05.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086190+0000", 2024-03-27T21:40:05.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013934399999999999 2024-03-27T21:40:05.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:05.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086905+0000", 2024-03-27T21:40:05.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071536199999999999 2024-03-27T21:40:05.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:05.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086937+0000", 2024-03-27T21:40:05.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2125000000000002e-05 2024-03-27T21:40:05.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:05.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086957+0000", 2024-03-27T21:40:05.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0157999999999999e-05 2024-03-27T21:40:05.192 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.192 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:05.192 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.193 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:05.193 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:05.193 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 25.087552676000001, 2024-03-27T21:40:05.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007405661, 2024-03-27T21:40:05.193 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:05.193 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.193 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:05.193 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:05.193 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:05.193 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 114 2024-03-27T21:40:05.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.193 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:05.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:05.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:05.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:05.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:05.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:05.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:05.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-03-27T21:40:05.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.195 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:05.195 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080353+0000", 2024-03-27T21:40:05.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.441e-06 2024-03-27T21:40:05.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.195 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:05.195 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080356+0000", 2024-03-27T21:40:05.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0769999999999999e-06 2024-03-27T21:40:05.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.195 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.195 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080358+0000", 2024-03-27T21:40:05.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3369999999999998e-06 2024-03-27T21:40:05.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.196 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:05.196 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085118+0000", 2024-03-27T21:40:05.196 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047603949999999997 2024-03-27T21:40:05.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.196 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:05.196 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085210+0000", 2024-03-27T21:40:05.196 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1983000000000003e-05 2024-03-27T21:40:05.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.196 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:05.196 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085332+0000", 2024-03-27T21:40:05.196 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121803 2024-03-27T21:40:05.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:05.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086292+0000", 2024-03-27T21:40:05.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095961400000000002 2024-03-27T21:40:05.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:05.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087710+0000", 2024-03-27T21:40:05.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014181459999999999 2024-03-27T21:40:05.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:05.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087736+0000", 2024-03-27T21:40:05.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6475e-05 2024-03-27T21:40:05.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:05.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087756+0000", 2024-03-27T21:40:05.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9626000000000001e-05 2024-03-27T21:40:05.198 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.198 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:05.198 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.198 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:05.198 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:05.198 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 25.087530241, 2024-03-27T21:40:05.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075011720000000004, 2024-03-27T21:40:05.199 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:05.199 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.199 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:05.199 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:05.199 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:05.199 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115 2024-03-27T21:40:05.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.199 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:05.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:05.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:05.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:05.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:05.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:05.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:05.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.201 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:05.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080458+0000", 2024-03-27T21:40:05.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5228000000000005e-05 2024-03-27T21:40:05.201 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:05.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080465+0000", 2024-03-27T21:40:05.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.238e-06 2024-03-27T21:40:05.201 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080468+0000", 2024-03-27T21:40:05.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5109999999999999e-06 2024-03-27T21:40:05.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:05.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086326+0000", 2024-03-27T21:40:05.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058579169999999998 2024-03-27T21:40:05.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:05.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086382+0000", 2024-03-27T21:40:05.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5309999999999997e-05 2024-03-27T21:40:05.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:05.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086457+0000", 2024-03-27T21:40:05.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5413000000000005e-05 2024-03-27T21:40:05.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:05.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087762+0000", 2024-03-27T21:40:05.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013052300000000001 2024-03-27T21:40:05.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:05.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087798+0000", 2024-03-27T21:40:05.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5614999999999997e-05 2024-03-27T21:40:05.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:05.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087840+0000", 2024-03-27T21:40:05.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2196000000000001e-05 2024-03-27T21:40:05.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:05.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087874+0000", 2024-03-27T21:40:05.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3785999999999998e-05 2024-03-27T21:40:05.204 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.204 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:05.204 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.205 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:05.205 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:05.205 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 25.087430158, 2024-03-27T21:40:05.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075050680000000002, 2024-03-27T21:40:05.205 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:05.205 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.205 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:05.205 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:05.205 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:05.205 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 116 2024-03-27T21:40:05.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.205 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:05.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:05.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:05.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:05.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:05.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:05.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080472+0000", 2024-03-27T21:40:05.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:05.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080475+0000", 2024-03-27T21:40:05.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.712e-06 2024-03-27T21:40:05.207 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:05.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080478+0000", 2024-03-27T21:40:05.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1070000000000001e-06 2024-03-27T21:40:05.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080481+0000", 2024-03-27T21:40:05.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4269999999999998e-06 2024-03-27T21:40:05.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:05.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086992+0000", 2024-03-27T21:40:05.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065114099999999996 2024-03-27T21:40:05.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:05.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087064+0000", 2024-03-27T21:40:05.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1927000000000004e-05 2024-03-27T21:40:05.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:05.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087188+0000", 2024-03-27T21:40:05.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123585 2024-03-27T21:40:05.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:05.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087766+0000", 2024-03-27T21:40:05.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057814300000000004 2024-03-27T21:40:05.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:05.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087934+0000", 2024-03-27T21:40:05.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000167894 2024-03-27T21:40:05.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:05.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087958+0000", 2024-03-27T21:40:05.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4663000000000001e-05 2024-03-27T21:40:05.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:05.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087978+0000", 2024-03-27T21:40:05.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9636e-05 2024-03-27T21:40:05.210 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.210 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:05.210 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.211 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:05.211 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:05.211 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 25.087398036, 2024-03-27T21:40:05.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076850679999999998, 2024-03-27T21:40:05.211 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:05.211 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.211 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:05.211 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:05.212 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:05.212 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 117 2024-03-27T21:40:05.212 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.212 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:05.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.212 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:05.212 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:05.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.212 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.212 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:05.212 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:05.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.212 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:05.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:05.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:05.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080508+0000", 2024-03-27T21:40:05.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.298e-06 2024-03-27T21:40:05.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:05.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080515+0000", 2024-03-27T21:40:05.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0790000000000003e-06 2024-03-27T21:40:05.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080518+0000", 2024-03-27T21:40:05.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8509999999999999e-06 2024-03-27T21:40:05.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:05.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087444+0000", 2024-03-27T21:40:05.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0069259120000000002 2024-03-27T21:40:05.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:05.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087491+0000", 2024-03-27T21:40:05.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7043999999999999e-05 2024-03-27T21:40:05.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:05.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087564+0000", 2024-03-27T21:40:05.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3603999999999995e-05 2024-03-27T21:40:05.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:05.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087902+0000", 2024-03-27T21:40:05.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033734100000000001 2024-03-27T21:40:05.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:05.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088117+0000", 2024-03-27T21:40:05.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000215164 2024-03-27T21:40:05.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:05.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088160+0000", 2024-03-27T21:40:05.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2933999999999999e-05 2024-03-27T21:40:05.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:05.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088190+0000", 2024-03-27T21:40:05.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0139999999999999e-05 2024-03-27T21:40:05.216 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.216 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:05.216 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.217 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:05.217 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:05.217 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 24.671117646999999, 2024-03-27T21:40:05.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052465439999999997, 2024-03-27T21:40:05.217 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:05.217 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.217 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:05.217 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:05.217 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:05.217 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 133 2024-03-27T21:40:05.217 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.217 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:05.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.217 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:05.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:05.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.218 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:05.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:05.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.218 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:05.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:05.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:05.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496788+0000", 2024-03-27T21:40:05.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-03-27T21:40:05.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:05.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496793+0000", 2024-03-27T21:40:05.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4790000000000004e-06 2024-03-27T21:40:05.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496796+0000", 2024-03-27T21:40:05.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1e-06 2024-03-27T21:40:05.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:05.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500447+0000", 2024-03-27T21:40:05.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036504459999999999 2024-03-27T21:40:05.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:05.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500492+0000", 2024-03-27T21:40:05.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5765e-05 2024-03-27T21:40:05.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:05.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500562+0000", 2024-03-27T21:40:05.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9206000000000001e-05 2024-03-27T21:40:05.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:05.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501959+0000", 2024-03-27T21:40:05.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001397187 2024-03-27T21:40:05.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:05.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501984+0000", 2024-03-27T21:40:05.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.552e-05 2024-03-27T21:40:05.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:05.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502010+0000", 2024-03-27T21:40:05.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5232000000000002e-05 2024-03-27T21:40:05.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:05.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502032+0000", 2024-03-27T21:40:05.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2337999999999999e-05 2024-03-27T21:40:05.222 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.222 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:05.222 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.223 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:05.223 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:05.223 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 24.671058988999999, 2024-03-27T21:40:05.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052892119999999997, 2024-03-27T21:40:05.223 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:05.223 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.223 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:05.223 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:05.223 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:05.223 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 134 2024-03-27T21:40:05.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.223 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:05.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:05.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:05.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:05.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:05.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:05.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:05.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:05.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496929+0000", 2024-03-27T21:40:05.225 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5098000000000004e-05 2024-03-27T21:40:05.225 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.225 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:05.225 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496937+0000", 2024-03-27T21:40:05.225 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9559999999999997e-06 2024-03-27T21:40:05.225 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.225 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.225 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496942+0000", 2024-03-27T21:40:05.225 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6350000000000003e-06 2024-03-27T21:40:05.225 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.225 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:05.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500749+0000", 2024-03-27T21:40:05.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038068529999999998 2024-03-27T21:40:05.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:05.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500792+0000", 2024-03-27T21:40:05.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2823e-05 2024-03-27T21:40:05.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:05.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500864+0000", 2024-03-27T21:40:05.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1715e-05 2024-03-27T21:40:05.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:05.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502068+0000", 2024-03-27T21:40:05.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012044860000000001 2024-03-27T21:40:05.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:05.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502073+0000", 2024-03-27T21:40:05.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5700000000000003e-06 2024-03-27T21:40:05.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:05.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502098+0000", 2024-03-27T21:40:05.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5120999999999999e-05 2024-03-27T21:40:05.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:05.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502133+0000", 2024-03-27T21:40:05.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5306999999999999e-05 2024-03-27T21:40:05.228 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.228 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:05.228 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.228 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.228 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:05.228 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:05.228 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 24.670953682, 2024-03-27T21:40:05.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062529040000000001, 2024-03-27T21:40:05.228 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:05.228 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.229 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:05.229 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:05.229 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:05.229 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135 2024-03-27T21:40:05.229 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.229 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:05.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.229 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:05.229 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:05.229 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.229 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.230 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:05.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:05.230 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.230 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.230 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:05.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:05.230 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.230 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.230 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:05.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496953+0000", 2024-03-27T21:40:05.230 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0709999999999996e-06 2024-03-27T21:40:05.230 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.231 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:05.231 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496961+0000", 2024-03-27T21:40:05.231 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1300000000000001e-06 2024-03-27T21:40:05.231 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.231 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.231 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.231 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496968+0000", 2024-03-27T21:40:05.231 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0999999999999998e-06 2024-03-27T21:40:05.231 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.231 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.231 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:05.231 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501244+0000", 2024-03-27T21:40:05.231 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042756310000000002 2024-03-27T21:40:05.231 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.231 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.232 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:05.232 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501377+0000", 2024-03-27T21:40:05.232 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013348799999999999 2024-03-27T21:40:05.232 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.232 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.232 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:05.232 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501455+0000", 2024-03-27T21:40:05.232 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7410000000000006e-05 2024-03-27T21:40:05.232 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.232 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.232 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:05.232 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502138+0000", 2024-03-27T21:40:05.232 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068386599999999999 2024-03-27T21:40:05.232 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.232 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.233 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:05.233 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503152+0000", 2024-03-27T21:40:05.233 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010139680000000001 2024-03-27T21:40:05.233 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.233 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.233 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:05.233 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503180+0000", 2024-03-27T21:40:05.233 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7426000000000001e-05 2024-03-27T21:40:05.233 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.233 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.233 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:05.233 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503202+0000", 2024-03-27T21:40:05.233 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2463000000000002e-05 2024-03-27T21:40:05.233 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.234 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:05.234 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.234 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.234 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:05.234 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:05.234 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 24.670929544, 2024-03-27T21:40:05.234 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066253680000000004, 2024-03-27T21:40:05.234 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:05.234 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.234 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:05.234 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:05.234 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:05.234 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 136 2024-03-27T21:40:05.234 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.235 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:05.235 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.235 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:05.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:05.235 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.235 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.235 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.235 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:05.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:05.235 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.235 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.235 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.235 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:05.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:05.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.236 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.236 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.236 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:05.236 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496977+0000", 2024-03-27T21:40:05.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5049999999999999e-06 2024-03-27T21:40:05.236 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.236 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.236 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:05.236 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496984+0000", 2024-03-27T21:40:05.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1520000000000003e-06 2024-03-27T21:40:05.236 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.236 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.236 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.237 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496990+0000", 2024-03-27T21:40:05.237 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-03-27T21:40:05.237 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.237 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.237 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:05.237 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501669+0000", 2024-03-27T21:40:05.237 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046793859999999998 2024-03-27T21:40:05.237 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.237 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.238 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:05.238 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501714+0000", 2024-03-27T21:40:05.238 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5154999999999997e-05 2024-03-27T21:40:05.238 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.238 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.238 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:05.238 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501783+0000", 2024-03-27T21:40:05.238 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8454999999999994e-05 2024-03-27T21:40:05.238 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.238 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.238 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:05.238 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503226+0000", 2024-03-27T21:40:05.238 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001443441 2024-03-27T21:40:05.238 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.239 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:05.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503540+0000", 2024-03-27T21:40:05.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031347100000000001 2024-03-27T21:40:05.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.239 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:05.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503563+0000", 2024-03-27T21:40:05.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3223000000000001e-05 2024-03-27T21:40:05.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.239 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:05.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503599+0000", 2024-03-27T21:40:05.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6105000000000001e-05 2024-03-27T21:40:05.240 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.240 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:05.240 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.240 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.240 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.240 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:05.240 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:05.240 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 24.670906733999999, 2024-03-27T21:40:05.240 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074261320000000002, 2024-03-27T21:40:05.240 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:05.240 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.240 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:05.240 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:05.240 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:05.240 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 137 2024-03-27T21:40:05.241 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.241 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:05.241 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.241 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:05.241 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:05.241 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.241 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.241 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.241 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:05.241 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:05.241 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.241 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.241 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.241 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:05.241 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:05.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:05.242 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497077+0000", 2024-03-27T21:40:05.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1599000000000003e-05 2024-03-27T21:40:05.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:05.242 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497085+0000", 2024-03-27T21:40:05.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6109999999999996e-06 2024-03-27T21:40:05.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.242 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497092+0000", 2024-03-27T21:40:05.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.742e-06 2024-03-27T21:40:05.243 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.243 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.243 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:05.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502159+0000", 2024-03-27T21:40:05.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050666870000000003 2024-03-27T21:40:05.243 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.243 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.243 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:05.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502200+0000", 2024-03-27T21:40:05.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1919999999999998e-05 2024-03-27T21:40:05.243 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.243 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.243 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:05.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502280+0000", 2024-03-27T21:40:05.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9417000000000002e-05 2024-03-27T21:40:05.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:05.244 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503605+0000", 2024-03-27T21:40:05.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001325641 2024-03-27T21:40:05.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:05.244 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504361+0000", 2024-03-27T21:40:05.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075555399999999997 2024-03-27T21:40:05.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:05.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504386+0000", 2024-03-27T21:40:05.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5369999999999999e-05 2024-03-27T21:40:05.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:05.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504422+0000", 2024-03-27T21:40:05.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6019000000000002e-05 2024-03-27T21:40:05.245 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.245 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:05.245 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.245 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:05.245 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:05.246 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 24.670802937000001, 2024-03-27T21:40:05.246 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074059750000000004, 2024-03-27T21:40:05.246 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:05.246 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.246 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:05.246 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:05.246 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:05.246 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-03-27T21:40:05.246 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.246 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:05.246 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.246 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:05.247 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:05.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.247 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.247 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:05.247 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:05.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.247 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.247 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:05.247 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:05.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:05.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497103+0000", 2024-03-27T21:40:05.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.856e-06 2024-03-27T21:40:05.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:05.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497121+0000", 2024-03-27T21:40:05.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7190999999999999e-05 2024-03-27T21:40:05.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497126+0000", 2024-03-27T21:40:05.249 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3770000000000004e-06 2024-03-27T21:40:05.249 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.249 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.249 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:05.249 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502833+0000", 2024-03-27T21:40:05.249 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057073510000000003 2024-03-27T21:40:05.249 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.249 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.249 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:05.249 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502881+0000", 2024-03-27T21:40:05.249 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7961999999999999e-05 2024-03-27T21:40:05.249 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.249 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.249 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:05.249 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502949+0000", 2024-03-27T21:40:05.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7957e-05 2024-03-27T21:40:05.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.250 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:05.250 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504430+0000", 2024-03-27T21:40:05.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001480763 2024-03-27T21:40:05.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.250 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:05.250 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504459+0000", 2024-03-27T21:40:05.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8983999999999999e-05 2024-03-27T21:40:05.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.250 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:05.251 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504486+0000", 2024-03-27T21:40:05.251 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6883999999999999e-05 2024-03-27T21:40:05.251 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.251 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.251 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:05.251 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504506+0000", 2024-03-27T21:40:05.251 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0214e-05 2024-03-27T21:40:05.251 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.251 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:05.251 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.251 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.251 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.251 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:05.252 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:05.252 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 24.670771103, 2024-03-27T21:40:05.252 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074754859999999999, 2024-03-27T21:40:05.252 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:05.252 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.252 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:05.252 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:05.252 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:05.252 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-03-27T21:40:05.252 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.252 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:05.252 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.252 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:05.252 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:05.253 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.253 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.253 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.253 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:05.253 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:05.253 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.253 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.253 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.253 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:05.253 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:05.253 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.253 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.253 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.253 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:05.253 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497137+0000", 2024-03-27T21:40:05.254 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0730000000000004e-06 2024-03-27T21:40:05.254 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.254 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:05.254 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497143+0000", 2024-03-27T21:40:05.254 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3759999999999996e-06 2024-03-27T21:40:05.254 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.254 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.255 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497150+0000", 2024-03-27T21:40:05.255 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.002e-06 2024-03-27T21:40:05.255 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.255 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.255 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:05.255 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503249+0000", 2024-03-27T21:40:05.255 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060989750000000004 2024-03-27T21:40:05.255 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.255 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.255 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:05.256 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503291+0000", 2024-03-27T21:40:05.256 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1563999999999997e-05 2024-03-27T21:40:05.256 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.256 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:05.256 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503376+0000", 2024-03-27T21:40:05.256 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5752000000000002e-05 2024-03-27T21:40:05.256 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.256 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:05.256 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504545+0000", 2024-03-27T21:40:05.256 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169172 2024-03-27T21:40:05.256 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.257 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:05.257 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504550+0000", 2024-03-27T21:40:05.257 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7419999999999997e-06 2024-03-27T21:40:05.257 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.257 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.257 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:05.257 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504574+0000", 2024-03-27T21:40:05.257 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4124000000000001e-05 2024-03-27T21:40:05.257 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.257 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.257 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:05.257 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504607+0000", 2024-03-27T21:40:05.257 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3118999999999999e-05 2024-03-27T21:40:05.257 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.258 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:05.258 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.258 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.258 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.258 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:05.258 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:05.258 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 24.670747436999999, 2024-03-27T21:40:05.258 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078150589999999992, 2024-03-27T21:40:05.258 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:05.258 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.258 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:05.258 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:05.258 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:05.258 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-03-27T21:40:05.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.259 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:05.259 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.259 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:05.259 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:05.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.259 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.259 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:05.259 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:05.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.259 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.259 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:05.259 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497155+0000", 2024-03-27T21:40:05.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.260 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.260 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.260 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:05.260 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497230+0000", 2024-03-27T21:40:05.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4789000000000005e-05 2024-03-27T21:40:05.260 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.260 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.260 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:05.260 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497241+0000", 2024-03-27T21:40:05.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1442e-05 2024-03-27T21:40:05.260 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.260 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.260 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.260 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497255+0000", 2024-03-27T21:40:05.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3087e-05 2024-03-27T21:40:05.261 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.261 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.261 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:05.261 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503630+0000", 2024-03-27T21:40:05.261 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063755269999999998 2024-03-27T21:40:05.261 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.261 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.261 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:05.261 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503673+0000", 2024-03-27T21:40:05.261 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2616e-05 2024-03-27T21:40:05.261 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.261 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.261 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:05.261 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503738+0000", 2024-03-27T21:40:05.261 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5114e-05 2024-03-27T21:40:05.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.262 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:05.262 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504907+0000", 2024-03-27T21:40:05.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169432 2024-03-27T21:40:05.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.262 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:05.262 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504926+0000", 2024-03-27T21:40:05.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9086999999999999e-05 2024-03-27T21:40:05.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.262 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:05.262 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504950+0000", 2024-03-27T21:40:05.263 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3598999999999999e-05 2024-03-27T21:40:05.263 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.263 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:05.263 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504971+0000", 2024-03-27T21:40:05.263 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0772999999999999e-05 2024-03-27T21:40:05.263 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.263 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:05.263 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.263 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.263 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:05.263 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:05.264 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 24.670640397, 2024-03-27T21:40:05.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082962590000000003, 2024-03-27T21:40:05.264 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:05.264 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.264 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:05.264 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:05.264 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:05.264 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-03-27T21:40:05.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.264 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:05.264 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.264 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:05.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:05.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.265 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:05.265 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:05.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.265 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:05.265 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497262+0000", 2024-03-27T21:40:05.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.266 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:05.266 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497267+0000", 2024-03-27T21:40:05.266 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.955e-06 2024-03-27T21:40:05.266 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.266 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.266 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:05.266 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497275+0000", 2024-03-27T21:40:05.266 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.824e-06 2024-03-27T21:40:05.266 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.266 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.266 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.266 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497280+0000", 2024-03-27T21:40:05.266 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8230000000000002e-06 2024-03-27T21:40:05.267 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.267 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.267 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:05.267 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504628+0000", 2024-03-27T21:40:05.267 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007348561 2024-03-27T21:40:05.267 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.267 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.267 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:05.267 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504673+0000", 2024-03-27T21:40:05.267 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4313000000000001e-05 2024-03-27T21:40:05.267 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.267 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.267 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:05.267 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504745+0000", 2024-03-27T21:40:05.268 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2303000000000002e-05 2024-03-27T21:40:05.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.268 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.268 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:05.268 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505358+0000", 2024-03-27T21:40:05.268 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061281300000000003 2024-03-27T21:40:05.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.268 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.268 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:05.268 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505503+0000", 2024-03-27T21:40:05.268 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000145134 2024-03-27T21:40:05.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.268 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.268 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:05.269 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505526+0000", 2024-03-27T21:40:05.269 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3312e-05 2024-03-27T21:40:05.269 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.269 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.269 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:05.269 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505559+0000", 2024-03-27T21:40:05.269 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2821999999999998e-05 2024-03-27T21:40:05.269 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.269 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:05.269 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.269 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.269 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.269 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:05.269 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:05.270 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 24.061418157999999, 2024-03-27T21:40:05.270 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053753339999999998, 2024-03-27T21:40:05.270 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:05.270 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.270 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:05.270 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:05.270 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:05.270 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159 2024-03-27T21:40:05.270 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.270 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:05.270 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.270 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:05.270 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:05.270 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.271 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.271 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.271 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:05.271 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:05.271 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:05.271 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.271 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.271 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:05.271 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:05.272 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.272 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.272 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.272 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:05.272 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106488+0000", 2024-03-27T21:40:05.272 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9830000000000001e-06 2024-03-27T21:40:05.272 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.272 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.272 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:05.272 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106491+0000", 2024-03-27T21:40:05.272 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3459999999999998e-06 2024-03-27T21:40:05.272 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.273 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.273 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.273 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106493+0000", 2024-03-27T21:40:05.273 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.509e-06 2024-03-27T21:40:05.273 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.273 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.273 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:05.273 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110636+0000", 2024-03-27T21:40:05.273 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041427260000000002 2024-03-27T21:40:05.273 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.273 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.273 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:05.273 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110683+0000", 2024-03-27T21:40:05.273 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6557000000000002e-05 2024-03-27T21:40:05.273 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.274 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.274 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:05.274 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110755+0000", 2024-03-27T21:40:05.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2119000000000005e-05 2024-03-27T21:40:05.274 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.274 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.274 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:05.274 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111415+0000", 2024-03-27T21:40:05.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066065199999999996 2024-03-27T21:40:05.274 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.274 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.274 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:05.274 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111793+0000", 2024-03-27T21:40:05.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037732100000000001 2024-03-27T21:40:05.274 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.275 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.275 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:05.275 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111829+0000", 2024-03-27T21:40:05.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418999999999998e-05 2024-03-27T21:40:05.275 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:05.275 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:05.275 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:05.275 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111860+0000", 2024-03-27T21:40:05.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1003000000000001e-05 2024-03-27T21:40:05.275 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.275 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:05.275 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.275 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:05.275 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:05.275 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:05.277 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:05.291+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:05.277 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:40:05.378 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:05.392+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:05.389 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:05.389 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:05.389 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:05.389 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:05.399 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:40:05.479 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:05.492+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:05.505 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:05.505 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:05.506 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:05.506 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:05.506 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:05.515 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:40:05.580 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:05.594+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:05.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.648+0000 7fdecdffb640 1 -- 172.21.15.60:0/2001375921 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v27) v1 ==== 87+0+0 (secure 0 0 0) 0x7fdec8057040 con 0x7fded813b020 2024-03-27T21:40:05.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.648+0000 7fdecdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:40:05.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.648+0000 7fdecdffb640 10 monclient: _finish_command 2 = system:0 noscrub is set 2024-03-27T21:40:05.635 INFO:teuthology.orchestra.run.smithi060.stderr:noscrub is set 2024-03-27T21:40:05.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.649+0000 7fcc26ffd640 1 -- 172.21.15.60:0/2177603658 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]=0 set osd.4 primary-affinity to 1 (8655362) v27) v1 ==== 133+0+0 (secure 0 0 0) 0x7fcc2c0170a0 con 0x7fcc3006f900 2024-03-27T21:40:05.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.649+0000 7fcc26ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}] 2024-03-27T21:40:05.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.649+0000 7fcc26ffd640 10 monclient: _finish_command 2 = system:0 set osd.4 primary-affinity to 1 (8655362) 2024-03-27T21:40:05.636 INFO:teuthology.orchestra.run.smithi060.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-27T21:40:05.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.650+0000 7fdedf599640 10 librados: watch_flush enter 2024-03-27T21:40:05.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.650+0000 7fdedf599640 10 librados: watch_flush exit 2024-03-27T21:40:05.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.650+0000 7fdedf599640 20 client.4512.objecter shutdown clearing up homeless session... 2024-03-27T21:40:05.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.650+0000 7fdedf599640 10 client.4512.objecter successfully canceled tick 2024-03-27T21:40:05.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.651+0000 7fdedf599640 1 -- 172.21.15.60:0/2001375921 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdebc06c590 msgr2=0x7fdebc06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:05.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.651+0000 7fdedf599640 1 --2- 172.21.15.60:0/2001375921 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdebc06c590 0x7fdebc06ea50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fded402b8e0 tx=0x7fded403c040 comp rx=0 tx=0).stop 2024-03-27T21:40:05.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.651+0000 7fdedf599640 10 monclient: shutdown 2024-03-27T21:40:05.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.651+0000 7fdedf599640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:05.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.651+0000 7fdedf599640 1 -- 172.21.15.60:0/2001375921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded813b020 msgr2=0x7fded8139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:05.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.651+0000 7fdedf599640 1 --2- 172.21.15.60:0/2001375921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded813b020 0x7fded8139000 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdec800c440 tx=0x7fdec8001e70 comp rx=0 tx=0).stop 2024-03-27T21:40:05.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.651+0000 7fdedf599640 1 -- 172.21.15.60:0/2001375921 shutdown_connections 2024-03-27T21:40:05.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.651+0000 7fdedf599640 1 --2- 172.21.15.60:0/2001375921 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdebc06c590 0x7fdebc06ea50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:05.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.651+0000 7fdedf599640 1 --2- 172.21.15.60:0/2001375921 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fded813b480 0x7fded8139540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:05.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.651+0000 7fdedf599640 1 --2- 172.21.15.60:0/2001375921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded813b020 0x7fded8139000 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:05.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.651+0000 7fdedf599640 1 --2- 172.21.15.60:0/2001375921 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fded80b40b0 0x7fded80afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:05.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.651+0000 7fdedf599640 1 -- 172.21.15.60:0/2001375921 >> 172.21.15.60:0/2001375921 conn(0x7fded80608e0 msgr2=0x7fded80b65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:05.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.651+0000 7fdedf599640 1 -- 172.21.15.60:0/2001375921 shutdown_connections 2024-03-27T21:40:05.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.651+0000 7fdedf599640 1 -- 172.21.15.60:0/2001375921 wait complete. 2024-03-27T21:40:05.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.651+0000 7fdedf599640 1 librados: shutdown 2024-03-27T21:40:05.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.652+0000 7fcc385e7640 10 librados: watch_flush enter 2024-03-27T21:40:05.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.652+0000 7fcc385e7640 10 librados: watch_flush exit 2024-03-27T21:40:05.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.652+0000 7fcc385e7640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-27T21:40:05.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.652+0000 7fcc385e7640 10 client.4495.objecter successfully canceled tick 2024-03-27T21:40:05.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.652+0000 7fcc385e7640 1 -- 172.21.15.60:0/2177603658 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcc1406c5c0 msgr2=0x7fcc1406ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:05.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.652+0000 7fcc385e7640 1 --2- 172.21.15.60:0/2177603658 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcc1406c5c0 0x7fcc1406ea80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcc20000c00 tx=0x7fcc2003c040 comp rx=0 tx=0).stop 2024-03-27T21:40:05.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.652+0000 7fcc385e7640 10 monclient: shutdown 2024-03-27T21:40:05.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.652+0000 7fcc385e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:05.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.652+0000 7fcc385e7640 1 -- 172.21.15.60:0/2177603658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc3006f900 msgr2=0x7fcc300a9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:05.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.652+0000 7fcc385e7640 1 --2- 172.21.15.60:0/2177603658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc3006f900 0x7fcc300a9940 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c00c450 tx=0x7fcc2c00c910 comp rx=0 tx=0).stop 2024-03-27T21:40:05.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.652+0000 7fcc385e7640 1 -- 172.21.15.60:0/2177603658 shutdown_connections 2024-03-27T21:40:05.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.652+0000 7fcc385e7640 1 --2- 172.21.15.60:0/2177603658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc30138bf0 0x7fcc30131c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:05.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.652+0000 7fcc385e7640 1 --2- 172.21.15.60:0/2177603658 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcc3006fce0 0x7fcc300ab720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:05.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.652+0000 7fcc385e7640 1 --2- 172.21.15.60:0/2177603658 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcc1406c5c0 0x7fcc1406ea80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:05.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.652+0000 7fcc385e7640 1 --2- 172.21.15.60:0/2177603658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc3006f900 0x7fcc300a9940 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:05.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.652+0000 7fcc385e7640 1 -- 172.21.15.60:0/2177603658 >> 172.21.15.60:0/2177603658 conn(0x7fcc300b4e50 msgr2=0x7fcc301384c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:05.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.652+0000 7fcc385e7640 1 -- 172.21.15.60:0/2177603658 shutdown_connections 2024-03-27T21:40:05.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.652+0000 7fcc385e7640 1 -- 172.21.15.60:0/2177603658 wait complete. 2024-03-27T21:40:05.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:05.652+0000 7fcc385e7640 1 librados: shutdown 2024-03-27T21:40:05.671 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:05.671 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:05.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:05.671 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-03-27T21:40:05.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.671 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:20 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2024-03-27T21:40:05.671 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:05.671 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.795108911, 2024-03-27T21:40:05.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015345630000000001, 2024-03-27T21:40:05.671 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:05.672 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.672 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:05.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.672 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:05.672 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:05.672 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.672 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.672 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:05.672 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:05.672 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.672 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.672 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:05.673 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888562+0000", 2024-03-27T21:40:05.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:05.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.673 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:05.673 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888735+0000", 2024-03-27T21:40:05.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017254799999999999 2024-03-27T21:40:05.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.673 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:05.673 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888747+0000", 2024-03-27T21:40:05.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2469999999999999e-05 2024-03-27T21:40:05.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.674 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.674 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888761+0000", 2024-03-27T21:40:05.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3254e-05 2024-03-27T21:40:05.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.674 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:05.674 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888804+0000", 2024-03-27T21:40:05.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3129e-05 2024-03-27T21:40:05.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.674 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:05.674 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888820+0000", 2024-03-27T21:40:05.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6426000000000002e-05 2024-03-27T21:40:05.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.675 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:05.675 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890060+0000", 2024-03-27T21:40:05.675 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012398050000000001 2024-03-27T21:40:05.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.675 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:05.675 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890098+0000", 2024-03-27T21:40:05.675 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7795e-05 2024-03-27T21:40:05.675 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.675 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:05.675 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.675 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2024-03-27T21:40:05.676 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:05.676 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.794284511000001, 2024-03-27T21:40:05.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001069555, 2024-03-27T21:40:05.676 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:05.676 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.676 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:05.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.676 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:05.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:05.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.676 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:05.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:05.677 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.677 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:05.677 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889387+0000", 2024-03-27T21:40:05.677 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:05.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.677 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:05.677 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889455+0000", 2024-03-27T21:40:05.677 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7930999999999997e-05 2024-03-27T21:40:05.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.677 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:05.677 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889470+0000", 2024-03-27T21:40:05.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5554e-05 2024-03-27T21:40:05.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.678 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889476+0000", 2024-03-27T21:40:05.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0619999999999997e-06 2024-03-27T21:40:05.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:05.678 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889513+0000", 2024-03-27T21:40:05.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6606999999999997e-05 2024-03-27T21:40:05.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:05.679 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889528+0000", 2024-03-27T21:40:05.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4982e-05 2024-03-27T21:40:05.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.679 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:05.679 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890421+0000", 2024-03-27T21:40:05.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00089293700000000001 2024-03-27T21:40:05.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.679 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:05.679 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890457+0000", 2024-03-27T21:40:05.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6260000000000002e-05 2024-03-27T21:40:05.679 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.679 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:05.680 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.680 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:38 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'24, mlcod=17'22)", 2024-03-27T21:40:05.680 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:05.680 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.777048545, 2024-03-27T21:40:05.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016182080000000001, 2024-03-27T21:40:05.680 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:05.680 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.680 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:05.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:05.680 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:05.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:05.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:05.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:05.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906623+0000", 2024-03-27T21:40:05.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:05.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:05.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906687+0000", 2024-03-27T21:40:05.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3706000000000003e-05 2024-03-27T21:40:05.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.682 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:05.682 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906698+0000", 2024-03-27T21:40:05.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1752e-05 2024-03-27T21:40:05.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.682 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.682 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906704+0000", 2024-03-27T21:40:05.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3940000000000004e-06 2024-03-27T21:40:05.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.683 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:05.683 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906741+0000", 2024-03-27T21:40:05.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6944999999999999e-05 2024-03-27T21:40:05.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.683 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:05.683 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906760+0000", 2024-03-27T21:40:05.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9293e-05 2024-03-27T21:40:05.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.683 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:05.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908211+0000", 2024-03-27T21:40:05.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014511350000000001 2024-03-27T21:40:05.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:05.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908242+0000", 2024-03-27T21:40:05.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0712e-05 2024-03-27T21:40:05.684 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.684 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:05.684 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.685 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:39 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'25, mlcod=17'22)", 2024-03-27T21:40:05.685 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:05.685 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.776203630000001, 2024-03-27T21:40:05.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012060199999999999, 2024-03-27T21:40:05.685 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:05.685 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.685 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:05.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:05.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:05.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.686 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:05.686 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:05.686 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.686 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:05.686 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:05.686 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:05.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.686 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:05.686 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907533+0000", 2024-03-27T21:40:05.686 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5675999999999999e-05 2024-03-27T21:40:05.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.687 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:05.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907545+0000", 2024-03-27T21:40:05.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.115e-05 2024-03-27T21:40:05.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.687 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907550+0000", 2024-03-27T21:40:05.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8320000000000002e-06 2024-03-27T21:40:05.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.687 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:05.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907592+0000", 2024-03-27T21:40:05.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1309e-05 2024-03-27T21:40:05.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:05.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907611+0000", 2024-03-27T21:40:05.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9009e-05 2024-03-27T21:40:05.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:05.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908638+0000", 2024-03-27T21:40:05.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001027533 2024-03-27T21:40:05.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:05.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908675+0000", 2024-03-27T21:40:05.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6329999999999999e-05 2024-03-27T21:40:05.689 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.689 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:05.689 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.689 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:57 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'35, mlcod=17'33)", 2024-03-27T21:40:05.689 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:05.689 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.762119198000001, 2024-03-27T21:40:05.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001346578, 2024-03-27T21:40:05.689 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:05.689 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.689 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:05.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:05.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:05.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:05.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:05.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:05.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921552+0000", 2024-03-27T21:40:05.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:05.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921615+0000", 2024-03-27T21:40:05.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2787999999999996e-05 2024-03-27T21:40:05.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:05.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921626+0000", 2024-03-27T21:40:05.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0888e-05 2024-03-27T21:40:05.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921635+0000", 2024-03-27T21:40:05.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4639999999999995e-06 2024-03-27T21:40:05.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:05.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921673+0000", 2024-03-27T21:40:05.692 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7891e-05 2024-03-27T21:40:05.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.692 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:05.692 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921694+0000", 2024-03-27T21:40:05.692 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0426000000000001e-05 2024-03-27T21:40:05.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.692 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:05.692 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922865+0000", 2024-03-27T21:40:05.692 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011709509999999999 2024-03-27T21:40:05.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.692 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:05.692 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922900+0000", 2024-03-27T21:40:05.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4819e-05 2024-03-27T21:40:05.693 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.693 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:05.693 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.693 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:71 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'45, mlcod=17'43)", 2024-03-27T21:40:05.693 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:05.693 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.746820486000001, 2024-03-27T21:40:05.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001460907, 2024-03-27T21:40:05.693 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:05.693 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.693 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:05.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.693 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:05.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:05.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:05.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:05.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:05.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936851+0000", 2024-03-27T21:40:05.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:05.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:05.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936903+0000", 2024-03-27T21:40:05.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2302999999999997e-05 2024-03-27T21:40:05.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:05.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936915+0000", 2024-03-27T21:40:05.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1967000000000001e-05 2024-03-27T21:40:05.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936923+0000", 2024-03-27T21:40:05.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9370000000000004e-06 2024-03-27T21:40:05.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:05.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936970+0000", 2024-03-27T21:40:05.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6987000000000003e-05 2024-03-27T21:40:05.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:05.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937002+0000", 2024-03-27T21:40:05.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2410999999999999e-05 2024-03-27T21:40:05.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:05.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938279+0000", 2024-03-27T21:40:05.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001276552 2024-03-27T21:40:05.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.697 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:05.697 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938313+0000", 2024-03-27T21:40:05.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3526999999999999e-05 2024-03-27T21:40:05.697 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.697 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:05.697 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.697 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:72 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'46, mlcod=17'43)", 2024-03-27T21:40:05.697 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:05.697 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.746182923999999, 2024-03-27T21:40:05.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014239490000000001, 2024-03-27T21:40:05.697 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:05.697 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.697 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:05.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:05.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:05.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:05.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:05.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:05.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937488+0000", 2024-03-27T21:40:05.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:05.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:05.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937547+0000", 2024-03-27T21:40:05.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8196999999999999e-05 2024-03-27T21:40:05.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:05.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937559+0000", 2024-03-27T21:40:05.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1928e-05 2024-03-27T21:40:05.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937564+0000", 2024-03-27T21:40:05.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4759999999999997e-06 2024-03-27T21:40:05.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:05.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937621+0000", 2024-03-27T21:40:05.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6712000000000002e-05 2024-03-27T21:40:05.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:05.701 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937640+0000", 2024-03-27T21:40:05.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9517e-05 2024-03-27T21:40:05.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.701 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:05.701 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938883+0000", 2024-03-27T21:40:05.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001242567 2024-03-27T21:40:05.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.701 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:05.701 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938913+0000", 2024-03-27T21:40:05.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0287e-05 2024-03-27T21:40:05.701 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.701 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:05.702 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.702 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:73 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'47, mlcod=17'43)", 2024-03-27T21:40:05.702 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:05.702 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.74550683, 2024-03-27T21:40:05.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010746239999999999, 2024-03-27T21:40:05.702 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:05.702 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.702 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:05.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.702 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:05.702 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:05.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:05.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:05.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:05.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:05.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:05.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:05.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938217+0000", 2024-03-27T21:40:05.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2743e-05 2024-03-27T21:40:05.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:05.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938229+0000", 2024-03-27T21:40:05.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1956999999999999e-05 2024-03-27T21:40:05.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938235+0000", 2024-03-27T21:40:05.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4589999999999997e-06 2024-03-27T21:40:05.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:05.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938341+0000", 2024-03-27T21:40:05.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000106054 2024-03-27T21:40:05.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:05.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938369+0000", 2024-03-27T21:40:05.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8291000000000001e-05 2024-03-27T21:40:05.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:05.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939209+0000", 2024-03-27T21:40:05.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084045400000000003 2024-03-27T21:40:05.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:05.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939240+0000", 2024-03-27T21:40:05.706 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0429999999999998e-05 2024-03-27T21:40:05.706 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.706 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:05.706 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.706 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.706 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:75 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'49, mlcod=17'47)", 2024-03-27T21:40:05.706 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:05.706 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.739069183000002, 2024-03-27T21:40:05.706 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001017001, 2024-03-27T21:40:05.706 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:05.706 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.706 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:05.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:05.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:05.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:05.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:05.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:05.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944601+0000", 2024-03-27T21:40:05.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999986 2024-03-27T21:40:05.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:05.708 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944614+0000", 2024-03-27T21:40:05.708 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2496e-05 2024-03-27T21:40:05.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.708 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:05.708 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944640+0000", 2024-03-27T21:40:05.708 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5865999999999999e-05 2024-03-27T21:40:05.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.708 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.708 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944651+0000", 2024-03-27T21:40:05.708 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1797000000000001e-05 2024-03-27T21:40:05.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.708 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:05.709 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944762+0000", 2024-03-27T21:40:05.709 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110412 2024-03-27T21:40:05.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.709 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:05.709 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944820+0000", 2024-03-27T21:40:05.709 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7649999999999999e-05 2024-03-27T21:40:05.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.709 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:05.709 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945585+0000", 2024-03-27T21:40:05.709 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076562600000000005 2024-03-27T21:40:05.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.710 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:05.710 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945620+0000", 2024-03-27T21:40:05.710 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4764000000000001e-05 2024-03-27T21:40:05.710 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.710 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:05.710 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.710 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.710 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:86 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'50)", 2024-03-27T21:40:05.710 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:05.710 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.731307696000002, 2024-03-27T21:40:05.710 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011392380000000001, 2024-03-27T21:40:05.710 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:05.710 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.710 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:05.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.711 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:05.711 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:05.711 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.711 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:05.711 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:05.711 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.711 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:05.711 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:05.711 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.712 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:05.712 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952373+0000", 2024-03-27T21:40:05.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9220000000000003e-06 2024-03-27T21:40:05.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.712 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:05.712 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952381+0000", 2024-03-27T21:40:05.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3070000000000003e-06 2024-03-27T21:40:05.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.712 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.712 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952385+0000", 2024-03-27T21:40:05.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.121e-06 2024-03-27T21:40:05.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:05.713 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952419+0000", 2024-03-27T21:40:05.713 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4082999999999999e-05 2024-03-27T21:40:05.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:05.713 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952440+0000", 2024-03-27T21:40:05.713 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0837999999999999e-05 2024-03-27T21:40:05.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:05.713 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953472+0000", 2024-03-27T21:40:05.713 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001032119 2024-03-27T21:40:05.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.714 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:05.714 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953504+0000", 2024-03-27T21:40:05.714 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1250000000000001e-05 2024-03-27T21:40:05.714 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.714 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:05.714 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.714 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:87 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-27T21:40:05.714 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:05.714 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.730940439000001, 2024-03-27T21:40:05.714 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00135519, 2024-03-27T21:40:05.715 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:05.715 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.715 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:05.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.715 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:05.715 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:05.715 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.715 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:05.715 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:05.715 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.716 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:05.716 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952731+0000", 2024-03-27T21:40:05.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.716 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:05.716 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952781+0000", 2024-03-27T21:40:05.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0161000000000003e-05 2024-03-27T21:40:05.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.716 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:05.717 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952794+0000", 2024-03-27T21:40:05.717 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2853e-05 2024-03-27T21:40:05.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.717 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.717 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952799+0000", 2024-03-27T21:40:05.717 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5560000000000001e-06 2024-03-27T21:40:05.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.717 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:05.717 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952832+0000", 2024-03-27T21:40:05.717 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3834999999999997e-05 2024-03-27T21:40:05.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.718 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:05.718 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952851+0000", 2024-03-27T21:40:05.718 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8289999999999999e-05 2024-03-27T21:40:05.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.718 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:05.718 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954055+0000", 2024-03-27T21:40:05.718 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012037949999999999 2024-03-27T21:40:05.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.718 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:05.718 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954087+0000", 2024-03-27T21:40:05.718 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2292999999999997e-05 2024-03-27T21:40:05.719 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.719 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:05.719 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.719 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.719 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:89 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'59, mlcod=17'57)", 2024-03-27T21:40:05.719 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:05.719 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.728666218000001, 2024-03-27T21:40:05.719 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00138893, 2024-03-27T21:40:05.719 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:05.719 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.719 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:05.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.719 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:05.719 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:05.720 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.720 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:05.720 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:05.720 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.720 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:05.720 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955005+0000", 2024-03-27T21:40:05.720 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.720 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:05.720 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955071+0000", 2024-03-27T21:40:05.721 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5290999999999995e-05 2024-03-27T21:40:05.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.721 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:05.721 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955081+0000", 2024-03-27T21:40:05.721 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0572e-05 2024-03-27T21:40:05.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.721 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.721 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955086+0000", 2024-03-27T21:40:05.721 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8380000000000001e-06 2024-03-27T21:40:05.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.721 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:05.721 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955122+0000", 2024-03-27T21:40:05.721 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5698000000000003e-05 2024-03-27T21:40:05.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.722 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:05.722 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955140+0000", 2024-03-27T21:40:05.722 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7900000000000001e-05 2024-03-27T21:40:05.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.722 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:05.722 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956366+0000", 2024-03-27T21:40:05.722 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001226471 2024-03-27T21:40:05.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.722 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:05.722 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956395+0000", 2024-03-27T21:40:05.723 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8782e-05 2024-03-27T21:40:05.723 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.723 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:05.723 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.723 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.723 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.723 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:90 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'60, mlcod=17'57)", 2024-03-27T21:40:05.723 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:05.723 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.727981641, 2024-03-27T21:40:05.723 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010778859999999999, 2024-03-27T21:40:05.723 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:05.723 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.723 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:05.723 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.723 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:05.724 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:05.724 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.724 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:05.724 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:05.724 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.724 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:05.724 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:05.724 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.724 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:05.724 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955747+0000", 2024-03-27T21:40:05.725 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7130999999999999e-05 2024-03-27T21:40:05.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.725 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:05.725 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955756+0000", 2024-03-27T21:40:05.725 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2750000000000001e-06 2024-03-27T21:40:05.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.725 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.725 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955761+0000", 2024-03-27T21:40:05.725 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3900000000000003e-06 2024-03-27T21:40:05.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.725 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:05.726 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955797+0000", 2024-03-27T21:40:05.726 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6650000000000003e-05 2024-03-27T21:40:05.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.726 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.726 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:05.726 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955816+0000", 2024-03-27T21:40:05.726 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8195000000000001e-05 2024-03-27T21:40:05.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.726 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.726 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:05.726 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956739+0000", 2024-03-27T21:40:05.726 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00092369700000000002 2024-03-27T21:40:05.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.727 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:05.727 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956768+0000", 2024-03-27T21:40:05.727 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9048e-05 2024-03-27T21:40:05.727 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.727 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:05.727 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.727 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.727 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:118 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'80, mlcod=17'78)", 2024-03-27T21:40:05.727 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:05.727 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.594990817999999, 2024-03-27T21:40:05.727 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014200669999999999, 2024-03-27T21:40:05.727 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:05.728 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.728 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:05.728 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.728 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:05.728 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:05.728 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.728 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.728 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.728 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:05.728 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:05.728 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.728 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.728 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.728 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:05.728 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:05.729 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.729 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.729 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.729 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:05.729 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088756+0000", 2024-03-27T21:40:05.729 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4902999999999998e-05 2024-03-27T21:40:05.729 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.729 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.729 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:05.729 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088764+0000", 2024-03-27T21:40:05.729 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8169999999999999e-06 2024-03-27T21:40:05.729 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.729 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.729 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.730 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088767+0000", 2024-03-27T21:40:05.730 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8310000000000003e-06 2024-03-27T21:40:05.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.730 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:05.730 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088798+0000", 2024-03-27T21:40:05.730 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0800999999999999e-05 2024-03-27T21:40:05.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.730 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:05.730 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088817+0000", 2024-03-27T21:40:05.730 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8467000000000001e-05 2024-03-27T21:40:05.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.730 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:05.731 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090074+0000", 2024-03-27T21:40:05.731 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012574089999999999 2024-03-27T21:40:05.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.731 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:05.731 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090101+0000", 2024-03-27T21:40:05.731 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7223e-05 2024-03-27T21:40:05.731 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.731 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:05.731 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.731 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:119 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'81, mlcod=17'78)", 2024-03-27T21:40:05.731 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:05.731 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.594256023, 2024-03-27T21:40:05.732 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014990870000000001, 2024-03-27T21:40:05.732 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:05.732 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.732 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:05.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.732 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:05.732 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:05.732 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.732 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.732 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:05.732 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:05.732 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.732 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.733 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:05.733 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:05.733 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.733 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:05.733 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089491+0000", 2024-03-27T21:40:05.733 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4805e-05 2024-03-27T21:40:05.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.733 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:05.733 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089498+0000", 2024-03-27T21:40:05.733 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0650000000000001e-06 2024-03-27T21:40:05.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.734 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.734 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.734 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089501+0000", 2024-03-27T21:40:05.734 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.54e-06 2024-03-27T21:40:05.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.734 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.734 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:05.734 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089533+0000", 2024-03-27T21:40:05.734 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1838999999999998e-05 2024-03-27T21:40:05.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.734 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.734 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:05.735 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089550+0000", 2024-03-27T21:40:05.735 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7465999999999998e-05 2024-03-27T21:40:05.735 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.735 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.735 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:05.735 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090890+0000", 2024-03-27T21:40:05.735 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013397979999999999 2024-03-27T21:40:05.735 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.736 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:05.736 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090915+0000", 2024-03-27T21:40:05.736 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4905e-05 2024-03-27T21:40:05.736 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.736 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:05.736 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.736 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:120 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'82, mlcod=17'78)", 2024-03-27T21:40:05.736 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:05.736 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.593456143000001, 2024-03-27T21:40:05.736 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001527093, 2024-03-27T21:40:05.736 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:05.736 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.737 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:05.737 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.737 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:05.737 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:05.737 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.737 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.737 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.737 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:05.737 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:05.737 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.737 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.737 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.737 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:05.737 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:05.737 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.738 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.738 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:05.738 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090267+0000", 2024-03-27T21:40:05.738 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1477000000000002e-05 2024-03-27T21:40:05.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.738 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.738 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:05.738 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090273+0000", 2024-03-27T21:40:05.738 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9519999999999999e-06 2024-03-27T21:40:05.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.738 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.738 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.738 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090276+0000", 2024-03-27T21:40:05.738 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.219e-06 2024-03-27T21:40:05.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.739 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.739 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:05.739 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090304+0000", 2024-03-27T21:40:05.739 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8073999999999999e-05 2024-03-27T21:40:05.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.739 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.739 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:05.739 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090316+0000", 2024-03-27T21:40:05.739 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2088e-05 2024-03-27T21:40:05.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.739 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.740 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:05.740 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091717+0000", 2024-03-27T21:40:05.740 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014003889999999999 2024-03-27T21:40:05.740 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.740 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.740 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:05.740 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091743+0000", 2024-03-27T21:40:05.740 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6234e-05 2024-03-27T21:40:05.740 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.740 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:05.740 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.740 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.740 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.740 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:121 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'83, mlcod=17'78)", 2024-03-27T21:40:05.741 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:05.741 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.592645778000001, 2024-03-27T21:40:05.741 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010226899999999999, 2024-03-27T21:40:05.741 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:05.741 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.741 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:05.741 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.741 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:05.741 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:05.741 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.741 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.741 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.741 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:05.741 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:05.741 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.742 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.742 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.742 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:05.742 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:05.742 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.742 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.742 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.742 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:05.742 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091142+0000", 2024-03-27T21:40:05.742 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011608300000000001 2024-03-27T21:40:05.742 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.742 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.742 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:05.742 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091151+0000", 2024-03-27T21:40:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2350000000000005e-06 2024-03-27T21:40:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091156+0000", 2024-03-27T21:40:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5419999999999999e-06 2024-03-27T21:40:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091191+0000", 2024-03-27T21:40:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5453000000000001e-05 2024-03-27T21:40:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:05.744 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091208+0000", 2024-03-27T21:40:05.744 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7119999999999999e-05 2024-03-27T21:40:05.744 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.744 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.744 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:05.744 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092024+0000", 2024-03-27T21:40:05.744 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000815539 2024-03-27T21:40:05.744 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.744 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.744 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:05.744 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092049+0000", 2024-03-27T21:40:05.744 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5171e-05 2024-03-27T21:40:05.744 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.744 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:05.745 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.745 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.745 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.745 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:142 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'100, mlcod=18'98)", 2024-03-27T21:40:05.745 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:05.745 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.177638823999999, 2024-03-27T21:40:05.745 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001198778, 2024-03-27T21:40:05.745 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:05.745 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.745 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:05.745 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.745 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:05.745 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:05.745 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.745 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.746 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.746 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:05.746 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:05.746 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.746 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.746 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.746 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:05.746 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:05.746 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.746 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.746 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.746 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:05.746 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506101+0000", 2024-03-27T21:40:05.746 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8202999999999997e-05 2024-03-27T21:40:05.747 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.747 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.747 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:05.747 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506109+0000", 2024-03-27T21:40:05.747 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3059999999999998e-06 2024-03-27T21:40:05.747 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.747 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.747 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.747 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506113+0000", 2024-03-27T21:40:05.747 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0119999999999999e-06 2024-03-27T21:40:05.747 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.747 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.747 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:05.747 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506137+0000", 2024-03-27T21:40:05.748 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4043999999999999e-05 2024-03-27T21:40:05.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.748 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.748 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:05.748 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506148+0000", 2024-03-27T21:40:05.748 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0446e-05 2024-03-27T21:40:05.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.748 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.748 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:05.748 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507213+0000", 2024-03-27T21:40:05.748 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001065215 2024-03-27T21:40:05.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.749 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:05.749 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507232+0000", 2024-03-27T21:40:05.749 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9038e-05 2024-03-27T21:40:05.749 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.750 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:05.750 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.750 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.750 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.750 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:143 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'101, mlcod=18'98)", 2024-03-27T21:40:05.750 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:05.750 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.177026205000001, 2024-03-27T21:40:05.750 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001357488, 2024-03-27T21:40:05.750 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:05.750 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.750 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:05.750 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.750 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:05.750 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:05.750 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.751 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:05.751 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:05.751 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.751 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:05.751 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:05.751 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.751 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:05.751 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506723+0000", 2024-03-27T21:40:05.752 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7404000000000006e-05 2024-03-27T21:40:05.752 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.752 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.752 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:05.752 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506730+0000", 2024-03-27T21:40:05.752 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5749999999999997e-06 2024-03-27T21:40:05.752 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.752 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.752 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.752 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506733+0000", 2024-03-27T21:40:05.752 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7979999999999999e-06 2024-03-27T21:40:05.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.753 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:05.753 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506758+0000", 2024-03-27T21:40:05.753 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4179e-05 2024-03-27T21:40:05.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.753 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:05.753 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506768+0000", 2024-03-27T21:40:05.753 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0740000000000001e-05 2024-03-27T21:40:05.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.753 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:05.753 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507978+0000", 2024-03-27T21:40:05.754 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012093900000000001 2024-03-27T21:40:05.754 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.754 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:05.754 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508003+0000", 2024-03-27T21:40:05.754 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5667e-05 2024-03-27T21:40:05.754 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.754 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:05.754 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.754 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.754 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:144 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'102, mlcod=18'98)", 2024-03-27T21:40:05.754 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:05.754 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 25.176367308, 2024-03-27T21:40:05.755 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013453390000000001, 2024-03-27T21:40:05.755 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:05.755 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:05.755 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:05.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.755 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:05.755 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:05.755 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.755 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:05.755 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:05.755 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:05.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.756 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:05.756 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507304+0000", 2024-03-27T21:40:05.756 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:05.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.756 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:05.756 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507397+0000", 2024-03-27T21:40:05.756 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2995e-05 2024-03-27T21:40:05.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.756 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:05.756 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507403+0000", 2024-03-27T21:40:05.756 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8640000000000001e-06 2024-03-27T21:40:05.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.757 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:05.757 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507407+0000", 2024-03-27T21:40:05.757 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5710000000000002e-06 2024-03-27T21:40:05.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.757 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:05.757 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507428+0000", 2024-03-27T21:40:05.757 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1192e-05 2024-03-27T21:40:05.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.757 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:05.757 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507438+0000", 2024-03-27T21:40:05.757 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8309999999999996e-06 2024-03-27T21:40:05.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.758 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:05.758 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508624+0000", 2024-03-27T21:40:05.758 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011861210000000001 2024-03-27T21:40:05.758 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:05.758 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:05.758 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:05.758 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508650+0000", 2024-03-27T21:40:05.758 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6118e-05 2024-03-27T21:40:05.758 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.758 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:05.758 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.758 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:05.758 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:05.758 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:05.760 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:40:05.760 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:05.773+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:05.861 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:05.875+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:05.936 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:05.937 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:05.937 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:05.937 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:05.946 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:40:05.962 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:05.976+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:06.063 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:06.077+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:06.097 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:06.097 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:06.097 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:06.098 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:06.098 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:06.106 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:40:06.166 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:06.180+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:06.221 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:06.221 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:06.221 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:06.221 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:40:06.221 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:06.230 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:40:06.267 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:06.281+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:06.370 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:06.385+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:06.378 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:06.378 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:06.379 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:06.379 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:06.388 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:40:06.471 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:06.486+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:06.503 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:06.503 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:06.503 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:06.503 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:06.503 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:06.512 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:40:06.574 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:06.588+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:06.628 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:06.629 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:06.629 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:06.629 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:40:06.629 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:06.639 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:40:06.684 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:06.698+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:06.756 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:06.756 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:06.757 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:06.757 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:06.765 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:40:06.807 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:06.821+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:06.877 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:06.890+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:06.913 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:06.913 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:06.913 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:06.914 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:06.914 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:06.922 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:40:06.978 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:06.991+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:07.026 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:07.026 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:07.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:07.027 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:07.027 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:07.035 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:40:07.079 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:07.094+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:07.119 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:40:07.141 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:07.142 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:07.142 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:07.142 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:07.150 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:40:07.191 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:07.205+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:07.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.242+0000 7f7d17e2f640 10 monclient: get_monmap_and_config 2024-03-27T21:40:07.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.242+0000 7f7d17e2f640 10 monclient: build_initial_monmap 2024-03-27T21:40:07.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.242+0000 7f7d17e2f640 10 monclient: monmap: 2024-03-27T21:40:07.230 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:07.230 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:07.230 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:07.243505+0000 2024-03-27T21:40:07.230 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:07.243505+0000 2024-03-27T21:40:07.231 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:07.231 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:07.231 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:07.231 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:07.231 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:07.231 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:07.232 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:07.232 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:07.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.243+0000 7f7d17e2f640 1 Processor -- start 2024-03-27T21:40:07.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.243+0000 7f7d17e2f640 1 -- start start 2024-03-27T21:40:07.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.243+0000 7f7d17e2f640 10 monclient: init 2024-03-27T21:40:07.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.243+0000 7f7d17e2f640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:07.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.243+0000 7f7d17e2f640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-03-27T21:40:07.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.243+0000 7f7d17e2f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7d100b40b0 0x7f7d100a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:07.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.243+0000 7f7d17e2f640 10 monclient(hunting): picked mon.noname-a con 0x7f7d100b40b0 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:07.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.243+0000 7f7d17e2f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7d1013b020 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:07.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.243+0000 7f7d17e2f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7d1013b480 0x7f7d10141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:07.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.243+0000 7f7d17e2f640 10 monclient(hunting): picked mon.noname-b con 0x7f7d1013b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:07.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.243+0000 7f7d17e2f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d100b3830 con 0x7f7d1013b020 2024-03-27T21:40:07.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.243+0000 7f7d17e2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:07.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.243+0000 7f7d17e2f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d1013b910 con 0x7f7d100b40b0 2024-03-27T21:40:07.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.243+0000 7f7d17e2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:07.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.243+0000 7f7d17e2f640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d100b4490 con 0x7f7d1013b480 2024-03-27T21:40:07.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.243+0000 7f7d17e2f640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:07.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.243+0000 7f7d17e2f640 10 monclient(hunting): authenticate will time out at 848.464849s 2024-03-27T21:40:07.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d15bcb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7d100b40b0 0x7f7d100a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:07.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d163cc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7d1013b480 0x7f7d10141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:07.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d15bcb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7d100b40b0 0x7f7d100a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52694/0 (socket says 172.21.15.60:52694) 2024-03-27T21:40:07.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d163cc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7d1013b480 0x7f7d10141e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:39452/0 (socket says 172.21.15.60:39452) 2024-03-27T21:40:07.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d15bcb640 1 -- 172.21.15.60:0/2106744372 learned_addr learned my addr 172.21.15.60:0/2106744372 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:07.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d15bcb640 10 monclient(hunting): get_auth_request con 0x7f7d100b40b0 auth_method 0 2024-03-27T21:40:07.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d15bcb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:07.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d15bcb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:07.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d15bcb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:07.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d163cc640 10 monclient(hunting): get_auth_request con 0x7f7d1013b480 auth_method 0 2024-03-27T21:40:07.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d163cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:07.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d163cc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:07.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d163cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:07.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d15bcb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:07.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d15bcb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:07.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d15bcb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:07.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d163cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:07.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d163cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:07.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d163cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:07.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d15bcb640 10 monclient(hunting): handle_auth_done global_id 4527 payload 274 2024-03-27T21:40:07.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d15bcb640 1 -- 172.21.15.60:0/2106744372 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7d1013b480 msgr2=0x7f7d10141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:07.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d15bcb640 1 --2- 172.21.15.60:0/2106744372 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7d1013b480 0x7f7d10141e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:07.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d15bcb640 1 -- 172.21.15.60:0/2106744372 >> v1:172.21.15.191:6789/0 conn(0x7f7d1013b020 legacy=0x7f7d101416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:07.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d15bcb640 10 monclient: _finish_hunting 0 2024-03-27T21:40:07.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d15bcb640 1 monclient: found mon.noname-a 2024-03-27T21:40:07.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d15bcb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:07.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d15bcb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:07.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d15bcb640 1 -- 172.21.15.60:0/2106744372 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d100aa520 con 0x7f7d100b40b0 2024-03-27T21:40:07.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.244+0000 7f7d163cc640 1 --2- 172.21.15.60:0/2106744372 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7d1013b480 0x7f7d10141e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:07.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d15bcb640 1 --2- 172.21.15.60:0/2106744372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d100b40b0 0x7f7d100a8b90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0098a0 tx=0x7f7d0c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d92b5f31fa293c server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:07.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d14bc9640 1 -- 172.21.15.60:0/2106744372 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c031640 con 0x7f7d100b40b0 2024-03-27T21:40:07.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d14bc9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:07.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d14bc9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:07.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d14bc9640 10 monclient: dump: 2024-03-27T21:40:07.248 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:07.248 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:07.249 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:07.249 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:07.249 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:07.249 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:07.249 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:07.249 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:07.249 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:07.249 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:07.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d14bc9640 10 monclient: _finish_auth 0 2024-03-27T21:40:07.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d14bc9640 10 monclient: _check_auth_tickets 2024-03-27T21:40:07.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d14bc9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:07.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d14bc9640 1 -- 172.21.15.60:0/2106744372 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0c0317a0 con 0x7f7d100b40b0 2024-03-27T21:40:07.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d17e2f640 5 monclient: authenticate success, global_id 4527 2024-03-27T21:40:07.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d17e2f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:07.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d14bc9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:07.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d17e2f640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:07.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d17e2f640 4 set_mon_vals no callback set 2024-03-27T21:40:07.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d14bc9640 1 -- 172.21.15.60:0/2106744372 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c031b30 con 0x7f7d100b40b0 2024-03-27T21:40:07.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d14bc9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:07.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d14bc9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:07.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d14bc9640 10 monclient: dump: 2024-03-27T21:40:07.252 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:07.252 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:07.252 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:07.252 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:07.252 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:07.252 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:07.253 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:07.253 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:07.253 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:07.253 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:07.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d17e2f640 10 monclient: shutdown 2024-03-27T21:40:07.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d17e2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:07.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d17e2f640 1 -- 172.21.15.60:0/2106744372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d100b40b0 msgr2=0x7f7d100a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:07.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d17e2f640 1 --2- 172.21.15.60:0/2106744372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d100b40b0 0x7f7d100a8b90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0098a0 tx=0x7f7d0c02b8f0 comp rx=0 tx=0).stop 2024-03-27T21:40:07.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d17e2f640 1 -- 172.21.15.60:0/2106744372 shutdown_connections 2024-03-27T21:40:07.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d17e2f640 1 --2- 172.21.15.60:0/2106744372 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7d1013b480 0x7f7d10141e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:07.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d17e2f640 1 --2- 172.21.15.60:0/2106744372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d100b40b0 0x7f7d100a8b90 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:07.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.245+0000 7f7d17e2f640 1 -- 172.21.15.60:0/2106744372 >> 172.21.15.60:0/2106744372 conn(0x7f7d100608e0 msgr2=0x7f7d10060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:07.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.246+0000 7f7d17e2f640 1 -- 172.21.15.60:0/2106744372 shutdown_connections 2024-03-27T21:40:07.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.246+0000 7f7d17e2f640 1 -- 172.21.15.60:0/2106744372 wait complete. 2024-03-27T21:40:07.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.246+0000 7f7d17e2f640 10 monclient: build_initial_monmap 2024-03-27T21:40:07.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.246+0000 7f7d17e2f640 10 monclient: monmap: 2024-03-27T21:40:07.255 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:07.255 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:07.255 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:07.255 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:07.255 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:07.255 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:07.256 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:07.256 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:07.256 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:07.256 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:07.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.246+0000 7f7d17e2f640 1 librados: starting msgr at 2024-03-27T21:40:07.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.246+0000 7f7d17e2f640 1 librados: starting objecter 2024-03-27T21:40:07.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d17e2f640 1 Processor -- start 2024-03-27T21:40:07.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d17e2f640 1 -- start start 2024-03-27T21:40:07.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d17e2f640 1 librados: setting wanted keys 2024-03-27T21:40:07.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d17e2f640 1 librados: calling monclient init 2024-03-27T21:40:07.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d17e2f640 10 monclient: init 2024-03-27T21:40:07.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d17e2f640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:07.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d17e2f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:40:07.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d17e2f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d100b40b0 0x7f7d1013e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:07.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d17e2f640 10 monclient(hunting): picked mon.noname-a con 0x7f7d100b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:07.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d17e2f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d1013b020 0x7f7d101d62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:07.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d17e2f640 10 monclient(hunting): picked mon.noname-b con 0x7f7d1013b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:07.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d17e2f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7d1013b480 0x7f7d1013bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:07.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d17e2f640 10 monclient(hunting): picked mon.noname-c con 0x7f7d1013b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:07.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d17e2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:07.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d17e2f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1013dd20 con 0x7f7d100b40b0 2024-03-27T21:40:07.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d17e2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:07.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d17e2f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1013e540 con 0x7f7d1013b020 2024-03-27T21:40:07.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d17e2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:07.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d17e2f640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1013e0d0 con 0x7f7d1013b480 2024-03-27T21:40:07.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d17e2f640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:07.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d17e2f640 10 monclient(hunting): authenticate will time out at 848.468539s 2024-03-27T21:40:07.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d15bcb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d100b40b0 0x7f7d1013e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:07.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d15bcb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d100b40b0 0x7f7d1013e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52708/0 (socket says 172.21.15.60:52708) 2024-03-27T21:40:07.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d15bcb640 1 -- 172.21.15.60:0/3018875909 learned_addr learned my addr 172.21.15.60:0/3018875909 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:07.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d15bcb640 10 monclient(hunting): get_auth_request con 0x7f7d100b40b0 auth_method 0 2024-03-27T21:40:07.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d15bcb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:07.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d15bcb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:07.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d163cc640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7d1013b480 0x7f7d1013bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:07.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d15bcb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:07.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d163cc640 10 monclient(hunting): get_auth_request con 0x7f7d1013b480 auth_method 0 2024-03-27T21:40:07.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d163cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:07.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d163cc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:07.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d163cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:07.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d153ca640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d1013b020 0x7f7d101d62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:07.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d15bcb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:07.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d15bcb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:07.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d15bcb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:07.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d163cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:07.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d163cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:07.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d163cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:07.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d153ca640 10 monclient(hunting): get_auth_request con 0x7f7d1013b020 auth_method 0 2024-03-27T21:40:07.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d153ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:07.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d153ca640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:07.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d153ca640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:07.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d153ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:07.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d153ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:07.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.247+0000 7f7d153ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:07.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d15bcb640 10 monclient(hunting): handle_auth_done global_id 4530 payload 824 2024-03-27T21:40:07.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d15bcb640 1 -- 172.21.15.60:0/3018875909 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7d1013b480 msgr2=0x7f7d1013bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:07.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d15bcb640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7d1013b480 0x7f7d1013bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:07.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d15bcb640 1 -- 172.21.15.60:0/3018875909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d1013b020 msgr2=0x7f7d101d62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:07.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d15bcb640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d1013b020 0x7f7d101d62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:07.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d15bcb640 10 monclient: _finish_hunting 0 2024-03-27T21:40:07.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d15bcb640 1 monclient: found mon.noname-a 2024-03-27T21:40:07.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d15bcb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:07.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d15bcb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:07.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d15bcb640 1 -- 172.21.15.60:0/3018875909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d08009a70 con 0x7f7d100b40b0 2024-03-27T21:40:07.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d163cc640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7d1013b480 0x7f7d1013bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:07.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d15bcb640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d100b40b0 0x7f7d1013e9d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c000c00 tx=0x7f7d0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:07.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d067fc640 10 client.?.objecter ms_handle_connect 0x7f7d100b40b0 2024-03-27T21:40:07.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d067fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:07.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d067fc640 1 -- 172.21.15.60:0/3018875909 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c03a930 con 0x7f7d100b40b0 2024-03-27T21:40:07.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:07.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:07.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d067fc640 10 monclient: dump: 2024-03-27T21:40:07.263 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:07.263 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:07.263 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:07.263 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:07.263 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:07.263 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:07.263 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:07.263 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:07.263 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:07.263 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:07.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d067fc640 10 monclient: _finish_auth 0 2024-03-27T21:40:07.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d067fc640 10 monclient: _check_auth_tickets 2024-03-27T21:40:07.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:07.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d067fc640 1 -- 172.21.15.60:0/3018875909 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0c03aab0 con 0x7f7d100b40b0 2024-03-27T21:40:07.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d17e2f640 5 monclient: authenticate success, global_id 4530 2024-03-27T21:40:07.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d17e2f640 10 monclient: _renew_subs 2024-03-27T21:40:07.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d17e2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:07.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d17e2f640 1 -- 172.21.15.60:0/3018875909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d0c0094d0 con 0x7f7d100b40b0 2024-03-27T21:40:07.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:07.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d067fc640 1 -- 172.21.15.60:0/3018875909 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c03ae60 con 0x7f7d100b40b0 2024-03-27T21:40:07.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:07.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:07.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d067fc640 10 monclient: dump: 2024-03-27T21:40:07.264 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:07.264 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:07.264 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:07.264 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:07.264 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:07.264 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:07.265 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:07.265 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:07.265 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:07.265 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:07.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d07fff640 4 set_mon_vals no callback set 2024-03-27T21:40:07.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d17e2f640 10 client.4530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:07.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d17e2f640 10 monclient: _renew_subs 2024-03-27T21:40:07.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d17e2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:07.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d17e2f640 1 -- 172.21.15.60:0/3018875909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d1013c210 con 0x7f7d100b40b0 2024-03-27T21:40:07.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.248+0000 7f7d17e2f640 1 librados: init done 2024-03-27T21:40:07.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.249+0000 7f7d17e2f640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:07.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.249+0000 7f7d17e2f640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:07.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.249+0000 7f7d17e2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:07.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.249+0000 7f7d17e2f640 1 -- 172.21.15.60:0/3018875909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f7d10069470 con 0x7f7d100b40b0 2024-03-27T21:40:07.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d067fc640 1 -- 172.21.15.60:0/3018875909 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7d0c02f050 con 0x7f7d100b40b0 2024-03-27T21:40:07.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d067fc640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f7cf406c590 0x7f7cf406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:07.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d067fc640 1 -- 172.21.15.60:0/3018875909 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f7d0c0ab3f0 con 0x7f7d100b40b0 2024-03-27T21:40:07.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d067fc640 10 client.4530.objecter ms_dispatch 0x7f7d10000b90 osd_map(27..27 src has 1..27) v4 2024-03-27T21:40:07.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d153ca640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f7cf406c590 0x7f7cf406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:07.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d067fc640 3 client.4530.objecter handle_osd_map got epochs [27,27] > 0 2024-03-27T21:40:07.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d067fc640 3 client.4530.objecter handle_osd_map decoding full epoch 27 2024-03-27T21:40:07.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d153ca640 10 monclient: get_auth_request con 0x7f7cf406c590 auth_method 0 2024-03-27T21:40:07.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d067fc640 20 client.4530.objecter dump_active .. 0 homeless 2024-03-27T21:40:07.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d067fc640 1 -- 172.21.15.60:0/3018875909 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v27) v1 ==== 54+0+16 (secure 0 0 0) 0x7f7d0c0ab840 con 0x7f7d100b40b0 2024-03-27T21:40:07.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:07.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d067fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:07.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d17e2f640 10 client.4530.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:07.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign -1 1 2024-03-27T21:40:07.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d17e2f640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f7d1013aba0 0x7f7d1013a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:07.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d101d69f0 osd=0 3 2024-03-27T21:40:07.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d101d69f0 osd=0 3 2024-03-27T21:40:07.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d101d69f0 osd=0 3 2024-03-27T21:40:07.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_remove -1 1 2024-03-27T21:40:07.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d17e2f640 20 client.4530.objecter get_session s=0x7f7d101d69f0 osd=0 3 2024-03-27T21:40:07.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign 0 1 2024-03-27T21:40:07.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d101d69f0 osd=0 4 2024-03-27T21:40:07.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d17e2f640 10 client.4530.objecter _send_command 1 2024-03-27T21:40:07.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.250+0000 7f7d17e2f640 1 -- 172.21.15.60:0/3018875909 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d100a90d0 con 0x7f7d1013aba0 2024-03-27T21:40:07.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.251+0000 7f7d163cc640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f7d1013aba0 0x7f7d1013a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:07.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.251+0000 7f7d153ca640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f7cf406c590 0x7f7cf406ea50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7d100b0fc0 tx=0x7f7d00009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:07.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.251+0000 7f7d067fc640 10 client.4530.objecter ms_handle_connect 0x7f7cf406c590 2024-03-27T21:40:07.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.251+0000 7f7d163cc640 10 monclient: get_auth_request con 0x7f7d1013aba0 auth_method 0 2024-03-27T21:40:07.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.251+0000 7f7d067fc640 10 client.4530.objecter ms_handle_connect 0x7f7d1013aba0 2024-03-27T21:40:07.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.251+0000 7f7d163cc640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f7d1013aba0 0x7f7d1013a230 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:07.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.252+0000 7f7d067fc640 1 -- 172.21.15.60:0/3018875909 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7d0800f040 con 0x7f7d1013aba0 2024-03-27T21:40:07.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.252+0000 7f7d067fc640 10 client.4530.objecter ms_dispatch 0x7f7d10000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:07.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.252+0000 7f7d067fc640 10 client.4530.objecter _finish_command 1 = system:0 2024-03-27T21:40:07.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.252+0000 7f7d067fc640 20 client.4530.objecter put_session s=0x7f7d101d69f0 osd=0 4 2024-03-27T21:40:07.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.252+0000 7f7d067fc640 15 client.4530.objecter _session_command_op_remove 0 1 2024-03-27T21:40:07.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.266+0000 7f7d17e2f640 10 client.4530.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:07.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.266+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign -1 2 2024-03-27T21:40:07.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.266+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d101d69f0 osd=0 3 2024-03-27T21:40:07.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.266+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d101d69f0 osd=0 3 2024-03-27T21:40:07.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.266+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d101d69f0 osd=0 3 2024-03-27T21:40:07.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.266+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_remove -1 2 2024-03-27T21:40:07.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.266+0000 7f7d17e2f640 20 client.4530.objecter get_session s=0x7f7d101d69f0 osd=0 3 2024-03-27T21:40:07.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.266+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign 0 2 2024-03-27T21:40:07.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.266+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d101d69f0 osd=0 4 2024-03-27T21:40:07.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.266+0000 7f7d17e2f640 10 client.4530.objecter _send_command 2 2024-03-27T21:40:07.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.266+0000 7f7d17e2f640 1 -- 172.21.15.60:0/3018875909 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7d10142340 con 0x7f7d1013aba0 2024-03-27T21:40:07.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.268+0000 7f7d067fc640 1 -- 172.21.15.60:0/3018875909 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7d0800e4f0 con 0x7f7d1013aba0 2024-03-27T21:40:07.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.268+0000 7f7d067fc640 10 client.4530.objecter ms_dispatch 0x7f7d10000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:07.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.268+0000 7f7d067fc640 10 client.4530.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:07.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.268+0000 7f7d067fc640 20 client.4530.objecter put_session s=0x7f7d101d69f0 osd=0 4 2024-03-27T21:40:07.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.268+0000 7f7d067fc640 15 client.4530.objecter _session_command_op_remove 0 2 2024-03-27T21:40:07.270 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:07.272 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:07.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.268+0000 7f7d17e2f640 10 client.4530.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:07.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.268+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign -1 3 2024-03-27T21:40:07.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.268+0000 7f7d17e2f640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f7d10065a90 0x7f7d10065e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:07.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.268+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d1014bd20 osd=1 3 2024-03-27T21:40:07.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.268+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d1014bd20 osd=1 3 2024-03-27T21:40:07.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.268+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d1014bd20 osd=1 3 2024-03-27T21:40:07.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.268+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_remove -1 3 2024-03-27T21:40:07.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.268+0000 7f7d17e2f640 20 client.4530.objecter get_session s=0x7f7d1014bd20 osd=1 3 2024-03-27T21:40:07.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.268+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign 1 3 2024-03-27T21:40:07.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.268+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d1014bd20 osd=1 4 2024-03-27T21:40:07.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.268+0000 7f7d17e2f640 10 client.4530.objecter _send_command 3 2024-03-27T21:40:07.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.268+0000 7f7d17e2f640 1 -- 172.21.15.60:0/3018875909 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d1013a770 con 0x7f7d10065a90 2024-03-27T21:40:07.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.268+0000 7f7d15bcb640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f7d10065a90 0x7f7d10065e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:07.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.268+0000 7f7d15bcb640 10 monclient: get_auth_request con 0x7f7d10065a90 auth_method 0 2024-03-27T21:40:07.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.268+0000 7f7d067fc640 10 client.4530.objecter ms_handle_connect 0x7f7d10065a90 2024-03-27T21:40:07.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.268+0000 7f7d15bcb640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f7d10065a90 0x7f7d10065e70 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:07.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.269+0000 7f7d067fc640 1 -- 172.21.15.60:0/3018875909 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7d1013a770 con 0x7f7d10065a90 2024-03-27T21:40:07.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.269+0000 7f7d067fc640 10 client.4530.objecter ms_dispatch 0x7f7d10000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:07.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.269+0000 7f7d067fc640 10 client.4530.objecter _finish_command 3 = system:0 2024-03-27T21:40:07.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.269+0000 7f7d067fc640 20 client.4530.objecter put_session s=0x7f7d1014bd20 osd=1 4 2024-03-27T21:40:07.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.269+0000 7f7d067fc640 15 client.4530.objecter _session_command_op_remove 1 3 2024-03-27T21:40:07.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.285+0000 7f7d17e2f640 10 client.4530.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:07.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.285+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign -1 4 2024-03-27T21:40:07.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.285+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d1014bd20 osd=1 3 2024-03-27T21:40:07.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.285+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d1014bd20 osd=1 3 2024-03-27T21:40:07.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.285+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d1014bd20 osd=1 3 2024-03-27T21:40:07.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.285+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_remove -1 4 2024-03-27T21:40:07.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.285+0000 7f7d17e2f640 20 client.4530.objecter get_session s=0x7f7d1014bd20 osd=1 3 2024-03-27T21:40:07.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.285+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign 1 4 2024-03-27T21:40:07.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.285+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d1014bd20 osd=1 4 2024-03-27T21:40:07.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.285+0000 7f7d17e2f640 10 client.4530.objecter _send_command 4 2024-03-27T21:40:07.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.285+0000 7f7d17e2f640 1 -- 172.21.15.60:0/3018875909 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7d101da4a0 con 0x7f7d10065a90 2024-03-27T21:40:07.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.287+0000 7f7d067fc640 1 -- 172.21.15.60:0/3018875909 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7d101da4a0 con 0x7f7d10065a90 2024-03-27T21:40:07.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.287+0000 7f7d067fc640 10 client.4530.objecter ms_dispatch 0x7f7d10000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:07.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.287+0000 7f7d067fc640 10 client.4530.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:07.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.287+0000 7f7d067fc640 20 client.4530.objecter put_session s=0x7f7d1014bd20 osd=1 4 2024-03-27T21:40:07.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.287+0000 7f7d067fc640 15 client.4530.objecter _session_command_op_remove 1 4 2024-03-27T21:40:07.276 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:07.278 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:07.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d17e2f640 10 client.4530.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:07.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign -1 5 2024-03-27T21:40:07.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d17e2f640 20 client.4530.objecter _get_session osd=-1 returning homeless 2024-03-27T21:40:07.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d17e2f640 10 client.4530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:07.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d17e2f640 10 monclient: _renew_subs 2024-03-27T21:40:07.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d17e2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:07.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d17e2f640 1 -- 172.21.15.60:0/3018875909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=28}) v3 -- 0x7f7d101d9a10 con 0x7f7d100b40b0 2024-03-27T21:40:07.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d17e2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:07.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d17e2f640 1 -- 172.21.15.60:0/3018875909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7d101da670 con 0x7f7d100b40b0 2024-03-27T21:40:07.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d067fc640 1 -- 172.21.15.60:0/3018875909 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d0c07e160 con 0x7f7d100b40b0 2024-03-27T21:40:07.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d067fc640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-03-27T21:40:07.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d07fff640 10 client.4530.objecter _check_command_map_dne tid 5 current 27 map_dne_bound 27 2024-03-27T21:40:07.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d07fff640 10 client.4530.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:40:07.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d07fff640 15 client.4530.objecter _session_command_op_remove -1 5 2024-03-27T21:40:07.279 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:40:07.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d17e2f640 10 client.4530.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:07.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign -1 6 2024-03-27T21:40:07.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d17e2f640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f7d1002b7f0 0x7f7d1002bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:07.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d1002b170 osd=3 3 2024-03-27T21:40:07.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d1002b170 osd=3 3 2024-03-27T21:40:07.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d1002b170 osd=3 3 2024-03-27T21:40:07.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_remove -1 6 2024-03-27T21:40:07.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d17e2f640 20 client.4530.objecter get_session s=0x7f7d1002b170 osd=3 3 2024-03-27T21:40:07.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign 3 6 2024-03-27T21:40:07.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d1002b170 osd=3 4 2024-03-27T21:40:07.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d17e2f640 10 client.4530.objecter _send_command 6 2024-03-27T21:40:07.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d17e2f640 1 -- 172.21.15.60:0/3018875909 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d1002c170 con 0x7f7d1002b7f0 2024-03-27T21:40:07.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d153ca640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f7d1002b7f0 0x7f7d1002bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:07.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d153ca640 10 monclient: get_auth_request con 0x7f7d1002b7f0 auth_method 0 2024-03-27T21:40:07.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d153ca640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f7d1002b7f0 0x7f7d1002bc30 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:07.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.288+0000 7f7d067fc640 10 client.4530.objecter ms_handle_connect 0x7f7d1002b7f0 2024-03-27T21:40:07.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.289+0000 7f7d067fc640 1 -- 172.21.15.60:0/3018875909 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7d1002c170 con 0x7f7d1002b7f0 2024-03-27T21:40:07.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.289+0000 7f7d067fc640 10 client.4530.objecter ms_dispatch 0x7f7d10000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:40:07.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.289+0000 7f7d067fc640 10 client.4530.objecter _finish_command 6 = system:0 2024-03-27T21:40:07.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.289+0000 7f7d067fc640 20 client.4530.objecter put_session s=0x7f7d1002b170 osd=3 4 2024-03-27T21:40:07.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.289+0000 7f7d067fc640 15 client.4530.objecter _session_command_op_remove 3 6 2024-03-27T21:40:07.282 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:07.297+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:07.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.304+0000 7f7d17e2f640 10 client.4530.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:07.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.304+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign -1 7 2024-03-27T21:40:07.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.304+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d1002b170 osd=3 3 2024-03-27T21:40:07.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.304+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d1002b170 osd=3 3 2024-03-27T21:40:07.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.304+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d1002b170 osd=3 3 2024-03-27T21:40:07.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.304+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_remove -1 7 2024-03-27T21:40:07.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.304+0000 7f7d17e2f640 20 client.4530.objecter get_session s=0x7f7d1002b170 osd=3 3 2024-03-27T21:40:07.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.304+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign 3 7 2024-03-27T21:40:07.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.304+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d1002b170 osd=3 4 2024-03-27T21:40:07.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.304+0000 7f7d17e2f640 10 client.4530.objecter _send_command 7 2024-03-27T21:40:07.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.304+0000 7f7d17e2f640 1 -- 172.21.15.60:0/3018875909 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7d1002c550 con 0x7f7d1002b7f0 2024-03-27T21:40:07.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.306+0000 7f7d067fc640 1 -- 172.21.15.60:0/3018875909 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7d1002c550 con 0x7f7d1002b7f0 2024-03-27T21:40:07.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.306+0000 7f7d067fc640 10 client.4530.objecter ms_dispatch 0x7f7d10000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:07.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.306+0000 7f7d067fc640 10 client.4530.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:07.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.306+0000 7f7d067fc640 20 client.4530.objecter put_session s=0x7f7d1002b170 osd=3 4 2024-03-27T21:40:07.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.306+0000 7f7d067fc640 15 client.4530.objecter _session_command_op_remove 3 7 2024-03-27T21:40:07.293 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:07.294 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:07.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.306+0000 7f7d17e2f640 10 client.4530.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:07.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.306+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign -1 8 2024-03-27T21:40:07.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.306+0000 7f7d17e2f640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f7d10066760 0x7f7d101e9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:07.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.306+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d100665f0 osd=4 3 2024-03-27T21:40:07.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.306+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d100665f0 osd=4 3 2024-03-27T21:40:07.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.306+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d100665f0 osd=4 3 2024-03-27T21:40:07.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.306+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_remove -1 8 2024-03-27T21:40:07.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.306+0000 7f7d17e2f640 20 client.4530.objecter get_session s=0x7f7d100665f0 osd=4 3 2024-03-27T21:40:07.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.306+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign 4 8 2024-03-27T21:40:07.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.306+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d100665f0 osd=4 4 2024-03-27T21:40:07.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.306+0000 7f7d17e2f640 10 client.4530.objecter _send_command 8 2024-03-27T21:40:07.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.306+0000 7f7d17e2f640 1 -- 172.21.15.60:0/3018875909 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d100b2620 con 0x7f7d10066760 2024-03-27T21:40:07.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.306+0000 7f7d163cc640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f7d10066760 0x7f7d101e9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:07.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.306+0000 7f7d163cc640 10 monclient: get_auth_request con 0x7f7d10066760 auth_method 0 2024-03-27T21:40:07.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.307+0000 7f7d067fc640 10 client.4530.objecter ms_handle_connect 0x7f7d10066760 2024-03-27T21:40:07.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.307+0000 7f7d163cc640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f7d10066760 0x7f7d101e9d00 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:07.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.308+0000 7f7d067fc640 1 -- 172.21.15.60:0/3018875909 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7d100b2620 con 0x7f7d10066760 2024-03-27T21:40:07.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.308+0000 7f7d067fc640 10 client.4530.objecter ms_dispatch 0x7f7d10000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:40:07.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.308+0000 7f7d067fc640 10 client.4530.objecter _finish_command 8 = system:0 2024-03-27T21:40:07.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.308+0000 7f7d067fc640 20 client.4530.objecter put_session s=0x7f7d100665f0 osd=4 4 2024-03-27T21:40:07.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.308+0000 7f7d067fc640 15 client.4530.objecter _session_command_op_remove 4 8 2024-03-27T21:40:07.296 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:07.296 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:07.296 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:07.296 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:07.296 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:07.305 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:40:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.324+0000 7f7d17e2f640 10 client.4530.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.324+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign -1 9 2024-03-27T21:40:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.324+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d100665f0 osd=4 3 2024-03-27T21:40:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.324+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d100665f0 osd=4 3 2024-03-27T21:40:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.324+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d100665f0 osd=4 3 2024-03-27T21:40:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.324+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_remove -1 9 2024-03-27T21:40:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.324+0000 7f7d17e2f640 20 client.4530.objecter get_session s=0x7f7d100665f0 osd=4 3 2024-03-27T21:40:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.324+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign 4 9 2024-03-27T21:40:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.324+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d100665f0 osd=4 4 2024-03-27T21:40:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.324+0000 7f7d17e2f640 10 client.4530.objecter _send_command 9 2024-03-27T21:40:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.324+0000 7f7d17e2f640 1 -- 172.21.15.60:0/3018875909 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7d1002c6c0 con 0x7f7d10066760 2024-03-27T21:40:07.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.326+0000 7f7d067fc640 1 -- 172.21.15.60:0/3018875909 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7d1002c6c0 con 0x7f7d10066760 2024-03-27T21:40:07.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.326+0000 7f7d067fc640 10 client.4530.objecter ms_dispatch 0x7f7d10000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:07.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.326+0000 7f7d067fc640 10 client.4530.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:07.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.326+0000 7f7d067fc640 20 client.4530.objecter put_session s=0x7f7d100665f0 osd=4 4 2024-03-27T21:40:07.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.326+0000 7f7d067fc640 15 client.4530.objecter _session_command_op_remove 4 9 2024-03-27T21:40:07.313 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:07.314 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:07.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.326+0000 7f7d17e2f640 10 client.4530.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.326+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign -1 10 2024-03-27T21:40:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.326+0000 7f7d17e2f640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f7d101ea750 0x7f7d101ecb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.326+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d1002c6c0 osd=5 3 2024-03-27T21:40:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.326+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d1002c6c0 osd=5 3 2024-03-27T21:40:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.326+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d1002c6c0 osd=5 3 2024-03-27T21:40:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.326+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_remove -1 10 2024-03-27T21:40:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.326+0000 7f7d17e2f640 20 client.4530.objecter get_session s=0x7f7d1002c6c0 osd=5 3 2024-03-27T21:40:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.326+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign 5 10 2024-03-27T21:40:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.326+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d1002c6c0 osd=5 4 2024-03-27T21:40:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.326+0000 7f7d17e2f640 10 client.4530.objecter _send_command 10 2024-03-27T21:40:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.326+0000 7f7d17e2f640 1 -- 172.21.15.60:0/3018875909 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d101ef910 con 0x7f7d101ea750 2024-03-27T21:40:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.326+0000 7f7d15bcb640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f7d101ea750 0x7f7d101ecb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.327+0000 7f7d15bcb640 10 monclient: get_auth_request con 0x7f7d101ea750 auth_method 0 2024-03-27T21:40:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.327+0000 7f7d067fc640 10 client.4530.objecter ms_handle_connect 0x7f7d101ea750 2024-03-27T21:40:07.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.327+0000 7f7d15bcb640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f7d101ea750 0x7f7d101ecb40 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:07.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.327+0000 7f7d067fc640 1 -- 172.21.15.60:0/3018875909 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7d101ef910 con 0x7f7d101ea750 2024-03-27T21:40:07.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.327+0000 7f7d067fc640 10 client.4530.objecter ms_dispatch 0x7f7d10000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:40:07.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.327+0000 7f7d067fc640 10 client.4530.objecter _finish_command 10 = system:0 2024-03-27T21:40:07.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.327+0000 7f7d067fc640 20 client.4530.objecter put_session s=0x7f7d1002c6c0 osd=5 4 2024-03-27T21:40:07.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.327+0000 7f7d067fc640 15 client.4530.objecter _session_command_op_remove 5 10 2024-03-27T21:40:07.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.343+0000 7f7d17e2f640 10 client.4530.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:07.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.343+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign -1 11 2024-03-27T21:40:07.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.343+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d1002c6c0 osd=5 3 2024-03-27T21:40:07.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.343+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d1002c6c0 osd=5 3 2024-03-27T21:40:07.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.343+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d1002c6c0 osd=5 3 2024-03-27T21:40:07.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.343+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_remove -1 11 2024-03-27T21:40:07.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.343+0000 7f7d17e2f640 20 client.4530.objecter get_session s=0x7f7d1002c6c0 osd=5 3 2024-03-27T21:40:07.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.343+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign 5 11 2024-03-27T21:40:07.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.343+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d1002c6c0 osd=5 4 2024-03-27T21:40:07.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.343+0000 7f7d17e2f640 10 client.4530.objecter _send_command 11 2024-03-27T21:40:07.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.343+0000 7f7d17e2f640 1 -- 172.21.15.60:0/3018875909 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7d101ed080 con 0x7f7d101ea750 2024-03-27T21:40:07.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.345+0000 7f7d067fc640 1 -- 172.21.15.60:0/3018875909 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7d101ed080 con 0x7f7d101ea750 2024-03-27T21:40:07.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.345+0000 7f7d067fc640 10 client.4530.objecter ms_dispatch 0x7f7d10000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:07.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.345+0000 7f7d067fc640 10 client.4530.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:07.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.345+0000 7f7d067fc640 20 client.4530.objecter put_session s=0x7f7d1002c6c0 osd=5 4 2024-03-27T21:40:07.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.345+0000 7f7d067fc640 15 client.4530.objecter _session_command_op_remove 5 11 2024-03-27T21:40:07.333 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:07.334 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:07.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.345+0000 7f7d17e2f640 10 client.4530.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:07.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.345+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign -1 12 2024-03-27T21:40:07.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.345+0000 7f7d17e2f640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f7d101ed700 0x7f7d101efaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:07.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.345+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d101ed080 osd=6 3 2024-03-27T21:40:07.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.345+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d101ed080 osd=6 3 2024-03-27T21:40:07.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.345+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d101ed080 osd=6 3 2024-03-27T21:40:07.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.345+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_remove -1 12 2024-03-27T21:40:07.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.345+0000 7f7d17e2f640 20 client.4530.objecter get_session s=0x7f7d101ed080 osd=6 3 2024-03-27T21:40:07.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.345+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign 6 12 2024-03-27T21:40:07.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.345+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d101ed080 osd=6 4 2024-03-27T21:40:07.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.345+0000 7f7d17e2f640 10 client.4530.objecter _send_command 12 2024-03-27T21:40:07.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.345+0000 7f7d17e2f640 1 -- 172.21.15.60:0/3018875909 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d101f28c0 con 0x7f7d101ed700 2024-03-27T21:40:07.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.345+0000 7f7d153ca640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f7d101ed700 0x7f7d101efaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:07.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.345+0000 7f7d153ca640 10 monclient: get_auth_request con 0x7f7d101ed700 auth_method 0 2024-03-27T21:40:07.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.345+0000 7f7d067fc640 10 client.4530.objecter ms_handle_connect 0x7f7d101ed700 2024-03-27T21:40:07.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.345+0000 7f7d153ca640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f7d101ed700 0x7f7d101efaf0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:07.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.346+0000 7f7d067fc640 1 -- 172.21.15.60:0/3018875909 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7d101f28c0 con 0x7f7d101ed700 2024-03-27T21:40:07.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.346+0000 7f7d067fc640 10 client.4530.objecter ms_dispatch 0x7f7d10000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:40:07.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.346+0000 7f7d067fc640 10 client.4530.objecter _finish_command 12 = system:0 2024-03-27T21:40:07.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.346+0000 7f7d067fc640 20 client.4530.objecter put_session s=0x7f7d101ed080 osd=6 4 2024-03-27T21:40:07.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.346+0000 7f7d067fc640 15 client.4530.objecter _session_command_op_remove 6 12 2024-03-27T21:40:07.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.361+0000 7f7d17e2f640 10 client.4530.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:07.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.361+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign -1 13 2024-03-27T21:40:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.361+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d101ed080 osd=6 3 2024-03-27T21:40:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.361+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d101ed080 osd=6 3 2024-03-27T21:40:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.361+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d101ed080 osd=6 3 2024-03-27T21:40:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.361+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_remove -1 13 2024-03-27T21:40:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.361+0000 7f7d17e2f640 20 client.4530.objecter get_session s=0x7f7d101ed080 osd=6 3 2024-03-27T21:40:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.361+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign 6 13 2024-03-27T21:40:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.361+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d101ed080 osd=6 4 2024-03-27T21:40:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.361+0000 7f7d17e2f640 10 client.4530.objecter _send_command 13 2024-03-27T21:40:07.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.361+0000 7f7d17e2f640 1 -- 172.21.15.60:0/3018875909 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7d101f0030 con 0x7f7d101ed700 2024-03-27T21:40:07.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.363+0000 7f7d067fc640 1 -- 172.21.15.60:0/3018875909 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7d101f0030 con 0x7f7d101ed700 2024-03-27T21:40:07.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.363+0000 7f7d067fc640 10 client.4530.objecter ms_dispatch 0x7f7d10000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:07.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.363+0000 7f7d067fc640 10 client.4530.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:07.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.363+0000 7f7d067fc640 20 client.4530.objecter put_session s=0x7f7d101ed080 osd=6 4 2024-03-27T21:40:07.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.363+0000 7f7d067fc640 15 client.4530.objecter _session_command_op_remove 6 13 2024-03-27T21:40:07.352 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:07.354 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:07.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.363+0000 7f7d17e2f640 10 client.4530.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:07.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.363+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign -1 14 2024-03-27T21:40:07.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.363+0000 7f7d17e2f640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f7d101f06b0 0x7f7d101f2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:07.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.363+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d101f0030 osd=7 3 2024-03-27T21:40:07.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.363+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d101f0030 osd=7 3 2024-03-27T21:40:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.363+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d101f0030 osd=7 3 2024-03-27T21:40:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.363+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_remove -1 14 2024-03-27T21:40:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.363+0000 7f7d17e2f640 20 client.4530.objecter get_session s=0x7f7d101f0030 osd=7 3 2024-03-27T21:40:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.363+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign 7 14 2024-03-27T21:40:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.363+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d101f0030 osd=7 4 2024-03-27T21:40:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.363+0000 7f7d17e2f640 10 client.4530.objecter _send_command 14 2024-03-27T21:40:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.363+0000 7f7d17e2f640 1 -- 172.21.15.60:0/3018875909 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d101f58c0 con 0x7f7d101f06b0 2024-03-27T21:40:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.363+0000 7f7d163cc640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f7d101f06b0 0x7f7d101f2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.363+0000 7f7d163cc640 10 monclient: get_auth_request con 0x7f7d101f06b0 auth_method 0 2024-03-27T21:40:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.363+0000 7f7d067fc640 10 client.4530.objecter ms_handle_connect 0x7f7d101f06b0 2024-03-27T21:40:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.363+0000 7f7d163cc640 1 --2- 172.21.15.60:0/3018875909 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f7d101f06b0 0x7f7d101f2aa0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.364+0000 7f7d067fc640 1 -- 172.21.15.60:0/3018875909 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7d101f58c0 con 0x7f7d101f06b0 2024-03-27T21:40:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.364+0000 7f7d067fc640 10 client.4530.objecter ms_dispatch 0x7f7d10000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:40:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.364+0000 7f7d067fc640 10 client.4530.objecter _finish_command 14 = system:0 2024-03-27T21:40:07.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.364+0000 7f7d067fc640 20 client.4530.objecter put_session s=0x7f7d101f0030 osd=7 4 2024-03-27T21:40:07.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.364+0000 7f7d067fc640 15 client.4530.objecter _session_command_op_remove 7 14 2024-03-27T21:40:07.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.381+0000 7f7d17e2f640 10 client.4530.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:07.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.381+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign -1 15 2024-03-27T21:40:07.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.381+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d101f0030 osd=7 3 2024-03-27T21:40:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.381+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d101f0030 osd=7 3 2024-03-27T21:40:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.381+0000 7f7d17e2f640 20 client.4530.objecter _get_session s=0x7f7d101f0030 osd=7 3 2024-03-27T21:40:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.381+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_remove -1 15 2024-03-27T21:40:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.381+0000 7f7d17e2f640 20 client.4530.objecter get_session s=0x7f7d101f0030 osd=7 3 2024-03-27T21:40:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.381+0000 7f7d17e2f640 15 client.4530.objecter _session_command_op_assign 7 15 2024-03-27T21:40:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.381+0000 7f7d17e2f640 20 client.4530.objecter put_session s=0x7f7d101f0030 osd=7 4 2024-03-27T21:40:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.381+0000 7f7d17e2f640 10 client.4530.objecter _send_command 15 2024-03-27T21:40:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.381+0000 7f7d17e2f640 1 -- 172.21.15.60:0/3018875909 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7d101f58c0 con 0x7f7d101f06b0 2024-03-27T21:40:07.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.382+0000 7f7d067fc640 1 -- 172.21.15.60:0/3018875909 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7d101f58c0 con 0x7f7d101f06b0 2024-03-27T21:40:07.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.382+0000 7f7d067fc640 10 client.4530.objecter ms_dispatch 0x7f7d10000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:07.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.382+0000 7f7d067fc640 10 client.4530.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:07.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.382+0000 7f7d067fc640 20 client.4530.objecter put_session s=0x7f7d101f0030 osd=7 4 2024-03-27T21:40:07.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.382+0000 7f7d067fc640 15 client.4530.objecter _session_command_op_remove 7 15 2024-03-27T21:40:07.369 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:07.370 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:07.372 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:40:07.372 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:40:07.390 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:07.404+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:07.450 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:07.451 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:07.459 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:40:07.484 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:07.498+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:07.585 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:07.599+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:07.604 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:07.604 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:07.613 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:40:07.653 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-27T21:40:07.697 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:07.711+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:07.718 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:07.718 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:07.726 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:40:07.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.771+0000 7f64f745f640 10 monclient: get_monmap_and_config 2024-03-27T21:40:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.771+0000 7f64f745f640 10 monclient: build_initial_monmap 2024-03-27T21:40:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.771+0000 7f64f745f640 10 monclient: monmap: 2024-03-27T21:40:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:07.772361+0000 2024-03-27T21:40:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:07.772361+0000 2024-03-27T21:40:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:07.759 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:07.759 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:07.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.771+0000 7f64f745f640 1 Processor -- start 2024-03-27T21:40:07.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f745f640 1 -- start start 2024-03-27T21:40:07.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f745f640 10 monclient: init 2024-03-27T21:40:07.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f745f640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:07.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f745f640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-03-27T21:40:07.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f745f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f64f00b40b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:07.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f745f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f64f013b020 0x7f64f013b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:07.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f745f640 10 monclient(hunting): picked mon.noname-c con 0x7f64f013b020 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:07.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f745f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f01416f0 0x7f64f0141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:07.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f745f640 10 monclient(hunting): picked mon.noname-b con 0x7f64f01416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:07.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f745f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f00b3830 con 0x7f64f00b40b0 2024-03-27T21:40:07.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f745f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:07.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f745f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f00b4490 con 0x7f64f013b020 2024-03-27T21:40:07.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f745f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:07.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f745f640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f00aa520 con 0x7f64f01416f0 2024-03-27T21:40:07.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f745f640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:07.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f745f640 10 monclient(hunting): authenticate will time out at 848.993751s 2024-03-27T21:40:07.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f51fb640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f64f00b40b0 0x7f64f00a8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42046/0 (socket says 172.21.15.60:42046) 2024-03-27T21:40:07.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f59fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f01416f0 0x7f64f0141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:07.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f51fb640 1 -- 172.21.15.60:0/3555780346 learned_addr learned my addr 172.21.15.60:0/3555780346 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:40:07.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f59fc640 10 monclient(hunting): get_auth_request con 0x7f64f01416f0 auth_method 0 2024-03-27T21:40:07.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f59fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:07.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.772+0000 7f64f59fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:07.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64f59fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:07.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64f49fa640 1 --2- 172.21.15.60:0/3555780346 >> v2:172.21.15.191:3300/0 conn(0x7f64f013b020 0x7f64f013b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:07.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64f59fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:07.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64f59fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:07.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64f59fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:07.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64f49fa640 10 monclient(hunting): get_auth_request con 0x7f64f013b020 auth_method 0 2024-03-27T21:40:07.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64f49fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:07.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64f49fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:07.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64f49fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:07.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64f49fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:07.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64f49fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:07.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64f49fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:07.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64e7fff640 1 -- 172.21.15.60:0/3555780346 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630630510 0 0) 0x7f64f00b3830 con 0x7f64f00b40b0 2024-03-27T21:40:07.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64f59fc640 10 monclient(hunting): handle_auth_done global_id 4451 payload 274 2024-03-27T21:40:07.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64f59fc640 1 -- 172.21.15.60:0/3555780346 >> v2:172.21.15.191:3300/0 conn(0x7f64f013b020 msgr2=0x7f64f013b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:07.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64f59fc640 1 --2- 172.21.15.60:0/3555780346 >> v2:172.21.15.191:3300/0 conn(0x7f64f013b020 0x7f64f013b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:07.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64f59fc640 1 -- 172.21.15.60:0/3555780346 >> v1:172.21.15.60:6789/0 conn(0x7f64f00b40b0 legacy=0x7f64f00a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:07.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64f59fc640 10 monclient: _finish_hunting 0 2024-03-27T21:40:07.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64f59fc640 1 monclient: found mon.noname-b 2024-03-27T21:40:07.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64f59fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:07.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64f59fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:07.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64f59fc640 1 -- 172.21.15.60:0/3555780346 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f00a9470 con 0x7f64f01416f0 2024-03-27T21:40:07.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.773+0000 7f64e7fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:40:07.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.774+0000 7f64f59fc640 1 --2- 172.21.15.60:0/3555780346 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f01416f0 0x7f64f0141b80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f64e8009d90 tx=0x7f64e802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7e2fd34af32ceab server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:07.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.774+0000 7f64e7fff640 1 -- 172.21.15.60:0/3555780346 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e802cc50 con 0x7f64f01416f0 2024-03-27T21:40:07.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.774+0000 7f64e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:07.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.774+0000 7f64e7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:07.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.774+0000 7f64e7fff640 10 monclient: dump: 2024-03-27T21:40:07.772 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:07.772 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:07.772 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:07.772 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:07.772 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:07.772 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:07.773 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:07.773 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:07.773 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:07.773 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:07.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.774+0000 7f64e7fff640 10 monclient: _finish_auth 0 2024-03-27T21:40:07.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.774+0000 7f64e7fff640 10 monclient: _check_auth_tickets 2024-03-27T21:40:07.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.774+0000 7f64e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:07.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.774+0000 7f64e7fff640 1 -- 172.21.15.60:0/3555780346 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e802cdb0 con 0x7f64f01416f0 2024-03-27T21:40:07.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.774+0000 7f64f745f640 5 monclient: authenticate success, global_id 4451 2024-03-27T21:40:07.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.774+0000 7f64f745f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:07.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.774+0000 7f64e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:07.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.774+0000 7f64e7fff640 1 -- 172.21.15.60:0/3555780346 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e8034660 con 0x7f64f01416f0 2024-03-27T21:40:07.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.774+0000 7f64f745f640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:07.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.774+0000 7f64f745f640 4 set_mon_vals no callback set 2024-03-27T21:40:07.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.774+0000 7f64e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:07.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.774+0000 7f64e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:07.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.774+0000 7f64e7fff640 10 monclient: dump: 2024-03-27T21:40:07.774 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:07.774 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:07.774 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:07.774 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:07.774 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:07.774 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:07.774 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:07.774 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:07.774 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:07.774 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:07.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.775+0000 7f64f745f640 10 monclient: shutdown 2024-03-27T21:40:07.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.775+0000 7f64f745f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:07.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.775+0000 7f64f745f640 1 -- 172.21.15.60:0/3555780346 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f01416f0 msgr2=0x7f64f0141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:07.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.775+0000 7f64f745f640 1 --2- 172.21.15.60:0/3555780346 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f01416f0 0x7f64f0141b80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f64e8009d90 tx=0x7f64e802bea0 comp rx=0 tx=0).stop 2024-03-27T21:40:07.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.775+0000 7f64f745f640 1 -- 172.21.15.60:0/3555780346 shutdown_connections 2024-03-27T21:40:07.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.775+0000 7f64f745f640 1 --2- 172.21.15.60:0/3555780346 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f01416f0 0x7f64f0141b80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:07.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.775+0000 7f64f745f640 1 --2- 172.21.15.60:0/3555780346 >> v2:172.21.15.191:3300/0 conn(0x7f64f013b020 0x7f64f013b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:07.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.775+0000 7f64f745f640 1 -- 172.21.15.60:0/3555780346 >> 172.21.15.60:0/3555780346 conn(0x7f64f00608e0 msgr2=0x7f64f0060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:07.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.775+0000 7f64f745f640 1 -- 172.21.15.60:0/3555780346 shutdown_connections 2024-03-27T21:40:07.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.775+0000 7f64f745f640 1 -- 172.21.15.60:0/3555780346 wait complete. 2024-03-27T21:40:07.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.776+0000 7f64f745f640 10 monclient: build_initial_monmap 2024-03-27T21:40:07.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.776+0000 7f64f745f640 10 monclient: monmap: 2024-03-27T21:40:07.775 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:07.775 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:07.775 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:07.776 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:07.776 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:07.776 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:07.776 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:07.776 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:07.776 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:07.776 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:07.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.776+0000 7f64f745f640 1 librados: starting msgr at 2024-03-27T21:40:07.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.776+0000 7f64f745f640 1 librados: starting objecter 2024-03-27T21:40:07.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.776+0000 7f64f745f640 1 Processor -- start 2024-03-27T21:40:07.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.776+0000 7f64f745f640 1 -- start start 2024-03-27T21:40:07.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.776+0000 7f64f745f640 1 librados: setting wanted keys 2024-03-27T21:40:07.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.776+0000 7f64f745f640 1 librados: calling monclient init 2024-03-27T21:40:07.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.776+0000 7f64f745f640 10 monclient: init 2024-03-27T21:40:07.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f745f640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f745f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:40:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f745f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64f00b40b0 0x7f64f013a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f745f640 10 monclient(hunting): picked mon.noname-b con 0x7f64f00b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f745f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f013b020 0x7f64f01342d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f745f640 10 monclient(hunting): picked mon.noname-c con 0x7f64f013b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f745f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f01416f0 0x7f64f0134810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f745f640 10 monclient(hunting): picked mon.noname-a con 0x7f64f01416f0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f745f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f745f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f013ea90 con 0x7f64f01416f0 2024-03-27T21:40:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f745f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f745f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f013e600 con 0x7f64f00b40b0 2024-03-27T21:40:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f745f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f745f640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f013d900 con 0x7f64f013b020 2024-03-27T21:40:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f745f640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f745f640 10 monclient(hunting): authenticate will time out at 848.998060s 2024-03-27T21:40:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f49fa640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f013b020 0x7f64f01342d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f49fa640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f013b020 0x7f64f01342d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:39478/0 (socket says 172.21.15.60:39478) 2024-03-27T21:40:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f49fa640 1 -- 172.21.15.60:0/3564859150 learned_addr learned my addr 172.21.15.60:0/3564859150 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f49fa640 10 monclient(hunting): get_auth_request con 0x7f64f013b020 auth_method 0 2024-03-27T21:40:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f49fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f49fa640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f49fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f59fc640 1 --2- 172.21.15.60:0/3564859150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f01416f0 0x7f64f0134810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f51fb640 1 --2- 172.21.15.60:0/3564859150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64f00b40b0 0x7f64f013a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f59fc640 10 monclient(hunting): get_auth_request con 0x7f64f01416f0 auth_method 0 2024-03-27T21:40:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f59fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f59fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f59fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:07.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f51fb640 10 monclient(hunting): get_auth_request con 0x7f64f00b40b0 auth_method 0 2024-03-27T21:40:07.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f51fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:07.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f51fb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:07.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f51fb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:07.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f49fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:07.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f49fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:07.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f49fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:07.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f59fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:07.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f59fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:07.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f59fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:07.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f51fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:07.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f51fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:07.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f51fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:07.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f49fa640 10 monclient(hunting): handle_auth_done global_id 4454 payload 824 2024-03-27T21:40:07.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f49fa640 1 -- 172.21.15.60:0/3564859150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64f00b40b0 msgr2=0x7f64f013a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:07.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f49fa640 1 --2- 172.21.15.60:0/3564859150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64f00b40b0 0x7f64f013a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:07.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f49fa640 1 -- 172.21.15.60:0/3564859150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f01416f0 msgr2=0x7f64f0134810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:07.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f49fa640 1 --2- 172.21.15.60:0/3564859150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f01416f0 0x7f64f0134810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:07.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f49fa640 10 monclient: _finish_hunting 0 2024-03-27T21:40:07.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f49fa640 1 monclient: found mon.noname-c 2024-03-27T21:40:07.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f49fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:07.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f49fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:40:07.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f49fa640 1 -- 172.21.15.60:0/3564859150 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e8009a70 con 0x7f64f013b020 2024-03-27T21:40:07.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f59fc640 1 --2- 172.21.15.60:0/3564859150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f01416f0 0x7f64f0134810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:07.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f51fb640 1 --2- 172.21.15.60:0/3564859150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64f00b40b0 0x7f64f013a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:07.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f49fa640 1 --2- 172.21.15.60:0/3564859150 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f013b020 0x7f64f01342d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f64e000e9e0 tx=0x7f64e000eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:07.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64e5ffb640 10 client.?.objecter ms_handle_connect 0x7f64f013b020 2024-03-27T21:40:07.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:07.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64e5ffb640 1 -- 172.21.15.60:0/3564859150 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e0009670 con 0x7f64f013b020 2024-03-27T21:40:07.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:07.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64e5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:07.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64e5ffb640 10 monclient: dump: 2024-03-27T21:40:07.781 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:07.781 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:07.781 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:07.781 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:07.781 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:07.782 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:07.782 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:07.782 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:07.782 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:07.782 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:07.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64e5ffb640 10 monclient: _finish_auth 0 2024-03-27T21:40:07.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64e5ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:40:07.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:07.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64e5ffb640 1 -- 172.21.15.60:0/3564859150 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e000c9d0 con 0x7f64f013b020 2024-03-27T21:40:07.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:07.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64e5ffb640 1 -- 172.21.15.60:0/3564859150 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e000cd80 con 0x7f64f013b020 2024-03-27T21:40:07.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f745f640 5 monclient: authenticate success, global_id 4454 2024-03-27T21:40:07.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f745f640 10 monclient: _renew_subs 2024-03-27T21:40:07.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f745f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:07.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64f745f640 1 -- 172.21.15.60:0/3564859150 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f013e170 con 0x7f64f013b020 2024-03-27T21:40:07.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.777+0000 7f64e77fe640 4 set_mon_vals no callback set 2024-03-27T21:40:07.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.778+0000 7f64e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:07.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.778+0000 7f64e5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:07.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.778+0000 7f64e5ffb640 10 monclient: dump: 2024-03-27T21:40:07.783 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:07.783 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:07.783 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:07.783 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:07.783 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:07.783 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:07.783 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:07.783 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:07.783 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:07.783 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:07.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.778+0000 7f64f745f640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:07.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.778+0000 7f64f745f640 10 monclient: _renew_subs 2024-03-27T21:40:07.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.778+0000 7f64f745f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:07.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.778+0000 7f64f745f640 1 -- 172.21.15.60:0/3564859150 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f00aac00 con 0x7f64f013b020 2024-03-27T21:40:07.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.778+0000 7f64f745f640 1 librados: init done 2024-03-27T21:40:07.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.779+0000 7f64f745f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:07.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.779+0000 7f64f745f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:07.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.779+0000 7f64f745f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:07.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.779+0000 7f64f745f640 1 -- 172.21.15.60:0/3564859150 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f013e390 con 0x7f64f013b020 2024-03-27T21:40:07.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.780+0000 7f64e5ffb640 1 -- 172.21.15.60:0/3564859150 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f64e0013470 con 0x7f64f013b020 2024-03-27T21:40:07.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.780+0000 7f64e5ffb640 1 --2- 172.21.15.60:0/3564859150 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f64d406c5c0 0x7f64d406ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:07.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.780+0000 7f64e5ffb640 1 -- 172.21.15.60:0/3564859150 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f64e00892d0 con 0x7f64f013b020 2024-03-27T21:40:07.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.780+0000 7f64e5ffb640 10 client.4454.objecter ms_dispatch 0x7f64f0000b90 osd_map(27..27 src has 1..27) v4 2024-03-27T21:40:07.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.780+0000 7f64e5ffb640 3 client.4454.objecter handle_osd_map got epochs [27,27] > 0 2024-03-27T21:40:07.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.780+0000 7f64e5ffb640 3 client.4454.objecter handle_osd_map decoding full epoch 27 2024-03-27T21:40:07.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.780+0000 7f64f51fb640 1 --2- 172.21.15.60:0/3564859150 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f64d406c5c0 0x7f64d406ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:07.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.780+0000 7f64f51fb640 10 monclient: get_auth_request con 0x7f64d406c5c0 auth_method 0 2024-03-27T21:40:07.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.780+0000 7f64e5ffb640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-27T21:40:07.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.781+0000 7f64f51fb640 1 --2- 172.21.15.60:0/3564859150 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f64d406c5c0 0x7f64d406ea80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f64ec007fd0 tx=0x7f64ec007de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:07.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.781+0000 7f64e5ffb640 10 client.4454.objecter ms_handle_connect 0x7f64d406c5c0 2024-03-27T21:40:07.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.785+0000 7f64e5ffb640 1 -- 172.21.15.60:0/3564859150 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f64e0054740 con 0x7f64f013b020 2024-03-27T21:40:07.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.785+0000 7f64e5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:07.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.785+0000 7f64e5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:07.786 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:07.800+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:07.830 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:07.830 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:07.839 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:40:07.887 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:07.901+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:07.943 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:07.943 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:07.952 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:40:07.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.979+0000 7f64f745f640 10 start_mon_command cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:40:07.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.979+0000 7f64f745f640 10 monclient: _send_command 2 [{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:40:07.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.979+0000 7f64f745f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:07.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:07.979+0000 7f64f745f640 1 -- 172.21.15.60:0/3564859150 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f64f013e770 con 0x7f64f013b020 2024-03-27T21:40:07.999 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:08.013+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:08.058 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:08.058 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:08.067 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:40:08.108 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:08.122+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:08.190 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:08.204+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:08.210 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:08.211 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:08.219 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:40:08.292 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:08.305+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:08.344 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:08.344 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:08.355 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:40:08.396 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:08.410+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:08.494 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:08.508+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:08.511 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:08.512 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:08.521 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:40:08.594 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:08.609+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:08.629 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:08.629 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:08.639 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:40:08.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.658+0000 7f64e5ffb640 1 -- 172.21.15.60:0/3564859150 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v28) v1 ==== 97+0+0 (secure 0 0 0) 0x7f64e0057d60 con 0x7f64f013b020 2024-03-27T21:40:08.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.658+0000 7f64e5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:40:08.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.658+0000 7f64e5ffb640 10 monclient: _finish_command 2 = system:0 nodeep-scrub is set 2024-03-27T21:40:08.646 INFO:teuthology.orchestra.run.smithi060.stderr:nodeep-scrub is set 2024-03-27T21:40:08.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.661+0000 7f64f745f640 10 librados: watch_flush enter 2024-03-27T21:40:08.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.661+0000 7f64f745f640 10 librados: watch_flush exit 2024-03-27T21:40:08.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.661+0000 7f64f745f640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-27T21:40:08.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.661+0000 7f64f745f640 10 client.4454.objecter successfully canceled tick 2024-03-27T21:40:08.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.661+0000 7f64f745f640 1 -- 172.21.15.60:0/3564859150 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f64d406c5c0 msgr2=0x7f64d406ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:08.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.661+0000 7f64f745f640 1 --2- 172.21.15.60:0/3564859150 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f64d406c5c0 0x7f64d406ea80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f64ec007fd0 tx=0x7f64ec007de0 comp rx=0 tx=0).stop 2024-03-27T21:40:08.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.661+0000 7f64f745f640 10 monclient: shutdown 2024-03-27T21:40:08.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.661+0000 7f64f745f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:08.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.661+0000 7f64f745f640 1 -- 172.21.15.60:0/3564859150 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f013b020 msgr2=0x7f64f01342d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:08.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.661+0000 7f64f745f640 1 --2- 172.21.15.60:0/3564859150 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f013b020 0x7f64f01342d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f64e000e9e0 tx=0x7f64e000eea0 comp rx=0 tx=0).stop 2024-03-27T21:40:08.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.661+0000 7f64f745f640 1 -- 172.21.15.60:0/3564859150 shutdown_connections 2024-03-27T21:40:08.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.661+0000 7f64f745f640 1 --2- 172.21.15.60:0/3564859150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f01416f0 0x7f64f0134810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:08.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.661+0000 7f64f745f640 1 --2- 172.21.15.60:0/3564859150 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f013b020 0x7f64f01342d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:08.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.661+0000 7f64f745f640 1 --2- 172.21.15.60:0/3564859150 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f64d406c5c0 0x7f64d406ea80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:08.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.661+0000 7f64f745f640 1 --2- 172.21.15.60:0/3564859150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64f00b40b0 0x7f64f013a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:08.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.661+0000 7f64f745f640 1 -- 172.21.15.60:0/3564859150 >> 172.21.15.60:0/3564859150 conn(0x7f64f00608e0 msgr2=0x7f64f00b1950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:08.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.661+0000 7f64f745f640 1 -- 172.21.15.60:0/3564859150 shutdown_connections 2024-03-27T21:40:08.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.661+0000 7f64f745f640 1 -- 172.21.15.60:0/3564859150 wait complete. 2024-03-27T21:40:08.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:08.661+0000 7f64f745f640 1 librados: shutdown 2024-03-27T21:40:08.696 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:08.710+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:08.752 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:08.752 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:08.761 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:40:08.797 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:08.811+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:08.873 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:08.873 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:08.882 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:40:08.898 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:08.912+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:08.989 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:08.989 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:08.998 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:40:08.999 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:09.014+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:09.100 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:09.114+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:09.131 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:09.131 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:09.140 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:40:09.224 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:09.238+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:09.292 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:09.292 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:09.301 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:40:09.302 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:09.317+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:09.373 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:40:09.403 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:09.417+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:09.451 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:09.451 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:09.460 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:40:09.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.497+0000 7efc33b42640 10 monclient: get_monmap_and_config 2024-03-27T21:40:09.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.497+0000 7efc33b42640 10 monclient: build_initial_monmap 2024-03-27T21:40:09.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.497+0000 7efc33b42640 10 monclient: monmap: 2024-03-27T21:40:09.484 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:09.484 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:09.484 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:09.498813+0000 2024-03-27T21:40:09.484 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:09.498813+0000 2024-03-27T21:40:09.484 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:09.485 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:09.485 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:09.485 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:09.485 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:09.485 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:09.485 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:09.485 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:09.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.498+0000 7efc33b42640 1 Processor -- start 2024-03-27T21:40:09.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.498+0000 7efc33b42640 1 -- start start 2024-03-27T21:40:09.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.498+0000 7efc33b42640 10 monclient: init 2024-03-27T21:40:09.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc33b42640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:09.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc33b42640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-03-27T21:40:09.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc33b42640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efc2c0b40b0 0x7efc2c0a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:09.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc33b42640 10 monclient(hunting): picked mon.noname-b con 0x7efc2c0b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:09.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc33b42640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efc2c13b020 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:09.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc33b42640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7efc2c13b480 0x7efc2c141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:09.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc33b42640 10 monclient(hunting): picked mon.noname-c con 0x7efc2c13b480 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:09.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc33b42640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc2c0b3830 con 0x7efc2c13b020 2024-03-27T21:40:09.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc33b42640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:09.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc33b42640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc2c13b910 con 0x7efc2c13b480 2024-03-27T21:40:09.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc33b42640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:09.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc33b42640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc2c0b4490 con 0x7efc2c0b40b0 2024-03-27T21:40:09.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc33b42640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:09.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc33b42640 10 monclient(hunting): authenticate will time out at 850.720093s 2024-03-27T21:40:09.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc318de640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efc2c0b40b0 0x7efc2c0a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:09.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc318de640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efc2c0b40b0 0x7efc2c0a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:39480/0 (socket says 172.21.15.60:39480) 2024-03-27T21:40:09.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc318de640 1 -- 172.21.15.60:0/2665154749 learned_addr learned my addr 172.21.15.60:0/2665154749 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:09.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc318de640 10 monclient(hunting): get_auth_request con 0x7efc2c0b40b0 auth_method 0 2024-03-27T21:40:09.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc318de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:09.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc318de640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:09.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc318de640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:09.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc318de640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:09.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc318de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:09.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc318de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:09.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc320df640 1 --2- 172.21.15.60:0/2665154749 >> v2:172.21.15.191:3300/0 conn(0x7efc2c13b480 0x7efc2c141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:09.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc320df640 10 monclient(hunting): get_auth_request con 0x7efc2c13b480 auth_method 0 2024-03-27T21:40:09.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc320df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:09.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc320df640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:09.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc320df640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:09.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc318de640 10 monclient(hunting): handle_auth_done global_id 4457 payload 274 2024-03-27T21:40:09.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc318de640 1 -- 172.21.15.60:0/2665154749 >> v2:172.21.15.191:3300/0 conn(0x7efc2c13b480 msgr2=0x7efc2c141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:09.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc318de640 1 --2- 172.21.15.60:0/2665154749 >> v2:172.21.15.191:3300/0 conn(0x7efc2c13b480 0x7efc2c141e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:09.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc318de640 1 -- 172.21.15.60:0/2665154749 >> v1:172.21.15.191:6789/0 conn(0x7efc2c13b020 legacy=0x7efc2c1416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:09.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc318de640 10 monclient: _finish_hunting 0 2024-03-27T21:40:09.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc318de640 1 monclient: found mon.noname-b 2024-03-27T21:40:09.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc318de640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:09.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc318de640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:09.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.499+0000 7efc318de640 1 -- 172.21.15.60:0/2665154749 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc2c0aa520 con 0x7efc2c0b40b0 2024-03-27T21:40:09.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.500+0000 7efc318de640 1 --2- 172.21.15.60:0/2665154749 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efc2c0b40b0 0x7efc2c0a8b90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7efc1c009870 tx=0x7efc1c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=817205da43f68f56 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:09.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.500+0000 7efc308dc640 1 -- 172.21.15.60:0/2665154749 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc1c031640 con 0x7efc2c0b40b0 2024-03-27T21:40:09.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.500+0000 7efc308dc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:09.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.500+0000 7efc308dc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:09.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.500+0000 7efc308dc640 10 monclient: dump: 2024-03-27T21:40:09.498 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:09.498 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:09.498 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:09.498 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:09.498 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:09.498 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:09.498 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:09.498 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:09.498 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:09.498 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:09.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.500+0000 7efc308dc640 10 monclient: _finish_auth 0 2024-03-27T21:40:09.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.500+0000 7efc308dc640 10 monclient: _check_auth_tickets 2024-03-27T21:40:09.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.500+0000 7efc308dc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:09.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.500+0000 7efc33b42640 5 monclient: authenticate success, global_id 4457 2024-03-27T21:40:09.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.500+0000 7efc308dc640 1 -- 172.21.15.60:0/2665154749 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc1c0317a0 con 0x7efc2c0b40b0 2024-03-27T21:40:09.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.500+0000 7efc33b42640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:09.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.500+0000 7efc308dc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:09.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.500+0000 7efc33b42640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:09.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.500+0000 7efc308dc640 1 -- 172.21.15.60:0/2665154749 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc1c031b30 con 0x7efc2c0b40b0 2024-03-27T21:40:09.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.500+0000 7efc308dc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:09.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.500+0000 7efc33b42640 4 set_mon_vals no callback set 2024-03-27T21:40:09.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.500+0000 7efc308dc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:09.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.500+0000 7efc308dc640 10 monclient: dump: 2024-03-27T21:40:09.499 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:09.499 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:09.499 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:09.500 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:09.500 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:09.500 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:09.500 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:09.500 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:09.500 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:09.500 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:09.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.501+0000 7efc33b42640 10 monclient: shutdown 2024-03-27T21:40:09.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.501+0000 7efc33b42640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:09.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.501+0000 7efc33b42640 1 -- 172.21.15.60:0/2665154749 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efc2c0b40b0 msgr2=0x7efc2c0a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:09.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.501+0000 7efc33b42640 1 --2- 172.21.15.60:0/2665154749 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efc2c0b40b0 0x7efc2c0a8b90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7efc1c009870 tx=0x7efc1c02b8f0 comp rx=0 tx=0).stop 2024-03-27T21:40:09.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.501+0000 7efc33b42640 1 -- 172.21.15.60:0/2665154749 shutdown_connections 2024-03-27T21:40:09.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.501+0000 7efc33b42640 1 --2- 172.21.15.60:0/2665154749 >> v2:172.21.15.191:3300/0 conn(0x7efc2c13b480 0x7efc2c141e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:09.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.501+0000 7efc33b42640 1 --2- 172.21.15.60:0/2665154749 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efc2c0b40b0 0x7efc2c0a8b90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:09.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.501+0000 7efc33b42640 1 -- 172.21.15.60:0/2665154749 >> 172.21.15.60:0/2665154749 conn(0x7efc2c0608e0 msgr2=0x7efc2c060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:09.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.501+0000 7efc33b42640 1 -- 172.21.15.60:0/2665154749 shutdown_connections 2024-03-27T21:40:09.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.501+0000 7efc33b42640 1 -- 172.21.15.60:0/2665154749 wait complete. 2024-03-27T21:40:09.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.501+0000 7efc33b42640 10 monclient: build_initial_monmap 2024-03-27T21:40:09.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.501+0000 7efc33b42640 10 monclient: monmap: 2024-03-27T21:40:09.501 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:09.501 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:09.501 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:09.501 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:09.501 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:09.501 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:09.501 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:09.501 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:09.501 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:09.502 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:09.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 1 librados: starting msgr at 2024-03-27T21:40:09.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 1 librados: starting objecter 2024-03-27T21:40:09.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 1 Processor -- start 2024-03-27T21:40:09.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 1 -- start start 2024-03-27T21:40:09.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 1 librados: setting wanted keys 2024-03-27T21:40:09.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 1 librados: calling monclient init 2024-03-27T21:40:09.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 10 monclient: init 2024-03-27T21:40:09.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:09.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:40:09.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc2c0b40b0 0x7efc2c13d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:09.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 10 monclient(hunting): picked mon.noname-a con 0x7efc2c0b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:09.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efc2c13b020 0x7efc2c13ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 10 monclient(hunting): picked mon.noname-c con 0x7efc2c13b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc2c13b480 0x7efc2c1d9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 10 monclient(hunting): picked mon.noname-b con 0x7efc2c13b480 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2c13ed90 con 0x7efc2c0b40b0 2024-03-27T21:40:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2c13e7e0 con 0x7efc2c13b480 2024-03-27T21:40:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc2c13e410 con 0x7efc2c13b020 2024-03-27T21:40:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc33b42640 10 monclient(hunting): authenticate will time out at 850.723729s 2024-03-27T21:40:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc318de640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc2c0b40b0 0x7efc2c13d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc310dd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efc2c13b020 0x7efc2c13ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc318de640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc2c0b40b0 0x7efc2c13d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52726/0 (socket says 172.21.15.60:52726) 2024-03-27T21:40:09.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc318de640 1 -- 172.21.15.60:0/2757661017 learned_addr learned my addr 172.21.15.60:0/2757661017 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:09.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc318de640 10 monclient(hunting): get_auth_request con 0x7efc2c0b40b0 auth_method 0 2024-03-27T21:40:09.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc318de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:09.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc318de640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:09.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc318de640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:09.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc310dd640 10 monclient(hunting): get_auth_request con 0x7efc2c13b020 auth_method 0 2024-03-27T21:40:09.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc310dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:09.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc310dd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:09.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc310dd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:09.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.502+0000 7efc320df640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc2c13b480 0x7efc2c1d9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:09.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc320df640 10 monclient(hunting): get_auth_request con 0x7efc2c13b480 auth_method 0 2024-03-27T21:40:09.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc320df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:09.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc320df640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:09.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc320df640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc318de640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc318de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc318de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc310dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc310dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc310dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc320df640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc320df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc320df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc318de640 10 monclient(hunting): handle_auth_done global_id 4539 payload 824 2024-03-27T21:40:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc318de640 1 -- 172.21.15.60:0/2757661017 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efc2c13b020 msgr2=0x7efc2c13ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc318de640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efc2c13b020 0x7efc2c13ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc318de640 1 -- 172.21.15.60:0/2757661017 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc2c13b480 msgr2=0x7efc2c1d9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc318de640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc2c13b480 0x7efc2c1d9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc318de640 10 monclient: _finish_hunting 0 2024-03-27T21:40:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc318de640 1 monclient: found mon.noname-a 2024-03-27T21:40:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc318de640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc318de640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc318de640 1 -- 172.21.15.60:0/2757661017 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1c0094d0 con 0x7efc2c0b40b0 2024-03-27T21:40:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc318de640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc2c0b40b0 0x7efc2c13d940 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efc1c02b930 tx=0x7efc1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc227fc640 10 client.?.objecter ms_handle_connect 0x7efc2c0b40b0 2024-03-27T21:40:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc227fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc227fc640 1 -- 172.21.15.60:0/2757661017 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc1c03ac60 con 0x7efc2c0b40b0 2024-03-27T21:40:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:09.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc227fc640 10 monclient: dump: 2024-03-27T21:40:09.507 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:09.507 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:09.507 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:09.507 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:09.507 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:09.507 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:09.507 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:09.507 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:09.507 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:09.508 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:09.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc227fc640 10 monclient: _finish_auth 0 2024-03-27T21:40:09.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc227fc640 10 monclient: _check_auth_tickets 2024-03-27T21:40:09.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:09.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc33b42640 5 monclient: authenticate success, global_id 4539 2024-03-27T21:40:09.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc227fc640 1 -- 172.21.15.60:0/2757661017 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc1c03ade0 con 0x7efc2c0b40b0 2024-03-27T21:40:09.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:09.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc227fc640 1 -- 172.21.15.60:0/2757661017 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc1c044630 con 0x7efc2c0b40b0 2024-03-27T21:40:09.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:09.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:09.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc227fc640 10 monclient: dump: 2024-03-27T21:40:09.508 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:09.508 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:09.509 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:09.509 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:09.509 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:09.509 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:09.509 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:09.509 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:09.509 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:09.509 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:09.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc23fff640 4 set_mon_vals no callback set 2024-03-27T21:40:09.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc33b42640 10 monclient: _renew_subs 2024-03-27T21:40:09.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc33b42640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:09.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc33b42640 1 -- 172.21.15.60:0/2757661017 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc2c13a790 con 0x7efc2c0b40b0 2024-03-27T21:40:09.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc33b42640 10 client.4539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:09.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc33b42640 10 monclient: _renew_subs 2024-03-27T21:40:09.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc33b42640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:09.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc33b42640 1 -- 172.21.15.60:0/2757661017 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc2c1da150 con 0x7efc2c0b40b0 2024-03-27T21:40:09.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.503+0000 7efc33b42640 1 librados: init done 2024-03-27T21:40:09.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.504+0000 7efc33b42640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:09.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.504+0000 7efc33b42640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:09.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.504+0000 7efc33b42640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:09.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.504+0000 7efc33b42640 1 -- 172.21.15.60:0/2757661017 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7efc2c13e9a0 con 0x7efc2c0b40b0 2024-03-27T21:40:09.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.505+0000 7efc227fc640 1 -- 172.21.15.60:0/2757661017 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7efc1c02f050 con 0x7efc2c0b40b0 2024-03-27T21:40:09.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.505+0000 7efc227fc640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7efc1006c590 0x7efc1006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:09.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.505+0000 7efc227fc640 1 -- 172.21.15.60:0/2757661017 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5242+0+0 (secure 0 0 0) 0x7efc1c0ab430 con 0x7efc2c0b40b0 2024-03-27T21:40:09.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.505+0000 7efc227fc640 10 client.4539.objecter ms_dispatch 0x7efc2c000b90 osd_map(28..28 src has 1..28) v4 2024-03-27T21:40:09.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.505+0000 7efc227fc640 3 client.4539.objecter handle_osd_map got epochs [28,28] > 0 2024-03-27T21:40:09.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.505+0000 7efc227fc640 3 client.4539.objecter handle_osd_map decoding full epoch 28 2024-03-27T21:40:09.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.505+0000 7efc310dd640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7efc1006c590 0x7efc1006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:09.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.505+0000 7efc310dd640 10 monclient: get_auth_request con 0x7efc1006c590 auth_method 0 2024-03-27T21:40:09.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.505+0000 7efc227fc640 20 client.4539.objecter dump_active .. 0 homeless 2024-03-27T21:40:09.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.505+0000 7efc227fc640 1 -- 172.21.15.60:0/2757661017 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v28) v1 ==== 54+0+16 (secure 0 0 0) 0x7efc1c0ab880 con 0x7efc2c0b40b0 2024-03-27T21:40:09.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.505+0000 7efc227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:09.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.505+0000 7efc227fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:09.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.506+0000 7efc33b42640 10 client.4539.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:09.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.506+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign -1 1 2024-03-27T21:40:09.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.506+0000 7efc33b42640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7efc2c0b04b0 0x7efc2c13ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:09.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.506+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c0a90d0 osd=0 3 2024-03-27T21:40:09.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.506+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c0a90d0 osd=0 3 2024-03-27T21:40:09.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.506+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c0a90d0 osd=0 3 2024-03-27T21:40:09.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.506+0000 7efc33b42640 15 client.4539.objecter _session_command_op_remove -1 1 2024-03-27T21:40:09.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.506+0000 7efc33b42640 20 client.4539.objecter get_session s=0x7efc2c0a90d0 osd=0 3 2024-03-27T21:40:09.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.506+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign 0 1 2024-03-27T21:40:09.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.506+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c0a90d0 osd=0 4 2024-03-27T21:40:09.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.506+0000 7efc33b42640 10 client.4539.objecter _send_command 1 2024-03-27T21:40:09.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.506+0000 7efc33b42640 1 -- 172.21.15.60:0/2757661017 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc2c142340 con 0x7efc2c0b04b0 2024-03-27T21:40:09.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.506+0000 7efc320df640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7efc2c0b04b0 0x7efc2c13ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:09.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.506+0000 7efc320df640 10 monclient: get_auth_request con 0x7efc2c0b04b0 auth_method 0 2024-03-27T21:40:09.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.506+0000 7efc227fc640 10 client.4539.objecter ms_handle_connect 0x7efc1006c590 2024-03-27T21:40:09.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.506+0000 7efc310dd640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7efc1006c590 0x7efc1006ea50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efc2c141a40 tx=0x7efc28009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:09.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.506+0000 7efc320df640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7efc2c0b04b0 0x7efc2c13ba60 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:09.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.506+0000 7efc227fc640 10 client.4539.objecter ms_handle_connect 0x7efc2c0b04b0 2024-03-27T21:40:09.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.507+0000 7efc227fc640 1 -- 172.21.15.60:0/2757661017 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7efc24009cb0 con 0x7efc2c0b04b0 2024-03-27T21:40:09.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.507+0000 7efc227fc640 10 client.4539.objecter ms_dispatch 0x7efc2c000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:09.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.507+0000 7efc227fc640 10 client.4539.objecter _finish_command 1 = system:0 2024-03-27T21:40:09.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.507+0000 7efc227fc640 20 client.4539.objecter put_session s=0x7efc2c0a90d0 osd=0 4 2024-03-27T21:40:09.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.507+0000 7efc227fc640 15 client.4539.objecter _session_command_op_remove 0 1 2024-03-27T21:40:09.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.522+0000 7efc33b42640 10 client.4539.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:09.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.522+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign -1 2 2024-03-27T21:40:09.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.522+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c0a90d0 osd=0 3 2024-03-27T21:40:09.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.522+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c0a90d0 osd=0 3 2024-03-27T21:40:09.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.522+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c0a90d0 osd=0 3 2024-03-27T21:40:09.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.522+0000 7efc33b42640 15 client.4539.objecter _session_command_op_remove -1 2 2024-03-27T21:40:09.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.522+0000 7efc33b42640 20 client.4539.objecter get_session s=0x7efc2c0a90d0 osd=0 3 2024-03-27T21:40:09.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.522+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign 0 2 2024-03-27T21:40:09.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.522+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c0a90d0 osd=0 4 2024-03-27T21:40:09.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.522+0000 7efc33b42640 10 client.4539.objecter _send_command 2 2024-03-27T21:40:09.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.522+0000 7efc33b42640 1 -- 172.21.15.60:0/2757661017 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc2c14bd00 con 0x7efc2c0b04b0 2024-03-27T21:40:09.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.524+0000 7efc227fc640 1 -- 172.21.15.60:0/2757661017 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7efc2c14bd00 con 0x7efc2c0b04b0 2024-03-27T21:40:09.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.524+0000 7efc227fc640 10 client.4539.objecter ms_dispatch 0x7efc2c000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:09.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.524+0000 7efc227fc640 10 client.4539.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:09.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.524+0000 7efc227fc640 20 client.4539.objecter put_session s=0x7efc2c0a90d0 osd=0 4 2024-03-27T21:40:09.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.524+0000 7efc227fc640 15 client.4539.objecter _session_command_op_remove 0 2 2024-03-27T21:40:09.515 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:09.516 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:09.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.524+0000 7efc33b42640 10 client.4539.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:09.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.524+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign -1 3 2024-03-27T21:40:09.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.524+0000 7efc33b42640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7efc2c0b0890 0x7efc2c13c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:09.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.524+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c13e630 osd=1 3 2024-03-27T21:40:09.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.524+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c13e630 osd=1 3 2024-03-27T21:40:09.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.524+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c13e630 osd=1 3 2024-03-27T21:40:09.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.524+0000 7efc33b42640 15 client.4539.objecter _session_command_op_remove -1 3 2024-03-27T21:40:09.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.524+0000 7efc33b42640 20 client.4539.objecter get_session s=0x7efc2c13e630 osd=1 3 2024-03-27T21:40:09.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.524+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign 1 3 2024-03-27T21:40:09.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.524+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c13e630 osd=1 4 2024-03-27T21:40:09.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.524+0000 7efc33b42640 10 client.4539.objecter _send_command 3 2024-03-27T21:40:09.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.524+0000 7efc33b42640 1 -- 172.21.15.60:0/2757661017 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc2c069470 con 0x7efc2c0b0890 2024-03-27T21:40:09.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.524+0000 7efc318de640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7efc2c0b0890 0x7efc2c13c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:09.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.524+0000 7efc318de640 10 monclient: get_auth_request con 0x7efc2c0b0890 auth_method 0 2024-03-27T21:40:09.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.524+0000 7efc227fc640 10 client.4539.objecter ms_handle_connect 0x7efc2c0b0890 2024-03-27T21:40:09.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.524+0000 7efc318de640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7efc2c0b0890 0x7efc2c13c4b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:09.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.525+0000 7efc227fc640 1 -- 172.21.15.60:0/2757661017 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7efc2c069470 con 0x7efc2c0b0890 2024-03-27T21:40:09.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.525+0000 7efc227fc640 10 client.4539.objecter ms_dispatch 0x7efc2c000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:09.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.525+0000 7efc227fc640 10 client.4539.objecter _finish_command 3 = system:0 2024-03-27T21:40:09.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.525+0000 7efc227fc640 20 client.4539.objecter put_session s=0x7efc2c13e630 osd=1 4 2024-03-27T21:40:09.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.525+0000 7efc227fc640 15 client.4539.objecter _session_command_op_remove 1 3 2024-03-27T21:40:09.518 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:09.527+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:09.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.540+0000 7efc33b42640 10 client.4539.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:09.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.540+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign -1 4 2024-03-27T21:40:09.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.540+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c13e630 osd=1 3 2024-03-27T21:40:09.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.540+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c13e630 osd=1 3 2024-03-27T21:40:09.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.540+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c13e630 osd=1 3 2024-03-27T21:40:09.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.540+0000 7efc33b42640 15 client.4539.objecter _session_command_op_remove -1 4 2024-03-27T21:40:09.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.540+0000 7efc33b42640 20 client.4539.objecter get_session s=0x7efc2c13e630 osd=1 3 2024-03-27T21:40:09.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.540+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign 1 4 2024-03-27T21:40:09.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.540+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c13e630 osd=1 4 2024-03-27T21:40:09.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.540+0000 7efc33b42640 10 client.4539.objecter _send_command 4 2024-03-27T21:40:09.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.540+0000 7efc33b42640 1 -- 172.21.15.60:0/2757661017 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc2c065a90 con 0x7efc2c0b0890 2024-03-27T21:40:09.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.541+0000 7efc227fc640 1 -- 172.21.15.60:0/2757661017 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7efc2c065a90 con 0x7efc2c0b0890 2024-03-27T21:40:09.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.541+0000 7efc227fc640 10 client.4539.objecter ms_dispatch 0x7efc2c000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:09.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.541+0000 7efc227fc640 10 client.4539.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:09.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.541+0000 7efc227fc640 20 client.4539.objecter put_session s=0x7efc2c13e630 osd=1 4 2024-03-27T21:40:09.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.541+0000 7efc227fc640 15 client.4539.objecter _session_command_op_remove 1 4 2024-03-27T21:40:09.529 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:09.530 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:09.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc33b42640 10 client.4539.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:09.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign -1 5 2024-03-27T21:40:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc33b42640 20 client.4539.objecter _get_session osd=-1 returning homeless 2024-03-27T21:40:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc33b42640 10 client.4539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc33b42640 10 monclient: _renew_subs 2024-03-27T21:40:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc33b42640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc33b42640 1 -- 172.21.15.60:0/2757661017 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=29}) v3 -- 0x7efc2c13fb10 con 0x7efc2c0b40b0 2024-03-27T21:40:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc33b42640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc33b42640 1 -- 172.21.15.60:0/2757661017 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efc2c13fe10 con 0x7efc2c0b40b0 2024-03-27T21:40:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc227fc640 1 -- 172.21.15.60:0/2757661017 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc1c0ab840 con 0x7efc2c0b40b0 2024-03-27T21:40:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc227fc640 10 monclient: handle_get_version_reply finishing 1 version 28 2024-03-27T21:40:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc23fff640 10 client.4539.objecter _check_command_map_dne tid 5 current 28 map_dne_bound 28 2024-03-27T21:40:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc23fff640 10 client.4539.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:40:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc23fff640 15 client.4539.objecter _session_command_op_remove -1 5 2024-03-27T21:40:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:40:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc33b42640 10 client.4539.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign -1 6 2024-03-27T21:40:09.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc33b42640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7efc2c0667d0 0x7efc2c02b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:09.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c066660 osd=3 3 2024-03-27T21:40:09.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c066660 osd=3 3 2024-03-27T21:40:09.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c066660 osd=3 3 2024-03-27T21:40:09.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc33b42640 15 client.4539.objecter _session_command_op_remove -1 6 2024-03-27T21:40:09.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc33b42640 20 client.4539.objecter get_session s=0x7efc2c066660 osd=3 3 2024-03-27T21:40:09.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign 3 6 2024-03-27T21:40:09.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c066660 osd=3 4 2024-03-27T21:40:09.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc33b42640 10 client.4539.objecter _send_command 6 2024-03-27T21:40:09.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc33b42640 1 -- 172.21.15.60:0/2757661017 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc2c02bbc0 con 0x7efc2c0667d0 2024-03-27T21:40:09.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc310dd640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7efc2c0667d0 0x7efc2c02b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:09.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc310dd640 10 monclient: get_auth_request con 0x7efc2c0667d0 auth_method 0 2024-03-27T21:40:09.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc227fc640 10 client.4539.objecter ms_handle_connect 0x7efc2c0667d0 2024-03-27T21:40:09.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.542+0000 7efc310dd640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7efc2c0667d0 0x7efc2c02b680 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:09.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.543+0000 7efc227fc640 1 -- 172.21.15.60:0/2757661017 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7efc2c02bbc0 con 0x7efc2c0667d0 2024-03-27T21:40:09.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.543+0000 7efc227fc640 10 client.4539.objecter ms_dispatch 0x7efc2c000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:40:09.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.543+0000 7efc227fc640 10 client.4539.objecter _finish_command 6 = system:0 2024-03-27T21:40:09.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.543+0000 7efc227fc640 20 client.4539.objecter put_session s=0x7efc2c066660 osd=3 4 2024-03-27T21:40:09.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.543+0000 7efc227fc640 15 client.4539.objecter _session_command_op_remove 3 6 2024-03-27T21:40:09.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.557+0000 7efc33b42640 10 client.4539.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:09.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.557+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign -1 7 2024-03-27T21:40:09.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.557+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c066660 osd=3 3 2024-03-27T21:40:09.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.557+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c066660 osd=3 3 2024-03-27T21:40:09.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.557+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c066660 osd=3 3 2024-03-27T21:40:09.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.557+0000 7efc33b42640 15 client.4539.objecter _session_command_op_remove -1 7 2024-03-27T21:40:09.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.557+0000 7efc33b42640 20 client.4539.objecter get_session s=0x7efc2c066660 osd=3 3 2024-03-27T21:40:09.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.557+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign 3 7 2024-03-27T21:40:09.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.557+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c066660 osd=3 4 2024-03-27T21:40:09.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.557+0000 7efc33b42640 10 client.4539.objecter _send_command 7 2024-03-27T21:40:09.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.557+0000 7efc33b42640 1 -- 172.21.15.60:0/2757661017 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc2c02c520 con 0x7efc2c0667d0 2024-03-27T21:40:09.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.559+0000 7efc227fc640 1 -- 172.21.15.60:0/2757661017 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7efc2c02c520 con 0x7efc2c0667d0 2024-03-27T21:40:09.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.559+0000 7efc227fc640 10 client.4539.objecter ms_dispatch 0x7efc2c000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:09.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.559+0000 7efc227fc640 10 client.4539.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:09.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.559+0000 7efc227fc640 20 client.4539.objecter put_session s=0x7efc2c066660 osd=3 4 2024-03-27T21:40:09.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.559+0000 7efc227fc640 15 client.4539.objecter _session_command_op_remove 3 7 2024-03-27T21:40:09.546 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:09.547 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:09.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.559+0000 7efc33b42640 10 client.4539.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:09.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.559+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign -1 8 2024-03-27T21:40:09.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.559+0000 7efc33b42640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7efc2c1e72a0 0x7efc2c1e9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:09.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.559+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c02c690 osd=4 3 2024-03-27T21:40:09.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.559+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c02c690 osd=4 3 2024-03-27T21:40:09.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.559+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c02c690 osd=4 3 2024-03-27T21:40:09.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.559+0000 7efc33b42640 15 client.4539.objecter _session_command_op_remove -1 8 2024-03-27T21:40:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.559+0000 7efc33b42640 20 client.4539.objecter get_session s=0x7efc2c02c690 osd=4 3 2024-03-27T21:40:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.559+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign 4 8 2024-03-27T21:40:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.559+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c02c690 osd=4 4 2024-03-27T21:40:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.559+0000 7efc33b42640 10 client.4539.objecter _send_command 8 2024-03-27T21:40:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.559+0000 7efc33b42640 1 -- 172.21.15.60:0/2757661017 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc2c066320 con 0x7efc2c1e72a0 2024-03-27T21:40:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.559+0000 7efc320df640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7efc2c1e72a0 0x7efc2c1e9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.559+0000 7efc320df640 10 monclient: get_auth_request con 0x7efc2c1e72a0 auth_method 0 2024-03-27T21:40:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.560+0000 7efc227fc640 10 client.4539.objecter ms_handle_connect 0x7efc2c1e72a0 2024-03-27T21:40:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.560+0000 7efc320df640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7efc2c1e72a0 0x7efc2c1e9690 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.561+0000 7efc227fc640 1 -- 172.21.15.60:0/2757661017 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7efc2c066320 con 0x7efc2c1e72a0 2024-03-27T21:40:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.561+0000 7efc227fc640 10 client.4539.objecter ms_dispatch 0x7efc2c000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:40:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.561+0000 7efc227fc640 10 client.4539.objecter _finish_command 8 = system:0 2024-03-27T21:40:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.561+0000 7efc227fc640 20 client.4539.objecter put_session s=0x7efc2c02c690 osd=4 4 2024-03-27T21:40:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.561+0000 7efc227fc640 15 client.4539.objecter _session_command_op_remove 4 8 2024-03-27T21:40:09.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.577+0000 7efc33b42640 10 client.4539.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:09.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.577+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign -1 9 2024-03-27T21:40:09.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.577+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c02c690 osd=4 3 2024-03-27T21:40:09.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.577+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c02c690 osd=4 3 2024-03-27T21:40:09.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.577+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c02c690 osd=4 3 2024-03-27T21:40:09.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.577+0000 7efc33b42640 15 client.4539.objecter _session_command_op_remove -1 9 2024-03-27T21:40:09.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.577+0000 7efc33b42640 20 client.4539.objecter get_session s=0x7efc2c02c690 osd=4 3 2024-03-27T21:40:09.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.577+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign 4 9 2024-03-27T21:40:09.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.577+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c02c690 osd=4 4 2024-03-27T21:40:09.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.577+0000 7efc33b42640 10 client.4539.objecter _send_command 9 2024-03-27T21:40:09.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.577+0000 7efc33b42640 1 -- 172.21.15.60:0/2757661017 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc2c0664c0 con 0x7efc2c1e72a0 2024-03-27T21:40:09.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.579+0000 7efc227fc640 1 -- 172.21.15.60:0/2757661017 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7efc2c0664c0 con 0x7efc2c1e72a0 2024-03-27T21:40:09.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.579+0000 7efc227fc640 10 client.4539.objecter ms_dispatch 0x7efc2c000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:09.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.579+0000 7efc227fc640 10 client.4539.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:09.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.579+0000 7efc227fc640 20 client.4539.objecter put_session s=0x7efc2c02c690 osd=4 4 2024-03-27T21:40:09.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.579+0000 7efc227fc640 15 client.4539.objecter _session_command_op_remove 4 9 2024-03-27T21:40:09.566 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:09.567 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:09.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.579+0000 7efc33b42640 10 client.4539.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:09.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.579+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign -1 10 2024-03-27T21:40:09.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.579+0000 7efc33b42640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7efc2c0b2110 0x7efc2c0b2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:09.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.579+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c0664c0 osd=5 3 2024-03-27T21:40:09.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.579+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c0664c0 osd=5 3 2024-03-27T21:40:09.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.579+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c0664c0 osd=5 3 2024-03-27T21:40:09.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.579+0000 7efc33b42640 15 client.4539.objecter _session_command_op_remove -1 10 2024-03-27T21:40:09.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.579+0000 7efc33b42640 20 client.4539.objecter get_session s=0x7efc2c0664c0 osd=5 3 2024-03-27T21:40:09.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.579+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign 5 10 2024-03-27T21:40:09.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.579+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c0664c0 osd=5 4 2024-03-27T21:40:09.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.579+0000 7efc33b42640 10 client.4539.objecter _send_command 10 2024-03-27T21:40:09.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.579+0000 7efc33b42640 1 -- 172.21.15.60:0/2757661017 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc2c1ef5f0 con 0x7efc2c0b2110 2024-03-27T21:40:09.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.579+0000 7efc318de640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7efc2c0b2110 0x7efc2c0b2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:09.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.579+0000 7efc318de640 10 monclient: get_auth_request con 0x7efc2c0b2110 auth_method 0 2024-03-27T21:40:09.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.580+0000 7efc227fc640 10 client.4539.objecter ms_handle_connect 0x7efc2c0b2110 2024-03-27T21:40:09.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.580+0000 7efc318de640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7efc2c0b2110 0x7efc2c0b2570 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:09.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.580+0000 7efc227fc640 1 -- 172.21.15.60:0/2757661017 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7efc2c1ef5f0 con 0x7efc2c0b2110 2024-03-27T21:40:09.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.580+0000 7efc227fc640 10 client.4539.objecter ms_dispatch 0x7efc2c000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:40:09.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.580+0000 7efc227fc640 10 client.4539.objecter _finish_command 10 = system:0 2024-03-27T21:40:09.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.580+0000 7efc227fc640 20 client.4539.objecter put_session s=0x7efc2c0664c0 osd=5 4 2024-03-27T21:40:09.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.580+0000 7efc227fc640 15 client.4539.objecter _session_command_op_remove 5 10 2024-03-27T21:40:09.576 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:09.576 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:09.576 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:09.576 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:09.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.595+0000 7efc33b42640 10 client.4539.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:09.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.595+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign -1 11 2024-03-27T21:40:09.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.595+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c0664c0 osd=5 3 2024-03-27T21:40:09.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.595+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c0664c0 osd=5 3 2024-03-27T21:40:09.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.595+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c0664c0 osd=5 3 2024-03-27T21:40:09.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.595+0000 7efc33b42640 15 client.4539.objecter _session_command_op_remove -1 11 2024-03-27T21:40:09.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.595+0000 7efc33b42640 20 client.4539.objecter get_session s=0x7efc2c0664c0 osd=5 3 2024-03-27T21:40:09.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.595+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign 5 11 2024-03-27T21:40:09.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.595+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c0664c0 osd=5 4 2024-03-27T21:40:09.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.595+0000 7efc33b42640 10 client.4539.objecter _send_command 11 2024-03-27T21:40:09.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.595+0000 7efc33b42640 1 -- 172.21.15.60:0/2757661017 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc2c1ecd60 con 0x7efc2c0b2110 2024-03-27T21:40:09.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.596+0000 7efc227fc640 1 -- 172.21.15.60:0/2757661017 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7efc2c1ecd60 con 0x7efc2c0b2110 2024-03-27T21:40:09.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.596+0000 7efc227fc640 10 client.4539.objecter ms_dispatch 0x7efc2c000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:09.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.596+0000 7efc227fc640 10 client.4539.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:09.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.596+0000 7efc227fc640 20 client.4539.objecter put_session s=0x7efc2c0664c0 osd=5 4 2024-03-27T21:40:09.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.596+0000 7efc227fc640 15 client.4539.objecter _session_command_op_remove 5 11 2024-03-27T21:40:09.584 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:09.585 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:09.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.597+0000 7efc33b42640 10 client.4539.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:09.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.597+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign -1 12 2024-03-27T21:40:09.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.597+0000 7efc33b42640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7efc2c1ed3e0 0x7efc2c1ef7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:09.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.597+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c1ecd60 osd=6 3 2024-03-27T21:40:09.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.597+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c1ecd60 osd=6 3 2024-03-27T21:40:09.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.597+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c1ecd60 osd=6 3 2024-03-27T21:40:09.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.597+0000 7efc33b42640 15 client.4539.objecter _session_command_op_remove -1 12 2024-03-27T21:40:09.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.597+0000 7efc33b42640 20 client.4539.objecter get_session s=0x7efc2c1ecd60 osd=6 3 2024-03-27T21:40:09.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.597+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign 6 12 2024-03-27T21:40:09.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.597+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c1ecd60 osd=6 4 2024-03-27T21:40:09.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.597+0000 7efc33b42640 10 client.4539.objecter _send_command 12 2024-03-27T21:40:09.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.597+0000 7efc33b42640 1 -- 172.21.15.60:0/2757661017 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc2c1f25a0 con 0x7efc2c1ed3e0 2024-03-27T21:40:09.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.597+0000 7efc310dd640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7efc2c1ed3e0 0x7efc2c1ef7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:09.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.597+0000 7efc310dd640 10 monclient: get_auth_request con 0x7efc2c1ed3e0 auth_method 0 2024-03-27T21:40:09.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.597+0000 7efc227fc640 10 client.4539.objecter ms_handle_connect 0x7efc2c1ed3e0 2024-03-27T21:40:09.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.597+0000 7efc310dd640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7efc2c1ed3e0 0x7efc2c1ef7d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:09.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.598+0000 7efc227fc640 1 -- 172.21.15.60:0/2757661017 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7efc2c1f25a0 con 0x7efc2c1ed3e0 2024-03-27T21:40:09.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.598+0000 7efc227fc640 10 client.4539.objecter ms_dispatch 0x7efc2c000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:40:09.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.598+0000 7efc227fc640 10 client.4539.objecter _finish_command 12 = system:0 2024-03-27T21:40:09.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.598+0000 7efc227fc640 20 client.4539.objecter put_session s=0x7efc2c1ecd60 osd=6 4 2024-03-27T21:40:09.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.598+0000 7efc227fc640 15 client.4539.objecter _session_command_op_remove 6 12 2024-03-27T21:40:09.588 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:40:09.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.616+0000 7efc33b42640 10 client.4539.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:09.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.616+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign -1 13 2024-03-27T21:40:09.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.616+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c1ecd60 osd=6 3 2024-03-27T21:40:09.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.616+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c1ecd60 osd=6 3 2024-03-27T21:40:09.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.616+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c1ecd60 osd=6 3 2024-03-27T21:40:09.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.616+0000 7efc33b42640 15 client.4539.objecter _session_command_op_remove -1 13 2024-03-27T21:40:09.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.616+0000 7efc33b42640 20 client.4539.objecter get_session s=0x7efc2c1ecd60 osd=6 3 2024-03-27T21:40:09.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.616+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign 6 13 2024-03-27T21:40:09.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.616+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c1ecd60 osd=6 4 2024-03-27T21:40:09.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.616+0000 7efc33b42640 10 client.4539.objecter _send_command 13 2024-03-27T21:40:09.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.616+0000 7efc33b42640 1 -- 172.21.15.60:0/2757661017 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc2c1efd10 con 0x7efc2c1ed3e0 2024-03-27T21:40:09.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.618+0000 7efc227fc640 1 -- 172.21.15.60:0/2757661017 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7efc2c1efd10 con 0x7efc2c1ed3e0 2024-03-27T21:40:09.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.618+0000 7efc227fc640 10 client.4539.objecter ms_dispatch 0x7efc2c000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:09.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.618+0000 7efc227fc640 10 client.4539.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:09.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.618+0000 7efc227fc640 20 client.4539.objecter put_session s=0x7efc2c1ecd60 osd=6 4 2024-03-27T21:40:09.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.618+0000 7efc227fc640 15 client.4539.objecter _session_command_op_remove 6 13 2024-03-27T21:40:09.606 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:09.607 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:09.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.618+0000 7efc33b42640 10 client.4539.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:09.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.618+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign -1 14 2024-03-27T21:40:09.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.618+0000 7efc33b42640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7efc2c1f03e0 0x7efc2c1f0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:09.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.618+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c1efd10 osd=7 3 2024-03-27T21:40:09.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.618+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c1efd10 osd=7 3 2024-03-27T21:40:09.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.618+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c1efd10 osd=7 3 2024-03-27T21:40:09.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.618+0000 7efc33b42640 15 client.4539.objecter _session_command_op_remove -1 14 2024-03-27T21:40:09.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.618+0000 7efc33b42640 20 client.4539.objecter get_session s=0x7efc2c1efd10 osd=7 3 2024-03-27T21:40:09.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.618+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign 7 14 2024-03-27T21:40:09.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.618+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c1efd10 osd=7 4 2024-03-27T21:40:09.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.618+0000 7efc33b42640 10 client.4539.objecter _send_command 14 2024-03-27T21:40:09.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.618+0000 7efc33b42640 1 -- 172.21.15.60:0/2757661017 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc2c1f58d0 con 0x7efc2c1f03e0 2024-03-27T21:40:09.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.619+0000 7efc320df640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7efc2c1f03e0 0x7efc2c1f0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:09.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.619+0000 7efc320df640 10 monclient: get_auth_request con 0x7efc2c1f03e0 auth_method 0 2024-03-27T21:40:09.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.619+0000 7efc320df640 1 --2- 172.21.15.60:0/2757661017 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7efc2c1f03e0 0x7efc2c1f0840 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:09.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.619+0000 7efc227fc640 10 client.4539.objecter ms_handle_connect 0x7efc2c1f03e0 2024-03-27T21:40:09.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.620+0000 7efc227fc640 1 -- 172.21.15.60:0/2757661017 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7efc2c1f58d0 con 0x7efc2c1f03e0 2024-03-27T21:40:09.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.620+0000 7efc227fc640 10 client.4539.objecter ms_dispatch 0x7efc2c000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:40:09.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.620+0000 7efc227fc640 10 client.4539.objecter _finish_command 14 = system:0 2024-03-27T21:40:09.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.620+0000 7efc227fc640 20 client.4539.objecter put_session s=0x7efc2c1efd10 osd=7 4 2024-03-27T21:40:09.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.620+0000 7efc227fc640 15 client.4539.objecter _session_command_op_remove 7 14 2024-03-27T21:40:09.614 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:09.628+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:09.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.634+0000 7efc33b42640 10 client.4539.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:09.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.634+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign -1 15 2024-03-27T21:40:09.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.634+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c1efd10 osd=7 3 2024-03-27T21:40:09.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.634+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c1efd10 osd=7 3 2024-03-27T21:40:09.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.634+0000 7efc33b42640 20 client.4539.objecter _get_session s=0x7efc2c1efd10 osd=7 3 2024-03-27T21:40:09.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.634+0000 7efc33b42640 15 client.4539.objecter _session_command_op_remove -1 15 2024-03-27T21:40:09.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.634+0000 7efc33b42640 20 client.4539.objecter get_session s=0x7efc2c1efd10 osd=7 3 2024-03-27T21:40:09.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.634+0000 7efc33b42640 15 client.4539.objecter _session_command_op_assign 7 15 2024-03-27T21:40:09.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.634+0000 7efc33b42640 20 client.4539.objecter put_session s=0x7efc2c1efd10 osd=7 4 2024-03-27T21:40:09.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.634+0000 7efc33b42640 10 client.4539.objecter _send_command 15 2024-03-27T21:40:09.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.634+0000 7efc33b42640 1 -- 172.21.15.60:0/2757661017 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc2c1f0e90 con 0x7efc2c1f03e0 2024-03-27T21:40:09.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.635+0000 7efc227fc640 1 -- 172.21.15.60:0/2757661017 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7efc2c1f0e90 con 0x7efc2c1f03e0 2024-03-27T21:40:09.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.635+0000 7efc227fc640 10 client.4539.objecter ms_dispatch 0x7efc2c000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:09.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.635+0000 7efc227fc640 10 client.4539.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:09.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.635+0000 7efc227fc640 20 client.4539.objecter put_session s=0x7efc2c1efd10 osd=7 4 2024-03-27T21:40:09.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:09.635+0000 7efc227fc640 15 client.4539.objecter _session_command_op_remove 7 15 2024-03-27T21:40:09.623 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:09.623 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:09.625 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:40:09.625 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:40:09.694 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:09.695 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:09.695 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:09.695 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:09.695 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:09.703 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:40:09.746 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:09.761+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:09.809 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:09.809 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:09.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:09.809 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:09.809 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:09.816 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:09.830+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:09.818 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:40:09.917 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:09.932+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:09.963 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:09.964 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:09.964 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:09.964 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:09.972 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:40:10.056 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:10.070+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:10.117 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:10.117 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:10.117 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:10.117 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:10.117 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:10.119 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:10.133+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:10.125 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:40:10.220 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:10.234+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:10.270 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:10.271 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:10.271 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:10.271 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:10.271 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:10.279 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:40:10.321 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:10.335+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:10.384 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:10.384 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:10.384 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:10.384 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:10.393 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:40:10.423 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:10.437+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:10.497 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:10.498 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:10.498 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:10.498 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:10.498 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:10.506 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:40:10.524 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:10.538+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:10.612 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:10.612 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:10.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:10.613 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:10.613 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:10.621 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:40:10.625 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:10.639+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:10.652 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-27T21:40:10.652 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T21:40:10.652 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-03-27T21:40:10.653 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-27T21:40:10.653 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-27T21:40:10.655 INFO:tasks.ceph.osd.2:Started 2024-03-27T21:40:10.655 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-27T21:40:10.660 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-27T21:40:10.725 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:10.726 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:10.726 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:10.726 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:10.726 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:10.740+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:10.734 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:40:10.800 INFO:teuthology.orchestra.run.smithi060.stderr:no valid command found; 10 closest matches: 2024-03-27T21:40:10.800 INFO:teuthology.orchestra.run.smithi060.stderr:0 2024-03-27T21:40:10.800 INFO:teuthology.orchestra.run.smithi060.stderr:1 2024-03-27T21:40:10.800 INFO:teuthology.orchestra.run.smithi060.stderr:2 2024-03-27T21:40:10.800 INFO:teuthology.orchestra.run.smithi060.stderr:abort 2024-03-27T21:40:10.800 INFO:teuthology.orchestra.run.smithi060.stderr:assert 2024-03-27T21:40:10.800 INFO:teuthology.orchestra.run.smithi060.stderr:config diff 2024-03-27T21:40:10.800 INFO:teuthology.orchestra.run.smithi060.stderr:config diff get 2024-03-27T21:40:10.800 INFO:teuthology.orchestra.run.smithi060.stderr:config get 2024-03-27T21:40:10.801 INFO:teuthology.orchestra.run.smithi060.stderr:config help [] 2024-03-27T21:40:10.801 INFO:teuthology.orchestra.run.smithi060.stderr:config set ... 2024-03-27T21:40:10.801 INFO:teuthology.orchestra.run.smithi060.stderr:admin_socket: invalid command 2024-03-27T21:40:10.803 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T21:40:10.803 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-03-27T21:40:10.827 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:10.842+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:10.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.848+0000 7f5c371cb640 10 monclient: get_monmap_and_config 2024-03-27T21:40:10.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.848+0000 7f5c371cb640 10 monclient: build_initial_monmap 2024-03-27T21:40:10.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.848+0000 7f5c371cb640 10 monclient: monmap: 2024-03-27T21:40:10.835 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:10.835 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:10.835 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:10.849466+0000 2024-03-27T21:40:10.835 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:10.849466+0000 2024-03-27T21:40:10.835 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:10.835 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:10.835 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:10.835 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:10.835 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:10.835 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:10.835 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:10.835 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:10.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.849+0000 7f5c371cb640 1 Processor -- start 2024-03-27T21:40:10.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.849+0000 7f5c371cb640 1 -- start start 2024-03-27T21:40:10.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.849+0000 7f5c371cb640 10 monclient: init 2024-03-27T21:40:10.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c371cb640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:10.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c371cb640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-03-27T21:40:10.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c371cb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5c300b40b0 0x7f5c300a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:10.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c371cb640 10 monclient(hunting): picked mon.noname-a con 0x7f5c300b40b0 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:10.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c371cb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5c3013b020 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:10.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c371cb640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c3013b480 0x7f5c30141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:10.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c371cb640 10 monclient(hunting): picked mon.noname-b con 0x7f5c3013b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:10.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c371cb640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c300b3830 con 0x7f5c3013b020 2024-03-27T21:40:10.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c371cb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:10.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c371cb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c3013b910 con 0x7f5c300b40b0 2024-03-27T21:40:10.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c371cb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:10.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c371cb640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c300b4490 con 0x7f5c3013b480 2024-03-27T21:40:10.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c371cb640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:10.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c371cb640 10 monclient(hunting): authenticate will time out at 852.071442s 2024-03-27T21:40:10.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c35768640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c3013b480 0x7f5c30141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:10.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c35768640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c3013b480 0x7f5c30141e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:39510/0 (socket says 172.21.15.60:39510) 2024-03-27T21:40:10.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c35768640 1 -- 172.21.15.60:0/2541999955 learned_addr learned my addr 172.21.15.60:0/2541999955 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:10.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c35768640 10 monclient(hunting): get_auth_request con 0x7f5c3013b480 auth_method 0 2024-03-27T21:40:10.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c35768640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:10.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c35768640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:10.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c35768640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:10.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c34f67640 1 --2- 172.21.15.60:0/2541999955 >> v2:172.21.15.60:3300/0 conn(0x7f5c300b40b0 0x7f5c300a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:10.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c34f67640 10 monclient(hunting): get_auth_request con 0x7f5c300b40b0 auth_method 0 2024-03-27T21:40:10.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c34f67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:10.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c34f67640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:10.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c34f67640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:10.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c35768640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:10.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c35768640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:10.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c35768640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:10.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c34f67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:10.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c34f67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:10.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c34f67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:10.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c35768640 10 monclient(hunting): handle_auth_done global_id 4463 payload 274 2024-03-27T21:40:10.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c35768640 1 -- 172.21.15.60:0/2541999955 >> v2:172.21.15.60:3300/0 conn(0x7f5c300b40b0 msgr2=0x7f5c300a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:10.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c35768640 1 --2- 172.21.15.60:0/2541999955 >> v2:172.21.15.60:3300/0 conn(0x7f5c300b40b0 0x7f5c300a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:10.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c35768640 1 -- 172.21.15.60:0/2541999955 >> v1:172.21.15.191:6789/0 conn(0x7f5c3013b020 legacy=0x7f5c301416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:10.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c35768640 10 monclient: _finish_hunting 0 2024-03-27T21:40:10.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c35768640 1 monclient: found mon.noname-b 2024-03-27T21:40:10.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c35768640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:10.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c35768640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:10.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c2f7fe640 1 -- 172.21.15.60:0/2541999955 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 460907790 0 0) 0x7f5c300b3830 con 0x7f5c3013b020 2024-03-27T21:40:10.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c35768640 1 -- 172.21.15.60:0/2541999955 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c300aa520 con 0x7f5c3013b480 2024-03-27T21:40:10.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.850+0000 7f5c2f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:40:10.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.851+0000 7f5c35768640 1 --2- 172.21.15.60:0/2541999955 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c3013b480 0x7f5c30141e00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5c28009e50 tx=0x7f5c2802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a30bf759c10b00ce server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:10.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c2f7fe640 1 -- 172.21.15.60:0/2541999955 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2802ca70 con 0x7f5c3013b480 2024-03-27T21:40:10.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:10.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c2f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:10.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c2f7fe640 10 monclient: dump: 2024-03-27T21:40:10.843 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:10.843 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:10.843 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:10.843 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:10.843 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:10.843 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:10.843 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:10.843 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:10.844 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:10.844 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:10.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c2f7fe640 10 monclient: _finish_auth 0 2024-03-27T21:40:10.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c2f7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:40:10.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:10.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c2f7fe640 1 -- 172.21.15.60:0/2541999955 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2802cbd0 con 0x7f5c3013b480 2024-03-27T21:40:10.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c371cb640 5 monclient: authenticate success, global_id 4463 2024-03-27T21:40:10.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c371cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:10.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c2f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:10.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c371cb640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:10.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c371cb640 4 set_mon_vals no callback set 2024-03-27T21:40:10.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c2f7fe640 1 -- 172.21.15.60:0/2541999955 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c280343f0 con 0x7f5c3013b480 2024-03-27T21:40:10.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:10.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c2f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:10.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c2f7fe640 10 monclient: dump: 2024-03-27T21:40:10.845 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:10.845 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:10.845 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:10.845 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:10.845 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:10.845 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:10.845 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:10.845 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:10.845 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:10.845 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:10.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c371cb640 10 monclient: shutdown 2024-03-27T21:40:10.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c371cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:10.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c371cb640 1 -- 172.21.15.60:0/2541999955 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c3013b480 msgr2=0x7f5c30141e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:10.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c371cb640 1 --2- 172.21.15.60:0/2541999955 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c3013b480 0x7f5c30141e00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5c28009e50 tx=0x7f5c2802bed0 comp rx=0 tx=0).stop 2024-03-27T21:40:10.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c371cb640 1 -- 172.21.15.60:0/2541999955 shutdown_connections 2024-03-27T21:40:10.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c371cb640 1 --2- 172.21.15.60:0/2541999955 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c3013b480 0x7f5c30141e00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:10.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c371cb640 1 --2- 172.21.15.60:0/2541999955 >> v2:172.21.15.60:3300/0 conn(0x7f5c300b40b0 0x7f5c300a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:10.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c371cb640 1 -- 172.21.15.60:0/2541999955 >> 172.21.15.60:0/2541999955 conn(0x7f5c300608e0 msgr2=0x7f5c30060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:10.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c371cb640 1 -- 172.21.15.60:0/2541999955 shutdown_connections 2024-03-27T21:40:10.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.852+0000 7f5c371cb640 1 -- 172.21.15.60:0/2541999955 wait complete. 2024-03-27T21:40:10.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.853+0000 7f5c371cb640 10 monclient: build_initial_monmap 2024-03-27T21:40:10.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.853+0000 7f5c371cb640 10 monclient: monmap: 2024-03-27T21:40:10.852 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:10.852 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:10.852 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:10.852 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:10.852 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:10.853 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:10.853 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:10.853 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:10.853 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:10.853 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:10.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.854+0000 7f5c371cb640 1 librados: starting msgr at 2024-03-27T21:40:10.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.854+0000 7f5c371cb640 1 librados: starting objecter 2024-03-27T21:40:10.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.854+0000 7f5c371cb640 1 Processor -- start 2024-03-27T21:40:10.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.854+0000 7f5c371cb640 1 -- start start 2024-03-27T21:40:10.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.854+0000 7f5c371cb640 1 librados: setting wanted keys 2024-03-27T21:40:10.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.854+0000 7f5c371cb640 1 librados: calling monclient init 2024-03-27T21:40:10.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.854+0000 7f5c371cb640 10 monclient: init 2024-03-27T21:40:10.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c371cb640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:10.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c371cb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:40:10.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c371cb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c300b40b0 0x7f5c3013a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:10.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c371cb640 10 monclient(hunting): picked mon.noname-a con 0x7f5c300b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:10.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c371cb640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c3013b020 0x7f5c3013e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:10.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c371cb640 10 monclient(hunting): picked mon.noname-b con 0x7f5c3013b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:10.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c371cb640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c3013b480 0x7f5c301340d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:10.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c371cb640 10 monclient(hunting): picked mon.noname-c con 0x7f5c3013b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:10.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c371cb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:10.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c371cb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c300ab390 con 0x7f5c300b40b0 2024-03-27T21:40:10.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c371cb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:10.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c371cb640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c300aac50 con 0x7f5c3013b020 2024-03-27T21:40:10.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c34f67640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c300b40b0 0x7f5c3013a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:10.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c371cb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:10.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c371cb640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3013d9e0 con 0x7f5c3013b480 2024-03-27T21:40:10.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c34f67640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c300b40b0 0x7f5c3013a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52748/0 (socket says 172.21.15.60:52748) 2024-03-27T21:40:10.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c34f67640 1 -- 172.21.15.60:0/3138508153 learned_addr learned my addr 172.21.15.60:0/3138508153 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:10.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c371cb640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:10.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c371cb640 10 monclient(hunting): authenticate will time out at 852.076625s 2024-03-27T21:40:10.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c34f67640 10 monclient(hunting): get_auth_request con 0x7f5c300b40b0 auth_method 0 2024-03-27T21:40:10.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c34f67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:10.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c34f67640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:10.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c34f67640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:10.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c35768640 1 --2- 172.21.15.60:0/3138508153 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c3013b480 0x7f5c301340d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:10.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c35768640 10 monclient(hunting): get_auth_request con 0x7f5c3013b480 auth_method 0 2024-03-27T21:40:10.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c35768640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:10.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c35768640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:10.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c35768640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:10.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c2ffff640 1 --2- 172.21.15.60:0/3138508153 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c3013b020 0x7f5c3013e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:10.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c34f67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:10.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c34f67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:10.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c34f67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:10.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c2ffff640 10 monclient(hunting): get_auth_request con 0x7f5c3013b020 auth_method 0 2024-03-27T21:40:10.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:10.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:10.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:10.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c35768640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:10.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c35768640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:10.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.855+0000 7f5c35768640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:10.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c34f67640 10 monclient(hunting): handle_auth_done global_id 4548 payload 824 2024-03-27T21:40:10.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c34f67640 1 -- 172.21.15.60:0/3138508153 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c3013b480 msgr2=0x7f5c301340d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:10.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c34f67640 1 --2- 172.21.15.60:0/3138508153 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c3013b480 0x7f5c301340d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:10.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c34f67640 1 -- 172.21.15.60:0/3138508153 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c3013b020 msgr2=0x7f5c3013e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:10.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c34f67640 1 --2- 172.21.15.60:0/3138508153 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c3013b020 0x7f5c3013e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:10.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c34f67640 10 monclient: _finish_hunting 0 2024-03-27T21:40:10.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c34f67640 1 monclient: found mon.noname-a 2024-03-27T21:40:10.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c34f67640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:10.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c34f67640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:10.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c34f67640 1 -- 172.21.15.60:0/3138508153 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c28009b30 con 0x7f5c300b40b0 2024-03-27T21:40:10.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c34f67640 1 --2- 172.21.15.60:0/3138508153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c300b40b0 0x7f5c3013a9a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5c2400e930 tx=0x7f5c2400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:10.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c2d7fa640 10 client.?.objecter ms_handle_connect 0x7f5c300b40b0 2024-03-27T21:40:10.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c2d7fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:10.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c2d7fa640 1 -- 172.21.15.60:0/3138508153 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c24009690 con 0x7f5c300b40b0 2024-03-27T21:40:10.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:10.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c2d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:10.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c2d7fa640 10 monclient: dump: 2024-03-27T21:40:10.858 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:10.858 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:10.858 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:10.858 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:10.858 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:10.858 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:10.858 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:10.858 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:10.858 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:10.858 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:10.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c2d7fa640 10 monclient: _finish_auth 0 2024-03-27T21:40:10.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c2d7fa640 10 monclient: _check_auth_tickets 2024-03-27T21:40:10.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:10.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c371cb640 5 monclient: authenticate success, global_id 4548 2024-03-27T21:40:10.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c371cb640 10 monclient: _renew_subs 2024-03-27T21:40:10.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c371cb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:10.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c2d7fa640 1 -- 172.21.15.60:0/3138508153 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2400c9a0 con 0x7f5c300b40b0 2024-03-27T21:40:10.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c371cb640 1 -- 172.21.15.60:0/3138508153 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c3013dcc0 con 0x7f5c300b40b0 2024-03-27T21:40:10.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c2d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:10.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c2d7fa640 1 -- 172.21.15.60:0/3138508153 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2400cd50 con 0x7f5c300b40b0 2024-03-27T21:40:10.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:10.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c2d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:10.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c2d7fa640 10 monclient: dump: 2024-03-27T21:40:10.859 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:10.859 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:10.859 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:10.860 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:10.860 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:10.860 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:10.860 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:10.860 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:10.860 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:10.860 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:10.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c2effd640 4 set_mon_vals no callback set 2024-03-27T21:40:10.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c371cb640 10 client.4548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:10.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c371cb640 10 monclient: _renew_subs 2024-03-27T21:40:10.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c371cb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:10.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c371cb640 1 -- 172.21.15.60:0/3138508153 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c301347f0 con 0x7f5c300b40b0 2024-03-27T21:40:10.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.856+0000 7f5c371cb640 1 librados: init done 2024-03-27T21:40:10.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.857+0000 7f5c371cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:10.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.857+0000 7f5c371cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:10.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.857+0000 7f5c371cb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:10.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.857+0000 7f5c371cb640 1 -- 172.21.15.60:0/3138508153 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c300aaa40 con 0x7f5c300b40b0 2024-03-27T21:40:10.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.858+0000 7f5c2d7fa640 1 -- 172.21.15.60:0/3138508153 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5c24013470 con 0x7f5c300b40b0 2024-03-27T21:40:10.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.859+0000 7f5c2d7fa640 1 --2- 172.21.15.60:0/3138508153 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5c1406c5c0 0x7f5c1406ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:10.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.859+0000 7f5c2ffff640 1 --2- 172.21.15.60:0/3138508153 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5c1406c5c0 0x7f5c1406ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:10.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.859+0000 7f5c2d7fa640 1 -- 172.21.15.60:0/3138508153 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f5c24089260 con 0x7f5c300b40b0 2024-03-27T21:40:10.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.859+0000 7f5c2d7fa640 10 client.4548.objecter ms_dispatch 0x7f5c30000b90 osd_map(28..28 src has 1..28) v4 2024-03-27T21:40:10.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.859+0000 7f5c2d7fa640 3 client.4548.objecter handle_osd_map got epochs [28,28] > 0 2024-03-27T21:40:10.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.859+0000 7f5c2d7fa640 3 client.4548.objecter handle_osd_map decoding full epoch 28 2024-03-27T21:40:10.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.859+0000 7f5c2ffff640 10 monclient: get_auth_request con 0x7f5c1406c5c0 auth_method 0 2024-03-27T21:40:10.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.859+0000 7f5c2d7fa640 20 client.4548.objecter dump_active .. 0 homeless 2024-03-27T21:40:10.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.859+0000 7f5c2ffff640 1 --2- 172.21.15.60:0/3138508153 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5c1406c5c0 0x7f5c1406ea80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5c300b3070 tx=0x7f5c1c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:10.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.859+0000 7f5c2d7fa640 10 client.4548.objecter ms_handle_connect 0x7f5c1406c5c0 2024-03-27T21:40:10.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.863+0000 7f5c2d7fa640 1 -- 172.21.15.60:0/3138508153 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5c24054790 con 0x7f5c300b40b0 2024-03-27T21:40:10.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.863+0000 7f5c2d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:10.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:10.863+0000 7f5c2d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:10.881 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:10.881 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:10.881 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:10.881 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:10.881 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:10.890 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:40:10.931 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:10.945+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:11.029 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:11.042+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:11.038 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:11.038 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:11.038 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:11.038 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-03-27T21:40:11.038 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.038 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:11.038 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:11.038 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 31.618188202999999, 2024-03-27T21:40:11.038 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43537978700000002, 2024-03-27T21:40:11.038 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:11.038 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.039 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:11.039 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:11.039 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:11.039 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-03-27T21:40:11.039 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.039 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:11.039 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.039 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:11.039 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:11.039 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.039 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.039 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.039 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:11.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:11.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:11.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432019+0000", 2024-03-27T21:40:11.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-03-27T21:40:11.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:11.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432027+0000", 2024-03-27T21:40:11.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1200000000000002e-06 2024-03-27T21:40:11.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:11.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432039+0000", 2024-03-27T21:40:11.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1123e-05 2024-03-27T21:40:11.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432051+0000", 2024-03-27T21:40:11.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-03-27T21:40:11.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:11.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432157+0000", 2024-03-27T21:40:11.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106846 2024-03-27T21:40:11.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-03-27T21:40:11.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432176+0000", 2024-03-27T21:40:11.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8490000000000001e-05 2024-03-27T21:40:11.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:11.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866170+0000", 2024-03-27T21:40:11.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43399370100000001 2024-03-27T21:40:11.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:11.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866335+0000", 2024-03-27T21:40:11.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016559699999999999 2024-03-27T21:40:11.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:11.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866495+0000", 2024-03-27T21:40:11.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015944200000000001 2024-03-27T21:40:11.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:11.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867012+0000", 2024-03-27T21:40:11.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0005176 2024-03-27T21:40:11.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.044 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:11.044 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867334+0000", 2024-03-27T21:40:11.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032158600000000001 2024-03-27T21:40:11.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.044 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:11.044 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867374+0000", 2024-03-27T21:40:11.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0547000000000003e-05 2024-03-27T21:40:11.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.044 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:11.044 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867400+0000", 2024-03-27T21:40:11.045 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5958000000000001e-05 2024-03-27T21:40:11.045 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.045 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:11.045 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.045 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.045 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.045 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:11.045 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:11.045 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 31.120805751999999, 2024-03-27T21:40:11.045 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064560310000000001, 2024-03-27T21:40:11.045 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:11.045 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.045 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:11.046 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:11.046 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:11.046 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 70 2024-03-27T21:40:11.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.046 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:11.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:11.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:11.046 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:11.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:11.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:11.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929402+0000", 2024-03-27T21:40:11.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:11.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929407+0000", 2024-03-27T21:40:11.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-06 2024-03-27T21:40:11.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:11.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929414+0000", 2024-03-27T21:40:11.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5689999999999999e-06 2024-03-27T21:40:11.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929419+0000", 2024-03-27T21:40:11.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-03-27T21:40:11.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:11.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930245+0000", 2024-03-27T21:40:11.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082521500000000002 2024-03-27T21:40:11.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:11.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930287+0000", 2024-03-27T21:40:11.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2305000000000002e-05 2024-03-27T21:40:11.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:11.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930367+0000", 2024-03-27T21:40:11.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9589e-05 2024-03-27T21:40:11.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:11.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.931214+0000", 2024-03-27T21:40:11.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000847827 2024-03-27T21:40:11.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:11.050 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935807+0000", 2024-03-27T21:40:11.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045923780000000003 2024-03-27T21:40:11.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:11.050 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935840+0000", 2024-03-27T21:40:11.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3501999999999999e-05 2024-03-27T21:40:11.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:11.050 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935859+0000", 2024-03-27T21:40:11.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8845e-05 2024-03-27T21:40:11.050 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.051 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:11.051 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.051 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:11.051 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:11.051 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 31.113546160999999, 2024-03-27T21:40:11.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071695429999999996, 2024-03-27T21:40:11.051 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:11.051 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.051 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:11.051 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:11.051 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:11.052 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-03-27T21:40:11.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.052 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:11.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:11.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:11.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:11.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:11.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:11.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936662+0000", 2024-03-27T21:40:11.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:11.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936737+0000", 2024-03-27T21:40:11.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5303999999999996e-05 2024-03-27T21:40:11.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:11.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936746+0000", 2024-03-27T21:40:11.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6340000000000007e-06 2024-03-27T21:40:11.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.054 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.054 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936753+0000", 2024-03-27T21:40:11.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-03-27T21:40:11.054 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.054 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:11.054 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937926+0000", 2024-03-27T21:40:11.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011731280000000001 2024-03-27T21:40:11.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:11.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937969+0000", 2024-03-27T21:40:11.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3285999999999999e-05 2024-03-27T21:40:11.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:11.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938040+0000", 2024-03-27T21:40:11.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0358000000000006e-05 2024-03-27T21:40:11.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:11.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939324+0000", 2024-03-27T21:40:11.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012844899999999999 2024-03-27T21:40:11.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:11.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943780+0000", 2024-03-27T21:40:11.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044555630000000001 2024-03-27T21:40:11.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:11.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943811+0000", 2024-03-27T21:40:11.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1415999999999999e-05 2024-03-27T21:40:11.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:11.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943832+0000", 2024-03-27T21:40:11.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0981000000000001e-05 2024-03-27T21:40:11.057 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.057 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:11.057 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.057 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:11.057 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:11.057 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 31.113448145, 2024-03-27T21:40:11.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071139740000000003, 2024-03-27T21:40:11.057 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:11.057 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.057 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:11.058 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:11.058 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:11.058 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-03-27T21:40:11.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.058 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:11.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:11.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:11.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:11.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:11.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:11.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936760+0000", 2024-03-27T21:40:11.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:11.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936765+0000", 2024-03-27T21:40:11.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6330000000000004e-06 2024-03-27T21:40:11.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:11.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936771+0000", 2024-03-27T21:40:11.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0560000000000001e-06 2024-03-27T21:40:11.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936776+0000", 2024-03-27T21:40:11.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1510000000000003e-06 2024-03-27T21:40:11.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:11.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938605+0000", 2024-03-27T21:40:11.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001829589 2024-03-27T21:40:11.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:11.061 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938657+0000", 2024-03-27T21:40:11.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1860000000000002e-05 2024-03-27T21:40:11.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.061 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:11.061 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938734+0000", 2024-03-27T21:40:11.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6639999999999998e-05 2024-03-27T21:40:11.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.061 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:11.061 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939562+0000", 2024-03-27T21:40:11.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082803100000000001 2024-03-27T21:40:11.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:11.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943838+0000", 2024-03-27T21:40:11.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042757050000000003 2024-03-27T21:40:11.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:11.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943859+0000", 2024-03-27T21:40:11.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1175e-05 2024-03-27T21:40:11.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:11.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943875+0000", 2024-03-27T21:40:11.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5744e-05 2024-03-27T21:40:11.062 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.063 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:11.063 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.063 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:11.063 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:11.063 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.969934093999999, 2024-03-27T21:40:11.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059712230000000003, 2024-03-27T21:40:11.063 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:11.063 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.063 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:11.063 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:11.063 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:11.063 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 112 2024-03-27T21:40:11.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.064 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:11.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:11.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:11.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:11.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:11.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:11.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080274+0000", 2024-03-27T21:40:11.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:11.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080318+0000", 2024-03-27T21:40:11.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3634999999999998e-05 2024-03-27T21:40:11.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:11.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080329+0000", 2024-03-27T21:40:11.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1232e-05 2024-03-27T21:40:11.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080333+0000", 2024-03-27T21:40:11.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6009999999999999e-06 2024-03-27T21:40:11.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:11.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083786+0000", 2024-03-27T21:40:11.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034528940000000002 2024-03-27T21:40:11.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:11.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083861+0000", 2024-03-27T21:40:11.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4893000000000003e-05 2024-03-27T21:40:11.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:11.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083981+0000", 2024-03-27T21:40:11.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012058299999999999 2024-03-27T21:40:11.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:11.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086148+0000", 2024-03-27T21:40:11.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021671469999999999 2024-03-27T21:40:11.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:11.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086195+0000", 2024-03-27T21:40:11.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6768999999999999e-05 2024-03-27T21:40:11.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:11.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086224+0000", 2024-03-27T21:40:11.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8758000000000001e-05 2024-03-27T21:40:11.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:11.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086246+0000", 2024-03-27T21:40:11.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2064000000000001e-05 2024-03-27T21:40:11.068 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.068 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:11.069 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.069 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:11.069 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:11.069 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.969871146999999, 2024-03-27T21:40:11.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066196320000000003, 2024-03-27T21:40:11.069 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:11.069 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.069 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:11.069 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:11.069 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:11.069 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113 2024-03-27T21:40:11.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.070 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:11.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:11.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:11.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:11.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:11.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:11.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080337+0000", 2024-03-27T21:40:11.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:11.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080340+0000", 2024-03-27T21:40:11.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6690000000000002e-06 2024-03-27T21:40:11.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:11.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080344+0000", 2024-03-27T21:40:11.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6550000000000002e-06 2024-03-27T21:40:11.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080348+0000", 2024-03-27T21:40:11.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9169999999999999e-06 2024-03-27T21:40:11.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:11.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084595+0000", 2024-03-27T21:40:11.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042471719999999996 2024-03-27T21:40:11.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:11.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084676+0000", 2024-03-27T21:40:11.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1147000000000001e-05 2024-03-27T21:40:11.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:11.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084796+0000", 2024-03-27T21:40:11.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012032900000000001 2024-03-27T21:40:11.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:11.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086190+0000", 2024-03-27T21:40:11.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013934399999999999 2024-03-27T21:40:11.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:11.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086905+0000", 2024-03-27T21:40:11.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071536199999999999 2024-03-27T21:40:11.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.074 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:11.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086937+0000", 2024-03-27T21:40:11.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2125000000000002e-05 2024-03-27T21:40:11.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.074 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:11.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086957+0000", 2024-03-27T21:40:11.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0157999999999999e-05 2024-03-27T21:40:11.075 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.075 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:11.075 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.075 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:11.075 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:11.075 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.969858430999999, 2024-03-27T21:40:11.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007405661, 2024-03-27T21:40:11.075 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:11.075 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.075 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:11.075 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:11.075 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:11.076 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 114 2024-03-27T21:40:11.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.076 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:11.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:11.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:11.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:11.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:11.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:11.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:11.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-03-27T21:40:11.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:11.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080353+0000", 2024-03-27T21:40:11.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.441e-06 2024-03-27T21:40:11.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:11.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080356+0000", 2024-03-27T21:40:11.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0769999999999999e-06 2024-03-27T21:40:11.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080358+0000", 2024-03-27T21:40:11.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3369999999999998e-06 2024-03-27T21:40:11.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:11.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085118+0000", 2024-03-27T21:40:11.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047603949999999997 2024-03-27T21:40:11.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.080 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:11.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085210+0000", 2024-03-27T21:40:11.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1983000000000003e-05 2024-03-27T21:40:11.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.080 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:11.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085332+0000", 2024-03-27T21:40:11.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121803 2024-03-27T21:40:11.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:11.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086292+0000", 2024-03-27T21:40:11.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095961400000000002 2024-03-27T21:40:11.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:11.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087710+0000", 2024-03-27T21:40:11.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014181459999999999 2024-03-27T21:40:11.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:11.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087736+0000", 2024-03-27T21:40:11.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6475e-05 2024-03-27T21:40:11.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.082 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:11.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087756+0000", 2024-03-27T21:40:11.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9626000000000001e-05 2024-03-27T21:40:11.082 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.082 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:11.082 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.082 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:11.083 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:11.083 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.969835996, 2024-03-27T21:40:11.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075011720000000004, 2024-03-27T21:40:11.083 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:11.083 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.083 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:11.083 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:11.083 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:11.083 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115 2024-03-27T21:40:11.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.083 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:11.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:11.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:11.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:11.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:11.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:11.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:11.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:11.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080458+0000", 2024-03-27T21:40:11.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5228000000000005e-05 2024-03-27T21:40:11.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:11.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080465+0000", 2024-03-27T21:40:11.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.238e-06 2024-03-27T21:40:11.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080468+0000", 2024-03-27T21:40:11.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5109999999999999e-06 2024-03-27T21:40:11.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:11.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086326+0000", 2024-03-27T21:40:11.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058579169999999998 2024-03-27T21:40:11.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:11.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086382+0000", 2024-03-27T21:40:11.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5309999999999997e-05 2024-03-27T21:40:11.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:11.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086457+0000", 2024-03-27T21:40:11.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5413000000000005e-05 2024-03-27T21:40:11.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:11.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087762+0000", 2024-03-27T21:40:11.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013052300000000001 2024-03-27T21:40:11.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:11.088 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087798+0000", 2024-03-27T21:40:11.088 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5614999999999997e-05 2024-03-27T21:40:11.088 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.088 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.088 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:11.088 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087840+0000", 2024-03-27T21:40:11.088 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2196000000000001e-05 2024-03-27T21:40:11.088 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.088 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.088 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:11.088 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087874+0000", 2024-03-27T21:40:11.088 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3785999999999998e-05 2024-03-27T21:40:11.088 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.088 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:11.089 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.089 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.089 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.089 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:11.089 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:11.089 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.969735913000001, 2024-03-27T21:40:11.089 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075050680000000002, 2024-03-27T21:40:11.089 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:11.089 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.089 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:11.089 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:11.089 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:11.089 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 116 2024-03-27T21:40:11.089 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.090 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:11.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:11.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:11.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:11.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:11.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:11.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080472+0000", 2024-03-27T21:40:11.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:11.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080475+0000", 2024-03-27T21:40:11.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.712e-06 2024-03-27T21:40:11.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:11.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080478+0000", 2024-03-27T21:40:11.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1070000000000001e-06 2024-03-27T21:40:11.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080481+0000", 2024-03-27T21:40:11.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4269999999999998e-06 2024-03-27T21:40:11.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:11.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086992+0000", 2024-03-27T21:40:11.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065114099999999996 2024-03-27T21:40:11.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:11.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087064+0000", 2024-03-27T21:40:11.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1927000000000004e-05 2024-03-27T21:40:11.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.093 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.093 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:11.093 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087188+0000", 2024-03-27T21:40:11.093 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123585 2024-03-27T21:40:11.093 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.093 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.093 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:11.093 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087766+0000", 2024-03-27T21:40:11.093 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057814300000000004 2024-03-27T21:40:11.093 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.093 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.093 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:11.093 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087934+0000", 2024-03-27T21:40:11.093 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000167894 2024-03-27T21:40:11.094 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.094 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.094 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:11.094 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087958+0000", 2024-03-27T21:40:11.094 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4663000000000001e-05 2024-03-27T21:40:11.094 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.094 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.094 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:11.094 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087978+0000", 2024-03-27T21:40:11.094 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9636e-05 2024-03-27T21:40:11.094 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.094 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:11.094 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.094 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.095 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.095 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:11.095 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:11.095 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.969703791000001, 2024-03-27T21:40:11.095 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076850679999999998, 2024-03-27T21:40:11.095 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:11.095 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.095 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:11.095 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:11.095 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:11.095 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 117 2024-03-27T21:40:11.095 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.095 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:11.095 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.096 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:11.096 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:11.096 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.096 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.096 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.096 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:11.096 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:11.096 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.096 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.096 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.096 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:11.096 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:11.096 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.096 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.096 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:11.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080508+0000", 2024-03-27T21:40:11.097 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.298e-06 2024-03-27T21:40:11.097 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:11.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080515+0000", 2024-03-27T21:40:11.097 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0790000000000003e-06 2024-03-27T21:40:11.097 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080518+0000", 2024-03-27T21:40:11.097 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8509999999999999e-06 2024-03-27T21:40:11.097 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.098 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:11.098 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087444+0000", 2024-03-27T21:40:11.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0069259120000000002 2024-03-27T21:40:11.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.098 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:11.098 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087491+0000", 2024-03-27T21:40:11.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7043999999999999e-05 2024-03-27T21:40:11.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.098 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:11.098 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087564+0000", 2024-03-27T21:40:11.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3603999999999995e-05 2024-03-27T21:40:11.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.099 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.099 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:11.099 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087902+0000", 2024-03-27T21:40:11.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033734100000000001 2024-03-27T21:40:11.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.099 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.099 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:11.099 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088117+0000", 2024-03-27T21:40:11.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000215164 2024-03-27T21:40:11.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.099 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.099 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:11.099 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088160+0000", 2024-03-27T21:40:11.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2933999999999999e-05 2024-03-27T21:40:11.100 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.100 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:11.100 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088190+0000", 2024-03-27T21:40:11.100 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0139999999999999e-05 2024-03-27T21:40:11.100 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.100 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:11.100 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.100 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.100 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:11.100 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:11.100 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.553423402, 2024-03-27T21:40:11.100 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052465439999999997, 2024-03-27T21:40:11.100 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:11.101 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.101 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:11.101 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:11.101 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:11.101 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 133 2024-03-27T21:40:11.101 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.101 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:11.101 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.101 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:11.101 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:11.101 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.101 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.101 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.101 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:11.101 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:11.102 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.102 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.102 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:11.102 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:11.102 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.102 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.102 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:11.102 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496788+0000", 2024-03-27T21:40:11.102 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-03-27T21:40:11.102 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.102 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:11.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496793+0000", 2024-03-27T21:40:11.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4790000000000004e-06 2024-03-27T21:40:11.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496796+0000", 2024-03-27T21:40:11.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1e-06 2024-03-27T21:40:11.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:11.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500447+0000", 2024-03-27T21:40:11.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036504459999999999 2024-03-27T21:40:11.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:11.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500492+0000", 2024-03-27T21:40:11.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5765e-05 2024-03-27T21:40:11.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:11.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500562+0000", 2024-03-27T21:40:11.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9206000000000001e-05 2024-03-27T21:40:11.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:11.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501959+0000", 2024-03-27T21:40:11.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001397187 2024-03-27T21:40:11.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:11.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501984+0000", 2024-03-27T21:40:11.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.552e-05 2024-03-27T21:40:11.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:11.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502010+0000", 2024-03-27T21:40:11.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5232000000000002e-05 2024-03-27T21:40:11.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:11.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502032+0000", 2024-03-27T21:40:11.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2337999999999999e-05 2024-03-27T21:40:11.106 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.106 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:11.106 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.106 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:11.106 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:11.106 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.553364744, 2024-03-27T21:40:11.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052892119999999997, 2024-03-27T21:40:11.106 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:11.106 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.106 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:11.106 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:11.106 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:11.107 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 134 2024-03-27T21:40:11.107 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.107 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:11.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.107 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:11.107 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:11.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.107 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.107 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:11.107 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:11.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.107 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.108 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:11.108 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:11.108 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.108 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.108 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:11.108 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496929+0000", 2024-03-27T21:40:11.108 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5098000000000004e-05 2024-03-27T21:40:11.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.108 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.108 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:11.108 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496937+0000", 2024-03-27T21:40:11.108 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9559999999999997e-06 2024-03-27T21:40:11.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.108 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.109 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.109 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496942+0000", 2024-03-27T21:40:11.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6350000000000003e-06 2024-03-27T21:40:11.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.109 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:11.109 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500749+0000", 2024-03-27T21:40:11.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038068529999999998 2024-03-27T21:40:11.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.109 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:11.109 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500792+0000", 2024-03-27T21:40:11.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2823e-05 2024-03-27T21:40:11.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:11.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500864+0000", 2024-03-27T21:40:11.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1715e-05 2024-03-27T21:40:11.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:11.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502068+0000", 2024-03-27T21:40:11.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012044860000000001 2024-03-27T21:40:11.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:11.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502073+0000", 2024-03-27T21:40:11.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5700000000000003e-06 2024-03-27T21:40:11.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:11.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502098+0000", 2024-03-27T21:40:11.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5120999999999999e-05 2024-03-27T21:40:11.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:11.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502133+0000", 2024-03-27T21:40:11.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5306999999999999e-05 2024-03-27T21:40:11.111 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.111 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:11.111 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.112 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:11.112 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:11.112 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.553259437000001, 2024-03-27T21:40:11.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062529040000000001, 2024-03-27T21:40:11.112 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:11.112 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.112 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:11.112 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:11.112 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:11.112 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135 2024-03-27T21:40:11.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.112 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:11.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:11.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:11.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:11.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:11.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:11.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:11.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.114 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:11.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496953+0000", 2024-03-27T21:40:11.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0709999999999996e-06 2024-03-27T21:40:11.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.114 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:11.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496961+0000", 2024-03-27T21:40:11.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1300000000000001e-06 2024-03-27T21:40:11.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.114 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496968+0000", 2024-03-27T21:40:11.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0999999999999998e-06 2024-03-27T21:40:11.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:11.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501244+0000", 2024-03-27T21:40:11.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042756310000000002 2024-03-27T21:40:11.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:11.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501377+0000", 2024-03-27T21:40:11.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013348799999999999 2024-03-27T21:40:11.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:11.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501455+0000", 2024-03-27T21:40:11.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7410000000000006e-05 2024-03-27T21:40:11.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:11.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502138+0000", 2024-03-27T21:40:11.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068386599999999999 2024-03-27T21:40:11.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:11.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503152+0000", 2024-03-27T21:40:11.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010139680000000001 2024-03-27T21:40:11.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:11.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503180+0000", 2024-03-27T21:40:11.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7426000000000001e-05 2024-03-27T21:40:11.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:11.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503202+0000", 2024-03-27T21:40:11.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2463000000000002e-05 2024-03-27T21:40:11.117 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.117 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:11.117 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.117 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:11.117 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:11.117 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.553235299000001, 2024-03-27T21:40:11.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066253680000000004, 2024-03-27T21:40:11.118 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:11.118 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.118 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:11.118 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:11.118 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:11.118 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 136 2024-03-27T21:40:11.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.118 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:11.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:11.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:11.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:11.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:11.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:11.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:11.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:11.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496977+0000", 2024-03-27T21:40:11.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5049999999999999e-06 2024-03-27T21:40:11.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:11.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496984+0000", 2024-03-27T21:40:11.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1520000000000003e-06 2024-03-27T21:40:11.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496990+0000", 2024-03-27T21:40:11.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-03-27T21:40:11.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:11.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501669+0000", 2024-03-27T21:40:11.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046793859999999998 2024-03-27T21:40:11.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:11.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501714+0000", 2024-03-27T21:40:11.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5154999999999997e-05 2024-03-27T21:40:11.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:11.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501783+0000", 2024-03-27T21:40:11.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8454999999999994e-05 2024-03-27T21:40:11.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:11.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503226+0000", 2024-03-27T21:40:11.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001443441 2024-03-27T21:40:11.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.122 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:11.122 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503540+0000", 2024-03-27T21:40:11.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031347100000000001 2024-03-27T21:40:11.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.122 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:11.122 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503563+0000", 2024-03-27T21:40:11.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3223000000000001e-05 2024-03-27T21:40:11.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.122 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:11.122 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503599+0000", 2024-03-27T21:40:11.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6105000000000001e-05 2024-03-27T21:40:11.123 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.123 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:11.123 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.123 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:11.123 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:11.123 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.553212489, 2024-03-27T21:40:11.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074261320000000002, 2024-03-27T21:40:11.123 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:11.123 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.123 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:11.123 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:11.123 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:11.124 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 137 2024-03-27T21:40:11.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.124 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:11.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:11.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:11.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:11.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:11.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:11.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:11.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:11.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497077+0000", 2024-03-27T21:40:11.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1599000000000003e-05 2024-03-27T21:40:11.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:11.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497085+0000", 2024-03-27T21:40:11.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6109999999999996e-06 2024-03-27T21:40:11.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497092+0000", 2024-03-27T21:40:11.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.742e-06 2024-03-27T21:40:11.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:11.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502159+0000", 2024-03-27T21:40:11.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050666870000000003 2024-03-27T21:40:11.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:11.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502200+0000", 2024-03-27T21:40:11.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1919999999999998e-05 2024-03-27T21:40:11.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.127 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:11.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502280+0000", 2024-03-27T21:40:11.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9417000000000002e-05 2024-03-27T21:40:11.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.127 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:11.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503605+0000", 2024-03-27T21:40:11.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001325641 2024-03-27T21:40:11.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.127 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:11.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504361+0000", 2024-03-27T21:40:11.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075555399999999997 2024-03-27T21:40:11.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:11.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504386+0000", 2024-03-27T21:40:11.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5369999999999999e-05 2024-03-27T21:40:11.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:11.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504422+0000", 2024-03-27T21:40:11.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6019000000000002e-05 2024-03-27T21:40:11.128 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.128 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:11.128 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.128 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:11.129 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:11.129 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.553108691999999, 2024-03-27T21:40:11.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074059750000000004, 2024-03-27T21:40:11.129 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:11.129 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.129 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:11.129 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:11.129 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:11.129 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-03-27T21:40:11.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.129 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:11.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:11.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:11.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:11.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:11.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:11.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:11.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:11.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497103+0000", 2024-03-27T21:40:11.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.856e-06 2024-03-27T21:40:11.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:11.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497121+0000", 2024-03-27T21:40:11.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7190999999999999e-05 2024-03-27T21:40:11.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497126+0000", 2024-03-27T21:40:11.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3770000000000004e-06 2024-03-27T21:40:11.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:11.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502833+0000", 2024-03-27T21:40:11.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057073510000000003 2024-03-27T21:40:11.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:11.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502881+0000", 2024-03-27T21:40:11.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7961999999999999e-05 2024-03-27T21:40:11.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:11.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502949+0000", 2024-03-27T21:40:11.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7957e-05 2024-03-27T21:40:11.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:11.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504430+0000", 2024-03-27T21:40:11.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001480763 2024-03-27T21:40:11.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:11.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504459+0000", 2024-03-27T21:40:11.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8983999999999999e-05 2024-03-27T21:40:11.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:11.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504486+0000", 2024-03-27T21:40:11.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6883999999999999e-05 2024-03-27T21:40:11.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:11.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504506+0000", 2024-03-27T21:40:11.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0214e-05 2024-03-27T21:40:11.134 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.134 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:11.134 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.134 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:11.134 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:11.134 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.553076858000001, 2024-03-27T21:40:11.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074754859999999999, 2024-03-27T21:40:11.134 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:11.134 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.135 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:11.135 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:11.135 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:11.135 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-03-27T21:40:11.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.135 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:11.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:11.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:11.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:11.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:11.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:11.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:11.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:11.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497137+0000", 2024-03-27T21:40:11.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0730000000000004e-06 2024-03-27T21:40:11.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:11.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497143+0000", 2024-03-27T21:40:11.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3759999999999996e-06 2024-03-27T21:40:11.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497150+0000", 2024-03-27T21:40:11.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.002e-06 2024-03-27T21:40:11.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:11.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503249+0000", 2024-03-27T21:40:11.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060989750000000004 2024-03-27T21:40:11.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:11.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503291+0000", 2024-03-27T21:40:11.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1563999999999997e-05 2024-03-27T21:40:11.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:11.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503376+0000", 2024-03-27T21:40:11.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5752000000000002e-05 2024-03-27T21:40:11.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:11.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504545+0000", 2024-03-27T21:40:11.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169172 2024-03-27T21:40:11.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:11.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504550+0000", 2024-03-27T21:40:11.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7419999999999997e-06 2024-03-27T21:40:11.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:11.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504574+0000", 2024-03-27T21:40:11.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4124000000000001e-05 2024-03-27T21:40:11.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:11.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504607+0000", 2024-03-27T21:40:11.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3118999999999999e-05 2024-03-27T21:40:11.139 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.140 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:11.140 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.140 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:11.140 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:11.140 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.553053192, 2024-03-27T21:40:11.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078150589999999992, 2024-03-27T21:40:11.140 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:11.140 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.140 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:11.140 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:11.140 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:11.140 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-03-27T21:40:11.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.141 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:11.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:11.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:11.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:11.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:11.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:11.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497155+0000", 2024-03-27T21:40:11.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:11.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497230+0000", 2024-03-27T21:40:11.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4789000000000005e-05 2024-03-27T21:40:11.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:11.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497241+0000", 2024-03-27T21:40:11.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1442e-05 2024-03-27T21:40:11.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497255+0000", 2024-03-27T21:40:11.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3087e-05 2024-03-27T21:40:11.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:11.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503630+0000", 2024-03-27T21:40:11.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063755269999999998 2024-03-27T21:40:11.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:11.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503673+0000", 2024-03-27T21:40:11.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2616e-05 2024-03-27T21:40:11.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:11.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503738+0000", 2024-03-27T21:40:11.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5114e-05 2024-03-27T21:40:11.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:11.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504907+0000", 2024-03-27T21:40:11.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169432 2024-03-27T21:40:11.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:11.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504926+0000", 2024-03-27T21:40:11.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9086999999999999e-05 2024-03-27T21:40:11.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:11.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504950+0000", 2024-03-27T21:40:11.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3598999999999999e-05 2024-03-27T21:40:11.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:11.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504971+0000", 2024-03-27T21:40:11.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0772999999999999e-05 2024-03-27T21:40:11.145 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.145 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:11.145 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.146 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:11.146 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:11.146 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 30.552946152000001, 2024-03-27T21:40:11.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082962590000000003, 2024-03-27T21:40:11.146 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:11.146 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.146 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:11.146 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:11.146 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:11.146 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-03-27T21:40:11.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.146 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:11.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:11.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:11.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:11.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:11.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:11.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497262+0000", 2024-03-27T21:40:11.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:11.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497267+0000", 2024-03-27T21:40:11.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.955e-06 2024-03-27T21:40:11.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:11.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497275+0000", 2024-03-27T21:40:11.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.824e-06 2024-03-27T21:40:11.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497280+0000", 2024-03-27T21:40:11.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8230000000000002e-06 2024-03-27T21:40:11.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:11.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504628+0000", 2024-03-27T21:40:11.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007348561 2024-03-27T21:40:11.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:11.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504673+0000", 2024-03-27T21:40:11.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4313000000000001e-05 2024-03-27T21:40:11.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:11.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504745+0000", 2024-03-27T21:40:11.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2303000000000002e-05 2024-03-27T21:40:11.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:11.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505358+0000", 2024-03-27T21:40:11.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061281300000000003 2024-03-27T21:40:11.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:11.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505503+0000", 2024-03-27T21:40:11.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000145134 2024-03-27T21:40:11.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:11.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505526+0000", 2024-03-27T21:40:11.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3312e-05 2024-03-27T21:40:11.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:11.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505559+0000", 2024-03-27T21:40:11.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2821999999999998e-05 2024-03-27T21:40:11.151 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.151 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:11.151 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.151 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:11.151 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:11.151 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 29.943723912999999, 2024-03-27T21:40:11.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053753339999999998, 2024-03-27T21:40:11.152 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:11.152 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.152 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:11.152 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:11.152 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:11.152 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159 2024-03-27T21:40:11.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.152 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:11.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:11.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:11.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:11.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:11.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:11.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:11.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:11.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:11.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106488+0000", 2024-03-27T21:40:11.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9830000000000001e-06 2024-03-27T21:40:11.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:11.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106491+0000", 2024-03-27T21:40:11.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3459999999999998e-06 2024-03-27T21:40:11.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106493+0000", 2024-03-27T21:40:11.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.509e-06 2024-03-27T21:40:11.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:11.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110636+0000", 2024-03-27T21:40:11.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041427260000000002 2024-03-27T21:40:11.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:11.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110683+0000", 2024-03-27T21:40:11.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6557000000000002e-05 2024-03-27T21:40:11.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:11.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110755+0000", 2024-03-27T21:40:11.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2119000000000005e-05 2024-03-27T21:40:11.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:11.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111415+0000", 2024-03-27T21:40:11.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066065199999999996 2024-03-27T21:40:11.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:11.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111793+0000", 2024-03-27T21:40:11.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037732100000000001 2024-03-27T21:40:11.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:11.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111829+0000", 2024-03-27T21:40:11.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418999999999998e-05 2024-03-27T21:40:11.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:11.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:11.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:11.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111860+0000", 2024-03-27T21:40:11.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1003000000000001e-05 2024-03-27T21:40:11.157 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.157 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:11.157 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.157 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:11.157 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:11.157 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:11.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.080+0000 7f5c371cb640 10 start_mon_command cmd=[{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:40:11.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.080+0000 7f5c371cb640 10 monclient: _send_command 2 [{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:40:11.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.080+0000 7f5c371cb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:11.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.080+0000 7f5c371cb640 1 -- 172.21.15.60:0/3138508153 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f5c30069470 con 0x7f5c300b40b0 2024-03-27T21:40:11.159 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:11.172+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:11.159 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:40:11.259 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:11.273+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:11.261 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:11.275+0000 7fa1b5c1a640 -1 Falling back to public interface 2024-03-27T21:40:11.294 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:11.294 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:11.294 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:11.294 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:11.306 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:40:11.360 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:11.374+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:11.456 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:11.456 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:11.456 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:11.457 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:11.457 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:11.461 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:11.475+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:11.465 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:40:11.562 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:11.576+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:11.616 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:11.616 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:11.616 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:11.616 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-03-27T21:40:11.616 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.616 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:20 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2024-03-27T21:40:11.616 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:11.616 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.739571474000002, 2024-03-27T21:40:11.616 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015345630000000001, 2024-03-27T21:40:11.616 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:11.617 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.617 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:11.617 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.617 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:11.617 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:11.617 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.617 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.617 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.617 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:11.617 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:11.617 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.617 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.617 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.617 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:11.617 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888562+0000", 2024-03-27T21:40:11.618 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:11.618 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.618 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.618 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:11.618 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888735+0000", 2024-03-27T21:40:11.618 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017254799999999999 2024-03-27T21:40:11.618 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.618 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.618 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:11.618 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888747+0000", 2024-03-27T21:40:11.618 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2469999999999999e-05 2024-03-27T21:40:11.618 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.618 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.618 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.619 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888761+0000", 2024-03-27T21:40:11.619 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3254e-05 2024-03-27T21:40:11.619 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.619 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.619 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:11.619 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888804+0000", 2024-03-27T21:40:11.619 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3129e-05 2024-03-27T21:40:11.619 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.619 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.619 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:11.619 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888820+0000", 2024-03-27T21:40:11.619 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6426000000000002e-05 2024-03-27T21:40:11.619 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.619 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.620 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:11.620 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890060+0000", 2024-03-27T21:40:11.620 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012398050000000001 2024-03-27T21:40:11.620 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.620 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.620 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:11.620 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890098+0000", 2024-03-27T21:40:11.620 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7795e-05 2024-03-27T21:40:11.620 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.620 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:11.620 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.620 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.620 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.620 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2024-03-27T21:40:11.620 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:11.621 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.738747073999999, 2024-03-27T21:40:11.621 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001069555, 2024-03-27T21:40:11.621 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:11.621 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.621 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:11.621 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.621 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:11.621 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:11.621 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.621 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.621 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.621 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:11.621 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:11.621 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.621 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.622 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.622 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:11.622 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889387+0000", 2024-03-27T21:40:11.622 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:11.622 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.622 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.622 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:11.622 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889455+0000", 2024-03-27T21:40:11.622 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7930999999999997e-05 2024-03-27T21:40:11.622 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.622 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.622 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:11.622 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889470+0000", 2024-03-27T21:40:11.622 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5554e-05 2024-03-27T21:40:11.623 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.623 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.623 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.623 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889476+0000", 2024-03-27T21:40:11.623 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0619999999999997e-06 2024-03-27T21:40:11.623 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.623 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.623 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:11.623 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889513+0000", 2024-03-27T21:40:11.623 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6606999999999997e-05 2024-03-27T21:40:11.623 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.623 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.623 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:11.623 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889528+0000", 2024-03-27T21:40:11.624 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4982e-05 2024-03-27T21:40:11.624 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.624 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.624 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:11.624 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890421+0000", 2024-03-27T21:40:11.624 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00089293700000000001 2024-03-27T21:40:11.624 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.624 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.624 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:11.624 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890457+0000", 2024-03-27T21:40:11.624 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6260000000000002e-05 2024-03-27T21:40:11.624 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.624 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:11.624 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.624 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.625 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.625 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:38 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'24, mlcod=17'22)", 2024-03-27T21:40:11.625 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:11.625 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.721511108000001, 2024-03-27T21:40:11.625 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016182080000000001, 2024-03-27T21:40:11.625 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:11.625 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.625 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:11.625 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.625 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:11.625 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:11.625 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.625 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.625 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.625 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:11.626 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:11.626 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.626 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.626 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.626 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:11.626 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906623+0000", 2024-03-27T21:40:11.626 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:11.626 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.626 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.626 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:11.626 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906687+0000", 2024-03-27T21:40:11.626 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3706000000000003e-05 2024-03-27T21:40:11.626 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.626 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.627 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:11.627 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906698+0000", 2024-03-27T21:40:11.627 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1752e-05 2024-03-27T21:40:11.627 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.627 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.627 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.627 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906704+0000", 2024-03-27T21:40:11.627 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3940000000000004e-06 2024-03-27T21:40:11.627 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.627 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.627 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:11.627 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906741+0000", 2024-03-27T21:40:11.627 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6944999999999999e-05 2024-03-27T21:40:11.627 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.627 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.628 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:11.628 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906760+0000", 2024-03-27T21:40:11.628 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9293e-05 2024-03-27T21:40:11.628 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.628 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.628 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:11.628 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908211+0000", 2024-03-27T21:40:11.628 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014511350000000001 2024-03-27T21:40:11.628 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.628 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.628 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:11.628 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908242+0000", 2024-03-27T21:40:11.628 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0712e-05 2024-03-27T21:40:11.628 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.628 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:11.629 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.629 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.629 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.629 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:39 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'25, mlcod=17'22)", 2024-03-27T21:40:11.629 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:11.629 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.720666193, 2024-03-27T21:40:11.629 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012060199999999999, 2024-03-27T21:40:11.629 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:11.629 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.629 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:11.629 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.629 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:11.629 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:11.629 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.630 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.630 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.630 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:11.630 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:11.630 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.630 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.630 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.630 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:11.630 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:11.630 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:11.630 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.630 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.630 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:11.630 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907533+0000", 2024-03-27T21:40:11.630 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5675999999999999e-05 2024-03-27T21:40:11.631 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.631 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.631 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:11.631 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907545+0000", 2024-03-27T21:40:11.631 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.115e-05 2024-03-27T21:40:11.631 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.631 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.631 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.631 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907550+0000", 2024-03-27T21:40:11.631 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8320000000000002e-06 2024-03-27T21:40:11.631 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.631 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.631 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:11.631 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907592+0000", 2024-03-27T21:40:11.632 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1309e-05 2024-03-27T21:40:11.632 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.632 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.632 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:11.632 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907611+0000", 2024-03-27T21:40:11.632 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9009e-05 2024-03-27T21:40:11.632 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.632 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.632 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:11.632 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908638+0000", 2024-03-27T21:40:11.632 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001027533 2024-03-27T21:40:11.632 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.632 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.632 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:11.633 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908675+0000", 2024-03-27T21:40:11.633 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6329999999999999e-05 2024-03-27T21:40:11.633 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.633 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:11.633 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.633 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.633 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.633 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:57 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'35, mlcod=17'33)", 2024-03-27T21:40:11.633 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:11.633 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.706581760999999, 2024-03-27T21:40:11.633 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001346578, 2024-03-27T21:40:11.633 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:11.633 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.633 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:11.633 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.634 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:11.634 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:11.634 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.634 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.634 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.634 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:11.634 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:11.634 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.634 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.634 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.634 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:11.634 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921552+0000", 2024-03-27T21:40:11.634 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.634 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.635 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.635 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:11.635 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921615+0000", 2024-03-27T21:40:11.635 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2787999999999996e-05 2024-03-27T21:40:11.635 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.635 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.635 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:11.635 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921626+0000", 2024-03-27T21:40:11.635 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0888e-05 2024-03-27T21:40:11.635 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.635 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.635 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.635 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921635+0000", 2024-03-27T21:40:11.635 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4639999999999995e-06 2024-03-27T21:40:11.636 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.636 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.636 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:11.636 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921673+0000", 2024-03-27T21:40:11.636 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7891e-05 2024-03-27T21:40:11.636 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.636 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.636 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:11.636 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921694+0000", 2024-03-27T21:40:11.636 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0426000000000001e-05 2024-03-27T21:40:11.636 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.636 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.636 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:11.636 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922865+0000", 2024-03-27T21:40:11.637 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011709509999999999 2024-03-27T21:40:11.637 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.637 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.637 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:11.637 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922900+0000", 2024-03-27T21:40:11.637 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4819e-05 2024-03-27T21:40:11.637 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.637 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:11.637 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.637 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.637 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.637 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:71 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'45, mlcod=17'43)", 2024-03-27T21:40:11.637 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:11.637 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.691283048999999, 2024-03-27T21:40:11.638 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001460907, 2024-03-27T21:40:11.638 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:11.638 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.638 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:11.638 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.638 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:11.638 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:11.638 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.638 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.638 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.638 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:11.638 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:11.638 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.638 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.639 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.639 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:11.639 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936851+0000", 2024-03-27T21:40:11.639 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:11.639 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.639 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.639 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:11.639 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936903+0000", 2024-03-27T21:40:11.639 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2302999999999997e-05 2024-03-27T21:40:11.639 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.639 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.639 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:11.639 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936915+0000", 2024-03-27T21:40:11.639 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1967000000000001e-05 2024-03-27T21:40:11.639 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.639 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.640 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.640 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936923+0000", 2024-03-27T21:40:11.640 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9370000000000004e-06 2024-03-27T21:40:11.640 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.640 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.640 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:11.640 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936970+0000", 2024-03-27T21:40:11.640 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6987000000000003e-05 2024-03-27T21:40:11.640 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.640 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.640 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:11.640 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937002+0000", 2024-03-27T21:40:11.640 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2410999999999999e-05 2024-03-27T21:40:11.640 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.641 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.641 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:11.641 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938279+0000", 2024-03-27T21:40:11.641 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001276552 2024-03-27T21:40:11.641 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.641 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.641 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:11.641 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938313+0000", 2024-03-27T21:40:11.641 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3526999999999999e-05 2024-03-27T21:40:11.641 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.641 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:11.641 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.641 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.641 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.641 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:72 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'46, mlcod=17'43)", 2024-03-27T21:40:11.642 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:11.642 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.690645487000001, 2024-03-27T21:40:11.642 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014239490000000001, 2024-03-27T21:40:11.642 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:11.642 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.642 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:11.642 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.642 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:11.642 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:11.642 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.642 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.642 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.642 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:11.642 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:11.642 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.643 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.643 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.643 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:11.643 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937488+0000", 2024-03-27T21:40:11.643 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:11.643 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.643 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.643 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:11.643 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937547+0000", 2024-03-27T21:40:11.643 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8196999999999999e-05 2024-03-27T21:40:11.643 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.643 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.643 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:11.643 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937559+0000", 2024-03-27T21:40:11.643 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1928e-05 2024-03-27T21:40:11.644 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.644 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.644 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.644 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937564+0000", 2024-03-27T21:40:11.644 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4759999999999997e-06 2024-03-27T21:40:11.644 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.644 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.644 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:11.644 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937621+0000", 2024-03-27T21:40:11.644 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6712000000000002e-05 2024-03-27T21:40:11.644 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.644 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.644 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:11.644 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937640+0000", 2024-03-27T21:40:11.645 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9517e-05 2024-03-27T21:40:11.645 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.645 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.645 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:11.645 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938883+0000", 2024-03-27T21:40:11.645 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001242567 2024-03-27T21:40:11.645 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.645 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.645 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:11.645 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938913+0000", 2024-03-27T21:40:11.645 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0287e-05 2024-03-27T21:40:11.645 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.645 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:11.645 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.645 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.646 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.646 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:73 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'47, mlcod=17'43)", 2024-03-27T21:40:11.646 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:11.646 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.689969392999998, 2024-03-27T21:40:11.646 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010746239999999999, 2024-03-27T21:40:11.646 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:11.646 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.646 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:11.646 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.646 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:11.646 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:11.646 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.646 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.647 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.647 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:11.647 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:11.647 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.647 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.647 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.647 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:11.647 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:11.647 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:11.647 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.647 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.647 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:11.647 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938217+0000", 2024-03-27T21:40:11.647 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2743e-05 2024-03-27T21:40:11.647 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.648 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.648 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:11.648 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938229+0000", 2024-03-27T21:40:11.648 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1956999999999999e-05 2024-03-27T21:40:11.648 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.648 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.648 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.648 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938235+0000", 2024-03-27T21:40:11.648 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4589999999999997e-06 2024-03-27T21:40:11.648 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.648 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.648 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:11.648 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938341+0000", 2024-03-27T21:40:11.648 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000106054 2024-03-27T21:40:11.648 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.649 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.649 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:11.649 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938369+0000", 2024-03-27T21:40:11.649 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8291000000000001e-05 2024-03-27T21:40:11.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.649 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.649 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:11.649 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939209+0000", 2024-03-27T21:40:11.649 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084045400000000003 2024-03-27T21:40:11.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.649 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.649 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:11.649 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939240+0000", 2024-03-27T21:40:11.649 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0429999999999998e-05 2024-03-27T21:40:11.649 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.650 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:11.650 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.650 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.650 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.650 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:75 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'49, mlcod=17'47)", 2024-03-27T21:40:11.650 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:11.650 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.683531746, 2024-03-27T21:40:11.650 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001017001, 2024-03-27T21:40:11.650 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:11.650 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.650 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:11.650 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.650 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:11.650 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:11.650 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.651 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.651 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.651 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:11.651 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:11.651 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.651 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.651 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.651 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:11.651 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944601+0000", 2024-03-27T21:40:11.651 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999986 2024-03-27T21:40:11.651 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.651 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.651 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:11.651 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944614+0000", 2024-03-27T21:40:11.652 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2496e-05 2024-03-27T21:40:11.652 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.652 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.652 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:11.652 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944640+0000", 2024-03-27T21:40:11.652 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5865999999999999e-05 2024-03-27T21:40:11.652 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.652 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.652 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.652 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944651+0000", 2024-03-27T21:40:11.652 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1797000000000001e-05 2024-03-27T21:40:11.652 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.652 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.652 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:11.652 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944762+0000", 2024-03-27T21:40:11.653 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110412 2024-03-27T21:40:11.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.653 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.653 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:11.653 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944820+0000", 2024-03-27T21:40:11.653 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7649999999999999e-05 2024-03-27T21:40:11.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.653 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.653 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:11.653 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945585+0000", 2024-03-27T21:40:11.653 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076562600000000005 2024-03-27T21:40:11.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.653 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.653 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:11.653 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945620+0000", 2024-03-27T21:40:11.654 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4764000000000001e-05 2024-03-27T21:40:11.654 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.654 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:11.654 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.654 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.654 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.654 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:86 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'50)", 2024-03-27T21:40:11.654 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:11.654 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.675770259, 2024-03-27T21:40:11.654 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011392380000000001, 2024-03-27T21:40:11.654 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:11.654 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.654 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:11.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.655 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:11.655 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:11.655 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.655 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:11.655 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:11.655 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.655 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:11.655 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:11.655 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.656 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:11.656 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952373+0000", 2024-03-27T21:40:11.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9220000000000003e-06 2024-03-27T21:40:11.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.656 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.656 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:11.656 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952381+0000", 2024-03-27T21:40:11.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3070000000000003e-06 2024-03-27T21:40:11.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.656 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.656 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.656 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952385+0000", 2024-03-27T21:40:11.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.121e-06 2024-03-27T21:40:11.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.657 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.657 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:11.657 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952419+0000", 2024-03-27T21:40:11.657 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4082999999999999e-05 2024-03-27T21:40:11.657 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.657 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.657 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:11.657 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952440+0000", 2024-03-27T21:40:11.657 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0837999999999999e-05 2024-03-27T21:40:11.657 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.657 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.657 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:11.657 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953472+0000", 2024-03-27T21:40:11.657 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001032119 2024-03-27T21:40:11.657 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.658 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.658 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:11.658 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953504+0000", 2024-03-27T21:40:11.658 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1250000000000001e-05 2024-03-27T21:40:11.658 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.658 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:11.658 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.658 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.658 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.658 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:87 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-27T21:40:11.658 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:11.658 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.675403001999999, 2024-03-27T21:40:11.658 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00135519, 2024-03-27T21:40:11.658 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:11.658 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.659 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:11.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.659 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:11.659 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:11.659 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.659 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.659 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:11.659 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:11.659 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.659 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.659 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:11.659 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952731+0000", 2024-03-27T21:40:11.659 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.660 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.660 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.660 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:11.660 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952781+0000", 2024-03-27T21:40:11.660 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0161000000000003e-05 2024-03-27T21:40:11.660 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.660 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.660 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:11.660 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952794+0000", 2024-03-27T21:40:11.660 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2853e-05 2024-03-27T21:40:11.660 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.660 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.660 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.660 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952799+0000", 2024-03-27T21:40:11.660 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5560000000000001e-06 2024-03-27T21:40:11.660 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.661 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:11.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952832+0000", 2024-03-27T21:40:11.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3834999999999997e-05 2024-03-27T21:40:11.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.661 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:11.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952851+0000", 2024-03-27T21:40:11.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8289999999999999e-05 2024-03-27T21:40:11.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.661 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:11.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954055+0000", 2024-03-27T21:40:11.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012037949999999999 2024-03-27T21:40:11.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.662 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.662 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:11.662 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954087+0000", 2024-03-27T21:40:11.662 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2292999999999997e-05 2024-03-27T21:40:11.662 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.662 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:11.662 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.662 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.662 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.662 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:89 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'59, mlcod=17'57)", 2024-03-27T21:40:11.662 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:11.662 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.673128780999999, 2024-03-27T21:40:11.662 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00138893, 2024-03-27T21:40:11.662 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:11.662 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.663 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:11.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.663 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:11.663 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:11.663 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.663 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.663 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:11.663 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:11.663 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.663 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.663 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:11.663 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955005+0000", 2024-03-27T21:40:11.663 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.664 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:11.664 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955071+0000", 2024-03-27T21:40:11.664 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5290999999999995e-05 2024-03-27T21:40:11.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.664 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:11.664 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955081+0000", 2024-03-27T21:40:11.664 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0572e-05 2024-03-27T21:40:11.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.664 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.664 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955086+0000", 2024-03-27T21:40:11.664 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8380000000000001e-06 2024-03-27T21:40:11.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.665 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:11.665 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955122+0000", 2024-03-27T21:40:11.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5698000000000003e-05 2024-03-27T21:40:11.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.665 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:11.665 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955140+0000", 2024-03-27T21:40:11.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7900000000000001e-05 2024-03-27T21:40:11.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.665 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:11.665 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956366+0000", 2024-03-27T21:40:11.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001226471 2024-03-27T21:40:11.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.666 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:11.666 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956395+0000", 2024-03-27T21:40:11.666 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8782e-05 2024-03-27T21:40:11.666 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.666 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:11.666 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.666 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:90 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'60, mlcod=17'57)", 2024-03-27T21:40:11.666 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:11.666 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.672444204000001, 2024-03-27T21:40:11.666 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010778859999999999, 2024-03-27T21:40:11.666 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:11.667 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.667 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:11.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:11.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:11.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:11.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:11.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:11.668 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:11.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.668 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:11.668 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955747+0000", 2024-03-27T21:40:11.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7130999999999999e-05 2024-03-27T21:40:11.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.668 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:11.668 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955756+0000", 2024-03-27T21:40:11.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2750000000000001e-06 2024-03-27T21:40:11.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.668 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.669 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955761+0000", 2024-03-27T21:40:11.669 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3900000000000003e-06 2024-03-27T21:40:11.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.669 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:11.669 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955797+0000", 2024-03-27T21:40:11.669 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6650000000000003e-05 2024-03-27T21:40:11.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.669 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:11.669 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955816+0000", 2024-03-27T21:40:11.669 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8195000000000001e-05 2024-03-27T21:40:11.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.670 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:11.670 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956739+0000", 2024-03-27T21:40:11.670 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00092369700000000002 2024-03-27T21:40:11.670 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.670 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:11.670 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956768+0000", 2024-03-27T21:40:11.670 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9048e-05 2024-03-27T21:40:11.670 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.670 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:11.670 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.670 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.670 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:118 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'80, mlcod=17'78)", 2024-03-27T21:40:11.670 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:11.671 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.539453381000001, 2024-03-27T21:40:11.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014200669999999999, 2024-03-27T21:40:11.671 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:11.671 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.671 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:11.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.671 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:11.671 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:11.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.671 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:11.671 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:11.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.672 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:11.672 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:11.672 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.672 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.672 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:11.672 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088756+0000", 2024-03-27T21:40:11.672 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4902999999999998e-05 2024-03-27T21:40:11.672 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.672 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:11.672 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088764+0000", 2024-03-27T21:40:11.672 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8169999999999999e-06 2024-03-27T21:40:11.672 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.673 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.673 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088767+0000", 2024-03-27T21:40:11.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8310000000000003e-06 2024-03-27T21:40:11.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.673 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:11.673 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088798+0000", 2024-03-27T21:40:11.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0800999999999999e-05 2024-03-27T21:40:11.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.673 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:11.674 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088817+0000", 2024-03-27T21:40:11.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8467000000000001e-05 2024-03-27T21:40:11.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.674 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:11.674 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090074+0000", 2024-03-27T21:40:11.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012574089999999999 2024-03-27T21:40:11.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.674 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:11.674 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090101+0000", 2024-03-27T21:40:11.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7223e-05 2024-03-27T21:40:11.674 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.674 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:11.675 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.675 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:119 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'81, mlcod=17'78)", 2024-03-27T21:40:11.675 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:11.675 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.538718586000002, 2024-03-27T21:40:11.675 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014990870000000001, 2024-03-27T21:40:11.675 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:11.675 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.675 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:11.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.675 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:11.675 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:11.675 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.676 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:11.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:11.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.676 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:11.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:11.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.676 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:11.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089491+0000", 2024-03-27T21:40:11.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4805e-05 2024-03-27T21:40:11.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.677 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:11.677 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089498+0000", 2024-03-27T21:40:11.677 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0650000000000001e-06 2024-03-27T21:40:11.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.677 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.677 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089501+0000", 2024-03-27T21:40:11.677 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.54e-06 2024-03-27T21:40:11.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.677 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:11.677 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089533+0000", 2024-03-27T21:40:11.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1838999999999998e-05 2024-03-27T21:40:11.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:11.678 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089550+0000", 2024-03-27T21:40:11.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7465999999999998e-05 2024-03-27T21:40:11.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:11.678 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090890+0000", 2024-03-27T21:40:11.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013397979999999999 2024-03-27T21:40:11.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:11.678 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090915+0000", 2024-03-27T21:40:11.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4905e-05 2024-03-27T21:40:11.679 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.679 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:11.679 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.679 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:120 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'82, mlcod=17'78)", 2024-03-27T21:40:11.679 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:11.679 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.537918705999999, 2024-03-27T21:40:11.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001527093, 2024-03-27T21:40:11.679 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:11.679 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.679 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:11.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.679 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:11.680 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:11.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:11.680 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:11.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:11.680 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:11.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:11.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090267+0000", 2024-03-27T21:40:11.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1477000000000002e-05 2024-03-27T21:40:11.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:11.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090273+0000", 2024-03-27T21:40:11.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9519999999999999e-06 2024-03-27T21:40:11.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090276+0000", 2024-03-27T21:40:11.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.219e-06 2024-03-27T21:40:11.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:11.682 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090304+0000", 2024-03-27T21:40:11.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8073999999999999e-05 2024-03-27T21:40:11.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.682 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:11.682 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090316+0000", 2024-03-27T21:40:11.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2088e-05 2024-03-27T21:40:11.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.682 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:11.682 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091717+0000", 2024-03-27T21:40:11.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014003889999999999 2024-03-27T21:40:11.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.683 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:11.683 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091743+0000", 2024-03-27T21:40:11.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6234e-05 2024-03-27T21:40:11.683 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.683 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:11.683 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.683 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:121 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'83, mlcod=17'78)", 2024-03-27T21:40:11.683 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:11.683 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.537108341, 2024-03-27T21:40:11.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010226899999999999, 2024-03-27T21:40:11.683 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:11.683 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.684 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:11.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:11.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:11.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:11.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:11.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:11.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:11.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:11.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091142+0000", 2024-03-27T21:40:11.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011608300000000001 2024-03-27T21:40:11.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:11.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091151+0000", 2024-03-27T21:40:11.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2350000000000005e-06 2024-03-27T21:40:11.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091156+0000", 2024-03-27T21:40:11.686 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5419999999999999e-06 2024-03-27T21:40:11.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.686 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:11.686 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091191+0000", 2024-03-27T21:40:11.686 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5453000000000001e-05 2024-03-27T21:40:11.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.686 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:11.686 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091208+0000", 2024-03-27T21:40:11.686 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7119999999999999e-05 2024-03-27T21:40:11.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.686 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:11.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092024+0000", 2024-03-27T21:40:11.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000815539 2024-03-27T21:40:11.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.687 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:11.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092049+0000", 2024-03-27T21:40:11.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5171e-05 2024-03-27T21:40:11.687 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.687 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:11.687 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.687 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:142 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'100, mlcod=18'98)", 2024-03-27T21:40:11.687 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:11.688 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.122101387000001, 2024-03-27T21:40:11.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001198778, 2024-03-27T21:40:11.688 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:11.688 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.688 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:11.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:11.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:11.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:11.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:11.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:11.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:11.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:11.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506101+0000", 2024-03-27T21:40:11.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8202999999999997e-05 2024-03-27T21:40:11.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:11.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506109+0000", 2024-03-27T21:40:11.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3059999999999998e-06 2024-03-27T21:40:11.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506113+0000", 2024-03-27T21:40:11.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0119999999999999e-06 2024-03-27T21:40:11.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:11.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506137+0000", 2024-03-27T21:40:11.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4043999999999999e-05 2024-03-27T21:40:11.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.693 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:11.693 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506148+0000", 2024-03-27T21:40:11.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0446e-05 2024-03-27T21:40:11.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:11.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507213+0000", 2024-03-27T21:40:11.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001065215 2024-03-27T21:40:11.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:11.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507232+0000", 2024-03-27T21:40:11.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9038e-05 2024-03-27T21:40:11.694 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.694 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:11.694 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.695 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:143 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'101, mlcod=18'98)", 2024-03-27T21:40:11.695 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:11.695 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.121488767999999, 2024-03-27T21:40:11.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001357488, 2024-03-27T21:40:11.695 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:11.695 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.695 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:11.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:11.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:11.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:11.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:11.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:11.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:11.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:11.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506723+0000", 2024-03-27T21:40:11.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7404000000000006e-05 2024-03-27T21:40:11.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.697 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:11.697 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506730+0000", 2024-03-27T21:40:11.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5749999999999997e-06 2024-03-27T21:40:11.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.697 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.697 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506733+0000", 2024-03-27T21:40:11.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7979999999999999e-06 2024-03-27T21:40:11.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.697 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:11.697 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506758+0000", 2024-03-27T21:40:11.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4179e-05 2024-03-27T21:40:11.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:11.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506768+0000", 2024-03-27T21:40:11.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0740000000000001e-05 2024-03-27T21:40:11.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:11.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507978+0000", 2024-03-27T21:40:11.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012093900000000001 2024-03-27T21:40:11.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:11.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508003+0000", 2024-03-27T21:40:11.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5667e-05 2024-03-27T21:40:11.699 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.699 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:11.699 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.699 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:144 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'102, mlcod=18'98)", 2024-03-27T21:40:11.699 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:11.699 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.120829871000002, 2024-03-27T21:40:11.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013453390000000001, 2024-03-27T21:40:11.699 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:11.699 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:11.699 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:11.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:11.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:11.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:11.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:11.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:11.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:11.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507304+0000", 2024-03-27T21:40:11.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:11.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:11.701 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507397+0000", 2024-03-27T21:40:11.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2995e-05 2024-03-27T21:40:11.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.701 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:11.701 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507403+0000", 2024-03-27T21:40:11.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8640000000000001e-06 2024-03-27T21:40:11.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.701 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:11.701 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507407+0000", 2024-03-27T21:40:11.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5710000000000002e-06 2024-03-27T21:40:11.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.701 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:11.702 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507428+0000", 2024-03-27T21:40:11.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1192e-05 2024-03-27T21:40:11.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.702 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:11.702 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507438+0000", 2024-03-27T21:40:11.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8309999999999996e-06 2024-03-27T21:40:11.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.702 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:11.702 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508624+0000", 2024-03-27T21:40:11.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011861210000000001 2024-03-27T21:40:11.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:11.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:11.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:11.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508650+0000", 2024-03-27T21:40:11.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6118e-05 2024-03-27T21:40:11.703 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.703 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:11.703 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.703 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:11.703 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:11.703 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:11.703 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:40:11.705 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:40:11.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.667+0000 7f5c2d7fa640 1 -- 172.21.15.60:0/3138508153 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v29) v1 ==== 91+0+0 (secure 0 0 0) 0x7f5c24057db0 con 0x7f5c300b40b0 2024-03-27T21:40:11.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.667+0000 7f5c2d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:40:11.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.667+0000 7f5c2d7fa640 10 monclient: _finish_command 2 = system:0 noscrub is unset 2024-03-27T21:40:11.707 INFO:teuthology.orchestra.run.smithi060.stderr:noscrub is unset 2024-03-27T21:40:11.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.669+0000 7f5c371cb640 10 librados: watch_flush enter 2024-03-27T21:40:11.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.669+0000 7f5c371cb640 10 librados: watch_flush exit 2024-03-27T21:40:11.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.669+0000 7f5c371cb640 20 client.4548.objecter shutdown clearing up homeless session... 2024-03-27T21:40:11.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.669+0000 7f5c371cb640 10 client.4548.objecter successfully canceled tick 2024-03-27T21:40:11.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.669+0000 7f5c371cb640 1 -- 172.21.15.60:0/3138508153 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5c1406c5c0 msgr2=0x7f5c1406ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:11.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.669+0000 7f5c371cb640 1 --2- 172.21.15.60:0/3138508153 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5c1406c5c0 0x7f5c1406ea80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5c300b3070 tx=0x7f5c1c009040 comp rx=0 tx=0).stop 2024-03-27T21:40:11.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.669+0000 7f5c371cb640 10 monclient: shutdown 2024-03-27T21:40:11.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.669+0000 7f5c371cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:11.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.669+0000 7f5c371cb640 1 -- 172.21.15.60:0/3138508153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c300b40b0 msgr2=0x7f5c3013a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:11.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.669+0000 7f5c371cb640 1 --2- 172.21.15.60:0/3138508153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c300b40b0 0x7f5c3013a9a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5c2400e930 tx=0x7f5c2400edf0 comp rx=0 tx=0).stop 2024-03-27T21:40:11.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.670+0000 7f5c371cb640 1 -- 172.21.15.60:0/3138508153 shutdown_connections 2024-03-27T21:40:11.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.670+0000 7f5c371cb640 1 --2- 172.21.15.60:0/3138508153 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c3013b480 0x7f5c301340d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:11.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.670+0000 7f5c371cb640 1 --2- 172.21.15.60:0/3138508153 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5c1406c5c0 0x7f5c1406ea80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:11.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.670+0000 7f5c371cb640 1 --2- 172.21.15.60:0/3138508153 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c3013b020 0x7f5c3013e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:11.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.670+0000 7f5c371cb640 1 --2- 172.21.15.60:0/3138508153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c300b40b0 0x7f5c3013a9a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:11.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.670+0000 7f5c371cb640 1 -- 172.21.15.60:0/3138508153 >> 172.21.15.60:0/3138508153 conn(0x7f5c300608e0 msgr2=0x7f5c30140560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:11.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.670+0000 7f5c371cb640 1 -- 172.21.15.60:0/3138508153 shutdown_connections 2024-03-27T21:40:11.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.670+0000 7f5c371cb640 1 -- 172.21.15.60:0/3138508153 wait complete. 2024-03-27T21:40:11.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.670+0000 7f5c371cb640 1 librados: shutdown 2024-03-27T21:40:11.709 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:11.718+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:11.806 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:11.820+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:11.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.913+0000 7f7ed268b640 10 monclient: get_monmap_and_config 2024-03-27T21:40:11.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.913+0000 7f7ed268b640 10 monclient: build_initial_monmap 2024-03-27T21:40:11.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.913+0000 7f7ed268b640 10 monclient: monmap: 2024-03-27T21:40:11.900 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:11.900 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:11.900 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:11.914181+0000 2024-03-27T21:40:11.900 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:11.914181+0000 2024-03-27T21:40:11.900 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:11.900 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:11.900 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:11.900 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:11.900 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:11.900 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:11.900 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:11.900 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:11.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.914+0000 7f7ed268b640 1 Processor -- start 2024-03-27T21:40:11.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.914+0000 7f7ed268b640 1 -- start start 2024-03-27T21:40:11.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.914+0000 7f7ed268b640 10 monclient: init 2024-03-27T21:40:11.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed268b640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:11.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed268b640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-03-27T21:40:11.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed268b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7ecc0b40b0 0x7f7ecc0a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:11.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed268b640 10 monclient(hunting): picked mon.noname-c con 0x7f7ecc0b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:11.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed268b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7ecc13b020 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:11.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed268b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7ecc13b480 0x7f7ecc141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:11.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed268b640 10 monclient(hunting): picked mon.noname-a con 0x7f7ecc13b480 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:11.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed268b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ecc13b8d0 con 0x7f7ecc13b020 2024-03-27T21:40:11.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed268b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:11.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed268b640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ecc0b4490 con 0x7f7ecc13b480 2024-03-27T21:40:11.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed268b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:11.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed268b640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ecc142340 con 0x7f7ecc0b40b0 2024-03-27T21:40:11.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed268b640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:11.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed268b640 10 monclient(hunting): authenticate will time out at 853.136262s 2024-03-27T21:40:11.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ecb7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7ecc13b020 0x7f7ecc1416f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39582/0 (socket says 172.21.15.60:39582) 2024-03-27T21:40:11.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed0c28640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7ecc13b480 0x7f7ecc141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:11.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ecb7fe640 1 -- 172.21.15.60:0/986387721 learned_addr learned my addr 172.21.15.60:0/986387721 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:40:11.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed0c28640 10 monclient(hunting): get_auth_request con 0x7f7ecc13b480 auth_method 0 2024-03-27T21:40:11.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed0c28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:11.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed0c28640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:11.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed0c28640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:11.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ecbfff640 1 --2- 172.21.15.60:0/986387721 >> v2:172.21.15.191:3300/0 conn(0x7f7ecc0b40b0 0x7f7ecc0a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:11.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ecbfff640 10 monclient(hunting): get_auth_request con 0x7f7ecc0b40b0 auth_method 0 2024-03-27T21:40:11.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ecbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:11.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ecbfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:11.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ecbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:11.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed0c28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:11.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed0c28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:11.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed0c28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:11.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ecbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:11.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ecbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:11.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ecbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:11.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ecbfff640 10 monclient(hunting): handle_auth_done global_id 4531 payload 274 2024-03-27T21:40:11.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ecbfff640 1 -- 172.21.15.60:0/986387721 >> v2:172.21.15.60:3300/0 conn(0x7f7ecc13b480 msgr2=0x7f7ecc141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ecbfff640 1 --2- 172.21.15.60:0/986387721 >> v2:172.21.15.60:3300/0 conn(0x7f7ecc13b480 0x7f7ecc141e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ecbfff640 1 -- 172.21.15.60:0/986387721 >> v1:172.21.15.60:6789/0 conn(0x7f7ecc13b020 legacy=0x7f7ecc1416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ecbfff640 10 monclient: _finish_hunting 0 2024-03-27T21:40:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ecbfff640 1 monclient: found mon.noname-c 2024-03-27T21:40:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ecaffd640 1 -- 172.21.15.60:0/986387721 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462844635 0 0) 0x7f7ecc13b8d0 con 0x7f7ecc13b020 2024-03-27T21:40:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ecbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ecbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:40:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ecbfff640 1 -- 172.21.15.60:0/986387721 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ecc0b3830 con 0x7f7ecc0b40b0 2024-03-27T21:40:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ecaffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:40:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.915+0000 7f7ed0c28640 1 --2- 172.21.15.60:0/986387721 >> v2:172.21.15.60:3300/0 conn(0x7f7ecc13b480 0x7f7ecc141e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.917+0000 7f7ecbfff640 1 --2- 172.21.15.60:0/986387721 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ecc0b40b0 0x7f7ecc0a8b90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc0b3a20 tx=0x7f7ebc034060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74fb3df6615a642 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.917+0000 7f7ecaffd640 1 -- 172.21.15.60:0/986387721 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc034e10 con 0x7f7ecc0b40b0 2024-03-27T21:40:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.917+0000 7f7ecaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.917+0000 7f7ecaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.917+0000 7f7ecaffd640 10 monclient: dump: 2024-03-27T21:40:11.910 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:11.910 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:11.910 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:11.910 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:11.910 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:11.910 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:11.910 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:11.910 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:11.910 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:11.910 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:11.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.917+0000 7f7ecaffd640 10 monclient: _finish_auth 0 2024-03-27T21:40:11.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.917+0000 7f7ecaffd640 10 monclient: _check_auth_tickets 2024-03-27T21:40:11.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.917+0000 7f7ecaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:11.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.917+0000 7f7ed268b640 5 monclient: authenticate success, global_id 4531 2024-03-27T21:40:11.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.917+0000 7f7ecaffd640 1 -- 172.21.15.60:0/986387721 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ebc03c440 con 0x7f7ecc0b40b0 2024-03-27T21:40:11.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.917+0000 7f7ed268b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:11.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.917+0000 7f7ecaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:11.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.917+0000 7f7ed268b640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:11.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.917+0000 7f7ed268b640 4 set_mon_vals no callback set 2024-03-27T21:40:11.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.917+0000 7f7ecaffd640 1 -- 172.21.15.60:0/986387721 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc03c7d0 con 0x7f7ecc0b40b0 2024-03-27T21:40:11.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.917+0000 7f7ecaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:11.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.917+0000 7f7ecaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:40:11.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.917+0000 7f7ecaffd640 10 monclient: dump: 2024-03-27T21:40:11.911 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:11.911 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:11.911 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:11.911 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:11.911 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:11.911 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:11.912 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:11.912 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:11.912 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:11.912 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:11.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.918+0000 7f7ed268b640 10 monclient: shutdown 2024-03-27T21:40:11.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.918+0000 7f7ed268b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:11.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.918+0000 7f7ed268b640 1 -- 172.21.15.60:0/986387721 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ecc0b40b0 msgr2=0x7f7ecc0a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:11.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.918+0000 7f7ed268b640 1 --2- 172.21.15.60:0/986387721 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ecc0b40b0 0x7f7ecc0a8b90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc0b3a20 tx=0x7f7ebc034060 comp rx=0 tx=0).stop 2024-03-27T21:40:11.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.918+0000 7f7ed268b640 1 -- 172.21.15.60:0/986387721 shutdown_connections 2024-03-27T21:40:11.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.918+0000 7f7ed268b640 1 --2- 172.21.15.60:0/986387721 >> v2:172.21.15.60:3300/0 conn(0x7f7ecc13b480 0x7f7ecc141e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:11.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.918+0000 7f7ed268b640 1 --2- 172.21.15.60:0/986387721 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ecc0b40b0 0x7f7ecc0a8b90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:11.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.918+0000 7f7ed268b640 1 -- 172.21.15.60:0/986387721 >> 172.21.15.60:0/986387721 conn(0x7f7ecc0608e0 msgr2=0x7f7ecc060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:11.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.918+0000 7f7ed268b640 1 -- 172.21.15.60:0/986387721 shutdown_connections 2024-03-27T21:40:11.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.918+0000 7f7ed268b640 1 -- 172.21.15.60:0/986387721 wait complete. 2024-03-27T21:40:11.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.919+0000 7f7ed268b640 10 monclient: build_initial_monmap 2024-03-27T21:40:11.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.919+0000 7f7ed268b640 10 monclient: monmap: 2024-03-27T21:40:11.913 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:11.913 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:11.913 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:11.913 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:11.913 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:11.913 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:11.913 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:11.913 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:11.913 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:11.913 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:11.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.919+0000 7f7ed268b640 1 librados: starting msgr at 2024-03-27T21:40:11.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.919+0000 7f7ed268b640 1 librados: starting objecter 2024-03-27T21:40:11.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.919+0000 7f7ed268b640 1 Processor -- start 2024-03-27T21:40:11.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.920+0000 7f7ed268b640 1 -- start start 2024-03-27T21:40:11.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.920+0000 7f7ed268b640 1 librados: setting wanted keys 2024-03-27T21:40:11.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.920+0000 7f7ed268b640 1 librados: calling monclient init 2024-03-27T21:40:11.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.920+0000 7f7ed268b640 10 monclient: init 2024-03-27T21:40:11.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.920+0000 7f7ed268b640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:11.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed268b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:40:11.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed268b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ecc0b40b0 0x7f7ecc0aac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:11.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed268b640 10 monclient(hunting): picked mon.noname-a con 0x7f7ecc0b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:11.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ecbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ecc0b40b0 0x7f7ecc0aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:11.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ecbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ecc0b40b0 0x7f7ecc0aac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35890/0 (socket says 172.21.15.60:35890) 2024-03-27T21:40:11.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ecbfff640 1 -- 172.21.15.60:0/2595194160 learned_addr learned my addr 172.21.15.60:0/2595194160 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:11.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed268b640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ecc13b020 0x7f7ecc1d2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:11.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed268b640 10 monclient(hunting): picked mon.noname-b con 0x7f7ecc13b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:11.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed268b640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7ecc13b480 0x7f7ecc139120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:11.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed268b640 10 monclient(hunting): picked mon.noname-c con 0x7f7ecc13b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:11.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed268b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:11.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed268b640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc1d1c80 con 0x7f7ecc0b40b0 2024-03-27T21:40:11.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed268b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:11.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed268b640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc1d1e90 con 0x7f7ecc13b020 2024-03-27T21:40:11.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ecb7fe640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ecc13b020 0x7f7ecc1d2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:11.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed268b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:11.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed268b640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc13e6d0 con 0x7f7ecc13b480 2024-03-27T21:40:11.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed268b640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:11.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed0c28640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7ecc13b480 0x7f7ecc139120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:11.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed268b640 10 monclient(hunting): authenticate will time out at 853.142610s 2024-03-27T21:40:11.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ecbfff640 10 monclient(hunting): get_auth_request con 0x7f7ecc0b40b0 auth_method 0 2024-03-27T21:40:11.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ecbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:11.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ecbfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:11.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ecbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:11.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ecb7fe640 10 monclient(hunting): get_auth_request con 0x7f7ecc13b020 auth_method 0 2024-03-27T21:40:11.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ecb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:11.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ecb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:11.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ecb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:11.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed0c28640 10 monclient(hunting): get_auth_request con 0x7f7ecc13b480 auth_method 0 2024-03-27T21:40:11.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed0c28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:11.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed0c28640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:11.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed0c28640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:11.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ecbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:11.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ecbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:11.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ecbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:11.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed0c28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:11.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed0c28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:11.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.921+0000 7f7ed0c28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:11.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ecb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:11.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ecb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:11.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ecb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:11.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ed0c28640 10 monclient(hunting): handle_auth_done global_id 4469 payload 824 2024-03-27T21:40:11.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ed0c28640 1 -- 172.21.15.60:0/2595194160 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ecc13b020 msgr2=0x7f7ecc1d2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:11.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ed0c28640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ecc13b020 0x7f7ecc1d2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:11.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ed0c28640 1 -- 172.21.15.60:0/2595194160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ecc0b40b0 msgr2=0x7f7ecc0aac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:11.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ed0c28640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ecc0b40b0 0x7f7ecc0aac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:11.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ed0c28640 10 monclient: _finish_hunting 0 2024-03-27T21:40:11.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ed0c28640 1 monclient: found mon.noname-c 2024-03-27T21:40:11.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ed0c28640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:11.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ed0c28640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:40:11.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ed0c28640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec4009a70 con 0x7f7ecc13b480 2024-03-27T21:40:11.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ecbfff640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ecc0b40b0 0x7f7ecc0aac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:11.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ed0c28640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7ecc13b480 0x7f7ecc139120 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7ec400efa0 tx=0x7f7ec400c550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:11.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ec8ff9640 10 client.?.objecter ms_handle_connect 0x7f7ecc13b480 2024-03-27T21:40:11.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ec8ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:11.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ec8ff9640 1 -- 172.21.15.60:0/2595194160 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec400ccd0 con 0x7f7ecc13b480 2024-03-27T21:40:11.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ec8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:11.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ec8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:11.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ec8ff9640 10 monclient: dump: 2024-03-27T21:40:11.925 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:11.925 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:11.925 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:11.925 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:11.925 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:11.925 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:11.925 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:11.926 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:11.926 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:11.926 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:11.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ec8ff9640 10 monclient: _finish_auth 0 2024-03-27T21:40:11.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ec8ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:40:11.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ec8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:11.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ec8ff9640 1 -- 172.21.15.60:0/2595194160 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec400f040 con 0x7f7ecc13b480 2024-03-27T21:40:11.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ec8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:11.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ec8ff9640 1 -- 172.21.15.60:0/2595194160 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4003bd0 con 0x7f7ecc13b480 2024-03-27T21:40:11.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ec8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:11.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ec8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:11.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7ec8ff9640 10 monclient: dump: 2024-03-27T21:40:11.926 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:11.926 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:11.927 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.922+0000 7f7eca7fc640 4 set_mon_vals no callback set 2024-03-27T21:40:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.923+0000 7f7ed268b640 5 monclient: authenticate success, global_id 4469 2024-03-27T21:40:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.923+0000 7f7ed268b640 10 monclient: _renew_subs 2024-03-27T21:40:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.923+0000 7f7ed268b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.923+0000 7f7ed268b640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ebc013a10 con 0x7f7ecc13b480 2024-03-27T21:40:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.923+0000 7f7ed268b640 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.923+0000 7f7ed268b640 10 monclient: _renew_subs 2024-03-27T21:40:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.923+0000 7f7ed268b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.923+0000 7f7ed268b640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ecc0ab360 con 0x7f7ecc13b480 2024-03-27T21:40:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.923+0000 7f7ed268b640 1 librados: init done 2024-03-27T21:40:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.925+0000 7f7ed268b640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.925+0000 7f7ed268b640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.925+0000 7f7ed268b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.925+0000 7f7ed268b640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f7ecc069470 con 0x7f7ecc13b480 2024-03-27T21:40:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.925+0000 7f7ec8ff9640 1 -- 172.21.15.60:0/2595194160 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7ec4014480 con 0x7f7ecc13b480 2024-03-27T21:40:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.926+0000 7f7ec8ff9640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f7eb806c5c0 0x7f7eb806ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.926+0000 7f7ec8ff9640 1 -- 172.21.15.60:0/2595194160 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f7ec40898b0 con 0x7f7ecc13b480 2024-03-27T21:40:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.926+0000 7f7ec8ff9640 10 client.4469.objecter ms_dispatch 0x7f7ecc000b90 osd_map(29..29 src has 1..29) v4 2024-03-27T21:40:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.926+0000 7f7ec8ff9640 3 client.4469.objecter handle_osd_map got epochs [29,29] > 0 2024-03-27T21:40:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.926+0000 7f7ec8ff9640 3 client.4469.objecter handle_osd_map decoding full epoch 29 2024-03-27T21:40:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.926+0000 7f7ec8ff9640 20 client.4469.objecter dump_active .. 0 homeless 2024-03-27T21:40:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.926+0000 7f7ec8ff9640 1 -- 172.21.15.60:0/2595194160 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v29) v1 ==== 54+0+16 (secure 0 0 0) 0x7f7ec408e240 con 0x7f7ecc13b480 2024-03-27T21:40:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.926+0000 7f7ec8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.926+0000 7f7ec8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.926+0000 7f7ecbfff640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f7eb806c5c0 0x7f7eb806ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.926+0000 7f7ecbfff640 10 monclient: get_auth_request con 0x7f7eb806c5c0 auth_method 0 2024-03-27T21:40:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.926+0000 7f7ecbfff640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f7eb806c5c0 0x7f7eb806ea80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc0091d0 tx=0x7f7ebc038000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.926+0000 7f7ec8ff9640 10 client.4469.objecter ms_handle_connect 0x7f7eb806c5c0 2024-03-27T21:40:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.927+0000 7f7eb27fc640 10 client.4469.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.927+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign -1 1 2024-03-27T21:40:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.927+0000 7f7eb27fc640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f7ecc1d28c0 0x7f7ecc1d5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.927+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc0a90d0 osd=0 3 2024-03-27T21:40:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.927+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc0a90d0 osd=0 3 2024-03-27T21:40:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.927+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc0a90d0 osd=0 3 2024-03-27T21:40:11.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.927+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_remove -1 1 2024-03-27T21:40:11.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.927+0000 7f7eb27fc640 20 client.4469.objecter get_session s=0x7f7ecc0a90d0 osd=0 3 2024-03-27T21:40:11.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.927+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign 0 1 2024-03-27T21:40:11.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.927+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc0a90d0 osd=0 4 2024-03-27T21:40:11.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.927+0000 7f7eb27fc640 10 client.4469.objecter _send_command 1 2024-03-27T21:40:11.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.927+0000 7f7eb27fc640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ecc0ab360 con 0x7f7ecc1d28c0 2024-03-27T21:40:11.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.927+0000 7f7ecb7fe640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f7ecc1d28c0 0x7f7ecc1d5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:11.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.927+0000 7f7ecb7fe640 10 monclient: get_auth_request con 0x7f7ecc1d28c0 auth_method 0 2024-03-27T21:40:11.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.927+0000 7f7ec8ff9640 10 client.4469.objecter ms_handle_connect 0x7f7ecc1d28c0 2024-03-27T21:40:11.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.927+0000 7f7ecb7fe640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f7ecc1d28c0 0x7f7ecc1d5940 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:11.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.928+0000 7f7ec8ff9640 1 -- 172.21.15.60:0/2595194160 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7ecc0ab360 con 0x7f7ecc1d28c0 2024-03-27T21:40:11.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.928+0000 7f7ec8ff9640 10 client.4469.objecter ms_dispatch 0x7f7ecc000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:11.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.928+0000 7f7ec8ff9640 10 client.4469.objecter _finish_command 1 = system:0 2024-03-27T21:40:11.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.928+0000 7f7ec8ff9640 20 client.4469.objecter put_session s=0x7f7ecc0a90d0 osd=0 4 2024-03-27T21:40:11.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.928+0000 7f7ec8ff9640 15 client.4469.objecter _session_command_op_remove 0 1 2024-03-27T21:40:11.931 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:11.931 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:11.931 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:11.929+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:11.934 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:40:11.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.958+0000 7f7eb27fc640 10 client.4469.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:11.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.958+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign -1 2 2024-03-27T21:40:11.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.958+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc0a90d0 osd=0 3 2024-03-27T21:40:11.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.958+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc0a90d0 osd=0 3 2024-03-27T21:40:11.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.958+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc0a90d0 osd=0 3 2024-03-27T21:40:11.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.958+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_remove -1 2 2024-03-27T21:40:11.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.958+0000 7f7eb27fc640 20 client.4469.objecter get_session s=0x7f7ecc0a90d0 osd=0 3 2024-03-27T21:40:11.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.958+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign 0 2 2024-03-27T21:40:11.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.958+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc0a90d0 osd=0 4 2024-03-27T21:40:11.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.958+0000 7f7eb27fc640 10 client.4469.objecter _send_command 2 2024-03-27T21:40:11.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.958+0000 7f7eb27fc640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7ecc1d2050 con 0x7f7ecc1d28c0 2024-03-27T21:40:11.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.961+0000 7f7ec8ff9640 1 -- 172.21.15.60:0/2595194160 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7ecc1d2050 con 0x7f7ecc1d28c0 2024-03-27T21:40:11.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.961+0000 7f7ec8ff9640 10 client.4469.objecter ms_dispatch 0x7f7ecc000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:11.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.961+0000 7f7ec8ff9640 10 client.4469.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:11.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.961+0000 7f7ec8ff9640 20 client.4469.objecter put_session s=0x7f7ecc0a90d0 osd=0 4 2024-03-27T21:40:11.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.961+0000 7f7ec8ff9640 15 client.4469.objecter _session_command_op_remove 0 2 2024-03-27T21:40:11.948 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:11.949 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:11.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.961+0000 7f7eb27fc640 10 client.4469.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:11.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.961+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign -1 3 2024-03-27T21:40:11.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.961+0000 7f7eb27fc640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f7ecc13bd20 0x7f7ecc065fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:11.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.961+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc13e810 osd=1 3 2024-03-27T21:40:11.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.961+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc13e810 osd=1 3 2024-03-27T21:40:11.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.961+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc13e810 osd=1 3 2024-03-27T21:40:11.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.961+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_remove -1 3 2024-03-27T21:40:11.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.961+0000 7f7eb27fc640 20 client.4469.objecter get_session s=0x7f7ecc13e810 osd=1 3 2024-03-27T21:40:11.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.961+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign 1 3 2024-03-27T21:40:11.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.961+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc13e810 osd=1 4 2024-03-27T21:40:11.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.961+0000 7f7eb27fc640 10 client.4469.objecter _send_command 3 2024-03-27T21:40:11.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.961+0000 7f7eb27fc640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ecc06bef0 con 0x7f7ecc13bd20 2024-03-27T21:40:11.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.962+0000 7f7ecbfff640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f7ecc13bd20 0x7f7ecc065fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:11.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.962+0000 7f7ecbfff640 10 monclient: get_auth_request con 0x7f7ecc13bd20 auth_method 0 2024-03-27T21:40:11.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.962+0000 7f7ecbfff640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f7ecc13bd20 0x7f7ecc065fa0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:11.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.962+0000 7f7ec8ff9640 10 client.4469.objecter ms_handle_connect 0x7f7ecc13bd20 2024-03-27T21:40:11.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.963+0000 7f7ec8ff9640 1 -- 172.21.15.60:0/2595194160 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7ecc06bef0 con 0x7f7ecc13bd20 2024-03-27T21:40:11.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.963+0000 7f7ec8ff9640 10 client.4469.objecter ms_dispatch 0x7f7ecc000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:11.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.963+0000 7f7ec8ff9640 10 client.4469.objecter _finish_command 3 = system:0 2024-03-27T21:40:11.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.963+0000 7f7ec8ff9640 20 client.4469.objecter put_session s=0x7f7ecc13e810 osd=1 4 2024-03-27T21:40:11.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.963+0000 7f7ec8ff9640 15 client.4469.objecter _session_command_op_remove 1 3 2024-03-27T21:40:11.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.993+0000 7f7eb27fc640 10 client.4469.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:11.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.993+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign -1 4 2024-03-27T21:40:11.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.993+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc13e810 osd=1 3 2024-03-27T21:40:11.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.993+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc13e810 osd=1 3 2024-03-27T21:40:11.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.993+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc13e810 osd=1 3 2024-03-27T21:40:11.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.993+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_remove -1 4 2024-03-27T21:40:11.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.993+0000 7f7eb27fc640 20 client.4469.objecter get_session s=0x7f7ecc13e810 osd=1 3 2024-03-27T21:40:11.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.993+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign 1 4 2024-03-27T21:40:11.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.993+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc13e810 osd=1 4 2024-03-27T21:40:11.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.993+0000 7f7eb27fc640 10 client.4469.objecter _send_command 4 2024-03-27T21:40:11.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.993+0000 7f7eb27fc640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7ecc06c060 con 0x7f7ecc13bd20 2024-03-27T21:40:11.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.996+0000 7f7ec8ff9640 1 -- 172.21.15.60:0/2595194160 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7ecc06c060 con 0x7f7ecc13bd20 2024-03-27T21:40:11.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.996+0000 7f7ec8ff9640 10 client.4469.objecter ms_dispatch 0x7f7ecc000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:11.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.996+0000 7f7ec8ff9640 10 client.4469.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:11.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.996+0000 7f7ec8ff9640 20 client.4469.objecter put_session s=0x7f7ecc13e810 osd=1 4 2024-03-27T21:40:11.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.996+0000 7f7ec8ff9640 15 client.4469.objecter _session_command_op_remove 1 4 2024-03-27T21:40:11.983 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:11.985 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:11.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.996+0000 7f7eb27fc640 10 client.4469.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:11.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.996+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign -1 5 2024-03-27T21:40:11.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.996+0000 7f7eb27fc640 20 client.4469.objecter _get_session osd=-1 returning homeless 2024-03-27T21:40:11.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.996+0000 7f7eb27fc640 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:11.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.996+0000 7f7eb27fc640 10 monclient: _renew_subs 2024-03-27T21:40:11.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.996+0000 7f7eb27fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:11.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.996+0000 7f7eb27fc640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=30}) v3 -- 0x7f7ecc0664e0 con 0x7f7ecc13b480 2024-03-27T21:40:11.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.996+0000 7f7eb27fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:11.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.996+0000 7f7eb27fc640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7ecc0667c0 con 0x7f7ecc13b480 2024-03-27T21:40:11.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.997+0000 7f7ec8ff9640 1 -- 172.21.15.60:0/2595194160 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ec4054da0 con 0x7f7ecc13b480 2024-03-27T21:40:11.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.997+0000 7f7ec8ff9640 10 monclient: handle_get_version_reply finishing 1 version 29 2024-03-27T21:40:11.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.997+0000 7f7eca7fc640 10 client.4469.objecter _check_command_map_dne tid 5 current 29 map_dne_bound 29 2024-03-27T21:40:11.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.997+0000 7f7eca7fc640 10 client.4469.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:40:11.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.997+0000 7f7eca7fc640 15 client.4469.objecter _session_command_op_remove -1 5 2024-03-27T21:40:11.986 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:40:11.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.997+0000 7f7eb27fc640 10 client.4469.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:11.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.997+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign -1 6 2024-03-27T21:40:11.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.997+0000 7f7eb27fc640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f7ecc0b2110 0x7f7ecc0b24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:11.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.997+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc066a00 osd=3 3 2024-03-27T21:40:11.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.997+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc066a00 osd=3 3 2024-03-27T21:40:11.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.997+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc066a00 osd=3 3 2024-03-27T21:40:11.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.997+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_remove -1 6 2024-03-27T21:40:11.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.997+0000 7f7eb27fc640 20 client.4469.objecter get_session s=0x7f7ecc066a00 osd=3 3 2024-03-27T21:40:11.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.997+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign 3 6 2024-03-27T21:40:11.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.997+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc066a00 osd=3 4 2024-03-27T21:40:11.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.997+0000 7f7eb27fc640 10 client.4469.objecter _send_command 6 2024-03-27T21:40:11.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.997+0000 7f7eb27fc640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ecc1e5ef0 con 0x7f7ecc0b2110 2024-03-27T21:40:11.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.997+0000 7f7ecb7fe640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f7ecc0b2110 0x7f7ecc0b24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:11.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.997+0000 7f7ecb7fe640 10 monclient: get_auth_request con 0x7f7ecc0b2110 auth_method 0 2024-03-27T21:40:11.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.998+0000 7f7ec8ff9640 10 client.4469.objecter ms_handle_connect 0x7f7ecc0b2110 2024-03-27T21:40:11.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.998+0000 7f7ecb7fe640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f7ecc0b2110 0x7f7ecc0b24f0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:11.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.998+0000 7f7ec8ff9640 1 -- 172.21.15.60:0/2595194160 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7ecc1e5ef0 con 0x7f7ecc0b2110 2024-03-27T21:40:11.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.999+0000 7f7ec8ff9640 10 client.4469.objecter ms_dispatch 0x7f7ecc000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:40:11.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.999+0000 7f7ec8ff9640 10 client.4469.objecter _finish_command 6 = system:0 2024-03-27T21:40:11.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.999+0000 7f7ec8ff9640 20 client.4469.objecter put_session s=0x7f7ecc066a00 osd=3 4 2024-03-27T21:40:11.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:11.999+0000 7f7ec8ff9640 15 client.4469.objecter _session_command_op_remove 3 6 2024-03-27T21:40:12.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.026+0000 7f7eb27fc640 10 client.4469.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:12.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.026+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign -1 7 2024-03-27T21:40:12.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.026+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc066a00 osd=3 3 2024-03-27T21:40:12.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.026+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc066a00 osd=3 3 2024-03-27T21:40:12.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.026+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc066a00 osd=3 3 2024-03-27T21:40:12.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.026+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_remove -1 7 2024-03-27T21:40:12.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.026+0000 7f7eb27fc640 20 client.4469.objecter get_session s=0x7f7ecc066a00 osd=3 3 2024-03-27T21:40:12.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.026+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign 3 7 2024-03-27T21:40:12.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.026+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc066a00 osd=3 4 2024-03-27T21:40:12.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.026+0000 7f7eb27fc640 10 client.4469.objecter _send_command 7 2024-03-27T21:40:12.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.026+0000 7f7eb27fc640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7ecc0b1b90 con 0x7f7ecc0b2110 2024-03-27T21:40:12.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.029+0000 7f7ec8ff9640 1 -- 172.21.15.60:0/2595194160 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7ecc0b1b90 con 0x7f7ecc0b2110 2024-03-27T21:40:12.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.029+0000 7f7ec8ff9640 10 client.4469.objecter ms_dispatch 0x7f7ecc000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:12.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.029+0000 7f7ec8ff9640 10 client.4469.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:12.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.029+0000 7f7ec8ff9640 20 client.4469.objecter put_session s=0x7f7ecc066a00 osd=3 4 2024-03-27T21:40:12.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.029+0000 7f7ec8ff9640 15 client.4469.objecter _session_command_op_remove 3 7 2024-03-27T21:40:12.016 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:12.017 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:12.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.029+0000 7f7eb27fc640 10 client.4469.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:12.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.029+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign -1 8 2024-03-27T21:40:12.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.029+0000 7f7eb27fc640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f7ecc1e11a0 0x7f7ecc1e47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:12.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.030+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc0b1f40 osd=4 3 2024-03-27T21:40:12.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.030+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc0b1f40 osd=4 3 2024-03-27T21:40:12.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.030+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc0b1f40 osd=4 3 2024-03-27T21:40:12.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.030+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_remove -1 8 2024-03-27T21:40:12.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.030+0000 7f7eb27fc640 20 client.4469.objecter get_session s=0x7f7ecc0b1f40 osd=4 3 2024-03-27T21:40:12.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.030+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign 4 8 2024-03-27T21:40:12.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.030+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc0b1f40 osd=4 4 2024-03-27T21:40:12.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.030+0000 7f7eb27fc640 10 client.4469.objecter _send_command 8 2024-03-27T21:40:12.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.030+0000 7f7eb27fc640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ecc1e9780 con 0x7f7ecc1e11a0 2024-03-27T21:40:12.018 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:12.030+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:12.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.030+0000 7f7ed0c28640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f7ecc1e11a0 0x7f7ecc1e47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:12.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.030+0000 7f7ed0c28640 10 monclient: get_auth_request con 0x7f7ecc1e11a0 auth_method 0 2024-03-27T21:40:12.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.030+0000 7f7ed0c28640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f7ecc1e11a0 0x7f7ecc1e47e0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:12.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.030+0000 7f7ec8ff9640 10 client.4469.objecter ms_handle_connect 0x7f7ecc1e11a0 2024-03-27T21:40:12.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.031+0000 7f7ec8ff9640 1 -- 172.21.15.60:0/2595194160 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7ecc1e9780 con 0x7f7ecc1e11a0 2024-03-27T21:40:12.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.031+0000 7f7ec8ff9640 10 client.4469.objecter ms_dispatch 0x7f7ecc000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:40:12.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.031+0000 7f7ec8ff9640 10 client.4469.objecter _finish_command 8 = system:0 2024-03-27T21:40:12.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.031+0000 7f7ec8ff9640 20 client.4469.objecter put_session s=0x7f7ecc0b1f40 osd=4 4 2024-03-27T21:40:12.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.031+0000 7f7ec8ff9640 15 client.4469.objecter _session_command_op_remove 4 8 2024-03-27T21:40:12.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.059+0000 7f7eb27fc640 10 client.4469.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:12.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.059+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign -1 9 2024-03-27T21:40:12.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.059+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc0b1f40 osd=4 3 2024-03-27T21:40:12.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.059+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc0b1f40 osd=4 3 2024-03-27T21:40:12.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.059+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc0b1f40 osd=4 3 2024-03-27T21:40:12.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.059+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_remove -1 9 2024-03-27T21:40:12.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.059+0000 7f7eb27fc640 20 client.4469.objecter get_session s=0x7f7ecc0b1f40 osd=4 3 2024-03-27T21:40:12.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.059+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign 4 9 2024-03-27T21:40:12.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.059+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc0b1f40 osd=4 4 2024-03-27T21:40:12.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.059+0000 7f7eb27fc640 10 client.4469.objecter _send_command 9 2024-03-27T21:40:12.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.059+0000 7f7eb27fc640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7ecc1e0710 con 0x7f7ecc1e11a0 2024-03-27T21:40:12.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.061+0000 7f7ec8ff9640 1 -- 172.21.15.60:0/2595194160 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7ecc1e0710 con 0x7f7ecc1e11a0 2024-03-27T21:40:12.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.061+0000 7f7ec8ff9640 10 client.4469.objecter ms_dispatch 0x7f7ecc000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:12.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.061+0000 7f7ec8ff9640 10 client.4469.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:12.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.061+0000 7f7ec8ff9640 20 client.4469.objecter put_session s=0x7f7ecc0b1f40 osd=4 4 2024-03-27T21:40:12.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.061+0000 7f7ec8ff9640 15 client.4469.objecter _session_command_op_remove 4 9 2024-03-27T21:40:12.049 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:12.050 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:12.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.061+0000 7f7eb27fc640 10 client.4469.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:12.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.061+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign -1 10 2024-03-27T21:40:12.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.061+0000 7f7eb27fc640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f7ecc1e0880 0x7f7ecc1e8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:12.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.062+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc1e0710 osd=5 3 2024-03-27T21:40:12.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.062+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc1e0710 osd=5 3 2024-03-27T21:40:12.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.062+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc1e0710 osd=5 3 2024-03-27T21:40:12.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.062+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_remove -1 10 2024-03-27T21:40:12.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.062+0000 7f7eb27fc640 20 client.4469.objecter get_session s=0x7f7ecc1e0710 osd=5 3 2024-03-27T21:40:12.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.062+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign 5 10 2024-03-27T21:40:12.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.062+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc1e0710 osd=5 4 2024-03-27T21:40:12.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.062+0000 7f7eb27fc640 10 client.4469.objecter _send_command 10 2024-03-27T21:40:12.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.062+0000 7f7eb27fc640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ecc1e4330 con 0x7f7ecc1e0880 2024-03-27T21:40:12.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.062+0000 7f7ecbfff640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f7ecc1e0880 0x7f7ecc1e8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:12.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.062+0000 7f7ecbfff640 10 monclient: get_auth_request con 0x7f7ecc1e0880 auth_method 0 2024-03-27T21:40:12.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.062+0000 7f7ec8ff9640 10 client.4469.objecter ms_handle_connect 0x7f7ecc1e0880 2024-03-27T21:40:12.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.062+0000 7f7ecbfff640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f7ecc1e0880 0x7f7ecc1e8070 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:12.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.063+0000 7f7ec8ff9640 1 -- 172.21.15.60:0/2595194160 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7ecc1e4330 con 0x7f7ecc1e0880 2024-03-27T21:40:12.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.063+0000 7f7ec8ff9640 10 client.4469.objecter ms_dispatch 0x7f7ecc000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:40:12.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.063+0000 7f7ec8ff9640 10 client.4469.objecter _finish_command 10 = system:0 2024-03-27T21:40:12.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.063+0000 7f7ec8ff9640 20 client.4469.objecter put_session s=0x7f7ecc1e0710 osd=5 4 2024-03-27T21:40:12.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.063+0000 7f7ec8ff9640 15 client.4469.objecter _session_command_op_remove 5 10 2024-03-27T21:40:12.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.090+0000 7f7eb27fc640 10 client.4469.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:12.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.090+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign -1 11 2024-03-27T21:40:12.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.090+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc1e0710 osd=5 3 2024-03-27T21:40:12.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.090+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc1e0710 osd=5 3 2024-03-27T21:40:12.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.090+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc1e0710 osd=5 3 2024-03-27T21:40:12.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.090+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_remove -1 11 2024-03-27T21:40:12.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.090+0000 7f7eb27fc640 20 client.4469.objecter get_session s=0x7f7ecc1e0710 osd=5 3 2024-03-27T21:40:12.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.090+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign 5 11 2024-03-27T21:40:12.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.090+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc1e0710 osd=5 4 2024-03-27T21:40:12.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.090+0000 7f7eb27fc640 10 client.4469.objecter _send_command 11 2024-03-27T21:40:12.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.090+0000 7f7eb27fc640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7ecc1e4330 con 0x7f7ecc1e0880 2024-03-27T21:40:12.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.092+0000 7f7ec8ff9640 1 -- 172.21.15.60:0/2595194160 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7ecc1e4330 con 0x7f7ecc1e0880 2024-03-27T21:40:12.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.092+0000 7f7ec8ff9640 10 client.4469.objecter ms_dispatch 0x7f7ecc000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:12.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.092+0000 7f7ec8ff9640 10 client.4469.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:12.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.092+0000 7f7ec8ff9640 20 client.4469.objecter put_session s=0x7f7ecc1e0710 osd=5 4 2024-03-27T21:40:12.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.092+0000 7f7ec8ff9640 15 client.4469.objecter _session_command_op_remove 5 11 2024-03-27T21:40:12.079 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:12.080 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:12.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.092+0000 7f7eb27fc640 10 client.4469.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:12.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.093+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign -1 12 2024-03-27T21:40:12.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.093+0000 7f7eb27fc640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f7ecc1e8ac0 0x7f7ecc1e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:12.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.093+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc1e4330 osd=6 3 2024-03-27T21:40:12.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.093+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc1e4330 osd=6 3 2024-03-27T21:40:12.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.093+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc1e4330 osd=6 3 2024-03-27T21:40:12.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.093+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_remove -1 12 2024-03-27T21:40:12.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.093+0000 7f7eb27fc640 20 client.4469.objecter get_session s=0x7f7ecc1e4330 osd=6 3 2024-03-27T21:40:12.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.093+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign 6 12 2024-03-27T21:40:12.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.093+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc1e4330 osd=6 4 2024-03-27T21:40:12.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.093+0000 7f7eb27fc640 10 client.4469.objecter _send_command 12 2024-03-27T21:40:12.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.093+0000 7f7eb27fc640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ecc1e9480 con 0x7f7ecc1e8ac0 2024-03-27T21:40:12.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.093+0000 7f7ecb7fe640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f7ecc1e8ac0 0x7f7ecc1e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:12.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.093+0000 7f7ecb7fe640 10 monclient: get_auth_request con 0x7f7ecc1e8ac0 auth_method 0 2024-03-27T21:40:12.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.093+0000 7f7ec8ff9640 10 client.4469.objecter ms_handle_connect 0x7f7ecc1e8ac0 2024-03-27T21:40:12.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.093+0000 7f7ecb7fe640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f7ecc1e8ac0 0x7f7ecc1e8f40 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:12.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.094+0000 7f7ec8ff9640 1 -- 172.21.15.60:0/2595194160 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7ecc1e9480 con 0x7f7ecc1e8ac0 2024-03-27T21:40:12.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.094+0000 7f7ec8ff9640 10 client.4469.objecter ms_dispatch 0x7f7ecc000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:40:12.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.094+0000 7f7ec8ff9640 10 client.4469.objecter _finish_command 12 = system:0 2024-03-27T21:40:12.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.094+0000 7f7ec8ff9640 20 client.4469.objecter put_session s=0x7f7ecc1e4330 osd=6 4 2024-03-27T21:40:12.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.094+0000 7f7ec8ff9640 15 client.4469.objecter _session_command_op_remove 6 12 2024-03-27T21:40:12.108 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:12.109 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:12.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.122+0000 7f7eb27fc640 10 client.4469.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:12.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.122+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign -1 13 2024-03-27T21:40:12.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.122+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc1e4330 osd=6 3 2024-03-27T21:40:12.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.122+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc1e4330 osd=6 3 2024-03-27T21:40:12.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.122+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc1e4330 osd=6 3 2024-03-27T21:40:12.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.122+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_remove -1 13 2024-03-27T21:40:12.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.122+0000 7f7eb27fc640 20 client.4469.objecter get_session s=0x7f7ecc1e4330 osd=6 3 2024-03-27T21:40:12.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.122+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign 6 13 2024-03-27T21:40:12.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.122+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc1e4330 osd=6 4 2024-03-27T21:40:12.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.122+0000 7f7eb27fc640 10 client.4469.objecter _send_command 13 2024-03-27T21:40:12.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.122+0000 7f7eb27fc640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7ecc1e9480 con 0x7f7ecc1e8ac0 2024-03-27T21:40:12.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.124+0000 7f7ec8ff9640 1 -- 172.21.15.60:0/2595194160 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7ecc1e9480 con 0x7f7ecc1e8ac0 2024-03-27T21:40:12.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.124+0000 7f7ec8ff9640 10 client.4469.objecter ms_dispatch 0x7f7ecc000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:12.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.124+0000 7f7ec8ff9640 10 client.4469.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:12.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.124+0000 7f7ec8ff9640 20 client.4469.objecter put_session s=0x7f7ecc1e4330 osd=6 4 2024-03-27T21:40:12.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.124+0000 7f7ec8ff9640 15 client.4469.objecter _session_command_op_remove 6 13 2024-03-27T21:40:12.111 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:12.112 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:12.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.125+0000 7f7eb27fc640 10 client.4469.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:12.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.125+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign -1 14 2024-03-27T21:40:12.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.125+0000 7f7eb27fc640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f7ecc1ede20 0x7f7ecc1f0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:12.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.125+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc1e9480 osd=7 3 2024-03-27T21:40:12.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.125+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc1e9480 osd=7 3 2024-03-27T21:40:12.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.125+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc1e9480 osd=7 3 2024-03-27T21:40:12.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.125+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_remove -1 14 2024-03-27T21:40:12.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.125+0000 7f7eb27fc640 20 client.4469.objecter get_session s=0x7f7ecc1e9480 osd=7 3 2024-03-27T21:40:12.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.125+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign 7 14 2024-03-27T21:40:12.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.125+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc1e9480 osd=7 4 2024-03-27T21:40:12.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.125+0000 7f7eb27fc640 10 client.4469.objecter _send_command 14 2024-03-27T21:40:12.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.125+0000 7f7eb27fc640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ecc1e4520 con 0x7f7ecc1ede20 2024-03-27T21:40:12.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.125+0000 7f7ed0c28640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f7ecc1ede20 0x7f7ecc1f0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:12.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.125+0000 7f7ed0c28640 10 monclient: get_auth_request con 0x7f7ecc1ede20 auth_method 0 2024-03-27T21:40:12.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.125+0000 7f7ec8ff9640 10 client.4469.objecter ms_handle_connect 0x7f7ecc1ede20 2024-03-27T21:40:12.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.125+0000 7f7ed0c28640 1 --2- 172.21.15.60:0/2595194160 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f7ecc1ede20 0x7f7ecc1f0210 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:12.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.126+0000 7f7ec8ff9640 1 -- 172.21.15.60:0/2595194160 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7ecc1e4520 con 0x7f7ecc1ede20 2024-03-27T21:40:12.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.126+0000 7f7ec8ff9640 10 client.4469.objecter ms_dispatch 0x7f7ecc000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:40:12.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.126+0000 7f7ec8ff9640 10 client.4469.objecter _finish_command 14 = system:0 2024-03-27T21:40:12.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.126+0000 7f7ec8ff9640 20 client.4469.objecter put_session s=0x7f7ecc1e9480 osd=7 4 2024-03-27T21:40:12.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.126+0000 7f7ec8ff9640 15 client.4469.objecter _session_command_op_remove 7 14 2024-03-27T21:40:12.117 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:12.131+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:12.122 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:40:12.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.154+0000 7f7eb27fc640 10 client.4469.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:12.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.154+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign -1 15 2024-03-27T21:40:12.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.154+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc1e9480 osd=7 3 2024-03-27T21:40:12.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.154+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc1e9480 osd=7 3 2024-03-27T21:40:12.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.154+0000 7f7eb27fc640 20 client.4469.objecter _get_session s=0x7f7ecc1e9480 osd=7 3 2024-03-27T21:40:12.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.154+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_remove -1 15 2024-03-27T21:40:12.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.154+0000 7f7eb27fc640 20 client.4469.objecter get_session s=0x7f7ecc1e9480 osd=7 3 2024-03-27T21:40:12.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.154+0000 7f7eb27fc640 15 client.4469.objecter _session_command_op_assign 7 15 2024-03-27T21:40:12.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.154+0000 7f7eb27fc640 20 client.4469.objecter put_session s=0x7f7ecc1e9480 osd=7 4 2024-03-27T21:40:12.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.154+0000 7f7eb27fc640 10 client.4469.objecter _send_command 15 2024-03-27T21:40:12.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.154+0000 7f7eb27fc640 1 -- 172.21.15.60:0/2595194160 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7ecc1e4520 con 0x7f7ecc1ede20 2024-03-27T21:40:12.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.156+0000 7f7ec8ff9640 1 -- 172.21.15.60:0/2595194160 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7ecc1e4520 con 0x7f7ecc1ede20 2024-03-27T21:40:12.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.156+0000 7f7ec8ff9640 10 client.4469.objecter ms_dispatch 0x7f7ecc000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:12.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.156+0000 7f7ec8ff9640 10 client.4469.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:12.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.156+0000 7f7ec8ff9640 20 client.4469.objecter put_session s=0x7f7ecc1e9480 osd=7 4 2024-03-27T21:40:12.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:12.156+0000 7f7ec8ff9640 15 client.4469.objecter _session_command_op_remove 7 15 2024-03-27T21:40:12.144 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:12.144 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:12.148 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:40:12.148 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:40:12.219 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:12.233+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:12.253 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:12.253 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:12.262 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:40:12.320 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:12.334+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:12.420 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:12.435+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:12.433 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:12.433 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:12.442 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:40:12.522 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:12.536+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:12.559 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:12.559 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:12.571 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:40:12.623 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:12.636+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:12.682 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:12.682 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:12.691 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:40:12.732 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:12.746+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:12.808 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:12.808 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:12.817 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:40:12.824 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:12.838+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:12.921 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:12.922 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:12.925 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:12.939+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:12.930 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:40:13.026 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:13.040+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:13.076 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:13.076 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:13.085 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:40:13.127 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:13.141+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:13.228 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:13.241+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:13.229 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:13.229 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:13.238 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:40:13.328 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:13.343+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:13.342 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:13.343 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:13.351 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:40:13.434 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:13.449+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:13.495 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:13.496 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:13.504 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:40:13.530 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:13.544+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:13.632 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:13.645+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:13.649 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:13.649 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:13.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:40:13.710 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-27T21:40:13.734 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:13.749+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:13.764 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:13.764 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:13.773 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:40:13.835 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:13.849+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.882+0000 7f3e74495640 10 monclient: get_monmap_and_config 2024-03-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.882+0000 7f3e74495640 10 monclient: build_initial_monmap 2024-03-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.882+0000 7f3e74495640 10 monclient: monmap: 2024-03-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:13.883488+0000 2024-03-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:13.883488+0000 2024-03-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:13.870 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:13.870 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:13.870 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:13.870 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:13.870 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:13.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.883+0000 7f3e74495640 1 Processor -- start 2024-03-27T21:40:13.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.883+0000 7f3e74495640 1 -- start start 2024-03-27T21:40:13.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.883+0000 7f3e74495640 10 monclient: init 2024-03-27T21:40:13.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.884+0000 7f3e74495640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:13.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.884+0000 7f3e74495640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-03-27T21:40:13.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.884+0000 7f3e74495640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3e6c0b40b0 0x7f3e6c0a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:13.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.884+0000 7f3e74495640 10 monclient(hunting): picked mon.noname-c con 0x7f3e6c0b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:13.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.884+0000 7f3e74495640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3e6c13b020 0x7f3e6c13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:13.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.884+0000 7f3e71a30640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3e6c13b020 0x7f3e6c13b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:13.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.884+0000 7f3e71a30640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3e6c13b020 0x7f3e6c13b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:56496/0 (socket says 172.21.15.60:56496) 2024-03-27T21:40:13.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.884+0000 7f3e74495640 10 monclient(hunting): picked mon.noname-b con 0x7f3e6c13b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:13.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.884+0000 7f3e71a30640 1 -- 172.21.15.60:0/1145262576 learned_addr learned my addr 172.21.15.60:0/1145262576 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:13.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.884+0000 7f3e74495640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e6c1416f0 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:13.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.884+0000 7f3e72231640 1 --2- 172.21.15.60:0/1145262576 >> v2:172.21.15.191:3300/0 conn(0x7f3e6c0b40b0 0x7f3e6c0a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:13.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.884+0000 7f3e74495640 1 -- 172.21.15.60:0/1145262576 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e6c0b3830 con 0x7f3e6c1416f0 2024-03-27T21:40:13.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.884+0000 7f3e74495640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:13.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.884+0000 7f3e74495640 1 -- 172.21.15.60:0/1145262576 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e6c0b4490 con 0x7f3e6c0b40b0 2024-03-27T21:40:13.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.884+0000 7f3e74495640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:13.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.884+0000 7f3e74495640 1 -- 172.21.15.60:0/1145262576 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c142290 con 0x7f3e6c13b020 2024-03-27T21:40:13.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.884+0000 7f3e74495640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:13.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.884+0000 7f3e74495640 10 monclient(hunting): authenticate will time out at 855.105861s 2024-03-27T21:40:13.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.885+0000 7f3e71a30640 10 monclient(hunting): get_auth_request con 0x7f3e6c13b020 auth_method 0 2024-03-27T21:40:13.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.885+0000 7f3e71a30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:13.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.885+0000 7f3e71a30640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:13.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.885+0000 7f3e71a30640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:13.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.885+0000 7f3e71a30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:13.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.885+0000 7f3e71a30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:13.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.885+0000 7f3e71a30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:13.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.885+0000 7f3e72231640 10 monclient(hunting): get_auth_request con 0x7f3e6c0b40b0 auth_method 0 2024-03-27T21:40:13.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.885+0000 7f3e72231640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:13.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.885+0000 7f3e72231640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.885+0000 7f3e72231640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.885+0000 7f3e71a30640 10 monclient(hunting): handle_auth_done global_id 4472 payload 274 2024-03-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.885+0000 7f3e71a30640 1 -- 172.21.15.60:0/1145262576 >> v2:172.21.15.191:3300/0 conn(0x7f3e6c0b40b0 msgr2=0x7f3e6c0a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.885+0000 7f3e71a30640 1 --2- 172.21.15.60:0/1145262576 >> v2:172.21.15.191:3300/0 conn(0x7f3e6c0b40b0 0x7f3e6c0a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.885+0000 7f3e71a30640 1 -- 172.21.15.60:0/1145262576 >> v1:172.21.15.60:6789/0 conn(0x7f3e6c1416f0 legacy=0x7f3e6c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.885+0000 7f3e71a30640 10 monclient: _finish_hunting 0 2024-03-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.885+0000 7f3e71a30640 1 monclient: found mon.noname-b 2024-03-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.885+0000 7f3e71a30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.885+0000 7f3e71a30640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.885+0000 7f3e71a30640 1 -- 172.21.15.60:0/1145262576 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6c0aa520 con 0x7f3e6c13b020 2024-03-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.886+0000 7f3e71a30640 1 --2- 172.21.15.60:0/1145262576 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3e6c13b020 0x7f3e6c13b480 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c009870 tx=0x7f3e5c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c1739ba4141e059 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.886+0000 7f3e7122f640 1 -- 172.21.15.60:0/1145262576 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c030840 con 0x7f3e6c13b020 2024-03-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.886+0000 7f3e7122f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.886+0000 7f3e7122f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.886+0000 7f3e7122f640 10 monclient: dump: 2024-03-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.886+0000 7f3e7122f640 10 monclient: _finish_auth 0 2024-03-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.886+0000 7f3e7122f640 10 monclient: _check_auth_tickets 2024-03-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.886+0000 7f3e7122f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.886+0000 7f3e74495640 5 monclient: authenticate success, global_id 4472 2024-03-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.886+0000 7f3e7122f640 1 -- 172.21.15.60:0/1145262576 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5c0309a0 con 0x7f3e6c13b020 2024-03-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.886+0000 7f3e74495640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.886+0000 7f3e7122f640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.886+0000 7f3e7122f640 1 -- 172.21.15.60:0/1145262576 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c030d30 con 0x7f3e6c13b020 2024-03-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.886+0000 7f3e74495640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.886+0000 7f3e74495640 4 set_mon_vals no callback set 2024-03-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.886+0000 7f3e7122f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.886+0000 7f3e7122f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.886+0000 7f3e7122f640 10 monclient: dump: 2024-03-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.887+0000 7f3e74495640 10 monclient: shutdown 2024-03-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.887+0000 7f3e74495640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.887+0000 7f3e74495640 1 -- 172.21.15.60:0/1145262576 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3e6c13b020 msgr2=0x7f3e6c13b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.887+0000 7f3e74495640 1 --2- 172.21.15.60:0/1145262576 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3e6c13b020 0x7f3e6c13b480 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c009870 tx=0x7f3e5c02b980 comp rx=0 tx=0).stop 2024-03-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.887+0000 7f3e74495640 1 -- 172.21.15.60:0/1145262576 shutdown_connections 2024-03-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.887+0000 7f3e74495640 1 --2- 172.21.15.60:0/1145262576 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3e6c13b020 0x7f3e6c13b480 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.887+0000 7f3e74495640 1 --2- 172.21.15.60:0/1145262576 >> v2:172.21.15.191:3300/0 conn(0x7f3e6c0b40b0 0x7f3e6c0a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.887+0000 7f3e74495640 1 -- 172.21.15.60:0/1145262576 >> 172.21.15.60:0/1145262576 conn(0x7f3e6c0608e0 msgr2=0x7f3e6c060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.888+0000 7f3e74495640 1 -- 172.21.15.60:0/1145262576 shutdown_connections 2024-03-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.888+0000 7f3e74495640 1 -- 172.21.15.60:0/1145262576 wait complete. 2024-03-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.888+0000 7f3e74495640 10 monclient: build_initial_monmap 2024-03-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.888+0000 7f3e74495640 10 monclient: monmap: 2024-03-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.889+0000 7f3e74495640 1 librados: starting msgr at 2024-03-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.889+0000 7f3e74495640 1 librados: starting objecter 2024-03-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.889+0000 7f3e74495640 1 Processor -- start 2024-03-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.889+0000 7f3e74495640 1 -- start start 2024-03-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.889+0000 7f3e74495640 1 librados: setting wanted keys 2024-03-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.889+0000 7f3e74495640 1 librados: calling monclient init 2024-03-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.889+0000 7f3e74495640 10 monclient: init 2024-03-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e74495640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e74495640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e74495640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e6c0b40b0 0x7f3e6c0aac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e74495640 10 monclient(hunting): picked mon.noname-b con 0x7f3e6c0b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e74495640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e6c13b020 0x7f3e6c13ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e74495640 10 monclient(hunting): picked mon.noname-a con 0x7f3e6c13b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e74495640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3e6c1416f0 0x7f3e6c1d6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e72231640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e6c0b40b0 0x7f3e6c0aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e72231640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e6c0b40b0 0x7f3e6c0aac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:53886/0 (socket says 172.21.15.60:53886) 2024-03-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e71a30640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e6c13b020 0x7f3e6c13ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e71a30640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e6c13b020 0x7f3e6c13ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35898/0 (socket says 172.21.15.60:35898) 2024-03-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e72a32640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3e6c1416f0 0x7f3e6c1d6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e72a32640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3e6c1416f0 0x7f3e6c1d6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:56502/0 (socket says 172.21.15.60:56502) 2024-03-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e74495640 10 monclient(hunting): picked mon.noname-c con 0x7f3e6c1416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e74495640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e72231640 1 -- 172.21.15.60:0/339128939 learned_addr learned my addr 172.21.15.60:0/339128939 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e74495640 1 -- 172.21.15.60:0/339128939 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c0ac180 con 0x7f3e6c13b020 2024-03-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e74495640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e74495640 1 -- 172.21.15.60:0/339128939 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c13bfa0 con 0x7f3e6c0b40b0 2024-03-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e74495640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e74495640 1 -- 172.21.15.60:0/339128939 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c1d64e0 con 0x7f3e6c1416f0 2024-03-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e74495640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.890+0000 7f3e74495640 10 monclient(hunting): authenticate will time out at 855.111903s 2024-03-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72231640 10 monclient(hunting): get_auth_request con 0x7f3e6c0b40b0 auth_method 0 2024-03-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72231640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72231640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72231640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e71a30640 10 monclient(hunting): get_auth_request con 0x7f3e6c13b020 auth_method 0 2024-03-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e71a30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e71a30640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e71a30640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72a32640 10 monclient(hunting): get_auth_request con 0x7f3e6c1416f0 auth_method 0 2024-03-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72a32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:13.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72a32640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:13.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72a32640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:13.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e71a30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:13.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e71a30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:13.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e71a30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:13.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72231640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:13.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72231640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:13.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72231640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:13.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72231640 10 monclient(hunting): handle_auth_done global_id 4540 payload 824 2024-03-27T21:40:13.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72231640 1 -- 172.21.15.60:0/339128939 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3e6c1416f0 msgr2=0x7f3e6c1d6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:13.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72231640 1 --2- 172.21.15.60:0/339128939 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3e6c1416f0 0x7f3e6c1d6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:13.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72231640 1 -- 172.21.15.60:0/339128939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e6c13b020 msgr2=0x7f3e6c13ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:13.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72231640 1 --2- 172.21.15.60:0/339128939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e6c13b020 0x7f3e6c13ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:13.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72231640 10 monclient: _finish_hunting 0 2024-03-27T21:40:13.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72231640 1 monclient: found mon.noname-b 2024-03-27T21:40:13.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72231640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:13.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72231640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:40:13.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72231640 1 -- 172.21.15.60:0/339128939 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5c0094d0 con 0x7f3e6c0b40b0 2024-03-27T21:40:13.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e71a30640 1 --2- 172.21.15.60:0/339128939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e6c13b020 0x7f3e6c13ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:13.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e72231640 1 --2- 172.21.15.60:0/339128939 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e6c0b40b0 0x7f3e6c0aac40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3e6800d800 tx=0x7f3e6800dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:13.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e62ffd640 10 client.?.objecter ms_handle_connect 0x7f3e6c0b40b0 2024-03-27T21:40:13.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e62ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:13.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e62ffd640 1 -- 172.21.15.60:0/339128939 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6800b6c0 con 0x7f3e6c0b40b0 2024-03-27T21:40:13.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:13.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e62ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:13.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e62ffd640 10 monclient: dump: 2024-03-27T21:40:13.904 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:13.904 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:13.904 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:13.904 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:13.904 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:13.904 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:13.904 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:13.904 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:13.904 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:13.904 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:13.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e62ffd640 10 monclient: _finish_auth 0 2024-03-27T21:40:13.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e62ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:40:13.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:13.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e74495640 5 monclient: authenticate success, global_id 4540 2024-03-27T21:40:13.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e62ffd640 1 -- 172.21.15.60:0/339128939 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e6800b820 con 0x7f3e6c0b40b0 2024-03-27T21:40:13.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e62ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:13.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.891+0000 7f3e62ffd640 1 -- 172.21.15.60:0/339128939 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6800bbb0 con 0x7f3e6c0b40b0 2024-03-27T21:40:13.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.892+0000 7f3e74495640 10 monclient: _renew_subs 2024-03-27T21:40:13.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.892+0000 7f3e74495640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:13.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.892+0000 7f3e74495640 1 -- 172.21.15.60:0/339128939 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e6c1d6f40 con 0x7f3e6c0b40b0 2024-03-27T21:40:13.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.892+0000 7f3e62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:13.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.892+0000 7f3e62ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:40:13.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.892+0000 7f3e62ffd640 10 monclient: dump: 2024-03-27T21:40:13.905 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:13.905 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:13.905 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:13.906 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:13.906 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:13.906 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:13.906 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:13.906 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:13.906 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:13.906 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:13.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.892+0000 7f3e70a2e640 4 set_mon_vals no callback set 2024-03-27T21:40:13.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.892+0000 7f3e74495640 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:13.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.892+0000 7f3e74495640 10 monclient: _renew_subs 2024-03-27T21:40:13.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.892+0000 7f3e74495640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:13.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.892+0000 7f3e74495640 1 -- 172.21.15.60:0/339128939 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e6c13e9f0 con 0x7f3e6c0b40b0 2024-03-27T21:40:13.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.892+0000 7f3e74495640 1 librados: init done 2024-03-27T21:40:13.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.894+0000 7f3e74495640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:13.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.894+0000 7f3e74495640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:13.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.894+0000 7f3e74495640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:13.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.894+0000 7f3e74495640 1 -- 172.21.15.60:0/339128939 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e6c1d66a0 con 0x7f3e6c0b40b0 2024-03-27T21:40:13.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.895+0000 7f3e62ffd640 1 -- 172.21.15.60:0/339128939 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3e6800bd10 con 0x7f3e6c0b40b0 2024-03-27T21:40:13.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.895+0000 7f3e62ffd640 1 --2- 172.21.15.60:0/339128939 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3e5006c590 0x7f3e5006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:13.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.896+0000 7f3e62ffd640 1 -- 172.21.15.60:0/339128939 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f3e680895d0 con 0x7f3e6c0b40b0 2024-03-27T21:40:13.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.896+0000 7f3e62ffd640 10 client.4540.objecter ms_dispatch 0x7f3e6c000b90 osd_map(29..29 src has 1..29) v4 2024-03-27T21:40:13.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.896+0000 7f3e62ffd640 3 client.4540.objecter handle_osd_map got epochs [29,29] > 0 2024-03-27T21:40:13.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.896+0000 7f3e62ffd640 3 client.4540.objecter handle_osd_map decoding full epoch 29 2024-03-27T21:40:13.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.896+0000 7f3e71a30640 1 --2- 172.21.15.60:0/339128939 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3e5006c590 0x7f3e5006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:13.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.896+0000 7f3e71a30640 10 monclient: get_auth_request con 0x7f3e5006c590 auth_method 0 2024-03-27T21:40:13.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.896+0000 7f3e62ffd640 20 client.4540.objecter dump_active .. 0 homeless 2024-03-27T21:40:13.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.896+0000 7f3e62ffd640 10 client.4540.objecter ms_handle_connect 0x7f3e5006c590 2024-03-27T21:40:13.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.896+0000 7f3e71a30640 1 --2- 172.21.15.60:0/339128939 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3e5006c590 0x7f3e5006ea50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c036000 tx=0x7f3e5c046040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:13.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.900+0000 7f3e62ffd640 1 -- 172.21.15.60:0/339128939 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3e68054a10 con 0x7f3e6c0b40b0 2024-03-27T21:40:13.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.900+0000 7f3e62ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:13.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:13.900+0000 7f3e62ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:13.920 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:13.920 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:13.928 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:40:13.936 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:13.949+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:14.033 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:14.033 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:14.036 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:14.050+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:14.041 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:40:14.137 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:14.151+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:14.146 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:14.146 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:14.149 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:40:14.154 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:40:14.238 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:14.253+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:14.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.263+0000 7f87d5e84640 10 monclient: get_monmap_and_config 2024-03-27T21:40:14.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.263+0000 7f87d5e84640 10 monclient: build_initial_monmap 2024-03-27T21:40:14.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.263+0000 7f87d5e84640 10 monclient: monmap: 2024-03-27T21:40:14.250 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:14.250 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:14.250 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:14.264588+0000 2024-03-27T21:40:14.250 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:14.264588+0000 2024-03-27T21:40:14.250 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:14.250 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:14.250 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:14.251 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:14.251 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:14.251 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:14.251 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:14.251 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:14.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.264+0000 7f87d5e84640 1 Processor -- start 2024-03-27T21:40:14.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.264+0000 7f87d5e84640 1 -- start start 2024-03-27T21:40:14.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.264+0000 7f87d5e84640 10 monclient: init 2024-03-27T21:40:14.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.264+0000 7f87d5e84640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:14.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87d5e84640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-03-27T21:40:14.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87d5e84640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f87d00b40b0 0x7f87d00a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:14.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87d5e84640 10 monclient(hunting): picked mon.noname-a con 0x7f87d00b40b0 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:14.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87d5e84640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f87d013b020 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:14.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87d5e84640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f87d013b480 0x7f87d0141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:14.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87d5e84640 10 monclient(hunting): picked mon.noname-c con 0x7f87d013b480 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:14.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87d5e84640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d013b8d0 con 0x7f87d013b020 2024-03-27T21:40:14.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87d5e84640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:14.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87d5e84640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87d00b4490 con 0x7f87d00b40b0 2024-03-27T21:40:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87d5e84640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87d5e84640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87d0142340 con 0x7f87d013b480 2024-03-27T21:40:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87d5e84640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87d5e84640 10 monclient(hunting): authenticate will time out at 855.486158s 2024-03-27T21:40:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87cffff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f87d013b480 0x7f87d0141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87ceffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f87d013b020 0x7f87d01416f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39610/0 (socket says 172.21.15.60:39610) 2024-03-27T21:40:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87ceffd640 1 -- 172.21.15.60:0/1118538372 learned_addr learned my addr 172.21.15.60:0/1118538372 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:40:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87cffff640 10 monclient(hunting): get_auth_request con 0x7f87d013b480 auth_method 0 2024-03-27T21:40:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87cffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87cf7fe640 1 --2- 172.21.15.60:0/1118538372 >> v2:172.21.15.60:3300/0 conn(0x7f87d00b40b0 0x7f87d00a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87cf7fe640 10 monclient(hunting): get_auth_request con 0x7f87d00b40b0 auth_method 0 2024-03-27T21:40:14.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:14.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:14.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:14.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:14.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:14.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:14.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:14.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:14.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.265+0000 7f87cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:14.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.266+0000 7f87cffff640 10 monclient(hunting): handle_auth_done global_id 4543 payload 274 2024-03-27T21:40:14.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.266+0000 7f87cffff640 1 -- 172.21.15.60:0/1118538372 >> v2:172.21.15.60:3300/0 conn(0x7f87d00b40b0 msgr2=0x7f87d00a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:14.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.266+0000 7f87cffff640 1 --2- 172.21.15.60:0/1118538372 >> v2:172.21.15.60:3300/0 conn(0x7f87d00b40b0 0x7f87d00a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:14.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.266+0000 7f87cffff640 1 -- 172.21.15.60:0/1118538372 >> v1:172.21.15.60:6789/0 conn(0x7f87d013b020 legacy=0x7f87d01416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:14.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.266+0000 7f87cffff640 10 monclient: _finish_hunting 0 2024-03-27T21:40:14.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.266+0000 7f87cffff640 1 monclient: found mon.noname-c 2024-03-27T21:40:14.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.266+0000 7f87cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:14.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.266+0000 7f87cffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:40:14.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.266+0000 7f87cffff640 1 -- 172.21.15.60:0/1118538372 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d00b3830 con 0x7f87d013b480 2024-03-27T21:40:14.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.267+0000 7f87cffff640 1 --2- 172.21.15.60:0/1118538372 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87d013b480 0x7f87d0141e00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f87d00b3a20 tx=0x7f87c802be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e101edbf073e015 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:14.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.267+0000 7f87ce7fc640 1 -- 172.21.15.60:0/1118538372 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c802cbe0 con 0x7f87d013b480 2024-03-27T21:40:14.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.267+0000 7f87ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:14.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.267+0000 7f87ce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:14.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.267+0000 7f87ce7fc640 10 monclient: dump: 2024-03-27T21:40:14.270 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:14.271 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:14.271 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:14.271 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:14.271 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:14.271 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:14.271 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:14.271 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:14.271 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:14.271 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:14.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.267+0000 7f87ce7fc640 10 monclient: _finish_auth 0 2024-03-27T21:40:14.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.267+0000 7f87ce7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:40:14.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.267+0000 7f87ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:14.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.267+0000 7f87ce7fc640 1 -- 172.21.15.60:0/1118538372 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c802cd40 con 0x7f87d013b480 2024-03-27T21:40:14.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.267+0000 7f87ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:14.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.267+0000 7f87ce7fc640 1 -- 172.21.15.60:0/1118538372 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c8034660 con 0x7f87d013b480 2024-03-27T21:40:14.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.267+0000 7f87ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:14.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.267+0000 7f87ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:40:14.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.267+0000 7f87ce7fc640 10 monclient: dump: 2024-03-27T21:40:14.272 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:14.272 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:14.272 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:14.272 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:14.272 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:14.272 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:14.272 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:14.272 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:14.272 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:14.272 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:14.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.267+0000 7f87d5e84640 5 monclient: authenticate success, global_id 4543 2024-03-27T21:40:14.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.267+0000 7f87d5e84640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:14.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.267+0000 7f87d5e84640 4 set_mon_vals no callback set 2024-03-27T21:40:14.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.268+0000 7f87d5e84640 10 monclient: shutdown 2024-03-27T21:40:14.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.268+0000 7f87d5e84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:14.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.268+0000 7f87d5e84640 1 -- 172.21.15.60:0/1118538372 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87d013b480 msgr2=0x7f87d0141e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:14.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.268+0000 7f87d5e84640 1 --2- 172.21.15.60:0/1118538372 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87d013b480 0x7f87d0141e00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f87d00b3a20 tx=0x7f87c802be30 comp rx=0 tx=0).stop 2024-03-27T21:40:14.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.268+0000 7f87d5e84640 1 -- 172.21.15.60:0/1118538372 shutdown_connections 2024-03-27T21:40:14.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.268+0000 7f87d5e84640 1 --2- 172.21.15.60:0/1118538372 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87d013b480 0x7f87d0141e00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:14.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.268+0000 7f87d5e84640 1 --2- 172.21.15.60:0/1118538372 >> v2:172.21.15.60:3300/0 conn(0x7f87d00b40b0 0x7f87d00a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:14.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.268+0000 7f87d5e84640 1 -- 172.21.15.60:0/1118538372 >> 172.21.15.60:0/1118538372 conn(0x7f87d00608e0 msgr2=0x7f87d0060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:14.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.268+0000 7f87d5e84640 1 -- 172.21.15.60:0/1118538372 shutdown_connections 2024-03-27T21:40:14.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.268+0000 7f87d5e84640 1 -- 172.21.15.60:0/1118538372 wait complete. 2024-03-27T21:40:14.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.269+0000 7f87d5e84640 10 monclient: build_initial_monmap 2024-03-27T21:40:14.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.269+0000 7f87d5e84640 10 monclient: monmap: 2024-03-27T21:40:14.274 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:14.274 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:14.274 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:14.274 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:14.274 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:14.274 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:14.274 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:14.274 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:14.274 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:14.275 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:14.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.269+0000 7f87d5e84640 1 librados: starting msgr at 2024-03-27T21:40:14.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.269+0000 7f87d5e84640 1 librados: starting objecter 2024-03-27T21:40:14.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.269+0000 7f87d5e84640 1 Processor -- start 2024-03-27T21:40:14.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.269+0000 7f87d5e84640 1 -- start start 2024-03-27T21:40:14.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.269+0000 7f87d5e84640 1 librados: setting wanted keys 2024-03-27T21:40:14.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.269+0000 7f87d5e84640 1 librados: calling monclient init 2024-03-27T21:40:14.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.269+0000 7f87d5e84640 10 monclient: init 2024-03-27T21:40:14.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87d5e84640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:14.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87d5e84640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:40:14.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87d5e84640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87d00b40b0 0x7f87d0139db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:14.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87d5e84640 10 monclient(hunting): picked mon.noname-b con 0x7f87d00b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:14.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87d5e84640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f87d013b020 0x7f87d013d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:14.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87d5e84640 10 monclient(hunting): picked mon.noname-c con 0x7f87d013b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:14.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87d5e84640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87d013b480 0x7f87d01330c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:14.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87d5e84640 10 monclient(hunting): picked mon.noname-a con 0x7f87d013b480 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:14.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87d5e84640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:14.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87d5e84640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d013e6c0 con 0x7f87d013b480 2024-03-27T21:40:14.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87d5e84640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:14.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87d5e84640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d013a8c0 con 0x7f87d00b40b0 2024-03-27T21:40:14.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87d5e84640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:14.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87d5e84640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d013dca0 con 0x7f87d013b020 2024-03-27T21:40:14.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87d5e84640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:14.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87d5e84640 10 monclient(hunting): authenticate will time out at 855.491570s 2024-03-27T21:40:14.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87ceffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f87d013b020 0x7f87d013d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:14.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87ceffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f87d013b020 0x7f87d013d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:56518/0 (socket says 172.21.15.60:56518) 2024-03-27T21:40:14.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87cffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87d013b480 0x7f87d01330c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:14.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87ceffd640 1 -- 172.21.15.60:0/1397817274 learned_addr learned my addr 172.21.15.60:0/1397817274 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:14.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87ceffd640 10 monclient(hunting): get_auth_request con 0x7f87d013b020 auth_method 0 2024-03-27T21:40:14.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:14.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:14.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:14.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:14.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:14.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:14.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87cffff640 10 monclient(hunting): get_auth_request con 0x7f87d013b480 auth_method 0 2024-03-27T21:40:14.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:14.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87cffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:14.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.270+0000 7f87cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:14.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87ceffd640 10 monclient(hunting): handle_auth_done global_id 4478 payload 824 2024-03-27T21:40:14.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87ceffd640 1 -- 172.21.15.60:0/1397817274 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87d00b40b0 msgr2=0x7f87d0139db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:14.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87ceffd640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87d00b40b0 0x7f87d0139db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:14.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87ceffd640 1 -- 172.21.15.60:0/1397817274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87d013b480 msgr2=0x7f87d01330c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:14.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87ceffd640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87d013b480 0x7f87d01330c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:14.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87ceffd640 10 monclient: _finish_hunting 0 2024-03-27T21:40:14.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87ceffd640 1 monclient: found mon.noname-c 2024-03-27T21:40:14.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:14.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87ceffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:40:14.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87ceffd640 1 -- 172.21.15.60:0/1397817274 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c8009a70 con 0x7f87d013b020 2024-03-27T21:40:14.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87cffff640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87d013b480 0x7f87d01330c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:40:14.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87ceffd640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f87d013b020 0x7f87d013d170 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f87c400c4d0 tx=0x7f87c400c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:14.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87b7fff640 10 client.?.objecter ms_handle_connect 0x7f87d013b020 2024-03-27T21:40:14.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87b7fff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:14.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87b7fff640 1 -- 172.21.15.60:0/1397817274 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c4005ce0 con 0x7f87d013b020 2024-03-27T21:40:14.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:14.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87b7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:14.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87b7fff640 10 monclient: dump: 2024-03-27T21:40:14.279 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:14.279 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:14.279 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:14.279 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:14.279 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:14.279 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:14.279 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:14.279 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:14.279 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:14.280 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:14.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87b7fff640 10 monclient: _finish_auth 0 2024-03-27T21:40:14.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87b7fff640 10 monclient: _check_auth_tickets 2024-03-27T21:40:14.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:14.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87b7fff640 1 -- 172.21.15.60:0/1397817274 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c4005e40 con 0x7f87d013b020 2024-03-27T21:40:14.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:14.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87b7fff640 1 -- 172.21.15.60:0/1397817274 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c4007270 con 0x7f87d013b020 2024-03-27T21:40:14.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:14.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87b7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:14.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87b7fff640 10 monclient: dump: 2024-03-27T21:40:14.280 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:14.280 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:14.280 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:14.281 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:14.281 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:14.281 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:14.281 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:14.281 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:14.281 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:14.281 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:14.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.271+0000 7f87cdffb640 4 set_mon_vals no callback set 2024-03-27T21:40:14.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.272+0000 7f87d5e84640 5 monclient: authenticate success, global_id 4478 2024-03-27T21:40:14.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.272+0000 7f87d5e84640 10 monclient: _renew_subs 2024-03-27T21:40:14.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.272+0000 7f87d5e84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:14.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.272+0000 7f87d5e84640 1 -- 172.21.15.60:0/1397817274 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d013d790 con 0x7f87d013b020 2024-03-27T21:40:14.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.272+0000 7f87d5e84640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:14.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.272+0000 7f87d5e84640 10 monclient: _renew_subs 2024-03-27T21:40:14.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.272+0000 7f87d5e84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:14.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.272+0000 7f87d5e84640 1 -- 172.21.15.60:0/1397817274 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d013da70 con 0x7f87d013b020 2024-03-27T21:40:14.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.272+0000 7f87d5e84640 1 librados: init done 2024-03-27T21:40:14.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.273+0000 7f87d5e84640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:14.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.273+0000 7f87d5e84640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:14.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.273+0000 7f87d5e84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:14.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.273+0000 7f87d5e84640 1 -- 172.21.15.60:0/1397817274 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f87d0069470 con 0x7f87d013b020 2024-03-27T21:40:14.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.273+0000 7f87b7fff640 1 -- 172.21.15.60:0/1397817274 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f87c40073d0 con 0x7f87d013b020 2024-03-27T21:40:14.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.274+0000 7f87b7fff640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f87bc06c590 0x7f87bc06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:14.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.274+0000 7f87b7fff640 1 -- 172.21.15.60:0/1397817274 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f87c4088550 con 0x7f87d013b020 2024-03-27T21:40:14.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.274+0000 7f87b7fff640 10 client.4478.objecter ms_dispatch 0x7f87d0000b90 osd_map(29..29 src has 1..29) v4 2024-03-27T21:40:14.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.274+0000 7f87b7fff640 3 client.4478.objecter handle_osd_map got epochs [29,29] > 0 2024-03-27T21:40:14.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.274+0000 7f87b7fff640 3 client.4478.objecter handle_osd_map decoding full epoch 29 2024-03-27T21:40:14.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.274+0000 7f87b7fff640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-27T21:40:14.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.274+0000 7f87cf7fe640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f87bc06c590 0x7f87bc06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:14.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.274+0000 7f87b7fff640 1 -- 172.21.15.60:0/1397817274 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v29) v1 ==== 54+0+16 (secure 0 0 0) 0x7f87c408f070 con 0x7f87d013b020 2024-03-27T21:40:14.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.274+0000 7f87b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:14.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.274+0000 7f87b7fff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:14.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.274+0000 7f87cf7fe640 10 monclient: get_auth_request con 0x7f87bc06c590 auth_method 0 2024-03-27T21:40:14.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.274+0000 7f87b5ffb640 10 client.4478.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:14.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.274+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign -1 1 2024-03-27T21:40:14.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.274+0000 7f87b5ffb640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f87a8002f70 0x7f87a8005430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:14.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.275+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a8002860 osd=0 3 2024-03-27T21:40:14.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.275+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a8002860 osd=0 3 2024-03-27T21:40:14.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.275+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a8002860 osd=0 3 2024-03-27T21:40:14.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.275+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_remove -1 1 2024-03-27T21:40:14.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.275+0000 7f87b5ffb640 20 client.4478.objecter get_session s=0x7f87a8002860 osd=0 3 2024-03-27T21:40:14.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.275+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign 0 1 2024-03-27T21:40:14.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.275+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a8002860 osd=0 4 2024-03-27T21:40:14.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.275+0000 7f87b5ffb640 10 client.4478.objecter _send_command 1 2024-03-27T21:40:14.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.275+0000 7f87b5ffb640 1 -- 172.21.15.60:0/1397817274 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87a8008420 con 0x7f87a8002f70 2024-03-27T21:40:14.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.275+0000 7f87b7fff640 10 client.4478.objecter ms_handle_connect 0x7f87bc06c590 2024-03-27T21:40:14.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.275+0000 7f87cf7fe640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f87bc06c590 0x7f87bc06ea50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f87c0006a90 tx=0x7f87c0006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:14.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.275+0000 7f87cffff640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f87a8002f70 0x7f87a8005430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:14.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.275+0000 7f87cffff640 10 monclient: get_auth_request con 0x7f87a8002f70 auth_method 0 2024-03-27T21:40:14.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.275+0000 7f87b7fff640 10 client.4478.objecter ms_handle_connect 0x7f87a8002f70 2024-03-27T21:40:14.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.275+0000 7f87cffff640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f87a8002f70 0x7f87a8005430 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:14.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.276+0000 7f87b7fff640 1 -- 172.21.15.60:0/1397817274 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f87a8008420 con 0x7f87a8002f70 2024-03-27T21:40:14.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.276+0000 7f87b7fff640 10 client.4478.objecter ms_dispatch 0x7f87d0000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:14.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.276+0000 7f87b7fff640 10 client.4478.objecter _finish_command 1 = system:0 2024-03-27T21:40:14.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.276+0000 7f87b7fff640 20 client.4478.objecter put_session s=0x7f87a8002860 osd=0 4 2024-03-27T21:40:14.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.276+0000 7f87b7fff640 15 client.4478.objecter _session_command_op_remove 0 1 2024-03-27T21:40:14.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.292+0000 7f3e74495640 10 start_mon_command cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:40:14.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.292+0000 7f3e74495640 10 monclient: _send_command 2 [{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:40:14.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.292+0000 7f3e74495640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:14.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.292+0000 7f3e74495640 1 -- 172.21.15.60:0/339128939 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f3e6c13ebe0 con 0x7f3e6c0b40b0 2024-03-27T21:40:14.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.305+0000 7f87b5ffb640 10 client.4478.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:14.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.305+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign -1 2 2024-03-27T21:40:14.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.305+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a8002860 osd=0 3 2024-03-27T21:40:14.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.305+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a8002860 osd=0 3 2024-03-27T21:40:14.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.305+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a8002860 osd=0 3 2024-03-27T21:40:14.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.305+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_remove -1 2 2024-03-27T21:40:14.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.305+0000 7f87b5ffb640 20 client.4478.objecter get_session s=0x7f87a8002860 osd=0 3 2024-03-27T21:40:14.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.305+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign 0 2 2024-03-27T21:40:14.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.305+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a8002860 osd=0 4 2024-03-27T21:40:14.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.305+0000 7f87b5ffb640 10 client.4478.objecter _send_command 2 2024-03-27T21:40:14.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.305+0000 7f87b5ffb640 1 -- 172.21.15.60:0/1397817274 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f87a8001100 con 0x7f87a8002f70 2024-03-27T21:40:14.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.307+0000 7f87b7fff640 1 -- 172.21.15.60:0/1397817274 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f87a8001100 con 0x7f87a8002f70 2024-03-27T21:40:14.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.307+0000 7f87b7fff640 10 client.4478.objecter ms_dispatch 0x7f87d0000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:14.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.307+0000 7f87b7fff640 10 client.4478.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:14.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.307+0000 7f87b7fff640 20 client.4478.objecter put_session s=0x7f87a8002860 osd=0 4 2024-03-27T21:40:14.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.307+0000 7f87b7fff640 15 client.4478.objecter _session_command_op_remove 0 2 2024-03-27T21:40:14.294 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:14.296 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:14.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.307+0000 7f87b5ffb640 10 client.4478.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:14.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.307+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign -1 3 2024-03-27T21:40:14.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.307+0000 7f87b5ffb640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f87a8005ab0 0x7f87a8005f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:14.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.307+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a8002560 osd=1 3 2024-03-27T21:40:14.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.307+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a8002560 osd=1 3 2024-03-27T21:40:14.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.307+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a8002560 osd=1 3 2024-03-27T21:40:14.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.307+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_remove -1 3 2024-03-27T21:40:14.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.307+0000 7f87b5ffb640 20 client.4478.objecter get_session s=0x7f87a8002560 osd=1 3 2024-03-27T21:40:14.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.307+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign 1 3 2024-03-27T21:40:14.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.307+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a8002560 osd=1 4 2024-03-27T21:40:14.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.307+0000 7f87b5ffb640 10 client.4478.objecter _send_command 3 2024-03-27T21:40:14.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.307+0000 7f87b5ffb640 1 -- 172.21.15.60:0/1397817274 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87a80064a0 con 0x7f87a8005ab0 2024-03-27T21:40:14.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.307+0000 7f87cf7fe640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f87a8005ab0 0x7f87a8005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:14.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.307+0000 7f87cf7fe640 10 monclient: get_auth_request con 0x7f87a8005ab0 auth_method 0 2024-03-27T21:40:14.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.308+0000 7f87cf7fe640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f87a8005ab0 0x7f87a8005f60 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:14.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.308+0000 7f87b7fff640 10 client.4478.objecter ms_handle_connect 0x7f87a8005ab0 2024-03-27T21:40:14.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.308+0000 7f87b7fff640 1 -- 172.21.15.60:0/1397817274 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f87a80064a0 con 0x7f87a8005ab0 2024-03-27T21:40:14.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.308+0000 7f87b7fff640 10 client.4478.objecter ms_dispatch 0x7f87d0000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:14.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.308+0000 7f87b7fff640 10 client.4478.objecter _finish_command 3 = system:0 2024-03-27T21:40:14.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.308+0000 7f87b7fff640 20 client.4478.objecter put_session s=0x7f87a8002560 osd=1 4 2024-03-27T21:40:14.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.308+0000 7f87b7fff640 15 client.4478.objecter _session_command_op_remove 1 3 2024-03-27T21:40:14.301 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:14.301 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:14.310 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:40:14.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.323+0000 7f87b5ffb640 10 client.4478.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:14.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.323+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign -1 4 2024-03-27T21:40:14.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.323+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a8002560 osd=1 3 2024-03-27T21:40:14.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.323+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a8002560 osd=1 3 2024-03-27T21:40:14.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.323+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a8002560 osd=1 3 2024-03-27T21:40:14.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.323+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_remove -1 4 2024-03-27T21:40:14.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.323+0000 7f87b5ffb640 20 client.4478.objecter get_session s=0x7f87a8002560 osd=1 3 2024-03-27T21:40:14.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.323+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign 1 4 2024-03-27T21:40:14.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.323+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a8002560 osd=1 4 2024-03-27T21:40:14.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.323+0000 7f87b5ffb640 10 client.4478.objecter _send_command 4 2024-03-27T21:40:14.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.323+0000 7f87b5ffb640 1 -- 172.21.15.60:0/1397817274 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f87a8006e60 con 0x7f87a8005ab0 2024-03-27T21:40:14.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.325+0000 7f87b7fff640 1 -- 172.21.15.60:0/1397817274 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f87a8006e60 con 0x7f87a8005ab0 2024-03-27T21:40:14.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.325+0000 7f87b7fff640 10 client.4478.objecter ms_dispatch 0x7f87d0000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:14.313 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:14.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.325+0000 7f87b7fff640 10 client.4478.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:14.316 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:14.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.325+0000 7f87b7fff640 20 client.4478.objecter put_session s=0x7f87a8002560 osd=1 4 2024-03-27T21:40:14.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.325+0000 7f87b7fff640 15 client.4478.objecter _session_command_op_remove 1 4 2024-03-27T21:40:14.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.325+0000 7f87b5ffb640 10 client.4478.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:14.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.325+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign -1 5 2024-03-27T21:40:14.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.325+0000 7f87b5ffb640 20 client.4478.objecter _get_session osd=-1 returning homeless 2024-03-27T21:40:14.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.325+0000 7f87b5ffb640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:14.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.325+0000 7f87b5ffb640 10 monclient: _renew_subs 2024-03-27T21:40:14.316 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:40:14.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.325+0000 7f87b5ffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:14.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.325+0000 7f87b5ffb640 1 -- 172.21.15.60:0/1397817274 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=30}) v3 -- 0x7f87a8006fd0 con 0x7f87d013b020 2024-03-27T21:40:14.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.325+0000 7f87b5ffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:14.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.325+0000 7f87b5ffb640 1 -- 172.21.15.60:0/1397817274 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f87a8007a10 con 0x7f87d013b020 2024-03-27T21:40:14.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.325+0000 7f87b7fff640 1 -- 172.21.15.60:0/1397817274 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87c4001200 con 0x7f87d013b020 2024-03-27T21:40:14.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.325+0000 7f87b7fff640 10 monclient: handle_get_version_reply finishing 1 version 29 2024-03-27T21:40:14.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.325+0000 7f87cdffb640 10 client.4478.objecter _check_command_map_dne tid 5 current 29 map_dne_bound 29 2024-03-27T21:40:14.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.325+0000 7f87cdffb640 10 client.4478.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:40:14.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.325+0000 7f87cdffb640 15 client.4478.objecter _session_command_op_remove -1 5 2024-03-27T21:40:14.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.326+0000 7f87b5ffb640 10 client.4478.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:14.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.326+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign -1 6 2024-03-27T21:40:14.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.326+0000 7f87b5ffb640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f87a8006a50 0x7f87a800b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:14.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.326+0000 7f87ceffd640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f87a8006a50 0x7f87a800b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:14.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.326+0000 7f87ceffd640 10 monclient: get_auth_request con 0x7f87a8006a50 auth_method 0 2024-03-27T21:40:14.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.326+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a80068e0 osd=3 3 2024-03-27T21:40:14.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.326+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a80068e0 osd=3 3 2024-03-27T21:40:14.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.326+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a80068e0 osd=3 3 2024-03-27T21:40:14.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.326+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_remove -1 6 2024-03-27T21:40:14.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.326+0000 7f87b5ffb640 20 client.4478.objecter get_session s=0x7f87a80068e0 osd=3 3 2024-03-27T21:40:14.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.326+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign 3 6 2024-03-27T21:40:14.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.326+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a80068e0 osd=3 4 2024-03-27T21:40:14.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.326+0000 7f87b5ffb640 10 client.4478.objecter _send_command 6 2024-03-27T21:40:14.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.326+0000 7f87b7fff640 10 client.4478.objecter ms_handle_connect 0x7f87a8006a50 2024-03-27T21:40:14.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.326+0000 7f87ceffd640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f87a8006a50 0x7f87a800b780 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:14.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.326+0000 7f87b5ffb640 1 -- 172.21.15.60:0/1397817274 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87a8002250 con 0x7f87a8006a50 2024-03-27T21:40:14.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.327+0000 7f87b7fff640 1 -- 172.21.15.60:0/1397817274 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f87a8002250 con 0x7f87a8006a50 2024-03-27T21:40:14.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.327+0000 7f87b7fff640 10 client.4478.objecter ms_dispatch 0x7f87d0000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:40:14.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.327+0000 7f87b7fff640 10 client.4478.objecter _finish_command 6 = system:0 2024-03-27T21:40:14.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.327+0000 7f87b7fff640 20 client.4478.objecter put_session s=0x7f87a80068e0 osd=3 4 2024-03-27T21:40:14.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.327+0000 7f87b7fff640 15 client.4478.objecter _session_command_op_remove 3 6 2024-03-27T21:40:14.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.344+0000 7f87b5ffb640 10 client.4478.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:14.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.345+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign -1 7 2024-03-27T21:40:14.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.345+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a80068e0 osd=3 3 2024-03-27T21:40:14.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.345+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a80068e0 osd=3 3 2024-03-27T21:40:14.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.345+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a80068e0 osd=3 3 2024-03-27T21:40:14.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.345+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_remove -1 7 2024-03-27T21:40:14.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.345+0000 7f87b5ffb640 20 client.4478.objecter get_session s=0x7f87a80068e0 osd=3 3 2024-03-27T21:40:14.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.345+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign 3 7 2024-03-27T21:40:14.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.345+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a80068e0 osd=3 4 2024-03-27T21:40:14.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.345+0000 7f87b5ffb640 10 client.4478.objecter _send_command 7 2024-03-27T21:40:14.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.345+0000 7f87b5ffb640 1 -- 172.21.15.60:0/1397817274 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f87a80015e0 con 0x7f87a8006a50 2024-03-27T21:40:14.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.347+0000 7f87b7fff640 1 -- 172.21.15.60:0/1397817274 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f87a80015e0 con 0x7f87a8006a50 2024-03-27T21:40:14.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.347+0000 7f87b7fff640 10 client.4478.objecter ms_dispatch 0x7f87d0000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:14.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.347+0000 7f87b7fff640 10 client.4478.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:14.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.347+0000 7f87b7fff640 20 client.4478.objecter put_session s=0x7f87a80068e0 osd=3 4 2024-03-27T21:40:14.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.347+0000 7f87b7fff640 15 client.4478.objecter _session_command_op_remove 3 7 2024-03-27T21:40:14.334 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:14.336 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:14.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.347+0000 7f87b5ffb640 10 client.4478.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:14.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.347+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign -1 8 2024-03-27T21:40:14.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.347+0000 7f87b5ffb640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f87a800b2d0 0x7f87a8010f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:14.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.347+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a8001990 osd=4 3 2024-03-27T21:40:14.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.347+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a8001990 osd=4 3 2024-03-27T21:40:14.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.347+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a8001990 osd=4 3 2024-03-27T21:40:14.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.347+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_remove -1 8 2024-03-27T21:40:14.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.347+0000 7f87b5ffb640 20 client.4478.objecter get_session s=0x7f87a8001990 osd=4 3 2024-03-27T21:40:14.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.347+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign 4 8 2024-03-27T21:40:14.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.347+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a8001990 osd=4 4 2024-03-27T21:40:14.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.347+0000 7f87b5ffb640 10 client.4478.objecter _send_command 8 2024-03-27T21:40:14.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.347+0000 7f87b5ffb640 1 -- 172.21.15.60:0/1397817274 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87a8013dd0 con 0x7f87a800b2d0 2024-03-27T21:40:14.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.348+0000 7f87cffff640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f87a800b2d0 0x7f87a8010f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:14.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.348+0000 7f87cffff640 10 monclient: get_auth_request con 0x7f87a800b2d0 auth_method 0 2024-03-27T21:40:14.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.348+0000 7f87b7fff640 10 client.4478.objecter ms_handle_connect 0x7f87a800b2d0 2024-03-27T21:40:14.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.348+0000 7f87cffff640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f87a800b2d0 0x7f87a8010f40 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:14.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.349+0000 7f87b7fff640 1 -- 172.21.15.60:0/1397817274 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f87a8013dd0 con 0x7f87a800b2d0 2024-03-27T21:40:14.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.349+0000 7f87b7fff640 10 client.4478.objecter ms_dispatch 0x7f87d0000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:40:14.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.349+0000 7f87b7fff640 10 client.4478.objecter _finish_command 8 = system:0 2024-03-27T21:40:14.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.349+0000 7f87b7fff640 20 client.4478.objecter put_session s=0x7f87a8001990 osd=4 4 2024-03-27T21:40:14.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.349+0000 7f87b7fff640 15 client.4478.objecter _session_command_op_remove 4 8 2024-03-27T21:40:14.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.364+0000 7f87b5ffb640 10 client.4478.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:14.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.364+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign -1 9 2024-03-27T21:40:14.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.364+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a8001990 osd=4 3 2024-03-27T21:40:14.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.364+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a8001990 osd=4 3 2024-03-27T21:40:14.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.364+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a8001990 osd=4 3 2024-03-27T21:40:14.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.364+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_remove -1 9 2024-03-27T21:40:14.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.364+0000 7f87b5ffb640 20 client.4478.objecter get_session s=0x7f87a8001990 osd=4 3 2024-03-27T21:40:14.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.364+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign 4 9 2024-03-27T21:40:14.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.364+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a8001990 osd=4 4 2024-03-27T21:40:14.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.364+0000 7f87b5ffb640 10 client.4478.objecter _send_command 9 2024-03-27T21:40:14.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.364+0000 7f87b5ffb640 1 -- 172.21.15.60:0/1397817274 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f87a800a840 con 0x7f87a800b2d0 2024-03-27T21:40:14.352 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:14.366+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:14.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.366+0000 7f87b7fff640 1 -- 172.21.15.60:0/1397817274 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f87a800a840 con 0x7f87a800b2d0 2024-03-27T21:40:14.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.366+0000 7f87b7fff640 10 client.4478.objecter ms_dispatch 0x7f87d0000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:14.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.366+0000 7f87b7fff640 10 client.4478.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:14.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.366+0000 7f87b7fff640 20 client.4478.objecter put_session s=0x7f87a8001990 osd=4 4 2024-03-27T21:40:14.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.366+0000 7f87b7fff640 15 client.4478.objecter _session_command_op_remove 4 9 2024-03-27T21:40:14.353 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:14.354 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:14.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.367+0000 7f87b5ffb640 10 client.4478.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:14.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.367+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign -1 10 2024-03-27T21:40:14.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.367+0000 7f87b5ffb640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f87a800c990 0x7f87a8011480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:14.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.367+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a800a600 osd=5 3 2024-03-27T21:40:14.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.367+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a800a600 osd=5 3 2024-03-27T21:40:14.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.367+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a800a600 osd=5 3 2024-03-27T21:40:14.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.367+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_remove -1 10 2024-03-27T21:40:14.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.367+0000 7f87b5ffb640 20 client.4478.objecter get_session s=0x7f87a800a600 osd=5 3 2024-03-27T21:40:14.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.367+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign 5 10 2024-03-27T21:40:14.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.367+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a800a600 osd=5 4 2024-03-27T21:40:14.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.367+0000 7f87b5ffb640 10 client.4478.objecter _send_command 10 2024-03-27T21:40:14.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.367+0000 7f87b5ffb640 1 -- 172.21.15.60:0/1397817274 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87a80176f0 con 0x7f87a800c990 2024-03-27T21:40:14.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.367+0000 7f87cf7fe640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f87a800c990 0x7f87a8011480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:14.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.367+0000 7f87cf7fe640 10 monclient: get_auth_request con 0x7f87a800c990 auth_method 0 2024-03-27T21:40:14.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.367+0000 7f87b7fff640 10 client.4478.objecter ms_handle_connect 0x7f87a800c990 2024-03-27T21:40:14.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.367+0000 7f87cf7fe640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f87a800c990 0x7f87a8011480 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:14.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.368+0000 7f87b7fff640 1 -- 172.21.15.60:0/1397817274 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f87a80176f0 con 0x7f87a800c990 2024-03-27T21:40:14.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.368+0000 7f87b7fff640 10 client.4478.objecter ms_dispatch 0x7f87d0000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:40:14.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.368+0000 7f87b7fff640 10 client.4478.objecter _finish_command 10 = system:0 2024-03-27T21:40:14.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.368+0000 7f87b7fff640 20 client.4478.objecter put_session s=0x7f87a800a600 osd=5 4 2024-03-27T21:40:14.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.368+0000 7f87b7fff640 15 client.4478.objecter _session_command_op_remove 5 10 2024-03-27T21:40:14.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.399+0000 7f87b5ffb640 10 client.4478.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:14.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.399+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign -1 11 2024-03-27T21:40:14.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.399+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a800a600 osd=5 3 2024-03-27T21:40:14.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.399+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a800a600 osd=5 3 2024-03-27T21:40:14.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.399+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a800a600 osd=5 3 2024-03-27T21:40:14.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.399+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_remove -1 11 2024-03-27T21:40:14.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.399+0000 7f87b5ffb640 20 client.4478.objecter get_session s=0x7f87a800a600 osd=5 3 2024-03-27T21:40:14.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.399+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign 5 11 2024-03-27T21:40:14.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.399+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a800a600 osd=5 4 2024-03-27T21:40:14.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.399+0000 7f87b5ffb640 10 client.4478.objecter _send_command 11 2024-03-27T21:40:14.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.399+0000 7f87b5ffb640 1 -- 172.21.15.60:0/1397817274 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f87a80119c0 con 0x7f87a800c990 2024-03-27T21:40:14.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.400+0000 7f87b7fff640 1 -- 172.21.15.60:0/1397817274 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f87a80119c0 con 0x7f87a800c990 2024-03-27T21:40:14.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.400+0000 7f87b7fff640 10 client.4478.objecter ms_dispatch 0x7f87d0000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:14.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.400+0000 7f87b7fff640 10 client.4478.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:14.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.400+0000 7f87b7fff640 20 client.4478.objecter put_session s=0x7f87a800a600 osd=5 4 2024-03-27T21:40:14.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.400+0000 7f87b7fff640 15 client.4478.objecter _session_command_op_remove 5 11 2024-03-27T21:40:14.387 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:14.389 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:14.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.401+0000 7f87b5ffb640 10 client.4478.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:14.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.401+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign -1 12 2024-03-27T21:40:14.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.401+0000 7f87b5ffb640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f87a80120d0 0x7f87a8012550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:14.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.401+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a80119c0 osd=6 3 2024-03-27T21:40:14.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.401+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a80119c0 osd=6 3 2024-03-27T21:40:14.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.401+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a80119c0 osd=6 3 2024-03-27T21:40:14.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.401+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_remove -1 12 2024-03-27T21:40:14.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.401+0000 7f87b5ffb640 20 client.4478.objecter get_session s=0x7f87a80119c0 osd=6 3 2024-03-27T21:40:14.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.401+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign 6 12 2024-03-27T21:40:14.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.401+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a80119c0 osd=6 4 2024-03-27T21:40:14.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.401+0000 7f87b5ffb640 10 client.4478.objecter _send_command 12 2024-03-27T21:40:14.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.401+0000 7f87b5ffb640 1 -- 172.21.15.60:0/1397817274 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87a801a880 con 0x7f87a80120d0 2024-03-27T21:40:14.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.401+0000 7f87ceffd640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f87a80120d0 0x7f87a8012550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:14.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.401+0000 7f87ceffd640 10 monclient: get_auth_request con 0x7f87a80120d0 auth_method 0 2024-03-27T21:40:14.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.401+0000 7f87b7fff640 10 client.4478.objecter ms_handle_connect 0x7f87a80120d0 2024-03-27T21:40:14.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.401+0000 7f87ceffd640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f87a80120d0 0x7f87a8012550 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:14.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.402+0000 7f87b7fff640 1 -- 172.21.15.60:0/1397817274 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f87a801a880 con 0x7f87a80120d0 2024-03-27T21:40:14.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.402+0000 7f87b7fff640 10 client.4478.objecter ms_dispatch 0x7f87d0000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:40:14.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.402+0000 7f87b7fff640 10 client.4478.objecter _finish_command 12 = system:0 2024-03-27T21:40:14.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.402+0000 7f87b7fff640 20 client.4478.objecter put_session s=0x7f87a80119c0 osd=6 4 2024-03-27T21:40:14.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.402+0000 7f87b7fff640 15 client.4478.objecter _session_command_op_remove 6 12 2024-03-27T21:40:14.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.431+0000 7f87b5ffb640 10 client.4478.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:14.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.431+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign -1 13 2024-03-27T21:40:14.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.431+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a80119c0 osd=6 3 2024-03-27T21:40:14.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.431+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a80119c0 osd=6 3 2024-03-27T21:40:14.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.431+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a80119c0 osd=6 3 2024-03-27T21:40:14.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.431+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_remove -1 13 2024-03-27T21:40:14.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.431+0000 7f87b5ffb640 20 client.4478.objecter get_session s=0x7f87a80119c0 osd=6 3 2024-03-27T21:40:14.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.431+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign 6 13 2024-03-27T21:40:14.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.431+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a80119c0 osd=6 4 2024-03-27T21:40:14.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.431+0000 7f87b5ffb640 10 client.4478.objecter _send_command 13 2024-03-27T21:40:14.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.431+0000 7f87b5ffb640 1 -- 172.21.15.60:0/1397817274 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f87a8017ff0 con 0x7f87a80120d0 2024-03-27T21:40:14.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.433+0000 7f87b7fff640 1 -- 172.21.15.60:0/1397817274 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f87a8017ff0 con 0x7f87a80120d0 2024-03-27T21:40:14.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.433+0000 7f87b7fff640 10 client.4478.objecter ms_dispatch 0x7f87d0000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:14.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.433+0000 7f87b7fff640 10 client.4478.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:14.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.433+0000 7f87b7fff640 20 client.4478.objecter put_session s=0x7f87a80119c0 osd=6 4 2024-03-27T21:40:14.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.433+0000 7f87b7fff640 15 client.4478.objecter _session_command_op_remove 6 13 2024-03-27T21:40:14.420 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:14.422 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.433+0000 7f87b5ffb640 10 client.4478.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.433+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign -1 14 2024-03-27T21:40:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.433+0000 7f87b5ffb640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f87a80186d0 0x7f87a8018b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.433+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a8017ff0 osd=7 3 2024-03-27T21:40:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.433+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a8017ff0 osd=7 3 2024-03-27T21:40:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.433+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a8017ff0 osd=7 3 2024-03-27T21:40:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.433+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_remove -1 14 2024-03-27T21:40:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.433+0000 7f87b5ffb640 20 client.4478.objecter get_session s=0x7f87a8017ff0 osd=7 3 2024-03-27T21:40:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.433+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign 7 14 2024-03-27T21:40:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.433+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a8017ff0 osd=7 4 2024-03-27T21:40:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.433+0000 7f87b5ffb640 10 client.4478.objecter _send_command 14 2024-03-27T21:40:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.433+0000 7f87b5ffb640 1 -- 172.21.15.60:0/1397817274 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87a801dbb0 con 0x7f87a80186d0 2024-03-27T21:40:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.434+0000 7f87cffff640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f87a80186d0 0x7f87a8018b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.434+0000 7f87cffff640 10 monclient: get_auth_request con 0x7f87a80186d0 auth_method 0 2024-03-27T21:40:14.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.434+0000 7f87cffff640 1 --2- 172.21.15.60:0/1397817274 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f87a80186d0 0x7f87a8018b30 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:14.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.434+0000 7f87b7fff640 10 client.4478.objecter ms_handle_connect 0x7f87a80186d0 2024-03-27T21:40:14.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.435+0000 7f87b7fff640 1 -- 172.21.15.60:0/1397817274 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f87a801dbb0 con 0x7f87a80186d0 2024-03-27T21:40:14.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.435+0000 7f87b7fff640 10 client.4478.objecter ms_dispatch 0x7f87d0000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:40:14.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.435+0000 7f87b7fff640 10 client.4478.objecter _finish_command 14 = system:0 2024-03-27T21:40:14.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.435+0000 7f87b7fff640 20 client.4478.objecter put_session s=0x7f87a8017ff0 osd=7 4 2024-03-27T21:40:14.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.435+0000 7f87b7fff640 15 client.4478.objecter _session_command_op_remove 7 14 2024-03-27T21:40:14.440 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:14.454+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.465+0000 7f87b5ffb640 10 client.4478.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.466+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign -1 15 2024-03-27T21:40:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.466+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a8017ff0 osd=7 3 2024-03-27T21:40:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.466+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a8017ff0 osd=7 3 2024-03-27T21:40:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.466+0000 7f87b5ffb640 20 client.4478.objecter _get_session s=0x7f87a8017ff0 osd=7 3 2024-03-27T21:40:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.466+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_remove -1 15 2024-03-27T21:40:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.466+0000 7f87b5ffb640 20 client.4478.objecter get_session s=0x7f87a8017ff0 osd=7 3 2024-03-27T21:40:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.466+0000 7f87b5ffb640 15 client.4478.objecter _session_command_op_assign 7 15 2024-03-27T21:40:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.466+0000 7f87b5ffb640 20 client.4478.objecter put_session s=0x7f87a8017ff0 osd=7 4 2024-03-27T21:40:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.466+0000 7f87b5ffb640 10 client.4478.objecter _send_command 15 2024-03-27T21:40:14.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.466+0000 7f87b5ffb640 1 -- 172.21.15.60:0/1397817274 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f87a8019180 con 0x7f87a80186d0 2024-03-27T21:40:14.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.468+0000 7f87b7fff640 1 -- 172.21.15.60:0/1397817274 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f87a8019180 con 0x7f87a80186d0 2024-03-27T21:40:14.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.468+0000 7f87b7fff640 10 client.4478.objecter ms_dispatch 0x7f87d0000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:14.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.468+0000 7f87b7fff640 10 client.4478.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:14.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.468+0000 7f87b7fff640 20 client.4478.objecter put_session s=0x7f87a8017ff0 osd=7 4 2024-03-27T21:40:14.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.468+0000 7f87b7fff640 15 client.4478.objecter _session_command_op_remove 7 15 2024-03-27T21:40:14.455 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:14.455 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:14.459 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:40:14.459 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:40:14.485 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:14.486 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:14.486 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:14.486 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:14.498 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:40:14.541 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:14.556+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:14.642 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:14.656+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:14.648 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:14.649 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:14.649 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:14.649 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:14.649 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:14.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.671+0000 7f3e62ffd640 1 -- 172.21.15.60:0/339128939 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v30) v1 ==== 101+0+0 (secure 0 0 0) 0x7f3e68058030 con 0x7f3e6c0b40b0 2024-03-27T21:40:14.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.671+0000 7f3e62ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:40:14.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.671+0000 7f3e62ffd640 10 monclient: _finish_command 2 = system:0 nodeep-scrub is unset 2024-03-27T21:40:14.658 INFO:teuthology.orchestra.run.smithi060.stderr:nodeep-scrub is unset 2024-03-27T21:40:14.659 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:40:14.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.673+0000 7f3e74495640 10 librados: watch_flush enter 2024-03-27T21:40:14.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.673+0000 7f3e74495640 10 librados: watch_flush exit 2024-03-27T21:40:14.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.673+0000 7f3e74495640 20 client.4540.objecter shutdown clearing up homeless session... 2024-03-27T21:40:14.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.673+0000 7f3e74495640 10 client.4540.objecter successfully canceled tick 2024-03-27T21:40:14.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.673+0000 7f3e74495640 1 -- 172.21.15.60:0/339128939 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3e5006c590 msgr2=0x7f3e5006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:14.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.673+0000 7f3e74495640 1 --2- 172.21.15.60:0/339128939 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3e5006c590 0x7f3e5006ea50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c036000 tx=0x7f3e5c046040 comp rx=0 tx=0).stop 2024-03-27T21:40:14.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.674+0000 7f3e74495640 10 monclient: shutdown 2024-03-27T21:40:14.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.674+0000 7f3e74495640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:14.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.674+0000 7f3e74495640 1 -- 172.21.15.60:0/339128939 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e6c0b40b0 msgr2=0x7f3e6c0aac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:14.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.674+0000 7f3e74495640 1 --2- 172.21.15.60:0/339128939 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e6c0b40b0 0x7f3e6c0aac40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3e6800d800 tx=0x7f3e6800dcc0 comp rx=0 tx=0).stop 2024-03-27T21:40:14.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.674+0000 7f3e74495640 1 -- 172.21.15.60:0/339128939 shutdown_connections 2024-03-27T21:40:14.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.674+0000 7f3e74495640 1 --2- 172.21.15.60:0/339128939 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3e6c1416f0 0x7f3e6c1d6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:14.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.674+0000 7f3e74495640 1 --2- 172.21.15.60:0/339128939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e6c13b020 0x7f3e6c13ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:14.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.674+0000 7f3e74495640 1 --2- 172.21.15.60:0/339128939 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3e5006c590 0x7f3e5006ea50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:14.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.674+0000 7f3e74495640 1 --2- 172.21.15.60:0/339128939 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e6c0b40b0 0x7f3e6c0aac40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:14.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.674+0000 7f3e74495640 1 -- 172.21.15.60:0/339128939 >> 172.21.15.60:0/339128939 conn(0x7f3e6c0608e0 msgr2=0x7f3e6c0b5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:14.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.674+0000 7f3e74495640 1 -- 172.21.15.60:0/339128939 shutdown_connections 2024-03-27T21:40:14.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.674+0000 7f3e74495640 1 -- 172.21.15.60:0/339128939 wait complete. 2024-03-27T21:40:14.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:14.674+0000 7f3e74495640 1 librados: shutdown 2024-03-27T21:40:14.742 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:14.757+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:14.747 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:14.761+0000 7fa1b5c1a640 -1 osd.2 21 log_to_monitors true 2024-03-27T21:40:14.786 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:14.786 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:14.786 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:14.786 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-03-27T21:40:14.786 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.786 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:20 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2024-03-27T21:40:14.786 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:14.786 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.909784199000001, 2024-03-27T21:40:14.786 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015345630000000001, 2024-03-27T21:40:14.787 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:14.787 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:14.787 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:14.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.787 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:14.787 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:14.787 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.787 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:14.787 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:14.787 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.788 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:14.788 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888562+0000", 2024-03-27T21:40:14.788 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:14.788 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.788 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.788 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:14.788 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888735+0000", 2024-03-27T21:40:14.788 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017254799999999999 2024-03-27T21:40:14.788 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.788 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.788 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:14.788 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888747+0000", 2024-03-27T21:40:14.788 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2469999999999999e-05 2024-03-27T21:40:14.788 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.789 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.789 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:14.789 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888761+0000", 2024-03-27T21:40:14.789 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3254e-05 2024-03-27T21:40:14.789 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.789 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.789 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:14.789 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888804+0000", 2024-03-27T21:40:14.789 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3129e-05 2024-03-27T21:40:14.789 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.789 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.789 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:14.789 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888820+0000", 2024-03-27T21:40:14.789 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6426000000000002e-05 2024-03-27T21:40:14.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.790 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.790 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:14.790 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890060+0000", 2024-03-27T21:40:14.790 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012398050000000001 2024-03-27T21:40:14.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.790 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.790 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:14.790 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890098+0000", 2024-03-27T21:40:14.790 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7795e-05 2024-03-27T21:40:14.790 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.791 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:14.791 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.791 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.791 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.791 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2024-03-27T21:40:14.791 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:14.791 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.908959799000002, 2024-03-27T21:40:14.791 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001069555, 2024-03-27T21:40:14.791 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:14.791 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:14.791 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:14.792 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.792 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:14.792 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:14.792 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.792 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.792 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.792 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:14.792 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:14.792 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.792 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.792 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.792 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:14.792 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889387+0000", 2024-03-27T21:40:14.792 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:14.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.793 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.793 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:14.793 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889455+0000", 2024-03-27T21:40:14.793 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7930999999999997e-05 2024-03-27T21:40:14.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.793 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.793 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:14.793 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889470+0000", 2024-03-27T21:40:14.793 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5554e-05 2024-03-27T21:40:14.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.793 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.793 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:14.793 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889476+0000", 2024-03-27T21:40:14.794 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0619999999999997e-06 2024-03-27T21:40:14.794 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.794 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.794 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:14.794 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889513+0000", 2024-03-27T21:40:14.794 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6606999999999997e-05 2024-03-27T21:40:14.794 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.794 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.794 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:14.794 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889528+0000", 2024-03-27T21:40:14.794 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4982e-05 2024-03-27T21:40:14.794 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.794 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.794 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:14.795 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890421+0000", 2024-03-27T21:40:14.795 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00089293700000000001 2024-03-27T21:40:14.795 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.795 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.795 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:14.795 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890457+0000", 2024-03-27T21:40:14.795 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6260000000000002e-05 2024-03-27T21:40:14.795 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.795 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:14.795 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.795 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.795 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.795 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:38 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'24, mlcod=17'22)", 2024-03-27T21:40:14.795 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:14.796 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.891723833, 2024-03-27T21:40:14.796 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016182080000000001, 2024-03-27T21:40:14.796 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:14.796 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:14.796 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:14.796 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.796 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:14.796 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:14.796 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.796 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.796 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.797 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:14.797 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:14.797 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.797 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.797 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.797 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:14.797 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906623+0000", 2024-03-27T21:40:14.797 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:14.797 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.797 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.797 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:14.797 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906687+0000", 2024-03-27T21:40:14.797 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3706000000000003e-05 2024-03-27T21:40:14.797 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.798 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.798 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:14.798 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906698+0000", 2024-03-27T21:40:14.798 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1752e-05 2024-03-27T21:40:14.798 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.798 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.798 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:14.798 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906704+0000", 2024-03-27T21:40:14.798 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3940000000000004e-06 2024-03-27T21:40:14.798 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.798 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.798 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:14.798 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906741+0000", 2024-03-27T21:40:14.799 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6944999999999999e-05 2024-03-27T21:40:14.799 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.799 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.799 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:14.799 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906760+0000", 2024-03-27T21:40:14.799 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9293e-05 2024-03-27T21:40:14.799 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.799 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.799 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:14.799 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908211+0000", 2024-03-27T21:40:14.799 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014511350000000001 2024-03-27T21:40:14.799 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.799 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.799 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:14.799 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908242+0000", 2024-03-27T21:40:14.800 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0712e-05 2024-03-27T21:40:14.800 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.800 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:14.800 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.800 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.800 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.800 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:39 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'25, mlcod=17'22)", 2024-03-27T21:40:14.800 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:14.800 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.890878917999999, 2024-03-27T21:40:14.800 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012060199999999999, 2024-03-27T21:40:14.800 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:14.801 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:14.801 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:14.801 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.801 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:14.801 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:14.801 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.801 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.801 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.801 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:14.801 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:14.802 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.802 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.802 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.802 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:14.802 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:14.802 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:14.802 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.802 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.802 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:14.802 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907533+0000", 2024-03-27T21:40:14.802 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5675999999999999e-05 2024-03-27T21:40:14.802 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.802 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.802 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:14.803 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907545+0000", 2024-03-27T21:40:14.803 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.115e-05 2024-03-27T21:40:14.803 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.803 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.803 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:14.803 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907550+0000", 2024-03-27T21:40:14.803 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8320000000000002e-06 2024-03-27T21:40:14.803 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.803 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.803 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:14.803 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907592+0000", 2024-03-27T21:40:14.803 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1309e-05 2024-03-27T21:40:14.803 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.804 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.804 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:14.804 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907611+0000", 2024-03-27T21:40:14.804 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9009e-05 2024-03-27T21:40:14.804 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.804 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.804 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:14.804 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908638+0000", 2024-03-27T21:40:14.804 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001027533 2024-03-27T21:40:14.804 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.804 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.804 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:14.804 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908675+0000", 2024-03-27T21:40:14.804 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6329999999999999e-05 2024-03-27T21:40:14.805 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.805 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:14.805 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.805 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.805 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.805 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:57 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'35, mlcod=17'33)", 2024-03-27T21:40:14.805 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:14.805 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.876794486000001, 2024-03-27T21:40:14.805 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001346578, 2024-03-27T21:40:14.805 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:14.805 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:14.805 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:14.805 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.805 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:14.806 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:14.806 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.806 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.806 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.806 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:14.806 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:14.806 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.806 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.806 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.806 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:14.807 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921552+0000", 2024-03-27T21:40:14.807 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:14.807 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.807 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.807 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:14.807 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921615+0000", 2024-03-27T21:40:14.807 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2787999999999996e-05 2024-03-27T21:40:14.807 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.807 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.807 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:14.807 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921626+0000", 2024-03-27T21:40:14.807 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0888e-05 2024-03-27T21:40:14.807 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.808 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.808 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:14.808 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921635+0000", 2024-03-27T21:40:14.808 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4639999999999995e-06 2024-03-27T21:40:14.808 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.808 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.808 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:14.808 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921673+0000", 2024-03-27T21:40:14.808 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7891e-05 2024-03-27T21:40:14.808 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.808 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.808 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:14.808 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921694+0000", 2024-03-27T21:40:14.808 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0426000000000001e-05 2024-03-27T21:40:14.809 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.809 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.809 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:14.809 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922865+0000", 2024-03-27T21:40:14.809 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011709509999999999 2024-03-27T21:40:14.809 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.809 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.809 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:14.809 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922900+0000", 2024-03-27T21:40:14.809 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4819e-05 2024-03-27T21:40:14.809 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.809 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:14.809 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.810 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.810 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.810 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:71 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'45, mlcod=17'43)", 2024-03-27T21:40:14.810 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:14.810 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.861495773999998, 2024-03-27T21:40:14.810 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001460907, 2024-03-27T21:40:14.810 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:14.810 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:14.810 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:14.810 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.810 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:14.810 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:14.810 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.810 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.811 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.811 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:14.811 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:14.811 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.811 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.811 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.811 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:14.811 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936851+0000", 2024-03-27T21:40:14.811 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:14.811 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.811 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.811 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:14.811 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936903+0000", 2024-03-27T21:40:14.811 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2302999999999997e-05 2024-03-27T21:40:14.812 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.812 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.812 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:14.812 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936915+0000", 2024-03-27T21:40:14.812 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1967000000000001e-05 2024-03-27T21:40:14.812 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.812 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.812 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:14.812 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936923+0000", 2024-03-27T21:40:14.812 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9370000000000004e-06 2024-03-27T21:40:14.812 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.812 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.812 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:14.813 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936970+0000", 2024-03-27T21:40:14.813 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6987000000000003e-05 2024-03-27T21:40:14.813 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.813 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.813 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:14.813 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937002+0000", 2024-03-27T21:40:14.813 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2410999999999999e-05 2024-03-27T21:40:14.813 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.813 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.813 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:14.813 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938279+0000", 2024-03-27T21:40:14.813 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001276552 2024-03-27T21:40:14.813 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.813 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.814 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:14.814 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938313+0000", 2024-03-27T21:40:14.814 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3526999999999999e-05 2024-03-27T21:40:14.814 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.814 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:14.814 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.814 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.814 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.814 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:72 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'46, mlcod=17'43)", 2024-03-27T21:40:14.814 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:14.814 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.860858211999997, 2024-03-27T21:40:14.814 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014239490000000001, 2024-03-27T21:40:14.814 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:14.814 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:14.814 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:14.815 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.815 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:14.815 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:14.815 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.815 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.815 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.815 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:14.815 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:14.815 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.815 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.815 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.815 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:14.815 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937488+0000", 2024-03-27T21:40:14.815 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:14.816 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.816 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.816 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:14.816 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937547+0000", 2024-03-27T21:40:14.816 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8196999999999999e-05 2024-03-27T21:40:14.816 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.816 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.816 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:14.816 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937559+0000", 2024-03-27T21:40:14.816 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1928e-05 2024-03-27T21:40:14.816 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.816 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.816 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:14.816 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937564+0000", 2024-03-27T21:40:14.817 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4759999999999997e-06 2024-03-27T21:40:14.817 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.817 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.817 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:14.817 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937621+0000", 2024-03-27T21:40:14.817 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6712000000000002e-05 2024-03-27T21:40:14.817 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.817 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.817 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:14.817 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937640+0000", 2024-03-27T21:40:14.817 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9517e-05 2024-03-27T21:40:14.817 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.817 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.817 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:14.818 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938883+0000", 2024-03-27T21:40:14.818 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001242567 2024-03-27T21:40:14.818 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.818 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.818 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:14.818 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938913+0000", 2024-03-27T21:40:14.818 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0287e-05 2024-03-27T21:40:14.818 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.818 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:14.818 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.818 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.818 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.818 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:73 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'47, mlcod=17'43)", 2024-03-27T21:40:14.818 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:14.819 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.860182117999997, 2024-03-27T21:40:14.819 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010746239999999999, 2024-03-27T21:40:14.819 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:14.819 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:14.819 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:14.819 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.819 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:14.819 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:14.819 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.819 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.819 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.819 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:14.819 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:14.819 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.820 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.820 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:14.820 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:14.820 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:14.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.820 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.820 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:14.820 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938217+0000", 2024-03-27T21:40:14.820 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2743e-05 2024-03-27T21:40:14.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.820 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.820 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:14.820 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938229+0000", 2024-03-27T21:40:14.821 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1956999999999999e-05 2024-03-27T21:40:14.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.821 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:14.821 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938235+0000", 2024-03-27T21:40:14.821 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4589999999999997e-06 2024-03-27T21:40:14.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.821 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:14.821 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938341+0000", 2024-03-27T21:40:14.821 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000106054 2024-03-27T21:40:14.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.821 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:14.822 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938369+0000", 2024-03-27T21:40:14.822 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8291000000000001e-05 2024-03-27T21:40:14.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.822 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.822 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:14.822 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939209+0000", 2024-03-27T21:40:14.822 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084045400000000003 2024-03-27T21:40:14.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.822 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.822 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:14.822 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939240+0000", 2024-03-27T21:40:14.822 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0429999999999998e-05 2024-03-27T21:40:14.822 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.822 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:14.823 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.823 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.823 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:75 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'49, mlcod=17'47)", 2024-03-27T21:40:14.823 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:14.823 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.853744470999999, 2024-03-27T21:40:14.823 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001017001, 2024-03-27T21:40:14.823 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:14.823 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:14.823 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:14.823 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.823 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:14.823 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:14.823 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.824 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.824 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.824 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:14.824 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:14.824 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.824 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.824 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.824 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:14.824 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944601+0000", 2024-03-27T21:40:14.824 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999986 2024-03-27T21:40:14.824 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.825 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.825 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:14.825 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944614+0000", 2024-03-27T21:40:14.825 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2496e-05 2024-03-27T21:40:14.825 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.825 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.825 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:14.825 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944640+0000", 2024-03-27T21:40:14.825 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5865999999999999e-05 2024-03-27T21:40:14.825 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.825 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.825 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:14.825 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944651+0000", 2024-03-27T21:40:14.825 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1797000000000001e-05 2024-03-27T21:40:14.826 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.826 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.826 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:14.826 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944762+0000", 2024-03-27T21:40:14.826 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110412 2024-03-27T21:40:14.826 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.826 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.826 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:14.826 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944820+0000", 2024-03-27T21:40:14.826 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7649999999999999e-05 2024-03-27T21:40:14.826 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.826 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.826 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:14.826 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945585+0000", 2024-03-27T21:40:14.827 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076562600000000005 2024-03-27T21:40:14.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.827 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.827 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:14.827 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945620+0000", 2024-03-27T21:40:14.827 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4764000000000001e-05 2024-03-27T21:40:14.827 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.827 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:14.827 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.827 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.827 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:86 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'50)", 2024-03-27T21:40:14.827 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:14.827 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.845982984000003, 2024-03-27T21:40:14.827 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011392380000000001, 2024-03-27T21:40:14.828 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:14.828 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:14.828 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:14.828 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.828 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:14.828 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:14.828 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.828 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.828 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:14.828 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:14.828 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.828 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.829 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:14.829 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:14.829 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:14.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.829 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.829 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:14.829 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952373+0000", 2024-03-27T21:40:14.829 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9220000000000003e-06 2024-03-27T21:40:14.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.829 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.829 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:14.829 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952381+0000", 2024-03-27T21:40:14.829 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3070000000000003e-06 2024-03-27T21:40:14.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.829 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.830 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:14.830 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952385+0000", 2024-03-27T21:40:14.830 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.121e-06 2024-03-27T21:40:14.830 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.830 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.830 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:14.830 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952419+0000", 2024-03-27T21:40:14.830 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4082999999999999e-05 2024-03-27T21:40:14.830 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.830 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.830 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:14.831 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952440+0000", 2024-03-27T21:40:14.831 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0837999999999999e-05 2024-03-27T21:40:14.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.831 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:14.831 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953472+0000", 2024-03-27T21:40:14.831 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001032119 2024-03-27T21:40:14.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.831 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:14.831 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953504+0000", 2024-03-27T21:40:14.831 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1250000000000001e-05 2024-03-27T21:40:14.831 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.831 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:14.832 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.832 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.832 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:87 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-27T21:40:14.832 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:14.832 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.845615727000002, 2024-03-27T21:40:14.832 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00135519, 2024-03-27T21:40:14.832 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:14.832 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:14.832 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:14.832 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.832 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:14.832 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:14.832 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.833 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.833 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:14.833 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:14.833 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.833 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.833 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:14.833 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952731+0000", 2024-03-27T21:40:14.833 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:14.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.833 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.833 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:14.833 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952781+0000", 2024-03-27T21:40:14.833 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0161000000000003e-05 2024-03-27T21:40:14.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.834 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.834 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:14.834 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952794+0000", 2024-03-27T21:40:14.834 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2853e-05 2024-03-27T21:40:14.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.834 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.834 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:14.834 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952799+0000", 2024-03-27T21:40:14.834 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5560000000000001e-06 2024-03-27T21:40:14.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.834 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.834 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:14.835 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952832+0000", 2024-03-27T21:40:14.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3834999999999997e-05 2024-03-27T21:40:14.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.835 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:14.835 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952851+0000", 2024-03-27T21:40:14.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8289999999999999e-05 2024-03-27T21:40:14.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.835 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:14.835 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954055+0000", 2024-03-27T21:40:14.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012037949999999999 2024-03-27T21:40:14.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.836 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:14.836 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954087+0000", 2024-03-27T21:40:14.836 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2292999999999997e-05 2024-03-27T21:40:14.836 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.836 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:14.836 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.836 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.836 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.836 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:89 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'59, mlcod=17'57)", 2024-03-27T21:40:14.836 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:14.836 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.843341506000002, 2024-03-27T21:40:14.836 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00138893, 2024-03-27T21:40:14.836 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:14.836 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:14.837 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:14.837 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.837 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:14.837 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:14.837 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.837 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.837 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.837 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:14.837 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:14.837 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.837 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.837 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.837 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:14.837 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955005+0000", 2024-03-27T21:40:14.838 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:14.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.838 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:14.838 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955071+0000", 2024-03-27T21:40:14.838 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5290999999999995e-05 2024-03-27T21:40:14.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.838 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:14.838 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955081+0000", 2024-03-27T21:40:14.838 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0572e-05 2024-03-27T21:40:14.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.838 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:14.839 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955086+0000", 2024-03-27T21:40:14.839 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8380000000000001e-06 2024-03-27T21:40:14.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.839 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.839 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:14.839 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955122+0000", 2024-03-27T21:40:14.839 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5698000000000003e-05 2024-03-27T21:40:14.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.839 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.839 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:14.839 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955140+0000", 2024-03-27T21:40:14.839 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7900000000000001e-05 2024-03-27T21:40:14.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.839 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.839 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:14.840 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956366+0000", 2024-03-27T21:40:14.840 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001226471 2024-03-27T21:40:14.840 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.840 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.840 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:14.840 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956395+0000", 2024-03-27T21:40:14.840 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8782e-05 2024-03-27T21:40:14.840 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.840 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:14.840 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.840 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.840 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.840 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:90 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'60, mlcod=17'57)", 2024-03-27T21:40:14.840 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:14.841 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.842656929, 2024-03-27T21:40:14.841 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010778859999999999, 2024-03-27T21:40:14.841 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:14.841 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:14.841 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:14.841 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.841 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:14.841 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:14.841 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.841 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.841 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.841 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:14.841 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:14.841 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.841 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.842 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.842 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:14.842 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:14.842 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:14.842 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.842 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.842 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:14.842 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955747+0000", 2024-03-27T21:40:14.842 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7130999999999999e-05 2024-03-27T21:40:14.842 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.842 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.842 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:14.842 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955756+0000", 2024-03-27T21:40:14.842 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2750000000000001e-06 2024-03-27T21:40:14.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.843 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.843 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:14.843 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955761+0000", 2024-03-27T21:40:14.843 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3900000000000003e-06 2024-03-27T21:40:14.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.843 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.843 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:14.843 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955797+0000", 2024-03-27T21:40:14.843 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6650000000000003e-05 2024-03-27T21:40:14.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.843 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.843 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:14.843 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955816+0000", 2024-03-27T21:40:14.844 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8195000000000001e-05 2024-03-27T21:40:14.844 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.844 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.844 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:14.844 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956739+0000", 2024-03-27T21:40:14.844 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00092369700000000002 2024-03-27T21:40:14.844 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.844 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.844 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:14.844 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956768+0000", 2024-03-27T21:40:14.844 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9048e-05 2024-03-27T21:40:14.844 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.844 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:14.844 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.844 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.845 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.845 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:118 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'80, mlcod=17'78)", 2024-03-27T21:40:14.845 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:14.845 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.709666106, 2024-03-27T21:40:14.845 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014200669999999999, 2024-03-27T21:40:14.845 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:14.845 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:14.845 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:14.845 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.845 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:14.845 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:14.845 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.845 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.845 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088756+0000", 2024-03-27T21:40:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4902999999999998e-05 2024-03-27T21:40:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088764+0000", 2024-03-27T21:40:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8169999999999999e-06 2024-03-27T21:40:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088767+0000", 2024-03-27T21:40:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8310000000000003e-06 2024-03-27T21:40:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088798+0000", 2024-03-27T21:40:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0800999999999999e-05 2024-03-27T21:40:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088817+0000", 2024-03-27T21:40:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8467000000000001e-05 2024-03-27T21:40:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090074+0000", 2024-03-27T21:40:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012574089999999999 2024-03-27T21:40:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090101+0000", 2024-03-27T21:40:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7223e-05 2024-03-27T21:40:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:119 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'81, mlcod=17'78)", 2024-03-27T21:40:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.708931311000001, 2024-03-27T21:40:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014990870000000001, 2024-03-27T21:40:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:14.850 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.850 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.850 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:14.850 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:14.850 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.850 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.850 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:14.850 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:14.850 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:14.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.850 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089491+0000", 2024-03-27T21:40:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4805e-05 2024-03-27T21:40:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089498+0000", 2024-03-27T21:40:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0650000000000001e-06 2024-03-27T21:40:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089501+0000", 2024-03-27T21:40:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.54e-06 2024-03-27T21:40:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089533+0000", 2024-03-27T21:40:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1838999999999998e-05 2024-03-27T21:40:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089550+0000", 2024-03-27T21:40:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7465999999999998e-05 2024-03-27T21:40:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090890+0000", 2024-03-27T21:40:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013397979999999999 2024-03-27T21:40:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090915+0000", 2024-03-27T21:40:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4905e-05 2024-03-27T21:40:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:120 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'82, mlcod=17'78)", 2024-03-27T21:40:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.708131430999998, 2024-03-27T21:40:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001527093, 2024-03-27T21:40:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090267+0000", 2024-03-27T21:40:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1477000000000002e-05 2024-03-27T21:40:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090273+0000", 2024-03-27T21:40:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9519999999999999e-06 2024-03-27T21:40:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090276+0000", 2024-03-27T21:40:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.219e-06 2024-03-27T21:40:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090304+0000", 2024-03-27T21:40:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8073999999999999e-05 2024-03-27T21:40:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090316+0000", 2024-03-27T21:40:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2088e-05 2024-03-27T21:40:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091717+0000", 2024-03-27T21:40:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014003889999999999 2024-03-27T21:40:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091743+0000", 2024-03-27T21:40:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6234e-05 2024-03-27T21:40:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:121 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'83, mlcod=17'78)", 2024-03-27T21:40:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.707321065999999, 2024-03-27T21:40:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010226899999999999, 2024-03-27T21:40:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091142+0000", 2024-03-27T21:40:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011608300000000001 2024-03-27T21:40:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091151+0000", 2024-03-27T21:40:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2350000000000005e-06 2024-03-27T21:40:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091156+0000", 2024-03-27T21:40:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5419999999999999e-06 2024-03-27T21:40:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091191+0000", 2024-03-27T21:40:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5453000000000001e-05 2024-03-27T21:40:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091208+0000", 2024-03-27T21:40:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7119999999999999e-05 2024-03-27T21:40:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092024+0000", 2024-03-27T21:40:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000815539 2024-03-27T21:40:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092049+0000", 2024-03-27T21:40:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5171e-05 2024-03-27T21:40:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.862 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:14.862 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.862 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.862 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.862 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:142 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'100, mlcod=18'98)", 2024-03-27T21:40:14.862 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:14.862 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.292314112, 2024-03-27T21:40:14.862 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001198778, 2024-03-27T21:40:14.862 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:14.862 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:14.862 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:14.862 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.862 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:14.862 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:14.863 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.863 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.863 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:14.863 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:14.863 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.863 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.863 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:14.863 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:14.863 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:14.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.863 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.864 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:14.864 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506101+0000", 2024-03-27T21:40:14.864 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8202999999999997e-05 2024-03-27T21:40:14.864 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.864 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.864 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:14.864 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506109+0000", 2024-03-27T21:40:14.864 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3059999999999998e-06 2024-03-27T21:40:14.864 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.864 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.864 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:14.864 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506113+0000", 2024-03-27T21:40:14.864 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0119999999999999e-06 2024-03-27T21:40:14.864 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.865 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.865 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:14.865 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506137+0000", 2024-03-27T21:40:14.865 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4043999999999999e-05 2024-03-27T21:40:14.865 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.865 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.865 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:14.865 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506148+0000", 2024-03-27T21:40:14.865 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0446e-05 2024-03-27T21:40:14.865 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.865 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.865 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:14.865 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507213+0000", 2024-03-27T21:40:14.865 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001065215 2024-03-27T21:40:14.866 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.866 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.866 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:14.866 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507232+0000", 2024-03-27T21:40:14.866 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9038e-05 2024-03-27T21:40:14.866 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.866 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:14.866 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.866 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.866 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.866 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:143 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'101, mlcod=18'98)", 2024-03-27T21:40:14.866 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:14.866 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.291701492999998, 2024-03-27T21:40:14.866 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001357488, 2024-03-27T21:40:14.867 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:14.867 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:14.867 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:14.867 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.867 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:14.867 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:14.867 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.867 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.867 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.867 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:14.867 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:14.867 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.867 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.867 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.868 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:14.868 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:14.868 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:14.868 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.868 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.868 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:14.868 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506723+0000", 2024-03-27T21:40:14.868 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7404000000000006e-05 2024-03-27T21:40:14.868 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.868 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.868 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:14.868 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506730+0000", 2024-03-27T21:40:14.868 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5749999999999997e-06 2024-03-27T21:40:14.868 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.869 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.869 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:14.869 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506733+0000", 2024-03-27T21:40:14.869 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7979999999999999e-06 2024-03-27T21:40:14.869 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.869 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.869 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:14.869 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506758+0000", 2024-03-27T21:40:14.869 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4179e-05 2024-03-27T21:40:14.869 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.869 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.869 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:14.869 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506768+0000", 2024-03-27T21:40:14.869 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0740000000000001e-05 2024-03-27T21:40:14.870 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.870 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.870 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:14.870 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507978+0000", 2024-03-27T21:40:14.870 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012093900000000001 2024-03-27T21:40:14.870 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.870 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.870 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:14.870 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508003+0000", 2024-03-27T21:40:14.870 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5667e-05 2024-03-27T21:40:14.870 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.870 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:14.870 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.870 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.871 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.871 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:144 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'102, mlcod=18'98)", 2024-03-27T21:40:14.871 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:14.871 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.291042595999997, 2024-03-27T21:40:14.871 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013453390000000001, 2024-03-27T21:40:14.871 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:14.871 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:14.871 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:14.871 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.871 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:14.871 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:14.871 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.871 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.871 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.872 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:14.872 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:14.872 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:14.872 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.872 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.872 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:14.872 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507304+0000", 2024-03-27T21:40:14.872 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:14.872 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.872 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.872 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:14.872 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507397+0000", 2024-03-27T21:40:14.872 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2995e-05 2024-03-27T21:40:14.873 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.873 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.873 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:14.873 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507403+0000", 2024-03-27T21:40:14.873 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8640000000000001e-06 2024-03-27T21:40:14.873 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.873 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.873 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:14.873 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507407+0000", 2024-03-27T21:40:14.873 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5710000000000002e-06 2024-03-27T21:40:14.873 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.873 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.873 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:14.873 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507428+0000", 2024-03-27T21:40:14.874 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1192e-05 2024-03-27T21:40:14.874 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.874 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.874 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:14.874 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507438+0000", 2024-03-27T21:40:14.874 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8309999999999996e-06 2024-03-27T21:40:14.874 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.874 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.874 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:14.874 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508624+0000", 2024-03-27T21:40:14.874 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011861210000000001 2024-03-27T21:40:14.874 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:14.874 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:14.874 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:14.875 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508650+0000", 2024-03-27T21:40:14.875 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6118e-05 2024-03-27T21:40:14.875 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.875 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:14.875 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.875 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:14.875 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:14.875 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:14.876 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:40:14.877 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:14.891+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:14.978 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:14.991+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:15.044 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:15.045 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:15.045 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:15.045 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:15.057 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:40:15.099 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:15.113+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:15.179 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:15.193+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:15.205 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:15.206 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:15.206 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:15.206 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:15.206 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:15.214 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:40:15.280 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:15.294+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:15.362 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:15.363 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:15.363 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:15.363 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:40:15.363 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:15.372 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:40:15.381 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:15.396+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:15.479 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:15.479 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:15.479 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:15.479 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:15.483 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:15.497+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:15.488 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:40:15.587 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:15.601+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:15.595 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:15.596 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:15.596 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:15.596 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:15.596 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:15.608 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:40:15.688 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:15.703+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:15.764 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:15.765 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:15.765 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:15.765 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:40:15.765 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:15.775 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:40:15.792 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:15.806+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:15.804 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-27T21:40:15.892 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:15.892 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:15.892 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:15.892 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:15.894 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:15.908+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:15.901 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:40:15.950 INFO:teuthology.orchestra.run.smithi060.stderr:no valid command found; 10 closest matches: 2024-03-27T21:40:15.950 INFO:teuthology.orchestra.run.smithi060.stderr:0 2024-03-27T21:40:15.950 INFO:teuthology.orchestra.run.smithi060.stderr:1 2024-03-27T21:40:15.951 INFO:teuthology.orchestra.run.smithi060.stderr:2 2024-03-27T21:40:15.951 INFO:teuthology.orchestra.run.smithi060.stderr:abort 2024-03-27T21:40:15.951 INFO:teuthology.orchestra.run.smithi060.stderr:assert 2024-03-27T21:40:15.951 INFO:teuthology.orchestra.run.smithi060.stderr:bluefs debug_inject_read_zeros 2024-03-27T21:40:15.951 INFO:teuthology.orchestra.run.smithi060.stderr:bluefs files list 2024-03-27T21:40:15.951 INFO:teuthology.orchestra.run.smithi060.stderr:bluefs stats 2024-03-27T21:40:15.951 INFO:teuthology.orchestra.run.smithi060.stderr:bluestore allocator dump block 2024-03-27T21:40:15.951 INFO:teuthology.orchestra.run.smithi060.stderr:bluestore allocator fragmentation block 2024-03-27T21:40:15.951 INFO:teuthology.orchestra.run.smithi060.stderr:admin_socket: invalid command 2024-03-27T21:40:15.954 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T21:40:15.954 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-03-27T21:40:15.995 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:16.008+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:16.046 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:16.047 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:16.047 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:16.047 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:16.047 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:16.055 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:40:16.098 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:16.113+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:16.160 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:16.160 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:16.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:16.160 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:16.160 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:16.168 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:40:16.210 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:16.224+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:16.300 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:16.314+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:16.315 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:16.315 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:16.315 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:16.315 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:16.324 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:40:16.407 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:16.422+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:16.460 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:40:16.469 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:16.469 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:16.469 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:16.469 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:16.469 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:16.478 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:40:16.519 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:16.533+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:16.604 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:16.618+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:16.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.626+0000 7f14769d2640 10 monclient: get_monmap_and_config 2024-03-27T21:40:16.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.626+0000 7f14769d2640 10 monclient: build_initial_monmap 2024-03-27T21:40:16.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.626+0000 7f14769d2640 10 monclient: monmap: 2024-03-27T21:40:16.613 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:16.613 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:16.613 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:16.627783+0000 2024-03-27T21:40:16.614 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:16.627783+0000 2024-03-27T21:40:16.614 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:16.614 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:16.614 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:16.614 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:16.614 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:16.614 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:16.614 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:16.614 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:16.627 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:16.628 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:16.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:16.628 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:16.628 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:16.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.627+0000 7f14769d2640 1 Processor -- start 2024-03-27T21:40:16.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.627+0000 7f14769d2640 1 -- start start 2024-03-27T21:40:16.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.628+0000 7f14769d2640 10 monclient: init 2024-03-27T21:40:16.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f14769d2640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:16.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f14769d2640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-03-27T21:40:16.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f14769d2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f14700b40b0 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:16.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f14769d2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f147013b020 0x7f147013b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:16.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f14769d2640 10 monclient(hunting): picked mon.noname-a con 0x7f147013b020 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:16.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f14769d2640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f14701416f0 0x7f1470141b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:16.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f14769d2640 10 monclient(hunting): picked mon.noname-c con 0x7f14701416f0 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:16.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f14769d2640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14700b3830 con 0x7f14700b40b0 2024-03-27T21:40:16.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f14769d2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:16.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f14769d2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14700b4490 con 0x7f147013b020 2024-03-27T21:40:16.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f14769d2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:16.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f14769d2640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14700aa520 con 0x7f14701416f0 2024-03-27T21:40:16.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f14769d2640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:16.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f14769d2640 10 monclient(hunting): authenticate will time out at 857.850385s 2024-03-27T21:40:16.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f1474f6f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f14701416f0 0x7f1470141b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:16.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f146ffff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f14700b40b0 0x7f14700a8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.60:59356/0 (socket says 172.21.15.60:59356) 2024-03-27T21:40:16.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f146ffff640 1 -- 172.21.15.60:0/4225391701 learned_addr learned my addr 172.21.15.60:0/4225391701 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:40:16.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f146f7fe640 1 --2- 172.21.15.60:0/4225391701 >> v2:172.21.15.60:3300/0 conn(0x7f147013b020 0x7f147013b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:16.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f1474f6f640 10 monclient(hunting): get_auth_request con 0x7f14701416f0 auth_method 0 2024-03-27T21:40:16.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f1474f6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:16.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f1474f6f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:16.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f1474f6f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:16.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f146f7fe640 10 monclient(hunting): get_auth_request con 0x7f147013b020 auth_method 0 2024-03-27T21:40:16.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f146f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:16.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f146f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:16.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f146f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:16.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f1474f6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:16.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f1474f6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:16.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f1474f6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:16.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f146f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:16.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f146f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:16.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f146f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:16.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f1474f6f640 10 monclient(hunting): handle_auth_done global_id 4549 payload 274 2024-03-27T21:40:16.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.629+0000 7f1474f6f640 1 -- 172.21.15.60:0/4225391701 >> v2:172.21.15.60:3300/0 conn(0x7f147013b020 msgr2=0x7f147013b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:16.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.630+0000 7f1474f6f640 1 --2- 172.21.15.60:0/4225391701 >> v2:172.21.15.60:3300/0 conn(0x7f147013b020 0x7f147013b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:16.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.630+0000 7f1474f6f640 1 -- 172.21.15.60:0/4225391701 >> v1:172.21.15.191:6789/0 conn(0x7f14700b40b0 legacy=0x7f14700a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:16.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.630+0000 7f1474f6f640 10 monclient: _finish_hunting 0 2024-03-27T21:40:16.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.630+0000 7f1474f6f640 1 monclient: found mon.noname-c 2024-03-27T21:40:16.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.630+0000 7f1474f6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:16.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.630+0000 7f1474f6f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:40:16.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.630+0000 7f1474f6f640 1 -- 172.21.15.60:0/4225391701 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14700a9470 con 0x7f14701416f0 2024-03-27T21:40:16.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.630+0000 7f146f7fe640 1 --2- 172.21.15.60:0/4225391701 >> v2:172.21.15.60:3300/0 conn(0x7f147013b020 0x7f147013b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:16.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.630+0000 7f146effd640 1 -- 172.21.15.60:0/4225391701 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849311110 0 0) 0x7f14700b3830 con 0x7f14700b40b0 2024-03-27T21:40:16.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.631+0000 7f146effd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:40:16.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.631+0000 7f1474f6f640 1 --2- 172.21.15.60:0/4225391701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14701416f0 0x7f1470141b40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f147006b940 tx=0x7f146802bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4b0eaafe33eff79 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:16.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.631+0000 7f146effd640 1 -- 172.21.15.60:0/4225391701 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146802cc70 con 0x7f14701416f0 2024-03-27T21:40:16.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.631+0000 7f146effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:16.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.631+0000 7f146effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:16.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.631+0000 7f146effd640 10 monclient: dump: 2024-03-27T21:40:16.635 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:16.635 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:16.635 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:16.635 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:16.635 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:16.635 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:16.635 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:16.635 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:16.635 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:16.635 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:16.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.631+0000 7f146effd640 10 monclient: _finish_auth 0 2024-03-27T21:40:16.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.631+0000 7f146effd640 10 monclient: _check_auth_tickets 2024-03-27T21:40:16.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.631+0000 7f146effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:16.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.632+0000 7f146effd640 1 -- 172.21.15.60:0/4225391701 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146802cdd0 con 0x7f14701416f0 2024-03-27T21:40:16.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.632+0000 7f14769d2640 5 monclient: authenticate success, global_id 4549 2024-03-27T21:40:16.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.632+0000 7f14769d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:16.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.632+0000 7f146effd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:16.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.632+0000 7f14769d2640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:16.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.632+0000 7f14769d2640 4 set_mon_vals no callback set 2024-03-27T21:40:16.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.632+0000 7f146effd640 1 -- 172.21.15.60:0/4225391701 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468034660 con 0x7f14701416f0 2024-03-27T21:40:16.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.632+0000 7f146effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:16.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.632+0000 7f146effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:40:16.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.632+0000 7f146effd640 10 monclient: dump: 2024-03-27T21:40:16.636 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:16.636 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:16.637 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:16.637 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:16.637 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:16.637 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:16.637 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:16.637 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:16.637 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:16.637 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:16.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.633+0000 7f14769d2640 10 monclient: shutdown 2024-03-27T21:40:16.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.633+0000 7f14769d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:16.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.633+0000 7f14769d2640 1 -- 172.21.15.60:0/4225391701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14701416f0 msgr2=0x7f1470141b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:16.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.633+0000 7f14769d2640 1 --2- 172.21.15.60:0/4225391701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14701416f0 0x7f1470141b40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f147006b940 tx=0x7f146802bec0 comp rx=0 tx=0).stop 2024-03-27T21:40:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.633+0000 7f14769d2640 1 -- 172.21.15.60:0/4225391701 shutdown_connections 2024-03-27T21:40:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.633+0000 7f14769d2640 1 --2- 172.21.15.60:0/4225391701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14701416f0 0x7f1470141b40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.633+0000 7f14769d2640 1 --2- 172.21.15.60:0/4225391701 >> v2:172.21.15.60:3300/0 conn(0x7f147013b020 0x7f147013b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.633+0000 7f14769d2640 1 -- 172.21.15.60:0/4225391701 >> 172.21.15.60:0/4225391701 conn(0x7f14700608e0 msgr2=0x7f1470060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.633+0000 7f14769d2640 1 -- 172.21.15.60:0/4225391701 shutdown_connections 2024-03-27T21:40:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.633+0000 7f14769d2640 1 -- 172.21.15.60:0/4225391701 wait complete. 2024-03-27T21:40:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.634+0000 7f14769d2640 10 monclient: build_initial_monmap 2024-03-27T21:40:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.634+0000 7f14769d2640 10 monclient: monmap: 2024-03-27T21:40:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:16.639 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:16.639 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:16.639 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:16.639 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:16.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.634+0000 7f14769d2640 1 librados: starting msgr at 2024-03-27T21:40:16.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.634+0000 7f14769d2640 1 librados: starting objecter 2024-03-27T21:40:16.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.634+0000 7f14769d2640 1 Processor -- start 2024-03-27T21:40:16.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.635+0000 7f14769d2640 1 -- start start 2024-03-27T21:40:16.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.635+0000 7f14769d2640 1 librados: setting wanted keys 2024-03-27T21:40:16.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.635+0000 7f14769d2640 1 librados: calling monclient init 2024-03-27T21:40:16.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.635+0000 7f14769d2640 10 monclient: init 2024-03-27T21:40:16.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.635+0000 7f14769d2640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:16.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f14769d2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:40:16.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f14769d2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14700b40b0 0x7f147013d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:16.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f14769d2640 10 monclient(hunting): picked mon.noname-b con 0x7f14700b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f14769d2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f147013b020 0x7f147014b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f146ffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14700b40b0 0x7f147013d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f146ffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14700b40b0 0x7f147013d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:53926/0 (socket says 172.21.15.60:53926) 2024-03-27T21:40:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f146f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f147013b020 0x7f147014b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f146f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f147013b020 0x7f147014b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35946/0 (socket says 172.21.15.60:35946) 2024-03-27T21:40:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f146ffff640 1 -- 172.21.15.60:0/789218575 learned_addr learned my addr 172.21.15.60:0/789218575 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f14769d2640 10 monclient(hunting): picked mon.noname-a con 0x7f147013b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f14769d2640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f14701416f0 0x7f1470134160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f1474f6f640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f14701416f0 0x7f1470134160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f14769d2640 10 monclient(hunting): picked mon.noname-c con 0x7f14701416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f14769d2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f14769d2640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147013e4e0 con 0x7f147013b020 2024-03-27T21:40:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f14769d2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f14769d2640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147013aaa0 con 0x7f14700b40b0 2024-03-27T21:40:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f14769d2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f14769d2640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f147013dfc0 con 0x7f14701416f0 2024-03-27T21:40:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f14769d2640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f14769d2640 10 monclient(hunting): authenticate will time out at 857.857764s 2024-03-27T21:40:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f146ffff640 10 monclient(hunting): get_auth_request con 0x7f14700b40b0 auth_method 0 2024-03-27T21:40:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f146ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f146ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f146ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f146f7fe640 10 monclient(hunting): get_auth_request con 0x7f147013b020 auth_method 0 2024-03-27T21:40:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f146f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f146f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.636+0000 7f146f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f1474f6f640 10 monclient(hunting): get_auth_request con 0x7f14701416f0 auth_method 0 2024-03-27T21:40:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f1474f6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f1474f6f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f1474f6f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146ffff640 10 monclient(hunting): handle_auth_done global_id 4555 payload 824 2024-03-27T21:40:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146ffff640 1 -- 172.21.15.60:0/789218575 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f14701416f0 msgr2=0x7f1470134160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146ffff640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f14701416f0 0x7f1470134160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:16.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146ffff640 1 -- 172.21.15.60:0/789218575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f147013b020 msgr2=0x7f147014b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:16.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146ffff640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f147013b020 0x7f147014b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:16.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146ffff640 10 monclient: _finish_hunting 0 2024-03-27T21:40:16.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146ffff640 1 monclient: found mon.noname-b 2024-03-27T21:40:16.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:16.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:40:16.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146ffff640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1464009400 con 0x7f14700b40b0 2024-03-27T21:40:16.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f1474f6f640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f14701416f0 0x7f1470134160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:40:16.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146f7fe640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f147013b020 0x7f147014b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:16.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146ffff640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14700b40b0 0x7f147013d9c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f146000c4d0 tx=0x7f146000c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:16.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146cff9640 10 client.?.objecter ms_handle_connect 0x7f14700b40b0 2024-03-27T21:40:16.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146cff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:16.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146cff9640 1 -- 172.21.15.60:0/789218575 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14600074e0 con 0x7f14700b40b0 2024-03-27T21:40:16.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146cff9640 10 monclient: dump: 2024-03-27T21:40:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:16.644 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146cff9640 10 monclient: _finish_auth 0 2024-03-27T21:40:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146cff9640 10 monclient: _check_auth_tickets 2024-03-27T21:40:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f146cff9640 1 -- 172.21.15.60:0/789218575 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1460007640 con 0x7f14700b40b0 2024-03-27T21:40:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f14769d2640 5 monclient: authenticate success, global_id 4555 2024-03-27T21:40:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f14769d2640 10 monclient: _renew_subs 2024-03-27T21:40:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.637+0000 7f14769d2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.638+0000 7f14769d2640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1468009a70 con 0x7f14700b40b0 2024-03-27T21:40:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.638+0000 7f146cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.638+0000 7f146cff9640 1 -- 172.21.15.60:0/789218575 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14600049e0 con 0x7f14700b40b0 2024-03-27T21:40:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.638+0000 7f146cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.638+0000 7f146cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:40:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.638+0000 7f146cff9640 10 monclient: dump: 2024-03-27T21:40:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:16.646 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:16.646 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:16.646 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:16.646 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:16.646 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:16.646 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:16.646 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:16.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.638+0000 7f14769d2640 10 client.4555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:16.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.638+0000 7f14769d2640 10 monclient: _renew_subs 2024-03-27T21:40:16.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.638+0000 7f14769d2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:16.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.638+0000 7f14769d2640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147013d290 con 0x7f14700b40b0 2024-03-27T21:40:16.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.638+0000 7f14769d2640 1 librados: init done 2024-03-27T21:40:16.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.638+0000 7f146e7fc640 4 set_mon_vals no callback set 2024-03-27T21:40:16.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.639+0000 7f14567fc640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.639+0000 7f14567fc640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.639+0000 7f14567fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.639+0000 7f14567fc640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f1470069470 con 0x7f14700b40b0 2024-03-27T21:40:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.640+0000 7f146cff9640 1 -- 172.21.15.60:0/789218575 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f146001c3b0 con 0x7f14700b40b0 2024-03-27T21:40:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.640+0000 7f146cff9640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f145c06c660 0x7f145c06eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f146f7fe640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f145c06c660 0x7f145c06eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f146cff9640 1 -- 172.21.15.60:0/789218575 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f1460088700 con 0x7f14700b40b0 2024-03-27T21:40:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f146cff9640 10 client.4555.objecter ms_dispatch 0x7f1470000b90 osd_map(31..31 src has 1..31) v4 2024-03-27T21:40:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f146cff9640 3 client.4555.objecter handle_osd_map got epochs [31,31] > 0 2024-03-27T21:40:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f146cff9640 3 client.4555.objecter handle_osd_map decoding full epoch 31 2024-03-27T21:40:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f146f7fe640 10 monclient: get_auth_request con 0x7f145c06c660 auth_method 0 2024-03-27T21:40:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f146cff9640 20 client.4555.objecter dump_active .. 0 homeless 2024-03-27T21:40:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f146cff9640 1 -- 172.21.15.60:0/789218575 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v31) v1 ==== 54+0+16 (secure 0 0 0) 0x7f146008f070 con 0x7f14700b40b0 2024-03-27T21:40:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f146cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f146cff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:16.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f14567fc640 10 client.4555.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:16.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f14567fc640 15 client.4555.objecter _session_command_op_assign -1 1 2024-03-27T21:40:16.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f14567fc640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f14701dac70 0x7f14701dda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:16.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f14567fc640 20 client.4555.objecter _get_session s=0x7f1470142290 osd=0 3 2024-03-27T21:40:16.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f14567fc640 20 client.4555.objecter put_session s=0x7f1470142290 osd=0 3 2024-03-27T21:40:16.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f14567fc640 20 client.4555.objecter _get_session s=0x7f1470142290 osd=0 3 2024-03-27T21:40:16.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f14567fc640 15 client.4555.objecter _session_command_op_remove -1 1 2024-03-27T21:40:16.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f14567fc640 20 client.4555.objecter get_session s=0x7f1470142290 osd=0 3 2024-03-27T21:40:16.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f14567fc640 15 client.4555.objecter _session_command_op_assign 0 1 2024-03-27T21:40:16.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f14567fc640 20 client.4555.objecter put_session s=0x7f1470142290 osd=0 4 2024-03-27T21:40:16.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f14567fc640 10 client.4555.objecter _send_command 1 2024-03-27T21:40:16.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f14567fc640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1470134bb0 con 0x7f14701dac70 2024-03-27T21:40:16.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f1474f6f640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f14701dac70 0x7f14701dda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:16.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.641+0000 7f1474f6f640 10 monclient: get_auth_request con 0x7f14701dac70 auth_method 0 2024-03-27T21:40:16.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.642+0000 7f146cff9640 10 client.4555.objecter ms_handle_connect 0x7f14701dac70 2024-03-27T21:40:16.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.642+0000 7f1474f6f640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f14701dac70 0x7f14701dda30 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:16.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.642+0000 7f146f7fe640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f145c06c660 0x7f145c06eb20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f14700702d0 tx=0x7f1464009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:16.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.642+0000 7f146cff9640 10 client.4555.objecter ms_handle_connect 0x7f145c06c660 2024-03-27T21:40:16.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.643+0000 7f146cff9640 1 -- 172.21.15.60:0/789218575 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f1470134bb0 con 0x7f14701dac70 2024-03-27T21:40:16.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.643+0000 7f146cff9640 10 client.4555.objecter ms_dispatch 0x7f1470000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:16.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.643+0000 7f146cff9640 10 client.4555.objecter _finish_command 1 = system:0 2024-03-27T21:40:16.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.643+0000 7f146cff9640 20 client.4555.objecter put_session s=0x7f1470142290 osd=0 4 2024-03-27T21:40:16.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.643+0000 7f146cff9640 15 client.4555.objecter _session_command_op_remove 0 1 2024-03-27T21:40:16.650 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:40:16.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.672+0000 7f14567fc640 10 client.4555.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:16.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.672+0000 7f14567fc640 15 client.4555.objecter _session_command_op_assign -1 2 2024-03-27T21:40:16.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.672+0000 7f14567fc640 20 client.4555.objecter _get_session s=0x7f1470142290 osd=0 3 2024-03-27T21:40:16.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.672+0000 7f14567fc640 20 client.4555.objecter put_session s=0x7f1470142290 osd=0 3 2024-03-27T21:40:16.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.672+0000 7f14567fc640 20 client.4555.objecter _get_session s=0x7f1470142290 osd=0 3 2024-03-27T21:40:16.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.672+0000 7f14567fc640 15 client.4555.objecter _session_command_op_remove -1 2 2024-03-27T21:40:16.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.672+0000 7f14567fc640 20 client.4555.objecter get_session s=0x7f1470142290 osd=0 3 2024-03-27T21:40:16.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.672+0000 7f14567fc640 15 client.4555.objecter _session_command_op_assign 0 2 2024-03-27T21:40:16.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.672+0000 7f14567fc640 20 client.4555.objecter put_session s=0x7f1470142290 osd=0 4 2024-03-27T21:40:16.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.672+0000 7f14567fc640 10 client.4555.objecter _send_command 2 2024-03-27T21:40:16.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.672+0000 7f14567fc640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f14700b13b0 con 0x7f14701dac70 2024-03-27T21:40:16.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.675+0000 7f146cff9640 1 -- 172.21.15.60:0/789218575 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f14700b13b0 con 0x7f14701dac70 2024-03-27T21:40:16.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.675+0000 7f146cff9640 10 client.4555.objecter ms_dispatch 0x7f1470000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:16.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.675+0000 7f146cff9640 10 client.4555.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:16.663 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:16.665 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:16.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.675+0000 7f146cff9640 20 client.4555.objecter put_session s=0x7f1470142290 osd=0 4 2024-03-27T21:40:16.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.675+0000 7f146cff9640 15 client.4555.objecter _session_command_op_remove 0 2 2024-03-27T21:40:16.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.675+0000 7f14567fc640 10 client.4555.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:16.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.675+0000 7f14567fc640 15 client.4555.objecter _session_command_op_assign -1 3 2024-03-27T21:40:16.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.675+0000 7f14567fc640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f1470065a90 0x7f1470065e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:16.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.675+0000 7f14567fc640 20 client.4555.objecter _get_session s=0x7f147013e180 osd=1 3 2024-03-27T21:40:16.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.675+0000 7f14567fc640 20 client.4555.objecter put_session s=0x7f147013e180 osd=1 3 2024-03-27T21:40:16.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.675+0000 7f14567fc640 20 client.4555.objecter _get_session s=0x7f147013e180 osd=1 3 2024-03-27T21:40:16.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.675+0000 7f14567fc640 15 client.4555.objecter _session_command_op_remove -1 3 2024-03-27T21:40:16.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.675+0000 7f14567fc640 20 client.4555.objecter get_session s=0x7f147013e180 osd=1 3 2024-03-27T21:40:16.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.675+0000 7f14567fc640 15 client.4555.objecter _session_command_op_assign 1 3 2024-03-27T21:40:16.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.675+0000 7f14567fc640 20 client.4555.objecter put_session s=0x7f147013e180 osd=1 4 2024-03-27T21:40:16.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.675+0000 7f14567fc640 10 client.4555.objecter _send_command 3 2024-03-27T21:40:16.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.675+0000 7f14567fc640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f14701de730 con 0x7f1470065a90 2024-03-27T21:40:16.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.675+0000 7f146ffff640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f1470065a90 0x7f1470065e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:16.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.675+0000 7f146ffff640 10 monclient: get_auth_request con 0x7f1470065a90 auth_method 0 2024-03-27T21:40:16.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.676+0000 7f146ffff640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f1470065a90 0x7f1470065e70 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:16.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.676+0000 7f146cff9640 10 client.4555.objecter ms_handle_connect 0x7f1470065a90 2024-03-27T21:40:16.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.676+0000 7f146cff9640 1 -- 172.21.15.60:0/789218575 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f14701de730 con 0x7f1470065a90 2024-03-27T21:40:16.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.676+0000 7f146cff9640 10 client.4555.objecter ms_dispatch 0x7f1470000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:16.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.676+0000 7f146cff9640 10 client.4555.objecter _finish_command 3 = system:0 2024-03-27T21:40:16.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.676+0000 7f146cff9640 20 client.4555.objecter put_session s=0x7f147013e180 osd=1 4 2024-03-27T21:40:16.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.676+0000 7f146cff9640 15 client.4555.objecter _session_command_op_remove 1 3 2024-03-27T21:40:16.676 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-27T21:40:16.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.706+0000 7f14567fc640 10 client.4555.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:16.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.706+0000 7f14567fc640 15 client.4555.objecter _session_command_op_assign -1 4 2024-03-27T21:40:16.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.706+0000 7f14567fc640 20 client.4555.objecter _get_session s=0x7f147013e180 osd=1 3 2024-03-27T21:40:16.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.706+0000 7f14567fc640 20 client.4555.objecter put_session s=0x7f147013e180 osd=1 3 2024-03-27T21:40:16.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.706+0000 7f14567fc640 20 client.4555.objecter _get_session s=0x7f147013e180 osd=1 3 2024-03-27T21:40:16.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.706+0000 7f14567fc640 15 client.4555.objecter _session_command_op_remove -1 4 2024-03-27T21:40:16.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.706+0000 7f14567fc640 20 client.4555.objecter get_session s=0x7f147013e180 osd=1 3 2024-03-27T21:40:16.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.706+0000 7f14567fc640 15 client.4555.objecter _session_command_op_assign 1 4 2024-03-27T21:40:16.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.706+0000 7f14567fc640 20 client.4555.objecter put_session s=0x7f147013e180 osd=1 4 2024-03-27T21:40:16.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.706+0000 7f14567fc640 10 client.4555.objecter _send_command 4 2024-03-27T21:40:16.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.706+0000 7f14567fc640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f14701de8a0 con 0x7f1470065a90 2024-03-27T21:40:16.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.709+0000 7f146cff9640 1 -- 172.21.15.60:0/789218575 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f14701de8a0 con 0x7f1470065a90 2024-03-27T21:40:16.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.709+0000 7f146cff9640 10 client.4555.objecter ms_dispatch 0x7f1470000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:16.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.709+0000 7f146cff9640 10 client.4555.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:16.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.709+0000 7f146cff9640 20 client.4555.objecter put_session s=0x7f147013e180 osd=1 4 2024-03-27T21:40:16.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.709+0000 7f146cff9640 15 client.4555.objecter _session_command_op_remove 1 4 2024-03-27T21:40:16.696 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:16.699 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:16.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.709+0000 7f14567fc640 10 client.4555.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.709+0000 7f14567fc640 15 client.4555.objecter _session_command_op_assign -1 5 2024-03-27T21:40:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.709+0000 7f14567fc640 20 client.4555.objecter _get_session osd=-1 returning homeless 2024-03-27T21:40:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.709+0000 7f14567fc640 10 client.4555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.710+0000 7f14567fc640 10 monclient: _renew_subs 2024-03-27T21:40:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.710+0000 7f14567fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.710+0000 7f14567fc640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=32}) v3 -- 0x7f14701de3f0 con 0x7f14700b40b0 2024-03-27T21:40:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.710+0000 7f14567fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.710+0000 7f14567fc640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f14701eb670 con 0x7f14700b40b0 2024-03-27T21:40:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.710+0000 7f146cff9640 1 -- 172.21.15.60:0/789218575 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f146005b610 con 0x7f14700b40b0 2024-03-27T21:40:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.710+0000 7f146cff9640 10 monclient: handle_get_version_reply finishing 1 version 31 2024-03-27T21:40:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.710+0000 7f146e7fc640 10 client.4555.objecter _check_command_map_dne tid 5 current 31 map_dne_bound 31 2024-03-27T21:40:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.710+0000 7f146e7fc640 10 client.4555.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:40:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.710+0000 7f146e7fc640 15 client.4555.objecter _session_command_op_remove -1 5 2024-03-27T21:40:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:40:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.711+0000 7f14769d2640 10 client.4555.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.711+0000 7f14769d2640 15 client.4555.objecter _session_command_op_assign -1 6 2024-03-27T21:40:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.711+0000 7f14769d2640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f144c002f70 0x7f144c005430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.711+0000 7f14769d2640 20 client.4555.objecter _get_session s=0x7f144c002860 osd=3 3 2024-03-27T21:40:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.711+0000 7f14769d2640 20 client.4555.objecter put_session s=0x7f144c002860 osd=3 3 2024-03-27T21:40:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.711+0000 7f14769d2640 20 client.4555.objecter _get_session s=0x7f144c002860 osd=3 3 2024-03-27T21:40:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.711+0000 7f14769d2640 15 client.4555.objecter _session_command_op_remove -1 6 2024-03-27T21:40:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.711+0000 7f14769d2640 20 client.4555.objecter get_session s=0x7f144c002860 osd=3 3 2024-03-27T21:40:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.711+0000 7f14769d2640 15 client.4555.objecter _session_command_op_assign 3 6 2024-03-27T21:40:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.711+0000 7f14769d2640 20 client.4555.objecter put_session s=0x7f144c002860 osd=3 4 2024-03-27T21:40:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.711+0000 7f14769d2640 10 client.4555.objecter _send_command 6 2024-03-27T21:40:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.711+0000 7f14769d2640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f144c008420 con 0x7f144c002f70 2024-03-27T21:40:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.711+0000 7f146f7fe640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f144c002f70 0x7f144c005430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.711+0000 7f146f7fe640 10 monclient: get_auth_request con 0x7f144c002f70 auth_method 0 2024-03-27T21:40:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.712+0000 7f146f7fe640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f144c002f70 0x7f144c005430 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.712+0000 7f146cff9640 10 client.4555.objecter ms_handle_connect 0x7f144c002f70 2024-03-27T21:40:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.713+0000 7f146cff9640 1 -- 172.21.15.60:0/789218575 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f144c008420 con 0x7f144c002f70 2024-03-27T21:40:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.713+0000 7f146cff9640 10 client.4555.objecter ms_dispatch 0x7f1470000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:40:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.713+0000 7f146cff9640 10 client.4555.objecter _finish_command 6 = system:0 2024-03-27T21:40:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.713+0000 7f146cff9640 20 client.4555.objecter put_session s=0x7f144c002860 osd=3 4 2024-03-27T21:40:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.713+0000 7f146cff9640 15 client.4555.objecter _session_command_op_remove 3 6 2024-03-27T21:40:16.705 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:16.720+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:16.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.743+0000 7f14769d2640 10 client.4555.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:16.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.743+0000 7f14769d2640 15 client.4555.objecter _session_command_op_assign -1 7 2024-03-27T21:40:16.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.743+0000 7f14769d2640 20 client.4555.objecter _get_session s=0x7f144c002860 osd=3 3 2024-03-27T21:40:16.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.743+0000 7f14769d2640 20 client.4555.objecter put_session s=0x7f144c002860 osd=3 3 2024-03-27T21:40:16.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.743+0000 7f14769d2640 20 client.4555.objecter _get_session s=0x7f144c002860 osd=3 3 2024-03-27T21:40:16.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.743+0000 7f14769d2640 15 client.4555.objecter _session_command_op_remove -1 7 2024-03-27T21:40:16.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.743+0000 7f14769d2640 20 client.4555.objecter get_session s=0x7f144c002860 osd=3 3 2024-03-27T21:40:16.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.743+0000 7f14769d2640 15 client.4555.objecter _session_command_op_assign 3 7 2024-03-27T21:40:16.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.743+0000 7f14769d2640 20 client.4555.objecter put_session s=0x7f144c002860 osd=3 4 2024-03-27T21:40:16.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.743+0000 7f14769d2640 10 client.4555.objecter _send_command 7 2024-03-27T21:40:16.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.743+0000 7f14769d2640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f144c001100 con 0x7f144c002f70 2024-03-27T21:40:16.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.746+0000 7f146cff9640 1 -- 172.21.15.60:0/789218575 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f144c001100 con 0x7f144c002f70 2024-03-27T21:40:16.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.746+0000 7f146cff9640 10 client.4555.objecter ms_dispatch 0x7f1470000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:16.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.746+0000 7f146cff9640 10 client.4555.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:16.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.746+0000 7f146cff9640 20 client.4555.objecter put_session s=0x7f144c002860 osd=3 4 2024-03-27T21:40:16.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.746+0000 7f146cff9640 15 client.4555.objecter _session_command_op_remove 3 7 2024-03-27T21:40:16.733 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:16.733 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:16.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.746+0000 7f14769d2640 10 client.4555.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:16.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.746+0000 7f14769d2640 15 client.4555.objecter _session_command_op_assign -1 8 2024-03-27T21:40:16.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.746+0000 7f14769d2640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f144c005ab0 0x7f144c005f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:16.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.746+0000 7f14769d2640 20 client.4555.objecter _get_session s=0x7f144c002550 osd=4 3 2024-03-27T21:40:16.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.746+0000 7f14769d2640 20 client.4555.objecter put_session s=0x7f144c002550 osd=4 3 2024-03-27T21:40:16.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.746+0000 7f14769d2640 20 client.4555.objecter _get_session s=0x7f144c002550 osd=4 3 2024-03-27T21:40:16.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.746+0000 7f14769d2640 15 client.4555.objecter _session_command_op_remove -1 8 2024-03-27T21:40:16.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.746+0000 7f14769d2640 20 client.4555.objecter get_session s=0x7f144c002550 osd=4 3 2024-03-27T21:40:16.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.746+0000 7f14769d2640 15 client.4555.objecter _session_command_op_assign 4 8 2024-03-27T21:40:16.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.746+0000 7f14769d2640 20 client.4555.objecter put_session s=0x7f144c002550 osd=4 4 2024-03-27T21:40:16.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.746+0000 7f14769d2640 10 client.4555.objecter _send_command 8 2024-03-27T21:40:16.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.746+0000 7f14769d2640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f144c0081b0 con 0x7f144c005ab0 2024-03-27T21:40:16.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.747+0000 7f1474f6f640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f144c005ab0 0x7f144c005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:16.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.748+0000 7f1474f6f640 10 monclient: get_auth_request con 0x7f144c005ab0 auth_method 0 2024-03-27T21:40:16.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.748+0000 7f146cff9640 10 client.4555.objecter ms_handle_connect 0x7f144c005ab0 2024-03-27T21:40:16.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.748+0000 7f1474f6f640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f144c005ab0 0x7f144c005f60 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:16.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.749+0000 7f146cff9640 1 -- 172.21.15.60:0/789218575 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f144c0081b0 con 0x7f144c005ab0 2024-03-27T21:40:16.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.749+0000 7f146cff9640 10 client.4555.objecter ms_dispatch 0x7f1470000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:40:16.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.749+0000 7f146cff9640 10 client.4555.objecter _finish_command 8 = system:0 2024-03-27T21:40:16.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.749+0000 7f146cff9640 20 client.4555.objecter put_session s=0x7f144c002550 osd=4 4 2024-03-27T21:40:16.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.749+0000 7f146cff9640 15 client.4555.objecter _session_command_op_remove 4 8 2024-03-27T21:40:16.756 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:16.756 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:16.757 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:16.757 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:16.765 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:40:16.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.778+0000 7f14769d2640 10 client.4555.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:16.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.778+0000 7f14769d2640 15 client.4555.objecter _session_command_op_assign -1 9 2024-03-27T21:40:16.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.778+0000 7f14769d2640 20 client.4555.objecter _get_session s=0x7f144c002550 osd=4 3 2024-03-27T21:40:16.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.778+0000 7f14769d2640 20 client.4555.objecter put_session s=0x7f144c002550 osd=4 3 2024-03-27T21:40:16.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.778+0000 7f14769d2640 20 client.4555.objecter _get_session s=0x7f144c002550 osd=4 3 2024-03-27T21:40:16.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.778+0000 7f14769d2640 15 client.4555.objecter _session_command_op_remove -1 9 2024-03-27T21:40:16.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.778+0000 7f14769d2640 20 client.4555.objecter get_session s=0x7f144c002550 osd=4 3 2024-03-27T21:40:16.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.778+0000 7f14769d2640 15 client.4555.objecter _session_command_op_assign 4 9 2024-03-27T21:40:16.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.778+0000 7f14769d2640 20 client.4555.objecter put_session s=0x7f144c002550 osd=4 4 2024-03-27T21:40:16.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.778+0000 7f14769d2640 10 client.4555.objecter _send_command 9 2024-03-27T21:40:16.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.778+0000 7f14769d2640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f144c006c90 con 0x7f144c005ab0 2024-03-27T21:40:16.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.780+0000 7f146cff9640 1 -- 172.21.15.60:0/789218575 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f144c006c90 con 0x7f144c005ab0 2024-03-27T21:40:16.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.780+0000 7f146cff9640 10 client.4555.objecter ms_dispatch 0x7f1470000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:16.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.780+0000 7f146cff9640 10 client.4555.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:16.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.780+0000 7f146cff9640 20 client.4555.objecter put_session s=0x7f144c002550 osd=4 4 2024-03-27T21:40:16.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.780+0000 7f146cff9640 15 client.4555.objecter _session_command_op_remove 4 9 2024-03-27T21:40:16.770 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:16.771 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:16.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.780+0000 7f14769d2640 10 client.4555.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:16.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.780+0000 7f14769d2640 15 client.4555.objecter _session_command_op_assign -1 10 2024-03-27T21:40:16.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.780+0000 7f14769d2640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f144c00a630 0x7f144c00ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:16.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.780+0000 7f14769d2640 20 client.4555.objecter _get_session s=0x7f144c0064a0 osd=5 3 2024-03-27T21:40:16.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.780+0000 7f14769d2640 20 client.4555.objecter put_session s=0x7f144c0064a0 osd=5 3 2024-03-27T21:40:16.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.780+0000 7f14769d2640 20 client.4555.objecter _get_session s=0x7f144c0064a0 osd=5 3 2024-03-27T21:40:16.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.780+0000 7f14769d2640 15 client.4555.objecter _session_command_op_remove -1 10 2024-03-27T21:40:16.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.780+0000 7f14769d2640 20 client.4555.objecter get_session s=0x7f144c0064a0 osd=5 3 2024-03-27T21:40:16.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.780+0000 7f14769d2640 15 client.4555.objecter _session_command_op_assign 5 10 2024-03-27T21:40:16.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.780+0000 7f14769d2640 20 client.4555.objecter put_session s=0x7f144c0064a0 osd=5 4 2024-03-27T21:40:16.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.780+0000 7f14769d2640 10 client.4555.objecter _send_command 10 2024-03-27T21:40:16.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.780+0000 7f14769d2640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f144c00f7f0 con 0x7f144c00a630 2024-03-27T21:40:16.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.783+0000 7f146ffff640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f144c00a630 0x7f144c00ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:16.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.783+0000 7f146ffff640 10 monclient: get_auth_request con 0x7f144c00a630 auth_method 0 2024-03-27T21:40:16.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.783+0000 7f146cff9640 10 client.4555.objecter ms_handle_connect 0x7f144c00a630 2024-03-27T21:40:16.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.783+0000 7f146ffff640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f144c00a630 0x7f144c00ca20 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:16.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.784+0000 7f146cff9640 1 -- 172.21.15.60:0/789218575 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f144c00f7f0 con 0x7f144c00a630 2024-03-27T21:40:16.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.784+0000 7f146cff9640 10 client.4555.objecter ms_dispatch 0x7f1470000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:40:16.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.784+0000 7f146cff9640 10 client.4555.objecter _finish_command 10 = system:0 2024-03-27T21:40:16.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.784+0000 7f146cff9640 20 client.4555.objecter put_session s=0x7f144c0064a0 osd=5 4 2024-03-27T21:40:16.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.784+0000 7f146cff9640 15 client.4555.objecter _session_command_op_remove 5 10 2024-03-27T21:40:16.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.812+0000 7f14769d2640 10 client.4555.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:16.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.812+0000 7f14769d2640 15 client.4555.objecter _session_command_op_assign -1 11 2024-03-27T21:40:16.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.812+0000 7f14769d2640 20 client.4555.objecter _get_session s=0x7f144c0064a0 osd=5 3 2024-03-27T21:40:16.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.812+0000 7f14769d2640 20 client.4555.objecter put_session s=0x7f144c0064a0 osd=5 3 2024-03-27T21:40:16.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.812+0000 7f14769d2640 20 client.4555.objecter _get_session s=0x7f144c0064a0 osd=5 3 2024-03-27T21:40:16.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.812+0000 7f14769d2640 15 client.4555.objecter _session_command_op_remove -1 11 2024-03-27T21:40:16.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.812+0000 7f14769d2640 20 client.4555.objecter get_session s=0x7f144c0064a0 osd=5 3 2024-03-27T21:40:16.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.812+0000 7f14769d2640 15 client.4555.objecter _session_command_op_assign 5 11 2024-03-27T21:40:16.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.812+0000 7f14769d2640 20 client.4555.objecter put_session s=0x7f144c0064a0 osd=5 4 2024-03-27T21:40:16.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.812+0000 7f14769d2640 10 client.4555.objecter _send_command 11 2024-03-27T21:40:16.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.812+0000 7f14769d2640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f144c00cf60 con 0x7f144c00a630 2024-03-27T21:40:16.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.814+0000 7f146cff9640 1 -- 172.21.15.60:0/789218575 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f144c00cf60 con 0x7f144c00a630 2024-03-27T21:40:16.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.814+0000 7f146cff9640 10 client.4555.objecter ms_dispatch 0x7f1470000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:16.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.814+0000 7f146cff9640 10 client.4555.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:16.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.814+0000 7f146cff9640 20 client.4555.objecter put_session s=0x7f144c0064a0 osd=5 4 2024-03-27T21:40:16.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.814+0000 7f146cff9640 15 client.4555.objecter _session_command_op_remove 5 11 2024-03-27T21:40:16.801 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:16.803 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:16.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.814+0000 7f14769d2640 10 client.4555.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:16.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.814+0000 7f14769d2640 15 client.4555.objecter _session_command_op_assign -1 12 2024-03-27T21:40:16.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.815+0000 7f14769d2640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f144c00d5e0 0x7f144c00fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:16.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.815+0000 7f14769d2640 20 client.4555.objecter _get_session s=0x7f144c00cf60 osd=6 3 2024-03-27T21:40:16.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.815+0000 7f14769d2640 20 client.4555.objecter put_session s=0x7f144c00cf60 osd=6 3 2024-03-27T21:40:16.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.815+0000 7f14769d2640 20 client.4555.objecter _get_session s=0x7f144c00cf60 osd=6 3 2024-03-27T21:40:16.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.815+0000 7f14769d2640 15 client.4555.objecter _session_command_op_remove -1 12 2024-03-27T21:40:16.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.815+0000 7f14769d2640 20 client.4555.objecter get_session s=0x7f144c00cf60 osd=6 3 2024-03-27T21:40:16.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.815+0000 7f14769d2640 15 client.4555.objecter _session_command_op_assign 6 12 2024-03-27T21:40:16.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.815+0000 7f14769d2640 20 client.4555.objecter put_session s=0x7f144c00cf60 osd=6 4 2024-03-27T21:40:16.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.815+0000 7f14769d2640 10 client.4555.objecter _send_command 12 2024-03-27T21:40:16.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.815+0000 7f14769d2640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f144c012a40 con 0x7f144c00d5e0 2024-03-27T21:40:16.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.815+0000 7f146f7fe640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f144c00d5e0 0x7f144c00fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:16.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.815+0000 7f146f7fe640 10 monclient: get_auth_request con 0x7f144c00d5e0 auth_method 0 2024-03-27T21:40:16.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.815+0000 7f146cff9640 10 client.4555.objecter ms_handle_connect 0x7f144c00d5e0 2024-03-27T21:40:16.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.815+0000 7f146f7fe640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f144c00d5e0 0x7f144c00fa10 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:16.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.816+0000 7f146cff9640 1 -- 172.21.15.60:0/789218575 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f144c012a40 con 0x7f144c00d5e0 2024-03-27T21:40:16.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.816+0000 7f146cff9640 10 client.4555.objecter ms_dispatch 0x7f1470000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:40:16.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.816+0000 7f146cff9640 10 client.4555.objecter _finish_command 12 = system:0 2024-03-27T21:40:16.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.816+0000 7f146cff9640 20 client.4555.objecter put_session s=0x7f144c00cf60 osd=6 4 2024-03-27T21:40:16.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.816+0000 7f146cff9640 15 client.4555.objecter _session_command_op_remove 6 12 2024-03-27T21:40:16.807 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:16.822+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:16.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.844+0000 7f14769d2640 10 client.4555.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:16.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.844+0000 7f14769d2640 15 client.4555.objecter _session_command_op_assign -1 13 2024-03-27T21:40:16.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.844+0000 7f14769d2640 20 client.4555.objecter _get_session s=0x7f144c00cf60 osd=6 3 2024-03-27T21:40:16.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.844+0000 7f14769d2640 20 client.4555.objecter put_session s=0x7f144c00cf60 osd=6 3 2024-03-27T21:40:16.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.844+0000 7f14769d2640 20 client.4555.objecter _get_session s=0x7f144c00cf60 osd=6 3 2024-03-27T21:40:16.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.844+0000 7f14769d2640 15 client.4555.objecter _session_command_op_remove -1 13 2024-03-27T21:40:16.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.844+0000 7f14769d2640 20 client.4555.objecter get_session s=0x7f144c00cf60 osd=6 3 2024-03-27T21:40:16.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.844+0000 7f14769d2640 15 client.4555.objecter _session_command_op_assign 6 13 2024-03-27T21:40:16.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.844+0000 7f14769d2640 20 client.4555.objecter put_session s=0x7f144c00cf60 osd=6 4 2024-03-27T21:40:16.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.844+0000 7f14769d2640 10 client.4555.objecter _send_command 13 2024-03-27T21:40:16.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.844+0000 7f14769d2640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f144c012a00 con 0x7f144c00d5e0 2024-03-27T21:40:16.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.846+0000 7f146cff9640 1 -- 172.21.15.60:0/789218575 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f1464018020 con 0x7f144c00d5e0 2024-03-27T21:40:16.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.846+0000 7f146cff9640 10 client.4555.objecter ms_dispatch 0x7f1470000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:16.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.846+0000 7f146cff9640 10 client.4555.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:16.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.846+0000 7f146cff9640 20 client.4555.objecter put_session s=0x7f144c00cf60 osd=6 4 2024-03-27T21:40:16.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.846+0000 7f146cff9640 15 client.4555.objecter _session_command_op_remove 6 13 2024-03-27T21:40:16.837 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:16.839 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:16.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.847+0000 7f14769d2640 10 client.4555.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:16.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.847+0000 7f14769d2640 15 client.4555.objecter _session_command_op_assign -1 14 2024-03-27T21:40:16.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.847+0000 7f14769d2640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f144c001b90 0x7f144c001fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:16.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.847+0000 7f14769d2640 20 client.4555.objecter _get_session s=0x7f144c0008d0 osd=7 3 2024-03-27T21:40:16.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.847+0000 7f14769d2640 20 client.4555.objecter put_session s=0x7f144c0008d0 osd=7 3 2024-03-27T21:40:16.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.847+0000 7f14769d2640 20 client.4555.objecter _get_session s=0x7f144c0008d0 osd=7 3 2024-03-27T21:40:16.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.847+0000 7f14769d2640 15 client.4555.objecter _session_command_op_remove -1 14 2024-03-27T21:40:16.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.847+0000 7f14769d2640 20 client.4555.objecter get_session s=0x7f144c0008d0 osd=7 3 2024-03-27T21:40:16.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.847+0000 7f14769d2640 15 client.4555.objecter _session_command_op_assign 7 14 2024-03-27T21:40:16.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.847+0000 7f14769d2640 20 client.4555.objecter put_session s=0x7f144c0008d0 osd=7 4 2024-03-27T21:40:16.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.847+0000 7f14769d2640 10 client.4555.objecter _send_command 14 2024-03-27T21:40:16.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.847+0000 7f14769d2640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f144c011310 con 0x7f144c001b90 2024-03-27T21:40:16.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.847+0000 7f1474f6f640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f144c001b90 0x7f144c001fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:16.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.847+0000 7f1474f6f640 10 monclient: get_auth_request con 0x7f144c001b90 auth_method 0 2024-03-27T21:40:16.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.847+0000 7f146cff9640 10 client.4555.objecter ms_handle_connect 0x7f144c001b90 2024-03-27T21:40:16.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.847+0000 7f1474f6f640 1 --2- 172.21.15.60:0/789218575 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f144c001b90 0x7f144c001fd0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:16.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.848+0000 7f146cff9640 1 -- 172.21.15.60:0/789218575 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f144c011310 con 0x7f144c001b90 2024-03-27T21:40:16.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.848+0000 7f146cff9640 10 client.4555.objecter ms_dispatch 0x7f1470000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:40:16.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.848+0000 7f146cff9640 10 client.4555.objecter _finish_command 14 = system:0 2024-03-27T21:40:16.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.848+0000 7f146cff9640 20 client.4555.objecter put_session s=0x7f144c0008d0 osd=7 4 2024-03-27T21:40:16.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.848+0000 7f146cff9640 15 client.4555.objecter _session_command_op_remove 7 14 2024-03-27T21:40:16.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.849+0000 7faa3e5c0640 10 monclient: get_monmap_and_config 2024-03-27T21:40:16.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.849+0000 7faa3e5c0640 10 monclient: build_initial_monmap 2024-03-27T21:40:16.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.849+0000 7faa3e5c0640 10 monclient: monmap: 2024-03-27T21:40:16.868 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:16.868 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:16.868 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:16.850046+0000 2024-03-27T21:40:16.868 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:16.850046+0000 2024-03-27T21:40:16.868 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:16.868 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:16.868 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:16.868 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:16.868 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:16.869 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:16.869 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:16.869 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:16.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.850+0000 7faa3e5c0640 1 Processor -- start 2024-03-27T21:40:16.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.850+0000 7faa3e5c0640 1 -- start start 2024-03-27T21:40:16.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.850+0000 7faa3e5c0640 10 monclient: init 2024-03-27T21:40:16.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.851+0000 7faa3e5c0640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:16.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.851+0000 7faa3e5c0640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-03-27T21:40:16.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.851+0000 7faa3e5c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faa380b40b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:16.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.851+0000 7faa3e5c0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faa3813b020 0x7faa3813b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:16.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.851+0000 7faa3e5c0640 10 monclient(hunting): picked mon.noname-a con 0x7faa3813b020 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:16.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.851+0000 7faa3e5c0640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faa381416f0 0x7faa38141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:16.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.851+0000 7faa377fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faa3813b020 0x7faa3813b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:16.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.851+0000 7faa377fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faa3813b020 0x7faa3813b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35960/0 (socket says 172.21.15.60:35960) 2024-03-27T21:40:16.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.851+0000 7faa37fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7faa380b40b0 0x7faa380a8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39620/0 (socket says 172.21.15.60:39620) 2024-03-27T21:40:16.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.851+0000 7faa3e5c0640 10 monclient(hunting): picked mon.noname-b con 0x7faa381416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:16.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.851+0000 7faa377fe640 1 -- 172.21.15.60:0/621095034 learned_addr learned my addr 172.21.15.60:0/621095034 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:16.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.852+0000 7faa3cb5d640 1 --2- 172.21.15.60:0/621095034 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faa381416f0 0x7faa38141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:16.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.852+0000 7faa3e5c0640 1 -- 172.21.15.60:0/621095034 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa380b3830 con 0x7faa380b40b0 2024-03-27T21:40:16.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.852+0000 7faa3e5c0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:16.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.852+0000 7faa3e5c0640 1 -- 172.21.15.60:0/621095034 --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa380b4490 con 0x7faa3813b020 2024-03-27T21:40:16.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.852+0000 7faa3e5c0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:16.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.852+0000 7faa3e5c0640 1 -- 172.21.15.60:0/621095034 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa380aa520 con 0x7faa381416f0 2024-03-27T21:40:16.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.852+0000 7faa3e5c0640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:16.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.852+0000 7faa36ffd640 1 -- 172.21.15.60:0/621095034 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63453833 0 0) 0x7faa380b3830 con 0x7faa380b40b0 2024-03-27T21:40:16.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.852+0000 7faa3e5c0640 10 monclient(hunting): authenticate will time out at 858.073605s 2024-03-27T21:40:16.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.852+0000 7faa377fe640 10 monclient(hunting): get_auth_request con 0x7faa3813b020 auth_method 0 2024-03-27T21:40:16.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.852+0000 7faa377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:16.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.852+0000 7faa377fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:16.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.852+0000 7faa377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:16.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa3cb5d640 10 monclient(hunting): get_auth_request con 0x7faa381416f0 auth_method 0 2024-03-27T21:40:16.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa3cb5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:16.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa3cb5d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:16.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa3cb5d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:16.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa36ffd640 10 monclient(hunting): _negotiate 2024-03-27T21:40:16.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:16.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:16.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa36ffd640 10 monclient(hunting): my global_id is 4584 2024-03-27T21:40:16.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa36ffd640 1 -- 172.21.15.60:0/621095034 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa20003520 con 0x7faa380b40b0 2024-03-27T21:40:16.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:16.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:16.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:16.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa377fe640 10 monclient(hunting): handle_auth_done global_id 4587 payload 274 2024-03-27T21:40:16.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa377fe640 1 -- 172.21.15.60:0/621095034 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faa381416f0 msgr2=0x7faa38141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:16.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa36ffd640 1 -- 172.21.15.60:0/621095034 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1824345618 0 0) 0x7faa20003520 con 0x7faa380b40b0 2024-03-27T21:40:16.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa377fe640 1 --2- 172.21.15.60:0/621095034 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faa381416f0 0x7faa38141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:16.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa377fe640 1 -- 172.21.15.60:0/621095034 >> v1:172.21.15.60:6789/0 conn(0x7faa380b40b0 legacy=0x7faa380a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:16.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa377fe640 10 monclient: _finish_hunting 0 2024-03-27T21:40:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa377fe640 1 monclient: found mon.noname-a 2024-03-27T21:40:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa377fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa377fe640 1 -- 172.21.15.60:0/621095034 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa380a9470 con 0x7faa3813b020 2024-03-27T21:40:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.853+0000 7faa36ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:40:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.854+0000 7faa3cb5d640 1 --2- 172.21.15.60:0/621095034 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faa381416f0 0x7faa38141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:40:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.855+0000 7faa377fe640 1 --2- 172.21.15.60:0/621095034 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa3813b020 0x7faa3813b480 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7faa2c009870 tx=0x7faa2c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4180a5e7c41a0ab server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.855+0000 7faa36ffd640 1 -- 172.21.15.60:0/621095034 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa300019f0 con 0x7faa3813b020 2024-03-27T21:40:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.855+0000 7faa36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.855+0000 7faa36ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.855+0000 7faa36ffd640 10 monclient: dump: 2024-03-27T21:40:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:16.874 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:16.874 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:16.874 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:16.874 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:16.874 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:16.874 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:16.874 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:16.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.855+0000 7faa36ffd640 10 monclient: _finish_auth 0 2024-03-27T21:40:16.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.855+0000 7faa36ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:40:16.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.855+0000 7faa36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:16.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.855+0000 7faa36ffd640 1 -- 172.21.15.60:0/621095034 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa2c031540 con 0x7faa3813b020 2024-03-27T21:40:16.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.855+0000 7faa36ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:16.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.855+0000 7faa36ffd640 1 -- 172.21.15.60:0/621095034 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa300019f0 con 0x7faa3813b020 2024-03-27T21:40:16.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.855+0000 7faa36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:16.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.855+0000 7faa36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:16.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.855+0000 7faa36ffd640 10 monclient: dump: 2024-03-27T21:40:16.875 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:16.875 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:16.875 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:16.875 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:16.875 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:16.875 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:16.875 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:16.875 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:16.875 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:16.875 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:16.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.855+0000 7faa3e5c0640 5 monclient: authenticate success, global_id 4587 2024-03-27T21:40:16.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.855+0000 7faa3e5c0640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:16.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.855+0000 7faa3e5c0640 4 set_mon_vals no callback set 2024-03-27T21:40:16.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.856+0000 7faa3e5c0640 10 monclient: shutdown 2024-03-27T21:40:16.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.856+0000 7faa3e5c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:16.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.856+0000 7faa3e5c0640 1 -- 172.21.15.60:0/621095034 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa3813b020 msgr2=0x7faa3813b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:16.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.856+0000 7faa3e5c0640 1 --2- 172.21.15.60:0/621095034 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa3813b020 0x7faa3813b480 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7faa2c009870 tx=0x7faa2c02b860 comp rx=0 tx=0).stop 2024-03-27T21:40:16.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.856+0000 7faa3e5c0640 1 -- 172.21.15.60:0/621095034 shutdown_connections 2024-03-27T21:40:16.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.856+0000 7faa3e5c0640 1 --2- 172.21.15.60:0/621095034 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faa381416f0 0x7faa38141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:16.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.856+0000 7faa3e5c0640 1 --2- 172.21.15.60:0/621095034 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa3813b020 0x7faa3813b480 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:16.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.856+0000 7faa3e5c0640 1 -- 172.21.15.60:0/621095034 >> 172.21.15.60:0/621095034 conn(0x7faa380608e0 msgr2=0x7faa38060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:16.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.856+0000 7faa3e5c0640 1 -- 172.21.15.60:0/621095034 shutdown_connections 2024-03-27T21:40:16.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.856+0000 7faa3e5c0640 1 -- 172.21.15.60:0/621095034 wait complete. 2024-03-27T21:40:16.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.857+0000 7faa3e5c0640 10 monclient: build_initial_monmap 2024-03-27T21:40:16.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.857+0000 7faa3e5c0640 10 monclient: monmap: 2024-03-27T21:40:16.876 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:16.876 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:16.877 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:16.877 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:16.877 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:16.877 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:16.877 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:16.877 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:16.877 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:16.877 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:16.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.858+0000 7faa3e5c0640 1 librados: starting msgr at 2024-03-27T21:40:16.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.858+0000 7faa3e5c0640 1 librados: starting objecter 2024-03-27T21:40:16.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.858+0000 7faa3e5c0640 1 Processor -- start 2024-03-27T21:40:16.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.858+0000 7faa3e5c0640 1 -- start start 2024-03-27T21:40:16.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.858+0000 7faa3e5c0640 1 librados: setting wanted keys 2024-03-27T21:40:16.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.858+0000 7faa3e5c0640 1 librados: calling monclient init 2024-03-27T21:40:16.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.858+0000 7faa3e5c0640 10 monclient: init 2024-03-27T21:40:16.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.859+0000 7faa3e5c0640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:16.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.859+0000 7faa3e5c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:40:16.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.859+0000 7faa3e5c0640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faa380b40b0 0x7faa3813a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:16.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.860+0000 7faa3e5c0640 10 monclient(hunting): picked mon.noname-c con 0x7faa380b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:16.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.860+0000 7faa3e5c0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa3813b020 0x7faa381d96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:16.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.860+0000 7faa37fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faa380b40b0 0x7faa3813a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:16.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.860+0000 7faa37fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faa380b40b0 0x7faa3813a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:56538/0 (socket says 172.21.15.60:56538) 2024-03-27T21:40:16.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.860+0000 7faa3e5c0640 10 monclient(hunting): picked mon.noname-b con 0x7faa3813b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:16.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.860+0000 7faa37fff640 1 -- 172.21.15.60:0/2731663916 learned_addr learned my addr 172.21.15.60:0/2731663916 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:16.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.860+0000 7faa3e5c0640 1 --2- 172.21.15.60:0/2731663916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa381416f0 0x7faa381d9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:16.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.860+0000 7faa377fe640 1 --2- 172.21.15.60:0/2731663916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa3813b020 0x7faa381d96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:16.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.860+0000 7faa3cb5d640 1 --2- 172.21.15.60:0/2731663916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa381416f0 0x7faa381d9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:16.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.860+0000 7faa3e5c0640 10 monclient(hunting): picked mon.noname-a con 0x7faa381416f0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:16.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.860+0000 7faa3e5c0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:16.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa3e5c0640 1 -- 172.21.15.60:0/2731663916 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa380abe30 con 0x7faa381416f0 2024-03-27T21:40:16.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa3e5c0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:16.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa3e5c0640 1 -- 172.21.15.60:0/2731663916 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3813ccb0 con 0x7faa3813b020 2024-03-27T21:40:16.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa3e5c0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:16.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa3e5c0640 1 -- 172.21.15.60:0/2731663916 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa3813dda0 con 0x7faa380b40b0 2024-03-27T21:40:16.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa3e5c0640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:16.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa3e5c0640 10 monclient(hunting): authenticate will time out at 858.082294s 2024-03-27T21:40:16.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa37fff640 10 monclient(hunting): get_auth_request con 0x7faa380b40b0 auth_method 0 2024-03-27T21:40:16.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:16.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa37fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:16.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:16.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa377fe640 10 monclient(hunting): get_auth_request con 0x7faa3813b020 auth_method 0 2024-03-27T21:40:16.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:16.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa377fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:16.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:16.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa3cb5d640 10 monclient(hunting): get_auth_request con 0x7faa381416f0 auth_method 0 2024-03-27T21:40:16.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa3cb5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:16.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa3cb5d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:16.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa3cb5d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:16.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa3cb5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:16.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa3cb5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:16.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa3cb5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:16.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:16.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:16.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:16.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa3cb5d640 10 monclient(hunting): handle_auth_done global_id 4590 payload 824 2024-03-27T21:40:16.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.861+0000 7faa3cb5d640 1 -- 172.21.15.60:0/2731663916 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faa380b40b0 msgr2=0x7faa3813a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:16.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.862+0000 7faa3cb5d640 1 --2- 172.21.15.60:0/2731663916 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faa380b40b0 0x7faa3813a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:16.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.862+0000 7faa3cb5d640 1 -- 172.21.15.60:0/2731663916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa3813b020 msgr2=0x7faa381d96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:16.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.862+0000 7faa3cb5d640 1 --2- 172.21.15.60:0/2731663916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa3813b020 0x7faa381d96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:16.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.862+0000 7faa3cb5d640 10 monclient: _finish_hunting 0 2024-03-27T21:40:16.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.862+0000 7faa3cb5d640 1 monclient: found mon.noname-a 2024-03-27T21:40:16.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.862+0000 7faa3cb5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:16.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.862+0000 7faa3cb5d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:16.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.862+0000 7faa3cb5d640 1 -- 172.21.15.60:0/2731663916 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa2c0094d0 con 0x7faa381416f0 2024-03-27T21:40:16.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.862+0000 7faa3cb5d640 1 --2- 172.21.15.60:0/2731663916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa381416f0 0x7faa381d9c30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7faa2800d7d0 tx=0x7faa2800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:16.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.863+0000 7faa37fff640 1 --2- 172.21.15.60:0/2731663916 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faa380b40b0 0x7faa3813a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:16.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.863+0000 7faa34ff9640 10 client.?.objecter ms_handle_connect 0x7faa381416f0 2024-03-27T21:40:16.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.863+0000 7faa34ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:16.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.863+0000 7faa34ff9640 1 -- 172.21.15.60:0/2731663916 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa28004030 con 0x7faa381416f0 2024-03-27T21:40:16.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.863+0000 7faa34ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:16.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.863+0000 7faa34ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:16.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.863+0000 7faa34ff9640 10 monclient: dump: 2024-03-27T21:40:16.884 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:16.885 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:16.885 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:16.885 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:16.885 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:16.885 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:16.885 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:16.885 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:16.886 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:16.886 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:16.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.863+0000 7faa34ff9640 10 monclient: _finish_auth 0 2024-03-27T21:40:16.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.863+0000 7faa34ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:40:16.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.863+0000 7faa34ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:16.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.863+0000 7faa34ff9640 1 -- 172.21.15.60:0/2731663916 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa28004190 con 0x7faa381416f0 2024-03-27T21:40:16.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.863+0000 7faa34ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:16.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.863+0000 7faa34ff9640 1 -- 172.21.15.60:0/2731663916 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa28004520 con 0x7faa381416f0 2024-03-27T21:40:16.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.863+0000 7faa34ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:16.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.863+0000 7faa34ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:16.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.863+0000 7faa34ff9640 10 monclient: dump: 2024-03-27T21:40:16.888 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:16.888 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:16.888 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:16.888 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:16.888 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:16.888 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:16.889 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:16.889 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:16.889 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:16.889 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:16.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.863+0000 7faa367fc640 4 set_mon_vals no callback set 2024-03-27T21:40:16.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.863+0000 7faa377fe640 1 --2- 172.21.15.60:0/2731663916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa3813b020 0x7faa381d96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:40:16.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.863+0000 7faa3e5c0640 5 monclient: authenticate success, global_id 4590 2024-03-27T21:40:16.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.863+0000 7faa3e5c0640 10 monclient: _renew_subs 2024-03-27T21:40:16.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.863+0000 7faa3e5c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:16.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.864+0000 7faa3e5c0640 1 -- 172.21.15.60:0/2731663916 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa380aa9d0 con 0x7faa381416f0 2024-03-27T21:40:16.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.864+0000 7faa3e5c0640 10 client.4590.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:16.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.864+0000 7faa3e5c0640 10 monclient: _renew_subs 2024-03-27T21:40:16.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.864+0000 7faa3e5c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:16.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.864+0000 7faa3e5c0640 1 -- 172.21.15.60:0/2731663916 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa381da350 con 0x7faa381416f0 2024-03-27T21:40:16.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.864+0000 7faa3e5c0640 1 librados: init done 2024-03-27T21:40:16.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.865+0000 7faa3e5c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:16.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.865+0000 7faa3e5c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:16.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.866+0000 7faa3e5c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:16.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.866+0000 7faa3e5c0640 1 -- 172.21.15.60:0/2731663916 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa3813df60 con 0x7faa381416f0 2024-03-27T21:40:16.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.869+0000 7faa34ff9640 1 -- 172.21.15.60:0/2731663916 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7faa2800b660 con 0x7faa381416f0 2024-03-27T21:40:16.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.869+0000 7faa34ff9640 1 --2- 172.21.15.60:0/2731663916 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7faa2406c660 0x7faa2406eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:16.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.869+0000 7faa34ff9640 1 -- 172.21.15.60:0/2731663916 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5242+0+0 (secure 0 0 0) 0x7faa280889f0 con 0x7faa381416f0 2024-03-27T21:40:16.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.869+0000 7faa34ff9640 10 client.4590.objecter ms_dispatch 0x7faa38000b90 osd_map(31..31 src has 1..31) v4 2024-03-27T21:40:16.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.869+0000 7faa34ff9640 3 client.4590.objecter handle_osd_map got epochs [31,31] > 0 2024-03-27T21:40:16.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.869+0000 7faa34ff9640 3 client.4590.objecter handle_osd_map decoding full epoch 31 2024-03-27T21:40:16.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.869+0000 7faa34ff9640 20 client.4590.objecter dump_active .. 0 homeless 2024-03-27T21:40:16.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.870+0000 7faa37fff640 1 --2- 172.21.15.60:0/2731663916 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7faa2406c660 0x7faa2406eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:16.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.870+0000 7faa37fff640 10 monclient: get_auth_request con 0x7faa2406c660 auth_method 0 2024-03-27T21:40:16.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.870+0000 7faa37fff640 1 --2- 172.21.15.60:0/2731663916 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7faa2406c660 0x7faa2406eb20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faa300012e0 tx=0x7faa30007ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:16.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.870+0000 7faa34ff9640 10 client.4590.objecter ms_handle_connect 0x7faa2406c660 2024-03-27T21:40:16.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.875+0000 7faa34ff9640 1 -- 172.21.15.60:0/2731663916 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faa28053ec0 con 0x7faa381416f0 2024-03-27T21:40:16.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.875+0000 7faa34ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:16.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.875+0000 7faa34ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:16.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.876+0000 7f14769d2640 10 client.4555.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:16.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.876+0000 7f14769d2640 15 client.4555.objecter _session_command_op_assign -1 15 2024-03-27T21:40:16.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.876+0000 7f14769d2640 20 client.4555.objecter _get_session s=0x7f144c0008d0 osd=7 3 2024-03-27T21:40:16.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.876+0000 7f14769d2640 20 client.4555.objecter put_session s=0x7f144c0008d0 osd=7 3 2024-03-27T21:40:16.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.876+0000 7f14769d2640 20 client.4555.objecter _get_session s=0x7f144c0008d0 osd=7 3 2024-03-27T21:40:16.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.876+0000 7f14769d2640 15 client.4555.objecter _session_command_op_remove -1 15 2024-03-27T21:40:16.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.876+0000 7f14769d2640 20 client.4555.objecter get_session s=0x7f144c0008d0 osd=7 3 2024-03-27T21:40:16.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.876+0000 7f14769d2640 15 client.4555.objecter _session_command_op_assign 7 15 2024-03-27T21:40:16.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.876+0000 7f14769d2640 20 client.4555.objecter put_session s=0x7f144c0008d0 osd=7 4 2024-03-27T21:40:16.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.876+0000 7f14769d2640 10 client.4555.objecter _send_command 15 2024-03-27T21:40:16.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.876+0000 7f14769d2640 1 -- 172.21.15.60:0/789218575 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f144c011310 con 0x7f144c001b90 2024-03-27T21:40:16.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.878+0000 7f146cff9640 1 -- 172.21.15.60:0/789218575 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f144c011310 con 0x7f144c001b90 2024-03-27T21:40:16.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.878+0000 7f146cff9640 10 client.4555.objecter ms_dispatch 0x7f1470000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:16.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.878+0000 7f146cff9640 10 client.4555.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:16.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.878+0000 7f146cff9640 20 client.4555.objecter put_session s=0x7f144c0008d0 osd=7 4 2024-03-27T21:40:16.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:16.878+0000 7f146cff9640 15 client.4555.objecter _session_command_op_remove 7 15 2024-03-27T21:40:16.899 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:16.900 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:40:16.900 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:16.901 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:40:16.907 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:16.922+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:16.912 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:16.912 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:16.921 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:40:17.008 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:17.022+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:17.066 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:17.105 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:17.106 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:40:17.109 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:17.123+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:17.210 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:17.224+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:17.211 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:17.211 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:17.219 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:40:17.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.263+0000 7faa3e5c0640 10 start_mon_command cmd=[{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:40:17.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.263+0000 7faa3e5c0640 10 monclient: _send_command 2 [{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:40:17.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.263+0000 7faa3e5c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:17.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.263+0000 7faa3e5c0640 1 -- 172.21.15.60:0/2731663916 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7faa380aab60 con 0x7faa381416f0 2024-03-27T21:40:17.311 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:17.325+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:17.324 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:17.324 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:17.333 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:40:17.412 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:17.426+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:17.438 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:17.438 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:17.447 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:40:17.513 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:17.527+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:17.614 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:17.628+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:17.621 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:17.621 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:17.634 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:40:17.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.690+0000 7faa34ff9640 1 -- 172.21.15.60:0/2731663916 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v32) v1 ==== 87+0+0 (secure 0 0 0) 0x7faa380aab60 con 0x7faa381416f0 2024-03-27T21:40:17.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.690+0000 7faa34ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:40:17.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.690+0000 7faa34ff9640 10 monclient: _finish_command 2 = system:0 noscrub is set 2024-03-27T21:40:17.678 INFO:teuthology.orchestra.run.smithi060.stderr:noscrub is set 2024-03-27T21:40:17.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.693+0000 7faa3e5c0640 10 librados: watch_flush enter 2024-03-27T21:40:17.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.693+0000 7faa3e5c0640 10 librados: watch_flush exit 2024-03-27T21:40:17.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.693+0000 7faa3e5c0640 20 client.4590.objecter shutdown clearing up homeless session... 2024-03-27T21:40:17.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.693+0000 7faa3e5c0640 10 client.4590.objecter successfully canceled tick 2024-03-27T21:40:17.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.693+0000 7faa3e5c0640 1 -- 172.21.15.60:0/2731663916 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7faa2406c660 msgr2=0x7faa2406eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:17.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.693+0000 7faa3e5c0640 1 --2- 172.21.15.60:0/2731663916 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7faa2406c660 0x7faa2406eb20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faa300012e0 tx=0x7faa30007ed0 comp rx=0 tx=0).stop 2024-03-27T21:40:17.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.693+0000 7faa3e5c0640 10 monclient: shutdown 2024-03-27T21:40:17.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.693+0000 7faa3e5c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:17.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.693+0000 7faa3e5c0640 1 -- 172.21.15.60:0/2731663916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa381416f0 msgr2=0x7faa381d9c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:17.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.693+0000 7faa3e5c0640 1 --2- 172.21.15.60:0/2731663916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa381416f0 0x7faa381d9c30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7faa2800d7d0 tx=0x7faa2800dc90 comp rx=0 tx=0).stop 2024-03-27T21:40:17.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.694+0000 7faa3e5c0640 1 -- 172.21.15.60:0/2731663916 shutdown_connections 2024-03-27T21:40:17.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.694+0000 7faa3e5c0640 1 --2- 172.21.15.60:0/2731663916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa381416f0 0x7faa381d9c30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:17.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.694+0000 7faa3e5c0640 1 --2- 172.21.15.60:0/2731663916 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7faa2406c660 0x7faa2406eb20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:17.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.694+0000 7faa3e5c0640 1 --2- 172.21.15.60:0/2731663916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa3813b020 0x7faa381d96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:17.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.694+0000 7faa3e5c0640 1 --2- 172.21.15.60:0/2731663916 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7faa380b40b0 0x7faa3813a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:17.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.694+0000 7faa3e5c0640 1 -- 172.21.15.60:0/2731663916 >> 172.21.15.60:0/2731663916 conn(0x7faa380608e0 msgr2=0x7faa38060cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:17.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.694+0000 7faa3e5c0640 1 -- 172.21.15.60:0/2731663916 shutdown_connections 2024-03-27T21:40:17.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.694+0000 7faa3e5c0640 1 -- 172.21.15.60:0/2731663916 wait complete. 2024-03-27T21:40:17.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:17.694+0000 7faa3e5c0640 1 librados: shutdown 2024-03-27T21:40:17.737 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:17.751+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:17.817 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:17.830+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:17.842 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:17.842 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:17.854 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:40:17.917 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:17.932+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:18.011 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:18.011 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:18.017 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:18.031+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:18.021 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:40:18.118 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:18.132+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:18.183 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:18.184 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:18.193 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:40:18.235 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:18.249+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:18.320 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:18.335+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:18.344 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:18.344 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:18.353 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:40:18.421 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:18.435+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:18.464 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:18.464 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:18.475 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:40:18.522 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:18.537+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:18.586 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:18.587 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:18.596 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:40:18.623 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:18.638+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:18.709 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:18.709 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:18.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:40:18.761 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:18.775+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:18.826 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:18.839+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:18.868 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:18.868 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:18.883 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:40:18.902 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:40:18.927 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:18.941+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:19.000 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:19.000 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:19.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:40:19.027 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:19.041+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:19.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.093+0000 7fc70ae82640 10 monclient: get_monmap_and_config 2024-03-27T21:40:19.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.093+0000 7fc70ae82640 10 monclient: build_initial_monmap 2024-03-27T21:40:19.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.093+0000 7fc70ae82640 10 monclient: monmap: 2024-03-27T21:40:19.080 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:19.080 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:19.081 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:19.094962+0000 2024-03-27T21:40:19.081 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:19.094962+0000 2024-03-27T21:40:19.081 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:19.081 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:19.081 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:19.081 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:19.081 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:19.081 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:19.081 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:19.081 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:19.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.095+0000 7fc70ae82640 1 Processor -- start 2024-03-27T21:40:19.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.095+0000 7fc70ae82640 1 -- start start 2024-03-27T21:40:19.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.095+0000 7fc70ae82640 10 monclient: init 2024-03-27T21:40:19.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc70ae82640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:19.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc70ae82640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-03-27T21:40:19.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc70ae82640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc7040b40b0 0x7fc7040a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:19.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc70ae82640 10 monclient(hunting): picked mon.noname-c con 0x7fc7040b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:19.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc70ae82640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc70413b020 0x7fc70413b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:19.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc70ae82640 10 monclient(hunting): picked mon.noname-a con 0x7fc70413b020 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:19.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc70ae82640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc7041416f0 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:19.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc70ae82640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7040b3830 con 0x7fc7041416f0 2024-03-27T21:40:19.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc70ae82640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:19.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc70ae82640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7040b4490 con 0x7fc70413b020 2024-03-27T21:40:19.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc70ae82640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:19.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc70ae82640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7040aa520 con 0x7fc7040b40b0 2024-03-27T21:40:19.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc70ae82640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:19.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc70ae82640 10 monclient(hunting): authenticate will time out at 860.317052s 2024-03-27T21:40:19.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc708c1e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc7040b40b0 0x7fc7040a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:19.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc703fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc70413b020 0x7fc70413b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:19.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc708c1e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc7040b40b0 0x7fc7040a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:53936/0 (socket says 172.21.15.60:53936) 2024-03-27T21:40:19.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc708c1e640 1 -- 172.21.15.60:0/749205217 learned_addr learned my addr 172.21.15.60:0/749205217 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:19.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc708c1e640 10 monclient(hunting): get_auth_request con 0x7fc7040b40b0 auth_method 0 2024-03-27T21:40:19.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc708c1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:19.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc708c1e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:19.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc708c1e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:19.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc703fff640 10 monclient(hunting): get_auth_request con 0x7fc70413b020 auth_method 0 2024-03-27T21:40:19.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc703fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:19.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc703fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:19.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc703fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:19.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc703fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:19.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc703fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:19.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc703fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:19.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc708c1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:19.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc708c1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:19.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc708c1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:19.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc703fff640 10 monclient(hunting): handle_auth_done global_id 4593 payload 274 2024-03-27T21:40:19.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc703fff640 1 -- 172.21.15.60:0/749205217 >> v2:172.21.15.191:3300/0 conn(0x7fc7040b40b0 msgr2=0x7fc7040a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:19.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc7037fe640 1 -- 172.21.15.60:0/749205217 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163132094 0 0) 0x7fc7040b3830 con 0x7fc7041416f0 2024-03-27T21:40:19.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.096+0000 7fc703fff640 1 --2- 172.21.15.60:0/749205217 >> v2:172.21.15.191:3300/0 conn(0x7fc7040b40b0 0x7fc7040a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:19.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.097+0000 7fc703fff640 1 -- 172.21.15.60:0/749205217 >> v1:172.21.15.60:6789/0 conn(0x7fc7041416f0 legacy=0x7fc704141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:19.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.097+0000 7fc703fff640 10 monclient: _finish_hunting 0 2024-03-27T21:40:19.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.097+0000 7fc703fff640 1 monclient: found mon.noname-a 2024-03-27T21:40:19.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.097+0000 7fc703fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:19.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.097+0000 7fc703fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:19.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.097+0000 7fc703fff640 1 -- 172.21.15.60:0/749205217 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7040a9470 con 0x7fc70413b020 2024-03-27T21:40:19.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.097+0000 7fc7037fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:40:19.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.098+0000 7fc703fff640 1 --2- 172.21.15.60:0/749205217 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc70413b020 0x7fc70413b480 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc7040b4810 tx=0x7fc6f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d512d4e03a86f75 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:19.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.098+0000 7fc7037fe640 1 -- 172.21.15.60:0/749205217 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8031830 con 0x7fc70413b020 2024-03-27T21:40:19.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.098+0000 7fc7037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:19.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.098+0000 7fc7037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:19.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.098+0000 7fc7037fe640 10 monclient: dump: 2024-03-27T21:40:19.088 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:19.089 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:19.089 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:19.089 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:19.089 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:19.089 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:19.089 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:19.089 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:19.089 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:19.089 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:19.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.098+0000 7fc7037fe640 10 monclient: _finish_auth 0 2024-03-27T21:40:19.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.098+0000 7fc7037fe640 10 monclient: _check_auth_tickets 2024-03-27T21:40:19.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.098+0000 7fc7037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:19.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.098+0000 7fc7037fe640 1 -- 172.21.15.60:0/749205217 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f8031990 con 0x7fc70413b020 2024-03-27T21:40:19.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.098+0000 7fc7037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.098+0000 7fc7037fe640 1 -- 172.21.15.60:0/749205217 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8031d20 con 0x7fc70413b020 2024-03-27T21:40:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.098+0000 7fc7037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.098+0000 7fc7037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.098+0000 7fc7037fe640 10 monclient: dump: 2024-03-27T21:40:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:19.090 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.098+0000 7fc70ae82640 5 monclient: authenticate success, global_id 4593 2024-03-27T21:40:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.098+0000 7fc70ae82640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.098+0000 7fc70ae82640 4 set_mon_vals no callback set 2024-03-27T21:40:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.099+0000 7fc70ae82640 10 monclient: shutdown 2024-03-27T21:40:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.099+0000 7fc70ae82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.099+0000 7fc70ae82640 1 -- 172.21.15.60:0/749205217 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc70413b020 msgr2=0x7fc70413b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.099+0000 7fc70ae82640 1 --2- 172.21.15.60:0/749205217 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc70413b020 0x7fc70413b480 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc7040b4810 tx=0x7fc6f802b8d0 comp rx=0 tx=0).stop 2024-03-27T21:40:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.100+0000 7fc70ae82640 1 -- 172.21.15.60:0/749205217 shutdown_connections 2024-03-27T21:40:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.100+0000 7fc70ae82640 1 --2- 172.21.15.60:0/749205217 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc70413b020 0x7fc70413b480 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.100+0000 7fc70ae82640 1 --2- 172.21.15.60:0/749205217 >> v2:172.21.15.191:3300/0 conn(0x7fc7040b40b0 0x7fc7040a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.100+0000 7fc70ae82640 1 -- 172.21.15.60:0/749205217 >> 172.21.15.60:0/749205217 conn(0x7fc7040608e0 msgr2=0x7fc704060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.102+0000 7fc70ae82640 1 -- 172.21.15.60:0/749205217 shutdown_connections 2024-03-27T21:40:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.102+0000 7fc70ae82640 1 -- 172.21.15.60:0/749205217 wait complete. 2024-03-27T21:40:19.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.103+0000 7fc70ae82640 10 monclient: build_initial_monmap 2024-03-27T21:40:19.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.103+0000 7fc70ae82640 10 monclient: monmap: 2024-03-27T21:40:19.101 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:19.101 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:19.101 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:19.101 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:19.101 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:19.102 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:19.102 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:19.102 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:19.102 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:19.102 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:19.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.104+0000 7fc70ae82640 1 librados: starting msgr at 2024-03-27T21:40:19.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.104+0000 7fc70ae82640 1 librados: starting objecter 2024-03-27T21:40:19.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.104+0000 7fc70ae82640 1 Processor -- start 2024-03-27T21:40:19.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.104+0000 7fc70ae82640 1 -- start start 2024-03-27T21:40:19.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.104+0000 7fc70ae82640 1 librados: setting wanted keys 2024-03-27T21:40:19.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.104+0000 7fc70ae82640 1 librados: calling monclient init 2024-03-27T21:40:19.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.104+0000 7fc70ae82640 10 monclient: init 2024-03-27T21:40:19.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.105+0000 7fc70ae82640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:19.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.105+0000 7fc70ae82640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:40:19.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.105+0000 7fc70ae82640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc7040b40b0 0x7fc70413e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:19.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.105+0000 7fc70ae82640 10 monclient(hunting): picked mon.noname-c con 0x7fc7040b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:19.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.106+0000 7fc708c1e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc7040b40b0 0x7fc70413e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:19.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.106+0000 7fc708c1e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc7040b40b0 0x7fc70413e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:56540/0 (socket says 172.21.15.60:56540) 2024-03-27T21:40:19.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.106+0000 7fc708c1e640 1 -- 172.21.15.60:0/1696298781 learned_addr learned my addr 172.21.15.60:0/1696298781 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:19.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.106+0000 7fc70ae82640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc70413b020 0x7fc7040aaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:19.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.106+0000 7fc70ae82640 10 monclient(hunting): picked mon.noname-b con 0x7fc70413b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:19.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.106+0000 7fc70ae82640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7041416f0 0x7fc7041da010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:19.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.106+0000 7fc703fff640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc70413b020 0x7fc7040aaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:19.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.106+0000 7fc70ae82640 10 monclient(hunting): picked mon.noname-a con 0x7fc7041416f0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:19.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.106+0000 7fc70941f640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7041416f0 0x7fc7041da010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:19.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.107+0000 7fc70ae82640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:19.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.107+0000 7fc70ae82640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7040ac110 con 0x7fc7041416f0 2024-03-27T21:40:19.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.107+0000 7fc70ae82640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:19.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.107+0000 7fc70ae82640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7041d98e0 con 0x7fc70413b020 2024-03-27T21:40:19.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.107+0000 7fc70ae82640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:19.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.107+0000 7fc70ae82640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7041d9b80 con 0x7fc7040b40b0 2024-03-27T21:40:19.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.107+0000 7fc70ae82640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:19.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.107+0000 7fc70ae82640 10 monclient(hunting): authenticate will time out at 860.328452s 2024-03-27T21:40:19.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.107+0000 7fc708c1e640 10 monclient(hunting): get_auth_request con 0x7fc7040b40b0 auth_method 0 2024-03-27T21:40:19.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.107+0000 7fc708c1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:19.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.107+0000 7fc708c1e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:19.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.107+0000 7fc708c1e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:19.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.107+0000 7fc703fff640 10 monclient(hunting): get_auth_request con 0x7fc70413b020 auth_method 0 2024-03-27T21:40:19.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.107+0000 7fc703fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:19.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.107+0000 7fc703fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:19.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.107+0000 7fc703fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:19.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.108+0000 7fc703fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:19.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.108+0000 7fc703fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:19.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.108+0000 7fc703fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:19.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.108+0000 7fc703fff640 10 monclient(hunting): handle_auth_done global_id 4564 payload 824 2024-03-27T21:40:19.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.108+0000 7fc703fff640 1 -- 172.21.15.60:0/1696298781 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc7040b40b0 msgr2=0x7fc70413e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:19.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.108+0000 7fc703fff640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc7040b40b0 0x7fc70413e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:19.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.108+0000 7fc703fff640 1 -- 172.21.15.60:0/1696298781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7041416f0 msgr2=0x7fc7041da010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:19.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.108+0000 7fc703fff640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7041416f0 0x7fc7041da010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:19.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.108+0000 7fc703fff640 10 monclient: _finish_hunting 0 2024-03-27T21:40:19.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.108+0000 7fc703fff640 1 monclient: found mon.noname-b 2024-03-27T21:40:19.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.108+0000 7fc703fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:19.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.108+0000 7fc703fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:40:19.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.108+0000 7fc703fff640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f8009400 con 0x7fc70413b020 2024-03-27T21:40:19.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.108+0000 7fc708c1e640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc7040b40b0 0x7fc70413e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:40:19.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.108+0000 7fc70941f640 10 monclient: get_auth_request con 0x7fc7041416f0 auth_method 0 2024-03-27T21:40:19.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.108+0000 7fc70941f640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7041416f0 0x7fc7041da010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:40:19.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.109+0000 7fc703fff640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc70413b020 0x7fc7040aaeb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8033000 tx=0x7fc6f80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:19.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.109+0000 7fc7017fa640 10 client.?.objecter ms_handle_connect 0x7fc70413b020 2024-03-27T21:40:19.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.109+0000 7fc7017fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:19.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.109+0000 7fc7017fa640 1 -- 172.21.15.60:0/1696298781 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f803bab0 con 0x7fc70413b020 2024-03-27T21:40:19.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.109+0000 7fc7017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:19.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.109+0000 7fc7017fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:19.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.109+0000 7fc7017fa640 10 monclient: dump: 2024-03-27T21:40:19.107 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:19.107 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:19.107 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:19.107 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:19.107 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:19.107 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:19.107 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:19.107 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:19.107 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:19.108 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:19.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.109+0000 7fc7017fa640 10 monclient: _finish_auth 0 2024-03-27T21:40:19.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.109+0000 7fc7017fa640 10 monclient: _check_auth_tickets 2024-03-27T21:40:19.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.109+0000 7fc7017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:19.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.109+0000 7fc7017fa640 1 -- 172.21.15.60:0/1696298781 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f803bc30 con 0x7fc70413b020 2024-03-27T21:40:19.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.109+0000 7fc70ae82640 5 monclient: authenticate success, global_id 4564 2024-03-27T21:40:19.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.109+0000 7fc70ae82640 10 monclient: _renew_subs 2024-03-27T21:40:19.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.110+0000 7fc70ae82640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:19.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.110+0000 7fc70ae82640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc70413ddd0 con 0x7fc70413b020 2024-03-27T21:40:19.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.110+0000 7fc7017fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:19.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.110+0000 7fc7017fa640 1 -- 172.21.15.60:0/1696298781 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f80433f0 con 0x7fc70413b020 2024-03-27T21:40:19.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.110+0000 7fc70ae82640 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:19.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.110+0000 7fc7017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:19.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.110+0000 7fc7017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:40:19.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.110+0000 7fc7017fa640 10 monclient: dump: 2024-03-27T21:40:19.109 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:19.109 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:19.109 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:19.109 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:19.109 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:19.109 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:19.109 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:19.109 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:19.109 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:19.109 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:19.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.110+0000 7fc70ae82640 10 monclient: _renew_subs 2024-03-27T21:40:19.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.110+0000 7fc70ae82640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:19.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.110+0000 7fc70ae82640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7041da730 con 0x7fc70413b020 2024-03-27T21:40:19.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.110+0000 7fc70ae82640 1 librados: init done 2024-03-27T21:40:19.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.110+0000 7fc7027fc640 4 set_mon_vals no callback set 2024-03-27T21:40:19.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.111+0000 7fc70ae82640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:19.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.111+0000 7fc70ae82640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:19.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.111+0000 7fc70ae82640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:19.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.111+0000 7fc70ae82640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fc704069470 con 0x7fc70413b020 2024-03-27T21:40:19.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.112+0000 7fc7017fa640 1 -- 172.21.15.60:0/1696298781 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc6f804c3b0 con 0x7fc70413b020 2024-03-27T21:40:19.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.113+0000 7fc7017fa640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc6f006c660 0x7fc6f006eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:19.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.113+0000 7fc7017fa640 1 -- 172.21.15.60:0/1696298781 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fc6f80ab100 con 0x7fc70413b020 2024-03-27T21:40:19.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.113+0000 7fc7017fa640 10 client.4564.objecter ms_dispatch 0x7fc704000b90 osd_map(32..32 src has 1..32) v4 2024-03-27T21:40:19.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.113+0000 7fc7017fa640 3 client.4564.objecter handle_osd_map got epochs [32,32] > 0 2024-03-27T21:40:19.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.113+0000 7fc708c1e640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc6f006c660 0x7fc6f006eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:19.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.113+0000 7fc708c1e640 10 monclient: get_auth_request con 0x7fc6f006c660 auth_method 0 2024-03-27T21:40:19.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.113+0000 7fc7017fa640 3 client.4564.objecter handle_osd_map decoding full epoch 32 2024-03-27T21:40:19.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.113+0000 7fc7017fa640 20 client.4564.objecter dump_active .. 0 homeless 2024-03-27T21:40:19.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.114+0000 7fc7017fa640 1 -- 172.21.15.60:0/1696298781 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v32) v1 ==== 54+0+16 (secure 0 0 0) 0x7fc6f80b2070 con 0x7fc70413b020 2024-03-27T21:40:19.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.114+0000 7fc7017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:19.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.114+0000 7fc7017fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:19.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.114+0000 7fc708c1e640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc6f006c660 0x7fc6f006eb20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc6f40094a0 tx=0x7fc6f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:19.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.114+0000 7fc7017fa640 10 client.4564.objecter ms_handle_connect 0x7fc6f006c660 2024-03-27T21:40:19.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.114+0000 7fc70ae82640 10 client.4564.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:19.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.114+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_assign -1 1 2024-03-27T21:40:19.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.114+0000 7fc70ae82640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fc70413ba60 0x7fc704134670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:19.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.114+0000 7fc70ae82640 20 client.4564.objecter _get_session s=0x7fc70413e010 osd=0 3 2024-03-27T21:40:19.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.114+0000 7fc70ae82640 20 client.4564.objecter put_session s=0x7fc70413e010 osd=0 3 2024-03-27T21:40:19.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.114+0000 7fc70ae82640 20 client.4564.objecter _get_session s=0x7fc70413e010 osd=0 3 2024-03-27T21:40:19.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.114+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_remove -1 1 2024-03-27T21:40:19.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.114+0000 7fc70ae82640 20 client.4564.objecter get_session s=0x7fc70413e010 osd=0 3 2024-03-27T21:40:19.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.114+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_assign 0 1 2024-03-27T21:40:19.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.114+0000 7fc70ae82640 20 client.4564.objecter put_session s=0x7fc70413e010 osd=0 4 2024-03-27T21:40:19.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.114+0000 7fc70ae82640 10 client.4564.objecter _send_command 1 2024-03-27T21:40:19.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.114+0000 7fc70ae82640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc704142250 con 0x7fc70413ba60 2024-03-27T21:40:19.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.114+0000 7fc70941f640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fc70413ba60 0x7fc704134670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:19.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.115+0000 7fc70941f640 10 monclient: get_auth_request con 0x7fc70413ba60 auth_method 0 2024-03-27T21:40:19.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.115+0000 7fc70941f640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fc70413ba60 0x7fc704134670 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:19.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.115+0000 7fc7017fa640 10 client.4564.objecter ms_handle_connect 0x7fc70413ba60 2024-03-27T21:40:19.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.116+0000 7fc7017fa640 1 -- 172.21.15.60:0/1696298781 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fc6fc001930 con 0x7fc70413ba60 2024-03-27T21:40:19.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.116+0000 7fc7017fa640 10 client.4564.objecter ms_dispatch 0x7fc704000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:19.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.116+0000 7fc7017fa640 10 client.4564.objecter _finish_command 1 = system:0 2024-03-27T21:40:19.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.116+0000 7fc7017fa640 20 client.4564.objecter put_session s=0x7fc70413e010 osd=0 4 2024-03-27T21:40:19.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.116+0000 7fc7017fa640 15 client.4564.objecter _session_command_op_remove 0 1 2024-03-27T21:40:19.128 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:19.141+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:19.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.144+0000 7fc70ae82640 10 client.4564.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:19.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.144+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_assign -1 2 2024-03-27T21:40:19.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.144+0000 7fc70ae82640 20 client.4564.objecter _get_session s=0x7fc70413e010 osd=0 3 2024-03-27T21:40:19.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.144+0000 7fc70ae82640 20 client.4564.objecter put_session s=0x7fc70413e010 osd=0 3 2024-03-27T21:40:19.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.144+0000 7fc70ae82640 20 client.4564.objecter _get_session s=0x7fc70413e010 osd=0 3 2024-03-27T21:40:19.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.144+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_remove -1 2 2024-03-27T21:40:19.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.144+0000 7fc70ae82640 20 client.4564.objecter get_session s=0x7fc70413e010 osd=0 3 2024-03-27T21:40:19.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.144+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_assign 0 2 2024-03-27T21:40:19.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.144+0000 7fc70ae82640 20 client.4564.objecter put_session s=0x7fc70413e010 osd=0 4 2024-03-27T21:40:19.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.144+0000 7fc70ae82640 10 client.4564.objecter _send_command 2 2024-03-27T21:40:19.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.144+0000 7fc70ae82640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc70413f100 con 0x7fc70413ba60 2024-03-27T21:40:19.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.147+0000 7fc7017fa640 1 -- 172.21.15.60:0/1696298781 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc6fc001e90 con 0x7fc70413ba60 2024-03-27T21:40:19.134 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.147+0000 7fc7017fa640 10 client.4564.objecter ms_dispatch 0x7fc704000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.147+0000 7fc7017fa640 10 client.4564.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.147+0000 7fc7017fa640 20 client.4564.objecter put_session s=0x7fc70413e010 osd=0 4 2024-03-27T21:40:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.147+0000 7fc7017fa640 15 client.4564.objecter _session_command_op_remove 0 2 2024-03-27T21:40:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.148+0000 7fc6eaffd640 10 client.4564.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.148+0000 7fc6eaffd640 15 client.4564.objecter _session_command_op_assign -1 3 2024-03-27T21:40:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.148+0000 7fc6eaffd640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fc7040b4bc0 0x7fc7040b4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.148+0000 7fc6eaffd640 20 client.4564.objecter _get_session s=0x7fc7041d9cc0 osd=1 3 2024-03-27T21:40:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.148+0000 7fc6eaffd640 20 client.4564.objecter put_session s=0x7fc7041d9cc0 osd=1 3 2024-03-27T21:40:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.148+0000 7fc6eaffd640 20 client.4564.objecter _get_session s=0x7fc7041d9cc0 osd=1 3 2024-03-27T21:40:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.148+0000 7fc6eaffd640 15 client.4564.objecter _session_command_op_remove -1 3 2024-03-27T21:40:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.148+0000 7fc708c1e640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fc7040b4bc0 0x7fc7040b4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.148+0000 7fc708c1e640 10 monclient: get_auth_request con 0x7fc7040b4bc0 auth_method 0 2024-03-27T21:40:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.148+0000 7fc6eaffd640 20 client.4564.objecter get_session s=0x7fc7041d9cc0 osd=1 3 2024-03-27T21:40:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.148+0000 7fc6eaffd640 15 client.4564.objecter _session_command_op_assign 1 3 2024-03-27T21:40:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.148+0000 7fc6eaffd640 20 client.4564.objecter put_session s=0x7fc7041d9cc0 osd=1 4 2024-03-27T21:40:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.148+0000 7fc6eaffd640 10 client.4564.objecter _send_command 3 2024-03-27T21:40:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.148+0000 7fc6eaffd640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc704066d70 con 0x7fc7040b4bc0 2024-03-27T21:40:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.148+0000 7fc708c1e640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fc7040b4bc0 0x7fc7040b4fa0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.148+0000 7fc7017fa640 10 client.4564.objecter ms_handle_connect 0x7fc7040b4bc0 2024-03-27T21:40:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.149+0000 7fc7017fa640 1 -- 172.21.15.60:0/1696298781 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fc704066d70 con 0x7fc7040b4bc0 2024-03-27T21:40:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.149+0000 7fc7017fa640 10 client.4564.objecter ms_dispatch 0x7fc704000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.149+0000 7fc7017fa640 10 client.4564.objecter _finish_command 3 = system:0 2024-03-27T21:40:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.149+0000 7fc7017fa640 20 client.4564.objecter put_session s=0x7fc7041d9cc0 osd=1 4 2024-03-27T21:40:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.149+0000 7fc7017fa640 15 client.4564.objecter _session_command_op_remove 1 3 2024-03-27T21:40:19.156 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:19.156 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:19.164 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:40:19.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.179+0000 7fc6eaffd640 10 client.4564.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:19.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.179+0000 7fc6eaffd640 15 client.4564.objecter _session_command_op_assign -1 4 2024-03-27T21:40:19.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.179+0000 7fc6eaffd640 20 client.4564.objecter _get_session s=0x7fc7041d9cc0 osd=1 3 2024-03-27T21:40:19.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.179+0000 7fc6eaffd640 20 client.4564.objecter put_session s=0x7fc7041d9cc0 osd=1 3 2024-03-27T21:40:19.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.179+0000 7fc6eaffd640 20 client.4564.objecter _get_session s=0x7fc7041d9cc0 osd=1 3 2024-03-27T21:40:19.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.179+0000 7fc6eaffd640 15 client.4564.objecter _session_command_op_remove -1 4 2024-03-27T21:40:19.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.179+0000 7fc6eaffd640 20 client.4564.objecter get_session s=0x7fc7041d9cc0 osd=1 3 2024-03-27T21:40:19.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.179+0000 7fc6eaffd640 15 client.4564.objecter _session_command_op_assign 1 4 2024-03-27T21:40:19.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.179+0000 7fc6eaffd640 20 client.4564.objecter put_session s=0x7fc7041d9cc0 osd=1 4 2024-03-27T21:40:19.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.179+0000 7fc6eaffd640 10 client.4564.objecter _send_command 4 2024-03-27T21:40:19.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.179+0000 7fc6eaffd640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc7041d7090 con 0x7fc7040b4bc0 2024-03-27T21:40:19.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.182+0000 7fc7017fa640 1 -- 172.21.15.60:0/1696298781 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc7041d7090 con 0x7fc7040b4bc0 2024-03-27T21:40:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.182+0000 7fc7017fa640 10 client.4564.objecter ms_dispatch 0x7fc704000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.182+0000 7fc7017fa640 10 client.4564.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.182+0000 7fc7017fa640 20 client.4564.objecter put_session s=0x7fc7041d9cc0 osd=1 4 2024-03-27T21:40:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.182+0000 7fc7017fa640 15 client.4564.objecter _session_command_op_remove 1 4 2024-03-27T21:40:19.169 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.182+0000 7fc6eaffd640 10 client.4564.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.182+0000 7fc6eaffd640 15 client.4564.objecter _session_command_op_assign -1 5 2024-03-27T21:40:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.182+0000 7fc6eaffd640 20 client.4564.objecter _get_session osd=-1 returning homeless 2024-03-27T21:40:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.182+0000 7fc6eaffd640 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.182+0000 7fc6eaffd640 10 monclient: _renew_subs 2024-03-27T21:40:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.183+0000 7fc6eaffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.183+0000 7fc6eaffd640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=33}) v3 -- 0x7fc7041d6600 con 0x7fc70413b020 2024-03-27T21:40:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.183+0000 7fc6eaffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.183+0000 7fc6eaffd640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc7041d7260 con 0x7fc70413b020 2024-03-27T21:40:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.183+0000 7fc7017fa640 1 -- 172.21.15.60:0/1696298781 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6f8076520 con 0x7fc70413b020 2024-03-27T21:40:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.183+0000 7fc7017fa640 10 monclient: handle_get_version_reply finishing 1 version 32 2024-03-27T21:40:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.183+0000 7fc7027fc640 10 client.4564.objecter _check_command_map_dne tid 5 current 32 map_dne_bound 32 2024-03-27T21:40:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.183+0000 7fc7027fc640 10 client.4564.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:40:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.183+0000 7fc7027fc640 15 client.4564.objecter _session_command_op_remove -1 5 2024-03-27T21:40:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:40:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.183+0000 7fc6eaffd640 10 client.4564.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.183+0000 7fc6eaffd640 15 client.4564.objecter _session_command_op_assign -1 6 2024-03-27T21:40:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.183+0000 7fc6eaffd640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fc704066250 0x7fc704066660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.183+0000 7fc6eaffd640 20 client.4564.objecter _get_session s=0x7fc7041d7260 osd=3 3 2024-03-27T21:40:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.183+0000 7fc6eaffd640 20 client.4564.objecter put_session s=0x7fc7041d7260 osd=3 3 2024-03-27T21:40:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.183+0000 7fc6eaffd640 20 client.4564.objecter _get_session s=0x7fc7041d7260 osd=3 3 2024-03-27T21:40:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.183+0000 7fc6eaffd640 15 client.4564.objecter _session_command_op_remove -1 6 2024-03-27T21:40:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.183+0000 7fc6eaffd640 20 client.4564.objecter get_session s=0x7fc7041d7260 osd=3 3 2024-03-27T21:40:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.183+0000 7fc6eaffd640 15 client.4564.objecter _session_command_op_assign 3 6 2024-03-27T21:40:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.183+0000 7fc6eaffd640 20 client.4564.objecter put_session s=0x7fc7041d7260 osd=3 4 2024-03-27T21:40:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.183+0000 7fc6eaffd640 10 client.4564.objecter _send_command 6 2024-03-27T21:40:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.184+0000 7fc6eaffd640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc7041d69f0 con 0x7fc704066250 2024-03-27T21:40:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.184+0000 7fc703fff640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fc704066250 0x7fc704066660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.184+0000 7fc703fff640 10 monclient: get_auth_request con 0x7fc704066250 auth_method 0 2024-03-27T21:40:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.184+0000 7fc7017fa640 10 client.4564.objecter ms_handle_connect 0x7fc704066250 2024-03-27T21:40:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.184+0000 7fc703fff640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fc704066250 0x7fc704066660 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.185+0000 7fc7017fa640 1 -- 172.21.15.60:0/1696298781 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fc7041d69f0 con 0x7fc704066250 2024-03-27T21:40:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.185+0000 7fc7017fa640 10 client.4564.objecter ms_dispatch 0x7fc704000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:40:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.185+0000 7fc7017fa640 10 client.4564.objecter _finish_command 6 = system:0 2024-03-27T21:40:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.185+0000 7fc7017fa640 20 client.4564.objecter put_session s=0x7fc7041d7260 osd=3 4 2024-03-27T21:40:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.186+0000 7fc7017fa640 15 client.4564.objecter _session_command_op_remove 3 6 2024-03-27T21:40:19.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.214+0000 7fc6eaffd640 10 client.4564.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:19.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.214+0000 7fc6eaffd640 15 client.4564.objecter _session_command_op_assign -1 7 2024-03-27T21:40:19.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.214+0000 7fc6eaffd640 20 client.4564.objecter _get_session s=0x7fc7041d7260 osd=3 3 2024-03-27T21:40:19.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.214+0000 7fc6eaffd640 20 client.4564.objecter put_session s=0x7fc7041d7260 osd=3 3 2024-03-27T21:40:19.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.214+0000 7fc6eaffd640 20 client.4564.objecter _get_session s=0x7fc7041d7260 osd=3 3 2024-03-27T21:40:19.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.214+0000 7fc6eaffd640 15 client.4564.objecter _session_command_op_remove -1 7 2024-03-27T21:40:19.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.214+0000 7fc6eaffd640 20 client.4564.objecter get_session s=0x7fc7041d7260 osd=3 3 2024-03-27T21:40:19.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.214+0000 7fc6eaffd640 15 client.4564.objecter _session_command_op_assign 3 7 2024-03-27T21:40:19.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.214+0000 7fc6eaffd640 20 client.4564.objecter put_session s=0x7fc7041d7260 osd=3 4 2024-03-27T21:40:19.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.214+0000 7fc6eaffd640 10 client.4564.objecter _send_command 7 2024-03-27T21:40:19.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.214+0000 7fc6eaffd640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc7040b5c30 con 0x7fc704066250 2024-03-27T21:40:19.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.217+0000 7fc7017fa640 1 -- 172.21.15.60:0/1696298781 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc7040b5c30 con 0x7fc704066250 2024-03-27T21:40:19.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.217+0000 7fc7017fa640 10 client.4564.objecter ms_dispatch 0x7fc704000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:19.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.217+0000 7fc7017fa640 10 client.4564.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:19.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.217+0000 7fc7017fa640 20 client.4564.objecter put_session s=0x7fc7041d7260 osd=3 4 2024-03-27T21:40:19.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.217+0000 7fc7017fa640 15 client.4564.objecter _session_command_op_remove 3 7 2024-03-27T21:40:19.204 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:19.206 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:19.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.217+0000 7fc70ae82640 10 client.4564.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:19.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.217+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_assign -1 8 2024-03-27T21:40:19.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.217+0000 7fc70ae82640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fc6e0002f70 0x7fc6e0005430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:19.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.217+0000 7fc70ae82640 20 client.4564.objecter _get_session s=0x7fc6e0002860 osd=4 3 2024-03-27T21:40:19.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.217+0000 7fc70ae82640 20 client.4564.objecter put_session s=0x7fc6e0002860 osd=4 3 2024-03-27T21:40:19.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.217+0000 7fc70ae82640 20 client.4564.objecter _get_session s=0x7fc6e0002860 osd=4 3 2024-03-27T21:40:19.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.217+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_remove -1 8 2024-03-27T21:40:19.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.217+0000 7fc70ae82640 20 client.4564.objecter get_session s=0x7fc6e0002860 osd=4 3 2024-03-27T21:40:19.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.217+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_assign 4 8 2024-03-27T21:40:19.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.217+0000 7fc70ae82640 20 client.4564.objecter put_session s=0x7fc6e0002860 osd=4 4 2024-03-27T21:40:19.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.217+0000 7fc70ae82640 10 client.4564.objecter _send_command 8 2024-03-27T21:40:19.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.217+0000 7fc70ae82640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6e0008420 con 0x7fc6e0002f70 2024-03-27T21:40:19.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.218+0000 7fc70941f640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fc6e0002f70 0x7fc6e0005430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:19.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.218+0000 7fc70941f640 10 monclient: get_auth_request con 0x7fc6e0002f70 auth_method 0 2024-03-27T21:40:19.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.218+0000 7fc7017fa640 10 client.4564.objecter ms_handle_connect 0x7fc6e0002f70 2024-03-27T21:40:19.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.218+0000 7fc70941f640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fc6e0002f70 0x7fc6e0005430 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:19.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.219+0000 7fc7017fa640 1 -- 172.21.15.60:0/1696298781 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fc6e0008420 con 0x7fc6e0002f70 2024-03-27T21:40:19.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.219+0000 7fc7017fa640 10 client.4564.objecter ms_dispatch 0x7fc704000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:40:19.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.219+0000 7fc7017fa640 10 client.4564.objecter _finish_command 8 = system:0 2024-03-27T21:40:19.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.219+0000 7fc7017fa640 20 client.4564.objecter put_session s=0x7fc6e0002860 osd=4 4 2024-03-27T21:40:19.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.219+0000 7fc7017fa640 15 client.4564.objecter _session_command_op_remove 4 8 2024-03-27T21:40:19.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.248+0000 7fc70ae82640 10 client.4564.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:19.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.248+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_assign -1 9 2024-03-27T21:40:19.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.248+0000 7fc70ae82640 20 client.4564.objecter _get_session s=0x7fc6e0002860 osd=4 3 2024-03-27T21:40:19.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.248+0000 7fc70ae82640 20 client.4564.objecter put_session s=0x7fc6e0002860 osd=4 3 2024-03-27T21:40:19.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.248+0000 7fc70ae82640 20 client.4564.objecter _get_session s=0x7fc6e0002860 osd=4 3 2024-03-27T21:40:19.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.248+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_remove -1 9 2024-03-27T21:40:19.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.248+0000 7fc70ae82640 20 client.4564.objecter get_session s=0x7fc6e0002860 osd=4 3 2024-03-27T21:40:19.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.248+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_assign 4 9 2024-03-27T21:40:19.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.248+0000 7fc70ae82640 20 client.4564.objecter put_session s=0x7fc6e0002860 osd=4 4 2024-03-27T21:40:19.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.248+0000 7fc70ae82640 10 client.4564.objecter _send_command 9 2024-03-27T21:40:19.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.248+0000 7fc70ae82640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc6e0001100 con 0x7fc6e0002f70 2024-03-27T21:40:19.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.249+0000 7fc7017fa640 1 -- 172.21.15.60:0/1696298781 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc6e0001100 con 0x7fc6e0002f70 2024-03-27T21:40:19.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.250+0000 7fc7017fa640 10 client.4564.objecter ms_dispatch 0x7fc704000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:19.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.250+0000 7fc7017fa640 10 client.4564.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:19.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.250+0000 7fc7017fa640 20 client.4564.objecter put_session s=0x7fc6e0002860 osd=4 4 2024-03-27T21:40:19.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.250+0000 7fc7017fa640 15 client.4564.objecter _session_command_op_remove 4 9 2024-03-27T21:40:19.237 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:19.239 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:19.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.250+0000 7fc70ae82640 10 client.4564.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:19.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.250+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_assign -1 10 2024-03-27T21:40:19.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.250+0000 7fc70ae82640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fc6e0007ec0 0x7fc6e0005ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:19.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.250+0000 7fc70ae82640 20 client.4564.objecter _get_session s=0x7fc6e0002560 osd=5 3 2024-03-27T21:40:19.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.250+0000 7fc70ae82640 20 client.4564.objecter put_session s=0x7fc6e0002560 osd=5 3 2024-03-27T21:40:19.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.250+0000 7fc70ae82640 20 client.4564.objecter _get_session s=0x7fc6e0002560 osd=5 3 2024-03-27T21:40:19.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.250+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_remove -1 10 2024-03-27T21:40:19.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.250+0000 7fc70ae82640 20 client.4564.objecter get_session s=0x7fc6e0002560 osd=5 3 2024-03-27T21:40:19.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.250+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_assign 5 10 2024-03-27T21:40:19.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.250+0000 7fc70ae82640 20 client.4564.objecter put_session s=0x7fc6e0002560 osd=5 4 2024-03-27T21:40:19.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.250+0000 7fc70ae82640 10 client.4564.objecter _send_command 10 2024-03-27T21:40:19.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.250+0000 7fc70ae82640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6e000bdf0 con 0x7fc6e0007ec0 2024-03-27T21:40:19.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.250+0000 7fc708c1e640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fc6e0007ec0 0x7fc6e0005ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:19.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.251+0000 7fc708c1e640 10 monclient: get_auth_request con 0x7fc6e0007ec0 auth_method 0 2024-03-27T21:40:19.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.251+0000 7fc708c1e640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fc6e0007ec0 0x7fc6e0005ab0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:19.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.251+0000 7fc7017fa640 10 client.4564.objecter ms_handle_connect 0x7fc6e0007ec0 2024-03-27T21:40:19.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.252+0000 7fc7017fa640 1 -- 172.21.15.60:0/1696298781 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fc6e000bdf0 con 0x7fc6e0007ec0 2024-03-27T21:40:19.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.252+0000 7fc7017fa640 10 client.4564.objecter ms_dispatch 0x7fc704000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:40:19.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.252+0000 7fc7017fa640 10 client.4564.objecter _finish_command 10 = system:0 2024-03-27T21:40:19.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.252+0000 7fc7017fa640 20 client.4564.objecter put_session s=0x7fc6e0002560 osd=5 4 2024-03-27T21:40:19.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.252+0000 7fc7017fa640 15 client.4564.objecter _session_command_op_remove 5 10 2024-03-27T21:40:19.253 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:19.267+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:19.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.280+0000 7fc70ae82640 10 client.4564.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.280+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_assign -1 11 2024-03-27T21:40:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.280+0000 7fc70ae82640 20 client.4564.objecter _get_session s=0x7fc6e0002560 osd=5 3 2024-03-27T21:40:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.280+0000 7fc70ae82640 20 client.4564.objecter put_session s=0x7fc6e0002560 osd=5 3 2024-03-27T21:40:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.280+0000 7fc70ae82640 20 client.4564.objecter _get_session s=0x7fc6e0002560 osd=5 3 2024-03-27T21:40:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.280+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_remove -1 11 2024-03-27T21:40:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.280+0000 7fc70ae82640 20 client.4564.objecter get_session s=0x7fc6e0002560 osd=5 3 2024-03-27T21:40:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.280+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_assign 5 11 2024-03-27T21:40:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.280+0000 7fc70ae82640 20 client.4564.objecter put_session s=0x7fc6e0002560 osd=5 4 2024-03-27T21:40:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.280+0000 7fc70ae82640 10 client.4564.objecter _send_command 11 2024-03-27T21:40:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.280+0000 7fc70ae82640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc6e00013a0 con 0x7fc6e0007ec0 2024-03-27T21:40:19.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.282+0000 7fc7017fa640 1 -- 172.21.15.60:0/1696298781 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc6e00013a0 con 0x7fc6e0007ec0 2024-03-27T21:40:19.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.282+0000 7fc7017fa640 10 client.4564.objecter ms_dispatch 0x7fc704000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:19.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.282+0000 7fc7017fa640 10 client.4564.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:19.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.282+0000 7fc7017fa640 20 client.4564.objecter put_session s=0x7fc6e0002560 osd=5 4 2024-03-27T21:40:19.269 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:19.271 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:19.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.282+0000 7fc7017fa640 15 client.4564.objecter _session_command_op_remove 5 11 2024-03-27T21:40:19.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.282+0000 7fc70ae82640 10 client.4564.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:19.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.283+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_assign -1 12 2024-03-27T21:40:19.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.283+0000 7fc70ae82640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fc6e0001a20 0x7fc6e0001ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:19.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.283+0000 7fc70ae82640 20 client.4564.objecter _get_session s=0x7fc6e00013a0 osd=6 3 2024-03-27T21:40:19.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.283+0000 7fc70ae82640 20 client.4564.objecter put_session s=0x7fc6e00013a0 osd=6 3 2024-03-27T21:40:19.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.283+0000 7fc70ae82640 20 client.4564.objecter _get_session s=0x7fc6e00013a0 osd=6 3 2024-03-27T21:40:19.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.283+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_remove -1 12 2024-03-27T21:40:19.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.283+0000 7fc70ae82640 20 client.4564.objecter get_session s=0x7fc6e00013a0 osd=6 3 2024-03-27T21:40:19.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.283+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_assign 6 12 2024-03-27T21:40:19.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.283+0000 7fc70ae82640 20 client.4564.objecter put_session s=0x7fc6e00013a0 osd=6 4 2024-03-27T21:40:19.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.283+0000 7fc70ae82640 10 client.4564.objecter _send_command 12 2024-03-27T21:40:19.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.283+0000 7fc70ae82640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6e000f180 con 0x7fc6e0001a20 2024-03-27T21:40:19.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.283+0000 7fc703fff640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fc6e0001a20 0x7fc6e0001ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:19.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.283+0000 7fc703fff640 10 monclient: get_auth_request con 0x7fc6e0001a20 auth_method 0 2024-03-27T21:40:19.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.284+0000 7fc703fff640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fc6e0001a20 0x7fc6e0001ea0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:19.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.284+0000 7fc7017fa640 10 client.4564.objecter ms_handle_connect 0x7fc6e0001a20 2024-03-27T21:40:19.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.285+0000 7fc7017fa640 1 -- 172.21.15.60:0/1696298781 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fc6e000f180 con 0x7fc6e0001a20 2024-03-27T21:40:19.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.285+0000 7fc7017fa640 10 client.4564.objecter ms_dispatch 0x7fc704000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:40:19.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.285+0000 7fc7017fa640 10 client.4564.objecter _finish_command 12 = system:0 2024-03-27T21:40:19.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.285+0000 7fc7017fa640 20 client.4564.objecter put_session s=0x7fc6e00013a0 osd=6 4 2024-03-27T21:40:19.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.285+0000 7fc7017fa640 15 client.4564.objecter _session_command_op_remove 6 12 2024-03-27T21:40:19.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.313+0000 7fc70ae82640 10 client.4564.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:19.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.313+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_assign -1 13 2024-03-27T21:40:19.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.313+0000 7fc70ae82640 20 client.4564.objecter _get_session s=0x7fc6e00013a0 osd=6 3 2024-03-27T21:40:19.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.313+0000 7fc70ae82640 20 client.4564.objecter put_session s=0x7fc6e00013a0 osd=6 3 2024-03-27T21:40:19.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.313+0000 7fc70ae82640 20 client.4564.objecter _get_session s=0x7fc6e00013a0 osd=6 3 2024-03-27T21:40:19.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.313+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_remove -1 13 2024-03-27T21:40:19.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.313+0000 7fc70ae82640 20 client.4564.objecter get_session s=0x7fc6e00013a0 osd=6 3 2024-03-27T21:40:19.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.313+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_assign 6 13 2024-03-27T21:40:19.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.313+0000 7fc70ae82640 20 client.4564.objecter put_session s=0x7fc6e00013a0 osd=6 4 2024-03-27T21:40:19.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.313+0000 7fc70ae82640 10 client.4564.objecter _send_command 13 2024-03-27T21:40:19.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.313+0000 7fc70ae82640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc6e000e000 con 0x7fc6e0001a20 2024-03-27T21:40:19.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.315+0000 7fc7017fa640 1 -- 172.21.15.60:0/1696298781 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc6e000e000 con 0x7fc6e0001a20 2024-03-27T21:40:19.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.315+0000 7fc7017fa640 10 client.4564.objecter ms_dispatch 0x7fc704000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:19.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.315+0000 7fc7017fa640 10 client.4564.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:19.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.315+0000 7fc7017fa640 20 client.4564.objecter put_session s=0x7fc6e00013a0 osd=6 4 2024-03-27T21:40:19.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.315+0000 7fc7017fa640 15 client.4564.objecter _session_command_op_remove 6 13 2024-03-27T21:40:19.302 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:19.304 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:19.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.315+0000 7fc70ae82640 10 client.4564.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:19.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.315+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_assign -1 14 2024-03-27T21:40:19.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.315+0000 7fc70ae82640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fc6e000ab40 0x7fc6e000aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:19.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.316+0000 7fc70ae82640 20 client.4564.objecter _get_session s=0x7fc6e000e000 osd=7 3 2024-03-27T21:40:19.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.316+0000 7fc70ae82640 20 client.4564.objecter put_session s=0x7fc6e000e000 osd=7 3 2024-03-27T21:40:19.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.316+0000 7fc70ae82640 20 client.4564.objecter _get_session s=0x7fc6e000e000 osd=7 3 2024-03-27T21:40:19.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.316+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_remove -1 14 2024-03-27T21:40:19.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.316+0000 7fc70ae82640 20 client.4564.objecter get_session s=0x7fc6e000e000 osd=7 3 2024-03-27T21:40:19.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.316+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_assign 7 14 2024-03-27T21:40:19.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.316+0000 7fc70ae82640 20 client.4564.objecter put_session s=0x7fc6e000e000 osd=7 4 2024-03-27T21:40:19.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.316+0000 7fc70ae82640 10 client.4564.objecter _send_command 14 2024-03-27T21:40:19.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.316+0000 7fc70ae82640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6e000b700 con 0x7fc6e000ab40 2024-03-27T21:40:19.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.316+0000 7fc70941f640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fc6e000ab40 0x7fc6e000aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:19.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.316+0000 7fc70941f640 10 monclient: get_auth_request con 0x7fc6e000ab40 auth_method 0 2024-03-27T21:40:19.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.316+0000 7fc7017fa640 10 client.4564.objecter ms_handle_connect 0x7fc6e000ab40 2024-03-27T21:40:19.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.316+0000 7fc70941f640 1 --2- 172.21.15.60:0/1696298781 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fc6e000ab40 0x7fc6e000aff0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:19.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.317+0000 7fc7017fa640 1 -- 172.21.15.60:0/1696298781 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fc6e000b700 con 0x7fc6e000ab40 2024-03-27T21:40:19.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.317+0000 7fc7017fa640 10 client.4564.objecter ms_dispatch 0x7fc704000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:40:19.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.317+0000 7fc7017fa640 10 client.4564.objecter _finish_command 14 = system:0 2024-03-27T21:40:19.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.317+0000 7fc7017fa640 20 client.4564.objecter put_session s=0x7fc6e000e000 osd=7 4 2024-03-27T21:40:19.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.317+0000 7fc7017fa640 15 client.4564.objecter _session_command_op_remove 7 14 2024-03-27T21:40:19.312 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:19.312 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:19.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:19.312 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:19.313 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:19.321 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:40:19.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.345+0000 7fc70ae82640 10 client.4564.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:19.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.345+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_assign -1 15 2024-03-27T21:40:19.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.346+0000 7fc70ae82640 20 client.4564.objecter _get_session s=0x7fc6e000e000 osd=7 3 2024-03-27T21:40:19.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.346+0000 7fc70ae82640 20 client.4564.objecter put_session s=0x7fc6e000e000 osd=7 3 2024-03-27T21:40:19.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.346+0000 7fc70ae82640 20 client.4564.objecter _get_session s=0x7fc6e000e000 osd=7 3 2024-03-27T21:40:19.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.346+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_remove -1 15 2024-03-27T21:40:19.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.346+0000 7fc70ae82640 20 client.4564.objecter get_session s=0x7fc6e000e000 osd=7 3 2024-03-27T21:40:19.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.346+0000 7fc70ae82640 15 client.4564.objecter _session_command_op_assign 7 15 2024-03-27T21:40:19.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.346+0000 7fc70ae82640 20 client.4564.objecter put_session s=0x7fc6e000e000 osd=7 4 2024-03-27T21:40:19.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.346+0000 7fc70ae82640 10 client.4564.objecter _send_command 15 2024-03-27T21:40:19.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.346+0000 7fc70ae82640 1 -- 172.21.15.60:0/1696298781 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc6e000b900 con 0x7fc6e000ab40 2024-03-27T21:40:19.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.348+0000 7fc7017fa640 1 -- 172.21.15.60:0/1696298781 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fc6e000b900 con 0x7fc6e000ab40 2024-03-27T21:40:19.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.348+0000 7fc7017fa640 10 client.4564.objecter ms_dispatch 0x7fc704000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:19.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.348+0000 7fc7017fa640 10 client.4564.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:19.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.348+0000 7fc7017fa640 20 client.4564.objecter put_session s=0x7fc6e000e000 osd=7 4 2024-03-27T21:40:19.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.348+0000 7fc7017fa640 15 client.4564.objecter _session_command_op_remove 7 15 2024-03-27T21:40:19.335 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:19.336 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:19.340 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:40:19.341 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:40:19.363 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:19.376+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:19.398 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:19.412+0000 7fa1a942a640 -1 osd.2 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-27T21:40:19.431 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:19.444+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:19.467 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:19.467 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:19.468 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:19.468 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:19.478 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:40:19.532 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:19.546+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:19.623 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:19.623 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:19.623 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:19.624 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:19.624 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:19.632 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:40:19.634 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:19.647+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:19.697 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-27T21:40:19.735 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:19.749+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:19.738 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:19.738 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:19.738 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:19.738 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:19.738 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:19.748 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:40:19.836 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:19.850+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:19.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.870+0000 7f7f09a5f640 10 monclient: get_monmap_and_config 2024-03-27T21:40:19.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.870+0000 7f7f09a5f640 10 monclient: build_initial_monmap 2024-03-27T21:40:19.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.870+0000 7f7f09a5f640 10 monclient: monmap: 2024-03-27T21:40:19.857 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:19.857 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:19.857 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:19.871068+0000 2024-03-27T21:40:19.857 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:19.871068+0000 2024-03-27T21:40:19.857 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:19.857 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:19.858 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:19.858 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:19.858 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:19.858 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:19.858 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:19.858 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:19.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.870+0000 7f7f09a5f640 1 Processor -- start 2024-03-27T21:40:19.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.871+0000 7f7f09a5f640 1 -- start start 2024-03-27T21:40:19.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.871+0000 7f7f09a5f640 10 monclient: init 2024-03-27T21:40:19.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.871+0000 7f7f09a5f640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:19.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.871+0000 7f7f09a5f640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-03-27T21:40:19.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.871+0000 7f7f09a5f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7f040b40b0 0x7f7f040a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:19.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.871+0000 7f7f09a5f640 10 monclient(hunting): picked mon.noname-a con 0x7f7f040b40b0 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:19.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f09a5f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7f0413b020 0x7f7f0413b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:19.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f09a5f640 10 monclient(hunting): picked mon.noname-c con 0x7f7f0413b020 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:19.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f09a5f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7f041416f0 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:19.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f09a5f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f040b3830 con 0x7f7f041416f0 2024-03-27T21:40:19.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f09a5f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:19.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f09a5f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f040b4490 con 0x7f7f040b40b0 2024-03-27T21:40:19.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f09a5f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:19.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f09a5f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f040aa520 con 0x7f7f0413b020 2024-03-27T21:40:19.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f09a5f640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:19.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f09a5f640 10 monclient(hunting): authenticate will time out at 861.093000s 2024-03-27T21:40:19.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f02ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7f040b40b0 0x7f7f040a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:19.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f02ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7f040b40b0 0x7f7f040a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35986/0 (socket says 172.21.15.60:35986) 2024-03-27T21:40:19.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f02ffd640 1 -- 172.21.15.60:0/4068465235 learned_addr learned my addr 172.21.15.60:0/4068465235 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:19.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f02ffd640 10 monclient(hunting): get_auth_request con 0x7f7f040b40b0 auth_method 0 2024-03-27T21:40:19.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:19.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:19.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:19.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f02ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:19.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f02ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:19.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f02ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:19.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f02ffd640 10 monclient(hunting): handle_auth_done global_id 4599 payload 274 2024-03-27T21:40:19.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f02ffd640 1 -- 172.21.15.60:0/4068465235 >> v2:172.21.15.191:3300/0 conn(0x7f7f0413b020 msgr2=0x7f7f0413b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:19.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f02ffd640 1 --2- 172.21.15.60:0/4068465235 >> v2:172.21.15.191:3300/0 conn(0x7f7f0413b020 0x7f7f0413b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:19.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f02ffd640 1 -- 172.21.15.60:0/4068465235 >> v1:172.21.15.60:6789/0 conn(0x7f7f041416f0 legacy=0x7f7f04141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:19.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f02ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:40:19.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f02ffd640 1 monclient: found mon.noname-a 2024-03-27T21:40:19.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:19.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f02ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:19.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.872+0000 7f7f02ffd640 1 -- 172.21.15.60:0/4068465235 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f040a9470 con 0x7f7f040b40b0 2024-03-27T21:40:19.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.873+0000 7f7f02ffd640 1 --2- 172.21.15.60:0/4068465235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f040b40b0 0x7f7f040a8b90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7f040b47e0 tx=0x7f7ef402b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e0bcf10ba73ad6 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:19.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.873+0000 7f7f01ffb640 1 -- 172.21.15.60:0/4068465235 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4030870 con 0x7f7f040b40b0 2024-03-27T21:40:19.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.873+0000 7f7f01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:19.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.874+0000 7f7f01ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:19.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.874+0000 7f7f01ffb640 10 monclient: dump: 2024-03-27T21:40:19.875 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:19.875 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:19.875 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:19.876 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:19.876 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:19.876 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:19.876 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:19.876 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:19.876 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:19.876 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:19.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.874+0000 7f7f01ffb640 10 monclient: _finish_auth 0 2024-03-27T21:40:19.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.874+0000 7f7f01ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:40:19.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.874+0000 7f7f01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:19.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.874+0000 7f7f09a5f640 5 monclient: authenticate success, global_id 4599 2024-03-27T21:40:19.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.874+0000 7f7f09a5f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:19.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.874+0000 7f7f01ffb640 1 -- 172.21.15.60:0/4068465235 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef40309d0 con 0x7f7f040b40b0 2024-03-27T21:40:19.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.874+0000 7f7f01ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:19.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.874+0000 7f7f09a5f640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:19.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.874+0000 7f7f09a5f640 4 set_mon_vals no callback set 2024-03-27T21:40:19.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.874+0000 7f7f01ffb640 1 -- 172.21.15.60:0/4068465235 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4030d60 con 0x7f7f040b40b0 2024-03-27T21:40:19.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.874+0000 7f7f01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:19.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.874+0000 7f7f01ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:19.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.874+0000 7f7f01ffb640 10 monclient: dump: 2024-03-27T21:40:19.877 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:19.877 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:19.877 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:19.877 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:19.877 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:19.877 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:19.877 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:19.877 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:19.877 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:19.878 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:19.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.875+0000 7f7f09a5f640 10 monclient: shutdown 2024-03-27T21:40:19.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.875+0000 7f7f09a5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:19.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.875+0000 7f7f09a5f640 1 -- 172.21.15.60:0/4068465235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f040b40b0 msgr2=0x7f7f040a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:19.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.875+0000 7f7f09a5f640 1 --2- 172.21.15.60:0/4068465235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f040b40b0 0x7f7f040a8b90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7f040b47e0 tx=0x7f7ef402b9b0 comp rx=0 tx=0).stop 2024-03-27T21:40:19.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.875+0000 7f7f09a5f640 1 -- 172.21.15.60:0/4068465235 shutdown_connections 2024-03-27T21:40:19.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.875+0000 7f7f09a5f640 1 --2- 172.21.15.60:0/4068465235 >> v2:172.21.15.191:3300/0 conn(0x7f7f0413b020 0x7f7f0413b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:19.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.875+0000 7f7f09a5f640 1 --2- 172.21.15.60:0/4068465235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f040b40b0 0x7f7f040a8b90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:19.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.875+0000 7f7f09a5f640 1 -- 172.21.15.60:0/4068465235 >> 172.21.15.60:0/4068465235 conn(0x7f7f040608e0 msgr2=0x7f7f04060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:19.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.875+0000 7f7f09a5f640 1 -- 172.21.15.60:0/4068465235 shutdown_connections 2024-03-27T21:40:19.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.875+0000 7f7f09a5f640 1 -- 172.21.15.60:0/4068465235 wait complete. 2024-03-27T21:40:19.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.876+0000 7f7f09a5f640 10 monclient: build_initial_monmap 2024-03-27T21:40:19.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.876+0000 7f7f09a5f640 10 monclient: monmap: 2024-03-27T21:40:19.878 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:19.878 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:19.879 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:19.879 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:19.879 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:19.879 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:19.879 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:19.879 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:19.879 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:19.879 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:19.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.876+0000 7f7f09a5f640 1 librados: starting msgr at 2024-03-27T21:40:19.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.876+0000 7f7f09a5f640 1 librados: starting objecter 2024-03-27T21:40:19.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.876+0000 7f7f09a5f640 1 Processor -- start 2024-03-27T21:40:19.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.877+0000 7f7f09a5f640 1 -- start start 2024-03-27T21:40:19.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.877+0000 7f7f09a5f640 1 librados: setting wanted keys 2024-03-27T21:40:19.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.877+0000 7f7f09a5f640 1 librados: calling monclient init 2024-03-27T21:40:19.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.877+0000 7f7f09a5f640 10 monclient: init 2024-03-27T21:40:19.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.877+0000 7f7f09a5f640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:19.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.877+0000 7f7f09a5f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:40:19.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.877+0000 7f7f09a5f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7f040b40b0 0x7f7f040ab320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:19.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.877+0000 7f7f09a5f640 10 monclient(hunting): picked mon.noname-c con 0x7f7f040b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:19.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f09a5f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f0413b020 0x7f7f041ddf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:19.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f09a5f640 10 monclient(hunting): picked mon.noname-b con 0x7f7f0413b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:19.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f09a5f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f041416f0 0x7f7f041de470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:19.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f09a5f640 10 monclient(hunting): picked mon.noname-a con 0x7f7f041416f0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:19.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f09a5f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:19.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f09a5f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f040aaa40 con 0x7f7f041416f0 2024-03-27T21:40:19.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f09a5f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:19.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f09a5f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0406f7a0 con 0x7f7f0413b020 2024-03-27T21:40:19.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f09a5f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:19.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f09a5f640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f040aae00 con 0x7f7f040b40b0 2024-03-27T21:40:19.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f09a5f640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:19.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f037fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f041416f0 0x7f7f041de470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:19.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f037fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f041416f0 0x7f7f041de470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35988/0 (socket says 172.21.15.60:35988) 2024-03-27T21:40:19.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f037fe640 1 -- 172.21.15.60:0/949680238 learned_addr learned my addr 172.21.15.60:0/949680238 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:19.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f09a5f640 10 monclient(hunting): authenticate will time out at 861.099085s 2024-03-27T21:40:19.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f027fc640 1 --2- 172.21.15.60:0/949680238 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f0413b020 0x7f7f041ddf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:19.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f02ffd640 1 --2- 172.21.15.60:0/949680238 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7f040b40b0 0x7f7f040ab320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:19.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f02ffd640 10 monclient(hunting): get_auth_request con 0x7f7f040b40b0 auth_method 0 2024-03-27T21:40:19.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:19.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:19.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:19.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f037fe640 10 monclient(hunting): get_auth_request con 0x7f7f041416f0 auth_method 0 2024-03-27T21:40:19.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:19.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f037fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:19.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:19.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f027fc640 10 monclient(hunting): get_auth_request con 0x7f7f0413b020 auth_method 0 2024-03-27T21:40:19.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f027fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:19.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f027fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:19.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:19.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:19.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:19.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:19.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f02ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:19.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f02ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:19.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f02ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:19.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f037fe640 10 monclient(hunting): handle_auth_done global_id 4605 payload 824 2024-03-27T21:40:19.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f037fe640 1 -- 172.21.15.60:0/949680238 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7f040b40b0 msgr2=0x7f7f040ab320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:19.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f037fe640 1 --2- 172.21.15.60:0/949680238 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7f040b40b0 0x7f7f040ab320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:19.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f037fe640 1 -- 172.21.15.60:0/949680238 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f0413b020 msgr2=0x7f7f041ddf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:19.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f037fe640 1 --2- 172.21.15.60:0/949680238 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f0413b020 0x7f7f041ddf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:19.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f037fe640 10 monclient: _finish_hunting 0 2024-03-27T21:40:19.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f037fe640 1 monclient: found mon.noname-a 2024-03-27T21:40:19.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:19.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f037fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:19.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f037fe640 1 -- 172.21.15.60:0/949680238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef40094d0 con 0x7f7f041416f0 2024-03-27T21:40:19.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f027fc640 1 --2- 172.21.15.60:0/949680238 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f0413b020 0x7f7f041ddf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:40:19.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f02ffd640 1 --2- 172.21.15.60:0/949680238 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7f040b40b0 0x7f7f040ab320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:19.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7f037fe640 1 --2- 172.21.15.60:0/949680238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f041416f0 0x7f7f041de470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7efc00c6b0 tx=0x7f7efc00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:19.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.878+0000 7f7eebfff640 10 client.?.objecter ms_handle_connect 0x7f7f041416f0 2024-03-27T21:40:19.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7eebfff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:19.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7eebfff640 1 -- 172.21.15.60:0/949680238 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc003b70 con 0x7f7f041416f0 2024-03-27T21:40:19.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7eebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:19.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7eebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:19.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7eebfff640 10 monclient: dump: 2024-03-27T21:40:19.884 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:19.884 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:19.884 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:19.884 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:19.884 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:19.884 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:19.884 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:19.884 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:19.885 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:19.885 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:19.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7eebfff640 10 monclient: _finish_auth 0 2024-03-27T21:40:19.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7eebfff640 10 monclient: _check_auth_tickets 2024-03-27T21:40:19.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7eebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:19.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7eebfff640 1 -- 172.21.15.60:0/949680238 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7efc003cf0 con 0x7f7f041416f0 2024-03-27T21:40:19.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7eebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:19.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7eebfff640 1 -- 172.21.15.60:0/949680238 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc003ea0 con 0x7f7f041416f0 2024-03-27T21:40:19.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7f09a5f640 5 monclient: authenticate success, global_id 4605 2024-03-27T21:40:19.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7eebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:19.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7eebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:19.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7eebfff640 10 monclient: dump: 2024-03-27T21:40:19.885 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:19.886 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:19.886 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:19.886 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:19.886 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:19.886 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:19.886 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:19.886 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:19.886 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:19.886 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:19.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7f09a5f640 10 monclient: _renew_subs 2024-03-27T21:40:19.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7f09a5f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:19.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7f09a5f640 1 -- 172.21.15.60:0/949680238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f041de9b0 con 0x7f7f041416f0 2024-03-27T21:40:19.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7f017fa640 4 set_mon_vals no callback set 2024-03-27T21:40:19.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7f09a5f640 10 client.4605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:19.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7f09a5f640 10 monclient: _renew_subs 2024-03-27T21:40:19.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7f09a5f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:19.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7f09a5f640 1 -- 172.21.15.60:0/949680238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f040abf20 con 0x7f7f041416f0 2024-03-27T21:40:19.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.879+0000 7f7f09a5f640 1 librados: init done 2024-03-27T21:40:19.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.881+0000 7f7f09a5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:19.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.881+0000 7f7f09a5f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:19.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.881+0000 7f7f09a5f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:19.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.881+0000 7f7f09a5f640 1 -- 172.21.15.60:0/949680238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f041ddc50 con 0x7f7f041416f0 2024-03-27T21:40:19.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.881+0000 7f7eebfff640 1 -- 172.21.15.60:0/949680238 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7efc005310 con 0x7f7f041416f0 2024-03-27T21:40:19.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.882+0000 7f7eebfff640 1 --2- 172.21.15.60:0/949680238 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f7ef006c590 0x7f7ef006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:19.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.882+0000 7f7eebfff640 1 -- 172.21.15.60:0/949680238 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f7efc088f10 con 0x7f7f041416f0 2024-03-27T21:40:19.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.882+0000 7f7eebfff640 10 client.4605.objecter ms_dispatch 0x7f7f04000b90 osd_map(32..32 src has 1..32) v4 2024-03-27T21:40:19.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.882+0000 7f7eebfff640 3 client.4605.objecter handle_osd_map got epochs [32,32] > 0 2024-03-27T21:40:19.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.882+0000 7f7eebfff640 3 client.4605.objecter handle_osd_map decoding full epoch 32 2024-03-27T21:40:19.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.882+0000 7f7eebfff640 20 client.4605.objecter dump_active .. 0 homeless 2024-03-27T21:40:19.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.882+0000 7f7f02ffd640 1 --2- 172.21.15.60:0/949680238 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f7ef006c590 0x7f7ef006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:19.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.882+0000 7f7f02ffd640 10 monclient: get_auth_request con 0x7f7ef006c590 auth_method 0 2024-03-27T21:40:19.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.883+0000 7f7f02ffd640 1 --2- 172.21.15.60:0/949680238 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f7ef006c590 0x7f7ef006ea50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4036000 tx=0x7f7ef4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:19.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.883+0000 7f7eebfff640 10 client.4605.objecter ms_handle_connect 0x7f7ef006c590 2024-03-27T21:40:19.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.890+0000 7f7eebfff640 1 -- 172.21.15.60:0/949680238 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7f041ddc50 con 0x7f7f041416f0 2024-03-27T21:40:19.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.890+0000 7f7eebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:19.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:19.890+0000 7f7eebfff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:19.894 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:19.894 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:19.894 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:19.894 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:19.903 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:40:19.937 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:19.951+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:20.006 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:20.007 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:20.007 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:20.007 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:20.007 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:20.015 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:40:20.038 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:20.051+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:20.119 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:20.119 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:20.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:20.120 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:20.120 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:20.128 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:40:20.139 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:20.153+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:20.233 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:20.233 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:20.233 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:20.233 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:20.240 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:20.253+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:20.242 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:40:20.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.275+0000 7f7f09a5f640 10 start_mon_command cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:40:20.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.275+0000 7f7f09a5f640 10 monclient: _send_command 2 [{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:40:20.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.275+0000 7f7f09a5f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:20.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.275+0000 7f7f09a5f640 1 -- 172.21.15.60:0/949680238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f7f041ddc50 con 0x7f7f041416f0 2024-03-27T21:40:20.341 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:20.354+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:20.345 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:20.345 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:20.346 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:20.346 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:20.346 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:20.354 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:40:20.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.409+0000 7f7eebfff640 1 -- 172.21.15.60:0/949680238 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v33) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7f041ddc50 con 0x7f7f041416f0 2024-03-27T21:40:20.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.409+0000 7f7eebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:40:20.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.409+0000 7f7eebfff640 10 monclient: _finish_command 2 = system:0 nodeep-scrub is set 2024-03-27T21:40:20.396 INFO:teuthology.orchestra.run.smithi060.stderr:nodeep-scrub is set 2024-03-27T21:40:20.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.412+0000 7f7f09a5f640 10 librados: watch_flush enter 2024-03-27T21:40:20.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.412+0000 7f7f09a5f640 10 librados: watch_flush exit 2024-03-27T21:40:20.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.412+0000 7f7f09a5f640 20 client.4605.objecter shutdown clearing up homeless session... 2024-03-27T21:40:20.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.412+0000 7f7f09a5f640 10 client.4605.objecter successfully canceled tick 2024-03-27T21:40:20.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.413+0000 7f7f09a5f640 1 -- 172.21.15.60:0/949680238 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f7ef006c590 msgr2=0x7f7ef006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:20.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.413+0000 7f7f09a5f640 1 --2- 172.21.15.60:0/949680238 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f7ef006c590 0x7f7ef006ea50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4036000 tx=0x7f7ef4008030 comp rx=0 tx=0).stop 2024-03-27T21:40:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.413+0000 7f7f09a5f640 10 monclient: shutdown 2024-03-27T21:40:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.413+0000 7f7f09a5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.413+0000 7f7f09a5f640 1 -- 172.21.15.60:0/949680238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f041416f0 msgr2=0x7f7f041de470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.413+0000 7f7f09a5f640 1 --2- 172.21.15.60:0/949680238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f041416f0 0x7f7f041de470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7efc00c6b0 tx=0x7f7efc00cb70 comp rx=0 tx=0).stop 2024-03-27T21:40:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.413+0000 7f7f09a5f640 1 -- 172.21.15.60:0/949680238 shutdown_connections 2024-03-27T21:40:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.413+0000 7f7f09a5f640 1 --2- 172.21.15.60:0/949680238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f041416f0 0x7f7f041de470 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.413+0000 7f7f09a5f640 1 --2- 172.21.15.60:0/949680238 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f7ef006c590 0x7f7ef006ea50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.413+0000 7f7f09a5f640 1 --2- 172.21.15.60:0/949680238 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f0413b020 0x7f7f041ddf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.413+0000 7f7f09a5f640 1 --2- 172.21.15.60:0/949680238 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7f040b40b0 0x7f7f040ab320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.413+0000 7f7f09a5f640 1 -- 172.21.15.60:0/949680238 >> 172.21.15.60:0/949680238 conn(0x7f7f040608e0 msgr2=0x7f7f0414b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:20.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.414+0000 7f7f09a5f640 1 -- 172.21.15.60:0/949680238 shutdown_connections 2024-03-27T21:40:20.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.414+0000 7f7f09a5f640 1 -- 172.21.15.60:0/949680238 wait complete. 2024-03-27T21:40:20.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:20.414+0000 7f7f09a5f640 1 librados: shutdown 2024-03-27T21:40:20.442 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:20.455+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:20.506 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:20.506 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:20.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:20.506 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-03-27T21:40:20.507 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.507 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:20.507 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:20.507 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 41.083034380000001, 2024-03-27T21:40:20.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43537978700000002, 2024-03-27T21:40:20.507 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:20.507 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:20.507 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:20.507 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:20.507 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:20.507 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-03-27T21:40:20.507 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.507 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:20.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:20.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:20.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:20.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:20.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:20.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432019+0000", 2024-03-27T21:40:20.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-03-27T21:40:20.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.509 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:20.509 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432027+0000", 2024-03-27T21:40:20.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1200000000000002e-06 2024-03-27T21:40:20.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.509 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:20.509 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432039+0000", 2024-03-27T21:40:20.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1123e-05 2024-03-27T21:40:20.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:20.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432051+0000", 2024-03-27T21:40:20.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-03-27T21:40:20.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.510 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:20.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432157+0000", 2024-03-27T21:40:20.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106846 2024-03-27T21:40:20.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.510 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-03-27T21:40:20.511 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432176+0000", 2024-03-27T21:40:20.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8490000000000001e-05 2024-03-27T21:40:20.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.511 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:20.511 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866170+0000", 2024-03-27T21:40:20.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43399370100000001 2024-03-27T21:40:20.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.511 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:20.511 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866335+0000", 2024-03-27T21:40:20.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016559699999999999 2024-03-27T21:40:20.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:20.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866495+0000", 2024-03-27T21:40:20.512 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015944200000000001 2024-03-27T21:40:20.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:20.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867012+0000", 2024-03-27T21:40:20.512 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0005176 2024-03-27T21:40:20.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:20.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867334+0000", 2024-03-27T21:40:20.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032158600000000001 2024-03-27T21:40:20.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.513 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:20.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867374+0000", 2024-03-27T21:40:20.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0547000000000003e-05 2024-03-27T21:40:20.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.513 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:20.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867400+0000", 2024-03-27T21:40:20.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5958000000000001e-05 2024-03-27T21:40:20.513 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.513 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:20.514 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.514 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:20.514 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:20.514 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.585651929000001, 2024-03-27T21:40:20.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064560310000000001, 2024-03-27T21:40:20.514 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:20.514 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:20.514 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:20.514 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:20.514 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:20.515 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 70 2024-03-27T21:40:20.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.515 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:20.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:20.515 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:20.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:20.515 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:20.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.516 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:20.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929402+0000", 2024-03-27T21:40:20.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:20.516 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.516 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.516 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:20.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929407+0000", 2024-03-27T21:40:20.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-06 2024-03-27T21:40:20.516 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.516 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.516 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:20.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929414+0000", 2024-03-27T21:40:20.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5689999999999999e-06 2024-03-27T21:40:20.516 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.516 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.517 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:20.517 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929419+0000", 2024-03-27T21:40:20.517 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-03-27T21:40:20.517 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.517 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:20.517 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930245+0000", 2024-03-27T21:40:20.517 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082521500000000002 2024-03-27T21:40:20.517 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.517 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:20.517 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930287+0000", 2024-03-27T21:40:20.517 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2305000000000002e-05 2024-03-27T21:40:20.517 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:20.518 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930367+0000", 2024-03-27T21:40:20.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9589e-05 2024-03-27T21:40:20.518 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:20.518 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.931214+0000", 2024-03-27T21:40:20.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000847827 2024-03-27T21:40:20.518 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:20.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935807+0000", 2024-03-27T21:40:20.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045923780000000003 2024-03-27T21:40:20.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:20.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935840+0000", 2024-03-27T21:40:20.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3501999999999999e-05 2024-03-27T21:40:20.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:20.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935859+0000", 2024-03-27T21:40:20.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8845e-05 2024-03-27T21:40:20.519 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.519 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:20.520 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.520 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:20.520 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:20.520 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.578392338, 2024-03-27T21:40:20.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071695429999999996, 2024-03-27T21:40:20.520 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:20.520 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:20.520 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:20.520 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:20.520 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:20.520 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-03-27T21:40:20.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.520 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:20.521 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.521 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:20.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:20.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.521 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.521 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.521 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:20.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:20.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.521 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.521 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.521 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:20.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936662+0000", 2024-03-27T21:40:20.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:20.522 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.522 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.522 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:20.522 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936737+0000", 2024-03-27T21:40:20.522 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5303999999999996e-05 2024-03-27T21:40:20.522 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.522 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.522 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:20.522 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936746+0000", 2024-03-27T21:40:20.522 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6340000000000007e-06 2024-03-27T21:40:20.522 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.522 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.522 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:20.523 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936753+0000", 2024-03-27T21:40:20.523 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-03-27T21:40:20.523 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.523 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.523 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:20.523 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937926+0000", 2024-03-27T21:40:20.523 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011731280000000001 2024-03-27T21:40:20.523 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.523 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.523 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:20.523 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937969+0000", 2024-03-27T21:40:20.523 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3285999999999999e-05 2024-03-27T21:40:20.523 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.523 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.523 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:20.524 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938040+0000", 2024-03-27T21:40:20.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0358000000000006e-05 2024-03-27T21:40:20.524 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.524 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:20.524 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939324+0000", 2024-03-27T21:40:20.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012844899999999999 2024-03-27T21:40:20.524 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.524 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:20.524 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943780+0000", 2024-03-27T21:40:20.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044555630000000001 2024-03-27T21:40:20.524 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.525 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:20.525 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943811+0000", 2024-03-27T21:40:20.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1415999999999999e-05 2024-03-27T21:40:20.525 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.525 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.525 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:20.525 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943832+0000", 2024-03-27T21:40:20.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0981000000000001e-05 2024-03-27T21:40:20.525 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.525 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:20.525 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.525 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.525 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.525 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:20.526 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:20.526 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.578294321999998, 2024-03-27T21:40:20.526 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071139740000000003, 2024-03-27T21:40:20.526 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:20.526 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:20.526 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:20.526 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:20.526 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:20.526 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-03-27T21:40:20.526 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.526 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:20.526 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.527 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:20.527 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:20.527 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.527 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.527 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.527 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:20.527 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:20.527 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.527 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.527 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.527 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:20.527 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936760+0000", 2024-03-27T21:40:20.527 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:20.528 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.528 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.528 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:20.528 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936765+0000", 2024-03-27T21:40:20.528 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6330000000000004e-06 2024-03-27T21:40:20.528 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.528 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.528 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:20.528 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936771+0000", 2024-03-27T21:40:20.528 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0560000000000001e-06 2024-03-27T21:40:20.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.529 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:20.529 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936776+0000", 2024-03-27T21:40:20.529 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1510000000000003e-06 2024-03-27T21:40:20.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.529 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:20.529 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938605+0000", 2024-03-27T21:40:20.529 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001829589 2024-03-27T21:40:20.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.529 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:20.529 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938657+0000", 2024-03-27T21:40:20.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1860000000000002e-05 2024-03-27T21:40:20.530 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.530 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.530 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:20.530 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938734+0000", 2024-03-27T21:40:20.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6639999999999998e-05 2024-03-27T21:40:20.530 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.530 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.530 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:20.530 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939562+0000", 2024-03-27T21:40:20.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082803100000000001 2024-03-27T21:40:20.530 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.530 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.530 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:20.531 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943838+0000", 2024-03-27T21:40:20.531 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042757050000000003 2024-03-27T21:40:20.531 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.531 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.531 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:20.531 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943859+0000", 2024-03-27T21:40:20.531 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1175e-05 2024-03-27T21:40:20.531 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.531 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.531 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:20.531 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943875+0000", 2024-03-27T21:40:20.531 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5744e-05 2024-03-27T21:40:20.532 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.532 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:20.532 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.532 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.532 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.532 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:20.532 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:20.532 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.434780271000001, 2024-03-27T21:40:20.532 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059712230000000003, 2024-03-27T21:40:20.532 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:20.532 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:20.532 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:20.532 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:20.533 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:20.533 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 112 2024-03-27T21:40:20.533 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.533 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:20.533 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.533 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:20.533 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:20.533 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.533 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.533 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.533 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:20.533 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:20.533 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.533 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.533 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.534 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:20.534 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080274+0000", 2024-03-27T21:40:20.534 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:20.534 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.534 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.534 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:20.534 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080318+0000", 2024-03-27T21:40:20.534 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3634999999999998e-05 2024-03-27T21:40:20.534 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.534 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.534 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:20.534 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080329+0000", 2024-03-27T21:40:20.534 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1232e-05 2024-03-27T21:40:20.535 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.535 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.535 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:20.535 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080333+0000", 2024-03-27T21:40:20.535 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6009999999999999e-06 2024-03-27T21:40:20.535 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.535 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.535 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:20.535 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083786+0000", 2024-03-27T21:40:20.535 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034528940000000002 2024-03-27T21:40:20.535 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.535 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.535 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:20.535 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083861+0000", 2024-03-27T21:40:20.536 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4893000000000003e-05 2024-03-27T21:40:20.536 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.536 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.536 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:20.536 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083981+0000", 2024-03-27T21:40:20.536 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012058299999999999 2024-03-27T21:40:20.536 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.536 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.536 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:20.536 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086148+0000", 2024-03-27T21:40:20.536 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021671469999999999 2024-03-27T21:40:20.536 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.536 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.536 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:20.537 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086195+0000", 2024-03-27T21:40:20.537 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6768999999999999e-05 2024-03-27T21:40:20.537 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.537 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.537 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:20.537 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086224+0000", 2024-03-27T21:40:20.537 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8758000000000001e-05 2024-03-27T21:40:20.537 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.537 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.537 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:20.537 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086246+0000", 2024-03-27T21:40:20.538 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2064000000000001e-05 2024-03-27T21:40:20.538 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.538 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:20.538 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.538 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.538 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.538 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:20.538 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:20.538 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.434717323999998, 2024-03-27T21:40:20.538 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066196320000000003, 2024-03-27T21:40:20.539 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:20.539 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:20.539 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:20.539 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:20.539 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:20.539 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113 2024-03-27T21:40:20.539 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.539 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:20.539 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.540 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:20.540 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:20.540 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.540 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.540 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.540 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:20.540 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:20.540 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.540 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.540 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.540 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:20.540 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080337+0000", 2024-03-27T21:40:20.540 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:20.540 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.540 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.541 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:20.541 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080340+0000", 2024-03-27T21:40:20.541 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6690000000000002e-06 2024-03-27T21:40:20.541 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.541 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.541 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:20.541 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080344+0000", 2024-03-27T21:40:20.541 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6550000000000002e-06 2024-03-27T21:40:20.541 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.541 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.541 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:20.541 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080348+0000", 2024-03-27T21:40:20.541 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9169999999999999e-06 2024-03-27T21:40:20.541 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.541 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.542 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:20.542 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084595+0000", 2024-03-27T21:40:20.542 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042471719999999996 2024-03-27T21:40:20.542 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.542 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.542 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:20.542 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084676+0000", 2024-03-27T21:40:20.542 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1147000000000001e-05 2024-03-27T21:40:20.542 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.542 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.542 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:20.542 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084796+0000", 2024-03-27T21:40:20.542 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012032900000000001 2024-03-27T21:40:20.543 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.543 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.543 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:20.543 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086190+0000", 2024-03-27T21:40:20.543 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013934399999999999 2024-03-27T21:40:20.543 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.543 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.543 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:20.543 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086905+0000", 2024-03-27T21:40:20.543 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071536199999999999 2024-03-27T21:40:20.543 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.543 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.543 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:20.543 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086937+0000", 2024-03-27T21:40:20.543 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2125000000000002e-05 2024-03-27T21:40:20.544 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.544 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.544 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:20.544 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086957+0000", 2024-03-27T21:40:20.544 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0157999999999999e-05 2024-03-27T21:40:20.544 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.544 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:20.544 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.544 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.544 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.544 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:20.544 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:20.544 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.434704607999997, 2024-03-27T21:40:20.544 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007405661, 2024-03-27T21:40:20.544 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:20.545 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:20.545 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:20.545 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:20.545 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:20.545 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 114 2024-03-27T21:40:20.545 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.545 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:20.545 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.545 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:20.545 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:20.545 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.545 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.545 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.545 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:20.545 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:20.546 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.546 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.546 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.546 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:20.546 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:20.546 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-03-27T21:40:20.546 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.546 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.546 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:20.546 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080353+0000", 2024-03-27T21:40:20.546 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.441e-06 2024-03-27T21:40:20.546 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.546 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.547 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:20.547 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080356+0000", 2024-03-27T21:40:20.547 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0769999999999999e-06 2024-03-27T21:40:20.547 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.547 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.547 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:20.547 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080358+0000", 2024-03-27T21:40:20.547 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3369999999999998e-06 2024-03-27T21:40:20.547 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.547 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.547 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:20.547 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085118+0000", 2024-03-27T21:40:20.548 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047603949999999997 2024-03-27T21:40:20.548 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.548 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.548 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:20.548 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085210+0000", 2024-03-27T21:40:20.548 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1983000000000003e-05 2024-03-27T21:40:20.548 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.548 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.548 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:20.548 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085332+0000", 2024-03-27T21:40:20.548 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121803 2024-03-27T21:40:20.548 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.548 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.548 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:20.549 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086292+0000", 2024-03-27T21:40:20.549 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095961400000000002 2024-03-27T21:40:20.549 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.549 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.549 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:20.549 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087710+0000", 2024-03-27T21:40:20.549 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014181459999999999 2024-03-27T21:40:20.549 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.549 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.549 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:20.549 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087736+0000", 2024-03-27T21:40:20.549 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6475e-05 2024-03-27T21:40:20.549 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.550 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.550 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:20.550 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087756+0000", 2024-03-27T21:40:20.550 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9626000000000001e-05 2024-03-27T21:40:20.550 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.550 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:20.550 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.550 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.550 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.550 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:20.551 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:20.551 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.434682172999999, 2024-03-27T21:40:20.551 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075011720000000004, 2024-03-27T21:40:20.551 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:20.551 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:20.551 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:20.551 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:20.551 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:20.551 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115 2024-03-27T21:40:20.551 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.551 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:20.551 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.551 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:20.551 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:20.552 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.552 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.552 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.552 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:20.552 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:20.552 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.552 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.552 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.552 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:20.552 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:20.552 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:20.552 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.552 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.552 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:20.552 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080458+0000", 2024-03-27T21:40:20.553 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5228000000000005e-05 2024-03-27T21:40:20.553 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.553 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.553 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:20.553 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080465+0000", 2024-03-27T21:40:20.553 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.238e-06 2024-03-27T21:40:20.553 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.553 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.553 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:20.553 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080468+0000", 2024-03-27T21:40:20.553 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5109999999999999e-06 2024-03-27T21:40:20.553 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.553 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.553 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:20.553 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086326+0000", 2024-03-27T21:40:20.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058579169999999998 2024-03-27T21:40:20.554 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.554 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.554 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:20.554 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086382+0000", 2024-03-27T21:40:20.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5309999999999997e-05 2024-03-27T21:40:20.554 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.554 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.554 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:20.554 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086457+0000", 2024-03-27T21:40:20.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5413000000000005e-05 2024-03-27T21:40:20.554 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.554 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.554 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:20.555 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087762+0000", 2024-03-27T21:40:20.555 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013052300000000001 2024-03-27T21:40:20.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.555 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:20.555 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087798+0000", 2024-03-27T21:40:20.555 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5614999999999997e-05 2024-03-27T21:40:20.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.555 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:20.555 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087840+0000", 2024-03-27T21:40:20.555 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2196000000000001e-05 2024-03-27T21:40:20.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.556 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:20.556 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087874+0000", 2024-03-27T21:40:20.556 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3785999999999998e-05 2024-03-27T21:40:20.556 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.556 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:20.556 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.556 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.556 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.556 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:20.556 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:20.556 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.434582089999999, 2024-03-27T21:40:20.556 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075050680000000002, 2024-03-27T21:40:20.556 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:20.556 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:20.556 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:20.556 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:20.557 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:20.557 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 116 2024-03-27T21:40:20.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.557 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:20.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.557 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:20.557 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:20.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.557 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:20.557 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:20.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:20.558 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080472+0000", 2024-03-27T21:40:20.558 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:20.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:20.558 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080475+0000", 2024-03-27T21:40:20.558 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.712e-06 2024-03-27T21:40:20.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:20.558 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080478+0000", 2024-03-27T21:40:20.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1070000000000001e-06 2024-03-27T21:40:20.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.559 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:20.559 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080481+0000", 2024-03-27T21:40:20.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4269999999999998e-06 2024-03-27T21:40:20.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.559 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:20.559 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086992+0000", 2024-03-27T21:40:20.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065114099999999996 2024-03-27T21:40:20.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:20.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087064+0000", 2024-03-27T21:40:20.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1927000000000004e-05 2024-03-27T21:40:20.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:20.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087188+0000", 2024-03-27T21:40:20.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123585 2024-03-27T21:40:20.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:20.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087766+0000", 2024-03-27T21:40:20.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057814300000000004 2024-03-27T21:40:20.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.561 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.561 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:20.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087934+0000", 2024-03-27T21:40:20.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000167894 2024-03-27T21:40:20.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.561 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.561 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:20.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087958+0000", 2024-03-27T21:40:20.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4663000000000001e-05 2024-03-27T21:40:20.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.561 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.561 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:20.562 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087978+0000", 2024-03-27T21:40:20.562 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9636e-05 2024-03-27T21:40:20.562 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.562 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:20.562 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.562 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.562 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.562 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:20.562 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:20.562 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.434549967999999, 2024-03-27T21:40:20.562 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076850679999999998, 2024-03-27T21:40:20.562 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:20.562 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:20.563 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:20.563 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:20.563 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:20.563 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 117 2024-03-27T21:40:20.563 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.563 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:20.563 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.563 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:20.563 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:20.563 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.563 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.563 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.564 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:20.564 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:20.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.564 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.564 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:20.564 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:20.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:20.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.564 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.564 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:20.564 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080508+0000", 2024-03-27T21:40:20.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.298e-06 2024-03-27T21:40:20.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:20.565 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080515+0000", 2024-03-27T21:40:20.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0790000000000003e-06 2024-03-27T21:40:20.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:20.565 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080518+0000", 2024-03-27T21:40:20.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8509999999999999e-06 2024-03-27T21:40:20.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:20.566 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087444+0000", 2024-03-27T21:40:20.566 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0069259120000000002 2024-03-27T21:40:20.566 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.566 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.566 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:20.566 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087491+0000", 2024-03-27T21:40:20.566 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7043999999999999e-05 2024-03-27T21:40:20.566 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.566 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.566 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:20.566 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087564+0000", 2024-03-27T21:40:20.566 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3603999999999995e-05 2024-03-27T21:40:20.566 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.567 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.567 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:20.567 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087902+0000", 2024-03-27T21:40:20.567 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033734100000000001 2024-03-27T21:40:20.567 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.567 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.567 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:20.567 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088117+0000", 2024-03-27T21:40:20.567 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000215164 2024-03-27T21:40:20.567 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.567 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.567 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:20.568 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088160+0000", 2024-03-27T21:40:20.568 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2933999999999999e-05 2024-03-27T21:40:20.568 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.568 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.568 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:20.568 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088190+0000", 2024-03-27T21:40:20.568 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0139999999999999e-05 2024-03-27T21:40:20.568 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.568 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:20.568 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.568 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.568 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.568 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:20.568 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:20.569 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.018269578999998, 2024-03-27T21:40:20.569 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052465439999999997, 2024-03-27T21:40:20.569 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:20.569 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:20.569 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:20.569 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:20.569 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:20.569 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 133 2024-03-27T21:40:20.569 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.569 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:20.569 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.569 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:20.569 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:20.569 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.569 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.570 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:20.570 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:20.570 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.570 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.570 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:20.570 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:20.570 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:20.570 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.570 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:20.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496788+0000", 2024-03-27T21:40:20.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-03-27T21:40:20.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.571 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:20.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496793+0000", 2024-03-27T21:40:20.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4790000000000004e-06 2024-03-27T21:40:20.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.571 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:20.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496796+0000", 2024-03-27T21:40:20.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1e-06 2024-03-27T21:40:20.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.572 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.572 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:20.572 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500447+0000", 2024-03-27T21:40:20.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036504459999999999 2024-03-27T21:40:20.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.572 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.572 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:20.572 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500492+0000", 2024-03-27T21:40:20.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5765e-05 2024-03-27T21:40:20.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.573 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.573 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:20.573 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500562+0000", 2024-03-27T21:40:20.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9206000000000001e-05 2024-03-27T21:40:20.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.573 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.573 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:20.573 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501959+0000", 2024-03-27T21:40:20.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001397187 2024-03-27T21:40:20.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.573 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.573 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:20.573 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501984+0000", 2024-03-27T21:40:20.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.552e-05 2024-03-27T21:40:20.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:20.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502010+0000", 2024-03-27T21:40:20.574 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5232000000000002e-05 2024-03-27T21:40:20.574 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:20.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502032+0000", 2024-03-27T21:40:20.574 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2337999999999999e-05 2024-03-27T21:40:20.574 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.574 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:20.574 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.575 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:20.575 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:20.575 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.018210920999998, 2024-03-27T21:40:20.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052892119999999997, 2024-03-27T21:40:20.575 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:20.575 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:20.575 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:20.575 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:20.575 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:20.575 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 134 2024-03-27T21:40:20.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.575 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:20.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.576 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:20.576 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:20.576 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.576 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.576 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.576 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:20.576 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:20.576 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.576 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.576 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.576 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:20.576 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:20.576 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:20.576 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.576 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.577 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:20.577 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496929+0000", 2024-03-27T21:40:20.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5098000000000004e-05 2024-03-27T21:40:20.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.577 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:20.577 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496937+0000", 2024-03-27T21:40:20.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9559999999999997e-06 2024-03-27T21:40:20.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.577 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:20.577 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496942+0000", 2024-03-27T21:40:20.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6350000000000003e-06 2024-03-27T21:40:20.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:20.578 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500749+0000", 2024-03-27T21:40:20.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038068529999999998 2024-03-27T21:40:20.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:20.578 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500792+0000", 2024-03-27T21:40:20.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2823e-05 2024-03-27T21:40:20.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:20.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500864+0000", 2024-03-27T21:40:20.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1715e-05 2024-03-27T21:40:20.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:20.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502068+0000", 2024-03-27T21:40:20.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012044860000000001 2024-03-27T21:40:20.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:20.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502073+0000", 2024-03-27T21:40:20.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5700000000000003e-06 2024-03-27T21:40:20.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.580 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:20.580 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502098+0000", 2024-03-27T21:40:20.580 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5120999999999999e-05 2024-03-27T21:40:20.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.580 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:20.580 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502133+0000", 2024-03-27T21:40:20.580 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5306999999999999e-05 2024-03-27T21:40:20.580 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.580 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:20.580 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.581 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:20.581 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:20.581 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.018105614, 2024-03-27T21:40:20.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062529040000000001, 2024-03-27T21:40:20.581 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:20.581 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:20.581 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:20.581 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:20.581 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:20.581 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135 2024-03-27T21:40:20.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.581 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:20.581 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.581 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:20.582 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:20.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.582 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.582 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:20.582 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:20.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.582 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.582 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:20.582 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:20.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:20.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.583 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:20.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496953+0000", 2024-03-27T21:40:20.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0709999999999996e-06 2024-03-27T21:40:20.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.583 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:20.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496961+0000", 2024-03-27T21:40:20.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1300000000000001e-06 2024-03-27T21:40:20.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.583 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:20.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496968+0000", 2024-03-27T21:40:20.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0999999999999998e-06 2024-03-27T21:40:20.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:20.584 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501244+0000", 2024-03-27T21:40:20.584 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042756310000000002 2024-03-27T21:40:20.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:20.584 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501377+0000", 2024-03-27T21:40:20.584 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013348799999999999 2024-03-27T21:40:20.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:20.584 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501455+0000", 2024-03-27T21:40:20.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7410000000000006e-05 2024-03-27T21:40:20.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.585 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:20.585 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502138+0000", 2024-03-27T21:40:20.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068386599999999999 2024-03-27T21:40:20.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.585 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:20.585 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503152+0000", 2024-03-27T21:40:20.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010139680000000001 2024-03-27T21:40:20.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.585 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:20.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503180+0000", 2024-03-27T21:40:20.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7426000000000001e-05 2024-03-27T21:40:20.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.586 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.586 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:20.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503202+0000", 2024-03-27T21:40:20.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2463000000000002e-05 2024-03-27T21:40:20.586 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.587 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:20.587 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.587 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:20.587 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:20.587 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.018081475999999, 2024-03-27T21:40:20.588 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066253680000000004, 2024-03-27T21:40:20.588 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:20.588 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:20.588 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:20.588 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:20.588 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:20.588 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 136 2024-03-27T21:40:20.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.588 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:20.588 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.588 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:20.588 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:20.588 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.589 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.589 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:20.589 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:20.589 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.589 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.589 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:20.589 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:20.589 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:20.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.589 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.589 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:20.589 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496977+0000", 2024-03-27T21:40:20.590 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5049999999999999e-06 2024-03-27T21:40:20.590 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.590 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.590 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:20.590 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496984+0000", 2024-03-27T21:40:20.590 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1520000000000003e-06 2024-03-27T21:40:20.590 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.590 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.590 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:20.591 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496990+0000", 2024-03-27T21:40:20.591 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-03-27T21:40:20.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.591 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:20.591 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501669+0000", 2024-03-27T21:40:20.591 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046793859999999998 2024-03-27T21:40:20.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.591 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:20.591 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501714+0000", 2024-03-27T21:40:20.591 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5154999999999997e-05 2024-03-27T21:40:20.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.591 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:20.592 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501783+0000", 2024-03-27T21:40:20.592 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8454999999999994e-05 2024-03-27T21:40:20.592 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.592 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.592 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:20.592 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503226+0000", 2024-03-27T21:40:20.592 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001443441 2024-03-27T21:40:20.592 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.592 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.592 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:20.592 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503540+0000", 2024-03-27T21:40:20.592 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031347100000000001 2024-03-27T21:40:20.592 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.593 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.593 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:20.593 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503563+0000", 2024-03-27T21:40:20.593 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3223000000000001e-05 2024-03-27T21:40:20.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.593 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.593 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:20.593 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503599+0000", 2024-03-27T21:40:20.593 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6105000000000001e-05 2024-03-27T21:40:20.594 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.594 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:20.594 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.594 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.594 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.594 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:20.594 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:20.594 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.018058666000002, 2024-03-27T21:40:20.594 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074261320000000002, 2024-03-27T21:40:20.594 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:20.594 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:20.594 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:20.594 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:20.594 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:20.595 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 137 2024-03-27T21:40:20.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.595 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:20.595 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.595 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:20.595 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:20.595 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.595 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.595 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:20.595 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:20.595 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.595 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.595 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:20.596 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:20.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:20.596 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.596 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.596 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:20.596 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497077+0000", 2024-03-27T21:40:20.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1599000000000003e-05 2024-03-27T21:40:20.596 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.596 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.596 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:20.596 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497085+0000", 2024-03-27T21:40:20.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6109999999999996e-06 2024-03-27T21:40:20.596 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.596 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.597 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:20.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497092+0000", 2024-03-27T21:40:20.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.742e-06 2024-03-27T21:40:20.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.597 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.597 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:20.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502159+0000", 2024-03-27T21:40:20.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050666870000000003 2024-03-27T21:40:20.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.597 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.597 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:20.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502200+0000", 2024-03-27T21:40:20.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1919999999999998e-05 2024-03-27T21:40:20.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:20.598 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502280+0000", 2024-03-27T21:40:20.598 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9417000000000002e-05 2024-03-27T21:40:20.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:20.598 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503605+0000", 2024-03-27T21:40:20.598 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001325641 2024-03-27T21:40:20.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:20.598 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504361+0000", 2024-03-27T21:40:20.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075555399999999997 2024-03-27T21:40:20.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.599 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.599 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:20.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504386+0000", 2024-03-27T21:40:20.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5369999999999999e-05 2024-03-27T21:40:20.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.599 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.599 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:20.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504422+0000", 2024-03-27T21:40:20.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6019000000000002e-05 2024-03-27T21:40:20.599 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.599 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:20.599 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.600 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.600 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:20.600 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:20.600 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.017954869, 2024-03-27T21:40:20.600 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074059750000000004, 2024-03-27T21:40:20.600 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:20.600 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:20.600 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:20.600 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:20.600 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:20.600 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-03-27T21:40:20.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.600 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:20.600 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:20.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:20.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:20.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:20.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:20.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:20.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:20.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:20.602 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497103+0000", 2024-03-27T21:40:20.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.856e-06 2024-03-27T21:40:20.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:20.602 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497121+0000", 2024-03-27T21:40:20.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7190999999999999e-05 2024-03-27T21:40:20.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:20.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497126+0000", 2024-03-27T21:40:20.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3770000000000004e-06 2024-03-27T21:40:20.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:20.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502833+0000", 2024-03-27T21:40:20.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057073510000000003 2024-03-27T21:40:20.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:20.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502881+0000", 2024-03-27T21:40:20.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7961999999999999e-05 2024-03-27T21:40:20.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:20.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502949+0000", 2024-03-27T21:40:20.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7957e-05 2024-03-27T21:40:20.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:20.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504430+0000", 2024-03-27T21:40:20.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001480763 2024-03-27T21:40:20.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:20.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504459+0000", 2024-03-27T21:40:20.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8983999999999999e-05 2024-03-27T21:40:20.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:20.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504486+0000", 2024-03-27T21:40:20.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6883999999999999e-05 2024-03-27T21:40:20.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:20.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504506+0000", 2024-03-27T21:40:20.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0214e-05 2024-03-27T21:40:20.605 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.605 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:20.605 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.605 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:20.605 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:20.606 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.017923035000003, 2024-03-27T21:40:20.606 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074754859999999999, 2024-03-27T21:40:20.606 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:20.606 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:20.606 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:20.606 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:20.606 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:20.606 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-03-27T21:40:20.606 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.606 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:20.606 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.606 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:20.606 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:20.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.607 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.607 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:20.607 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:20.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.607 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.607 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:20.607 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:20.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:20.607 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.607 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:20.608 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497137+0000", 2024-03-27T21:40:20.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0730000000000004e-06 2024-03-27T21:40:20.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:20.608 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497143+0000", 2024-03-27T21:40:20.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3759999999999996e-06 2024-03-27T21:40:20.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:20.608 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497150+0000", 2024-03-27T21:40:20.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.002e-06 2024-03-27T21:40:20.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:20.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503249+0000", 2024-03-27T21:40:20.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060989750000000004 2024-03-27T21:40:20.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:20.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503291+0000", 2024-03-27T21:40:20.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1563999999999997e-05 2024-03-27T21:40:20.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:20.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503376+0000", 2024-03-27T21:40:20.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5752000000000002e-05 2024-03-27T21:40:20.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:20.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504545+0000", 2024-03-27T21:40:20.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169172 2024-03-27T21:40:20.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.610 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:20.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504550+0000", 2024-03-27T21:40:20.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7419999999999997e-06 2024-03-27T21:40:20.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.610 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:20.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504574+0000", 2024-03-27T21:40:20.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4124000000000001e-05 2024-03-27T21:40:20.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.611 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:20.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504607+0000", 2024-03-27T21:40:20.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3118999999999999e-05 2024-03-27T21:40:20.611 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.611 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:20.611 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.611 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:20.611 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:20.611 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.017899368999998, 2024-03-27T21:40:20.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078150589999999992, 2024-03-27T21:40:20.612 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:20.612 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:20.612 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:20.612 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:20.612 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:20.612 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-03-27T21:40:20.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.612 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:20.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:20.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:20.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:20.613 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:20.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:20.613 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497155+0000", 2024-03-27T21:40:20.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:20.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:20.613 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497230+0000", 2024-03-27T21:40:20.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4789000000000005e-05 2024-03-27T21:40:20.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.614 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.614 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:20.614 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497241+0000", 2024-03-27T21:40:20.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1442e-05 2024-03-27T21:40:20.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.614 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.614 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:20.614 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497255+0000", 2024-03-27T21:40:20.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3087e-05 2024-03-27T21:40:20.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.615 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:20.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503630+0000", 2024-03-27T21:40:20.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063755269999999998 2024-03-27T21:40:20.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.615 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:20.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503673+0000", 2024-03-27T21:40:20.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2616e-05 2024-03-27T21:40:20.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.615 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:20.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503738+0000", 2024-03-27T21:40:20.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5114e-05 2024-03-27T21:40:20.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:20.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504907+0000", 2024-03-27T21:40:20.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169432 2024-03-27T21:40:20.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:20.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504926+0000", 2024-03-27T21:40:20.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9086999999999999e-05 2024-03-27T21:40:20.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:20.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504950+0000", 2024-03-27T21:40:20.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3598999999999999e-05 2024-03-27T21:40:20.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:20.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504971+0000", 2024-03-27T21:40:20.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0772999999999999e-05 2024-03-27T21:40:20.617 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.617 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:20.617 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.617 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:20.618 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:20.618 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 40.017792329000002, 2024-03-27T21:40:20.618 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082962590000000003, 2024-03-27T21:40:20.618 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:20.618 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:20.618 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:20.618 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:20.618 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:20.618 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-03-27T21:40:20.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.618 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:20.618 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.618 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:20.618 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:20.618 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.619 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.619 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:20.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:20.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.619 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.619 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:20.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497262+0000", 2024-03-27T21:40:20.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:20.619 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.619 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:20.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497267+0000", 2024-03-27T21:40:20.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.955e-06 2024-03-27T21:40:20.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.620 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.620 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:20.620 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497275+0000", 2024-03-27T21:40:20.620 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.824e-06 2024-03-27T21:40:20.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.620 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.620 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:20.620 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497280+0000", 2024-03-27T21:40:20.620 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8230000000000002e-06 2024-03-27T21:40:20.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.620 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.620 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:20.621 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504628+0000", 2024-03-27T21:40:20.621 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007348561 2024-03-27T21:40:20.621 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.621 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:20.621 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504673+0000", 2024-03-27T21:40:20.621 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4313000000000001e-05 2024-03-27T21:40:20.621 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.621 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:20.621 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504745+0000", 2024-03-27T21:40:20.621 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2303000000000002e-05 2024-03-27T21:40:20.621 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:20.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505358+0000", 2024-03-27T21:40:20.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061281300000000003 2024-03-27T21:40:20.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:20.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505503+0000", 2024-03-27T21:40:20.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000145134 2024-03-27T21:40:20.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:20.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505526+0000", 2024-03-27T21:40:20.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3312e-05 2024-03-27T21:40:20.623 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:20.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505559+0000", 2024-03-27T21:40:20.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2821999999999998e-05 2024-03-27T21:40:20.623 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.623 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:20.623 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.623 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.623 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:20.623 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:20.623 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 39.408570089999998, 2024-03-27T21:40:20.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053753339999999998, 2024-03-27T21:40:20.624 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:20.624 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:20.624 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:20.624 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:20.624 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:20.624 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159 2024-03-27T21:40:20.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.624 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:20.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.625 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:20.625 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:20.625 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.625 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.625 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.625 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:20.625 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:20.625 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:20.626 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.626 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.626 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:20.626 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:20.626 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:20.626 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.626 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:20.627 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106488+0000", 2024-03-27T21:40:20.627 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9830000000000001e-06 2024-03-27T21:40:20.627 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:20.628 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106491+0000", 2024-03-27T21:40:20.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3459999999999998e-06 2024-03-27T21:40:20.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.628 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:20.628 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106493+0000", 2024-03-27T21:40:20.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.509e-06 2024-03-27T21:40:20.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.629 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:20.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110636+0000", 2024-03-27T21:40:20.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041427260000000002 2024-03-27T21:40:20.630 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.630 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:20.630 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110683+0000", 2024-03-27T21:40:20.630 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6557000000000002e-05 2024-03-27T21:40:20.630 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.630 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:20.631 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110755+0000", 2024-03-27T21:40:20.631 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2119000000000005e-05 2024-03-27T21:40:20.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.631 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.631 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:20.631 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111415+0000", 2024-03-27T21:40:20.631 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066065199999999996 2024-03-27T21:40:20.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.632 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.632 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:20.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111793+0000", 2024-03-27T21:40:20.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037732100000000001 2024-03-27T21:40:20.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:20.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111829+0000", 2024-03-27T21:40:20.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418999999999998e-05 2024-03-27T21:40:20.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:20.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:20.634 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:20.634 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111860+0000", 2024-03-27T21:40:20.634 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1003000000000001e-05 2024-03-27T21:40:20.634 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.634 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:20.634 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.634 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:20.635 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:20.635 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:20.636 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:40:20.637 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:20.650+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:20.737 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:20.751+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:20.810 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:20.810 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:20.810 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:20.810 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:20.823 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:40:20.863 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:20.877+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:20.939 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:20.953+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:20.955 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-27T21:40:20.976 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:20.977 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:20.977 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:20.977 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:20.977 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:20.988 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:40:21.040 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:21.054+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:21.103 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:21.103 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:21.103 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:21.103 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:21.141 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:21.155+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:21.206 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:21.207 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:21.207 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:21.207 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-03-27T21:40:21.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.207 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:20 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2024-03-27T21:40:21.207 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:21.207 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 41.330086193, 2024-03-27T21:40:21.207 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015345630000000001, 2024-03-27T21:40:21.207 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:21.207 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:21.207 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:21.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.207 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:21.207 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:21.207 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.208 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.208 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.208 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:21.208 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:21.208 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.208 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.208 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.208 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:21.208 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888562+0000", 2024-03-27T21:40:21.208 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:21.208 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.208 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.208 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:21.208 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888735+0000", 2024-03-27T21:40:21.208 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017254799999999999 2024-03-27T21:40:21.209 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.209 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.209 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:21.209 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888747+0000", 2024-03-27T21:40:21.209 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2469999999999999e-05 2024-03-27T21:40:21.209 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.209 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.209 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:21.209 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888761+0000", 2024-03-27T21:40:21.209 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3254e-05 2024-03-27T21:40:21.209 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.209 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.209 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:21.209 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888804+0000", 2024-03-27T21:40:21.209 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3129e-05 2024-03-27T21:40:21.210 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.210 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.210 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:21.210 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888820+0000", 2024-03-27T21:40:21.210 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6426000000000002e-05 2024-03-27T21:40:21.210 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.210 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.210 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:21.210 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890060+0000", 2024-03-27T21:40:21.210 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012398050000000001 2024-03-27T21:40:21.210 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.210 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.210 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:21.211 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890098+0000", 2024-03-27T21:40:21.211 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7795e-05 2024-03-27T21:40:21.211 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.211 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:21.211 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.211 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.211 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2024-03-27T21:40:21.211 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:21.211 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 41.329261793000001, 2024-03-27T21:40:21.211 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001069555, 2024-03-27T21:40:21.211 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:21.211 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:21.211 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889387+0000", 2024-03-27T21:40:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:21.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.213 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.213 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:21.213 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889455+0000", 2024-03-27T21:40:21.213 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7930999999999997e-05 2024-03-27T21:40:21.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.213 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.213 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:21.213 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889470+0000", 2024-03-27T21:40:21.213 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5554e-05 2024-03-27T21:40:21.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.213 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.213 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:21.213 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889476+0000", 2024-03-27T21:40:21.213 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0619999999999997e-06 2024-03-27T21:40:21.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.214 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.214 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:21.214 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889513+0000", 2024-03-27T21:40:21.214 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6606999999999997e-05 2024-03-27T21:40:21.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.214 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.214 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:21.214 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889528+0000", 2024-03-27T21:40:21.214 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4982e-05 2024-03-27T21:40:21.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.214 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.214 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:21.214 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890421+0000", 2024-03-27T21:40:21.214 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00089293700000000001 2024-03-27T21:40:21.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.215 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.215 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:21.215 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890457+0000", 2024-03-27T21:40:21.215 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6260000000000002e-05 2024-03-27T21:40:21.215 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.215 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:21.215 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.215 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.215 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.215 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:38 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'24, mlcod=17'22)", 2024-03-27T21:40:21.215 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:21.215 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 41.312025826999999, 2024-03-27T21:40:21.215 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016182080000000001, 2024-03-27T21:40:21.215 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:21.215 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:21.215 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:21.215 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.216 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:21.216 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:21.216 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.216 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.216 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:21.216 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:21.216 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.216 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.216 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:21.216 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906623+0000", 2024-03-27T21:40:21.216 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:21.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.216 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.217 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:21.217 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906687+0000", 2024-03-27T21:40:21.217 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3706000000000003e-05 2024-03-27T21:40:21.217 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.217 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.217 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:21.217 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906698+0000", 2024-03-27T21:40:21.217 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1752e-05 2024-03-27T21:40:21.217 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.217 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.217 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:21.217 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906704+0000", 2024-03-27T21:40:21.217 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3940000000000004e-06 2024-03-27T21:40:21.217 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.218 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:21.218 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906741+0000", 2024-03-27T21:40:21.218 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6944999999999999e-05 2024-03-27T21:40:21.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.218 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:21.218 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906760+0000", 2024-03-27T21:40:21.218 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9293e-05 2024-03-27T21:40:21.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.218 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:21.218 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908211+0000", 2024-03-27T21:40:21.218 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014511350000000001 2024-03-27T21:40:21.219 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.219 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.219 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:21.219 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908242+0000", 2024-03-27T21:40:21.219 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0712e-05 2024-03-27T21:40:21.219 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.219 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:21.219 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.219 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.219 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.219 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:39 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'25, mlcod=17'22)", 2024-03-27T21:40:21.219 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:21.219 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 41.311180911999998, 2024-03-27T21:40:21.219 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012060199999999999, 2024-03-27T21:40:21.220 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:21.220 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:21.220 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:21.220 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.220 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:21.220 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:21.220 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.220 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.220 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.220 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:21.221 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:21.221 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.221 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.221 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.221 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:21.221 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:21.221 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:21.221 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.221 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.221 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:21.221 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907533+0000", 2024-03-27T21:40:21.221 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5675999999999999e-05 2024-03-27T21:40:21.221 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.221 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.221 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:21.221 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907545+0000", 2024-03-27T21:40:21.222 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.115e-05 2024-03-27T21:40:21.222 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.222 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.222 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:21.222 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907550+0000", 2024-03-27T21:40:21.222 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8320000000000002e-06 2024-03-27T21:40:21.222 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.222 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.222 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:21.222 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907592+0000", 2024-03-27T21:40:21.222 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1309e-05 2024-03-27T21:40:21.223 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.223 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.223 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:21.223 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907611+0000", 2024-03-27T21:40:21.223 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9009e-05 2024-03-27T21:40:21.223 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.223 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.223 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:21.223 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908638+0000", 2024-03-27T21:40:21.223 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001027533 2024-03-27T21:40:21.223 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.223 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.223 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:21.223 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908675+0000", 2024-03-27T21:40:21.223 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6329999999999999e-05 2024-03-27T21:40:21.224 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.224 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:21.224 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.224 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.224 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.224 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:57 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'35, mlcod=17'33)", 2024-03-27T21:40:21.224 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:21.224 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 41.29709648, 2024-03-27T21:40:21.224 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001346578, 2024-03-27T21:40:21.224 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:21.224 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:21.224 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:21.225 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.225 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:21.225 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:21.225 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.225 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.225 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.225 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:21.225 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:21.225 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.225 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.225 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.225 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:21.225 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921552+0000", 2024-03-27T21:40:21.225 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:21.225 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.226 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.226 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:21.226 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921615+0000", 2024-03-27T21:40:21.226 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2787999999999996e-05 2024-03-27T21:40:21.226 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.226 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.226 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:21.226 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921626+0000", 2024-03-27T21:40:21.226 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0888e-05 2024-03-27T21:40:21.226 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.226 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.226 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:21.226 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921635+0000", 2024-03-27T21:40:21.227 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4639999999999995e-06 2024-03-27T21:40:21.227 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.227 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.227 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:21.227 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921673+0000", 2024-03-27T21:40:21.227 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7891e-05 2024-03-27T21:40:21.227 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.227 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.227 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:21.227 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921694+0000", 2024-03-27T21:40:21.227 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0426000000000001e-05 2024-03-27T21:40:21.227 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.228 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.228 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:21.228 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922865+0000", 2024-03-27T21:40:21.228 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011709509999999999 2024-03-27T21:40:21.228 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.228 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.228 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:21.228 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922900+0000", 2024-03-27T21:40:21.228 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4819e-05 2024-03-27T21:40:21.228 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.228 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:21.228 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.228 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.228 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.228 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:71 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'45, mlcod=17'43)", 2024-03-27T21:40:21.229 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:21.229 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 41.281797767999997, 2024-03-27T21:40:21.229 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001460907, 2024-03-27T21:40:21.229 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:21.229 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:21.229 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:21.229 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.229 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:21.229 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:21.229 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.229 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.229 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.229 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:21.229 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:21.229 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.230 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.230 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.230 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:21.230 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936851+0000", 2024-03-27T21:40:21.230 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:21.230 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.230 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.230 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:21.230 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936903+0000", 2024-03-27T21:40:21.230 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2302999999999997e-05 2024-03-27T21:40:21.230 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.230 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.230 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:21.230 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936915+0000", 2024-03-27T21:40:21.230 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1967000000000001e-05 2024-03-27T21:40:21.231 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.231 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.231 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:21.231 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936923+0000", 2024-03-27T21:40:21.231 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9370000000000004e-06 2024-03-27T21:40:21.231 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.231 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.231 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:21.231 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936970+0000", 2024-03-27T21:40:21.231 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6987000000000003e-05 2024-03-27T21:40:21.231 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.231 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.232 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:21.232 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937002+0000", 2024-03-27T21:40:21.232 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2410999999999999e-05 2024-03-27T21:40:21.232 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.232 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.232 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:21.232 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938279+0000", 2024-03-27T21:40:21.232 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001276552 2024-03-27T21:40:21.232 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.232 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.232 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:21.232 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938313+0000", 2024-03-27T21:40:21.232 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3526999999999999e-05 2024-03-27T21:40:21.232 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.232 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:21.233 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.233 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.233 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.233 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:72 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'46, mlcod=17'43)", 2024-03-27T21:40:21.233 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:21.233 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 41.281160206000003, 2024-03-27T21:40:21.233 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014239490000000001, 2024-03-27T21:40:21.233 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:21.233 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:21.233 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:21.233 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.233 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:21.233 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:21.233 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.233 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.234 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.234 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:21.234 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:21.234 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.234 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.234 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.234 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:21.234 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937488+0000", 2024-03-27T21:40:21.234 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:21.234 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.234 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.234 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:21.234 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937547+0000", 2024-03-27T21:40:21.234 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8196999999999999e-05 2024-03-27T21:40:21.234 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.235 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.235 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:21.235 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937559+0000", 2024-03-27T21:40:21.235 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1928e-05 2024-03-27T21:40:21.235 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.235 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.235 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:21.235 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937564+0000", 2024-03-27T21:40:21.235 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4759999999999997e-06 2024-03-27T21:40:21.235 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.235 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.235 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:21.235 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937621+0000", 2024-03-27T21:40:21.236 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6712000000000002e-05 2024-03-27T21:40:21.236 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.236 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.236 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:21.236 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937640+0000", 2024-03-27T21:40:21.236 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9517e-05 2024-03-27T21:40:21.236 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.236 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.236 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:21.236 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938883+0000", 2024-03-27T21:40:21.236 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001242567 2024-03-27T21:40:21.236 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.236 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.236 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:21.236 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938913+0000", 2024-03-27T21:40:21.237 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0287e-05 2024-03-27T21:40:21.237 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.237 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:21.237 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.237 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.237 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.237 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:73 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'47, mlcod=17'43)", 2024-03-27T21:40:21.237 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:21.237 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 41.280484112000003, 2024-03-27T21:40:21.237 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010746239999999999, 2024-03-27T21:40:21.237 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:21.237 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:21.237 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:21.238 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.238 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:21.238 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:21.238 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.238 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.238 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.238 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:21.238 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:21.238 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.238 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.238 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.238 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:21.238 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:21.238 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:21.239 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.239 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.239 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:21.239 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938217+0000", 2024-03-27T21:40:21.239 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2743e-05 2024-03-27T21:40:21.239 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.239 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.239 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:21.239 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938229+0000", 2024-03-27T21:40:21.239 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1956999999999999e-05 2024-03-27T21:40:21.239 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.239 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.239 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:21.239 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938235+0000", 2024-03-27T21:40:21.239 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4589999999999997e-06 2024-03-27T21:40:21.240 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.240 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.240 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:21.240 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938341+0000", 2024-03-27T21:40:21.240 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000106054 2024-03-27T21:40:21.240 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.240 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.240 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:21.240 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938369+0000", 2024-03-27T21:40:21.240 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8291000000000001e-05 2024-03-27T21:40:21.240 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.240 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.240 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:21.240 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939209+0000", 2024-03-27T21:40:21.241 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084045400000000003 2024-03-27T21:40:21.241 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.241 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.241 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:21.241 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939240+0000", 2024-03-27T21:40:21.241 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0429999999999998e-05 2024-03-27T21:40:21.241 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.241 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:21.241 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.241 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.241 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.241 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:75 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'49, mlcod=17'47)", 2024-03-27T21:40:21.241 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:21.241 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 41.274046464999998, 2024-03-27T21:40:21.242 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001017001, 2024-03-27T21:40:21.242 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:21.242 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:21.242 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:21.242 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.242 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:21.242 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:21.242 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.242 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.242 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.242 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:21.242 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:21.242 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.242 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.243 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.243 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:21.243 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944601+0000", 2024-03-27T21:40:21.243 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999986 2024-03-27T21:40:21.243 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.243 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.243 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:21.243 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944614+0000", 2024-03-27T21:40:21.243 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2496e-05 2024-03-27T21:40:21.243 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.243 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.243 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:21.243 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944640+0000", 2024-03-27T21:40:21.243 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5865999999999999e-05 2024-03-27T21:40:21.244 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.244 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.244 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:21.244 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944651+0000", 2024-03-27T21:40:21.244 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1797000000000001e-05 2024-03-27T21:40:21.244 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.244 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.244 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:21.244 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944762+0000", 2024-03-27T21:40:21.244 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110412 2024-03-27T21:40:21.244 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.244 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.244 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:21.244 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944820+0000", 2024-03-27T21:40:21.244 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7649999999999999e-05 2024-03-27T21:40:21.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.245 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.245 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:21.245 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945585+0000", 2024-03-27T21:40:21.245 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076562600000000005 2024-03-27T21:40:21.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.245 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.245 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:21.245 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945620+0000", 2024-03-27T21:40:21.245 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4764000000000001e-05 2024-03-27T21:40:21.245 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.245 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:21.245 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.245 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.246 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:86 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'50)", 2024-03-27T21:40:21.246 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:21.246 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 41.266284978000002, 2024-03-27T21:40:21.246 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011392380000000001, 2024-03-27T21:40:21.246 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:21.246 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:21.246 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:21.246 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.246 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:21.246 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:21.246 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.246 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.246 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.246 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:21.247 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:21.247 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.247 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.247 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.247 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:21.247 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:21.247 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:21.247 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.247 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.247 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:21.247 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952373+0000", 2024-03-27T21:40:21.247 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9220000000000003e-06 2024-03-27T21:40:21.247 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.247 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.248 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:21.248 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952381+0000", 2024-03-27T21:40:21.248 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3070000000000003e-06 2024-03-27T21:40:21.248 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.248 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.248 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:21.248 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952385+0000", 2024-03-27T21:40:21.248 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.121e-06 2024-03-27T21:40:21.248 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.248 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.248 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:21.248 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952419+0000", 2024-03-27T21:40:21.248 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4082999999999999e-05 2024-03-27T21:40:21.248 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.248 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.249 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:21.249 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952440+0000", 2024-03-27T21:40:21.249 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0837999999999999e-05 2024-03-27T21:40:21.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.249 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.249 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:21.249 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953472+0000", 2024-03-27T21:40:21.249 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001032119 2024-03-27T21:40:21.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.249 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.249 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:21.249 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953504+0000", 2024-03-27T21:40:21.249 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1250000000000001e-05 2024-03-27T21:40:21.249 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.249 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:21.250 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.250 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.250 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:87 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-27T21:40:21.250 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:21.250 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 41.265917721000001, 2024-03-27T21:40:21.250 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00135519, 2024-03-27T21:40:21.250 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:21.250 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:21.250 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:21.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.250 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:21.250 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:21.250 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.251 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:21.251 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:21.251 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.251 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:21.251 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952731+0000", 2024-03-27T21:40:21.251 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:21.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.251 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:21.251 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952781+0000", 2024-03-27T21:40:21.251 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0161000000000003e-05 2024-03-27T21:40:21.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.252 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:21.252 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952794+0000", 2024-03-27T21:40:21.252 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2853e-05 2024-03-27T21:40:21.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.252 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:21.252 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952799+0000", 2024-03-27T21:40:21.252 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5560000000000001e-06 2024-03-27T21:40:21.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.252 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:21.252 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952832+0000", 2024-03-27T21:40:21.252 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3834999999999997e-05 2024-03-27T21:40:21.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.253 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:21.253 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952851+0000", 2024-03-27T21:40:21.253 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8289999999999999e-05 2024-03-27T21:40:21.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.253 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:21.253 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954055+0000", 2024-03-27T21:40:21.253 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012037949999999999 2024-03-27T21:40:21.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.253 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:21.254 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954087+0000", 2024-03-27T21:40:21.254 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2292999999999997e-05 2024-03-27T21:40:21.254 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.254 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:21.254 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.254 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.254 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.254 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:89 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'59, mlcod=17'57)", 2024-03-27T21:40:21.254 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:21.254 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 41.263643500000001, 2024-03-27T21:40:21.254 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00138893, 2024-03-27T21:40:21.254 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:21.254 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:21.255 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:21.255 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.255 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:21.255 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:21.255 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.255 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.255 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.255 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:21.255 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:21.255 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.255 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.255 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.255 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:21.255 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955005+0000", 2024-03-27T21:40:21.256 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:21.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.256 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.256 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:21.256 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955071+0000", 2024-03-27T21:40:21.256 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5290999999999995e-05 2024-03-27T21:40:21.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.256 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.256 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:21.256 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955081+0000", 2024-03-27T21:40:21.256 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0572e-05 2024-03-27T21:40:21.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.256 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.256 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:21.256 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955086+0000", 2024-03-27T21:40:21.257 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8380000000000001e-06 2024-03-27T21:40:21.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.257 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:21.257 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955122+0000", 2024-03-27T21:40:21.257 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5698000000000003e-05 2024-03-27T21:40:21.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.257 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:21.257 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955140+0000", 2024-03-27T21:40:21.257 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7900000000000001e-05 2024-03-27T21:40:21.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.257 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:21.257 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956366+0000", 2024-03-27T21:40:21.257 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001226471 2024-03-27T21:40:21.258 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.258 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.258 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:21.258 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956395+0000", 2024-03-27T21:40:21.258 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8782e-05 2024-03-27T21:40:21.258 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.258 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:21.258 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.258 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.258 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.258 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:90 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'60, mlcod=17'57)", 2024-03-27T21:40:21.258 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:21.258 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 41.262958922999999, 2024-03-27T21:40:21.258 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010778859999999999, 2024-03-27T21:40:21.259 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:21.259 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:21.259 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:21.259 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.259 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:21.259 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:21.259 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.259 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.259 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:21.259 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:21.259 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.259 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.259 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:21.260 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:21.260 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:21.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.260 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.260 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:21.260 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955747+0000", 2024-03-27T21:40:21.260 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7130999999999999e-05 2024-03-27T21:40:21.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.260 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.260 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:21.260 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955756+0000", 2024-03-27T21:40:21.261 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2750000000000001e-06 2024-03-27T21:40:21.261 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.261 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.261 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:21.261 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955761+0000", 2024-03-27T21:40:21.261 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3900000000000003e-06 2024-03-27T21:40:21.261 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.262 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.262 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:21.262 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955797+0000", 2024-03-27T21:40:21.262 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6650000000000003e-05 2024-03-27T21:40:21.262 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.262 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.262 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:21.263 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955816+0000", 2024-03-27T21:40:21.263 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8195000000000001e-05 2024-03-27T21:40:21.263 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.263 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.263 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:21.263 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956739+0000", 2024-03-27T21:40:21.264 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00092369700000000002 2024-03-27T21:40:21.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.264 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.264 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:21.264 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956768+0000", 2024-03-27T21:40:21.264 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9048e-05 2024-03-27T21:40:21.264 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.265 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:21.265 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.265 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.265 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.265 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:118 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'80, mlcod=17'78)", 2024-03-27T21:40:21.265 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:21.265 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 41.129968099999999, 2024-03-27T21:40:21.266 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014200669999999999, 2024-03-27T21:40:21.266 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:21.266 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:21.266 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:21.266 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.266 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:21.266 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:21.267 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.267 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.267 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.267 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:21.267 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:21.267 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.267 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.268 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.268 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:21.268 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:21.268 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:21.268 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.268 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.268 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:21.269 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088756+0000", 2024-03-27T21:40:21.269 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4902999999999998e-05 2024-03-27T21:40:21.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.269 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.269 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:21.269 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088764+0000", 2024-03-27T21:40:21.270 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8169999999999999e-06 2024-03-27T21:40:21.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.270 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.270 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:21.270 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088767+0000", 2024-03-27T21:40:21.270 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8310000000000003e-06 2024-03-27T21:40:21.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.270 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.270 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:21.271 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088798+0000", 2024-03-27T21:40:21.271 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0800999999999999e-05 2024-03-27T21:40:21.271 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.271 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.271 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:21.271 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088817+0000", 2024-03-27T21:40:21.271 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8467000000000001e-05 2024-03-27T21:40:21.271 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.271 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.271 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:21.271 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090074+0000", 2024-03-27T21:40:21.271 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012574089999999999 2024-03-27T21:40:21.271 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.272 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.272 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:21.272 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090101+0000", 2024-03-27T21:40:21.272 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7223e-05 2024-03-27T21:40:21.272 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.272 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:21.272 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.272 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.272 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:119 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'81, mlcod=17'78)", 2024-03-27T21:40:21.272 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:21.272 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 41.129233305, 2024-03-27T21:40:21.272 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014990870000000001, 2024-03-27T21:40:21.272 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:21.272 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089491+0000", 2024-03-27T21:40:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4805e-05 2024-03-27T21:40:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:21.274 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089498+0000", 2024-03-27T21:40:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0650000000000001e-06 2024-03-27T21:40:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089501+0000", 2024-03-27T21:40:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.54e-06 2024-03-27T21:40:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089533+0000", 2024-03-27T21:40:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1838999999999998e-05 2024-03-27T21:40:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089550+0000", 2024-03-27T21:40:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7465999999999998e-05 2024-03-27T21:40:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:21.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090890+0000", 2024-03-27T21:40:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013397979999999999 2024-03-27T21:40:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090915+0000", 2024-03-27T21:40:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4905e-05 2024-03-27T21:40:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:120 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'82, mlcod=17'78)", 2024-03-27T21:40:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:21.277 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 41.128433424999997, 2024-03-27T21:40:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001527093, 2024-03-27T21:40:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.279 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090267+0000", 2024-03-27T21:40:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1477000000000002e-05 2024-03-27T21:40:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090273+0000", 2024-03-27T21:40:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9519999999999999e-06 2024-03-27T21:40:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090276+0000", 2024-03-27T21:40:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.219e-06 2024-03-27T21:40:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:21.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090304+0000", 2024-03-27T21:40:21.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8073999999999999e-05 2024-03-27T21:40:21.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:21.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090316+0000", 2024-03-27T21:40:21.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2088e-05 2024-03-27T21:40:21.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:21.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091717+0000", 2024-03-27T21:40:21.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014003889999999999 2024-03-27T21:40:21.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:21.282 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091743+0000", 2024-03-27T21:40:21.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6234e-05 2024-03-27T21:40:21.282 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.282 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:21.282 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.283 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:121 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'83, mlcod=17'78)", 2024-03-27T21:40:21.283 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:21.283 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 41.127623059999998, 2024-03-27T21:40:21.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010226899999999999, 2024-03-27T21:40:21.283 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:21.283 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:21.283 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:21.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.283 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:21.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:21.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:21.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:21.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:21.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:21.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:21.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:21.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091142+0000", 2024-03-27T21:40:21.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011608300000000001 2024-03-27T21:40:21.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:21.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091151+0000", 2024-03-27T21:40:21.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2350000000000005e-06 2024-03-27T21:40:21.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:21.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091156+0000", 2024-03-27T21:40:21.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5419999999999999e-06 2024-03-27T21:40:21.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.286 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:21.286 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091191+0000", 2024-03-27T21:40:21.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5453000000000001e-05 2024-03-27T21:40:21.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.286 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:21.286 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091208+0000", 2024-03-27T21:40:21.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7119999999999999e-05 2024-03-27T21:40:21.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:21.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092024+0000", 2024-03-27T21:40:21.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000815539 2024-03-27T21:40:21.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:21.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092049+0000", 2024-03-27T21:40:21.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5171e-05 2024-03-27T21:40:21.287 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.287 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:21.287 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.287 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:142 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'100, mlcod=18'98)", 2024-03-27T21:40:21.288 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:21.288 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 40.712616105999999, 2024-03-27T21:40:21.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001198778, 2024-03-27T21:40:21.288 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:21.288 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:21.288 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:21.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.288 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:21.288 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:21.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:21.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:21.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:21.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:21.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:21.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.290 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:21.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506101+0000", 2024-03-27T21:40:21.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8202999999999997e-05 2024-03-27T21:40:21.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:21.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506109+0000", 2024-03-27T21:40:21.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3059999999999998e-06 2024-03-27T21:40:21.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:21.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506113+0000", 2024-03-27T21:40:21.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0119999999999999e-06 2024-03-27T21:40:21.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:21.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506137+0000", 2024-03-27T21:40:21.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4043999999999999e-05 2024-03-27T21:40:21.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:21.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506148+0000", 2024-03-27T21:40:21.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0446e-05 2024-03-27T21:40:21.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:21.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507213+0000", 2024-03-27T21:40:21.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001065215 2024-03-27T21:40:21.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.294 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:21.294 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507232+0000", 2024-03-27T21:40:21.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9038e-05 2024-03-27T21:40:21.294 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.294 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:21.294 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.295 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:143 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'101, mlcod=18'98)", 2024-03-27T21:40:21.295 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:21.295 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 40.712003486999997, 2024-03-27T21:40:21.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001357488, 2024-03-27T21:40:21.295 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:21.296 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:21.296 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:21.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.296 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:21.296 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:21.296 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.296 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:21.296 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:21.296 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:21.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:21.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:21.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:21.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506723+0000", 2024-03-27T21:40:21.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7404000000000006e-05 2024-03-27T21:40:21.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:21.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506730+0000", 2024-03-27T21:40:21.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5749999999999997e-06 2024-03-27T21:40:21.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:21.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506733+0000", 2024-03-27T21:40:21.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7979999999999999e-06 2024-03-27T21:40:21.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:21.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506758+0000", 2024-03-27T21:40:21.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4179e-05 2024-03-27T21:40:21.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:21.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506768+0000", 2024-03-27T21:40:21.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0740000000000001e-05 2024-03-27T21:40:21.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:21.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507978+0000", 2024-03-27T21:40:21.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012093900000000001 2024-03-27T21:40:21.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:21.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508003+0000", 2024-03-27T21:40:21.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5667e-05 2024-03-27T21:40:21.299 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.299 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:21.299 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.300 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:144 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'102, mlcod=18'98)", 2024-03-27T21:40:21.300 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:21.300 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 40.711344590000003, 2024-03-27T21:40:21.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013453390000000001, 2024-03-27T21:40:21.300 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:21.300 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:21.300 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:21.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:21.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:21.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:21.301 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:21.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:21.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.301 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:21.301 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507304+0000", 2024-03-27T21:40:21.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:21.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.301 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:21.301 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507397+0000", 2024-03-27T21:40:21.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2995e-05 2024-03-27T21:40:21.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.301 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:21.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507403+0000", 2024-03-27T21:40:21.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8640000000000001e-06 2024-03-27T21:40:21.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:21.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507407+0000", 2024-03-27T21:40:21.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5710000000000002e-06 2024-03-27T21:40:21.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:21.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507428+0000", 2024-03-27T21:40:21.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1192e-05 2024-03-27T21:40:21.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:21.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507438+0000", 2024-03-27T21:40:21.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8309999999999996e-06 2024-03-27T21:40:21.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:21.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508624+0000", 2024-03-27T21:40:21.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011861210000000001 2024-03-27T21:40:21.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:21.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:21.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:21.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508650+0000", 2024-03-27T21:40:21.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6118e-05 2024-03-27T21:40:21.303 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.304 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:21.304 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.304 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:21.304 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:21.304 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:21.305 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:40:21.306 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:21.319+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:21.341 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:40:21.406 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:21.420+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:21.465 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:21.465 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:21.465 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:21.465 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:21.476 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:40:21.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.505+0000 7f292ca70640 10 monclient: get_monmap_and_config 2024-03-27T21:40:21.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.505+0000 7f292ca70640 10 monclient: build_initial_monmap 2024-03-27T21:40:21.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.505+0000 7f292ca70640 10 monclient: monmap: 2024-03-27T21:40:21.492 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:21.492 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:21.492 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:21.506407+0000 2024-03-27T21:40:21.492 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:21.506407+0000 2024-03-27T21:40:21.492 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:21.492 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:21.492 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:21.492 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:21.492 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:21.492 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:21.492 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:21.493 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:21.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.506+0000 7f292ca70640 1 Processor -- start 2024-03-27T21:40:21.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.506+0000 7f292ca70640 1 -- start start 2024-03-27T21:40:21.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.506+0000 7f292ca70640 10 monclient: init 2024-03-27T21:40:21.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.507+0000 7f292ca70640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:21.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.507+0000 7f292ca70640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-03-27T21:40:21.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.507+0000 7f292ca70640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f29280b40b0 0x7f29280a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:21.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.507+0000 7f292ca70640 10 monclient(hunting): picked mon.noname-a con 0x7f29280b40b0 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:21.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.507+0000 7f292ca70640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f292813b020 0x7f292813b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:21.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.507+0000 7f292ca70640 10 monclient(hunting): picked mon.noname-c con 0x7f292813b020 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:21.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.507+0000 7f292659c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f29280b40b0 0x7f29280a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:21.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.507+0000 7f292659c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f29280b40b0 0x7f29280a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36000/0 (socket says 172.21.15.60:36000) 2024-03-27T21:40:21.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.507+0000 7f292659c640 1 -- 172.21.15.60:0/3668954634 learned_addr learned my addr 172.21.15.60:0/3668954634 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:21.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.507+0000 7f292ca70640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f29281416f0 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:21.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.507+0000 7f291dd9b640 1 --2- 172.21.15.60:0/3668954634 >> v2:172.21.15.191:3300/0 conn(0x7f292813b020 0x7f292813b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:21.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.507+0000 7f292ca70640 1 -- 172.21.15.60:0/3668954634 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29280b3830 con 0x7f29281416f0 2024-03-27T21:40:21.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.507+0000 7f292ca70640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:21.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.507+0000 7f292ca70640 1 -- 172.21.15.60:0/3668954634 --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29280b4490 con 0x7f29280b40b0 2024-03-27T21:40:21.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.507+0000 7f292ca70640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:21.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.507+0000 7f292ca70640 1 -- 172.21.15.60:0/3668954634 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29280aa520 con 0x7f292813b020 2024-03-27T21:40:21.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.507+0000 7f292ca70640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:21.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.507+0000 7f292ca70640 10 monclient(hunting): authenticate will time out at 862.728832s 2024-03-27T21:40:21.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 1 -- 172.21.15.60:0/3668954634 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134743785 0 0) 0x7f29280b3830 con 0x7f29281416f0 2024-03-27T21:40:21.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 10 monclient(hunting): _negotiate 2024-03-27T21:40:21.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:21.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:21.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 10 monclient(hunting): my global_id is 4608 2024-03-27T21:40:21.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 1 -- 172.21.15.60:0/3668954634 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2914003520 con 0x7f29281416f0 2024-03-27T21:40:21.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 1 -- 172.21.15.60:0/3668954634 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3611950994 0 0) 0x7f2914003520 con 0x7f29281416f0 2024-03-27T21:40:21.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 1 -- 172.21.15.60:0/3668954634 >> v2:172.21.15.191:3300/0 conn(0x7f292813b020 msgr2=0x7f292813b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:21.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 1 --2- 172.21.15.60:0/3668954634 >> v2:172.21.15.191:3300/0 conn(0x7f292813b020 0x7f292813b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:21.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 1 -- 172.21.15.60:0/3668954634 >> v2:172.21.15.60:3300/0 conn(0x7f29280b40b0 msgr2=0x7f29280a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:21.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 1 --2- 172.21.15.60:0/3668954634 >> v2:172.21.15.60:3300/0 conn(0x7f29280b40b0 0x7f29280a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:21.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 10 monclient: _finish_hunting 0 2024-03-27T21:40:21.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 1 monclient: found mon.noname-a-legacy 2024-03-27T21:40:21.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:21.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.60:6789/0 2024-03-27T21:40:21.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 1 -- 172.21.15.60:0/3668954634 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29280a9470 con 0x7f29281416f0 2024-03-27T21:40:21.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 10 monclient: _finish_auth 0 2024-03-27T21:40:21.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 10 monclient: _check_auth_tickets 2024-03-27T21:40:21.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:21.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f292ca70640 5 monclient: authenticate success, global_id 4608 2024-03-27T21:40:21.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f291dd9b640 10 monclient: get_auth_request con 0x7f292813b020 auth_method 0 2024-03-27T21:40:21.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f291dd9b640 1 --2- 172.21.15.60:0/3668954634 >> v2:172.21.15.191:3300/0 conn(0x7f292813b020 0x7f292813b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:40:21.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 1 -- 172.21.15.60:0/3668954634 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2933500084 0 0) 0x7f2920001650 con 0x7f29281416f0 2024-03-27T21:40:21.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:21.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-03-27T21:40:21.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 10 monclient: dump: 2024-03-27T21:40:21.512 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:21.512 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:21.512 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:21.512 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:21.512 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:21.512 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:21.512 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:21.513 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:21.513 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:21.513 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:21.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] but i'm connected to v1:172.21.15.60:6789/0, reconnecting 2024-03-27T21:40:21.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:21.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 1 -- 172.21.15.60:0/3668954634 >> v1:172.21.15.60:6789/0 conn(0x7f29281416f0 legacy=0x7f2928141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:21.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:40:21.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 1 --2- 172.21.15.60:0/3668954634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29140039b0 0x7f2914003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:21.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 10 monclient(hunting): picked mon.a con 0x7f29140039b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:21.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 1 --2- 172.21.15.60:0/3668954634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29140047c0 0x7f2914004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:21.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 10 monclient(hunting): picked mon.b con 0x7f29140047c0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:21.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 1 --2- 172.21.15.60:0/3668954634 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f29140052a0 0x7f291400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:21.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 10 monclient(hunting): picked mon.c con 0x7f29140052a0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:21.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:21.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 1 -- 172.21.15.60:0/3668954634 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29280b4490 con 0x7f29140039b0 2024-03-27T21:40:21.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:21.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 1 -- 172.21.15.60:0/3668954634 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29280aa520 con 0x7f29140047c0 2024-03-27T21:40:21.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:21.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 1 -- 172.21.15.60:0/3668954634 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f291401d2e0 con 0x7f29140052a0 2024-03-27T21:40:21.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:21.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2925d9b640 10 monclient(hunting): _finish_auth 0 2024-03-27T21:40:21.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f292ca70640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:21.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f291dd9b640 1 --2- 172.21.15.60:0/3668954634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29140047c0 0x7f2914004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:21.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2926d9d640 1 --2- 172.21.15.60:0/3668954634 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f29140052a0 0x7f291400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:21.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f291dd9b640 10 monclient(hunting): get_auth_request con 0x7f29140047c0 auth_method 0 2024-03-27T21:40:21.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f291dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:21.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f291dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:21.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f291dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:40:21.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2926d9d640 10 monclient(hunting): get_auth_request con 0x7f29140052a0 auth_method 0 2024-03-27T21:40:21.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2926d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:21.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2926d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:21.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2926d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:40:21.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f291dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:21.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f291dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:21.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f291dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:40:21.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2926d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:21.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2926d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:21.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.508+0000 7f2926d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:40:21.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.509+0000 7f2926d9d640 10 monclient(hunting): handle_auth_done global_id 4608 payload 293 2024-03-27T21:40:21.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.509+0000 7f2926d9d640 1 -- 172.21.15.60:0/3668954634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29140047c0 msgr2=0x7f2914004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:21.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.509+0000 7f2926d9d640 1 --2- 172.21.15.60:0/3668954634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29140047c0 0x7f2914004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:21.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.509+0000 7f2926d9d640 1 -- 172.21.15.60:0/3668954634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29140039b0 msgr2=0x7f2914003df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T21:40:21.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.509+0000 7f2926d9d640 1 --2- 172.21.15.60:0/3668954634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29140039b0 0x7f2914003df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.509+0000 7f2926d9d640 10 monclient: _finish_hunting 0 2024-03-27T21:40:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.509+0000 7f2926d9d640 1 monclient: found mon.c 2024-03-27T21:40:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.509+0000 7f2926d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.509+0000 7f2926d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.509+0000 7f2926d9d640 1 -- 172.21.15.60:0/3668954634 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29280a9470 con 0x7f29140052a0 2024-03-27T21:40:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.509+0000 7f2926d9d640 10 monclient: _finish_auth 0 2024-03-27T21:40:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.509+0000 7f2926d9d640 10 monclient: _check_auth_tickets 2024-03-27T21:40:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.509+0000 7f2926d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.509+0000 7f291dd9b640 1 --2- 172.21.15.60:0/3668954634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29140047c0 0x7f2914004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.509+0000 7f2926d9d640 1 -- 172.21.15.60:0/3668954634 reap_dead start 2024-03-27T21:40:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.510+0000 7f2926d9d640 1 --2- 172.21.15.60:0/3668954634 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f29140052a0 0x7f291400b980 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f292000c1e0 tx=0x7f29140057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c69a281f4b1ad22a server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.510+0000 7f2925d9b640 1 -- 172.21.15.60:0/3668954634 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292002ca10 con 0x7f29140052a0 2024-03-27T21:40:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.510+0000 7f2925d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.510+0000 7f2925d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.510+0000 7f2925d9b640 10 monclient: dump: 2024-03-27T21:40:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:21.517 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.510+0000 7f292ca70640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.510+0000 7f2925d9b640 1 -- 172.21.15.60:0/3668954634 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f292002cb70 con 0x7f29140052a0 2024-03-27T21:40:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.510+0000 7f2925d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.510+0000 7f292ca70640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.510+0000 7f292ca70640 4 set_mon_vals no callback set 2024-03-27T21:40:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.510+0000 7f2925d9b640 1 -- 172.21.15.60:0/3668954634 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292002ca10 con 0x7f29140052a0 2024-03-27T21:40:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.510+0000 7f2925d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.510+0000 7f2925d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.510+0000 7f2925d9b640 10 monclient: dump: 2024-03-27T21:40:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:21.519 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.510+0000 7f292659c640 10 monclient: get_auth_request con 0x7f29280b40b0 auth_method 0 2024-03-27T21:40:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.510+0000 7f292659c640 1 --2- 172.21.15.60:0/3668954634 >> v2:172.21.15.60:3300/0 conn(0x7f29280b40b0 0x7f29280a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:40:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.511+0000 7f292ca70640 10 monclient: shutdown 2024-03-27T21:40:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.511+0000 7f292ca70640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.511+0000 7f292ca70640 1 -- 172.21.15.60:0/3668954634 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f29140052a0 msgr2=0x7f291400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.511+0000 7f292ca70640 1 --2- 172.21.15.60:0/3668954634 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f29140052a0 0x7f291400b980 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f292000c1e0 tx=0x7f29140057f0 comp rx=0 tx=0).stop 2024-03-27T21:40:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.511+0000 7f292ca70640 1 -- 172.21.15.60:0/3668954634 shutdown_connections 2024-03-27T21:40:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.511+0000 7f292ca70640 1 --2- 172.21.15.60:0/3668954634 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f29140052a0 0x7f291400b980 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.511+0000 7f292ca70640 1 -- 172.21.15.60:0/3668954634 >> 172.21.15.60:0/3668954634 conn(0x7f29280608e0 msgr2=0x7f2928060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.511+0000 7f292ca70640 1 -- 172.21.15.60:0/3668954634 shutdown_connections 2024-03-27T21:40:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.511+0000 7f292ca70640 1 -- 172.21.15.60:0/3668954634 wait complete. 2024-03-27T21:40:21.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.512+0000 7f292ca70640 10 monclient: build_initial_monmap 2024-03-27T21:40:21.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.512+0000 7f292ca70640 10 monclient: monmap: 2024-03-27T21:40:21.520 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:21.520 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:21.520 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:21.520 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:21.520 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:21.520 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:21.520 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:21.520 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:21.520 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:21.520 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:21.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.512+0000 7f292ca70640 1 librados: starting msgr at 2024-03-27T21:40:21.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.512+0000 7f292ca70640 1 librados: starting objecter 2024-03-27T21:40:21.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.512+0000 7f292ca70640 1 Processor -- start 2024-03-27T21:40:21.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.512+0000 7f292ca70640 1 -- start start 2024-03-27T21:40:21.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.512+0000 7f292ca70640 1 librados: setting wanted keys 2024-03-27T21:40:21.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.512+0000 7f292ca70640 1 librados: calling monclient init 2024-03-27T21:40:21.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.512+0000 7f292ca70640 10 monclient: init 2024-03-27T21:40:21.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292ca70640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:21.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292ca70640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:40:21.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292ca70640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f29140052a0 0x7f29280ab250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:21.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292ca70640 10 monclient(hunting): picked mon.noname-c con 0x7f29140052a0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:21.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292ca70640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29280abeb0 0x7f29280ac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:21.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292ca70640 10 monclient(hunting): picked mon.noname-a con 0x7f29280abeb0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:21.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292ca70640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29281d2310 0x7f29281d2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:21.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292ca70640 10 monclient(hunting): picked mon.noname-b con 0x7f29281d2310 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:21.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292ca70640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:21.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292ca70640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29280ad4b0 con 0x7f29280abeb0 2024-03-27T21:40:21.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292ca70640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:21.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292ca70640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29281d2cb0 con 0x7f29281d2310 2024-03-27T21:40:21.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292ca70640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:21.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292ca70640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29281d1e80 con 0x7f29140052a0 2024-03-27T21:40:21.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292ca70640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:21.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292ca70640 10 monclient(hunting): authenticate will time out at 862.734329s 2024-03-27T21:40:21.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f291dd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29280abeb0 0x7f29280ac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:21.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f291dd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29280abeb0 0x7f29280ac2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36016/0 (socket says 172.21.15.60:36016) 2024-03-27T21:40:21.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f291dd9b640 1 -- 172.21.15.60:0/2310078750 learned_addr learned my addr 172.21.15.60:0/2310078750 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f291dd9b640 10 monclient(hunting): get_auth_request con 0x7f29280abeb0 auth_method 0 2024-03-27T21:40:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f291dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f291dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f291dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292659c640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f29140052a0 0x7f29280ab250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f2926d9d640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29281d2310 0x7f29281d2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f291dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f291dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f291dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292659c640 10 monclient(hunting): get_auth_request con 0x7f29140052a0 auth_method 0 2024-03-27T21:40:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292659c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f2926d9d640 10 monclient(hunting): get_auth_request con 0x7f29281d2310 auth_method 0 2024-03-27T21:40:21.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f2926d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:21.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f2926d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:21.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f2926d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:21.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:21.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:21.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f292659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:21.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f2926d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:21.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f2926d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:21.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f2926d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:21.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f291dd9b640 10 monclient(hunting): handle_auth_done global_id 4611 payload 824 2024-03-27T21:40:21.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.513+0000 7f291dd9b640 1 -- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f29140052a0 msgr2=0x7f29280ab250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:21.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291dd9b640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f29140052a0 0x7f29280ab250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291dd9b640 1 -- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29281d2310 msgr2=0x7f29281d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291dd9b640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29281d2310 0x7f29281d2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291dd9b640 10 monclient: _finish_hunting 0 2024-03-27T21:40:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291dd9b640 1 monclient: found mon.noname-a 2024-03-27T21:40:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291dd9b640 1 -- 172.21.15.60:0/2310078750 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f292806fd20 con 0x7f29280abeb0 2024-03-27T21:40:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f2926d9d640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29281d2310 0x7f29281d2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f292659c640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f29140052a0 0x7f29280ab250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291dd9b640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29280abeb0 0x7f29280ac2f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f291000fa50 tx=0x7f291000ff10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291f7fe640 10 client.?.objecter ms_handle_connect 0x7f29280abeb0 2024-03-27T21:40:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291f7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291000ddd0 con 0x7f29280abeb0 2024-03-27T21:40:21.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:21.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:21.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291f7fe640 10 monclient: dump: 2024-03-27T21:40:21.526 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:21.526 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:21.526 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:21.526 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:21.526 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:21.526 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:21.526 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:21.526 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:21.526 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:21.526 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:21.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291f7fe640 10 monclient: _finish_auth 0 2024-03-27T21:40:21.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291f7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:40:21.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:21.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f292ca70640 5 monclient: authenticate success, global_id 4611 2024-03-27T21:40:21.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f292ca70640 10 monclient: _renew_subs 2024-03-27T21:40:21.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f292ca70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:21.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f292ca70640 1 -- 172.21.15.60:0/2310078750 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f292806dff0 con 0x7f29280abeb0 2024-03-27T21:40:21.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f292ca70640 10 client.4611.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:21.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f292ca70640 10 monclient: _renew_subs 2024-03-27T21:40:21.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f292ca70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:21.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f292ca70640 1 -- 172.21.15.60:0/2310078750 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2928139490 con 0x7f29280abeb0 2024-03-27T21:40:21.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f292ca70640 1 librados: init done 2024-03-27T21:40:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f291000c040 con 0x7f29280abeb0 2024-03-27T21:40:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2910005b90 con 0x7f29280abeb0 2024-03-27T21:40:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f291f7fe640 10 monclient: dump: 2024-03-27T21:40:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:21.529 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:21.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.514+0000 7f292559a640 4 set_mon_vals no callback set 2024-03-27T21:40:21.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.516+0000 7f291cd99640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:21.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.516+0000 7f291cd99640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:21.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.516+0000 7f291cd99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:21.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.516+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f2928069470 con 0x7f29280abeb0 2024-03-27T21:40:21.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.516+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2910005cf0 con 0x7f29280abeb0 2024-03-27T21:40:21.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f291f7fe640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f290806c5c0 0x7f290806ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:21.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f29100899a0 con 0x7f29280abeb0 2024-03-27T21:40:21.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f291f7fe640 10 client.4611.objecter ms_dispatch 0x7f2928000b90 osd_map(34..34 src has 1..34) v4 2024-03-27T21:40:21.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f291f7fe640 3 client.4611.objecter handle_osd_map got epochs [34,34] > 0 2024-03-27T21:40:21.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f291f7fe640 3 client.4611.objecter handle_osd_map decoding full epoch 34 2024-03-27T21:40:21.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f291f7fe640 20 client.4611.objecter dump_active .. 0 homeless 2024-03-27T21:40:21.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v34) v1 ==== 54+0+16 (secure 0 0 0) 0x7f291008e220 con 0x7f29280abeb0 2024-03-27T21:40:21.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f291f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:21.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f291f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:21.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f292659c640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f290806c5c0 0x7f290806ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:21.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f292659c640 10 monclient: get_auth_request con 0x7f290806c5c0 auth_method 0 2024-03-27T21:40:21.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f291cd99640 10 client.4611.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:21.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign -1 1 2024-03-27T21:40:21.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f291cd99640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f292813b480 0x7f29281d57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:21.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281d1fc0 osd=0 3 2024-03-27T21:40:21.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281d1fc0 osd=0 3 2024-03-27T21:40:21.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281d1fc0 osd=0 3 2024-03-27T21:40:21.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f291cd99640 15 client.4611.objecter _session_command_op_remove -1 1 2024-03-27T21:40:21.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f291cd99640 20 client.4611.objecter get_session s=0x7f29281d1fc0 osd=0 3 2024-03-27T21:40:21.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign 0 1 2024-03-27T21:40:21.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281d1fc0 osd=0 4 2024-03-27T21:40:21.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f291cd99640 10 client.4611.objecter _send_command 1 2024-03-27T21:40:21.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.517+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f29281d5ce0 con 0x7f292813b480 2024-03-27T21:40:21.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.518+0000 7f2926d9d640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f292813b480 0x7f29281d57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:21.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.518+0000 7f2926d9d640 10 monclient: get_auth_request con 0x7f292813b480 auth_method 0 2024-03-27T21:40:21.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.518+0000 7f291f7fe640 10 client.4611.objecter ms_handle_connect 0x7f292813b480 2024-03-27T21:40:21.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.518+0000 7f2926d9d640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f292813b480 0x7f29281d57a0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:21.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.519+0000 7f291f7fe640 10 client.4611.objecter ms_handle_connect 0x7f290806c5c0 2024-03-27T21:40:21.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.519+0000 7f292659c640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f290806c5c0 0x7f290806ea80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2918007fd0 tx=0x7f291800c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:21.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.519+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f29281d5ce0 con 0x7f292813b480 2024-03-27T21:40:21.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.519+0000 7f291f7fe640 10 client.4611.objecter ms_dispatch 0x7f2928000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:21.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.519+0000 7f291f7fe640 10 client.4611.objecter _finish_command 1 = system:0 2024-03-27T21:40:21.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.519+0000 7f291f7fe640 20 client.4611.objecter put_session s=0x7f29281d1fc0 osd=0 4 2024-03-27T21:40:21.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.519+0000 7f291f7fe640 15 client.4611.objecter _session_command_op_remove 0 1 2024-03-27T21:40:21.533 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:21.547+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:21.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.548+0000 7f291cd99640 10 client.4611.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:21.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.548+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign -1 2 2024-03-27T21:40:21.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.548+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281d1fc0 osd=0 3 2024-03-27T21:40:21.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.548+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281d1fc0 osd=0 3 2024-03-27T21:40:21.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.548+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281d1fc0 osd=0 3 2024-03-27T21:40:21.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.548+0000 7f291cd99640 15 client.4611.objecter _session_command_op_remove -1 2 2024-03-27T21:40:21.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.548+0000 7f291cd99640 20 client.4611.objecter get_session s=0x7f29281d1fc0 osd=0 3 2024-03-27T21:40:21.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.548+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign 0 2 2024-03-27T21:40:21.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.548+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281d1fc0 osd=0 4 2024-03-27T21:40:21.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.548+0000 7f291cd99640 10 client.4611.objecter _send_command 2 2024-03-27T21:40:21.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.548+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f29281d6090 con 0x7f292813b480 2024-03-27T21:40:21.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.551+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f29281d6090 con 0x7f292813b480 2024-03-27T21:40:21.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.551+0000 7f291f7fe640 10 client.4611.objecter ms_dispatch 0x7f2928000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:21.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.551+0000 7f291f7fe640 10 client.4611.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:21.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.551+0000 7f291f7fe640 20 client.4611.objecter put_session s=0x7f29281d1fc0 osd=0 4 2024-03-27T21:40:21.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.551+0000 7f291f7fe640 15 client.4611.objecter _session_command_op_remove 0 2 2024-03-27T21:40:21.538 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:21.540 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:21.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.551+0000 7f291cd99640 10 client.4611.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:21.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.551+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign -1 3 2024-03-27T21:40:21.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.551+0000 7f291cd99640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f292813e740 0x7f29281e05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:21.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.552+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281399a0 osd=1 3 2024-03-27T21:40:21.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.552+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281399a0 osd=1 3 2024-03-27T21:40:21.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.552+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281399a0 osd=1 3 2024-03-27T21:40:21.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.552+0000 7f291cd99640 15 client.4611.objecter _session_command_op_remove -1 3 2024-03-27T21:40:21.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.552+0000 7f292659c640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f292813e740 0x7f29281e05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:21.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.552+0000 7f292659c640 10 monclient: get_auth_request con 0x7f292813e740 auth_method 0 2024-03-27T21:40:21.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.552+0000 7f291cd99640 20 client.4611.objecter get_session s=0x7f29281399a0 osd=1 3 2024-03-27T21:40:21.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.552+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign 1 3 2024-03-27T21:40:21.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.552+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281399a0 osd=1 4 2024-03-27T21:40:21.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.552+0000 7f291cd99640 10 client.4611.objecter _send_command 3 2024-03-27T21:40:21.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.552+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2928139b10 con 0x7f292813e740 2024-03-27T21:40:21.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.552+0000 7f291f7fe640 10 client.4611.objecter ms_handle_connect 0x7f292813e740 2024-03-27T21:40:21.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.552+0000 7f292659c640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f292813e740 0x7f29281e05e0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:21.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.553+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f2928139b10 con 0x7f292813e740 2024-03-27T21:40:21.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.553+0000 7f291f7fe640 10 client.4611.objecter ms_dispatch 0x7f2928000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:21.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.553+0000 7f291f7fe640 10 client.4611.objecter _finish_command 3 = system:0 2024-03-27T21:40:21.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.553+0000 7f291f7fe640 20 client.4611.objecter put_session s=0x7f29281399a0 osd=1 4 2024-03-27T21:40:21.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.553+0000 7f291f7fe640 15 client.4611.objecter _session_command_op_remove 1 3 2024-03-27T21:40:21.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.583+0000 7f291cd99640 10 client.4611.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:21.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.583+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign -1 4 2024-03-27T21:40:21.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.583+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281399a0 osd=1 3 2024-03-27T21:40:21.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.583+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281399a0 osd=1 3 2024-03-27T21:40:21.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.583+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281399a0 osd=1 3 2024-03-27T21:40:21.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.583+0000 7f291cd99640 15 client.4611.objecter _session_command_op_remove -1 4 2024-03-27T21:40:21.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.583+0000 7f291cd99640 20 client.4611.objecter get_session s=0x7f29281399a0 osd=1 3 2024-03-27T21:40:21.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.583+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign 1 4 2024-03-27T21:40:21.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.583+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281399a0 osd=1 4 2024-03-27T21:40:21.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.583+0000 7f291cd99640 10 client.4611.objecter _send_command 4 2024-03-27T21:40:21.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.583+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f29281e0b20 con 0x7f292813e740 2024-03-27T21:40:21.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.586+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f29281e0b20 con 0x7f292813e740 2024-03-27T21:40:21.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.586+0000 7f291f7fe640 10 client.4611.objecter ms_dispatch 0x7f2928000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:21.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.586+0000 7f291f7fe640 10 client.4611.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:21.573 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:21.575 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:21.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.586+0000 7f291f7fe640 20 client.4611.objecter put_session s=0x7f29281399a0 osd=1 4 2024-03-27T21:40:21.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.586+0000 7f291f7fe640 15 client.4611.objecter _session_command_op_remove 1 4 2024-03-27T21:40:21.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.586+0000 7f291cd99640 10 client.4611.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:21.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.586+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign -1 5 2024-03-27T21:40:21.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.586+0000 7f291cd99640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f292813de20 0x7f29281e3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.586+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f292813dcb0 osd=2 3 2024-03-27T21:40:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.586+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f292813dcb0 osd=2 3 2024-03-27T21:40:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.586+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f292813dcb0 osd=2 3 2024-03-27T21:40:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.586+0000 7f291cd99640 15 client.4611.objecter _session_command_op_remove -1 5 2024-03-27T21:40:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.586+0000 7f291cd99640 20 client.4611.objecter get_session s=0x7f292813dcb0 osd=2 3 2024-03-27T21:40:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.586+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign 2 5 2024-03-27T21:40:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.586+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f292813dcb0 osd=2 4 2024-03-27T21:40:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.586+0000 7f291cd99640 10 client.4611.objecter _send_command 5 2024-03-27T21:40:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.586+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2928066760 con 0x7f292813de20 2024-03-27T21:40:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.586+0000 7f291dd9b640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f292813de20 0x7f29281e3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.586+0000 7f291dd9b640 10 monclient: get_auth_request con 0x7f292813de20 auth_method 0 2024-03-27T21:40:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.587+0000 7f291f7fe640 10 client.4611.objecter ms_handle_connect 0x7f292813de20 2024-03-27T21:40:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.587+0000 7f291dd9b640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f292813de20 0x7f29281e3e80 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.588+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== osd.2 v2:172.21.15.60:6824/44654 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f2928066760 con 0x7f292813de20 2024-03-27T21:40:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.588+0000 7f291f7fe640 10 client.4611.objecter ms_dispatch 0x7f2928000b90 command_reply(tid 5: 0 ) v1 2024-03-27T21:40:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.588+0000 7f291f7fe640 10 client.4611.objecter _finish_command 5 = system:0 2024-03-27T21:40:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.588+0000 7f291f7fe640 20 client.4611.objecter put_session s=0x7f292813dcb0 osd=2 4 2024-03-27T21:40:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.588+0000 7f291f7fe640 15 client.4611.objecter _session_command_op_remove 2 5 2024-03-27T21:40:21.590 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:21.590 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:21.600 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:40:21.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.616+0000 7f291cd99640 10 client.4611.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:21.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.616+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign -1 6 2024-03-27T21:40:21.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.616+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f292813dcb0 osd=2 3 2024-03-27T21:40:21.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.616+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f292813dcb0 osd=2 3 2024-03-27T21:40:21.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.616+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f292813dcb0 osd=2 3 2024-03-27T21:40:21.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.616+0000 7f291cd99640 15 client.4611.objecter _session_command_op_remove -1 6 2024-03-27T21:40:21.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.616+0000 7f291cd99640 20 client.4611.objecter get_session s=0x7f292813dcb0 osd=2 3 2024-03-27T21:40:21.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.616+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign 2 6 2024-03-27T21:40:21.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.616+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f292813dcb0 osd=2 4 2024-03-27T21:40:21.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.616+0000 7f291cd99640 10 client.4611.objecter _send_command 6 2024-03-27T21:40:21.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.616+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f29281e43c0 con 0x7f292813de20 2024-03-27T21:40:21.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.619+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== osd.2 v2:172.21.15.60:6824/44654 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f29281e43c0 con 0x7f292813de20 2024-03-27T21:40:21.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.619+0000 7f291f7fe640 10 client.4611.objecter ms_dispatch 0x7f2928000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:21.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.619+0000 7f291f7fe640 10 client.4611.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:21.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.619+0000 7f291f7fe640 20 client.4611.objecter put_session s=0x7f292813dcb0 osd=2 4 2024-03-27T21:40:21.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.619+0000 7f291f7fe640 15 client.4611.objecter _session_command_op_remove 2 6 2024-03-27T21:40:21.606 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2: {} 2024-03-27T21:40:21.607 INFO:teuthology.orchestra.run.smithi060.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-27T21:40:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.619+0000 7f291cd99640 10 client.4611.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.619+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign -1 7 2024-03-27T21:40:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.619+0000 7f291cd99640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f2928065e40 0x7f29281e76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.619+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f2928065cd0 osd=3 3 2024-03-27T21:40:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.619+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f2928065cd0 osd=3 3 2024-03-27T21:40:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.619+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f2928065cd0 osd=3 3 2024-03-27T21:40:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.619+0000 7f291cd99640 15 client.4611.objecter _session_command_op_remove -1 7 2024-03-27T21:40:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.619+0000 7f291cd99640 20 client.4611.objecter get_session s=0x7f2928065cd0 osd=3 3 2024-03-27T21:40:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.619+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign 3 7 2024-03-27T21:40:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.619+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f2928065cd0 osd=3 4 2024-03-27T21:40:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.619+0000 7f291cd99640 10 client.4611.objecter _send_command 7 2024-03-27T21:40:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.619+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f29281e1960 con 0x7f2928065e40 2024-03-27T21:40:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.619+0000 7f2926d9d640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f2928065e40 0x7f29281e76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.619+0000 7f2926d9d640 10 monclient: get_auth_request con 0x7f2928065e40 auth_method 0 2024-03-27T21:40:21.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.620+0000 7f2926d9d640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f2928065e40 0x7f29281e76c0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:21.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.620+0000 7f291f7fe640 10 client.4611.objecter ms_handle_connect 0x7f2928065e40 2024-03-27T21:40:21.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.621+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f29281e1960 con 0x7f2928065e40 2024-03-27T21:40:21.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.621+0000 7f291f7fe640 10 client.4611.objecter ms_dispatch 0x7f2928000b90 command_reply(tid 7: 0 ) v1 2024-03-27T21:40:21.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.621+0000 7f291f7fe640 10 client.4611.objecter _finish_command 7 = system:0 2024-03-27T21:40:21.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.621+0000 7f291f7fe640 20 client.4611.objecter put_session s=0x7f2928065cd0 osd=3 4 2024-03-27T21:40:21.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.621+0000 7f291f7fe640 15 client.4611.objecter _session_command_op_remove 3 7 2024-03-27T21:40:21.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.637+0000 7f291cd99640 10 client.4611.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:21.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.638+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign -1 8 2024-03-27T21:40:21.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.638+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f2928065cd0 osd=3 3 2024-03-27T21:40:21.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.638+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f2928065cd0 osd=3 3 2024-03-27T21:40:21.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.638+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f2928065cd0 osd=3 3 2024-03-27T21:40:21.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.638+0000 7f291cd99640 15 client.4611.objecter _session_command_op_remove -1 8 2024-03-27T21:40:21.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.638+0000 7f291cd99640 20 client.4611.objecter get_session s=0x7f2928065cd0 osd=3 3 2024-03-27T21:40:21.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.638+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign 3 8 2024-03-27T21:40:21.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.638+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f2928065cd0 osd=3 4 2024-03-27T21:40:21.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.638+0000 7f291cd99640 10 client.4611.objecter _send_command 8 2024-03-27T21:40:21.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.638+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f29281e7c00 con 0x7f2928065e40 2024-03-27T21:40:21.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.639+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f29281e7c00 con 0x7f2928065e40 2024-03-27T21:40:21.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.639+0000 7f291f7fe640 10 client.4611.objecter ms_dispatch 0x7f2928000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:21.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.639+0000 7f291f7fe640 10 client.4611.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:21.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.639+0000 7f291f7fe640 20 client.4611.objecter put_session s=0x7f2928065cd0 osd=3 4 2024-03-27T21:40:21.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.639+0000 7f291f7fe640 15 client.4611.objecter _session_command_op_remove 3 8 2024-03-27T21:40:21.626 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:21.627 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:21.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.639+0000 7f291cd99640 10 client.4611.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:21.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.639+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign -1 9 2024-03-27T21:40:21.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.639+0000 7f291cd99640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f29281e8450 0x7f29281ea840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:21.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.639+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281e7dd0 osd=4 3 2024-03-27T21:40:21.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.639+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281e7dd0 osd=4 3 2024-03-27T21:40:21.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.639+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281e7dd0 osd=4 3 2024-03-27T21:40:21.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.639+0000 7f291cd99640 15 client.4611.objecter _session_command_op_remove -1 9 2024-03-27T21:40:21.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.639+0000 7f291cd99640 20 client.4611.objecter get_session s=0x7f29281e7dd0 osd=4 3 2024-03-27T21:40:21.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.639+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign 4 9 2024-03-27T21:40:21.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.639+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281e7dd0 osd=4 4 2024-03-27T21:40:21.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.639+0000 7f291cd99640 10 client.4611.objecter _send_command 9 2024-03-27T21:40:21.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.639+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f29281ed610 con 0x7f29281e8450 2024-03-27T21:40:21.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.640+0000 7f292659c640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f29281e8450 0x7f29281ea840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:21.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.640+0000 7f292659c640 10 monclient: get_auth_request con 0x7f29281e8450 auth_method 0 2024-03-27T21:40:21.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.640+0000 7f291f7fe640 10 client.4611.objecter ms_handle_connect 0x7f29281e8450 2024-03-27T21:40:21.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.640+0000 7f292659c640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f29281e8450 0x7f29281ea840 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:21.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.641+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f29281ed610 con 0x7f29281e8450 2024-03-27T21:40:21.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.641+0000 7f291f7fe640 10 client.4611.objecter ms_dispatch 0x7f2928000b90 command_reply(tid 9: 0 ) v1 2024-03-27T21:40:21.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.641+0000 7f291f7fe640 10 client.4611.objecter _finish_command 9 = system:0 2024-03-27T21:40:21.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.641+0000 7f291f7fe640 20 client.4611.objecter put_session s=0x7f29281e7dd0 osd=4 4 2024-03-27T21:40:21.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.641+0000 7f291f7fe640 15 client.4611.objecter _session_command_op_remove 4 9 2024-03-27T21:40:21.634 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:21.648+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:21.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.665+0000 7f291cd99640 10 client.4611.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:21.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.665+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign -1 10 2024-03-27T21:40:21.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.665+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281e7dd0 osd=4 3 2024-03-27T21:40:21.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.665+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281e7dd0 osd=4 3 2024-03-27T21:40:21.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.665+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281e7dd0 osd=4 3 2024-03-27T21:40:21.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.665+0000 7f291cd99640 15 client.4611.objecter _session_command_op_remove -1 10 2024-03-27T21:40:21.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.665+0000 7f291cd99640 20 client.4611.objecter get_session s=0x7f29281e7dd0 osd=4 3 2024-03-27T21:40:21.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.665+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign 4 10 2024-03-27T21:40:21.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.665+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281e7dd0 osd=4 4 2024-03-27T21:40:21.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.665+0000 7f291cd99640 10 client.4611.objecter _send_command 10 2024-03-27T21:40:21.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.665+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f29281ead80 con 0x7f29281e8450 2024-03-27T21:40:21.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.667+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f29281ead80 con 0x7f29281e8450 2024-03-27T21:40:21.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.667+0000 7f291f7fe640 10 client.4611.objecter ms_dispatch 0x7f2928000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:21.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.667+0000 7f291f7fe640 10 client.4611.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:21.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.667+0000 7f291f7fe640 20 client.4611.objecter put_session s=0x7f29281e7dd0 osd=4 4 2024-03-27T21:40:21.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.667+0000 7f291f7fe640 15 client.4611.objecter _session_command_op_remove 4 10 2024-03-27T21:40:21.654 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:21.656 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:21.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.667+0000 7f291cd99640 10 client.4611.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:21.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.667+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign -1 11 2024-03-27T21:40:21.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.667+0000 7f291cd99640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f29281eb400 0x7f29281ed7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:21.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.667+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281ead80 osd=5 3 2024-03-27T21:40:21.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.667+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281ead80 osd=5 3 2024-03-27T21:40:21.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.667+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281ead80 osd=5 3 2024-03-27T21:40:21.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.667+0000 7f291cd99640 15 client.4611.objecter _session_command_op_remove -1 11 2024-03-27T21:40:21.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.667+0000 7f291cd99640 20 client.4611.objecter get_session s=0x7f29281ead80 osd=5 3 2024-03-27T21:40:21.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.667+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign 5 11 2024-03-27T21:40:21.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.667+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281ead80 osd=5 4 2024-03-27T21:40:21.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.667+0000 7f291cd99640 10 client.4611.objecter _send_command 11 2024-03-27T21:40:21.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.667+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f29281f05c0 con 0x7f29281eb400 2024-03-27T21:40:21.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.668+0000 7f291dd9b640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f29281eb400 0x7f29281ed7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:21.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.668+0000 7f291dd9b640 10 monclient: get_auth_request con 0x7f29281eb400 auth_method 0 2024-03-27T21:40:21.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.668+0000 7f291f7fe640 10 client.4611.objecter ms_handle_connect 0x7f29281eb400 2024-03-27T21:40:21.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.668+0000 7f291dd9b640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f29281eb400 0x7f29281ed7f0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:21.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.669+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f29281f05c0 con 0x7f29281eb400 2024-03-27T21:40:21.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.669+0000 7f291f7fe640 10 client.4611.objecter ms_dispatch 0x7f2928000b90 command_reply(tid 11: 0 ) v1 2024-03-27T21:40:21.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.669+0000 7f291f7fe640 10 client.4611.objecter _finish_command 11 = system:0 2024-03-27T21:40:21.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.669+0000 7f291f7fe640 20 client.4611.objecter put_session s=0x7f29281ead80 osd=5 4 2024-03-27T21:40:21.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.669+0000 7f291f7fe640 15 client.4611.objecter _session_command_op_remove 5 11 2024-03-27T21:40:21.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.690+0000 7f291cd99640 10 client.4611.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:21.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.690+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign -1 12 2024-03-27T21:40:21.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.690+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281ead80 osd=5 3 2024-03-27T21:40:21.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.690+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281ead80 osd=5 3 2024-03-27T21:40:21.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.690+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281ead80 osd=5 3 2024-03-27T21:40:21.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.690+0000 7f291cd99640 15 client.4611.objecter _session_command_op_remove -1 12 2024-03-27T21:40:21.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.690+0000 7f291cd99640 20 client.4611.objecter get_session s=0x7f29281ead80 osd=5 3 2024-03-27T21:40:21.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.690+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign 5 12 2024-03-27T21:40:21.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.690+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281ead80 osd=5 4 2024-03-27T21:40:21.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.690+0000 7f291cd99640 10 client.4611.objecter _send_command 12 2024-03-27T21:40:21.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.690+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f29281edd30 con 0x7f29281eb400 2024-03-27T21:40:21.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.692+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f29281edd30 con 0x7f29281eb400 2024-03-27T21:40:21.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.692+0000 7f291f7fe640 10 client.4611.objecter ms_dispatch 0x7f2928000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:21.680 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:21.681 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:21.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.692+0000 7f291f7fe640 10 client.4611.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:21.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.692+0000 7f291f7fe640 20 client.4611.objecter put_session s=0x7f29281ead80 osd=5 4 2024-03-27T21:40:21.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.692+0000 7f291f7fe640 15 client.4611.objecter _session_command_op_remove 5 12 2024-03-27T21:40:21.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.693+0000 7f291cd99640 10 client.4611.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:21.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.693+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign -1 13 2024-03-27T21:40:21.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.693+0000 7f291cd99640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f29281ee3b0 0x7f29281f07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:21.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.693+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281edd30 osd=6 3 2024-03-27T21:40:21.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.693+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281edd30 osd=6 3 2024-03-27T21:40:21.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.693+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281edd30 osd=6 3 2024-03-27T21:40:21.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.693+0000 7f291cd99640 15 client.4611.objecter _session_command_op_remove -1 13 2024-03-27T21:40:21.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.693+0000 7f291cd99640 20 client.4611.objecter get_session s=0x7f29281edd30 osd=6 3 2024-03-27T21:40:21.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.693+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign 6 13 2024-03-27T21:40:21.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.693+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281edd30 osd=6 4 2024-03-27T21:40:21.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.693+0000 7f291cd99640 10 client.4611.objecter _send_command 13 2024-03-27T21:40:21.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.693+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f29281f3710 con 0x7f29281ee3b0 2024-03-27T21:40:21.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.693+0000 7f2926d9d640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f29281ee3b0 0x7f29281f07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:21.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.693+0000 7f2926d9d640 10 monclient: get_auth_request con 0x7f29281ee3b0 auth_method 0 2024-03-27T21:40:21.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.693+0000 7f291f7fe640 10 client.4611.objecter ms_handle_connect 0x7f29281ee3b0 2024-03-27T21:40:21.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.693+0000 7f2926d9d640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f29281ee3b0 0x7f29281f07a0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:21.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.694+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f29281f3710 con 0x7f29281ee3b0 2024-03-27T21:40:21.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.694+0000 7f291f7fe640 10 client.4611.objecter ms_dispatch 0x7f2928000b90 command_reply(tid 13: 0 ) v1 2024-03-27T21:40:21.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.694+0000 7f291f7fe640 10 client.4611.objecter _finish_command 13 = system:0 2024-03-27T21:40:21.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.694+0000 7f291f7fe640 20 client.4611.objecter put_session s=0x7f29281edd30 osd=6 4 2024-03-27T21:40:21.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.694+0000 7f291f7fe640 15 client.4611.objecter _session_command_op_remove 6 13 2024-03-27T21:40:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.722+0000 7f291cd99640 10 client.4611.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.722+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign -1 14 2024-03-27T21:40:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.722+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281edd30 osd=6 3 2024-03-27T21:40:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.722+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281edd30 osd=6 3 2024-03-27T21:40:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.722+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281edd30 osd=6 3 2024-03-27T21:40:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.722+0000 7f291cd99640 15 client.4611.objecter _session_command_op_remove -1 14 2024-03-27T21:40:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.722+0000 7f291cd99640 20 client.4611.objecter get_session s=0x7f29281edd30 osd=6 3 2024-03-27T21:40:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.722+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign 6 14 2024-03-27T21:40:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.722+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281edd30 osd=6 4 2024-03-27T21:40:21.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.722+0000 7f291cd99640 10 client.4611.objecter _send_command 14 2024-03-27T21:40:21.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.722+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f29281f3710 con 0x7f29281ee3b0 2024-03-27T21:40:21.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.724+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f29281f3710 con 0x7f29281ee3b0 2024-03-27T21:40:21.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.724+0000 7f291f7fe640 10 client.4611.objecter ms_dispatch 0x7f2928000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:21.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.724+0000 7f291f7fe640 10 client.4611.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:21.711 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:21.713 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:21.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.724+0000 7f291f7fe640 20 client.4611.objecter put_session s=0x7f29281edd30 osd=6 4 2024-03-27T21:40:21.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.724+0000 7f291f7fe640 15 client.4611.objecter _session_command_op_remove 6 14 2024-03-27T21:40:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.724+0000 7f291cd99640 10 client.4611.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.724+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign -1 15 2024-03-27T21:40:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.724+0000 7f291cd99640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f29281f12e0 0x7f29281f1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.725+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281f3710 osd=7 3 2024-03-27T21:40:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.725+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281f3710 osd=7 3 2024-03-27T21:40:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.725+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281f3710 osd=7 3 2024-03-27T21:40:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.725+0000 7f291cd99640 15 client.4611.objecter _session_command_op_remove -1 15 2024-03-27T21:40:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.725+0000 7f291cd99640 20 client.4611.objecter get_session s=0x7f29281f3710 osd=7 3 2024-03-27T21:40:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.725+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign 7 15 2024-03-27T21:40:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.725+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281f3710 osd=7 4 2024-03-27T21:40:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.725+0000 7f291cd99640 10 client.4611.objecter _send_command 15 2024-03-27T21:40:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.725+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f29281f6a90 con 0x7f29281f12e0 2024-03-27T21:40:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.725+0000 7f292659c640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f29281f12e0 0x7f29281f1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:21.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.725+0000 7f292659c640 10 monclient: get_auth_request con 0x7f29281f12e0 auth_method 0 2024-03-27T21:40:21.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.725+0000 7f291f7fe640 10 client.4611.objecter ms_handle_connect 0x7f29281f12e0 2024-03-27T21:40:21.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.725+0000 7f292659c640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f29281f12e0 0x7f29281f1740 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:21.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.726+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f29281f6a90 con 0x7f29281f12e0 2024-03-27T21:40:21.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.726+0000 7f291f7fe640 10 client.4611.objecter ms_dispatch 0x7f2928000b90 command_reply(tid 15: 0 ) v1 2024-03-27T21:40:21.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.726+0000 7f291f7fe640 10 client.4611.objecter _finish_command 15 = system:0 2024-03-27T21:40:21.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.726+0000 7f291f7fe640 20 client.4611.objecter put_session s=0x7f29281f3710 osd=7 4 2024-03-27T21:40:21.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.726+0000 7f291f7fe640 15 client.4611.objecter _session_command_op_remove 7 15 2024-03-27T21:40:21.735 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:21.749+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:21.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.754+0000 7f291cd99640 10 client.4611.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:21.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.754+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign -1 16 2024-03-27T21:40:21.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.754+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281f3710 osd=7 3 2024-03-27T21:40:21.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.754+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281f3710 osd=7 3 2024-03-27T21:40:21.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.754+0000 7f291cd99640 20 client.4611.objecter _get_session s=0x7f29281f3710 osd=7 3 2024-03-27T21:40:21.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.754+0000 7f291cd99640 15 client.4611.objecter _session_command_op_remove -1 16 2024-03-27T21:40:21.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.754+0000 7f291cd99640 20 client.4611.objecter get_session s=0x7f29281f3710 osd=7 3 2024-03-27T21:40:21.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.754+0000 7f291cd99640 15 client.4611.objecter _session_command_op_assign 7 16 2024-03-27T21:40:21.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.754+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281f3710 osd=7 4 2024-03-27T21:40:21.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.754+0000 7f291cd99640 10 client.4611.objecter _send_command 16 2024-03-27T21:40:21.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.754+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f29281f1d90 con 0x7f29281f12e0 2024-03-27T21:40:21.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.756+0000 7f291f7fe640 1 -- 172.21.15.60:0/2310078750 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f29281f1d90 con 0x7f29281f12e0 2024-03-27T21:40:21.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.756+0000 7f291f7fe640 10 client.4611.objecter ms_dispatch 0x7f2928000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:21.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.756+0000 7f291f7fe640 10 client.4611.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:21.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.756+0000 7f291f7fe640 20 client.4611.objecter put_session s=0x7f29281f3710 osd=7 4 2024-03-27T21:40:21.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.756+0000 7f291f7fe640 15 client.4611.objecter _session_command_op_remove 7 16 2024-03-27T21:40:21.745 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:21.749 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:21.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f291cd99640 10 librados: watch_flush enter 2024-03-27T21:40:21.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f291cd99640 10 librados: watch_flush exit 2024-03-27T21:40:21.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f291cd99640 10 client.4611.objecter close_session for osd.0 2024-03-27T21:40:21.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f292813b480 msgr2=0x7f29281d57a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:21.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f291cd99640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f292813b480 0x7f29281d57a0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:21.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281d1fc0 osd=0 1 2024-03-27T21:40:21.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f291cd99640 10 client.4611.objecter close_session for osd.1 2024-03-27T21:40:21.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f292813e740 msgr2=0x7f29281e05e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:21.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f291cd99640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f292813e740 0x7f29281e05e0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:21.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281399a0 osd=1 1 2024-03-27T21:40:21.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f291cd99640 10 client.4611.objecter close_session for osd.2 2024-03-27T21:40:21.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f292813de20 msgr2=0x7f29281e3e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:21.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f291cd99640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f292813de20 0x7f29281e3e80 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:21.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f2926d9d640 1 -- 172.21.15.60:0/2310078750 reap_dead start 2024-03-27T21:40:21.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f292813dcb0 osd=2 1 2024-03-27T21:40:21.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f291cd99640 10 client.4611.objecter close_session for osd.3 2024-03-27T21:40:21.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f2928065e40 msgr2=0x7f29281e76c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:21.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f291cd99640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f2928065e40 0x7f29281e76c0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:21.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f2928065cd0 osd=3 1 2024-03-27T21:40:21.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f291cd99640 10 client.4611.objecter close_session for osd.4 2024-03-27T21:40:21.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f29281e8450 msgr2=0x7f29281ea840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:21.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.757+0000 7f291cd99640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f29281e8450 0x7f29281ea840 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:21.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281e7dd0 osd=4 1 2024-03-27T21:40:21.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 10 client.4611.objecter close_session for osd.5 2024-03-27T21:40:21.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f29281eb400 msgr2=0x7f29281ed7f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:21.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f29281eb400 0x7f29281ed7f0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:21.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281ead80 osd=5 1 2024-03-27T21:40:21.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 10 client.4611.objecter close_session for osd.6 2024-03-27T21:40:21.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f29281ee3b0 msgr2=0x7f29281f07a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:21.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f29281ee3b0 0x7f29281f07a0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:21.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281edd30 osd=6 1 2024-03-27T21:40:21.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 10 client.4611.objecter close_session for osd.7 2024-03-27T21:40:21.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f29281f12e0 msgr2=0x7f29281f1740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:21.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f29281f12e0 0x7f29281f1740 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:21.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f2926d9d640 1 -- 172.21.15.60:0/2310078750 reap_dead start 2024-03-27T21:40:21.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 20 client.4611.objecter put_session s=0x7f29281f3710 osd=7 1 2024-03-27T21:40:21.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 20 client.4611.objecter shutdown clearing up homeless session... 2024-03-27T21:40:21.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 10 client.4611.objecter successfully canceled tick 2024-03-27T21:40:21.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f290806c5c0 msgr2=0x7f290806ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:21.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f290806c5c0 0x7f290806ea80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2918007fd0 tx=0x7f291800c040 comp rx=0 tx=0).stop 2024-03-27T21:40:21.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 10 monclient: shutdown 2024-03-27T21:40:21.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:21.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29280abeb0 msgr2=0x7f29280ac2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:21.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29280abeb0 0x7f29280ac2f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f291000fa50 tx=0x7f291000ff10 comp rx=0 tx=0).stop 2024-03-27T21:40:21.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 shutdown_connections 2024-03-27T21:40:21.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f290806c5c0 0x7f290806ea80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:21.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 1 --2- 172.21.15.60:0/2310078750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29280abeb0 0x7f29280ac2f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:21.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 >> 172.21.15.60:0/2310078750 conn(0x7f29280608e0 msgr2=0x7f2928141b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:21.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 shutdown_connections 2024-03-27T21:40:21.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.758+0000 7f291cd99640 1 -- 172.21.15.60:0/2310078750 wait complete. 2024-03-27T21:40:21.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:21.759+0000 7f291cd99640 1 librados: shutdown 2024-03-27T21:40:21.754 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:21.754 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:21.763 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:40:21.849 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:21.862+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:21.922 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:21.923 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:21.933 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:40:21.973 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:21.987+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:22.039 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:22.053+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:22.084 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:22.084 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:22.094 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:40:22.140 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:22.154+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:22.203 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:22.203 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:22.212 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:40:22.253 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:22.267+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:22.329 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:22.330 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:22.339 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:40:22.382 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:22.396+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:22.416 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-27T21:40:22.445 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:22.445 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:22.454 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:40:22.460 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:22.474+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:22.543 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:22.557+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:22.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.569+0000 7f6f648b4640 10 monclient: get_monmap_and_config 2024-03-27T21:40:22.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.569+0000 7f6f648b4640 10 monclient: build_initial_monmap 2024-03-27T21:40:22.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.569+0000 7f6f648b4640 10 monclient: monmap: 2024-03-27T21:40:22.556 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:22.557 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:22.557 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:22.570748+0000 2024-03-27T21:40:22.557 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:22.570748+0000 2024-03-27T21:40:22.557 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:22.557 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:22.557 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:22.557 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:22.557 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:22.557 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:22.557 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:22.557 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:22.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.570+0000 7f6f648b4640 1 Processor -- start 2024-03-27T21:40:22.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.570+0000 7f6f648b4640 1 -- start start 2024-03-27T21:40:22.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.570+0000 7f6f648b4640 10 monclient: init 2024-03-27T21:40:22.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.571+0000 7f6f648b4640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:22.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.571+0000 7f6f648b4640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-03-27T21:40:22.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.571+0000 7f6f648b4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6f600b40b0 0x7f6f600a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:22.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.571+0000 7f6f648b4640 10 monclient(hunting): picked mon.noname-c con 0x7f6f600b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:22.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.571+0000 7f6f648b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6f6013b020 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:22.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.571+0000 7f6f648b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6f6013b480 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:22.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.571+0000 7f6f648b4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f6013b8d0 con 0x7f6f6013b480 2024-03-27T21:40:22.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.571+0000 7f6f648b4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f600b3830 con 0x7f6f6013b020 2024-03-27T21:40:22.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.571+0000 7f6f648b4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:22.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.571+0000 7f6f648b4640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f600b4490 con 0x7f6f600b40b0 2024-03-27T21:40:22.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.571+0000 7f6f648b4640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:22.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.571+0000 7f6f648b4640 10 monclient(hunting): authenticate will time out at 863.792678s 2024-03-27T21:40:22.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.571+0000 7f6f5ed9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6f6013b480 0x7f6f6014bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58140/0 (socket says 172.21.15.60:58140) 2024-03-27T21:40:22.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.571+0000 7f6f5ed9d640 1 -- 172.21.15.60:0/2915834277 learned_addr learned my addr 172.21.15.60:0/2915834277 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:40:22.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.571+0000 7f6f5e59c640 1 --2- 172.21.15.60:0/2915834277 >> v2:172.21.15.191:3300/0 conn(0x7f6f600b40b0 0x7f6f600a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:22.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.571+0000 7f6f5e59c640 10 monclient(hunting): get_auth_request con 0x7f6f600b40b0 auth_method 0 2024-03-27T21:40:22.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.571+0000 7f6f5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:22.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.571+0000 7f6f5e59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:22.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.571+0000 7f6f5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:22.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.572+0000 7f6f5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:22.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.572+0000 7f6f5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:22.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.572+0000 7f6f5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:22.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.572+0000 7f6f5e59c640 10 monclient(hunting): handle_auth_done global_id 4573 payload 274 2024-03-27T21:40:22.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.572+0000 7f6f5e59c640 1 -- 172.21.15.60:0/2915834277 >> v1:172.21.15.191:6789/0 conn(0x7f6f6013b020 legacy=0x7f6f601416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:22.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.572+0000 7f6f5d59a640 1 -- 172.21.15.60:0/2915834277 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446892712 0 0) 0x7f6f600b3830 con 0x7f6f6013b020 2024-03-27T21:40:22.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.572+0000 7f6f5e59c640 1 -- 172.21.15.60:0/2915834277 >> v1:172.21.15.60:6789/0 conn(0x7f6f6013b480 legacy=0x7f6f6014bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:22.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.572+0000 7f6f5e59c640 10 monclient: _finish_hunting 0 2024-03-27T21:40:22.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.572+0000 7f6f5e59c640 1 monclient: found mon.noname-c 2024-03-27T21:40:22.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.572+0000 7f6f5e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:22.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.572+0000 7f6f5e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:40:22.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.572+0000 7f6f5e59c640 1 -- 172.21.15.60:0/2915834277 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f600aa520 con 0x7f6f600b40b0 2024-03-27T21:40:22.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.572+0000 7f6f5d59a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:40:22.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f5e59c640 1 --2- 172.21.15.60:0/2915834277 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f600b40b0 0x7f6f600a8b90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6f500097a0 tx=0x7f6f5002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c225eef1886195f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:22.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f5d59a640 1 -- 172.21.15.60:0/2915834277 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f50031830 con 0x7f6f600b40b0 2024-03-27T21:40:22.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f5d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:22.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f5d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:22.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f5d59a640 10 monclient: dump: 2024-03-27T21:40:22.566 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:22.566 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:22.567 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:22.567 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:22.567 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:22.567 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:22.567 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:22.567 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:22.567 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:22.567 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:22.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f5d59a640 10 monclient: _finish_auth 0 2024-03-27T21:40:22.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f5d59a640 10 monclient: _check_auth_tickets 2024-03-27T21:40:22.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f5d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:22.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f648b4640 5 monclient: authenticate success, global_id 4573 2024-03-27T21:40:22.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f5d59a640 1 -- 172.21.15.60:0/2915834277 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f50031990 con 0x7f6f600b40b0 2024-03-27T21:40:22.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f5d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:22.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f648b4640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:22.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f5d59a640 1 -- 172.21.15.60:0/2915834277 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f50031d20 con 0x7f6f600b40b0 2024-03-27T21:40:22.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f648b4640 4 set_mon_vals no callback set 2024-03-27T21:40:22.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f5d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:22.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f5d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:40:22.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f5d59a640 10 monclient: dump: 2024-03-27T21:40:22.568 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:22.568 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:22.568 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:22.568 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:22.568 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:22.569 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:22.569 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:22.569 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:22.569 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:22.569 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:22.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f648b4640 10 monclient: shutdown 2024-03-27T21:40:22.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f648b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:22.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f648b4640 1 -- 172.21.15.60:0/2915834277 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f600b40b0 msgr2=0x7f6f600a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:22.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f648b4640 1 --2- 172.21.15.60:0/2915834277 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f600b40b0 0x7f6f600a8b90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6f500097a0 tx=0x7f6f5002b8d0 comp rx=0 tx=0).stop 2024-03-27T21:40:22.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f648b4640 1 -- 172.21.15.60:0/2915834277 shutdown_connections 2024-03-27T21:40:22.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f648b4640 1 --2- 172.21.15.60:0/2915834277 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f600b40b0 0x7f6f600a8b90 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:22.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f648b4640 1 -- 172.21.15.60:0/2915834277 >> 172.21.15.60:0/2915834277 conn(0x7f6f600608e0 msgr2=0x7f6f60060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:22.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f648b4640 1 -- 172.21.15.60:0/2915834277 shutdown_connections 2024-03-27T21:40:22.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.573+0000 7f6f648b4640 1 -- 172.21.15.60:0/2915834277 wait complete. 2024-03-27T21:40:22.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.574+0000 7f6f648b4640 10 monclient: build_initial_monmap 2024-03-27T21:40:22.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.574+0000 7f6f648b4640 10 monclient: monmap: 2024-03-27T21:40:22.570 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:22.570 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:22.570 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:22.570 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:22.570 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:22.570 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:22.571 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:22.571 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:22.571 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:22.571 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:22.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.575+0000 7f6f648b4640 1 librados: starting msgr at 2024-03-27T21:40:22.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.575+0000 7f6f648b4640 1 librados: starting objecter 2024-03-27T21:40:22.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.575+0000 7f6f648b4640 1 Processor -- start 2024-03-27T21:40:22.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.575+0000 7f6f648b4640 1 -- start start 2024-03-27T21:40:22.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.575+0000 7f6f648b4640 1 librados: setting wanted keys 2024-03-27T21:40:22.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.575+0000 7f6f648b4640 1 librados: calling monclient init 2024-03-27T21:40:22.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.575+0000 7f6f648b4640 10 monclient: init 2024-03-27T21:40:22.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f648b4640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:22.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f648b4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:40:22.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f648b4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f600b40b0 0x7f6f600af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:22.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f648b4640 10 monclient(hunting): picked mon.noname-b con 0x7f6f600b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:22.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f648b4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f6013b020 0x7f6f600b0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:22.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f648b4640 10 monclient(hunting): picked mon.noname-a con 0x7f6f6013b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:22.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f648b4640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6f6013b480 0x7f6f601d69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:22.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5dd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f6013b020 0x7f6f600b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:22.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5dd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f6013b020 0x7f6f600b0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40310/0 (socket says 172.21.15.60:40310) 2024-03-27T21:40:22.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5dd9b640 1 -- 172.21.15.60:0/1635163315 learned_addr learned my addr 172.21.15.60:0/1635163315 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:22.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f648b4640 10 monclient(hunting): picked mon.noname-c con 0x7f6f6013b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:22.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f648b4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:22.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f648b4640 1 -- 172.21.15.60:0/1635163315 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f600aeb40 con 0x7f6f6013b020 2024-03-27T21:40:22.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f648b4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:22.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f648b4640 1 -- 172.21.15.60:0/1635163315 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f601d64f0 con 0x7f6f600b40b0 2024-03-27T21:40:22.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f648b4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:22.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f648b4640 1 -- 172.21.15.60:0/1635163315 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f600acc10 con 0x7f6f6013b480 2024-03-27T21:40:22.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5e59c640 1 --2- 172.21.15.60:0/1635163315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f600b40b0 0x7f6f600af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:22.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f648b4640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:22.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f648b4640 10 monclient(hunting): authenticate will time out at 863.797619s 2024-03-27T21:40:22.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5dd9b640 10 monclient(hunting): get_auth_request con 0x7f6f6013b020 auth_method 0 2024-03-27T21:40:22.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:22.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:22.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:22.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5e59c640 10 monclient(hunting): get_auth_request con 0x7f6f600b40b0 auth_method 0 2024-03-27T21:40:22.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:22.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5e59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:22.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:22.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5ed9d640 1 --2- 172.21.15.60:0/1635163315 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6f6013b480 0x7f6f601d69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:22.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5ed9d640 10 monclient(hunting): get_auth_request con 0x7f6f6013b480 auth_method 0 2024-03-27T21:40:22.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:22.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5ed9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:22.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:22.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:22.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:22.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:22.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:22.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:22.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:22.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:22.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:22.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:22.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.576+0000 7f6f5e59c640 10 monclient(hunting): handle_auth_done global_id 4579 payload 824 2024-03-27T21:40:22.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f5e59c640 1 -- 172.21.15.60:0/1635163315 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6f6013b480 msgr2=0x7f6f601d69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:22.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f5e59c640 1 --2- 172.21.15.60:0/1635163315 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6f6013b480 0x7f6f601d69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:22.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f5e59c640 1 -- 172.21.15.60:0/1635163315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f6013b020 msgr2=0x7f6f600b0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:22.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f5e59c640 1 --2- 172.21.15.60:0/1635163315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f6013b020 0x7f6f600b0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:22.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f5e59c640 10 monclient: _finish_hunting 0 2024-03-27T21:40:22.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f5e59c640 1 monclient: found mon.noname-b 2024-03-27T21:40:22.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f5e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:22.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f5e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:40:22.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f5e59c640 1 -- 172.21.15.60:0/1635163315 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f50009400 con 0x7f6f600b40b0 2024-03-27T21:40:22.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f5dd9b640 1 --2- 172.21.15.60:0/1635163315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f6013b020 0x7f6f600b0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:22.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f5e59c640 1 --2- 172.21.15.60:0/1635163315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f600b40b0 0x7f6f600af5c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6f50033000 tx=0x7f6f500046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:22.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f4effd640 10 client.?.objecter ms_handle_connect 0x7f6f600b40b0 2024-03-27T21:40:22.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f4effd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:22.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f4effd640 1 -- 172.21.15.60:0/1635163315 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5003ba70 con 0x7f6f600b40b0 2024-03-27T21:40:22.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:22.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f4effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:22.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f4effd640 10 monclient: dump: 2024-03-27T21:40:22.586 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:22.586 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:22.587 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:22.587 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:22.587 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:22.587 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:22.587 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:22.587 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:22.587 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:22.587 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:22.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f4effd640 10 monclient: _finish_auth 0 2024-03-27T21:40:22.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f4effd640 10 monclient: _check_auth_tickets 2024-03-27T21:40:22.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:22.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f648b4640 5 monclient: authenticate success, global_id 4579 2024-03-27T21:40:22.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f4effd640 1 -- 172.21.15.60:0/1635163315 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5003bc10 con 0x7f6f600b40b0 2024-03-27T21:40:22.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:22.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f4effd640 1 -- 172.21.15.60:0/1635163315 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f500453f0 con 0x7f6f600b40b0 2024-03-27T21:40:22.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:22.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:40:22.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f5cd99640 4 set_mon_vals no callback set 2024-03-27T21:40:22.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f4effd640 10 monclient: dump: 2024-03-27T21:40:22.588 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:22.588 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:22.588 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:22.589 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:22.589 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:22.589 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:22.589 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:22.589 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:22.589 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:22.589 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:22.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f648b4640 10 monclient: _renew_subs 2024-03-27T21:40:22.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f648b4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:22.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f648b4640 1 -- 172.21.15.60:0/1635163315 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f600b08c0 con 0x7f6f600b40b0 2024-03-27T21:40:22.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f648b4640 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:22.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f648b4640 10 monclient: _renew_subs 2024-03-27T21:40:22.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f648b4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:22.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f648b4640 1 -- 172.21.15.60:0/1635163315 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f601d6f10 con 0x7f6f600b40b0 2024-03-27T21:40:22.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.577+0000 7f6f648b4640 1 librados: init done 2024-03-27T21:40:22.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.578+0000 7f6f648b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:22.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.578+0000 7f6f648b4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:22.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.578+0000 7f6f648b4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:22.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.578+0000 7f6f648b4640 1 -- 172.21.15.60:0/1635163315 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f600b0ed0 con 0x7f6f600b40b0 2024-03-27T21:40:22.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.578+0000 7f6f4effd640 1 -- 172.21.15.60:0/1635163315 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6f5004e450 con 0x7f6f600b40b0 2024-03-27T21:40:22.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.579+0000 7f6f4effd640 1 --2- 172.21.15.60:0/1635163315 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6f4006c5c0 0x7f6f4006ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:22.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.579+0000 7f6f4effd640 1 -- 172.21.15.60:0/1635163315 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f6f500ab1b0 con 0x7f6f600b40b0 2024-03-27T21:40:22.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.579+0000 7f6f4effd640 10 client.4579.objecter ms_dispatch 0x7f6f60000b90 osd_map(34..34 src has 1..34) v4 2024-03-27T21:40:22.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.579+0000 7f6f4effd640 3 client.4579.objecter handle_osd_map got epochs [34,34] > 0 2024-03-27T21:40:22.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.579+0000 7f6f4effd640 3 client.4579.objecter handle_osd_map decoding full epoch 34 2024-03-27T21:40:22.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.579+0000 7f6f4effd640 20 client.4579.objecter dump_active .. 0 homeless 2024-03-27T21:40:22.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.579+0000 7f6f5dd9b640 1 --2- 172.21.15.60:0/1635163315 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6f4006c5c0 0x7f6f4006ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:22.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.579+0000 7f6f5dd9b640 10 monclient: get_auth_request con 0x7f6f4006c5c0 auth_method 0 2024-03-27T21:40:22.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.580+0000 7f6f5dd9b640 1 --2- 172.21.15.60:0/1635163315 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6f4006c5c0 0x7f6f4006ea80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6f6011f370 tx=0x7f6f54009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:22.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.580+0000 7f6f4effd640 10 client.4579.objecter ms_handle_connect 0x7f6f4006c5c0 2024-03-27T21:40:22.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.584+0000 7f6f4effd640 1 -- 172.21.15.60:0/1635163315 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6f500766a0 con 0x7f6f600b40b0 2024-03-27T21:40:22.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.584+0000 7f6f4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:22.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.584+0000 7f6f4effd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:22.600 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:22.600 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:22.610 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:40:22.644 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:22.658+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:22.745 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:22.759+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:22.757 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:22.758 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:22.764 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:40:22.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.840+0000 7f6f648b4640 10 start_mon_command cmd=[{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:40:22.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.840+0000 7f6f648b4640 10 monclient: _send_command 2 [{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:40:22.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.840+0000 7f6f648b4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:22.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:22.840+0000 7f6f648b4640 1 -- 172.21.15.60:0/1635163315 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f6f60069470 con 0x7f6f600b40b0 2024-03-27T21:40:22.847 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:22.860+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:22.909 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:22.909 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:22.918 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:40:22.948 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:22.962+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:23.023 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:23.023 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:23.032 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:40:23.075 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:23.089+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:23.138 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:23.138 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:23.146 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:40:23.150 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:23.164+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:23.250 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:23.264+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:23.292 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:23.292 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:23.301 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:40:23.351 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:23.365+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:23.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.417+0000 7f6f4effd640 1 -- 172.21.15.60:0/1635163315 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v35) v1 ==== 91+0+0 (secure 0 0 0) 0x7f6f50079cc0 con 0x7f6f600b40b0 2024-03-27T21:40:23.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.417+0000 7f6f4effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:40:23.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.417+0000 7f6f4effd640 10 monclient: _finish_command 2 = system:0 noscrub is unset 2024-03-27T21:40:23.404 INFO:teuthology.orchestra.run.smithi060.stderr:noscrub is unset 2024-03-27T21:40:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.419+0000 7f6f648b4640 10 librados: watch_flush enter 2024-03-27T21:40:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.419+0000 7f6f648b4640 10 librados: watch_flush exit 2024-03-27T21:40:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.419+0000 7f6f648b4640 20 client.4579.objecter shutdown clearing up homeless session... 2024-03-27T21:40:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.419+0000 7f6f648b4640 10 client.4579.objecter successfully canceled tick 2024-03-27T21:40:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.419+0000 7f6f648b4640 1 -- 172.21.15.60:0/1635163315 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6f4006c5c0 msgr2=0x7f6f4006ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.419+0000 7f6f648b4640 1 --2- 172.21.15.60:0/1635163315 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6f4006c5c0 0x7f6f4006ea80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6f6011f370 tx=0x7f6f54009040 comp rx=0 tx=0).stop 2024-03-27T21:40:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.419+0000 7f6f648b4640 10 monclient: shutdown 2024-03-27T21:40:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.419+0000 7f6f648b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.419+0000 7f6f648b4640 1 -- 172.21.15.60:0/1635163315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f600b40b0 msgr2=0x7f6f600af5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.419+0000 7f6f648b4640 1 --2- 172.21.15.60:0/1635163315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f600b40b0 0x7f6f600af5c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6f50033000 tx=0x7f6f500046c0 comp rx=0 tx=0).stop 2024-03-27T21:40:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.419+0000 7f6f648b4640 1 -- 172.21.15.60:0/1635163315 shutdown_connections 2024-03-27T21:40:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.419+0000 7f6f648b4640 1 --2- 172.21.15.60:0/1635163315 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6f6013b480 0x7f6f601d69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:23.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.419+0000 7f6f648b4640 1 --2- 172.21.15.60:0/1635163315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f6013b020 0x7f6f600b0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:23.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.419+0000 7f6f648b4640 1 --2- 172.21.15.60:0/1635163315 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6f4006c5c0 0x7f6f4006ea80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:23.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.419+0000 7f6f648b4640 1 --2- 172.21.15.60:0/1635163315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f600b40b0 0x7f6f600af5c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:23.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.419+0000 7f6f648b4640 1 -- 172.21.15.60:0/1635163315 >> 172.21.15.60:0/1635163315 conn(0x7f6f600608e0 msgr2=0x7f6f600b2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:23.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.419+0000 7f6f648b4640 1 -- 172.21.15.60:0/1635163315 shutdown_connections 2024-03-27T21:40:23.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.420+0000 7f6f648b4640 1 -- 172.21.15.60:0/1635163315 wait complete. 2024-03-27T21:40:23.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.420+0000 7f6f648b4640 1 librados: shutdown 2024-03-27T21:40:23.447 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:23.447 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:23.456 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:40:23.462 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:23.476+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:23.553 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:23.567+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:23.561 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:23.561 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:23.570 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:40:23.655 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:23.668+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:23.715 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:23.715 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:23.724 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:40:23.756 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:23.770+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:23.761 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:40:23.857 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:23.871+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:23.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.874+0000 7feb202ca640 10 monclient: get_monmap_and_config 2024-03-27T21:40:23.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.874+0000 7feb202ca640 10 monclient: build_initial_monmap 2024-03-27T21:40:23.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.874+0000 7feb202ca640 10 monclient: monmap: 2024-03-27T21:40:23.861 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:23.861 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:23.861 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:23.875693+0000 2024-03-27T21:40:23.861 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:23.875693+0000 2024-03-27T21:40:23.861 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:23.861 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:23.861 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:23.861 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:23.862 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:23.862 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:23.862 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:23.862 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:23.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.875+0000 7feb202ca640 1 Processor -- start 2024-03-27T21:40:23.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.875+0000 7feb202ca640 1 -- start start 2024-03-27T21:40:23.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.875+0000 7feb202ca640 10 monclient: init 2024-03-27T21:40:23.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb202ca640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:23.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb202ca640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-03-27T21:40:23.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb202ca640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb180b40b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:23.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb202ca640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7feb1813b020 0x7feb1813b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:23.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb202ca640 10 monclient(hunting): picked mon.noname-b con 0x7feb1813b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:23.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb202ca640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7feb181416f0 0x7feb18141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:23.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb202ca640 10 monclient(hunting): picked mon.noname-a con 0x7feb181416f0 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:23.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb202ca640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb180b3830 con 0x7feb180b40b0 2024-03-27T21:40:23.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb202ca640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:23.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb202ca640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb180b4490 con 0x7feb181416f0 2024-03-27T21:40:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb202ca640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb202ca640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb180aa520 con 0x7feb1813b020 2024-03-27T21:40:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb202ca640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb202ca640 10 monclient(hunting): authenticate will time out at 865.097183s 2024-03-27T21:40:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1d865640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7feb1813b020 0x7feb1813b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1e066640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7feb180b40b0 0x7feb180a8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58146/0 (socket says 172.21.15.60:58146) 2024-03-27T21:40:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1e066640 1 -- 172.21.15.60:0/2826495902 learned_addr learned my addr 172.21.15.60:0/2826495902 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:40:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1d865640 10 monclient(hunting): get_auth_request con 0x7feb1813b020 auth_method 0 2024-03-27T21:40:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1d865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1d865640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1d865640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1d865640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1d865640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1d865640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1e867640 1 --2- 172.21.15.60:0/2826495902 >> v2:172.21.15.60:3300/0 conn(0x7feb181416f0 0x7feb18141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:23.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1e867640 10 monclient(hunting): get_auth_request con 0x7feb181416f0 auth_method 0 2024-03-27T21:40:23.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1e867640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:23.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1e867640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:23.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1e867640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:23.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1e867640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:23.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1e867640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:23.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1e867640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:23.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1d865640 10 monclient(hunting): handle_auth_done global_id 4505 payload 274 2024-03-27T21:40:23.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1d865640 1 -- 172.21.15.60:0/2826495902 >> v2:172.21.15.60:3300/0 conn(0x7feb181416f0 msgr2=0x7feb18141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:23.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1d865640 1 --2- 172.21.15.60:0/2826495902 >> v2:172.21.15.60:3300/0 conn(0x7feb181416f0 0x7feb18141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:23.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1d865640 1 -- 172.21.15.60:0/2826495902 >> v1:172.21.15.60:6789/0 conn(0x7feb180b40b0 legacy=0x7feb180a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:23.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1d865640 10 monclient: _finish_hunting 0 2024-03-27T21:40:23.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1d865640 1 monclient: found mon.noname-b 2024-03-27T21:40:23.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1d865640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:23.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1d865640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:23.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.876+0000 7feb1d865640 1 -- 172.21.15.60:0/2826495902 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb180a9470 con 0x7feb1813b020 2024-03-27T21:40:23.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.877+0000 7feb1d865640 1 --2- 172.21.15.60:0/2826495902 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7feb1813b020 0x7feb1813b480 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7feb080097a0 tx=0x7feb0802b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a651bd3de77e402 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:23.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.877+0000 7feb1d064640 1 -- 172.21.15.60:0/2826495902 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb08030710 con 0x7feb1813b020 2024-03-27T21:40:23.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.877+0000 7feb1d064640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:23.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.877+0000 7feb1d064640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:23.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.877+0000 7feb1d064640 10 monclient: dump: 2024-03-27T21:40:23.868 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:23.868 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:23.868 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:23.868 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:23.868 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:23.868 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:23.868 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:23.868 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:23.868 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:23.869 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:23.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.877+0000 7feb1d064640 10 monclient: _finish_auth 0 2024-03-27T21:40:23.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.877+0000 7feb1d064640 10 monclient: _check_auth_tickets 2024-03-27T21:40:23.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.877+0000 7feb1d064640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:23.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.877+0000 7feb1d064640 1 -- 172.21.15.60:0/2826495902 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0802e050 con 0x7feb1813b020 2024-03-27T21:40:23.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.877+0000 7feb202ca640 5 monclient: authenticate success, global_id 4505 2024-03-27T21:40:23.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.877+0000 7feb202ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:23.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.877+0000 7feb1d064640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:23.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.877+0000 7feb202ca640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:23.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.877+0000 7feb1d064640 1 -- 172.21.15.60:0/2826495902 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb08030a90 con 0x7feb1813b020 2024-03-27T21:40:23.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.877+0000 7feb202ca640 4 set_mon_vals no callback set 2024-03-27T21:40:23.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.877+0000 7feb1d064640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:23.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.877+0000 7feb1d064640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:23.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.877+0000 7feb1d064640 10 monclient: dump: 2024-03-27T21:40:23.870 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:23.870 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:23.870 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:23.870 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:23.870 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:23.870 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:23.870 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:23.870 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:23.870 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:23.870 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:23.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.878+0000 7feb202ca640 10 monclient: shutdown 2024-03-27T21:40:23.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.878+0000 7feb202ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:23.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.878+0000 7feb202ca640 1 -- 172.21.15.60:0/2826495902 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7feb1813b020 msgr2=0x7feb1813b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:23.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.878+0000 7feb202ca640 1 --2- 172.21.15.60:0/2826495902 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7feb1813b020 0x7feb1813b480 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7feb080097a0 tx=0x7feb0802b710 comp rx=0 tx=0).stop 2024-03-27T21:40:23.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.878+0000 7feb202ca640 1 -- 172.21.15.60:0/2826495902 shutdown_connections 2024-03-27T21:40:23.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.878+0000 7feb202ca640 1 --2- 172.21.15.60:0/2826495902 >> v2:172.21.15.60:3300/0 conn(0x7feb181416f0 0x7feb18141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:23.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.878+0000 7feb202ca640 1 --2- 172.21.15.60:0/2826495902 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7feb1813b020 0x7feb1813b480 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:23.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.878+0000 7feb202ca640 1 -- 172.21.15.60:0/2826495902 >> 172.21.15.60:0/2826495902 conn(0x7feb180608e0 msgr2=0x7feb18060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:23.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.878+0000 7feb202ca640 1 -- 172.21.15.60:0/2826495902 shutdown_connections 2024-03-27T21:40:23.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.878+0000 7feb202ca640 1 -- 172.21.15.60:0/2826495902 wait complete. 2024-03-27T21:40:23.871 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:23.871 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:23.878 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-27T21:40:23.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.879+0000 7feb202ca640 10 monclient: build_initial_monmap 2024-03-27T21:40:23.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.879+0000 7feb202ca640 10 monclient: monmap: 2024-03-27T21:40:23.880 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:23.880 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:23.880 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:23.880 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:23.880 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:23.880 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:23.880 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:23.880 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:23.880 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:23.880 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:23.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.879+0000 7feb202ca640 1 librados: starting msgr at 2024-03-27T21:40:23.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.879+0000 7feb202ca640 1 librados: starting objecter 2024-03-27T21:40:23.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.879+0000 7feb202ca640 1 Processor -- start 2024-03-27T21:40:23.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.879+0000 7feb202ca640 1 -- start start 2024-03-27T21:40:23.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.879+0000 7feb202ca640 1 librados: setting wanted keys 2024-03-27T21:40:23.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.879+0000 7feb202ca640 1 librados: calling monclient init 2024-03-27T21:40:23.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.879+0000 7feb202ca640 10 monclient: init 2024-03-27T21:40:23.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb202ca640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:23.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb202ca640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:40:23.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb202ca640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb180b40b0 0x7feb1813a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:23.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb202ca640 10 monclient(hunting): picked mon.noname-b con 0x7feb180b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:23.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb202ca640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7feb1813b020 0x7feb1813dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:23.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb202ca640 10 monclient(hunting): picked mon.noname-c con 0x7feb1813b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:23.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb202ca640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb181416f0 0x7feb180b0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:23.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb202ca640 10 monclient(hunting): picked mon.noname-a con 0x7feb181416f0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:23.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb202ca640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:23.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb202ca640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb180abb00 con 0x7feb181416f0 2024-03-27T21:40:23.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb202ca640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:23.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb202ca640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb1813d970 con 0x7feb180b40b0 2024-03-27T21:40:23.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb202ca640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:23.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb202ca640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb1813e480 con 0x7feb1813b020 2024-03-27T21:40:23.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb202ca640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:23.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb202ca640 10 monclient(hunting): authenticate will time out at 865.101125s 2024-03-27T21:40:23.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1d865640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7feb1813b020 0x7feb1813dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:23.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1d865640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7feb1813b020 0x7feb1813dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47528/0 (socket says 172.21.15.60:47528) 2024-03-27T21:40:23.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1d865640 1 -- 172.21.15.60:0/2962606661 learned_addr learned my addr 172.21.15.60:0/2962606661 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:23.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1d865640 10 monclient(hunting): get_auth_request con 0x7feb1813b020 auth_method 0 2024-03-27T21:40:23.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1d865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:23.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1d865640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:23.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1d865640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:23.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1e867640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb181416f0 0x7feb180b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:23.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1d865640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:23.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1d865640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:23.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1e066640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb180b40b0 0x7feb1813a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:23.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1d865640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:23.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1e867640 10 monclient(hunting): get_auth_request con 0x7feb181416f0 auth_method 0 2024-03-27T21:40:23.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1e867640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:23.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1e867640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:23.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1e867640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:23.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1e066640 10 monclient(hunting): get_auth_request con 0x7feb180b40b0 auth_method 0 2024-03-27T21:40:23.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1e066640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:23.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1e066640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:23.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1e066640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:23.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1e867640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:23.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1e867640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:23.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1e867640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:23.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1d865640 10 monclient(hunting): handle_auth_done global_id 4508 payload 824 2024-03-27T21:40:23.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1d865640 1 -- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb180b40b0 msgr2=0x7feb1813a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:23.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1d865640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb180b40b0 0x7feb1813a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:23.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1d865640 1 -- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb181416f0 msgr2=0x7feb180b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:23.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1d865640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb181416f0 0x7feb180b0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:23.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1d865640 10 monclient: _finish_hunting 0 2024-03-27T21:40:23.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1d865640 1 monclient: found mon.noname-c 2024-03-27T21:40:23.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1d865640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:23.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1d865640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:40:23.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1d865640 1 -- 172.21.15.60:0/2962606661 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb08009400 con 0x7feb1813b020 2024-03-27T21:40:23.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1e066640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb180b40b0 0x7feb1813a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:40:23.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.880+0000 7feb1e867640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb181416f0 0x7feb180b0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:23.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.881+0000 7feb1d865640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7feb1813b020 0x7feb1813dc20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feb08035040 tx=0x7feb080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:23.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.881+0000 7feb0effd640 10 client.?.objecter ms_handle_connect 0x7feb1813b020 2024-03-27T21:40:23.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.881+0000 7feb0effd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:23.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.881+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0803b8d0 con 0x7feb1813b020 2024-03-27T21:40:23.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.881+0000 7feb0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:23.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.881+0000 7feb0effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:23.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.881+0000 7feb0effd640 10 monclient: dump: 2024-03-27T21:40:23.885 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:23.885 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:23.885 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:23.885 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:23.885 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:23.885 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:23.885 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:23.885 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:23.885 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:23.885 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:23.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.881+0000 7feb0effd640 10 monclient: _finish_auth 0 2024-03-27T21:40:23.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.881+0000 7feb0effd640 10 monclient: _check_auth_tickets 2024-03-27T21:40:23.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.881+0000 7feb0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:23.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.881+0000 7feb202ca640 5 monclient: authenticate success, global_id 4508 2024-03-27T21:40:23.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.881+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0803ba50 con 0x7feb1813b020 2024-03-27T21:40:23.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.881+0000 7feb0effd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:23.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.881+0000 7feb1c863640 4 set_mon_vals no callback set 2024-03-27T21:40:23.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.882+0000 7feb202ca640 10 monclient: _renew_subs 2024-03-27T21:40:23.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.882+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0803be00 con 0x7feb1813b020 2024-03-27T21:40:23.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.882+0000 7feb202ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:23.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.882+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb1813e160 con 0x7feb1813b020 2024-03-27T21:40:23.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.882+0000 7feb0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:23.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.882+0000 7feb0effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:23.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.882+0000 7feb0effd640 10 monclient: dump: 2024-03-27T21:40:23.886 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:23.887 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:23.887 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:23.887 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:23.887 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:23.887 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:23.887 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:23.887 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:23.887 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:23.887 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:23.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.882+0000 7feb202ca640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:23.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.882+0000 7feb202ca640 10 monclient: _renew_subs 2024-03-27T21:40:23.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.882+0000 7feb202ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:23.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.882+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb180b0940 con 0x7feb1813b020 2024-03-27T21:40:23.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.882+0000 7feb202ca640 1 librados: init done 2024-03-27T21:40:23.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.883+0000 7feb202ca640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:23.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.883+0000 7feb202ca640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:23.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.883+0000 7feb202ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:23.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.883+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7feb1813b480 con 0x7feb1813b020 2024-03-27T21:40:23.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.884+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7feb08039040 con 0x7feb1813b020 2024-03-27T21:40:23.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.884+0000 7feb0effd640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7feafc06c590 0x7feafc06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:23.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.884+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5242+0+0 (secure 0 0 0) 0x7feb080aa8e0 con 0x7feb1813b020 2024-03-27T21:40:23.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.884+0000 7feb0effd640 10 client.4508.objecter ms_dispatch 0x7feb18000b90 osd_map(35..35 src has 1..35) v4 2024-03-27T21:40:23.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.884+0000 7feb0effd640 3 client.4508.objecter handle_osd_map got epochs [35,35] > 0 2024-03-27T21:40:23.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.884+0000 7feb0effd640 3 client.4508.objecter handle_osd_map decoding full epoch 35 2024-03-27T21:40:23.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.884+0000 7feb1e066640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7feafc06c590 0x7feafc06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:23.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.884+0000 7feb1e066640 10 monclient: get_auth_request con 0x7feafc06c590 auth_method 0 2024-03-27T21:40:23.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb0effd640 20 client.4508.objecter dump_active .. 0 homeless 2024-03-27T21:40:23.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v35) v1 ==== 54+0+16 (secure 0 0 0) 0x7feb080af240 con 0x7feb1813b020 2024-03-27T21:40:23.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:23.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb0effd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:23.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb202ca640 10 client.4508.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:23.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign -1 1 2024-03-27T21:40:23.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb202ca640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7feb1814bd20 0x7feb181da1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:23.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb18069470 osd=0 3 2024-03-27T21:40:23.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb18069470 osd=0 3 2024-03-27T21:40:23.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb18069470 osd=0 3 2024-03-27T21:40:23.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb202ca640 15 client.4508.objecter _session_command_op_remove -1 1 2024-03-27T21:40:23.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb202ca640 20 client.4508.objecter get_session s=0x7feb18069470 osd=0 3 2024-03-27T21:40:23.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign 0 1 2024-03-27T21:40:23.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb18069470 osd=0 4 2024-03-27T21:40:23.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb202ca640 10 client.4508.objecter _send_command 1 2024-03-27T21:40:23.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb180b2e80 con 0x7feb1814bd20 2024-03-27T21:40:23.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb1e066640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7feafc06c590 0x7feafc06ea50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feb14000f60 tx=0x7feb14007ce0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:23.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb0effd640 10 client.4508.objecter ms_handle_connect 0x7feafc06c590 2024-03-27T21:40:23.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb1e867640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7feb1814bd20 0x7feb181da1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:23.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb1e867640 10 monclient: get_auth_request con 0x7feb1814bd20 auth_method 0 2024-03-27T21:40:23.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb0effd640 10 client.4508.objecter ms_handle_connect 0x7feb1814bd20 2024-03-27T21:40:23.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.885+0000 7feb1e867640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7feb1814bd20 0x7feb181da1a0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:23.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.886+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7feb180b2e80 con 0x7feb1814bd20 2024-03-27T21:40:23.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.886+0000 7feb0effd640 10 client.4508.objecter ms_dispatch 0x7feb18000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:23.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.886+0000 7feb0effd640 10 client.4508.objecter _finish_command 1 = system:0 2024-03-27T21:40:23.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.886+0000 7feb0effd640 20 client.4508.objecter put_session s=0x7feb18069470 osd=0 4 2024-03-27T21:40:23.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.886+0000 7feb0effd640 15 client.4508.objecter _session_command_op_remove 0 1 2024-03-27T21:40:23.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.901+0000 7feb202ca640 10 client.4508.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:23.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.901+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign -1 2 2024-03-27T21:40:23.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.901+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb18069470 osd=0 3 2024-03-27T21:40:23.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.901+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb18069470 osd=0 3 2024-03-27T21:40:23.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.901+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb18069470 osd=0 3 2024-03-27T21:40:23.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.901+0000 7feb202ca640 15 client.4508.objecter _session_command_op_remove -1 2 2024-03-27T21:40:23.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.901+0000 7feb202ca640 20 client.4508.objecter get_session s=0x7feb18069470 osd=0 3 2024-03-27T21:40:23.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.901+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign 0 2 2024-03-27T21:40:23.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.901+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb18069470 osd=0 4 2024-03-27T21:40:23.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.901+0000 7feb202ca640 10 client.4508.objecter _send_command 2 2024-03-27T21:40:23.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.901+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7feb180b0d70 con 0x7feb1814bd20 2024-03-27T21:40:23.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.903+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7feb180b0d70 con 0x7feb1814bd20 2024-03-27T21:40:23.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.903+0000 7feb0effd640 10 client.4508.objecter ms_dispatch 0x7feb18000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:23.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.903+0000 7feb0effd640 10 client.4508.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:23.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.903+0000 7feb0effd640 20 client.4508.objecter put_session s=0x7feb18069470 osd=0 4 2024-03-27T21:40:23.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.903+0000 7feb0effd640 15 client.4508.objecter _session_command_op_remove 0 2 2024-03-27T21:40:23.892 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:23.893 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.903+0000 7feb202ca640 10 client.4508.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.903+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign -1 3 2024-03-27T21:40:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.903+0000 7feb202ca640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7feb181d6c60 0x7feb181d7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.903+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb181da920 osd=1 3 2024-03-27T21:40:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.903+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb181da920 osd=1 3 2024-03-27T21:40:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.903+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb181da920 osd=1 3 2024-03-27T21:40:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.903+0000 7feb202ca640 15 client.4508.objecter _session_command_op_remove -1 3 2024-03-27T21:40:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.903+0000 7feb202ca640 20 client.4508.objecter get_session s=0x7feb181da920 osd=1 3 2024-03-27T21:40:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.903+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign 1 3 2024-03-27T21:40:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.903+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb181da920 osd=1 4 2024-03-27T21:40:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.904+0000 7feb202ca640 10 client.4508.objecter _send_command 3 2024-03-27T21:40:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.904+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb181d7580 con 0x7feb181d6c60 2024-03-27T21:40:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.904+0000 7feb1e066640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7feb181d6c60 0x7feb181d7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.904+0000 7feb1e066640 10 monclient: get_auth_request con 0x7feb181d6c60 auth_method 0 2024-03-27T21:40:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.904+0000 7feb0effd640 10 client.4508.objecter ms_handle_connect 0x7feb181d6c60 2024-03-27T21:40:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.904+0000 7feb1e066640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7feb181d6c60 0x7feb181d7040 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.905+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7feb181d7580 con 0x7feb181d6c60 2024-03-27T21:40:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.905+0000 7feb0effd640 10 client.4508.objecter ms_dispatch 0x7feb18000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.905+0000 7feb0effd640 10 client.4508.objecter _finish_command 3 = system:0 2024-03-27T21:40:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.905+0000 7feb0effd640 20 client.4508.objecter put_session s=0x7feb181da920 osd=1 4 2024-03-27T21:40:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.905+0000 7feb0effd640 15 client.4508.objecter _session_command_op_remove 1 3 2024-03-27T21:40:23.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.932+0000 7feb202ca640 10 client.4508.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:23.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.932+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign -1 4 2024-03-27T21:40:23.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.933+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb181da920 osd=1 3 2024-03-27T21:40:23.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.933+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb181da920 osd=1 3 2024-03-27T21:40:23.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.933+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb181da920 osd=1 3 2024-03-27T21:40:23.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.933+0000 7feb202ca640 15 client.4508.objecter _session_command_op_remove -1 4 2024-03-27T21:40:23.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.933+0000 7feb202ca640 20 client.4508.objecter get_session s=0x7feb181da920 osd=1 3 2024-03-27T21:40:23.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.933+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign 1 4 2024-03-27T21:40:23.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.933+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb181da920 osd=1 4 2024-03-27T21:40:23.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.933+0000 7feb202ca640 10 client.4508.objecter _send_command 4 2024-03-27T21:40:23.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.933+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7feb1813fba0 con 0x7feb181d6c60 2024-03-27T21:40:23.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.935+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7feb1813fba0 con 0x7feb181d6c60 2024-03-27T21:40:23.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.935+0000 7feb0effd640 10 client.4508.objecter ms_dispatch 0x7feb18000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:23.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.935+0000 7feb0effd640 10 client.4508.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:23.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.935+0000 7feb0effd640 20 client.4508.objecter put_session s=0x7feb181da920 osd=1 4 2024-03-27T21:40:23.922 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:23.924 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:23.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.935+0000 7feb0effd640 15 client.4508.objecter _session_command_op_remove 1 4 2024-03-27T21:40:23.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.935+0000 7feb202ca640 10 client.4508.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:23.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.935+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign -1 5 2024-03-27T21:40:23.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.936+0000 7feb202ca640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7feb1802b170 0x7feb1802b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:23.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.936+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb180b55b0 osd=2 3 2024-03-27T21:40:23.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.936+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb180b55b0 osd=2 3 2024-03-27T21:40:23.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.936+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb180b55b0 osd=2 3 2024-03-27T21:40:23.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.936+0000 7feb202ca640 15 client.4508.objecter _session_command_op_remove -1 5 2024-03-27T21:40:23.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.936+0000 7feb202ca640 20 client.4508.objecter get_session s=0x7feb180b55b0 osd=2 3 2024-03-27T21:40:23.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.936+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign 2 5 2024-03-27T21:40:23.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.936+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb180b55b0 osd=2 4 2024-03-27T21:40:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.936+0000 7feb202ca640 10 client.4508.objecter _send_command 5 2024-03-27T21:40:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.936+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb180b5c30 con 0x7feb1802b170 2024-03-27T21:40:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.937+0000 7feb1d865640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7feb1802b170 0x7feb1802b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.937+0000 7feb1d865640 10 monclient: get_auth_request con 0x7feb1802b170 auth_method 0 2024-03-27T21:40:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.937+0000 7feb0effd640 10 client.4508.objecter ms_handle_connect 0x7feb1802b170 2024-03-27T21:40:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.937+0000 7feb1d865640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7feb1802b170 0x7feb1802b550 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.938+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== osd.2 v2:172.21.15.60:6824/44654 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7feb080892b0 con 0x7feb1802b170 2024-03-27T21:40:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.938+0000 7feb0effd640 10 client.4508.objecter ms_dispatch 0x7feb18000b90 command_reply(tid 5: 0 ) v1 2024-03-27T21:40:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.938+0000 7feb0effd640 10 client.4508.objecter _finish_command 5 = system:0 2024-03-27T21:40:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.938+0000 7feb0effd640 20 client.4508.objecter put_session s=0x7feb180b55b0 osd=2 4 2024-03-27T21:40:23.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.938+0000 7feb0effd640 15 client.4508.objecter _session_command_op_remove 2 5 2024-03-27T21:40:23.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.966+0000 7feb202ca640 10 client.4508.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:23.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.966+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign -1 6 2024-03-27T21:40:23.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.966+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb180b55b0 osd=2 3 2024-03-27T21:40:23.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.966+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb180b55b0 osd=2 3 2024-03-27T21:40:23.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.966+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb180b55b0 osd=2 3 2024-03-27T21:40:23.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.966+0000 7feb202ca640 15 client.4508.objecter _session_command_op_remove -1 6 2024-03-27T21:40:23.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.966+0000 7feb202ca640 20 client.4508.objecter get_session s=0x7feb180b55b0 osd=2 3 2024-03-27T21:40:23.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.966+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign 2 6 2024-03-27T21:40:23.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.966+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb180b55b0 osd=2 4 2024-03-27T21:40:23.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.966+0000 7feb202ca640 10 client.4508.objecter _send_command 6 2024-03-27T21:40:23.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.966+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7feb180b5e00 con 0x7feb1802b170 2024-03-27T21:40:23.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.969+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== osd.2 v2:172.21.15.60:6824/44654 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7feb180b5e00 con 0x7feb1802b170 2024-03-27T21:40:23.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.969+0000 7feb0effd640 10 client.4508.objecter ms_dispatch 0x7feb18000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:23.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.969+0000 7feb0effd640 10 client.4508.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:23.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.969+0000 7feb0effd640 20 client.4508.objecter put_session s=0x7feb180b55b0 osd=2 4 2024-03-27T21:40:23.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.969+0000 7feb0effd640 15 client.4508.objecter _session_command_op_remove 2 6 2024-03-27T21:40:23.956 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2: {} 2024-03-27T21:40:23.958 INFO:teuthology.orchestra.run.smithi060.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-27T21:40:23.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.969+0000 7feb202ca640 10 client.4508.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:23.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.969+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign -1 7 2024-03-27T21:40:23.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.969+0000 7feb202ca640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7feb181e7050 0x7feb181e9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:23.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.969+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb180b5270 osd=3 3 2024-03-27T21:40:23.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.969+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb180b5270 osd=3 3 2024-03-27T21:40:23.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.969+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb180b5270 osd=3 3 2024-03-27T21:40:23.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.969+0000 7feb202ca640 15 client.4508.objecter _session_command_op_remove -1 7 2024-03-27T21:40:23.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.969+0000 7feb202ca640 20 client.4508.objecter get_session s=0x7feb180b5270 osd=3 3 2024-03-27T21:40:23.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.969+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign 3 7 2024-03-27T21:40:23.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.969+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb180b5270 osd=3 4 2024-03-27T21:40:23.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.969+0000 7feb1e867640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7feb181e7050 0x7feb181e9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:23.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.969+0000 7feb202ca640 10 client.4508.objecter _send_command 7 2024-03-27T21:40:23.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.969+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb180b53e0 con 0x7feb181e7050 2024-03-27T21:40:23.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.970+0000 7feb1e867640 10 monclient: get_auth_request con 0x7feb181e7050 auth_method 0 2024-03-27T21:40:23.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.970+0000 7feb1e867640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7feb181e7050 0x7feb181e9440 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:23.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.970+0000 7feb0effd640 10 client.4508.objecter ms_handle_connect 0x7feb181e7050 2024-03-27T21:40:23.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.971+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7feb180b53e0 con 0x7feb181e7050 2024-03-27T21:40:23.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.971+0000 7feb0effd640 10 client.4508.objecter ms_dispatch 0x7feb18000b90 command_reply(tid 7: 0 ) v1 2024-03-27T21:40:23.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.971+0000 7feb0effd640 10 client.4508.objecter _finish_command 7 = system:0 2024-03-27T21:40:23.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.971+0000 7feb0effd640 20 client.4508.objecter put_session s=0x7feb180b5270 osd=3 4 2024-03-27T21:40:23.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.971+0000 7feb0effd640 15 client.4508.objecter _session_command_op_remove 3 7 2024-03-27T21:40:23.961 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:23.974+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:23.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.999+0000 7feb202ca640 10 client.4508.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:23.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.999+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign -1 8 2024-03-27T21:40:23.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.999+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb180b5270 osd=3 3 2024-03-27T21:40:23.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.999+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb180b5270 osd=3 3 2024-03-27T21:40:23.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.999+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb180b5270 osd=3 3 2024-03-27T21:40:23.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.999+0000 7feb202ca640 15 client.4508.objecter _session_command_op_remove -1 8 2024-03-27T21:40:23.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.999+0000 7feb202ca640 20 client.4508.objecter get_session s=0x7feb180b5270 osd=3 3 2024-03-27T21:40:23.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.999+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign 3 8 2024-03-27T21:40:23.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.999+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb180b5270 osd=3 4 2024-03-27T21:40:23.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.999+0000 7feb202ca640 10 client.4508.objecter _send_command 8 2024-03-27T21:40:23.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:23.999+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7feb180b53e0 con 0x7feb181e7050 2024-03-27T21:40:23.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.001+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7feb180b53e0 con 0x7feb181e7050 2024-03-27T21:40:23.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.001+0000 7feb0effd640 10 client.4508.objecter ms_dispatch 0x7feb18000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:23.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.001+0000 7feb0effd640 10 client.4508.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:23.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.001+0000 7feb0effd640 20 client.4508.objecter put_session s=0x7feb180b5270 osd=3 4 2024-03-27T21:40:23.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.001+0000 7feb0effd640 15 client.4508.objecter _session_command_op_remove 3 8 2024-03-27T21:40:23.988 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:23.990 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:23.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.002+0000 7feb202ca640 10 client.4508.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:23.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.002+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign -1 9 2024-03-27T21:40:23.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.002+0000 7feb202ca640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7feb18066250 0x7feb18066690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:23.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.002+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb180b53e0 osd=4 3 2024-03-27T21:40:23.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.002+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb180b53e0 osd=4 3 2024-03-27T21:40:23.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.002+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb180b53e0 osd=4 3 2024-03-27T21:40:23.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.002+0000 7feb202ca640 15 client.4508.objecter _session_command_op_remove -1 9 2024-03-27T21:40:23.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.002+0000 7feb202ca640 20 client.4508.objecter get_session s=0x7feb180b53e0 osd=4 3 2024-03-27T21:40:23.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.002+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign 4 9 2024-03-27T21:40:23.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.002+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb180b53e0 osd=4 4 2024-03-27T21:40:23.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.002+0000 7feb202ca640 10 client.4508.objecter _send_command 9 2024-03-27T21:40:23.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.002+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb181ef3a0 con 0x7feb18066250 2024-03-27T21:40:23.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.002+0000 7feb1e066640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7feb18066250 0x7feb18066690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:23.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.002+0000 7feb1e066640 10 monclient: get_auth_request con 0x7feb18066250 auth_method 0 2024-03-27T21:40:23.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.002+0000 7feb0effd640 10 client.4508.objecter ms_handle_connect 0x7feb18066250 2024-03-27T21:40:23.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.002+0000 7feb1e066640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7feb18066250 0x7feb18066690 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:23.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.003+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7feb181ef3a0 con 0x7feb18066250 2024-03-27T21:40:23.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.003+0000 7feb0effd640 10 client.4508.objecter ms_dispatch 0x7feb18000b90 command_reply(tid 9: 0 ) v1 2024-03-27T21:40:23.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.003+0000 7feb0effd640 10 client.4508.objecter _finish_command 9 = system:0 2024-03-27T21:40:23.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.003+0000 7feb0effd640 20 client.4508.objecter put_session s=0x7feb180b53e0 osd=4 4 2024-03-27T21:40:23.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.003+0000 7feb0effd640 15 client.4508.objecter _session_command_op_remove 4 9 2024-03-27T21:40:24.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.031+0000 7feb202ca640 10 client.4508.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:24.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.031+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign -1 10 2024-03-27T21:40:24.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.031+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb180b53e0 osd=4 3 2024-03-27T21:40:24.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.031+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb180b53e0 osd=4 3 2024-03-27T21:40:24.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.031+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb180b53e0 osd=4 3 2024-03-27T21:40:24.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.031+0000 7feb202ca640 15 client.4508.objecter _session_command_op_remove -1 10 2024-03-27T21:40:24.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.031+0000 7feb202ca640 20 client.4508.objecter get_session s=0x7feb180b53e0 osd=4 3 2024-03-27T21:40:24.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.031+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign 4 10 2024-03-27T21:40:24.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.031+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb180b53e0 osd=4 4 2024-03-27T21:40:24.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.031+0000 7feb202ca640 10 client.4508.objecter _send_command 10 2024-03-27T21:40:24.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.031+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7feb181ecb10 con 0x7feb18066250 2024-03-27T21:40:24.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.032+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7feb181ecb10 con 0x7feb18066250 2024-03-27T21:40:24.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.032+0000 7feb0effd640 10 client.4508.objecter ms_dispatch 0x7feb18000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:24.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.032+0000 7feb0effd640 10 client.4508.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:24.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.032+0000 7feb0effd640 20 client.4508.objecter put_session s=0x7feb180b53e0 osd=4 4 2024-03-27T21:40:24.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.032+0000 7feb0effd640 15 client.4508.objecter _session_command_op_remove 4 10 2024-03-27T21:40:24.019 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:24.021 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:24.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.033+0000 7feb202ca640 10 client.4508.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:24.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.033+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign -1 11 2024-03-27T21:40:24.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.033+0000 7feb202ca640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7feb181ed190 0x7feb181ef580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:24.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.033+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb181ecb10 osd=5 3 2024-03-27T21:40:24.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.033+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb181ecb10 osd=5 3 2024-03-27T21:40:24.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.033+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb181ecb10 osd=5 3 2024-03-27T21:40:24.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.033+0000 7feb202ca640 15 client.4508.objecter _session_command_op_remove -1 11 2024-03-27T21:40:24.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.033+0000 7feb202ca640 20 client.4508.objecter get_session s=0x7feb181ecb10 osd=5 3 2024-03-27T21:40:24.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.033+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign 5 11 2024-03-27T21:40:24.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.033+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb181ecb10 osd=5 4 2024-03-27T21:40:24.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.033+0000 7feb202ca640 10 client.4508.objecter _send_command 11 2024-03-27T21:40:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.033+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb181f2350 con 0x7feb181ed190 2024-03-27T21:40:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.033+0000 7feb1d865640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7feb181ed190 0x7feb181ef580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.033+0000 7feb1d865640 10 monclient: get_auth_request con 0x7feb181ed190 auth_method 0 2024-03-27T21:40:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.033+0000 7feb0effd640 10 client.4508.objecter ms_handle_connect 0x7feb181ed190 2024-03-27T21:40:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.033+0000 7feb1d865640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7feb181ed190 0x7feb181ef580 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.034+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7feb181f2350 con 0x7feb181ed190 2024-03-27T21:40:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.034+0000 7feb0effd640 10 client.4508.objecter ms_dispatch 0x7feb18000b90 command_reply(tid 11: 0 ) v1 2024-03-27T21:40:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.034+0000 7feb0effd640 10 client.4508.objecter _finish_command 11 = system:0 2024-03-27T21:40:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.034+0000 7feb0effd640 20 client.4508.objecter put_session s=0x7feb181ecb10 osd=5 4 2024-03-27T21:40:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.034+0000 7feb0effd640 15 client.4508.objecter _session_command_op_remove 5 11 2024-03-27T21:40:24.038 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:24.039 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:24.039 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:24.039 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:24.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.055+0000 7feb202ca640 10 client.4508.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:24.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.055+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign -1 12 2024-03-27T21:40:24.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.055+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb181ecb10 osd=5 3 2024-03-27T21:40:24.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.055+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb181ecb10 osd=5 3 2024-03-27T21:40:24.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.055+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb181ecb10 osd=5 3 2024-03-27T21:40:24.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.055+0000 7feb202ca640 15 client.4508.objecter _session_command_op_remove -1 12 2024-03-27T21:40:24.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.055+0000 7feb202ca640 20 client.4508.objecter get_session s=0x7feb181ecb10 osd=5 3 2024-03-27T21:40:24.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.055+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign 5 12 2024-03-27T21:40:24.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.055+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb181ecb10 osd=5 4 2024-03-27T21:40:24.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.055+0000 7feb202ca640 10 client.4508.objecter _send_command 12 2024-03-27T21:40:24.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.055+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7feb181efac0 con 0x7feb181ed190 2024-03-27T21:40:24.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.057+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7feb181efac0 con 0x7feb181ed190 2024-03-27T21:40:24.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.057+0000 7feb0effd640 10 client.4508.objecter ms_dispatch 0x7feb18000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:24.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.057+0000 7feb0effd640 10 client.4508.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:24.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.057+0000 7feb0effd640 20 client.4508.objecter put_session s=0x7feb181ecb10 osd=5 4 2024-03-27T21:40:24.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.057+0000 7feb0effd640 15 client.4508.objecter _session_command_op_remove 5 12 2024-03-27T21:40:24.045 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:24.047 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:24.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.057+0000 7feb202ca640 10 client.4508.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:24.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.057+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign -1 13 2024-03-27T21:40:24.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.057+0000 7feb202ca640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7feb181f0140 0x7feb181f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:24.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.057+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb181efac0 osd=6 3 2024-03-27T21:40:24.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.057+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb181efac0 osd=6 3 2024-03-27T21:40:24.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.057+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb181efac0 osd=6 3 2024-03-27T21:40:24.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.057+0000 7feb202ca640 15 client.4508.objecter _session_command_op_remove -1 13 2024-03-27T21:40:24.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.057+0000 7feb202ca640 20 client.4508.objecter get_session s=0x7feb181efac0 osd=6 3 2024-03-27T21:40:24.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.057+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign 6 13 2024-03-27T21:40:24.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.057+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb181efac0 osd=6 4 2024-03-27T21:40:24.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.057+0000 7feb202ca640 10 client.4508.objecter _send_command 13 2024-03-27T21:40:24.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.057+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb181f5300 con 0x7feb181f0140 2024-03-27T21:40:24.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.057+0000 7feb1e867640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7feb181f0140 0x7feb181f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:24.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.057+0000 7feb1e867640 10 monclient: get_auth_request con 0x7feb181f0140 auth_method 0 2024-03-27T21:40:24.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.058+0000 7feb0effd640 10 client.4508.objecter ms_handle_connect 0x7feb181f0140 2024-03-27T21:40:24.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.058+0000 7feb1e867640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7feb181f0140 0x7feb181f2530 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:24.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.058+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7feb181f5300 con 0x7feb181f0140 2024-03-27T21:40:24.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.058+0000 7feb0effd640 10 client.4508.objecter ms_dispatch 0x7feb18000b90 command_reply(tid 13: 0 ) v1 2024-03-27T21:40:24.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.058+0000 7feb0effd640 10 client.4508.objecter _finish_command 13 = system:0 2024-03-27T21:40:24.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.058+0000 7feb0effd640 20 client.4508.objecter put_session s=0x7feb181efac0 osd=6 4 2024-03-27T21:40:24.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.058+0000 7feb0effd640 15 client.4508.objecter _session_command_op_remove 6 13 2024-03-27T21:40:24.051 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-27T21:40:24.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.073+0000 7feb202ca640 10 client.4508.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:24.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.073+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign -1 14 2024-03-27T21:40:24.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.073+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb181efac0 osd=6 3 2024-03-27T21:40:24.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.073+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb181efac0 osd=6 3 2024-03-27T21:40:24.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.073+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb181efac0 osd=6 3 2024-03-27T21:40:24.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.073+0000 7feb202ca640 15 client.4508.objecter _session_command_op_remove -1 14 2024-03-27T21:40:24.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.073+0000 7feb202ca640 20 client.4508.objecter get_session s=0x7feb181efac0 osd=6 3 2024-03-27T21:40:24.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.073+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign 6 14 2024-03-27T21:40:24.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.073+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb181efac0 osd=6 4 2024-03-27T21:40:24.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.073+0000 7feb202ca640 10 client.4508.objecter _send_command 14 2024-03-27T21:40:24.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.073+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7feb181f2a70 con 0x7feb181f0140 2024-03-27T21:40:24.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.075+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7feb181f2a70 con 0x7feb181f0140 2024-03-27T21:40:24.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.075+0000 7feb0effd640 10 client.4508.objecter ms_dispatch 0x7feb18000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:24.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.075+0000 7feb0effd640 10 client.4508.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:24.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.075+0000 7feb0effd640 20 client.4508.objecter put_session s=0x7feb181efac0 osd=6 4 2024-03-27T21:40:24.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.075+0000 7feb0effd640 15 client.4508.objecter _session_command_op_remove 6 14 2024-03-27T21:40:24.062 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:24.063 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:24.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.075+0000 7feb202ca640 10 client.4508.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:24.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.075+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign -1 15 2024-03-27T21:40:24.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.075+0000 7feb202ca640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7feb181f30f0 0x7feb181f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:24.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.075+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb181f2a70 osd=7 3 2024-03-27T21:40:24.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.075+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb181f2a70 osd=7 3 2024-03-27T21:40:24.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.075+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb181f2a70 osd=7 3 2024-03-27T21:40:24.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.075+0000 7feb202ca640 15 client.4508.objecter _session_command_op_remove -1 15 2024-03-27T21:40:24.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.075+0000 7feb202ca640 20 client.4508.objecter get_session s=0x7feb181f2a70 osd=7 3 2024-03-27T21:40:24.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.075+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign 7 15 2024-03-27T21:40:24.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.075+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb181f2a70 osd=7 4 2024-03-27T21:40:24.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.075+0000 7feb202ca640 10 client.4508.objecter _send_command 15 2024-03-27T21:40:24.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.075+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb181f8430 con 0x7feb181f30f0 2024-03-27T21:40:24.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.076+0000 7feb1e066640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7feb181f30f0 0x7feb181f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:24.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.076+0000 7feb1e066640 10 monclient: get_auth_request con 0x7feb181f30f0 auth_method 0 2024-03-27T21:40:24.067 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:24.076+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:24.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.076+0000 7feb0effd640 10 client.4508.objecter ms_handle_connect 0x7feb181f30f0 2024-03-27T21:40:24.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.076+0000 7feb1e066640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7feb181f30f0 0x7feb181f54e0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:24.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.077+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7feb181f8430 con 0x7feb181f30f0 2024-03-27T21:40:24.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.077+0000 7feb0effd640 10 client.4508.objecter ms_dispatch 0x7feb18000b90 command_reply(tid 15: 0 ) v1 2024-03-27T21:40:24.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.077+0000 7feb0effd640 10 client.4508.objecter _finish_command 15 = system:0 2024-03-27T21:40:24.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.077+0000 7feb0effd640 20 client.4508.objecter put_session s=0x7feb181f2a70 osd=7 4 2024-03-27T21:40:24.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.077+0000 7feb0effd640 15 client.4508.objecter _session_command_op_remove 7 15 2024-03-27T21:40:24.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.092+0000 7feb202ca640 10 client.4508.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:24.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.092+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign -1 16 2024-03-27T21:40:24.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.092+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb181f2a70 osd=7 3 2024-03-27T21:40:24.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.092+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb181f2a70 osd=7 3 2024-03-27T21:40:24.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.092+0000 7feb202ca640 20 client.4508.objecter _get_session s=0x7feb181f2a70 osd=7 3 2024-03-27T21:40:24.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.092+0000 7feb202ca640 15 client.4508.objecter _session_command_op_remove -1 16 2024-03-27T21:40:24.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.092+0000 7feb202ca640 20 client.4508.objecter get_session s=0x7feb181f2a70 osd=7 3 2024-03-27T21:40:24.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.092+0000 7feb202ca640 15 client.4508.objecter _session_command_op_assign 7 16 2024-03-27T21:40:24.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.092+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb181f2a70 osd=7 4 2024-03-27T21:40:24.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.092+0000 7feb202ca640 10 client.4508.objecter _send_command 16 2024-03-27T21:40:24.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.092+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7feb181f7200 con 0x7feb181f30f0 2024-03-27T21:40:24.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb0effd640 1 -- 172.21.15.60:0/2962606661 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7feb181f7200 con 0x7feb181f30f0 2024-03-27T21:40:24.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb0effd640 10 client.4508.objecter ms_dispatch 0x7feb18000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:24.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb0effd640 10 client.4508.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:24.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb0effd640 20 client.4508.objecter put_session s=0x7feb181f2a70 osd=7 4 2024-03-27T21:40:24.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb0effd640 15 client.4508.objecter _session_command_op_remove 7 16 2024-03-27T21:40:24.082 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:24.087 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:24.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 10 librados: watch_flush enter 2024-03-27T21:40:24.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 10 librados: watch_flush exit 2024-03-27T21:40:24.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 10 client.4508.objecter close_session for osd.0 2024-03-27T21:40:24.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7feb1814bd20 msgr2=0x7feb181da1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:24.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7feb1814bd20 0x7feb181da1a0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:24.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb18069470 osd=0 1 2024-03-27T21:40:24.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 10 client.4508.objecter close_session for osd.1 2024-03-27T21:40:24.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7feb181d6c60 msgr2=0x7feb181d7040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:24.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7feb181d6c60 0x7feb181d7040 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:24.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb181da920 osd=1 1 2024-03-27T21:40:24.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 10 client.4508.objecter close_session for osd.2 2024-03-27T21:40:24.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7feb1802b170 msgr2=0x7feb1802b550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:24.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7feb1802b170 0x7feb1802b550 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:24.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb180b55b0 osd=2 1 2024-03-27T21:40:24.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 10 client.4508.objecter close_session for osd.3 2024-03-27T21:40:24.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7feb181e7050 msgr2=0x7feb181e9440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:24.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7feb181e7050 0x7feb181e9440 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:24.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb180b5270 osd=3 1 2024-03-27T21:40:24.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 10 client.4508.objecter close_session for osd.4 2024-03-27T21:40:24.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7feb18066250 msgr2=0x7feb18066690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:24.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb1e867640 1 -- 172.21.15.60:0/2962606661 reap_dead start 2024-03-27T21:40:24.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7feb18066250 0x7feb18066690 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:24.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb1e867640 1 -- 172.21.15.60:0/2962606661 reap_dead start 2024-03-27T21:40:24.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb180b53e0 osd=4 1 2024-03-27T21:40:24.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 10 client.4508.objecter close_session for osd.5 2024-03-27T21:40:24.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7feb181ed190 msgr2=0x7feb181ef580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:24.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7feb181ed190 0x7feb181ef580 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:24.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb181ecb10 osd=5 1 2024-03-27T21:40:24.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 10 client.4508.objecter close_session for osd.6 2024-03-27T21:40:24.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7feb181f0140 msgr2=0x7feb181f2530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:24.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7feb181f0140 0x7feb181f2530 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:24.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb181efac0 osd=6 1 2024-03-27T21:40:24.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 10 client.4508.objecter close_session for osd.7 2024-03-27T21:40:24.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7feb181f30f0 msgr2=0x7feb181f54e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:24.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7feb181f30f0 0x7feb181f54e0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:24.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 20 client.4508.objecter put_session s=0x7feb181f2a70 osd=7 1 2024-03-27T21:40:24.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 20 client.4508.objecter shutdown clearing up homeless session... 2024-03-27T21:40:24.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.094+0000 7feb202ca640 10 client.4508.objecter successfully canceled tick 2024-03-27T21:40:24.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.095+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7feafc06c590 msgr2=0x7feafc06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:24.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.095+0000 7feb202ca640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7feafc06c590 0x7feafc06ea50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feb14000f60 tx=0x7feb14007ce0 comp rx=0 tx=0).stop 2024-03-27T21:40:24.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.095+0000 7feb202ca640 10 monclient: shutdown 2024-03-27T21:40:24.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.095+0000 7feb202ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:24.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.095+0000 7feb1e867640 1 -- 172.21.15.60:0/2962606661 reap_dead start 2024-03-27T21:40:24.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.095+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7feb1813b020 msgr2=0x7feb1813dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:24.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.095+0000 7feb202ca640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7feb1813b020 0x7feb1813dc20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feb08035040 tx=0x7feb080046c0 comp rx=0 tx=0).stop 2024-03-27T21:40:24.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.095+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 shutdown_connections 2024-03-27T21:40:24.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.095+0000 7feb202ca640 1 --2- 172.21.15.60:0/2962606661 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7feb1813b020 0x7feb1813dc20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:24.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.095+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 >> 172.21.15.60:0/2962606661 conn(0x7feb180608e0 msgr2=0x7feb180b6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:24.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.095+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 shutdown_connections 2024-03-27T21:40:24.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.095+0000 7feb202ca640 1 -- 172.21.15.60:0/2962606661 wait complete. 2024-03-27T21:40:24.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:24.095+0000 7feb202ca640 1 librados: shutdown 2024-03-27T21:40:24.162 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:24.176+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:24.172 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:24.172 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:24.172 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:24.172 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:24.173 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:24.182 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-27T21:40:24.263 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:24.277+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:24.330 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:24.331 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:24.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:24.331 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:40:24.331 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:24.340 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:40:24.364 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:24.378+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:24.449 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:24.450 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:24.450 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:24.450 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:24.459 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:40:24.465 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:24.478+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:24.565 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:24.579+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:24.582 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:24.582 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:24.582 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:24.582 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:24.582 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:24.594 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:40:24.679 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:24.693+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:24.747 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:24.747 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:24.747 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:24.747 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-03-27T21:40:24.747 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.747 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:20 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2024-03-27T21:40:24.748 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:24.748 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.871076916, 2024-03-27T21:40:24.748 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015345630000000001, 2024-03-27T21:40:24.748 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:24.748 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:24.748 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:24.748 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.748 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:24.748 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:24.748 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.748 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.748 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:24.748 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:24.748 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.749 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.749 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:24.749 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888562+0000", 2024-03-27T21:40:24.749 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:24.749 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.749 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:24.749 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888735+0000", 2024-03-27T21:40:24.749 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017254799999999999 2024-03-27T21:40:24.749 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.749 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:24.749 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888747+0000", 2024-03-27T21:40:24.749 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2469999999999999e-05 2024-03-27T21:40:24.750 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.750 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.750 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:24.750 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888761+0000", 2024-03-27T21:40:24.750 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3254e-05 2024-03-27T21:40:24.750 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.750 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.750 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:24.750 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888804+0000", 2024-03-27T21:40:24.750 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3129e-05 2024-03-27T21:40:24.750 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.750 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.750 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:24.750 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888820+0000", 2024-03-27T21:40:24.751 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6426000000000002e-05 2024-03-27T21:40:24.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.751 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:24.751 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890060+0000", 2024-03-27T21:40:24.751 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012398050000000001 2024-03-27T21:40:24.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.751 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:24.751 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890098+0000", 2024-03-27T21:40:24.751 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7795e-05 2024-03-27T21:40:24.751 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.751 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:24.751 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.752 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2024-03-27T21:40:24.752 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:24.752 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.870252516000001, 2024-03-27T21:40:24.752 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001069555, 2024-03-27T21:40:24.752 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:24.752 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:24.752 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:24.752 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.752 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:24.752 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:24.752 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.752 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.752 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.752 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:24.752 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:24.753 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.753 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:24.753 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889387+0000", 2024-03-27T21:40:24.753 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:24.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.753 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:24.753 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889455+0000", 2024-03-27T21:40:24.753 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7930999999999997e-05 2024-03-27T21:40:24.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.753 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:24.753 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889470+0000", 2024-03-27T21:40:24.754 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5554e-05 2024-03-27T21:40:24.754 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.754 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:24.754 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889476+0000", 2024-03-27T21:40:24.754 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0619999999999997e-06 2024-03-27T21:40:24.754 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.754 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:24.754 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889513+0000", 2024-03-27T21:40:24.754 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6606999999999997e-05 2024-03-27T21:40:24.754 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.754 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:24.755 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889528+0000", 2024-03-27T21:40:24.755 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4982e-05 2024-03-27T21:40:24.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.755 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:24.755 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890421+0000", 2024-03-27T21:40:24.755 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00089293700000000001 2024-03-27T21:40:24.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.755 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:24.755 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890457+0000", 2024-03-27T21:40:24.755 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6260000000000002e-05 2024-03-27T21:40:24.755 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.755 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:24.755 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.756 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:38 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'24, mlcod=17'22)", 2024-03-27T21:40:24.756 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:24.756 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.85301655, 2024-03-27T21:40:24.756 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016182080000000001, 2024-03-27T21:40:24.756 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:24.756 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:24.756 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:24.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.756 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:24.756 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:24.756 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.757 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:24.757 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:24.757 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.757 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:24.757 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906623+0000", 2024-03-27T21:40:24.757 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:24.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.757 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:24.757 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906687+0000", 2024-03-27T21:40:24.757 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3706000000000003e-05 2024-03-27T21:40:24.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.757 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:24.758 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906698+0000", 2024-03-27T21:40:24.758 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1752e-05 2024-03-27T21:40:24.758 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.758 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.758 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:24.758 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906704+0000", 2024-03-27T21:40:24.758 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3940000000000004e-06 2024-03-27T21:40:24.758 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.758 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.758 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:24.759 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906741+0000", 2024-03-27T21:40:24.759 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6944999999999999e-05 2024-03-27T21:40:24.759 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.759 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.759 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:24.759 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906760+0000", 2024-03-27T21:40:24.759 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9293e-05 2024-03-27T21:40:24.759 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.759 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.759 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:24.759 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908211+0000", 2024-03-27T21:40:24.759 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014511350000000001 2024-03-27T21:40:24.759 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.759 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.759 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:24.760 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908242+0000", 2024-03-27T21:40:24.760 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0712e-05 2024-03-27T21:40:24.760 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.760 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:24.760 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.760 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.760 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.760 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:39 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'25, mlcod=17'22)", 2024-03-27T21:40:24.760 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:24.760 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.852171634999998, 2024-03-27T21:40:24.760 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012060199999999999, 2024-03-27T21:40:24.760 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:24.760 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:24.760 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:24.760 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.760 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:24.761 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:24.761 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.761 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.761 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.761 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:24.761 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:24.761 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.761 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.761 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.762 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:24.762 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:24.762 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:24.762 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.762 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.762 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:24.762 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907533+0000", 2024-03-27T21:40:24.762 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5675999999999999e-05 2024-03-27T21:40:24.762 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.762 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.762 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:24.762 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907545+0000", 2024-03-27T21:40:24.762 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.115e-05 2024-03-27T21:40:24.763 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.763 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.763 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:24.763 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907550+0000", 2024-03-27T21:40:24.763 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8320000000000002e-06 2024-03-27T21:40:24.763 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.763 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.763 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:24.763 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907592+0000", 2024-03-27T21:40:24.763 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1309e-05 2024-03-27T21:40:24.763 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.763 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.764 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:24.764 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907611+0000", 2024-03-27T21:40:24.764 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9009e-05 2024-03-27T21:40:24.764 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.764 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.764 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:24.764 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908638+0000", 2024-03-27T21:40:24.764 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001027533 2024-03-27T21:40:24.764 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.764 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.764 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:24.764 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908675+0000", 2024-03-27T21:40:24.765 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6329999999999999e-05 2024-03-27T21:40:24.765 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.765 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:24.765 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.765 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.765 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.765 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:57 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'35, mlcod=17'33)", 2024-03-27T21:40:24.765 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:24.765 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.838087203000001, 2024-03-27T21:40:24.766 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001346578, 2024-03-27T21:40:24.766 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:24.766 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:24.766 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:24.766 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.766 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:24.766 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:24.766 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.766 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.766 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.766 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:24.766 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:24.766 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.766 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.767 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.767 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:24.767 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921552+0000", 2024-03-27T21:40:24.767 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:24.767 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.767 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.767 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:24.767 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921615+0000", 2024-03-27T21:40:24.767 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2787999999999996e-05 2024-03-27T21:40:24.767 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.767 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.767 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:24.767 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921626+0000", 2024-03-27T21:40:24.767 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0888e-05 2024-03-27T21:40:24.767 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.767 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.768 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:24.768 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921635+0000", 2024-03-27T21:40:24.768 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4639999999999995e-06 2024-03-27T21:40:24.768 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.768 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.768 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:24.768 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921673+0000", 2024-03-27T21:40:24.768 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7891e-05 2024-03-27T21:40:24.768 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.768 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.768 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:24.768 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921694+0000", 2024-03-27T21:40:24.768 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0426000000000001e-05 2024-03-27T21:40:24.768 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.768 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.769 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:24.769 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922865+0000", 2024-03-27T21:40:24.769 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011709509999999999 2024-03-27T21:40:24.769 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.769 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.769 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:24.770 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922900+0000", 2024-03-27T21:40:24.770 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4819e-05 2024-03-27T21:40:24.770 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.770 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:24.770 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.770 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.770 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.770 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:71 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'45, mlcod=17'43)", 2024-03-27T21:40:24.770 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:24.770 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.822788490999997, 2024-03-27T21:40:24.770 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001460907, 2024-03-27T21:40:24.770 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:24.770 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:24.770 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:24.771 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.771 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:24.771 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:24.771 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.771 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.771 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:24.771 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:24.771 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.771 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.771 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:24.771 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936851+0000", 2024-03-27T21:40:24.771 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:24.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.772 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.772 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:24.772 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936903+0000", 2024-03-27T21:40:24.772 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2302999999999997e-05 2024-03-27T21:40:24.772 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.772 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.772 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:24.772 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936915+0000", 2024-03-27T21:40:24.772 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1967000000000001e-05 2024-03-27T21:40:24.772 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.772 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.773 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:24.773 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936923+0000", 2024-03-27T21:40:24.773 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9370000000000004e-06 2024-03-27T21:40:24.773 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.773 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:24.773 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936970+0000", 2024-03-27T21:40:24.773 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6987000000000003e-05 2024-03-27T21:40:24.773 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.773 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:24.773 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937002+0000", 2024-03-27T21:40:24.773 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2410999999999999e-05 2024-03-27T21:40:24.773 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.774 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:24.774 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938279+0000", 2024-03-27T21:40:24.774 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001276552 2024-03-27T21:40:24.774 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.774 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.774 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:24.774 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938313+0000", 2024-03-27T21:40:24.774 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3526999999999999e-05 2024-03-27T21:40:24.774 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.774 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:24.774 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.774 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.774 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.774 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:72 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'46, mlcod=17'43)", 2024-03-27T21:40:24.775 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:24.775 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.822150929000003, 2024-03-27T21:40:24.775 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014239490000000001, 2024-03-27T21:40:24.775 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:24.775 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:24.775 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:24.775 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.775 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:24.775 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:24.775 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.775 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.775 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.775 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:24.775 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:24.775 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.776 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.776 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:24.776 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937488+0000", 2024-03-27T21:40:24.776 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:24.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.776 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.776 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:24.776 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937547+0000", 2024-03-27T21:40:24.776 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8196999999999999e-05 2024-03-27T21:40:24.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.776 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.776 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:24.776 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937559+0000", 2024-03-27T21:40:24.776 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1928e-05 2024-03-27T21:40:24.777 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.777 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.777 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:24.777 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937564+0000", 2024-03-27T21:40:24.777 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4759999999999997e-06 2024-03-27T21:40:24.777 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.777 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.777 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:24.777 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937621+0000", 2024-03-27T21:40:24.777 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6712000000000002e-05 2024-03-27T21:40:24.777 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.777 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.777 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:24.777 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937640+0000", 2024-03-27T21:40:24.777 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9517e-05 2024-03-27T21:40:24.778 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.778 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.778 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:24.778 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938883+0000", 2024-03-27T21:40:24.778 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001242567 2024-03-27T21:40:24.778 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.778 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.778 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:24.778 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938913+0000", 2024-03-27T21:40:24.778 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0287e-05 2024-03-27T21:40:24.778 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.778 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:24.778 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.778 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.778 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.779 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:73 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'47, mlcod=17'43)", 2024-03-27T21:40:24.779 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:24.779 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.821474834999997, 2024-03-27T21:40:24.779 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010746239999999999, 2024-03-27T21:40:24.779 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:24.779 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:24.779 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:24.779 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.779 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:24.779 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:24.779 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.779 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.779 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.779 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:24.779 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:24.780 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.780 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.780 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.780 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:24.780 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:24.780 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:24.780 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.780 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.780 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:24.780 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938217+0000", 2024-03-27T21:40:24.780 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2743e-05 2024-03-27T21:40:24.781 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.781 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.781 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:24.781 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938229+0000", 2024-03-27T21:40:24.781 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1956999999999999e-05 2024-03-27T21:40:24.781 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.781 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.781 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:24.781 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938235+0000", 2024-03-27T21:40:24.781 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4589999999999997e-06 2024-03-27T21:40:24.781 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.781 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.781 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:24.781 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938341+0000", 2024-03-27T21:40:24.781 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000106054 2024-03-27T21:40:24.782 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.782 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.782 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:24.782 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938369+0000", 2024-03-27T21:40:24.782 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8291000000000001e-05 2024-03-27T21:40:24.782 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.782 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.782 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:24.782 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939209+0000", 2024-03-27T21:40:24.782 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084045400000000003 2024-03-27T21:40:24.782 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.782 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.782 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:24.783 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939240+0000", 2024-03-27T21:40:24.783 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0429999999999998e-05 2024-03-27T21:40:24.783 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.783 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:24.783 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.783 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.783 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.783 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:75 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'49, mlcod=17'47)", 2024-03-27T21:40:24.783 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:24.783 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.815037187999998, 2024-03-27T21:40:24.783 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001017001, 2024-03-27T21:40:24.783 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:24.783 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:24.783 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:24.783 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.784 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:24.784 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:24.784 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.784 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.784 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.784 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:24.784 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:24.784 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.784 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.784 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.784 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:24.785 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944601+0000", 2024-03-27T21:40:24.785 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999986 2024-03-27T21:40:24.785 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.785 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.785 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:24.785 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944614+0000", 2024-03-27T21:40:24.785 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2496e-05 2024-03-27T21:40:24.785 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.785 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.785 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:24.786 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944640+0000", 2024-03-27T21:40:24.786 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5865999999999999e-05 2024-03-27T21:40:24.786 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.786 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.786 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:24.786 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944651+0000", 2024-03-27T21:40:24.786 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1797000000000001e-05 2024-03-27T21:40:24.786 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.786 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.786 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:24.786 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944762+0000", 2024-03-27T21:40:24.787 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110412 2024-03-27T21:40:24.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.787 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:24.787 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944820+0000", 2024-03-27T21:40:24.787 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7649999999999999e-05 2024-03-27T21:40:24.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.787 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:24.787 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945585+0000", 2024-03-27T21:40:24.787 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076562600000000005 2024-03-27T21:40:24.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.787 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:24.787 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945620+0000", 2024-03-27T21:40:24.788 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4764000000000001e-05 2024-03-27T21:40:24.788 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.788 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:24.788 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.788 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.788 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.788 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:86 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'50)", 2024-03-27T21:40:24.788 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:24.788 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.807275701000002, 2024-03-27T21:40:24.788 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011392380000000001, 2024-03-27T21:40:24.788 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:24.788 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:24.788 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:24.788 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.788 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:24.789 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:24.789 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.789 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.789 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.789 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:24.789 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:24.789 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.789 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.789 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.789 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:24.789 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:24.789 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:24.789 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.789 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.789 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:24.790 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952373+0000", 2024-03-27T21:40:24.790 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9220000000000003e-06 2024-03-27T21:40:24.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.790 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.790 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:24.790 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952381+0000", 2024-03-27T21:40:24.790 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3070000000000003e-06 2024-03-27T21:40:24.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.790 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.790 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:24.790 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952385+0000", 2024-03-27T21:40:24.790 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.121e-06 2024-03-27T21:40:24.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.790 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.790 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:24.791 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952419+0000", 2024-03-27T21:40:24.791 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4082999999999999e-05 2024-03-27T21:40:24.791 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.791 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.791 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:24.791 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952440+0000", 2024-03-27T21:40:24.791 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0837999999999999e-05 2024-03-27T21:40:24.791 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.791 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.791 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:24.791 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953472+0000", 2024-03-27T21:40:24.791 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001032119 2024-03-27T21:40:24.791 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.791 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.791 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:24.792 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953504+0000", 2024-03-27T21:40:24.792 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1250000000000001e-05 2024-03-27T21:40:24.792 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.792 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:24.792 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.792 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.792 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.792 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:87 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-27T21:40:24.792 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:24.792 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.806908444000001, 2024-03-27T21:40:24.792 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00135519, 2024-03-27T21:40:24.792 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:24.793 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:24.793 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:24.793 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.793 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:24.793 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:24.793 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.793 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.793 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:24.793 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:24.793 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.794 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.794 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.794 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:24.794 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952731+0000", 2024-03-27T21:40:24.794 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:24.794 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.794 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.794 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:24.794 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952781+0000", 2024-03-27T21:40:24.794 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0161000000000003e-05 2024-03-27T21:40:24.794 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.794 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.794 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:24.794 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952794+0000", 2024-03-27T21:40:24.795 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2853e-05 2024-03-27T21:40:24.795 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.795 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.795 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:24.795 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952799+0000", 2024-03-27T21:40:24.795 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5560000000000001e-06 2024-03-27T21:40:24.795 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.795 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.795 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:24.795 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952832+0000", 2024-03-27T21:40:24.795 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3834999999999997e-05 2024-03-27T21:40:24.795 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.795 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.795 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:24.795 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952851+0000", 2024-03-27T21:40:24.796 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8289999999999999e-05 2024-03-27T21:40:24.796 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.796 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.796 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:24.796 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954055+0000", 2024-03-27T21:40:24.796 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012037949999999999 2024-03-27T21:40:24.796 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.796 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.796 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:24.796 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954087+0000", 2024-03-27T21:40:24.796 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2292999999999997e-05 2024-03-27T21:40:24.796 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.796 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:24.796 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.796 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.797 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.797 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:89 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'59, mlcod=17'57)", 2024-03-27T21:40:24.797 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:24.797 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.804634223000001, 2024-03-27T21:40:24.797 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00138893, 2024-03-27T21:40:24.797 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:24.797 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:24.797 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:24.797 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.797 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:24.797 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:24.797 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.797 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.797 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.797 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:24.798 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:24.798 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.798 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.798 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.798 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:24.798 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955005+0000", 2024-03-27T21:40:24.798 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:24.798 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.798 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.798 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:24.798 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955071+0000", 2024-03-27T21:40:24.798 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5290999999999995e-05 2024-03-27T21:40:24.798 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.798 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.798 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:24.799 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955081+0000", 2024-03-27T21:40:24.799 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0572e-05 2024-03-27T21:40:24.799 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.799 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.799 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:24.799 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955086+0000", 2024-03-27T21:40:24.799 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8380000000000001e-06 2024-03-27T21:40:24.799 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.799 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.799 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:24.799 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955122+0000", 2024-03-27T21:40:24.799 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5698000000000003e-05 2024-03-27T21:40:24.799 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.799 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.799 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:24.800 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955140+0000", 2024-03-27T21:40:24.800 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7900000000000001e-05 2024-03-27T21:40:24.800 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.800 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.800 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:24.800 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956366+0000", 2024-03-27T21:40:24.800 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001226471 2024-03-27T21:40:24.800 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.800 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.800 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:24.800 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956395+0000", 2024-03-27T21:40:24.800 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8782e-05 2024-03-27T21:40:24.800 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.801 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:24.801 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.801 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.801 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.801 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:90 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'60, mlcod=17'57)", 2024-03-27T21:40:24.801 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:24.801 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.803949646, 2024-03-27T21:40:24.801 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010778859999999999, 2024-03-27T21:40:24.801 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:24.801 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:24.801 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:24.801 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.801 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:24.801 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:24.801 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.802 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.802 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.802 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:24.802 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:24.802 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.802 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.802 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.802 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:24.802 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:24.802 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:24.802 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.802 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.802 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:24.802 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955747+0000", 2024-03-27T21:40:24.802 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7130999999999999e-05 2024-03-27T21:40:24.803 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.803 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.803 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:24.803 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955756+0000", 2024-03-27T21:40:24.803 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2750000000000001e-06 2024-03-27T21:40:24.803 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.803 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.803 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:24.803 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955761+0000", 2024-03-27T21:40:24.803 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3900000000000003e-06 2024-03-27T21:40:24.803 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.803 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.803 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:24.803 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955797+0000", 2024-03-27T21:40:24.803 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6650000000000003e-05 2024-03-27T21:40:24.803 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.804 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.804 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:24.804 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955816+0000", 2024-03-27T21:40:24.804 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8195000000000001e-05 2024-03-27T21:40:24.804 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.804 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.804 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:24.804 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956739+0000", 2024-03-27T21:40:24.804 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00092369700000000002 2024-03-27T21:40:24.804 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.804 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.804 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:24.804 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956768+0000", 2024-03-27T21:40:24.804 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9048e-05 2024-03-27T21:40:24.805 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.805 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:24.805 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.805 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.805 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.805 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:118 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'80, mlcod=17'78)", 2024-03-27T21:40:24.805 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:24.805 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.670958822999999, 2024-03-27T21:40:24.805 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014200669999999999, 2024-03-27T21:40:24.805 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:24.805 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:24.805 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:24.805 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.805 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:24.805 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:24.805 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.806 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.806 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.806 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:24.806 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:24.806 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.806 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.806 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.806 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:24.806 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:24.806 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:24.806 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.806 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.806 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:24.806 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088756+0000", 2024-03-27T21:40:24.806 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4902999999999998e-05 2024-03-27T21:40:24.807 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.807 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.807 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:24.807 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088764+0000", 2024-03-27T21:40:24.807 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8169999999999999e-06 2024-03-27T21:40:24.807 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.807 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.807 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:24.807 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088767+0000", 2024-03-27T21:40:24.807 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8310000000000003e-06 2024-03-27T21:40:24.807 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.807 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.807 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:24.807 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088798+0000", 2024-03-27T21:40:24.807 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0800999999999999e-05 2024-03-27T21:40:24.808 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.808 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.808 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:24.808 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088817+0000", 2024-03-27T21:40:24.808 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8467000000000001e-05 2024-03-27T21:40:24.808 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.808 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.808 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:24.808 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090074+0000", 2024-03-27T21:40:24.808 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012574089999999999 2024-03-27T21:40:24.808 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.808 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.808 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:24.808 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090101+0000", 2024-03-27T21:40:24.808 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7223e-05 2024-03-27T21:40:24.809 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.809 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:24.809 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.809 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.809 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.809 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:119 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'81, mlcod=17'78)", 2024-03-27T21:40:24.809 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:24.809 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.670224028, 2024-03-27T21:40:24.809 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014990870000000001, 2024-03-27T21:40:24.809 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:24.809 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:24.809 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:24.809 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.809 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:24.809 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:24.809 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.810 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.810 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.810 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:24.810 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:24.810 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.810 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.810 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.810 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:24.810 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:24.810 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:24.810 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.810 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.810 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:24.810 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089491+0000", 2024-03-27T21:40:24.810 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4805e-05 2024-03-27T21:40:24.811 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.811 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.811 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:24.811 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089498+0000", 2024-03-27T21:40:24.811 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0650000000000001e-06 2024-03-27T21:40:24.811 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.811 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.811 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:24.811 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089501+0000", 2024-03-27T21:40:24.811 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.54e-06 2024-03-27T21:40:24.811 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.811 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.811 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:24.811 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089533+0000", 2024-03-27T21:40:24.811 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1838999999999998e-05 2024-03-27T21:40:24.812 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.812 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.812 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:24.812 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089550+0000", 2024-03-27T21:40:24.812 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7465999999999998e-05 2024-03-27T21:40:24.812 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.812 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.812 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:24.812 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090890+0000", 2024-03-27T21:40:24.812 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013397979999999999 2024-03-27T21:40:24.812 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.812 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.812 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:24.812 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090915+0000", 2024-03-27T21:40:24.812 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4905e-05 2024-03-27T21:40:24.812 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.813 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:24.813 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.813 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.813 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.813 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:120 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'82, mlcod=17'78)", 2024-03-27T21:40:24.813 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:24.813 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.669424147999997, 2024-03-27T21:40:24.813 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001527093, 2024-03-27T21:40:24.813 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:24.813 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:24.813 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:24.813 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.813 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:24.813 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:24.813 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.814 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.814 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.814 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:24.814 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:24.814 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.814 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.814 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.814 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:24.814 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:24.814 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:24.814 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.814 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.814 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:24.814 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090267+0000", 2024-03-27T21:40:24.814 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1477000000000002e-05 2024-03-27T21:40:24.815 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.815 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.815 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:24.818 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090273+0000", 2024-03-27T21:40:24.818 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9519999999999999e-06 2024-03-27T21:40:24.818 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.818 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.818 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:24.818 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090276+0000", 2024-03-27T21:40:24.819 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.219e-06 2024-03-27T21:40:24.819 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.819 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.819 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:24.819 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090304+0000", 2024-03-27T21:40:24.819 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8073999999999999e-05 2024-03-27T21:40:24.819 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.819 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.819 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:24.819 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090316+0000", 2024-03-27T21:40:24.819 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2088e-05 2024-03-27T21:40:24.819 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.819 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.819 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:24.819 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091717+0000", 2024-03-27T21:40:24.820 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014003889999999999 2024-03-27T21:40:24.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.820 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.820 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:24.820 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091743+0000", 2024-03-27T21:40:24.820 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6234e-05 2024-03-27T21:40:24.820 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.820 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:24.820 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.820 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.820 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:121 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'83, mlcod=17'78)", 2024-03-27T21:40:24.820 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:24.820 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.668613782999998, 2024-03-27T21:40:24.821 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010226899999999999, 2024-03-27T21:40:24.821 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:24.821 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:24.821 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:24.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.821 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:24.821 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:24.821 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.821 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:24.821 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:24.821 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.821 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:24.822 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:24.822 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:24.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.822 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.822 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:24.822 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091142+0000", 2024-03-27T21:40:24.822 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011608300000000001 2024-03-27T21:40:24.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.822 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.823 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:24.823 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091151+0000", 2024-03-27T21:40:24.823 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2350000000000005e-06 2024-03-27T21:40:24.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.823 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.823 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:24.823 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091156+0000", 2024-03-27T21:40:24.823 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5419999999999999e-06 2024-03-27T21:40:24.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.823 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.823 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:24.823 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091191+0000", 2024-03-27T21:40:24.823 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5453000000000001e-05 2024-03-27T21:40:24.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.823 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.824 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:24.824 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091208+0000", 2024-03-27T21:40:24.824 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7119999999999999e-05 2024-03-27T21:40:24.824 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.824 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.824 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:24.824 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092024+0000", 2024-03-27T21:40:24.824 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000815539 2024-03-27T21:40:24.824 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.824 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.824 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:24.824 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092049+0000", 2024-03-27T21:40:24.825 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5171e-05 2024-03-27T21:40:24.825 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.825 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:24.825 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.825 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.825 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.825 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:142 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'100, mlcod=18'98)", 2024-03-27T21:40:24.825 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:24.825 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.253606828999999, 2024-03-27T21:40:24.825 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001198778, 2024-03-27T21:40:24.825 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:24.825 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:24.825 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:24.825 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.826 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:24.826 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:24.826 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.826 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.826 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.826 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:24.826 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:24.826 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.826 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.826 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.826 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:24.827 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:24.827 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:24.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.827 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.827 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:24.827 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506101+0000", 2024-03-27T21:40:24.827 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8202999999999997e-05 2024-03-27T21:40:24.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.827 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.827 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:24.827 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506109+0000", 2024-03-27T21:40:24.827 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3059999999999998e-06 2024-03-27T21:40:24.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.827 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.828 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:24.828 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506113+0000", 2024-03-27T21:40:24.828 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0119999999999999e-06 2024-03-27T21:40:24.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.828 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.828 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:24.828 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506137+0000", 2024-03-27T21:40:24.828 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4043999999999999e-05 2024-03-27T21:40:24.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.828 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.828 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:24.828 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506148+0000", 2024-03-27T21:40:24.828 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0446e-05 2024-03-27T21:40:24.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.829 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.829 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:24.829 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507213+0000", 2024-03-27T21:40:24.829 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001065215 2024-03-27T21:40:24.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.829 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.829 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:24.829 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507232+0000", 2024-03-27T21:40:24.829 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9038e-05 2024-03-27T21:40:24.829 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.829 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:24.829 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.830 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.830 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:143 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'101, mlcod=18'98)", 2024-03-27T21:40:24.830 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:24.830 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.252994209999997, 2024-03-27T21:40:24.830 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001357488, 2024-03-27T21:40:24.830 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:24.830 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:24.830 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:24.830 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.830 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:24.830 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:24.830 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.830 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.831 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:24.831 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:24.831 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.831 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:24.831 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:24.831 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:24.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.831 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:24.831 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506723+0000", 2024-03-27T21:40:24.831 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7404000000000006e-05 2024-03-27T21:40:24.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.832 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.832 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:24.832 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506730+0000", 2024-03-27T21:40:24.832 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5749999999999997e-06 2024-03-27T21:40:24.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.834 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.834 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:24.834 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506733+0000", 2024-03-27T21:40:24.834 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7979999999999999e-06 2024-03-27T21:40:24.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.834 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.834 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:24.834 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506758+0000", 2024-03-27T21:40:24.834 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4179e-05 2024-03-27T21:40:24.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.835 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:24.835 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506768+0000", 2024-03-27T21:40:24.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0740000000000001e-05 2024-03-27T21:40:24.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.835 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:24.835 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507978+0000", 2024-03-27T21:40:24.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012093900000000001 2024-03-27T21:40:24.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.835 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:24.835 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508003+0000", 2024-03-27T21:40:24.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5667e-05 2024-03-27T21:40:24.836 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.836 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:24.836 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.836 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.836 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.836 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:144 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'102, mlcod=18'98)", 2024-03-27T21:40:24.836 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:24.836 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.252335313000003, 2024-03-27T21:40:24.836 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013453390000000001, 2024-03-27T21:40:24.836 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:24.836 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:24.836 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:24.836 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.836 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:24.836 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:24.837 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.837 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.837 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.837 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:24.837 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:24.837 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:24.837 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.837 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.837 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:24.837 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507304+0000", 2024-03-27T21:40:24.837 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:24.837 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.837 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.837 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:24.837 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507397+0000", 2024-03-27T21:40:24.838 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2995e-05 2024-03-27T21:40:24.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.838 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:24.838 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507403+0000", 2024-03-27T21:40:24.838 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8640000000000001e-06 2024-03-27T21:40:24.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.838 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:24.838 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507407+0000", 2024-03-27T21:40:24.838 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5710000000000002e-06 2024-03-27T21:40:24.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.839 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:24.839 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507428+0000", 2024-03-27T21:40:24.839 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1192e-05 2024-03-27T21:40:24.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.839 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.839 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:24.839 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507438+0000", 2024-03-27T21:40:24.839 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8309999999999996e-06 2024-03-27T21:40:24.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.839 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.839 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:24.839 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508624+0000", 2024-03-27T21:40:24.839 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011861210000000001 2024-03-27T21:40:24.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:24.839 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:24.839 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:24.840 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508650+0000", 2024-03-27T21:40:24.840 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6118e-05 2024-03-27T21:40:24.840 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.840 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:24.840 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.840 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:24.840 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:24.840 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:24.842 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:40:24.842 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:24.855+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:24.942 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:24.956+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:25.028 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:25.028 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:25.028 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:25.028 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:25.040 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:40:25.081 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:25.095+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:25.144 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:25.157+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:25.189 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:25.189 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:25.189 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:25.189 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:25.189 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:25.198 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:40:25.245 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:25.258+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:25.306 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:25.306 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:25.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:25.306 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:40:25.306 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:25.315 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:40:25.359 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:25.373+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:25.422 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-27T21:40:25.429 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:25.430 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:25.430 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:25.430 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:25.441 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:40:25.446 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:25.460+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:25.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.541+0000 7fb3d0bfd640 10 monclient: get_monmap_and_config 2024-03-27T21:40:25.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.541+0000 7fb3d0bfd640 10 monclient: build_initial_monmap 2024-03-27T21:40:25.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.541+0000 7fb3d0bfd640 10 monclient: monmap: 2024-03-27T21:40:25.529 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:25.529 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:25.529 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:25.542881+0000 2024-03-27T21:40:25.529 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:25.542881+0000 2024-03-27T21:40:25.529 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:25.529 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:25.529 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:25.530 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:25.530 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:25.530 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:25.530 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:25.530 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:25.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.542+0000 7fb3d0bfd640 1 Processor -- start 2024-03-27T21:40:25.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.542+0000 7fb3d0bfd640 1 -- start start 2024-03-27T21:40:25.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.542+0000 7fb3d0bfd640 10 monclient: init 2024-03-27T21:40:25.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.543+0000 7fb3d0bfd640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:25.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.543+0000 7fb3d0bfd640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-03-27T21:40:25.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.543+0000 7fb3d0bfd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb3cc071a50 0x7fb3cc0a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:25.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.543+0000 7fb3d0bfd640 10 monclient(hunting): picked mon.noname-c con 0x7fb3cc071a50 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:25.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.543+0000 7fb3d0bfd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb3cc13e810 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:25.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.543+0000 7fb3d0bfd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb3cc13ec50 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:25.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.543+0000 7fb3d0bfd640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3cc0a8e90 con 0x7fb3cc13e810 2024-03-27T21:40:25.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.543+0000 7fb3d0bfd640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3cc0aa040 con 0x7fb3cc13ec50 2024-03-27T21:40:25.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.543+0000 7fb3d0bfd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:25.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.543+0000 7fb3d0bfd640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3cc071e30 con 0x7fb3cc071a50 2024-03-27T21:40:25.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.543+0000 7fb3d0bfd640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.543+0000 7fb3d0bfd640 10 monclient(hunting): authenticate will time out at 866.764499s 2024-03-27T21:40:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.543+0000 7fb3c9d9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb3cc13e810 0x7fb3cc141630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58160/0 (socket says 172.21.15.60:58160) 2024-03-27T21:40:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.543+0000 7fb3ca59c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb3cc071a50 0x7fb3cc0a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.543+0000 7fb3c9d9b640 1 -- 172.21.15.60:0/2960513974 learned_addr learned my addr 172.21.15.60:0/2960513974 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:40:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.543+0000 7fb3ca59c640 10 monclient(hunting): get_auth_request con 0x7fb3cc071a50 auth_method 0 2024-03-27T21:40:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.543+0000 7fb3ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.543+0000 7fb3ca59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.543+0000 7fb3ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.543+0000 7fb3ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.544+0000 7fb3ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.544+0000 7fb3ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.544+0000 7fb3ca59c640 10 monclient(hunting): handle_auth_done global_id 4585 payload 274 2024-03-27T21:40:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.544+0000 7fb3ca59c640 1 -- 172.21.15.60:0/2960513974 >> v1:172.21.15.191:6789/0 conn(0x7fb3cc13ec50 legacy=0x7fb3cc141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.544+0000 7fb3ca59c640 1 -- 172.21.15.60:0/2960513974 >> v1:172.21.15.60:6789/0 conn(0x7fb3cc13e810 legacy=0x7fb3cc141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:25.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.544+0000 7fb3ca59c640 10 monclient: _finish_hunting 0 2024-03-27T21:40:25.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.544+0000 7fb3ca59c640 1 monclient: found mon.noname-c 2024-03-27T21:40:25.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.544+0000 7fb3ca59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:25.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.544+0000 7fb3ca59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:40:25.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.544+0000 7fb3ca59c640 1 -- 172.21.15.60:0/2960513974 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3cc0aa2b0 con 0x7fb3cc071a50 2024-03-27T21:40:25.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.545+0000 7fb3ca59c640 1 --2- 172.21.15.60:0/2960513974 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3cc071a50 0x7fb3cc0a87e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc0097a0 tx=0x7fb3bc02b600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a04e74087c343ac server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:25.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.547+0000 7fb3c959a640 1 -- 172.21.15.60:0/2960513974 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc030510 con 0x7fb3cc071a50 2024-03-27T21:40:25.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.547+0000 7fb3c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:25.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.547+0000 7fb3c959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:25.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.547+0000 7fb3c959a640 10 monclient: dump: 2024-03-27T21:40:25.546 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:25.546 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:25.546 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:25.546 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:25.547 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:25.547 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:25.547 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:25.547 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:25.547 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:25.547 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:25.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.547+0000 7fb3c959a640 10 monclient: _finish_auth 0 2024-03-27T21:40:25.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.547+0000 7fb3c959a640 10 monclient: _check_auth_tickets 2024-03-27T21:40:25.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.547+0000 7fb3c959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:25.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.547+0000 7fb3c959a640 1 -- 172.21.15.60:0/2960513974 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3bc02e050 con 0x7fb3cc071a50 2024-03-27T21:40:25.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.547+0000 7fb3c959a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:25.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.547+0000 7fb3c959a640 1 -- 172.21.15.60:0/2960513974 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc030890 con 0x7fb3cc071a50 2024-03-27T21:40:25.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.547+0000 7fb3c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:25.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.547+0000 7fb3c959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:40:25.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.547+0000 7fb3c959a640 10 monclient: dump: 2024-03-27T21:40:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:25.548 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.547+0000 7fb3d0bfd640 5 monclient: authenticate success, global_id 4585 2024-03-27T21:40:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.547+0000 7fb3d0bfd640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.547+0000 7fb3d0bfd640 4 set_mon_vals no callback set 2024-03-27T21:40:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.547+0000 7fb3d0bfd640 10 monclient: shutdown 2024-03-27T21:40:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.547+0000 7fb3d0bfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.547+0000 7fb3d0bfd640 1 -- 172.21.15.60:0/2960513974 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3cc071a50 msgr2=0x7fb3cc0a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.547+0000 7fb3d0bfd640 1 --2- 172.21.15.60:0/2960513974 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3cc071a50 0x7fb3cc0a87e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc0097a0 tx=0x7fb3bc02b600 comp rx=0 tx=0).stop 2024-03-27T21:40:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.548+0000 7fb3d0bfd640 1 -- 172.21.15.60:0/2960513974 shutdown_connections 2024-03-27T21:40:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.548+0000 7fb3d0bfd640 1 --2- 172.21.15.60:0/2960513974 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3cc071a50 0x7fb3cc0a87e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.548+0000 7fb3d0bfd640 1 -- 172.21.15.60:0/2960513974 >> 172.21.15.60:0/2960513974 conn(0x7fb3cc0b7300 msgr2=0x7fb3cc0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.548+0000 7fb3d0bfd640 1 -- 172.21.15.60:0/2960513974 shutdown_connections 2024-03-27T21:40:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.548+0000 7fb3d0bfd640 1 -- 172.21.15.60:0/2960513974 wait complete. 2024-03-27T21:40:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.548+0000 7fb3d0bfd640 10 monclient: build_initial_monmap 2024-03-27T21:40:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.549+0000 7fb3d0bfd640 10 monclient: monmap: 2024-03-27T21:40:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:25.550 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.549+0000 7fb3d0bfd640 1 librados: starting msgr at 2024-03-27T21:40:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.549+0000 7fb3d0bfd640 1 librados: starting objecter 2024-03-27T21:40:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.549+0000 7fb3d0bfd640 1 Processor -- start 2024-03-27T21:40:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.549+0000 7fb3d0bfd640 1 -- start start 2024-03-27T21:40:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.549+0000 7fb3d0bfd640 1 librados: setting wanted keys 2024-03-27T21:40:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.549+0000 7fb3d0bfd640 1 librados: calling monclient init 2024-03-27T21:40:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.549+0000 7fb3d0bfd640 10 monclient: init 2024-03-27T21:40:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3d0bfd640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3d0bfd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:40:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3d0bfd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3cc071a50 0x7fb3cc1387a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3d0bfd640 10 monclient(hunting): picked mon.noname-b con 0x7fb3cc071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3d0bfd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb3cc13e810 0x7fb3cc139160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3d0bfd640 10 monclient(hunting): picked mon.noname-c con 0x7fb3cc13e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3d0bfd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3cc13ec50 0x7fb3cc1396a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3d0bfd640 10 monclient(hunting): picked mon.noname-a con 0x7fb3cc13ec50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3d0bfd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3d0bfd640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc138490 con 0x7fb3cc13ec50 2024-03-27T21:40:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3d0bfd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3d0bfd640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc137460 con 0x7fb3cc071a50 2024-03-27T21:40:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3d0bfd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3d0bfd640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc138ce0 con 0x7fb3cc13e810 2024-03-27T21:40:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3d0bfd640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3d0bfd640 10 monclient(hunting): authenticate will time out at 866.771101s 2024-03-27T21:40:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3cad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3cc13ec50 0x7fb3cc1396a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:25.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3cad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3cc13ec50 0x7fb3cc1396a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40342/0 (socket says 172.21.15.60:40342) 2024-03-27T21:40:25.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3cad9d640 1 -- 172.21.15.60:0/1976441434 learned_addr learned my addr 172.21.15.60:0/1976441434 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:25.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3cad9d640 10 monclient(hunting): get_auth_request con 0x7fb3cc13ec50 auth_method 0 2024-03-27T21:40:25.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:25.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3cad9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:25.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:25.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3ca59c640 1 --2- 172.21.15.60:0/1976441434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3cc071a50 0x7fb3cc1387a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:25.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3ca59c640 10 monclient(hunting): get_auth_request con 0x7fb3cc071a50 auth_method 0 2024-03-27T21:40:25.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:25.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3ca59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:25.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:25.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:25.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:25.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3c9d9b640 1 --2- 172.21.15.60:0/1976441434 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb3cc13e810 0x7fb3cc139160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:25.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:25.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:25.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:25.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:25.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3cad9d640 10 monclient(hunting): handle_auth_done global_id 4629 payload 824 2024-03-27T21:40:25.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3cad9d640 1 -- 172.21.15.60:0/1976441434 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb3cc13e810 msgr2=0x7fb3cc139160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:25.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3cad9d640 1 --2- 172.21.15.60:0/1976441434 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb3cc13e810 0x7fb3cc139160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:25.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3cad9d640 1 -- 172.21.15.60:0/1976441434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3cc071a50 msgr2=0x7fb3cc1387a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:25.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3cad9d640 1 --2- 172.21.15.60:0/1976441434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3cc071a50 0x7fb3cc1387a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:25.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3cad9d640 10 monclient: _finish_hunting 0 2024-03-27T21:40:25.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3cad9d640 1 monclient: found mon.noname-a 2024-03-27T21:40:25.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3cad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:25.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3cad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:25.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.550+0000 7fb3cad9d640 1 -- 172.21.15.60:0/1976441434 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc009400 con 0x7fb3cc13ec50 2024-03-27T21:40:25.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3ca59c640 1 --2- 172.21.15.60:0/1976441434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3cc071a50 0x7fb3cc1387a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:25.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3c9d9b640 10 monclient: get_auth_request con 0x7fb3cc13e810 auth_method 0 2024-03-27T21:40:25.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3c9d9b640 1 --2- 172.21.15.60:0/1976441434 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb3cc13e810 0x7fb3cc139160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:40:25.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3cad9d640 1 --2- 172.21.15.60:0/1976441434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3cc13ec50 0x7fb3cc1396a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400c7f0 tx=0x7fb3c400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:25.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3baffd640 10 client.?.objecter ms_handle_connect 0x7fb3cc13ec50 2024-03-27T21:40:25.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3baffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:25.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3baffd640 1 -- 172.21.15.60:0/1976441434 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4003da0 con 0x7fb3cc13ec50 2024-03-27T21:40:25.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:25.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:25.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3baffd640 10 monclient: dump: 2024-03-27T21:40:25.554 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:25.554 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:25.554 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:25.554 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:25.554 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:25.555 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:25.555 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:25.555 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:25.555 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:25.555 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:25.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3baffd640 10 monclient: _finish_auth 0 2024-03-27T21:40:25.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3baffd640 10 monclient: _check_auth_tickets 2024-03-27T21:40:25.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:25.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3d0bfd640 5 monclient: authenticate success, global_id 4629 2024-03-27T21:40:25.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3baffd640 1 -- 172.21.15.60:0/1976441434 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c4005070 con 0x7fb3cc13ec50 2024-03-27T21:40:25.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3d0bfd640 10 monclient: _renew_subs 2024-03-27T21:40:25.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3d0bfd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:25.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3d0bfd640 1 -- 172.21.15.60:0/1976441434 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3cc1378f0 con 0x7fb3cc13ec50 2024-03-27T21:40:25.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:25.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3baffd640 1 -- 172.21.15.60:0/1976441434 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4005420 con 0x7fb3cc13ec50 2024-03-27T21:40:25.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:25.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:25.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3c8d99640 4 set_mon_vals no callback set 2024-03-27T21:40:25.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.551+0000 7fb3baffd640 10 monclient: dump: 2024-03-27T21:40:25.556 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:25.556 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:25.556 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:25.556 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:25.556 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:25.556 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:25.556 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:25.556 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:25.556 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:25.556 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:25.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.552+0000 7fb3d0bfd640 10 client.4629.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:25.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.552+0000 7fb3d0bfd640 10 monclient: _renew_subs 2024-03-27T21:40:25.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.552+0000 7fb3d0bfd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:25.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.552+0000 7fb3d0bfd640 1 -- 172.21.15.60:0/1976441434 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3cc139be0 con 0x7fb3cc13ec50 2024-03-27T21:40:25.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.552+0000 7fb3d0bfd640 1 librados: init done 2024-03-27T21:40:25.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.553+0000 7fb3d0bfd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:25.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.553+0000 7fb3d0bfd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:25.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.553+0000 7fb3d0bfd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:25.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.553+0000 7fb3d0bfd640 1 -- 172.21.15.60:0/1976441434 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3cc1375d0 con 0x7fb3cc13ec50 2024-03-27T21:40:25.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.553+0000 7fb3baffd640 1 -- 172.21.15.60:0/1976441434 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb3c4005580 con 0x7fb3cc13ec50 2024-03-27T21:40:25.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.554+0000 7fb3baffd640 1 --2- 172.21.15.60:0/1976441434 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb3ac06c5c0 0x7fb3ac06ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:25.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.554+0000 7fb3baffd640 1 -- 172.21.15.60:0/1976441434 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fb3c40887b0 con 0x7fb3cc13ec50 2024-03-27T21:40:25.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.554+0000 7fb3baffd640 10 client.4629.objecter ms_dispatch 0x7fb3cc000b90 osd_map(35..35 src has 1..35) v4 2024-03-27T21:40:25.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.554+0000 7fb3baffd640 3 client.4629.objecter handle_osd_map got epochs [35,35] > 0 2024-03-27T21:40:25.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.554+0000 7fb3baffd640 3 client.4629.objecter handle_osd_map decoding full epoch 35 2024-03-27T21:40:25.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.554+0000 7fb3baffd640 20 client.4629.objecter dump_active .. 0 homeless 2024-03-27T21:40:25.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.554+0000 7fb3ca59c640 1 --2- 172.21.15.60:0/1976441434 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb3ac06c5c0 0x7fb3ac06ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:25.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.554+0000 7fb3ca59c640 10 monclient: get_auth_request con 0x7fb3ac06c5c0 auth_method 0 2024-03-27T21:40:25.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.555+0000 7fb3ca59c640 1 --2- 172.21.15.60:0/1976441434 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb3ac06c5c0 0x7fb3ac06ea80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc034000 tx=0x7fb3bc008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:25.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.555+0000 7fb3baffd640 10 client.4629.objecter ms_handle_connect 0x7fb3ac06c5c0 2024-03-27T21:40:25.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.559+0000 7fb3baffd640 1 -- 172.21.15.60:0/1976441434 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb3c4053b80 con 0x7fb3cc13ec50 2024-03-27T21:40:25.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.559+0000 7fb3baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:25.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.559+0000 7fb3baffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:25.559 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:25.573+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:25.598 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:25.598 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:25.598 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:25.598 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:25.598 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:25.607 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:40:25.660 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:25.674+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:25.761 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:25.775+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:25.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.782+0000 7fb3d0bfd640 10 start_mon_command cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:40:25.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.782+0000 7fb3d0bfd640 10 monclient: _send_command 2 [{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:40:25.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.782+0000 7fb3d0bfd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:25.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:25.782+0000 7fb3d0bfd640 1 -- 172.21.15.60:0/1976441434 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fb3cc069470 con 0x7fb3cc13ec50 2024-03-27T21:40:25.777 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:25.777 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:25.777 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:25.777 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:40:25.777 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:25.789 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:40:25.862 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:25.875+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:25.907 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:25.907 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:25.907 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:25.907 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:25.916 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:40:25.963 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:25.977+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:26.063 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:26.063 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:26.064 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:26.064 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:26.064 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:26.064 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:26.077+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:26.072 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:40:26.098 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:40:26.118 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-03-27T21:40:26.118 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T21:40:26.118 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-27T21:40:26.118 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 1.000000 2024-03-27T21:40:26.119 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-27T21:40:26.165 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:26.178+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:26.216 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:26.217 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:26.225 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:40:26.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.262+0000 7f1f2f7ce640 10 monclient: get_monmap_and_config 2024-03-27T21:40:26.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.262+0000 7f1f2f7ce640 10 monclient: build_initial_monmap 2024-03-27T21:40:26.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.262+0000 7f1f2f7ce640 10 monclient: monmap: 2024-03-27T21:40:26.249 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:26.249 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:26.249 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:26.263617+0000 2024-03-27T21:40:26.249 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:26.263617+0000 2024-03-27T21:40:26.249 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:26.249 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:26.249 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:26.249 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:26.249 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:26.249 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:26.249 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:26.249 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:26.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.263+0000 7f1f2f7ce640 1 Processor -- start 2024-03-27T21:40:26.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.263+0000 7f1f2f7ce640 1 -- start start 2024-03-27T21:40:26.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.263+0000 7f1f2f7ce640 10 monclient: init 2024-03-27T21:40:26.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2f7ce640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:26.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2f7ce640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-03-27T21:40:26.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2f7ce640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1f280b40b0 0x7f1f280a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:26.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2f7ce640 10 monclient(hunting): picked mon.noname-a con 0x7f1f280b40b0 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:26.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2f7ce640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1f2813b020 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:26.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2f7ce640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1f2813b480 0x7f1f28141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:26.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2f7ce640 10 monclient(hunting): picked mon.noname-c con 0x7f1f2813b480 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:26.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2f7ce640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2813b8d0 con 0x7f1f2813b020 2024-03-27T21:40:26.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2f7ce640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:26.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2f7ce640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f280b4490 con 0x7f1f280b40b0 2024-03-27T21:40:26.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2f7ce640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:26.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2f7ce640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f28142340 con 0x7f1f2813b480 2024-03-27T21:40:26.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2f7ce640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:26.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2f7ce640 10 monclient(hunting): authenticate will time out at 867.485570s 2024-03-27T21:40:26.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2d56a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1f280b40b0 0x7f1f280a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:26.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2d56a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1f280b40b0 0x7f1f280a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40350/0 (socket says 172.21.15.60:40350) 2024-03-27T21:40:26.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2d56a640 1 -- 172.21.15.60:0/3757699058 learned_addr learned my addr 172.21.15.60:0/3757699058 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:26.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2d56a640 10 monclient(hunting): get_auth_request con 0x7f1f280b40b0 auth_method 0 2024-03-27T21:40:26.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2d56a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:26.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2d56a640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:26.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2d56a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:26.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2d56a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:26.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2d56a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:26.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.264+0000 7f1f2d56a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:26.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.265+0000 7f1f2d56a640 10 monclient(hunting): handle_auth_done global_id 4632 payload 274 2024-03-27T21:40:26.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.265+0000 7f1f2d56a640 1 -- 172.21.15.60:0/3757699058 >> v2:172.21.15.191:3300/0 conn(0x7f1f2813b480 msgr2=0x7f1f28141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:26.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.265+0000 7f1f2d56a640 1 --2- 172.21.15.60:0/3757699058 >> v2:172.21.15.191:3300/0 conn(0x7f1f2813b480 0x7f1f28141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.265+0000 7f1f2d56a640 1 -- 172.21.15.60:0/3757699058 >> v1:172.21.15.191:6789/0 conn(0x7f1f2813b020 legacy=0x7f1f281416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:26.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.265+0000 7f1f2d56a640 10 monclient: _finish_hunting 0 2024-03-27T21:40:26.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.265+0000 7f1f2d56a640 1 monclient: found mon.noname-a 2024-03-27T21:40:26.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.265+0000 7f1f2d56a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:26.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.265+0000 7f1f2d56a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:26.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.265+0000 7f1f2d56a640 1 -- 172.21.15.60:0/3757699058 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f280b3830 con 0x7f1f280b40b0 2024-03-27T21:40:26.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.265+0000 7f1f2d56a640 1 --2- 172.21.15.60:0/3757699058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f280b40b0 0x7f1f280a8b90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1f280b3a20 tx=0x7f1f2402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32dba76a1a2a5a1b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:26.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f1ffff640 1 -- 172.21.15.60:0/3757699058 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2402c9e0 con 0x7f1f280b40b0 2024-03-27T21:40:26.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:26.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f1ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:26.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f1ffff640 10 monclient: dump: 2024-03-27T21:40:26.261 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:26.262 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:26.262 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:26.262 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:26.262 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:26.262 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:26.262 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:26.262 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:26.262 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:26.262 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:26.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f1ffff640 10 monclient: _finish_auth 0 2024-03-27T21:40:26.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f1ffff640 10 monclient: _check_auth_tickets 2024-03-27T21:40:26.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:26.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f1ffff640 1 -- 172.21.15.60:0/3757699058 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2402cb40 con 0x7f1f280b40b0 2024-03-27T21:40:26.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f2f7ce640 5 monclient: authenticate success, global_id 4632 2024-03-27T21:40:26.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f2f7ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:26.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f1ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:26.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f1ffff640 1 -- 172.21.15.60:0/3757699058 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f240343f0 con 0x7f1f280b40b0 2024-03-27T21:40:26.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:26.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f1ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:26.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f1ffff640 10 monclient: dump: 2024-03-27T21:40:26.263 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:26.263 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:26.263 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:26.263 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:26.263 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:26.263 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:26.263 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:26.263 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:26.264 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:26.264 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:26.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f2f7ce640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:26.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f2f7ce640 4 set_mon_vals no callback set 2024-03-27T21:40:26.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f2f7ce640 10 monclient: shutdown 2024-03-27T21:40:26.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f2f7ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:26.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3757699058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f280b40b0 msgr2=0x7f1f280a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:26.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3757699058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f280b40b0 0x7f1f280a8b90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1f280b3a20 tx=0x7f1f2402be40 comp rx=0 tx=0).stop 2024-03-27T21:40:26.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3757699058 shutdown_connections 2024-03-27T21:40:26.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.266+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3757699058 >> v2:172.21.15.191:3300/0 conn(0x7f1f2813b480 0x7f1f28141e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.267+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3757699058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f280b40b0 0x7f1f280a8b90 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.267+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3757699058 >> 172.21.15.60:0/3757699058 conn(0x7f1f280608e0 msgr2=0x7f1f28060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:26.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.267+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3757699058 shutdown_connections 2024-03-27T21:40:26.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.267+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3757699058 wait complete. 2024-03-27T21:40:26.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.268+0000 7f1f2f7ce640 10 monclient: build_initial_monmap 2024-03-27T21:40:26.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.268+0000 7f1f2f7ce640 10 monclient: monmap: 2024-03-27T21:40:26.265 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:26.265 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:26.265 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:26.265 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:26.265 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:26.265 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:26.265 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:26.265 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:26.265 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:26.265 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:26.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.268+0000 7f1f2f7ce640 1 librados: starting msgr at 2024-03-27T21:40:26.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.268+0000 7f1f2f7ce640 1 librados: starting objecter 2024-03-27T21:40:26.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.268+0000 7f1f2f7ce640 1 Processor -- start 2024-03-27T21:40:26.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.268+0000 7f1f2f7ce640 1 -- start start 2024-03-27T21:40:26.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.268+0000 7f1f2f7ce640 1 librados: setting wanted keys 2024-03-27T21:40:26.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.268+0000 7f1f2f7ce640 1 librados: calling monclient init 2024-03-27T21:40:26.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.268+0000 7f1f2f7ce640 10 monclient: init 2024-03-27T21:40:26.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2f7ce640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:26.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2f7ce640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:40:26.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2f7ce640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1f2813b020 0x7f1f281336f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:26.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2f7ce640 10 monclient(hunting): picked mon.noname-c con 0x7f1f2813b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:26.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2f7ce640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f2813b480 0x7f1f2813ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:26.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2f7ce640 10 monclient(hunting): picked mon.noname-a con 0x7f1f2813b480 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:26.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2f7ce640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f281312a0 0x7f1f28131730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:26.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2f7ce640 10 monclient(hunting): picked mon.noname-b con 0x7f1f281312a0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:26.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2d56a640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1f2813b020 0x7f1f281336f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:26.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2dd6b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f281312a0 0x7f1f28131730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:26.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2f7ce640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:26.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2d56a640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1f2813b020 0x7f1f281336f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47550/0 (socket says 172.21.15.60:47550) 2024-03-27T21:40:26.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2f7ce640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2813ab60 con 0x7f1f2813b480 2024-03-27T21:40:26.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2d56a640 1 -- 172.21.15.60:0/3646919696 learned_addr learned my addr 172.21.15.60:0/3646919696 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:26.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2f7ce640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:26.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2813a6f0 con 0x7f1f281312a0 2024-03-27T21:40:26.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2f7ce640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:26.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f28131c70 con 0x7f1f2813b020 2024-03-27T21:40:26.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2f7ce640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:26.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2f7ce640 10 monclient(hunting): authenticate will time out at 867.490737s 2024-03-27T21:40:26.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2cd69640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f2813b480 0x7f1f2813ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:26.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2d56a640 10 monclient(hunting): get_auth_request con 0x7f1f2813b020 auth_method 0 2024-03-27T21:40:26.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2d56a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:26.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2d56a640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:26.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2d56a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:26.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2cd69640 10 monclient(hunting): get_auth_request con 0x7f1f2813b480 auth_method 0 2024-03-27T21:40:26.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2cd69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:26.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2cd69640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:26.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2cd69640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:26.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2d56a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:26.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2d56a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:26.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.269+0000 7f1f2d56a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:26.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f2d56a640 10 monclient(hunting): handle_auth_done global_id 4511 payload 824 2024-03-27T21:40:26.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f2d56a640 1 -- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f281312a0 msgr2=0x7f1f28131730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:26.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f2d56a640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f281312a0 0x7f1f28131730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f2d56a640 1 -- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f2813b480 msgr2=0x7f1f2813ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:26.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f2d56a640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f2813b480 0x7f1f2813ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f2d56a640 10 monclient: _finish_hunting 0 2024-03-27T21:40:26.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f2d56a640 1 monclient: found mon.noname-c 2024-03-27T21:40:26.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.271+0000 7f1bb5fd5640 10 monclient: get_monmap_and_config 2024-03-27T21:40:26.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.271+0000 7f1bb5fd5640 10 monclient: build_initial_monmap 2024-03-27T21:40:26.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.271+0000 7f1bb5fd5640 10 monclient: monmap: 2024-03-27T21:40:26.269 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:26.270 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:26.270 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:26.272224+0000 2024-03-27T21:40:26.270 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:26.272224+0000 2024-03-27T21:40:26.270 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:26.270 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:26.270 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:26.270 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:26.270 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:26.270 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:26.270 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:26.270 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:26.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f2d56a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:26.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f2d56a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:40:26.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f2d56a640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f24009b30 con 0x7f1f2813b020 2024-03-27T21:40:26.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f2d56a640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1f2813b020 0x7f1f281336f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1f24036040 tx=0x7f1f24004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:26.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f1dffb640 10 client.?.objecter ms_handle_connect 0x7f1f2813b020 2024-03-27T21:40:26.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f1dffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:26.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2403e450 con 0x7f1f2813b020 2024-03-27T21:40:26.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:26.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f1dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:26.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f1dffb640 10 monclient: dump: 2024-03-27T21:40:26.272 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:26.272 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:26.272 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:26.272 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:26.272 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:26.272 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:26.272 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:26.272 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:26.272 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:26.272 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:26.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f1dffb640 10 monclient: _finish_auth 0 2024-03-27T21:40:26.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f1dffb640 10 monclient: _check_auth_tickets 2024-03-27T21:40:26.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:26.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2403e5b0 con 0x7f1f2813b020 2024-03-27T21:40:26.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f1dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:26.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f1f7fe640 4 set_mon_vals no callback set 2024-03-27T21:40:26.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2403e920 con 0x7f1f2813b020 2024-03-27T21:40:26.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:26.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f1dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:26.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f1dffb640 10 monclient: dump: 2024-03-27T21:40:26.273 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:26.273 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:26.274 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f2f7ce640 5 monclient: authenticate success, global_id 4511 2024-03-27T21:40:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f2f7ce640 10 monclient: _renew_subs 2024-03-27T21:40:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f2f7ce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f2813bfa0 con 0x7f1f2813b020 2024-03-27T21:40:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f2f7ce640 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:26.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f2f7ce640 10 monclient: _renew_subs 2024-03-27T21:40:26.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f2f7ce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:26.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2813de00 con 0x7f1f2813b020 2024-03-27T21:40:26.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.270+0000 7f1f2f7ce640 1 librados: init done 2024-03-27T21:40:26.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.271+0000 7f1f2f7ce640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:26.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.271+0000 7f1f2f7ce640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:26.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.271+0000 7f1f2f7ce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:26.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.271+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f1f2813d660 con 0x7f1f2813b020 2024-03-27T21:40:26.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.272+0000 7f1bb5fd5640 1 Processor -- start 2024-03-27T21:40:26.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.272+0000 7f1bb5fd5640 1 -- start start 2024-03-27T21:40:26.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.272+0000 7f1bb5fd5640 10 monclient: init 2024-03-27T21:40:26.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.272+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1f2403ea80 con 0x7f1f2813b020 2024-03-27T21:40:26.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f1dffb640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1f0c074da0 0x7f1f0c077260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:26.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f1f240ac500 con 0x7f1f2813b020 2024-03-27T21:40:26.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f1dffb640 10 client.4511.objecter ms_dispatch 0x7f1f28000b90 osd_map(35..35 src has 1..35) v4 2024-03-27T21:40:26.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1bb5fd5640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:26.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f1dffb640 3 client.4511.objecter handle_osd_map got epochs [35,35] > 0 2024-03-27T21:40:26.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f1dffb640 3 client.4511.objecter handle_osd_map decoding full epoch 35 2024-03-27T21:40:26.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f1dffb640 20 client.4511.objecter dump_active .. 0 homeless 2024-03-27T21:40:26.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v35) v1 ==== 54+0+16 (secure 0 0 0) 0x7f1f240ac8b0 con 0x7f1f2813b020 2024-03-27T21:40:26.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f1dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:26.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f1dffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:26.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1bb5fd5640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-03-27T21:40:26.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1bb5fd5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1bb00b40b0 0x7f1bb00a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:26.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1bb5fd5640 10 monclient(hunting): picked mon.noname-a con 0x7f1bb00b40b0 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:26.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1bb5fd5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1bb013b020 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:26.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1bb5fd5640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1bb013b480 0x7f1bb0141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:26.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1bb5fd5640 10 monclient(hunting): picked mon.noname-b con 0x7f1bb013b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:26.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1bb5fd5640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb00b3830 con 0x7f1bb013b020 2024-03-27T21:40:26.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1bb5fd5640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:26.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1bb5fd5640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bb013b910 con 0x7f1bb00b40b0 2024-03-27T21:40:26.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1bb5fd5640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:26.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1bb5fd5640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb00b4490 con 0x7f1bb013b480 2024-03-27T21:40:26.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1bb5fd5640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:26.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1bb5fd5640 10 monclient(hunting): authenticate will time out at 867.494196s 2024-03-27T21:40:26.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baf7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1bb00b40b0 0x7f1bb00a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:26.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baf7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1bb00b40b0 0x7f1bb00a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40374/0 (socket says 172.21.15.60:40374) 2024-03-27T21:40:26.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baf7fe640 1 -- 172.21.15.60:0/134804363 learned_addr learned my addr 172.21.15.60:0/134804363 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:26.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baf7fe640 10 monclient(hunting): get_auth_request con 0x7f1bb00b40b0 auth_method 0 2024-03-27T21:40:26.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:26.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:26.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:26.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baffff640 1 --2- 172.21.15.60:0/134804363 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1bb013b480 0x7f1bb0141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:26.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baffff640 10 monclient(hunting): get_auth_request con 0x7f1bb013b480 auth_method 0 2024-03-27T21:40:26.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:26.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:26.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:26.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:26.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:26.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f2f7ce640 10 client.4511.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:26.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f2cd69640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1f0c074da0 0x7f1f0c077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:26.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign -1 1 2024-03-27T21:40:26.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f1f2813e5f0 0x7f1f28138a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:26.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f2cd69640 10 monclient: get_auth_request con 0x7f1f0c074da0 auth_method 0 2024-03-27T21:40:26.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f280a90d0 osd=0 3 2024-03-27T21:40:26.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f280a90d0 osd=0 3 2024-03-27T21:40:26.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f280a90d0 osd=0 3 2024-03-27T21:40:26.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_remove -1 1 2024-03-27T21:40:26.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f2f7ce640 20 client.4511.objecter get_session s=0x7f1f280a90d0 osd=0 3 2024-03-27T21:40:26.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign 0 1 2024-03-27T21:40:26.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f280a90d0 osd=0 4 2024-03-27T21:40:26.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f2f7ce640 10 client.4511.objecter _send_command 1 2024-03-27T21:40:26.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1f280a9470 con 0x7f1f2813e5f0 2024-03-27T21:40:26.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f2dd6b640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f1f2813e5f0 0x7f1f28138a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:26.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:26.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:26.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:26.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:26.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1f2dd6b640 10 monclient: get_auth_request con 0x7f1f2813e5f0 auth_method 0 2024-03-27T21:40:26.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baf7fe640 10 monclient(hunting): handle_auth_done global_id 4638 payload 274 2024-03-27T21:40:26.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baf7fe640 1 -- 172.21.15.60:0/134804363 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1bb013b480 msgr2=0x7f1bb0141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:26.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baf7fe640 1 --2- 172.21.15.60:0/134804363 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1bb013b480 0x7f1bb0141e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.274+0000 7f1f1dffb640 10 client.4511.objecter ms_handle_connect 0x7f1f2813e5f0 2024-03-27T21:40:26.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baf7fe640 1 -- 172.21.15.60:0/134804363 >> v1:172.21.15.60:6789/0 conn(0x7f1bb013b020 legacy=0x7f1bb01416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:26.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baf7fe640 10 monclient: _finish_hunting 0 2024-03-27T21:40:26.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baf7fe640 1 monclient: found mon.noname-a 2024-03-27T21:40:26.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:26.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:26.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.273+0000 7f1baf7fe640 1 -- 172.21.15.60:0/134804363 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb00aa520 con 0x7f1bb00b40b0 2024-03-27T21:40:26.286 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:26.292+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:26.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.274+0000 7f1f2dd6b640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f1f2813e5f0 0x7f1f28138a70 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:26.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.274+0000 7f1f2cd69640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1f0c074da0 0x7f1f0c077260 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1f18007fd0 tx=0x7f1f18009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:26.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.274+0000 7f1f1dffb640 10 client.4511.objecter ms_handle_connect 0x7f1f0c074da0 2024-03-27T21:40:26.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.274+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f1f2000a2f0 con 0x7f1f2813e5f0 2024-03-27T21:40:26.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.274+0000 7f1f1dffb640 10 client.4511.objecter ms_dispatch 0x7f1f28000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:26.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.274+0000 7f1f1dffb640 10 client.4511.objecter _finish_command 1 = system:0 2024-03-27T21:40:26.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.274+0000 7f1f1dffb640 20 client.4511.objecter put_session s=0x7f1f280a90d0 osd=0 4 2024-03-27T21:40:26.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.274+0000 7f1f1dffb640 15 client.4511.objecter _session_command_op_remove 0 1 2024-03-27T21:40:26.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.275+0000 7f1baf7fe640 1 --2- 172.21.15.60:0/134804363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bb00b40b0 0x7f1bb00a8b90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1ba00098a0 tx=0x7f1ba002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=323687c06b3b75f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:26.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.275+0000 7f1bae7fc640 1 -- 172.21.15.60:0/134804363 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0031640 con 0x7f1bb00b40b0 2024-03-27T21:40:26.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.275+0000 7f1bae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:26.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.275+0000 7f1bae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:26.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.275+0000 7f1bae7fc640 10 monclient: dump: 2024-03-27T21:40:26.294 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:26.294 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:26.294 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:26.294 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:26.295 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:26.295 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:26.295 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:26.295 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:26.295 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:26.295 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:26.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.275+0000 7f1bae7fc640 10 monclient: _finish_auth 0 2024-03-27T21:40:26.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.275+0000 7f1bae7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:40:26.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.275+0000 7f1bae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:26.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.275+0000 7f1bae7fc640 1 -- 172.21.15.60:0/134804363 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba00317a0 con 0x7f1bb00b40b0 2024-03-27T21:40:26.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.275+0000 7f1bb5fd5640 5 monclient: authenticate success, global_id 4638 2024-03-27T21:40:26.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.275+0000 7f1bb5fd5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:26.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.275+0000 7f1bae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:26.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.275+0000 7f1bb5fd5640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:26.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.275+0000 7f1bae7fc640 1 -- 172.21.15.60:0/134804363 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0031b30 con 0x7f1bb00b40b0 2024-03-27T21:40:26.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.275+0000 7f1bb5fd5640 4 set_mon_vals no callback set 2024-03-27T21:40:26.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.275+0000 7f1bae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:26.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.275+0000 7f1bae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:26.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.275+0000 7f1bae7fc640 10 monclient: dump: 2024-03-27T21:40:26.296 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:26.296 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:26.296 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:26.296 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:26.296 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:26.296 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:26.296 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:26.296 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:26.297 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:26.297 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:26.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.276+0000 7f1bb5fd5640 10 monclient: shutdown 2024-03-27T21:40:26.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.276+0000 7f1bb5fd5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:26.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.276+0000 7f1bb5fd5640 1 -- 172.21.15.60:0/134804363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bb00b40b0 msgr2=0x7f1bb00a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:26.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.276+0000 7f1bb5fd5640 1 --2- 172.21.15.60:0/134804363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bb00b40b0 0x7f1bb00a8b90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1ba00098a0 tx=0x7f1ba002b8f0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.276+0000 7f1bb5fd5640 1 -- 172.21.15.60:0/134804363 shutdown_connections 2024-03-27T21:40:26.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.276+0000 7f1bb5fd5640 1 --2- 172.21.15.60:0/134804363 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1bb013b480 0x7f1bb0141e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.276+0000 7f1bb5fd5640 1 --2- 172.21.15.60:0/134804363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bb00b40b0 0x7f1bb00a8b90 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.276+0000 7f1bb5fd5640 1 -- 172.21.15.60:0/134804363 >> 172.21.15.60:0/134804363 conn(0x7f1bb00608e0 msgr2=0x7f1bb0060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:26.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.276+0000 7f1bb5fd5640 1 -- 172.21.15.60:0/134804363 shutdown_connections 2024-03-27T21:40:26.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.276+0000 7f1bb5fd5640 1 -- 172.21.15.60:0/134804363 wait complete. 2024-03-27T21:40:26.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.277+0000 7f1bb5fd5640 10 monclient: build_initial_monmap 2024-03-27T21:40:26.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.277+0000 7f1bb5fd5640 10 monclient: monmap: 2024-03-27T21:40:26.298 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:26.298 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:26.298 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:26.298 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:26.298 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:26.298 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:26.298 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:26.298 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:26.298 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:26.298 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:26.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.277+0000 7f1bb5fd5640 1 librados: starting msgr at 2024-03-27T21:40:26.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.277+0000 7f1bb5fd5640 1 librados: starting objecter 2024-03-27T21:40:26.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.277+0000 7f1bb5fd5640 1 Processor -- start 2024-03-27T21:40:26.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.278+0000 7f1bb5fd5640 1 -- start start 2024-03-27T21:40:26.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.278+0000 7f1bb5fd5640 1 librados: setting wanted keys 2024-03-27T21:40:26.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.278+0000 7f1bb5fd5640 1 librados: calling monclient init 2024-03-27T21:40:26.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.278+0000 7f1bb5fd5640 10 monclient: init 2024-03-27T21:40:26.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.278+0000 7f1bb5fd5640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:26.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.278+0000 7f1bb5fd5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:40:26.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.278+0000 7f1bb5fd5640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1bb00b40b0 0x7f1bb013aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:26.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.278+0000 7f1bb5fd5640 10 monclient(hunting): picked mon.noname-c con 0x7f1bb00b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:26.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.278+0000 7f1bb5fd5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bb013b020 0x7f1bb01311a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:26.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.278+0000 7f1bb5fd5640 10 monclient(hunting): picked mon.noname-a con 0x7f1bb013b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:26.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.278+0000 7f1bb5fd5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bb013b480 0x7f1bb01316e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:26.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1bb5fd5640 10 monclient(hunting): picked mon.noname-b con 0x7f1bb013b480 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:26.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1bb5fd5640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:26.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baf7fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1bb00b40b0 0x7f1bb013aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:26.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1bb5fd5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb013beb0 con 0x7f1bb013b020 2024-03-27T21:40:26.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baf7fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1bb00b40b0 0x7f1bb013aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47558/0 (socket says 172.21.15.60:47558) 2024-03-27T21:40:26.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1bb5fd5640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:26.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baf7fe640 1 -- 172.21.15.60:0/2824031251 learned_addr learned my addr 172.21.15.60:0/2824031251 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:26.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1bb5fd5640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb0131c20 con 0x7f1bb013b480 2024-03-27T21:40:26.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1bb5fd5640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:26.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1bb5fd5640 1 -- 172.21.15.60:0/2824031251 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb013c2d0 con 0x7f1bb00b40b0 2024-03-27T21:40:26.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baeffd640 1 --2- 172.21.15.60:0/2824031251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bb013b020 0x7f1bb01311a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:26.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1bb5fd5640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:26.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baffff640 1 --2- 172.21.15.60:0/2824031251 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bb013b480 0x7f1bb01316e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:26.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1bb5fd5640 10 monclient(hunting): authenticate will time out at 867.500234s 2024-03-27T21:40:26.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baf7fe640 10 monclient(hunting): get_auth_request con 0x7f1bb00b40b0 auth_method 0 2024-03-27T21:40:26.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:26.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:26.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:26.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baeffd640 10 monclient(hunting): get_auth_request con 0x7f1bb013b020 auth_method 0 2024-03-27T21:40:26.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:26.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baeffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:26.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:26.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baffff640 10 monclient(hunting): get_auth_request con 0x7f1bb013b480 auth_method 0 2024-03-27T21:40:26.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:26.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:26.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:26.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:26.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:26.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:26.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:26.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:26.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:26.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:26.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:26.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:26.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baffff640 10 monclient(hunting): handle_auth_done global_id 4594 payload 824 2024-03-27T21:40:26.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baffff640 1 -- 172.21.15.60:0/2824031251 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1bb00b40b0 msgr2=0x7f1bb013aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:26.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baffff640 1 --2- 172.21.15.60:0/2824031251 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1bb00b40b0 0x7f1bb013aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baffff640 1 -- 172.21.15.60:0/2824031251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bb013b020 msgr2=0x7f1bb01311a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:26.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baffff640 1 --2- 172.21.15.60:0/2824031251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bb013b020 0x7f1bb01311a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baffff640 10 monclient: _finish_hunting 0 2024-03-27T21:40:26.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baffff640 1 monclient: found mon.noname-b 2024-03-27T21:40:26.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:26.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:40:26.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baffff640 1 -- 172.21.15.60:0/2824031251 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba00094d0 con 0x7f1bb013b480 2024-03-27T21:40:26.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.279+0000 7f1baf7fe640 1 --2- 172.21.15.60:0/2824031251 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1bb00b40b0 0x7f1bb013aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:26.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1baffff640 1 --2- 172.21.15.60:0/2824031251 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bb013b480 0x7f1bb01316e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800ebd0 tx=0x7f1ba800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:26.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1b97fff640 10 client.?.objecter ms_handle_connect 0x7f1bb013b480 2024-03-27T21:40:26.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1b97fff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:26.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1b97fff640 1 -- 172.21.15.60:0/2824031251 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba800cd20 con 0x7f1bb013b480 2024-03-27T21:40:26.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1b97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:26.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1b97fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:26.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1b97fff640 10 monclient: dump: 2024-03-27T21:40:26.303 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:26.303 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:26.303 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:26.303 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:26.303 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:26.303 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:26.304 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:26.304 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:26.304 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:26.304 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:26.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1b97fff640 10 monclient: _finish_auth 0 2024-03-27T21:40:26.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1b97fff640 10 monclient: _check_auth_tickets 2024-03-27T21:40:26.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1b97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:26.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1b97fff640 1 -- 172.21.15.60:0/2824031251 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba800f040 con 0x7f1bb013b480 2024-03-27T21:40:26.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1b97fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:26.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1b97fff640 1 -- 172.21.15.60:0/2824031251 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba80146c0 con 0x7f1bb013b480 2024-03-27T21:40:26.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1b97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:26.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1b97fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:40:26.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1b97fff640 10 monclient: dump: 2024-03-27T21:40:26.304 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:26.304 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:26.305 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:26.305 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:26.305 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:26.305 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:26.305 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:26.305 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:26.305 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:26.305 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:26.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1bb5fd5640 5 monclient: authenticate success, global_id 4594 2024-03-27T21:40:26.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1bb5fd5640 10 monclient: _renew_subs 2024-03-27T21:40:26.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1bb5fd5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:26.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1badffb640 4 set_mon_vals no callback set 2024-03-27T21:40:26.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1bb5fd5640 1 -- 172.21.15.60:0/2824031251 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb013ba60 con 0x7f1bb013b480 2024-03-27T21:40:26.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1bb5fd5640 10 client.4594.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:26.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1bb5fd5640 10 monclient: _renew_subs 2024-03-27T21:40:26.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1bb5fd5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:26.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1bb5fd5640 1 -- 172.21.15.60:0/2824031251 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb0137170 con 0x7f1bb013b480 2024-03-27T21:40:26.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.280+0000 7f1bb5fd5640 1 librados: init done 2024-03-27T21:40:26.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.281+0000 7f1bb5fd5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:26.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.281+0000 7f1bb5fd5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:26.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.281+0000 7f1bb5fd5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:26.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.281+0000 7f1bb5fd5640 1 -- 172.21.15.60:0/2824031251 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb013bff0 con 0x7f1bb013b480 2024-03-27T21:40:26.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.282+0000 7f1b97fff640 1 -- 172.21.15.60:0/2824031251 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1ba8014820 con 0x7f1bb013b480 2024-03-27T21:40:26.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.283+0000 7f1b97fff640 1 --2- 172.21.15.60:0/2824031251 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1b9c06c590 0x7f1b9c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:26.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.283+0000 7f1b97fff640 1 -- 172.21.15.60:0/2824031251 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f1ba8089d70 con 0x7f1bb013b480 2024-03-27T21:40:26.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.283+0000 7f1b97fff640 10 client.4594.objecter ms_dispatch 0x7f1bb0000b90 osd_map(35..35 src has 1..35) v4 2024-03-27T21:40:26.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.283+0000 7f1b97fff640 3 client.4594.objecter handle_osd_map got epochs [35,35] > 0 2024-03-27T21:40:26.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.283+0000 7f1b97fff640 3 client.4594.objecter handle_osd_map decoding full epoch 35 2024-03-27T21:40:26.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.283+0000 7f1baf7fe640 1 --2- 172.21.15.60:0/2824031251 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1b9c06c590 0x7f1b9c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:26.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.283+0000 7f1baf7fe640 10 monclient: get_auth_request con 0x7f1b9c06c590 auth_method 0 2024-03-27T21:40:26.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.283+0000 7f1b97fff640 20 client.4594.objecter dump_active .. 0 homeless 2024-03-27T21:40:26.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.284+0000 7f1baf7fe640 1 --2- 172.21.15.60:0/2824031251 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1b9c06c590 0x7f1b9c06ea50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1ba00098a0 tx=0x7f1ba003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:26.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.284+0000 7f1b97fff640 10 client.4594.objecter ms_handle_connect 0x7f1b9c06c590 2024-03-27T21:40:26.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.287+0000 7f1b97fff640 1 -- 172.21.15.60:0/2824031251 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1bb013bff0 con 0x7f1bb013b480 2024-03-27T21:40:26.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.287+0000 7f1b97fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:26.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.287+0000 7f1b97fff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:26.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.302+0000 7f1f2f7ce640 10 client.4511.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:26.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.302+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign -1 2 2024-03-27T21:40:26.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.302+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f280a90d0 osd=0 3 2024-03-27T21:40:26.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.302+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f280a90d0 osd=0 3 2024-03-27T21:40:26.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.302+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f280a90d0 osd=0 3 2024-03-27T21:40:26.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.302+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_remove -1 2 2024-03-27T21:40:26.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.302+0000 7f1f2f7ce640 20 client.4511.objecter get_session s=0x7f1f280a90d0 osd=0 3 2024-03-27T21:40:26.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.302+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign 0 2 2024-03-27T21:40:26.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.302+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f280a90d0 osd=0 4 2024-03-27T21:40:26.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.302+0000 7f1f2f7ce640 10 client.4511.objecter _send_command 2 2024-03-27T21:40:26.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.302+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1f28131e00 con 0x7f1f2813e5f0 2024-03-27T21:40:26.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.307+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f1f28131e00 con 0x7f1f2813e5f0 2024-03-27T21:40:26.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.307+0000 7f1f1dffb640 10 client.4511.objecter ms_dispatch 0x7f1f28000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:26.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.307+0000 7f1f1dffb640 10 client.4511.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:26.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.307+0000 7f1f1dffb640 20 client.4511.objecter put_session s=0x7f1f280a90d0 osd=0 4 2024-03-27T21:40:26.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.307+0000 7f1f1dffb640 15 client.4511.objecter _session_command_op_remove 0 2 2024-03-27T21:40:26.309 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:26.310 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:26.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.307+0000 7f1f2f7ce640 10 client.4511.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:26.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.307+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign -1 3 2024-03-27T21:40:26.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.307+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f1f28065a90 0x7f1f28065e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:26.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.307+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f2813aca0 osd=1 3 2024-03-27T21:40:26.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.307+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f2813aca0 osd=1 3 2024-03-27T21:40:26.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.307+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f2813aca0 osd=1 3 2024-03-27T21:40:26.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.307+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_remove -1 3 2024-03-27T21:40:26.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.307+0000 7f1f2f7ce640 20 client.4511.objecter get_session s=0x7f1f2813aca0 osd=1 3 2024-03-27T21:40:26.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.307+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign 1 3 2024-03-27T21:40:26.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.307+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f2813aca0 osd=1 4 2024-03-27T21:40:26.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.307+0000 7f1f2f7ce640 10 client.4511.objecter _send_command 3 2024-03-27T21:40:26.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.307+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1f2813d660 con 0x7f1f28065a90 2024-03-27T21:40:26.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.307+0000 7f1f2d56a640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f1f28065a90 0x7f1f28065e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:26.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.307+0000 7f1f2d56a640 10 monclient: get_auth_request con 0x7f1f28065a90 auth_method 0 2024-03-27T21:40:26.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.308+0000 7f1f1dffb640 10 client.4511.objecter ms_handle_connect 0x7f1f28065a90 2024-03-27T21:40:26.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.308+0000 7f1f2d56a640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f1f28065a90 0x7f1f28065e70 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:26.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.308+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f1f2813d660 con 0x7f1f28065a90 2024-03-27T21:40:26.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.309+0000 7f1f1dffb640 10 client.4511.objecter ms_dispatch 0x7f1f28000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:26.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.309+0000 7f1f1dffb640 10 client.4511.objecter _finish_command 3 = system:0 2024-03-27T21:40:26.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.309+0000 7f1f1dffb640 20 client.4511.objecter put_session s=0x7f1f2813aca0 osd=1 4 2024-03-27T21:40:26.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.309+0000 7f1f1dffb640 15 client.4511.objecter _session_command_op_remove 1 3 2024-03-27T21:40:26.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.338+0000 7f1f2f7ce640 10 client.4511.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:26.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.338+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign -1 4 2024-03-27T21:40:26.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.338+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f2813aca0 osd=1 3 2024-03-27T21:40:26.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.338+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f2813aca0 osd=1 3 2024-03-27T21:40:26.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.338+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f2813aca0 osd=1 3 2024-03-27T21:40:26.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.338+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_remove -1 4 2024-03-27T21:40:26.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.338+0000 7f1f2f7ce640 20 client.4511.objecter get_session s=0x7f1f2813aca0 osd=1 3 2024-03-27T21:40:26.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.338+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign 1 4 2024-03-27T21:40:26.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.338+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f2813aca0 osd=1 4 2024-03-27T21:40:26.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.338+0000 7f1f2f7ce640 10 client.4511.objecter _send_command 4 2024-03-27T21:40:26.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.338+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1f2813a8b0 con 0x7f1f28065a90 2024-03-27T21:40:26.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.340+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f1f2813a8b0 con 0x7f1f28065a90 2024-03-27T21:40:26.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.340+0000 7f1f1dffb640 10 client.4511.objecter ms_dispatch 0x7f1f28000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:26.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.340+0000 7f1f1dffb640 10 client.4511.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:26.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.340+0000 7f1f1dffb640 20 client.4511.objecter put_session s=0x7f1f2813aca0 osd=1 4 2024-03-27T21:40:26.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.340+0000 7f1f1dffb640 15 client.4511.objecter _session_command_op_remove 1 4 2024-03-27T21:40:26.327 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:26.329 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:26.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.341+0000 7f1f2f7ce640 10 client.4511.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:26.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.341+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign -1 5 2024-03-27T21:40:26.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.341+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f1f2802b170 0x7f1f2802b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:26.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.341+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f28070150 osd=2 3 2024-03-27T21:40:26.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.341+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f28070150 osd=2 3 2024-03-27T21:40:26.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.341+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f28070150 osd=2 3 2024-03-27T21:40:26.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.341+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_remove -1 5 2024-03-27T21:40:26.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.341+0000 7f1f2f7ce640 20 client.4511.objecter get_session s=0x7f1f28070150 osd=2 3 2024-03-27T21:40:26.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.341+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign 2 5 2024-03-27T21:40:26.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.341+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f28070150 osd=2 4 2024-03-27T21:40:26.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.341+0000 7f1f2f7ce640 10 client.4511.objecter _send_command 5 2024-03-27T21:40:26.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.341+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1f28069470 con 0x7f1f2802b170 2024-03-27T21:40:26.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.341+0000 7f1f2cd69640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f1f2802b170 0x7f1f2802b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:26.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.341+0000 7f1f2cd69640 10 monclient: get_auth_request con 0x7f1f2802b170 auth_method 0 2024-03-27T21:40:26.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.341+0000 7f1f1dffb640 10 client.4511.objecter ms_handle_connect 0x7f1f2802b170 2024-03-27T21:40:26.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.341+0000 7f1f2cd69640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f1f2802b170 0x7f1f2802b550 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:26.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.342+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== osd.2 v2:172.21.15.60:6824/44654 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f1f28069470 con 0x7f1f2802b170 2024-03-27T21:40:26.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.342+0000 7f1f1dffb640 10 client.4511.objecter ms_dispatch 0x7f1f28000b90 command_reply(tid 5: 0 ) v1 2024-03-27T21:40:26.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.342+0000 7f1f1dffb640 10 client.4511.objecter _finish_command 5 = system:0 2024-03-27T21:40:26.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.342+0000 7f1f1dffb640 20 client.4511.objecter put_session s=0x7f1f28070150 osd=2 4 2024-03-27T21:40:26.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.342+0000 7f1f1dffb640 15 client.4511.objecter _session_command_op_remove 2 5 2024-03-27T21:40:26.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.373+0000 7f1f2f7ce640 10 client.4511.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:26.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.373+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign -1 6 2024-03-27T21:40:26.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.373+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f28070150 osd=2 3 2024-03-27T21:40:26.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.373+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f28070150 osd=2 3 2024-03-27T21:40:26.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.373+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f28070150 osd=2 3 2024-03-27T21:40:26.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.373+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_remove -1 6 2024-03-27T21:40:26.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.373+0000 7f1f2f7ce640 20 client.4511.objecter get_session s=0x7f1f28070150 osd=2 3 2024-03-27T21:40:26.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.373+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign 2 6 2024-03-27T21:40:26.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.373+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f28070150 osd=2 4 2024-03-27T21:40:26.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.373+0000 7f1f2f7ce640 10 client.4511.objecter _send_command 6 2024-03-27T21:40:26.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.373+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1f2802ba90 con 0x7f1f2802b170 2024-03-27T21:40:26.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.375+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== osd.2 v2:172.21.15.60:6824/44654 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f1f2802ba90 con 0x7f1f2802b170 2024-03-27T21:40:26.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.375+0000 7f1f1dffb640 10 client.4511.objecter ms_dispatch 0x7f1f28000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:26.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.375+0000 7f1f1dffb640 10 client.4511.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:26.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.375+0000 7f1f1dffb640 20 client.4511.objecter put_session s=0x7f1f28070150 osd=2 4 2024-03-27T21:40:26.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.375+0000 7f1f1dffb640 15 client.4511.objecter _session_command_op_remove 2 6 2024-03-27T21:40:26.362 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2: {} 2024-03-27T21:40:26.363 INFO:teuthology.orchestra.run.smithi060.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-27T21:40:26.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.376+0000 7f1f2f7ce640 10 client.4511.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:26.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.376+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign -1 7 2024-03-27T21:40:26.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.376+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f1f281e2cb0 0x7f1f281e50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:26.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.376+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f2802c420 osd=3 3 2024-03-27T21:40:26.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.376+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f2802c420 osd=3 3 2024-03-27T21:40:26.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.376+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f2802c420 osd=3 3 2024-03-27T21:40:26.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.376+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_remove -1 7 2024-03-27T21:40:26.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.376+0000 7f1f2f7ce640 20 client.4511.objecter get_session s=0x7f1f2802c420 osd=3 3 2024-03-27T21:40:26.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.376+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign 3 7 2024-03-27T21:40:26.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.376+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f2802c420 osd=3 4 2024-03-27T21:40:26.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.376+0000 7f1f2f7ce640 10 client.4511.objecter _send_command 7 2024-03-27T21:40:26.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.376+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1f2802c590 con 0x7f1f281e2cb0 2024-03-27T21:40:26.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.376+0000 7f1f2dd6b640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f1f281e2cb0 0x7f1f281e50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:26.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.376+0000 7f1f2dd6b640 10 monclient: get_auth_request con 0x7f1f281e2cb0 auth_method 0 2024-03-27T21:40:26.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.376+0000 7f1f2dd6b640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f1f281e2cb0 0x7f1f281e50a0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:26.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.376+0000 7f1f1dffb640 10 client.4511.objecter ms_handle_connect 0x7f1f281e2cb0 2024-03-27T21:40:26.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.377+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f1f2802c590 con 0x7f1f281e2cb0 2024-03-27T21:40:26.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.377+0000 7f1f1dffb640 10 client.4511.objecter ms_dispatch 0x7f1f28000b90 command_reply(tid 7: 0 ) v1 2024-03-27T21:40:26.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.377+0000 7f1f1dffb640 10 client.4511.objecter _finish_command 7 = system:0 2024-03-27T21:40:26.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.377+0000 7f1f1dffb640 20 client.4511.objecter put_session s=0x7f1f2802c420 osd=3 4 2024-03-27T21:40:26.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.377+0000 7f1f1dffb640 15 client.4511.objecter _session_command_op_remove 3 7 2024-03-27T21:40:26.379 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:26.393+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:26.383 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:26.383 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:26.383 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:26.383 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:26.392 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:40:26.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.405+0000 7f1f2f7ce640 10 client.4511.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:26.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.405+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign -1 8 2024-03-27T21:40:26.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.405+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f2802c420 osd=3 3 2024-03-27T21:40:26.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.405+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f2802c420 osd=3 3 2024-03-27T21:40:26.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.405+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f2802c420 osd=3 3 2024-03-27T21:40:26.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.405+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_remove -1 8 2024-03-27T21:40:26.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.405+0000 7f1f2f7ce640 20 client.4511.objecter get_session s=0x7f1f2802c420 osd=3 3 2024-03-27T21:40:26.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.405+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign 3 8 2024-03-27T21:40:26.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.405+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f2802c420 osd=3 4 2024-03-27T21:40:26.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.405+0000 7f1f2f7ce640 10 client.4511.objecter _send_command 8 2024-03-27T21:40:26.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.405+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1f2802be40 con 0x7f1f281e2cb0 2024-03-27T21:40:26.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.408+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f1f2802be40 con 0x7f1f281e2cb0 2024-03-27T21:40:26.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.408+0000 7f1f1dffb640 10 client.4511.objecter ms_dispatch 0x7f1f28000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:26.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.408+0000 7f1f1dffb640 10 client.4511.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:26.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.408+0000 7f1f1dffb640 20 client.4511.objecter put_session s=0x7f1f2802c420 osd=3 4 2024-03-27T21:40:26.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.408+0000 7f1f1dffb640 15 client.4511.objecter _session_command_op_remove 3 8 2024-03-27T21:40:26.395 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:26.396 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:26.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.408+0000 7f1f2f7ce640 10 client.4511.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:26.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.408+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign -1 9 2024-03-27T21:40:26.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.408+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f1f280b2620 0x7f1f281e8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:26.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.408+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f2802be40 osd=4 3 2024-03-27T21:40:26.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.408+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f2802be40 osd=4 3 2024-03-27T21:40:26.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.408+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f2802be40 osd=4 3 2024-03-27T21:40:26.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.408+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_remove -1 9 2024-03-27T21:40:26.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.408+0000 7f1f2f7ce640 20 client.4511.objecter get_session s=0x7f1f2802be40 osd=4 3 2024-03-27T21:40:26.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.408+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign 4 9 2024-03-27T21:40:26.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.408+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f2802be40 osd=4 4 2024-03-27T21:40:26.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.408+0000 7f1f2f7ce640 10 client.4511.objecter _send_command 9 2024-03-27T21:40:26.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.408+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1f2802c280 con 0x7f1f280b2620 2024-03-27T21:40:26.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.409+0000 7f1f2d56a640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f1f280b2620 0x7f1f281e8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:26.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.409+0000 7f1f2d56a640 10 monclient: get_auth_request con 0x7f1f280b2620 auth_method 0 2024-03-27T21:40:26.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.409+0000 7f1f1dffb640 10 client.4511.objecter ms_handle_connect 0x7f1f280b2620 2024-03-27T21:40:26.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.409+0000 7f1f2d56a640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f1f280b2620 0x7f1f281e8770 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:26.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.410+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f1f2802c280 con 0x7f1f280b2620 2024-03-27T21:40:26.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.410+0000 7f1f1dffb640 10 client.4511.objecter ms_dispatch 0x7f1f28000b90 command_reply(tid 9: 0 ) v1 2024-03-27T21:40:26.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.410+0000 7f1f1dffb640 10 client.4511.objecter _finish_command 9 = system:0 2024-03-27T21:40:26.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.410+0000 7f1f1dffb640 20 client.4511.objecter put_session s=0x7f1f2802be40 osd=4 4 2024-03-27T21:40:26.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.410+0000 7f1f1dffb640 15 client.4511.objecter _session_command_op_remove 4 9 2024-03-27T21:40:26.408 INFO:teuthology.orchestra.run.smithi060.stderr:nodeep-scrub is unset 2024-03-27T21:40:26.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.421+0000 7fb3baffd640 1 -- 172.21.15.60:0/1976441434 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v36) v1 ==== 101+0+0 (secure 0 0 0) 0x7fb3c40571a0 con 0x7fb3cc13ec50 2024-03-27T21:40:26.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.421+0000 7fb3baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:40:26.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.421+0000 7fb3baffd640 10 monclient: _finish_command 2 = system:0 nodeep-scrub is unset 2024-03-27T21:40:26.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.423+0000 7fb3d0bfd640 10 librados: watch_flush enter 2024-03-27T21:40:26.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.423+0000 7fb3d0bfd640 10 librados: watch_flush exit 2024-03-27T21:40:26.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.423+0000 7fb3d0bfd640 20 client.4629.objecter shutdown clearing up homeless session... 2024-03-27T21:40:26.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.423+0000 7fb3d0bfd640 10 client.4629.objecter successfully canceled tick 2024-03-27T21:40:26.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.423+0000 7fb3d0bfd640 1 -- 172.21.15.60:0/1976441434 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb3ac06c5c0 msgr2=0x7fb3ac06ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:26.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.423+0000 7fb3d0bfd640 1 --2- 172.21.15.60:0/1976441434 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb3ac06c5c0 0x7fb3ac06ea80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc034000 tx=0x7fb3bc008030 comp rx=0 tx=0).stop 2024-03-27T21:40:26.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.423+0000 7fb3d0bfd640 10 monclient: shutdown 2024-03-27T21:40:26.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.423+0000 7fb3d0bfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:26.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.423+0000 7fb3d0bfd640 1 -- 172.21.15.60:0/1976441434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3cc13ec50 msgr2=0x7fb3cc1396a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:26.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.423+0000 7fb3d0bfd640 1 --2- 172.21.15.60:0/1976441434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3cc13ec50 0x7fb3cc1396a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400c7f0 tx=0x7fb3c400ccb0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.424+0000 7fb3d0bfd640 1 -- 172.21.15.60:0/1976441434 shutdown_connections 2024-03-27T21:40:26.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.424+0000 7fb3d0bfd640 1 --2- 172.21.15.60:0/1976441434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3cc13ec50 0x7fb3cc1396a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.424+0000 7fb3d0bfd640 1 --2- 172.21.15.60:0/1976441434 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb3cc13e810 0x7fb3cc139160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.424+0000 7fb3d0bfd640 1 --2- 172.21.15.60:0/1976441434 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb3ac06c5c0 0x7fb3ac06ea80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.424+0000 7fb3d0bfd640 1 --2- 172.21.15.60:0/1976441434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3cc071a50 0x7fb3cc1387a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.424+0000 7fb3d0bfd640 1 -- 172.21.15.60:0/1976441434 >> 172.21.15.60:0/1976441434 conn(0x7fb3cc0b7300 msgr2=0x7fb3cc13d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:26.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.424+0000 7fb3d0bfd640 1 -- 172.21.15.60:0/1976441434 shutdown_connections 2024-03-27T21:40:26.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.424+0000 7fb3d0bfd640 1 -- 172.21.15.60:0/1976441434 wait complete. 2024-03-27T21:40:26.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.424+0000 7fb3d0bfd640 1 librados: shutdown 2024-03-27T21:40:26.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.437+0000 7f1f2f7ce640 10 client.4511.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.437+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign -1 10 2024-03-27T21:40:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.437+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f2802be40 osd=4 3 2024-03-27T21:40:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.437+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f2802be40 osd=4 3 2024-03-27T21:40:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.437+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f2802be40 osd=4 3 2024-03-27T21:40:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.437+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_remove -1 10 2024-03-27T21:40:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.437+0000 7f1f2f7ce640 20 client.4511.objecter get_session s=0x7f1f2802be40 osd=4 3 2024-03-27T21:40:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.437+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign 4 10 2024-03-27T21:40:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.437+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f2802be40 osd=4 4 2024-03-27T21:40:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.437+0000 7f1f2f7ce640 10 client.4511.objecter _send_command 10 2024-03-27T21:40:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.437+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1f2802c1f0 con 0x7f1f280b2620 2024-03-27T21:40:26.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.439+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f1f2802c1f0 con 0x7f1f280b2620 2024-03-27T21:40:26.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.439+0000 7f1f1dffb640 10 client.4511.objecter ms_dispatch 0x7f1f28000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:26.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.439+0000 7f1f1dffb640 10 client.4511.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:26.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.439+0000 7f1f1dffb640 20 client.4511.objecter put_session s=0x7f1f2802be40 osd=4 4 2024-03-27T21:40:26.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.439+0000 7f1f1dffb640 15 client.4511.objecter _session_command_op_remove 4 10 2024-03-27T21:40:26.426 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:26.428 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:26.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.439+0000 7f1f2f7ce640 10 client.4511.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:26.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.439+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign -1 11 2024-03-27T21:40:26.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.439+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f1f281e91c0 0x7f1f281eb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:26.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.439+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f2802c1f0 osd=5 3 2024-03-27T21:40:26.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.439+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f2802c1f0 osd=5 3 2024-03-27T21:40:26.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.439+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f2802c1f0 osd=5 3 2024-03-27T21:40:26.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.439+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_remove -1 11 2024-03-27T21:40:26.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.439+0000 7f1f2f7ce640 20 client.4511.objecter get_session s=0x7f1f2802c1f0 osd=5 3 2024-03-27T21:40:26.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.439+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign 5 11 2024-03-27T21:40:26.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.440+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f2802c1f0 osd=5 4 2024-03-27T21:40:26.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.440+0000 7f1f2f7ce640 10 client.4511.objecter _send_command 11 2024-03-27T21:40:26.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.440+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1f281ee380 con 0x7f1f281e91c0 2024-03-27T21:40:26.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.440+0000 7f1f2cd69640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f1f281e91c0 0x7f1f281eb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:26.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.440+0000 7f1f2cd69640 10 monclient: get_auth_request con 0x7f1f281e91c0 auth_method 0 2024-03-27T21:40:26.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.440+0000 7f1f1dffb640 10 client.4511.objecter ms_handle_connect 0x7f1f281e91c0 2024-03-27T21:40:26.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.440+0000 7f1f2cd69640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f1f281e91c0 0x7f1f281eb5b0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:26.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.441+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f1f281ee380 con 0x7f1f281e91c0 2024-03-27T21:40:26.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.441+0000 7f1f1dffb640 10 client.4511.objecter ms_dispatch 0x7f1f28000b90 command_reply(tid 11: 0 ) v1 2024-03-27T21:40:26.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.441+0000 7f1f1dffb640 10 client.4511.objecter _finish_command 11 = system:0 2024-03-27T21:40:26.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.441+0000 7f1f1dffb640 20 client.4511.objecter put_session s=0x7f1f2802c1f0 osd=5 4 2024-03-27T21:40:26.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.441+0000 7f1f1dffb640 15 client.4511.objecter _session_command_op_remove 5 11 2024-03-27T21:40:26.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.468+0000 7f1f2f7ce640 10 client.4511.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:26.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.468+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign -1 12 2024-03-27T21:40:26.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.468+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f2802c1f0 osd=5 3 2024-03-27T21:40:26.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.468+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f2802c1f0 osd=5 3 2024-03-27T21:40:26.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.468+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f2802c1f0 osd=5 3 2024-03-27T21:40:26.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.468+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_remove -1 12 2024-03-27T21:40:26.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.468+0000 7f1f2f7ce640 20 client.4511.objecter get_session s=0x7f1f2802c1f0 osd=5 3 2024-03-27T21:40:26.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.468+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign 5 12 2024-03-27T21:40:26.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.468+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f2802c1f0 osd=5 4 2024-03-27T21:40:26.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.468+0000 7f1f2f7ce640 10 client.4511.objecter _send_command 12 2024-03-27T21:40:26.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.468+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1f281ebaf0 con 0x7f1f281e91c0 2024-03-27T21:40:26.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.470+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f1f281ebaf0 con 0x7f1f281e91c0 2024-03-27T21:40:26.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.470+0000 7f1f1dffb640 10 client.4511.objecter ms_dispatch 0x7f1f28000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:26.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.470+0000 7f1f1dffb640 10 client.4511.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:26.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.470+0000 7f1f1dffb640 20 client.4511.objecter put_session s=0x7f1f2802c1f0 osd=5 4 2024-03-27T21:40:26.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.470+0000 7f1f1dffb640 15 client.4511.objecter _session_command_op_remove 5 12 2024-03-27T21:40:26.457 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:26.458 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:26.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.470+0000 7f1f2f7ce640 10 client.4511.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:26.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.470+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign -1 13 2024-03-27T21:40:26.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.470+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f1f281ec1a0 0x7f1f281ee590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:26.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.470+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f281ebaf0 osd=6 3 2024-03-27T21:40:26.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.470+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f281ebaf0 osd=6 3 2024-03-27T21:40:26.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.470+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f281ebaf0 osd=6 3 2024-03-27T21:40:26.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.470+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_remove -1 13 2024-03-27T21:40:26.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.470+0000 7f1f2f7ce640 20 client.4511.objecter get_session s=0x7f1f281ebaf0 osd=6 3 2024-03-27T21:40:26.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.470+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign 6 13 2024-03-27T21:40:26.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.470+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f281ebaf0 osd=6 4 2024-03-27T21:40:26.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.470+0000 7f1f2f7ce640 10 client.4511.objecter _send_command 13 2024-03-27T21:40:26.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.470+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1f281f1360 con 0x7f1f281ec1a0 2024-03-27T21:40:26.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.471+0000 7f1f2dd6b640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f1f281ec1a0 0x7f1f281ee590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:26.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.471+0000 7f1f2dd6b640 10 monclient: get_auth_request con 0x7f1f281ec1a0 auth_method 0 2024-03-27T21:40:26.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.471+0000 7f1f1dffb640 10 client.4511.objecter ms_handle_connect 0x7f1f281ec1a0 2024-03-27T21:40:26.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.471+0000 7f1f2dd6b640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f1f281ec1a0 0x7f1f281ee590 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:26.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.472+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f1f281f1360 con 0x7f1f281ec1a0 2024-03-27T21:40:26.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.472+0000 7f1f1dffb640 10 client.4511.objecter ms_dispatch 0x7f1f28000b90 command_reply(tid 13: 0 ) v1 2024-03-27T21:40:26.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.472+0000 7f1f1dffb640 10 client.4511.objecter _finish_command 13 = system:0 2024-03-27T21:40:26.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.472+0000 7f1f1dffb640 20 client.4511.objecter put_session s=0x7f1f281ebaf0 osd=6 4 2024-03-27T21:40:26.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.472+0000 7f1f1dffb640 15 client.4511.objecter _session_command_op_remove 6 13 2024-03-27T21:40:26.480 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:26.494+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:26.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.500+0000 7f1f2f7ce640 10 client.4511.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:26.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.500+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign -1 14 2024-03-27T21:40:26.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.500+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f281ebaf0 osd=6 3 2024-03-27T21:40:26.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.500+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f281ebaf0 osd=6 3 2024-03-27T21:40:26.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.500+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f281ebaf0 osd=6 3 2024-03-27T21:40:26.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.500+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_remove -1 14 2024-03-27T21:40:26.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.500+0000 7f1f2f7ce640 20 client.4511.objecter get_session s=0x7f1f281ebaf0 osd=6 3 2024-03-27T21:40:26.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.500+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign 6 14 2024-03-27T21:40:26.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.500+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f281ebaf0 osd=6 4 2024-03-27T21:40:26.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.500+0000 7f1f2f7ce640 10 client.4511.objecter _send_command 14 2024-03-27T21:40:26.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.500+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1f281eead0 con 0x7f1f281ec1a0 2024-03-27T21:40:26.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.501+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f1f281eead0 con 0x7f1f281ec1a0 2024-03-27T21:40:26.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.501+0000 7f1f1dffb640 10 client.4511.objecter ms_dispatch 0x7f1f28000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:26.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.501+0000 7f1f1dffb640 10 client.4511.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:26.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.501+0000 7f1f1dffb640 20 client.4511.objecter put_session s=0x7f1f281ebaf0 osd=6 4 2024-03-27T21:40:26.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.501+0000 7f1f1dffb640 15 client.4511.objecter _session_command_op_remove 6 14 2024-03-27T21:40:26.488 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:26.490 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:26.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.502+0000 7f1f2f7ce640 10 client.4511.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:26.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.502+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign -1 15 2024-03-27T21:40:26.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.502+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f1f281ef1a0 0x7f1f281ef600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:26.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.502+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f281eead0 osd=7 3 2024-03-27T21:40:26.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.502+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f281eead0 osd=7 3 2024-03-27T21:40:26.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.502+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f281eead0 osd=7 3 2024-03-27T21:40:26.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.502+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_remove -1 15 2024-03-27T21:40:26.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.502+0000 7f1f2f7ce640 20 client.4511.objecter get_session s=0x7f1f281eead0 osd=7 3 2024-03-27T21:40:26.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.502+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign 7 15 2024-03-27T21:40:26.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.502+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f281eead0 osd=7 4 2024-03-27T21:40:26.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.502+0000 7f1f2f7ce640 10 client.4511.objecter _send_command 15 2024-03-27T21:40:26.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.502+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1f281f4690 con 0x7f1f281ef1a0 2024-03-27T21:40:26.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.502+0000 7f1f2d56a640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f1f281ef1a0 0x7f1f281ef600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:26.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.502+0000 7f1f2d56a640 10 monclient: get_auth_request con 0x7f1f281ef1a0 auth_method 0 2024-03-27T21:40:26.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.503+0000 7f1f2d56a640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f1f281ef1a0 0x7f1f281ef600 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:26.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.503+0000 7f1f1dffb640 10 client.4511.objecter ms_handle_connect 0x7f1f281ef1a0 2024-03-27T21:40:26.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.503+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f1f281f4690 con 0x7f1f281ef1a0 2024-03-27T21:40:26.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.503+0000 7f1f1dffb640 10 client.4511.objecter ms_dispatch 0x7f1f28000b90 command_reply(tid 15: 0 ) v1 2024-03-27T21:40:26.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.503+0000 7f1f1dffb640 10 client.4511.objecter _finish_command 15 = system:0 2024-03-27T21:40:26.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.503+0000 7f1f1dffb640 20 client.4511.objecter put_session s=0x7f1f281eead0 osd=7 4 2024-03-27T21:40:26.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.503+0000 7f1f1dffb640 15 client.4511.objecter _session_command_op_remove 7 15 2024-03-27T21:40:26.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.531+0000 7f1f2f7ce640 10 client.4511.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:26.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.531+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign -1 16 2024-03-27T21:40:26.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.531+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f281eead0 osd=7 3 2024-03-27T21:40:26.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.531+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f281eead0 osd=7 3 2024-03-27T21:40:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.531+0000 7f1f2f7ce640 20 client.4511.objecter _get_session s=0x7f1f281eead0 osd=7 3 2024-03-27T21:40:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.531+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_remove -1 16 2024-03-27T21:40:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.531+0000 7f1f2f7ce640 20 client.4511.objecter get_session s=0x7f1f281eead0 osd=7 3 2024-03-27T21:40:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.531+0000 7f1f2f7ce640 15 client.4511.objecter _session_command_op_assign 7 16 2024-03-27T21:40:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.531+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f281eead0 osd=7 4 2024-03-27T21:40:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.531+0000 7f1f2f7ce640 10 client.4511.objecter _send_command 16 2024-03-27T21:40:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.531+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1f281efc50 con 0x7f1f281ef1a0 2024-03-27T21:40:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.533+0000 7f1f1dffb640 1 -- 172.21.15.60:0/3646919696 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f1f281efc50 con 0x7f1f281ef1a0 2024-03-27T21:40:26.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.533+0000 7f1f1dffb640 10 client.4511.objecter ms_dispatch 0x7f1f28000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:26.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.533+0000 7f1f1dffb640 10 client.4511.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:26.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.533+0000 7f1f1dffb640 20 client.4511.objecter put_session s=0x7f1f281eead0 osd=7 4 2024-03-27T21:40:26.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.533+0000 7f1f1dffb640 15 client.4511.objecter _session_command_op_remove 7 16 2024-03-27T21:40:26.520 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:26.526 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:26.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 10 librados: watch_flush enter 2024-03-27T21:40:26.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 10 librados: watch_flush exit 2024-03-27T21:40:26.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 10 client.4511.objecter close_session for osd.0 2024-03-27T21:40:26.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f1f2813e5f0 msgr2=0x7f1f28138a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:26.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f1f2813e5f0 0x7f1f28138a70 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f280a90d0 osd=0 1 2024-03-27T21:40:26.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 10 client.4511.objecter close_session for osd.1 2024-03-27T21:40:26.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f1f28065a90 msgr2=0x7f1f28065e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:26.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f1f28065a90 0x7f1f28065e70 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f2813aca0 osd=1 1 2024-03-27T21:40:26.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 10 client.4511.objecter close_session for osd.2 2024-03-27T21:40:26.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f1f2802b170 msgr2=0x7f1f2802b550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:26.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f1f2802b170 0x7f1f2802b550 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2dd6b640 1 -- 172.21.15.60:0/3646919696 reap_dead start 2024-03-27T21:40:26.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f28070150 osd=2 1 2024-03-27T21:40:26.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 10 client.4511.objecter close_session for osd.3 2024-03-27T21:40:26.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f1f281e2cb0 msgr2=0x7f1f281e50a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:26.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f1f281e2cb0 0x7f1f281e50a0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f2802c420 osd=3 1 2024-03-27T21:40:26.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 10 client.4511.objecter close_session for osd.4 2024-03-27T21:40:26.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f1f280b2620 msgr2=0x7f1f281e8770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f1f280b2620 0x7f1f281e8770 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f2802be40 osd=4 1 2024-03-27T21:40:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 10 client.4511.objecter close_session for osd.5 2024-03-27T21:40:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f1f281e91c0 msgr2=0x7f1f281eb5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f1f281e91c0 0x7f1f281eb5b0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f2802c1f0 osd=5 1 2024-03-27T21:40:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 10 client.4511.objecter close_session for osd.6 2024-03-27T21:40:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f1f281ec1a0 msgr2=0x7f1f281ee590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f1f281ec1a0 0x7f1f281ee590 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f281ebaf0 osd=6 1 2024-03-27T21:40:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 10 client.4511.objecter close_session for osd.7 2024-03-27T21:40:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f1f281ef1a0 msgr2=0x7f1f281ef600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f1f281ef1a0 0x7f1f281ef600 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2dd6b640 1 -- 172.21.15.60:0/3646919696 reap_dead start 2024-03-27T21:40:26.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 20 client.4511.objecter put_session s=0x7f1f281eead0 osd=7 1 2024-03-27T21:40:26.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 20 client.4511.objecter shutdown clearing up homeless session... 2024-03-27T21:40:26.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 10 client.4511.objecter successfully canceled tick 2024-03-27T21:40:26.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1f0c074da0 msgr2=0x7f1f0c077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:26.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1f0c074da0 0x7f1f0c077260 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1f18007fd0 tx=0x7f1f18009040 comp rx=0 tx=0).stop 2024-03-27T21:40:26.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 10 monclient: shutdown 2024-03-27T21:40:26.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:26.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1f2813b020 msgr2=0x7f1f281336f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:26.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.534+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1f2813b020 0x7f1f281336f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1f24036040 tx=0x7f1f24004a40 comp rx=0 tx=0).stop 2024-03-27T21:40:26.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.535+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 shutdown_connections 2024-03-27T21:40:26.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.535+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1f0c074da0 0x7f1f0c077260 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.535+0000 7f1f2f7ce640 1 --2- 172.21.15.60:0/3646919696 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1f2813b020 0x7f1f281336f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:26.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.535+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 >> 172.21.15.60:0/3646919696 conn(0x7f1f280608e0 msgr2=0x7f1f280b4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:26.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.535+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 shutdown_connections 2024-03-27T21:40:26.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.535+0000 7f1f2f7ce640 1 -- 172.21.15.60:0/3646919696 wait complete. 2024-03-27T21:40:26.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.535+0000 7f1f2f7ce640 1 librados: shutdown 2024-03-27T21:40:26.537 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:26.537 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:26.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.559+0000 7f1bb5fd5640 10 start_mon_command cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}] 2024-03-27T21:40:26.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.559+0000 7f1bb5fd5640 10 monclient: _send_command 2 [{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}] 2024-03-27T21:40:26.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.559+0000 7f1bb5fd5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:26.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:26.559+0000 7f1bb5fd5640 1 -- 172.21.15.60:0/2824031251 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} v 0) v1 -- 0x7f1bb013bff0 con 0x7f1bb013b480 2024-03-27T21:40:26.546 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:40:26.581 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:26.595+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:26.682 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:26.696+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:26.692 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:26.692 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:26.701 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:40:26.785 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:26.799+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:26.847 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:26.847 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:26.855 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:40:26.884 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:26.898+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:26.985 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:26.999+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:27.000 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:27.000 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:27.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:40:27.086 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:27.100+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:27.155 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:27.155 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:27.164 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:40:27.188 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:27.201+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:27.269 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:27.269 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:27.278 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:40:27.321 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:27.335+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:27.383 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:27.383 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:27.389 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:27.403+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:27.392 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:40:27.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.429+0000 7f1b97fff640 1 -- 172.21.15.60:0/2824031251 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]=0 set osd.4 primary-affinity to 1 (8655362) v37) v1 ==== 133+0+0 (secure 0 0 0) 0x7f1bb013bff0 con 0x7f1bb013b480 2024-03-27T21:40:27.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.429+0000 7f1b97fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}] 2024-03-27T21:40:27.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.429+0000 7f1b97fff640 10 monclient: _finish_command 2 = system:0 set osd.4 primary-affinity to 1 (8655362) 2024-03-27T21:40:27.416 INFO:teuthology.orchestra.run.smithi060.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-27T21:40:27.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.431+0000 7f1bb5fd5640 10 librados: watch_flush enter 2024-03-27T21:40:27.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.431+0000 7f1bb5fd5640 10 librados: watch_flush exit 2024-03-27T21:40:27.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.431+0000 7f1bb5fd5640 20 client.4594.objecter shutdown clearing up homeless session... 2024-03-27T21:40:27.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.431+0000 7f1bb5fd5640 10 client.4594.objecter successfully canceled tick 2024-03-27T21:40:27.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.432+0000 7f1bb5fd5640 1 -- 172.21.15.60:0/2824031251 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1b9c06c590 msgr2=0x7f1b9c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:27.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.432+0000 7f1bb5fd5640 1 --2- 172.21.15.60:0/2824031251 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1b9c06c590 0x7f1b9c06ea50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1ba00098a0 tx=0x7f1ba003c040 comp rx=0 tx=0).stop 2024-03-27T21:40:27.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.432+0000 7f1bb5fd5640 10 monclient: shutdown 2024-03-27T21:40:27.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.432+0000 7f1bb5fd5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:27.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.432+0000 7f1bb5fd5640 1 -- 172.21.15.60:0/2824031251 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bb013b480 msgr2=0x7f1bb01316e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:27.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.432+0000 7f1bb5fd5640 1 --2- 172.21.15.60:0/2824031251 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bb013b480 0x7f1bb01316e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800ebd0 tx=0x7f1ba800c490 comp rx=0 tx=0).stop 2024-03-27T21:40:27.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.432+0000 7f1bb5fd5640 1 -- 172.21.15.60:0/2824031251 shutdown_connections 2024-03-27T21:40:27.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.432+0000 7f1bb5fd5640 1 --2- 172.21.15.60:0/2824031251 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1b9c06c590 0x7f1b9c06ea50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:27.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.432+0000 7f1bb5fd5640 1 --2- 172.21.15.60:0/2824031251 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bb013b480 0x7f1bb01316e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:27.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.432+0000 7f1bb5fd5640 1 --2- 172.21.15.60:0/2824031251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bb013b020 0x7f1bb01311a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:27.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.432+0000 7f1bb5fd5640 1 --2- 172.21.15.60:0/2824031251 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1bb00b40b0 0x7f1bb013aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:27.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.432+0000 7f1bb5fd5640 1 -- 172.21.15.60:0/2824031251 >> 172.21.15.60:0/2824031251 conn(0x7f1bb00608e0 msgr2=0x7f1bb01416f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:27.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.432+0000 7f1bb5fd5640 1 -- 172.21.15.60:0/2824031251 shutdown_connections 2024-03-27T21:40:27.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.432+0000 7f1bb5fd5640 1 -- 172.21.15.60:0/2824031251 wait complete. 2024-03-27T21:40:27.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:27.432+0000 7f1bb5fd5640 1 librados: shutdown 2024-03-27T21:40:27.491 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:27.504+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:27.537 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:27.537 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:27.546 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-27T21:40:27.591 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:27.605+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:27.692 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:27.706+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:27.717 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:27.717 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:27.730 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-27T21:40:27.816 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:27.830+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:27.894 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:27.907+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:27.943 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:27.943 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:27.955 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-27T21:40:27.996 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:28.010+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:28.097 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:28.110+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:28.166 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:28.167 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:28.179 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:40:28.197 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:28.211+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:28.298 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:28.311+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:28.334 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:28.334 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:28.344 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:40:28.399 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:28.413+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:28.426 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-27T21:40:28.459 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:28.459 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:28.469 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:40:28.510 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:28.524+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:28.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.546+0000 7f161d9c4640 10 monclient: get_monmap_and_config 2024-03-27T21:40:28.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.546+0000 7f161d9c4640 10 monclient: build_initial_monmap 2024-03-27T21:40:28.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.546+0000 7f161d9c4640 10 monclient: monmap: 2024-03-27T21:40:28.533 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:28.533 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:28.533 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:28.547780+0000 2024-03-27T21:40:28.533 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:28.547780+0000 2024-03-27T21:40:28.533 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:28.534 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:28.534 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:28.534 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:28.534 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:28.534 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:28.534 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:28.534 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:28.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.547+0000 7f161d9c4640 1 Processor -- start 2024-03-27T21:40:28.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.547+0000 7f161d9c4640 1 -- start start 2024-03-27T21:40:28.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.547+0000 7f161d9c4640 10 monclient: init 2024-03-27T21:40:28.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f161d9c4640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:28.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f161d9c4640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-03-27T21:40:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f161d9c4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16180b40b0 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f161d9c4640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f161813b020 0x7f161813b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f161d9c4640 10 monclient(hunting): picked mon.noname-b con 0x7f161813b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f161d9c4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16181416f0 0x7f1618141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f161d9c4640 10 monclient(hunting): picked mon.noname-a con 0x7f16181416f0 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f161d9c4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16180b3830 con 0x7f16180b40b0 2024-03-27T21:40:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f161d9c4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f161d9c4640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16180b4490 con 0x7f16181416f0 2024-03-27T21:40:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f161d9c4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f161d9c4640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16180aa520 con 0x7f161813b020 2024-03-27T21:40:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f161d9c4640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f161d9c4640 10 monclient(hunting): authenticate will time out at 869.769480s 2024-03-27T21:40:28.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f16167fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f161813b020 0x7f161813b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:28.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f16167fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f161813b020 0x7f161813b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47566/0 (socket says 172.21.15.60:47566) 2024-03-27T21:40:28.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f16167fc640 1 -- 172.21.15.60:0/1092617407 learned_addr learned my addr 172.21.15.60:0/1092617407 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:28.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f16167fc640 10 monclient(hunting): get_auth_request con 0x7f161813b020 auth_method 0 2024-03-27T21:40:28.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f16167fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:28.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f16167fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:28.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f16167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:28.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f16177fe640 1 --2- 172.21.15.60:0/1092617407 >> v2:172.21.15.60:3300/0 conn(0x7f16181416f0 0x7f1618141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:28.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f16167fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:28.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f16167fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:28.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.548+0000 7f16167fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:28.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.549+0000 7f16177fe640 10 monclient(hunting): get_auth_request con 0x7f16181416f0 auth_method 0 2024-03-27T21:40:28.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.549+0000 7f16177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:28.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.549+0000 7f16177fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:28.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.549+0000 7f16177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:28.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.549+0000 7f16167fc640 10 monclient(hunting): handle_auth_done global_id 4520 payload 274 2024-03-27T21:40:28.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.549+0000 7f16167fc640 1 -- 172.21.15.60:0/1092617407 >> v2:172.21.15.60:3300/0 conn(0x7f16181416f0 msgr2=0x7f1618141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:28.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.549+0000 7f1615ffb640 1 -- 172.21.15.60:0/1092617407 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886224327 0 0) 0x7f16180b3830 con 0x7f16180b40b0 2024-03-27T21:40:28.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.549+0000 7f16167fc640 1 --2- 172.21.15.60:0/1092617407 >> v2:172.21.15.60:3300/0 conn(0x7f16181416f0 0x7f1618141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.549+0000 7f16167fc640 1 -- 172.21.15.60:0/1092617407 >> v1:172.21.15.191:6789/0 conn(0x7f16180b40b0 legacy=0x7f16180a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:28.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.549+0000 7f16167fc640 10 monclient: _finish_hunting 0 2024-03-27T21:40:28.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.549+0000 7f16167fc640 1 monclient: found mon.noname-b 2024-03-27T21:40:28.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.549+0000 7f16167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:28.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.549+0000 7f16167fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:28.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.549+0000 7f16167fc640 1 -- 172.21.15.60:0/1092617407 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16180a9470 con 0x7f161813b020 2024-03-27T21:40:28.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.549+0000 7f1615ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:40:28.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.549+0000 7f16177fe640 1 --2- 172.21.15.60:0/1092617407 >> v2:172.21.15.60:3300/0 conn(0x7f16181416f0 0x7f1618141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:40:28.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f16167fc640 1 --2- 172.21.15.60:0/1092617407 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f161813b020 0x7f161813b480 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f160c009870 tx=0x7f160c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80b49ff6a213b051 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:28.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f1615ffb640 1 -- 172.21.15.60:0/1092617407 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160c030840 con 0x7f161813b020 2024-03-27T21:40:28.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f1615ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:28.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f1615ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:28.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f1615ffb640 10 monclient: dump: 2024-03-27T21:40:28.540 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:28.540 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:28.540 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:28.540 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:28.540 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:28.540 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:28.540 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:28.540 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:28.541 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:28.541 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:28.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f1615ffb640 10 monclient: _finish_auth 0 2024-03-27T21:40:28.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f1615ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:40:28.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f1615ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:28.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f161d9c4640 5 monclient: authenticate success, global_id 4520 2024-03-27T21:40:28.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f1615ffb640 1 -- 172.21.15.60:0/1092617407 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160c0309a0 con 0x7f161813b020 2024-03-27T21:40:28.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f161d9c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:28.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f1615ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:28.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f161d9c4640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:28.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f161d9c4640 4 set_mon_vals no callback set 2024-03-27T21:40:28.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f1615ffb640 1 -- 172.21.15.60:0/1092617407 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160c030d30 con 0x7f161813b020 2024-03-27T21:40:28.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f1615ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:28.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f1615ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:28.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f1615ffb640 10 monclient: dump: 2024-03-27T21:40:28.542 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:28.542 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:28.542 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:28.542 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:28.542 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:28.542 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:28.542 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:28.542 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:28.542 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:28.542 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:28.542 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:40:28.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f161d9c4640 10 monclient: shutdown 2024-03-27T21:40:28.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f161d9c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:28.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f161d9c4640 1 -- 172.21.15.60:0/1092617407 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f161813b020 msgr2=0x7f161813b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:28.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.550+0000 7f161d9c4640 1 --2- 172.21.15.60:0/1092617407 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f161813b020 0x7f161813b480 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f160c009870 tx=0x7f160c02b980 comp rx=0 tx=0).stop 2024-03-27T21:40:28.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.551+0000 7f161d9c4640 1 -- 172.21.15.60:0/1092617407 shutdown_connections 2024-03-27T21:40:28.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.551+0000 7f161d9c4640 1 --2- 172.21.15.60:0/1092617407 >> v2:172.21.15.60:3300/0 conn(0x7f16181416f0 0x7f1618141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.551+0000 7f161d9c4640 1 --2- 172.21.15.60:0/1092617407 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f161813b020 0x7f161813b480 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.551+0000 7f161d9c4640 1 -- 172.21.15.60:0/1092617407 >> 172.21.15.60:0/1092617407 conn(0x7f16180608e0 msgr2=0x7f1618060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:28.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.551+0000 7f161d9c4640 1 -- 172.21.15.60:0/1092617407 shutdown_connections 2024-03-27T21:40:28.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.551+0000 7f161d9c4640 1 -- 172.21.15.60:0/1092617407 wait complete. 2024-03-27T21:40:28.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.551+0000 7f161d9c4640 10 monclient: build_initial_monmap 2024-03-27T21:40:28.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.551+0000 7f161d9c4640 10 monclient: monmap: 2024-03-27T21:40:28.549 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:28.549 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:28.549 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:28.549 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:28.549 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:28.550 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:28.550 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:28.550 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:28.550 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:28.550 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:28.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 1 librados: starting msgr at 2024-03-27T21:40:28.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 1 librados: starting objecter 2024-03-27T21:40:28.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 1 Processor -- start 2024-03-27T21:40:28.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 1 -- start start 2024-03-27T21:40:28.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 1 librados: setting wanted keys 2024-03-27T21:40:28.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 1 librados: calling monclient init 2024-03-27T21:40:28.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 10 monclient: init 2024-03-27T21:40:28.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:28.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:40:28.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16180b40b0 0x7f16180aaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:28.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 10 monclient(hunting): picked mon.noname-b con 0x7f16180b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:28.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f16181416f0 0x7f161813e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:28.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 10 monclient(hunting): picked mon.noname-c con 0x7f16181416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:28.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16180aaa20 0x7f16181d9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:28.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 10 monclient(hunting): picked mon.noname-a con 0x7f16180aaa20 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:28.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:28.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16181d99c0 con 0x7f16180aaa20 2024-03-27T21:40:28.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:28.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161813d480 con 0x7f16180b40b0 2024-03-27T21:40:28.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:28.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f161813b9c0 con 0x7f16181416f0 2024-03-27T21:40:28.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:28.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.552+0000 7f161d9c4640 10 monclient(hunting): authenticate will time out at 869.773881s 2024-03-27T21:40:28.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16167fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f16181416f0 0x7f161813e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:28.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16167fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f16181416f0 0x7f161813e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47578/0 (socket says 172.21.15.60:47578) 2024-03-27T21:40:28.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16167fc640 1 -- 172.21.15.60:0/599651106 learned_addr learned my addr 172.21.15.60:0/599651106 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:28.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16167fc640 10 monclient(hunting): get_auth_request con 0x7f16181416f0 auth_method 0 2024-03-27T21:40:28.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16167fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:28.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16167fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:28.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:28.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16167fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:28.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16167fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:28.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16167fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:28.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16177fe640 1 --2- 172.21.15.60:0/599651106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16180aaa20 0x7f16181d9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:28.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f1616ffd640 1 --2- 172.21.15.60:0/599651106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16180b40b0 0x7f16180aaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:28.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f1616ffd640 10 monclient(hunting): get_auth_request con 0x7f16180b40b0 auth_method 0 2024-03-27T21:40:28.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f1616ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:28.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f1616ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:28.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f1616ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:28.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16177fe640 10 monclient(hunting): get_auth_request con 0x7f16180aaa20 auth_method 0 2024-03-27T21:40:28.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:28.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16177fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:28.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:28.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f1616ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:28.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f1616ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:28.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f1616ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:28.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16167fc640 10 monclient(hunting): handle_auth_done global_id 4523 payload 824 2024-03-27T21:40:28.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16167fc640 1 -- 172.21.15.60:0/599651106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16180b40b0 msgr2=0x7f16180aaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:28.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16167fc640 1 --2- 172.21.15.60:0/599651106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16180b40b0 0x7f16180aaee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16167fc640 1 -- 172.21.15.60:0/599651106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16180aaa20 msgr2=0x7f16181d9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:28.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16167fc640 1 --2- 172.21.15.60:0/599651106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16180aaa20 0x7f16181d9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16167fc640 10 monclient: _finish_hunting 0 2024-03-27T21:40:28.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16167fc640 1 monclient: found mon.noname-c 2024-03-27T21:40:28.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:28.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16167fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:40:28.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16167fc640 1 -- 172.21.15.60:0/599651106 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160c0094d0 con 0x7f16181416f0 2024-03-27T21:40:28.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16177fe640 1 --2- 172.21.15.60:0/599651106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16180aaa20 0x7f16181d9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:40:28.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f16167fc640 1 --2- 172.21.15.60:0/599651106 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f16181416f0 0x7f161813e9c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f160c02bf10 tx=0x7f160c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:28.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f15fffff640 10 client.?.objecter ms_handle_connect 0x7f16181416f0 2024-03-27T21:40:28.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.553+0000 7f15fffff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:28.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.554+0000 7f15fffff640 1 -- 172.21.15.60:0/599651106 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160c039d60 con 0x7f16181416f0 2024-03-27T21:40:28.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.554+0000 7f15fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:28.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.554+0000 7f15fffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:28.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.554+0000 7f15fffff640 10 monclient: dump: 2024-03-27T21:40:28.554 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:28.554 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:28.555 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:28.555 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:28.555 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:28.555 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:28.555 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:28.555 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:28.555 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:28.555 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:28.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.554+0000 7f15fffff640 10 monclient: _finish_auth 0 2024-03-27T21:40:28.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.554+0000 7f15fffff640 10 monclient: _check_auth_tickets 2024-03-27T21:40:28.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.554+0000 7f15fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:28.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.554+0000 7f161d9c4640 5 monclient: authenticate success, global_id 4523 2024-03-27T21:40:28.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.554+0000 7f15fffff640 1 -- 172.21.15.60:0/599651106 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160c044430 con 0x7f16181416f0 2024-03-27T21:40:28.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.554+0000 7f161d9c4640 10 monclient: _renew_subs 2024-03-27T21:40:28.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.554+0000 7f161d9c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:28.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.554+0000 7f161d9c4640 1 -- 172.21.15.60:0/599651106 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16181da340 con 0x7f16181416f0 2024-03-27T21:40:28.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.554+0000 7f15fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:28.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.554+0000 7f161d9c4640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:28.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.554+0000 7f161d9c4640 10 monclient: _renew_subs 2024-03-27T21:40:28.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.554+0000 7f15fffff640 1 -- 172.21.15.60:0/599651106 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160c0447e0 con 0x7f16181416f0 2024-03-27T21:40:28.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.554+0000 7f161d9c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:28.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.554+0000 7f161d9c4640 1 -- 172.21.15.60:0/599651106 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16181da7e0 con 0x7f16181416f0 2024-03-27T21:40:28.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.554+0000 7f161d9c4640 1 librados: init done 2024-03-27T21:40:28.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.554+0000 7f16157fa640 4 set_mon_vals no callback set 2024-03-27T21:40:28.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.555+0000 7f15fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:28.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.555+0000 7f15fffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:28.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.555+0000 7f15fffff640 10 monclient: dump: 2024-03-27T21:40:28.556 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:28.556 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:28.556 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:28.557 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:28.557 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:28.557 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:28.557 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:28.557 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:28.557 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:28.557 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:28.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.555+0000 7f161d9c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:28.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.555+0000 7f161d9c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:28.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.555+0000 7f161d9c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:28.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.555+0000 7f161d9c4640 1 -- 172.21.15.60:0/599651106 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f161813bb80 con 0x7f16181416f0 2024-03-27T21:40:28.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.556+0000 7f15fffff640 1 -- 172.21.15.60:0/599651106 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f160c0433e0 con 0x7f16181416f0 2024-03-27T21:40:28.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.557+0000 7f15fffff640 1 --2- 172.21.15.60:0/599651106 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f160406c590 0x7f160406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:28.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.557+0000 7f15fffff640 1 -- 172.21.15.60:0/599651106 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f160c03e070 con 0x7f16181416f0 2024-03-27T21:40:28.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.557+0000 7f1616ffd640 1 --2- 172.21.15.60:0/599651106 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f160406c590 0x7f160406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:28.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.557+0000 7f15fffff640 10 client.4523.objecter ms_dispatch 0x7f1618000b90 osd_map(37..37 src has 1..37) v4 2024-03-27T21:40:28.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.557+0000 7f15fffff640 3 client.4523.objecter handle_osd_map got epochs [37,37] > 0 2024-03-27T21:40:28.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.557+0000 7f15fffff640 3 client.4523.objecter handle_osd_map decoding full epoch 37 2024-03-27T21:40:28.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.557+0000 7f1616ffd640 10 monclient: get_auth_request con 0x7f160406c590 auth_method 0 2024-03-27T21:40:28.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.557+0000 7f1616ffd640 1 --2- 172.21.15.60:0/599651106 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f160406c590 0x7f160406ea50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f16080012e0 tx=0x7f1608009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:28.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.557+0000 7f15fffff640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-27T21:40:28.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.558+0000 7f15fffff640 10 client.4523.objecter ms_handle_connect 0x7f160406c590 2024-03-27T21:40:28.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.561+0000 7f15fffff640 1 -- 172.21.15.60:0/599651106 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f160c0765d0 con 0x7f16181416f0 2024-03-27T21:40:28.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.561+0000 7f15fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:28.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.561+0000 7f15fffff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:28.601 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:28.615+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:28.628 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:28.629 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:28.638 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:40:28.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.692+0000 7fd231dc0640 10 monclient: get_monmap_and_config 2024-03-27T21:40:28.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.692+0000 7fd231dc0640 10 monclient: build_initial_monmap 2024-03-27T21:40:28.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.692+0000 7fd231dc0640 10 monclient: monmap: 2024-03-27T21:40:28.682 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:28.682 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:28.682 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:28.693770+0000 2024-03-27T21:40:28.682 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:28.693770+0000 2024-03-27T21:40:28.682 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:28.683 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:28.683 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:28.683 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:28.683 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:28.683 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:28.683 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:28.683 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:28.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.693+0000 7fd231dc0640 1 Processor -- start 2024-03-27T21:40:28.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.693+0000 7fd231dc0640 1 -- start start 2024-03-27T21:40:28.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.693+0000 7fd231dc0640 10 monclient: init 2024-03-27T21:40:28.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd231dc0640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:28.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd231dc0640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-03-27T21:40:28.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd231dc0640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd22c0b40b0 0x7fd22c0a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:28.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd231dc0640 10 monclient(hunting): picked mon.noname-c con 0x7fd22c0b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:28.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd231dc0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd22c13b020 0x7fd22c13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:28.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd231dc0640 10 monclient(hunting): picked mon.noname-a con 0x7fd22c13b020 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:28.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd231dc0640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd22c1416f0 0x7fd22c141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:28.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd231dc0640 10 monclient(hunting): picked mon.noname-b con 0x7fd22c1416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:28.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd231dc0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:28.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd231dc0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd22c0b4490 con 0x7fd22c13b020 2024-03-27T21:40:28.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd231dc0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:28.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd231dc0640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd22c0b3830 con 0x7fd22c0b40b0 2024-03-27T21:40:28.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd231dc0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:28.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd231dc0640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c0aa520 con 0x7fd22c1416f0 2024-03-27T21:40:28.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd231dc0640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:28.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd231dc0640 10 monclient(hunting): authenticate will time out at 869.915425s 2024-03-27T21:40:28.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd22affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd22c13b020 0x7fd22c13b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:28.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd22affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd22c13b020 0x7fd22c13b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40394/0 (socket says 172.21.15.60:40394) 2024-03-27T21:40:28.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd22affd640 1 -- 172.21.15.60:0/3171362001 learned_addr learned my addr 172.21.15.60:0/3171362001 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:28.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd22affd640 10 monclient(hunting): get_auth_request con 0x7fd22c13b020 auth_method 0 2024-03-27T21:40:28.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd22affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:28.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd22affd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:28.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd22affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:28.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd22bfff640 1 --2- 172.21.15.60:0/3171362001 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd22c1416f0 0x7fd22c141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:28.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd22bfff640 10 monclient(hunting): get_auth_request con 0x7fd22c1416f0 auth_method 0 2024-03-27T21:40:28.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd22bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:28.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd22bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:28.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd22bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:28.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd22affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:28.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd22affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:28.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd22affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:28.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd22bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:28.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.694+0000 7fd22bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:28.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.695+0000 7fd22bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:28.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.695+0000 7fd22affd640 10 monclient(hunting): handle_auth_done global_id 4653 payload 274 2024-03-27T21:40:28.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.695+0000 7fd22affd640 1 -- 172.21.15.60:0/3171362001 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd22c1416f0 msgr2=0x7fd22c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:28.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.695+0000 7fd22affd640 1 --2- 172.21.15.60:0/3171362001 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd22c1416f0 0x7fd22c141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.695+0000 7fd22affd640 1 -- 172.21.15.60:0/3171362001 >> v2:172.21.15.191:3300/0 conn(0x7fd22c0b40b0 msgr2=0x7fd22c0a8b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T21:40:28.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.695+0000 7fd22affd640 1 --2- 172.21.15.60:0/3171362001 >> v2:172.21.15.191:3300/0 conn(0x7fd22c0b40b0 0x7fd22c0a8b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.695+0000 7fd22affd640 10 monclient: _finish_hunting 0 2024-03-27T21:40:28.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.695+0000 7fd22affd640 1 monclient: found mon.noname-a 2024-03-27T21:40:28.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.695+0000 7fd22affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:28.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.695+0000 7fd22affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:28.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.695+0000 7fd22affd640 1 -- 172.21.15.60:0/3171362001 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22c0a9470 con 0x7fd22c13b020 2024-03-27T21:40:28.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.696+0000 7fd22affd640 1 --2- 172.21.15.60:0/3171362001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd22c13b020 0x7fd22c13b480 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0b4810 tx=0x7fd22002b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=324099fa957868c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:28.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.696+0000 7fd22a7fc640 1 -- 172.21.15.60:0/3171362001 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd220031890 con 0x7fd22c13b020 2024-03-27T21:40:28.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.696+0000 7fd22a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:28.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.696+0000 7fd22a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:28.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.696+0000 7fd22a7fc640 10 monclient: dump: 2024-03-27T21:40:28.696 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:28.696 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:28.696 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:28.696 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:28.696 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:28.696 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:28.696 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:28.696 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:28.696 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:28.696 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:28.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.696+0000 7fd22a7fc640 10 monclient: _finish_auth 0 2024-03-27T21:40:28.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.696+0000 7fd22a7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:40:28.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.696+0000 7fd22a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:28.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.696+0000 7fd22a7fc640 1 -- 172.21.15.60:0/3171362001 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2200319f0 con 0x7fd22c13b020 2024-03-27T21:40:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.696+0000 7fd22a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.696+0000 7fd22a7fc640 1 -- 172.21.15.60:0/3171362001 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd220031d80 con 0x7fd22c13b020 2024-03-27T21:40:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.696+0000 7fd22a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.696+0000 7fd22a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.696+0000 7fd22a7fc640 10 monclient: dump: 2024-03-27T21:40:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:28.697 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:28.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.696+0000 7fd231dc0640 5 monclient: authenticate success, global_id 4653 2024-03-27T21:40:28.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.696+0000 7fd231dc0640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:28.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.696+0000 7fd231dc0640 4 set_mon_vals no callback set 2024-03-27T21:40:28.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.697+0000 7fd231dc0640 10 monclient: shutdown 2024-03-27T21:40:28.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.697+0000 7fd231dc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:28.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.697+0000 7fd231dc0640 1 -- 172.21.15.60:0/3171362001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd22c13b020 msgr2=0x7fd22c13b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:28.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.697+0000 7fd231dc0640 1 --2- 172.21.15.60:0/3171362001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd22c13b020 0x7fd22c13b480 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0b4810 tx=0x7fd22002b930 comp rx=0 tx=0).stop 2024-03-27T21:40:28.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.697+0000 7fd231dc0640 1 -- 172.21.15.60:0/3171362001 shutdown_connections 2024-03-27T21:40:28.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.697+0000 7fd231dc0640 1 --2- 172.21.15.60:0/3171362001 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd22c1416f0 0x7fd22c141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.697+0000 7fd231dc0640 1 --2- 172.21.15.60:0/3171362001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd22c13b020 0x7fd22c13b480 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.697+0000 7fd231dc0640 1 --2- 172.21.15.60:0/3171362001 >> v2:172.21.15.191:3300/0 conn(0x7fd22c0b40b0 0x7fd22c0a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.697+0000 7fd231dc0640 1 -- 172.21.15.60:0/3171362001 >> 172.21.15.60:0/3171362001 conn(0x7fd22c0608e0 msgr2=0x7fd22c060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:28.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.697+0000 7fd231dc0640 1 -- 172.21.15.60:0/3171362001 shutdown_connections 2024-03-27T21:40:28.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.698+0000 7fd231dc0640 1 -- 172.21.15.60:0/3171362001 wait complete. 2024-03-27T21:40:28.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.698+0000 7fd231dc0640 10 monclient: build_initial_monmap 2024-03-27T21:40:28.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.698+0000 7fd231dc0640 10 monclient: monmap: 2024-03-27T21:40:28.699 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:28.699 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:28.699 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:28.699 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:28.699 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:28.699 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:28.699 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:28.699 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:28.699 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:28.699 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:28.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.698+0000 7fd231dc0640 1 librados: starting msgr at 2024-03-27T21:40:28.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.698+0000 7fd231dc0640 1 librados: starting objecter 2024-03-27T21:40:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.699+0000 7fd231dc0640 1 Processor -- start 2024-03-27T21:40:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.699+0000 7fd231dc0640 1 -- start start 2024-03-27T21:40:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.699+0000 7fd231dc0640 1 librados: setting wanted keys 2024-03-27T21:40:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.699+0000 7fd231dc0640 1 librados: calling monclient init 2024-03-27T21:40:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.699+0000 7fd231dc0640 10 monclient: init 2024-03-27T21:40:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.699+0000 7fd231dc0640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.699+0000 7fd231dc0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:40:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.699+0000 7fd231dc0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd22c0b40b0 0x7fd22c13e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.699+0000 7fd231dc0640 10 monclient(hunting): picked mon.noname-a con 0x7fd22c0b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.699+0000 7fd231dc0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd22c13b020 0x7fd22c0aae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.700+0000 7fd231dc0640 10 monclient(hunting): picked mon.noname-b con 0x7fd22c13b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.700+0000 7fd231dc0640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd22c1416f0 0x7fd22c06eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.700+0000 7fd231dc0640 10 monclient(hunting): picked mon.noname-c con 0x7fd22c1416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.700+0000 7fd231dc0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.700+0000 7fd231dc0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c0ab340 con 0x7fd22c0b40b0 2024-03-27T21:40:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.700+0000 7fd231dc0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.700+0000 7fd231dc0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c0aa9e0 con 0x7fd22c13b020 2024-03-27T21:40:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.700+0000 7fd231dc0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.700+0000 7fd231dc0640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c06e640 con 0x7fd22c1416f0 2024-03-27T21:40:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.700+0000 7fd22bfff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd22c1416f0 0x7fd22c06eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.700+0000 7fd22bfff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd22c1416f0 0x7fd22c06eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:47592/0 (socket says 172.21.15.60:47592) 2024-03-27T21:40:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.700+0000 7fd22bfff640 1 -- 172.21.15.60:0/2638420870 learned_addr learned my addr 172.21.15.60:0/2638420870 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.700+0000 7fd231dc0640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.700+0000 7fd231dc0640 10 monclient(hunting): authenticate will time out at 869.921838s 2024-03-27T21:40:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.700+0000 7fd22bfff640 10 monclient(hunting): get_auth_request con 0x7fd22c1416f0 auth_method 0 2024-03-27T21:40:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.700+0000 7fd22bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.700+0000 7fd22bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22b7fe640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd22c0b40b0 0x7fd22c13e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22b7fe640 10 monclient(hunting): get_auth_request con 0x7fd22c0b40b0 auth_method 0 2024-03-27T21:40:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22affd640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd22c13b020 0x7fd22c0aae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22affd640 10 monclient(hunting): get_auth_request con 0x7fd22c13b020 auth_method 0 2024-03-27T21:40:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22affd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22b7fe640 10 monclient(hunting): handle_auth_done global_id 4656 payload 824 2024-03-27T21:40:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22b7fe640 1 -- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd22c1416f0 msgr2=0x7fd22c06eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22b7fe640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd22c1416f0 0x7fd22c06eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22b7fe640 1 -- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd22c13b020 msgr2=0x7fd22c0aae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22b7fe640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd22c13b020 0x7fd22c0aae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22b7fe640 10 monclient: _finish_hunting 0 2024-03-27T21:40:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22b7fe640 1 monclient: found mon.noname-a 2024-03-27T21:40:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22b7fe640 1 -- 172.21.15.60:0/2638420870 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2200094d0 con 0x7fd22c0b40b0 2024-03-27T21:40:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd213fff640 10 client.?.objecter ms_handle_connect 0x7fd22c0b40b0 2024-03-27T21:40:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd213fff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.701+0000 7fd22b7fe640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd22c0b40b0 0x7fd22c13e520 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd21c009fd0 tx=0x7fd21c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21c011830 con 0x7fd22c0b40b0 2024-03-27T21:40:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd213fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd213fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd213fff640 10 monclient: dump: 2024-03-27T21:40:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:28.705 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd213fff640 10 monclient: _finish_auth 0 2024-03-27T21:40:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd213fff640 10 monclient: _check_auth_tickets 2024-03-27T21:40:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd213fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd21c0119b0 con 0x7fd22c0b40b0 2024-03-27T21:40:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd213fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21c011da0 con 0x7fd22c0b40b0 2024-03-27T21:40:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd213fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd213fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd213fff640 10 monclient: dump: 2024-03-27T21:40:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:28.706 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd229ffb640 4 set_mon_vals no callback set 2024-03-27T21:40:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd231dc0640 5 monclient: authenticate success, global_id 4656 2024-03-27T21:40:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd231dc0640 10 monclient: _renew_subs 2024-03-27T21:40:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd231dc0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd231dc0640 1 -- 172.21.15.60:0/2638420870 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd22c06d0f0 con 0x7fd22c0b40b0 2024-03-27T21:40:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd231dc0640 10 client.4656.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd231dc0640 10 monclient: _renew_subs 2024-03-27T21:40:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd231dc0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd231dc0640 1 -- 172.21.15.60:0/2638420870 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd22c06f0c0 con 0x7fd22c0b40b0 2024-03-27T21:40:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.702+0000 7fd231dc0640 1 librados: init done 2024-03-27T21:40:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.704+0000 7fd211ffb640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.704+0000 7fd211ffb640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.704+0000 7fd211ffb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.704+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd22c069470 con 0x7fd22c0b40b0 2024-03-27T21:40:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.705+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd21c01a400 con 0x7fd22c0b40b0 2024-03-27T21:40:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.705+0000 7fd213fff640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd21806c590 0x7fd21806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.705+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fd21c089190 con 0x7fd22c0b40b0 2024-03-27T21:40:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.705+0000 7fd213fff640 10 client.4656.objecter ms_dispatch 0x7fd22c000b90 osd_map(37..37 src has 1..37) v4 2024-03-27T21:40:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.705+0000 7fd213fff640 3 client.4656.objecter handle_osd_map got epochs [37,37] > 0 2024-03-27T21:40:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.705+0000 7fd213fff640 3 client.4656.objecter handle_osd_map decoding full epoch 37 2024-03-27T21:40:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.705+0000 7fd213fff640 20 client.4656.objecter dump_active .. 0 homeless 2024-03-27T21:40:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.705+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v37) v1 ==== 54+0+16 (secure 0 0 0) 0x7fd21c090070 con 0x7fd22c0b40b0 2024-03-27T21:40:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.705+0000 7fd213fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.705+0000 7fd213fff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.706+0000 7fd22affd640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd21806c590 0x7fd21806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.706+0000 7fd22affd640 10 monclient: get_auth_request con 0x7fd21806c590 auth_method 0 2024-03-27T21:40:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.706+0000 7fd211ffb640 10 client.4656.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.706+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign -1 1 2024-03-27T21:40:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.706+0000 7fd211ffb640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fd22c06dcc0 0x7fd22c1ddc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.706+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c073e70 osd=0 3 2024-03-27T21:40:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.706+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c073e70 osd=0 3 2024-03-27T21:40:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.706+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c073e70 osd=0 3 2024-03-27T21:40:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.706+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_remove -1 1 2024-03-27T21:40:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.706+0000 7fd211ffb640 20 client.4656.objecter get_session s=0x7fd22c073e70 osd=0 3 2024-03-27T21:40:28.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.706+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign 0 1 2024-03-27T21:40:28.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.706+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c073e70 osd=0 4 2024-03-27T21:40:28.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.706+0000 7fd211ffb640 10 client.4656.objecter _send_command 1 2024-03-27T21:40:28.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.706+0000 7fd22affd640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd21806c590 0x7fd21806ea50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd220036000 tx=0x7fd22003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:28.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.706+0000 7fd213fff640 10 client.4656.objecter ms_handle_connect 0x7fd21806c590 2024-03-27T21:40:28.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.706+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd22c144990 con 0x7fd22c06dcc0 2024-03-27T21:40:28.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.707+0000 7fd22bfff640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fd22c06dcc0 0x7fd22c1ddc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:28.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.707+0000 7fd22bfff640 10 monclient: get_auth_request con 0x7fd22c06dcc0 auth_method 0 2024-03-27T21:40:28.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.707+0000 7fd213fff640 10 client.4656.objecter ms_handle_connect 0x7fd22c06dcc0 2024-03-27T21:40:28.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.707+0000 7fd22bfff640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fd22c06dcc0 0x7fd22c1ddc50 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:28.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.708+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fd22400f040 con 0x7fd22c06dcc0 2024-03-27T21:40:28.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.708+0000 7fd213fff640 10 client.4656.objecter ms_dispatch 0x7fd22c000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:28.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.708+0000 7fd213fff640 10 client.4656.objecter _finish_command 1 = system:0 2024-03-27T21:40:28.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.708+0000 7fd213fff640 20 client.4656.objecter put_session s=0x7fd22c073e70 osd=0 4 2024-03-27T21:40:28.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.708+0000 7fd213fff640 15 client.4656.objecter _session_command_op_remove 0 1 2024-03-27T21:40:28.711 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:28.724+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:28.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.730+0000 7fd211ffb640 10 client.4656.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:28.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.730+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign -1 2 2024-03-27T21:40:28.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.730+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c073e70 osd=0 3 2024-03-27T21:40:28.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.730+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c073e70 osd=0 3 2024-03-27T21:40:28.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.730+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c073e70 osd=0 3 2024-03-27T21:40:28.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.730+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_remove -1 2 2024-03-27T21:40:28.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.730+0000 7fd211ffb640 20 client.4656.objecter get_session s=0x7fd22c073e70 osd=0 3 2024-03-27T21:40:28.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.730+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign 0 2 2024-03-27T21:40:28.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.730+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c073e70 osd=0 4 2024-03-27T21:40:28.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.730+0000 7fd211ffb640 10 client.4656.objecter _send_command 2 2024-03-27T21:40:28.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.730+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd22c06cbc0 con 0x7fd22c06dcc0 2024-03-27T21:40:28.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.732+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd22c06cbc0 con 0x7fd22c06dcc0 2024-03-27T21:40:28.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.732+0000 7fd213fff640 10 client.4656.objecter ms_dispatch 0x7fd22c000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:28.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.732+0000 7fd213fff640 10 client.4656.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:28.719 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:28.722 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:28.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.732+0000 7fd213fff640 20 client.4656.objecter put_session s=0x7fd22c073e70 osd=0 4 2024-03-27T21:40:28.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.732+0000 7fd213fff640 15 client.4656.objecter _session_command_op_remove 0 2 2024-03-27T21:40:28.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.733+0000 7fd211ffb640 10 client.4656.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:28.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.733+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign -1 3 2024-03-27T21:40:28.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.733+0000 7fd211ffb640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fd22c065a90 0x7fd22c065e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:28.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.733+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c06cd30 osd=1 3 2024-03-27T21:40:28.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.733+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c06cd30 osd=1 3 2024-03-27T21:40:28.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.733+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c06cd30 osd=1 3 2024-03-27T21:40:28.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.733+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_remove -1 3 2024-03-27T21:40:28.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.733+0000 7fd211ffb640 20 client.4656.objecter get_session s=0x7fd22c06cd30 osd=1 3 2024-03-27T21:40:28.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.733+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign 1 3 2024-03-27T21:40:28.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.733+0000 7fd22b7fe640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fd22c065a90 0x7fd22c065e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:28.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.733+0000 7fd22b7fe640 10 monclient: get_auth_request con 0x7fd22c065a90 auth_method 0 2024-03-27T21:40:28.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.733+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c06cd30 osd=1 4 2024-03-27T21:40:28.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.733+0000 7fd211ffb640 10 client.4656.objecter _send_command 3 2024-03-27T21:40:28.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.734+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd22c1de950 con 0x7fd22c065a90 2024-03-27T21:40:28.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.734+0000 7fd213fff640 10 client.4656.objecter ms_handle_connect 0x7fd22c065a90 2024-03-27T21:40:28.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.734+0000 7fd22b7fe640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fd22c065a90 0x7fd22c065e70 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:28.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.735+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fd22c1de950 con 0x7fd22c065a90 2024-03-27T21:40:28.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.735+0000 7fd213fff640 10 client.4656.objecter ms_dispatch 0x7fd22c000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:28.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.735+0000 7fd213fff640 10 client.4656.objecter _finish_command 3 = system:0 2024-03-27T21:40:28.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.735+0000 7fd213fff640 20 client.4656.objecter put_session s=0x7fd22c06cd30 osd=1 4 2024-03-27T21:40:28.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.735+0000 7fd213fff640 15 client.4656.objecter _session_command_op_remove 1 3 2024-03-27T21:40:28.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.764+0000 7fd211ffb640 10 client.4656.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:28.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.764+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign -1 4 2024-03-27T21:40:28.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.764+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c06cd30 osd=1 3 2024-03-27T21:40:28.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.764+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c06cd30 osd=1 3 2024-03-27T21:40:28.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.764+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c06cd30 osd=1 3 2024-03-27T21:40:28.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.764+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_remove -1 4 2024-03-27T21:40:28.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.764+0000 7fd211ffb640 20 client.4656.objecter get_session s=0x7fd22c06cd30 osd=1 3 2024-03-27T21:40:28.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.764+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign 1 4 2024-03-27T21:40:28.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.764+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c06cd30 osd=1 4 2024-03-27T21:40:28.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.764+0000 7fd211ffb640 10 client.4656.objecter _send_command 4 2024-03-27T21:40:28.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.764+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd22c1de3d0 con 0x7fd22c065a90 2024-03-27T21:40:28.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.766+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd22c1de3d0 con 0x7fd22c065a90 2024-03-27T21:40:28.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.766+0000 7fd213fff640 10 client.4656.objecter ms_dispatch 0x7fd22c000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:28.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.766+0000 7fd213fff640 10 client.4656.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:28.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.766+0000 7fd213fff640 20 client.4656.objecter put_session s=0x7fd22c06cd30 osd=1 4 2024-03-27T21:40:28.753 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:28.754 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:28.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.766+0000 7fd213fff640 15 client.4656.objecter _session_command_op_remove 1 4 2024-03-27T21:40:28.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.766+0000 7fd211ffb640 10 client.4656.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:28.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.766+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign -1 5 2024-03-27T21:40:28.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.766+0000 7fd211ffb640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7fd22c1e9650 0x7fd22c1eba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:28.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.766+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c1de780 osd=2 3 2024-03-27T21:40:28.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.766+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c1de780 osd=2 3 2024-03-27T21:40:28.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.766+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c1de780 osd=2 3 2024-03-27T21:40:28.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.766+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_remove -1 5 2024-03-27T21:40:28.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.766+0000 7fd211ffb640 20 client.4656.objecter get_session s=0x7fd22c1de780 osd=2 3 2024-03-27T21:40:28.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.766+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign 2 5 2024-03-27T21:40:28.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.766+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c1de780 osd=2 4 2024-03-27T21:40:28.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.766+0000 7fd211ffb640 10 client.4656.objecter _send_command 5 2024-03-27T21:40:28.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.766+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd22c1ee810 con 0x7fd22c1e9650 2024-03-27T21:40:28.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.766+0000 7fd22affd640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7fd22c1e9650 0x7fd22c1eba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:28.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.766+0000 7fd22affd640 10 monclient: get_auth_request con 0x7fd22c1e9650 auth_method 0 2024-03-27T21:40:28.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.767+0000 7fd213fff640 10 client.4656.objecter ms_handle_connect 0x7fd22c1e9650 2024-03-27T21:40:28.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.767+0000 7fd22affd640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7fd22c1e9650 0x7fd22c1eba40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:28.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.768+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== osd.2 v2:172.21.15.60:6824/44654 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fd22c1ee810 con 0x7fd22c1e9650 2024-03-27T21:40:28.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.768+0000 7fd213fff640 10 client.4656.objecter ms_dispatch 0x7fd22c000b90 command_reply(tid 5: 0 ) v1 2024-03-27T21:40:28.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.768+0000 7fd213fff640 10 client.4656.objecter _finish_command 5 = system:0 2024-03-27T21:40:28.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.768+0000 7fd213fff640 20 client.4656.objecter put_session s=0x7fd22c1de780 osd=2 4 2024-03-27T21:40:28.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.768+0000 7fd213fff640 15 client.4656.objecter _session_command_op_remove 2 5 2024-03-27T21:40:28.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.796+0000 7fd211ffb640 10 client.4656.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:28.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.796+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign -1 6 2024-03-27T21:40:28.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.796+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c1de780 osd=2 3 2024-03-27T21:40:28.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.796+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c1de780 osd=2 3 2024-03-27T21:40:28.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.796+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c1de780 osd=2 3 2024-03-27T21:40:28.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.796+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_remove -1 6 2024-03-27T21:40:28.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.796+0000 7fd211ffb640 20 client.4656.objecter get_session s=0x7fd22c1de780 osd=2 3 2024-03-27T21:40:28.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.796+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign 2 6 2024-03-27T21:40:28.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.796+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c1de780 osd=2 4 2024-03-27T21:40:28.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.796+0000 7fd211ffb640 10 client.4656.objecter _send_command 6 2024-03-27T21:40:28.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.796+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd22c0665f0 con 0x7fd22c1e9650 2024-03-27T21:40:28.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.798+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== osd.2 v2:172.21.15.60:6824/44654 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd22c0665f0 con 0x7fd22c1e9650 2024-03-27T21:40:28.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.798+0000 7fd213fff640 10 client.4656.objecter ms_dispatch 0x7fd22c000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:28.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.798+0000 7fd213fff640 10 client.4656.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:28.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.798+0000 7fd213fff640 20 client.4656.objecter put_session s=0x7fd22c1de780 osd=2 4 2024-03-27T21:40:28.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.798+0000 7fd213fff640 15 client.4656.objecter _session_command_op_remove 2 6 2024-03-27T21:40:28.785 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2: {} 2024-03-27T21:40:28.786 INFO:teuthology.orchestra.run.smithi060.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-27T21:40:28.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.798+0000 7fd211ffb640 10 client.4656.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:28.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.799+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign -1 7 2024-03-27T21:40:28.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.799+0000 7fd211ffb640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fd22c13c730 0x7fd22c1ed100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:28.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.799+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c0669a0 osd=3 3 2024-03-27T21:40:28.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.799+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c0669a0 osd=3 3 2024-03-27T21:40:28.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.799+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c0669a0 osd=3 3 2024-03-27T21:40:28.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.799+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_remove -1 7 2024-03-27T21:40:28.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.799+0000 7fd211ffb640 20 client.4656.objecter get_session s=0x7fd22c0669a0 osd=3 3 2024-03-27T21:40:28.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.799+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign 3 7 2024-03-27T21:40:28.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.799+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c0669a0 osd=3 4 2024-03-27T21:40:28.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.799+0000 7fd211ffb640 10 client.4656.objecter _send_command 7 2024-03-27T21:40:28.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.799+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd22c1f20a0 con 0x7fd22c13c730 2024-03-27T21:40:28.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.799+0000 7fd22bfff640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fd22c13c730 0x7fd22c1ed100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:28.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.799+0000 7fd22bfff640 10 monclient: get_auth_request con 0x7fd22c13c730 auth_method 0 2024-03-27T21:40:28.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.799+0000 7fd22bfff640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fd22c13c730 0x7fd22c1ed100 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:28.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.799+0000 7fd213fff640 10 client.4656.objecter ms_handle_connect 0x7fd22c13c730 2024-03-27T21:40:28.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.800+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fd22c1f20a0 con 0x7fd22c13c730 2024-03-27T21:40:28.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.800+0000 7fd213fff640 10 client.4656.objecter ms_dispatch 0x7fd22c000b90 command_reply(tid 7: 0 ) v1 2024-03-27T21:40:28.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.800+0000 7fd213fff640 10 client.4656.objecter _finish_command 7 = system:0 2024-03-27T21:40:28.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.800+0000 7fd213fff640 20 client.4656.objecter put_session s=0x7fd22c0669a0 osd=3 4 2024-03-27T21:40:28.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.800+0000 7fd213fff640 15 client.4656.objecter _session_command_op_remove 3 7 2024-03-27T21:40:28.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.814+0000 7fd211ffb640 10 client.4656.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:28.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.814+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign -1 8 2024-03-27T21:40:28.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.814+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c0669a0 osd=3 3 2024-03-27T21:40:28.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.814+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c0669a0 osd=3 3 2024-03-27T21:40:28.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.814+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c0669a0 osd=3 3 2024-03-27T21:40:28.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.814+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_remove -1 8 2024-03-27T21:40:28.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.814+0000 7fd211ffb640 20 client.4656.objecter get_session s=0x7fd22c0669a0 osd=3 3 2024-03-27T21:40:28.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.814+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign 3 8 2024-03-27T21:40:28.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.814+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c0669a0 osd=3 4 2024-03-27T21:40:28.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.815+0000 7fd211ffb640 10 client.4656.objecter _send_command 8 2024-03-27T21:40:28.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.815+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd22c13bca0 con 0x7fd22c13c730 2024-03-27T21:40:28.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.816+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd22c13bca0 con 0x7fd22c13c730 2024-03-27T21:40:28.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.816+0000 7fd213fff640 10 client.4656.objecter ms_dispatch 0x7fd22c000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:28.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.816+0000 7fd213fff640 10 client.4656.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:28.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.816+0000 7fd213fff640 20 client.4656.objecter put_session s=0x7fd22c0669a0 osd=3 4 2024-03-27T21:40:28.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.816+0000 7fd213fff640 15 client.4656.objecter _session_command_op_remove 3 8 2024-03-27T21:40:28.805 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:28.806 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:28.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.817+0000 7fd211ffb640 10 client.4656.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:28.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.817+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign -1 9 2024-03-27T21:40:28.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.817+0000 7fd211ffb640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fd22c13be10 0x7fd22c1f0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:28.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.817+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c13bca0 osd=4 3 2024-03-27T21:40:28.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.817+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c13bca0 osd=4 3 2024-03-27T21:40:28.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.817+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c13bca0 osd=4 3 2024-03-27T21:40:28.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.817+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_remove -1 9 2024-03-27T21:40:28.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.817+0000 7fd211ffb640 20 client.4656.objecter get_session s=0x7fd22c13bca0 osd=4 3 2024-03-27T21:40:28.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.817+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign 4 9 2024-03-27T21:40:28.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.817+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c13bca0 osd=4 4 2024-03-27T21:40:28.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.817+0000 7fd211ffb640 10 client.4656.objecter _send_command 9 2024-03-27T21:40:28.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.817+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd22c1ecc50 con 0x7fd22c13be10 2024-03-27T21:40:28.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.817+0000 7fd22b7fe640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fd22c13be10 0x7fd22c1f0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:28.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.817+0000 7fd22b7fe640 10 monclient: get_auth_request con 0x7fd22c13be10 auth_method 0 2024-03-27T21:40:28.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.817+0000 7fd22b7fe640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fd22c13be10 0x7fd22c1f0990 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:28.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.817+0000 7fd213fff640 10 client.4656.objecter ms_handle_connect 0x7fd22c13be10 2024-03-27T21:40:28.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.818+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fd22c1ecc50 con 0x7fd22c13be10 2024-03-27T21:40:28.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.818+0000 7fd213fff640 10 client.4656.objecter ms_dispatch 0x7fd22c000b90 command_reply(tid 9: 0 ) v1 2024-03-27T21:40:28.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.818+0000 7fd213fff640 10 client.4656.objecter _finish_command 9 = system:0 2024-03-27T21:40:28.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.818+0000 7fd213fff640 20 client.4656.objecter put_session s=0x7fd22c13bca0 osd=4 4 2024-03-27T21:40:28.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.818+0000 7fd213fff640 15 client.4656.objecter _session_command_op_remove 4 9 2024-03-27T21:40:28.811 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:28.825+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:28.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.832+0000 7fd211ffb640 10 client.4656.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:28.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.832+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign -1 10 2024-03-27T21:40:28.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.832+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c13bca0 osd=4 3 2024-03-27T21:40:28.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.832+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c13bca0 osd=4 3 2024-03-27T21:40:28.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.832+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c13bca0 osd=4 3 2024-03-27T21:40:28.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.832+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_remove -1 10 2024-03-27T21:40:28.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.832+0000 7fd211ffb640 20 client.4656.objecter get_session s=0x7fd22c13bca0 osd=4 3 2024-03-27T21:40:28.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.832+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign 4 10 2024-03-27T21:40:28.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.832+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c13bca0 osd=4 4 2024-03-27T21:40:28.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.832+0000 7fd211ffb640 10 client.4656.objecter _send_command 10 2024-03-27T21:40:28.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.832+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd22c1ecc50 con 0x7fd22c13be10 2024-03-27T21:40:28.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.835+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd22c1ecc50 con 0x7fd22c13be10 2024-03-27T21:40:28.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.835+0000 7fd213fff640 10 client.4656.objecter ms_dispatch 0x7fd22c000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:28.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.835+0000 7fd213fff640 10 client.4656.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:28.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.835+0000 7fd213fff640 20 client.4656.objecter put_session s=0x7fd22c13bca0 osd=4 4 2024-03-27T21:40:28.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.835+0000 7fd213fff640 15 client.4656.objecter _session_command_op_remove 4 10 2024-03-27T21:40:28.821 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:28.822 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:28.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.835+0000 7fd211ffb640 10 client.4656.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:28.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.835+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign -1 11 2024-03-27T21:40:28.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.835+0000 7fd211ffb640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fd22c1f13e0 0x7fd22c1f1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:28.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.835+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c1ecc50 osd=5 3 2024-03-27T21:40:28.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.835+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c1ecc50 osd=5 3 2024-03-27T21:40:28.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.835+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c1ecc50 osd=5 3 2024-03-27T21:40:28.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.835+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_remove -1 11 2024-03-27T21:40:28.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.835+0000 7fd211ffb640 20 client.4656.objecter get_session s=0x7fd22c1ecc50 osd=5 3 2024-03-27T21:40:28.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.835+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign 5 11 2024-03-27T21:40:28.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.835+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c1ecc50 osd=5 4 2024-03-27T21:40:28.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.835+0000 7fd211ffb640 10 client.4656.objecter _send_command 11 2024-03-27T21:40:28.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.835+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd22c1f1d80 con 0x7fd22c1f13e0 2024-03-27T21:40:28.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.836+0000 7fd22affd640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fd22c1f13e0 0x7fd22c1f1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:28.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.836+0000 7fd22affd640 10 monclient: get_auth_request con 0x7fd22c1f13e0 auth_method 0 2024-03-27T21:40:28.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.836+0000 7fd22affd640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fd22c1f13e0 0x7fd22c1f1840 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:28.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.836+0000 7fd213fff640 10 client.4656.objecter ms_handle_connect 0x7fd22c1f13e0 2024-03-27T21:40:28.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.837+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fd22c1f1d80 con 0x7fd22c1f13e0 2024-03-27T21:40:28.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.837+0000 7fd213fff640 10 client.4656.objecter ms_dispatch 0x7fd22c000b90 command_reply(tid 11: 0 ) v1 2024-03-27T21:40:28.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.837+0000 7fd213fff640 10 client.4656.objecter _finish_command 11 = system:0 2024-03-27T21:40:28.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.837+0000 7fd213fff640 20 client.4656.objecter put_session s=0x7fd22c1ecc50 osd=5 4 2024-03-27T21:40:28.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.837+0000 7fd213fff640 15 client.4656.objecter _session_command_op_remove 5 11 2024-03-27T21:40:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.854+0000 7fd211ffb640 10 client.4656.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.854+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign -1 12 2024-03-27T21:40:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.854+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c1ecc50 osd=5 3 2024-03-27T21:40:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.854+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c1ecc50 osd=5 3 2024-03-27T21:40:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.854+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c1ecc50 osd=5 3 2024-03-27T21:40:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.854+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_remove -1 12 2024-03-27T21:40:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.854+0000 7fd211ffb640 20 client.4656.objecter get_session s=0x7fd22c1ecc50 osd=5 3 2024-03-27T21:40:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.854+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign 5 12 2024-03-27T21:40:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.854+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c1ecc50 osd=5 4 2024-03-27T21:40:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.854+0000 7fd211ffb640 10 client.4656.objecter _send_command 12 2024-03-27T21:40:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.854+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd22c1f1d80 con 0x7fd22c1f13e0 2024-03-27T21:40:28.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.856+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd22c1f1d80 con 0x7fd22c1f13e0 2024-03-27T21:40:28.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.856+0000 7fd213fff640 10 client.4656.objecter ms_dispatch 0x7fd22c000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:28.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.856+0000 7fd213fff640 10 client.4656.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:28.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.856+0000 7fd213fff640 20 client.4656.objecter put_session s=0x7fd22c1ecc50 osd=5 4 2024-03-27T21:40:28.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.856+0000 7fd213fff640 15 client.4656.objecter _session_command_op_remove 5 12 2024-03-27T21:40:28.843 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.856+0000 7fd211ffb640 10 client.4656.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.856+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign -1 13 2024-03-27T21:40:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.856+0000 7fd211ffb640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fd22c1f6740 0x7fd22c1f8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.856+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c1f1d80 osd=6 3 2024-03-27T21:40:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.856+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c1f1d80 osd=6 3 2024-03-27T21:40:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.856+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c1f1d80 osd=6 3 2024-03-27T21:40:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.856+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_remove -1 13 2024-03-27T21:40:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.857+0000 7fd211ffb640 20 client.4656.objecter get_session s=0x7fd22c1f1d80 osd=6 3 2024-03-27T21:40:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.857+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign 6 13 2024-03-27T21:40:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.857+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c1f1d80 osd=6 4 2024-03-27T21:40:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.857+0000 7fd211ffb640 10 client.4656.objecter _send_command 13 2024-03-27T21:40:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.857+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd22c1ece20 con 0x7fd22c1f6740 2024-03-27T21:40:28.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.857+0000 7fd22bfff640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fd22c1f6740 0x7fd22c1f8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:28.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.857+0000 7fd22bfff640 10 monclient: get_auth_request con 0x7fd22c1f6740 auth_method 0 2024-03-27T21:40:28.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.857+0000 7fd22bfff640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fd22c1f6740 0x7fd22c1f8b30 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:28.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.857+0000 7fd213fff640 10 client.4656.objecter ms_handle_connect 0x7fd22c1f6740 2024-03-27T21:40:28.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.858+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fd22c1ece20 con 0x7fd22c1f6740 2024-03-27T21:40:28.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.858+0000 7fd213fff640 10 client.4656.objecter ms_dispatch 0x7fd22c000b90 command_reply(tid 13: 0 ) v1 2024-03-27T21:40:28.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.858+0000 7fd213fff640 10 client.4656.objecter _finish_command 13 = system:0 2024-03-27T21:40:28.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.858+0000 7fd213fff640 20 client.4656.objecter put_session s=0x7fd22c1f1d80 osd=6 4 2024-03-27T21:40:28.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.858+0000 7fd213fff640 15 client.4656.objecter _session_command_op_remove 6 13 2024-03-27T21:40:28.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.872+0000 7f161d9c4640 10 start_mon_command cmd=[{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:40:28.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.872+0000 7f161d9c4640 10 monclient: _send_command 2 [{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:40:28.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.872+0000 7f161d9c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:28.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.872+0000 7f161d9c4640 1 -- 172.21.15.60:0/599651106 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f1618069470 con 0x7f16181416f0 2024-03-27T21:40:28.860 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:28.860 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:28.860 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:28.860 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:28.871 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:40:28.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.887+0000 7fd211ffb640 10 client.4656.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:28.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.887+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign -1 14 2024-03-27T21:40:28.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.887+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c1f1d80 osd=6 3 2024-03-27T21:40:28.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.887+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c1f1d80 osd=6 3 2024-03-27T21:40:28.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.887+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c1f1d80 osd=6 3 2024-03-27T21:40:28.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.887+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_remove -1 14 2024-03-27T21:40:28.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.887+0000 7fd211ffb640 20 client.4656.objecter get_session s=0x7fd22c1f1d80 osd=6 3 2024-03-27T21:40:28.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.887+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign 6 14 2024-03-27T21:40:28.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.887+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c1f1d80 osd=6 4 2024-03-27T21:40:28.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.887+0000 7fd211ffb640 10 client.4656.objecter _send_command 14 2024-03-27T21:40:28.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.887+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd22c1ece20 con 0x7fd22c1f6740 2024-03-27T21:40:28.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.889+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd22c1ece20 con 0x7fd22c1f6740 2024-03-27T21:40:28.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.889+0000 7fd213fff640 10 client.4656.objecter ms_dispatch 0x7fd22c000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:28.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.889+0000 7fd213fff640 10 client.4656.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:28.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.889+0000 7fd213fff640 20 client.4656.objecter put_session s=0x7fd22c1f1d80 osd=6 4 2024-03-27T21:40:28.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.889+0000 7fd213fff640 15 client.4656.objecter _session_command_op_remove 6 14 2024-03-27T21:40:28.877 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:28.878 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:28.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.890+0000 7fd211ffb640 10 client.4656.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:28.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.890+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign -1 15 2024-03-27T21:40:28.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.890+0000 7fd211ffb640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fd22c1f9580 0x7fd22c1fb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:28.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.890+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c1ece20 osd=7 3 2024-03-27T21:40:28.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.890+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c1ece20 osd=7 3 2024-03-27T21:40:28.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.890+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c1ece20 osd=7 3 2024-03-27T21:40:28.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.890+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_remove -1 15 2024-03-27T21:40:28.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.890+0000 7fd211ffb640 20 client.4656.objecter get_session s=0x7fd22c1ece20 osd=7 3 2024-03-27T21:40:28.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.890+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign 7 15 2024-03-27T21:40:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.890+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c1ece20 osd=7 4 2024-03-27T21:40:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.890+0000 7fd211ffb640 10 client.4656.objecter _send_command 15 2024-03-27T21:40:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.890+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd22c1fe740 con 0x7fd22c1f9580 2024-03-27T21:40:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.890+0000 7fd22b7fe640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fd22c1f9580 0x7fd22c1fb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.890+0000 7fd22b7fe640 10 monclient: get_auth_request con 0x7fd22c1f9580 auth_method 0 2024-03-27T21:40:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.890+0000 7fd213fff640 10 client.4656.objecter ms_handle_connect 0x7fd22c1f9580 2024-03-27T21:40:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.890+0000 7fd22b7fe640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fd22c1f9580 0x7fd22c1fb970 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.891+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fd22c1fe740 con 0x7fd22c1f9580 2024-03-27T21:40:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.891+0000 7fd213fff640 10 client.4656.objecter ms_dispatch 0x7fd22c000b90 command_reply(tid 15: 0 ) v1 2024-03-27T21:40:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.891+0000 7fd213fff640 10 client.4656.objecter _finish_command 15 = system:0 2024-03-27T21:40:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.891+0000 7fd213fff640 20 client.4656.objecter put_session s=0x7fd22c1ece20 osd=7 4 2024-03-27T21:40:28.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.891+0000 7fd213fff640 15 client.4656.objecter _session_command_op_remove 7 15 2024-03-27T21:40:28.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.919+0000 7fd211ffb640 10 client.4656.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:28.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.919+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign -1 16 2024-03-27T21:40:28.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.919+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c1ece20 osd=7 3 2024-03-27T21:40:28.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.919+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c1ece20 osd=7 3 2024-03-27T21:40:28.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.919+0000 7fd211ffb640 20 client.4656.objecter _get_session s=0x7fd22c1ece20 osd=7 3 2024-03-27T21:40:28.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.919+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_remove -1 16 2024-03-27T21:40:28.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.919+0000 7fd211ffb640 20 client.4656.objecter get_session s=0x7fd22c1ece20 osd=7 3 2024-03-27T21:40:28.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.919+0000 7fd211ffb640 15 client.4656.objecter _session_command_op_assign 7 16 2024-03-27T21:40:28.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.919+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c1ece20 osd=7 4 2024-03-27T21:40:28.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.919+0000 7fd211ffb640 10 client.4656.objecter _send_command 16 2024-03-27T21:40:28.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.919+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd22c1fbeb0 con 0x7fd22c1f9580 2024-03-27T21:40:28.907 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:28.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.921+0000 7fd213fff640 1 -- 172.21.15.60:0/2638420870 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fd22c1fbeb0 con 0x7fd22c1f9580 2024-03-27T21:40:28.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.921+0000 7fd213fff640 10 client.4656.objecter ms_dispatch 0x7fd22c000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:28.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.921+0000 7fd213fff640 10 client.4656.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:28.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.921+0000 7fd213fff640 20 client.4656.objecter put_session s=0x7fd22c1ece20 osd=7 4 2024-03-27T21:40:28.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.921+0000 7fd213fff640 15 client.4656.objecter _session_command_op_remove 7 16 2024-03-27T21:40:28.915 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:28.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.921+0000 7fd211ffb640 10 librados: watch_flush enter 2024-03-27T21:40:28.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.921+0000 7fd211ffb640 10 librados: watch_flush exit 2024-03-27T21:40:28.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.921+0000 7fd211ffb640 10 client.4656.objecter close_session for osd.0 2024-03-27T21:40:28.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.921+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fd22c06dcc0 msgr2=0x7fd22c1ddc50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:28.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.921+0000 7fd211ffb640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fd22c06dcc0 0x7fd22c1ddc50 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.921+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c073e70 osd=0 1 2024-03-27T21:40:28.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.921+0000 7fd211ffb640 10 client.4656.objecter close_session for osd.1 2024-03-27T21:40:28.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.921+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fd22c065a90 msgr2=0x7fd22c065e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:28.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.921+0000 7fd211ffb640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fd22c065a90 0x7fd22c065e70 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c06cd30 osd=1 1 2024-03-27T21:40:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 10 client.4656.objecter close_session for osd.2 2024-03-27T21:40:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7fd22c1e9650 msgr2=0x7fd22c1eba40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7fd22c1e9650 0x7fd22c1eba40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd22bfff640 1 -- 172.21.15.60:0/2638420870 reap_dead start 2024-03-27T21:40:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c1de780 osd=2 1 2024-03-27T21:40:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 10 client.4656.objecter close_session for osd.3 2024-03-27T21:40:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fd22c13c730 msgr2=0x7fd22c1ed100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fd22c13c730 0x7fd22c1ed100 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c0669a0 osd=3 1 2024-03-27T21:40:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 10 client.4656.objecter close_session for osd.4 2024-03-27T21:40:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fd22c13be10 msgr2=0x7fd22c1f0990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:28.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fd22c13be10 0x7fd22c1f0990 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c13bca0 osd=4 1 2024-03-27T21:40:28.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 10 client.4656.objecter close_session for osd.5 2024-03-27T21:40:28.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fd22c1f13e0 msgr2=0x7fd22c1f1840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:28.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fd22c1f13e0 0x7fd22c1f1840 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c1ecc50 osd=5 1 2024-03-27T21:40:28.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 10 client.4656.objecter close_session for osd.6 2024-03-27T21:40:28.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fd22c1f6740 msgr2=0x7fd22c1f8b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:28.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fd22c1f6740 0x7fd22c1f8b30 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c1f1d80 osd=6 1 2024-03-27T21:40:28.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 10 client.4656.objecter close_session for osd.7 2024-03-27T21:40:28.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fd22c1f9580 msgr2=0x7fd22c1fb970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:28.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fd22c1f9580 0x7fd22c1fb970 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd22bfff640 1 -- 172.21.15.60:0/2638420870 reap_dead start 2024-03-27T21:40:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 20 client.4656.objecter put_session s=0x7fd22c1ece20 osd=7 1 2024-03-27T21:40:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 20 client.4656.objecter shutdown clearing up homeless session... 2024-03-27T21:40:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 10 client.4656.objecter successfully canceled tick 2024-03-27T21:40:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd21806c590 msgr2=0x7fd21806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd21806c590 0x7fd21806ea50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd220036000 tx=0x7fd22003c040 comp rx=0 tx=0).stop 2024-03-27T21:40:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 10 monclient: shutdown 2024-03-27T21:40:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd22c0b40b0 msgr2=0x7fd22c13e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd22c0b40b0 0x7fd22c13e520 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd21c009fd0 tx=0x7fd21c00b870 comp rx=0 tx=0).stop 2024-03-27T21:40:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 shutdown_connections 2024-03-27T21:40:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd21806c590 0x7fd21806ea50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.922+0000 7fd211ffb640 1 --2- 172.21.15.60:0/2638420870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd22c0b40b0 0x7fd22c13e520 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.923+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 >> 172.21.15.60:0/2638420870 conn(0x7fd22c0608e0 msgr2=0x7fd22c140350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:28.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.923+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 shutdown_connections 2024-03-27T21:40:28.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.923+0000 7fd211ffb640 1 -- 172.21.15.60:0/2638420870 wait complete. 2024-03-27T21:40:28.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:28.923+0000 7fd211ffb640 1 librados: shutdown 2024-03-27T21:40:28.920 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:28.934+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:28.986 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:28.986 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:28.986 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:28.987 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:28.987 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:28.995 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:40:29.021 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:29.034+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:29.106 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:29.106 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:29.106 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:29.106 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:40:29.106 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:29.118 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:40:29.162 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:29.176+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:29.222 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:29.235+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:29.226 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:29.226 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:29.226 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:29.226 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:29.236 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:40:29.323 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:29.337+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:29.387 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:29.387 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:29.387 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:29.388 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:29.388 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:29.397 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:40:29.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.436+0000 7f15fffff640 1 -- 172.21.15.60:0/599651106 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v38) v1 ==== 87+0+0 (secure 0 0 0) 0x7f160c079bf0 con 0x7f16181416f0 2024-03-27T21:40:29.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.436+0000 7f15fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:40:29.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.436+0000 7f15fffff640 10 monclient: _finish_command 2 = system:0 noscrub is set 2024-03-27T21:40:29.424 INFO:teuthology.orchestra.run.smithi060.stderr:noscrub is set 2024-03-27T21:40:29.424 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:29.437+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:29.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.439+0000 7f161d9c4640 10 librados: watch_flush enter 2024-03-27T21:40:29.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.439+0000 7f161d9c4640 10 librados: watch_flush exit 2024-03-27T21:40:29.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.439+0000 7f161d9c4640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-27T21:40:29.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.439+0000 7f161d9c4640 10 client.4523.objecter successfully canceled tick 2024-03-27T21:40:29.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.439+0000 7f161d9c4640 1 -- 172.21.15.60:0/599651106 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f160406c590 msgr2=0x7f160406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:29.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.439+0000 7f161d9c4640 1 --2- 172.21.15.60:0/599651106 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f160406c590 0x7f160406ea50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f16080012e0 tx=0x7f1608009040 comp rx=0 tx=0).stop 2024-03-27T21:40:29.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.439+0000 7f161d9c4640 10 monclient: shutdown 2024-03-27T21:40:29.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.439+0000 7f161d9c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:29.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.439+0000 7f161d9c4640 1 -- 172.21.15.60:0/599651106 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f16181416f0 msgr2=0x7f161813e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:29.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.439+0000 7f161d9c4640 1 --2- 172.21.15.60:0/599651106 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f16181416f0 0x7f161813e9c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f160c02bf10 tx=0x7f160c004730 comp rx=0 tx=0).stop 2024-03-27T21:40:29.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.439+0000 7f161d9c4640 1 -- 172.21.15.60:0/599651106 shutdown_connections 2024-03-27T21:40:29.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.439+0000 7f161d9c4640 1 --2- 172.21.15.60:0/599651106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16180aaa20 0x7f16181d9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:29.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.439+0000 7f161d9c4640 1 --2- 172.21.15.60:0/599651106 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f16181416f0 0x7f161813e9c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:29.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.439+0000 7f161d9c4640 1 --2- 172.21.15.60:0/599651106 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f160406c590 0x7f160406ea50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:29.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.439+0000 7f161d9c4640 1 --2- 172.21.15.60:0/599651106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16180b40b0 0x7f16180aaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:29.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.439+0000 7f161d9c4640 1 -- 172.21.15.60:0/599651106 >> 172.21.15.60:0/599651106 conn(0x7f16180608e0 msgr2=0x7f16180b2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:29.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.439+0000 7f161d9c4640 1 -- 172.21.15.60:0/599651106 shutdown_connections 2024-03-27T21:40:29.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.439+0000 7f161d9c4640 1 -- 172.21.15.60:0/599651106 wait complete. 2024-03-27T21:40:29.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:29.439+0000 7f161d9c4640 1 librados: shutdown 2024-03-27T21:40:29.512 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:29.512 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:29.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:29.512 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:40:29.512 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:29.522 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:40:29.526 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:29.539+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:29.627 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:29.640+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:29.641 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:29.641 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:29.641 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:29.641 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:29.650 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:40:29.728 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:29.741+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:29.755 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:29.755 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:29.756 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:29.756 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:29.756 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:29.764 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:40:29.849 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:29.862+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:29.911 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:29.911 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:29.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:29.911 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:29.911 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:29.919 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:40:29.960 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:29.974+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:30.030 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:30.044+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:30.064 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:30.064 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:30.064 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:30.064 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:30.073 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:40:30.131 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:30.145+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:30.177 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:30.178 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:30.178 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:30.178 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:30.178 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:30.186 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:40:30.232 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:30.246+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:30.291 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:30.291 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:30.292 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:30.292 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:30.292 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:30.301 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:40:30.333 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:30.347+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:30.405 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:30.406 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:30.406 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:30.406 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:30.415 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:40:30.434 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:30.448+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:30.520 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:30.520 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:30.520 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:30.520 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:30.520 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:30.529 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:40:30.535 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:30.549+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:30.633 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:30.633 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:30.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:30.633 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:30.633 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:30.636 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:30.649+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:30.642 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:40:30.736 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:30.750+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:30.787 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:30.787 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:30.787 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:30.787 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:30.796 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:40:30.838 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:30.851+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:30.925 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:40:30.941 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:30.942 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:30.942 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:30.942 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:30.942 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:30.950 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:40:30.969 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:30.983+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:31.041 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:31.053+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:31.057 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:31.057 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:31.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:31.058 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-03-27T21:40:31.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.058 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:31.058 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:31.058 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 51.638037734999997, 2024-03-27T21:40:31.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43537978700000002, 2024-03-27T21:40:31.058 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:31.058 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:31.058 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:31.058 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:31.058 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:31.058 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-03-27T21:40:31.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.059 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:31.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:31.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:31.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:31.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:31.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:31.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432019+0000", 2024-03-27T21:40:31.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-03-27T21:40:31.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:31.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432027+0000", 2024-03-27T21:40:31.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1200000000000002e-06 2024-03-27T21:40:31.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:31.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432039+0000", 2024-03-27T21:40:31.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1123e-05 2024-03-27T21:40:31.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:31.061 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432051+0000", 2024-03-27T21:40:31.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-03-27T21:40:31.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.061 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:31.061 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432157+0000", 2024-03-27T21:40:31.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106846 2024-03-27T21:40:31.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.061 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-03-27T21:40:31.061 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432176+0000", 2024-03-27T21:40:31.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8490000000000001e-05 2024-03-27T21:40:31.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:31.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866170+0000", 2024-03-27T21:40:31.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43399370100000001 2024-03-27T21:40:31.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:31.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866335+0000", 2024-03-27T21:40:31.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016559699999999999 2024-03-27T21:40:31.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:31.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866495+0000", 2024-03-27T21:40:31.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015944200000000001 2024-03-27T21:40:31.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:31.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867012+0000", 2024-03-27T21:40:31.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0005176 2024-03-27T21:40:31.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:31.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867334+0000", 2024-03-27T21:40:31.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032158600000000001 2024-03-27T21:40:31.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:31.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867374+0000", 2024-03-27T21:40:31.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0547000000000003e-05 2024-03-27T21:40:31.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:31.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867400+0000", 2024-03-27T21:40:31.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5958000000000001e-05 2024-03-27T21:40:31.064 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.064 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:31.064 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.064 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:31.064 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:31.064 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 51.140655283999997, 2024-03-27T21:40:31.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064560310000000001, 2024-03-27T21:40:31.065 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:31.065 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:31.065 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:31.065 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:31.065 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:31.065 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 70 2024-03-27T21:40:31.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.065 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:31.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:31.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:31.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:31.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:31.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:31.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929402+0000", 2024-03-27T21:40:31.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:31.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:31.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929407+0000", 2024-03-27T21:40:31.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-06 2024-03-27T21:40:31.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:31.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929414+0000", 2024-03-27T21:40:31.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5689999999999999e-06 2024-03-27T21:40:31.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:31.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929419+0000", 2024-03-27T21:40:31.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-03-27T21:40:31.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:31.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930245+0000", 2024-03-27T21:40:31.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082521500000000002 2024-03-27T21:40:31.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:31.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930287+0000", 2024-03-27T21:40:31.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2305000000000002e-05 2024-03-27T21:40:31.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:31.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930367+0000", 2024-03-27T21:40:31.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9589e-05 2024-03-27T21:40:31.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:31.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.931214+0000", 2024-03-27T21:40:31.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000847827 2024-03-27T21:40:31.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:31.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935807+0000", 2024-03-27T21:40:31.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045923780000000003 2024-03-27T21:40:31.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:31.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935840+0000", 2024-03-27T21:40:31.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3501999999999999e-05 2024-03-27T21:40:31.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:31.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935859+0000", 2024-03-27T21:40:31.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8845e-05 2024-03-27T21:40:31.069 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.070 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:31.070 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.070 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:31.070 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:31.070 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 51.133395692999997, 2024-03-27T21:40:31.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071695429999999996, 2024-03-27T21:40:31.070 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:31.070 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:31.070 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:31.070 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:31.070 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:31.070 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-03-27T21:40:31.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.071 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:31.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:31.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:31.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:31.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:31.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:31.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936662+0000", 2024-03-27T21:40:31.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:31.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:31.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936737+0000", 2024-03-27T21:40:31.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5303999999999996e-05 2024-03-27T21:40:31.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:31.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936746+0000", 2024-03-27T21:40:31.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6340000000000007e-06 2024-03-27T21:40:31.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:31.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936753+0000", 2024-03-27T21:40:31.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-03-27T21:40:31.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:31.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937926+0000", 2024-03-27T21:40:31.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011731280000000001 2024-03-27T21:40:31.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:31.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937969+0000", 2024-03-27T21:40:31.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3285999999999999e-05 2024-03-27T21:40:31.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:31.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938040+0000", 2024-03-27T21:40:31.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0358000000000006e-05 2024-03-27T21:40:31.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.074 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:31.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939324+0000", 2024-03-27T21:40:31.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012844899999999999 2024-03-27T21:40:31.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.074 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:31.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943780+0000", 2024-03-27T21:40:31.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044555630000000001 2024-03-27T21:40:31.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:31.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943811+0000", 2024-03-27T21:40:31.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1415999999999999e-05 2024-03-27T21:40:31.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:31.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943832+0000", 2024-03-27T21:40:31.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0981000000000001e-05 2024-03-27T21:40:31.075 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.075 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:31.075 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.075 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:31.075 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:31.076 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 51.133297677000002, 2024-03-27T21:40:31.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071139740000000003, 2024-03-27T21:40:31.076 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:31.076 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:31.076 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:31.076 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:31.076 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:31.076 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-03-27T21:40:31.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.076 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:31.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:31.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:31.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:31.077 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:31.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:31.077 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936760+0000", 2024-03-27T21:40:31.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:31.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:31.077 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936765+0000", 2024-03-27T21:40:31.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6330000000000004e-06 2024-03-27T21:40:31.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:31.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936771+0000", 2024-03-27T21:40:31.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0560000000000001e-06 2024-03-27T21:40:31.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:31.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936776+0000", 2024-03-27T21:40:31.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1510000000000003e-06 2024-03-27T21:40:31.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:31.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938605+0000", 2024-03-27T21:40:31.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001829589 2024-03-27T21:40:31.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:31.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938657+0000", 2024-03-27T21:40:31.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1860000000000002e-05 2024-03-27T21:40:31.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.080 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:31.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938734+0000", 2024-03-27T21:40:31.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6639999999999998e-05 2024-03-27T21:40:31.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.080 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:31.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939562+0000", 2024-03-27T21:40:31.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082803100000000001 2024-03-27T21:40:31.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.080 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:31.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943838+0000", 2024-03-27T21:40:31.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042757050000000003 2024-03-27T21:40:31.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:31.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943859+0000", 2024-03-27T21:40:31.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1175e-05 2024-03-27T21:40:31.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:31.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943875+0000", 2024-03-27T21:40:31.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5744e-05 2024-03-27T21:40:31.081 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.081 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:31.081 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.082 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:31.082 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:31.082 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.989783625999998, 2024-03-27T21:40:31.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059712230000000003, 2024-03-27T21:40:31.082 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:31.082 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:31.082 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:31.082 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:31.082 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:31.082 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 112 2024-03-27T21:40:31.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.082 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:31.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:31.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:31.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:31.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:31.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:31.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080274+0000", 2024-03-27T21:40:31.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:31.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:31.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080318+0000", 2024-03-27T21:40:31.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3634999999999998e-05 2024-03-27T21:40:31.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:31.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080329+0000", 2024-03-27T21:40:31.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1232e-05 2024-03-27T21:40:31.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:31.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080333+0000", 2024-03-27T21:40:31.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6009999999999999e-06 2024-03-27T21:40:31.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:31.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083786+0000", 2024-03-27T21:40:31.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034528940000000002 2024-03-27T21:40:31.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:31.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083861+0000", 2024-03-27T21:40:31.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4893000000000003e-05 2024-03-27T21:40:31.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:31.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083981+0000", 2024-03-27T21:40:31.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012058299999999999 2024-03-27T21:40:31.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:31.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086148+0000", 2024-03-27T21:40:31.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021671469999999999 2024-03-27T21:40:31.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:31.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086195+0000", 2024-03-27T21:40:31.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6768999999999999e-05 2024-03-27T21:40:31.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:31.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086224+0000", 2024-03-27T21:40:31.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8758000000000001e-05 2024-03-27T21:40:31.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:31.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086246+0000", 2024-03-27T21:40:31.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2064000000000001e-05 2024-03-27T21:40:31.087 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.087 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:31.087 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.087 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:31.088 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:31.088 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.989720679000001, 2024-03-27T21:40:31.088 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066196320000000003, 2024-03-27T21:40:31.088 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:31.088 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:31.088 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:31.088 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:31.088 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:31.088 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113 2024-03-27T21:40:31.088 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.088 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:31.088 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.088 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:31.088 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:31.089 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.089 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.089 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.089 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:31.089 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:31.089 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.089 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.089 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.089 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:31.089 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080337+0000", 2024-03-27T21:40:31.089 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:31.089 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.089 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.089 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:31.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080340+0000", 2024-03-27T21:40:31.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6690000000000002e-06 2024-03-27T21:40:31.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:31.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080344+0000", 2024-03-27T21:40:31.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6550000000000002e-06 2024-03-27T21:40:31.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:31.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080348+0000", 2024-03-27T21:40:31.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9169999999999999e-06 2024-03-27T21:40:31.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:31.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084595+0000", 2024-03-27T21:40:31.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042471719999999996 2024-03-27T21:40:31.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:31.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084676+0000", 2024-03-27T21:40:31.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1147000000000001e-05 2024-03-27T21:40:31.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:31.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084796+0000", 2024-03-27T21:40:31.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012032900000000001 2024-03-27T21:40:31.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:31.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086190+0000", 2024-03-27T21:40:31.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013934399999999999 2024-03-27T21:40:31.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:31.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086905+0000", 2024-03-27T21:40:31.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071536199999999999 2024-03-27T21:40:31.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:31.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086937+0000", 2024-03-27T21:40:31.093 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2125000000000002e-05 2024-03-27T21:40:31.093 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.093 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.093 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:31.093 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086957+0000", 2024-03-27T21:40:31.093 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0157999999999999e-05 2024-03-27T21:40:31.093 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.093 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:31.093 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.093 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.093 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.093 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:31.093 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:31.093 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.989707963000001, 2024-03-27T21:40:31.094 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007405661, 2024-03-27T21:40:31.094 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:31.094 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:31.094 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:31.094 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:31.094 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:31.094 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 114 2024-03-27T21:40:31.094 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.094 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:31.094 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.094 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:31.094 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:31.094 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.094 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.095 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.095 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:31.095 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:31.095 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.095 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.095 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.095 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:31.095 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:31.095 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-03-27T21:40:31.095 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.095 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.095 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:31.095 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080353+0000", 2024-03-27T21:40:31.095 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.441e-06 2024-03-27T21:40:31.095 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.096 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.096 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:31.096 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080356+0000", 2024-03-27T21:40:31.096 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0769999999999999e-06 2024-03-27T21:40:31.096 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.096 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.096 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:31.096 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080358+0000", 2024-03-27T21:40:31.096 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3369999999999998e-06 2024-03-27T21:40:31.096 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.096 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.096 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:31.096 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085118+0000", 2024-03-27T21:40:31.097 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047603949999999997 2024-03-27T21:40:31.097 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:31.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085210+0000", 2024-03-27T21:40:31.097 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1983000000000003e-05 2024-03-27T21:40:31.097 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:31.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085332+0000", 2024-03-27T21:40:31.097 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121803 2024-03-27T21:40:31.097 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:31.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086292+0000", 2024-03-27T21:40:31.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095961400000000002 2024-03-27T21:40:31.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.098 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:31.098 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087710+0000", 2024-03-27T21:40:31.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014181459999999999 2024-03-27T21:40:31.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.098 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:31.098 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087736+0000", 2024-03-27T21:40:31.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6475e-05 2024-03-27T21:40:31.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.099 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:31.099 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087756+0000", 2024-03-27T21:40:31.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9626000000000001e-05 2024-03-27T21:40:31.099 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.099 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:31.099 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.099 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.099 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:31.099 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:31.099 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.989685528000003, 2024-03-27T21:40:31.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075011720000000004, 2024-03-27T21:40:31.099 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:31.099 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:31.099 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:31.100 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:31.100 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:31.100 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115 2024-03-27T21:40:31.100 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.100 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:31.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.100 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:31.100 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:31.100 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.100 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.100 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:31.100 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:31.100 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.101 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.101 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.101 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:31.101 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:31.101 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:31.101 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.101 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.101 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:31.101 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080458+0000", 2024-03-27T21:40:31.101 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5228000000000005e-05 2024-03-27T21:40:31.101 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.101 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.101 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:31.101 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080465+0000", 2024-03-27T21:40:31.101 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.238e-06 2024-03-27T21:40:31.102 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.102 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:31.102 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080468+0000", 2024-03-27T21:40:31.102 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5109999999999999e-06 2024-03-27T21:40:31.102 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.102 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:31.102 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086326+0000", 2024-03-27T21:40:31.102 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058579169999999998 2024-03-27T21:40:31.102 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.102 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:31.102 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086382+0000", 2024-03-27T21:40:31.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5309999999999997e-05 2024-03-27T21:40:31.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:31.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086457+0000", 2024-03-27T21:40:31.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5413000000000005e-05 2024-03-27T21:40:31.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:31.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087762+0000", 2024-03-27T21:40:31.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013052300000000001 2024-03-27T21:40:31.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:31.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087798+0000", 2024-03-27T21:40:31.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5614999999999997e-05 2024-03-27T21:40:31.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:31.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087840+0000", 2024-03-27T21:40:31.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2196000000000001e-05 2024-03-27T21:40:31.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:31.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087874+0000", 2024-03-27T21:40:31.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3785999999999998e-05 2024-03-27T21:40:31.104 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.104 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:31.104 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.105 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:31.105 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:31.105 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.989585445000003, 2024-03-27T21:40:31.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075050680000000002, 2024-03-27T21:40:31.105 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:31.105 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:31.105 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:31.105 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:31.105 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:31.105 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 116 2024-03-27T21:40:31.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.105 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:31.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:31.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:31.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:31.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:31.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:31.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080472+0000", 2024-03-27T21:40:31.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:31.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.107 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:31.107 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080475+0000", 2024-03-27T21:40:31.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.712e-06 2024-03-27T21:40:31.107 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.107 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:31.107 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080478+0000", 2024-03-27T21:40:31.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1070000000000001e-06 2024-03-27T21:40:31.107 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.107 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:31.107 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080481+0000", 2024-03-27T21:40:31.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4269999999999998e-06 2024-03-27T21:40:31.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.108 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.108 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:31.108 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086992+0000", 2024-03-27T21:40:31.108 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065114099999999996 2024-03-27T21:40:31.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.108 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.108 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:31.108 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087064+0000", 2024-03-27T21:40:31.108 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1927000000000004e-05 2024-03-27T21:40:31.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.108 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.108 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:31.108 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087188+0000", 2024-03-27T21:40:31.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123585 2024-03-27T21:40:31.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.109 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:31.109 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087766+0000", 2024-03-27T21:40:31.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057814300000000004 2024-03-27T21:40:31.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.109 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:31.109 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087934+0000", 2024-03-27T21:40:31.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000167894 2024-03-27T21:40:31.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.109 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:31.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087958+0000", 2024-03-27T21:40:31.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4663000000000001e-05 2024-03-27T21:40:31.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:31.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087978+0000", 2024-03-27T21:40:31.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9636e-05 2024-03-27T21:40:31.110 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.110 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:31.110 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.110 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:31.110 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:31.111 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.989553323000003, 2024-03-27T21:40:31.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076850679999999998, 2024-03-27T21:40:31.111 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:31.111 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:31.111 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:31.111 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:31.111 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:31.111 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 117 2024-03-27T21:40:31.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.111 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:31.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:31.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:31.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:31.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:31.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:31.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:31.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:31.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:31.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080508+0000", 2024-03-27T21:40:31.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.298e-06 2024-03-27T21:40:31.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:31.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080515+0000", 2024-03-27T21:40:31.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0790000000000003e-06 2024-03-27T21:40:31.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:31.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080518+0000", 2024-03-27T21:40:31.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8509999999999999e-06 2024-03-27T21:40:31.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:31.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087444+0000", 2024-03-27T21:40:31.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0069259120000000002 2024-03-27T21:40:31.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.114 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:31.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087491+0000", 2024-03-27T21:40:31.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7043999999999999e-05 2024-03-27T21:40:31.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.114 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:31.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087564+0000", 2024-03-27T21:40:31.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3603999999999995e-05 2024-03-27T21:40:31.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:31.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087902+0000", 2024-03-27T21:40:31.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033734100000000001 2024-03-27T21:40:31.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:31.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088117+0000", 2024-03-27T21:40:31.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000215164 2024-03-27T21:40:31.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:31.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088160+0000", 2024-03-27T21:40:31.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2933999999999999e-05 2024-03-27T21:40:31.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:31.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088190+0000", 2024-03-27T21:40:31.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0139999999999999e-05 2024-03-27T21:40:31.116 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.116 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:31.116 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.116 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:31.116 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:31.116 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.573272934000002, 2024-03-27T21:40:31.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052465439999999997, 2024-03-27T21:40:31.116 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:31.116 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:31.117 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:31.117 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:31.117 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:31.117 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 133 2024-03-27T21:40:31.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.117 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:31.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:31.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:31.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:31.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:31.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:31.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:31.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:31.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:31.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496788+0000", 2024-03-27T21:40:31.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-03-27T21:40:31.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:31.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496793+0000", 2024-03-27T21:40:31.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4790000000000004e-06 2024-03-27T21:40:31.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:31.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496796+0000", 2024-03-27T21:40:31.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1e-06 2024-03-27T21:40:31.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:31.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500447+0000", 2024-03-27T21:40:31.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036504459999999999 2024-03-27T21:40:31.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:31.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500492+0000", 2024-03-27T21:40:31.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5765e-05 2024-03-27T21:40:31.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:31.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500562+0000", 2024-03-27T21:40:31.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9206000000000001e-05 2024-03-27T21:40:31.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:31.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501959+0000", 2024-03-27T21:40:31.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001397187 2024-03-27T21:40:31.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:31.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501984+0000", 2024-03-27T21:40:31.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.552e-05 2024-03-27T21:40:31.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:31.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502010+0000", 2024-03-27T21:40:31.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5232000000000002e-05 2024-03-27T21:40:31.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:31.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502032+0000", 2024-03-27T21:40:31.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2337999999999999e-05 2024-03-27T21:40:31.121 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.121 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:31.122 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.122 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:31.122 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:31.122 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.573214276000002, 2024-03-27T21:40:31.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052892119999999997, 2024-03-27T21:40:31.122 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:31.122 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:31.122 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:31.122 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:31.122 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:31.122 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 134 2024-03-27T21:40:31.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.122 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:31.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:31.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:31.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:31.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:31.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:31.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:31.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:31.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:31.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496929+0000", 2024-03-27T21:40:31.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5098000000000004e-05 2024-03-27T21:40:31.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:31.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496937+0000", 2024-03-27T21:40:31.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9559999999999997e-06 2024-03-27T21:40:31.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:31.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496942+0000", 2024-03-27T21:40:31.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6350000000000003e-06 2024-03-27T21:40:31.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:31.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500749+0000", 2024-03-27T21:40:31.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038068529999999998 2024-03-27T21:40:31.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:31.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500792+0000", 2024-03-27T21:40:31.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2823e-05 2024-03-27T21:40:31.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:31.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500864+0000", 2024-03-27T21:40:31.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1715e-05 2024-03-27T21:40:31.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:31.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502068+0000", 2024-03-27T21:40:31.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012044860000000001 2024-03-27T21:40:31.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:31.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502073+0000", 2024-03-27T21:40:31.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5700000000000003e-06 2024-03-27T21:40:31.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:31.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502098+0000", 2024-03-27T21:40:31.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5120999999999999e-05 2024-03-27T21:40:31.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.127 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:31.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502133+0000", 2024-03-27T21:40:31.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5306999999999999e-05 2024-03-27T21:40:31.127 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.127 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:31.127 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.127 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:31.127 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:31.127 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.573108969000003, 2024-03-27T21:40:31.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062529040000000001, 2024-03-27T21:40:31.128 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:31.128 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:31.128 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:31.128 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:31.128 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:31.128 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135 2024-03-27T21:40:31.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.128 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:31.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:31.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:31.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:31.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:31.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:31.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:31.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:31.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:31.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496953+0000", 2024-03-27T21:40:31.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0709999999999996e-06 2024-03-27T21:40:31.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:31.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496961+0000", 2024-03-27T21:40:31.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1300000000000001e-06 2024-03-27T21:40:31.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:31.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496968+0000", 2024-03-27T21:40:31.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0999999999999998e-06 2024-03-27T21:40:31.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:31.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501244+0000", 2024-03-27T21:40:31.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042756310000000002 2024-03-27T21:40:31.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:31.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501377+0000", 2024-03-27T21:40:31.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013348799999999999 2024-03-27T21:40:31.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:31.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501455+0000", 2024-03-27T21:40:31.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7410000000000006e-05 2024-03-27T21:40:31.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:31.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502138+0000", 2024-03-27T21:40:31.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068386599999999999 2024-03-27T21:40:31.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:31.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503152+0000", 2024-03-27T21:40:31.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010139680000000001 2024-03-27T21:40:31.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:31.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503180+0000", 2024-03-27T21:40:31.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7426000000000001e-05 2024-03-27T21:40:31.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:31.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503202+0000", 2024-03-27T21:40:31.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2463000000000002e-05 2024-03-27T21:40:31.133 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.133 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:31.133 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.133 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:31.133 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:31.133 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.573084831000003, 2024-03-27T21:40:31.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066253680000000004, 2024-03-27T21:40:31.133 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:31.133 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:31.133 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:31.133 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:31.133 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:31.134 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 136 2024-03-27T21:40:31.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.134 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:31.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:31.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:31.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:31.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:31.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:31.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:31.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:31.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:31.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496977+0000", 2024-03-27T21:40:31.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5049999999999999e-06 2024-03-27T21:40:31.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:31.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496984+0000", 2024-03-27T21:40:31.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1520000000000003e-06 2024-03-27T21:40:31.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:31.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496990+0000", 2024-03-27T21:40:31.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-03-27T21:40:31.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:31.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501669+0000", 2024-03-27T21:40:31.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046793859999999998 2024-03-27T21:40:31.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:31.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501714+0000", 2024-03-27T21:40:31.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5154999999999997e-05 2024-03-27T21:40:31.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:31.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501783+0000", 2024-03-27T21:40:31.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8454999999999994e-05 2024-03-27T21:40:31.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:31.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503226+0000", 2024-03-27T21:40:31.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001443441 2024-03-27T21:40:31.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:31.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503540+0000", 2024-03-27T21:40:31.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031347100000000001 2024-03-27T21:40:31.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:31.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503563+0000", 2024-03-27T21:40:31.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3223000000000001e-05 2024-03-27T21:40:31.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:31.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503599+0000", 2024-03-27T21:40:31.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6105000000000001e-05 2024-03-27T21:40:31.138 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.138 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:31.138 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.138 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:31.139 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:31.139 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.573062020999998, 2024-03-27T21:40:31.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074261320000000002, 2024-03-27T21:40:31.139 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:31.139 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:31.139 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:31.139 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:31.139 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:31.139 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 137 2024-03-27T21:40:31.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.139 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:31.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:31.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:31.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:31.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:31.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:31.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:31.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:31.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:31.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497077+0000", 2024-03-27T21:40:31.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1599000000000003e-05 2024-03-27T21:40:31.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:31.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497085+0000", 2024-03-27T21:40:31.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6109999999999996e-06 2024-03-27T21:40:31.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:31.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497092+0000", 2024-03-27T21:40:31.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.742e-06 2024-03-27T21:40:31.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:31.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502159+0000", 2024-03-27T21:40:31.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050666870000000003 2024-03-27T21:40:31.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:31.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502200+0000", 2024-03-27T21:40:31.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1919999999999998e-05 2024-03-27T21:40:31.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:31.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502280+0000", 2024-03-27T21:40:31.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9417000000000002e-05 2024-03-27T21:40:31.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:31.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503605+0000", 2024-03-27T21:40:31.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001325641 2024-03-27T21:40:31.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:31.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504361+0000", 2024-03-27T21:40:31.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075555399999999997 2024-03-27T21:40:31.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:31.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504386+0000", 2024-03-27T21:40:31.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5369999999999999e-05 2024-03-27T21:40:31.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:31.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504422+0000", 2024-03-27T21:40:31.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6019000000000002e-05 2024-03-27T21:40:31.144 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.144 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:31.144 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.144 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:31.144 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:31.144 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.572958223999997, 2024-03-27T21:40:31.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074059750000000004, 2024-03-27T21:40:31.144 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:31.144 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:31.144 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:31.145 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:31.145 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:31.145 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-03-27T21:40:31.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.145 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:31.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:31.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:31.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:31.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:31.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:31.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:31.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:31.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:31.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497103+0000", 2024-03-27T21:40:31.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.856e-06 2024-03-27T21:40:31.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:31.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497121+0000", 2024-03-27T21:40:31.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7190999999999999e-05 2024-03-27T21:40:31.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:31.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497126+0000", 2024-03-27T21:40:31.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3770000000000004e-06 2024-03-27T21:40:31.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:31.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502833+0000", 2024-03-27T21:40:31.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057073510000000003 2024-03-27T21:40:31.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:31.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502881+0000", 2024-03-27T21:40:31.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7961999999999999e-05 2024-03-27T21:40:31.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:31.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502949+0000", 2024-03-27T21:40:31.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7957e-05 2024-03-27T21:40:31.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:31.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504430+0000", 2024-03-27T21:40:31.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001480763 2024-03-27T21:40:31.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:31.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504459+0000", 2024-03-27T21:40:31.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8983999999999999e-05 2024-03-27T21:40:31.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:31.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504486+0000", 2024-03-27T21:40:31.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6883999999999999e-05 2024-03-27T21:40:31.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:31.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504506+0000", 2024-03-27T21:40:31.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0214e-05 2024-03-27T21:40:31.149 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.149 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:31.150 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.150 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:31.150 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:31.150 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.572926389999999, 2024-03-27T21:40:31.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074754859999999999, 2024-03-27T21:40:31.150 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:31.150 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:31.150 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:31.150 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:31.150 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:31.150 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-03-27T21:40:31.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.151 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:31.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:31.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:31.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:31.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:31.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:31.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:31.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:31.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:31.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497137+0000", 2024-03-27T21:40:31.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0730000000000004e-06 2024-03-27T21:40:31.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:31.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497143+0000", 2024-03-27T21:40:31.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3759999999999996e-06 2024-03-27T21:40:31.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:31.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497150+0000", 2024-03-27T21:40:31.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.002e-06 2024-03-27T21:40:31.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:31.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503249+0000", 2024-03-27T21:40:31.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060989750000000004 2024-03-27T21:40:31.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:31.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503291+0000", 2024-03-27T21:40:31.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1563999999999997e-05 2024-03-27T21:40:31.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:31.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503376+0000", 2024-03-27T21:40:31.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5752000000000002e-05 2024-03-27T21:40:31.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:31.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504545+0000", 2024-03-27T21:40:31.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169172 2024-03-27T21:40:31.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:31.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504550+0000", 2024-03-27T21:40:31.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7419999999999997e-06 2024-03-27T21:40:31.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:31.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504574+0000", 2024-03-27T21:40:31.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4124000000000001e-05 2024-03-27T21:40:31.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:31.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504607+0000", 2024-03-27T21:40:31.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3118999999999999e-05 2024-03-27T21:40:31.155 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.155 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:31.155 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.155 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:31.156 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:31.156 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.572902724000002, 2024-03-27T21:40:31.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078150589999999992, 2024-03-27T21:40:31.156 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:31.156 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:31.156 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:31.156 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:31.156 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:31.156 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-03-27T21:40:31.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.156 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:31.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:31.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:31.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:31.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:31.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:31.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497155+0000", 2024-03-27T21:40:31.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:31.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:31.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497230+0000", 2024-03-27T21:40:31.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4789000000000005e-05 2024-03-27T21:40:31.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:31.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497241+0000", 2024-03-27T21:40:31.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1442e-05 2024-03-27T21:40:31.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:31.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497255+0000", 2024-03-27T21:40:31.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3087e-05 2024-03-27T21:40:31.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:31.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503630+0000", 2024-03-27T21:40:31.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063755269999999998 2024-03-27T21:40:31.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:31.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503673+0000", 2024-03-27T21:40:31.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2616e-05 2024-03-27T21:40:31.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:31.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503738+0000", 2024-03-27T21:40:31.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5114e-05 2024-03-27T21:40:31.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:31.160 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504907+0000", 2024-03-27T21:40:31.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169432 2024-03-27T21:40:31.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.160 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:31.160 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504926+0000", 2024-03-27T21:40:31.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9086999999999999e-05 2024-03-27T21:40:31.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.160 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:31.160 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504950+0000", 2024-03-27T21:40:31.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3598999999999999e-05 2024-03-27T21:40:31.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.160 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:31.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504971+0000", 2024-03-27T21:40:31.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0772999999999999e-05 2024-03-27T21:40:31.161 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.161 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:31.161 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.161 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:31.161 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:31.161 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 50.572795683999999, 2024-03-27T21:40:31.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082962590000000003, 2024-03-27T21:40:31.161 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:31.161 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:31.161 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:31.161 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:31.162 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:31.162 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-03-27T21:40:31.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.162 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:31.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:31.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:31.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:31.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:31.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:31.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497262+0000", 2024-03-27T21:40:31.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:31.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:31.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497267+0000", 2024-03-27T21:40:31.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.955e-06 2024-03-27T21:40:31.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:31.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497275+0000", 2024-03-27T21:40:31.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.824e-06 2024-03-27T21:40:31.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:31.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497280+0000", 2024-03-27T21:40:31.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8230000000000002e-06 2024-03-27T21:40:31.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:31.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504628+0000", 2024-03-27T21:40:31.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007348561 2024-03-27T21:40:31.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:31.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504673+0000", 2024-03-27T21:40:31.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4313000000000001e-05 2024-03-27T21:40:31.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:31.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504745+0000", 2024-03-27T21:40:31.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2303000000000002e-05 2024-03-27T21:40:31.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:31.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505358+0000", 2024-03-27T21:40:31.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061281300000000003 2024-03-27T21:40:31.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:31.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505503+0000", 2024-03-27T21:40:31.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000145134 2024-03-27T21:40:31.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:31.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505526+0000", 2024-03-27T21:40:31.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3312e-05 2024-03-27T21:40:31.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:31.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505559+0000", 2024-03-27T21:40:31.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2821999999999998e-05 2024-03-27T21:40:31.166 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.166 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:31.166 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.167 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:31.167 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:31.167 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 49.963573445000002, 2024-03-27T21:40:31.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053753339999999998, 2024-03-27T21:40:31.167 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:31.167 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:31.167 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:31.167 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:31.167 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:31.167 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159 2024-03-27T21:40:31.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.167 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:31.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:31.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:31.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:31.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:31.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:31.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:31.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:31.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:31.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:31.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106488+0000", 2024-03-27T21:40:31.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9830000000000001e-06 2024-03-27T21:40:31.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:31.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106491+0000", 2024-03-27T21:40:31.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3459999999999998e-06 2024-03-27T21:40:31.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:31.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106493+0000", 2024-03-27T21:40:31.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.509e-06 2024-03-27T21:40:31.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:31.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110636+0000", 2024-03-27T21:40:31.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041427260000000002 2024-03-27T21:40:31.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:31.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110683+0000", 2024-03-27T21:40:31.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6557000000000002e-05 2024-03-27T21:40:31.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:31.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110755+0000", 2024-03-27T21:40:31.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2119000000000005e-05 2024-03-27T21:40:31.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.173 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:31.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111415+0000", 2024-03-27T21:40:31.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066065199999999996 2024-03-27T21:40:31.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.173 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:31.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111793+0000", 2024-03-27T21:40:31.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037732100000000001 2024-03-27T21:40:31.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.173 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:31.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111829+0000", 2024-03-27T21:40:31.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418999999999998e-05 2024-03-27T21:40:31.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:31.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:31.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:31.174 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111860+0000", 2024-03-27T21:40:31.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1003000000000001e-05 2024-03-27T21:40:31.174 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.174 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:31.174 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.174 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:31.174 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:31.174 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:31.176 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-27T21:40:31.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.089+0000 7feccf918640 10 monclient: get_monmap_and_config 2024-03-27T21:40:31.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.089+0000 7feccf918640 10 monclient: build_initial_monmap 2024-03-27T21:40:31.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.089+0000 7feccf918640 10 monclient: monmap: 2024-03-27T21:40:31.189 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:31.189 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:31.189 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:31.090957+0000 2024-03-27T21:40:31.189 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:31.090957+0000 2024-03-27T21:40:31.189 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:31.189 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:31.189 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:31.189 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:31.190 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.090+0000 7feccf918640 1 Processor -- start 2024-03-27T21:40:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.091+0000 7feccf918640 1 -- start start 2024-03-27T21:40:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.091+0000 7feccf918640 10 monclient: init 2024-03-27T21:40:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.091+0000 7feccf918640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccf918640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-03-27T21:40:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccf918640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fecc80b40b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccf918640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fecc813b020 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccf918640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fecc813b480 0x7fecc8141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccf918640 10 monclient(hunting): picked mon.noname-a con 0x7fecc813b480 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccf918640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc813b8d0 con 0x7fecc80b40b0 2024-03-27T21:40:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccf918640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc80b3830 con 0x7fecc813b020 2024-03-27T21:40:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccf918640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccf918640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecc80b4490 con 0x7fecc813b480 2024-03-27T21:40:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccf918640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccf918640 10 monclient(hunting): authenticate will time out at 872.313139s 2024-03-27T21:40:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccdeb5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fecc813b480 0x7fecc8141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccdeb5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fecc813b480 0x7fecc8141e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40414/0 (socket says 172.21.15.60:40414) 2024-03-27T21:40:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccdeb5640 1 -- 172.21.15.60:0/1514221016 learned_addr learned my addr 172.21.15.60:0/1514221016 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccdeb5640 10 monclient(hunting): get_auth_request con 0x7fecc813b480 auth_method 0 2024-03-27T21:40:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccdeb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccdeb5640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccdeb5640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccdeb5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccdeb5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccdeb5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccdeb5640 10 monclient(hunting): handle_auth_done global_id 4659 payload 274 2024-03-27T21:40:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccdeb5640 1 -- 172.21.15.60:0/1514221016 >> v1:172.21.15.191:6789/0 conn(0x7fecc813b020 legacy=0x7fecc81416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccdeb5640 1 -- 172.21.15.60:0/1514221016 >> v1:172.21.15.60:6789/0 conn(0x7fecc80b40b0 legacy=0x7fecc80a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccdeb5640 10 monclient: _finish_hunting 0 2024-03-27T21:40:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccdeb5640 1 monclient: found mon.noname-a 2024-03-27T21:40:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccdeb5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccdeb5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.092+0000 7feccdeb5640 1 -- 172.21.15.60:0/1514221016 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc80aa520 con 0x7fecc813b480 2024-03-27T21:40:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.093+0000 7feccdeb5640 1 --2- 172.21.15.60:0/1514221016 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecc813b480 0x7fecc8141e00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fecc0009e50 tx=0x7fecc002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c2ed31cdb92a502 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.094+0000 7fecbffff640 1 -- 172.21.15.60:0/1514221016 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc002cbb0 con 0x7fecc813b480 2024-03-27T21:40:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.094+0000 7fecbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.094+0000 7fecbffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:31.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.094+0000 7fecbffff640 10 monclient: dump: 2024-03-27T21:40:31.193 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:31.193 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:31.193 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:31.193 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:31.193 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:31.193 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:31.193 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:31.193 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:31.193 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:31.193 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:31.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.094+0000 7fecbffff640 10 monclient: _finish_auth 0 2024-03-27T21:40:31.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.094+0000 7fecbffff640 10 monclient: _check_auth_tickets 2024-03-27T21:40:31.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.094+0000 7fecbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:31.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.094+0000 7feccf918640 5 monclient: authenticate success, global_id 4659 2024-03-27T21:40:31.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.094+0000 7fecbffff640 1 -- 172.21.15.60:0/1514221016 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc002cd10 con 0x7fecc813b480 2024-03-27T21:40:31.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.094+0000 7fecbffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:31.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.094+0000 7feccf918640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:31.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.094+0000 7fecbffff640 1 -- 172.21.15.60:0/1514221016 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc0034610 con 0x7fecc813b480 2024-03-27T21:40:31.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.094+0000 7feccf918640 4 set_mon_vals no callback set 2024-03-27T21:40:31.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.094+0000 7fecbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:31.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.094+0000 7fecbffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:31.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.094+0000 7fecbffff640 10 monclient: dump: 2024-03-27T21:40:31.194 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:31.194 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:31.194 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:31.194 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:31.194 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:31.195 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:31.195 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:31.195 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:31.195 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:31.195 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:31.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.094+0000 7feccf918640 10 monclient: shutdown 2024-03-27T21:40:31.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.095+0000 7feccf918640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:31.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.095+0000 7feccf918640 1 -- 172.21.15.60:0/1514221016 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecc813b480 msgr2=0x7fecc8141e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:31.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.095+0000 7feccf918640 1 --2- 172.21.15.60:0/1514221016 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecc813b480 0x7fecc8141e00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fecc0009e50 tx=0x7fecc002c010 comp rx=0 tx=0).stop 2024-03-27T21:40:31.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.095+0000 7feccf918640 1 -- 172.21.15.60:0/1514221016 shutdown_connections 2024-03-27T21:40:31.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.095+0000 7feccf918640 1 --2- 172.21.15.60:0/1514221016 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecc813b480 0x7fecc8141e00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:31.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.095+0000 7feccf918640 1 -- 172.21.15.60:0/1514221016 >> 172.21.15.60:0/1514221016 conn(0x7fecc80608e0 msgr2=0x7fecc8060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:31.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.095+0000 7feccf918640 1 -- 172.21.15.60:0/1514221016 shutdown_connections 2024-03-27T21:40:31.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.095+0000 7feccf918640 1 -- 172.21.15.60:0/1514221016 wait complete. 2024-03-27T21:40:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.096+0000 7feccf918640 10 monclient: build_initial_monmap 2024-03-27T21:40:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.096+0000 7feccf918640 10 monclient: monmap: 2024-03-27T21:40:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:31.196 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.096+0000 7feccf918640 1 librados: starting msgr at 2024-03-27T21:40:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.096+0000 7feccf918640 1 librados: starting objecter 2024-03-27T21:40:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.096+0000 7feccf918640 1 Processor -- start 2024-03-27T21:40:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.096+0000 7feccf918640 1 -- start start 2024-03-27T21:40:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.096+0000 7feccf918640 1 librados: setting wanted keys 2024-03-27T21:40:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.096+0000 7feccf918640 1 librados: calling monclient init 2024-03-27T21:40:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.096+0000 7feccf918640 10 monclient: init 2024-03-27T21:40:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.097+0000 7feccf918640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.097+0000 7feccf918640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:40:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.097+0000 7feccf918640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecc80b40b0 0x7fecc8138eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.097+0000 7feccf918640 10 monclient(hunting): picked mon.noname-b con 0x7fecc80b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.097+0000 7feccf918640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecc813b020 0x7fecc81d66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.097+0000 7feccf918640 10 monclient(hunting): picked mon.noname-a con 0x7fecc813b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.097+0000 7feccf918640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fecc813b480 0x7fecc81d6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.097+0000 7feccceb3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecc813b020 0x7fecc81d66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.097+0000 7feccceb3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecc813b020 0x7fecc81d66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40420/0 (socket says 172.21.15.60:40420) 2024-03-27T21:40:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.097+0000 7feccceb3640 1 -- 172.21.15.60:0/242917654 learned_addr learned my addr 172.21.15.60:0/242917654 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccf918640 10 monclient(hunting): picked mon.noname-c con 0x7fecc813b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccf918640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.097+0000 7feccd6b4640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecc80b40b0 0x7fecc8138eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc81d6220 con 0x7fecc813b020 2024-03-27T21:40:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccf918640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc81394f0 con 0x7fecc80b40b0 2024-03-27T21:40:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccf918640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccdeb5640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fecc813b480 0x7fecc81d6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc81d7130 con 0x7fecc813b480 2024-03-27T21:40:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccf918640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccf918640 10 monclient(hunting): authenticate will time out at 872.319001s 2024-03-27T21:40:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccceb3640 10 monclient(hunting): get_auth_request con 0x7fecc813b020 auth_method 0 2024-03-27T21:40:31.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccceb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:31.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccceb3640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:31.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccceb3640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:31.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccd6b4640 10 monclient(hunting): get_auth_request con 0x7fecc80b40b0 auth_method 0 2024-03-27T21:40:31.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccd6b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:31.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccd6b4640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:31.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccd6b4640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:31.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccdeb5640 10 monclient(hunting): get_auth_request con 0x7fecc813b480 auth_method 0 2024-03-27T21:40:31.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccdeb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:31.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccdeb5640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:31.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccdeb5640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:31.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccceb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:31.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccceb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:31.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccceb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:31.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccdeb5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:31.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccdeb5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:31.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccdeb5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:31.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccd6b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:31.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccd6b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:31.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccd6b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:31.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccceb3640 10 monclient(hunting): handle_auth_done global_id 4665 payload 824 2024-03-27T21:40:31.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccceb3640 1 -- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fecc813b480 msgr2=0x7fecc81d6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:31.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccceb3640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fecc813b480 0x7fecc81d6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:31.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccceb3640 1 -- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecc80b40b0 msgr2=0x7fecc8138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:31.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccceb3640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecc80b40b0 0x7fecc8138eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:31.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccceb3640 10 monclient: _finish_hunting 0 2024-03-27T21:40:31.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccceb3640 1 monclient: found mon.noname-a 2024-03-27T21:40:31.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccceb3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:31.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccceb3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:31.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccceb3640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc0009b30 con 0x7fecc813b020 2024-03-27T21:40:31.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccd6b4640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecc80b40b0 0x7fecc8138eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:31.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccdeb5640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fecc813b480 0x7fecc81d6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:31.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7feccceb3640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecc813b020 0x7fecc81d66b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fecb800c480 tx=0x7fecb800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:31.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7fecbdffb640 10 client.?.objecter ms_handle_connect 0x7fecc813b020 2024-03-27T21:40:31.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7fecbdffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:31.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.098+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb80052d0 con 0x7fecc813b020 2024-03-27T21:40:31.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.099+0000 7fecbdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:31.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.099+0000 7fecbdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:31.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.099+0000 7fecbdffb640 10 monclient: dump: 2024-03-27T21:40:31.201 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:31.201 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:31.201 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:31.201 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:31.202 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.099+0000 7fecbdffb640 10 monclient: _finish_auth 0 2024-03-27T21:40:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.099+0000 7fecbdffb640 10 monclient: _check_auth_tickets 2024-03-27T21:40:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.099+0000 7fecbdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.099+0000 7feccf918640 5 monclient: authenticate success, global_id 4665 2024-03-27T21:40:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.099+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb8005430 con 0x7fecc813b020 2024-03-27T21:40:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.099+0000 7fecbdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.099+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb8012870 con 0x7fecc813b020 2024-03-27T21:40:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.099+0000 7feccf918640 10 monclient: _renew_subs 2024-03-27T21:40:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.099+0000 7feccf918640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.099+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc813ce40 con 0x7fecc813b020 2024-03-27T21:40:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.099+0000 7fecbdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.099+0000 7fecbdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.099+0000 7fecbdffb640 10 monclient: dump: 2024-03-27T21:40:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:31.204 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.099+0000 7fecbf7fe640 4 set_mon_vals no callback set 2024-03-27T21:40:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.100+0000 7feccf918640 10 client.4665.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.100+0000 7feccf918640 10 monclient: _renew_subs 2024-03-27T21:40:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.100+0000 7feccf918640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.100+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc8138110 con 0x7fecc813b020 2024-03-27T21:40:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.100+0000 7feccf918640 1 librados: init done 2024-03-27T21:40:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.101+0000 7feccf918640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.101+0000 7feccf918640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.101+0000 7feccf918640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.101+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fecc8138300 con 0x7fecc813b020 2024-03-27T21:40:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.101+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fecb80129d0 con 0x7fecc813b020 2024-03-27T21:40:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.101+0000 7fecbdffb640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fecac06c5c0 0x7fecac06ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.101+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fecb8088350 con 0x7fecc813b020 2024-03-27T21:40:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.101+0000 7fecbdffb640 10 client.4665.objecter ms_dispatch 0x7fecc8000b90 osd_map(38..38 src has 1..38) v4 2024-03-27T21:40:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.101+0000 7fecbdffb640 3 client.4665.objecter handle_osd_map got epochs [38,38] > 0 2024-03-27T21:40:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.101+0000 7fecbdffb640 3 client.4665.objecter handle_osd_map decoding full epoch 38 2024-03-27T21:40:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7feccd6b4640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fecac06c5c0 0x7fecac06ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7fecbdffb640 20 client.4665.objecter dump_active .. 0 homeless 2024-03-27T21:40:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7feccd6b4640 10 monclient: get_auth_request con 0x7fecac06c5c0 auth_method 0 2024-03-27T21:40:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v38) v1 ==== 54+0+16 (secure 0 0 0) 0x7fecb8088650 con 0x7fecc813b020 2024-03-27T21:40:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7fecbdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7fecbdffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7feccf918640 10 client.4665.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign -1 1 2024-03-27T21:40:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7feccf918640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fecc813ab90 0x7fecc8132650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7fecbdffb640 10 client.4665.objecter ms_handle_connect 0x7fecac06c5c0 2024-03-27T21:40:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7feccd6b4640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fecac06c5c0 0x7fecac06ea80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fecc4000f60 tx=0x7fecc4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc8142340 osd=0 3 2024-03-27T21:40:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc8142340 osd=0 3 2024-03-27T21:40:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc8142340 osd=0 3 2024-03-27T21:40:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7feccf918640 15 client.4665.objecter _session_command_op_remove -1 1 2024-03-27T21:40:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7feccf918640 20 client.4665.objecter get_session s=0x7fecc8142340 osd=0 3 2024-03-27T21:40:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign 0 1 2024-03-27T21:40:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc8142340 osd=0 4 2024-03-27T21:40:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7feccf918640 10 client.4665.objecter _send_command 1 2024-03-27T21:40:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecc81d63d0 con 0x7fecc813ab90 2024-03-27T21:40:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7feccdeb5640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fecc813ab90 0x7fecc8132650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.102+0000 7feccdeb5640 10 monclient: get_auth_request con 0x7fecc813ab90 auth_method 0 2024-03-27T21:40:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.103+0000 7fecbdffb640 10 client.4665.objecter ms_handle_connect 0x7fecc813ab90 2024-03-27T21:40:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.103+0000 7feccdeb5640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fecc813ab90 0x7fecc8132650 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.104+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fecc81d63d0 con 0x7fecc813ab90 2024-03-27T21:40:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.104+0000 7fecbdffb640 10 client.4665.objecter ms_dispatch 0x7fecc8000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.104+0000 7fecbdffb640 10 client.4665.objecter _finish_command 1 = system:0 2024-03-27T21:40:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.104+0000 7fecbdffb640 20 client.4665.objecter put_session s=0x7fecc8142340 osd=0 4 2024-03-27T21:40:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.104+0000 7fecbdffb640 15 client.4665.objecter _session_command_op_remove 0 1 2024-03-27T21:40:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.132+0000 7feccf918640 10 client.4665.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.132+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign -1 2 2024-03-27T21:40:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.132+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc8142340 osd=0 3 2024-03-27T21:40:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.132+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc8142340 osd=0 3 2024-03-27T21:40:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.132+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc8142340 osd=0 3 2024-03-27T21:40:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.132+0000 7feccf918640 15 client.4665.objecter _session_command_op_remove -1 2 2024-03-27T21:40:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.132+0000 7feccf918640 20 client.4665.objecter get_session s=0x7fecc8142340 osd=0 3 2024-03-27T21:40:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.132+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign 0 2 2024-03-27T21:40:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.132+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc8142340 osd=0 4 2024-03-27T21:40:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.132+0000 7feccf918640 10 client.4665.objecter _send_command 2 2024-03-27T21:40:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.132+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fecc8069470 con 0x7fecc813ab90 2024-03-27T21:40:31.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.135+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fecc8069470 con 0x7fecc813ab90 2024-03-27T21:40:31.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.135+0000 7fecbdffb640 10 client.4665.objecter ms_dispatch 0x7fecc8000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:31.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.135+0000 7fecbdffb640 10 client.4665.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:31.208 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:31.211 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:31.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.135+0000 7fecbdffb640 20 client.4665.objecter put_session s=0x7fecc8142340 osd=0 4 2024-03-27T21:40:31.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.135+0000 7fecbdffb640 15 client.4665.objecter _session_command_op_remove 0 2 2024-03-27T21:40:31.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.135+0000 7feccf918640 10 client.4665.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:31.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.135+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign -1 3 2024-03-27T21:40:31.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.135+0000 7feccf918640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fecc81d9da0 0x7fecc81da180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:31.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.135+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc81d9720 osd=1 3 2024-03-27T21:40:31.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.135+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc81d9720 osd=1 3 2024-03-27T21:40:31.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.135+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc81d9720 osd=1 3 2024-03-27T21:40:31.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.135+0000 7feccf918640 15 client.4665.objecter _session_command_op_remove -1 3 2024-03-27T21:40:31.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.135+0000 7feccf918640 20 client.4665.objecter get_session s=0x7fecc81d9720 osd=1 3 2024-03-27T21:40:31.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.135+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign 1 3 2024-03-27T21:40:31.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.135+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc81d9720 osd=1 4 2024-03-27T21:40:31.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.135+0000 7feccf918640 10 client.4665.objecter _send_command 3 2024-03-27T21:40:31.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.135+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecc81e4a70 con 0x7fecc81d9da0 2024-03-27T21:40:31.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.135+0000 7feccd6b4640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fecc81d9da0 0x7fecc81da180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:31.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.135+0000 7feccd6b4640 10 monclient: get_auth_request con 0x7fecc81d9da0 auth_method 0 2024-03-27T21:40:31.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.136+0000 7fecbdffb640 10 client.4665.objecter ms_handle_connect 0x7fecc81d9da0 2024-03-27T21:40:31.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.136+0000 7feccd6b4640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fecc81d9da0 0x7fecc81da180 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:31.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.136+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fecc81e4a70 con 0x7fecc81d9da0 2024-03-27T21:40:31.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.136+0000 7fecbdffb640 10 client.4665.objecter ms_dispatch 0x7fecc8000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:31.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.136+0000 7fecbdffb640 10 client.4665.objecter _finish_command 3 = system:0 2024-03-27T21:40:31.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.136+0000 7fecbdffb640 20 client.4665.objecter put_session s=0x7fecc81d9720 osd=1 4 2024-03-27T21:40:31.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.136+0000 7fecbdffb640 15 client.4665.objecter _session_command_op_remove 1 3 2024-03-27T21:40:31.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.165+0000 7feccf918640 10 client.4665.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:31.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.165+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign -1 4 2024-03-27T21:40:31.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.165+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc81d9720 osd=1 3 2024-03-27T21:40:31.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.165+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc81d9720 osd=1 3 2024-03-27T21:40:31.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.165+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc81d9720 osd=1 3 2024-03-27T21:40:31.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.165+0000 7feccf918640 15 client.4665.objecter _session_command_op_remove -1 4 2024-03-27T21:40:31.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.165+0000 7feccf918640 20 client.4665.objecter get_session s=0x7fecc81d9720 osd=1 3 2024-03-27T21:40:31.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.165+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign 1 4 2024-03-27T21:40:31.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.165+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc81d9720 osd=1 4 2024-03-27T21:40:31.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.165+0000 7feccf918640 10 client.4665.objecter _send_command 4 2024-03-27T21:40:31.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.165+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fecc80b1d00 con 0x7fecc81d9da0 2024-03-27T21:40:31.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.168+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fecc80b1d00 con 0x7fecc81d9da0 2024-03-27T21:40:31.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.168+0000 7fecbdffb640 10 client.4665.objecter ms_dispatch 0x7fecc8000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:31.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.168+0000 7fecbdffb640 10 client.4665.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:31.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.168+0000 7fecbdffb640 20 client.4665.objecter put_session s=0x7fecc81d9720 osd=1 4 2024-03-27T21:40:31.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.168+0000 7fecbdffb640 15 client.4665.objecter _session_command_op_remove 1 4 2024-03-27T21:40:31.214 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:31.215 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:31.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.168+0000 7feccf918640 10 client.4665.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:31.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.168+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign -1 5 2024-03-27T21:40:31.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.168+0000 7feccf918640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7fecc813c730 0x7fecc802b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:31.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.168+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc80b2110 osd=2 3 2024-03-27T21:40:31.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.168+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc80b2110 osd=2 3 2024-03-27T21:40:31.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.168+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc80b2110 osd=2 3 2024-03-27T21:40:31.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.168+0000 7feccf918640 15 client.4665.objecter _session_command_op_remove -1 5 2024-03-27T21:40:31.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.168+0000 7feccf918640 20 client.4665.objecter get_session s=0x7fecc80b2110 osd=2 3 2024-03-27T21:40:31.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.168+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign 2 5 2024-03-27T21:40:31.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.168+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc80b2110 osd=2 4 2024-03-27T21:40:31.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.168+0000 7feccf918640 10 client.4665.objecter _send_command 5 2024-03-27T21:40:31.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.168+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecc81e8300 con 0x7fecc813c730 2024-03-27T21:40:31.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.168+0000 7feccceb3640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7fecc813c730 0x7fecc802b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:31.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.168+0000 7feccceb3640 10 monclient: get_auth_request con 0x7fecc813c730 auth_method 0 2024-03-27T21:40:31.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.169+0000 7fecbdffb640 10 client.4665.objecter ms_handle_connect 0x7fecc813c730 2024-03-27T21:40:31.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.169+0000 7feccceb3640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7fecc813c730 0x7fecc802b170 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:31.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.170+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== osd.2 v2:172.21.15.60:6824/44654 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fecc81e8300 con 0x7fecc813c730 2024-03-27T21:40:31.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.170+0000 7fecbdffb640 10 client.4665.objecter ms_dispatch 0x7fecc8000b90 command_reply(tid 5: 0 ) v1 2024-03-27T21:40:31.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.170+0000 7fecbdffb640 10 client.4665.objecter _finish_command 5 = system:0 2024-03-27T21:40:31.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.170+0000 7fecbdffb640 20 client.4665.objecter put_session s=0x7fecc80b2110 osd=2 4 2024-03-27T21:40:31.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.170+0000 7fecbdffb640 15 client.4665.objecter _session_command_op_remove 2 5 2024-03-27T21:40:31.217 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:31.189+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:31.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.198+0000 7feccf918640 10 client.4665.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:31.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.198+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign -1 6 2024-03-27T21:40:31.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.198+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc80b2110 osd=2 3 2024-03-27T21:40:31.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.198+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc80b2110 osd=2 3 2024-03-27T21:40:31.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.198+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc80b2110 osd=2 3 2024-03-27T21:40:31.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.198+0000 7feccf918640 15 client.4665.objecter _session_command_op_remove -1 6 2024-03-27T21:40:31.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.198+0000 7feccf918640 20 client.4665.objecter get_session s=0x7fecc80b2110 osd=2 3 2024-03-27T21:40:31.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.198+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign 2 6 2024-03-27T21:40:31.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.198+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc80b2110 osd=2 4 2024-03-27T21:40:31.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.198+0000 7feccf918640 10 client.4665.objecter _send_command 6 2024-03-27T21:40:31.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.198+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fecc813bca0 con 0x7fecc813c730 2024-03-27T21:40:31.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.200+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== osd.2 v2:172.21.15.60:6824/44654 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fecc813bca0 con 0x7fecc813c730 2024-03-27T21:40:31.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.200+0000 7fecbdffb640 10 client.4665.objecter ms_dispatch 0x7fecc8000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:31.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.200+0000 7fecbdffb640 10 client.4665.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:31.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.200+0000 7fecbdffb640 20 client.4665.objecter put_session s=0x7fecc80b2110 osd=2 4 2024-03-27T21:40:31.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.200+0000 7fecbdffb640 15 client.4665.objecter _session_command_op_remove 2 6 2024-03-27T21:40:31.220 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2: {} 2024-03-27T21:40:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-27T21:40:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.200+0000 7feccf918640 10 client.4665.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.200+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign -1 7 2024-03-27T21:40:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.200+0000 7feccf918640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fecc802c380 0x7fecc81e6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.200+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc813c050 osd=3 3 2024-03-27T21:40:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.200+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc813c050 osd=3 3 2024-03-27T21:40:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.200+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc813c050 osd=3 3 2024-03-27T21:40:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.200+0000 7feccf918640 15 client.4665.objecter _session_command_op_remove -1 7 2024-03-27T21:40:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.200+0000 7feccf918640 20 client.4665.objecter get_session s=0x7fecc813c050 osd=3 3 2024-03-27T21:40:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.200+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign 3 7 2024-03-27T21:40:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.200+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc813c050 osd=3 4 2024-03-27T21:40:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.200+0000 7feccf918640 10 client.4665.objecter _send_command 7 2024-03-27T21:40:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.200+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecc81ebb90 con 0x7fecc802c380 2024-03-27T21:40:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.201+0000 7feccdeb5640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fecc802c380 0x7fecc81e6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.201+0000 7feccdeb5640 10 monclient: get_auth_request con 0x7fecc802c380 auth_method 0 2024-03-27T21:40:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.201+0000 7fecbdffb640 10 client.4665.objecter ms_handle_connect 0x7fecc802c380 2024-03-27T21:40:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.201+0000 7feccdeb5640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fecc802c380 0x7fecc81e6bf0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.202+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fecc81ebb90 con 0x7fecc802c380 2024-03-27T21:40:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.202+0000 7fecbdffb640 10 client.4665.objecter ms_dispatch 0x7fecc8000b90 command_reply(tid 7: 0 ) v1 2024-03-27T21:40:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.202+0000 7fecbdffb640 10 client.4665.objecter _finish_command 7 = system:0 2024-03-27T21:40:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.202+0000 7fecbdffb640 20 client.4665.objecter put_session s=0x7fecc813c050 osd=3 4 2024-03-27T21:40:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.202+0000 7fecbdffb640 15 client.4665.objecter _session_command_op_remove 3 7 2024-03-27T21:40:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.230+0000 7feccf918640 10 client.4665.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.230+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign -1 8 2024-03-27T21:40:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.230+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc813c050 osd=3 3 2024-03-27T21:40:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.230+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc813c050 osd=3 3 2024-03-27T21:40:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.230+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc813c050 osd=3 3 2024-03-27T21:40:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.230+0000 7feccf918640 15 client.4665.objecter _session_command_op_remove -1 8 2024-03-27T21:40:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.230+0000 7feccf918640 20 client.4665.objecter get_session s=0x7fecc813c050 osd=3 3 2024-03-27T21:40:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.230+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign 3 8 2024-03-27T21:40:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.230+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc813c050 osd=3 4 2024-03-27T21:40:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.230+0000 7feccf918640 10 client.4665.objecter _send_command 8 2024-03-27T21:40:31.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.230+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fecc802b8f0 con 0x7fecc802c380 2024-03-27T21:40:31.224 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.233+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fecc802b8f0 con 0x7fecc802c380 2024-03-27T21:40:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.233+0000 7fecbdffb640 10 client.4665.objecter ms_dispatch 0x7fecc8000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.233+0000 7fecbdffb640 10 client.4665.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.233+0000 7fecbdffb640 20 client.4665.objecter put_session s=0x7fecc813c050 osd=3 4 2024-03-27T21:40:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.233+0000 7fecbdffb640 15 client.4665.objecter _session_command_op_remove 3 8 2024-03-27T21:40:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.233+0000 7feccf918640 10 client.4665.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.233+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign -1 9 2024-03-27T21:40:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.233+0000 7feccf918640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fecc81e7e00 0x7fecc81ea480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.233+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc802bb30 osd=4 3 2024-03-27T21:40:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.233+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc802bb30 osd=4 3 2024-03-27T21:40:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.233+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc802bb30 osd=4 3 2024-03-27T21:40:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.233+0000 7feccf918640 15 client.4665.objecter _session_command_op_remove -1 9 2024-03-27T21:40:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.233+0000 7feccf918640 20 client.4665.objecter get_session s=0x7fecc802bb30 osd=4 3 2024-03-27T21:40:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.233+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign 4 9 2024-03-27T21:40:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.233+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc802bb30 osd=4 4 2024-03-27T21:40:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.233+0000 7feccf918640 10 client.4665.objecter _send_command 9 2024-03-27T21:40:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.233+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecc802bca0 con 0x7fecc81e7e00 2024-03-27T21:40:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.233+0000 7feccd6b4640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fecc81e7e00 0x7fecc81ea480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.233+0000 7feccd6b4640 10 monclient: get_auth_request con 0x7fecc81e7e00 auth_method 0 2024-03-27T21:40:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.234+0000 7fecbdffb640 10 client.4665.objecter ms_handle_connect 0x7fecc81e7e00 2024-03-27T21:40:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.234+0000 7feccd6b4640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fecc81e7e00 0x7fecc81ea480 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.234+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fecc802bca0 con 0x7fecc81e7e00 2024-03-27T21:40:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.234+0000 7fecbdffb640 10 client.4665.objecter ms_dispatch 0x7fecc8000b90 command_reply(tid 9: 0 ) v1 2024-03-27T21:40:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.234+0000 7fecbdffb640 10 client.4665.objecter _finish_command 9 = system:0 2024-03-27T21:40:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.234+0000 7fecbdffb640 20 client.4665.objecter put_session s=0x7fecc802bb30 osd=4 4 2024-03-27T21:40:31.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.234+0000 7fecbdffb640 15 client.4665.objecter _session_command_op_remove 4 9 2024-03-27T21:40:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.262+0000 7feccf918640 10 client.4665.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.262+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign -1 10 2024-03-27T21:40:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.262+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc802bb30 osd=4 3 2024-03-27T21:40:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.262+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc802bb30 osd=4 3 2024-03-27T21:40:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.262+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc802bb30 osd=4 3 2024-03-27T21:40:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.262+0000 7feccf918640 15 client.4665.objecter _session_command_op_remove -1 10 2024-03-27T21:40:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.262+0000 7feccf918640 20 client.4665.objecter get_session s=0x7fecc802bb30 osd=4 3 2024-03-27T21:40:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.262+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign 4 10 2024-03-27T21:40:31.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.262+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc802bb30 osd=4 4 2024-03-27T21:40:31.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.262+0000 7feccf918640 10 client.4665.objecter _send_command 10 2024-03-27T21:40:31.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.262+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fecc802bca0 con 0x7fecc81e7e00 2024-03-27T21:40:31.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.264+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fecc802bca0 con 0x7fecc81e7e00 2024-03-27T21:40:31.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.264+0000 7fecbdffb640 10 client.4665.objecter ms_dispatch 0x7fecc8000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:31.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.264+0000 7fecbdffb640 10 client.4665.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:31.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.264+0000 7fecbdffb640 20 client.4665.objecter put_session s=0x7fecc802bb30 osd=4 4 2024-03-27T21:40:31.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.264+0000 7fecbdffb640 15 client.4665.objecter _session_command_op_remove 4 10 2024-03-27T21:40:31.251 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:31.252 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.265+0000 7feccf918640 10 client.4665.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.265+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign -1 11 2024-03-27T21:40:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.265+0000 7feccf918640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fecc8065fa0 0x7fecc8066400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.265+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc802bca0 osd=5 3 2024-03-27T21:40:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.265+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc802bca0 osd=5 3 2024-03-27T21:40:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.265+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc802bca0 osd=5 3 2024-03-27T21:40:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.265+0000 7feccf918640 15 client.4665.objecter _session_command_op_remove -1 11 2024-03-27T21:40:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.265+0000 7feccf918640 20 client.4665.objecter get_session s=0x7fecc802bca0 osd=5 3 2024-03-27T21:40:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.265+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign 5 11 2024-03-27T21:40:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.265+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc802bca0 osd=5 4 2024-03-27T21:40:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.265+0000 7feccf918640 10 client.4665.objecter _send_command 11 2024-03-27T21:40:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.265+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecc8066940 con 0x7fecc8065fa0 2024-03-27T21:40:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.265+0000 7feccceb3640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fecc8065fa0 0x7fecc8066400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.265+0000 7feccceb3640 10 monclient: get_auth_request con 0x7fecc8065fa0 auth_method 0 2024-03-27T21:40:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.266+0000 7feccceb3640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fecc8065fa0 0x7fecc8066400 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.266+0000 7fecbdffb640 10 client.4665.objecter ms_handle_connect 0x7fecc8065fa0 2024-03-27T21:40:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.266+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fecc8066940 con 0x7fecc8065fa0 2024-03-27T21:40:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.266+0000 7fecbdffb640 10 client.4665.objecter ms_dispatch 0x7fecc8000b90 command_reply(tid 11: 0 ) v1 2024-03-27T21:40:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.266+0000 7fecbdffb640 10 client.4665.objecter _finish_command 11 = system:0 2024-03-27T21:40:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.266+0000 7fecbdffb640 20 client.4665.objecter put_session s=0x7fecc802bca0 osd=5 4 2024-03-27T21:40:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.266+0000 7fecbdffb640 15 client.4665.objecter _session_command_op_remove 5 11 2024-03-27T21:40:31.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.281+0000 7feccf918640 10 client.4665.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:31.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.281+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign -1 12 2024-03-27T21:40:31.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.281+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc802bca0 osd=5 3 2024-03-27T21:40:31.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.281+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc802bca0 osd=5 3 2024-03-27T21:40:31.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.281+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc802bca0 osd=5 3 2024-03-27T21:40:31.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.281+0000 7feccf918640 15 client.4665.objecter _session_command_op_remove -1 12 2024-03-27T21:40:31.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.281+0000 7feccf918640 20 client.4665.objecter get_session s=0x7fecc802bca0 osd=5 3 2024-03-27T21:40:31.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.281+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign 5 12 2024-03-27T21:40:31.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.281+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc802bca0 osd=5 4 2024-03-27T21:40:31.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.281+0000 7feccf918640 10 client.4665.objecter _send_command 12 2024-03-27T21:40:31.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.281+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fecc8066940 con 0x7fecc8065fa0 2024-03-27T21:40:31.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.285+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fecc8066940 con 0x7fecc8065fa0 2024-03-27T21:40:31.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.285+0000 7fecbdffb640 10 client.4665.objecter ms_dispatch 0x7fecc8000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:31.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.285+0000 7fecbdffb640 10 client.4665.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:31.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.285+0000 7fecbdffb640 20 client.4665.objecter put_session s=0x7fecc802bca0 osd=5 4 2024-03-27T21:40:31.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.285+0000 7fecbdffb640 15 client.4665.objecter _session_command_op_remove 5 12 2024-03-27T21:40:31.271 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:31.272 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:31.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.285+0000 7feccf918640 10 client.4665.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:31.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.285+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign -1 13 2024-03-27T21:40:31.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.285+0000 7feccf918640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fecc81f0230 0x7fecc81f2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:31.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.285+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc8066940 osd=6 3 2024-03-27T21:40:31.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.285+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc8066940 osd=6 3 2024-03-27T21:40:31.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.285+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc8066940 osd=6 3 2024-03-27T21:40:31.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.285+0000 7feccf918640 15 client.4665.objecter _session_command_op_remove -1 13 2024-03-27T21:40:31.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.285+0000 7feccf918640 20 client.4665.objecter get_session s=0x7fecc8066940 osd=6 3 2024-03-27T21:40:31.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.285+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign 6 13 2024-03-27T21:40:31.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.285+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc8066940 osd=6 4 2024-03-27T21:40:31.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.285+0000 7feccf918640 10 client.4665.objecter _send_command 13 2024-03-27T21:40:31.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.285+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecc81f5430 con 0x7fecc81f0230 2024-03-27T21:40:31.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.285+0000 7feccdeb5640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fecc81f0230 0x7fecc81f2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:31.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.285+0000 7feccdeb5640 10 monclient: get_auth_request con 0x7fecc81f0230 auth_method 0 2024-03-27T21:40:31.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.285+0000 7feccdeb5640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fecc81f0230 0x7fecc81f2620 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:31.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.285+0000 7fecbdffb640 10 client.4665.objecter ms_handle_connect 0x7fecc81f0230 2024-03-27T21:40:31.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.286+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fecc81f5430 con 0x7fecc81f0230 2024-03-27T21:40:31.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.286+0000 7fecbdffb640 10 client.4665.objecter ms_dispatch 0x7fecc8000b90 command_reply(tid 13: 0 ) v1 2024-03-27T21:40:31.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.286+0000 7fecbdffb640 10 client.4665.objecter _finish_command 13 = system:0 2024-03-27T21:40:31.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.286+0000 7fecbdffb640 20 client.4665.objecter put_session s=0x7fecc8066940 osd=6 4 2024-03-27T21:40:31.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.286+0000 7fecbdffb640 15 client.4665.objecter _session_command_op_remove 6 13 2024-03-27T21:40:31.276 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:31.290+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.300+0000 7feccf918640 10 client.4665.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.300+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign -1 14 2024-03-27T21:40:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.300+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc8066940 osd=6 3 2024-03-27T21:40:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.300+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc8066940 osd=6 3 2024-03-27T21:40:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.300+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc8066940 osd=6 3 2024-03-27T21:40:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.300+0000 7feccf918640 15 client.4665.objecter _session_command_op_remove -1 14 2024-03-27T21:40:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.300+0000 7feccf918640 20 client.4665.objecter get_session s=0x7fecc8066940 osd=6 3 2024-03-27T21:40:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.300+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign 6 14 2024-03-27T21:40:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.300+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc8066940 osd=6 4 2024-03-27T21:40:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.300+0000 7feccf918640 10 client.4665.objecter _send_command 14 2024-03-27T21:40:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.300+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fecc81f2b60 con 0x7fecc81f0230 2024-03-27T21:40:31.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.302+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fecc81f2b60 con 0x7fecc81f0230 2024-03-27T21:40:31.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.302+0000 7fecbdffb640 10 client.4665.objecter ms_dispatch 0x7fecc8000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:31.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.302+0000 7fecbdffb640 10 client.4665.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:31.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.302+0000 7fecbdffb640 20 client.4665.objecter put_session s=0x7fecc8066940 osd=6 4 2024-03-27T21:40:31.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.302+0000 7fecbdffb640 15 client.4665.objecter _session_command_op_remove 6 14 2024-03-27T21:40:31.289 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:31.290 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:31.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.302+0000 7feccf918640 10 client.4665.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:31.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.302+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign -1 15 2024-03-27T21:40:31.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.302+0000 7feccf918640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fecc81f3230 0x7fecc81f3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:31.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.302+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc81f2b60 osd=7 3 2024-03-27T21:40:31.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.302+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc81f2b60 osd=7 3 2024-03-27T21:40:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.302+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc81f2b60 osd=7 3 2024-03-27T21:40:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.302+0000 7feccf918640 15 client.4665.objecter _session_command_op_remove -1 15 2024-03-27T21:40:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.302+0000 7feccf918640 20 client.4665.objecter get_session s=0x7fecc81f2b60 osd=7 3 2024-03-27T21:40:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.302+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign 7 15 2024-03-27T21:40:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.302+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc81f2b60 osd=7 4 2024-03-27T21:40:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.302+0000 7feccf918640 10 client.4665.objecter _send_command 15 2024-03-27T21:40:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.302+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecc81f8760 con 0x7fecc81f3230 2024-03-27T21:40:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.303+0000 7feccd6b4640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fecc81f3230 0x7fecc81f3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.303+0000 7feccd6b4640 10 monclient: get_auth_request con 0x7fecc81f3230 auth_method 0 2024-03-27T21:40:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.303+0000 7fecbdffb640 10 client.4665.objecter ms_handle_connect 0x7fecc81f3230 2024-03-27T21:40:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.303+0000 7feccd6b4640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fecc81f3230 0x7fecc81f3670 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.304+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fecc81f8760 con 0x7fecc81f3230 2024-03-27T21:40:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.304+0000 7fecbdffb640 10 client.4665.objecter ms_dispatch 0x7fecc8000b90 command_reply(tid 15: 0 ) v1 2024-03-27T21:40:31.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.304+0000 7fecbdffb640 10 client.4665.objecter _finish_command 15 = system:0 2024-03-27T21:40:31.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.304+0000 7fecbdffb640 20 client.4665.objecter put_session s=0x7fecc81f2b60 osd=7 4 2024-03-27T21:40:31.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.304+0000 7fecbdffb640 15 client.4665.objecter _session_command_op_remove 7 15 2024-03-27T21:40:31.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.318+0000 7feccf918640 10 client.4665.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:31.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.318+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign -1 16 2024-03-27T21:40:31.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.318+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc81f2b60 osd=7 3 2024-03-27T21:40:31.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.318+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc81f2b60 osd=7 3 2024-03-27T21:40:31.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.318+0000 7feccf918640 20 client.4665.objecter _get_session s=0x7fecc81f2b60 osd=7 3 2024-03-27T21:40:31.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.318+0000 7feccf918640 15 client.4665.objecter _session_command_op_remove -1 16 2024-03-27T21:40:31.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.318+0000 7feccf918640 20 client.4665.objecter get_session s=0x7fecc81f2b60 osd=7 3 2024-03-27T21:40:31.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.318+0000 7feccf918640 15 client.4665.objecter _session_command_op_assign 7 16 2024-03-27T21:40:31.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.318+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc81f2b60 osd=7 4 2024-03-27T21:40:31.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.318+0000 7feccf918640 10 client.4665.objecter _send_command 16 2024-03-27T21:40:31.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.318+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fecc81f3cc0 con 0x7fecc81f3230 2024-03-27T21:40:31.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.319+0000 7fecbdffb640 1 -- 172.21.15.60:0/242917654 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fecc81f3cc0 con 0x7fecc81f3230 2024-03-27T21:40:31.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.319+0000 7fecbdffb640 10 client.4665.objecter ms_dispatch 0x7fecc8000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:31.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.319+0000 7fecbdffb640 10 client.4665.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:31.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.319+0000 7fecbdffb640 20 client.4665.objecter put_session s=0x7fecc81f2b60 osd=7 4 2024-03-27T21:40:31.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.319+0000 7fecbdffb640 15 client.4665.objecter _session_command_op_remove 7 16 2024-03-27T21:40:31.306 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:31.308 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:31.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 10 librados: watch_flush enter 2024-03-27T21:40:31.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 10 librados: watch_flush exit 2024-03-27T21:40:31.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 10 client.4665.objecter close_session for osd.0 2024-03-27T21:40:31.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fecc813ab90 msgr2=0x7fecc8132650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:31.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fecc813ab90 0x7fecc8132650 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:31.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc8142340 osd=0 1 2024-03-27T21:40:31.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 10 client.4665.objecter close_session for osd.1 2024-03-27T21:40:31.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fecc81d9da0 msgr2=0x7fecc81da180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:31.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fecc81d9da0 0x7fecc81da180 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:31.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc81d9720 osd=1 1 2024-03-27T21:40:31.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 10 client.4665.objecter close_session for osd.2 2024-03-27T21:40:31.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7fecc813c730 msgr2=0x7fecc802b170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:31.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7fecc813c730 0x7fecc802b170 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:31.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccdeb5640 1 -- 172.21.15.60:0/242917654 reap_dead start 2024-03-27T21:40:31.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc80b2110 osd=2 1 2024-03-27T21:40:31.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 10 client.4665.objecter close_session for osd.3 2024-03-27T21:40:31.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fecc802c380 msgr2=0x7fecc81e6bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:31.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fecc802c380 0x7fecc81e6bf0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:31.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc813c050 osd=3 1 2024-03-27T21:40:31.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 10 client.4665.objecter close_session for osd.4 2024-03-27T21:40:31.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fecc81e7e00 msgr2=0x7fecc81ea480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:31.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fecc81e7e00 0x7fecc81ea480 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:31.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc802bb30 osd=4 1 2024-03-27T21:40:31.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 10 client.4665.objecter close_session for osd.5 2024-03-27T21:40:31.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fecc8065fa0 msgr2=0x7fecc8066400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:31.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fecc8065fa0 0x7fecc8066400 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:31.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc802bca0 osd=5 1 2024-03-27T21:40:31.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 10 client.4665.objecter close_session for osd.6 2024-03-27T21:40:31.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fecc81f0230 msgr2=0x7fecc81f2620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:31.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fecc81f0230 0x7fecc81f2620 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:31.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc8066940 osd=6 1 2024-03-27T21:40:31.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 10 client.4665.objecter close_session for osd.7 2024-03-27T21:40:31.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fecc81f3230 msgr2=0x7fecc81f3670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:31.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fecc81f3230 0x7fecc81f3670 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:31.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccdeb5640 1 -- 172.21.15.60:0/242917654 reap_dead start 2024-03-27T21:40:31.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 20 client.4665.objecter put_session s=0x7fecc81f2b60 osd=7 1 2024-03-27T21:40:31.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 20 client.4665.objecter shutdown clearing up homeless session... 2024-03-27T21:40:31.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 10 client.4665.objecter successfully canceled tick 2024-03-27T21:40:31.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fecac06c5c0 msgr2=0x7fecac06ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:31.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fecac06c5c0 0x7fecac06ea80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fecc4000f60 tx=0x7fecc4009040 comp rx=0 tx=0).stop 2024-03-27T21:40:31.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 10 monclient: shutdown 2024-03-27T21:40:31.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:31.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecc813b020 msgr2=0x7fecc81d66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:31.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecc813b020 0x7fecc81d66b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fecb800c480 tx=0x7fecb800c940 comp rx=0 tx=0).stop 2024-03-27T21:40:31.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 shutdown_connections 2024-03-27T21:40:31.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecc813b020 0x7fecc81d66b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:31.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 --2- 172.21.15.60:0/242917654 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fecac06c5c0 0x7fecac06ea80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:31.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.320+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 >> 172.21.15.60:0/242917654 conn(0x7fecc80608e0 msgr2=0x7fecc80a8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:31.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.321+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 shutdown_connections 2024-03-27T21:40:31.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.321+0000 7feccf918640 1 -- 172.21.15.60:0/242917654 wait complete. 2024-03-27T21:40:31.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.321+0000 7feccf918640 1 librados: shutdown 2024-03-27T21:40:31.377 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:31.391+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:31.388 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:31.388 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:31.401 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-27T21:40:31.442 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-27T21:40:31.486 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:31.500+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:31.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.560+0000 7f4fc2767640 10 monclient: get_monmap_and_config 2024-03-27T21:40:31.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.560+0000 7f4fc2767640 10 monclient: build_initial_monmap 2024-03-27T21:40:31.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.560+0000 7f4fc2767640 10 monclient: monmap: 2024-03-27T21:40:31.547 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:31.547 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:31.547 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:31.561599+0000 2024-03-27T21:40:31.547 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:31.561599+0000 2024-03-27T21:40:31.547 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:31.547 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:31.548 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:31.548 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:31.548 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:31.548 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:31.548 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:31.548 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:31.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.561+0000 7f4fc2767640 1 Processor -- start 2024-03-27T21:40:31.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.561+0000 7f4fc2767640 1 -- start start 2024-03-27T21:40:31.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.561+0000 7f4fc2767640 10 monclient: init 2024-03-27T21:40:31.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc2767640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:31.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc2767640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-03-27T21:40:31.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc2767640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4fbc0b40b0 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:31.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc2767640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fbc13b020 0x7f4fbc13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:31.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc2767640 10 monclient(hunting): picked mon.noname-b con 0x7f4fbc13b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:31.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc2767640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4fbc1416f0 0x7f4fbc141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:31.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc2767640 10 monclient(hunting): picked mon.noname-a con 0x7f4fbc1416f0 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:31.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc2767640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fbc0b3830 con 0x7f4fbc0b40b0 2024-03-27T21:40:31.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc2767640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:31.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc2767640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fbc0b4490 con 0x7f4fbc1416f0 2024-03-27T21:40:31.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc2767640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc2767640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc0aa520 con 0x7f4fbc13b020 2024-03-27T21:40:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc2767640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc2767640 10 monclient(hunting): authenticate will time out at 872.783145s 2024-03-27T21:40:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fbb7fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fbc13b020 0x7f4fbc13b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fbbfff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f4fbc0b40b0 0x7f4fbc0a8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.60:50394/0 (socket says 172.21.15.60:50394) 2024-03-27T21:40:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fbbfff640 1 -- 172.21.15.60:0/954853255 learned_addr learned my addr 172.21.15.60:0/954853255 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:40:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fbb7fe640 10 monclient(hunting): get_auth_request con 0x7f4fbc13b020 auth_method 0 2024-03-27T21:40:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fbb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc0d04640 1 --2- 172.21.15.60:0/954853255 >> v2:172.21.15.60:3300/0 conn(0x7f4fbc1416f0 0x7f4fbc141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc0d04640 10 monclient(hunting): get_auth_request con 0x7f4fbc1416f0 auth_method 0 2024-03-27T21:40:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc0d04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc0d04640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc0d04640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fbb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fbb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fbb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc0d04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc0d04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fc0d04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fbb7fe640 10 monclient(hunting): handle_auth_done global_id 4535 payload 274 2024-03-27T21:40:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fbb7fe640 1 -- 172.21.15.60:0/954853255 >> v2:172.21.15.60:3300/0 conn(0x7f4fbc1416f0 msgr2=0x7f4fbc141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fbb7fe640 1 --2- 172.21.15.60:0/954853255 >> v2:172.21.15.60:3300/0 conn(0x7f4fbc1416f0 0x7f4fbc141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fbb7fe640 1 -- 172.21.15.60:0/954853255 >> v1:172.21.15.191:6789/0 conn(0x7f4fbc0b40b0 legacy=0x7f4fbc0a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fbb7fe640 10 monclient: _finish_hunting 0 2024-03-27T21:40:31.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fbb7fe640 1 monclient: found mon.noname-b 2024-03-27T21:40:31.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fbb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:31.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fbb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:31.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fbb7fe640 1 -- 172.21.15.60:0/954853255 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fbc0a9470 con 0x7f4fbc13b020 2024-03-27T21:40:31.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.563+0000 7f4fc0d04640 1 --2- 172.21.15.60:0/954853255 >> v2:172.21.15.60:3300/0 conn(0x7f4fbc1416f0 0x7f4fbc141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:31.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.562+0000 7f4fbaffd640 1 -- 172.21.15.60:0/954853255 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637797702 0 0) 0x7f4fbc0b3830 con 0x7f4fbc0b40b0 2024-03-27T21:40:31.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.563+0000 7f4fbaffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:40:31.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.564+0000 7f4fbb7fe640 1 --2- 172.21.15.60:0/954853255 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fbc13b020 0x7f4fbc13b480 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0009a00 tx=0x7f4fb002bb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4a632e084170102 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:31.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.564+0000 7f4fbaffd640 1 -- 172.21.15.60:0/954853255 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb002d6c0 con 0x7f4fbc13b020 2024-03-27T21:40:31.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.564+0000 7f4fbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:31.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.564+0000 7f4fbaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:31.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.564+0000 7f4fbaffd640 10 monclient: dump: 2024-03-27T21:40:31.564 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:31.564 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:31.565 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:31.565 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:31.565 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:31.565 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:31.565 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:31.565 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:31.565 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:31.565 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:31.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.564+0000 7f4fbaffd640 10 monclient: _finish_auth 0 2024-03-27T21:40:31.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.564+0000 7f4fbaffd640 10 monclient: _check_auth_tickets 2024-03-27T21:40:31.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.564+0000 7f4fbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:31.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.564+0000 7f4fbaffd640 1 -- 172.21.15.60:0/954853255 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb002d820 con 0x7f4fbc13b020 2024-03-27T21:40:31.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.564+0000 7f4fbaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:31.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.564+0000 7f4fbaffd640 1 -- 172.21.15.60:0/954853255 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb002dbb0 con 0x7f4fbc13b020 2024-03-27T21:40:31.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.564+0000 7f4fbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:31.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.564+0000 7f4fbaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:31.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.564+0000 7f4fbaffd640 10 monclient: dump: 2024-03-27T21:40:31.566 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:31.566 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:31.566 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:31.566 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:31.566 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:31.566 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:31.566 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:31.566 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:31.566 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:31.566 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:31.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.564+0000 7f4fc2767640 5 monclient: authenticate success, global_id 4535 2024-03-27T21:40:31.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.564+0000 7f4fc2767640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:31.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.564+0000 7f4fc2767640 4 set_mon_vals no callback set 2024-03-27T21:40:31.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.565+0000 7f4fc2767640 10 monclient: shutdown 2024-03-27T21:40:31.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.565+0000 7f4fc2767640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:31.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.565+0000 7f4fc2767640 1 -- 172.21.15.60:0/954853255 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fbc13b020 msgr2=0x7f4fbc13b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:31.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.565+0000 7f4fc2767640 1 --2- 172.21.15.60:0/954853255 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fbc13b020 0x7f4fbc13b480 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0009a00 tx=0x7f4fb002bb10 comp rx=0 tx=0).stop 2024-03-27T21:40:31.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.565+0000 7f4fc2767640 1 -- 172.21.15.60:0/954853255 shutdown_connections 2024-03-27T21:40:31.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.565+0000 7f4fc2767640 1 --2- 172.21.15.60:0/954853255 >> v2:172.21.15.60:3300/0 conn(0x7f4fbc1416f0 0x7f4fbc141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:31.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.565+0000 7f4fc2767640 1 --2- 172.21.15.60:0/954853255 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fbc13b020 0x7f4fbc13b480 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:31.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.565+0000 7f4fc2767640 1 -- 172.21.15.60:0/954853255 >> 172.21.15.60:0/954853255 conn(0x7f4fbc0608e0 msgr2=0x7f4fbc060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:31.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.565+0000 7f4fc2767640 1 -- 172.21.15.60:0/954853255 shutdown_connections 2024-03-27T21:40:31.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.565+0000 7f4fc2767640 1 -- 172.21.15.60:0/954853255 wait complete. 2024-03-27T21:40:31.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.566+0000 7f4fc2767640 10 monclient: build_initial_monmap 2024-03-27T21:40:31.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.566+0000 7f4fc2767640 10 monclient: monmap: 2024-03-27T21:40:31.568 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:31.568 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:31.568 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:31.568 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:31.568 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:31.568 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:31.568 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:31.568 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:31.568 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:31.568 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:31.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.567+0000 7f4fc2767640 1 librados: starting msgr at 2024-03-27T21:40:31.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.567+0000 7f4fc2767640 1 librados: starting objecter 2024-03-27T21:40:31.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.567+0000 7f4fc2767640 1 Processor -- start 2024-03-27T21:40:31.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.567+0000 7f4fc2767640 1 -- start start 2024-03-27T21:40:31.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.567+0000 7f4fc2767640 1 librados: setting wanted keys 2024-03-27T21:40:31.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.567+0000 7f4fc2767640 1 librados: calling monclient init 2024-03-27T21:40:31.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.567+0000 7f4fc2767640 10 monclient: init 2024-03-27T21:40:31.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc2767640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:31.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc2767640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:40:31.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc2767640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fbc0b40b0 0x7f4fbc0ac7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:31.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc2767640 10 monclient(hunting): picked mon.noname-a con 0x7f4fbc0b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:31.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc2767640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fbc13b020 0x7f4fbc0accf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:31.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc2767640 10 monclient(hunting): picked mon.noname-b con 0x7f4fbc13b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:31.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc2767640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fbc1416f0 0x7f4fbc0ae590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:31.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc0d04640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fbc1416f0 0x7f4fbc0ae590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:31.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fbbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fbc0b40b0 0x7f4fbc0ac7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:31.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc0d04640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fbc1416f0 0x7f4fbc0ae590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:59440/0 (socket says 172.21.15.60:59440) 2024-03-27T21:40:31.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fbbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fbc0b40b0 0x7f4fbc0ac7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58196/0 (socket says 172.21.15.60:58196) 2024-03-27T21:40:31.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc0d04640 1 -- 172.21.15.60:0/3945352668 learned_addr learned my addr 172.21.15.60:0/3945352668 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:31.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc2767640 10 monclient(hunting): picked mon.noname-c con 0x7f4fbc1416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:31.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc2767640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:31.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc2767640 1 -- 172.21.15.60:0/3945352668 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc0aadf0 con 0x7f4fbc0b40b0 2024-03-27T21:40:31.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fbb7fe640 1 --2- 172.21.15.60:0/3945352668 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fbc13b020 0x7f4fbc0accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:31.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc2767640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:31.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc2767640 1 -- 172.21.15.60:0/3945352668 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc0aaf30 con 0x7f4fbc13b020 2024-03-27T21:40:31.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc2767640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:31.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc2767640 1 -- 172.21.15.60:0/3945352668 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc13e670 con 0x7f4fbc1416f0 2024-03-27T21:40:31.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc2767640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:31.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc2767640 10 monclient(hunting): authenticate will time out at 872.789829s 2024-03-27T21:40:31.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fbbfff640 10 monclient(hunting): get_auth_request con 0x7f4fbc0b40b0 auth_method 0 2024-03-27T21:40:31.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:31.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:31.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:31.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc0d04640 10 monclient(hunting): get_auth_request con 0x7f4fbc1416f0 auth_method 0 2024-03-27T21:40:31.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc0d04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:31.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc0d04640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:31.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fc0d04640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:31.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fbb7fe640 10 monclient(hunting): get_auth_request con 0x7f4fbc13b020 auth_method 0 2024-03-27T21:40:31.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:31.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fbb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:31.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.568+0000 7f4fbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:31.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fbb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:31.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fbb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:31.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fbb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:31.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:31.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:31.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:31.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fc0d04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:31.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fc0d04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:31.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fc0d04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:31.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fc0d04640 10 monclient(hunting): handle_auth_done global_id 4538 payload 824 2024-03-27T21:40:31.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fc0d04640 1 -- 172.21.15.60:0/3945352668 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fbc13b020 msgr2=0x7f4fbc0accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:31.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fc0d04640 1 --2- 172.21.15.60:0/3945352668 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fbc13b020 0x7f4fbc0accf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:31.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fc0d04640 1 -- 172.21.15.60:0/3945352668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fbc0b40b0 msgr2=0x7f4fbc0ac7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:31.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fc0d04640 1 --2- 172.21.15.60:0/3945352668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fbc0b40b0 0x7f4fbc0ac7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:31.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fc0d04640 10 monclient: _finish_hunting 0 2024-03-27T21:40:31.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fc0d04640 1 monclient: found mon.noname-c 2024-03-27T21:40:31.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fc0d04640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:31.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fc0d04640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:40:31.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fc0d04640 1 -- 172.21.15.60:0/3945352668 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb4009a70 con 0x7f4fbc1416f0 2024-03-27T21:40:31.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fbb7fe640 1 --2- 172.21.15.60:0/3945352668 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fbc13b020 0x7f4fbc0accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:31.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fbbfff640 1 --2- 172.21.15.60:0/3945352668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fbc0b40b0 0x7f4fbc0ac7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:31.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fc0d04640 1 --2- 172.21.15.60:0/3945352668 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fbc1416f0 0x7f4fbc0ae590 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400efc0 tx=0x7f4fb400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:31.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fb8ff9640 10 client.?.objecter ms_handle_connect 0x7f4fbc1416f0 2024-03-27T21:40:31.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fb8ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:31.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fb8ff9640 1 -- 172.21.15.60:0/3945352668 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4009cd0 con 0x7f4fbc1416f0 2024-03-27T21:40:31.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fb8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:31.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fb8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:31.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fb8ff9640 10 monclient: dump: 2024-03-27T21:40:31.573 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:31.573 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:31.573 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:31.574 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:31.574 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:31.574 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:31.574 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:31.574 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:31.574 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:31.574 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:31.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fb8ff9640 10 monclient: _finish_auth 0 2024-03-27T21:40:31.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fb8ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:40:31.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fb8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:31.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fb8ff9640 1 -- 172.21.15.60:0/3945352668 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb400f040 con 0x7f4fbc1416f0 2024-03-27T21:40:31.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fb8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:31.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fb8ff9640 1 -- 172.21.15.60:0/3945352668 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4003990 con 0x7f4fbc1416f0 2024-03-27T21:40:31.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fb8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:31.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fb8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:31.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fb8ff9640 10 monclient: dump: 2024-03-27T21:40:31.575 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:31.575 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:31.575 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:31.575 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:31.575 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:31.575 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:31.575 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:31.575 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:31.575 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:31.575 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:31.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.569+0000 7f4fba7fc640 4 set_mon_vals no callback set 2024-03-27T21:40:31.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.570+0000 7f4fc2767640 5 monclient: authenticate success, global_id 4538 2024-03-27T21:40:31.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.570+0000 7f4fc2767640 10 monclient: _renew_subs 2024-03-27T21:40:31.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.570+0000 7f4fc2767640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:31.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.570+0000 7f4fc2767640 1 -- 172.21.15.60:0/3945352668 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb0009400 con 0x7f4fbc1416f0 2024-03-27T21:40:31.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.570+0000 7f4fc2767640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:31.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.570+0000 7f4fc2767640 10 monclient: _renew_subs 2024-03-27T21:40:31.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.570+0000 7f4fc2767640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:31.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.570+0000 7f4fc2767640 1 -- 172.21.15.60:0/3945352668 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fbc0ac060 con 0x7f4fbc1416f0 2024-03-27T21:40:31.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.570+0000 7f4fc2767640 1 librados: init done 2024-03-27T21:40:31.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.571+0000 7f4fc2767640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:31.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.571+0000 7f4fc2767640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:31.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.571+0000 7f4fc2767640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:31.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.571+0000 7f4fc2767640 1 -- 172.21.15.60:0/3945352668 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fbc069470 con 0x7f4fbc1416f0 2024-03-27T21:40:31.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.572+0000 7f4fb8ff9640 1 -- 172.21.15.60:0/3945352668 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4fb4014480 con 0x7f4fbc1416f0 2024-03-27T21:40:31.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.572+0000 7f4fb8ff9640 1 --2- 172.21.15.60:0/3945352668 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4fa806c590 0x7f4fa806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:31.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.572+0000 7f4fb8ff9640 1 -- 172.21.15.60:0/3945352668 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f4fb4089ba0 con 0x7f4fbc1416f0 2024-03-27T21:40:31.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.572+0000 7f4fb8ff9640 10 client.4538.objecter ms_dispatch 0x7f4fbc000b90 osd_map(38..38 src has 1..38) v4 2024-03-27T21:40:31.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.572+0000 7f4fb8ff9640 3 client.4538.objecter handle_osd_map got epochs [38,38] > 0 2024-03-27T21:40:31.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.572+0000 7f4fb8ff9640 3 client.4538.objecter handle_osd_map decoding full epoch 38 2024-03-27T21:40:31.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.573+0000 7f4fb8ff9640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-27T21:40:31.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.573+0000 7f4fbbfff640 1 --2- 172.21.15.60:0/3945352668 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4fa806c590 0x7f4fa806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:31.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.573+0000 7f4fbbfff640 10 monclient: get_auth_request con 0x7f4fa806c590 auth_method 0 2024-03-27T21:40:31.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.573+0000 7f4fb8ff9640 10 client.4538.objecter ms_handle_connect 0x7f4fa806c590 2024-03-27T21:40:31.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.573+0000 7f4fbbfff640 1 --2- 172.21.15.60:0/3945352668 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4fa806c590 0x7f4fa806ea50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4fac000f60 tx=0x7f4fac007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:31.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.580+0000 7f4fb8ff9640 1 -- 172.21.15.60:0/3945352668 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4fb4055090 con 0x7f4fbc1416f0 2024-03-27T21:40:31.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.580+0000 7f4fb8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:31.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.580+0000 7f4fb8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:31.580 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:31.580 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:31.580 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:31.594+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:31.589 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-27T21:40:31.681 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:31.695+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:31.782 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:31.782 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:31.782 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:31.796+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:31.790 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:40:31.883 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:31.897+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:31.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.938+0000 7f4fc2767640 10 start_mon_command cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:40:31.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.938+0000 7f4fc2767640 10 monclient: _send_command 2 [{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:40:31.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.938+0000 7f4fc2767640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:31.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:31.938+0000 7f4fc2767640 1 -- 172.21.15.60:0/3945352668 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f4fbc0b69a0 con 0x7f4fbc1416f0 2024-03-27T21:40:31.942 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:31.943 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:31.952 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:40:31.985 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:31.998+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:32.061 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:32.061 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:32.070 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:40:32.114 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:32.128+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:32.179 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:32.180 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:32.190 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:40:32.191 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:32.204+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:32.291 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:32.305+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:32.353 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:32.353 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:32.365 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:40:32.409 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:32.423+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:32.429 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-03-27T21:40:32.430 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T21:40:32.430 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-03-27T21:40:32.430 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-27T21:40:32.430 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-27T21:40:32.430 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-03-27T21:40:32.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.444+0000 7f4fb8ff9640 1 -- 172.21.15.60:0/3945352668 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v39) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4fb4018090 con 0x7f4fbc1416f0 2024-03-27T21:40:32.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.444+0000 7f4fb8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:40:32.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.444+0000 7f4fb8ff9640 10 monclient: _finish_command 2 = system:0 nodeep-scrub is set 2024-03-27T21:40:32.431 INFO:teuthology.orchestra.run.smithi060.stderr:nodeep-scrub is set 2024-03-27T21:40:32.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.447+0000 7f4fc2767640 10 librados: watch_flush enter 2024-03-27T21:40:32.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.447+0000 7f4fc2767640 10 librados: watch_flush exit 2024-03-27T21:40:32.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.447+0000 7f4fc2767640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-27T21:40:32.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.447+0000 7f4fc2767640 10 client.4538.objecter successfully canceled tick 2024-03-27T21:40:32.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.447+0000 7f4fc2767640 1 -- 172.21.15.60:0/3945352668 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4fa806c590 msgr2=0x7f4fa806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:32.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.447+0000 7f4fc2767640 1 --2- 172.21.15.60:0/3945352668 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4fa806c590 0x7f4fa806ea50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4fac000f60 tx=0x7f4fac007d50 comp rx=0 tx=0).stop 2024-03-27T21:40:32.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.447+0000 7f4fc2767640 10 monclient: shutdown 2024-03-27T21:40:32.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.447+0000 7f4fc2767640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:32.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.447+0000 7f4fc2767640 1 -- 172.21.15.60:0/3945352668 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fbc1416f0 msgr2=0x7f4fbc0ae590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:32.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.447+0000 7f4fc2767640 1 --2- 172.21.15.60:0/3945352668 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fbc1416f0 0x7f4fbc0ae590 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400efc0 tx=0x7f4fb400c490 comp rx=0 tx=0).stop 2024-03-27T21:40:32.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.447+0000 7f4fc2767640 1 -- 172.21.15.60:0/3945352668 shutdown_connections 2024-03-27T21:40:32.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.447+0000 7f4fc2767640 1 --2- 172.21.15.60:0/3945352668 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fbc1416f0 0x7f4fbc0ae590 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:32.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.447+0000 7f4fc2767640 1 --2- 172.21.15.60:0/3945352668 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4fa806c590 0x7f4fa806ea50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:32.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.447+0000 7f4fc2767640 1 --2- 172.21.15.60:0/3945352668 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fbc13b020 0x7f4fbc0accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:32.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.447+0000 7f4fc2767640 1 --2- 172.21.15.60:0/3945352668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fbc0b40b0 0x7f4fbc0ac7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:32.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.447+0000 7f4fc2767640 1 -- 172.21.15.60:0/3945352668 >> 172.21.15.60:0/3945352668 conn(0x7f4fbc0608e0 msgr2=0x7f4fbc0b5dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:32.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.447+0000 7f4fc2767640 1 -- 172.21.15.60:0/3945352668 shutdown_connections 2024-03-27T21:40:32.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.447+0000 7f4fc2767640 1 -- 172.21.15.60:0/3945352668 wait complete. 2024-03-27T21:40:32.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:32.447+0000 7f4fc2767640 1 librados: shutdown 2024-03-27T21:40:32.492 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:32.506+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:32.517 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:32.517 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:32.528 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:40:32.544 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:32.544 INFO:teuthology.orchestra.run.smithi191.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-27T21:40:32.544 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:32.594 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:32.607+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:32.676 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:32.676 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:32.685 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:40:32.729 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:32.743+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:32.794 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:32.794 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:32.795 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:32.809+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:32.803 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:40:32.896 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:32.910+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:32.952 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:32.952 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:32.961 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:40:33.002 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:33.016+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:33.098 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:33.112+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:33.111 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:33.111 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:33.120 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:40:33.199 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:33.213+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:33.237 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:33.237 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:33.246 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:40:33.319 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:40:33.330 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:33.344+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:33.393 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:33.393 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:33.401 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:40:33.402 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:33.414+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:33.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.436+0000 7f626c7c2640 10 monclient: get_monmap_and_config 2024-03-27T21:40:33.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.436+0000 7f626c7c2640 10 monclient: build_initial_monmap 2024-03-27T21:40:33.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.436+0000 7f626c7c2640 10 monclient: monmap: 2024-03-27T21:40:33.422 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:33.422 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:33.423 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:33.437355+0000 2024-03-27T21:40:33.423 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:33.437355+0000 2024-03-27T21:40:33.423 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:33.423 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:33.423 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:33.423 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:33.423 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:33.423 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:33.423 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:33.423 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:33.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.436+0000 7f626c7c2640 1 Processor -- start 2024-03-27T21:40:33.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.437+0000 7f626c7c2640 1 -- start start 2024-03-27T21:40:33.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.437+0000 7f626c7c2640 10 monclient: init 2024-03-27T21:40:33.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.437+0000 7f626c7c2640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:33.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.437+0000 7f626c7c2640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-03-27T21:40:33.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.437+0000 7f626c7c2640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6264071a50 0x7f62640a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:33.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.437+0000 7f626c7c2640 10 monclient(hunting): picked mon.noname-b con 0x7f6264071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:33.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.437+0000 7f626c7c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f626413e810 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:33.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.437+0000 7f626c7c2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f626413ec50 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:33.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.437+0000 7f626c7c2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62640a8e90 con 0x7f626413ec50 2024-03-27T21:40:33.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.437+0000 7f626c7c2640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62640aa040 con 0x7f626413e810 2024-03-27T21:40:33.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.437+0000 7f626c7c2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:33.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.437+0000 7f626c7c2640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6264071e30 con 0x7f6264071a50 2024-03-27T21:40:33.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.437+0000 7f626c7c2640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:33.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.437+0000 7f626c7c2640 10 monclient(hunting): authenticate will time out at 874.658781s 2024-03-27T21:40:33.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626ad5f640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f626413ec50 0x7f6264141d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59974/0 (socket says 172.21.15.60:59974) 2024-03-27T21:40:33.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626ad5f640 1 -- 172.21.15.60:0/1934818755 learned_addr learned my addr 172.21.15.60:0/1934818755 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:40:33.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626a55e640 1 --2- 172.21.15.60:0/1934818755 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6264071a50 0x7f62640a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:33.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626a55e640 10 monclient(hunting): get_auth_request con 0x7f6264071a50 auth_method 0 2024-03-27T21:40:33.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626a55e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:33.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626a55e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:33.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626a55e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:33.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626a55e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:33.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626a55e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:33.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626a55e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:33.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626955c640 1 -- 172.21.15.60:0/1934818755 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573916593 0 0) 0x7f62640aa040 con 0x7f626413e810 2024-03-27T21:40:33.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626955c640 10 monclient(hunting): _negotiate 2024-03-27T21:40:33.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626955c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:33.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626955c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:33.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626955c640 10 monclient(hunting): my global_id is 4618 2024-03-27T21:40:33.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626955c640 1 -- 172.21.15.60:0/1934818755 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6250003520 con 0x7f626413e810 2024-03-27T21:40:33.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626955c640 1 -- 172.21.15.60:0/1934818755 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4163550054 0 0) 0x7f62640a8e90 con 0x7f626413ec50 2024-03-27T21:40:33.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626a55e640 10 monclient(hunting): handle_auth_done global_id 4541 payload 274 2024-03-27T21:40:33.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626a55e640 1 -- 172.21.15.60:0/1934818755 >> v1:172.21.15.191:6789/0 conn(0x7f626413e810 legacy=0x7f6264141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:33.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626a55e640 1 -- 172.21.15.60:0/1934818755 >> v1:172.21.15.60:6789/0 conn(0x7f626413ec50 legacy=0x7f6264141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:33.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626a55e640 10 monclient: _finish_hunting 0 2024-03-27T21:40:33.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626a55e640 1 monclient: found mon.noname-b 2024-03-27T21:40:33.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626a55e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:33.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626a55e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:33.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626a55e640 1 -- 172.21.15.60:0/1934818755 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62640aa2b0 con 0x7f6264071a50 2024-03-27T21:40:33.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.438+0000 7f626955c640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:40:33.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.439+0000 7f626a55e640 1 --2- 172.21.15.60:0/1934818755 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6264071a50 0x7f62640a87e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6260009a30 tx=0x7f626002bba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55b2ca15c3190add server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:33.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.440+0000 7f626955c640 1 -- 172.21.15.60:0/1934818755 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626002c740 con 0x7f6264071a50 2024-03-27T21:40:33.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.440+0000 7f626955c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:33.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.440+0000 7f626955c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:33.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.440+0000 7f626955c640 10 monclient: dump: 2024-03-27T21:40:33.437 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:33.437 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:33.437 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:33.437 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:33.437 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:33.437 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:33.437 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:33.437 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:33.437 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:33.437 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:33.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.440+0000 7f626955c640 10 monclient: _finish_auth 0 2024-03-27T21:40:33.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.440+0000 7f626955c640 10 monclient: _check_auth_tickets 2024-03-27T21:40:33.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.440+0000 7f626955c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:33.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.440+0000 7f626955c640 1 -- 172.21.15.60:0/1934818755 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f626002c8a0 con 0x7f6264071a50 2024-03-27T21:40:33.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.440+0000 7f626c7c2640 5 monclient: authenticate success, global_id 4541 2024-03-27T21:40:33.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.440+0000 7f626c7c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:33.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.440+0000 7f626955c640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:33.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.440+0000 7f626955c640 1 -- 172.21.15.60:0/1934818755 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626002cc30 con 0x7f6264071a50 2024-03-27T21:40:33.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.440+0000 7f626c7c2640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:33.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.440+0000 7f626c7c2640 4 set_mon_vals no callback set 2024-03-27T21:40:33.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.440+0000 7f626955c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:33.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.440+0000 7f626955c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:33.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.440+0000 7f626955c640 10 monclient: dump: 2024-03-27T21:40:33.438 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:33.438 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:33.438 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:33.439 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:33.439 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:33.439 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:33.439 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:33.439 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:33.439 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:33.439 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:33.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.441+0000 7f626c7c2640 10 monclient: shutdown 2024-03-27T21:40:33.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.441+0000 7f626c7c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:33.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.441+0000 7f626c7c2640 1 -- 172.21.15.60:0/1934818755 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6264071a50 msgr2=0x7f62640a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:33.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.441+0000 7f626c7c2640 1 --2- 172.21.15.60:0/1934818755 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6264071a50 0x7f62640a87e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6260009a30 tx=0x7f626002bba0 comp rx=0 tx=0).stop 2024-03-27T21:40:33.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.441+0000 7f626c7c2640 1 -- 172.21.15.60:0/1934818755 shutdown_connections 2024-03-27T21:40:33.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.441+0000 7f626c7c2640 1 --2- 172.21.15.60:0/1934818755 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6264071a50 0x7f62640a87e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:33.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.441+0000 7f626c7c2640 1 -- 172.21.15.60:0/1934818755 >> 172.21.15.60:0/1934818755 conn(0x7f62640b7300 msgr2=0x7f62640b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:33.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.441+0000 7f626c7c2640 1 -- 172.21.15.60:0/1934818755 shutdown_connections 2024-03-27T21:40:33.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.441+0000 7f626c7c2640 1 -- 172.21.15.60:0/1934818755 wait complete. 2024-03-27T21:40:33.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.442+0000 7f626c7c2640 10 monclient: build_initial_monmap 2024-03-27T21:40:33.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.442+0000 7f626c7c2640 10 monclient: monmap: 2024-03-27T21:40:33.440 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:33.440 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:33.440 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:33.440 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:33.440 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:33.440 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:33.440 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:33.440 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:33.441 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:33.441 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:33.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.442+0000 7f626c7c2640 1 librados: starting msgr at 2024-03-27T21:40:33.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.442+0000 7f626c7c2640 1 librados: starting objecter 2024-03-27T21:40:33.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.442+0000 7f626c7c2640 1 Processor -- start 2024-03-27T21:40:33.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.442+0000 7f626c7c2640 1 -- start start 2024-03-27T21:40:33.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.442+0000 7f626c7c2640 1 librados: setting wanted keys 2024-03-27T21:40:33.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.442+0000 7f626c7c2640 1 librados: calling monclient init 2024-03-27T21:40:33.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.442+0000 7f626c7c2640 10 monclient: init 2024-03-27T21:40:33.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626c7c2640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:33.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626c7c2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:40:33.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626c7c2640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6264071a50 0x7f62641d6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:33.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626c7c2640 10 monclient(hunting): picked mon.noname-c con 0x7f6264071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:33.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626c7c2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f626413e810 0x7f62641d6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:33.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626c7c2640 10 monclient(hunting): picked mon.noname-b con 0x7f626413e810 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:33.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626c7c2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f626413ec50 0x7f626413b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:33.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626c7c2640 10 monclient(hunting): picked mon.noname-a con 0x7f626413ec50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:33.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626c7c2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:33.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626c7c2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626413a530 con 0x7f626413ec50 2024-03-27T21:40:33.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626c7c2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:33.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626c7c2640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62641d6590 con 0x7f626413e810 2024-03-27T21:40:33.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626c7c2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:33.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626c7c2640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626413a730 con 0x7f6264071a50 2024-03-27T21:40:33.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626c7c2640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:33.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626c7c2640 10 monclient(hunting): authenticate will time out at 874.664414s 2024-03-27T21:40:33.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f6269d5d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f626413e810 0x7f62641d6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:33.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f6269d5d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f626413e810 0x7f62641d6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:54610/0 (socket says 172.21.15.60:54610) 2024-03-27T21:40:33.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f6269d5d640 1 -- 172.21.15.60:0/3101508738 learned_addr learned my addr 172.21.15.60:0/3101508738 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:33.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f6269d5d640 10 monclient(hunting): get_auth_request con 0x7f626413e810 auth_method 0 2024-03-27T21:40:33.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f6269d5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:33.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f6269d5d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:33.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f6269d5d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:33.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626ad5f640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f626413ec50 0x7f626413b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:33.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f6269d5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:33.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f6269d5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:33.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f6269d5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:33.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626a55e640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6264071a50 0x7f62641d6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:33.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626ad5f640 10 monclient(hunting): get_auth_request con 0x7f626413ec50 auth_method 0 2024-03-27T21:40:33.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626ad5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:33.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626ad5f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:33.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626ad5f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:33.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626a55e640 10 monclient(hunting): get_auth_request con 0x7f6264071a50 auth_method 0 2024-03-27T21:40:33.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626a55e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:33.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626a55e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:33.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626a55e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:33.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626a55e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:33.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626a55e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:33.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f626a55e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:33.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f6269d5d640 10 monclient(hunting): handle_auth_done global_id 4621 payload 824 2024-03-27T21:40:33.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.443+0000 7f6269d5d640 1 -- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6264071a50 msgr2=0x7f62641d6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:33.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f6269d5d640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6264071a50 0x7f62641d6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:33.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f6269d5d640 1 -- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f626413ec50 msgr2=0x7f626413b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:33.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f6269d5d640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f626413ec50 0x7f626413b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:33.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f6269d5d640 10 monclient: _finish_hunting 0 2024-03-27T21:40:33.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f6269d5d640 1 monclient: found mon.noname-b 2024-03-27T21:40:33.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f6269d5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:33.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f6269d5d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:40:33.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f6269d5d640 1 -- 172.21.15.60:0/3101508738 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6260009400 con 0x7f626413e810 2024-03-27T21:40:33.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f626ad5f640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f626413ec50 0x7f626413b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:40:33.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f626a55e640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6264071a50 0x7f62641d6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:33.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f6269d5d640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f626413e810 0x7f62641d6f60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f625400c4c0 tx=0x7f625400c980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:33.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f625affd640 10 client.?.objecter ms_handle_connect 0x7f626413e810 2024-03-27T21:40:33.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f625affd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:33.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62540052d0 con 0x7f626413e810 2024-03-27T21:40:33.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f625affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:33.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f625affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:33.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f625affd640 10 monclient: dump: 2024-03-27T21:40:33.445 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:33.445 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:33.445 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:33.445 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:33.445 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:33.445 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:33.445 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:33.446 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:33.446 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:33.446 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:33.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f625affd640 10 monclient: _finish_auth 0 2024-03-27T21:40:33.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f625affd640 10 monclient: _check_auth_tickets 2024-03-27T21:40:33.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f625affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:33.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6254005430 con 0x7f626413e810 2024-03-27T21:40:33.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f625affd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:33.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f6268d5b640 4 set_mon_vals no callback set 2024-03-27T21:40:33.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6254012930 con 0x7f626413e810 2024-03-27T21:40:33.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f625affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:33.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f625affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:40:33.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f625affd640 10 monclient: dump: 2024-03-27T21:40:33.446 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:33.447 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:33.447 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:33.447 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:33.447 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:33.447 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:33.447 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:33.447 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:33.447 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:33.447 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:33.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f626c7c2640 5 monclient: authenticate success, global_id 4621 2024-03-27T21:40:33.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f626c7c2640 10 monclient: _renew_subs 2024-03-27T21:40:33.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f626c7c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:33.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f626413abb0 con 0x7f626413e810 2024-03-27T21:40:33.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f626c7c2640 10 client.4621.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:33.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f626c7c2640 10 monclient: _renew_subs 2024-03-27T21:40:33.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f626c7c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:33.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626413b940 con 0x7f626413e810 2024-03-27T21:40:33.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.444+0000 7f626c7c2640 1 librados: init done 2024-03-27T21:40:33.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.445+0000 7f626c7c2640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:33.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.445+0000 7f626c7c2640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:33.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.445+0000 7f626c7c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:33.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.445+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f626413add0 con 0x7f626413e810 2024-03-27T21:40:33.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.446+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6254012a90 con 0x7f626413e810 2024-03-27T21:40:33.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.446+0000 7f625affd640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f624806c590 0x7f624806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:33.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.446+0000 7f626a55e640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f624806c590 0x7f624806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:33.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.446+0000 7f626a55e640 10 monclient: get_auth_request con 0x7f624806c590 auth_method 0 2024-03-27T21:40:33.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f626a55e640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f624806c590 0x7f624806ea50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6260036000 tx=0x7f6260030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:33.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f6254088ce0 con 0x7f626413e810 2024-03-27T21:40:33.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f625affd640 10 client.4621.objecter ms_dispatch 0x7f6264000b90 osd_map(39..39 src has 1..39) v4 2024-03-27T21:40:33.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f625affd640 3 client.4621.objecter handle_osd_map got epochs [39,39] > 0 2024-03-27T21:40:33.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f625affd640 3 client.4621.objecter handle_osd_map decoding full epoch 39 2024-03-27T21:40:33.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f625affd640 20 client.4621.objecter dump_active .. 0 homeless 2024-03-27T21:40:33.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f625affd640 10 client.4621.objecter ms_handle_connect 0x7f624806c590 2024-03-27T21:40:33.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v39) v1 ==== 54+0+16 (secure 0 0 0) 0x7f625408b320 con 0x7f626413e810 2024-03-27T21:40:33.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f625affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:33.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f625affd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:33.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f626c7c2640 10 client.4621.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:33.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign -1 1 2024-03-27T21:40:33.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f626c7c2640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f6264138ee0 0x7f62641392c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:33.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f626413f0c0 osd=0 3 2024-03-27T21:40:33.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f626413f0c0 osd=0 3 2024-03-27T21:40:33.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f626413f0c0 osd=0 3 2024-03-27T21:40:33.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_remove -1 1 2024-03-27T21:40:33.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f626c7c2640 20 client.4621.objecter get_session s=0x7f626413f0c0 osd=0 3 2024-03-27T21:40:33.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign 0 1 2024-03-27T21:40:33.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f626413f0c0 osd=0 4 2024-03-27T21:40:33.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f626c7c2640 10 client.4621.objecter _send_command 1 2024-03-27T21:40:33.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.447+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6264069470 con 0x7f6264138ee0 2024-03-27T21:40:33.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.448+0000 7f626ad5f640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f6264138ee0 0x7f62641392c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:33.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.448+0000 7f626ad5f640 10 monclient: get_auth_request con 0x7f6264138ee0 auth_method 0 2024-03-27T21:40:33.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.449+0000 7f625affd640 10 client.4621.objecter ms_handle_connect 0x7f6264138ee0 2024-03-27T21:40:33.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.449+0000 7f626ad5f640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f6264138ee0 0x7f62641392c0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:33.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.450+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6264069470 con 0x7f6264138ee0 2024-03-27T21:40:33.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.450+0000 7f625affd640 10 client.4621.objecter ms_dispatch 0x7f6264000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:33.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.450+0000 7f625affd640 10 client.4621.objecter _finish_command 1 = system:0 2024-03-27T21:40:33.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.450+0000 7f625affd640 20 client.4621.objecter put_session s=0x7f626413f0c0 osd=0 4 2024-03-27T21:40:33.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.450+0000 7f625affd640 15 client.4621.objecter _session_command_op_remove 0 1 2024-03-27T21:40:33.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.478+0000 7f626c7c2640 10 client.4621.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:33.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.478+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign -1 2 2024-03-27T21:40:33.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.478+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f626413f0c0 osd=0 3 2024-03-27T21:40:33.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.478+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f626413f0c0 osd=0 3 2024-03-27T21:40:33.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.478+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f626413f0c0 osd=0 3 2024-03-27T21:40:33.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.478+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_remove -1 2 2024-03-27T21:40:33.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.478+0000 7f626c7c2640 20 client.4621.objecter get_session s=0x7f626413f0c0 osd=0 3 2024-03-27T21:40:33.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.478+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign 0 2 2024-03-27T21:40:33.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.478+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f626413f0c0 osd=0 4 2024-03-27T21:40:33.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.478+0000 7f626c7c2640 10 client.4621.objecter _send_command 2 2024-03-27T21:40:33.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.478+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f62641d66d0 con 0x7f6264138ee0 2024-03-27T21:40:33.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.481+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f62641d66d0 con 0x7f6264138ee0 2024-03-27T21:40:33.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.481+0000 7f625affd640 10 client.4621.objecter ms_dispatch 0x7f6264000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:33.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.481+0000 7f625affd640 10 client.4621.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:33.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.481+0000 7f625affd640 20 client.4621.objecter put_session s=0x7f626413f0c0 osd=0 4 2024-03-27T21:40:33.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.481+0000 7f625affd640 15 client.4621.objecter _session_command_op_remove 0 2 2024-03-27T21:40:33.468 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:33.469 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:33.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.481+0000 7f626c7c2640 10 client.4621.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:33.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.481+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign -1 3 2024-03-27T21:40:33.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.481+0000 7f626c7c2640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f62641d9e70 0x7f62641da250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:33.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.481+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f62640a9260 osd=1 3 2024-03-27T21:40:33.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.481+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f62640a9260 osd=1 3 2024-03-27T21:40:33.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.481+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f62640a9260 osd=1 3 2024-03-27T21:40:33.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.481+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_remove -1 3 2024-03-27T21:40:33.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.481+0000 7f626c7c2640 20 client.4621.objecter get_session s=0x7f62640a9260 osd=1 3 2024-03-27T21:40:33.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.481+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign 1 3 2024-03-27T21:40:33.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.481+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f62640a9260 osd=1 4 2024-03-27T21:40:33.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.481+0000 7f626c7c2640 10 client.4621.objecter _send_command 3 2024-03-27T21:40:33.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.481+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62641da790 con 0x7f62641d9e70 2024-03-27T21:40:33.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.481+0000 7f626a55e640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f62641d9e70 0x7f62641da250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:33.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.481+0000 7f626a55e640 10 monclient: get_auth_request con 0x7f62641d9e70 auth_method 0 2024-03-27T21:40:33.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.482+0000 7f626a55e640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f62641d9e70 0x7f62641da250 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:33.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.482+0000 7f625affd640 10 client.4621.objecter ms_handle_connect 0x7f62641d9e70 2024-03-27T21:40:33.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.482+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f62641da790 con 0x7f62641d9e70 2024-03-27T21:40:33.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.483+0000 7f625affd640 10 client.4621.objecter ms_dispatch 0x7f6264000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:33.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.483+0000 7f625affd640 10 client.4621.objecter _finish_command 3 = system:0 2024-03-27T21:40:33.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.483+0000 7f625affd640 20 client.4621.objecter put_session s=0x7f62640a9260 osd=1 4 2024-03-27T21:40:33.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.483+0000 7f625affd640 15 client.4621.objecter _session_command_op_remove 1 3 2024-03-27T21:40:33.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.497+0000 7f626c7c2640 10 client.4621.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:33.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.497+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign -1 4 2024-03-27T21:40:33.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.497+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f62640a9260 osd=1 3 2024-03-27T21:40:33.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.497+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f62640a9260 osd=1 3 2024-03-27T21:40:33.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.497+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f62640a9260 osd=1 3 2024-03-27T21:40:33.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.497+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_remove -1 4 2024-03-27T21:40:33.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.497+0000 7f626c7c2640 20 client.4621.objecter get_session s=0x7f62640a9260 osd=1 3 2024-03-27T21:40:33.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.497+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign 1 4 2024-03-27T21:40:33.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.497+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f62640a9260 osd=1 4 2024-03-27T21:40:33.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.497+0000 7f626c7c2640 10 client.4621.objecter _send_command 4 2024-03-27T21:40:33.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.497+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f626413bdd0 con 0x7f62641d9e70 2024-03-27T21:40:33.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.500+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f626002c6c0 con 0x7f62641d9e70 2024-03-27T21:40:33.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.500+0000 7f625affd640 10 client.4621.objecter ms_dispatch 0x7f6264000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:33.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.500+0000 7f625affd640 10 client.4621.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:33.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.500+0000 7f625affd640 20 client.4621.objecter put_session s=0x7f62640a9260 osd=1 4 2024-03-27T21:40:33.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.500+0000 7f625affd640 15 client.4621.objecter _session_command_op_remove 1 4 2024-03-27T21:40:33.487 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:33.489 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:33.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.500+0000 7f626c7c2640 10 client.4621.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:33.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.500+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign -1 5 2024-03-27T21:40:33.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.500+0000 7f626c7c2640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f62641419c0 0x7f626402b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:33.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.500+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f6264141850 osd=2 3 2024-03-27T21:40:33.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.500+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f6264141850 osd=2 3 2024-03-27T21:40:33.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.500+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f6264141850 osd=2 3 2024-03-27T21:40:33.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.500+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_remove -1 5 2024-03-27T21:40:33.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.500+0000 7f626c7c2640 20 client.4621.objecter get_session s=0x7f6264141850 osd=2 3 2024-03-27T21:40:33.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.500+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign 2 5 2024-03-27T21:40:33.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.500+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f6264141850 osd=2 4 2024-03-27T21:40:33.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.500+0000 7f626c7c2640 10 client.4621.objecter _send_command 5 2024-03-27T21:40:33.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.500+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f626413dbf0 con 0x7f62641419c0 2024-03-27T21:40:33.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.500+0000 7f6269d5d640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f62641419c0 0x7f626402b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:33.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.500+0000 7f6269d5d640 10 monclient: get_auth_request con 0x7f62641419c0 auth_method 0 2024-03-27T21:40:33.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.500+0000 7f625affd640 10 client.4621.objecter ms_handle_connect 0x7f62641419c0 2024-03-27T21:40:33.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.500+0000 7f6269d5d640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f62641419c0 0x7f626402b170 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:33.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.501+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== osd.2 v2:172.21.15.60:6824/44654 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f626413dbf0 con 0x7f62641419c0 2024-03-27T21:40:33.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.501+0000 7f625affd640 10 client.4621.objecter ms_dispatch 0x7f6264000b90 command_reply(tid 5: 0 ) v1 2024-03-27T21:40:33.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.501+0000 7f625affd640 10 client.4621.objecter _finish_command 5 = system:0 2024-03-27T21:40:33.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.501+0000 7f625affd640 20 client.4621.objecter put_session s=0x7f6264141850 osd=2 4 2024-03-27T21:40:33.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.501+0000 7f625affd640 15 client.4621.objecter _session_command_op_remove 2 5 2024-03-27T21:40:33.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.515+0000 7f626c7c2640 10 client.4621.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:33.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.515+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign -1 6 2024-03-27T21:40:33.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.515+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f6264141850 osd=2 3 2024-03-27T21:40:33.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.515+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f6264141850 osd=2 3 2024-03-27T21:40:33.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.515+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f6264141850 osd=2 3 2024-03-27T21:40:33.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.515+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_remove -1 6 2024-03-27T21:40:33.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.515+0000 7f626c7c2640 20 client.4621.objecter get_session s=0x7f6264141850 osd=2 3 2024-03-27T21:40:33.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.515+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign 2 6 2024-03-27T21:40:33.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.515+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f6264141850 osd=2 4 2024-03-27T21:40:33.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.515+0000 7f626c7c2640 10 client.4621.objecter _send_command 6 2024-03-27T21:40:33.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.515+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6264065a90 con 0x7f62641419c0 2024-03-27T21:40:33.503 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:33.515+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:33.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.517+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== osd.2 v2:172.21.15.60:6824/44654 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6264065a90 con 0x7f62641419c0 2024-03-27T21:40:33.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.517+0000 7f625affd640 10 client.4621.objecter ms_dispatch 0x7f6264000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:33.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.517+0000 7f625affd640 10 client.4621.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:33.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.517+0000 7f625affd640 20 client.4621.objecter put_session s=0x7f6264141850 osd=2 4 2024-03-27T21:40:33.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.517+0000 7f625affd640 15 client.4621.objecter _session_command_op_remove 2 6 2024-03-27T21:40:33.504 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2: {} 2024-03-27T21:40:33.505 INFO:teuthology.orchestra.run.smithi060.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-27T21:40:33.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.517+0000 7f626c7c2640 10 client.4621.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:33.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.517+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign -1 7 2024-03-27T21:40:33.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.517+0000 7f626c7c2640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f626413d2d0 0x7f62641e8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:33.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.517+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f626413d160 osd=3 3 2024-03-27T21:40:33.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.517+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f626413d160 osd=3 3 2024-03-27T21:40:33.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.517+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f626413d160 osd=3 3 2024-03-27T21:40:33.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.517+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_remove -1 7 2024-03-27T21:40:33.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.517+0000 7f626c7c2640 20 client.4621.objecter get_session s=0x7f626413d160 osd=3 3 2024-03-27T21:40:33.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.517+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign 3 7 2024-03-27T21:40:33.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.517+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f626413d160 osd=3 4 2024-03-27T21:40:33.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.517+0000 7f626c7c2640 10 client.4621.objecter _send_command 7 2024-03-27T21:40:33.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.517+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6264066930 con 0x7f626413d2d0 2024-03-27T21:40:33.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.517+0000 7f626ad5f640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f626413d2d0 0x7f62641e8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:33.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.517+0000 7f626ad5f640 10 monclient: get_auth_request con 0x7f626413d2d0 auth_method 0 2024-03-27T21:40:33.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.517+0000 7f625affd640 10 client.4621.objecter ms_handle_connect 0x7f626413d2d0 2024-03-27T21:40:33.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.517+0000 7f626ad5f640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f626413d2d0 0x7f62641e8cd0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:33.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.518+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6264066930 con 0x7f626413d2d0 2024-03-27T21:40:33.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.518+0000 7f625affd640 10 client.4621.objecter ms_dispatch 0x7f6264000b90 command_reply(tid 7: 0 ) v1 2024-03-27T21:40:33.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.518+0000 7f625affd640 10 client.4621.objecter _finish_command 7 = system:0 2024-03-27T21:40:33.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.518+0000 7f625affd640 20 client.4621.objecter put_session s=0x7f626413d160 osd=3 4 2024-03-27T21:40:33.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.518+0000 7f625affd640 15 client.4621.objecter _session_command_op_remove 3 7 2024-03-27T21:40:33.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.531+0000 7f626c7c2640 10 client.4621.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:33.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.531+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign -1 8 2024-03-27T21:40:33.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.531+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f626413d160 osd=3 3 2024-03-27T21:40:33.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.531+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f626413d160 osd=3 3 2024-03-27T21:40:33.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.532+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f626413d160 osd=3 3 2024-03-27T21:40:33.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.532+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_remove -1 8 2024-03-27T21:40:33.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.532+0000 7f626c7c2640 20 client.4621.objecter get_session s=0x7f626413d160 osd=3 3 2024-03-27T21:40:33.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.532+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign 3 8 2024-03-27T21:40:33.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.532+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f626413d160 osd=3 4 2024-03-27T21:40:33.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.532+0000 7f626c7c2640 10 client.4621.objecter _send_command 8 2024-03-27T21:40:33.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.532+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6264066aa0 con 0x7f626413d2d0 2024-03-27T21:40:33.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.533+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f625c007940 con 0x7f626413d2d0 2024-03-27T21:40:33.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.533+0000 7f625affd640 10 client.4621.objecter ms_dispatch 0x7f6264000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:33.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.533+0000 7f625affd640 10 client.4621.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:33.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.533+0000 7f625affd640 20 client.4621.objecter put_session s=0x7f626413d160 osd=3 4 2024-03-27T21:40:33.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.533+0000 7f625affd640 15 client.4621.objecter _session_command_op_remove 3 8 2024-03-27T21:40:33.520 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:33.521 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:33.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.533+0000 7f626c7c2640 10 client.4621.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:33.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.533+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign -1 9 2024-03-27T21:40:33.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.533+0000 7f626c7c2640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f6264065fb0 0x7f62641ec3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:33.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.533+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f6264065e40 osd=4 3 2024-03-27T21:40:33.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.533+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f6264065e40 osd=4 3 2024-03-27T21:40:33.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.533+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f6264065e40 osd=4 3 2024-03-27T21:40:33.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.533+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_remove -1 9 2024-03-27T21:40:33.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.533+0000 7f626c7c2640 20 client.4621.objecter get_session s=0x7f6264065e40 osd=4 3 2024-03-27T21:40:33.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.533+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign 4 9 2024-03-27T21:40:33.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.533+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f6264065e40 osd=4 4 2024-03-27T21:40:33.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.533+0000 7f626c7c2640 10 client.4621.objecter _send_command 9 2024-03-27T21:40:33.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.533+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f626402c380 con 0x7f6264065fb0 2024-03-27T21:40:33.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.534+0000 7f626a55e640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f6264065fb0 0x7f62641ec3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:33.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.534+0000 7f626a55e640 10 monclient: get_auth_request con 0x7f6264065fb0 auth_method 0 2024-03-27T21:40:33.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.534+0000 7f626a55e640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f6264065fb0 0x7f62641ec3a0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:33.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.534+0000 7f625affd640 10 client.4621.objecter ms_handle_connect 0x7f6264065fb0 2024-03-27T21:40:33.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.535+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f626402c380 con 0x7f6264065fb0 2024-03-27T21:40:33.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.535+0000 7f625affd640 10 client.4621.objecter ms_dispatch 0x7f6264000b90 command_reply(tid 9: 0 ) v1 2024-03-27T21:40:33.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.535+0000 7f625affd640 10 client.4621.objecter _finish_command 9 = system:0 2024-03-27T21:40:33.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.535+0000 7f625affd640 20 client.4621.objecter put_session s=0x7f6264065e40 osd=4 4 2024-03-27T21:40:33.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.535+0000 7f625affd640 15 client.4621.objecter _session_command_op_remove 4 9 2024-03-27T21:40:33.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.564+0000 7f626c7c2640 10 client.4621.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.564+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign -1 10 2024-03-27T21:40:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.564+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f6264065e40 osd=4 3 2024-03-27T21:40:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.564+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f6264065e40 osd=4 3 2024-03-27T21:40:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.564+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f6264065e40 osd=4 3 2024-03-27T21:40:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.564+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_remove -1 10 2024-03-27T21:40:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.564+0000 7f626c7c2640 20 client.4621.objecter get_session s=0x7f6264065e40 osd=4 3 2024-03-27T21:40:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.564+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign 4 10 2024-03-27T21:40:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.564+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f6264065e40 osd=4 4 2024-03-27T21:40:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.564+0000 7f626c7c2640 10 client.4621.objecter _send_command 10 2024-03-27T21:40:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.564+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f626402c380 con 0x7f6264065fb0 2024-03-27T21:40:33.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.567+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f626402c380 con 0x7f6264065fb0 2024-03-27T21:40:33.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.567+0000 7f625affd640 10 client.4621.objecter ms_dispatch 0x7f6264000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:33.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.567+0000 7f625affd640 10 client.4621.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:33.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.567+0000 7f625affd640 20 client.4621.objecter put_session s=0x7f6264065e40 osd=4 4 2024-03-27T21:40:33.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.567+0000 7f625affd640 15 client.4621.objecter _session_command_op_remove 4 10 2024-03-27T21:40:33.554 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.567+0000 7f626c7c2640 10 client.4621.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.567+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign -1 11 2024-03-27T21:40:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.567+0000 7f626c7c2640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f62641ecdf0 0x7f62641ef1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.567+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f626402c380 osd=5 3 2024-03-27T21:40:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.567+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f626402c380 osd=5 3 2024-03-27T21:40:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.567+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f626402c380 osd=5 3 2024-03-27T21:40:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.567+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_remove -1 11 2024-03-27T21:40:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.567+0000 7f626c7c2640 20 client.4621.objecter get_session s=0x7f626402c380 osd=5 3 2024-03-27T21:40:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.567+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign 5 11 2024-03-27T21:40:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.567+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f626402c380 osd=5 4 2024-03-27T21:40:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.567+0000 7f626c7c2640 10 client.4621.objecter _send_command 11 2024-03-27T21:40:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.567+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f626402c5d0 con 0x7f62641ecdf0 2024-03-27T21:40:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.567+0000 7f6269d5d640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f62641ecdf0 0x7f62641ef1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.567+0000 7f6269d5d640 10 monclient: get_auth_request con 0x7f62641ecdf0 auth_method 0 2024-03-27T21:40:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.568+0000 7f625affd640 10 client.4621.objecter ms_handle_connect 0x7f62641ecdf0 2024-03-27T21:40:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.568+0000 7f6269d5d640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f62641ecdf0 0x7f62641ef1e0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.568+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f626402c5d0 con 0x7f62641ecdf0 2024-03-27T21:40:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.568+0000 7f625affd640 10 client.4621.objecter ms_dispatch 0x7f6264000b90 command_reply(tid 11: 0 ) v1 2024-03-27T21:40:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.568+0000 7f625affd640 10 client.4621.objecter _finish_command 11 = system:0 2024-03-27T21:40:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.568+0000 7f625affd640 20 client.4621.objecter put_session s=0x7f626402c380 osd=5 4 2024-03-27T21:40:33.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.568+0000 7f625affd640 15 client.4621.objecter _session_command_op_remove 5 11 2024-03-27T21:40:33.557 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:33.557 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:33.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:33.557 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:33.557 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:33.565 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:40:33.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.582+0000 7f626c7c2640 10 client.4621.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:33.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.582+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign -1 12 2024-03-27T21:40:33.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.582+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f626402c380 osd=5 3 2024-03-27T21:40:33.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.582+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f626402c380 osd=5 3 2024-03-27T21:40:33.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.582+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f626402c380 osd=5 3 2024-03-27T21:40:33.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.582+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_remove -1 12 2024-03-27T21:40:33.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.582+0000 7f626c7c2640 20 client.4621.objecter get_session s=0x7f626402c380 osd=5 3 2024-03-27T21:40:33.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.582+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign 5 12 2024-03-27T21:40:33.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.582+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f626402c380 osd=5 4 2024-03-27T21:40:33.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.582+0000 7f626c7c2640 10 client.4621.objecter _send_command 12 2024-03-27T21:40:33.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.582+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f626402c5d0 con 0x7f62641ecdf0 2024-03-27T21:40:33.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.584+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f626402c5d0 con 0x7f62641ecdf0 2024-03-27T21:40:33.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.584+0000 7f625affd640 10 client.4621.objecter ms_dispatch 0x7f6264000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:33.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.584+0000 7f625affd640 10 client.4621.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:33.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.584+0000 7f625affd640 20 client.4621.objecter put_session s=0x7f626402c380 osd=5 4 2024-03-27T21:40:33.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.584+0000 7f625affd640 15 client.4621.objecter _session_command_op_remove 5 12 2024-03-27T21:40:33.571 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:33.572 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:33.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.584+0000 7f626c7c2640 10 client.4621.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:33.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.584+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign -1 13 2024-03-27T21:40:33.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.584+0000 7f626c7c2640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f62641efc30 0x7f62641f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:33.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.584+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f626402c5d0 osd=6 3 2024-03-27T21:40:33.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.584+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f626402c5d0 osd=6 3 2024-03-27T21:40:33.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.584+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f626402c5d0 osd=6 3 2024-03-27T21:40:33.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.584+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_remove -1 13 2024-03-27T21:40:33.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.584+0000 7f626c7c2640 20 client.4621.objecter get_session s=0x7f626402c5d0 osd=6 3 2024-03-27T21:40:33.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.584+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign 6 13 2024-03-27T21:40:33.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.584+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f626402c5d0 osd=6 4 2024-03-27T21:40:33.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.584+0000 7f626c7c2640 10 client.4621.objecter _send_command 13 2024-03-27T21:40:33.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.584+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62641f5000 con 0x7f62641efc30 2024-03-27T21:40:33.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.584+0000 7f626ad5f640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f62641efc30 0x7f62641f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:33.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.584+0000 7f626ad5f640 10 monclient: get_auth_request con 0x7f62641efc30 auth_method 0 2024-03-27T21:40:33.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.585+0000 7f625affd640 10 client.4621.objecter ms_handle_connect 0x7f62641efc30 2024-03-27T21:40:33.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.585+0000 7f626ad5f640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f62641efc30 0x7f62641f2040 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:33.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.585+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f62641f5000 con 0x7f62641efc30 2024-03-27T21:40:33.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.585+0000 7f625affd640 10 client.4621.objecter ms_dispatch 0x7f6264000b90 command_reply(tid 13: 0 ) v1 2024-03-27T21:40:33.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.585+0000 7f625affd640 10 client.4621.objecter _finish_command 13 = system:0 2024-03-27T21:40:33.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.585+0000 7f625affd640 20 client.4621.objecter put_session s=0x7f626402c5d0 osd=6 4 2024-03-27T21:40:33.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.585+0000 7f625affd640 15 client.4621.objecter _session_command_op_remove 6 13 2024-03-27T21:40:33.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.599+0000 7f626c7c2640 10 client.4621.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:33.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.599+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign -1 14 2024-03-27T21:40:33.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.599+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f626402c5d0 osd=6 3 2024-03-27T21:40:33.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.599+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f626402c5d0 osd=6 3 2024-03-27T21:40:33.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.599+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f626402c5d0 osd=6 3 2024-03-27T21:40:33.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.599+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_remove -1 14 2024-03-27T21:40:33.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.599+0000 7f626c7c2640 20 client.4621.objecter get_session s=0x7f626402c5d0 osd=6 3 2024-03-27T21:40:33.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.599+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign 6 14 2024-03-27T21:40:33.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.599+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f626402c5d0 osd=6 4 2024-03-27T21:40:33.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.599+0000 7f626c7c2640 10 client.4621.objecter _send_command 14 2024-03-27T21:40:33.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.599+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f62641f5000 con 0x7f62641efc30 2024-03-27T21:40:33.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.601+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f62641f5000 con 0x7f62641efc30 2024-03-27T21:40:33.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.601+0000 7f625affd640 10 client.4621.objecter ms_dispatch 0x7f6264000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:33.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.601+0000 7f625affd640 10 client.4621.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:33.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.601+0000 7f625affd640 20 client.4621.objecter put_session s=0x7f626402c5d0 osd=6 4 2024-03-27T21:40:33.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.601+0000 7f625affd640 15 client.4621.objecter _session_command_op_remove 6 14 2024-03-27T21:40:33.588 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:33.589 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:33.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.602+0000 7f626c7c2640 10 client.4621.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:33.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.602+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign -1 15 2024-03-27T21:40:33.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.602+0000 7f626c7c2640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f62641f4330 0x7f62641f47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:33.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.602+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f62641f5000 osd=7 3 2024-03-27T21:40:33.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.602+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f62641f5000 osd=7 3 2024-03-27T21:40:33.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.602+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f62641f5000 osd=7 3 2024-03-27T21:40:33.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.602+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_remove -1 15 2024-03-27T21:40:33.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.602+0000 7f626c7c2640 20 client.4621.objecter get_session s=0x7f62641f5000 osd=7 3 2024-03-27T21:40:33.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.602+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign 7 15 2024-03-27T21:40:33.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.602+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f62641f5000 osd=7 4 2024-03-27T21:40:33.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.602+0000 7f626c7c2640 10 client.4621.objecter _send_command 15 2024-03-27T21:40:33.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.602+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62641f83e0 con 0x7f62641f4330 2024-03-27T21:40:33.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.602+0000 7f626a55e640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f62641f4330 0x7f62641f47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:33.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.602+0000 7f626a55e640 10 monclient: get_auth_request con 0x7f62641f4330 auth_method 0 2024-03-27T21:40:33.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.602+0000 7f625affd640 10 client.4621.objecter ms_handle_connect 0x7f62641f4330 2024-03-27T21:40:33.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.602+0000 7f626a55e640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f62641f4330 0x7f62641f47b0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:33.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.603+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f62641f83e0 con 0x7f62641f4330 2024-03-27T21:40:33.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.603+0000 7f625affd640 10 client.4621.objecter ms_dispatch 0x7f6264000b90 command_reply(tid 15: 0 ) v1 2024-03-27T21:40:33.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.603+0000 7f625affd640 10 client.4621.objecter _finish_command 15 = system:0 2024-03-27T21:40:33.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.603+0000 7f625affd640 20 client.4621.objecter put_session s=0x7f62641f5000 osd=7 4 2024-03-27T21:40:33.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.603+0000 7f625affd640 15 client.4621.objecter _session_command_op_remove 7 15 2024-03-27T21:40:33.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.617+0000 7f626c7c2640 10 client.4621.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:33.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.617+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign -1 16 2024-03-27T21:40:33.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.617+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f62641f5000 osd=7 3 2024-03-27T21:40:33.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.617+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f62641f5000 osd=7 3 2024-03-27T21:40:33.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.617+0000 7f626c7c2640 20 client.4621.objecter _get_session s=0x7f62641f5000 osd=7 3 2024-03-27T21:40:33.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.617+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_remove -1 16 2024-03-27T21:40:33.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.617+0000 7f626c7c2640 20 client.4621.objecter get_session s=0x7f62641f5000 osd=7 3 2024-03-27T21:40:33.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.617+0000 7f626c7c2640 15 client.4621.objecter _session_command_op_assign 7 16 2024-03-27T21:40:33.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.617+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f62641f5000 osd=7 4 2024-03-27T21:40:33.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.617+0000 7f626c7c2640 10 client.4621.objecter _send_command 16 2024-03-27T21:40:33.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.617+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f62641f2690 con 0x7f62641f4330 2024-03-27T21:40:33.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.618+0000 7f625affd640 1 -- 172.21.15.60:0/3101508738 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f62641f2690 con 0x7f62641f4330 2024-03-27T21:40:33.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.618+0000 7f625affd640 10 client.4621.objecter ms_dispatch 0x7f6264000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:33.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.618+0000 7f625affd640 10 client.4621.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:33.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.618+0000 7f625affd640 20 client.4621.objecter put_session s=0x7f62641f5000 osd=7 4 2024-03-27T21:40:33.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.618+0000 7f625affd640 15 client.4621.objecter _session_command_op_remove 7 16 2024-03-27T21:40:33.605 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 10 librados: watch_flush enter 2024-03-27T21:40:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 10 librados: watch_flush exit 2024-03-27T21:40:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 10 client.4621.objecter close_session for osd.0 2024-03-27T21:40:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f6264138ee0 msgr2=0x7f62641392c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f6264138ee0 0x7f62641392c0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f626413f0c0 osd=0 1 2024-03-27T21:40:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 10 client.4621.objecter close_session for osd.1 2024-03-27T21:40:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f62641d9e70 msgr2=0x7f62641da250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f62641d9e70 0x7f62641da250 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f62640a9260 osd=1 1 2024-03-27T21:40:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 10 client.4621.objecter close_session for osd.2 2024-03-27T21:40:33.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f62641419c0 msgr2=0x7f626402b170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:33.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f62641419c0 0x7f626402b170 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:33.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f6264141850 osd=2 1 2024-03-27T21:40:33.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 10 client.4621.objecter close_session for osd.3 2024-03-27T21:40:33.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f626413d2d0 msgr2=0x7f62641e8cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:33.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f626413d2d0 0x7f62641e8cd0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:33.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f626413d160 osd=3 1 2024-03-27T21:40:33.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 10 client.4621.objecter close_session for osd.4 2024-03-27T21:40:33.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f6264065fb0 msgr2=0x7f62641ec3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:33.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f6264065fb0 0x7f62641ec3a0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f6264065e40 osd=4 1 2024-03-27T21:40:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 10 client.4621.objecter close_session for osd.5 2024-03-27T21:40:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f62641ecdf0 msgr2=0x7f62641ef1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f62641ecdf0 0x7f62641ef1e0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f626402c380 osd=5 1 2024-03-27T21:40:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 10 client.4621.objecter close_session for osd.6 2024-03-27T21:40:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f62641efc30 msgr2=0x7f62641f2040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f62641efc30 0x7f62641f2040 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f626402c5d0 osd=6 1 2024-03-27T21:40:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 10 client.4621.objecter close_session for osd.7 2024-03-27T21:40:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f62641f4330 msgr2=0x7f62641f47b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f62641f4330 0x7f62641f47b0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 20 client.4621.objecter put_session s=0x7f62641f5000 osd=7 1 2024-03-27T21:40:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 20 client.4621.objecter shutdown clearing up homeless session... 2024-03-27T21:40:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 10 client.4621.objecter successfully canceled tick 2024-03-27T21:40:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626ad5f640 1 -- 172.21.15.60:0/3101508738 reap_dead start 2024-03-27T21:40:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626ad5f640 1 -- 172.21.15.60:0/3101508738 reap_dead start 2024-03-27T21:40:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626ad5f640 1 -- 172.21.15.60:0/3101508738 reap_dead start 2024-03-27T21:40:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626ad5f640 1 -- 172.21.15.60:0/3101508738 reap_dead start 2024-03-27T21:40:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626ad5f640 1 -- 172.21.15.60:0/3101508738 reap_dead start 2024-03-27T21:40:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f624806c590 msgr2=0x7f624806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f624806c590 0x7f624806ea50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6260036000 tx=0x7f6260030000 comp rx=0 tx=0).stop 2024-03-27T21:40:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 10 monclient: shutdown 2024-03-27T21:40:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f626413e810 msgr2=0x7f62641d6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f626413e810 0x7f62641d6f60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f625400c4c0 tx=0x7f625400c980 comp rx=0 tx=0).stop 2024-03-27T21:40:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 shutdown_connections 2024-03-27T21:40:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f624806c590 0x7f624806ea50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:33.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 --2- 172.21.15.60:0/3101508738 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f626413e810 0x7f62641d6f60 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:33.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 >> 172.21.15.60:0/3101508738 conn(0x7f62640b7300 msgr2=0x7f62640a8520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:33.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 shutdown_connections 2024-03-27T21:40:33.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 -- 172.21.15.60:0/3101508738 wait complete. 2024-03-27T21:40:33.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:33.619+0000 7f626c7c2640 1 librados: shutdown 2024-03-27T21:40:33.612 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:33.623+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:33.671 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:33.672 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:33.672 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:33.672 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:33.681 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:40:33.703 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:33.717+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:33.804 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:33.818+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:33.825 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:33.825 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:33.826 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:33.826 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:33.826 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:33.834 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:40:33.905 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:33.918+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:33.939 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:33.939 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:33.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:33.939 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:33.939 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:33.948 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:40:34.005 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:34.019+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:34.092 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:34.092 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:34.092 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:34.093 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:34.101 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:40:34.106 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:34.120+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:34.205 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:34.206 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:34.206 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:34.206 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:34.206 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:34.207 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:34.221+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:34.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:40:34.307 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:34.321+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:34.319 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:34.319 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:34.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:34.319 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:34.319 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:34.327 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:40:34.408 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:34.422+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:34.433 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:34.434 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:34.434 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:34.434 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:34.442 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:40:34.445 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-27T21:40:34.509 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:34.523+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:34.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.560+0000 7fe6bbf99640 10 monclient: get_monmap_and_config 2024-03-27T21:40:34.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.560+0000 7fe6bbf99640 10 monclient: build_initial_monmap 2024-03-27T21:40:34.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.560+0000 7fe6bbf99640 10 monclient: monmap: 2024-03-27T21:40:34.547 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:34.547 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:34.547 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:34.561750+0000 2024-03-27T21:40:34.547 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:34.561750+0000 2024-03-27T21:40:34.547 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:34.547 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:34.548 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:34.548 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:34.548 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:34.548 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:34.548 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:34.548 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:34.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.561+0000 7fe6bbf99640 1 Processor -- start 2024-03-27T21:40:34.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.561+0000 7fe6bbf99640 1 -- start start 2024-03-27T21:40:34.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.561+0000 7fe6bbf99640 10 monclient: init 2024-03-27T21:40:34.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6bbf99640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:34.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6bbf99640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-03-27T21:40:34.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6bbf99640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe6b4071a50 0x7fe6b40a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:34.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6bbf99640 10 monclient(hunting): picked mon.noname-a con 0x7fe6b4071a50 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:34.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6bbf99640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe6b413e810 0x7fe6b413ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:34.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6bbf99640 10 monclient(hunting): picked mon.noname-c con 0x7fe6b413e810 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:34.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6bbf99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe6b4141630 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:34.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6bbf99640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b413f190 con 0x7fe6b4141630 2024-03-27T21:40:34.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6bbf99640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:34.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6bbf99640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6b4071e30 con 0x7fe6b4071a50 2024-03-27T21:40:34.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6bbf99640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:34.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6bbf99640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6b40a8e90 con 0x7fe6b413e810 2024-03-27T21:40:34.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6bbf99640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:34.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6bbf99640 10 monclient(hunting): authenticate will time out at 875.783181s 2024-03-27T21:40:34.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9d35640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe6b4071a50 0x7fe6b40a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:34.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6ba536640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe6b4141630 0x7fe6b4141aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59988/0 (socket says 172.21.15.60:59988) 2024-03-27T21:40:34.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6ba536640 1 -- 172.21.15.60:0/996586767 learned_addr learned my addr 172.21.15.60:0/996586767 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:40:34.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9d35640 10 monclient(hunting): get_auth_request con 0x7fe6b4071a50 auth_method 0 2024-03-27T21:40:34.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9d35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:34.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9d35640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:34.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9d35640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:34.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9534640 1 --2- 172.21.15.60:0/996586767 >> v2:172.21.15.191:3300/0 conn(0x7fe6b413e810 0x7fe6b413ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:34.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9534640 10 monclient(hunting): get_auth_request con 0x7fe6b413e810 auth_method 0 2024-03-27T21:40:34.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9534640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:34.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9534640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:34.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9534640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:34.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9d35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:34.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9d35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:34.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9d35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:34.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9534640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:34.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9534640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:34.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9534640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:34.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9d35640 10 monclient(hunting): handle_auth_done global_id 4680 payload 274 2024-03-27T21:40:34.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9d35640 1 -- 172.21.15.60:0/996586767 >> v2:172.21.15.191:3300/0 conn(0x7fe6b413e810 msgr2=0x7fe6b413ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:34.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9d35640 1 --2- 172.21.15.60:0/996586767 >> v2:172.21.15.191:3300/0 conn(0x7fe6b413e810 0x7fe6b413ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:34.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9d35640 1 -- 172.21.15.60:0/996586767 >> v1:172.21.15.60:6789/0 conn(0x7fe6b4141630 legacy=0x7fe6b4141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:34.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b8d33640 1 -- 172.21.15.60:0/996586767 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296363408 0 0) 0x7fe6ac001490 con 0x7fe6b4141630 2024-03-27T21:40:34.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9d35640 10 monclient: _finish_hunting 0 2024-03-27T21:40:34.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9d35640 1 monclient: found mon.noname-a 2024-03-27T21:40:34.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9d35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:34.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9d35640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:34.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b9d35640 1 -- 172.21.15.60:0/996586767 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b40aa040 con 0x7fe6b4071a50 2024-03-27T21:40:34.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.562+0000 7fe6b8d33640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:40:34.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.563+0000 7fe6b9d35640 1 --2- 172.21.15.60:0/996586767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6b4071a50 0x7fe6b40a87e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe6b40709f0 tx=0x7fe6b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c88c52dfc91e2b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:34.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.563+0000 7fe6b8d33640 1 -- 172.21.15.60:0/996586767 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0031830 con 0x7fe6b4071a50 2024-03-27T21:40:34.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.563+0000 7fe6b8d33640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:34.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.563+0000 7fe6b8d33640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:34.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.563+0000 7fe6b8d33640 10 monclient: dump: 2024-03-27T21:40:34.553 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:34.553 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:34.553 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:34.553 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:34.553 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:34.554 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:34.554 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:34.554 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:34.554 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:34.554 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:34.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.563+0000 7fe6b8d33640 10 monclient: _finish_auth 0 2024-03-27T21:40:34.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.563+0000 7fe6b8d33640 10 monclient: _check_auth_tickets 2024-03-27T21:40:34.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.563+0000 7fe6b8d33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:34.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.563+0000 7fe6bbf99640 5 monclient: authenticate success, global_id 4680 2024-03-27T21:40:34.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.563+0000 7fe6bbf99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:34.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.563+0000 7fe6b8d33640 1 -- 172.21.15.60:0/996586767 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b0031990 con 0x7fe6b4071a50 2024-03-27T21:40:34.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.563+0000 7fe6b8d33640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:34.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.563+0000 7fe6b8d33640 1 -- 172.21.15.60:0/996586767 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0031d20 con 0x7fe6b4071a50 2024-03-27T21:40:34.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.563+0000 7fe6bbf99640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:34.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.563+0000 7fe6bbf99640 4 set_mon_vals no callback set 2024-03-27T21:40:34.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.563+0000 7fe6b8d33640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:34.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.563+0000 7fe6b8d33640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:34.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.563+0000 7fe6b8d33640 10 monclient: dump: 2024-03-27T21:40:34.555 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:34.555 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:34.555 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:34.555 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:34.555 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:34.555 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:34.555 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:34.555 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:34.555 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:34.555 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:34.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.564+0000 7fe6bbf99640 10 monclient: shutdown 2024-03-27T21:40:34.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.564+0000 7fe6bbf99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:34.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.564+0000 7fe6bbf99640 1 -- 172.21.15.60:0/996586767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6b4071a50 msgr2=0x7fe6b40a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:34.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.564+0000 7fe6bbf99640 1 --2- 172.21.15.60:0/996586767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6b4071a50 0x7fe6b40a87e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe6b40709f0 tx=0x7fe6b002b8d0 comp rx=0 tx=0).stop 2024-03-27T21:40:34.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.564+0000 7fe6bbf99640 1 -- 172.21.15.60:0/996586767 shutdown_connections 2024-03-27T21:40:34.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.564+0000 7fe6bbf99640 1 --2- 172.21.15.60:0/996586767 >> v2:172.21.15.191:3300/0 conn(0x7fe6b413e810 0x7fe6b413ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:34.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.564+0000 7fe6bbf99640 1 --2- 172.21.15.60:0/996586767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6b4071a50 0x7fe6b40a87e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:34.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.564+0000 7fe6bbf99640 1 -- 172.21.15.60:0/996586767 >> 172.21.15.60:0/996586767 conn(0x7fe6b40b7300 msgr2=0x7fe6b40b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:34.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.564+0000 7fe6bbf99640 1 -- 172.21.15.60:0/996586767 shutdown_connections 2024-03-27T21:40:34.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.565+0000 7fe6bbf99640 1 -- 172.21.15.60:0/996586767 wait complete. 2024-03-27T21:40:34.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.565+0000 7fe6bbf99640 10 monclient: build_initial_monmap 2024-03-27T21:40:34.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.565+0000 7fe6bbf99640 10 monclient: monmap: 2024-03-27T21:40:34.562 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:34.562 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:34.562 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:34.562 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:34.563 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:34.563 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:34.563 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:34.563 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:34.563 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:34.563 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:34.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.566+0000 7fe6bbf99640 1 librados: starting msgr at 2024-03-27T21:40:34.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.566+0000 7fe6bbf99640 1 librados: starting objecter 2024-03-27T21:40:34.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.566+0000 7fe6bbf99640 1 Processor -- start 2024-03-27T21:40:34.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.566+0000 7fe6bbf99640 1 -- start start 2024-03-27T21:40:34.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.566+0000 7fe6bbf99640 1 librados: setting wanted keys 2024-03-27T21:40:34.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.566+0000 7fe6bbf99640 1 librados: calling monclient init 2024-03-27T21:40:34.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.566+0000 7fe6bbf99640 10 monclient: init 2024-03-27T21:40:34.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6bbf99640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:34.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6bbf99640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:40:34.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6bbf99640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6b4071a50 0x7fe6b41d6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:34.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6bbf99640 10 monclient(hunting): picked mon.noname-b con 0x7fe6b4071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:34.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6bbf99640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe6b413e810 0x7fe6b4133050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:34.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6b9534640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe6b413e810 0x7fe6b4133050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:34.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6b9d35640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6b4071a50 0x7fe6b41d6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:34.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6b9534640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe6b413e810 0x7fe6b4133050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:59460/0 (socket says 172.21.15.60:59460) 2024-03-27T21:40:34.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6b9534640 1 -- 172.21.15.60:0/2998563414 learned_addr learned my addr 172.21.15.60:0/2998563414 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:34.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6bbf99640 10 monclient(hunting): picked mon.noname-c con 0x7fe6b413e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:34.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6bbf99640 1 --2- 172.21.15.60:0/2998563414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6b4141630 0x7fe6b4133590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:34.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6bbf99640 10 monclient(hunting): picked mon.noname-a con 0x7fe6b4141630 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:34.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6bbf99640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:34.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6bbf99640 1 -- 172.21.15.60:0/2998563414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b414bc60 con 0x7fe6b4141630 2024-03-27T21:40:34.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6bbf99640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:34.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6bbf99640 1 -- 172.21.15.60:0/2998563414 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b41374d0 con 0x7fe6b4071a50 2024-03-27T21:40:34.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6bbf99640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:34.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6bbf99640 1 -- 172.21.15.60:0/2998563414 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b4133ad0 con 0x7fe6b413e810 2024-03-27T21:40:34.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6bbf99640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:34.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6bbf99640 10 monclient(hunting): authenticate will time out at 875.788641s 2024-03-27T21:40:34.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6b9534640 10 monclient(hunting): get_auth_request con 0x7fe6b413e810 auth_method 0 2024-03-27T21:40:34.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6b9534640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:34.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6b9534640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:34.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6b9534640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:34.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6ba536640 1 --2- 172.21.15.60:0/2998563414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6b4141630 0x7fe6b4133590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:34.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6b9534640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:34.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6b9534640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:34.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6b9534640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:34.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6ba536640 10 monclient(hunting): get_auth_request con 0x7fe6b4141630 auth_method 0 2024-03-27T21:40:34.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6ba536640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:34.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6ba536640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:34.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6ba536640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:34.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6b9d35640 10 monclient(hunting): get_auth_request con 0x7fe6b4071a50 auth_method 0 2024-03-27T21:40:34.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6b9d35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:34.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6b9d35640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:34.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6b9d35640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:34.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6b9d35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:34.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.567+0000 7fe6b9d35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:34.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6b9d35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:34.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6ba536640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:34.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6ba536640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:34.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6ba536640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:34.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6b9534640 10 monclient(hunting): handle_auth_done global_id 4547 payload 824 2024-03-27T21:40:34.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6b9534640 1 -- 172.21.15.60:0/2998563414 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6b4071a50 msgr2=0x7fe6b41d6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:34.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6b9534640 1 --2- 172.21.15.60:0/2998563414 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6b4071a50 0x7fe6b41d6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:34.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6b9534640 1 -- 172.21.15.60:0/2998563414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6b4141630 msgr2=0x7fe6b4133590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:34.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6b9534640 1 --2- 172.21.15.60:0/2998563414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6b4141630 0x7fe6b4133590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:34.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6b9534640 10 monclient: _finish_hunting 0 2024-03-27T21:40:34.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6b9534640 1 monclient: found mon.noname-c 2024-03-27T21:40:34.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6b9534640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:34.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6b9534640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:40:34.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6b9534640 1 -- 172.21.15.60:0/2998563414 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b0009400 con 0x7fe6b413e810 2024-03-27T21:40:34.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6b9d35640 1 --2- 172.21.15.60:0/2998563414 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6b4071a50 0x7fe6b41d6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:34.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6b9534640 1 --2- 172.21.15.60:0/2998563414 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe6b413e810 0x7fe6b4133050 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe6a400e9e0 tx=0x7fe6a400eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:34.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6aa7fc640 10 client.?.objecter ms_handle_connect 0x7fe6b413e810 2024-03-27T21:40:34.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6aa7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:34.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6aa7fc640 1 -- 172.21.15.60:0/2998563414 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4009670 con 0x7fe6b413e810 2024-03-27T21:40:34.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:34.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6aa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:34.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6aa7fc640 10 monclient: dump: 2024-03-27T21:40:34.568 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:34.568 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:34.568 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:34.568 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:34.568 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:34.568 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:34.568 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:34.568 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:34.568 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:34.568 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:34.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6aa7fc640 10 monclient: _finish_auth 0 2024-03-27T21:40:34.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6aa7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:40:34.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:34.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6aa7fc640 1 -- 172.21.15.60:0/2998563414 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a400c9d0 con 0x7fe6b413e810 2024-03-27T21:40:34.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:34.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6aa7fc640 1 -- 172.21.15.60:0/2998563414 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a400cd80 con 0x7fe6b413e810 2024-03-27T21:40:34.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:34.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:34.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6aa7fc640 10 monclient: dump: 2024-03-27T21:40:34.569 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:34.569 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:34.569 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:34.569 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:34.569 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:34.569 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:34.569 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:34.569 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:34.569 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:34.569 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:34.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6bbf99640 5 monclient: authenticate success, global_id 4547 2024-03-27T21:40:34.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6abfff640 4 set_mon_vals no callback set 2024-03-27T21:40:34.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6bbf99640 10 monclient: _renew_subs 2024-03-27T21:40:34.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6bbf99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:34.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6bbf99640 1 -- 172.21.15.60:0/2998563414 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b413a240 con 0x7fe6b413e810 2024-03-27T21:40:34.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6bbf99640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:34.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6bbf99640 10 monclient: _renew_subs 2024-03-27T21:40:34.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6bbf99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:34.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6bbf99640 1 -- 172.21.15.60:0/2998563414 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b4139f90 con 0x7fe6b413e810 2024-03-27T21:40:34.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.568+0000 7fe6bbf99640 1 librados: init done 2024-03-27T21:40:34.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.570+0000 7fe6bbf99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:34.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.570+0000 7fe6bbf99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:34.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.570+0000 7fe6bbf99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:34.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.570+0000 7fe6bbf99640 1 -- 172.21.15.60:0/2998563414 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b413a3d0 con 0x7fe6b413e810 2024-03-27T21:40:34.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.570+0000 7fe6aa7fc640 1 -- 172.21.15.60:0/2998563414 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe6a4013470 con 0x7fe6b413e810 2024-03-27T21:40:34.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.571+0000 7fe6aa7fc640 1 --2- 172.21.15.60:0/2998563414 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe69806c590 0x7fe69806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:34.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.571+0000 7fe6aa7fc640 1 -- 172.21.15.60:0/2998563414 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fe6a40892c0 con 0x7fe6b413e810 2024-03-27T21:40:34.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.571+0000 7fe6aa7fc640 10 client.4547.objecter ms_dispatch 0x7fe6b4000b90 osd_map(39..39 src has 1..39) v4 2024-03-27T21:40:34.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.571+0000 7fe6aa7fc640 3 client.4547.objecter handle_osd_map got epochs [39,39] > 0 2024-03-27T21:40:34.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.571+0000 7fe6aa7fc640 3 client.4547.objecter handle_osd_map decoding full epoch 39 2024-03-27T21:40:34.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.571+0000 7fe6b9d35640 1 --2- 172.21.15.60:0/2998563414 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe69806c590 0x7fe69806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:34.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.571+0000 7fe6aa7fc640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-27T21:40:34.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.571+0000 7fe6b9d35640 10 monclient: get_auth_request con 0x7fe69806c590 auth_method 0 2024-03-27T21:40:34.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.572+0000 7fe6b9d35640 1 --2- 172.21.15.60:0/2998563414 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe69806c590 0x7fe69806ea50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0036000 tx=0x7fe6b003e040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:34.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.572+0000 7fe6aa7fc640 10 client.4547.objecter ms_handle_connect 0x7fe69806c590 2024-03-27T21:40:34.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.579+0000 7fe6aa7fc640 1 -- 172.21.15.60:0/2998563414 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe6a4054710 con 0x7fe6b413e810 2024-03-27T21:40:34.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.579+0000 7fe6aa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:34.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.579+0000 7fe6aa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:34.588 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:34.588 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:34.588 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:34.588 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:34.588 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:34.596 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:40:34.641 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:34.655+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:34.703 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:34.703 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:34.703 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:34.703 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-03-27T21:40:34.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.704 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:34.704 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:34.704 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 55.283803826000003, 2024-03-27T21:40:34.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43537978700000002, 2024-03-27T21:40:34.704 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:34.704 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:34.704 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:34.704 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:34.704 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:34.704 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-03-27T21:40:34.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.704 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:34.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:34.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:34.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.705 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:34.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:34.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.705 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:34.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432019+0000", 2024-03-27T21:40:34.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-03-27T21:40:34.705 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:34.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432027+0000", 2024-03-27T21:40:34.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1200000000000002e-06 2024-03-27T21:40:34.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:34.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432039+0000", 2024-03-27T21:40:34.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1123e-05 2024-03-27T21:40:34.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:34.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432051+0000", 2024-03-27T21:40:34.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-03-27T21:40:34.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.707 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:34.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432157+0000", 2024-03-27T21:40:34.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106846 2024-03-27T21:40:34.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.707 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-03-27T21:40:34.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432176+0000", 2024-03-27T21:40:34.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8490000000000001e-05 2024-03-27T21:40:34.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.707 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:34.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866170+0000", 2024-03-27T21:40:34.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43399370100000001 2024-03-27T21:40:34.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.708 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.708 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:34.708 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866335+0000", 2024-03-27T21:40:34.708 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016559699999999999 2024-03-27T21:40:34.708 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.708 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.708 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:34.708 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866495+0000", 2024-03-27T21:40:34.708 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015944200000000001 2024-03-27T21:40:34.708 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.708 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.708 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:34.708 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867012+0000", 2024-03-27T21:40:34.708 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0005176 2024-03-27T21:40:34.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:34.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867334+0000", 2024-03-27T21:40:34.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032158600000000001 2024-03-27T21:40:34.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:34.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867374+0000", 2024-03-27T21:40:34.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0547000000000003e-05 2024-03-27T21:40:34.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:34.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867400+0000", 2024-03-27T21:40:34.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5958000000000001e-05 2024-03-27T21:40:34.710 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.710 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:34.710 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.710 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.710 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:34.710 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:34.710 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.786421375000003, 2024-03-27T21:40:34.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064560310000000001, 2024-03-27T21:40:34.710 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:34.710 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:34.710 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:34.710 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:34.710 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:34.711 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 70 2024-03-27T21:40:34.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.711 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:34.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:34.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:34.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:34.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:34.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:34.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929402+0000", 2024-03-27T21:40:34.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:34.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:34.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929407+0000", 2024-03-27T21:40:34.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-06 2024-03-27T21:40:34.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:34.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929414+0000", 2024-03-27T21:40:34.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5689999999999999e-06 2024-03-27T21:40:34.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:34.713 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929419+0000", 2024-03-27T21:40:34.713 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-03-27T21:40:34.713 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.713 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:34.713 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930245+0000", 2024-03-27T21:40:34.713 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082521500000000002 2024-03-27T21:40:34.713 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.713 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:34.713 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930287+0000", 2024-03-27T21:40:34.713 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2305000000000002e-05 2024-03-27T21:40:34.713 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.713 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:34.714 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930367+0000", 2024-03-27T21:40:34.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9589e-05 2024-03-27T21:40:34.714 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.714 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.714 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:34.714 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.931214+0000", 2024-03-27T21:40:34.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000847827 2024-03-27T21:40:34.714 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.714 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.714 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:34.714 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935807+0000", 2024-03-27T21:40:34.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045923780000000003 2024-03-27T21:40:34.714 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.714 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.714 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:34.715 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935840+0000", 2024-03-27T21:40:34.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3501999999999999e-05 2024-03-27T21:40:34.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.715 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.715 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:34.715 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935859+0000", 2024-03-27T21:40:34.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8845e-05 2024-03-27T21:40:34.715 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.715 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:34.715 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.715 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.715 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:34.715 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:34.715 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.779161784000003, 2024-03-27T21:40:34.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071695429999999996, 2024-03-27T21:40:34.716 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:34.716 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:34.716 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:34.716 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:34.716 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:34.716 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-03-27T21:40:34.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.716 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:34.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:34.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:34.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:34.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:34.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:34.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936662+0000", 2024-03-27T21:40:34.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:34.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:34.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936737+0000", 2024-03-27T21:40:34.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5303999999999996e-05 2024-03-27T21:40:34.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:34.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936746+0000", 2024-03-27T21:40:34.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6340000000000007e-06 2024-03-27T21:40:34.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:34.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936753+0000", 2024-03-27T21:40:34.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-03-27T21:40:34.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:34.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937926+0000", 2024-03-27T21:40:34.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011731280000000001 2024-03-27T21:40:34.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:34.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937969+0000", 2024-03-27T21:40:34.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3285999999999999e-05 2024-03-27T21:40:34.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:34.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938040+0000", 2024-03-27T21:40:34.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0358000000000006e-05 2024-03-27T21:40:34.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:34.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939324+0000", 2024-03-27T21:40:34.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012844899999999999 2024-03-27T21:40:34.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.720 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.720 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:34.720 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943780+0000", 2024-03-27T21:40:34.720 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044555630000000001 2024-03-27T21:40:34.720 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.720 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.720 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:34.720 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943811+0000", 2024-03-27T21:40:34.720 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1415999999999999e-05 2024-03-27T21:40:34.720 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.720 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.720 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:34.720 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943832+0000", 2024-03-27T21:40:34.720 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0981000000000001e-05 2024-03-27T21:40:34.720 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.721 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:34.721 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.721 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.721 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.721 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:34.721 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:34.721 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.779063768, 2024-03-27T21:40:34.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071139740000000003, 2024-03-27T21:40:34.721 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:34.721 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:34.721 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:34.721 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:34.721 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:34.721 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-03-27T21:40:34.721 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.722 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:34.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.722 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:34.722 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:34.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.722 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.722 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:34.722 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:34.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.722 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.722 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:34.722 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936760+0000", 2024-03-27T21:40:34.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:34.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:34.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936765+0000", 2024-03-27T21:40:34.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6330000000000004e-06 2024-03-27T21:40:34.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:34.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936771+0000", 2024-03-27T21:40:34.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0560000000000001e-06 2024-03-27T21:40:34.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:34.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936776+0000", 2024-03-27T21:40:34.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1510000000000003e-06 2024-03-27T21:40:34.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:34.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938605+0000", 2024-03-27T21:40:34.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001829589 2024-03-27T21:40:34.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:34.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938657+0000", 2024-03-27T21:40:34.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1860000000000002e-05 2024-03-27T21:40:34.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:34.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938734+0000", 2024-03-27T21:40:34.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6639999999999998e-05 2024-03-27T21:40:34.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:34.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939562+0000", 2024-03-27T21:40:34.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082803100000000001 2024-03-27T21:40:34.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:34.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943838+0000", 2024-03-27T21:40:34.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042757050000000003 2024-03-27T21:40:34.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:34.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943859+0000", 2024-03-27T21:40:34.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1175e-05 2024-03-27T21:40:34.726 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.726 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.726 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:34.726 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943875+0000", 2024-03-27T21:40:34.726 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5744e-05 2024-03-27T21:40:34.726 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.726 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:34.726 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.726 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.726 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.726 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:34.726 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:34.726 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.635549717000004, 2024-03-27T21:40:34.726 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059712230000000003, 2024-03-27T21:40:34.727 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:34.727 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:34.727 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:34.727 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:34.727 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:34.727 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 112 2024-03-27T21:40:34.727 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.727 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:34.727 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.727 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:34.727 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:34.727 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.727 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.727 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.727 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:34.728 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:34.728 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.728 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.728 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.728 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:34.728 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080274+0000", 2024-03-27T21:40:34.728 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:34.728 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.728 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.728 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:34.728 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080318+0000", 2024-03-27T21:40:34.728 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3634999999999998e-05 2024-03-27T21:40:34.728 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.728 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.728 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:34.728 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080329+0000", 2024-03-27T21:40:34.729 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1232e-05 2024-03-27T21:40:34.729 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.729 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.729 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:34.729 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080333+0000", 2024-03-27T21:40:34.729 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6009999999999999e-06 2024-03-27T21:40:34.729 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.729 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.729 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:34.729 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083786+0000", 2024-03-27T21:40:34.729 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034528940000000002 2024-03-27T21:40:34.729 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.729 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.729 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:34.730 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083861+0000", 2024-03-27T21:40:34.730 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4893000000000003e-05 2024-03-27T21:40:34.730 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.730 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.730 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:34.730 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083981+0000", 2024-03-27T21:40:34.730 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012058299999999999 2024-03-27T21:40:34.730 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.730 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.730 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:34.730 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086148+0000", 2024-03-27T21:40:34.730 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021671469999999999 2024-03-27T21:40:34.730 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.730 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.731 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:34.731 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086195+0000", 2024-03-27T21:40:34.731 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6768999999999999e-05 2024-03-27T21:40:34.731 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.731 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.731 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:34.731 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086224+0000", 2024-03-27T21:40:34.731 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8758000000000001e-05 2024-03-27T21:40:34.731 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.731 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.731 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:34.731 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086246+0000", 2024-03-27T21:40:34.731 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2064000000000001e-05 2024-03-27T21:40:34.731 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.731 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:34.731 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.732 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.732 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.732 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:34.732 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:34.732 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.63548677, 2024-03-27T21:40:34.732 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066196320000000003, 2024-03-27T21:40:34.732 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:34.732 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:34.732 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:34.732 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:34.732 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:34.732 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113 2024-03-27T21:40:34.732 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.732 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:34.732 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.733 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:34.733 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:34.733 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.733 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.733 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.733 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:34.733 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:34.733 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.733 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.733 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.733 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:34.733 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080337+0000", 2024-03-27T21:40:34.733 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:34.733 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.733 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.734 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:34.734 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080340+0000", 2024-03-27T21:40:34.734 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6690000000000002e-06 2024-03-27T21:40:34.734 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.734 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:34.734 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080344+0000", 2024-03-27T21:40:34.734 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6550000000000002e-06 2024-03-27T21:40:34.734 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.734 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:34.734 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080348+0000", 2024-03-27T21:40:34.734 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9169999999999999e-06 2024-03-27T21:40:34.734 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.735 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:34.735 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084595+0000", 2024-03-27T21:40:34.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042471719999999996 2024-03-27T21:40:34.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.735 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.735 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:34.735 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084676+0000", 2024-03-27T21:40:34.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1147000000000001e-05 2024-03-27T21:40:34.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.735 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.735 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:34.735 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084796+0000", 2024-03-27T21:40:34.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012032900000000001 2024-03-27T21:40:34.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.735 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:34.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086190+0000", 2024-03-27T21:40:34.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013934399999999999 2024-03-27T21:40:34.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:34.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086905+0000", 2024-03-27T21:40:34.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071536199999999999 2024-03-27T21:40:34.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:34.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086937+0000", 2024-03-27T21:40:34.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2125000000000002e-05 2024-03-27T21:40:34.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.737 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:34.737 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086957+0000", 2024-03-27T21:40:34.737 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0157999999999999e-05 2024-03-27T21:40:34.737 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.737 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:34.737 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.737 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.737 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.737 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:34.737 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:34.737 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.635474053999999, 2024-03-27T21:40:34.737 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007405661, 2024-03-27T21:40:34.737 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:34.737 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:34.737 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:34.738 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:34.738 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:34.738 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 114 2024-03-27T21:40:34.738 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.738 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:34.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.738 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:34.738 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:34.738 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.738 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.738 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:34.738 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:34.738 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.738 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.739 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.739 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:34.739 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:34.739 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-03-27T21:40:34.739 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.739 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.739 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:34.739 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080353+0000", 2024-03-27T21:40:34.739 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.441e-06 2024-03-27T21:40:34.739 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.739 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.739 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:34.739 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080356+0000", 2024-03-27T21:40:34.739 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0769999999999999e-06 2024-03-27T21:40:34.739 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.739 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:34.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080358+0000", 2024-03-27T21:40:34.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3369999999999998e-06 2024-03-27T21:40:34.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:34.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085118+0000", 2024-03-27T21:40:34.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047603949999999997 2024-03-27T21:40:34.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:34.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085210+0000", 2024-03-27T21:40:34.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1983000000000003e-05 2024-03-27T21:40:34.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:34.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085332+0000", 2024-03-27T21:40:34.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121803 2024-03-27T21:40:34.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:34.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086292+0000", 2024-03-27T21:40:34.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095961400000000002 2024-03-27T21:40:34.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:34.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087710+0000", 2024-03-27T21:40:34.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014181459999999999 2024-03-27T21:40:34.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.742 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:34.742 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087736+0000", 2024-03-27T21:40:34.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6475e-05 2024-03-27T21:40:34.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.742 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:34.742 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087756+0000", 2024-03-27T21:40:34.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9626000000000001e-05 2024-03-27T21:40:34.742 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.742 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:34.742 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.742 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:34.742 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:34.743 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.635451619000001, 2024-03-27T21:40:34.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075011720000000004, 2024-03-27T21:40:34.743 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:34.743 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:34.743 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:34.743 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:34.743 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:34.743 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115 2024-03-27T21:40:34.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.743 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:34.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:34.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:34.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.744 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:34.744 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:34.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.744 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:34.744 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:34.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:34.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.744 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:34.744 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080458+0000", 2024-03-27T21:40:34.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5228000000000005e-05 2024-03-27T21:40:34.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.745 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:34.745 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080465+0000", 2024-03-27T21:40:34.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.238e-06 2024-03-27T21:40:34.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.745 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:34.745 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080468+0000", 2024-03-27T21:40:34.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5109999999999999e-06 2024-03-27T21:40:34.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.745 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:34.745 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086326+0000", 2024-03-27T21:40:34.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058579169999999998 2024-03-27T21:40:34.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:34.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086382+0000", 2024-03-27T21:40:34.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5309999999999997e-05 2024-03-27T21:40:34.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:34.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086457+0000", 2024-03-27T21:40:34.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5413000000000005e-05 2024-03-27T21:40:34.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:34.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087762+0000", 2024-03-27T21:40:34.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013052300000000001 2024-03-27T21:40:34.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:34.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087798+0000", 2024-03-27T21:40:34.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5614999999999997e-05 2024-03-27T21:40:34.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:34.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087840+0000", 2024-03-27T21:40:34.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2196000000000001e-05 2024-03-27T21:40:34.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:34.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087874+0000", 2024-03-27T21:40:34.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3785999999999998e-05 2024-03-27T21:40:34.748 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.748 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:34.748 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.748 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.748 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.748 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:34.748 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:34.748 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.635351536000002, 2024-03-27T21:40:34.748 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075050680000000002, 2024-03-27T21:40:34.748 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:34.748 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:34.748 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:34.748 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:34.748 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:34.748 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 116 2024-03-27T21:40:34.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.749 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:34.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.749 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:34.749 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:34.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.749 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:34.749 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:34.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.749 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:34.749 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080472+0000", 2024-03-27T21:40:34.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:34.750 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:34.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080475+0000", 2024-03-27T21:40:34.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.712e-06 2024-03-27T21:40:34.750 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:34.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080478+0000", 2024-03-27T21:40:34.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1070000000000001e-06 2024-03-27T21:40:34.750 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:34.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080481+0000", 2024-03-27T21:40:34.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4269999999999998e-06 2024-03-27T21:40:34.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:34.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086992+0000", 2024-03-27T21:40:34.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065114099999999996 2024-03-27T21:40:34.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:34.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087064+0000", 2024-03-27T21:40:34.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1927000000000004e-05 2024-03-27T21:40:34.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:34.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087188+0000", 2024-03-27T21:40:34.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123585 2024-03-27T21:40:34.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:34.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087766+0000", 2024-03-27T21:40:34.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057814300000000004 2024-03-27T21:40:34.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:34.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087934+0000", 2024-03-27T21:40:34.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000167894 2024-03-27T21:40:34.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:34.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087958+0000", 2024-03-27T21:40:34.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4663000000000001e-05 2024-03-27T21:40:34.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:34.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087978+0000", 2024-03-27T21:40:34.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9636e-05 2024-03-27T21:40:34.753 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.753 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:34.753 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.753 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:34.753 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:34.753 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.635319414000001, 2024-03-27T21:40:34.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076850679999999998, 2024-03-27T21:40:34.753 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:34.754 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:34.754 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:34.754 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:34.754 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:34.754 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 117 2024-03-27T21:40:34.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.754 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:34.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:34.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:34.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:34.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:34.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:34.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:34.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:34.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:34.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080508+0000", 2024-03-27T21:40:34.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.298e-06 2024-03-27T21:40:34.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:34.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080515+0000", 2024-03-27T21:40:34.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0790000000000003e-06 2024-03-27T21:40:34.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.756 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:34.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080518+0000", 2024-03-27T21:40:34.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8509999999999999e-06 2024-03-27T21:40:34.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.756 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:34.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087444+0000", 2024-03-27T21:40:34.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0069259120000000002 2024-03-27T21:40:34.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:34.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087491+0000", 2024-03-27T21:40:34.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7043999999999999e-05 2024-03-27T21:40:34.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:34.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087564+0000", 2024-03-27T21:40:34.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3603999999999995e-05 2024-03-27T21:40:34.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:34.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087902+0000", 2024-03-27T21:40:34.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033734100000000001 2024-03-27T21:40:34.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:34.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088117+0000", 2024-03-27T21:40:34.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000215164 2024-03-27T21:40:34.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:34.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088160+0000", 2024-03-27T21:40:34.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2933999999999999e-05 2024-03-27T21:40:34.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:34.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088190+0000", 2024-03-27T21:40:34.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0139999999999999e-05 2024-03-27T21:40:34.758 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.759 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:34.759 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.759 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:34.759 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:34.759 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.219039025000001, 2024-03-27T21:40:34.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052465439999999997, 2024-03-27T21:40:34.759 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:34.759 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:34.759 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:34.759 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:34.759 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:34.759 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 133 2024-03-27T21:40:34.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.760 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:34.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:34.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:34.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:34.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:34.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:34.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:34.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:34.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:34.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496788+0000", 2024-03-27T21:40:34.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-03-27T21:40:34.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:34.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496793+0000", 2024-03-27T21:40:34.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4790000000000004e-06 2024-03-27T21:40:34.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:34.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496796+0000", 2024-03-27T21:40:34.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1e-06 2024-03-27T21:40:34.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:34.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500447+0000", 2024-03-27T21:40:34.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036504459999999999 2024-03-27T21:40:34.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:34.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500492+0000", 2024-03-27T21:40:34.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5765e-05 2024-03-27T21:40:34.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:34.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500562+0000", 2024-03-27T21:40:34.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9206000000000001e-05 2024-03-27T21:40:34.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:34.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501959+0000", 2024-03-27T21:40:34.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001397187 2024-03-27T21:40:34.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:34.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501984+0000", 2024-03-27T21:40:34.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.552e-05 2024-03-27T21:40:34.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:34.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502010+0000", 2024-03-27T21:40:34.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5232000000000002e-05 2024-03-27T21:40:34.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:34.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502032+0000", 2024-03-27T21:40:34.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2337999999999999e-05 2024-03-27T21:40:34.764 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.764 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:34.764 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.764 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:34.764 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:34.764 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.218980367, 2024-03-27T21:40:34.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052892119999999997, 2024-03-27T21:40:34.765 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:34.765 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:34.765 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:34.765 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:34.765 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:34.765 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 134 2024-03-27T21:40:34.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.765 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:34.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:34.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:34.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:34.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:34.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:34.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:34.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:34.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:34.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496929+0000", 2024-03-27T21:40:34.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5098000000000004e-05 2024-03-27T21:40:34.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:34.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496937+0000", 2024-03-27T21:40:34.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9559999999999997e-06 2024-03-27T21:40:34.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:34.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496942+0000", 2024-03-27T21:40:34.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6350000000000003e-06 2024-03-27T21:40:34.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:34.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500749+0000", 2024-03-27T21:40:34.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038068529999999998 2024-03-27T21:40:34.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:34.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500792+0000", 2024-03-27T21:40:34.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2823e-05 2024-03-27T21:40:34.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:34.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500864+0000", 2024-03-27T21:40:34.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1715e-05 2024-03-27T21:40:34.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:34.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502068+0000", 2024-03-27T21:40:34.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012044860000000001 2024-03-27T21:40:34.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:34.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502073+0000", 2024-03-27T21:40:34.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5700000000000003e-06 2024-03-27T21:40:34.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:34.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502098+0000", 2024-03-27T21:40:34.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5120999999999999e-05 2024-03-27T21:40:34.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:34.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502133+0000", 2024-03-27T21:40:34.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5306999999999999e-05 2024-03-27T21:40:34.770 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.770 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:34.770 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.771 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:34.771 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:34.771 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.218875060000002, 2024-03-27T21:40:34.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062529040000000001, 2024-03-27T21:40:34.771 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:34.771 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:34.771 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:34.771 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:34.771 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:34.771 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135 2024-03-27T21:40:34.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.771 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:34.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:34.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:34.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:34.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:34.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:34.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:34.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:34.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:34.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496953+0000", 2024-03-27T21:40:34.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0709999999999996e-06 2024-03-27T21:40:34.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:34.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496961+0000", 2024-03-27T21:40:34.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1300000000000001e-06 2024-03-27T21:40:34.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:34.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496968+0000", 2024-03-27T21:40:34.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0999999999999998e-06 2024-03-27T21:40:34.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:34.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501244+0000", 2024-03-27T21:40:34.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042756310000000002 2024-03-27T21:40:34.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:34.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501377+0000", 2024-03-27T21:40:34.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013348799999999999 2024-03-27T21:40:34.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:34.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501455+0000", 2024-03-27T21:40:34.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7410000000000006e-05 2024-03-27T21:40:34.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:34.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502138+0000", 2024-03-27T21:40:34.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068386599999999999 2024-03-27T21:40:34.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:34.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503152+0000", 2024-03-27T21:40:34.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010139680000000001 2024-03-27T21:40:34.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:34.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503180+0000", 2024-03-27T21:40:34.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7426000000000001e-05 2024-03-27T21:40:34.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:34.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503202+0000", 2024-03-27T21:40:34.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2463000000000002e-05 2024-03-27T21:40:34.776 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.776 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:34.776 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.776 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:34.776 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:34.776 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.218850922000001, 2024-03-27T21:40:34.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066253680000000004, 2024-03-27T21:40:34.776 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:34.777 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:34.777 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:34.777 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:34.777 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:34.777 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 136 2024-03-27T21:40:34.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.777 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:34.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:34.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:34.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:34.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:34.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:34.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:34.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:34.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:34.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496977+0000", 2024-03-27T21:40:34.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5049999999999999e-06 2024-03-27T21:40:34.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:34.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496984+0000", 2024-03-27T21:40:34.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1520000000000003e-06 2024-03-27T21:40:34.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:34.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496990+0000", 2024-03-27T21:40:34.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-03-27T21:40:34.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:34.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501669+0000", 2024-03-27T21:40:34.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046793859999999998 2024-03-27T21:40:34.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:34.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501714+0000", 2024-03-27T21:40:34.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5154999999999997e-05 2024-03-27T21:40:34.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:34.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501783+0000", 2024-03-27T21:40:34.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8454999999999994e-05 2024-03-27T21:40:34.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:34.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503226+0000", 2024-03-27T21:40:34.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001443441 2024-03-27T21:40:34.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:34.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503540+0000", 2024-03-27T21:40:34.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031347100000000001 2024-03-27T21:40:34.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:34.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503563+0000", 2024-03-27T21:40:34.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3223000000000001e-05 2024-03-27T21:40:34.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:34.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503599+0000", 2024-03-27T21:40:34.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6105000000000001e-05 2024-03-27T21:40:34.781 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.781 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:34.782 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.782 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:34.782 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:34.782 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.218828111999997, 2024-03-27T21:40:34.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074261320000000002, 2024-03-27T21:40:34.782 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:34.782 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:34.782 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:34.782 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:34.782 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:34.782 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 137 2024-03-27T21:40:34.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.783 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:34.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:34.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:34.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:34.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:34.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:34.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:34.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:34.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:34.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497077+0000", 2024-03-27T21:40:34.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1599000000000003e-05 2024-03-27T21:40:34.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:34.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497085+0000", 2024-03-27T21:40:34.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6109999999999996e-06 2024-03-27T21:40:34.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:34.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497092+0000", 2024-03-27T21:40:34.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.742e-06 2024-03-27T21:40:34.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.785 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:34.785 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502159+0000", 2024-03-27T21:40:34.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050666870000000003 2024-03-27T21:40:34.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.785 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:34.785 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502200+0000", 2024-03-27T21:40:34.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1919999999999998e-05 2024-03-27T21:40:34.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.785 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:34.785 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502280+0000", 2024-03-27T21:40:34.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9417000000000002e-05 2024-03-27T21:40:34.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:34.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503605+0000", 2024-03-27T21:40:34.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001325641 2024-03-27T21:40:34.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:34.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504361+0000", 2024-03-27T21:40:34.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075555399999999997 2024-03-27T21:40:34.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:34.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504386+0000", 2024-03-27T21:40:34.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5369999999999999e-05 2024-03-27T21:40:34.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:34.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504422+0000", 2024-03-27T21:40:34.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6019000000000002e-05 2024-03-27T21:40:34.787 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.787 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:34.787 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.787 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:34.787 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:34.787 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.218724315000003, 2024-03-27T21:40:34.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074059750000000004, 2024-03-27T21:40:34.788 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:34.788 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:34.788 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:34.788 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:34.788 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:34.788 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-03-27T21:40:34.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.788 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:34.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:34.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:34.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:34.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:34.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:34.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:34.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:34.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:34.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497103+0000", 2024-03-27T21:40:34.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.856e-06 2024-03-27T21:40:34.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:34.790 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497121+0000", 2024-03-27T21:40:34.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7190999999999999e-05 2024-03-27T21:40:34.790 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.790 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:34.790 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497126+0000", 2024-03-27T21:40:34.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3770000000000004e-06 2024-03-27T21:40:34.790 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.790 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:34.790 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502833+0000", 2024-03-27T21:40:34.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057073510000000003 2024-03-27T21:40:34.790 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:34.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502881+0000", 2024-03-27T21:40:34.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7961999999999999e-05 2024-03-27T21:40:34.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:34.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502949+0000", 2024-03-27T21:40:34.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7957e-05 2024-03-27T21:40:34.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:34.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504430+0000", 2024-03-27T21:40:34.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001480763 2024-03-27T21:40:34.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:34.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504459+0000", 2024-03-27T21:40:34.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8983999999999999e-05 2024-03-27T21:40:34.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:34.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504486+0000", 2024-03-27T21:40:34.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6883999999999999e-05 2024-03-27T21:40:34.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:34.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504506+0000", 2024-03-27T21:40:34.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0214e-05 2024-03-27T21:40:34.792 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.792 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:34.793 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.793 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:34.793 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:34.793 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.218692480999998, 2024-03-27T21:40:34.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074754859999999999, 2024-03-27T21:40:34.793 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:34.793 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:34.793 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:34.793 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:34.793 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:34.793 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-03-27T21:40:34.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.793 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:34.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:34.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:34.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:34.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:34.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:34.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:34.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:34.795 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.795 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:34.795 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497137+0000", 2024-03-27T21:40:34.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0730000000000004e-06 2024-03-27T21:40:34.795 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.795 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:34.795 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497143+0000", 2024-03-27T21:40:34.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3759999999999996e-06 2024-03-27T21:40:34.795 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.795 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:34.795 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497150+0000", 2024-03-27T21:40:34.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.002e-06 2024-03-27T21:40:34.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:34.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503249+0000", 2024-03-27T21:40:34.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060989750000000004 2024-03-27T21:40:34.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:34.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503291+0000", 2024-03-27T21:40:34.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1563999999999997e-05 2024-03-27T21:40:34.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:34.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503376+0000", 2024-03-27T21:40:34.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5752000000000002e-05 2024-03-27T21:40:34.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:34.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504545+0000", 2024-03-27T21:40:34.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169172 2024-03-27T21:40:34.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:34.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504550+0000", 2024-03-27T21:40:34.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7419999999999997e-06 2024-03-27T21:40:34.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:34.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504574+0000", 2024-03-27T21:40:34.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4124000000000001e-05 2024-03-27T21:40:34.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:34.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504607+0000", 2024-03-27T21:40:34.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3118999999999999e-05 2024-03-27T21:40:34.798 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.798 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:34.798 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.798 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:34.798 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:34.798 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.218668815000001, 2024-03-27T21:40:34.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078150589999999992, 2024-03-27T21:40:34.798 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:34.799 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:34.799 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:34.799 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:34.799 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:34.799 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-03-27T21:40:34.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.799 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:34.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:34.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:34.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:34.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:34.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:34.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497155+0000", 2024-03-27T21:40:34.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:34.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:34.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497230+0000", 2024-03-27T21:40:34.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4789000000000005e-05 2024-03-27T21:40:34.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:34.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497241+0000", 2024-03-27T21:40:34.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1442e-05 2024-03-27T21:40:34.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:34.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497255+0000", 2024-03-27T21:40:34.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3087e-05 2024-03-27T21:40:34.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:34.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503630+0000", 2024-03-27T21:40:34.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063755269999999998 2024-03-27T21:40:34.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:34.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503673+0000", 2024-03-27T21:40:34.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2616e-05 2024-03-27T21:40:34.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:34.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503738+0000", 2024-03-27T21:40:34.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5114e-05 2024-03-27T21:40:34.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:34.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504907+0000", 2024-03-27T21:40:34.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169432 2024-03-27T21:40:34.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:34.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504926+0000", 2024-03-27T21:40:34.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9086999999999999e-05 2024-03-27T21:40:34.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:34.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504950+0000", 2024-03-27T21:40:34.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3598999999999999e-05 2024-03-27T21:40:34.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:34.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504971+0000", 2024-03-27T21:40:34.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0772999999999999e-05 2024-03-27T21:40:34.803 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.803 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:34.803 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.804 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:34.804 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:34.804 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.218561774999998, 2024-03-27T21:40:34.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082962590000000003, 2024-03-27T21:40:34.804 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:34.804 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:34.804 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:34.804 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:34.804 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:34.804 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-03-27T21:40:34.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.804 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:34.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.805 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:34.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:34.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.805 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.805 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:34.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:34.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.805 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.805 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:34.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497262+0000", 2024-03-27T21:40:34.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:34.805 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:34.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497267+0000", 2024-03-27T21:40:34.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.955e-06 2024-03-27T21:40:34.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:34.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497275+0000", 2024-03-27T21:40:34.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.824e-06 2024-03-27T21:40:34.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:34.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497280+0000", 2024-03-27T21:40:34.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8230000000000002e-06 2024-03-27T21:40:34.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:34.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504628+0000", 2024-03-27T21:40:34.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007348561 2024-03-27T21:40:34.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:34.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504673+0000", 2024-03-27T21:40:34.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4313000000000001e-05 2024-03-27T21:40:34.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:34.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504745+0000", 2024-03-27T21:40:34.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2303000000000002e-05 2024-03-27T21:40:34.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:34.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505358+0000", 2024-03-27T21:40:34.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061281300000000003 2024-03-27T21:40:34.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:34.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505503+0000", 2024-03-27T21:40:34.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000145134 2024-03-27T21:40:34.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:34.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505526+0000", 2024-03-27T21:40:34.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3312e-05 2024-03-27T21:40:34.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:34.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505559+0000", 2024-03-27T21:40:34.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2821999999999998e-05 2024-03-27T21:40:34.809 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.809 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:34.809 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.809 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:34.809 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:34.809 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 53.609339536, 2024-03-27T21:40:34.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053753339999999998, 2024-03-27T21:40:34.809 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:34.810 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:34.810 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:34.810 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:34.810 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:34.810 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159 2024-03-27T21:40:34.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.810 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:34.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.810 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:34.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:34.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.810 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:34.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:34.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:34.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:34.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:34.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:34.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:34.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106488+0000", 2024-03-27T21:40:34.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9830000000000001e-06 2024-03-27T21:40:34.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:34.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106491+0000", 2024-03-27T21:40:34.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3459999999999998e-06 2024-03-27T21:40:34.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:34.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106493+0000", 2024-03-27T21:40:34.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.509e-06 2024-03-27T21:40:34.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:34.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110636+0000", 2024-03-27T21:40:34.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041427260000000002 2024-03-27T21:40:34.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.813 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:34.813 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110683+0000", 2024-03-27T21:40:34.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6557000000000002e-05 2024-03-27T21:40:34.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.813 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:34.813 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110755+0000", 2024-03-27T21:40:34.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2119000000000005e-05 2024-03-27T21:40:34.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.813 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:34.813 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111415+0000", 2024-03-27T21:40:34.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066065199999999996 2024-03-27T21:40:34.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:34.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111793+0000", 2024-03-27T21:40:34.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037732100000000001 2024-03-27T21:40:34.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:34.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111829+0000", 2024-03-27T21:40:34.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418999999999998e-05 2024-03-27T21:40:34.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:34.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:34.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:34.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111860+0000", 2024-03-27T21:40:34.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1003000000000001e-05 2024-03-27T21:40:34.814 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.814 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:34.815 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.815 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:34.815 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:34.815 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:34.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.772+0000 7fe6bbf99640 10 start_mon_command cmd=[{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:40:34.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.772+0000 7fe6bbf99640 10 monclient: _send_command 2 [{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:40:34.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.772+0000 7fe6bbf99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:34.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:34.772+0000 7fe6bbf99640 1 -- 172.21.15.60:0/2998563414 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fe6b4069470 con 0x7fe6b413e810 2024-03-27T21:40:34.817 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-27T21:40:34.817 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:34.830+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:34.916 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:34.930+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:34.927 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:34.927 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:34.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-27T21:40:35.017 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:35.030+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:35.099 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:35.099 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:35.099 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:35.099 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:35.099 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:35.111 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-27T21:40:35.155 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:35.169+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:35.219 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:35.232+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:35.219 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:35.219 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:35.219 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:35.219 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:40:35.219 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:35.229 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:40:35.320 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:35.334+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:35.380 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:35.380 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:35.380 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:35.380 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:35.389 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:40:35.421 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:35.434+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:35.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.457+0000 7fe6aa7fc640 1 -- 172.21.15.60:0/2998563414 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v40) v1 ==== 91+0+0 (secure 0 0 0) 0x7fe6a4057d30 con 0x7fe6b413e810 2024-03-27T21:40:35.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.457+0000 7fe6aa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:40:35.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.457+0000 7fe6aa7fc640 10 monclient: _finish_command 2 = system:0 noscrub is unset 2024-03-27T21:40:35.444 INFO:teuthology.orchestra.run.smithi060.stderr:noscrub is unset 2024-03-27T21:40:35.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.459+0000 7fe6bbf99640 10 librados: watch_flush enter 2024-03-27T21:40:35.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.460+0000 7fe6bbf99640 10 librados: watch_flush exit 2024-03-27T21:40:35.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.460+0000 7fe6bbf99640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-27T21:40:35.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.460+0000 7fe6bbf99640 10 client.4547.objecter successfully canceled tick 2024-03-27T21:40:35.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.460+0000 7fe6bbf99640 1 -- 172.21.15.60:0/2998563414 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe69806c590 msgr2=0x7fe69806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:35.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.460+0000 7fe6bbf99640 1 --2- 172.21.15.60:0/2998563414 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe69806c590 0x7fe69806ea50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0036000 tx=0x7fe6b003e040 comp rx=0 tx=0).stop 2024-03-27T21:40:35.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.460+0000 7fe6bbf99640 10 monclient: shutdown 2024-03-27T21:40:35.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.460+0000 7fe6bbf99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:35.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.460+0000 7fe6bbf99640 1 -- 172.21.15.60:0/2998563414 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe6b413e810 msgr2=0x7fe6b4133050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:35.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.460+0000 7fe6bbf99640 1 --2- 172.21.15.60:0/2998563414 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe6b413e810 0x7fe6b4133050 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe6a400e9e0 tx=0x7fe6a400eea0 comp rx=0 tx=0).stop 2024-03-27T21:40:35.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.460+0000 7fe6bbf99640 1 -- 172.21.15.60:0/2998563414 shutdown_connections 2024-03-27T21:40:35.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.460+0000 7fe6bbf99640 1 --2- 172.21.15.60:0/2998563414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6b4141630 0x7fe6b4133590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:35.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.460+0000 7fe6bbf99640 1 --2- 172.21.15.60:0/2998563414 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe6b413e810 0x7fe6b4133050 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:35.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.460+0000 7fe6bbf99640 1 --2- 172.21.15.60:0/2998563414 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe69806c590 0x7fe69806ea50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:35.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.460+0000 7fe6bbf99640 1 --2- 172.21.15.60:0/2998563414 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6b4071a50 0x7fe6b41d6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:35.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.460+0000 7fe6bbf99640 1 -- 172.21.15.60:0/2998563414 >> 172.21.15.60:0/2998563414 conn(0x7fe6b40b7300 msgr2=0x7fe6b4141aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:35.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.461+0000 7fe6bbf99640 1 -- 172.21.15.60:0/2998563414 shutdown_connections 2024-03-27T21:40:35.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.461+0000 7fe6bbf99640 1 -- 172.21.15.60:0/2998563414 wait complete. 2024-03-27T21:40:35.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.461+0000 7fe6bbf99640 1 librados: shutdown 2024-03-27T21:40:35.521 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:35.535+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:35.543 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:35.543 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:35.543 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:35.543 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:35.543 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:35.553 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:40:35.620 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:40:35.622 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:35.636+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:35.687 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:35.688 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:35.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:35.688 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-03-27T21:40:35.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.688 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:20 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2024-03-27T21:40:35.688 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:35.688 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 55.811404697, 2024-03-27T21:40:35.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015345630000000001, 2024-03-27T21:40:35.688 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:35.688 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:35.688 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:35.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:35.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:35.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:35.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:35.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:35.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888562+0000", 2024-03-27T21:40:35.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:35.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:35.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888735+0000", 2024-03-27T21:40:35.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017254799999999999 2024-03-27T21:40:35.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:35.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888747+0000", 2024-03-27T21:40:35.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2469999999999999e-05 2024-03-27T21:40:35.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:35.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888761+0000", 2024-03-27T21:40:35.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3254e-05 2024-03-27T21:40:35.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:35.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888804+0000", 2024-03-27T21:40:35.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3129e-05 2024-03-27T21:40:35.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:35.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888820+0000", 2024-03-27T21:40:35.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6426000000000002e-05 2024-03-27T21:40:35.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:35.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890060+0000", 2024-03-27T21:40:35.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012398050000000001 2024-03-27T21:40:35.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.692 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:35.692 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890098+0000", 2024-03-27T21:40:35.692 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7795e-05 2024-03-27T21:40:35.692 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.692 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:35.692 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.692 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2024-03-27T21:40:35.692 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:35.692 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 55.810580297000001, 2024-03-27T21:40:35.692 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001069555, 2024-03-27T21:40:35.692 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:35.693 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:35.693 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:35.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.693 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:35.693 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:35.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.693 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:35.693 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:35.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:35.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889387+0000", 2024-03-27T21:40:35.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:35.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:35.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889455+0000", 2024-03-27T21:40:35.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7930999999999997e-05 2024-03-27T21:40:35.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:35.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889470+0000", 2024-03-27T21:40:35.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5554e-05 2024-03-27T21:40:35.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:35.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889476+0000", 2024-03-27T21:40:35.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0619999999999997e-06 2024-03-27T21:40:35.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:35.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889513+0000", 2024-03-27T21:40:35.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6606999999999997e-05 2024-03-27T21:40:35.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:35.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889528+0000", 2024-03-27T21:40:35.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4982e-05 2024-03-27T21:40:35.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:35.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890421+0000", 2024-03-27T21:40:35.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00089293700000000001 2024-03-27T21:40:35.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:35.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890457+0000", 2024-03-27T21:40:35.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6260000000000002e-05 2024-03-27T21:40:35.696 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.696 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:35.697 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.697 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:38 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'24, mlcod=17'22)", 2024-03-27T21:40:35.697 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:35.697 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 55.793344331, 2024-03-27T21:40:35.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016182080000000001, 2024-03-27T21:40:35.697 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:35.697 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:35.697 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:35.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.697 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:35.697 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:35.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:35.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:35.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:35.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906623+0000", 2024-03-27T21:40:35.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:35.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:35.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906687+0000", 2024-03-27T21:40:35.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3706000000000003e-05 2024-03-27T21:40:35.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:35.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906698+0000", 2024-03-27T21:40:35.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1752e-05 2024-03-27T21:40:35.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:35.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906704+0000", 2024-03-27T21:40:35.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3940000000000004e-06 2024-03-27T21:40:35.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:35.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906741+0000", 2024-03-27T21:40:35.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6944999999999999e-05 2024-03-27T21:40:35.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:35.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906760+0000", 2024-03-27T21:40:35.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9293e-05 2024-03-27T21:40:35.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:35.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908211+0000", 2024-03-27T21:40:35.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014511350000000001 2024-03-27T21:40:35.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:35.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908242+0000", 2024-03-27T21:40:35.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0712e-05 2024-03-27T21:40:35.701 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.701 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:35.701 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.701 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:39 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'25, mlcod=17'22)", 2024-03-27T21:40:35.701 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:35.701 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 55.792499415999998, 2024-03-27T21:40:35.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012060199999999999, 2024-03-27T21:40:35.701 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:35.701 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:35.701 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:35.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.701 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:35.702 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:35.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.702 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:35.702 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:35.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.702 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:35.702 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:35.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:35.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:35.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907533+0000", 2024-03-27T21:40:35.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5675999999999999e-05 2024-03-27T21:40:35.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:35.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907545+0000", 2024-03-27T21:40:35.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.115e-05 2024-03-27T21:40:35.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:35.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907550+0000", 2024-03-27T21:40:35.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8320000000000002e-06 2024-03-27T21:40:35.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:35.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907592+0000", 2024-03-27T21:40:35.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1309e-05 2024-03-27T21:40:35.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:35.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907611+0000", 2024-03-27T21:40:35.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9009e-05 2024-03-27T21:40:35.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:35.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908638+0000", 2024-03-27T21:40:35.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001027533 2024-03-27T21:40:35.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:35.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908675+0000", 2024-03-27T21:40:35.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6329999999999999e-05 2024-03-27T21:40:35.705 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.705 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:35.705 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.705 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:57 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'35, mlcod=17'33)", 2024-03-27T21:40:35.705 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:35.705 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 55.778414984000001, 2024-03-27T21:40:35.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001346578, 2024-03-27T21:40:35.705 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:35.705 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:35.706 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:35.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.706 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:35.706 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:35.706 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.706 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.706 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:35.706 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:35.706 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:35.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921552+0000", 2024-03-27T21:40:35.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:35.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:35.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921615+0000", 2024-03-27T21:40:35.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2787999999999996e-05 2024-03-27T21:40:35.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:35.708 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921626+0000", 2024-03-27T21:40:35.708 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0888e-05 2024-03-27T21:40:35.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.708 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:35.708 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921635+0000", 2024-03-27T21:40:35.708 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4639999999999995e-06 2024-03-27T21:40:35.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.708 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:35.708 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921673+0000", 2024-03-27T21:40:35.708 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7891e-05 2024-03-27T21:40:35.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.708 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:35.709 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921694+0000", 2024-03-27T21:40:35.709 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0426000000000001e-05 2024-03-27T21:40:35.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.709 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:35.709 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922865+0000", 2024-03-27T21:40:35.709 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011709509999999999 2024-03-27T21:40:35.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.709 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:35.709 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922900+0000", 2024-03-27T21:40:35.709 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4819e-05 2024-03-27T21:40:35.709 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.709 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:35.710 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.710 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.710 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:71 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'45, mlcod=17'43)", 2024-03-27T21:40:35.710 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:35.710 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 55.763116271999998, 2024-03-27T21:40:35.710 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001460907, 2024-03-27T21:40:35.710 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:35.710 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:35.710 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:35.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.710 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:35.710 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:35.711 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.711 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:35.711 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:35.711 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.711 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:35.711 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936851+0000", 2024-03-27T21:40:35.711 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:35.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.711 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:35.711 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936903+0000", 2024-03-27T21:40:35.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2302999999999997e-05 2024-03-27T21:40:35.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.712 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:35.712 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936915+0000", 2024-03-27T21:40:35.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1967000000000001e-05 2024-03-27T21:40:35.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.712 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:35.712 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936923+0000", 2024-03-27T21:40:35.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9370000000000004e-06 2024-03-27T21:40:35.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:35.713 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936970+0000", 2024-03-27T21:40:35.713 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6987000000000003e-05 2024-03-27T21:40:35.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:35.713 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937002+0000", 2024-03-27T21:40:35.713 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2410999999999999e-05 2024-03-27T21:40:35.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:35.714 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938279+0000", 2024-03-27T21:40:35.714 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001276552 2024-03-27T21:40:35.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.714 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:35.714 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938313+0000", 2024-03-27T21:40:35.714 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3526999999999999e-05 2024-03-27T21:40:35.714 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.714 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:35.714 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.714 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:72 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'46, mlcod=17'43)", 2024-03-27T21:40:35.714 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:35.714 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 55.762478710000003, 2024-03-27T21:40:35.715 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014239490000000001, 2024-03-27T21:40:35.715 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:35.715 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:35.715 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:35.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.715 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:35.715 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:35.715 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.716 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:35.716 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:35.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.716 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:35.716 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937488+0000", 2024-03-27T21:40:35.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:35.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.716 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:35.716 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937547+0000", 2024-03-27T21:40:35.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8196999999999999e-05 2024-03-27T21:40:35.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.717 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:35.717 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937559+0000", 2024-03-27T21:40:35.717 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1928e-05 2024-03-27T21:40:35.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.717 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:35.717 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937564+0000", 2024-03-27T21:40:35.717 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4759999999999997e-06 2024-03-27T21:40:35.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.717 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:35.717 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937621+0000", 2024-03-27T21:40:35.718 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6712000000000002e-05 2024-03-27T21:40:35.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.718 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:35.718 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937640+0000", 2024-03-27T21:40:35.718 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9517e-05 2024-03-27T21:40:35.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.718 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:35.718 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938883+0000", 2024-03-27T21:40:35.718 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001242567 2024-03-27T21:40:35.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.718 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:35.719 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938913+0000", 2024-03-27T21:40:35.719 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0287e-05 2024-03-27T21:40:35.719 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.719 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:35.719 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.719 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.719 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:73 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'47, mlcod=17'43)", 2024-03-27T21:40:35.719 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:35.719 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 55.761802615999997, 2024-03-27T21:40:35.719 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010746239999999999, 2024-03-27T21:40:35.719 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:35.719 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:35.719 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:35.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.720 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:35.720 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:35.720 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.720 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:35.720 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:35.720 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.720 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:35.720 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:35.720 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:35.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.721 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:35.721 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938217+0000", 2024-03-27T21:40:35.721 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2743e-05 2024-03-27T21:40:35.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.721 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:35.721 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938229+0000", 2024-03-27T21:40:35.721 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1956999999999999e-05 2024-03-27T21:40:35.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.721 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:35.721 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938235+0000", 2024-03-27T21:40:35.721 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4589999999999997e-06 2024-03-27T21:40:35.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.722 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:35.722 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938341+0000", 2024-03-27T21:40:35.722 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000106054 2024-03-27T21:40:35.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.722 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:35.722 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938369+0000", 2024-03-27T21:40:35.722 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8291000000000001e-05 2024-03-27T21:40:35.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.722 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:35.722 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939209+0000", 2024-03-27T21:40:35.722 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084045400000000003 2024-03-27T21:40:35.723 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.723 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.723 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:35.723 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939240+0000", 2024-03-27T21:40:35.723 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0429999999999998e-05 2024-03-27T21:40:35.723 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.723 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:35.723 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.723 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.723 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.723 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:75 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'49, mlcod=17'47)", 2024-03-27T21:40:35.723 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:35.723 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 55.755364968999999, 2024-03-27T21:40:35.723 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001017001, 2024-03-27T21:40:35.724 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:35.724 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:35.724 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:35.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.724 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:35.724 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:35.724 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.724 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:35.724 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:35.724 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.724 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:35.725 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944601+0000", 2024-03-27T21:40:35.725 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999986 2024-03-27T21:40:35.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.725 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:35.725 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944614+0000", 2024-03-27T21:40:35.725 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2496e-05 2024-03-27T21:40:35.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.725 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:35.725 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944640+0000", 2024-03-27T21:40:35.725 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5865999999999999e-05 2024-03-27T21:40:35.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.726 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:35.726 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944651+0000", 2024-03-27T21:40:35.726 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1797000000000001e-05 2024-03-27T21:40:35.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.726 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.726 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:35.726 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944762+0000", 2024-03-27T21:40:35.726 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110412 2024-03-27T21:40:35.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.726 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.726 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:35.726 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944820+0000", 2024-03-27T21:40:35.726 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7649999999999999e-05 2024-03-27T21:40:35.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.727 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:35.727 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945585+0000", 2024-03-27T21:40:35.727 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076562600000000005 2024-03-27T21:40:35.727 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.727 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:35.727 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945620+0000", 2024-03-27T21:40:35.727 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4764000000000001e-05 2024-03-27T21:40:35.727 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.727 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:35.727 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.727 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.728 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:86 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'50)", 2024-03-27T21:40:35.728 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:35.728 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 55.747603482000002, 2024-03-27T21:40:35.728 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011392380000000001, 2024-03-27T21:40:35.728 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:35.728 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:35.728 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:35.728 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.728 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:35.728 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:35.728 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.728 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.728 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.728 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:35.728 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:35.729 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.729 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.729 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.729 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:35.729 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:35.729 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:35.729 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.729 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.729 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:35.729 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952373+0000", 2024-03-27T21:40:35.729 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9220000000000003e-06 2024-03-27T21:40:35.729 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.729 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.729 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:35.730 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952381+0000", 2024-03-27T21:40:35.730 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3070000000000003e-06 2024-03-27T21:40:35.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.730 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:35.730 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952385+0000", 2024-03-27T21:40:35.730 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.121e-06 2024-03-27T21:40:35.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.730 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:35.730 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952419+0000", 2024-03-27T21:40:35.730 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4082999999999999e-05 2024-03-27T21:40:35.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.731 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:35.731 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952440+0000", 2024-03-27T21:40:35.731 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0837999999999999e-05 2024-03-27T21:40:35.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.731 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:35.731 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953472+0000", 2024-03-27T21:40:35.731 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001032119 2024-03-27T21:40:35.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.731 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:35.731 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953504+0000", 2024-03-27T21:40:35.731 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1250000000000001e-05 2024-03-27T21:40:35.731 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.731 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:35.732 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.732 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.732 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:87 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-27T21:40:35.732 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:35.732 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 55.747236225000002, 2024-03-27T21:40:35.732 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00135519, 2024-03-27T21:40:35.732 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:35.732 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:35.732 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:35.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.732 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:35.732 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:35.732 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.732 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.733 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:35.733 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:35.733 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.733 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:35.733 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952731+0000", 2024-03-27T21:40:35.733 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:35.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.733 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:35.733 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952781+0000", 2024-03-27T21:40:35.733 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0161000000000003e-05 2024-03-27T21:40:35.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.734 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.734 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:35.734 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952794+0000", 2024-03-27T21:40:35.734 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2853e-05 2024-03-27T21:40:35.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.734 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.734 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:35.734 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952799+0000", 2024-03-27T21:40:35.734 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5560000000000001e-06 2024-03-27T21:40:35.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.734 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.734 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:35.734 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952832+0000", 2024-03-27T21:40:35.735 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3834999999999997e-05 2024-03-27T21:40:35.735 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.735 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.735 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:35.735 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952851+0000", 2024-03-27T21:40:35.735 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8289999999999999e-05 2024-03-27T21:40:35.735 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.735 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.735 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:35.735 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954055+0000", 2024-03-27T21:40:35.735 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012037949999999999 2024-03-27T21:40:35.735 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.735 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.735 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:35.735 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954087+0000", 2024-03-27T21:40:35.736 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2292999999999997e-05 2024-03-27T21:40:35.736 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.736 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:35.736 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.736 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:89 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'59, mlcod=17'57)", 2024-03-27T21:40:35.736 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:35.736 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 55.744962004000001, 2024-03-27T21:40:35.736 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00138893, 2024-03-27T21:40:35.736 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:35.736 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:35.736 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:35.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.736 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:35.737 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:35.737 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.737 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.737 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.737 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:35.737 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:35.737 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.737 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.737 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.737 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:35.737 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955005+0000", 2024-03-27T21:40:35.737 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:35.737 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.737 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.738 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:35.738 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955071+0000", 2024-03-27T21:40:35.738 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5290999999999995e-05 2024-03-27T21:40:35.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.738 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.738 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:35.738 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955081+0000", 2024-03-27T21:40:35.738 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0572e-05 2024-03-27T21:40:35.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.738 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.738 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:35.738 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955086+0000", 2024-03-27T21:40:35.738 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8380000000000001e-06 2024-03-27T21:40:35.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.739 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.739 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:35.739 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955122+0000", 2024-03-27T21:40:35.739 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5698000000000003e-05 2024-03-27T21:40:35.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.739 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.739 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:35.739 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955140+0000", 2024-03-27T21:40:35.739 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7900000000000001e-05 2024-03-27T21:40:35.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.739 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.739 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:35.739 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956366+0000", 2024-03-27T21:40:35.739 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001226471 2024-03-27T21:40:35.740 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.740 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.740 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:35.740 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956395+0000", 2024-03-27T21:40:35.740 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8782e-05 2024-03-27T21:40:35.740 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.740 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:35.740 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.740 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.740 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.740 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:90 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'60, mlcod=17'57)", 2024-03-27T21:40:35.740 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:35.740 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 55.744277427, 2024-03-27T21:40:35.740 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010778859999999999, 2024-03-27T21:40:35.740 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:35.741 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:35.741 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:35.741 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.741 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:35.741 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:35.741 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.741 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.741 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.741 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:35.741 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:35.741 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.741 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.741 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.741 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:35.742 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:35.742 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:35.742 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.742 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.742 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:35.742 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955747+0000", 2024-03-27T21:40:35.742 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7130999999999999e-05 2024-03-27T21:40:35.742 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.742 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.742 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:35.742 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955756+0000", 2024-03-27T21:40:35.742 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2750000000000001e-06 2024-03-27T21:40:35.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.743 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:35.743 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955761+0000", 2024-03-27T21:40:35.743 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3900000000000003e-06 2024-03-27T21:40:35.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.744 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.744 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:35.744 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955797+0000", 2024-03-27T21:40:35.744 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6650000000000003e-05 2024-03-27T21:40:35.744 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.744 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.745 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:35.745 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955816+0000", 2024-03-27T21:40:35.745 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8195000000000001e-05 2024-03-27T21:40:35.745 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.745 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.745 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:35.746 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956739+0000", 2024-03-27T21:40:35.746 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00092369700000000002 2024-03-27T21:40:35.746 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.746 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.746 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:35.746 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956768+0000", 2024-03-27T21:40:35.747 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9048e-05 2024-03-27T21:40:35.747 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.747 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:35.747 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.747 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.748 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.748 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:118 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'80, mlcod=17'78)", 2024-03-27T21:40:35.748 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:35.748 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 55.611286604, 2024-03-27T21:40:35.748 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014200669999999999, 2024-03-27T21:40:35.749 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:35.749 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:35.749 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:35.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.749 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:35.750 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:35.750 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.750 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.750 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.751 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:35.751 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:35.751 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.751 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:35.752 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:35.752 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:35.752 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.752 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.752 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:35.752 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088756+0000", 2024-03-27T21:40:35.752 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4902999999999998e-05 2024-03-27T21:40:35.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.753 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:35.753 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088764+0000", 2024-03-27T21:40:35.753 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8169999999999999e-06 2024-03-27T21:40:35.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.754 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:35.754 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088767+0000", 2024-03-27T21:40:35.754 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8310000000000003e-06 2024-03-27T21:40:35.754 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.755 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:35.755 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088798+0000", 2024-03-27T21:40:35.755 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0800999999999999e-05 2024-03-27T21:40:35.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.756 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:35.756 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088817+0000", 2024-03-27T21:40:35.756 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8467000000000001e-05 2024-03-27T21:40:35.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.757 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:35.757 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090074+0000", 2024-03-27T21:40:35.757 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012574089999999999 2024-03-27T21:40:35.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.758 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:35.758 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090101+0000", 2024-03-27T21:40:35.758 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7223e-05 2024-03-27T21:40:35.758 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.758 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:35.758 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.759 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.759 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.759 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:119 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'81, mlcod=17'78)", 2024-03-27T21:40:35.759 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:35.759 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 55.610551809, 2024-03-27T21:40:35.759 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014990870000000001, 2024-03-27T21:40:35.760 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:35.760 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:35.760 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:35.760 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.760 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:35.760 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:35.761 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.761 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.761 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.761 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:35.761 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:35.761 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.762 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.762 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.762 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:35.762 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:35.762 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:35.762 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.763 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.763 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:35.763 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089491+0000", 2024-03-27T21:40:35.763 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4805e-05 2024-03-27T21:40:35.763 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.763 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.763 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:35.763 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089498+0000", 2024-03-27T21:40:35.764 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0650000000000001e-06 2024-03-27T21:40:35.764 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.764 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.764 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:35.764 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089501+0000", 2024-03-27T21:40:35.764 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.54e-06 2024-03-27T21:40:35.764 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.764 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.765 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:35.765 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089533+0000", 2024-03-27T21:40:35.765 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1838999999999998e-05 2024-03-27T21:40:35.765 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.765 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.765 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:35.765 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089550+0000", 2024-03-27T21:40:35.765 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7465999999999998e-05 2024-03-27T21:40:35.766 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.766 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.766 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:35.766 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090890+0000", 2024-03-27T21:40:35.766 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013397979999999999 2024-03-27T21:40:35.766 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.766 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.767 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:35.767 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090915+0000", 2024-03-27T21:40:35.767 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4905e-05 2024-03-27T21:40:35.767 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.767 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:35.767 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.768 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.768 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.768 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:120 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'82, mlcod=17'78)", 2024-03-27T21:40:35.768 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:35.768 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 55.609751928999998, 2024-03-27T21:40:35.768 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001527093, 2024-03-27T21:40:35.769 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:35.769 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:35.769 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:35.769 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.769 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:35.769 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:35.769 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.769 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.769 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.769 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:35.769 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:35.769 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.769 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.770 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.770 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:35.770 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:35.770 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:35.770 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.770 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.770 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:35.770 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090267+0000", 2024-03-27T21:40:35.770 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1477000000000002e-05 2024-03-27T21:40:35.770 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.770 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.770 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:35.771 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090273+0000", 2024-03-27T21:40:35.771 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9519999999999999e-06 2024-03-27T21:40:35.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.771 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.771 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:35.771 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090276+0000", 2024-03-27T21:40:35.771 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.219e-06 2024-03-27T21:40:35.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.771 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.771 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:35.771 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090304+0000", 2024-03-27T21:40:35.771 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8073999999999999e-05 2024-03-27T21:40:35.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.772 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.772 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:35.772 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090316+0000", 2024-03-27T21:40:35.772 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2088e-05 2024-03-27T21:40:35.772 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.772 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.772 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:35.772 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091717+0000", 2024-03-27T21:40:35.772 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014003889999999999 2024-03-27T21:40:35.772 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.772 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.772 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:35.772 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091743+0000", 2024-03-27T21:40:35.772 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6234e-05 2024-03-27T21:40:35.772 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.773 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:35.773 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.773 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.773 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:121 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'83, mlcod=17'78)", 2024-03-27T21:40:35.773 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:35.773 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 55.608941563999998, 2024-03-27T21:40:35.773 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010226899999999999, 2024-03-27T21:40:35.773 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:35.773 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:35.773 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:35.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.774 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:35.774 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:35.774 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.774 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.774 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.774 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:35.774 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:35.774 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.774 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.774 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.774 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:35.774 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:35.774 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:35.775 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.775 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.775 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:35.775 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091142+0000", 2024-03-27T21:40:35.775 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011608300000000001 2024-03-27T21:40:35.775 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.775 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.775 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:35.775 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091151+0000", 2024-03-27T21:40:35.775 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2350000000000005e-06 2024-03-27T21:40:35.775 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.775 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.775 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:35.775 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091156+0000", 2024-03-27T21:40:35.775 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5419999999999999e-06 2024-03-27T21:40:35.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.776 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.776 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:35.776 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091191+0000", 2024-03-27T21:40:35.776 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5453000000000001e-05 2024-03-27T21:40:35.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.776 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.776 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:35.776 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091208+0000", 2024-03-27T21:40:35.776 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7119999999999999e-05 2024-03-27T21:40:35.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.776 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.776 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:35.777 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092024+0000", 2024-03-27T21:40:35.777 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000815539 2024-03-27T21:40:35.777 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.777 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.777 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:35.777 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092049+0000", 2024-03-27T21:40:35.777 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5171e-05 2024-03-27T21:40:35.777 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.777 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:35.777 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.777 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.777 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.777 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:142 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'100, mlcod=18'98)", 2024-03-27T21:40:35.777 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:35.778 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 55.193934609999999, 2024-03-27T21:40:35.778 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001198778, 2024-03-27T21:40:35.778 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:35.778 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:35.778 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:35.778 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.778 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:35.778 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:35.778 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.778 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.779 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.779 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:35.779 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:35.779 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.779 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.779 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.779 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:35.779 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:35.779 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:35.779 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.779 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.779 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:35.779 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506101+0000", 2024-03-27T21:40:35.780 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8202999999999997e-05 2024-03-27T21:40:35.780 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.780 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.780 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:35.780 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506109+0000", 2024-03-27T21:40:35.780 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3059999999999998e-06 2024-03-27T21:40:35.780 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.780 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.780 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:35.780 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506113+0000", 2024-03-27T21:40:35.780 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0119999999999999e-06 2024-03-27T21:40:35.780 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.780 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.780 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:35.781 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506137+0000", 2024-03-27T21:40:35.781 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4043999999999999e-05 2024-03-27T21:40:35.781 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.781 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.781 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:35.781 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506148+0000", 2024-03-27T21:40:35.781 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0446e-05 2024-03-27T21:40:35.781 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.781 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.781 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:35.781 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507213+0000", 2024-03-27T21:40:35.781 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001065215 2024-03-27T21:40:35.781 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.781 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.782 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:35.782 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507232+0000", 2024-03-27T21:40:35.782 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9038e-05 2024-03-27T21:40:35.782 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.782 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:35.782 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.782 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.782 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.782 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:143 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'101, mlcod=18'98)", 2024-03-27T21:40:35.782 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:35.782 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 55.193321990999998, 2024-03-27T21:40:35.782 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001357488, 2024-03-27T21:40:35.782 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:35.783 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:35.783 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:35.783 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.783 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:35.783 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:35.783 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.783 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.783 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.783 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:35.783 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:35.783 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.783 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.784 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.784 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:35.784 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:35.784 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:35.784 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.784 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.784 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:35.784 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506723+0000", 2024-03-27T21:40:35.784 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7404000000000006e-05 2024-03-27T21:40:35.784 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.784 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.784 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:35.784 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506730+0000", 2024-03-27T21:40:35.784 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5749999999999997e-06 2024-03-27T21:40:35.784 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.785 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.785 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:35.785 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506733+0000", 2024-03-27T21:40:35.785 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7979999999999999e-06 2024-03-27T21:40:35.785 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.785 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.785 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:35.785 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506758+0000", 2024-03-27T21:40:35.785 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4179e-05 2024-03-27T21:40:35.785 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.785 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.785 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:35.785 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506768+0000", 2024-03-27T21:40:35.785 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0740000000000001e-05 2024-03-27T21:40:35.785 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.786 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.786 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:35.786 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507978+0000", 2024-03-27T21:40:35.786 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012093900000000001 2024-03-27T21:40:35.786 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.786 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.786 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:35.786 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508003+0000", 2024-03-27T21:40:35.786 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5667e-05 2024-03-27T21:40:35.786 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.786 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:35.786 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.786 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.786 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.787 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:144 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'102, mlcod=18'98)", 2024-03-27T21:40:35.787 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:35.787 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 55.192663093999997, 2024-03-27T21:40:35.787 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013453390000000001, 2024-03-27T21:40:35.787 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:35.787 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:35.787 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:35.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.787 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:35.787 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:35.787 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.788 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.788 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:35.788 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:35.788 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:35.788 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.788 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.788 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:35.788 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507304+0000", 2024-03-27T21:40:35.788 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:35.788 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.788 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.788 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:35.788 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507397+0000", 2024-03-27T21:40:35.788 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2995e-05 2024-03-27T21:40:35.789 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.789 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.789 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:35.789 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507403+0000", 2024-03-27T21:40:35.789 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8640000000000001e-06 2024-03-27T21:40:35.789 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.789 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.789 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:35.789 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507407+0000", 2024-03-27T21:40:35.789 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5710000000000002e-06 2024-03-27T21:40:35.789 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.789 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.789 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:35.789 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507428+0000", 2024-03-27T21:40:35.790 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1192e-05 2024-03-27T21:40:35.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.790 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.790 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:35.790 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507438+0000", 2024-03-27T21:40:35.790 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8309999999999996e-06 2024-03-27T21:40:35.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.790 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.790 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:35.790 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508624+0000", 2024-03-27T21:40:35.790 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011861210000000001 2024-03-27T21:40:35.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:35.790 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:35.791 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:35.791 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508650+0000", 2024-03-27T21:40:35.791 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6118e-05 2024-03-27T21:40:35.791 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.791 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:35.791 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.791 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:35.791 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:35.791 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:35.793 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:40:35.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.760+0000 7fdb3d69d640 10 monclient: get_monmap_and_config 2024-03-27T21:40:35.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.760+0000 7fdb3d69d640 10 monclient: build_initial_monmap 2024-03-27T21:40:35.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.760+0000 7fdb3d69d640 10 monclient: monmap: 2024-03-27T21:40:35.805 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:35.805 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:35.805 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:35.761533+0000 2024-03-27T21:40:35.805 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:35.761533+0000 2024-03-27T21:40:35.805 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:35.805 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:35.805 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:35.805 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:35.805 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:35.805 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:35.806 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:35.806 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:35.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.761+0000 7fdb3d69d640 1 Processor -- start 2024-03-27T21:40:35.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.761+0000 7fdb3d69d640 1 -- start start 2024-03-27T21:40:35.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.761+0000 7fdb3d69d640 10 monclient: init 2024-03-27T21:40:35.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.761+0000 7fdb3d69d640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:35.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.761+0000 7fdb3d69d640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-03-27T21:40:35.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.761+0000 7fdb3d69d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fdb380b40b0 0x7fdb380a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:35.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.761+0000 7fdb3d69d640 10 monclient(hunting): picked mon.noname-c con 0x7fdb380b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:35.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.761+0000 7fdb3d69d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdb3813b020 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:35.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.761+0000 7fdb3d69d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb3813b480 0x7fdb38141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:35.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.761+0000 7fdb3d69d640 10 monclient(hunting): picked mon.noname-a con 0x7fdb3813b480 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:35.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.761+0000 7fdb3d69d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3813b8d0 con 0x7fdb3813b020 2024-03-27T21:40:35.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.761+0000 7fdb3d69d640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:35.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.761+0000 7fdb3d69d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb380b4490 con 0x7fdb3813b480 2024-03-27T21:40:35.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.761+0000 7fdb3d69d640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:35.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.761+0000 7fdb3d69d640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb38142340 con 0x7fdb380b40b0 2024-03-27T21:40:35.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.761+0000 7fdb3d69d640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:35.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.761+0000 7fdb3d69d640 10 monclient(hunting): authenticate will time out at 876.982811s 2024-03-27T21:40:35.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb367fc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdb3813b020 0x7fdb381416f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60000/0 (socket says 172.21.15.60:60000) 2024-03-27T21:40:35.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb367fc640 1 -- 172.21.15.60:0/23460439 learned_addr learned my addr 172.21.15.60:0/23460439 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:40:35.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb377fe640 1 --2- 172.21.15.60:0/23460439 >> v2:172.21.15.60:3300/0 conn(0x7fdb3813b480 0x7fdb38141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:35.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb377fe640 10 monclient(hunting): get_auth_request con 0x7fdb3813b480 auth_method 0 2024-03-27T21:40:35.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:35.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb377fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:35.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:35.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:35.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:35.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:35.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb36ffd640 1 --2- 172.21.15.60:0/23460439 >> v2:172.21.15.191:3300/0 conn(0x7fdb380b40b0 0x7fdb380a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:35.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb36ffd640 10 monclient(hunting): get_auth_request con 0x7fdb380b40b0 auth_method 0 2024-03-27T21:40:35.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:35.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:35.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:35.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb377fe640 10 monclient(hunting): handle_auth_done global_id 4689 payload 274 2024-03-27T21:40:35.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb377fe640 1 -- 172.21.15.60:0/23460439 >> v2:172.21.15.191:3300/0 conn(0x7fdb380b40b0 msgr2=0x7fdb380a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:35.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb377fe640 1 --2- 172.21.15.60:0/23460439 >> v2:172.21.15.191:3300/0 conn(0x7fdb380b40b0 0x7fdb380a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:35.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb377fe640 1 -- 172.21.15.60:0/23460439 >> v1:172.21.15.60:6789/0 conn(0x7fdb3813b020 legacy=0x7fdb381416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:35.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb377fe640 10 monclient: _finish_hunting 0 2024-03-27T21:40:35.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb377fe640 1 monclient: found mon.noname-a 2024-03-27T21:40:35.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:35.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb377fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:35.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.762+0000 7fdb377fe640 1 -- 172.21.15.60:0/23460439 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb380b3830 con 0x7fdb3813b480 2024-03-27T21:40:35.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.763+0000 7fdb377fe640 1 --2- 172.21.15.60:0/23460439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb3813b480 0x7fdb38141e00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdb380b3a20 tx=0x7fdb3002be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=569da060b2a35e7c server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:35.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.763+0000 7fdb35ffb640 1 -- 172.21.15.60:0/23460439 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3002cbe0 con 0x7fdb3813b480 2024-03-27T21:40:35.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.763+0000 7fdb35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:35.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.763+0000 7fdb35ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:35.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.763+0000 7fdb35ffb640 10 monclient: dump: 2024-03-27T21:40:35.809 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:35.809 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:35.809 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:35.809 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:35.809 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:35.809 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:35.810 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:35.810 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:35.810 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:35.810 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:35.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.763+0000 7fdb35ffb640 10 monclient: _finish_auth 0 2024-03-27T21:40:35.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.763+0000 7fdb35ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:40:35.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.763+0000 7fdb35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:35.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.763+0000 7fdb35ffb640 1 -- 172.21.15.60:0/23460439 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3002cd40 con 0x7fdb3813b480 2024-03-27T21:40:35.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.763+0000 7fdb3d69d640 5 monclient: authenticate success, global_id 4689 2024-03-27T21:40:35.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.763+0000 7fdb3d69d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:35.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.763+0000 7fdb35ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:35.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.763+0000 7fdb3d69d640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:35.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.763+0000 7fdb3d69d640 4 set_mon_vals no callback set 2024-03-27T21:40:35.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.763+0000 7fdb35ffb640 1 -- 172.21.15.60:0/23460439 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb30034660 con 0x7fdb3813b480 2024-03-27T21:40:35.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.763+0000 7fdb35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:35.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.763+0000 7fdb35ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:35.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.763+0000 7fdb35ffb640 10 monclient: dump: 2024-03-27T21:40:35.811 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:35.811 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:35.811 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:35.811 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:35.811 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:35.811 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:35.811 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:35.812 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:35.812 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:35.812 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:35.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.764+0000 7fdb3d69d640 10 monclient: shutdown 2024-03-27T21:40:35.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.764+0000 7fdb3d69d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:35.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.764+0000 7fdb3d69d640 1 -- 172.21.15.60:0/23460439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb3813b480 msgr2=0x7fdb38141e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:35.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.764+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/23460439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb3813b480 0x7fdb38141e00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdb380b3a20 tx=0x7fdb3002be30 comp rx=0 tx=0).stop 2024-03-27T21:40:35.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.764+0000 7fdb3d69d640 1 -- 172.21.15.60:0/23460439 shutdown_connections 2024-03-27T21:40:35.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.764+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/23460439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb3813b480 0x7fdb38141e00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:35.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.764+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/23460439 >> v2:172.21.15.191:3300/0 conn(0x7fdb380b40b0 0x7fdb380a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:35.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.764+0000 7fdb3d69d640 1 -- 172.21.15.60:0/23460439 >> 172.21.15.60:0/23460439 conn(0x7fdb380608e0 msgr2=0x7fdb38060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:35.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.764+0000 7fdb3d69d640 1 -- 172.21.15.60:0/23460439 shutdown_connections 2024-03-27T21:40:35.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.764+0000 7fdb3d69d640 1 -- 172.21.15.60:0/23460439 wait complete. 2024-03-27T21:40:35.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.765+0000 7fdb3d69d640 10 monclient: build_initial_monmap 2024-03-27T21:40:35.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.765+0000 7fdb3d69d640 10 monclient: monmap: 2024-03-27T21:40:35.813 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:35.813 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:35.813 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:35.814 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:35.814 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:35.814 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:35.814 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:35.814 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:35.814 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:35.814 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:35.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.766+0000 7fdb3d69d640 1 librados: starting msgr at 2024-03-27T21:40:35.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.766+0000 7fdb3d69d640 1 librados: starting objecter 2024-03-27T21:40:35.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.766+0000 7fdb3d69d640 1 Processor -- start 2024-03-27T21:40:35.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.766+0000 7fdb3d69d640 1 -- start start 2024-03-27T21:40:35.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.766+0000 7fdb3d69d640 1 librados: setting wanted keys 2024-03-27T21:40:35.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.766+0000 7fdb3d69d640 1 librados: calling monclient init 2024-03-27T21:40:35.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.766+0000 7fdb3d69d640 10 monclient: init 2024-03-27T21:40:35.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.767+0000 7fdb3d69d640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:35.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.767+0000 7fdb3d69d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:40:35.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.767+0000 7fdb3d69d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb380b40b0 0x7fdb3813df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:35.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.767+0000 7fdb3d69d640 10 monclient(hunting): picked mon.noname-a con 0x7fdb380b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:35.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.767+0000 7fdb3d69d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb3813b020 0x7fdb380aac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:35.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.767+0000 7fdb3d69d640 10 monclient(hunting): picked mon.noname-b con 0x7fdb3813b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:35.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.767+0000 7fdb36ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb380b40b0 0x7fdb3813df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:35.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.767+0000 7fdb3d69d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdb3813b480 0x7fdb381da090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:35.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.767+0000 7fdb3d69d640 10 monclient(hunting): picked mon.noname-c con 0x7fdb3813b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:35.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.767+0000 7fdb3d69d640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:35.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.767+0000 7fdb3d69d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3813d430 con 0x7fdb380b40b0 2024-03-27T21:40:35.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.767+0000 7fdb3d69d640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:35.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.767+0000 7fdb3d69d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb381d9b70 con 0x7fdb3813b020 2024-03-27T21:40:35.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.767+0000 7fdb3d69d640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:35.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.767+0000 7fdb3d69d640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3813ba60 con 0x7fdb3813b480 2024-03-27T21:40:35.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.767+0000 7fdb3d69d640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:35.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.767+0000 7fdb36ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb380b40b0 0x7fdb3813df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58252/0 (socket says 172.21.15.60:58252) 2024-03-27T21:40:35.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.767+0000 7fdb36ffd640 1 -- 172.21.15.60:0/448764475 learned_addr learned my addr 172.21.15.60:0/448764475 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:35.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.767+0000 7fdb3d69d640 10 monclient(hunting): authenticate will time out at 876.988868s 2024-03-27T21:40:35.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb367fc640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb3813b020 0x7fdb380aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:35.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb367fc640 10 monclient(hunting): get_auth_request con 0x7fdb3813b020 auth_method 0 2024-03-27T21:40:35.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:35.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb367fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:35.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:35.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:35.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:35.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:35.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb377fe640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdb3813b480 0x7fdb381da090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:35.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb377fe640 10 monclient(hunting): get_auth_request con 0x7fdb3813b480 auth_method 0 2024-03-27T21:40:35.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:35.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb377fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:35.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:35.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb36ffd640 10 monclient(hunting): get_auth_request con 0x7fdb380b40b0 auth_method 0 2024-03-27T21:40:35.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:35.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:35.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:35.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:35.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:35.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:35.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:35.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:35.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:35.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb367fc640 10 monclient(hunting): handle_auth_done global_id 4633 payload 824 2024-03-27T21:40:35.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb367fc640 1 -- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdb3813b480 msgr2=0x7fdb381da090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:35.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb367fc640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdb3813b480 0x7fdb381da090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:35.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb367fc640 1 -- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb380b40b0 msgr2=0x7fdb3813df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:35.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb367fc640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb380b40b0 0x7fdb3813df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:35.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb367fc640 10 monclient: _finish_hunting 0 2024-03-27T21:40:35.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb367fc640 1 monclient: found mon.noname-b 2024-03-27T21:40:35.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:35.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb367fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:40:35.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb367fc640 1 -- 172.21.15.60:0/448764475 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb30009a70 con 0x7fdb3813b020 2024-03-27T21:40:35.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb36ffd640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb380b40b0 0x7fdb3813df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:35.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.768+0000 7fdb377fe640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdb3813b480 0x7fdb381da090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:35.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.769+0000 7fdb367fc640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb3813b020 0x7fdb380aac40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c00c440 tx=0x7fdb2c001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:35.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.769+0000 7fdb1f7fe640 10 client.?.objecter ms_handle_connect 0x7fdb3813b020 2024-03-27T21:40:35.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.769+0000 7fdb1f7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:35.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.769+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c005ce0 con 0x7fdb3813b020 2024-03-27T21:40:35.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.769+0000 7fdb1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:35.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.769+0000 7fdb1f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:35.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.769+0000 7fdb1f7fe640 10 monclient: dump: 2024-03-27T21:40:35.825 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:35.826 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:35.826 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:35.826 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:35.826 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:35.826 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:35.827 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:35.827 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:35.827 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:35.827 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:35.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.769+0000 7fdb1f7fe640 10 monclient: _finish_auth 0 2024-03-27T21:40:35.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.769+0000 7fdb1f7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:40:35.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.769+0000 7fdb1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:35.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.769+0000 7fdb3d69d640 5 monclient: authenticate success, global_id 4633 2024-03-27T21:40:35.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.769+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb2c005e40 con 0x7fdb3813b020 2024-03-27T21:40:35.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.769+0000 7fdb1f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:35.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.769+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c0072d0 con 0x7fdb3813b020 2024-03-27T21:40:35.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.769+0000 7fdb1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:35.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.769+0000 7fdb1f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:40:35.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.769+0000 7fdb1f7fe640 10 monclient: dump: 2024-03-27T21:40:35.829 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:35.829 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:35.829 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:35.829 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:35.830 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:35.830 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:35.830 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:35.830 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:35.830 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:35.830 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:35.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.769+0000 7fdb357fa640 4 set_mon_vals no callback set 2024-03-27T21:40:35.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.770+0000 7fdb3d69d640 10 monclient: _renew_subs 2024-03-27T21:40:35.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.770+0000 7fdb3d69d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:35.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.770+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb380ab180 con 0x7fdb3813b020 2024-03-27T21:40:35.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.770+0000 7fdb3d69d640 10 client.4633.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:35.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.770+0000 7fdb3d69d640 10 monclient: _renew_subs 2024-03-27T21:40:35.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.770+0000 7fdb3d69d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:35.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.770+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb381da5d0 con 0x7fdb3813b020 2024-03-27T21:40:35.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.770+0000 7fdb3d69d640 1 librados: init done 2024-03-27T21:40:35.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.771+0000 7fdb3d69d640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:35.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.771+0000 7fdb3d69d640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:35.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.771+0000 7fdb3d69d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:35.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.771+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fdb381d9d30 con 0x7fdb3813b020 2024-03-27T21:40:35.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.772+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fdb2c0074f0 con 0x7fdb3813b020 2024-03-27T21:40:35.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.772+0000 7fdb1f7fe640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdb2406c590 0x7fdb2406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:35.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.772+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fdb2c088580 con 0x7fdb3813b020 2024-03-27T21:40:35.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.772+0000 7fdb1f7fe640 10 client.4633.objecter ms_dispatch 0x7fdb38000b90 osd_map(40..40 src has 1..40) v4 2024-03-27T21:40:35.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.772+0000 7fdb1f7fe640 3 client.4633.objecter handle_osd_map got epochs [40,40] > 0 2024-03-27T21:40:35.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.772+0000 7fdb1f7fe640 3 client.4633.objecter handle_osd_map decoding full epoch 40 2024-03-27T21:40:35.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb1f7fe640 20 client.4633.objecter dump_active .. 0 homeless 2024-03-27T21:40:35.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v40) v1 ==== 54+0+16 (secure 0 0 0) 0x7fdb2c08f070 con 0x7fdb3813b020 2024-03-27T21:40:35.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb1f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:35.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb1f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:35.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb36ffd640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdb2406c590 0x7fdb2406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:35.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb36ffd640 10 monclient: get_auth_request con 0x7fdb2406c590 auth_method 0 2024-03-27T21:40:35.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb3d69d640 10 client.4633.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:35.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign -1 1 2024-03-27T21:40:35.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fdb38134810 0x7fdb381d6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:35.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb380a90d0 osd=0 3 2024-03-27T21:40:35.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb380a90d0 osd=0 3 2024-03-27T21:40:35.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb380a90d0 osd=0 3 2024-03-27T21:40:35.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_remove -1 1 2024-03-27T21:40:35.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb3d69d640 20 client.4633.objecter get_session s=0x7fdb380a90d0 osd=0 3 2024-03-27T21:40:35.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign 0 1 2024-03-27T21:40:35.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb380a90d0 osd=0 4 2024-03-27T21:40:35.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb3d69d640 10 client.4633.objecter _send_command 1 2024-03-27T21:40:35.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb38069470 con 0x7fdb38134810 2024-03-27T21:40:35.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb377fe640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fdb38134810 0x7fdb381d6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:35.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb377fe640 10 monclient: get_auth_request con 0x7fdb38134810 auth_method 0 2024-03-27T21:40:35.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb36ffd640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdb2406c590 0x7fdb2406ea50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdb28006fd0 tx=0x7fdb28008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:35.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb1f7fe640 10 client.4633.objecter ms_handle_connect 0x7fdb2406c590 2024-03-27T21:40:35.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb377fe640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fdb38134810 0x7fdb381d6560 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:35.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.773+0000 7fdb1f7fe640 10 client.4633.objecter ms_handle_connect 0x7fdb38134810 2024-03-27T21:40:35.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.774+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fdb38069470 con 0x7fdb38134810 2024-03-27T21:40:35.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.774+0000 7fdb1f7fe640 10 client.4633.objecter ms_dispatch 0x7fdb38000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:35.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.774+0000 7fdb1f7fe640 10 client.4633.objecter _finish_command 1 = system:0 2024-03-27T21:40:35.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.774+0000 7fdb1f7fe640 20 client.4633.objecter put_session s=0x7fdb380a90d0 osd=0 4 2024-03-27T21:40:35.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.774+0000 7fdb1f7fe640 15 client.4633.objecter _session_command_op_remove 0 1 2024-03-27T21:40:35.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.803+0000 7fdb3d69d640 10 client.4633.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:35.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.803+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign -1 2 2024-03-27T21:40:35.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.803+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb380a90d0 osd=0 3 2024-03-27T21:40:35.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.803+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb380a90d0 osd=0 3 2024-03-27T21:40:35.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.803+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb380a90d0 osd=0 3 2024-03-27T21:40:35.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.803+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_remove -1 2 2024-03-27T21:40:35.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.803+0000 7fdb3d69d640 20 client.4633.objecter get_session s=0x7fdb380a90d0 osd=0 3 2024-03-27T21:40:35.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.803+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign 0 2 2024-03-27T21:40:35.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.803+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb380a90d0 osd=0 4 2024-03-27T21:40:35.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.803+0000 7fdb3d69d640 10 client.4633.objecter _send_command 2 2024-03-27T21:40:35.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.803+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdb380a87d0 con 0x7fdb38134810 2024-03-27T21:40:35.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.804+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdb380a87d0 con 0x7fdb38134810 2024-03-27T21:40:35.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.804+0000 7fdb1f7fe640 10 client.4633.objecter ms_dispatch 0x7fdb38000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:35.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.804+0000 7fdb1f7fe640 10 client.4633.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:35.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.804+0000 7fdb1f7fe640 20 client.4633.objecter put_session s=0x7fdb380a90d0 osd=0 4 2024-03-27T21:40:35.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.804+0000 7fdb1f7fe640 15 client.4633.objecter _session_command_op_remove 0 2 2024-03-27T21:40:35.838 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:35.839 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:35.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.805+0000 7fdb3d69d640 10 client.4633.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:35.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.805+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign -1 3 2024-03-27T21:40:35.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.805+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fdb381d6fb0 0x7fdb38065a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:35.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.805+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb380b0e90 osd=1 3 2024-03-27T21:40:35.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.805+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb380b0e90 osd=1 3 2024-03-27T21:40:35.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.805+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb380b0e90 osd=1 3 2024-03-27T21:40:35.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.805+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_remove -1 3 2024-03-27T21:40:35.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.805+0000 7fdb3d69d640 20 client.4633.objecter get_session s=0x7fdb380b0e90 osd=1 3 2024-03-27T21:40:35.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.805+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign 1 3 2024-03-27T21:40:35.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.805+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb380b0e90 osd=1 4 2024-03-27T21:40:35.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.805+0000 7fdb3d69d640 10 client.4633.objecter _send_command 3 2024-03-27T21:40:35.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.805+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb381d7390 con 0x7fdb381d6fb0 2024-03-27T21:40:35.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.805+0000 7fdb36ffd640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fdb381d6fb0 0x7fdb38065a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:35.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.805+0000 7fdb36ffd640 10 monclient: get_auth_request con 0x7fdb381d6fb0 auth_method 0 2024-03-27T21:40:35.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.805+0000 7fdb1f7fe640 10 client.4633.objecter ms_handle_connect 0x7fdb381d6fb0 2024-03-27T21:40:35.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.805+0000 7fdb36ffd640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fdb381d6fb0 0x7fdb38065a90 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:35.841 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:35.806+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:35.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.806+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fdb381d7390 con 0x7fdb381d6fb0 2024-03-27T21:40:35.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.806+0000 7fdb1f7fe640 10 client.4633.objecter ms_dispatch 0x7fdb38000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:35.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.806+0000 7fdb1f7fe640 10 client.4633.objecter _finish_command 3 = system:0 2024-03-27T21:40:35.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.806+0000 7fdb1f7fe640 20 client.4633.objecter put_session s=0x7fdb380b0e90 osd=1 4 2024-03-27T21:40:35.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.806+0000 7fdb1f7fe640 15 client.4633.objecter _session_command_op_remove 1 3 2024-03-27T21:40:35.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.821+0000 7fdb3d69d640 10 client.4633.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:35.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.821+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign -1 4 2024-03-27T21:40:35.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.821+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb380b0e90 osd=1 3 2024-03-27T21:40:35.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.821+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb380b0e90 osd=1 3 2024-03-27T21:40:35.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.821+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb380b0e90 osd=1 3 2024-03-27T21:40:35.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.821+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_remove -1 4 2024-03-27T21:40:35.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.821+0000 7fdb3d69d640 20 client.4633.objecter get_session s=0x7fdb380b0e90 osd=1 3 2024-03-27T21:40:35.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.821+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign 1 4 2024-03-27T21:40:35.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.821+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb380b0e90 osd=1 4 2024-03-27T21:40:35.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.821+0000 7fdb3d69d640 10 client.4633.objecter _send_command 4 2024-03-27T21:40:35.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.821+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdb38065fd0 con 0x7fdb381d6fb0 2024-03-27T21:40:35.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.823+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdb38065fd0 con 0x7fdb381d6fb0 2024-03-27T21:40:35.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.823+0000 7fdb1f7fe640 10 client.4633.objecter ms_dispatch 0x7fdb38000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:35.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.823+0000 7fdb1f7fe640 10 client.4633.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:35.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.823+0000 7fdb1f7fe640 20 client.4633.objecter put_session s=0x7fdb380b0e90 osd=1 4 2024-03-27T21:40:35.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.823+0000 7fdb1f7fe640 15 client.4633.objecter _session_command_op_remove 1 4 2024-03-27T21:40:35.844 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:35.847 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:35.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.823+0000 7fdb3d69d640 10 client.4633.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:35.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.824+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign -1 5 2024-03-27T21:40:35.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.824+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7fdb3802b170 0x7fdb3802b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:35.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.824+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb38066960 osd=2 3 2024-03-27T21:40:35.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.824+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb38066960 osd=2 3 2024-03-27T21:40:35.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.824+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb38066960 osd=2 3 2024-03-27T21:40:35.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.824+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_remove -1 5 2024-03-27T21:40:35.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.824+0000 7fdb3d69d640 20 client.4633.objecter get_session s=0x7fdb38066960 osd=2 3 2024-03-27T21:40:35.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.824+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign 2 5 2024-03-27T21:40:35.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.824+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb38066960 osd=2 4 2024-03-27T21:40:35.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.824+0000 7fdb3d69d640 10 client.4633.objecter _send_command 5 2024-03-27T21:40:35.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.824+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb3802bad0 con 0x7fdb3802b170 2024-03-27T21:40:35.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.824+0000 7fdb367fc640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7fdb3802b170 0x7fdb3802b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:35.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.824+0000 7fdb367fc640 10 monclient: get_auth_request con 0x7fdb3802b170 auth_method 0 2024-03-27T21:40:35.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.824+0000 7fdb1f7fe640 10 client.4633.objecter ms_handle_connect 0x7fdb3802b170 2024-03-27T21:40:35.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.824+0000 7fdb367fc640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7fdb3802b170 0x7fdb3802b590 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:35.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.825+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== osd.2 v2:172.21.15.60:6824/44654 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fdb3802bad0 con 0x7fdb3802b170 2024-03-27T21:40:35.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.825+0000 7fdb1f7fe640 10 client.4633.objecter ms_dispatch 0x7fdb38000b90 command_reply(tid 5: 0 ) v1 2024-03-27T21:40:35.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.825+0000 7fdb1f7fe640 10 client.4633.objecter _finish_command 5 = system:0 2024-03-27T21:40:35.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.825+0000 7fdb1f7fe640 20 client.4633.objecter put_session s=0x7fdb38066960 osd=2 4 2024-03-27T21:40:35.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.825+0000 7fdb1f7fe640 15 client.4633.objecter _session_command_op_remove 2 5 2024-03-27T21:40:35.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.838+0000 7fdb3d69d640 10 client.4633.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:35.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.839+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign -1 6 2024-03-27T21:40:35.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.839+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb38066960 osd=2 3 2024-03-27T21:40:35.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.839+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb38066960 osd=2 3 2024-03-27T21:40:35.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.839+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb38066960 osd=2 3 2024-03-27T21:40:35.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.839+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_remove -1 6 2024-03-27T21:40:35.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.839+0000 7fdb3d69d640 20 client.4633.objecter get_session s=0x7fdb38066960 osd=2 3 2024-03-27T21:40:35.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.839+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign 2 6 2024-03-27T21:40:35.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.839+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb38066960 osd=2 4 2024-03-27T21:40:35.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.839+0000 7fdb3d69d640 10 client.4633.objecter _send_command 6 2024-03-27T21:40:35.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.839+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdb3802c490 con 0x7fdb3802b170 2024-03-27T21:40:35.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.841+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== osd.2 v2:172.21.15.60:6824/44654 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdb3802c490 con 0x7fdb3802b170 2024-03-27T21:40:35.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.841+0000 7fdb1f7fe640 10 client.4633.objecter ms_dispatch 0x7fdb38000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:35.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.841+0000 7fdb1f7fe640 10 client.4633.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:35.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.841+0000 7fdb1f7fe640 20 client.4633.objecter put_session s=0x7fdb38066960 osd=2 4 2024-03-27T21:40:35.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.841+0000 7fdb1f7fe640 15 client.4633.objecter _session_command_op_remove 2 6 2024-03-27T21:40:35.851 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2: {} 2024-03-27T21:40:35.852 INFO:teuthology.orchestra.run.smithi060.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-27T21:40:35.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.841+0000 7fdb3d69d640 10 client.4633.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:35.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.841+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign -1 7 2024-03-27T21:40:35.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.841+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fdb380664f0 0x7fdb381e9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:35.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.841+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb38066380 osd=3 3 2024-03-27T21:40:35.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.841+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb38066380 osd=3 3 2024-03-27T21:40:35.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.841+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb38066380 osd=3 3 2024-03-27T21:40:35.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.841+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_remove -1 7 2024-03-27T21:40:35.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.841+0000 7fdb3d69d640 20 client.4633.objecter get_session s=0x7fdb38066380 osd=3 3 2024-03-27T21:40:35.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.841+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign 3 7 2024-03-27T21:40:35.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.841+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb38066380 osd=3 4 2024-03-27T21:40:35.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.841+0000 7fdb3d69d640 10 client.4633.objecter _send_command 7 2024-03-27T21:40:35.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.841+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb381ec110 con 0x7fdb380664f0 2024-03-27T21:40:35.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.841+0000 7fdb377fe640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fdb380664f0 0x7fdb381e9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:35.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.841+0000 7fdb377fe640 10 monclient: get_auth_request con 0x7fdb380664f0 auth_method 0 2024-03-27T21:40:35.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.842+0000 7fdb1f7fe640 10 client.4633.objecter ms_handle_connect 0x7fdb380664f0 2024-03-27T21:40:35.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.842+0000 7fdb377fe640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fdb380664f0 0x7fdb381e9340 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:35.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.842+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fdb381ec110 con 0x7fdb380664f0 2024-03-27T21:40:35.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.842+0000 7fdb1f7fe640 10 client.4633.objecter ms_dispatch 0x7fdb38000b90 command_reply(tid 7: 0 ) v1 2024-03-27T21:40:35.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.842+0000 7fdb1f7fe640 10 client.4633.objecter _finish_command 7 = system:0 2024-03-27T21:40:35.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.842+0000 7fdb1f7fe640 20 client.4633.objecter put_session s=0x7fdb38066380 osd=3 4 2024-03-27T21:40:35.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.842+0000 7fdb1f7fe640 15 client.4633.objecter _session_command_op_remove 3 7 2024-03-27T21:40:35.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.856+0000 7fdb3d69d640 10 client.4633.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:35.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.856+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign -1 8 2024-03-27T21:40:35.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.856+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb38066380 osd=3 3 2024-03-27T21:40:35.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.856+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb38066380 osd=3 3 2024-03-27T21:40:35.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.856+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb38066380 osd=3 3 2024-03-27T21:40:35.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.856+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_remove -1 8 2024-03-27T21:40:35.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.856+0000 7fdb3d69d640 20 client.4633.objecter get_session s=0x7fdb38066380 osd=3 3 2024-03-27T21:40:35.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.856+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign 3 8 2024-03-27T21:40:35.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.856+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb38066380 osd=3 4 2024-03-27T21:40:35.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.856+0000 7fdb3d69d640 10 client.4633.objecter _send_command 8 2024-03-27T21:40:35.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.856+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdb3802bf10 con 0x7fdb380664f0 2024-03-27T21:40:35.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.859+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdb3802bf10 con 0x7fdb380664f0 2024-03-27T21:40:35.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.859+0000 7fdb1f7fe640 10 client.4633.objecter ms_dispatch 0x7fdb38000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:35.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.859+0000 7fdb1f7fe640 10 client.4633.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:35.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.859+0000 7fdb1f7fe640 20 client.4633.objecter put_session s=0x7fdb38066380 osd=3 4 2024-03-27T21:40:35.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.859+0000 7fdb1f7fe640 15 client.4633.objecter _session_command_op_remove 3 8 2024-03-27T21:40:35.856 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:35.857 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:35.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.859+0000 7fdb3d69d640 10 client.4633.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:35.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.859+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign -1 9 2024-03-27T21:40:35.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.859+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fdb3802c080 0x7fdb381eaa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:35.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.859+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb3802bf10 osd=4 3 2024-03-27T21:40:35.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.859+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb3802bf10 osd=4 3 2024-03-27T21:40:35.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.859+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb3802bf10 osd=4 3 2024-03-27T21:40:35.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.859+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_remove -1 9 2024-03-27T21:40:35.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.859+0000 7fdb3d69d640 20 client.4633.objecter get_session s=0x7fdb3802bf10 osd=4 3 2024-03-27T21:40:35.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.859+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign 4 9 2024-03-27T21:40:35.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.859+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb3802bf10 osd=4 4 2024-03-27T21:40:35.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.859+0000 7fdb3d69d640 10 client.4633.objecter _send_command 9 2024-03-27T21:40:35.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.859+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb38139380 con 0x7fdb3802c080 2024-03-27T21:40:35.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.859+0000 7fdb36ffd640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fdb3802c080 0x7fdb381eaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:35.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.859+0000 7fdb36ffd640 10 monclient: get_auth_request con 0x7fdb3802c080 auth_method 0 2024-03-27T21:40:35.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.860+0000 7fdb1f7fe640 10 client.4633.objecter ms_handle_connect 0x7fdb3802c080 2024-03-27T21:40:35.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.860+0000 7fdb36ffd640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fdb3802c080 0x7fdb381eaa00 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:35.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.861+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fdb38139380 con 0x7fdb3802c080 2024-03-27T21:40:35.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.861+0000 7fdb1f7fe640 10 client.4633.objecter ms_dispatch 0x7fdb38000b90 command_reply(tid 9: 0 ) v1 2024-03-27T21:40:35.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.861+0000 7fdb1f7fe640 10 client.4633.objecter _finish_command 9 = system:0 2024-03-27T21:40:35.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.861+0000 7fdb1f7fe640 20 client.4633.objecter put_session s=0x7fdb3802bf10 osd=4 4 2024-03-27T21:40:35.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.861+0000 7fdb1f7fe640 15 client.4633.objecter _session_command_op_remove 4 9 2024-03-27T21:40:35.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.889+0000 7fdb3d69d640 10 client.4633.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:35.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.889+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign -1 10 2024-03-27T21:40:35.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.889+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb3802bf10 osd=4 3 2024-03-27T21:40:35.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.889+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb3802bf10 osd=4 3 2024-03-27T21:40:35.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.889+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb3802bf10 osd=4 3 2024-03-27T21:40:35.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.889+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_remove -1 10 2024-03-27T21:40:35.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.889+0000 7fdb3d69d640 20 client.4633.objecter get_session s=0x7fdb3802bf10 osd=4 3 2024-03-27T21:40:35.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.889+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign 4 10 2024-03-27T21:40:35.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.889+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb3802bf10 osd=4 4 2024-03-27T21:40:35.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.889+0000 7fdb3d69d640 10 client.4633.objecter _send_command 10 2024-03-27T21:40:35.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.889+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdb38139380 con 0x7fdb3802c080 2024-03-27T21:40:35.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.891+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdb38139380 con 0x7fdb3802c080 2024-03-27T21:40:35.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.891+0000 7fdb1f7fe640 10 client.4633.objecter ms_dispatch 0x7fdb38000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:35.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.891+0000 7fdb1f7fe640 10 client.4633.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:35.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.891+0000 7fdb1f7fe640 20 client.4633.objecter put_session s=0x7fdb3802bf10 osd=4 4 2024-03-27T21:40:35.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.891+0000 7fdb1f7fe640 15 client.4633.objecter _session_command_op_remove 4 10 2024-03-27T21:40:35.880 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:35.884 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:35.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.891+0000 7fdb3d69d640 10 client.4633.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:35.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.891+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign -1 11 2024-03-27T21:40:35.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.891+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fdb381eb450 0x7fdb381eb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:35.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.891+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb38139380 osd=5 3 2024-03-27T21:40:35.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.891+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb38139380 osd=5 3 2024-03-27T21:40:35.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.891+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb38139380 osd=5 3 2024-03-27T21:40:35.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.891+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_remove -1 11 2024-03-27T21:40:35.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.891+0000 7fdb3d69d640 20 client.4633.objecter get_session s=0x7fdb38139380 osd=5 3 2024-03-27T21:40:35.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.891+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign 5 11 2024-03-27T21:40:35.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.891+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb38139380 osd=5 4 2024-03-27T21:40:35.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.891+0000 7fdb3d69d640 10 client.4633.objecter _send_command 11 2024-03-27T21:40:35.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.891+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb381ebdf0 con 0x7fdb381eb450 2024-03-27T21:40:35.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.892+0000 7fdb367fc640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fdb381eb450 0x7fdb381eb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:35.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.892+0000 7fdb367fc640 10 monclient: get_auth_request con 0x7fdb381eb450 auth_method 0 2024-03-27T21:40:35.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.892+0000 7fdb1f7fe640 10 client.4633.objecter ms_handle_connect 0x7fdb381eb450 2024-03-27T21:40:35.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.892+0000 7fdb367fc640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fdb381eb450 0x7fdb381eb8b0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:35.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.893+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fdb381ebdf0 con 0x7fdb381eb450 2024-03-27T21:40:35.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.893+0000 7fdb1f7fe640 10 client.4633.objecter ms_dispatch 0x7fdb38000b90 command_reply(tid 11: 0 ) v1 2024-03-27T21:40:35.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.893+0000 7fdb1f7fe640 10 client.4633.objecter _finish_command 11 = system:0 2024-03-27T21:40:35.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.893+0000 7fdb1f7fe640 20 client.4633.objecter put_session s=0x7fdb38139380 osd=5 4 2024-03-27T21:40:35.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.893+0000 7fdb1f7fe640 15 client.4633.objecter _session_command_op_remove 5 11 2024-03-27T21:40:35.893 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:35.906+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.923+0000 7fdb3d69d640 10 client.4633.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.923+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign -1 12 2024-03-27T21:40:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.923+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb38139380 osd=5 3 2024-03-27T21:40:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.923+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb38139380 osd=5 3 2024-03-27T21:40:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.923+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb38139380 osd=5 3 2024-03-27T21:40:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.923+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_remove -1 12 2024-03-27T21:40:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.923+0000 7fdb3d69d640 20 client.4633.objecter get_session s=0x7fdb38139380 osd=5 3 2024-03-27T21:40:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.923+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign 5 12 2024-03-27T21:40:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.923+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb38139380 osd=5 4 2024-03-27T21:40:35.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.923+0000 7fdb3d69d640 10 client.4633.objecter _send_command 12 2024-03-27T21:40:35.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.923+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdb381ebdf0 con 0x7fdb381eb450 2024-03-27T21:40:35.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.925+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdb381ebdf0 con 0x7fdb381eb450 2024-03-27T21:40:35.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.925+0000 7fdb1f7fe640 10 client.4633.objecter ms_dispatch 0x7fdb38000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:35.912 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:35.914 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:35.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.925+0000 7fdb1f7fe640 10 client.4633.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:35.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.925+0000 7fdb1f7fe640 20 client.4633.objecter put_session s=0x7fdb38139380 osd=5 4 2024-03-27T21:40:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.925+0000 7fdb1f7fe640 15 client.4633.objecter _session_command_op_remove 5 12 2024-03-27T21:40:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.925+0000 7fdb3d69d640 10 client.4633.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.925+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign -1 13 2024-03-27T21:40:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.925+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fdb381f07b0 0x7fdb381f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.925+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb381ebdf0 osd=6 3 2024-03-27T21:40:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.925+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb381ebdf0 osd=6 3 2024-03-27T21:40:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.925+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb381ebdf0 osd=6 3 2024-03-27T21:40:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.925+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_remove -1 13 2024-03-27T21:40:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.925+0000 7fdb3d69d640 20 client.4633.objecter get_session s=0x7fdb381ebdf0 osd=6 3 2024-03-27T21:40:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.925+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign 6 13 2024-03-27T21:40:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.925+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb381ebdf0 osd=6 4 2024-03-27T21:40:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.925+0000 7fdb3d69d640 10 client.4633.objecter _send_command 13 2024-03-27T21:40:35.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.925+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb381394f0 con 0x7fdb381f07b0 2024-03-27T21:40:35.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.925+0000 7fdb377fe640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fdb381f07b0 0x7fdb381f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:35.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.925+0000 7fdb377fe640 10 monclient: get_auth_request con 0x7fdb381f07b0 auth_method 0 2024-03-27T21:40:35.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.926+0000 7fdb1f7fe640 10 client.4633.objecter ms_handle_connect 0x7fdb381f07b0 2024-03-27T21:40:35.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.926+0000 7fdb377fe640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fdb381f07b0 0x7fdb381f2ba0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:35.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.927+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fdb381394f0 con 0x7fdb381f07b0 2024-03-27T21:40:35.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.927+0000 7fdb1f7fe640 10 client.4633.objecter ms_dispatch 0x7fdb38000b90 command_reply(tid 13: 0 ) v1 2024-03-27T21:40:35.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.927+0000 7fdb1f7fe640 10 client.4633.objecter _finish_command 13 = system:0 2024-03-27T21:40:35.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.927+0000 7fdb1f7fe640 20 client.4633.objecter put_session s=0x7fdb381ebdf0 osd=6 4 2024-03-27T21:40:35.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.927+0000 7fdb1f7fe640 15 client.4633.objecter _session_command_op_remove 6 13 2024-03-27T21:40:35.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.955+0000 7fdb3d69d640 10 client.4633.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:35.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.955+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign -1 14 2024-03-27T21:40:35.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.955+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb381ebdf0 osd=6 3 2024-03-27T21:40:35.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.955+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb381ebdf0 osd=6 3 2024-03-27T21:40:35.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.955+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb381ebdf0 osd=6 3 2024-03-27T21:40:35.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.955+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_remove -1 14 2024-03-27T21:40:35.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.955+0000 7fdb3d69d640 20 client.4633.objecter get_session s=0x7fdb381ebdf0 osd=6 3 2024-03-27T21:40:35.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.955+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign 6 14 2024-03-27T21:40:35.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.955+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb381ebdf0 osd=6 4 2024-03-27T21:40:35.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.955+0000 7fdb3d69d640 10 client.4633.objecter _send_command 14 2024-03-27T21:40:35.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.955+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdb381394f0 con 0x7fdb381f07b0 2024-03-27T21:40:35.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.957+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdb381394f0 con 0x7fdb381f07b0 2024-03-27T21:40:35.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.957+0000 7fdb1f7fe640 10 client.4633.objecter ms_dispatch 0x7fdb38000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:35.944 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:35.947 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:35.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.957+0000 7fdb1f7fe640 10 client.4633.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:35.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.957+0000 7fdb1f7fe640 20 client.4633.objecter put_session s=0x7fdb381ebdf0 osd=6 4 2024-03-27T21:40:35.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.957+0000 7fdb1f7fe640 15 client.4633.objecter _session_command_op_remove 6 14 2024-03-27T21:40:35.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.957+0000 7fdb3d69d640 10 client.4633.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:35.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.958+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign -1 15 2024-03-27T21:40:35.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.958+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fdb381f35f0 0x7fdb381f59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:35.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.958+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb381394f0 osd=7 3 2024-03-27T21:40:35.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.958+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb381394f0 osd=7 3 2024-03-27T21:40:35.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.958+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb381394f0 osd=7 3 2024-03-27T21:40:35.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.958+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_remove -1 15 2024-03-27T21:40:35.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.958+0000 7fdb3d69d640 20 client.4633.objecter get_session s=0x7fdb381394f0 osd=7 3 2024-03-27T21:40:35.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.958+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign 7 15 2024-03-27T21:40:35.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.958+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb381394f0 osd=7 4 2024-03-27T21:40:35.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.958+0000 7fdb3d69d640 10 client.4633.objecter _send_command 15 2024-03-27T21:40:35.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.958+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb38139690 con 0x7fdb381f35f0 2024-03-27T21:40:35.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.958+0000 7fdb36ffd640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fdb381f35f0 0x7fdb381f59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:35.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.958+0000 7fdb36ffd640 10 monclient: get_auth_request con 0x7fdb381f35f0 auth_method 0 2024-03-27T21:40:35.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.958+0000 7fdb1f7fe640 10 client.4633.objecter ms_handle_connect 0x7fdb381f35f0 2024-03-27T21:40:35.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.958+0000 7fdb36ffd640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fdb381f35f0 0x7fdb381f59e0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:35.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.959+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fdb38139690 con 0x7fdb381f35f0 2024-03-27T21:40:35.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.959+0000 7fdb1f7fe640 10 client.4633.objecter ms_dispatch 0x7fdb38000b90 command_reply(tid 15: 0 ) v1 2024-03-27T21:40:35.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.959+0000 7fdb1f7fe640 10 client.4633.objecter _finish_command 15 = system:0 2024-03-27T21:40:35.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.959+0000 7fdb1f7fe640 20 client.4633.objecter put_session s=0x7fdb381394f0 osd=7 4 2024-03-27T21:40:35.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.959+0000 7fdb1f7fe640 15 client.4633.objecter _session_command_op_remove 7 15 2024-03-27T21:40:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.988+0000 7fdb3d69d640 10 client.4633.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.988+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign -1 16 2024-03-27T21:40:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.988+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb381394f0 osd=7 3 2024-03-27T21:40:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.988+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb381394f0 osd=7 3 2024-03-27T21:40:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.988+0000 7fdb3d69d640 20 client.4633.objecter _get_session s=0x7fdb381394f0 osd=7 3 2024-03-27T21:40:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.988+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_remove -1 16 2024-03-27T21:40:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.988+0000 7fdb3d69d640 20 client.4633.objecter get_session s=0x7fdb381394f0 osd=7 3 2024-03-27T21:40:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.988+0000 7fdb3d69d640 15 client.4633.objecter _session_command_op_assign 7 16 2024-03-27T21:40:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.988+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb381394f0 osd=7 4 2024-03-27T21:40:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.988+0000 7fdb3d69d640 10 client.4633.objecter _send_command 16 2024-03-27T21:40:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.988+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdb38139690 con 0x7fdb381f35f0 2024-03-27T21:40:35.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.990+0000 7fdb1f7fe640 1 -- 172.21.15.60:0/448764475 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdb38139690 con 0x7fdb381f35f0 2024-03-27T21:40:35.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.990+0000 7fdb1f7fe640 10 client.4633.objecter ms_dispatch 0x7fdb38000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:35.977 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:35.982 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:35.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.990+0000 7fdb1f7fe640 10 client.4633.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:35.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.990+0000 7fdb1f7fe640 20 client.4633.objecter put_session s=0x7fdb381394f0 osd=7 4 2024-03-27T21:40:35.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.990+0000 7fdb1f7fe640 15 client.4633.objecter _session_command_op_remove 7 16 2024-03-27T21:40:35.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.990+0000 7fdb3d69d640 10 librados: watch_flush enter 2024-03-27T21:40:35.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.990+0000 7fdb3d69d640 10 librados: watch_flush exit 2024-03-27T21:40:35.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.990+0000 7fdb3d69d640 10 client.4633.objecter close_session for osd.0 2024-03-27T21:40:35.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.991+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fdb38134810 msgr2=0x7fdb381d6560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:35.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.991+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fdb38134810 0x7fdb381d6560 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:35.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.991+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb380a90d0 osd=0 1 2024-03-27T21:40:35.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.991+0000 7fdb3d69d640 10 client.4633.objecter close_session for osd.1 2024-03-27T21:40:35.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.991+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fdb381d6fb0 msgr2=0x7fdb38065a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:35.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.991+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fdb381d6fb0 0x7fdb38065a90 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:35.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.991+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb380b0e90 osd=1 1 2024-03-27T21:40:35.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.991+0000 7fdb3d69d640 10 client.4633.objecter close_session for osd.2 2024-03-27T21:40:35.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.991+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7fdb3802b170 msgr2=0x7fdb3802b590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:35.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.991+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7fdb3802b170 0x7fdb3802b590 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:35.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.991+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb38066960 osd=2 1 2024-03-27T21:40:35.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.991+0000 7fdb377fe640 1 -- 172.21.15.60:0/448764475 reap_dead start 2024-03-27T21:40:35.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.991+0000 7fdb3d69d640 10 client.4633.objecter close_session for osd.3 2024-03-27T21:40:35.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.991+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fdb380664f0 msgr2=0x7fdb381e9340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:35.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.991+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fdb380664f0 0x7fdb381e9340 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:35.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.991+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb38066380 osd=3 1 2024-03-27T21:40:35.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.991+0000 7fdb3d69d640 10 client.4633.objecter close_session for osd.4 2024-03-27T21:40:35.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.991+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fdb3802c080 msgr2=0x7fdb381eaa00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:35.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.992+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fdb3802c080 0x7fdb381eaa00 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:35.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.992+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb3802bf10 osd=4 1 2024-03-27T21:40:35.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.992+0000 7fdb3d69d640 10 client.4633.objecter close_session for osd.5 2024-03-27T21:40:35.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.992+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fdb381eb450 msgr2=0x7fdb381eb8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:35.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.992+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fdb381eb450 0x7fdb381eb8b0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:35.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.992+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb38139380 osd=5 1 2024-03-27T21:40:35.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.992+0000 7fdb3d69d640 10 client.4633.objecter close_session for osd.6 2024-03-27T21:40:35.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.992+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fdb381f07b0 msgr2=0x7fdb381f2ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:35.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.992+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fdb381f07b0 0x7fdb381f2ba0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:35.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.992+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb381ebdf0 osd=6 1 2024-03-27T21:40:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.992+0000 7fdb3d69d640 10 client.4633.objecter close_session for osd.7 2024-03-27T21:40:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.992+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fdb381f35f0 msgr2=0x7fdb381f59e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.992+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fdb381f35f0 0x7fdb381f59e0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.992+0000 7fdb3d69d640 20 client.4633.objecter put_session s=0x7fdb381394f0 osd=7 1 2024-03-27T21:40:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.992+0000 7fdb377fe640 1 -- 172.21.15.60:0/448764475 reap_dead start 2024-03-27T21:40:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.992+0000 7fdb3d69d640 20 client.4633.objecter shutdown clearing up homeless session... 2024-03-27T21:40:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.992+0000 7fdb3d69d640 10 client.4633.objecter successfully canceled tick 2024-03-27T21:40:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.992+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdb2406c590 msgr2=0x7fdb2406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.993+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdb2406c590 0x7fdb2406ea50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdb28006fd0 tx=0x7fdb28008040 comp rx=0 tx=0).stop 2024-03-27T21:40:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.993+0000 7fdb3d69d640 10 monclient: shutdown 2024-03-27T21:40:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.993+0000 7fdb3d69d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.993+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb3813b020 msgr2=0x7fdb380aac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.993+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb3813b020 0x7fdb380aac40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c00c440 tx=0x7fdb2c001e70 comp rx=0 tx=0).stop 2024-03-27T21:40:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.993+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 shutdown_connections 2024-03-27T21:40:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.993+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdb2406c590 0x7fdb2406ea50 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:35.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.993+0000 7fdb3d69d640 1 --2- 172.21.15.60:0/448764475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb3813b020 0x7fdb380aac40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:35.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.993+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 >> 172.21.15.60:0/448764475 conn(0x7fdb380608e0 msgr2=0x7fdb380b60e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:35.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.993+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 shutdown_connections 2024-03-27T21:40:35.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.993+0000 7fdb3d69d640 1 -- 172.21.15.60:0/448764475 wait complete. 2024-03-27T21:40:35.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:35.993+0000 7fdb3d69d640 1 librados: shutdown 2024-03-27T21:40:35.993 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:36.006+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:35.994 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:35.994 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:36.007 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:40:36.094 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:36.108+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:36.157 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:36.157 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:36.166 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:40:36.207 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:36.221+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:36.295 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:36.309+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:36.315 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:36.315 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:36.325 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:40:36.396 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:36.410+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:36.497 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:36.511+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:36.531 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:36.531 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:36.544 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:40:36.628 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:36.642+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:36.694 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:36.695 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:36.699 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:36.713+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:36.705 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:40:36.800 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:36.813+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:36.852 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:36.852 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:36.861 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:40:36.906 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:36.919+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:36.980 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:36.980 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:36.989 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:40:37.030 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:37.043+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:37.102 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:37.116+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:37.135 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:37.136 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:37.144 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:40:37.228 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:37.242+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:37.290 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:37.290 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:37.300 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:40:37.306 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:37.320+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:37.407 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:37.421+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:37.445 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:37.445 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:37.454 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:40:37.464 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-27T21:40:37.538 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:37.552+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:37.554 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-03-27T21:40:37.555 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T21:40:37.555 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-27T21:40:37.555 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-03-27T21:40:37.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.583+0000 7fbc76acf640 10 monclient: get_monmap_and_config 2024-03-27T21:40:37.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.583+0000 7fbc76acf640 10 monclient: build_initial_monmap 2024-03-27T21:40:37.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.583+0000 7fbc76acf640 10 monclient: monmap: 2024-03-27T21:40:37.569 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:37.569 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:37.569 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:37.584271+0000 2024-03-27T21:40:37.569 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:37.584271+0000 2024-03-27T21:40:37.570 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:37.570 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:37.570 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:37.570 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:37.570 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:37.570 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:37.570 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:37.570 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:37.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.584+0000 7fbc76acf640 1 Processor -- start 2024-03-27T21:40:37.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.584+0000 7fbc76acf640 1 -- start start 2024-03-27T21:40:37.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.584+0000 7fbc76acf640 10 monclient: init 2024-03-27T21:40:37.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.585+0000 7fbc76acf640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:37.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.585+0000 7fbc76acf640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-03-27T21:40:37.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.585+0000 7fbc76acf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc700b40b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:37.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.585+0000 7fbc76acf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc7013b020 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:37.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.585+0000 7fbc76acf640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbc7013b480 0x7fbc70141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:37.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.585+0000 7fbc7486b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbc700b40b0 0x7fbc700a8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60016/0 (socket says 172.21.15.60:60016) 2024-03-27T21:40:37.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.585+0000 7fbc7506c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbc7013b480 0x7fbc70141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:37.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.585+0000 7fbc7506c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbc7013b480 0x7fbc70141e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:59478/0 (socket says 172.21.15.60:59478) 2024-03-27T21:40:37.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.585+0000 7fbc6ffff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fbc7013b020 0x7fbc701416f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.60:50420/0 (socket says 172.21.15.60:50420) 2024-03-27T21:40:37.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc76acf640 10 monclient(hunting): picked mon.noname-b con 0x7fbc7013b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:37.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc7486b640 1 -- 172.21.15.60:0/231849231 learned_addr learned my addr 172.21.15.60:0/231849231 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:40:37.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc76acf640 1 -- 172.21.15.60:0/231849231 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc700b3830 con 0x7fbc700b40b0 2024-03-27T21:40:37.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc76acf640 1 -- 172.21.15.60:0/231849231 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc700aa520 con 0x7fbc7013b020 2024-03-27T21:40:37.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc76acf640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:37.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc76acf640 1 -- 172.21.15.60:0/231849231 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7013b910 con 0x7fbc7013b480 2024-03-27T21:40:37.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc76acf640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:37.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc76acf640 10 monclient(hunting): authenticate will time out at 878.807046s 2024-03-27T21:40:37.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc7506c640 10 monclient(hunting): get_auth_request con 0x7fbc7013b480 auth_method 0 2024-03-27T21:40:37.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc7506c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:37.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc7506c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:37.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc7506c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:37.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc6f7fe640 1 -- 172.21.15.60:0/231849231 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352437843 0 0) 0x7fbc700b3830 con 0x7fbc700b40b0 2024-03-27T21:40:37.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc6f7fe640 10 monclient(hunting): _negotiate 2024-03-27T21:40:37.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:37.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:37.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc6f7fe640 10 monclient(hunting): my global_id is 4695 2024-03-27T21:40:37.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc6f7fe640 1 -- 172.21.15.60:0/231849231 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc60003520 con 0x7fbc700b40b0 2024-03-27T21:40:37.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc7506c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:37.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc7506c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:37.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc7506c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:37.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc6f7fe640 1 -- 172.21.15.60:0/231849231 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1876699760 0 0) 0x7fbc700aa520 con 0x7fbc7013b020 2024-03-27T21:40:37.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc6f7fe640 10 monclient(hunting): _negotiate 2024-03-27T21:40:37.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:37.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:37.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc6f7fe640 10 monclient(hunting): my global_id is 4636 2024-03-27T21:40:37.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.586+0000 7fbc6f7fe640 1 -- 172.21.15.60:0/231849231 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc700b3830 con 0x7fbc7013b020 2024-03-27T21:40:37.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 1 -- 172.21.15.60:0/231849231 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2701145704 0 0) 0x7fbc700b3830 con 0x7fbc7013b020 2024-03-27T21:40:37.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 1 -- 172.21.15.60:0/231849231 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbc7013b480 msgr2=0x7fbc70141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:37.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 1 --2- 172.21.15.60:0/231849231 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbc7013b480 0x7fbc70141e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:37.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 1 -- 172.21.15.60:0/231849231 >> v1:172.21.15.60:6789/0 conn(0x7fbc700b40b0 legacy=0x7fbc700a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:37.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 10 monclient: _finish_hunting 0 2024-03-27T21:40:37.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 1 monclient: found mon.noname-c-legacy 2024-03-27T21:40:37.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:37.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.191:6789/0 2024-03-27T21:40:37.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 1 -- 172.21.15.60:0/231849231 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc700a9470 con 0x7fbc7013b020 2024-03-27T21:40:37.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 10 monclient: _finish_auth 0 2024-03-27T21:40:37.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:40:37.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:37.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 1 -- 172.21.15.60:0/231849231 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2933500084 0 0) 0x7fbc5c001670 con 0x7fbc7013b020 2024-03-27T21:40:37.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:37.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-03-27T21:40:37.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 10 monclient: dump: 2024-03-27T21:40:37.585 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:37.585 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:37.585 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:37.585 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:37.586 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:37.586 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:37.586 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:37.586 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:37.586 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:37.586 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:37.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] but i'm connected to v1:172.21.15.191:6789/0, reconnecting 2024-03-27T21:40:37.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:37.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 1 -- 172.21.15.60:0/231849231 >> v1:172.21.15.191:6789/0 conn(0x7fbc7013b020 legacy=0x7fbc701416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:37.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:40:37.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 1 --2- 172.21.15.60:0/231849231 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbc7013b480 0x7fbc60003b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:37.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 10 monclient(hunting): picked mon.c con 0x7fbc7013b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:37.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 1 --2- 172.21.15.60:0/231849231 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc60004060 0x7fbc600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:37.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 10 monclient(hunting): picked mon.b con 0x7fbc60004060 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:37.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 1 --2- 172.21.15.60:0/231849231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc60004d00 0x7fbc60005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:37.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 10 monclient(hunting): picked mon.a con 0x7fbc60004d00 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:37.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:37.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 1 -- 172.21.15.60:0/231849231 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6000ba30 con 0x7fbc60004d00 2024-03-27T21:40:37.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:37.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 1 -- 172.21.15.60:0/231849231 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6000bb70 con 0x7fbc60004060 2024-03-27T21:40:37.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:37.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 1 -- 172.21.15.60:0/231849231 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6000bd30 con 0x7fbc7013b480 2024-03-27T21:40:37.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:37.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-27T21:40:37.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6ffff640 1 --2- 172.21.15.60:0/231849231 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc60004060 0x7fbc600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:37.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc7506c640 1 --2- 172.21.15.60:0/231849231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc60004d00 0x7fbc60005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:37.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc7506c640 10 monclient(hunting): get_auth_request con 0x7fbc60004d00 auth_method 0 2024-03-27T21:40:37.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc7506c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:37.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc7506c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:37.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc7506c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:40:37.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc7486b640 1 --2- 172.21.15.60:0/231849231 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbc7013b480 0x7fbc60003b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:37.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc7486b640 10 monclient(hunting): get_auth_request con 0x7fbc7013b480 auth_method 0 2024-03-27T21:40:37.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc7486b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:37.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc7486b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:37.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc7486b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:40:37.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6ffff640 10 monclient(hunting): get_auth_request con 0x7fbc60004060 auth_method 0 2024-03-27T21:40:37.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:37.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:37.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc6ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:40:37.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc7506c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:37.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc7506c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:37.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.587+0000 7fbc7506c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:40:37.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.588+0000 7fbc7486b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:37.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.588+0000 7fbc7486b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:37.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.588+0000 7fbc7486b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:40:37.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.588+0000 7fbc6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:37.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.588+0000 7fbc6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:37.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.588+0000 7fbc6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:40:37.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.588+0000 7fbc6ffff640 10 monclient(hunting): handle_auth_done global_id 4636 payload 293 2024-03-27T21:40:37.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.588+0000 7fbc6ffff640 1 -- 172.21.15.60:0/231849231 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbc7013b480 msgr2=0x7fbc60003b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:37.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.588+0000 7fbc6ffff640 1 --2- 172.21.15.60:0/231849231 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbc7013b480 0x7fbc60003b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:37.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.588+0000 7fbc6ffff640 1 -- 172.21.15.60:0/231849231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc60004d00 msgr2=0x7fbc60005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:37.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.588+0000 7fbc6ffff640 1 --2- 172.21.15.60:0/231849231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc60004d00 0x7fbc60005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:37.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.588+0000 7fbc6ffff640 10 monclient: _finish_hunting 0 2024-03-27T21:40:37.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.588+0000 7fbc6ffff640 1 monclient: found mon.b 2024-03-27T21:40:37.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.588+0000 7fbc6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:37.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.588+0000 7fbc6ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:37.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.588+0000 7fbc6ffff640 1 -- 172.21.15.60:0/231849231 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc700b3830 con 0x7fbc60004060 2024-03-27T21:40:37.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.588+0000 7fbc6ffff640 10 monclient: _finish_auth 0 2024-03-27T21:40:37.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.588+0000 7fbc6ffff640 10 monclient: _check_auth_tickets 2024-03-27T21:40:37.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.588+0000 7fbc6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:37.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.588+0000 7fbc76acf640 5 monclient: authenticate success, global_id 4636 2024-03-27T21:40:37.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.588+0000 7fbc76acf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:37.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.589+0000 7fbc6ffff640 1 --2- 172.21.15.60:0/231849231 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc60004060 0x7fbc600047c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c007b60 tx=0x7fbc600057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d51c9e9b3f7294f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:37.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.589+0000 7fbc6f7fe640 1 -- 172.21.15.60:0/231849231 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c007270 con 0x7fbc60004060 2024-03-27T21:40:37.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.589+0000 7fbc6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:37.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.589+0000 7fbc6f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:40:37.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.589+0000 7fbc6f7fe640 10 monclient: dump: 2024-03-27T21:40:37.591 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:37.591 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:37.591 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:37.591 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:37.591 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:37.592 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:37.592 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:37.592 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:37.592 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:37.592 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:37.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.589+0000 7fbc6f7fe640 1 -- 172.21.15.60:0/231849231 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5c0073f0 con 0x7fbc60004060 2024-03-27T21:40:37.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.589+0000 7fbc6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:37.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.589+0000 7fbc6f7fe640 1 -- 172.21.15.60:0/231849231 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c005810 con 0x7fbc60004060 2024-03-27T21:40:37.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.589+0000 7fbc6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:37.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.589+0000 7fbc6f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:40:37.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.589+0000 7fbc6f7fe640 10 monclient: dump: 2024-03-27T21:40:37.593 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:37.593 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:37.593 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:37.593 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:37.593 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:37.593 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:37.593 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:37.593 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:37.593 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:37.593 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:37.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.589+0000 7fbc76acf640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:37.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.589+0000 7fbc76acf640 4 set_mon_vals no callback set 2024-03-27T21:40:37.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.590+0000 7fbc76acf640 10 monclient: shutdown 2024-03-27T21:40:37.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.590+0000 7fbc76acf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:37.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.590+0000 7fbc76acf640 1 -- 172.21.15.60:0/231849231 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc60004060 msgr2=0x7fbc600047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:37.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.590+0000 7fbc76acf640 1 --2- 172.21.15.60:0/231849231 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc60004060 0x7fbc600047c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c007b60 tx=0x7fbc600057f0 comp rx=0 tx=0).stop 2024-03-27T21:40:37.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.590+0000 7fbc7506c640 1 -- 172.21.15.60:0/231849231 reap_dead start 2024-03-27T21:40:37.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.590+0000 7fbc76acf640 1 -- 172.21.15.60:0/231849231 shutdown_connections 2024-03-27T21:40:37.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.590+0000 7fbc76acf640 1 -- 172.21.15.60:0/231849231 >> 172.21.15.60:0/231849231 conn(0x7fbc700608e0 msgr2=0x7fbc70060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:37.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.590+0000 7fbc76acf640 1 -- 172.21.15.60:0/231849231 shutdown_connections 2024-03-27T21:40:37.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.590+0000 7fbc76acf640 1 -- 172.21.15.60:0/231849231 wait complete. 2024-03-27T21:40:37.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.591+0000 7fbc76acf640 10 monclient: build_initial_monmap 2024-03-27T21:40:37.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.591+0000 7fbc76acf640 10 monclient: monmap: 2024-03-27T21:40:37.594 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:37.594 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:37.594 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:37.594 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:37.594 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:37.595 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:37.595 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:37.595 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:37.595 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:37.595 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:37.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.592+0000 7fbc76acf640 1 librados: starting msgr at 2024-03-27T21:40:37.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.592+0000 7fbc76acf640 1 librados: starting objecter 2024-03-27T21:40:37.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.592+0000 7fbc76acf640 1 Processor -- start 2024-03-27T21:40:37.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.592+0000 7fbc76acf640 1 -- start start 2024-03-27T21:40:37.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.592+0000 7fbc76acf640 1 librados: setting wanted keys 2024-03-27T21:40:37.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.592+0000 7fbc76acf640 1 librados: calling monclient init 2024-03-27T21:40:37.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.592+0000 7fbc76acf640 10 monclient: init 2024-03-27T21:40:37.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.593+0000 7fbc76acf640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:37.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.593+0000 7fbc76acf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:40:37.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.593+0000 7fbc76acf640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbc700aac40 0x7fbc7006caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:37.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.593+0000 7fbc76acf640 10 monclient(hunting): picked mon.noname-c con 0x7fbc700aac40 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:37.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc7486b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbc700aac40 0x7fbc7006caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:37.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc7486b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbc700aac40 0x7fbc7006caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:59508/0 (socket says 172.21.15.60:59508) 2024-03-27T21:40:37.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc7486b640 1 -- 172.21.15.60:0/2437008184 learned_addr learned my addr 172.21.15.60:0/2437008184 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:37.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc76acf640 1 --2- 172.21.15.60:0/2437008184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc7006f9d0 0x7fbc701d2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:37.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc76acf640 10 monclient(hunting): picked mon.noname-a con 0x7fbc7006f9d0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:37.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc6ffff640 1 --2- 172.21.15.60:0/2437008184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc7006f9d0 0x7fbc701d2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:37.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc76acf640 1 --2- 172.21.15.60:0/2437008184 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc701d2a70 0x7fbc701d5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:37.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc76acf640 10 monclient(hunting): picked mon.noname-b con 0x7fbc701d2a70 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:37.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc76acf640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:37.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc76acf640 1 -- 172.21.15.60:0/2437008184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc700ab400 con 0x7fbc7006f9d0 2024-03-27T21:40:37.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc7506c640 1 --2- 172.21.15.60:0/2437008184 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc701d2a70 0x7fbc701d5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:37.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc76acf640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:37.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc76acf640 1 -- 172.21.15.60:0/2437008184 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc701d1e00 con 0x7fbc701d2a70 2024-03-27T21:40:37.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc76acf640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:37.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc76acf640 1 -- 172.21.15.60:0/2437008184 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc700ab020 con 0x7fbc700aac40 2024-03-27T21:40:37.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc76acf640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:37.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc76acf640 10 monclient(hunting): authenticate will time out at 878.815668s 2024-03-27T21:40:37.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc7486b640 10 monclient(hunting): get_auth_request con 0x7fbc700aac40 auth_method 0 2024-03-27T21:40:37.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc7486b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:37.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc7486b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:37.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc7486b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:37.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc6ffff640 10 monclient(hunting): get_auth_request con 0x7fbc7006f9d0 auth_method 0 2024-03-27T21:40:37.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:37.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:37.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:37.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc7506c640 10 monclient(hunting): get_auth_request con 0x7fbc701d2a70 auth_method 0 2024-03-27T21:40:37.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc7506c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:37.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc7506c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:37.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc7506c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:37.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:37.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:37.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.594+0000 7fbc6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:37.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.595+0000 7fbc7506c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:37.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.595+0000 7fbc7506c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:37.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.595+0000 7fbc7506c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:37.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.595+0000 7fbc7486b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:37.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.595+0000 7fbc7486b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:37.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.595+0000 7fbc7486b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:37.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.595+0000 7fbc6ffff640 10 monclient(hunting): handle_auth_done global_id 4698 payload 824 2024-03-27T21:40:37.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.595+0000 7fbc6ffff640 1 -- 172.21.15.60:0/2437008184 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbc700aac40 msgr2=0x7fbc7006caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:37.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.595+0000 7fbc6ffff640 1 --2- 172.21.15.60:0/2437008184 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbc700aac40 0x7fbc7006caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:37.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.595+0000 7fbc6ffff640 1 -- 172.21.15.60:0/2437008184 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc701d2a70 msgr2=0x7fbc701d5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:37.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.595+0000 7fbc6ffff640 1 --2- 172.21.15.60:0/2437008184 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc701d2a70 0x7fbc701d5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:37.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.595+0000 7fbc6ffff640 10 monclient: _finish_hunting 0 2024-03-27T21:40:37.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.595+0000 7fbc6ffff640 1 monclient: found mon.noname-a 2024-03-27T21:40:37.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.595+0000 7fbc6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:37.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.595+0000 7fbc6ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:37.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.595+0000 7fbc6ffff640 1 -- 172.21.15.60:0/2437008184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc700ac3d0 con 0x7fbc7006f9d0 2024-03-27T21:40:37.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.595+0000 7fbc6ffff640 1 --2- 172.21.15.60:0/2437008184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc7006f9d0 0x7fbc701d2530 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c042000 tx=0x7fbc5c001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:37.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.595+0000 7fbc6d7fa640 10 client.?.objecter ms_handle_connect 0x7fbc7006f9d0 2024-03-27T21:40:37.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.595+0000 7fbc6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:37.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc6d7fa640 1 -- 172.21.15.60:0/2437008184 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c04d040 con 0x7fbc7006f9d0 2024-03-27T21:40:37.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc6d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:37.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc6d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:37.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc6d7fa640 10 monclient: dump: 2024-03-27T21:40:37.603 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:37.603 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:37.603 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:37.603 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:37.603 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:37.603 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:37.603 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:37.603 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:37.603 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:37.603 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:37.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc6d7fa640 10 monclient: _finish_auth 0 2024-03-27T21:40:37.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc6d7fa640 10 monclient: _check_auth_tickets 2024-03-27T21:40:37.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc6d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:37.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc6d7fa640 1 -- 172.21.15.60:0/2437008184 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5c007270 con 0x7fbc7006f9d0 2024-03-27T21:40:37.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc6d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:37.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc6d7fa640 1 -- 172.21.15.60:0/2437008184 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c03d970 con 0x7fbc7006f9d0 2024-03-27T21:40:37.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc6d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:37.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc6d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:37.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc6d7fa640 10 monclient: dump: 2024-03-27T21:40:37.604 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:37.604 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:37.604 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:37.604 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:37.604 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:37.604 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:37.604 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:37.604 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:37.604 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:37.605 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:37.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc76acf640 5 monclient: authenticate success, global_id 4698 2024-03-27T21:40:37.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc76acf640 10 monclient: _renew_subs 2024-03-27T21:40:37.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc76acf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:37.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc76acf640 1 -- 172.21.15.60:0/2437008184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc7006fe50 con 0x7fbc7006f9d0 2024-03-27T21:40:37.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc6effd640 4 set_mon_vals no callback set 2024-03-27T21:40:37.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc76acf640 10 client.4698.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:37.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc76acf640 10 monclient: _renew_subs 2024-03-27T21:40:37.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc76acf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:37.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc76acf640 1 -- 172.21.15.60:0/2437008184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc701d58c0 con 0x7fbc7006f9d0 2024-03-27T21:40:37.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.596+0000 7fbc76acf640 1 librados: init done 2024-03-27T21:40:37.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.598+0000 7fbc76acf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:37.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.598+0000 7fbc76acf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:37.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.598+0000 7fbc76acf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:37.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.598+0000 7fbc76acf640 1 -- 172.21.15.60:0/2437008184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc70069470 con 0x7fbc7006f9d0 2024-03-27T21:40:37.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.598+0000 7fbc6d7fa640 1 -- 172.21.15.60:0/2437008184 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fbc5c043070 con 0x7fbc7006f9d0 2024-03-27T21:40:37.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.599+0000 7fbc6d7fa640 1 --2- 172.21.15.60:0/2437008184 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fbc5406c590 0x7fbc5406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:37.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.599+0000 7fbc6d7fa640 1 -- 172.21.15.60:0/2437008184 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fbc5c0bb350 con 0x7fbc7006f9d0 2024-03-27T21:40:37.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.599+0000 7fbc6d7fa640 10 client.4698.objecter ms_dispatch 0x7fbc70000b90 osd_map(40..40 src has 1..40) v4 2024-03-27T21:40:37.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.599+0000 7fbc6d7fa640 3 client.4698.objecter handle_osd_map got epochs [40,40] > 0 2024-03-27T21:40:37.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.599+0000 7fbc6d7fa640 3 client.4698.objecter handle_osd_map decoding full epoch 40 2024-03-27T21:40:37.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.599+0000 7fbc7486b640 1 --2- 172.21.15.60:0/2437008184 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fbc5406c590 0x7fbc5406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:37.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.599+0000 7fbc6d7fa640 20 client.4698.objecter dump_active .. 0 homeless 2024-03-27T21:40:37.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.599+0000 7fbc7486b640 10 monclient: get_auth_request con 0x7fbc5406c590 auth_method 0 2024-03-27T21:40:37.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.600+0000 7fbc6d7fa640 10 client.4698.objecter ms_handle_connect 0x7fbc5406c590 2024-03-27T21:40:37.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.600+0000 7fbc7486b640 1 --2- 172.21.15.60:0/2437008184 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fbc5406c590 0x7fbc5406ea50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbc64004e70 tx=0x7fbc64007d70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:37.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.604+0000 7fbc6d7fa640 1 -- 172.21.15.60:0/2437008184 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbc5c086840 con 0x7fbc7006f9d0 2024-03-27T21:40:37.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.604+0000 7fbc6d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:37.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.604+0000 7fbc6d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:37.607 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:37.607 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:37.609 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:40:37.611 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:37.625+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:37.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.717+0000 7f08aaeb6640 10 monclient: get_monmap_and_config 2024-03-27T21:40:37.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.717+0000 7f08aaeb6640 10 monclient: build_initial_monmap 2024-03-27T21:40:37.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.717+0000 7f08aaeb6640 10 monclient: monmap: 2024-03-27T21:40:37.704 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:37.704 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:37.704 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:37.718710+0000 2024-03-27T21:40:37.704 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:37.718710+0000 2024-03-27T21:40:37.704 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:37.704 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:37.704 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:37.704 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:37.705 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:37.705 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:37.705 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:37.705 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.718+0000 7f08aaeb6640 1 Processor -- start 2024-03-27T21:40:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.718+0000 7f08aaeb6640 1 -- start start 2024-03-27T21:40:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.718+0000 7f08aaeb6640 10 monclient: init 2024-03-27T21:40:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08aaeb6640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08aaeb6640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-03-27T21:40:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08aaeb6640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f08a4071a50 0x7f08a40a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08aaeb6640 10 monclient(hunting): picked mon.noname-b con 0x7f08a4071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08aaeb6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f08a413e810 0x7f08a413ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08aaeb6640 10 monclient(hunting): picked mon.noname-c con 0x7f08a413e810 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08aaeb6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f08a4141630 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08aaeb6640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a413f190 con 0x7f08a4141630 2024-03-27T21:40:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08aaeb6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08a8c52640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f08a4071a50 0x7f08a40a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08a3fff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f08a413e810 0x7f08a413ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08a3fff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f08a413e810 0x7f08a413ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:54678/0 (socket says 172.21.15.60:54678) 2024-03-27T21:40:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08a3fff640 1 -- 172.21.15.60:0/4021129748 learned_addr learned my addr 172.21.15.60:0/4021129748 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08aaeb6640 1 -- 172.21.15.60:0/4021129748 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a4071e30 con 0x7f08a413e810 2024-03-27T21:40:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08aaeb6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08aaeb6640 1 -- 172.21.15.60:0/4021129748 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a40a8e90 con 0x7f08a4071a50 2024-03-27T21:40:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08aaeb6640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08aaeb6640 10 monclient(hunting): authenticate will time out at 878.940657s 2024-03-27T21:40:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08a3fff640 10 monclient(hunting): get_auth_request con 0x7f08a413e810 auth_method 0 2024-03-27T21:40:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08a8c52640 10 monclient(hunting): get_auth_request con 0x7f08a4071a50 auth_method 0 2024-03-27T21:40:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08a8c52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08a8c52640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08a8c52640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08a8c52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08a8c52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.719+0000 7f08a8c52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.720+0000 7f08a3fff640 10 monclient(hunting): handle_auth_done global_id 4642 payload 274 2024-03-27T21:40:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.720+0000 7f08a3fff640 1 -- 172.21.15.60:0/4021129748 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f08a4071a50 msgr2=0x7f08a40a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.720+0000 7f08a37fe640 1 -- 172.21.15.60:0/4021129748 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185426359 0 0) 0x7f089c001490 con 0x7f08a4141630 2024-03-27T21:40:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.720+0000 7f08a3fff640 1 --2- 172.21.15.60:0/4021129748 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f08a4071a50 0x7f08a40a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.720+0000 7f08a3fff640 1 -- 172.21.15.60:0/4021129748 >> v1:172.21.15.60:6789/0 conn(0x7f08a4141630 legacy=0x7f08a4141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.720+0000 7f08a3fff640 10 monclient: _finish_hunting 0 2024-03-27T21:40:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.720+0000 7f08a3fff640 1 monclient: found mon.noname-c 2024-03-27T21:40:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.720+0000 7f08a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.720+0000 7f08a3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:40:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.720+0000 7f08a3fff640 1 -- 172.21.15.60:0/4021129748 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a40aa040 con 0x7f08a413e810 2024-03-27T21:40:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.720+0000 7f08a37fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:40:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.720+0000 7f08a8c52640 1 --2- 172.21.15.60:0/4021129748 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f08a4071a50 0x7f08a40a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.720+0000 7f08a3fff640 1 --2- 172.21.15.60:0/4021129748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08a413e810 0x7f08a413ec50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0890009870 tx=0x7f089002b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7858b28cad038773 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.721+0000 7f08a37fe640 1 -- 172.21.15.60:0/4021129748 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0890030880 con 0x7f08a413e810 2024-03-27T21:40:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.721+0000 7f08a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.721+0000 7f08a37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.721+0000 7f08a37fe640 10 monclient: dump: 2024-03-27T21:40:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:37.714 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:37.714 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:37.714 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:37.714 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:37.714 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:37.714 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:37.714 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:37.714 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:37.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.721+0000 7f08a37fe640 10 monclient: _finish_auth 0 2024-03-27T21:40:37.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.721+0000 7f08a37fe640 10 monclient: _check_auth_tickets 2024-03-27T21:40:37.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.721+0000 7f08a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:37.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.721+0000 7f08a37fe640 1 -- 172.21.15.60:0/4021129748 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08900309e0 con 0x7f08a413e810 2024-03-27T21:40:37.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.721+0000 7f08a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:37.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.721+0000 7f08a37fe640 1 -- 172.21.15.60:0/4021129748 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0890030d50 con 0x7f08a413e810 2024-03-27T21:40:37.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.721+0000 7f08aaeb6640 5 monclient: authenticate success, global_id 4642 2024-03-27T21:40:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.721+0000 7f08aaeb6640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.721+0000 7f08aaeb6640 4 set_mon_vals no callback set 2024-03-27T21:40:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.721+0000 7f08a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.721+0000 7f08a37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:40:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.721+0000 7f08a37fe640 10 monclient: dump: 2024-03-27T21:40:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:37.715 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.721+0000 7f08aaeb6640 10 monclient: shutdown 2024-03-27T21:40:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.721+0000 7f08aaeb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.721+0000 7f08aaeb6640 1 -- 172.21.15.60:0/4021129748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08a413e810 msgr2=0x7f08a413ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.721+0000 7f08aaeb6640 1 --2- 172.21.15.60:0/4021129748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08a413e810 0x7f08a413ec50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0890009870 tx=0x7f089002b9c0 comp rx=0 tx=0).stop 2024-03-27T21:40:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.722+0000 7f08aaeb6640 1 -- 172.21.15.60:0/4021129748 shutdown_connections 2024-03-27T21:40:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.722+0000 7f08aaeb6640 1 --2- 172.21.15.60:0/4021129748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08a413e810 0x7f08a413ec50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.722+0000 7f08aaeb6640 1 --2- 172.21.15.60:0/4021129748 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f08a4071a50 0x7f08a40a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.722+0000 7f08aaeb6640 1 -- 172.21.15.60:0/4021129748 >> 172.21.15.60:0/4021129748 conn(0x7f08a40b7300 msgr2=0x7f08a40b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.722+0000 7f08aaeb6640 1 -- 172.21.15.60:0/4021129748 shutdown_connections 2024-03-27T21:40:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.722+0000 7f08aaeb6640 1 -- 172.21.15.60:0/4021129748 wait complete. 2024-03-27T21:40:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.723+0000 7f08aaeb6640 10 monclient: build_initial_monmap 2024-03-27T21:40:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.723+0000 7f08aaeb6640 10 monclient: monmap: 2024-03-27T21:40:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:37.717 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:37.717 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:37.717 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:37.717 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:37.717 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:37.717 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:37.717 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:37.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.723+0000 7f08aaeb6640 1 librados: starting msgr at 2024-03-27T21:40:37.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.723+0000 7f08aaeb6640 1 librados: starting objecter 2024-03-27T21:40:37.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.723+0000 7f08aaeb6640 1 Processor -- start 2024-03-27T21:40:37.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.723+0000 7f08aaeb6640 1 -- start start 2024-03-27T21:40:37.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.723+0000 7f08aaeb6640 1 librados: setting wanted keys 2024-03-27T21:40:37.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.723+0000 7f08aaeb6640 1 librados: calling monclient init 2024-03-27T21:40:37.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.724+0000 7f08aaeb6640 10 monclient: init 2024-03-27T21:40:37.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.725+0000 7f08aaeb6640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:37.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.725+0000 7f08aaeb6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:40:37.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.725+0000 7f08aaeb6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08a4071a50 0x7f08a4136190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:37.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.725+0000 7f08aaeb6640 10 monclient(hunting): picked mon.noname-a con 0x7f08a4071a50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:37.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.725+0000 7f08aaeb6640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f08a413e810 0x7f08a4139180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:37.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.725+0000 7f08a8c52640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08a4071a50 0x7f08a4136190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:37.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.725+0000 7f08a8c52640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08a4071a50 0x7f08a4136190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58284/0 (socket says 172.21.15.60:58284) 2024-03-27T21:40:37.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.725+0000 7f08a3fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f08a413e810 0x7f08a4139180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:37.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.725+0000 7f08a3fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f08a413e810 0x7f08a4139180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:59514/0 (socket says 172.21.15.60:59514) 2024-03-27T21:40:37.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.725+0000 7f08aaeb6640 10 monclient(hunting): picked mon.noname-c con 0x7f08a413e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:37.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.725+0000 7f08a8c52640 1 -- 172.21.15.60:0/2822571180 learned_addr learned my addr 172.21.15.60:0/2822571180 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:37.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.725+0000 7f08aaeb6640 1 --2- 172.21.15.60:0/2822571180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08a4141630 0x7f08a41d6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:37.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.725+0000 7f08aaeb6640 10 monclient(hunting): picked mon.noname-b con 0x7f08a4141630 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:37.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.725+0000 7f08aaeb6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:37.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.725+0000 7f08aaeb6640 1 -- 172.21.15.60:0/2822571180 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a413af00 con 0x7f08a4071a50 2024-03-27T21:40:37.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.725+0000 7f08aaeb6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:37.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.725+0000 7f08aaeb6640 1 -- 172.21.15.60:0/2822571180 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a41396c0 con 0x7f08a4141630 2024-03-27T21:40:37.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.725+0000 7f08aaeb6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:37.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.725+0000 7f08aaeb6640 1 -- 172.21.15.60:0/2822571180 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a4139970 con 0x7f08a413e810 2024-03-27T21:40:37.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.725+0000 7f08aaeb6640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:37.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.725+0000 7f08aaeb6640 10 monclient(hunting): authenticate will time out at 878.946915s 2024-03-27T21:40:37.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a8c52640 10 monclient(hunting): get_auth_request con 0x7f08a4071a50 auth_method 0 2024-03-27T21:40:37.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a8c52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:37.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a8c52640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:37.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a8c52640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:37.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a3fff640 10 monclient(hunting): get_auth_request con 0x7f08a413e810 auth_method 0 2024-03-27T21:40:37.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:37.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:37.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:37.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a9453640 1 --2- 172.21.15.60:0/2822571180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08a4141630 0x7f08a41d6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:37.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a9453640 10 monclient(hunting): get_auth_request con 0x7f08a4141630 auth_method 0 2024-03-27T21:40:37.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a9453640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:37.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a9453640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:37.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a9453640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:37.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a8c52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:37.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a8c52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:37.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a8c52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:37.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:37.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:37.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:37.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a9453640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:37.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a9453640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:37.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a9453640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:37.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a9453640 10 monclient(hunting): handle_auth_done global_id 4645 payload 824 2024-03-27T21:40:37.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a9453640 1 -- 172.21.15.60:0/2822571180 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f08a413e810 msgr2=0x7f08a4139180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:37.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a9453640 1 --2- 172.21.15.60:0/2822571180 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f08a413e810 0x7f08a4139180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:37.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a9453640 1 -- 172.21.15.60:0/2822571180 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08a4071a50 msgr2=0x7f08a4136190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:37.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a9453640 1 --2- 172.21.15.60:0/2822571180 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08a4071a50 0x7f08a4136190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:37.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a9453640 10 monclient: _finish_hunting 0 2024-03-27T21:40:37.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a9453640 1 monclient: found mon.noname-b 2024-03-27T21:40:37.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a9453640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:37.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a9453640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:40:37.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a9453640 1 -- 172.21.15.60:0/2822571180 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0898009400 con 0x7f08a4141630 2024-03-27T21:40:37.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a8c52640 1 --2- 172.21.15.60:0/2822571180 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08a4071a50 0x7f08a4136190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:37.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a3fff640 1 --2- 172.21.15.60:0/2822571180 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f08a413e810 0x7f08a4139180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:37.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a9453640 1 --2- 172.21.15.60:0/2822571180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08a4141630 0x7f08a41d6710 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f089c00c780 tx=0x7f089c00cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:37.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a17fa640 10 client.?.objecter ms_handle_connect 0x7f08a4141630 2024-03-27T21:40:37.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a17fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:37.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a17fa640 1 -- 172.21.15.60:0/2822571180 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089c003b70 con 0x7f08a4141630 2024-03-27T21:40:37.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:37.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:37.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a17fa640 10 monclient: dump: 2024-03-27T21:40:37.727 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:37.727 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:37.727 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:37.727 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:37.727 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:37.727 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:37.728 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:37.728 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:37.728 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:37.728 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:37.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a17fa640 10 monclient: _finish_auth 0 2024-03-27T21:40:37.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a17fa640 10 monclient: _check_auth_tickets 2024-03-27T21:40:37.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:37.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08aaeb6640 5 monclient: authenticate success, global_id 4645 2024-03-27T21:40:37.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a17fa640 1 -- 172.21.15.60:0/2822571180 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f089c003cd0 con 0x7f08a4141630 2024-03-27T21:40:37.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:37.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a17fa640 1 -- 172.21.15.60:0/2822571180 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089c005290 con 0x7f08a4141630 2024-03-27T21:40:37.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08aaeb6640 10 monclient: _renew_subs 2024-03-27T21:40:37.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08aaeb6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:37.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08aaeb6640 1 -- 172.21.15.60:0/2822571180 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08900094d0 con 0x7f08a4141630 2024-03-27T21:40:37.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.726+0000 7f08a2ffd640 4 set_mon_vals no callback set 2024-03-27T21:40:37.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.727+0000 7f08a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:37.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.727+0000 7f08a17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:40:37.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.727+0000 7f08a17fa640 10 monclient: dump: 2024-03-27T21:40:37.729 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:37.729 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:37.729 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:37.729 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:37.729 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:37.729 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:37.729 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:37.729 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:37.729 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:37.729 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:37.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.727+0000 7f08aaeb6640 10 client.4645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:37.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.727+0000 7f08aaeb6640 10 monclient: _renew_subs 2024-03-27T21:40:37.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.727+0000 7f08aaeb6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:37.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.727+0000 7f08aaeb6640 1 -- 172.21.15.60:0/2822571180 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a4139e90 con 0x7f08a4141630 2024-03-27T21:40:37.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.727+0000 7f08aaeb6640 1 librados: init done 2024-03-27T21:40:37.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.728+0000 7f08a17fa640 1 -- 172.21.15.60:0/2822571180 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f089c0053f0 con 0x7f08a4141630 2024-03-27T21:40:37.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.728+0000 7f08aaeb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:37.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.728+0000 7f08aaeb6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:37.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.728+0000 7f08aaeb6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:37.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.728+0000 7f08aaeb6640 1 -- 172.21.15.60:0/2822571180 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08a40aa550 con 0x7f08a4141630 2024-03-27T21:40:37.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.728+0000 7f08a17fa640 1 --2- 172.21.15.60:0/2822571180 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f088806c5c0 0x7f088806ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:37.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.728+0000 7f08a17fa640 1 -- 172.21.15.60:0/2822571180 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f089c088750 con 0x7f08a4141630 2024-03-27T21:40:37.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.728+0000 7f08a17fa640 10 client.4645.objecter ms_dispatch 0x7f08a4000b90 osd_map(40..40 src has 1..40) v4 2024-03-27T21:40:37.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.728+0000 7f08a17fa640 3 client.4645.objecter handle_osd_map got epochs [40,40] > 0 2024-03-27T21:40:37.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.728+0000 7f08a17fa640 3 client.4645.objecter handle_osd_map decoding full epoch 40 2024-03-27T21:40:37.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.728+0000 7f08a8c52640 1 --2- 172.21.15.60:0/2822571180 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f088806c5c0 0x7f088806ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:37.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.728+0000 7f08a8c52640 10 monclient: get_auth_request con 0x7f088806c5c0 auth_method 0 2024-03-27T21:40:37.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.729+0000 7f08a17fa640 20 client.4645.objecter dump_active .. 0 homeless 2024-03-27T21:40:37.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.729+0000 7f08a17fa640 10 client.4645.objecter ms_handle_connect 0x7f088806c5c0 2024-03-27T21:40:37.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.729+0000 7f08a8c52640 1 --2- 172.21.15.60:0/2822571180 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f088806c5c0 0x7f088806ea80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0898009fd0 tx=0x7f0898009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:37.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.733+0000 7f08a17fa640 1 -- 172.21.15.60:0/2822571180 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f089c053ba0 con 0x7f08a4141630 2024-03-27T21:40:37.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.733+0000 7f08a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:37.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.733+0000 7f08a17fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:37.731 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:37.732+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:37.755 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:37.755 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:37.763 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:40:37.819 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:37.833+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:37.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.887+0000 7fbc76acf640 10 start_mon_command cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:40:37.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.887+0000 7fbc76acf640 10 monclient: _send_command 2 [{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:40:37.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.887+0000 7fbc76acf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:37.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.887+0000 7fbc76acf640 1 -- 172.21.15.60:0/2437008184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fbc700b68e0 con 0x7fbc7006f9d0 2024-03-27T21:40:37.908 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:37.908 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:37.917 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:40:37.920 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:37.934+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:37.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.956+0000 7f08aaeb6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json-pretty"}] 2024-03-27T21:40:37.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.956+0000 7f08aaeb6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json-pretty"}] 2024-03-27T21:40:37.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.956+0000 7f08aaeb6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:37.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.956+0000 7f08aaeb6640 1 -- 172.21.15.60:0/2822571180 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f08a4069470 con 0x7f08a4141630 2024-03-27T21:40:37.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.957+0000 7f08a17fa640 1 -- 172.21.15.60:0/2822571180 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v40) v1 ==== 81+0+30239 (secure 0 0 0) 0x7f089c017020 con 0x7f08a4141630 2024-03-27T21:40:37.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.957+0000 7f08a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json-pretty"}] 2024-03-27T21:40:37.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.957+0000 7f08a17fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:40:37.944 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:40:37.945 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:37.945 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 40, 2024-03-27T21:40:37.945 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "dcd19d9a-535c-4b41-9095-d588268e2860", 2024-03-27T21:40:37.945 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2024-03-27T21:39:27.092191+0000", 2024-03-27T21:40:37.945 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2024-03-27T21:40:35.449663+0000", 2024-03-27T21:40:37.945 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2024-03-27T21:40:20.406185+0000", 2024-03-27T21:40:37.945 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2024-03-27T21:39:29.291620+0000", 2024-03-27T21:40:37.945 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-27T21:40:37.945 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5804032, 2024-03-27T21:40:37.945 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2024-03-27T21:40:37.945 INFO:teuthology.orchestra.run.smithi060.stdout: "nodeep-scrub", 2024-03-27T21:40:37.945 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2024-03-27T21:40:37.945 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2024-03-27T21:40:37.945 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2024-03-27T21:40:37.946 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2024-03-27T21:40:37.946 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-27T21:40:37.946 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 6, 2024-03-27T21:40:37.946 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2024-03-27T21:40:37.946 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-27T21:40:37.946 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-27T21:40:37.946 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2024-03-27T21:40:37.946 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 2, 2024-03-27T21:40:37.946 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2024-03-27T21:40:37.946 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "luminous", 2024-03-27T21:40:37.946 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2024-03-27T21:40:37.946 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "reef", 2024-03-27T21:40:37.946 INFO:teuthology.orchestra.run.smithi060.stdout: "allow_crimson": false, 2024-03-27T21:40:37.947 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2024-03-27T21:40:37.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.947 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-03-27T21:40:37.947 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": ".mgr", 2024-03-27T21:40:37.947 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2024-03-27T21:39:37.438493+0000", 2024-03-27T21:40:37.947 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-27T21:40:37.947 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2024-03-27T21:40:37.947 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2024-03-27T21:40:37.947 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2024-03-27T21:40:37.947 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2024-03-27T21:40:37.947 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2024-03-27T21:40:37.947 INFO:teuthology.orchestra.run.smithi060.stdout: "peering_crush_bucket_count": 0, 2024-03-27T21:40:37.947 INFO:teuthology.orchestra.run.smithi060.stdout: "peering_crush_bucket_target": 0, 2024-03-27T21:40:37.947 INFO:teuthology.orchestra.run.smithi060.stdout: "peering_crush_bucket_barrier": 0, 2024-03-27T21:40:37.948 INFO:teuthology.orchestra.run.smithi060.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-27T21:40:37.948 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2024-03-27T21:40:37.948 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2024-03-27T21:40:37.948 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2024-03-27T21:40:37.948 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2024-03-27T21:40:37.948 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2024-03-27T21:40:37.948 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2024-03-27T21:40:37.948 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2024-03-27T21:40:37.948 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2024-03-27T21:40:37.948 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2024-03-27T21:40:37.948 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2024-03-27T21:40:37.948 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2024-03-27T21:40:37.949 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2024-03-27T21:40:37.949 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2024-03-27T21:40:37.949 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2024-03-27T21:40:37.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.949 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2024-03-27T21:40:37.949 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2024-03-27T21:40:37.949 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-27T21:40:37.949 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2024-03-27T21:40:37.949 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2024-03-27T21:40:37.949 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2024-03-27T21:40:37.949 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2024-03-27T21:40:37.949 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2024-03-27T21:40:37.950 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2024-03-27T21:40:37.950 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2024-03-27T21:40:37.950 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2024-03-27T21:40:37.950 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2024-03-27T21:40:37.950 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2024-03-27T21:40:37.950 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2024-03-27T21:40:37.950 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2024-03-27T21:40:37.950 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2024-03-27T21:40:37.950 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2024-03-27T21:40:37.950 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2024-03-27T21:40:37.950 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2024-03-27T21:40:37.950 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-27T21:40:37.950 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-27T21:40:37.951 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-27T21:40:37.951 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2024-03-27T21:40:37.951 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2024-03-27T21:40:37.951 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2024-03-27T21:40:37.951 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2024-03-27T21:40:37.951 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2024-03-27T21:40:37.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.951 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2024-03-27T21:40:37.951 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2024-03-27T21:40:37.951 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2024-03-27T21:40:37.951 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2024-03-27T21:40:37.951 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2024-03-27T21:40:37.952 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2024-03-27T21:40:37.952 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2024-03-27T21:40:37.952 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2024-03-27T21:40:37.952 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2024-03-27T21:40:37.952 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2024-03-27T21:40:37.952 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2024-03-27T21:40:37.952 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2024-03-27T21:40:37.952 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_max": 32, 2024-03-27T21:40:37.952 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2024-03-27T21:40:37.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.952 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2024-03-27T21:40:37.952 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": {} 2024-03-27T21:40:37.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.953 INFO:teuthology.orchestra.run.smithi060.stdout: "read_balance": { 2024-03-27T21:40:37.953 INFO:teuthology.orchestra.run.smithi060.stdout: "score_acting": 8, 2024-03-27T21:40:37.953 INFO:teuthology.orchestra.run.smithi060.stdout: "score_stable": 8, 2024-03-27T21:40:37.953 INFO:teuthology.orchestra.run.smithi060.stdout: "optimal_score": 0.25, 2024-03-27T21:40:37.953 INFO:teuthology.orchestra.run.smithi060.stdout: "raw_score_acting": 2, 2024-03-27T21:40:37.953 INFO:teuthology.orchestra.run.smithi060.stdout: "raw_score_stable": 2, 2024-03-27T21:40:37.953 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity_weighted": 1, 2024-03-27T21:40:37.953 INFO:teuthology.orchestra.run.smithi060.stdout: "average_primary_affinity": 1, 2024-03-27T21:40:37.953 INFO:teuthology.orchestra.run.smithi060.stdout: "average_primary_affinity_weighted": 1 2024-03-27T21:40:37.953 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.953 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2024-03-27T21:40:37.954 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2024-03-27T21:40:37.954 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2024-03-27T21:39:45.042103+0000", 2024-03-27T21:40:37.954 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-27T21:40:37.954 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2024-03-27T21:40:37.954 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2024-03-27T21:40:37.954 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2024-03-27T21:40:37.954 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2024-03-27T21:40:37.954 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2024-03-27T21:40:37.954 INFO:teuthology.orchestra.run.smithi060.stdout: "peering_crush_bucket_count": 0, 2024-03-27T21:40:37.954 INFO:teuthology.orchestra.run.smithi060.stdout: "peering_crush_bucket_target": 0, 2024-03-27T21:40:37.954 INFO:teuthology.orchestra.run.smithi060.stdout: "peering_crush_bucket_barrier": 0, 2024-03-27T21:40:37.954 INFO:teuthology.orchestra.run.smithi060.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-27T21:40:37.954 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2024-03-27T21:40:37.955 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2024-03-27T21:40:37.955 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2024-03-27T21:40:37.955 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2024-03-27T21:40:37.955 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2024-03-27T21:40:37.955 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2024-03-27T21:40:37.955 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2024-03-27T21:40:37.955 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2024-03-27T21:40:37.955 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2024-03-27T21:40:37.955 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2024-03-27T21:40:37.955 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2024-03-27T21:40:37.955 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2024-03-27T21:40:37.955 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2024-03-27T21:40:37.955 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2024-03-27T21:40:37.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.956 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "21", 2024-03-27T21:40:37.956 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2024-03-27T21:40:37.956 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-27T21:40:37.956 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2024-03-27T21:40:37.956 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2024-03-27T21:40:37.956 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2024-03-27T21:40:37.956 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2024-03-27T21:40:37.956 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2024-03-27T21:40:37.956 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2024-03-27T21:40:37.956 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2024-03-27T21:40:37.957 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2024-03-27T21:40:37.957 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2024-03-27T21:40:37.957 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2024-03-27T21:40:37.957 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2024-03-27T21:40:37.957 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2024-03-27T21:40:37.957 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2024-03-27T21:40:37.957 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2024-03-27T21:40:37.957 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2024-03-27T21:40:37.957 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2024-03-27T21:40:37.957 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-27T21:40:37.957 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-27T21:40:37.957 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-27T21:40:37.957 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2024-03-27T21:40:37.958 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2024-03-27T21:40:37.958 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2024-03-27T21:40:37.958 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2024-03-27T21:40:37.958 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2024-03-27T21:40:37.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.958 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2024-03-27T21:40:37.958 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2024-03-27T21:40:37.958 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2024-03-27T21:40:37.958 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2024-03-27T21:40:37.958 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2024-03-27T21:40:37.958 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2024-03-27T21:40:37.958 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2024-03-27T21:40:37.958 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2024-03-27T21:40:37.959 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2024-03-27T21:40:37.959 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2024-03-27T21:40:37.959 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2024-03-27T21:40:37.959 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2024-03-27T21:40:37.959 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2024-03-27T21:40:37.959 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2024-03-27T21:40:37.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.959 INFO:teuthology.orchestra.run.smithi060.stdout: "read_balance": { 2024-03-27T21:40:37.959 INFO:teuthology.orchestra.run.smithi060.stdout: "score_acting": 1.9900000095367432, 2024-03-27T21:40:37.959 INFO:teuthology.orchestra.run.smithi060.stdout: "score_stable": 1.9900000095367432, 2024-03-27T21:40:37.959 INFO:teuthology.orchestra.run.smithi060.stdout: "optimal_score": 0.87999999523162842, 2024-03-27T21:40:37.959 INFO:teuthology.orchestra.run.smithi060.stdout: "raw_score_acting": 1.75, 2024-03-27T21:40:37.959 INFO:teuthology.orchestra.run.smithi060.stdout: "raw_score_stable": 1.75, 2024-03-27T21:40:37.960 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity_weighted": 1, 2024-03-27T21:40:37.960 INFO:teuthology.orchestra.run.smithi060.stdout: "average_primary_affinity": 1, 2024-03-27T21:40:37.960 INFO:teuthology.orchestra.run.smithi060.stdout: "average_primary_affinity_weighted": 1 2024-03-27T21:40:37.960 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.960 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.960 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-27T21:40:37.960 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2024-03-27T21:40:37.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.960 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2024-03-27T21:40:37.960 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "778aaa64-6720-495b-bcdc-86fbcece9d58", 2024-03-27T21:40:37.960 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2024-03-27T21:40:37.960 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2024-03-27T21:40:37.961 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2024-03-27T21:40:37.961 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2024-03-27T21:40:37.961 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2024-03-27T21:40:37.961 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2024-03-27T21:40:37.961 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 14, 2024-03-27T21:40:37.961 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 0, 2024-03-27T21:40:37.961 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2024-03-27T21:40:37.961 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2024-03-27T21:40:37.961 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2024-03-27T21:40:37.961 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.961 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.961 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2024-03-27T21:40:37.961 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34603 2024-03-27T21:40:37.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.962 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.962 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2024-03-27T21:40:37.962 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34603 2024-03-27T21:40:37.962 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.962 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.962 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2024-03-27T21:40:37.962 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.962 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.962 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2024-03-27T21:40:37.962 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34603 2024-03-27T21:40:37.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.963 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.963 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2024-03-27T21:40:37.963 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34603 2024-03-27T21:40:37.963 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.963 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.963 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2024-03-27T21:40:37.963 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.963 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.963 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2024-03-27T21:40:37.963 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34603 2024-03-27T21:40:37.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.964 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.964 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2024-03-27T21:40:37.964 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34603 2024-03-27T21:40:37.964 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.964 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.964 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2024-03-27T21:40:37.964 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.964 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.964 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2024-03-27T21:40:37.964 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34603 2024-03-27T21:40:37.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.965 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.965 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2024-03-27T21:40:37.965 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34603 2024-03-27T21:40:37.965 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.965 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.965 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6817/34603", 2024-03-27T21:40:37.965 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6819/34603", 2024-03-27T21:40:37.965 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6823/34603", 2024-03-27T21:40:37.965 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6821/34603", 2024-03-27T21:40:37.965 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2024-03-27T21:40:37.965 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2024-03-27T21:40:37.966 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2024-03-27T21:40:37.966 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.966 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2024-03-27T21:40:37.966 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "d960adf1-3d83-4e24-a612-8df53fa43d0c", 2024-03-27T21:40:37.966 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2024-03-27T21:40:37.966 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2024-03-27T21:40:37.966 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2024-03-27T21:40:37.966 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2024-03-27T21:40:37.966 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2024-03-27T21:40:37.966 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2024-03-27T21:40:37.966 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2024-03-27T21:40:37.966 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 19, 2024-03-27T21:40:37.967 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2024-03-27T21:40:37.967 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2024-03-27T21:40:37.967 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2024-03-27T21:40:37.967 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.967 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.967 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2024-03-27T21:40:37.967 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34602 2024-03-27T21:40:37.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.967 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.967 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2024-03-27T21:40:37.967 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34602 2024-03-27T21:40:37.967 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.968 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.968 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2024-03-27T21:40:37.968 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.968 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.968 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2024-03-27T21:40:37.968 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34602 2024-03-27T21:40:37.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.968 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.968 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2024-03-27T21:40:37.968 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34602 2024-03-27T21:40:37.968 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.969 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.969 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2024-03-27T21:40:37.969 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.969 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.969 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2024-03-27T21:40:37.969 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34602 2024-03-27T21:40:37.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.969 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.969 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2024-03-27T21:40:37.969 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34602 2024-03-27T21:40:37.969 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.970 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.970 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2024-03-27T21:40:37.970 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.970 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.970 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2024-03-27T21:40:37.970 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34602 2024-03-27T21:40:37.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.970 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.970 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2024-03-27T21:40:37.970 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34602 2024-03-27T21:40:37.970 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.971 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.971 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/34602", 2024-03-27T21:40:37.971 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/34602", 2024-03-27T21:40:37.971 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/34602", 2024-03-27T21:40:37.971 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/34602", 2024-03-27T21:40:37.971 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2024-03-27T21:40:37.971 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2024-03-27T21:40:37.971 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2024-03-27T21:40:37.971 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.971 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2024-03-27T21:40:37.971 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "b8ffc48b-cf6f-4701-a6aa-c25fdb0f7977", 2024-03-27T21:40:37.972 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2024-03-27T21:40:37.972 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2024-03-27T21:40:37.972 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2024-03-27T21:40:37.972 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2024-03-27T21:40:37.972 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 14, 2024-03-27T21:40:37.972 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 21, 2024-03-27T21:40:37.972 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 33, 2024-03-27T21:40:37.972 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 33, 2024-03-27T21:40:37.972 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 22, 2024-03-27T21:40:37.972 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2024-03-27T21:40:37.972 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2024-03-27T21:40:37.972 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.972 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.972 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2024-03-27T21:40:37.972 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 44654 2024-03-27T21:40:37.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.973 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.973 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2024-03-27T21:40:37.973 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 44654 2024-03-27T21:40:37.973 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.973 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.973 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2024-03-27T21:40:37.973 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.973 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.973 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2024-03-27T21:40:37.973 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 44654 2024-03-27T21:40:37.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.974 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.974 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2024-03-27T21:40:37.974 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 44654 2024-03-27T21:40:37.974 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.974 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.974 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2024-03-27T21:40:37.974 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.974 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.974 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2024-03-27T21:40:37.974 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 44654 2024-03-27T21:40:37.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.975 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.975 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2024-03-27T21:40:37.975 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 44654 2024-03-27T21:40:37.975 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.975 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.975 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2024-03-27T21:40:37.975 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.975 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.975 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2024-03-27T21:40:37.975 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 44654 2024-03-27T21:40:37.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.976 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.976 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2024-03-27T21:40:37.976 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 44654 2024-03-27T21:40:37.976 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.976 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.976 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/44654", 2024-03-27T21:40:37.976 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/44654", 2024-03-27T21:40:37.976 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/44654", 2024-03-27T21:40:37.976 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/44654", 2024-03-27T21:40:37.976 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2024-03-27T21:40:37.976 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2024-03-27T21:40:37.976 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2024-03-27T21:40:37.976 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.977 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2024-03-27T21:40:37.977 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "0f14f2d8-b271-48e4-a198-aa5279a01c06", 2024-03-27T21:40:37.977 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2024-03-27T21:40:37.977 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2024-03-27T21:40:37.977 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2024-03-27T21:40:37.977 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2024-03-27T21:40:37.977 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2024-03-27T21:40:37.977 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2024-03-27T21:40:37.977 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 16, 2024-03-27T21:40:37.977 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 22, 2024-03-27T21:40:37.977 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2024-03-27T21:40:37.977 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2024-03-27T21:40:37.977 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2024-03-27T21:40:37.978 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.978 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.978 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2024-03-27T21:40:37.978 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34613 2024-03-27T21:40:37.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.978 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.978 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2024-03-27T21:40:37.978 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34613 2024-03-27T21:40:37.978 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.978 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.978 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2024-03-27T21:40:37.979 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.979 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.979 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2024-03-27T21:40:37.979 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34613 2024-03-27T21:40:37.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.979 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.979 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2024-03-27T21:40:37.979 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34613 2024-03-27T21:40:37.979 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.979 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.979 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2024-03-27T21:40:37.980 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.980 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.980 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2024-03-27T21:40:37.980 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34613 2024-03-27T21:40:37.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.980 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.980 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2024-03-27T21:40:37.980 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34613 2024-03-27T21:40:37.980 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.980 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.980 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2024-03-27T21:40:37.980 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.981 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.981 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2024-03-27T21:40:37.981 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34613 2024-03-27T21:40:37.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.981 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.981 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2024-03-27T21:40:37.981 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34613 2024-03-27T21:40:37.981 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.981 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.981 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/34613", 2024-03-27T21:40:37.981 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/34613", 2024-03-27T21:40:37.981 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6815/34613", 2024-03-27T21:40:37.982 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/34613", 2024-03-27T21:40:37.982 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2024-03-27T21:40:37.982 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2024-03-27T21:40:37.982 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2024-03-27T21:40:37.982 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.982 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2024-03-27T21:40:37.982 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8cdeb7ae-1c6e-4d0d-ba44-80248eff2fb6", 2024-03-27T21:40:37.982 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2024-03-27T21:40:37.982 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2024-03-27T21:40:37.982 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2024-03-27T21:40:37.982 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2024-03-27T21:40:37.982 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2024-03-27T21:40:37.982 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2024-03-27T21:40:37.983 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 14, 2024-03-27T21:40:37.983 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 0, 2024-03-27T21:40:37.983 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2024-03-27T21:40:37.983 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2024-03-27T21:40:37.983 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2024-03-27T21:40:37.983 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.983 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.983 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6800", 2024-03-27T21:40:37.983 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34332 2024-03-27T21:40:37.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.983 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.983 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6801", 2024-03-27T21:40:37.983 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34332 2024-03-27T21:40:37.984 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.984 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.984 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2024-03-27T21:40:37.984 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.984 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.984 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6802", 2024-03-27T21:40:37.984 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34332 2024-03-27T21:40:37.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.984 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.984 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6803", 2024-03-27T21:40:37.984 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34332 2024-03-27T21:40:37.984 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.984 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.985 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2024-03-27T21:40:37.985 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.985 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.985 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6806", 2024-03-27T21:40:37.985 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34332 2024-03-27T21:40:37.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.985 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.985 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6807", 2024-03-27T21:40:37.985 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34332 2024-03-27T21:40:37.985 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.985 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.986 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2024-03-27T21:40:37.986 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.986 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.986 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6804", 2024-03-27T21:40:37.986 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34332 2024-03-27T21:40:37.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.986 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.986 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6805", 2024-03-27T21:40:37.986 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34332 2024-03-27T21:40:37.986 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.986 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.987 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.191:6801/34332", 2024-03-27T21:40:37.987 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.191:6803/34332", 2024-03-27T21:40:37.987 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.191:6807/34332", 2024-03-27T21:40:37.987 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.191:6805/34332", 2024-03-27T21:40:37.987 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2024-03-27T21:40:37.987 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2024-03-27T21:40:37.987 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2024-03-27T21:40:37.987 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.987 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2024-03-27T21:40:37.987 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "93748cf8-d3fe-46b8-950f-e01e28a5e02c", 2024-03-27T21:40:37.987 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2024-03-27T21:40:37.987 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2024-03-27T21:40:37.988 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2024-03-27T21:40:37.988 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2024-03-27T21:40:37.988 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2024-03-27T21:40:37.988 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2024-03-27T21:40:37.988 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 16, 2024-03-27T21:40:37.988 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 33, 2024-03-27T21:40:37.988 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2024-03-27T21:40:37.988 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2024-03-27T21:40:37.988 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2024-03-27T21:40:37.988 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.988 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.988 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6809", 2024-03-27T21:40:37.988 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34333 2024-03-27T21:40:37.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.989 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.989 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6811", 2024-03-27T21:40:37.989 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34333 2024-03-27T21:40:37.989 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.989 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.989 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2024-03-27T21:40:37.989 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.989 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.989 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6813", 2024-03-27T21:40:37.989 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34333 2024-03-27T21:40:37.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.990 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.990 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6816", 2024-03-27T21:40:37.990 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34333 2024-03-27T21:40:37.990 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.990 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.990 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2024-03-27T21:40:37.990 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.990 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.990 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6822", 2024-03-27T21:40:37.990 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34333 2024-03-27T21:40:37.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.991 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.991 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6823", 2024-03-27T21:40:37.991 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34333 2024-03-27T21:40:37.991 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.991 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.991 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2024-03-27T21:40:37.991 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.991 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.991 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6818", 2024-03-27T21:40:37.991 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34333 2024-03-27T21:40:37.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.992 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.992 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6820", 2024-03-27T21:40:37.992 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34333 2024-03-27T21:40:37.992 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.992 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.992 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.191:6811/34333", 2024-03-27T21:40:37.992 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.191:6816/34333", 2024-03-27T21:40:37.992 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.191:6823/34333", 2024-03-27T21:40:37.992 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.191:6820/34333", 2024-03-27T21:40:37.992 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2024-03-27T21:40:37.992 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2024-03-27T21:40:37.992 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2024-03-27T21:40:37.992 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.993 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2024-03-27T21:40:37.993 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "1ad21434-cbc5-4de9-8d63-76f894f7e736", 2024-03-27T21:40:37.993 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2024-03-27T21:40:37.993 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2024-03-27T21:40:37.993 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2024-03-27T21:40:37.993 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2024-03-27T21:40:37.993 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2024-03-27T21:40:37.993 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2024-03-27T21:40:37.993 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 14, 2024-03-27T21:40:37.993 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 19, 2024-03-27T21:40:37.993 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2024-03-27T21:40:37.993 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2024-03-27T21:40:37.993 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2024-03-27T21:40:37.994 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.994 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.994 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6824", 2024-03-27T21:40:37.994 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34361 2024-03-27T21:40:37.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.994 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.994 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6825", 2024-03-27T21:40:37.994 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34361 2024-03-27T21:40:37.994 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.994 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.994 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2024-03-27T21:40:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6826", 2024-03-27T21:40:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34361 2024-03-27T21:40:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6827", 2024-03-27T21:40:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34361 2024-03-27T21:40:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2024-03-27T21:40:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6830", 2024-03-27T21:40:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34361 2024-03-27T21:40:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6831", 2024-03-27T21:40:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34361 2024-03-27T21:40:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2024-03-27T21:40:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6828", 2024-03-27T21:40:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34361 2024-03-27T21:40:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6829", 2024-03-27T21:40:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34361 2024-03-27T21:40:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.191:6825/34361", 2024-03-27T21:40:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.191:6827/34361", 2024-03-27T21:40:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.191:6831/34361", 2024-03-27T21:40:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.191:6829/34361", 2024-03-27T21:40:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2024-03-27T21:40:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2024-03-27T21:40:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2024-03-27T21:40:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2024-03-27T21:40:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "9fbd0927-ee2f-471b-8df8-c4e0fc40b9b0", 2024-03-27T21:40:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2024-03-27T21:40:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2024-03-27T21:40:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2024-03-27T21:40:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2024-03-27T21:40:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2024-03-27T21:40:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2024-03-27T21:40:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 14, 2024-03-27T21:40:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 19, 2024-03-27T21:40:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2024-03-27T21:40:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2024-03-27T21:40:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2024-03-27T21:40:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6808", 2024-03-27T21:40:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34334 2024-03-27T21:40:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6810", 2024-03-27T21:40:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34334 2024-03-27T21:40:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2024-03-27T21:40:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6812", 2024-03-27T21:40:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34334 2024-03-27T21:40:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6814", 2024-03-27T21:40:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34334 2024-03-27T21:40:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2024-03-27T21:40:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6819", 2024-03-27T21:40:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34334 2024-03-27T21:40:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6821", 2024-03-27T21:40:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34334 2024-03-27T21:40:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2024-03-27T21:40:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6815", 2024-03-27T21:40:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34334 2024-03-27T21:40:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6817", 2024-03-27T21:40:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34334 2024-03-27T21:40:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.191:6810/34334", 2024-03-27T21:40:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.191:6814/34334", 2024-03-27T21:40:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.191:6821/34334", 2024-03-27T21:40:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.191:6817/34334", 2024-03-27T21:40:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2024-03-27T21:40:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2024-03-27T21:40:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2024-03-27T21:40:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-27T21:40:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2024-03-27T21:40:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2024-03-27T21:40:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2024-03-27T21:40:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2024-03-27T21:40:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2024-03-27T21:40:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138322906710015, 2024-03-27T21:40:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2024-03-27T21:40:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2024-03-27T21:39:34.476097+0000", 2024-03-27T21:40:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2024-03-27T21:40:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2024-03-27T21:40:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2024-03-27T21:40:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2024-03-27T21:40:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2024-03-27T21:40:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138322906710015, 2024-03-27T21:40:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2024-03-27T21:40:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2024-03-27T21:39:34.411243+0000", 2024-03-27T21:40:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2024-03-27T21:40:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2024-03-27T21:40:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2024-03-27T21:40:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2024-03-27T21:40:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2024-03-27T21:40:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138322906710015, 2024-03-27T21:40:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2024-03-27T21:40:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2024-03-27T21:39:34.458154+0000", 2024-03-27T21:40:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 22 2024-03-27T21:40:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2024-03-27T21:40:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2024-03-27T21:40:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2024-03-27T21:40:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2024-03-27T21:40:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138322906710015, 2024-03-27T21:40:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2024-03-27T21:40:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2024-03-27T21:39:34.415285+0000", 2024-03-27T21:40:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2024-03-27T21:40:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2024-03-27T21:40:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2024-03-27T21:40:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2024-03-27T21:40:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2024-03-27T21:40:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138322906710015, 2024-03-27T21:40:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2024-03-27T21:40:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2024-03-27T21:39:34.431201+0000", 2024-03-27T21:40:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2024-03-27T21:40:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2024-03-27T21:40:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2024-03-27T21:40:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2024-03-27T21:40:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2024-03-27T21:40:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138322906710015, 2024-03-27T21:40:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2024-03-27T21:40:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2024-03-27T21:39:34.456013+0000", 2024-03-27T21:40:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2024-03-27T21:40:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2024-03-27T21:40:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2024-03-27T21:40:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2024-03-27T21:40:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2024-03-27T21:40:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138322906710015, 2024-03-27T21:40:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2024-03-27T21:40:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2024-03-27T21:39:34.474448+0000", 2024-03-27T21:40:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2024-03-27T21:40:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2024-03-27T21:40:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2024-03-27T21:40:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2024-03-27T21:40:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2024-03-27T21:40:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138322906710015, 2024-03-27T21:40:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2024-03-27T21:40:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2024-03-27T21:39:34.414076+0000", 2024-03-27T21:40:38.010 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2024-03-27T21:40:38.010 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:38.010 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-27T21:40:38.010 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2024-03-27T21:40:38.010 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2024-03-27T21:40:38.010 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_primaries": [], 2024-03-27T21:40:38.010 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2024-03-27T21:40:38.010 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2024-03-27T21:40:38.010 INFO:teuthology.orchestra.run.smithi060.stdout: "blocklist": {}, 2024-03-27T21:40:38.010 INFO:teuthology.orchestra.run.smithi060.stdout: "range_blocklist": {}, 2024-03-27T21:40:38.010 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2024-03-27T21:40:38.010 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2024-03-27T21:40:38.010 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2024-03-27T21:40:38.011 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2024-03-27T21:40:38.011 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2024-03-27T21:40:38.011 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2024-03-27T21:40:38.011 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2024-03-27T21:40:38.011 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:38.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:38.011 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2024-03-27T21:40:38.011 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2024-03-27T21:40:38.011 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2024-03-27T21:40:38.011 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2024-03-27T21:40:38.011 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {}, 2024-03-27T21:40:38.011 INFO:teuthology.orchestra.run.smithi060.stdout: "stretch_mode": { 2024-03-27T21:40:38.011 INFO:teuthology.orchestra.run.smithi060.stdout: "stretch_mode_enabled": false, 2024-03-27T21:40:38.011 INFO:teuthology.orchestra.run.smithi060.stdout: "stretch_bucket_count": 0, 2024-03-27T21:40:38.012 INFO:teuthology.orchestra.run.smithi060.stdout: "degraded_stretch_mode": 0, 2024-03-27T21:40:38.012 INFO:teuthology.orchestra.run.smithi060.stdout: "recovering_stretch_mode": 0, 2024-03-27T21:40:38.012 INFO:teuthology.orchestra.run.smithi060.stdout: "stretch_mode_bucket": 0 2024-03-27T21:40:38.012 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:38.012 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:38.012 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:40:38.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.960+0000 7f08aaeb6640 10 librados: watch_flush enter 2024-03-27T21:40:38.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.960+0000 7f08aaeb6640 10 librados: watch_flush exit 2024-03-27T21:40:38.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.960+0000 7f08aaeb6640 20 client.4645.objecter shutdown clearing up homeless session... 2024-03-27T21:40:38.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.960+0000 7f08aaeb6640 10 client.4645.objecter successfully canceled tick 2024-03-27T21:40:38.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.960+0000 7f08aaeb6640 1 -- 172.21.15.60:0/2822571180 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f088806c5c0 msgr2=0x7f088806ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:38.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.960+0000 7f08aaeb6640 1 --2- 172.21.15.60:0/2822571180 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f088806c5c0 0x7f088806ea80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0898009fd0 tx=0x7f0898009190 comp rx=0 tx=0).stop 2024-03-27T21:40:38.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.960+0000 7f08aaeb6640 10 monclient: shutdown 2024-03-27T21:40:38.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.960+0000 7f08aaeb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:38.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.960+0000 7f08aaeb6640 1 -- 172.21.15.60:0/2822571180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08a4141630 msgr2=0x7f08a41d6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:38.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.960+0000 7f08aaeb6640 1 --2- 172.21.15.60:0/2822571180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08a4141630 0x7f08a41d6710 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f089c00c780 tx=0x7f089c00cc40 comp rx=0 tx=0).stop 2024-03-27T21:40:38.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.960+0000 7f08aaeb6640 1 -- 172.21.15.60:0/2822571180 shutdown_connections 2024-03-27T21:40:38.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.960+0000 7f08aaeb6640 1 --2- 172.21.15.60:0/2822571180 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f088806c5c0 0x7f088806ea80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.960+0000 7f08aaeb6640 1 --2- 172.21.15.60:0/2822571180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08a4141630 0x7f08a41d6710 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.960+0000 7f08aaeb6640 1 --2- 172.21.15.60:0/2822571180 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f08a413e810 0x7f08a4139180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.960+0000 7f08aaeb6640 1 --2- 172.21.15.60:0/2822571180 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08a4071a50 0x7f08a4136190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.960+0000 7f08aaeb6640 1 -- 172.21.15.60:0/2822571180 >> 172.21.15.60:0/2822571180 conn(0x7f08a40b7300 msgr2=0x7f08a413dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:38.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.960+0000 7f08aaeb6640 1 -- 172.21.15.60:0/2822571180 shutdown_connections 2024-03-27T21:40:38.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.960+0000 7f08aaeb6640 1 -- 172.21.15.60:0/2822571180 wait complete. 2024-03-27T21:40:38.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:37.960+0000 7f08aaeb6640 1 librados: shutdown 2024-03-27T21:40:38.016 INFO:tasks.thrashosds.thrasher:j is {'epoch': 40, 'fsid': 'dcd19d9a-535c-4b41-9095-d588268e2860', 'created': '2024-03-27T21:39:27.092191+0000', 'modified': '2024-03-27T21:40:35.449663+0000', 'last_up_change': '2024-03-27T21:40:20.406185+0000', 'last_in_change': '2024-03-27T21:39:29.291620+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-27T21:39:37.438493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-03-27T21:39:45.042103+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '778aaa64-6720-495b-bcdc-86fbcece9d58', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6816', 'nonce': 34603}, {'type': 'v1', 'addr': '172.21.15.60:6817', 'nonce': 34603}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6818', 'nonce': 34603}, {'type': 'v1', 'addr': '172.21.15.60:6819', 'nonce': 34603}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6822', 'nonce': 34603}, {'type': 'v1', 'addr': '172.21.15.60:6823', 'nonce': 34603}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6820', 'nonce': 34603}, {'type': 'v1', 'addr': '172.21.15.60:6821', 'nonce': 34603}]}, 'public_addr': '172.21.15.60:6817/34603', 'cluster_addr': '172.21.15.60:6819/34603', 'heartbeat_back_addr': '172.21.15.60:6823/34603', 'heartbeat_front_addr': '172.21.15.60:6821/34603', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'd960adf1-3d83-4e24-a612-8df53fa43d0c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6800', 'nonce': 34602}, {'type': 'v1', 'addr': '172.21.15.60:6801', 'nonce': 34602}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6802', 'nonce': 34602}, {'type': 'v1', 'addr': '172.21.15.60:6803', 'nonce': 34602}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6806', 'nonce': 34602}, {'type': 'v1', 'addr': '172.21.15.60:6807', 'nonce': 34602}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6804', 'nonce': 34602}, {'type': 'v1', 'addr': '172.21.15.60:6805', 'nonce': 34602}]}, 'public_addr': '172.21.15.60:6801/34602', 'cluster_addr': '172.21.15.60:6803/34602', 'heartbeat_back_addr': '172.21.15.60:6807/34602', 'heartbeat_front_addr': '172.21.15.60:6805/34602', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b8ffc48b-cf6f-4701-a6aa-c25fdb0f7977', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 14, 'last_clean_end': 21, 'up_from': 33, 'up_thru': 33, 'down_at': 22, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6824', 'nonce': 44654}, {'type': 'v1', 'addr': '172.21.15.60:6825', 'nonce': 44654}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6826', 'nonce': 44654}, {'type': 'v1', 'addr': '172.21.15.60:6827', 'nonce': 44654}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6830', 'nonce': 44654}, {'type': 'v1', 'addr': '172.21.15.60:6831', 'nonce': 44654}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6828', 'nonce': 44654}, {'type': 'v1', 'addr': '172.21.15.60:6829', 'nonce': 44654}]}, 'public_addr': '172.21.15.60:6825/44654', 'cluster_addr': '172.21.15.60:6827/44654', 'heartbeat_back_addr': '172.21.15.60:6831/44654', 'heartbeat_front_addr': '172.21.15.60:6829/44654', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '0f14f2d8-b271-48e4-a198-aa5279a01c06', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6808', 'nonce': 34613}, {'type': 'v1', 'addr': '172.21.15.60:6809', 'nonce': 34613}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6810', 'nonce': 34613}, {'type': 'v1', 'addr': '172.21.15.60:6811', 'nonce': 34613}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6814', 'nonce': 34613}, {'type': 'v1', 'addr': '172.21.15.60:6815', 'nonce': 34613}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6812', 'nonce': 34613}, {'type': 'v1', 'addr': '172.21.15.60:6813', 'nonce': 34613}]}, 'public_addr': '172.21.15.60:6809/34613', 'cluster_addr': '172.21.15.60:6811/34613', 'heartbeat_back_addr': '172.21.15.60:6815/34613', 'heartbeat_front_addr': '172.21.15.60:6813/34613', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '8cdeb7ae-1c6e-4d0d-ba44-80248eff2fb6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6800', 'nonce': 34332}, {'type': 'v1', 'addr': '172.21.15.191:6801', 'nonce': 34332}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6802', 'nonce': 34332}, {'type': 'v1', 'addr': '172.21.15.191:6803', 'nonce': 34332}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6806', 'nonce': 34332}, {'type': 'v1', 'addr': '172.21.15.191:6807', 'nonce': 34332}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6804', 'nonce': 34332}, {'type': 'v1', 'addr': '172.21.15.191:6805', 'nonce': 34332}]}, 'public_addr': '172.21.15.191:6801/34332', 'cluster_addr': '172.21.15.191:6803/34332', 'heartbeat_back_addr': '172.21.15.191:6807/34332', 'heartbeat_front_addr': '172.21.15.191:6805/34332', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '93748cf8-d3fe-46b8-950f-e01e28a5e02c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 33, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6809', 'nonce': 34333}, {'type': 'v1', 'addr': '172.21.15.191:6811', 'nonce': 34333}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6813', 'nonce': 34333}, {'type': 'v1', 'addr': '172.21.15.191:6816', 'nonce': 34333}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6822', 'nonce': 34333}, {'type': 'v1', 'addr': '172.21.15.191:6823', 'nonce': 34333}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6818', 'nonce': 34333}, {'type': 'v1', 'addr': '172.21.15.191:6820', 'nonce': 34333}]}, 'public_addr': '172.21.15.191:6811/34333', 'cluster_addr': '172.21.15.191:6816/34333', 'heartbeat_back_addr': '172.21.15.191:6823/34333', 'heartbeat_front_addr': '172.21.15.191:6820/34333', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '1ad21434-cbc5-4de9-8d63-76f894f7e736', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6824', 'nonce': 34361}, {'type': 'v1', 'addr': '172.21.15.191:6825', 'nonce': 34361}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6826', 'nonce': 34361}, {'type': 'v1', 'addr': '172.21.15.191:6827', 'nonce': 34361}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6830', 'nonce': 34361}, {'type': 'v1', 'addr': '172.21.15.191:6831', 'nonce': 34361}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6828', 'nonce': 34361}, {'type': 'v1', 'addr': '172.21.15.191:6829', 'nonce': 34361}]}, 'public_addr': '172.21.15.191:6825/34361', 'cluster_addr': '172.21.15.191:6827/34361', 'heartbeat_back_addr': '172.21.15.191:6831/34361', 'heartbeat_front_addr': '172.21.15.191:6829/34361', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '9fbd0927-ee2f-471b-8df8-c4e0fc40b9b0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6808', 'nonce': 34334}, {'type': 'v1', 'addr': '172.21.15.191:6810', 'nonce': 34334}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6812', 'nonce': 34334}, {'type': 'v1', 'addr': '172.21.15.191:6814', 'nonce': 34334}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6819', 'nonce': 34334}, {'type': 'v1', 'addr': '172.21.15.191:6821', 'nonce': 34334}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6815', 'nonce': 34334}, {'type': 'v1', 'addr': '172.21.15.191:6817', 'nonce': 34334}]}, 'public_addr': '172.21.15.191:6810/34334', 'cluster_addr': '172.21.15.191:6814/34334', 'heartbeat_back_addr': '172.21.15.191:6821/34334', 'heartbeat_front_addr': '172.21.15.191:6817/34334', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-27T21:39:34.476097+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-27T21:39:34.411243+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-27T21:39:34.458154+0000', 'dead_epoch': 22}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-27T21:39:34.415285+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-27T21:39:34.431201+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-27T21:39:34.456013+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-27T21:39:34.474448+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-27T21:39:34.414076+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-27T21:40:38.017 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-03-27T21:40:38.021 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:38.034+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:38.022 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:38.022 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:38.033 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:40:38.122 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:38.135+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:38.178 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:38.179 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:38.187 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:40:38.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.206+0000 7fcd9d84a640 10 monclient: get_monmap_and_config 2024-03-27T21:40:38.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.206+0000 7fcd9d84a640 10 monclient: build_initial_monmap 2024-03-27T21:40:38.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.206+0000 7fcd9d84a640 10 monclient: monmap: 2024-03-27T21:40:38.193 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:38.193 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:38.193 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:38.207475+0000 2024-03-27T21:40:38.193 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:38.207475+0000 2024-03-27T21:40:38.193 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:38.193 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:38.193 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:38.193 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:38.193 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:38.193 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:38.193 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:38.193 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:38.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.207+0000 7fcd9d84a640 1 Processor -- start 2024-03-27T21:40:38.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.207+0000 7fcd9d84a640 1 -- start start 2024-03-27T21:40:38.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.207+0000 7fcd9d84a640 10 monclient: init 2024-03-27T21:40:38.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd9d84a640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:38.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd9d84a640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-03-27T21:40:38.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd9d84a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcd980b40b0 0x7fcd980a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:38.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd9d84a640 10 monclient(hunting): picked mon.noname-c con 0x7fcd980b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:38.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd9d84a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcd9813b020 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:38.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd9d84a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcd9813b480 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:38.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd9d84a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd9813b8d0 con 0x7fcd9813b480 2024-03-27T21:40:38.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd9d84a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd980b3830 con 0x7fcd9813b020 2024-03-27T21:40:38.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd9d84a640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:38.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd9d84a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd980b4490 con 0x7fcd980b40b0 2024-03-27T21:40:38.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd9d84a640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:38.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd9d84a640 10 monclient(hunting): authenticate will time out at 879.429328s 2024-03-27T21:40:38.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd977fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcd9813b480 0x7fcd9814bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60034/0 (socket says 172.21.15.60:60034) 2024-03-27T21:40:38.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd977fe640 1 -- 172.21.15.60:0/2848875469 learned_addr learned my addr 172.21.15.60:0/2848875469 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:40:38.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd96ffd640 1 --2- 172.21.15.60:0/2848875469 >> v2:172.21.15.191:3300/0 conn(0x7fcd980b40b0 0x7fcd980a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:38.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd96ffd640 10 monclient(hunting): get_auth_request con 0x7fcd980b40b0 auth_method 0 2024-03-27T21:40:38.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:38.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:38.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:38.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd96ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:38.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd96ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:38.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd96ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:38.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd95ffb640 1 -- 172.21.15.60:0/2848875469 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076484194 0 0) 0x7fcd9813b8d0 con 0x7fcd9813b480 2024-03-27T21:40:38.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd95ffb640 10 monclient(hunting): _negotiate 2024-03-27T21:40:38.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd95ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:38.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:38.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd95ffb640 10 monclient(hunting): my global_id is 4707 2024-03-27T21:40:38.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd95ffb640 1 -- 172.21.15.60:0/2848875469 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd80003520 con 0x7fcd9813b480 2024-03-27T21:40:38.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd95ffb640 1 -- 172.21.15.60:0/2848875469 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494530171 0 0) 0x7fcd980b3830 con 0x7fcd9813b020 2024-03-27T21:40:38.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd95ffb640 10 monclient(hunting): _negotiate 2024-03-27T21:40:38.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd95ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:38.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:38.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd95ffb640 10 monclient(hunting): my global_id is 4651 2024-03-27T21:40:38.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd95ffb640 1 -- 172.21.15.60:0/2848875469 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd9813b8d0 con 0x7fcd9813b020 2024-03-27T21:40:38.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd96ffd640 10 monclient(hunting): handle_auth_done global_id 4648 payload 274 2024-03-27T21:40:38.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd96ffd640 1 -- 172.21.15.60:0/2848875469 >> v1:172.21.15.191:6789/0 conn(0x7fcd9813b020 legacy=0x7fcd981416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:38.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd96ffd640 1 -- 172.21.15.60:0/2848875469 >> v1:172.21.15.60:6789/0 conn(0x7fcd9813b480 legacy=0x7fcd9814bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:38.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd96ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:40:38.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd96ffd640 1 monclient: found mon.noname-c 2024-03-27T21:40:38.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:38.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd96ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:40:38.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.208+0000 7fcd96ffd640 1 -- 172.21.15.60:0/2848875469 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd980aa520 con 0x7fcd980b40b0 2024-03-27T21:40:38.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.209+0000 7fcd96ffd640 1 --2- 172.21.15.60:0/2848875469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd980b40b0 0x7fcd980a8b90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcd880097a0 tx=0x7fcd880340e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b59db12c3141c3d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:38.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.209+0000 7fcd95ffb640 1 -- 172.21.15.60:0/2848875469 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd88034e90 con 0x7fcd980b40b0 2024-03-27T21:40:38.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.209+0000 7fcd95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:38.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd95ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:38.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd95ffb640 10 monclient: dump: 2024-03-27T21:40:38.208 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:38.208 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:38.208 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:38.208 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:38.208 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:38.208 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:38.209 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:38.209 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:38.209 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:38.209 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:38.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd95ffb640 10 monclient: _finish_auth 0 2024-03-27T21:40:38.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd95ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:40:38.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:38.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd9d84a640 5 monclient: authenticate success, global_id 4648 2024-03-27T21:40:38.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd9d84a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:38.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd95ffb640 1 -- 172.21.15.60:0/2848875469 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8803c440 con 0x7fcd980b40b0 2024-03-27T21:40:38.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd95ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:38.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd9d84a640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:38.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd9d84a640 4 set_mon_vals no callback set 2024-03-27T21:40:38.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd95ffb640 1 -- 172.21.15.60:0/2848875469 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8803c7b0 con 0x7fcd980b40b0 2024-03-27T21:40:38.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:38.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd95ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:40:38.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd95ffb640 10 monclient: dump: 2024-03-27T21:40:38.210 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:38.210 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:38.210 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:38.210 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:38.210 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:38.210 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:38.210 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:38.210 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:38.210 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:38.210 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:38.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd9d84a640 10 monclient: shutdown 2024-03-27T21:40:38.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd9d84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:38.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd9d84a640 1 -- 172.21.15.60:0/2848875469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd980b40b0 msgr2=0x7fcd980a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:38.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/2848875469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd980b40b0 0x7fcd980a8b90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcd880097a0 tx=0x7fcd880340e0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd9d84a640 1 -- 172.21.15.60:0/2848875469 shutdown_connections 2024-03-27T21:40:38.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/2848875469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd980b40b0 0x7fcd980a8b90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd9d84a640 1 -- 172.21.15.60:0/2848875469 >> 172.21.15.60:0/2848875469 conn(0x7fcd980608e0 msgr2=0x7fcd98060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:38.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd9d84a640 1 -- 172.21.15.60:0/2848875469 shutdown_connections 2024-03-27T21:40:38.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.210+0000 7fcd9d84a640 1 -- 172.21.15.60:0/2848875469 wait complete. 2024-03-27T21:40:38.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.211+0000 7fcd9d84a640 10 monclient: build_initial_monmap 2024-03-27T21:40:38.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.211+0000 7fcd9d84a640 10 monclient: monmap: 2024-03-27T21:40:38.211 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:38.211 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:38.211 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:38.211 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:38.211 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:38.212 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:38.212 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:38.212 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:38.212 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:38.212 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:38.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.211+0000 7fcd9d84a640 1 librados: starting msgr at 2024-03-27T21:40:38.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.211+0000 7fcd9d84a640 1 librados: starting objecter 2024-03-27T21:40:38.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.211+0000 7fcd9d84a640 1 Processor -- start 2024-03-27T21:40:38.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.211+0000 7fcd9d84a640 1 -- start start 2024-03-27T21:40:38.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.211+0000 7fcd9d84a640 1 librados: setting wanted keys 2024-03-27T21:40:38.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.211+0000 7fcd9d84a640 1 librados: calling monclient init 2024-03-27T21:40:38.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.211+0000 7fcd9d84a640 10 monclient: init 2024-03-27T21:40:38.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd9d84a640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:38.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd9d84a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:40:38.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd9d84a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd980b40b0 0x7fcd9813dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:38.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd9d84a640 10 monclient(hunting): picked mon.noname-b con 0x7fcd980b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:38.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd9d84a640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcd9813b020 0x7fcd981d5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:38.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd9d84a640 10 monclient(hunting): picked mon.noname-c con 0x7fcd9813b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:38.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd9d84a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd9813b480 0x7fcd981362d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:38.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd9d84a640 10 monclient(hunting): picked mon.noname-a con 0x7fcd9813b480 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:38.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd9d84a640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:38.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd96ffd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd980b40b0 0x7fcd9813dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:38.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd96ffd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd980b40b0 0x7fcd9813dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:54702/0 (socket says 172.21.15.60:54702) 2024-03-27T21:40:38.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd967fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcd9813b020 0x7fcd981d5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:38.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd96ffd640 1 -- 172.21.15.60:0/292188241 learned_addr learned my addr 172.21.15.60:0/292188241 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:38.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9813eaa0 con 0x7fcd9813b480 2024-03-27T21:40:38.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd9d84a640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:38.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9806d960 con 0x7fcd980b40b0 2024-03-27T21:40:38.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd9d84a640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:38.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd977fe640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd9813b480 0x7fcd981362d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:38.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9806d430 con 0x7fcd9813b020 2024-03-27T21:40:38.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd9d84a640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:38.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd9d84a640 10 monclient(hunting): authenticate will time out at 879.433835s 2024-03-27T21:40:38.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd967fc640 10 monclient(hunting): get_auth_request con 0x7fcd9813b020 auth_method 0 2024-03-27T21:40:38.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:38.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd967fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:38.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:38.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd96ffd640 10 monclient(hunting): get_auth_request con 0x7fcd980b40b0 auth_method 0 2024-03-27T21:40:38.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:38.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:38.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:38.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.212+0000 7fcd977fe640 10 monclient(hunting): get_auth_request con 0x7fcd9813b480 auth_method 0 2024-03-27T21:40:38.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:38.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd977fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:38.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:38.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:38.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:38.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:38.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd96ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:38.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd96ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:38.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd96ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:38.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:38.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:38.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:38.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd967fc640 10 monclient(hunting): handle_auth_done global_id 4565 payload 824 2024-03-27T21:40:38.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd967fc640 1 -- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd980b40b0 msgr2=0x7fcd9813dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:38.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd967fc640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd980b40b0 0x7fcd9813dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd967fc640 1 -- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd9813b480 msgr2=0x7fcd981362d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:38.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd967fc640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd9813b480 0x7fcd981362d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd967fc640 10 monclient: _finish_hunting 0 2024-03-27T21:40:38.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd967fc640 1 monclient: found mon.noname-c 2024-03-27T21:40:38.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:38.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd967fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:40:38.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd967fc640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd88009400 con 0x7fcd9813b020 2024-03-27T21:40:38.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd967fc640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcd9813b020 0x7fcd981d5480 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c00c3c0 tx=0x7fcd8c00c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:38.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd7ffff640 10 client.?.objecter ms_handle_connect 0x7fcd9813b020 2024-03-27T21:40:38.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd7ffff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:38.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c0052d0 con 0x7fcd9813b020 2024-03-27T21:40:38.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:38.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd7ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:38.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd7ffff640 10 monclient: dump: 2024-03-27T21:40:38.217 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:38.217 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:38.217 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:38.217 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:38.217 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:38.217 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:38.217 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:38.217 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:38.217 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:38.217 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:38.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd7ffff640 10 monclient: _finish_auth 0 2024-03-27T21:40:38.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd7ffff640 10 monclient: _check_auth_tickets 2024-03-27T21:40:38.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:38.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd9d84a640 5 monclient: authenticate success, global_id 4565 2024-03-27T21:40:38.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd9d84a640 10 monclient: _renew_subs 2024-03-27T21:40:38.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd9d84a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:38.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9806e8f0 con 0x7fcd9813b020 2024-03-27T21:40:38.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd9d84a640 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:38.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd9d84a640 10 monclient: _renew_subs 2024-03-27T21:40:38.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd9d84a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:38.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd98070100 con 0x7fcd9813b020 2024-03-27T21:40:38.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8c007df0 con 0x7fcd9813b020 2024-03-27T21:40:38.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd7ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:38.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd9d84a640 1 librados: init done 2024-03-27T21:40:38.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd957fa640 4 set_mon_vals no callback set 2024-03-27T21:40:38.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c0035f0 con 0x7fcd9813b020 2024-03-27T21:40:38.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:38.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd7ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:38.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.213+0000 7fcd7ffff640 10 monclient: dump: 2024-03-27T21:40:38.219 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:38.219 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:38.219 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:38.219 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:38.219 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:38.219 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:38.219 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:38.219 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:38.219 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:38.219 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:38.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.214+0000 7fcd9d84a640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:38.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.214+0000 7fcd9d84a640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:38.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.214+0000 7fcd9d84a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:38.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.214+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fcd78005020 con 0x7fcd9813b020 2024-03-27T21:40:38.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.215+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcd8c003750 con 0x7fcd9813b020 2024-03-27T21:40:38.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.216+0000 7fcd7ffff640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcd8406c590 0x7fcd8406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:38.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.216+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fcd8c088690 con 0x7fcd9813b020 2024-03-27T21:40:38.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.216+0000 7fcd7ffff640 10 client.4565.objecter ms_dispatch 0x7fcd98000b90 osd_map(40..40 src has 1..40) v4 2024-03-27T21:40:38.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.216+0000 7fcd7ffff640 3 client.4565.objecter handle_osd_map got epochs [40,40] > 0 2024-03-27T21:40:38.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.216+0000 7fcd7ffff640 3 client.4565.objecter handle_osd_map decoding full epoch 40 2024-03-27T21:40:38.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.216+0000 7fcd96ffd640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcd8406c590 0x7fcd8406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:38.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.216+0000 7fcd7ffff640 20 client.4565.objecter dump_active .. 0 homeless 2024-03-27T21:40:38.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.216+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v40) v1 ==== 54+0+16 (secure 0 0 0) 0x7fcd8c08f070 con 0x7fcd9813b020 2024-03-27T21:40:38.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.216+0000 7fcd7ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:38.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.216+0000 7fcd7ffff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:38.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.216+0000 7fcd96ffd640 10 monclient: get_auth_request con 0x7fcd8406c590 auth_method 0 2024-03-27T21:40:38.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.216+0000 7fcd9d84a640 10 client.4565.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:38.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.216+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign -1 1 2024-03-27T21:40:38.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.216+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fcd78004640 0x7fcd78004a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:38.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.217+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd780052c0 osd=0 3 2024-03-27T21:40:38.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.217+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd780052c0 osd=0 3 2024-03-27T21:40:38.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.217+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd780052c0 osd=0 3 2024-03-27T21:40:38.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.217+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_remove -1 1 2024-03-27T21:40:38.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.217+0000 7fcd9d84a640 20 client.4565.objecter get_session s=0x7fcd780052c0 osd=0 3 2024-03-27T21:40:38.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.217+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign 0 1 2024-03-27T21:40:38.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.217+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd780052c0 osd=0 4 2024-03-27T21:40:38.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.217+0000 7fcd9d84a640 10 client.4565.objecter _send_command 1 2024-03-27T21:40:38.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.217+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd78008a50 con 0x7fcd78004640 2024-03-27T21:40:38.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.217+0000 7fcd977fe640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fcd78004640 0x7fcd78004a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:38.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.217+0000 7fcd977fe640 10 monclient: get_auth_request con 0x7fcd78004640 auth_method 0 2024-03-27T21:40:38.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.217+0000 7fcd7ffff640 10 client.4565.objecter ms_handle_connect 0x7fcd8406c590 2024-03-27T21:40:38.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.217+0000 7fcd96ffd640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcd8406c590 0x7fcd8406ea50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcd880097a0 tx=0x7fcd88038000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:38.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.217+0000 7fcd7ffff640 10 client.4565.objecter ms_handle_connect 0x7fcd78004640 2024-03-27T21:40:38.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.217+0000 7fcd977fe640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fcd78004640 0x7fcd78004a80 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:38.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.218+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fcd78008a50 con 0x7fcd78004640 2024-03-27T21:40:38.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.218+0000 7fcd7ffff640 10 client.4565.objecter ms_dispatch 0x7fcd98000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:38.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.218+0000 7fcd7ffff640 10 client.4565.objecter _finish_command 1 = system:0 2024-03-27T21:40:38.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.218+0000 7fcd7ffff640 20 client.4565.objecter put_session s=0x7fcd780052c0 osd=0 4 2024-03-27T21:40:38.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.218+0000 7fcd7ffff640 15 client.4565.objecter _session_command_op_remove 0 1 2024-03-27T21:40:38.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.232+0000 7fcd9d84a640 10 client.4565.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:38.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.232+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign -1 2 2024-03-27T21:40:38.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.232+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd780052c0 osd=0 3 2024-03-27T21:40:38.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.232+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd780052c0 osd=0 3 2024-03-27T21:40:38.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.232+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd780052c0 osd=0 3 2024-03-27T21:40:38.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.232+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_remove -1 2 2024-03-27T21:40:38.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.232+0000 7fcd9d84a640 20 client.4565.objecter get_session s=0x7fcd780052c0 osd=0 3 2024-03-27T21:40:38.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.232+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign 0 2 2024-03-27T21:40:38.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.232+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd780052c0 osd=0 4 2024-03-27T21:40:38.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.232+0000 7fcd9d84a640 10 client.4565.objecter _send_command 2 2024-03-27T21:40:38.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.232+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcd780040f0 con 0x7fcd78004640 2024-03-27T21:40:38.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.234+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fcd780040f0 con 0x7fcd78004640 2024-03-27T21:40:38.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.234+0000 7fcd7ffff640 10 client.4565.objecter ms_dispatch 0x7fcd98000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:38.224 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.234+0000 7fcd7ffff640 10 client.4565.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.234+0000 7fcd7ffff640 20 client.4565.objecter put_session s=0x7fcd780052c0 osd=0 4 2024-03-27T21:40:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.234+0000 7fcd7ffff640 15 client.4565.objecter _session_command_op_remove 0 2 2024-03-27T21:40:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.234+0000 7fcd9d84a640 10 client.4565.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.234+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign -1 3 2024-03-27T21:40:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.234+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fcd78002700 0x7fcd78002b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.234+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd78008090 osd=1 3 2024-03-27T21:40:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.234+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78008090 osd=1 3 2024-03-27T21:40:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.234+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd78008090 osd=1 3 2024-03-27T21:40:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.234+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_remove -1 3 2024-03-27T21:40:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.234+0000 7fcd9d84a640 20 client.4565.objecter get_session s=0x7fcd78008090 osd=1 3 2024-03-27T21:40:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.234+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign 1 3 2024-03-27T21:40:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.234+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78008090 osd=1 4 2024-03-27T21:40:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.234+0000 7fcd9d84a640 10 client.4565.objecter _send_command 3 2024-03-27T21:40:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.234+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd780030c0 con 0x7fcd78002700 2024-03-27T21:40:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.234+0000 7fcd96ffd640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fcd78002700 0x7fcd78002b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.234+0000 7fcd96ffd640 10 monclient: get_auth_request con 0x7fcd78002700 auth_method 0 2024-03-27T21:40:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.235+0000 7fcd96ffd640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fcd78002700 0x7fcd78002b80 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.235+0000 7fcd7ffff640 10 client.4565.objecter ms_handle_connect 0x7fcd78002700 2024-03-27T21:40:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.236+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fcd780030c0 con 0x7fcd78002700 2024-03-27T21:40:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.236+0000 7fcd7ffff640 10 client.4565.objecter ms_dispatch 0x7fcd98000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.236+0000 7fcd7ffff640 10 client.4565.objecter _finish_command 3 = system:0 2024-03-27T21:40:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.236+0000 7fcd7ffff640 20 client.4565.objecter put_session s=0x7fcd78008090 osd=1 4 2024-03-27T21:40:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.236+0000 7fcd7ffff640 15 client.4565.objecter _session_command_op_remove 1 3 2024-03-27T21:40:38.227 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:38.237+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:38.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.250+0000 7fcd9d84a640 10 client.4565.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:38.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.250+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign -1 4 2024-03-27T21:40:38.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.250+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd78008090 osd=1 3 2024-03-27T21:40:38.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.250+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78008090 osd=1 3 2024-03-27T21:40:38.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.250+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd78008090 osd=1 3 2024-03-27T21:40:38.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.250+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_remove -1 4 2024-03-27T21:40:38.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.250+0000 7fcd9d84a640 20 client.4565.objecter get_session s=0x7fcd78008090 osd=1 3 2024-03-27T21:40:38.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.250+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign 1 4 2024-03-27T21:40:38.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.250+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78008090 osd=1 4 2024-03-27T21:40:38.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.250+0000 7fcd9d84a640 10 client.4565.objecter _send_command 4 2024-03-27T21:40:38.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.250+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcd78003a80 con 0x7fcd78002700 2024-03-27T21:40:38.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.253+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fcd78003a80 con 0x7fcd78002700 2024-03-27T21:40:38.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.253+0000 7fcd7ffff640 10 client.4565.objecter ms_dispatch 0x7fcd98000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:38.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.253+0000 7fcd7ffff640 10 client.4565.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:38.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.253+0000 7fcd7ffff640 20 client.4565.objecter put_session s=0x7fcd78008090 osd=1 4 2024-03-27T21:40:38.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.253+0000 7fcd7ffff640 15 client.4565.objecter _session_command_op_remove 1 4 2024-03-27T21:40:38.240 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:38.241 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:38.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.253+0000 7fcd9d84a640 10 client.4565.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:38.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.253+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign -1 5 2024-03-27T21:40:38.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.253+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7fcd7800b0e0 0x7fcd7800d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:38.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.253+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd78003bf0 osd=2 3 2024-03-27T21:40:38.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.253+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78003bf0 osd=2 3 2024-03-27T21:40:38.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.253+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd78003bf0 osd=2 3 2024-03-27T21:40:38.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.253+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_remove -1 5 2024-03-27T21:40:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.253+0000 7fcd9d84a640 20 client.4565.objecter get_session s=0x7fcd78003bf0 osd=2 3 2024-03-27T21:40:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.253+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign 2 5 2024-03-27T21:40:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.253+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78003bf0 osd=2 4 2024-03-27T21:40:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.253+0000 7fcd9d84a640 10 client.4565.objecter _send_command 5 2024-03-27T21:40:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.253+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd780102a0 con 0x7fcd7800b0e0 2024-03-27T21:40:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.253+0000 7fcd967fc640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7fcd7800b0e0 0x7fcd7800d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.253+0000 7fcd967fc640 10 monclient: get_auth_request con 0x7fcd7800b0e0 auth_method 0 2024-03-27T21:40:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.253+0000 7fcd7ffff640 10 client.4565.objecter ms_handle_connect 0x7fcd7800b0e0 2024-03-27T21:40:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.253+0000 7fcd967fc640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7fcd7800b0e0 0x7fcd7800d4d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.254+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== osd.2 v2:172.21.15.60:6824/44654 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fcd780102a0 con 0x7fcd7800b0e0 2024-03-27T21:40:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.254+0000 7fcd7ffff640 10 client.4565.objecter ms_dispatch 0x7fcd98000b90 command_reply(tid 5: 0 ) v1 2024-03-27T21:40:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.254+0000 7fcd7ffff640 10 client.4565.objecter _finish_command 5 = system:0 2024-03-27T21:40:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.254+0000 7fcd7ffff640 20 client.4565.objecter put_session s=0x7fcd78003bf0 osd=2 4 2024-03-27T21:40:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.254+0000 7fcd7ffff640 15 client.4565.objecter _session_command_op_remove 2 5 2024-03-27T21:40:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.282+0000 7fcd9d84a640 10 client.4565.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.282+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign -1 6 2024-03-27T21:40:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.282+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd78003bf0 osd=2 3 2024-03-27T21:40:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.282+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78003bf0 osd=2 3 2024-03-27T21:40:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.282+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd78003bf0 osd=2 3 2024-03-27T21:40:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.282+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_remove -1 6 2024-03-27T21:40:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.282+0000 7fcd9d84a640 20 client.4565.objecter get_session s=0x7fcd78003bf0 osd=2 3 2024-03-27T21:40:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.282+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign 2 6 2024-03-27T21:40:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.282+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78003bf0 osd=2 4 2024-03-27T21:40:38.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.282+0000 7fcd9d84a640 10 client.4565.objecter _send_command 6 2024-03-27T21:40:38.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.282+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcd78003500 con 0x7fcd7800b0e0 2024-03-27T21:40:38.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.284+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== osd.2 v2:172.21.15.60:6824/44654 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fcd78003500 con 0x7fcd7800b0e0 2024-03-27T21:40:38.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.284+0000 7fcd7ffff640 10 client.4565.objecter ms_dispatch 0x7fcd98000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:38.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.284+0000 7fcd7ffff640 10 client.4565.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:38.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.284+0000 7fcd7ffff640 20 client.4565.objecter put_session s=0x7fcd78003bf0 osd=2 4 2024-03-27T21:40:38.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.284+0000 7fcd7ffff640 15 client.4565.objecter _session_command_op_remove 2 6 2024-03-27T21:40:38.271 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2: {} 2024-03-27T21:40:38.272 INFO:teuthology.orchestra.run.smithi060.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-27T21:40:38.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.284+0000 7fcd9d84a640 10 client.4565.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:38.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.284+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign -1 7 2024-03-27T21:40:38.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.284+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fcd78002070 0x7fcd7800eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:38.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.284+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd780038b0 osd=3 3 2024-03-27T21:40:38.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.284+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd780038b0 osd=3 3 2024-03-27T21:40:38.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.284+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd780038b0 osd=3 3 2024-03-27T21:40:38.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.284+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_remove -1 7 2024-03-27T21:40:38.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.284+0000 7fcd9d84a640 20 client.4565.objecter get_session s=0x7fcd780038b0 osd=3 3 2024-03-27T21:40:38.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.284+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign 3 7 2024-03-27T21:40:38.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.284+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd780038b0 osd=3 4 2024-03-27T21:40:38.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.284+0000 7fcd9d84a640 10 client.4565.objecter _send_command 7 2024-03-27T21:40:38.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.284+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd7800f1e0 con 0x7fcd78002070 2024-03-27T21:40:38.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.284+0000 7fcd977fe640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fcd78002070 0x7fcd7800eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:38.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.284+0000 7fcd977fe640 10 monclient: get_auth_request con 0x7fcd78002070 auth_method 0 2024-03-27T21:40:38.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.285+0000 7fcd7ffff640 10 client.4565.objecter ms_handle_connect 0x7fcd78002070 2024-03-27T21:40:38.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.285+0000 7fcd977fe640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fcd78002070 0x7fcd7800eb90 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:38.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.285+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fcd7800f1e0 con 0x7fcd78002070 2024-03-27T21:40:38.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.285+0000 7fcd7ffff640 10 client.4565.objecter ms_dispatch 0x7fcd98000b90 command_reply(tid 7: 0 ) v1 2024-03-27T21:40:38.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.285+0000 7fcd7ffff640 10 client.4565.objecter _finish_command 7 = system:0 2024-03-27T21:40:38.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.285+0000 7fcd7ffff640 20 client.4565.objecter put_session s=0x7fcd780038b0 osd=3 4 2024-03-27T21:40:38.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.285+0000 7fcd7ffff640 15 client.4565.objecter _session_command_op_remove 3 7 2024-03-27T21:40:38.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.313+0000 7fcd9d84a640 10 client.4565.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:38.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.313+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign -1 8 2024-03-27T21:40:38.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.313+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd780038b0 osd=3 3 2024-03-27T21:40:38.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.313+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd780038b0 osd=3 3 2024-03-27T21:40:38.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.313+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd780038b0 osd=3 3 2024-03-27T21:40:38.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.313+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_remove -1 8 2024-03-27T21:40:38.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.313+0000 7fcd9d84a640 20 client.4565.objecter get_session s=0x7fcd780038b0 osd=3 3 2024-03-27T21:40:38.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.313+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign 3 8 2024-03-27T21:40:38.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.313+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd780038b0 osd=3 4 2024-03-27T21:40:38.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.313+0000 7fcd9d84a640 10 client.4565.objecter _send_command 8 2024-03-27T21:40:38.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.313+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcd780015e0 con 0x7fcd78002070 2024-03-27T21:40:38.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.315+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fcd780015e0 con 0x7fcd78002070 2024-03-27T21:40:38.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.315+0000 7fcd7ffff640 10 client.4565.objecter ms_dispatch 0x7fcd98000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:38.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.315+0000 7fcd7ffff640 10 client.4565.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:38.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.315+0000 7fcd7ffff640 20 client.4565.objecter put_session s=0x7fcd780038b0 osd=3 4 2024-03-27T21:40:38.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.315+0000 7fcd7ffff640 15 client.4565.objecter _session_command_op_remove 3 8 2024-03-27T21:40:38.302 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:38.303 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:38.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.315+0000 7fcd9d84a640 10 client.4565.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:38.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.315+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign -1 9 2024-03-27T21:40:38.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.315+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fcd78012420 0x7fcd78014810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:38.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.315+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd780013a0 osd=4 3 2024-03-27T21:40:38.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.315+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd780013a0 osd=4 3 2024-03-27T21:40:38.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.315+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd780013a0 osd=4 3 2024-03-27T21:40:38.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.315+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_remove -1 9 2024-03-27T21:40:38.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.315+0000 7fcd9d84a640 20 client.4565.objecter get_session s=0x7fcd780013a0 osd=4 3 2024-03-27T21:40:38.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.315+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign 4 9 2024-03-27T21:40:38.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.315+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd780013a0 osd=4 4 2024-03-27T21:40:38.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.315+0000 7fcd9d84a640 10 client.4565.objecter _send_command 9 2024-03-27T21:40:38.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.315+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd780015a0 con 0x7fcd78012420 2024-03-27T21:40:38.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.316+0000 7fcd96ffd640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fcd78012420 0x7fcd78014810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:38.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.316+0000 7fcd96ffd640 10 monclient: get_auth_request con 0x7fcd78012420 auth_method 0 2024-03-27T21:40:38.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.316+0000 7fcd7ffff640 10 client.4565.objecter ms_handle_connect 0x7fcd78012420 2024-03-27T21:40:38.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.316+0000 7fcd96ffd640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fcd78012420 0x7fcd78014810 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:38.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.317+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fcd780015a0 con 0x7fcd78012420 2024-03-27T21:40:38.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.317+0000 7fcd7ffff640 10 client.4565.objecter ms_dispatch 0x7fcd98000b90 command_reply(tid 9: 0 ) v1 2024-03-27T21:40:38.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.317+0000 7fcd7ffff640 10 client.4565.objecter _finish_command 9 = system:0 2024-03-27T21:40:38.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.317+0000 7fcd7ffff640 20 client.4565.objecter put_session s=0x7fcd780013a0 osd=4 4 2024-03-27T21:40:38.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.317+0000 7fcd7ffff640 15 client.4565.objecter _session_command_op_remove 4 9 2024-03-27T21:40:38.324 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:38.337+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.344+0000 7fcd9d84a640 10 client.4565.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.344+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign -1 10 2024-03-27T21:40:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.344+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd780013a0 osd=4 3 2024-03-27T21:40:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.344+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd780013a0 osd=4 3 2024-03-27T21:40:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.344+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd780013a0 osd=4 3 2024-03-27T21:40:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.344+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_remove -1 10 2024-03-27T21:40:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.344+0000 7fcd9d84a640 20 client.4565.objecter get_session s=0x7fcd780013a0 osd=4 3 2024-03-27T21:40:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.344+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign 4 10 2024-03-27T21:40:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.344+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd780013a0 osd=4 4 2024-03-27T21:40:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.344+0000 7fcd9d84a640 10 client.4565.objecter _send_command 10 2024-03-27T21:40:38.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.344+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcd78001560 con 0x7fcd78012420 2024-03-27T21:40:38.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.346+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fcd78001560 con 0x7fcd78012420 2024-03-27T21:40:38.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.346+0000 7fcd7ffff640 10 client.4565.objecter ms_dispatch 0x7fcd98000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:38.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.346+0000 7fcd7ffff640 10 client.4565.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:38.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.346+0000 7fcd7ffff640 20 client.4565.objecter put_session s=0x7fcd780013a0 osd=4 4 2024-03-27T21:40:38.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.346+0000 7fcd7ffff640 15 client.4565.objecter _session_command_op_remove 4 10 2024-03-27T21:40:38.333 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:38.334 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:38.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.346+0000 7fcd9d84a640 10 client.4565.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.346+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign -1 11 2024-03-27T21:40:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.346+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fcd78015260 0x7fcd780176b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.346+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd78001560 osd=5 3 2024-03-27T21:40:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.346+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78001560 osd=5 3 2024-03-27T21:40:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.346+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd78001560 osd=5 3 2024-03-27T21:40:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.346+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_remove -1 11 2024-03-27T21:40:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.346+0000 7fcd9d84a640 20 client.4565.objecter get_session s=0x7fcd78001560 osd=5 3 2024-03-27T21:40:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.346+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign 5 11 2024-03-27T21:40:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.346+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78001560 osd=5 4 2024-03-27T21:40:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.346+0000 7fcd9d84a640 10 client.4565.objecter _send_command 11 2024-03-27T21:40:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.346+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd7801a6b0 con 0x7fcd78015260 2024-03-27T21:40:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.347+0000 7fcd967fc640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fcd78015260 0x7fcd780176b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.347+0000 7fcd967fc640 10 monclient: get_auth_request con 0x7fcd78015260 auth_method 0 2024-03-27T21:40:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.347+0000 7fcd7ffff640 10 client.4565.objecter ms_handle_connect 0x7fcd78015260 2024-03-27T21:40:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.347+0000 7fcd967fc640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fcd78015260 0x7fcd780176b0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.348+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fcd7801a6b0 con 0x7fcd78015260 2024-03-27T21:40:38.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.348+0000 7fcd7ffff640 10 client.4565.objecter ms_dispatch 0x7fcd98000b90 command_reply(tid 11: 0 ) v1 2024-03-27T21:40:38.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.348+0000 7fcd7ffff640 10 client.4565.objecter _finish_command 11 = system:0 2024-03-27T21:40:38.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.348+0000 7fcd7ffff640 20 client.4565.objecter put_session s=0x7fcd78001560 osd=5 4 2024-03-27T21:40:38.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.348+0000 7fcd7ffff640 15 client.4565.objecter _session_command_op_remove 5 11 2024-03-27T21:40:38.336 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:38.336 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:38.343 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:40:38.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.375+0000 7fcd9d84a640 10 client.4565.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:38.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.375+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign -1 12 2024-03-27T21:40:38.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.375+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd78001560 osd=5 3 2024-03-27T21:40:38.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.375+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78001560 osd=5 3 2024-03-27T21:40:38.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.375+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd78001560 osd=5 3 2024-03-27T21:40:38.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.375+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_remove -1 12 2024-03-27T21:40:38.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.375+0000 7fcd9d84a640 20 client.4565.objecter get_session s=0x7fcd78001560 osd=5 3 2024-03-27T21:40:38.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.375+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign 5 12 2024-03-27T21:40:38.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.375+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78001560 osd=5 4 2024-03-27T21:40:38.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.375+0000 7fcd9d84a640 10 client.4565.objecter _send_command 12 2024-03-27T21:40:38.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.375+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcd78019440 con 0x7fcd78015260 2024-03-27T21:40:38.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.377+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fcd78019440 con 0x7fcd78015260 2024-03-27T21:40:38.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.377+0000 7fcd7ffff640 10 client.4565.objecter ms_dispatch 0x7fcd98000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:38.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.377+0000 7fcd7ffff640 10 client.4565.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:38.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.377+0000 7fcd7ffff640 20 client.4565.objecter put_session s=0x7fcd78001560 osd=5 4 2024-03-27T21:40:38.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.377+0000 7fcd7ffff640 15 client.4565.objecter _session_command_op_remove 5 12 2024-03-27T21:40:38.364 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:38.365 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.377+0000 7fcd9d84a640 10 client.4565.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.377+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign -1 13 2024-03-27T21:40:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.377+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fcd78019b70 0x7fcd78019ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.377+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd78019440 osd=6 3 2024-03-27T21:40:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.377+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78019440 osd=6 3 2024-03-27T21:40:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.377+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd78019440 osd=6 3 2024-03-27T21:40:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.377+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_remove -1 13 2024-03-27T21:40:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.377+0000 7fcd9d84a640 20 client.4565.objecter get_session s=0x7fcd78019440 osd=6 3 2024-03-27T21:40:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.377+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign 6 13 2024-03-27T21:40:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.377+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78019440 osd=6 4 2024-03-27T21:40:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.377+0000 7fcd9d84a640 10 client.4565.objecter _send_command 13 2024-03-27T21:40:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.377+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd78018fc0 con 0x7fcd78019b70 2024-03-27T21:40:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.378+0000 7fcd977fe640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fcd78019b70 0x7fcd78019ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:38.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.378+0000 7fcd977fe640 10 monclient: get_auth_request con 0x7fcd78019b70 auth_method 0 2024-03-27T21:40:38.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.378+0000 7fcd7ffff640 10 client.4565.objecter ms_handle_connect 0x7fcd78019b70 2024-03-27T21:40:38.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.378+0000 7fcd977fe640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fcd78019b70 0x7fcd78019ff0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:38.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.379+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fcd78018fc0 con 0x7fcd78019b70 2024-03-27T21:40:38.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.379+0000 7fcd7ffff640 10 client.4565.objecter ms_dispatch 0x7fcd98000b90 command_reply(tid 13: 0 ) v1 2024-03-27T21:40:38.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.379+0000 7fcd7ffff640 10 client.4565.objecter _finish_command 13 = system:0 2024-03-27T21:40:38.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.379+0000 7fcd7ffff640 20 client.4565.objecter put_session s=0x7fcd78019440 osd=6 4 2024-03-27T21:40:38.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.379+0000 7fcd7ffff640 15 client.4565.objecter _session_command_op_remove 6 13 2024-03-27T21:40:38.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.394+0000 7fcd9d84a640 10 client.4565.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:38.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.394+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign -1 14 2024-03-27T21:40:38.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.394+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd78019440 osd=6 3 2024-03-27T21:40:38.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.394+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78019440 osd=6 3 2024-03-27T21:40:38.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.394+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd78019440 osd=6 3 2024-03-27T21:40:38.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.394+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_remove -1 14 2024-03-27T21:40:38.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.394+0000 7fcd9d84a640 20 client.4565.objecter get_session s=0x7fcd78019440 osd=6 3 2024-03-27T21:40:38.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.394+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign 6 14 2024-03-27T21:40:38.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.394+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78019440 osd=6 4 2024-03-27T21:40:38.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.394+0000 7fcd9d84a640 10 client.4565.objecter _send_command 14 2024-03-27T21:40:38.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.394+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcd78018f80 con 0x7fcd78019b70 2024-03-27T21:40:38.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.396+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fcd78018f80 con 0x7fcd78019b70 2024-03-27T21:40:38.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.396+0000 7fcd7ffff640 10 client.4565.objecter ms_dispatch 0x7fcd98000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:38.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.396+0000 7fcd7ffff640 10 client.4565.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:38.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.396+0000 7fcd7ffff640 20 client.4565.objecter put_session s=0x7fcd78019440 osd=6 4 2024-03-27T21:40:38.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.396+0000 7fcd7ffff640 15 client.4565.objecter _session_command_op_remove 6 14 2024-03-27T21:40:38.383 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:38.384 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:38.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.396+0000 7fcd9d84a640 10 client.4565.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:38.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.396+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign -1 15 2024-03-27T21:40:38.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.396+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fcd78018260 0x7fcd780186e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:38.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.396+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd78018f80 osd=7 3 2024-03-27T21:40:38.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.396+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78018f80 osd=7 3 2024-03-27T21:40:38.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.396+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd78018f80 osd=7 3 2024-03-27T21:40:38.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.396+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_remove -1 15 2024-03-27T21:40:38.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.396+0000 7fcd9d84a640 20 client.4565.objecter get_session s=0x7fcd78018f80 osd=7 3 2024-03-27T21:40:38.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.396+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign 7 15 2024-03-27T21:40:38.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.396+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78018f80 osd=7 4 2024-03-27T21:40:38.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.396+0000 7fcd9d84a640 10 client.4565.objecter _send_command 15 2024-03-27T21:40:38.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.396+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd78018c20 con 0x7fcd78018260 2024-03-27T21:40:38.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.396+0000 7fcd96ffd640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fcd78018260 0x7fcd780186e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:38.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.396+0000 7fcd96ffd640 10 monclient: get_auth_request con 0x7fcd78018260 auth_method 0 2024-03-27T21:40:38.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.396+0000 7fcd7ffff640 10 client.4565.objecter ms_handle_connect 0x7fcd78018260 2024-03-27T21:40:38.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.396+0000 7fcd96ffd640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fcd78018260 0x7fcd780186e0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:38.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.397+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fcd78018c20 con 0x7fcd78018260 2024-03-27T21:40:38.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.397+0000 7fcd7ffff640 10 client.4565.objecter ms_dispatch 0x7fcd98000b90 command_reply(tid 15: 0 ) v1 2024-03-27T21:40:38.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.397+0000 7fcd7ffff640 10 client.4565.objecter _finish_command 15 = system:0 2024-03-27T21:40:38.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.397+0000 7fcd7ffff640 20 client.4565.objecter put_session s=0x7fcd78018f80 osd=7 4 2024-03-27T21:40:38.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.397+0000 7fcd7ffff640 15 client.4565.objecter _session_command_op_remove 7 15 2024-03-27T21:40:38.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.411+0000 7fcd9d84a640 10 client.4565.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:38.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.411+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign -1 16 2024-03-27T21:40:38.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.411+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd78018f80 osd=7 3 2024-03-27T21:40:38.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.411+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78018f80 osd=7 3 2024-03-27T21:40:38.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.411+0000 7fcd9d84a640 20 client.4565.objecter _get_session s=0x7fcd78018f80 osd=7 3 2024-03-27T21:40:38.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.411+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_remove -1 16 2024-03-27T21:40:38.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.411+0000 7fcd9d84a640 20 client.4565.objecter get_session s=0x7fcd78018f80 osd=7 3 2024-03-27T21:40:38.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.411+0000 7fcd9d84a640 15 client.4565.objecter _session_command_op_assign 7 16 2024-03-27T21:40:38.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.411+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78018f80 osd=7 4 2024-03-27T21:40:38.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.411+0000 7fcd9d84a640 10 client.4565.objecter _send_command 16 2024-03-27T21:40:38.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.411+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcd78018c20 con 0x7fcd78018260 2024-03-27T21:40:38.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.412+0000 7fcd7ffff640 1 -- 172.21.15.60:0/292188241 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fcd78018c20 con 0x7fcd78018260 2024-03-27T21:40:38.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.412+0000 7fcd7ffff640 10 client.4565.objecter ms_dispatch 0x7fcd98000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:38.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.412+0000 7fcd7ffff640 10 client.4565.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:38.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.412+0000 7fcd7ffff640 20 client.4565.objecter put_session s=0x7fcd78018f80 osd=7 4 2024-03-27T21:40:38.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.412+0000 7fcd7ffff640 15 client.4565.objecter _session_command_op_remove 7 16 2024-03-27T21:40:38.399 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:38.401 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:38.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.412+0000 7fcd9d84a640 10 librados: watch_flush enter 2024-03-27T21:40:38.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.412+0000 7fcd9d84a640 10 librados: watch_flush exit 2024-03-27T21:40:38.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.412+0000 7fcd9d84a640 10 client.4565.objecter close_session for osd.0 2024-03-27T21:40:38.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.412+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fcd78004640 msgr2=0x7fcd78004a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:38.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.412+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fcd78004640 0x7fcd78004a80 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd780052c0 osd=0 1 2024-03-27T21:40:38.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 10 client.4565.objecter close_session for osd.1 2024-03-27T21:40:38.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fcd78002700 msgr2=0x7fcd78002b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:38.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fcd78002700 0x7fcd78002b80 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78008090 osd=1 1 2024-03-27T21:40:38.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 10 client.4565.objecter close_session for osd.2 2024-03-27T21:40:38.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7fcd7800b0e0 msgr2=0x7fcd7800d4d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:38.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7fcd7800b0e0 0x7fcd7800d4d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78003bf0 osd=2 1 2024-03-27T21:40:38.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 10 client.4565.objecter close_session for osd.3 2024-03-27T21:40:38.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fcd78002070 msgr2=0x7fcd7800eb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:38.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fcd78002070 0x7fcd7800eb90 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd780038b0 osd=3 1 2024-03-27T21:40:38.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 10 client.4565.objecter close_session for osd.4 2024-03-27T21:40:38.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fcd78012420 msgr2=0x7fcd78014810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:38.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fcd78012420 0x7fcd78014810 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd780013a0 osd=4 1 2024-03-27T21:40:38.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 10 client.4565.objecter close_session for osd.5 2024-03-27T21:40:38.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fcd78015260 msgr2=0x7fcd780176b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:38.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fcd78015260 0x7fcd780176b0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78001560 osd=5 1 2024-03-27T21:40:38.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 10 client.4565.objecter close_session for osd.6 2024-03-27T21:40:38.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fcd78019b70 msgr2=0x7fcd78019ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:38.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fcd78019b70 0x7fcd78019ff0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78019440 osd=6 1 2024-03-27T21:40:38.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 10 client.4565.objecter close_session for osd.7 2024-03-27T21:40:38.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fcd78018260 msgr2=0x7fcd780186e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:38.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fcd78018260 0x7fcd780186e0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 20 client.4565.objecter put_session s=0x7fcd78018f80 osd=7 1 2024-03-27T21:40:38.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 20 client.4565.objecter shutdown clearing up homeless session... 2024-03-27T21:40:38.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 10 client.4565.objecter successfully canceled tick 2024-03-27T21:40:38.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd977fe640 1 -- 172.21.15.60:0/292188241 reap_dead start 2024-03-27T21:40:38.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd977fe640 1 -- 172.21.15.60:0/292188241 reap_dead start 2024-03-27T21:40:38.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd977fe640 1 -- 172.21.15.60:0/292188241 reap_dead start 2024-03-27T21:40:38.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd977fe640 1 -- 172.21.15.60:0/292188241 reap_dead start 2024-03-27T21:40:38.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd977fe640 1 -- 172.21.15.60:0/292188241 reap_dead start 2024-03-27T21:40:38.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcd8406c590 msgr2=0x7fcd8406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:38.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcd8406c590 0x7fcd8406ea50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcd880097a0 tx=0x7fcd88038000 comp rx=0 tx=0).stop 2024-03-27T21:40:38.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 10 monclient: shutdown 2024-03-27T21:40:38.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:38.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcd9813b020 msgr2=0x7fcd981d5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:38.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcd9813b020 0x7fcd981d5480 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c00c3c0 tx=0x7fcd8c00c880 comp rx=0 tx=0).stop 2024-03-27T21:40:38.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 shutdown_connections 2024-03-27T21:40:38.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcd9813b020 0x7fcd981d5480 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 --2- 172.21.15.60:0/292188241 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcd8406c590 0x7fcd8406ea50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 >> 172.21.15.60:0/292188241 conn(0x7fcd980608e0 msgr2=0x7fcd98140630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:38.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 shutdown_connections 2024-03-27T21:40:38.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 -- 172.21.15.60:0/292188241 wait complete. 2024-03-27T21:40:38.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.413+0000 7fcd9d84a640 1 librados: shutdown 2024-03-27T21:40:38.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.458+0000 7fbc6d7fa640 1 -- 172.21.15.60:0/2437008184 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v41) v1 ==== 101+0+0 (secure 0 0 0) 0x7fbc5c089e60 con 0x7fbc7006f9d0 2024-03-27T21:40:38.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.458+0000 7fbc6d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:40:38.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.458+0000 7fbc6d7fa640 10 monclient: _finish_command 2 = system:0 nodeep-scrub is unset 2024-03-27T21:40:38.445 INFO:teuthology.orchestra.run.smithi060.stderr:nodeep-scrub is unset 2024-03-27T21:40:38.445 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:38.458+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:38.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.461+0000 7fbc76acf640 10 librados: watch_flush enter 2024-03-27T21:40:38.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.461+0000 7fbc76acf640 10 librados: watch_flush exit 2024-03-27T21:40:38.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.461+0000 7fbc76acf640 20 client.4698.objecter shutdown clearing up homeless session... 2024-03-27T21:40:38.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.461+0000 7fbc76acf640 10 client.4698.objecter successfully canceled tick 2024-03-27T21:40:38.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.461+0000 7fbc76acf640 1 -- 172.21.15.60:0/2437008184 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fbc5406c590 msgr2=0x7fbc5406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:38.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.461+0000 7fbc76acf640 1 --2- 172.21.15.60:0/2437008184 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fbc5406c590 0x7fbc5406ea50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbc64004e70 tx=0x7fbc64007d70 comp rx=0 tx=0).stop 2024-03-27T21:40:38.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.461+0000 7fbc76acf640 10 monclient: shutdown 2024-03-27T21:40:38.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.461+0000 7fbc76acf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:38.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.461+0000 7fbc76acf640 1 -- 172.21.15.60:0/2437008184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc7006f9d0 msgr2=0x7fbc701d2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:38.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.461+0000 7fbc76acf640 1 --2- 172.21.15.60:0/2437008184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc7006f9d0 0x7fbc701d2530 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c042000 tx=0x7fbc5c001980 comp rx=0 tx=0).stop 2024-03-27T21:40:38.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.461+0000 7fbc76acf640 1 -- 172.21.15.60:0/2437008184 shutdown_connections 2024-03-27T21:40:38.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.461+0000 7fbc76acf640 1 --2- 172.21.15.60:0/2437008184 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fbc5406c590 0x7fbc5406ea50 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.461+0000 7fbc76acf640 1 --2- 172.21.15.60:0/2437008184 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc701d2a70 0x7fbc701d5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.461+0000 7fbc76acf640 1 --2- 172.21.15.60:0/2437008184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc7006f9d0 0x7fbc701d2530 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.461+0000 7fbc76acf640 1 --2- 172.21.15.60:0/2437008184 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbc700aac40 0x7fbc7006caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:38.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.461+0000 7fbc76acf640 1 -- 172.21.15.60:0/2437008184 >> 172.21.15.60:0/2437008184 conn(0x7fbc700608e0 msgr2=0x7fbc700a8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:38.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.461+0000 7fbc76acf640 1 -- 172.21.15.60:0/2437008184 shutdown_connections 2024-03-27T21:40:38.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.461+0000 7fbc76acf640 1 -- 172.21.15.60:0/2437008184 wait complete. 2024-03-27T21:40:38.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:38.461+0000 7fbc76acf640 1 librados: shutdown 2024-03-27T21:40:38.490 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:38.490 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:38.490 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:38.490 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:38.491 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:38.499 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:40:38.525 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:38.539+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:38.607 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:38.607 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:38.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:38.608 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-03-27T21:40:38.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.608 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:38.608 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:38.608 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 59.187844003999999, 2024-03-27T21:40:38.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43537978700000002, 2024-03-27T21:40:38.608 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:38.608 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:38.608 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:38.608 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:38.608 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:38.608 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-03-27T21:40:38.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.609 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:38.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:38.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:38.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:38.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:38.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:38.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432019+0000", 2024-03-27T21:40:38.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-03-27T21:40:38.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.610 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:38.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432027+0000", 2024-03-27T21:40:38.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1200000000000002e-06 2024-03-27T21:40:38.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.610 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:38.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432039+0000", 2024-03-27T21:40:38.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1123e-05 2024-03-27T21:40:38.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.610 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:38.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432051+0000", 2024-03-27T21:40:38.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-03-27T21:40:38.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.611 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:38.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432157+0000", 2024-03-27T21:40:38.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106846 2024-03-27T21:40:38.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.611 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-03-27T21:40:38.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432176+0000", 2024-03-27T21:40:38.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8490000000000001e-05 2024-03-27T21:40:38.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.611 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:38.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866170+0000", 2024-03-27T21:40:38.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43399370100000001 2024-03-27T21:40:38.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:38.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866335+0000", 2024-03-27T21:40:38.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016559699999999999 2024-03-27T21:40:38.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:38.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866495+0000", 2024-03-27T21:40:38.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015944200000000001 2024-03-27T21:40:38.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:38.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867012+0000", 2024-03-27T21:40:38.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0005176 2024-03-27T21:40:38.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:38.613 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867334+0000", 2024-03-27T21:40:38.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032158600000000001 2024-03-27T21:40:38.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:38.613 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867374+0000", 2024-03-27T21:40:38.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0547000000000003e-05 2024-03-27T21:40:38.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:38.613 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867400+0000", 2024-03-27T21:40:38.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5958000000000001e-05 2024-03-27T21:40:38.614 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.614 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:38.614 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.614 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.614 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:38.614 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:38.614 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.690461552999999, 2024-03-27T21:40:38.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064560310000000001, 2024-03-27T21:40:38.614 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:38.614 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:38.614 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:38.614 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:38.614 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:38.615 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 70 2024-03-27T21:40:38.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.615 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:38.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.615 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:38.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:38.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.615 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:38.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:38.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.615 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:38.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929402+0000", 2024-03-27T21:40:38.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:38.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:38.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929407+0000", 2024-03-27T21:40:38.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-06 2024-03-27T21:40:38.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:38.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929414+0000", 2024-03-27T21:40:38.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5689999999999999e-06 2024-03-27T21:40:38.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:38.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929419+0000", 2024-03-27T21:40:38.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-03-27T21:40:38.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:38.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930245+0000", 2024-03-27T21:40:38.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082521500000000002 2024-03-27T21:40:38.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:38.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930287+0000", 2024-03-27T21:40:38.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2305000000000002e-05 2024-03-27T21:40:38.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:38.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930367+0000", 2024-03-27T21:40:38.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9589e-05 2024-03-27T21:40:38.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.618 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.618 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:38.618 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.931214+0000", 2024-03-27T21:40:38.618 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000847827 2024-03-27T21:40:38.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.618 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.618 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:38.618 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935807+0000", 2024-03-27T21:40:38.618 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045923780000000003 2024-03-27T21:40:38.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.618 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.618 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:38.618 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935840+0000", 2024-03-27T21:40:38.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3501999999999999e-05 2024-03-27T21:40:38.619 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.619 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:38.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935859+0000", 2024-03-27T21:40:38.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8845e-05 2024-03-27T21:40:38.619 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.619 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:38.619 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.619 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.619 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:38.619 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:38.619 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.683201961999998, 2024-03-27T21:40:38.620 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071695429999999996, 2024-03-27T21:40:38.620 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:38.620 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:38.620 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:38.620 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:38.620 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:38.620 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-03-27T21:40:38.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.620 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:38.620 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.620 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:38.620 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:38.620 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.620 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.621 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:38.621 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:38.621 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.621 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.621 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:38.621 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936662+0000", 2024-03-27T21:40:38.621 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:38.621 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.621 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:38.621 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936737+0000", 2024-03-27T21:40:38.621 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5303999999999996e-05 2024-03-27T21:40:38.621 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:38.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936746+0000", 2024-03-27T21:40:38.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6340000000000007e-06 2024-03-27T21:40:38.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:38.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936753+0000", 2024-03-27T21:40:38.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-03-27T21:40:38.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:38.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937926+0000", 2024-03-27T21:40:38.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011731280000000001 2024-03-27T21:40:38.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:38.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937969+0000", 2024-03-27T21:40:38.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3285999999999999e-05 2024-03-27T21:40:38.623 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:38.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938040+0000", 2024-03-27T21:40:38.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0358000000000006e-05 2024-03-27T21:40:38.623 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:38.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939324+0000", 2024-03-27T21:40:38.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012844899999999999 2024-03-27T21:40:38.623 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.624 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:38.624 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943780+0000", 2024-03-27T21:40:38.624 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044555630000000001 2024-03-27T21:40:38.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.624 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:38.624 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943811+0000", 2024-03-27T21:40:38.624 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1415999999999999e-05 2024-03-27T21:40:38.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.624 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:38.624 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943832+0000", 2024-03-27T21:40:38.624 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0981000000000001e-05 2024-03-27T21:40:38.624 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.625 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:38.625 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.625 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.625 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.625 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:38.625 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:38.625 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.683103946000003, 2024-03-27T21:40:38.625 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071139740000000003, 2024-03-27T21:40:38.625 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:38.625 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:38.625 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:38.625 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:38.625 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:38.625 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-03-27T21:40:38.625 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.625 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:38.626 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.626 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:38.626 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:38.626 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.626 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.626 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.626 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:38.626 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:38.626 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.626 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.626 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.626 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:38.626 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936760+0000", 2024-03-27T21:40:38.626 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:38.627 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:38.627 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936765+0000", 2024-03-27T21:40:38.627 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6330000000000004e-06 2024-03-27T21:40:38.627 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:38.627 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936771+0000", 2024-03-27T21:40:38.627 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0560000000000001e-06 2024-03-27T21:40:38.627 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.627 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:38.627 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936776+0000", 2024-03-27T21:40:38.627 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1510000000000003e-06 2024-03-27T21:40:38.627 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.628 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:38.628 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938605+0000", 2024-03-27T21:40:38.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001829589 2024-03-27T21:40:38.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.628 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:38.628 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938657+0000", 2024-03-27T21:40:38.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1860000000000002e-05 2024-03-27T21:40:38.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.628 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.628 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:38.628 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938734+0000", 2024-03-27T21:40:38.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6639999999999998e-05 2024-03-27T21:40:38.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.629 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:38.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939562+0000", 2024-03-27T21:40:38.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082803100000000001 2024-03-27T21:40:38.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.629 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:38.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943838+0000", 2024-03-27T21:40:38.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042757050000000003 2024-03-27T21:40:38.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.629 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:38.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943859+0000", 2024-03-27T21:40:38.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1175e-05 2024-03-27T21:40:38.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.630 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:38.630 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943875+0000", 2024-03-27T21:40:38.630 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5744e-05 2024-03-27T21:40:38.630 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.630 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:38.630 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.630 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.630 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:38.630 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:38.630 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.539589894999999, 2024-03-27T21:40:38.630 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059712230000000003, 2024-03-27T21:40:38.630 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:38.630 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:38.631 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:38.631 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:38.631 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:38.631 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 112 2024-03-27T21:40:38.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.631 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:38.631 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.631 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:38.631 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:38.631 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.631 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.631 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:38.631 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:38.631 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.632 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.632 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:38.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080274+0000", 2024-03-27T21:40:38.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:38.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.632 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.632 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:38.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080318+0000", 2024-03-27T21:40:38.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3634999999999998e-05 2024-03-27T21:40:38.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.632 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.632 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:38.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080329+0000", 2024-03-27T21:40:38.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1232e-05 2024-03-27T21:40:38.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:38.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080333+0000", 2024-03-27T21:40:38.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6009999999999999e-06 2024-03-27T21:40:38.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:38.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083786+0000", 2024-03-27T21:40:38.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034528940000000002 2024-03-27T21:40:38.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:38.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083861+0000", 2024-03-27T21:40:38.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4893000000000003e-05 2024-03-27T21:40:38.634 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.634 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.634 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:38.634 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083981+0000", 2024-03-27T21:40:38.634 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012058299999999999 2024-03-27T21:40:38.634 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.634 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.634 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:38.634 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086148+0000", 2024-03-27T21:40:38.634 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021671469999999999 2024-03-27T21:40:38.634 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.634 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.634 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:38.634 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086195+0000", 2024-03-27T21:40:38.634 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6768999999999999e-05 2024-03-27T21:40:38.635 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.635 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.635 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:38.635 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086224+0000", 2024-03-27T21:40:38.635 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8758000000000001e-05 2024-03-27T21:40:38.635 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.635 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.635 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:38.635 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086246+0000", 2024-03-27T21:40:38.635 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2064000000000001e-05 2024-03-27T21:40:38.635 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.635 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:38.635 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.635 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.635 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.636 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:38.636 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:38.636 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.539526948000002, 2024-03-27T21:40:38.636 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066196320000000003, 2024-03-27T21:40:38.636 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:38.636 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:38.636 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:38.636 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:38.636 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:38.636 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113 2024-03-27T21:40:38.636 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.636 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:38.636 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.636 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:38.636 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:38.636 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.637 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.637 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:38.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:38.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.637 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.637 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:38.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080337+0000", 2024-03-27T21:40:38.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:38.637 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.637 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:38.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080340+0000", 2024-03-27T21:40:38.638 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6690000000000002e-06 2024-03-27T21:40:38.638 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.638 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:38.638 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080344+0000", 2024-03-27T21:40:38.638 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6550000000000002e-06 2024-03-27T21:40:38.638 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.638 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:38.638 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080348+0000", 2024-03-27T21:40:38.638 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9169999999999999e-06 2024-03-27T21:40:38.638 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.638 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:38.638 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084595+0000", 2024-03-27T21:40:38.639 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042471719999999996 2024-03-27T21:40:38.639 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.639 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.639 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:38.639 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084676+0000", 2024-03-27T21:40:38.639 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1147000000000001e-05 2024-03-27T21:40:38.639 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.639 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.639 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:38.639 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084796+0000", 2024-03-27T21:40:38.639 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012032900000000001 2024-03-27T21:40:38.639 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.639 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.639 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:38.639 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086190+0000", 2024-03-27T21:40:38.639 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013934399999999999 2024-03-27T21:40:38.640 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.640 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.640 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:38.640 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086905+0000", 2024-03-27T21:40:38.640 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071536199999999999 2024-03-27T21:40:38.640 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.640 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.640 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:38.640 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086937+0000", 2024-03-27T21:40:38.640 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2125000000000002e-05 2024-03-27T21:40:38.640 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.640 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.640 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:38.640 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086957+0000", 2024-03-27T21:40:38.641 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0157999999999999e-05 2024-03-27T21:40:38.641 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.641 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:38.641 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.641 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.641 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.641 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:38.641 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:38.641 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.539514232000002, 2024-03-27T21:40:38.641 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007405661, 2024-03-27T21:40:38.641 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:38.641 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:38.641 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:38.641 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:38.641 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:38.642 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 114 2024-03-27T21:40:38.642 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.642 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:38.642 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.642 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:38.642 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:38.642 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.642 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.642 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.642 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:38.642 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:38.642 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.642 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.642 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.642 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:38.643 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:38.643 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-03-27T21:40:38.643 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.643 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.643 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:38.643 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080353+0000", 2024-03-27T21:40:38.643 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.441e-06 2024-03-27T21:40:38.643 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.643 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.643 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:38.643 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080356+0000", 2024-03-27T21:40:38.643 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0769999999999999e-06 2024-03-27T21:40:38.643 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.643 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.643 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:38.643 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080358+0000", 2024-03-27T21:40:38.644 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3369999999999998e-06 2024-03-27T21:40:38.644 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.644 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.644 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:38.644 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085118+0000", 2024-03-27T21:40:38.644 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047603949999999997 2024-03-27T21:40:38.644 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.644 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.644 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:38.644 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085210+0000", 2024-03-27T21:40:38.644 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1983000000000003e-05 2024-03-27T21:40:38.644 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.644 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.644 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:38.644 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085332+0000", 2024-03-27T21:40:38.645 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121803 2024-03-27T21:40:38.645 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.645 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.645 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:38.645 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086292+0000", 2024-03-27T21:40:38.645 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095961400000000002 2024-03-27T21:40:38.645 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.645 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.645 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:38.645 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087710+0000", 2024-03-27T21:40:38.645 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014181459999999999 2024-03-27T21:40:38.645 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.645 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.645 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:38.645 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087736+0000", 2024-03-27T21:40:38.646 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6475e-05 2024-03-27T21:40:38.646 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.646 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.646 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:38.646 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087756+0000", 2024-03-27T21:40:38.646 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9626000000000001e-05 2024-03-27T21:40:38.646 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.646 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:38.646 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.646 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.646 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.646 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:38.646 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:38.646 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.539491796999997, 2024-03-27T21:40:38.647 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075011720000000004, 2024-03-27T21:40:38.647 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:38.647 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:38.647 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:38.647 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:38.647 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:38.647 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115 2024-03-27T21:40:38.647 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.647 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:38.647 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.647 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:38.647 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:38.647 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.647 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.647 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.647 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:38.648 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:38.648 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.648 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.648 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.648 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:38.648 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:38.648 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:38.648 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.648 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.648 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:38.648 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080458+0000", 2024-03-27T21:40:38.649 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5228000000000005e-05 2024-03-27T21:40:38.649 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.649 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.649 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:38.649 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080465+0000", 2024-03-27T21:40:38.649 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.238e-06 2024-03-27T21:40:38.649 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.649 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.649 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:38.649 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080468+0000", 2024-03-27T21:40:38.649 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5109999999999999e-06 2024-03-27T21:40:38.649 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.649 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.649 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:38.650 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086326+0000", 2024-03-27T21:40:38.650 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058579169999999998 2024-03-27T21:40:38.650 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.650 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.650 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:38.650 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086382+0000", 2024-03-27T21:40:38.650 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5309999999999997e-05 2024-03-27T21:40:38.650 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.650 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.650 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:38.650 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086457+0000", 2024-03-27T21:40:38.650 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5413000000000005e-05 2024-03-27T21:40:38.650 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.650 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.651 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:38.651 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087762+0000", 2024-03-27T21:40:38.651 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013052300000000001 2024-03-27T21:40:38.651 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.651 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.651 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:38.651 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087798+0000", 2024-03-27T21:40:38.651 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5614999999999997e-05 2024-03-27T21:40:38.651 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.651 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.651 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:38.651 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087840+0000", 2024-03-27T21:40:38.651 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2196000000000001e-05 2024-03-27T21:40:38.651 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.651 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.652 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:38.652 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087874+0000", 2024-03-27T21:40:38.652 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3785999999999998e-05 2024-03-27T21:40:38.652 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.652 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:38.652 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.652 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.652 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.653 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:38.653 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:38.653 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.539391713999997, 2024-03-27T21:40:38.653 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075050680000000002, 2024-03-27T21:40:38.653 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:38.653 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:38.653 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:38.653 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:38.653 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:38.653 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 116 2024-03-27T21:40:38.653 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.653 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:38.653 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.653 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:38.654 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:38.654 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.654 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.654 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.654 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:38.654 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:38.654 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.654 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.654 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.654 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:38.654 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080472+0000", 2024-03-27T21:40:38.654 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:38.654 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.654 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.654 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:38.655 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080475+0000", 2024-03-27T21:40:38.655 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.712e-06 2024-03-27T21:40:38.655 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.655 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.655 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:38.655 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080478+0000", 2024-03-27T21:40:38.655 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1070000000000001e-06 2024-03-27T21:40:38.655 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.655 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.655 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:38.655 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080481+0000", 2024-03-27T21:40:38.655 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4269999999999998e-06 2024-03-27T21:40:38.655 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.655 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.656 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:38.656 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086992+0000", 2024-03-27T21:40:38.656 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065114099999999996 2024-03-27T21:40:38.656 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.656 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.656 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:38.656 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087064+0000", 2024-03-27T21:40:38.656 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1927000000000004e-05 2024-03-27T21:40:38.656 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.656 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.656 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:38.656 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087188+0000", 2024-03-27T21:40:38.656 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123585 2024-03-27T21:40:38.656 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.656 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.657 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:38.657 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087766+0000", 2024-03-27T21:40:38.657 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057814300000000004 2024-03-27T21:40:38.657 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.657 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.657 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:38.657 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087934+0000", 2024-03-27T21:40:38.657 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000167894 2024-03-27T21:40:38.657 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.657 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.657 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:38.657 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087958+0000", 2024-03-27T21:40:38.657 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4663000000000001e-05 2024-03-27T21:40:38.657 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.657 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.657 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:38.658 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087978+0000", 2024-03-27T21:40:38.658 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9636e-05 2024-03-27T21:40:38.658 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.658 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:38.658 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.658 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.658 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.658 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:38.658 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:38.658 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.539359591999997, 2024-03-27T21:40:38.658 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076850679999999998, 2024-03-27T21:40:38.658 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:38.658 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:38.658 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:38.659 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:38.659 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:38.659 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 117 2024-03-27T21:40:38.659 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.659 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:38.659 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.659 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:38.659 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:38.659 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.659 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.659 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.659 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:38.659 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:38.659 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.660 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.660 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.660 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:38.660 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:38.660 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:38.660 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.660 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.660 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:38.660 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080508+0000", 2024-03-27T21:40:38.660 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.298e-06 2024-03-27T21:40:38.660 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.660 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.660 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:38.660 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080515+0000", 2024-03-27T21:40:38.660 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0790000000000003e-06 2024-03-27T21:40:38.661 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.661 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.661 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:38.661 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080518+0000", 2024-03-27T21:40:38.661 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8509999999999999e-06 2024-03-27T21:40:38.661 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.661 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.661 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:38.661 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087444+0000", 2024-03-27T21:40:38.661 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0069259120000000002 2024-03-27T21:40:38.661 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.661 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.661 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:38.661 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087491+0000", 2024-03-27T21:40:38.661 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7043999999999999e-05 2024-03-27T21:40:38.662 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.662 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.662 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:38.662 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087564+0000", 2024-03-27T21:40:38.662 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3603999999999995e-05 2024-03-27T21:40:38.662 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.662 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.662 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:38.662 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087902+0000", 2024-03-27T21:40:38.662 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033734100000000001 2024-03-27T21:40:38.663 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.663 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.663 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:38.663 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088117+0000", 2024-03-27T21:40:38.663 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000215164 2024-03-27T21:40:38.663 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.663 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.663 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:38.663 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088160+0000", 2024-03-27T21:40:38.663 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2933999999999999e-05 2024-03-27T21:40:38.663 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.663 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.663 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:38.663 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088190+0000", 2024-03-27T21:40:38.663 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0139999999999999e-05 2024-03-27T21:40:38.664 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.664 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:38.664 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.664 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.664 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.664 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:38.664 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:38.664 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.123079203000003, 2024-03-27T21:40:38.664 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052465439999999997, 2024-03-27T21:40:38.664 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:38.664 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:38.664 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:38.664 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:38.664 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:38.664 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 133 2024-03-27T21:40:38.665 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.665 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:38.665 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.665 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:38.665 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:38.665 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.665 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.665 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.665 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:38.665 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:38.665 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.665 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.665 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.665 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:38.666 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:38.666 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:38.666 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.666 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.666 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:38.666 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496788+0000", 2024-03-27T21:40:38.666 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-03-27T21:40:38.666 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.666 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.666 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:38.666 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496793+0000", 2024-03-27T21:40:38.666 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4790000000000004e-06 2024-03-27T21:40:38.666 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.666 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.667 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:38.667 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496796+0000", 2024-03-27T21:40:38.667 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1e-06 2024-03-27T21:40:38.667 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.667 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.667 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:38.667 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500447+0000", 2024-03-27T21:40:38.667 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036504459999999999 2024-03-27T21:40:38.667 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.667 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.667 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:38.667 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500492+0000", 2024-03-27T21:40:38.667 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5765e-05 2024-03-27T21:40:38.667 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.667 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.668 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:38.668 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500562+0000", 2024-03-27T21:40:38.668 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9206000000000001e-05 2024-03-27T21:40:38.668 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.668 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.668 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:38.668 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501959+0000", 2024-03-27T21:40:38.668 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001397187 2024-03-27T21:40:38.668 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.668 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.668 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:38.668 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501984+0000", 2024-03-27T21:40:38.668 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.552e-05 2024-03-27T21:40:38.668 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.668 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.669 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:38.669 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502010+0000", 2024-03-27T21:40:38.669 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5232000000000002e-05 2024-03-27T21:40:38.669 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.669 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.669 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:38.669 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502032+0000", 2024-03-27T21:40:38.669 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2337999999999999e-05 2024-03-27T21:40:38.669 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.669 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:38.669 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.669 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.669 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.669 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:38.669 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:38.669 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.123020545000003, 2024-03-27T21:40:38.670 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052892119999999997, 2024-03-27T21:40:38.670 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:38.670 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:38.670 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:38.670 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:38.670 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:38.670 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 134 2024-03-27T21:40:38.670 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.670 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:38.670 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.670 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:38.670 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:38.670 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.670 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.670 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.671 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:38.671 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:38.671 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.671 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.671 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.671 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:38.671 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:38.671 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:38.671 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.671 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.671 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:38.671 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496929+0000", 2024-03-27T21:40:38.671 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5098000000000004e-05 2024-03-27T21:40:38.671 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.671 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.672 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:38.672 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496937+0000", 2024-03-27T21:40:38.672 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9559999999999997e-06 2024-03-27T21:40:38.672 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.672 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.672 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:38.672 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496942+0000", 2024-03-27T21:40:38.672 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6350000000000003e-06 2024-03-27T21:40:38.672 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.672 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.672 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:38.672 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500749+0000", 2024-03-27T21:40:38.672 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038068529999999998 2024-03-27T21:40:38.672 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.673 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.673 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:38.673 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500792+0000", 2024-03-27T21:40:38.673 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2823e-05 2024-03-27T21:40:38.673 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.673 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.673 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:38.673 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500864+0000", 2024-03-27T21:40:38.673 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1715e-05 2024-03-27T21:40:38.673 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.673 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.673 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:38.673 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502068+0000", 2024-03-27T21:40:38.673 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012044860000000001 2024-03-27T21:40:38.673 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.674 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.674 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:38.674 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502073+0000", 2024-03-27T21:40:38.674 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5700000000000003e-06 2024-03-27T21:40:38.674 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.674 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.674 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:38.674 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502098+0000", 2024-03-27T21:40:38.674 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5120999999999999e-05 2024-03-27T21:40:38.674 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.674 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.674 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:38.674 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502133+0000", 2024-03-27T21:40:38.674 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5306999999999999e-05 2024-03-27T21:40:38.674 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.675 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:38.675 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.675 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.675 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.675 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:38.675 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:38.675 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.122915237999997, 2024-03-27T21:40:38.675 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062529040000000001, 2024-03-27T21:40:38.675 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:38.675 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:38.675 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:38.675 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:38.675 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:38.675 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135 2024-03-27T21:40:38.675 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.676 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:38.676 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.676 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:38.676 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:38.676 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.676 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.676 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.676 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:38.676 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:38.676 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.676 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.676 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.676 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:38.676 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:38.676 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:38.677 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.677 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.677 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:38.677 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496953+0000", 2024-03-27T21:40:38.677 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0709999999999996e-06 2024-03-27T21:40:38.677 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.677 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.677 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:38.677 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496961+0000", 2024-03-27T21:40:38.677 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1300000000000001e-06 2024-03-27T21:40:38.677 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.677 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.677 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:38.677 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496968+0000", 2024-03-27T21:40:38.678 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0999999999999998e-06 2024-03-27T21:40:38.678 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.678 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.678 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:38.678 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501244+0000", 2024-03-27T21:40:38.678 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042756310000000002 2024-03-27T21:40:38.678 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.678 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.678 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:38.678 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501377+0000", 2024-03-27T21:40:38.678 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013348799999999999 2024-03-27T21:40:38.678 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.678 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.678 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:38.679 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501455+0000", 2024-03-27T21:40:38.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7410000000000006e-05 2024-03-27T21:40:38.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:38.679 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502138+0000", 2024-03-27T21:40:38.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068386599999999999 2024-03-27T21:40:38.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:38.679 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503152+0000", 2024-03-27T21:40:38.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010139680000000001 2024-03-27T21:40:38.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:38.680 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503180+0000", 2024-03-27T21:40:38.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7426000000000001e-05 2024-03-27T21:40:38.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.680 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.680 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:38.680 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503202+0000", 2024-03-27T21:40:38.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2463000000000002e-05 2024-03-27T21:40:38.680 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.680 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:38.680 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.680 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.680 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:38.680 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:38.681 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.122891099999997, 2024-03-27T21:40:38.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066253680000000004, 2024-03-27T21:40:38.681 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:38.681 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:38.681 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:38.681 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:38.681 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:38.681 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 136 2024-03-27T21:40:38.681 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.681 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:38.681 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.681 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:38.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:38.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.682 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:38.682 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:38.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.682 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:38.682 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:38.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:38.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.682 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:38.682 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496977+0000", 2024-03-27T21:40:38.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5049999999999999e-06 2024-03-27T21:40:38.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.683 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:38.683 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496984+0000", 2024-03-27T21:40:38.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1520000000000003e-06 2024-03-27T21:40:38.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.683 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:38.683 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496990+0000", 2024-03-27T21:40:38.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-03-27T21:40:38.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.683 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:38.683 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501669+0000", 2024-03-27T21:40:38.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046793859999999998 2024-03-27T21:40:38.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:38.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501714+0000", 2024-03-27T21:40:38.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5154999999999997e-05 2024-03-27T21:40:38.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:38.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501783+0000", 2024-03-27T21:40:38.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8454999999999994e-05 2024-03-27T21:40:38.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:38.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503226+0000", 2024-03-27T21:40:38.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001443441 2024-03-27T21:40:38.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:38.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503540+0000", 2024-03-27T21:40:38.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031347100000000001 2024-03-27T21:40:38.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:38.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503563+0000", 2024-03-27T21:40:38.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3223000000000001e-05 2024-03-27T21:40:38.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:38.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503599+0000", 2024-03-27T21:40:38.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6105000000000001e-05 2024-03-27T21:40:38.686 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.686 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:38.686 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.686 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.686 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.686 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:38.686 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:38.686 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.12286829, 2024-03-27T21:40:38.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074261320000000002, 2024-03-27T21:40:38.686 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:38.686 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:38.686 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:38.686 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:38.686 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:38.687 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 137 2024-03-27T21:40:38.687 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.687 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:38.687 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.687 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:38.687 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:38.687 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.687 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.687 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.687 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:38.687 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:38.687 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.687 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.687 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.687 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:38.688 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:38.688 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:38.688 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.688 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.688 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:38.688 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497077+0000", 2024-03-27T21:40:38.688 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1599000000000003e-05 2024-03-27T21:40:38.688 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.688 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.688 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:38.688 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497085+0000", 2024-03-27T21:40:38.688 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6109999999999996e-06 2024-03-27T21:40:38.688 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.688 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.688 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:38.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497092+0000", 2024-03-27T21:40:38.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.742e-06 2024-03-27T21:40:38.689 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:38.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502159+0000", 2024-03-27T21:40:38.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050666870000000003 2024-03-27T21:40:38.689 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:38.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502200+0000", 2024-03-27T21:40:38.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1919999999999998e-05 2024-03-27T21:40:38.689 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:38.690 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502280+0000", 2024-03-27T21:40:38.690 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9417000000000002e-05 2024-03-27T21:40:38.690 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.690 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:38.690 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503605+0000", 2024-03-27T21:40:38.690 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001325641 2024-03-27T21:40:38.690 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.690 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:38.690 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504361+0000", 2024-03-27T21:40:38.690 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075555399999999997 2024-03-27T21:40:38.690 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.691 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:38.691 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504386+0000", 2024-03-27T21:40:38.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5369999999999999e-05 2024-03-27T21:40:38.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.691 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.691 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:38.691 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504422+0000", 2024-03-27T21:40:38.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6019000000000002e-05 2024-03-27T21:40:38.691 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.691 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:38.691 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.691 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.691 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:38.691 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:38.692 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.122764492999998, 2024-03-27T21:40:38.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074059750000000004, 2024-03-27T21:40:38.692 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:38.692 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:38.692 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:38.692 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:38.692 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:38.692 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-03-27T21:40:38.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.692 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:38.692 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:38.692 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:38.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.693 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.693 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:38.693 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:38.693 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.693 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.693 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.693 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:38.693 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:38.693 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:38.693 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.693 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.693 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:38.693 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497103+0000", 2024-03-27T21:40:38.693 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.856e-06 2024-03-27T21:40:38.693 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.694 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.694 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:38.694 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497121+0000", 2024-03-27T21:40:38.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7190999999999999e-05 2024-03-27T21:40:38.694 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.694 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.694 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:38.694 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497126+0000", 2024-03-27T21:40:38.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3770000000000004e-06 2024-03-27T21:40:38.694 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.694 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.694 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:38.694 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502833+0000", 2024-03-27T21:40:38.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057073510000000003 2024-03-27T21:40:38.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:38.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502881+0000", 2024-03-27T21:40:38.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7961999999999999e-05 2024-03-27T21:40:38.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:38.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502949+0000", 2024-03-27T21:40:38.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7957e-05 2024-03-27T21:40:38.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:38.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504430+0000", 2024-03-27T21:40:38.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001480763 2024-03-27T21:40:38.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:38.696 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504459+0000", 2024-03-27T21:40:38.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8983999999999999e-05 2024-03-27T21:40:38.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:38.696 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504486+0000", 2024-03-27T21:40:38.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6883999999999999e-05 2024-03-27T21:40:38.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:38.696 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504506+0000", 2024-03-27T21:40:38.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0214e-05 2024-03-27T21:40:38.697 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.697 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:38.697 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.697 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.697 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.697 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:38.697 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:38.697 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.122732659, 2024-03-27T21:40:38.697 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074754859999999999, 2024-03-27T21:40:38.697 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:38.697 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:38.697 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:38.697 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:38.697 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:38.697 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-03-27T21:40:38.698 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.698 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:38.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.698 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:38.698 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:38.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.698 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.698 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:38.698 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:38.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.698 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.698 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:38.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:38.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:38.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:38.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497137+0000", 2024-03-27T21:40:38.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0730000000000004e-06 2024-03-27T21:40:38.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:38.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497143+0000", 2024-03-27T21:40:38.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3759999999999996e-06 2024-03-27T21:40:38.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:38.700 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497150+0000", 2024-03-27T21:40:38.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.002e-06 2024-03-27T21:40:38.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:38.700 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503249+0000", 2024-03-27T21:40:38.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060989750000000004 2024-03-27T21:40:38.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:38.700 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503291+0000", 2024-03-27T21:40:38.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1563999999999997e-05 2024-03-27T21:40:38.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:38.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503376+0000", 2024-03-27T21:40:38.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5752000000000002e-05 2024-03-27T21:40:38.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:38.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504545+0000", 2024-03-27T21:40:38.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169172 2024-03-27T21:40:38.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:38.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504550+0000", 2024-03-27T21:40:38.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7419999999999997e-06 2024-03-27T21:40:38.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.702 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:38.702 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504574+0000", 2024-03-27T21:40:38.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4124000000000001e-05 2024-03-27T21:40:38.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.702 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.702 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:38.702 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504607+0000", 2024-03-27T21:40:38.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3118999999999999e-05 2024-03-27T21:40:38.702 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.702 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:38.702 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.702 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.702 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:38.702 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:38.703 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.122708993000003, 2024-03-27T21:40:38.703 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078150589999999992, 2024-03-27T21:40:38.703 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:38.703 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:38.703 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:38.703 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:38.703 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:38.703 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-03-27T21:40:38.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.703 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:38.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.703 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:38.703 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:38.703 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:38.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:38.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:38.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497155+0000", 2024-03-27T21:40:38.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:38.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:38.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497230+0000", 2024-03-27T21:40:38.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4789000000000005e-05 2024-03-27T21:40:38.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:38.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497241+0000", 2024-03-27T21:40:38.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1442e-05 2024-03-27T21:40:38.705 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:38.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497255+0000", 2024-03-27T21:40:38.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3087e-05 2024-03-27T21:40:38.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:38.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503630+0000", 2024-03-27T21:40:38.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063755269999999998 2024-03-27T21:40:38.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:38.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503673+0000", 2024-03-27T21:40:38.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2616e-05 2024-03-27T21:40:38.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:38.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503738+0000", 2024-03-27T21:40:38.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5114e-05 2024-03-27T21:40:38.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.707 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:38.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504907+0000", 2024-03-27T21:40:38.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169432 2024-03-27T21:40:38.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.707 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:38.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504926+0000", 2024-03-27T21:40:38.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9086999999999999e-05 2024-03-27T21:40:38.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.707 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:38.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504950+0000", 2024-03-27T21:40:38.708 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3598999999999999e-05 2024-03-27T21:40:38.708 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.708 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.708 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:38.708 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504971+0000", 2024-03-27T21:40:38.708 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0772999999999999e-05 2024-03-27T21:40:38.708 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.708 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:38.708 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.708 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.708 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.708 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:38.708 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:38.709 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 58.122601953, 2024-03-27T21:40:38.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082962590000000003, 2024-03-27T21:40:38.709 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:38.709 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:38.709 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:38.709 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:38.709 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:38.709 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-03-27T21:40:38.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.709 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:38.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:38.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:38.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.710 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:38.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:38.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.710 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.710 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:38.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497262+0000", 2024-03-27T21:40:38.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:38.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.710 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.710 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:38.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497267+0000", 2024-03-27T21:40:38.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.955e-06 2024-03-27T21:40:38.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:38.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497275+0000", 2024-03-27T21:40:38.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.824e-06 2024-03-27T21:40:38.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:38.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497280+0000", 2024-03-27T21:40:38.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8230000000000002e-06 2024-03-27T21:40:38.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:38.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504628+0000", 2024-03-27T21:40:38.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007348561 2024-03-27T21:40:38.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:38.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504673+0000", 2024-03-27T21:40:38.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4313000000000001e-05 2024-03-27T21:40:38.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:38.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504745+0000", 2024-03-27T21:40:38.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2303000000000002e-05 2024-03-27T21:40:38.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:38.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505358+0000", 2024-03-27T21:40:38.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061281300000000003 2024-03-27T21:40:38.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.713 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:38.713 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505503+0000", 2024-03-27T21:40:38.713 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000145134 2024-03-27T21:40:38.713 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.713 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:38.713 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505526+0000", 2024-03-27T21:40:38.713 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3312e-05 2024-03-27T21:40:38.713 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.713 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:38.713 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505559+0000", 2024-03-27T21:40:38.713 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2821999999999998e-05 2024-03-27T21:40:38.713 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.714 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:38.714 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.714 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.714 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.714 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:38.714 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:38.714 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 57.513379714000003, 2024-03-27T21:40:38.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053753339999999998, 2024-03-27T21:40:38.714 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:38.714 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:38.714 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:38.714 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:38.714 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:38.714 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159 2024-03-27T21:40:38.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.715 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:38.715 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.715 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:38.715 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:38.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.715 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.715 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:38.715 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:38.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:38.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.715 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.715 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:38.715 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:38.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:38.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:38.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106488+0000", 2024-03-27T21:40:38.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9830000000000001e-06 2024-03-27T21:40:38.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:38.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106491+0000", 2024-03-27T21:40:38.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3459999999999998e-06 2024-03-27T21:40:38.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:38.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106493+0000", 2024-03-27T21:40:38.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.509e-06 2024-03-27T21:40:38.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:38.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110636+0000", 2024-03-27T21:40:38.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041427260000000002 2024-03-27T21:40:38.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:38.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110683+0000", 2024-03-27T21:40:38.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6557000000000002e-05 2024-03-27T21:40:38.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:38.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110755+0000", 2024-03-27T21:40:38.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2119000000000005e-05 2024-03-27T21:40:38.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:38.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111415+0000", 2024-03-27T21:40:38.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066065199999999996 2024-03-27T21:40:38.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:38.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111793+0000", 2024-03-27T21:40:38.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037732100000000001 2024-03-27T21:40:38.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:38.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111829+0000", 2024-03-27T21:40:38.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418999999999998e-05 2024-03-27T21:40:38.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:38.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:38.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:38.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111860+0000", 2024-03-27T21:40:38.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1003000000000001e-05 2024-03-27T21:40:38.719 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.719 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:38.719 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.719 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:38.719 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:38.719 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:38.721 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-27T21:40:38.721 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:38.734+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:38.821 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:38.835+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:38.830 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:38.830 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:38.830 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:38.831 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:38.840 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-27T21:40:38.924 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:38.938+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:38.999 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:38.999 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:38.999 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:38.999 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:38.999 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:39.009 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-27T21:40:39.024 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:39.037+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:39.125 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:39.139+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:39.159 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:39.160 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:39.160 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:39.160 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:40:39.160 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:39.169 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:40:39.226 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:39.240+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:39.291 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:39.291 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:39.291 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:39.291 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:39.303 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:40:39.326 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:39.340+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:39.411 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:39.411 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:39.411 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:39.411 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:39.411 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:39.420 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:40:39.465 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:39.479+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:39.527 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:39.541+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:39.531 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:39.531 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:39.531 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:39.531 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-03-27T21:40:39.532 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.532 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:20 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2024-03-27T21:40:39.532 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:39.532 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.656276267999999, 2024-03-27T21:40:39.532 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015345630000000001, 2024-03-27T21:40:39.532 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:39.532 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:39.532 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:39.532 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.532 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:39.532 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:39.532 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.533 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.533 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:39.533 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:39.533 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.533 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.533 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:39.533 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888562+0000", 2024-03-27T21:40:39.533 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:39.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.533 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.533 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:39.534 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888735+0000", 2024-03-27T21:40:39.534 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017254799999999999 2024-03-27T21:40:39.534 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.534 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.534 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:39.534 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888747+0000", 2024-03-27T21:40:39.534 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2469999999999999e-05 2024-03-27T21:40:39.534 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.534 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.534 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:39.534 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888761+0000", 2024-03-27T21:40:39.534 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3254e-05 2024-03-27T21:40:39.534 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.535 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.535 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:39.535 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888804+0000", 2024-03-27T21:40:39.535 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3129e-05 2024-03-27T21:40:39.535 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.535 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.535 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:39.535 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888820+0000", 2024-03-27T21:40:39.535 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6426000000000002e-05 2024-03-27T21:40:39.535 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.535 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.535 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:39.536 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890060+0000", 2024-03-27T21:40:39.536 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012398050000000001 2024-03-27T21:40:39.536 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.536 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.536 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:39.536 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890098+0000", 2024-03-27T21:40:39.536 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7795e-05 2024-03-27T21:40:39.536 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.536 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:39.536 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.536 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.536 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.537 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2024-03-27T21:40:39.537 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:39.537 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.655451868, 2024-03-27T21:40:39.537 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001069555, 2024-03-27T21:40:39.537 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:39.537 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:39.537 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:39.537 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.537 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:39.537 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:39.537 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.537 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.537 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.537 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:39.538 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:39.538 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.538 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.538 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.538 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:39.538 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889387+0000", 2024-03-27T21:40:39.538 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:39.538 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.538 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.538 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:39.538 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889455+0000", 2024-03-27T21:40:39.538 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7930999999999997e-05 2024-03-27T21:40:39.538 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.539 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.539 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:39.539 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889470+0000", 2024-03-27T21:40:39.539 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5554e-05 2024-03-27T21:40:39.539 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.539 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.539 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:39.539 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889476+0000", 2024-03-27T21:40:39.539 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0619999999999997e-06 2024-03-27T21:40:39.539 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.539 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.539 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:39.539 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889513+0000", 2024-03-27T21:40:39.540 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6606999999999997e-05 2024-03-27T21:40:39.540 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.540 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.540 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:39.540 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889528+0000", 2024-03-27T21:40:39.540 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4982e-05 2024-03-27T21:40:39.540 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.540 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.540 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:39.540 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890421+0000", 2024-03-27T21:40:39.540 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00089293700000000001 2024-03-27T21:40:39.540 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.540 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.540 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:39.541 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890457+0000", 2024-03-27T21:40:39.541 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6260000000000002e-05 2024-03-27T21:40:39.541 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.541 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:39.541 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.541 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.541 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.541 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:38 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'24, mlcod=17'22)", 2024-03-27T21:40:39.541 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:39.542 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.638215901999999, 2024-03-27T21:40:39.542 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016182080000000001, 2024-03-27T21:40:39.542 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:39.542 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:39.543 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:39.543 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.543 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:39.543 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:39.543 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.544 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.544 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.544 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:39.544 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:39.544 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.544 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.544 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.544 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:39.544 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906623+0000", 2024-03-27T21:40:39.544 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:39.545 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.545 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.545 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:39.545 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906687+0000", 2024-03-27T21:40:39.545 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3706000000000003e-05 2024-03-27T21:40:39.545 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.545 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.545 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:39.545 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906698+0000", 2024-03-27T21:40:39.545 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1752e-05 2024-03-27T21:40:39.545 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.545 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.545 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:39.545 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906704+0000", 2024-03-27T21:40:39.545 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3940000000000004e-06 2024-03-27T21:40:39.546 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.546 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.546 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:39.546 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906741+0000", 2024-03-27T21:40:39.546 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6944999999999999e-05 2024-03-27T21:40:39.546 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.549 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.549 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:39.549 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906760+0000", 2024-03-27T21:40:39.549 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9293e-05 2024-03-27T21:40:39.550 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.550 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.550 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:39.550 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908211+0000", 2024-03-27T21:40:39.550 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014511350000000001 2024-03-27T21:40:39.550 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.550 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.551 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:39.551 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908242+0000", 2024-03-27T21:40:39.551 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0712e-05 2024-03-27T21:40:39.551 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.551 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:39.551 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.551 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.551 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.551 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:39 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'25, mlcod=17'22)", 2024-03-27T21:40:39.551 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:39.551 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.637370986999997, 2024-03-27T21:40:39.551 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012060199999999999, 2024-03-27T21:40:39.551 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:39.552 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:39.552 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:39.552 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.552 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:39.552 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:39.552 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.552 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.552 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.552 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:39.552 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:39.552 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.552 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.553 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.553 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:39.553 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:39.553 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:39.553 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.553 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.553 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:39.553 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907533+0000", 2024-03-27T21:40:39.553 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5675999999999999e-05 2024-03-27T21:40:39.553 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.553 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.553 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:39.553 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907545+0000", 2024-03-27T21:40:39.553 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.115e-05 2024-03-27T21:40:39.554 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.554 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.554 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:39.554 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907550+0000", 2024-03-27T21:40:39.554 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8320000000000002e-06 2024-03-27T21:40:39.554 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.554 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.554 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:39.554 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907592+0000", 2024-03-27T21:40:39.554 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1309e-05 2024-03-27T21:40:39.554 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.555 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.555 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:39.555 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907611+0000", 2024-03-27T21:40:39.555 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9009e-05 2024-03-27T21:40:39.556 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.556 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.557 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:39.557 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908638+0000", 2024-03-27T21:40:39.557 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001027533 2024-03-27T21:40:39.557 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.557 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.557 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:39.557 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908675+0000", 2024-03-27T21:40:39.557 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6329999999999999e-05 2024-03-27T21:40:39.557 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.557 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:39.557 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.557 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.557 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.557 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:57 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'35, mlcod=17'33)", 2024-03-27T21:40:39.558 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:39.558 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.623286555, 2024-03-27T21:40:39.558 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001346578, 2024-03-27T21:40:39.558 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:39.558 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:39.558 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:39.558 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.558 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:39.558 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:39.558 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.558 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.559 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.559 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:39.559 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:39.559 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.559 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.559 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.559 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:39.559 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921552+0000", 2024-03-27T21:40:39.559 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:39.559 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.559 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.559 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:39.559 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921615+0000", 2024-03-27T21:40:39.559 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2787999999999996e-05 2024-03-27T21:40:39.559 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.560 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.560 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:39.560 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921626+0000", 2024-03-27T21:40:39.560 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0888e-05 2024-03-27T21:40:39.560 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.560 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.560 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:39.560 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921635+0000", 2024-03-27T21:40:39.560 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4639999999999995e-06 2024-03-27T21:40:39.560 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.560 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.560 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:39.560 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921673+0000", 2024-03-27T21:40:39.560 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7891e-05 2024-03-27T21:40:39.561 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.561 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.561 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:39.561 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921694+0000", 2024-03-27T21:40:39.561 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0426000000000001e-05 2024-03-27T21:40:39.561 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.561 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.561 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:39.561 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922865+0000", 2024-03-27T21:40:39.561 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011709509999999999 2024-03-27T21:40:39.561 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.561 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.561 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:39.561 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922900+0000", 2024-03-27T21:40:39.561 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4819e-05 2024-03-27T21:40:39.562 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.562 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:39.562 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.562 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.562 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.562 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:71 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'45, mlcod=17'43)", 2024-03-27T21:40:39.562 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:39.562 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.607987842999997, 2024-03-27T21:40:39.562 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001460907, 2024-03-27T21:40:39.562 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:39.562 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:39.562 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:39.562 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.563 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:39.563 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:39.563 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.563 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.563 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.563 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:39.563 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:39.563 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.563 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.563 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.563 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:39.563 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936851+0000", 2024-03-27T21:40:39.563 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:39.563 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.563 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.563 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:39.564 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936903+0000", 2024-03-27T21:40:39.564 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2302999999999997e-05 2024-03-27T21:40:39.564 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.564 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.564 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:39.564 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936915+0000", 2024-03-27T21:40:39.564 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1967000000000001e-05 2024-03-27T21:40:39.564 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.564 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.564 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:39.564 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936923+0000", 2024-03-27T21:40:39.564 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9370000000000004e-06 2024-03-27T21:40:39.564 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.565 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.565 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:39.565 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936970+0000", 2024-03-27T21:40:39.565 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6987000000000003e-05 2024-03-27T21:40:39.565 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.565 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.565 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:39.565 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937002+0000", 2024-03-27T21:40:39.565 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2410999999999999e-05 2024-03-27T21:40:39.565 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.565 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.565 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:39.565 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938279+0000", 2024-03-27T21:40:39.566 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001276552 2024-03-27T21:40:39.566 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.566 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.566 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:39.566 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938313+0000", 2024-03-27T21:40:39.566 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3526999999999999e-05 2024-03-27T21:40:39.566 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.566 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:39.566 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.566 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.566 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.566 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:72 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'46, mlcod=17'43)", 2024-03-27T21:40:39.566 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:39.567 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.607350281000002, 2024-03-27T21:40:39.567 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014239490000000001, 2024-03-27T21:40:39.567 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:39.567 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:39.567 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:39.567 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.567 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:39.567 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:39.567 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.567 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.567 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.568 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:39.568 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:39.568 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.568 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.568 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.568 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:39.568 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937488+0000", 2024-03-27T21:40:39.568 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:39.568 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.568 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.568 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:39.568 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937547+0000", 2024-03-27T21:40:39.568 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8196999999999999e-05 2024-03-27T21:40:39.568 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.569 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.569 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:39.569 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937559+0000", 2024-03-27T21:40:39.569 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1928e-05 2024-03-27T21:40:39.569 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.569 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.569 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:39.569 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937564+0000", 2024-03-27T21:40:39.569 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4759999999999997e-06 2024-03-27T21:40:39.569 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.569 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.569 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:39.570 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937621+0000", 2024-03-27T21:40:39.570 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6712000000000002e-05 2024-03-27T21:40:39.570 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.570 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.570 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:39.570 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937640+0000", 2024-03-27T21:40:39.570 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9517e-05 2024-03-27T21:40:39.570 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.570 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.570 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:39.570 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938883+0000", 2024-03-27T21:40:39.570 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001242567 2024-03-27T21:40:39.570 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.570 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.571 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:39.571 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938913+0000", 2024-03-27T21:40:39.571 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0287e-05 2024-03-27T21:40:39.571 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.571 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:39.571 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.571 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.571 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.571 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:73 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'47, mlcod=17'43)", 2024-03-27T21:40:39.571 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:39.571 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.606674187000003, 2024-03-27T21:40:39.571 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010746239999999999, 2024-03-27T21:40:39.571 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:39.571 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:39.571 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.573 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.573 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:39.573 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938217+0000", 2024-03-27T21:40:39.573 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2743e-05 2024-03-27T21:40:39.573 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.573 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.573 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:39.573 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938229+0000", 2024-03-27T21:40:39.573 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1956999999999999e-05 2024-03-27T21:40:39.573 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.573 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.573 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:39.574 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938235+0000", 2024-03-27T21:40:39.574 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4589999999999997e-06 2024-03-27T21:40:39.574 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.574 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.574 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:39.574 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938341+0000", 2024-03-27T21:40:39.574 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000106054 2024-03-27T21:40:39.574 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.574 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.574 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:39.574 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938369+0000", 2024-03-27T21:40:39.574 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8291000000000001e-05 2024-03-27T21:40:39.574 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.575 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.575 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:39.575 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939209+0000", 2024-03-27T21:40:39.575 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084045400000000003 2024-03-27T21:40:39.575 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.576 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.576 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:39.576 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939240+0000", 2024-03-27T21:40:39.576 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0429999999999998e-05 2024-03-27T21:40:39.576 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.576 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:39.576 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.576 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.576 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.576 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:75 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'49, mlcod=17'47)", 2024-03-27T21:40:39.576 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:39.577 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.600236539999997, 2024-03-27T21:40:39.577 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001017001, 2024-03-27T21:40:39.577 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:39.577 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:39.577 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:39.577 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.577 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:39.577 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:39.577 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.577 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.577 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.577 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:39.578 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:39.578 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.578 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.578 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.578 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:39.578 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944601+0000", 2024-03-27T21:40:39.578 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999986 2024-03-27T21:40:39.578 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.578 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.578 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:39.578 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944614+0000", 2024-03-27T21:40:39.578 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2496e-05 2024-03-27T21:40:39.579 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.579 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.579 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:39.579 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944640+0000", 2024-03-27T21:40:39.579 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5865999999999999e-05 2024-03-27T21:40:39.579 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.579 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.579 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:39.579 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944651+0000", 2024-03-27T21:40:39.579 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1797000000000001e-05 2024-03-27T21:40:39.579 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.579 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.579 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:39.579 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944762+0000", 2024-03-27T21:40:39.579 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110412 2024-03-27T21:40:39.580 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.580 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.580 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:39.580 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944820+0000", 2024-03-27T21:40:39.580 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7649999999999999e-05 2024-03-27T21:40:39.580 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.580 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.580 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:39.580 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945585+0000", 2024-03-27T21:40:39.580 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076562600000000005 2024-03-27T21:40:39.580 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.580 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.580 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:39.580 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945620+0000", 2024-03-27T21:40:39.581 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4764000000000001e-05 2024-03-27T21:40:39.581 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.581 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:39.581 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.581 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.581 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.581 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:86 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'50)", 2024-03-27T21:40:39.581 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:39.581 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.592475053000001, 2024-03-27T21:40:39.581 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011392380000000001, 2024-03-27T21:40:39.581 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:39.582 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:39.582 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:39.582 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.582 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:39.582 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:39.582 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.582 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.582 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.582 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:39.582 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:39.582 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.582 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.583 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.583 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:39.583 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:39.583 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:39.583 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.583 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.583 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:39.583 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952373+0000", 2024-03-27T21:40:39.583 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9220000000000003e-06 2024-03-27T21:40:39.583 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.583 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.583 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:39.583 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952381+0000", 2024-03-27T21:40:39.583 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3070000000000003e-06 2024-03-27T21:40:39.583 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.584 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.584 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:39.584 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952385+0000", 2024-03-27T21:40:39.584 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.121e-06 2024-03-27T21:40:39.584 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.584 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.584 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:39.584 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952419+0000", 2024-03-27T21:40:39.584 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4082999999999999e-05 2024-03-27T21:40:39.584 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.584 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.584 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:39.584 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952440+0000", 2024-03-27T21:40:39.584 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0837999999999999e-05 2024-03-27T21:40:39.585 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.585 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.585 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:39.585 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953472+0000", 2024-03-27T21:40:39.585 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001032119 2024-03-27T21:40:39.585 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.585 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.585 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:39.585 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953504+0000", 2024-03-27T21:40:39.585 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1250000000000001e-05 2024-03-27T21:40:39.585 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.585 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:39.585 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.585 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.585 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.586 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:87 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-27T21:40:39.586 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:39.586 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.592107796000001, 2024-03-27T21:40:39.586 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00135519, 2024-03-27T21:40:39.586 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:39.586 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:39.586 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:39.586 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.586 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:39.586 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:39.586 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.586 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.586 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.587 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:39.587 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:39.587 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.587 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.587 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.587 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:39.587 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952731+0000", 2024-03-27T21:40:39.587 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:39.587 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.587 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.587 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:39.587 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952781+0000", 2024-03-27T21:40:39.587 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0161000000000003e-05 2024-03-27T21:40:39.587 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.587 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.588 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:39.588 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952794+0000", 2024-03-27T21:40:39.588 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2853e-05 2024-03-27T21:40:39.588 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.588 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.588 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:39.588 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952799+0000", 2024-03-27T21:40:39.588 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5560000000000001e-06 2024-03-27T21:40:39.588 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.588 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.588 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:39.588 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952832+0000", 2024-03-27T21:40:39.588 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3834999999999997e-05 2024-03-27T21:40:39.588 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.589 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.589 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:39.589 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952851+0000", 2024-03-27T21:40:39.589 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8289999999999999e-05 2024-03-27T21:40:39.589 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.589 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.589 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:39.589 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954055+0000", 2024-03-27T21:40:39.589 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012037949999999999 2024-03-27T21:40:39.589 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.589 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.589 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:39.589 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954087+0000", 2024-03-27T21:40:39.589 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2292999999999997e-05 2024-03-27T21:40:39.589 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.590 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:39.590 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.590 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.590 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.590 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:89 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'59, mlcod=17'57)", 2024-03-27T21:40:39.590 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:39.590 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.589833575, 2024-03-27T21:40:39.590 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00138893, 2024-03-27T21:40:39.590 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:39.590 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:39.590 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:39.590 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.590 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:39.590 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:39.591 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.591 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.591 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.591 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:39.591 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:39.591 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.591 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.591 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.591 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:39.591 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955005+0000", 2024-03-27T21:40:39.591 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:39.591 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.591 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.591 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:39.592 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955071+0000", 2024-03-27T21:40:39.592 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5290999999999995e-05 2024-03-27T21:40:39.592 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.592 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.592 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:39.592 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955081+0000", 2024-03-27T21:40:39.592 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0572e-05 2024-03-27T21:40:39.592 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.592 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.592 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:39.592 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955086+0000", 2024-03-27T21:40:39.592 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8380000000000001e-06 2024-03-27T21:40:39.592 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.592 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.592 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:39.593 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955122+0000", 2024-03-27T21:40:39.593 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5698000000000003e-05 2024-03-27T21:40:39.593 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.593 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.593 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:39.593 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955140+0000", 2024-03-27T21:40:39.594 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7900000000000001e-05 2024-03-27T21:40:39.594 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.594 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.594 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:39.594 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956366+0000", 2024-03-27T21:40:39.594 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001226471 2024-03-27T21:40:39.595 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.595 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.595 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:39.595 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956395+0000", 2024-03-27T21:40:39.595 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8782e-05 2024-03-27T21:40:39.595 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.595 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:39.596 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.596 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.596 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.596 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:90 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'60, mlcod=17'57)", 2024-03-27T21:40:39.596 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:39.596 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.589148997999999, 2024-03-27T21:40:39.596 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010778859999999999, 2024-03-27T21:40:39.596 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:39.596 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:39.596 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:39.596 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.596 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:39.596 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:39.596 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.597 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.597 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.597 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:39.597 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:39.597 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.597 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.597 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.597 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:39.597 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:39.597 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:39.597 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.597 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.597 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:39.597 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955747+0000", 2024-03-27T21:40:39.598 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7130999999999999e-05 2024-03-27T21:40:39.598 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.598 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.598 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:39.598 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955756+0000", 2024-03-27T21:40:39.598 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2750000000000001e-06 2024-03-27T21:40:39.598 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.598 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.598 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:39.598 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955761+0000", 2024-03-27T21:40:39.598 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3900000000000003e-06 2024-03-27T21:40:39.598 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.599 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.599 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:39.599 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955797+0000", 2024-03-27T21:40:39.599 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6650000000000003e-05 2024-03-27T21:40:39.599 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.599 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.599 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:39.599 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955816+0000", 2024-03-27T21:40:39.599 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8195000000000001e-05 2024-03-27T21:40:39.599 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.599 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.599 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:39.599 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956739+0000", 2024-03-27T21:40:39.600 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00092369700000000002 2024-03-27T21:40:39.600 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.600 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.600 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:39.600 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956768+0000", 2024-03-27T21:40:39.600 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9048e-05 2024-03-27T21:40:39.600 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.600 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:39.600 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.600 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.600 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.600 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:118 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'80, mlcod=17'78)", 2024-03-27T21:40:39.600 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:39.600 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.456158174999999, 2024-03-27T21:40:39.601 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014200669999999999, 2024-03-27T21:40:39.601 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:39.601 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:39.601 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:39.601 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.601 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:39.601 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:39.601 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.601 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.601 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.601 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:39.601 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:39.601 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.601 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.602 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.602 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:39.602 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:39.602 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:39.602 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.602 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.602 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:39.602 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088756+0000", 2024-03-27T21:40:39.602 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4902999999999998e-05 2024-03-27T21:40:39.602 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.602 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.602 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:39.602 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088764+0000", 2024-03-27T21:40:39.603 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8169999999999999e-06 2024-03-27T21:40:39.603 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.603 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.603 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:39.603 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088767+0000", 2024-03-27T21:40:39.603 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8310000000000003e-06 2024-03-27T21:40:39.603 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.603 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.603 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:39.603 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088798+0000", 2024-03-27T21:40:39.603 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0800999999999999e-05 2024-03-27T21:40:39.603 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.603 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.604 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:39.604 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088817+0000", 2024-03-27T21:40:39.604 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8467000000000001e-05 2024-03-27T21:40:39.604 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.604 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.604 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:39.604 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090074+0000", 2024-03-27T21:40:39.604 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012574089999999999 2024-03-27T21:40:39.604 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.604 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.604 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:39.604 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090101+0000", 2024-03-27T21:40:39.604 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7223e-05 2024-03-27T21:40:39.605 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.605 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:39.605 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.605 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.605 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.605 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:119 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'81, mlcod=17'78)", 2024-03-27T21:40:39.605 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:39.605 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.455423379999999, 2024-03-27T21:40:39.605 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014990870000000001, 2024-03-27T21:40:39.605 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:39.605 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:39.605 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:39.605 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.605 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:39.606 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:39.606 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.606 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.606 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.606 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:39.606 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:39.606 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.606 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.606 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.606 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:39.606 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:39.606 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:39.607 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.607 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.607 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:39.607 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089491+0000", 2024-03-27T21:40:39.607 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4805e-05 2024-03-27T21:40:39.607 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.607 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.607 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:39.607 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089498+0000", 2024-03-27T21:40:39.607 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0650000000000001e-06 2024-03-27T21:40:39.607 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.607 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.607 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:39.607 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089501+0000", 2024-03-27T21:40:39.608 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.54e-06 2024-03-27T21:40:39.608 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.608 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.608 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:39.608 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089533+0000", 2024-03-27T21:40:39.608 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1838999999999998e-05 2024-03-27T21:40:39.608 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.608 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.608 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:39.608 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089550+0000", 2024-03-27T21:40:39.608 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7465999999999998e-05 2024-03-27T21:40:39.608 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.608 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.608 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:39.608 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090890+0000", 2024-03-27T21:40:39.609 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013397979999999999 2024-03-27T21:40:39.609 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.609 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.609 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:39.609 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090915+0000", 2024-03-27T21:40:39.609 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4905e-05 2024-03-27T21:40:39.609 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.609 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:39.609 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.609 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.609 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.609 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:120 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'82, mlcod=17'78)", 2024-03-27T21:40:39.609 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:39.609 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.454623499999997, 2024-03-27T21:40:39.610 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001527093, 2024-03-27T21:40:39.610 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:39.610 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:39.610 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:39.610 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.610 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:39.610 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:39.610 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.610 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.610 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.610 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:39.610 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:39.610 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.610 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.610 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.611 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:39.611 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:39.611 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:39.611 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.611 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.611 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:39.611 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090267+0000", 2024-03-27T21:40:39.611 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1477000000000002e-05 2024-03-27T21:40:39.611 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.611 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.611 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:39.611 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090273+0000", 2024-03-27T21:40:39.611 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9519999999999999e-06 2024-03-27T21:40:39.611 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.611 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.612 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:39.612 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090276+0000", 2024-03-27T21:40:39.612 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.219e-06 2024-03-27T21:40:39.612 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.612 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.612 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:39.612 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090304+0000", 2024-03-27T21:40:39.612 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8073999999999999e-05 2024-03-27T21:40:39.612 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.612 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.612 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:39.612 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090316+0000", 2024-03-27T21:40:39.612 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2088e-05 2024-03-27T21:40:39.612 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.612 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.613 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:39.613 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091717+0000", 2024-03-27T21:40:39.613 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014003889999999999 2024-03-27T21:40:39.613 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.613 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.613 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:39.613 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091743+0000", 2024-03-27T21:40:39.613 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6234e-05 2024-03-27T21:40:39.613 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.613 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:39.613 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.613 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.614 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.614 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:121 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'83, mlcod=17'78)", 2024-03-27T21:40:39.614 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:39.614 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.453813134999997, 2024-03-27T21:40:39.614 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010226899999999999, 2024-03-27T21:40:39.614 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:39.614 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:39.614 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:39.614 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.614 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:39.615 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:39.615 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.615 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.615 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.615 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:39.615 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:39.615 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.615 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.615 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.615 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:39.615 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:39.615 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:39.615 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.616 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.616 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:39.616 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091142+0000", 2024-03-27T21:40:39.616 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011608300000000001 2024-03-27T21:40:39.616 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.616 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.616 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:39.616 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091151+0000", 2024-03-27T21:40:39.616 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2350000000000005e-06 2024-03-27T21:40:39.616 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.616 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.616 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:39.616 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091156+0000", 2024-03-27T21:40:39.616 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5419999999999999e-06 2024-03-27T21:40:39.616 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.617 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.617 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:39.617 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091191+0000", 2024-03-27T21:40:39.617 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5453000000000001e-05 2024-03-27T21:40:39.617 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.617 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.617 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:39.617 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091208+0000", 2024-03-27T21:40:39.617 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7119999999999999e-05 2024-03-27T21:40:39.617 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.617 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.618 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:39.618 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092024+0000", 2024-03-27T21:40:39.618 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000815539 2024-03-27T21:40:39.618 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.618 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.618 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:39.618 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092049+0000", 2024-03-27T21:40:39.618 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5171e-05 2024-03-27T21:40:39.618 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.618 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:39.618 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.618 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.618 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.618 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:142 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'100, mlcod=18'98)", 2024-03-27T21:40:39.619 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:39.619 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.038806180999998, 2024-03-27T21:40:39.619 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001198778, 2024-03-27T21:40:39.619 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:39.619 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:39.619 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:39.619 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.619 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:39.619 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:39.619 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.619 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.619 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.619 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:39.619 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:39.620 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.620 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.620 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.620 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:39.620 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:39.620 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:39.620 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.620 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.620 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:39.620 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506101+0000", 2024-03-27T21:40:39.620 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8202999999999997e-05 2024-03-27T21:40:39.620 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.620 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.620 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:39.621 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506109+0000", 2024-03-27T21:40:39.621 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3059999999999998e-06 2024-03-27T21:40:39.621 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.621 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.621 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:39.621 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506113+0000", 2024-03-27T21:40:39.621 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0119999999999999e-06 2024-03-27T21:40:39.621 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.621 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.621 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:39.621 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506137+0000", 2024-03-27T21:40:39.621 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4043999999999999e-05 2024-03-27T21:40:39.621 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.621 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.622 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:39.622 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506148+0000", 2024-03-27T21:40:39.622 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0446e-05 2024-03-27T21:40:39.622 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.622 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.622 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:39.622 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507213+0000", 2024-03-27T21:40:39.622 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001065215 2024-03-27T21:40:39.622 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.622 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.622 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:39.622 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507232+0000", 2024-03-27T21:40:39.622 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9038e-05 2024-03-27T21:40:39.622 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.623 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:39.623 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.623 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.623 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.623 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:143 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'101, mlcod=18'98)", 2024-03-27T21:40:39.623 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:39.623 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.038193561999996, 2024-03-27T21:40:39.623 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001357488, 2024-03-27T21:40:39.623 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:39.623 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:39.623 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:39.623 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.623 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:39.624 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:39.624 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.624 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.624 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.624 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:39.624 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:39.624 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.624 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.624 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.624 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:39.624 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:39.624 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:39.624 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.625 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.625 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:39.625 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506723+0000", 2024-03-27T21:40:39.625 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7404000000000006e-05 2024-03-27T21:40:39.625 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.625 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.625 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:39.625 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506730+0000", 2024-03-27T21:40:39.625 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5749999999999997e-06 2024-03-27T21:40:39.625 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.626 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.626 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:39.626 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506733+0000", 2024-03-27T21:40:39.626 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7979999999999999e-06 2024-03-27T21:40:39.626 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.626 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.626 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:39.626 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506758+0000", 2024-03-27T21:40:39.626 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4179e-05 2024-03-27T21:40:39.626 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.626 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.626 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:39.626 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506768+0000", 2024-03-27T21:40:39.627 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0740000000000001e-05 2024-03-27T21:40:39.627 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.627 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.627 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:39.627 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507978+0000", 2024-03-27T21:40:39.627 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012093900000000001 2024-03-27T21:40:39.627 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.627 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.627 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:39.627 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508003+0000", 2024-03-27T21:40:39.627 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5667e-05 2024-03-27T21:40:39.627 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.627 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:39.628 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.628 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.628 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.628 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:144 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'102, mlcod=18'98)", 2024-03-27T21:40:39.628 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:39.628 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.037534665000003, 2024-03-27T21:40:39.628 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013453390000000001, 2024-03-27T21:40:39.628 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:39.628 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:39.628 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:39.628 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.628 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:39.629 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:39.629 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.629 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.629 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.629 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:39.629 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:39.629 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:39.629 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.629 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.629 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:39.629 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507304+0000", 2024-03-27T21:40:39.629 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:39.629 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.629 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.629 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:39.630 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507397+0000", 2024-03-27T21:40:39.630 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2995e-05 2024-03-27T21:40:39.630 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.630 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.630 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:39.630 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507403+0000", 2024-03-27T21:40:39.630 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8640000000000001e-06 2024-03-27T21:40:39.630 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.630 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.630 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:39.630 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507407+0000", 2024-03-27T21:40:39.630 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5710000000000002e-06 2024-03-27T21:40:39.630 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.631 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.631 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:39.631 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507428+0000", 2024-03-27T21:40:39.631 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1192e-05 2024-03-27T21:40:39.631 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.631 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.631 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:39.631 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507438+0000", 2024-03-27T21:40:39.631 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8309999999999996e-06 2024-03-27T21:40:39.631 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.631 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.631 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:39.631 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508624+0000", 2024-03-27T21:40:39.631 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011861210000000001 2024-03-27T21:40:39.631 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:39.632 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:39.632 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:39.632 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508650+0000", 2024-03-27T21:40:39.632 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6118e-05 2024-03-27T21:40:39.632 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.632 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:39.632 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.632 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:39.632 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:39.632 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:39.634 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:40:39.634 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:39.647+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:39.734 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:39.748+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:39.835 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:39.849+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:39.847 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:39.847 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:39.847 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:39.847 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:39.860 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:40:39.944 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:39.958+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:40.012 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:40.013 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:40.013 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:40.013 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:40.013 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:40.022 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:40:40.064 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:40.078+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:40.138 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:40.152+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:40.172 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:40.172 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:40.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:40.173 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:40:40.173 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:40.182 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:40:40.239 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:40.253+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:40.332 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:40.332 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:40.332 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:40.333 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:40.340 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:40.354+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:40.342 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:40:40.415 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:40:40.459 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-27T21:40:40.460 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:40.473+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:40.477 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:40.477 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:40.478 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:40.478 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:40.478 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:40.491 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:40:40.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.573+0000 7f622f38e640 10 monclient: get_monmap_and_config 2024-03-27T21:40:40.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.573+0000 7f622f38e640 10 monclient: build_initial_monmap 2024-03-27T21:40:40.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.573+0000 7f622f38e640 10 monclient: monmap: 2024-03-27T21:40:40.560 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:40.560 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:40.560 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:40.574563+0000 2024-03-27T21:40:40.560 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:40.574563+0000 2024-03-27T21:40:40.560 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:40.561 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:40.561 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:40.561 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:40.561 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:40.561 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:40.561 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:40.561 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:40.561 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:40.574+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:40.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.574+0000 7f622f38e640 1 Processor -- start 2024-03-27T21:40:40.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.574+0000 7f622f38e640 1 -- start start 2024-03-27T21:40:40.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.574+0000 7f622f38e640 10 monclient: init 2024-03-27T21:40:40.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.575+0000 7f622f38e640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:40.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.575+0000 7f622f38e640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-03-27T21:40:40.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.575+0000 7f622f38e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f62280b40b0 0x7f62280a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:40.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.575+0000 7f622f38e640 10 monclient(hunting): picked mon.noname-a con 0x7f62280b40b0 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:40.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.575+0000 7f622f38e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f622813b020 0x7f622813b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:40.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.575+0000 7f622f38e640 10 monclient(hunting): picked mon.noname-b con 0x7f622813b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:40.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.575+0000 7f622f38e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62281416f0 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:40.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.575+0000 7f622f38e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62280b3830 con 0x7f62281416f0 2024-03-27T21:40:40.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.575+0000 7f622f38e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:40.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.575+0000 7f622f38e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62280b4490 con 0x7f62280b40b0 2024-03-27T21:40:40.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.575+0000 7f622f38e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:40.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.575+0000 7f622f38e640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6228142290 con 0x7f622813b020 2024-03-27T21:40:40.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.575+0000 7f622f38e640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:40.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.575+0000 7f622f38e640 10 monclient(hunting): authenticate will time out at 881.796614s 2024-03-27T21:40:40.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622d12a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f62280b40b0 0x7f62280a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:40.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622d12a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f62280b40b0 0x7f62280a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58306/0 (socket says 172.21.15.60:58306) 2024-03-27T21:40:40.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622d12a640 1 -- 172.21.15.60:0/774651551 learned_addr learned my addr 172.21.15.60:0/774651551 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:40.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622d12a640 10 monclient(hunting): get_auth_request con 0x7f62280b40b0 auth_method 0 2024-03-27T21:40:40.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622d12a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:40.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622d12a640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:40.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622d12a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:40.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622c929640 1 --2- 172.21.15.60:0/774651551 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f622813b020 0x7f622813b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:40.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622c929640 10 monclient(hunting): get_auth_request con 0x7f622813b020 auth_method 0 2024-03-27T21:40:40.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622c929640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:40.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622c929640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:40.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622c929640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:40.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622c929640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:40.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622c929640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:40.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622c929640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:40.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622d12a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:40.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622d12a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:40.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622d12a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:40.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622c929640 10 monclient(hunting): handle_auth_done global_id 4568 payload 274 2024-03-27T21:40:40.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622c929640 1 -- 172.21.15.60:0/774651551 >> v2:172.21.15.60:3300/0 conn(0x7f62280b40b0 msgr2=0x7f62280a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:40.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622c929640 1 --2- 172.21.15.60:0/774651551 >> v2:172.21.15.60:3300/0 conn(0x7f62280b40b0 0x7f62280a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:40.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622c929640 1 -- 172.21.15.60:0/774651551 >> v1:172.21.15.60:6789/0 conn(0x7f62281416f0 legacy=0x7f6228141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:40.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622c929640 10 monclient: _finish_hunting 0 2024-03-27T21:40:40.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622c929640 1 monclient: found mon.noname-b 2024-03-27T21:40:40.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622c929640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:40.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622c929640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:40.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.576+0000 7f622c929640 1 -- 172.21.15.60:0/774651551 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62280aa520 con 0x7f622813b020 2024-03-27T21:40:40.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.578+0000 7f622c929640 1 --2- 172.21.15.60:0/774651551 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f622813b020 0x7f622813b480 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6218009a00 tx=0x7f621802bb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=584817bde8fe7813 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:40.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.578+0000 7f621ffff640 1 -- 172.21.15.60:0/774651551 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621802d6c0 con 0x7f622813b020 2024-03-27T21:40:40.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.578+0000 7f621ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:40.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.578+0000 7f621ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:40.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.578+0000 7f621ffff640 10 monclient: dump: 2024-03-27T21:40:40.568 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:40.569 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.578+0000 7f621ffff640 10 monclient: _finish_auth 0 2024-03-27T21:40:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.578+0000 7f621ffff640 10 monclient: _check_auth_tickets 2024-03-27T21:40:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.578+0000 7f621ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.578+0000 7f622f38e640 5 monclient: authenticate success, global_id 4568 2024-03-27T21:40:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.578+0000 7f621ffff640 1 -- 172.21.15.60:0/774651551 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621802d820 con 0x7f622813b020 2024-03-27T21:40:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.578+0000 7f621ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.578+0000 7f621ffff640 1 -- 172.21.15.60:0/774651551 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621802dbb0 con 0x7f622813b020 2024-03-27T21:40:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.578+0000 7f622f38e640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.578+0000 7f622f38e640 4 set_mon_vals no callback set 2024-03-27T21:40:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.578+0000 7f621ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.578+0000 7f621ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.578+0000 7f621ffff640 10 monclient: dump: 2024-03-27T21:40:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:40.571 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.579+0000 7f622f38e640 10 monclient: shutdown 2024-03-27T21:40:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.579+0000 7f622f38e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.579+0000 7f622f38e640 1 -- 172.21.15.60:0/774651551 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f622813b020 msgr2=0x7f622813b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.579+0000 7f622f38e640 1 --2- 172.21.15.60:0/774651551 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f622813b020 0x7f622813b480 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6218009a00 tx=0x7f621802bb10 comp rx=0 tx=0).stop 2024-03-27T21:40:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.579+0000 7f622f38e640 1 -- 172.21.15.60:0/774651551 shutdown_connections 2024-03-27T21:40:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.579+0000 7f622f38e640 1 --2- 172.21.15.60:0/774651551 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f622813b020 0x7f622813b480 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.579+0000 7f622f38e640 1 --2- 172.21.15.60:0/774651551 >> v2:172.21.15.60:3300/0 conn(0x7f62280b40b0 0x7f62280a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.579+0000 7f622f38e640 1 -- 172.21.15.60:0/774651551 >> 172.21.15.60:0/774651551 conn(0x7f62280608e0 msgr2=0x7f6228060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.580+0000 7f622f38e640 1 -- 172.21.15.60:0/774651551 shutdown_connections 2024-03-27T21:40:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.580+0000 7f622f38e640 1 -- 172.21.15.60:0/774651551 wait complete. 2024-03-27T21:40:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.581+0000 7f622f38e640 10 monclient: build_initial_monmap 2024-03-27T21:40:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.581+0000 7f622f38e640 10 monclient: monmap: 2024-03-27T21:40:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:40.572 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.581+0000 7f622f38e640 1 librados: starting msgr at 2024-03-27T21:40:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.581+0000 7f622f38e640 1 librados: starting objecter 2024-03-27T21:40:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.581+0000 7f622f38e640 1 Processor -- start 2024-03-27T21:40:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.581+0000 7f622f38e640 1 -- start start 2024-03-27T21:40:40.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.581+0000 7f622f38e640 1 librados: setting wanted keys 2024-03-27T21:40:40.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.581+0000 7f622f38e640 1 librados: calling monclient init 2024-03-27T21:40:40.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.581+0000 7f622f38e640 10 monclient: init 2024-03-27T21:40:40.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.583+0000 7f622f38e640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:40.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.583+0000 7f622f38e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:40:40.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.583+0000 7f622f38e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62280b40b0 0x7f6228136c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:40.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.583+0000 7f622f38e640 10 monclient(hunting): picked mon.noname-b con 0x7f62280b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:40.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.583+0000 7f622f38e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f622813b020 0x7f622813ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:40.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.583+0000 7f622f38e640 10 monclient(hunting): picked mon.noname-c con 0x7f622813b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:40.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.583+0000 7f622d12a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62280b40b0 0x7f6228136c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.583+0000 7f622d12a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62280b40b0 0x7f6228136c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:54710/0 (socket says 172.21.15.60:54710) 2024-03-27T21:40:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.583+0000 7f622d12a640 1 -- 172.21.15.60:0/2204675202 learned_addr learned my addr 172.21.15.60:0/2204675202 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.583+0000 7f622c929640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f622813b020 0x7f622813ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.583+0000 7f622f38e640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62281416f0 0x7f622813daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622f38e640 10 monclient(hunting): picked mon.noname-a con 0x7f62281416f0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622f38e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622f38e640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622813bfa0 con 0x7f62281416f0 2024-03-27T21:40:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622d92b640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62281416f0 0x7f622813daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622f38e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622f38e640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622813a9c0 con 0x7f62280b40b0 2024-03-27T21:40:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622f38e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622f38e640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62281399f0 con 0x7f622813b020 2024-03-27T21:40:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622f38e640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622f38e640 10 monclient(hunting): authenticate will time out at 881.805507s 2024-03-27T21:40:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622d12a640 10 monclient(hunting): get_auth_request con 0x7f62280b40b0 auth_method 0 2024-03-27T21:40:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622d12a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622d12a640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622d12a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622c929640 10 monclient(hunting): get_auth_request con 0x7f622813b020 auth_method 0 2024-03-27T21:40:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622c929640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622c929640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622c929640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622d92b640 10 monclient(hunting): get_auth_request con 0x7f62281416f0 auth_method 0 2024-03-27T21:40:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622d92b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622d92b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.584+0000 7f622d92b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.585+0000 7f622d92b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.585+0000 7f622d92b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.585+0000 7f622d92b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.585+0000 7f622d92b640 10 monclient(hunting): handle_auth_done global_id 4719 payload 824 2024-03-27T21:40:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.585+0000 7f622d92b640 1 -- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f622813b020 msgr2=0x7f622813ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.585+0000 7f622d92b640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f622813b020 0x7f622813ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.585+0000 7f622d92b640 1 -- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62280b40b0 msgr2=0x7f6228136c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.585+0000 7f622d92b640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62280b40b0 0x7f6228136c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.585+0000 7f622d92b640 10 monclient: _finish_hunting 0 2024-03-27T21:40:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.585+0000 7f622d92b640 1 monclient: found mon.noname-a 2024-03-27T21:40:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.585+0000 7f622d92b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.585+0000 7f622d92b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.585+0000 7f622d92b640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6218009400 con 0x7f62281416f0 2024-03-27T21:40:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.586+0000 7f622c929640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f622813b020 0x7f622813ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:40:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.586+0000 7f622d12a640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62280b40b0 0x7f6228136c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:40:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.586+0000 7f622d92b640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62281416f0 0x7f622813daf0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f622000c700 tx=0x7f622000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.586+0000 7f621dffb640 10 client.?.objecter ms_handle_connect 0x7f62281416f0 2024-03-27T21:40:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.586+0000 7f621dffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.586+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6220003b70 con 0x7f62281416f0 2024-03-27T21:40:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.586+0000 7f621dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.586+0000 7f621dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.586+0000 7f621dffb640 10 monclient: dump: 2024-03-27T21:40:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:40.587 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:40.587 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:40.587 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:40.587 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:40.587 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:40.587 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:40.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.586+0000 7f621dffb640 10 monclient: _finish_auth 0 2024-03-27T21:40:40.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.586+0000 7f621dffb640 10 monclient: _check_auth_tickets 2024-03-27T21:40:40.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.586+0000 7f621dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:40.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.587+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6220003cd0 con 0x7f62281416f0 2024-03-27T21:40:40.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.587+0000 7f621dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:40.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.587+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6220005310 con 0x7f62281416f0 2024-03-27T21:40:40.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.587+0000 7f621dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:40.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.587+0000 7f621dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:40.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.587+0000 7f621dffb640 10 monclient: dump: 2024-03-27T21:40:40.588 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:40.588 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:40.588 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:40.588 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:40.588 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:40.588 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:40.588 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:40.588 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:40.588 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:40.589 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:40.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.587+0000 7f621f7fe640 4 set_mon_vals no callback set 2024-03-27T21:40:40.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.587+0000 7f622f38e640 5 monclient: authenticate success, global_id 4719 2024-03-27T21:40:40.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.587+0000 7f622f38e640 10 monclient: _renew_subs 2024-03-27T21:40:40.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.587+0000 7f622f38e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:40.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.587+0000 7f622f38e640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f622813a5c0 con 0x7f62281416f0 2024-03-27T21:40:40.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.587+0000 7f622f38e640 10 client.4719.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:40.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.587+0000 7f622f38e640 10 monclient: _renew_subs 2024-03-27T21:40:40.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.587+0000 7f622f38e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:40.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.587+0000 7f622f38e640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f622813e210 con 0x7f62281416f0 2024-03-27T21:40:40.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.587+0000 7f622f38e640 1 librados: init done 2024-03-27T21:40:40.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.589+0000 7f620b7fe640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:40.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.589+0000 7f620b7fe640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:40.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.589+0000 7f620b7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:40.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.589+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6228069470 con 0x7f62281416f0 2024-03-27T21:40:40.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.589+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6220005470 con 0x7f62281416f0 2024-03-27T21:40:40.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.590+0000 7f621dffb640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f620c06c590 0x7f620c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:40.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.590+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f6220088df0 con 0x7f62281416f0 2024-03-27T21:40:40.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.590+0000 7f621dffb640 10 client.4719.objecter ms_dispatch 0x7f6228000b90 osd_map(41..41 src has 1..41) v4 2024-03-27T21:40:40.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.590+0000 7f621dffb640 3 client.4719.objecter handle_osd_map got epochs [41,41] > 0 2024-03-27T21:40:40.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.590+0000 7f621dffb640 3 client.4719.objecter handle_osd_map decoding full epoch 41 2024-03-27T21:40:40.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.590+0000 7f621dffb640 20 client.4719.objecter dump_active .. 0 homeless 2024-03-27T21:40:40.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.590+0000 7f622d12a640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f620c06c590 0x7f620c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:40.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.590+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v41) v1 ==== 54+0+16 (secure 0 0 0) 0x7f6220054230 con 0x7f62281416f0 2024-03-27T21:40:40.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.590+0000 7f621dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:40.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.590+0000 7f621dffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:40.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.590+0000 7f622d12a640 10 monclient: get_auth_request con 0x7f620c06c590 auth_method 0 2024-03-27T21:40:40.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.591+0000 7f621dffb640 10 client.4719.objecter ms_handle_connect 0x7f620c06c590 2024-03-27T21:40:40.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.591+0000 7f622d12a640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f620c06c590 0x7f620c06ea50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f62240094a0 tx=0x7f6224008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:40.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.593+0000 7f620b7fe640 10 client.4719.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:40.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.593+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign -1 1 2024-03-27T21:40:40.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.593+0000 7f620b7fe640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f6228138f90 0x7f6228130d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:40.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.593+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f62280a90d0 osd=0 3 2024-03-27T21:40:40.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.593+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62280a90d0 osd=0 3 2024-03-27T21:40:40.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.593+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f62280a90d0 osd=0 3 2024-03-27T21:40:40.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.593+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_remove -1 1 2024-03-27T21:40:40.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.593+0000 7f622c929640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f6228138f90 0x7f6228130d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:40.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.593+0000 7f622c929640 10 monclient: get_auth_request con 0x7f6228138f90 auth_method 0 2024-03-27T21:40:40.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.593+0000 7f620b7fe640 20 client.4719.objecter get_session s=0x7f62280a90d0 osd=0 3 2024-03-27T21:40:40.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.593+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign 0 1 2024-03-27T21:40:40.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.593+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62280a90d0 osd=0 4 2024-03-27T21:40:40.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.593+0000 7f620b7fe640 10 client.4719.objecter _send_command 1 2024-03-27T21:40:40.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.593+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f622813a7b0 con 0x7f6228138f90 2024-03-27T21:40:40.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.593+0000 7f5ffabba640 10 monclient: get_monmap_and_config 2024-03-27T21:40:40.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.593+0000 7f5ffabba640 10 monclient: build_initial_monmap 2024-03-27T21:40:40.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.594+0000 7f5ffabba640 10 monclient: monmap: 2024-03-27T21:40:40.592 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:40.592 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:40.592 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:40.595020+0000 2024-03-27T21:40:40.592 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:40.595020+0000 2024-03-27T21:40:40.592 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:40.592 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:40.592 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:40.593 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:40.593 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:40.593 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:40.593 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:40.593 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:40.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.594+0000 7f622c929640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f6228138f90 0x7f6228130d50 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:40.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.594+0000 7f621dffb640 10 client.4719.objecter ms_handle_connect 0x7f6228138f90 2024-03-27T21:40:40.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.594+0000 7f5ffabba640 1 Processor -- start 2024-03-27T21:40:40.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.595+0000 7f5ffabba640 1 -- start start 2024-03-27T21:40:40.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.595+0000 7f5ffabba640 10 monclient: init 2024-03-27T21:40:40.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.595+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f621802d6a0 con 0x7f6228138f90 2024-03-27T21:40:40.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.595+0000 7f621dffb640 10 client.4719.objecter ms_dispatch 0x7f6228000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:40.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.595+0000 7f621dffb640 10 client.4719.objecter _finish_command 1 = system:0 2024-03-27T21:40:40.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.595+0000 7f621dffb640 20 client.4719.objecter put_session s=0x7f62280a90d0 osd=0 4 2024-03-27T21:40:40.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.595+0000 7f621dffb640 15 client.4719.objecter _session_command_op_remove 0 1 2024-03-27T21:40:40.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.596+0000 7f5ffabba640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:40.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.596+0000 7f5ffabba640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-03-27T21:40:40.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.596+0000 7f5ffabba640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5fec037540 0x7f5fec0c4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:40.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.596+0000 7f5ffabba640 10 monclient(hunting): picked mon.noname-b con 0x7f5fec037540 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:40.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.596+0000 7f5ffabba640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5fec02b4c0 0x7f5fec0c5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:40.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.596+0000 7f5ffabba640 10 monclient(hunting): picked mon.noname-c con 0x7f5fec02b4c0 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:40.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.596+0000 7f5ffabba640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5fec0cb300 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:40.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.596+0000 7f5ffabba640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fec0c5570 con 0x7f5fec0cb300 2024-03-27T21:40:40.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.596+0000 7f5ffabba640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:40.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.596+0000 7f5ffabba640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fec02b240 con 0x7f5fec02b4c0 2024-03-27T21:40:40.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.596+0000 7f5ffabba640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.596+0000 7f5ffabba640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec037920 con 0x7f5fec037540 2024-03-27T21:40:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.596+0000 7f5ffabba640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.596+0000 7f5ffabba640 10 monclient(hunting): authenticate will time out at 881.817148s 2024-03-27T21:40:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.596+0000 7f5ffa3b9640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f5fec0cb300 0x7f5fec0cb790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.60:50446/0 (socket says 172.21.15.60:50446) 2024-03-27T21:40:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.596+0000 7f5ffa3b9640 1 -- 172.21.15.60:0/3773637299 learned_addr learned my addr 172.21.15.60:0/3773637299 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:40:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff9bb8640 1 --2- 172.21.15.60:0/3773637299 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5fec037540 0x7f5fec0c4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff9bb8640 10 monclient(hunting): get_auth_request con 0x7f5fec037540 auth_method 0 2024-03-27T21:40:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff9bb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff9bb8640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff9bb8640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff93b7640 1 --2- 172.21.15.60:0/3773637299 >> v2:172.21.15.191:3300/0 conn(0x7f5fec02b4c0 0x7f5fec0c5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff9bb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff9bb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff9bb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff93b7640 10 monclient(hunting): get_auth_request con 0x7f5fec02b4c0 auth_method 0 2024-03-27T21:40:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff93b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff93b7640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff93b7640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff9bb8640 10 monclient(hunting): handle_auth_done global_id 4574 payload 274 2024-03-27T21:40:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff9bb8640 1 -- 172.21.15.60:0/3773637299 >> v2:172.21.15.191:3300/0 conn(0x7f5fec02b4c0 msgr2=0x7f5fec0c5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff9bb8640 1 --2- 172.21.15.60:0/3773637299 >> v2:172.21.15.191:3300/0 conn(0x7f5fec02b4c0 0x7f5fec0c5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff9bb8640 1 -- 172.21.15.60:0/3773637299 >> v1:172.21.15.191:6789/0 conn(0x7f5fec0cb300 legacy=0x7f5fec0cb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff9bb8640 10 monclient: _finish_hunting 0 2024-03-27T21:40:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff9bb8640 1 monclient: found mon.noname-b 2024-03-27T21:40:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff9bb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff9bb8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.597+0000 7f5ff9bb8640 1 -- 172.21.15.60:0/3773637299 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec0308b0 con 0x7f5fec037540 2024-03-27T21:40:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.599+0000 7f5ff9bb8640 1 --2- 172.21.15.60:0/3773637299 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5fec037540 0x7f5fec0c4af0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00097a0 tx=0x7f5ff002b640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=629a5a667ce24284 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.599+0000 7f5ff8bb6640 1 -- 172.21.15.60:0/3773637299 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff0030710 con 0x7f5fec037540 2024-03-27T21:40:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.599+0000 7f5ff8bb6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.599+0000 7f5ff8bb6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.599+0000 7f5ff8bb6640 10 monclient: dump: 2024-03-27T21:40:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:40.610 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.599+0000 7f5ff8bb6640 10 monclient: _finish_auth 0 2024-03-27T21:40:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.599+0000 7f5ff8bb6640 10 monclient: _check_auth_tickets 2024-03-27T21:40:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.599+0000 7f5ff8bb6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.599+0000 7f5ff8bb6640 1 -- 172.21.15.60:0/3773637299 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff002e050 con 0x7f5fec037540 2024-03-27T21:40:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.599+0000 7f5ff8bb6640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.599+0000 7f5ff8bb6640 1 -- 172.21.15.60:0/3773637299 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff0030a90 con 0x7f5fec037540 2024-03-27T21:40:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.599+0000 7f5ff8bb6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.599+0000 7f5ff8bb6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.599+0000 7f5ff8bb6640 10 monclient: dump: 2024-03-27T21:40:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:40.611 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.599+0000 7f5ffabba640 5 monclient: authenticate success, global_id 4574 2024-03-27T21:40:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.599+0000 7f5ffabba640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:40.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.599+0000 7f5ffabba640 4 set_mon_vals no callback set 2024-03-27T21:40:40.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.600+0000 7f5ffabba640 10 monclient: shutdown 2024-03-27T21:40:40.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.600+0000 7f5ffabba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:40.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.600+0000 7f5ffabba640 1 -- 172.21.15.60:0/3773637299 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5fec037540 msgr2=0x7f5fec0c4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:40.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.600+0000 7f5ffabba640 1 --2- 172.21.15.60:0/3773637299 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5fec037540 0x7f5fec0c4af0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00097a0 tx=0x7f5ff002b640 comp rx=0 tx=0).stop 2024-03-27T21:40:40.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.600+0000 7f5ffabba640 1 -- 172.21.15.60:0/3773637299 shutdown_connections 2024-03-27T21:40:40.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.601+0000 7f5ffabba640 1 --2- 172.21.15.60:0/3773637299 >> v2:172.21.15.191:3300/0 conn(0x7f5fec02b4c0 0x7f5fec0c5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:40.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.601+0000 7f5ffabba640 1 --2- 172.21.15.60:0/3773637299 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5fec037540 0x7f5fec0c4af0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:40.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.601+0000 7f5ffabba640 1 -- 172.21.15.60:0/3773637299 >> 172.21.15.60:0/3773637299 conn(0x7f5fec03a8b0 msgr2=0x7f5fec036b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:40.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.601+0000 7f5ffabba640 1 -- 172.21.15.60:0/3773637299 shutdown_connections 2024-03-27T21:40:40.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.601+0000 7f5ffabba640 1 -- 172.21.15.60:0/3773637299 wait complete. 2024-03-27T21:40:40.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.602+0000 7f5ffabba640 10 monclient: build_initial_monmap 2024-03-27T21:40:40.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.602+0000 7f5ffabba640 10 monclient: monmap: 2024-03-27T21:40:40.612 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:40.613 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.602+0000 7f5ffabba640 1 librados: starting msgr at 2024-03-27T21:40:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.602+0000 7f5ffabba640 1 librados: starting objecter 2024-03-27T21:40:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.602+0000 7f5ffabba640 1 Processor -- start 2024-03-27T21:40:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.602+0000 7f5ffabba640 1 -- start start 2024-03-27T21:40:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.602+0000 7f5ffabba640 1 librados: setting wanted keys 2024-03-27T21:40:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.602+0000 7f5ffabba640 1 librados: calling monclient init 2024-03-27T21:40:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.602+0000 7f5ffabba640 10 monclient: init 2024-03-27T21:40:40.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.603+0000 7f5ffabba640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:40.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.603+0000 7f5ffabba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:40:40.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.603+0000 7f5ffabba640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5fec02b4c0 0x7f5fec0c07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:40.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.603+0000 7f5ffabba640 10 monclient(hunting): picked mon.noname-c con 0x7f5fec02b4c0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:40.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.603+0000 7f5ff9bb8640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5fec02b4c0 0x7f5fec0c07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:40.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.603+0000 7f5ff9bb8640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5fec02b4c0 0x7f5fec0c07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:59528/0 (socket says 172.21.15.60:59528) 2024-03-27T21:40:40.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.603+0000 7f5ff9bb8640 1 -- 172.21.15.60:0/116778768 learned_addr learned my addr 172.21.15.60:0/116778768 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:40.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ffabba640 1 --2- 172.21.15.60:0/116778768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fec037540 0x7f5fec0b7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:40.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ffabba640 10 monclient(hunting): picked mon.noname-a con 0x7f5fec037540 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:40.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ffabba640 1 --2- 172.21.15.60:0/116778768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5fec0cb300 0x7f5fec0b7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:40.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ffabba640 10 monclient(hunting): picked mon.noname-b con 0x7f5fec0cb300 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:40.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ffabba640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:40.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ffabba640 1 -- 172.21.15.60:0/116778768 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec0bfdb0 con 0x7f5fec037540 2024-03-27T21:40:40.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ffabba640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:40.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ffabba640 1 -- 172.21.15.60:0/116778768 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec0bee90 con 0x7f5fec0cb300 2024-03-27T21:40:40.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ffabba640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:40.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ffabba640 1 -- 172.21.15.60:0/116778768 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec0b8e10 con 0x7f5fec02b4c0 2024-03-27T21:40:40.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ffabba640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:40.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ffabba640 10 monclient(hunting): authenticate will time out at 881.825174s 2024-03-27T21:40:40.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff9bb8640 10 monclient(hunting): get_auth_request con 0x7f5fec02b4c0 auth_method 0 2024-03-27T21:40:40.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff9bb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:40.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff9bb8640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:40.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff9bb8640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:40.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff93b7640 1 --2- 172.21.15.60:0/116778768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fec037540 0x7f5fec0b7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:40.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff93b7640 10 monclient(hunting): get_auth_request con 0x7f5fec037540 auth_method 0 2024-03-27T21:40:40.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff93b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:40.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff93b7640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:40.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff93b7640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:40.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff9bb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:40.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff9bb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:40.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff9bb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:40.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff93b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:40.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff93b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:40.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff93b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:40.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ffa3b9640 1 --2- 172.21.15.60:0/116778768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5fec0cb300 0x7f5fec0b7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:40.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ffa3b9640 10 monclient(hunting): get_auth_request con 0x7f5fec0cb300 auth_method 0 2024-03-27T21:40:40.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ffa3b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:40.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ffa3b9640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:40.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ffa3b9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:40.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff9bb8640 10 monclient(hunting): handle_auth_done global_id 4577 payload 824 2024-03-27T21:40:40.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff9bb8640 1 -- 172.21.15.60:0/116778768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5fec0cb300 msgr2=0x7f5fec0b7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:40.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff9bb8640 1 --2- 172.21.15.60:0/116778768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5fec0cb300 0x7f5fec0b7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:40.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff9bb8640 1 -- 172.21.15.60:0/116778768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fec037540 msgr2=0x7f5fec0b7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:40.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff9bb8640 1 --2- 172.21.15.60:0/116778768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fec037540 0x7f5fec0b7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:40.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff9bb8640 10 monclient: _finish_hunting 0 2024-03-27T21:40:40.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff9bb8640 1 monclient: found mon.noname-c 2024-03-27T21:40:40.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff9bb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:40.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff9bb8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:40:40.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff9bb8640 1 -- 172.21.15.60:0/116778768 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff0009400 con 0x7f5fec02b4c0 2024-03-27T21:40:40.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.604+0000 7f5ff93b7640 1 --2- 172.21.15.60:0/116778768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fec037540 0x7f5fec0b7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:40.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.605+0000 7f5ff9bb8640 1 --2- 172.21.15.60:0/116778768 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5fec02b4c0 0x7f5fec0c07b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0035040 tx=0x7f5ff00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:40.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.605+0000 7f5fea7fc640 10 client.?.objecter ms_handle_connect 0x7f5fec02b4c0 2024-03-27T21:40:40.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.605+0000 7f5fea7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:40.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.605+0000 7f5fea7fc640 1 -- 172.21.15.60:0/116778768 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff004b430 con 0x7f5fec02b4c0 2024-03-27T21:40:40.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.605+0000 7f5fea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:40.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.605+0000 7f5fea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:40.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.605+0000 7f5fea7fc640 10 monclient: dump: 2024-03-27T21:40:40.618 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:40.618 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:40.618 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:40.618 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:40.618 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:40.618 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:40.618 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:40.618 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:40.618 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:40.618 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:40.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.605+0000 7f5fea7fc640 10 monclient: _finish_auth 0 2024-03-27T21:40:40.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.605+0000 7f5fea7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:40:40.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.605+0000 7f5fea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:40.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.605+0000 7f5fea7fc640 1 -- 172.21.15.60:0/116778768 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff0045070 con 0x7f5fec02b4c0 2024-03-27T21:40:40.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.605+0000 7f5fea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:40.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.605+0000 7f5fea7fc640 1 -- 172.21.15.60:0/116778768 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff004b7f0 con 0x7f5fec02b4c0 2024-03-27T21:40:40.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.605+0000 7f5fea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:40.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.605+0000 7f5fea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:40.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.605+0000 7f5fea7fc640 10 monclient: dump: 2024-03-27T21:40:40.619 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:40.619 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:40.619 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:40.619 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:40.619 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:40.619 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:40.619 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:40.619 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:40.620 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:40.620 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:40.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.605+0000 7f5febfff640 4 set_mon_vals no callback set 2024-03-27T21:40:40.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.606+0000 7f5ffabba640 5 monclient: authenticate success, global_id 4577 2024-03-27T21:40:40.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.606+0000 7f5ffabba640 10 monclient: _renew_subs 2024-03-27T21:40:40.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.606+0000 7f5ffabba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:40.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.606+0000 7f5ffabba640 1 -- 172.21.15.60:0/116778768 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fec0b7ec0 con 0x7f5fec02b4c0 2024-03-27T21:40:40.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.606+0000 7f5ffabba640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:40.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.606+0000 7f5ffabba640 10 monclient: _renew_subs 2024-03-27T21:40:40.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.606+0000 7f5ffabba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:40.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.606+0000 7f5ffabba640 1 -- 172.21.15.60:0/116778768 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fec0c0cf0 con 0x7f5fec02b4c0 2024-03-27T21:40:40.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.606+0000 7f5ffabba640 1 librados: init done 2024-03-27T21:40:40.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.607+0000 7f5ffabba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:40.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.608+0000 7f5ffabba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:40.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.608+0000 7f5ffabba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:40.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.608+0000 7f5ffabba640 1 -- 172.21.15.60:0/116778768 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fec0c0ff0 con 0x7f5fec02b4c0 2024-03-27T21:40:40.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.608+0000 7f5fea7fc640 1 -- 172.21.15.60:0/116778768 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5ff0039040 con 0x7f5fec02b4c0 2024-03-27T21:40:40.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.609+0000 7f5fea7fc640 1 --2- 172.21.15.60:0/116778768 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5fd806c590 0x7f5fd806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:40.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.609+0000 7f5fea7fc640 1 -- 172.21.15.60:0/116778768 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f5ff0047070 con 0x7f5fec02b4c0 2024-03-27T21:40:40.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.609+0000 7f5fea7fc640 10 client.4577.objecter ms_dispatch 0x7f5ff4000b90 osd_map(41..41 src has 1..41) v4 2024-03-27T21:40:40.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.609+0000 7f5fea7fc640 3 client.4577.objecter handle_osd_map got epochs [41,41] > 0 2024-03-27T21:40:40.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.609+0000 7f5fea7fc640 3 client.4577.objecter handle_osd_map decoding full epoch 41 2024-03-27T21:40:40.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.609+0000 7f5fea7fc640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-27T21:40:40.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.610+0000 7f5ff93b7640 1 --2- 172.21.15.60:0/116778768 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5fd806c590 0x7f5fd806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:40.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.610+0000 7f5ff93b7640 10 monclient: get_auth_request con 0x7f5fd806c590 auth_method 0 2024-03-27T21:40:40.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.610+0000 7f5ff93b7640 1 --2- 172.21.15.60:0/116778768 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5fd806c590 0x7f5fd806ea50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5fec0cab30 tx=0x7f5fe4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:40.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.611+0000 7f5fea7fc640 10 client.4577.objecter ms_handle_connect 0x7f5fd806c590 2024-03-27T21:40:40.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.618+0000 7f5fea7fc640 1 -- 172.21.15.60:0/116778768 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5fec0c0ff0 con 0x7f5fec02b4c0 2024-03-27T21:40:40.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.618+0000 7f5fea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:40.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.618+0000 7f5fea7fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:40.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.624+0000 7f620b7fe640 10 client.4719.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:40.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.625+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign -1 2 2024-03-27T21:40:40.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.625+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f62280a90d0 osd=0 3 2024-03-27T21:40:40.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.625+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62280a90d0 osd=0 3 2024-03-27T21:40:40.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.625+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f62280a90d0 osd=0 3 2024-03-27T21:40:40.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.625+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_remove -1 2 2024-03-27T21:40:40.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.625+0000 7f620b7fe640 20 client.4719.objecter get_session s=0x7f62280a90d0 osd=0 3 2024-03-27T21:40:40.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.625+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign 0 2 2024-03-27T21:40:40.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.625+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62280a90d0 osd=0 4 2024-03-27T21:40:40.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.625+0000 7f620b7fe640 10 client.4719.objecter _send_command 2 2024-03-27T21:40:40.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.625+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f622813ab00 con 0x7f6228138f90 2024-03-27T21:40:40.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.627+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f622813ab00 con 0x7f6228138f90 2024-03-27T21:40:40.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.627+0000 7f621dffb640 10 client.4719.objecter ms_dispatch 0x7f6228000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:40.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.627+0000 7f621dffb640 10 client.4719.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:40.624 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:40.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.627+0000 7f621dffb640 20 client.4719.objecter put_session s=0x7f62280a90d0 osd=0 4 2024-03-27T21:40:40.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.627+0000 7f621dffb640 15 client.4719.objecter _session_command_op_remove 0 2 2024-03-27T21:40:40.625 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:40.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.627+0000 7f620b7fe640 10 client.4719.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:40.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.627+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign -1 3 2024-03-27T21:40:40.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.627+0000 7f620b7fe640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f62281317a0 0x7f62280b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:40.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.627+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f62281395b0 osd=1 3 2024-03-27T21:40:40.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.627+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62281395b0 osd=1 3 2024-03-27T21:40:40.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.627+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f62281395b0 osd=1 3 2024-03-27T21:40:40.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.627+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_remove -1 3 2024-03-27T21:40:40.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.627+0000 7f620b7fe640 20 client.4719.objecter get_session s=0x7f62281395b0 osd=1 3 2024-03-27T21:40:40.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.627+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign 1 3 2024-03-27T21:40:40.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.627+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62281395b0 osd=1 4 2024-03-27T21:40:40.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.627+0000 7f620b7fe640 10 client.4719.objecter _send_command 3 2024-03-27T21:40:40.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.627+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6228131b80 con 0x7f62281317a0 2024-03-27T21:40:40.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.628+0000 7f622d12a640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f62281317a0 0x7f62280b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:40.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.628+0000 7f622d12a640 10 monclient: get_auth_request con 0x7f62281317a0 auth_method 0 2024-03-27T21:40:40.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.628+0000 7f622d12a640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f62281317a0 0x7f62280b4bc0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:40.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.628+0000 7f621dffb640 10 client.4719.objecter ms_handle_connect 0x7f62281317a0 2024-03-27T21:40:40.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.629+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6228131b80 con 0x7f62281317a0 2024-03-27T21:40:40.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.629+0000 7f621dffb640 10 client.4719.objecter ms_dispatch 0x7f6228000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:40.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.629+0000 7f621dffb640 10 client.4719.objecter _finish_command 3 = system:0 2024-03-27T21:40:40.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.629+0000 7f621dffb640 20 client.4719.objecter put_session s=0x7f62281395b0 osd=1 4 2024-03-27T21:40:40.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.629+0000 7f621dffb640 15 client.4719.objecter _session_command_op_remove 1 3 2024-03-27T21:40:40.643 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:40.657+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:40.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.657+0000 7f620b7fe640 10 client.4719.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:40.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.657+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign -1 4 2024-03-27T21:40:40.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.657+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f62281395b0 osd=1 3 2024-03-27T21:40:40.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.657+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62281395b0 osd=1 3 2024-03-27T21:40:40.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.657+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f62281395b0 osd=1 3 2024-03-27T21:40:40.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.657+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_remove -1 4 2024-03-27T21:40:40.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.657+0000 7f620b7fe640 20 client.4719.objecter get_session s=0x7f62281395b0 osd=1 3 2024-03-27T21:40:40.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.657+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign 1 4 2024-03-27T21:40:40.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.657+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62281395b0 osd=1 4 2024-03-27T21:40:40.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.657+0000 7f620b7fe640 10 client.4719.objecter _send_command 4 2024-03-27T21:40:40.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.658+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f62280b58c0 con 0x7f62281317a0 2024-03-27T21:40:40.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.660+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f62280b58c0 con 0x7f62281317a0 2024-03-27T21:40:40.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.660+0000 7f621dffb640 10 client.4719.objecter ms_dispatch 0x7f6228000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:40.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.660+0000 7f621dffb640 10 client.4719.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:40.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.660+0000 7f621dffb640 20 client.4719.objecter put_session s=0x7f62281395b0 osd=1 4 2024-03-27T21:40:40.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.660+0000 7f621dffb640 15 client.4719.objecter _session_command_op_remove 1 4 2024-03-27T21:40:40.647 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:40.647 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:40.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.660+0000 7f620b7fe640 10 client.4719.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:40.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.660+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign -1 5 2024-03-27T21:40:40.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.660+0000 7f620b7fe640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f6228138b80 0x7f62281e7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:40.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.660+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f6228138a10 osd=2 3 2024-03-27T21:40:40.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.660+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f6228138a10 osd=2 3 2024-03-27T21:40:40.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.660+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f6228138a10 osd=2 3 2024-03-27T21:40:40.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.660+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_remove -1 5 2024-03-27T21:40:40.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.660+0000 7f620b7fe640 20 client.4719.objecter get_session s=0x7f6228138a10 osd=2 3 2024-03-27T21:40:40.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.660+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign 2 5 2024-03-27T21:40:40.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.660+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f6228138a10 osd=2 4 2024-03-27T21:40:40.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.660+0000 7f620b7fe640 10 client.4719.objecter _send_command 5 2024-03-27T21:40:40.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.660+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62281ea010 con 0x7f6228138b80 2024-03-27T21:40:40.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.660+0000 7f622c929640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f6228138b80 0x7f62281e7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:40.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.660+0000 7f622c929640 10 monclient: get_auth_request con 0x7f6228138b80 auth_method 0 2024-03-27T21:40:40.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.661+0000 7f622c929640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f6228138b80 0x7f62281e7240 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:40.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.661+0000 7f621dffb640 10 client.4719.objecter ms_handle_connect 0x7f6228138b80 2024-03-27T21:40:40.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.662+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== osd.2 v2:172.21.15.60:6824/44654 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f62281ea010 con 0x7f6228138b80 2024-03-27T21:40:40.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.662+0000 7f621dffb640 10 client.4719.objecter ms_dispatch 0x7f6228000b90 command_reply(tid 5: 0 ) v1 2024-03-27T21:40:40.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.662+0000 7f621dffb640 10 client.4719.objecter _finish_command 5 = system:0 2024-03-27T21:40:40.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.662+0000 7f621dffb640 20 client.4719.objecter put_session s=0x7f6228138a10 osd=2 4 2024-03-27T21:40:40.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.662+0000 7f621dffb640 15 client.4719.objecter _session_command_op_remove 2 5 2024-03-27T21:40:40.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.688+0000 7f620b7fe640 10 client.4719.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:40.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.688+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign -1 6 2024-03-27T21:40:40.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.688+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f6228138a10 osd=2 3 2024-03-27T21:40:40.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.688+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f6228138a10 osd=2 3 2024-03-27T21:40:40.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.688+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f6228138a10 osd=2 3 2024-03-27T21:40:40.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.688+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_remove -1 6 2024-03-27T21:40:40.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.688+0000 7f620b7fe640 20 client.4719.objecter get_session s=0x7f6228138a10 osd=2 3 2024-03-27T21:40:40.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.688+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign 2 6 2024-03-27T21:40:40.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.688+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f6228138a10 osd=2 4 2024-03-27T21:40:40.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.688+0000 7f620b7fe640 10 client.4719.objecter _send_command 6 2024-03-27T21:40:40.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.688+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f62280b5340 con 0x7f6228138b80 2024-03-27T21:40:40.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.691+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== osd.2 v2:172.21.15.60:6824/44654 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f62280b5340 con 0x7f6228138b80 2024-03-27T21:40:40.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.691+0000 7f621dffb640 10 client.4719.objecter ms_dispatch 0x7f6228000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:40.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.691+0000 7f621dffb640 10 client.4719.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:40.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.691+0000 7f621dffb640 20 client.4719.objecter put_session s=0x7f6228138a10 osd=2 4 2024-03-27T21:40:40.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.691+0000 7f621dffb640 15 client.4719.objecter _session_command_op_remove 2 6 2024-03-27T21:40:40.678 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2: {} 2024-03-27T21:40:40.678 INFO:teuthology.orchestra.run.smithi060.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-27T21:40:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.691+0000 7f620b7fe640 10 client.4719.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.691+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign -1 7 2024-03-27T21:40:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.691+0000 7f620b7fe640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f6228066760 0x7f62281e8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.691+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f62280b5750 osd=3 3 2024-03-27T21:40:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.691+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62280b5750 osd=3 3 2024-03-27T21:40:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.691+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f62280b5750 osd=3 3 2024-03-27T21:40:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.691+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_remove -1 7 2024-03-27T21:40:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.691+0000 7f620b7fe640 20 client.4719.objecter get_session s=0x7f62280b5750 osd=3 3 2024-03-27T21:40:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.691+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign 3 7 2024-03-27T21:40:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.691+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62280b5750 osd=3 4 2024-03-27T21:40:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.691+0000 7f620b7fe640 10 client.4719.objecter _send_command 7 2024-03-27T21:40:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.691+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62281ed8a0 con 0x7f6228066760 2024-03-27T21:40:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.691+0000 7f622d92b640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f6228066760 0x7f62281e8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.691+0000 7f622d92b640 10 monclient: get_auth_request con 0x7f6228066760 auth_method 0 2024-03-27T21:40:40.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.692+0000 7f621dffb640 10 client.4719.objecter ms_handle_connect 0x7f6228066760 2024-03-27T21:40:40.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.692+0000 7f622d92b640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f6228066760 0x7f62281e8900 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:40.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.693+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f62281ed8a0 con 0x7f6228066760 2024-03-27T21:40:40.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.693+0000 7f621dffb640 10 client.4719.objecter ms_dispatch 0x7f6228000b90 command_reply(tid 7: 0 ) v1 2024-03-27T21:40:40.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.693+0000 7f621dffb640 10 client.4719.objecter _finish_command 7 = system:0 2024-03-27T21:40:40.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.693+0000 7f621dffb640 20 client.4719.objecter put_session s=0x7f62280b5750 osd=3 4 2024-03-27T21:40:40.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.693+0000 7f621dffb640 15 client.4719.objecter _session_command_op_remove 3 7 2024-03-27T21:40:40.706 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:40.706 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:40.706 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:40.707 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:40:40.707 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:40.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.722+0000 7f620b7fe640 10 client.4719.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:40.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.722+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign -1 8 2024-03-27T21:40:40.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.722+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f62280b5750 osd=3 3 2024-03-27T21:40:40.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.722+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62280b5750 osd=3 3 2024-03-27T21:40:40.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.722+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f62280b5750 osd=3 3 2024-03-27T21:40:40.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.722+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_remove -1 8 2024-03-27T21:40:40.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.722+0000 7f620b7fe640 20 client.4719.objecter get_session s=0x7f62280b5750 osd=3 3 2024-03-27T21:40:40.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.722+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign 3 8 2024-03-27T21:40:40.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.722+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62280b5750 osd=3 4 2024-03-27T21:40:40.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.722+0000 7f620b7fe640 10 client.4719.objecter _send_command 8 2024-03-27T21:40:40.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.722+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6228065cd0 con 0x7f6228066760 2024-03-27T21:40:40.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.724+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6228065cd0 con 0x7f6228066760 2024-03-27T21:40:40.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.724+0000 7f621dffb640 10 client.4719.objecter ms_dispatch 0x7f6228000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:40.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.724+0000 7f621dffb640 10 client.4719.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:40.711 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:40.712 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:40.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.724+0000 7f621dffb640 20 client.4719.objecter put_session s=0x7f62280b5750 osd=3 4 2024-03-27T21:40:40.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.724+0000 7f621dffb640 15 client.4719.objecter _session_command_op_remove 3 8 2024-03-27T21:40:40.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.725+0000 7f620b7fe640 10 client.4719.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:40.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.725+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign -1 9 2024-03-27T21:40:40.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.725+0000 7f620b7fe640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f62281e8450 0x7f62281ec190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:40.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.725+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f6228065e40 osd=4 3 2024-03-27T21:40:40.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.725+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f6228065e40 osd=4 3 2024-03-27T21:40:40.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.725+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f6228065e40 osd=4 3 2024-03-27T21:40:40.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.725+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_remove -1 9 2024-03-27T21:40:40.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.725+0000 7f620b7fe640 20 client.4719.objecter get_session s=0x7f6228065e40 osd=4 3 2024-03-27T21:40:40.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.725+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign 4 9 2024-03-27T21:40:40.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.725+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f6228065e40 osd=4 4 2024-03-27T21:40:40.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.725+0000 7f620b7fe640 10 client.4719.objecter _send_command 9 2024-03-27T21:40:40.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.725+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6228065fb0 con 0x7f62281e8450 2024-03-27T21:40:40.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.726+0000 7f622d12a640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f62281e8450 0x7f62281ec190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:40.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.726+0000 7f622d12a640 10 monclient: get_auth_request con 0x7f62281e8450 auth_method 0 2024-03-27T21:40:40.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.726+0000 7f622d12a640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f62281e8450 0x7f62281ec190 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:40.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.726+0000 7f621dffb640 10 client.4719.objecter ms_handle_connect 0x7f62281e8450 2024-03-27T21:40:40.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.727+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6228065fb0 con 0x7f62281e8450 2024-03-27T21:40:40.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.727+0000 7f621dffb640 10 client.4719.objecter ms_dispatch 0x7f6228000b90 command_reply(tid 9: 0 ) v1 2024-03-27T21:40:40.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.727+0000 7f621dffb640 10 client.4719.objecter _finish_command 9 = system:0 2024-03-27T21:40:40.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.727+0000 7f621dffb640 20 client.4719.objecter put_session s=0x7f6228065e40 osd=4 4 2024-03-27T21:40:40.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.727+0000 7f621dffb640 15 client.4719.objecter _session_command_op_remove 4 9 2024-03-27T21:40:40.719 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:40:40.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.755+0000 7f620b7fe640 10 client.4719.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:40.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.755+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign -1 10 2024-03-27T21:40:40.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.755+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f6228065e40 osd=4 3 2024-03-27T21:40:40.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.755+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f6228065e40 osd=4 3 2024-03-27T21:40:40.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.755+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f6228065e40 osd=4 3 2024-03-27T21:40:40.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.755+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_remove -1 10 2024-03-27T21:40:40.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.755+0000 7f620b7fe640 20 client.4719.objecter get_session s=0x7f6228065e40 osd=4 3 2024-03-27T21:40:40.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.755+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign 4 10 2024-03-27T21:40:40.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.755+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f6228065e40 osd=4 4 2024-03-27T21:40:40.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.755+0000 7f620b7fe640 10 client.4719.objecter _send_command 10 2024-03-27T21:40:40.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.755+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6228065fb0 con 0x7f62281e8450 2024-03-27T21:40:40.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.757+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6228065fb0 con 0x7f62281e8450 2024-03-27T21:40:40.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.757+0000 7f621dffb640 10 client.4719.objecter ms_dispatch 0x7f6228000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:40.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.757+0000 7f621dffb640 10 client.4719.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:40.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.757+0000 7f621dffb640 20 client.4719.objecter put_session s=0x7f6228065e40 osd=4 4 2024-03-27T21:40:40.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.757+0000 7f621dffb640 15 client.4719.objecter _session_command_op_remove 4 10 2024-03-27T21:40:40.744 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:40.745 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:40.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.757+0000 7f620b7fe640 10 client.4719.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:40.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.757+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign -1 11 2024-03-27T21:40:40.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.757+0000 7f620b7fe640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f62281ecbe0 0x7f62281ed020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:40.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.758+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f6228065fb0 osd=5 3 2024-03-27T21:40:40.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.758+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f6228065fb0 osd=5 3 2024-03-27T21:40:40.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.758+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f6228065fb0 osd=5 3 2024-03-27T21:40:40.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.758+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_remove -1 11 2024-03-27T21:40:40.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.758+0000 7f620b7fe640 20 client.4719.objecter get_session s=0x7f6228065fb0 osd=5 3 2024-03-27T21:40:40.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.758+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign 5 11 2024-03-27T21:40:40.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.758+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f6228065fb0 osd=5 4 2024-03-27T21:40:40.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.758+0000 7f620b7fe640 10 client.4719.objecter _send_command 11 2024-03-27T21:40:40.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.758+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62281ed5b0 con 0x7f62281ecbe0 2024-03-27T21:40:40.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.758+0000 7f622c929640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f62281ecbe0 0x7f62281ed020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:40.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.758+0000 7f622c929640 10 monclient: get_auth_request con 0x7f62281ecbe0 auth_method 0 2024-03-27T21:40:40.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.758+0000 7f621dffb640 10 client.4719.objecter ms_handle_connect 0x7f62281ecbe0 2024-03-27T21:40:40.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.758+0000 7f622c929640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f62281ecbe0 0x7f62281ed020 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:40.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.759+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f62281ed5b0 con 0x7f62281ecbe0 2024-03-27T21:40:40.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.759+0000 7f621dffb640 10 client.4719.objecter ms_dispatch 0x7f6228000b90 command_reply(tid 11: 0 ) v1 2024-03-27T21:40:40.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.759+0000 7f621dffb640 10 client.4719.objecter _finish_command 11 = system:0 2024-03-27T21:40:40.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.759+0000 7f621dffb640 20 client.4719.objecter put_session s=0x7f6228065fb0 osd=5 4 2024-03-27T21:40:40.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.759+0000 7f621dffb640 15 client.4719.objecter _session_command_op_remove 5 11 2024-03-27T21:40:40.762 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:40.776+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:40.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.787+0000 7f620b7fe640 10 client.4719.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:40.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.787+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign -1 12 2024-03-27T21:40:40.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.787+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f6228065fb0 osd=5 3 2024-03-27T21:40:40.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.787+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f6228065fb0 osd=5 3 2024-03-27T21:40:40.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.787+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f6228065fb0 osd=5 3 2024-03-27T21:40:40.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.787+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_remove -1 12 2024-03-27T21:40:40.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.787+0000 7f620b7fe640 20 client.4719.objecter get_session s=0x7f6228065fb0 osd=5 3 2024-03-27T21:40:40.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.787+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign 5 12 2024-03-27T21:40:40.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.787+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f6228065fb0 osd=5 4 2024-03-27T21:40:40.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.787+0000 7f620b7fe640 10 client.4719.objecter _send_command 12 2024-03-27T21:40:40.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.787+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f62281ed5b0 con 0x7f62281ecbe0 2024-03-27T21:40:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.789+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f62281ed5b0 con 0x7f62281ecbe0 2024-03-27T21:40:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.789+0000 7f621dffb640 10 client.4719.objecter ms_dispatch 0x7f6228000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.789+0000 7f621dffb640 10 client.4719.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.789+0000 7f621dffb640 20 client.4719.objecter put_session s=0x7f6228065fb0 osd=5 4 2024-03-27T21:40:40.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.789+0000 7f621dffb640 15 client.4719.objecter _session_command_op_remove 5 12 2024-03-27T21:40:40.776 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:40.776 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:40.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.789+0000 7f620b7fe640 10 client.4719.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:40.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.789+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign -1 13 2024-03-27T21:40:40.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.789+0000 7f620b7fe640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f62281f1f40 0x7f62281f4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:40.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.789+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f62281ed5b0 osd=6 3 2024-03-27T21:40:40.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.789+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62281ed5b0 osd=6 3 2024-03-27T21:40:40.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.789+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f62281ed5b0 osd=6 3 2024-03-27T21:40:40.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.789+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_remove -1 13 2024-03-27T21:40:40.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.789+0000 7f620b7fe640 20 client.4719.objecter get_session s=0x7f62281ed5b0 osd=6 3 2024-03-27T21:40:40.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.789+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign 6 13 2024-03-27T21:40:40.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.789+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62281ed5b0 osd=6 4 2024-03-27T21:40:40.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.789+0000 7f620b7fe640 10 client.4719.objecter _send_command 13 2024-03-27T21:40:40.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.789+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62281f7280 con 0x7f62281f1f40 2024-03-27T21:40:40.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.790+0000 7f622d92b640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f62281f1f40 0x7f62281f4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:40.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.790+0000 7f622d92b640 10 monclient: get_auth_request con 0x7f62281f1f40 auth_method 0 2024-03-27T21:40:40.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.790+0000 7f621dffb640 10 client.4719.objecter ms_handle_connect 0x7f62281f1f40 2024-03-27T21:40:40.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.790+0000 7f622d92b640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f62281f1f40 0x7f62281f4330 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:40.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.791+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f62281f7280 con 0x7f62281f1f40 2024-03-27T21:40:40.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.791+0000 7f621dffb640 10 client.4719.objecter ms_dispatch 0x7f6228000b90 command_reply(tid 13: 0 ) v1 2024-03-27T21:40:40.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.791+0000 7f621dffb640 10 client.4719.objecter _finish_command 13 = system:0 2024-03-27T21:40:40.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.791+0000 7f621dffb640 20 client.4719.objecter put_session s=0x7f62281ed5b0 osd=6 4 2024-03-27T21:40:40.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.791+0000 7f621dffb640 15 client.4719.objecter _session_command_op_remove 6 13 2024-03-27T21:40:40.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.819+0000 7f620b7fe640 10 client.4719.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:40.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.819+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign -1 14 2024-03-27T21:40:40.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.819+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f62281ed5b0 osd=6 3 2024-03-27T21:40:40.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.819+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62281ed5b0 osd=6 3 2024-03-27T21:40:40.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.819+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f62281ed5b0 osd=6 3 2024-03-27T21:40:40.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.819+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_remove -1 14 2024-03-27T21:40:40.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.819+0000 7f620b7fe640 20 client.4719.objecter get_session s=0x7f62281ed5b0 osd=6 3 2024-03-27T21:40:40.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.819+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign 6 14 2024-03-27T21:40:40.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.819+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62281ed5b0 osd=6 4 2024-03-27T21:40:40.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.819+0000 7f620b7fe640 10 client.4719.objecter _send_command 14 2024-03-27T21:40:40.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.819+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f62281f7280 con 0x7f62281f1f40 2024-03-27T21:40:40.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.821+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f62281f7280 con 0x7f62281f1f40 2024-03-27T21:40:40.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.821+0000 7f621dffb640 10 client.4719.objecter ms_dispatch 0x7f6228000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:40.807 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:40.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.821+0000 7f621dffb640 10 client.4719.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:40.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.821+0000 7f621dffb640 20 client.4719.objecter put_session s=0x7f62281ed5b0 osd=6 4 2024-03-27T21:40:40.809 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:40.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.821+0000 7f621dffb640 15 client.4719.objecter _session_command_op_remove 6 14 2024-03-27T21:40:40.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.821+0000 7f620b7fe640 10 client.4719.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:40.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.821+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign -1 15 2024-03-27T21:40:40.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.821+0000 7f620b7fe640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f62281f65b0 0x7f62281f69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:40.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.821+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f62281f7280 osd=7 3 2024-03-27T21:40:40.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.821+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62281f7280 osd=7 3 2024-03-27T21:40:40.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.821+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f62281f7280 osd=7 3 2024-03-27T21:40:40.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.821+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_remove -1 15 2024-03-27T21:40:40.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.821+0000 7f620b7fe640 20 client.4719.objecter get_session s=0x7f62281f7280 osd=7 3 2024-03-27T21:40:40.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.821+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign 7 15 2024-03-27T21:40:40.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.821+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62281f7280 osd=7 4 2024-03-27T21:40:40.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.821+0000 7f620b7fe640 10 client.4719.objecter _send_command 15 2024-03-27T21:40:40.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.821+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62281fa600 con 0x7f62281f65b0 2024-03-27T21:40:40.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.821+0000 7f622d12a640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f62281f65b0 0x7f62281f69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:40.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.821+0000 7f622d12a640 10 monclient: get_auth_request con 0x7f62281f65b0 auth_method 0 2024-03-27T21:40:40.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.822+0000 7f621dffb640 10 client.4719.objecter ms_handle_connect 0x7f62281f65b0 2024-03-27T21:40:40.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.822+0000 7f622d12a640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f62281f65b0 0x7f62281f69f0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:40.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.823+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f62281fa600 con 0x7f62281f65b0 2024-03-27T21:40:40.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.823+0000 7f621dffb640 10 client.4719.objecter ms_dispatch 0x7f6228000b90 command_reply(tid 15: 0 ) v1 2024-03-27T21:40:40.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.823+0000 7f621dffb640 10 client.4719.objecter _finish_command 15 = system:0 2024-03-27T21:40:40.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.823+0000 7f621dffb640 20 client.4719.objecter put_session s=0x7f62281f7280 osd=7 4 2024-03-27T21:40:40.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.823+0000 7f621dffb640 15 client.4719.objecter _session_command_op_remove 7 15 2024-03-27T21:40:40.835 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:40.835 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:40.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.851+0000 7f620b7fe640 10 client.4719.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:40.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.851+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign -1 16 2024-03-27T21:40:40.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.851+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f62281f7280 osd=7 3 2024-03-27T21:40:40.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.851+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62281f7280 osd=7 3 2024-03-27T21:40:40.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.851+0000 7f620b7fe640 20 client.4719.objecter _get_session s=0x7f62281f7280 osd=7 3 2024-03-27T21:40:40.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.851+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_remove -1 16 2024-03-27T21:40:40.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.851+0000 7f620b7fe640 20 client.4719.objecter get_session s=0x7f62281f7280 osd=7 3 2024-03-27T21:40:40.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.851+0000 7f620b7fe640 15 client.4719.objecter _session_command_op_assign 7 16 2024-03-27T21:40:40.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.851+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62281f7280 osd=7 4 2024-03-27T21:40:40.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.851+0000 7f620b7fe640 10 client.4719.objecter _send_command 16 2024-03-27T21:40:40.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.851+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f62281f4910 con 0x7f62281f65b0 2024-03-27T21:40:40.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f621dffb640 1 -- 172.21.15.60:0/2204675202 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f62281f4910 con 0x7f62281f65b0 2024-03-27T21:40:40.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f621dffb640 10 client.4719.objecter ms_dispatch 0x7f6228000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:40.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f621dffb640 10 client.4719.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:40.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f621dffb640 20 client.4719.objecter put_session s=0x7f62281f7280 osd=7 4 2024-03-27T21:40:40.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f621dffb640 15 client.4719.objecter _session_command_op_remove 7 16 2024-03-27T21:40:40.840 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:40.842 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:40.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 10 librados: watch_flush enter 2024-03-27T21:40:40.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 10 librados: watch_flush exit 2024-03-27T21:40:40.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 10 client.4719.objecter close_session for osd.0 2024-03-27T21:40:40.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f6228138f90 msgr2=0x7f6228130d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:40.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f6228138f90 0x7f6228130d50 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:40.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62280a90d0 osd=0 1 2024-03-27T21:40:40.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 10 client.4719.objecter close_session for osd.1 2024-03-27T21:40:40.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f62281317a0 msgr2=0x7f62280b4bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:40.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f62281317a0 0x7f62280b4bc0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:40.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62281395b0 osd=1 1 2024-03-27T21:40:40.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 10 client.4719.objecter close_session for osd.2 2024-03-27T21:40:40.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f6228138b80 msgr2=0x7f62281e7240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:40.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f6228138b80 0x7f62281e7240 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:40.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f622d92b640 1 -- 172.21.15.60:0/2204675202 reap_dead start 2024-03-27T21:40:40.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f6228138a10 osd=2 1 2024-03-27T21:40:40.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 10 client.4719.objecter close_session for osd.3 2024-03-27T21:40:40.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f6228066760 msgr2=0x7f62281e8900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:40.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f6228066760 0x7f62281e8900 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:40.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62280b5750 osd=3 1 2024-03-27T21:40:40.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 10 client.4719.objecter close_session for osd.4 2024-03-27T21:40:40.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f62281e8450 msgr2=0x7f62281ec190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:40.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f62281e8450 0x7f62281ec190 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:40.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f6228065e40 osd=4 1 2024-03-27T21:40:40.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 10 client.4719.objecter close_session for osd.5 2024-03-27T21:40:40.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f62281ecbe0 msgr2=0x7f62281ed020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f62281ecbe0 0x7f62281ed020 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f6228065fb0 osd=5 1 2024-03-27T21:40:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 10 client.4719.objecter close_session for osd.6 2024-03-27T21:40:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f62281f1f40 msgr2=0x7f62281f4330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f62281f1f40 0x7f62281f4330 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62281ed5b0 osd=6 1 2024-03-27T21:40:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 10 client.4719.objecter close_session for osd.7 2024-03-27T21:40:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f62281f65b0 msgr2=0x7f62281f69f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f62281f65b0 0x7f62281f69f0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f622d92b640 1 -- 172.21.15.60:0/2204675202 reap_dead start 2024-03-27T21:40:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 20 client.4719.objecter put_session s=0x7f62281f7280 osd=7 1 2024-03-27T21:40:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 20 client.4719.objecter shutdown clearing up homeless session... 2024-03-27T21:40:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 10 client.4719.objecter successfully canceled tick 2024-03-27T21:40:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f620c06c590 msgr2=0x7f620c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f620c06c590 0x7f620c06ea50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f62240094a0 tx=0x7f6224008040 comp rx=0 tx=0).stop 2024-03-27T21:40:40.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 10 monclient: shutdown 2024-03-27T21:40:40.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:40.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62281416f0 msgr2=0x7f622813daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:40.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.853+0000 7f620b7fe640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62281416f0 0x7f622813daf0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f622000c700 tx=0x7f622000cbc0 comp rx=0 tx=0).stop 2024-03-27T21:40:40.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.854+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 shutdown_connections 2024-03-27T21:40:40.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.854+0000 7f620b7fe640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62281416f0 0x7f622813daf0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:40.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.854+0000 7f620b7fe640 1 --2- 172.21.15.60:0/2204675202 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f620c06c590 0x7f620c06ea50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:40.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.854+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 >> 172.21.15.60:0/2204675202 conn(0x7f62280608e0 msgr2=0x7f6228140cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:40.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.854+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 shutdown_connections 2024-03-27T21:40:40.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.854+0000 7f620b7fe640 1 -- 172.21.15.60:0/2204675202 wait complete. 2024-03-27T21:40:40.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:40.854+0000 7f620b7fe640 1 librados: shutdown 2024-03-27T21:40:40.846 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:40:40.847 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:40.860+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:40.947 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:40.960+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:40.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.001+0000 7f5ffabba640 10 start_mon_command cmd=[{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:40:40.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.001+0000 7f5ffabba640 10 monclient: _send_command 2 [{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:40:40.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.001+0000 7f5ffabba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:40.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.001+0000 7f5ffabba640 1 -- 172.21.15.60:0/116778768 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f5fec0c0ff0 con 0x7f5fec02b4c0 2024-03-27T21:40:40.993 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:40.993 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:41.000 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:40:41.048 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:41.062+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:41.146 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:41.146 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:41.150 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:41.164+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:41.155 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:40:41.251 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:41.265+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:41.258 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:41.259 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:41.267 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:40:41.352 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:41.365+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:41.372 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:41.372 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:41.381 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:40:41.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.467+0000 7f5fea7fc640 1 -- 172.21.15.60:0/116778768 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v42) v1 ==== 87+0+0 (secure 0 0 0) 0x7f5fec0c0ff0 con 0x7f5fec02b4c0 2024-03-27T21:40:41.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.467+0000 7f5fea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:40:41.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.467+0000 7f5fea7fc640 10 monclient: _finish_command 2 = system:0 noscrub is set 2024-03-27T21:40:41.454 INFO:teuthology.orchestra.run.smithi060.stderr:noscrub is set 2024-03-27T21:40:41.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.470+0000 7f5ffabba640 10 librados: watch_flush enter 2024-03-27T21:40:41.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.470+0000 7f5ffabba640 10 librados: watch_flush exit 2024-03-27T21:40:41.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.470+0000 7f5ffabba640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-27T21:40:41.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.470+0000 7f5ffabba640 10 client.4577.objecter successfully canceled tick 2024-03-27T21:40:41.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.470+0000 7f5ffabba640 1 -- 172.21.15.60:0/116778768 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5fd806c590 msgr2=0x7f5fd806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:41.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.470+0000 7f5ffabba640 1 --2- 172.21.15.60:0/116778768 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5fd806c590 0x7f5fd806ea50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5fec0cab30 tx=0x7f5fe4008040 comp rx=0 tx=0).stop 2024-03-27T21:40:41.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.470+0000 7f5ffabba640 10 monclient: shutdown 2024-03-27T21:40:41.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.470+0000 7f5ffabba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:41.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.470+0000 7f5ffabba640 1 -- 172.21.15.60:0/116778768 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5fec02b4c0 msgr2=0x7f5fec0c07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:41.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.470+0000 7f5ffabba640 1 --2- 172.21.15.60:0/116778768 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5fec02b4c0 0x7f5fec0c07b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0035040 tx=0x7f5ff00046c0 comp rx=0 tx=0).stop 2024-03-27T21:40:41.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.470+0000 7f5ffabba640 1 -- 172.21.15.60:0/116778768 shutdown_connections 2024-03-27T21:40:41.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.470+0000 7f5ffabba640 1 --2- 172.21.15.60:0/116778768 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5fd806c590 0x7f5fd806ea50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:41.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.470+0000 7f5ffabba640 1 --2- 172.21.15.60:0/116778768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5fec0cb300 0x7f5fec0b7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:41.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.470+0000 7f5ffabba640 1 --2- 172.21.15.60:0/116778768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fec037540 0x7f5fec0b7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:41.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.470+0000 7f5ffabba640 1 --2- 172.21.15.60:0/116778768 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5fec02b4c0 0x7f5fec0c07b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:41.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.470+0000 7f5ffabba640 1 -- 172.21.15.60:0/116778768 >> 172.21.15.60:0/116778768 conn(0x7f5fec03a8b0 msgr2=0x7f5fec037ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:41.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.470+0000 7f5ffabba640 1 -- 172.21.15.60:0/116778768 shutdown_connections 2024-03-27T21:40:41.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.470+0000 7f5ffabba640 1 -- 172.21.15.60:0/116778768 wait complete. 2024-03-27T21:40:41.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:41.470+0000 7f5ffabba640 1 librados: shutdown 2024-03-27T21:40:41.465 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:41.479+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:41.527 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:41.527 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:41.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:40:41.577 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:41.590+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:41.655 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:41.669+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:41.681 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:41.681 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:41.690 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:40:41.775 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:41.788+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:41.836 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:41.836 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:41.845 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:40:41.858 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:41.872+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:41.959 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:41.973+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:41.991 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:41.991 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:42.001 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:40:42.060 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:42.074+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:42.106 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:42.107 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:42.115 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:40:42.161 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:42.175+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:42.219 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:42.219 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:42.228 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:40:42.263 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:42.277+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:42.364 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:42.378+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:42.374 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:42.374 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:42.383 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-27T21:40:42.466 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:42.480+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:42.496 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:42.496 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:42.505 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-27T21:40:42.567 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:42.581+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:42.615 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:42.615 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:42.624 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-27T21:40:42.667 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:42.681+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:42.732 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:42.732 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:42.744 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:40:42.769 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:42.783+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:42.852 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:40:42.855 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:42.855 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:42.864 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:40:42.905 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:42.919+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:42.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.971+0000 7f42a495c640 10 monclient: get_monmap_and_config 2024-03-27T21:40:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.971+0000 7f42a495c640 10 monclient: build_initial_monmap 2024-03-27T21:40:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.971+0000 7f42a495c640 10 monclient: monmap: 2024-03-27T21:40:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:42.972200+0000 2024-03-27T21:40:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:42.972200+0000 2024-03-27T21:40:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:42.959 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:42.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.971+0000 7f42a495c640 1 Processor -- start 2024-03-27T21:40:42.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.971+0000 7f42a495c640 1 -- start start 2024-03-27T21:40:42.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.971+0000 7f42a495c640 10 monclient: init 2024-03-27T21:40:42.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.972+0000 7f42a495c640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:42.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.972+0000 7f42a495c640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-03-27T21:40:42.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.972+0000 7f42a495c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42a00b40b0 0x7f42a00a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:42.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.972+0000 7f42a495c640 10 monclient(hunting): picked mon.noname-b con 0x7f42a00b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:42.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.972+0000 7f42a495c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f42a013b020 0x7f42a013b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:42.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.972+0000 7f42a495c640 10 monclient(hunting): picked mon.noname-c con 0x7f42a013b020 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:42.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.972+0000 7f42a495c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f42a01416f0 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:42.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.972+0000 7f42a495c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42a00b3830 con 0x7f42a01416f0 2024-03-27T21:40:42.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.972+0000 7f42a495c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:42.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.972+0000 7f42a495c640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42a00b4490 con 0x7f42a013b020 2024-03-27T21:40:42.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.972+0000 7f42a495c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:42.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.972+0000 7f42a495c640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a0142290 con 0x7f42a00b40b0 2024-03-27T21:40:42.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.972+0000 7f42a495c640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:42.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.972+0000 7f42a495c640 10 monclient(hunting): authenticate will time out at 884.193728s 2024-03-27T21:40:42.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.972+0000 7f429e59c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42a00b40b0 0x7f42a00a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:42.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.972+0000 7f429e59c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42a00b40b0 0x7f42a00a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:57364/0 (socket says 172.21.15.60:57364) 2024-03-27T21:40:42.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.972+0000 7f429e59c640 1 -- 172.21.15.60:0/1828189207 learned_addr learned my addr 172.21.15.60:0/1828189207 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:42.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.972+0000 7f429e59c640 10 monclient(hunting): get_auth_request con 0x7f42a00b40b0 auth_method 0 2024-03-27T21:40:42.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.972+0000 7f429e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:42.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.972+0000 7f429e59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:42.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.972+0000 7f429e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:42.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.973+0000 7f429e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:42.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.973+0000 7f429e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:42.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.973+0000 7f429e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:42.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.973+0000 7f429e59c640 10 monclient(hunting): handle_auth_done global_id 4580 payload 274 2024-03-27T21:40:42.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.973+0000 7f429e59c640 1 -- 172.21.15.60:0/1828189207 >> v2:172.21.15.191:3300/0 conn(0x7f42a013b020 msgr2=0x7f42a013b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:42.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.973+0000 7f429e59c640 1 --2- 172.21.15.60:0/1828189207 >> v2:172.21.15.191:3300/0 conn(0x7f42a013b020 0x7f42a013b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:42.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.973+0000 7f429e59c640 1 -- 172.21.15.60:0/1828189207 >> v1:172.21.15.60:6789/0 conn(0x7f42a01416f0 legacy=0x7f42a0141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:42.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.973+0000 7f429e59c640 10 monclient: _finish_hunting 0 2024-03-27T21:40:42.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.973+0000 7f429e59c640 1 monclient: found mon.noname-b 2024-03-27T21:40:42.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.973+0000 7f429e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:42.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.973+0000 7f429e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:42.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.973+0000 7f429e59c640 1 -- 172.21.15.60:0/1828189207 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a00aa520 con 0x7f42a00b40b0 2024-03-27T21:40:42.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.974+0000 7f429e59c640 1 --2- 172.21.15.60:0/1828189207 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42a00b40b0 0x7f42a00a8b90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4290009870 tx=0x7f429002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dde239b8e6fe1db server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:42.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.974+0000 7f429d59a640 1 -- 172.21.15.60:0/1828189207 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4290030840 con 0x7f42a00b40b0 2024-03-27T21:40:42.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.974+0000 7f429d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:42.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.974+0000 7f429d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:42.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.974+0000 7f429d59a640 10 monclient: dump: 2024-03-27T21:40:42.966 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:42.966 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:42.967 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:42.967 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:42.967 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:42.967 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:42.967 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:42.967 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:42.967 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:42.967 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:42.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.974+0000 7f429d59a640 10 monclient: _finish_auth 0 2024-03-27T21:40:42.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.974+0000 7f429d59a640 10 monclient: _check_auth_tickets 2024-03-27T21:40:42.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.974+0000 7f429d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:42.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.974+0000 7f42a495c640 5 monclient: authenticate success, global_id 4580 2024-03-27T21:40:42.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.974+0000 7f42a495c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:42.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.974+0000 7f429d59a640 1 -- 172.21.15.60:0/1828189207 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42900309a0 con 0x7f42a00b40b0 2024-03-27T21:40:42.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.974+0000 7f429d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:42.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.974+0000 7f429d59a640 1 -- 172.21.15.60:0/1828189207 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4290030d30 con 0x7f42a00b40b0 2024-03-27T21:40:42.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.974+0000 7f429d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:42.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.974+0000 7f429d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:42.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.974+0000 7f429d59a640 10 monclient: dump: 2024-03-27T21:40:42.968 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:42.968 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:42.968 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:42.968 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:42.968 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:42.968 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:42.968 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:42.968 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:42.968 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:42.968 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:42.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.974+0000 7f42a495c640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:42.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.974+0000 7f42a495c640 4 set_mon_vals no callback set 2024-03-27T21:40:42.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.975+0000 7f42a495c640 10 monclient: shutdown 2024-03-27T21:40:42.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.975+0000 7f42a495c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:42.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.975+0000 7f42a495c640 1 -- 172.21.15.60:0/1828189207 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42a00b40b0 msgr2=0x7f42a00a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:42.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.975+0000 7f42a495c640 1 --2- 172.21.15.60:0/1828189207 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42a00b40b0 0x7f42a00a8b90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4290009870 tx=0x7f429002b980 comp rx=0 tx=0).stop 2024-03-27T21:40:42.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.975+0000 7f42a495c640 1 -- 172.21.15.60:0/1828189207 shutdown_connections 2024-03-27T21:40:42.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.975+0000 7f42a495c640 1 --2- 172.21.15.60:0/1828189207 >> v2:172.21.15.191:3300/0 conn(0x7f42a013b020 0x7f42a013b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:42.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.975+0000 7f42a495c640 1 --2- 172.21.15.60:0/1828189207 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42a00b40b0 0x7f42a00a8b90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:42.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.975+0000 7f42a495c640 1 -- 172.21.15.60:0/1828189207 >> 172.21.15.60:0/1828189207 conn(0x7f42a00608e0 msgr2=0x7f42a0060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:42.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.975+0000 7f42a495c640 1 -- 172.21.15.60:0/1828189207 shutdown_connections 2024-03-27T21:40:42.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.975+0000 7f42a495c640 1 -- 172.21.15.60:0/1828189207 wait complete. 2024-03-27T21:40:42.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.976+0000 7f42a495c640 10 monclient: build_initial_monmap 2024-03-27T21:40:42.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.976+0000 7f42a495c640 10 monclient: monmap: 2024-03-27T21:40:42.969 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:42.970 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:42.970 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:42.970 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:42.970 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:42.970 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:42.970 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:42.970 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:42.970 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:42.970 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:42.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.977+0000 7f42a495c640 1 librados: starting msgr at 2024-03-27T21:40:42.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.977+0000 7f42a495c640 1 librados: starting objecter 2024-03-27T21:40:42.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.977+0000 7f42a495c640 1 Processor -- start 2024-03-27T21:40:42.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.977+0000 7f42a495c640 1 -- start start 2024-03-27T21:40:42.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.977+0000 7f42a495c640 1 librados: setting wanted keys 2024-03-27T21:40:42.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.977+0000 7f42a495c640 1 librados: calling monclient init 2024-03-27T21:40:42.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.977+0000 7f42a495c640 10 monclient: init 2024-03-27T21:40:42.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.978+0000 7f42a495c640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:42.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.978+0000 7f42a495c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:40:42.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.978+0000 7f42a495c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42a00b40b0 0x7f42a0139ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:42.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.978+0000 7f42a495c640 10 monclient(hunting): picked mon.noname-a con 0x7f42a00b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:42.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.978+0000 7f429e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42a00b40b0 0x7f42a0139ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:42.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42a00b40b0 0x7f42a0139ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39694/0 (socket says 172.21.15.60:39694) 2024-03-27T21:40:42.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429e59c640 1 -- 172.21.15.60:0/976326530 learned_addr learned my addr 172.21.15.60:0/976326530 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:42.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f42a495c640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42a013b020 0x7f42a01333f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:42.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f42a495c640 10 monclient(hunting): picked mon.noname-c con 0x7f42a013b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:42.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f42a495c640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42a01416f0 0x7f42a01d6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:42.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429dd9b640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42a013b020 0x7f42a01333f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:42.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f42a495c640 10 monclient(hunting): picked mon.noname-b con 0x7f42a01416f0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:42.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f42a495c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:42.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f42a495c640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a01d7200 con 0x7f42a00b40b0 2024-03-27T21:40:42.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f42a495c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:42.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f42a495c640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a013e870 con 0x7f42a01416f0 2024-03-27T21:40:42.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f42a495c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:42.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f42a495c640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a0133930 con 0x7f42a013b020 2024-03-27T21:40:42.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f42a495c640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:42.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f42a495c640 10 monclient(hunting): authenticate will time out at 884.200430s 2024-03-27T21:40:42.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429e59c640 10 monclient(hunting): get_auth_request con 0x7f42a00b40b0 auth_method 0 2024-03-27T21:40:42.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:42.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429e59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:42.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:42.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429dd9b640 10 monclient(hunting): get_auth_request con 0x7f42a013b020 auth_method 0 2024-03-27T21:40:42.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:42.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429ed9d640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42a01416f0 0x7f42a01d6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:42.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:42.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:42.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429ed9d640 10 monclient(hunting): get_auth_request con 0x7f42a01416f0 auth_method 0 2024-03-27T21:40:42.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:42.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429ed9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:42.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:42.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:42.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:42.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:42.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:42.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:42.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.979+0000 7f429ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:42.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f429ed9d640 10 monclient(hunting): handle_auth_done global_id 4669 payload 824 2024-03-27T21:40:42.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f429ed9d640 1 -- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42a013b020 msgr2=0x7f42a01333f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:42.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f429ed9d640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42a013b020 0x7f42a01333f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:42.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f429ed9d640 1 -- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42a00b40b0 msgr2=0x7f42a0139ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:42.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f429ed9d640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42a00b40b0 0x7f42a0139ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:42.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f429ed9d640 10 monclient: _finish_hunting 0 2024-03-27T21:40:42.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f429ed9d640 1 monclient: found mon.noname-b 2024-03-27T21:40:42.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f429ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:42.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f429ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:40:42.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f429ed9d640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42900094d0 con 0x7f42a01416f0 2024-03-27T21:40:42.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f429dd9b640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42a013b020 0x7f42a01333f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:42.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f429e59c640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42a00b40b0 0x7f42a0139ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:40:42.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f428effd640 10 client.?.objecter ms_handle_connect 0x7f42a01416f0 2024-03-27T21:40:42.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f428effd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:42.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f429ed9d640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42a01416f0 0x7f42a01d6730 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f429800c760 tx=0x7f429800cb60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:42.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429800cea0 con 0x7f42a01416f0 2024-03-27T21:40:42.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f428effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:42.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f428effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:42.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f428effd640 10 monclient: dump: 2024-03-27T21:40:42.984 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:42.984 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:42.984 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:42.984 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:42.984 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:42.984 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:42.984 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:42.984 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:42.984 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:42.985 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:42.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f428effd640 10 monclient: _finish_auth 0 2024-03-27T21:40:42.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f428effd640 10 monclient: _check_auth_tickets 2024-03-27T21:40:42.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f428effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:42.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f42a495c640 5 monclient: authenticate success, global_id 4669 2024-03-27T21:40:42.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f42a495c640 10 monclient: _renew_subs 2024-03-27T21:40:42.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4298006d50 con 0x7f42a01416f0 2024-03-27T21:40:42.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f42a495c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:42.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f42a495c640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a0133bb0 con 0x7f42a01416f0 2024-03-27T21:40:42.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f428effd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:42.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429800cea0 con 0x7f42a01416f0 2024-03-27T21:40:42.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f428effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:42.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f428effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:40:42.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f428effd640 10 monclient: dump: 2024-03-27T21:40:42.986 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:42.986 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:42.986 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:42.986 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:42.986 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:42.986 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:42.986 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:42.986 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:42.986 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:42.986 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:42.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.980+0000 7f429cd99640 4 set_mon_vals no callback set 2024-03-27T21:40:42.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.981+0000 7f42a495c640 10 client.4669.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:42.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.981+0000 7f42a495c640 10 monclient: _renew_subs 2024-03-27T21:40:42.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.981+0000 7f42a495c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:42.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.981+0000 7f42a495c640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a01d6e50 con 0x7f42a01416f0 2024-03-27T21:40:42.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.981+0000 7f42a495c640 1 librados: init done 2024-03-27T21:40:42.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.982+0000 7f42a495c640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:42.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.982+0000 7f42a495c640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:42.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.982+0000 7f42a495c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:40:42.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.982+0000 7f42a495c640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f42a01d7070 con 0x7f42a01416f0 2024-03-27T21:40:42.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.982+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4298003970 con 0x7f42a01416f0 2024-03-27T21:40:42.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.982+0000 7f428effd640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f428006c5f0 0x7f428006eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:42.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f4298088820 con 0x7f42a01416f0 2024-03-27T21:40:42.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f428effd640 10 client.4669.objecter ms_dispatch 0x7f42a0000b90 osd_map(42..42 src has 1..42) v4 2024-03-27T21:40:42.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f428effd640 3 client.4669.objecter handle_osd_map got epochs [42,42] > 0 2024-03-27T21:40:42.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f428effd640 3 client.4669.objecter handle_osd_map decoding full epoch 42 2024-03-27T21:40:42.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f428effd640 20 client.4669.objecter dump_active .. 0 homeless 2024-03-27T21:40:42.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f429e59c640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f428006c5f0 0x7f428006eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:42.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v42) v1 ==== 54+0+16 (secure 0 0 0) 0x7f4298088bc0 con 0x7f42a01416f0 2024-03-27T21:40:42.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f428effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:42.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f428effd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:42.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f429e59c640 10 monclient: get_auth_request con 0x7f428006c5f0 auth_method 0 2024-03-27T21:40:42.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f428cff9640 10 client.4669.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:42.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign -1 1 2024-03-27T21:40:42.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f428cff9640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f42a01d9770 0x7f42a01d9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:42.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a00a90d0 osd=0 3 2024-03-27T21:40:42.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a00a90d0 osd=0 3 2024-03-27T21:40:42.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a00a90d0 osd=0 3 2024-03-27T21:40:42.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f428cff9640 15 client.4669.objecter _session_command_op_remove -1 1 2024-03-27T21:40:42.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f428cff9640 20 client.4669.objecter get_session s=0x7f42a00a90d0 osd=0 3 2024-03-27T21:40:42.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign 0 1 2024-03-27T21:40:42.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a00a90d0 osd=0 4 2024-03-27T21:40:42.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f428cff9640 10 client.4669.objecter _send_command 1 2024-03-27T21:40:42.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f42a0069470 con 0x7f42a01d9770 2024-03-27T21:40:42.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f428effd640 10 client.4669.objecter ms_handle_connect 0x7f428006c5f0 2024-03-27T21:40:42.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f429e59c640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f428006c5f0 0x7f428006eab0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4290009870 tx=0x7f4290008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:42.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f429dd9b640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f42a01d9770 0x7f42a01d9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:42.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.983+0000 7f429dd9b640 10 monclient: get_auth_request con 0x7f42a01d9770 auth_method 0 2024-03-27T21:40:42.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.984+0000 7f428effd640 10 client.4669.objecter ms_handle_connect 0x7f42a01d9770 2024-03-27T21:40:42.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.984+0000 7f429dd9b640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f42a01d9770 0x7f42a01d9b50 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:42.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.985+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f42a0069470 con 0x7f42a01d9770 2024-03-27T21:40:42.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.985+0000 7f428effd640 10 client.4669.objecter ms_dispatch 0x7f42a0000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:42.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.985+0000 7f428effd640 10 client.4669.objecter _finish_command 1 = system:0 2024-03-27T21:40:42.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.985+0000 7f428effd640 20 client.4669.objecter put_session s=0x7f42a00a90d0 osd=0 4 2024-03-27T21:40:42.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.985+0000 7f428effd640 15 client.4669.objecter _session_command_op_remove 0 1 2024-03-27T21:40:42.990 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:42.985+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:42.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.999+0000 7f428cff9640 10 client.4669.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:42.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.999+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign -1 2 2024-03-27T21:40:42.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.999+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a00a90d0 osd=0 3 2024-03-27T21:40:42.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.999+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a00a90d0 osd=0 3 2024-03-27T21:40:42.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.999+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a00a90d0 osd=0 3 2024-03-27T21:40:42.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.999+0000 7f428cff9640 15 client.4669.objecter _session_command_op_remove -1 2 2024-03-27T21:40:42.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.999+0000 7f428cff9640 20 client.4669.objecter get_session s=0x7f42a00a90d0 osd=0 3 2024-03-27T21:40:42.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.999+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign 0 2 2024-03-27T21:40:42.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.999+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a00a90d0 osd=0 4 2024-03-27T21:40:42.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.999+0000 7f428cff9640 10 client.4669.objecter _send_command 2 2024-03-27T21:40:42.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:42.999+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f42a014bd20 con 0x7f42a01d9770 2024-03-27T21:40:42.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.001+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f42a014bd20 con 0x7f42a01d9770 2024-03-27T21:40:42.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.001+0000 7f428effd640 10 client.4669.objecter ms_dispatch 0x7f42a0000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:42.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.001+0000 7f428effd640 10 client.4669.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:42.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.001+0000 7f428effd640 20 client.4669.objecter put_session s=0x7f42a00a90d0 osd=0 4 2024-03-27T21:40:42.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.001+0000 7f428effd640 15 client.4669.objecter _session_command_op_remove 0 2 2024-03-27T21:40:42.993 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:42.994 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:42.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.001+0000 7f428cff9640 10 client.4669.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:42.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.001+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign -1 3 2024-03-27T21:40:42.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.001+0000 7f428cff9640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f42a01da2d0 0x7f42a00b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:42.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.001+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a014be90 osd=1 3 2024-03-27T21:40:42.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.001+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a014be90 osd=1 3 2024-03-27T21:40:42.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.001+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a014be90 osd=1 3 2024-03-27T21:40:42.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.001+0000 7f428cff9640 15 client.4669.objecter _session_command_op_remove -1 3 2024-03-27T21:40:42.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.001+0000 7f428cff9640 20 client.4669.objecter get_session s=0x7f42a014be90 osd=1 3 2024-03-27T21:40:42.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.001+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign 1 3 2024-03-27T21:40:42.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.001+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a014be90 osd=1 4 2024-03-27T21:40:42.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.001+0000 7f428cff9640 10 client.4669.objecter _send_command 3 2024-03-27T21:40:42.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.001+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f42a00b5610 con 0x7f42a01da2d0 2024-03-27T21:40:42.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.001+0000 7f429e59c640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f42a01da2d0 0x7f42a00b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.001+0000 7f429e59c640 10 monclient: get_auth_request con 0x7f42a01da2d0 auth_method 0 2024-03-27T21:40:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.002+0000 7f428effd640 10 client.4669.objecter ms_handle_connect 0x7f42a01da2d0 2024-03-27T21:40:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.002+0000 7f429e59c640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f42a01da2d0 0x7f42a00b50d0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.002+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f42a00b5610 con 0x7f42a01da2d0 2024-03-27T21:40:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.002+0000 7f428effd640 10 client.4669.objecter ms_dispatch 0x7f42a0000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.002+0000 7f428effd640 10 client.4669.objecter _finish_command 3 = system:0 2024-03-27T21:40:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.002+0000 7f428effd640 20 client.4669.objecter put_session s=0x7f42a014be90 osd=1 4 2024-03-27T21:40:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.002+0000 7f428effd640 15 client.4669.objecter _session_command_op_remove 1 3 2024-03-27T21:40:43.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.018+0000 7f428cff9640 10 client.4669.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:43.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.018+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign -1 4 2024-03-27T21:40:43.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.018+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a014be90 osd=1 3 2024-03-27T21:40:43.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.018+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a014be90 osd=1 3 2024-03-27T21:40:43.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.018+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a014be90 osd=1 3 2024-03-27T21:40:43.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.018+0000 7f428cff9640 15 client.4669.objecter _session_command_op_remove -1 4 2024-03-27T21:40:43.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.018+0000 7f428cff9640 20 client.4669.objecter get_session s=0x7f42a014be90 osd=1 3 2024-03-27T21:40:43.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.018+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign 1 4 2024-03-27T21:40:43.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.018+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a014be90 osd=1 4 2024-03-27T21:40:43.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.018+0000 7f428cff9640 10 client.4669.objecter _send_command 4 2024-03-27T21:40:43.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.018+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f42a00b59c0 con 0x7f42a01da2d0 2024-03-27T21:40:43.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.020+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f42a00b59c0 con 0x7f42a01da2d0 2024-03-27T21:40:43.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.020+0000 7f428effd640 10 client.4669.objecter ms_dispatch 0x7f42a0000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:43.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.020+0000 7f428effd640 10 client.4669.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:43.007 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:43.007 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:43.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.020+0000 7f428effd640 20 client.4669.objecter put_session s=0x7f42a014be90 osd=1 4 2024-03-27T21:40:43.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.020+0000 7f428effd640 15 client.4669.objecter _session_command_op_remove 1 4 2024-03-27T21:40:43.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.020+0000 7f428cff9640 10 client.4669.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:43.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.020+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign -1 5 2024-03-27T21:40:43.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.020+0000 7f428cff9640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f42a0066250 0x7f42a0066630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:43.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.020+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a00b5b90 osd=2 3 2024-03-27T21:40:43.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.020+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a00b5b90 osd=2 3 2024-03-27T21:40:43.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.020+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a00b5b90 osd=2 3 2024-03-27T21:40:43.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.020+0000 7f428cff9640 15 client.4669.objecter _session_command_op_remove -1 5 2024-03-27T21:40:43.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.020+0000 7f428cff9640 20 client.4669.objecter get_session s=0x7f42a00b5b90 osd=2 3 2024-03-27T21:40:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.020+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign 2 5 2024-03-27T21:40:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.020+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a00b5b90 osd=2 4 2024-03-27T21:40:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.020+0000 7f428cff9640 10 client.4669.objecter _send_command 5 2024-03-27T21:40:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.020+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f42a01ea4e0 con 0x7f42a0066250 2024-03-27T21:40:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.020+0000 7f429dd9b640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f42a0066250 0x7f42a0066630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.020+0000 7f429dd9b640 10 monclient: get_auth_request con 0x7f42a0066250 auth_method 0 2024-03-27T21:40:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.021+0000 7f428effd640 10 client.4669.objecter ms_handle_connect 0x7f42a0066250 2024-03-27T21:40:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.021+0000 7f429dd9b640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f42a0066250 0x7f42a0066630 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.022+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== osd.2 v2:172.21.15.60:6824/44654 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f42a01ea4e0 con 0x7f42a0066250 2024-03-27T21:40:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.022+0000 7f428effd640 10 client.4669.objecter ms_dispatch 0x7f42a0000b90 command_reply(tid 5: 0 ) v1 2024-03-27T21:40:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.022+0000 7f428effd640 10 client.4669.objecter _finish_command 5 = system:0 2024-03-27T21:40:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.022+0000 7f428effd640 20 client.4669.objecter put_session s=0x7f42a00b5b90 osd=2 4 2024-03-27T21:40:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.022+0000 7f428effd640 15 client.4669.objecter _session_command_op_remove 2 5 2024-03-27T21:40:43.018 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-03-27T21:40:43.018 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T21:40:43.018 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-27T21:40:43.018 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-03-27T21:40:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.038+0000 7f428cff9640 10 client.4669.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.038+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign -1 6 2024-03-27T21:40:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.038+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a00b5b90 osd=2 3 2024-03-27T21:40:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.038+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a00b5b90 osd=2 3 2024-03-27T21:40:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.038+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a00b5b90 osd=2 3 2024-03-27T21:40:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.038+0000 7f428cff9640 15 client.4669.objecter _session_command_op_remove -1 6 2024-03-27T21:40:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.038+0000 7f428cff9640 20 client.4669.objecter get_session s=0x7f42a00b5b90 osd=2 3 2024-03-27T21:40:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.038+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign 2 6 2024-03-27T21:40:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.038+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a00b5b90 osd=2 4 2024-03-27T21:40:43.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.038+0000 7f428cff9640 10 client.4669.objecter _send_command 6 2024-03-27T21:40:43.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.038+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f42a0065cd0 con 0x7f42a0066250 2024-03-27T21:40:43.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.040+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== osd.2 v2:172.21.15.60:6824/44654 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f42a0065cd0 con 0x7f42a0066250 2024-03-27T21:40:43.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.040+0000 7f428effd640 10 client.4669.objecter ms_dispatch 0x7f42a0000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:43.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.040+0000 7f428effd640 10 client.4669.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:43.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.040+0000 7f428effd640 20 client.4669.objecter put_session s=0x7f42a00b5b90 osd=2 4 2024-03-27T21:40:43.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.040+0000 7f428effd640 15 client.4669.objecter _session_command_op_remove 2 6 2024-03-27T21:40:43.030 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2: {} 2024-03-27T21:40:43.033 INFO:teuthology.orchestra.run.smithi060.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-27T21:40:43.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.040+0000 7f428cff9640 10 client.4669.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:43.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.041+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign -1 7 2024-03-27T21:40:43.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.041+0000 7f428cff9640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f42a01e5790 0x7f42a01e8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:43.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.041+0000 7f429ed9d640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f42a01e5790 0x7f42a01e8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:43.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.041+0000 7f429ed9d640 10 monclient: get_auth_request con 0x7f42a01e5790 auth_method 0 2024-03-27T21:40:43.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.041+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a0066080 osd=3 3 2024-03-27T21:40:43.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.041+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a0066080 osd=3 3 2024-03-27T21:40:43.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.041+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a0066080 osd=3 3 2024-03-27T21:40:43.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.041+0000 7f428cff9640 15 client.4669.objecter _session_command_op_remove -1 7 2024-03-27T21:40:43.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.041+0000 7f428cff9640 20 client.4669.objecter get_session s=0x7f42a0066080 osd=3 3 2024-03-27T21:40:43.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.041+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign 3 7 2024-03-27T21:40:43.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.041+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a0066080 osd=3 4 2024-03-27T21:40:43.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.041+0000 7f428cff9640 10 client.4669.objecter _send_command 7 2024-03-27T21:40:43.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.041+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f42a01edd70 con 0x7f42a01e5790 2024-03-27T21:40:43.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.042+0000 7f429ed9d640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f42a01e5790 0x7f42a01e8dd0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:43.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.042+0000 7f428effd640 10 client.4669.objecter ms_handle_connect 0x7f42a01e5790 2024-03-27T21:40:43.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.043+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f42a01edd70 con 0x7f42a01e5790 2024-03-27T21:40:43.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.043+0000 7f428effd640 10 client.4669.objecter ms_dispatch 0x7f42a0000b90 command_reply(tid 7: 0 ) v1 2024-03-27T21:40:43.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.043+0000 7f428effd640 10 client.4669.objecter _finish_command 7 = system:0 2024-03-27T21:40:43.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.043+0000 7f428effd640 20 client.4669.objecter put_session s=0x7f42a0066080 osd=3 4 2024-03-27T21:40:43.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.043+0000 7f428effd640 15 client.4669.objecter _session_command_op_remove 3 7 2024-03-27T21:40:43.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.059+0000 7f428cff9640 10 client.4669.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:43.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.059+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign -1 8 2024-03-27T21:40:43.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.059+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a0066080 osd=3 3 2024-03-27T21:40:43.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.059+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a0066080 osd=3 3 2024-03-27T21:40:43.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.059+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a0066080 osd=3 3 2024-03-27T21:40:43.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.059+0000 7f428cff9640 15 client.4669.objecter _session_command_op_remove -1 8 2024-03-27T21:40:43.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.059+0000 7f428cff9640 20 client.4669.objecter get_session s=0x7f42a0066080 osd=3 3 2024-03-27T21:40:43.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.059+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign 3 8 2024-03-27T21:40:43.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.059+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a0066080 osd=3 4 2024-03-27T21:40:43.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.059+0000 7f428cff9640 10 client.4669.objecter _send_command 8 2024-03-27T21:40:43.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.059+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f42a01e4d00 con 0x7f42a01e5790 2024-03-27T21:40:43.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.062+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f42a01e4d00 con 0x7f42a01e5790 2024-03-27T21:40:43.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.062+0000 7f428effd640 10 client.4669.objecter ms_dispatch 0x7f42a0000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:43.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.062+0000 7f428effd640 10 client.4669.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:43.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.062+0000 7f428effd640 20 client.4669.objecter put_session s=0x7f42a0066080 osd=3 4 2024-03-27T21:40:43.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.062+0000 7f428effd640 15 client.4669.objecter _session_command_op_remove 3 8 2024-03-27T21:40:43.048 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:43.049 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:43.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.062+0000 7f428cff9640 10 client.4669.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:43.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.062+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign -1 9 2024-03-27T21:40:43.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.062+0000 7f428cff9640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f42a01e4e70 0x7f42a01ec660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:43.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.062+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a01e4d00 osd=4 3 2024-03-27T21:40:43.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.062+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a01e4d00 osd=4 3 2024-03-27T21:40:43.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.062+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a01e4d00 osd=4 3 2024-03-27T21:40:43.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.062+0000 7f428cff9640 15 client.4669.objecter _session_command_op_remove -1 9 2024-03-27T21:40:43.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.062+0000 7f428cff9640 20 client.4669.objecter get_session s=0x7f42a01e4d00 osd=4 3 2024-03-27T21:40:43.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.062+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign 4 9 2024-03-27T21:40:43.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.062+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a01e4d00 osd=4 4 2024-03-27T21:40:43.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.062+0000 7f428cff9640 10 client.4669.objecter _send_command 9 2024-03-27T21:40:43.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.062+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f42a01e8920 con 0x7f42a01e4e70 2024-03-27T21:40:43.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.062+0000 7f429e59c640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f42a01e4e70 0x7f42a01ec660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:43.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.062+0000 7f429e59c640 10 monclient: get_auth_request con 0x7f42a01e4e70 auth_method 0 2024-03-27T21:40:43.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.063+0000 7f429e59c640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f42a01e4e70 0x7f42a01ec660 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:43.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.063+0000 7f428effd640 10 client.4669.objecter ms_handle_connect 0x7f42a01e4e70 2024-03-27T21:40:43.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.064+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f42a01e8920 con 0x7f42a01e4e70 2024-03-27T21:40:43.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.064+0000 7f428effd640 10 client.4669.objecter ms_dispatch 0x7f42a0000b90 command_reply(tid 9: 0 ) v1 2024-03-27T21:40:43.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.064+0000 7f428effd640 10 client.4669.objecter _finish_command 9 = system:0 2024-03-27T21:40:43.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.064+0000 7f428effd640 20 client.4669.objecter put_session s=0x7f42a01e4d00 osd=4 4 2024-03-27T21:40:43.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.064+0000 7f428effd640 15 client.4669.objecter _session_command_op_remove 4 9 2024-03-27T21:40:43.072 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:43.086+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:43.077 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:43.077 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:43.077 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:43.077 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:43.077 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.090+0000 7f428cff9640 10 client.4669.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.090+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign -1 10 2024-03-27T21:40:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.090+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a01e4d00 osd=4 3 2024-03-27T21:40:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.090+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a01e4d00 osd=4 3 2024-03-27T21:40:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.090+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a01e4d00 osd=4 3 2024-03-27T21:40:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.090+0000 7f428cff9640 15 client.4669.objecter _session_command_op_remove -1 10 2024-03-27T21:40:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.090+0000 7f428cff9640 20 client.4669.objecter get_session s=0x7f42a01e4d00 osd=4 3 2024-03-27T21:40:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.090+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign 4 10 2024-03-27T21:40:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.090+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a01e4d00 osd=4 4 2024-03-27T21:40:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.090+0000 7f428cff9640 10 client.4669.objecter _send_command 10 2024-03-27T21:40:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.090+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f42a01e8920 con 0x7f42a01e4e70 2024-03-27T21:40:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.092+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f42a01e8920 con 0x7f42a01e4e70 2024-03-27T21:40:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.092+0000 7f428effd640 10 client.4669.objecter ms_dispatch 0x7f42a0000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.092+0000 7f428effd640 10 client.4669.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.092+0000 7f428effd640 20 client.4669.objecter put_session s=0x7f42a01e4d00 osd=4 4 2024-03-27T21:40:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.092+0000 7f428effd640 15 client.4669.objecter _session_command_op_remove 4 10 2024-03-27T21:40:43.080 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.093+0000 7f428cff9640 10 client.4669.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.093+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign -1 11 2024-03-27T21:40:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.093+0000 7f428cff9640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f42a01ed0b0 0x7f42a01ed530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.093+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a01e8920 osd=5 3 2024-03-27T21:40:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.093+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a01e8920 osd=5 3 2024-03-27T21:40:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.093+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a01e8920 osd=5 3 2024-03-27T21:40:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.093+0000 7f428cff9640 15 client.4669.objecter _session_command_op_remove -1 11 2024-03-27T21:40:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.093+0000 7f428cff9640 20 client.4669.objecter get_session s=0x7f42a01e8920 osd=5 3 2024-03-27T21:40:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.093+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign 5 11 2024-03-27T21:40:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.093+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a01e8920 osd=5 4 2024-03-27T21:40:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.093+0000 7f428cff9640 10 client.4669.objecter _send_command 11 2024-03-27T21:40:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.093+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f42a01eda70 con 0x7f42a01ed0b0 2024-03-27T21:40:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.093+0000 7f429dd9b640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f42a01ed0b0 0x7f42a01ed530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.093+0000 7f429dd9b640 10 monclient: get_auth_request con 0x7f42a01ed0b0 auth_method 0 2024-03-27T21:40:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.094+0000 7f428effd640 10 client.4669.objecter ms_handle_connect 0x7f42a01ed0b0 2024-03-27T21:40:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.094+0000 7f429dd9b640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f42a01ed0b0 0x7f42a01ed530 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.094+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f42a01eda70 con 0x7f42a01ed0b0 2024-03-27T21:40:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.094+0000 7f428effd640 10 client.4669.objecter ms_dispatch 0x7f42a0000b90 command_reply(tid 11: 0 ) v1 2024-03-27T21:40:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.094+0000 7f428effd640 10 client.4669.objecter _finish_command 11 = system:0 2024-03-27T21:40:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.094+0000 7f428effd640 20 client.4669.objecter put_session s=0x7f42a01e8920 osd=5 4 2024-03-27T21:40:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.094+0000 7f428effd640 15 client.4669.objecter _session_command_op_remove 5 11 2024-03-27T21:40:43.089 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:40:43.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.120+0000 7f428cff9640 10 client.4669.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:43.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.120+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign -1 12 2024-03-27T21:40:43.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.120+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a01e8920 osd=5 3 2024-03-27T21:40:43.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.120+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a01e8920 osd=5 3 2024-03-27T21:40:43.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.120+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a01e8920 osd=5 3 2024-03-27T21:40:43.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.120+0000 7f428cff9640 15 client.4669.objecter _session_command_op_remove -1 12 2024-03-27T21:40:43.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.120+0000 7f428cff9640 20 client.4669.objecter get_session s=0x7f42a01e8920 osd=5 3 2024-03-27T21:40:43.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.120+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign 5 12 2024-03-27T21:40:43.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.120+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a01e8920 osd=5 4 2024-03-27T21:40:43.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.120+0000 7f428cff9640 10 client.4669.objecter _send_command 12 2024-03-27T21:40:43.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.120+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f42a01eda70 con 0x7f42a01ed0b0 2024-03-27T21:40:43.109 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:43.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.122+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f42a01eda70 con 0x7f42a01ed0b0 2024-03-27T21:40:43.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.122+0000 7f428effd640 10 client.4669.objecter ms_dispatch 0x7f42a0000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:43.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.122+0000 7f428effd640 10 client.4669.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:43.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.122+0000 7f428effd640 20 client.4669.objecter put_session s=0x7f42a01e8920 osd=5 4 2024-03-27T21:40:43.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.122+0000 7f428effd640 15 client.4669.objecter _session_command_op_remove 5 12 2024-03-27T21:40:43.110 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:43.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.122+0000 7f428cff9640 10 client.4669.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:43.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.122+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign -1 13 2024-03-27T21:40:43.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.122+0000 7f428cff9640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f42a01f2410 0x7f42a01f4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:43.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.122+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a01eda70 osd=6 3 2024-03-27T21:40:43.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.122+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a01eda70 osd=6 3 2024-03-27T21:40:43.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.122+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a01eda70 osd=6 3 2024-03-27T21:40:43.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.122+0000 7f428cff9640 15 client.4669.objecter _session_command_op_remove -1 13 2024-03-27T21:40:43.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.122+0000 7f428cff9640 20 client.4669.objecter get_session s=0x7f42a01eda70 osd=6 3 2024-03-27T21:40:43.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.122+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign 6 13 2024-03-27T21:40:43.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.122+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a01eda70 osd=6 4 2024-03-27T21:40:43.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.122+0000 7f428cff9640 10 client.4669.objecter _send_command 13 2024-03-27T21:40:43.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.122+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f42a01e8a90 con 0x7f42a01f2410 2024-03-27T21:40:43.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.122+0000 7f429ed9d640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f42a01f2410 0x7f42a01f4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:43.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.122+0000 7f429ed9d640 10 monclient: get_auth_request con 0x7f42a01f2410 auth_method 0 2024-03-27T21:40:43.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.123+0000 7f428effd640 10 client.4669.objecter ms_handle_connect 0x7f42a01f2410 2024-03-27T21:40:43.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.123+0000 7f429ed9d640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f42a01f2410 0x7f42a01f4800 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:43.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.124+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f42a01e8a90 con 0x7f42a01f2410 2024-03-27T21:40:43.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.124+0000 7f428effd640 10 client.4669.objecter ms_dispatch 0x7f42a0000b90 command_reply(tid 13: 0 ) v1 2024-03-27T21:40:43.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.124+0000 7f428effd640 10 client.4669.objecter _finish_command 13 = system:0 2024-03-27T21:40:43.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.124+0000 7f428effd640 20 client.4669.objecter put_session s=0x7f42a01eda70 osd=6 4 2024-03-27T21:40:43.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.124+0000 7f428effd640 15 client.4669.objecter _session_command_op_remove 6 13 2024-03-27T21:40:43.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.151+0000 7f428cff9640 10 client.4669.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:43.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.151+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign -1 14 2024-03-27T21:40:43.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.151+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a01eda70 osd=6 3 2024-03-27T21:40:43.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.151+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a01eda70 osd=6 3 2024-03-27T21:40:43.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.151+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a01eda70 osd=6 3 2024-03-27T21:40:43.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.151+0000 7f428cff9640 15 client.4669.objecter _session_command_op_remove -1 14 2024-03-27T21:40:43.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.151+0000 7f428cff9640 20 client.4669.objecter get_session s=0x7f42a01eda70 osd=6 3 2024-03-27T21:40:43.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.151+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign 6 14 2024-03-27T21:40:43.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.151+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a01eda70 osd=6 4 2024-03-27T21:40:43.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.151+0000 7f428cff9640 10 client.4669.objecter _send_command 14 2024-03-27T21:40:43.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.151+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f42a01e8a90 con 0x7f42a01f2410 2024-03-27T21:40:43.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.152+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f42a01e8a90 con 0x7f42a01f2410 2024-03-27T21:40:43.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.152+0000 7f428effd640 10 client.4669.objecter ms_dispatch 0x7f42a0000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:43.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.152+0000 7f428effd640 10 client.4669.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:43.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.152+0000 7f428effd640 20 client.4669.objecter put_session s=0x7f42a01eda70 osd=6 4 2024-03-27T21:40:43.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.153+0000 7f428effd640 15 client.4669.objecter _session_command_op_remove 6 14 2024-03-27T21:40:43.139 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:43.140 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:43.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.153+0000 7f428cff9640 10 client.4669.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:43.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.153+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign -1 15 2024-03-27T21:40:43.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.153+0000 7f428cff9640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f42a01f5250 0x7f42a01f7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:43.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.153+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a01e8a90 osd=7 3 2024-03-27T21:40:43.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.153+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a01e8a90 osd=7 3 2024-03-27T21:40:43.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.153+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a01e8a90 osd=7 3 2024-03-27T21:40:43.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.153+0000 7f428cff9640 15 client.4669.objecter _session_command_op_remove -1 15 2024-03-27T21:40:43.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.153+0000 7f428cff9640 20 client.4669.objecter get_session s=0x7f42a01e8a90 osd=7 3 2024-03-27T21:40:43.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.153+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign 7 15 2024-03-27T21:40:43.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.153+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a01e8a90 osd=7 4 2024-03-27T21:40:43.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.153+0000 7f428cff9640 10 client.4669.objecter _send_command 15 2024-03-27T21:40:43.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.153+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f42a01fa560 con 0x7f42a01f5250 2024-03-27T21:40:43.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.153+0000 7f429e59c640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f42a01f5250 0x7f42a01f7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:43.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.153+0000 7f429e59c640 10 monclient: get_auth_request con 0x7f42a01f5250 auth_method 0 2024-03-27T21:40:43.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.154+0000 7f429e59c640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f42a01f5250 0x7f42a01f7640 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:43.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.154+0000 7f428effd640 10 client.4669.objecter ms_handle_connect 0x7f42a01f5250 2024-03-27T21:40:43.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.155+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f42a01fa560 con 0x7f42a01f5250 2024-03-27T21:40:43.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.155+0000 7f428effd640 10 client.4669.objecter ms_dispatch 0x7f42a0000b90 command_reply(tid 15: 0 ) v1 2024-03-27T21:40:43.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.155+0000 7f428effd640 10 client.4669.objecter _finish_command 15 = system:0 2024-03-27T21:40:43.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.155+0000 7f428effd640 20 client.4669.objecter put_session s=0x7f42a01e8a90 osd=7 4 2024-03-27T21:40:43.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.155+0000 7f428effd640 15 client.4669.objecter _session_command_op_remove 7 15 2024-03-27T21:40:43.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.181+0000 7fd9685d2640 10 monclient: get_monmap_and_config 2024-03-27T21:40:43.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.181+0000 7fd9685d2640 10 monclient: build_initial_monmap 2024-03-27T21:40:43.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.181+0000 7fd9685d2640 10 monclient: monmap: 2024-03-27T21:40:43.168 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:43.168 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:43.168 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:43.182477+0000 2024-03-27T21:40:43.168 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:43.182477+0000 2024-03-27T21:40:43.168 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:43.168 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:43.168 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:43.168 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:43.168 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:43.168 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:43.168 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:43.168 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:43.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.182+0000 7fd9685d2640 1 Processor -- start 2024-03-27T21:40:43.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.182+0000 7fd9685d2640 1 -- start start 2024-03-27T21:40:43.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.182+0000 7fd9685d2640 10 monclient: init 2024-03-27T21:40:43.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd9685d2640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:43.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd9685d2640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-03-27T21:40:43.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd9685d2640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd9600b0ec0 0x7fd960142cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:43.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd9685d2640 10 monclient(hunting): picked mon.noname-b con 0x7fd9600b0ec0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:43.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd9685d2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd960143200 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:43.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd9685d2640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd9600b32c0 0x7fd960149710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:43.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd9685d2640 10 monclient(hunting): picked mon.noname-c con 0x7fd9600b32c0 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:43.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd9685d2640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96006d370 con 0x7fd960143200 2024-03-27T21:40:43.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd9685d2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:43.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd9685d2640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9600b3930 con 0x7fd9600b32c0 2024-03-27T21:40:43.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd9685d2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:43.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd9685d2640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9600b12a0 con 0x7fd9600b0ec0 2024-03-27T21:40:43.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd9685d2640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:43.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd9685d2640 10 monclient(hunting): authenticate will time out at 884.404111s 2024-03-27T21:40:43.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd96636e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd9600b0ec0 0x7fd960142cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:43.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd96636e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd9600b0ec0 0x7fd960142cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:57388/0 (socket says 172.21.15.60:57388) 2024-03-27T21:40:43.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd96636e640 1 -- 172.21.15.60:0/1898421075 learned_addr learned my addr 172.21.15.60:0/1898421075 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:43.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd96636e640 10 monclient(hunting): get_auth_request con 0x7fd9600b0ec0 auth_method 0 2024-03-27T21:40:43.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd96636e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:43.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd96636e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:43.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd96636e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:43.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd966b6f640 1 --2- 172.21.15.60:0/1898421075 >> v2:172.21.15.191:3300/0 conn(0x7fd9600b32c0 0x7fd960149710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:43.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd966b6f640 10 monclient(hunting): get_auth_request con 0x7fd9600b32c0 auth_method 0 2024-03-27T21:40:43.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd966b6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:43.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd966b6f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:43.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd966b6f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:43.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd96636e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:43.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd96636e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:43.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd96636e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:43.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd966b6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:43.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd966b6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:43.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd966b6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:43.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd96636e640 10 monclient(hunting): handle_auth_done global_id 4586 payload 274 2024-03-27T21:40:43.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd96636e640 1 -- 172.21.15.60:0/1898421075 >> v2:172.21.15.191:3300/0 conn(0x7fd9600b32c0 msgr2=0x7fd960149710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:43.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd96636e640 1 --2- 172.21.15.60:0/1898421075 >> v2:172.21.15.191:3300/0 conn(0x7fd9600b32c0 0x7fd960149710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd96636e640 1 -- 172.21.15.60:0/1898421075 >> v1:172.21.15.191:6789/0 conn(0x7fd960143200 legacy=0x7fd960149000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:43.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd96636e640 10 monclient: _finish_hunting 0 2024-03-27T21:40:43.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd96636e640 1 monclient: found mon.noname-b 2024-03-27T21:40:43.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd96636e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:43.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd96636e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:43.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.183+0000 7fd96636e640 1 -- 172.21.15.60:0/1898421075 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd960149c50 con 0x7fd9600b0ec0 2024-03-27T21:40:43.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.184+0000 7fd966b6f640 1 --2- 172.21.15.60:0/1898421075 >> v2:172.21.15.191:3300/0 conn(0x7fd9600b32c0 0x7fd960149710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:43.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.185+0000 7fd96636e640 1 --2- 172.21.15.60:0/1898421075 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd9600b0ec0 0x7fd960142cc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009870 tx=0x7fd95c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4cb64c2175dd0d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:43.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.185+0000 7fd96536c640 1 -- 172.21.15.60:0/1898421075 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c030840 con 0x7fd9600b0ec0 2024-03-27T21:40:43.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.185+0000 7fd96536c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:43.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.185+0000 7fd96536c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:43.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.185+0000 7fd96536c640 10 monclient: dump: 2024-03-27T21:40:43.177 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:43.177 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:43.177 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:43.177 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:43.177 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:43.177 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:43.177 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:43.177 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:43.177 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:43.177 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:43.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.185+0000 7fd96536c640 10 monclient: _finish_auth 0 2024-03-27T21:40:43.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.185+0000 7fd96536c640 10 monclient: _check_auth_tickets 2024-03-27T21:40:43.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.185+0000 7fd96536c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:43.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.185+0000 7fd96536c640 1 -- 172.21.15.60:0/1898421075 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95c0309a0 con 0x7fd9600b0ec0 2024-03-27T21:40:43.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.185+0000 7fd9685d2640 5 monclient: authenticate success, global_id 4586 2024-03-27T21:40:43.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.185+0000 7fd9685d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:43.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.185+0000 7fd96536c640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:43.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.185+0000 7fd96536c640 1 -- 172.21.15.60:0/1898421075 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c030d30 con 0x7fd9600b0ec0 2024-03-27T21:40:43.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.185+0000 7fd96536c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:43.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.185+0000 7fd96536c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:43.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.185+0000 7fd96536c640 10 monclient: dump: 2024-03-27T21:40:43.178 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:43.178 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:43.178 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:43.178 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:43.178 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:43.179 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:43.179 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:43.179 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:43.179 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:43.179 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:43.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.185+0000 7fd9685d2640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:43.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.185+0000 7fd9685d2640 4 set_mon_vals no callback set 2024-03-27T21:40:43.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.186+0000 7fd9685d2640 10 monclient: shutdown 2024-03-27T21:40:43.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.186+0000 7fd9685d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:43.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.186+0000 7fd9685d2640 1 -- 172.21.15.60:0/1898421075 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd9600b0ec0 msgr2=0x7fd960142cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:43.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.186+0000 7fd9685d2640 1 --2- 172.21.15.60:0/1898421075 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd9600b0ec0 0x7fd960142cc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009870 tx=0x7fd95c02b980 comp rx=0 tx=0).stop 2024-03-27T21:40:43.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.186+0000 7fd9685d2640 1 -- 172.21.15.60:0/1898421075 shutdown_connections 2024-03-27T21:40:43.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.186+0000 7fd9685d2640 1 --2- 172.21.15.60:0/1898421075 >> v2:172.21.15.191:3300/0 conn(0x7fd9600b32c0 0x7fd960149710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.186+0000 7fd9685d2640 1 --2- 172.21.15.60:0/1898421075 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd9600b0ec0 0x7fd960142cc0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.186+0000 7fd9685d2640 1 -- 172.21.15.60:0/1898421075 >> 172.21.15.60:0/1898421075 conn(0x7fd9600608e0 msgr2=0x7fd960060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:43.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.186+0000 7fd9685d2640 1 -- 172.21.15.60:0/1898421075 shutdown_connections 2024-03-27T21:40:43.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.186+0000 7fd9685d2640 1 -- 172.21.15.60:0/1898421075 wait complete. 2024-03-27T21:40:43.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.187+0000 7fd9685d2640 10 monclient: build_initial_monmap 2024-03-27T21:40:43.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.187+0000 7fd9685d2640 10 monclient: monmap: 2024-03-27T21:40:43.180 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:43.180 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:43.180 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:43.180 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:43.180 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:43.180 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:43.180 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:43.180 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:43.180 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:43.180 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:43.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.187+0000 7fd9685d2640 1 librados: starting msgr at 2024-03-27T21:40:43.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.187+0000 7fd9685d2640 1 librados: starting objecter 2024-03-27T21:40:43.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.187+0000 7fd9685d2640 1 Processor -- start 2024-03-27T21:40:43.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd9685d2640 1 -- start start 2024-03-27T21:40:43.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd9685d2640 1 librados: setting wanted keys 2024-03-27T21:40:43.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd9685d2640 1 librados: calling monclient init 2024-03-27T21:40:43.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd9685d2640 10 monclient: init 2024-03-27T21:40:43.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd9685d2640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:43.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd9685d2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:40:43.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd9685d2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9600b0ec0 0x7fd96013d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:43.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd9685d2640 10 monclient(hunting): picked mon.noname-b con 0x7fd9600b0ec0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:43.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd9685d2640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd9600b32c0 0x7fd9601dad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:43.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd9685d2640 10 monclient(hunting): picked mon.noname-c con 0x7fd9600b32c0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:43.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd9685d2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd960143200 0x7fd9601db250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:43.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7f428cff9640 10 client.4669.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:43.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign -1 16 2024-03-27T21:40:43.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a01e8a90 osd=7 3 2024-03-27T21:40:43.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a01e8a90 osd=7 3 2024-03-27T21:40:43.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7f428cff9640 20 client.4669.objecter _get_session s=0x7f42a01e8a90 osd=7 3 2024-03-27T21:40:43.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7f428cff9640 15 client.4669.objecter _session_command_op_remove -1 16 2024-03-27T21:40:43.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7f428cff9640 20 client.4669.objecter get_session s=0x7f42a01e8a90 osd=7 3 2024-03-27T21:40:43.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7f428cff9640 15 client.4669.objecter _session_command_op_assign 7 16 2024-03-27T21:40:43.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a01e8a90 osd=7 4 2024-03-27T21:40:43.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7f428cff9640 10 client.4669.objecter _send_command 16 2024-03-27T21:40:43.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f42a01f9330 con 0x7f42a01f5250 2024-03-27T21:40:43.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd9685d2640 10 monclient(hunting): picked mon.noname-a con 0x7fd960143200 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:43.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd9685d2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:43.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd9685d2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9601da7f0 con 0x7fd960143200 2024-03-27T21:40:43.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd9685d2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:43.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd9685d2640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd960145210 con 0x7fd9600b0ec0 2024-03-27T21:40:43.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd9685d2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:43.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd9685d2640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9601ddc90 con 0x7fd9600b32c0 2024-03-27T21:40:43.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd9685d2640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:43.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd9685d2640 10 monclient(hunting): authenticate will time out at 884.409591s 2024-03-27T21:40:43.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd965b6d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd9600b32c0 0x7fd9601dad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:43.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd966b6f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd960143200 0x7fd9601db250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:43.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd965b6d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd9600b32c0 0x7fd9601dad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:57390/0 (socket says 172.21.15.60:57390) 2024-03-27T21:40:43.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd965b6d640 1 -- 172.21.15.60:0/1081439290 learned_addr learned my addr 172.21.15.60:0/1081439290 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:43.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd965b6d640 10 monclient(hunting): get_auth_request con 0x7fd9600b32c0 auth_method 0 2024-03-27T21:40:43.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd965b6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:43.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd965b6d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:43.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd965b6d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:43.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd966b6f640 10 monclient(hunting): get_auth_request con 0x7fd960143200 auth_method 0 2024-03-27T21:40:43.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd966b6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:43.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd966b6f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:43.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.188+0000 7fd966b6f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:43.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.189+0000 7fd966b6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:43.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.189+0000 7fd966b6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:43.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.189+0000 7fd966b6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:43.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.189+0000 7fd965b6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:43.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.189+0000 7fd965b6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:43.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.189+0000 7fd965b6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:43.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.189+0000 7fd966b6f640 10 monclient(hunting): handle_auth_done global_id 4731 payload 824 2024-03-27T21:40:43.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.189+0000 7fd966b6f640 1 -- 172.21.15.60:0/1081439290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd9600b32c0 msgr2=0x7fd9601dad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:43.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.189+0000 7fd966b6f640 1 --2- 172.21.15.60:0/1081439290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd9600b32c0 0x7fd9601dad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.189+0000 7fd966b6f640 1 -- 172.21.15.60:0/1081439290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9600b0ec0 msgr2=0x7fd96013d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:43.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.189+0000 7fd96636e640 1 --2- 172.21.15.60:0/1081439290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9600b0ec0 0x7fd96013d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:43.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.189+0000 7fd966b6f640 1 --2- 172.21.15.60:0/1081439290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9600b0ec0 0x7fd96013d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.189+0000 7fd966b6f640 10 monclient: _finish_hunting 0 2024-03-27T21:40:43.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.189+0000 7fd966b6f640 1 monclient: found mon.noname-a 2024-03-27T21:40:43.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.189+0000 7fd966b6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:43.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.189+0000 7fd966b6f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:43.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.189+0000 7fd966b6f640 1 -- 172.21.15.60:0/1081439290 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd958009a70 con 0x7fd960143200 2024-03-27T21:40:43.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.189+0000 7fd965b6d640 1 --2- 172.21.15.60:0/1081439290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd9600b32c0 0x7fd9601dad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:43.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.189+0000 7fd966b6f640 1 --2- 172.21.15.60:0/1081439290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd960143200 0x7fd9601db250 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd95800efc0 tx=0x7fd95800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:43.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.189+0000 7fd956ffd640 10 client.?.objecter ms_handle_connect 0x7fd960143200 2024-03-27T21:40:43.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.189+0000 7fd956ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:43.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd956ffd640 1 -- 172.21.15.60:0/1081439290 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95800cd30 con 0x7fd960143200 2024-03-27T21:40:43.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd956ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:43.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd956ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:43.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd956ffd640 10 monclient: dump: 2024-03-27T21:40:43.191 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:43.191 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:43.191 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:43.191 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:43.191 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:43.191 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:43.191 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:43.191 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:43.192 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:43.192 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:43.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd956ffd640 10 monclient: _finish_auth 0 2024-03-27T21:40:43.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd956ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:40:43.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd956ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:43.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd956ffd640 1 -- 172.21.15.60:0/1081439290 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95800f040 con 0x7fd960143200 2024-03-27T21:40:43.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd956ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:43.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd964b6b640 4 set_mon_vals no callback set 2024-03-27T21:40:43.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd9685d2640 5 monclient: authenticate success, global_id 4731 2024-03-27T21:40:43.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7f428effd640 1 -- 172.21.15.60:0/976326530 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f42a01f9330 con 0x7f42a01f5250 2024-03-27T21:40:43.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7f428effd640 10 client.4669.objecter ms_dispatch 0x7f42a0000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:43.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7f428effd640 10 client.4669.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:43.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7f428effd640 20 client.4669.objecter put_session s=0x7f42a01e8a90 osd=7 4 2024-03-27T21:40:43.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7f428effd640 15 client.4669.objecter _session_command_op_remove 7 16 2024-03-27T21:40:43.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd9685d2640 10 monclient: _renew_subs 2024-03-27T21:40:43.193 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:43.194 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:43.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd9685d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:43.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd9685d2640 1 -- 172.21.15.60:0/1081439290 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95c0094d0 con 0x7fd960143200 2024-03-27T21:40:43.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd9685d2640 10 client.4731.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:43.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd9685d2640 10 monclient: _renew_subs 2024-03-27T21:40:43.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd9685d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:43.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd9685d2640 1 -- 172.21.15.60:0/1081439290 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96013dcb0 con 0x7fd960143200 2024-03-27T21:40:43.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd9685d2640 1 librados: init done 2024-03-27T21:40:43.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd956ffd640 1 -- 172.21.15.60:0/1081439290 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd958003bd0 con 0x7fd960143200 2024-03-27T21:40:43.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd956ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:43.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd956ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:43.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.190+0000 7fd956ffd640 10 monclient: dump: 2024-03-27T21:40:43.195 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:43.195 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:43.195 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:43.195 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:43.196 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:43.196 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:43.196 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:43.196 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:43.196 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:43.196 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:43.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 10 librados: watch_flush enter 2024-03-27T21:40:43.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 10 librados: watch_flush exit 2024-03-27T21:40:43.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 10 client.4669.objecter close_session for osd.0 2024-03-27T21:40:43.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f42a01d9770 msgr2=0x7f42a01d9b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:43.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f42a01d9770 0x7f42a01d9b50 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a00a90d0 osd=0 1 2024-03-27T21:40:43.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 10 client.4669.objecter close_session for osd.1 2024-03-27T21:40:43.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f42a01da2d0 msgr2=0x7f42a00b50d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:43.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f42a01da2d0 0x7f42a00b50d0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a014be90 osd=1 1 2024-03-27T21:40:43.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 10 client.4669.objecter close_session for osd.2 2024-03-27T21:40:43.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f42a0066250 msgr2=0x7f42a0066630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:43.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f42a0066250 0x7f42a0066630 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f429ed9d640 1 -- 172.21.15.60:0/976326530 reap_dead start 2024-03-27T21:40:43.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a00b5b90 osd=2 1 2024-03-27T21:40:43.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 10 client.4669.objecter close_session for osd.3 2024-03-27T21:40:43.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f42a01e5790 msgr2=0x7f42a01e8dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:43.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f42a01e5790 0x7f42a01e8dd0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a0066080 osd=3 1 2024-03-27T21:40:43.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 10 client.4669.objecter close_session for osd.4 2024-03-27T21:40:43.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f42a01e4e70 msgr2=0x7f42a01ec660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:43.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f42a01e4e70 0x7f42a01ec660 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a01e4d00 osd=4 1 2024-03-27T21:40:43.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 10 client.4669.objecter close_session for osd.5 2024-03-27T21:40:43.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f42a01ed0b0 msgr2=0x7f42a01ed530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:43.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f42a01ed0b0 0x7f42a01ed530 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a01e8920 osd=5 1 2024-03-27T21:40:43.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 10 client.4669.objecter close_session for osd.6 2024-03-27T21:40:43.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f42a01f2410 msgr2=0x7f42a01f4800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:43.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f42a01f2410 0x7f42a01f4800 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a01eda70 osd=6 1 2024-03-27T21:40:43.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 10 client.4669.objecter close_session for osd.7 2024-03-27T21:40:43.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f42a01f5250 msgr2=0x7f42a01f7640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:43.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f42a01f5250 0x7f42a01f7640 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f429ed9d640 1 -- 172.21.15.60:0/976326530 reap_dead start 2024-03-27T21:40:43.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 20 client.4669.objecter put_session s=0x7f42a01e8a90 osd=7 1 2024-03-27T21:40:43.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 20 client.4669.objecter shutdown clearing up homeless session... 2024-03-27T21:40:43.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 10 client.4669.objecter successfully canceled tick 2024-03-27T21:40:43.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f428006c5f0 msgr2=0x7f428006eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:43.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f428006c5f0 0x7f428006eab0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4290009870 tx=0x7f4290008030 comp rx=0 tx=0).stop 2024-03-27T21:40:43.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 10 monclient: shutdown 2024-03-27T21:40:43.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:43.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42a01416f0 msgr2=0x7f42a01d6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:43.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42a01416f0 0x7f42a01d6730 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f429800c760 tx=0x7f429800cb60 comp rx=0 tx=0).stop 2024-03-27T21:40:43.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7fd9685d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:43.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7fd9685d2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:43.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 shutdown_connections 2024-03-27T21:40:43.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.192+0000 7f428cff9640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f428006c5f0 0x7f428006eab0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.192+0000 7f428cff9640 1 --2- 172.21.15.60:0/976326530 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42a01416f0 0x7f42a01d6730 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.192+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 >> 172.21.15.60:0/976326530 conn(0x7f42a00608e0 msgr2=0x7f42a00b1950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:43.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7fd9685d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:43.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.191+0000 7fd9685d2640 1 -- 172.21.15.60:0/1081439290 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd96006f940 con 0x7fd960143200 2024-03-27T21:40:43.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.192+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 shutdown_connections 2024-03-27T21:40:43.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.192+0000 7f428cff9640 1 -- 172.21.15.60:0/976326530 wait complete. 2024-03-27T21:40:43.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.192+0000 7f428cff9640 1 librados: shutdown 2024-03-27T21:40:43.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.192+0000 7fd956ffd640 1 -- 172.21.15.60:0/1081439290 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd958014480 con 0x7fd960143200 2024-03-27T21:40:43.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.193+0000 7fd956ffd640 1 --2- 172.21.15.60:0/1081439290 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd94406c590 0x7fd94406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:43.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.193+0000 7fd956ffd640 1 -- 172.21.15.60:0/1081439290 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fd958089c90 con 0x7fd960143200 2024-03-27T21:40:43.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.193+0000 7fd956ffd640 10 client.4731.objecter ms_dispatch 0x7fd960000b90 osd_map(42..42 src has 1..42) v4 2024-03-27T21:40:43.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.193+0000 7fd956ffd640 3 client.4731.objecter handle_osd_map got epochs [42,42] > 0 2024-03-27T21:40:43.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.193+0000 7fd956ffd640 3 client.4731.objecter handle_osd_map decoding full epoch 42 2024-03-27T21:40:43.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.193+0000 7fd956ffd640 20 client.4731.objecter dump_active .. 0 homeless 2024-03-27T21:40:43.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.193+0000 7fd96636e640 1 --2- 172.21.15.60:0/1081439290 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd94406c590 0x7fd94406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:43.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.193+0000 7fd96636e640 10 monclient: get_auth_request con 0x7fd94406c590 auth_method 0 2024-03-27T21:40:43.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.194+0000 7fd956ffd640 10 client.4731.objecter ms_handle_connect 0x7fd94406c590 2024-03-27T21:40:43.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.194+0000 7fd96636e640 1 --2- 172.21.15.60:0/1081439290 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd94406c590 0x7fd94406ea50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009870 tx=0x7fd95c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:43.205 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:43.195+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:43.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.198+0000 7fd956ffd640 1 -- 172.21.15.60:0/1081439290 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd958055040 con 0x7fd960143200 2024-03-27T21:40:43.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.198+0000 7fd956ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:43.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.198+0000 7fd956ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:43.273 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:43.273 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:43.273 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:43.273 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-03-27T21:40:43.273 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.273 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:20 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2024-03-27T21:40:43.273 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:43.274 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 63.396702052999999, 2024-03-27T21:40:43.274 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015345630000000001, 2024-03-27T21:40:43.274 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:43.274 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:43.274 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:43.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.274 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:43.274 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:43.274 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.275 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:43.275 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:43.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.275 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:43.275 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888562+0000", 2024-03-27T21:40:43.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:43.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.275 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:43.275 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888735+0000", 2024-03-27T21:40:43.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017254799999999999 2024-03-27T21:40:43.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:43.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888747+0000", 2024-03-27T21:40:43.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2469999999999999e-05 2024-03-27T21:40:43.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:43.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888761+0000", 2024-03-27T21:40:43.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3254e-05 2024-03-27T21:40:43.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:43.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888804+0000", 2024-03-27T21:40:43.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3129e-05 2024-03-27T21:40:43.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:43.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888820+0000", 2024-03-27T21:40:43.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6426000000000002e-05 2024-03-27T21:40:43.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:43.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890060+0000", 2024-03-27T21:40:43.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012398050000000001 2024-03-27T21:40:43.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:43.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890098+0000", 2024-03-27T21:40:43.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7795e-05 2024-03-27T21:40:43.278 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.278 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.278 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.278 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2024-03-27T21:40:43.278 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:43.278 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 63.395877652999999, 2024-03-27T21:40:43.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001069555, 2024-03-27T21:40:43.278 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:43.278 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:43.279 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:43.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.279 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:43.279 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:43.279 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.279 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:43.279 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:43.279 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:43.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889387+0000", 2024-03-27T21:40:43.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:43.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:43.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889455+0000", 2024-03-27T21:40:43.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7930999999999997e-05 2024-03-27T21:40:43.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:43.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889470+0000", 2024-03-27T21:40:43.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5554e-05 2024-03-27T21:40:43.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:43.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889476+0000", 2024-03-27T21:40:43.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0619999999999997e-06 2024-03-27T21:40:43.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:43.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889513+0000", 2024-03-27T21:40:43.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6606999999999997e-05 2024-03-27T21:40:43.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:43.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889528+0000", 2024-03-27T21:40:43.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4982e-05 2024-03-27T21:40:43.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:43.282 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890421+0000", 2024-03-27T21:40:43.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00089293700000000001 2024-03-27T21:40:43.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:43.282 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890457+0000", 2024-03-27T21:40:43.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6260000000000002e-05 2024-03-27T21:40:43.282 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.282 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.282 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.283 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:38 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'24, mlcod=17'22)", 2024-03-27T21:40:43.283 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:43.283 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 63.378641686999998, 2024-03-27T21:40:43.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016182080000000001, 2024-03-27T21:40:43.283 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:43.283 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:43.283 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:43.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.283 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:43.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:43.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.283 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:43.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:43.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:43.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906623+0000", 2024-03-27T21:40:43.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:43.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:43.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906687+0000", 2024-03-27T21:40:43.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3706000000000003e-05 2024-03-27T21:40:43.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:43.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906698+0000", 2024-03-27T21:40:43.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1752e-05 2024-03-27T21:40:43.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:43.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906704+0000", 2024-03-27T21:40:43.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3940000000000004e-06 2024-03-27T21:40:43.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:43.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906741+0000", 2024-03-27T21:40:43.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6944999999999999e-05 2024-03-27T21:40:43.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.286 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:43.286 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906760+0000", 2024-03-27T21:40:43.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9293e-05 2024-03-27T21:40:43.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.286 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:43.286 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908211+0000", 2024-03-27T21:40:43.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014511350000000001 2024-03-27T21:40:43.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:43.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908242+0000", 2024-03-27T21:40:43.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0712e-05 2024-03-27T21:40:43.287 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.287 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.287 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.287 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:39 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'25, mlcod=17'22)", 2024-03-27T21:40:43.287 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:43.287 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 63.377796772000004, 2024-03-27T21:40:43.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012060199999999999, 2024-03-27T21:40:43.287 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:43.288 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:43.288 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:43.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.288 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:43.288 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:43.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.288 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:43.288 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:43.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:43.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:43.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:43.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:43.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907533+0000", 2024-03-27T21:40:43.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5675999999999999e-05 2024-03-27T21:40:43.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:43.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907545+0000", 2024-03-27T21:40:43.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.115e-05 2024-03-27T21:40:43.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.290 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:43.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907550+0000", 2024-03-27T21:40:43.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8320000000000002e-06 2024-03-27T21:40:43.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.290 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:43.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907592+0000", 2024-03-27T21:40:43.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1309e-05 2024-03-27T21:40:43.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.290 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:43.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907611+0000", 2024-03-27T21:40:43.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9009e-05 2024-03-27T21:40:43.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:43.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908638+0000", 2024-03-27T21:40:43.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001027533 2024-03-27T21:40:43.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:43.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908675+0000", 2024-03-27T21:40:43.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6329999999999999e-05 2024-03-27T21:40:43.291 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.291 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.292 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.292 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:57 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'35, mlcod=17'33)", 2024-03-27T21:40:43.292 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:43.292 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 63.363712339999999, 2024-03-27T21:40:43.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001346578, 2024-03-27T21:40:43.292 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:43.292 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:43.292 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:43.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:43.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:43.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:43.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:43.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:43.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921552+0000", 2024-03-27T21:40:43.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:43.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:43.294 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921615+0000", 2024-03-27T21:40:43.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2787999999999996e-05 2024-03-27T21:40:43.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.294 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:43.294 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921626+0000", 2024-03-27T21:40:43.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0888e-05 2024-03-27T21:40:43.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.294 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:43.294 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921635+0000", 2024-03-27T21:40:43.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4639999999999995e-06 2024-03-27T21:40:43.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:43.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921673+0000", 2024-03-27T21:40:43.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7891e-05 2024-03-27T21:40:43.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:43.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921694+0000", 2024-03-27T21:40:43.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0426000000000001e-05 2024-03-27T21:40:43.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:43.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922865+0000", 2024-03-27T21:40:43.296 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011709509999999999 2024-03-27T21:40:43.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.296 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:43.296 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922900+0000", 2024-03-27T21:40:43.296 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4819e-05 2024-03-27T21:40:43.296 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.296 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.296 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.296 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:71 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'45, mlcod=17'43)", 2024-03-27T21:40:43.296 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:43.296 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 63.348413628000003, 2024-03-27T21:40:43.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001460907, 2024-03-27T21:40:43.297 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:43.297 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:43.297 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:43.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:43.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:43.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:43.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:43.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:43.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936851+0000", 2024-03-27T21:40:43.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:43.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:43.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936903+0000", 2024-03-27T21:40:43.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2302999999999997e-05 2024-03-27T21:40:43.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:43.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936915+0000", 2024-03-27T21:40:43.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1967000000000001e-05 2024-03-27T21:40:43.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:43.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936923+0000", 2024-03-27T21:40:43.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9370000000000004e-06 2024-03-27T21:40:43.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:43.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936970+0000", 2024-03-27T21:40:43.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6987000000000003e-05 2024-03-27T21:40:43.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:43.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937002+0000", 2024-03-27T21:40:43.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2410999999999999e-05 2024-03-27T21:40:43.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:43.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938279+0000", 2024-03-27T21:40:43.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001276552 2024-03-27T21:40:43.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:43.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938313+0000", 2024-03-27T21:40:43.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3526999999999999e-05 2024-03-27T21:40:43.301 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.301 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.301 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.301 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:72 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'46, mlcod=17'43)", 2024-03-27T21:40:43.301 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:43.301 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 63.347776066000002, 2024-03-27T21:40:43.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014239490000000001, 2024-03-27T21:40:43.301 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:43.301 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:43.301 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:43.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:43.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:43.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:43.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:43.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:43.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937488+0000", 2024-03-27T21:40:43.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:43.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:43.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937547+0000", 2024-03-27T21:40:43.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8196999999999999e-05 2024-03-27T21:40:43.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:43.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937559+0000", 2024-03-27T21:40:43.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1928e-05 2024-03-27T21:40:43.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:43.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937564+0000", 2024-03-27T21:40:43.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4759999999999997e-06 2024-03-27T21:40:43.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:43.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937621+0000", 2024-03-27T21:40:43.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6712000000000002e-05 2024-03-27T21:40:43.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:43.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937640+0000", 2024-03-27T21:40:43.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9517e-05 2024-03-27T21:40:43.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:43.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938883+0000", 2024-03-27T21:40:43.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001242567 2024-03-27T21:40:43.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:43.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938913+0000", 2024-03-27T21:40:43.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0287e-05 2024-03-27T21:40:43.305 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.305 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.305 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.305 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:73 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'47, mlcod=17'43)", 2024-03-27T21:40:43.305 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:43.305 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 63.347099972000002, 2024-03-27T21:40:43.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010746239999999999, 2024-03-27T21:40:43.306 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:43.306 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:43.306 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:43.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:43.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:43.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:43.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:43.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:43.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:43.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:43.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:43.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938217+0000", 2024-03-27T21:40:43.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2743e-05 2024-03-27T21:40:43.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:43.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938229+0000", 2024-03-27T21:40:43.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1956999999999999e-05 2024-03-27T21:40:43.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:43.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938235+0000", 2024-03-27T21:40:43.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4589999999999997e-06 2024-03-27T21:40:43.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:43.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938341+0000", 2024-03-27T21:40:43.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000106054 2024-03-27T21:40:43.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:43.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938369+0000", 2024-03-27T21:40:43.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8291000000000001e-05 2024-03-27T21:40:43.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:43.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939209+0000", 2024-03-27T21:40:43.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084045400000000003 2024-03-27T21:40:43.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:43.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939240+0000", 2024-03-27T21:40:43.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0429999999999998e-05 2024-03-27T21:40:43.309 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.309 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.309 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.309 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:75 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'49, mlcod=17'47)", 2024-03-27T21:40:43.309 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:43.309 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 63.340662324999997, 2024-03-27T21:40:43.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001017001, 2024-03-27T21:40:43.310 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:43.310 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:43.310 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:43.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:43.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:43.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:43.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:43.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:43.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944601+0000", 2024-03-27T21:40:43.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999986 2024-03-27T21:40:43.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:43.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944614+0000", 2024-03-27T21:40:43.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2496e-05 2024-03-27T21:40:43.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:43.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944640+0000", 2024-03-27T21:40:43.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5865999999999999e-05 2024-03-27T21:40:43.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:43.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944651+0000", 2024-03-27T21:40:43.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1797000000000001e-05 2024-03-27T21:40:43.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:43.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944762+0000", 2024-03-27T21:40:43.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110412 2024-03-27T21:40:43.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:43.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944820+0000", 2024-03-27T21:40:43.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7649999999999999e-05 2024-03-27T21:40:43.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:43.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945585+0000", 2024-03-27T21:40:43.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076562600000000005 2024-03-27T21:40:43.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:43.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945620+0000", 2024-03-27T21:40:43.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4764000000000001e-05 2024-03-27T21:40:43.313 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.313 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.313 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.314 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:86 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'50)", 2024-03-27T21:40:43.314 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:43.314 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 63.332900838, 2024-03-27T21:40:43.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011392380000000001, 2024-03-27T21:40:43.314 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:43.314 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:43.314 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:43.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:43.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:43.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:43.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:43.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:43.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:43.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:43.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:43.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952373+0000", 2024-03-27T21:40:43.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9220000000000003e-06 2024-03-27T21:40:43.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:43.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952381+0000", 2024-03-27T21:40:43.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3070000000000003e-06 2024-03-27T21:40:43.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:43.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952385+0000", 2024-03-27T21:40:43.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.121e-06 2024-03-27T21:40:43.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:43.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952419+0000", 2024-03-27T21:40:43.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4082999999999999e-05 2024-03-27T21:40:43.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.317 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:43.317 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952440+0000", 2024-03-27T21:40:43.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0837999999999999e-05 2024-03-27T21:40:43.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.317 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:43.317 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953472+0000", 2024-03-27T21:40:43.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001032119 2024-03-27T21:40:43.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:43.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953504+0000", 2024-03-27T21:40:43.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1250000000000001e-05 2024-03-27T21:40:43.318 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.318 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.318 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.318 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:87 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-27T21:40:43.318 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:43.318 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 63.332533581, 2024-03-27T21:40:43.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00135519, 2024-03-27T21:40:43.318 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:43.319 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:43.319 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:43.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:43.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:43.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:43.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:43.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:43.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952731+0000", 2024-03-27T21:40:43.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:43.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:43.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952781+0000", 2024-03-27T21:40:43.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0161000000000003e-05 2024-03-27T21:40:43.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:43.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952794+0000", 2024-03-27T21:40:43.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2853e-05 2024-03-27T21:40:43.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:43.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952799+0000", 2024-03-27T21:40:43.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5560000000000001e-06 2024-03-27T21:40:43.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:43.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952832+0000", 2024-03-27T21:40:43.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3834999999999997e-05 2024-03-27T21:40:43.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:43.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952851+0000", 2024-03-27T21:40:43.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8289999999999999e-05 2024-03-27T21:40:43.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.322 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:43.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954055+0000", 2024-03-27T21:40:43.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012037949999999999 2024-03-27T21:40:43.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.322 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:43.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954087+0000", 2024-03-27T21:40:43.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2292999999999997e-05 2024-03-27T21:40:43.322 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.322 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.322 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.323 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:89 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'59, mlcod=17'57)", 2024-03-27T21:40:43.323 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:43.323 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 63.330259359999999, 2024-03-27T21:40:43.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00138893, 2024-03-27T21:40:43.323 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:43.323 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:43.323 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:43.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:43.323 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:43.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:43.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:43.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:43.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955005+0000", 2024-03-27T21:40:43.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:43.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:43.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955071+0000", 2024-03-27T21:40:43.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5290999999999995e-05 2024-03-27T21:40:43.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:43.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955081+0000", 2024-03-27T21:40:43.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0572e-05 2024-03-27T21:40:43.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:43.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955086+0000", 2024-03-27T21:40:43.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8380000000000001e-06 2024-03-27T21:40:43.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:43.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955122+0000", 2024-03-27T21:40:43.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5698000000000003e-05 2024-03-27T21:40:43.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:43.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955140+0000", 2024-03-27T21:40:43.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7900000000000001e-05 2024-03-27T21:40:43.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:43.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956366+0000", 2024-03-27T21:40:43.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001226471 2024-03-27T21:40:43.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:43.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956395+0000", 2024-03-27T21:40:43.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8782e-05 2024-03-27T21:40:43.327 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.327 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.327 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.327 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:90 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'60, mlcod=17'57)", 2024-03-27T21:40:43.327 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:43.327 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 63.329574782999998, 2024-03-27T21:40:43.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010778859999999999, 2024-03-27T21:40:43.327 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:43.327 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:43.327 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:43.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:43.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:43.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:43.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:43.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:43.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:43.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:43.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:43.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955747+0000", 2024-03-27T21:40:43.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7130999999999999e-05 2024-03-27T21:40:43.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:43.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955756+0000", 2024-03-27T21:40:43.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2750000000000001e-06 2024-03-27T21:40:43.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:43.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955761+0000", 2024-03-27T21:40:43.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3900000000000003e-06 2024-03-27T21:40:43.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:43.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955797+0000", 2024-03-27T21:40:43.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6650000000000003e-05 2024-03-27T21:40:43.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:43.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955816+0000", 2024-03-27T21:40:43.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8195000000000001e-05 2024-03-27T21:40:43.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:43.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956739+0000", 2024-03-27T21:40:43.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00092369700000000002 2024-03-27T21:40:43.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.331 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:43.331 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956768+0000", 2024-03-27T21:40:43.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9048e-05 2024-03-27T21:40:43.331 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.331 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.331 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.331 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:118 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'80, mlcod=17'78)", 2024-03-27T21:40:43.331 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:43.331 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 63.196583959999998, 2024-03-27T21:40:43.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014200669999999999, 2024-03-27T21:40:43.332 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:43.332 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:43.332 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:43.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.332 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:43.332 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:43.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.332 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:43.332 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:43.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.332 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:43.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:43.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:43.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:43.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088756+0000", 2024-03-27T21:40:43.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4902999999999998e-05 2024-03-27T21:40:43.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:43.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088764+0000", 2024-03-27T21:40:43.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8169999999999999e-06 2024-03-27T21:40:43.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:43.334 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088767+0000", 2024-03-27T21:40:43.334 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8310000000000003e-06 2024-03-27T21:40:43.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.334 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.334 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:43.334 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088798+0000", 2024-03-27T21:40:43.334 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0800999999999999e-05 2024-03-27T21:40:43.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.334 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.334 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:43.334 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088817+0000", 2024-03-27T21:40:43.334 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8467000000000001e-05 2024-03-27T21:40:43.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.335 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:43.335 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090074+0000", 2024-03-27T21:40:43.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012574089999999999 2024-03-27T21:40:43.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.335 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:43.335 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090101+0000", 2024-03-27T21:40:43.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7223e-05 2024-03-27T21:40:43.335 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.335 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.336 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.336 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:119 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'81, mlcod=17'78)", 2024-03-27T21:40:43.336 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:43.336 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 63.195849164999998, 2024-03-27T21:40:43.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014990870000000001, 2024-03-27T21:40:43.336 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:43.336 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:43.336 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:43.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.336 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:43.336 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:43.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.337 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:43.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:43.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.337 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:43.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:43.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:43.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.337 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:43.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089491+0000", 2024-03-27T21:40:43.338 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4805e-05 2024-03-27T21:40:43.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.338 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:43.338 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089498+0000", 2024-03-27T21:40:43.338 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0650000000000001e-06 2024-03-27T21:40:43.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.338 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:43.338 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089501+0000", 2024-03-27T21:40:43.338 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.54e-06 2024-03-27T21:40:43.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:43.339 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089533+0000", 2024-03-27T21:40:43.339 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1838999999999998e-05 2024-03-27T21:40:43.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:43.339 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089550+0000", 2024-03-27T21:40:43.339 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7465999999999998e-05 2024-03-27T21:40:43.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:43.339 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090890+0000", 2024-03-27T21:40:43.339 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013397979999999999 2024-03-27T21:40:43.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.340 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:43.340 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090915+0000", 2024-03-27T21:40:43.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4905e-05 2024-03-27T21:40:43.340 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.340 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.340 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.340 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:120 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'82, mlcod=17'78)", 2024-03-27T21:40:43.340 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:43.340 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 63.195049285000003, 2024-03-27T21:40:43.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001527093, 2024-03-27T21:40:43.340 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:43.340 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:43.341 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:43.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:43.341 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:43.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:43.341 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:43.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:43.342 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:43.342 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:43.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.342 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:43.342 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090267+0000", 2024-03-27T21:40:43.342 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1477000000000002e-05 2024-03-27T21:40:43.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.342 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:43.342 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090273+0000", 2024-03-27T21:40:43.342 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9519999999999999e-06 2024-03-27T21:40:43.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.343 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:43.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090276+0000", 2024-03-27T21:40:43.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.219e-06 2024-03-27T21:40:43.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.343 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:43.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090304+0000", 2024-03-27T21:40:43.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8073999999999999e-05 2024-03-27T21:40:43.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.343 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:43.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090316+0000", 2024-03-27T21:40:43.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2088e-05 2024-03-27T21:40:43.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:43.344 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091717+0000", 2024-03-27T21:40:43.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014003889999999999 2024-03-27T21:40:43.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:43.344 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091743+0000", 2024-03-27T21:40:43.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6234e-05 2024-03-27T21:40:43.344 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.344 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.344 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.345 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:121 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'83, mlcod=17'78)", 2024-03-27T21:40:43.345 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:43.345 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 63.194238919999997, 2024-03-27T21:40:43.345 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010226899999999999, 2024-03-27T21:40:43.345 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:43.345 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:43.345 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:43.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.345 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:43.345 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:43.345 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.346 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:43.346 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:43.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.346 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:43.346 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:43.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:43.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.346 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:43.346 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091142+0000", 2024-03-27T21:40:43.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011608300000000001 2024-03-27T21:40:43.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.347 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:43.347 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091151+0000", 2024-03-27T21:40:43.347 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2350000000000005e-06 2024-03-27T21:40:43.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.347 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:43.347 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091156+0000", 2024-03-27T21:40:43.347 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5419999999999999e-06 2024-03-27T21:40:43.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.347 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:43.347 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091191+0000", 2024-03-27T21:40:43.347 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5453000000000001e-05 2024-03-27T21:40:43.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.348 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:43.348 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091208+0000", 2024-03-27T21:40:43.348 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7119999999999999e-05 2024-03-27T21:40:43.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.348 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:43.348 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092024+0000", 2024-03-27T21:40:43.348 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000815539 2024-03-27T21:40:43.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.348 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:43.348 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092049+0000", 2024-03-27T21:40:43.349 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5171e-05 2024-03-27T21:40:43.349 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.349 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.349 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.349 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.349 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:142 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'100, mlcod=18'98)", 2024-03-27T21:40:43.349 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:43.349 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 62.779231965999998, 2024-03-27T21:40:43.349 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001198778, 2024-03-27T21:40:43.349 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:43.349 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:43.349 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:43.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.350 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:43.350 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:43.350 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.350 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:43.350 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:43.350 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.350 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:43.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:43.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:43.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.351 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:43.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506101+0000", 2024-03-27T21:40:43.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8202999999999997e-05 2024-03-27T21:40:43.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.351 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:43.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506109+0000", 2024-03-27T21:40:43.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3059999999999998e-06 2024-03-27T21:40:43.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.352 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:43.352 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506113+0000", 2024-03-27T21:40:43.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0119999999999999e-06 2024-03-27T21:40:43.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.352 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:43.352 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506137+0000", 2024-03-27T21:40:43.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4043999999999999e-05 2024-03-27T21:40:43.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.352 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:43.352 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506148+0000", 2024-03-27T21:40:43.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0446e-05 2024-03-27T21:40:43.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.353 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:43.353 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507213+0000", 2024-03-27T21:40:43.353 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001065215 2024-03-27T21:40:43.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.353 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:43.353 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507232+0000", 2024-03-27T21:40:43.353 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9038e-05 2024-03-27T21:40:43.353 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.353 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.353 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.354 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:143 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'101, mlcod=18'98)", 2024-03-27T21:40:43.354 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:43.354 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 62.778619347000003, 2024-03-27T21:40:43.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001357488, 2024-03-27T21:40:43.354 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:43.354 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:43.354 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:43.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.354 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:43.354 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:43.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:43.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:43.355 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:43.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:43.355 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:43.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:43.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506723+0000", 2024-03-27T21:40:43.355 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7404000000000006e-05 2024-03-27T21:40:43.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.356 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:43.356 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506730+0000", 2024-03-27T21:40:43.356 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5749999999999997e-06 2024-03-27T21:40:43.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.356 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:43.356 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506733+0000", 2024-03-27T21:40:43.356 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7979999999999999e-06 2024-03-27T21:40:43.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.356 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:43.357 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506758+0000", 2024-03-27T21:40:43.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4179e-05 2024-03-27T21:40:43.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.357 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:43.357 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506768+0000", 2024-03-27T21:40:43.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0740000000000001e-05 2024-03-27T21:40:43.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.357 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:43.357 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507978+0000", 2024-03-27T21:40:43.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012093900000000001 2024-03-27T21:40:43.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.358 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:43.358 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508003+0000", 2024-03-27T21:40:43.358 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5667e-05 2024-03-27T21:40:43.358 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.358 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.358 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.358 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.358 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.358 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:144 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'102, mlcod=18'98)", 2024-03-27T21:40:43.358 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:43.358 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 62.777960450000002, 2024-03-27T21:40:43.358 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013453390000000001, 2024-03-27T21:40:43.358 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:43.358 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:43.359 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:43.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.359 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:43.359 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:43.359 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.359 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.359 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:43.359 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:43.359 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:43.359 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.359 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:43.359 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507304+0000", 2024-03-27T21:40:43.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:43.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.360 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:43.360 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507397+0000", 2024-03-27T21:40:43.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2995e-05 2024-03-27T21:40:43.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.360 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:43.360 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507403+0000", 2024-03-27T21:40:43.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8640000000000001e-06 2024-03-27T21:40:43.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.360 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:43.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507407+0000", 2024-03-27T21:40:43.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5710000000000002e-06 2024-03-27T21:40:43.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.361 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:43.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507428+0000", 2024-03-27T21:40:43.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1192e-05 2024-03-27T21:40:43.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.361 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:43.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507438+0000", 2024-03-27T21:40:43.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8309999999999996e-06 2024-03-27T21:40:43.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.362 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:43.362 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508624+0000", 2024-03-27T21:40:43.362 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011861210000000001 2024-03-27T21:40:43.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.362 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:43.362 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508650+0000", 2024-03-27T21:40:43.362 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6118e-05 2024-03-27T21:40:43.362 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.362 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.362 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.362 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.362 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.363 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:43.364 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:40:43.365 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:43.377+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:43.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.464+0000 7fd9685d2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json-pretty"}] 2024-03-27T21:40:43.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.464+0000 7fd9685d2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json-pretty"}] 2024-03-27T21:40:43.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.464+0000 7fd9685d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:43.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.464+0000 7fd9685d2640 1 -- 172.21.15.60:0/1081439290 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fd960069470 con 0x7fd960143200 2024-03-27T21:40:43.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.465+0000 7fd956ffd640 1 -- 172.21.15.60:0/1081439290 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v42) v1 ==== 81+0+30229 (secure 0 0 0) 0x7fd958058660 con 0x7fd960143200 2024-03-27T21:40:43.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.465+0000 7fd956ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json-pretty"}] 2024-03-27T21:40:43.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.465+0000 7fd956ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:40:43.452 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:40:43.452 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:43.452 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 42, 2024-03-27T21:40:43.453 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "dcd19d9a-535c-4b41-9095-d588268e2860", 2024-03-27T21:40:43.453 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2024-03-27T21:39:27.092191+0000", 2024-03-27T21:40:43.453 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2024-03-27T21:40:41.459857+0000", 2024-03-27T21:40:43.453 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2024-03-27T21:40:20.406185+0000", 2024-03-27T21:40:43.453 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2024-03-27T21:39:29.291620+0000", 2024-03-27T21:40:43.453 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-27T21:40:43.453 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5801984, 2024-03-27T21:40:43.453 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2024-03-27T21:40:43.453 INFO:teuthology.orchestra.run.smithi060.stdout: "noscrub", 2024-03-27T21:40:43.453 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2024-03-27T21:40:43.454 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2024-03-27T21:40:43.454 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2024-03-27T21:40:43.454 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2024-03-27T21:40:43.454 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-27T21:40:43.454 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 6, 2024-03-27T21:40:43.454 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2024-03-27T21:40:43.454 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-27T21:40:43.454 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-27T21:40:43.454 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2024-03-27T21:40:43.454 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 2, 2024-03-27T21:40:43.454 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2024-03-27T21:40:43.454 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "luminous", 2024-03-27T21:40:43.454 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2024-03-27T21:40:43.454 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "reef", 2024-03-27T21:40:43.455 INFO:teuthology.orchestra.run.smithi060.stdout: "allow_crimson": false, 2024-03-27T21:40:43.455 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2024-03-27T21:40:43.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.455 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-03-27T21:40:43.455 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": ".mgr", 2024-03-27T21:40:43.455 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2024-03-27T21:39:37.438493+0000", 2024-03-27T21:40:43.455 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-27T21:40:43.455 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2024-03-27T21:40:43.455 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2024-03-27T21:40:43.455 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2024-03-27T21:40:43.455 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2024-03-27T21:40:43.456 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2024-03-27T21:40:43.456 INFO:teuthology.orchestra.run.smithi060.stdout: "peering_crush_bucket_count": 0, 2024-03-27T21:40:43.456 INFO:teuthology.orchestra.run.smithi060.stdout: "peering_crush_bucket_target": 0, 2024-03-27T21:40:43.456 INFO:teuthology.orchestra.run.smithi060.stdout: "peering_crush_bucket_barrier": 0, 2024-03-27T21:40:43.456 INFO:teuthology.orchestra.run.smithi060.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-27T21:40:43.456 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2024-03-27T21:40:43.456 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2024-03-27T21:40:43.456 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2024-03-27T21:40:43.456 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2024-03-27T21:40:43.456 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2024-03-27T21:40:43.456 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2024-03-27T21:40:43.456 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2024-03-27T21:40:43.456 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2024-03-27T21:40:43.456 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2024-03-27T21:40:43.457 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2024-03-27T21:40:43.457 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2024-03-27T21:40:43.457 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2024-03-27T21:40:43.457 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2024-03-27T21:40:43.457 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2024-03-27T21:40:43.457 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.457 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2024-03-27T21:40:43.457 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2024-03-27T21:40:43.457 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-27T21:40:43.457 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2024-03-27T21:40:43.457 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2024-03-27T21:40:43.457 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2024-03-27T21:40:43.457 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2024-03-27T21:40:43.457 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2024-03-27T21:40:43.457 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2024-03-27T21:40:43.458 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2024-03-27T21:40:43.458 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2024-03-27T21:40:43.458 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2024-03-27T21:40:43.458 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2024-03-27T21:40:43.458 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2024-03-27T21:40:43.458 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2024-03-27T21:40:43.458 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2024-03-27T21:40:43.458 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2024-03-27T21:40:43.458 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2024-03-27T21:40:43.458 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2024-03-27T21:40:43.458 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-27T21:40:43.458 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-27T21:40:43.458 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-27T21:40:43.459 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2024-03-27T21:40:43.459 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2024-03-27T21:40:43.459 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2024-03-27T21:40:43.459 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2024-03-27T21:40:43.459 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2024-03-27T21:40:43.459 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.459 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2024-03-27T21:40:43.459 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2024-03-27T21:40:43.459 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2024-03-27T21:40:43.459 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2024-03-27T21:40:43.459 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2024-03-27T21:40:43.459 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2024-03-27T21:40:43.459 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2024-03-27T21:40:43.460 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2024-03-27T21:40:43.460 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2024-03-27T21:40:43.460 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2024-03-27T21:40:43.460 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2024-03-27T21:40:43.460 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2024-03-27T21:40:43.460 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_max": 32, 2024-03-27T21:40:43.460 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2024-03-27T21:40:43.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.460 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2024-03-27T21:40:43.460 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": {} 2024-03-27T21:40:43.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.460 INFO:teuthology.orchestra.run.smithi060.stdout: "read_balance": { 2024-03-27T21:40:43.460 INFO:teuthology.orchestra.run.smithi060.stdout: "score_acting": 8, 2024-03-27T21:40:43.461 INFO:teuthology.orchestra.run.smithi060.stdout: "score_stable": 8, 2024-03-27T21:40:43.461 INFO:teuthology.orchestra.run.smithi060.stdout: "optimal_score": 0.25, 2024-03-27T21:40:43.461 INFO:teuthology.orchestra.run.smithi060.stdout: "raw_score_acting": 2, 2024-03-27T21:40:43.461 INFO:teuthology.orchestra.run.smithi060.stdout: "raw_score_stable": 2, 2024-03-27T21:40:43.461 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity_weighted": 1, 2024-03-27T21:40:43.461 INFO:teuthology.orchestra.run.smithi060.stdout: "average_primary_affinity": 1, 2024-03-27T21:40:43.461 INFO:teuthology.orchestra.run.smithi060.stdout: "average_primary_affinity_weighted": 1 2024-03-27T21:40:43.461 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.461 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.461 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.461 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2024-03-27T21:40:43.461 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2024-03-27T21:40:43.461 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2024-03-27T21:39:45.042103+0000", 2024-03-27T21:40:43.462 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-27T21:40:43.462 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2024-03-27T21:40:43.462 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2024-03-27T21:40:43.462 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2024-03-27T21:40:43.462 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2024-03-27T21:40:43.462 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2024-03-27T21:40:43.462 INFO:teuthology.orchestra.run.smithi060.stdout: "peering_crush_bucket_count": 0, 2024-03-27T21:40:43.462 INFO:teuthology.orchestra.run.smithi060.stdout: "peering_crush_bucket_target": 0, 2024-03-27T21:40:43.462 INFO:teuthology.orchestra.run.smithi060.stdout: "peering_crush_bucket_barrier": 0, 2024-03-27T21:40:43.462 INFO:teuthology.orchestra.run.smithi060.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-27T21:40:43.462 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2024-03-27T21:40:43.462 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2024-03-27T21:40:43.462 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2024-03-27T21:40:43.463 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2024-03-27T21:40:43.463 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2024-03-27T21:40:43.463 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2024-03-27T21:40:43.463 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2024-03-27T21:40:43.463 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2024-03-27T21:40:43.463 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2024-03-27T21:40:43.463 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2024-03-27T21:40:43.463 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2024-03-27T21:40:43.463 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2024-03-27T21:40:43.463 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2024-03-27T21:40:43.463 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2024-03-27T21:40:43.463 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.464 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "21", 2024-03-27T21:40:43.464 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2024-03-27T21:40:43.464 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-27T21:40:43.464 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2024-03-27T21:40:43.464 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2024-03-27T21:40:43.464 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2024-03-27T21:40:43.464 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2024-03-27T21:40:43.464 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2024-03-27T21:40:43.464 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2024-03-27T21:40:43.464 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2024-03-27T21:40:43.464 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2024-03-27T21:40:43.464 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2024-03-27T21:40:43.465 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2024-03-27T21:40:43.465 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2024-03-27T21:40:43.465 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2024-03-27T21:40:43.465 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2024-03-27T21:40:43.465 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2024-03-27T21:40:43.465 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2024-03-27T21:40:43.465 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2024-03-27T21:40:43.465 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-27T21:40:43.465 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-27T21:40:43.465 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-27T21:40:43.465 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2024-03-27T21:40:43.465 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2024-03-27T21:40:43.465 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2024-03-27T21:40:43.465 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2024-03-27T21:40:43.466 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2024-03-27T21:40:43.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.466 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2024-03-27T21:40:43.466 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2024-03-27T21:40:43.466 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2024-03-27T21:40:43.466 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2024-03-27T21:40:43.466 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2024-03-27T21:40:43.466 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2024-03-27T21:40:43.466 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2024-03-27T21:40:43.466 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2024-03-27T21:40:43.466 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2024-03-27T21:40:43.466 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2024-03-27T21:40:43.466 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2024-03-27T21:40:43.466 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2024-03-27T21:40:43.467 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2024-03-27T21:40:43.467 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2024-03-27T21:40:43.467 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.467 INFO:teuthology.orchestra.run.smithi060.stdout: "read_balance": { 2024-03-27T21:40:43.467 INFO:teuthology.orchestra.run.smithi060.stdout: "score_acting": 1.9900000095367432, 2024-03-27T21:40:43.467 INFO:teuthology.orchestra.run.smithi060.stdout: "score_stable": 1.9900000095367432, 2024-03-27T21:40:43.467 INFO:teuthology.orchestra.run.smithi060.stdout: "optimal_score": 0.87999999523162842, 2024-03-27T21:40:43.467 INFO:teuthology.orchestra.run.smithi060.stdout: "raw_score_acting": 1.75, 2024-03-27T21:40:43.467 INFO:teuthology.orchestra.run.smithi060.stdout: "raw_score_stable": 1.75, 2024-03-27T21:40:43.467 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity_weighted": 1, 2024-03-27T21:40:43.467 INFO:teuthology.orchestra.run.smithi060.stdout: "average_primary_affinity": 1, 2024-03-27T21:40:43.467 INFO:teuthology.orchestra.run.smithi060.stdout: "average_primary_affinity_weighted": 1 2024-03-27T21:40:43.467 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.467 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.467 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-27T21:40:43.468 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2024-03-27T21:40:43.468 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.468 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2024-03-27T21:40:43.468 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "778aaa64-6720-495b-bcdc-86fbcece9d58", 2024-03-27T21:40:43.468 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2024-03-27T21:40:43.468 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2024-03-27T21:40:43.468 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2024-03-27T21:40:43.468 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2024-03-27T21:40:43.468 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2024-03-27T21:40:43.468 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2024-03-27T21:40:43.468 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 14, 2024-03-27T21:40:43.468 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 0, 2024-03-27T21:40:43.468 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2024-03-27T21:40:43.468 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2024-03-27T21:40:43.469 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2024-03-27T21:40:43.469 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.469 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.469 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.469 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2024-03-27T21:40:43.469 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34603 2024-03-27T21:40:43.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.469 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.469 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.469 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2024-03-27T21:40:43.469 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34603 2024-03-27T21:40:43.469 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.469 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.470 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2024-03-27T21:40:43.470 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.470 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.470 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.470 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2024-03-27T21:40:43.470 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34603 2024-03-27T21:40:43.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.470 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.470 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.470 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2024-03-27T21:40:43.470 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34603 2024-03-27T21:40:43.470 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.470 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.471 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2024-03-27T21:40:43.471 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.471 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.471 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2024-03-27T21:40:43.471 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34603 2024-03-27T21:40:43.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.471 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.471 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2024-03-27T21:40:43.471 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34603 2024-03-27T21:40:43.471 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.471 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.472 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.472 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2024-03-27T21:40:43.472 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.472 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.472 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.472 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2024-03-27T21:40:43.472 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34603 2024-03-27T21:40:43.472 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.472 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.472 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.472 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2024-03-27T21:40:43.472 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34603 2024-03-27T21:40:43.472 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.472 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.473 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.473 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6817/34603", 2024-03-27T21:40:43.473 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6819/34603", 2024-03-27T21:40:43.473 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6823/34603", 2024-03-27T21:40:43.473 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6821/34603", 2024-03-27T21:40:43.473 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2024-03-27T21:40:43.473 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2024-03-27T21:40:43.473 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2024-03-27T21:40:43.473 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.473 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.473 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.473 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2024-03-27T21:40:43.474 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "d960adf1-3d83-4e24-a612-8df53fa43d0c", 2024-03-27T21:40:43.474 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2024-03-27T21:40:43.474 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2024-03-27T21:40:43.474 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2024-03-27T21:40:43.474 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2024-03-27T21:40:43.474 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2024-03-27T21:40:43.474 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2024-03-27T21:40:43.474 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2024-03-27T21:40:43.474 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 19, 2024-03-27T21:40:43.474 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2024-03-27T21:40:43.474 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2024-03-27T21:40:43.474 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2024-03-27T21:40:43.474 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.475 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.475 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.475 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2024-03-27T21:40:43.475 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34602 2024-03-27T21:40:43.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.475 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.475 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.475 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2024-03-27T21:40:43.475 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34602 2024-03-27T21:40:43.475 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.475 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.475 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2024-03-27T21:40:43.475 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.475 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.476 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.476 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2024-03-27T21:40:43.476 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34602 2024-03-27T21:40:43.476 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.476 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.476 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.476 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2024-03-27T21:40:43.476 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34602 2024-03-27T21:40:43.476 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.476 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.476 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.476 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2024-03-27T21:40:43.476 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.476 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.476 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.476 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2024-03-27T21:40:43.477 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34602 2024-03-27T21:40:43.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.477 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.477 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2024-03-27T21:40:43.477 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34602 2024-03-27T21:40:43.477 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.477 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.477 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2024-03-27T21:40:43.477 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.477 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.477 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2024-03-27T21:40:43.477 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34602 2024-03-27T21:40:43.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.478 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.478 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.478 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2024-03-27T21:40:43.478 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34602 2024-03-27T21:40:43.478 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.478 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.478 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.478 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/34602", 2024-03-27T21:40:43.478 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/34602", 2024-03-27T21:40:43.478 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/34602", 2024-03-27T21:40:43.478 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/34602", 2024-03-27T21:40:43.478 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2024-03-27T21:40:43.478 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2024-03-27T21:40:43.479 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2024-03-27T21:40:43.479 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.479 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.479 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2024-03-27T21:40:43.479 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "b8ffc48b-cf6f-4701-a6aa-c25fdb0f7977", 2024-03-27T21:40:43.479 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2024-03-27T21:40:43.479 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2024-03-27T21:40:43.479 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2024-03-27T21:40:43.479 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2024-03-27T21:40:43.479 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 14, 2024-03-27T21:40:43.479 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 21, 2024-03-27T21:40:43.479 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 33, 2024-03-27T21:40:43.479 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 33, 2024-03-27T21:40:43.479 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 22, 2024-03-27T21:40:43.480 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2024-03-27T21:40:43.480 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2024-03-27T21:40:43.480 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.480 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.480 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.480 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2024-03-27T21:40:43.480 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 44654 2024-03-27T21:40:43.480 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.480 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.480 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.480 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2024-03-27T21:40:43.480 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 44654 2024-03-27T21:40:43.480 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.480 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.480 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.480 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2024-03-27T21:40:43.481 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.481 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.481 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.481 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2024-03-27T21:40:43.481 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 44654 2024-03-27T21:40:43.481 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.481 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.481 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.481 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2024-03-27T21:40:43.481 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 44654 2024-03-27T21:40:43.481 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.481 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.481 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.481 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2024-03-27T21:40:43.482 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.482 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.482 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.482 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2024-03-27T21:40:43.482 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 44654 2024-03-27T21:40:43.482 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.482 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.482 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.482 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2024-03-27T21:40:43.482 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 44654 2024-03-27T21:40:43.482 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.482 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.482 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.482 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2024-03-27T21:40:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2024-03-27T21:40:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 44654 2024-03-27T21:40:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2024-03-27T21:40:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 44654 2024-03-27T21:40:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/44654", 2024-03-27T21:40:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/44654", 2024-03-27T21:40:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/44654", 2024-03-27T21:40:43.484 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/44654", 2024-03-27T21:40:43.484 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2024-03-27T21:40:43.484 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2024-03-27T21:40:43.484 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2024-03-27T21:40:43.484 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.484 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.484 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.484 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2024-03-27T21:40:43.484 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "0f14f2d8-b271-48e4-a198-aa5279a01c06", 2024-03-27T21:40:43.485 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2024-03-27T21:40:43.485 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2024-03-27T21:40:43.485 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2024-03-27T21:40:43.485 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2024-03-27T21:40:43.485 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2024-03-27T21:40:43.485 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2024-03-27T21:40:43.485 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 16, 2024-03-27T21:40:43.485 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 22, 2024-03-27T21:40:43.485 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2024-03-27T21:40:43.485 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2024-03-27T21:40:43.485 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2024-03-27T21:40:43.485 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2024-03-27T21:40:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34613 2024-03-27T21:40:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2024-03-27T21:40:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34613 2024-03-27T21:40:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2024-03-27T21:40:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2024-03-27T21:40:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34613 2024-03-27T21:40:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2024-03-27T21:40:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34613 2024-03-27T21:40:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2024-03-27T21:40:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2024-03-27T21:40:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34613 2024-03-27T21:40:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2024-03-27T21:40:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34613 2024-03-27T21:40:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2024-03-27T21:40:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2024-03-27T21:40:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34613 2024-03-27T21:40:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2024-03-27T21:40:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34613 2024-03-27T21:40:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/34613", 2024-03-27T21:40:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/34613", 2024-03-27T21:40:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6815/34613", 2024-03-27T21:40:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/34613", 2024-03-27T21:40:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2024-03-27T21:40:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2024-03-27T21:40:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2024-03-27T21:40:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2024-03-27T21:40:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8cdeb7ae-1c6e-4d0d-ba44-80248eff2fb6", 2024-03-27T21:40:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2024-03-27T21:40:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2024-03-27T21:40:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2024-03-27T21:40:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2024-03-27T21:40:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2024-03-27T21:40:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2024-03-27T21:40:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 14, 2024-03-27T21:40:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 0, 2024-03-27T21:40:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2024-03-27T21:40:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2024-03-27T21:40:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2024-03-27T21:40:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6800", 2024-03-27T21:40:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34332 2024-03-27T21:40:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6801", 2024-03-27T21:40:43.492 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34332 2024-03-27T21:40:43.492 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.492 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.492 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2024-03-27T21:40:43.492 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.492 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.492 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6802", 2024-03-27T21:40:43.492 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34332 2024-03-27T21:40:43.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.493 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.493 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.493 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6803", 2024-03-27T21:40:43.493 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34332 2024-03-27T21:40:43.493 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.493 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.493 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2024-03-27T21:40:43.493 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.493 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.493 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.493 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6806", 2024-03-27T21:40:43.494 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34332 2024-03-27T21:40:43.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.494 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.494 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.494 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6807", 2024-03-27T21:40:43.494 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34332 2024-03-27T21:40:43.494 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.494 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.494 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2024-03-27T21:40:43.494 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.494 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.494 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6804", 2024-03-27T21:40:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34332 2024-03-27T21:40:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6805", 2024-03-27T21:40:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34332 2024-03-27T21:40:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.191:6801/34332", 2024-03-27T21:40:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.191:6803/34332", 2024-03-27T21:40:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.191:6807/34332", 2024-03-27T21:40:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.191:6805/34332", 2024-03-27T21:40:43.496 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2024-03-27T21:40:43.496 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2024-03-27T21:40:43.496 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2024-03-27T21:40:43.496 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.496 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.496 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.496 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2024-03-27T21:40:43.496 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "93748cf8-d3fe-46b8-950f-e01e28a5e02c", 2024-03-27T21:40:43.496 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2024-03-27T21:40:43.496 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2024-03-27T21:40:43.496 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2024-03-27T21:40:43.496 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2024-03-27T21:40:43.496 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2024-03-27T21:40:43.496 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2024-03-27T21:40:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 16, 2024-03-27T21:40:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 33, 2024-03-27T21:40:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2024-03-27T21:40:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2024-03-27T21:40:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2024-03-27T21:40:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6809", 2024-03-27T21:40:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34333 2024-03-27T21:40:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6811", 2024-03-27T21:40:43.498 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34333 2024-03-27T21:40:43.498 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.498 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.498 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2024-03-27T21:40:43.498 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.498 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.498 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.498 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6813", 2024-03-27T21:40:43.498 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34333 2024-03-27T21:40:43.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.498 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.498 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6816", 2024-03-27T21:40:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34333 2024-03-27T21:40:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2024-03-27T21:40:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6822", 2024-03-27T21:40:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34333 2024-03-27T21:40:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6823", 2024-03-27T21:40:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34333 2024-03-27T21:40:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2024-03-27T21:40:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6818", 2024-03-27T21:40:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34333 2024-03-27T21:40:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.501 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.501 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6820", 2024-03-27T21:40:43.501 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34333 2024-03-27T21:40:43.501 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.501 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.501 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.501 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.191:6811/34333", 2024-03-27T21:40:43.501 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.191:6816/34333", 2024-03-27T21:40:43.501 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.191:6823/34333", 2024-03-27T21:40:43.501 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.191:6820/34333", 2024-03-27T21:40:43.501 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2024-03-27T21:40:43.501 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2024-03-27T21:40:43.501 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2024-03-27T21:40:43.502 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.502 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.502 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.502 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2024-03-27T21:40:43.502 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "1ad21434-cbc5-4de9-8d63-76f894f7e736", 2024-03-27T21:40:43.502 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2024-03-27T21:40:43.502 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2024-03-27T21:40:43.502 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2024-03-27T21:40:43.502 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2024-03-27T21:40:43.502 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2024-03-27T21:40:43.502 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2024-03-27T21:40:43.502 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 14, 2024-03-27T21:40:43.502 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 19, 2024-03-27T21:40:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2024-03-27T21:40:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2024-03-27T21:40:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2024-03-27T21:40:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6824", 2024-03-27T21:40:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34361 2024-03-27T21:40:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6825", 2024-03-27T21:40:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34361 2024-03-27T21:40:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2024-03-27T21:40:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6826", 2024-03-27T21:40:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34361 2024-03-27T21:40:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6827", 2024-03-27T21:40:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34361 2024-03-27T21:40:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2024-03-27T21:40:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6830", 2024-03-27T21:40:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34361 2024-03-27T21:40:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6831", 2024-03-27T21:40:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34361 2024-03-27T21:40:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2024-03-27T21:40:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6828", 2024-03-27T21:40:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34361 2024-03-27T21:40:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6829", 2024-03-27T21:40:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34361 2024-03-27T21:40:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.191:6825/34361", 2024-03-27T21:40:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.191:6827/34361", 2024-03-27T21:40:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.191:6831/34361", 2024-03-27T21:40:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.191:6829/34361", 2024-03-27T21:40:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2024-03-27T21:40:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2024-03-27T21:40:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2024-03-27T21:40:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2024-03-27T21:40:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "9fbd0927-ee2f-471b-8df8-c4e0fc40b9b0", 2024-03-27T21:40:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2024-03-27T21:40:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2024-03-27T21:40:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2024-03-27T21:40:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2024-03-27T21:40:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2024-03-27T21:40:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2024-03-27T21:40:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 14, 2024-03-27T21:40:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 19, 2024-03-27T21:40:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2024-03-27T21:40:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2024-03-27T21:40:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2024-03-27T21:40:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6808", 2024-03-27T21:40:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34334 2024-03-27T21:40:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6810", 2024-03-27T21:40:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34334 2024-03-27T21:40:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2024-03-27T21:40:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6812", 2024-03-27T21:40:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34334 2024-03-27T21:40:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.510 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.510 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6814", 2024-03-27T21:40:43.510 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34334 2024-03-27T21:40:43.510 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.510 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.510 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2024-03-27T21:40:43.510 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.510 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.510 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6819", 2024-03-27T21:40:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34334 2024-03-27T21:40:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6821", 2024-03-27T21:40:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34334 2024-03-27T21:40:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2024-03-27T21:40:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-27T21:40:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-27T21:40:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6815", 2024-03-27T21:40:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34334 2024-03-27T21:40:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-27T21:40:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.191:6817", 2024-03-27T21:40:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 34334 2024-03-27T21:40:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.191:6810/34334", 2024-03-27T21:40:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.191:6814/34334", 2024-03-27T21:40:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.191:6821/34334", 2024-03-27T21:40:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.191:6817/34334", 2024-03-27T21:40:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2024-03-27T21:40:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2024-03-27T21:40:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2024-03-27T21:40:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-27T21:40:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2024-03-27T21:40:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2024-03-27T21:40:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2024-03-27T21:40:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2024-03-27T21:40:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2024-03-27T21:40:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138322906710015, 2024-03-27T21:40:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2024-03-27T21:40:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2024-03-27T21:39:34.476097+0000", 2024-03-27T21:40:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2024-03-27T21:40:43.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.514 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.514 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2024-03-27T21:40:43.514 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2024-03-27T21:40:43.514 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2024-03-27T21:40:43.514 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2024-03-27T21:40:43.514 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138322906710015, 2024-03-27T21:40:43.514 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2024-03-27T21:40:43.514 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2024-03-27T21:39:34.411243+0000", 2024-03-27T21:40:43.514 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2024-03-27T21:40:43.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.514 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.514 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2024-03-27T21:40:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2024-03-27T21:40:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2024-03-27T21:40:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2024-03-27T21:40:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138322906710015, 2024-03-27T21:40:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2024-03-27T21:40:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2024-03-27T21:39:34.458154+0000", 2024-03-27T21:40:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 22 2024-03-27T21:40:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2024-03-27T21:40:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2024-03-27T21:40:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2024-03-27T21:40:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2024-03-27T21:40:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138322906710015, 2024-03-27T21:40:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2024-03-27T21:40:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2024-03-27T21:39:34.415285+0000", 2024-03-27T21:40:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2024-03-27T21:40:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2024-03-27T21:40:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2024-03-27T21:40:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2024-03-27T21:40:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2024-03-27T21:40:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138322906710015, 2024-03-27T21:40:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2024-03-27T21:40:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2024-03-27T21:39:34.431201+0000", 2024-03-27T21:40:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2024-03-27T21:40:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2024-03-27T21:40:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2024-03-27T21:40:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2024-03-27T21:40:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2024-03-27T21:40:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138322906710015, 2024-03-27T21:40:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2024-03-27T21:40:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2024-03-27T21:39:34.456013+0000", 2024-03-27T21:40:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2024-03-27T21:40:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2024-03-27T21:40:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2024-03-27T21:40:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2024-03-27T21:40:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2024-03-27T21:40:43.518 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138322906710015, 2024-03-27T21:40:43.518 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2024-03-27T21:40:43.518 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2024-03-27T21:39:34.474448+0000", 2024-03-27T21:40:43.518 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2024-03-27T21:40:43.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:43.518 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2024-03-27T21:40:43.518 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2024-03-27T21:40:43.518 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2024-03-27T21:40:43.518 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2024-03-27T21:40:43.518 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138322906710015, 2024-03-27T21:40:43.518 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2024-03-27T21:40:43.518 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2024-03-27T21:39:34.414076+0000", 2024-03-27T21:40:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2024-03-27T21:40:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-27T21:40:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2024-03-27T21:40:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2024-03-27T21:40:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_primaries": [], 2024-03-27T21:40:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2024-03-27T21:40:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2024-03-27T21:40:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: "blocklist": {}, 2024-03-27T21:40:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: "range_blocklist": {}, 2024-03-27T21:40:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2024-03-27T21:40:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2024-03-27T21:40:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2024-03-27T21:40:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2024-03-27T21:40:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2024-03-27T21:40:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2024-03-27T21:40:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2024-03-27T21:40:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2024-03-27T21:40:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2024-03-27T21:40:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2024-03-27T21:40:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2024-03-27T21:40:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {}, 2024-03-27T21:40:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: "stretch_mode": { 2024-03-27T21:40:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: "stretch_mode_enabled": false, 2024-03-27T21:40:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: "stretch_bucket_count": 0, 2024-03-27T21:40:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: "degraded_stretch_mode": 0, 2024-03-27T21:40:43.521 INFO:teuthology.orchestra.run.smithi060.stdout: "recovering_stretch_mode": 0, 2024-03-27T21:40:43.521 INFO:teuthology.orchestra.run.smithi060.stdout: "stretch_mode_bucket": 0 2024-03-27T21:40:43.521 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:43.521 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:43.521 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-27T21:40:43.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.468+0000 7fd9685d2640 10 librados: watch_flush enter 2024-03-27T21:40:43.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.468+0000 7fd9685d2640 10 librados: watch_flush exit 2024-03-27T21:40:43.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.469+0000 7fd9685d2640 20 client.4731.objecter shutdown clearing up homeless session... 2024-03-27T21:40:43.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.469+0000 7fd9685d2640 10 client.4731.objecter successfully canceled tick 2024-03-27T21:40:43.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.469+0000 7fd9685d2640 1 -- 172.21.15.60:0/1081439290 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd94406c590 msgr2=0x7fd94406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:43.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.469+0000 7fd9685d2640 1 --2- 172.21.15.60:0/1081439290 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd94406c590 0x7fd94406ea50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009870 tx=0x7fd95c03b040 comp rx=0 tx=0).stop 2024-03-27T21:40:43.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.469+0000 7fd9685d2640 10 monclient: shutdown 2024-03-27T21:40:43.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.469+0000 7fd9685d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:43.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.469+0000 7fd9685d2640 1 -- 172.21.15.60:0/1081439290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd960143200 msgr2=0x7fd9601db250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:43.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.469+0000 7fd9685d2640 1 --2- 172.21.15.60:0/1081439290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd960143200 0x7fd9601db250 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd95800efc0 tx=0x7fd95800c490 comp rx=0 tx=0).stop 2024-03-27T21:40:43.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.469+0000 7fd9685d2640 1 -- 172.21.15.60:0/1081439290 shutdown_connections 2024-03-27T21:40:43.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.469+0000 7fd9685d2640 1 --2- 172.21.15.60:0/1081439290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd960143200 0x7fd9601db250 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.469+0000 7fd9685d2640 1 --2- 172.21.15.60:0/1081439290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd9600b32c0 0x7fd9601dad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.469+0000 7fd9685d2640 1 --2- 172.21.15.60:0/1081439290 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd94406c590 0x7fd94406ea50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.469+0000 7fd9685d2640 1 --2- 172.21.15.60:0/1081439290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9600b0ec0 0x7fd96013d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.469+0000 7fd9685d2640 1 -- 172.21.15.60:0/1081439290 >> 172.21.15.60:0/1081439290 conn(0x7fd9600608e0 msgr2=0x7fd96013cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:43.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.469+0000 7fd9685d2640 1 -- 172.21.15.60:0/1081439290 shutdown_connections 2024-03-27T21:40:43.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.469+0000 7fd9685d2640 1 -- 172.21.15.60:0/1081439290 wait complete. 2024-03-27T21:40:43.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.469+0000 7fd9685d2640 1 librados: shutdown 2024-03-27T21:40:43.526 INFO:tasks.thrashosds.thrasher:j is {'epoch': 42, 'fsid': 'dcd19d9a-535c-4b41-9095-d588268e2860', 'created': '2024-03-27T21:39:27.092191+0000', 'modified': '2024-03-27T21:40:41.459857+0000', 'last_up_change': '2024-03-27T21:40:20.406185+0000', 'last_in_change': '2024-03-27T21:39:29.291620+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-27T21:39:37.438493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-03-27T21:39:45.042103+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '778aaa64-6720-495b-bcdc-86fbcece9d58', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6816', 'nonce': 34603}, {'type': 'v1', 'addr': '172.21.15.60:6817', 'nonce': 34603}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6818', 'nonce': 34603}, {'type': 'v1', 'addr': '172.21.15.60:6819', 'nonce': 34603}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6822', 'nonce': 34603}, {'type': 'v1', 'addr': '172.21.15.60:6823', 'nonce': 34603}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6820', 'nonce': 34603}, {'type': 'v1', 'addr': '172.21.15.60:6821', 'nonce': 34603}]}, 'public_addr': '172.21.15.60:6817/34603', 'cluster_addr': '172.21.15.60:6819/34603', 'heartbeat_back_addr': '172.21.15.60:6823/34603', 'heartbeat_front_addr': '172.21.15.60:6821/34603', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'd960adf1-3d83-4e24-a612-8df53fa43d0c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6800', 'nonce': 34602}, {'type': 'v1', 'addr': '172.21.15.60:6801', 'nonce': 34602}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6802', 'nonce': 34602}, {'type': 'v1', 'addr': '172.21.15.60:6803', 'nonce': 34602}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6806', 'nonce': 34602}, {'type': 'v1', 'addr': '172.21.15.60:6807', 'nonce': 34602}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6804', 'nonce': 34602}, {'type': 'v1', 'addr': '172.21.15.60:6805', 'nonce': 34602}]}, 'public_addr': '172.21.15.60:6801/34602', 'cluster_addr': '172.21.15.60:6803/34602', 'heartbeat_back_addr': '172.21.15.60:6807/34602', 'heartbeat_front_addr': '172.21.15.60:6805/34602', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b8ffc48b-cf6f-4701-a6aa-c25fdb0f7977', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 14, 'last_clean_end': 21, 'up_from': 33, 'up_thru': 33, 'down_at': 22, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6824', 'nonce': 44654}, {'type': 'v1', 'addr': '172.21.15.60:6825', 'nonce': 44654}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6826', 'nonce': 44654}, {'type': 'v1', 'addr': '172.21.15.60:6827', 'nonce': 44654}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6830', 'nonce': 44654}, {'type': 'v1', 'addr': '172.21.15.60:6831', 'nonce': 44654}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6828', 'nonce': 44654}, {'type': 'v1', 'addr': '172.21.15.60:6829', 'nonce': 44654}]}, 'public_addr': '172.21.15.60:6825/44654', 'cluster_addr': '172.21.15.60:6827/44654', 'heartbeat_back_addr': '172.21.15.60:6831/44654', 'heartbeat_front_addr': '172.21.15.60:6829/44654', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '0f14f2d8-b271-48e4-a198-aa5279a01c06', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6808', 'nonce': 34613}, {'type': 'v1', 'addr': '172.21.15.60:6809', 'nonce': 34613}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6810', 'nonce': 34613}, {'type': 'v1', 'addr': '172.21.15.60:6811', 'nonce': 34613}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6814', 'nonce': 34613}, {'type': 'v1', 'addr': '172.21.15.60:6815', 'nonce': 34613}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6812', 'nonce': 34613}, {'type': 'v1', 'addr': '172.21.15.60:6813', 'nonce': 34613}]}, 'public_addr': '172.21.15.60:6809/34613', 'cluster_addr': '172.21.15.60:6811/34613', 'heartbeat_back_addr': '172.21.15.60:6815/34613', 'heartbeat_front_addr': '172.21.15.60:6813/34613', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '8cdeb7ae-1c6e-4d0d-ba44-80248eff2fb6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6800', 'nonce': 34332}, {'type': 'v1', 'addr': '172.21.15.191:6801', 'nonce': 34332}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6802', 'nonce': 34332}, {'type': 'v1', 'addr': '172.21.15.191:6803', 'nonce': 34332}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6806', 'nonce': 34332}, {'type': 'v1', 'addr': '172.21.15.191:6807', 'nonce': 34332}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6804', 'nonce': 34332}, {'type': 'v1', 'addr': '172.21.15.191:6805', 'nonce': 34332}]}, 'public_addr': '172.21.15.191:6801/34332', 'cluster_addr': '172.21.15.191:6803/34332', 'heartbeat_back_addr': '172.21.15.191:6807/34332', 'heartbeat_front_addr': '172.21.15.191:6805/34332', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '93748cf8-d3fe-46b8-950f-e01e28a5e02c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 33, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6809', 'nonce': 34333}, {'type': 'v1', 'addr': '172.21.15.191:6811', 'nonce': 34333}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6813', 'nonce': 34333}, {'type': 'v1', 'addr': '172.21.15.191:6816', 'nonce': 34333}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6822', 'nonce': 34333}, {'type': 'v1', 'addr': '172.21.15.191:6823', 'nonce': 34333}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6818', 'nonce': 34333}, {'type': 'v1', 'addr': '172.21.15.191:6820', 'nonce': 34333}]}, 'public_addr': '172.21.15.191:6811/34333', 'cluster_addr': '172.21.15.191:6816/34333', 'heartbeat_back_addr': '172.21.15.191:6823/34333', 'heartbeat_front_addr': '172.21.15.191:6820/34333', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '1ad21434-cbc5-4de9-8d63-76f894f7e736', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6824', 'nonce': 34361}, {'type': 'v1', 'addr': '172.21.15.191:6825', 'nonce': 34361}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6826', 'nonce': 34361}, {'type': 'v1', 'addr': '172.21.15.191:6827', 'nonce': 34361}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6830', 'nonce': 34361}, {'type': 'v1', 'addr': '172.21.15.191:6831', 'nonce': 34361}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6828', 'nonce': 34361}, {'type': 'v1', 'addr': '172.21.15.191:6829', 'nonce': 34361}]}, 'public_addr': '172.21.15.191:6825/34361', 'cluster_addr': '172.21.15.191:6827/34361', 'heartbeat_back_addr': '172.21.15.191:6831/34361', 'heartbeat_front_addr': '172.21.15.191:6829/34361', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '9fbd0927-ee2f-471b-8df8-c4e0fc40b9b0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6808', 'nonce': 34334}, {'type': 'v1', 'addr': '172.21.15.191:6810', 'nonce': 34334}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6812', 'nonce': 34334}, {'type': 'v1', 'addr': '172.21.15.191:6814', 'nonce': 34334}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6819', 'nonce': 34334}, {'type': 'v1', 'addr': '172.21.15.191:6821', 'nonce': 34334}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6815', 'nonce': 34334}, {'type': 'v1', 'addr': '172.21.15.191:6817', 'nonce': 34334}]}, 'public_addr': '172.21.15.191:6810/34334', 'cluster_addr': '172.21.15.191:6814/34334', 'heartbeat_back_addr': '172.21.15.191:6821/34334', 'heartbeat_front_addr': '172.21.15.191:6817/34334', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-27T21:39:34.476097+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-27T21:39:34.411243+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-27T21:39:34.458154+0000', 'dead_epoch': 22}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-27T21:39:34.415285+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-27T21:39:34.431201+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-27T21:39:34.456013+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-27T21:39:34.474448+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-27T21:39:34.414076+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-27T21:40:43.526 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-03-27T21:40:43.526 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:43.536+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:43.548 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:43.548 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:43.548 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:43.548 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:43.558 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:40:43.623 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:43.637+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:43.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.670+0000 7f945dfc9640 10 monclient: get_monmap_and_config 2024-03-27T21:40:43.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.670+0000 7f945dfc9640 10 monclient: build_initial_monmap 2024-03-27T21:40:43.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.670+0000 7f945dfc9640 10 monclient: monmap: 2024-03-27T21:40:43.657 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:43.657 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:43.657 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:43.671895+0000 2024-03-27T21:40:43.658 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:43.671895+0000 2024-03-27T21:40:43.658 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:43.658 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:43.658 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:43.658 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:43.658 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:43.658 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:43.658 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:43.658 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:43.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.671+0000 7f945dfc9640 1 Processor -- start 2024-03-27T21:40:43.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.671+0000 7f945dfc9640 1 -- start start 2024-03-27T21:40:43.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.671+0000 7f945dfc9640 10 monclient: init 2024-03-27T21:40:43.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f945dfc9640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:43.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f945dfc9640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-03-27T21:40:43.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f945dfc9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f94580b40b0 0x7f94580a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:43.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f945dfc9640 10 monclient(hunting): picked mon.noname-a con 0x7f94580b40b0 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:43.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f945dfc9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f945813b020 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:43.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f94577fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f94580b40b0 0x7f94580a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:43.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f9456ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f945813b020 0x7f94581416f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:48028/0 (socket says 172.21.15.60:48028) 2024-03-27T21:40:43.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f94577fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f94580b40b0 0x7f94580a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39726/0 (socket says 172.21.15.60:39726) 2024-03-27T21:40:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f9456ffd640 1 -- 172.21.15.60:0/2637821127 learned_addr learned my addr 172.21.15.60:0/2637821127 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:40:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f945dfc9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f945813b480 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f945dfc9640 1 -- 172.21.15.60:0/2637821127 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f945813b8d0 con 0x7f945813b020 2024-03-27T21:40:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f945dfc9640 1 -- 172.21.15.60:0/2637821127 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94580b3830 con 0x7f945813b480 2024-03-27T21:40:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f945dfc9640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f945dfc9640 1 -- 172.21.15.60:0/2637821127 --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94580b4490 con 0x7f94580b40b0 2024-03-27T21:40:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f945dfc9640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f945dfc9640 10 monclient(hunting): authenticate will time out at 884.893634s 2024-03-27T21:40:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f94577fe640 10 monclient(hunting): get_auth_request con 0x7f94580b40b0 auth_method 0 2024-03-27T21:40:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f94577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f94577fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f94577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f94577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f94577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.672+0000 7f94577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.673+0000 7f94567fc640 1 -- 172.21.15.60:0/2637821127 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3100265515 0 0) 0x7f945813b8d0 con 0x7f945813b020 2024-03-27T21:40:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.673+0000 7f94567fc640 10 monclient(hunting): _negotiate 2024-03-27T21:40:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.673+0000 7f94567fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.673+0000 7f94567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.673+0000 7f94567fc640 10 monclient(hunting): my global_id is 4737 2024-03-27T21:40:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.673+0000 7f94567fc640 1 -- 172.21.15.60:0/2637821127 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9440003520 con 0x7f945813b020 2024-03-27T21:40:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.673+0000 7f94577fe640 10 monclient(hunting): handle_auth_done global_id 4734 payload 274 2024-03-27T21:40:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.673+0000 7f94577fe640 1 -- 172.21.15.60:0/2637821127 >> v1:172.21.15.191:6789/0 conn(0x7f945813b480 legacy=0x7f945814bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.673+0000 7f94577fe640 1 -- 172.21.15.60:0/2637821127 >> v1:172.21.15.60:6789/0 conn(0x7f945813b020 legacy=0x7f94581416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.673+0000 7f94577fe640 10 monclient: _finish_hunting 0 2024-03-27T21:40:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.673+0000 7f94577fe640 1 monclient: found mon.noname-a 2024-03-27T21:40:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.673+0000 7f94577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.673+0000 7f94577fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.673+0000 7f94577fe640 1 -- 172.21.15.60:0/2637821127 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94580aa520 con 0x7f94580b40b0 2024-03-27T21:40:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.674+0000 7f94577fe640 1 --2- 172.21.15.60:0/2637821127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94580b40b0 0x7f94580a8b90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f94480097a0 tx=0x7f944802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c684b78d4c4d91e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.674+0000 7f94567fc640 1 -- 172.21.15.60:0/2637821127 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944802e070 con 0x7f94580b40b0 2024-03-27T21:40:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.674+0000 7f94567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.674+0000 7f94567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.674+0000 7f94567fc640 10 monclient: dump: 2024-03-27T21:40:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:43.666 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:43.666 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:43.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.674+0000 7f94567fc640 10 monclient: _finish_auth 0 2024-03-27T21:40:43.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.674+0000 7f94567fc640 10 monclient: _check_auth_tickets 2024-03-27T21:40:43.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.674+0000 7f94567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:43.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.674+0000 7f945dfc9640 5 monclient: authenticate success, global_id 4734 2024-03-27T21:40:43.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.674+0000 7f945dfc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:43.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.674+0000 7f94567fc640 1 -- 172.21.15.60:0/2637821127 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94480315a0 con 0x7f94580b40b0 2024-03-27T21:40:43.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.674+0000 7f94567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:43.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.674+0000 7f94567fc640 1 -- 172.21.15.60:0/2637821127 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9448031910 con 0x7f94580b40b0 2024-03-27T21:40:43.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.674+0000 7f94567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:43.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.674+0000 7f94567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:43.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.674+0000 7f94567fc640 10 monclient: dump: 2024-03-27T21:40:43.667 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:43.667 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:43.667 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:43.667 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:43.667 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:43.667 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:43.667 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:43.667 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:43.667 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:43.667 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:43.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.674+0000 7f945dfc9640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:43.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.674+0000 7f945dfc9640 4 set_mon_vals no callback set 2024-03-27T21:40:43.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.675+0000 7f945dfc9640 10 monclient: shutdown 2024-03-27T21:40:43.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.675+0000 7f945dfc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:43.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.675+0000 7f945dfc9640 1 -- 172.21.15.60:0/2637821127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94580b40b0 msgr2=0x7f94580a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:43.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.675+0000 7f945dfc9640 1 --2- 172.21.15.60:0/2637821127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94580b40b0 0x7f94580a8b90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f94480097a0 tx=0x7f944802b840 comp rx=0 tx=0).stop 2024-03-27T21:40:43.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.675+0000 7f945dfc9640 1 -- 172.21.15.60:0/2637821127 shutdown_connections 2024-03-27T21:40:43.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.675+0000 7f945dfc9640 1 --2- 172.21.15.60:0/2637821127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94580b40b0 0x7f94580a8b90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.675+0000 7f945dfc9640 1 -- 172.21.15.60:0/2637821127 >> 172.21.15.60:0/2637821127 conn(0x7f94580608e0 msgr2=0x7f9458060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:43.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.675+0000 7f945dfc9640 1 -- 172.21.15.60:0/2637821127 shutdown_connections 2024-03-27T21:40:43.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.675+0000 7f945dfc9640 1 -- 172.21.15.60:0/2637821127 wait complete. 2024-03-27T21:40:43.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.676+0000 7f945dfc9640 10 monclient: build_initial_monmap 2024-03-27T21:40:43.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.676+0000 7f945dfc9640 10 monclient: monmap: 2024-03-27T21:40:43.676 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:43.676 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:43.676 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:43.676 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:43.676 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:43.676 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:43.676 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:43.676 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:43.676 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:43.677 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:43.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.676+0000 7f945dfc9640 1 librados: starting msgr at 2024-03-27T21:40:43.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.676+0000 7f945dfc9640 1 librados: starting objecter 2024-03-27T21:40:43.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.676+0000 7f945dfc9640 1 Processor -- start 2024-03-27T21:40:43.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.677+0000 7f945dfc9640 1 -- start start 2024-03-27T21:40:43.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.677+0000 7f945dfc9640 1 librados: setting wanted keys 2024-03-27T21:40:43.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.677+0000 7f945dfc9640 1 librados: calling monclient init 2024-03-27T21:40:43.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.677+0000 7f945dfc9640 10 monclient: init 2024-03-27T21:40:43.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f945dfc9640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:43.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f945dfc9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:40:43.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f945dfc9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94580b40b0 0x7f94581c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:43.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f945dfc9640 10 monclient(hunting): picked mon.noname-a con 0x7f94580b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:43.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f945dfc9640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f945813b020 0x7f94581d6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:43.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f94577fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94580b40b0 0x7f94581c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:43.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f94577fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94580b40b0 0x7f94581c9c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39740/0 (socket says 172.21.15.60:39740) 2024-03-27T21:40:43.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f94577fe640 1 -- 172.21.15.60:0/2203465828 learned_addr learned my addr 172.21.15.60:0/2203465828 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:43.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f945dfc9640 10 monclient(hunting): picked mon.noname-c con 0x7f945813b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:43.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f945dfc9640 1 --2- 172.21.15.60:0/2203465828 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f945813b480 0x7f945813b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:43.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f9456ffd640 1 --2- 172.21.15.60:0/2203465828 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f945813b020 0x7f94581d6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:43.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f945dfc9640 10 monclient(hunting): picked mon.noname-b con 0x7f945813b480 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:43.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f945dfc9640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:43.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f945dfc9640 1 -- 172.21.15.60:0/2203465828 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94581d6780 con 0x7f94580b40b0 2024-03-27T21:40:43.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f945dfc9640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:43.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f945dfc9640 1 -- 172.21.15.60:0/2203465828 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94581d7150 con 0x7f945813b480 2024-03-27T21:40:43.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f945dfc9640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:43.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f945dfc9640 1 -- 172.21.15.60:0/2203465828 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94581ca7a0 con 0x7f945813b020 2024-03-27T21:40:43.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f945dfc9640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:43.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f945dfc9640 10 monclient(hunting): authenticate will time out at 884.899384s 2024-03-27T21:40:43.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f94577fe640 10 monclient(hunting): get_auth_request con 0x7f94580b40b0 auth_method 0 2024-03-27T21:40:43.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f94577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:43.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f94577fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:43.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f94577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:43.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f9456ffd640 10 monclient(hunting): get_auth_request con 0x7f945813b020 auth_method 0 2024-03-27T21:40:43.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f9456ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:43.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f9456ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:43.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f9456ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:43.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f9456ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:43.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f9456ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:43.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f9456ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:43.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f9457fff640 1 --2- 172.21.15.60:0/2203465828 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f945813b480 0x7f945813b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:43.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f9457fff640 10 monclient(hunting): get_auth_request con 0x7f945813b480 auth_method 0 2024-03-27T21:40:43.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f9457fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:43.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f9457fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:43.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f9457fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:43.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f94577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:43.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f94577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:43.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f94577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:43.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f9457fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:43.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f9457fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:43.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.678+0000 7f9457fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:43.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f9456ffd640 10 monclient(hunting): handle_auth_done global_id 4592 payload 824 2024-03-27T21:40:43.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f9456ffd640 1 -- 172.21.15.60:0/2203465828 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f945813b480 msgr2=0x7f945813b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:43.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f9456ffd640 1 --2- 172.21.15.60:0/2203465828 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f945813b480 0x7f945813b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f9456ffd640 1 -- 172.21.15.60:0/2203465828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94580b40b0 msgr2=0x7f94581c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:43.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f9456ffd640 1 --2- 172.21.15.60:0/2203465828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94580b40b0 0x7f94581c9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:43.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f9456ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:40:43.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f9456ffd640 1 monclient: found mon.noname-c 2024-03-27T21:40:43.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f9456ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:43.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f9456ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:40:43.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f9456ffd640 1 -- 172.21.15.60:0/2203465828 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9448009400 con 0x7f945813b020 2024-03-27T21:40:43.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f9457fff640 1 --2- 172.21.15.60:0/2203465828 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f945813b480 0x7f945813b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:43.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f94577fe640 1 --2- 172.21.15.60:0/2203465828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94580b40b0 0x7f94581c9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:43.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f9456ffd640 1 --2- 172.21.15.60:0/2203465828 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f945813b020 0x7f94581d6c10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f944c00c430 tx=0x7f944c00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:43.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f943ffff640 10 client.?.objecter ms_handle_connect 0x7f945813b020 2024-03-27T21:40:43.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f943ffff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:43.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f943ffff640 1 -- 172.21.15.60:0/2203465828 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c005ce0 con 0x7f945813b020 2024-03-27T21:40:43.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f943ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:43.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f943ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:43.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f943ffff640 10 monclient: dump: 2024-03-27T21:40:43.682 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:43.682 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:43.682 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:43.682 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:43.682 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:43.682 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:43.683 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:43.683 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:43.683 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:43.683 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:43.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f943ffff640 10 monclient: _finish_auth 0 2024-03-27T21:40:43.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f943ffff640 10 monclient: _check_auth_tickets 2024-03-27T21:40:43.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f943ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:43.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f945dfc9640 5 monclient: authenticate success, global_id 4592 2024-03-27T21:40:43.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f945dfc9640 10 monclient: _renew_subs 2024-03-27T21:40:43.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f943ffff640 1 -- 172.21.15.60:0/2203465828 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944c005e40 con 0x7f945813b020 2024-03-27T21:40:43.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f945dfc9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:43.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f945dfc9640 1 -- 172.21.15.60:0/2203465828 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945813bdf0 con 0x7f945813b020 2024-03-27T21:40:43.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f943ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:43.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f943ffff640 1 -- 172.21.15.60:0/2203465828 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c0072d0 con 0x7f945813b020 2024-03-27T21:40:43.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f943ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:43.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f943ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:43.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.679+0000 7f943ffff640 10 monclient: dump: 2024-03-27T21:40:43.684 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:43.684 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:43.684 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:43.684 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:43.684 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:43.684 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:43.684 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:43.684 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:43.684 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:43.684 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:43.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.680+0000 7f945dfc9640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:43.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.680+0000 7f9455ffb640 4 set_mon_vals no callback set 2024-03-27T21:40:43.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.680+0000 7f945dfc9640 10 monclient: _renew_subs 2024-03-27T21:40:43.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.680+0000 7f945dfc9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:43.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.680+0000 7f945dfc9640 1 -- 172.21.15.60:0/2203465828 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94581d6480 con 0x7f945813b020 2024-03-27T21:40:43.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.680+0000 7f945dfc9640 1 librados: init done 2024-03-27T21:40:43.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.681+0000 7f943ffff640 1 -- 172.21.15.60:0/2203465828 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f944c007430 con 0x7f945813b020 2024-03-27T21:40:43.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.682+0000 7f943ffff640 1 --2- 172.21.15.60:0/2203465828 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f944406c590 0x7f944406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:43.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.682+0000 7f945dfc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:43.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.682+0000 7f945dfc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:43.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.682+0000 7f945dfc9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:43.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.682+0000 7f945dfc9640 1 -- 172.21.15.60:0/2203465828 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9458069470 con 0x7f945813b020 2024-03-27T21:40:43.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.682+0000 7f943ffff640 1 -- 172.21.15.60:0/2203465828 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f944c088680 con 0x7f945813b020 2024-03-27T21:40:43.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.682+0000 7f943ffff640 10 client.4592.objecter ms_dispatch 0x7f9458000b90 osd_map(42..42 src has 1..42) v4 2024-03-27T21:40:43.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.682+0000 7f943ffff640 3 client.4592.objecter handle_osd_map got epochs [42,42] > 0 2024-03-27T21:40:43.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.682+0000 7f943ffff640 3 client.4592.objecter handle_osd_map decoding full epoch 42 2024-03-27T21:40:43.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.682+0000 7f943ffff640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-27T21:40:43.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.682+0000 7f94577fe640 1 --2- 172.21.15.60:0/2203465828 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f944406c590 0x7f944406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:43.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.682+0000 7f94577fe640 10 monclient: get_auth_request con 0x7f944406c590 auth_method 0 2024-03-27T21:40:43.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.683+0000 7f94577fe640 1 --2- 172.21.15.60:0/2203465828 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f944406c590 0x7f944406ea50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f944802b880 tx=0x7f9448008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:43.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.683+0000 7f943ffff640 10 client.4592.objecter ms_handle_connect 0x7f944406c590 2024-03-27T21:40:43.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.691+0000 7f943ffff640 1 -- 172.21.15.60:0/2203465828 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f944c021020 con 0x7f945813b020 2024-03-27T21:40:43.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.691+0000 7f943ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:43.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:43.691+0000 7f943ffff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:43.724 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:43.738+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:43.764 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:43.764 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:43.764 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:43.764 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:43.764 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:43.773 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:40:43.857 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:43.871+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:43.925 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:43.939+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:43.983 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:40:43.984 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:43.996 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:40:44.026 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:44.040+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:44.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.078+0000 7f945dfc9640 10 start_mon_command cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:40:44.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.078+0000 7f945dfc9640 10 monclient: _send_command 2 [{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:40:44.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.078+0000 7f945dfc9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:44.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.078+0000 7f945dfc9640 1 -- 172.21.15.60:0/2203465828 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f94581d6930 con 0x7f945813b020 2024-03-27T21:40:44.128 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:44.142+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:44.165 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:44.165 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:44.165 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:44.165 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:44.174 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:40:44.229 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:44.242+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:44.324 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:44.324 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:44.324 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:44.325 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:44.325 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:44.330 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:44.344+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:44.333 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:40:44.431 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:44.445+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:44.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.477+0000 7f943ffff640 1 -- 172.21.15.60:0/2203465828 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v43) v1 ==== 97+0+0 (secure 0 0 0) 0x7f944c017020 con 0x7f945813b020 2024-03-27T21:40:44.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.478+0000 7f943ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:40:44.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.478+0000 7f943ffff640 10 monclient: _finish_command 2 = system:0 nodeep-scrub is set 2024-03-27T21:40:44.465 INFO:teuthology.orchestra.run.smithi060.stderr:nodeep-scrub is set 2024-03-27T21:40:44.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.480+0000 7f945dfc9640 10 librados: watch_flush enter 2024-03-27T21:40:44.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.480+0000 7f945dfc9640 10 librados: watch_flush exit 2024-03-27T21:40:44.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.480+0000 7f945dfc9640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-27T21:40:44.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.480+0000 7f945dfc9640 10 client.4592.objecter successfully canceled tick 2024-03-27T21:40:44.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.480+0000 7f945dfc9640 1 -- 172.21.15.60:0/2203465828 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f944406c590 msgr2=0x7f944406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:44.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.480+0000 7f945dfc9640 1 --2- 172.21.15.60:0/2203465828 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f944406c590 0x7f944406ea50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f944802b880 tx=0x7f9448008030 comp rx=0 tx=0).stop 2024-03-27T21:40:44.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.480+0000 7f945dfc9640 10 monclient: shutdown 2024-03-27T21:40:44.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.480+0000 7f945dfc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:44.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.480+0000 7f945dfc9640 1 -- 172.21.15.60:0/2203465828 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f945813b020 msgr2=0x7f94581d6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:44.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.480+0000 7f945dfc9640 1 --2- 172.21.15.60:0/2203465828 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f945813b020 0x7f94581d6c10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f944c00c430 tx=0x7f944c00c8f0 comp rx=0 tx=0).stop 2024-03-27T21:40:44.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.480+0000 7f945dfc9640 1 -- 172.21.15.60:0/2203465828 shutdown_connections 2024-03-27T21:40:44.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.480+0000 7f945dfc9640 1 --2- 172.21.15.60:0/2203465828 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f944406c590 0x7f944406ea50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:44.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.480+0000 7f945dfc9640 1 --2- 172.21.15.60:0/2203465828 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f945813b480 0x7f945813b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:44.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.480+0000 7f945dfc9640 1 --2- 172.21.15.60:0/2203465828 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f945813b020 0x7f94581d6c10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:44.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.480+0000 7f945dfc9640 1 --2- 172.21.15.60:0/2203465828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94580b40b0 0x7f94581c9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:44.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.480+0000 7f945dfc9640 1 -- 172.21.15.60:0/2203465828 >> 172.21.15.60:0/2203465828 conn(0x7f94580608e0 msgr2=0x7f9458140930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:44.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.480+0000 7f945dfc9640 1 -- 172.21.15.60:0/2203465828 shutdown_connections 2024-03-27T21:40:44.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.480+0000 7f945dfc9640 1 -- 172.21.15.60:0/2203465828 wait complete. 2024-03-27T21:40:44.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:44.480+0000 7f945dfc9640 1 librados: shutdown 2024-03-27T21:40:44.511 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:44.511 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:44.511 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:44.511 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:40:44.511 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:44.520 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:40:44.561 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:44.575+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:44.633 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:44.646+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:44.636 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:44.636 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:44.636 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:44.636 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:44.645 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:40:44.733 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:44.747+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:44.791 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:44.791 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:44.791 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:44.791 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:44.791 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:44.799 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:40:44.835 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:44.849+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:44.904 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:44.904 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:44.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:44.904 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:44.904 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:44.912 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:40:44.936 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:44.949+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:45.017 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:45.017 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:45.017 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:45.017 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:45.026 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:40:45.036 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:45.050+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:45.130 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:45.130 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:45.130 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:45.137 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:45.151+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:45.139 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:40:45.207 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:40:45.243 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:45.244 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:45.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:45.244 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:45.244 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:45.251 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:45.265+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:45.252 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:40:45.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.321+0000 7f4d7668b640 10 monclient: get_monmap_and_config 2024-03-27T21:40:45.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.321+0000 7f4d7668b640 10 monclient: build_initial_monmap 2024-03-27T21:40:45.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.321+0000 7f4d7668b640 10 monclient: monmap: 2024-03-27T21:40:45.307 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:45.322150+0000 2024-03-27T21:40:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:45.322150+0000 2024-03-27T21:40:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:45.308 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:45.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.321+0000 7f4d7668b640 1 Processor -- start 2024-03-27T21:40:45.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.321+0000 7f4d7668b640 1 -- start start 2024-03-27T21:40:45.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.321+0000 7f4d7668b640 10 monclient: init 2024-03-27T21:40:45.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d7668b640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:45.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d7668b640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-03-27T21:40:45.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d7668b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d70071a50 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:45.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d7668b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4d7013ef20 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:45.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d7668b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d700a87e0 0x7f4d70141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d7668b640 10 monclient(hunting): picked mon.noname-b con 0x7f4d700a87e0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d7668b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d700a8e90 con 0x7f4d7013ef20 2024-03-27T21:40:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d7668b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d700aa040 con 0x7f4d70071a50 2024-03-27T21:40:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d7668b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d7668b640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d70071e30 con 0x7f4d700a87e0 2024-03-27T21:40:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d7668b640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d7668b640 10 monclient(hunting): authenticate will time out at 886.543529s 2024-03-27T21:40:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d6f7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4d7013ef20 0x7f4d70141630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:48044/0 (socket says 172.21.15.60:48044) 2024-03-27T21:40:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d6f7fe640 1 -- 172.21.15.60:0/475376279 learned_addr learned my addr 172.21.15.60:0/475376279 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:40:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d74c28640 1 --2- 172.21.15.60:0/475376279 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d700a87e0 0x7f4d70141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d74c28640 10 monclient(hunting): get_auth_request con 0x7f4d700a87e0 auth_method 0 2024-03-27T21:40:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d74c28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d74c28640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d74c28640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:45.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d74c28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:45.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d74c28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:45.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d74c28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:45.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d74c28640 10 monclient(hunting): handle_auth_done global_id 4595 payload 274 2024-03-27T21:40:45.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d74c28640 1 -- 172.21.15.60:0/475376279 >> v1:172.21.15.191:6789/0 conn(0x7f4d70071a50 legacy=0x7f4d7013e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:45.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d6effd640 1 -- 172.21.15.60:0/475376279 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782735864 0 0) 0x7f4d700a8e90 con 0x7f4d7013ef20 2024-03-27T21:40:45.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.322+0000 7f4d74c28640 1 -- 172.21.15.60:0/475376279 >> v1:172.21.15.60:6789/0 conn(0x7f4d7013ef20 legacy=0x7f4d70141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:45.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d74c28640 10 monclient: _finish_hunting 0 2024-03-27T21:40:45.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d74c28640 1 monclient: found mon.noname-b 2024-03-27T21:40:45.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d74c28640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:45.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d74c28640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:45.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d74c28640 1 -- 172.21.15.60:0/475376279 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d70142280 con 0x7f4d700a87e0 2024-03-27T21:40:45.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d6effd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:40:45.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d74c28640 1 --2- 172.21.15.60:0/475376279 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d700a87e0 0x7f4d70141d40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4d68009dc0 tx=0x7f4d6802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0d07ca82c11b94d server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:45.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d6effd640 1 -- 172.21.15.60:0/475376279 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6802ccd0 con 0x7f4d700a87e0 2024-03-27T21:40:45.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:45.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d6effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:45.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d6effd640 10 monclient: dump: 2024-03-27T21:40:45.315 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:45.315 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:45.315 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:45.315 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:45.315 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:45.315 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:45.315 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:45.315 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:45.315 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:45.315 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:45.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d6effd640 10 monclient: _finish_auth 0 2024-03-27T21:40:45.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d6effd640 10 monclient: _check_auth_tickets 2024-03-27T21:40:45.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:45.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d7668b640 5 monclient: authenticate success, global_id 4595 2024-03-27T21:40:45.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d7668b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:45.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d6effd640 1 -- 172.21.15.60:0/475376279 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d6802ce30 con 0x7f4d700a87e0 2024-03-27T21:40:45.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:45.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d7668b640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:45.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d7668b640 4 set_mon_vals no callback set 2024-03-27T21:40:45.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d6effd640 1 -- 172.21.15.60:0/475376279 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d68034660 con 0x7f4d700a87e0 2024-03-27T21:40:45.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:45.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d6effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:45.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.323+0000 7f4d6effd640 10 monclient: dump: 2024-03-27T21:40:45.316 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:45.316 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:45.317 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:45.317 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:45.317 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:45.317 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:45.317 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:45.317 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:45.317 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:45.317 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:45.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.324+0000 7f4d7668b640 10 monclient: shutdown 2024-03-27T21:40:45.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.324+0000 7f4d7668b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:45.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.324+0000 7f4d7668b640 1 -- 172.21.15.60:0/475376279 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d700a87e0 msgr2=0x7f4d70141d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:45.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.324+0000 7f4d7668b640 1 --2- 172.21.15.60:0/475376279 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d700a87e0 0x7f4d70141d40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4d68009dc0 tx=0x7f4d6802bf20 comp rx=0 tx=0).stop 2024-03-27T21:40:45.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.324+0000 7f4d7668b640 1 -- 172.21.15.60:0/475376279 shutdown_connections 2024-03-27T21:40:45.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.324+0000 7f4d7668b640 1 --2- 172.21.15.60:0/475376279 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d700a87e0 0x7f4d70141d40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:45.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.324+0000 7f4d7668b640 1 -- 172.21.15.60:0/475376279 >> 172.21.15.60:0/475376279 conn(0x7f4d700b7300 msgr2=0x7f4d700b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:45.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.324+0000 7f4d7668b640 1 -- 172.21.15.60:0/475376279 shutdown_connections 2024-03-27T21:40:45.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.324+0000 7f4d7668b640 1 -- 172.21.15.60:0/475376279 wait complete. 2024-03-27T21:40:45.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.325+0000 7f4d7668b640 10 monclient: build_initial_monmap 2024-03-27T21:40:45.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.325+0000 7f4d7668b640 10 monclient: monmap: 2024-03-27T21:40:45.318 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:45.318 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:45.318 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:45.318 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:45.318 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:45.318 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:45.318 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:45.318 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:45.318 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:45.318 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:45.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.325+0000 7f4d7668b640 1 librados: starting msgr at 2024-03-27T21:40:45.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.325+0000 7f4d7668b640 1 librados: starting objecter 2024-03-27T21:40:45.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.325+0000 7f4d7668b640 1 Processor -- start 2024-03-27T21:40:45.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d7668b640 1 -- start start 2024-03-27T21:40:45.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d7668b640 1 librados: setting wanted keys 2024-03-27T21:40:45.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d7668b640 1 librados: calling monclient init 2024-03-27T21:40:45.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d7668b640 10 monclient: init 2024-03-27T21:40:45.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d7668b640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:45.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d7668b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:40:45.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d7668b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d70071a50 0x7f4d701da490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:45.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d7668b640 10 monclient(hunting): picked mon.noname-a con 0x7f4d70071a50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:45.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d7668b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d700a87e0 0x7f4d701da9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:45.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d7668b640 10 monclient(hunting): picked mon.noname-b con 0x7f4d700a87e0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:45.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d7668b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d7013ef20 0x7f4d70136a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:45.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d7668b640 10 monclient(hunting): picked mon.noname-c con 0x7f4d7013ef20 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:45.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d7668b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:45.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d7668b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7013af30 con 0x7f4d70071a50 2024-03-27T21:40:45.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d7668b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:45.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d7668b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d700aabe0 con 0x7f4d700a87e0 2024-03-27T21:40:45.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d7668b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:45.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d7668b640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d70136540 con 0x7f4d7013ef20 2024-03-27T21:40:45.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d7668b640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:45.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d7668b640 10 monclient(hunting): authenticate will time out at 886.547665s 2024-03-27T21:40:45.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d74c28640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d7013ef20 0x7f4d70136a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:45.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d6ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d70071a50 0x7f4d701da490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:45.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d74c28640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d7013ef20 0x7f4d70136a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:57424/0 (socket says 172.21.15.60:57424) 2024-03-27T21:40:45.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d74c28640 1 -- 172.21.15.60:0/1888060103 learned_addr learned my addr 172.21.15.60:0/1888060103 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:45.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d74c28640 10 monclient(hunting): get_auth_request con 0x7f4d7013ef20 auth_method 0 2024-03-27T21:40:45.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d74c28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:45.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d74c28640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:45.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d74c28640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:45.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d6ffff640 10 monclient(hunting): get_auth_request con 0x7f4d70071a50 auth_method 0 2024-03-27T21:40:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d6f7fe640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d700a87e0 0x7f4d701da9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d6f7fe640 10 monclient(hunting): get_auth_request con 0x7f4d700a87e0 auth_method 0 2024-03-27T21:40:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d74c28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d74c28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d74c28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.326+0000 7f4d6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d74c28640 10 monclient(hunting): handle_auth_done global_id 4598 payload 824 2024-03-27T21:40:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d74c28640 1 -- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d700a87e0 msgr2=0x7f4d701da9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d74c28640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d700a87e0 0x7f4d701da9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d74c28640 1 -- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d70071a50 msgr2=0x7f4d701da490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d74c28640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d70071a50 0x7f4d701da490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d74c28640 10 monclient: _finish_hunting 0 2024-03-27T21:40:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d74c28640 1 monclient: found mon.noname-c 2024-03-27T21:40:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d74c28640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d74c28640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:40:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d74c28640 1 -- 172.21.15.60:0/1888060103 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d68009a70 con 0x7f4d7013ef20 2024-03-27T21:40:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d6ffff640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d70071a50 0x7f4d701da490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d74c28640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d7013ef20 0x7f4d70136a40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4d6802c530 tx=0x7f4d68004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d6cff9640 10 client.?.objecter ms_handle_connect 0x7f4d7013ef20 2024-03-27T21:40:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d6cff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6803f680 con 0x7f4d7013ef20 2024-03-27T21:40:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d6cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d6cff9640 10 monclient: dump: 2024-03-27T21:40:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:45.330 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d6cff9640 10 monclient: _finish_auth 0 2024-03-27T21:40:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d6cff9640 10 monclient: _check_auth_tickets 2024-03-27T21:40:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d6cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d7668b640 5 monclient: authenticate success, global_id 4598 2024-03-27T21:40:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d6803f7e0 con 0x7f4d7013ef20 2024-03-27T21:40:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d6cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6803fb50 con 0x7f4d7013ef20 2024-03-27T21:40:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d7668b640 10 monclient: _renew_subs 2024-03-27T21:40:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d7668b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d701daf10 con 0x7f4d7013ef20 2024-03-27T21:40:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d6cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d6cff9640 10 monclient: dump: 2024-03-27T21:40:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:45.331 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d6e7fc640 4 set_mon_vals no callback set 2024-03-27T21:40:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d7668b640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d7668b640 10 monclient: _renew_subs 2024-03-27T21:40:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d7668b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d70137160 con 0x7f4d7013ef20 2024-03-27T21:40:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.327+0000 7f4d7668b640 1 librados: init done 2024-03-27T21:40:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.328+0000 7f4d7668b640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.328+0000 7f4d7668b640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.328+0000 7f4d7668b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.328+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4d70136700 con 0x7f4d7013ef20 2024-03-27T21:40:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.328+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4d6803b040 con 0x7f4d7013ef20 2024-03-27T21:40:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d6cff9640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4d5006c590 0x7f4d5006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d6ffff640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4d5006c590 0x7f4d5006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f4d680abc30 con 0x7f4d7013ef20 2024-03-27T21:40:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d6ffff640 10 monclient: get_auth_request con 0x7f4d5006c590 auth_method 0 2024-03-27T21:40:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d6cff9640 10 client.4598.objecter ms_dispatch 0x7f4d70000b90 osd_map(43..43 src has 1..43) v4 2024-03-27T21:40:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d6cff9640 3 client.4598.objecter handle_osd_map got epochs [43,43] > 0 2024-03-27T21:40:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d6cff9640 3 client.4598.objecter handle_osd_map decoding full epoch 43 2024-03-27T21:40:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d6cff9640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-27T21:40:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v43) v1 ==== 54+0+16 (secure 0 0 0) 0x7f4d680ae210 con 0x7f4d7013ef20 2024-03-27T21:40:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d6cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d6cff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d7668b640 10 client.4598.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign -1 1 2024-03-27T21:40:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d6cff9640 10 client.4598.objecter ms_handle_connect 0x7f4d5006c590 2024-03-27T21:40:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d6ffff640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4d5006c590 0x7f4d5006ea50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4d60000f60 tx=0x7f4d60009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d7668b640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f4d70137410 0x7f4d70139250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d700aa590 osd=0 3 2024-03-27T21:40:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d700aa590 osd=0 3 2024-03-27T21:40:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d700aa590 osd=0 3 2024-03-27T21:40:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_remove -1 1 2024-03-27T21:40:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d7668b640 20 client.4598.objecter get_session s=0x7f4d700aa590 osd=0 3 2024-03-27T21:40:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign 0 1 2024-03-27T21:40:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d700aa590 osd=0 4 2024-03-27T21:40:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d7668b640 10 client.4598.objecter _send_command 1 2024-03-27T21:40:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d70069470 con 0x7f4d70137410 2024-03-27T21:40:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.329+0000 7f4d6f7fe640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f4d70137410 0x7f4d70139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.330+0000 7f4d6f7fe640 10 monclient: get_auth_request con 0x7f4d70137410 auth_method 0 2024-03-27T21:40:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.330+0000 7f4d6cff9640 10 client.4598.objecter ms_handle_connect 0x7f4d70137410 2024-03-27T21:40:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.330+0000 7f4d6f7fe640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f4d70137410 0x7f4d70139250 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.330+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4d70069470 con 0x7f4d70137410 2024-03-27T21:40:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.330+0000 7f4d6cff9640 10 client.4598.objecter ms_dispatch 0x7f4d70000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.330+0000 7f4d6cff9640 10 client.4598.objecter _finish_command 1 = system:0 2024-03-27T21:40:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.330+0000 7f4d6cff9640 20 client.4598.objecter put_session s=0x7f4d700aa590 osd=0 4 2024-03-27T21:40:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.330+0000 7f4d6cff9640 15 client.4598.objecter _session_command_op_remove 0 1 2024-03-27T21:40:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.345+0000 7f4d7668b640 10 client.4598.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.345+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign -1 2 2024-03-27T21:40:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.345+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d700aa590 osd=0 3 2024-03-27T21:40:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.345+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d700aa590 osd=0 3 2024-03-27T21:40:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.345+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d700aa590 osd=0 3 2024-03-27T21:40:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.345+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_remove -1 2 2024-03-27T21:40:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.345+0000 7f4d7668b640 20 client.4598.objecter get_session s=0x7f4d700aa590 osd=0 3 2024-03-27T21:40:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.345+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign 0 2 2024-03-27T21:40:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.345+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d700aa590 osd=0 4 2024-03-27T21:40:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.345+0000 7f4d7668b640 10 client.4598.objecter _send_command 2 2024-03-27T21:40:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.345+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4d700a9260 con 0x7f4d70137410 2024-03-27T21:40:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.346+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4d700a9260 con 0x7f4d70137410 2024-03-27T21:40:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.346+0000 7f4d6cff9640 10 client.4598.objecter ms_dispatch 0x7f4d70000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.346+0000 7f4d6cff9640 10 client.4598.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.346+0000 7f4d6cff9640 20 client.4598.objecter put_session s=0x7f4d700aa590 osd=0 4 2024-03-27T21:40:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.346+0000 7f4d6cff9640 15 client.4598.objecter _session_command_op_remove 0 2 2024-03-27T21:40:45.336 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.346+0000 7f4d7668b640 10 client.4598.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.346+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign -1 3 2024-03-27T21:40:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.346+0000 7f4d7668b640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f4d70131bf0 0x7f4d70131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.346+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d70069dd0 osd=1 3 2024-03-27T21:40:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.346+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d70069dd0 osd=1 3 2024-03-27T21:40:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.346+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d70069dd0 osd=1 3 2024-03-27T21:40:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.346+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_remove -1 3 2024-03-27T21:40:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.346+0000 7f4d7668b640 20 client.4598.objecter get_session s=0x7f4d70069dd0 osd=1 3 2024-03-27T21:40:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.346+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign 1 3 2024-03-27T21:40:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.346+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d70069dd0 osd=1 4 2024-03-27T21:40:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.346+0000 7f4d7668b640 10 client.4598.objecter _send_command 3 2024-03-27T21:40:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.346+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d700b25a0 con 0x7f4d70131bf0 2024-03-27T21:40:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.347+0000 7f4d6ffff640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f4d70131bf0 0x7f4d70131fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.347+0000 7f4d6ffff640 10 monclient: get_auth_request con 0x7f4d70131bf0 auth_method 0 2024-03-27T21:40:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.347+0000 7f4d6cff9640 10 client.4598.objecter ms_handle_connect 0x7f4d70131bf0 2024-03-27T21:40:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.347+0000 7f4d6ffff640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f4d70131bf0 0x7f4d70131fd0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.347+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4d700b25a0 con 0x7f4d70131bf0 2024-03-27T21:40:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.347+0000 7f4d6cff9640 10 client.4598.objecter ms_dispatch 0x7f4d70000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.347+0000 7f4d6cff9640 10 client.4598.objecter _finish_command 3 = system:0 2024-03-27T21:40:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.347+0000 7f4d6cff9640 20 client.4598.objecter put_session s=0x7f4d70069dd0 osd=1 4 2024-03-27T21:40:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.347+0000 7f4d6cff9640 15 client.4598.objecter _session_command_op_remove 1 3 2024-03-27T21:40:45.340 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:45.354+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:45.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.362+0000 7f4d7668b640 10 client.4598.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:45.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.362+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign -1 4 2024-03-27T21:40:45.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.362+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d70069dd0 osd=1 3 2024-03-27T21:40:45.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.362+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d70069dd0 osd=1 3 2024-03-27T21:40:45.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.362+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d70069dd0 osd=1 3 2024-03-27T21:40:45.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.362+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_remove -1 4 2024-03-27T21:40:45.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.362+0000 7f4d7668b640 20 client.4598.objecter get_session s=0x7f4d70069dd0 osd=1 3 2024-03-27T21:40:45.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.362+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign 1 4 2024-03-27T21:40:45.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.362+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d70069dd0 osd=1 4 2024-03-27T21:40:45.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.362+0000 7f4d7668b640 10 client.4598.objecter _send_command 4 2024-03-27T21:40:45.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.362+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4d70132510 con 0x7f4d70131bf0 2024-03-27T21:40:45.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.364+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4d70132510 con 0x7f4d70131bf0 2024-03-27T21:40:45.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.364+0000 7f4d6cff9640 10 client.4598.objecter ms_dispatch 0x7f4d70000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:45.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.364+0000 7f4d6cff9640 10 client.4598.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:45.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.364+0000 7f4d6cff9640 20 client.4598.objecter put_session s=0x7f4d70069dd0 osd=1 4 2024-03-27T21:40:45.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.364+0000 7f4d6cff9640 15 client.4598.objecter _session_command_op_remove 1 4 2024-03-27T21:40:45.350 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.364+0000 7f4d7668b640 10 client.4598.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.364+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign -1 5 2024-03-27T21:40:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.364+0000 7f4d7668b640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f4d70065fa0 0x7f4d70066380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.364+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d70132950 osd=2 3 2024-03-27T21:40:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.364+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d70132950 osd=2 3 2024-03-27T21:40:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.364+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d70132950 osd=2 3 2024-03-27T21:40:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.364+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_remove -1 5 2024-03-27T21:40:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.364+0000 7f4d7668b640 20 client.4598.objecter get_session s=0x7f4d70132950 osd=2 3 2024-03-27T21:40:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.364+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign 2 5 2024-03-27T21:40:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.364+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d70132950 osd=2 4 2024-03-27T21:40:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.364+0000 7f4d7668b640 10 client.4598.objecter _send_command 5 2024-03-27T21:40:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.364+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d700668c0 con 0x7f4d70065fa0 2024-03-27T21:40:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.364+0000 7f4d6f7fe640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f4d70065fa0 0x7f4d70066380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:45.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.364+0000 7f4d6f7fe640 10 monclient: get_auth_request con 0x7f4d70065fa0 auth_method 0 2024-03-27T21:40:45.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.364+0000 7f4d6cff9640 10 client.4598.objecter ms_handle_connect 0x7f4d70065fa0 2024-03-27T21:40:45.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.364+0000 7f4d6f7fe640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f4d70065fa0 0x7f4d70066380 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:45.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.365+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== osd.2 v2:172.21.15.60:6824/44654 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4d700668c0 con 0x7f4d70065fa0 2024-03-27T21:40:45.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.365+0000 7f4d6cff9640 10 client.4598.objecter ms_dispatch 0x7f4d70000b90 command_reply(tid 5: 0 ) v1 2024-03-27T21:40:45.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.365+0000 7f4d6cff9640 10 client.4598.objecter _finish_command 5 = system:0 2024-03-27T21:40:45.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.365+0000 7f4d6cff9640 20 client.4598.objecter put_session s=0x7f4d70132950 osd=2 4 2024-03-27T21:40:45.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.365+0000 7f4d6cff9640 15 client.4598.objecter _session_command_op_remove 2 5 2024-03-27T21:40:45.358 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:45.358 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:45.358 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:45.358 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:45.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.379+0000 7f4d7668b640 10 client.4598.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:45.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.379+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign -1 6 2024-03-27T21:40:45.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.379+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d70132950 osd=2 3 2024-03-27T21:40:45.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.379+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d70132950 osd=2 3 2024-03-27T21:40:45.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.379+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d70132950 osd=2 3 2024-03-27T21:40:45.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.379+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_remove -1 6 2024-03-27T21:40:45.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.379+0000 7f4d7668b640 20 client.4598.objecter get_session s=0x7f4d70132950 osd=2 3 2024-03-27T21:40:45.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.379+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign 2 6 2024-03-27T21:40:45.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.379+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d70132950 osd=2 4 2024-03-27T21:40:45.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.379+0000 7f4d7668b640 10 client.4598.objecter _send_command 6 2024-03-27T21:40:45.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.379+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4d7013b970 con 0x7f4d70065fa0 2024-03-27T21:40:45.367 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:40:45.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.380+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== osd.2 v2:172.21.15.60:6824/44654 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4d7013b970 con 0x7f4d70065fa0 2024-03-27T21:40:45.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.380+0000 7f4d6cff9640 10 client.4598.objecter ms_dispatch 0x7f4d70000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:45.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.380+0000 7f4d6cff9640 10 client.4598.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:45.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.380+0000 7f4d6cff9640 20 client.4598.objecter put_session s=0x7f4d70132950 osd=2 4 2024-03-27T21:40:45.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.380+0000 7f4d6cff9640 15 client.4598.objecter _session_command_op_remove 2 6 2024-03-27T21:40:45.368 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2: {} 2024-03-27T21:40:45.369 INFO:teuthology.orchestra.run.smithi060.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-27T21:40:45.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.381+0000 7f4d7668b640 10 client.4598.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:45.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.381+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign -1 7 2024-03-27T21:40:45.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.381+0000 7f4d7668b640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f4d701335d0 0x7f4d701339b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:45.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.381+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d7013bd20 osd=3 3 2024-03-27T21:40:45.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.381+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d7013bd20 osd=3 3 2024-03-27T21:40:45.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.381+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d7013bd20 osd=3 3 2024-03-27T21:40:45.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.381+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_remove -1 7 2024-03-27T21:40:45.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.381+0000 7f4d7668b640 20 client.4598.objecter get_session s=0x7f4d7013bd20 osd=3 3 2024-03-27T21:40:45.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.381+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign 3 7 2024-03-27T21:40:45.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.381+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d7013bd20 osd=3 4 2024-03-27T21:40:45.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.381+0000 7f4d7668b640 10 client.4598.objecter _send_command 7 2024-03-27T21:40:45.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.381+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d701ec030 con 0x7f4d701335d0 2024-03-27T21:40:45.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.381+0000 7f4d74c28640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f4d701335d0 0x7f4d701339b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:45.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.381+0000 7f4d74c28640 10 monclient: get_auth_request con 0x7f4d701335d0 auth_method 0 2024-03-27T21:40:45.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.381+0000 7f4d6cff9640 10 client.4598.objecter ms_handle_connect 0x7f4d701335d0 2024-03-27T21:40:45.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.381+0000 7f4d74c28640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f4d701335d0 0x7f4d701339b0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:45.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.382+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4d701ec030 con 0x7f4d701335d0 2024-03-27T21:40:45.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.382+0000 7f4d6cff9640 10 client.4598.objecter ms_dispatch 0x7f4d70000b90 command_reply(tid 7: 0 ) v1 2024-03-27T21:40:45.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.382+0000 7f4d6cff9640 10 client.4598.objecter _finish_command 7 = system:0 2024-03-27T21:40:45.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.382+0000 7f4d6cff9640 20 client.4598.objecter put_session s=0x7f4d7013bd20 osd=3 4 2024-03-27T21:40:45.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.382+0000 7f4d6cff9640 15 client.4598.objecter _session_command_op_remove 3 7 2024-03-27T21:40:45.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.404+0000 7f4d7668b640 10 client.4598.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:45.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.404+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign -1 8 2024-03-27T21:40:45.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.404+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d7013bd20 osd=3 3 2024-03-27T21:40:45.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.404+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d7013bd20 osd=3 3 2024-03-27T21:40:45.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.404+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d7013bd20 osd=3 3 2024-03-27T21:40:45.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.404+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_remove -1 8 2024-03-27T21:40:45.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.404+0000 7f4d7668b640 20 client.4598.objecter get_session s=0x7f4d7013bd20 osd=3 3 2024-03-27T21:40:45.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.404+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign 3 8 2024-03-27T21:40:45.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.404+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d7013bd20 osd=3 4 2024-03-27T21:40:45.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.404+0000 7f4d7668b640 10 client.4598.objecter _send_command 8 2024-03-27T21:40:45.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.404+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4d70133050 con 0x7f4d701335d0 2024-03-27T21:40:45.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.405+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4d70133050 con 0x7f4d701335d0 2024-03-27T21:40:45.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.405+0000 7f4d6cff9640 10 client.4598.objecter ms_dispatch 0x7f4d70000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:45.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.405+0000 7f4d6cff9640 10 client.4598.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:45.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.405+0000 7f4d6cff9640 20 client.4598.objecter put_session s=0x7f4d7013bd20 osd=3 4 2024-03-27T21:40:45.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.405+0000 7f4d6cff9640 15 client.4598.objecter _session_command_op_remove 3 8 2024-03-27T21:40:45.392 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:45.393 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:45.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.406+0000 7f4d7668b640 10 client.4598.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:45.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.406+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign -1 9 2024-03-27T21:40:45.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.406+0000 7f4d7668b640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f4d701331c0 0x7f4d701ea920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.406+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d70133050 osd=4 3 2024-03-27T21:40:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.406+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d70133050 osd=4 3 2024-03-27T21:40:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.406+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d70133050 osd=4 3 2024-03-27T21:40:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.406+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_remove -1 9 2024-03-27T21:40:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.406+0000 7f4d7668b640 20 client.4598.objecter get_session s=0x7f4d70133050 osd=4 3 2024-03-27T21:40:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.406+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign 4 9 2024-03-27T21:40:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.406+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d70133050 osd=4 4 2024-03-27T21:40:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.406+0000 7f4d7668b640 10 client.4598.objecter _send_command 9 2024-03-27T21:40:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.406+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d7002be40 con 0x7f4d701331c0 2024-03-27T21:40:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.406+0000 7f4d6ffff640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f4d701331c0 0x7f4d701ea920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.406+0000 7f4d6ffff640 10 monclient: get_auth_request con 0x7f4d701331c0 auth_method 0 2024-03-27T21:40:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.406+0000 7f4d6cff9640 10 client.4598.objecter ms_handle_connect 0x7f4d701331c0 2024-03-27T21:40:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.406+0000 7f4d6ffff640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f4d701331c0 0x7f4d701ea920 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.407+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4d7002be40 con 0x7f4d701331c0 2024-03-27T21:40:45.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.407+0000 7f4d6cff9640 10 client.4598.objecter ms_dispatch 0x7f4d70000b90 command_reply(tid 9: 0 ) v1 2024-03-27T21:40:45.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.407+0000 7f4d6cff9640 10 client.4598.objecter _finish_command 9 = system:0 2024-03-27T21:40:45.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.407+0000 7f4d6cff9640 20 client.4598.objecter put_session s=0x7f4d70133050 osd=4 4 2024-03-27T21:40:45.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.407+0000 7f4d6cff9640 15 client.4598.objecter _session_command_op_remove 4 9 2024-03-27T21:40:45.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.429+0000 7f4d7668b640 10 client.4598.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:45.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.429+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign -1 10 2024-03-27T21:40:45.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.429+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d70133050 osd=4 3 2024-03-27T21:40:45.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.429+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d70133050 osd=4 3 2024-03-27T21:40:45.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.429+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d70133050 osd=4 3 2024-03-27T21:40:45.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.429+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_remove -1 10 2024-03-27T21:40:45.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.429+0000 7f4d7668b640 20 client.4598.objecter get_session s=0x7f4d70133050 osd=4 3 2024-03-27T21:40:45.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.429+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign 4 10 2024-03-27T21:40:45.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.429+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d70133050 osd=4 4 2024-03-27T21:40:45.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.429+0000 7f4d7668b640 10 client.4598.objecter _send_command 10 2024-03-27T21:40:45.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.429+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4d7002be40 con 0x7f4d701331c0 2024-03-27T21:40:45.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.430+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4d7002be40 con 0x7f4d701331c0 2024-03-27T21:40:45.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.430+0000 7f4d6cff9640 10 client.4598.objecter ms_dispatch 0x7f4d70000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:45.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.430+0000 7f4d6cff9640 10 client.4598.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:45.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.430+0000 7f4d6cff9640 20 client.4598.objecter put_session s=0x7f4d70133050 osd=4 4 2024-03-27T21:40:45.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.430+0000 7f4d6cff9640 15 client.4598.objecter _session_command_op_remove 4 10 2024-03-27T21:40:45.417 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:45.418 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:45.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.431+0000 7f4d7668b640 10 client.4598.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:45.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.431+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign -1 11 2024-03-27T21:40:45.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.431+0000 7f4d7668b640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f4d701eb370 0x7f4d701eb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:45.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.431+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d7002be40 osd=5 3 2024-03-27T21:40:45.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.431+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d7002be40 osd=5 3 2024-03-27T21:40:45.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.431+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d7002be40 osd=5 3 2024-03-27T21:40:45.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.431+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_remove -1 11 2024-03-27T21:40:45.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.431+0000 7f4d7668b640 20 client.4598.objecter get_session s=0x7f4d7002be40 osd=5 3 2024-03-27T21:40:45.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.431+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign 5 11 2024-03-27T21:40:45.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.431+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d7002be40 osd=5 4 2024-03-27T21:40:45.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.431+0000 7f4d7668b640 10 client.4598.objecter _send_command 11 2024-03-27T21:40:45.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.431+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d701ebd30 con 0x7f4d701eb370 2024-03-27T21:40:45.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.431+0000 7f4d6f7fe640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f4d701eb370 0x7f4d701eb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:45.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.431+0000 7f4d6f7fe640 10 monclient: get_auth_request con 0x7f4d701eb370 auth_method 0 2024-03-27T21:40:45.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.431+0000 7f4d6cff9640 10 client.4598.objecter ms_handle_connect 0x7f4d701eb370 2024-03-27T21:40:45.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.431+0000 7f4d6f7fe640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f4d701eb370 0x7f4d701eb7f0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:45.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.432+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4d701ebd30 con 0x7f4d701eb370 2024-03-27T21:40:45.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.432+0000 7f4d6cff9640 10 client.4598.objecter ms_dispatch 0x7f4d70000b90 command_reply(tid 11: 0 ) v1 2024-03-27T21:40:45.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.432+0000 7f4d6cff9640 10 client.4598.objecter _finish_command 11 = system:0 2024-03-27T21:40:45.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.432+0000 7f4d6cff9640 20 client.4598.objecter put_session s=0x7f4d7002be40 osd=5 4 2024-03-27T21:40:45.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.432+0000 7f4d6cff9640 15 client.4598.objecter _session_command_op_remove 5 11 2024-03-27T21:40:45.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.446+0000 7f4d7668b640 10 client.4598.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:45.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.446+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign -1 12 2024-03-27T21:40:45.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.446+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d7002be40 osd=5 3 2024-03-27T21:40:45.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.446+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d7002be40 osd=5 3 2024-03-27T21:40:45.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.446+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d7002be40 osd=5 3 2024-03-27T21:40:45.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.446+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_remove -1 12 2024-03-27T21:40:45.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.446+0000 7f4d7668b640 20 client.4598.objecter get_session s=0x7f4d7002be40 osd=5 3 2024-03-27T21:40:45.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.446+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign 5 12 2024-03-27T21:40:45.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.446+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d7002be40 osd=5 4 2024-03-27T21:40:45.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.446+0000 7f4d7668b640 10 client.4598.objecter _send_command 12 2024-03-27T21:40:45.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.446+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4d701ebd30 con 0x7f4d701eb370 2024-03-27T21:40:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.447+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4d701ebd30 con 0x7f4d701eb370 2024-03-27T21:40:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.447+0000 7f4d6cff9640 10 client.4598.objecter ms_dispatch 0x7f4d70000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.447+0000 7f4d6cff9640 10 client.4598.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.447+0000 7f4d6cff9640 20 client.4598.objecter put_session s=0x7f4d7002be40 osd=5 4 2024-03-27T21:40:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.447+0000 7f4d6cff9640 15 client.4598.objecter _session_command_op_remove 5 12 2024-03-27T21:40:45.434 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:45.435 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.448+0000 7f4d7668b640 10 client.4598.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.448+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign -1 13 2024-03-27T21:40:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.448+0000 7f4d7668b640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f4d701f06d0 0x7f4d701f2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.448+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d701ebd30 osd=6 3 2024-03-27T21:40:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.448+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d701ebd30 osd=6 3 2024-03-27T21:40:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.448+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d701ebd30 osd=6 3 2024-03-27T21:40:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.448+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_remove -1 13 2024-03-27T21:40:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.448+0000 7f4d7668b640 20 client.4598.objecter get_session s=0x7f4d701ebd30 osd=6 3 2024-03-27T21:40:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.448+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign 6 13 2024-03-27T21:40:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.448+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d701ebd30 osd=6 4 2024-03-27T21:40:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.448+0000 7f4d7668b640 10 client.4598.objecter _send_command 13 2024-03-27T21:40:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.448+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d7002bfb0 con 0x7f4d701f06d0 2024-03-27T21:40:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.448+0000 7f4d74c28640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f4d701f06d0 0x7f4d701f2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.448+0000 7f4d74c28640 10 monclient: get_auth_request con 0x7f4d701f06d0 auth_method 0 2024-03-27T21:40:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.448+0000 7f4d74c28640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f4d701f06d0 0x7f4d701f2ac0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.448+0000 7f4d6cff9640 10 client.4598.objecter ms_handle_connect 0x7f4d701f06d0 2024-03-27T21:40:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.449+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4d7002bfb0 con 0x7f4d701f06d0 2024-03-27T21:40:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.449+0000 7f4d6cff9640 10 client.4598.objecter ms_dispatch 0x7f4d70000b90 command_reply(tid 13: 0 ) v1 2024-03-27T21:40:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.449+0000 7f4d6cff9640 10 client.4598.objecter _finish_command 13 = system:0 2024-03-27T21:40:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.449+0000 7f4d6cff9640 20 client.4598.objecter put_session s=0x7f4d701ebd30 osd=6 4 2024-03-27T21:40:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.449+0000 7f4d6cff9640 15 client.4598.objecter _session_command_op_remove 6 13 2024-03-27T21:40:45.441 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:45.455+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:45.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.463+0000 7f4d7668b640 10 client.4598.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:45.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.463+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign -1 14 2024-03-27T21:40:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.463+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d701ebd30 osd=6 3 2024-03-27T21:40:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.463+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d701ebd30 osd=6 3 2024-03-27T21:40:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.463+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d701ebd30 osd=6 3 2024-03-27T21:40:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.463+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_remove -1 14 2024-03-27T21:40:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.463+0000 7f4d7668b640 20 client.4598.objecter get_session s=0x7f4d701ebd30 osd=6 3 2024-03-27T21:40:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.463+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign 6 14 2024-03-27T21:40:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.463+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d701ebd30 osd=6 4 2024-03-27T21:40:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.463+0000 7f4d7668b640 10 client.4598.objecter _send_command 14 2024-03-27T21:40:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.463+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4d7002bfb0 con 0x7f4d701f06d0 2024-03-27T21:40:45.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.466+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4d7002bfb0 con 0x7f4d701f06d0 2024-03-27T21:40:45.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.466+0000 7f4d6cff9640 10 client.4598.objecter ms_dispatch 0x7f4d70000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:45.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.466+0000 7f4d6cff9640 10 client.4598.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:45.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.466+0000 7f4d6cff9640 20 client.4598.objecter put_session s=0x7f4d701ebd30 osd=6 4 2024-03-27T21:40:45.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.466+0000 7f4d6cff9640 15 client.4598.objecter _session_command_op_remove 6 14 2024-03-27T21:40:45.453 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:45.454 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:45.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.466+0000 7f4d7668b640 10 client.4598.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:45.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.466+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign -1 15 2024-03-27T21:40:45.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.466+0000 7f4d7668b640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f4d701f3510 0x7f4d701f5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:45.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.466+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d7002bfb0 osd=7 3 2024-03-27T21:40:45.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.466+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d7002bfb0 osd=7 3 2024-03-27T21:40:45.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.466+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d7002bfb0 osd=7 3 2024-03-27T21:40:45.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.466+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_remove -1 15 2024-03-27T21:40:45.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.466+0000 7f4d7668b640 20 client.4598.objecter get_session s=0x7f4d7002bfb0 osd=7 3 2024-03-27T21:40:45.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.466+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign 7 15 2024-03-27T21:40:45.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.466+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d7002bfb0 osd=7 4 2024-03-27T21:40:45.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.466+0000 7f4d7668b640 10 client.4598.objecter _send_command 15 2024-03-27T21:40:45.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.466+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d7002c120 con 0x7f4d701f3510 2024-03-27T21:40:45.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.466+0000 7f4d6ffff640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f4d701f3510 0x7f4d701f5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:45.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.466+0000 7f4d6ffff640 10 monclient: get_auth_request con 0x7f4d701f3510 auth_method 0 2024-03-27T21:40:45.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.467+0000 7f4d6cff9640 10 client.4598.objecter ms_handle_connect 0x7f4d701f3510 2024-03-27T21:40:45.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.467+0000 7f4d6ffff640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f4d701f3510 0x7f4d701f5900 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:45.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.467+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4d7002c120 con 0x7f4d701f3510 2024-03-27T21:40:45.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.467+0000 7f4d6cff9640 10 client.4598.objecter ms_dispatch 0x7f4d70000b90 command_reply(tid 15: 0 ) v1 2024-03-27T21:40:45.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.467+0000 7f4d6cff9640 10 client.4598.objecter _finish_command 15 = system:0 2024-03-27T21:40:45.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.467+0000 7f4d6cff9640 20 client.4598.objecter put_session s=0x7f4d7002bfb0 osd=7 4 2024-03-27T21:40:45.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.467+0000 7f4d6cff9640 15 client.4598.objecter _session_command_op_remove 7 15 2024-03-27T21:40:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.481+0000 7f4d7668b640 10 client.4598.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.481+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign -1 16 2024-03-27T21:40:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.481+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d7002bfb0 osd=7 3 2024-03-27T21:40:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.481+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d7002bfb0 osd=7 3 2024-03-27T21:40:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.481+0000 7f4d7668b640 20 client.4598.objecter _get_session s=0x7f4d7002bfb0 osd=7 3 2024-03-27T21:40:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.481+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_remove -1 16 2024-03-27T21:40:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.481+0000 7f4d7668b640 20 client.4598.objecter get_session s=0x7f4d7002bfb0 osd=7 3 2024-03-27T21:40:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.481+0000 7f4d7668b640 15 client.4598.objecter _session_command_op_assign 7 16 2024-03-27T21:40:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.481+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d7002bfb0 osd=7 4 2024-03-27T21:40:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.481+0000 7f4d7668b640 10 client.4598.objecter _send_command 16 2024-03-27T21:40:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.481+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4d7002c120 con 0x7f4d701f3510 2024-03-27T21:40:45.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d6cff9640 1 -- 172.21.15.60:0/1888060103 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4d7002c120 con 0x7f4d701f3510 2024-03-27T21:40:45.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d6cff9640 10 client.4598.objecter ms_dispatch 0x7f4d70000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:45.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d6cff9640 10 client.4598.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:45.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d6cff9640 20 client.4598.objecter put_session s=0x7f4d7002bfb0 osd=7 4 2024-03-27T21:40:45.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d6cff9640 15 client.4598.objecter _session_command_op_remove 7 16 2024-03-27T21:40:45.470 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:45.472 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:45.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 10 librados: watch_flush enter 2024-03-27T21:40:45.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 10 librados: watch_flush exit 2024-03-27T21:40:45.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 10 client.4598.objecter close_session for osd.0 2024-03-27T21:40:45.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f4d70137410 msgr2=0x7f4d70139250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:45.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f4d70137410 0x7f4d70139250 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:45.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d700aa590 osd=0 1 2024-03-27T21:40:45.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 10 client.4598.objecter close_session for osd.1 2024-03-27T21:40:45.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f4d70131bf0 msgr2=0x7f4d70131fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:45.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f4d70131bf0 0x7f4d70131fd0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:45.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d70069dd0 osd=1 1 2024-03-27T21:40:45.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 10 client.4598.objecter close_session for osd.2 2024-03-27T21:40:45.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f4d70065fa0 msgr2=0x7f4d70066380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:45.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7f4d70065fa0 0x7f4d70066380 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:45.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d70132950 osd=2 1 2024-03-27T21:40:45.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 10 client.4598.objecter close_session for osd.3 2024-03-27T21:40:45.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f4d701335d0 msgr2=0x7f4d701339b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:45.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f4d701335d0 0x7f4d701339b0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:45.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d7013bd20 osd=3 1 2024-03-27T21:40:45.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 10 client.4598.objecter close_session for osd.4 2024-03-27T21:40:45.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d74c28640 1 -- 172.21.15.60:0/1888060103 reap_dead start 2024-03-27T21:40:45.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f4d701331c0 msgr2=0x7f4d701ea920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:45.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f4d701331c0 0x7f4d701ea920 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:45.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d74c28640 1 -- 172.21.15.60:0/1888060103 reap_dead start 2024-03-27T21:40:45.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.483+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d70133050 osd=4 1 2024-03-27T21:40:45.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 10 client.4598.objecter close_session for osd.5 2024-03-27T21:40:45.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f4d701eb370 msgr2=0x7f4d701eb7f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:45.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f4d701eb370 0x7f4d701eb7f0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:45.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d7002be40 osd=5 1 2024-03-27T21:40:45.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 10 client.4598.objecter close_session for osd.6 2024-03-27T21:40:45.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f4d701f06d0 msgr2=0x7f4d701f2ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:45.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f4d701f06d0 0x7f4d701f2ac0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:45.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d701ebd30 osd=6 1 2024-03-27T21:40:45.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 10 client.4598.objecter close_session for osd.7 2024-03-27T21:40:45.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f4d701f3510 msgr2=0x7f4d701f5900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:45.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f4d701f3510 0x7f4d701f5900 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:45.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 20 client.4598.objecter put_session s=0x7f4d7002bfb0 osd=7 1 2024-03-27T21:40:45.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-27T21:40:45.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 10 client.4598.objecter successfully canceled tick 2024-03-27T21:40:45.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4d5006c590 msgr2=0x7f4d5006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:45.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4d5006c590 0x7f4d5006ea50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4d60000f60 tx=0x7f4d60009040 comp rx=0 tx=0).stop 2024-03-27T21:40:45.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 10 monclient: shutdown 2024-03-27T21:40:45.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:45.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d7013ef20 msgr2=0x7f4d70136a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:45.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d74c28640 1 -- 172.21.15.60:0/1888060103 reap_dead start 2024-03-27T21:40:45.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d7013ef20 0x7f4d70136a40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4d6802c530 tx=0x7f4d68004a40 comp rx=0 tx=0).stop 2024-03-27T21:40:45.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 shutdown_connections 2024-03-27T21:40:45.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 1 --2- 172.21.15.60:0/1888060103 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d7013ef20 0x7f4d70136a40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:45.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 >> 172.21.15.60:0/1888060103 conn(0x7f4d700b7300 msgr2=0x7f4d7013d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:45.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 shutdown_connections 2024-03-27T21:40:45.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 1 -- 172.21.15.60:0/1888060103 wait complete. 2024-03-27T21:40:45.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:45.484+0000 7f4d7668b640 1 librados: shutdown 2024-03-27T21:40:45.478 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:45.478 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:45.487 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:40:45.543 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:45.556+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:45.592 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:45.593 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:45.600 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:40:45.643 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:45.657+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:45.743 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:45.757+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:45.746 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:45.746 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:45.754 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:40:45.845 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:45.858+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:45.858 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:45.859 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:45.867 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:40:45.951 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:45.965+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:46.013 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:46.013 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:46.022 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-27T21:40:46.065 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:46.079+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:46.130 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:46.130 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:46.139 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-27T21:40:46.148 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:46.162+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:46.249 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:46.263+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:46.286 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:46.286 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:46.295 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-27T21:40:46.350 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:46.364+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:46.403 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:46.403 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:46.412 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:40:46.451 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:46.464+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:46.478 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-27T21:40:46.523 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:46.523 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:46.534 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:40:46.577 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:46.591+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:46.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.598+0000 7f070112e640 10 monclient: get_monmap_and_config 2024-03-27T21:40:46.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.598+0000 7f070112e640 10 monclient: build_initial_monmap 2024-03-27T21:40:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.598+0000 7f070112e640 10 monclient: monmap: 2024-03-27T21:40:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:46.599624+0000 2024-03-27T21:40:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:46.599624+0000 2024-03-27T21:40:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:46.585 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:46.585 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.599+0000 7f070112e640 1 Processor -- start 2024-03-27T21:40:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.599+0000 7f070112e640 1 -- start start 2024-03-27T21:40:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.599+0000 7f070112e640 10 monclient: init 2024-03-27T21:40:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.599+0000 7f070112e640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.599+0000 7f070112e640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-03-27T21:40:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.599+0000 7f070112e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f06fc06f900 0x7f06fc13fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.599+0000 7f070112e640 10 monclient(hunting): picked mon.noname-b con 0x7f06fc06f900 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.599+0000 7f070112e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f06fc06fce0 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.599+0000 7f070112e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f06fc138bf0 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.599+0000 7f070112e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06fc071b50 con 0x7f06fc138bf0 2024-03-27T21:40:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.599+0000 7f070112e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06fc13f690 con 0x7f06fc06fce0 2024-03-27T21:40:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.599+0000 7f070112e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.599+0000 7f070112e640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06fc140100 con 0x7f06fc06f900 2024-03-27T21:40:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.599+0000 7f070112e640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.599+0000 7f070112e640 10 monclient(hunting): authenticate will time out at 887.820860s 2024-03-27T21:40:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.600+0000 7f06fad9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f06fc06f900 0x7f06fc13fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.600+0000 7f06fad9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f06fc06f900 0x7f06fc13fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:57428/0 (socket says 172.21.15.60:57428) 2024-03-27T21:40:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.600+0000 7f06fad9d640 1 -- 172.21.15.60:0/3372059373 learned_addr learned my addr 172.21.15.60:0/3372059373 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.600+0000 7f06fad9d640 10 monclient(hunting): get_auth_request con 0x7f06fc06f900 auth_method 0 2024-03-27T21:40:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.600+0000 7f06fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.600+0000 7f06fad9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:46.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.600+0000 7f06fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.600+0000 7f06fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.600+0000 7f06fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.600+0000 7f06fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.600+0000 7f06fad9d640 10 monclient(hunting): handle_auth_done global_id 4601 payload 274 2024-03-27T21:40:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.600+0000 7f06fad9d640 1 -- 172.21.15.60:0/3372059373 >> v1:172.21.15.191:6789/0 conn(0x7f06fc06fce0 legacy=0x7f06fc0af850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.600+0000 7f06fad9d640 1 -- 172.21.15.60:0/3372059373 >> v1:172.21.15.60:6789/0 conn(0x7f06fc138bf0 legacy=0x7f06fc139080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.600+0000 7f06fad9d640 10 monclient: _finish_hunting 0 2024-03-27T21:40:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.600+0000 7f06fad9d640 1 monclient: found mon.noname-b 2024-03-27T21:40:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.600+0000 7f06fad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.600+0000 7f06fad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.600+0000 7f06fad9d640 1 -- 172.21.15.60:0/3372059373 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06fc06f130 con 0x7f06fc06f900 2024-03-27T21:40:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.601+0000 7f06fad9d640 1 --2- 172.21.15.60:0/3372059373 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f06fc06f900 0x7f06fc13fbc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0098a0 tx=0x7f06ec02ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5a5925a7bc4094c server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.601+0000 7f06f9d9b640 1 -- 172.21.15.60:0/3372059373 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec02f830 con 0x7f06fc06f900 2024-03-27T21:40:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.601+0000 7f06f9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.601+0000 7f06f9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.601+0000 7f06f9d9b640 10 monclient: dump: 2024-03-27T21:40:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:46.592 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.601+0000 7f06f9d9b640 10 monclient: _finish_auth 0 2024-03-27T21:40:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.601+0000 7f06f9d9b640 10 monclient: _check_auth_tickets 2024-03-27T21:40:46.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.601+0000 7f06f9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:46.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.601+0000 7f070112e640 5 monclient: authenticate success, global_id 4601 2024-03-27T21:40:46.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.601+0000 7f06f9d9b640 1 -- 172.21.15.60:0/3372059373 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ec02f990 con 0x7f06fc06f900 2024-03-27T21:40:46.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.601+0000 7f06f9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:46.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.601+0000 7f070112e640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:46.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.601+0000 7f06f9d9b640 1 -- 172.21.15.60:0/3372059373 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec02fd20 con 0x7f06fc06f900 2024-03-27T21:40:46.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.601+0000 7f070112e640 4 set_mon_vals no callback set 2024-03-27T21:40:46.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.601+0000 7f06f9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:46.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.601+0000 7f06f9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:46.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.601+0000 7f06f9d9b640 10 monclient: dump: 2024-03-27T21:40:46.593 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:46.593 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:46.593 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:46.593 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:46.594 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:46.594 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:46.594 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:46.594 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:46.594 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:46.594 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:46.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.601+0000 7f070112e640 10 monclient: shutdown 2024-03-27T21:40:46.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.601+0000 7f070112e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:46.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.601+0000 7f070112e640 1 -- 172.21.15.60:0/3372059373 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f06fc06f900 msgr2=0x7f06fc13fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:46.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.601+0000 7f070112e640 1 --2- 172.21.15.60:0/3372059373 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f06fc06f900 0x7f06fc13fbc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0098a0 tx=0x7f06ec02ba10 comp rx=0 tx=0).stop 2024-03-27T21:40:46.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.602+0000 7f070112e640 1 -- 172.21.15.60:0/3372059373 shutdown_connections 2024-03-27T21:40:46.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.602+0000 7f070112e640 1 --2- 172.21.15.60:0/3372059373 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f06fc06f900 0x7f06fc13fbc0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:46.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.602+0000 7f070112e640 1 -- 172.21.15.60:0/3372059373 >> 172.21.15.60:0/3372059373 conn(0x7f06fc0b4e50 msgr2=0x7f06fc0b5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:46.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.602+0000 7f070112e640 1 -- 172.21.15.60:0/3372059373 shutdown_connections 2024-03-27T21:40:46.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.602+0000 7f070112e640 1 -- 172.21.15.60:0/3372059373 wait complete. 2024-03-27T21:40:46.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.602+0000 7f070112e640 10 monclient: build_initial_monmap 2024-03-27T21:40:46.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.602+0000 7f070112e640 10 monclient: monmap: 2024-03-27T21:40:46.595 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:46.595 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:46.595 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:46.595 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:46.595 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:46.595 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:46.595 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:46.595 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:46.595 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:46.595 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:46.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.603+0000 7f070112e640 1 librados: starting msgr at 2024-03-27T21:40:46.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.603+0000 7f070112e640 1 librados: starting objecter 2024-03-27T21:40:46.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.603+0000 7f070112e640 1 Processor -- start 2024-03-27T21:40:46.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.603+0000 7f070112e640 1 -- start start 2024-03-27T21:40:46.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.603+0000 7f070112e640 1 librados: setting wanted keys 2024-03-27T21:40:46.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.603+0000 7f070112e640 1 librados: calling monclient init 2024-03-27T21:40:46.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.603+0000 7f070112e640 10 monclient: init 2024-03-27T21:40:46.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.603+0000 7f070112e640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:46.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.603+0000 7f070112e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:40:46.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.603+0000 7f070112e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f06fc06f900 0x7f06fc142ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:46.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f070112e640 10 monclient(hunting): picked mon.noname-b con 0x7f06fc06f900 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:46.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f070112e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06fc06fce0 0x7f06fc0ae4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:46.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f070112e640 10 monclient(hunting): picked mon.noname-a con 0x7f06fc06fce0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:46.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f070112e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f06fc138bf0 0x7f06fc137940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:46.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f070112e640 10 monclient(hunting): picked mon.noname-c con 0x7f06fc138bf0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:46.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f070112e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:46.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f070112e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06fc06b5b0 con 0x7f06fc06fce0 2024-03-27T21:40:46.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f070112e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:46.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f070112e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06fc132d40 con 0x7f06fc06f900 2024-03-27T21:40:46.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f070112e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:46.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f070112e640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06fc06b7e0 con 0x7f06fc138bf0 2024-03-27T21:40:46.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f070112e640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:46.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f070112e640 10 monclient(hunting): authenticate will time out at 887.824982s 2024-03-27T21:40:46.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fb59e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f06fc138bf0 0x7f06fc137940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:46.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fa59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06fc06fce0 0x7f06fc0ae4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:46.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fa59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06fc06fce0 0x7f06fc0ae4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39760/0 (socket says 172.21.15.60:39760) 2024-03-27T21:40:46.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fb59e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f06fc138bf0 0x7f06fc137940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:57430/0 (socket says 172.21.15.60:57430) 2024-03-27T21:40:46.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fa59c640 1 -- 172.21.15.60:0/3331848634 learned_addr learned my addr 172.21.15.60:0/3331848634 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:46.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fa59c640 10 monclient(hunting): get_auth_request con 0x7f06fc06fce0 auth_method 0 2024-03-27T21:40:46.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:46.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fa59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:46.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:46.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fb59e640 10 monclient(hunting): get_auth_request con 0x7f06fc138bf0 auth_method 0 2024-03-27T21:40:46.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:46.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fb59e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:46.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fb59e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:46.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fad9d640 1 --2- 172.21.15.60:0/3331848634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f06fc06f900 0x7f06fc142ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:46.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fad9d640 10 monclient(hunting): get_auth_request con 0x7f06fc06f900 auth_method 0 2024-03-27T21:40:46.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:46.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fad9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:46.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:46.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:46.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:46.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:46.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:46.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:46.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:46.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:46.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:46.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:46.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fa59c640 10 monclient(hunting): handle_auth_done global_id 4749 payload 824 2024-03-27T21:40:46.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fa59c640 1 -- 172.21.15.60:0/3331848634 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f06fc138bf0 msgr2=0x7f06fc137940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:46.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fa59c640 1 --2- 172.21.15.60:0/3331848634 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f06fc138bf0 0x7f06fc137940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:46.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fa59c640 1 -- 172.21.15.60:0/3331848634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f06fc06f900 msgr2=0x7f06fc142ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:46.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fa59c640 1 --2- 172.21.15.60:0/3331848634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f06fc06f900 0x7f06fc142ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:46.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fa59c640 10 monclient: _finish_hunting 0 2024-03-27T21:40:46.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fa59c640 1 monclient: found mon.noname-a 2024-03-27T21:40:46.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:46.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fa59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:46.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fa59c640 1 -- 172.21.15.60:0/3331848634 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ec0094d0 con 0x7f06fc06fce0 2024-03-27T21:40:46.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fb59e640 1 --2- 172.21.15.60:0/3331848634 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f06fc138bf0 0x7f06fc137940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:46.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06fa59c640 1 --2- 172.21.15.60:0/3331848634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06fc06fce0 0x7f06fc0ae4b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f06f000c450 tx=0x7f06f000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:46.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06e37fe640 10 client.?.objecter ms_handle_connect 0x7f06fc06fce0 2024-03-27T21:40:46.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06e37fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:46.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.604+0000 7f06e37fe640 1 -- 172.21.15.60:0/3331848634 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f00052d0 con 0x7f06fc06fce0 2024-03-27T21:40:46.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f06e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:46.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f06e37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:46.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f06e37fe640 10 monclient: dump: 2024-03-27T21:40:46.606 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:46.606 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:46.606 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:46.606 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:46.607 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:46.607 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:46.607 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:46.607 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:46.607 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:46.607 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:46.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f06e37fe640 10 monclient: _finish_auth 0 2024-03-27T21:40:46.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f06e37fe640 10 monclient: _check_auth_tickets 2024-03-27T21:40:46.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f06e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:46.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f06e37fe640 1 -- 172.21.15.60:0/3331848634 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f0005430 con 0x7f06fc06fce0 2024-03-27T21:40:46.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f070112e640 5 monclient: authenticate success, global_id 4749 2024-03-27T21:40:46.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f070112e640 10 monclient: _renew_subs 2024-03-27T21:40:46.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f070112e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:46.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f070112e640 1 -- 172.21.15.60:0/3331848634 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06fc06bac0 con 0x7f06fc06fce0 2024-03-27T21:40:46.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f06e37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:46.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f06e37fe640 1 -- 172.21.15.60:0/3331848634 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f0012870 con 0x7f06fc06fce0 2024-03-27T21:40:46.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f06e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:46.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f06e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:46.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f06e37fe640 10 monclient: dump: 2024-03-27T21:40:46.608 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:46.608 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:46.608 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:46.608 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:46.608 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:46.608 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:46.608 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:46.609 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:46.609 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:46.609 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:46.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f06f959a640 4 set_mon_vals no callback set 2024-03-27T21:40:46.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f070112e640 10 client.4749.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:46.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f070112e640 10 monclient: _renew_subs 2024-03-27T21:40:46.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f070112e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:46.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f070112e640 1 -- 172.21.15.60:0/3331848634 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06fc1d9de0 con 0x7f06fc06fce0 2024-03-27T21:40:46.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.605+0000 7f070112e640 1 librados: init done 2024-03-27T21:40:46.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.606+0000 7f070112e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:46.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.606+0000 7f070112e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:46.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.606+0000 7f070112e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:46.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.606+0000 7f070112e640 1 -- 172.21.15.60:0/3331848634 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06fc0acd80 con 0x7f06fc06fce0 2024-03-27T21:40:46.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.606+0000 7f06e37fe640 1 -- 172.21.15.60:0/3331848634 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f06f00129d0 con 0x7f06fc06fce0 2024-03-27T21:40:46.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.607+0000 7f06e37fe640 1 --2- 172.21.15.60:0/3331848634 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f06e806c590 0x7f06e806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:46.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.607+0000 7f06fad9d640 1 --2- 172.21.15.60:0/3331848634 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f06e806c590 0x7f06e806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:46.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.607+0000 7f06e37fe640 1 -- 172.21.15.60:0/3331848634 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f06f0088b10 con 0x7f06fc06fce0 2024-03-27T21:40:46.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.607+0000 7f06e37fe640 10 client.4749.objecter ms_dispatch 0x7f06fc000b90 osd_map(43..43 src has 1..43) v4 2024-03-27T21:40:46.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.607+0000 7f06e37fe640 3 client.4749.objecter handle_osd_map got epochs [43,43] > 0 2024-03-27T21:40:46.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.607+0000 7f06e37fe640 3 client.4749.objecter handle_osd_map decoding full epoch 43 2024-03-27T21:40:46.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.607+0000 7f06fad9d640 10 monclient: get_auth_request con 0x7f06e806c590 auth_method 0 2024-03-27T21:40:46.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.607+0000 7f06e37fe640 20 client.4749.objecter dump_active .. 0 homeless 2024-03-27T21:40:46.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.607+0000 7f06e37fe640 10 client.4749.objecter ms_handle_connect 0x7f06e806c590 2024-03-27T21:40:46.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.607+0000 7f06fad9d640 1 --2- 172.21.15.60:0/3331848634 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f06e806c590 0x7f06e806ea50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f06ec036000 tx=0x7f06ec008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:46.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.611+0000 7f06e37fe640 1 -- 172.21.15.60:0/3331848634 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f06fc0acd80 con 0x7f06fc06fce0 2024-03-27T21:40:46.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.611+0000 7f06e37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:46.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.611+0000 7f06e37fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:46.653 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:46.667+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:46.655 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:46.655 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:46.664 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:40:46.754 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:46.768+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:46.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.814+0000 7f070112e640 10 start_mon_command cmd=[{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:40:46.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.814+0000 7f070112e640 10 monclient: _send_command 2 [{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:40:46.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.814+0000 7f070112e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:46.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.814+0000 7f070112e640 1 -- 172.21.15.60:0/3331848634 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f06fc0acd80 con 0x7f06fc06fce0 2024-03-27T21:40:46.812 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:46.813 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:46.822 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:40:46.866 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:46.880+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:46.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.926+0000 7f06e37fe640 1 -- 172.21.15.60:0/3331848634 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v44) v1 ==== 91+0+0 (secure 0 0 0) 0x7f06fc0acd80 con 0x7f06fc06fce0 2024-03-27T21:40:46.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.926+0000 7f06e37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:40:46.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.926+0000 7f06e37fe640 10 monclient: _finish_command 2 = system:0 noscrub is unset 2024-03-27T21:40:46.912 INFO:teuthology.orchestra.run.smithi060.stderr:noscrub is unset 2024-03-27T21:40:46.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.927+0000 7f070112e640 10 librados: watch_flush enter 2024-03-27T21:40:46.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.927+0000 7f070112e640 10 librados: watch_flush exit 2024-03-27T21:40:46.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.927+0000 7f070112e640 20 client.4749.objecter shutdown clearing up homeless session... 2024-03-27T21:40:46.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.927+0000 7f070112e640 10 client.4749.objecter successfully canceled tick 2024-03-27T21:40:46.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.928+0000 7f070112e640 1 -- 172.21.15.60:0/3331848634 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f06e806c590 msgr2=0x7f06e806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:46.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.928+0000 7f070112e640 1 --2- 172.21.15.60:0/3331848634 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f06e806c590 0x7f06e806ea50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f06ec036000 tx=0x7f06ec008030 comp rx=0 tx=0).stop 2024-03-27T21:40:46.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.928+0000 7f070112e640 10 monclient: shutdown 2024-03-27T21:40:46.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.928+0000 7f070112e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:46.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.928+0000 7f070112e640 1 -- 172.21.15.60:0/3331848634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06fc06fce0 msgr2=0x7f06fc0ae4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:46.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.928+0000 7f070112e640 1 --2- 172.21.15.60:0/3331848634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06fc06fce0 0x7f06fc0ae4b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f06f000c450 tx=0x7f06f000c910 comp rx=0 tx=0).stop 2024-03-27T21:40:46.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.928+0000 7f070112e640 1 -- 172.21.15.60:0/3331848634 shutdown_connections 2024-03-27T21:40:46.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.928+0000 7f070112e640 1 --2- 172.21.15.60:0/3331848634 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f06fc138bf0 0x7f06fc137940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:46.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.928+0000 7f070112e640 1 --2- 172.21.15.60:0/3331848634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06fc06fce0 0x7f06fc0ae4b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:46.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.928+0000 7f070112e640 1 --2- 172.21.15.60:0/3331848634 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f06e806c590 0x7f06e806ea50 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:46.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.928+0000 7f070112e640 1 --2- 172.21.15.60:0/3331848634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f06fc06f900 0x7f06fc142ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:46.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.928+0000 7f070112e640 1 -- 172.21.15.60:0/3331848634 >> 172.21.15.60:0/3331848634 conn(0x7f06fc0b4e50 msgr2=0x7f06fc0b5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:46.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.928+0000 7f070112e640 1 -- 172.21.15.60:0/3331848634 shutdown_connections 2024-03-27T21:40:46.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.928+0000 7f070112e640 1 -- 172.21.15.60:0/3331848634 wait complete. 2024-03-27T21:40:46.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:46.928+0000 7f070112e640 1 librados: shutdown 2024-03-27T21:40:46.919 INFO:tasks.workunit.client.0.smithi060.stderr:Updating files: 94% (11473/12088) Updating files: 95% (11484/12088) Updating files: 96% (11605/12088) Updating files: 97% (11726/12088) Updating files: 98% (11847/12088) Updating files: 99% (11968/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-03-27T21:40:46.932 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:46.932 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:46.941 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:40:46.952 DEBUG:teuthology.orchestra.run:got remote process result: 128 2024-03-27T21:40:46.953 INFO:tasks.workunit.client.0.smithi060.stderr:fatal: reference is not a tree: 7c37d80f7a88f4eb561c4fe02558660a650b10ac 2024-03-27T21:40:46.953 INFO:tasks.workunit:failed to check out '7c37d80f7a88f4eb561c4fe02558660a650b10ac' from https://github.com/ceph/ceph-ci.git; will also try in https://github.com/ceph/ceph.git 2024-03-27T21:40:46.953 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7c37d80f7a88f4eb561c4fe02558660a650b10ac 2024-03-27T21:40:46.956 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:46.971+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:47.057 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:47.071+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:47.100 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:47.100 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:47.109 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:40:47.193 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:47.207+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:47.225 INFO:tasks.workunit.client.0.smithi060.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-27T21:40:47.257 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:47.257 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:47.260 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:47.274+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:47.266 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:40:47.361 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:47.375+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:47.417 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:47.417 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:47.427 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:40:47.470 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:47.484+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:47.481 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:40:47.534 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:47.534 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:47.544 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:40:47.562 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:47.576+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:47.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.595+0000 7ff463ff6640 10 monclient: get_monmap_and_config 2024-03-27T21:40:47.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.595+0000 7ff463ff6640 10 monclient: build_initial_monmap 2024-03-27T21:40:47.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.595+0000 7ff463ff6640 10 monclient: monmap: 2024-03-27T21:40:47.582 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:47.582 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:47.582 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:47.596431+0000 2024-03-27T21:40:47.582 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:47.596431+0000 2024-03-27T21:40:47.582 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:47.582 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:47.582 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:47.582 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:47.582 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:47.582 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:47.582 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:47.582 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:47.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.595+0000 7ff463ff6640 1 Processor -- start 2024-03-27T21:40:47.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.596+0000 7ff463ff6640 1 -- start start 2024-03-27T21:40:47.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.596+0000 7ff463ff6640 10 monclient: init 2024-03-27T21:40:47.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff463ff6640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:47.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff463ff6640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-03-27T21:40:47.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff463ff6640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff45c0b40b0 0x7ff45c0a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:47.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff463ff6640 10 monclient(hunting): picked mon.noname-b con 0x7ff45c0b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:47.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff463ff6640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff45c13b020 0x7ff45c13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:47.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff463ff6640 10 monclient(hunting): picked mon.noname-a con 0x7ff45c13b020 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:47.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff463ff6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff45c1416f0 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:47.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff463ff6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff45c0b3830 con 0x7ff45c1416f0 2024-03-27T21:40:47.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff463ff6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:47.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff463ff6640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff45c0b4490 con 0x7ff45c13b020 2024-03-27T21:40:47.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff463ff6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:47.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff463ff6640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c142290 con 0x7ff45c0b40b0 2024-03-27T21:40:47.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff463ff6640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:47.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff463ff6640 10 monclient(hunting): authenticate will time out at 888.818247s 2024-03-27T21:40:47.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461d92640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff45c0b40b0 0x7ff45c0a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:47.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461591640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff45c13b020 0x7ff45c13b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:47.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461d92640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff45c0b40b0 0x7ff45c0a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:57444/0 (socket says 172.21.15.60:57444) 2024-03-27T21:40:47.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461d92640 1 -- 172.21.15.60:0/2320069285 learned_addr learned my addr 172.21.15.60:0/2320069285 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:47.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461d92640 10 monclient(hunting): get_auth_request con 0x7ff45c0b40b0 auth_method 0 2024-03-27T21:40:47.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461d92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:47.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461d92640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:47.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461d92640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:47.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461591640 10 monclient(hunting): get_auth_request con 0x7ff45c13b020 auth_method 0 2024-03-27T21:40:47.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461591640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:47.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461591640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:47.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461591640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:47.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461d92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:47.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461d92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:47.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461d92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:47.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461591640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:47.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461591640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:47.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461591640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:47.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461d92640 10 monclient(hunting): handle_auth_done global_id 4607 payload 274 2024-03-27T21:40:47.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461d92640 1 -- 172.21.15.60:0/2320069285 >> v2:172.21.15.60:3300/0 conn(0x7ff45c13b020 msgr2=0x7ff45c13b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:47.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461d92640 1 --2- 172.21.15.60:0/2320069285 >> v2:172.21.15.60:3300/0 conn(0x7ff45c13b020 0x7ff45c13b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:47.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461d92640 1 -- 172.21.15.60:0/2320069285 >> v1:172.21.15.191:6789/0 conn(0x7ff45c1416f0 legacy=0x7ff45c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:47.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461d92640 10 monclient: _finish_hunting 0 2024-03-27T21:40:47.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461d92640 1 monclient: found mon.noname-b 2024-03-27T21:40:47.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461d92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:47.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461d92640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:47.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461d92640 1 -- 172.21.15.60:0/2320069285 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45c0aa520 con 0x7ff45c0b40b0 2024-03-27T21:40:47.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.597+0000 7ff461591640 1 --2- 172.21.15.60:0/2320069285 >> v2:172.21.15.60:3300/0 conn(0x7ff45c13b020 0x7ff45c13b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:47.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.598+0000 7ff461d92640 1 --2- 172.21.15.60:0/2320069285 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff45c0b40b0 0x7ff45c0a8b90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff458009e50 tx=0x7ff45802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cc46595d0c38bb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:47.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.598+0000 7ff460d90640 1 -- 172.21.15.60:0/2320069285 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45802ca70 con 0x7ff45c0b40b0 2024-03-27T21:40:47.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.598+0000 7ff460d90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:47.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.598+0000 7ff460d90640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:47.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.598+0000 7ff460d90640 10 monclient: dump: 2024-03-27T21:40:47.604 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:47.604 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:47.604 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:47.604 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:47.604 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:47.604 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:47.605 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:47.605 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:47.605 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:47.605 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:47.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.598+0000 7ff460d90640 10 monclient: _finish_auth 0 2024-03-27T21:40:47.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.598+0000 7ff460d90640 10 monclient: _check_auth_tickets 2024-03-27T21:40:47.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.598+0000 7ff460d90640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:47.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.598+0000 7ff460d90640 1 -- 172.21.15.60:0/2320069285 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff45802cbd0 con 0x7ff45c0b40b0 2024-03-27T21:40:47.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.598+0000 7ff463ff6640 5 monclient: authenticate success, global_id 4607 2024-03-27T21:40:47.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.598+0000 7ff463ff6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:47.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.598+0000 7ff460d90640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:47.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.598+0000 7ff463ff6640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:47.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.598+0000 7ff463ff6640 4 set_mon_vals no callback set 2024-03-27T21:40:47.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.598+0000 7ff460d90640 1 -- 172.21.15.60:0/2320069285 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4580343f0 con 0x7ff45c0b40b0 2024-03-27T21:40:47.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.598+0000 7ff460d90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:47.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.598+0000 7ff460d90640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:47.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.598+0000 7ff460d90640 10 monclient: dump: 2024-03-27T21:40:47.607 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:47.607 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:47.607 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:47.607 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:47.607 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:47.607 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:47.607 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:47.607 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:47.607 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:47.607 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.599+0000 7ff463ff6640 10 monclient: shutdown 2024-03-27T21:40:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.599+0000 7ff463ff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.599+0000 7ff463ff6640 1 -- 172.21.15.60:0/2320069285 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff45c0b40b0 msgr2=0x7ff45c0a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.599+0000 7ff463ff6640 1 --2- 172.21.15.60:0/2320069285 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff45c0b40b0 0x7ff45c0a8b90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff458009e50 tx=0x7ff45802bed0 comp rx=0 tx=0).stop 2024-03-27T21:40:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.599+0000 7ff463ff6640 1 -- 172.21.15.60:0/2320069285 shutdown_connections 2024-03-27T21:40:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.599+0000 7ff463ff6640 1 --2- 172.21.15.60:0/2320069285 >> v2:172.21.15.60:3300/0 conn(0x7ff45c13b020 0x7ff45c13b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.599+0000 7ff463ff6640 1 --2- 172.21.15.60:0/2320069285 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff45c0b40b0 0x7ff45c0a8b90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.599+0000 7ff463ff6640 1 -- 172.21.15.60:0/2320069285 >> 172.21.15.60:0/2320069285 conn(0x7ff45c0608e0 msgr2=0x7ff45c060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.599+0000 7ff463ff6640 1 -- 172.21.15.60:0/2320069285 shutdown_connections 2024-03-27T21:40:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.599+0000 7ff463ff6640 1 -- 172.21.15.60:0/2320069285 wait complete. 2024-03-27T21:40:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.600+0000 7ff463ff6640 10 monclient: build_initial_monmap 2024-03-27T21:40:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.600+0000 7ff463ff6640 10 monclient: monmap: 2024-03-27T21:40:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:47.609 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:47.609 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:47.609 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:47.609 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:47.609 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:47.609 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:47.609 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:47.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.600+0000 7ff463ff6640 1 librados: starting msgr at 2024-03-27T21:40:47.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.600+0000 7ff463ff6640 1 librados: starting objecter 2024-03-27T21:40:47.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.601+0000 7ff463ff6640 1 Processor -- start 2024-03-27T21:40:47.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.601+0000 7ff463ff6640 1 -- start start 2024-03-27T21:40:47.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.601+0000 7ff463ff6640 1 librados: setting wanted keys 2024-03-27T21:40:47.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.601+0000 7ff463ff6640 1 librados: calling monclient init 2024-03-27T21:40:47.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.601+0000 7ff463ff6640 10 monclient: init 2024-03-27T21:40:47.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.601+0000 7ff463ff6640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:47.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.601+0000 7ff463ff6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:40:47.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.601+0000 7ff463ff6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff45c0b40b0 0x7ff45c13df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:47.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.601+0000 7ff463ff6640 10 monclient(hunting): picked mon.noname-a con 0x7ff45c0b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:47.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.601+0000 7ff463ff6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff45c13b020 0x7ff45c1d69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:47.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff463ff6640 10 monclient(hunting): picked mon.noname-b con 0x7ff45c13b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:47.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff463ff6640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff45c1416f0 0x7ff45c1d6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:47.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff463ff6640 10 monclient(hunting): picked mon.noname-c con 0x7ff45c1416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:47.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff463ff6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:47.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff463ff6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c1d6530 con 0x7ff45c0b40b0 2024-03-27T21:40:47.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff463ff6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:47.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff463ff6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c13cd30 con 0x7ff45c13b020 2024-03-27T21:40:47.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff463ff6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:47.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff463ff6640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c13d900 con 0x7ff45c1416f0 2024-03-27T21:40:47.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff463ff6640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:47.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff463ff6640 10 monclient(hunting): authenticate will time out at 888.822970s 2024-03-27T21:40:47.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461d92640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff45c0b40b0 0x7ff45c13df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:47.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461d92640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff45c0b40b0 0x7ff45c13df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39790/0 (socket says 172.21.15.60:39790) 2024-03-27T21:40:47.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff462593640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff45c1416f0 0x7ff45c1d6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:47.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461d92640 1 -- 172.21.15.60:0/682874290 learned_addr learned my addr 172.21.15.60:0/682874290 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:47.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461d92640 10 monclient(hunting): get_auth_request con 0x7ff45c0b40b0 auth_method 0 2024-03-27T21:40:47.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461d92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:47.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461d92640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:47.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461d92640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:47.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff462593640 10 monclient(hunting): get_auth_request con 0x7ff45c1416f0 auth_method 0 2024-03-27T21:40:47.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff462593640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:47.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff462593640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff462593640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461d92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461d92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461d92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff462593640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff462593640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff462593640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461591640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff45c13b020 0x7ff45c1d69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461591640 10 monclient(hunting): get_auth_request con 0x7ff45c13b020 auth_method 0 2024-03-27T21:40:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461591640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461591640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461591640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461d92640 10 monclient(hunting): handle_auth_done global_id 4755 payload 824 2024-03-27T21:40:47.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461d92640 1 -- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff45c1416f0 msgr2=0x7ff45c1d6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:47.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461d92640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff45c1416f0 0x7ff45c1d6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:47.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461d92640 1 -- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff45c13b020 msgr2=0x7ff45c1d69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:47.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461d92640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff45c13b020 0x7ff45c1d69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:47.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461d92640 10 monclient: _finish_hunting 0 2024-03-27T21:40:47.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461d92640 1 monclient: found mon.noname-a 2024-03-27T21:40:47.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461d92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:47.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461d92640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:47.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461d92640 1 -- 172.21.15.60:0/682874290 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44c009400 con 0x7ff45c0b40b0 2024-03-27T21:40:47.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff462593640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff45c1416f0 0x7ff45c1d6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:47.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff461d92640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff45c0b40b0 0x7ff45c13df30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff458000f80 tx=0x7ff458004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:47.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff4527fc640 10 client.?.objecter ms_handle_connect 0x7ff45c0b40b0 2024-03-27T21:40:47.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff4527fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:47.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff458009510 con 0x7ff45c0b40b0 2024-03-27T21:40:47.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff4527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:47.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff4527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:47.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff4527fc640 10 monclient: dump: 2024-03-27T21:40:47.614 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:47.614 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:47.614 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:47.614 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:47.614 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:47.614 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:47.614 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:47.615 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:47.615 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:47.615 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:47.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff4527fc640 10 monclient: _finish_auth 0 2024-03-27T21:40:47.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff4527fc640 10 monclient: _check_auth_tickets 2024-03-27T21:40:47.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff4527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:47.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff463ff6640 5 monclient: authenticate success, global_id 4755 2024-03-27T21:40:47.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff45803e450 con 0x7ff45c0b40b0 2024-03-27T21:40:47.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff4527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:47.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45803e800 con 0x7ff45c0b40b0 2024-03-27T21:40:47.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff463ff6640 10 monclient: _renew_subs 2024-03-27T21:40:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff463ff6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff458009b30 con 0x7ff45c0b40b0 2024-03-27T21:40:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff453fff640 4 set_mon_vals no callback set 2024-03-27T21:40:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff4527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff4527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff463ff6640 10 client.4755.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff4527fc640 10 monclient: dump: 2024-03-27T21:40:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:47.617 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff463ff6640 10 monclient: _renew_subs 2024-03-27T21:40:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff463ff6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff45c13b660 con 0x7ff45c0b40b0 2024-03-27T21:40:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.602+0000 7ff463ff6640 1 librados: init done 2024-03-27T21:40:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.603+0000 7ff463ff6640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.603+0000 7ff463ff6640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.603+0000 7ff463ff6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.603+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff45c069470 con 0x7ff45c0b40b0 2024-03-27T21:40:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.604+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff45803e960 con 0x7ff45c0b40b0 2024-03-27T21:40:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.604+0000 7ff4527fc640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff44006c590 0x7ff44006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.604+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5242+0+0 (secure 0 0 0) 0x7ff4580ac350 con 0x7ff45c0b40b0 2024-03-27T21:40:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.604+0000 7ff4527fc640 10 client.4755.objecter ms_dispatch 0x7ff45c000b90 osd_map(44..44 src has 1..44) v4 2024-03-27T21:40:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.604+0000 7ff4527fc640 3 client.4755.objecter handle_osd_map got epochs [44,44] > 0 2024-03-27T21:40:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.604+0000 7ff4527fc640 3 client.4755.objecter handle_osd_map decoding full epoch 44 2024-03-27T21:40:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.604+0000 7ff461591640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff44006c590 0x7ff44006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.604+0000 7ff461591640 10 monclient: get_auth_request con 0x7ff44006c590 auth_method 0 2024-03-27T21:40:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.604+0000 7ff4527fc640 20 client.4755.objecter dump_active .. 0 homeless 2024-03-27T21:40:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.604+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v44) v1 ==== 54+0+16 (secure 0 0 0) 0x7ff4580ac7a0 con 0x7ff45c0b40b0 2024-03-27T21:40:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.604+0000 7ff4527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.604+0000 7ff4527fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.605+0000 7ff463ff6640 10 client.4755.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.605+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign -1 1 2024-03-27T21:40:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.605+0000 7ff463ff6640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7ff45c1d9870 0x7ff45c1d9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.605+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c0a90d0 osd=0 3 2024-03-27T21:40:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.605+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c0a90d0 osd=0 3 2024-03-27T21:40:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.605+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c0a90d0 osd=0 3 2024-03-27T21:40:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.605+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_remove -1 1 2024-03-27T21:40:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.605+0000 7ff463ff6640 20 client.4755.objecter get_session s=0x7ff45c0a90d0 osd=0 3 2024-03-27T21:40:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.605+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign 0 1 2024-03-27T21:40:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.605+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c0a90d0 osd=0 4 2024-03-27T21:40:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.605+0000 7ff463ff6640 10 client.4755.objecter _send_command 1 2024-03-27T21:40:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.605+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff45c0b62c0 con 0x7ff45c1d9870 2024-03-27T21:40:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.605+0000 7ff4527fc640 10 client.4755.objecter ms_handle_connect 0x7ff44006c590 2024-03-27T21:40:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.605+0000 7ff461591640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff44006c590 0x7ff44006ea50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff45c0b5f60 tx=0x7ff44c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.605+0000 7ff462593640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7ff45c1d9870 0x7ff45c1d9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.605+0000 7ff462593640 10 monclient: get_auth_request con 0x7ff45c1d9870 auth_method 0 2024-03-27T21:40:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.605+0000 7ff4527fc640 10 client.4755.objecter ms_handle_connect 0x7ff45c1d9870 2024-03-27T21:40:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.605+0000 7ff462593640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7ff45c1d9870 0x7ff45c1d9c50 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.606+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7ff45c0b62c0 con 0x7ff45c1d9870 2024-03-27T21:40:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.606+0000 7ff4527fc640 10 client.4755.objecter ms_dispatch 0x7ff45c000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:47.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.606+0000 7ff4527fc640 10 client.4755.objecter _finish_command 1 = system:0 2024-03-27T21:40:47.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.606+0000 7ff4527fc640 20 client.4755.objecter put_session s=0x7ff45c0a90d0 osd=0 4 2024-03-27T21:40:47.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.606+0000 7ff4527fc640 15 client.4755.objecter _session_command_op_remove 0 1 2024-03-27T21:40:47.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.620+0000 7ff463ff6640 10 client.4755.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.620+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign -1 2 2024-03-27T21:40:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.620+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c0a90d0 osd=0 3 2024-03-27T21:40:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.620+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c0a90d0 osd=0 3 2024-03-27T21:40:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.620+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c0a90d0 osd=0 3 2024-03-27T21:40:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.620+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_remove -1 2 2024-03-27T21:40:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.620+0000 7ff463ff6640 20 client.4755.objecter get_session s=0x7ff45c0a90d0 osd=0 3 2024-03-27T21:40:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.620+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign 0 2 2024-03-27T21:40:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.620+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c0a90d0 osd=0 4 2024-03-27T21:40:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.620+0000 7ff463ff6640 10 client.4755.objecter _send_command 2 2024-03-27T21:40:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.620+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff45c1d6670 con 0x7ff45c1d9870 2024-03-27T21:40:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.621+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff454003ef0 con 0x7ff45c1d9870 2024-03-27T21:40:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.621+0000 7ff4527fc640 10 client.4755.objecter ms_dispatch 0x7ff45c000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:47.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.621+0000 7ff4527fc640 10 client.4755.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:47.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.621+0000 7ff4527fc640 20 client.4755.objecter put_session s=0x7ff45c0a90d0 osd=0 4 2024-03-27T21:40:47.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.621+0000 7ff4527fc640 15 client.4755.objecter _session_command_op_remove 0 2 2024-03-27T21:40:47.623 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.622+0000 7ff463ff6640 10 client.4755.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.622+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign -1 3 2024-03-27T21:40:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.622+0000 7ff463ff6640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7ff45c134290 0x7ff45c134670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.622+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c1da3d0 osd=1 3 2024-03-27T21:40:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.622+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c1da3d0 osd=1 3 2024-03-27T21:40:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.622+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c1da3d0 osd=1 3 2024-03-27T21:40:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.622+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_remove -1 3 2024-03-27T21:40:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.622+0000 7ff463ff6640 20 client.4755.objecter get_session s=0x7ff45c1da3d0 osd=1 3 2024-03-27T21:40:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.622+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign 1 3 2024-03-27T21:40:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.622+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c1da3d0 osd=1 4 2024-03-27T21:40:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.622+0000 7ff463ff6640 10 client.4755.objecter _send_command 3 2024-03-27T21:40:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.622+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff45c134bb0 con 0x7ff45c134290 2024-03-27T21:40:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.622+0000 7ff461d92640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7ff45c134290 0x7ff45c134670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.622+0000 7ff461d92640 10 monclient: get_auth_request con 0x7ff45c134290 auth_method 0 2024-03-27T21:40:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.622+0000 7ff4527fc640 10 client.4755.objecter ms_handle_connect 0x7ff45c134290 2024-03-27T21:40:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.622+0000 7ff461d92640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7ff45c134290 0x7ff45c134670 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.623+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7ff45c134bb0 con 0x7ff45c134290 2024-03-27T21:40:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.623+0000 7ff4527fc640 10 client.4755.objecter ms_dispatch 0x7ff45c000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.623+0000 7ff4527fc640 10 client.4755.objecter _finish_command 3 = system:0 2024-03-27T21:40:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.623+0000 7ff4527fc640 20 client.4755.objecter put_session s=0x7ff45c1da3d0 osd=1 4 2024-03-27T21:40:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.623+0000 7ff4527fc640 15 client.4755.objecter _session_command_op_remove 1 3 2024-03-27T21:40:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.637+0000 7ff463ff6640 10 client.4755.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.637+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign -1 4 2024-03-27T21:40:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.637+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c1da3d0 osd=1 3 2024-03-27T21:40:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.637+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c1da3d0 osd=1 3 2024-03-27T21:40:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.637+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c1da3d0 osd=1 3 2024-03-27T21:40:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.637+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_remove -1 4 2024-03-27T21:40:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.637+0000 7ff463ff6640 20 client.4755.objecter get_session s=0x7ff45c1da3d0 osd=1 3 2024-03-27T21:40:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.637+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign 1 4 2024-03-27T21:40:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.637+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c1da3d0 osd=1 4 2024-03-27T21:40:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.637+0000 7ff463ff6640 10 client.4755.objecter _send_command 4 2024-03-27T21:40:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.637+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff45c065a90 con 0x7ff45c134290 2024-03-27T21:40:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.639+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff45c065a90 con 0x7ff45c134290 2024-03-27T21:40:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.639+0000 7ff4527fc640 10 client.4755.objecter ms_dispatch 0x7ff45c000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.639+0000 7ff4527fc640 10 client.4755.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.639+0000 7ff4527fc640 20 client.4755.objecter put_session s=0x7ff45c1da3d0 osd=1 4 2024-03-27T21:40:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.639+0000 7ff4527fc640 15 client.4755.objecter _session_command_op_remove 1 4 2024-03-27T21:40:47.628 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:47.629 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:47.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.639+0000 7ff463ff6640 10 client.4755.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:47.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.639+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign -1 5 2024-03-27T21:40:47.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.639+0000 7ff463ff6640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7ff45c137720 0x7ff45c02b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:47.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.639+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c1375b0 osd=2 3 2024-03-27T21:40:47.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.639+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c1375b0 osd=2 3 2024-03-27T21:40:47.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.639+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c1375b0 osd=2 3 2024-03-27T21:40:47.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.639+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_remove -1 5 2024-03-27T21:40:47.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.639+0000 7ff463ff6640 20 client.4755.objecter get_session s=0x7ff45c1375b0 osd=2 3 2024-03-27T21:40:47.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.639+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign 2 5 2024-03-27T21:40:47.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.639+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c1375b0 osd=2 4 2024-03-27T21:40:47.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.639+0000 7ff463ff6640 10 client.4755.objecter _send_command 5 2024-03-27T21:40:47.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.639+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff45c066930 con 0x7ff45c137720 2024-03-27T21:40:47.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.639+0000 7ff461591640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7ff45c137720 0x7ff45c02b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:47.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.639+0000 7ff461591640 10 monclient: get_auth_request con 0x7ff45c137720 auth_method 0 2024-03-27T21:40:47.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.640+0000 7ff4527fc640 10 client.4755.objecter ms_handle_connect 0x7ff45c137720 2024-03-27T21:40:47.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.640+0000 7ff461591640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7ff45c137720 0x7ff45c02b170 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:47.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.640+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== osd.2 v2:172.21.15.60:6824/44654 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7ff45c066930 con 0x7ff45c137720 2024-03-27T21:40:47.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.640+0000 7ff4527fc640 10 client.4755.objecter ms_dispatch 0x7ff45c000b90 command_reply(tid 5: 0 ) v1 2024-03-27T21:40:47.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.640+0000 7ff4527fc640 10 client.4755.objecter _finish_command 5 = system:0 2024-03-27T21:40:47.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.640+0000 7ff4527fc640 20 client.4755.objecter put_session s=0x7ff45c1375b0 osd=2 4 2024-03-27T21:40:47.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.640+0000 7ff4527fc640 15 client.4755.objecter _session_command_op_remove 2 5 2024-03-27T21:40:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.654+0000 7ff463ff6640 10 client.4755.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.654+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign -1 6 2024-03-27T21:40:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.654+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c1375b0 osd=2 3 2024-03-27T21:40:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.654+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c1375b0 osd=2 3 2024-03-27T21:40:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.654+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c1375b0 osd=2 3 2024-03-27T21:40:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.654+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_remove -1 6 2024-03-27T21:40:47.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.654+0000 7ff463ff6640 20 client.4755.objecter get_session s=0x7ff45c1375b0 osd=2 3 2024-03-27T21:40:47.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.654+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign 2 6 2024-03-27T21:40:47.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.654+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c1375b0 osd=2 4 2024-03-27T21:40:47.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.654+0000 7ff463ff6640 10 client.4755.objecter _send_command 6 2024-03-27T21:40:47.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.654+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 --> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff45c066aa0 con 0x7ff45c137720 2024-03-27T21:40:47.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.656+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== osd.2 v2:172.21.15.60:6824/44654 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff44c009ea0 con 0x7ff45c137720 2024-03-27T21:40:47.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.656+0000 7ff4527fc640 10 client.4755.objecter ms_dispatch 0x7ff45c000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:47.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.656+0000 7ff4527fc640 10 client.4755.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:47.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.656+0000 7ff4527fc640 20 client.4755.objecter put_session s=0x7ff45c1375b0 osd=2 4 2024-03-27T21:40:47.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.656+0000 7ff4527fc640 15 client.4755.objecter _session_command_op_remove 2 6 2024-03-27T21:40:47.643 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2: {} 2024-03-27T21:40:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-27T21:40:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.656+0000 7ff463ff6640 10 client.4755.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.656+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign -1 7 2024-03-27T21:40:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.656+0000 7ff463ff6640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7ff45c140ca0 0x7ff45c1e8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.656+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c066080 osd=3 3 2024-03-27T21:40:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.656+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c066080 osd=3 3 2024-03-27T21:40:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.656+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c066080 osd=3 3 2024-03-27T21:40:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.656+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_remove -1 7 2024-03-27T21:40:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.656+0000 7ff463ff6640 20 client.4755.objecter get_session s=0x7ff45c066080 osd=3 3 2024-03-27T21:40:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.656+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign 3 7 2024-03-27T21:40:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.656+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c066080 osd=3 4 2024-03-27T21:40:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.656+0000 7ff463ff6640 10 client.4755.objecter _send_command 7 2024-03-27T21:40:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.656+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff45c0661f0 con 0x7ff45c140ca0 2024-03-27T21:40:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.656+0000 7ff462593640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7ff45c140ca0 0x7ff45c1e8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.656+0000 7ff462593640 10 monclient: get_auth_request con 0x7ff45c140ca0 auth_method 0 2024-03-27T21:40:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.656+0000 7ff4527fc640 10 client.4755.objecter ms_handle_connect 0x7ff45c140ca0 2024-03-27T21:40:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.656+0000 7ff462593640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7ff45c140ca0 0x7ff45c1e8df0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.657+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7ff45c0661f0 con 0x7ff45c140ca0 2024-03-27T21:40:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.657+0000 7ff4527fc640 10 client.4755.objecter ms_dispatch 0x7ff45c000b90 command_reply(tid 7: 0 ) v1 2024-03-27T21:40:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.657+0000 7ff4527fc640 10 client.4755.objecter _finish_command 7 = system:0 2024-03-27T21:40:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.657+0000 7ff4527fc640 20 client.4755.objecter put_session s=0x7ff45c066080 osd=3 4 2024-03-27T21:40:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.657+0000 7ff4527fc640 15 client.4755.objecter _session_command_op_remove 3 7 2024-03-27T21:40:47.654 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:47.654 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.671+0000 7ff463ff6640 10 client.4755.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.671+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign -1 8 2024-03-27T21:40:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.671+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c066080 osd=3 3 2024-03-27T21:40:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.671+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c066080 osd=3 3 2024-03-27T21:40:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.671+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c066080 osd=3 3 2024-03-27T21:40:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.671+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_remove -1 8 2024-03-27T21:40:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.671+0000 7ff463ff6640 20 client.4755.objecter get_session s=0x7ff45c066080 osd=3 3 2024-03-27T21:40:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.671+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign 3 8 2024-03-27T21:40:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.671+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c066080 osd=3 4 2024-03-27T21:40:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.671+0000 7ff463ff6640 10 client.4755.objecter _send_command 8 2024-03-27T21:40:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.671+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff45c140210 con 0x7ff45c140ca0 2024-03-27T21:40:47.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.672+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff45c140210 con 0x7ff45c140ca0 2024-03-27T21:40:47.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.672+0000 7ff4527fc640 10 client.4755.objecter ms_dispatch 0x7ff45c000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:47.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.672+0000 7ff4527fc640 10 client.4755.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:47.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.672+0000 7ff4527fc640 20 client.4755.objecter put_session s=0x7ff45c066080 osd=3 4 2024-03-27T21:40:47.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.672+0000 7ff4527fc640 15 client.4755.objecter _session_command_op_remove 3 8 2024-03-27T21:40:47.659 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:47.660 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:47.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.672+0000 7ff463ff6640 10 client.4755.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:47.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.672+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign -1 9 2024-03-27T21:40:47.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.672+0000 7ff463ff6640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7ff45c140380 0x7ff45c1ec4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:47.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.672+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c140210 osd=4 3 2024-03-27T21:40:47.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.672+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c140210 osd=4 3 2024-03-27T21:40:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.672+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c140210 osd=4 3 2024-03-27T21:40:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.672+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_remove -1 9 2024-03-27T21:40:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.672+0000 7ff463ff6640 20 client.4755.objecter get_session s=0x7ff45c140210 osd=4 3 2024-03-27T21:40:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.672+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign 4 9 2024-03-27T21:40:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.672+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c140210 osd=4 4 2024-03-27T21:40:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.672+0000 7ff463ff6640 10 client.4755.objecter _send_command 9 2024-03-27T21:40:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.672+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff45c02c380 con 0x7ff45c140380 2024-03-27T21:40:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.673+0000 7ff461d92640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7ff45c140380 0x7ff45c1ec4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.673+0000 7ff461d92640 10 monclient: get_auth_request con 0x7ff45c140380 auth_method 0 2024-03-27T21:40:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.673+0000 7ff4527fc640 10 client.4755.objecter ms_handle_connect 0x7ff45c140380 2024-03-27T21:40:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.673+0000 7ff461d92640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7ff45c140380 0x7ff45c1ec4c0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.674+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7ff45c02c380 con 0x7ff45c140380 2024-03-27T21:40:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.674+0000 7ff4527fc640 10 client.4755.objecter ms_dispatch 0x7ff45c000b90 command_reply(tid 9: 0 ) v1 2024-03-27T21:40:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.674+0000 7ff4527fc640 10 client.4755.objecter _finish_command 9 = system:0 2024-03-27T21:40:47.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.674+0000 7ff4527fc640 20 client.4755.objecter put_session s=0x7ff45c140210 osd=4 4 2024-03-27T21:40:47.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.674+0000 7ff4527fc640 15 client.4755.objecter _session_command_op_remove 4 9 2024-03-27T21:40:47.662 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:40:47.676+0000 7fa1b2621640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44633) UID: 0 2024-03-27T21:40:47.664 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:40:47.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.688+0000 7ff463ff6640 10 client.4755.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:47.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.688+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign -1 10 2024-03-27T21:40:47.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.688+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c140210 osd=4 3 2024-03-27T21:40:47.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.688+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c140210 osd=4 3 2024-03-27T21:40:47.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.688+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c140210 osd=4 3 2024-03-27T21:40:47.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.688+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_remove -1 10 2024-03-27T21:40:47.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.688+0000 7ff463ff6640 20 client.4755.objecter get_session s=0x7ff45c140210 osd=4 3 2024-03-27T21:40:47.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.688+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign 4 10 2024-03-27T21:40:47.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.688+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c140210 osd=4 4 2024-03-27T21:40:47.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.688+0000 7ff463ff6640 10 client.4755.objecter _send_command 10 2024-03-27T21:40:47.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.688+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff45c02c380 con 0x7ff45c140380 2024-03-27T21:40:47.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.690+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff45c02c380 con 0x7ff45c140380 2024-03-27T21:40:47.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.690+0000 7ff4527fc640 10 client.4755.objecter ms_dispatch 0x7ff45c000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:47.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.690+0000 7ff4527fc640 10 client.4755.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:47.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.690+0000 7ff4527fc640 20 client.4755.objecter put_session s=0x7ff45c140210 osd=4 4 2024-03-27T21:40:47.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.690+0000 7ff4527fc640 15 client.4755.objecter _session_command_op_remove 4 10 2024-03-27T21:40:47.676 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:47.678 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:47.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.690+0000 7ff463ff6640 10 client.4755.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:47.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.690+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign -1 11 2024-03-27T21:40:47.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.690+0000 7ff463ff6640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7ff45c1ecf10 0x7ff45c1ef300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:47.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.690+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c02c380 osd=5 3 2024-03-27T21:40:47.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.690+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c02c380 osd=5 3 2024-03-27T21:40:47.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.690+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c02c380 osd=5 3 2024-03-27T21:40:47.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.690+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_remove -1 11 2024-03-27T21:40:47.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.690+0000 7ff463ff6640 20 client.4755.objecter get_session s=0x7ff45c02c380 osd=5 3 2024-03-27T21:40:47.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.690+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign 5 11 2024-03-27T21:40:47.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.690+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c02c380 osd=5 4 2024-03-27T21:40:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.690+0000 7ff463ff6640 10 client.4755.objecter _send_command 11 2024-03-27T21:40:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.690+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff45c02c520 con 0x7ff45c1ecf10 2024-03-27T21:40:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.690+0000 7ff461591640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7ff45c1ecf10 0x7ff45c1ef300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.690+0000 7ff461591640 10 monclient: get_auth_request con 0x7ff45c1ecf10 auth_method 0 2024-03-27T21:40:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.691+0000 7ff4527fc640 10 client.4755.objecter ms_handle_connect 0x7ff45c1ecf10 2024-03-27T21:40:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.691+0000 7ff461591640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7ff45c1ecf10 0x7ff45c1ef300 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.691+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7ff45c02c520 con 0x7ff45c1ecf10 2024-03-27T21:40:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.692+0000 7ff4527fc640 10 client.4755.objecter ms_dispatch 0x7ff45c000b90 command_reply(tid 11: 0 ) v1 2024-03-27T21:40:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.692+0000 7ff4527fc640 10 client.4755.objecter _finish_command 11 = system:0 2024-03-27T21:40:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.692+0000 7ff4527fc640 20 client.4755.objecter put_session s=0x7ff45c02c380 osd=5 4 2024-03-27T21:40:47.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.692+0000 7ff4527fc640 15 client.4755.objecter _session_command_op_remove 5 11 2024-03-27T21:40:47.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.705+0000 7ff463ff6640 10 client.4755.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:47.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.705+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign -1 12 2024-03-27T21:40:47.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.705+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c02c380 osd=5 3 2024-03-27T21:40:47.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.705+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c02c380 osd=5 3 2024-03-27T21:40:47.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.705+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c02c380 osd=5 3 2024-03-27T21:40:47.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.705+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_remove -1 12 2024-03-27T21:40:47.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.705+0000 7ff463ff6640 20 client.4755.objecter get_session s=0x7ff45c02c380 osd=5 3 2024-03-27T21:40:47.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.705+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign 5 12 2024-03-27T21:40:47.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.705+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c02c380 osd=5 4 2024-03-27T21:40:47.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.705+0000 7ff463ff6640 10 client.4755.objecter _send_command 12 2024-03-27T21:40:47.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.705+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff45c02c6c0 con 0x7ff45c1ecf10 2024-03-27T21:40:47.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.707+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff45c02c6c0 con 0x7ff45c1ecf10 2024-03-27T21:40:47.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.707+0000 7ff4527fc640 10 client.4755.objecter ms_dispatch 0x7ff45c000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:47.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.707+0000 7ff4527fc640 10 client.4755.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:47.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.707+0000 7ff4527fc640 20 client.4755.objecter put_session s=0x7ff45c02c380 osd=5 4 2024-03-27T21:40:47.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.707+0000 7ff4527fc640 15 client.4755.objecter _session_command_op_remove 5 12 2024-03-27T21:40:47.694 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.707+0000 7ff463ff6640 10 client.4755.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.707+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign -1 13 2024-03-27T21:40:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.707+0000 7ff463ff6640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7ff45c1efde0 0x7ff45c1f2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.707+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c02c520 osd=6 3 2024-03-27T21:40:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.707+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c02c520 osd=6 3 2024-03-27T21:40:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.707+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c02c520 osd=6 3 2024-03-27T21:40:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.707+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_remove -1 13 2024-03-27T21:40:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.707+0000 7ff463ff6640 20 client.4755.objecter get_session s=0x7ff45c02c520 osd=6 3 2024-03-27T21:40:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.707+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign 6 13 2024-03-27T21:40:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.707+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c02c520 osd=6 4 2024-03-27T21:40:47.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.707+0000 7ff463ff6640 10 client.4755.objecter _send_command 13 2024-03-27T21:40:47.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.707+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff45c02c6c0 con 0x7ff45c1efde0 2024-03-27T21:40:47.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.707+0000 7ff462593640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7ff45c1efde0 0x7ff45c1f2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:47.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.707+0000 7ff462593640 10 monclient: get_auth_request con 0x7ff45c1efde0 auth_method 0 2024-03-27T21:40:47.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.708+0000 7ff4527fc640 10 client.4755.objecter ms_handle_connect 0x7ff45c1efde0 2024-03-27T21:40:47.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.708+0000 7ff462593640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7ff45c1efde0 0x7ff45c1f2250 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:47.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.708+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7ff45c02c6c0 con 0x7ff45c1efde0 2024-03-27T21:40:47.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.708+0000 7ff4527fc640 10 client.4755.objecter ms_dispatch 0x7ff45c000b90 command_reply(tid 13: 0 ) v1 2024-03-27T21:40:47.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.708+0000 7ff4527fc640 10 client.4755.objecter _finish_command 13 = system:0 2024-03-27T21:40:47.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.708+0000 7ff4527fc640 20 client.4755.objecter put_session s=0x7ff45c02c520 osd=6 4 2024-03-27T21:40:47.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.708+0000 7ff4527fc640 15 client.4755.objecter _session_command_op_remove 6 13 2024-03-27T21:40:47.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.722+0000 7ff463ff6640 10 client.4755.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:47.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.722+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign -1 14 2024-03-27T21:40:47.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.722+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c02c520 osd=6 3 2024-03-27T21:40:47.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.722+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c02c520 osd=6 3 2024-03-27T21:40:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.722+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c02c520 osd=6 3 2024-03-27T21:40:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.722+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_remove -1 14 2024-03-27T21:40:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.722+0000 7ff463ff6640 20 client.4755.objecter get_session s=0x7ff45c02c520 osd=6 3 2024-03-27T21:40:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.722+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign 6 14 2024-03-27T21:40:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.722+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c02c520 osd=6 4 2024-03-27T21:40:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.722+0000 7ff463ff6640 10 client.4755.objecter _send_command 14 2024-03-27T21:40:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.722+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff45c02c6c0 con 0x7ff45c1efde0 2024-03-27T21:40:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.724+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff45c02c6c0 con 0x7ff45c1efde0 2024-03-27T21:40:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.724+0000 7ff4527fc640 10 client.4755.objecter ms_dispatch 0x7ff45c000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.724+0000 7ff4527fc640 10 client.4755.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.724+0000 7ff4527fc640 20 client.4755.objecter put_session s=0x7ff45c02c520 osd=6 4 2024-03-27T21:40:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.724+0000 7ff4527fc640 15 client.4755.objecter _session_command_op_remove 6 14 2024-03-27T21:40:47.711 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.724+0000 7ff463ff6640 10 client.4755.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.724+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign -1 15 2024-03-27T21:40:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.724+0000 7ff463ff6640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7ff45c1f4540 0x7ff45c1f49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.724+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c02c6c0 osd=7 3 2024-03-27T21:40:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.724+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c02c6c0 osd=7 3 2024-03-27T21:40:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.724+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c02c6c0 osd=7 3 2024-03-27T21:40:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.724+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_remove -1 15 2024-03-27T21:40:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.724+0000 7ff463ff6640 20 client.4755.objecter get_session s=0x7ff45c02c6c0 osd=7 3 2024-03-27T21:40:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.724+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign 7 15 2024-03-27T21:40:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.724+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c02c6c0 osd=7 4 2024-03-27T21:40:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.724+0000 7ff463ff6640 10 client.4755.objecter _send_command 15 2024-03-27T21:40:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.724+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff45c1f83d0 con 0x7ff45c1f4540 2024-03-27T21:40:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.724+0000 7ff461d92640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7ff45c1f4540 0x7ff45c1f49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.724+0000 7ff461d92640 10 monclient: get_auth_request con 0x7ff45c1f4540 auth_method 0 2024-03-27T21:40:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.725+0000 7ff461d92640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7ff45c1f4540 0x7ff45c1f49c0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.725+0000 7ff4527fc640 10 client.4755.objecter ms_handle_connect 0x7ff45c1f4540 2024-03-27T21:40:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.725+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7ff45c1f83d0 con 0x7ff45c1f4540 2024-03-27T21:40:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.725+0000 7ff4527fc640 10 client.4755.objecter ms_dispatch 0x7ff45c000b90 command_reply(tid 15: 0 ) v1 2024-03-27T21:40:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.725+0000 7ff4527fc640 10 client.4755.objecter _finish_command 15 = system:0 2024-03-27T21:40:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.725+0000 7ff4527fc640 20 client.4755.objecter put_session s=0x7ff45c02c6c0 osd=7 4 2024-03-27T21:40:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.725+0000 7ff4527fc640 15 client.4755.objecter _session_command_op_remove 7 15 2024-03-27T21:40:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.739+0000 7ff463ff6640 10 client.4755.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.739+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign -1 16 2024-03-27T21:40:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.739+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c02c6c0 osd=7 3 2024-03-27T21:40:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.739+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c02c6c0 osd=7 3 2024-03-27T21:40:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.739+0000 7ff463ff6640 20 client.4755.objecter _get_session s=0x7ff45c02c6c0 osd=7 3 2024-03-27T21:40:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.739+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_remove -1 16 2024-03-27T21:40:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.739+0000 7ff463ff6640 20 client.4755.objecter get_session s=0x7ff45c02c6c0 osd=7 3 2024-03-27T21:40:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.739+0000 7ff463ff6640 15 client.4755.objecter _session_command_op_assign 7 16 2024-03-27T21:40:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.739+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c02c6c0 osd=7 4 2024-03-27T21:40:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.739+0000 7ff463ff6640 10 client.4755.objecter _send_command 16 2024-03-27T21:40:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.739+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff45c1f28a0 con 0x7ff45c1f4540 2024-03-27T21:40:47.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff4527fc640 1 -- 172.21.15.60:0/682874290 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7ff45c1f28a0 con 0x7ff45c1f4540 2024-03-27T21:40:47.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff4527fc640 10 client.4755.objecter ms_dispatch 0x7ff45c000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:47.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff4527fc640 10 client.4755.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:47.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff4527fc640 20 client.4755.objecter put_session s=0x7ff45c02c6c0 osd=7 4 2024-03-27T21:40:47.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff4527fc640 15 client.4755.objecter _session_command_op_remove 7 16 2024-03-27T21:40:47.728 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 10 librados: watch_flush enter 2024-03-27T21:40:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 10 librados: watch_flush exit 2024-03-27T21:40:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 10 client.4755.objecter close_session for osd.0 2024-03-27T21:40:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7ff45c1d9870 msgr2=0x7ff45c1d9c50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7ff45c1d9870 0x7ff45c1d9c50 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c0a90d0 osd=0 1 2024-03-27T21:40:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 10 client.4755.objecter close_session for osd.1 2024-03-27T21:40:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7ff45c134290 msgr2=0x7ff45c134670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7ff45c134290 0x7ff45c134670 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c1da3d0 osd=1 1 2024-03-27T21:40:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 10 client.4755.objecter close_session for osd.2 2024-03-27T21:40:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7ff45c137720 msgr2=0x7ff45c02b170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6824/44654,v1:172.21.15.60:6825/44654] conn(0x7ff45c137720 0x7ff45c02b170 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c1375b0 osd=2 1 2024-03-27T21:40:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 10 client.4755.objecter close_session for osd.3 2024-03-27T21:40:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7ff45c140ca0 msgr2=0x7ff45c1e8df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff462593640 1 -- 172.21.15.60:0/682874290 reap_dead start 2024-03-27T21:40:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7ff45c140ca0 0x7ff45c1e8df0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c066080 osd=3 1 2024-03-27T21:40:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 10 client.4755.objecter close_session for osd.4 2024-03-27T21:40:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7ff45c140380 msgr2=0x7ff45c1ec4c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7ff45c140380 0x7ff45c1ec4c0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c140210 osd=4 1 2024-03-27T21:40:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 10 client.4755.objecter close_session for osd.5 2024-03-27T21:40:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7ff45c1ecf10 msgr2=0x7ff45c1ef300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7ff45c1ecf10 0x7ff45c1ef300 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c02c380 osd=5 1 2024-03-27T21:40:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 10 client.4755.objecter close_session for osd.6 2024-03-27T21:40:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7ff45c1efde0 msgr2=0x7ff45c1f2250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.741+0000 7ff463ff6640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7ff45c1efde0 0x7ff45c1f2250 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.742+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c02c520 osd=6 1 2024-03-27T21:40:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.742+0000 7ff463ff6640 10 client.4755.objecter close_session for osd.7 2024-03-27T21:40:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.742+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7ff45c1f4540 msgr2=0x7ff45c1f49c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.742+0000 7ff463ff6640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7ff45c1f4540 0x7ff45c1f49c0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.742+0000 7ff462593640 1 -- 172.21.15.60:0/682874290 reap_dead start 2024-03-27T21:40:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.742+0000 7ff463ff6640 20 client.4755.objecter put_session s=0x7ff45c02c6c0 osd=7 1 2024-03-27T21:40:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.742+0000 7ff463ff6640 20 client.4755.objecter shutdown clearing up homeless session... 2024-03-27T21:40:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.742+0000 7ff463ff6640 10 client.4755.objecter successfully canceled tick 2024-03-27T21:40:47.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.742+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff44006c590 msgr2=0x7ff44006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:47.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.742+0000 7ff463ff6640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff44006c590 0x7ff44006ea50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff45c0b5f60 tx=0x7ff44c008040 comp rx=0 tx=0).stop 2024-03-27T21:40:47.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.742+0000 7ff463ff6640 10 monclient: shutdown 2024-03-27T21:40:47.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.742+0000 7ff463ff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:47.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.742+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff45c0b40b0 msgr2=0x7ff45c13df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:47.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.742+0000 7ff463ff6640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff45c0b40b0 0x7ff45c13df30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff458000f80 tx=0x7ff458004a40 comp rx=0 tx=0).stop 2024-03-27T21:40:47.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.742+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 shutdown_connections 2024-03-27T21:40:47.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.742+0000 7ff463ff6640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff44006c590 0x7ff44006ea50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:47.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.742+0000 7ff463ff6640 1 --2- 172.21.15.60:0/682874290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff45c0b40b0 0x7ff45c13df30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:47.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.742+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 >> 172.21.15.60:0/682874290 conn(0x7ff45c0608e0 msgr2=0x7ff45c068be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:47.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.742+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 shutdown_connections 2024-03-27T21:40:47.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.742+0000 7ff463ff6640 1 -- 172.21.15.60:0/682874290 wait complete. 2024-03-27T21:40:47.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:47.742+0000 7ff463ff6640 1 librados: shutdown 2024-03-27T21:40:47.764 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:47.777+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:47.782 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:47.782 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:47.782 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:47.782 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:47.793 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:40:47.875 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:47.889+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:47.977 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:47.990+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:47.983 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:47.983 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:47.983 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:47.983 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:47.983 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:47.992 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:40:48.077 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:48.092+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:48.138 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:48.138 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:48.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:48.138 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:48.138 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:48.147 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:40:48.192 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:48.206+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:48.254 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:48.254 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:48.255 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:48.255 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:48.262 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:40:48.279 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:48.293+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:48.379 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:48.394+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:48.408 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:48.408 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:48.408 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:48.408 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:48.408 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:48.416 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:40:48.480 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:48.494+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:48.521 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:48.521 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:48.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:48.521 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:48.522 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:48.527 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-03-27T21:40:48.527 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T21:40:48.527 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 3, 4, 5, 6, 7, 2] 2024-03-27T21:40:48.527 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-27T21:40:48.527 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T21:40:48.530 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:40:48.558 INFO:tasks.ceph.osd.2:Stopped 2024-03-27T21:40:48.558 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-03-27T21:40:48.558 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi060.front.sepia.ceph.com 2024-03-27T21:40:48.558 DEBUG:teuthology.orchestra.run.smithi060:> type ceph-objectstore-tool 2024-03-27T21:40:48.599 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:48.613+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:48.615 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-03-27T21:40:48.615 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-03-27T21:40:48.635 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:48.635 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:48.635 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:48.635 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:48.644 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:40:48.687 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:48.701+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:48.748 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:48.749 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:48.749 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:48.749 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:48.749 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:48.757 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:40:48.799 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:48.812+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:48.883 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:48.897+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:48.902 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:48.902 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:48.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:48.902 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:48.902 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:48.911 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:40:48.926 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-27T21:40:48.995 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:49.009+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:49.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.061+0000 7f5551edc640 10 monclient: get_monmap_and_config 2024-03-27T21:40:49.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.061+0000 7f5551edc640 10 monclient: build_initial_monmap 2024-03-27T21:40:49.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.061+0000 7f5551edc640 10 monclient: monmap: 2024-03-27T21:40:49.048 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:49.048 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:49.048 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:49.062356+0000 2024-03-27T21:40:49.048 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:49.062356+0000 2024-03-27T21:40:49.048 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:49.048 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:49.048 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:49.048 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:49.048 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:49.048 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:49.048 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:49.048 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:49.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.062+0000 7f5551edc640 1 Processor -- start 2024-03-27T21:40:49.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.062+0000 7f5551edc640 1 -- start start 2024-03-27T21:40:49.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.062+0000 7f5551edc640 10 monclient: init 2024-03-27T21:40:49.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f5551edc640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:49.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f5551edc640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-03-27T21:40:49.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f5551edc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f554c0b40b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:49.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f5551edc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f554c13b020 0x7f554c13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:49.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f5551edc640 10 monclient(hunting): picked mon.noname-a con 0x7f554c13b020 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:49.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f5551edc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f554c1416f0 0x7f554c141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:49.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f5551edc640 10 monclient(hunting): picked mon.noname-b con 0x7f554c1416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:49.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f5551edc640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554c0b3830 con 0x7f554c0b40b0 2024-03-27T21:40:49.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f5551edc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:49.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f5551edc640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f554c0b4490 con 0x7f554c13b020 2024-03-27T21:40:49.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f5551edc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:49.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f5551edc640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f554c0aa520 con 0x7f554c1416f0 2024-03-27T21:40:49.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f5551edc640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:49.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f5551edc640 10 monclient(hunting): authenticate will time out at 890.284321s 2024-03-27T21:40:49.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f554c13b020 0x7f554c13b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:49.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554bfff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f554c1416f0 0x7f554c141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:49.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f554c13b020 0x7f554c13b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39804/0 (socket says 172.21.15.60:39804) 2024-03-27T21:40:49.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554b7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f554c0b40b0 0x7f554c0a8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:48060/0 (socket says 172.21.15.60:48060) 2024-03-27T21:40:49.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554affd640 1 -- 172.21.15.60:0/2086704556 learned_addr learned my addr 172.21.15.60:0/2086704556 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:49.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554affd640 10 monclient(hunting): get_auth_request con 0x7f554c13b020 auth_method 0 2024-03-27T21:40:49.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:49.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554affd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:49.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:49.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554bfff640 10 monclient(hunting): get_auth_request con 0x7f554c1416f0 auth_method 0 2024-03-27T21:40:49.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:49.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:49.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:49.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:49.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:49.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:49.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:49.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:49.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:49.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554affd640 10 monclient(hunting): handle_auth_done global_id 4758 payload 274 2024-03-27T21:40:49.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554affd640 1 -- 172.21.15.60:0/2086704556 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f554c1416f0 msgr2=0x7f554c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:49.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554affd640 1 --2- 172.21.15.60:0/2086704556 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f554c1416f0 0x7f554c141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:49.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554affd640 1 -- 172.21.15.60:0/2086704556 >> v1:172.21.15.60:6789/0 conn(0x7f554c0b40b0 legacy=0x7f554c0a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:49.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554affd640 10 monclient: _finish_hunting 0 2024-03-27T21:40:49.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554affd640 1 monclient: found mon.noname-a 2024-03-27T21:40:49.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:49.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:49.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554affd640 1 -- 172.21.15.60:0/2086704556 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c0a9470 con 0x7f554c13b020 2024-03-27T21:40:49.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554a7fc640 1 -- 172.21.15.60:0/2086704556 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257590109 0 0) 0x7f554c0b3830 con 0x7f554c0b40b0 2024-03-27T21:40:49.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554a7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:40:49.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.063+0000 7f554bfff640 1 --2- 172.21.15.60:0/2086704556 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f554c1416f0 0x7f554c141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:49.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.064+0000 7f554affd640 1 --2- 172.21.15.60:0/2086704556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554c13b020 0x7f554c13b480 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5540009870 tx=0x7f554002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38979c0970acb498 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:49.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.064+0000 7f554a7fc640 1 -- 172.21.15.60:0/2086704556 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5540030840 con 0x7f554c13b020 2024-03-27T21:40:49.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.064+0000 7f554a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:49.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.064+0000 7f554a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:49.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.064+0000 7f554a7fc640 10 monclient: dump: 2024-03-27T21:40:49.063 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:49.063 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:49.063 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:49.063 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:49.063 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:49.063 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:49.063 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:49.063 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:49.064 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:49.064 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:49.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.064+0000 7f554a7fc640 10 monclient: _finish_auth 0 2024-03-27T21:40:49.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.064+0000 7f554a7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:40:49.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.064+0000 7f554a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:49.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.064+0000 7f5551edc640 5 monclient: authenticate success, global_id 4758 2024-03-27T21:40:49.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.064+0000 7f554a7fc640 1 -- 172.21.15.60:0/2086704556 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55400309a0 con 0x7f554c13b020 2024-03-27T21:40:49.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.064+0000 7f554a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:49.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.064+0000 7f554a7fc640 1 -- 172.21.15.60:0/2086704556 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5540030d30 con 0x7f554c13b020 2024-03-27T21:40:49.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.064+0000 7f554a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:49.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.064+0000 7f554a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:49.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.064+0000 7f554a7fc640 10 monclient: dump: 2024-03-27T21:40:49.065 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:49.065 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:49.065 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:49.065 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:49.065 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:49.065 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:49.065 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:49.065 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:49.065 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:49.065 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:49.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.064+0000 7f5551edc640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:49.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.064+0000 7f5551edc640 4 set_mon_vals no callback set 2024-03-27T21:40:49.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.065+0000 7f5551edc640 10 monclient: shutdown 2024-03-27T21:40:49.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.065+0000 7f5551edc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:49.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.065+0000 7f5551edc640 1 -- 172.21.15.60:0/2086704556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554c13b020 msgr2=0x7f554c13b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:49.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.065+0000 7f5551edc640 1 --2- 172.21.15.60:0/2086704556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554c13b020 0x7f554c13b480 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5540009870 tx=0x7f554002b980 comp rx=0 tx=0).stop 2024-03-27T21:40:49.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.065+0000 7f5551edc640 1 -- 172.21.15.60:0/2086704556 shutdown_connections 2024-03-27T21:40:49.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.065+0000 7f5551edc640 1 --2- 172.21.15.60:0/2086704556 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f554c1416f0 0x7f554c141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:49.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.065+0000 7f5551edc640 1 --2- 172.21.15.60:0/2086704556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554c13b020 0x7f554c13b480 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:49.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.065+0000 7f5551edc640 1 -- 172.21.15.60:0/2086704556 >> 172.21.15.60:0/2086704556 conn(0x7f554c0608e0 msgr2=0x7f554c060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:49.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.065+0000 7f5551edc640 1 -- 172.21.15.60:0/2086704556 shutdown_connections 2024-03-27T21:40:49.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.065+0000 7f5551edc640 1 -- 172.21.15.60:0/2086704556 wait complete. 2024-03-27T21:40:49.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.066+0000 7f5551edc640 10 monclient: build_initial_monmap 2024-03-27T21:40:49.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.066+0000 7f5551edc640 10 monclient: monmap: 2024-03-27T21:40:49.067 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:49.067 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:49.067 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:49.067 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:49.067 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:49.067 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:49.067 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:49.067 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:49.067 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:49.067 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:49.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.066+0000 7f5551edc640 1 librados: starting msgr at 2024-03-27T21:40:49.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.066+0000 7f5551edc640 1 librados: starting objecter 2024-03-27T21:40:49.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.066+0000 7f5551edc640 1 Processor -- start 2024-03-27T21:40:49.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.066+0000 7f5551edc640 1 -- start start 2024-03-27T21:40:49.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.066+0000 7f5551edc640 1 librados: setting wanted keys 2024-03-27T21:40:49.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.066+0000 7f5551edc640 1 librados: calling monclient init 2024-03-27T21:40:49.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.066+0000 7f5551edc640 10 monclient: init 2024-03-27T21:40:49.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f5551edc640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:49.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f5551edc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:40:49.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f5551edc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f554c0b40b0 0x7f554c134690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:49.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f5551edc640 10 monclient(hunting): picked mon.noname-c con 0x7f554c0b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:49.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f5551edc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f554c13b020 0x7f554c13aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:49.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f5551edc640 10 monclient(hunting): picked mon.noname-b con 0x7f554c13b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:49.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f5551edc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554c1416f0 0x7f554c131110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:49.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f5551edc640 10 monclient(hunting): picked mon.noname-a con 0x7f554c1416f0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:49.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f5551edc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:49.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f5551edc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554c13bdb0 con 0x7f554c1416f0 2024-03-27T21:40:49.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f5551edc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:49.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f5551edc640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554c139980 con 0x7f554c13b020 2024-03-27T21:40:49.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f5551edc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:49.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f5551edc640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f554c13ba60 con 0x7f554c0b40b0 2024-03-27T21:40:49.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f5551edc640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:49.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f5551edc640 10 monclient(hunting): authenticate will time out at 890.288510s 2024-03-27T21:40:49.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554c1416f0 0x7f554c131110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:49.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554c1416f0 0x7f554c131110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39816/0 (socket says 172.21.15.60:39816) 2024-03-27T21:40:49.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554bfff640 1 -- 172.21.15.60:0/3726029195 learned_addr learned my addr 172.21.15.60:0/3726029195 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:49.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554bfff640 10 monclient(hunting): get_auth_request con 0x7f554c1416f0 auth_method 0 2024-03-27T21:40:49.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:49.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:49.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:49.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554b7fe640 1 --2- 172.21.15.60:0/3726029195 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f554c0b40b0 0x7f554c134690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:49.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554affd640 1 --2- 172.21.15.60:0/3726029195 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f554c13b020 0x7f554c13aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:49.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:49.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:49.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:49.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554b7fe640 10 monclient(hunting): get_auth_request con 0x7f554c0b40b0 auth_method 0 2024-03-27T21:40:49.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:49.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:49.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:49.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554affd640 10 monclient(hunting): get_auth_request con 0x7f554c13b020 auth_method 0 2024-03-27T21:40:49.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:49.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554affd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:49.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:49.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:49.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.067+0000 7f554b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:49.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f554b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:49.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f554affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:49.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f554affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:49.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f554affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:49.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f554bfff640 10 monclient(hunting): handle_auth_done global_id 4764 payload 824 2024-03-27T21:40:49.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f554bfff640 1 -- 172.21.15.60:0/3726029195 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f554c0b40b0 msgr2=0x7f554c134690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:49.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f554bfff640 1 --2- 172.21.15.60:0/3726029195 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f554c0b40b0 0x7f554c134690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:49.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f554bfff640 1 -- 172.21.15.60:0/3726029195 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f554c13b020 msgr2=0x7f554c13aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:49.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f554bfff640 1 --2- 172.21.15.60:0/3726029195 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f554c13b020 0x7f554c13aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:49.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f554bfff640 10 monclient: _finish_hunting 0 2024-03-27T21:40:49.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f554bfff640 1 monclient: found mon.noname-a 2024-03-27T21:40:49.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f554bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:49.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f554bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:49.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f554bfff640 1 -- 172.21.15.60:0/3726029195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5544009a70 con 0x7f554c1416f0 2024-03-27T21:40:49.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f554bfff640 1 --2- 172.21.15.60:0/3726029195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554c1416f0 0x7f554c131110 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f554400efc0 tx=0x7f554400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:49.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f5533fff640 10 client.?.objecter ms_handle_connect 0x7f554c1416f0 2024-03-27T21:40:49.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f5533fff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:49.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f5533fff640 1 -- 172.21.15.60:0/3726029195 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5544009cd0 con 0x7f554c1416f0 2024-03-27T21:40:49.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f5533fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:49.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f5533fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:49.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f5533fff640 10 monclient: dump: 2024-03-27T21:40:49.073 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:49.073 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:49.073 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:49.073 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:49.073 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:49.073 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:49.073 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:49.073 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:49.074 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:49.074 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:49.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f5533fff640 10 monclient: _finish_auth 0 2024-03-27T21:40:49.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f5533fff640 10 monclient: _check_auth_tickets 2024-03-27T21:40:49.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f5533fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:49.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f5533fff640 1 -- 172.21.15.60:0/3726029195 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f554400f040 con 0x7f554c1416f0 2024-03-27T21:40:49.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f5551edc640 5 monclient: authenticate success, global_id 4764 2024-03-27T21:40:49.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f5551edc640 10 monclient: _renew_subs 2024-03-27T21:40:49.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f5551edc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:49.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f5551edc640 1 -- 172.21.15.60:0/3726029195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55400094d0 con 0x7f554c1416f0 2024-03-27T21:40:49.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f5533fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:49.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f5533fff640 1 -- 172.21.15.60:0/3726029195 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5544003990 con 0x7f554c1416f0 2024-03-27T21:40:49.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f5533fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:49.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f5533fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:49.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f5549ffb640 4 set_mon_vals no callback set 2024-03-27T21:40:49.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.068+0000 7f5533fff640 10 monclient: dump: 2024-03-27T21:40:49.075 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:49.075 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:49.075 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:49.075 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:49.075 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:49.075 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:49.075 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:49.075 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:49.075 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:49.075 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:49.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.069+0000 7f5551edc640 10 client.4764.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:49.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.069+0000 7f5551edc640 10 monclient: _renew_subs 2024-03-27T21:40:49.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.069+0000 7f5551edc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:49.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.069+0000 7f5551edc640 1 -- 172.21.15.60:0/3726029195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f554c131650 con 0x7f554c1416f0 2024-03-27T21:40:49.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.069+0000 7f5551edc640 1 librados: init done 2024-03-27T21:40:49.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.070+0000 7f5533fff640 1 -- 172.21.15.60:0/3726029195 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5544014480 con 0x7f554c1416f0 2024-03-27T21:40:49.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.070+0000 7f5551edc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:49.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.070+0000 7f5551edc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:49.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.070+0000 7f5551edc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:49.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.070+0000 7f5551edc640 1 -- 172.21.15.60:0/3726029195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f554c13d760 con 0x7f554c1416f0 2024-03-27T21:40:49.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.071+0000 7f5533fff640 1 --2- 172.21.15.60:0/3726029195 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f553806c5e0 0x7f553806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:49.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.071+0000 7f5533fff640 1 -- 172.21.15.60:0/3726029195 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f5544089ac0 con 0x7f554c1416f0 2024-03-27T21:40:49.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.071+0000 7f5533fff640 10 client.4764.objecter ms_dispatch 0x7f554c000b90 osd_map(45..45 src has 1..45) v4 2024-03-27T21:40:49.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.071+0000 7f5533fff640 3 client.4764.objecter handle_osd_map got epochs [45,45] > 0 2024-03-27T21:40:49.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.071+0000 7f5533fff640 3 client.4764.objecter handle_osd_map decoding full epoch 45 2024-03-27T21:40:49.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.071+0000 7f5533fff640 20 client.4764.objecter dump_active .. 0 homeless 2024-03-27T21:40:49.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.071+0000 7f554b7fe640 1 --2- 172.21.15.60:0/3726029195 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f553806c5e0 0x7f553806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:49.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.071+0000 7f554b7fe640 10 monclient: get_auth_request con 0x7f553806c5e0 auth_method 0 2024-03-27T21:40:49.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.071+0000 7f554b7fe640 1 --2- 172.21.15.60:0/3726029195 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f553806c5e0 0x7f553806eaa0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f553c000f60 tx=0x7f553c0017f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:49.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.071+0000 7f5533fff640 10 client.4764.objecter ms_handle_connect 0x7f553806c5e0 2024-03-27T21:40:49.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.075+0000 7f5533fff640 1 -- 172.21.15.60:0/3726029195 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5544055090 con 0x7f554c1416f0 2024-03-27T21:40:49.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.075+0000 7f5533fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:49.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.075+0000 7f5533fff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:49.078 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:49.079 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:49.079 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:49.079 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:49.079 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:40:49.119 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:49.133+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:49.187 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:49.200+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:49.223 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:49.223 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:49.223 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:49.223 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:49.223 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:49.232 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:40:49.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.276+0000 7f5551edc640 10 start_mon_command cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:40:49.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.277+0000 7f5551edc640 10 monclient: _send_command 2 [{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:40:49.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.277+0000 7f5551edc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:49.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.277+0000 7f5551edc640 1 -- 172.21.15.60:0/3726029195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f554c13bc20 con 0x7f554c1416f0 2024-03-27T21:40:49.316 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:49.330+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:49.379 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:49.379 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:49.379 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:49.379 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-03-27T21:40:49.379 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.380 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:49.380 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:49.380 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 69.959728561999995, 2024-03-27T21:40:49.380 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43537978700000002, 2024-03-27T21:40:49.380 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:49.380 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:49.380 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:49.380 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:49.380 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:49.380 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-03-27T21:40:49.380 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.380 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:49.380 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.381 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:49.381 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:49.381 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.381 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.381 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.381 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:49.381 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:49.381 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.381 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.381 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.381 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:49.382 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432019+0000", 2024-03-27T21:40:49.382 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-03-27T21:40:49.382 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.382 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.382 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:49.382 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432027+0000", 2024-03-27T21:40:49.382 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1200000000000002e-06 2024-03-27T21:40:49.382 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.382 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.382 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:49.382 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432039+0000", 2024-03-27T21:40:49.382 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1123e-05 2024-03-27T21:40:49.382 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.383 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:49.383 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432051+0000", 2024-03-27T21:40:49.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-03-27T21:40:49.383 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.383 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:49.383 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432157+0000", 2024-03-27T21:40:49.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106846 2024-03-27T21:40:49.383 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.383 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-03-27T21:40:49.383 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432176+0000", 2024-03-27T21:40:49.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8490000000000001e-05 2024-03-27T21:40:49.383 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.384 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:49.384 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866170+0000", 2024-03-27T21:40:49.384 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43399370100000001 2024-03-27T21:40:49.384 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.384 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.384 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:49.384 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866335+0000", 2024-03-27T21:40:49.384 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016559699999999999 2024-03-27T21:40:49.384 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.384 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.384 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:49.384 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866495+0000", 2024-03-27T21:40:49.384 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015944200000000001 2024-03-27T21:40:49.384 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.384 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.385 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:49.385 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867012+0000", 2024-03-27T21:40:49.385 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0005176 2024-03-27T21:40:49.385 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.385 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.385 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:49.385 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867334+0000", 2024-03-27T21:40:49.385 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032158600000000001 2024-03-27T21:40:49.385 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.385 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.385 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:49.385 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867374+0000", 2024-03-27T21:40:49.385 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0547000000000003e-05 2024-03-27T21:40:49.385 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.386 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.386 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:49.386 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867400+0000", 2024-03-27T21:40:49.386 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5958000000000001e-05 2024-03-27T21:40:49.386 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.386 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:49.386 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.386 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.386 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.386 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:49.386 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:49.386 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 69.462346111000002, 2024-03-27T21:40:49.386 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064560310000000001, 2024-03-27T21:40:49.387 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:49.387 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:49.387 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:49.387 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:49.387 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:49.387 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 70 2024-03-27T21:40:49.387 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.387 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:49.387 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.387 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:49.387 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:49.387 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.387 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.387 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.387 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:49.387 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:49.388 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.388 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.388 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.388 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:49.388 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929402+0000", 2024-03-27T21:40:49.388 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:49.388 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.388 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.388 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:49.388 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929407+0000", 2024-03-27T21:40:49.388 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-06 2024-03-27T21:40:49.388 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.388 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.388 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:49.389 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929414+0000", 2024-03-27T21:40:49.389 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5689999999999999e-06 2024-03-27T21:40:49.389 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.389 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.389 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:49.389 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929419+0000", 2024-03-27T21:40:49.389 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-03-27T21:40:49.389 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.389 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.389 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:49.389 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930245+0000", 2024-03-27T21:40:49.389 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082521500000000002 2024-03-27T21:40:49.389 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.390 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.390 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:49.390 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930287+0000", 2024-03-27T21:40:49.391 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2305000000000002e-05 2024-03-27T21:40:49.391 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.391 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.391 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:49.391 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930367+0000", 2024-03-27T21:40:49.391 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9589e-05 2024-03-27T21:40:49.391 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.391 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.391 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:49.391 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.931214+0000", 2024-03-27T21:40:49.391 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000847827 2024-03-27T21:40:49.391 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.391 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.391 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:49.391 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935807+0000", 2024-03-27T21:40:49.392 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045923780000000003 2024-03-27T21:40:49.392 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.392 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.392 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:49.392 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935840+0000", 2024-03-27T21:40:49.392 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3501999999999999e-05 2024-03-27T21:40:49.392 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.392 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.392 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:49.392 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935859+0000", 2024-03-27T21:40:49.392 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8845e-05 2024-03-27T21:40:49.392 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.392 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:49.392 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.392 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.392 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.393 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:49.393 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:49.393 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 69.455086519999995, 2024-03-27T21:40:49.393 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071695429999999996, 2024-03-27T21:40:49.393 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:49.393 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:49.393 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:49.393 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:49.393 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:49.393 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-03-27T21:40:49.393 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.393 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:49.393 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.394 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:49.394 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:49.394 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.394 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.394 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.394 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:49.394 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:49.394 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.394 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.394 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.394 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:49.394 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936662+0000", 2024-03-27T21:40:49.394 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:49.394 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.394 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.395 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:49.395 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936737+0000", 2024-03-27T21:40:49.395 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5303999999999996e-05 2024-03-27T21:40:49.395 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.395 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.395 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:49.395 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936746+0000", 2024-03-27T21:40:49.395 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6340000000000007e-06 2024-03-27T21:40:49.395 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.395 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.395 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:49.395 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936753+0000", 2024-03-27T21:40:49.395 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-03-27T21:40:49.395 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.396 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.396 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:49.396 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937926+0000", 2024-03-27T21:40:49.396 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011731280000000001 2024-03-27T21:40:49.396 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.396 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.396 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:49.396 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937969+0000", 2024-03-27T21:40:49.396 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3285999999999999e-05 2024-03-27T21:40:49.396 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.396 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.396 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:49.396 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938040+0000", 2024-03-27T21:40:49.396 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0358000000000006e-05 2024-03-27T21:40:49.396 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.397 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.397 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:49.397 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939324+0000", 2024-03-27T21:40:49.397 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012844899999999999 2024-03-27T21:40:49.397 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.397 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.397 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:49.397 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943780+0000", 2024-03-27T21:40:49.397 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044555630000000001 2024-03-27T21:40:49.397 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.397 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.397 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:49.397 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943811+0000", 2024-03-27T21:40:49.397 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1415999999999999e-05 2024-03-27T21:40:49.397 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.398 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.398 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:49.398 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943832+0000", 2024-03-27T21:40:49.398 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0981000000000001e-05 2024-03-27T21:40:49.398 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.398 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:49.398 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.398 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.398 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.398 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:49.398 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:49.398 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 69.454988503999999, 2024-03-27T21:40:49.398 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071139740000000003, 2024-03-27T21:40:49.398 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:49.399 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:49.399 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:49.399 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:49.399 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:49.399 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-03-27T21:40:49.399 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.399 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:49.399 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.399 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:49.399 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:49.399 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.399 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.399 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.399 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:49.400 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:49.400 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.400 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.400 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.400 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:49.400 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936760+0000", 2024-03-27T21:40:49.400 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:49.400 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.400 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.400 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:49.401 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936765+0000", 2024-03-27T21:40:49.401 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6330000000000004e-06 2024-03-27T21:40:49.401 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.401 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.401 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:49.401 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936771+0000", 2024-03-27T21:40:49.401 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0560000000000001e-06 2024-03-27T21:40:49.401 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.401 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.401 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:49.401 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936776+0000", 2024-03-27T21:40:49.401 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1510000000000003e-06 2024-03-27T21:40:49.401 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.402 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.402 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:49.402 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938605+0000", 2024-03-27T21:40:49.402 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001829589 2024-03-27T21:40:49.402 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.402 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.402 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:49.402 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938657+0000", 2024-03-27T21:40:49.402 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1860000000000002e-05 2024-03-27T21:40:49.402 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.402 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.402 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:49.402 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938734+0000", 2024-03-27T21:40:49.402 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6639999999999998e-05 2024-03-27T21:40:49.403 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.403 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.403 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:49.403 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939562+0000", 2024-03-27T21:40:49.403 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082803100000000001 2024-03-27T21:40:49.403 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.403 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.403 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:49.403 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943838+0000", 2024-03-27T21:40:49.403 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042757050000000003 2024-03-27T21:40:49.403 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.403 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.403 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:49.403 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943859+0000", 2024-03-27T21:40:49.404 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1175e-05 2024-03-27T21:40:49.404 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.404 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.404 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:49.404 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943875+0000", 2024-03-27T21:40:49.404 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5744e-05 2024-03-27T21:40:49.404 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.404 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:49.404 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.404 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.404 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.404 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:49.404 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:49.404 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 69.311474453000002, 2024-03-27T21:40:49.404 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059712230000000003, 2024-03-27T21:40:49.405 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:49.405 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:49.405 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:49.405 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:49.405 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:49.405 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 112 2024-03-27T21:40:49.405 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.405 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:49.405 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.405 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:49.405 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:49.405 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.405 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.405 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.406 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:49.406 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:49.406 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.406 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.406 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.406 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:49.406 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080274+0000", 2024-03-27T21:40:49.406 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:49.406 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.406 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.406 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:49.406 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080318+0000", 2024-03-27T21:40:49.406 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3634999999999998e-05 2024-03-27T21:40:49.406 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.407 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.407 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:49.407 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080329+0000", 2024-03-27T21:40:49.407 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1232e-05 2024-03-27T21:40:49.407 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.407 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.407 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:49.407 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080333+0000", 2024-03-27T21:40:49.407 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6009999999999999e-06 2024-03-27T21:40:49.407 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.407 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.407 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:49.407 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083786+0000", 2024-03-27T21:40:49.407 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034528940000000002 2024-03-27T21:40:49.407 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.408 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.408 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:49.408 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083861+0000", 2024-03-27T21:40:49.408 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4893000000000003e-05 2024-03-27T21:40:49.408 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.408 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.408 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:49.408 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083981+0000", 2024-03-27T21:40:49.408 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012058299999999999 2024-03-27T21:40:49.408 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.408 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.408 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:49.408 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086148+0000", 2024-03-27T21:40:49.408 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021671469999999999 2024-03-27T21:40:49.408 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.408 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.409 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:49.409 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086195+0000", 2024-03-27T21:40:49.409 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6768999999999999e-05 2024-03-27T21:40:49.409 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.409 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.409 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:49.409 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086224+0000", 2024-03-27T21:40:49.409 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8758000000000001e-05 2024-03-27T21:40:49.409 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.409 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.409 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:49.409 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086246+0000", 2024-03-27T21:40:49.409 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2064000000000001e-05 2024-03-27T21:40:49.409 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.409 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:49.410 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.410 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.410 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.410 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:49.410 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:49.410 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 69.311411505999999, 2024-03-27T21:40:49.410 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066196320000000003, 2024-03-27T21:40:49.410 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:49.410 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:49.410 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:49.410 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:49.410 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:49.410 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113 2024-03-27T21:40:49.410 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.411 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:49.411 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.411 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:49.411 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:49.411 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.411 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.411 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.411 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:49.411 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:49.411 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.411 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.411 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.411 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:49.411 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080337+0000", 2024-03-27T21:40:49.411 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:49.412 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.412 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.412 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:49.412 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080340+0000", 2024-03-27T21:40:49.412 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6690000000000002e-06 2024-03-27T21:40:49.412 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.412 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.412 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:49.412 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080344+0000", 2024-03-27T21:40:49.412 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6550000000000002e-06 2024-03-27T21:40:49.412 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.412 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.412 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:49.412 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080348+0000", 2024-03-27T21:40:49.413 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9169999999999999e-06 2024-03-27T21:40:49.413 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.413 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.413 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:49.413 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084595+0000", 2024-03-27T21:40:49.413 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042471719999999996 2024-03-27T21:40:49.413 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.413 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.413 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:49.413 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084676+0000", 2024-03-27T21:40:49.413 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1147000000000001e-05 2024-03-27T21:40:49.413 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.413 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.413 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:49.413 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084796+0000", 2024-03-27T21:40:49.414 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012032900000000001 2024-03-27T21:40:49.414 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.414 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.414 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:49.414 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086190+0000", 2024-03-27T21:40:49.414 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013934399999999999 2024-03-27T21:40:49.414 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.414 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.414 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:49.414 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086905+0000", 2024-03-27T21:40:49.414 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071536199999999999 2024-03-27T21:40:49.414 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.414 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.414 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:49.415 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086937+0000", 2024-03-27T21:40:49.415 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2125000000000002e-05 2024-03-27T21:40:49.415 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.415 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.415 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:49.415 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086957+0000", 2024-03-27T21:40:49.415 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0157999999999999e-05 2024-03-27T21:40:49.415 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.415 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:49.415 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.415 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.415 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.415 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:49.415 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:49.415 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 69.311398789999998, 2024-03-27T21:40:49.416 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007405661, 2024-03-27T21:40:49.416 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:49.416 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:49.416 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:49.416 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:49.416 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:49.416 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 114 2024-03-27T21:40:49.416 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.416 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:49.416 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.416 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:49.416 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:49.416 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.416 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.417 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.417 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:49.417 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:49.417 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.417 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.417 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.417 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:49.417 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:49.417 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-03-27T21:40:49.417 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.417 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.417 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:49.417 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080353+0000", 2024-03-27T21:40:49.417 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.441e-06 2024-03-27T21:40:49.417 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.418 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.418 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:49.418 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080356+0000", 2024-03-27T21:40:49.418 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0769999999999999e-06 2024-03-27T21:40:49.418 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.418 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.418 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:49.418 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080358+0000", 2024-03-27T21:40:49.418 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3369999999999998e-06 2024-03-27T21:40:49.418 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.418 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.418 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:49.418 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085118+0000", 2024-03-27T21:40:49.418 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047603949999999997 2024-03-27T21:40:49.419 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.419 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.419 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:49.419 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085210+0000", 2024-03-27T21:40:49.419 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1983000000000003e-05 2024-03-27T21:40:49.419 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.419 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.419 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:49.419 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085332+0000", 2024-03-27T21:40:49.419 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121803 2024-03-27T21:40:49.419 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.419 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.419 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:49.419 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086292+0000", 2024-03-27T21:40:49.419 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095961400000000002 2024-03-27T21:40:49.420 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.420 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.420 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:49.420 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087710+0000", 2024-03-27T21:40:49.420 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014181459999999999 2024-03-27T21:40:49.420 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.420 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.420 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:49.420 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087736+0000", 2024-03-27T21:40:49.420 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6475e-05 2024-03-27T21:40:49.420 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.420 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.420 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:49.420 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087756+0000", 2024-03-27T21:40:49.421 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9626000000000001e-05 2024-03-27T21:40:49.421 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.421 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:49.421 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.421 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.421 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.421 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:49.421 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:49.421 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 69.311376354999993, 2024-03-27T21:40:49.421 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075011720000000004, 2024-03-27T21:40:49.421 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:49.421 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:49.421 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:49.421 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:49.422 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:49.422 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115 2024-03-27T21:40:49.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.422 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:49.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.422 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:49.422 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:49.422 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.422 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:49.422 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:49.422 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.423 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:49.423 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:49.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:49.423 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.423 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.423 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:49.423 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080458+0000", 2024-03-27T21:40:49.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5228000000000005e-05 2024-03-27T21:40:49.423 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.423 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.423 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:49.423 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080465+0000", 2024-03-27T21:40:49.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.238e-06 2024-03-27T21:40:49.423 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.424 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.424 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:49.424 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080468+0000", 2024-03-27T21:40:49.424 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5109999999999999e-06 2024-03-27T21:40:49.424 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.424 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.424 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:49.424 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086326+0000", 2024-03-27T21:40:49.424 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058579169999999998 2024-03-27T21:40:49.424 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.424 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.424 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:49.424 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086382+0000", 2024-03-27T21:40:49.424 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5309999999999997e-05 2024-03-27T21:40:49.424 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:49.425 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086457+0000", 2024-03-27T21:40:49.425 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5413000000000005e-05 2024-03-27T21:40:49.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:49.425 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087762+0000", 2024-03-27T21:40:49.425 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013052300000000001 2024-03-27T21:40:49.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:49.425 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087798+0000", 2024-03-27T21:40:49.425 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5614999999999997e-05 2024-03-27T21:40:49.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:49.426 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087840+0000", 2024-03-27T21:40:49.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2196000000000001e-05 2024-03-27T21:40:49.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:49.426 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087874+0000", 2024-03-27T21:40:49.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3785999999999998e-05 2024-03-27T21:40:49.426 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.426 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:49.426 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.427 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:49.427 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:49.427 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 69.311276272000001, 2024-03-27T21:40:49.427 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075050680000000002, 2024-03-27T21:40:49.427 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:49.427 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:49.427 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:49.427 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:49.427 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:49.427 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 116 2024-03-27T21:40:49.427 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.427 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:49.427 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.427 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:49.427 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:49.428 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.428 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:49.428 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:49.428 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.428 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:49.428 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080472+0000", 2024-03-27T21:40:49.428 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:49.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.428 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:49.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080475+0000", 2024-03-27T21:40:49.429 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.712e-06 2024-03-27T21:40:49.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.429 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.429 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:49.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080478+0000", 2024-03-27T21:40:49.429 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1070000000000001e-06 2024-03-27T21:40:49.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.429 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.429 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:49.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080481+0000", 2024-03-27T21:40:49.429 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4269999999999998e-06 2024-03-27T21:40:49.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.429 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.429 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:49.430 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086992+0000", 2024-03-27T21:40:49.430 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065114099999999996 2024-03-27T21:40:49.430 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.430 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.430 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:49.430 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087064+0000", 2024-03-27T21:40:49.430 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1927000000000004e-05 2024-03-27T21:40:49.430 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.430 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.430 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:49.430 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087188+0000", 2024-03-27T21:40:49.430 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123585 2024-03-27T21:40:49.430 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.430 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:49.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087766+0000", 2024-03-27T21:40:49.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057814300000000004 2024-03-27T21:40:49.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.431 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:49.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087934+0000", 2024-03-27T21:40:49.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000167894 2024-03-27T21:40:49.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.431 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:49.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087958+0000", 2024-03-27T21:40:49.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4663000000000001e-05 2024-03-27T21:40:49.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:49.432 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087978+0000", 2024-03-27T21:40:49.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9636e-05 2024-03-27T21:40:49.432 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.432 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:49.432 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.432 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:49.432 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:49.432 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 69.311244149999993, 2024-03-27T21:40:49.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076850679999999998, 2024-03-27T21:40:49.432 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:49.432 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:49.433 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:49.433 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:49.433 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:49.433 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 117 2024-03-27T21:40:49.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.433 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:49.433 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.433 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:49.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:49.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.433 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.433 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:49.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:49.434 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.434 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.434 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:49.434 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:49.434 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:49.434 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.434 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:49.434 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080508+0000", 2024-03-27T21:40:49.434 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.298e-06 2024-03-27T21:40:49.434 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.434 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:49.435 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080515+0000", 2024-03-27T21:40:49.435 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0790000000000003e-06 2024-03-27T21:40:49.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.435 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:49.435 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080518+0000", 2024-03-27T21:40:49.435 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8509999999999999e-06 2024-03-27T21:40:49.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.435 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:49.435 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087444+0000", 2024-03-27T21:40:49.435 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0069259120000000002 2024-03-27T21:40:49.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.435 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:49.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087491+0000", 2024-03-27T21:40:49.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7043999999999999e-05 2024-03-27T21:40:49.436 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.436 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.436 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:49.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087564+0000", 2024-03-27T21:40:49.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3603999999999995e-05 2024-03-27T21:40:49.436 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.436 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.436 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:49.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087902+0000", 2024-03-27T21:40:49.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033734100000000001 2024-03-27T21:40:49.436 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.436 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.436 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:49.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088117+0000", 2024-03-27T21:40:49.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000215164 2024-03-27T21:40:49.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:49.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088160+0000", 2024-03-27T21:40:49.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2933999999999999e-05 2024-03-27T21:40:49.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:49.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088190+0000", 2024-03-27T21:40:49.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0139999999999999e-05 2024-03-27T21:40:49.437 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.437 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:49.438 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.438 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:49.438 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:49.438 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 68.894963761, 2024-03-27T21:40:49.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052465439999999997, 2024-03-27T21:40:49.438 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:49.438 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:49.438 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:49.438 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:49.438 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:49.438 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 133 2024-03-27T21:40:49.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.438 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:49.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:49.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:49.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:49.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:49.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:49.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:49.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:49.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:49.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496788+0000", 2024-03-27T21:40:49.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-03-27T21:40:49.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:49.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496793+0000", 2024-03-27T21:40:49.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4790000000000004e-06 2024-03-27T21:40:49.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:49.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496796+0000", 2024-03-27T21:40:49.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1e-06 2024-03-27T21:40:49.441 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.441 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.441 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:49.441 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500447+0000", 2024-03-27T21:40:49.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036504459999999999 2024-03-27T21:40:49.441 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.441 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.441 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:49.441 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500492+0000", 2024-03-27T21:40:49.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5765e-05 2024-03-27T21:40:49.441 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.441 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.441 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:49.441 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500562+0000", 2024-03-27T21:40:49.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9206000000000001e-05 2024-03-27T21:40:49.442 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.442 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.442 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:49.442 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501959+0000", 2024-03-27T21:40:49.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001397187 2024-03-27T21:40:49.442 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.442 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.442 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:49.442 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501984+0000", 2024-03-27T21:40:49.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.552e-05 2024-03-27T21:40:49.442 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.442 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.442 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:49.443 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502010+0000", 2024-03-27T21:40:49.443 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5232000000000002e-05 2024-03-27T21:40:49.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.443 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.443 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:49.443 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502032+0000", 2024-03-27T21:40:49.443 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2337999999999999e-05 2024-03-27T21:40:49.443 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.443 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:49.443 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.443 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.443 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:49.443 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:49.444 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 68.894905102999999, 2024-03-27T21:40:49.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052892119999999997, 2024-03-27T21:40:49.444 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:49.444 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:49.444 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:49.444 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:49.444 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:49.444 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 134 2024-03-27T21:40:49.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.444 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:49.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:49.444 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:49.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:49.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:49.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:49.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:49.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:49.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:49.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496929+0000", 2024-03-27T21:40:49.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5098000000000004e-05 2024-03-27T21:40:49.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:49.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496937+0000", 2024-03-27T21:40:49.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9559999999999997e-06 2024-03-27T21:40:49.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:49.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496942+0000", 2024-03-27T21:40:49.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6350000000000003e-06 2024-03-27T21:40:49.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:49.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500749+0000", 2024-03-27T21:40:49.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038068529999999998 2024-03-27T21:40:49.447 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.447 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:49.447 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500792+0000", 2024-03-27T21:40:49.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2823e-05 2024-03-27T21:40:49.447 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.447 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:49.447 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500864+0000", 2024-03-27T21:40:49.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1715e-05 2024-03-27T21:40:49.447 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.447 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:49.447 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502068+0000", 2024-03-27T21:40:49.448 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012044860000000001 2024-03-27T21:40:49.448 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.448 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.448 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:49.448 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502073+0000", 2024-03-27T21:40:49.448 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5700000000000003e-06 2024-03-27T21:40:49.448 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.448 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.448 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:49.448 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502098+0000", 2024-03-27T21:40:49.448 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5120999999999999e-05 2024-03-27T21:40:49.448 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.448 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.448 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:49.448 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502133+0000", 2024-03-27T21:40:49.448 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5306999999999999e-05 2024-03-27T21:40:49.449 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.449 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:49.449 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.449 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.449 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.449 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:49.449 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:49.449 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 68.894799796000001, 2024-03-27T21:40:49.449 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062529040000000001, 2024-03-27T21:40:49.449 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:49.449 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:49.449 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:49.449 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:49.450 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:49.450 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135 2024-03-27T21:40:49.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.450 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:49.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.450 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:49.450 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:49.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:49.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:49.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:49.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:49.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:49.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:49.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496953+0000", 2024-03-27T21:40:49.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0709999999999996e-06 2024-03-27T21:40:49.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:49.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496961+0000", 2024-03-27T21:40:49.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1300000000000001e-06 2024-03-27T21:40:49.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:49.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496968+0000", 2024-03-27T21:40:49.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0999999999999998e-06 2024-03-27T21:40:49.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:49.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501244+0000", 2024-03-27T21:40:49.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042756310000000002 2024-03-27T21:40:49.453 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.453 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.453 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:49.453 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501377+0000", 2024-03-27T21:40:49.453 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013348799999999999 2024-03-27T21:40:49.453 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.453 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.453 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:49.453 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501455+0000", 2024-03-27T21:40:49.453 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7410000000000006e-05 2024-03-27T21:40:49.453 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.453 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.453 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:49.453 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502138+0000", 2024-03-27T21:40:49.454 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068386599999999999 2024-03-27T21:40:49.454 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.454 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.454 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:49.454 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503152+0000", 2024-03-27T21:40:49.454 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010139680000000001 2024-03-27T21:40:49.454 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.454 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.454 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:49.454 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503180+0000", 2024-03-27T21:40:49.454 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7426000000000001e-05 2024-03-27T21:40:49.454 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.454 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.454 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:49.455 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503202+0000", 2024-03-27T21:40:49.455 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2463000000000002e-05 2024-03-27T21:40:49.455 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.455 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:49.455 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.455 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.455 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.455 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:49.455 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:49.455 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 68.894775658, 2024-03-27T21:40:49.455 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066253680000000004, 2024-03-27T21:40:49.455 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:49.455 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:49.455 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:49.455 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:49.456 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:49.456 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 136 2024-03-27T21:40:49.456 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.456 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:49.456 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.456 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:49.456 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:49.456 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.456 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.456 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.456 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:49.456 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:49.456 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.456 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.457 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.457 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:49.457 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:49.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:49.457 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.457 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.457 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:49.457 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496977+0000", 2024-03-27T21:40:49.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5049999999999999e-06 2024-03-27T21:40:49.457 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.457 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.457 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:49.457 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496984+0000", 2024-03-27T21:40:49.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1520000000000003e-06 2024-03-27T21:40:49.458 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:49.458 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496990+0000", 2024-03-27T21:40:49.458 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-03-27T21:40:49.458 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:49.458 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501669+0000", 2024-03-27T21:40:49.458 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046793859999999998 2024-03-27T21:40:49.458 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:49.459 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501714+0000", 2024-03-27T21:40:49.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5154999999999997e-05 2024-03-27T21:40:49.459 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.459 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.459 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:49.459 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501783+0000", 2024-03-27T21:40:49.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8454999999999994e-05 2024-03-27T21:40:49.459 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.459 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.460 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:49.460 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503226+0000", 2024-03-27T21:40:49.460 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001443441 2024-03-27T21:40:49.460 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.460 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.460 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:49.460 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503540+0000", 2024-03-27T21:40:49.460 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031347100000000001 2024-03-27T21:40:49.460 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.460 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.460 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:49.460 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503563+0000", 2024-03-27T21:40:49.460 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3223000000000001e-05 2024-03-27T21:40:49.461 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.461 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.461 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:49.461 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503599+0000", 2024-03-27T21:40:49.461 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6105000000000001e-05 2024-03-27T21:40:49.461 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.461 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:49.461 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.461 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.461 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.461 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:49.461 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:49.461 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 68.894752847999996, 2024-03-27T21:40:49.462 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074261320000000002, 2024-03-27T21:40:49.462 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:49.462 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:49.462 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:49.462 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:49.462 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:49.462 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 137 2024-03-27T21:40:49.462 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.462 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:49.462 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.462 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:49.462 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:49.462 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.462 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.463 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.463 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:49.463 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:49.463 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.463 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.463 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.463 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:49.463 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:49.463 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:49.463 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.464 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.464 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:49.464 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497077+0000", 2024-03-27T21:40:49.464 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1599000000000003e-05 2024-03-27T21:40:49.464 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.464 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.464 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:49.464 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497085+0000", 2024-03-27T21:40:49.464 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6109999999999996e-06 2024-03-27T21:40:49.464 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.464 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.464 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:49.464 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497092+0000", 2024-03-27T21:40:49.465 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.742e-06 2024-03-27T21:40:49.465 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.465 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.465 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:49.465 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502159+0000", 2024-03-27T21:40:49.465 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050666870000000003 2024-03-27T21:40:49.465 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.465 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.465 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:49.465 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502200+0000", 2024-03-27T21:40:49.465 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1919999999999998e-05 2024-03-27T21:40:49.465 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.465 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.465 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:49.466 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502280+0000", 2024-03-27T21:40:49.466 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9417000000000002e-05 2024-03-27T21:40:49.466 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.466 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.466 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:49.466 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503605+0000", 2024-03-27T21:40:49.466 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001325641 2024-03-27T21:40:49.466 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.466 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.466 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:49.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504361+0000", 2024-03-27T21:40:49.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075555399999999997 2024-03-27T21:40:49.467 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.467 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:49.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504386+0000", 2024-03-27T21:40:49.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5369999999999999e-05 2024-03-27T21:40:49.467 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.467 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:49.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504422+0000", 2024-03-27T21:40:49.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6019000000000002e-05 2024-03-27T21:40:49.467 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.468 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:49.468 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.468 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.468 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.468 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:49.468 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:49.468 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 68.894649051000002, 2024-03-27T21:40:49.468 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074059750000000004, 2024-03-27T21:40:49.468 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:49.468 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:49.468 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:49.468 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:49.468 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:49.468 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-03-27T21:40:49.469 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.469 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:49.469 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.469 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:49.469 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:49.469 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.469 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.469 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.469 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:49.470 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:49.470 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.470 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.470 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.470 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:49.470 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:49.470 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:49.470 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.470 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.470 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:49.470 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497103+0000", 2024-03-27T21:40:49.470 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.856e-06 2024-03-27T21:40:49.470 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.471 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.471 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:49.471 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497121+0000", 2024-03-27T21:40:49.471 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7190999999999999e-05 2024-03-27T21:40:49.471 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.471 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.471 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:49.471 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497126+0000", 2024-03-27T21:40:49.471 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3770000000000004e-06 2024-03-27T21:40:49.471 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.471 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.471 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:49.471 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502833+0000", 2024-03-27T21:40:49.471 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057073510000000003 2024-03-27T21:40:49.472 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.472 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.472 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:49.472 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502881+0000", 2024-03-27T21:40:49.472 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7961999999999999e-05 2024-03-27T21:40:49.472 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.472 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.472 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:49.472 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502949+0000", 2024-03-27T21:40:49.472 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7957e-05 2024-03-27T21:40:49.472 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.472 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.472 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:49.472 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504430+0000", 2024-03-27T21:40:49.473 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001480763 2024-03-27T21:40:49.473 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.473 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.473 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:49.473 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504459+0000", 2024-03-27T21:40:49.473 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8983999999999999e-05 2024-03-27T21:40:49.473 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.473 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.473 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:49.473 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504486+0000", 2024-03-27T21:40:49.473 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6883999999999999e-05 2024-03-27T21:40:49.473 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.473 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.474 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:49.474 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504506+0000", 2024-03-27T21:40:49.474 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0214e-05 2024-03-27T21:40:49.474 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.474 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:49.474 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.474 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.474 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.474 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:49.474 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:49.474 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 68.894617217000004, 2024-03-27T21:40:49.474 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074754859999999999, 2024-03-27T21:40:49.474 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:49.474 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:49.475 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:49.475 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:49.475 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:49.475 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-03-27T21:40:49.475 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.475 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:49.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.475 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:49.475 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:49.475 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.475 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.475 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:49.475 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:49.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.476 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.476 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.476 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:49.476 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:49.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:49.476 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.476 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.476 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:49.476 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497137+0000", 2024-03-27T21:40:49.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0730000000000004e-06 2024-03-27T21:40:49.476 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.476 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.476 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:49.476 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497143+0000", 2024-03-27T21:40:49.477 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3759999999999996e-06 2024-03-27T21:40:49.477 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.477 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.477 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:49.477 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497150+0000", 2024-03-27T21:40:49.477 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.002e-06 2024-03-27T21:40:49.477 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.477 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.477 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:49.477 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503249+0000", 2024-03-27T21:40:49.477 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060989750000000004 2024-03-27T21:40:49.477 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.477 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.477 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:49.477 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503291+0000", 2024-03-27T21:40:49.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1563999999999997e-05 2024-03-27T21:40:49.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.478 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.478 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:49.478 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503376+0000", 2024-03-27T21:40:49.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5752000000000002e-05 2024-03-27T21:40:49.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.478 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.478 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:49.478 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504545+0000", 2024-03-27T21:40:49.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169172 2024-03-27T21:40:49.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.478 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.478 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:49.479 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504550+0000", 2024-03-27T21:40:49.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7419999999999997e-06 2024-03-27T21:40:49.479 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.479 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.479 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:49.479 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504574+0000", 2024-03-27T21:40:49.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4124000000000001e-05 2024-03-27T21:40:49.479 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.479 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.479 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:49.479 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504607+0000", 2024-03-27T21:40:49.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3118999999999999e-05 2024-03-27T21:40:49.479 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.479 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:49.480 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.480 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.480 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.480 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:49.480 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:49.480 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 68.894593551, 2024-03-27T21:40:49.480 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078150589999999992, 2024-03-27T21:40:49.480 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:49.480 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:49.480 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:49.480 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:49.480 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:49.480 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-03-27T21:40:49.480 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.480 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:49.481 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.481 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:49.481 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:49.481 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.481 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.481 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.481 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:49.481 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:49.481 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.481 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.481 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.481 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:49.481 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497155+0000", 2024-03-27T21:40:49.481 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:49.482 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.482 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.482 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:49.482 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497230+0000", 2024-03-27T21:40:49.482 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4789000000000005e-05 2024-03-27T21:40:49.482 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.482 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.482 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:49.482 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497241+0000", 2024-03-27T21:40:49.482 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1442e-05 2024-03-27T21:40:49.482 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.482 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.482 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:49.482 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497255+0000", 2024-03-27T21:40:49.483 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3087e-05 2024-03-27T21:40:49.483 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:49.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503630+0000", 2024-03-27T21:40:49.483 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063755269999999998 2024-03-27T21:40:49.483 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:49.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503673+0000", 2024-03-27T21:40:49.483 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2616e-05 2024-03-27T21:40:49.483 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:49.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503738+0000", 2024-03-27T21:40:49.484 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5114e-05 2024-03-27T21:40:49.484 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.484 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.484 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:49.484 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504907+0000", 2024-03-27T21:40:49.484 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169432 2024-03-27T21:40:49.484 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.484 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.484 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:49.484 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504926+0000", 2024-03-27T21:40:49.484 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9086999999999999e-05 2024-03-27T21:40:49.484 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.484 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.484 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:49.484 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504950+0000", 2024-03-27T21:40:49.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3598999999999999e-05 2024-03-27T21:40:49.485 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.485 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.485 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:49.485 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504971+0000", 2024-03-27T21:40:49.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0772999999999999e-05 2024-03-27T21:40:49.485 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.485 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:49.485 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.485 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.485 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.485 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:49.485 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:49.485 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 68.894486510999997, 2024-03-27T21:40:49.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082962590000000003, 2024-03-27T21:40:49.486 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:49.486 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:49.486 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:49.486 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:49.486 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:49.486 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-03-27T21:40:49.486 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.486 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:49.486 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.486 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:49.486 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:49.486 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.486 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.486 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.487 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:49.487 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:49.487 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.487 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.487 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:49.487 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497262+0000", 2024-03-27T21:40:49.487 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:49.487 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.487 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:49.487 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497267+0000", 2024-03-27T21:40:49.487 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.955e-06 2024-03-27T21:40:49.487 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:49.488 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497275+0000", 2024-03-27T21:40:49.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.824e-06 2024-03-27T21:40:49.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:49.488 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497280+0000", 2024-03-27T21:40:49.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8230000000000002e-06 2024-03-27T21:40:49.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:49.488 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504628+0000", 2024-03-27T21:40:49.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007348561 2024-03-27T21:40:49.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:49.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504673+0000", 2024-03-27T21:40:49.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4313000000000001e-05 2024-03-27T21:40:49.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:49.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504745+0000", 2024-03-27T21:40:49.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2303000000000002e-05 2024-03-27T21:40:49.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:49.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505358+0000", 2024-03-27T21:40:49.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061281300000000003 2024-03-27T21:40:49.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.490 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.490 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:49.490 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505503+0000", 2024-03-27T21:40:49.490 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000145134 2024-03-27T21:40:49.490 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.490 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.490 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:49.490 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505526+0000", 2024-03-27T21:40:49.490 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3312e-05 2024-03-27T21:40:49.490 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.490 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.490 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:49.490 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505559+0000", 2024-03-27T21:40:49.490 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2821999999999998e-05 2024-03-27T21:40:49.491 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.491 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:49.491 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.491 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.491 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.491 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:49.491 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:49.491 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 68.285264272000006, 2024-03-27T21:40:49.491 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053753339999999998, 2024-03-27T21:40:49.491 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:49.491 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:49.491 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:49.492 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:49.492 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:49.492 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159 2024-03-27T21:40:49.492 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.492 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:49.492 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.492 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:49.492 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:49.492 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.492 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.492 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.492 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:49.492 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:49.492 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:49.492 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:49.493 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:49.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:49.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:49.493 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106488+0000", 2024-03-27T21:40:49.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9830000000000001e-06 2024-03-27T21:40:49.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:49.493 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106491+0000", 2024-03-27T21:40:49.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3459999999999998e-06 2024-03-27T21:40:49.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.494 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.494 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:49.494 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106493+0000", 2024-03-27T21:40:49.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.509e-06 2024-03-27T21:40:49.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.494 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.494 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:49.494 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110636+0000", 2024-03-27T21:40:49.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041427260000000002 2024-03-27T21:40:49.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.494 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.494 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:49.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110683+0000", 2024-03-27T21:40:49.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6557000000000002e-05 2024-03-27T21:40:49.495 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:49.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110755+0000", 2024-03-27T21:40:49.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2119000000000005e-05 2024-03-27T21:40:49.495 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:49.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111415+0000", 2024-03-27T21:40:49.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066065199999999996 2024-03-27T21:40:49.495 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.496 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:49.496 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111793+0000", 2024-03-27T21:40:49.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037732100000000001 2024-03-27T21:40:49.496 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.496 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.496 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:49.496 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111829+0000", 2024-03-27T21:40:49.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418999999999998e-05 2024-03-27T21:40:49.496 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:49.496 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:49.496 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:49.496 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111860+0000", 2024-03-27T21:40:49.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1003000000000001e-05 2024-03-27T21:40:49.497 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.497 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:49.497 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.497 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:49.497 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:49.497 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:49.498 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:40:49.499 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:49.512+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:49.600 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:49.614+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:49.611 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:49.611 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:49.611 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:49.611 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:49.620 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:40:49.701 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:49.714+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:49.740 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:40:49.741 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:49.741 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:49.741 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:49.741 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:49.741 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:49.754 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:40:49.810 INFO:teuthology.orchestra.run.smithi060.stdout:2.3 2024-03-27T21:40:49.811 INFO:teuthology.orchestra.run.smithi060.stdout:2.1 2024-03-27T21:40:49.811 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:49.825+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:49.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.866+0000 7efc8275c640 10 monclient: get_monmap_and_config 2024-03-27T21:40:49.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.866+0000 7efc8275c640 10 monclient: build_initial_monmap 2024-03-27T21:40:49.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.866+0000 7efc8275c640 10 monclient: monmap: 2024-03-27T21:40:49.852 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:49.852 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:49.852 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:49.867106+0000 2024-03-27T21:40:49.852 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:49.867106+0000 2024-03-27T21:40:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:49.853 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:49.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.866+0000 7efc8275c640 1 Processor -- start 2024-03-27T21:40:49.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.866+0000 7efc8275c640 1 -- start start 2024-03-27T21:40:49.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.866+0000 7efc8275c640 10 monclient: init 2024-03-27T21:40:49.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc8275c640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:49.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc8275c640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-03-27T21:40:49.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc8275c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efc7c071a50 0x7efc7c0a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc8275c640 10 monclient(hunting): picked mon.noname-a con 0x7efc7c071a50 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc8275c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efc7c13e810 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc8275c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7efc7c13ec50 0x7efc7c141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc8275c640 10 monclient(hunting): picked mon.noname-c con 0x7efc7c13ec50 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc8275c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7c0a8e90 con 0x7efc7c13e810 2024-03-27T21:40:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc8275c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc8275c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc7c071e30 con 0x7efc7c071a50 2024-03-27T21:40:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc8275c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc8275c640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc7c0aa040 con 0x7efc7c13ec50 2024-03-27T21:40:49.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc8275c640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:49.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc8275c640 10 monclient(hunting): authenticate will time out at 891.088456s 2024-03-27T21:40:49.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc7bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efc7c071a50 0x7efc7c0a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:49.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc7bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efc7c071a50 0x7efc7c0a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39822/0 (socket says 172.21.15.60:39822) 2024-03-27T21:40:49.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc7bfff640 1 -- 172.21.15.60:0/2081784921 learned_addr learned my addr 172.21.15.60:0/2081784921 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:49.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc7bfff640 10 monclient(hunting): get_auth_request con 0x7efc7c071a50 auth_method 0 2024-03-27T21:40:49.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:49.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:49.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:49.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:49.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:49.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:49.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc80cf9640 1 --2- 172.21.15.60:0/2081784921 >> v2:172.21.15.191:3300/0 conn(0x7efc7c13ec50 0x7efc7c141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:49.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc80cf9640 10 monclient(hunting): get_auth_request con 0x7efc7c13ec50 auth_method 0 2024-03-27T21:40:49.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc80cf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:49.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc80cf9640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:49.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc80cf9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:49.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc7bfff640 10 monclient(hunting): handle_auth_done global_id 4767 payload 274 2024-03-27T21:40:49.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc7bfff640 1 -- 172.21.15.60:0/2081784921 >> v2:172.21.15.191:3300/0 conn(0x7efc7c13ec50 msgr2=0x7efc7c141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:49.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc7bfff640 1 --2- 172.21.15.60:0/2081784921 >> v2:172.21.15.191:3300/0 conn(0x7efc7c13ec50 0x7efc7c141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:49.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc7bfff640 1 -- 172.21.15.60:0/2081784921 >> v1:172.21.15.191:6789/0 conn(0x7efc7c13e810 legacy=0x7efc7c141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:49.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.867+0000 7efc7bfff640 10 monclient: _finish_hunting 0 2024-03-27T21:40:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.868+0000 7efc7bfff640 1 monclient: found mon.noname-a 2024-03-27T21:40:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.868+0000 7efc7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.868+0000 7efc7bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.868+0000 7efc7bfff640 1 -- 172.21.15.60:0/2081784921 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c142280 con 0x7efc7c071a50 2024-03-27T21:40:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.868+0000 7efc80cf9640 1 --2- 172.21.15.60:0/2081784921 >> v2:172.21.15.191:3300/0 conn(0x7efc7c13ec50 0x7efc7c141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:40:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.869+0000 7efc7bfff640 1 --2- 172.21.15.60:0/2081784921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc7c071a50 0x7efc7c0a87e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efc7c0b3460 tx=0x7efc6c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51fec665932ca48b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.869+0000 7efc7affd640 1 -- 172.21.15.60:0/2081784921 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c030860 con 0x7efc7c071a50 2024-03-27T21:40:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.869+0000 7efc7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.869+0000 7efc7affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.869+0000 7efc7affd640 10 monclient: dump: 2024-03-27T21:40:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:49.877 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.869+0000 7efc7affd640 10 monclient: _finish_auth 0 2024-03-27T21:40:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.869+0000 7efc7affd640 10 monclient: _check_auth_tickets 2024-03-27T21:40:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.869+0000 7efc7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.869+0000 7efc7affd640 1 -- 172.21.15.60:0/2081784921 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6c0309c0 con 0x7efc7c071a50 2024-03-27T21:40:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.869+0000 7efc7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:49.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.869+0000 7efc7affd640 1 -- 172.21.15.60:0/2081784921 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c030d50 con 0x7efc7c071a50 2024-03-27T21:40:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.869+0000 7efc8275c640 5 monclient: authenticate success, global_id 4767 2024-03-27T21:40:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.869+0000 7efc8275c640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.869+0000 7efc8275c640 4 set_mon_vals no callback set 2024-03-27T21:40:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.869+0000 7efc7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.869+0000 7efc7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.869+0000 7efc7affd640 10 monclient: dump: 2024-03-27T21:40:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:49.879 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.869+0000 7efc8275c640 10 monclient: shutdown 2024-03-27T21:40:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.869+0000 7efc8275c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.869+0000 7efc8275c640 1 -- 172.21.15.60:0/2081784921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc7c071a50 msgr2=0x7efc7c0a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.869+0000 7efc8275c640 1 --2- 172.21.15.60:0/2081784921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc7c071a50 0x7efc7c0a87e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efc7c0b3460 tx=0x7efc6c02b9a0 comp rx=0 tx=0).stop 2024-03-27T21:40:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.870+0000 7efc8275c640 1 -- 172.21.15.60:0/2081784921 shutdown_connections 2024-03-27T21:40:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.870+0000 7efc8275c640 1 --2- 172.21.15.60:0/2081784921 >> v2:172.21.15.191:3300/0 conn(0x7efc7c13ec50 0x7efc7c141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.870+0000 7efc8275c640 1 --2- 172.21.15.60:0/2081784921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc7c071a50 0x7efc7c0a87e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.870+0000 7efc8275c640 1 -- 172.21.15.60:0/2081784921 >> 172.21.15.60:0/2081784921 conn(0x7efc7c0b7300 msgr2=0x7efc7c0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:49.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.870+0000 7efc8275c640 1 -- 172.21.15.60:0/2081784921 shutdown_connections 2024-03-27T21:40:49.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.870+0000 7efc8275c640 1 -- 172.21.15.60:0/2081784921 wait complete. 2024-03-27T21:40:49.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.870+0000 7efc8275c640 10 monclient: build_initial_monmap 2024-03-27T21:40:49.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.870+0000 7efc8275c640 10 monclient: monmap: 2024-03-27T21:40:49.880 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:49.881 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:49.881 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:49.881 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:49.881 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:49.881 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:49.881 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:49.881 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:49.881 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:49.881 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:49.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 1 librados: starting msgr at 2024-03-27T21:40:49.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 1 librados: starting objecter 2024-03-27T21:40:49.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 1 Processor -- start 2024-03-27T21:40:49.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 1 -- start start 2024-03-27T21:40:49.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 1 librados: setting wanted keys 2024-03-27T21:40:49.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 1 librados: calling monclient init 2024-03-27T21:40:49.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 10 monclient: init 2024-03-27T21:40:49.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:49.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:40:49.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc7c071a50 0x7efc7c135dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:49.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 10 monclient(hunting): picked mon.noname-b con 0x7efc7c071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:49.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc7c13e810 0x7efc7c131fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:49.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 10 monclient(hunting): picked mon.noname-a con 0x7efc7c13e810 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:49.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efc7c13ec50 0x7efc7c1324f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:49.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 10 monclient(hunting): picked mon.noname-c con 0x7efc7c13ec50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:49.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:49.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c13a180 con 0x7efc7c13e810 2024-03-27T21:40:49.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:49.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c132a60 con 0x7efc7c071a50 2024-03-27T21:40:49.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:49.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c139940 con 0x7efc7c13ec50 2024-03-27T21:40:49.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:49.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc8275c640 10 monclient(hunting): authenticate will time out at 891.092813s 2024-03-27T21:40:49.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.871+0000 7efc7b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc7c13e810 0x7efc7c131fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:49.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc7c13e810 0x7efc7c131fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39828/0 (socket says 172.21.15.60:39828) 2024-03-27T21:40:49.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc80cf9640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efc7c13ec50 0x7efc7c1324f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:49.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7b7fe640 1 -- 172.21.15.60:0/2978648585 learned_addr learned my addr 172.21.15.60:0/2978648585 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:49.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7b7fe640 10 monclient(hunting): get_auth_request con 0x7efc7c13e810 auth_method 0 2024-03-27T21:40:49.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:49.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:49.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:49.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc80cf9640 10 monclient(hunting): get_auth_request con 0x7efc7c13ec50 auth_method 0 2024-03-27T21:40:49.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc80cf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:49.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc80cf9640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:49.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc80cf9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:49.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:49.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:49.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:49.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc80cf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:49.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc80cf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:49.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc80cf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:49.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7bfff640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc7c071a50 0x7efc7c135dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:49.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7bfff640 10 monclient(hunting): get_auth_request con 0x7efc7c071a50 auth_method 0 2024-03-27T21:40:49.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:49.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:49.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:49.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7b7fe640 10 monclient(hunting): handle_auth_done global_id 4770 payload 824 2024-03-27T21:40:49.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7b7fe640 1 -- 172.21.15.60:0/2978648585 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efc7c13ec50 msgr2=0x7efc7c1324f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:49.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7b7fe640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efc7c13ec50 0x7efc7c1324f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:49.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7b7fe640 1 -- 172.21.15.60:0/2978648585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc7c071a50 msgr2=0x7efc7c135dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:49.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7b7fe640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc7c071a50 0x7efc7c135dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:49.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7b7fe640 10 monclient: _finish_hunting 0 2024-03-27T21:40:49.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7b7fe640 1 monclient: found mon.noname-a 2024-03-27T21:40:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7b7fe640 1 -- 172.21.15.60:0/2978648585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6c0094d0 con 0x7efc7c13e810 2024-03-27T21:40:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc80cf9640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efc7c13ec50 0x7efc7c1324f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7b7fe640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc7c13e810 0x7efc7c131fb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efc7000c450 tx=0x7efc7000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc78ff9640 10 client.?.objecter ms_handle_connect 0x7efc7c13e810 2024-03-27T21:40:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc78ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc700052d0 con 0x7efc7c13e810 2024-03-27T21:40:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc78ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc78ff9640 10 monclient: dump: 2024-03-27T21:40:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:49.888 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:49.888 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:49.888 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:49.888 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:49.888 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:49.888 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:49.888 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:49.888 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:49.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc78ff9640 10 monclient: _finish_auth 0 2024-03-27T21:40:49.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc78ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:40:49.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:49.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc8275c640 5 monclient: authenticate success, global_id 4770 2024-03-27T21:40:49.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc8275c640 10 monclient: _renew_subs 2024-03-27T21:40:49.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc8275c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:49.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc8275c640 1 -- 172.21.15.60:0/2978648585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc7c1393f0 con 0x7efc7c13e810 2024-03-27T21:40:49.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc8275c640 10 client.4770.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:49.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc8275c640 10 monclient: _renew_subs 2024-03-27T21:40:49.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc8275c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:49.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc8275c640 1 -- 172.21.15.60:0/2978648585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7c139b50 con 0x7efc7c13e810 2024-03-27T21:40:49.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc8275c640 1 librados: init done 2024-03-27T21:40:49.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc70005430 con 0x7efc7c13e810 2024-03-27T21:40:49.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc78ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:49.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc70003830 con 0x7efc7c13e810 2024-03-27T21:40:49.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:49.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc78ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:49.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc7a7fc640 4 set_mon_vals no callback set 2024-03-27T21:40:49.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.872+0000 7efc78ff9640 10 monclient: dump: 2024-03-27T21:40:49.889 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:49.889 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:49.889 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:49.889 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:49.890 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:49.890 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:49.890 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:49.890 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:49.890 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:49.890 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:49.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.873+0000 7efc8275c640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:49.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.873+0000 7efc8275c640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:49.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.873+0000 7efc8275c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:49.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.873+0000 7efc8275c640 1 -- 172.21.15.60:0/2978648585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7efc5c005020 con 0x7efc7c13e810 2024-03-27T21:40:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.874+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7efc70003990 con 0x7efc7c13e810 2024-03-27T21:40:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.874+0000 7efc78ff9640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7efc6806c5c0 0x7efc6806ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5242+0+0 (secure 0 0 0) 0x7efc70088650 con 0x7efc7c13e810 2024-03-27T21:40:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc78ff9640 10 client.4770.objecter ms_dispatch 0x7efc7c000b90 osd_map(45..45 src has 1..45) v4 2024-03-27T21:40:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc78ff9640 3 client.4770.objecter handle_osd_map got epochs [45,45] > 0 2024-03-27T21:40:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc78ff9640 3 client.4770.objecter handle_osd_map decoding full epoch 45 2024-03-27T21:40:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc7bfff640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7efc6806c5c0 0x7efc6806ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc7bfff640 10 monclient: get_auth_request con 0x7efc6806c5c0 auth_method 0 2024-03-27T21:40:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc78ff9640 20 client.4770.objecter dump_active .. 0 homeless 2024-03-27T21:40:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v45) v1 ==== 54+0+16 (secure 0 0 0) 0x7efc7008f070 con 0x7efc7c13e810 2024-03-27T21:40:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc78ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc8275c640 10 client.4770.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign -1 1 2024-03-27T21:40:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc8275c640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7efc5c004640 0x7efc5c004a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c0052c0 osd=0 3 2024-03-27T21:40:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c0052c0 osd=0 3 2024-03-27T21:40:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c0052c0 osd=0 3 2024-03-27T21:40:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc8275c640 15 client.4770.objecter _session_command_op_remove -1 1 2024-03-27T21:40:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc8275c640 20 client.4770.objecter get_session s=0x7efc5c0052c0 osd=0 3 2024-03-27T21:40:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign 0 1 2024-03-27T21:40:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c0052c0 osd=0 4 2024-03-27T21:40:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc8275c640 10 client.4770.objecter _send_command 1 2024-03-27T21:40:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc8275c640 1 -- 172.21.15.60:0/2978648585 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc5c008a50 con 0x7efc5c004640 2024-03-27T21:40:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc80cf9640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7efc5c004640 0x7efc5c004a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc80cf9640 10 monclient: get_auth_request con 0x7efc5c004640 auth_method 0 2024-03-27T21:40:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc7bfff640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7efc6806c5c0 0x7efc6806ea80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efc6c036000 tx=0x7efc6c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:49.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc78ff9640 10 client.4770.objecter ms_handle_connect 0x7efc6806c5c0 2024-03-27T21:40:49.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc78ff9640 10 client.4770.objecter ms_handle_connect 0x7efc5c004640 2024-03-27T21:40:49.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.875+0000 7efc80cf9640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7efc5c004640 0x7efc5c004a80 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:49.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.876+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7efc5c008a50 con 0x7efc5c004640 2024-03-27T21:40:49.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.876+0000 7efc78ff9640 10 client.4770.objecter ms_dispatch 0x7efc7c000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:49.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.876+0000 7efc78ff9640 10 client.4770.objecter _finish_command 1 = system:0 2024-03-27T21:40:49.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.876+0000 7efc78ff9640 20 client.4770.objecter put_session s=0x7efc5c0052c0 osd=0 4 2024-03-27T21:40:49.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.876+0000 7efc78ff9640 15 client.4770.objecter _session_command_op_remove 0 1 2024-03-27T21:40:49.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.906+0000 7efc8275c640 10 client.4770.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:49.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.906+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign -1 2 2024-03-27T21:40:49.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.906+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c0052c0 osd=0 3 2024-03-27T21:40:49.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.906+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c0052c0 osd=0 3 2024-03-27T21:40:49.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.906+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c0052c0 osd=0 3 2024-03-27T21:40:49.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.906+0000 7efc8275c640 15 client.4770.objecter _session_command_op_remove -1 2 2024-03-27T21:40:49.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.906+0000 7efc8275c640 20 client.4770.objecter get_session s=0x7efc5c0052c0 osd=0 3 2024-03-27T21:40:49.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.906+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign 0 2 2024-03-27T21:40:49.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.906+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c0052c0 osd=0 4 2024-03-27T21:40:49.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.906+0000 7efc8275c640 10 client.4770.objecter _send_command 2 2024-03-27T21:40:49.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.906+0000 7efc8275c640 1 -- 172.21.15.60:0/2978648585 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efc5c0040f0 con 0x7efc5c004640 2024-03-27T21:40:49.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.907+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7efc5c0040f0 con 0x7efc5c004640 2024-03-27T21:40:49.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.907+0000 7efc78ff9640 10 client.4770.objecter ms_dispatch 0x7efc7c000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:49.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.907+0000 7efc78ff9640 10 client.4770.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:49.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.907+0000 7efc78ff9640 20 client.4770.objecter put_session s=0x7efc5c0052c0 osd=0 4 2024-03-27T21:40:49.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.907+0000 7efc78ff9640 15 client.4770.objecter _session_command_op_remove 0 2 2024-03-27T21:40:49.896 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:49.897 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:49.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.908+0000 7efc8275c640 10 client.4770.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.908+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign -1 3 2024-03-27T21:40:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.908+0000 7efc8275c640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7efc5c002700 0x7efc5c002b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.908+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c008090 osd=1 3 2024-03-27T21:40:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.908+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c008090 osd=1 3 2024-03-27T21:40:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.908+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c008090 osd=1 3 2024-03-27T21:40:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.908+0000 7efc8275c640 15 client.4770.objecter _session_command_op_remove -1 3 2024-03-27T21:40:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.908+0000 7efc8275c640 20 client.4770.objecter get_session s=0x7efc5c008090 osd=1 3 2024-03-27T21:40:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.908+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign 1 3 2024-03-27T21:40:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.908+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c008090 osd=1 4 2024-03-27T21:40:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.908+0000 7efc8275c640 10 client.4770.objecter _send_command 3 2024-03-27T21:40:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.908+0000 7efc8275c640 1 -- 172.21.15.60:0/2978648585 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc5c0030c0 con 0x7efc5c002700 2024-03-27T21:40:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.908+0000 7efc7bfff640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7efc5c002700 0x7efc5c002b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:49.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.908+0000 7efc7bfff640 10 monclient: get_auth_request con 0x7efc5c002700 auth_method 0 2024-03-27T21:40:49.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.908+0000 7efc78ff9640 10 client.4770.objecter ms_handle_connect 0x7efc5c002700 2024-03-27T21:40:49.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.908+0000 7efc7bfff640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7efc5c002700 0x7efc5c002b80 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:49.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.909+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7efc5c0030c0 con 0x7efc5c002700 2024-03-27T21:40:49.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.909+0000 7efc78ff9640 10 client.4770.objecter ms_dispatch 0x7efc7c000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:49.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.909+0000 7efc78ff9640 10 client.4770.objecter _finish_command 3 = system:0 2024-03-27T21:40:49.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.909+0000 7efc78ff9640 20 client.4770.objecter put_session s=0x7efc5c008090 osd=1 4 2024-03-27T21:40:49.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.909+0000 7efc78ff9640 15 client.4770.objecter _session_command_op_remove 1 3 2024-03-27T21:40:49.901 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:49.915+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.924+0000 7efc8275c640 10 client.4770.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.924+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign -1 4 2024-03-27T21:40:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.924+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c008090 osd=1 3 2024-03-27T21:40:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.924+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c008090 osd=1 3 2024-03-27T21:40:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.924+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c008090 osd=1 3 2024-03-27T21:40:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.924+0000 7efc8275c640 15 client.4770.objecter _session_command_op_remove -1 4 2024-03-27T21:40:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.924+0000 7efc8275c640 20 client.4770.objecter get_session s=0x7efc5c008090 osd=1 3 2024-03-27T21:40:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.924+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign 1 4 2024-03-27T21:40:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.924+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c008090 osd=1 4 2024-03-27T21:40:49.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.924+0000 7efc8275c640 10 client.4770.objecter _send_command 4 2024-03-27T21:40:49.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.924+0000 7efc8275c640 1 -- 172.21.15.60:0/2978648585 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efc5c003a80 con 0x7efc5c002700 2024-03-27T21:40:49.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.925+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7efc5c003a80 con 0x7efc5c002700 2024-03-27T21:40:49.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.925+0000 7efc78ff9640 10 client.4770.objecter ms_dispatch 0x7efc7c000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:49.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.925+0000 7efc78ff9640 10 client.4770.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:49.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.925+0000 7efc78ff9640 20 client.4770.objecter put_session s=0x7efc5c008090 osd=1 4 2024-03-27T21:40:49.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.925+0000 7efc78ff9640 15 client.4770.objecter _session_command_op_remove 1 4 2024-03-27T21:40:49.912 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.925+0000 7efc8275c640 10 client.4770.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.925+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign -1 5 2024-03-27T21:40:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.925+0000 7efc8275c640 20 client.4770.objecter _get_session osd=-1 returning homeless 2024-03-27T21:40:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.925+0000 7efc8275c640 10 client.4770.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.925+0000 7efc8275c640 10 monclient: _renew_subs 2024-03-27T21:40:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.925+0000 7efc8275c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.925+0000 7efc8275c640 1 -- 172.21.15.60:0/2978648585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=46}) v3 -- 0x7efc5c003bf0 con 0x7efc7c13e810 2024-03-27T21:40:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.925+0000 7efc8275c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.925+0000 7efc8275c640 1 -- 172.21.15.60:0/2978648585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efc5c007eb0 con 0x7efc7c13e810 2024-03-27T21:40:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.926+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_get_version_reply(handle=1 version=45) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc700011e0 con 0x7efc7c13e810 2024-03-27T21:40:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.926+0000 7efc78ff9640 10 monclient: handle_get_version_reply finishing 1 version 45 2024-03-27T21:40:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.926+0000 7efc7a7fc640 10 client.4770.objecter _check_command_map_dne tid 5 current 45 map_dne_bound 45 2024-03-27T21:40:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.926+0000 7efc7a7fc640 10 client.4770.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:40:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.926+0000 7efc7a7fc640 15 client.4770.objecter _session_command_op_remove -1 5 2024-03-27T21:40:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:40:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.926+0000 7efc8275c640 10 client.4770.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.926+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign -1 6 2024-03-27T21:40:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.926+0000 7efc8275c640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7efc5c002070 0x7efc5c00bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.926+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c003530 osd=3 3 2024-03-27T21:40:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.926+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c003530 osd=3 3 2024-03-27T21:40:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.926+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c003530 osd=3 3 2024-03-27T21:40:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.926+0000 7efc8275c640 15 client.4770.objecter _session_command_op_remove -1 6 2024-03-27T21:40:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.926+0000 7efc8275c640 20 client.4770.objecter get_session s=0x7efc5c003530 osd=3 3 2024-03-27T21:40:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.926+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign 3 6 2024-03-27T21:40:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.926+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c003530 osd=3 4 2024-03-27T21:40:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.926+0000 7efc8275c640 10 client.4770.objecter _send_command 6 2024-03-27T21:40:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.926+0000 7efc8275c640 1 -- 172.21.15.60:0/2978648585 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc5c003870 con 0x7efc5c002070 2024-03-27T21:40:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.926+0000 7efc7b7fe640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7efc5c002070 0x7efc5c00bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.926+0000 7efc7b7fe640 10 monclient: get_auth_request con 0x7efc5c002070 auth_method 0 2024-03-27T21:40:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.926+0000 7efc78ff9640 10 client.4770.objecter ms_handle_connect 0x7efc5c002070 2024-03-27T21:40:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.926+0000 7efc7b7fe640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7efc5c002070 0x7efc5c00bd50 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:49.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.927+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7efc5c003870 con 0x7efc5c002070 2024-03-27T21:40:49.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.927+0000 7efc78ff9640 10 client.4770.objecter ms_dispatch 0x7efc7c000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:40:49.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.927+0000 7efc78ff9640 10 client.4770.objecter _finish_command 6 = system:0 2024-03-27T21:40:49.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.927+0000 7efc78ff9640 20 client.4770.objecter put_session s=0x7efc5c003530 osd=3 4 2024-03-27T21:40:49.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.927+0000 7efc78ff9640 15 client.4770.objecter _session_command_op_remove 3 6 2024-03-27T21:40:49.917 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:49.917 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.934+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== mon.0 v2:172.21.15.60:3300/0 8 ==== osd_map(46..46 src has 1..46) v4 ==== 304+0+0 (secure 0 0 0) 0x7efc70057760 con 0x7efc7c13e810 2024-03-27T21:40:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.934+0000 7efc78ff9640 10 client.4770.objecter ms_dispatch 0x7efc7c000b90 osd_map(46..46 src has 1..46) v4 2024-03-27T21:40:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.934+0000 7efc78ff9640 3 client.4770.objecter handle_osd_map got epochs [46,46] > 45 2024-03-27T21:40:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.934+0000 7efc78ff9640 3 client.4770.objecter handle_osd_map decoding incremental epoch 46 2024-03-27T21:40:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.934+0000 7efc78ff9640 20 client.4770.objecter dump_active .. 0 homeless 2024-03-27T21:40:49.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.936+0000 7f5533fff640 1 -- 172.21.15.60:0/3726029195 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v46) v1 ==== 101+0+0 (secure 0 0 0) 0x7f5544018090 con 0x7f554c1416f0 2024-03-27T21:40:49.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.936+0000 7f5533fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:40:49.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.936+0000 7f5533fff640 10 monclient: _finish_command 2 = system:0 nodeep-scrub is unset 2024-03-27T21:40:49.923 INFO:teuthology.orchestra.run.smithi060.stderr:nodeep-scrub is unset 2024-03-27T21:40:49.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.939+0000 7f5551edc640 10 librados: watch_flush enter 2024-03-27T21:40:49.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.939+0000 7f5551edc640 10 librados: watch_flush exit 2024-03-27T21:40:49.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.939+0000 7f5551edc640 20 client.4764.objecter shutdown clearing up homeless session... 2024-03-27T21:40:49.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.939+0000 7f5551edc640 10 client.4764.objecter successfully canceled tick 2024-03-27T21:40:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.939+0000 7f5551edc640 1 -- 172.21.15.60:0/3726029195 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f553806c5e0 msgr2=0x7f553806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.939+0000 7f5551edc640 1 --2- 172.21.15.60:0/3726029195 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f553806c5e0 0x7f553806eaa0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f553c000f60 tx=0x7f553c0017f0 comp rx=0 tx=0).stop 2024-03-27T21:40:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.939+0000 7f5551edc640 10 monclient: shutdown 2024-03-27T21:40:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.939+0000 7f5551edc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.939+0000 7f5551edc640 1 -- 172.21.15.60:0/3726029195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554c1416f0 msgr2=0x7f554c131110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.939+0000 7f5551edc640 1 --2- 172.21.15.60:0/3726029195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554c1416f0 0x7f554c131110 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f554400efc0 tx=0x7f554400c490 comp rx=0 tx=0).stop 2024-03-27T21:40:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.939+0000 7f5551edc640 1 -- 172.21.15.60:0/3726029195 shutdown_connections 2024-03-27T21:40:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.939+0000 7f5551edc640 1 --2- 172.21.15.60:0/3726029195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554c1416f0 0x7f554c131110 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.939+0000 7f5551edc640 1 --2- 172.21.15.60:0/3726029195 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f553806c5e0 0x7f553806eaa0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.939+0000 7f5551edc640 1 --2- 172.21.15.60:0/3726029195 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f554c13b020 0x7f554c13aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.939+0000 7f5551edc640 1 --2- 172.21.15.60:0/3726029195 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f554c0b40b0 0x7f554c134690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.939+0000 7f5551edc640 1 -- 172.21.15.60:0/3726029195 >> 172.21.15.60:0/3726029195 conn(0x7f554c0608e0 msgr2=0x7f554c0b63d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.940+0000 7f5551edc640 1 -- 172.21.15.60:0/3726029195 shutdown_connections 2024-03-27T21:40:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.940+0000 7f5551edc640 1 -- 172.21.15.60:0/3726029195 wait complete. 2024-03-27T21:40:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.940+0000 7f5551edc640 1 librados: shutdown 2024-03-27T21:40:49.929 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:40:49.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.944+0000 7efc8275c640 10 client.4770.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:49.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.944+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign -1 7 2024-03-27T21:40:49.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.944+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c003530 osd=3 3 2024-03-27T21:40:49.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.944+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c003530 osd=3 3 2024-03-27T21:40:49.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.944+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c003530 osd=3 3 2024-03-27T21:40:49.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.944+0000 7efc8275c640 15 client.4770.objecter _session_command_op_remove -1 7 2024-03-27T21:40:49.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.944+0000 7efc8275c640 20 client.4770.objecter get_session s=0x7efc5c003530 osd=3 3 2024-03-27T21:40:49.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.944+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign 3 7 2024-03-27T21:40:49.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.944+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c003530 osd=3 4 2024-03-27T21:40:49.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.944+0000 7efc8275c640 10 client.4770.objecter _send_command 7 2024-03-27T21:40:49.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.944+0000 7efc8275c640 1 -- 172.21.15.60:0/2978648585 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efc5c0015e0 con 0x7efc5c002070 2024-03-27T21:40:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.946+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7efc5c0015e0 con 0x7efc5c002070 2024-03-27T21:40:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.946+0000 7efc78ff9640 10 client.4770.objecter ms_dispatch 0x7efc7c000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.946+0000 7efc78ff9640 10 client.4770.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.946+0000 7efc78ff9640 20 client.4770.objecter put_session s=0x7efc5c003530 osd=3 4 2024-03-27T21:40:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.946+0000 7efc78ff9640 15 client.4770.objecter _session_command_op_remove 3 7 2024-03-27T21:40:49.935 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:49.935 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.946+0000 7efc8275c640 10 client.4770.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.946+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign -1 8 2024-03-27T21:40:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.946+0000 7efc8275c640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7efc5c00b8a0 0x7efc5c011510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.946+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c001990 osd=4 3 2024-03-27T21:40:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.946+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c001990 osd=4 3 2024-03-27T21:40:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.946+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c001990 osd=4 3 2024-03-27T21:40:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.946+0000 7efc8275c640 15 client.4770.objecter _session_command_op_remove -1 8 2024-03-27T21:40:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.946+0000 7efc8275c640 20 client.4770.objecter get_session s=0x7efc5c001990 osd=4 3 2024-03-27T21:40:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.946+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign 4 8 2024-03-27T21:40:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.946+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c001990 osd=4 4 2024-03-27T21:40:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.946+0000 7efc8275c640 10 client.4770.objecter _send_command 8 2024-03-27T21:40:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.946+0000 7efc8275c640 1 -- 172.21.15.60:0/2978648585 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc5c0143c0 con 0x7efc5c00b8a0 2024-03-27T21:40:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.947+0000 7efc80cf9640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7efc5c00b8a0 0x7efc5c011510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:49.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.947+0000 7efc80cf9640 10 monclient: get_auth_request con 0x7efc5c00b8a0 auth_method 0 2024-03-27T21:40:49.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.947+0000 7efc78ff9640 10 client.4770.objecter ms_handle_connect 0x7efc5c00b8a0 2024-03-27T21:40:49.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.947+0000 7efc80cf9640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7efc5c00b8a0 0x7efc5c011510 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:49.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.948+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7efc5c0143c0 con 0x7efc5c00b8a0 2024-03-27T21:40:49.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.948+0000 7efc78ff9640 10 client.4770.objecter ms_dispatch 0x7efc7c000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:40:49.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.948+0000 7efc78ff9640 10 client.4770.objecter _finish_command 8 = system:0 2024-03-27T21:40:49.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.948+0000 7efc78ff9640 20 client.4770.objecter put_session s=0x7efc5c001990 osd=4 4 2024-03-27T21:40:49.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.948+0000 7efc78ff9640 15 client.4770.objecter _session_command_op_remove 4 8 2024-03-27T21:40:49.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.964+0000 7efc8275c640 10 client.4770.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.964+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign -1 9 2024-03-27T21:40:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.964+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c001990 osd=4 3 2024-03-27T21:40:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.964+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c001990 osd=4 3 2024-03-27T21:40:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.964+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c001990 osd=4 3 2024-03-27T21:40:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.964+0000 7efc8275c640 15 client.4770.objecter _session_command_op_remove -1 9 2024-03-27T21:40:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.964+0000 7efc8275c640 20 client.4770.objecter get_session s=0x7efc5c001990 osd=4 3 2024-03-27T21:40:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.964+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign 4 9 2024-03-27T21:40:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.964+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c001990 osd=4 4 2024-03-27T21:40:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.964+0000 7efc8275c640 10 client.4770.objecter _send_command 9 2024-03-27T21:40:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.964+0000 7efc8275c640 1 -- 172.21.15.60:0/2978648585 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efc5c00ae10 con 0x7efc5c00b8a0 2024-03-27T21:40:49.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.966+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7efc5c00ae10 con 0x7efc5c00b8a0 2024-03-27T21:40:49.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.966+0000 7efc78ff9640 10 client.4770.objecter ms_dispatch 0x7efc7c000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.966+0000 7efc78ff9640 10 client.4770.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.966+0000 7efc78ff9640 20 client.4770.objecter put_session s=0x7efc5c001990 osd=4 4 2024-03-27T21:40:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.966+0000 7efc78ff9640 15 client.4770.objecter _session_command_op_remove 4 9 2024-03-27T21:40:49.953 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.966+0000 7efc8275c640 10 client.4770.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.966+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign -1 10 2024-03-27T21:40:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.966+0000 7efc8275c640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7efc5c00cf60 0x7efc5c011a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.966+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c00abd0 osd=5 3 2024-03-27T21:40:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.966+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c00abd0 osd=5 3 2024-03-27T21:40:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.966+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c00abd0 osd=5 3 2024-03-27T21:40:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.966+0000 7efc8275c640 15 client.4770.objecter _session_command_op_remove -1 10 2024-03-27T21:40:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.966+0000 7efc8275c640 20 client.4770.objecter get_session s=0x7efc5c00abd0 osd=5 3 2024-03-27T21:40:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.966+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign 5 10 2024-03-27T21:40:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.966+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c00abd0 osd=5 4 2024-03-27T21:40:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.966+0000 7efc8275c640 10 client.4770.objecter _send_command 10 2024-03-27T21:40:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.966+0000 7efc8275c640 1 -- 172.21.15.60:0/2978648585 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc5c017ce0 con 0x7efc5c00cf60 2024-03-27T21:40:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.967+0000 7efc7bfff640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7efc5c00cf60 0x7efc5c011a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.967+0000 7efc7bfff640 10 monclient: get_auth_request con 0x7efc5c00cf60 auth_method 0 2024-03-27T21:40:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.967+0000 7efc78ff9640 10 client.4770.objecter ms_handle_connect 0x7efc5c00cf60 2024-03-27T21:40:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.967+0000 7efc7bfff640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7efc5c00cf60 0x7efc5c011a50 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.968+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7efc5c017ce0 con 0x7efc5c00cf60 2024-03-27T21:40:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.968+0000 7efc78ff9640 10 client.4770.objecter ms_dispatch 0x7efc7c000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:40:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.968+0000 7efc78ff9640 10 client.4770.objecter _finish_command 10 = system:0 2024-03-27T21:40:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.968+0000 7efc78ff9640 20 client.4770.objecter put_session s=0x7efc5c00abd0 osd=5 4 2024-03-27T21:40:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.968+0000 7efc78ff9640 15 client.4770.objecter _session_command_op_remove 5 10 2024-03-27T21:40:49.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.984+0000 7efc8275c640 10 client.4770.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:49.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.984+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign -1 11 2024-03-27T21:40:49.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.984+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c00abd0 osd=5 3 2024-03-27T21:40:49.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.984+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c00abd0 osd=5 3 2024-03-27T21:40:49.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.984+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c00abd0 osd=5 3 2024-03-27T21:40:49.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.984+0000 7efc8275c640 15 client.4770.objecter _session_command_op_remove -1 11 2024-03-27T21:40:49.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.984+0000 7efc8275c640 20 client.4770.objecter get_session s=0x7efc5c00abd0 osd=5 3 2024-03-27T21:40:49.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.984+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign 5 11 2024-03-27T21:40:49.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.984+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c00abd0 osd=5 4 2024-03-27T21:40:49.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.984+0000 7efc8275c640 10 client.4770.objecter _send_command 11 2024-03-27T21:40:49.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.984+0000 7efc8275c640 1 -- 172.21.15.60:0/2978648585 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efc5c011f90 con 0x7efc5c00cf60 2024-03-27T21:40:49.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.985+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7efc5c011f90 con 0x7efc5c00cf60 2024-03-27T21:40:49.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.985+0000 7efc78ff9640 10 client.4770.objecter ms_dispatch 0x7efc7c000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:49.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.985+0000 7efc78ff9640 10 client.4770.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:49.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.985+0000 7efc78ff9640 20 client.4770.objecter put_session s=0x7efc5c00abd0 osd=5 4 2024-03-27T21:40:49.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.985+0000 7efc78ff9640 15 client.4770.objecter _session_command_op_remove 5 11 2024-03-27T21:40:49.972 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:49.973 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:49.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.985+0000 7efc8275c640 10 client.4770.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:49.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.986+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign -1 12 2024-03-27T21:40:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.986+0000 7efc8275c640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7efc5c012670 0x7efc5c012af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.986+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c011f90 osd=6 3 2024-03-27T21:40:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.986+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c011f90 osd=6 3 2024-03-27T21:40:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.986+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c011f90 osd=6 3 2024-03-27T21:40:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.986+0000 7efc8275c640 15 client.4770.objecter _session_command_op_remove -1 12 2024-03-27T21:40:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.986+0000 7efc8275c640 20 client.4770.objecter get_session s=0x7efc5c011f90 osd=6 3 2024-03-27T21:40:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.986+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign 6 12 2024-03-27T21:40:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.986+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c011f90 osd=6 4 2024-03-27T21:40:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.986+0000 7efc8275c640 10 client.4770.objecter _send_command 12 2024-03-27T21:40:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.986+0000 7efc8275c640 1 -- 172.21.15.60:0/2978648585 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc5c01ae70 con 0x7efc5c012670 2024-03-27T21:40:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.986+0000 7efc7b7fe640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7efc5c012670 0x7efc5c012af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.986+0000 7efc7b7fe640 10 monclient: get_auth_request con 0x7efc5c012670 auth_method 0 2024-03-27T21:40:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.986+0000 7efc78ff9640 10 client.4770.objecter ms_handle_connect 0x7efc5c012670 2024-03-27T21:40:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.986+0000 7efc7b7fe640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7efc5c012670 0x7efc5c012af0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.987+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7efc5c01ae70 con 0x7efc5c012670 2024-03-27T21:40:49.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.987+0000 7efc78ff9640 10 client.4770.objecter ms_dispatch 0x7efc7c000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:40:49.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.987+0000 7efc78ff9640 10 client.4770.objecter _finish_command 12 = system:0 2024-03-27T21:40:49.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.987+0000 7efc78ff9640 20 client.4770.objecter put_session s=0x7efc5c011f90 osd=6 4 2024-03-27T21:40:49.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:49.987+0000 7efc78ff9640 15 client.4770.objecter _session_command_op_remove 6 12 2024-03-27T21:40:49.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.001+0000 7efc8275c640 10 client.4770.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:49.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.001+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign -1 13 2024-03-27T21:40:49.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.001+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c011f90 osd=6 3 2024-03-27T21:40:49.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.001+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c011f90 osd=6 3 2024-03-27T21:40:49.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.001+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c011f90 osd=6 3 2024-03-27T21:40:49.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.001+0000 7efc8275c640 15 client.4770.objecter _session_command_op_remove -1 13 2024-03-27T21:40:49.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.001+0000 7efc8275c640 20 client.4770.objecter get_session s=0x7efc5c011f90 osd=6 3 2024-03-27T21:40:49.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.001+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign 6 13 2024-03-27T21:40:49.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.001+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c011f90 osd=6 4 2024-03-27T21:40:49.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.001+0000 7efc8275c640 10 client.4770.objecter _send_command 13 2024-03-27T21:40:49.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.001+0000 7efc8275c640 1 -- 172.21.15.60:0/2978648585 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efc5c0185e0 con 0x7efc5c012670 2024-03-27T21:40:49.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.003+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7efc5c0185e0 con 0x7efc5c012670 2024-03-27T21:40:49.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.003+0000 7efc78ff9640 10 client.4770.objecter ms_dispatch 0x7efc7c000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:49.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.003+0000 7efc78ff9640 10 client.4770.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:49.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.003+0000 7efc78ff9640 20 client.4770.objecter put_session s=0x7efc5c011f90 osd=6 4 2024-03-27T21:40:49.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.003+0000 7efc78ff9640 15 client.4770.objecter _session_command_op_remove 6 13 2024-03-27T21:40:49.989 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:49.990 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:49.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.003+0000 7efc8275c640 10 client.4770.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.003+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign -1 14 2024-03-27T21:40:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.003+0000 7efc8275c640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7efc5c018c60 0x7efc5c01b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.003+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c0185e0 osd=7 3 2024-03-27T21:40:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.003+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c0185e0 osd=7 3 2024-03-27T21:40:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.003+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c0185e0 osd=7 3 2024-03-27T21:40:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.003+0000 7efc8275c640 15 client.4770.objecter _session_command_op_remove -1 14 2024-03-27T21:40:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.003+0000 7efc8275c640 20 client.4770.objecter get_session s=0x7efc5c0185e0 osd=7 3 2024-03-27T21:40:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.003+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign 7 14 2024-03-27T21:40:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.003+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c0185e0 osd=7 4 2024-03-27T21:40:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.003+0000 7efc8275c640 10 client.4770.objecter _send_command 14 2024-03-27T21:40:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.003+0000 7efc8275c640 1 -- 172.21.15.60:0/2978648585 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc5c01e090 con 0x7efc5c018c60 2024-03-27T21:40:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.003+0000 7efc80cf9640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7efc5c018c60 0x7efc5c01b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.003+0000 7efc80cf9640 10 monclient: get_auth_request con 0x7efc5c018c60 auth_method 0 2024-03-27T21:40:49.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.003+0000 7efc78ff9640 10 client.4770.objecter ms_handle_connect 0x7efc5c018c60 2024-03-27T21:40:49.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.003+0000 7efc80cf9640 1 --2- 172.21.15.60:0/2978648585 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7efc5c018c60 0x7efc5c01b0d0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:49.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.004+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7efc5c01e090 con 0x7efc5c018c60 2024-03-27T21:40:49.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.004+0000 7efc78ff9640 10 client.4770.objecter ms_dispatch 0x7efc7c000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:40:49.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.004+0000 7efc78ff9640 10 client.4770.objecter _finish_command 14 = system:0 2024-03-27T21:40:49.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.004+0000 7efc78ff9640 20 client.4770.objecter put_session s=0x7efc5c0185e0 osd=7 4 2024-03-27T21:40:49.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.004+0000 7efc78ff9640 15 client.4770.objecter _session_command_op_remove 7 14 2024-03-27T21:40:50.002 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:50.016+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:50.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.018+0000 7efc8275c640 10 client.4770.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:50.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.018+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign -1 15 2024-03-27T21:40:50.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.018+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c0185e0 osd=7 3 2024-03-27T21:40:50.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.018+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c0185e0 osd=7 3 2024-03-27T21:40:50.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.018+0000 7efc8275c640 20 client.4770.objecter _get_session s=0x7efc5c0185e0 osd=7 3 2024-03-27T21:40:50.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.018+0000 7efc8275c640 15 client.4770.objecter _session_command_op_remove -1 15 2024-03-27T21:40:50.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.018+0000 7efc8275c640 20 client.4770.objecter get_session s=0x7efc5c0185e0 osd=7 3 2024-03-27T21:40:50.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.018+0000 7efc8275c640 15 client.4770.objecter _session_command_op_assign 7 15 2024-03-27T21:40:50.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.018+0000 7efc8275c640 20 client.4770.objecter put_session s=0x7efc5c0185e0 osd=7 4 2024-03-27T21:40:50.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.018+0000 7efc8275c640 10 client.4770.objecter _send_command 15 2024-03-27T21:40:50.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.018+0000 7efc8275c640 1 -- 172.21.15.60:0/2978648585 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efc5c01e090 con 0x7efc5c018c60 2024-03-27T21:40:50.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.020+0000 7efc78ff9640 1 -- 172.21.15.60:0/2978648585 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7efc5c01e090 con 0x7efc5c018c60 2024-03-27T21:40:50.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.020+0000 7efc78ff9640 10 client.4770.objecter ms_dispatch 0x7efc7c000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:50.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.020+0000 7efc78ff9640 10 client.4770.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:50.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.020+0000 7efc78ff9640 20 client.4770.objecter put_session s=0x7efc5c0185e0 osd=7 4 2024-03-27T21:40:50.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:50.020+0000 7efc78ff9640 15 client.4770.objecter _session_command_op_remove 7 15 2024-03-27T21:40:50.007 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:50.007 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:50.009 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:40:50.010 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:40:50.044 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:50.045 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:50.054 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:40:50.138 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:50.152+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:50.202 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:50.203 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:50.204 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:50.218+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:50.211 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:40:50.305 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:50.319+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:50.336 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 2.3 --file /var/log/ceph/exp.2.3.2 2024-03-27T21:40:50.361 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:50.361 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:50.371 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:40:50.406 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:50.420+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:50.507 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:50.521+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:50.519 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:50.520 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:50.528 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:40:50.608 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:50.622+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:50.680 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:50.680 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:50.691 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:40:50.708 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:50.722+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:50.807 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:50.807 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:50.809 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:50.823+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:50.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:40:50.910 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:50.924+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:50.935 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:50.935 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:50.944 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:40:51.012 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:51.025+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:51.050 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:51.050 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:51.059 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:40:51.142 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:51.156+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:51.204 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:51.204 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:51.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:40:51.214 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:51.227+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:51.315 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:51.328+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:51.360 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:51.360 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:51.368 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:40:51.417 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:51.429+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:51.420 INFO:teuthology.orchestra.run.smithi060.stderr:Exporting 2.3 info 2.3( empty local-lis/les=33/34 n=0 ec=19/19 lis/c=33/33 les/c/f=34/34/0 sis=33) 2024-03-27T21:40:51.426 INFO:teuthology.orchestra.run.smithi060.stderr:Export successful 2024-03-27T21:40:51.513 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:51.513 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:51.515 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:51.529+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:51.522 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:40:51.616 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:51.630+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:51.627 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:51.627 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:51.635 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:40:51.717 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:51.731+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:51.741 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:51.741 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:51.750 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:40:51.819 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:51.833+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:51.855 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:51.855 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:51.864 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:40:51.920 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:51.933+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:51.935 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 2.3 2024-03-27T21:40:51.942 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-27T21:40:51.969 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:51.970 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:51.978 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:40:52.011 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:40:52.021 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:52.035+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:52.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.097+0000 7f8bfaee0640 10 monclient: get_monmap_and_config 2024-03-27T21:40:52.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.097+0000 7f8bfaee0640 10 monclient: build_initial_monmap 2024-03-27T21:40:52.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.097+0000 7f8bfaee0640 10 monclient: monmap: 2024-03-27T21:40:52.084 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:52.084 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:52.084 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:52.098911+0000 2024-03-27T21:40:52.084 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:52.098911+0000 2024-03-27T21:40:52.084 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:52.085 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:52.085 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:52.085 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:52.085 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:52.085 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:52.085 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:52.085 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:52.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.098+0000 7f8bfaee0640 1 Processor -- start 2024-03-27T21:40:52.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.098+0000 7f8bfaee0640 1 -- start start 2024-03-27T21:40:52.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.098+0000 7f8bfaee0640 10 monclient: init 2024-03-27T21:40:52.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bfaee0640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:52.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bfaee0640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-03-27T21:40:52.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bfaee0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8bf40b40b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:52.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bfaee0640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8bf413b020 0x7f8bf413b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:52.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bfaee0640 10 monclient(hunting): picked mon.noname-b con 0x7f8bf413b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:52.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bfaee0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8bf41416f0 0x7f8bf4141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:52.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bfaee0640 10 monclient(hunting): picked mon.noname-a con 0x7f8bf41416f0 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:52.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bfaee0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bf40b3830 con 0x7f8bf40b40b0 2024-03-27T21:40:52.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bfaee0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:52.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bfaee0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bf40b4490 con 0x7f8bf41416f0 2024-03-27T21:40:52.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bfaee0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:52.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bfaee0640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf40aa520 con 0x7f8bf413b020 2024-03-27T21:40:52.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bfaee0640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:52.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bfaee0640 10 monclient(hunting): authenticate will time out at 893.320297s 2024-03-27T21:40:52.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf3fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8bf413b020 0x7f8bf413b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:52.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf947d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8bf41416f0 0x7f8bf4141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:52.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf3fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8bf413b020 0x7f8bf413b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:57756/0 (socket says 172.21.15.60:57756) 2024-03-27T21:40:52.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf3fff640 1 -- 172.21.15.60:0/1739165334 learned_addr learned my addr 172.21.15.60:0/1739165334 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:52.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf947d640 10 monclient(hunting): get_auth_request con 0x7f8bf41416f0 auth_method 0 2024-03-27T21:40:52.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf947d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:52.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf947d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:52.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf947d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:52.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf3fff640 10 monclient(hunting): get_auth_request con 0x7f8bf413b020 auth_method 0 2024-03-27T21:40:52.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:52.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:52.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:52.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:52.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:52.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:52.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf947d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:52.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf947d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:52.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf947d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:52.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf947d640 10 monclient(hunting): handle_auth_done global_id 4773 payload 274 2024-03-27T21:40:52.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf37fe640 1 -- 172.21.15.60:0/1739165334 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147900040 0 0) 0x7f8bf40b3830 con 0x7f8bf40b40b0 2024-03-27T21:40:52.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf947d640 1 -- 172.21.15.60:0/1739165334 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8bf413b020 msgr2=0x7f8bf413b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:52.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf947d640 1 --2- 172.21.15.60:0/1739165334 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8bf413b020 0x7f8bf413b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:52.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf947d640 1 -- 172.21.15.60:0/1739165334 >> v1:172.21.15.60:6789/0 conn(0x7f8bf40b40b0 legacy=0x7f8bf40a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:52.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf947d640 10 monclient: _finish_hunting 0 2024-03-27T21:40:52.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf947d640 1 monclient: found mon.noname-a 2024-03-27T21:40:52.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf947d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:52.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf947d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:52.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf947d640 1 -- 172.21.15.60:0/1739165334 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf40a9470 con 0x7f8bf41416f0 2024-03-27T21:40:52.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf37fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:40:52.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.099+0000 7f8bf3fff640 1 --2- 172.21.15.60:0/1739165334 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8bf413b020 0x7f8bf413b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:52.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.101+0000 7f8bf947d640 1 --2- 172.21.15.60:0/1739165334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bf41416f0 0x7f8bf4141b80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8bec009d90 tx=0x7f8bec02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5941c23ff36a48a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:52.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.101+0000 7f8bf37fe640 1 -- 172.21.15.60:0/1739165334 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec02cc50 con 0x7f8bf41416f0 2024-03-27T21:40:52.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.101+0000 7f8bf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:52.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.101+0000 7f8bf37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:52.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.101+0000 7f8bf37fe640 10 monclient: dump: 2024-03-27T21:40:52.096 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:52.096 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:52.096 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:52.096 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:52.096 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:52.097 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:52.097 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:52.097 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:52.097 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:52.097 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:52.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.101+0000 7f8bf37fe640 10 monclient: _finish_auth 0 2024-03-27T21:40:52.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.101+0000 7f8bf37fe640 10 monclient: _check_auth_tickets 2024-03-27T21:40:52.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.101+0000 7f8bf37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:52.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.101+0000 7f8bfaee0640 5 monclient: authenticate success, global_id 4773 2024-03-27T21:40:52.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.101+0000 7f8bfaee0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:52.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.101+0000 7f8bf37fe640 1 -- 172.21.15.60:0/1739165334 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bec02cdb0 con 0x7f8bf41416f0 2024-03-27T21:40:52.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.101+0000 7f8bf37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:52.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.101+0000 7f8bfaee0640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:52.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.101+0000 7f8bf37fe640 1 -- 172.21.15.60:0/1739165334 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec034660 con 0x7f8bf41416f0 2024-03-27T21:40:52.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.101+0000 7f8bf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:52.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.101+0000 7f8bfaee0640 4 set_mon_vals no callback set 2024-03-27T21:40:52.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.101+0000 7f8bf37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:52.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.101+0000 7f8bf37fe640 10 monclient: dump: 2024-03-27T21:40:52.098 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:52.098 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:52.098 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:52.098 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:52.098 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:52.098 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:52.098 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:52.098 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:52.098 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:52.098 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:52.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.102+0000 7f8bfaee0640 10 monclient: shutdown 2024-03-27T21:40:52.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.102+0000 7f8bfaee0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:52.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.102+0000 7f8bfaee0640 1 -- 172.21.15.60:0/1739165334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bf41416f0 msgr2=0x7f8bf4141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:52.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.102+0000 7f8bfaee0640 1 --2- 172.21.15.60:0/1739165334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bf41416f0 0x7f8bf4141b80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8bec009d90 tx=0x7f8bec02bea0 comp rx=0 tx=0).stop 2024-03-27T21:40:52.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.102+0000 7f8bfaee0640 1 -- 172.21.15.60:0/1739165334 shutdown_connections 2024-03-27T21:40:52.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.102+0000 7f8bfaee0640 1 --2- 172.21.15.60:0/1739165334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bf41416f0 0x7f8bf4141b80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:52.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.102+0000 7f8bfaee0640 1 --2- 172.21.15.60:0/1739165334 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8bf413b020 0x7f8bf413b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:52.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.102+0000 7f8bfaee0640 1 -- 172.21.15.60:0/1739165334 >> 172.21.15.60:0/1739165334 conn(0x7f8bf40608e0 msgr2=0x7f8bf4060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:52.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.102+0000 7f8bfaee0640 1 -- 172.21.15.60:0/1739165334 shutdown_connections 2024-03-27T21:40:52.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.102+0000 7f8bfaee0640 1 -- 172.21.15.60:0/1739165334 wait complete. 2024-03-27T21:40:52.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.102+0000 7f8bfaee0640 10 monclient: build_initial_monmap 2024-03-27T21:40:52.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.102+0000 7f8bfaee0640 10 monclient: monmap: 2024-03-27T21:40:52.100 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:52.100 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:52.100 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:52.100 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:52.100 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:52.100 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:52.101 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:52.101 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:52.101 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:52.101 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:52.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 1 librados: starting msgr at 2024-03-27T21:40:52.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 1 librados: starting objecter 2024-03-27T21:40:52.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 1 Processor -- start 2024-03-27T21:40:52.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 1 -- start start 2024-03-27T21:40:52.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 1 librados: setting wanted keys 2024-03-27T21:40:52.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 1 librados: calling monclient init 2024-03-27T21:40:52.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 10 monclient: init 2024-03-27T21:40:52.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:52.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:40:52.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bf40b40b0 0x7f8bf413d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:52.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 10 monclient(hunting): picked mon.noname-b con 0x7f8bf40b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:52.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bf413b020 0x7f8bf41d9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:52.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 10 monclient(hunting): picked mon.noname-a con 0x7f8bf413b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:52.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8bf41416f0 0x7f8bf41361f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:52.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 10 monclient(hunting): picked mon.noname-c con 0x7f8bf41416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:52.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:52.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf413e380 con 0x7f8bf413b020 2024-03-27T21:40:52.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:52.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf40aca90 con 0x7f8bf40b40b0 2024-03-27T21:40:52.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:52.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf40acf70 con 0x7f8bf41416f0 2024-03-27T21:40:52.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:52.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.103+0000 7f8bfaee0640 10 monclient(hunting): authenticate will time out at 893.324821s 2024-03-27T21:40:52.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bf413b020 0x7f8bf41d9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:52.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf947d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8bf41416f0 0x7f8bf41361f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:52.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bf413b020 0x7f8bf41d9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47126/0 (socket says 172.21.15.60:47126) 2024-03-27T21:40:52.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf3fff640 1 -- 172.21.15.60:0/2605805512 learned_addr learned my addr 172.21.15.60:0/2605805512 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:52.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf3fff640 10 monclient(hunting): get_auth_request con 0x7f8bf413b020 auth_method 0 2024-03-27T21:40:52.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:52.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:52.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:52.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf947d640 10 monclient(hunting): get_auth_request con 0x7f8bf41416f0 auth_method 0 2024-03-27T21:40:52.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf947d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:52.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf947d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:52.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf947d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:52.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf947d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:52.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf947d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:52.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf947d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:52.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:52.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:52.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf8c7c640 1 --2- 172.21.15.60:0/2605805512 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bf40b40b0 0x7f8bf413d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:52.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:52.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf8c7c640 10 monclient(hunting): get_auth_request con 0x7f8bf40b40b0 auth_method 0 2024-03-27T21:40:52.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf8c7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:52.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf8c7c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:52.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf8c7c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:52.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf3fff640 10 monclient(hunting): handle_auth_done global_id 4779 payload 824 2024-03-27T21:40:52.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf3fff640 1 -- 172.21.15.60:0/2605805512 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8bf41416f0 msgr2=0x7f8bf41361f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:52.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf3fff640 1 --2- 172.21.15.60:0/2605805512 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8bf41416f0 0x7f8bf41361f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:52.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf3fff640 1 -- 172.21.15.60:0/2605805512 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bf40b40b0 msgr2=0x7f8bf413d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:52.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf3fff640 1 --2- 172.21.15.60:0/2605805512 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bf40b40b0 0x7f8bf413d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:52.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf3fff640 10 monclient: _finish_hunting 0 2024-03-27T21:40:52.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf3fff640 1 monclient: found mon.noname-a 2024-03-27T21:40:52.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:52.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:52.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf3fff640 1 -- 172.21.15.60:0/2605805512 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be00094d0 con 0x7f8bf413b020 2024-03-27T21:40:52.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf947d640 1 --2- 172.21.15.60:0/2605805512 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8bf41416f0 0x7f8bf41361f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:52.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf8c7c640 1 --2- 172.21.15.60:0/2605805512 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bf40b40b0 0x7f8bf413d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:40:52.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf3fff640 1 --2- 172.21.15.60:0/2605805512 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bf413b020 0x7f8bf41d9610 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8be000e9d0 tx=0x7f8be000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:52.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf17fa640 10 client.?.objecter ms_handle_connect 0x7f8bf413b020 2024-03-27T21:40:52.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf17fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:52.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf17fa640 1 -- 172.21.15.60:0/2605805512 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be000ca60 con 0x7f8bf413b020 2024-03-27T21:40:52.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:52.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:52.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf17fa640 10 monclient: dump: 2024-03-27T21:40:52.106 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:52.106 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:52.106 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:52.106 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:52.106 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:52.106 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:52.106 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:52.106 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:52.106 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:52.106 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:52.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf17fa640 10 monclient: _finish_auth 0 2024-03-27T21:40:52.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf17fa640 10 monclient: _check_auth_tickets 2024-03-27T21:40:52.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:52.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf17fa640 1 -- 172.21.15.60:0/2605805512 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be000cbc0 con 0x7f8bf413b020 2024-03-27T21:40:52.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bfaee0640 5 monclient: authenticate success, global_id 4779 2024-03-27T21:40:52.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bfaee0640 10 monclient: _renew_subs 2024-03-27T21:40:52.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bfaee0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:52.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bfaee0640 1 -- 172.21.15.60:0/2605805512 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bec009a70 con 0x7f8bf413b020 2024-03-27T21:40:52.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:52.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf17fa640 1 -- 172.21.15.60:0/2605805512 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be0013470 con 0x7f8bf413b020 2024-03-27T21:40:52.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:52.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:52.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf17fa640 10 monclient: dump: 2024-03-27T21:40:52.108 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:52.108 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:52.108 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:52.108 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:52.108 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:52.108 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:52.108 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:52.108 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:52.108 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:52.108 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:52.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bf2ffd640 4 set_mon_vals no callback set 2024-03-27T21:40:52.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bfaee0640 10 client.4779.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:52.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bfaee0640 10 monclient: _renew_subs 2024-03-27T21:40:52.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bfaee0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:52.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bfaee0640 1 -- 172.21.15.60:0/2605805512 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf414ba00 con 0x7f8bf413b020 2024-03-27T21:40:52.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.104+0000 7f8bfaee0640 1 librados: init done 2024-03-27T21:40:52.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.105+0000 7f8bfaee0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:52.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.106+0000 7f8bfaee0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:52.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.106+0000 7f8bfaee0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:52.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.106+0000 7f8bfaee0640 1 -- 172.21.15.60:0/2605805512 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf4069470 con 0x7f8bf413b020 2024-03-27T21:40:52.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.106+0000 7f8bf17fa640 1 -- 172.21.15.60:0/2605805512 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8be00135d0 con 0x7f8bf413b020 2024-03-27T21:40:52.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.107+0000 7f8bf17fa640 1 --2- 172.21.15.60:0/2605805512 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f8bd806c590 0x7f8bd806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:52.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.107+0000 7f8bf17fa640 1 -- 172.21.15.60:0/2605805512 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f8be0089420 con 0x7f8bf413b020 2024-03-27T21:40:52.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.107+0000 7f8bf17fa640 10 client.4779.objecter ms_dispatch 0x7f8bf4000b90 osd_map(46..46 src has 1..46) v4 2024-03-27T21:40:52.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.107+0000 7f8bf17fa640 3 client.4779.objecter handle_osd_map got epochs [46,46] > 0 2024-03-27T21:40:52.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.107+0000 7f8bf17fa640 3 client.4779.objecter handle_osd_map decoding full epoch 46 2024-03-27T21:40:52.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.107+0000 7f8bf17fa640 20 client.4779.objecter dump_active .. 0 homeless 2024-03-27T21:40:52.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.107+0000 7f8bf8c7c640 1 --2- 172.21.15.60:0/2605805512 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f8bd806c590 0x7f8bd806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:52.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.107+0000 7f8bf8c7c640 10 monclient: get_auth_request con 0x7f8bd806c590 auth_method 0 2024-03-27T21:40:52.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.107+0000 7f8bf17fa640 10 client.4779.objecter ms_handle_connect 0x7f8bd806c590 2024-03-27T21:40:52.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.107+0000 7f8bf8c7c640 1 --2- 172.21.15.60:0/2605805512 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f8bd806c590 0x7f8bd806ea50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8be8000f60 tx=0x7f8be8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:52.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.112+0000 7f8bf17fa640 1 -- 172.21.15.60:0/2605805512 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8be0054890 con 0x7f8bf413b020 2024-03-27T21:40:52.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.112+0000 7f8bf17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:52.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.112+0000 7f8bf17fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:52.121 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:52.135+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:52.124 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:52.125 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:52.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.141+0000 7fdeb0922640 10 monclient: get_monmap_and_config 2024-03-27T21:40:52.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.141+0000 7fdeb0922640 10 monclient: build_initial_monmap 2024-03-27T21:40:52.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.141+0000 7fdeb0922640 10 monclient: monmap: 2024-03-27T21:40:52.128 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:52.128 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:52.128 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:52.142794+0000 2024-03-27T21:40:52.128 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:52.142794+0000 2024-03-27T21:40:52.128 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:52.128 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:52.128 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:52.128 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:52.128 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:52.128 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:52.128 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:52.129 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:52.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.142+0000 7fdeb0922640 1 Processor -- start 2024-03-27T21:40:52.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.142+0000 7fdeb0922640 1 -- start start 2024-03-27T21:40:52.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.142+0000 7fdeb0922640 10 monclient: init 2024-03-27T21:40:52.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdeb0922640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:52.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdeb0922640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-03-27T21:40:52.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdeb0922640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdeac071a50 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:52.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdeb0922640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdeac13ef20 0x7fdeac0a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:52.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdeb0922640 10 monclient(hunting): picked mon.noname-b con 0x7fdeac13ef20 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:52.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdeb0922640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdeac141630 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:52.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdeb0922640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeac0a8e90 con 0x7fdeac071a50 2024-03-27T21:40:52.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdeb0922640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeac0aa040 con 0x7fdeac141630 2024-03-27T21:40:52.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdeb0922640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:52.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdeb0922640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac071e30 con 0x7fdeac13ef20 2024-03-27T21:40:52.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdeb0922640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:52.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdeb0922640 10 monclient(hunting): authenticate will time out at 893.364181s 2024-03-27T21:40:52.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdea9d9b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdeac13ef20 0x7fdeac0a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:52.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdeaa59c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdeac071a50 0x7fdeac13e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37052/0 (socket says 172.21.15.60:37052) 2024-03-27T21:40:52.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdeaa59c640 1 -- 172.21.15.60:0/1654891954 learned_addr learned my addr 172.21.15.60:0/1654891954 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:40:52.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdea9d9b640 10 monclient(hunting): get_auth_request con 0x7fdeac13ef20 auth_method 0 2024-03-27T21:40:52.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdea9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:52.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdea9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:52.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdea9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:52.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdea9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:52.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdea9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:52.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdea9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:52.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdea9d9b640 10 monclient(hunting): handle_auth_done global_id 4628 payload 274 2024-03-27T21:40:52.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdea9d9b640 1 -- 172.21.15.60:0/1654891954 >> v1:172.21.15.191:6789/0 conn(0x7fdeac141630 legacy=0x7fdeac141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:52.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdea9d9b640 1 -- 172.21.15.60:0/1654891954 >> v1:172.21.15.60:6789/0 conn(0x7fdeac071a50 legacy=0x7fdeac13e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:52.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdea9d9b640 10 monclient: _finish_hunting 0 2024-03-27T21:40:52.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdea9d9b640 1 monclient: found mon.noname-b 2024-03-27T21:40:52.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdea9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:52.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdea9d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:52.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.143+0000 7fdea9d9b640 1 -- 172.21.15.60:0/1654891954 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeac0aa2b0 con 0x7fdeac13ef20 2024-03-27T21:40:52.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.144+0000 7fdea9d9b640 1 --2- 172.21.15.60:0/1654891954 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdeac13ef20 0x7fdeac0a87e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdea00097d0 tx=0x7fdea002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b625c595fb12e2c server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:52.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.144+0000 7fdea959a640 1 -- 172.21.15.60:0/1654891954 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea0031890 con 0x7fdeac13ef20 2024-03-27T21:40:52.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.144+0000 7fdea959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:52.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.144+0000 7fdea959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:52.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.144+0000 7fdea959a640 10 monclient: dump: 2024-03-27T21:40:52.135 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:52.135 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:52.135 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:52.135 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:52.136 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:52.136 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:52.136 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:52.136 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:52.136 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:52.136 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:52.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.144+0000 7fdea959a640 10 monclient: _finish_auth 0 2024-03-27T21:40:52.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.144+0000 7fdea959a640 10 monclient: _check_auth_tickets 2024-03-27T21:40:52.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.144+0000 7fdea959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:52.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.144+0000 7fdeb0922640 5 monclient: authenticate success, global_id 4628 2024-03-27T21:40:52.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.144+0000 7fdeb0922640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:52.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.144+0000 7fdea959a640 1 -- 172.21.15.60:0/1654891954 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea00319f0 con 0x7fdeac13ef20 2024-03-27T21:40:52.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.144+0000 7fdea959a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:52.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.144+0000 7fdeb0922640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:52.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.144+0000 7fdea959a640 1 -- 172.21.15.60:0/1654891954 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea0031d80 con 0x7fdeac13ef20 2024-03-27T21:40:52.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.144+0000 7fdeb0922640 4 set_mon_vals no callback set 2024-03-27T21:40:52.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.144+0000 7fdea959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:52.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.144+0000 7fdea959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:52.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.144+0000 7fdea959a640 10 monclient: dump: 2024-03-27T21:40:52.137 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:52.137 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:52.137 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:52.137 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:52.137 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:52.137 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:52.137 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:52.137 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:52.137 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:52.137 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:52.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.145+0000 7fdeb0922640 10 monclient: shutdown 2024-03-27T21:40:52.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.145+0000 7fdeb0922640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:52.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.145+0000 7fdeb0922640 1 -- 172.21.15.60:0/1654891954 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdeac13ef20 msgr2=0x7fdeac0a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:52.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.145+0000 7fdeb0922640 1 --2- 172.21.15.60:0/1654891954 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdeac13ef20 0x7fdeac0a87e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdea00097d0 tx=0x7fdea002b930 comp rx=0 tx=0).stop 2024-03-27T21:40:52.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.145+0000 7fdeb0922640 1 -- 172.21.15.60:0/1654891954 shutdown_connections 2024-03-27T21:40:52.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.145+0000 7fdeb0922640 1 --2- 172.21.15.60:0/1654891954 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdeac13ef20 0x7fdeac0a87e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:52.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.145+0000 7fdeb0922640 1 -- 172.21.15.60:0/1654891954 >> 172.21.15.60:0/1654891954 conn(0x7fdeac0b7300 msgr2=0x7fdeac0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:52.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.145+0000 7fdeb0922640 1 -- 172.21.15.60:0/1654891954 shutdown_connections 2024-03-27T21:40:52.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.145+0000 7fdeb0922640 1 -- 172.21.15.60:0/1654891954 wait complete. 2024-03-27T21:40:52.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.146+0000 7fdeb0922640 10 monclient: build_initial_monmap 2024-03-27T21:40:52.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.146+0000 7fdeb0922640 10 monclient: monmap: 2024-03-27T21:40:52.138 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:52.138 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:52.138 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:52.138 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:52.139 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:52.139 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:52.139 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:52.139 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:52.139 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:52.139 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:52.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.146+0000 7fdeb0922640 1 librados: starting msgr at 2024-03-27T21:40:52.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.146+0000 7fdeb0922640 1 librados: starting objecter 2024-03-27T21:40:52.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.146+0000 7fdeb0922640 1 Processor -- start 2024-03-27T21:40:52.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.146+0000 7fdeb0922640 1 -- start start 2024-03-27T21:40:52.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.146+0000 7fdeb0922640 1 librados: setting wanted keys 2024-03-27T21:40:52.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.146+0000 7fdeb0922640 1 librados: calling monclient init 2024-03-27T21:40:52.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.146+0000 7fdeb0922640 10 monclient: init 2024-03-27T21:40:52.141 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:40:52.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeb0922640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:52.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeb0922640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:40:52.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeb0922640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdeac071a50 0x7fdeac137430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:52.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeb0922640 10 monclient(hunting): picked mon.noname-b con 0x7fdeac071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:52.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeb0922640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdeac13ef20 0x7fdeac0aaae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:52.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeb0922640 10 monclient(hunting): picked mon.noname-a con 0x7fdeac13ef20 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:52.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeb0922640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdeac141630 0x7fdeac139310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:52.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeb0922640 10 monclient(hunting): picked mon.noname-c con 0x7fdeac141630 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:52.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeb0922640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:52.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeb0922640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac139c70 con 0x7fdeac13ef20 2024-03-27T21:40:52.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeb0922640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:52.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeb0922640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac137970 con 0x7fdeac071a50 2024-03-27T21:40:52.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeb0922640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:52.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeb0922640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac13a960 con 0x7fdeac141630 2024-03-27T21:40:52.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeb0922640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:52.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeb0922640 10 monclient(hunting): authenticate will time out at 893.368230s 2024-03-27T21:40:52.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeaad9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdeac141630 0x7fdeac139310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:52.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdea9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdeac13ef20 0x7fdeac0aaae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:52.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeaa59c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdeac071a50 0x7fdeac137430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:52.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdea9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdeac13ef20 0x7fdeac0aaae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47128/0 (socket says 172.21.15.60:47128) 2024-03-27T21:40:52.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdea9d9b640 1 -- 172.21.15.60:0/1168413398 learned_addr learned my addr 172.21.15.60:0/1168413398 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:52.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeaad9d640 10 monclient(hunting): get_auth_request con 0x7fdeac141630 auth_method 0 2024-03-27T21:40:52.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeaad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:52.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeaad9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:52.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeaad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:52.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdea9d9b640 10 monclient(hunting): get_auth_request con 0x7fdeac13ef20 auth_method 0 2024-03-27T21:40:52.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdea9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:52.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdea9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:52.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdea9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:52.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeaa59c640 10 monclient(hunting): get_auth_request con 0x7fdeac071a50 auth_method 0 2024-03-27T21:40:52.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeaa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:52.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeaa59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:52.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeaa59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:52.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdea9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:52.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdea9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:52.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdea9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:52.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeaa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:52.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeaa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:52.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeaa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:52.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeaad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:52.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeaad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:52.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdeaad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:52.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdea9d9b640 10 monclient(hunting): handle_auth_done global_id 4785 payload 824 2024-03-27T21:40:52.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdea9d9b640 1 -- 172.21.15.60:0/1168413398 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdeac141630 msgr2=0x7fdeac139310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:52.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdea9d9b640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdeac141630 0x7fdeac139310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:52.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdea9d9b640 1 -- 172.21.15.60:0/1168413398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdeac071a50 msgr2=0x7fdeac137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:52.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdea9d9b640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdeac071a50 0x7fdeac137430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:52.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdea9d9b640 10 monclient: _finish_hunting 0 2024-03-27T21:40:52.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdea9d9b640 1 monclient: found mon.noname-a 2024-03-27T21:40:52.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdea9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:52.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdea9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:52.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.147+0000 7fdea9d9b640 1 -- 172.21.15.60:0/1168413398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea0009400 con 0x7fdeac13ef20 2024-03-27T21:40:52.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fdea9d9b640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdeac13ef20 0x7fdeac0aaae0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdea002bf40 tx=0x7fdea00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:52.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fde9affd640 10 client.?.objecter ms_handle_connect 0x7fdeac13ef20 2024-03-27T21:40:52.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fde9affd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:52.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fde9affd640 1 -- 172.21.15.60:0/1168413398 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea0031830 con 0x7fdeac13ef20 2024-03-27T21:40:52.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fde9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:52.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fde9affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:52.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fde9affd640 10 monclient: dump: 2024-03-27T21:40:52.149 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:52.149 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:52.150 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:52.150 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:52.150 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:52.150 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:52.150 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:52.150 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:52.150 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:52.150 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:52.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fde9affd640 10 monclient: _finish_auth 0 2024-03-27T21:40:52.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fde9affd640 10 monclient: _check_auth_tickets 2024-03-27T21:40:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fde9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fde9affd640 1 -- 172.21.15.60:0/1168413398 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea0046450 con 0x7fdeac13ef20 2024-03-27T21:40:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fde9affd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fde9affd640 1 -- 172.21.15.60:0/1168413398 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea0046800 con 0x7fdeac13ef20 2024-03-27T21:40:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fde9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fde9affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fde9affd640 10 monclient: dump: 2024-03-27T21:40:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:52.152 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fdeb0922640 5 monclient: authenticate success, global_id 4785 2024-03-27T21:40:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fdea8d99640 4 set_mon_vals no callback set 2024-03-27T21:40:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fdeb0922640 10 monclient: _renew_subs 2024-03-27T21:40:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fdeb0922640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fdeb0922640 1 -- 172.21.15.60:0/1168413398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeac13ac40 con 0x7fdeac13ef20 2024-03-27T21:40:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fdeb0922640 10 client.4785.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fdeb0922640 10 monclient: _renew_subs 2024-03-27T21:40:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fdeb0922640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fdeb0922640 1 -- 172.21.15.60:0/1168413398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeac1388c0 con 0x7fdeac13ef20 2024-03-27T21:40:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.148+0000 7fdeb0922640 1 librados: init done 2024-03-27T21:40:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.149+0000 7fdeb0922640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.149+0000 7fdeb0922640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.149+0000 7fdeb0922640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.149+0000 7fdeb0922640 1 -- 172.21.15.60:0/1168413398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fdeac139de0 con 0x7fdeac13ef20 2024-03-27T21:40:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.150+0000 7fde9affd640 1 -- 172.21.15.60:0/1168413398 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fdea00303e0 con 0x7fdeac13ef20 2024-03-27T21:40:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.150+0000 7fde9affd640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fde8c06c590 0x7fde8c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fde9affd640 1 -- 172.21.15.60:0/1168413398 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fdea00abac0 con 0x7fdeac13ef20 2024-03-27T21:40:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fde9affd640 10 client.4785.objecter ms_dispatch 0x7fdeac000b90 osd_map(46..46 src has 1..46) v4 2024-03-27T21:40:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fde9affd640 3 client.4785.objecter handle_osd_map got epochs [46,46] > 0 2024-03-27T21:40:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fde9affd640 3 client.4785.objecter handle_osd_map decoding full epoch 46 2024-03-27T21:40:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fdeaa59c640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fde8c06c590 0x7fde8c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fde9affd640 20 client.4785.objecter dump_active .. 0 homeless 2024-03-27T21:40:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fdeaa59c640 10 monclient: get_auth_request con 0x7fde8c06c590 auth_method 0 2024-03-27T21:40:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fde9affd640 1 -- 172.21.15.60:0/1168413398 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v46) v1 ==== 54+0+16 (secure 0 0 0) 0x7fdea00abe60 con 0x7fdeac13ef20 2024-03-27T21:40:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fde9affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fde9affd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:52.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fdeb0922640 10 client.4785.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:52.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign -1 1 2024-03-27T21:40:52.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fdeb0922640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fdeac139850 0x7fdeac1da4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:52.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac142210 osd=0 3 2024-03-27T21:40:52.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac142210 osd=0 3 2024-03-27T21:40:52.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac142210 osd=0 3 2024-03-27T21:40:52.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_remove -1 1 2024-03-27T21:40:52.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fdeb0922640 20 client.4785.objecter get_session s=0x7fdeac142210 osd=0 3 2024-03-27T21:40:52.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign 0 1 2024-03-27T21:40:52.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac142210 osd=0 4 2024-03-27T21:40:52.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fdeb0922640 10 client.4785.objecter _send_command 1 2024-03-27T21:40:52.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fdeb0922640 1 -- 172.21.15.60:0/1168413398 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdeac069470 con 0x7fdeac139850 2024-03-27T21:40:52.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fdeaad9d640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fdeac139850 0x7fdeac1da4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:52.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fdeaad9d640 10 monclient: get_auth_request con 0x7fdeac139850 auth_method 0 2024-03-27T21:40:52.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fdeaa59c640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fde8c06c590 0x7fde8c06ea50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fde9c005880 tx=0x7fde9c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:52.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fde9affd640 10 client.4785.objecter ms_handle_connect 0x7fde8c06c590 2024-03-27T21:40:52.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fdeaad9d640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fdeac139850 0x7fdeac1da4b0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:52.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.151+0000 7fde9affd640 10 client.4785.objecter ms_handle_connect 0x7fdeac139850 2024-03-27T21:40:52.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.152+0000 7fde9affd640 1 -- 172.21.15.60:0/1168413398 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fdeac069470 con 0x7fdeac139850 2024-03-27T21:40:52.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.152+0000 7fde9affd640 10 client.4785.objecter ms_dispatch 0x7fdeac000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:52.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.152+0000 7fde9affd640 10 client.4785.objecter _finish_command 1 = system:0 2024-03-27T21:40:52.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.152+0000 7fde9affd640 20 client.4785.objecter put_session s=0x7fdeac142210 osd=0 4 2024-03-27T21:40:52.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.152+0000 7fde9affd640 15 client.4785.objecter _session_command_op_remove 0 1 2024-03-27T21:40:52.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.168+0000 7fdeb0922640 10 client.4785.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:52.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.168+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign -1 2 2024-03-27T21:40:52.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.168+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac142210 osd=0 3 2024-03-27T21:40:52.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.168+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac142210 osd=0 3 2024-03-27T21:40:52.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.168+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac142210 osd=0 3 2024-03-27T21:40:52.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.168+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_remove -1 2 2024-03-27T21:40:52.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.168+0000 7fdeb0922640 20 client.4785.objecter get_session s=0x7fdeac142210 osd=0 3 2024-03-27T21:40:52.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.168+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign 0 2 2024-03-27T21:40:52.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.168+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac142210 osd=0 4 2024-03-27T21:40:52.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.168+0000 7fdeb0922640 10 client.4785.objecter _send_command 2 2024-03-27T21:40:52.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.168+0000 7fdeb0922640 1 -- 172.21.15.60:0/1168413398 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdeac0a9260 con 0x7fdeac139850 2024-03-27T21:40:52.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.171+0000 7fde9affd640 1 -- 172.21.15.60:0/1168413398 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdeac0a9260 con 0x7fdeac139850 2024-03-27T21:40:52.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.171+0000 7fde9affd640 10 client.4785.objecter ms_dispatch 0x7fdeac000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:52.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.171+0000 7fde9affd640 10 client.4785.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:52.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.171+0000 7fde9affd640 20 client.4785.objecter put_session s=0x7fdeac142210 osd=0 4 2024-03-27T21:40:52.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.171+0000 7fde9affd640 15 client.4785.objecter _session_command_op_remove 0 2 2024-03-27T21:40:52.158 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:52.159 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:52.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.171+0000 7fdeb0922640 10 client.4785.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:52.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.171+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign -1 3 2024-03-27T21:40:52.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.171+0000 7fdeb0922640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fdeac1dab60 0x7fdeac136830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:52.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.171+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac1da9f0 osd=1 3 2024-03-27T21:40:52.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.171+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac1da9f0 osd=1 3 2024-03-27T21:40:52.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.171+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac1da9f0 osd=1 3 2024-03-27T21:40:52.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.171+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_remove -1 3 2024-03-27T21:40:52.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.171+0000 7fdeb0922640 20 client.4785.objecter get_session s=0x7fdeac1da9f0 osd=1 3 2024-03-27T21:40:52.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.171+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign 1 3 2024-03-27T21:40:52.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.171+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac1da9f0 osd=1 4 2024-03-27T21:40:52.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.171+0000 7fdeb0922640 10 client.4785.objecter _send_command 3 2024-03-27T21:40:52.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.171+0000 7fdeb0922640 1 -- 172.21.15.60:0/1168413398 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdeac136d70 con 0x7fdeac1dab60 2024-03-27T21:40:52.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.171+0000 7fdeaa59c640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fdeac1dab60 0x7fdeac136830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:52.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.171+0000 7fdeaa59c640 10 monclient: get_auth_request con 0x7fdeac1dab60 auth_method 0 2024-03-27T21:40:52.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.171+0000 7fde9affd640 10 client.4785.objecter ms_handle_connect 0x7fdeac1dab60 2024-03-27T21:40:52.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.171+0000 7fdeaa59c640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fdeac1dab60 0x7fdeac136830 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:52.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.172+0000 7fde9affd640 1 -- 172.21.15.60:0/1168413398 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fdeac136d70 con 0x7fdeac1dab60 2024-03-27T21:40:52.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.172+0000 7fde9affd640 10 client.4785.objecter ms_dispatch 0x7fdeac000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:52.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.172+0000 7fde9affd640 10 client.4785.objecter _finish_command 3 = system:0 2024-03-27T21:40:52.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.172+0000 7fde9affd640 20 client.4785.objecter put_session s=0x7fdeac1da9f0 osd=1 4 2024-03-27T21:40:52.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.172+0000 7fde9affd640 15 client.4785.objecter _session_command_op_remove 1 3 2024-03-27T21:40:52.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.189+0000 7fdeb0922640 10 client.4785.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:52.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.189+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign -1 4 2024-03-27T21:40:52.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.189+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac1da9f0 osd=1 3 2024-03-27T21:40:52.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.189+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac1da9f0 osd=1 3 2024-03-27T21:40:52.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.189+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac1da9f0 osd=1 3 2024-03-27T21:40:52.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.189+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_remove -1 4 2024-03-27T21:40:52.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.189+0000 7fdeb0922640 20 client.4785.objecter get_session s=0x7fdeac1da9f0 osd=1 3 2024-03-27T21:40:52.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.189+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign 1 4 2024-03-27T21:40:52.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.189+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac1da9f0 osd=1 4 2024-03-27T21:40:52.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.189+0000 7fdeb0922640 10 client.4785.objecter _send_command 4 2024-03-27T21:40:52.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.189+0000 7fdeb0922640 1 -- 172.21.15.60:0/1168413398 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdeac137180 con 0x7fdeac1dab60 2024-03-27T21:40:52.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fde9affd640 1 -- 172.21.15.60:0/1168413398 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdeac137180 con 0x7fdeac1dab60 2024-03-27T21:40:52.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fde9affd640 10 client.4785.objecter ms_dispatch 0x7fdeac000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:52.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fde9affd640 10 client.4785.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:52.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fde9affd640 20 client.4785.objecter put_session s=0x7fdeac1da9f0 osd=1 4 2024-03-27T21:40:52.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fde9affd640 15 client.4785.objecter _session_command_op_remove 1 4 2024-03-27T21:40:52.179 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:52.180 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:52.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdeb0922640 10 client.4785.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:52.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign -1 5 2024-03-27T21:40:52.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdeb0922640 20 client.4785.objecter _get_session osd=-1 returning homeless 2024-03-27T21:40:52.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdeb0922640 10 client.4785.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:52.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdeb0922640 10 monclient: _renew_subs 2024-03-27T21:40:52.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdeb0922640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:52.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdeb0922640 1 -- 172.21.15.60:0/1168413398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=47}) v3 -- 0x7fdeac13d550 con 0x7fdeac13ef20 2024-03-27T21:40:52.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdeb0922640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:52.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdeb0922640 1 -- 172.21.15.60:0/1168413398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdeac13d830 con 0x7fdeac13ef20 2024-03-27T21:40:52.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fde9affd640 1 -- 172.21.15.60:0/1168413398 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdea0076fb0 con 0x7fdeac13ef20 2024-03-27T21:40:52.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fde9affd640 10 monclient: handle_get_version_reply finishing 1 version 46 2024-03-27T21:40:52.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdea8d99640 10 client.4785.objecter _check_command_map_dne tid 5 current 46 map_dne_bound 46 2024-03-27T21:40:52.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdea8d99640 10 client.4785.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:40:52.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdea8d99640 15 client.4785.objecter _session_command_op_remove -1 5 2024-03-27T21:40:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:40:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdeb0922640 10 client.4785.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign -1 6 2024-03-27T21:40:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdeb0922640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fdeac02b170 0x7fdeac02b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac133930 osd=3 3 2024-03-27T21:40:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac133930 osd=3 3 2024-03-27T21:40:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac133930 osd=3 3 2024-03-27T21:40:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_remove -1 6 2024-03-27T21:40:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdeb0922640 20 client.4785.objecter get_session s=0x7fdeac133930 osd=3 3 2024-03-27T21:40:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign 3 6 2024-03-27T21:40:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac133930 osd=3 4 2024-03-27T21:40:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdeb0922640 10 client.4785.objecter _send_command 6 2024-03-27T21:40:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.192+0000 7fdeb0922640 1 -- 172.21.15.60:0/1168413398 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdeac02ba90 con 0x7fdeac02b170 2024-03-27T21:40:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.193+0000 7fdea9d9b640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fdeac02b170 0x7fdeac02b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:52.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.193+0000 7fdea9d9b640 10 monclient: get_auth_request con 0x7fdeac02b170 auth_method 0 2024-03-27T21:40:52.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.193+0000 7fdea9d9b640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fdeac02b170 0x7fdeac02b550 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:52.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.193+0000 7fde9affd640 10 client.4785.objecter ms_handle_connect 0x7fdeac02b170 2024-03-27T21:40:52.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.194+0000 7fde9affd640 1 -- 172.21.15.60:0/1168413398 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fdeac02ba90 con 0x7fdeac02b170 2024-03-27T21:40:52.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.194+0000 7fde9affd640 10 client.4785.objecter ms_dispatch 0x7fdeac000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:40:52.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.194+0000 7fde9affd640 10 client.4785.objecter _finish_command 6 = system:0 2024-03-27T21:40:52.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.194+0000 7fde9affd640 20 client.4785.objecter put_session s=0x7fdeac133930 osd=3 4 2024-03-27T21:40:52.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.194+0000 7fde9affd640 15 client.4785.objecter _session_command_op_remove 3 6 2024-03-27T21:40:52.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.208+0000 7fdeb0922640 10 client.4785.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:52.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.208+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign -1 7 2024-03-27T21:40:52.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.208+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac133930 osd=3 3 2024-03-27T21:40:52.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.208+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac133930 osd=3 3 2024-03-27T21:40:52.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.208+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac133930 osd=3 3 2024-03-27T21:40:52.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.208+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_remove -1 7 2024-03-27T21:40:52.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.208+0000 7fdeb0922640 20 client.4785.objecter get_session s=0x7fdeac133930 osd=3 3 2024-03-27T21:40:52.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.208+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign 3 7 2024-03-27T21:40:52.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.208+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac133930 osd=3 4 2024-03-27T21:40:52.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.208+0000 7fdeb0922640 10 client.4785.objecter _send_command 7 2024-03-27T21:40:52.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.208+0000 7fdeb0922640 1 -- 172.21.15.60:0/1168413398 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdeac02c450 con 0x7fdeac02b170 2024-03-27T21:40:52.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.211+0000 7fde9affd640 1 -- 172.21.15.60:0/1168413398 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdeac02c450 con 0x7fdeac02b170 2024-03-27T21:40:52.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.211+0000 7fde9affd640 10 client.4785.objecter ms_dispatch 0x7fdeac000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:52.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.211+0000 7fde9affd640 10 client.4785.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:52.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.211+0000 7fde9affd640 20 client.4785.objecter put_session s=0x7fdeac133930 osd=3 4 2024-03-27T21:40:52.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.211+0000 7fde9affd640 15 client.4785.objecter _session_command_op_remove 3 7 2024-03-27T21:40:52.198 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:52.199 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:52.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.211+0000 7fdeb0922640 10 client.4785.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:52.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.211+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign -1 8 2024-03-27T21:40:52.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.211+0000 7fdeb0922640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fdeac133520 0x7fdeac1e9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:52.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.211+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac1333b0 osd=4 3 2024-03-27T21:40:52.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.211+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac1333b0 osd=4 3 2024-03-27T21:40:52.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.211+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac1333b0 osd=4 3 2024-03-27T21:40:52.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.211+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_remove -1 8 2024-03-27T21:40:52.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.211+0000 7fdeb0922640 20 client.4785.objecter get_session s=0x7fdeac1333b0 osd=4 3 2024-03-27T21:40:52.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.211+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign 4 8 2024-03-27T21:40:52.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.211+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac1333b0 osd=4 4 2024-03-27T21:40:52.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.211+0000 7fdeb0922640 10 client.4785.objecter _send_command 8 2024-03-27T21:40:52.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.211+0000 7fdeb0922640 1 -- 172.21.15.60:0/1168413398 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdeac1ec510 con 0x7fdeac133520 2024-03-27T21:40:52.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.211+0000 7fdeaad9d640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fdeac133520 0x7fdeac1e9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:52.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.211+0000 7fdeaad9d640 10 monclient: get_auth_request con 0x7fdeac133520 auth_method 0 2024-03-27T21:40:52.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.212+0000 7fde9affd640 10 client.4785.objecter ms_handle_connect 0x7fdeac133520 2024-03-27T21:40:52.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.212+0000 7fdeaad9d640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fdeac133520 0x7fdeac1e9740 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:52.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.213+0000 7fde9affd640 1 -- 172.21.15.60:0/1168413398 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fdeac1ec510 con 0x7fdeac133520 2024-03-27T21:40:52.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.213+0000 7fde9affd640 10 client.4785.objecter ms_dispatch 0x7fdeac000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:40:52.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.213+0000 7fde9affd640 10 client.4785.objecter _finish_command 8 = system:0 2024-03-27T21:40:52.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.213+0000 7fde9affd640 20 client.4785.objecter put_session s=0x7fdeac1333b0 osd=4 4 2024-03-27T21:40:52.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.213+0000 7fde9affd640 15 client.4785.objecter _session_command_op_remove 4 8 2024-03-27T21:40:52.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.228+0000 7fdeb0922640 10 client.4785.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:52.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.228+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign -1 9 2024-03-27T21:40:52.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.228+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac1333b0 osd=4 3 2024-03-27T21:40:52.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.228+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac1333b0 osd=4 3 2024-03-27T21:40:52.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.228+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac1333b0 osd=4 3 2024-03-27T21:40:52.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.228+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_remove -1 9 2024-03-27T21:40:52.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.228+0000 7fdeb0922640 20 client.4785.objecter get_session s=0x7fdeac1333b0 osd=4 3 2024-03-27T21:40:52.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.228+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign 4 9 2024-03-27T21:40:52.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.228+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac1333b0 osd=4 4 2024-03-27T21:40:52.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.228+0000 7fdeb0922640 10 client.4785.objecter _send_command 9 2024-03-27T21:40:52.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.228+0000 7fdeb0922640 1 -- 172.21.15.60:0/1168413398 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdeac02bed0 con 0x7fdeac133520 2024-03-27T21:40:52.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.229+0000 7fde9affd640 1 -- 172.21.15.60:0/1168413398 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdeac02bed0 con 0x7fdeac133520 2024-03-27T21:40:52.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.229+0000 7fde9affd640 10 client.4785.objecter ms_dispatch 0x7fdeac000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:52.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.229+0000 7fde9affd640 10 client.4785.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:52.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.229+0000 7fde9affd640 20 client.4785.objecter put_session s=0x7fdeac1333b0 osd=4 4 2024-03-27T21:40:52.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.229+0000 7fde9affd640 15 client.4785.objecter _session_command_op_remove 4 9 2024-03-27T21:40:52.216 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:52.217 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:52.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.230+0000 7fdeb0922640 10 client.4785.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:52.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.230+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign -1 10 2024-03-27T21:40:52.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.230+0000 7fdeb0922640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fdeac02c040 0x7fdeac1eae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.230+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac02bed0 osd=5 3 2024-03-27T21:40:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.230+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac02bed0 osd=5 3 2024-03-27T21:40:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.230+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac02bed0 osd=5 3 2024-03-27T21:40:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.230+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_remove -1 10 2024-03-27T21:40:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.230+0000 7fdeb0922640 20 client.4785.objecter get_session s=0x7fdeac02bed0 osd=5 3 2024-03-27T21:40:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.230+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign 5 10 2024-03-27T21:40:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.230+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac02bed0 osd=5 4 2024-03-27T21:40:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.230+0000 7fdeb0922640 10 client.4785.objecter _send_command 10 2024-03-27T21:40:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.230+0000 7fdeb0922640 1 -- 172.21.15.60:0/1168413398 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdeac066760 con 0x7fdeac02c040 2024-03-27T21:40:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.230+0000 7fdeaa59c640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fdeac02c040 0x7fdeac1eae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.230+0000 7fdeaa59c640 10 monclient: get_auth_request con 0x7fdeac02c040 auth_method 0 2024-03-27T21:40:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.230+0000 7fde9affd640 10 client.4785.objecter ms_handle_connect 0x7fdeac02c040 2024-03-27T21:40:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.230+0000 7fdeaa59c640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fdeac02c040 0x7fdeac1eae00 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:52.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.231+0000 7fde9affd640 1 -- 172.21.15.60:0/1168413398 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fdeac066760 con 0x7fdeac02c040 2024-03-27T21:40:52.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.231+0000 7fde9affd640 10 client.4785.objecter ms_dispatch 0x7fdeac000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:40:52.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.231+0000 7fde9affd640 10 client.4785.objecter _finish_command 10 = system:0 2024-03-27T21:40:52.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.231+0000 7fde9affd640 20 client.4785.objecter put_session s=0x7fdeac02bed0 osd=5 4 2024-03-27T21:40:52.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.231+0000 7fde9affd640 15 client.4785.objecter _session_command_op_remove 5 10 2024-03-27T21:40:52.222 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:52.236+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:52.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.246+0000 7fdeb0922640 10 client.4785.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:52.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.246+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign -1 11 2024-03-27T21:40:52.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.246+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac02bed0 osd=5 3 2024-03-27T21:40:52.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.246+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac02bed0 osd=5 3 2024-03-27T21:40:52.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.246+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac02bed0 osd=5 3 2024-03-27T21:40:52.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.246+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_remove -1 11 2024-03-27T21:40:52.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.246+0000 7fdeb0922640 20 client.4785.objecter get_session s=0x7fdeac02bed0 osd=5 3 2024-03-27T21:40:52.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.246+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign 5 11 2024-03-27T21:40:52.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.246+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac02bed0 osd=5 4 2024-03-27T21:40:52.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.246+0000 7fdeb0922640 10 client.4785.objecter _send_command 11 2024-03-27T21:40:52.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.246+0000 7fdeb0922640 1 -- 172.21.15.60:0/1168413398 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdeac066760 con 0x7fdeac02c040 2024-03-27T21:40:52.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.248+0000 7fde9affd640 1 -- 172.21.15.60:0/1168413398 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdeac066760 con 0x7fdeac02c040 2024-03-27T21:40:52.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.248+0000 7fde9affd640 10 client.4785.objecter ms_dispatch 0x7fdeac000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:52.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.248+0000 7fde9affd640 10 client.4785.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:52.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.248+0000 7fde9affd640 20 client.4785.objecter put_session s=0x7fdeac02bed0 osd=5 4 2024-03-27T21:40:52.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.248+0000 7fde9affd640 15 client.4785.objecter _session_command_op_remove 5 11 2024-03-27T21:40:52.235 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:52.236 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:52.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.248+0000 7fdeb0922640 10 client.4785.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:52.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.248+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign -1 12 2024-03-27T21:40:52.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.248+0000 7fdeb0922640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fdeac1eb850 0x7fdeac1ebcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:52.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.248+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac066760 osd=6 3 2024-03-27T21:40:52.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.248+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac066760 osd=6 3 2024-03-27T21:40:52.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.248+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac066760 osd=6 3 2024-03-27T21:40:52.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.248+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_remove -1 12 2024-03-27T21:40:52.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.248+0000 7fdeb0922640 20 client.4785.objecter get_session s=0x7fdeac066760 osd=6 3 2024-03-27T21:40:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.248+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign 6 12 2024-03-27T21:40:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.248+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac066760 osd=6 4 2024-03-27T21:40:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.248+0000 7fdeb0922640 10 client.4785.objecter _send_command 12 2024-03-27T21:40:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.248+0000 7fdeb0922640 1 -- 172.21.15.60:0/1168413398 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdeac1ec1f0 con 0x7fdeac1eb850 2024-03-27T21:40:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.249+0000 7fdea9d9b640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fdeac1eb850 0x7fdeac1ebcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.249+0000 7fdea9d9b640 10 monclient: get_auth_request con 0x7fdeac1eb850 auth_method 0 2024-03-27T21:40:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.249+0000 7fde9affd640 10 client.4785.objecter ms_handle_connect 0x7fdeac1eb850 2024-03-27T21:40:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.249+0000 7fdea9d9b640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fdeac1eb850 0x7fdeac1ebcb0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.250+0000 7fde9affd640 1 -- 172.21.15.60:0/1168413398 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fdeac1ec1f0 con 0x7fdeac1eb850 2024-03-27T21:40:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.250+0000 7fde9affd640 10 client.4785.objecter ms_dispatch 0x7fdeac000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:40:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.250+0000 7fde9affd640 10 client.4785.objecter _finish_command 12 = system:0 2024-03-27T21:40:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.250+0000 7fde9affd640 20 client.4785.objecter put_session s=0x7fdeac066760 osd=6 4 2024-03-27T21:40:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.250+0000 7fde9affd640 15 client.4785.objecter _session_command_op_remove 6 12 2024-03-27T21:40:52.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.264+0000 7fdeb0922640 10 client.4785.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:52.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.264+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign -1 13 2024-03-27T21:40:52.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.264+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac066760 osd=6 3 2024-03-27T21:40:52.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.264+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac066760 osd=6 3 2024-03-27T21:40:52.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.264+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac066760 osd=6 3 2024-03-27T21:40:52.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.264+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_remove -1 13 2024-03-27T21:40:52.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.264+0000 7fdeb0922640 20 client.4785.objecter get_session s=0x7fdeac066760 osd=6 3 2024-03-27T21:40:52.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.264+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign 6 13 2024-03-27T21:40:52.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.264+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac066760 osd=6 4 2024-03-27T21:40:52.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.264+0000 7fdeb0922640 10 client.4785.objecter _send_command 13 2024-03-27T21:40:52.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.264+0000 7fdeb0922640 1 -- 172.21.15.60:0/1168413398 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdeac1ec1f0 con 0x7fdeac1eb850 2024-03-27T21:40:52.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.266+0000 7fde9affd640 1 -- 172.21.15.60:0/1168413398 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdeac1ec1f0 con 0x7fdeac1eb850 2024-03-27T21:40:52.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.266+0000 7fde9affd640 10 client.4785.objecter ms_dispatch 0x7fdeac000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:52.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.266+0000 7fde9affd640 10 client.4785.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:52.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.266+0000 7fde9affd640 20 client.4785.objecter put_session s=0x7fdeac066760 osd=6 4 2024-03-27T21:40:52.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.266+0000 7fde9affd640 15 client.4785.objecter _session_command_op_remove 6 13 2024-03-27T21:40:52.252 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:52.253 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:52.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.266+0000 7fdeb0922640 10 client.4785.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:52.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.266+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign -1 14 2024-03-27T21:40:52.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.266+0000 7fdeb0922640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fdeac1f0bb0 0x7fdeac1f2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:52.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.266+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac1ec1f0 osd=7 3 2024-03-27T21:40:52.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.266+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac1ec1f0 osd=7 3 2024-03-27T21:40:52.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.266+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac1ec1f0 osd=7 3 2024-03-27T21:40:52.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.266+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_remove -1 14 2024-03-27T21:40:52.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.266+0000 7fdeb0922640 20 client.4785.objecter get_session s=0x7fdeac1ec1f0 osd=7 3 2024-03-27T21:40:52.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.266+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign 7 14 2024-03-27T21:40:52.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.266+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac1ec1f0 osd=7 4 2024-03-27T21:40:52.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.266+0000 7fdeb0922640 10 client.4785.objecter _send_command 14 2024-03-27T21:40:52.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.266+0000 7fdeb0922640 1 -- 172.21.15.60:0/1168413398 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdeac066990 con 0x7fdeac1f0bb0 2024-03-27T21:40:52.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.266+0000 7fdeaad9d640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fdeac1f0bb0 0x7fdeac1f2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:52.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.266+0000 7fdeaad9d640 10 monclient: get_auth_request con 0x7fdeac1f0bb0 auth_method 0 2024-03-27T21:40:52.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.267+0000 7fde9affd640 10 client.4785.objecter ms_handle_connect 0x7fdeac1f0bb0 2024-03-27T21:40:52.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.267+0000 7fdeaad9d640 1 --2- 172.21.15.60:0/1168413398 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fdeac1f0bb0 0x7fdeac1f2fa0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:52.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.267+0000 7fde9affd640 1 -- 172.21.15.60:0/1168413398 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fdeac066990 con 0x7fdeac1f0bb0 2024-03-27T21:40:52.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.267+0000 7fde9affd640 10 client.4785.objecter ms_dispatch 0x7fdeac000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:40:52.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.267+0000 7fde9affd640 10 client.4785.objecter _finish_command 14 = system:0 2024-03-27T21:40:52.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.267+0000 7fde9affd640 20 client.4785.objecter put_session s=0x7fdeac1ec1f0 osd=7 4 2024-03-27T21:40:52.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.267+0000 7fde9affd640 15 client.4785.objecter _session_command_op_remove 7 14 2024-03-27T21:40:52.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.282+0000 7fdeb0922640 10 client.4785.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:52.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.282+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign -1 15 2024-03-27T21:40:52.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.282+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac1ec1f0 osd=7 3 2024-03-27T21:40:52.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.282+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac1ec1f0 osd=7 3 2024-03-27T21:40:52.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.282+0000 7fdeb0922640 20 client.4785.objecter _get_session s=0x7fdeac1ec1f0 osd=7 3 2024-03-27T21:40:52.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.282+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_remove -1 15 2024-03-27T21:40:52.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.282+0000 7fdeb0922640 20 client.4785.objecter get_session s=0x7fdeac1ec1f0 osd=7 3 2024-03-27T21:40:52.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.282+0000 7fdeb0922640 15 client.4785.objecter _session_command_op_assign 7 15 2024-03-27T21:40:52.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.282+0000 7fdeb0922640 20 client.4785.objecter put_session s=0x7fdeac1ec1f0 osd=7 4 2024-03-27T21:40:52.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.282+0000 7fdeb0922640 10 client.4785.objecter _send_command 15 2024-03-27T21:40:52.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.282+0000 7fdeb0922640 1 -- 172.21.15.60:0/1168413398 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdeac066990 con 0x7fdeac1f0bb0 2024-03-27T21:40:52.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.284+0000 7fde9affd640 1 -- 172.21.15.60:0/1168413398 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fdeac066990 con 0x7fdeac1f0bb0 2024-03-27T21:40:52.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.284+0000 7fde9affd640 10 client.4785.objecter ms_dispatch 0x7fdeac000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:52.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.284+0000 7fde9affd640 10 client.4785.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:52.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.284+0000 7fde9affd640 20 client.4785.objecter put_session s=0x7fdeac1ec1f0 osd=7 4 2024-03-27T21:40:52.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.284+0000 7fde9affd640 15 client.4785.objecter _session_command_op_remove 7 15 2024-03-27T21:40:52.270 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:52.271 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:52.273 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:40:52.274 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:40:52.290 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:52.290 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:52.290 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:52.290 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:52.290 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:52.298 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:40:52.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.329+0000 7f8bfaee0640 10 start_mon_command cmd=[{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:40:52.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.329+0000 7f8bfaee0640 10 monclient: _send_command 2 [{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:40:52.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.329+0000 7f8bfaee0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:52.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.329+0000 7f8bfaee0640 1 -- 172.21.15.60:0/2605805512 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f8bf41411f0 con 0x7f8bf413b020 2024-03-27T21:40:52.342 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:52.356+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:52.406 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:52.406 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:52.406 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:52.407 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-03-27T21:40:52.407 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.407 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:52.407 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:52.407 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 72.986799679000001, 2024-03-27T21:40:52.407 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43537978700000002, 2024-03-27T21:40:52.407 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:52.407 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.407 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:52.407 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:52.407 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:52.407 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-03-27T21:40:52.407 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.408 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:52.408 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.408 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:52.408 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:52.408 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.408 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.408 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.408 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:52.408 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:52.408 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.408 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.408 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.409 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:52.409 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432019+0000", 2024-03-27T21:40:52.409 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-03-27T21:40:52.409 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.409 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.409 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:52.409 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432027+0000", 2024-03-27T21:40:52.409 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1200000000000002e-06 2024-03-27T21:40:52.409 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.409 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.409 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:52.409 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432039+0000", 2024-03-27T21:40:52.409 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1123e-05 2024-03-27T21:40:52.409 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.410 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.410 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.410 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432051+0000", 2024-03-27T21:40:52.410 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-03-27T21:40:52.410 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.410 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.410 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:52.410 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432157+0000", 2024-03-27T21:40:52.410 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106846 2024-03-27T21:40:52.410 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.410 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.410 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-03-27T21:40:52.410 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432176+0000", 2024-03-27T21:40:52.410 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8490000000000001e-05 2024-03-27T21:40:52.411 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.411 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.411 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:52.411 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866170+0000", 2024-03-27T21:40:52.411 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43399370100000001 2024-03-27T21:40:52.411 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.411 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.411 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:52.411 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866335+0000", 2024-03-27T21:40:52.411 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016559699999999999 2024-03-27T21:40:52.411 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.411 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.411 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:52.411 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866495+0000", 2024-03-27T21:40:52.411 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015944200000000001 2024-03-27T21:40:52.412 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.412 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.412 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:52.412 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867012+0000", 2024-03-27T21:40:52.412 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0005176 2024-03-27T21:40:52.412 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.412 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.412 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:52.412 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867334+0000", 2024-03-27T21:40:52.412 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032158600000000001 2024-03-27T21:40:52.412 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.412 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.412 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:52.412 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867374+0000", 2024-03-27T21:40:52.412 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0547000000000003e-05 2024-03-27T21:40:52.413 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.413 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.413 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:52.413 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867400+0000", 2024-03-27T21:40:52.413 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5958000000000001e-05 2024-03-27T21:40:52.413 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.413 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:52.413 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.413 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.413 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.413 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:52.413 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:52.413 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 72.489417227999994, 2024-03-27T21:40:52.413 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064560310000000001, 2024-03-27T21:40:52.413 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:52.414 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.414 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:52.414 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:52.414 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:52.414 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 70 2024-03-27T21:40:52.414 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.414 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:52.414 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.414 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:52.414 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:52.414 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.414 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.414 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.415 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:52.415 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:52.415 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.415 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.415 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.415 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:52.415 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929402+0000", 2024-03-27T21:40:52.415 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.415 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.415 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.415 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:52.415 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929407+0000", 2024-03-27T21:40:52.415 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-06 2024-03-27T21:40:52.415 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.416 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.416 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:52.416 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929414+0000", 2024-03-27T21:40:52.416 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5689999999999999e-06 2024-03-27T21:40:52.416 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.416 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.416 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.416 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929419+0000", 2024-03-27T21:40:52.416 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-03-27T21:40:52.416 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.416 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.416 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:52.416 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930245+0000", 2024-03-27T21:40:52.416 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082521500000000002 2024-03-27T21:40:52.416 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.417 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.417 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:52.417 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930287+0000", 2024-03-27T21:40:52.417 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2305000000000002e-05 2024-03-27T21:40:52.417 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.417 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.417 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:52.417 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930367+0000", 2024-03-27T21:40:52.417 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9589e-05 2024-03-27T21:40:52.417 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.417 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.417 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:52.417 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.931214+0000", 2024-03-27T21:40:52.417 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000847827 2024-03-27T21:40:52.417 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.418 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.418 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:52.418 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935807+0000", 2024-03-27T21:40:52.418 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045923780000000003 2024-03-27T21:40:52.418 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.418 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.418 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:52.418 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935840+0000", 2024-03-27T21:40:52.418 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3501999999999999e-05 2024-03-27T21:40:52.418 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.418 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.418 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:52.418 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935859+0000", 2024-03-27T21:40:52.419 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8845e-05 2024-03-27T21:40:52.419 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.419 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:52.419 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.419 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.419 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.419 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:52.419 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:52.419 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 72.482157637, 2024-03-27T21:40:52.419 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071695429999999996, 2024-03-27T21:40:52.419 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:52.419 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.419 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:52.420 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:52.420 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:52.420 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-03-27T21:40:52.420 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.420 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:52.420 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.420 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:52.420 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:52.420 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.420 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.420 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.420 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:52.420 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:52.421 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.421 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.421 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.421 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:52.421 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936662+0000", 2024-03-27T21:40:52.421 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.421 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.421 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.421 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:52.421 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936737+0000", 2024-03-27T21:40:52.421 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5303999999999996e-05 2024-03-27T21:40:52.421 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.421 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.421 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:52.421 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936746+0000", 2024-03-27T21:40:52.422 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6340000000000007e-06 2024-03-27T21:40:52.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.422 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.422 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936753+0000", 2024-03-27T21:40:52.422 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-03-27T21:40:52.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.422 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:52.422 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937926+0000", 2024-03-27T21:40:52.422 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011731280000000001 2024-03-27T21:40:52.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.423 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:52.423 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937969+0000", 2024-03-27T21:40:52.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3285999999999999e-05 2024-03-27T21:40:52.423 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.423 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.423 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:52.423 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938040+0000", 2024-03-27T21:40:52.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0358000000000006e-05 2024-03-27T21:40:52.423 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.423 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.423 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:52.423 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939324+0000", 2024-03-27T21:40:52.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012844899999999999 2024-03-27T21:40:52.423 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.423 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.424 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:52.424 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943780+0000", 2024-03-27T21:40:52.424 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044555630000000001 2024-03-27T21:40:52.424 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.424 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.424 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:52.424 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943811+0000", 2024-03-27T21:40:52.424 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1415999999999999e-05 2024-03-27T21:40:52.424 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.424 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.424 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:52.424 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943832+0000", 2024-03-27T21:40:52.424 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0981000000000001e-05 2024-03-27T21:40:52.424 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.424 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:52.424 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.424 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.425 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:52.425 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:52.425 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 72.482059621000005, 2024-03-27T21:40:52.425 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071139740000000003, 2024-03-27T21:40:52.425 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:52.425 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.425 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:52.425 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:52.425 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:52.425 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-03-27T21:40:52.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.425 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:52.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:52.425 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:52.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:52.426 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:52.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:52.426 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936760+0000", 2024-03-27T21:40:52.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:52.426 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936765+0000", 2024-03-27T21:40:52.427 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6330000000000004e-06 2024-03-27T21:40:52.427 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.427 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.427 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:52.427 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936771+0000", 2024-03-27T21:40:52.427 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0560000000000001e-06 2024-03-27T21:40:52.427 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.427 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.427 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.427 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936776+0000", 2024-03-27T21:40:52.427 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1510000000000003e-06 2024-03-27T21:40:52.427 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.427 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.427 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:52.427 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938605+0000", 2024-03-27T21:40:52.428 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001829589 2024-03-27T21:40:52.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.428 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:52.428 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938657+0000", 2024-03-27T21:40:52.428 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1860000000000002e-05 2024-03-27T21:40:52.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.428 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:52.428 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938734+0000", 2024-03-27T21:40:52.428 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6639999999999998e-05 2024-03-27T21:40:52.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.428 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:52.428 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939562+0000", 2024-03-27T21:40:52.428 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082803100000000001 2024-03-27T21:40:52.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.429 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.429 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:52.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943838+0000", 2024-03-27T21:40:52.429 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042757050000000003 2024-03-27T21:40:52.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.429 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.429 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:52.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943859+0000", 2024-03-27T21:40:52.429 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1175e-05 2024-03-27T21:40:52.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.429 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.429 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:52.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943875+0000", 2024-03-27T21:40:52.430 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5744e-05 2024-03-27T21:40:52.430 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.430 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:52.430 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.430 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.430 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.430 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:52.430 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:52.430 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 72.338545569999994, 2024-03-27T21:40:52.430 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059712230000000003, 2024-03-27T21:40:52.430 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:52.430 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.431 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:52.431 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:52.431 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:52.431 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 112 2024-03-27T21:40:52.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.431 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:52.431 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:52.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:52.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.431 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:52.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:52.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:52.432 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080274+0000", 2024-03-27T21:40:52.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:52.432 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080318+0000", 2024-03-27T21:40:52.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3634999999999998e-05 2024-03-27T21:40:52.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.433 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:52.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080329+0000", 2024-03-27T21:40:52.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1232e-05 2024-03-27T21:40:52.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.433 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.433 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080333+0000", 2024-03-27T21:40:52.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6009999999999999e-06 2024-03-27T21:40:52.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.433 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.433 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:52.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083786+0000", 2024-03-27T21:40:52.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034528940000000002 2024-03-27T21:40:52.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.434 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:52.434 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083861+0000", 2024-03-27T21:40:52.434 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4893000000000003e-05 2024-03-27T21:40:52.434 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.434 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:52.434 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083981+0000", 2024-03-27T21:40:52.434 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012058299999999999 2024-03-27T21:40:52.434 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.434 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:52.434 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086148+0000", 2024-03-27T21:40:52.434 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021671469999999999 2024-03-27T21:40:52.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.435 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:52.435 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086195+0000", 2024-03-27T21:40:52.435 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6768999999999999e-05 2024-03-27T21:40:52.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.435 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:52.435 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086224+0000", 2024-03-27T21:40:52.435 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8758000000000001e-05 2024-03-27T21:40:52.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.435 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:52.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086246+0000", 2024-03-27T21:40:52.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2064000000000001e-05 2024-03-27T21:40:52.436 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.436 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:52.436 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.436 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.436 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.436 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:52.436 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:52.436 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 72.338482623000004, 2024-03-27T21:40:52.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066196320000000003, 2024-03-27T21:40:52.436 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:52.436 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.436 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:52.437 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:52.437 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:52.437 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113 2024-03-27T21:40:52.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.437 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:52.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:52.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:52.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:52.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:52.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.438 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:52.438 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080337+0000", 2024-03-27T21:40:52.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.438 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:52.438 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080340+0000", 2024-03-27T21:40:52.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6690000000000002e-06 2024-03-27T21:40:52.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.438 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:52.438 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080344+0000", 2024-03-27T21:40:52.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6550000000000002e-06 2024-03-27T21:40:52.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080348+0000", 2024-03-27T21:40:52.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9169999999999999e-06 2024-03-27T21:40:52.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:52.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084595+0000", 2024-03-27T21:40:52.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042471719999999996 2024-03-27T21:40:52.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:52.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084676+0000", 2024-03-27T21:40:52.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1147000000000001e-05 2024-03-27T21:40:52.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:52.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084796+0000", 2024-03-27T21:40:52.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012032900000000001 2024-03-27T21:40:52.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:52.441 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086190+0000", 2024-03-27T21:40:52.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013934399999999999 2024-03-27T21:40:52.441 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.441 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.441 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:52.441 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086905+0000", 2024-03-27T21:40:52.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071536199999999999 2024-03-27T21:40:52.442 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.442 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.442 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:52.442 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086937+0000", 2024-03-27T21:40:52.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2125000000000002e-05 2024-03-27T21:40:52.442 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.442 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.442 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:52.442 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086957+0000", 2024-03-27T21:40:52.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0157999999999999e-05 2024-03-27T21:40:52.442 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.442 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:52.443 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.443 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.443 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:52.443 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:52.443 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 72.338469907000004, 2024-03-27T21:40:52.443 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007405661, 2024-03-27T21:40:52.443 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:52.443 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.443 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:52.443 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:52.443 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:52.443 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 114 2024-03-27T21:40:52.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.443 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:52.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:52.444 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:52.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:52.444 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:52.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:52.444 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:52.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-03-27T21:40:52.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:52.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080353+0000", 2024-03-27T21:40:52.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.441e-06 2024-03-27T21:40:52.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:52.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080356+0000", 2024-03-27T21:40:52.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0769999999999999e-06 2024-03-27T21:40:52.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080358+0000", 2024-03-27T21:40:52.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3369999999999998e-06 2024-03-27T21:40:52.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:52.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085118+0000", 2024-03-27T21:40:52.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047603949999999997 2024-03-27T21:40:52.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:52.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085210+0000", 2024-03-27T21:40:52.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1983000000000003e-05 2024-03-27T21:40:52.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:52.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085332+0000", 2024-03-27T21:40:52.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121803 2024-03-27T21:40:52.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.447 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:52.447 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086292+0000", 2024-03-27T21:40:52.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095961400000000002 2024-03-27T21:40:52.447 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.447 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:52.447 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087710+0000", 2024-03-27T21:40:52.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014181459999999999 2024-03-27T21:40:52.447 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.447 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:52.447 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087736+0000", 2024-03-27T21:40:52.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6475e-05 2024-03-27T21:40:52.448 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.448 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.448 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:52.448 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087756+0000", 2024-03-27T21:40:52.448 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9626000000000001e-05 2024-03-27T21:40:52.448 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.448 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:52.448 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.448 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.448 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.448 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:52.448 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:52.448 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 72.338447471999999, 2024-03-27T21:40:52.449 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075011720000000004, 2024-03-27T21:40:52.449 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:52.449 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.449 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:52.449 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:52.449 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:52.449 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115 2024-03-27T21:40:52.449 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.449 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:52.449 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.449 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:52.449 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:52.449 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.449 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.450 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:52.450 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:52.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.450 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:52.450 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:52.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.450 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:52.450 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080458+0000", 2024-03-27T21:40:52.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5228000000000005e-05 2024-03-27T21:40:52.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:52.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080465+0000", 2024-03-27T21:40:52.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.238e-06 2024-03-27T21:40:52.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080468+0000", 2024-03-27T21:40:52.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5109999999999999e-06 2024-03-27T21:40:52.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:52.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086326+0000", 2024-03-27T21:40:52.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058579169999999998 2024-03-27T21:40:52.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:52.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086382+0000", 2024-03-27T21:40:52.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5309999999999997e-05 2024-03-27T21:40:52.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:52.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086457+0000", 2024-03-27T21:40:52.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5413000000000005e-05 2024-03-27T21:40:52.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:52.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087762+0000", 2024-03-27T21:40:52.453 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013052300000000001 2024-03-27T21:40:52.453 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.453 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.453 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:52.453 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087798+0000", 2024-03-27T21:40:52.453 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5614999999999997e-05 2024-03-27T21:40:52.453 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.453 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.453 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:52.453 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087840+0000", 2024-03-27T21:40:52.453 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2196000000000001e-05 2024-03-27T21:40:52.453 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.453 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.453 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:52.453 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087874+0000", 2024-03-27T21:40:52.454 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3785999999999998e-05 2024-03-27T21:40:52.454 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.454 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:52.454 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.454 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.454 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.454 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:52.454 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:52.454 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 72.338347389000006, 2024-03-27T21:40:52.454 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075050680000000002, 2024-03-27T21:40:52.454 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:52.454 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.455 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:52.455 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:52.455 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:52.455 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 116 2024-03-27T21:40:52.455 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.455 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:52.455 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.455 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:52.455 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:52.455 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.455 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.455 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.455 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:52.455 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:52.456 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.456 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.456 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.456 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:52.456 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080472+0000", 2024-03-27T21:40:52.456 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.456 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.456 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.456 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:52.456 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080475+0000", 2024-03-27T21:40:52.456 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.712e-06 2024-03-27T21:40:52.456 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.456 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.456 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:52.456 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080478+0000", 2024-03-27T21:40:52.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1070000000000001e-06 2024-03-27T21:40:52.457 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.457 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.457 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.457 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080481+0000", 2024-03-27T21:40:52.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4269999999999998e-06 2024-03-27T21:40:52.457 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.457 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.457 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:52.457 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086992+0000", 2024-03-27T21:40:52.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065114099999999996 2024-03-27T21:40:52.457 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.457 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.457 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:52.457 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087064+0000", 2024-03-27T21:40:52.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1927000000000004e-05 2024-03-27T21:40:52.458 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:52.458 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087188+0000", 2024-03-27T21:40:52.458 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123585 2024-03-27T21:40:52.458 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:52.458 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087766+0000", 2024-03-27T21:40:52.458 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057814300000000004 2024-03-27T21:40:52.458 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:52.458 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087934+0000", 2024-03-27T21:40:52.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000167894 2024-03-27T21:40:52.459 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.459 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.459 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:52.459 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087958+0000", 2024-03-27T21:40:52.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4663000000000001e-05 2024-03-27T21:40:52.459 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.459 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.459 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:52.459 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087978+0000", 2024-03-27T21:40:52.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9636e-05 2024-03-27T21:40:52.459 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.459 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:52.459 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.459 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.460 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.460 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:52.460 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:52.460 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 72.338315266999999, 2024-03-27T21:40:52.460 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076850679999999998, 2024-03-27T21:40:52.460 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:52.460 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.460 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:52.460 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:52.460 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:52.460 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 117 2024-03-27T21:40:52.460 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.460 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:52.460 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.461 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:52.461 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:52.461 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.461 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.461 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.461 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:52.461 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:52.461 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.461 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.461 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.461 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:52.461 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:52.461 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.461 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.461 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.461 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:52.462 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080508+0000", 2024-03-27T21:40:52.462 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.298e-06 2024-03-27T21:40:52.462 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.462 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.462 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:52.462 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080515+0000", 2024-03-27T21:40:52.462 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0790000000000003e-06 2024-03-27T21:40:52.462 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.462 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.462 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.462 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080518+0000", 2024-03-27T21:40:52.462 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8509999999999999e-06 2024-03-27T21:40:52.462 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.462 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.463 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:52.463 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087444+0000", 2024-03-27T21:40:52.463 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0069259120000000002 2024-03-27T21:40:52.463 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.463 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.463 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:52.463 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087491+0000", 2024-03-27T21:40:52.463 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7043999999999999e-05 2024-03-27T21:40:52.463 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.463 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.463 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:52.463 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087564+0000", 2024-03-27T21:40:52.463 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3603999999999995e-05 2024-03-27T21:40:52.463 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.463 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.464 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:52.464 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087902+0000", 2024-03-27T21:40:52.464 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033734100000000001 2024-03-27T21:40:52.464 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.464 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.464 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:52.464 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088117+0000", 2024-03-27T21:40:52.464 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000215164 2024-03-27T21:40:52.464 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.464 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.464 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:52.464 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088160+0000", 2024-03-27T21:40:52.465 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2933999999999999e-05 2024-03-27T21:40:52.465 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.465 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.465 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:52.465 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088190+0000", 2024-03-27T21:40:52.465 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0139999999999999e-05 2024-03-27T21:40:52.465 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.465 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:52.465 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.465 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.465 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.465 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:52.465 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:52.465 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 71.922034878000005, 2024-03-27T21:40:52.465 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052465439999999997, 2024-03-27T21:40:52.465 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:52.466 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.466 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:52.466 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:52.466 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:52.466 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 133 2024-03-27T21:40:52.466 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.466 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:52.466 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.466 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:52.466 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:52.466 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.466 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.466 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.466 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:52.466 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:52.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.467 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.467 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:52.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:52.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.467 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.467 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:52.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496788+0000", 2024-03-27T21:40:52.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-03-27T21:40:52.467 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.467 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:52.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496793+0000", 2024-03-27T21:40:52.468 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4790000000000004e-06 2024-03-27T21:40:52.468 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.468 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.468 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.468 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496796+0000", 2024-03-27T21:40:52.468 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1e-06 2024-03-27T21:40:52.468 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.468 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.468 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:52.468 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500447+0000", 2024-03-27T21:40:52.468 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036504459999999999 2024-03-27T21:40:52.468 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.468 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.468 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:52.469 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500492+0000", 2024-03-27T21:40:52.469 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5765e-05 2024-03-27T21:40:52.469 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.469 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.469 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:52.469 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500562+0000", 2024-03-27T21:40:52.469 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9206000000000001e-05 2024-03-27T21:40:52.469 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.469 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.469 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:52.469 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501959+0000", 2024-03-27T21:40:52.469 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001397187 2024-03-27T21:40:52.469 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.469 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.469 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:52.469 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501984+0000", 2024-03-27T21:40:52.470 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.552e-05 2024-03-27T21:40:52.470 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.470 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.470 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:52.470 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502010+0000", 2024-03-27T21:40:52.470 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5232000000000002e-05 2024-03-27T21:40:52.470 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.470 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.470 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:52.470 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502032+0000", 2024-03-27T21:40:52.470 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2337999999999999e-05 2024-03-27T21:40:52.470 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.470 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:52.470 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.470 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.470 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.471 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:52.471 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:52.471 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 71.921976220000005, 2024-03-27T21:40:52.471 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052892119999999997, 2024-03-27T21:40:52.471 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:52.471 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.471 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:52.471 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:52.471 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:52.471 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 134 2024-03-27T21:40:52.471 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.471 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:52.471 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.471 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:52.472 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:52.472 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.472 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.472 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.472 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:52.472 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:52.472 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.472 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.472 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.472 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:52.472 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:52.472 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.472 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.472 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.472 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:52.472 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496929+0000", 2024-03-27T21:40:52.473 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5098000000000004e-05 2024-03-27T21:40:52.473 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.473 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.473 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:52.473 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496937+0000", 2024-03-27T21:40:52.473 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9559999999999997e-06 2024-03-27T21:40:52.473 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.473 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.473 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.473 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496942+0000", 2024-03-27T21:40:52.473 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6350000000000003e-06 2024-03-27T21:40:52.473 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.473 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.473 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:52.474 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500749+0000", 2024-03-27T21:40:52.474 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038068529999999998 2024-03-27T21:40:52.474 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.474 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.474 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:52.474 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500792+0000", 2024-03-27T21:40:52.474 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2823e-05 2024-03-27T21:40:52.474 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.474 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.474 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:52.474 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500864+0000", 2024-03-27T21:40:52.474 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1715e-05 2024-03-27T21:40:52.474 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.474 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.474 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:52.475 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502068+0000", 2024-03-27T21:40:52.475 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012044860000000001 2024-03-27T21:40:52.475 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.475 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:52.475 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502073+0000", 2024-03-27T21:40:52.475 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5700000000000003e-06 2024-03-27T21:40:52.475 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.475 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:52.475 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502098+0000", 2024-03-27T21:40:52.475 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5120999999999999e-05 2024-03-27T21:40:52.475 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.475 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:52.475 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502133+0000", 2024-03-27T21:40:52.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5306999999999999e-05 2024-03-27T21:40:52.476 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.476 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:52.476 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.476 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.476 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.476 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:52.476 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:52.476 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 71.921870913000006, 2024-03-27T21:40:52.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062529040000000001, 2024-03-27T21:40:52.476 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:52.476 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.476 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:52.476 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:52.477 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:52.477 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135 2024-03-27T21:40:52.477 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.477 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:52.477 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.477 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:52.477 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:52.477 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.477 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.477 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.477 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:52.477 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:52.477 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.477 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.477 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.477 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:52.477 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:52.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.478 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.478 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:52.478 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496953+0000", 2024-03-27T21:40:52.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0709999999999996e-06 2024-03-27T21:40:52.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.478 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.478 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:52.478 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496961+0000", 2024-03-27T21:40:52.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1300000000000001e-06 2024-03-27T21:40:52.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.478 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.479 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.479 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496968+0000", 2024-03-27T21:40:52.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0999999999999998e-06 2024-03-27T21:40:52.479 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.479 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.479 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:52.479 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501244+0000", 2024-03-27T21:40:52.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042756310000000002 2024-03-27T21:40:52.479 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.479 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.479 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:52.479 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501377+0000", 2024-03-27T21:40:52.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013348799999999999 2024-03-27T21:40:52.480 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.480 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.480 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:52.480 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501455+0000", 2024-03-27T21:40:52.480 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7410000000000006e-05 2024-03-27T21:40:52.480 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.480 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.480 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:52.480 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502138+0000", 2024-03-27T21:40:52.480 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068386599999999999 2024-03-27T21:40:52.480 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.480 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.480 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:52.480 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503152+0000", 2024-03-27T21:40:52.481 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010139680000000001 2024-03-27T21:40:52.481 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.481 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.481 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:52.481 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503180+0000", 2024-03-27T21:40:52.481 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7426000000000001e-05 2024-03-27T21:40:52.481 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.481 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.481 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:52.481 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503202+0000", 2024-03-27T21:40:52.481 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2463000000000002e-05 2024-03-27T21:40:52.481 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.481 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:52.482 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.482 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.482 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.482 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:52.482 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:52.482 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 71.921846775000006, 2024-03-27T21:40:52.482 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066253680000000004, 2024-03-27T21:40:52.482 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:52.482 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.482 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:52.482 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:52.482 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:52.482 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 136 2024-03-27T21:40:52.482 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.483 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:52.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:52.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:52.483 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.483 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:52.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:52.483 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.483 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:52.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:52.483 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.484 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.484 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.484 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:52.484 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496977+0000", 2024-03-27T21:40:52.484 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5049999999999999e-06 2024-03-27T21:40:52.484 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.484 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.484 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:52.484 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496984+0000", 2024-03-27T21:40:52.484 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1520000000000003e-06 2024-03-27T21:40:52.484 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.484 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.484 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.484 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496990+0000", 2024-03-27T21:40:52.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-03-27T21:40:52.485 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.485 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.485 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:52.485 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501669+0000", 2024-03-27T21:40:52.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046793859999999998 2024-03-27T21:40:52.485 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.485 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.485 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:52.485 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501714+0000", 2024-03-27T21:40:52.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5154999999999997e-05 2024-03-27T21:40:52.485 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.485 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.485 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:52.486 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501783+0000", 2024-03-27T21:40:52.486 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8454999999999994e-05 2024-03-27T21:40:52.486 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.486 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.486 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:52.486 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503226+0000", 2024-03-27T21:40:52.486 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001443441 2024-03-27T21:40:52.486 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.486 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.486 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:52.486 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503540+0000", 2024-03-27T21:40:52.486 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031347100000000001 2024-03-27T21:40:52.487 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.487 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:52.487 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503563+0000", 2024-03-27T21:40:52.487 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3223000000000001e-05 2024-03-27T21:40:52.487 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.487 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:52.487 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503599+0000", 2024-03-27T21:40:52.487 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6105000000000001e-05 2024-03-27T21:40:52.487 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.487 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:52.487 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.487 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.488 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:52.488 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:52.488 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 71.921823965000002, 2024-03-27T21:40:52.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074261320000000002, 2024-03-27T21:40:52.488 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:52.488 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.488 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:52.488 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:52.488 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:52.488 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 137 2024-03-27T21:40:52.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.488 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:52.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:52.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:52.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:52.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:52.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:52.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:52.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.490 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.490 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:52.490 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497077+0000", 2024-03-27T21:40:52.490 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1599000000000003e-05 2024-03-27T21:40:52.490 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.490 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.490 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:52.490 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497085+0000", 2024-03-27T21:40:52.490 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6109999999999996e-06 2024-03-27T21:40:52.490 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.490 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.490 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.490 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497092+0000", 2024-03-27T21:40:52.490 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.742e-06 2024-03-27T21:40:52.491 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.491 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.491 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:52.491 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502159+0000", 2024-03-27T21:40:52.491 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050666870000000003 2024-03-27T21:40:52.491 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.491 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.491 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:52.491 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502200+0000", 2024-03-27T21:40:52.491 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1919999999999998e-05 2024-03-27T21:40:52.491 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.491 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.491 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:52.492 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502280+0000", 2024-03-27T21:40:52.492 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9417000000000002e-05 2024-03-27T21:40:52.492 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.492 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.492 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:52.492 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503605+0000", 2024-03-27T21:40:52.492 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001325641 2024-03-27T21:40:52.492 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.492 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.492 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:52.492 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504361+0000", 2024-03-27T21:40:52.492 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075555399999999997 2024-03-27T21:40:52.492 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.492 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.492 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:52.493 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504386+0000", 2024-03-27T21:40:52.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5369999999999999e-05 2024-03-27T21:40:52.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:52.493 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504422+0000", 2024-03-27T21:40:52.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6019000000000002e-05 2024-03-27T21:40:52.493 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.493 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:52.493 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.493 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:52.494 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:52.494 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 71.921720167999993, 2024-03-27T21:40:52.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074059750000000004, 2024-03-27T21:40:52.494 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:52.494 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.494 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:52.494 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:52.494 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:52.494 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-03-27T21:40:52.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.494 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:52.494 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.494 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:52.494 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:52.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.495 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:52.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:52.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.495 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:52.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:52.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.495 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:52.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497103+0000", 2024-03-27T21:40:52.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.856e-06 2024-03-27T21:40:52.496 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.496 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.496 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:52.496 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497121+0000", 2024-03-27T21:40:52.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7190999999999999e-05 2024-03-27T21:40:52.496 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.496 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.496 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.496 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497126+0000", 2024-03-27T21:40:52.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3770000000000004e-06 2024-03-27T21:40:52.496 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.496 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.496 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:52.496 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502833+0000", 2024-03-27T21:40:52.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057073510000000003 2024-03-27T21:40:52.496 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.497 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.497 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:52.497 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502881+0000", 2024-03-27T21:40:52.497 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7961999999999999e-05 2024-03-27T21:40:52.497 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.497 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.497 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:52.497 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502949+0000", 2024-03-27T21:40:52.497 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7957e-05 2024-03-27T21:40:52.497 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.497 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.497 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:52.497 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504430+0000", 2024-03-27T21:40:52.498 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001480763 2024-03-27T21:40:52.498 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.498 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.498 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:52.498 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504459+0000", 2024-03-27T21:40:52.498 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8983999999999999e-05 2024-03-27T21:40:52.498 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.498 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.498 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:52.498 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504486+0000", 2024-03-27T21:40:52.498 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6883999999999999e-05 2024-03-27T21:40:52.498 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.498 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.499 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:52.499 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504506+0000", 2024-03-27T21:40:52.499 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0214e-05 2024-03-27T21:40:52.499 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.499 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:52.499 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.499 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.499 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.499 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:52.499 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:52.499 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 71.921688333999995, 2024-03-27T21:40:52.499 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074754859999999999, 2024-03-27T21:40:52.499 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:52.499 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.499 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:52.500 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:52.500 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:52.500 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-03-27T21:40:52.500 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.500 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:52.500 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.500 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:52.500 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:52.500 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.500 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.500 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.500 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:52.500 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:52.501 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.501 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.501 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.501 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:52.501 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:52.501 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.501 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.501 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.501 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:52.501 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497137+0000", 2024-03-27T21:40:52.501 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0730000000000004e-06 2024-03-27T21:40:52.501 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.501 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.501 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:52.501 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497143+0000", 2024-03-27T21:40:52.502 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3759999999999996e-06 2024-03-27T21:40:52.502 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.502 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.502 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.502 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497150+0000", 2024-03-27T21:40:52.502 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.002e-06 2024-03-27T21:40:52.502 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.502 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.502 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:52.502 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503249+0000", 2024-03-27T21:40:52.502 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060989750000000004 2024-03-27T21:40:52.502 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.502 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.503 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:52.503 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503291+0000", 2024-03-27T21:40:52.503 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1563999999999997e-05 2024-03-27T21:40:52.503 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.503 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.503 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:52.503 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503376+0000", 2024-03-27T21:40:52.503 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5752000000000002e-05 2024-03-27T21:40:52.503 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.503 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.503 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:52.503 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504545+0000", 2024-03-27T21:40:52.503 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169172 2024-03-27T21:40:52.503 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.504 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.504 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:52.504 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504550+0000", 2024-03-27T21:40:52.504 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7419999999999997e-06 2024-03-27T21:40:52.504 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.504 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.504 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:52.504 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504574+0000", 2024-03-27T21:40:52.504 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4124000000000001e-05 2024-03-27T21:40:52.504 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.504 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.504 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:52.504 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504607+0000", 2024-03-27T21:40:52.505 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3118999999999999e-05 2024-03-27T21:40:52.505 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.505 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:52.505 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.505 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.505 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.505 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:52.505 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:52.505 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 71.921664668000005, 2024-03-27T21:40:52.505 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078150589999999992, 2024-03-27T21:40:52.505 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:52.505 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.505 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:52.505 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:52.505 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:52.505 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-03-27T21:40:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497155+0000", 2024-03-27T21:40:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497230+0000", 2024-03-27T21:40:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4789000000000005e-05 2024-03-27T21:40:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497241+0000", 2024-03-27T21:40:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1442e-05 2024-03-27T21:40:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497255+0000", 2024-03-27T21:40:52.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3087e-05 2024-03-27T21:40:52.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:52.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503630+0000", 2024-03-27T21:40:52.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063755269999999998 2024-03-27T21:40:52.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:52.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503673+0000", 2024-03-27T21:40:52.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2616e-05 2024-03-27T21:40:52.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:52.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503738+0000", 2024-03-27T21:40:52.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5114e-05 2024-03-27T21:40:52.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.509 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:52.509 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504907+0000", 2024-03-27T21:40:52.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169432 2024-03-27T21:40:52.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.509 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:52.509 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504926+0000", 2024-03-27T21:40:52.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9086999999999999e-05 2024-03-27T21:40:52.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:52.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504950+0000", 2024-03-27T21:40:52.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3598999999999999e-05 2024-03-27T21:40:52.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.510 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:52.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504971+0000", 2024-03-27T21:40:52.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0772999999999999e-05 2024-03-27T21:40:52.510 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.510 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:52.510 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.510 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.510 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:52.510 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:52.511 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 71.921557628000002, 2024-03-27T21:40:52.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082962590000000003, 2024-03-27T21:40:52.511 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:52.511 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.511 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:52.511 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:52.511 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:52.511 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-03-27T21:40:52.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.511 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:52.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.511 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:52.511 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:52.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:52.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:52.512 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:52.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497262+0000", 2024-03-27T21:40:52.512 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:52.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497267+0000", 2024-03-27T21:40:52.512 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.955e-06 2024-03-27T21:40:52.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.513 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:52.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497275+0000", 2024-03-27T21:40:52.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.824e-06 2024-03-27T21:40:52.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.513 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497280+0000", 2024-03-27T21:40:52.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8230000000000002e-06 2024-03-27T21:40:52.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.513 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:52.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504628+0000", 2024-03-27T21:40:52.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007348561 2024-03-27T21:40:52.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.514 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:52.514 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504673+0000", 2024-03-27T21:40:52.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4313000000000001e-05 2024-03-27T21:40:52.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.514 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:52.514 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504745+0000", 2024-03-27T21:40:52.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2303000000000002e-05 2024-03-27T21:40:52.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:52.515 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505358+0000", 2024-03-27T21:40:52.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061281300000000003 2024-03-27T21:40:52.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:52.515 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505503+0000", 2024-03-27T21:40:52.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000145134 2024-03-27T21:40:52.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:52.515 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505526+0000", 2024-03-27T21:40:52.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3312e-05 2024-03-27T21:40:52.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.516 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:52.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505559+0000", 2024-03-27T21:40:52.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2821999999999998e-05 2024-03-27T21:40:52.516 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.516 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:52.516 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.516 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.516 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.516 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:52.516 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:52.516 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 71.312335388999998, 2024-03-27T21:40:52.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053753339999999998, 2024-03-27T21:40:52.516 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:52.516 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.516 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:52.517 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:52.517 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:52.517 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159 2024-03-27T21:40:52.517 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.517 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:52.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.517 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:52.517 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:52.517 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.517 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.517 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:52.517 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:52.517 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:52.517 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:52.518 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:52.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.518 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:52.518 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106488+0000", 2024-03-27T21:40:52.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9830000000000001e-06 2024-03-27T21:40:52.518 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:52.518 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106491+0000", 2024-03-27T21:40:52.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3459999999999998e-06 2024-03-27T21:40:52.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106493+0000", 2024-03-27T21:40:52.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.509e-06 2024-03-27T21:40:52.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:52.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110636+0000", 2024-03-27T21:40:52.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041427260000000002 2024-03-27T21:40:52.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:52.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110683+0000", 2024-03-27T21:40:52.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6557000000000002e-05 2024-03-27T21:40:52.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:52.520 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110755+0000", 2024-03-27T21:40:52.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2119000000000005e-05 2024-03-27T21:40:52.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:52.520 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111415+0000", 2024-03-27T21:40:52.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066065199999999996 2024-03-27T21:40:52.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:52.520 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111793+0000", 2024-03-27T21:40:52.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037732100000000001 2024-03-27T21:40:52.521 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.521 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.521 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:52.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111829+0000", 2024-03-27T21:40:52.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418999999999998e-05 2024-03-27T21:40:52.521 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:52.521 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:52.521 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:52.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111860+0000", 2024-03-27T21:40:52.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1003000000000001e-05 2024-03-27T21:40:52.521 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.521 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:52.521 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.521 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:52.522 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:52.522 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:52.523 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:40:52.524 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:52.537+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:52.624 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:52.638+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:52.632 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:52.632 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:52.632 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:52.632 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:52.641 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:40:52.725 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:52.739+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:52.791 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:52.791 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:52.791 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:52.791 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:52.791 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:52.800 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:40:52.844 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:52.858+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:52.912 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:52.912 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:52.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:52.912 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-03-27T21:40:52.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.912 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:20 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2024-03-27T21:40:52.912 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:52.913 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 73.036786359000004, 2024-03-27T21:40:52.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015345630000000001, 2024-03-27T21:40:52.913 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:52.913 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.913 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:52.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:52.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:52.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:52.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:52.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:52.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888562+0000", 2024-03-27T21:40:52.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:52.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:52.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888735+0000", 2024-03-27T21:40:52.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017254799999999999 2024-03-27T21:40:52.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:52.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888747+0000", 2024-03-27T21:40:52.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2469999999999999e-05 2024-03-27T21:40:52.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.915 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888761+0000", 2024-03-27T21:40:52.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3254e-05 2024-03-27T21:40:52.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.915 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:52.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888804+0000", 2024-03-27T21:40:52.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3129e-05 2024-03-27T21:40:52.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.916 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:52.916 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888820+0000", 2024-03-27T21:40:52.916 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6426000000000002e-05 2024-03-27T21:40:52.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.916 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:52.916 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890060+0000", 2024-03-27T21:40:52.916 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012398050000000001 2024-03-27T21:40:52.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.916 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:52.916 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890098+0000", 2024-03-27T21:40:52.916 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7795e-05 2024-03-27T21:40:52.917 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.917 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:52.917 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.917 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2024-03-27T21:40:52.917 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:52.917 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 73.035961959000005, 2024-03-27T21:40:52.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001069555, 2024-03-27T21:40:52.917 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:52.917 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.917 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:52.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.917 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:52.917 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:52.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.918 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:52.918 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:52.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.918 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:52.918 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889387+0000", 2024-03-27T21:40:52.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:52.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:52.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889455+0000", 2024-03-27T21:40:52.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7930999999999997e-05 2024-03-27T21:40:52.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:52.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889470+0000", 2024-03-27T21:40:52.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5554e-05 2024-03-27T21:40:52.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889476+0000", 2024-03-27T21:40:52.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0619999999999997e-06 2024-03-27T21:40:52.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.920 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:52.920 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889513+0000", 2024-03-27T21:40:52.920 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6606999999999997e-05 2024-03-27T21:40:52.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.920 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:52.920 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889528+0000", 2024-03-27T21:40:52.920 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4982e-05 2024-03-27T21:40:52.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.920 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:52.920 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890421+0000", 2024-03-27T21:40:52.920 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00089293700000000001 2024-03-27T21:40:52.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:52.921 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890457+0000", 2024-03-27T21:40:52.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6260000000000002e-05 2024-03-27T21:40:52.921 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.921 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:52.921 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.921 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:38 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'24, mlcod=17'22)", 2024-03-27T21:40:52.921 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:52.921 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 73.018725993000004, 2024-03-27T21:40:52.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016182080000000001, 2024-03-27T21:40:52.921 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:52.921 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.921 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:52.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:52.922 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:52.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.922 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:52.922 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:52.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.922 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:52.922 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906623+0000", 2024-03-27T21:40:52.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:52.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.923 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:52.923 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906687+0000", 2024-03-27T21:40:52.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3706000000000003e-05 2024-03-27T21:40:52.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.923 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:52.923 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906698+0000", 2024-03-27T21:40:52.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1752e-05 2024-03-27T21:40:52.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.923 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.923 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906704+0000", 2024-03-27T21:40:52.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3940000000000004e-06 2024-03-27T21:40:52.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.924 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:52.924 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906741+0000", 2024-03-27T21:40:52.924 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6944999999999999e-05 2024-03-27T21:40:52.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.924 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.924 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:52.924 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906760+0000", 2024-03-27T21:40:52.924 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9293e-05 2024-03-27T21:40:52.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.924 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.924 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:52.924 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908211+0000", 2024-03-27T21:40:52.924 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014511350000000001 2024-03-27T21:40:52.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.925 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:52.925 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908242+0000", 2024-03-27T21:40:52.925 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0712e-05 2024-03-27T21:40:52.925 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.925 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:52.925 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.925 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:39 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'25, mlcod=17'22)", 2024-03-27T21:40:52.925 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:52.925 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 73.017881078000002, 2024-03-27T21:40:52.926 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012060199999999999, 2024-03-27T21:40:52.926 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:52.926 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.926 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:52.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.926 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:52.926 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:52.926 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.926 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:52.926 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:52.926 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.927 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.927 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:52.927 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:52.927 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:52.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.927 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.927 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:52.927 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907533+0000", 2024-03-27T21:40:52.927 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5675999999999999e-05 2024-03-27T21:40:52.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.927 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.927 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:52.927 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907545+0000", 2024-03-27T21:40:52.927 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.115e-05 2024-03-27T21:40:52.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.928 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.928 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.928 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907550+0000", 2024-03-27T21:40:52.928 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8320000000000002e-06 2024-03-27T21:40:52.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.928 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.928 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:52.928 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907592+0000", 2024-03-27T21:40:52.928 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1309e-05 2024-03-27T21:40:52.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.928 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.928 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:52.928 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907611+0000", 2024-03-27T21:40:52.928 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9009e-05 2024-03-27T21:40:52.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.929 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:52.929 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908638+0000", 2024-03-27T21:40:52.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001027533 2024-03-27T21:40:52.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.929 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:52.929 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908675+0000", 2024-03-27T21:40:52.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6329999999999999e-05 2024-03-27T21:40:52.929 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.929 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:52.929 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.930 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:57 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'35, mlcod=17'33)", 2024-03-27T21:40:52.930 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:52.930 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 73.003796645999998, 2024-03-27T21:40:52.930 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001346578, 2024-03-27T21:40:52.930 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:52.930 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.930 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:52.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.930 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:52.930 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:52.930 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.931 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:52.931 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:52.931 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.931 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:52.931 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921552+0000", 2024-03-27T21:40:52.931 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.931 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:52.931 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921615+0000", 2024-03-27T21:40:52.931 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2787999999999996e-05 2024-03-27T21:40:52.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:52.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921626+0000", 2024-03-27T21:40:52.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0888e-05 2024-03-27T21:40:52.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921635+0000", 2024-03-27T21:40:52.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4639999999999995e-06 2024-03-27T21:40:52.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:52.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921673+0000", 2024-03-27T21:40:52.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7891e-05 2024-03-27T21:40:52.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.933 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:52.933 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921694+0000", 2024-03-27T21:40:52.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0426000000000001e-05 2024-03-27T21:40:52.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.933 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:52.933 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922865+0000", 2024-03-27T21:40:52.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011709509999999999 2024-03-27T21:40:52.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.933 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:52.933 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922900+0000", 2024-03-27T21:40:52.934 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4819e-05 2024-03-27T21:40:52.934 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.934 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:52.934 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.934 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:71 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'45, mlcod=17'43)", 2024-03-27T21:40:52.934 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:52.934 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 72.988497933999994, 2024-03-27T21:40:52.934 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001460907, 2024-03-27T21:40:52.934 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:52.934 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.934 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:52.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.934 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:52.935 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:52.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:52.935 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:52.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:52.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936851+0000", 2024-03-27T21:40:52.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:52.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.936 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:52.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936903+0000", 2024-03-27T21:40:52.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2302999999999997e-05 2024-03-27T21:40:52.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.936 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:52.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936915+0000", 2024-03-27T21:40:52.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1967000000000001e-05 2024-03-27T21:40:52.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.936 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.937 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936923+0000", 2024-03-27T21:40:52.937 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9370000000000004e-06 2024-03-27T21:40:52.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.937 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:52.937 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936970+0000", 2024-03-27T21:40:52.937 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6987000000000003e-05 2024-03-27T21:40:52.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.937 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:52.937 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937002+0000", 2024-03-27T21:40:52.937 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2410999999999999e-05 2024-03-27T21:40:52.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.937 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:52.938 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938279+0000", 2024-03-27T21:40:52.938 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001276552 2024-03-27T21:40:52.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.938 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:52.938 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938313+0000", 2024-03-27T21:40:52.938 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3526999999999999e-05 2024-03-27T21:40:52.938 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.938 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:52.938 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.938 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:72 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'46, mlcod=17'43)", 2024-03-27T21:40:52.938 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:52.939 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 72.987860372, 2024-03-27T21:40:52.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014239490000000001, 2024-03-27T21:40:52.939 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:52.939 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.939 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:52.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.939 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:52.939 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:52.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.939 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:52.939 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:52.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.940 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:52.940 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937488+0000", 2024-03-27T21:40:52.940 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:52.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.940 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:52.940 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937547+0000", 2024-03-27T21:40:52.940 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8196999999999999e-05 2024-03-27T21:40:52.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.940 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:52.940 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937559+0000", 2024-03-27T21:40:52.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1928e-05 2024-03-27T21:40:52.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.941 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937564+0000", 2024-03-27T21:40:52.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4759999999999997e-06 2024-03-27T21:40:52.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:52.941 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937621+0000", 2024-03-27T21:40:52.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6712000000000002e-05 2024-03-27T21:40:52.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:52.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937640+0000", 2024-03-27T21:40:52.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9517e-05 2024-03-27T21:40:52.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:52.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938883+0000", 2024-03-27T21:40:52.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001242567 2024-03-27T21:40:52.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:52.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938913+0000", 2024-03-27T21:40:52.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0287e-05 2024-03-27T21:40:52.942 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.942 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:52.942 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.943 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:73 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'47, mlcod=17'43)", 2024-03-27T21:40:52.943 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:52.943 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 72.987184278000001, 2024-03-27T21:40:52.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010746239999999999, 2024-03-27T21:40:52.943 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:52.943 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.943 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:52.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.943 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:52.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:52.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.944 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:52.944 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:52.944 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.944 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:52.944 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:52.944 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:52.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.944 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:52.944 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938217+0000", 2024-03-27T21:40:52.945 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2743e-05 2024-03-27T21:40:52.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.945 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.945 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:52.945 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938229+0000", 2024-03-27T21:40:52.945 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1956999999999999e-05 2024-03-27T21:40:52.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.945 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.945 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.945 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938235+0000", 2024-03-27T21:40:52.945 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4589999999999997e-06 2024-03-27T21:40:52.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.946 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:52.946 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938341+0000", 2024-03-27T21:40:52.946 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000106054 2024-03-27T21:40:52.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.946 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:52.946 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938369+0000", 2024-03-27T21:40:52.946 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8291000000000001e-05 2024-03-27T21:40:52.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.946 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:52.946 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939209+0000", 2024-03-27T21:40:52.946 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084045400000000003 2024-03-27T21:40:52.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:52.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939240+0000", 2024-03-27T21:40:52.947 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0429999999999998e-05 2024-03-27T21:40:52.947 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.947 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:52.947 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.947 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:75 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'49, mlcod=17'47)", 2024-03-27T21:40:52.947 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:52.947 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 72.980746631000002, 2024-03-27T21:40:52.947 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001017001, 2024-03-27T21:40:52.948 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:52.948 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.948 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:52.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.948 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:52.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:52.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.948 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:52.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:52.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.949 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:52.949 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944601+0000", 2024-03-27T21:40:52.949 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999986 2024-03-27T21:40:52.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.949 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.949 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:52.949 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944614+0000", 2024-03-27T21:40:52.949 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2496e-05 2024-03-27T21:40:52.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.949 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.949 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:52.949 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944640+0000", 2024-03-27T21:40:52.949 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5865999999999999e-05 2024-03-27T21:40:52.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.950 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.950 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.950 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944651+0000", 2024-03-27T21:40:52.950 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1797000000000001e-05 2024-03-27T21:40:52.950 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.950 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.950 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:52.950 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944762+0000", 2024-03-27T21:40:52.950 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110412 2024-03-27T21:40:52.950 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.950 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.950 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:52.950 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944820+0000", 2024-03-27T21:40:52.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7649999999999999e-05 2024-03-27T21:40:52.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.951 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:52.951 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945585+0000", 2024-03-27T21:40:52.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076562600000000005 2024-03-27T21:40:52.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.951 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:52.951 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945620+0000", 2024-03-27T21:40:52.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4764000000000001e-05 2024-03-27T21:40:52.951 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.951 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:52.951 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.952 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:86 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'50)", 2024-03-27T21:40:52.952 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:52.952 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 72.972985144000006, 2024-03-27T21:40:52.952 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011392380000000001, 2024-03-27T21:40:52.952 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:52.952 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.952 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:52.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.952 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:52.952 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:52.952 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.953 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:52.953 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:52.953 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.953 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:52.953 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:52.953 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.954 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:52.954 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952373+0000", 2024-03-27T21:40:52.954 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9220000000000003e-06 2024-03-27T21:40:52.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.954 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:52.954 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952381+0000", 2024-03-27T21:40:52.954 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3070000000000003e-06 2024-03-27T21:40:52.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.954 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952385+0000", 2024-03-27T21:40:52.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.121e-06 2024-03-27T21:40:52.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.955 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:52.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952419+0000", 2024-03-27T21:40:52.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4082999999999999e-05 2024-03-27T21:40:52.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:52.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952440+0000", 2024-03-27T21:40:52.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0837999999999999e-05 2024-03-27T21:40:52.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.957 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:52.957 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953472+0000", 2024-03-27T21:40:52.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001032119 2024-03-27T21:40:52.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.957 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:52.957 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953504+0000", 2024-03-27T21:40:52.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1250000000000001e-05 2024-03-27T21:40:52.957 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.957 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:52.958 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.958 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.958 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:87 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-27T21:40:52.958 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:52.958 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 72.972617886999998, 2024-03-27T21:40:52.958 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00135519, 2024-03-27T21:40:52.958 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:52.958 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.958 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:52.958 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.958 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:52.958 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:52.958 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:52.959 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:52.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:52.959 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952731+0000", 2024-03-27T21:40:52.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:52.959 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952781+0000", 2024-03-27T21:40:52.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0161000000000003e-05 2024-03-27T21:40:52.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:52.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952794+0000", 2024-03-27T21:40:52.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2853e-05 2024-03-27T21:40:52.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952799+0000", 2024-03-27T21:40:52.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5560000000000001e-06 2024-03-27T21:40:52.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:52.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952832+0000", 2024-03-27T21:40:52.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3834999999999997e-05 2024-03-27T21:40:52.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:52.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952851+0000", 2024-03-27T21:40:52.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8289999999999999e-05 2024-03-27T21:40:52.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:52.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954055+0000", 2024-03-27T21:40:52.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012037949999999999 2024-03-27T21:40:52.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:52.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954087+0000", 2024-03-27T21:40:52.962 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2292999999999997e-05 2024-03-27T21:40:52.962 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.962 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:52.962 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.962 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:89 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'59, mlcod=17'57)", 2024-03-27T21:40:52.962 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:52.962 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 72.970343666000005, 2024-03-27T21:40:52.962 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00138893, 2024-03-27T21:40:52.962 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:52.962 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.962 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:52.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.963 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:52.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:52.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.963 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:52.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:52.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.963 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:52.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955005+0000", 2024-03-27T21:40:52.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:52.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955071+0000", 2024-03-27T21:40:52.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5290999999999995e-05 2024-03-27T21:40:52.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:52.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955081+0000", 2024-03-27T21:40:52.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0572e-05 2024-03-27T21:40:52.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955086+0000", 2024-03-27T21:40:52.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8380000000000001e-06 2024-03-27T21:40:52.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:52.965 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955122+0000", 2024-03-27T21:40:52.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5698000000000003e-05 2024-03-27T21:40:52.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:52.965 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955140+0000", 2024-03-27T21:40:52.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7900000000000001e-05 2024-03-27T21:40:52.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:52.965 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956366+0000", 2024-03-27T21:40:52.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001226471 2024-03-27T21:40:52.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:52.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956395+0000", 2024-03-27T21:40:52.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8782e-05 2024-03-27T21:40:52.966 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.966 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:52.966 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.966 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:90 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'60, mlcod=17'57)", 2024-03-27T21:40:52.966 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:52.966 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 72.969659089000004, 2024-03-27T21:40:52.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010778859999999999, 2024-03-27T21:40:52.966 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:52.966 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.966 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:52.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:52.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:52.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:52.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:52.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:52.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:52.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:52.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955747+0000", 2024-03-27T21:40:52.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7130999999999999e-05 2024-03-27T21:40:52.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:52.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955756+0000", 2024-03-27T21:40:52.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2750000000000001e-06 2024-03-27T21:40:52.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955761+0000", 2024-03-27T21:40:52.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3900000000000003e-06 2024-03-27T21:40:52.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:52.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955797+0000", 2024-03-27T21:40:52.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6650000000000003e-05 2024-03-27T21:40:52.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:52.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955816+0000", 2024-03-27T21:40:52.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8195000000000001e-05 2024-03-27T21:40:52.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:52.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956739+0000", 2024-03-27T21:40:52.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00092369700000000002 2024-03-27T21:40:52.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:52.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956768+0000", 2024-03-27T21:40:52.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9048e-05 2024-03-27T21:40:52.970 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.970 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:52.970 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.970 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:118 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'80, mlcod=17'78)", 2024-03-27T21:40:52.970 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:52.970 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 72.836668266000004, 2024-03-27T21:40:52.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014200669999999999, 2024-03-27T21:40:52.971 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:52.971 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.971 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:52.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:52.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:52.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:52.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:52.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:52.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:52.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:52.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088756+0000", 2024-03-27T21:40:52.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4902999999999998e-05 2024-03-27T21:40:52.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:52.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088764+0000", 2024-03-27T21:40:52.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8169999999999999e-06 2024-03-27T21:40:52.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088767+0000", 2024-03-27T21:40:52.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8310000000000003e-06 2024-03-27T21:40:52.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:52.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088798+0000", 2024-03-27T21:40:52.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0800999999999999e-05 2024-03-27T21:40:52.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:52.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088817+0000", 2024-03-27T21:40:52.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8467000000000001e-05 2024-03-27T21:40:52.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.974 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:52.974 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090074+0000", 2024-03-27T21:40:52.974 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012574089999999999 2024-03-27T21:40:52.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.974 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:52.974 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090101+0000", 2024-03-27T21:40:52.974 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7223e-05 2024-03-27T21:40:52.974 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.974 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:52.974 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.975 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:119 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'81, mlcod=17'78)", 2024-03-27T21:40:52.975 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:52.975 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 72.835933471000004, 2024-03-27T21:40:52.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014990870000000001, 2024-03-27T21:40:52.975 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:52.975 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.975 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:52.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:52.975 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:52.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:52.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:52.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:52.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:52.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:52.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089491+0000", 2024-03-27T21:40:52.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4805e-05 2024-03-27T21:40:52.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:52.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089498+0000", 2024-03-27T21:40:52.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0650000000000001e-06 2024-03-27T21:40:52.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089501+0000", 2024-03-27T21:40:52.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.54e-06 2024-03-27T21:40:52.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:52.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089533+0000", 2024-03-27T21:40:52.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1838999999999998e-05 2024-03-27T21:40:52.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:52.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089550+0000", 2024-03-27T21:40:52.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7465999999999998e-05 2024-03-27T21:40:52.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:52.979 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090890+0000", 2024-03-27T21:40:52.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013397979999999999 2024-03-27T21:40:52.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.979 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:52.979 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090915+0000", 2024-03-27T21:40:52.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4905e-05 2024-03-27T21:40:52.979 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.979 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:52.979 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.980 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:120 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'82, mlcod=17'78)", 2024-03-27T21:40:52.980 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:52.980 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 72.835133591000002, 2024-03-27T21:40:52.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001527093, 2024-03-27T21:40:52.980 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:52.980 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.980 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:52.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.980 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:52.980 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:52.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:52.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:52.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:52.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:52.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:52.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090267+0000", 2024-03-27T21:40:52.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1477000000000002e-05 2024-03-27T21:40:52.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:52.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090273+0000", 2024-03-27T21:40:52.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9519999999999999e-06 2024-03-27T21:40:52.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090276+0000", 2024-03-27T21:40:52.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.219e-06 2024-03-27T21:40:52.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:52.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090304+0000", 2024-03-27T21:40:52.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8073999999999999e-05 2024-03-27T21:40:52.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:52.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090316+0000", 2024-03-27T21:40:52.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2088e-05 2024-03-27T21:40:52.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:52.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091717+0000", 2024-03-27T21:40:52.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014003889999999999 2024-03-27T21:40:52.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:52.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091743+0000", 2024-03-27T21:40:52.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6234e-05 2024-03-27T21:40:52.984 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.984 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:52.984 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.984 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:121 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'83, mlcod=17'78)", 2024-03-27T21:40:52.984 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:52.984 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 72.834323225999995, 2024-03-27T21:40:52.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010226899999999999, 2024-03-27T21:40:52.984 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:52.984 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.984 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:52.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:52.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:52.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:52.985 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:52.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:52.985 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:52.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:52.985 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091142+0000", 2024-03-27T21:40:52.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011608300000000001 2024-03-27T21:40:52.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.986 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:52.986 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091151+0000", 2024-03-27T21:40:52.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2350000000000005e-06 2024-03-27T21:40:52.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.986 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.986 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091156+0000", 2024-03-27T21:40:52.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5419999999999999e-06 2024-03-27T21:40:52.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.986 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:52.987 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091191+0000", 2024-03-27T21:40:52.987 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5453000000000001e-05 2024-03-27T21:40:52.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.987 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:52.987 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091208+0000", 2024-03-27T21:40:52.987 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7119999999999999e-05 2024-03-27T21:40:52.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.987 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:52.987 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092024+0000", 2024-03-27T21:40:52.987 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000815539 2024-03-27T21:40:52.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.987 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:52.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092049+0000", 2024-03-27T21:40:52.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5171e-05 2024-03-27T21:40:52.988 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.988 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:52.988 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.988 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:142 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'100, mlcod=18'98)", 2024-03-27T21:40:52.988 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:52.989 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 72.419316272000003, 2024-03-27T21:40:52.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001198778, 2024-03-27T21:40:52.989 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:52.989 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.989 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:52.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:52.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:52.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:52.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:52.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:52.990 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:52.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:52.990 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506101+0000", 2024-03-27T21:40:52.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8202999999999997e-05 2024-03-27T21:40:52.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:52.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506109+0000", 2024-03-27T21:40:52.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3059999999999998e-06 2024-03-27T21:40:52.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.991 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506113+0000", 2024-03-27T21:40:52.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0119999999999999e-06 2024-03-27T21:40:52.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.991 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:52.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506137+0000", 2024-03-27T21:40:52.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4043999999999999e-05 2024-03-27T21:40:52.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.992 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:52.992 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506148+0000", 2024-03-27T21:40:52.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0446e-05 2024-03-27T21:40:52.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.992 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:52.992 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507213+0000", 2024-03-27T21:40:52.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001065215 2024-03-27T21:40:52.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.992 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:52.992 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507232+0000", 2024-03-27T21:40:52.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9038e-05 2024-03-27T21:40:52.993 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.993 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:52.993 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.993 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:143 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'101, mlcod=18'98)", 2024-03-27T21:40:52.993 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:52.993 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 72.418703652999994, 2024-03-27T21:40:52.993 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001357488, 2024-03-27T21:40:52.993 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:52.993 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.993 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:52.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.993 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:52.993 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:52.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:52.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:52.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:52.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:52.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:52.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506723+0000", 2024-03-27T21:40:52.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7404000000000006e-05 2024-03-27T21:40:52.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:52.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506730+0000", 2024-03-27T21:40:52.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5749999999999997e-06 2024-03-27T21:40:52.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:52.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506733+0000", 2024-03-27T21:40:52.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7979999999999999e-06 2024-03-27T21:40:52.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:52.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506758+0000", 2024-03-27T21:40:52.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4179e-05 2024-03-27T21:40:52.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:52.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506768+0000", 2024-03-27T21:40:52.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0740000000000001e-05 2024-03-27T21:40:52.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:52.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507978+0000", 2024-03-27T21:40:52.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012093900000000001 2024-03-27T21:40:52.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.997 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:52.997 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508003+0000", 2024-03-27T21:40:52.997 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5667e-05 2024-03-27T21:40:52.997 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.997 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:52.997 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:52.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.997 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:144 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'102, mlcod=18'98)", 2024-03-27T21:40:52.997 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:52.997 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 72.418044756, 2024-03-27T21:40:52.997 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013453390000000001, 2024-03-27T21:40:52.997 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:52.998 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:52.998 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:52.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.998 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:52.998 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:52.998 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.998 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:52.998 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:52.998 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:52.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.998 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:52.999 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507304+0000", 2024-03-27T21:40:52.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:52.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.999 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:52.999 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507397+0000", 2024-03-27T21:40:52.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2995e-05 2024-03-27T21:40:52.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:52.999 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:52.999 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507403+0000", 2024-03-27T21:40:52.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8640000000000001e-06 2024-03-27T21:40:52.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:52.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:53.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:53.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507407+0000", 2024-03-27T21:40:53.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5710000000000002e-06 2024-03-27T21:40:53.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:53.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:53.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:53.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507428+0000", 2024-03-27T21:40:53.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1192e-05 2024-03-27T21:40:53.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:53.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:53.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:53.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507438+0000", 2024-03-27T21:40:53.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8309999999999996e-06 2024-03-27T21:40:53.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:53.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:53.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:53.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508624+0000", 2024-03-27T21:40:53.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011861210000000001 2024-03-27T21:40:53.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:53.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:53.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:53.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508650+0000", 2024-03-27T21:40:53.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6118e-05 2024-03-27T21:40:53.001 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:53.001 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:53.001 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:53.001 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:53.001 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:53.001 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:53.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.931+0000 7f8bf17fa640 1 -- 172.21.15.60:0/2605805512 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v47) v1 ==== 87+0+0 (secure 0 0 0) 0x7f8be0057eb0 con 0x7f8bf413b020 2024-03-27T21:40:53.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.931+0000 7f8bf17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:40:53.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.931+0000 7f8bf17fa640 10 monclient: _finish_command 2 = system:0 noscrub is set 2024-03-27T21:40:53.004 INFO:teuthology.orchestra.run.smithi060.stderr:noscrub is set 2024-03-27T21:40:53.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.934+0000 7f8bfaee0640 10 librados: watch_flush enter 2024-03-27T21:40:53.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.934+0000 7f8bfaee0640 10 librados: watch_flush exit 2024-03-27T21:40:53.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.934+0000 7f8bfaee0640 20 client.4779.objecter shutdown clearing up homeless session... 2024-03-27T21:40:53.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.934+0000 7f8bfaee0640 10 client.4779.objecter successfully canceled tick 2024-03-27T21:40:53.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.934+0000 7f8bfaee0640 1 -- 172.21.15.60:0/2605805512 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f8bd806c590 msgr2=0x7f8bd806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:53.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.934+0000 7f8bfaee0640 1 --2- 172.21.15.60:0/2605805512 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f8bd806c590 0x7f8bd806ea50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8be8000f60 tx=0x7f8be8009040 comp rx=0 tx=0).stop 2024-03-27T21:40:53.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.934+0000 7f8bfaee0640 10 monclient: shutdown 2024-03-27T21:40:53.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.934+0000 7f8bfaee0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:53.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.934+0000 7f8bfaee0640 1 -- 172.21.15.60:0/2605805512 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bf413b020 msgr2=0x7f8bf41d9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:53.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.934+0000 7f8bfaee0640 1 --2- 172.21.15.60:0/2605805512 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bf413b020 0x7f8bf41d9610 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8be000e9d0 tx=0x7f8be000ee90 comp rx=0 tx=0).stop 2024-03-27T21:40:53.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.934+0000 7f8bfaee0640 1 -- 172.21.15.60:0/2605805512 shutdown_connections 2024-03-27T21:40:53.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.934+0000 7f8bfaee0640 1 --2- 172.21.15.60:0/2605805512 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8bf41416f0 0x7f8bf41361f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:53.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.934+0000 7f8bfaee0640 1 --2- 172.21.15.60:0/2605805512 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bf413b020 0x7f8bf41d9610 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:53.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.934+0000 7f8bfaee0640 1 --2- 172.21.15.60:0/2605805512 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f8bd806c590 0x7f8bd806ea50 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:53.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.934+0000 7f8bfaee0640 1 --2- 172.21.15.60:0/2605805512 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bf40b40b0 0x7f8bf413d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:53.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.934+0000 7f8bfaee0640 1 -- 172.21.15.60:0/2605805512 >> 172.21.15.60:0/2605805512 conn(0x7f8bf40608e0 msgr2=0x7f8bf413f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:53.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.934+0000 7f8bfaee0640 1 -- 172.21.15.60:0/2605805512 shutdown_connections 2024-03-27T21:40:53.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.934+0000 7f8bfaee0640 1 -- 172.21.15.60:0/2605805512 wait complete. 2024-03-27T21:40:53.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:52.934+0000 7f8bfaee0640 1 librados: shutdown 2024-03-27T21:40:53.006 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:40:53.007 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:53.017+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:53.079 INFO:teuthology.orchestra.run.smithi060.stdout: marking collection for removal 2024-03-27T21:40:53.080 INFO:teuthology.orchestra.run.smithi060.stdout:setting '_remove' omap key 2024-03-27T21:40:53.080 INFO:teuthology.orchestra.run.smithi060.stdout:finish_remove_pgs 2.3_head removing 2.3 2024-03-27T21:40:53.086 INFO:teuthology.orchestra.run.smithi060.stdout:Remove successful 2024-03-27T21:40:53.104 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:53.118+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:53.158 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:53.158 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:53.158 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:53.159 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:53.167 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:40:53.205 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:53.219+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:53.306 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:53.319+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:53.316 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:53.316 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:53.316 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:53.316 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:53.316 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:53.325 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:40:53.407 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:53.421+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:53.441 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:53.441 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:53.441 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:53.442 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:40:53.442 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:53.451 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:40:53.508 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:53.522+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:53.559 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:53.559 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:53.559 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:53.559 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:53.568 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:40:53.599 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.3.2 2024-03-27T21:40:53.641 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:53.655+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:53.710 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:53.723+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:53.764 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:53.765 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:53.765 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:53.765 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:53.765 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:53.774 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:40:53.811 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:53.824+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:53.884 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:53.884 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:53.884 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:53.884 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:40:53.884 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:53.893 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:40:53.937 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:53.951+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:54.009 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:54.009 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:54.009 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:54.009 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:54.011 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:54.025+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:54.018 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:40:54.112 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:54.126+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:54.164 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:54.164 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:54.164 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:54.164 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:54.165 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:54.173 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:40:54.213 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:54.227+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:54.275 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:40:54.314 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:54.328+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:54.318 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:54.318 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:54.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:54.318 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:54.318 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:54.327 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:40:54.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.389+0000 7f4ff1d99640 10 monclient: get_monmap_and_config 2024-03-27T21:40:54.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.389+0000 7f4ff1d99640 10 monclient: build_initial_monmap 2024-03-27T21:40:54.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.389+0000 7f4ff1d99640 10 monclient: monmap: 2024-03-27T21:40:54.376 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:54.376 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:54.376 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:54.390913+0000 2024-03-27T21:40:54.376 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:54.390913+0000 2024-03-27T21:40:54.376 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:54.376 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:54.376 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:54.376 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:54.376 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:54.377 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:54.377 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:54.377 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:54.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.390+0000 7f4ff1d99640 1 Processor -- start 2024-03-27T21:40:54.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.390+0000 7f4ff1d99640 1 -- start start 2024-03-27T21:40:54.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.390+0000 7f4ff1d99640 10 monclient: init 2024-03-27T21:40:54.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4ff1d99640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:54.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4ff1d99640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-03-27T21:40:54.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4ff1d99640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4fec0b40b0 0x7f4fec0a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:54.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4ff1d99640 10 monclient(hunting): picked mon.noname-c con 0x7f4fec0b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:54.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4ff1d99640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4fec13b020 0x7f4fec13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:54.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4ff1d99640 10 monclient(hunting): picked mon.noname-a con 0x7f4fec13b020 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:54.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4ff1d99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4fec1416f0 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:54.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4ff1d99640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fec0b3830 con 0x7f4fec1416f0 2024-03-27T21:40:54.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4ff1d99640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:54.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4ff1d99640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fec0b4490 con 0x7f4fec13b020 2024-03-27T21:40:54.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4ff1d99640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:54.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4ff1d99640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fec0aa520 con 0x7f4fec0b40b0 2024-03-27T21:40:54.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4ff1d99640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:54.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4ff1d99640 10 monclient(hunting): authenticate will time out at 895.612289s 2024-03-27T21:40:54.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feaffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4fec13b020 0x7f4fec13b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:54.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feaffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4fec13b020 0x7f4fec13b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47138/0 (socket says 172.21.15.60:47138) 2024-03-27T21:40:54.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feaffd640 1 -- 172.21.15.60:0/3785407859 learned_addr learned my addr 172.21.15.60:0/3785407859 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:54.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feaffd640 10 monclient(hunting): get_auth_request con 0x7f4fec13b020 auth_method 0 2024-03-27T21:40:54.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:54.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feaffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:54.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:54.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feb7fe640 1 --2- 172.21.15.60:0/3785407859 >> v2:172.21.15.191:3300/0 conn(0x7f4fec0b40b0 0x7f4fec0a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:54.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:54.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:54.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:54.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feb7fe640 10 monclient(hunting): get_auth_request con 0x7f4fec0b40b0 auth_method 0 2024-03-27T21:40:54.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:54.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:54.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:54.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:54.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:54.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:54.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feaffd640 10 monclient(hunting): handle_auth_done global_id 4791 payload 274 2024-03-27T21:40:54.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feaffd640 1 -- 172.21.15.60:0/3785407859 >> v2:172.21.15.191:3300/0 conn(0x7f4fec0b40b0 msgr2=0x7f4fec0a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:54.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feaffd640 1 --2- 172.21.15.60:0/3785407859 >> v2:172.21.15.191:3300/0 conn(0x7f4fec0b40b0 0x7f4fec0a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:54.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feaffd640 1 -- 172.21.15.60:0/3785407859 >> v1:172.21.15.191:6789/0 conn(0x7f4fec1416f0 legacy=0x7f4fec141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:54.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feaffd640 10 monclient: _finish_hunting 0 2024-03-27T21:40:54.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feaffd640 1 monclient: found mon.noname-a 2024-03-27T21:40:54.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:54.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:54.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.391+0000 7f4feaffd640 1 -- 172.21.15.60:0/3785407859 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fec0a9470 con 0x7f4fec13b020 2024-03-27T21:40:54.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.392+0000 7f4feaffd640 1 --2- 172.21.15.60:0/3785407859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fec13b020 0x7f4fec13b480 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0b4810 tx=0x7f4fe002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b9281dc710c0dc server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:54.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.392+0000 7f4fea7fc640 1 -- 172.21.15.60:0/3785407859 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe0030860 con 0x7f4fec13b020 2024-03-27T21:40:54.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.392+0000 7f4fea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:54.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.392+0000 7f4fea7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:54.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.392+0000 7f4fea7fc640 10 monclient: dump: 2024-03-27T21:40:54.390 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:54.390 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:54.390 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:54.390 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:54.391 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:54.391 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:54.391 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:54.391 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:54.391 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:54.391 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:54.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.392+0000 7f4fea7fc640 10 monclient: _finish_auth 0 2024-03-27T21:40:54.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.392+0000 7f4fea7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:40:54.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.392+0000 7f4fea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:54.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.392+0000 7f4fea7fc640 1 -- 172.21.15.60:0/3785407859 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe00309c0 con 0x7f4fec13b020 2024-03-27T21:40:54.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.392+0000 7f4ff1d99640 5 monclient: authenticate success, global_id 4791 2024-03-27T21:40:54.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.392+0000 7f4ff1d99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:54.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.392+0000 7f4fea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:54.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.392+0000 7f4fea7fc640 1 -- 172.21.15.60:0/3785407859 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe0030d50 con 0x7f4fec13b020 2024-03-27T21:40:54.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.392+0000 7f4ff1d99640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:54.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.392+0000 7f4ff1d99640 4 set_mon_vals no callback set 2024-03-27T21:40:54.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.392+0000 7f4fea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:54.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.392+0000 7f4fea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:54.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.392+0000 7f4fea7fc640 10 monclient: dump: 2024-03-27T21:40:54.392 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:54.392 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:54.392 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:54.392 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:54.392 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:54.392 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:54.392 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:54.392 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:54.392 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:54.393 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:54.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.393+0000 7f4ff1d99640 10 monclient: shutdown 2024-03-27T21:40:54.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.393+0000 7f4ff1d99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:54.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.393+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3785407859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fec13b020 msgr2=0x7f4fec13b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:54.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.393+0000 7f4ff1d99640 1 --2- 172.21.15.60:0/3785407859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fec13b020 0x7f4fec13b480 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0b4810 tx=0x7f4fe002b9a0 comp rx=0 tx=0).stop 2024-03-27T21:40:54.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.393+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3785407859 shutdown_connections 2024-03-27T21:40:54.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.393+0000 7f4ff1d99640 1 --2- 172.21.15.60:0/3785407859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fec13b020 0x7f4fec13b480 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:54.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.393+0000 7f4ff1d99640 1 --2- 172.21.15.60:0/3785407859 >> v2:172.21.15.191:3300/0 conn(0x7f4fec0b40b0 0x7f4fec0a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:54.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.393+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3785407859 >> 172.21.15.60:0/3785407859 conn(0x7f4fec0608e0 msgr2=0x7f4fec060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:54.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.393+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3785407859 shutdown_connections 2024-03-27T21:40:54.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.393+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3785407859 wait complete. 2024-03-27T21:40:54.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.394+0000 7f4ff1d99640 10 monclient: build_initial_monmap 2024-03-27T21:40:54.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.394+0000 7f4ff1d99640 10 monclient: monmap: 2024-03-27T21:40:54.393 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:54.394 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:54.394 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:54.394 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:54.394 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:54.394 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:54.394 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:54.394 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:54.394 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:54.394 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:54.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.394+0000 7f4ff1d99640 1 librados: starting msgr at 2024-03-27T21:40:54.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.394+0000 7f4ff1d99640 1 librados: starting objecter 2024-03-27T21:40:54.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.394+0000 7f4ff1d99640 1 Processor -- start 2024-03-27T21:40:54.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.394+0000 7f4ff1d99640 1 -- start start 2024-03-27T21:40:54.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.394+0000 7f4ff1d99640 1 librados: setting wanted keys 2024-03-27T21:40:54.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.394+0000 7f4ff1d99640 1 librados: calling monclient init 2024-03-27T21:40:54.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.394+0000 7f4ff1d99640 10 monclient: init 2024-03-27T21:40:54.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4ff1d99640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:54.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4ff1d99640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:40:54.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4ff1d99640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fec0b40b0 0x7f4fec13a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:54.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4ff1d99640 10 monclient(hunting): picked mon.noname-c con 0x7f4fec0b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:54.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4ff1d99640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fec13b020 0x7f4fec1d9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:54.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4ff1d99640 10 monclient(hunting): picked mon.noname-a con 0x7f4fec13b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:54.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4ff1d99640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fec1416f0 0x7f4fec1331c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:54.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4ff1d99640 10 monclient(hunting): picked mon.noname-b con 0x7f4fec1416f0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:54.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4ff1d99640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:54.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4ff1d99640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec1da6c0 con 0x7f4fec13b020 2024-03-27T21:40:54.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4ff1d99640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:54.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4ff1d99640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec13abe0 con 0x7f4fec1416f0 2024-03-27T21:40:54.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4ff1d99640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:54.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4ff1d99640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec1da2c0 con 0x7f4fec0b40b0 2024-03-27T21:40:54.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4ff1d99640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:54.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4ff1d99640 10 monclient(hunting): authenticate will time out at 895.616198s 2024-03-27T21:40:54.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feaffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fec13b020 0x7f4fec1d9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:54.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feb7fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fec0b40b0 0x7f4fec13a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:54.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feaffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fec13b020 0x7f4fec1d9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47152/0 (socket says 172.21.15.60:47152) 2024-03-27T21:40:54.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feb7fe640 10 monclient(hunting): get_auth_request con 0x7f4fec0b40b0 auth_method 0 2024-03-27T21:40:54.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feaffd640 1 -- 172.21.15.60:0/3495346115 learned_addr learned my addr 172.21.15.60:0/3495346115 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:54.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:54.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:54.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:54.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feaffd640 10 monclient(hunting): get_auth_request con 0x7f4fec13b020 auth_method 0 2024-03-27T21:40:54.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:54.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feaffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:54.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:54.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:54.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:54.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4febfff640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fec1416f0 0x7f4fec1331c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:54.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:54.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:54.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:54.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:54.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feb7fe640 10 monclient(hunting): handle_auth_done global_id 4634 payload 824 2024-03-27T21:40:54.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feb7fe640 1 -- 172.21.15.60:0/3495346115 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fec1416f0 msgr2=0x7f4fec1331c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:54.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feb7fe640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fec1416f0 0x7f4fec1331c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:54.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feb7fe640 1 -- 172.21.15.60:0/3495346115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fec13b020 msgr2=0x7f4fec1d9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:54.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feb7fe640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fec13b020 0x7f4fec1d9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:54.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feb7fe640 10 monclient: _finish_hunting 0 2024-03-27T21:40:54.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feb7fe640 1 monclient: found mon.noname-c 2024-03-27T21:40:54.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:54.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:40:54.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feb7fe640 1 -- 172.21.15.60:0/3495346115 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe00094d0 con 0x7f4fec0b40b0 2024-03-27T21:40:54.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4febfff640 10 monclient: get_auth_request con 0x7f4fec1416f0 auth_method 0 2024-03-27T21:40:54.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feaffd640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fec13b020 0x7f4fec1d9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:54.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4febfff640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fec1416f0 0x7f4fec1331c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:40:54.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4feb7fe640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fec0b40b0 0x7f4fec13a0a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc00e880 tx=0x7f4fdc00ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:54.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4fd3fff640 10 client.?.objecter ms_handle_connect 0x7f4fec0b40b0 2024-03-27T21:40:54.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.395+0000 7f4fd3fff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:54.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4fd3fff640 1 -- 172.21.15.60:0/3495346115 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc00c940 con 0x7f4fec0b40b0 2024-03-27T21:40:54.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4fd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:54.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4fd3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:54.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4fd3fff640 10 monclient: dump: 2024-03-27T21:40:54.399 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:54.399 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:54.399 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:54.399 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:54.399 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:54.399 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:54.399 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:54.399 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:54.400 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:54.400 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:54.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4fd3fff640 10 monclient: _finish_auth 0 2024-03-27T21:40:54.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4fd3fff640 10 monclient: _check_auth_tickets 2024-03-27T21:40:54.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4fd3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:54.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4fd3fff640 1 -- 172.21.15.60:0/3495346115 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fdc00caa0 con 0x7f4fec0b40b0 2024-03-27T21:40:54.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4ff1d99640 5 monclient: authenticate success, global_id 4634 2024-03-27T21:40:54.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4ff1d99640 10 monclient: _renew_subs 2024-03-27T21:40:54.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4ff1d99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:54.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3495346115 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fec13d150 con 0x7f4fec0b40b0 2024-03-27T21:40:54.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4fd3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:54.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4fd3fff640 1 -- 172.21.15.60:0/3495346115 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc00ce50 con 0x7f4fec0b40b0 2024-03-27T21:40:54.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4fd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:54.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4fd3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:54.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4fd3fff640 10 monclient: dump: 2024-03-27T21:40:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:54.401 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4ff1d99640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4ff1d99640 10 monclient: _renew_subs 2024-03-27T21:40:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4ff1d99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4fe9ffb640 4 set_mon_vals no callback set 2024-03-27T21:40:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3495346115 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fec1338e0 con 0x7f4fec0b40b0 2024-03-27T21:40:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4ff1d99640 1 librados: init done 2024-03-27T21:40:54.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4ff1d99640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:54.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4ff1d99640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:54.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4ff1d99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:54.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.396+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3495346115 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4fec069470 con 0x7f4fec0b40b0 2024-03-27T21:40:54.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.397+0000 7f4fd3fff640 1 -- 172.21.15.60:0/3495346115 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4fdc013470 con 0x7f4fec0b40b0 2024-03-27T21:40:54.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4fd3fff640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4fd806c590 0x7f4fd806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:54.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4feaffd640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4fd806c590 0x7f4fd806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:54.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4fd3fff640 1 -- 172.21.15.60:0/3495346115 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f4fdc089220 con 0x7f4fec0b40b0 2024-03-27T21:40:54.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4fd3fff640 10 client.4634.objecter ms_dispatch 0x7f4fec000b90 osd_map(47..47 src has 1..47) v4 2024-03-27T21:40:54.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4fd3fff640 3 client.4634.objecter handle_osd_map got epochs [47,47] > 0 2024-03-27T21:40:54.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4fd3fff640 3 client.4634.objecter handle_osd_map decoding full epoch 47 2024-03-27T21:40:54.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4feaffd640 10 monclient: get_auth_request con 0x7f4fd806c590 auth_method 0 2024-03-27T21:40:54.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4fd3fff640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-27T21:40:54.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4fd3fff640 1 -- 172.21.15.60:0/3495346115 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v47) v1 ==== 54+0+16 (secure 0 0 0) 0x7f4fdc090070 con 0x7f4fec0b40b0 2024-03-27T21:40:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4fd3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4fd3fff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4ff1d99640 10 client.4634.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign -1 1 2024-03-27T21:40:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4ff1d99640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f4fec1d69d0 0x7f4fec1d6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec142250 osd=0 3 2024-03-27T21:40:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec142250 osd=0 3 2024-03-27T21:40:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec142250 osd=0 3 2024-03-27T21:40:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_remove -1 1 2024-03-27T21:40:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4ff1d99640 20 client.4634.objecter get_session s=0x7f4fec142250 osd=0 3 2024-03-27T21:40:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign 0 1 2024-03-27T21:40:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec142250 osd=0 4 2024-03-27T21:40:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4ff1d99640 10 client.4634.objecter _send_command 1 2024-03-27T21:40:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4fd3fff640 10 client.4634.objecter ms_handle_connect 0x7f4fd806c590 2024-03-27T21:40:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3495346115 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4fec0b6d70 con 0x7f4fec1d69d0 2024-03-27T21:40:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4feaffd640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4fd806c590 0x7f4fd806ea50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0b0f20 tx=0x7f4fe0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4febfff640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f4fec1d69d0 0x7f4fec1d6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.398+0000 7f4febfff640 10 monclient: get_auth_request con 0x7f4fec1d69d0 auth_method 0 2024-03-27T21:40:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.399+0000 7f4fd3fff640 10 client.4634.objecter ms_handle_connect 0x7f4fec1d69d0 2024-03-27T21:40:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.399+0000 7f4febfff640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f4fec1d69d0 0x7f4fec1d6db0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.399+0000 7f4fd3fff640 1 -- 172.21.15.60:0/3495346115 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4fec0b6d70 con 0x7f4fec1d69d0 2024-03-27T21:40:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.399+0000 7f4fd3fff640 10 client.4634.objecter ms_dispatch 0x7f4fec000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.399+0000 7f4fd3fff640 10 client.4634.objecter _finish_command 1 = system:0 2024-03-27T21:40:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.399+0000 7f4fd3fff640 20 client.4634.objecter put_session s=0x7f4fec142250 osd=0 4 2024-03-27T21:40:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.399+0000 7f4fd3fff640 15 client.4634.objecter _session_command_op_remove 0 1 2024-03-27T21:40:54.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.415+0000 7f4ff1d99640 10 client.4634.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:54.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.415+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign -1 2 2024-03-27T21:40:54.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.415+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec142250 osd=0 3 2024-03-27T21:40:54.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.415+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec142250 osd=0 3 2024-03-27T21:40:54.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.415+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec142250 osd=0 3 2024-03-27T21:40:54.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.415+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_remove -1 2 2024-03-27T21:40:54.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.415+0000 7f4ff1d99640 20 client.4634.objecter get_session s=0x7f4fec142250 osd=0 3 2024-03-27T21:40:54.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.415+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign 0 2 2024-03-27T21:40:54.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.415+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec142250 osd=0 4 2024-03-27T21:40:54.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.415+0000 7f4ff1d99640 10 client.4634.objecter _send_command 2 2024-03-27T21:40:54.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.415+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3495346115 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4fec133b30 con 0x7f4fec1d69d0 2024-03-27T21:40:54.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.417+0000 7f4fd3fff640 1 -- 172.21.15.60:0/3495346115 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4fec133b30 con 0x7f4fec1d69d0 2024-03-27T21:40:54.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.417+0000 7f4fd3fff640 10 client.4634.objecter ms_dispatch 0x7f4fec000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:54.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.417+0000 7f4fd3fff640 10 client.4634.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:54.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.417+0000 7f4fd3fff640 20 client.4634.objecter put_session s=0x7f4fec142250 osd=0 4 2024-03-27T21:40:54.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.417+0000 7f4fd3fff640 15 client.4634.objecter _session_command_op_remove 0 2 2024-03-27T21:40:54.406 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:54.408 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:54.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.417+0000 7f4ff1d99640 10 client.4634.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:54.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.417+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign -1 3 2024-03-27T21:40:54.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.417+0000 7f4ff1d99640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f4fec140f80 0x7f4fec0b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:54.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.417+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec140e10 osd=1 3 2024-03-27T21:40:54.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.417+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec140e10 osd=1 3 2024-03-27T21:40:54.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.417+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec140e10 osd=1 3 2024-03-27T21:40:54.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.417+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_remove -1 3 2024-03-27T21:40:54.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.417+0000 7f4ff1d99640 20 client.4634.objecter get_session s=0x7f4fec140e10 osd=1 3 2024-03-27T21:40:54.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.417+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign 1 3 2024-03-27T21:40:54.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.417+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec140e10 osd=1 4 2024-03-27T21:40:54.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.417+0000 7f4ff1d99640 10 client.4634.objecter _send_command 3 2024-03-27T21:40:54.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.417+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3495346115 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4fec13c730 con 0x7f4fec140f80 2024-03-27T21:40:54.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.417+0000 7f4feb7fe640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f4fec140f80 0x7f4fec0b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:54.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.417+0000 7f4feb7fe640 10 monclient: get_auth_request con 0x7f4fec140f80 auth_method 0 2024-03-27T21:40:54.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.417+0000 7f4fd3fff640 10 client.4634.objecter ms_handle_connect 0x7f4fec140f80 2024-03-27T21:40:54.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.417+0000 7f4feb7fe640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f4fec140f80 0x7f4fec0b4bc0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:54.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.418+0000 7f4fd3fff640 1 -- 172.21.15.60:0/3495346115 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4fec13c730 con 0x7f4fec140f80 2024-03-27T21:40:54.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.418+0000 7f4fd3fff640 10 client.4634.objecter ms_dispatch 0x7f4fec000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:54.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.418+0000 7f4fd3fff640 10 client.4634.objecter _finish_command 3 = system:0 2024-03-27T21:40:54.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.418+0000 7f4fd3fff640 20 client.4634.objecter put_session s=0x7f4fec140e10 osd=1 4 2024-03-27T21:40:54.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.418+0000 7f4fd3fff640 15 client.4634.objecter _session_command_op_remove 1 3 2024-03-27T21:40:54.414 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:54.428+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:54.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.432+0000 7f4ff1d99640 10 client.4634.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:54.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.432+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign -1 4 2024-03-27T21:40:54.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.432+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec140e10 osd=1 3 2024-03-27T21:40:54.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.432+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec140e10 osd=1 3 2024-03-27T21:40:54.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.432+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec140e10 osd=1 3 2024-03-27T21:40:54.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.432+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_remove -1 4 2024-03-27T21:40:54.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.432+0000 7f4ff1d99640 20 client.4634.objecter get_session s=0x7f4fec140e10 osd=1 3 2024-03-27T21:40:54.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.432+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign 1 4 2024-03-27T21:40:54.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.432+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec140e10 osd=1 4 2024-03-27T21:40:54.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.432+0000 7f4ff1d99640 10 client.4634.objecter _send_command 4 2024-03-27T21:40:54.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.432+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3495346115 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4fec133ca0 con 0x7f4fec140f80 2024-03-27T21:40:54.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4fd3fff640 1 -- 172.21.15.60:0/3495346115 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4fec133ca0 con 0x7f4fec140f80 2024-03-27T21:40:54.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4fd3fff640 10 client.4634.objecter ms_dispatch 0x7f4fec000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:54.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4fd3fff640 10 client.4634.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:54.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4fd3fff640 20 client.4634.objecter put_session s=0x7f4fec140e10 osd=1 4 2024-03-27T21:40:54.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4fd3fff640 15 client.4634.objecter _session_command_op_remove 1 4 2024-03-27T21:40:54.421 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:54.423 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:54.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4ff1d99640 10 client.4634.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:54.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign -1 5 2024-03-27T21:40:54.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4ff1d99640 20 client.4634.objecter _get_session osd=-1 returning homeless 2024-03-27T21:40:54.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4ff1d99640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:54.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4ff1d99640 10 monclient: _renew_subs 2024-03-27T21:40:54.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4ff1d99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:54.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3495346115 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=48}) v3 -- 0x7f4fec13c8a0 con 0x7f4fec0b40b0 2024-03-27T21:40:54.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4ff1d99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:54.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3495346115 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4fec13bee0 con 0x7f4fec0b40b0 2024-03-27T21:40:54.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4fd3fff640 1 -- 172.21.15.60:0/3495346115 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4fdc0890b0 con 0x7f4fec0b40b0 2024-03-27T21:40:54.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4fd3fff640 10 monclient: handle_get_version_reply finishing 1 version 47 2024-03-27T21:40:54.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4fe9ffb640 10 client.4634.objecter _check_command_map_dne tid 5 current 47 map_dne_bound 47 2024-03-27T21:40:54.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4fe9ffb640 10 client.4634.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:40:54.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4fe9ffb640 15 client.4634.objecter _session_command_op_remove -1 5 2024-03-27T21:40:54.424 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:40:54.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4ff1d99640 10 client.4634.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:54.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign -1 6 2024-03-27T21:40:54.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4ff1d99640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f4fec02b680 0x7f4fec02ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:54.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec13c0a0 osd=3 3 2024-03-27T21:40:54.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec13c0a0 osd=3 3 2024-03-27T21:40:54.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec13c0a0 osd=3 3 2024-03-27T21:40:54.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_remove -1 6 2024-03-27T21:40:54.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4ff1d99640 20 client.4634.objecter get_session s=0x7f4fec13c0a0 osd=3 3 2024-03-27T21:40:54.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign 3 6 2024-03-27T21:40:54.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec13c0a0 osd=3 4 2024-03-27T21:40:54.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4ff1d99640 10 client.4634.objecter _send_command 6 2024-03-27T21:40:54.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.434+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3495346115 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4fec0b5b00 con 0x7f4fec02b680 2024-03-27T21:40:54.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.435+0000 7f4feaffd640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f4fec02b680 0x7f4fec02ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:54.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.435+0000 7f4feaffd640 10 monclient: get_auth_request con 0x7f4fec02b680 auth_method 0 2024-03-27T21:40:54.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.435+0000 7f4feaffd640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f4fec02b680 0x7f4fec02ba60 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:54.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.435+0000 7f4fd3fff640 10 client.4634.objecter ms_handle_connect 0x7f4fec02b680 2024-03-27T21:40:54.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.435+0000 7f4fd3fff640 1 -- 172.21.15.60:0/3495346115 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4fec0b5b00 con 0x7f4fec02b680 2024-03-27T21:40:54.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.435+0000 7f4fd3fff640 10 client.4634.objecter ms_dispatch 0x7f4fec000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:40:54.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.435+0000 7f4fd3fff640 10 client.4634.objecter _finish_command 6 = system:0 2024-03-27T21:40:54.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.435+0000 7f4fd3fff640 20 client.4634.objecter put_session s=0x7f4fec13c0a0 osd=3 4 2024-03-27T21:40:54.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.435+0000 7f4fd3fff640 15 client.4634.objecter _session_command_op_remove 3 6 2024-03-27T21:40:54.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.462+0000 7f4ff1d99640 10 client.4634.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.462+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign -1 7 2024-03-27T21:40:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.462+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec13c0a0 osd=3 3 2024-03-27T21:40:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.462+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec13c0a0 osd=3 3 2024-03-27T21:40:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.462+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec13c0a0 osd=3 3 2024-03-27T21:40:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.462+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_remove -1 7 2024-03-27T21:40:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.462+0000 7f4ff1d99640 20 client.4634.objecter get_session s=0x7f4fec13c0a0 osd=3 3 2024-03-27T21:40:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.462+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign 3 7 2024-03-27T21:40:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.462+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec13c0a0 osd=3 4 2024-03-27T21:40:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.462+0000 7f4ff1d99640 10 client.4634.objecter _send_command 7 2024-03-27T21:40:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.462+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3495346115 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4fec0b5580 con 0x7f4fec02b680 2024-03-27T21:40:54.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.464+0000 7f4fd3fff640 1 -- 172.21.15.60:0/3495346115 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4fec0b5580 con 0x7f4fec02b680 2024-03-27T21:40:54.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.464+0000 7f4fd3fff640 10 client.4634.objecter ms_dispatch 0x7f4fec000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:54.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.464+0000 7f4fd3fff640 10 client.4634.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:54.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.464+0000 7f4fd3fff640 20 client.4634.objecter put_session s=0x7f4fec13c0a0 osd=3 4 2024-03-27T21:40:54.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.464+0000 7f4fd3fff640 15 client.4634.objecter _session_command_op_remove 3 7 2024-03-27T21:40:54.451 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:54.451 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.464+0000 7f4ff1d99640 10 client.4634.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.464+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign -1 8 2024-03-27T21:40:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.464+0000 7f4ff1d99640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f4fec1e7130 0x7f4fec1e9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.464+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec0b5930 osd=4 3 2024-03-27T21:40:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.464+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec0b5930 osd=4 3 2024-03-27T21:40:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.464+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec0b5930 osd=4 3 2024-03-27T21:40:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.464+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_remove -1 8 2024-03-27T21:40:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.464+0000 7f4ff1d99640 20 client.4634.objecter get_session s=0x7f4fec0b5930 osd=4 3 2024-03-27T21:40:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.464+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign 4 8 2024-03-27T21:40:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.464+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec0b5930 osd=4 4 2024-03-27T21:40:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.464+0000 7f4ff1d99640 10 client.4634.objecter _send_command 8 2024-03-27T21:40:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.464+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3495346115 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4fec1ec2f0 con 0x7f4fec1e7130 2024-03-27T21:40:54.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.464+0000 7f4febfff640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f4fec1e7130 0x7f4fec1e9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:54.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.464+0000 7f4febfff640 10 monclient: get_auth_request con 0x7f4fec1e7130 auth_method 0 2024-03-27T21:40:54.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.464+0000 7f4fd3fff640 10 client.4634.objecter ms_handle_connect 0x7f4fec1e7130 2024-03-27T21:40:54.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.464+0000 7f4febfff640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f4fec1e7130 0x7f4fec1e9520 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:54.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.465+0000 7f4fd3fff640 1 -- 172.21.15.60:0/3495346115 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4fec1ec2f0 con 0x7f4fec1e7130 2024-03-27T21:40:54.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.465+0000 7f4fd3fff640 10 client.4634.objecter ms_dispatch 0x7f4fec000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:40:54.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.465+0000 7f4fd3fff640 10 client.4634.objecter _finish_command 8 = system:0 2024-03-27T21:40:54.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.465+0000 7f4fd3fff640 20 client.4634.objecter put_session s=0x7f4fec0b5930 osd=4 4 2024-03-27T21:40:54.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.465+0000 7f4fd3fff640 15 client.4634.objecter _session_command_op_remove 4 8 2024-03-27T21:40:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.484+0000 7f4ff1d99640 10 client.4634.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.484+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign -1 9 2024-03-27T21:40:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.484+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec0b5930 osd=4 3 2024-03-27T21:40:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.484+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec0b5930 osd=4 3 2024-03-27T21:40:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.484+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec0b5930 osd=4 3 2024-03-27T21:40:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.484+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_remove -1 9 2024-03-27T21:40:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.484+0000 7f4ff1d99640 20 client.4634.objecter get_session s=0x7f4fec0b5930 osd=4 3 2024-03-27T21:40:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.484+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign 4 9 2024-03-27T21:40:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.484+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec0b5930 osd=4 4 2024-03-27T21:40:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.484+0000 7f4ff1d99640 10 client.4634.objecter _send_command 9 2024-03-27T21:40:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.484+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3495346115 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4fec13d2e0 con 0x7f4fec1e7130 2024-03-27T21:40:54.472 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:54.472 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:54.472 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:54.472 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:54.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.486+0000 7f4fd3fff640 1 -- 172.21.15.60:0/3495346115 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4fec13d2e0 con 0x7f4fec1e7130 2024-03-27T21:40:54.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.486+0000 7f4fd3fff640 10 client.4634.objecter ms_dispatch 0x7f4fec000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:54.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.486+0000 7f4fd3fff640 10 client.4634.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:54.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.486+0000 7f4fd3fff640 20 client.4634.objecter put_session s=0x7f4fec0b5930 osd=4 4 2024-03-27T21:40:54.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.486+0000 7f4fd3fff640 15 client.4634.objecter _session_command_op_remove 4 9 2024-03-27T21:40:54.473 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:54.474 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:54.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.486+0000 7f4ff1d99640 10 client.4634.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:54.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.486+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign -1 10 2024-03-27T21:40:54.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.486+0000 7f4ff1d99640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f4fec066250 0x7f4fec066670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:54.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.486+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec13d2e0 osd=5 3 2024-03-27T21:40:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.486+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec13d2e0 osd=5 3 2024-03-27T21:40:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.486+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec13d2e0 osd=5 3 2024-03-27T21:40:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.486+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_remove -1 10 2024-03-27T21:40:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.486+0000 7f4ff1d99640 20 client.4634.objecter get_session s=0x7f4fec13d2e0 osd=5 3 2024-03-27T21:40:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.486+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign 5 10 2024-03-27T21:40:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.486+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec13d2e0 osd=5 4 2024-03-27T21:40:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.486+0000 7f4ff1d99640 10 client.4634.objecter _send_command 10 2024-03-27T21:40:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.486+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3495346115 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4fec02c6f0 con 0x7f4fec066250 2024-03-27T21:40:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.486+0000 7f4feb7fe640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f4fec066250 0x7f4fec066670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.486+0000 7f4feb7fe640 10 monclient: get_auth_request con 0x7f4fec066250 auth_method 0 2024-03-27T21:40:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.487+0000 7f4fd3fff640 10 client.4634.objecter ms_handle_connect 0x7f4fec066250 2024-03-27T21:40:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.487+0000 7f4feb7fe640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f4fec066250 0x7f4fec066670 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.487+0000 7f4fd3fff640 1 -- 172.21.15.60:0/3495346115 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4fec02c6f0 con 0x7f4fec066250 2024-03-27T21:40:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.487+0000 7f4fd3fff640 10 client.4634.objecter ms_dispatch 0x7f4fec000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:40:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.487+0000 7f4fd3fff640 10 client.4634.objecter _finish_command 10 = system:0 2024-03-27T21:40:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.487+0000 7f4fd3fff640 20 client.4634.objecter put_session s=0x7f4fec13d2e0 osd=5 4 2024-03-27T21:40:54.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.487+0000 7f4fd3fff640 15 client.4634.objecter _session_command_op_remove 5 10 2024-03-27T21:40:54.481 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:40:54.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.503+0000 7f4ff1d99640 10 client.4634.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:54.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.503+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign -1 11 2024-03-27T21:40:54.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.503+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec13d2e0 osd=5 3 2024-03-27T21:40:54.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.503+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec13d2e0 osd=5 3 2024-03-27T21:40:54.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.503+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec13d2e0 osd=5 3 2024-03-27T21:40:54.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.503+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_remove -1 11 2024-03-27T21:40:54.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.503+0000 7f4ff1d99640 20 client.4634.objecter get_session s=0x7f4fec13d2e0 osd=5 3 2024-03-27T21:40:54.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.503+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign 5 11 2024-03-27T21:40:54.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.503+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec13d2e0 osd=5 4 2024-03-27T21:40:54.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.503+0000 7f4ff1d99640 10 client.4634.objecter _send_command 11 2024-03-27T21:40:54.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.503+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3495346115 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4fec02c6f0 con 0x7f4fec066250 2024-03-27T21:40:54.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.504+0000 7f4fd3fff640 1 -- 172.21.15.60:0/3495346115 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4fec02c6f0 con 0x7f4fec066250 2024-03-27T21:40:54.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.504+0000 7f4fd3fff640 10 client.4634.objecter ms_dispatch 0x7f4fec000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:54.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.504+0000 7f4fd3fff640 10 client.4634.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:54.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.504+0000 7f4fd3fff640 20 client.4634.objecter put_session s=0x7f4fec13d2e0 osd=5 4 2024-03-27T21:40:54.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.504+0000 7f4fd3fff640 15 client.4634.objecter _session_command_op_remove 5 11 2024-03-27T21:40:54.491 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:54.492 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:54.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.505+0000 7f4ff1d99640 10 client.4634.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:54.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.505+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign -1 12 2024-03-27T21:40:54.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.505+0000 7f4ff1d99640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f4fec1ed100 0x7f4fec1ef4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:54.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.505+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec02c6f0 osd=6 3 2024-03-27T21:40:54.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.505+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec02c6f0 osd=6 3 2024-03-27T21:40:54.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.505+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec02c6f0 osd=6 3 2024-03-27T21:40:54.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.505+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_remove -1 12 2024-03-27T21:40:54.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.505+0000 7f4ff1d99640 20 client.4634.objecter get_session s=0x7f4fec02c6f0 osd=6 3 2024-03-27T21:40:54.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.505+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign 6 12 2024-03-27T21:40:54.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.505+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec02c6f0 osd=6 4 2024-03-27T21:40:54.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.505+0000 7f4ff1d99640 10 client.4634.objecter _send_command 12 2024-03-27T21:40:54.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.505+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3495346115 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4fec1f23b0 con 0x7f4fec1ed100 2024-03-27T21:40:54.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.505+0000 7f4feaffd640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f4fec1ed100 0x7f4fec1ef4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:54.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.505+0000 7f4feaffd640 10 monclient: get_auth_request con 0x7f4fec1ed100 auth_method 0 2024-03-27T21:40:54.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.505+0000 7f4fd3fff640 10 client.4634.objecter ms_handle_connect 0x7f4fec1ed100 2024-03-27T21:40:54.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.505+0000 7f4feaffd640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f4fec1ed100 0x7f4fec1ef4f0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:54.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.506+0000 7f4fd3fff640 1 -- 172.21.15.60:0/3495346115 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4fec1f23b0 con 0x7f4fec1ed100 2024-03-27T21:40:54.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.506+0000 7f4fd3fff640 10 client.4634.objecter ms_dispatch 0x7f4fec000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:40:54.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.506+0000 7f4fd3fff640 10 client.4634.objecter _finish_command 12 = system:0 2024-03-27T21:40:54.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.506+0000 7f4fd3fff640 20 client.4634.objecter put_session s=0x7f4fec02c6f0 osd=6 4 2024-03-27T21:40:54.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.506+0000 7f4fd3fff640 15 client.4634.objecter _session_command_op_remove 6 12 2024-03-27T21:40:54.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.521+0000 7f4ff1d99640 10 client.4634.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:54.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.521+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign -1 13 2024-03-27T21:40:54.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.521+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec02c6f0 osd=6 3 2024-03-27T21:40:54.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.521+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec02c6f0 osd=6 3 2024-03-27T21:40:54.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.521+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec02c6f0 osd=6 3 2024-03-27T21:40:54.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.521+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_remove -1 13 2024-03-27T21:40:54.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.521+0000 7f4ff1d99640 20 client.4634.objecter get_session s=0x7f4fec02c6f0 osd=6 3 2024-03-27T21:40:54.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.521+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign 6 13 2024-03-27T21:40:54.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.521+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec02c6f0 osd=6 4 2024-03-27T21:40:54.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.521+0000 7f4ff1d99640 10 client.4634.objecter _send_command 13 2024-03-27T21:40:54.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.521+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3495346115 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4fec1f23b0 con 0x7f4fec1ed100 2024-03-27T21:40:54.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.523+0000 7f4fd3fff640 1 -- 172.21.15.60:0/3495346115 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4fec1f23b0 con 0x7f4fec1ed100 2024-03-27T21:40:54.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.523+0000 7f4fd3fff640 10 client.4634.objecter ms_dispatch 0x7f4fec000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:54.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.523+0000 7f4fd3fff640 10 client.4634.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:54.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.523+0000 7f4fd3fff640 20 client.4634.objecter put_session s=0x7f4fec02c6f0 osd=6 4 2024-03-27T21:40:54.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.523+0000 7f4fd3fff640 15 client.4634.objecter _session_command_op_remove 6 13 2024-03-27T21:40:54.509 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:54.511 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:54.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.523+0000 7f4ff1d99640 10 client.4634.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:54.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.523+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign -1 14 2024-03-27T21:40:54.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.523+0000 7f4ff1d99640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f4fec1f1720 0x7f4fec1f1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:54.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.523+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec1f23b0 osd=7 3 2024-03-27T21:40:54.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.523+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec1f23b0 osd=7 3 2024-03-27T21:40:54.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.523+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec1f23b0 osd=7 3 2024-03-27T21:40:54.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.523+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_remove -1 14 2024-03-27T21:40:54.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.523+0000 7f4ff1d99640 20 client.4634.objecter get_session s=0x7f4fec1f23b0 osd=7 3 2024-03-27T21:40:54.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.523+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign 7 14 2024-03-27T21:40:54.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.523+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec1f23b0 osd=7 4 2024-03-27T21:40:54.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.523+0000 7f4ff1d99640 10 client.4634.objecter _send_command 14 2024-03-27T21:40:54.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.523+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3495346115 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4fec1f5740 con 0x7f4fec1f1720 2024-03-27T21:40:54.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.523+0000 7f4febfff640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f4fec1f1720 0x7f4fec1f1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:54.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.523+0000 7f4febfff640 10 monclient: get_auth_request con 0x7f4fec1f1720 auth_method 0 2024-03-27T21:40:54.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.524+0000 7f4fd3fff640 10 client.4634.objecter ms_handle_connect 0x7f4fec1f1720 2024-03-27T21:40:54.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.524+0000 7f4febfff640 1 --2- 172.21.15.60:0/3495346115 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f4fec1f1720 0x7f4fec1f1b80 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:54.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.525+0000 7f4fd3fff640 1 -- 172.21.15.60:0/3495346115 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4fec1f5740 con 0x7f4fec1f1720 2024-03-27T21:40:54.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.525+0000 7f4fd3fff640 10 client.4634.objecter ms_dispatch 0x7f4fec000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:40:54.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.525+0000 7f4fd3fff640 10 client.4634.objecter _finish_command 14 = system:0 2024-03-27T21:40:54.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.525+0000 7f4fd3fff640 20 client.4634.objecter put_session s=0x7f4fec1f23b0 osd=7 4 2024-03-27T21:40:54.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.525+0000 7f4fd3fff640 15 client.4634.objecter _session_command_op_remove 7 14 2024-03-27T21:40:54.515 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:54.529+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:54.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.540+0000 7f4ff1d99640 10 client.4634.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:54.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.540+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign -1 15 2024-03-27T21:40:54.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.540+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec1f23b0 osd=7 3 2024-03-27T21:40:54.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.540+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec1f23b0 osd=7 3 2024-03-27T21:40:54.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.540+0000 7f4ff1d99640 20 client.4634.objecter _get_session s=0x7f4fec1f23b0 osd=7 3 2024-03-27T21:40:54.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.540+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_remove -1 15 2024-03-27T21:40:54.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.540+0000 7f4ff1d99640 20 client.4634.objecter get_session s=0x7f4fec1f23b0 osd=7 3 2024-03-27T21:40:54.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.540+0000 7f4ff1d99640 15 client.4634.objecter _session_command_op_assign 7 15 2024-03-27T21:40:54.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.540+0000 7f4ff1d99640 20 client.4634.objecter put_session s=0x7f4fec1f23b0 osd=7 4 2024-03-27T21:40:54.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.540+0000 7f4ff1d99640 10 client.4634.objecter _send_command 15 2024-03-27T21:40:54.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.540+0000 7f4ff1d99640 1 -- 172.21.15.60:0/3495346115 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4fec1efa80 con 0x7f4fec1f1720 2024-03-27T21:40:54.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.542+0000 7f4fd3fff640 1 -- 172.21.15.60:0/3495346115 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f4fec1efa80 con 0x7f4fec1f1720 2024-03-27T21:40:54.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.542+0000 7f4fd3fff640 10 client.4634.objecter ms_dispatch 0x7f4fec000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:54.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.542+0000 7f4fd3fff640 10 client.4634.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:54.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.542+0000 7f4fd3fff640 20 client.4634.objecter put_session s=0x7f4fec1f23b0 osd=7 4 2024-03-27T21:40:54.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:54.542+0000 7f4fd3fff640 15 client.4634.objecter _session_command_op_remove 7 15 2024-03-27T21:40:54.528 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:54.529 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:54.531 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:40:54.531 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:40:54.616 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:54.630+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:54.626 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:54.626 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:54.634 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:40:54.716 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:54.730+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:54.735 INFO:teuthology.orchestra.run.smithi060.stdout:get_pg_num_history pg_num_history pg_num_history(e44 pg_nums {1={16=1},2={19=8}} deleted_pools ) 2024-03-27T21:40:54.735 INFO:teuthology.orchestra.run.smithi060.stdout:Importing pgid 2.3 2024-03-27T21:40:54.739 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:54.740 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:54.740 INFO:teuthology.orchestra.run.smithi060.stdout:write_pg epoch 34 info 2.3( empty local-lis/les=33/34 n=0 ec=19/19 lis/c=33/33 les/c/f=34/34/0 sis=33) 2024-03-27T21:40:54.741 INFO:teuthology.orchestra.run.smithi060.stdout:Import successful 2024-03-27T21:40:54.749 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:40:54.818 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:54.832+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:54.854 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:54.854 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:54.862 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:40:54.919 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:54.933+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:54.968 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:54.969 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:54.977 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:40:55.007 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-27T21:40:55.051 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:55.065+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:55.082 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:55.082 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:55.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:40:55.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.127+0000 7f38f0f00640 10 monclient: get_monmap_and_config 2024-03-27T21:40:55.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.127+0000 7f38f0f00640 10 monclient: build_initial_monmap 2024-03-27T21:40:55.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.127+0000 7f38f0f00640 10 monclient: monmap: 2024-03-27T21:40:55.113 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:55.113 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:55.114 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:55.128427+0000 2024-03-27T21:40:55.114 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:55.128427+0000 2024-03-27T21:40:55.114 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:55.114 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:55.114 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:55.114 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:55.114 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:55.114 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:55.114 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:55.114 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:55.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.128+0000 7f38f0f00640 1 Processor -- start 2024-03-27T21:40:55.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.128+0000 7f38f0f00640 1 -- start start 2024-03-27T21:40:55.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.128+0000 7f38f0f00640 10 monclient: init 2024-03-27T21:40:55.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.128+0000 7f38f0f00640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:55.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.128+0000 7f38f0f00640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-03-27T21:40:55.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.128+0000 7f38f0f00640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f38ec06f900 0x7f38ec13fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:55.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.128+0000 7f38f0f00640 10 monclient(hunting): picked mon.noname-b con 0x7f38ec06f900 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:55.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.128+0000 7f38f0f00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f38ec06fce0 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:55.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.128+0000 7f38f0f00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f38ec138bf0 0x7f38ec139080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:55.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.128+0000 7f38f0f00640 10 monclient(hunting): picked mon.noname-c con 0x7f38ec138bf0 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:55.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.128+0000 7f38f0f00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38ec071b50 con 0x7f38ec06fce0 2024-03-27T21:40:55.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.128+0000 7f38f0f00640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:55.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.128+0000 7f38f0f00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38ec140100 con 0x7f38ec138bf0 2024-03-27T21:40:55.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.128+0000 7f38f0f00640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:55.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.128+0000 7f38f0f00640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec1395c0 con 0x7f38ec06f900 2024-03-27T21:40:55.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.128+0000 7f38f0f00640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:55.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.128+0000 7f38f0f00640 10 monclient(hunting): authenticate will time out at 896.349853s 2024-03-27T21:40:55.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ea59c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f38ec06f900 0x7f38ec13fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:55.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ea59c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f38ec06f900 0x7f38ec13fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:57820/0 (socket says 172.21.15.60:57820) 2024-03-27T21:40:55.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ea59c640 1 -- 172.21.15.60:0/1037736617 learned_addr learned my addr 172.21.15.60:0/1037736617 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:55.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ea59c640 10 monclient(hunting): get_auth_request con 0x7f38ec06f900 auth_method 0 2024-03-27T21:40:55.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:55.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ea59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:55.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:55.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ead9d640 1 --2- 172.21.15.60:0/1037736617 >> v2:172.21.15.191:3300/0 conn(0x7f38ec138bf0 0x7f38ec139080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:55.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ead9d640 10 monclient(hunting): get_auth_request con 0x7f38ec138bf0 auth_method 0 2024-03-27T21:40:55.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:55.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ead9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:55.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:55.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:55.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:55.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:55.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:55.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:55.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:55.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ea59c640 10 monclient(hunting): handle_auth_done global_id 4637 payload 274 2024-03-27T21:40:55.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ea59c640 1 -- 172.21.15.60:0/1037736617 >> v2:172.21.15.191:3300/0 conn(0x7f38ec138bf0 msgr2=0x7f38ec139080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:55.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ea59c640 1 --2- 172.21.15.60:0/1037736617 >> v2:172.21.15.191:3300/0 conn(0x7f38ec138bf0 0x7f38ec139080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:55.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ea59c640 1 -- 172.21.15.60:0/1037736617 >> v1:172.21.15.191:6789/0 conn(0x7f38ec06fce0 legacy=0x7f38ec0af850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:55.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ea59c640 10 monclient: _finish_hunting 0 2024-03-27T21:40:55.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ea59c640 1 monclient: found mon.noname-b 2024-03-27T21:40:55.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ea59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:55.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ea59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:55.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.129+0000 7f38ea59c640 1 -- 172.21.15.60:0/1037736617 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ec13f690 con 0x7f38ec06f900 2024-03-27T21:40:55.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38ea59c640 1 --2- 172.21.15.60:0/1037736617 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f38ec06f900 0x7f38ec13fbc0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f38dc009870 tx=0x7f38dc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14a352488a513947 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:55.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38e9d9b640 1 -- 172.21.15.60:0/1037736617 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38dc030840 con 0x7f38ec06f900 2024-03-27T21:40:55.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38e9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:55.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38e9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:55.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38e9d9b640 10 monclient: dump: 2024-03-27T21:40:55.126 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:55.126 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:55.126 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:55.126 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:55.126 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:55.126 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:55.126 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:55.126 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:55.127 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38e9d9b640 10 monclient: _finish_auth 0 2024-03-27T21:40:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38e9d9b640 10 monclient: _check_auth_tickets 2024-03-27T21:40:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38e9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38f0f00640 5 monclient: authenticate success, global_id 4637 2024-03-27T21:40:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38f0f00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38e9d9b640 1 -- 172.21.15.60:0/1037736617 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38dc0309a0 con 0x7f38ec06f900 2024-03-27T21:40:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38e9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38f0f00640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38e9d9b640 1 -- 172.21.15.60:0/1037736617 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38dc030d30 con 0x7f38ec06f900 2024-03-27T21:40:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38f0f00640 4 set_mon_vals no callback set 2024-03-27T21:40:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38e9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38e9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38e9d9b640 10 monclient: dump: 2024-03-27T21:40:55.128 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:55.128 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:55.128 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:55.128 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:55.128 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:55.128 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:55.128 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:55.128 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:55.128 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:55.128 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:55.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38f0f00640 10 monclient: shutdown 2024-03-27T21:40:55.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38f0f00640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:55.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38f0f00640 1 -- 172.21.15.60:0/1037736617 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f38ec06f900 msgr2=0x7f38ec13fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:55.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38f0f00640 1 --2- 172.21.15.60:0/1037736617 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f38ec06f900 0x7f38ec13fbc0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f38dc009870 tx=0x7f38dc02b980 comp rx=0 tx=0).stop 2024-03-27T21:40:55.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38f0f00640 1 -- 172.21.15.60:0/1037736617 shutdown_connections 2024-03-27T21:40:55.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38f0f00640 1 --2- 172.21.15.60:0/1037736617 >> v2:172.21.15.191:3300/0 conn(0x7f38ec138bf0 0x7f38ec139080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:55.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38f0f00640 1 --2- 172.21.15.60:0/1037736617 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f38ec06f900 0x7f38ec13fbc0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:55.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38f0f00640 1 -- 172.21.15.60:0/1037736617 >> 172.21.15.60:0/1037736617 conn(0x7f38ec0b4e50 msgr2=0x7f38ec0b5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:55.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38f0f00640 1 -- 172.21.15.60:0/1037736617 shutdown_connections 2024-03-27T21:40:55.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.130+0000 7f38f0f00640 1 -- 172.21.15.60:0/1037736617 wait complete. 2024-03-27T21:40:55.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.131+0000 7f38f0f00640 10 monclient: build_initial_monmap 2024-03-27T21:40:55.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.131+0000 7f38f0f00640 10 monclient: monmap: 2024-03-27T21:40:55.129 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:55.129 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:55.129 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:55.129 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:55.129 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:55.130 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:55.130 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:55.130 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:55.130 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:55.130 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:55.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.131+0000 7f38f0f00640 1 librados: starting msgr at 2024-03-27T21:40:55.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 1 librados: starting objecter 2024-03-27T21:40:55.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 1 Processor -- start 2024-03-27T21:40:55.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 1 -- start start 2024-03-27T21:40:55.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 1 librados: setting wanted keys 2024-03-27T21:40:55.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 1 librados: calling monclient init 2024-03-27T21:40:55.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 10 monclient: init 2024-03-27T21:40:55.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:40:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38ec06f900 0x7f38ec13b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 10 monclient(hunting): picked mon.noname-a con 0x7f38ec06f900 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f38ec06fce0 0x7f38ec137fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 10 monclient(hunting): picked mon.noname-c con 0x7f38ec06fce0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38ec138bf0 0x7f38ec138500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 10 monclient(hunting): picked mon.noname-b con 0x7f38ec138bf0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec138a40 con 0x7f38ec06f900 2024-03-27T21:40:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec142fa0 con 0x7f38ec138bf0 2024-03-27T21:40:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec13c290 con 0x7f38ec06fce0 2024-03-27T21:40:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38f0f00640 10 monclient(hunting): authenticate will time out at 896.353667s 2024-03-27T21:40:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38ea59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38ec06f900 0x7f38ec13b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38e3fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f38ec06fce0 0x7f38ec137fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38ea59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38ec06f900 0x7f38ec13b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47182/0 (socket says 172.21.15.60:47182) 2024-03-27T21:40:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38ea59c640 1 -- 172.21.15.60:0/1090468503 learned_addr learned my addr 172.21.15.60:0/1090468503 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38ea59c640 10 monclient(hunting): get_auth_request con 0x7f38ec06f900 auth_method 0 2024-03-27T21:40:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38ea59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38e3fff640 10 monclient(hunting): get_auth_request con 0x7f38ec06fce0 auth_method 0 2024-03-27T21:40:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38ead9d640 1 --2- 172.21.15.60:0/1090468503 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38ec138bf0 0x7f38ec138500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38ead9d640 10 monclient(hunting): get_auth_request con 0x7f38ec138bf0 auth_method 0 2024-03-27T21:40:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38ead9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.132+0000 7f38ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e3fff640 10 monclient(hunting): handle_auth_done global_id 4640 payload 824 2024-03-27T21:40:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e3fff640 1 -- 172.21.15.60:0/1090468503 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38ec138bf0 msgr2=0x7f38ec138500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e3fff640 1 --2- 172.21.15.60:0/1090468503 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38ec138bf0 0x7f38ec138500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e3fff640 1 -- 172.21.15.60:0/1090468503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38ec06f900 msgr2=0x7f38ec13b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e3fff640 1 --2- 172.21.15.60:0/1090468503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38ec06f900 0x7f38ec13b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e3fff640 10 monclient: _finish_hunting 0 2024-03-27T21:40:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e3fff640 1 monclient: found mon.noname-c 2024-03-27T21:40:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:40:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e3fff640 1 -- 172.21.15.60:0/1090468503 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38dc0094d0 con 0x7f38ec06fce0 2024-03-27T21:40:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38ea59c640 1 --2- 172.21.15.60:0/1090468503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38ec06f900 0x7f38ec13b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e3fff640 1 --2- 172.21.15.60:0/1090468503 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f38ec06fce0 0x7f38ec137fc0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f38d400c340 tx=0x7f38d400c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e2ffd640 10 client.?.objecter ms_handle_connect 0x7f38ec06fce0 2024-03-27T21:40:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e2ffd640 1 -- 172.21.15.60:0/1090468503 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d40052d0 con 0x7f38ec06fce0 2024-03-27T21:40:55.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:55.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:55.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e2ffd640 10 monclient: dump: 2024-03-27T21:40:55.135 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:55.135 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:55.135 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:55.135 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:55.135 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:55.135 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:55.135 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:55.135 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:55.135 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:55.135 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:55.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e2ffd640 10 monclient: _finish_auth 0 2024-03-27T21:40:55.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e2ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:40:55.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:55.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e2ffd640 1 -- 172.21.15.60:0/1090468503 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d4005450 con 0x7f38ec06fce0 2024-03-27T21:40:55.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38f0f00640 5 monclient: authenticate success, global_id 4640 2024-03-27T21:40:55.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38f0f00640 10 monclient: _renew_subs 2024-03-27T21:40:55.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38f0f00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:55.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38f0f00640 1 -- 172.21.15.60:0/1090468503 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38ec142a80 con 0x7f38ec06fce0 2024-03-27T21:40:55.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:55.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e2ffd640 1 -- 172.21.15.60:0/1090468503 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d4007260 con 0x7f38ec06fce0 2024-03-27T21:40:55.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:55.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:55.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e2ffd640 10 monclient: dump: 2024-03-27T21:40:55.136 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:55.136 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:55.136 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:55.136 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:55.137 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38e959a640 4 set_mon_vals no callback set 2024-03-27T21:40:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38f0f00640 10 client.4640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38f0f00640 10 monclient: _renew_subs 2024-03-27T21:40:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38f0f00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38f0f00640 1 -- 172.21.15.60:0/1090468503 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38ec142750 con 0x7f38ec06fce0 2024-03-27T21:40:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.133+0000 7f38f0f00640 1 librados: init done 2024-03-27T21:40:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.134+0000 7f38f0f00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.134+0000 7f38f0f00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.134+0000 7f38f0f00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:55.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.134+0000 7f38f0f00640 1 -- 172.21.15.60:0/1090468503 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38ec142c70 con 0x7f38ec06fce0 2024-03-27T21:40:55.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.134+0000 7f38e2ffd640 1 -- 172.21.15.60:0/1090468503 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f38d40073c0 con 0x7f38ec06fce0 2024-03-27T21:40:55.138 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:55.135+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:55.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.135+0000 7f38e2ffd640 1 --2- 172.21.15.60:0/1090468503 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f38cc06c590 0x7f38cc06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:55.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.135+0000 7f38e2ffd640 1 -- 172.21.15.60:0/1090468503 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f38d40883c0 con 0x7f38ec06fce0 2024-03-27T21:40:55.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.135+0000 7f38e2ffd640 10 client.4640.objecter ms_dispatch 0x7f38ec000b90 osd_map(47..47 src has 1..47) v4 2024-03-27T21:40:55.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.135+0000 7f38e2ffd640 3 client.4640.objecter handle_osd_map got epochs [47,47] > 0 2024-03-27T21:40:55.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.135+0000 7f38e2ffd640 3 client.4640.objecter handle_osd_map decoding full epoch 47 2024-03-27T21:40:55.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.135+0000 7f38ea59c640 1 --2- 172.21.15.60:0/1090468503 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f38cc06c590 0x7f38cc06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:55.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.135+0000 7f38ea59c640 10 monclient: get_auth_request con 0x7f38cc06c590 auth_method 0 2024-03-27T21:40:55.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.135+0000 7f38e2ffd640 20 client.4640.objecter dump_active .. 0 homeless 2024-03-27T21:40:55.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.135+0000 7f38e2ffd640 10 client.4640.objecter ms_handle_connect 0x7f38cc06c590 2024-03-27T21:40:55.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.135+0000 7f38ea59c640 1 --2- 172.21.15.60:0/1090468503 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f38cc06c590 0x7f38cc06ea50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f38dc009870 tx=0x7f38dc008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:55.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.139+0000 7f38e2ffd640 1 -- 172.21.15.60:0/1090468503 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f38d40538b0 con 0x7f38ec06fce0 2024-03-27T21:40:55.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.139+0000 7f38e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:55.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.139+0000 7f38e2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:55.196 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:55.196 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:55.205 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:40:55.250 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:55.263+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:55.251 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f /var/log/ceph/exp.2.3.2 2024-03-27T21:40:55.309 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:55.309 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:55.318 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:40:55.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.351+0000 7f38f0f00640 10 start_mon_command cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:40:55.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.351+0000 7f38f0f00640 10 monclient: _send_command 2 [{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:40:55.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.351+0000 7f38f0f00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:55.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.351+0000 7f38f0f00640 1 -- 172.21.15.60:0/1090468503 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f38ec069470 con 0x7f38ec06fce0 2024-03-27T21:40:55.338 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:55.351+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:55.424 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:55.438+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:55.464 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:55.464 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:55.473 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:40:55.525 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:55.539+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:55.583 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:55.583 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:55.592 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:40:55.626 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:55.641+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:55.727 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:55.741+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:55.746 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:55.746 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:55.756 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:40:55.827 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:55.841+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:55.865 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:55.865 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:55.874 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:40:55.929 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:55.942+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:55.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.944+0000 7f38e2ffd640 1 -- 172.21.15.60:0/1090468503 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v48) v1 ==== 97+0+0 (secure 0 0 0) 0x7f38d4056ed0 con 0x7f38ec06fce0 2024-03-27T21:40:55.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.944+0000 7f38e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:40:55.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.944+0000 7f38e2ffd640 10 monclient: _finish_command 2 = system:0 nodeep-scrub is set 2024-03-27T21:40:55.932 INFO:teuthology.orchestra.run.smithi060.stderr:nodeep-scrub is set 2024-03-27T21:40:55.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.946+0000 7f38f0f00640 10 librados: watch_flush enter 2024-03-27T21:40:55.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.946+0000 7f38f0f00640 10 librados: watch_flush exit 2024-03-27T21:40:55.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.946+0000 7f38f0f00640 20 client.4640.objecter shutdown clearing up homeless session... 2024-03-27T21:40:55.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.946+0000 7f38f0f00640 10 client.4640.objecter successfully canceled tick 2024-03-27T21:40:55.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.947+0000 7f38f0f00640 1 -- 172.21.15.60:0/1090468503 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f38cc06c590 msgr2=0x7f38cc06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:55.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.947+0000 7f38f0f00640 1 --2- 172.21.15.60:0/1090468503 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f38cc06c590 0x7f38cc06ea50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f38dc009870 tx=0x7f38dc008030 comp rx=0 tx=0).stop 2024-03-27T21:40:55.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.947+0000 7f38f0f00640 10 monclient: shutdown 2024-03-27T21:40:55.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.947+0000 7f38f0f00640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:55.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.947+0000 7f38f0f00640 1 -- 172.21.15.60:0/1090468503 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f38ec06fce0 msgr2=0x7f38ec137fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:55.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.947+0000 7f38f0f00640 1 --2- 172.21.15.60:0/1090468503 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f38ec06fce0 0x7f38ec137fc0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f38d400c340 tx=0x7f38d400c800 comp rx=0 tx=0).stop 2024-03-27T21:40:55.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.947+0000 7f38f0f00640 1 -- 172.21.15.60:0/1090468503 shutdown_connections 2024-03-27T21:40:55.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.947+0000 7f38f0f00640 1 --2- 172.21.15.60:0/1090468503 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f38cc06c590 0x7f38cc06ea50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:55.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.947+0000 7f38f0f00640 1 --2- 172.21.15.60:0/1090468503 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38ec138bf0 0x7f38ec138500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:55.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.947+0000 7f38f0f00640 1 --2- 172.21.15.60:0/1090468503 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f38ec06fce0 0x7f38ec137fc0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:55.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.947+0000 7f38f0f00640 1 --2- 172.21.15.60:0/1090468503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38ec06f900 0x7f38ec13b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:55.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.947+0000 7f38f0f00640 1 -- 172.21.15.60:0/1090468503 >> 172.21.15.60:0/1090468503 conn(0x7f38ec0b4e50 msgr2=0x7f38ec0b0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:55.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.947+0000 7f38f0f00640 1 -- 172.21.15.60:0/1090468503 shutdown_connections 2024-03-27T21:40:55.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.947+0000 7f38f0f00640 1 -- 172.21.15.60:0/1090468503 wait complete. 2024-03-27T21:40:55.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:55.947+0000 7f38f0f00640 1 librados: shutdown 2024-03-27T21:40:56.027 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:56.028 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:56.030 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:56.044+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:56.037 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:40:56.131 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:56.145+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:56.162 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:56.163 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:56.174 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:40:56.232 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:56.246+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:56.283 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:56.283 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:56.292 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:40:56.333 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:56.346+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:56.400 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:56.400 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:56.409 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:40:56.434 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:56.447+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:56.520 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:56.520 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:56.529 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:40:56.534 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:40:56.575 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:40:56.589+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:40:56.636 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:56.650+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:56.646 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:56.646 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:56.656 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:40:56.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.700+0000 7f599a355640 10 monclient: get_monmap_and_config 2024-03-27T21:40:56.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.700+0000 7f599a355640 10 monclient: build_initial_monmap 2024-03-27T21:40:56.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.700+0000 7f599a355640 10 monclient: monmap: 2024-03-27T21:40:56.686 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:56.686 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:56.686 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:56.701238+0000 2024-03-27T21:40:56.687 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:56.701238+0000 2024-03-27T21:40:56.687 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:56.687 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:56.687 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:56.687 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:56.687 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:56.687 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:56.687 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:56.687 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:56.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f599a355640 1 Processor -- start 2024-03-27T21:40:56.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f599a355640 1 -- start start 2024-03-27T21:40:56.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f599a355640 10 monclient: init 2024-03-27T21:40:56.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f599a355640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:56.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f599a355640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-03-27T21:40:56.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f599a355640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5994073640 0x7f59941379e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:56.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f599a355640 10 monclient(hunting): picked mon.noname-c con 0x7f5994073640 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:56.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f599a355640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f59940ad850 0x7f5994137f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:56.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f599a355640 10 monclient(hunting): picked mon.noname-b con 0x7f59940ad850 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:56.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f599a355640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f599413e0c0 0x7f599413e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:56.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f599a355640 10 monclient(hunting): picked mon.noname-a con 0x7f599413e0c0 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:56.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f599a355640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:56.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f599a355640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5994073a20 con 0x7f599413e0c0 2024-03-27T21:40:56.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f599a355640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:56.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f599a355640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5994071220 con 0x7f5994073640 2024-03-27T21:40:56.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f599a355640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:56.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f599a355640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5994070510 con 0x7f59940ad850 2024-03-27T21:40:56.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f599a355640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:56.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f599a355640 10 monclient(hunting): authenticate will time out at 897.922601s 2024-03-27T21:40:56.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f59937fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f59940ad850 0x7f5994137f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:56.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f59988f2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f599413e0c0 0x7f599413e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:56.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f59937fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f59940ad850 0x7f5994137f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:57836/0 (socket says 172.21.15.60:57836) 2024-03-27T21:40:56.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f59937fe640 1 -- 172.21.15.60:0/3805569431 learned_addr learned my addr 172.21.15.60:0/3805569431 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:56.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f59937fe640 10 monclient(hunting): get_auth_request con 0x7f59940ad850 auth_method 0 2024-03-27T21:40:56.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f59937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:56.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f59937fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:56.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f59937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:56.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f59988f2640 10 monclient(hunting): get_auth_request con 0x7f599413e0c0 auth_method 0 2024-03-27T21:40:56.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f59988f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:56.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f59988f2640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:56.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f59988f2640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:56.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f59937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:56.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f59937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:56.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f59937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:56.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f59988f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:56.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f59988f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:56.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f59988f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:56.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.701+0000 7f5993fff640 1 --2- 172.21.15.60:0/3805569431 >> v2:172.21.15.191:3300/0 conn(0x7f5994073640 0x7f59941379e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:56.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.702+0000 7f5993fff640 10 monclient(hunting): get_auth_request con 0x7f5994073640 auth_method 0 2024-03-27T21:40:56.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.702+0000 7f5993fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:56.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.702+0000 7f5993fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:56.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.702+0000 7f5993fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:56.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.702+0000 7f59937fe640 10 monclient(hunting): handle_auth_done global_id 4643 payload 274 2024-03-27T21:40:56.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.702+0000 7f59937fe640 1 -- 172.21.15.60:0/3805569431 >> v2:172.21.15.191:3300/0 conn(0x7f5994073640 msgr2=0x7f59941379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:56.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.702+0000 7f59937fe640 1 --2- 172.21.15.60:0/3805569431 >> v2:172.21.15.191:3300/0 conn(0x7f5994073640 0x7f59941379e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:56.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.702+0000 7f59937fe640 1 -- 172.21.15.60:0/3805569431 >> v2:172.21.15.60:3300/0 conn(0x7f599413e0c0 msgr2=0x7f599413e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:56.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.702+0000 7f59937fe640 1 --2- 172.21.15.60:0/3805569431 >> v2:172.21.15.60:3300/0 conn(0x7f599413e0c0 0x7f599413e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:56.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.702+0000 7f59937fe640 10 monclient: _finish_hunting 0 2024-03-27T21:40:56.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.702+0000 7f59937fe640 1 monclient: found mon.noname-b 2024-03-27T21:40:56.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.702+0000 7f59937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:56.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.702+0000 7f59937fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:56.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.702+0000 7f59937fe640 1 -- 172.21.15.60:0/3805569431 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59940707f0 con 0x7f59940ad850 2024-03-27T21:40:56.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.702+0000 7f59988f2640 1 --2- 172.21.15.60:0/3805569431 >> v2:172.21.15.60:3300/0 conn(0x7f599413e0c0 0x7f599413e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:56.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.702+0000 7f5993fff640 1 --2- 172.21.15.60:0/3805569431 >> v2:172.21.15.191:3300/0 conn(0x7f5994073640 0x7f59941379e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:40:56.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.702+0000 7f59937fe640 1 --2- 172.21.15.60:0/3805569431 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f59940ad850 0x7f5994137f20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5994073c30 tx=0x7f598802b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93a28c635ad5e23e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:56.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.702+0000 7f5992ffd640 1 -- 172.21.15.60:0/3805569431 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988031860 con 0x7f59940ad850 2024-03-27T21:40:56.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f5992ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:56.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f5992ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:56.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f5992ffd640 10 monclient: dump: 2024-03-27T21:40:56.701 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:56.701 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:56.701 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:56.701 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:56.701 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:56.701 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:56.702 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:56.702 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:56.702 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:56.702 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:56.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f5992ffd640 10 monclient: _finish_auth 0 2024-03-27T21:40:56.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f5992ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:40:56.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f5992ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:56.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f599a355640 5 monclient: authenticate success, global_id 4643 2024-03-27T21:40:56.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f5992ffd640 1 -- 172.21.15.60:0/3805569431 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59880319c0 con 0x7f59940ad850 2024-03-27T21:40:56.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f5992ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:56.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f599a355640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:56.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f5992ffd640 1 -- 172.21.15.60:0/3805569431 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988031d50 con 0x7f59940ad850 2024-03-27T21:40:56.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f599a355640 4 set_mon_vals no callback set 2024-03-27T21:40:56.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f5992ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:56.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f5992ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:56.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f5992ffd640 10 monclient: dump: 2024-03-27T21:40:56.703 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:56.703 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:56.703 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:56.703 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:56.703 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:56.703 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:56.703 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:56.703 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:56.703 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:56.703 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:56.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f599a355640 10 monclient: shutdown 2024-03-27T21:40:56.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f599a355640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:56.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f599a355640 1 -- 172.21.15.60:0/3805569431 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f59940ad850 msgr2=0x7f5994137f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:56.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f599a355640 1 --2- 172.21.15.60:0/3805569431 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f59940ad850 0x7f5994137f20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5994073c30 tx=0x7f598802b900 comp rx=0 tx=0).stop 2024-03-27T21:40:56.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f599a355640 1 -- 172.21.15.60:0/3805569431 shutdown_connections 2024-03-27T21:40:56.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f599a355640 1 --2- 172.21.15.60:0/3805569431 >> v2:172.21.15.60:3300/0 conn(0x7f599413e0c0 0x7f599413e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:56.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f599a355640 1 --2- 172.21.15.60:0/3805569431 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f59940ad850 0x7f5994137f20 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:56.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f599a355640 1 --2- 172.21.15.60:0/3805569431 >> v2:172.21.15.191:3300/0 conn(0x7f5994073640 0x7f59941379e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:56.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f599a355640 1 -- 172.21.15.60:0/3805569431 >> 172.21.15.60:0/3805569431 conn(0x7f59940b3de0 msgr2=0x7f5994069ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:56.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f599a355640 1 -- 172.21.15.60:0/3805569431 shutdown_connections 2024-03-27T21:40:56.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.703+0000 7f599a355640 1 -- 172.21.15.60:0/3805569431 wait complete. 2024-03-27T21:40:56.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.704+0000 7f599a355640 10 monclient: build_initial_monmap 2024-03-27T21:40:56.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.704+0000 7f599a355640 10 monclient: monmap: 2024-03-27T21:40:56.704 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:56.705 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:56.705 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:56.705 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:56.705 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:56.705 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:56.705 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:56.705 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:56.705 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:56.705 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:56.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.704+0000 7f599a355640 1 librados: starting msgr at 2024-03-27T21:40:56.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.704+0000 7f599a355640 1 librados: starting objecter 2024-03-27T21:40:56.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.704+0000 7f599a355640 1 Processor -- start 2024-03-27T21:40:56.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.704+0000 7f599a355640 1 -- start start 2024-03-27T21:40:56.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.704+0000 7f599a355640 1 librados: setting wanted keys 2024-03-27T21:40:56.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.704+0000 7f599a355640 1 librados: calling monclient init 2024-03-27T21:40:56.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.704+0000 7f599a355640 10 monclient: init 2024-03-27T21:40:56.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f599a355640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:56.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f599a355640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:40:56.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f599a355640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5994073640 0x7f59940acac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:56.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f599a355640 10 monclient(hunting): picked mon.noname-c con 0x7f5994073640 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:56.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f599a355640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59940ad850 0x7f5994138b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:56.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f599a355640 10 monclient(hunting): picked mon.noname-b con 0x7f59940ad850 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:56.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f599a355640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f599413e0c0 0x7f59941d9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:56.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f599a355640 10 monclient(hunting): picked mon.noname-a con 0x7f599413e0c0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:56.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f599a355640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:56.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f599a355640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59940b3260 con 0x7f599413e0c0 2024-03-27T21:40:56.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f599a355640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:56.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f599a355640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599413d280 con 0x7f59940ad850 2024-03-27T21:40:56.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f599a355640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:56.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f599a355640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f599413daf0 con 0x7f5994073640 2024-03-27T21:40:56.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f599a355640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:56.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f599a355640 10 monclient(hunting): authenticate will time out at 897.926394s 2024-03-27T21:40:56.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f5993fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5994073640 0x7f59940acac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:56.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f5993fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5994073640 0x7f59940acac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:57838/0 (socket says 172.21.15.60:57838) 2024-03-27T21:40:56.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f5993fff640 1 -- 172.21.15.60:0/1529533293 learned_addr learned my addr 172.21.15.60:0/1529533293 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:56.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f5993fff640 10 monclient(hunting): get_auth_request con 0x7f5994073640 auth_method 0 2024-03-27T21:40:56.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f5993fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:56.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f5993fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:56.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f5993fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:56.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f59988f2640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f599413e0c0 0x7f59941d9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:56.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f59988f2640 10 monclient(hunting): get_auth_request con 0x7f599413e0c0 auth_method 0 2024-03-27T21:40:56.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f59988f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:56.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f59988f2640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:56.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f59988f2640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:56.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f5993fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:56.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f5993fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:56.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f59937fe640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59940ad850 0x7f5994138b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:56.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f5993fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:56.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f59988f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:56.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f59988f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:56.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f59988f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:56.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f59937fe640 10 monclient(hunting): get_auth_request con 0x7f59940ad850 auth_method 0 2024-03-27T21:40:56.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f59937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:56.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f59937fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:56.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f59937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:56.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f59937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:56.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f59937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:56.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f59937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:56.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f5993fff640 10 monclient(hunting): handle_auth_done global_id 4646 payload 824 2024-03-27T21:40:56.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f5993fff640 1 -- 172.21.15.60:0/1529533293 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59940ad850 msgr2=0x7f5994138b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:56.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f5993fff640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59940ad850 0x7f5994138b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:56.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f5993fff640 1 -- 172.21.15.60:0/1529533293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f599413e0c0 msgr2=0x7f59941d9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:56.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f5993fff640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f599413e0c0 0x7f59941d9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:56.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f5993fff640 10 monclient: _finish_hunting 0 2024-03-27T21:40:56.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f5993fff640 1 monclient: found mon.noname-c 2024-03-27T21:40:56.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f5993fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:56.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f5993fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:40:56.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f5993fff640 1 -- 172.21.15.60:0/1529533293 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c009a70 con 0x7f5994073640 2024-03-27T21:40:56.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.705+0000 7f59988f2640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f599413e0c0 0x7f59941d9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:56.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f5993fff640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5994073640 0x7f59940acac0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f598400e730 tx=0x7f598400eb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:56.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f5990ff9640 10 client.?.objecter ms_handle_connect 0x7f5994073640 2024-03-27T21:40:56.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f5990ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:56.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f5990ff9640 1 -- 172.21.15.60:0/1529533293 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984004030 con 0x7f5994073640 2024-03-27T21:40:56.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f5990ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:56.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f5990ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:56.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f5990ff9640 10 monclient: dump: 2024-03-27T21:40:56.710 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:56.710 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:56.710 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:56.710 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:56.710 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:56.710 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:56.711 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:56.711 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:56.711 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:56.711 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:56.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f5990ff9640 10 monclient: _finish_auth 0 2024-03-27T21:40:56.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f5990ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:40:56.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f5990ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:56.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f599a355640 5 monclient: authenticate success, global_id 4646 2024-03-27T21:40:56.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f5990ff9640 1 -- 172.21.15.60:0/1529533293 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5984004190 con 0x7f5994073640 2024-03-27T21:40:56.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f5990ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:56.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f5990ff9640 1 -- 172.21.15.60:0/1529533293 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984004520 con 0x7f5994073640 2024-03-27T21:40:56.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f599a355640 10 monclient: _renew_subs 2024-03-27T21:40:56.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f599a355640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:56.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f599a355640 1 -- 172.21.15.60:0/1529533293 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59880094d0 con 0x7f5994073640 2024-03-27T21:40:56.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f5990ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:56.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f59927fc640 4 set_mon_vals no callback set 2024-03-27T21:40:56.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f5990ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:56.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f5990ff9640 10 monclient: dump: 2024-03-27T21:40:56.712 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:56.712 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:56.712 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:56.712 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:56.712 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:56.712 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:56.712 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:56.712 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:56.712 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:56.712 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:56.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f599a355640 10 client.4646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:56.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f599a355640 10 monclient: _renew_subs 2024-03-27T21:40:56.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f599a355640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:56.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f599a355640 1 -- 172.21.15.60:0/1529533293 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59941da150 con 0x7f5994073640 2024-03-27T21:40:56.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.706+0000 7f599a355640 1 librados: init done 2024-03-27T21:40:56.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.707+0000 7f599a355640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:56.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.707+0000 7f599a355640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:56.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.707+0000 7f599a355640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:56.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.707+0000 7f599a355640 1 -- 172.21.15.60:0/1529533293 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f59940ad6c0 con 0x7f5994073640 2024-03-27T21:40:56.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.707+0000 7f5990ff9640 1 -- 172.21.15.60:0/1529533293 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f598400c6d0 con 0x7f5994073640 2024-03-27T21:40:56.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f5990ff9640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f598006c590 0x7f598006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:56.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f5990ff9640 1 -- 172.21.15.60:0/1529533293 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f5984088b70 con 0x7f5994073640 2024-03-27T21:40:56.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f5990ff9640 10 client.4646.objecter ms_dispatch 0x7f5994000b90 osd_map(48..48 src has 1..48) v4 2024-03-27T21:40:56.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f5990ff9640 3 client.4646.objecter handle_osd_map got epochs [48,48] > 0 2024-03-27T21:40:56.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f5990ff9640 3 client.4646.objecter handle_osd_map decoding full epoch 48 2024-03-27T21:40:56.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f59937fe640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f598006c590 0x7f598006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:56.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f59937fe640 10 monclient: get_auth_request con 0x7f598006c590 auth_method 0 2024-03-27T21:40:56.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f5990ff9640 20 client.4646.objecter dump_active .. 0 homeless 2024-03-27T21:40:56.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f5990ff9640 1 -- 172.21.15.60:0/1529533293 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v48) v1 ==== 54+0+16 (secure 0 0 0) 0x7f598408d240 con 0x7f5994073640 2024-03-27T21:40:56.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f5990ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:56.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f5990ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:56.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f599a355640 10 client.4646.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:56.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign -1 1 2024-03-27T21:40:56.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f599a355640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f5994132560 0x7f5994132940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:56.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f5994138460 osd=0 3 2024-03-27T21:40:56.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f5994138460 osd=0 3 2024-03-27T21:40:56.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f5994138460 osd=0 3 2024-03-27T21:40:56.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f599a355640 15 client.4646.objecter _session_command_op_remove -1 1 2024-03-27T21:40:56.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f599a355640 20 client.4646.objecter get_session s=0x7f5994138460 osd=0 3 2024-03-27T21:40:56.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign 0 1 2024-03-27T21:40:56.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f5994138460 osd=0 4 2024-03-27T21:40:56.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f599a355640 10 client.4646.objecter _send_command 1 2024-03-27T21:40:56.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f599a355640 1 -- 172.21.15.60:0/1529533293 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f599413dd70 con 0x7f5994132560 2024-03-27T21:40:56.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f59988f2640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f5994132560 0x7f5994132940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:56.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f5990ff9640 10 client.4646.objecter ms_handle_connect 0x7f598006c590 2024-03-27T21:40:56.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f59937fe640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f598006c590 0x7f598006ea50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5988036000 tx=0x7f598803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:56.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f59988f2640 10 monclient: get_auth_request con 0x7f5994132560 auth_method 0 2024-03-27T21:40:56.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f5990ff9640 10 client.4646.objecter ms_handle_connect 0x7f5994132560 2024-03-27T21:40:56.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.708+0000 7f59988f2640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f5994132560 0x7f5994132940 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:56.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.709+0000 7f5990ff9640 1 -- 172.21.15.60:0/1529533293 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f599413dd70 con 0x7f5994132560 2024-03-27T21:40:56.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.709+0000 7f5990ff9640 10 client.4646.objecter ms_dispatch 0x7f5994000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:56.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.709+0000 7f5990ff9640 10 client.4646.objecter _finish_command 1 = system:0 2024-03-27T21:40:56.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.709+0000 7f5990ff9640 20 client.4646.objecter put_session s=0x7f5994138460 osd=0 4 2024-03-27T21:40:56.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.709+0000 7f5990ff9640 15 client.4646.objecter _session_command_op_remove 0 1 2024-03-27T21:40:56.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.724+0000 7f599a355640 10 client.4646.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:56.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.724+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign -1 2 2024-03-27T21:40:56.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.724+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f5994138460 osd=0 3 2024-03-27T21:40:56.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.724+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f5994138460 osd=0 3 2024-03-27T21:40:56.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.724+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f5994138460 osd=0 3 2024-03-27T21:40:56.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.724+0000 7f599a355640 15 client.4646.objecter _session_command_op_remove -1 2 2024-03-27T21:40:56.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.724+0000 7f599a355640 20 client.4646.objecter get_session s=0x7f5994138460 osd=0 3 2024-03-27T21:40:56.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.724+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign 0 2 2024-03-27T21:40:56.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.724+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f5994138460 osd=0 4 2024-03-27T21:40:56.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.724+0000 7f599a355640 10 client.4646.objecter _send_command 2 2024-03-27T21:40:56.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.724+0000 7f599a355640 1 -- 172.21.15.60:0/1529533293 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5994069470 con 0x7f5994132560 2024-03-27T21:40:56.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.725+0000 7f5990ff9640 1 -- 172.21.15.60:0/1529533293 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5994069470 con 0x7f5994132560 2024-03-27T21:40:56.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.725+0000 7f5990ff9640 10 client.4646.objecter ms_dispatch 0x7f5994000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:56.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.725+0000 7f5990ff9640 10 client.4646.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:56.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.725+0000 7f5990ff9640 20 client.4646.objecter put_session s=0x7f5994138460 osd=0 4 2024-03-27T21:40:56.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.725+0000 7f5990ff9640 15 client.4646.objecter _session_command_op_remove 0 2 2024-03-27T21:40:56.718 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:56.719 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:56.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.725+0000 7f599a355640 10 client.4646.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:56.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.725+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign -1 3 2024-03-27T21:40:56.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.725+0000 7f599a355640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f59941d6a40 0x7f59941d6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:56.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.725+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f599406ff50 osd=1 3 2024-03-27T21:40:56.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.725+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f599406ff50 osd=1 3 2024-03-27T21:40:56.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.725+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f599406ff50 osd=1 3 2024-03-27T21:40:56.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.725+0000 7f599a355640 15 client.4646.objecter _session_command_op_remove -1 3 2024-03-27T21:40:56.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.725+0000 7f599a355640 20 client.4646.objecter get_session s=0x7f599406ff50 osd=1 3 2024-03-27T21:40:56.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.725+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign 1 3 2024-03-27T21:40:56.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.725+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f599406ff50 osd=1 4 2024-03-27T21:40:56.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.725+0000 7f599a355640 10 client.4646.objecter _send_command 3 2024-03-27T21:40:56.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.725+0000 7f599a355640 1 -- 172.21.15.60:0/1529533293 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f59941d7360 con 0x7f59941d6a40 2024-03-27T21:40:56.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.725+0000 7f5993fff640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f59941d6a40 0x7f59941d6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:56.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.725+0000 7f5993fff640 10 monclient: get_auth_request con 0x7f59941d6a40 auth_method 0 2024-03-27T21:40:56.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.726+0000 7f5990ff9640 10 client.4646.objecter ms_handle_connect 0x7f59941d6a40 2024-03-27T21:40:56.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.726+0000 7f5993fff640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f59941d6a40 0x7f59941d6e20 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:56.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.726+0000 7f5990ff9640 1 -- 172.21.15.60:0/1529533293 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f59941d7360 con 0x7f59941d6a40 2024-03-27T21:40:56.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.726+0000 7f5990ff9640 10 client.4646.objecter ms_dispatch 0x7f5994000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:56.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.726+0000 7f5990ff9640 10 client.4646.objecter _finish_command 3 = system:0 2024-03-27T21:40:56.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.726+0000 7f5990ff9640 20 client.4646.objecter put_session s=0x7f599406ff50 osd=1 4 2024-03-27T21:40:56.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.726+0000 7f5990ff9640 15 client.4646.objecter _session_command_op_remove 1 3 2024-03-27T21:40:56.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.742+0000 7f599a355640 10 client.4646.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:56.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.742+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign -1 4 2024-03-27T21:40:56.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.742+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f599406ff50 osd=1 3 2024-03-27T21:40:56.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.742+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f599406ff50 osd=1 3 2024-03-27T21:40:56.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.742+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f599406ff50 osd=1 3 2024-03-27T21:40:56.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.742+0000 7f599a355640 15 client.4646.objecter _session_command_op_remove -1 4 2024-03-27T21:40:56.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.742+0000 7f599a355640 20 client.4646.objecter get_session s=0x7f599406ff50 osd=1 3 2024-03-27T21:40:56.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.742+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign 1 4 2024-03-27T21:40:56.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.742+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f599406ff50 osd=1 4 2024-03-27T21:40:56.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.742+0000 7f599a355640 10 client.4646.objecter _send_command 4 2024-03-27T21:40:56.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.742+0000 7f599a355640 1 -- 172.21.15.60:0/1529533293 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5994065a90 con 0x7f59941d6a40 2024-03-27T21:40:56.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.743+0000 7f5990ff9640 1 -- 172.21.15.60:0/1529533293 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5994065a90 con 0x7f59941d6a40 2024-03-27T21:40:56.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.743+0000 7f5990ff9640 10 client.4646.objecter ms_dispatch 0x7f5994000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:56.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.743+0000 7f5990ff9640 10 client.4646.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:56.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.743+0000 7f5990ff9640 20 client.4646.objecter put_session s=0x7f599406ff50 osd=1 4 2024-03-27T21:40:56.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.743+0000 7f5990ff9640 15 client.4646.objecter _session_command_op_remove 1 4 2024-03-27T21:40:56.730 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:56.732 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:56.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.743+0000 7f599a355640 10 client.4646.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:56.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.743+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign -1 5 2024-03-27T21:40:56.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.743+0000 7f599a355640 20 client.4646.objecter _get_session osd=-1 returning homeless 2024-03-27T21:40:56.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.743+0000 7f599a355640 10 client.4646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:56.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.743+0000 7f599a355640 10 monclient: _renew_subs 2024-03-27T21:40:56.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.743+0000 7f599a355640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:56.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.743+0000 7f599a355640 1 -- 172.21.15.60:0/1529533293 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=49}) v3 -- 0x7f59941c6910 con 0x7f5994073640 2024-03-27T21:40:56.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.743+0000 7f599a355640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:40:56.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.743+0000 7f599a355640 1 -- 172.21.15.60:0/1529533293 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f59941c6bc0 con 0x7f5994073640 2024-03-27T21:40:56.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.743+0000 7f5990ff9640 1 -- 172.21.15.60:0/1529533293 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f598408d220 con 0x7f5994073640 2024-03-27T21:40:56.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.743+0000 7f5990ff9640 10 monclient: handle_get_version_reply finishing 1 version 48 2024-03-27T21:40:56.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.744+0000 7f59927fc640 10 client.4646.objecter _check_command_map_dne tid 5 current 48 map_dne_bound 48 2024-03-27T21:40:56.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.744+0000 7f59927fc640 10 client.4646.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:40:56.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.744+0000 7f59927fc640 15 client.4646.objecter _session_command_op_remove -1 5 2024-03-27T21:40:56.733 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:40:56.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.744+0000 7f599a355640 10 client.4646.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:56.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.744+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign -1 6 2024-03-27T21:40:56.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.744+0000 7f599a355640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f59940667d0 0x7f599402b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:56.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.744+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f5994066660 osd=3 3 2024-03-27T21:40:56.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.744+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f5994066660 osd=3 3 2024-03-27T21:40:56.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.744+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f5994066660 osd=3 3 2024-03-27T21:40:56.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.744+0000 7f599a355640 15 client.4646.objecter _session_command_op_remove -1 6 2024-03-27T21:40:56.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.744+0000 7f599a355640 20 client.4646.objecter get_session s=0x7f5994066660 osd=3 3 2024-03-27T21:40:56.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.744+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign 3 6 2024-03-27T21:40:56.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.744+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f5994066660 osd=3 4 2024-03-27T21:40:56.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.744+0000 7f599a355640 10 client.4646.objecter _send_command 6 2024-03-27T21:40:56.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.744+0000 7f599a355640 1 -- 172.21.15.60:0/1529533293 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f599402bbc0 con 0x7f59940667d0 2024-03-27T21:40:56.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.744+0000 7f59937fe640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f59940667d0 0x7f599402b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:56.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.744+0000 7f59937fe640 10 monclient: get_auth_request con 0x7f59940667d0 auth_method 0 2024-03-27T21:40:56.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.744+0000 7f5990ff9640 10 client.4646.objecter ms_handle_connect 0x7f59940667d0 2024-03-27T21:40:56.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.744+0000 7f59937fe640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f59940667d0 0x7f599402b680 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:56.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.745+0000 7f5990ff9640 1 -- 172.21.15.60:0/1529533293 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f599402bbc0 con 0x7f59940667d0 2024-03-27T21:40:56.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.745+0000 7f5990ff9640 10 client.4646.objecter ms_dispatch 0x7f5994000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:40:56.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.745+0000 7f5990ff9640 10 client.4646.objecter _finish_command 6 = system:0 2024-03-27T21:40:56.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.745+0000 7f5990ff9640 20 client.4646.objecter put_session s=0x7f5994066660 osd=3 4 2024-03-27T21:40:56.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.745+0000 7f5990ff9640 15 client.4646.objecter _session_command_op_remove 3 6 2024-03-27T21:40:56.737 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:56.750+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.758+0000 7f599a355640 10 client.4646.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.758+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign -1 7 2024-03-27T21:40:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.758+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f5994066660 osd=3 3 2024-03-27T21:40:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.758+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f5994066660 osd=3 3 2024-03-27T21:40:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.758+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f5994066660 osd=3 3 2024-03-27T21:40:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.758+0000 7f599a355640 15 client.4646.objecter _session_command_op_remove -1 7 2024-03-27T21:40:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.758+0000 7f599a355640 20 client.4646.objecter get_session s=0x7f5994066660 osd=3 3 2024-03-27T21:40:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.758+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign 3 7 2024-03-27T21:40:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.758+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f5994066660 osd=3 4 2024-03-27T21:40:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.758+0000 7f599a355640 10 client.4646.objecter _send_command 7 2024-03-27T21:40:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.758+0000 7f599a355640 1 -- 172.21.15.60:0/1529533293 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f599402c550 con 0x7f59940667d0 2024-03-27T21:40:56.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.760+0000 7f5990ff9640 1 -- 172.21.15.60:0/1529533293 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f599402c550 con 0x7f59940667d0 2024-03-27T21:40:56.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.760+0000 7f5990ff9640 10 client.4646.objecter ms_dispatch 0x7f5994000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:56.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.760+0000 7f5990ff9640 10 client.4646.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:56.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.760+0000 7f5990ff9640 20 client.4646.objecter put_session s=0x7f5994066660 osd=3 4 2024-03-27T21:40:56.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.760+0000 7f5990ff9640 15 client.4646.objecter _session_command_op_remove 3 7 2024-03-27T21:40:56.747 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.760+0000 7f599a355640 10 client.4646.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.760+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign -1 8 2024-03-27T21:40:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.760+0000 7f599a355640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f59941e72e0 0x7f59941e96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.760+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f599402c6c0 osd=4 3 2024-03-27T21:40:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.760+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f599402c6c0 osd=4 3 2024-03-27T21:40:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.760+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f599402c6c0 osd=4 3 2024-03-27T21:40:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.760+0000 7f599a355640 15 client.4646.objecter _session_command_op_remove -1 8 2024-03-27T21:40:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.760+0000 7f599a355640 20 client.4646.objecter get_session s=0x7f599402c6c0 osd=4 3 2024-03-27T21:40:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.760+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign 4 8 2024-03-27T21:40:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.760+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f599402c6c0 osd=4 4 2024-03-27T21:40:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.760+0000 7f599a355640 10 client.4646.objecter _send_command 8 2024-03-27T21:40:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.760+0000 7f599a355640 1 -- 172.21.15.60:0/1529533293 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5994066320 con 0x7f59941e72e0 2024-03-27T21:40:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.760+0000 7f59988f2640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f59941e72e0 0x7f59941e96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.760+0000 7f59988f2640 10 monclient: get_auth_request con 0x7f59941e72e0 auth_method 0 2024-03-27T21:40:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.761+0000 7f59988f2640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f59941e72e0 0x7f59941e96d0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.761+0000 7f5990ff9640 10 client.4646.objecter ms_handle_connect 0x7f59941e72e0 2024-03-27T21:40:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.761+0000 7f5990ff9640 1 -- 172.21.15.60:0/1529533293 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f5994066320 con 0x7f59941e72e0 2024-03-27T21:40:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.761+0000 7f5990ff9640 10 client.4646.objecter ms_dispatch 0x7f5994000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:40:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.761+0000 7f5990ff9640 10 client.4646.objecter _finish_command 8 = system:0 2024-03-27T21:40:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.761+0000 7f5990ff9640 20 client.4646.objecter put_session s=0x7f599402c6c0 osd=4 4 2024-03-27T21:40:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.761+0000 7f5990ff9640 15 client.4646.objecter _session_command_op_remove 4 8 2024-03-27T21:40:56.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.776+0000 7f599a355640 10 client.4646.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:56.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.776+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign -1 9 2024-03-27T21:40:56.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.776+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f599402c6c0 osd=4 3 2024-03-27T21:40:56.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.776+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f599402c6c0 osd=4 3 2024-03-27T21:40:56.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.776+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f599402c6c0 osd=4 3 2024-03-27T21:40:56.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.776+0000 7f599a355640 15 client.4646.objecter _session_command_op_remove -1 9 2024-03-27T21:40:56.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.776+0000 7f599a355640 20 client.4646.objecter get_session s=0x7f599402c6c0 osd=4 3 2024-03-27T21:40:56.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.776+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign 4 9 2024-03-27T21:40:56.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.776+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f599402c6c0 osd=4 4 2024-03-27T21:40:56.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.776+0000 7f599a355640 10 client.4646.objecter _send_command 9 2024-03-27T21:40:56.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.776+0000 7f599a355640 1 -- 172.21.15.60:0/1529533293 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f59940664c0 con 0x7f59941e72e0 2024-03-27T21:40:56.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.778+0000 7f5990ff9640 1 -- 172.21.15.60:0/1529533293 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f59940664c0 con 0x7f59941e72e0 2024-03-27T21:40:56.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.778+0000 7f5990ff9640 10 client.4646.objecter ms_dispatch 0x7f5994000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:56.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.778+0000 7f5990ff9640 10 client.4646.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:56.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.778+0000 7f5990ff9640 20 client.4646.objecter put_session s=0x7f599402c6c0 osd=4 4 2024-03-27T21:40:56.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.778+0000 7f5990ff9640 15 client.4646.objecter _session_command_op_remove 4 9 2024-03-27T21:40:56.765 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:56.766 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:56.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.778+0000 7f599a355640 10 client.4646.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:56.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.778+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign -1 10 2024-03-27T21:40:56.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.778+0000 7f599a355640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f59940ae4f0 0x7f59940ae910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:56.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.778+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f59940664c0 osd=5 3 2024-03-27T21:40:56.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.778+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f59940664c0 osd=5 3 2024-03-27T21:40:56.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.778+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f59940664c0 osd=5 3 2024-03-27T21:40:56.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.778+0000 7f599a355640 15 client.4646.objecter _session_command_op_remove -1 10 2024-03-27T21:40:56.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.778+0000 7f599a355640 20 client.4646.objecter get_session s=0x7f59940664c0 osd=5 3 2024-03-27T21:40:56.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.778+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign 5 10 2024-03-27T21:40:56.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.778+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f59940664c0 osd=5 4 2024-03-27T21:40:56.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.778+0000 7f599a355640 10 client.4646.objecter _send_command 10 2024-03-27T21:40:56.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.778+0000 7f599a355640 1 -- 172.21.15.60:0/1529533293 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f59941ef630 con 0x7f59940ae4f0 2024-03-27T21:40:56.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.778+0000 7f5993fff640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f59940ae4f0 0x7f59940ae910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:56.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.778+0000 7f5993fff640 10 monclient: get_auth_request con 0x7f59940ae4f0 auth_method 0 2024-03-27T21:40:56.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.778+0000 7f5993fff640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f59940ae4f0 0x7f59940ae910 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:56.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.778+0000 7f5990ff9640 10 client.4646.objecter ms_handle_connect 0x7f59940ae4f0 2024-03-27T21:40:56.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.779+0000 7f5990ff9640 1 -- 172.21.15.60:0/1529533293 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f59941ef630 con 0x7f59940ae4f0 2024-03-27T21:40:56.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.779+0000 7f5990ff9640 10 client.4646.objecter ms_dispatch 0x7f5994000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:40:56.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.779+0000 7f5990ff9640 10 client.4646.objecter _finish_command 10 = system:0 2024-03-27T21:40:56.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.779+0000 7f5990ff9640 20 client.4646.objecter put_session s=0x7f59940664c0 osd=5 4 2024-03-27T21:40:56.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.779+0000 7f5990ff9640 15 client.4646.objecter _session_command_op_remove 5 10 2024-03-27T21:40:56.774 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:56.774 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:56.774 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:56.774 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:56.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.794+0000 7f599a355640 10 client.4646.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:56.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.794+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign -1 11 2024-03-27T21:40:56.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.794+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f59940664c0 osd=5 3 2024-03-27T21:40:56.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.794+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f59940664c0 osd=5 3 2024-03-27T21:40:56.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.794+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f59940664c0 osd=5 3 2024-03-27T21:40:56.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.794+0000 7f599a355640 15 client.4646.objecter _session_command_op_remove -1 11 2024-03-27T21:40:56.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.794+0000 7f599a355640 20 client.4646.objecter get_session s=0x7f59940664c0 osd=5 3 2024-03-27T21:40:56.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.794+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign 5 11 2024-03-27T21:40:56.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.794+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f59940664c0 osd=5 4 2024-03-27T21:40:56.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.794+0000 7f599a355640 10 client.4646.objecter _send_command 11 2024-03-27T21:40:56.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.794+0000 7f599a355640 1 -- 172.21.15.60:0/1529533293 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f59941ecda0 con 0x7f59940ae4f0 2024-03-27T21:40:56.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.795+0000 7f5990ff9640 1 -- 172.21.15.60:0/1529533293 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f59941ecda0 con 0x7f59940ae4f0 2024-03-27T21:40:56.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.795+0000 7f5990ff9640 10 client.4646.objecter ms_dispatch 0x7f5994000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:56.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.795+0000 7f5990ff9640 10 client.4646.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:56.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.795+0000 7f5990ff9640 20 client.4646.objecter put_session s=0x7f59940664c0 osd=5 4 2024-03-27T21:40:56.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.795+0000 7f5990ff9640 15 client.4646.objecter _session_command_op_remove 5 11 2024-03-27T21:40:56.782 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:56.783 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:56.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.795+0000 7f599a355640 10 client.4646.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:56.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.796+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign -1 12 2024-03-27T21:40:56.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.796+0000 7f599a355640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f59941ed420 0x7f59941ef810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:56.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.796+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f59941ecda0 osd=6 3 2024-03-27T21:40:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.796+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f59941ecda0 osd=6 3 2024-03-27T21:40:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.796+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f59941ecda0 osd=6 3 2024-03-27T21:40:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.796+0000 7f599a355640 15 client.4646.objecter _session_command_op_remove -1 12 2024-03-27T21:40:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.796+0000 7f599a355640 20 client.4646.objecter get_session s=0x7f59941ecda0 osd=6 3 2024-03-27T21:40:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.796+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign 6 12 2024-03-27T21:40:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.796+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f59941ecda0 osd=6 4 2024-03-27T21:40:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.796+0000 7f599a355640 10 client.4646.objecter _send_command 12 2024-03-27T21:40:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.796+0000 7f599a355640 1 -- 172.21.15.60:0/1529533293 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f59941f25e0 con 0x7f59941ed420 2024-03-27T21:40:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.796+0000 7f59937fe640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f59941ed420 0x7f59941ef810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.796+0000 7f59937fe640 10 monclient: get_auth_request con 0x7f59941ed420 auth_method 0 2024-03-27T21:40:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.796+0000 7f5990ff9640 10 client.4646.objecter ms_handle_connect 0x7f59941ed420 2024-03-27T21:40:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.796+0000 7f59937fe640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f59941ed420 0x7f59941ef810 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.797+0000 7f5990ff9640 1 -- 172.21.15.60:0/1529533293 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f59941f25e0 con 0x7f59941ed420 2024-03-27T21:40:56.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.797+0000 7f5990ff9640 10 client.4646.objecter ms_dispatch 0x7f5994000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:40:56.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.797+0000 7f5990ff9640 10 client.4646.objecter _finish_command 12 = system:0 2024-03-27T21:40:56.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.797+0000 7f5990ff9640 20 client.4646.objecter put_session s=0x7f59941ecda0 osd=6 4 2024-03-27T21:40:56.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.797+0000 7f5990ff9640 15 client.4646.objecter _session_command_op_remove 6 12 2024-03-27T21:40:56.786 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:40:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.810+0000 7f599a355640 10 client.4646.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.810+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign -1 13 2024-03-27T21:40:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.810+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f59941ecda0 osd=6 3 2024-03-27T21:40:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.810+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f59941ecda0 osd=6 3 2024-03-27T21:40:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.810+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f59941ecda0 osd=6 3 2024-03-27T21:40:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.810+0000 7f599a355640 15 client.4646.objecter _session_command_op_remove -1 13 2024-03-27T21:40:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.810+0000 7f599a355640 20 client.4646.objecter get_session s=0x7f59941ecda0 osd=6 3 2024-03-27T21:40:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.810+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign 6 13 2024-03-27T21:40:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.810+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f59941ecda0 osd=6 4 2024-03-27T21:40:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.810+0000 7f599a355640 10 client.4646.objecter _send_command 13 2024-03-27T21:40:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.810+0000 7f599a355640 1 -- 172.21.15.60:0/1529533293 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f59941efd50 con 0x7f59941ed420 2024-03-27T21:40:56.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.812+0000 7f5990ff9640 1 -- 172.21.15.60:0/1529533293 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f59941efd50 con 0x7f59941ed420 2024-03-27T21:40:56.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.812+0000 7f5990ff9640 10 client.4646.objecter ms_dispatch 0x7f5994000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:56.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.812+0000 7f5990ff9640 10 client.4646.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:56.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.812+0000 7f5990ff9640 20 client.4646.objecter put_session s=0x7f59941ecda0 osd=6 4 2024-03-27T21:40:56.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.812+0000 7f5990ff9640 15 client.4646.objecter _session_command_op_remove 6 13 2024-03-27T21:40:56.799 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:56.800 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:56.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.812+0000 7f599a355640 10 client.4646.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:56.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.812+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign -1 14 2024-03-27T21:40:56.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.812+0000 7f599a355640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f59941f03d0 0x7f59941f27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:56.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.812+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f59941efd50 osd=7 3 2024-03-27T21:40:56.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.812+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f59941efd50 osd=7 3 2024-03-27T21:40:56.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.812+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f59941efd50 osd=7 3 2024-03-27T21:40:56.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.812+0000 7f599a355640 15 client.4646.objecter _session_command_op_remove -1 14 2024-03-27T21:40:56.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.812+0000 7f599a355640 20 client.4646.objecter get_session s=0x7f59941efd50 osd=7 3 2024-03-27T21:40:56.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.812+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign 7 14 2024-03-27T21:40:56.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.812+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f59941efd50 osd=7 4 2024-03-27T21:40:56.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.812+0000 7f599a355640 10 client.4646.objecter _send_command 14 2024-03-27T21:40:56.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.812+0000 7f599a355640 1 -- 172.21.15.60:0/1529533293 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f59941f5590 con 0x7f59941f03d0 2024-03-27T21:40:56.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.813+0000 7f59988f2640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f59941f03d0 0x7f59941f27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:56.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.813+0000 7f59988f2640 10 monclient: get_auth_request con 0x7f59941f03d0 auth_method 0 2024-03-27T21:40:56.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.813+0000 7f59988f2640 1 --2- 172.21.15.60:0/1529533293 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f59941f03d0 0x7f59941f27c0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:56.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.813+0000 7f5990ff9640 10 client.4646.objecter ms_handle_connect 0x7f59941f03d0 2024-03-27T21:40:56.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.814+0000 7f5990ff9640 1 -- 172.21.15.60:0/1529533293 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f59941f5590 con 0x7f59941f03d0 2024-03-27T21:40:56.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.814+0000 7f5990ff9640 10 client.4646.objecter ms_dispatch 0x7f5994000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:40:56.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.814+0000 7f5990ff9640 10 client.4646.objecter _finish_command 14 = system:0 2024-03-27T21:40:56.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.814+0000 7f5990ff9640 20 client.4646.objecter put_session s=0x7f59941efd50 osd=7 4 2024-03-27T21:40:56.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.814+0000 7f5990ff9640 15 client.4646.objecter _session_command_op_remove 7 14 2024-03-27T21:40:56.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.827+0000 7f599a355640 10 client.4646.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:40:56.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.827+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign -1 15 2024-03-27T21:40:56.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.827+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f59941efd50 osd=7 3 2024-03-27T21:40:56.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.827+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f59941efd50 osd=7 3 2024-03-27T21:40:56.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.827+0000 7f599a355640 20 client.4646.objecter _get_session s=0x7f59941efd50 osd=7 3 2024-03-27T21:40:56.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.827+0000 7f599a355640 15 client.4646.objecter _session_command_op_remove -1 15 2024-03-27T21:40:56.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.827+0000 7f599a355640 20 client.4646.objecter get_session s=0x7f59941efd50 osd=7 3 2024-03-27T21:40:56.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.827+0000 7f599a355640 15 client.4646.objecter _session_command_op_assign 7 15 2024-03-27T21:40:56.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.827+0000 7f599a355640 20 client.4646.objecter put_session s=0x7f59941efd50 osd=7 4 2024-03-27T21:40:56.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.827+0000 7f599a355640 10 client.4646.objecter _send_command 15 2024-03-27T21:40:56.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.827+0000 7f599a355640 1 -- 172.21.15.60:0/1529533293 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f59941f2d00 con 0x7f59941f03d0 2024-03-27T21:40:56.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.829+0000 7f5990ff9640 1 -- 172.21.15.60:0/1529533293 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f59941f2d00 con 0x7f59941f03d0 2024-03-27T21:40:56.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.829+0000 7f5990ff9640 10 client.4646.objecter ms_dispatch 0x7f5994000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:56.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.829+0000 7f5990ff9640 10 client.4646.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:56.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.829+0000 7f5990ff9640 20 client.4646.objecter put_session s=0x7f59941efd50 osd=7 4 2024-03-27T21:40:56.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:56.829+0000 7f5990ff9640 15 client.4646.objecter _session_command_op_remove 7 15 2024-03-27T21:40:56.816 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:56.816 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:56.819 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:40:56.819 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:40:56.870 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:56.884+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:56.932 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:56.932 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:56.932 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:56.932 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:56.932 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:56.938 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:56.952+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:56.941 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:40:57.039 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:57.053+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:57.086 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:57.086 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:57.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:57.086 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:57.087 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:57.095 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:40:57.140 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:57.154+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:57.199 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:57.199 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:57.199 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:57.199 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:57.208 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:40:57.251 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:57.265+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:57.312 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:57.313 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:57.313 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:57.313 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:57.313 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:57.321 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:40:57.363 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:57.376+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:57.442 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:57.456+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:57.467 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:57.467 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:57.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:57.467 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:57.468 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:57.476 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:40:57.560 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:57.574+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:57.622 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:57.622 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:57.622 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:57.622 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:57.631 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:40:57.673 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:57.686+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:57.745 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:57.759+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:57.776 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:57.777 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:57.777 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:57.777 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:57.777 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:57.785 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:40:57.869 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:57.883+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:57.930 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:57.931 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:57.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:57.931 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:40:57.931 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:57.939 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:40:57.946 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-27T21:40:57.947 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:57.961+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:58.048 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:58.062+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:58.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.072+0000 7f85bc9eb640 10 monclient: get_monmap_and_config 2024-03-27T21:40:58.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.072+0000 7f85bc9eb640 10 monclient: build_initial_monmap 2024-03-27T21:40:58.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.072+0000 7f85bc9eb640 10 monclient: monmap: 2024-03-27T21:40:58.059 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:58.059 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:58.059 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:58.073502+0000 2024-03-27T21:40:58.059 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:58.073502+0000 2024-03-27T21:40:58.059 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:58.059 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:58.059 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:58.059 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:58.059 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:58.059 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:58.059 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:58.059 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:58.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.073+0000 7f85bc9eb640 1 Processor -- start 2024-03-27T21:40:58.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.073+0000 7f85bc9eb640 1 -- start start 2024-03-27T21:40:58.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.073+0000 7f85bc9eb640 10 monclient: init 2024-03-27T21:40:58.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.073+0000 7f85bc9eb640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:58.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.073+0000 7f85bc9eb640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-03-27T21:40:58.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.073+0000 7f85bc9eb640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f85b80b40b0 0x7f85b80a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:58.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.073+0000 7f85bc9eb640 10 monclient(hunting): picked mon.noname-c con 0x7f85b80b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:58.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.073+0000 7f85bc9eb640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f85b813b020 0x7f85b813b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:58.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.073+0000 7f85bc9eb640 10 monclient(hunting): picked mon.noname-b con 0x7f85b813b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:58.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.073+0000 7f85bc9eb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f85b81416f0 0x7f85b8141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:58.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.073+0000 7f85bc9eb640 10 monclient(hunting): picked mon.noname-a con 0x7f85b81416f0 addr v2:172.21.15.60:3300/0 2024-03-27T21:40:58.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.073+0000 7f85bc9eb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:58.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.073+0000 7f85bc9eb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85b80b4490 con 0x7f85b81416f0 2024-03-27T21:40:58.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.073+0000 7f85bc9eb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:58.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.073+0000 7f85bc9eb640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85b80b3830 con 0x7f85b80b40b0 2024-03-27T21:40:58.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.073+0000 7f85bc9eb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:58.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.073+0000 7f85bc9eb640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b80aa520 con 0x7f85b813b020 2024-03-27T21:40:58.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.073+0000 7f85bc9eb640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:58.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.073+0000 7f85bc9eb640 10 monclient(hunting): authenticate will time out at 899.294823s 2024-03-27T21:40:58.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b6d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f85b81416f0 0x7f85b8141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:58.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b5d9b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f85b813b020 0x7f85b813b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:58.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b6d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f85b81416f0 0x7f85b8141b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47210/0 (socket says 172.21.15.60:47210) 2024-03-27T21:40:58.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b6d9d640 1 -- 172.21.15.60:0/3210612310 learned_addr learned my addr 172.21.15.60:0/3210612310 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:58.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b6d9d640 10 monclient(hunting): get_auth_request con 0x7f85b81416f0 auth_method 0 2024-03-27T21:40:58.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:58.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b6d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:58.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:58.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b5d9b640 10 monclient(hunting): get_auth_request con 0x7f85b813b020 auth_method 0 2024-03-27T21:40:58.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:58.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:58.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:58.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b659c640 1 --2- 172.21.15.60:0/3210612310 >> v2:172.21.15.191:3300/0 conn(0x7f85b80b40b0 0x7f85b80a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:58.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:58.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:58.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:58.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:58.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:58.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:58.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b659c640 10 monclient(hunting): get_auth_request con 0x7f85b80b40b0 auth_method 0 2024-03-27T21:40:58.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:58.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b659c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:58.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:58.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:58.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:58.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:58.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b5d9b640 10 monclient(hunting): handle_auth_done global_id 4649 payload 274 2024-03-27T21:40:58.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b5d9b640 1 -- 172.21.15.60:0/3210612310 >> v2:172.21.15.191:3300/0 conn(0x7f85b80b40b0 msgr2=0x7f85b80a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:58.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b5d9b640 1 --2- 172.21.15.60:0/3210612310 >> v2:172.21.15.191:3300/0 conn(0x7f85b80b40b0 0x7f85b80a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:58.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b5d9b640 1 -- 172.21.15.60:0/3210612310 >> v2:172.21.15.60:3300/0 conn(0x7f85b81416f0 msgr2=0x7f85b8141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:40:58.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b5d9b640 1 --2- 172.21.15.60:0/3210612310 >> v2:172.21.15.60:3300/0 conn(0x7f85b81416f0 0x7f85b8141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:58.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b5d9b640 10 monclient: _finish_hunting 0 2024-03-27T21:40:58.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b5d9b640 1 monclient: found mon.noname-b 2024-03-27T21:40:58.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:58.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:40:58.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b5d9b640 1 -- 172.21.15.60:0/3210612310 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b80a9470 con 0x7f85b813b020 2024-03-27T21:40:58.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.074+0000 7f85b6d9d640 1 --2- 172.21.15.60:0/3210612310 >> v2:172.21.15.60:3300/0 conn(0x7f85b81416f0 0x7f85b8141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:58.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85b5d9b640 1 --2- 172.21.15.60:0/3210612310 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f85b813b020 0x7f85b813b480 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f85b80b4810 tx=0x7f85ac02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34e3a927c962ae4 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:58.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85b559a640 1 -- 172.21.15.60:0/3210612310 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac02e070 con 0x7f85b813b020 2024-03-27T21:40:58.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85b559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:58.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85b559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:40:58.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85b559a640 10 monclient: dump: 2024-03-27T21:40:58.073 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:58.073 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:58.073 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:58.073 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:58.073 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:58.073 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:58.073 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:58.073 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:58.073 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:58.073 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:58.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85b559a640 10 monclient: _finish_auth 0 2024-03-27T21:40:58.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85b559a640 10 monclient: _check_auth_tickets 2024-03-27T21:40:58.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85b559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:58.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85b559a640 1 -- 172.21.15.60:0/3210612310 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85ac0315a0 con 0x7f85b813b020 2024-03-27T21:40:58.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85bc9eb640 5 monclient: authenticate success, global_id 4649 2024-03-27T21:40:58.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85bc9eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:58.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85b559a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:58.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85b559a640 1 -- 172.21.15.60:0/3210612310 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac031930 con 0x7f85b813b020 2024-03-27T21:40:58.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85bc9eb640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:58.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85bc9eb640 4 set_mon_vals no callback set 2024-03-27T21:40:58.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85b559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:58.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85b559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:40:58.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85b559a640 10 monclient: dump: 2024-03-27T21:40:58.074 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:58.074 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:58.074 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:58.074 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:58.074 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:58.074 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:58.074 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:58.075 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:58.075 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:58.075 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:58.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85bc9eb640 10 monclient: shutdown 2024-03-27T21:40:58.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85bc9eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:58.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85bc9eb640 1 -- 172.21.15.60:0/3210612310 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f85b813b020 msgr2=0x7f85b813b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:58.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85bc9eb640 1 --2- 172.21.15.60:0/3210612310 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f85b813b020 0x7f85b813b480 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f85b80b4810 tx=0x7f85ac02b840 comp rx=0 tx=0).stop 2024-03-27T21:40:58.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85bc9eb640 1 -- 172.21.15.60:0/3210612310 shutdown_connections 2024-03-27T21:40:58.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85bc9eb640 1 --2- 172.21.15.60:0/3210612310 >> v2:172.21.15.60:3300/0 conn(0x7f85b81416f0 0x7f85b8141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:58.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85bc9eb640 1 --2- 172.21.15.60:0/3210612310 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f85b813b020 0x7f85b813b480 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:58.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85bc9eb640 1 --2- 172.21.15.60:0/3210612310 >> v2:172.21.15.191:3300/0 conn(0x7f85b80b40b0 0x7f85b80a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:58.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85bc9eb640 1 -- 172.21.15.60:0/3210612310 >> 172.21.15.60:0/3210612310 conn(0x7f85b80608e0 msgr2=0x7f85b8060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:58.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85bc9eb640 1 -- 172.21.15.60:0/3210612310 shutdown_connections 2024-03-27T21:40:58.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.075+0000 7f85bc9eb640 1 -- 172.21.15.60:0/3210612310 wait complete. 2024-03-27T21:40:58.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.076+0000 7f85bc9eb640 10 monclient: build_initial_monmap 2024-03-27T21:40:58.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.076+0000 7f85bc9eb640 10 monclient: monmap: 2024-03-27T21:40:58.076 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:58.076 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:58.076 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:58.076 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:58.076 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:58.076 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:58.076 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:58.076 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:58.076 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:58.076 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:58.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.076+0000 7f85bc9eb640 1 librados: starting msgr at 2024-03-27T21:40:58.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.076+0000 7f85bc9eb640 1 librados: starting objecter 2024-03-27T21:40:58.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85bc9eb640 1 Processor -- start 2024-03-27T21:40:58.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85bc9eb640 1 -- start start 2024-03-27T21:40:58.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85bc9eb640 1 librados: setting wanted keys 2024-03-27T21:40:58.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85bc9eb640 1 librados: calling monclient init 2024-03-27T21:40:58.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85bc9eb640 10 monclient: init 2024-03-27T21:40:58.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85bc9eb640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:58.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85bc9eb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:40:58.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85bc9eb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f85b80b40b0 0x7f85b8139f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:58.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85bc9eb640 10 monclient(hunting): picked mon.noname-a con 0x7f85b80b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:58.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85bc9eb640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85b813b020 0x7f85b8143ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:58.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85bc9eb640 10 monclient(hunting): picked mon.noname-b con 0x7f85b813b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:58.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85bc9eb640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f85b81416f0 0x7f85b81d6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:58.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85bc9eb640 10 monclient(hunting): picked mon.noname-c con 0x7f85b81416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:58.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85bc9eb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:58.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85bc9eb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b81d7320 con 0x7f85b80b40b0 2024-03-27T21:40:58.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85bc9eb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:58.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85bc9eb640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b81d9900 con 0x7f85b813b020 2024-03-27T21:40:58.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85bc9eb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:58.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85bc9eb640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b80ab980 con 0x7f85b81416f0 2024-03-27T21:40:58.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85bc9eb640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:58.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85bc9eb640 10 monclient(hunting): authenticate will time out at 899.298596s 2024-03-27T21:40:58.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f85b80b40b0 0x7f85b8139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:58.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f85b80b40b0 0x7f85b8139f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47220/0 (socket says 172.21.15.60:47220) 2024-03-27T21:40:58.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b659c640 1 -- 172.21.15.60:0/778281124 learned_addr learned my addr 172.21.15.60:0/778281124 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:58.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b6d9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f85b81416f0 0x7f85b81d6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:58.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b659c640 10 monclient(hunting): get_auth_request con 0x7f85b80b40b0 auth_method 0 2024-03-27T21:40:58.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:58.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b659c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:58.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:58.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b6d9d640 10 monclient(hunting): get_auth_request con 0x7f85b81416f0 auth_method 0 2024-03-27T21:40:58.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:58.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b6d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:58.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:58.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:58.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:58.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:58.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:58.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:58.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b5d9b640 1 --2- 172.21.15.60:0/778281124 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85b813b020 0x7f85b8143ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:58.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:58.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b5d9b640 10 monclient(hunting): get_auth_request con 0x7f85b813b020 auth_method 0 2024-03-27T21:40:58.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:58.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:58.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:58.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.077+0000 7f85b5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:58.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85b5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:58.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85b5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:58.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85b659c640 10 monclient(hunting): handle_auth_done global_id 4809 payload 824 2024-03-27T21:40:58.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85b659c640 1 -- 172.21.15.60:0/778281124 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f85b81416f0 msgr2=0x7f85b81d6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:58.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85b659c640 1 --2- 172.21.15.60:0/778281124 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f85b81416f0 0x7f85b81d6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:58.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85b659c640 1 -- 172.21.15.60:0/778281124 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85b813b020 msgr2=0x7f85b8143ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:58.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85b659c640 1 --2- 172.21.15.60:0/778281124 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85b813b020 0x7f85b8143ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:58.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85b659c640 10 monclient: _finish_hunting 0 2024-03-27T21:40:58.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85b659c640 1 monclient: found mon.noname-a 2024-03-27T21:40:58.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85b659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:58.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85b659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:58.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85b659c640 1 -- 172.21.15.60:0/778281124 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b0009b30 con 0x7f85b80b40b0 2024-03-27T21:40:58.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85b6d9d640 1 --2- 172.21.15.60:0/778281124 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f85b81416f0 0x7f85b81d6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:40:58.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85b659c640 1 --2- 172.21.15.60:0/778281124 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f85b80b40b0 0x7f85b8139f70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f85a800e9b0 tx=0x7f85a800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:58.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85a6ffd640 10 client.?.objecter ms_handle_connect 0x7f85b80b40b0 2024-03-27T21:40:58.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:58.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85a6ffd640 1 -- 172.21.15.60:0/778281124 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a8009670 con 0x7f85b80b40b0 2024-03-27T21:40:58.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:58.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:58.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85a6ffd640 10 monclient: dump: 2024-03-27T21:40:58.081 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:58.082 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:58.082 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:58.082 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:58.082 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:58.082 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:58.082 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:58.082 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:58.082 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:58.082 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:58.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85a6ffd640 10 monclient: _finish_auth 0 2024-03-27T21:40:58.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85a6ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:40:58.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:58.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85bc9eb640 5 monclient: authenticate success, global_id 4809 2024-03-27T21:40:58.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85a6ffd640 1 -- 172.21.15.60:0/778281124 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a800c9a0 con 0x7f85b80b40b0 2024-03-27T21:40:58.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:58.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85a6ffd640 1 -- 172.21.15.60:0/778281124 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a800cd50 con 0x7f85b80b40b0 2024-03-27T21:40:58.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:58.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:58.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85a6ffd640 10 monclient: dump: 2024-03-27T21:40:58.083 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:58.083 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:58.083 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:58.083 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:58.083 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:58.083 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:58.083 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:58.083 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:58.084 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:58.084 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:58.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85bc9eb640 10 monclient: _renew_subs 2024-03-27T21:40:58.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85b4d99640 4 set_mon_vals no callback set 2024-03-27T21:40:58.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85bc9eb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:58.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85bc9eb640 1 -- 172.21.15.60:0/778281124 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ac009400 con 0x7f85b80b40b0 2024-03-27T21:40:58.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85bc9eb640 10 client.4809.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:58.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85bc9eb640 10 monclient: _renew_subs 2024-03-27T21:40:58.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85bc9eb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:58.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85bc9eb640 1 -- 172.21.15.60:0/778281124 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b81d9ba0 con 0x7f85b80b40b0 2024-03-27T21:40:58.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.078+0000 7f85bc9eb640 1 librados: init done 2024-03-27T21:40:58.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.079+0000 7f85bc9eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:58.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.079+0000 7f85bc9eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:58.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.079+0000 7f85bc9eb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:58.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.079+0000 7f85bc9eb640 1 -- 172.21.15.60:0/778281124 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b8069470 con 0x7f85b80b40b0 2024-03-27T21:40:58.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.079+0000 7f85a6ffd640 1 -- 172.21.15.60:0/778281124 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f85a8013470 con 0x7f85b80b40b0 2024-03-27T21:40:58.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.080+0000 7f85a6ffd640 1 --2- 172.21.15.60:0/778281124 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f859806c590 0x7f859806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:58.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.080+0000 7f85b5d9b640 1 --2- 172.21.15.60:0/778281124 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f859806c590 0x7f859806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:58.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.080+0000 7f85a6ffd640 1 -- 172.21.15.60:0/778281124 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f85a8089370 con 0x7f85b80b40b0 2024-03-27T21:40:58.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.080+0000 7f85a6ffd640 10 client.4809.objecter ms_dispatch 0x7f85b8000b90 osd_map(48..48 src has 1..48) v4 2024-03-27T21:40:58.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.080+0000 7f85a6ffd640 3 client.4809.objecter handle_osd_map got epochs [48,48] > 0 2024-03-27T21:40:58.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.080+0000 7f85a6ffd640 3 client.4809.objecter handle_osd_map decoding full epoch 48 2024-03-27T21:40:58.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.080+0000 7f85b5d9b640 10 monclient: get_auth_request con 0x7f859806c590 auth_method 0 2024-03-27T21:40:58.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.080+0000 7f85a6ffd640 20 client.4809.objecter dump_active .. 0 homeless 2024-03-27T21:40:58.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.080+0000 7f85a6ffd640 10 client.4809.objecter ms_handle_connect 0x7f859806c590 2024-03-27T21:40:58.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.080+0000 7f85b5d9b640 1 --2- 172.21.15.60:0/778281124 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f859806c590 0x7f859806ea50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f85ac036000 tx=0x7f85ac03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:58.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.085+0000 7f85a6ffd640 1 -- 172.21.15.60:0/778281124 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f85a8054740 con 0x7f85b80b40b0 2024-03-27T21:40:58.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.085+0000 7f85a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:58.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.085+0000 7f85a6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:58.086 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:58.087 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:58.087 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:40:58.087 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:58.094 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:40:58.149 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:58.163+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:58.239 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:58.239 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:40:58.239 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:40:58.239 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:40:58.239 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:58.247 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:40:58.249 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:58.263+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:58.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.295+0000 7f85bc9eb640 10 start_mon_command cmd=[{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:40:58.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.295+0000 7f85bc9eb640 10 monclient: _send_command 2 [{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:40:58.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.295+0000 7f85bc9eb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:58.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.295+0000 7f85bc9eb640 1 -- 172.21.15.60:0/778281124 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f85b813b480 con 0x7f85b80b40b0 2024-03-27T21:40:58.350 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:58.364+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:58.353 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:40:58.354 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:40:58.354 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:40:58.354 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-03-27T21:40:58.354 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.354 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:58.354 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:58.354 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 78.934513342000002, 2024-03-27T21:40:58.354 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43537978700000002, 2024-03-27T21:40:58.354 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:58.354 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.354 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:58.354 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:58.355 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:58.355 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-03-27T21:40:58.355 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.355 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:58.355 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.355 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:58.355 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:58.355 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.355 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.355 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.355 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:58.355 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:40:58.355 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.355 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:58.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432019+0000", 2024-03-27T21:40:58.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-03-27T21:40:58.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:58.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432027+0000", 2024-03-27T21:40:58.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1200000000000002e-06 2024-03-27T21:40:58.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:58.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432039+0000", 2024-03-27T21:40:58.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1123e-05 2024-03-27T21:40:58.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432051+0000", 2024-03-27T21:40:58.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-03-27T21:40:58.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:58.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432157+0000", 2024-03-27T21:40:58.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106846 2024-03-27T21:40:58.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-03-27T21:40:58.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432176+0000", 2024-03-27T21:40:58.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8490000000000001e-05 2024-03-27T21:40:58.358 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.358 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:58.358 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866170+0000", 2024-03-27T21:40:58.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43399370100000001 2024-03-27T21:40:58.358 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.358 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:58.358 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866335+0000", 2024-03-27T21:40:58.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016559699999999999 2024-03-27T21:40:58.358 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.358 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:58.359 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866495+0000", 2024-03-27T21:40:58.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015944200000000001 2024-03-27T21:40:58.359 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.359 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:58.359 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867012+0000", 2024-03-27T21:40:58.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0005176 2024-03-27T21:40:58.359 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.359 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:58.359 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867334+0000", 2024-03-27T21:40:58.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032158600000000001 2024-03-27T21:40:58.359 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.360 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:58.360 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867374+0000", 2024-03-27T21:40:58.360 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0547000000000003e-05 2024-03-27T21:40:58.360 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.360 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.360 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:58.360 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867400+0000", 2024-03-27T21:40:58.360 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5958000000000001e-05 2024-03-27T21:40:58.360 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.360 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:58.360 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.360 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.360 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.360 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:58.361 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:58.361 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 78.437130890999995, 2024-03-27T21:40:58.361 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064560310000000001, 2024-03-27T21:40:58.361 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:58.361 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.361 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:58.361 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:58.361 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:58.361 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 70 2024-03-27T21:40:58.361 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.361 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:58.361 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.361 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:58.361 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:58.361 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.361 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.362 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.362 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:58.362 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:40:58.362 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.362 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.362 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.362 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:58.362 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929402+0000", 2024-03-27T21:40:58.362 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.362 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.362 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.362 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:58.362 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929407+0000", 2024-03-27T21:40:58.362 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-06 2024-03-27T21:40:58.362 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.363 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.363 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:58.363 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929414+0000", 2024-03-27T21:40:58.363 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5689999999999999e-06 2024-03-27T21:40:58.363 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.363 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.363 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.363 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929419+0000", 2024-03-27T21:40:58.363 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-03-27T21:40:58.363 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.363 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.363 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:58.363 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930245+0000", 2024-03-27T21:40:58.363 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082521500000000002 2024-03-27T21:40:58.364 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.364 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.364 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:58.364 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930287+0000", 2024-03-27T21:40:58.364 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2305000000000002e-05 2024-03-27T21:40:58.364 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.364 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.364 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:58.364 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930367+0000", 2024-03-27T21:40:58.364 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9589e-05 2024-03-27T21:40:58.364 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.364 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.364 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:58.364 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.931214+0000", 2024-03-27T21:40:58.364 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000847827 2024-03-27T21:40:58.364 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.365 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.365 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:58.365 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935807+0000", 2024-03-27T21:40:58.365 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045923780000000003 2024-03-27T21:40:58.365 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.365 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.365 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:58.365 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935840+0000", 2024-03-27T21:40:58.365 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3501999999999999e-05 2024-03-27T21:40:58.365 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.365 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.365 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:58.365 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935859+0000", 2024-03-27T21:40:58.365 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8845e-05 2024-03-27T21:40:58.366 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.366 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:58.366 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.366 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.366 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.366 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:58.366 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:58.366 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 78.429871300000002, 2024-03-27T21:40:58.366 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071695429999999996, 2024-03-27T21:40:58.366 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:58.366 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.366 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:58.366 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:58.366 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:58.367 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-03-27T21:40:58.367 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.367 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:58.367 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.367 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:58.367 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:58.367 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.367 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.367 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.367 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:58.367 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:40:58.367 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.367 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.367 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.367 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:58.368 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936662+0000", 2024-03-27T21:40:58.368 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.368 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.368 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.368 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:58.368 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936737+0000", 2024-03-27T21:40:58.368 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5303999999999996e-05 2024-03-27T21:40:58.368 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.368 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.368 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:58.368 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936746+0000", 2024-03-27T21:40:58.368 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6340000000000007e-06 2024-03-27T21:40:58.368 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.368 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.369 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.369 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936753+0000", 2024-03-27T21:40:58.369 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-03-27T21:40:58.369 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.369 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.369 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:58.369 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937926+0000", 2024-03-27T21:40:58.369 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011731280000000001 2024-03-27T21:40:58.369 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.369 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.369 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:58.369 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937969+0000", 2024-03-27T21:40:58.369 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3285999999999999e-05 2024-03-27T21:40:58.369 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.369 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.370 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:58.370 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938040+0000", 2024-03-27T21:40:58.370 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0358000000000006e-05 2024-03-27T21:40:58.370 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.370 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.370 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:58.370 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939324+0000", 2024-03-27T21:40:58.370 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012844899999999999 2024-03-27T21:40:58.370 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.370 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.370 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:58.370 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943780+0000", 2024-03-27T21:40:58.370 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044555630000000001 2024-03-27T21:40:58.371 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.371 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.371 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:58.371 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943811+0000", 2024-03-27T21:40:58.371 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1415999999999999e-05 2024-03-27T21:40:58.371 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.371 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.371 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:58.371 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943832+0000", 2024-03-27T21:40:58.371 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0981000000000001e-05 2024-03-27T21:40:58.371 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.371 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:58.371 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.371 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.371 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.372 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:58.372 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:58.372 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 78.429773284000007, 2024-03-27T21:40:58.372 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071139740000000003, 2024-03-27T21:40:58.374 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:58.374 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.374 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:58.374 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:58.374 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:58.374 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-03-27T21:40:58.374 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.374 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:58.374 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.374 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:58.374 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:58.375 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.375 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.375 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.375 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:58.375 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:40:58.375 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.375 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.375 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.375 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:58.375 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936760+0000", 2024-03-27T21:40:58.375 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.375 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.375 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.376 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:58.376 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936765+0000", 2024-03-27T21:40:58.376 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6330000000000004e-06 2024-03-27T21:40:58.376 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.376 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.376 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:58.376 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936771+0000", 2024-03-27T21:40:58.376 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0560000000000001e-06 2024-03-27T21:40:58.376 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.376 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.376 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.376 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936776+0000", 2024-03-27T21:40:58.376 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1510000000000003e-06 2024-03-27T21:40:58.377 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.377 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.377 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:58.377 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938605+0000", 2024-03-27T21:40:58.377 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001829589 2024-03-27T21:40:58.377 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.377 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.377 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:58.377 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938657+0000", 2024-03-27T21:40:58.377 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1860000000000002e-05 2024-03-27T21:40:58.377 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.377 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.378 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:58.378 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938734+0000", 2024-03-27T21:40:58.378 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6639999999999998e-05 2024-03-27T21:40:58.378 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.378 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.378 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:58.378 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939562+0000", 2024-03-27T21:40:58.378 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082803100000000001 2024-03-27T21:40:58.378 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.378 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.378 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:58.378 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943838+0000", 2024-03-27T21:40:58.378 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042757050000000003 2024-03-27T21:40:58.378 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.379 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.379 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:58.379 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943859+0000", 2024-03-27T21:40:58.379 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1175e-05 2024-03-27T21:40:58.379 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.379 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.379 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:58.379 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943875+0000", 2024-03-27T21:40:58.379 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5744e-05 2024-03-27T21:40:58.379 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.379 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:58.379 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.379 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.379 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.380 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:58.380 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:58.380 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 78.286259232999996, 2024-03-27T21:40:58.380 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059712230000000003, 2024-03-27T21:40:58.380 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:58.380 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.380 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:58.380 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:58.380 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:58.381 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 112 2024-03-27T21:40:58.381 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.381 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:58.381 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.381 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:58.381 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:58.381 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.381 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.381 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.381 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:58.381 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:40:58.381 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.381 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.382 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.382 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:58.382 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080274+0000", 2024-03-27T21:40:58.382 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.382 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.382 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.382 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:58.382 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080318+0000", 2024-03-27T21:40:58.382 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3634999999999998e-05 2024-03-27T21:40:58.382 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.382 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.382 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:58.382 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080329+0000", 2024-03-27T21:40:58.382 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1232e-05 2024-03-27T21:40:58.383 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.383 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.383 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080333+0000", 2024-03-27T21:40:58.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6009999999999999e-06 2024-03-27T21:40:58.383 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.383 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:58.383 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083786+0000", 2024-03-27T21:40:58.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034528940000000002 2024-03-27T21:40:58.383 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.383 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:58.384 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083861+0000", 2024-03-27T21:40:58.384 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4893000000000003e-05 2024-03-27T21:40:58.384 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.384 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.384 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:58.384 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083981+0000", 2024-03-27T21:40:58.384 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012058299999999999 2024-03-27T21:40:58.384 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.384 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.384 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:58.384 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086148+0000", 2024-03-27T21:40:58.384 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021671469999999999 2024-03-27T21:40:58.385 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.385 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.385 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:58.385 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086195+0000", 2024-03-27T21:40:58.385 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6768999999999999e-05 2024-03-27T21:40:58.385 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.385 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.385 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:58.385 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086224+0000", 2024-03-27T21:40:58.385 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8758000000000001e-05 2024-03-27T21:40:58.386 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.386 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.386 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:58.386 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086246+0000", 2024-03-27T21:40:58.386 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2064000000000001e-05 2024-03-27T21:40:58.386 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.386 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:58.386 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.386 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.386 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.386 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:58.386 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:58.386 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 78.286196286000006, 2024-03-27T21:40:58.386 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066196320000000003, 2024-03-27T21:40:58.387 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:58.387 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.387 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:58.387 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:58.387 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:58.387 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113 2024-03-27T21:40:58.387 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.387 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:58.387 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.387 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:58.387 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:58.387 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.387 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.388 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.388 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:58.388 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:40:58.388 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.388 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.388 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.388 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:58.388 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080337+0000", 2024-03-27T21:40:58.389 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.389 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.389 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.389 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:58.389 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080340+0000", 2024-03-27T21:40:58.389 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6690000000000002e-06 2024-03-27T21:40:58.389 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.389 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.389 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:58.389 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080344+0000", 2024-03-27T21:40:58.389 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6550000000000002e-06 2024-03-27T21:40:58.389 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.389 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.389 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.390 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080348+0000", 2024-03-27T21:40:58.390 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9169999999999999e-06 2024-03-27T21:40:58.390 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.390 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.390 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:58.390 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084595+0000", 2024-03-27T21:40:58.390 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042471719999999996 2024-03-27T21:40:58.390 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.390 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.390 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:58.390 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084676+0000", 2024-03-27T21:40:58.390 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1147000000000001e-05 2024-03-27T21:40:58.390 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.391 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.391 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:58.391 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084796+0000", 2024-03-27T21:40:58.391 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012032900000000001 2024-03-27T21:40:58.391 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.391 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.391 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:58.391 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086190+0000", 2024-03-27T21:40:58.391 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013934399999999999 2024-03-27T21:40:58.391 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.391 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.391 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:58.391 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086905+0000", 2024-03-27T21:40:58.392 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071536199999999999 2024-03-27T21:40:58.392 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.392 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.392 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:58.392 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086937+0000", 2024-03-27T21:40:58.392 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2125000000000002e-05 2024-03-27T21:40:58.392 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.392 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.392 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:58.392 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086957+0000", 2024-03-27T21:40:58.392 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0157999999999999e-05 2024-03-27T21:40:58.392 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.392 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:58.392 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.392 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.393 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.393 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:58.393 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:58.393 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 78.286183570000006, 2024-03-27T21:40:58.393 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007405661, 2024-03-27T21:40:58.393 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:58.393 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.393 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:58.393 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:58.393 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:58.393 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 114 2024-03-27T21:40:58.393 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.393 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:58.393 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.394 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:58.394 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:58.394 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.394 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.394 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.394 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:58.394 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:58.394 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.394 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.394 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.394 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:58.394 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:40:58.395 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-03-27T21:40:58.395 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.395 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.395 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:58.395 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080353+0000", 2024-03-27T21:40:58.395 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.441e-06 2024-03-27T21:40:58.395 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.395 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.395 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:58.395 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080356+0000", 2024-03-27T21:40:58.395 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0769999999999999e-06 2024-03-27T21:40:58.395 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.395 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.396 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.396 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080358+0000", 2024-03-27T21:40:58.396 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3369999999999998e-06 2024-03-27T21:40:58.396 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.396 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.396 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:58.396 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085118+0000", 2024-03-27T21:40:58.396 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047603949999999997 2024-03-27T21:40:58.396 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.396 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.396 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:58.396 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085210+0000", 2024-03-27T21:40:58.396 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1983000000000003e-05 2024-03-27T21:40:58.397 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.397 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.397 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:58.397 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085332+0000", 2024-03-27T21:40:58.397 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121803 2024-03-27T21:40:58.397 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.397 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.397 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:58.397 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086292+0000", 2024-03-27T21:40:58.397 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095961400000000002 2024-03-27T21:40:58.397 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.397 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.397 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:58.398 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087710+0000", 2024-03-27T21:40:58.398 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014181459999999999 2024-03-27T21:40:58.398 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.398 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.398 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:58.398 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087736+0000", 2024-03-27T21:40:58.398 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6475e-05 2024-03-27T21:40:58.398 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.398 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.398 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:58.398 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087756+0000", 2024-03-27T21:40:58.398 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9626000000000001e-05 2024-03-27T21:40:58.398 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.399 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:58.399 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.399 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.399 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.399 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:58.399 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:58.399 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 78.286161135, 2024-03-27T21:40:58.399 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075011720000000004, 2024-03-27T21:40:58.399 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:58.399 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.399 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:58.399 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:58.399 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:58.399 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115 2024-03-27T21:40:58.400 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.400 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:58.400 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.400 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:58.400 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:58.400 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.400 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.400 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.400 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:58.400 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:58.400 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.400 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.400 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.400 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:58.401 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:40:58.401 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.401 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.401 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.401 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:58.401 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080458+0000", 2024-03-27T21:40:58.401 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5228000000000005e-05 2024-03-27T21:40:58.401 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.401 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.401 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:58.401 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080465+0000", 2024-03-27T21:40:58.401 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.238e-06 2024-03-27T21:40:58.402 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.402 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.402 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.402 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080468+0000", 2024-03-27T21:40:58.402 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5109999999999999e-06 2024-03-27T21:40:58.402 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.402 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.402 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:58.402 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086326+0000", 2024-03-27T21:40:58.402 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058579169999999998 2024-03-27T21:40:58.402 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.402 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.402 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:58.403 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086382+0000", 2024-03-27T21:40:58.403 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5309999999999997e-05 2024-03-27T21:40:58.403 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.403 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.403 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:58.403 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086457+0000", 2024-03-27T21:40:58.403 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5413000000000005e-05 2024-03-27T21:40:58.403 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.403 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.403 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:58.403 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087762+0000", 2024-03-27T21:40:58.403 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013052300000000001 2024-03-27T21:40:58.403 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.404 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.404 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:58.404 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087798+0000", 2024-03-27T21:40:58.404 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5614999999999997e-05 2024-03-27T21:40:58.404 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.404 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.404 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:58.404 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087840+0000", 2024-03-27T21:40:58.404 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2196000000000001e-05 2024-03-27T21:40:58.404 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.404 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.404 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:58.404 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087874+0000", 2024-03-27T21:40:58.404 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3785999999999998e-05 2024-03-27T21:40:58.405 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.405 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:58.405 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.405 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.405 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.405 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:58.405 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:58.405 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 78.286061051999994, 2024-03-27T21:40:58.405 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075050680000000002, 2024-03-27T21:40:58.405 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:58.405 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.405 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:58.405 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:58.406 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:58.406 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 116 2024-03-27T21:40:58.406 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.406 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:58.406 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.406 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:58.406 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:58.406 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.406 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.406 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.406 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:58.406 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:40:58.407 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.407 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.407 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.407 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:58.407 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080472+0000", 2024-03-27T21:40:58.407 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.407 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.407 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.407 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:58.407 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080475+0000", 2024-03-27T21:40:58.407 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.712e-06 2024-03-27T21:40:58.407 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.407 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.407 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:58.408 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080478+0000", 2024-03-27T21:40:58.408 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1070000000000001e-06 2024-03-27T21:40:58.408 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.408 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.408 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.408 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080481+0000", 2024-03-27T21:40:58.408 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4269999999999998e-06 2024-03-27T21:40:58.408 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.408 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.408 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:58.408 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086992+0000", 2024-03-27T21:40:58.408 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065114099999999996 2024-03-27T21:40:58.408 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.409 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.409 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:58.409 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087064+0000", 2024-03-27T21:40:58.409 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1927000000000004e-05 2024-03-27T21:40:58.409 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.409 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.409 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:58.409 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087188+0000", 2024-03-27T21:40:58.409 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123585 2024-03-27T21:40:58.409 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.409 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.409 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:58.409 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087766+0000", 2024-03-27T21:40:58.409 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057814300000000004 2024-03-27T21:40:58.410 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.410 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.410 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:58.410 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087934+0000", 2024-03-27T21:40:58.410 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000167894 2024-03-27T21:40:58.410 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.410 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.410 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:58.410 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087958+0000", 2024-03-27T21:40:58.410 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4663000000000001e-05 2024-03-27T21:40:58.411 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.411 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.411 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:58.411 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087978+0000", 2024-03-27T21:40:58.411 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9636e-05 2024-03-27T21:40:58.411 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.411 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:58.411 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.411 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.411 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.411 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:40:58.411 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:58.411 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 78.286028930000001, 2024-03-27T21:40:58.412 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076850679999999998, 2024-03-27T21:40:58.412 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:58.412 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.412 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:58.412 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:58.412 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:58.412 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 117 2024-03-27T21:40:58.412 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.412 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:58.412 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.412 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:58.412 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:58.412 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.413 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.413 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.413 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:58.413 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:58.413 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.413 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.413 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.413 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:58.413 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:40:58.413 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.413 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.414 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.414 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:58.414 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080508+0000", 2024-03-27T21:40:58.414 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.298e-06 2024-03-27T21:40:58.414 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.414 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.414 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:58.414 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080515+0000", 2024-03-27T21:40:58.414 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0790000000000003e-06 2024-03-27T21:40:58.414 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.414 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.414 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.414 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080518+0000", 2024-03-27T21:40:58.415 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8509999999999999e-06 2024-03-27T21:40:58.415 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.415 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.415 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:58.415 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087444+0000", 2024-03-27T21:40:58.415 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0069259120000000002 2024-03-27T21:40:58.415 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.415 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.415 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:58.415 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087491+0000", 2024-03-27T21:40:58.415 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7043999999999999e-05 2024-03-27T21:40:58.415 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.415 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.415 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:58.416 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087564+0000", 2024-03-27T21:40:58.416 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3603999999999995e-05 2024-03-27T21:40:58.416 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.416 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.416 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:58.416 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087902+0000", 2024-03-27T21:40:58.416 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033734100000000001 2024-03-27T21:40:58.416 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.416 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.416 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:58.416 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088117+0000", 2024-03-27T21:40:58.416 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000215164 2024-03-27T21:40:58.417 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.417 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.417 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:58.417 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088160+0000", 2024-03-27T21:40:58.417 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2933999999999999e-05 2024-03-27T21:40:58.417 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.417 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.417 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:58.417 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088190+0000", 2024-03-27T21:40:58.417 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0139999999999999e-05 2024-03-27T21:40:58.417 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.417 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:58.417 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.417 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.418 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.418 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:58.418 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:58.418 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 77.869748541000007, 2024-03-27T21:40:58.418 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052465439999999997, 2024-03-27T21:40:58.418 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:58.418 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.418 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:58.418 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:58.418 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:58.418 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 133 2024-03-27T21:40:58.418 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.418 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:58.418 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.419 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:58.419 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:58.419 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.419 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.419 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.419 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:58.419 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:58.419 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.419 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.419 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.419 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:58.419 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:40:58.419 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.419 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.420 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.420 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:58.420 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496788+0000", 2024-03-27T21:40:58.420 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-03-27T21:40:58.420 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.420 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.420 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:58.420 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496793+0000", 2024-03-27T21:40:58.420 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4790000000000004e-06 2024-03-27T21:40:58.420 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.420 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.420 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.420 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496796+0000", 2024-03-27T21:40:58.421 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1e-06 2024-03-27T21:40:58.421 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.421 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.421 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:58.421 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500447+0000", 2024-03-27T21:40:58.421 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036504459999999999 2024-03-27T21:40:58.421 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.421 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.421 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:58.421 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500492+0000", 2024-03-27T21:40:58.421 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5765e-05 2024-03-27T21:40:58.421 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.421 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.421 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:58.421 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500562+0000", 2024-03-27T21:40:58.421 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9206000000000001e-05 2024-03-27T21:40:58.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.422 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:58.422 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501959+0000", 2024-03-27T21:40:58.422 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001397187 2024-03-27T21:40:58.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.422 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:58.422 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501984+0000", 2024-03-27T21:40:58.422 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.552e-05 2024-03-27T21:40:58.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.422 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:58.422 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502010+0000", 2024-03-27T21:40:58.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5232000000000002e-05 2024-03-27T21:40:58.423 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.423 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.423 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:58.423 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502032+0000", 2024-03-27T21:40:58.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2337999999999999e-05 2024-03-27T21:40:58.423 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.423 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:58.423 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.423 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.423 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.423 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:58.423 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:58.424 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 77.869689883000007, 2024-03-27T21:40:58.424 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052892119999999997, 2024-03-27T21:40:58.424 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:58.424 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.424 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:58.424 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:58.424 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:58.424 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 134 2024-03-27T21:40:58.424 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.424 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:58.424 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.424 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:58.424 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:58.424 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:58.425 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:58.425 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:58.425 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:40:58.425 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:58.425 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496929+0000", 2024-03-27T21:40:58.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5098000000000004e-05 2024-03-27T21:40:58.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:58.426 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496937+0000", 2024-03-27T21:40:58.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9559999999999997e-06 2024-03-27T21:40:58.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.426 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496942+0000", 2024-03-27T21:40:58.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6350000000000003e-06 2024-03-27T21:40:58.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.427 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.427 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:58.427 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500749+0000", 2024-03-27T21:40:58.427 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038068529999999998 2024-03-27T21:40:58.427 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.427 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.427 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:58.427 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500792+0000", 2024-03-27T21:40:58.427 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2823e-05 2024-03-27T21:40:58.427 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.427 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.427 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:58.427 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500864+0000", 2024-03-27T21:40:58.428 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1715e-05 2024-03-27T21:40:58.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.428 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:58.428 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502068+0000", 2024-03-27T21:40:58.428 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012044860000000001 2024-03-27T21:40:58.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.428 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:58.428 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502073+0000", 2024-03-27T21:40:58.428 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5700000000000003e-06 2024-03-27T21:40:58.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.428 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:58.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502098+0000", 2024-03-27T21:40:58.429 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5120999999999999e-05 2024-03-27T21:40:58.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.429 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.429 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:58.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502133+0000", 2024-03-27T21:40:58.429 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5306999999999999e-05 2024-03-27T21:40:58.429 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.429 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:58.429 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.429 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.429 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:58.429 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:58.430 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 77.869584575999994, 2024-03-27T21:40:58.430 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062529040000000001, 2024-03-27T21:40:58.430 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:58.430 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.430 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:58.430 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:58.430 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:58.430 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135 2024-03-27T21:40:58.430 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.430 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:58.430 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.430 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:58.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:58.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.431 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:58.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:58.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.431 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:58.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:40:58.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.431 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:58.432 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496953+0000", 2024-03-27T21:40:58.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0709999999999996e-06 2024-03-27T21:40:58.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:58.432 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496961+0000", 2024-03-27T21:40:58.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1300000000000001e-06 2024-03-27T21:40:58.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.432 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496968+0000", 2024-03-27T21:40:58.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0999999999999998e-06 2024-03-27T21:40:58.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501244+0000", 2024-03-27T21:40:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042756310000000002 2024-03-27T21:40:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501377+0000", 2024-03-27T21:40:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013348799999999999 2024-03-27T21:40:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501455+0000", 2024-03-27T21:40:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7410000000000006e-05 2024-03-27T21:40:58.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502138+0000", 2024-03-27T21:40:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068386599999999999 2024-03-27T21:40:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503152+0000", 2024-03-27T21:40:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010139680000000001 2024-03-27T21:40:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:58.434 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503180+0000", 2024-03-27T21:40:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7426000000000001e-05 2024-03-27T21:40:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503202+0000", 2024-03-27T21:40:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2463000000000002e-05 2024-03-27T21:40:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:58.435 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 77.869560437999993, 2024-03-27T21:40:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066253680000000004, 2024-03-27T21:40:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 136 2024-03-27T21:40:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:58.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:40:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496977+0000", 2024-03-27T21:40:58.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5049999999999999e-06 2024-03-27T21:40:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496984+0000", 2024-03-27T21:40:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1520000000000003e-06 2024-03-27T21:40:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496990+0000", 2024-03-27T21:40:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-03-27T21:40:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:58.438 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501669+0000", 2024-03-27T21:40:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046793859999999998 2024-03-27T21:40:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501714+0000", 2024-03-27T21:40:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5154999999999997e-05 2024-03-27T21:40:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501783+0000", 2024-03-27T21:40:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8454999999999994e-05 2024-03-27T21:40:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503226+0000", 2024-03-27T21:40:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001443441 2024-03-27T21:40:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503540+0000", 2024-03-27T21:40:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031347100000000001 2024-03-27T21:40:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503563+0000", 2024-03-27T21:40:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3223000000000001e-05 2024-03-27T21:40:58.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503599+0000", 2024-03-27T21:40:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6105000000000001e-05 2024-03-27T21:40:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 77.869537628000003, 2024-03-27T21:40:58.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074261320000000002, 2024-03-27T21:40:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 137 2024-03-27T21:40:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:58.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:40:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:58.443 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497077+0000", 2024-03-27T21:40:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1599000000000003e-05 2024-03-27T21:40:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497085+0000", 2024-03-27T21:40:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6109999999999996e-06 2024-03-27T21:40:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497092+0000", 2024-03-27T21:40:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.742e-06 2024-03-27T21:40:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502159+0000", 2024-03-27T21:40:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050666870000000003 2024-03-27T21:40:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502200+0000", 2024-03-27T21:40:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1919999999999998e-05 2024-03-27T21:40:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502280+0000", 2024-03-27T21:40:58.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9417000000000002e-05 2024-03-27T21:40:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503605+0000", 2024-03-27T21:40:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001325641 2024-03-27T21:40:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504361+0000", 2024-03-27T21:40:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075555399999999997 2024-03-27T21:40:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504386+0000", 2024-03-27T21:40:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5369999999999999e-05 2024-03-27T21:40:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504422+0000", 2024-03-27T21:40:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6019000000000002e-05 2024-03-27T21:40:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:58.447 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 77.869433830999995, 2024-03-27T21:40:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074059750000000004, 2024-03-27T21:40:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-03-27T21:40:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.448 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:40:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.449 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497103+0000", 2024-03-27T21:40:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.856e-06 2024-03-27T21:40:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497121+0000", 2024-03-27T21:40:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7190999999999999e-05 2024-03-27T21:40:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497126+0000", 2024-03-27T21:40:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3770000000000004e-06 2024-03-27T21:40:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502833+0000", 2024-03-27T21:40:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057073510000000003 2024-03-27T21:40:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502881+0000", 2024-03-27T21:40:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7961999999999999e-05 2024-03-27T21:40:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502949+0000", 2024-03-27T21:40:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7957e-05 2024-03-27T21:40:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504430+0000", 2024-03-27T21:40:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001480763 2024-03-27T21:40:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504459+0000", 2024-03-27T21:40:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8983999999999999e-05 2024-03-27T21:40:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504486+0000", 2024-03-27T21:40:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6883999999999999e-05 2024-03-27T21:40:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504506+0000", 2024-03-27T21:40:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0214e-05 2024-03-27T21:40:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 77.869401996999997, 2024-03-27T21:40:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074754859999999999, 2024-03-27T21:40:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:58.453 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.454 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:58.454 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:58.454 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:58.454 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-03-27T21:40:58.454 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.454 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:58.454 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.454 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:58.454 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:58.454 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.454 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.454 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.454 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:58.455 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:58.455 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.455 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.455 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.455 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:58.455 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:40:58.455 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.455 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.455 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.455 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:58.455 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497137+0000", 2024-03-27T21:40:58.455 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0730000000000004e-06 2024-03-27T21:40:58.455 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.455 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.455 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:58.456 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497143+0000", 2024-03-27T21:40:58.456 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3759999999999996e-06 2024-03-27T21:40:58.456 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.456 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.456 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.456 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497150+0000", 2024-03-27T21:40:58.456 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.002e-06 2024-03-27T21:40:58.456 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.456 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.456 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:58.456 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503249+0000", 2024-03-27T21:40:58.456 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060989750000000004 2024-03-27T21:40:58.456 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.456 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.457 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:58.457 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503291+0000", 2024-03-27T21:40:58.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1563999999999997e-05 2024-03-27T21:40:58.457 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.457 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.457 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:58.457 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503376+0000", 2024-03-27T21:40:58.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5752000000000002e-05 2024-03-27T21:40:58.457 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.457 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.457 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:58.457 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504545+0000", 2024-03-27T21:40:58.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169172 2024-03-27T21:40:58.457 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:58.458 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504550+0000", 2024-03-27T21:40:58.458 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7419999999999997e-06 2024-03-27T21:40:58.458 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:58.458 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504574+0000", 2024-03-27T21:40:58.458 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4124000000000001e-05 2024-03-27T21:40:58.458 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:58.458 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504607+0000", 2024-03-27T21:40:58.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3118999999999999e-05 2024-03-27T21:40:58.459 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.459 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:58.459 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.459 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.459 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.459 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:58.459 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:58.459 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 77.869378330999993, 2024-03-27T21:40:58.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078150589999999992, 2024-03-27T21:40:58.459 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:58.459 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.459 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:58.459 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:58.460 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:58.460 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-03-27T21:40:58.460 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.460 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:58.460 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.460 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:58.460 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:58.460 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.460 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.460 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.460 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:58.460 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:40:58.460 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.460 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.460 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.461 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:58.461 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497155+0000", 2024-03-27T21:40:58.461 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.461 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.461 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.461 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:58.461 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497230+0000", 2024-03-27T21:40:58.461 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4789000000000005e-05 2024-03-27T21:40:58.461 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.461 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.461 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:58.461 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497241+0000", 2024-03-27T21:40:58.461 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1442e-05 2024-03-27T21:40:58.461 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.461 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.462 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.462 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497255+0000", 2024-03-27T21:40:58.462 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3087e-05 2024-03-27T21:40:58.462 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.462 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.462 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:58.462 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503630+0000", 2024-03-27T21:40:58.462 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063755269999999998 2024-03-27T21:40:58.462 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.462 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.462 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:58.462 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503673+0000", 2024-03-27T21:40:58.462 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2616e-05 2024-03-27T21:40:58.463 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.463 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.463 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:58.463 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503738+0000", 2024-03-27T21:40:58.463 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5114e-05 2024-03-27T21:40:58.463 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.463 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.463 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:58.463 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504907+0000", 2024-03-27T21:40:58.463 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169432 2024-03-27T21:40:58.463 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.463 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.463 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:58.463 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504926+0000", 2024-03-27T21:40:58.463 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9086999999999999e-05 2024-03-27T21:40:58.464 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.464 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.464 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:58.464 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504950+0000", 2024-03-27T21:40:58.464 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3598999999999999e-05 2024-03-27T21:40:58.464 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.464 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.464 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:58.464 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504971+0000", 2024-03-27T21:40:58.464 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0772999999999999e-05 2024-03-27T21:40:58.464 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.464 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:58.464 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.464 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.465 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.465 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:58.465 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:58.465 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 77.869271291000004, 2024-03-27T21:40:58.465 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082962590000000003, 2024-03-27T21:40:58.465 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:58.465 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.465 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:58.465 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:58.465 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:58.465 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-03-27T21:40:58.465 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.465 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:58.465 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.466 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:58.466 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:58.466 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.466 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.466 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.466 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:58.466 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:40:58.466 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.466 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.466 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.466 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:58.466 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497262+0000", 2024-03-27T21:40:58.466 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.466 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.467 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:58.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497267+0000", 2024-03-27T21:40:58.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.955e-06 2024-03-27T21:40:58.467 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.467 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:58.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497275+0000", 2024-03-27T21:40:58.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.824e-06 2024-03-27T21:40:58.467 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.467 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497280+0000", 2024-03-27T21:40:58.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8230000000000002e-06 2024-03-27T21:40:58.467 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.468 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.468 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:58.468 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504628+0000", 2024-03-27T21:40:58.468 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007348561 2024-03-27T21:40:58.468 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.468 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.468 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:58.468 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504673+0000", 2024-03-27T21:40:58.468 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4313000000000001e-05 2024-03-27T21:40:58.468 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.468 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.468 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:58.469 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504745+0000", 2024-03-27T21:40:58.469 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2303000000000002e-05 2024-03-27T21:40:58.469 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.469 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.469 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:58.469 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505358+0000", 2024-03-27T21:40:58.469 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061281300000000003 2024-03-27T21:40:58.469 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.469 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.469 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:58.469 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505503+0000", 2024-03-27T21:40:58.469 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000145134 2024-03-27T21:40:58.470 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.470 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.470 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:58.470 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505526+0000", 2024-03-27T21:40:58.470 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3312e-05 2024-03-27T21:40:58.470 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.470 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.470 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:58.470 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505559+0000", 2024-03-27T21:40:58.470 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2821999999999998e-05 2024-03-27T21:40:58.470 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.470 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:58.470 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.471 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.471 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.471 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:40:58.471 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:58.471 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 77.260049051999999, 2024-03-27T21:40:58.471 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053753339999999998, 2024-03-27T21:40:58.471 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:40:58.471 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.471 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:40:58.471 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:40:58.471 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:40:58.471 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159 2024-03-27T21:40:58.471 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.472 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:40:58.472 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.472 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:40:58.472 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:58.472 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.472 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.472 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.472 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:40:58.472 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:58.472 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:40:58.472 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.472 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.472 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:40:58.472 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:40:58.472 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.473 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.473 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.473 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:40:58.473 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106488+0000", 2024-03-27T21:40:58.473 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9830000000000001e-06 2024-03-27T21:40:58.473 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.473 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.473 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:40:58.473 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106491+0000", 2024-03-27T21:40:58.473 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3459999999999998e-06 2024-03-27T21:40:58.473 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.473 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.473 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.473 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106493+0000", 2024-03-27T21:40:58.474 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.509e-06 2024-03-27T21:40:58.474 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.474 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.474 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:40:58.474 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110636+0000", 2024-03-27T21:40:58.474 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041427260000000002 2024-03-27T21:40:58.474 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.474 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.474 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:40:58.474 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110683+0000", 2024-03-27T21:40:58.474 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6557000000000002e-05 2024-03-27T21:40:58.474 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.474 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.474 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:40:58.475 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110755+0000", 2024-03-27T21:40:58.475 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2119000000000005e-05 2024-03-27T21:40:58.475 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.475 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:40:58.475 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111415+0000", 2024-03-27T21:40:58.475 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066065199999999996 2024-03-27T21:40:58.475 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.475 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:40:58.475 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111793+0000", 2024-03-27T21:40:58.475 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037732100000000001 2024-03-27T21:40:58.475 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.475 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:40:58.476 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111829+0000", 2024-03-27T21:40:58.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418999999999998e-05 2024-03-27T21:40:58.476 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:40:58.476 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:40:58.476 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:40:58.476 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111860+0000", 2024-03-27T21:40:58.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1003000000000001e-05 2024-03-27T21:40:58.476 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.476 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:58.476 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.476 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:40:58.476 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:40:58.476 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:40:58.478 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:40:58.479 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:58.491+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:58.579 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:58.593+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:58.588 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:58.589 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:58.589 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:40:58.589 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:58.598 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:40:58.680 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:58.694+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:58.707 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:58.708 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:40:58.708 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:40:58.708 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:40:58.708 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:58.717 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:40:58.780 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:58.795+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:58.820 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:40:58.882 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:58.896+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:58.884 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:40:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:40:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:40:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-03-27T21:40:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:20 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2024-03-27T21:40:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 79.008866772999994, 2024-03-27T21:40:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015345630000000001, 2024-03-27T21:40:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:40:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888562+0000", 2024-03-27T21:40:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888735+0000", 2024-03-27T21:40:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017254799999999999 2024-03-27T21:40:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888747+0000", 2024-03-27T21:40:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2469999999999999e-05 2024-03-27T21:40:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888761+0000", 2024-03-27T21:40:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3254e-05 2024-03-27T21:40:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888804+0000", 2024-03-27T21:40:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3129e-05 2024-03-27T21:40:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888820+0000", 2024-03-27T21:40:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6426000000000002e-05 2024-03-27T21:40:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890060+0000", 2024-03-27T21:40:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012398050000000001 2024-03-27T21:40:58.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.889 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.889 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:58.889 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890098+0000", 2024-03-27T21:40:58.889 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7795e-05 2024-03-27T21:40:58.889 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.889 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:58.889 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.889 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.890 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2024-03-27T21:40:58.890 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:58.890 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 79.008042372999995, 2024-03-27T21:40:58.890 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001069555, 2024-03-27T21:40:58.890 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:58.890 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.890 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:58.891 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.892 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:58.892 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:58.892 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.892 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.892 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.892 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:40:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889387+0000", 2024-03-27T21:40:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889455+0000", 2024-03-27T21:40:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7930999999999997e-05 2024-03-27T21:40:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889470+0000", 2024-03-27T21:40:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5554e-05 2024-03-27T21:40:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889476+0000", 2024-03-27T21:40:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0619999999999997e-06 2024-03-27T21:40:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889513+0000", 2024-03-27T21:40:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6606999999999997e-05 2024-03-27T21:40:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889528+0000", 2024-03-27T21:40:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4982e-05 2024-03-27T21:40:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890421+0000", 2024-03-27T21:40:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00089293700000000001 2024-03-27T21:40:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890457+0000", 2024-03-27T21:40:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6260000000000002e-05 2024-03-27T21:40:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:38 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'24, mlcod=17'22)", 2024-03-27T21:40:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 78.990806406999994, 2024-03-27T21:40:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016182080000000001, 2024-03-27T21:40:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:40:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906623+0000", 2024-03-27T21:40:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906687+0000", 2024-03-27T21:40:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3706000000000003e-05 2024-03-27T21:40:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906698+0000", 2024-03-27T21:40:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1752e-05 2024-03-27T21:40:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906704+0000", 2024-03-27T21:40:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3940000000000004e-06 2024-03-27T21:40:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906741+0000", 2024-03-27T21:40:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6944999999999999e-05 2024-03-27T21:40:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906760+0000", 2024-03-27T21:40:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9293e-05 2024-03-27T21:40:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908211+0000", 2024-03-27T21:40:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014511350000000001 2024-03-27T21:40:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908242+0000", 2024-03-27T21:40:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0712e-05 2024-03-27T21:40:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:39 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'25, mlcod=17'22)", 2024-03-27T21:40:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 78.989961492000006, 2024-03-27T21:40:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012060199999999999, 2024-03-27T21:40:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:40:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907533+0000", 2024-03-27T21:40:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5675999999999999e-05 2024-03-27T21:40:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907545+0000", 2024-03-27T21:40:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.115e-05 2024-03-27T21:40:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907550+0000", 2024-03-27T21:40:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8320000000000002e-06 2024-03-27T21:40:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907592+0000", 2024-03-27T21:40:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1309e-05 2024-03-27T21:40:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907611+0000", 2024-03-27T21:40:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9009e-05 2024-03-27T21:40:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908638+0000", 2024-03-27T21:40:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001027533 2024-03-27T21:40:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908675+0000", 2024-03-27T21:40:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6329999999999999e-05 2024-03-27T21:40:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:57 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'35, mlcod=17'33)", 2024-03-27T21:40:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 78.975877060000002, 2024-03-27T21:40:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001346578, 2024-03-27T21:40:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:40:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921552+0000", 2024-03-27T21:40:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921615+0000", 2024-03-27T21:40:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2787999999999996e-05 2024-03-27T21:40:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921626+0000", 2024-03-27T21:40:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0888e-05 2024-03-27T21:40:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921635+0000", 2024-03-27T21:40:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4639999999999995e-06 2024-03-27T21:40:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921673+0000", 2024-03-27T21:40:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7891e-05 2024-03-27T21:40:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921694+0000", 2024-03-27T21:40:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0426000000000001e-05 2024-03-27T21:40:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922865+0000", 2024-03-27T21:40:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011709509999999999 2024-03-27T21:40:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922900+0000", 2024-03-27T21:40:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4819e-05 2024-03-27T21:40:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:71 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'45, mlcod=17'43)", 2024-03-27T21:40:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 78.960578347999999, 2024-03-27T21:40:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001460907, 2024-03-27T21:40:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:40:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936851+0000", 2024-03-27T21:40:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936903+0000", 2024-03-27T21:40:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2302999999999997e-05 2024-03-27T21:40:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936915+0000", 2024-03-27T21:40:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1967000000000001e-05 2024-03-27T21:40:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936923+0000", 2024-03-27T21:40:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9370000000000004e-06 2024-03-27T21:40:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936970+0000", 2024-03-27T21:40:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6987000000000003e-05 2024-03-27T21:40:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937002+0000", 2024-03-27T21:40:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2410999999999999e-05 2024-03-27T21:40:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938279+0000", 2024-03-27T21:40:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001276552 2024-03-27T21:40:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938313+0000", 2024-03-27T21:40:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3526999999999999e-05 2024-03-27T21:40:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:72 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'46, mlcod=17'43)", 2024-03-27T21:40:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 78.959940786000004, 2024-03-27T21:40:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014239490000000001, 2024-03-27T21:40:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:40:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937488+0000", 2024-03-27T21:40:58.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:58.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:58.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937547+0000", 2024-03-27T21:40:58.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8196999999999999e-05 2024-03-27T21:40:58.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:58.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937559+0000", 2024-03-27T21:40:58.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1928e-05 2024-03-27T21:40:58.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937564+0000", 2024-03-27T21:40:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4759999999999997e-06 2024-03-27T21:40:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937621+0000", 2024-03-27T21:40:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6712000000000002e-05 2024-03-27T21:40:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937640+0000", 2024-03-27T21:40:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9517e-05 2024-03-27T21:40:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938883+0000", 2024-03-27T21:40:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001242567 2024-03-27T21:40:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938913+0000", 2024-03-27T21:40:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0287e-05 2024-03-27T21:40:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:73 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'47, mlcod=17'43)", 2024-03-27T21:40:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 78.959264692000005, 2024-03-27T21:40:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010746239999999999, 2024-03-27T21:40:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:40:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:40:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938217+0000", 2024-03-27T21:40:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2743e-05 2024-03-27T21:40:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938229+0000", 2024-03-27T21:40:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1956999999999999e-05 2024-03-27T21:40:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938235+0000", 2024-03-27T21:40:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4589999999999997e-06 2024-03-27T21:40:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938341+0000", 2024-03-27T21:40:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000106054 2024-03-27T21:40:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938369+0000", 2024-03-27T21:40:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8291000000000001e-05 2024-03-27T21:40:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939209+0000", 2024-03-27T21:40:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084045400000000003 2024-03-27T21:40:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939240+0000", 2024-03-27T21:40:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0429999999999998e-05 2024-03-27T21:40:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:75 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'49, mlcod=17'47)", 2024-03-27T21:40:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 78.952827045000006, 2024-03-27T21:40:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001017001, 2024-03-27T21:40:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:40:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944601+0000", 2024-03-27T21:40:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999986 2024-03-27T21:40:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944614+0000", 2024-03-27T21:40:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2496e-05 2024-03-27T21:40:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944640+0000", 2024-03-27T21:40:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5865999999999999e-05 2024-03-27T21:40:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944651+0000", 2024-03-27T21:40:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1797000000000001e-05 2024-03-27T21:40:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944762+0000", 2024-03-27T21:40:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110412 2024-03-27T21:40:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944820+0000", 2024-03-27T21:40:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7649999999999999e-05 2024-03-27T21:40:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945585+0000", 2024-03-27T21:40:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076562600000000005 2024-03-27T21:40:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945620+0000", 2024-03-27T21:40:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4764000000000001e-05 2024-03-27T21:40:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:86 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'50)", 2024-03-27T21:40:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 78.945065557999996, 2024-03-27T21:40:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011392380000000001, 2024-03-27T21:40:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:40:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952373+0000", 2024-03-27T21:40:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9220000000000003e-06 2024-03-27T21:40:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952381+0000", 2024-03-27T21:40:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3070000000000003e-06 2024-03-27T21:40:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952385+0000", 2024-03-27T21:40:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.121e-06 2024-03-27T21:40:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952419+0000", 2024-03-27T21:40:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4082999999999999e-05 2024-03-27T21:40:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952440+0000", 2024-03-27T21:40:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0837999999999999e-05 2024-03-27T21:40:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953472+0000", 2024-03-27T21:40:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001032119 2024-03-27T21:40:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953504+0000", 2024-03-27T21:40:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1250000000000001e-05 2024-03-27T21:40:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:87 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-27T21:40:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 78.944698301000003, 2024-03-27T21:40:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00135519, 2024-03-27T21:40:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:40:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952731+0000", 2024-03-27T21:40:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952781+0000", 2024-03-27T21:40:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0161000000000003e-05 2024-03-27T21:40:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952794+0000", 2024-03-27T21:40:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2853e-05 2024-03-27T21:40:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952799+0000", 2024-03-27T21:40:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5560000000000001e-06 2024-03-27T21:40:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952832+0000", 2024-03-27T21:40:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3834999999999997e-05 2024-03-27T21:40:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952851+0000", 2024-03-27T21:40:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8289999999999999e-05 2024-03-27T21:40:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954055+0000", 2024-03-27T21:40:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012037949999999999 2024-03-27T21:40:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954087+0000", 2024-03-27T21:40:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2292999999999997e-05 2024-03-27T21:40:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:89 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'59, mlcod=17'57)", 2024-03-27T21:40:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 78.942424079999995, 2024-03-27T21:40:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00138893, 2024-03-27T21:40:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:40:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955005+0000", 2024-03-27T21:40:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955071+0000", 2024-03-27T21:40:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5290999999999995e-05 2024-03-27T21:40:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955081+0000", 2024-03-27T21:40:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0572e-05 2024-03-27T21:40:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955086+0000", 2024-03-27T21:40:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8380000000000001e-06 2024-03-27T21:40:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955122+0000", 2024-03-27T21:40:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5698000000000003e-05 2024-03-27T21:40:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955140+0000", 2024-03-27T21:40:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7900000000000001e-05 2024-03-27T21:40:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956366+0000", 2024-03-27T21:40:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001226471 2024-03-27T21:40:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956395+0000", 2024-03-27T21:40:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8782e-05 2024-03-27T21:40:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:90 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'60, mlcod=17'57)", 2024-03-27T21:40:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 78.941739502999994, 2024-03-27T21:40:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010778859999999999, 2024-03-27T21:40:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:40:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955747+0000", 2024-03-27T21:40:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7130999999999999e-05 2024-03-27T21:40:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955756+0000", 2024-03-27T21:40:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2750000000000001e-06 2024-03-27T21:40:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955761+0000", 2024-03-27T21:40:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3900000000000003e-06 2024-03-27T21:40:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.940 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:58.940 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955797+0000", 2024-03-27T21:40:58.940 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6650000000000003e-05 2024-03-27T21:40:58.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.940 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:58.940 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955816+0000", 2024-03-27T21:40:58.940 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8195000000000001e-05 2024-03-27T21:40:58.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.940 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:58.940 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956739+0000", 2024-03-27T21:40:58.940 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00092369700000000002 2024-03-27T21:40:58.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956768+0000", 2024-03-27T21:40:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9048e-05 2024-03-27T21:40:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:118 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'80, mlcod=17'78)", 2024-03-27T21:40:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 78.808748679999994, 2024-03-27T21:40:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014200669999999999, 2024-03-27T21:40:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:40:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088756+0000", 2024-03-27T21:40:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4902999999999998e-05 2024-03-27T21:40:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088764+0000", 2024-03-27T21:40:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8169999999999999e-06 2024-03-27T21:40:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088767+0000", 2024-03-27T21:40:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8310000000000003e-06 2024-03-27T21:40:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088798+0000", 2024-03-27T21:40:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0800999999999999e-05 2024-03-27T21:40:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088817+0000", 2024-03-27T21:40:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8467000000000001e-05 2024-03-27T21:40:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090074+0000", 2024-03-27T21:40:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012574089999999999 2024-03-27T21:40:58.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.945 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.945 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:58.945 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090101+0000", 2024-03-27T21:40:58.945 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7223e-05 2024-03-27T21:40:58.945 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.945 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:58.945 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.945 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.945 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:119 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'81, mlcod=17'78)", 2024-03-27T21:40:58.945 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:58.945 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 78.808013884999994, 2024-03-27T21:40:58.945 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014990870000000001, 2024-03-27T21:40:58.945 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:58.946 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.946 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:58.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.946 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:58.946 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:58.946 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.946 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:58.946 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:58.947 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:58.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:40:58.947 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:58.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089491+0000", 2024-03-27T21:40:58.947 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4805e-05 2024-03-27T21:40:58.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:58.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089498+0000", 2024-03-27T21:40:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0650000000000001e-06 2024-03-27T21:40:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089501+0000", 2024-03-27T21:40:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.54e-06 2024-03-27T21:40:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089533+0000", 2024-03-27T21:40:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1838999999999998e-05 2024-03-27T21:40:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089550+0000", 2024-03-27T21:40:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7465999999999998e-05 2024-03-27T21:40:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090890+0000", 2024-03-27T21:40:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013397979999999999 2024-03-27T21:40:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090915+0000", 2024-03-27T21:40:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4905e-05 2024-03-27T21:40:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:120 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'82, mlcod=17'78)", 2024-03-27T21:40:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 78.807214005000006, 2024-03-27T21:40:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001527093, 2024-03-27T21:40:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:40:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090267+0000", 2024-03-27T21:40:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1477000000000002e-05 2024-03-27T21:40:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090273+0000", 2024-03-27T21:40:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9519999999999999e-06 2024-03-27T21:40:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090276+0000", 2024-03-27T21:40:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.219e-06 2024-03-27T21:40:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090304+0000", 2024-03-27T21:40:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8073999999999999e-05 2024-03-27T21:40:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090316+0000", 2024-03-27T21:40:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2088e-05 2024-03-27T21:40:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091717+0000", 2024-03-27T21:40:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014003889999999999 2024-03-27T21:40:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091743+0000", 2024-03-27T21:40:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6234e-05 2024-03-27T21:40:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:121 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'83, mlcod=17'78)", 2024-03-27T21:40:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 78.806403639999999, 2024-03-27T21:40:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010226899999999999, 2024-03-27T21:40:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:40:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091142+0000", 2024-03-27T21:40:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011608300000000001 2024-03-27T21:40:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091151+0000", 2024-03-27T21:40:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2350000000000005e-06 2024-03-27T21:40:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091156+0000", 2024-03-27T21:40:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5419999999999999e-06 2024-03-27T21:40:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091191+0000", 2024-03-27T21:40:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5453000000000001e-05 2024-03-27T21:40:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091208+0000", 2024-03-27T21:40:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7119999999999999e-05 2024-03-27T21:40:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092024+0000", 2024-03-27T21:40:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000815539 2024-03-27T21:40:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092049+0000", 2024-03-27T21:40:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5171e-05 2024-03-27T21:40:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:142 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'100, mlcod=18'98)", 2024-03-27T21:40:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 78.391396685999993, 2024-03-27T21:40:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001198778, 2024-03-27T21:40:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:40:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506101+0000", 2024-03-27T21:40:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8202999999999997e-05 2024-03-27T21:40:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506109+0000", 2024-03-27T21:40:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3059999999999998e-06 2024-03-27T21:40:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506113+0000", 2024-03-27T21:40:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0119999999999999e-06 2024-03-27T21:40:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506137+0000", 2024-03-27T21:40:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4043999999999999e-05 2024-03-27T21:40:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506148+0000", 2024-03-27T21:40:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0446e-05 2024-03-27T21:40:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507213+0000", 2024-03-27T21:40:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001065215 2024-03-27T21:40:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:58.962 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507232+0000", 2024-03-27T21:40:58.962 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9038e-05 2024-03-27T21:40:58.962 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.962 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:58.962 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.962 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:143 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'101, mlcod=18'98)", 2024-03-27T21:40:58.962 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:58.962 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 78.390784066999998, 2024-03-27T21:40:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001357488, 2024-03-27T21:40:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:40:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506723+0000", 2024-03-27T21:40:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7404000000000006e-05 2024-03-27T21:40:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506730+0000", 2024-03-27T21:40:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5749999999999997e-06 2024-03-27T21:40:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506733+0000", 2024-03-27T21:40:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7979999999999999e-06 2024-03-27T21:40:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506758+0000", 2024-03-27T21:40:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4179e-05 2024-03-27T21:40:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506768+0000", 2024-03-27T21:40:58.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0740000000000001e-05 2024-03-27T21:40:58.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:58.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507978+0000", 2024-03-27T21:40:58.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012093900000000001 2024-03-27T21:40:58.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:58.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508003+0000", 2024-03-27T21:40:58.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5667e-05 2024-03-27T21:40:58.966 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.966 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:58.967 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.967 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:144 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'102, mlcod=18'98)", 2024-03-27T21:40:58.967 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:58.967 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 78.390125170000005, 2024-03-27T21:40:58.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013453390000000001, 2024-03-27T21:40:58.967 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:40:58.967 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:40:58.967 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:40:58.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:40:58.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:40:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:40:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:40:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:40:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507304+0000", 2024-03-27T21:40:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:40:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:40:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507397+0000", 2024-03-27T21:40:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2995e-05 2024-03-27T21:40:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:40:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507403+0000", 2024-03-27T21:40:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8640000000000001e-06 2024-03-27T21:40:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:40:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507407+0000", 2024-03-27T21:40:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5710000000000002e-06 2024-03-27T21:40:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:40:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507428+0000", 2024-03-27T21:40:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1192e-05 2024-03-27T21:40:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:40:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507438+0000", 2024-03-27T21:40:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8309999999999996e-06 2024-03-27T21:40:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:40:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508624+0000", 2024-03-27T21:40:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011861210000000001 2024-03-27T21:40:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:40:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:40:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:40:58.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508650+0000", 2024-03-27T21:40:58.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6118e-05 2024-03-27T21:40:58.971 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.971 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:58.971 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.971 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:40:58.971 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:40:58.971 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:40:58.972 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:40:58.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.937+0000 7f5cf14db640 10 monclient: get_monmap_and_config 2024-03-27T21:40:58.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.937+0000 7f5cf14db640 10 monclient: build_initial_monmap 2024-03-27T21:40:58.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.937+0000 7f5cf14db640 10 monclient: monmap: 2024-03-27T21:40:58.980 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:58.980 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:58.980 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:40:58.938038+0000 2024-03-27T21:40:58.980 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:40:58.938038+0000 2024-03-27T21:40:58.980 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:58.980 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:58.980 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:40:58.980 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:40:58.980 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:40:58.980 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:40:58.980 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:40:58.981 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:58.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.937+0000 7f5cf14db640 1 Processor -- start 2024-03-27T21:40:58.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.937+0000 7f5cf14db640 1 -- start start 2024-03-27T21:40:58.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.937+0000 7f5cf14db640 10 monclient: init 2024-03-27T21:40:58.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5cf14db640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:58.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5cf14db640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-03-27T21:40:58.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5cf14db640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5cec0b40b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:40:58.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5cf14db640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5cec13b020 0x7f5cec13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:58.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5cf14db640 10 monclient(hunting): picked mon.noname-c con 0x7f5cec13b020 addr v2:172.21.15.191:3300/0 2024-03-27T21:40:58.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5cf14db640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5cec1416f0 addr v1:172.21.15.191:6789/0 2024-03-27T21:40:58.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5cf14db640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cec0b3830 con 0x7f5cec0b40b0 2024-03-27T21:40:58.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5cf14db640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cec0aa520 con 0x7f5cec1416f0 2024-03-27T21:40:58.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5cf14db640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:58.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5cf14db640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cec0b4490 con 0x7f5cec13b020 2024-03-27T21:40:58.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5cf14db640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:58.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5cf14db640 10 monclient(hunting): authenticate will time out at 900.159444s 2024-03-27T21:40:58.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5ceaffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5cec0b40b0 0x7f5cec0a8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37064/0 (socket says 172.21.15.60:37064) 2024-03-27T21:40:58.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5ceaffd640 1 -- 172.21.15.60:0/3836470810 learned_addr learned my addr 172.21.15.60:0/3836470810 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:40:58.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5ce3fff640 1 --2- 172.21.15.60:0/3836470810 >> v2:172.21.15.191:3300/0 conn(0x7f5cec13b020 0x7f5cec13b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:58.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5ce3fff640 10 monclient(hunting): get_auth_request con 0x7f5cec13b020 auth_method 0 2024-03-27T21:40:58.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5ce3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:58.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5ce3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:58.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5ce3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:58.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5ce3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:58.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5ce3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:58.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5ce3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:58.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5cea7fc640 1 -- 172.21.15.60:0/3836470810 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426474983 0 0) 0x7f5cec0b3830 con 0x7f5cec0b40b0 2024-03-27T21:40:58.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5cea7fc640 10 monclient(hunting): _negotiate 2024-03-27T21:40:58.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5cea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:58.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5cea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:58.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5cea7fc640 10 monclient(hunting): my global_id is 4812 2024-03-27T21:40:58.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5cea7fc640 1 -- 172.21.15.60:0/3836470810 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cd8003520 con 0x7f5cec0b40b0 2024-03-27T21:40:58.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5ce3fff640 10 monclient(hunting): handle_auth_done global_id 4750 payload 274 2024-03-27T21:40:58.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.938+0000 7f5ce3fff640 1 -- 172.21.15.60:0/3836470810 >> v1:172.21.15.191:6789/0 conn(0x7f5cec1416f0 legacy=0x7f5cec141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:58.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.939+0000 7f5cea7fc640 1 -- 172.21.15.60:0/3836470810 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328421073 0 0) 0x7f5cec0aa520 con 0x7f5cec1416f0 2024-03-27T21:40:58.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.939+0000 7f5ce3fff640 1 -- 172.21.15.60:0/3836470810 >> v1:172.21.15.60:6789/0 conn(0x7f5cec0b40b0 legacy=0x7f5cec0a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:58.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.939+0000 7f5ce3fff640 10 monclient: _finish_hunting 0 2024-03-27T21:40:58.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.939+0000 7f5ce3fff640 1 monclient: found mon.noname-c 2024-03-27T21:40:58.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.939+0000 7f5ce3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:58.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.939+0000 7f5ce3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:40:58.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.939+0000 7f5ce3fff640 1 -- 172.21.15.60:0/3836470810 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cec0a9470 con 0x7f5cec13b020 2024-03-27T21:40:58.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.939+0000 7f5cea7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:40:58.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.939+0000 7f5ce3fff640 1 --2- 172.21.15.60:0/3836470810 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cec13b020 0x7f5cec13b480 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5cd40097a0 tx=0x7f5cd402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce73a25859839b57 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:58.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cea7fc640 1 -- 172.21.15.60:0/3836470810 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4030610 con 0x7f5cec13b020 2024-03-27T21:40:58.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:58.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:40:58.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cea7fc640 10 monclient: dump: 2024-03-27T21:40:58.984 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:58.984 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:58.984 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:58.984 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:58.984 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:58.984 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:58.984 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:58.984 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:58.984 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:58.984 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:58.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cea7fc640 10 monclient: _finish_auth 0 2024-03-27T21:40:58.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cea7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:40:58.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:58.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cea7fc640 1 -- 172.21.15.60:0/3836470810 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd4030770 con 0x7f5cec13b020 2024-03-27T21:40:58.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cf14db640 5 monclient: authenticate success, global_id 4750 2024-03-27T21:40:58.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cf14db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:40:58.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:58.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cea7fc640 1 -- 172.21.15.60:0/3836470810 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4030b00 con 0x7f5cec13b020 2024-03-27T21:40:58.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cf14db640 10 monclient: get_monmap_and_config success 2024-03-27T21:40:58.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:58.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cf14db640 4 set_mon_vals no callback set 2024-03-27T21:40:58.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:40:58.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cea7fc640 10 monclient: dump: 2024-03-27T21:40:58.985 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:58.985 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:58.985 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:58.986 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:58.986 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:58.986 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:58.986 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:58.986 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:58.986 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:58.986 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:58.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cf14db640 10 monclient: shutdown 2024-03-27T21:40:58.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cf14db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:58.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cf14db640 1 -- 172.21.15.60:0/3836470810 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cec13b020 msgr2=0x7f5cec13b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:58.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cf14db640 1 --2- 172.21.15.60:0/3836470810 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cec13b020 0x7f5cec13b480 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5cd40097a0 tx=0x7f5cd402b960 comp rx=0 tx=0).stop 2024-03-27T21:40:58.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cf14db640 1 -- 172.21.15.60:0/3836470810 shutdown_connections 2024-03-27T21:40:58.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cf14db640 1 --2- 172.21.15.60:0/3836470810 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cec13b020 0x7f5cec13b480 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:58.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.940+0000 7f5cf14db640 1 -- 172.21.15.60:0/3836470810 >> 172.21.15.60:0/3836470810 conn(0x7f5cec0608e0 msgr2=0x7f5cec060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:58.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.941+0000 7f5cf14db640 1 -- 172.21.15.60:0/3836470810 shutdown_connections 2024-03-27T21:40:58.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.941+0000 7f5cf14db640 1 -- 172.21.15.60:0/3836470810 wait complete. 2024-03-27T21:40:58.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.941+0000 7f5cf14db640 10 monclient: build_initial_monmap 2024-03-27T21:40:58.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.941+0000 7f5cf14db640 10 monclient: monmap: 2024-03-27T21:40:58.990 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:40:58.990 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:40:58.990 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:40:58.990 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:40:58.990 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:40:58.990 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:40:58.990 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:40:58.990 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:40:58.990 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:40:58.990 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:58.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.942+0000 7f5cf14db640 1 librados: starting msgr at 2024-03-27T21:40:58.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.942+0000 7f5cf14db640 1 librados: starting objecter 2024-03-27T21:40:58.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.942+0000 7f5cf14db640 1 Processor -- start 2024-03-27T21:40:58.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.942+0000 7f5cf14db640 1 -- start start 2024-03-27T21:40:58.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.942+0000 7f5cf14db640 1 librados: setting wanted keys 2024-03-27T21:40:58.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.942+0000 7f5cf14db640 1 librados: calling monclient init 2024-03-27T21:40:58.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.942+0000 7f5cf14db640 10 monclient: init 2024-03-27T21:40:58.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5cf14db640 10 monclient: _reopen_session rank -1 2024-03-27T21:40:58.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5cf14db640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:40:58.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5cf14db640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5cec0b40b0 0x7f5cec0aac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:58.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5cf14db640 10 monclient(hunting): picked mon.noname-c con 0x7f5cec0b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:40:58.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5cf14db640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5cec13b020 0x7f5cec1398f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:58.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5cf14db640 10 monclient(hunting): picked mon.noname-a con 0x7f5cec13b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:40:58.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5cf14db640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cec1416f0 0x7f5cec1d6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:58.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5cf14db640 10 monclient(hunting): picked mon.noname-b con 0x7f5cec1416f0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:40:58.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5cf14db640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:58.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5cf14db640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cec1d6210 con 0x7f5cec13b020 2024-03-27T21:40:58.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5cf14db640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:58.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5cf14db640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cec0ac7b0 con 0x7f5cec1416f0 2024-03-27T21:40:58.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5cf14db640 10 monclient(hunting): start opening mon connection 2024-03-27T21:40:58.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5cf14db640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cec139e30 con 0x7f5cec0b40b0 2024-03-27T21:40:58.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5cf14db640 10 monclient(hunting): _renew_subs 2024-03-27T21:40:58.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5ce3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5cec13b020 0x7f5cec1398f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:58.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5ceaffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5cec0b40b0 0x7f5cec0aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:58.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5cf14db640 10 monclient(hunting): authenticate will time out at 900.164634s 2024-03-27T21:40:58.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5ceaffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5cec0b40b0 0x7f5cec0aac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:57868/0 (socket says 172.21.15.60:57868) 2024-03-27T21:40:58.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5ce3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5cec13b020 0x7f5cec1398f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47226/0 (socket says 172.21.15.60:47226) 2024-03-27T21:40:58.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5ceaffd640 1 -- 172.21.15.60:0/461868192 learned_addr learned my addr 172.21.15.60:0/461868192 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:40:58.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5ceaffd640 10 monclient(hunting): get_auth_request con 0x7f5cec0b40b0 auth_method 0 2024-03-27T21:40:58.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5ceaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:58.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5ceaffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:58.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5ceaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:58.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5ce3fff640 10 monclient(hunting): get_auth_request con 0x7f5cec13b020 auth_method 0 2024-03-27T21:40:58.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5ce3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:58.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5ce3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:58.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5ce3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:58.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5ce3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:58.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5ce3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:58.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5ce3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:58.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5ceaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:40:58.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5ceaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:40:58.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.943+0000 7f5ceaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:40:58.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.944+0000 7f5ceb7fe640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cec1416f0 0x7f5cec1d6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:58.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.944+0000 7f5ceb7fe640 10 monclient(hunting): get_auth_request con 0x7f5cec1416f0 auth_method 0 2024-03-27T21:40:58.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.944+0000 7f5ceb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:40:58.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.944+0000 7f5ceb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:40:58.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.944+0000 7f5ceb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:40:58.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.944+0000 7f5ce3fff640 10 monclient(hunting): handle_auth_done global_id 4815 payload 824 2024-03-27T21:40:58.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.944+0000 7f5ce3fff640 1 -- 172.21.15.60:0/461868192 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5cec0b40b0 msgr2=0x7f5cec0aac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:58.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.944+0000 7f5ce3fff640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5cec0b40b0 0x7f5cec0aac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:58.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.944+0000 7f5ce3fff640 1 -- 172.21.15.60:0/461868192 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cec1416f0 msgr2=0x7f5cec1d6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:58.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.944+0000 7f5ce3fff640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cec1416f0 0x7f5cec1d6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:58.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.944+0000 7f5ce3fff640 10 monclient: _finish_hunting 0 2024-03-27T21:40:58.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.944+0000 7f5ce3fff640 1 monclient: found mon.noname-a 2024-03-27T21:40:58.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.944+0000 7f5ce3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:40:58.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.944+0000 7f5ce3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:40:58.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.944+0000 7f5ce3fff640 1 -- 172.21.15.60:0/461868192 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd4009400 con 0x7f5cec13b020 2024-03-27T21:40:58.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.944+0000 7f5ce3fff640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5cec13b020 0x7f5cec1398f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5cec0a9e10 tx=0x7f5cd4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:58.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.944+0000 7f5ce37fe640 10 client.?.objecter ms_handle_connect 0x7f5cec13b020 2024-03-27T21:40:58.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.944+0000 7f5ce37fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:40:58.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.946+0000 7f5ce37fe640 1 -- 172.21.15.60:0/461868192 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd403e040 con 0x7f5cec13b020 2024-03-27T21:40:58.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.946+0000 7f5ce37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:58.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.946+0000 7f5ce37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:40:58.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.946+0000 7f5ce37fe640 10 monclient: dump: 2024-03-27T21:40:58.995 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:58.995 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:58.995 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:58.995 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:58.995 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:58.995 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:58.995 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:58.995 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:58.995 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:58.995 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:58.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.946+0000 7f5ce37fe640 10 monclient: _finish_auth 0 2024-03-27T21:40:58.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.946+0000 7f5ce37fe640 10 monclient: _check_auth_tickets 2024-03-27T21:40:58.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.946+0000 7f5ce37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:40:58.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.946+0000 7f5cf14db640 5 monclient: authenticate success, global_id 4815 2024-03-27T21:40:58.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.946+0000 7f5cf14db640 10 monclient: _renew_subs 2024-03-27T21:40:58.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.946+0000 7f5cf14db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:58.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.946+0000 7f5cf14db640 1 -- 172.21.15.60:0/461868192 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cec133e10 con 0x7f5cec13b020 2024-03-27T21:40:58.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.947+0000 7f5ce37fe640 1 -- 172.21.15.60:0/461868192 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd4039c10 con 0x7f5cec13b020 2024-03-27T21:40:58.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.947+0000 7f5ce37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:40:58.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.947+0000 7f5ce37fe640 1 -- 172.21.15.60:0/461868192 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4030e30 con 0x7f5cec13b020 2024-03-27T21:40:58.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.947+0000 7f5ce37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:40:58.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.947+0000 7f5ce37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:40:58.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.947+0000 7f5ce37fe640 10 monclient: dump: 2024-03-27T21:40:58.996 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:40:58.996 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:40:58.996 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:58.996 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:40:58.997 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:40:58.997 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:40:58.997 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:40:58.997 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:40:58.997 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:40:58.997 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:40:58.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.947+0000 7f5cf14db640 10 client.4815.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:58.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.947+0000 7f5cf14db640 10 monclient: _renew_subs 2024-03-27T21:40:58.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.947+0000 7f5cf14db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:58.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.947+0000 7f5cf14db640 1 -- 172.21.15.60:0/461868192 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cec1d6b70 con 0x7f5cec13b020 2024-03-27T21:40:58.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.947+0000 7f5cf14db640 1 librados: init done 2024-03-27T21:40:58.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.947+0000 7f5ce9ffb640 4 set_mon_vals no callback set 2024-03-27T21:40:58.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.947+0000 7f5cf14db640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:40:58.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.947+0000 7f5cf14db640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:58.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.947+0000 7f5cf14db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:58.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.947+0000 7f5cf14db640 1 -- 172.21.15.60:0/461868192 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f5cec1d6350 con 0x7f5cec13b020 2024-03-27T21:40:58.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.949+0000 7f5ce37fe640 1 -- 172.21.15.60:0/461868192 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5cd4043390 con 0x7f5cec13b020 2024-03-27T21:40:58.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.949+0000 7f5ce37fe640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5ccc06c590 0x7f5ccc06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:58.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.949+0000 7f5ceaffd640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5ccc06c590 0x7f5ccc06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:58.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.949+0000 7f5ceaffd640 10 monclient: get_auth_request con 0x7f5ccc06c590 auth_method 0 2024-03-27T21:40:58.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f85a6ffd640 1 -- 172.21.15.60:0/778281124 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v49) v1 ==== 91+0+0 (secure 0 0 0) 0x7f85a8057d60 con 0x7f85b80b40b0 2024-03-27T21:40:58.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f85a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:40:58.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f85a6ffd640 10 monclient: _finish_command 2 = system:0 noscrub is unset 2024-03-27T21:40:58.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5ce37fe640 1 -- 172.21.15.60:0/461868192 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f5cd407aa90 con 0x7f5cec13b020 2024-03-27T21:40:58.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5ce37fe640 10 client.4815.objecter ms_dispatch 0x7f5cec000b90 osd_map(49..49 src has 1..49) v4 2024-03-27T21:40:58.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5ce37fe640 3 client.4815.objecter handle_osd_map got epochs [49,49] > 0 2024-03-27T21:40:58.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5ce37fe640 3 client.4815.objecter handle_osd_map decoding full epoch 49 2024-03-27T21:40:58.999 INFO:teuthology.orchestra.run.smithi060.stderr:noscrub is unset 2024-03-27T21:40:59.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5ce37fe640 20 client.4815.objecter dump_active .. 0 homeless 2024-03-27T21:40:59.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5ceaffd640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5ccc06c590 0x7f5ccc06ea50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc005880 tx=0x7f5cdc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:59.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5ce37fe640 10 client.4815.objecter ms_handle_connect 0x7f5ccc06c590 2024-03-27T21:40:59.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5ce37fe640 1 -- 172.21.15.60:0/461868192 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v49) v1 ==== 54+0+16 (secure 0 0 0) 0x7f5cd4076dd0 con 0x7f5cec13b020 2024-03-27T21:40:59.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5ce37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:40:59.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5ce37fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:40:59.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5cf14db640 10 client.4815.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:59.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign -1 1 2024-03-27T21:40:59.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5cf14db640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f5cec14cd30 0x7f5cec1d9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:59.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec069470 osd=0 3 2024-03-27T21:40:59.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec069470 osd=0 3 2024-03-27T21:40:59.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec069470 osd=0 3 2024-03-27T21:40:59.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_remove -1 1 2024-03-27T21:40:59.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5cf14db640 20 client.4815.objecter get_session s=0x7f5cec069470 osd=0 3 2024-03-27T21:40:59.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign 0 1 2024-03-27T21:40:59.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec069470 osd=0 4 2024-03-27T21:40:59.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5cf14db640 10 client.4815.objecter _send_command 1 2024-03-27T21:40:59.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.950+0000 7f5cf14db640 1 -- 172.21.15.60:0/461868192 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5cec13ebb0 con 0x7f5cec14cd30 2024-03-27T21:40:59.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.951+0000 7f5ceb7fe640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f5cec14cd30 0x7f5cec1d9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:59.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.951+0000 7f5ceb7fe640 10 monclient: get_auth_request con 0x7f5cec14cd30 auth_method 0 2024-03-27T21:40:59.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.951+0000 7f5ce37fe640 10 client.4815.objecter ms_handle_connect 0x7f5cec14cd30 2024-03-27T21:40:59.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.951+0000 7f5ceb7fe640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f5cec14cd30 0x7f5cec1d9c20 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:59.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f5ce37fe640 1 -- 172.21.15.60:0/461868192 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f5cec13ebb0 con 0x7f5cec14cd30 2024-03-27T21:40:59.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f5ce37fe640 10 client.4815.objecter ms_dispatch 0x7f5cec000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:40:59.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f5ce37fe640 10 client.4815.objecter _finish_command 1 = system:0 2024-03-27T21:40:59.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f5ce37fe640 20 client.4815.objecter put_session s=0x7f5cec069470 osd=0 4 2024-03-27T21:40:59.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f5ce37fe640 15 client.4815.objecter _session_command_op_remove 0 1 2024-03-27T21:40:59.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f85bc9eb640 10 librados: watch_flush enter 2024-03-27T21:40:59.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f85bc9eb640 10 librados: watch_flush exit 2024-03-27T21:40:59.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f85bc9eb640 20 client.4809.objecter shutdown clearing up homeless session... 2024-03-27T21:40:59.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f85bc9eb640 10 client.4809.objecter successfully canceled tick 2024-03-27T21:40:59.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f85bc9eb640 1 -- 172.21.15.60:0/778281124 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f859806c590 msgr2=0x7f859806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:59.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f85bc9eb640 1 --2- 172.21.15.60:0/778281124 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f859806c590 0x7f859806ea50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f85ac036000 tx=0x7f85ac03d040 comp rx=0 tx=0).stop 2024-03-27T21:40:59.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f85bc9eb640 10 monclient: shutdown 2024-03-27T21:40:59.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f85bc9eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:40:59.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f85bc9eb640 1 -- 172.21.15.60:0/778281124 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f85b80b40b0 msgr2=0x7f85b8139f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:40:59.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f85bc9eb640 1 --2- 172.21.15.60:0/778281124 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f85b80b40b0 0x7f85b8139f70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f85a800e9b0 tx=0x7f85a800ee70 comp rx=0 tx=0).stop 2024-03-27T21:40:59.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f85bc9eb640 1 -- 172.21.15.60:0/778281124 shutdown_connections 2024-03-27T21:40:59.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f85bc9eb640 1 --2- 172.21.15.60:0/778281124 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f85b81416f0 0x7f85b81d6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:59.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f85bc9eb640 1 --2- 172.21.15.60:0/778281124 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f859806c590 0x7f859806ea50 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:59.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f85bc9eb640 1 --2- 172.21.15.60:0/778281124 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85b813b020 0x7f85b8143ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:59.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f85bc9eb640 1 --2- 172.21.15.60:0/778281124 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f85b80b40b0 0x7f85b8139f70 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:40:59.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f85bc9eb640 1 -- 172.21.15.60:0/778281124 >> 172.21.15.60:0/778281124 conn(0x7f85b80608e0 msgr2=0x7f85b8060cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:40:59.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f85bc9eb640 1 -- 172.21.15.60:0/778281124 shutdown_connections 2024-03-27T21:40:59.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f85bc9eb640 1 -- 172.21.15.60:0/778281124 wait complete. 2024-03-27T21:40:59.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.952+0000 7f85bc9eb640 1 librados: shutdown 2024-03-27T21:40:59.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.969+0000 7f5cf14db640 10 client.4815.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:59.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.969+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign -1 2 2024-03-27T21:40:59.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.969+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec069470 osd=0 3 2024-03-27T21:40:59.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.969+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec069470 osd=0 3 2024-03-27T21:40:59.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.969+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec069470 osd=0 3 2024-03-27T21:40:59.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.969+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_remove -1 2 2024-03-27T21:40:59.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.969+0000 7f5cf14db640 20 client.4815.objecter get_session s=0x7f5cec069470 osd=0 3 2024-03-27T21:40:59.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.969+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign 0 2 2024-03-27T21:40:59.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.969+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec069470 osd=0 4 2024-03-27T21:40:59.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.969+0000 7f5cf14db640 10 client.4815.objecter _send_command 2 2024-03-27T21:40:59.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.969+0000 7f5cf14db640 1 -- 172.21.15.60:0/461868192 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5cec1d7060 con 0x7f5cec14cd30 2024-03-27T21:40:59.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.971+0000 7f5ce37fe640 1 -- 172.21.15.60:0/461868192 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5cec1d7060 con 0x7f5cec14cd30 2024-03-27T21:40:59.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.971+0000 7f5ce37fe640 10 client.4815.objecter ms_dispatch 0x7f5cec000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:59.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.971+0000 7f5ce37fe640 10 client.4815.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:59.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.971+0000 7f5ce37fe640 20 client.4815.objecter put_session s=0x7f5cec069470 osd=0 4 2024-03-27T21:40:59.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.971+0000 7f5ce37fe640 15 client.4815.objecter _session_command_op_remove 0 2 2024-03-27T21:40:59.008 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:40:59.008 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:40:59.016+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:40:59.010 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:40:59.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.971+0000 7f5cf14db640 10 client.4815.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:59.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.971+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign -1 3 2024-03-27T21:40:59.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.971+0000 7f5cf14db640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f5cec133280 0x7f5cec133660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:59.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.971+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec1da3a0 osd=1 3 2024-03-27T21:40:59.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.971+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec1da3a0 osd=1 3 2024-03-27T21:40:59.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.971+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec1da3a0 osd=1 3 2024-03-27T21:40:59.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.971+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_remove -1 3 2024-03-27T21:40:59.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.971+0000 7f5cf14db640 20 client.4815.objecter get_session s=0x7f5cec1da3a0 osd=1 3 2024-03-27T21:40:59.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.971+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign 1 3 2024-03-27T21:40:59.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.971+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec1da3a0 osd=1 4 2024-03-27T21:40:59.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.971+0000 7f5cf14db640 10 client.4815.objecter _send_command 3 2024-03-27T21:40:59.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.971+0000 7f5cf14db640 1 -- 172.21.15.60:0/461868192 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5cec133ba0 con 0x7f5cec133280 2024-03-27T21:40:59.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.972+0000 7f5ceaffd640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f5cec133280 0x7f5cec133660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:59.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.972+0000 7f5ceaffd640 10 monclient: get_auth_request con 0x7f5cec133280 auth_method 0 2024-03-27T21:40:59.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.972+0000 7f5ce37fe640 10 client.4815.objecter ms_handle_connect 0x7f5cec133280 2024-03-27T21:40:59.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.972+0000 7f5ceaffd640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f5cec133280 0x7f5cec133660 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:59.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.973+0000 7f5ce37fe640 1 -- 172.21.15.60:0/461868192 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f5cec133ba0 con 0x7f5cec133280 2024-03-27T21:40:59.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.973+0000 7f5ce37fe640 10 client.4815.objecter ms_dispatch 0x7f5cec000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:40:59.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.973+0000 7f5ce37fe640 10 client.4815.objecter _finish_command 3 = system:0 2024-03-27T21:40:59.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.973+0000 7f5ce37fe640 20 client.4815.objecter put_session s=0x7f5cec1da3a0 osd=1 4 2024-03-27T21:40:59.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.973+0000 7f5ce37fe640 15 client.4815.objecter _session_command_op_remove 1 3 2024-03-27T21:40:59.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.988+0000 7f5cf14db640 10 client.4815.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:59.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.988+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign -1 4 2024-03-27T21:40:59.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.988+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec1da3a0 osd=1 3 2024-03-27T21:40:59.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.988+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec1da3a0 osd=1 3 2024-03-27T21:40:59.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.988+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec1da3a0 osd=1 3 2024-03-27T21:40:59.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.988+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_remove -1 4 2024-03-27T21:40:59.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.988+0000 7f5cf14db640 20 client.4815.objecter get_session s=0x7f5cec1da3a0 osd=1 3 2024-03-27T21:40:59.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.988+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign 1 4 2024-03-27T21:40:59.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.988+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec1da3a0 osd=1 4 2024-03-27T21:40:59.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.988+0000 7f5cf14db640 10 client.4815.objecter _send_command 4 2024-03-27T21:40:59.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.988+0000 7f5cf14db640 1 -- 172.21.15.60:0/461868192 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5cec1da510 con 0x7f5cec133280 2024-03-27T21:40:59.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.990+0000 7f5ce37fe640 1 -- 172.21.15.60:0/461868192 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5cec1da510 con 0x7f5cec133280 2024-03-27T21:40:59.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.990+0000 7f5ce37fe640 10 client.4815.objecter ms_dispatch 0x7f5cec000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:59.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.990+0000 7f5ce37fe640 10 client.4815.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:59.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.990+0000 7f5ce37fe640 20 client.4815.objecter put_session s=0x7f5cec1da3a0 osd=1 4 2024-03-27T21:40:59.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.990+0000 7f5ce37fe640 15 client.4815.objecter _session_command_op_remove 1 4 2024-03-27T21:40:59.013 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:40:59.015 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:40:59.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.990+0000 7f5cf14db640 10 client.4815.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.990+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign -1 5 2024-03-27T21:40:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.990+0000 7f5cf14db640 20 client.4815.objecter _get_session osd=-1 returning homeless 2024-03-27T21:40:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.990+0000 7f5cf14db640 10 client.4815.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:40:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.990+0000 7f5cf14db640 10 monclient: _renew_subs 2024-03-27T21:40:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.990+0000 7f5cf14db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.990+0000 7f5cf14db640 1 -- 172.21.15.60:0/461868192 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=50}) v3 -- 0x7f5cec13a8c0 con 0x7f5cec13b020 2024-03-27T21:40:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.990+0000 7f5cf14db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:40:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.990+0000 7f5cf14db640 1 -- 172.21.15.60:0/461868192 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5cec13e1f0 con 0x7f5cec13b020 2024-03-27T21:40:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.991+0000 7f5ce37fe640 1 -- 172.21.15.60:0/461868192 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5cd4076dd0 con 0x7f5cec13b020 2024-03-27T21:40:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.991+0000 7f5ce37fe640 10 monclient: handle_get_version_reply finishing 1 version 49 2024-03-27T21:40:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.991+0000 7f5ce9ffb640 10 client.4815.objecter _check_command_map_dne tid 5 current 49 map_dne_bound 49 2024-03-27T21:40:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.991+0000 7f5ce9ffb640 10 client.4815.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:40:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.991+0000 7f5ce9ffb640 15 client.4815.objecter _session_command_op_remove -1 5 2024-03-27T21:40:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:40:59.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.991+0000 7f5cf14db640 10 client.4815.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:59.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.991+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign -1 6 2024-03-27T21:40:59.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.991+0000 7f5cf14db640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f5cec13dde0 0x7f5cec02b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:59.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.991+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec13dc70 osd=3 3 2024-03-27T21:40:59.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.991+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec13dc70 osd=3 3 2024-03-27T21:40:59.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.991+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec13dc70 osd=3 3 2024-03-27T21:40:59.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.991+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_remove -1 6 2024-03-27T21:40:59.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.991+0000 7f5cf14db640 20 client.4815.objecter get_session s=0x7f5cec13dc70 osd=3 3 2024-03-27T21:40:59.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.991+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign 3 6 2024-03-27T21:40:59.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.991+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec13dc70 osd=3 4 2024-03-27T21:40:59.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.991+0000 7f5cf14db640 10 client.4815.objecter _send_command 6 2024-03-27T21:40:59.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.991+0000 7f5cf14db640 1 -- 172.21.15.60:0/461868192 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5cec02bbc0 con 0x7f5cec13dde0 2024-03-27T21:40:59.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.991+0000 7f5ce3fff640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f5cec13dde0 0x7f5cec02b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:59.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.991+0000 7f5ce3fff640 10 monclient: get_auth_request con 0x7f5cec13dde0 auth_method 0 2024-03-27T21:40:59.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.991+0000 7f5ce37fe640 10 client.4815.objecter ms_handle_connect 0x7f5cec13dde0 2024-03-27T21:40:59.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.991+0000 7f5ce3fff640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f5cec13dde0 0x7f5cec02b680 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:59.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.992+0000 7f5ce37fe640 1 -- 172.21.15.60:0/461868192 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f5cec02bbc0 con 0x7f5cec13dde0 2024-03-27T21:40:59.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.992+0000 7f5ce37fe640 10 client.4815.objecter ms_dispatch 0x7f5cec000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:40:59.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.992+0000 7f5ce37fe640 10 client.4815.objecter _finish_command 6 = system:0 2024-03-27T21:40:59.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.992+0000 7f5ce37fe640 20 client.4815.objecter put_session s=0x7f5cec13dc70 osd=3 4 2024-03-27T21:40:59.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:58.992+0000 7f5ce37fe640 15 client.4815.objecter _session_command_op_remove 3 6 2024-03-27T21:40:59.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.005+0000 7f5cf14db640 10 client.4815.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:59.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.005+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign -1 7 2024-03-27T21:40:59.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.005+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec13dc70 osd=3 3 2024-03-27T21:40:59.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.005+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec13dc70 osd=3 3 2024-03-27T21:40:59.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.005+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec13dc70 osd=3 3 2024-03-27T21:40:59.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.005+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_remove -1 7 2024-03-27T21:40:59.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.005+0000 7f5cf14db640 20 client.4815.objecter get_session s=0x7f5cec13dc70 osd=3 3 2024-03-27T21:40:59.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.005+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign 3 7 2024-03-27T21:40:59.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.005+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec13dc70 osd=3 4 2024-03-27T21:40:59.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.005+0000 7f5cf14db640 10 client.4815.objecter _send_command 7 2024-03-27T21:40:59.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.005+0000 7f5cf14db640 1 -- 172.21.15.60:0/461868192 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5cec02c580 con 0x7f5cec13dde0 2024-03-27T21:40:59.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.007+0000 7f5ce37fe640 1 -- 172.21.15.60:0/461868192 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5cec02c580 con 0x7f5cec13dde0 2024-03-27T21:40:59.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.007+0000 7f5ce37fe640 10 client.4815.objecter ms_dispatch 0x7f5cec000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:59.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.007+0000 7f5ce37fe640 10 client.4815.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:59.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.007+0000 7f5ce37fe640 20 client.4815.objecter put_session s=0x7f5cec13dc70 osd=3 4 2024-03-27T21:40:59.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.007+0000 7f5ce37fe640 15 client.4815.objecter _session_command_op_remove 3 7 2024-03-27T21:40:59.019 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:40:59.021 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:40:59.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.007+0000 7f5cf14db640 10 client.4815.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:59.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.007+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign -1 8 2024-03-27T21:40:59.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.007+0000 7f5cf14db640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f5cec1e7190 0x7f5cec1e9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:59.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.007+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec13e7e0 osd=4 3 2024-03-27T21:40:59.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.007+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec13e7e0 osd=4 3 2024-03-27T21:40:59.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.007+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec13e7e0 osd=4 3 2024-03-27T21:40:59.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.007+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_remove -1 8 2024-03-27T21:40:59.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.007+0000 7f5cf14db640 20 client.4815.objecter get_session s=0x7f5cec13e7e0 osd=4 3 2024-03-27T21:40:59.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.007+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign 4 8 2024-03-27T21:40:59.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.007+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec13e7e0 osd=4 4 2024-03-27T21:40:59.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.007+0000 7f5cf14db640 10 client.4815.objecter _send_command 8 2024-03-27T21:40:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.007+0000 7f5cf14db640 1 -- 172.21.15.60:0/461868192 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5cec13e950 con 0x7f5cec1e7190 2024-03-27T21:40:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.007+0000 7f5ceb7fe640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f5cec1e7190 0x7f5cec1e9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.007+0000 7f5ceb7fe640 10 monclient: get_auth_request con 0x7f5cec1e7190 auth_method 0 2024-03-27T21:40:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.008+0000 7f5ce37fe640 10 client.4815.objecter ms_handle_connect 0x7f5cec1e7190 2024-03-27T21:40:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.008+0000 7f5ceb7fe640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f5cec1e7190 0x7f5cec1e9580 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.008+0000 7f5ce37fe640 1 -- 172.21.15.60:0/461868192 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f5cec13e950 con 0x7f5cec1e7190 2024-03-27T21:40:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.008+0000 7f5ce37fe640 10 client.4815.objecter ms_dispatch 0x7f5cec000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:40:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.008+0000 7f5ce37fe640 10 client.4815.objecter _finish_command 8 = system:0 2024-03-27T21:40:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.009+0000 7f5ce37fe640 20 client.4815.objecter put_session s=0x7f5cec13e7e0 osd=4 4 2024-03-27T21:40:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.009+0000 7f5ce37fe640 15 client.4815.objecter _session_command_op_remove 4 8 2024-03-27T21:40:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.025+0000 7f5cf14db640 10 client.4815.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.025+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign -1 9 2024-03-27T21:40:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.025+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec13e7e0 osd=4 3 2024-03-27T21:40:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.025+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec13e7e0 osd=4 3 2024-03-27T21:40:59.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.025+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec13e7e0 osd=4 3 2024-03-27T21:40:59.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.025+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_remove -1 9 2024-03-27T21:40:59.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.025+0000 7f5cf14db640 20 client.4815.objecter get_session s=0x7f5cec13e7e0 osd=4 3 2024-03-27T21:40:59.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.025+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign 4 9 2024-03-27T21:40:59.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.025+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec13e7e0 osd=4 4 2024-03-27T21:40:59.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.025+0000 7f5cf14db640 10 client.4815.objecter _send_command 9 2024-03-27T21:40:59.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.025+0000 7f5cf14db640 1 -- 172.21.15.60:0/461868192 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5cec02c000 con 0x7f5cec1e7190 2024-03-27T21:40:59.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.027+0000 7f5ce37fe640 1 -- 172.21.15.60:0/461868192 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5cec02c000 con 0x7f5cec1e7190 2024-03-27T21:40:59.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.027+0000 7f5ce37fe640 10 client.4815.objecter ms_dispatch 0x7f5cec000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:59.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.027+0000 7f5ce37fe640 10 client.4815.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:59.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.027+0000 7f5ce37fe640 20 client.4815.objecter put_session s=0x7f5cec13e7e0 osd=4 4 2024-03-27T21:40:59.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.027+0000 7f5ce37fe640 15 client.4815.objecter _session_command_op_remove 4 9 2024-03-27T21:40:59.024 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:40:59.026 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:40:59.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.028+0000 7f5cf14db640 10 client.4815.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:59.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.028+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign -1 10 2024-03-27T21:40:59.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.028+0000 7f5cf14db640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f5cec02c1a0 0x7f5cec1ecc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:59.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.028+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec02c000 osd=5 3 2024-03-27T21:40:59.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.028+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec02c000 osd=5 3 2024-03-27T21:40:59.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.028+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec02c000 osd=5 3 2024-03-27T21:40:59.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.028+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_remove -1 10 2024-03-27T21:40:59.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.028+0000 7f5cf14db640 20 client.4815.objecter get_session s=0x7f5cec02c000 osd=5 3 2024-03-27T21:40:59.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.028+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign 5 10 2024-03-27T21:40:59.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.028+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec02c000 osd=5 4 2024-03-27T21:40:59.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.028+0000 7f5cf14db640 10 client.4815.objecter _send_command 10 2024-03-27T21:40:59.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.028+0000 7f5cf14db640 1 -- 172.21.15.60:0/461868192 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5cec066760 con 0x7f5cec02c1a0 2024-03-27T21:40:59.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.028+0000 7f5ceaffd640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f5cec02c1a0 0x7f5cec1ecc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:59.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.028+0000 7f5ceaffd640 10 monclient: get_auth_request con 0x7f5cec02c1a0 auth_method 0 2024-03-27T21:40:59.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.028+0000 7f5ce37fe640 10 client.4815.objecter ms_handle_connect 0x7f5cec02c1a0 2024-03-27T21:40:59.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.028+0000 7f5ceaffd640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f5cec02c1a0 0x7f5cec1ecc50 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:59.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.029+0000 7f5ce37fe640 1 -- 172.21.15.60:0/461868192 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f5cec066760 con 0x7f5cec02c1a0 2024-03-27T21:40:59.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.029+0000 7f5ce37fe640 10 client.4815.objecter ms_dispatch 0x7f5cec000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:40:59.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.029+0000 7f5ce37fe640 10 client.4815.objecter _finish_command 10 = system:0 2024-03-27T21:40:59.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.029+0000 7f5ce37fe640 20 client.4815.objecter put_session s=0x7f5cec02c000 osd=5 4 2024-03-27T21:40:59.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.029+0000 7f5ce37fe640 15 client.4815.objecter _session_command_op_remove 5 10 2024-03-27T21:40:59.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.046+0000 7f5cf14db640 10 client.4815.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:59.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.046+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign -1 11 2024-03-27T21:40:59.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.046+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec02c000 osd=5 3 2024-03-27T21:40:59.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.046+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec02c000 osd=5 3 2024-03-27T21:40:59.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.046+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec02c000 osd=5 3 2024-03-27T21:40:59.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.046+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_remove -1 11 2024-03-27T21:40:59.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.046+0000 7f5cf14db640 20 client.4815.objecter get_session s=0x7f5cec02c000 osd=5 3 2024-03-27T21:40:59.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.046+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign 5 11 2024-03-27T21:40:59.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.046+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec02c000 osd=5 4 2024-03-27T21:40:59.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.046+0000 7f5cf14db640 10 client.4815.objecter _send_command 11 2024-03-27T21:40:59.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.046+0000 7f5cf14db640 1 -- 172.21.15.60:0/461868192 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5cec066760 con 0x7f5cec02c1a0 2024-03-27T21:40:59.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.048+0000 7f5ce37fe640 1 -- 172.21.15.60:0/461868192 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5cec066760 con 0x7f5cec02c1a0 2024-03-27T21:40:59.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.048+0000 7f5ce37fe640 10 client.4815.objecter ms_dispatch 0x7f5cec000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:59.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.048+0000 7f5ce37fe640 10 client.4815.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:59.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.048+0000 7f5ce37fe640 20 client.4815.objecter put_session s=0x7f5cec02c000 osd=5 4 2024-03-27T21:40:59.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.048+0000 7f5ce37fe640 15 client.4815.objecter _session_command_op_remove 5 11 2024-03-27T21:40:59.034 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:40:59.035 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:40:59.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.048+0000 7f5cf14db640 10 client.4815.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:59.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.048+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign -1 12 2024-03-27T21:40:59.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.048+0000 7f5cf14db640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f5cec1ed6a0 0x7f5cec1efa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:59.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.048+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec066760 osd=6 3 2024-03-27T21:40:59.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.048+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec066760 osd=6 3 2024-03-27T21:40:59.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.048+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec066760 osd=6 3 2024-03-27T21:40:59.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.048+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_remove -1 12 2024-03-27T21:40:59.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.048+0000 7f5cf14db640 20 client.4815.objecter get_session s=0x7f5cec066760 osd=6 3 2024-03-27T21:40:59.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.048+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign 6 12 2024-03-27T21:40:59.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.048+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec066760 osd=6 4 2024-03-27T21:40:59.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.048+0000 7f5cf14db640 10 client.4815.objecter _send_command 12 2024-03-27T21:40:59.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.048+0000 7f5cf14db640 1 -- 172.21.15.60:0/461868192 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5cec066900 con 0x7f5cec1ed6a0 2024-03-27T21:40:59.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.048+0000 7f5ce3fff640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f5cec1ed6a0 0x7f5cec1efa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:59.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.048+0000 7f5ce3fff640 10 monclient: get_auth_request con 0x7f5cec1ed6a0 auth_method 0 2024-03-27T21:40:59.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.049+0000 7f5ce3fff640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f5cec1ed6a0 0x7f5cec1efa90 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:59.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.049+0000 7f5ce37fe640 10 client.4815.objecter ms_handle_connect 0x7f5cec1ed6a0 2024-03-27T21:40:59.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.049+0000 7f5ce37fe640 1 -- 172.21.15.60:0/461868192 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f5cec066900 con 0x7f5cec1ed6a0 2024-03-27T21:40:59.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.049+0000 7f5ce37fe640 10 client.4815.objecter ms_dispatch 0x7f5cec000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:40:59.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.049+0000 7f5ce37fe640 10 client.4815.objecter _finish_command 12 = system:0 2024-03-27T21:40:59.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.049+0000 7f5ce37fe640 20 client.4815.objecter put_session s=0x7f5cec066760 osd=6 4 2024-03-27T21:40:59.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.049+0000 7f5ce37fe640 15 client.4815.objecter _session_command_op_remove 6 12 2024-03-27T21:40:59.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.066+0000 7f5cf14db640 10 client.4815.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:59.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.066+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign -1 13 2024-03-27T21:40:59.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.066+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec066760 osd=6 3 2024-03-27T21:40:59.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.066+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec066760 osd=6 3 2024-03-27T21:40:59.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.066+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec066760 osd=6 3 2024-03-27T21:40:59.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.066+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_remove -1 13 2024-03-27T21:40:59.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.066+0000 7f5cf14db640 20 client.4815.objecter get_session s=0x7f5cec066760 osd=6 3 2024-03-27T21:40:59.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.066+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign 6 13 2024-03-27T21:40:59.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.066+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec066760 osd=6 4 2024-03-27T21:40:59.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.066+0000 7f5cf14db640 10 client.4815.objecter _send_command 13 2024-03-27T21:40:59.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.066+0000 7f5cf14db640 1 -- 172.21.15.60:0/461868192 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5cec066900 con 0x7f5cec1ed6a0 2024-03-27T21:40:59.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.068+0000 7f5ce37fe640 1 -- 172.21.15.60:0/461868192 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5cec066900 con 0x7f5cec1ed6a0 2024-03-27T21:40:59.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.068+0000 7f5ce37fe640 10 client.4815.objecter ms_dispatch 0x7f5cec000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:59.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.068+0000 7f5ce37fe640 10 client.4815.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:59.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.068+0000 7f5ce37fe640 20 client.4815.objecter put_session s=0x7f5cec066760 osd=6 4 2024-03-27T21:40:59.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.068+0000 7f5ce37fe640 15 client.4815.objecter _session_command_op_remove 6 13 2024-03-27T21:40:59.055 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:40:59.056 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:40:59.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.068+0000 7f5cf14db640 10 client.4815.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:40:59.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.068+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign -1 14 2024-03-27T21:40:59.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.068+0000 7f5cf14db640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f5cec1f04e0 0x7f5cec1f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:40:59.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.068+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec066900 osd=7 3 2024-03-27T21:40:59.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.068+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec066900 osd=7 3 2024-03-27T21:40:59.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.068+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec066900 osd=7 3 2024-03-27T21:40:59.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.068+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_remove -1 14 2024-03-27T21:40:59.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.068+0000 7f5cf14db640 20 client.4815.objecter get_session s=0x7f5cec066900 osd=7 3 2024-03-27T21:40:59.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.068+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign 7 14 2024-03-27T21:40:59.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.068+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec066900 osd=7 4 2024-03-27T21:40:59.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.068+0000 7f5cf14db640 10 client.4815.objecter _send_command 14 2024-03-27T21:40:59.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.068+0000 7f5cf14db640 1 -- 172.21.15.60:0/461868192 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5cec1f56a0 con 0x7f5cec1f04e0 2024-03-27T21:40:59.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.068+0000 7f5ceb7fe640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f5cec1f04e0 0x7f5cec1f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:40:59.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.068+0000 7f5ceb7fe640 10 monclient: get_auth_request con 0x7f5cec1f04e0 auth_method 0 2024-03-27T21:40:59.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.069+0000 7f5ce37fe640 10 client.4815.objecter ms_handle_connect 0x7f5cec1f04e0 2024-03-27T21:40:59.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.069+0000 7f5ceb7fe640 1 --2- 172.21.15.60:0/461868192 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f5cec1f04e0 0x7f5cec1f28d0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:40:59.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.070+0000 7f5ce37fe640 1 -- 172.21.15.60:0/461868192 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f5cec1f56a0 con 0x7f5cec1f04e0 2024-03-27T21:40:59.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.070+0000 7f5ce37fe640 10 client.4815.objecter ms_dispatch 0x7f5cec000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:40:59.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.070+0000 7f5ce37fe640 10 client.4815.objecter _finish_command 14 = system:0 2024-03-27T21:40:59.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.070+0000 7f5ce37fe640 20 client.4815.objecter put_session s=0x7f5cec066900 osd=7 4 2024-03-27T21:40:59.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.070+0000 7f5ce37fe640 15 client.4815.objecter _session_command_op_remove 7 14 2024-03-27T21:40:59.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.099+0000 7f5cf14db640 10 client.4815.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:40:59.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.099+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign -1 15 2024-03-27T21:40:59.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.099+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec066900 osd=7 3 2024-03-27T21:40:59.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.099+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec066900 osd=7 3 2024-03-27T21:40:59.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.099+0000 7f5cf14db640 20 client.4815.objecter _get_session s=0x7f5cec066900 osd=7 3 2024-03-27T21:40:59.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.099+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_remove -1 15 2024-03-27T21:40:59.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.099+0000 7f5cf14db640 20 client.4815.objecter get_session s=0x7f5cec066900 osd=7 3 2024-03-27T21:40:59.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.099+0000 7f5cf14db640 15 client.4815.objecter _session_command_op_assign 7 15 2024-03-27T21:40:59.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.099+0000 7f5cf14db640 20 client.4815.objecter put_session s=0x7f5cec066900 osd=7 4 2024-03-27T21:40:59.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.099+0000 7f5cf14db640 10 client.4815.objecter _send_command 15 2024-03-27T21:40:59.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.099+0000 7f5cf14db640 1 -- 172.21.15.60:0/461868192 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5cec1f2e10 con 0x7f5cec1f04e0 2024-03-27T21:40:59.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.101+0000 7f5ce37fe640 1 -- 172.21.15.60:0/461868192 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5cec1f2e10 con 0x7f5cec1f04e0 2024-03-27T21:40:59.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.101+0000 7f5ce37fe640 10 client.4815.objecter ms_dispatch 0x7f5cec000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:40:59.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.101+0000 7f5ce37fe640 10 client.4815.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:40:59.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.101+0000 7f5ce37fe640 20 client.4815.objecter put_session s=0x7f5cec066900 osd=7 4 2024-03-27T21:40:59.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:40:59.101+0000 7f5ce37fe640 15 client.4815.objecter _session_command_op_remove 7 15 2024-03-27T21:40:59.088 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:40:59.088 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:40:59.091 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:40:59.092 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:40:59.133 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:59.147+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:59.133 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:59.134 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:59.143 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:40:59.203 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:59.218+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:59.291 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:59.291 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:59.300 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:40:59.305 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:59.319+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:59.405 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:40:59.419+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:40:59.411 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:59.411 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:59.420 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:40:59.506 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:40:59.520+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:40:59.568 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:59.568 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:59.577 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:40:59.607 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:40:59.621+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:40:59.685 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:59.685 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:59.693 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:40:59.707 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:59.722+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:59.808 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:40:59.823+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:40:59.817 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:59.817 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:59.826 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:40:59.910 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:40:59.924+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:40:59.943 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:40:59.943 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:40:59.953 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:41:00.010 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:00.025+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:00.060 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:00.060 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:00.068 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:41:00.111 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:00.126+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:00.213 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:00.226+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:00.214 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:00.214 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:00.222 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:41:00.313 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:00.327+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:00.317 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-27T21:41:00.318 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T21:41:00.318 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-03-27T21:41:00.318 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-27T21:41:00.318 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-27T21:41:00.318 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok config set filestore_inject_stall 3 2024-03-27T21:41:00.327 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:00.327 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:00.337 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:41:00.421 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:00.436+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:00.425 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:00.425 INFO:teuthology.orchestra.run.smithi191.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-27T21:41:00.425 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:00.485 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:00.485 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:00.494 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:41:00.537 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:00.552+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:00.598 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:00.598 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:00.606 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:41:00.648 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:00.662+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:00.717 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:00.732+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:00.752 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:00.752 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:00.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:41:00.818 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:00.832+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:00.906 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:00.906 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:00.914 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:41:00.958 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:00.973+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:01.007 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-27T21:41:01.019 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:01.019 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:01.028 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:41:01.051 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:01.065+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:01.092 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:41:01.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.128+0000 7fae7fd27640 10 monclient: get_monmap_and_config 2024-03-27T21:41:01.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.128+0000 7fae7fd27640 10 monclient: build_initial_monmap 2024-03-27T21:41:01.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.128+0000 7fae7fd27640 10 monclient: monmap: 2024-03-27T21:41:01.114 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:01.114 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:01.114 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:01.129261+0000 2024-03-27T21:41:01.114 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:01.129261+0000 2024-03-27T21:41:01.115 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:01.115 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:01.115 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:01.115 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:01.115 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:01.115 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:01.115 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:01.115 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:01.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.128+0000 7fae7fd27640 1 Processor -- start 2024-03-27T21:41:01.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7fd27640 1 -- start start 2024-03-27T21:41:01.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7fd27640 10 monclient: init 2024-03-27T21:41:01.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7fd27640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:01.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7fd27640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-03-27T21:41:01.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7fd27640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fae78060970 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:01.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7fd27640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fae780aefd0 0x7fae7813e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:01.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7fd27640 10 monclient(hunting): picked mon.noname-a con 0x7fae780aefd0 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:01.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7fd27640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fae7813e790 0x7fae7814bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:01.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7fd27640 10 monclient(hunting): picked mon.noname-b con 0x7fae7813e790 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:01.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7fd27640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae780b0580 con 0x7fae78060970 2024-03-27T21:41:01.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7fd27640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:01.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7fd27640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae78060d50 con 0x7fae780aefd0 2024-03-27T21:41:01.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7fd27640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:01.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7fd27640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae780af480 con 0x7fae7813e790 2024-03-27T21:41:01.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7fd27640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:01.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7fd27640 10 monclient(hunting): authenticate will time out at 902.350652s 2024-03-27T21:41:01.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7d2c2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fae780aefd0 0x7fae7813e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:01.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7e2c4640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fae7813e790 0x7fae7814bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:01.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7d2c2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fae780aefd0 0x7fae7813e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47240/0 (socket says 172.21.15.60:47240) 2024-03-27T21:41:01.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7d2c2640 1 -- 172.21.15.60:0/3429495339 learned_addr learned my addr 172.21.15.60:0/3429495339 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:01.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7d2c2640 10 monclient(hunting): get_auth_request con 0x7fae780aefd0 auth_method 0 2024-03-27T21:41:01.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7d2c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:01.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7d2c2640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:01.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7d2c2640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:01.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7e2c4640 10 monclient(hunting): get_auth_request con 0x7fae7813e790 auth_method 0 2024-03-27T21:41:01.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7e2c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:01.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7e2c4640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:01.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7e2c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:01.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7d2c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:01.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7d2c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:01.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7d2c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:01.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7e2c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:01.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.129+0000 7fae7e2c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:01.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.130+0000 7fae7e2c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:01.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.130+0000 7fae7e2c4640 10 monclient(hunting): handle_auth_done global_id 4658 payload 274 2024-03-27T21:41:01.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.130+0000 7fae7e2c4640 1 -- 172.21.15.60:0/3429495339 >> v2:172.21.15.60:3300/0 conn(0x7fae780aefd0 msgr2=0x7fae7813e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:01.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.130+0000 7fae7e2c4640 1 --2- 172.21.15.60:0/3429495339 >> v2:172.21.15.60:3300/0 conn(0x7fae780aefd0 0x7fae7813e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:01.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.130+0000 7fae7e2c4640 1 -- 172.21.15.60:0/3429495339 >> v1:172.21.15.60:6789/0 conn(0x7fae78060970 legacy=0x7fae78141350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:01.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.130+0000 7fae7e2c4640 10 monclient: _finish_hunting 0 2024-03-27T21:41:01.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.130+0000 7fae7e2c4640 1 monclient: found mon.noname-b 2024-03-27T21:41:01.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.130+0000 7fae7e2c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:01.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.130+0000 7fae7e2c4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:01.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.130+0000 7fae7e2c4640 1 -- 172.21.15.60:0/3429495339 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7813ee90 con 0x7fae7813e790 2024-03-27T21:41:01.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.130+0000 7fae7d2c2640 1 --2- 172.21.15.60:0/3429495339 >> v2:172.21.15.60:3300/0 conn(0x7fae780aefd0 0x7fae7813e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:01.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.131+0000 7fae7e2c4640 1 --2- 172.21.15.60:0/3429495339 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fae7813e790 0x7fae7814bc60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fae70009d90 tx=0x7fae7002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55d74d277cb8e291 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:01.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.131+0000 7fae7cac1640 1 -- 172.21.15.60:0/3429495339 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7002cbc0 con 0x7fae7813e790 2024-03-27T21:41:01.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.131+0000 7fae7cac1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:01.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.131+0000 7fae7cac1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:01.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.131+0000 7fae7cac1640 10 monclient: dump: 2024-03-27T21:41:01.122 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:01.123 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:01.123 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:01.123 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:01.123 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:01.123 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:01.123 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:01.123 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:01.123 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:01.123 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:01.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.131+0000 7fae7cac1640 10 monclient: _finish_auth 0 2024-03-27T21:41:01.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.131+0000 7fae7cac1640 10 monclient: _check_auth_tickets 2024-03-27T21:41:01.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.131+0000 7fae7cac1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:01.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.131+0000 7fae7cac1640 1 -- 172.21.15.60:0/3429495339 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7002cd20 con 0x7fae7813e790 2024-03-27T21:41:01.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.131+0000 7fae7fd27640 5 monclient: authenticate success, global_id 4658 2024-03-27T21:41:01.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.131+0000 7fae7fd27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:01.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.131+0000 7fae7cac1640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:01.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.131+0000 7fae7fd27640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:01.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.131+0000 7fae7cac1640 1 -- 172.21.15.60:0/3429495339 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae70034660 con 0x7fae7813e790 2024-03-27T21:41:01.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.131+0000 7fae7fd27640 4 set_mon_vals no callback set 2024-03-27T21:41:01.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.131+0000 7fae7cac1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:01.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.131+0000 7fae7cac1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:01.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.131+0000 7fae7cac1640 10 monclient: dump: 2024-03-27T21:41:01.124 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:01.125 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:01.125 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:01.125 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:01.125 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:01.125 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:01.125 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:01.125 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:01.125 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:01.125 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:01.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.132+0000 7fae7fd27640 10 monclient: shutdown 2024-03-27T21:41:01.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.132+0000 7fae7fd27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:01.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.132+0000 7fae7fd27640 1 -- 172.21.15.60:0/3429495339 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fae7813e790 msgr2=0x7fae7814bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:01.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.132+0000 7fae7fd27640 1 --2- 172.21.15.60:0/3429495339 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fae7813e790 0x7fae7814bc60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fae70009d90 tx=0x7fae7002be10 comp rx=0 tx=0).stop 2024-03-27T21:41:01.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.132+0000 7fae7fd27640 1 -- 172.21.15.60:0/3429495339 shutdown_connections 2024-03-27T21:41:01.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.132+0000 7fae7fd27640 1 --2- 172.21.15.60:0/3429495339 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fae7813e790 0x7fae7814bc60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:01.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.132+0000 7fae7fd27640 1 --2- 172.21.15.60:0/3429495339 >> v2:172.21.15.60:3300/0 conn(0x7fae780aefd0 0x7fae7813e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:01.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.132+0000 7fae7fd27640 1 -- 172.21.15.60:0/3429495339 >> 172.21.15.60:0/3429495339 conn(0x7fae780b4850 msgr2=0x7fae78069ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:01.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.132+0000 7fae7fd27640 1 -- 172.21.15.60:0/3429495339 shutdown_connections 2024-03-27T21:41:01.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.132+0000 7fae7fd27640 1 -- 172.21.15.60:0/3429495339 wait complete. 2024-03-27T21:41:01.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.133+0000 7fae7fd27640 10 monclient: build_initial_monmap 2024-03-27T21:41:01.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.133+0000 7fae7fd27640 10 monclient: monmap: 2024-03-27T21:41:01.131 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:01.131 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:01.131 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:01.131 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:01.132 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.134+0000 7fae7fd27640 1 librados: starting msgr at 2024-03-27T21:41:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.134+0000 7fae7fd27640 1 librados: starting objecter 2024-03-27T21:41:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.134+0000 7fae7fd27640 1 Processor -- start 2024-03-27T21:41:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.134+0000 7fae7fd27640 1 -- start start 2024-03-27T21:41:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.134+0000 7fae7fd27640 1 librados: setting wanted keys 2024-03-27T21:41:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.134+0000 7fae7fd27640 1 librados: calling monclient init 2024-03-27T21:41:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.134+0000 7fae7fd27640 10 monclient: init 2024-03-27T21:41:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7fd27640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7fd27640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:41:01.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7fd27640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae78060970 0x7fae7811f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:01.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7fd27640 10 monclient(hunting): picked mon.noname-b con 0x7fae78060970 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:01.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7fd27640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fae780aefd0 0x7fae78120440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:01.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7fd27640 10 monclient(hunting): picked mon.noname-c con 0x7fae780aefd0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:01.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7fd27640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae7813e790 0x7fae781d9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:01.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7d2c2640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fae780aefd0 0x7fae78120440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:01.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7d2c2640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fae780aefd0 0x7fae78120440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:57886/0 (socket says 172.21.15.60:57886) 2024-03-27T21:41:01.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7dac3640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae78060970 0x7fae7811f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:01.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7d2c2640 1 -- 172.21.15.60:0/461285645 learned_addr learned my addr 172.21.15.60:0/461285645 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:01.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7fd27640 10 monclient(hunting): picked mon.noname-a con 0x7fae7813e790 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:01.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7fd27640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:01.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7fd27640 1 -- 172.21.15.60:0/461285645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae781d96d0 con 0x7fae7813e790 2024-03-27T21:41:01.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7fd27640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:01.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7fd27640 1 -- 172.21.15.60:0/461285645 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae78121a60 con 0x7fae78060970 2024-03-27T21:41:01.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7fd27640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:01.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7fd27640 1 -- 172.21.15.60:0/461285645 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae78120980 con 0x7fae780aefd0 2024-03-27T21:41:01.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7fd27640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:01.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7fd27640 10 monclient(hunting): authenticate will time out at 902.356419s 2024-03-27T21:41:01.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7d2c2640 10 monclient(hunting): get_auth_request con 0x7fae780aefd0 auth_method 0 2024-03-27T21:41:01.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7d2c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:01.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7d2c2640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:01.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7d2c2640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:01.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7dac3640 10 monclient(hunting): get_auth_request con 0x7fae78060970 auth_method 0 2024-03-27T21:41:01.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7dac3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:01.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7dac3640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:01.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7dac3640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:01.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7e2c4640 1 --2- 172.21.15.60:0/461285645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae7813e790 0x7fae781d9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:01.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7dac3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:01.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7dac3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:01.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7dac3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:01.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7e2c4640 10 monclient(hunting): get_auth_request con 0x7fae7813e790 auth_method 0 2024-03-27T21:41:01.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7e2c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:01.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7e2c4640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:01.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7e2c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:01.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7e2c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:01.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7e2c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:01.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7e2c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:01.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7dac3640 10 monclient(hunting): handle_auth_done global_id 4759 payload 824 2024-03-27T21:41:01.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.135+0000 7fae7dac3640 1 -- 172.21.15.60:0/461285645 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fae780aefd0 msgr2=0x7fae78120440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:01.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae7dac3640 1 --2- 172.21.15.60:0/461285645 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fae780aefd0 0x7fae78120440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:01.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae7dac3640 1 -- 172.21.15.60:0/461285645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae7813e790 msgr2=0x7fae781d9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:01.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae7dac3640 1 --2- 172.21.15.60:0/461285645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae7813e790 0x7fae781d9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:01.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae7dac3640 10 monclient: _finish_hunting 0 2024-03-27T21:41:01.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae7dac3640 1 monclient: found mon.noname-b 2024-03-27T21:41:01.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae7dac3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:01.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae7dac3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:41:01.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae7dac3640 1 -- 172.21.15.60:0/461285645 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae680094d0 con 0x7fae78060970 2024-03-27T21:41:01.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae7d2c2640 1 --2- 172.21.15.60:0/461285645 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fae780aefd0 0x7fae78120440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:41:01.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae7dac3640 1 --2- 172.21.15.60:0/461285645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae78060970 0x7fae7811f600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fae7400c390 tx=0x7fae7400c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:01.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae6e7fc640 10 client.?.objecter ms_handle_connect 0x7fae78060970 2024-03-27T21:41:01.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:01.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae7e2c4640 1 --2- 172.21.15.60:0/461285645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae7813e790 0x7fae781d9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:01.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae6e7fc640 1 -- 172.21.15.60:0/461285645 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae74005220 con 0x7fae78060970 2024-03-27T21:41:01.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:01.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae6e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:01.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae6e7fc640 10 monclient: dump: 2024-03-27T21:41:01.136 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:01.136 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:01.136 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:01.137 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:01.137 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:01.137 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:01.137 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:01.137 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:01.137 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:01.137 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:01.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae6e7fc640 10 monclient: _finish_auth 0 2024-03-27T21:41:01.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae6e7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:41:01.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:01.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae6e7fc640 1 -- 172.21.15.60:0/461285645 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae74005380 con 0x7fae78060970 2024-03-27T21:41:01.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:01.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.136+0000 7fae6ffff640 4 set_mon_vals no callback set 2024-03-27T21:41:01.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.137+0000 7fae6e7fc640 1 -- 172.21.15.60:0/461285645 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae740054f0 con 0x7fae78060970 2024-03-27T21:41:01.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.137+0000 7fae6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:01.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.137+0000 7fae6e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:01.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.137+0000 7fae6e7fc640 10 monclient: dump: 2024-03-27T21:41:01.138 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:01.138 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:01.138 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:01.138 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:01.138 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:01.138 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:01.138 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:01.138 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:01.138 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:01.138 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:01.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.137+0000 7fae7fd27640 5 monclient: authenticate success, global_id 4759 2024-03-27T21:41:01.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.137+0000 7fae7fd27640 10 monclient: _renew_subs 2024-03-27T21:41:01.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.137+0000 7fae7fd27640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:01.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.137+0000 7fae7fd27640 1 -- 172.21.15.60:0/461285645 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae70009a70 con 0x7fae78060970 2024-03-27T21:41:01.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.137+0000 7fae7fd27640 10 client.4759.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:01.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.137+0000 7fae7fd27640 10 monclient: _renew_subs 2024-03-27T21:41:01.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.137+0000 7fae7fd27640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:01.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.137+0000 7fae7fd27640 1 -- 172.21.15.60:0/461285645 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7811ca20 con 0x7fae78060970 2024-03-27T21:41:01.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.137+0000 7fae7fd27640 1 librados: init done 2024-03-27T21:41:01.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.138+0000 7fae7fd27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:01.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.138+0000 7fae7fd27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:01.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.138+0000 7fae7fd27640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:01.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.138+0000 7fae7fd27640 1 -- 172.21.15.60:0/461285645 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae78069470 con 0x7fae78060970 2024-03-27T21:41:01.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.139+0000 7fae6e7fc640 1 -- 172.21.15.60:0/461285645 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fae74007230 con 0x7fae78060970 2024-03-27T21:41:01.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.139+0000 7fae6e7fc640 1 --2- 172.21.15.60:0/461285645 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fae5c06c590 0x7fae5c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:01.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.139+0000 7fae6e7fc640 1 -- 172.21.15.60:0/461285645 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fae74088610 con 0x7fae78060970 2024-03-27T21:41:01.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.139+0000 7fae6e7fc640 10 client.4759.objecter ms_dispatch 0x7fae78000b90 osd_map(49..49 src has 1..49) v4 2024-03-27T21:41:01.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.139+0000 7fae6e7fc640 3 client.4759.objecter handle_osd_map got epochs [49,49] > 0 2024-03-27T21:41:01.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.139+0000 7fae6e7fc640 3 client.4759.objecter handle_osd_map decoding full epoch 49 2024-03-27T21:41:01.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.139+0000 7fae7d2c2640 1 --2- 172.21.15.60:0/461285645 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fae5c06c590 0x7fae5c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:01.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.139+0000 7fae7d2c2640 10 monclient: get_auth_request con 0x7fae5c06c590 auth_method 0 2024-03-27T21:41:01.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.139+0000 7fae6e7fc640 20 client.4759.objecter dump_active .. 0 homeless 2024-03-27T21:41:01.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.139+0000 7fae6e7fc640 10 client.4759.objecter ms_handle_connect 0x7fae5c06c590 2024-03-27T21:41:01.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.139+0000 7fae7d2c2640 1 --2- 172.21.15.60:0/461285645 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fae5c06c590 0x7fae5c06ea50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fae780ae3d0 tx=0x7fae68008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:01.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.144+0000 7fae6e7fc640 1 -- 172.21.15.60:0/461285645 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fae740539d0 con 0x7fae78060970 2024-03-27T21:41:01.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.144+0000 7fae6e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:01.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.144+0000 7fae6e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:01.141 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:01.141+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:01.174 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:01.175 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:01.184 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:41:01.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.217+0000 7f57851a4640 10 monclient: get_monmap_and_config 2024-03-27T21:41:01.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.217+0000 7f57851a4640 10 monclient: build_initial_monmap 2024-03-27T21:41:01.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.217+0000 7f57851a4640 10 monclient: monmap: 2024-03-27T21:41:01.204 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:01.204 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:01.204 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:01.218604+0000 2024-03-27T21:41:01.204 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:01.218604+0000 2024-03-27T21:41:01.204 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:01.204 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:01.204 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:01.205 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:01.205 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:01.205 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:01.205 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:01.205 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:01.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.218+0000 7f57851a4640 1 Processor -- start 2024-03-27T21:41:01.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.218+0000 7f57851a4640 1 -- start start 2024-03-27T21:41:01.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.218+0000 7f57851a4640 10 monclient: init 2024-03-27T21:41:01.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f57851a4640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:01.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f57851a4640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-03-27T21:41:01.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f57851a4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5780071a50 0x7f57800a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:01.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f57851a4640 10 monclient(hunting): picked mon.noname-a con 0x7f5780071a50 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:01.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f57851a4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f578013e810 0x7f578013ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:01.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f57851a4640 10 monclient(hunting): picked mon.noname-c con 0x7f578013e810 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:01.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f57851a4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5780141630 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:01.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f57851a4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f578013f190 con 0x7f5780141630 2024-03-27T21:41:01.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f57851a4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:01.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f57851a4640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5780071e30 con 0x7f5780071a50 2024-03-27T21:41:01.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f57851a4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:01.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f57851a4640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57800a8e90 con 0x7f578013e810 2024-03-27T21:41:01.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f57851a4640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:01.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f57851a4640 10 monclient(hunting): authenticate will time out at 902.440459s 2024-03-27T21:41:01.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f577ed9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5780071a50 0x7f57800a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:01.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f577ed9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5780071a50 0x7f57800a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47262/0 (socket says 172.21.15.60:47262) 2024-03-27T21:41:01.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f577ed9d640 1 -- 172.21.15.60:0/3494260904 learned_addr learned my addr 172.21.15.60:0/3494260904 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:01.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f577ed9d640 10 monclient(hunting): get_auth_request con 0x7f5780071a50 auth_method 0 2024-03-27T21:41:01.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f577ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:01.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f577ed9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:01.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f577ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:01.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f5777fff640 1 --2- 172.21.15.60:0/3494260904 >> v2:172.21.15.191:3300/0 conn(0x7f578013e810 0x7f578013ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:01.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f5777fff640 10 monclient(hunting): get_auth_request con 0x7f578013e810 auth_method 0 2024-03-27T21:41:01.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f5777fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:01.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f5777fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:01.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f5777fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:01.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f577ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:01.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f577ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:01.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f577ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:01.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f5777fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:01.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f5777fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:01.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.219+0000 7f5777fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:01.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.220+0000 7f577ed9d640 10 monclient(hunting): handle_auth_done global_id 4824 payload 274 2024-03-27T21:41:01.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.220+0000 7f577ed9d640 1 -- 172.21.15.60:0/3494260904 >> v2:172.21.15.191:3300/0 conn(0x7f578013e810 msgr2=0x7f578013ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:01.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.220+0000 7f577ed9d640 1 --2- 172.21.15.60:0/3494260904 >> v2:172.21.15.191:3300/0 conn(0x7f578013e810 0x7f578013ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:01.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.220+0000 7f577ed9d640 1 -- 172.21.15.60:0/3494260904 >> v1:172.21.15.191:6789/0 conn(0x7f5780141630 legacy=0x7f5780141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:01.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.220+0000 7f577ed9d640 10 monclient: _finish_hunting 0 2024-03-27T21:41:01.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.220+0000 7f577ed9d640 1 monclient: found mon.noname-a 2024-03-27T21:41:01.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.220+0000 7f577ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:01.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.220+0000 7f577ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:01.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.220+0000 7f577ed9d640 1 -- 172.21.15.60:0/3494260904 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57800aa040 con 0x7f5780071a50 2024-03-27T21:41:01.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f577ed9d640 1 --2- 172.21.15.60:0/3494260904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5780071a50 0x7f57800a87e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f57800709f0 tx=0x7f577002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f467bff1c09d25 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:01.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f577e59c640 1 -- 172.21.15.60:0/3494260904 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5770031870 con 0x7f5780071a50 2024-03-27T21:41:01.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f577e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:01.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f577e59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:01.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f577e59c640 10 monclient: dump: 2024-03-27T21:41:01.224 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:01.224 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:01.224 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:01.224 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:01.224 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:01.224 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:01.224 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:01.224 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:01.225 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:01.225 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:01.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f577e59c640 10 monclient: _finish_auth 0 2024-03-27T21:41:01.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f577e59c640 10 monclient: _check_auth_tickets 2024-03-27T21:41:01.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f577e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:01.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f577e59c640 1 -- 172.21.15.60:0/3494260904 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57700319d0 con 0x7f5780071a50 2024-03-27T21:41:01.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f577e59c640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:01.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f577e59c640 1 -- 172.21.15.60:0/3494260904 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5770031d60 con 0x7f5780071a50 2024-03-27T21:41:01.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f57851a4640 5 monclient: authenticate success, global_id 4824 2024-03-27T21:41:01.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f57851a4640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:01.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f57851a4640 4 set_mon_vals no callback set 2024-03-27T21:41:01.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f577e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:01.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f577e59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:01.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f577e59c640 10 monclient: dump: 2024-03-27T21:41:01.226 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:01.226 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:01.226 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:01.227 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:01.227 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:01.227 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:01.227 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:01.227 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:01.227 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:01.227 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:01.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f57851a4640 10 monclient: shutdown 2024-03-27T21:41:01.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f57851a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:01.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f57851a4640 1 -- 172.21.15.60:0/3494260904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5780071a50 msgr2=0x7f57800a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:01.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f57851a4640 1 --2- 172.21.15.60:0/3494260904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5780071a50 0x7f57800a87e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f57800709f0 tx=0x7f577002b910 comp rx=0 tx=0).stop 2024-03-27T21:41:01.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f57851a4640 1 -- 172.21.15.60:0/3494260904 shutdown_connections 2024-03-27T21:41:01.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f57851a4640 1 --2- 172.21.15.60:0/3494260904 >> v2:172.21.15.191:3300/0 conn(0x7f578013e810 0x7f578013ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:01.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f57851a4640 1 --2- 172.21.15.60:0/3494260904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5780071a50 0x7f57800a87e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:01.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f57851a4640 1 -- 172.21.15.60:0/3494260904 >> 172.21.15.60:0/3494260904 conn(0x7f57800b7300 msgr2=0x7f57800b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:01.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f57851a4640 1 -- 172.21.15.60:0/3494260904 shutdown_connections 2024-03-27T21:41:01.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.221+0000 7f57851a4640 1 -- 172.21.15.60:0/3494260904 wait complete. 2024-03-27T21:41:01.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.222+0000 7f57851a4640 10 monclient: build_initial_monmap 2024-03-27T21:41:01.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.222+0000 7f57851a4640 10 monclient: monmap: 2024-03-27T21:41:01.229 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:01.229 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:01.229 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:01.229 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:01.229 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:01.230 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:01.230 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:01.230 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:01.230 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:01.230 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:01.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 1 librados: starting msgr at 2024-03-27T21:41:01.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 1 librados: starting objecter 2024-03-27T21:41:01.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 1 Processor -- start 2024-03-27T21:41:01.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 1 -- start start 2024-03-27T21:41:01.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 1 librados: setting wanted keys 2024-03-27T21:41:01.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 1 librados: calling monclient init 2024-03-27T21:41:01.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 10 monclient: init 2024-03-27T21:41:01.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:01.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:41:01.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5780071a50 0x7f5780135f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:01.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 10 monclient(hunting): picked mon.noname-c con 0x7f5780071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:01.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f578013e810 0x7f57801392f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:01.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 10 monclient(hunting): picked mon.noname-b con 0x7f578013e810 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:01.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5780141630 0x7f5780131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:01.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 10 monclient(hunting): picked mon.noname-a con 0x7f5780141630 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:01.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:01.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5780139c50 con 0x7f5780141630 2024-03-27T21:41:01.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:01.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5780138a50 con 0x7f578013e810 2024-03-27T21:41:01.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:01.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5780138bc0 con 0x7f5780071a50 2024-03-27T21:41:01.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:01.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.223+0000 7f57851a4640 10 monclient(hunting): authenticate will time out at 902.444870s 2024-03-27T21:41:01.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577f59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5780141630 0x7f5780131fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:01.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577f59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5780141630 0x7f5780131fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47268/0 (socket says 172.21.15.60:47268) 2024-03-27T21:41:01.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577ed9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5780071a50 0x7f5780135f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:01.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577f59e640 1 -- 172.21.15.60:0/4056101675 learned_addr learned my addr 172.21.15.60:0/4056101675 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:01.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577f59e640 10 monclient(hunting): get_auth_request con 0x7f5780141630 auth_method 0 2024-03-27T21:41:01.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:01.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577f59e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:01.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577f59e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:01.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577ed9d640 10 monclient(hunting): get_auth_request con 0x7f5780071a50 auth_method 0 2024-03-27T21:41:01.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:01.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577ed9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:01.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:01.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:01.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:01.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f5777fff640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f578013e810 0x7f57801392f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:01.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:01.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:01.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:01.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:01.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f5777fff640 10 monclient(hunting): get_auth_request con 0x7f578013e810 auth_method 0 2024-03-27T21:41:01.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f5777fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:01.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f5777fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:01.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f5777fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:01.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f5777fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:01.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f5777fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:01.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f5777fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:01.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577ed9d640 10 monclient(hunting): handle_auth_done global_id 4664 payload 824 2024-03-27T21:41:01.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577ed9d640 1 -- 172.21.15.60:0/4056101675 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f578013e810 msgr2=0x7f57801392f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:01.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577ed9d640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f578013e810 0x7f57801392f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:01.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577ed9d640 1 -- 172.21.15.60:0/4056101675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5780141630 msgr2=0x7f5780131fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:01.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577ed9d640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5780141630 0x7f5780131fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:01.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577ed9d640 10 monclient: _finish_hunting 0 2024-03-27T21:41:01.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577ed9d640 1 monclient: found mon.noname-c 2024-03-27T21:41:01.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:01.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:01.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577ed9d640 1 -- 172.21.15.60:0/4056101675 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57700094d0 con 0x7f5780071a50 2024-03-27T21:41:01.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577f59e640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5780141630 0x7f5780131fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:01.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577ed9d640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5780071a50 0x7f5780135f90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5770033000 tx=0x7f57700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:01.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f57777fe640 10 client.?.objecter ms_handle_connect 0x7f5780071a50 2024-03-27T21:41:01.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f57777fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:01.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f57777fe640 1 -- 172.21.15.60:0/4056101675 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5770043c60 con 0x7f5780071a50 2024-03-27T21:41:01.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f57777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:01.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f57777fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:01.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f57777fe640 10 monclient: dump: 2024-03-27T21:41:01.235 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:01.235 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:01.236 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:01.236 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:01.236 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:01.236 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:01.236 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:01.236 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:01.236 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:01.236 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:01.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f57777fe640 10 monclient: _finish_auth 0 2024-03-27T21:41:01.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f57777fe640 10 monclient: _check_auth_tickets 2024-03-27T21:41:01.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f57777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:01.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f57851a4640 5 monclient: authenticate success, global_id 4664 2024-03-27T21:41:01.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f57777fe640 1 -- 172.21.15.60:0/4056101675 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5770047070 con 0x7f5780071a50 2024-03-27T21:41:01.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f57851a4640 10 monclient: _renew_subs 2024-03-27T21:41:01.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f57851a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:01.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f57851a4640 1 -- 172.21.15.60:0/4056101675 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5780139830 con 0x7f5780071a50 2024-03-27T21:41:01.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f57777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:01.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f57777fe640 1 -- 172.21.15.60:0/4056101675 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577003a7d0 con 0x7f5780071a50 2024-03-27T21:41:01.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f57777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:01.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f57851a4640 10 client.4664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:01.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f57777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:01.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f57777fe640 10 monclient: dump: 2024-03-27T21:41:01.237 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:01.237 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:01.237 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:01.237 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:01.237 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:01.237 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:01.237 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:01.238 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:01.238 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:01.238 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:01.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.224+0000 7f577dd9b640 4 set_mon_vals no callback set 2024-03-27T21:41:01.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.225+0000 7f57851a4640 10 monclient: _renew_subs 2024-03-27T21:41:01.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.225+0000 7f57851a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:01.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.225+0000 7f57851a4640 1 -- 172.21.15.60:0/4056101675 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57801326f0 con 0x7f5780071a50 2024-03-27T21:41:01.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.225+0000 7f57851a4640 1 librados: init done 2024-03-27T21:41:01.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.226+0000 7f57777fe640 1 -- 172.21.15.60:0/4056101675 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f577003a930 con 0x7f5780071a50 2024-03-27T21:41:01.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.226+0000 7f57851a4640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:41:01.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.226+0000 7f57851a4640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:01.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.226+0000 7f57851a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:01.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.226+0000 7f57851a4640 1 -- 172.21.15.60:0/4056101675 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f5780069470 con 0x7f5780071a50 2024-03-27T21:41:01.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.226+0000 7f57777fe640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f576006c5c0 0x7f576006ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:01.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.226+0000 7f57777fe640 1 -- 172.21.15.60:0/4056101675 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f577002d040 con 0x7f5780071a50 2024-03-27T21:41:01.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.226+0000 7f57777fe640 10 client.4664.objecter ms_dispatch 0x7f5780000b90 osd_map(49..49 src has 1..49) v4 2024-03-27T21:41:01.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.226+0000 7f57777fe640 3 client.4664.objecter handle_osd_map got epochs [49,49] > 0 2024-03-27T21:41:01.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.226+0000 7f57777fe640 3 client.4664.objecter handle_osd_map decoding full epoch 49 2024-03-27T21:41:01.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.226+0000 7f5777fff640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f576006c5c0 0x7f576006ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:01.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.226+0000 7f5777fff640 10 monclient: get_auth_request con 0x7f576006c5c0 auth_method 0 2024-03-27T21:41:01.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.226+0000 7f57777fe640 20 client.4664.objecter dump_active .. 0 homeless 2024-03-27T21:41:01.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.226+0000 7f57777fe640 1 -- 172.21.15.60:0/4056101675 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v49) v1 ==== 54+0+16 (secure 0 0 0) 0x7f57700ba070 con 0x7f5780071a50 2024-03-27T21:41:01.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.226+0000 7f57777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:01.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.226+0000 7f57777fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:01.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.227+0000 7f57851a4640 10 client.4664.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:01.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.227+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign -1 1 2024-03-27T21:41:01.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.227+0000 7f57851a4640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f57801d9e90 0x7f57801da270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:01.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.227+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f5780139ad0 osd=0 3 2024-03-27T21:41:01.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.227+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f5780139ad0 osd=0 3 2024-03-27T21:41:01.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.227+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f5780139ad0 osd=0 3 2024-03-27T21:41:01.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.227+0000 7f57851a4640 15 client.4664.objecter _session_command_op_remove -1 1 2024-03-27T21:41:01.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.227+0000 7f57851a4640 20 client.4664.objecter get_session s=0x7f5780139ad0 osd=0 3 2024-03-27T21:41:01.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.227+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign 0 1 2024-03-27T21:41:01.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.227+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f5780139ad0 osd=0 4 2024-03-27T21:41:01.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.227+0000 7f57851a4640 10 client.4664.objecter _send_command 1 2024-03-27T21:41:01.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.227+0000 7f57851a4640 1 -- 172.21.15.60:0/4056101675 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57801421b0 con 0x7f57801d9e90 2024-03-27T21:41:01.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.227+0000 7f577f59e640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f57801d9e90 0x7f57801da270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:01.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.227+0000 7f57777fe640 10 client.4664.objecter ms_handle_connect 0x7f576006c5c0 2024-03-27T21:41:01.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.227+0000 7f577f59e640 10 monclient: get_auth_request con 0x7f57801d9e90 auth_method 0 2024-03-27T21:41:01.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.227+0000 7f5777fff640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f576006c5c0 0x7f576006ea80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f57801daa60 tx=0x7f5768009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:01.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.227+0000 7f57777fe640 10 client.4664.objecter ms_handle_connect 0x7f57801d9e90 2024-03-27T21:41:01.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.227+0000 7f577f59e640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f57801d9e90 0x7f57801da270 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:01.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.228+0000 7f57777fe640 1 -- 172.21.15.60:0/4056101675 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f577800f040 con 0x7f57801d9e90 2024-03-27T21:41:01.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.228+0000 7f57777fe640 10 client.4664.objecter ms_dispatch 0x7f5780000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:41:01.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.228+0000 7f57777fe640 10 client.4664.objecter _finish_command 1 = system:0 2024-03-27T21:41:01.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.228+0000 7f57777fe640 20 client.4664.objecter put_session s=0x7f5780139ad0 osd=0 4 2024-03-27T21:41:01.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.228+0000 7f57777fe640 15 client.4664.objecter _session_command_op_remove 0 1 2024-03-27T21:41:01.242 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:01.256+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:01.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.257+0000 7f57851a4640 10 client.4664.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:01.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.257+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign -1 2 2024-03-27T21:41:01.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.257+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f5780139ad0 osd=0 3 2024-03-27T21:41:01.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.257+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f5780139ad0 osd=0 3 2024-03-27T21:41:01.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.257+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f5780139ad0 osd=0 3 2024-03-27T21:41:01.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.257+0000 7f57851a4640 15 client.4664.objecter _session_command_op_remove -1 2 2024-03-27T21:41:01.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.257+0000 7f57851a4640 20 client.4664.objecter get_session s=0x7f5780139ad0 osd=0 3 2024-03-27T21:41:01.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.257+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign 0 2 2024-03-27T21:41:01.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.257+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f5780139ad0 osd=0 4 2024-03-27T21:41:01.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.257+0000 7f57851a4640 10 client.4664.objecter _send_command 2 2024-03-27T21:41:01.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.257+0000 7f57851a4640 1 -- 172.21.15.60:0/4056101675 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f578013f190 con 0x7f57801d9e90 2024-03-27T21:41:01.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.259+0000 7f57777fe640 1 -- 172.21.15.60:0/4056101675 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f578013f190 con 0x7f57801d9e90 2024-03-27T21:41:01.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.259+0000 7f57777fe640 10 client.4664.objecter ms_dispatch 0x7f5780000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:01.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.259+0000 7f57777fe640 10 client.4664.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:01.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.259+0000 7f57777fe640 20 client.4664.objecter put_session s=0x7f5780139ad0 osd=0 4 2024-03-27T21:41:01.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.259+0000 7f57777fe640 15 client.4664.objecter _session_command_op_remove 0 2 2024-03-27T21:41:01.247 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:41:01.248 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:41:01.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.259+0000 7f57851a4640 10 client.4664.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.259+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign -1 3 2024-03-27T21:41:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.259+0000 7f57851a4640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f57801d6990 0x7f57801d6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.259+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f57800a9260 osd=1 3 2024-03-27T21:41:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.259+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f57800a9260 osd=1 3 2024-03-27T21:41:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.259+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f57800a9260 osd=1 3 2024-03-27T21:41:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.259+0000 7f57851a4640 15 client.4664.objecter _session_command_op_remove -1 3 2024-03-27T21:41:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.259+0000 7f57851a4640 20 client.4664.objecter get_session s=0x7f57800a9260 osd=1 3 2024-03-27T21:41:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.259+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign 1 3 2024-03-27T21:41:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.259+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f57800a9260 osd=1 4 2024-03-27T21:41:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.259+0000 7f57851a4640 10 client.4664.objecter _send_command 3 2024-03-27T21:41:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.259+0000 7f57851a4640 1 -- 172.21.15.60:0/4056101675 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57801d72d0 con 0x7f57801d6990 2024-03-27T21:41:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.259+0000 7f577ed9d640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f57801d6990 0x7f57801d6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.259+0000 7f577ed9d640 10 monclient: get_auth_request con 0x7f57801d6990 auth_method 0 2024-03-27T21:41:01.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.260+0000 7f57777fe640 10 client.4664.objecter ms_handle_connect 0x7f57801d6990 2024-03-27T21:41:01.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.260+0000 7f577ed9d640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f57801d6990 0x7f57801d6d90 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:01.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.260+0000 7f57777fe640 1 -- 172.21.15.60:0/4056101675 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f57801d72d0 con 0x7f57801d6990 2024-03-27T21:41:01.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.260+0000 7f57777fe640 10 client.4664.objecter ms_dispatch 0x7f5780000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:41:01.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.260+0000 7f57777fe640 10 client.4664.objecter _finish_command 3 = system:0 2024-03-27T21:41:01.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.260+0000 7f57777fe640 20 client.4664.objecter put_session s=0x7f57800a9260 osd=1 4 2024-03-27T21:41:01.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.260+0000 7f57777fe640 15 client.4664.objecter _session_command_op_remove 1 3 2024-03-27T21:41:01.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.282+0000 7f57851a4640 10 client.4664.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:01.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.282+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign -1 4 2024-03-27T21:41:01.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.282+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f57800a9260 osd=1 3 2024-03-27T21:41:01.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.282+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f57800a9260 osd=1 3 2024-03-27T21:41:01.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.282+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f57800a9260 osd=1 3 2024-03-27T21:41:01.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.282+0000 7f57851a4640 15 client.4664.objecter _session_command_op_remove -1 4 2024-03-27T21:41:01.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.282+0000 7f57851a4640 20 client.4664.objecter get_session s=0x7f57800a9260 osd=1 3 2024-03-27T21:41:01.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.282+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign 1 4 2024-03-27T21:41:01.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.282+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f57800a9260 osd=1 4 2024-03-27T21:41:01.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.282+0000 7f57851a4640 10 client.4664.objecter _send_command 4 2024-03-27T21:41:01.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.282+0000 7f57851a4640 1 -- 172.21.15.60:0/4056101675 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f578006ff10 con 0x7f57801d6990 2024-03-27T21:41:01.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57777fe640 1 -- 172.21.15.60:0/4056101675 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f578006ff10 con 0x7f57801d6990 2024-03-27T21:41:01.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57777fe640 10 client.4664.objecter ms_dispatch 0x7f5780000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:01.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57777fe640 10 client.4664.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:01.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57777fe640 20 client.4664.objecter put_session s=0x7f57800a9260 osd=1 4 2024-03-27T21:41:01.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57777fe640 15 client.4664.objecter _session_command_op_remove 1 4 2024-03-27T21:41:01.270 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:41:01.272 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:41:01.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57851a4640 10 client.4664.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:01.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign -1 5 2024-03-27T21:41:01.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57851a4640 20 client.4664.objecter _get_session osd=-1 returning homeless 2024-03-27T21:41:01.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57851a4640 10 client.4664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:01.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57851a4640 10 monclient: _renew_subs 2024-03-27T21:41:01.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57851a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:01.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57851a4640 1 -- 172.21.15.60:0/4056101675 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=50}) v3 -- 0x7f57800700e0 con 0x7f5780071a50 2024-03-27T21:41:01.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57851a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:01.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57851a4640 1 -- 172.21.15.60:0/4056101675 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5780132eb0 con 0x7f5780071a50 2024-03-27T21:41:01.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57777fe640 1 -- 172.21.15.60:0/4056101675 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5770057e80 con 0x7f5780071a50 2024-03-27T21:41:01.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57777fe640 10 monclient: handle_get_version_reply finishing 1 version 49 2024-03-27T21:41:01.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f577dd9b640 10 client.4664.objecter _check_command_map_dne tid 5 current 49 map_dne_bound 49 2024-03-27T21:41:01.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f577dd9b640 10 client.4664.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:41:01.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f577dd9b640 15 client.4664.objecter _session_command_op_remove -1 5 2024-03-27T21:41:01.273 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:41:01.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57851a4640 10 client.4664.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:01.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign -1 6 2024-03-27T21:41:01.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57851a4640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f578002b170 0x7f578002b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:01.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f57800b62a0 osd=3 3 2024-03-27T21:41:01.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f57800b62a0 osd=3 3 2024-03-27T21:41:01.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f57800b62a0 osd=3 3 2024-03-27T21:41:01.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57851a4640 15 client.4664.objecter _session_command_op_remove -1 6 2024-03-27T21:41:01.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57851a4640 20 client.4664.objecter get_session s=0x7f57800b62a0 osd=3 3 2024-03-27T21:41:01.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign 3 6 2024-03-27T21:41:01.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f57800b62a0 osd=3 4 2024-03-27T21:41:01.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57851a4640 10 client.4664.objecter _send_command 6 2024-03-27T21:41:01.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f57851a4640 1 -- 172.21.15.60:0/4056101675 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57800b6920 con 0x7f578002b170 2024-03-27T21:41:01.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f5777fff640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f578002b170 0x7f578002b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:01.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.284+0000 7f5777fff640 10 monclient: get_auth_request con 0x7f578002b170 auth_method 0 2024-03-27T21:41:01.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.285+0000 7f57777fe640 10 client.4664.objecter ms_handle_connect 0x7f578002b170 2024-03-27T21:41:01.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.285+0000 7f5777fff640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f578002b170 0x7f578002b550 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:01.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.286+0000 7f57777fe640 1 -- 172.21.15.60:0/4056101675 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f57800b6920 con 0x7f578002b170 2024-03-27T21:41:01.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.286+0000 7f57777fe640 10 client.4664.objecter ms_dispatch 0x7f5780000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:41:01.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.286+0000 7f57777fe640 10 client.4664.objecter _finish_command 6 = system:0 2024-03-27T21:41:01.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.286+0000 7f57777fe640 20 client.4664.objecter put_session s=0x7f57800b62a0 osd=3 4 2024-03-27T21:41:01.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.286+0000 7f57777fe640 15 client.4664.objecter _session_command_op_remove 3 6 2024-03-27T21:41:01.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.304+0000 7f57851a4640 10 client.4664.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:01.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.304+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign -1 7 2024-03-27T21:41:01.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.304+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f57800b62a0 osd=3 3 2024-03-27T21:41:01.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.304+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f57800b62a0 osd=3 3 2024-03-27T21:41:01.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.304+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f57800b62a0 osd=3 3 2024-03-27T21:41:01.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.304+0000 7f57851a4640 15 client.4664.objecter _session_command_op_remove -1 7 2024-03-27T21:41:01.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.304+0000 7f57851a4640 20 client.4664.objecter get_session s=0x7f57800b62a0 osd=3 3 2024-03-27T21:41:01.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.304+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign 3 7 2024-03-27T21:41:01.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.304+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f57800b62a0 osd=3 4 2024-03-27T21:41:01.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.304+0000 7f57851a4640 10 client.4664.objecter _send_command 7 2024-03-27T21:41:01.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.304+0000 7f57851a4640 1 -- 172.21.15.60:0/4056101675 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f57800b6af0 con 0x7f578002b170 2024-03-27T21:41:01.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.307+0000 7f57777fe640 1 -- 172.21.15.60:0/4056101675 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f57800b6af0 con 0x7f578002b170 2024-03-27T21:41:01.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.307+0000 7f57777fe640 10 client.4664.objecter ms_dispatch 0x7f5780000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:01.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.307+0000 7f57777fe640 10 client.4664.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:01.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.307+0000 7f57777fe640 20 client.4664.objecter put_session s=0x7f57800b62a0 osd=3 4 2024-03-27T21:41:01.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.307+0000 7f57777fe640 15 client.4664.objecter _session_command_op_remove 3 7 2024-03-27T21:41:01.293 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:41:01.295 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:41:01.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.307+0000 7f57851a4640 10 client.4664.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:01.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.307+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign -1 8 2024-03-27T21:41:01.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.307+0000 7f57851a4640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f57801e2d00 0x7f57801e50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:01.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.307+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f57800b5f60 osd=4 3 2024-03-27T21:41:01.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.307+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f57800b5f60 osd=4 3 2024-03-27T21:41:01.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.307+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f57800b5f60 osd=4 3 2024-03-27T21:41:01.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.307+0000 7f57851a4640 15 client.4664.objecter _session_command_op_remove -1 8 2024-03-27T21:41:01.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.307+0000 7f57851a4640 20 client.4664.objecter get_session s=0x7f57800b5f60 osd=4 3 2024-03-27T21:41:01.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.307+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign 4 8 2024-03-27T21:41:01.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.307+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f57800b5f60 osd=4 4 2024-03-27T21:41:01.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.307+0000 7f57851a4640 10 client.4664.objecter _send_command 8 2024-03-27T21:41:01.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.307+0000 7f57851a4640 1 -- 172.21.15.60:0/4056101675 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57800b60d0 con 0x7f57801e2d00 2024-03-27T21:41:01.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.307+0000 7f577f59e640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f57801e2d00 0x7f57801e50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:01.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.307+0000 7f577f59e640 10 monclient: get_auth_request con 0x7f57801e2d00 auth_method 0 2024-03-27T21:41:01.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.307+0000 7f577f59e640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f57801e2d00 0x7f57801e50f0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:01.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.307+0000 7f57777fe640 10 client.4664.objecter ms_handle_connect 0x7f57801e2d00 2024-03-27T21:41:01.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.308+0000 7f57777fe640 1 -- 172.21.15.60:0/4056101675 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f57800b60d0 con 0x7f57801e2d00 2024-03-27T21:41:01.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.308+0000 7f57777fe640 10 client.4664.objecter ms_dispatch 0x7f5780000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:41:01.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.308+0000 7f57777fe640 10 client.4664.objecter _finish_command 8 = system:0 2024-03-27T21:41:01.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.308+0000 7f57777fe640 20 client.4664.objecter put_session s=0x7f57800b5f60 osd=4 4 2024-03-27T21:41:01.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.308+0000 7f57777fe640 15 client.4664.objecter _session_command_op_remove 4 8 2024-03-27T21:41:01.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.324+0000 7f57851a4640 10 client.4664.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:01.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.324+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign -1 9 2024-03-27T21:41:01.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.324+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f57800b5f60 osd=4 3 2024-03-27T21:41:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.324+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f57800b5f60 osd=4 3 2024-03-27T21:41:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.324+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f57800b5f60 osd=4 3 2024-03-27T21:41:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.324+0000 7f57851a4640 15 client.4664.objecter _session_command_op_remove -1 9 2024-03-27T21:41:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.324+0000 7f57851a4640 20 client.4664.objecter get_session s=0x7f57800b5f60 osd=4 3 2024-03-27T21:41:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.324+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign 4 9 2024-03-27T21:41:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.324+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f57800b5f60 osd=4 4 2024-03-27T21:41:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.324+0000 7f57851a4640 10 client.4664.objecter _send_command 9 2024-03-27T21:41:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.324+0000 7f57851a4640 1 -- 172.21.15.60:0/4056101675 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f578002bcd0 con 0x7f57801e2d00 2024-03-27T21:41:01.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.326+0000 7f57777fe640 1 -- 172.21.15.60:0/4056101675 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f578002bcd0 con 0x7f57801e2d00 2024-03-27T21:41:01.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.326+0000 7f57777fe640 10 client.4664.objecter ms_dispatch 0x7f5780000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:01.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.326+0000 7f57777fe640 10 client.4664.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:01.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.326+0000 7f57777fe640 20 client.4664.objecter put_session s=0x7f57800b5f60 osd=4 4 2024-03-27T21:41:01.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.326+0000 7f57777fe640 15 client.4664.objecter _session_command_op_remove 4 9 2024-03-27T21:41:01.315 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:41:01.317 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:41:01.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.326+0000 7f57851a4640 10 client.4664.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:01.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.326+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign -1 10 2024-03-27T21:41:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.326+0000 7f57851a4640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f578002be40 0x7f57801e87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.326+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f578002bcd0 osd=5 3 2024-03-27T21:41:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.326+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f578002bcd0 osd=5 3 2024-03-27T21:41:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.326+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f578002bcd0 osd=5 3 2024-03-27T21:41:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.326+0000 7f57851a4640 15 client.4664.objecter _session_command_op_remove -1 10 2024-03-27T21:41:01.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.326+0000 7f57851a4640 20 client.4664.objecter get_session s=0x7f578002bcd0 osd=5 3 2024-03-27T21:41:01.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.326+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign 5 10 2024-03-27T21:41:01.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.326+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f578002bcd0 osd=5 4 2024-03-27T21:41:01.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.326+0000 7f57851a4640 10 client.4664.objecter _send_command 10 2024-03-27T21:41:01.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.326+0000 7f57851a4640 1 -- 172.21.15.60:0/4056101675 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5780066760 con 0x7f578002be40 2024-03-27T21:41:01.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.327+0000 7f577ed9d640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f578002be40 0x7f57801e87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.327+0000 7f577ed9d640 10 monclient: get_auth_request con 0x7f578002be40 auth_method 0 2024-03-27T21:41:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.327+0000 7f57777fe640 10 client.4664.objecter ms_handle_connect 0x7f578002be40 2024-03-27T21:41:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.327+0000 7f577ed9d640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f578002be40 0x7f57801e87c0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.328+0000 7f57777fe640 1 -- 172.21.15.60:0/4056101675 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f5780066760 con 0x7f578002be40 2024-03-27T21:41:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.328+0000 7f57777fe640 10 client.4664.objecter ms_dispatch 0x7f5780000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:41:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.328+0000 7f57777fe640 10 client.4664.objecter _finish_command 10 = system:0 2024-03-27T21:41:01.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.328+0000 7f57777fe640 20 client.4664.objecter put_session s=0x7f578002bcd0 osd=5 4 2024-03-27T21:41:01.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.328+0000 7f57777fe640 15 client.4664.objecter _session_command_op_remove 5 10 2024-03-27T21:41:01.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.342+0000 7f57851a4640 10 client.4664.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:01.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.342+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign -1 11 2024-03-27T21:41:01.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.342+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f578002bcd0 osd=5 3 2024-03-27T21:41:01.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.342+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f578002bcd0 osd=5 3 2024-03-27T21:41:01.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.342+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f578002bcd0 osd=5 3 2024-03-27T21:41:01.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.342+0000 7f57851a4640 15 client.4664.objecter _session_command_op_remove -1 11 2024-03-27T21:41:01.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.342+0000 7f57851a4640 20 client.4664.objecter get_session s=0x7f578002bcd0 osd=5 3 2024-03-27T21:41:01.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.342+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign 5 11 2024-03-27T21:41:01.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.342+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f578002bcd0 osd=5 4 2024-03-27T21:41:01.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.342+0000 7f57851a4640 10 client.4664.objecter _send_command 11 2024-03-27T21:41:01.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.342+0000 7f57851a4640 1 -- 172.21.15.60:0/4056101675 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5780066760 con 0x7f578002be40 2024-03-27T21:41:01.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.344+0000 7f57777fe640 1 -- 172.21.15.60:0/4056101675 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5780066760 con 0x7f578002be40 2024-03-27T21:41:01.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.344+0000 7f57777fe640 10 client.4664.objecter ms_dispatch 0x7f5780000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:01.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.344+0000 7f57777fe640 10 client.4664.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:01.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.344+0000 7f57777fe640 20 client.4664.objecter put_session s=0x7f578002bcd0 osd=5 4 2024-03-27T21:41:01.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.344+0000 7f57777fe640 15 client.4664.objecter _session_command_op_remove 5 11 2024-03-27T21:41:01.331 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:41:01.333 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:41:01.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.344+0000 7f57851a4640 10 client.4664.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:01.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.344+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign -1 12 2024-03-27T21:41:01.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.344+0000 7f57851a4640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f57801e9210 0x7f57801eb600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:01.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.344+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f5780066760 osd=6 3 2024-03-27T21:41:01.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.344+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f5780066760 osd=6 3 2024-03-27T21:41:01.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.344+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f5780066760 osd=6 3 2024-03-27T21:41:01.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.344+0000 7f57851a4640 15 client.4664.objecter _session_command_op_remove -1 12 2024-03-27T21:41:01.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.344+0000 7f57851a4640 20 client.4664.objecter get_session s=0x7f5780066760 osd=6 3 2024-03-27T21:41:01.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.344+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign 6 12 2024-03-27T21:41:01.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.344+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f5780066760 osd=6 4 2024-03-27T21:41:01.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.344+0000 7f57851a4640 10 client.4664.objecter _send_command 12 2024-03-27T21:41:01.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.344+0000 7f57851a4640 1 -- 172.21.15.60:0/4056101675 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57800669d0 con 0x7f57801e9210 2024-03-27T21:41:01.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.345+0000 7f5777fff640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f57801e9210 0x7f57801eb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:01.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.345+0000 7f5777fff640 10 monclient: get_auth_request con 0x7f57801e9210 auth_method 0 2024-03-27T21:41:01.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.345+0000 7f5777fff640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f57801e9210 0x7f57801eb600 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:01.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.345+0000 7f57777fe640 10 client.4664.objecter ms_handle_connect 0x7f57801e9210 2024-03-27T21:41:01.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.346+0000 7f57777fe640 1 -- 172.21.15.60:0/4056101675 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f57800669d0 con 0x7f57801e9210 2024-03-27T21:41:01.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.346+0000 7f57777fe640 10 client.4664.objecter ms_dispatch 0x7f5780000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:41:01.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.346+0000 7f57777fe640 10 client.4664.objecter _finish_command 12 = system:0 2024-03-27T21:41:01.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.346+0000 7f57777fe640 20 client.4664.objecter put_session s=0x7f5780066760 osd=6 4 2024-03-27T21:41:01.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.346+0000 7f57777fe640 15 client.4664.objecter _session_command_op_remove 6 12 2024-03-27T21:41:01.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.346+0000 7fae7fd27640 10 start_mon_command cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:41:01.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.346+0000 7fae7fd27640 10 monclient: _send_command 2 [{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:41:01.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.346+0000 7fae7fd27640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:01.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.346+0000 7fae7fd27640 1 -- 172.21.15.60:0/461285645 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fae7806af90 con 0x7fae78060970 2024-03-27T21:41:01.343 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:01.357+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:01.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.360+0000 7f57851a4640 10 client.4664.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:01.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.360+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign -1 13 2024-03-27T21:41:01.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.360+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f5780066760 osd=6 3 2024-03-27T21:41:01.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.360+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f5780066760 osd=6 3 2024-03-27T21:41:01.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.360+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f5780066760 osd=6 3 2024-03-27T21:41:01.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.360+0000 7f57851a4640 15 client.4664.objecter _session_command_op_remove -1 13 2024-03-27T21:41:01.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.360+0000 7f57851a4640 20 client.4664.objecter get_session s=0x7f5780066760 osd=6 3 2024-03-27T21:41:01.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.360+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign 6 13 2024-03-27T21:41:01.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.360+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f5780066760 osd=6 4 2024-03-27T21:41:01.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.360+0000 7f57851a4640 10 client.4664.objecter _send_command 13 2024-03-27T21:41:01.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.360+0000 7f57851a4640 1 -- 172.21.15.60:0/4056101675 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5780066990 con 0x7f57801e9210 2024-03-27T21:41:01.348 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:01.348 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:01.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.362+0000 7f57777fe640 1 -- 172.21.15.60:0/4056101675 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f5780066990 con 0x7f57801e9210 2024-03-27T21:41:01.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.362+0000 7f57777fe640 10 client.4664.objecter ms_dispatch 0x7f5780000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:01.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.362+0000 7f57777fe640 10 client.4664.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:01.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.362+0000 7f57777fe640 20 client.4664.objecter put_session s=0x7f5780066760 osd=6 4 2024-03-27T21:41:01.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.362+0000 7f57777fe640 15 client.4664.objecter _session_command_op_remove 6 13 2024-03-27T21:41:01.349 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:41:01.350 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:41:01.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.362+0000 7f57851a4640 10 client.4664.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:01.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.362+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign -1 14 2024-03-27T21:41:01.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.362+0000 7f57851a4640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f57801ec050 0x7f57801ee440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:01.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.362+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f5780066990 osd=7 3 2024-03-27T21:41:01.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.362+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f5780066990 osd=7 3 2024-03-27T21:41:01.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.362+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f5780066990 osd=7 3 2024-03-27T21:41:01.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.362+0000 7f57851a4640 15 client.4664.objecter _session_command_op_remove -1 14 2024-03-27T21:41:01.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.362+0000 7f57851a4640 20 client.4664.objecter get_session s=0x7f5780066990 osd=7 3 2024-03-27T21:41:01.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.362+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign 7 14 2024-03-27T21:41:01.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.362+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f5780066990 osd=7 4 2024-03-27T21:41:01.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.362+0000 7f57851a4640 10 client.4664.objecter _send_command 14 2024-03-27T21:41:01.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.362+0000 7f57851a4640 1 -- 172.21.15.60:0/4056101675 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57801f13e0 con 0x7f57801ec050 2024-03-27T21:41:01.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.362+0000 7f577f59e640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f57801ec050 0x7f57801ee440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:01.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.362+0000 7f577f59e640 10 monclient: get_auth_request con 0x7f57801ec050 auth_method 0 2024-03-27T21:41:01.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.363+0000 7f57777fe640 10 client.4664.objecter ms_handle_connect 0x7f57801ec050 2024-03-27T21:41:01.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.363+0000 7f577f59e640 1 --2- 172.21.15.60:0/4056101675 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f57801ec050 0x7f57801ee440 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:01.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.363+0000 7f57777fe640 1 -- 172.21.15.60:0/4056101675 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f57801f13e0 con 0x7f57801ec050 2024-03-27T21:41:01.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.363+0000 7f57777fe640 10 client.4664.objecter ms_dispatch 0x7f5780000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:41:01.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.363+0000 7f57777fe640 10 client.4664.objecter _finish_command 14 = system:0 2024-03-27T21:41:01.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.363+0000 7f57777fe640 20 client.4664.objecter put_session s=0x7f5780066990 osd=7 4 2024-03-27T21:41:01.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.363+0000 7f57777fe640 15 client.4664.objecter _session_command_op_remove 7 14 2024-03-27T21:41:01.357 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:41:01.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.377+0000 7f57851a4640 10 client.4664.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:01.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.377+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign -1 15 2024-03-27T21:41:01.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.377+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f5780066990 osd=7 3 2024-03-27T21:41:01.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.377+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f5780066990 osd=7 3 2024-03-27T21:41:01.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.377+0000 7f57851a4640 20 client.4664.objecter _get_session s=0x7f5780066990 osd=7 3 2024-03-27T21:41:01.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.377+0000 7f57851a4640 15 client.4664.objecter _session_command_op_remove -1 15 2024-03-27T21:41:01.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.377+0000 7f57851a4640 20 client.4664.objecter get_session s=0x7f5780066990 osd=7 3 2024-03-27T21:41:01.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.377+0000 7f57851a4640 15 client.4664.objecter _session_command_op_assign 7 15 2024-03-27T21:41:01.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.377+0000 7f57851a4640 20 client.4664.objecter put_session s=0x7f5780066990 osd=7 4 2024-03-27T21:41:01.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.377+0000 7f57851a4640 10 client.4664.objecter _send_command 15 2024-03-27T21:41:01.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.377+0000 7f57851a4640 1 -- 172.21.15.60:0/4056101675 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f57801f01b0 con 0x7f57801ec050 2024-03-27T21:41:01.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.378+0000 7f57777fe640 1 -- 172.21.15.60:0/4056101675 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f57801f01b0 con 0x7f57801ec050 2024-03-27T21:41:01.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.378+0000 7f57777fe640 10 client.4664.objecter ms_dispatch 0x7f5780000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:01.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.378+0000 7f57777fe640 10 client.4664.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:01.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.378+0000 7f57777fe640 20 client.4664.objecter put_session s=0x7f5780066990 osd=7 4 2024-03-27T21:41:01.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.378+0000 7f57777fe640 15 client.4664.objecter _session_command_op_remove 7 15 2024-03-27T21:41:01.365 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:41:01.366 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:41:01.367 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:41:01.368 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:41:01.444 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:01.458+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:01.504 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:01.504 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:01.504 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:01.504 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-03-27T21:41:01.504 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.504 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:01.505 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:01.505 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 82.084687978000005, 2024-03-27T21:41:01.505 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43537978700000002, 2024-03-27T21:41:01.505 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:01.505 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:01.505 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:01.505 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:01.505 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:01.505 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-03-27T21:41:01.505 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.505 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:01.505 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.505 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:01.505 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:01.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:01.506 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:01.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:01.506 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432019+0000", 2024-03-27T21:41:01.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-03-27T21:41:01.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.507 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:01.507 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432027+0000", 2024-03-27T21:41:01.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1200000000000002e-06 2024-03-27T21:41:01.507 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.507 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.507 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:01.507 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432039+0000", 2024-03-27T21:41:01.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1123e-05 2024-03-27T21:41:01.507 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.507 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.507 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:01.507 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432051+0000", 2024-03-27T21:41:01.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-03-27T21:41:01.507 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:01.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432157+0000", 2024-03-27T21:41:01.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106846 2024-03-27T21:41:01.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-03-27T21:41:01.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432176+0000", 2024-03-27T21:41:01.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8490000000000001e-05 2024-03-27T21:41:01.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:01.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866170+0000", 2024-03-27T21:41:01.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43399370100000001 2024-03-27T21:41:01.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.509 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:01.509 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866335+0000", 2024-03-27T21:41:01.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016559699999999999 2024-03-27T21:41:01.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.509 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:01.509 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866495+0000", 2024-03-27T21:41:01.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015944200000000001 2024-03-27T21:41:01.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.509 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:01.509 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867012+0000", 2024-03-27T21:41:01.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0005176 2024-03-27T21:41:01.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:01.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867334+0000", 2024-03-27T21:41:01.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032158600000000001 2024-03-27T21:41:01.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.510 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:01.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867374+0000", 2024-03-27T21:41:01.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0547000000000003e-05 2024-03-27T21:41:01.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.510 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:01.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867400+0000", 2024-03-27T21:41:01.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5958000000000001e-05 2024-03-27T21:41:01.510 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.511 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:01.511 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.511 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:01.511 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:01.511 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.587305526999998, 2024-03-27T21:41:01.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064560310000000001, 2024-03-27T21:41:01.511 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:01.511 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:01.511 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:01.511 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:01.511 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:01.511 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 70 2024-03-27T21:41:01.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.512 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:01.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:01.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:01.512 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:01.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:01.512 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:01.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929402+0000", 2024-03-27T21:41:01.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:01.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.513 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:01.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929407+0000", 2024-03-27T21:41:01.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-06 2024-03-27T21:41:01.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.513 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:01.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929414+0000", 2024-03-27T21:41:01.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5689999999999999e-06 2024-03-27T21:41:01.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.513 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:01.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929419+0000", 2024-03-27T21:41:01.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-03-27T21:41:01.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.514 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:01.514 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930245+0000", 2024-03-27T21:41:01.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082521500000000002 2024-03-27T21:41:01.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.514 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:01.514 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930287+0000", 2024-03-27T21:41:01.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2305000000000002e-05 2024-03-27T21:41:01.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.514 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:01.515 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930367+0000", 2024-03-27T21:41:01.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9589e-05 2024-03-27T21:41:01.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:01.515 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.931214+0000", 2024-03-27T21:41:01.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000847827 2024-03-27T21:41:01.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:01.515 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935807+0000", 2024-03-27T21:41:01.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045923780000000003 2024-03-27T21:41:01.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.516 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:01.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935840+0000", 2024-03-27T21:41:01.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3501999999999999e-05 2024-03-27T21:41:01.516 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.516 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.516 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:01.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935859+0000", 2024-03-27T21:41:01.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8845e-05 2024-03-27T21:41:01.516 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.516 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:01.516 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.516 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.516 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.516 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:01.516 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:01.517 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.580045936000005, 2024-03-27T21:41:01.517 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071695429999999996, 2024-03-27T21:41:01.517 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:01.517 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:01.517 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:01.517 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:01.517 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:01.517 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-03-27T21:41:01.517 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.517 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:01.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.517 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:01.517 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:01.517 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.518 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:01.518 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:01.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.518 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:01.518 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936662+0000", 2024-03-27T21:41:01.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:01.518 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:01.518 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936737+0000", 2024-03-27T21:41:01.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5303999999999996e-05 2024-03-27T21:41:01.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:01.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936746+0000", 2024-03-27T21:41:01.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6340000000000007e-06 2024-03-27T21:41:01.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:01.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936753+0000", 2024-03-27T21:41:01.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-03-27T21:41:01.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:01.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937926+0000", 2024-03-27T21:41:01.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011731280000000001 2024-03-27T21:41:01.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:01.520 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937969+0000", 2024-03-27T21:41:01.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3285999999999999e-05 2024-03-27T21:41:01.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:01.520 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938040+0000", 2024-03-27T21:41:01.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0358000000000006e-05 2024-03-27T21:41:01.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:01.520 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939324+0000", 2024-03-27T21:41:01.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012844899999999999 2024-03-27T21:41:01.521 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.521 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.521 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:01.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943780+0000", 2024-03-27T21:41:01.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044555630000000001 2024-03-27T21:41:01.521 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.521 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.521 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:01.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943811+0000", 2024-03-27T21:41:01.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1415999999999999e-05 2024-03-27T21:41:01.521 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.521 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.521 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:01.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943832+0000", 2024-03-27T21:41:01.522 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0981000000000001e-05 2024-03-27T21:41:01.522 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.522 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:01.522 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.522 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.522 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.522 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:01.522 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:01.522 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.579947919999995, 2024-03-27T21:41:01.522 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071139740000000003, 2024-03-27T21:41:01.522 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:01.522 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:01.522 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:01.523 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:01.523 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:01.523 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-03-27T21:41:01.523 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.523 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:01.523 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.523 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:01.523 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:01.523 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.523 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.523 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.523 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:01.523 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:01.523 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.523 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.523 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.524 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:01.524 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936760+0000", 2024-03-27T21:41:01.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:01.524 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.524 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:01.524 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936765+0000", 2024-03-27T21:41:01.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6330000000000004e-06 2024-03-27T21:41:01.524 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.524 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:01.524 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936771+0000", 2024-03-27T21:41:01.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0560000000000001e-06 2024-03-27T21:41:01.524 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.525 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:01.525 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936776+0000", 2024-03-27T21:41:01.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1510000000000003e-06 2024-03-27T21:41:01.525 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.525 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.525 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:01.525 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938605+0000", 2024-03-27T21:41:01.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001829589 2024-03-27T21:41:01.525 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.525 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.525 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:01.525 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938657+0000", 2024-03-27T21:41:01.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1860000000000002e-05 2024-03-27T21:41:01.525 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.526 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.526 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:01.526 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938734+0000", 2024-03-27T21:41:01.526 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6639999999999998e-05 2024-03-27T21:41:01.526 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.526 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.526 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:01.526 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939562+0000", 2024-03-27T21:41:01.526 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082803100000000001 2024-03-27T21:41:01.526 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.526 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.526 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:01.526 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943838+0000", 2024-03-27T21:41:01.526 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042757050000000003 2024-03-27T21:41:01.527 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.527 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.527 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:01.527 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943859+0000", 2024-03-27T21:41:01.527 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1175e-05 2024-03-27T21:41:01.527 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.527 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.527 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:01.527 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943875+0000", 2024-03-27T21:41:01.527 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5744e-05 2024-03-27T21:41:01.527 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.527 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:01.527 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.527 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.527 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.528 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:01.528 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:01.528 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.436433868999998, 2024-03-27T21:41:01.528 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059712230000000003, 2024-03-27T21:41:01.528 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:01.528 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:01.528 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:01.528 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:01.528 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:01.528 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 112 2024-03-27T21:41:01.528 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.528 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:01.528 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.528 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:01.528 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:01.529 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.529 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:01.529 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:01.529 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.529 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:01.529 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080274+0000", 2024-03-27T21:41:01.529 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:01.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.529 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:01.529 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080318+0000", 2024-03-27T21:41:01.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3634999999999998e-05 2024-03-27T21:41:01.530 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.530 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.530 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:01.530 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080329+0000", 2024-03-27T21:41:01.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1232e-05 2024-03-27T21:41:01.530 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.530 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.530 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:01.530 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080333+0000", 2024-03-27T21:41:01.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6009999999999999e-06 2024-03-27T21:41:01.530 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.530 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.531 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:01.531 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083786+0000", 2024-03-27T21:41:01.531 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034528940000000002 2024-03-27T21:41:01.531 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.531 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.531 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:01.531 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083861+0000", 2024-03-27T21:41:01.531 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4893000000000003e-05 2024-03-27T21:41:01.531 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.531 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.531 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:01.531 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083981+0000", 2024-03-27T21:41:01.531 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012058299999999999 2024-03-27T21:41:01.531 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.531 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.531 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:01.532 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086148+0000", 2024-03-27T21:41:01.532 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021671469999999999 2024-03-27T21:41:01.532 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.532 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.532 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:01.532 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086195+0000", 2024-03-27T21:41:01.532 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6768999999999999e-05 2024-03-27T21:41:01.532 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.532 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.532 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:01.532 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086224+0000", 2024-03-27T21:41:01.532 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8758000000000001e-05 2024-03-27T21:41:01.532 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.532 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.532 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:01.533 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086246+0000", 2024-03-27T21:41:01.533 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2064000000000001e-05 2024-03-27T21:41:01.533 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.533 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:01.533 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.533 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.533 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.533 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:01.533 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:01.533 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.436370921999995, 2024-03-27T21:41:01.533 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066196320000000003, 2024-03-27T21:41:01.533 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:01.533 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:01.533 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:01.533 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:01.534 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:01.534 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113 2024-03-27T21:41:01.534 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.534 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:01.534 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.534 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:01.534 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:01.534 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.534 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.534 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.534 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:01.534 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:01.534 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.534 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.535 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.535 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:01.535 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080337+0000", 2024-03-27T21:41:01.535 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:01.535 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.535 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.535 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:01.535 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080340+0000", 2024-03-27T21:41:01.535 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6690000000000002e-06 2024-03-27T21:41:01.535 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.535 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.535 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:01.535 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080344+0000", 2024-03-27T21:41:01.535 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6550000000000002e-06 2024-03-27T21:41:01.535 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.536 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.536 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:01.536 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080348+0000", 2024-03-27T21:41:01.536 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9169999999999999e-06 2024-03-27T21:41:01.536 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.536 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.536 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:01.536 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084595+0000", 2024-03-27T21:41:01.536 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042471719999999996 2024-03-27T21:41:01.536 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.536 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.536 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:01.536 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084676+0000", 2024-03-27T21:41:01.536 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1147000000000001e-05 2024-03-27T21:41:01.537 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.537 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.537 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:01.537 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084796+0000", 2024-03-27T21:41:01.537 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012032900000000001 2024-03-27T21:41:01.537 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.537 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.537 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:01.537 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086190+0000", 2024-03-27T21:41:01.537 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013934399999999999 2024-03-27T21:41:01.537 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.537 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.537 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:01.537 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086905+0000", 2024-03-27T21:41:01.538 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071536199999999999 2024-03-27T21:41:01.538 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.538 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.538 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:01.538 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086937+0000", 2024-03-27T21:41:01.538 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2125000000000002e-05 2024-03-27T21:41:01.538 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.538 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.538 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:01.538 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086957+0000", 2024-03-27T21:41:01.538 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0157999999999999e-05 2024-03-27T21:41:01.538 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.538 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:01.538 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.539 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.539 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.539 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:01.539 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:01.539 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.436358205999994, 2024-03-27T21:41:01.539 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007405661, 2024-03-27T21:41:01.539 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:01.539 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:01.539 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:01.539 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:01.539 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:01.539 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 114 2024-03-27T21:41:01.539 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.539 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:01.539 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.540 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:01.540 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:01.540 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.540 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.540 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.540 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:01.540 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:01.540 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.540 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.540 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.540 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:01.540 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:01.540 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-03-27T21:41:01.540 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.540 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.541 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:01.541 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080353+0000", 2024-03-27T21:41:01.541 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.441e-06 2024-03-27T21:41:01.541 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.541 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.541 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:01.541 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080356+0000", 2024-03-27T21:41:01.541 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0769999999999999e-06 2024-03-27T21:41:01.541 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.541 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.541 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:01.541 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080358+0000", 2024-03-27T21:41:01.541 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3369999999999998e-06 2024-03-27T21:41:01.541 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.541 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.541 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:01.542 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085118+0000", 2024-03-27T21:41:01.542 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047603949999999997 2024-03-27T21:41:01.542 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.542 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.542 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:01.542 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085210+0000", 2024-03-27T21:41:01.542 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1983000000000003e-05 2024-03-27T21:41:01.542 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.542 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.542 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:01.542 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085332+0000", 2024-03-27T21:41:01.542 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121803 2024-03-27T21:41:01.542 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.542 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.542 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:01.543 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086292+0000", 2024-03-27T21:41:01.543 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095961400000000002 2024-03-27T21:41:01.543 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.543 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.543 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:01.543 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087710+0000", 2024-03-27T21:41:01.543 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014181459999999999 2024-03-27T21:41:01.543 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.543 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.543 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:01.543 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087736+0000", 2024-03-27T21:41:01.543 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6475e-05 2024-03-27T21:41:01.543 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.543 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.544 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:01.544 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087756+0000", 2024-03-27T21:41:01.544 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9626000000000001e-05 2024-03-27T21:41:01.544 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.544 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:01.544 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.544 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.544 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.544 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:01.544 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:01.544 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.436335771000003, 2024-03-27T21:41:01.544 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075011720000000004, 2024-03-27T21:41:01.544 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:01.544 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:01.544 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:01.544 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:01.545 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:01.545 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115 2024-03-27T21:41:01.545 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.545 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:01.545 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.545 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:01.545 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:01.545 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.545 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.545 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.545 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:01.545 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:01.545 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.545 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.545 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.546 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:01.546 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:01.546 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:01.546 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.546 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.546 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:01.546 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080458+0000", 2024-03-27T21:41:01.546 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5228000000000005e-05 2024-03-27T21:41:01.546 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.546 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.546 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:01.546 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080465+0000", 2024-03-27T21:41:01.546 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.238e-06 2024-03-27T21:41:01.546 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.547 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.547 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:01.547 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080468+0000", 2024-03-27T21:41:01.547 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5109999999999999e-06 2024-03-27T21:41:01.547 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.547 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.547 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:01.547 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086326+0000", 2024-03-27T21:41:01.547 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058579169999999998 2024-03-27T21:41:01.547 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.547 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.547 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:01.547 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086382+0000", 2024-03-27T21:41:01.547 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5309999999999997e-05 2024-03-27T21:41:01.548 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.548 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.548 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:01.548 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086457+0000", 2024-03-27T21:41:01.548 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5413000000000005e-05 2024-03-27T21:41:01.548 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.548 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.548 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:01.548 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087762+0000", 2024-03-27T21:41:01.548 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013052300000000001 2024-03-27T21:41:01.549 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.549 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.549 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:01.549 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087798+0000", 2024-03-27T21:41:01.549 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5614999999999997e-05 2024-03-27T21:41:01.549 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.549 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.549 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:01.549 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087840+0000", 2024-03-27T21:41:01.549 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2196000000000001e-05 2024-03-27T21:41:01.549 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.549 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.549 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:01.549 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087874+0000", 2024-03-27T21:41:01.550 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3785999999999998e-05 2024-03-27T21:41:01.550 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.550 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:01.550 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.550 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.550 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.550 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:01.550 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:01.550 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.436235687999996, 2024-03-27T21:41:01.550 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075050680000000002, 2024-03-27T21:41:01.550 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:01.550 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:01.550 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:01.550 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:01.551 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:01.551 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 116 2024-03-27T21:41:01.551 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.551 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:01.551 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.551 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:01.551 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:01.551 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.551 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.551 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.551 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:01.551 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:01.551 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.551 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.551 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.552 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:01.552 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080472+0000", 2024-03-27T21:41:01.552 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:01.552 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.552 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.552 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:01.552 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080475+0000", 2024-03-27T21:41:01.552 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.712e-06 2024-03-27T21:41:01.552 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.552 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.552 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:01.552 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080478+0000", 2024-03-27T21:41:01.552 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1070000000000001e-06 2024-03-27T21:41:01.552 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.552 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.553 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:01.553 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080481+0000", 2024-03-27T21:41:01.553 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4269999999999998e-06 2024-03-27T21:41:01.553 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.553 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.553 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:01.553 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086992+0000", 2024-03-27T21:41:01.553 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065114099999999996 2024-03-27T21:41:01.553 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.553 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.553 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:01.553 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087064+0000", 2024-03-27T21:41:01.553 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1927000000000004e-05 2024-03-27T21:41:01.553 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.553 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.554 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:01.554 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087188+0000", 2024-03-27T21:41:01.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123585 2024-03-27T21:41:01.554 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.554 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.554 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:01.554 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087766+0000", 2024-03-27T21:41:01.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057814300000000004 2024-03-27T21:41:01.554 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.554 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.554 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:01.554 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087934+0000", 2024-03-27T21:41:01.554 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000167894 2024-03-27T21:41:01.554 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.555 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:01.555 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087958+0000", 2024-03-27T21:41:01.555 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4663000000000001e-05 2024-03-27T21:41:01.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.555 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:01.555 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087978+0000", 2024-03-27T21:41:01.555 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9636e-05 2024-03-27T21:41:01.555 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.555 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:01.555 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.555 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.555 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:01.556 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:01.556 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.436203566000003, 2024-03-27T21:41:01.556 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076850679999999998, 2024-03-27T21:41:01.556 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:01.556 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:01.556 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:01.556 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:01.556 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:01.556 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 117 2024-03-27T21:41:01.556 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.556 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:01.556 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.556 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:01.556 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:01.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.557 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:01.557 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:01.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.557 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:01.557 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:01.557 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:01.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.557 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.557 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:01.557 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080508+0000", 2024-03-27T21:41:01.558 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.298e-06 2024-03-27T21:41:01.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:01.558 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080515+0000", 2024-03-27T21:41:01.558 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0790000000000003e-06 2024-03-27T21:41:01.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:01.558 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080518+0000", 2024-03-27T21:41:01.558 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8509999999999999e-06 2024-03-27T21:41:01.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.558 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.558 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:01.559 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087444+0000", 2024-03-27T21:41:01.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0069259120000000002 2024-03-27T21:41:01.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.559 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:01.559 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087491+0000", 2024-03-27T21:41:01.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7043999999999999e-05 2024-03-27T21:41:01.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.559 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:01.559 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087564+0000", 2024-03-27T21:41:01.559 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3603999999999995e-05 2024-03-27T21:41:01.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.559 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:01.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087902+0000", 2024-03-27T21:41:01.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033734100000000001 2024-03-27T21:41:01.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:01.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088117+0000", 2024-03-27T21:41:01.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000215164 2024-03-27T21:41:01.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.560 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:01.560 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088160+0000", 2024-03-27T21:41:01.560 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2933999999999999e-05 2024-03-27T21:41:01.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.560 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.561 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:01.561 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088190+0000", 2024-03-27T21:41:01.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0139999999999999e-05 2024-03-27T21:41:01.561 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.561 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:01.561 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.561 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.561 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:01.561 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:01.561 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.019923176999995, 2024-03-27T21:41:01.561 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052465439999999997, 2024-03-27T21:41:01.561 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:01.561 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:01.562 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:01.562 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:01.562 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:01.562 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 133 2024-03-27T21:41:01.562 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.562 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:01.562 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.562 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:01.562 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:01.562 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.562 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.562 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.562 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:01.562 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:01.563 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.563 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.563 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.563 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:01.563 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:01.563 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:01.563 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.563 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.563 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:01.563 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496788+0000", 2024-03-27T21:41:01.563 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-03-27T21:41:01.563 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.563 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.563 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:01.563 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496793+0000", 2024-03-27T21:41:01.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4790000000000004e-06 2024-03-27T21:41:01.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.564 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.564 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:01.564 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496796+0000", 2024-03-27T21:41:01.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1e-06 2024-03-27T21:41:01.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.564 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.564 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:01.564 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500447+0000", 2024-03-27T21:41:01.564 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036504459999999999 2024-03-27T21:41:01.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.564 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.564 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:01.565 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500492+0000", 2024-03-27T21:41:01.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5765e-05 2024-03-27T21:41:01.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:01.565 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500562+0000", 2024-03-27T21:41:01.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9206000000000001e-05 2024-03-27T21:41:01.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:01.565 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501959+0000", 2024-03-27T21:41:01.565 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001397187 2024-03-27T21:41:01.565 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.565 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.565 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:01.566 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501984+0000", 2024-03-27T21:41:01.566 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.552e-05 2024-03-27T21:41:01.566 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.566 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.566 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:01.566 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502010+0000", 2024-03-27T21:41:01.566 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5232000000000002e-05 2024-03-27T21:41:01.566 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.566 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.566 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:01.566 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502032+0000", 2024-03-27T21:41:01.566 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2337999999999999e-05 2024-03-27T21:41:01.566 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.566 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:01.566 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.567 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.567 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.567 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:01.567 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:01.567 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.019864518999995, 2024-03-27T21:41:01.567 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052892119999999997, 2024-03-27T21:41:01.567 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:01.567 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:01.567 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:01.567 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:01.567 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:01.567 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 134 2024-03-27T21:41:01.567 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.567 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:01.568 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.568 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:01.568 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:01.568 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.568 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.568 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.568 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:01.568 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:01.568 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.568 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.568 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.568 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:01.568 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:01.568 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:01.568 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.569 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.569 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:01.569 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496929+0000", 2024-03-27T21:41:01.569 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5098000000000004e-05 2024-03-27T21:41:01.569 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.569 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.569 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:01.569 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496937+0000", 2024-03-27T21:41:01.569 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9559999999999997e-06 2024-03-27T21:41:01.569 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.569 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.569 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:01.569 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496942+0000", 2024-03-27T21:41:01.569 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6350000000000003e-06 2024-03-27T21:41:01.570 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.570 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:01.570 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500749+0000", 2024-03-27T21:41:01.570 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038068529999999998 2024-03-27T21:41:01.570 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.570 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:01.570 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500792+0000", 2024-03-27T21:41:01.570 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2823e-05 2024-03-27T21:41:01.570 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.570 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:01.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500864+0000", 2024-03-27T21:41:01.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1715e-05 2024-03-27T21:41:01.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.571 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:01.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502068+0000", 2024-03-27T21:41:01.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012044860000000001 2024-03-27T21:41:01.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.571 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:01.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502073+0000", 2024-03-27T21:41:01.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5700000000000003e-06 2024-03-27T21:41:01.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.572 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:01.572 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502098+0000", 2024-03-27T21:41:01.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5120999999999999e-05 2024-03-27T21:41:01.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.572 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.572 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:01.572 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502133+0000", 2024-03-27T21:41:01.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5306999999999999e-05 2024-03-27T21:41:01.572 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.572 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:01.572 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.572 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.572 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:01.572 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:01.573 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.019759211999997, 2024-03-27T21:41:01.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062529040000000001, 2024-03-27T21:41:01.573 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:01.573 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:01.573 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:01.573 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:01.573 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:01.573 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135 2024-03-27T21:41:01.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.573 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:01.573 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.573 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:01.573 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:01.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:01.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:01.574 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.574 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:01.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:01.574 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:01.574 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.574 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:01.574 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496953+0000", 2024-03-27T21:41:01.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0709999999999996e-06 2024-03-27T21:41:01.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.575 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:01.575 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496961+0000", 2024-03-27T21:41:01.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1300000000000001e-06 2024-03-27T21:41:01.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.575 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:01.575 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496968+0000", 2024-03-27T21:41:01.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0999999999999998e-06 2024-03-27T21:41:01.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.575 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:01.576 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501244+0000", 2024-03-27T21:41:01.576 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042756310000000002 2024-03-27T21:41:01.576 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.576 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.576 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:01.576 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501377+0000", 2024-03-27T21:41:01.576 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013348799999999999 2024-03-27T21:41:01.576 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.576 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.576 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:01.576 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501455+0000", 2024-03-27T21:41:01.576 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7410000000000006e-05 2024-03-27T21:41:01.576 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.577 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:01.577 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502138+0000", 2024-03-27T21:41:01.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068386599999999999 2024-03-27T21:41:01.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.577 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:01.577 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503152+0000", 2024-03-27T21:41:01.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010139680000000001 2024-03-27T21:41:01.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.577 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:01.577 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503180+0000", 2024-03-27T21:41:01.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7426000000000001e-05 2024-03-27T21:41:01.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.578 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:01.578 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503202+0000", 2024-03-27T21:41:01.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2463000000000002e-05 2024-03-27T21:41:01.578 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.578 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:01.578 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.578 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:01.578 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:01.578 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.019735073999996, 2024-03-27T21:41:01.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066253680000000004, 2024-03-27T21:41:01.578 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:01.579 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:01.579 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:01.579 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:01.579 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:01.579 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 136 2024-03-27T21:41:01.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.579 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:01.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:01.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:01.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:01.580 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:01.580 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.580 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:01.580 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:01.580 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:01.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.580 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:01.580 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496977+0000", 2024-03-27T21:41:01.580 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5049999999999999e-06 2024-03-27T21:41:01.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.581 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:01.581 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496984+0000", 2024-03-27T21:41:01.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1520000000000003e-06 2024-03-27T21:41:01.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.581 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.581 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:01.581 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496990+0000", 2024-03-27T21:41:01.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-03-27T21:41:01.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.581 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.581 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:01.581 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501669+0000", 2024-03-27T21:41:01.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046793859999999998 2024-03-27T21:41:01.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.582 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.582 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:01.582 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501714+0000", 2024-03-27T21:41:01.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5154999999999997e-05 2024-03-27T21:41:01.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.582 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.582 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:01.582 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501783+0000", 2024-03-27T21:41:01.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8454999999999994e-05 2024-03-27T21:41:01.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.582 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.582 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:01.582 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503226+0000", 2024-03-27T21:41:01.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001443441 2024-03-27T21:41:01.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.583 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:01.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503540+0000", 2024-03-27T21:41:01.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031347100000000001 2024-03-27T21:41:01.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.583 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:01.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503563+0000", 2024-03-27T21:41:01.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3223000000000001e-05 2024-03-27T21:41:01.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.583 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:01.584 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503599+0000", 2024-03-27T21:41:01.584 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6105000000000001e-05 2024-03-27T21:41:01.584 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.584 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:01.584 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.584 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:01.584 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:01.584 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.019712264000006, 2024-03-27T21:41:01.584 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074261320000000002, 2024-03-27T21:41:01.584 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:01.584 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:01.584 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:01.584 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:01.585 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:01.585 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 137 2024-03-27T21:41:01.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.585 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:01.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.585 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:01.585 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:01.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.585 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:01.585 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:01.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.586 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.586 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:01.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:01.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:01.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.586 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.586 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:01.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497077+0000", 2024-03-27T21:41:01.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1599000000000003e-05 2024-03-27T21:41:01.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.586 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.586 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:01.586 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497085+0000", 2024-03-27T21:41:01.586 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6109999999999996e-06 2024-03-27T21:41:01.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.587 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:01.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497092+0000", 2024-03-27T21:41:01.587 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.742e-06 2024-03-27T21:41:01.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.587 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:01.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502159+0000", 2024-03-27T21:41:01.587 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050666870000000003 2024-03-27T21:41:01.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.587 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.587 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:01.587 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502200+0000", 2024-03-27T21:41:01.587 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1919999999999998e-05 2024-03-27T21:41:01.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.588 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.588 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:01.588 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502280+0000", 2024-03-27T21:41:01.588 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9417000000000002e-05 2024-03-27T21:41:01.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.588 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.588 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:01.588 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503605+0000", 2024-03-27T21:41:01.588 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001325641 2024-03-27T21:41:01.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.588 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.588 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:01.588 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504361+0000", 2024-03-27T21:41:01.589 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075555399999999997 2024-03-27T21:41:01.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.589 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.589 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:01.589 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504386+0000", 2024-03-27T21:41:01.589 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5369999999999999e-05 2024-03-27T21:41:01.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.589 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.589 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:01.589 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504422+0000", 2024-03-27T21:41:01.589 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6019000000000002e-05 2024-03-27T21:41:01.589 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.589 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:01.589 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.590 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.590 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.590 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:01.590 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:01.590 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.019608466999998, 2024-03-27T21:41:01.590 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074059750000000004, 2024-03-27T21:41:01.590 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:01.590 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:01.590 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:01.590 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:01.590 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:01.590 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-03-27T21:41:01.590 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.590 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:01.591 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:01.591 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:01.591 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.591 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:01.591 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:01.591 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.591 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.591 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:01.591 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:01.591 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:01.592 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.592 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.592 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:01.592 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497103+0000", 2024-03-27T21:41:01.592 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.856e-06 2024-03-27T21:41:01.592 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.592 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.592 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:01.592 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497121+0000", 2024-03-27T21:41:01.592 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7190999999999999e-05 2024-03-27T21:41:01.592 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.592 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.592 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:01.592 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497126+0000", 2024-03-27T21:41:01.592 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3770000000000004e-06 2024-03-27T21:41:01.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.593 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.593 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:01.593 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502833+0000", 2024-03-27T21:41:01.593 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057073510000000003 2024-03-27T21:41:01.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.593 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.593 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:01.593 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502881+0000", 2024-03-27T21:41:01.593 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7961999999999999e-05 2024-03-27T21:41:01.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.593 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.593 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:01.593 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502949+0000", 2024-03-27T21:41:01.593 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7957e-05 2024-03-27T21:41:01.594 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.594 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.594 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:01.594 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504430+0000", 2024-03-27T21:41:01.594 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001480763 2024-03-27T21:41:01.594 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.594 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.594 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:01.594 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504459+0000", 2024-03-27T21:41:01.594 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8983999999999999e-05 2024-03-27T21:41:01.594 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.594 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.594 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:01.595 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504486+0000", 2024-03-27T21:41:01.595 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6883999999999999e-05 2024-03-27T21:41:01.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.595 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.595 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:01.595 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504506+0000", 2024-03-27T21:41:01.595 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0214e-05 2024-03-27T21:41:01.595 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.595 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:01.595 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.595 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.595 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:01.595 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:01.596 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.019576633, 2024-03-27T21:41:01.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074754859999999999, 2024-03-27T21:41:01.596 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:01.596 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:01.596 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:01.596 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:01.596 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:01.596 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-03-27T21:41:01.596 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.596 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:01.596 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.596 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:01.596 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:01.596 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.597 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.597 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:01.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:01.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.597 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.597 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:01.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:01.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:01.597 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.597 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.597 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:01.597 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497137+0000", 2024-03-27T21:41:01.597 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0730000000000004e-06 2024-03-27T21:41:01.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:01.598 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497143+0000", 2024-03-27T21:41:01.598 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3759999999999996e-06 2024-03-27T21:41:01.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:01.598 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497150+0000", 2024-03-27T21:41:01.598 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.002e-06 2024-03-27T21:41:01.598 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.598 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.598 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:01.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503249+0000", 2024-03-27T21:41:01.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060989750000000004 2024-03-27T21:41:01.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.599 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.599 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:01.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503291+0000", 2024-03-27T21:41:01.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1563999999999997e-05 2024-03-27T21:41:01.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.599 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.599 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:01.599 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503376+0000", 2024-03-27T21:41:01.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5752000000000002e-05 2024-03-27T21:41:01.599 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.599 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.600 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:01.600 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504545+0000", 2024-03-27T21:41:01.600 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169172 2024-03-27T21:41:01.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.600 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.600 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:01.600 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504550+0000", 2024-03-27T21:41:01.600 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7419999999999997e-06 2024-03-27T21:41:01.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.600 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.600 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:01.600 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504574+0000", 2024-03-27T21:41:01.600 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4124000000000001e-05 2024-03-27T21:41:01.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:01.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504607+0000", 2024-03-27T21:41:01.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3118999999999999e-05 2024-03-27T21:41:01.601 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.601 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:01.601 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.601 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:01.601 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:01.601 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.019552966999996, 2024-03-27T21:41:01.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078150589999999992, 2024-03-27T21:41:01.601 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:01.602 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:01.602 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:01.602 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:01.602 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:01.602 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-03-27T21:41:01.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.602 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:01.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:01.602 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:01.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:01.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:01.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:01.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497155+0000", 2024-03-27T21:41:01.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:01.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:01.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497230+0000", 2024-03-27T21:41:01.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4789000000000005e-05 2024-03-27T21:41:01.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:01.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497241+0000", 2024-03-27T21:41:01.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1442e-05 2024-03-27T21:41:01.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:01.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497255+0000", 2024-03-27T21:41:01.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3087e-05 2024-03-27T21:41:01.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:01.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503630+0000", 2024-03-27T21:41:01.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063755269999999998 2024-03-27T21:41:01.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:01.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503673+0000", 2024-03-27T21:41:01.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2616e-05 2024-03-27T21:41:01.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:01.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503738+0000", 2024-03-27T21:41:01.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5114e-05 2024-03-27T21:41:01.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:01.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504907+0000", 2024-03-27T21:41:01.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169432 2024-03-27T21:41:01.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.606 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:01.606 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504926+0000", 2024-03-27T21:41:01.606 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9086999999999999e-05 2024-03-27T21:41:01.606 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.606 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.606 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:01.606 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504950+0000", 2024-03-27T21:41:01.606 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3598999999999999e-05 2024-03-27T21:41:01.606 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.606 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.606 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:01.606 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504971+0000", 2024-03-27T21:41:01.606 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0772999999999999e-05 2024-03-27T21:41:01.607 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.607 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:01.607 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.607 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.607 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.607 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:01.607 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:01.607 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 81.019445927000007, 2024-03-27T21:41:01.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082962590000000003, 2024-03-27T21:41:01.607 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:01.607 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:01.607 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:01.607 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:01.607 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:01.608 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-03-27T21:41:01.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.608 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:01.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:01.608 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:01.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:01.608 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:01.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:01.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497262+0000", 2024-03-27T21:41:01.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:01.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:01.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497267+0000", 2024-03-27T21:41:01.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.955e-06 2024-03-27T21:41:01.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:01.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497275+0000", 2024-03-27T21:41:01.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.824e-06 2024-03-27T21:41:01.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:01.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497280+0000", 2024-03-27T21:41:01.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8230000000000002e-06 2024-03-27T21:41:01.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.610 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:01.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504628+0000", 2024-03-27T21:41:01.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007348561 2024-03-27T21:41:01.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.610 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:01.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504673+0000", 2024-03-27T21:41:01.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4313000000000001e-05 2024-03-27T21:41:01.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.611 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:01.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504745+0000", 2024-03-27T21:41:01.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2303000000000002e-05 2024-03-27T21:41:01.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.611 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:01.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505358+0000", 2024-03-27T21:41:01.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061281300000000003 2024-03-27T21:41:01.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.611 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:01.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505503+0000", 2024-03-27T21:41:01.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000145134 2024-03-27T21:41:01.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:01.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505526+0000", 2024-03-27T21:41:01.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3312e-05 2024-03-27T21:41:01.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:01.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505559+0000", 2024-03-27T21:41:01.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2821999999999998e-05 2024-03-27T21:41:01.612 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.612 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:01.612 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.613 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:01.613 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:01.613 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 80.410223688000002, 2024-03-27T21:41:01.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053753339999999998, 2024-03-27T21:41:01.613 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:01.613 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:01.613 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:01.613 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:01.613 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:01.613 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159 2024-03-27T21:41:01.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.613 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:01.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.613 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:01.614 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:01.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.614 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.614 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:01.614 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:01.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:01.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.614 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.614 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:01.614 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:01.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:01.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.614 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.615 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:01.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106488+0000", 2024-03-27T21:41:01.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9830000000000001e-06 2024-03-27T21:41:01.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.615 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:01.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106491+0000", 2024-03-27T21:41:01.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3459999999999998e-06 2024-03-27T21:41:01.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.615 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:01.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106493+0000", 2024-03-27T21:41:01.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.509e-06 2024-03-27T21:41:01.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:01.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110636+0000", 2024-03-27T21:41:01.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041427260000000002 2024-03-27T21:41:01.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:01.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110683+0000", 2024-03-27T21:41:01.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6557000000000002e-05 2024-03-27T21:41:01.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:01.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110755+0000", 2024-03-27T21:41:01.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2119000000000005e-05 2024-03-27T21:41:01.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:01.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111415+0000", 2024-03-27T21:41:01.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066065199999999996 2024-03-27T21:41:01.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:01.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111793+0000", 2024-03-27T21:41:01.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037732100000000001 2024-03-27T21:41:01.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:01.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111829+0000", 2024-03-27T21:41:01.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418999999999998e-05 2024-03-27T21:41:01.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:01.618 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:01.618 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:01.618 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111860+0000", 2024-03-27T21:41:01.618 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1003000000000001e-05 2024-03-27T21:41:01.618 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.618 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:01.618 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.618 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:01.618 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:01.618 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:01.620 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:41:01.620 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:01.633+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:01.721 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:01.735+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:01.731 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:01.732 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:01.732 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:01.732 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:01.741 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:41:01.821 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:01.835+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:01.850 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:01.851 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:01.851 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:41:01.851 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:41:01.851 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:01.859 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:41:01.921 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:01.936+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:01.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.957+0000 7fae6e7fc640 1 -- 172.21.15.60:0/461285645 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v50) v1 ==== 101+0+0 (secure 0 0 0) 0x7fae74056ff0 con 0x7fae78060970 2024-03-27T21:41:01.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.957+0000 7fae6e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:41:01.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.957+0000 7fae6e7fc640 10 monclient: _finish_command 2 = system:0 nodeep-scrub is unset 2024-03-27T21:41:01.943 INFO:teuthology.orchestra.run.smithi060.stderr:nodeep-scrub is unset 2024-03-27T21:41:01.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.958+0000 7fae7fd27640 10 librados: watch_flush enter 2024-03-27T21:41:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.959+0000 7fae7fd27640 10 librados: watch_flush exit 2024-03-27T21:41:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.959+0000 7fae7fd27640 20 client.4759.objecter shutdown clearing up homeless session... 2024-03-27T21:41:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.959+0000 7fae7fd27640 10 client.4759.objecter successfully canceled tick 2024-03-27T21:41:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.959+0000 7fae7fd27640 1 -- 172.21.15.60:0/461285645 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fae5c06c590 msgr2=0x7fae5c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.959+0000 7fae7fd27640 1 --2- 172.21.15.60:0/461285645 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fae5c06c590 0x7fae5c06ea50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fae780ae3d0 tx=0x7fae68008040 comp rx=0 tx=0).stop 2024-03-27T21:41:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.959+0000 7fae7fd27640 10 monclient: shutdown 2024-03-27T21:41:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.959+0000 7fae7fd27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.959+0000 7fae7fd27640 1 -- 172.21.15.60:0/461285645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae78060970 msgr2=0x7fae7811f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.959+0000 7fae7fd27640 1 --2- 172.21.15.60:0/461285645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae78060970 0x7fae7811f600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fae7400c390 tx=0x7fae7400c850 comp rx=0 tx=0).stop 2024-03-27T21:41:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.959+0000 7fae7fd27640 1 -- 172.21.15.60:0/461285645 shutdown_connections 2024-03-27T21:41:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.959+0000 7fae7fd27640 1 --2- 172.21.15.60:0/461285645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae7813e790 0x7fae781d9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.959+0000 7fae7fd27640 1 --2- 172.21.15.60:0/461285645 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fae780aefd0 0x7fae78120440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.959+0000 7fae7fd27640 1 --2- 172.21.15.60:0/461285645 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fae5c06c590 0x7fae5c06ea50 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.959+0000 7fae7fd27640 1 --2- 172.21.15.60:0/461285645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae78060970 0x7fae7811f600 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.959+0000 7fae7fd27640 1 -- 172.21.15.60:0/461285645 >> 172.21.15.60:0/461285645 conn(0x7fae780b4850 msgr2=0x7fae780ad0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.959+0000 7fae7fd27640 1 -- 172.21.15.60:0/461285645 shutdown_connections 2024-03-27T21:41:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.959+0000 7fae7fd27640 1 -- 172.21.15.60:0/461285645 wait complete. 2024-03-27T21:41:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:01.959+0000 7fae7fd27640 1 librados: shutdown 2024-03-27T21:41:02.016 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:02.016 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:41:02.016 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:41:02.016 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-03-27T21:41:02.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.016 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:20 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2024-03-27T21:41:02.016 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:41:02.016 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 82.140049360999996, 2024-03-27T21:41:02.016 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015345630000000001, 2024-03-27T21:41:02.017 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:02.017 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:02.017 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:02.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.017 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:02.017 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:41:02.017 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.017 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:02.017 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:41:02.017 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.017 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:02.018 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888562+0000", 2024-03-27T21:41:02.018 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:02.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.018 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:02.018 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888735+0000", 2024-03-27T21:41:02.018 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017254799999999999 2024-03-27T21:41:02.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.018 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:02.018 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888747+0000", 2024-03-27T21:41:02.018 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2469999999999999e-05 2024-03-27T21:41:02.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.019 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:02.019 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888761+0000", 2024-03-27T21:41:02.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3254e-05 2024-03-27T21:41:02.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.019 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:02.019 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888804+0000", 2024-03-27T21:41:02.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3129e-05 2024-03-27T21:41:02.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.019 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:02.019 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888820+0000", 2024-03-27T21:41:02.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6426000000000002e-05 2024-03-27T21:41:02.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:02.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890060+0000", 2024-03-27T21:41:02.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012398050000000001 2024-03-27T21:41:02.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:02.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890098+0000", 2024-03-27T21:41:02.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7795e-05 2024-03-27T21:41:02.020 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.020 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:02.020 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.021 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2024-03-27T21:41:02.021 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:41:02.021 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 82.139224960999996, 2024-03-27T21:41:02.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001069555, 2024-03-27T21:41:02.021 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:02.021 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:02.021 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:02.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:02.021 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:41:02.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:02.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:41:02.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:02.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889387+0000", 2024-03-27T21:41:02.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:02.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:02.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889455+0000", 2024-03-27T21:41:02.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7930999999999997e-05 2024-03-27T21:41:02.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.023 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:02.023 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889470+0000", 2024-03-27T21:41:02.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5554e-05 2024-03-27T21:41:02.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.023 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:02.023 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889476+0000", 2024-03-27T21:41:02.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0619999999999997e-06 2024-03-27T21:41:02.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.023 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:02.023 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889513+0000", 2024-03-27T21:41:02.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6606999999999997e-05 2024-03-27T21:41:02.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.024 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:02.024 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889528+0000", 2024-03-27T21:41:02.024 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4982e-05 2024-03-27T21:41:02.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.024 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:02.024 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890421+0000", 2024-03-27T21:41:02.024 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00089293700000000001 2024-03-27T21:41:02.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.025 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:02.025 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890457+0000", 2024-03-27T21:41:02.025 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6260000000000002e-05 2024-03-27T21:41:02.025 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.025 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:02.025 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.025 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:38 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'24, mlcod=17'22)", 2024-03-27T21:41:02.025 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:41:02.025 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 82.121988994999995, 2024-03-27T21:41:02.025 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016182080000000001, 2024-03-27T21:41:02.025 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:02.025 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:02.025 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:02.026 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.026 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:02.026 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:41:02.026 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.026 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.026 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:02.026 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:41:02.026 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.026 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.026 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:02.026 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906623+0000", 2024-03-27T21:41:02.026 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:02.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.027 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:02.027 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906687+0000", 2024-03-27T21:41:02.027 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3706000000000003e-05 2024-03-27T21:41:02.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.027 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:02.027 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906698+0000", 2024-03-27T21:41:02.027 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1752e-05 2024-03-27T21:41:02.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.027 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:02.027 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906704+0000", 2024-03-27T21:41:02.027 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3940000000000004e-06 2024-03-27T21:41:02.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:02.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906741+0000", 2024-03-27T21:41:02.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6944999999999999e-05 2024-03-27T21:41:02.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:02.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906760+0000", 2024-03-27T21:41:02.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9293e-05 2024-03-27T21:41:02.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:02.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908211+0000", 2024-03-27T21:41:02.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014511350000000001 2024-03-27T21:41:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908242+0000", 2024-03-27T21:41:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0712e-05 2024-03-27T21:41:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:39 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'25, mlcod=17'22)", 2024-03-27T21:41:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 82.121144079999993, 2024-03-27T21:41:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012060199999999999, 2024-03-27T21:41:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:02.030 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:02.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.030 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:02.030 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:02.030 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.030 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:02.030 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:02.030 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.030 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:02.030 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907533+0000", 2024-03-27T21:41:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5675999999999999e-05 2024-03-27T21:41:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907545+0000", 2024-03-27T21:41:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.115e-05 2024-03-27T21:41:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907550+0000", 2024-03-27T21:41:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8320000000000002e-06 2024-03-27T21:41:02.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:02.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907592+0000", 2024-03-27T21:41:02.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1309e-05 2024-03-27T21:41:02.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:02.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907611+0000", 2024-03-27T21:41:02.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9009e-05 2024-03-27T21:41:02.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:02.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908638+0000", 2024-03-27T21:41:02.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001027533 2024-03-27T21:41:02.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.033 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:02.033 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908675+0000", 2024-03-27T21:41:02.033 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6329999999999999e-05 2024-03-27T21:41:02.033 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.033 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:02.033 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.033 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:57 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'35, mlcod=17'33)", 2024-03-27T21:41:02.033 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:41:02.033 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 82.107059648000003, 2024-03-27T21:41:02.033 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001346578, 2024-03-27T21:41:02.033 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:02.034 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:02.034 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:02.034 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.034 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:02.034 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:41:02.034 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.034 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.034 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.034 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:02.034 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:41:02.034 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.034 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.034 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.034 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:02.034 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921552+0000", 2024-03-27T21:41:02.035 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:02.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.035 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:02.035 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921615+0000", 2024-03-27T21:41:02.035 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2787999999999996e-05 2024-03-27T21:41:02.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.035 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:02.035 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921626+0000", 2024-03-27T21:41:02.035 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0888e-05 2024-03-27T21:41:02.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.036 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:02.036 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921635+0000", 2024-03-27T21:41:02.036 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4639999999999995e-06 2024-03-27T21:41:02.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.036 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:02.036 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921673+0000", 2024-03-27T21:41:02.036 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7891e-05 2024-03-27T21:41:02.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.036 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:02.036 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921694+0000", 2024-03-27T21:41:02.036 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0426000000000001e-05 2024-03-27T21:41:02.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.037 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:02.037 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922865+0000", 2024-03-27T21:41:02.037 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011709509999999999 2024-03-27T21:41:02.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.037 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:02.037 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922900+0000", 2024-03-27T21:41:02.037 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4819e-05 2024-03-27T21:41:02.037 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.037 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:02.037 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.037 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:71 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'45, mlcod=17'43)", 2024-03-27T21:41:02.038 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:41:02.038 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 82.091760936, 2024-03-27T21:41:02.038 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001460907, 2024-03-27T21:41:02.038 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:02.038 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:02.038 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:02.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.038 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:02.038 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:41:02.038 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.038 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:02.038 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:41:02.039 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.039 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.039 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:02.039 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936851+0000", 2024-03-27T21:41:02.039 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:02.039 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.039 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:02.039 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936903+0000", 2024-03-27T21:41:02.039 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2302999999999997e-05 2024-03-27T21:41:02.039 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.039 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:02.040 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936915+0000", 2024-03-27T21:41:02.040 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1967000000000001e-05 2024-03-27T21:41:02.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.040 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.040 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:02.040 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936923+0000", 2024-03-27T21:41:02.040 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9370000000000004e-06 2024-03-27T21:41:02.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.040 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.040 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:02.040 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936970+0000", 2024-03-27T21:41:02.040 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6987000000000003e-05 2024-03-27T21:41:02.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.040 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.040 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:02.041 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937002+0000", 2024-03-27T21:41:02.041 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2410999999999999e-05 2024-03-27T21:41:02.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.041 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.041 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:02.041 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938279+0000", 2024-03-27T21:41:02.041 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001276552 2024-03-27T21:41:02.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.041 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.041 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:02.041 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938313+0000", 2024-03-27T21:41:02.041 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3526999999999999e-05 2024-03-27T21:41:02.041 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.041 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:02.041 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.042 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.042 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.042 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:72 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'46, mlcod=17'43)", 2024-03-27T21:41:02.042 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:41:02.042 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 82.091123374000006, 2024-03-27T21:41:02.042 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014239490000000001, 2024-03-27T21:41:02.042 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:02.042 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:02.042 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:02.042 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.042 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:02.042 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:41:02.042 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.042 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.043 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.043 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:02.043 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:41:02.043 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.043 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.043 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.043 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:02.043 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937488+0000", 2024-03-27T21:41:02.043 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:02.043 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.043 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.043 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:02.043 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937547+0000", 2024-03-27T21:41:02.044 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8196999999999999e-05 2024-03-27T21:41:02.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.044 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.044 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:02.044 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937559+0000", 2024-03-27T21:41:02.044 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1928e-05 2024-03-27T21:41:02.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.044 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.044 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:02.044 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937564+0000", 2024-03-27T21:41:02.044 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4759999999999997e-06 2024-03-27T21:41:02.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.044 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.044 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:02.044 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937621+0000", 2024-03-27T21:41:02.044 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6712000000000002e-05 2024-03-27T21:41:02.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.045 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.045 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:02.045 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937640+0000", 2024-03-27T21:41:02.045 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9517e-05 2024-03-27T21:41:02.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.045 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.045 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:02.045 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938883+0000", 2024-03-27T21:41:02.045 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001242567 2024-03-27T21:41:02.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.045 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.045 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:02.045 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938913+0000", 2024-03-27T21:41:02.045 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0287e-05 2024-03-27T21:41:02.045 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.046 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:02.046 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.046 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.046 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.046 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:73 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'47, mlcod=17'43)", 2024-03-27T21:41:02.046 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:02.046 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 82.090447280000006, 2024-03-27T21:41:02.046 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010746239999999999, 2024-03-27T21:41:02.046 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:02.046 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:02.046 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:02.046 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.046 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:02.046 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:02.047 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.047 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.047 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:02.047 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:02.047 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.047 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.047 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:02.047 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:02.047 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:02.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.047 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.047 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:02.047 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938217+0000", 2024-03-27T21:41:02.047 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2743e-05 2024-03-27T21:41:02.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.048 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:02.048 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938229+0000", 2024-03-27T21:41:02.048 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1956999999999999e-05 2024-03-27T21:41:02.048 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.048 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:02.048 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938235+0000", 2024-03-27T21:41:02.048 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4589999999999997e-06 2024-03-27T21:41:02.048 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.048 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:02.048 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938341+0000", 2024-03-27T21:41:02.048 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000106054 2024-03-27T21:41:02.048 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.049 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:02.049 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938369+0000", 2024-03-27T21:41:02.049 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8291000000000001e-05 2024-03-27T21:41:02.049 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.049 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.049 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:02.049 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939209+0000", 2024-03-27T21:41:02.049 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084045400000000003 2024-03-27T21:41:02.049 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.049 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.049 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:02.049 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939240+0000", 2024-03-27T21:41:02.049 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0429999999999998e-05 2024-03-27T21:41:02.049 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.049 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:02.050 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.050 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.050 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:75 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'49, mlcod=17'47)", 2024-03-27T21:41:02.050 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:41:02.050 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 82.084009632999994, 2024-03-27T21:41:02.050 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001017001, 2024-03-27T21:41:02.050 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:02.050 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:02.050 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:02.050 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.050 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:02.050 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:41:02.051 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.051 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:02.051 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:41:02.051 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.051 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:02.051 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944601+0000", 2024-03-27T21:41:02.051 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999986 2024-03-27T21:41:02.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.051 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:02.051 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944614+0000", 2024-03-27T21:41:02.051 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2496e-05 2024-03-27T21:41:02.052 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.052 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:02.052 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944640+0000", 2024-03-27T21:41:02.052 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5865999999999999e-05 2024-03-27T21:41:02.052 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.052 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:02.052 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944651+0000", 2024-03-27T21:41:02.052 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1797000000000001e-05 2024-03-27T21:41:02.052 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.052 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:02.052 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944762+0000", 2024-03-27T21:41:02.052 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110412 2024-03-27T21:41:02.052 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.053 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:02.053 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944820+0000", 2024-03-27T21:41:02.053 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7649999999999999e-05 2024-03-27T21:41:02.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.053 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:02.053 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945585+0000", 2024-03-27T21:41:02.053 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076562600000000005 2024-03-27T21:41:02.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.053 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:02.053 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945620+0000", 2024-03-27T21:41:02.053 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4764000000000001e-05 2024-03-27T21:41:02.053 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.054 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:02.054 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.054 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.054 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:86 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'50)", 2024-03-27T21:41:02.054 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:02.054 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 82.076248145999998, 2024-03-27T21:41:02.054 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011392380000000001, 2024-03-27T21:41:02.054 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:02.054 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:02.054 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:02.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.054 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:02.054 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:02.055 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.055 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:02.055 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:02.055 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.055 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:02.055 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:02.055 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:02.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.055 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:02.056 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952373+0000", 2024-03-27T21:41:02.056 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9220000000000003e-06 2024-03-27T21:41:02.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.056 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:02.056 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952381+0000", 2024-03-27T21:41:02.056 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3070000000000003e-06 2024-03-27T21:41:02.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.056 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:02.056 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952385+0000", 2024-03-27T21:41:02.056 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.121e-06 2024-03-27T21:41:02.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.057 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:02.057 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952419+0000", 2024-03-27T21:41:02.057 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4082999999999999e-05 2024-03-27T21:41:02.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.057 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:02.057 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952440+0000", 2024-03-27T21:41:02.057 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0837999999999999e-05 2024-03-27T21:41:02.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.057 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:02.057 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953472+0000", 2024-03-27T21:41:02.057 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001032119 2024-03-27T21:41:02.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.057 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:02.058 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953504+0000", 2024-03-27T21:41:02.058 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1250000000000001e-05 2024-03-27T21:41:02.058 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.058 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:02.058 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.058 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.058 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:87 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-27T21:41:02.058 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:41:02.058 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 82.075880889000004, 2024-03-27T21:41:02.058 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00135519, 2024-03-27T21:41:02.058 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:02.058 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:02.058 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:02.058 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.059 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:02.059 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:41:02.059 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.059 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.059 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:02.059 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:41:02.059 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.059 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.059 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:02.059 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952731+0000", 2024-03-27T21:41:02.059 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:02.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.059 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.059 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:02.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952781+0000", 2024-03-27T21:41:02.060 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0161000000000003e-05 2024-03-27T21:41:02.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.060 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:02.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952794+0000", 2024-03-27T21:41:02.060 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2853e-05 2024-03-27T21:41:02.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.060 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:02.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952799+0000", 2024-03-27T21:41:02.060 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5560000000000001e-06 2024-03-27T21:41:02.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:02.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952832+0000", 2024-03-27T21:41:02.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3834999999999997e-05 2024-03-27T21:41:02.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:02.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952851+0000", 2024-03-27T21:41:02.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8289999999999999e-05 2024-03-27T21:41:02.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:02.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954055+0000", 2024-03-27T21:41:02.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012037949999999999 2024-03-27T21:41:02.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.062 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.062 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:02.062 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954087+0000", 2024-03-27T21:41:02.062 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2292999999999997e-05 2024-03-27T21:41:02.062 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.062 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:02.062 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.062 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.062 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:89 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'59, mlcod=17'57)", 2024-03-27T21:41:02.062 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:41:02.062 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 82.073606667999996, 2024-03-27T21:41:02.062 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00138893, 2024-03-27T21:41:02.062 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:02.062 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:02.063 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:02.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.063 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:02.063 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:41:02.063 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.063 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:02.063 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:41:02.063 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.063 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:02.063 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955005+0000", 2024-03-27T21:41:02.064 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:02.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.064 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.064 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:02.064 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955071+0000", 2024-03-27T21:41:02.064 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5290999999999995e-05 2024-03-27T21:41:02.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.064 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.064 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:02.064 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955081+0000", 2024-03-27T21:41:02.064 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0572e-05 2024-03-27T21:41:02.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.064 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.064 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:02.064 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955086+0000", 2024-03-27T21:41:02.065 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8380000000000001e-06 2024-03-27T21:41:02.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.065 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.065 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:02.065 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955122+0000", 2024-03-27T21:41:02.065 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5698000000000003e-05 2024-03-27T21:41:02.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.065 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.065 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:02.065 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955140+0000", 2024-03-27T21:41:02.065 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7900000000000001e-05 2024-03-27T21:41:02.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.065 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.065 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:02.065 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956366+0000", 2024-03-27T21:41:02.065 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001226471 2024-03-27T21:41:02.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.066 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:02.066 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956395+0000", 2024-03-27T21:41:02.066 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8782e-05 2024-03-27T21:41:02.066 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.066 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:02.066 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.066 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.066 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:90 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'60, mlcod=17'57)", 2024-03-27T21:41:02.066 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:02.066 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 82.072922090999995, 2024-03-27T21:41:02.066 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010778859999999999, 2024-03-27T21:41:02.067 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:02.067 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:02.067 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:02.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.067 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:02.067 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:02.067 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.067 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:02.067 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:02.067 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.067 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:02.068 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:02.068 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:02.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:02.068 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955747+0000", 2024-03-27T21:41:02.068 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7130999999999999e-05 2024-03-27T21:41:02.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:02.068 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955756+0000", 2024-03-27T21:41:02.068 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2750000000000001e-06 2024-03-27T21:41:02.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:02.068 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955761+0000", 2024-03-27T21:41:02.068 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3900000000000003e-06 2024-03-27T21:41:02.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.069 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:02.069 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955797+0000", 2024-03-27T21:41:02.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6650000000000003e-05 2024-03-27T21:41:02.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.069 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:02.069 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955816+0000", 2024-03-27T21:41:02.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8195000000000001e-05 2024-03-27T21:41:02.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.069 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:02.069 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956739+0000", 2024-03-27T21:41:02.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00092369700000000002 2024-03-27T21:41:02.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.070 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.070 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:02.070 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956768+0000", 2024-03-27T21:41:02.070 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9048e-05 2024-03-27T21:41:02.070 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.070 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:02.070 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.070 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.070 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:118 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'80, mlcod=17'78)", 2024-03-27T21:41:02.070 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:02.070 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 81.939931267999995, 2024-03-27T21:41:02.071 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014200669999999999, 2024-03-27T21:41:02.071 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:02.071 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:02.071 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:02.071 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.071 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:02.071 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:02.071 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.071 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.071 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.071 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:02.071 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:02.071 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.071 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.072 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:02.072 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:02.072 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:02.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.072 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:02.072 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088756+0000", 2024-03-27T21:41:02.072 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4902999999999998e-05 2024-03-27T21:41:02.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.072 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:02.072 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088764+0000", 2024-03-27T21:41:02.072 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8169999999999999e-06 2024-03-27T21:41:02.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.073 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:02.073 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088767+0000", 2024-03-27T21:41:02.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8310000000000003e-06 2024-03-27T21:41:02.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.073 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:02.073 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088798+0000", 2024-03-27T21:41:02.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0800999999999999e-05 2024-03-27T21:41:02.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.073 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:02.073 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088817+0000", 2024-03-27T21:41:02.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8467000000000001e-05 2024-03-27T21:41:02.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.074 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:02.074 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090074+0000", 2024-03-27T21:41:02.074 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012574089999999999 2024-03-27T21:41:02.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.074 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:02.074 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090101+0000", 2024-03-27T21:41:02.074 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7223e-05 2024-03-27T21:41:02.074 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.074 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:02.074 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.074 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:119 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'81, mlcod=17'78)", 2024-03-27T21:41:02.075 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:02.075 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 81.939196472999996, 2024-03-27T21:41:02.075 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014990870000000001, 2024-03-27T21:41:02.075 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:02.075 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:02.075 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:02.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.075 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:02.075 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:02.075 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.075 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:02.075 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:02.075 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.076 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:02.076 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:02.076 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:02.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.076 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:02.076 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089491+0000", 2024-03-27T21:41:02.076 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4805e-05 2024-03-27T21:41:02.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.076 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:02.076 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089498+0000", 2024-03-27T21:41:02.076 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0650000000000001e-06 2024-03-27T21:41:02.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.076 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:02.077 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089501+0000", 2024-03-27T21:41:02.077 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.54e-06 2024-03-27T21:41:02.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.077 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.077 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:02.077 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089533+0000", 2024-03-27T21:41:02.077 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1838999999999998e-05 2024-03-27T21:41:02.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.077 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.077 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:02.077 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089550+0000", 2024-03-27T21:41:02.077 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7465999999999998e-05 2024-03-27T21:41:02.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.077 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.078 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:02.078 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090890+0000", 2024-03-27T21:41:02.078 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013397979999999999 2024-03-27T21:41:02.078 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.078 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.078 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:02.078 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090915+0000", 2024-03-27T21:41:02.078 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4905e-05 2024-03-27T21:41:02.078 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.078 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:02.078 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.078 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.078 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.078 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:120 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'82, mlcod=17'78)", 2024-03-27T21:41:02.079 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:02.079 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 81.938396592999993, 2024-03-27T21:41:02.079 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001527093, 2024-03-27T21:41:02.079 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:02.079 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:02.079 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:02.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.079 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:02.079 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:02.079 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.079 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:02.079 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:02.079 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.080 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:02.080 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:02.080 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:02.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.080 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:02.080 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090267+0000", 2024-03-27T21:41:02.080 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1477000000000002e-05 2024-03-27T21:41:02.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.080 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:02.080 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090273+0000", 2024-03-27T21:41:02.081 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9519999999999999e-06 2024-03-27T21:41:02.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.081 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.081 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:02.081 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090276+0000", 2024-03-27T21:41:02.081 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.219e-06 2024-03-27T21:41:02.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.081 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.081 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:02.081 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090304+0000", 2024-03-27T21:41:02.081 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8073999999999999e-05 2024-03-27T21:41:02.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.081 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.081 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:02.081 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090316+0000", 2024-03-27T21:41:02.082 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2088e-05 2024-03-27T21:41:02.082 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.082 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.082 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:02.082 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091717+0000", 2024-03-27T21:41:02.082 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014003889999999999 2024-03-27T21:41:02.082 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.082 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.082 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:02.082 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091743+0000", 2024-03-27T21:41:02.082 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6234e-05 2024-03-27T21:41:02.082 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.082 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:02.082 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.082 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.083 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:121 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'83, mlcod=17'78)", 2024-03-27T21:41:02.083 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:02.083 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 81.937586228000001, 2024-03-27T21:41:02.083 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010226899999999999, 2024-03-27T21:41:02.083 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:02.083 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:02.083 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:02.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.083 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:02.083 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:02.083 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.084 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:02.084 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:02.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.084 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:02.084 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:02.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:02.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.084 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:02.084 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091142+0000", 2024-03-27T21:41:02.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011608300000000001 2024-03-27T21:41:02.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:02.085 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091151+0000", 2024-03-27T21:41:02.085 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2350000000000005e-06 2024-03-27T21:41:02.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:02.085 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091156+0000", 2024-03-27T21:41:02.085 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5419999999999999e-06 2024-03-27T21:41:02.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:02.085 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091191+0000", 2024-03-27T21:41:02.085 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5453000000000001e-05 2024-03-27T21:41:02.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:02.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091208+0000", 2024-03-27T21:41:02.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7119999999999999e-05 2024-03-27T21:41:02.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:02.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092024+0000", 2024-03-27T21:41:02.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000815539 2024-03-27T21:41:02.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:02.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092049+0000", 2024-03-27T21:41:02.087 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5171e-05 2024-03-27T21:41:02.087 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.087 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:02.087 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.087 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:142 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'100, mlcod=18'98)", 2024-03-27T21:41:02.087 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:02.087 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 81.522579273999995, 2024-03-27T21:41:02.087 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001198778, 2024-03-27T21:41:02.087 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:02.087 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:02.087 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:02.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.087 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:02.088 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:02.088 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.088 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:02.088 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:02.088 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.088 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:02.088 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:02.088 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:02.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:02.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506101+0000", 2024-03-27T21:41:02.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8202999999999997e-05 2024-03-27T21:41:02.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:02.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506109+0000", 2024-03-27T21:41:02.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3059999999999998e-06 2024-03-27T21:41:02.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:02.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506113+0000", 2024-03-27T21:41:02.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0119999999999999e-06 2024-03-27T21:41:02.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:02.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506137+0000", 2024-03-27T21:41:02.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4043999999999999e-05 2024-03-27T21:41:02.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:02.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506148+0000", 2024-03-27T21:41:02.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0446e-05 2024-03-27T21:41:02.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:02.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507213+0000", 2024-03-27T21:41:02.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001065215 2024-03-27T21:41:02.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.091 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:02.091 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507232+0000", 2024-03-27T21:41:02.091 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9038e-05 2024-03-27T21:41:02.091 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.091 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:02.091 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.091 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:143 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'101, mlcod=18'98)", 2024-03-27T21:41:02.091 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:02.091 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 81.521966655, 2024-03-27T21:41:02.091 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001357488, 2024-03-27T21:41:02.092 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:02.092 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:02.092 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:02.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:02.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:02.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:02.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:02.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:02.093 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:02.093 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:02.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.093 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:02.093 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506723+0000", 2024-03-27T21:41:02.093 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7404000000000006e-05 2024-03-27T21:41:02.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.093 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:02.093 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506730+0000", 2024-03-27T21:41:02.093 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5749999999999997e-06 2024-03-27T21:41:02.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.094 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:02.094 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506733+0000", 2024-03-27T21:41:02.094 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7979999999999999e-06 2024-03-27T21:41:02.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.094 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.094 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:02.094 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506758+0000", 2024-03-27T21:41:02.094 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4179e-05 2024-03-27T21:41:02.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.094 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.094 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:02.094 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506768+0000", 2024-03-27T21:41:02.094 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0740000000000001e-05 2024-03-27T21:41:02.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.094 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.095 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:02.095 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507978+0000", 2024-03-27T21:41:02.095 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012093900000000001 2024-03-27T21:41:02.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.095 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:02.095 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508003+0000", 2024-03-27T21:41:02.095 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5667e-05 2024-03-27T21:41:02.095 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.095 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:02.095 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.095 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:144 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'102, mlcod=18'98)", 2024-03-27T21:41:02.096 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:41:02.096 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 81.521307758000006, 2024-03-27T21:41:02.096 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013453390000000001, 2024-03-27T21:41:02.096 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:02.096 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:02.096 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:02.096 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.096 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:02.096 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:41:02.096 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.097 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:02.097 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:41:02.097 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:02.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.097 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:02.097 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507304+0000", 2024-03-27T21:41:02.097 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:02.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.097 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:02.097 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507397+0000", 2024-03-27T21:41:02.097 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2995e-05 2024-03-27T21:41:02.098 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.098 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.098 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:02.098 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507403+0000", 2024-03-27T21:41:02.098 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8640000000000001e-06 2024-03-27T21:41:02.098 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.098 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.098 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:02.098 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507407+0000", 2024-03-27T21:41:02.098 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5710000000000002e-06 2024-03-27T21:41:02.098 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.098 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.098 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:02.098 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507428+0000", 2024-03-27T21:41:02.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1192e-05 2024-03-27T21:41:02.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.099 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:02.099 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507438+0000", 2024-03-27T21:41:02.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8309999999999996e-06 2024-03-27T21:41:02.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.099 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:02.099 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508624+0000", 2024-03-27T21:41:02.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011861210000000001 2024-03-27T21:41:02.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:02.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:02.099 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:02.100 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508650+0000", 2024-03-27T21:41:02.100 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6118e-05 2024-03-27T21:41:02.100 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.100 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:02.100 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.100 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:02.100 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:02.100 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:02.101 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:41:02.102 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:02.116+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:02.203 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:02.217+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:02.223 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:02.223 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:02.223 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:02.223 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:02.232 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:41:02.304 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:02.319+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:02.344 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:02.344 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:02.345 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:41:02.345 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:41:02.345 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:02.354 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:41:02.405 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:02.420+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:02.466 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:02.467 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:41:02.467 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:41:02.467 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:41:02.467 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:02.479 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:41:02.506 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:02.520+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:02.591 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:02.591 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:02.591 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:02.591 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:02.600 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:41:02.607 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:02.621+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:02.708 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:02.722+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:02.710 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:02.710 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:02.710 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:41:02.710 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:41:02.710 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:02.720 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:41:02.809 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:02.823+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:02.837 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:02.837 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:41:02.838 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:41:02.838 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:41:02.838 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:02.847 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:41:02.910 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:02.925+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:02.964 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:02.965 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:02.965 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:02.965 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:02.974 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:41:03.015 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:03.030+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:03.112 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:03.126+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:03.120 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:03.120 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:03.120 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:03.121 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:03.121 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:03.129 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:41:03.213 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:03.228+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:03.275 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:03.275 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:03.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:03.275 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:41:03.275 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:03.284 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:41:03.324 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:03.339+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:03.369 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:41:03.415 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:03.429+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:03.428 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:03.429 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:03.429 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:03.429 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:03.437 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:41:03.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.481+0000 7f16ab32e640 10 monclient: get_monmap_and_config 2024-03-27T21:41:03.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.481+0000 7f16ab32e640 10 monclient: build_initial_monmap 2024-03-27T21:41:03.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.481+0000 7f16ab32e640 10 monclient: monmap: 2024-03-27T21:41:03.467 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:03.468 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:03.468 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:03.482510+0000 2024-03-27T21:41:03.468 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:03.482510+0000 2024-03-27T21:41:03.468 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:03.468 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:03.468 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:03.468 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:03.468 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:03.468 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:03.468 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:03.468 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:03.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.482+0000 7f16ab32e640 1 Processor -- start 2024-03-27T21:41:03.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.482+0000 7f16ab32e640 1 -- start start 2024-03-27T21:41:03.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.482+0000 7f16ab32e640 10 monclient: init 2024-03-27T21:41:03.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16ab32e640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:03.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16ab32e640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-03-27T21:41:03.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16ab32e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f16a40717b0 0x7f16a41414c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:03.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16ab32e640 10 monclient(hunting): picked mon.noname-c con 0x7f16a40717b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:03.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16ab32e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f16a4071b90 0x7f16a40a8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:03.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16ab32e640 10 monclient(hunting): picked mon.noname-b con 0x7f16a4071b90 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:03.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16ab32e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f16a40b0fa0 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:03.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16ab32e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16a40b0880 con 0x7f16a40b0fa0 2024-03-27T21:41:03.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16ab32e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:03.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16ab32e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16a4072300 con 0x7f16a40717b0 2024-03-27T21:41:03.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16ab32e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:03.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16ab32e640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a4141a00 con 0x7f16a4071b90 2024-03-27T21:41:03.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16ab32e640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:03.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16ab32e640 10 monclient(hunting): authenticate will time out at 904.704093s 2024-03-27T21:41:03.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a88c9640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f16a4071b90 0x7f16a40a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:03.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a98cb640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f16a40b0fa0 0x7f16a40b0170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50900/0 (socket says 172.21.15.60:50900) 2024-03-27T21:41:03.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a88c9640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f16a4071b90 0x7f16a40a8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:39966/0 (socket says 172.21.15.60:39966) 2024-03-27T21:41:03.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a98cb640 1 -- 172.21.15.60:0/1497757524 learned_addr learned my addr 172.21.15.60:0/1497757524 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:03.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a88c9640 10 monclient(hunting): get_auth_request con 0x7f16a4071b90 auth_method 0 2024-03-27T21:41:03.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a88c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:03.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a88c9640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:03.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a88c9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:03.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a88c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:03.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a88c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:03.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a90ca640 1 --2- 172.21.15.60:0/1497757524 >> v2:172.21.15.191:3300/0 conn(0x7f16a40717b0 0x7f16a41414c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:03.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a88c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:03.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a90ca640 10 monclient(hunting): get_auth_request con 0x7f16a40717b0 auth_method 0 2024-03-27T21:41:03.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a90ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:03.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a90ca640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:03.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a90ca640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:03.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a90ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:03.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a90ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:03.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a90ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:03.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a88c9640 10 monclient(hunting): handle_auth_done global_id 4667 payload 274 2024-03-27T21:41:03.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f169bfff640 1 -- 172.21.15.60:0/1497757524 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395419966 0 0) 0x7f16a40b0880 con 0x7f16a40b0fa0 2024-03-27T21:41:03.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a88c9640 1 -- 172.21.15.60:0/1497757524 >> v2:172.21.15.191:3300/0 conn(0x7f16a40717b0 msgr2=0x7f16a41414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:03.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a88c9640 1 --2- 172.21.15.60:0/1497757524 >> v2:172.21.15.191:3300/0 conn(0x7f16a40717b0 0x7f16a41414c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:03.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a88c9640 1 -- 172.21.15.60:0/1497757524 >> v1:172.21.15.60:6789/0 conn(0x7f16a40b0fa0 legacy=0x7f16a40b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:03.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a88c9640 10 monclient: _finish_hunting 0 2024-03-27T21:41:03.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a88c9640 1 monclient: found mon.noname-b 2024-03-27T21:41:03.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a88c9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:03.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a88c9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:03.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f16a88c9640 1 -- 172.21.15.60:0/1497757524 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a40a8980 con 0x7f16a4071b90 2024-03-27T21:41:03.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.483+0000 7f169bfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:41:03.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.484+0000 7f16a88c9640 1 --2- 172.21.15.60:0/1497757524 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f16a4071b90 0x7f16a40a8440 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f16940097a0 tx=0x7f169402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e7d731e022c0199 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:03.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.484+0000 7f169bfff640 1 -- 172.21.15.60:0/1497757524 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169402e070 con 0x7f16a4071b90 2024-03-27T21:41:03.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.484+0000 7f169bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:03.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.484+0000 7f169bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:03.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.484+0000 7f169bfff640 10 monclient: dump: 2024-03-27T21:41:03.482 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:03.482 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:03.483 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:03.483 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:03.483 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:03.483 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:03.483 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:03.483 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:03.483 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:03.483 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:03.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.484+0000 7f169bfff640 10 monclient: _finish_auth 0 2024-03-27T21:41:03.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.484+0000 7f169bfff640 10 monclient: _check_auth_tickets 2024-03-27T21:41:03.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.484+0000 7f169bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:03.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.484+0000 7f16ab32e640 5 monclient: authenticate success, global_id 4667 2024-03-27T21:41:03.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.484+0000 7f16ab32e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:03.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.484+0000 7f169bfff640 1 -- 172.21.15.60:0/1497757524 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16940315a0 con 0x7f16a4071b90 2024-03-27T21:41:03.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.484+0000 7f169bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:03.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.484+0000 7f16ab32e640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:03.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.484+0000 7f16ab32e640 4 set_mon_vals no callback set 2024-03-27T21:41:03.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.484+0000 7f169bfff640 1 -- 172.21.15.60:0/1497757524 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1694031930 con 0x7f16a4071b90 2024-03-27T21:41:03.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.484+0000 7f169bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:03.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.484+0000 7f169bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:03.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.484+0000 7f169bfff640 10 monclient: dump: 2024-03-27T21:41:03.484 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:03.484 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:03.484 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:03.484 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:03.484 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:03.484 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:03.484 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:03.484 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:03.485 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:03.485 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:03.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.485+0000 7f16ab32e640 10 monclient: shutdown 2024-03-27T21:41:03.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.485+0000 7f16ab32e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:03.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.485+0000 7f16ab32e640 1 -- 172.21.15.60:0/1497757524 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f16a4071b90 msgr2=0x7f16a40a8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:03.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.485+0000 7f16ab32e640 1 --2- 172.21.15.60:0/1497757524 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f16a4071b90 0x7f16a40a8440 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f16940097a0 tx=0x7f169402b840 comp rx=0 tx=0).stop 2024-03-27T21:41:03.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.485+0000 7f16ab32e640 1 -- 172.21.15.60:0/1497757524 shutdown_connections 2024-03-27T21:41:03.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.485+0000 7f16ab32e640 1 --2- 172.21.15.60:0/1497757524 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f16a4071b90 0x7f16a40a8440 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:03.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.485+0000 7f16ab32e640 1 --2- 172.21.15.60:0/1497757524 >> v2:172.21.15.191:3300/0 conn(0x7f16a40717b0 0x7f16a41414c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:03.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.485+0000 7f16ab32e640 1 -- 172.21.15.60:0/1497757524 >> 172.21.15.60:0/1497757524 conn(0x7f16a40b7290 msgr2=0x7f16a40b76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:03.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.485+0000 7f16ab32e640 1 -- 172.21.15.60:0/1497757524 shutdown_connections 2024-03-27T21:41:03.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.485+0000 7f16ab32e640 1 -- 172.21.15.60:0/1497757524 wait complete. 2024-03-27T21:41:03.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.485+0000 7f16ab32e640 10 monclient: build_initial_monmap 2024-03-27T21:41:03.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.485+0000 7f16ab32e640 10 monclient: monmap: 2024-03-27T21:41:03.486 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:03.486 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:03.486 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:03.486 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:03.486 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:03.486 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:03.486 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:03.487 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:03.487 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:03.487 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:03.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 1 librados: starting msgr at 2024-03-27T21:41:03.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 1 librados: starting objecter 2024-03-27T21:41:03.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 1 Processor -- start 2024-03-27T21:41:03.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 1 -- start start 2024-03-27T21:41:03.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 1 librados: setting wanted keys 2024-03-27T21:41:03.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 1 librados: calling monclient init 2024-03-27T21:41:03.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 10 monclient: init 2024-03-27T21:41:03.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:03.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:41:03.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f16a40717b0 0x7f16a40ae990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:03.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 10 monclient(hunting): picked mon.noname-c con 0x7f16a40717b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:03.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16a4071b90 0x7f16a40aeed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:03.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 10 monclient(hunting): picked mon.noname-b con 0x7f16a4071b90 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:03.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16a40b0fa0 0x7f16a41d9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:03.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 10 monclient(hunting): picked mon.noname-a con 0x7f16a40b0fa0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:03.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:03.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a41426f0 con 0x7f16a40b0fa0 2024-03-27T21:41:03.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:03.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a414bb40 con 0x7f16a4071b90 2024-03-27T21:41:03.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:03.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a413a680 con 0x7f16a40717b0 2024-03-27T21:41:03.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:03.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16ab32e640 10 monclient(hunting): authenticate will time out at 904.707739s 2024-03-27T21:41:03.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16a90ca640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f16a40717b0 0x7f16a40ae990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:03.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16a90ca640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f16a40717b0 0x7f16a40ae990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:39978/0 (socket says 172.21.15.60:39978) 2024-03-27T21:41:03.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16a90ca640 1 -- 172.21.15.60:0/435548250 learned_addr learned my addr 172.21.15.60:0/435548250 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:03.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16a90ca640 10 monclient(hunting): get_auth_request con 0x7f16a40717b0 auth_method 0 2024-03-27T21:41:03.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16a90ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:03.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16a90ca640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:03.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16a90ca640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:03.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16a98cb640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16a40b0fa0 0x7f16a41d9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:03.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16a98cb640 10 monclient(hunting): get_auth_request con 0x7f16a40b0fa0 auth_method 0 2024-03-27T21:41:03.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16a98cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:03.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16a98cb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:03.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16a98cb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:03.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16a90ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:03.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16a90ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:03.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16a90ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:03.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.486+0000 7f16a88c9640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16a4071b90 0x7f16a40aeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:03.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a88c9640 10 monclient(hunting): get_auth_request con 0x7f16a4071b90 auth_method 0 2024-03-27T21:41:03.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a88c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:03.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a88c9640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:03.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a88c9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:03.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a98cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:03.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a98cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:03.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a98cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:03.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a88c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:03.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a88c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:03.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a88c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:03.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a90ca640 10 monclient(hunting): handle_auth_done global_id 4670 payload 824 2024-03-27T21:41:03.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a90ca640 1 -- 172.21.15.60:0/435548250 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16a4071b90 msgr2=0x7f16a40aeed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:03.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a90ca640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16a4071b90 0x7f16a40aeed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:03.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a90ca640 1 -- 172.21.15.60:0/435548250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16a40b0fa0 msgr2=0x7f16a41d9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:03.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a90ca640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16a40b0fa0 0x7f16a41d9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:03.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a90ca640 10 monclient: _finish_hunting 0 2024-03-27T21:41:03.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a90ca640 1 monclient: found mon.noname-c 2024-03-27T21:41:03.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a90ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:03.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a90ca640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:03.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a90ca640 1 -- 172.21.15.60:0/435548250 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1694009400 con 0x7f16a40717b0 2024-03-27T21:41:03.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a98cb640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16a40b0fa0 0x7f16a41d9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:03.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16a90ca640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f16a40717b0 0x7f16a40ae990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f16a000e8a0 tx=0x7f16a000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:03.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f1699ffb640 10 client.?.objecter ms_handle_connect 0x7f16a40717b0 2024-03-27T21:41:03.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f1699ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:03.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f1699ffb640 1 -- 172.21.15.60:0/435548250 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a00094f0 con 0x7f16a40717b0 2024-03-27T21:41:03.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f1699ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:03.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f1699ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:03.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f1699ffb640 10 monclient: dump: 2024-03-27T21:41:03.492 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:03.492 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:03.492 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:03.492 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:03.492 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:03.492 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:03.493 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:03.493 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:03.493 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:03.493 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:03.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f1699ffb640 10 monclient: _finish_auth 0 2024-03-27T21:41:03.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f1699ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:41:03.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f1699ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:03.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16ab32e640 5 monclient: authenticate success, global_id 4670 2024-03-27T21:41:03.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f1699ffb640 1 -- 172.21.15.60:0/435548250 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a000c940 con 0x7f16a40717b0 2024-03-27T21:41:03.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f1699ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:03.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f1699ffb640 1 -- 172.21.15.60:0/435548250 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a000ccf0 con 0x7f16a40717b0 2024-03-27T21:41:03.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16ab32e640 10 monclient: _renew_subs 2024-03-27T21:41:03.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16ab32e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:03.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16ab32e640 1 -- 172.21.15.60:0/435548250 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16a40adb50 con 0x7f16a40717b0 2024-03-27T21:41:03.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f1699ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:03.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f169b7fe640 4 set_mon_vals no callback set 2024-03-27T21:41:03.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f1699ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:03.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f1699ffb640 10 monclient: dump: 2024-03-27T21:41:03.494 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:03.494 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:03.494 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:03.494 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:03.494 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:03.494 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:03.494 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:03.494 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:03.494 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:03.495 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:03.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16ab32e640 10 client.4670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:03.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16ab32e640 10 monclient: _renew_subs 2024-03-27T21:41:03.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16ab32e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:03.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16ab32e640 1 -- 172.21.15.60:0/435548250 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16a414b2d0 con 0x7f16a40717b0 2024-03-27T21:41:03.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.487+0000 7f16ab32e640 1 librados: init done 2024-03-27T21:41:03.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.488+0000 7f16ab32e640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:41:03.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.488+0000 7f16ab32e640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:03.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.488+0000 7f16ab32e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:03.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.488+0000 7f16ab32e640 1 -- 172.21.15.60:0/435548250 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f16a4069470 con 0x7f16a40717b0 2024-03-27T21:41:03.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.489+0000 7f1699ffb640 1 -- 172.21.15.60:0/435548250 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f16a0013470 con 0x7f16a40717b0 2024-03-27T21:41:03.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.489+0000 7f1699ffb640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f168806c590 0x7f168806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:03.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.489+0000 7f1699ffb640 1 -- 172.21.15.60:0/435548250 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f16a0089110 con 0x7f16a40717b0 2024-03-27T21:41:03.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.489+0000 7f1699ffb640 10 client.4670.objecter ms_dispatch 0x7f16a4000b90 osd_map(50..50 src has 1..50) v4 2024-03-27T21:41:03.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.489+0000 7f1699ffb640 3 client.4670.objecter handle_osd_map got epochs [50,50] > 0 2024-03-27T21:41:03.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.489+0000 7f1699ffb640 3 client.4670.objecter handle_osd_map decoding full epoch 50 2024-03-27T21:41:03.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.489+0000 7f16a88c9640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f168806c590 0x7f168806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:03.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.489+0000 7f16a88c9640 10 monclient: get_auth_request con 0x7f168806c590 auth_method 0 2024-03-27T21:41:03.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.489+0000 7f1699ffb640 20 client.4670.objecter dump_active .. 0 homeless 2024-03-27T21:41:03.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.489+0000 7f1699ffb640 1 -- 172.21.15.60:0/435548250 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v50) v1 ==== 54+0+16 (secure 0 0 0) 0x7f16a0090070 con 0x7f16a40717b0 2024-03-27T21:41:03.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.489+0000 7f1699ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:03.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.489+0000 7f1699ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:03.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.489+0000 7f16ab32e640 10 client.4670.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:03.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.490+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign -1 1 2024-03-27T21:41:03.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.490+0000 7f16ab32e640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f16a41da3e0 0x7f16a4065a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:03.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.490+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a414bcb0 osd=0 3 2024-03-27T21:41:03.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.490+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a414bcb0 osd=0 3 2024-03-27T21:41:03.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.490+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a414bcb0 osd=0 3 2024-03-27T21:41:03.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.490+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_remove -1 1 2024-03-27T21:41:03.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.490+0000 7f16ab32e640 20 client.4670.objecter get_session s=0x7f16a414bcb0 osd=0 3 2024-03-27T21:41:03.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.490+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign 0 1 2024-03-27T21:41:03.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.490+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a414bcb0 osd=0 4 2024-03-27T21:41:03.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.490+0000 7f16ab32e640 10 client.4670.objecter _send_command 1 2024-03-27T21:41:03.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.490+0000 7f16ab32e640 1 -- 172.21.15.60:0/435548250 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16a41da7c0 con 0x7f16a41da3e0 2024-03-27T21:41:03.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.490+0000 7f16a98cb640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f16a41da3e0 0x7f16a4065a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:03.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.490+0000 7f16a98cb640 10 monclient: get_auth_request con 0x7f16a41da3e0 auth_method 0 2024-03-27T21:41:03.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.490+0000 7f16a88c9640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f168806c590 0x7f168806ea50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1694036000 tx=0x7f169403e040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:03.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.490+0000 7f1699ffb640 10 client.4670.objecter ms_handle_connect 0x7f168806c590 2024-03-27T21:41:03.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.490+0000 7f1699ffb640 10 client.4670.objecter ms_handle_connect 0x7f16a41da3e0 2024-03-27T21:41:03.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.490+0000 7f16a98cb640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f16a41da3e0 0x7f16a4065a90 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:03.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.491+0000 7f1699ffb640 1 -- 172.21.15.60:0/435548250 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f16a41da7c0 con 0x7f16a41da3e0 2024-03-27T21:41:03.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.491+0000 7f1699ffb640 10 client.4670.objecter ms_dispatch 0x7f16a4000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:41:03.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.491+0000 7f1699ffb640 10 client.4670.objecter _finish_command 1 = system:0 2024-03-27T21:41:03.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.491+0000 7f1699ffb640 20 client.4670.objecter put_session s=0x7f16a414bcb0 osd=0 4 2024-03-27T21:41:03.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.491+0000 7f1699ffb640 15 client.4670.objecter _session_command_op_remove 0 1 2024-03-27T21:41:03.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.505+0000 7f16ab32e640 10 client.4670.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:03.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.505+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign -1 2 2024-03-27T21:41:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.505+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a414bcb0 osd=0 3 2024-03-27T21:41:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.505+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a414bcb0 osd=0 3 2024-03-27T21:41:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.505+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a414bcb0 osd=0 3 2024-03-27T21:41:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.505+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_remove -1 2 2024-03-27T21:41:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.505+0000 7f16ab32e640 20 client.4670.objecter get_session s=0x7f16a414bcb0 osd=0 3 2024-03-27T21:41:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.505+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign 0 2 2024-03-27T21:41:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.505+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a414bcb0 osd=0 4 2024-03-27T21:41:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.505+0000 7f16ab32e640 10 client.4670.objecter _send_command 2 2024-03-27T21:41:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.505+0000 7f16ab32e640 1 -- 172.21.15.60:0/435548250 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f16a40a9380 con 0x7f16a41da3e0 2024-03-27T21:41:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.507+0000 7f1699ffb640 1 -- 172.21.15.60:0/435548250 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f16a40a9380 con 0x7f16a41da3e0 2024-03-27T21:41:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.507+0000 7f1699ffb640 10 client.4670.objecter ms_dispatch 0x7f16a4000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.507+0000 7f1699ffb640 10 client.4670.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.507+0000 7f1699ffb640 20 client.4670.objecter put_session s=0x7f16a414bcb0 osd=0 4 2024-03-27T21:41:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.507+0000 7f1699ffb640 15 client.4670.objecter _session_command_op_remove 0 2 2024-03-27T21:41:03.502 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:41:03.503 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:41:03.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.507+0000 7f16ab32e640 10 client.4670.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:03.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.507+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign -1 3 2024-03-27T21:41:03.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.507+0000 7f16ab32e640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f16a402b170 0x7f16a402b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:03.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.507+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a4066790 osd=1 3 2024-03-27T21:41:03.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.507+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a4066790 osd=1 3 2024-03-27T21:41:03.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.507+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a4066790 osd=1 3 2024-03-27T21:41:03.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.507+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_remove -1 3 2024-03-27T21:41:03.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.507+0000 7f16ab32e640 20 client.4670.objecter get_session s=0x7f16a4066790 osd=1 3 2024-03-27T21:41:03.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.507+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign 1 3 2024-03-27T21:41:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.507+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a4066790 osd=1 4 2024-03-27T21:41:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.507+0000 7f16ab32e640 10 client.4670.objecter _send_command 3 2024-03-27T21:41:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.507+0000 7f16ab32e640 1 -- 172.21.15.60:0/435548250 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16a40ad6b0 con 0x7f16a402b170 2024-03-27T21:41:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.507+0000 7f16a90ca640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f16a402b170 0x7f16a402b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.507+0000 7f16a90ca640 10 monclient: get_auth_request con 0x7f16a402b170 auth_method 0 2024-03-27T21:41:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.507+0000 7f1699ffb640 10 client.4670.objecter ms_handle_connect 0x7f16a402b170 2024-03-27T21:41:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.507+0000 7f16a90ca640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f16a402b170 0x7f16a402b550 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.508+0000 7f1699ffb640 1 -- 172.21.15.60:0/435548250 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f16a40ad6b0 con 0x7f16a402b170 2024-03-27T21:41:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.508+0000 7f1699ffb640 10 client.4670.objecter ms_dispatch 0x7f16a4000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:41:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.508+0000 7f1699ffb640 10 client.4670.objecter _finish_command 3 = system:0 2024-03-27T21:41:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.508+0000 7f1699ffb640 20 client.4670.objecter put_session s=0x7f16a4066790 osd=1 4 2024-03-27T21:41:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.508+0000 7f1699ffb640 15 client.4670.objecter _session_command_op_remove 1 3 2024-03-27T21:41:03.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.522+0000 7f16ab32e640 10 client.4670.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:03.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.522+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign -1 4 2024-03-27T21:41:03.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.522+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a4066790 osd=1 3 2024-03-27T21:41:03.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.522+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a4066790 osd=1 3 2024-03-27T21:41:03.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.522+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a4066790 osd=1 3 2024-03-27T21:41:03.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.522+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_remove -1 4 2024-03-27T21:41:03.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.522+0000 7f16ab32e640 20 client.4670.objecter get_session s=0x7f16a4066790 osd=1 3 2024-03-27T21:41:03.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.522+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign 1 4 2024-03-27T21:41:03.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.522+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a4066790 osd=1 4 2024-03-27T21:41:03.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.522+0000 7f16ab32e640 10 client.4670.objecter _send_command 4 2024-03-27T21:41:03.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.522+0000 7f16ab32e640 1 -- 172.21.15.60:0/435548250 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f16a402ba90 con 0x7f16a402b170 2024-03-27T21:41:03.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f1699ffb640 1 -- 172.21.15.60:0/435548250 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f16a402ba90 con 0x7f16a402b170 2024-03-27T21:41:03.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f1699ffb640 10 client.4670.objecter ms_dispatch 0x7f16a4000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:03.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f1699ffb640 10 client.4670.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:03.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f1699ffb640 20 client.4670.objecter put_session s=0x7f16a4066790 osd=1 4 2024-03-27T21:41:03.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f1699ffb640 15 client.4670.objecter _session_command_op_remove 1 4 2024-03-27T21:41:03.511 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:41:03.513 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:41:03.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f16ab32e640 10 client.4670.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:03.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign -1 5 2024-03-27T21:41:03.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f16ab32e640 20 client.4670.objecter _get_session osd=-1 returning homeless 2024-03-27T21:41:03.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f16ab32e640 10 client.4670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:03.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f16ab32e640 10 monclient: _renew_subs 2024-03-27T21:41:03.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f16ab32e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:03.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f16ab32e640 1 -- 172.21.15.60:0/435548250 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=51}) v3 -- 0x7f16a402c420 con 0x7f16a40717b0 2024-03-27T21:41:03.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f16ab32e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:03.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f16ab32e640 1 -- 172.21.15.60:0/435548250 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f16a4066210 con 0x7f16a40717b0 2024-03-27T21:41:03.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f1699ffb640 1 -- 172.21.15.60:0/435548250 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16a00582c0 con 0x7f16a40717b0 2024-03-27T21:41:03.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f1699ffb640 10 monclient: handle_get_version_reply finishing 1 version 50 2024-03-27T21:41:03.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f169b7fe640 10 client.4670.objecter _check_command_map_dne tid 5 current 50 map_dne_bound 50 2024-03-27T21:41:03.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f169b7fe640 10 client.4670.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:41:03.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f169b7fe640 15 client.4670.objecter _session_command_op_remove -1 5 2024-03-27T21:41:03.514 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:41:03.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f16ab32e640 10 client.4670.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:03.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign -1 6 2024-03-27T21:41:03.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f16ab32e640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f16a40ac620 0x7f16a40aca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:03.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a40abfa0 osd=3 3 2024-03-27T21:41:03.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a40abfa0 osd=3 3 2024-03-27T21:41:03.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a40abfa0 osd=3 3 2024-03-27T21:41:03.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_remove -1 6 2024-03-27T21:41:03.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f16ab32e640 20 client.4670.objecter get_session s=0x7f16a40abfa0 osd=3 3 2024-03-27T21:41:03.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign 3 6 2024-03-27T21:41:03.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a40abfa0 osd=3 4 2024-03-27T21:41:03.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f16ab32e640 10 client.4670.objecter _send_command 6 2024-03-27T21:41:03.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.524+0000 7f16ab32e640 1 -- 172.21.15.60:0/435548250 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16a40acf40 con 0x7f16a40ac620 2024-03-27T21:41:03.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.525+0000 7f16a88c9640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f16a40ac620 0x7f16a40aca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:03.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.525+0000 7f16a88c9640 10 monclient: get_auth_request con 0x7f16a40ac620 auth_method 0 2024-03-27T21:41:03.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.525+0000 7f1699ffb640 10 client.4670.objecter ms_handle_connect 0x7f16a40ac620 2024-03-27T21:41:03.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.525+0000 7f16a88c9640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f16a40ac620 0x7f16a40aca00 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:03.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.525+0000 7f1699ffb640 1 -- 172.21.15.60:0/435548250 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f16a40acf40 con 0x7f16a40ac620 2024-03-27T21:41:03.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.525+0000 7f1699ffb640 10 client.4670.objecter ms_dispatch 0x7f16a4000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:41:03.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.525+0000 7f1699ffb640 10 client.4670.objecter _finish_command 6 = system:0 2024-03-27T21:41:03.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.525+0000 7f1699ffb640 20 client.4670.objecter put_session s=0x7f16a40abfa0 osd=3 4 2024-03-27T21:41:03.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.525+0000 7f1699ffb640 15 client.4670.objecter _session_command_op_remove 3 6 2024-03-27T21:41:03.516 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:03.531+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:03.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.540+0000 7f16ab32e640 10 client.4670.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:03.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.540+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign -1 7 2024-03-27T21:41:03.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.540+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a40abfa0 osd=3 3 2024-03-27T21:41:03.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.540+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a40abfa0 osd=3 3 2024-03-27T21:41:03.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.540+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a40abfa0 osd=3 3 2024-03-27T21:41:03.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.540+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_remove -1 7 2024-03-27T21:41:03.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.540+0000 7f16ab32e640 20 client.4670.objecter get_session s=0x7f16a40abfa0 osd=3 3 2024-03-27T21:41:03.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.540+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign 3 7 2024-03-27T21:41:03.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.540+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a40abfa0 osd=3 4 2024-03-27T21:41:03.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.540+0000 7f16ab32e640 10 client.4670.objecter _send_command 7 2024-03-27T21:41:03.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.540+0000 7f16ab32e640 1 -- 172.21.15.60:0/435548250 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f16a40ad2f0 con 0x7f16a40ac620 2024-03-27T21:41:03.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.542+0000 7f1699ffb640 1 -- 172.21.15.60:0/435548250 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f16a40ad2f0 con 0x7f16a40ac620 2024-03-27T21:41:03.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.542+0000 7f1699ffb640 10 client.4670.objecter ms_dispatch 0x7f16a4000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:03.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.542+0000 7f1699ffb640 10 client.4670.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:03.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.542+0000 7f1699ffb640 20 client.4670.objecter put_session s=0x7f16a40abfa0 osd=3 4 2024-03-27T21:41:03.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.542+0000 7f1699ffb640 15 client.4670.objecter _session_command_op_remove 3 7 2024-03-27T21:41:03.529 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:41:03.530 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:41:03.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.542+0000 7f16ab32e640 10 client.4670.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:03.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.542+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign -1 8 2024-03-27T21:41:03.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.542+0000 7f16ab32e640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f16a41e7300 0x7f16a41e96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:03.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.542+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a41e6c80 osd=4 3 2024-03-27T21:41:03.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.542+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a41e6c80 osd=4 3 2024-03-27T21:41:03.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.542+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a41e6c80 osd=4 3 2024-03-27T21:41:03.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.542+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_remove -1 8 2024-03-27T21:41:03.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.542+0000 7f16ab32e640 20 client.4670.objecter get_session s=0x7f16a41e6c80 osd=4 3 2024-03-27T21:41:03.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.542+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign 4 8 2024-03-27T21:41:03.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.542+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a41e6c80 osd=4 4 2024-03-27T21:41:03.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.542+0000 7f16ab32e640 10 client.4670.objecter _send_command 8 2024-03-27T21:41:03.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.542+0000 7f16ab32e640 1 -- 172.21.15.60:0/435548250 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16a41ec4c0 con 0x7f16a41e7300 2024-03-27T21:41:03.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.542+0000 7f16a98cb640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f16a41e7300 0x7f16a41e96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:03.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.542+0000 7f16a98cb640 10 monclient: get_auth_request con 0x7f16a41e7300 auth_method 0 2024-03-27T21:41:03.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.543+0000 7f1699ffb640 10 client.4670.objecter ms_handle_connect 0x7f16a41e7300 2024-03-27T21:41:03.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.543+0000 7f16a98cb640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f16a41e7300 0x7f16a41e96f0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:03.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.543+0000 7f1699ffb640 1 -- 172.21.15.60:0/435548250 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f16a41ec4c0 con 0x7f16a41e7300 2024-03-27T21:41:03.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.544+0000 7f1699ffb640 10 client.4670.objecter ms_dispatch 0x7f16a4000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:41:03.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.544+0000 7f1699ffb640 10 client.4670.objecter _finish_command 8 = system:0 2024-03-27T21:41:03.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.544+0000 7f1699ffb640 20 client.4670.objecter put_session s=0x7f16a41e6c80 osd=4 4 2024-03-27T21:41:03.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.544+0000 7f1699ffb640 15 client.4670.objecter _session_command_op_remove 4 8 2024-03-27T21:41:03.542 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:03.542 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:03.543 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:03.543 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:03.543 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:03.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.558+0000 7f16ab32e640 10 client.4670.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:03.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.558+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign -1 9 2024-03-27T21:41:03.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.558+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a41e6c80 osd=4 3 2024-03-27T21:41:03.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.558+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a41e6c80 osd=4 3 2024-03-27T21:41:03.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.558+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a41e6c80 osd=4 3 2024-03-27T21:41:03.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.558+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_remove -1 9 2024-03-27T21:41:03.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.558+0000 7f16ab32e640 20 client.4670.objecter get_session s=0x7f16a41e6c80 osd=4 3 2024-03-27T21:41:03.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.558+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign 4 9 2024-03-27T21:41:03.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.558+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a41e6c80 osd=4 4 2024-03-27T21:41:03.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.558+0000 7f16ab32e640 10 client.4670.objecter _send_command 9 2024-03-27T21:41:03.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.558+0000 7f16ab32e640 1 -- 172.21.15.60:0/435548250 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f16a402bea0 con 0x7f16a41e7300 2024-03-27T21:41:03.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.559+0000 7f1699ffb640 1 -- 172.21.15.60:0/435548250 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f16a402bea0 con 0x7f16a41e7300 2024-03-27T21:41:03.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.559+0000 7f1699ffb640 10 client.4670.objecter ms_dispatch 0x7f16a4000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:03.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.559+0000 7f1699ffb640 10 client.4670.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:03.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.559+0000 7f1699ffb640 20 client.4670.objecter put_session s=0x7f16a41e6c80 osd=4 4 2024-03-27T21:41:03.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.559+0000 7f1699ffb640 15 client.4670.objecter _session_command_op_remove 4 9 2024-03-27T21:41:03.546 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:41:03.547 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:41:03.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.559+0000 7f16ab32e640 10 client.4670.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:03.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.559+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign -1 10 2024-03-27T21:41:03.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.559+0000 7f16ab32e640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f16a40ab670 0x7f16a41eadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:03.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.559+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a402c0e0 osd=5 3 2024-03-27T21:41:03.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.559+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a402c0e0 osd=5 3 2024-03-27T21:41:03.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.559+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a402c0e0 osd=5 3 2024-03-27T21:41:03.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.559+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_remove -1 10 2024-03-27T21:41:03.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.559+0000 7f16ab32e640 20 client.4670.objecter get_session s=0x7f16a402c0e0 osd=5 3 2024-03-27T21:41:03.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.559+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign 5 10 2024-03-27T21:41:03.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.559+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a402c0e0 osd=5 4 2024-03-27T21:41:03.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.559+0000 7f16ab32e640 10 client.4670.objecter _send_command 10 2024-03-27T21:41:03.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.559+0000 7f16ab32e640 1 -- 172.21.15.60:0/435548250 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16a402c250 con 0x7f16a40ab670 2024-03-27T21:41:03.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.560+0000 7f16a90ca640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f16a40ab670 0x7f16a41eadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:03.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.560+0000 7f16a90ca640 10 monclient: get_auth_request con 0x7f16a40ab670 auth_method 0 2024-03-27T21:41:03.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.560+0000 7f16a90ca640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f16a40ab670 0x7f16a41eadb0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:03.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.560+0000 7f1699ffb640 10 client.4670.objecter ms_handle_connect 0x7f16a40ab670 2024-03-27T21:41:03.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.561+0000 7f1699ffb640 1 -- 172.21.15.60:0/435548250 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f16a402c250 con 0x7f16a40ab670 2024-03-27T21:41:03.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.561+0000 7f1699ffb640 10 client.4670.objecter ms_dispatch 0x7f16a4000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:41:03.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.561+0000 7f1699ffb640 10 client.4670.objecter _finish_command 10 = system:0 2024-03-27T21:41:03.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.561+0000 7f1699ffb640 20 client.4670.objecter put_session s=0x7f16a402c0e0 osd=5 4 2024-03-27T21:41:03.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.561+0000 7f1699ffb640 15 client.4670.objecter _session_command_op_remove 5 10 2024-03-27T21:41:03.551 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:41:03.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.585+0000 7f16ab32e640 10 client.4670.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:03.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.585+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign -1 11 2024-03-27T21:41:03.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.585+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a402c0e0 osd=5 3 2024-03-27T21:41:03.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.585+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a402c0e0 osd=5 3 2024-03-27T21:41:03.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.585+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a402c0e0 osd=5 3 2024-03-27T21:41:03.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.585+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_remove -1 11 2024-03-27T21:41:03.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.585+0000 7f16ab32e640 20 client.4670.objecter get_session s=0x7f16a402c0e0 osd=5 3 2024-03-27T21:41:03.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.585+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign 5 11 2024-03-27T21:41:03.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.585+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a402c0e0 osd=5 4 2024-03-27T21:41:03.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.585+0000 7f16ab32e640 10 client.4670.objecter _send_command 11 2024-03-27T21:41:03.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.585+0000 7f16ab32e640 1 -- 172.21.15.60:0/435548250 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f16a402c250 con 0x7f16a40ab670 2024-03-27T21:41:03.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.586+0000 7f1699ffb640 1 -- 172.21.15.60:0/435548250 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f16a402c250 con 0x7f16a40ab670 2024-03-27T21:41:03.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.586+0000 7f1699ffb640 10 client.4670.objecter ms_dispatch 0x7f16a4000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:03.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.586+0000 7f1699ffb640 10 client.4670.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:03.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.586+0000 7f1699ffb640 20 client.4670.objecter put_session s=0x7f16a402c0e0 osd=5 4 2024-03-27T21:41:03.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.586+0000 7f1699ffb640 15 client.4670.objecter _session_command_op_remove 5 11 2024-03-27T21:41:03.573 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:41:03.574 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:41:03.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.587+0000 7f16ab32e640 10 client.4670.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:03.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.587+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign -1 12 2024-03-27T21:41:03.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.587+0000 7f16ab32e640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f16a41eb800 0x7f16a41ebc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.587+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a402c250 osd=6 3 2024-03-27T21:41:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.587+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a402c250 osd=6 3 2024-03-27T21:41:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.587+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a402c250 osd=6 3 2024-03-27T21:41:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.587+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_remove -1 12 2024-03-27T21:41:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.587+0000 7f16ab32e640 20 client.4670.objecter get_session s=0x7f16a402c250 osd=6 3 2024-03-27T21:41:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.587+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign 6 12 2024-03-27T21:41:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.587+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a402c250 osd=6 4 2024-03-27T21:41:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.587+0000 7f16ab32e640 10 client.4670.objecter _send_command 12 2024-03-27T21:41:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.587+0000 7f16ab32e640 1 -- 172.21.15.60:0/435548250 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16a41ec1a0 con 0x7f16a41eb800 2024-03-27T21:41:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.587+0000 7f16a88c9640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f16a41eb800 0x7f16a41ebc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.587+0000 7f16a88c9640 10 monclient: get_auth_request con 0x7f16a41eb800 auth_method 0 2024-03-27T21:41:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.587+0000 7f16a88c9640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f16a41eb800 0x7f16a41ebc60 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.587+0000 7f1699ffb640 10 client.4670.objecter ms_handle_connect 0x7f16a41eb800 2024-03-27T21:41:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.588+0000 7f1699ffb640 1 -- 172.21.15.60:0/435548250 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f16a41ec1a0 con 0x7f16a41eb800 2024-03-27T21:41:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.588+0000 7f1699ffb640 10 client.4670.objecter ms_dispatch 0x7f16a4000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:41:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.588+0000 7f1699ffb640 10 client.4670.objecter _finish_command 12 = system:0 2024-03-27T21:41:03.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.588+0000 7f1699ffb640 20 client.4670.objecter put_session s=0x7f16a402c250 osd=6 4 2024-03-27T21:41:03.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.588+0000 7f1699ffb640 15 client.4670.objecter _session_command_op_remove 6 12 2024-03-27T21:41:03.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.602+0000 7f16ab32e640 10 client.4670.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:03.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.602+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign -1 13 2024-03-27T21:41:03.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.602+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a402c250 osd=6 3 2024-03-27T21:41:03.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.602+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a402c250 osd=6 3 2024-03-27T21:41:03.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.602+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a402c250 osd=6 3 2024-03-27T21:41:03.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.602+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_remove -1 13 2024-03-27T21:41:03.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.602+0000 7f16ab32e640 20 client.4670.objecter get_session s=0x7f16a402c250 osd=6 3 2024-03-27T21:41:03.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.602+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign 6 13 2024-03-27T21:41:03.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.602+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a402c250 osd=6 4 2024-03-27T21:41:03.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.602+0000 7f16ab32e640 10 client.4670.objecter _send_command 13 2024-03-27T21:41:03.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.602+0000 7f16ab32e640 1 -- 172.21.15.60:0/435548250 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f16a41ec1a0 con 0x7f16a41eb800 2024-03-27T21:41:03.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.603+0000 7f1699ffb640 1 -- 172.21.15.60:0/435548250 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f16a41ec1a0 con 0x7f16a41eb800 2024-03-27T21:41:03.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.603+0000 7f1699ffb640 10 client.4670.objecter ms_dispatch 0x7f16a4000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:03.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.603+0000 7f1699ffb640 10 client.4670.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:03.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.603+0000 7f1699ffb640 20 client.4670.objecter put_session s=0x7f16a402c250 osd=6 4 2024-03-27T21:41:03.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.603+0000 7f1699ffb640 15 client.4670.objecter _session_command_op_remove 6 13 2024-03-27T21:41:03.590 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:41:03.591 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:41:03.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.603+0000 7f16ab32e640 10 client.4670.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:03.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.603+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign -1 14 2024-03-27T21:41:03.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.603+0000 7f16ab32e640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f16a41f0b60 0x7f16a41f2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:03.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.603+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a41ec1a0 osd=7 3 2024-03-27T21:41:03.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.603+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a41ec1a0 osd=7 3 2024-03-27T21:41:03.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.603+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a41ec1a0 osd=7 3 2024-03-27T21:41:03.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.603+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_remove -1 14 2024-03-27T21:41:03.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.603+0000 7f16ab32e640 20 client.4670.objecter get_session s=0x7f16a41ec1a0 osd=7 3 2024-03-27T21:41:03.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.603+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign 7 14 2024-03-27T21:41:03.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.603+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a41ec1a0 osd=7 4 2024-03-27T21:41:03.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.603+0000 7f16ab32e640 10 client.4670.objecter _send_command 14 2024-03-27T21:41:03.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.603+0000 7f16ab32e640 1 -- 172.21.15.60:0/435548250 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16a41f5d20 con 0x7f16a41f0b60 2024-03-27T21:41:03.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.604+0000 7f16a98cb640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f16a41f0b60 0x7f16a41f2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:03.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.604+0000 7f16a98cb640 10 monclient: get_auth_request con 0x7f16a41f0b60 auth_method 0 2024-03-27T21:41:03.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.604+0000 7f1699ffb640 10 client.4670.objecter ms_handle_connect 0x7f16a41f0b60 2024-03-27T21:41:03.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.604+0000 7f16a98cb640 1 --2- 172.21.15.60:0/435548250 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f16a41f0b60 0x7f16a41f2f50 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:03.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.605+0000 7f1699ffb640 1 -- 172.21.15.60:0/435548250 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f16a41f5d20 con 0x7f16a41f0b60 2024-03-27T21:41:03.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.605+0000 7f1699ffb640 10 client.4670.objecter ms_dispatch 0x7f16a4000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:41:03.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.605+0000 7f1699ffb640 10 client.4670.objecter _finish_command 14 = system:0 2024-03-27T21:41:03.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.605+0000 7f1699ffb640 20 client.4670.objecter put_session s=0x7f16a41ec1a0 osd=7 4 2024-03-27T21:41:03.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.605+0000 7f1699ffb640 15 client.4670.objecter _session_command_op_remove 7 14 2024-03-27T21:41:03.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.625+0000 7f16ab32e640 10 client.4670.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:03.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.625+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign -1 15 2024-03-27T21:41:03.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.625+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a41ec1a0 osd=7 3 2024-03-27T21:41:03.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.625+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a41ec1a0 osd=7 3 2024-03-27T21:41:03.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.625+0000 7f16ab32e640 20 client.4670.objecter _get_session s=0x7f16a41ec1a0 osd=7 3 2024-03-27T21:41:03.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.625+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_remove -1 15 2024-03-27T21:41:03.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.625+0000 7f16ab32e640 20 client.4670.objecter get_session s=0x7f16a41ec1a0 osd=7 3 2024-03-27T21:41:03.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.625+0000 7f16ab32e640 15 client.4670.objecter _session_command_op_assign 7 15 2024-03-27T21:41:03.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.625+0000 7f16ab32e640 20 client.4670.objecter put_session s=0x7f16a41ec1a0 osd=7 4 2024-03-27T21:41:03.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.625+0000 7f16ab32e640 10 client.4670.objecter _send_command 15 2024-03-27T21:41:03.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.625+0000 7f16ab32e640 1 -- 172.21.15.60:0/435548250 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f16a41f3490 con 0x7f16a41f0b60 2024-03-27T21:41:03.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.627+0000 7f1699ffb640 1 -- 172.21.15.60:0/435548250 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f16a41f3490 con 0x7f16a41f0b60 2024-03-27T21:41:03.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.627+0000 7f1699ffb640 10 client.4670.objecter ms_dispatch 0x7f16a4000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:03.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.627+0000 7f1699ffb640 10 client.4670.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:03.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.627+0000 7f1699ffb640 20 client.4670.objecter put_session s=0x7f16a41ec1a0 osd=7 4 2024-03-27T21:41:03.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:03.627+0000 7f1699ffb640 15 client.4670.objecter _session_command_op_remove 7 15 2024-03-27T21:41:03.614 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:41:03.614 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:41:03.616 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:41:03.617 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:41:03.636 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:03.651+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:03.698 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:03.698 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:03.707 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:41:03.717 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:03.731+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:03.818 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:03.833+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:03.851 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:03.851 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:03.860 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:41:03.947 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:03.962+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:03.958 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-27T21:41:04.006 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:04.007 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:04.015 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:41:04.057 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:04.071+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:04.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.087+0000 7f2bbd851640 10 monclient: get_monmap_and_config 2024-03-27T21:41:04.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.087+0000 7f2bbd851640 10 monclient: build_initial_monmap 2024-03-27T21:41:04.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.087+0000 7f2bbd851640 10 monclient: monmap: 2024-03-27T21:41:04.074 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:04.074 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:04.074 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:04.088916+0000 2024-03-27T21:41:04.074 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:04.088916+0000 2024-03-27T21:41:04.074 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:04.074 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:04.075 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:04.075 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:04.075 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:04.075 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:04.075 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:04.075 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:04.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.088+0000 7f2bbd851640 1 Processor -- start 2024-03-27T21:41:04.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.088+0000 7f2bbd851640 1 -- start start 2024-03-27T21:41:04.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.088+0000 7f2bbd851640 10 monclient: init 2024-03-27T21:41:04.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bbd851640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:04.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bbd851640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-03-27T21:41:04.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bbd851640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2bb80b40b0 0x7f2bb80a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:04.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bbd851640 10 monclient(hunting): picked mon.noname-a con 0x7f2bb80b40b0 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:04.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bbd851640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2bb813b020 0x7f2bb813b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:04.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bbd851640 10 monclient(hunting): picked mon.noname-c con 0x7f2bb813b020 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:04.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bbd851640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2bb81416f0 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:04.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bbd851640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bb80b3830 con 0x7f2bb81416f0 2024-03-27T21:41:04.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bbd851640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:04.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bbd851640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb80b4490 con 0x7f2bb80b40b0 2024-03-27T21:41:04.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bbd851640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:04.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bbd851640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb80aa520 con 0x7f2bb813b020 2024-03-27T21:41:04.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bbd851640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:04.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bbd851640 10 monclient(hunting): authenticate will time out at 905.310216s 2024-03-27T21:41:04.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb6ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2bb80b40b0 0x7f2bb80a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:04.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb6ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2bb80b40b0 0x7f2bb80a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44746/0 (socket says 172.21.15.60:44746) 2024-03-27T21:41:04.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb6ffd640 1 -- 172.21.15.60:0/3857599244 learned_addr learned my addr 172.21.15.60:0/3857599244 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:04.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb77fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2bb81416f0 0x7f2bb8141b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50910/0 (socket says 172.21.15.60:50910) 2024-03-27T21:41:04.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb6ffd640 10 monclient(hunting): get_auth_request con 0x7f2bb80b40b0 auth_method 0 2024-03-27T21:41:04.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:04.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:04.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:04.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb67fc640 1 --2- 172.21.15.60:0/3857599244 >> v2:172.21.15.191:3300/0 conn(0x7f2bb813b020 0x7f2bb813b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:04.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb67fc640 10 monclient(hunting): get_auth_request con 0x7f2bb813b020 auth_method 0 2024-03-27T21:41:04.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:04.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb67fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:04.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:04.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:04.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:04.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:04.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:04.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:04.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:04.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb6ffd640 10 monclient(hunting): handle_auth_done global_id 4836 payload 274 2024-03-27T21:41:04.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb6ffd640 1 -- 172.21.15.60:0/3857599244 >> v2:172.21.15.191:3300/0 conn(0x7f2bb813b020 msgr2=0x7f2bb813b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:04.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb6ffd640 1 --2- 172.21.15.60:0/3857599244 >> v2:172.21.15.191:3300/0 conn(0x7f2bb813b020 0x7f2bb813b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:04.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb6ffd640 1 -- 172.21.15.60:0/3857599244 >> v1:172.21.15.60:6789/0 conn(0x7f2bb81416f0 legacy=0x7f2bb8141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:04.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb6ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:41:04.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb6ffd640 1 monclient: found mon.noname-a 2024-03-27T21:41:04.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:04.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:04.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.089+0000 7f2bb6ffd640 1 -- 172.21.15.60:0/3857599244 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb80a9470 con 0x7f2bb80b40b0 2024-03-27T21:41:04.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.090+0000 7f2bb6ffd640 1 --2- 172.21.15.60:0/3857599244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bb80b40b0 0x7f2bb80a8b90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2bb80b47e0 tx=0x7f2ba802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ae553a740195292 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:04.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.090+0000 7f2bb5ffb640 1 -- 172.21.15.60:0/3857599244 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8030860 con 0x7f2bb80b40b0 2024-03-27T21:41:04.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.090+0000 7f2bb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:04.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.090+0000 7f2bb5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:04.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.090+0000 7f2bb5ffb640 10 monclient: dump: 2024-03-27T21:41:04.087 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:04.087 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:04.087 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:04.087 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:04.087 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:04.087 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:04.087 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:04.087 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:04.087 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:04.087 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:04.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.090+0000 7f2bb5ffb640 10 monclient: _finish_auth 0 2024-03-27T21:41:04.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.090+0000 7f2bb5ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:41:04.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.090+0000 7f2bb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:04.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.090+0000 7f2bb5ffb640 1 -- 172.21.15.60:0/3857599244 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba80309c0 con 0x7f2bb80b40b0 2024-03-27T21:41:04.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.090+0000 7f2bbd851640 5 monclient: authenticate success, global_id 4836 2024-03-27T21:41:04.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.090+0000 7f2bbd851640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:04.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.090+0000 7f2bb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:04.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.090+0000 7f2bbd851640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:04.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.090+0000 7f2bb5ffb640 1 -- 172.21.15.60:0/3857599244 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8030d50 con 0x7f2bb80b40b0 2024-03-27T21:41:04.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.090+0000 7f2bb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:04.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.090+0000 7f2bbd851640 4 set_mon_vals no callback set 2024-03-27T21:41:04.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.090+0000 7f2bb5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:04.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.090+0000 7f2bb5ffb640 10 monclient: dump: 2024-03-27T21:41:04.088 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:04.089 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:04.089 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:04.089 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:04.089 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:04.089 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:04.089 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:04.089 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:04.089 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:04.089 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:04.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.091+0000 7f2bbd851640 10 monclient: shutdown 2024-03-27T21:41:04.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.091+0000 7f2bbd851640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:04.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.091+0000 7f2bbd851640 1 -- 172.21.15.60:0/3857599244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bb80b40b0 msgr2=0x7f2bb80a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:04.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.091+0000 7f2bbd851640 1 --2- 172.21.15.60:0/3857599244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bb80b40b0 0x7f2bb80a8b90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2bb80b47e0 tx=0x7f2ba802b9a0 comp rx=0 tx=0).stop 2024-03-27T21:41:04.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.091+0000 7f2bbd851640 1 -- 172.21.15.60:0/3857599244 shutdown_connections 2024-03-27T21:41:04.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.091+0000 7f2bbd851640 1 --2- 172.21.15.60:0/3857599244 >> v2:172.21.15.191:3300/0 conn(0x7f2bb813b020 0x7f2bb813b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:04.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.091+0000 7f2bbd851640 1 --2- 172.21.15.60:0/3857599244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bb80b40b0 0x7f2bb80a8b90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:04.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.091+0000 7f2bbd851640 1 -- 172.21.15.60:0/3857599244 >> 172.21.15.60:0/3857599244 conn(0x7f2bb80608e0 msgr2=0x7f2bb8060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:04.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.091+0000 7f2bbd851640 1 -- 172.21.15.60:0/3857599244 shutdown_connections 2024-03-27T21:41:04.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.091+0000 7f2bbd851640 1 -- 172.21.15.60:0/3857599244 wait complete. 2024-03-27T21:41:04.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.091+0000 7f2bbd851640 10 monclient: build_initial_monmap 2024-03-27T21:41:04.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.091+0000 7f2bbd851640 10 monclient: monmap: 2024-03-27T21:41:04.090 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:04.090 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:04.090 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:04.090 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:04.090 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:04.090 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:04.090 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:04.090 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:04.090 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:04.091 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:04.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 1 librados: starting msgr at 2024-03-27T21:41:04.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 1 librados: starting objecter 2024-03-27T21:41:04.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 1 Processor -- start 2024-03-27T21:41:04.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 1 -- start start 2024-03-27T21:41:04.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 1 librados: setting wanted keys 2024-03-27T21:41:04.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 1 librados: calling monclient init 2024-03-27T21:41:04.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 10 monclient: init 2024-03-27T21:41:04.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:04.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:41:04.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bb80b40b0 0x7f2bb80aa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:04.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 10 monclient(hunting): picked mon.noname-a con 0x7f2bb80b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:04.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2bb813b020 0x7f2bb80aaf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:04.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 10 monclient(hunting): picked mon.noname-b con 0x7f2bb813b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:04.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2bb81416f0 0x7f2bb80afc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:04.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 10 monclient(hunting): picked mon.noname-c con 0x7f2bb81416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:04.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:04.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb80ac500 con 0x7f2bb80b40b0 2024-03-27T21:41:04.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:04.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb813e460 con 0x7f2bb813b020 2024-03-27T21:41:04.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:04.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb813e760 con 0x7f2bb81416f0 2024-03-27T21:41:04.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:04.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.092+0000 7f2bbd851640 10 monclient(hunting): authenticate will time out at 905.313902s 2024-03-27T21:41:04.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bb80b40b0 0x7f2bb80aa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:04.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bb80b40b0 0x7f2bb80aa9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44760/0 (socket says 172.21.15.60:44760) 2024-03-27T21:41:04.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb6ffd640 1 -- 172.21.15.60:0/3424533624 learned_addr learned my addr 172.21.15.60:0/3424533624 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:04.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb6ffd640 10 monclient(hunting): get_auth_request con 0x7f2bb80b40b0 auth_method 0 2024-03-27T21:41:04.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:04.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:04.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:04.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb77fe640 1 --2- 172.21.15.60:0/3424533624 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2bb81416f0 0x7f2bb80afc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:04.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb77fe640 10 monclient(hunting): get_auth_request con 0x7f2bb81416f0 auth_method 0 2024-03-27T21:41:04.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:04.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:04.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:04.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:04.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:04.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:04.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:04.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:04.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:04.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb67fc640 1 --2- 172.21.15.60:0/3424533624 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2bb813b020 0x7f2bb80aaf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:04.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb67fc640 10 monclient(hunting): get_auth_request con 0x7f2bb813b020 auth_method 0 2024-03-27T21:41:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb67fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb6ffd640 10 monclient(hunting): handle_auth_done global_id 4839 payload 824 2024-03-27T21:41:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb6ffd640 1 -- 172.21.15.60:0/3424533624 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2bb81416f0 msgr2=0x7f2bb80afc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb6ffd640 1 --2- 172.21.15.60:0/3424533624 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2bb81416f0 0x7f2bb80afc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb6ffd640 1 -- 172.21.15.60:0/3424533624 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2bb813b020 msgr2=0x7f2bb80aaf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb6ffd640 1 --2- 172.21.15.60:0/3424533624 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2bb813b020 0x7f2bb80aaf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb6ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:41:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb6ffd640 1 monclient: found mon.noname-a 2024-03-27T21:41:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb6ffd640 1 -- 172.21.15.60:0/3424533624 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba80094d0 con 0x7f2bb80b40b0 2024-03-27T21:41:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb67fc640 1 --2- 172.21.15.60:0/3424533624 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2bb813b020 0x7f2bb80aaf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:41:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb6ffd640 1 --2- 172.21.15.60:0/3424533624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bb80b40b0 0x7f2bb80aa9d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2ba802b9e0 tx=0x7f2ba8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2b9ffff640 10 client.?.objecter ms_handle_connect 0x7f2bb80b40b0 2024-03-27T21:41:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2b9ffff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2b9ffff640 1 -- 172.21.15.60:0/3424533624 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8039c90 con 0x7f2bb80b40b0 2024-03-27T21:41:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2b9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2b9ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2b9ffff640 10 monclient: dump: 2024-03-27T21:41:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:04.096 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:04.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2b9ffff640 10 monclient: _finish_auth 0 2024-03-27T21:41:04.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2b9ffff640 10 monclient: _check_auth_tickets 2024-03-27T21:41:04.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2b9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:04.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bbd851640 5 monclient: authenticate success, global_id 4839 2024-03-27T21:41:04.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2b9ffff640 1 -- 172.21.15.60:0/3424533624 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba8039e10 con 0x7f2bb80b40b0 2024-03-27T21:41:04.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bbd851640 10 monclient: _renew_subs 2024-03-27T21:41:04.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bbd851640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:04.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bbd851640 1 -- 172.21.15.60:0/3424533624 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bb813e980 con 0x7f2bb80b40b0 2024-03-27T21:41:04.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2b9ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:04.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2b9ffff640 1 -- 172.21.15.60:0/3424533624 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8043640 con 0x7f2bb80b40b0 2024-03-27T21:41:04.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2b9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:04.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2b9ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:04.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2b9ffff640 10 monclient: dump: 2024-03-27T21:41:04.097 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:04.097 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:04.097 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:04.097 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:04.097 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:04.097 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:04.097 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:04.097 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:04.097 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:04.097 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:04.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bbd851640 10 client.4839.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:04.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bbd851640 10 monclient: _renew_subs 2024-03-27T21:41:04.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bb57fa640 4 set_mon_vals no callback set 2024-03-27T21:41:04.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bbd851640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:04.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bbd851640 1 -- 172.21.15.60:0/3424533624 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb80b0330 con 0x7f2bb80b40b0 2024-03-27T21:41:04.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.093+0000 7f2bbd851640 1 librados: init done 2024-03-27T21:41:04.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.094+0000 7f2bbd851640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:04.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.094+0000 7f2bbd851640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:04.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.094+0000 7f2bbd851640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:04.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.094+0000 7f2bbd851640 1 -- 172.21.15.60:0/3424533624 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bb8069470 con 0x7f2bb80b40b0 2024-03-27T21:41:04.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.095+0000 7f2b9ffff640 1 -- 172.21.15.60:0/3424533624 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2ba80437a0 con 0x7f2bb80b40b0 2024-03-27T21:41:04.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.095+0000 7f2b9ffff640 1 --2- 172.21.15.60:0/3424533624 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2ba406c590 0x7f2ba406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:04.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.096+0000 7f2b9ffff640 1 -- 172.21.15.60:0/3424533624 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f2ba803e070 con 0x7f2bb80b40b0 2024-03-27T21:41:04.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.096+0000 7f2b9ffff640 10 client.4839.objecter ms_dispatch 0x7f2bb8000b90 osd_map(50..50 src has 1..50) v4 2024-03-27T21:41:04.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.096+0000 7f2b9ffff640 3 client.4839.objecter handle_osd_map got epochs [50,50] > 0 2024-03-27T21:41:04.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.096+0000 7f2b9ffff640 3 client.4839.objecter handle_osd_map decoding full epoch 50 2024-03-27T21:41:04.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.096+0000 7f2bb67fc640 1 --2- 172.21.15.60:0/3424533624 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2ba406c590 0x7f2ba406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:04.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.096+0000 7f2bb67fc640 10 monclient: get_auth_request con 0x7f2ba406c590 auth_method 0 2024-03-27T21:41:04.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.096+0000 7f2b9ffff640 20 client.4839.objecter dump_active .. 0 homeless 2024-03-27T21:41:04.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.096+0000 7f2b9ffff640 10 client.4839.objecter ms_handle_connect 0x7f2ba406c590 2024-03-27T21:41:04.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.096+0000 7f2bb67fc640 1 --2- 172.21.15.60:0/3424533624 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2ba406c590 0x7f2ba406ea50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2bb813ae50 tx=0x7f2bac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:04.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.101+0000 7f2b9ffff640 1 -- 172.21.15.60:0/3424533624 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2ba80df580 con 0x7f2bb80b40b0 2024-03-27T21:41:04.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.101+0000 7f2b9ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:04.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.101+0000 7f2b9ffff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:04.120 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:04.135+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:04.162 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:04.162 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:04.170 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:41:04.255 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:04.270+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:04.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.293+0000 7f2bbd851640 10 start_mon_command cmd=[{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:41:04.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.293+0000 7f2bbd851640 10 monclient: _send_command 2 [{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:41:04.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.293+0000 7f2bbd851640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:04.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.293+0000 7f2bbd851640 1 -- 172.21.15.60:0/3424533624 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f2bb8069e40 con 0x7f2bb80b40b0 2024-03-27T21:41:04.315 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:04.316 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:04.322 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:04.336+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:04.324 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:41:04.423 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:04.437+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:04.469 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:04.469 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:04.478 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:41:04.524 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:04.539+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:04.583 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:04.583 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:04.592 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:41:04.636 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:04.650+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:04.703 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:04.703 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:04.712 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:41:04.726 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:04.741+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:04.827 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:04.842+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:04.882 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:04.883 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:04.892 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:41:04.935 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:04.949+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.962+0000 7f2b9ffff640 1 -- 172.21.15.60:0/3424533624 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v51) v1 ==== 87+0+0 (secure 0 0 0) 0x7f2ba80776a0 con 0x7f2bb80b40b0 2024-03-27T21:41:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.962+0000 7f2b9ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:41:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.962+0000 7f2b9ffff640 10 monclient: _finish_command 2 = system:0 noscrub is set 2024-03-27T21:41:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:noscrub is set 2024-03-27T21:41:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.964+0000 7f2bbd851640 10 librados: watch_flush enter 2024-03-27T21:41:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.964+0000 7f2bbd851640 10 librados: watch_flush exit 2024-03-27T21:41:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.964+0000 7f2bbd851640 20 client.4839.objecter shutdown clearing up homeless session... 2024-03-27T21:41:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.964+0000 7f2bbd851640 10 client.4839.objecter successfully canceled tick 2024-03-27T21:41:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.964+0000 7f2bbd851640 1 -- 172.21.15.60:0/3424533624 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2ba406c590 msgr2=0x7f2ba406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.964+0000 7f2bbd851640 1 --2- 172.21.15.60:0/3424533624 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2ba406c590 0x7f2ba406ea50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2bb813ae50 tx=0x7f2bac008040 comp rx=0 tx=0).stop 2024-03-27T21:41:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.965+0000 7f2bbd851640 10 monclient: shutdown 2024-03-27T21:41:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.965+0000 7f2bbd851640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.965+0000 7f2bbd851640 1 -- 172.21.15.60:0/3424533624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bb80b40b0 msgr2=0x7f2bb80aa9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.965+0000 7f2bbd851640 1 --2- 172.21.15.60:0/3424533624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bb80b40b0 0x7f2bb80aa9d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2ba802b9e0 tx=0x7f2ba8004730 comp rx=0 tx=0).stop 2024-03-27T21:41:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.965+0000 7f2bbd851640 1 -- 172.21.15.60:0/3424533624 shutdown_connections 2024-03-27T21:41:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.965+0000 7f2bbd851640 1 --2- 172.21.15.60:0/3424533624 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2bb81416f0 0x7f2bb80afc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.965+0000 7f2bbd851640 1 --2- 172.21.15.60:0/3424533624 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2ba406c590 0x7f2ba406ea50 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.965+0000 7f2bbd851640 1 --2- 172.21.15.60:0/3424533624 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2bb813b020 0x7f2bb80aaf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.965+0000 7f2bbd851640 1 --2- 172.21.15.60:0/3424533624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bb80b40b0 0x7f2bb80aa9d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.965+0000 7f2bbd851640 1 -- 172.21.15.60:0/3424533624 >> 172.21.15.60:0/3424533624 conn(0x7f2bb80608e0 msgr2=0x7f2bb813f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.965+0000 7f2bbd851640 1 -- 172.21.15.60:0/3424533624 shutdown_connections 2024-03-27T21:41:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.965+0000 7f2bbd851640 1 -- 172.21.15.60:0/3424533624 wait complete. 2024-03-27T21:41:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:04.965+0000 7f2bbd851640 1 librados: shutdown 2024-03-27T21:41:05.009 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:05.009 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:05.019 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:41:05.063 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:05.077+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:05.133 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:05.133 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:05.133 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:05.148+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:05.141 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:41:05.234 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:05.248+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:05.292 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:05.292 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:05.302 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:41:05.338 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:05.353+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:05.436 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-27T21:41:05.436 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T21:41:05.436 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-03-27T21:41:05.436 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 3.0 2024-03-27T21:41:05.439 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:05.454+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:05.456 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:05.456 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:05.465 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:41:05.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.557+0000 7f7044700640 10 monclient: get_monmap_and_config 2024-03-27T21:41:05.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.557+0000 7f7044700640 10 monclient: build_initial_monmap 2024-03-27T21:41:05.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.557+0000 7f7044700640 10 monclient: monmap: 2024-03-27T21:41:05.543 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:05.543 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:05.544 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:05.558065+0000 2024-03-27T21:41:05.544 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:05.558065+0000 2024-03-27T21:41:05.544 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:05.544 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:05.544 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:05.544 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:05.544 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:05.544 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:05.544 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:05.544 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:05.545 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:05.557+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:05.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.557+0000 7f7044700640 1 Processor -- start 2024-03-27T21:41:05.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.557+0000 7f7044700640 1 -- start start 2024-03-27T21:41:05.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.557+0000 7f7044700640 10 monclient: init 2024-03-27T21:41:05.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7044700640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:05.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7044700640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-03-27T21:41:05.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7044700640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f703c071a50 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:05.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7044700640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f703c13ef20 0x7f703c0a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7044700640 10 monclient(hunting): picked mon.noname-c con 0x7f703c13ef20 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:05.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7044700640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f703c141630 0x7f703c141ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7044700640 10 monclient(hunting): picked mon.noname-b con 0x7f703c141630 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:05.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7044700640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703c0a8e90 con 0x7f703c071a50 2024-03-27T21:41:05.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7044700640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:05.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7044700640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f703c071e30 con 0x7f703c13ef20 2024-03-27T21:41:05.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7044700640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:05.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7044700640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f703c0aa040 con 0x7f703c141630 2024-03-27T21:41:05.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7044700640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:05.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7044700640 10 monclient(hunting): authenticate will time out at 906.779505s 2024-03-27T21:41:05.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7042c9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f703c141630 0x7f703c141ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7042c9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f703c141630 0x7f703c141ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:39988/0 (socket says 172.21.15.60:39988) 2024-03-27T21:41:05.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7042c9d640 1 -- 172.21.15.60:0/4231762225 learned_addr learned my addr 172.21.15.60:0/4231762225 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:05.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7042c9d640 10 monclient(hunting): get_auth_request con 0x7f703c141630 auth_method 0 2024-03-27T21:41:05.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7042c9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:05.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7042c9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:05.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7042c9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:05.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7041c9b640 1 --2- 172.21.15.60:0/4231762225 >> v2:172.21.15.191:3300/0 conn(0x7f703c13ef20 0x7f703c0a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7041c9b640 10 monclient(hunting): get_auth_request con 0x7f703c13ef20 auth_method 0 2024-03-27T21:41:05.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7041c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:05.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7041c9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:05.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7041c9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:05.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7042c9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:05.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7042c9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:05.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7042c9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:05.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7041c9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:05.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.558+0000 7f7041c9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:05.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.559+0000 7f7041c9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:05.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.559+0000 7f7042c9d640 10 monclient(hunting): handle_auth_done global_id 4676 payload 274 2024-03-27T21:41:05.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.559+0000 7f7042c9d640 1 -- 172.21.15.60:0/4231762225 >> v2:172.21.15.191:3300/0 conn(0x7f703c13ef20 msgr2=0x7f703c0a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:05.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.559+0000 7f7042c9d640 1 --2- 172.21.15.60:0/4231762225 >> v2:172.21.15.191:3300/0 conn(0x7f703c13ef20 0x7f703c0a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:05.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.559+0000 7f7042c9d640 1 -- 172.21.15.60:0/4231762225 >> v1:172.21.15.191:6789/0 conn(0x7f703c071a50 legacy=0x7f703c13e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:05.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.559+0000 7f7042c9d640 10 monclient: _finish_hunting 0 2024-03-27T21:41:05.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.559+0000 7f7042c9d640 1 monclient: found mon.noname-b 2024-03-27T21:41:05.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.559+0000 7f7042c9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:05.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.559+0000 7f7042c9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:05.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.559+0000 7f7042c9d640 1 -- 172.21.15.60:0/4231762225 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703c0aa2b0 con 0x7f703c141630 2024-03-27T21:41:05.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.560+0000 7f7042c9d640 1 --2- 172.21.15.60:0/4231762225 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f703c141630 0x7f703c141ac0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7034009e50 tx=0x7f703402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4b605d1c19b9fcb server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:05.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.560+0000 7f704149a640 1 -- 172.21.15.60:0/4231762225 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703402cb00 con 0x7f703c141630 2024-03-27T21:41:05.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.560+0000 7f704149a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:05.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.560+0000 7f704149a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:05.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.560+0000 7f704149a640 10 monclient: dump: 2024-03-27T21:41:05.556 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:05.556 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:05.557 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.557 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.557 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:05.557 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:05.557 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:05.557 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:05.557 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:05.557 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:05.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.560+0000 7f704149a640 10 monclient: _finish_auth 0 2024-03-27T21:41:05.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.560+0000 7f704149a640 10 monclient: _check_auth_tickets 2024-03-27T21:41:05.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.560+0000 7f704149a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:05.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.560+0000 7f704149a640 1 -- 172.21.15.60:0/4231762225 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703402cc60 con 0x7f703c141630 2024-03-27T21:41:05.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.560+0000 7f704149a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:05.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.560+0000 7f704149a640 1 -- 172.21.15.60:0/4231762225 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70340343f0 con 0x7f703c141630 2024-03-27T21:41:05.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.560+0000 7f7044700640 5 monclient: authenticate success, global_id 4676 2024-03-27T21:41:05.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.560+0000 7f7044700640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:05.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.560+0000 7f7044700640 4 set_mon_vals no callback set 2024-03-27T21:41:05.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.560+0000 7f704149a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:05.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.560+0000 7f704149a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:05.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.560+0000 7f704149a640 10 monclient: dump: 2024-03-27T21:41:05.558 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:05.558 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:05.558 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.558 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.558 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:05.558 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:05.558 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:05.558 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:05.559 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:05.559 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:05.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.560+0000 7f7044700640 10 monclient: shutdown 2024-03-27T21:41:05.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.560+0000 7f7044700640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:05.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.561+0000 7f7044700640 1 -- 172.21.15.60:0/4231762225 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f703c141630 msgr2=0x7f703c141ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:05.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.561+0000 7f7044700640 1 --2- 172.21.15.60:0/4231762225 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f703c141630 0x7f703c141ac0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7034009e50 tx=0x7f703402bf60 comp rx=0 tx=0).stop 2024-03-27T21:41:05.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.561+0000 7f7044700640 1 -- 172.21.15.60:0/4231762225 shutdown_connections 2024-03-27T21:41:05.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.561+0000 7f7044700640 1 --2- 172.21.15.60:0/4231762225 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f703c141630 0x7f703c141ac0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:05.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.561+0000 7f7044700640 1 --2- 172.21.15.60:0/4231762225 >> v2:172.21.15.191:3300/0 conn(0x7f703c13ef20 0x7f703c0a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:05.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.561+0000 7f7044700640 1 -- 172.21.15.60:0/4231762225 >> 172.21.15.60:0/4231762225 conn(0x7f703c0b7300 msgr2=0x7f703c0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:05.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.561+0000 7f7044700640 1 -- 172.21.15.60:0/4231762225 shutdown_connections 2024-03-27T21:41:05.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.561+0000 7f7044700640 1 -- 172.21.15.60:0/4231762225 wait complete. 2024-03-27T21:41:05.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.561+0000 7f7044700640 10 monclient: build_initial_monmap 2024-03-27T21:41:05.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.561+0000 7f7044700640 10 monclient: monmap: 2024-03-27T21:41:05.559 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:05.560 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:05.560 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:05.560 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:05.560 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:05.560 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:05.560 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:05.560 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:05.560 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:05.560 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:05.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 1 librados: starting msgr at 2024-03-27T21:41:05.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 1 librados: starting objecter 2024-03-27T21:41:05.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 1 Processor -- start 2024-03-27T21:41:05.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 1 -- start start 2024-03-27T21:41:05.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 1 librados: setting wanted keys 2024-03-27T21:41:05.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 1 librados: calling monclient init 2024-03-27T21:41:05.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 10 monclient: init 2024-03-27T21:41:05.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:05.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:41:05.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f703c071a50 0x7f703c1361f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 10 monclient(hunting): picked mon.noname-b con 0x7f703c071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:05.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f703c13ef20 0x7f703c138ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 10 monclient(hunting): picked mon.noname-c con 0x7f703c13ef20 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:05.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f703c141630 0x7f703c1396a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 10 monclient(hunting): picked mon.noname-a con 0x7f703c141630 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:05.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:05.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703c139100 con 0x7f703c141630 2024-03-27T21:41:05.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:05.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703c13ac60 con 0x7f703c071a50 2024-03-27T21:41:05.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:05.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f703c13aed0 con 0x7f703c13ef20 2024-03-27T21:41:05.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:05.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7044700640 10 monclient(hunting): authenticate will time out at 906.783751s 2024-03-27T21:41:05.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7041c9b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f703c13ef20 0x7f703c138ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7042c9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f703c141630 0x7f703c1396a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7041c9b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f703c13ef20 0x7f703c138ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:39994/0 (socket says 172.21.15.60:39994) 2024-03-27T21:41:05.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7041c9b640 1 -- 172.21.15.60:0/825085854 learned_addr learned my addr 172.21.15.60:0/825085854 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:05.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7041c9b640 10 monclient(hunting): get_auth_request con 0x7f703c13ef20 auth_method 0 2024-03-27T21:41:05.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7041c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:05.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7041c9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:05.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.562+0000 7f7041c9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:05.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7042c9d640 10 monclient(hunting): get_auth_request con 0x7f703c141630 auth_method 0 2024-03-27T21:41:05.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7042c9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:05.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7042c9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:05.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7042c9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:05.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7041c9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:05.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7041c9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:05.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7041c9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:05.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7042c9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:05.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7042c9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:05.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7042c9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:05.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f704249c640 1 --2- 172.21.15.60:0/825085854 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f703c071a50 0x7f703c1361f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f704249c640 10 monclient(hunting): get_auth_request con 0x7f703c071a50 auth_method 0 2024-03-27T21:41:05.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f704249c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:05.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f704249c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:05.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f704249c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:05.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f704249c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:05.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f704249c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:05.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f704249c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:05.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7041c9b640 10 monclient(hunting): handle_auth_done global_id 4679 payload 824 2024-03-27T21:41:05.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7041c9b640 1 -- 172.21.15.60:0/825085854 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f703c071a50 msgr2=0x7f703c1361f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:05.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7041c9b640 1 --2- 172.21.15.60:0/825085854 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f703c071a50 0x7f703c1361f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:05.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7041c9b640 1 -- 172.21.15.60:0/825085854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f703c141630 msgr2=0x7f703c1396a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:05.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7041c9b640 1 --2- 172.21.15.60:0/825085854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f703c141630 0x7f703c1396a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:05.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7041c9b640 10 monclient: _finish_hunting 0 2024-03-27T21:41:05.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7041c9b640 1 monclient: found mon.noname-c 2024-03-27T21:41:05.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7041c9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:05.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7041c9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:05.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7041c9b640 1 -- 172.21.15.60:0/825085854 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7034009b30 con 0x7f703c13ef20 2024-03-27T21:41:05.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7042c9d640 1 --2- 172.21.15.60:0/825085854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f703c141630 0x7f703c1396a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:05.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7041c9b640 1 --2- 172.21.15.60:0/825085854 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f703c13ef20 0x7f703c138ae0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f702c00e8b0 tx=0x7f702c00ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:05.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7032ffd640 10 client.?.objecter ms_handle_connect 0x7f703c13ef20 2024-03-27T21:41:05.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7032ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:05.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7032ffd640 1 -- 172.21.15.60:0/825085854 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c009610 con 0x7f703c13ef20 2024-03-27T21:41:05.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7032ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:05.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7032ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:05.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7032ffd640 10 monclient: dump: 2024-03-27T21:41:05.565 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:05.565 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:05.565 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.565 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.565 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:05.565 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:05.565 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:05.565 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:05.565 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:05.565 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:05.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7032ffd640 10 monclient: _finish_auth 0 2024-03-27T21:41:05.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7032ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:41:05.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7032ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:05.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7032ffd640 1 -- 172.21.15.60:0/825085854 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f702c00c940 con 0x7f703c13ef20 2024-03-27T21:41:05.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7044700640 5 monclient: authenticate success, global_id 4679 2024-03-27T21:41:05.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7044700640 10 monclient: _renew_subs 2024-03-27T21:41:05.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7044700640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:05.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7044700640 1 -- 172.21.15.60:0/825085854 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f703c136b80 con 0x7f703c13ef20 2024-03-27T21:41:05.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7032ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:05.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7032ffd640 1 -- 172.21.15.60:0/825085854 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c00ccf0 con 0x7f703c13ef20 2024-03-27T21:41:05.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7032ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:05.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7032ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:05.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7032ffd640 10 monclient: dump: 2024-03-27T21:41:05.566 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:05.566 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:05.566 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.566 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.566 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:05.567 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:05.567 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:05.567 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:05.567 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:05.567 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:05.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7044700640 10 client.4679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:05.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7044700640 10 monclient: _renew_subs 2024-03-27T21:41:05.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7044700640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:05.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7044700640 1 -- 172.21.15.60:0/825085854 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f703c139be0 con 0x7f703c13ef20 2024-03-27T21:41:05.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7044700640 1 librados: init done 2024-03-27T21:41:05.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.563+0000 7f7040c99640 4 set_mon_vals no callback set 2024-03-27T21:41:05.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.565+0000 7f7032ffd640 1 -- 172.21.15.60:0/825085854 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f702c013470 con 0x7f703c13ef20 2024-03-27T21:41:05.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.565+0000 7f7044700640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:05.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.565+0000 7f7044700640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:05.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.565+0000 7f7044700640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:05.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.565+0000 7f7044700640 1 -- 172.21.15.60:0/825085854 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f703c067900 con 0x7f703c13ef20 2024-03-27T21:41:05.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.565+0000 7f7032ffd640 1 --2- 172.21.15.60:0/825085854 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f702006c590 0x7f702006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.565+0000 7f7032ffd640 1 -- 172.21.15.60:0/825085854 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f702c089320 con 0x7f703c13ef20 2024-03-27T21:41:05.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.565+0000 7f7032ffd640 10 client.4679.objecter ms_dispatch 0x7f703c000b90 osd_map(51..51 src has 1..51) v4 2024-03-27T21:41:05.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.565+0000 7f7032ffd640 3 client.4679.objecter handle_osd_map got epochs [51,51] > 0 2024-03-27T21:41:05.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.565+0000 7f7032ffd640 3 client.4679.objecter handle_osd_map decoding full epoch 51 2024-03-27T21:41:05.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.565+0000 7f704249c640 1 --2- 172.21.15.60:0/825085854 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f702006c590 0x7f702006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.565+0000 7f704249c640 10 monclient: get_auth_request con 0x7f702006c590 auth_method 0 2024-03-27T21:41:05.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.565+0000 7f7032ffd640 20 client.4679.objecter dump_active .. 0 homeless 2024-03-27T21:41:05.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.566+0000 7f7032ffd640 10 client.4679.objecter ms_handle_connect 0x7f702006c590 2024-03-27T21:41:05.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.566+0000 7f704249c640 1 --2- 172.21.15.60:0/825085854 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f702006c590 0x7f702006ea50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7038000f60 tx=0x7f7038009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:05.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.574+0000 7f7032ffd640 1 -- 172.21.15.60:0/825085854 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f702c054770 con 0x7f703c13ef20 2024-03-27T21:41:05.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.574+0000 7f7032ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:05.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.574+0000 7f7032ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:05.618 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:41:05.630 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:05.630 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:05.641 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:41:05.683 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:05.697+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.736+0000 7f7215342640 10 monclient: get_monmap_and_config 2024-03-27T21:41:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.736+0000 7f7215342640 10 monclient: build_initial_monmap 2024-03-27T21:41:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.736+0000 7f7215342640 10 monclient: monmap: 2024-03-27T21:41:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:05.737283+0000 2024-03-27T21:41:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:05.737283+0000 2024-03-27T21:41:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:05.723 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:05.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.736+0000 7f7215342640 1 Processor -- start 2024-03-27T21:41:05.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.737+0000 7f7215342640 1 -- start start 2024-03-27T21:41:05.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.737+0000 7f7215342640 10 monclient: init 2024-03-27T21:41:05.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.737+0000 7f7215342640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:05.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.737+0000 7f7215342640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-03-27T21:41:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.737+0000 7f7215342640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f72100b40b0 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.737+0000 7f7215342640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f721013b020 0x7f721013b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.737+0000 7f7215342640 10 monclient(hunting): picked mon.noname-c con 0x7f721013b020 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.737+0000 7f7215342640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f72101416f0 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.737+0000 7f7215342640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72100b3830 con 0x7f72101416f0 2024-03-27T21:41:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.737+0000 7f7215342640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72100aa520 con 0x7f72100b40b0 2024-03-27T21:41:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.737+0000 7f7215342640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.737+0000 7f7215342640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72100b4490 con 0x7f721013b020 2024-03-27T21:41:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.737+0000 7f7215342640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.737+0000 7f7215342640 10 monclient(hunting): authenticate will time out at 906.958618s 2024-03-27T21:41:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.737+0000 7f720f7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f72101416f0 0x7f7210141b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50918/0 (socket says 172.21.15.60:50918) 2024-03-27T21:41:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.737+0000 7f720f7fe640 1 -- 172.21.15.60:0/354041458 learned_addr learned my addr 172.21.15.60:0/354041458 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720e7fc640 1 --2- 172.21.15.60:0/354041458 >> v2:172.21.15.191:3300/0 conn(0x7f721013b020 0x7f721013b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720e7fc640 10 monclient(hunting): get_auth_request con 0x7f721013b020 auth_method 0 2024-03-27T21:41:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720dffb640 1 -- 172.21.15.60:0/354041458 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624719573 0 0) 0x7f72100b3830 con 0x7f72101416f0 2024-03-27T21:41:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720dffb640 10 monclient(hunting): _negotiate 2024-03-27T21:41:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720dffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720dffb640 10 monclient(hunting): my global_id is 4845 2024-03-27T21:41:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720dffb640 1 -- 172.21.15.60:0/354041458 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71f8003520 con 0x7f72101416f0 2024-03-27T21:41:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720e7fc640 10 monclient(hunting): handle_auth_done global_id 4789 payload 274 2024-03-27T21:41:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720e7fc640 1 -- 172.21.15.60:0/354041458 >> v1:172.21.15.191:6789/0 conn(0x7f72100b40b0 legacy=0x7f72100a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720dffb640 1 -- 172.21.15.60:0/354041458 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401539328 0 0) 0x7f72100aa520 con 0x7f72100b40b0 2024-03-27T21:41:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720e7fc640 1 -- 172.21.15.60:0/354041458 >> v1:172.21.15.60:6789/0 conn(0x7f72101416f0 legacy=0x7f7210141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720e7fc640 10 monclient: _finish_hunting 0 2024-03-27T21:41:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720e7fc640 1 monclient: found mon.noname-c 2024-03-27T21:41:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:41:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720e7fc640 1 -- 172.21.15.60:0/354041458 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72100a9470 con 0x7f721013b020 2024-03-27T21:41:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.738+0000 7f720dffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:41:05.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.740+0000 7f720e7fc640 1 --2- 172.21.15.60:0/354041458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f721013b020 0x7f721013b480 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f72040097a0 tx=0x7f720402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6cd872b4397c1e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:05.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.740+0000 7f720dffb640 1 -- 172.21.15.60:0/354041458 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7204030610 con 0x7f721013b020 2024-03-27T21:41:05.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.740+0000 7f720dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:05.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.740+0000 7f720dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:05.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.740+0000 7f720dffb640 10 monclient: dump: 2024-03-27T21:41:05.737 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:05.737 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:05.737 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.737 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.737 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:05.737 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:05.737 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:05.737 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:05.737 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:05.737 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:05.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.740+0000 7f720dffb640 10 monclient: _finish_auth 0 2024-03-27T21:41:05.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.740+0000 7f720dffb640 10 monclient: _check_auth_tickets 2024-03-27T21:41:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.740+0000 7f720dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.740+0000 7f720dffb640 1 -- 172.21.15.60:0/354041458 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7204030770 con 0x7f721013b020 2024-03-27T21:41:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.740+0000 7f720dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.740+0000 7f720dffb640 1 -- 172.21.15.60:0/354041458 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7204030b00 con 0x7f721013b020 2024-03-27T21:41:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.740+0000 7f720dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.740+0000 7f720dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.740+0000 7f720dffb640 10 monclient: dump: 2024-03-27T21:41:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:05.739 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:05.739 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:05.739 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:05.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.740+0000 7f7215342640 5 monclient: authenticate success, global_id 4789 2024-03-27T21:41:05.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.740+0000 7f7215342640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:05.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.740+0000 7f7215342640 4 set_mon_vals no callback set 2024-03-27T21:41:05.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.740+0000 7f7215342640 10 monclient: shutdown 2024-03-27T21:41:05.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.740+0000 7f7215342640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:05.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.740+0000 7f7215342640 1 -- 172.21.15.60:0/354041458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f721013b020 msgr2=0x7f721013b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:05.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.740+0000 7f7215342640 1 --2- 172.21.15.60:0/354041458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f721013b020 0x7f721013b480 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f72040097a0 tx=0x7f720402b960 comp rx=0 tx=0).stop 2024-03-27T21:41:05.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.741+0000 7f7215342640 1 -- 172.21.15.60:0/354041458 shutdown_connections 2024-03-27T21:41:05.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.741+0000 7f7215342640 1 --2- 172.21.15.60:0/354041458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f721013b020 0x7f721013b480 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:05.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.741+0000 7f7215342640 1 -- 172.21.15.60:0/354041458 >> 172.21.15.60:0/354041458 conn(0x7f72100608e0 msgr2=0x7f7210060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:05.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.741+0000 7f7215342640 1 -- 172.21.15.60:0/354041458 shutdown_connections 2024-03-27T21:41:05.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.741+0000 7f7215342640 1 -- 172.21.15.60:0/354041458 wait complete. 2024-03-27T21:41:05.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.741+0000 7f7215342640 10 monclient: build_initial_monmap 2024-03-27T21:41:05.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.741+0000 7f7215342640 10 monclient: monmap: 2024-03-27T21:41:05.740 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:05.740 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:05.740 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:05.740 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:05.740 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:05.740 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:05.740 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:05.740 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:05.741 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:05.741 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:05.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.742+0000 7f7215342640 1 librados: starting msgr at 2024-03-27T21:41:05.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.742+0000 7f7215342640 1 librados: starting objecter 2024-03-27T21:41:05.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.742+0000 7f7215342640 1 Processor -- start 2024-03-27T21:41:05.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.742+0000 7f7215342640 1 -- start start 2024-03-27T21:41:05.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.742+0000 7f7215342640 1 librados: setting wanted keys 2024-03-27T21:41:05.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.742+0000 7f7215342640 1 librados: calling monclient init 2024-03-27T21:41:05.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.742+0000 7f7215342640 10 monclient: init 2024-03-27T21:41:05.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f7215342640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:05.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f7215342640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:41:05.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f7215342640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f72100b40b0 0x7f7210138660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f7215342640 10 monclient(hunting): picked mon.noname-c con 0x7f72100b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:05.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f7215342640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f721013b020 0x7f72100aada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f7215342640 10 monclient(hunting): picked mon.noname-a con 0x7f721013b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:05.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f7215342640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72101416f0 0x7f72101d6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f7215342640 10 monclient(hunting): picked mon.noname-b con 0x7f72101416f0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:05.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f7215342640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:05.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f7215342640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72101d6540 con 0x7f721013b020 2024-03-27T21:41:05.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f7215342640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:05.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f7215342640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72100ab2e0 con 0x7f72101416f0 2024-03-27T21:41:05.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f7215342640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:05.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f7215342640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721013a440 con 0x7f72100b40b0 2024-03-27T21:41:05.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f7215342640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:05.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f7215342640 10 monclient(hunting): authenticate will time out at 906.964395s 2024-03-27T21:41:05.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720e7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f721013b020 0x7f72100aada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720effd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f72100b40b0 0x7f7210138660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720e7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f721013b020 0x7f72100aada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44776/0 (socket says 172.21.15.60:44776) 2024-03-27T21:41:05.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720e7fc640 1 -- 172.21.15.60:0/2816281179 learned_addr learned my addr 172.21.15.60:0/2816281179 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:05.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720e7fc640 10 monclient(hunting): get_auth_request con 0x7f721013b020 auth_method 0 2024-03-27T21:41:05.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:05.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:05.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720effd640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f72100b40b0 0x7f7210138660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:40004/0 (socket says 172.21.15.60:40004) 2024-03-27T21:41:05.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:05.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720effd640 10 monclient(hunting): get_auth_request con 0x7f72100b40b0 auth_method 0 2024-03-27T21:41:05.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:05.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:05.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:05.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720f7fe640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72101416f0 0x7f72101d6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720f7fe640 10 monclient(hunting): get_auth_request con 0x7f72101416f0 auth_method 0 2024-03-27T21:41:05.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:05.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:05.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:05.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:05.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:05.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:05.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:05.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:05.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:05.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720e7fc640 10 monclient(hunting): handle_auth_done global_id 4848 payload 824 2024-03-27T21:41:05.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720e7fc640 1 -- 172.21.15.60:0/2816281179 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f72100b40b0 msgr2=0x7f7210138660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:05.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720e7fc640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f72100b40b0 0x7f7210138660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:05.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720e7fc640 1 -- 172.21.15.60:0/2816281179 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72101416f0 msgr2=0x7f72101d6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:05.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720e7fc640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72101416f0 0x7f72101d6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:05.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720e7fc640 10 monclient: _finish_hunting 0 2024-03-27T21:41:05.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.743+0000 7f720e7fc640 1 monclient: found mon.noname-a 2024-03-27T21:41:05.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f720e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:05.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f720e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:05.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f720e7fc640 1 -- 172.21.15.60:0/2816281179 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7204009400 con 0x7f721013b020 2024-03-27T21:41:05.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f720f7fe640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72101416f0 0x7f72101d6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:05.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f720effd640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f72100b40b0 0x7f7210138660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:41:05.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f720e7fc640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f721013b020 0x7f72100aada0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f72040098c0 tx=0x7f7204004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:05.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f71f77fe640 10 client.?.objecter ms_handle_connect 0x7f721013b020 2024-03-27T21:41:05.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f71f77fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:05.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f71f77fe640 1 -- 172.21.15.60:0/2816281179 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720403e040 con 0x7f721013b020 2024-03-27T21:41:05.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f71f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:05.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f71f77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:05.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f71f77fe640 10 monclient: dump: 2024-03-27T21:41:05.749 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:05.749 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:05.750 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.750 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.750 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:05.750 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:05.750 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:05.750 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:05.750 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:05.750 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:05.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f71f77fe640 10 monclient: _finish_auth 0 2024-03-27T21:41:05.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f71f77fe640 10 monclient: _check_auth_tickets 2024-03-27T21:41:05.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f71f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:05.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f7215342640 5 monclient: authenticate success, global_id 4848 2024-03-27T21:41:05.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f7215342640 10 monclient: _renew_subs 2024-03-27T21:41:05.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f7215342640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:05.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f7215342640 1 -- 172.21.15.60:0/2816281179 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72101d6210 con 0x7f721013b020 2024-03-27T21:41:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f71f77fe640 1 -- 172.21.15.60:0/2816281179 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7204039c10 con 0x7f721013b020 2024-03-27T21:41:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f71f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f71f77fe640 1 -- 172.21.15.60:0/2816281179 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7204030e30 con 0x7f721013b020 2024-03-27T21:41:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f71f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f71f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f71f77fe640 10 monclient: dump: 2024-03-27T21:41:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:05.752 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:05.752 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:05.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f720d7fa640 4 set_mon_vals no callback set 2024-03-27T21:41:05.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f7215342640 10 client.4848.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:05.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f7215342640 10 monclient: _renew_subs 2024-03-27T21:41:05.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f7215342640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:05.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f7215342640 1 -- 172.21.15.60:0/2816281179 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72100abba0 con 0x7f721013b020 2024-03-27T21:41:05.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.744+0000 7f7215342640 1 librados: init done 2024-03-27T21:41:05.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.745+0000 7f7215342640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:41:05.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.745+0000 7f7215342640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.745+0000 7f7215342640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.745+0000 7f7215342640 1 -- 172.21.15.60:0/2816281179 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f72100acf50 con 0x7f721013b020 2024-03-27T21:41:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.747+0000 7f71f77fe640 1 -- 172.21.15.60:0/2816281179 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7204043390 con 0x7f721013b020 2024-03-27T21:41:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.747+0000 7f71f77fe640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f71fc06c5c0 0x7f71fc06ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.747+0000 7f71f77fe640 1 -- 172.21.15.60:0/2816281179 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f72040ab9c0 con 0x7f721013b020 2024-03-27T21:41:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.747+0000 7f71f77fe640 10 client.4848.objecter ms_dispatch 0x7f7210000b90 osd_map(51..51 src has 1..51) v4 2024-03-27T21:41:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.747+0000 7f71f77fe640 3 client.4848.objecter handle_osd_map got epochs [51,51] > 0 2024-03-27T21:41:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.747+0000 7f71f77fe640 3 client.4848.objecter handle_osd_map decoding full epoch 51 2024-03-27T21:41:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.747+0000 7f71f77fe640 20 client.4848.objecter dump_active .. 0 homeless 2024-03-27T21:41:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.747+0000 7f71f77fe640 1 -- 172.21.15.60:0/2816281179 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v51) v1 ==== 54+0+16 (secure 0 0 0) 0x7f72040ac0c0 con 0x7f721013b020 2024-03-27T21:41:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.747+0000 7f71f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.747+0000 7f71f77fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.747+0000 7f720effd640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f71fc06c5c0 0x7f71fc06ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.747+0000 7f720effd640 10 monclient: get_auth_request con 0x7f71fc06c5c0 auth_method 0 2024-03-27T21:41:05.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.747+0000 7f7215342640 10 client.4848.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:05.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.747+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign -1 1 2024-03-27T21:41:05.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.747+0000 7f7215342640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f721014cd30 0x7f7210135760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.748+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f72100abd90 osd=0 3 2024-03-27T21:41:05.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.748+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f72100abd90 osd=0 3 2024-03-27T21:41:05.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.748+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f72100abd90 osd=0 3 2024-03-27T21:41:05.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.748+0000 7f7215342640 15 client.4848.objecter _session_command_op_remove -1 1 2024-03-27T21:41:05.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.748+0000 7f7215342640 20 client.4848.objecter get_session s=0x7f72100abd90 osd=0 3 2024-03-27T21:41:05.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.748+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign 0 1 2024-03-27T21:41:05.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.748+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f72100abd90 osd=0 4 2024-03-27T21:41:05.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.748+0000 7f7215342640 10 client.4848.objecter _send_command 1 2024-03-27T21:41:05.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.748+0000 7f7215342640 1 -- 172.21.15.60:0/2816281179 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7210069470 con 0x7f721014cd30 2024-03-27T21:41:05.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.748+0000 7f720f7fe640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f721014cd30 0x7f7210135760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.748+0000 7f720f7fe640 10 monclient: get_auth_request con 0x7f721014cd30 auth_method 0 2024-03-27T21:41:05.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.748+0000 7f71f77fe640 10 client.4848.objecter ms_handle_connect 0x7f71fc06c5c0 2024-03-27T21:41:05.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.748+0000 7f720effd640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f71fc06c5c0 0x7f71fc06ea80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7200000f60 tx=0x7f7200009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:05.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.748+0000 7f71f77fe640 10 client.4848.objecter ms_handle_connect 0x7f721014cd30 2024-03-27T21:41:05.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.748+0000 7f720f7fe640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f721014cd30 0x7f7210135760 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:05.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.749+0000 7f71f77fe640 1 -- 172.21.15.60:0/2816281179 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7210069470 con 0x7f721014cd30 2024-03-27T21:41:05.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.749+0000 7f71f77fe640 10 client.4848.objecter ms_dispatch 0x7f7210000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:41:05.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.749+0000 7f71f77fe640 10 client.4848.objecter _finish_command 1 = system:0 2024-03-27T21:41:05.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.749+0000 7f71f77fe640 20 client.4848.objecter put_session s=0x7f72100abd90 osd=0 4 2024-03-27T21:41:05.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.749+0000 7f71f77fe640 15 client.4848.objecter _session_command_op_remove 0 1 2024-03-27T21:41:05.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.765+0000 7f7215342640 10 client.4848.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:05.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.765+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign -1 2 2024-03-27T21:41:05.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.765+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f72100abd90 osd=0 3 2024-03-27T21:41:05.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.765+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f72100abd90 osd=0 3 2024-03-27T21:41:05.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.765+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f72100abd90 osd=0 3 2024-03-27T21:41:05.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.765+0000 7f7215342640 15 client.4848.objecter _session_command_op_remove -1 2 2024-03-27T21:41:05.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.765+0000 7f7215342640 20 client.4848.objecter get_session s=0x7f72100abd90 osd=0 3 2024-03-27T21:41:05.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.765+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign 0 2 2024-03-27T21:41:05.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.765+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f72100abd90 osd=0 4 2024-03-27T21:41:05.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.765+0000 7f7215342640 10 client.4848.objecter _send_command 2 2024-03-27T21:41:05.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.765+0000 7f7215342640 1 -- 172.21.15.60:0/2816281179 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7210141ed0 con 0x7f721014cd30 2024-03-27T21:41:05.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.768+0000 7f71f77fe640 1 -- 172.21.15.60:0/2816281179 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7210141ed0 con 0x7f721014cd30 2024-03-27T21:41:05.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.768+0000 7f71f77fe640 10 client.4848.objecter ms_dispatch 0x7f7210000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:05.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.768+0000 7f71f77fe640 10 client.4848.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:05.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.768+0000 7f71f77fe640 20 client.4848.objecter put_session s=0x7f72100abd90 osd=0 4 2024-03-27T21:41:05.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.768+0000 7f71f77fe640 15 client.4848.objecter _session_command_op_remove 0 2 2024-03-27T21:41:05.759 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:41:05.760 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:41:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.768+0000 7f7215342640 10 client.4848.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.768+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign -1 3 2024-03-27T21:41:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.768+0000 7f7215342640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f72101d9d90 0x7f72101da190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.768+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f72101d9710 osd=1 3 2024-03-27T21:41:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.768+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f72101d9710 osd=1 3 2024-03-27T21:41:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.768+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f72101d9710 osd=1 3 2024-03-27T21:41:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.768+0000 7f7215342640 15 client.4848.objecter _session_command_op_remove -1 3 2024-03-27T21:41:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.768+0000 7f7215342640 20 client.4848.objecter get_session s=0x7f72101d9710 osd=1 3 2024-03-27T21:41:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.768+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign 1 3 2024-03-27T21:41:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.768+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f72101d9710 osd=1 4 2024-03-27T21:41:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.768+0000 7f7215342640 10 client.4848.objecter _send_command 3 2024-03-27T21:41:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.768+0000 7f7215342640 1 -- 172.21.15.60:0/2816281179 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72101e4a60 con 0x7f72101d9d90 2024-03-27T21:41:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.768+0000 7f720effd640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f72101d9d90 0x7f72101da190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.768+0000 7f720effd640 10 monclient: get_auth_request con 0x7f72101d9d90 auth_method 0 2024-03-27T21:41:05.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.768+0000 7f71f77fe640 10 client.4848.objecter ms_handle_connect 0x7f72101d9d90 2024-03-27T21:41:05.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.768+0000 7f720effd640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f72101d9d90 0x7f72101da190 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:05.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.769+0000 7f71f77fe640 1 -- 172.21.15.60:0/2816281179 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f72101e4a60 con 0x7f72101d9d90 2024-03-27T21:41:05.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.769+0000 7f71f77fe640 10 client.4848.objecter ms_dispatch 0x7f7210000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:41:05.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.769+0000 7f71f77fe640 10 client.4848.objecter _finish_command 3 = system:0 2024-03-27T21:41:05.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.769+0000 7f71f77fe640 20 client.4848.objecter put_session s=0x7f72101d9710 osd=1 4 2024-03-27T21:41:05.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.769+0000 7f71f77fe640 15 client.4848.objecter _session_command_op_remove 1 3 2024-03-27T21:41:05.762 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:05.772+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:05.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.795+0000 7f7215342640 10 client.4848.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:05.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.795+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign -1 4 2024-03-27T21:41:05.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.795+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f72101d9710 osd=1 3 2024-03-27T21:41:05.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.795+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f72101d9710 osd=1 3 2024-03-27T21:41:05.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.795+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f72101d9710 osd=1 3 2024-03-27T21:41:05.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.795+0000 7f7215342640 15 client.4848.objecter _session_command_op_remove -1 4 2024-03-27T21:41:05.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.795+0000 7f7215342640 20 client.4848.objecter get_session s=0x7f72101d9710 osd=1 3 2024-03-27T21:41:05.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.795+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign 1 4 2024-03-27T21:41:05.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.795+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f72101d9710 osd=1 4 2024-03-27T21:41:05.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.795+0000 7f7215342640 10 client.4848.objecter _send_command 4 2024-03-27T21:41:05.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.795+0000 7f7215342640 1 -- 172.21.15.60:0/2816281179 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f72100b5590 con 0x7f72101d9d90 2024-03-27T21:41:05.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f71f77fe640 1 -- 172.21.15.60:0/2816281179 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f72100b5590 con 0x7f72101d9d90 2024-03-27T21:41:05.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f71f77fe640 10 client.4848.objecter ms_dispatch 0x7f7210000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:05.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f71f77fe640 10 client.4848.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:05.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f71f77fe640 20 client.4848.objecter put_session s=0x7f72101d9710 osd=1 4 2024-03-27T21:41:05.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f71f77fe640 15 client.4848.objecter _session_command_op_remove 1 4 2024-03-27T21:41:05.784 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:41:05.786 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:41:05.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f7215342640 10 client.4848.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:05.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign -1 5 2024-03-27T21:41:05.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f7215342640 20 client.4848.objecter _get_session osd=-1 returning homeless 2024-03-27T21:41:05.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f7215342640 10 client.4848.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:05.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f7215342640 10 monclient: _renew_subs 2024-03-27T21:41:05.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f7215342640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:05.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f7215342640 1 -- 172.21.15.60:0/2816281179 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=52}) v3 -- 0x7f721013fb60 con 0x7f721013b020 2024-03-27T21:41:05.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f7215342640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:05.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f7215342640 1 -- 172.21.15.60:0/2816281179 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f72100b59a0 con 0x7f721013b020 2024-03-27T21:41:05.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f71f77fe640 1 -- 172.21.15.60:0/2816281179 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7204076f50 con 0x7f721013b020 2024-03-27T21:41:05.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f71f77fe640 10 monclient: handle_get_version_reply finishing 1 version 51 2024-03-27T21:41:05.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f720d7fa640 10 client.4848.objecter _check_command_map_dne tid 5 current 51 map_dne_bound 51 2024-03-27T21:41:05.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f720d7fa640 10 client.4848.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:41:05.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f720d7fa640 15 client.4848.objecter _session_command_op_remove -1 5 2024-03-27T21:41:05.787 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:41:05.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f7215342640 10 client.4848.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:05.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign -1 6 2024-03-27T21:41:05.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f7215342640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f721002b680 0x7f721002bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f72101400d0 osd=3 3 2024-03-27T21:41:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f72101400d0 osd=3 3 2024-03-27T21:41:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f72101400d0 osd=3 3 2024-03-27T21:41:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f7215342640 15 client.4848.objecter _session_command_op_remove -1 6 2024-03-27T21:41:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f7215342640 20 client.4848.objecter get_session s=0x7f72101400d0 osd=3 3 2024-03-27T21:41:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign 3 6 2024-03-27T21:41:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f72101400d0 osd=3 4 2024-03-27T21:41:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f7215342640 10 client.4848.objecter _send_command 6 2024-03-27T21:41:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.797+0000 7f7215342640 1 -- 172.21.15.60:0/2816281179 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7210140240 con 0x7f721002b680 2024-03-27T21:41:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.798+0000 7f720e7fc640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f721002b680 0x7f721002bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.798+0000 7f720e7fc640 10 monclient: get_auth_request con 0x7f721002b680 auth_method 0 2024-03-27T21:41:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.798+0000 7f71f77fe640 10 client.4848.objecter ms_handle_connect 0x7f721002b680 2024-03-27T21:41:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.798+0000 7f720e7fc640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f721002b680 0x7f721002bab0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.799+0000 7f71f77fe640 1 -- 172.21.15.60:0/2816281179 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7210140240 con 0x7f721002b680 2024-03-27T21:41:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.799+0000 7f71f77fe640 10 client.4848.objecter ms_dispatch 0x7f7210000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:41:05.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.799+0000 7f71f77fe640 10 client.4848.objecter _finish_command 6 = system:0 2024-03-27T21:41:05.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.799+0000 7f71f77fe640 20 client.4848.objecter put_session s=0x7f72101400d0 osd=3 4 2024-03-27T21:41:05.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.799+0000 7f71f77fe640 15 client.4848.objecter _session_command_op_remove 3 6 2024-03-27T21:41:05.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.803+0000 7f7044700640 1 -- 172.21.15.60:0/825085854 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}) v1 -- 0x7f703c13cf70 con 0x7f702006c590 2024-03-27T21:41:05.789 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:41:05.803+0000 7f63d8d11640 -1 mgr.server reply reply (2) No such file or directory no change 2024-03-27T21:41:05.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.803+0000 7f7032ffd640 1 -- 172.21.15.60:0/825085854 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 2 no change) v1 ==== 17+0+337 (secure 0 0 0) 0x7f703c13cf70 con 0x7f702006c590 2024-03-27T21:41:05.790 INFO:teuthology.orchestra.run.smithi060.stdout:moved 0 / 18 (0%) 2024-03-27T21:41:05.790 INFO:teuthology.orchestra.run.smithi060.stdout:avg 2.57143 2024-03-27T21:41:05.790 INFO:teuthology.orchestra.run.smithi060.stdout:stddev 1.41421 -> 1.41421 (expected baseline 1.48461) 2024-03-27T21:41:05.790 INFO:teuthology.orchestra.run.smithi060.stdout:min osd.4 with 0 -> 0 pgs (0 -> 0 * mean) 2024-03-27T21:41:05.790 INFO:teuthology.orchestra.run.smithi060.stdout:max osd.1 with 4 -> 4 pgs (1.55556 -> 1.55556 * mean) 2024-03-27T21:41:05.790 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:41:05.790 INFO:teuthology.orchestra.run.smithi060.stdout:oload 110 2024-03-27T21:41:05.790 INFO:teuthology.orchestra.run.smithi060.stdout:max_change 3 2024-03-27T21:41:05.791 INFO:teuthology.orchestra.run.smithi060.stdout:max_change_osds 4 2024-03-27T21:41:05.791 INFO:teuthology.orchestra.run.smithi060.stdout:average_utilization 0.0003 2024-03-27T21:41:05.791 INFO:teuthology.orchestra.run.smithi060.stdout:overload_utilization 0.0003 2024-03-27T21:41:05.791 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 weight 1.0000 -> 0.8795 2024-03-27T21:41:05.791 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 weight 1.0000 -> 0.9070 2024-03-27T21:41:05.791 INFO:teuthology.orchestra.run.smithi060.stderr:no change 2024-03-27T21:41:05.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.806+0000 7f7044700640 10 librados: watch_flush enter 2024-03-27T21:41:05.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.806+0000 7f7044700640 10 librados: watch_flush exit 2024-03-27T21:41:05.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.806+0000 7f7044700640 20 client.4679.objecter shutdown clearing up homeless session... 2024-03-27T21:41:05.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.806+0000 7f7044700640 10 client.4679.objecter successfully canceled tick 2024-03-27T21:41:05.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.806+0000 7f7044700640 1 -- 172.21.15.60:0/825085854 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f702006c590 msgr2=0x7f702006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:05.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.806+0000 7f7044700640 1 --2- 172.21.15.60:0/825085854 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f702006c590 0x7f702006ea50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7038000f60 tx=0x7f7038009040 comp rx=0 tx=0).stop 2024-03-27T21:41:05.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.806+0000 7f7044700640 10 monclient: shutdown 2024-03-27T21:41:05.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.806+0000 7f7044700640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:05.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.806+0000 7f7044700640 1 -- 172.21.15.60:0/825085854 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f703c13ef20 msgr2=0x7f703c138ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:05.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.806+0000 7f7044700640 1 --2- 172.21.15.60:0/825085854 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f703c13ef20 0x7f703c138ae0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f702c00e8b0 tx=0x7f702c00ed70 comp rx=0 tx=0).stop 2024-03-27T21:41:05.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.806+0000 7f7044700640 1 -- 172.21.15.60:0/825085854 shutdown_connections 2024-03-27T21:41:05.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.806+0000 7f7044700640 1 --2- 172.21.15.60:0/825085854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f703c141630 0x7f703c1396a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:05.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.806+0000 7f7044700640 1 --2- 172.21.15.60:0/825085854 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f703c13ef20 0x7f703c138ae0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:05.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.806+0000 7f7044700640 1 --2- 172.21.15.60:0/825085854 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f702006c590 0x7f702006ea50 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:05.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.806+0000 7f7044700640 1 --2- 172.21.15.60:0/825085854 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f703c071a50 0x7f703c1361f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:05.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.806+0000 7f7044700640 1 -- 172.21.15.60:0/825085854 >> 172.21.15.60:0/825085854 conn(0x7f703c0b7300 msgr2=0x7f703c13dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:05.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.806+0000 7f7044700640 1 -- 172.21.15.60:0/825085854 shutdown_connections 2024-03-27T21:41:05.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.806+0000 7f7044700640 1 -- 172.21.15.60:0/825085854 wait complete. 2024-03-27T21:41:05.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.806+0000 7f7044700640 1 librados: shutdown 2024-03-27T21:41:05.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.814+0000 7f7215342640 10 client.4848.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:05.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.814+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign -1 7 2024-03-27T21:41:05.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.814+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f72101400d0 osd=3 3 2024-03-27T21:41:05.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.814+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f72101400d0 osd=3 3 2024-03-27T21:41:05.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.814+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f72101400d0 osd=3 3 2024-03-27T21:41:05.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.814+0000 7f7215342640 15 client.4848.objecter _session_command_op_remove -1 7 2024-03-27T21:41:05.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.814+0000 7f7215342640 20 client.4848.objecter get_session s=0x7f72101400d0 osd=3 3 2024-03-27T21:41:05.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.814+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign 3 7 2024-03-27T21:41:05.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.814+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f72101400d0 osd=3 4 2024-03-27T21:41:05.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.814+0000 7f7215342640 10 client.4848.objecter _send_command 7 2024-03-27T21:41:05.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.814+0000 7f7215342640 1 -- 172.21.15.60:0/2816281179 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f721013f5e0 con 0x7f721002b680 2024-03-27T21:41:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.815+0000 7f71f77fe640 1 -- 172.21.15.60:0/2816281179 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f721013f5e0 con 0x7f721002b680 2024-03-27T21:41:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.816+0000 7f71f77fe640 10 client.4848.objecter ms_dispatch 0x7f7210000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.816+0000 7f71f77fe640 10 client.4848.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.816+0000 7f71f77fe640 20 client.4848.objecter put_session s=0x7f72101400d0 osd=3 4 2024-03-27T21:41:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.816+0000 7f71f77fe640 15 client.4848.objecter _session_command_op_remove 3 7 2024-03-27T21:41:05.802 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:41:05.803 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:41:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.816+0000 7f7215342640 10 client.4848.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.816+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign -1 8 2024-03-27T21:41:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.816+0000 7f7215342640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f72101e70f0 0x7f72101e94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.816+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f721013f990 osd=4 3 2024-03-27T21:41:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.816+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f721013f990 osd=4 3 2024-03-27T21:41:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.816+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f721013f990 osd=4 3 2024-03-27T21:41:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.816+0000 7f7215342640 15 client.4848.objecter _session_command_op_remove -1 8 2024-03-27T21:41:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.816+0000 7f7215342640 20 client.4848.objecter get_session s=0x7f721013f990 osd=4 3 2024-03-27T21:41:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.816+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign 4 8 2024-03-27T21:41:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.816+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f721013f990 osd=4 4 2024-03-27T21:41:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.816+0000 7f7215342640 10 client.4848.objecter _send_command 8 2024-03-27T21:41:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.816+0000 7f7215342640 1 -- 172.21.15.60:0/2816281179 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72101ec2b0 con 0x7f72101e70f0 2024-03-27T21:41:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.816+0000 7f720f7fe640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f72101e70f0 0x7f72101e94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.816+0000 7f720f7fe640 10 monclient: get_auth_request con 0x7f72101e70f0 auth_method 0 2024-03-27T21:41:05.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.817+0000 7f71f77fe640 10 client.4848.objecter ms_handle_connect 0x7f72101e70f0 2024-03-27T21:41:05.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.817+0000 7f720f7fe640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f72101e70f0 0x7f72101e94e0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:05.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.817+0000 7f71f77fe640 1 -- 172.21.15.60:0/2816281179 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f72101ec2b0 con 0x7f72101e70f0 2024-03-27T21:41:05.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.817+0000 7f71f77fe640 10 client.4848.objecter ms_dispatch 0x7f7210000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:41:05.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.817+0000 7f71f77fe640 10 client.4848.objecter _finish_command 8 = system:0 2024-03-27T21:41:05.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.817+0000 7f71f77fe640 20 client.4848.objecter put_session s=0x7f721013f990 osd=4 4 2024-03-27T21:41:05.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.817+0000 7f71f77fe640 15 client.4848.objecter _session_command_op_remove 4 8 2024-03-27T21:41:05.808 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-03-27T21:41:05.808 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 3.0 2024-03-27T21:41:05.808 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:05.808 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:05.808 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:41:05.808 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:41:05.808 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:05.818 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:41:05.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.832+0000 7f7215342640 10 client.4848.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:05.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.832+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign -1 9 2024-03-27T21:41:05.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.832+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f721013f990 osd=4 3 2024-03-27T21:41:05.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.832+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f721013f990 osd=4 3 2024-03-27T21:41:05.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.832+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f721013f990 osd=4 3 2024-03-27T21:41:05.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.832+0000 7f7215342640 15 client.4848.objecter _session_command_op_remove -1 9 2024-03-27T21:41:05.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.832+0000 7f7215342640 20 client.4848.objecter get_session s=0x7f721013f990 osd=4 3 2024-03-27T21:41:05.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.832+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign 4 9 2024-03-27T21:41:05.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.832+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f721013f990 osd=4 4 2024-03-27T21:41:05.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.832+0000 7f7215342640 10 client.4848.objecter _send_command 9 2024-03-27T21:41:05.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.832+0000 7f7215342640 1 -- 172.21.15.60:0/2816281179 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f721002c230 con 0x7f72101e70f0 2024-03-27T21:41:05.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.834+0000 7f71f77fe640 1 -- 172.21.15.60:0/2816281179 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f721002c230 con 0x7f72101e70f0 2024-03-27T21:41:05.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.834+0000 7f71f77fe640 10 client.4848.objecter ms_dispatch 0x7f7210000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:05.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.834+0000 7f71f77fe640 10 client.4848.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:05.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.834+0000 7f71f77fe640 20 client.4848.objecter put_session s=0x7f721013f990 osd=4 4 2024-03-27T21:41:05.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.834+0000 7f71f77fe640 15 client.4848.objecter _session_command_op_remove 4 9 2024-03-27T21:41:05.821 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:41:05.822 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:41:05.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.835+0000 7f7215342640 10 client.4848.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:05.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.835+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign -1 10 2024-03-27T21:41:05.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.835+0000 7f7215342640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f721002c3a0 0x7f72101eaba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.835+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f721002c230 osd=5 3 2024-03-27T21:41:05.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.835+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f721002c230 osd=5 3 2024-03-27T21:41:05.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.835+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f721002c230 osd=5 3 2024-03-27T21:41:05.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.835+0000 7f7215342640 15 client.4848.objecter _session_command_op_remove -1 10 2024-03-27T21:41:05.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.835+0000 7f7215342640 20 client.4848.objecter get_session s=0x7f721002c230 osd=5 3 2024-03-27T21:41:05.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.835+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign 5 10 2024-03-27T21:41:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.835+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f721002c230 osd=5 4 2024-03-27T21:41:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.835+0000 7f7215342640 10 client.4848.objecter _send_command 10 2024-03-27T21:41:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.835+0000 7f7215342640 1 -- 172.21.15.60:0/2816281179 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7210066760 con 0x7f721002c3a0 2024-03-27T21:41:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.835+0000 7f720effd640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f721002c3a0 0x7f72101eaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.835+0000 7f720effd640 10 monclient: get_auth_request con 0x7f721002c3a0 auth_method 0 2024-03-27T21:41:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.835+0000 7f720effd640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f721002c3a0 0x7f72101eaba0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.835+0000 7f71f77fe640 10 client.4848.objecter ms_handle_connect 0x7f721002c3a0 2024-03-27T21:41:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.836+0000 7f71f77fe640 1 -- 172.21.15.60:0/2816281179 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7210066760 con 0x7f721002c3a0 2024-03-27T21:41:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.836+0000 7f71f77fe640 10 client.4848.objecter ms_dispatch 0x7f7210000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:41:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.836+0000 7f71f77fe640 10 client.4848.objecter _finish_command 10 = system:0 2024-03-27T21:41:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.836+0000 7f71f77fe640 20 client.4848.objecter put_session s=0x7f721002c230 osd=5 4 2024-03-27T21:41:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.836+0000 7f71f77fe640 15 client.4848.objecter _session_command_op_remove 5 10 2024-03-27T21:41:05.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.855+0000 7f7215342640 10 client.4848.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:05.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.855+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign -1 11 2024-03-27T21:41:05.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.855+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f721002c230 osd=5 3 2024-03-27T21:41:05.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.855+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f721002c230 osd=5 3 2024-03-27T21:41:05.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.855+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f721002c230 osd=5 3 2024-03-27T21:41:05.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.855+0000 7f7215342640 15 client.4848.objecter _session_command_op_remove -1 11 2024-03-27T21:41:05.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.855+0000 7f7215342640 20 client.4848.objecter get_session s=0x7f721002c230 osd=5 3 2024-03-27T21:41:05.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.855+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign 5 11 2024-03-27T21:41:05.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.855+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f721002c230 osd=5 4 2024-03-27T21:41:05.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.855+0000 7f7215342640 10 client.4848.objecter _send_command 11 2024-03-27T21:41:05.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.855+0000 7f7215342640 1 -- 172.21.15.60:0/2816281179 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7210066760 con 0x7f721002c3a0 2024-03-27T21:41:05.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.857+0000 7f71f77fe640 1 -- 172.21.15.60:0/2816281179 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f7210066760 con 0x7f721002c3a0 2024-03-27T21:41:05.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.857+0000 7f71f77fe640 10 client.4848.objecter ms_dispatch 0x7f7210000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:05.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.857+0000 7f71f77fe640 10 client.4848.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:05.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.857+0000 7f71f77fe640 20 client.4848.objecter put_session s=0x7f721002c230 osd=5 4 2024-03-27T21:41:05.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.857+0000 7f71f77fe640 15 client.4848.objecter _session_command_op_remove 5 11 2024-03-27T21:41:05.843 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:41:05.844 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:41:05.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.857+0000 7f7215342640 10 client.4848.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:05.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.857+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign -1 12 2024-03-27T21:41:05.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.857+0000 7f7215342640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f72101eb5f0 0x7f72101eba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.857+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f7210066760 osd=6 3 2024-03-27T21:41:05.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.857+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f7210066760 osd=6 3 2024-03-27T21:41:05.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.857+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f7210066760 osd=6 3 2024-03-27T21:41:05.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.857+0000 7f7215342640 15 client.4848.objecter _session_command_op_remove -1 12 2024-03-27T21:41:05.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.857+0000 7f7215342640 20 client.4848.objecter get_session s=0x7f7210066760 osd=6 3 2024-03-27T21:41:05.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.857+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign 6 12 2024-03-27T21:41:05.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.857+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f7210066760 osd=6 4 2024-03-27T21:41:05.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.857+0000 7f7215342640 10 client.4848.objecter _send_command 12 2024-03-27T21:41:05.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.857+0000 7f7215342640 1 -- 172.21.15.60:0/2816281179 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72101ebfb0 con 0x7f72101eb5f0 2024-03-27T21:41:05.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.857+0000 7f720e7fc640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f72101eb5f0 0x7f72101eba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.857+0000 7f720e7fc640 10 monclient: get_auth_request con 0x7f72101eb5f0 auth_method 0 2024-03-27T21:41:05.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.858+0000 7f71f77fe640 10 client.4848.objecter ms_handle_connect 0x7f72101eb5f0 2024-03-27T21:41:05.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.858+0000 7f720e7fc640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f72101eb5f0 0x7f72101eba70 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:05.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.859+0000 7f71f77fe640 1 -- 172.21.15.60:0/2816281179 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f72101ebfb0 con 0x7f72101eb5f0 2024-03-27T21:41:05.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.859+0000 7f71f77fe640 10 client.4848.objecter ms_dispatch 0x7f7210000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:41:05.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.859+0000 7f71f77fe640 10 client.4848.objecter _finish_command 12 = system:0 2024-03-27T21:41:05.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.859+0000 7f71f77fe640 20 client.4848.objecter put_session s=0x7f7210066760 osd=6 4 2024-03-27T21:41:05.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.859+0000 7f71f77fe640 15 client.4848.objecter _session_command_op_remove 6 12 2024-03-27T21:41:05.859 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:05.874+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:05.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.874+0000 7f7215342640 10 client.4848.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:05.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.874+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign -1 13 2024-03-27T21:41:05.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.874+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f7210066760 osd=6 3 2024-03-27T21:41:05.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.874+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f7210066760 osd=6 3 2024-03-27T21:41:05.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.874+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f7210066760 osd=6 3 2024-03-27T21:41:05.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.874+0000 7f7215342640 15 client.4848.objecter _session_command_op_remove -1 13 2024-03-27T21:41:05.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.874+0000 7f7215342640 20 client.4848.objecter get_session s=0x7f7210066760 osd=6 3 2024-03-27T21:41:05.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.874+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign 6 13 2024-03-27T21:41:05.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.874+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f7210066760 osd=6 4 2024-03-27T21:41:05.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.874+0000 7f7215342640 10 client.4848.objecter _send_command 13 2024-03-27T21:41:05.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.874+0000 7f7215342640 1 -- 172.21.15.60:0/2816281179 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f72101ebfb0 con 0x7f72101eb5f0 2024-03-27T21:41:05.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.876+0000 7f71f77fe640 1 -- 172.21.15.60:0/2816281179 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f72101ebfb0 con 0x7f72101eb5f0 2024-03-27T21:41:05.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.876+0000 7f71f77fe640 10 client.4848.objecter ms_dispatch 0x7f7210000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:05.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.876+0000 7f71f77fe640 10 client.4848.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:05.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.876+0000 7f71f77fe640 20 client.4848.objecter put_session s=0x7f7210066760 osd=6 4 2024-03-27T21:41:05.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.876+0000 7f71f77fe640 15 client.4848.objecter _session_command_op_remove 6 13 2024-03-27T21:41:05.863 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:41:05.864 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:41:05.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.876+0000 7f7215342640 10 client.4848.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:05.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.876+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign -1 14 2024-03-27T21:41:05.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.876+0000 7f7215342640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f72101f0950 0x7f72101f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.876+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f72101ebfb0 osd=7 3 2024-03-27T21:41:05.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.876+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f72101ebfb0 osd=7 3 2024-03-27T21:41:05.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.876+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f72101ebfb0 osd=7 3 2024-03-27T21:41:05.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.876+0000 7f7215342640 15 client.4848.objecter _session_command_op_remove -1 14 2024-03-27T21:41:05.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.876+0000 7f7215342640 20 client.4848.objecter get_session s=0x7f72101ebfb0 osd=7 3 2024-03-27T21:41:05.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.876+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign 7 14 2024-03-27T21:41:05.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.876+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f72101ebfb0 osd=7 4 2024-03-27T21:41:05.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.876+0000 7f7215342640 10 client.4848.objecter _send_command 14 2024-03-27T21:41:05.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.876+0000 7f7215342640 1 -- 172.21.15.60:0/2816281179 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72100668d0 con 0x7f72101f0950 2024-03-27T21:41:05.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.877+0000 7f720f7fe640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f72101f0950 0x7f72101f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.877+0000 7f720f7fe640 10 monclient: get_auth_request con 0x7f72101f0950 auth_method 0 2024-03-27T21:41:05.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.877+0000 7f71f77fe640 10 client.4848.objecter ms_handle_connect 0x7f72101f0950 2024-03-27T21:41:05.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.877+0000 7f720f7fe640 1 --2- 172.21.15.60:0/2816281179 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f72101f0950 0x7f72101f2d40 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:05.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.878+0000 7f71f77fe640 1 -- 172.21.15.60:0/2816281179 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f72100668d0 con 0x7f72101f0950 2024-03-27T21:41:05.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.878+0000 7f71f77fe640 10 client.4848.objecter ms_dispatch 0x7f7210000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:41:05.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.878+0000 7f71f77fe640 10 client.4848.objecter _finish_command 14 = system:0 2024-03-27T21:41:05.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.878+0000 7f71f77fe640 20 client.4848.objecter put_session s=0x7f72101ebfb0 osd=7 4 2024-03-27T21:41:05.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.878+0000 7f71f77fe640 15 client.4848.objecter _session_command_op_remove 7 14 2024-03-27T21:41:05.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.908+0000 7f7215342640 10 client.4848.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:05.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.908+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign -1 15 2024-03-27T21:41:05.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.908+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f72101ebfb0 osd=7 3 2024-03-27T21:41:05.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.908+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f72101ebfb0 osd=7 3 2024-03-27T21:41:05.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.908+0000 7f7215342640 20 client.4848.objecter _get_session s=0x7f72101ebfb0 osd=7 3 2024-03-27T21:41:05.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.908+0000 7f7215342640 15 client.4848.objecter _session_command_op_remove -1 15 2024-03-27T21:41:05.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.908+0000 7f7215342640 20 client.4848.objecter get_session s=0x7f72101ebfb0 osd=7 3 2024-03-27T21:41:05.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.908+0000 7f7215342640 15 client.4848.objecter _session_command_op_assign 7 15 2024-03-27T21:41:05.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.908+0000 7f7215342640 20 client.4848.objecter put_session s=0x7f72101ebfb0 osd=7 4 2024-03-27T21:41:05.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.908+0000 7f7215342640 10 client.4848.objecter _send_command 15 2024-03-27T21:41:05.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.908+0000 7f7215342640 1 -- 172.21.15.60:0/2816281179 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f72100668d0 con 0x7f72101f0950 2024-03-27T21:41:05.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.910+0000 7f71f77fe640 1 -- 172.21.15.60:0/2816281179 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f72100668d0 con 0x7f72101f0950 2024-03-27T21:41:05.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.910+0000 7f71f77fe640 10 client.4848.objecter ms_dispatch 0x7f7210000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:05.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.910+0000 7f71f77fe640 10 client.4848.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:05.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.910+0000 7f71f77fe640 20 client.4848.objecter put_session s=0x7f72101ebfb0 osd=7 4 2024-03-27T21:41:05.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.910+0000 7f71f77fe640 15 client.4848.objecter _session_command_op_remove 7 15 2024-03-27T21:41:05.897 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:41:05.897 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:41:05.901 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:41:05.901 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:41:05.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.942+0000 7f09fe342640 10 monclient: get_monmap_and_config 2024-03-27T21:41:05.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.942+0000 7f09fe342640 10 monclient: build_initial_monmap 2024-03-27T21:41:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.942+0000 7f09fe342640 10 monclient: monmap: 2024-03-27T21:41:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:05.943453+0000 2024-03-27T21:41:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:05.943453+0000 2024-03-27T21:41:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:05.929 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.943+0000 7f09fe342640 1 Processor -- start 2024-03-27T21:41:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.943+0000 7f09fe342640 1 -- start start 2024-03-27T21:41:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.943+0000 7f09fe342640 10 monclient: init 2024-03-27T21:41:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.943+0000 7f09fe342640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.943+0000 7f09fe342640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-03-27T21:41:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.943+0000 7f09fe342640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f09f8071a50 0x7f09f80a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.943+0000 7f09fe342640 10 monclient(hunting): picked mon.noname-c con 0x7f09f8071a50 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.943+0000 7f09fe342640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f09f813e810 0x7f09f813ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.943+0000 7f09fe342640 10 monclient(hunting): picked mon.noname-b con 0x7f09f813e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:05.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.943+0000 7f09fe342640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f09f8141630 0x7f09f8141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.943+0000 7f09fe342640 10 monclient(hunting): picked mon.noname-a con 0x7f09f8141630 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:05.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.943+0000 7f09fe342640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:05.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.943+0000 7f09fe342640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09f8071e30 con 0x7f09f8141630 2024-03-27T21:41:05.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.943+0000 7f09fe342640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:05.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.943+0000 7f09fe342640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09f813f190 con 0x7f09f8071a50 2024-03-27T21:41:05.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.943+0000 7f09fe342640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:05.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.943+0000 7f09fe342640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f80a8e90 con 0x7f09f813e810 2024-03-27T21:41:05.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.943+0000 7f09fe342640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:05.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.943+0000 7f09fe342640 10 monclient(hunting): authenticate will time out at 907.164804s 2024-03-27T21:41:05.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09f77fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f09f813e810 0x7f09f813ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09f77fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f09f813e810 0x7f09f813ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:40018/0 (socket says 172.21.15.60:40018) 2024-03-27T21:41:05.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09f77fe640 1 -- 172.21.15.60:0/3477409373 learned_addr learned my addr 172.21.15.60:0/3477409373 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:05.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09f77fe640 10 monclient(hunting): get_auth_request con 0x7f09f813e810 auth_method 0 2024-03-27T21:41:05.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:05.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:05.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:05.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09fc8df640 1 --2- 172.21.15.60:0/3477409373 >> v2:172.21.15.60:3300/0 conn(0x7f09f8141630 0x7f09f8141aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09fc8df640 10 monclient(hunting): get_auth_request con 0x7f09f8141630 auth_method 0 2024-03-27T21:41:05.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09fc8df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:05.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09fc8df640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:05.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09fc8df640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:05.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:05.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:05.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:05.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09fc8df640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:05.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09fc8df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09fc8df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09f7fff640 1 --2- 172.21.15.60:0/3477409373 >> v2:172.21.15.191:3300/0 conn(0x7f09f8071a50 0x7f09f80a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09f7fff640 10 monclient(hunting): get_auth_request con 0x7f09f8071a50 auth_method 0 2024-03-27T21:41:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09fc8df640 10 monclient(hunting): handle_auth_done global_id 4851 payload 274 2024-03-27T21:41:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09fc8df640 1 -- 172.21.15.60:0/3477409373 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f09f813e810 msgr2=0x7f09f813ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09fc8df640 1 --2- 172.21.15.60:0/3477409373 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f09f813e810 0x7f09f813ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09fc8df640 1 -- 172.21.15.60:0/3477409373 >> v2:172.21.15.191:3300/0 conn(0x7f09f8071a50 msgr2=0x7f09f80a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09fc8df640 1 --2- 172.21.15.60:0/3477409373 >> v2:172.21.15.191:3300/0 conn(0x7f09f8071a50 0x7f09f80a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09fc8df640 10 monclient: _finish_hunting 0 2024-03-27T21:41:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09fc8df640 1 monclient: found mon.noname-a 2024-03-27T21:41:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09fc8df640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09fc8df640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09fc8df640 1 -- 172.21.15.60:0/3477409373 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f80aa040 con 0x7f09f8141630 2024-03-27T21:41:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.944+0000 7f09f77fe640 1 --2- 172.21.15.60:0/3477409373 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f09f813e810 0x7f09f813ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.945+0000 7f09fc8df640 1 --2- 172.21.15.60:0/3477409373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09f8141630 0x7f09f8141aa0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f09f80b4a80 tx=0x7f09f002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d4393f38a3ebe8e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.945+0000 7f09f6ffd640 1 -- 172.21.15.60:0/3477409373 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f002cbd0 con 0x7f09f8141630 2024-03-27T21:41:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.945+0000 7f09f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.945+0000 7f09f6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.945+0000 7f09f6ffd640 10 monclient: dump: 2024-03-27T21:41:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:05.938 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.945+0000 7f09f6ffd640 10 monclient: _finish_auth 0 2024-03-27T21:41:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.945+0000 7f09f6ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:41:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.945+0000 7f09f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.945+0000 7f09fe342640 5 monclient: authenticate success, global_id 4851 2024-03-27T21:41:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.945+0000 7f09fe342640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.945+0000 7f09f6ffd640 1 -- 172.21.15.60:0/3477409373 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f002cd30 con 0x7f09f8141630 2024-03-27T21:41:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.945+0000 7f09f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.945+0000 7f09f6ffd640 1 -- 172.21.15.60:0/3477409373 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f002cea0 con 0x7f09f8141630 2024-03-27T21:41:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.945+0000 7f09fe342640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.945+0000 7f09fe342640 4 set_mon_vals no callback set 2024-03-27T21:41:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.945+0000 7f09f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.945+0000 7f09f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.945+0000 7f09f6ffd640 10 monclient: dump: 2024-03-27T21:41:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:05.940 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.946+0000 7f09fe342640 10 monclient: shutdown 2024-03-27T21:41:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.946+0000 7f09fe342640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.946+0000 7f09fe342640 1 -- 172.21.15.60:0/3477409373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09f8141630 msgr2=0x7f09f8141aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.946+0000 7f09fe342640 1 --2- 172.21.15.60:0/3477409373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09f8141630 0x7f09f8141aa0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f09f80b4a80 tx=0x7f09f002be20 comp rx=0 tx=0).stop 2024-03-27T21:41:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.946+0000 7f09fe342640 1 -- 172.21.15.60:0/3477409373 shutdown_connections 2024-03-27T21:41:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.946+0000 7f09fe342640 1 --2- 172.21.15.60:0/3477409373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09f8141630 0x7f09f8141aa0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.946+0000 7f09fe342640 1 --2- 172.21.15.60:0/3477409373 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f09f813e810 0x7f09f813ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.946+0000 7f09fe342640 1 --2- 172.21.15.60:0/3477409373 >> v2:172.21.15.191:3300/0 conn(0x7f09f8071a50 0x7f09f80a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.946+0000 7f09fe342640 1 -- 172.21.15.60:0/3477409373 >> 172.21.15.60:0/3477409373 conn(0x7f09f80b7300 msgr2=0x7f09f80b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.946+0000 7f09fe342640 1 -- 172.21.15.60:0/3477409373 shutdown_connections 2024-03-27T21:41:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.946+0000 7f09fe342640 1 -- 172.21.15.60:0/3477409373 wait complete. 2024-03-27T21:41:05.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.947+0000 7f09fe342640 10 monclient: build_initial_monmap 2024-03-27T21:41:05.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.947+0000 7f09fe342640 10 monclient: monmap: 2024-03-27T21:41:05.946 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:05.946 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:05.946 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:05.946 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:05.946 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:05.946 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:05.946 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:05.946 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:05.946 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:05.946 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.948+0000 7f09fe342640 1 librados: starting msgr at 2024-03-27T21:41:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.948+0000 7f09fe342640 1 librados: starting objecter 2024-03-27T21:41:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.948+0000 7f09fe342640 1 Processor -- start 2024-03-27T21:41:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.948+0000 7f09fe342640 1 -- start start 2024-03-27T21:41:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.948+0000 7f09fe342640 1 librados: setting wanted keys 2024-03-27T21:41:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.948+0000 7f09fe342640 1 librados: calling monclient init 2024-03-27T21:41:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.948+0000 7f09fe342640 10 monclient: init 2024-03-27T21:41:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.949+0000 7f09fe342640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.949+0000 7f09fe342640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:41:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.949+0000 7f09fe342640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09f8071a50 0x7f09f80afd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.949+0000 7f09fe342640 10 monclient(hunting): picked mon.noname-b con 0x7f09f8071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.949+0000 7f09fe342640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09f813e810 0x7f09f8138de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.949+0000 7f09fe342640 10 monclient(hunting): picked mon.noname-a con 0x7f09f813e810 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.949+0000 7f09fe342640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f09f8141630 0x7f09f8139320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.949+0000 7f09fe342640 10 monclient(hunting): picked mon.noname-c con 0x7f09f8141630 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.949+0000 7f09fe342640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.949+0000 7f09fe342640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f80ae560 con 0x7f09f813e810 2024-03-27T21:41:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.949+0000 7f09fe342640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.949+0000 7f09fe342640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f80b02c0 con 0x7f09f8071a50 2024-03-27T21:41:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.949+0000 7f09fe342640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.949+0000 7f09fe342640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f8139860 con 0x7f09f8141630 2024-03-27T21:41:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.949+0000 7f09fe342640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.949+0000 7f09fe342640 10 monclient(hunting): authenticate will time out at 907.170871s 2024-03-27T21:41:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09f813e810 0x7f09f8138de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09f813e810 0x7f09f8138de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44796/0 (socket says 172.21.15.60:44796) 2024-03-27T21:41:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09fc8df640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f09f8141630 0x7f09f8139320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f77fe640 1 -- 172.21.15.60:0/3007456848 learned_addr learned my addr 172.21.15.60:0/3007456848 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f77fe640 10 monclient(hunting): get_auth_request con 0x7f09f813e810 auth_method 0 2024-03-27T21:41:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09fc8df640 10 monclient(hunting): get_auth_request con 0x7f09f8141630 auth_method 0 2024-03-27T21:41:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09fc8df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09fc8df640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09fc8df640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f7fff640 1 --2- 172.21.15.60:0/3007456848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09f8071a50 0x7f09f80afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f7fff640 10 monclient(hunting): get_auth_request con 0x7f09f8071a50 auth_method 0 2024-03-27T21:41:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09fc8df640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09fc8df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09fc8df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f77fe640 10 monclient(hunting): handle_auth_done global_id 4854 payload 824 2024-03-27T21:41:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f77fe640 1 -- 172.21.15.60:0/3007456848 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f09f8141630 msgr2=0x7f09f8139320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f77fe640 1 --2- 172.21.15.60:0/3007456848 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f09f8141630 0x7f09f8139320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f77fe640 1 -- 172.21.15.60:0/3007456848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09f8071a50 msgr2=0x7f09f80afd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f77fe640 1 --2- 172.21.15.60:0/3007456848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09f8071a50 0x7f09f80afd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f77fe640 10 monclient: _finish_hunting 0 2024-03-27T21:41:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f77fe640 1 monclient: found mon.noname-a 2024-03-27T21:41:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f77fe640 1 -- 172.21.15.60:0/3007456848 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ec0094d0 con 0x7f09f813e810 2024-03-27T21:41:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f77fe640 1 --2- 172.21.15.60:0/3007456848 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09f813e810 0x7f09f8138de0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f09ec00e9d0 tx=0x7f09ec00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f4ff9640 10 client.?.objecter ms_handle_connect 0x7f09f813e810 2024-03-27T21:41:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f4ff9640 1 -- 172.21.15.60:0/3007456848 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec00ca60 con 0x7f09f813e810 2024-03-27T21:41:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f4ff9640 10 monclient: dump: 2024-03-27T21:41:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:05.952 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:05.952 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:05.952 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:05.952 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:05.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f4ff9640 10 monclient: _finish_auth 0 2024-03-27T21:41:05.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f4ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:41:05.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:05.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f4ff9640 1 -- 172.21.15.60:0/3007456848 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ec00cbc0 con 0x7f09f813e810 2024-03-27T21:41:05.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09fe342640 5 monclient: authenticate success, global_id 4854 2024-03-27T21:41:05.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:05.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f4ff9640 1 -- 172.21.15.60:0/3007456848 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec013470 con 0x7f09f813e810 2024-03-27T21:41:05.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:05.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:05.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f4ff9640 10 monclient: dump: 2024-03-27T21:41:05.952 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:05.953 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:05.953 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.953 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:05.953 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:05.953 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:05.953 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:05.953 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:05.953 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:05.953 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:05.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09fe342640 10 monclient: _renew_subs 2024-03-27T21:41:05.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09f67fc640 4 set_mon_vals no callback set 2024-03-27T21:41:05.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09fe342640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:05.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09fe342640 1 -- 172.21.15.60:0/3007456848 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f0009a70 con 0x7f09f813e810 2024-03-27T21:41:05.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.950+0000 7f09fe342640 10 client.4854.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:05.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.951+0000 7f09fe342640 10 monclient: _renew_subs 2024-03-27T21:41:05.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.951+0000 7f09fe342640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:05.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.951+0000 7f09fe342640 1 -- 172.21.15.60:0/3007456848 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f8139af0 con 0x7f09f813e810 2024-03-27T21:41:05.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.951+0000 7f09fe342640 1 librados: init done 2024-03-27T21:41:05.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.952+0000 7f09fe342640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:05.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.952+0000 7f09fe342640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:05.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.952+0000 7f09fe342640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:05.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.952+0000 7f09fe342640 1 -- 172.21.15.60:0/3007456848 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f8139d40 con 0x7f09f813e810 2024-03-27T21:41:05.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.952+0000 7f09f4ff9640 1 -- 172.21.15.60:0/3007456848 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f09ec0135d0 con 0x7f09f813e810 2024-03-27T21:41:05.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.952+0000 7f09f4ff9640 1 --2- 172.21.15.60:0/3007456848 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f09e406c590 0x7f09e406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:05.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.953+0000 7f09f4ff9640 1 -- 172.21.15.60:0/3007456848 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f09ec089310 con 0x7f09f813e810 2024-03-27T21:41:05.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.953+0000 7f09f4ff9640 10 client.4854.objecter ms_dispatch 0x7f09f8000b90 osd_map(51..51 src has 1..51) v4 2024-03-27T21:41:05.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.953+0000 7f09f4ff9640 3 client.4854.objecter handle_osd_map got epochs [51,51] > 0 2024-03-27T21:41:05.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.953+0000 7f09f4ff9640 3 client.4854.objecter handle_osd_map decoding full epoch 51 2024-03-27T21:41:05.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.953+0000 7f09f4ff9640 20 client.4854.objecter dump_active .. 0 homeless 2024-03-27T21:41:05.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.953+0000 7f09f7fff640 1 --2- 172.21.15.60:0/3007456848 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f09e406c590 0x7f09e406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:05.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.953+0000 7f09f7fff640 10 monclient: get_auth_request con 0x7f09e406c590 auth_method 0 2024-03-27T21:41:05.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.953+0000 7f09f7fff640 1 --2- 172.21.15.60:0/3007456848 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f09e406c590 0x7f09e406ea50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f09e8006fd0 tx=0x7f09e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:05.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.953+0000 7f09f4ff9640 10 client.4854.objecter ms_handle_connect 0x7f09e406c590 2024-03-27T21:41:05.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.957+0000 7f09f4ff9640 1 -- 172.21.15.60:0/3007456848 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f09ec054800 con 0x7f09f813e810 2024-03-27T21:41:05.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.957+0000 7f09f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:05.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:05.957+0000 7f09f4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:05.956 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:05.956 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:41:05.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:41:05.956 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:41:05.956 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:05.960 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:05.974+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:05.962 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:41:06.061 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:06.075+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:06.076 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:06.077 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:06.077 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:06.077 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:06.086 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:41:06.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.156+0000 7f09fe342640 1 -- 172.21.15.60:0/3007456848 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}) v1 -- 0x7f09f80aa590 con 0x7f09e406c590 2024-03-27T21:41:06.143 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:41:06.156+0000 7f63d8d11640 -1 mgr.server reply reply (2) No such file or directory no change 2024-03-27T21:41:06.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.157+0000 7f09f4ff9640 1 -- 172.21.15.60:0/3007456848 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 2 no change) v1 ==== 17+0+337 (secure 0 0 0) 0x7f09e8010990 con 0x7f09e406c590 2024-03-27T21:41:06.144 INFO:teuthology.orchestra.run.smithi060.stdout:moved 0 / 18 (0%) 2024-03-27T21:41:06.144 INFO:teuthology.orchestra.run.smithi060.stdout:avg 2.57143 2024-03-27T21:41:06.144 INFO:teuthology.orchestra.run.smithi060.stdout:stddev 1.41421 -> 1.41421 (expected baseline 1.48461) 2024-03-27T21:41:06.144 INFO:teuthology.orchestra.run.smithi060.stdout:min osd.4 with 0 -> 0 pgs (0 -> 0 * mean) 2024-03-27T21:41:06.144 INFO:teuthology.orchestra.run.smithi060.stdout:max osd.1 with 4 -> 4 pgs (1.55556 -> 1.55556 * mean) 2024-03-27T21:41:06.144 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:41:06.144 INFO:teuthology.orchestra.run.smithi060.stdout:oload 110 2024-03-27T21:41:06.144 INFO:teuthology.orchestra.run.smithi060.stdout:max_change 3 2024-03-27T21:41:06.144 INFO:teuthology.orchestra.run.smithi060.stdout:max_change_osds 4 2024-03-27T21:41:06.145 INFO:teuthology.orchestra.run.smithi060.stdout:average_utilization 0.0003 2024-03-27T21:41:06.145 INFO:teuthology.orchestra.run.smithi060.stdout:overload_utilization 0.0003 2024-03-27T21:41:06.145 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 weight 1.0000 -> 0.8795 2024-03-27T21:41:06.145 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 weight 1.0000 -> 0.9070 2024-03-27T21:41:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:no change 2024-03-27T21:41:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.158+0000 7f09fe342640 10 librados: watch_flush enter 2024-03-27T21:41:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.158+0000 7f09fe342640 10 librados: watch_flush exit 2024-03-27T21:41:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.158+0000 7f09fe342640 20 client.4854.objecter shutdown clearing up homeless session... 2024-03-27T21:41:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.158+0000 7f09fe342640 10 client.4854.objecter successfully canceled tick 2024-03-27T21:41:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.158+0000 7f09fe342640 1 -- 172.21.15.60:0/3007456848 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f09e406c590 msgr2=0x7f09e406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.158+0000 7f09fe342640 1 --2- 172.21.15.60:0/3007456848 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f09e406c590 0x7f09e406ea50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f09e8006fd0 tx=0x7f09e8008040 comp rx=0 tx=0).stop 2024-03-27T21:41:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.158+0000 7f09fe342640 10 monclient: shutdown 2024-03-27T21:41:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.158+0000 7f09fe342640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.158+0000 7f09fe342640 1 -- 172.21.15.60:0/3007456848 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09f813e810 msgr2=0x7f09f8138de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.158+0000 7f09fe342640 1 --2- 172.21.15.60:0/3007456848 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09f813e810 0x7f09f8138de0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f09ec00e9d0 tx=0x7f09ec00ee90 comp rx=0 tx=0).stop 2024-03-27T21:41:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.159+0000 7f09fe342640 1 -- 172.21.15.60:0/3007456848 shutdown_connections 2024-03-27T21:41:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.159+0000 7f09fe342640 1 --2- 172.21.15.60:0/3007456848 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f09f8141630 0x7f09f8139320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.159+0000 7f09fe342640 1 --2- 172.21.15.60:0/3007456848 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09f813e810 0x7f09f8138de0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.159+0000 7f09fe342640 1 --2- 172.21.15.60:0/3007456848 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f09e406c590 0x7f09e406ea50 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.159+0000 7f09fe342640 1 --2- 172.21.15.60:0/3007456848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09f8071a50 0x7f09f80afd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.159+0000 7f09fe342640 1 -- 172.21.15.60:0/3007456848 >> 172.21.15.60:0/3007456848 conn(0x7f09f80b7300 msgr2=0x7f09f813dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.159+0000 7f09fe342640 1 -- 172.21.15.60:0/3007456848 shutdown_connections 2024-03-27T21:41:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.159+0000 7f09fe342640 1 -- 172.21.15.60:0/3007456848 wait complete. 2024-03-27T21:41:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.159+0000 7f09fe342640 1 librados: shutdown 2024-03-27T21:41:06.156 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-03-27T21:41:06.156 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 3.0 2024-03-27T21:41:06.191 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:06.192 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:06.192 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:06.192 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:06.192 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:06.197 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:06.211+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:06.200 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:41:06.263 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:06.277+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:06.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.316+0000 7f18efbc2640 10 monclient: get_monmap_and_config 2024-03-27T21:41:06.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.316+0000 7f18efbc2640 10 monclient: build_initial_monmap 2024-03-27T21:41:06.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.316+0000 7f18efbc2640 10 monclient: monmap: 2024-03-27T21:41:06.303 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:06.303 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:06.303 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:06.317650+0000 2024-03-27T21:41:06.303 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:06.317650+0000 2024-03-27T21:41:06.303 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:06.303 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:06.303 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:06.303 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:06.303 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:06.303 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:06.303 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:06.304 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:06.309 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:06.309 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:06.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:06.309 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:41:06.310 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:06.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.317+0000 7f18efbc2640 1 Processor -- start 2024-03-27T21:41:06.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.317+0000 7f18efbc2640 1 -- start start 2024-03-27T21:41:06.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.317+0000 7f18efbc2640 10 monclient: init 2024-03-27T21:41:06.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18efbc2640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:06.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18efbc2640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-03-27T21:41:06.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18efbc2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f18e806a1f0 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:06.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18efbc2640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f18e80af850 0x7f18e8071630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:06.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18efbc2640 10 monclient(hunting): picked mon.noname-c con 0x7f18e80af850 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:06.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18efbc2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f18e8071b70 0x7f18e8141a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:06.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18efbc2640 10 monclient(hunting): picked mon.noname-a con 0x7f18e8071b70 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:06.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18efbc2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e806f660 con 0x7f18e806a1f0 2024-03-27T21:41:06.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18efbc2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:06.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18efbc2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e806f480 con 0x7f18e8071b70 2024-03-27T21:41:06.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18efbc2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:06.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18efbc2640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e806f940 con 0x7f18e80af850 2024-03-27T21:41:06.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18efbc2640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:06.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18efbc2640 10 monclient(hunting): authenticate will time out at 907.539039s 2024-03-27T21:41:06.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ee15f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f18e8071b70 0x7f18e8141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:06.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ee15f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f18e8071b70 0x7f18e8141a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44802/0 (socket says 172.21.15.60:44802) 2024-03-27T21:41:06.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ee15f640 1 -- 172.21.15.60:0/2738381453 learned_addr learned my addr 172.21.15.60:0/2738381453 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:06.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ee15f640 10 monclient(hunting): get_auth_request con 0x7f18e8071b70 auth_method 0 2024-03-27T21:41:06.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ee15f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:06.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ee15f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:06.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ee15f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:06.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ed15d640 1 --2- 172.21.15.60:0/2738381453 >> v2:172.21.15.191:3300/0 conn(0x7f18e80af850 0x7f18e8071630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:06.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ee15f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:06.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ee15f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:06.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ee15f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:06.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ed15d640 10 monclient(hunting): get_auth_request con 0x7f18e80af850 auth_method 0 2024-03-27T21:41:06.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ed15d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:06.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ed15d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:06.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ed15d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:06.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ed15d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:06.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ed15d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:06.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ed15d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:06.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ee15f640 10 monclient(hunting): handle_auth_done global_id 4857 payload 274 2024-03-27T21:41:06.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ee15f640 1 -- 172.21.15.60:0/2738381453 >> v2:172.21.15.191:3300/0 conn(0x7f18e80af850 msgr2=0x7f18e8071630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:06.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ee15f640 1 --2- 172.21.15.60:0/2738381453 >> v2:172.21.15.191:3300/0 conn(0x7f18e80af850 0x7f18e8071630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ee15f640 1 -- 172.21.15.60:0/2738381453 >> v1:172.21.15.60:6789/0 conn(0x7f18e806a1f0 legacy=0x7f18e806a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:06.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ee15f640 10 monclient: _finish_hunting 0 2024-03-27T21:41:06.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ee15f640 1 monclient: found mon.noname-a 2024-03-27T21:41:06.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ee15f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:06.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ee15f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:06.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.318+0000 7f18ee15f640 1 -- 172.21.15.60:0/2738381453 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e806fc20 con 0x7f18e8071b70 2024-03-27T21:41:06.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18ee15f640 1 --2- 172.21.15.60:0/2738381453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18e8071b70 0x7f18e8141a80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f18e806f0c0 tx=0x7f18e002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43b2f7a62676d406 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:06.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18ec95c640 1 -- 172.21.15.60:0/2738381453 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e002ca80 con 0x7f18e8071b70 2024-03-27T21:41:06.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18ec95c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:06.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18ec95c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:06.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18ec95c640 10 monclient: dump: 2024-03-27T21:41:06.316 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:06.316 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:06.316 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:06.316 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:06.316 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:06.316 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:06.316 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:06.316 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:06.316 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:06.316 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:06.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18ec95c640 10 monclient: _finish_auth 0 2024-03-27T21:41:06.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18ec95c640 10 monclient: _check_auth_tickets 2024-03-27T21:41:06.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18ec95c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:06.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18ec95c640 1 -- 172.21.15.60:0/2738381453 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e002cbe0 con 0x7f18e8071b70 2024-03-27T21:41:06.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18efbc2640 5 monclient: authenticate success, global_id 4857 2024-03-27T21:41:06.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18efbc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:06.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18ec95c640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:06.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18efbc2640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:06.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18ec95c640 1 -- 172.21.15.60:0/2738381453 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e00343f0 con 0x7f18e8071b70 2024-03-27T21:41:06.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18efbc2640 4 set_mon_vals no callback set 2024-03-27T21:41:06.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18ec95c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:06.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18ec95c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:06.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18ec95c640 10 monclient: dump: 2024-03-27T21:41:06.317 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:06.317 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:06.317 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:06.317 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:06.317 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:06.317 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:06.318 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:06.318 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:06.318 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:06.318 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:06.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18efbc2640 10 monclient: shutdown 2024-03-27T21:41:06.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18efbc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:06.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18efbc2640 1 -- 172.21.15.60:0/2738381453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18e8071b70 msgr2=0x7f18e8141a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:06.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.319+0000 7f18efbc2640 1 --2- 172.21.15.60:0/2738381453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18e8071b70 0x7f18e8141a80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f18e806f0c0 tx=0x7f18e002bee0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.320+0000 7f18efbc2640 1 -- 172.21.15.60:0/2738381453 shutdown_connections 2024-03-27T21:41:06.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.320+0000 7f18efbc2640 1 --2- 172.21.15.60:0/2738381453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18e8071b70 0x7f18e8141a80 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.320+0000 7f18efbc2640 1 --2- 172.21.15.60:0/2738381453 >> v2:172.21.15.191:3300/0 conn(0x7f18e80af850 0x7f18e8071630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.320+0000 7f18efbc2640 1 -- 172.21.15.60:0/2738381453 >> 172.21.15.60:0/2738381453 conn(0x7f18e80b4e50 msgr2=0x7f18e80b5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:06.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.320+0000 7f18efbc2640 1 -- 172.21.15.60:0/2738381453 shutdown_connections 2024-03-27T21:41:06.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.320+0000 7f18efbc2640 1 -- 172.21.15.60:0/2738381453 wait complete. 2024-03-27T21:41:06.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.320+0000 7f18efbc2640 10 monclient: build_initial_monmap 2024-03-27T21:41:06.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.320+0000 7f18efbc2640 10 monclient: monmap: 2024-03-27T21:41:06.319 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:06.319 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:06.319 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:06.319 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:06.319 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:06.319 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:06.319 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:06.319 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:06.319 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:06.319 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:06.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.320+0000 7f18efbc2640 1 librados: starting msgr at 2024-03-27T21:41:06.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.320+0000 7f18efbc2640 1 librados: starting objecter 2024-03-27T21:41:06.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18efbc2640 1 Processor -- start 2024-03-27T21:41:06.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18efbc2640 1 -- start start 2024-03-27T21:41:06.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18efbc2640 1 librados: setting wanted keys 2024-03-27T21:41:06.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18efbc2640 1 librados: calling monclient init 2024-03-27T21:41:06.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18efbc2640 10 monclient: init 2024-03-27T21:41:06.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18efbc2640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:06.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18efbc2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:41:06.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18efbc2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18e806a1f0 0x7f18e80add70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:06.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18efbc2640 10 monclient(hunting): picked mon.noname-b con 0x7f18e806a1f0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:06.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18efbc2640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f18e8071b70 0x7f18e8143fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:06.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18efbc2640 10 monclient(hunting): picked mon.noname-c con 0x7f18e8071b70 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:06.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18efbc2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18e80af850 0x7f18e8144520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:06.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18efbc2640 10 monclient(hunting): picked mon.noname-a con 0x7f18e80af850 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:06.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18efbc2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:06.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18efbc2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e8144c30 con 0x7f18e80af850 2024-03-27T21:41:06.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18efbc2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:06.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18efbc2640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e81d9850 con 0x7f18e806a1f0 2024-03-27T21:41:06.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18efbc2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:06.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18efbc2640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e81d9a10 con 0x7f18e8071b70 2024-03-27T21:41:06.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18efbc2640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:06.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18efbc2640 10 monclient(hunting): authenticate will time out at 907.542572s 2024-03-27T21:41:06.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ee15f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18e80af850 0x7f18e8144520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:06.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ed15d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f18e8071b70 0x7f18e8143fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:06.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ee15f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18e80af850 0x7f18e8144520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44814/0 (socket says 172.21.15.60:44814) 2024-03-27T21:41:06.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ed15d640 10 monclient(hunting): get_auth_request con 0x7f18e8071b70 auth_method 0 2024-03-27T21:41:06.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ee15f640 1 -- 172.21.15.60:0/3764299188 learned_addr learned my addr 172.21.15.60:0/3764299188 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:06.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ed15d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:06.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ed15d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:06.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ed15d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:06.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ee15f640 10 monclient(hunting): get_auth_request con 0x7f18e80af850 auth_method 0 2024-03-27T21:41:06.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ee15f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:06.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ee15f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:06.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ee15f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:06.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ed95e640 1 --2- 172.21.15.60:0/3764299188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18e806a1f0 0x7f18e80add70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:06.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ed15d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:06.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ed15d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:06.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ed15d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:06.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ee15f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:06.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ee15f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:06.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ee15f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:06.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ed95e640 10 monclient(hunting): get_auth_request con 0x7f18e806a1f0 auth_method 0 2024-03-27T21:41:06.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ed95e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:06.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ed95e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:06.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.321+0000 7f18ed95e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:06.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18ed95e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:06.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18ed95e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:06.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18ed95e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:06.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18ed15d640 10 monclient(hunting): handle_auth_done global_id 4691 payload 824 2024-03-27T21:41:06.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18ed15d640 1 -- 172.21.15.60:0/3764299188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18e806a1f0 msgr2=0x7f18e80add70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:06.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18ed15d640 1 --2- 172.21.15.60:0/3764299188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18e806a1f0 0x7f18e80add70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18ed15d640 1 -- 172.21.15.60:0/3764299188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18e80af850 msgr2=0x7f18e8144520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:06.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18ed15d640 1 --2- 172.21.15.60:0/3764299188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18e80af850 0x7f18e8144520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18ed15d640 10 monclient: _finish_hunting 0 2024-03-27T21:41:06.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18ed15d640 1 monclient: found mon.noname-c 2024-03-27T21:41:06.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18ed15d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:06.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18ed15d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:06.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18ed15d640 1 -- 172.21.15.60:0/3764299188 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e0009b30 con 0x7f18e8071b70 2024-03-27T21:41:06.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18ee15f640 1 --2- 172.21.15.60:0/3764299188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18e80af850 0x7f18e8144520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:06.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18ed15d640 1 --2- 172.21.15.60:0/3764299188 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f18e8071b70 0x7f18e8143fe0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f18d800e850 tx=0x7f18d800ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:06.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18de7fc640 10 client.?.objecter ms_handle_connect 0x7f18e8071b70 2024-03-27T21:41:06.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18de7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:06.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18de7fc640 1 -- 172.21.15.60:0/3764299188 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d80095b0 con 0x7f18e8071b70 2024-03-27T21:41:06.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:06.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18de7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:06.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18de7fc640 10 monclient: dump: 2024-03-27T21:41:06.324 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:06.324 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:06.324 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:06.324 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:06.324 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:06.325 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:06.325 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:06.325 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:06.325 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:06.325 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:06.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18de7fc640 10 monclient: _finish_auth 0 2024-03-27T21:41:06.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18de7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:41:06.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:06.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18de7fc640 1 -- 172.21.15.60:0/3764299188 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d800c940 con 0x7f18e8071b70 2024-03-27T21:41:06.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18efbc2640 5 monclient: authenticate success, global_id 4691 2024-03-27T21:41:06.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18efbc2640 10 monclient: _renew_subs 2024-03-27T21:41:06.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18efbc2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:06.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18efbc2640 1 -- 172.21.15.60:0/3764299188 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e81d9c90 con 0x7f18e8071b70 2024-03-27T21:41:06.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:06.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18de7fc640 1 -- 172.21.15.60:0/3764299188 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d800ccd0 con 0x7f18e8071b70 2024-03-27T21:41:06.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:06.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18de7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:06.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18efbc2640 10 client.4691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:06.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18de7fc640 10 monclient: dump: 2024-03-27T21:41:06.326 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:06.326 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:06.326 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:06.326 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:06.326 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:06.326 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:06.326 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:06.326 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:06.326 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:06.326 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18dffff640 4 set_mon_vals no callback set 2024-03-27T21:41:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18efbc2640 10 monclient: _renew_subs 2024-03-27T21:41:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18efbc2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18efbc2640 1 -- 172.21.15.60:0/3764299188 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e81da1f0 con 0x7f18e8071b70 2024-03-27T21:41:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.322+0000 7f18efbc2640 1 librados: init done 2024-03-27T21:41:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.323+0000 7f18efbc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.323+0000 7f18efbc2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.323+0000 7f18efbc2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.323+0000 7f18efbc2640 1 -- 172.21.15.60:0/3764299188 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e8144a60 con 0x7f18e8071b70 2024-03-27T21:41:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.323+0000 7f18de7fc640 1 -- 172.21.15.60:0/3764299188 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f18d8013470 con 0x7f18e8071b70 2024-03-27T21:41:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.324+0000 7f18de7fc640 1 --2- 172.21.15.60:0/3764299188 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f18cc06c590 0x7f18cc06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.324+0000 7f18ed95e640 1 --2- 172.21.15.60:0/3764299188 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f18cc06c590 0x7f18cc06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.324+0000 7f18de7fc640 1 -- 172.21.15.60:0/3764299188 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f18d8089220 con 0x7f18e8071b70 2024-03-27T21:41:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.324+0000 7f18de7fc640 10 client.4691.objecter ms_dispatch 0x7f18e8000b90 osd_map(51..51 src has 1..51) v4 2024-03-27T21:41:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.324+0000 7f18ed95e640 10 monclient: get_auth_request con 0x7f18cc06c590 auth_method 0 2024-03-27T21:41:06.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.324+0000 7f18de7fc640 3 client.4691.objecter handle_osd_map got epochs [51,51] > 0 2024-03-27T21:41:06.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.324+0000 7f18de7fc640 3 client.4691.objecter handle_osd_map decoding full epoch 51 2024-03-27T21:41:06.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.324+0000 7f18de7fc640 20 client.4691.objecter dump_active .. 0 homeless 2024-03-27T21:41:06.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.325+0000 7f18ed95e640 1 --2- 172.21.15.60:0/3764299188 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f18cc06c590 0x7f18cc06ea50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f18e4000f60 tx=0x7f18e4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:06.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.325+0000 7f18de7fc640 10 client.4691.objecter ms_handle_connect 0x7f18cc06c590 2024-03-27T21:41:06.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.328+0000 7f18de7fc640 1 -- 172.21.15.60:0/3764299188 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f18d8054710 con 0x7f18e8071b70 2024-03-27T21:41:06.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.328+0000 7f18de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:06.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.328+0000 7f18de7fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:06.329 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:41:06.370 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:06.385+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:06.467 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:06.482+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:06.476 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:06.476 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:06.476 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:06.476 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:06.484 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:41:06.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.574+0000 7f18efbc2640 1 -- 172.21.15.60:0/3764299188 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}) v1 -- 0x7f18e81d9e20 con 0x7f18cc06c590 2024-03-27T21:41:06.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.574+0000 7f18de7fc640 1 -- 172.21.15.60:0/3764299188 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 no change) v1 ==== 17+0+278 (secure 0 0 0) 0x7f18e81d9e20 con 0x7f18cc06c590 2024-03-27T21:41:06.561 INFO:teuthology.orchestra.run.smithi060.stdout:moved 0 / 18 (0%) 2024-03-27T21:41:06.561 INFO:teuthology.orchestra.run.smithi060.stdout:avg 2.57143 2024-03-27T21:41:06.561 INFO:teuthology.orchestra.run.smithi060.stdout:stddev 1.41421 -> 1.41421 (expected baseline 1.48461) 2024-03-27T21:41:06.561 INFO:teuthology.orchestra.run.smithi060.stdout:min osd.4 with 0 -> 0 pgs (0 -> 0 * mean) 2024-03-27T21:41:06.561 INFO:teuthology.orchestra.run.smithi060.stdout:max osd.1 with 4 -> 4 pgs (1.55556 -> 1.55556 * mean) 2024-03-27T21:41:06.561 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:41:06.561 INFO:teuthology.orchestra.run.smithi060.stdout:oload 1000 2024-03-27T21:41:06.561 INFO:teuthology.orchestra.run.smithi060.stdout:max_change 3 2024-03-27T21:41:06.561 INFO:teuthology.orchestra.run.smithi060.stdout:max_change_osds 4 2024-03-27T21:41:06.561 INFO:teuthology.orchestra.run.smithi060.stdout:average_utilization 0.0003 2024-03-27T21:41:06.561 INFO:teuthology.orchestra.run.smithi060.stdout:overload_utilization 0.0030 2024-03-27T21:41:06.562 INFO:teuthology.orchestra.run.smithi060.stderr:no change 2024-03-27T21:41:06.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.576+0000 7f18efbc2640 10 librados: watch_flush enter 2024-03-27T21:41:06.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.576+0000 7f18efbc2640 10 librados: watch_flush exit 2024-03-27T21:41:06.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.576+0000 7f18efbc2640 20 client.4691.objecter shutdown clearing up homeless session... 2024-03-27T21:41:06.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.576+0000 7f18efbc2640 10 client.4691.objecter successfully canceled tick 2024-03-27T21:41:06.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.576+0000 7f18efbc2640 1 -- 172.21.15.60:0/3764299188 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f18cc06c590 msgr2=0x7f18cc06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:06.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.576+0000 7f18efbc2640 1 --2- 172.21.15.60:0/3764299188 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f18cc06c590 0x7f18cc06ea50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f18e4000f60 tx=0x7f18e4009040 comp rx=0 tx=0).stop 2024-03-27T21:41:06.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.576+0000 7f18efbc2640 10 monclient: shutdown 2024-03-27T21:41:06.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.576+0000 7f18efbc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:06.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.576+0000 7f18efbc2640 1 -- 172.21.15.60:0/3764299188 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f18e8071b70 msgr2=0x7f18e8143fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:06.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.576+0000 7f18efbc2640 1 --2- 172.21.15.60:0/3764299188 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f18e8071b70 0x7f18e8143fe0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f18d800e850 tx=0x7f18d800ed10 comp rx=0 tx=0).stop 2024-03-27T21:41:06.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.576+0000 7f18efbc2640 1 -- 172.21.15.60:0/3764299188 shutdown_connections 2024-03-27T21:41:06.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.576+0000 7f18efbc2640 1 --2- 172.21.15.60:0/3764299188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18e80af850 0x7f18e8144520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.576+0000 7f18efbc2640 1 --2- 172.21.15.60:0/3764299188 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f18e8071b70 0x7f18e8143fe0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.576+0000 7f18efbc2640 1 --2- 172.21.15.60:0/3764299188 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f18cc06c590 0x7f18cc06ea50 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.576+0000 7f18efbc2640 1 --2- 172.21.15.60:0/3764299188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18e806a1f0 0x7f18e80add70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.576+0000 7f18efbc2640 1 -- 172.21.15.60:0/3764299188 >> 172.21.15.60:0/3764299188 conn(0x7f18e80b4e50 msgr2=0x7f18e813a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:06.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.576+0000 7f18efbc2640 1 -- 172.21.15.60:0/3764299188 shutdown_connections 2024-03-27T21:41:06.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.576+0000 7f18efbc2640 1 -- 172.21.15.60:0/3764299188 wait complete. 2024-03-27T21:41:06.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.576+0000 7f18efbc2640 1 librados: shutdown 2024-03-27T21:41:06.570 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:06.585+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:06.574 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-03-27T21:41:06.574 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 0.05 2024-03-27T21:41:06.629 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:06.629 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:06.629 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:06.629 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:06.629 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:06.638 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:41:06.671 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:06.685+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:06.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.696+0000 7f4018b92640 10 monclient: get_monmap_and_config 2024-03-27T21:41:06.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.696+0000 7f4018b92640 10 monclient: build_initial_monmap 2024-03-27T21:41:06.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.696+0000 7f4018b92640 10 monclient: monmap: 2024-03-27T21:41:06.683 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:06.683 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:06.683 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:06.697598+0000 2024-03-27T21:41:06.683 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:06.697598+0000 2024-03-27T21:41:06.683 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:06.683 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:06.683 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:06.683 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:06.683 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:06.683 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:06.683 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:06.683 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:06.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.697+0000 7f4018b92640 1 Processor -- start 2024-03-27T21:41:06.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.697+0000 7f4018b92640 1 -- start start 2024-03-27T21:41:06.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.697+0000 7f4018b92640 10 monclient: init 2024-03-27T21:41:06.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.697+0000 7f4018b92640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:06.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.697+0000 7f4018b92640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-03-27T21:41:06.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.697+0000 7f4018b92640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f40140b40b0 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:06.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.697+0000 7f4018b92640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f401413b020 0x7f401413b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:06.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4018b92640 10 monclient(hunting): picked mon.noname-b con 0x7f401413b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:06.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4018b92640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f40141416f0 0x7f4014141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:06.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4018b92640 10 monclient(hunting): picked mon.noname-c con 0x7f40141416f0 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:06.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4018b92640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40140b3830 con 0x7f40140b40b0 2024-03-27T21:41:06.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4018b92640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:06.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4018b92640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40140b4490 con 0x7f40141416f0 2024-03-27T21:41:06.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4018b92640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:06.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4018b92640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40140aa520 con 0x7f401413b020 2024-03-27T21:41:06.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4018b92640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:06.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4018b92640 10 monclient(hunting): authenticate will time out at 907.918956s 2024-03-27T21:41:06.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4011d9b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f401413b020 0x7f401413b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:06.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4011d9b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f401413b020 0x7f401413b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:40056/0 (socket says 172.21.15.60:40056) 2024-03-27T21:41:06.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4011d9b640 1 -- 172.21.15.60:0/293930043 learned_addr learned my addr 172.21.15.60:0/293930043 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:06.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4011d9b640 10 monclient(hunting): get_auth_request con 0x7f401413b020 auth_method 0 2024-03-27T21:41:06.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4011d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:06.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4011d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:06.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4011d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:06.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4012d9d640 1 --2- 172.21.15.60:0/293930043 >> v2:172.21.15.191:3300/0 conn(0x7f40141416f0 0x7f4014141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:06.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4012d9d640 10 monclient(hunting): get_auth_request con 0x7f40141416f0 auth_method 0 2024-03-27T21:41:06.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4012d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:06.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4012d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:06.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4012d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:06.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4011d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:06.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4011d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:06.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4011d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:06.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4012d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:06.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4012d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4012d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4011d9b640 10 monclient(hunting): handle_auth_done global_id 4694 payload 274 2024-03-27T21:41:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4011d9b640 1 -- 172.21.15.60:0/293930043 >> v2:172.21.15.191:3300/0 conn(0x7f40141416f0 msgr2=0x7f4014141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4011d9b640 1 --2- 172.21.15.60:0/293930043 >> v2:172.21.15.191:3300/0 conn(0x7f40141416f0 0x7f4014141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4011d9b640 1 -- 172.21.15.60:0/293930043 >> v1:172.21.15.191:6789/0 conn(0x7f40140b40b0 legacy=0x7f40140a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4011d9b640 10 monclient: _finish_hunting 0 2024-03-27T21:41:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4011d9b640 1 monclient: found mon.noname-b 2024-03-27T21:41:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4011d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4011d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.698+0000 7f4011d9b640 1 -- 172.21.15.60:0/293930043 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40140a9470 con 0x7f401413b020 2024-03-27T21:41:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.699+0000 7f4011d9b640 1 --2- 172.21.15.60:0/293930043 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f401413b020 0x7f401413b480 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4008009870 tx=0x7f400802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a44c84b6a623d93 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.699+0000 7f401159a640 1 -- 172.21.15.60:0/293930043 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4008030840 con 0x7f401413b020 2024-03-27T21:41:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.699+0000 7f401159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.699+0000 7f401159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.699+0000 7f401159a640 10 monclient: dump: 2024-03-27T21:41:06.697 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:06.697 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:06.697 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:06.697 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:06.697 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:06.697 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:06.697 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:06.697 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:06.697 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:06.697 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:06.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.699+0000 7f401159a640 10 monclient: _finish_auth 0 2024-03-27T21:41:06.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.699+0000 7f401159a640 10 monclient: _check_auth_tickets 2024-03-27T21:41:06.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.699+0000 7f401159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:06.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.699+0000 7f401159a640 1 -- 172.21.15.60:0/293930043 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40080309a0 con 0x7f401413b020 2024-03-27T21:41:06.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.699+0000 7f401159a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:06.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.699+0000 7f401159a640 1 -- 172.21.15.60:0/293930043 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4008030d30 con 0x7f401413b020 2024-03-27T21:41:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.699+0000 7f4018b92640 5 monclient: authenticate success, global_id 4694 2024-03-27T21:41:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.699+0000 7f4018b92640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.699+0000 7f4018b92640 4 set_mon_vals no callback set 2024-03-27T21:41:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.699+0000 7f401159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.699+0000 7f401159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.699+0000 7f401159a640 10 monclient: dump: 2024-03-27T21:41:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:06.699 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:06.699 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:06.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.699+0000 7f4018b92640 10 monclient: shutdown 2024-03-27T21:41:06.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.699+0000 7f4018b92640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:06.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.699+0000 7f4018b92640 1 -- 172.21.15.60:0/293930043 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f401413b020 msgr2=0x7f401413b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:06.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.699+0000 7f4018b92640 1 --2- 172.21.15.60:0/293930043 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f401413b020 0x7f401413b480 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4008009870 tx=0x7f400802b980 comp rx=0 tx=0).stop 2024-03-27T21:41:06.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.700+0000 7f4018b92640 1 -- 172.21.15.60:0/293930043 shutdown_connections 2024-03-27T21:41:06.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.700+0000 7f4018b92640 1 --2- 172.21.15.60:0/293930043 >> v2:172.21.15.191:3300/0 conn(0x7f40141416f0 0x7f4014141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.700+0000 7f4018b92640 1 --2- 172.21.15.60:0/293930043 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f401413b020 0x7f401413b480 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.700+0000 7f4018b92640 1 -- 172.21.15.60:0/293930043 >> 172.21.15.60:0/293930043 conn(0x7f40140608e0 msgr2=0x7f4014060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:06.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.700+0000 7f4018b92640 1 -- 172.21.15.60:0/293930043 shutdown_connections 2024-03-27T21:41:06.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.700+0000 7f4018b92640 1 -- 172.21.15.60:0/293930043 wait complete. 2024-03-27T21:41:06.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.701+0000 7f4018b92640 10 monclient: build_initial_monmap 2024-03-27T21:41:06.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.701+0000 7f4018b92640 10 monclient: monmap: 2024-03-27T21:41:06.700 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:06.700 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:06.700 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:06.700 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:06.701 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:06.701 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:06.701 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:06.701 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:06.701 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:06.701 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:06.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.701+0000 7f4018b92640 1 librados: starting msgr at 2024-03-27T21:41:06.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.701+0000 7f4018b92640 1 librados: starting objecter 2024-03-27T21:41:06.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.701+0000 7f4018b92640 1 Processor -- start 2024-03-27T21:41:06.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.701+0000 7f4018b92640 1 -- start start 2024-03-27T21:41:06.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.701+0000 7f4018b92640 1 librados: setting wanted keys 2024-03-27T21:41:06.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.701+0000 7f4018b92640 1 librados: calling monclient init 2024-03-27T21:41:06.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.701+0000 7f4018b92640 10 monclient: init 2024-03-27T21:41:06.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4018b92640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:06.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4018b92640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:41:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4018b92640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40140b40b0 0x7f40140aac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4018b92640 10 monclient(hunting): picked mon.noname-b con 0x7f40140b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4018b92640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f401413b020 0x7f40141d66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4018b92640 10 monclient(hunting): picked mon.noname-a con 0x7f401413b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4018b92640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f40141416f0 0x7f40141d6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4018b92640 10 monclient(hunting): picked mon.noname-c con 0x7f40141416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4018b92640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4018b92640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40141d7170 con 0x7f401413b020 2024-03-27T21:41:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4018b92640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4018b92640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40140abfa0 con 0x7f40140b40b0 2024-03-27T21:41:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4018b92640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4018b92640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f401413d310 con 0x7f40141416f0 2024-03-27T21:41:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4018b92640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4018b92640 10 monclient(hunting): authenticate will time out at 907.923203s 2024-03-27T21:41:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4012d9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f40141416f0 0x7f40141d6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4011d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f401413b020 0x7f40141d66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f401259c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40140b40b0 0x7f40140aac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4012d9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f40141416f0 0x7f40141d6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:40072/0 (socket says 172.21.15.60:40072) 2024-03-27T21:41:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4012d9d640 1 -- 172.21.15.60:0/2765447307 learned_addr learned my addr 172.21.15.60:0/2765447307 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4012d9d640 10 monclient(hunting): get_auth_request con 0x7f40141416f0 auth_method 0 2024-03-27T21:41:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4012d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4012d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4012d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f401259c640 10 monclient(hunting): get_auth_request con 0x7f40140b40b0 auth_method 0 2024-03-27T21:41:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f401259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f401259c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f401259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4011d9b640 10 monclient(hunting): get_auth_request con 0x7f401413b020 auth_method 0 2024-03-27T21:41:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4011d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4011d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:06.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4011d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:06.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4012d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:06.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4012d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:06.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4011d9b640 1 -- 172.21.15.60:0/2765447307 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f401413b020 msgr2=0x7f40141d66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-27T21:41:06.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4011d9b640 1 -- 172.21.15.60:0/2765447307 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f401413b020 msgr2=0x7f40141d66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-27T21:41:06.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4012d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:06.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4011d9b640 1 --2- 172.21.15.60:0/2765447307 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f401413b020 0x7f40141d66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-27T21:41:06.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f401259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:06.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f401259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:06.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4011d9b640 1 --2- 172.21.15.60:0/2765447307 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f401413b020 0x7f40141d66f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-27T21:41:06.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f401259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:06.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4012d9d640 10 monclient(hunting): handle_auth_done global_id 4697 payload 824 2024-03-27T21:41:06.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4012d9d640 1 -- 172.21.15.60:0/2765447307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40140b40b0 msgr2=0x7f40140aac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:06.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4012d9d640 1 --2- 172.21.15.60:0/2765447307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40140b40b0 0x7f40140aac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4012d9d640 1 -- 172.21.15.60:0/2765447307 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f401413b020 msgr2=0x7f40141d66f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T21:41:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4012d9d640 1 --2- 172.21.15.60:0/2765447307 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f401413b020 0x7f40141d66f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4012d9d640 10 monclient: _finish_hunting 0 2024-03-27T21:41:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4012d9d640 1 monclient: found mon.noname-c 2024-03-27T21:41:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4012d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4012d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4012d9d640 1 -- 172.21.15.60:0/2765447307 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40080094d0 con 0x7f40141416f0 2024-03-27T21:41:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f401259c640 1 --2- 172.21.15.60:0/2765447307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40140b40b0 0x7f40140aac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4012d9d640 1 --2- 172.21.15.60:0/2765447307 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f40141416f0 0x7f40141d6c30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f400c00efc0 tx=0x7f400c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4002ffd640 10 client.?.objecter ms_handle_connect 0x7f40141416f0 2024-03-27T21:41:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.702+0000 7f4002ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4002ffd640 1 -- 172.21.15.60:0/2765447307 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c014480 con 0x7f40141416f0 2024-03-27T21:41:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4002ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4002ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4002ffd640 10 monclient: dump: 2024-03-27T21:41:06.706 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:06.706 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:06.706 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:06.706 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:06.706 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:06.706 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:06.706 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:06.706 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:06.706 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:06.706 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:06.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4002ffd640 10 monclient: _finish_auth 0 2024-03-27T21:41:06.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4002ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:41:06.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4002ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:06.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4002ffd640 1 -- 172.21.15.60:0/2765447307 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400c00f040 con 0x7f40141416f0 2024-03-27T21:41:06.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4018b92640 5 monclient: authenticate success, global_id 4697 2024-03-27T21:41:06.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4018b92640 10 monclient: _renew_subs 2024-03-27T21:41:06.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4018b92640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:06.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4018b92640 1 -- 172.21.15.60:0/2765447307 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f401413b9c0 con 0x7f40141416f0 2024-03-27T21:41:06.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4002ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:06.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4002ffd640 1 -- 172.21.15.60:0/2765447307 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c014820 con 0x7f40141416f0 2024-03-27T21:41:06.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4002ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:06.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4002ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:06.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4002ffd640 10 monclient: dump: 2024-03-27T21:41:06.707 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:06.707 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:06.707 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:06.707 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:06.707 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:06.707 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:06.708 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:06.708 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:06.708 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:06.708 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:06.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4010d99640 4 set_mon_vals no callback set 2024-03-27T21:41:06.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4018b92640 10 client.4697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:06.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4018b92640 10 monclient: _renew_subs 2024-03-27T21:41:06.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4018b92640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:06.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4018b92640 1 -- 172.21.15.60:0/2765447307 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401413e530 con 0x7f40141416f0 2024-03-27T21:41:06.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.703+0000 7f4018b92640 1 librados: init done 2024-03-27T21:41:06.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.704+0000 7f4018b92640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:06.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.704+0000 7f4018b92640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:06.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.704+0000 7f4018b92640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:06.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.704+0000 7f4018b92640 1 -- 172.21.15.60:0/2765447307 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4014069470 con 0x7f40141416f0 2024-03-27T21:41:06.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.704+0000 7f4002ffd640 1 -- 172.21.15.60:0/2765447307 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f400c014980 con 0x7f40141416f0 2024-03-27T21:41:06.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.705+0000 7f4002ffd640 1 --2- 172.21.15.60:0/2765447307 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3ff406c590 0x7f3ff406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:06.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.705+0000 7f4002ffd640 1 -- 172.21.15.60:0/2765447307 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f400c089c00 con 0x7f40141416f0 2024-03-27T21:41:06.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.705+0000 7f4002ffd640 10 client.4697.objecter ms_dispatch 0x7f4014000b90 osd_map(51..51 src has 1..51) v4 2024-03-27T21:41:06.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.705+0000 7f4002ffd640 3 client.4697.objecter handle_osd_map got epochs [51,51] > 0 2024-03-27T21:41:06.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.705+0000 7f4002ffd640 3 client.4697.objecter handle_osd_map decoding full epoch 51 2024-03-27T21:41:06.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.705+0000 7f401259c640 1 --2- 172.21.15.60:0/2765447307 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3ff406c590 0x7f3ff406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:06.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.705+0000 7f401259c640 10 monclient: get_auth_request con 0x7f3ff406c590 auth_method 0 2024-03-27T21:41:06.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.705+0000 7f4002ffd640 20 client.4697.objecter dump_active .. 0 homeless 2024-03-27T21:41:06.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.705+0000 7f4002ffd640 10 client.4697.objecter ms_handle_connect 0x7f3ff406c590 2024-03-27T21:41:06.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.705+0000 7f401259c640 1 --2- 172.21.15.60:0/2765447307 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3ff406c590 0x7f3ff406ea50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4004000f60 tx=0x7f4004009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:06.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.709+0000 7f4002ffd640 1 -- 172.21.15.60:0/2765447307 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f400c090080 con 0x7f40141416f0 2024-03-27T21:41:06.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.709+0000 7f4002ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:06.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.709+0000 7f4002ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:06.774 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:06.789+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:06.783 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:06.783 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:06.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:06.783 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:41:06.783 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:06.792 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:41:06.875 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:06.889+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:06.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.937+0000 7f4018b92640 1 -- 172.21.15.60:0/2765447307 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}) v1 -- 0x7f401413e780 con 0x7f3ff406c590 2024-03-27T21:41:06.924 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-27T21:41:06.937+0000 7f63d8d11640 -1 mgr.server reply reply (2) No such file or directory no change 2024-03-27T21:41:06.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.938+0000 7f4002ffd640 1 -- 172.21.15.60:0/2765447307 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 2 no change) v1 ==== 17+0+340 (secure 0 0 0) 0x7f401413e780 con 0x7f3ff406c590 2024-03-27T21:41:06.924 INFO:teuthology.orchestra.run.smithi060.stdout:moved 0 / 18 (0%) 2024-03-27T21:41:06.925 INFO:teuthology.orchestra.run.smithi060.stdout:avg 2.57143 2024-03-27T21:41:06.925 INFO:teuthology.orchestra.run.smithi060.stdout:stddev 1.41421 -> 1.41421 (expected baseline 1.48461) 2024-03-27T21:41:06.925 INFO:teuthology.orchestra.run.smithi060.stdout:min osd.4 with 0 -> 0 pgs (0 -> 0 * mean) 2024-03-27T21:41:06.925 INFO:teuthology.orchestra.run.smithi060.stdout:max osd.1 with 4 -> 4 pgs (1.55556 -> 1.55556 * mean) 2024-03-27T21:41:06.925 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:41:06.925 INFO:teuthology.orchestra.run.smithi060.stdout:oload 110 2024-03-27T21:41:06.925 INFO:teuthology.orchestra.run.smithi060.stdout:max_change 0.05 2024-03-27T21:41:06.925 INFO:teuthology.orchestra.run.smithi060.stdout:max_change_osds 4 2024-03-27T21:41:06.925 INFO:teuthology.orchestra.run.smithi060.stdout:average_utilization 0.0003 2024-03-27T21:41:06.925 INFO:teuthology.orchestra.run.smithi060.stdout:overload_utilization 0.0003 2024-03-27T21:41:06.925 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 weight 1.0000 -> 0.9500 2024-03-27T21:41:06.925 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 weight 1.0000 -> 0.9500 2024-03-27T21:41:06.927 INFO:teuthology.orchestra.run.smithi060.stderr:no change 2024-03-27T21:41:06.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.939+0000 7f4018b92640 10 librados: watch_flush enter 2024-03-27T21:41:06.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.939+0000 7f4018b92640 10 librados: watch_flush exit 2024-03-27T21:41:06.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.939+0000 7f4018b92640 20 client.4697.objecter shutdown clearing up homeless session... 2024-03-27T21:41:06.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.939+0000 7f4018b92640 10 client.4697.objecter successfully canceled tick 2024-03-27T21:41:06.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.939+0000 7f4018b92640 1 -- 172.21.15.60:0/2765447307 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3ff406c590 msgr2=0x7f3ff406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:06.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.939+0000 7f4018b92640 1 --2- 172.21.15.60:0/2765447307 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3ff406c590 0x7f3ff406ea50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4004000f60 tx=0x7f4004009040 comp rx=0 tx=0).stop 2024-03-27T21:41:06.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.939+0000 7f4018b92640 10 monclient: shutdown 2024-03-27T21:41:06.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.939+0000 7f4018b92640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:06.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.939+0000 7f4018b92640 1 -- 172.21.15.60:0/2765447307 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f40141416f0 msgr2=0x7f40141d6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:06.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.939+0000 7f4018b92640 1 --2- 172.21.15.60:0/2765447307 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f40141416f0 0x7f40141d6c30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f400c00efc0 tx=0x7f400c00c490 comp rx=0 tx=0).stop 2024-03-27T21:41:06.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.939+0000 7f4018b92640 1 -- 172.21.15.60:0/2765447307 shutdown_connections 2024-03-27T21:41:06.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.939+0000 7f4018b92640 1 --2- 172.21.15.60:0/2765447307 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f40141416f0 0x7f40141d6c30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.939+0000 7f4018b92640 1 --2- 172.21.15.60:0/2765447307 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f401413b020 0x7f40141d66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.939+0000 7f4018b92640 1 --2- 172.21.15.60:0/2765447307 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3ff406c590 0x7f3ff406ea50 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.939+0000 7f4018b92640 1 --2- 172.21.15.60:0/2765447307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40140b40b0 0x7f40140aac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:06.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.939+0000 7f4018b92640 1 -- 172.21.15.60:0/2765447307 >> 172.21.15.60:0/2765447307 conn(0x7f40140608e0 msgr2=0x7f4014140c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:06.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.940+0000 7f4018b92640 1 -- 172.21.15.60:0/2765447307 shutdown_connections 2024-03-27T21:41:06.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.940+0000 7f4018b92640 1 -- 172.21.15.60:0/2765447307 wait complete. 2024-03-27T21:41:06.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:06.940+0000 7f4018b92640 1 librados: shutdown 2024-03-27T21:41:06.938 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-03-27T21:41:06.938 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 3.0 2024-03-27T21:41:06.938 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:06.938 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:06.938 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:06.939 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:06.947 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:41:06.965 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-27T21:41:06.990 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:07.005+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:07.052 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:07.053 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:07.053 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:07.053 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:07.053 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:07.061 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:41:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.102+0000 7fd48b813640 10 monclient: get_monmap_and_config 2024-03-27T21:41:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.102+0000 7fd48b813640 10 monclient: build_initial_monmap 2024-03-27T21:41:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.102+0000 7fd48b813640 10 monclient: monmap: 2024-03-27T21:41:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:07.103202+0000 2024-03-27T21:41:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:07.103202+0000 2024-03-27T21:41:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:07.089 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:07.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.102+0000 7fd48b813640 1 Processor -- start 2024-03-27T21:41:07.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.102+0000 7fd48b813640 1 -- start start 2024-03-27T21:41:07.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.102+0000 7fd48b813640 10 monclient: init 2024-03-27T21:41:07.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd48b813640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:07.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd48b813640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-03-27T21:41:07.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd48b813640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd4840b40b0 0x7fd4840a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:07.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd48b813640 10 monclient(hunting): picked mon.noname-b con 0x7fd4840b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:07.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd48b813640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd48413b020 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:07.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd48b813640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd48413b480 0x7fd484141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:07.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd48b813640 10 monclient(hunting): picked mon.noname-c con 0x7fd48413b480 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:07.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd48b813640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4840b3830 con 0x7fd48413b020 2024-03-27T21:41:07.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd48b813640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:07.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd48b813640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd48413b910 con 0x7fd48413b480 2024-03-27T21:41:07.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd48b813640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:07.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd48b813640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4840b4490 con 0x7fd4840b40b0 2024-03-27T21:41:07.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd48b813640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:07.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd48b813640 10 monclient(hunting): authenticate will time out at 908.324664s 2024-03-27T21:41:07.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd4895af640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd4840b40b0 0x7fd4840a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:07.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd4895af640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd4840b40b0 0x7fd4840a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:40086/0 (socket says 172.21.15.60:40086) 2024-03-27T21:41:07.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd4895af640 1 -- 172.21.15.60:0/3736001536 learned_addr learned my addr 172.21.15.60:0/3736001536 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:07.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd4895af640 10 monclient(hunting): get_auth_request con 0x7fd4840b40b0 auth_method 0 2024-03-27T21:41:07.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd4895af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:07.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd4895af640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:07.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd4895af640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7f8ffc069640 10 monclient: get_monmap_and_config 2024-03-27T21:41:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7f8ffc069640 10 monclient: build_initial_monmap 2024-03-27T21:41:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7f8ffc069640 10 monclient: monmap: 2024-03-27T21:41:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:07.104838+0000 2024-03-27T21:41:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:07.104838+0000 2024-03-27T21:41:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:07.094 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:07.094 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:07.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd4895af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:07.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd4895af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:07.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd489db0640 1 --2- 172.21.15.60:0/3736001536 >> v2:172.21.15.191:3300/0 conn(0x7fd48413b480 0x7fd484141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:07.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.103+0000 7fd4895af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:07.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.104+0000 7fd489db0640 10 monclient(hunting): get_auth_request con 0x7fd48413b480 auth_method 0 2024-03-27T21:41:07.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.104+0000 7fd489db0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:07.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.104+0000 7fd489db0640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:07.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.104+0000 7fd489db0640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:07.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.104+0000 7fd4895af640 10 monclient(hunting): handle_auth_done global_id 4700 payload 274 2024-03-27T21:41:07.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.104+0000 7fd4895af640 1 -- 172.21.15.60:0/3736001536 >> v2:172.21.15.191:3300/0 conn(0x7fd48413b480 msgr2=0x7fd484141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:07.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.104+0000 7fd4895af640 1 --2- 172.21.15.60:0/3736001536 >> v2:172.21.15.191:3300/0 conn(0x7fd48413b480 0x7fd484141e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:07.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.104+0000 7fd4895af640 1 -- 172.21.15.60:0/3736001536 >> v1:172.21.15.191:6789/0 conn(0x7fd48413b020 legacy=0x7fd4841416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:07.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.104+0000 7fd4895af640 10 monclient: _finish_hunting 0 2024-03-27T21:41:07.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.104+0000 7fd4895af640 1 monclient: found mon.noname-b 2024-03-27T21:41:07.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.104+0000 7fd4895af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:07.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.104+0000 7fd4895af640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:07.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.104+0000 7fd4895af640 1 -- 172.21.15.60:0/3736001536 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4840aa520 con 0x7fd4840b40b0 2024-03-27T21:41:07.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.104+0000 7fd489db0640 1 --2- 172.21.15.60:0/3736001536 >> v2:172.21.15.191:3300/0 conn(0x7fd48413b480 0x7fd484141e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:41:07.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.104+0000 7f8ffc069640 1 Processor -- start 2024-03-27T21:41:07.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.104+0000 7f8ffc069640 1 -- start start 2024-03-27T21:41:07.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.104+0000 7f8ffc069640 10 monclient: init 2024-03-27T21:41:07.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7fd4895af640 1 --2- 172.21.15.60:0/3736001536 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd4840b40b0 0x7fd4840a8b90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd480009870 tx=0x7fd48002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88dd80a568de3846 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:07.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffc069640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:07.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffc069640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-03-27T21:41:07.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffc069640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8ff40b40b0 0x7f8ff40a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:07.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffc069640 10 monclient(hunting): picked mon.noname-b con 0x7f8ff40b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:07.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffc069640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8ff413b020 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:07.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffc069640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8ff413b480 0x7f8ff4141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:07.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ff9e05640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8ff40b40b0 0x7f8ff40a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:07.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffc069640 10 monclient(hunting): picked mon.noname-a con 0x7f8ff413b480 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:07.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7fd47bfff640 1 -- 172.21.15.60:0/3736001536 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd480031640 con 0x7fd4840b40b0 2024-03-27T21:41:07.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7fd47bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:07.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7fd47bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:07.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7fd47bfff640 10 monclient: dump: 2024-03-27T21:41:07.100 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:07.100 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:07.100 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:07.100 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:07.100 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:07.100 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:07.100 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:07.100 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:07.100 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:07.100 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:07.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7fd47bfff640 10 monclient: _finish_auth 0 2024-03-27T21:41:07.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7fd47bfff640 10 monclient: _check_auth_tickets 2024-03-27T21:41:07.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7fd47bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:07.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7fd47bfff640 1 -- 172.21.15.60:0/3736001536 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4800317a0 con 0x7fd4840b40b0 2024-03-27T21:41:07.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7fd47bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:07.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7fd47bfff640 1 -- 172.21.15.60:0/3736001536 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd480031b30 con 0x7fd4840b40b0 2024-03-27T21:41:07.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7fd47bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:07.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7fd47bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:07.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ff9e05640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8ff40b40b0 0x7f8ff40a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:40102/0 (socket says 172.21.15.60:40102) 2024-03-27T21:41:07.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ff9e05640 1 -- 172.21.15.60:0/3607296112 learned_addr learned my addr 172.21.15.60:0/3607296112 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7fd47bfff640 10 monclient: dump: 2024-03-27T21:41:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:07.102 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7fd48b813640 5 monclient: authenticate success, global_id 4700 2024-03-27T21:41:07.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7fd48b813640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:07.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7fd48b813640 4 set_mon_vals no callback set 2024-03-27T21:41:07.104 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:07.118+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:07.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffc069640 1 -- 172.21.15.60:0/3607296112 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff40b3830 con 0x7f8ff413b020 2024-03-27T21:41:07.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffc069640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffc069640 1 -- 172.21.15.60:0/3607296112 --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff413b910 con 0x7f8ff413b480 2024-03-27T21:41:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffc069640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffc069640 1 -- 172.21.15.60:0/3607296112 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40b4490 con 0x7f8ff40b40b0 2024-03-27T21:41:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffc069640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffc069640 10 monclient(hunting): authenticate will time out at 908.326647s 2024-03-27T21:41:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffa606640 1 --2- 172.21.15.60:0/3607296112 >> v2:172.21.15.60:3300/0 conn(0x7f8ff413b480 0x7f8ff4141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ff9e05640 10 monclient(hunting): get_auth_request con 0x7f8ff40b40b0 auth_method 0 2024-03-27T21:41:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ff9e05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ff9e05640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ff9e05640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffa606640 10 monclient(hunting): get_auth_request con 0x7f8ff413b480 auth_method 0 2024-03-27T21:41:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffa606640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffa606640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffa606640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffa606640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffa606640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ffa606640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ff9e05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ff9e05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ff9e05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ff8e03640 1 -- 172.21.15.60:0/3607296112 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676731113 0 0) 0x7f8ff40b3830 con 0x7f8ff413b020 2024-03-27T21:41:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ff8e03640 10 monclient(hunting): _negotiate 2024-03-27T21:41:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ff8e03640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ff8e03640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ff8e03640 10 monclient(hunting): my global_id is 4825 2024-03-27T21:41:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.105+0000 7f8ff8e03640 1 -- 172.21.15.60:0/3607296112 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe0003520 con 0x7f8ff413b020 2024-03-27T21:41:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.106+0000 7f8ff9e05640 10 monclient(hunting): handle_auth_done global_id 4703 payload 274 2024-03-27T21:41:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.106+0000 7f8ff9e05640 1 -- 172.21.15.60:0/3607296112 >> v2:172.21.15.60:3300/0 conn(0x7f8ff413b480 msgr2=0x7f8ff4141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.106+0000 7f8ff9e05640 1 --2- 172.21.15.60:0/3607296112 >> v2:172.21.15.60:3300/0 conn(0x7f8ff413b480 0x7f8ff4141e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.106+0000 7f8ff9e05640 1 -- 172.21.15.60:0/3607296112 >> v1:172.21.15.191:6789/0 conn(0x7f8ff413b020 legacy=0x7f8ff41416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.106+0000 7f8ff9e05640 10 monclient: _finish_hunting 0 2024-03-27T21:41:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.106+0000 7f8ff9e05640 1 monclient: found mon.noname-b 2024-03-27T21:41:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.106+0000 7f8ff9e05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.106+0000 7f8ff9e05640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.106+0000 7f8ff9e05640 1 -- 172.21.15.60:0/3607296112 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff40aa520 con 0x7f8ff40b40b0 2024-03-27T21:41:07.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.106+0000 7fd48b813640 10 monclient: shutdown 2024-03-27T21:41:07.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.106+0000 7fd48b813640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:07.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.106+0000 7fd48b813640 1 -- 172.21.15.60:0/3736001536 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd4840b40b0 msgr2=0x7fd4840a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:07.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.106+0000 7fd48b813640 1 --2- 172.21.15.60:0/3736001536 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd4840b40b0 0x7fd4840a8b90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd480009870 tx=0x7fd48002b8f0 comp rx=0 tx=0).stop 2024-03-27T21:41:07.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.106+0000 7fd48b813640 1 -- 172.21.15.60:0/3736001536 shutdown_connections 2024-03-27T21:41:07.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.106+0000 7fd48b813640 1 --2- 172.21.15.60:0/3736001536 >> v2:172.21.15.191:3300/0 conn(0x7fd48413b480 0x7fd484141e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:07.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.106+0000 7fd48b813640 1 --2- 172.21.15.60:0/3736001536 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd4840b40b0 0x7fd4840a8b90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:07.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.106+0000 7fd48b813640 1 -- 172.21.15.60:0/3736001536 >> 172.21.15.60:0/3736001536 conn(0x7fd4840608e0 msgr2=0x7fd484060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:07.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.106+0000 7fd48b813640 1 -- 172.21.15.60:0/3736001536 shutdown_connections 2024-03-27T21:41:07.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.106+0000 7fd48b813640 1 -- 172.21.15.60:0/3736001536 wait complete. 2024-03-27T21:41:07.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.106+0000 7f8ff9e05640 1 --2- 172.21.15.60:0/3607296112 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8ff40b40b0 0x7f8ff40a8b90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009870 tx=0x7f8ff002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec79c9d7ad81637e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:07.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ff8e03640 1 -- 172.21.15.60:0/3607296112 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0031640 con 0x7f8ff40b40b0 2024-03-27T21:41:07.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ff8e03640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:07.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ff8e03640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ff8e03640 10 monclient: dump: 2024-03-27T21:41:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:07.110 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ff8e03640 10 monclient: _finish_auth 0 2024-03-27T21:41:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ff8e03640 10 monclient: _check_auth_tickets 2024-03-27T21:41:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ff8e03640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ff8e03640 1 -- 172.21.15.60:0/3607296112 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff00317a0 con 0x7f8ff40b40b0 2024-03-27T21:41:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ff8e03640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ffc069640 5 monclient: authenticate success, global_id 4703 2024-03-27T21:41:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ff8e03640 1 -- 172.21.15.60:0/3607296112 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0031b30 con 0x7f8ff40b40b0 2024-03-27T21:41:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ffc069640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ffc069640 4 set_mon_vals no callback set 2024-03-27T21:41:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ff8e03640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ff8e03640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ff8e03640 10 monclient: dump: 2024-03-27T21:41:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:07.112 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7fd48b813640 10 monclient: build_initial_monmap 2024-03-27T21:41:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7fd48b813640 10 monclient: monmap: 2024-03-27T21:41:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:07.113 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7fd48b813640 1 librados: starting msgr at 2024-03-27T21:41:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7fd48b813640 1 librados: starting objecter 2024-03-27T21:41:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7fd48b813640 1 Processor -- start 2024-03-27T21:41:07.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ffc069640 10 monclient: shutdown 2024-03-27T21:41:07.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ffc069640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:07.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ffc069640 1 -- 172.21.15.60:0/3607296112 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8ff40b40b0 msgr2=0x7f8ff40a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:07.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ffc069640 1 --2- 172.21.15.60:0/3607296112 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8ff40b40b0 0x7f8ff40a8b90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009870 tx=0x7f8ff002b8f0 comp rx=0 tx=0).stop 2024-03-27T21:41:07.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7fd48b813640 1 -- start start 2024-03-27T21:41:07.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7fd48b813640 1 librados: setting wanted keys 2024-03-27T21:41:07.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7fd48b813640 1 librados: calling monclient init 2024-03-27T21:41:07.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7fd48b813640 10 monclient: init 2024-03-27T21:41:07.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ffc069640 1 -- 172.21.15.60:0/3607296112 shutdown_connections 2024-03-27T21:41:07.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ffc069640 1 --2- 172.21.15.60:0/3607296112 >> v2:172.21.15.60:3300/0 conn(0x7f8ff413b480 0x7f8ff4141e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:07.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ffc069640 1 --2- 172.21.15.60:0/3607296112 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8ff40b40b0 0x7f8ff40a8b90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:07.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.107+0000 7f8ffc069640 1 -- 172.21.15.60:0/3607296112 >> 172.21.15.60:0/3607296112 conn(0x7f8ff40608e0 msgr2=0x7f8ff4060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:07.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7f8ffc069640 1 -- 172.21.15.60:0/3607296112 shutdown_connections 2024-03-27T21:41:07.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7f8ffc069640 1 -- 172.21.15.60:0/3607296112 wait complete. 2024-03-27T21:41:07.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd48b813640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:07.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd48b813640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:41:07.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd48b813640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4840b40b0 0x7fd48413daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:07.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd48b813640 10 monclient(hunting): picked mon.noname-a con 0x7fd4840b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:07.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd48b813640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd48413b020 0x7fd48413a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:07.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd48b813640 10 monclient(hunting): picked mon.noname-b con 0x7fd48413b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:07.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd48b813640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd48413b480 0x7fd48413e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:07.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd4895af640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4840b40b0 0x7fd48413daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:07.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd48b813640 10 monclient(hunting): picked mon.noname-c con 0x7fd48413b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:07.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd48b813640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:07.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd48b813640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48413aa20 con 0x7fd4840b40b0 2024-03-27T21:41:07.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd48b813640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:07.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd48b813640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4840ab6b0 con 0x7fd48413b020 2024-03-27T21:41:07.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd48b813640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:07.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd48b813640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd48413cc60 con 0x7fd48413b480 2024-03-27T21:41:07.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd4895af640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4840b40b0 0x7fd48413daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44854/0 (socket says 172.21.15.60:44854) 2024-03-27T21:41:07.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd48b813640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:07.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd4895af640 1 -- 172.21.15.60:0/233084351 learned_addr learned my addr 172.21.15.60:0/233084351 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:07.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd489db0640 1 --2- 172.21.15.60:0/233084351 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd48413b480 0x7fd48413e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:07.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd48b813640 10 monclient(hunting): authenticate will time out at 908.329778s 2024-03-27T21:41:07.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd4895af640 10 monclient(hunting): get_auth_request con 0x7fd4840b40b0 auth_method 0 2024-03-27T21:41:07.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd4895af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:07.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd4895af640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:07.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7f8ffc069640 10 monclient: build_initial_monmap 2024-03-27T21:41:07.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7f8ffc069640 10 monclient: monmap: 2024-03-27T21:41:07.119 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:07.119 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:07.119 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:07.119 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:07.119 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:07.119 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:07.119 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:07.119 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:07.119 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:07.120 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd4895af640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd489db0640 10 monclient(hunting): get_auth_request con 0x7fd48413b480 auth_method 0 2024-03-27T21:41:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd489db0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd489db0640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd489db0640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd489db0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.108+0000 7fd489db0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd489db0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd4895af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd4895af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd4895af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd488dae640 1 --2- 172.21.15.60:0/233084351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd48413b020 0x7fd48413a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd488dae640 10 monclient(hunting): get_auth_request con 0x7fd48413b020 auth_method 0 2024-03-27T21:41:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd488dae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd488dae640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd488dae640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7f8ffc069640 1 librados: starting msgr at 2024-03-27T21:41:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7f8ffc069640 1 librados: starting objecter 2024-03-27T21:41:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd489db0640 10 monclient(hunting): handle_auth_done global_id 4706 payload 824 2024-03-27T21:41:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd489db0640 1 -- 172.21.15.60:0/233084351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd48413b020 msgr2=0x7fd48413a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd489db0640 1 --2- 172.21.15.60:0/233084351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd48413b020 0x7fd48413a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd489db0640 1 -- 172.21.15.60:0/233084351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4840b40b0 msgr2=0x7fd48413daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd489db0640 1 --2- 172.21.15.60:0/233084351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4840b40b0 0x7fd48413daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd489db0640 10 monclient: _finish_hunting 0 2024-03-27T21:41:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd489db0640 1 monclient: found mon.noname-c 2024-03-27T21:41:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd489db0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd489db0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd489db0640 1 -- 172.21.15.60:0/233084351 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4800094d0 con 0x7fd48413b480 2024-03-27T21:41:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd488dae640 1 --2- 172.21.15.60:0/233084351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd48413b020 0x7fd48413a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:41:07.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7f8ffc069640 1 Processor -- start 2024-03-27T21:41:07.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7f8ffc069640 1 -- start start 2024-03-27T21:41:07.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7f8ffc069640 1 librados: setting wanted keys 2024-03-27T21:41:07.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7f8ffc069640 1 librados: calling monclient init 2024-03-27T21:41:07.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7f8ffc069640 10 monclient: init 2024-03-27T21:41:07.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd489db0640 1 --2- 172.21.15.60:0/233084351 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd48413b480 0x7fd48413e480 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd47c007de0 tx=0x7fd47c00d4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:07.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd479ffb640 10 client.?.objecter ms_handle_connect 0x7fd48413b480 2024-03-27T21:41:07.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd479ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:07.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd479ffb640 1 -- 172.21.15.60:0/233084351 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47c003990 con 0x7fd48413b480 2024-03-27T21:41:07.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd479ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:07.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd479ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:07.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd479ffb640 10 monclient: dump: 2024-03-27T21:41:07.126 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:07.126 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:07.126 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:07.127 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd479ffb640 10 monclient: _finish_auth 0 2024-03-27T21:41:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd479ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:41:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd479ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd479ffb640 1 -- 172.21.15.60:0/233084351 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd47c00f040 con 0x7fd48413b480 2024-03-27T21:41:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd48b813640 5 monclient: authenticate success, global_id 4706 2024-03-27T21:41:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd48b813640 10 monclient: _renew_subs 2024-03-27T21:41:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd48b813640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd48b813640 1 -- 172.21.15.60:0/233084351 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4841d9e70 con 0x7fd48413b480 2024-03-27T21:41:07.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd479ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:07.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd479ffb640 1 -- 172.21.15.60:0/233084351 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47c003d30 con 0x7fd48413b480 2024-03-27T21:41:07.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd479ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:07.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd479ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:07.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd479ffb640 10 monclient: dump: 2024-03-27T21:41:07.128 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:07.128 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:07.128 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:07.128 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:07.128 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:07.128 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:07.128 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:07.128 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:07.128 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:07.129 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:07.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.109+0000 7fd47b7fe640 4 set_mon_vals no callback set 2024-03-27T21:41:07.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7fd48b813640 10 client.4706.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:07.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7fd48b813640 10 monclient: _renew_subs 2024-03-27T21:41:07.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7fd48b813640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:07.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7fd48b813640 1 -- 172.21.15.60:0/233084351 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd48413eba0 con 0x7fd48413b480 2024-03-27T21:41:07.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7fd48b813640 1 librados: init done 2024-03-27T21:41:07.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffc069640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:07.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffc069640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:41:07.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffc069640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8ff40b40b0 0x7f8ff40aacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:07.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffc069640 10 monclient(hunting): picked mon.noname-c con 0x7f8ff40b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:07.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffc069640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ff413b020 0x7f8ff413e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:07.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffc069640 10 monclient(hunting): picked mon.noname-a con 0x7f8ff413b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:07.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffc069640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ff413b480 0x7f8ff41d6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:07.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffc069640 10 monclient(hunting): picked mon.noname-b con 0x7f8ff413b480 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:07.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffc069640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:07.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffc069640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff41d61d0 con 0x7f8ff413b020 2024-03-27T21:41:07.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffc069640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:07.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffc069640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff413d3f0 con 0x7f8ff413b480 2024-03-27T21:41:07.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffc069640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:07.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffc069640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff413ba60 con 0x7f8ff40b40b0 2024-03-27T21:41:07.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffc069640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:07.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffa606640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ff413b480 0x7f8ff41d6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:07.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffc069640 10 monclient(hunting): authenticate will time out at 908.331507s 2024-03-27T21:41:07.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffa606640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ff413b480 0x7f8ff41d6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:56790/0 (socket says 172.21.15.60:56790) 2024-03-27T21:41:07.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffa606640 1 -- 172.21.15.60:0/2968200703 learned_addr learned my addr 172.21.15.60:0/2968200703 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:07.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffa606640 10 monclient(hunting): get_auth_request con 0x7f8ff413b480 auth_method 0 2024-03-27T21:41:07.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffa606640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:07.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffa606640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:07.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffa606640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:07.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffa606640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:07.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffa606640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:07.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffa606640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:07.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ff9604640 1 --2- 172.21.15.60:0/2968200703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ff413b020 0x7f8ff413e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:07.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ff9604640 10 monclient(hunting): get_auth_request con 0x7f8ff413b020 auth_method 0 2024-03-27T21:41:07.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ff9604640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:07.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ff9604640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:07.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ff9604640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:07.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ff9e05640 1 --2- 172.21.15.60:0/2968200703 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8ff40b40b0 0x7f8ff40aacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:07.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffa606640 10 monclient(hunting): handle_auth_done global_id 4831 payload 824 2024-03-27T21:41:07.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffa606640 1 -- 172.21.15.60:0/2968200703 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8ff40b40b0 msgr2=0x7f8ff40aacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:07.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffa606640 1 --2- 172.21.15.60:0/2968200703 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8ff40b40b0 0x7f8ff40aacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:07.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffa606640 1 -- 172.21.15.60:0/2968200703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ff413b020 msgr2=0x7f8ff413e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:07.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffa606640 1 --2- 172.21.15.60:0/2968200703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ff413b020 0x7f8ff413e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:07.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffa606640 10 monclient: _finish_hunting 0 2024-03-27T21:41:07.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.110+0000 7f8ffa606640 1 monclient: found mon.noname-b 2024-03-27T21:41:07.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8ffa606640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:07.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8ffa606640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:41:07.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8ffa606640 1 -- 172.21.15.60:0/2968200703 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff00094d0 con 0x7f8ff413b480 2024-03-27T21:41:07.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8ff9e05640 10 monclient: get_auth_request con 0x7f8ff40b40b0 auth_method 0 2024-03-27T21:41:07.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8ff9e05640 1 --2- 172.21.15.60:0/2968200703 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8ff40b40b0 0x7f8ff40aacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:41:07.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8ff9604640 1 --2- 172.21.15.60:0/2968200703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ff413b020 0x7f8ff413e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:41:07.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8ffa606640 1 --2- 172.21.15.60:0/2968200703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ff413b480 0x7f8ff41d6660 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8fec00efc0 tx=0x7f8fec00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:07.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8fea7fc640 10 client.?.objecter ms_handle_connect 0x7f8ff413b480 2024-03-27T21:41:07.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8fea7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:07.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8fea7fc640 1 -- 172.21.15.60:0/2968200703 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec009ba0 con 0x7f8ff413b480 2024-03-27T21:41:07.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8fea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:07.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8fea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:07.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8fea7fc640 10 monclient: dump: 2024-03-27T21:41:07.136 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:07.136 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:07.136 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:07.136 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:07.136 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:07.136 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:07.136 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:07.136 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:07.136 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:07.136 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:07.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8fea7fc640 10 monclient: _finish_auth 0 2024-03-27T21:41:07.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8fea7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:41:07.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8fea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:07.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8fea7fc640 1 -- 172.21.15.60:0/2968200703 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fec00f040 con 0x7f8ff413b480 2024-03-27T21:41:07.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8fea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:07.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8fea7fc640 1 -- 172.21.15.60:0/2968200703 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec0146c0 con 0x7f8ff413b480 2024-03-27T21:41:07.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8fea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:07.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8fea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:07.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8fea7fc640 10 monclient: dump: 2024-03-27T21:41:07.137 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:07.137 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:07.137 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:07.137 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:07.137 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:07.137 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:07.138 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:07.138 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:07.138 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:07.138 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:07.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8febfff640 4 set_mon_vals no callback set 2024-03-27T21:41:07.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7fd479ffb640 1 -- 172.21.15.60:0/233084351 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd47c004030 con 0x7fd48413b480 2024-03-27T21:41:07.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8ffc069640 5 monclient: authenticate success, global_id 4831 2024-03-27T21:41:07.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8ffc069640 10 monclient: _renew_subs 2024-03-27T21:41:07.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8ffc069640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:07.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8ffc069640 1 -- 172.21.15.60:0/2968200703 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff413dfc0 con 0x7f8ff413b480 2024-03-27T21:41:07.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8ffc069640 10 client.4831.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:07.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8ffc069640 10 monclient: _renew_subs 2024-03-27T21:41:07.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.111+0000 7f8ffc069640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:07.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.112+0000 7f8ffc069640 1 -- 172.21.15.60:0/2968200703 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff41d6d80 con 0x7f8ff413b480 2024-03-27T21:41:07.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.112+0000 7f8ffc069640 1 librados: init done 2024-03-27T21:41:07.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.112+0000 7fd48b813640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:07.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.112+0000 7fd48b813640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:07.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.112+0000 7fd48b813640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:07.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.112+0000 7fd48b813640 1 -- 172.21.15.60:0/233084351 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd484069470 con 0x7fd48413b480 2024-03-27T21:41:07.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.112+0000 7fd479ffb640 1 --2- 172.21.15.60:0/233084351 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd46806c590 0x7fd46806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:07.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.112+0000 7fd479ffb640 1 -- 172.21.15.60:0/233084351 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fd47c089410 con 0x7fd48413b480 2024-03-27T21:41:07.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.112+0000 7fd479ffb640 10 client.4706.objecter ms_dispatch 0x7fd484000b90 osd_map(51..51 src has 1..51) v4 2024-03-27T21:41:07.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.112+0000 7fd479ffb640 3 client.4706.objecter handle_osd_map got epochs [51,51] > 0 2024-03-27T21:41:07.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.112+0000 7fd479ffb640 3 client.4706.objecter handle_osd_map decoding full epoch 51 2024-03-27T21:41:07.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.112+0000 7fd4895af640 1 --2- 172.21.15.60:0/233084351 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd46806c590 0x7fd46806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:07.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.112+0000 7fd479ffb640 20 client.4706.objecter dump_active .. 0 homeless 2024-03-27T21:41:07.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.112+0000 7fd4895af640 10 monclient: get_auth_request con 0x7fd46806c590 auth_method 0 2024-03-27T21:41:07.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.113+0000 7f8ffc069640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:07.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.113+0000 7f8ffc069640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:07.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.113+0000 7f8ffc069640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:07.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.113+0000 7f8ffc069640 1 -- 172.21.15.60:0/2968200703 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff40ac380 con 0x7f8ff413b480 2024-03-27T21:41:07.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.113+0000 7fd479ffb640 10 client.4706.objecter ms_handle_connect 0x7fd46806c590 2024-03-27T21:41:07.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.113+0000 7fd4895af640 1 --2- 172.21.15.60:0/233084351 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd46806c590 0x7fd46806ea50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd4800090b0 tx=0x7fd48003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:07.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.113+0000 7f8fea7fc640 1 -- 172.21.15.60:0/2968200703 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8fec014820 con 0x7f8ff413b480 2024-03-27T21:41:07.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.114+0000 7f8fea7fc640 1 --2- 172.21.15.60:0/2968200703 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f8fd806c590 0x7f8fd806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:07.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.114+0000 7f8fea7fc640 1 -- 172.21.15.60:0/2968200703 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f8fec089d30 con 0x7f8ff413b480 2024-03-27T21:41:07.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.114+0000 7f8fea7fc640 10 client.4831.objecter ms_dispatch 0x7f8ff4000b90 osd_map(51..51 src has 1..51) v4 2024-03-27T21:41:07.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.114+0000 7f8fea7fc640 3 client.4831.objecter handle_osd_map got epochs [51,51] > 0 2024-03-27T21:41:07.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.114+0000 7f8fea7fc640 3 client.4831.objecter handle_osd_map decoding full epoch 51 2024-03-27T21:41:07.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.114+0000 7f8ff9e05640 1 --2- 172.21.15.60:0/2968200703 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f8fd806c590 0x7f8fd806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:07.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.114+0000 7f8ff9e05640 10 monclient: get_auth_request con 0x7f8fd806c590 auth_method 0 2024-03-27T21:41:07.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.114+0000 7f8fea7fc640 20 client.4831.objecter dump_active .. 0 homeless 2024-03-27T21:41:07.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.115+0000 7f8fea7fc640 10 client.4831.objecter ms_handle_connect 0x7f8fd806c590 2024-03-27T21:41:07.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.115+0000 7f8ff9e05640 1 --2- 172.21.15.60:0/2968200703 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f8fd806c590 0x7f8fd806ea50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009870 tx=0x7f8ff003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:07.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.118+0000 7fd479ffb640 1 -- 172.21.15.60:0/233084351 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd47c054880 con 0x7fd48413b480 2024-03-27T21:41:07.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.118+0000 7fd479ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:07.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.118+0000 7fd479ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:07.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.118+0000 7f8fea7fc640 1 -- 172.21.15.60:0/2968200703 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8fec055040 con 0x7f8ff413b480 2024-03-27T21:41:07.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.118+0000 7f8fea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:07.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.118+0000 7f8fea7fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:07.178 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:07.192+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:07.207 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:07.208 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:07.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:07.208 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:41:07.208 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:07.216 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:41:07.300 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:07.315+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:07.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.321+0000 7f8ffc069640 1 -- 172.21.15.60:0/2968200703 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}) v1 -- 0x7f8ff40a90d0 con 0x7f8fd806c590 2024-03-27T21:41:07.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.321+0000 7f8fea7fc640 1 -- 172.21.15.60:0/2968200703 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 no change) v1 ==== 17+0+278 (secure 0 0 0) 0x7f8ff0031a10 con 0x7f8fd806c590 2024-03-27T21:41:07.308 INFO:teuthology.orchestra.run.smithi060.stdout:moved 0 / 18 (0%) 2024-03-27T21:41:07.308 INFO:teuthology.orchestra.run.smithi060.stdout:avg 2.57143 2024-03-27T21:41:07.308 INFO:teuthology.orchestra.run.smithi060.stdout:stddev 1.41421 -> 1.41421 (expected baseline 1.48461) 2024-03-27T21:41:07.308 INFO:teuthology.orchestra.run.smithi060.stdout:min osd.4 with 0 -> 0 pgs (0 -> 0 * mean) 2024-03-27T21:41:07.308 INFO:teuthology.orchestra.run.smithi060.stdout:max osd.1 with 4 -> 4 pgs (1.55556 -> 1.55556 * mean) 2024-03-27T21:41:07.308 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:41:07.308 INFO:teuthology.orchestra.run.smithi060.stdout:oload 1000 2024-03-27T21:41:07.308 INFO:teuthology.orchestra.run.smithi060.stdout:max_change 3 2024-03-27T21:41:07.308 INFO:teuthology.orchestra.run.smithi060.stdout:max_change_osds 4 2024-03-27T21:41:07.308 INFO:teuthology.orchestra.run.smithi060.stdout:average_utilization 0.0003 2024-03-27T21:41:07.308 INFO:teuthology.orchestra.run.smithi060.stdout:overload_utilization 0.0030 2024-03-27T21:41:07.309 INFO:teuthology.orchestra.run.smithi060.stderr:no change 2024-03-27T21:41:07.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.323+0000 7f8ffc069640 10 librados: watch_flush enter 2024-03-27T21:41:07.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.323+0000 7f8ffc069640 10 librados: watch_flush exit 2024-03-27T21:41:07.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.323+0000 7f8ffc069640 20 client.4831.objecter shutdown clearing up homeless session... 2024-03-27T21:41:07.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.323+0000 7f8ffc069640 10 client.4831.objecter successfully canceled tick 2024-03-27T21:41:07.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.323+0000 7f8ffc069640 1 -- 172.21.15.60:0/2968200703 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f8fd806c590 msgr2=0x7f8fd806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:07.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.323+0000 7f8ffc069640 1 --2- 172.21.15.60:0/2968200703 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f8fd806c590 0x7f8fd806ea50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009870 tx=0x7f8ff003c040 comp rx=0 tx=0).stop 2024-03-27T21:41:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.323+0000 7f8ffc069640 10 monclient: shutdown 2024-03-27T21:41:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.323+0000 7f8ffc069640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.323+0000 7f8ffc069640 1 -- 172.21.15.60:0/2968200703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ff413b480 msgr2=0x7f8ff41d6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.323+0000 7f8ffc069640 1 --2- 172.21.15.60:0/2968200703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ff413b480 0x7f8ff41d6660 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8fec00efc0 tx=0x7f8fec00c490 comp rx=0 tx=0).stop 2024-03-27T21:41:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.323+0000 7f8ffc069640 1 -- 172.21.15.60:0/2968200703 shutdown_connections 2024-03-27T21:41:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.323+0000 7f8ffc069640 1 --2- 172.21.15.60:0/2968200703 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f8fd806c590 0x7f8fd806ea50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.323+0000 7f8ffc069640 1 --2- 172.21.15.60:0/2968200703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ff413b480 0x7f8ff41d6660 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.323+0000 7f8ffc069640 1 --2- 172.21.15.60:0/2968200703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ff413b020 0x7f8ff413e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.323+0000 7f8ffc069640 1 --2- 172.21.15.60:0/2968200703 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8ff40b40b0 0x7f8ff40aacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.323+0000 7f8ffc069640 1 -- 172.21.15.60:0/2968200703 >> 172.21.15.60:0/2968200703 conn(0x7f8ff40608e0 msgr2=0x7f8ff40b6090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.324+0000 7f8ffc069640 1 -- 172.21.15.60:0/2968200703 shutdown_connections 2024-03-27T21:41:07.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.324+0000 7f8ffc069640 1 -- 172.21.15.60:0/2968200703 wait complete. 2024-03-27T21:41:07.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.324+0000 7f8ffc069640 1 librados: shutdown 2024-03-27T21:41:07.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.327+0000 7fd48b813640 10 start_mon_command cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:41:07.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.327+0000 7fd48b813640 10 monclient: _send_command 2 [{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:41:07.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.328+0000 7fd48b813640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:07.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.328+0000 7fd48b813640 1 -- 172.21.15.60:0/233084351 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fd4840b6b20 con 0x7fd48413b480 2024-03-27T21:41:07.362 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:07.363 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:07.363 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:07.363 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:07.373 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:41:07.413 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:07.427+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:07.480 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:07.495+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:07.516 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:07.516 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:07.516 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:07.517 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:07.517 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:07.525 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:41:07.609 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:07.624+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:07.672 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:07.672 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:07.673 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:07.673 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-03-27T21:41:07.673 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.673 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:07.673 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:07.673 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 88.253140920000007, 2024-03-27T21:41:07.673 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43537978700000002, 2024-03-27T21:41:07.673 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:07.673 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:07.673 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:07.673 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:07.673 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:07.674 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-03-27T21:41:07.674 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.674 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:07.674 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.674 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:07.674 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:07.674 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.674 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.674 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.674 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:07.674 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:07.674 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.674 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.674 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.675 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:07.675 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432019+0000", 2024-03-27T21:41:07.675 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-03-27T21:41:07.675 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.675 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.675 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:07.675 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432027+0000", 2024-03-27T21:41:07.675 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1200000000000002e-06 2024-03-27T21:41:07.675 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.675 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.675 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:07.675 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432039+0000", 2024-03-27T21:41:07.675 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1123e-05 2024-03-27T21:41:07.675 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.676 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.676 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:07.676 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432051+0000", 2024-03-27T21:41:07.676 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-03-27T21:41:07.676 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.676 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.676 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:07.676 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432157+0000", 2024-03-27T21:41:07.676 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106846 2024-03-27T21:41:07.676 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.676 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.676 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-03-27T21:41:07.676 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432176+0000", 2024-03-27T21:41:07.676 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8490000000000001e-05 2024-03-27T21:41:07.676 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.677 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.677 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:07.677 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866170+0000", 2024-03-27T21:41:07.677 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43399370100000001 2024-03-27T21:41:07.677 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.677 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.677 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:07.677 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866335+0000", 2024-03-27T21:41:07.677 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016559699999999999 2024-03-27T21:41:07.677 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.677 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.677 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:07.677 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866495+0000", 2024-03-27T21:41:07.677 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015944200000000001 2024-03-27T21:41:07.678 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.678 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.678 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:07.678 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867012+0000", 2024-03-27T21:41:07.678 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0005176 2024-03-27T21:41:07.678 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.678 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.678 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:07.678 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867334+0000", 2024-03-27T21:41:07.678 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032158600000000001 2024-03-27T21:41:07.678 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.678 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.678 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:07.678 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867374+0000", 2024-03-27T21:41:07.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0547000000000003e-05 2024-03-27T21:41:07.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:07.679 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867400+0000", 2024-03-27T21:41:07.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5958000000000001e-05 2024-03-27T21:41:07.679 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.679 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:07.679 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.679 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:07.679 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:07.679 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 87.755758469, 2024-03-27T21:41:07.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064560310000000001, 2024-03-27T21:41:07.680 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:07.680 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:07.680 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:07.680 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:07.680 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:07.680 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 70 2024-03-27T21:41:07.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.680 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:07.680 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.680 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:07.680 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:07.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.681 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.681 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:07.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:07.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.681 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.681 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.681 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:07.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929402+0000", 2024-03-27T21:41:07.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:07.681 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.681 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.681 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:07.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929407+0000", 2024-03-27T21:41:07.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-06 2024-03-27T21:41:07.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.682 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:07.682 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929414+0000", 2024-03-27T21:41:07.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5689999999999999e-06 2024-03-27T21:41:07.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.682 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:07.682 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929419+0000", 2024-03-27T21:41:07.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-03-27T21:41:07.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.682 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:07.683 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930245+0000", 2024-03-27T21:41:07.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082521500000000002 2024-03-27T21:41:07.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.683 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:07.683 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930287+0000", 2024-03-27T21:41:07.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2305000000000002e-05 2024-03-27T21:41:07.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.683 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:07.683 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930367+0000", 2024-03-27T21:41:07.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9589e-05 2024-03-27T21:41:07.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:07.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.931214+0000", 2024-03-27T21:41:07.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000847827 2024-03-27T21:41:07.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:07.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935807+0000", 2024-03-27T21:41:07.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045923780000000003 2024-03-27T21:41:07.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:07.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935840+0000", 2024-03-27T21:41:07.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3501999999999999e-05 2024-03-27T21:41:07.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:07.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935859+0000", 2024-03-27T21:41:07.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8845e-05 2024-03-27T21:41:07.685 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.685 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:07.685 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.685 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:07.685 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:07.685 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 87.748498878000007, 2024-03-27T21:41:07.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071695429999999996, 2024-03-27T21:41:07.686 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:07.686 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:07.686 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:07.686 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:07.686 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:07.686 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-03-27T21:41:07.686 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.686 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:07.686 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.686 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:07.686 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:07.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.687 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.687 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.687 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:07.687 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:07.687 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.687 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.687 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.687 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:07.687 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936662+0000", 2024-03-27T21:41:07.687 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:07.687 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.687 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.687 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:07.687 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936737+0000", 2024-03-27T21:41:07.687 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5303999999999996e-05 2024-03-27T21:41:07.688 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.688 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.688 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:07.688 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936746+0000", 2024-03-27T21:41:07.688 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6340000000000007e-06 2024-03-27T21:41:07.688 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.688 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.688 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:07.688 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936753+0000", 2024-03-27T21:41:07.688 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-03-27T21:41:07.688 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.688 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.688 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:07.688 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937926+0000", 2024-03-27T21:41:07.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011731280000000001 2024-03-27T21:41:07.689 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:07.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937969+0000", 2024-03-27T21:41:07.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3285999999999999e-05 2024-03-27T21:41:07.689 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:07.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938040+0000", 2024-03-27T21:41:07.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0358000000000006e-05 2024-03-27T21:41:07.689 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:07.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939324+0000", 2024-03-27T21:41:07.690 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012844899999999999 2024-03-27T21:41:07.690 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.690 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:07.690 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943780+0000", 2024-03-27T21:41:07.690 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044555630000000001 2024-03-27T21:41:07.690 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.690 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:07.690 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943811+0000", 2024-03-27T21:41:07.690 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1415999999999999e-05 2024-03-27T21:41:07.690 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.691 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:07.691 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943832+0000", 2024-03-27T21:41:07.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0981000000000001e-05 2024-03-27T21:41:07.691 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.691 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:07.691 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.691 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.691 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:07.691 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:07.691 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 87.748400861999997, 2024-03-27T21:41:07.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071139740000000003, 2024-03-27T21:41:07.691 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:07.691 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:07.691 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:07.691 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:07.692 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:07.692 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-03-27T21:41:07.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.692 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:07.692 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:07.692 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:07.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.692 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:07.692 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:07.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.692 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:07.693 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936760+0000", 2024-03-27T21:41:07.693 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:07.693 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.693 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.693 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:07.693 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936765+0000", 2024-03-27T21:41:07.693 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6330000000000004e-06 2024-03-27T21:41:07.693 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.693 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.693 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:07.693 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936771+0000", 2024-03-27T21:41:07.693 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0560000000000001e-06 2024-03-27T21:41:07.693 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.693 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.694 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:07.694 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936776+0000", 2024-03-27T21:41:07.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1510000000000003e-06 2024-03-27T21:41:07.694 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.694 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.694 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:07.694 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938605+0000", 2024-03-27T21:41:07.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001829589 2024-03-27T21:41:07.694 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.694 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.694 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:07.694 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938657+0000", 2024-03-27T21:41:07.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1860000000000002e-05 2024-03-27T21:41:07.694 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:07.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938734+0000", 2024-03-27T21:41:07.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6639999999999998e-05 2024-03-27T21:41:07.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:07.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939562+0000", 2024-03-27T21:41:07.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082803100000000001 2024-03-27T21:41:07.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:07.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943838+0000", 2024-03-27T21:41:07.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042757050000000003 2024-03-27T21:41:07.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:07.696 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943859+0000", 2024-03-27T21:41:07.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1175e-05 2024-03-27T21:41:07.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:07.696 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943875+0000", 2024-03-27T21:41:07.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5744e-05 2024-03-27T21:41:07.696 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.696 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:07.696 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.696 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:07.696 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:07.697 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 87.604886811, 2024-03-27T21:41:07.697 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059712230000000003, 2024-03-27T21:41:07.697 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:07.697 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:07.697 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:07.697 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:07.697 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:07.697 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 112 2024-03-27T21:41:07.697 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.697 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:07.697 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.697 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:07.697 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:07.697 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.697 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.697 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.698 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:07.698 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:07.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.698 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.698 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:07.698 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080274+0000", 2024-03-27T21:41:07.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:07.698 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.698 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:07.698 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080318+0000", 2024-03-27T21:41:07.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3634999999999998e-05 2024-03-27T21:41:07.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:07.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080329+0000", 2024-03-27T21:41:07.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1232e-05 2024-03-27T21:41:07.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:07.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080333+0000", 2024-03-27T21:41:07.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6009999999999999e-06 2024-03-27T21:41:07.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:07.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083786+0000", 2024-03-27T21:41:07.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034528940000000002 2024-03-27T21:41:07.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:07.700 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083861+0000", 2024-03-27T21:41:07.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4893000000000003e-05 2024-03-27T21:41:07.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:07.700 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083981+0000", 2024-03-27T21:41:07.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012058299999999999 2024-03-27T21:41:07.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:07.700 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086148+0000", 2024-03-27T21:41:07.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021671469999999999 2024-03-27T21:41:07.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:07.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086195+0000", 2024-03-27T21:41:07.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6768999999999999e-05 2024-03-27T21:41:07.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:07.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086224+0000", 2024-03-27T21:41:07.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8758000000000001e-05 2024-03-27T21:41:07.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.702 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:07.702 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086246+0000", 2024-03-27T21:41:07.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2064000000000001e-05 2024-03-27T21:41:07.702 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.702 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:07.702 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.702 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.702 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:07.702 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:07.702 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 87.604823863999997, 2024-03-27T21:41:07.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066196320000000003, 2024-03-27T21:41:07.702 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:07.702 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:07.703 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:07.703 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:07.703 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:07.703 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113 2024-03-27T21:41:07.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.703 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:07.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.703 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:07.703 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:07.703 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.703 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:07.703 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:07.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:07.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080337+0000", 2024-03-27T21:41:07.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:07.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:07.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080340+0000", 2024-03-27T21:41:07.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6690000000000002e-06 2024-03-27T21:41:07.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:07.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080344+0000", 2024-03-27T21:41:07.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6550000000000002e-06 2024-03-27T21:41:07.705 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:07.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080348+0000", 2024-03-27T21:41:07.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9169999999999999e-06 2024-03-27T21:41:07.705 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:07.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084595+0000", 2024-03-27T21:41:07.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042471719999999996 2024-03-27T21:41:07.705 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:07.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084676+0000", 2024-03-27T21:41:07.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1147000000000001e-05 2024-03-27T21:41:07.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:07.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084796+0000", 2024-03-27T21:41:07.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012032900000000001 2024-03-27T21:41:07.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:07.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086190+0000", 2024-03-27T21:41:07.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013934399999999999 2024-03-27T21:41:07.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:07.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086905+0000", 2024-03-27T21:41:07.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071536199999999999 2024-03-27T21:41:07.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.707 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:07.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086937+0000", 2024-03-27T21:41:07.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2125000000000002e-05 2024-03-27T21:41:07.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.707 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:07.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086957+0000", 2024-03-27T21:41:07.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0157999999999999e-05 2024-03-27T21:41:07.707 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.707 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:07.708 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.708 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.708 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.708 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:07.708 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:07.708 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 87.604811147999996, 2024-03-27T21:41:07.708 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007405661, 2024-03-27T21:41:07.708 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:07.708 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:07.708 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:07.708 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:07.708 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:07.708 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 114 2024-03-27T21:41:07.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.709 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:07.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:07.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:07.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:07.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:07.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.710 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:07.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:07.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-03-27T21:41:07.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.710 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.710 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:07.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080353+0000", 2024-03-27T21:41:07.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.441e-06 2024-03-27T21:41:07.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.710 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.710 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:07.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080356+0000", 2024-03-27T21:41:07.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0769999999999999e-06 2024-03-27T21:41:07.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:07.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080358+0000", 2024-03-27T21:41:07.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3369999999999998e-06 2024-03-27T21:41:07.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:07.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085118+0000", 2024-03-27T21:41:07.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047603949999999997 2024-03-27T21:41:07.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:07.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085210+0000", 2024-03-27T21:41:07.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1983000000000003e-05 2024-03-27T21:41:07.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:07.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085332+0000", 2024-03-27T21:41:07.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121803 2024-03-27T21:41:07.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:07.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086292+0000", 2024-03-27T21:41:07.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095961400000000002 2024-03-27T21:41:07.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:07.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087710+0000", 2024-03-27T21:41:07.713 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014181459999999999 2024-03-27T21:41:07.713 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.713 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:07.713 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087736+0000", 2024-03-27T21:41:07.713 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6475e-05 2024-03-27T21:41:07.713 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.713 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:07.713 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087756+0000", 2024-03-27T21:41:07.713 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9626000000000001e-05 2024-03-27T21:41:07.713 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.713 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:07.714 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.714 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.714 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.714 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:07.714 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:07.714 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 87.604788713000005, 2024-03-27T21:41:07.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075011720000000004, 2024-03-27T21:41:07.714 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:07.714 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:07.714 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:07.714 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:07.714 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:07.714 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115 2024-03-27T21:41:07.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.715 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:07.715 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.715 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:07.715 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:07.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.715 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.715 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:07.715 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:07.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.715 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:07.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:07.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:07.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:07.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080458+0000", 2024-03-27T21:41:07.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5228000000000005e-05 2024-03-27T21:41:07.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:07.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080465+0000", 2024-03-27T21:41:07.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.238e-06 2024-03-27T21:41:07.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:07.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080468+0000", 2024-03-27T21:41:07.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5109999999999999e-06 2024-03-27T21:41:07.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:07.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086326+0000", 2024-03-27T21:41:07.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058579169999999998 2024-03-27T21:41:07.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:07.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086382+0000", 2024-03-27T21:41:07.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5309999999999997e-05 2024-03-27T21:41:07.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:07.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086457+0000", 2024-03-27T21:41:07.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5413000000000005e-05 2024-03-27T21:41:07.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:07.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087762+0000", 2024-03-27T21:41:07.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013052300000000001 2024-03-27T21:41:07.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:07.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087798+0000", 2024-03-27T21:41:07.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5614999999999997e-05 2024-03-27T21:41:07.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:07.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087840+0000", 2024-03-27T21:41:07.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2196000000000001e-05 2024-03-27T21:41:07.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:07.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087874+0000", 2024-03-27T21:41:07.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3785999999999998e-05 2024-03-27T21:41:07.719 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.719 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:07.719 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 87.604688629999998, 2024-03-27T21:41:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075050680000000002, 2024-03-27T21:41:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 116 2024-03-27T21:41:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.721 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:07.721 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:07.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.721 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.721 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.721 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:07.721 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:07.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.721 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.721 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.721 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:07.721 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080472+0000", 2024-03-27T21:41:07.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:07.721 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.722 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:07.722 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080475+0000", 2024-03-27T21:41:07.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.712e-06 2024-03-27T21:41:07.722 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.722 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:07.722 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080478+0000", 2024-03-27T21:41:07.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1070000000000001e-06 2024-03-27T21:41:07.722 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.722 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:07.722 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080481+0000", 2024-03-27T21:41:07.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4269999999999998e-06 2024-03-27T21:41:07.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:07.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086992+0000", 2024-03-27T21:41:07.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065114099999999996 2024-03-27T21:41:07.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:07.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087064+0000", 2024-03-27T21:41:07.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1927000000000004e-05 2024-03-27T21:41:07.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:07.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087188+0000", 2024-03-27T21:41:07.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123585 2024-03-27T21:41:07.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:07.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087766+0000", 2024-03-27T21:41:07.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057814300000000004 2024-03-27T21:41:07.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:07.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087934+0000", 2024-03-27T21:41:07.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000167894 2024-03-27T21:41:07.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:07.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087958+0000", 2024-03-27T21:41:07.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4663000000000001e-05 2024-03-27T21:41:07.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:07.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087978+0000", 2024-03-27T21:41:07.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9636e-05 2024-03-27T21:41:07.725 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.725 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:07.725 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.725 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:07.725 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:07.726 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 87.604656508000005, 2024-03-27T21:41:07.726 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076850679999999998, 2024-03-27T21:41:07.726 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:07.726 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:07.726 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:07.726 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:07.726 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:07.726 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 117 2024-03-27T21:41:07.726 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.726 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:07.726 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.726 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:07.727 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:07.727 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.727 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.727 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.727 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:07.727 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:07.727 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.727 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.727 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.727 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:07.727 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:07.727 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:07.727 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.727 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.727 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:07.728 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080508+0000", 2024-03-27T21:41:07.728 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.298e-06 2024-03-27T21:41:07.728 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.728 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.728 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:07.728 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080515+0000", 2024-03-27T21:41:07.728 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0790000000000003e-06 2024-03-27T21:41:07.728 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.728 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.728 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:07.728 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080518+0000", 2024-03-27T21:41:07.728 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8509999999999999e-06 2024-03-27T21:41:07.728 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.728 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.729 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:07.729 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087444+0000", 2024-03-27T21:41:07.729 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0069259120000000002 2024-03-27T21:41:07.729 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.729 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.729 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:07.729 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087491+0000", 2024-03-27T21:41:07.729 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7043999999999999e-05 2024-03-27T21:41:07.729 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.729 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.729 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:07.729 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087564+0000", 2024-03-27T21:41:07.729 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3603999999999995e-05 2024-03-27T21:41:07.729 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.729 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.729 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:07.730 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087902+0000", 2024-03-27T21:41:07.730 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033734100000000001 2024-03-27T21:41:07.730 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.730 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.730 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:07.730 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088117+0000", 2024-03-27T21:41:07.730 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000215164 2024-03-27T21:41:07.730 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.730 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.730 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:07.730 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088160+0000", 2024-03-27T21:41:07.730 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2933999999999999e-05 2024-03-27T21:41:07.730 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.731 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.731 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:07.731 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088190+0000", 2024-03-27T21:41:07.731 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0139999999999999e-05 2024-03-27T21:41:07.731 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.731 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:07.731 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.731 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.731 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.731 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:07.731 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:07.731 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 87.188376118999997, 2024-03-27T21:41:07.731 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052465439999999997, 2024-03-27T21:41:07.731 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:07.731 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:07.732 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:07.732 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:07.732 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:07.732 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 133 2024-03-27T21:41:07.732 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.732 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:07.732 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.732 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:07.732 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:07.732 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.732 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.732 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.732 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:07.732 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:07.732 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.732 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.732 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.733 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:07.733 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:07.733 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:07.733 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.733 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.733 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:07.733 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496788+0000", 2024-03-27T21:41:07.733 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-03-27T21:41:07.733 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.733 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.733 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:07.733 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496793+0000", 2024-03-27T21:41:07.733 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4790000000000004e-06 2024-03-27T21:41:07.733 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.733 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.733 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:07.734 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496796+0000", 2024-03-27T21:41:07.734 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1e-06 2024-03-27T21:41:07.734 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.734 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:07.734 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500447+0000", 2024-03-27T21:41:07.734 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036504459999999999 2024-03-27T21:41:07.734 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.734 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:07.735 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500492+0000", 2024-03-27T21:41:07.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5765e-05 2024-03-27T21:41:07.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.735 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.735 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:07.735 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500562+0000", 2024-03-27T21:41:07.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9206000000000001e-05 2024-03-27T21:41:07.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.735 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.735 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:07.735 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501959+0000", 2024-03-27T21:41:07.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001397187 2024-03-27T21:41:07.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.735 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.735 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:07.735 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501984+0000", 2024-03-27T21:41:07.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.552e-05 2024-03-27T21:41:07.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:07.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502010+0000", 2024-03-27T21:41:07.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5232000000000002e-05 2024-03-27T21:41:07.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:07.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502032+0000", 2024-03-27T21:41:07.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2337999999999999e-05 2024-03-27T21:41:07.736 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.736 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:07.736 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.736 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:07.737 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:07.737 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 87.188317460999997, 2024-03-27T21:41:07.737 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052892119999999997, 2024-03-27T21:41:07.737 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:07.737 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:07.737 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:07.737 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:07.737 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:07.737 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 134 2024-03-27T21:41:07.737 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.737 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:07.737 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.737 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:07.737 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:07.737 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.737 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.738 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:07.738 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:07.738 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.738 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.738 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:07.738 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:07.738 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:07.738 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.738 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:07.738 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496929+0000", 2024-03-27T21:41:07.738 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5098000000000004e-05 2024-03-27T21:41:07.738 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.739 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.739 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:07.739 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496937+0000", 2024-03-27T21:41:07.739 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9559999999999997e-06 2024-03-27T21:41:07.739 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.739 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.739 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:07.739 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496942+0000", 2024-03-27T21:41:07.739 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6350000000000003e-06 2024-03-27T21:41:07.739 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.739 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.739 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:07.739 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500749+0000", 2024-03-27T21:41:07.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038068529999999998 2024-03-27T21:41:07.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:07.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500792+0000", 2024-03-27T21:41:07.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2823e-05 2024-03-27T21:41:07.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:07.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500864+0000", 2024-03-27T21:41:07.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1715e-05 2024-03-27T21:41:07.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:07.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502068+0000", 2024-03-27T21:41:07.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012044860000000001 2024-03-27T21:41:07.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:07.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502073+0000", 2024-03-27T21:41:07.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5700000000000003e-06 2024-03-27T21:41:07.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:07.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502098+0000", 2024-03-27T21:41:07.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5120999999999999e-05 2024-03-27T21:41:07.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.742 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:07.742 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502133+0000", 2024-03-27T21:41:07.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5306999999999999e-05 2024-03-27T21:41:07.742 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.742 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:07.742 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.742 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:07.742 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:07.742 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 87.188212153999999, 2024-03-27T21:41:07.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062529040000000001, 2024-03-27T21:41:07.742 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:07.742 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:07.743 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:07.743 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:07.743 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:07.743 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135 2024-03-27T21:41:07.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.743 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:07.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:07.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:07.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:07.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:07.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.744 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:07.744 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:07.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:07.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.744 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:07.744 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496953+0000", 2024-03-27T21:41:07.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0709999999999996e-06 2024-03-27T21:41:07.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.744 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:07.745 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496961+0000", 2024-03-27T21:41:07.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1300000000000001e-06 2024-03-27T21:41:07.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.745 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:07.745 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496968+0000", 2024-03-27T21:41:07.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0999999999999998e-06 2024-03-27T21:41:07.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.745 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:07.745 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501244+0000", 2024-03-27T21:41:07.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042756310000000002 2024-03-27T21:41:07.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:07.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501377+0000", 2024-03-27T21:41:07.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013348799999999999 2024-03-27T21:41:07.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:07.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501455+0000", 2024-03-27T21:41:07.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7410000000000006e-05 2024-03-27T21:41:07.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:07.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502138+0000", 2024-03-27T21:41:07.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068386599999999999 2024-03-27T21:41:07.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:07.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503152+0000", 2024-03-27T21:41:07.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010139680000000001 2024-03-27T21:41:07.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:07.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503180+0000", 2024-03-27T21:41:07.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7426000000000001e-05 2024-03-27T21:41:07.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:07.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503202+0000", 2024-03-27T21:41:07.748 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2463000000000002e-05 2024-03-27T21:41:07.748 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.748 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:07.748 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.748 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.748 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.748 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:07.748 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:07.748 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 87.188188015999998, 2024-03-27T21:41:07.748 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066253680000000004, 2024-03-27T21:41:07.748 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:07.748 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:07.748 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:07.748 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:07.749 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:07.749 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 136 2024-03-27T21:41:07.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.749 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:07.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.749 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:07.749 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:07.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.749 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:07.749 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:07.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:07.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:07.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:07.750 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:07.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496977+0000", 2024-03-27T21:41:07.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5049999999999999e-06 2024-03-27T21:41:07.750 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:07.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496984+0000", 2024-03-27T21:41:07.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1520000000000003e-06 2024-03-27T21:41:07.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:07.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496990+0000", 2024-03-27T21:41:07.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-03-27T21:41:07.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:07.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501669+0000", 2024-03-27T21:41:07.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046793859999999998 2024-03-27T21:41:07.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:07.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501714+0000", 2024-03-27T21:41:07.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5154999999999997e-05 2024-03-27T21:41:07.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:07.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501783+0000", 2024-03-27T21:41:07.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8454999999999994e-05 2024-03-27T21:41:07.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:07.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503226+0000", 2024-03-27T21:41:07.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001443441 2024-03-27T21:41:07.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:07.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503540+0000", 2024-03-27T21:41:07.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031347100000000001 2024-03-27T21:41:07.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:07.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503563+0000", 2024-03-27T21:41:07.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3223000000000001e-05 2024-03-27T21:41:07.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:07.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503599+0000", 2024-03-27T21:41:07.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6105000000000001e-05 2024-03-27T21:41:07.753 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.753 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:07.753 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.754 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:07.754 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:07.754 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 87.188165205999994, 2024-03-27T21:41:07.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074261320000000002, 2024-03-27T21:41:07.754 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:07.754 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:07.754 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:07.754 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:07.754 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:07.754 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 137 2024-03-27T21:41:07.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.754 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:07.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:07.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:07.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:07.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:07.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:07.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:07.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:07.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:07.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497077+0000", 2024-03-27T21:41:07.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1599000000000003e-05 2024-03-27T21:41:07.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.756 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:07.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497085+0000", 2024-03-27T21:41:07.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6109999999999996e-06 2024-03-27T21:41:07.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.756 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:07.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497092+0000", 2024-03-27T21:41:07.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.742e-06 2024-03-27T21:41:07.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.756 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:07.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502159+0000", 2024-03-27T21:41:07.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050666870000000003 2024-03-27T21:41:07.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:07.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502200+0000", 2024-03-27T21:41:07.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1919999999999998e-05 2024-03-27T21:41:07.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:07.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502280+0000", 2024-03-27T21:41:07.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9417000000000002e-05 2024-03-27T21:41:07.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:07.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503605+0000", 2024-03-27T21:41:07.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001325641 2024-03-27T21:41:07.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:07.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504361+0000", 2024-03-27T21:41:07.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075555399999999997 2024-03-27T21:41:07.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:07.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504386+0000", 2024-03-27T21:41:07.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5369999999999999e-05 2024-03-27T21:41:07.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:07.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504422+0000", 2024-03-27T21:41:07.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6019000000000002e-05 2024-03-27T21:41:07.758 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.759 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:07.759 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.759 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:07.759 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:07.759 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 87.188061408999999, 2024-03-27T21:41:07.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074059750000000004, 2024-03-27T21:41:07.759 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:07.759 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:07.759 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:07.759 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:07.759 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:07.759 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-03-27T21:41:07.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.760 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:07.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:07.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:07.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:07.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:07.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:07.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:07.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:07.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:07.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497103+0000", 2024-03-27T21:41:07.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.856e-06 2024-03-27T21:41:07.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:07.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497121+0000", 2024-03-27T21:41:07.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7190999999999999e-05 2024-03-27T21:41:07.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:07.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497126+0000", 2024-03-27T21:41:07.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3770000000000004e-06 2024-03-27T21:41:07.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:07.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502833+0000", 2024-03-27T21:41:07.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057073510000000003 2024-03-27T21:41:07.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:07.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502881+0000", 2024-03-27T21:41:07.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7961999999999999e-05 2024-03-27T21:41:07.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:07.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502949+0000", 2024-03-27T21:41:07.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7957e-05 2024-03-27T21:41:07.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:07.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504430+0000", 2024-03-27T21:41:07.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001480763 2024-03-27T21:41:07.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:07.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504459+0000", 2024-03-27T21:41:07.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8983999999999999e-05 2024-03-27T21:41:07.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:07.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504486+0000", 2024-03-27T21:41:07.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6883999999999999e-05 2024-03-27T21:41:07.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:07.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504506+0000", 2024-03-27T21:41:07.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0214e-05 2024-03-27T21:41:07.764 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.764 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:07.764 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.764 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:07.764 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:07.764 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 87.188029575000002, 2024-03-27T21:41:07.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074754859999999999, 2024-03-27T21:41:07.765 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:07.765 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:07.765 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:07.765 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:07.765 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:07.765 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-03-27T21:41:07.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.765 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:07.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:07.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:07.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:07.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:07.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:07.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:07.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:07.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:07.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497137+0000", 2024-03-27T21:41:07.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0730000000000004e-06 2024-03-27T21:41:07.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:07.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497143+0000", 2024-03-27T21:41:07.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3759999999999996e-06 2024-03-27T21:41:07.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:07.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497150+0000", 2024-03-27T21:41:07.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.002e-06 2024-03-27T21:41:07.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:07.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503249+0000", 2024-03-27T21:41:07.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060989750000000004 2024-03-27T21:41:07.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:07.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503291+0000", 2024-03-27T21:41:07.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1563999999999997e-05 2024-03-27T21:41:07.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:07.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503376+0000", 2024-03-27T21:41:07.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5752000000000002e-05 2024-03-27T21:41:07.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:07.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504545+0000", 2024-03-27T21:41:07.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169172 2024-03-27T21:41:07.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:07.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504550+0000", 2024-03-27T21:41:07.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7419999999999997e-06 2024-03-27T21:41:07.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:07.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504574+0000", 2024-03-27T21:41:07.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4124000000000001e-05 2024-03-27T21:41:07.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:07.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504607+0000", 2024-03-27T21:41:07.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3118999999999999e-05 2024-03-27T21:41:07.769 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.770 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:07.770 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.770 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:07.770 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:07.770 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 87.188005908999997, 2024-03-27T21:41:07.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078150589999999992, 2024-03-27T21:41:07.770 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:07.770 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:07.770 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:07.770 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:07.770 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:07.771 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-03-27T21:41:07.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.771 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:07.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:07.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:07.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:07.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:07.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:07.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497155+0000", 2024-03-27T21:41:07.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:07.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:07.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497230+0000", 2024-03-27T21:41:07.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4789000000000005e-05 2024-03-27T21:41:07.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:07.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497241+0000", 2024-03-27T21:41:07.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1442e-05 2024-03-27T21:41:07.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:07.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497255+0000", 2024-03-27T21:41:07.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3087e-05 2024-03-27T21:41:07.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:07.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503630+0000", 2024-03-27T21:41:07.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063755269999999998 2024-03-27T21:41:07.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:07.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503673+0000", 2024-03-27T21:41:07.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2616e-05 2024-03-27T21:41:07.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:07.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503738+0000", 2024-03-27T21:41:07.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5114e-05 2024-03-27T21:41:07.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:07.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504907+0000", 2024-03-27T21:41:07.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169432 2024-03-27T21:41:07.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:07.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504926+0000", 2024-03-27T21:41:07.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9086999999999999e-05 2024-03-27T21:41:07.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:07.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504950+0000", 2024-03-27T21:41:07.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3598999999999999e-05 2024-03-27T21:41:07.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:07.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504971+0000", 2024-03-27T21:41:07.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0772999999999999e-05 2024-03-27T21:41:07.775 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.775 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:07.775 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.775 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:07.775 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:07.775 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 87.187898868999994, 2024-03-27T21:41:07.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082962590000000003, 2024-03-27T21:41:07.776 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:07.776 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:07.776 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:07.776 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:07.776 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:07.776 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-03-27T21:41:07.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.776 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:07.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:07.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:07.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:07.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:07.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:07.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497262+0000", 2024-03-27T21:41:07.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:07.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:07.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497267+0000", 2024-03-27T21:41:07.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.955e-06 2024-03-27T21:41:07.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:07.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497275+0000", 2024-03-27T21:41:07.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.824e-06 2024-03-27T21:41:07.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:07.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497280+0000", 2024-03-27T21:41:07.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8230000000000002e-06 2024-03-27T21:41:07.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:07.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504628+0000", 2024-03-27T21:41:07.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007348561 2024-03-27T21:41:07.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:07.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504673+0000", 2024-03-27T21:41:07.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4313000000000001e-05 2024-03-27T21:41:07.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:07.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504745+0000", 2024-03-27T21:41:07.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2303000000000002e-05 2024-03-27T21:41:07.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:07.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505358+0000", 2024-03-27T21:41:07.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061281300000000003 2024-03-27T21:41:07.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:07.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505503+0000", 2024-03-27T21:41:07.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000145134 2024-03-27T21:41:07.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:07.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505526+0000", 2024-03-27T21:41:07.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3312e-05 2024-03-27T21:41:07.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:07.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505559+0000", 2024-03-27T21:41:07.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2821999999999998e-05 2024-03-27T21:41:07.781 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.781 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:07.781 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.781 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:07.781 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:07.781 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 86.578676630000004, 2024-03-27T21:41:07.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053753339999999998, 2024-03-27T21:41:07.781 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:07.781 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:07.781 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:07.782 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:07.782 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:07.782 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159 2024-03-27T21:41:07.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.782 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:07.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:07.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:07.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:07.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:07.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:07.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:07.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:07.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:07.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:07.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106488+0000", 2024-03-27T21:41:07.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9830000000000001e-06 2024-03-27T21:41:07.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:07.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106491+0000", 2024-03-27T21:41:07.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3459999999999998e-06 2024-03-27T21:41:07.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:07.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106493+0000", 2024-03-27T21:41:07.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.509e-06 2024-03-27T21:41:07.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:07.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110636+0000", 2024-03-27T21:41:07.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041427260000000002 2024-03-27T21:41:07.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:07.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110683+0000", 2024-03-27T21:41:07.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6557000000000002e-05 2024-03-27T21:41:07.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.785 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:07.785 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110755+0000", 2024-03-27T21:41:07.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2119000000000005e-05 2024-03-27T21:41:07.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.785 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:07.785 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111415+0000", 2024-03-27T21:41:07.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066065199999999996 2024-03-27T21:41:07.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.785 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:07.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111793+0000", 2024-03-27T21:41:07.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037732100000000001 2024-03-27T21:41:07.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:07.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111829+0000", 2024-03-27T21:41:07.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418999999999998e-05 2024-03-27T21:41:07.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:07.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:07.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:07.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111860+0000", 2024-03-27T21:41:07.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1003000000000001e-05 2024-03-27T21:41:07.786 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.786 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:07.787 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.787 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:07.787 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:07.787 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:07.788 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:41:07.789 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:07.802+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:07.889 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:07.903+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:07.900 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:07.901 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:07.901 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:07.901 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:07.902 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:41:07.910 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:41:07.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.981+0000 7fd479ffb640 1 -- 172.21.15.60:0/233084351 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v52) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd47c003e90 con 0x7fd48413b480 2024-03-27T21:41:07.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.981+0000 7fd479ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:41:07.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.981+0000 7fd479ffb640 10 monclient: _finish_command 2 = system:0 nodeep-scrub is set 2024-03-27T21:41:07.968 INFO:teuthology.orchestra.run.smithi060.stderr:nodeep-scrub is set 2024-03-27T21:41:07.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.985+0000 7fd48b813640 10 librados: watch_flush enter 2024-03-27T21:41:07.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.985+0000 7fd48b813640 10 librados: watch_flush exit 2024-03-27T21:41:07.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.985+0000 7fd48b813640 20 client.4706.objecter shutdown clearing up homeless session... 2024-03-27T21:41:07.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.985+0000 7fd48b813640 10 client.4706.objecter successfully canceled tick 2024-03-27T21:41:07.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.985+0000 7fd48b813640 1 -- 172.21.15.60:0/233084351 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd46806c590 msgr2=0x7fd46806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:07.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.985+0000 7fd48b813640 1 --2- 172.21.15.60:0/233084351 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd46806c590 0x7fd46806ea50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd4800090b0 tx=0x7fd48003c040 comp rx=0 tx=0).stop 2024-03-27T21:41:07.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.985+0000 7fd48b813640 10 monclient: shutdown 2024-03-27T21:41:07.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.985+0000 7fd48b813640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:07.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.985+0000 7fd48b813640 1 -- 172.21.15.60:0/233084351 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd48413b480 msgr2=0x7fd48413e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:07.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.985+0000 7fd48b813640 1 --2- 172.21.15.60:0/233084351 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd48413b480 0x7fd48413e480 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd47c007de0 tx=0x7fd47c00d4a0 comp rx=0 tx=0).stop 2024-03-27T21:41:07.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.985+0000 7fd48b813640 1 -- 172.21.15.60:0/233084351 shutdown_connections 2024-03-27T21:41:07.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.985+0000 7fd48b813640 1 --2- 172.21.15.60:0/233084351 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd48413b480 0x7fd48413e480 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:07.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.985+0000 7fd48b813640 1 --2- 172.21.15.60:0/233084351 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd46806c590 0x7fd46806ea50 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:07.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.985+0000 7fd48b813640 1 --2- 172.21.15.60:0/233084351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd48413b020 0x7fd48413a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:07.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.985+0000 7fd48b813640 1 --2- 172.21.15.60:0/233084351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4840b40b0 0x7fd48413daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:07.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.985+0000 7fd48b813640 1 -- 172.21.15.60:0/233084351 >> 172.21.15.60:0/233084351 conn(0x7fd4840608e0 msgr2=0x7fd4841416f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:07.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.985+0000 7fd48b813640 1 -- 172.21.15.60:0/233084351 shutdown_connections 2024-03-27T21:41:07.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.985+0000 7fd48b813640 1 -- 172.21.15.60:0/233084351 wait complete. 2024-03-27T21:41:07.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:07.985+0000 7fd48b813640 1 librados: shutdown 2024-03-27T21:41:08.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.027+0000 7f48636ed640 10 monclient: get_monmap_and_config 2024-03-27T21:41:08.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.027+0000 7f48636ed640 10 monclient: build_initial_monmap 2024-03-27T21:41:08.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.027+0000 7f48636ed640 10 monclient: monmap: 2024-03-27T21:41:08.014 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:08.014 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:08.014 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:08.028367+0000 2024-03-27T21:41:08.014 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:08.028367+0000 2024-03-27T21:41:08.014 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:08.014 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:08.014 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:08.014 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:08.015 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:08.015 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:08.015 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:08.015 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:08.015 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:08.027+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:08.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.027+0000 7f48636ed640 1 Processor -- start 2024-03-27T21:41:08.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.028+0000 7f48636ed640 1 -- start start 2024-03-27T21:41:08.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.028+0000 7f48636ed640 10 monclient: init 2024-03-27T21:41:08.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.028+0000 7f48636ed640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:08.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.028+0000 7f48636ed640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-03-27T21:41:08.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.028+0000 7f48636ed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f485c0b40b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:08.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.028+0000 7f48636ed640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f485c13b020 0x7f485c13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:08.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.028+0000 7f48636ed640 10 monclient(hunting): picked mon.noname-c con 0x7f485c13b020 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:08.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.028+0000 7f48636ed640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f485c1416f0 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:08.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.028+0000 7f48636ed640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c0b3830 con 0x7f485c0b40b0 2024-03-27T21:41:08.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.028+0000 7f48636ed640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c0aa520 con 0x7f485c1416f0 2024-03-27T21:41:08.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.028+0000 7f48636ed640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.028+0000 7f48636ed640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485c0b4490 con 0x7f485c13b020 2024-03-27T21:41:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.028+0000 7f48636ed640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.028+0000 7f4861489640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f485c0b40b0 0x7f485c0a8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50940/0 (socket says 172.21.15.60:50940) 2024-03-27T21:41:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.028+0000 7f4861489640 1 -- 172.21.15.60:0/2963851425 learned_addr learned my addr 172.21.15.60:0/2963851425 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.028+0000 7f48636ed640 10 monclient(hunting): authenticate will time out at 909.249870s 2024-03-27T21:41:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4860c88640 1 --2- 172.21.15.60:0/2963851425 >> v2:172.21.15.191:3300/0 conn(0x7f485c13b020 0x7f485c13b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4860c88640 10 monclient(hunting): get_auth_request con 0x7f485c13b020 auth_method 0 2024-03-27T21:41:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4860c88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4860c88640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4860c88640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4860c88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4860c88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4860c88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4853fff640 1 -- 172.21.15.60:0/2963851425 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788955612 0 0) 0x7f485c0b3830 con 0x7f485c0b40b0 2024-03-27T21:41:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4853fff640 10 monclient(hunting): _negotiate 2024-03-27T21:41:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4853fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4853fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4853fff640 10 monclient(hunting): my global_id is 4875 2024-03-27T21:41:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4853fff640 1 -- 172.21.15.60:0/2963851425 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4848003520 con 0x7f485c0b40b0 2024-03-27T21:41:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4860c88640 10 monclient(hunting): handle_auth_done global_id 4834 payload 274 2024-03-27T21:41:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4860c88640 1 -- 172.21.15.60:0/2963851425 >> v1:172.21.15.191:6789/0 conn(0x7f485c1416f0 legacy=0x7f485c141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4853fff640 1 -- 172.21.15.60:0/2963851425 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160285879 0 0) 0x7f485c0aa520 con 0x7f485c1416f0 2024-03-27T21:41:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4860c88640 1 -- 172.21.15.60:0/2963851425 >> v1:172.21.15.60:6789/0 conn(0x7f485c0b40b0 legacy=0x7f485c0a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4860c88640 10 monclient: _finish_hunting 0 2024-03-27T21:41:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4860c88640 1 monclient: found mon.noname-c 2024-03-27T21:41:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4860c88640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4860c88640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:41:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4860c88640 1 -- 172.21.15.60:0/2963851425 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c0a9470 con 0x7f485c13b020 2024-03-27T21:41:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.029+0000 7f4853fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:41:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f4860c88640 1 --2- 172.21.15.60:0/2963851425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f485c13b020 0x7f485c13b480 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f484c0097a0 tx=0x7f484c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2da8368646ae84c server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f4853fff640 1 -- 172.21.15.60:0/2963851425 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c030610 con 0x7f485c13b020 2024-03-27T21:41:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f4853fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f4853fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f4853fff640 10 monclient: dump: 2024-03-27T21:41:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:08.029 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f4853fff640 10 monclient: _finish_auth 0 2024-03-27T21:41:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f4853fff640 10 monclient: _check_auth_tickets 2024-03-27T21:41:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f4853fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f4853fff640 1 -- 172.21.15.60:0/2963851425 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484c030770 con 0x7f485c13b020 2024-03-27T21:41:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f48636ed640 5 monclient: authenticate success, global_id 4834 2024-03-27T21:41:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f48636ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f4853fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f48636ed640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f48636ed640 4 set_mon_vals no callback set 2024-03-27T21:41:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f4853fff640 1 -- 172.21.15.60:0/2963851425 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c030ae0 con 0x7f485c13b020 2024-03-27T21:41:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f4853fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f4853fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f4853fff640 10 monclient: dump: 2024-03-27T21:41:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:08.031 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:08.031 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:08.031 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:08.031 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:08.031 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:08.031 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:08.031 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:08.031 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:08.031 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:08.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f48636ed640 10 monclient: shutdown 2024-03-27T21:41:08.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f48636ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:08.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f48636ed640 1 -- 172.21.15.60:0/2963851425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f485c13b020 msgr2=0x7f485c13b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:08.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.030+0000 7f48636ed640 1 --2- 172.21.15.60:0/2963851425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f485c13b020 0x7f485c13b480 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f484c0097a0 tx=0x7f484c02b960 comp rx=0 tx=0).stop 2024-03-27T21:41:08.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.031+0000 7f48636ed640 1 -- 172.21.15.60:0/2963851425 shutdown_connections 2024-03-27T21:41:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.031+0000 7f48636ed640 1 --2- 172.21.15.60:0/2963851425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f485c13b020 0x7f485c13b480 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.031+0000 7f48636ed640 1 -- 172.21.15.60:0/2963851425 >> 172.21.15.60:0/2963851425 conn(0x7f485c0608e0 msgr2=0x7f485c060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.031+0000 7f48636ed640 1 -- 172.21.15.60:0/2963851425 shutdown_connections 2024-03-27T21:41:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.031+0000 7f48636ed640 1 -- 172.21.15.60:0/2963851425 wait complete. 2024-03-27T21:41:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.031+0000 7f48636ed640 10 monclient: build_initial_monmap 2024-03-27T21:41:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.031+0000 7f48636ed640 10 monclient: monmap: 2024-03-27T21:41:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:08.033 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.031+0000 7f48636ed640 1 librados: starting msgr at 2024-03-27T21:41:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.031+0000 7f48636ed640 1 librados: starting objecter 2024-03-27T21:41:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f48636ed640 1 Processor -- start 2024-03-27T21:41:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f48636ed640 1 -- start start 2024-03-27T21:41:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f48636ed640 1 librados: setting wanted keys 2024-03-27T21:41:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f48636ed640 1 librados: calling monclient init 2024-03-27T21:41:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f48636ed640 10 monclient: init 2024-03-27T21:41:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f48636ed640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f48636ed640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:41:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f48636ed640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f485c0b40b0 0x7f485c13d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f48636ed640 10 monclient(hunting): picked mon.noname-b con 0x7f485c0b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f48636ed640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f485c13b020 0x7f485c138880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f48636ed640 10 monclient(hunting): picked mon.noname-a con 0x7f485c13b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f48636ed640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f485c1416f0 0x7f485c139620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f48636ed640 10 monclient(hunting): picked mon.noname-c con 0x7f485c1416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f48636ed640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f48636ed640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c1390e0 con 0x7f485c13b020 2024-03-27T21:41:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f48636ed640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f48636ed640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c13ace0 con 0x7f485c0b40b0 2024-03-27T21:41:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f48636ed640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f48636ed640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485c13e2f0 con 0x7f485c1416f0 2024-03-27T21:41:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f48636ed640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f48636ed640 10 monclient(hunting): authenticate will time out at 909.253561s 2024-03-27T21:41:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4860c88640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f485c13b020 0x7f485c138880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4861c8a640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f485c1416f0 0x7f485c139620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4860c88640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f485c13b020 0x7f485c138880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44868/0 (socket says 172.21.15.60:44868) 2024-03-27T21:41:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4860c88640 1 -- 172.21.15.60:0/889321766 learned_addr learned my addr 172.21.15.60:0/889321766 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4860c88640 10 monclient(hunting): get_auth_request con 0x7f485c13b020 auth_method 0 2024-03-27T21:41:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4860c88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4860c88640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4860c88640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4861c8a640 10 monclient(hunting): get_auth_request con 0x7f485c1416f0 auth_method 0 2024-03-27T21:41:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4861c8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4861c8a640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4861c8a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4861489640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f485c0b40b0 0x7f485c13d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4860c88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4860c88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4860c88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4861489640 10 monclient(hunting): get_auth_request con 0x7f485c0b40b0 auth_method 0 2024-03-27T21:41:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4861489640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4861489640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4861489640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4861c8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4861c8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.032+0000 7f4861c8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4861489640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4861489640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4861489640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4860c88640 10 monclient(hunting): handle_auth_done global_id 4878 payload 824 2024-03-27T21:41:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4860c88640 1 -- 172.21.15.60:0/889321766 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f485c1416f0 msgr2=0x7f485c139620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4860c88640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f485c1416f0 0x7f485c139620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4860c88640 1 -- 172.21.15.60:0/889321766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f485c0b40b0 msgr2=0x7f485c13d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4860c88640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f485c0b40b0 0x7f485c13d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4860c88640 10 monclient: _finish_hunting 0 2024-03-27T21:41:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4860c88640 1 monclient: found mon.noname-a 2024-03-27T21:41:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4860c88640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4860c88640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4860c88640 1 -- 172.21.15.60:0/889321766 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c009400 con 0x7f485c13b020 2024-03-27T21:41:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4861489640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f485c0b40b0 0x7f485c13d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4861c8a640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f485c1416f0 0x7f485c139620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4860c88640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f485c13b020 0x7f485c138880 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f484c000c00 tx=0x7f484c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4851ffb640 10 client.?.objecter ms_handle_connect 0x7f485c13b020 2024-03-27T21:41:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4851ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4851ffb640 1 -- 172.21.15.60:0/889321766 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c03e040 con 0x7f485c13b020 2024-03-27T21:41:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4851ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4851ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4851ffb640 10 monclient: dump: 2024-03-27T21:41:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:08.038 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:08.038 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:08.038 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:08.038 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:08.038 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:08.038 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:08.038 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:08.038 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:08.038 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:08.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4851ffb640 10 monclient: _finish_auth 0 2024-03-27T21:41:08.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4851ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:41:08.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4851ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:08.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4851ffb640 1 -- 172.21.15.60:0/889321766 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484c039c90 con 0x7f485c13b020 2024-03-27T21:41:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f48636ed640 5 monclient: authenticate success, global_id 4878 2024-03-27T21:41:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f48636ed640 10 monclient: _renew_subs 2024-03-27T21:41:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f48636ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f48636ed640 1 -- 172.21.15.60:0/889321766 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485c138dc0 con 0x7f485c13b020 2024-03-27T21:41:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4851ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4851ffb640 1 -- 172.21.15.60:0/889321766 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c030e10 con 0x7f485c13b020 2024-03-27T21:41:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4851ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4851ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f4851ffb640 10 monclient: dump: 2024-03-27T21:41:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:08.040 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.033+0000 7f48537fe640 4 set_mon_vals no callback set 2024-03-27T21:41:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.034+0000 7f48636ed640 10 client.4878.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.034+0000 7f48636ed640 10 monclient: _renew_subs 2024-03-27T21:41:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.034+0000 7f48636ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.034+0000 7f48636ed640 1 -- 172.21.15.60:0/889321766 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485c1d9600 con 0x7f485c13b020 2024-03-27T21:41:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.034+0000 7f48636ed640 1 librados: init done 2024-03-27T21:41:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.034+0000 7f48636ed640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:41:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.034+0000 7f48636ed640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.034+0000 7f48636ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.034+0000 7f48636ed640 1 -- 172.21.15.60:0/889321766 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f485c069470 con 0x7f485c13b020 2024-03-27T21:41:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.035+0000 7f4851ffb640 1 -- 172.21.15.60:0/889321766 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f484c04d3b0 con 0x7f485c13b020 2024-03-27T21:41:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f4851ffb640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f484006c590 0x7f484006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f4861489640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f484006c590 0x7f484006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f4851ffb640 1 -- 172.21.15.60:0/889321766 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f484c0ab550 con 0x7f485c13b020 2024-03-27T21:41:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f4851ffb640 10 client.4878.objecter ms_dispatch 0x7f485c000b90 osd_map(52..52 src has 1..52) v4 2024-03-27T21:41:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f4861489640 10 monclient: get_auth_request con 0x7f484006c590 auth_method 0 2024-03-27T21:41:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f4851ffb640 3 client.4878.objecter handle_osd_map got epochs [52,52] > 0 2024-03-27T21:41:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f4851ffb640 3 client.4878.objecter handle_osd_map decoding full epoch 52 2024-03-27T21:41:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f4851ffb640 20 client.4878.objecter dump_active .. 0 homeless 2024-03-27T21:41:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f4851ffb640 1 -- 172.21.15.60:0/889321766 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v52) v1 ==== 54+0+16 (secure 0 0 0) 0x7f484c0b2070 con 0x7f485c13b020 2024-03-27T21:41:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f4851ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f4851ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f48636ed640 10 client.4878.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign -1 1 2024-03-27T21:41:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f48636ed640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f485c133150 0x7f485c133530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c14cd30 osd=0 3 2024-03-27T21:41:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c14cd30 osd=0 3 2024-03-27T21:41:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c14cd30 osd=0 3 2024-03-27T21:41:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f48636ed640 15 client.4878.objecter _session_command_op_remove -1 1 2024-03-27T21:41:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f48636ed640 20 client.4878.objecter get_session s=0x7f485c14cd30 osd=0 3 2024-03-27T21:41:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign 0 1 2024-03-27T21:41:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c14cd30 osd=0 4 2024-03-27T21:41:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f48636ed640 10 client.4878.objecter _send_command 1 2024-03-27T21:41:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f48636ed640 1 -- 172.21.15.60:0/889321766 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f485c14cea0 con 0x7f485c133150 2024-03-27T21:41:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f4851ffb640 10 client.4878.objecter ms_handle_connect 0x7f484006c590 2024-03-27T21:41:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f4861489640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f484006c590 0x7f484006ea50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4858005880 tx=0x7f4858009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f4861c8a640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f485c133150 0x7f485c133530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.036+0000 7f4861c8a640 10 monclient: get_auth_request con 0x7f485c133150 auth_method 0 2024-03-27T21:41:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.037+0000 7f4851ffb640 10 client.4878.objecter ms_handle_connect 0x7f485c133150 2024-03-27T21:41:08.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.037+0000 7f4861c8a640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f485c133150 0x7f485c133530 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:08.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.037+0000 7f4851ffb640 1 -- 172.21.15.60:0/889321766 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f485c14cea0 con 0x7f485c133150 2024-03-27T21:41:08.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.037+0000 7f4851ffb640 10 client.4878.objecter ms_dispatch 0x7f485c000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:41:08.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.037+0000 7f4851ffb640 10 client.4878.objecter _finish_command 1 = system:0 2024-03-27T21:41:08.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.037+0000 7f4851ffb640 20 client.4878.objecter put_session s=0x7f485c14cd30 osd=0 4 2024-03-27T21:41:08.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.037+0000 7f4851ffb640 15 client.4878.objecter _session_command_op_remove 0 1 2024-03-27T21:41:08.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.051+0000 7f48636ed640 10 client.4878.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:08.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.052+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign -1 2 2024-03-27T21:41:08.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.052+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c14cd30 osd=0 3 2024-03-27T21:41:08.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.052+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c14cd30 osd=0 3 2024-03-27T21:41:08.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.052+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c14cd30 osd=0 3 2024-03-27T21:41:08.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.052+0000 7f48636ed640 15 client.4878.objecter _session_command_op_remove -1 2 2024-03-27T21:41:08.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.052+0000 7f48636ed640 20 client.4878.objecter get_session s=0x7f485c14cd30 osd=0 3 2024-03-27T21:41:08.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.052+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign 0 2 2024-03-27T21:41:08.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.052+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c14cd30 osd=0 4 2024-03-27T21:41:08.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.052+0000 7f48636ed640 10 client.4878.objecter _send_command 2 2024-03-27T21:41:08.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.052+0000 7f48636ed640 1 -- 172.21.15.60:0/889321766 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f485c14d010 con 0x7f485c133150 2024-03-27T21:41:08.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.053+0000 7f4851ffb640 1 -- 172.21.15.60:0/889321766 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f485c14d010 con 0x7f485c133150 2024-03-27T21:41:08.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.053+0000 7f4851ffb640 10 client.4878.objecter ms_dispatch 0x7f485c000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:08.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.053+0000 7f4851ffb640 10 client.4878.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:08.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.053+0000 7f4851ffb640 20 client.4878.objecter put_session s=0x7f485c14cd30 osd=0 4 2024-03-27T21:41:08.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.053+0000 7f4851ffb640 15 client.4878.objecter _session_command_op_remove 0 2 2024-03-27T21:41:08.045 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:41:08.046 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:41:08.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.053+0000 7f48636ed640 10 client.4878.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:08.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.053+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign -1 3 2024-03-27T21:41:08.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.053+0000 7f48636ed640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f485c0b1e70 0x7f485c0b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:08.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.053+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c0b1d00 osd=1 3 2024-03-27T21:41:08.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.053+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c0b1d00 osd=1 3 2024-03-27T21:41:08.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.053+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c0b1d00 osd=1 3 2024-03-27T21:41:08.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.053+0000 7f48636ed640 15 client.4878.objecter _session_command_op_remove -1 3 2024-03-27T21:41:08.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.053+0000 7f48636ed640 20 client.4878.objecter get_session s=0x7f485c0b1d00 osd=1 3 2024-03-27T21:41:08.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.053+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign 1 3 2024-03-27T21:41:08.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.053+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c0b1d00 osd=1 4 2024-03-27T21:41:08.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.053+0000 7f48636ed640 10 client.4878.objecter _send_command 3 2024-03-27T21:41:08.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.053+0000 7f48636ed640 1 -- 172.21.15.60:0/889321766 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f485c1381e0 con 0x7f485c0b1e70 2024-03-27T21:41:08.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.054+0000 7f4861489640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f485c0b1e70 0x7f485c0b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:08.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.054+0000 7f4861489640 10 monclient: get_auth_request con 0x7f485c0b1e70 auth_method 0 2024-03-27T21:41:08.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.054+0000 7f4851ffb640 10 client.4878.objecter ms_handle_connect 0x7f485c0b1e70 2024-03-27T21:41:08.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.054+0000 7f4861489640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f485c0b1e70 0x7f485c0b4bc0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:08.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.055+0000 7f4851ffb640 1 -- 172.21.15.60:0/889321766 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f485c1381e0 con 0x7f485c0b1e70 2024-03-27T21:41:08.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.055+0000 7f4851ffb640 10 client.4878.objecter ms_dispatch 0x7f485c000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:41:08.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.055+0000 7f4851ffb640 10 client.4878.objecter _finish_command 3 = system:0 2024-03-27T21:41:08.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.055+0000 7f4851ffb640 20 client.4878.objecter put_session s=0x7f485c0b1d00 osd=1 4 2024-03-27T21:41:08.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.055+0000 7f4851ffb640 15 client.4878.objecter _session_command_op_remove 1 3 2024-03-27T21:41:08.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.070+0000 7f48636ed640 10 client.4878.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:08.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.070+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign -1 4 2024-03-27T21:41:08.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.070+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c0b1d00 osd=1 3 2024-03-27T21:41:08.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.070+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c0b1d00 osd=1 3 2024-03-27T21:41:08.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.070+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c0b1d00 osd=1 3 2024-03-27T21:41:08.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.070+0000 7f48636ed640 15 client.4878.objecter _session_command_op_remove -1 4 2024-03-27T21:41:08.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.070+0000 7f48636ed640 20 client.4878.objecter get_session s=0x7f485c0b1d00 osd=1 3 2024-03-27T21:41:08.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.070+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign 1 4 2024-03-27T21:41:08.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.070+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c0b1d00 osd=1 4 2024-03-27T21:41:08.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.070+0000 7f48636ed640 10 client.4878.objecter _send_command 4 2024-03-27T21:41:08.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.070+0000 7f48636ed640 1 -- 172.21.15.60:0/889321766 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f485c138350 con 0x7f485c0b1e70 2024-03-27T21:41:08.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.072+0000 7f4851ffb640 1 -- 172.21.15.60:0/889321766 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f485c138350 con 0x7f485c0b1e70 2024-03-27T21:41:08.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.072+0000 7f4851ffb640 10 client.4878.objecter ms_dispatch 0x7f485c000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:08.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.072+0000 7f4851ffb640 10 client.4878.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:08.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.072+0000 7f4851ffb640 20 client.4878.objecter put_session s=0x7f485c0b1d00 osd=1 4 2024-03-27T21:41:08.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.072+0000 7f4851ffb640 15 client.4878.objecter _session_command_op_remove 1 4 2024-03-27T21:41:08.060 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:41:08.062 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:41:08.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.072+0000 7f48636ed640 10 client.4878.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:08.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.072+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign -1 5 2024-03-27T21:41:08.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.072+0000 7f48636ed640 20 client.4878.objecter _get_session osd=-1 returning homeless 2024-03-27T21:41:08.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f48636ed640 10 client.4878.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:08.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f48636ed640 10 monclient: _renew_subs 2024-03-27T21:41:08.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f48636ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:08.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f48636ed640 1 -- 172.21.15.60:0/889321766 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=53}) v3 -- 0x7f485c137990 con 0x7f485c13b020 2024-03-27T21:41:08.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f48636ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:08.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f48636ed640 1 -- 172.21.15.60:0/889321766 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f485c0b58c0 con 0x7f485c13b020 2024-03-27T21:41:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f4851ffb640 1 -- 172.21.15.60:0/889321766 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_get_version_reply(handle=1 version=52) v2 ==== 24+0+0 (secure 0 0 0) 0x7f484c0769a0 con 0x7f485c13b020 2024-03-27T21:41:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f4851ffb640 10 monclient: handle_get_version_reply finishing 1 version 52 2024-03-27T21:41:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f48537fe640 10 client.4878.objecter _check_command_map_dne tid 5 current 52 map_dne_bound 52 2024-03-27T21:41:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f48537fe640 10 client.4878.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:41:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f48537fe640 15 client.4878.objecter _session_command_op_remove -1 5 2024-03-27T21:41:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:41:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f48636ed640 10 client.4878.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign -1 6 2024-03-27T21:41:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f48636ed640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f485c02b7f0 0x7f485c02bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c02b170 osd=3 3 2024-03-27T21:41:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c02b170 osd=3 3 2024-03-27T21:41:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c02b170 osd=3 3 2024-03-27T21:41:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f48636ed640 15 client.4878.objecter _session_command_op_remove -1 6 2024-03-27T21:41:08.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f48636ed640 20 client.4878.objecter get_session s=0x7f485c02b170 osd=3 3 2024-03-27T21:41:08.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign 3 6 2024-03-27T21:41:08.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c02b170 osd=3 4 2024-03-27T21:41:08.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f48636ed640 10 client.4878.objecter _send_command 6 2024-03-27T21:41:08.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f48636ed640 1 -- 172.21.15.60:0/889321766 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f485c02c110 con 0x7f485c02b7f0 2024-03-27T21:41:08.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f4860c88640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f485c02b7f0 0x7f485c02bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:08.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f4860c88640 10 monclient: get_auth_request con 0x7f485c02b7f0 auth_method 0 2024-03-27T21:41:08.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f4851ffb640 10 client.4878.objecter ms_handle_connect 0x7f485c02b7f0 2024-03-27T21:41:08.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.073+0000 7f4860c88640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f485c02b7f0 0x7f485c02bbd0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:08.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.074+0000 7f4851ffb640 1 -- 172.21.15.60:0/889321766 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f485c02c110 con 0x7f485c02b7f0 2024-03-27T21:41:08.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.074+0000 7f4851ffb640 10 client.4878.objecter ms_dispatch 0x7f485c000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:41:08.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.074+0000 7f4851ffb640 10 client.4878.objecter _finish_command 6 = system:0 2024-03-27T21:41:08.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.074+0000 7f4851ffb640 20 client.4878.objecter put_session s=0x7f485c02b170 osd=3 4 2024-03-27T21:41:08.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.074+0000 7f4851ffb640 15 client.4878.objecter _session_command_op_remove 3 6 2024-03-27T21:41:08.070 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:08.070 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:08.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.088+0000 7f48636ed640 10 client.4878.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:08.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.089+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign -1 7 2024-03-27T21:41:08.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.089+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c02b170 osd=3 3 2024-03-27T21:41:08.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.089+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c02b170 osd=3 3 2024-03-27T21:41:08.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.089+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c02b170 osd=3 3 2024-03-27T21:41:08.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.089+0000 7f48636ed640 15 client.4878.objecter _session_command_op_remove -1 7 2024-03-27T21:41:08.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.089+0000 7f48636ed640 20 client.4878.objecter get_session s=0x7f485c02b170 osd=3 3 2024-03-27T21:41:08.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.089+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign 3 7 2024-03-27T21:41:08.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.089+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c02b170 osd=3 4 2024-03-27T21:41:08.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.089+0000 7f48636ed640 10 client.4878.objecter _send_command 7 2024-03-27T21:41:08.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.089+0000 7f48636ed640 1 -- 172.21.15.60:0/889321766 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f485c02c520 con 0x7f485c02b7f0 2024-03-27T21:41:08.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.091+0000 7f4851ffb640 1 -- 172.21.15.60:0/889321766 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f485c02c520 con 0x7f485c02b7f0 2024-03-27T21:41:08.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.091+0000 7f4851ffb640 10 client.4878.objecter ms_dispatch 0x7f485c000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:08.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.091+0000 7f4851ffb640 10 client.4878.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:08.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.091+0000 7f4851ffb640 20 client.4878.objecter put_session s=0x7f485c02b170 osd=3 4 2024-03-27T21:41:08.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.091+0000 7f4851ffb640 15 client.4878.objecter _session_command_op_remove 3 7 2024-03-27T21:41:08.078 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:41:08.079 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:41:08.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.091+0000 7f48636ed640 10 client.4878.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:08.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.091+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign -1 8 2024-03-27T21:41:08.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.091+0000 7f48636ed640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f485c066250 0x7f485c066690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:08.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.091+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c02c690 osd=4 3 2024-03-27T21:41:08.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.091+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c02c690 osd=4 3 2024-03-27T21:41:08.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.091+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c02c690 osd=4 3 2024-03-27T21:41:08.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.091+0000 7f48636ed640 15 client.4878.objecter _session_command_op_remove -1 8 2024-03-27T21:41:08.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.091+0000 7f48636ed640 20 client.4878.objecter get_session s=0x7f485c02c690 osd=4 3 2024-03-27T21:41:08.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.091+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign 4 8 2024-03-27T21:41:08.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.091+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c02c690 osd=4 4 2024-03-27T21:41:08.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.091+0000 7f48636ed640 10 client.4878.objecter _send_command 8 2024-03-27T21:41:08.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.091+0000 7f48636ed640 1 -- 172.21.15.60:0/889321766 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f485c1ec1a0 con 0x7f485c066250 2024-03-27T21:41:08.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.092+0000 7f4861c8a640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f485c066250 0x7f485c066690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:08.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.092+0000 7f4861c8a640 10 monclient: get_auth_request con 0x7f485c066250 auth_method 0 2024-03-27T21:41:08.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.092+0000 7f4851ffb640 10 client.4878.objecter ms_handle_connect 0x7f485c066250 2024-03-27T21:41:08.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.092+0000 7f4861c8a640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f485c066250 0x7f485c066690 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:08.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.093+0000 7f4851ffb640 1 -- 172.21.15.60:0/889321766 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f485c1ec1a0 con 0x7f485c066250 2024-03-27T21:41:08.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.093+0000 7f4851ffb640 10 client.4878.objecter ms_dispatch 0x7f485c000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:41:08.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.093+0000 7f4851ffb640 10 client.4878.objecter _finish_command 8 = system:0 2024-03-27T21:41:08.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.093+0000 7f4851ffb640 20 client.4878.objecter put_session s=0x7f485c02c690 osd=4 4 2024-03-27T21:41:08.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.093+0000 7f4851ffb640 15 client.4878.objecter _session_command_op_remove 4 8 2024-03-27T21:41:08.082 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:41:08.090 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:08.105+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:08.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.107+0000 7f48636ed640 10 client.4878.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:08.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.107+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign -1 9 2024-03-27T21:41:08.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.107+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c02c690 osd=4 3 2024-03-27T21:41:08.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.107+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c02c690 osd=4 3 2024-03-27T21:41:08.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.107+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c02c690 osd=4 3 2024-03-27T21:41:08.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.107+0000 7f48636ed640 15 client.4878.objecter _session_command_op_remove -1 9 2024-03-27T21:41:08.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.107+0000 7f48636ed640 20 client.4878.objecter get_session s=0x7f485c02c690 osd=4 3 2024-03-27T21:41:08.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.107+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign 4 9 2024-03-27T21:41:08.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.107+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c02c690 osd=4 4 2024-03-27T21:41:08.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.107+0000 7f48636ed640 10 client.4878.objecter _send_command 9 2024-03-27T21:41:08.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.107+0000 7f48636ed640 1 -- 172.21.15.60:0/889321766 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f485c065cd0 con 0x7f485c066250 2024-03-27T21:41:08.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.109+0000 7f4851ffb640 1 -- 172.21.15.60:0/889321766 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f485c065cd0 con 0x7f485c066250 2024-03-27T21:41:08.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.109+0000 7f4851ffb640 10 client.4878.objecter ms_dispatch 0x7f485c000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:08.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.109+0000 7f4851ffb640 10 client.4878.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:08.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.109+0000 7f4851ffb640 20 client.4878.objecter put_session s=0x7f485c02c690 osd=4 4 2024-03-27T21:41:08.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.109+0000 7f4851ffb640 15 client.4878.objecter _session_command_op_remove 4 9 2024-03-27T21:41:08.097 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:41:08.098 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:41:08.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.109+0000 7f48636ed640 10 client.4878.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:08.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.109+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign -1 10 2024-03-27T21:41:08.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.109+0000 7f48636ed640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f485c065e70 0x7f485c1eaa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:08.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.109+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c065cd0 osd=5 3 2024-03-27T21:41:08.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.109+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c065cd0 osd=5 3 2024-03-27T21:41:08.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.109+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c065cd0 osd=5 3 2024-03-27T21:41:08.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.109+0000 7f48636ed640 15 client.4878.objecter _session_command_op_remove -1 10 2024-03-27T21:41:08.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.109+0000 7f48636ed640 20 client.4878.objecter get_session s=0x7f485c065cd0 osd=5 3 2024-03-27T21:41:08.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.109+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign 5 10 2024-03-27T21:41:08.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.109+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c065cd0 osd=5 4 2024-03-27T21:41:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.109+0000 7f48636ed640 10 client.4878.objecter _send_command 10 2024-03-27T21:41:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.109+0000 7f48636ed640 1 -- 172.21.15.60:0/889321766 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f485c1e7450 con 0x7f485c065e70 2024-03-27T21:41:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.110+0000 7f4861489640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f485c065e70 0x7f485c1eaa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.110+0000 7f4861489640 10 monclient: get_auth_request con 0x7f485c065e70 auth_method 0 2024-03-27T21:41:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.110+0000 7f4861489640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f485c065e70 0x7f485c1eaa90 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.110+0000 7f4851ffb640 10 client.4878.objecter ms_handle_connect 0x7f485c065e70 2024-03-27T21:41:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.111+0000 7f4851ffb640 1 -- 172.21.15.60:0/889321766 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f485c1e7450 con 0x7f485c065e70 2024-03-27T21:41:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.111+0000 7f4851ffb640 10 client.4878.objecter ms_dispatch 0x7f485c000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:41:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.111+0000 7f4851ffb640 10 client.4878.objecter _finish_command 10 = system:0 2024-03-27T21:41:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.111+0000 7f4851ffb640 20 client.4878.objecter put_session s=0x7f485c065cd0 osd=5 4 2024-03-27T21:41:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.111+0000 7f4851ffb640 15 client.4878.objecter _session_command_op_remove 5 10 2024-03-27T21:41:08.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.133+0000 7f48636ed640 10 client.4878.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:08.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.133+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign -1 11 2024-03-27T21:41:08.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.133+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c065cd0 osd=5 3 2024-03-27T21:41:08.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.133+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c065cd0 osd=5 3 2024-03-27T21:41:08.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.133+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c065cd0 osd=5 3 2024-03-27T21:41:08.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.133+0000 7f48636ed640 15 client.4878.objecter _session_command_op_remove -1 11 2024-03-27T21:41:08.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.133+0000 7f48636ed640 20 client.4878.objecter get_session s=0x7f485c065cd0 osd=5 3 2024-03-27T21:41:08.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.133+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign 5 11 2024-03-27T21:41:08.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.133+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c065cd0 osd=5 4 2024-03-27T21:41:08.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.133+0000 7f48636ed640 10 client.4878.objecter _send_command 11 2024-03-27T21:41:08.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.133+0000 7f48636ed640 1 -- 172.21.15.60:0/889321766 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f485c1e7450 con 0x7f485c065e70 2024-03-27T21:41:08.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.135+0000 7f4851ffb640 1 -- 172.21.15.60:0/889321766 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f485c1e7450 con 0x7f485c065e70 2024-03-27T21:41:08.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.135+0000 7f4851ffb640 10 client.4878.objecter ms_dispatch 0x7f485c000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:08.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.135+0000 7f4851ffb640 10 client.4878.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:08.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.135+0000 7f4851ffb640 20 client.4878.objecter put_session s=0x7f485c065cd0 osd=5 4 2024-03-27T21:41:08.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.135+0000 7f4851ffb640 15 client.4878.objecter _session_command_op_remove 5 11 2024-03-27T21:41:08.122 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:41:08.123 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:41:08.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.135+0000 7f48636ed640 10 client.4878.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:08.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.135+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign -1 12 2024-03-27T21:41:08.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.135+0000 7f48636ed640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f485c1eb4e0 0x7f485c1eb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:08.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.135+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c1e7450 osd=6 3 2024-03-27T21:41:08.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.135+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c1e7450 osd=6 3 2024-03-27T21:41:08.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.135+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c1e7450 osd=6 3 2024-03-27T21:41:08.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.135+0000 7f48636ed640 15 client.4878.objecter _session_command_op_remove -1 12 2024-03-27T21:41:08.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.135+0000 7f48636ed640 20 client.4878.objecter get_session s=0x7f485c1e7450 osd=6 3 2024-03-27T21:41:08.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.135+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign 6 12 2024-03-27T21:41:08.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.135+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c1e7450 osd=6 4 2024-03-27T21:41:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.135+0000 7f48636ed640 10 client.4878.objecter _send_command 12 2024-03-27T21:41:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.135+0000 7f48636ed640 1 -- 172.21.15.60:0/889321766 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f485c1ebea0 con 0x7f485c1eb4e0 2024-03-27T21:41:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.135+0000 7f4860c88640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f485c1eb4e0 0x7f485c1eb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.135+0000 7f4860c88640 10 monclient: get_auth_request con 0x7f485c1eb4e0 auth_method 0 2024-03-27T21:41:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.136+0000 7f4851ffb640 10 client.4878.objecter ms_handle_connect 0x7f485c1eb4e0 2024-03-27T21:41:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.136+0000 7f4860c88640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f485c1eb4e0 0x7f485c1eb960 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.136+0000 7f4851ffb640 1 -- 172.21.15.60:0/889321766 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f485c1ebea0 con 0x7f485c1eb4e0 2024-03-27T21:41:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.136+0000 7f4851ffb640 10 client.4878.objecter ms_dispatch 0x7f485c000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:41:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.136+0000 7f4851ffb640 10 client.4878.objecter _finish_command 12 = system:0 2024-03-27T21:41:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.136+0000 7f4851ffb640 20 client.4878.objecter put_session s=0x7f485c1e7450 osd=6 4 2024-03-27T21:41:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.136+0000 7f4851ffb640 15 client.4878.objecter _session_command_op_remove 6 12 2024-03-27T21:41:08.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.151+0000 7f48636ed640 10 client.4878.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:08.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.151+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign -1 13 2024-03-27T21:41:08.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.151+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c1e7450 osd=6 3 2024-03-27T21:41:08.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.151+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c1e7450 osd=6 3 2024-03-27T21:41:08.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.151+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c1e7450 osd=6 3 2024-03-27T21:41:08.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.151+0000 7f48636ed640 15 client.4878.objecter _session_command_op_remove -1 13 2024-03-27T21:41:08.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.151+0000 7f48636ed640 20 client.4878.objecter get_session s=0x7f485c1e7450 osd=6 3 2024-03-27T21:41:08.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.151+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign 6 13 2024-03-27T21:41:08.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.151+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c1e7450 osd=6 4 2024-03-27T21:41:08.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.151+0000 7f48636ed640 10 client.4878.objecter _send_command 13 2024-03-27T21:41:08.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.151+0000 7f48636ed640 1 -- 172.21.15.60:0/889321766 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f485c1ebea0 con 0x7f485c1eb4e0 2024-03-27T21:41:08.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.153+0000 7f4851ffb640 1 -- 172.21.15.60:0/889321766 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f485c1ebea0 con 0x7f485c1eb4e0 2024-03-27T21:41:08.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.153+0000 7f4851ffb640 10 client.4878.objecter ms_dispatch 0x7f485c000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:08.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.153+0000 7f4851ffb640 10 client.4878.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:08.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.153+0000 7f4851ffb640 20 client.4878.objecter put_session s=0x7f485c1e7450 osd=6 4 2024-03-27T21:41:08.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.153+0000 7f4851ffb640 15 client.4878.objecter _session_command_op_remove 6 13 2024-03-27T21:41:08.140 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:41:08.141 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:41:08.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.153+0000 7f48636ed640 10 client.4878.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:08.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.153+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign -1 14 2024-03-27T21:41:08.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.153+0000 7f48636ed640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f485c1f0840 0x7f485c1f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:08.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.153+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c1ebea0 osd=7 3 2024-03-27T21:41:08.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.153+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c1ebea0 osd=7 3 2024-03-27T21:41:08.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.153+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c1ebea0 osd=7 3 2024-03-27T21:41:08.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.153+0000 7f48636ed640 15 client.4878.objecter _session_command_op_remove -1 14 2024-03-27T21:41:08.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.153+0000 7f48636ed640 20 client.4878.objecter get_session s=0x7f485c1ebea0 osd=7 3 2024-03-27T21:41:08.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.153+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign 7 14 2024-03-27T21:41:08.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.153+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c1ebea0 osd=7 4 2024-03-27T21:41:08.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.153+0000 7f48636ed640 10 client.4878.objecter _send_command 14 2024-03-27T21:41:08.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.153+0000 7f48636ed640 1 -- 172.21.15.60:0/889321766 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f485c1e7620 con 0x7f485c1f0840 2024-03-27T21:41:08.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.154+0000 7f4861c8a640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f485c1f0840 0x7f485c1f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:08.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.154+0000 7f4861c8a640 10 monclient: get_auth_request con 0x7f485c1f0840 auth_method 0 2024-03-27T21:41:08.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.154+0000 7f4851ffb640 10 client.4878.objecter ms_handle_connect 0x7f485c1f0840 2024-03-27T21:41:08.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.154+0000 7f4861c8a640 1 --2- 172.21.15.60:0/889321766 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f485c1f0840 0x7f485c1f2c30 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:08.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.155+0000 7f4851ffb640 1 -- 172.21.15.60:0/889321766 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f485c1e7620 con 0x7f485c1f0840 2024-03-27T21:41:08.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.155+0000 7f4851ffb640 10 client.4878.objecter ms_dispatch 0x7f485c000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:41:08.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.155+0000 7f4851ffb640 10 client.4878.objecter _finish_command 14 = system:0 2024-03-27T21:41:08.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.155+0000 7f4851ffb640 20 client.4878.objecter put_session s=0x7f485c1ebea0 osd=7 4 2024-03-27T21:41:08.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.155+0000 7f4851ffb640 15 client.4878.objecter _session_command_op_remove 7 14 2024-03-27T21:41:08.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.170+0000 7f48636ed640 10 client.4878.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:08.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.170+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign -1 15 2024-03-27T21:41:08.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.170+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c1ebea0 osd=7 3 2024-03-27T21:41:08.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.170+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c1ebea0 osd=7 3 2024-03-27T21:41:08.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.170+0000 7f48636ed640 20 client.4878.objecter _get_session s=0x7f485c1ebea0 osd=7 3 2024-03-27T21:41:08.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.170+0000 7f48636ed640 15 client.4878.objecter _session_command_op_remove -1 15 2024-03-27T21:41:08.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.170+0000 7f48636ed640 20 client.4878.objecter get_session s=0x7f485c1ebea0 osd=7 3 2024-03-27T21:41:08.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.170+0000 7f48636ed640 15 client.4878.objecter _session_command_op_assign 7 15 2024-03-27T21:41:08.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.170+0000 7f48636ed640 20 client.4878.objecter put_session s=0x7f485c1ebea0 osd=7 4 2024-03-27T21:41:08.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.170+0000 7f48636ed640 10 client.4878.objecter _send_command 15 2024-03-27T21:41:08.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.170+0000 7f48636ed640 1 -- 172.21.15.60:0/889321766 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f485c1e7620 con 0x7f485c1f0840 2024-03-27T21:41:08.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.172+0000 7f4851ffb640 1 -- 172.21.15.60:0/889321766 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f485c1e7620 con 0x7f485c1f0840 2024-03-27T21:41:08.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.172+0000 7f4851ffb640 10 client.4878.objecter ms_dispatch 0x7f485c000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:08.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.172+0000 7f4851ffb640 10 client.4878.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:08.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.172+0000 7f4851ffb640 20 client.4878.objecter put_session s=0x7f485c1ebea0 osd=7 4 2024-03-27T21:41:08.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:08.172+0000 7f4851ffb640 15 client.4878.objecter _session_command_op_remove 7 15 2024-03-27T21:41:08.158 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:41:08.159 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:41:08.161 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:41:08.162 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:41:08.203 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:08.217+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:08.207 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:08.207 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:08.216 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:41:08.301 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:08.315+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:08.365 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:08.365 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:08.374 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:41:08.393 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:08.407+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:08.494 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:08.508+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:08.545 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:08.546 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:08.554 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:41:08.595 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:08.610+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:08.696 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:08.711+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:08.703 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:08.703 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:08.711 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:41:08.797 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:08.811+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:08.826 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:08.826 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:08.835 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:41:08.919 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:08.933+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:08.984 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:08.984 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:08.992 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:41:08.999 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:09.014+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:09.100 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:09.115+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:09.140 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:09.141 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:09.149 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:41:09.200 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:09.214+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:09.265 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:09.266 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:09.274 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:41:09.317 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:09.332+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:09.379 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:09.380 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:09.388 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:41:09.429 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:09.444+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:09.503 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:09.518+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:09.534 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:09.534 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:09.543 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:41:09.627 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:09.642+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:09.688 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:09.688 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:09.697 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:41:09.705 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:09.719+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:09.806 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:09.820+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:09.842 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:09.842 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:09.851 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:41:09.907 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:09.921+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:09.956 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:09.956 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:09.964 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:41:09.989 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-27T21:41:10.008 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:10.023+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:10.069 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:10.069 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:10.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:41:10.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.104+0000 7f5240f22640 10 monclient: get_monmap_and_config 2024-03-27T21:41:10.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.104+0000 7f5240f22640 10 monclient: build_initial_monmap 2024-03-27T21:41:10.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.104+0000 7f5240f22640 10 monclient: monmap: 2024-03-27T21:41:10.091 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:10.091 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:10.091 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:10.105907+0000 2024-03-27T21:41:10.091 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:10.105907+0000 2024-03-27T21:41:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:10.092 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:10.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.105+0000 7f5240f22640 1 Processor -- start 2024-03-27T21:41:10.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.105+0000 7f5240f22640 1 -- start start 2024-03-27T21:41:10.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.105+0000 7f5240f22640 10 monclient: init 2024-03-27T21:41:10.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5240f22640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:10.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5240f22640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-03-27T21:41:10.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5240f22640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f523c071a50 0x7f523c0a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:10.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5240f22640 10 monclient(hunting): picked mon.noname-a con 0x7f523c071a50 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:10.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5240f22640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f523c13e810 0x7f523c13ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:10.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5240f22640 10 monclient(hunting): picked mon.noname-b con 0x7f523c13e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:10.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5240f22640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f523c141630 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:10.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5240f22640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f523c13f190 con 0x7f523c141630 2024-03-27T21:41:10.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5240f22640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:10.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5240f22640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f523c071e30 con 0x7f523c071a50 2024-03-27T21:41:10.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5240f22640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:10.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5240f22640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523c0a8e90 con 0x7f523c13e810 2024-03-27T21:41:10.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5240f22640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:10.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5240f22640 10 monclient(hunting): authenticate will time out at 911.327285s 2024-03-27T21:41:10.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5233fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f523c13e810 0x7f523c13ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:10.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5233fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f523c13e810 0x7f523c13ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:40160/0 (socket says 172.21.15.60:40160) 2024-03-27T21:41:10.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5233fff640 1 -- 172.21.15.60:0/527993123 learned_addr learned my addr 172.21.15.60:0/527993123 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:10.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5233fff640 10 monclient(hunting): get_auth_request con 0x7f523c13e810 auth_method 0 2024-03-27T21:41:10.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5233fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:10.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5233fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:10.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5233fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:10.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f523a59c640 1 --2- 172.21.15.60:0/527993123 >> v2:172.21.15.60:3300/0 conn(0x7f523c071a50 0x7f523c0a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:10.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f523a59c640 10 monclient(hunting): get_auth_request con 0x7f523c071a50 auth_method 0 2024-03-27T21:41:10.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f523a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:10.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f523a59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:10.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f523a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:10.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5233fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:10.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5233fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:10.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5233fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:10.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f523a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:10.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f523a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:10.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f523a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:10.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5233fff640 10 monclient(hunting): handle_auth_done global_id 4712 payload 274 2024-03-27T21:41:10.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5233fff640 1 -- 172.21.15.60:0/527993123 >> v2:172.21.15.60:3300/0 conn(0x7f523c071a50 msgr2=0x7f523c0a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:10.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5233fff640 1 --2- 172.21.15.60:0/527993123 >> v2:172.21.15.60:3300/0 conn(0x7f523c071a50 0x7f523c0a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:10.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5233fff640 1 -- 172.21.15.60:0/527993123 >> v1:172.21.15.191:6789/0 conn(0x7f523c141630 legacy=0x7f523c141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:10.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5233fff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:10.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5233fff640 1 monclient: found mon.noname-b 2024-03-27T21:41:10.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5233fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:10.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5233fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:10.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f5233fff640 1 -- 172.21.15.60:0/527993123 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523c0aa040 con 0x7f523c13e810 2024-03-27T21:41:10.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.106+0000 7f523a59c640 1 --2- 172.21.15.60:0/527993123 >> v2:172.21.15.60:3300/0 conn(0x7f523c071a50 0x7f523c0a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:10.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.107+0000 7f5233fff640 1 --2- 172.21.15.60:0/527993123 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f523c13e810 0x7f523c13ec50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5224009870 tx=0x7f522402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d6195c64ddb34da server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:10.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.107+0000 7f5239d9b640 1 -- 172.21.15.60:0/527993123 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5224030840 con 0x7f523c13e810 2024-03-27T21:41:10.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.107+0000 7f5239d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:10.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.107+0000 7f5239d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:10.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.107+0000 7f5239d9b640 10 monclient: dump: 2024-03-27T21:41:10.104 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:10.104 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:10.104 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:10.104 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:10.104 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:10.105 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:10.105 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:10.105 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:10.105 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:10.105 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:10.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.107+0000 7f5239d9b640 10 monclient: _finish_auth 0 2024-03-27T21:41:10.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.107+0000 7f5239d9b640 10 monclient: _check_auth_tickets 2024-03-27T21:41:10.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.107+0000 7f5239d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:10.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.107+0000 7f5240f22640 5 monclient: authenticate success, global_id 4712 2024-03-27T21:41:10.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.107+0000 7f5239d9b640 1 -- 172.21.15.60:0/527993123 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52240309a0 con 0x7f523c13e810 2024-03-27T21:41:10.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.107+0000 7f5240f22640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:10.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.107+0000 7f5239d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:10.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.107+0000 7f5240f22640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:10.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.107+0000 7f5239d9b640 1 -- 172.21.15.60:0/527993123 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5224030d30 con 0x7f523c13e810 2024-03-27T21:41:10.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.107+0000 7f5240f22640 4 set_mon_vals no callback set 2024-03-27T21:41:10.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.107+0000 7f5239d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:10.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.107+0000 7f5239d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:10.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.107+0000 7f5239d9b640 10 monclient: dump: 2024-03-27T21:41:10.107 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:10.107 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:10.108 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:10.108 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:10.108 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:10.108 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:10.108 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:10.108 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:10.108 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:10.109 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:10.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.108+0000 7f5240f22640 10 monclient: shutdown 2024-03-27T21:41:10.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.108+0000 7f5240f22640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:10.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.108+0000 7f5240f22640 1 -- 172.21.15.60:0/527993123 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f523c13e810 msgr2=0x7f523c13ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:10.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.108+0000 7f5240f22640 1 --2- 172.21.15.60:0/527993123 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f523c13e810 0x7f523c13ec50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5224009870 tx=0x7f522402b980 comp rx=0 tx=0).stop 2024-03-27T21:41:10.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.108+0000 7f5240f22640 1 -- 172.21.15.60:0/527993123 shutdown_connections 2024-03-27T21:41:10.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.108+0000 7f5240f22640 1 --2- 172.21.15.60:0/527993123 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f523c13e810 0x7f523c13ec50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:10.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.108+0000 7f5240f22640 1 --2- 172.21.15.60:0/527993123 >> v2:172.21.15.60:3300/0 conn(0x7f523c071a50 0x7f523c0a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:10.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.108+0000 7f5240f22640 1 -- 172.21.15.60:0/527993123 >> 172.21.15.60:0/527993123 conn(0x7f523c0b7300 msgr2=0x7f523c0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:10.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.108+0000 7f5240f22640 1 -- 172.21.15.60:0/527993123 shutdown_connections 2024-03-27T21:41:10.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.108+0000 7f5240f22640 1 -- 172.21.15.60:0/527993123 wait complete. 2024-03-27T21:41:10.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.109+0000 7f5240f22640 10 monclient: build_initial_monmap 2024-03-27T21:41:10.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.109+0000 7f5240f22640 10 monclient: monmap: 2024-03-27T21:41:10.110 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:10.110 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:10.110 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:10.110 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:10.110 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:10.110 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:10.110 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:10.110 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:10.110 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:10.110 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:10.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.109+0000 7f5240f22640 1 librados: starting msgr at 2024-03-27T21:41:10.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.109+0000 7f5240f22640 1 librados: starting objecter 2024-03-27T21:41:10.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.109+0000 7f5240f22640 1 Processor -- start 2024-03-27T21:41:10.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.109+0000 7f5240f22640 1 -- start start 2024-03-27T21:41:10.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.109+0000 7f5240f22640 1 librados: setting wanted keys 2024-03-27T21:41:10.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.109+0000 7f5240f22640 1 librados: calling monclient init 2024-03-27T21:41:10.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.109+0000 7f5240f22640 10 monclient: init 2024-03-27T21:41:10.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5240f22640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:10.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5240f22640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:41:10.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5240f22640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f523c071a50 0x7f523c1d69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:10.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5240f22640 10 monclient(hunting): picked mon.noname-b con 0x7f523c071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:10.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5240f22640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f523c13e810 0x7f523c1330e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:10.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5240f22640 10 monclient(hunting): picked mon.noname-c con 0x7f523c13e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:10.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5240f22640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f523c141630 0x7f523c133620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:10.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5240f22640 10 monclient(hunting): picked mon.noname-a con 0x7f523c141630 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:10.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5240f22640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:10.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5240f22640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523c138da0 con 0x7f523c141630 2024-03-27T21:41:10.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5240f22640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:10.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5240f22640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523c1d64a0 con 0x7f523c071a50 2024-03-27T21:41:10.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5240f22640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:10.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5240f22640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523c1398c0 con 0x7f523c13e810 2024-03-27T21:41:10.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5240f22640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:10.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5240f22640 10 monclient(hunting): authenticate will time out at 911.331073s 2024-03-27T21:41:10.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5233fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f523c13e810 0x7f523c1330e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:10.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f523c141630 0x7f523c133620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:10.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5233fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f523c13e810 0x7f523c1330e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:40168/0 (socket says 172.21.15.60:40168) 2024-03-27T21:41:10.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5233fff640 1 -- 172.21.15.60:0/140813696 learned_addr learned my addr 172.21.15.60:0/140813696 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:10.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5233fff640 10 monclient(hunting): get_auth_request con 0x7f523c13e810 auth_method 0 2024-03-27T21:41:10.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5233fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:10.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5233fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:10.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5233fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:10.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523ad9d640 10 monclient(hunting): get_auth_request con 0x7f523c141630 auth_method 0 2024-03-27T21:41:10.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:10.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:10.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:10.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:10.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:10.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:10.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5233fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:10.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5233fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:10.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5233fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:10.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523a59c640 1 --2- 172.21.15.60:0/140813696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f523c071a50 0x7f523c1d69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:10.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523ad9d640 10 monclient(hunting): handle_auth_done global_id 4884 payload 824 2024-03-27T21:41:10.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523ad9d640 1 -- 172.21.15.60:0/140813696 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f523c13e810 msgr2=0x7f523c1330e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:10.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523ad9d640 1 --2- 172.21.15.60:0/140813696 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f523c13e810 0x7f523c1330e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:10.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523ad9d640 1 -- 172.21.15.60:0/140813696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f523c071a50 msgr2=0x7f523c1d69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:10.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523ad9d640 1 --2- 172.21.15.60:0/140813696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f523c071a50 0x7f523c1d69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:10.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523ad9d640 10 monclient: _finish_hunting 0 2024-03-27T21:41:10.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523ad9d640 1 monclient: found mon.noname-a 2024-03-27T21:41:10.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:10.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:10.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523ad9d640 1 -- 172.21.15.60:0/140813696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522c009400 con 0x7f523c141630 2024-03-27T21:41:10.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523a59c640 10 monclient: get_auth_request con 0x7f523c071a50 auth_method 0 2024-03-27T21:41:10.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523a59c640 1 --2- 172.21.15.60:0/140813696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f523c071a50 0x7f523c1d69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:41:10.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f523ad9d640 1 --2- 172.21.15.60:0/140813696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f523c141630 0x7f523c133620 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f523400c700 tx=0x7f523400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:10.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5232ffd640 10 client.?.objecter ms_handle_connect 0x7f523c141630 2024-03-27T21:41:10.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5232ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:10.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5232ffd640 1 -- 172.21.15.60:0/140813696 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5234003b70 con 0x7f523c141630 2024-03-27T21:41:10.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5232ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:10.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5232ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:10.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5232ffd640 10 monclient: dump: 2024-03-27T21:41:10.115 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:10.115 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:10.115 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:10.115 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:10.115 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:10.115 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:10.115 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:10.115 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:10.115 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:10.116 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:10.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5232ffd640 10 monclient: _finish_auth 0 2024-03-27T21:41:10.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5232ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:41:10.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5232ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:10.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5232ffd640 1 -- 172.21.15.60:0/140813696 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5234003cd0 con 0x7f523c141630 2024-03-27T21:41:10.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5240f22640 5 monclient: authenticate success, global_id 4884 2024-03-27T21:41:10.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5240f22640 10 monclient: _renew_subs 2024-03-27T21:41:10.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5240f22640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:10.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5240f22640 1 -- 172.21.15.60:0/140813696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52240094d0 con 0x7f523c141630 2024-03-27T21:41:10.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.110+0000 7f5232ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:10.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.111+0000 7f5232ffd640 1 -- 172.21.15.60:0/140813696 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5234005310 con 0x7f523c141630 2024-03-27T21:41:10.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.111+0000 7f5232ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:10.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.111+0000 7f5232ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:10.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.111+0000 7f5232ffd640 10 monclient: dump: 2024-03-27T21:41:10.117 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:10.117 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:10.117 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:10.117 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:10.117 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:10.117 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:10.117 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:10.117 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:10.117 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:10.117 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:10.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.111+0000 7f523959a640 4 set_mon_vals no callback set 2024-03-27T21:41:10.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.111+0000 7f5240f22640 10 client.4884.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:10.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.111+0000 7f5240f22640 10 monclient: _renew_subs 2024-03-27T21:41:10.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.111+0000 7f5240f22640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:10.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.111+0000 7f5240f22640 1 -- 172.21.15.60:0/140813696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f523c137580 con 0x7f523c141630 2024-03-27T21:41:10.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.111+0000 7f5240f22640 1 librados: init done 2024-03-27T21:41:10.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.111+0000 7f5240f22640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:10.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.111+0000 7f5240f22640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:10.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.111+0000 7f5240f22640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:10.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.111+0000 7f5240f22640 1 -- 172.21.15.60:0/140813696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f523c1d6300 con 0x7f523c141630 2024-03-27T21:41:10.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.112+0000 7f5232ffd640 1 -- 172.21.15.60:0/140813696 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5234005470 con 0x7f523c141630 2024-03-27T21:41:10.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.112+0000 7f5232ffd640 1 --2- 172.21.15.60:0/140813696 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f521c06c590 0x7f521c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:10.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.112+0000 7f5232ffd640 1 -- 172.21.15.60:0/140813696 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f5234088bc0 con 0x7f523c141630 2024-03-27T21:41:10.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.112+0000 7f5232ffd640 10 client.4884.objecter ms_dispatch 0x7f523c000b90 osd_map(52..52 src has 1..52) v4 2024-03-27T21:41:10.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.112+0000 7f5232ffd640 3 client.4884.objecter handle_osd_map got epochs [52,52] > 0 2024-03-27T21:41:10.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.112+0000 7f5232ffd640 3 client.4884.objecter handle_osd_map decoding full epoch 52 2024-03-27T21:41:10.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.112+0000 7f523a59c640 1 --2- 172.21.15.60:0/140813696 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f521c06c590 0x7f521c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:10.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.112+0000 7f5232ffd640 20 client.4884.objecter dump_active .. 0 homeless 2024-03-27T21:41:10.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.112+0000 7f523a59c640 10 monclient: get_auth_request con 0x7f521c06c590 auth_method 0 2024-03-27T21:41:10.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.113+0000 7f5232ffd640 10 client.4884.objecter ms_handle_connect 0x7f521c06c590 2024-03-27T21:41:10.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.113+0000 7f523a59c640 1 --2- 172.21.15.60:0/140813696 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f521c06c590 0x7f521c06ea50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f522c009c50 tx=0x7f522c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:10.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.117+0000 7f5232ffd640 1 -- 172.21.15.60:0/140813696 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f523c1d6300 con 0x7f523c141630 2024-03-27T21:41:10.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.117+0000 7f5232ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:10.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.117+0000 7f5232ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:10.120 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:10.135+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:10.163 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:41:10.220 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:10.234+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:10.224 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:10.224 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:10.233 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:41:10.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.280+0000 7f6c9efa4640 10 monclient: get_monmap_and_config 2024-03-27T21:41:10.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.280+0000 7f6c9efa4640 10 monclient: build_initial_monmap 2024-03-27T21:41:10.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.280+0000 7f6c9efa4640 10 monclient: monmap: 2024-03-27T21:41:10.266 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:10.266 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:10.266 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:10.281286+0000 2024-03-27T21:41:10.267 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:10.281286+0000 2024-03-27T21:41:10.267 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:10.267 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:10.267 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:10.267 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:10.267 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:10.267 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:10.267 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:10.267 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:10.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9efa4640 1 Processor -- start 2024-03-27T21:41:10.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9efa4640 1 -- start start 2024-03-27T21:41:10.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9efa4640 10 monclient: init 2024-03-27T21:41:10.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9efa4640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:10.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9efa4640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-03-27T21:41:10.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9efa4640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6c98071a50 0x7f6c980a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:10.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9efa4640 10 monclient(hunting): picked mon.noname-b con 0x7f6c98071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:10.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9efa4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6c9813e810 0x7f6c9813ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:10.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9efa4640 10 monclient(hunting): picked mon.noname-a con 0x7f6c9813e810 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:10.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9efa4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6c98141630 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:10.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9efa4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c9813f190 con 0x7f6c98141630 2024-03-27T21:41:10.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9efa4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:10.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9efa4640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c98071e30 con 0x7f6c9813e810 2024-03-27T21:41:10.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9efa4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:10.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9efa4640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c980a8e90 con 0x7f6c98071a50 2024-03-27T21:41:10.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9efa4640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:10.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9efa4640 10 monclient(hunting): authenticate will time out at 911.502637s 2024-03-27T21:41:10.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9cd40640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6c98071a50 0x7f6c980a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:10.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c97fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6c9813e810 0x7f6c9813ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:10.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9cd40640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6c98071a50 0x7f6c980a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:40180/0 (socket says 172.21.15.60:40180) 2024-03-27T21:41:10.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c97fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6c9813e810 0x7f6c9813ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44896/0 (socket says 172.21.15.60:44896) 2024-03-27T21:41:10.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9cd40640 1 -- 172.21.15.60:0/3618223978 learned_addr learned my addr 172.21.15.60:0/3618223978 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:10.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9cd40640 10 monclient(hunting): get_auth_request con 0x7f6c98071a50 auth_method 0 2024-03-27T21:41:10.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9cd40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:10.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9cd40640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:10.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9cd40640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:10.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c97fff640 10 monclient(hunting): get_auth_request con 0x7f6c9813e810 auth_method 0 2024-03-27T21:41:10.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:10.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c97fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:10.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:10.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9cd40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:10.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.281+0000 7f6c9cd40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:10.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.282+0000 7f6c9cd40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:10.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.282+0000 7f6c97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:10.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.282+0000 7f6c97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:10.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.282+0000 7f6c97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:10.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.282+0000 7f6c9cd40640 10 monclient(hunting): handle_auth_done global_id 4718 payload 274 2024-03-27T21:41:10.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.282+0000 7f6c9cd40640 1 -- 172.21.15.60:0/3618223978 >> v2:172.21.15.60:3300/0 conn(0x7f6c9813e810 msgr2=0x7f6c9813ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:10.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.282+0000 7f6c9cd40640 1 --2- 172.21.15.60:0/3618223978 >> v2:172.21.15.60:3300/0 conn(0x7f6c9813e810 0x7f6c9813ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:10.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.282+0000 7f6c9cd40640 1 -- 172.21.15.60:0/3618223978 >> v1:172.21.15.60:6789/0 conn(0x7f6c98141630 legacy=0x7f6c98141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:10.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.282+0000 7f6c9cd40640 10 monclient: _finish_hunting 0 2024-03-27T21:41:10.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.282+0000 7f6c9cd40640 1 monclient: found mon.noname-b 2024-03-27T21:41:10.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.282+0000 7f6c9cd40640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:10.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.282+0000 7f6c9cd40640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:10.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.282+0000 7f6c9cd40640 1 -- 172.21.15.60:0/3618223978 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c980aa040 con 0x7f6c98071a50 2024-03-27T21:41:10.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.282+0000 7f6c977fe640 1 -- 172.21.15.60:0/3618223978 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2290606374 0 0) 0x7f6c90001490 con 0x7f6c98141630 2024-03-27T21:41:10.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.282+0000 7f6c977fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:41:10.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.282+0000 7f6c97fff640 1 --2- 172.21.15.60:0/3618223978 >> v2:172.21.15.60:3300/0 conn(0x7f6c9813e810 0x7f6c9813ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:10.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.282+0000 7f6c9cd40640 1 --2- 172.21.15.60:0/3618223978 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6c98071a50 0x7f6c980a87e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c009870 tx=0x7f6c8c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d73397419b53aaaa server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:10.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c977fe640 1 -- 172.21.15.60:0/3618223978 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c030840 con 0x7f6c98071a50 2024-03-27T21:41:10.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:10.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:10.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c977fe640 10 monclient: dump: 2024-03-27T21:41:10.281 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:10.281 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:10.281 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:10.281 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:10.282 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:10.282 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:10.282 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:10.282 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:10.282 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:10.282 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:10.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c977fe640 10 monclient: _finish_auth 0 2024-03-27T21:41:10.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c977fe640 10 monclient: _check_auth_tickets 2024-03-27T21:41:10.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:10.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c977fe640 1 -- 172.21.15.60:0/3618223978 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8c0309a0 con 0x7f6c98071a50 2024-03-27T21:41:10.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c9efa4640 5 monclient: authenticate success, global_id 4718 2024-03-27T21:41:10.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c9efa4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:10.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:10.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c9efa4640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:10.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c977fe640 1 -- 172.21.15.60:0/3618223978 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c030d30 con 0x7f6c98071a50 2024-03-27T21:41:10.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:10.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c9efa4640 4 set_mon_vals no callback set 2024-03-27T21:41:10.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:10.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c977fe640 10 monclient: dump: 2024-03-27T21:41:10.283 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:10.283 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:10.283 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:10.283 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:10.283 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:10.283 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:10.283 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:10.283 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:10.283 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:10.284 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:10.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c9efa4640 10 monclient: shutdown 2024-03-27T21:41:10.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c9efa4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:10.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c9efa4640 1 -- 172.21.15.60:0/3618223978 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6c98071a50 msgr2=0x7f6c980a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:10.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c9efa4640 1 --2- 172.21.15.60:0/3618223978 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6c98071a50 0x7f6c980a87e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c009870 tx=0x7f6c8c02b980 comp rx=0 tx=0).stop 2024-03-27T21:41:10.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c9efa4640 1 -- 172.21.15.60:0/3618223978 shutdown_connections 2024-03-27T21:41:10.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c9efa4640 1 --2- 172.21.15.60:0/3618223978 >> v2:172.21.15.60:3300/0 conn(0x7f6c9813e810 0x7f6c9813ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:10.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c9efa4640 1 --2- 172.21.15.60:0/3618223978 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6c98071a50 0x7f6c980a87e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:10.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c9efa4640 1 -- 172.21.15.60:0/3618223978 >> 172.21.15.60:0/3618223978 conn(0x7f6c980b7300 msgr2=0x7f6c980b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:10.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c9efa4640 1 -- 172.21.15.60:0/3618223978 shutdown_connections 2024-03-27T21:41:10.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.283+0000 7f6c9efa4640 1 -- 172.21.15.60:0/3618223978 wait complete. 2024-03-27T21:41:10.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.284+0000 7f6c9efa4640 10 monclient: build_initial_monmap 2024-03-27T21:41:10.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.284+0000 7f6c9efa4640 10 monclient: monmap: 2024-03-27T21:41:10.284 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:10.285 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:10.285 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:10.285 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:10.285 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:10.285 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:10.285 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:10.285 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:10.285 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:10.285 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:10.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.284+0000 7f6c9efa4640 1 librados: starting msgr at 2024-03-27T21:41:10.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.284+0000 7f6c9efa4640 1 librados: starting objecter 2024-03-27T21:41:10.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.284+0000 7f6c9efa4640 1 Processor -- start 2024-03-27T21:41:10.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.284+0000 7f6c9efa4640 1 -- start start 2024-03-27T21:41:10.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.284+0000 7f6c9efa4640 1 librados: setting wanted keys 2024-03-27T21:41:10.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.284+0000 7f6c9efa4640 1 librados: calling monclient init 2024-03-27T21:41:10.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.284+0000 7f6c9efa4640 10 monclient: init 2024-03-27T21:41:10.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9efa4640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:10.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9efa4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:41:10.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9efa4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c98071a50 0x7f6c9813a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:10.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9efa4640 10 monclient(hunting): picked mon.noname-a con 0x7f6c98071a50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:10.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9efa4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c9813e810 0x7f6c981d9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:10.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9efa4640 10 monclient(hunting): picked mon.noname-b con 0x7f6c9813e810 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:10.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9efa4640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6c98141630 0x7f6c981da220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:10.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9efa4640 10 monclient(hunting): picked mon.noname-c con 0x7f6c98141630 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:10.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9efa4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:10.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9efa4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c980ab160 con 0x7f6c98071a50 2024-03-27T21:41:10.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9efa4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:10.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9efa4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c981d9830 con 0x7f6c9813e810 2024-03-27T21:41:10.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9efa4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:10.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9efa4640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c98138b80 con 0x7f6c98141630 2024-03-27T21:41:10.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9efa4640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:10.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9efa4640 10 monclient(hunting): authenticate will time out at 911.506636s 2024-03-27T21:41:10.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9cd40640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c98071a50 0x7f6c9813a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:10.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9d541640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6c98141630 0x7f6c981da220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:10.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9cd40640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c98071a50 0x7f6c9813a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44906/0 (socket says 172.21.15.60:44906) 2024-03-27T21:41:10.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9cd40640 1 -- 172.21.15.60:0/818722907 learned_addr learned my addr 172.21.15.60:0/818722907 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:10.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9cd40640 10 monclient(hunting): get_auth_request con 0x7f6c98071a50 auth_method 0 2024-03-27T21:41:10.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9cd40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:10.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9cd40640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:10.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9cd40640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:10.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9d541640 10 monclient(hunting): get_auth_request con 0x7f6c98141630 auth_method 0 2024-03-27T21:41:10.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9d541640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:10.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9d541640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:10.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9d541640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:10.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c97fff640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c9813e810 0x7f6c981d9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:10.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9cd40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:10.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9cd40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:10.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9cd40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:10.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9d541640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:10.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9d541640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:10.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c9d541640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:10.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c97fff640 10 monclient(hunting): get_auth_request con 0x7f6c9813e810 auth_method 0 2024-03-27T21:41:10.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:10.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c97fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:10.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.285+0000 7f6c97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:10.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:10.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:10.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:10.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c9cd40640 10 monclient(hunting): handle_auth_done global_id 4893 payload 824 2024-03-27T21:41:10.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c9cd40640 1 -- 172.21.15.60:0/818722907 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6c98141630 msgr2=0x7f6c981da220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:10.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c9cd40640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6c98141630 0x7f6c981da220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:10.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c9cd40640 1 -- 172.21.15.60:0/818722907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c9813e810 msgr2=0x7f6c981d9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:10.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c9cd40640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c9813e810 0x7f6c981d9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:10.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c9cd40640 10 monclient: _finish_hunting 0 2024-03-27T21:41:10.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c9cd40640 1 monclient: found mon.noname-a 2024-03-27T21:41:10.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c9cd40640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:10.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c9cd40640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:10.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c9cd40640 1 -- 172.21.15.60:0/818722907 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c84009400 con 0x7f6c98071a50 2024-03-27T21:41:10.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c9d541640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6c98141630 0x7f6c981da220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:10.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c9cd40640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c98071a50 0x7f6c9813a4f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c02b9c0 tx=0x7f6c8c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:10.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c957fa640 10 client.?.objecter ms_handle_connect 0x7f6c98071a50 2024-03-27T21:41:10.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c957fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:10.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c957fa640 1 -- 172.21.15.60:0/818722907 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c039c90 con 0x7f6c98071a50 2024-03-27T21:41:10.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:10.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c957fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:10.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c957fa640 10 monclient: dump: 2024-03-27T21:41:10.290 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:10.290 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:10.290 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:10.290 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:10.290 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:10.290 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:10.290 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:10.290 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:10.290 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:10.290 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:10.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c957fa640 10 monclient: _finish_auth 0 2024-03-27T21:41:10.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c957fa640 10 monclient: _check_auth_tickets 2024-03-27T21:41:10.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:10.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c9efa4640 5 monclient: authenticate success, global_id 4893 2024-03-27T21:41:10.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c9efa4640 10 monclient: _renew_subs 2024-03-27T21:41:10.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c9efa4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:10.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c957fa640 1 -- 172.21.15.60:0/818722907 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8c039df0 con 0x7f6c98071a50 2024-03-27T21:41:10.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c9efa4640 1 -- 172.21.15.60:0/818722907 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c8c0094d0 con 0x7f6c98071a50 2024-03-27T21:41:10.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:10.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c957fa640 1 -- 172.21.15.60:0/818722907 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c0446c0 con 0x7f6c98071a50 2024-03-27T21:41:10.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:10.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:10.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c957fa640 10 monclient: dump: 2024-03-27T21:41:10.291 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:10.291 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:10.292 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:10.292 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:10.292 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:10.292 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:10.292 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:10.292 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:10.292 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:10.292 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:10.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c9efa4640 10 client.4893.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:10.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c9efa4640 10 monclient: _renew_subs 2024-03-27T21:41:10.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c9efa4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:10.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c96ffd640 4 set_mon_vals no callback set 2024-03-27T21:41:10.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c9efa4640 1 -- 172.21.15.60:0/818722907 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9813a010 con 0x7f6c98071a50 2024-03-27T21:41:10.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.286+0000 7f6c9efa4640 1 librados: init done 2024-03-27T21:41:10.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.287+0000 7f6c9efa4640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:41:10.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.287+0000 7f6c9efa4640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:10.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.287+0000 7f6c9efa4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:10.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.287+0000 7f6c9efa4640 1 -- 172.21.15.60:0/818722907 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6c98069470 con 0x7f6c98071a50 2024-03-27T21:41:10.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.288+0000 7f6c957fa640 1 -- 172.21.15.60:0/818722907 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6c8c02f390 con 0x7f6c98071a50 2024-03-27T21:41:10.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.288+0000 7f6c957fa640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6c7c06c590 0x7f6c7c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:10.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.288+0000 7f6c957fa640 1 -- 172.21.15.60:0/818722907 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f6c8c03e070 con 0x7f6c98071a50 2024-03-27T21:41:10.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.288+0000 7f6c957fa640 10 client.4893.objecter ms_dispatch 0x7f6c98000b90 osd_map(52..52 src has 1..52) v4 2024-03-27T21:41:10.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.288+0000 7f6c957fa640 3 client.4893.objecter handle_osd_map got epochs [52,52] > 0 2024-03-27T21:41:10.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.288+0000 7f6c957fa640 3 client.4893.objecter handle_osd_map decoding full epoch 52 2024-03-27T21:41:10.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.288+0000 7f6c97fff640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6c7c06c590 0x7f6c7c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:10.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.288+0000 7f6c97fff640 10 monclient: get_auth_request con 0x7f6c7c06c590 auth_method 0 2024-03-27T21:41:10.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.288+0000 7f6c957fa640 20 client.4893.objecter dump_active .. 0 homeless 2024-03-27T21:41:10.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.288+0000 7f6c957fa640 1 -- 172.21.15.60:0/818722907 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v52) v1 ==== 54+0+16 (secure 0 0 0) 0x7f6c8c0b2070 con 0x7f6c98071a50 2024-03-27T21:41:10.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.288+0000 7f6c957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:10.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.288+0000 7f6c957fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:10.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.289+0000 7f6c9efa4640 10 client.4893.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:10.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.289+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign -1 1 2024-03-27T21:41:10.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.289+0000 7f6c9efa4640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f6c98132fc0 0x7f6c981333a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:10.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.289+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c981421b0 osd=0 3 2024-03-27T21:41:10.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.289+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c981421b0 osd=0 3 2024-03-27T21:41:10.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.289+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c981421b0 osd=0 3 2024-03-27T21:41:10.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.289+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_remove -1 1 2024-03-27T21:41:10.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.289+0000 7f6c9efa4640 20 client.4893.objecter get_session s=0x7f6c981421b0 osd=0 3 2024-03-27T21:41:10.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.289+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign 0 1 2024-03-27T21:41:10.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.289+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c981421b0 osd=0 4 2024-03-27T21:41:10.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.289+0000 7f6c9efa4640 10 client.4893.objecter _send_command 1 2024-03-27T21:41:10.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.289+0000 7f6c9efa4640 1 -- 172.21.15.60:0/818722907 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c980aa590 con 0x7f6c98132fc0 2024-03-27T21:41:10.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.289+0000 7f6c957fa640 10 client.4893.objecter ms_handle_connect 0x7f6c7c06c590 2024-03-27T21:41:10.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.289+0000 7f6c97fff640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6c7c06c590 0x7f6c7c06ea50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6c981d7350 tx=0x7f6c84009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:10.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.289+0000 7f6c9d541640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f6c98132fc0 0x7f6c981333a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:10.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.289+0000 7f6c9d541640 10 monclient: get_auth_request con 0x7f6c98132fc0 auth_method 0 2024-03-27T21:41:10.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.289+0000 7f6c957fa640 10 client.4893.objecter ms_handle_connect 0x7f6c98132fc0 2024-03-27T21:41:10.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.289+0000 7f6c9d541640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f6c98132fc0 0x7f6c981333a0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:10.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.290+0000 7f6c957fa640 1 -- 172.21.15.60:0/818722907 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6c90001e90 con 0x7f6c98132fc0 2024-03-27T21:41:10.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.290+0000 7f6c957fa640 10 client.4893.objecter ms_dispatch 0x7f6c98000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:41:10.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.290+0000 7f6c957fa640 10 client.4893.objecter _finish_command 1 = system:0 2024-03-27T21:41:10.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.290+0000 7f6c957fa640 20 client.4893.objecter put_session s=0x7f6c981421b0 osd=0 4 2024-03-27T21:41:10.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.290+0000 7f6c957fa640 15 client.4893.objecter _session_command_op_remove 0 1 2024-03-27T21:41:10.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.304+0000 7f6c9efa4640 10 client.4893.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:10.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.304+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign -1 2 2024-03-27T21:41:10.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.304+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c981421b0 osd=0 3 2024-03-27T21:41:10.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.304+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c981421b0 osd=0 3 2024-03-27T21:41:10.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.304+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c981421b0 osd=0 3 2024-03-27T21:41:10.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.304+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_remove -1 2 2024-03-27T21:41:10.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.304+0000 7f6c9efa4640 20 client.4893.objecter get_session s=0x7f6c981421b0 osd=0 3 2024-03-27T21:41:10.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.304+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign 0 2 2024-03-27T21:41:10.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.304+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c981421b0 osd=0 4 2024-03-27T21:41:10.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.304+0000 7f6c9efa4640 10 client.4893.objecter _send_command 2 2024-03-27T21:41:10.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.304+0000 7f6c9efa4640 1 -- 172.21.15.60:0/818722907 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6c9814bde0 con 0x7f6c98132fc0 2024-03-27T21:41:10.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.306+0000 7f6c957fa640 1 -- 172.21.15.60:0/818722907 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6c90013390 con 0x7f6c98132fc0 2024-03-27T21:41:10.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.306+0000 7f6c957fa640 10 client.4893.objecter ms_dispatch 0x7f6c98000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:10.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.306+0000 7f6c957fa640 10 client.4893.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:10.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.306+0000 7f6c957fa640 20 client.4893.objecter put_session s=0x7f6c981421b0 osd=0 4 2024-03-27T21:41:10.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.306+0000 7f6c957fa640 15 client.4893.objecter _session_command_op_remove 0 2 2024-03-27T21:41:10.298 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:41:10.299 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:41:10.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.306+0000 7f6c9efa4640 10 client.4893.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:10.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.306+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign -1 3 2024-03-27T21:41:10.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.306+0000 7f6c9efa4640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f6c981d67a0 0x7f6c981d6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:10.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.306+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c980a9260 osd=1 3 2024-03-27T21:41:10.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.306+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c980a9260 osd=1 3 2024-03-27T21:41:10.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.306+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c980a9260 osd=1 3 2024-03-27T21:41:10.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.306+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_remove -1 3 2024-03-27T21:41:10.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.306+0000 7f6c9efa4640 20 client.4893.objecter get_session s=0x7f6c980a9260 osd=1 3 2024-03-27T21:41:10.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.306+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign 1 3 2024-03-27T21:41:10.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.306+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c980a9260 osd=1 4 2024-03-27T21:41:10.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.306+0000 7f6c9efa4640 10 client.4893.objecter _send_command 3 2024-03-27T21:41:10.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.306+0000 7f6c9efa4640 1 -- 172.21.15.60:0/818722907 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c981d70c0 con 0x7f6c981d67a0 2024-03-27T21:41:10.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.306+0000 7f6c9cd40640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f6c981d67a0 0x7f6c981d6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:10.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.306+0000 7f6c9cd40640 10 monclient: get_auth_request con 0x7f6c981d67a0 auth_method 0 2024-03-27T21:41:10.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.306+0000 7f6c9cd40640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f6c981d67a0 0x7f6c981d6b80 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:10.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.306+0000 7f6c957fa640 10 client.4893.objecter ms_handle_connect 0x7f6c981d67a0 2024-03-27T21:41:10.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.307+0000 7f6c957fa640 1 -- 172.21.15.60:0/818722907 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6c981d70c0 con 0x7f6c981d67a0 2024-03-27T21:41:10.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.307+0000 7f6c957fa640 10 client.4893.objecter ms_dispatch 0x7f6c98000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:41:10.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.307+0000 7f6c957fa640 10 client.4893.objecter _finish_command 3 = system:0 2024-03-27T21:41:10.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.307+0000 7f6c957fa640 20 client.4893.objecter put_session s=0x7f6c980a9260 osd=1 4 2024-03-27T21:41:10.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.307+0000 7f6c957fa640 15 client.4893.objecter _session_command_op_remove 1 3 2024-03-27T21:41:10.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.326+0000 7f5240f22640 10 start_mon_command cmd=[{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:41:10.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.326+0000 7f5240f22640 10 monclient: _send_command 2 [{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:41:10.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.326+0000 7f5240f22640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:10.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.326+0000 7f5240f22640 1 -- 172.21.15.60:0/140813696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f523c13f190 con 0x7f523c141630 2024-03-27T21:41:10.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.332+0000 7f6c9efa4640 10 client.4893.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:10.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.332+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign -1 4 2024-03-27T21:41:10.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.332+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c980a9260 osd=1 3 2024-03-27T21:41:10.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.332+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c980a9260 osd=1 3 2024-03-27T21:41:10.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.332+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c980a9260 osd=1 3 2024-03-27T21:41:10.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.332+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_remove -1 4 2024-03-27T21:41:10.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.332+0000 7f6c9efa4640 20 client.4893.objecter get_session s=0x7f6c980a9260 osd=1 3 2024-03-27T21:41:10.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.332+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign 1 4 2024-03-27T21:41:10.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.332+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c980a9260 osd=1 4 2024-03-27T21:41:10.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.332+0000 7f6c9efa4640 10 client.4893.objecter _send_command 4 2024-03-27T21:41:10.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.332+0000 7f6c9efa4640 1 -- 172.21.15.60:0/818722907 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6c98136aa0 con 0x7f6c981d67a0 2024-03-27T21:41:10.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.333+0000 7f6c957fa640 1 -- 172.21.15.60:0/818722907 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6c98136aa0 con 0x7f6c981d67a0 2024-03-27T21:41:10.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.333+0000 7f6c957fa640 10 client.4893.objecter ms_dispatch 0x7f6c98000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:10.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.333+0000 7f6c957fa640 10 client.4893.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:10.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.333+0000 7f6c957fa640 20 client.4893.objecter put_session s=0x7f6c980a9260 osd=1 4 2024-03-27T21:41:10.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.333+0000 7f6c957fa640 15 client.4893.objecter _session_command_op_remove 1 4 2024-03-27T21:41:10.320 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:41:10.322 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:41:10.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.333+0000 7f6c9efa4640 10 client.4893.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:10.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.333+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign -1 5 2024-03-27T21:41:10.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.333+0000 7f6c9efa4640 20 client.4893.objecter _get_session osd=-1 returning homeless 2024-03-27T21:41:10.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.333+0000 7f6c9efa4640 10 client.4893.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:10.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.333+0000 7f6c9efa4640 10 monclient: _renew_subs 2024-03-27T21:41:10.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.333+0000 7f6c9efa4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:10.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.333+0000 7f6c9efa4640 1 -- 172.21.15.60:0/818722907 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=53}) v3 -- 0x7f6c98134700 con 0x7f6c98071a50 2024-03-27T21:41:10.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.333+0000 7f6c9efa4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:10.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.333+0000 7f6c9efa4640 1 -- 172.21.15.60:0/818722907 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6c98134a50 con 0x7f6c98071a50 2024-03-27T21:41:10.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.334+0000 7f6c957fa640 1 -- 172.21.15.60:0/818722907 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_get_version_reply(handle=1 version=52) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c8c07abd0 con 0x7f6c98071a50 2024-03-27T21:41:10.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.334+0000 7f6c957fa640 10 monclient: handle_get_version_reply finishing 1 version 52 2024-03-27T21:41:10.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.334+0000 7f6c96ffd640 10 client.4893.objecter _check_command_map_dne tid 5 current 52 map_dne_bound 52 2024-03-27T21:41:10.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.334+0000 7f6c96ffd640 10 client.4893.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:41:10.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.334+0000 7f6c96ffd640 15 client.4893.objecter _session_command_op_remove -1 5 2024-03-27T21:41:10.323 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:41:10.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.334+0000 7f6c9efa4640 10 client.4893.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:10.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.334+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign -1 6 2024-03-27T21:41:10.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.334+0000 7f6c9efa4640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f6c9802b680 0x7f6c9802ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:10.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.334+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c98136eb0 osd=3 3 2024-03-27T21:41:10.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.334+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c98136eb0 osd=3 3 2024-03-27T21:41:10.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.334+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c98136eb0 osd=3 3 2024-03-27T21:41:10.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.334+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_remove -1 6 2024-03-27T21:41:10.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.334+0000 7f6c9efa4640 20 client.4893.objecter get_session s=0x7f6c98136eb0 osd=3 3 2024-03-27T21:41:10.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.334+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign 3 6 2024-03-27T21:41:10.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.334+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c98136eb0 osd=3 4 2024-03-27T21:41:10.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.334+0000 7f6c9efa4640 10 client.4893.objecter _send_command 6 2024-03-27T21:41:10.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.334+0000 7f6c9efa4640 1 -- 172.21.15.60:0/818722907 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c98134ed0 con 0x7f6c9802b680 2024-03-27T21:41:10.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.334+0000 7f6c97fff640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f6c9802b680 0x7f6c9802ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:10.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.334+0000 7f6c97fff640 10 monclient: get_auth_request con 0x7f6c9802b680 auth_method 0 2024-03-27T21:41:10.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.334+0000 7f6c957fa640 10 client.4893.objecter ms_handle_connect 0x7f6c9802b680 2024-03-27T21:41:10.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.334+0000 7f6c97fff640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f6c9802b680 0x7f6c9802ba80 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:10.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.335+0000 7f6c957fa640 1 -- 172.21.15.60:0/818722907 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6c98134ed0 con 0x7f6c9802b680 2024-03-27T21:41:10.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.335+0000 7f6c957fa640 10 client.4893.objecter ms_dispatch 0x7f6c98000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:41:10.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.335+0000 7f6c957fa640 10 client.4893.objecter _finish_command 6 = system:0 2024-03-27T21:41:10.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.335+0000 7f6c957fa640 20 client.4893.objecter put_session s=0x7f6c98136eb0 osd=3 4 2024-03-27T21:41:10.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.335+0000 7f6c957fa640 15 client.4893.objecter _session_command_op_remove 3 6 2024-03-27T21:41:10.325 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:10.340+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:10.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.348+0000 7f6c9efa4640 10 client.4893.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:10.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.348+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign -1 7 2024-03-27T21:41:10.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.348+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c98136eb0 osd=3 3 2024-03-27T21:41:10.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.348+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c98136eb0 osd=3 3 2024-03-27T21:41:10.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.348+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c98136eb0 osd=3 3 2024-03-27T21:41:10.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.349+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_remove -1 7 2024-03-27T21:41:10.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.349+0000 7f6c9efa4640 20 client.4893.objecter get_session s=0x7f6c98136eb0 osd=3 3 2024-03-27T21:41:10.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.349+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign 3 7 2024-03-27T21:41:10.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.349+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c98136eb0 osd=3 4 2024-03-27T21:41:10.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.349+0000 7f6c9efa4640 10 client.4893.objecter _send_command 7 2024-03-27T21:41:10.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.349+0000 7f6c9efa4640 1 -- 172.21.15.60:0/818722907 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6c981374a0 con 0x7f6c9802b680 2024-03-27T21:41:10.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.350+0000 7f6c957fa640 1 -- 172.21.15.60:0/818722907 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6c981374a0 con 0x7f6c9802b680 2024-03-27T21:41:10.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.350+0000 7f6c957fa640 10 client.4893.objecter ms_dispatch 0x7f6c98000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:10.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.350+0000 7f6c957fa640 10 client.4893.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:10.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.350+0000 7f6c957fa640 20 client.4893.objecter put_session s=0x7f6c98136eb0 osd=3 4 2024-03-27T21:41:10.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.350+0000 7f6c957fa640 15 client.4893.objecter _session_command_op_remove 3 7 2024-03-27T21:41:10.337 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:41:10.338 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:41:10.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.350+0000 7f6c9efa4640 10 client.4893.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:10.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.350+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign -1 8 2024-03-27T21:41:10.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.350+0000 7f6c9efa4640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f6c981e7060 0x7f6c981e9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:10.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.350+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c981343c0 osd=4 3 2024-03-27T21:41:10.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.350+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c981343c0 osd=4 3 2024-03-27T21:41:10.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.350+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c981343c0 osd=4 3 2024-03-27T21:41:10.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.350+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_remove -1 8 2024-03-27T21:41:10.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.350+0000 7f6c9efa4640 20 client.4893.objecter get_session s=0x7f6c981343c0 osd=4 3 2024-03-27T21:41:10.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.350+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign 4 8 2024-03-27T21:41:10.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.350+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c981343c0 osd=4 4 2024-03-27T21:41:10.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.350+0000 7f6c9efa4640 10 client.4893.objecter _send_command 8 2024-03-27T21:41:10.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.350+0000 7f6c9efa4640 1 -- 172.21.15.60:0/818722907 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c98134530 con 0x7f6c981e7060 2024-03-27T21:41:10.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.350+0000 7f6c9d541640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f6c981e7060 0x7f6c981e9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:10.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.350+0000 7f6c9d541640 10 monclient: get_auth_request con 0x7f6c981e7060 auth_method 0 2024-03-27T21:41:10.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.351+0000 7f6c9d541640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f6c981e7060 0x7f6c981e9450 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:10.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.351+0000 7f6c957fa640 10 client.4893.objecter ms_handle_connect 0x7f6c981e7060 2024-03-27T21:41:10.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.351+0000 7f6c957fa640 1 -- 172.21.15.60:0/818722907 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6c98134530 con 0x7f6c981e7060 2024-03-27T21:41:10.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.351+0000 7f6c957fa640 10 client.4893.objecter ms_dispatch 0x7f6c98000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:41:10.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.351+0000 7f6c957fa640 10 client.4893.objecter _finish_command 8 = system:0 2024-03-27T21:41:10.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.351+0000 7f6c957fa640 20 client.4893.objecter put_session s=0x7f6c981343c0 osd=4 4 2024-03-27T21:41:10.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.351+0000 7f6c957fa640 15 client.4893.objecter _session_command_op_remove 4 8 2024-03-27T21:41:10.340 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:10.340 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:10.347 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:41:10.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.365+0000 7f6c9efa4640 10 client.4893.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:10.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.365+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign -1 9 2024-03-27T21:41:10.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.365+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c981343c0 osd=4 3 2024-03-27T21:41:10.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.365+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c981343c0 osd=4 3 2024-03-27T21:41:10.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.365+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c981343c0 osd=4 3 2024-03-27T21:41:10.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.365+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_remove -1 9 2024-03-27T21:41:10.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.365+0000 7f6c9efa4640 20 client.4893.objecter get_session s=0x7f6c981343c0 osd=4 3 2024-03-27T21:41:10.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.365+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign 4 9 2024-03-27T21:41:10.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.365+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c981343c0 osd=4 4 2024-03-27T21:41:10.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.365+0000 7f6c9efa4640 10 client.4893.objecter _send_command 9 2024-03-27T21:41:10.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.365+0000 7f6c9efa4640 1 -- 172.21.15.60:0/818722907 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6c9802c200 con 0x7f6c981e7060 2024-03-27T21:41:10.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.367+0000 7f6c957fa640 1 -- 172.21.15.60:0/818722907 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6c9802c200 con 0x7f6c981e7060 2024-03-27T21:41:10.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.367+0000 7f6c957fa640 10 client.4893.objecter ms_dispatch 0x7f6c98000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:10.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.367+0000 7f6c957fa640 10 client.4893.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:10.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.367+0000 7f6c957fa640 20 client.4893.objecter put_session s=0x7f6c981343c0 osd=4 4 2024-03-27T21:41:10.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.367+0000 7f6c957fa640 15 client.4893.objecter _session_command_op_remove 4 9 2024-03-27T21:41:10.354 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:41:10.355 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:41:10.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.367+0000 7f6c9efa4640 10 client.4893.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:10.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.367+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign -1 10 2024-03-27T21:41:10.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.367+0000 7f6c9efa4640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f6c98066760 0x7f6c981ecb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:10.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.367+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c9802c440 osd=5 3 2024-03-27T21:41:10.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.367+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c9802c440 osd=5 3 2024-03-27T21:41:10.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.367+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c9802c440 osd=5 3 2024-03-27T21:41:10.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.367+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_remove -1 10 2024-03-27T21:41:10.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.367+0000 7f6c9efa4640 20 client.4893.objecter get_session s=0x7f6c9802c440 osd=5 3 2024-03-27T21:41:10.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.367+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign 5 10 2024-03-27T21:41:10.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.367+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c9802c440 osd=5 4 2024-03-27T21:41:10.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.367+0000 7f6c9efa4640 10 client.4893.objecter _send_command 10 2024-03-27T21:41:10.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.367+0000 7f6c9efa4640 1 -- 172.21.15.60:0/818722907 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c9802c5b0 con 0x7f6c98066760 2024-03-27T21:41:10.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.367+0000 7f6c9cd40640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f6c98066760 0x7f6c981ecb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:10.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.367+0000 7f6c9cd40640 10 monclient: get_auth_request con 0x7f6c98066760 auth_method 0 2024-03-27T21:41:10.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.367+0000 7f6c957fa640 10 client.4893.objecter ms_handle_connect 0x7f6c98066760 2024-03-27T21:41:10.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.367+0000 7f6c9cd40640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f6c98066760 0x7f6c981ecb20 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:10.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.368+0000 7f6c957fa640 1 -- 172.21.15.60:0/818722907 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6c9802c5b0 con 0x7f6c98066760 2024-03-27T21:41:10.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.368+0000 7f6c957fa640 10 client.4893.objecter ms_dispatch 0x7f6c98000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:41:10.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.368+0000 7f6c957fa640 10 client.4893.objecter _finish_command 10 = system:0 2024-03-27T21:41:10.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.368+0000 7f6c957fa640 20 client.4893.objecter put_session s=0x7f6c9802c440 osd=5 4 2024-03-27T21:41:10.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.368+0000 7f6c957fa640 15 client.4893.objecter _session_command_op_remove 5 10 2024-03-27T21:41:10.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.382+0000 7f6c9efa4640 10 client.4893.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:10.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.382+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign -1 11 2024-03-27T21:41:10.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.382+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c9802c440 osd=5 3 2024-03-27T21:41:10.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.382+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c9802c440 osd=5 3 2024-03-27T21:41:10.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.382+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c9802c440 osd=5 3 2024-03-27T21:41:10.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.382+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_remove -1 11 2024-03-27T21:41:10.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.382+0000 7f6c9efa4640 20 client.4893.objecter get_session s=0x7f6c9802c440 osd=5 3 2024-03-27T21:41:10.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.382+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign 5 11 2024-03-27T21:41:10.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.382+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c9802c440 osd=5 4 2024-03-27T21:41:10.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.382+0000 7f6c9efa4640 10 client.4893.objecter _send_command 11 2024-03-27T21:41:10.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.382+0000 7f6c9efa4640 1 -- 172.21.15.60:0/818722907 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6c9802c5b0 con 0x7f6c98066760 2024-03-27T21:41:10.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.383+0000 7f6c957fa640 1 -- 172.21.15.60:0/818722907 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6c9802c5b0 con 0x7f6c98066760 2024-03-27T21:41:10.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.383+0000 7f6c957fa640 10 client.4893.objecter ms_dispatch 0x7f6c98000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:10.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.383+0000 7f6c957fa640 10 client.4893.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:10.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.383+0000 7f6c957fa640 20 client.4893.objecter put_session s=0x7f6c9802c440 osd=5 4 2024-03-27T21:41:10.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.383+0000 7f6c957fa640 15 client.4893.objecter _session_command_op_remove 5 11 2024-03-27T21:41:10.370 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:41:10.371 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:41:10.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.384+0000 7f6c9efa4640 10 client.4893.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:10.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.384+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign -1 12 2024-03-27T21:41:10.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.384+0000 7f6c9efa4640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f6c981ed570 0x7f6c981ef9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:10.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.384+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c9802c5b0 osd=6 3 2024-03-27T21:41:10.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.384+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c9802c5b0 osd=6 3 2024-03-27T21:41:10.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.384+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c9802c5b0 osd=6 3 2024-03-27T21:41:10.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.384+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_remove -1 12 2024-03-27T21:41:10.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.384+0000 7f6c9efa4640 20 client.4893.objecter get_session s=0x7f6c9802c5b0 osd=6 3 2024-03-27T21:41:10.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.384+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign 6 12 2024-03-27T21:41:10.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.384+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c9802c5b0 osd=6 4 2024-03-27T21:41:10.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.384+0000 7f6c9efa4640 10 client.4893.objecter _send_command 12 2024-03-27T21:41:10.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.384+0000 7f6c9efa4640 1 -- 172.21.15.60:0/818722907 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c981f2980 con 0x7f6c981ed570 2024-03-27T21:41:10.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.384+0000 7f6c97fff640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f6c981ed570 0x7f6c981ef9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:10.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.384+0000 7f6c97fff640 10 monclient: get_auth_request con 0x7f6c981ed570 auth_method 0 2024-03-27T21:41:10.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.384+0000 7f6c957fa640 10 client.4893.objecter ms_handle_connect 0x7f6c981ed570 2024-03-27T21:41:10.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.384+0000 7f6c97fff640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f6c981ed570 0x7f6c981ef9c0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:10.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.385+0000 7f6c957fa640 1 -- 172.21.15.60:0/818722907 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6c981f2980 con 0x7f6c981ed570 2024-03-27T21:41:10.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.385+0000 7f6c957fa640 10 client.4893.objecter ms_dispatch 0x7f6c98000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:41:10.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.385+0000 7f6c957fa640 10 client.4893.objecter _finish_command 12 = system:0 2024-03-27T21:41:10.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.385+0000 7f6c957fa640 20 client.4893.objecter put_session s=0x7f6c9802c5b0 osd=6 4 2024-03-27T21:41:10.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.385+0000 7f6c957fa640 15 client.4893.objecter _session_command_op_remove 6 12 2024-03-27T21:41:10.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.398+0000 7f6c9efa4640 10 client.4893.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:10.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.398+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign -1 13 2024-03-27T21:41:10.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.398+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c9802c5b0 osd=6 3 2024-03-27T21:41:10.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.398+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c9802c5b0 osd=6 3 2024-03-27T21:41:10.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.398+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c9802c5b0 osd=6 3 2024-03-27T21:41:10.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.398+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_remove -1 13 2024-03-27T21:41:10.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.398+0000 7f6c9efa4640 20 client.4893.objecter get_session s=0x7f6c9802c5b0 osd=6 3 2024-03-27T21:41:10.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.398+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign 6 13 2024-03-27T21:41:10.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.398+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c9802c5b0 osd=6 4 2024-03-27T21:41:10.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.398+0000 7f6c9efa4640 10 client.4893.objecter _send_command 13 2024-03-27T21:41:10.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.398+0000 7f6c9efa4640 1 -- 172.21.15.60:0/818722907 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6c981f2980 con 0x7f6c981ed570 2024-03-27T21:41:10.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.400+0000 7f6c957fa640 1 -- 172.21.15.60:0/818722907 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6c981f2980 con 0x7f6c981ed570 2024-03-27T21:41:10.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.400+0000 7f6c957fa640 10 client.4893.objecter ms_dispatch 0x7f6c98000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:10.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.400+0000 7f6c957fa640 10 client.4893.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:10.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.400+0000 7f6c957fa640 20 client.4893.objecter put_session s=0x7f6c9802c5b0 osd=6 4 2024-03-27T21:41:10.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.400+0000 7f6c957fa640 15 client.4893.objecter _session_command_op_remove 6 13 2024-03-27T21:41:10.387 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:41:10.388 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:41:10.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.400+0000 7f6c9efa4640 10 client.4893.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:10.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.400+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign -1 14 2024-03-27T21:41:10.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.400+0000 7f6c9efa4640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f6c981f1cb0 0x7f6c981f2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:10.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.400+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c981f2980 osd=7 3 2024-03-27T21:41:10.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.400+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c981f2980 osd=7 3 2024-03-27T21:41:10.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.400+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c981f2980 osd=7 3 2024-03-27T21:41:10.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.400+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_remove -1 14 2024-03-27T21:41:10.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.400+0000 7f6c9efa4640 20 client.4893.objecter get_session s=0x7f6c981f2980 osd=7 3 2024-03-27T21:41:10.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.400+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign 7 14 2024-03-27T21:41:10.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.400+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c981f2980 osd=7 4 2024-03-27T21:41:10.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.400+0000 7f6c9efa4640 10 client.4893.objecter _send_command 14 2024-03-27T21:41:10.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.400+0000 7f6c9efa4640 1 -- 172.21.15.60:0/818722907 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c981f5cb0 con 0x7f6c981f1cb0 2024-03-27T21:41:10.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.400+0000 7f6c9d541640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f6c981f1cb0 0x7f6c981f2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:10.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.400+0000 7f6c9d541640 10 monclient: get_auth_request con 0x7f6c981f1cb0 auth_method 0 2024-03-27T21:41:10.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.401+0000 7f6c9d541640 1 --2- 172.21.15.60:0/818722907 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f6c981f1cb0 0x7f6c981f2130 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:10.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.401+0000 7f6c957fa640 10 client.4893.objecter ms_handle_connect 0x7f6c981f1cb0 2024-03-27T21:41:10.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.401+0000 7f6c957fa640 1 -- 172.21.15.60:0/818722907 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6c981f5cb0 con 0x7f6c981f1cb0 2024-03-27T21:41:10.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.401+0000 7f6c957fa640 10 client.4893.objecter ms_dispatch 0x7f6c98000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:41:10.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.401+0000 7f6c957fa640 10 client.4893.objecter _finish_command 14 = system:0 2024-03-27T21:41:10.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.401+0000 7f6c957fa640 20 client.4893.objecter put_session s=0x7f6c981f2980 osd=7 4 2024-03-27T21:41:10.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.401+0000 7f6c957fa640 15 client.4893.objecter _session_command_op_remove 7 14 2024-03-27T21:41:10.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.415+0000 7f6c9efa4640 10 client.4893.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:10.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.415+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign -1 15 2024-03-27T21:41:10.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.415+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c981f2980 osd=7 3 2024-03-27T21:41:10.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.415+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c981f2980 osd=7 3 2024-03-27T21:41:10.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.415+0000 7f6c9efa4640 20 client.4893.objecter _get_session s=0x7f6c981f2980 osd=7 3 2024-03-27T21:41:10.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.415+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_remove -1 15 2024-03-27T21:41:10.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.415+0000 7f6c9efa4640 20 client.4893.objecter get_session s=0x7f6c981f2980 osd=7 3 2024-03-27T21:41:10.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.415+0000 7f6c9efa4640 15 client.4893.objecter _session_command_op_assign 7 15 2024-03-27T21:41:10.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.415+0000 7f6c9efa4640 20 client.4893.objecter put_session s=0x7f6c981f2980 osd=7 4 2024-03-27T21:41:10.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.415+0000 7f6c9efa4640 10 client.4893.objecter _send_command 15 2024-03-27T21:41:10.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.415+0000 7f6c9efa4640 1 -- 172.21.15.60:0/818722907 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6c981f0010 con 0x7f6c981f1cb0 2024-03-27T21:41:10.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.417+0000 7f6c957fa640 1 -- 172.21.15.60:0/818722907 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6c981f0010 con 0x7f6c981f1cb0 2024-03-27T21:41:10.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.417+0000 7f6c957fa640 10 client.4893.objecter ms_dispatch 0x7f6c98000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:10.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.417+0000 7f6c957fa640 10 client.4893.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:10.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.417+0000 7f6c957fa640 20 client.4893.objecter put_session s=0x7f6c981f2980 osd=7 4 2024-03-27T21:41:10.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.417+0000 7f6c957fa640 15 client.4893.objecter _session_command_op_remove 7 15 2024-03-27T21:41:10.404 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:41:10.404 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:41:10.406 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:41:10.406 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:41:10.433 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:10.448+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:10.496 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:10.496 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:10.496 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:10.496 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:10.505 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:41:10.547 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:10.561+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:10.628 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:10.642+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:10.650 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:10.650 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:10.650 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:10.650 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:10.650 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:10.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:41:10.744 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:10.759+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:10.807 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:10.807 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:10.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:10.808 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-03-27T21:41:10.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.808 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:10.808 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:10.808 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 91.387450829000002, 2024-03-27T21:41:10.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43537978700000002, 2024-03-27T21:41:10.809 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:10.809 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:10.809 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:10.809 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:10.809 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:10.809 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-03-27T21:41:10.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.810 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:10.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.810 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:10.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:10.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:10.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:10.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:10.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432019+0000", 2024-03-27T21:41:10.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-03-27T21:41:10.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:10.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432027+0000", 2024-03-27T21:41:10.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1200000000000002e-06 2024-03-27T21:41:10.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.813 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:10.813 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432039+0000", 2024-03-27T21:41:10.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1123e-05 2024-03-27T21:41:10.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:10.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432051+0000", 2024-03-27T21:41:10.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-03-27T21:41:10.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:10.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432157+0000", 2024-03-27T21:41:10.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106846 2024-03-27T21:41:10.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-03-27T21:41:10.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432176+0000", 2024-03-27T21:41:10.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8490000000000001e-05 2024-03-27T21:41:10.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:10.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866170+0000", 2024-03-27T21:41:10.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43399370100000001 2024-03-27T21:41:10.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:10.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866335+0000", 2024-03-27T21:41:10.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016559699999999999 2024-03-27T21:41:10.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:10.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866495+0000", 2024-03-27T21:41:10.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015944200000000001 2024-03-27T21:41:10.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.818 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:10.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867012+0000", 2024-03-27T21:41:10.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0005176 2024-03-27T21:41:10.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.818 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:10.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867334+0000", 2024-03-27T21:41:10.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032158600000000001 2024-03-27T21:41:10.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:10.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867374+0000", 2024-03-27T21:41:10.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0547000000000003e-05 2024-03-27T21:41:10.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:10.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867400+0000", 2024-03-27T21:41:10.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5958000000000001e-05 2024-03-27T21:41:10.819 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.819 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:10.819 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.819 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:10.819 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:10.820 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 90.890068377999995, 2024-03-27T21:41:10.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064560310000000001, 2024-03-27T21:41:10.820 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:10.820 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:10.820 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:10.820 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:10.820 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:10.820 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 70 2024-03-27T21:41:10.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.820 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:10.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:10.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:10.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:10.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:10.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:10.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929402+0000", 2024-03-27T21:41:10.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:10.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:10.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929407+0000", 2024-03-27T21:41:10.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-06 2024-03-27T21:41:10.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.822 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:10.822 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929414+0000", 2024-03-27T21:41:10.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5689999999999999e-06 2024-03-27T21:41:10.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.822 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:10.822 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929419+0000", 2024-03-27T21:41:10.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-03-27T21:41:10.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:10.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930245+0000", 2024-03-27T21:41:10.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082521500000000002 2024-03-27T21:41:10.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:10.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930287+0000", 2024-03-27T21:41:10.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2305000000000002e-05 2024-03-27T21:41:10.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:10.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930367+0000", 2024-03-27T21:41:10.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9589e-05 2024-03-27T21:41:10.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:10.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.931214+0000", 2024-03-27T21:41:10.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000847827 2024-03-27T21:41:10.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:10.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935807+0000", 2024-03-27T21:41:10.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045923780000000003 2024-03-27T21:41:10.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:10.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935840+0000", 2024-03-27T21:41:10.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3501999999999999e-05 2024-03-27T21:41:10.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:10.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935859+0000", 2024-03-27T21:41:10.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8845e-05 2024-03-27T21:41:10.825 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.825 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:10.825 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.825 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:10.825 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:10.825 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 90.882808787000002, 2024-03-27T21:41:10.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071695429999999996, 2024-03-27T21:41:10.825 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:10.825 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:10.826 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:10.826 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:10.826 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:10.826 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-03-27T21:41:10.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.826 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:10.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:10.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:10.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:10.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:10.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:10.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936662+0000", 2024-03-27T21:41:10.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:10.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:10.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936737+0000", 2024-03-27T21:41:10.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5303999999999996e-05 2024-03-27T21:41:10.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.828 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.828 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:10.828 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936746+0000", 2024-03-27T21:41:10.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6340000000000007e-06 2024-03-27T21:41:10.828 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.828 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.828 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:10.828 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936753+0000", 2024-03-27T21:41:10.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-03-27T21:41:10.828 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.828 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.828 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:10.828 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937926+0000", 2024-03-27T21:41:10.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011731280000000001 2024-03-27T21:41:10.828 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:10.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937969+0000", 2024-03-27T21:41:10.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3285999999999999e-05 2024-03-27T21:41:10.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:10.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938040+0000", 2024-03-27T21:41:10.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0358000000000006e-05 2024-03-27T21:41:10.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:10.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939324+0000", 2024-03-27T21:41:10.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012844899999999999 2024-03-27T21:41:10.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:10.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943780+0000", 2024-03-27T21:41:10.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044555630000000001 2024-03-27T21:41:10.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:10.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943811+0000", 2024-03-27T21:41:10.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1415999999999999e-05 2024-03-27T21:41:10.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:10.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943832+0000", 2024-03-27T21:41:10.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0981000000000001e-05 2024-03-27T21:41:10.831 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.831 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:10.831 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.831 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:10.831 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:10.831 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 90.882710771000006, 2024-03-27T21:41:10.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071139740000000003, 2024-03-27T21:41:10.832 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:10.832 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:10.832 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:10.832 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:10.832 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:10.832 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-03-27T21:41:10.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.833 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:10.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:10.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:10.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:10.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:10.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.834 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:10.834 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936760+0000", 2024-03-27T21:41:10.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:10.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.834 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:10.834 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936765+0000", 2024-03-27T21:41:10.835 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6330000000000004e-06 2024-03-27T21:41:10.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.835 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.835 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:10.835 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936771+0000", 2024-03-27T21:41:10.835 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0560000000000001e-06 2024-03-27T21:41:10.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.835 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.835 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:10.835 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936776+0000", 2024-03-27T21:41:10.835 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1510000000000003e-06 2024-03-27T21:41:10.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.836 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:10.836 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938605+0000", 2024-03-27T21:41:10.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001829589 2024-03-27T21:41:10.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.836 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:10.836 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938657+0000", 2024-03-27T21:41:10.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1860000000000002e-05 2024-03-27T21:41:10.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.836 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:10.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938734+0000", 2024-03-27T21:41:10.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6639999999999998e-05 2024-03-27T21:41:10.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:10.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939562+0000", 2024-03-27T21:41:10.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082803100000000001 2024-03-27T21:41:10.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:10.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943838+0000", 2024-03-27T21:41:10.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042757050000000003 2024-03-27T21:41:10.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:10.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943859+0000", 2024-03-27T21:41:10.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1175e-05 2024-03-27T21:41:10.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:10.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943875+0000", 2024-03-27T21:41:10.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5744e-05 2024-03-27T21:41:10.838 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.838 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:10.838 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.839 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:10.839 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:10.839 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 90.739196719999995, 2024-03-27T21:41:10.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059712230000000003, 2024-03-27T21:41:10.839 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:10.839 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:10.839 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:10.839 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:10.839 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:10.839 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 112 2024-03-27T21:41:10.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.840 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:10.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:10.840 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:10.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:10.840 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:10.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:10.841 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080274+0000", 2024-03-27T21:41:10.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:10.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.841 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.841 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:10.841 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080318+0000", 2024-03-27T21:41:10.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3634999999999998e-05 2024-03-27T21:41:10.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.841 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.841 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:10.841 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080329+0000", 2024-03-27T21:41:10.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1232e-05 2024-03-27T21:41:10.842 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.842 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.842 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:10.842 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080333+0000", 2024-03-27T21:41:10.842 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6009999999999999e-06 2024-03-27T21:41:10.842 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.842 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.842 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:10.842 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083786+0000", 2024-03-27T21:41:10.842 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034528940000000002 2024-03-27T21:41:10.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.843 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.843 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:10.843 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083861+0000", 2024-03-27T21:41:10.843 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4893000000000003e-05 2024-03-27T21:41:10.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.843 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.843 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:10.843 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083981+0000", 2024-03-27T21:41:10.843 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012058299999999999 2024-03-27T21:41:10.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.844 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.844 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:10.844 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086148+0000", 2024-03-27T21:41:10.844 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021671469999999999 2024-03-27T21:41:10.844 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.844 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.844 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:10.844 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086195+0000", 2024-03-27T21:41:10.844 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6768999999999999e-05 2024-03-27T21:41:10.844 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.844 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.844 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:10.844 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086224+0000", 2024-03-27T21:41:10.844 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8758000000000001e-05 2024-03-27T21:41:10.845 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.845 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.845 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:10.845 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086246+0000", 2024-03-27T21:41:10.845 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2064000000000001e-05 2024-03-27T21:41:10.845 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.845 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:10.845 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.845 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.845 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.845 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:10.845 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:10.845 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 90.739133773000006, 2024-03-27T21:41:10.845 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066196320000000003, 2024-03-27T21:41:10.845 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:10.846 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:10.846 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:10.846 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:10.846 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:10.846 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113 2024-03-27T21:41:10.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.846 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:10.846 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.846 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:10.846 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:10.846 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.846 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.846 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:10.847 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:10.847 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.847 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.847 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:10.847 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080337+0000", 2024-03-27T21:41:10.847 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:10.847 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.847 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:10.847 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080340+0000", 2024-03-27T21:41:10.847 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6690000000000002e-06 2024-03-27T21:41:10.847 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.847 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:10.847 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080344+0000", 2024-03-27T21:41:10.848 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6550000000000002e-06 2024-03-27T21:41:10.848 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.848 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.848 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:10.848 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080348+0000", 2024-03-27T21:41:10.848 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9169999999999999e-06 2024-03-27T21:41:10.848 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.848 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.848 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:10.848 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084595+0000", 2024-03-27T21:41:10.848 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042471719999999996 2024-03-27T21:41:10.848 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.848 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.848 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:10.848 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084676+0000", 2024-03-27T21:41:10.849 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1147000000000001e-05 2024-03-27T21:41:10.849 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.849 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.849 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:10.849 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084796+0000", 2024-03-27T21:41:10.849 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012032900000000001 2024-03-27T21:41:10.849 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.849 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.849 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:10.849 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086190+0000", 2024-03-27T21:41:10.849 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013934399999999999 2024-03-27T21:41:10.849 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.849 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.849 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:10.849 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086905+0000", 2024-03-27T21:41:10.850 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071536199999999999 2024-03-27T21:41:10.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.850 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.850 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:10.850 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086937+0000", 2024-03-27T21:41:10.850 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2125000000000002e-05 2024-03-27T21:41:10.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.850 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.850 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:10.850 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086957+0000", 2024-03-27T21:41:10.850 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0157999999999999e-05 2024-03-27T21:41:10.850 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.850 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:10.850 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.851 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.851 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:10.851 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:10.851 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 90.739121057000006, 2024-03-27T21:41:10.851 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007405661, 2024-03-27T21:41:10.851 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:10.851 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:10.851 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:10.851 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:10.851 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:10.851 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 114 2024-03-27T21:41:10.852 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.852 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:10.852 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.852 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:10.852 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:10.852 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.852 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.852 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.852 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:10.852 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:10.853 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.853 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.853 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.853 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:10.853 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:10.853 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-03-27T21:41:10.853 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.853 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.854 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:10.854 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080353+0000", 2024-03-27T21:41:10.854 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.441e-06 2024-03-27T21:41:10.854 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.854 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.854 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:10.854 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080356+0000", 2024-03-27T21:41:10.854 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0769999999999999e-06 2024-03-27T21:41:10.854 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.854 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.854 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:10.854 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080358+0000", 2024-03-27T21:41:10.854 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3369999999999998e-06 2024-03-27T21:41:10.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.855 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:10.855 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085118+0000", 2024-03-27T21:41:10.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047603949999999997 2024-03-27T21:41:10.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.855 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:10.855 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085210+0000", 2024-03-27T21:41:10.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1983000000000003e-05 2024-03-27T21:41:10.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.855 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:10.855 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085332+0000", 2024-03-27T21:41:10.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121803 2024-03-27T21:41:10.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:10.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086292+0000", 2024-03-27T21:41:10.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095961400000000002 2024-03-27T21:41:10.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:10.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087710+0000", 2024-03-27T21:41:10.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014181459999999999 2024-03-27T21:41:10.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:10.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087736+0000", 2024-03-27T21:41:10.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6475e-05 2024-03-27T21:41:10.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:10.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087756+0000", 2024-03-27T21:41:10.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9626000000000001e-05 2024-03-27T21:41:10.857 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.857 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:10.857 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.857 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:10.858 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:10.858 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 90.739098622, 2024-03-27T21:41:10.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075011720000000004, 2024-03-27T21:41:10.858 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:10.858 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:10.858 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:10.858 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:10.858 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:10.858 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115 2024-03-27T21:41:10.858 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.858 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:10.858 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.859 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:10.859 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:10.859 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.859 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.859 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.859 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:10.859 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:10.859 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.859 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.859 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.859 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:10.859 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:10.859 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:10.859 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.859 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.860 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:10.860 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080458+0000", 2024-03-27T21:41:10.860 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5228000000000005e-05 2024-03-27T21:41:10.860 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.860 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.860 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:10.860 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080465+0000", 2024-03-27T21:41:10.860 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.238e-06 2024-03-27T21:41:10.860 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.860 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.860 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:10.860 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080468+0000", 2024-03-27T21:41:10.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5109999999999999e-06 2024-03-27T21:41:10.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.861 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.861 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:10.861 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086326+0000", 2024-03-27T21:41:10.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058579169999999998 2024-03-27T21:41:10.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.861 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.861 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:10.861 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086382+0000", 2024-03-27T21:41:10.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5309999999999997e-05 2024-03-27T21:41:10.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.861 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.861 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:10.861 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086457+0000", 2024-03-27T21:41:10.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5413000000000005e-05 2024-03-27T21:41:10.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:10.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087762+0000", 2024-03-27T21:41:10.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013052300000000001 2024-03-27T21:41:10.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:10.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087798+0000", 2024-03-27T21:41:10.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5614999999999997e-05 2024-03-27T21:41:10.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:10.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087840+0000", 2024-03-27T21:41:10.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2196000000000001e-05 2024-03-27T21:41:10.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.863 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:10.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087874+0000", 2024-03-27T21:41:10.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3785999999999998e-05 2024-03-27T21:41:10.863 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.863 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:10.863 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.863 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:10.864 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:10.864 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 90.738998538999994, 2024-03-27T21:41:10.864 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075050680000000002, 2024-03-27T21:41:10.864 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:10.864 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:10.864 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:10.864 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:10.864 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:10.864 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 116 2024-03-27T21:41:10.864 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.864 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:10.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:10.865 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:10.865 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.865 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:10.865 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:10.865 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.865 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:10.865 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080472+0000", 2024-03-27T21:41:10.865 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:10.865 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.866 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:10.866 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080475+0000", 2024-03-27T21:41:10.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.712e-06 2024-03-27T21:41:10.866 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.866 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.866 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:10.866 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080478+0000", 2024-03-27T21:41:10.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1070000000000001e-06 2024-03-27T21:41:10.866 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.866 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.866 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:10.866 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080481+0000", 2024-03-27T21:41:10.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4269999999999998e-06 2024-03-27T21:41:10.867 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.867 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.867 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:10.867 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086992+0000", 2024-03-27T21:41:10.867 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065114099999999996 2024-03-27T21:41:10.867 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.867 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.867 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:10.867 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087064+0000", 2024-03-27T21:41:10.867 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1927000000000004e-05 2024-03-27T21:41:10.867 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.867 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.867 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:10.867 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087188+0000", 2024-03-27T21:41:10.867 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123585 2024-03-27T21:41:10.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:10.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087766+0000", 2024-03-27T21:41:10.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057814300000000004 2024-03-27T21:41:10.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:10.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087934+0000", 2024-03-27T21:41:10.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000167894 2024-03-27T21:41:10.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:10.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087958+0000", 2024-03-27T21:41:10.870 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4663000000000001e-05 2024-03-27T21:41:10.870 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:10.870 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087978+0000", 2024-03-27T21:41:10.870 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9636e-05 2024-03-27T21:41:10.870 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.870 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:10.870 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.870 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.871 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:10.871 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:10.871 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 90.738966417, 2024-03-27T21:41:10.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076850679999999998, 2024-03-27T21:41:10.871 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:10.871 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:10.871 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:10.871 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:10.871 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:10.871 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 117 2024-03-27T21:41:10.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.871 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:10.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:10.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:10.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.872 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:10.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:10.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.872 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:10.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:10.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:10.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.873 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:10.873 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080508+0000", 2024-03-27T21:41:10.873 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.298e-06 2024-03-27T21:41:10.873 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.873 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.873 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:10.873 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080515+0000", 2024-03-27T21:41:10.873 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0790000000000003e-06 2024-03-27T21:41:10.873 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.873 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.873 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:10.874 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080518+0000", 2024-03-27T21:41:10.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8509999999999999e-06 2024-03-27T21:41:10.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.874 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:10.874 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087444+0000", 2024-03-27T21:41:10.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0069259120000000002 2024-03-27T21:41:10.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.874 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:10.874 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087491+0000", 2024-03-27T21:41:10.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7043999999999999e-05 2024-03-27T21:41:10.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:10.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087564+0000", 2024-03-27T21:41:10.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3603999999999995e-05 2024-03-27T21:41:10.875 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:10.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087902+0000", 2024-03-27T21:41:10.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033734100000000001 2024-03-27T21:41:10.875 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:10.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088117+0000", 2024-03-27T21:41:10.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000215164 2024-03-27T21:41:10.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.876 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:10.876 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088160+0000", 2024-03-27T21:41:10.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2933999999999999e-05 2024-03-27T21:41:10.877 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.877 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.877 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:10.877 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088190+0000", 2024-03-27T21:41:10.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0139999999999999e-05 2024-03-27T21:41:10.877 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.878 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:10.878 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.878 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.878 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.878 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:10.878 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:10.878 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 90.322686028000007, 2024-03-27T21:41:10.878 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052465439999999997, 2024-03-27T21:41:10.878 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:10.879 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:10.879 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:10.879 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:10.879 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:10.879 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 133 2024-03-27T21:41:10.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.879 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:10.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:10.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:10.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:10.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:10.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:10.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:10.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:10.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:10.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496788+0000", 2024-03-27T21:41:10.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-03-27T21:41:10.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:10.881 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496793+0000", 2024-03-27T21:41:10.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4790000000000004e-06 2024-03-27T21:41:10.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:10.881 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496796+0000", 2024-03-27T21:41:10.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1e-06 2024-03-27T21:41:10.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:10.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500447+0000", 2024-03-27T21:41:10.882 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036504459999999999 2024-03-27T21:41:10.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:10.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500492+0000", 2024-03-27T21:41:10.882 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5765e-05 2024-03-27T21:41:10.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:10.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500562+0000", 2024-03-27T21:41:10.882 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9206000000000001e-05 2024-03-27T21:41:10.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:10.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501959+0000", 2024-03-27T21:41:10.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001397187 2024-03-27T21:41:10.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:10.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501984+0000", 2024-03-27T21:41:10.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.552e-05 2024-03-27T21:41:10.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:10.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502010+0000", 2024-03-27T21:41:10.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5232000000000002e-05 2024-03-27T21:41:10.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:10.884 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502032+0000", 2024-03-27T21:41:10.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2337999999999999e-05 2024-03-27T21:41:10.884 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.884 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:10.884 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.884 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.884 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.884 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:10.884 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:10.884 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 90.322627370000006, 2024-03-27T21:41:10.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052892119999999997, 2024-03-27T21:41:10.884 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:10.884 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:10.884 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:10.884 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:10.885 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:10.885 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 134 2024-03-27T21:41:10.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.885 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:10.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:10.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:10.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:10.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:10.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.886 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:10.886 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:10.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:10.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.886 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:10.886 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496929+0000", 2024-03-27T21:41:10.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5098000000000004e-05 2024-03-27T21:41:10.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.886 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:10.886 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496937+0000", 2024-03-27T21:41:10.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9559999999999997e-06 2024-03-27T21:41:10.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.887 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.887 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:10.887 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496942+0000", 2024-03-27T21:41:10.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6350000000000003e-06 2024-03-27T21:41:10.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.887 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.887 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:10.887 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500749+0000", 2024-03-27T21:41:10.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038068529999999998 2024-03-27T21:41:10.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.887 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.887 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:10.887 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500792+0000", 2024-03-27T21:41:10.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2823e-05 2024-03-27T21:41:10.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:10.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500864+0000", 2024-03-27T21:41:10.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1715e-05 2024-03-27T21:41:10.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:10.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502068+0000", 2024-03-27T21:41:10.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012044860000000001 2024-03-27T21:41:10.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:10.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502073+0000", 2024-03-27T21:41:10.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5700000000000003e-06 2024-03-27T21:41:10.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:10.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502098+0000", 2024-03-27T21:41:10.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5120999999999999e-05 2024-03-27T21:41:10.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:10.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502133+0000", 2024-03-27T21:41:10.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5306999999999999e-05 2024-03-27T21:41:10.889 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.889 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:10.889 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.890 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:10.890 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:10.890 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 90.322522062999994, 2024-03-27T21:41:10.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062529040000000001, 2024-03-27T21:41:10.890 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:10.890 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:10.890 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:10.890 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:10.890 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:10.890 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135 2024-03-27T21:41:10.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.890 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:10.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:10.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:10.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:10.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:10.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:10.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:10.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:10.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:10.892 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496953+0000", 2024-03-27T21:41:10.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0709999999999996e-06 2024-03-27T21:41:10.892 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:10.892 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496961+0000", 2024-03-27T21:41:10.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1300000000000001e-06 2024-03-27T21:41:10.892 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:10.892 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496968+0000", 2024-03-27T21:41:10.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0999999999999998e-06 2024-03-27T21:41:10.892 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:10.893 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501244+0000", 2024-03-27T21:41:10.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042756310000000002 2024-03-27T21:41:10.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:10.893 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501377+0000", 2024-03-27T21:41:10.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013348799999999999 2024-03-27T21:41:10.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:10.893 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501455+0000", 2024-03-27T21:41:10.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7410000000000006e-05 2024-03-27T21:41:10.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:10.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502138+0000", 2024-03-27T21:41:10.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068386599999999999 2024-03-27T21:41:10.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:10.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503152+0000", 2024-03-27T21:41:10.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010139680000000001 2024-03-27T21:41:10.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:10.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503180+0000", 2024-03-27T21:41:10.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7426000000000001e-05 2024-03-27T21:41:10.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:10.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503202+0000", 2024-03-27T21:41:10.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2463000000000002e-05 2024-03-27T21:41:10.895 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.895 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:10.895 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.895 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:10.895 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:10.895 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 90.322497924999993, 2024-03-27T21:41:10.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066253680000000004, 2024-03-27T21:41:10.896 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:10.896 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:10.896 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:10.896 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:10.896 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:10.896 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 136 2024-03-27T21:41:10.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.896 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:10.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:10.896 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:10.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:10.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:10.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:10.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:10.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:10.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:10.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496977+0000", 2024-03-27T21:41:10.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5049999999999999e-06 2024-03-27T21:41:10.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.898 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:10.898 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496984+0000", 2024-03-27T21:41:10.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1520000000000003e-06 2024-03-27T21:41:10.898 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.898 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.898 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:10.898 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496990+0000", 2024-03-27T21:41:10.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-03-27T21:41:10.898 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.898 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.898 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:10.898 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501669+0000", 2024-03-27T21:41:10.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046793859999999998 2024-03-27T21:41:10.898 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:10.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501714+0000", 2024-03-27T21:41:10.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5154999999999997e-05 2024-03-27T21:41:10.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:10.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501783+0000", 2024-03-27T21:41:10.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8454999999999994e-05 2024-03-27T21:41:10.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:10.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503226+0000", 2024-03-27T21:41:10.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001443441 2024-03-27T21:41:10.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:10.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503540+0000", 2024-03-27T21:41:10.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031347100000000001 2024-03-27T21:41:10.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:10.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503563+0000", 2024-03-27T21:41:10.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3223000000000001e-05 2024-03-27T21:41:10.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:10.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503599+0000", 2024-03-27T21:41:10.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6105000000000001e-05 2024-03-27T21:41:10.901 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.901 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:10.901 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.901 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:10.901 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:10.901 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 90.322475115000003, 2024-03-27T21:41:10.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074261320000000002, 2024-03-27T21:41:10.901 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:10.901 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:10.902 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:10.902 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:10.902 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:10.902 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 137 2024-03-27T21:41:10.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.902 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:10.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:10.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:10.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:10.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:10.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.903 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:10.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:10.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:10.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.903 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:10.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497077+0000", 2024-03-27T21:41:10.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1599000000000003e-05 2024-03-27T21:41:10.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.903 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:10.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497085+0000", 2024-03-27T21:41:10.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6109999999999996e-06 2024-03-27T21:41:10.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:10.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497092+0000", 2024-03-27T21:41:10.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.742e-06 2024-03-27T21:41:10.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:10.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502159+0000", 2024-03-27T21:41:10.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050666870000000003 2024-03-27T21:41:10.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:10.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502200+0000", 2024-03-27T21:41:10.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1919999999999998e-05 2024-03-27T21:41:10.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.905 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:10.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502280+0000", 2024-03-27T21:41:10.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9417000000000002e-05 2024-03-27T21:41:10.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.905 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:10.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503605+0000", 2024-03-27T21:41:10.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001325641 2024-03-27T21:41:10.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.905 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:10.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504361+0000", 2024-03-27T21:41:10.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075555399999999997 2024-03-27T21:41:10.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:10.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504386+0000", 2024-03-27T21:41:10.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5369999999999999e-05 2024-03-27T21:41:10.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:10.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504422+0000", 2024-03-27T21:41:10.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6019000000000002e-05 2024-03-27T21:41:10.907 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.907 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:10.907 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.907 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:10.907 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:10.907 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 90.322371317999995, 2024-03-27T21:41:10.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074059750000000004, 2024-03-27T21:41:10.907 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:10.908 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:10.908 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:10.908 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:10.908 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:10.908 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-03-27T21:41:10.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.908 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:10.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:10.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:10.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:10.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:10.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:10.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:10.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:10.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:10.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497103+0000", 2024-03-27T21:41:10.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.856e-06 2024-03-27T21:41:10.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:10.910 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497121+0000", 2024-03-27T21:41:10.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7190999999999999e-05 2024-03-27T21:41:10.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:10.910 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497126+0000", 2024-03-27T21:41:10.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3770000000000004e-06 2024-03-27T21:41:10.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:10.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502833+0000", 2024-03-27T21:41:10.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057073510000000003 2024-03-27T21:41:10.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:10.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502881+0000", 2024-03-27T21:41:10.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7961999999999999e-05 2024-03-27T21:41:10.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:10.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502949+0000", 2024-03-27T21:41:10.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7957e-05 2024-03-27T21:41:10.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:10.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504430+0000", 2024-03-27T21:41:10.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001480763 2024-03-27T21:41:10.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:10.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504459+0000", 2024-03-27T21:41:10.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8983999999999999e-05 2024-03-27T21:41:10.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:10.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504486+0000", 2024-03-27T21:41:10.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6883999999999999e-05 2024-03-27T21:41:10.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:10.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504506+0000", 2024-03-27T21:41:10.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0214e-05 2024-03-27T21:41:10.913 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.913 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:10.913 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.913 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:10.913 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:10.913 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 90.322339483999997, 2024-03-27T21:41:10.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074754859999999999, 2024-03-27T21:41:10.914 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:10.914 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:10.914 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:10.914 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:10.914 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:10.914 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-03-27T21:41:10.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.914 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:10.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.914 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:10.914 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:10.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:10.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:10.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:10.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:10.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:10.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:10.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497137+0000", 2024-03-27T21:41:10.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0730000000000004e-06 2024-03-27T21:41:10.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:10.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497143+0000", 2024-03-27T21:41:10.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3759999999999996e-06 2024-03-27T21:41:10.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:10.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497150+0000", 2024-03-27T21:41:10.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.002e-06 2024-03-27T21:41:10.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:10.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503249+0000", 2024-03-27T21:41:10.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060989750000000004 2024-03-27T21:41:10.917 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.917 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:10.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503291+0000", 2024-03-27T21:41:10.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1563999999999997e-05 2024-03-27T21:41:10.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.918 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:10.918 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503376+0000", 2024-03-27T21:41:10.918 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5752000000000002e-05 2024-03-27T21:41:10.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.918 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:10.918 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504545+0000", 2024-03-27T21:41:10.918 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169172 2024-03-27T21:41:10.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.919 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:10.919 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504550+0000", 2024-03-27T21:41:10.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7419999999999997e-06 2024-03-27T21:41:10.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.919 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:10.919 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504574+0000", 2024-03-27T21:41:10.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4124000000000001e-05 2024-03-27T21:41:10.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.919 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:10.919 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504607+0000", 2024-03-27T21:41:10.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3118999999999999e-05 2024-03-27T21:41:10.919 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.919 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:10.920 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.920 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:10.920 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:10.920 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 90.322315818000007, 2024-03-27T21:41:10.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078150589999999992, 2024-03-27T21:41:10.920 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:10.920 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:10.920 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:10.920 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:10.920 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:10.920 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-03-27T21:41:10.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.921 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:10.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:10.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:10.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:10.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:10.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:10.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497155+0000", 2024-03-27T21:41:10.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:10.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.922 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:10.922 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497230+0000", 2024-03-27T21:41:10.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4789000000000005e-05 2024-03-27T21:41:10.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.922 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:10.922 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497241+0000", 2024-03-27T21:41:10.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1442e-05 2024-03-27T21:41:10.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.923 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:10.923 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497255+0000", 2024-03-27T21:41:10.923 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3087e-05 2024-03-27T21:41:10.923 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.924 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:10.924 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503630+0000", 2024-03-27T21:41:10.924 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063755269999999998 2024-03-27T21:41:10.924 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.924 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.924 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:10.924 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503673+0000", 2024-03-27T21:41:10.925 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2616e-05 2024-03-27T21:41:10.925 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.925 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.925 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:10.925 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503738+0000", 2024-03-27T21:41:10.925 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5114e-05 2024-03-27T21:41:10.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.926 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.926 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:10.926 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504907+0000", 2024-03-27T21:41:10.926 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169432 2024-03-27T21:41:10.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.926 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:10.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504926+0000", 2024-03-27T21:41:10.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9086999999999999e-05 2024-03-27T21:41:10.927 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:10.928 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504950+0000", 2024-03-27T21:41:10.928 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3598999999999999e-05 2024-03-27T21:41:10.928 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.928 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.928 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:10.928 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504971+0000", 2024-03-27T21:41:10.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0772999999999999e-05 2024-03-27T21:41:10.929 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.929 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:10.929 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.929 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.929 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.929 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:10.930 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:10.930 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 90.322208778000004, 2024-03-27T21:41:10.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082962590000000003, 2024-03-27T21:41:10.930 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:10.930 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:10.930 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:10.931 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:10.931 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:10.931 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-03-27T21:41:10.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.931 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:10.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:10.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:10.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:10.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:10.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:10.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497262+0000", 2024-03-27T21:41:10.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:10.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.934 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:10.934 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497267+0000", 2024-03-27T21:41:10.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.955e-06 2024-03-27T21:41:10.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.934 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:10.935 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497275+0000", 2024-03-27T21:41:10.935 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.824e-06 2024-03-27T21:41:10.935 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.935 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.935 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:10.935 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497280+0000", 2024-03-27T21:41:10.935 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8230000000000002e-06 2024-03-27T21:41:10.936 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.936 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.936 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:10.936 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504628+0000", 2024-03-27T21:41:10.936 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007348561 2024-03-27T21:41:10.936 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.937 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.937 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:10.937 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504673+0000", 2024-03-27T21:41:10.937 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4313000000000001e-05 2024-03-27T21:41:10.937 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.937 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.938 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:10.938 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504745+0000", 2024-03-27T21:41:10.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2303000000000002e-05 2024-03-27T21:41:10.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.938 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:10.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505358+0000", 2024-03-27T21:41:10.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061281300000000003 2024-03-27T21:41:10.939 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.939 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.939 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:10.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505503+0000", 2024-03-27T21:41:10.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000145134 2024-03-27T21:41:10.940 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.940 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.940 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:10.940 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505526+0000", 2024-03-27T21:41:10.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3312e-05 2024-03-27T21:41:10.941 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.941 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.941 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:10.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505559+0000", 2024-03-27T21:41:10.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2821999999999998e-05 2024-03-27T21:41:10.941 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.942 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:10.942 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.942 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.942 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.942 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:10.942 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:10.943 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 89.712986538999999, 2024-03-27T21:41:10.943 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053753339999999998, 2024-03-27T21:41:10.943 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:10.943 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:10.943 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:10.943 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:10.944 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:10.944 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159 2024-03-27T21:41:10.944 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.944 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:10.944 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.944 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:10.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:10.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.945 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.945 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:10.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:10.946 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:10.946 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.946 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.946 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:10.946 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:10.946 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:10.947 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.947 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.947 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:10.947 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106488+0000", 2024-03-27T21:41:10.947 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9830000000000001e-06 2024-03-27T21:41:10.947 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.948 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.948 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:10.948 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106491+0000", 2024-03-27T21:41:10.948 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3459999999999998e-06 2024-03-27T21:41:10.948 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.948 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.949 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:10.949 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106493+0000", 2024-03-27T21:41:10.949 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.509e-06 2024-03-27T21:41:10.949 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.949 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.949 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:10.950 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110636+0000", 2024-03-27T21:41:10.950 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041427260000000002 2024-03-27T21:41:10.950 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.950 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.950 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:10.950 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110683+0000", 2024-03-27T21:41:10.951 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6557000000000002e-05 2024-03-27T21:41:10.951 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.951 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.951 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:10.951 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110755+0000", 2024-03-27T21:41:10.951 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2119000000000005e-05 2024-03-27T21:41:10.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.952 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.952 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:10.952 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111415+0000", 2024-03-27T21:41:10.952 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066065199999999996 2024-03-27T21:41:10.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:10.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111793+0000", 2024-03-27T21:41:10.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037732100000000001 2024-03-27T21:41:10.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.954 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:10.954 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111829+0000", 2024-03-27T21:41:10.954 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418999999999998e-05 2024-03-27T21:41:10.954 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:10.954 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:10.954 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:10.955 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111860+0000", 2024-03-27T21:41:10.955 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1003000000000001e-05 2024-03-27T21:41:10.955 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.955 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:10.955 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.955 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:10.955 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:10.956 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:10.958 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:10.972+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:10.958 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:41:10.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.983+0000 7f5232ffd640 1 -- 172.21.15.60:0/140813696 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v53) v1 ==== 91+0+0 (secure 0 0 0) 0x7f523c13f190 con 0x7f523c141630 2024-03-27T21:41:10.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.983+0000 7f5232ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:41:10.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.983+0000 7f5232ffd640 10 monclient: _finish_command 2 = system:0 noscrub is unset 2024-03-27T21:41:10.970 INFO:teuthology.orchestra.run.smithi060.stderr:noscrub is unset 2024-03-27T21:41:10.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.985+0000 7f5240f22640 10 librados: watch_flush enter 2024-03-27T21:41:10.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.985+0000 7f5240f22640 10 librados: watch_flush exit 2024-03-27T21:41:10.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.985+0000 7f5240f22640 20 client.4884.objecter shutdown clearing up homeless session... 2024-03-27T21:41:10.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.985+0000 7f5240f22640 10 client.4884.objecter successfully canceled tick 2024-03-27T21:41:10.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.985+0000 7f5240f22640 1 -- 172.21.15.60:0/140813696 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f521c06c590 msgr2=0x7f521c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:10.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.985+0000 7f5240f22640 1 --2- 172.21.15.60:0/140813696 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f521c06c590 0x7f521c06ea50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f522c009c50 tx=0x7f522c008040 comp rx=0 tx=0).stop 2024-03-27T21:41:10.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.985+0000 7f5240f22640 10 monclient: shutdown 2024-03-27T21:41:10.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.985+0000 7f5240f22640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:10.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.985+0000 7f5240f22640 1 -- 172.21.15.60:0/140813696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f523c141630 msgr2=0x7f523c133620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:10.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.985+0000 7f5240f22640 1 --2- 172.21.15.60:0/140813696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f523c141630 0x7f523c133620 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f523400c700 tx=0x7f523400cbc0 comp rx=0 tx=0).stop 2024-03-27T21:41:10.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.985+0000 7f5240f22640 1 -- 172.21.15.60:0/140813696 shutdown_connections 2024-03-27T21:41:10.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.985+0000 7f5240f22640 1 --2- 172.21.15.60:0/140813696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f523c141630 0x7f523c133620 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:10.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.985+0000 7f5240f22640 1 --2- 172.21.15.60:0/140813696 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f523c13e810 0x7f523c1330e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:10.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.985+0000 7f5240f22640 1 --2- 172.21.15.60:0/140813696 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f521c06c590 0x7f521c06ea50 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:10.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.985+0000 7f5240f22640 1 --2- 172.21.15.60:0/140813696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f523c071a50 0x7f523c1d69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:10.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.985+0000 7f5240f22640 1 -- 172.21.15.60:0/140813696 >> 172.21.15.60:0/140813696 conn(0x7f523c0b7300 msgr2=0x7f523c0b17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:10.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.986+0000 7f5240f22640 1 -- 172.21.15.60:0/140813696 shutdown_connections 2024-03-27T21:41:10.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.986+0000 7f5240f22640 1 -- 172.21.15.60:0/140813696 wait complete. 2024-03-27T21:41:10.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:10.986+0000 7f5240f22640 1 librados: shutdown 2024-03-27T21:41:11.058 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:11.073+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:11.067 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:11.067 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:11.067 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:11.068 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:11.076 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:41:11.160 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:11.174+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:11.185 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:11.185 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:11.185 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:41:11.185 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:41:11.185 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:11.194 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:41:11.261 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:11.275+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:11.305 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:11.306 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:41:11.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:41:11.306 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-03-27T21:41:11.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.306 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:20 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2024-03-27T21:41:11.306 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:41:11.306 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 91.429855652000001, 2024-03-27T21:41:11.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015345630000000001, 2024-03-27T21:41:11.306 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:11.306 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:11.306 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:11.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:11.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:41:11.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:11.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:41:11.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:11.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888562+0000", 2024-03-27T21:41:11.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:11.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:11.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888735+0000", 2024-03-27T21:41:11.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017254799999999999 2024-03-27T21:41:11.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:11.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888747+0000", 2024-03-27T21:41:11.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2469999999999999e-05 2024-03-27T21:41:11.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:11.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888761+0000", 2024-03-27T21:41:11.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3254e-05 2024-03-27T21:41:11.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:11.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888804+0000", 2024-03-27T21:41:11.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3129e-05 2024-03-27T21:41:11.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:11.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888820+0000", 2024-03-27T21:41:11.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6426000000000002e-05 2024-03-27T21:41:11.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:11.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890060+0000", 2024-03-27T21:41:11.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012398050000000001 2024-03-27T21:41:11.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:11.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890098+0000", 2024-03-27T21:41:11.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7795e-05 2024-03-27T21:41:11.310 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.310 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:11.310 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.310 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2024-03-27T21:41:11.310 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:41:11.310 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 91.429031252000001, 2024-03-27T21:41:11.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001069555, 2024-03-27T21:41:11.311 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:11.311 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:11.311 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:11.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:11.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:41:11.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:11.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:41:11.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:11.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889387+0000", 2024-03-27T21:41:11.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:11.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:11.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889455+0000", 2024-03-27T21:41:11.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7930999999999997e-05 2024-03-27T21:41:11.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:11.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889470+0000", 2024-03-27T21:41:11.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5554e-05 2024-03-27T21:41:11.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:11.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889476+0000", 2024-03-27T21:41:11.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0619999999999997e-06 2024-03-27T21:41:11.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:11.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889513+0000", 2024-03-27T21:41:11.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6606999999999997e-05 2024-03-27T21:41:11.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:11.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889528+0000", 2024-03-27T21:41:11.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4982e-05 2024-03-27T21:41:11.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:11.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890421+0000", 2024-03-27T21:41:11.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00089293700000000001 2024-03-27T21:41:11.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:11.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890457+0000", 2024-03-27T21:41:11.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6260000000000002e-05 2024-03-27T21:41:11.314 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.314 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:11.314 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.315 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:38 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'24, mlcod=17'22)", 2024-03-27T21:41:11.315 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:41:11.315 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 91.411795286, 2024-03-27T21:41:11.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016182080000000001, 2024-03-27T21:41:11.315 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:11.315 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:11.315 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:11.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:11.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:41:11.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:11.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:41:11.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:11.317 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906623+0000", 2024-03-27T21:41:11.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:11.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.317 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:11.317 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906687+0000", 2024-03-27T21:41:11.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3706000000000003e-05 2024-03-27T21:41:11.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.317 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:11.317 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906698+0000", 2024-03-27T21:41:11.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1752e-05 2024-03-27T21:41:11.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:11.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906704+0000", 2024-03-27T21:41:11.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3940000000000004e-06 2024-03-27T21:41:11.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:11.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906741+0000", 2024-03-27T21:41:11.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6944999999999999e-05 2024-03-27T21:41:11.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:11.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906760+0000", 2024-03-27T21:41:11.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9293e-05 2024-03-27T21:41:11.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:11.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908211+0000", 2024-03-27T21:41:11.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014511350000000001 2024-03-27T21:41:11.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:11.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908242+0000", 2024-03-27T21:41:11.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0712e-05 2024-03-27T21:41:11.319 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.319 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:11.320 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.320 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:39 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'25, mlcod=17'22)", 2024-03-27T21:41:11.320 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:11.320 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 91.410950370999998, 2024-03-27T21:41:11.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012060199999999999, 2024-03-27T21:41:11.320 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:11.320 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:11.320 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:11.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:11.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:11.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:11.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:11.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:11.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:11.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:11.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:11.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907533+0000", 2024-03-27T21:41:11.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5675999999999999e-05 2024-03-27T21:41:11.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.322 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:11.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907545+0000", 2024-03-27T21:41:11.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.115e-05 2024-03-27T21:41:11.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.322 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:11.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907550+0000", 2024-03-27T21:41:11.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8320000000000002e-06 2024-03-27T21:41:11.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:11.323 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907592+0000", 2024-03-27T21:41:11.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1309e-05 2024-03-27T21:41:11.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:11.323 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907611+0000", 2024-03-27T21:41:11.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9009e-05 2024-03-27T21:41:11.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:11.323 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908638+0000", 2024-03-27T21:41:11.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001027533 2024-03-27T21:41:11.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:11.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908675+0000", 2024-03-27T21:41:11.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6329999999999999e-05 2024-03-27T21:41:11.324 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.324 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:11.324 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.324 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:57 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'35, mlcod=17'33)", 2024-03-27T21:41:11.324 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:41:11.324 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 91.396865938999994, 2024-03-27T21:41:11.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001346578, 2024-03-27T21:41:11.324 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:11.324 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:11.324 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:11.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:11.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:41:11.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:11.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:41:11.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:11.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921552+0000", 2024-03-27T21:41:11.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:11.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:11.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921615+0000", 2024-03-27T21:41:11.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2787999999999996e-05 2024-03-27T21:41:11.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:11.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921626+0000", 2024-03-27T21:41:11.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0888e-05 2024-03-27T21:41:11.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:11.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921635+0000", 2024-03-27T21:41:11.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4639999999999995e-06 2024-03-27T21:41:11.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:11.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921673+0000", 2024-03-27T21:41:11.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7891e-05 2024-03-27T21:41:11.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:11.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921694+0000", 2024-03-27T21:41:11.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0426000000000001e-05 2024-03-27T21:41:11.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:11.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922865+0000", 2024-03-27T21:41:11.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011709509999999999 2024-03-27T21:41:11.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:11.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922900+0000", 2024-03-27T21:41:11.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4819e-05 2024-03-27T21:41:11.328 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.328 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:11.328 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.328 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:71 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'45, mlcod=17'43)", 2024-03-27T21:41:11.328 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:41:11.328 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 91.381567227000005, 2024-03-27T21:41:11.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001460907, 2024-03-27T21:41:11.329 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:11.329 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:11.329 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:11.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:11.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:41:11.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:11.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:41:11.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:11.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936851+0000", 2024-03-27T21:41:11.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:11.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:11.331 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936903+0000", 2024-03-27T21:41:11.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2302999999999997e-05 2024-03-27T21:41:11.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.331 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:11.331 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936915+0000", 2024-03-27T21:41:11.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1967000000000001e-05 2024-03-27T21:41:11.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.331 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:11.331 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936923+0000", 2024-03-27T21:41:11.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9370000000000004e-06 2024-03-27T21:41:11.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.331 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:11.332 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936970+0000", 2024-03-27T21:41:11.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6987000000000003e-05 2024-03-27T21:41:11.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.332 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:11.332 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937002+0000", 2024-03-27T21:41:11.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2410999999999999e-05 2024-03-27T21:41:11.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.332 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:11.332 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938279+0000", 2024-03-27T21:41:11.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001276552 2024-03-27T21:41:11.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:11.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938313+0000", 2024-03-27T21:41:11.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3526999999999999e-05 2024-03-27T21:41:11.333 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.333 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:11.333 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.333 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:72 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'46, mlcod=17'43)", 2024-03-27T21:41:11.333 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:41:11.333 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 91.380929664999996, 2024-03-27T21:41:11.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014239490000000001, 2024-03-27T21:41:11.333 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:11.333 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:11.334 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:11.334 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.334 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:11.334 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:41:11.334 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.334 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.334 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:11.334 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:41:11.334 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.334 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.334 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:11.334 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937488+0000", 2024-03-27T21:41:11.334 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:11.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.335 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:11.335 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937547+0000", 2024-03-27T21:41:11.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8196999999999999e-05 2024-03-27T21:41:11.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.335 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:11.335 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937559+0000", 2024-03-27T21:41:11.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1928e-05 2024-03-27T21:41:11.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.336 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:11.336 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937564+0000", 2024-03-27T21:41:11.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4759999999999997e-06 2024-03-27T21:41:11.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.336 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:11.336 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937621+0000", 2024-03-27T21:41:11.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6712000000000002e-05 2024-03-27T21:41:11.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.336 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:11.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937640+0000", 2024-03-27T21:41:11.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9517e-05 2024-03-27T21:41:11.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.337 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:11.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938883+0000", 2024-03-27T21:41:11.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001242567 2024-03-27T21:41:11.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.337 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:11.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938913+0000", 2024-03-27T21:41:11.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0287e-05 2024-03-27T21:41:11.337 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.337 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:11.337 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.338 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:73 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'47, mlcod=17'43)", 2024-03-27T21:41:11.338 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:11.338 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 91.380253570999997, 2024-03-27T21:41:11.338 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010746239999999999, 2024-03-27T21:41:11.338 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:11.338 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:11.338 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:11.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.338 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:11.338 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:11.338 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:11.339 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:11.339 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:11.339 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:11.339 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:11.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:11.339 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938217+0000", 2024-03-27T21:41:11.339 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2743e-05 2024-03-27T21:41:11.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.340 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:11.340 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938229+0000", 2024-03-27T21:41:11.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1956999999999999e-05 2024-03-27T21:41:11.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.340 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:11.340 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938235+0000", 2024-03-27T21:41:11.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4589999999999997e-06 2024-03-27T21:41:11.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.340 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:11.340 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938341+0000", 2024-03-27T21:41:11.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000106054 2024-03-27T21:41:11.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:11.341 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938369+0000", 2024-03-27T21:41:11.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8291000000000001e-05 2024-03-27T21:41:11.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:11.341 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939209+0000", 2024-03-27T21:41:11.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084045400000000003 2024-03-27T21:41:11.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:11.341 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939240+0000", 2024-03-27T21:41:11.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0429999999999998e-05 2024-03-27T21:41:11.341 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.342 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:11.342 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.342 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:75 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'49, mlcod=17'47)", 2024-03-27T21:41:11.342 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:41:11.342 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 91.373815923999999, 2024-03-27T21:41:11.342 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001017001, 2024-03-27T21:41:11.342 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:11.342 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:11.342 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:11.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.342 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:11.342 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:41:11.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.343 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:11.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:41:11.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.343 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:11.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944601+0000", 2024-03-27T21:41:11.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999986 2024-03-27T21:41:11.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.343 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:11.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944614+0000", 2024-03-27T21:41:11.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2496e-05 2024-03-27T21:41:11.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:11.344 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944640+0000", 2024-03-27T21:41:11.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5865999999999999e-05 2024-03-27T21:41:11.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:11.344 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944651+0000", 2024-03-27T21:41:11.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1797000000000001e-05 2024-03-27T21:41:11.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:11.344 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944762+0000", 2024-03-27T21:41:11.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110412 2024-03-27T21:41:11.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.345 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:11.345 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944820+0000", 2024-03-27T21:41:11.345 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7649999999999999e-05 2024-03-27T21:41:11.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.345 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:11.345 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945585+0000", 2024-03-27T21:41:11.345 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076562600000000005 2024-03-27T21:41:11.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.345 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:11.345 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945620+0000", 2024-03-27T21:41:11.345 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4764000000000001e-05 2024-03-27T21:41:11.346 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.346 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:11.346 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.346 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:86 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'50)", 2024-03-27T21:41:11.346 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:11.346 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 91.366054437000003, 2024-03-27T21:41:11.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011392380000000001, 2024-03-27T21:41:11.346 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:11.346 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:11.346 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:11.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.346 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:11.346 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:11.347 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.347 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:11.347 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:11.347 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.347 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:11.347 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:11.347 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:11.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.347 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:11.347 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952373+0000", 2024-03-27T21:41:11.348 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9220000000000003e-06 2024-03-27T21:41:11.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.348 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:11.348 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952381+0000", 2024-03-27T21:41:11.348 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3070000000000003e-06 2024-03-27T21:41:11.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.348 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:11.348 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952385+0000", 2024-03-27T21:41:11.348 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.121e-06 2024-03-27T21:41:11.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.348 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:11.349 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952419+0000", 2024-03-27T21:41:11.349 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4082999999999999e-05 2024-03-27T21:41:11.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.349 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.349 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:11.349 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952440+0000", 2024-03-27T21:41:11.349 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0837999999999999e-05 2024-03-27T21:41:11.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.349 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.349 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:11.349 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953472+0000", 2024-03-27T21:41:11.349 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001032119 2024-03-27T21:41:11.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.349 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.349 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:11.349 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953504+0000", 2024-03-27T21:41:11.350 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1250000000000001e-05 2024-03-27T21:41:11.350 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.350 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:11.350 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.350 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:87 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-27T21:41:11.350 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:41:11.350 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 91.365687179999995, 2024-03-27T21:41:11.350 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00135519, 2024-03-27T21:41:11.350 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:11.350 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:11.350 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:11.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.351 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:11.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:41:11.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.351 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:11.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:41:11.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.351 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:11.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952731+0000", 2024-03-27T21:41:11.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:11.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.352 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:11.352 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952781+0000", 2024-03-27T21:41:11.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0161000000000003e-05 2024-03-27T21:41:11.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.352 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:11.352 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952794+0000", 2024-03-27T21:41:11.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2853e-05 2024-03-27T21:41:11.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.352 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:11.352 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952799+0000", 2024-03-27T21:41:11.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5560000000000001e-06 2024-03-27T21:41:11.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.353 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:11.353 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952832+0000", 2024-03-27T21:41:11.353 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3834999999999997e-05 2024-03-27T21:41:11.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.353 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:11.353 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952851+0000", 2024-03-27T21:41:11.353 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8289999999999999e-05 2024-03-27T21:41:11.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.353 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:11.353 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954055+0000", 2024-03-27T21:41:11.353 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012037949999999999 2024-03-27T21:41:11.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.354 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:11.354 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954087+0000", 2024-03-27T21:41:11.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2292999999999997e-05 2024-03-27T21:41:11.354 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.354 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:11.354 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.354 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:89 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'59, mlcod=17'57)", 2024-03-27T21:41:11.354 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:41:11.354 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 91.363412959000001, 2024-03-27T21:41:11.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00138893, 2024-03-27T21:41:11.354 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:11.354 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:11.355 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:11.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:11.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:41:11.355 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:11.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:41:11.355 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:11.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955005+0000", 2024-03-27T21:41:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955071+0000", 2024-03-27T21:41:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5290999999999995e-05 2024-03-27T21:41:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955081+0000", 2024-03-27T21:41:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0572e-05 2024-03-27T21:41:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955086+0000", 2024-03-27T21:41:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8380000000000001e-06 2024-03-27T21:41:11.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.357 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:11.357 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955122+0000", 2024-03-27T21:41:11.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5698000000000003e-05 2024-03-27T21:41:11.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.357 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:11.357 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955140+0000", 2024-03-27T21:41:11.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7900000000000001e-05 2024-03-27T21:41:11.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.357 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:11.357 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956366+0000", 2024-03-27T21:41:11.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001226471 2024-03-27T21:41:11.358 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.358 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.358 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:11.358 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956395+0000", 2024-03-27T21:41:11.358 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8782e-05 2024-03-27T21:41:11.358 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.358 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:11.358 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.358 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.358 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.358 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:90 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'60, mlcod=17'57)", 2024-03-27T21:41:11.358 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:11.359 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 91.362728382, 2024-03-27T21:41:11.359 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010778859999999999, 2024-03-27T21:41:11.359 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:11.359 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:11.359 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:11.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.359 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:11.359 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:11.359 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.359 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.360 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:11.360 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:11.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.360 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:11.360 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:11.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:11.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.360 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:11.360 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955747+0000", 2024-03-27T21:41:11.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7130999999999999e-05 2024-03-27T21:41:11.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.361 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:11.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955756+0000", 2024-03-27T21:41:11.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2750000000000001e-06 2024-03-27T21:41:11.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.361 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:11.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955761+0000", 2024-03-27T21:41:11.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3900000000000003e-06 2024-03-27T21:41:11.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.361 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:11.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955797+0000", 2024-03-27T21:41:11.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6650000000000003e-05 2024-03-27T21:41:11.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.362 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:11.362 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955816+0000", 2024-03-27T21:41:11.362 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8195000000000001e-05 2024-03-27T21:41:11.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.362 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:11.362 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956739+0000", 2024-03-27T21:41:11.362 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00092369700000000002 2024-03-27T21:41:11.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.362 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:11.362 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956768+0000", 2024-03-27T21:41:11.362 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9048e-05 2024-03-27T21:41:11.363 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.363 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:11.363 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.363 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.363 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.363 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:118 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'80, mlcod=17'78)", 2024-03-27T21:41:11.363 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:11.363 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 91.229737559, 2024-03-27T21:41:11.363 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014200669999999999, 2024-03-27T21:41:11.363 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:11.363 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:11.363 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:11.363 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.364 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:11.364 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:11.364 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.364 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.364 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.364 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:11.364 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:11.364 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.364 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.364 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.364 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:11.365 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:11.365 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:11.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.365 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:11.365 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088756+0000", 2024-03-27T21:41:11.365 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4902999999999998e-05 2024-03-27T21:41:11.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.365 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:11.365 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088764+0000", 2024-03-27T21:41:11.365 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8169999999999999e-06 2024-03-27T21:41:11.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.365 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:11.366 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088767+0000", 2024-03-27T21:41:11.366 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8310000000000003e-06 2024-03-27T21:41:11.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.366 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.366 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:11.366 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088798+0000", 2024-03-27T21:41:11.366 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0800999999999999e-05 2024-03-27T21:41:11.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.366 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.366 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:11.366 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088817+0000", 2024-03-27T21:41:11.366 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8467000000000001e-05 2024-03-27T21:41:11.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.366 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.366 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:11.367 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090074+0000", 2024-03-27T21:41:11.367 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012574089999999999 2024-03-27T21:41:11.367 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.367 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.367 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:11.367 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090101+0000", 2024-03-27T21:41:11.367 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7223e-05 2024-03-27T21:41:11.367 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.367 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:11.367 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.367 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.367 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.367 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:119 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'81, mlcod=17'78)", 2024-03-27T21:41:11.367 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:11.367 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 91.229002764000001, 2024-03-27T21:41:11.367 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014990870000000001, 2024-03-27T21:41:11.368 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:11.368 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:11.368 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:11.368 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.368 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:11.368 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:11.368 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.368 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.368 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.368 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:11.368 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:11.368 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.368 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.368 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.368 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:11.369 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:11.369 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:11.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.369 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.369 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:11.369 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089491+0000", 2024-03-27T21:41:11.369 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4805e-05 2024-03-27T21:41:11.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.369 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.369 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:11.369 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089498+0000", 2024-03-27T21:41:11.369 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0650000000000001e-06 2024-03-27T21:41:11.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.369 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.369 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:11.370 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089501+0000", 2024-03-27T21:41:11.370 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.54e-06 2024-03-27T21:41:11.370 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.370 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.370 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:11.370 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089533+0000", 2024-03-27T21:41:11.370 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1838999999999998e-05 2024-03-27T21:41:11.370 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.370 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.370 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:11.370 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089550+0000", 2024-03-27T21:41:11.370 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7465999999999998e-05 2024-03-27T21:41:11.370 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.370 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.370 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:11.371 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090890+0000", 2024-03-27T21:41:11.371 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013397979999999999 2024-03-27T21:41:11.371 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.371 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.371 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:11.371 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090915+0000", 2024-03-27T21:41:11.371 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4905e-05 2024-03-27T21:41:11.371 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.371 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:11.371 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.371 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.371 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.371 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:120 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'82, mlcod=17'78)", 2024-03-27T21:41:11.371 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:11.371 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 91.228202883999998, 2024-03-27T21:41:11.372 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001527093, 2024-03-27T21:41:11.372 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:11.372 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:11.372 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:11.372 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.372 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:11.372 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:11.372 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.372 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.372 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:11.372 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:11.373 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.373 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.373 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:11.373 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:11.373 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:11.373 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.373 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:11.373 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090267+0000", 2024-03-27T21:41:11.373 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1477000000000002e-05 2024-03-27T21:41:11.373 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.374 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:11.374 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090273+0000", 2024-03-27T21:41:11.374 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9519999999999999e-06 2024-03-27T21:41:11.374 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.374 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.374 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:11.374 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090276+0000", 2024-03-27T21:41:11.374 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.219e-06 2024-03-27T21:41:11.374 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.374 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.374 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:11.374 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090304+0000", 2024-03-27T21:41:11.374 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8073999999999999e-05 2024-03-27T21:41:11.374 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.374 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.375 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:11.375 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090316+0000", 2024-03-27T21:41:11.375 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2088e-05 2024-03-27T21:41:11.375 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.375 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.375 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:11.375 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091717+0000", 2024-03-27T21:41:11.375 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014003889999999999 2024-03-27T21:41:11.375 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.375 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.375 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:11.375 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091743+0000", 2024-03-27T21:41:11.375 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6234e-05 2024-03-27T21:41:11.375 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.375 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:11.376 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.376 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.376 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.376 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:121 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'83, mlcod=17'78)", 2024-03-27T21:41:11.376 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:11.376 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 91.227392519000006, 2024-03-27T21:41:11.376 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010226899999999999, 2024-03-27T21:41:11.376 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:11.376 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:11.376 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:11.376 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.376 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:11.376 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:11.376 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.377 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.377 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:11.377 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:11.377 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.377 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.377 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:11.377 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:11.377 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:11.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.377 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.377 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:11.378 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091142+0000", 2024-03-27T21:41:11.378 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011608300000000001 2024-03-27T21:41:11.378 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.378 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.378 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:11.378 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091151+0000", 2024-03-27T21:41:11.378 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2350000000000005e-06 2024-03-27T21:41:11.378 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.378 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.378 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:11.378 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091156+0000", 2024-03-27T21:41:11.378 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5419999999999999e-06 2024-03-27T21:41:11.378 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.378 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.378 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:11.379 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091191+0000", 2024-03-27T21:41:11.379 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5453000000000001e-05 2024-03-27T21:41:11.379 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.379 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.379 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:11.379 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091208+0000", 2024-03-27T21:41:11.379 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7119999999999999e-05 2024-03-27T21:41:11.379 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.379 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.379 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:11.379 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092024+0000", 2024-03-27T21:41:11.380 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000815539 2024-03-27T21:41:11.380 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.380 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.380 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:11.380 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092049+0000", 2024-03-27T21:41:11.380 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5171e-05 2024-03-27T21:41:11.380 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.380 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:11.380 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.380 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.380 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.380 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:142 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'100, mlcod=18'98)", 2024-03-27T21:41:11.380 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:11.380 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.812385565, 2024-03-27T21:41:11.380 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001198778, 2024-03-27T21:41:11.381 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:11.381 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:11.381 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:11.381 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.381 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:11.381 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:11.381 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.381 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.381 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.381 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:11.381 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:11.381 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.381 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.381 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.381 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:11.381 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:11.382 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:11.382 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.382 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.382 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:11.382 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506101+0000", 2024-03-27T21:41:11.382 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8202999999999997e-05 2024-03-27T21:41:11.382 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.382 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.382 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:11.382 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506109+0000", 2024-03-27T21:41:11.382 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3059999999999998e-06 2024-03-27T21:41:11.382 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.382 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.382 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:11.382 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506113+0000", 2024-03-27T21:41:11.383 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0119999999999999e-06 2024-03-27T21:41:11.383 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.383 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.383 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:11.383 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506137+0000", 2024-03-27T21:41:11.383 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4043999999999999e-05 2024-03-27T21:41:11.383 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.383 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.383 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:11.383 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506148+0000", 2024-03-27T21:41:11.383 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0446e-05 2024-03-27T21:41:11.384 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.384 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.384 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:11.384 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507213+0000", 2024-03-27T21:41:11.384 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001065215 2024-03-27T21:41:11.384 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.384 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.384 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:11.384 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507232+0000", 2024-03-27T21:41:11.384 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9038e-05 2024-03-27T21:41:11.384 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.384 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:11.384 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.384 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.384 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.385 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:143 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'101, mlcod=18'98)", 2024-03-27T21:41:11.385 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:11.385 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.811772946000005, 2024-03-27T21:41:11.385 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001357488, 2024-03-27T21:41:11.385 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:11.385 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:11.385 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:11.385 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.385 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:11.385 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:11.385 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.385 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.385 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.385 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:11.385 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:11.386 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.386 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.386 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.386 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:11.386 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:11.386 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:11.386 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.386 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.386 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:11.386 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506723+0000", 2024-03-27T21:41:11.386 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7404000000000006e-05 2024-03-27T21:41:11.386 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.386 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.386 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:11.386 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506730+0000", 2024-03-27T21:41:11.387 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5749999999999997e-06 2024-03-27T21:41:11.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.387 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:11.387 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506733+0000", 2024-03-27T21:41:11.387 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7979999999999999e-06 2024-03-27T21:41:11.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.387 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:11.387 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506758+0000", 2024-03-27T21:41:11.387 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4179e-05 2024-03-27T21:41:11.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.387 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:11.387 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506768+0000", 2024-03-27T21:41:11.388 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0740000000000001e-05 2024-03-27T21:41:11.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.388 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.388 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:11.388 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507978+0000", 2024-03-27T21:41:11.388 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012093900000000001 2024-03-27T21:41:11.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.388 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.388 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:11.388 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508003+0000", 2024-03-27T21:41:11.388 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5667e-05 2024-03-27T21:41:11.388 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.388 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:11.388 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.389 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.389 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:144 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'102, mlcod=18'98)", 2024-03-27T21:41:11.389 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:41:11.389 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.811114048999997, 2024-03-27T21:41:11.389 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013453390000000001, 2024-03-27T21:41:11.389 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:11.389 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:11.389 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:11.389 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.389 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:11.389 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:41:11.389 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.389 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.389 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.389 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:11.390 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:41:11.390 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:11.390 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.390 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.390 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:11.390 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507304+0000", 2024-03-27T21:41:11.390 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:11.390 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.390 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.390 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:11.390 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507397+0000", 2024-03-27T21:41:11.390 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2995e-05 2024-03-27T21:41:11.390 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.390 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.391 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:11.391 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507403+0000", 2024-03-27T21:41:11.391 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8640000000000001e-06 2024-03-27T21:41:11.391 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.391 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.391 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:11.391 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507407+0000", 2024-03-27T21:41:11.391 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5710000000000002e-06 2024-03-27T21:41:11.391 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.391 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.391 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:11.391 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507428+0000", 2024-03-27T21:41:11.391 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1192e-05 2024-03-27T21:41:11.391 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.391 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.392 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:11.392 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507438+0000", 2024-03-27T21:41:11.392 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8309999999999996e-06 2024-03-27T21:41:11.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.392 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.392 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:11.392 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508624+0000", 2024-03-27T21:41:11.392 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011861210000000001 2024-03-27T21:41:11.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:11.392 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:11.392 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:11.392 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508650+0000", 2024-03-27T21:41:11.392 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6118e-05 2024-03-27T21:41:11.392 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.393 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:11.393 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.393 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:11.393 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:11.393 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:11.394 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:41:11.395 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:11.409+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:11.495 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:11.510+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:11.504 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:11.504 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:11.504 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:11.504 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:11.513 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:41:11.596 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:11.610+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:11.629 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:11.629 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:11.629 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:41:11.629 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:41:11.629 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:11.638 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:41:11.696 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:11.710+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:11.747 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:11.747 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:41:11.747 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:41:11.748 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:41:11.748 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:11.757 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:41:11.797 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:11.812+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:11.898 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:11.912+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:11.922 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:11.922 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:11.922 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:11.923 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:11.931 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:41:11.999 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:12.013+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:12.050 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:12.050 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:12.050 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:41:12.050 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:41:12.050 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:12.059 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:41:12.100 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:12.114+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:12.201 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:12.215+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:12.206 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:12.207 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:41:12.207 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:41:12.207 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:41:12.207 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:12.216 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:41:12.302 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:12.316+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:12.324 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-27T21:41:12.325 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T21:41:12.325 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-27T21:41:12.325 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2024-03-27T21:41:12.332 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:12.333 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:12.333 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:12.333 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:12.343 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:41:12.407 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:41:12.426 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:12.441+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:12.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.442+0000 7fb443bf6640 10 monclient: get_monmap_and_config 2024-03-27T21:41:12.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.442+0000 7fb443bf6640 10 monclient: build_initial_monmap 2024-03-27T21:41:12.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.442+0000 7fb443bf6640 10 monclient: monmap: 2024-03-27T21:41:12.429 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:12.429 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:12.429 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:12.443821+0000 2024-03-27T21:41:12.429 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:12.443821+0000 2024-03-27T21:41:12.429 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:12.429 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:12.429 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:12.429 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:12.430 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:12.430 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:12.430 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:12.430 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:12.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.443+0000 7fb443bf6640 1 Processor -- start 2024-03-27T21:41:12.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.443+0000 7fb443bf6640 1 -- start start 2024-03-27T21:41:12.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.443+0000 7fb443bf6640 10 monclient: init 2024-03-27T21:41:12.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb443bf6640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:12.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb443bf6640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-03-27T21:41:12.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb443bf6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb43c0b40b0 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:12.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb443bf6640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb43c13b020 0x7fb43c13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:12.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb443bf6640 10 monclient(hunting): picked mon.noname-a con 0x7fb43c13b020 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:12.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb443bf6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb43c1416f0 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:12.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb443bf6640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb43c0b3830 con 0x7fb43c1416f0 2024-03-27T21:41:12.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb443bf6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb43c0aa520 con 0x7fb43c0b40b0 2024-03-27T21:41:12.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb443bf6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:12.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb443bf6640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb43c0b4490 con 0x7fb43c13b020 2024-03-27T21:41:12.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb443bf6640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:12.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb443bf6640 10 monclient(hunting): authenticate will time out at 913.665227s 2024-03-27T21:41:12.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb441191640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb43c13b020 0x7fb43c13b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:12.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb441191640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb43c13b020 0x7fb43c13b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37210/0 (socket says 172.21.15.60:37210) 2024-03-27T21:41:12.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb441191640 1 -- 172.21.15.60:0/2117924391 learned_addr learned my addr 172.21.15.60:0/2117924391 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:12.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb441191640 10 monclient(hunting): get_auth_request con 0x7fb43c13b020 auth_method 0 2024-03-27T21:41:12.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb441191640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:12.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb441191640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:12.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb441191640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:12.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb441191640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:12.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb441191640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:12.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb441191640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:12.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb441191640 10 monclient(hunting): handle_auth_done global_id 4896 payload 274 2024-03-27T21:41:12.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb441191640 1 -- 172.21.15.60:0/2117924391 >> v1:172.21.15.191:6789/0 conn(0x7fb43c0b40b0 legacy=0x7fb43c0a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:12.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb441191640 1 -- 172.21.15.60:0/2117924391 >> v1:172.21.15.60:6789/0 conn(0x7fb43c1416f0 legacy=0x7fb43c141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:12.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb441191640 10 monclient: _finish_hunting 0 2024-03-27T21:41:12.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb441191640 1 monclient: found mon.noname-a 2024-03-27T21:41:12.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb441191640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:12.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb441191640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:12.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.444+0000 7fb441191640 1 -- 172.21.15.60:0/2117924391 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43c0a9470 con 0x7fb43c13b020 2024-03-27T21:41:12.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.445+0000 7fb441191640 1 --2- 172.21.15.60:0/2117924391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb43c13b020 0x7fb43c13b480 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb430009870 tx=0x7fb43002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0c047351c080a86 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:12.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.445+0000 7fb440990640 1 -- 172.21.15.60:0/2117924391 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43002f840 con 0x7fb43c13b020 2024-03-27T21:41:12.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.445+0000 7fb440990640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:12.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.445+0000 7fb440990640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:12.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.445+0000 7fb440990640 10 monclient: dump: 2024-03-27T21:41:12.440 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:12.440 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:12.440 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:12.441 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:12.441 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:12.441 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:12.441 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:12.441 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:12.441 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:12.441 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:12.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.445+0000 7fb440990640 10 monclient: _finish_auth 0 2024-03-27T21:41:12.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.445+0000 7fb440990640 10 monclient: _check_auth_tickets 2024-03-27T21:41:12.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.445+0000 7fb440990640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:12.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.445+0000 7fb443bf6640 5 monclient: authenticate success, global_id 4896 2024-03-27T21:41:12.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.445+0000 7fb440990640 1 -- 172.21.15.60:0/2117924391 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb43002f9a0 con 0x7fb43c13b020 2024-03-27T21:41:12.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.445+0000 7fb440990640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:12.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.445+0000 7fb443bf6640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:12.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.445+0000 7fb440990640 1 -- 172.21.15.60:0/2117924391 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43002fd30 con 0x7fb43c13b020 2024-03-27T21:41:12.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.445+0000 7fb440990640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:12.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.445+0000 7fb443bf6640 4 set_mon_vals no callback set 2024-03-27T21:41:12.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.445+0000 7fb440990640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:12.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.445+0000 7fb440990640 10 monclient: dump: 2024-03-27T21:41:12.442 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:12.442 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:12.442 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:12.442 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:12.442 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:12.442 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:12.442 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:12.442 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:12.442 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:12.442 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:12.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.446+0000 7fb443bf6640 10 monclient: shutdown 2024-03-27T21:41:12.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.446+0000 7fb443bf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:12.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.446+0000 7fb443bf6640 1 -- 172.21.15.60:0/2117924391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb43c13b020 msgr2=0x7fb43c13b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:12.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.446+0000 7fb443bf6640 1 --2- 172.21.15.60:0/2117924391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb43c13b020 0x7fb43c13b480 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb430009870 tx=0x7fb43002ba20 comp rx=0 tx=0).stop 2024-03-27T21:41:12.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.446+0000 7fb443bf6640 1 -- 172.21.15.60:0/2117924391 shutdown_connections 2024-03-27T21:41:12.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.446+0000 7fb443bf6640 1 --2- 172.21.15.60:0/2117924391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb43c13b020 0x7fb43c13b480 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:12.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.446+0000 7fb443bf6640 1 -- 172.21.15.60:0/2117924391 >> 172.21.15.60:0/2117924391 conn(0x7fb43c0608e0 msgr2=0x7fb43c060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:12.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.446+0000 7fb443bf6640 1 -- 172.21.15.60:0/2117924391 shutdown_connections 2024-03-27T21:41:12.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.446+0000 7fb443bf6640 1 -- 172.21.15.60:0/2117924391 wait complete. 2024-03-27T21:41:12.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.447+0000 7fb443bf6640 10 monclient: build_initial_monmap 2024-03-27T21:41:12.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.447+0000 7fb443bf6640 10 monclient: monmap: 2024-03-27T21:41:12.443 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:12.443 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:12.443 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:12.444 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.447+0000 7fb443bf6640 1 librados: starting msgr at 2024-03-27T21:41:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.447+0000 7fb443bf6640 1 librados: starting objecter 2024-03-27T21:41:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.447+0000 7fb443bf6640 1 Processor -- start 2024-03-27T21:41:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.447+0000 7fb443bf6640 1 -- start start 2024-03-27T21:41:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.447+0000 7fb443bf6640 1 librados: setting wanted keys 2024-03-27T21:41:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.447+0000 7fb443bf6640 1 librados: calling monclient init 2024-03-27T21:41:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.447+0000 7fb443bf6640 10 monclient: init 2024-03-27T21:41:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb443bf6640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb443bf6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:41:12.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb443bf6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb43c0b40b0 0x7fb43c13e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:12.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb443bf6640 10 monclient(hunting): picked mon.noname-a con 0x7fb43c0b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:12.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb443bf6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb43c13b020 0x7fb43c139d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:12.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb443bf6640 10 monclient(hunting): picked mon.noname-b con 0x7fb43c13b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:12.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb443bf6640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb43c1416f0 0x7fb43c133230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:12.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb443bf6640 10 monclient(hunting): picked mon.noname-c con 0x7fb43c1416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:12.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb443bf6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:12.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb443bf6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c13ae60 con 0x7fb43c0b40b0 2024-03-27T21:41:12.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb443bf6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:12.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb443bf6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c13a3f0 con 0x7fb43c13b020 2024-03-27T21:41:12.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb443bf6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:12.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb443bf6640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c139910 con 0x7fb43c1416f0 2024-03-27T21:41:12.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb443bf6640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:12.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb443bf6640 10 monclient(hunting): authenticate will time out at 913.669168s 2024-03-27T21:41:12.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441992640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb43c0b40b0 0x7fb43c13e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:12.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441992640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb43c0b40b0 0x7fb43c13e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37212/0 (socket says 172.21.15.60:37212) 2024-03-27T21:41:12.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441992640 1 -- 172.21.15.60:0/719544222 learned_addr learned my addr 172.21.15.60:0/719544222 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:12.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441992640 10 monclient(hunting): get_auth_request con 0x7fb43c0b40b0 auth_method 0 2024-03-27T21:41:12.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441992640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:12.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441992640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:12.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441992640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:12.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb442193640 1 --2- 172.21.15.60:0/719544222 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb43c1416f0 0x7fb43c133230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:12.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441191640 1 --2- 172.21.15.60:0/719544222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb43c13b020 0x7fb43c139d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:12.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441992640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:12.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441992640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:12.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441992640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:12.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb442193640 10 monclient(hunting): get_auth_request con 0x7fb43c1416f0 auth_method 0 2024-03-27T21:41:12.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb442193640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:12.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb442193640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:12.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb442193640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:12.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441191640 10 monclient(hunting): get_auth_request con 0x7fb43c13b020 auth_method 0 2024-03-27T21:41:12.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441191640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:12.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441191640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:12.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441191640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:12.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441191640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:12.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441191640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:12.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441191640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:12.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441992640 10 monclient(hunting): handle_auth_done global_id 4899 payload 824 2024-03-27T21:41:12.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441992640 1 -- 172.21.15.60:0/719544222 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb43c1416f0 msgr2=0x7fb43c133230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441992640 1 --2- 172.21.15.60:0/719544222 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb43c1416f0 0x7fb43c133230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441992640 1 -- 172.21.15.60:0/719544222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb43c13b020 msgr2=0x7fb43c139d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441992640 1 --2- 172.21.15.60:0/719544222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb43c13b020 0x7fb43c139d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441992640 10 monclient: _finish_hunting 0 2024-03-27T21:41:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441992640 1 monclient: found mon.noname-a 2024-03-27T21:41:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441992640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441992640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441992640 1 -- 172.21.15.60:0/719544222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4300094d0 con 0x7fb43c0b40b0 2024-03-27T21:41:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb442193640 1 --2- 172.21.15.60:0/719544222 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb43c1416f0 0x7fb43c133230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:41:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.448+0000 7fb441992640 1 --2- 172.21.15.60:0/719544222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb43c0b40b0 0x7fb43c13e110 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb43800c450 tx=0x7fb43800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb42e7fc640 10 client.?.objecter ms_handle_connect 0x7fb43c0b40b0 2024-03-27T21:41:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb42e7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb42e7fc640 1 -- 172.21.15.60:0/719544222 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4380052d0 con 0x7fb43c0b40b0 2024-03-27T21:41:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb42e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:12.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb42e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:12.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb42e7fc640 10 monclient: dump: 2024-03-27T21:41:12.449 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:12.449 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:12.449 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:12.449 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:12.449 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:12.449 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:12.449 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:12.449 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:12.449 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:12.449 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:12.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb42e7fc640 10 monclient: _finish_auth 0 2024-03-27T21:41:12.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb42e7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:41:12.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb42e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:12.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb42e7fc640 1 -- 172.21.15.60:0/719544222 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb438005430 con 0x7fb43c0b40b0 2024-03-27T21:41:12.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb443bf6640 5 monclient: authenticate success, global_id 4899 2024-03-27T21:41:12.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb443bf6640 10 monclient: _renew_subs 2024-03-27T21:41:12.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb443bf6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:12.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb443bf6640 1 -- 172.21.15.60:0/719544222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb43c139b40 con 0x7fb43c0b40b0 2024-03-27T21:41:12.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb42e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:12.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb42e7fc640 1 -- 172.21.15.60:0/719544222 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb438012870 con 0x7fb43c0b40b0 2024-03-27T21:41:12.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb42e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:12.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb42e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:12.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb42e7fc640 10 monclient: dump: 2024-03-27T21:41:12.450 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:12.450 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:12.450 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:12.450 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:12.451 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:12.451 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:12.451 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:12.451 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:12.451 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:12.451 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:12.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb443bf6640 10 client.4899.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:12.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb443bf6640 10 monclient: _renew_subs 2024-03-27T21:41:12.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb42ffff640 4 set_mon_vals no callback set 2024-03-27T21:41:12.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb443bf6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:12.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb443bf6640 1 -- 172.21.15.60:0/719544222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb43c133950 con 0x7fb43c0b40b0 2024-03-27T21:41:12.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.449+0000 7fb443bf6640 1 librados: init done 2024-03-27T21:41:12.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.450+0000 7fb443bf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:12.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.450+0000 7fb443bf6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:12.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.450+0000 7fb443bf6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:12.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.450+0000 7fb443bf6640 1 -- 172.21.15.60:0/719544222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb43c069470 con 0x7fb43c0b40b0 2024-03-27T21:41:12.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.450+0000 7fb42e7fc640 1 -- 172.21.15.60:0/719544222 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb4380129d0 con 0x7fb43c0b40b0 2024-03-27T21:41:12.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.451+0000 7fb42e7fc640 1 --2- 172.21.15.60:0/719544222 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb42006c590 0x7fb42006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:12.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.451+0000 7fb441191640 1 --2- 172.21.15.60:0/719544222 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb42006c590 0x7fb42006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:12.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.451+0000 7fb42e7fc640 1 -- 172.21.15.60:0/719544222 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fb438088c20 con 0x7fb43c0b40b0 2024-03-27T21:41:12.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.451+0000 7fb42e7fc640 10 client.4899.objecter ms_dispatch 0x7fb43c000b90 osd_map(53..53 src has 1..53) v4 2024-03-27T21:41:12.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.451+0000 7fb42e7fc640 3 client.4899.objecter handle_osd_map got epochs [53,53] > 0 2024-03-27T21:41:12.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.451+0000 7fb42e7fc640 3 client.4899.objecter handle_osd_map decoding full epoch 53 2024-03-27T21:41:12.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.451+0000 7fb441191640 10 monclient: get_auth_request con 0x7fb42006c590 auth_method 0 2024-03-27T21:41:12.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.451+0000 7fb42e7fc640 20 client.4899.objecter dump_active .. 0 homeless 2024-03-27T21:41:12.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.451+0000 7fb42e7fc640 10 client.4899.objecter ms_handle_connect 0x7fb42006c590 2024-03-27T21:41:12.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.451+0000 7fb441191640 1 --2- 172.21.15.60:0/719544222 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb42006c590 0x7fb42006ea50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb430036000 tx=0x7fb430008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:12.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.455+0000 7fb42e7fc640 1 -- 172.21.15.60:0/719544222 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb438053fd0 con 0x7fb43c0b40b0 2024-03-27T21:41:12.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.456+0000 7fb42e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:12.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.456+0000 7fb42e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:12.488 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:12.488 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:12.489 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:12.489 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:12.489 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:12.497 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:41:12.504 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:12.518+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:12.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.529+0000 7fa316869640 10 monclient: get_monmap_and_config 2024-03-27T21:41:12.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.529+0000 7fa316869640 10 monclient: build_initial_monmap 2024-03-27T21:41:12.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.529+0000 7fa316869640 10 monclient: monmap: 2024-03-27T21:41:12.515 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:12.515 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:12.515 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:12.530376+0000 2024-03-27T21:41:12.516 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:12.530376+0000 2024-03-27T21:41:12.516 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:12.516 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:12.516 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:12.516 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:12.516 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:12.516 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:12.516 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:12.516 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:12.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa316869640 1 Processor -- start 2024-03-27T21:41:12.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa316869640 1 -- start start 2024-03-27T21:41:12.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa316869640 10 monclient: init 2024-03-27T21:41:12.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa316869640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:12.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa316869640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-03-27T21:41:12.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa316869640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa3100b40b0 0x7fa3100a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:12.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa316869640 10 monclient(hunting): picked mon.noname-b con 0x7fa3100b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:12.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa316869640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa31013b020 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:12.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa316869640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa31013b480 0x7fa310141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:12.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa316869640 10 monclient(hunting): picked mon.noname-a con 0x7fa31013b480 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:12.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa316869640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3100b3830 con 0x7fa31013b020 2024-03-27T21:41:12.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa316869640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:12.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa316869640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa31013b910 con 0x7fa31013b480 2024-03-27T21:41:12.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa316869640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:12.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa316869640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3100b4490 con 0x7fa3100b40b0 2024-03-27T21:41:12.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa316869640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:12.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa316869640 10 monclient(hunting): authenticate will time out at 913.751616s 2024-03-27T21:41:12.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa314e06640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa31013b480 0x7fa310141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:12.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa30ffff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa3100b40b0 0x7fa3100a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:12.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa314e06640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa31013b480 0x7fa310141e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37218/0 (socket says 172.21.15.60:37218) 2024-03-27T21:41:12.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa30f7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa31013b020 0x7fa3101416f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40192/0 (socket says 172.21.15.60:40192) 2024-03-27T21:41:12.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa314e06640 1 -- 172.21.15.60:0/1217977078 learned_addr learned my addr 172.21.15.60:0/1217977078 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:12.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa314e06640 10 monclient(hunting): get_auth_request con 0x7fa31013b480 auth_method 0 2024-03-27T21:41:12.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa314e06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:12.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa314e06640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:12.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa314e06640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:12.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa30ffff640 10 monclient(hunting): get_auth_request con 0x7fa3100b40b0 auth_method 0 2024-03-27T21:41:12.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa30ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:12.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa30ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:12.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa30ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:12.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa30ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:12.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa30ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:12.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.530+0000 7fa30ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:12.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.531+0000 7fa314e06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:12.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.531+0000 7fa314e06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:12.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.531+0000 7fa314e06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:12.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.531+0000 7fa30ffff640 10 monclient(hunting): handle_auth_done global_id 4727 payload 274 2024-03-27T21:41:12.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.531+0000 7fa30ffff640 1 -- 172.21.15.60:0/1217977078 >> v2:172.21.15.60:3300/0 conn(0x7fa31013b480 msgr2=0x7fa310141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:12.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.531+0000 7fa30ffff640 1 --2- 172.21.15.60:0/1217977078 >> v2:172.21.15.60:3300/0 conn(0x7fa31013b480 0x7fa310141e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:12.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.531+0000 7fa30ffff640 1 -- 172.21.15.60:0/1217977078 >> v1:172.21.15.60:6789/0 conn(0x7fa31013b020 legacy=0x7fa3101416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:12.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.531+0000 7fa30ffff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:12.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.531+0000 7fa30ffff640 1 monclient: found mon.noname-b 2024-03-27T21:41:12.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.531+0000 7fa30ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:12.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.531+0000 7fa30ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:12.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.531+0000 7fa30ffff640 1 -- 172.21.15.60:0/1217977078 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3100aa520 con 0x7fa3100b40b0 2024-03-27T21:41:12.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.531+0000 7fa30ffff640 1 --2- 172.21.15.60:0/1217977078 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa3100b40b0 0x7fa3100a8b90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa3000097a0 tx=0x7fa30002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba2cee15ced4e085 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:12.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa30effd640 1 -- 172.21.15.60:0/1217977078 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30002e070 con 0x7fa3100b40b0 2024-03-27T21:41:12.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa30effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:12.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa30effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:12.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa30effd640 10 monclient: dump: 2024-03-27T21:41:12.535 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:12.535 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:12.535 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:12.535 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:12.535 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:12.535 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:12.535 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:12.535 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:12.535 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:12.535 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:12.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa30effd640 10 monclient: _finish_auth 0 2024-03-27T21:41:12.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa30effd640 10 monclient: _check_auth_tickets 2024-03-27T21:41:12.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa30effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:12.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa30effd640 1 -- 172.21.15.60:0/1217977078 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3000315a0 con 0x7fa3100b40b0 2024-03-27T21:41:12.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa316869640 5 monclient: authenticate success, global_id 4727 2024-03-27T21:41:12.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa316869640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:12.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa30effd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:12.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa316869640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:12.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa30effd640 1 -- 172.21.15.60:0/1217977078 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa300031930 con 0x7fa3100b40b0 2024-03-27T21:41:12.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa30effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:12.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa316869640 4 set_mon_vals no callback set 2024-03-27T21:41:12.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa30effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:12.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa30effd640 10 monclient: dump: 2024-03-27T21:41:12.539 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:12.539 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:12.539 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:12.539 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:12.539 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:12.539 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:12.539 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:12.539 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:12.539 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:12.539 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:12.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa316869640 10 monclient: shutdown 2024-03-27T21:41:12.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa316869640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:12.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa316869640 1 -- 172.21.15.60:0/1217977078 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa3100b40b0 msgr2=0x7fa3100a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:12.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa316869640 1 --2- 172.21.15.60:0/1217977078 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa3100b40b0 0x7fa3100a8b90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa3000097a0 tx=0x7fa30002b840 comp rx=0 tx=0).stop 2024-03-27T21:41:12.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa316869640 1 -- 172.21.15.60:0/1217977078 shutdown_connections 2024-03-27T21:41:12.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa316869640 1 --2- 172.21.15.60:0/1217977078 >> v2:172.21.15.60:3300/0 conn(0x7fa31013b480 0x7fa310141e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:12.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa316869640 1 --2- 172.21.15.60:0/1217977078 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa3100b40b0 0x7fa3100a8b90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:12.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa316869640 1 -- 172.21.15.60:0/1217977078 >> 172.21.15.60:0/1217977078 conn(0x7fa3100608e0 msgr2=0x7fa310060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:12.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa316869640 1 -- 172.21.15.60:0/1217977078 shutdown_connections 2024-03-27T21:41:12.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.532+0000 7fa316869640 1 -- 172.21.15.60:0/1217977078 wait complete. 2024-03-27T21:41:12.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.533+0000 7fa316869640 10 monclient: build_initial_monmap 2024-03-27T21:41:12.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.533+0000 7fa316869640 10 monclient: monmap: 2024-03-27T21:41:12.540 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:12.540 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:12.540 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:12.540 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:12.541 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:12.541 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:12.541 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:12.541 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:12.541 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:12.541 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:12.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.533+0000 7fa316869640 1 librados: starting msgr at 2024-03-27T21:41:12.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.533+0000 7fa316869640 1 librados: starting objecter 2024-03-27T21:41:12.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.533+0000 7fa316869640 1 Processor -- start 2024-03-27T21:41:12.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.533+0000 7fa316869640 1 -- start start 2024-03-27T21:41:12.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.533+0000 7fa316869640 1 librados: setting wanted keys 2024-03-27T21:41:12.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.533+0000 7fa316869640 1 librados: calling monclient init 2024-03-27T21:41:12.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.533+0000 7fa316869640 10 monclient: init 2024-03-27T21:41:12.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa316869640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:12.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa316869640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:41:12.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa316869640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa3100b40b0 0x7fa31013d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:12.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa316869640 10 monclient(hunting): picked mon.noname-c con 0x7fa3100b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:12.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa316869640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa31013b020 0x7fa31013ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:12.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa316869640 10 monclient(hunting): picked mon.noname-a con 0x7fa31013b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:12.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa316869640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa31013b480 0x7fa3101d9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:12.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa316869640 10 monclient(hunting): picked mon.noname-b con 0x7fa31013b480 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:12.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa316869640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:12.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa316869640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31013ed90 con 0x7fa31013b020 2024-03-27T21:41:12.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa316869640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:12.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa316869640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31013e7e0 con 0x7fa31013b480 2024-03-27T21:41:12.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa316869640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:12.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa316869640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa31013e410 con 0x7fa3100b40b0 2024-03-27T21:41:12.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa316869640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:12.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa316869640 10 monclient(hunting): authenticate will time out at 913.755442s 2024-03-27T21:41:12.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30ffff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa3100b40b0 0x7fa31013d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:12.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa31013b020 0x7fa31013ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:12.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa31013b020 0x7fa31013ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37224/0 (socket says 172.21.15.60:37224) 2024-03-27T21:41:12.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30ffff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa3100b40b0 0x7fa31013d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:37806/0 (socket says 172.21.15.60:37806) 2024-03-27T21:41:12.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30f7fe640 1 -- 172.21.15.60:0/293553658 learned_addr learned my addr 172.21.15.60:0/293553658 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:12.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30f7fe640 10 monclient(hunting): get_auth_request con 0x7fa31013b020 auth_method 0 2024-03-27T21:41:12.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:12.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:12.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:12.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30ffff640 10 monclient(hunting): get_auth_request con 0x7fa3100b40b0 auth_method 0 2024-03-27T21:41:12.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:12.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:12.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:12.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:12.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:12.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:12.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:12.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:12.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:12.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa314e06640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa31013b480 0x7fa3101d9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:12.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa314e06640 10 monclient(hunting): get_auth_request con 0x7fa31013b480 auth_method 0 2024-03-27T21:41:12.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa314e06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:12.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa314e06640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:12.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa314e06640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:12.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30ffff640 10 monclient(hunting): handle_auth_done global_id 4730 payload 824 2024-03-27T21:41:12.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30ffff640 1 -- 172.21.15.60:0/293553658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa31013b480 msgr2=0x7fa3101d9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:12.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30ffff640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa31013b480 0x7fa3101d9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:12.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30ffff640 1 -- 172.21.15.60:0/293553658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa31013b020 msgr2=0x7fa31013ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:12.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30ffff640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa31013b020 0x7fa31013ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:12.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30ffff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:12.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30ffff640 1 monclient: found mon.noname-c 2024-03-27T21:41:12.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:12.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:12.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30ffff640 1 -- 172.21.15.60:0/293553658 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa300009400 con 0x7fa3100b40b0 2024-03-27T21:41:12.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.534+0000 7fa30f7fe640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa31013b020 0x7fa31013ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:12.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa30ffff640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa3100b40b0 0x7fa31013d940 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa3100b8060 tx=0x7fa3000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:12.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa30cff9640 10 client.?.objecter ms_handle_connect 0x7fa3100b40b0 2024-03-27T21:41:12.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa30cff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:12.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa30cff9640 1 -- 172.21.15.60:0/293553658 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa300032840 con 0x7fa3100b40b0 2024-03-27T21:41:12.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa30cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:12.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa30cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:12.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa30cff9640 10 monclient: dump: 2024-03-27T21:41:12.546 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:12.546 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:12.546 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:12.546 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:12.546 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:12.546 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:12.546 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:12.546 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:12.547 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:12.547 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:12.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa30cff9640 10 monclient: _finish_auth 0 2024-03-27T21:41:12.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa30cff9640 10 monclient: _check_auth_tickets 2024-03-27T21:41:12.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa30cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:12.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa30cff9640 1 -- 172.21.15.60:0/293553658 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa300031f00 con 0x7fa3100b40b0 2024-03-27T21:41:12.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa316869640 5 monclient: authenticate success, global_id 4730 2024-03-27T21:41:12.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa316869640 10 monclient: _renew_subs 2024-03-27T21:41:12.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa316869640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:12.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa316869640 1 -- 172.21.15.60:0/293553658 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa31013a790 con 0x7fa3100b40b0 2024-03-27T21:41:12.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa30cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:12.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa30cff9640 1 -- 172.21.15.60:0/293553658 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa300032c00 con 0x7fa3100b40b0 2024-03-27T21:41:12.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa30cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:12.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa30cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:12.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa30cff9640 10 monclient: dump: 2024-03-27T21:41:12.548 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:12.548 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:12.548 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:12.548 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:12.548 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:12.548 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:12.548 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:12.548 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:12.548 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:12.548 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:12.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa316869640 10 client.4730.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:12.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa316869640 10 monclient: _renew_subs 2024-03-27T21:41:12.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa316869640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:12.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa316869640 1 -- 172.21.15.60:0/293553658 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3101da150 con 0x7fa3100b40b0 2024-03-27T21:41:12.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa316869640 1 librados: init done 2024-03-27T21:41:12.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.535+0000 7fa30e7fc640 4 set_mon_vals no callback set 2024-03-27T21:41:12.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.536+0000 7fa316869640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:41:12.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.536+0000 7fa316869640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:12.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.536+0000 7fa316869640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:12.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.536+0000 7fa316869640 1 -- 172.21.15.60:0/293553658 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fa31013e9a0 con 0x7fa3100b40b0 2024-03-27T21:41:12.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.536+0000 7fa30cff9640 1 -- 172.21.15.60:0/293553658 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa30004e430 con 0x7fa3100b40b0 2024-03-27T21:41:12.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa30cff9640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa2fc06c590 0x7fa2fc06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:12.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa30cff9640 1 -- 172.21.15.60:0/293553658 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fa3000aafa0 con 0x7fa3100b40b0 2024-03-27T21:41:12.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa30cff9640 10 client.4730.objecter ms_dispatch 0x7fa310000b90 osd_map(53..53 src has 1..53) v4 2024-03-27T21:41:12.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa30cff9640 3 client.4730.objecter handle_osd_map got epochs [53,53] > 0 2024-03-27T21:41:12.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa30cff9640 3 client.4730.objecter handle_osd_map decoding full epoch 53 2024-03-27T21:41:12.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa30f7fe640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa2fc06c590 0x7fa2fc06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:12.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa30cff9640 20 client.4730.objecter dump_active .. 0 homeless 2024-03-27T21:41:12.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa30f7fe640 10 monclient: get_auth_request con 0x7fa2fc06c590 auth_method 0 2024-03-27T21:41:12.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa30cff9640 1 -- 172.21.15.60:0/293553658 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v53) v1 ==== 54+0+16 (secure 0 0 0) 0x7fa3000ad580 con 0x7fa3100b40b0 2024-03-27T21:41:12.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa30cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:12.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa30cff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:12.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa316869640 10 client.4730.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:12.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign -1 1 2024-03-27T21:41:12.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa316869640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fa3100b04b0 0x7fa31013ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:12.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa31013a920 osd=0 3 2024-03-27T21:41:12.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa31013a920 osd=0 3 2024-03-27T21:41:12.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa31013a920 osd=0 3 2024-03-27T21:41:12.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa316869640 15 client.4730.objecter _session_command_op_remove -1 1 2024-03-27T21:41:12.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa316869640 20 client.4730.objecter get_session s=0x7fa31013a920 osd=0 3 2024-03-27T21:41:12.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign 0 1 2024-03-27T21:41:12.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa31013a920 osd=0 4 2024-03-27T21:41:12.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa316869640 10 client.4730.objecter _send_command 1 2024-03-27T21:41:12.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa316869640 1 -- 172.21.15.60:0/293553658 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3100a90d0 con 0x7fa3100b04b0 2024-03-27T21:41:12.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa314e06640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fa3100b04b0 0x7fa31013ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:12.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa30cff9640 10 client.4730.objecter ms_handle_connect 0x7fa2fc06c590 2024-03-27T21:41:12.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa314e06640 10 monclient: get_auth_request con 0x7fa3100b04b0 auth_method 0 2024-03-27T21:41:12.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.537+0000 7fa30f7fe640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa2fc06c590 0x7fa2fc06ea50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa310141a40 tx=0x7fa304007f00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:12.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.538+0000 7fa30cff9640 10 client.4730.objecter ms_handle_connect 0x7fa3100b04b0 2024-03-27T21:41:12.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.538+0000 7fa314e06640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fa3100b04b0 0x7fa31013ba60 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:12.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.538+0000 7fa30cff9640 1 -- 172.21.15.60:0/293553658 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fa30800f040 con 0x7fa3100b04b0 2024-03-27T21:41:12.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.538+0000 7fa30cff9640 10 client.4730.objecter ms_dispatch 0x7fa310000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:41:12.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.538+0000 7fa30cff9640 10 client.4730.objecter _finish_command 1 = system:0 2024-03-27T21:41:12.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.538+0000 7fa30cff9640 20 client.4730.objecter put_session s=0x7fa31013a920 osd=0 4 2024-03-27T21:41:12.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.538+0000 7fa30cff9640 15 client.4730.objecter _session_command_op_remove 0 1 2024-03-27T21:41:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.553+0000 7fa316869640 10 client.4730.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.553+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign -1 2 2024-03-27T21:41:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.553+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa31013a920 osd=0 3 2024-03-27T21:41:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.553+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa31013a920 osd=0 3 2024-03-27T21:41:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.553+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa31013a920 osd=0 3 2024-03-27T21:41:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.553+0000 7fa316869640 15 client.4730.objecter _session_command_op_remove -1 2 2024-03-27T21:41:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.553+0000 7fa316869640 20 client.4730.objecter get_session s=0x7fa31013a920 osd=0 3 2024-03-27T21:41:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.553+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign 0 2 2024-03-27T21:41:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.553+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa31013a920 osd=0 4 2024-03-27T21:41:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.553+0000 7fa316869640 10 client.4730.objecter _send_command 2 2024-03-27T21:41:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.553+0000 7fa316869640 1 -- 172.21.15.60:0/293553658 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa310142340 con 0x7fa3100b04b0 2024-03-27T21:41:12.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.555+0000 7fa30cff9640 1 -- 172.21.15.60:0/293553658 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa308009d40 con 0x7fa3100b04b0 2024-03-27T21:41:12.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.555+0000 7fa30cff9640 10 client.4730.objecter ms_dispatch 0x7fa310000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:12.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.555+0000 7fa30cff9640 10 client.4730.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:12.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.555+0000 7fa30cff9640 20 client.4730.objecter put_session s=0x7fa31013a920 osd=0 4 2024-03-27T21:41:12.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.555+0000 7fa30cff9640 15 client.4730.objecter _session_command_op_remove 0 2 2024-03-27T21:41:12.554 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:41:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:41:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.555+0000 7fa316869640 10 client.4730.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.555+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign -1 3 2024-03-27T21:41:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.555+0000 7fa316869640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fa3100b0890 0x7fa31013c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.555+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa31014bce0 osd=1 3 2024-03-27T21:41:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.555+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa31014bce0 osd=1 3 2024-03-27T21:41:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.555+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa31014bce0 osd=1 3 2024-03-27T21:41:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.555+0000 7fa316869640 15 client.4730.objecter _session_command_op_remove -1 3 2024-03-27T21:41:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.555+0000 7fa316869640 20 client.4730.objecter get_session s=0x7fa31014bce0 osd=1 3 2024-03-27T21:41:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.555+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign 1 3 2024-03-27T21:41:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.555+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa31014bce0 osd=1 4 2024-03-27T21:41:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.555+0000 7fa316869640 10 client.4730.objecter _send_command 3 2024-03-27T21:41:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.555+0000 7fa316869640 1 -- 172.21.15.60:0/293553658 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa310069470 con 0x7fa3100b0890 2024-03-27T21:41:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.555+0000 7fa30ffff640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fa3100b0890 0x7fa31013c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.555+0000 7fa30ffff640 10 monclient: get_auth_request con 0x7fa3100b0890 auth_method 0 2024-03-27T21:41:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.555+0000 7fa30cff9640 10 client.4730.objecter ms_handle_connect 0x7fa3100b0890 2024-03-27T21:41:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.555+0000 7fa30ffff640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fa3100b0890 0x7fa31013c4b0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.556+0000 7fa30cff9640 1 -- 172.21.15.60:0/293553658 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fa310069470 con 0x7fa3100b0890 2024-03-27T21:41:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.556+0000 7fa30cff9640 10 client.4730.objecter ms_dispatch 0x7fa310000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:41:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.556+0000 7fa30cff9640 10 client.4730.objecter _finish_command 3 = system:0 2024-03-27T21:41:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.556+0000 7fa30cff9640 20 client.4730.objecter put_session s=0x7fa31014bce0 osd=1 4 2024-03-27T21:41:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.556+0000 7fa30cff9640 15 client.4730.objecter _session_command_op_remove 1 3 2024-03-27T21:41:12.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.572+0000 7fa316869640 10 client.4730.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:12.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.572+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign -1 4 2024-03-27T21:41:12.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.572+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa31014bce0 osd=1 3 2024-03-27T21:41:12.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.572+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa31014bce0 osd=1 3 2024-03-27T21:41:12.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.572+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa31014bce0 osd=1 3 2024-03-27T21:41:12.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.572+0000 7fa316869640 15 client.4730.objecter _session_command_op_remove -1 4 2024-03-27T21:41:12.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.572+0000 7fa316869640 20 client.4730.objecter get_session s=0x7fa31014bce0 osd=1 3 2024-03-27T21:41:12.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.572+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign 1 4 2024-03-27T21:41:12.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.572+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa31014bce0 osd=1 4 2024-03-27T21:41:12.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.572+0000 7fa316869640 10 client.4730.objecter _send_command 4 2024-03-27T21:41:12.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.572+0000 7fa316869640 1 -- 172.21.15.60:0/293553658 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa310065a90 con 0x7fa3100b0890 2024-03-27T21:41:12.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa30cff9640 1 -- 172.21.15.60:0/293553658 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa310065a90 con 0x7fa3100b0890 2024-03-27T21:41:12.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa30cff9640 10 client.4730.objecter ms_dispatch 0x7fa310000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:12.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa30cff9640 10 client.4730.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:12.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa30cff9640 20 client.4730.objecter put_session s=0x7fa31014bce0 osd=1 4 2024-03-27T21:41:12.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa30cff9640 15 client.4730.objecter _session_command_op_remove 1 4 2024-03-27T21:41:12.560 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:41:12.562 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:41:12.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa316869640 10 client.4730.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:12.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign -1 5 2024-03-27T21:41:12.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa316869640 20 client.4730.objecter _get_session osd=-1 returning homeless 2024-03-27T21:41:12.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa316869640 10 client.4730.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:12.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa316869640 10 monclient: _renew_subs 2024-03-27T21:41:12.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa316869640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:12.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa316869640 1 -- 172.21.15.60:0/293553658 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=54}) v3 -- 0x7fa31013fb10 con 0x7fa3100b40b0 2024-03-27T21:41:12.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa316869640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:12.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa316869640 1 -- 172.21.15.60:0/293553658 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa31013fdb0 con 0x7fa3100b40b0 2024-03-27T21:41:12.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa30cff9640 1 -- 172.21.15.60:0/293553658 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3000ad560 con 0x7fa3100b40b0 2024-03-27T21:41:12.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa30cff9640 10 monclient: handle_get_version_reply finishing 1 version 53 2024-03-27T21:41:12.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa30e7fc640 10 client.4730.objecter _check_command_map_dne tid 5 current 53 map_dne_bound 53 2024-03-27T21:41:12.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa30e7fc640 10 client.4730.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:41:12.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa30e7fc640 15 client.4730.objecter _session_command_op_remove -1 5 2024-03-27T21:41:12.564 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:41:12.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa316869640 10 client.4730.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:12.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign -1 6 2024-03-27T21:41:12.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa316869640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fa31002b170 0x7fa31002b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:12.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa31013fb10 osd=3 3 2024-03-27T21:41:12.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa31013fb10 osd=3 3 2024-03-27T21:41:12.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa31013fb10 osd=3 3 2024-03-27T21:41:12.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa316869640 15 client.4730.objecter _session_command_op_remove -1 6 2024-03-27T21:41:12.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa316869640 20 client.4730.objecter get_session s=0x7fa31013fb10 osd=3 3 2024-03-27T21:41:12.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign 3 6 2024-03-27T21:41:12.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa31013fb10 osd=3 4 2024-03-27T21:41:12.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa316869640 10 client.4730.objecter _send_command 6 2024-03-27T21:41:12.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa316869640 1 -- 172.21.15.60:0/293553658 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa31002ba90 con 0x7fa31002b170 2024-03-27T21:41:12.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa30f7fe640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fa31002b170 0x7fa31002b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:12.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.574+0000 7fa30f7fe640 10 monclient: get_auth_request con 0x7fa31002b170 auth_method 0 2024-03-27T21:41:12.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.575+0000 7fa30cff9640 10 client.4730.objecter ms_handle_connect 0x7fa31002b170 2024-03-27T21:41:12.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.575+0000 7fa30f7fe640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fa31002b170 0x7fa31002b550 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:12.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.575+0000 7fa30cff9640 1 -- 172.21.15.60:0/293553658 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fa31002ba90 con 0x7fa31002b170 2024-03-27T21:41:12.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.575+0000 7fa30cff9640 10 client.4730.objecter ms_dispatch 0x7fa310000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:41:12.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.575+0000 7fa30cff9640 10 client.4730.objecter _finish_command 6 = system:0 2024-03-27T21:41:12.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.575+0000 7fa30cff9640 20 client.4730.objecter put_session s=0x7fa31013fb10 osd=3 4 2024-03-27T21:41:12.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.575+0000 7fa30cff9640 15 client.4730.objecter _session_command_op_remove 3 6 2024-03-27T21:41:12.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.592+0000 7fa316869640 10 client.4730.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:12.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.592+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign -1 7 2024-03-27T21:41:12.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.592+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa31013fb10 osd=3 3 2024-03-27T21:41:12.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.592+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa31013fb10 osd=3 3 2024-03-27T21:41:12.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.592+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa31013fb10 osd=3 3 2024-03-27T21:41:12.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.592+0000 7fa316869640 15 client.4730.objecter _session_command_op_remove -1 7 2024-03-27T21:41:12.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.592+0000 7fa316869640 20 client.4730.objecter get_session s=0x7fa31013fb10 osd=3 3 2024-03-27T21:41:12.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.592+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign 3 7 2024-03-27T21:41:12.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.592+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa31013fb10 osd=3 4 2024-03-27T21:41:12.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.592+0000 7fa316869640 10 client.4730.objecter _send_command 7 2024-03-27T21:41:12.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.592+0000 7fa316869640 1 -- 172.21.15.60:0/293553658 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa31002c450 con 0x7fa31002b170 2024-03-27T21:41:12.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.593+0000 7fa30cff9640 1 -- 172.21.15.60:0/293553658 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa31002c450 con 0x7fa31002b170 2024-03-27T21:41:12.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.593+0000 7fa30cff9640 10 client.4730.objecter ms_dispatch 0x7fa310000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:12.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.593+0000 7fa30cff9640 10 client.4730.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:12.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.593+0000 7fa30cff9640 20 client.4730.objecter put_session s=0x7fa31013fb10 osd=3 4 2024-03-27T21:41:12.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.593+0000 7fa30cff9640 15 client.4730.objecter _session_command_op_remove 3 7 2024-03-27T21:41:12.580 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:41:12.581 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:41:12.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.593+0000 7fa316869640 10 client.4730.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:12.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.593+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign -1 8 2024-03-27T21:41:12.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.593+0000 7fa316869640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fa310066250 0x7fa3101e92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:12.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.593+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa3100660e0 osd=4 3 2024-03-27T21:41:12.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.593+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa3100660e0 osd=4 3 2024-03-27T21:41:12.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.593+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa3100660e0 osd=4 3 2024-03-27T21:41:12.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.593+0000 7fa316869640 15 client.4730.objecter _session_command_op_remove -1 8 2024-03-27T21:41:12.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.593+0000 7fa316869640 20 client.4730.objecter get_session s=0x7fa3100660e0 osd=4 3 2024-03-27T21:41:12.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.593+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign 4 8 2024-03-27T21:41:12.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.593+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa3100660e0 osd=4 4 2024-03-27T21:41:12.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.593+0000 7fa316869640 10 client.4730.objecter _send_command 8 2024-03-27T21:41:12.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.593+0000 7fa316869640 1 -- 172.21.15.60:0/293553658 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3101ec080 con 0x7fa310066250 2024-03-27T21:41:12.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.594+0000 7fa314e06640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fa310066250 0x7fa3101e92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:12.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.594+0000 7fa314e06640 10 monclient: get_auth_request con 0x7fa310066250 auth_method 0 2024-03-27T21:41:12.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.594+0000 7fa314e06640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fa310066250 0x7fa3101e92b0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:12.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.594+0000 7fa30cff9640 10 client.4730.objecter ms_handle_connect 0x7fa310066250 2024-03-27T21:41:12.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.595+0000 7fa30cff9640 1 -- 172.21.15.60:0/293553658 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fa3101ec080 con 0x7fa310066250 2024-03-27T21:41:12.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.595+0000 7fa30cff9640 10 client.4730.objecter ms_dispatch 0x7fa310000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:41:12.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.595+0000 7fa30cff9640 10 client.4730.objecter _finish_command 8 = system:0 2024-03-27T21:41:12.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.595+0000 7fa30cff9640 20 client.4730.objecter put_session s=0x7fa3100660e0 osd=4 4 2024-03-27T21:41:12.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.595+0000 7fa30cff9640 15 client.4730.objecter _session_command_op_remove 4 8 2024-03-27T21:41:12.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.610+0000 7fa316869640 10 client.4730.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:12.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.610+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign -1 9 2024-03-27T21:41:12.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.610+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa3100660e0 osd=4 3 2024-03-27T21:41:12.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.610+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa3100660e0 osd=4 3 2024-03-27T21:41:12.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.610+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa3100660e0 osd=4 3 2024-03-27T21:41:12.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.610+0000 7fa316869640 15 client.4730.objecter _session_command_op_remove -1 9 2024-03-27T21:41:12.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.610+0000 7fa316869640 20 client.4730.objecter get_session s=0x7fa3100660e0 osd=4 3 2024-03-27T21:41:12.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.610+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign 4 9 2024-03-27T21:41:12.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.610+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa3100660e0 osd=4 4 2024-03-27T21:41:12.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.610+0000 7fa316869640 10 client.4730.objecter _send_command 9 2024-03-27T21:41:12.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.610+0000 7fa316869640 1 -- 172.21.15.60:0/293553658 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa31002bed0 con 0x7fa310066250 2024-03-27T21:41:12.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.611+0000 7fa30cff9640 1 -- 172.21.15.60:0/293553658 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa31002bed0 con 0x7fa310066250 2024-03-27T21:41:12.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.611+0000 7fa30cff9640 10 client.4730.objecter ms_dispatch 0x7fa310000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:12.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.611+0000 7fa30cff9640 10 client.4730.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:12.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.611+0000 7fa30cff9640 20 client.4730.objecter put_session s=0x7fa3100660e0 osd=4 4 2024-03-27T21:41:12.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.611+0000 7fa30cff9640 15 client.4730.objecter _session_command_op_remove 4 9 2024-03-27T21:41:12.598 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:41:12.599 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:41:12.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.611+0000 7fa316869640 10 client.4730.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:12.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.611+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign -1 10 2024-03-27T21:41:12.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.611+0000 7fa316869640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fa31002c040 0x7fa3101ea970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:12.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.611+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa31002bed0 osd=5 3 2024-03-27T21:41:12.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.611+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa31002bed0 osd=5 3 2024-03-27T21:41:12.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.611+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa31002bed0 osd=5 3 2024-03-27T21:41:12.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.611+0000 7fa316869640 15 client.4730.objecter _session_command_op_remove -1 10 2024-03-27T21:41:12.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.611+0000 7fa316869640 20 client.4730.objecter get_session s=0x7fa31002bed0 osd=5 3 2024-03-27T21:41:12.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.611+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign 5 10 2024-03-27T21:41:12.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.611+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa31002bed0 osd=5 4 2024-03-27T21:41:12.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.611+0000 7fa316869640 10 client.4730.objecter _send_command 10 2024-03-27T21:41:12.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.611+0000 7fa316869640 1 -- 172.21.15.60:0/293553658 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3100b2620 con 0x7fa31002c040 2024-03-27T21:41:12.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.612+0000 7fa30ffff640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fa31002c040 0x7fa3101ea970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:12.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.612+0000 7fa30ffff640 10 monclient: get_auth_request con 0x7fa31002c040 auth_method 0 2024-03-27T21:41:12.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.612+0000 7fa30cff9640 10 client.4730.objecter ms_handle_connect 0x7fa31002c040 2024-03-27T21:41:12.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.612+0000 7fa30ffff640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fa31002c040 0x7fa3101ea970 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:12.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.613+0000 7fa30cff9640 1 -- 172.21.15.60:0/293553658 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fa3100b2620 con 0x7fa31002c040 2024-03-27T21:41:12.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.613+0000 7fa30cff9640 10 client.4730.objecter ms_dispatch 0x7fa310000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:41:12.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.613+0000 7fa30cff9640 10 client.4730.objecter _finish_command 10 = system:0 2024-03-27T21:41:12.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.613+0000 7fa30cff9640 20 client.4730.objecter put_session s=0x7fa31002bed0 osd=5 4 2024-03-27T21:41:12.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.613+0000 7fa30cff9640 15 client.4730.objecter _session_command_op_remove 5 10 2024-03-27T21:41:12.605 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:12.619+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:12.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.626+0000 7fa316869640 10 client.4730.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:12.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.626+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign -1 11 2024-03-27T21:41:12.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.626+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa31002bed0 osd=5 3 2024-03-27T21:41:12.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.626+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa31002bed0 osd=5 3 2024-03-27T21:41:12.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.626+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa31002bed0 osd=5 3 2024-03-27T21:41:12.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.626+0000 7fa316869640 15 client.4730.objecter _session_command_op_remove -1 11 2024-03-27T21:41:12.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.626+0000 7fa316869640 20 client.4730.objecter get_session s=0x7fa31002bed0 osd=5 3 2024-03-27T21:41:12.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.626+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign 5 11 2024-03-27T21:41:12.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.626+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa31002bed0 osd=5 4 2024-03-27T21:41:12.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.626+0000 7fa316869640 10 client.4730.objecter _send_command 11 2024-03-27T21:41:12.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.626+0000 7fa316869640 1 -- 172.21.15.60:0/293553658 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa3100b2620 con 0x7fa31002c040 2024-03-27T21:41:12.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.628+0000 7fa30cff9640 1 -- 172.21.15.60:0/293553658 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa3100b2620 con 0x7fa31002c040 2024-03-27T21:41:12.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.628+0000 7fa30cff9640 10 client.4730.objecter ms_dispatch 0x7fa310000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:12.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.628+0000 7fa30cff9640 10 client.4730.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:12.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.628+0000 7fa30cff9640 20 client.4730.objecter put_session s=0x7fa31002bed0 osd=5 4 2024-03-27T21:41:12.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.628+0000 7fa30cff9640 15 client.4730.objecter _session_command_op_remove 5 11 2024-03-27T21:41:12.615 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:41:12.616 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:41:12.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.628+0000 7fa316869640 10 client.4730.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:12.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.628+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign -1 12 2024-03-27T21:41:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.628+0000 7fa316869640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fa3101eb3c0 0x7fa3101eb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.628+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa3100b2620 osd=6 3 2024-03-27T21:41:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.628+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa3100b2620 osd=6 3 2024-03-27T21:41:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.628+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa3100b2620 osd=6 3 2024-03-27T21:41:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.628+0000 7fa316869640 15 client.4730.objecter _session_command_op_remove -1 12 2024-03-27T21:41:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.628+0000 7fa316869640 20 client.4730.objecter get_session s=0x7fa3100b2620 osd=6 3 2024-03-27T21:41:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.628+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign 6 12 2024-03-27T21:41:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.628+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa3100b2620 osd=6 4 2024-03-27T21:41:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.628+0000 7fa316869640 10 client.4730.objecter _send_command 12 2024-03-27T21:41:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.628+0000 7fa316869640 1 -- 172.21.15.60:0/293553658 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3101ebd80 con 0x7fa3101eb3c0 2024-03-27T21:41:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.628+0000 7fa30f7fe640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fa3101eb3c0 0x7fa3101eb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.628+0000 7fa30f7fe640 10 monclient: get_auth_request con 0x7fa3101eb3c0 auth_method 0 2024-03-27T21:41:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.629+0000 7fa30cff9640 10 client.4730.objecter ms_handle_connect 0x7fa3101eb3c0 2024-03-27T21:41:12.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.629+0000 7fa30f7fe640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fa3101eb3c0 0x7fa3101eb840 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:12.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.629+0000 7fa30cff9640 1 -- 172.21.15.60:0/293553658 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fa3101ebd80 con 0x7fa3101eb3c0 2024-03-27T21:41:12.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.629+0000 7fa30cff9640 10 client.4730.objecter ms_dispatch 0x7fa310000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:41:12.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.629+0000 7fa30cff9640 10 client.4730.objecter _finish_command 12 = system:0 2024-03-27T21:41:12.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.629+0000 7fa30cff9640 20 client.4730.objecter put_session s=0x7fa3100b2620 osd=6 4 2024-03-27T21:41:12.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.629+0000 7fa30cff9640 15 client.4730.objecter _session_command_op_remove 6 12 2024-03-27T21:41:12.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.644+0000 7fa316869640 10 client.4730.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:12.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.644+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign -1 13 2024-03-27T21:41:12.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.644+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa3100b2620 osd=6 3 2024-03-27T21:41:12.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.644+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa3100b2620 osd=6 3 2024-03-27T21:41:12.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.644+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa3100b2620 osd=6 3 2024-03-27T21:41:12.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.644+0000 7fa316869640 15 client.4730.objecter _session_command_op_remove -1 13 2024-03-27T21:41:12.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.644+0000 7fa316869640 20 client.4730.objecter get_session s=0x7fa3100b2620 osd=6 3 2024-03-27T21:41:12.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.644+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign 6 13 2024-03-27T21:41:12.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.644+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa3100b2620 osd=6 4 2024-03-27T21:41:12.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.644+0000 7fa316869640 10 client.4730.objecter _send_command 13 2024-03-27T21:41:12.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.644+0000 7fa316869640 1 -- 172.21.15.60:0/293553658 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa3101ebd80 con 0x7fa3101eb3c0 2024-03-27T21:41:12.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.646+0000 7fa30cff9640 1 -- 172.21.15.60:0/293553658 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa3101ebd80 con 0x7fa3101eb3c0 2024-03-27T21:41:12.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.646+0000 7fa30cff9640 10 client.4730.objecter ms_dispatch 0x7fa310000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:12.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.646+0000 7fa30cff9640 10 client.4730.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:12.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.646+0000 7fa30cff9640 20 client.4730.objecter put_session s=0x7fa3100b2620 osd=6 4 2024-03-27T21:41:12.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.646+0000 7fa30cff9640 15 client.4730.objecter _session_command_op_remove 6 13 2024-03-27T21:41:12.632 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:41:12.633 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:41:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.646+0000 7fa316869640 10 client.4730.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.646+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign -1 14 2024-03-27T21:41:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.646+0000 7fa316869640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fa3101f0720 0x7fa3101f2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.646+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa3101ebd80 osd=7 3 2024-03-27T21:41:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.646+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa3101ebd80 osd=7 3 2024-03-27T21:41:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.646+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa3101ebd80 osd=7 3 2024-03-27T21:41:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.646+0000 7fa316869640 15 client.4730.objecter _session_command_op_remove -1 14 2024-03-27T21:41:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.646+0000 7fa316869640 20 client.4730.objecter get_session s=0x7fa3101ebd80 osd=7 3 2024-03-27T21:41:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.646+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign 7 14 2024-03-27T21:41:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.646+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa3101ebd80 osd=7 4 2024-03-27T21:41:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.646+0000 7fa316869640 10 client.4730.objecter _send_command 14 2024-03-27T21:41:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.646+0000 7fa316869640 1 -- 172.21.15.60:0/293553658 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3101f58e0 con 0x7fa3101f0720 2024-03-27T21:41:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.646+0000 7fa314e06640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fa3101f0720 0x7fa3101f2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:12.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.646+0000 7fa314e06640 10 monclient: get_auth_request con 0x7fa3101f0720 auth_method 0 2024-03-27T21:41:12.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.646+0000 7fa30cff9640 10 client.4730.objecter ms_handle_connect 0x7fa3101f0720 2024-03-27T21:41:12.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.646+0000 7fa314e06640 1 --2- 172.21.15.60:0/293553658 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fa3101f0720 0x7fa3101f2b10 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:12.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.647+0000 7fa30cff9640 1 -- 172.21.15.60:0/293553658 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fa3101f58e0 con 0x7fa3101f0720 2024-03-27T21:41:12.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.647+0000 7fa30cff9640 10 client.4730.objecter ms_dispatch 0x7fa310000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:41:12.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.647+0000 7fa30cff9640 10 client.4730.objecter _finish_command 14 = system:0 2024-03-27T21:41:12.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.647+0000 7fa30cff9640 20 client.4730.objecter put_session s=0x7fa3101ebd80 osd=7 4 2024-03-27T21:41:12.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.647+0000 7fa30cff9640 15 client.4730.objecter _session_command_op_remove 7 14 2024-03-27T21:41:12.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.651+0000 7fb443bf6640 10 start_mon_command cmd=[{"prefix": "osd out", "ids": ["1"]}] 2024-03-27T21:41:12.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.651+0000 7fb443bf6640 10 monclient: _send_command 2 [{"prefix": "osd out", "ids": ["1"]}] 2024-03-27T21:41:12.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.651+0000 7fb443bf6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:12.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.651+0000 7fb443bf6640 1 -- 172.21.15.60:0/719544222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd out", "ids": ["1"]} v 0) v1 -- 0x7fb43c069e40 con 0x7fb43c0b40b0 2024-03-27T21:41:12.638 INFO:tasks.ceph.mon.a.smithi060.stderr:2024-03-27T21:41:12.651+0000 7ff34bd31640 -1 mon.a@0(leader).osd e53 definitely_dead 0 2024-03-27T21:41:12.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.662+0000 7fa316869640 10 client.4730.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:12.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.662+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign -1 15 2024-03-27T21:41:12.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.662+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa3101ebd80 osd=7 3 2024-03-27T21:41:12.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.662+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa3101ebd80 osd=7 3 2024-03-27T21:41:12.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.662+0000 7fa316869640 20 client.4730.objecter _get_session s=0x7fa3101ebd80 osd=7 3 2024-03-27T21:41:12.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.662+0000 7fa316869640 15 client.4730.objecter _session_command_op_remove -1 15 2024-03-27T21:41:12.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.662+0000 7fa316869640 20 client.4730.objecter get_session s=0x7fa3101ebd80 osd=7 3 2024-03-27T21:41:12.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.662+0000 7fa316869640 15 client.4730.objecter _session_command_op_assign 7 15 2024-03-27T21:41:12.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.662+0000 7fa316869640 20 client.4730.objecter put_session s=0x7fa3101ebd80 osd=7 4 2024-03-27T21:41:12.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.662+0000 7fa316869640 10 client.4730.objecter _send_command 15 2024-03-27T21:41:12.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.662+0000 7fa316869640 1 -- 172.21.15.60:0/293553658 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa3101f3050 con 0x7fa3101f0720 2024-03-27T21:41:12.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.663+0000 7fa30cff9640 1 -- 172.21.15.60:0/293553658 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa3101f3050 con 0x7fa3101f0720 2024-03-27T21:41:12.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.663+0000 7fa30cff9640 10 client.4730.objecter ms_dispatch 0x7fa310000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:12.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.663+0000 7fa30cff9640 10 client.4730.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:12.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.663+0000 7fa30cff9640 20 client.4730.objecter put_session s=0x7fa3101ebd80 osd=7 4 2024-03-27T21:41:12.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.663+0000 7fa30cff9640 15 client.4730.objecter _session_command_op_remove 7 15 2024-03-27T21:41:12.650 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:41:12.650 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:41:12.652 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:41:12.653 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:41:12.658 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:12.658 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:12.667 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:41:12.706 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:12.720+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:12.807 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:12.821+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:12.812 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:12.812 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:12.821 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:41:12.908 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:12.922+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:12.966 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:12.966 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:12.974 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:41:12.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.992+0000 7fb42e7fc640 1 -- 172.21.15.60:0/719544222 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd out", "ids": ["1"]}]=0 marked out osd.1. v54) v1 ==== 87+0+0 (secure 0 0 0) 0x7fb4380575f0 con 0x7fb43c0b40b0 2024-03-27T21:41:12.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.992+0000 7fb42e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd out", "ids": ["1"]}] 2024-03-27T21:41:12.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.992+0000 7fb42e7fc640 10 monclient: _finish_command 2 = system:0 marked out osd.1. 2024-03-27T21:41:12.978 INFO:teuthology.orchestra.run.smithi060.stderr:marked out osd.1. 2024-03-27T21:41:12.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.994+0000 7fb443bf6640 10 librados: watch_flush enter 2024-03-27T21:41:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.995+0000 7fb443bf6640 10 librados: watch_flush exit 2024-03-27T21:41:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.995+0000 7fb443bf6640 20 client.4899.objecter shutdown clearing up homeless session... 2024-03-27T21:41:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.995+0000 7fb443bf6640 10 client.4899.objecter successfully canceled tick 2024-03-27T21:41:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.995+0000 7fb443bf6640 1 -- 172.21.15.60:0/719544222 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb42006c590 msgr2=0x7fb42006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.995+0000 7fb443bf6640 1 --2- 172.21.15.60:0/719544222 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb42006c590 0x7fb42006ea50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb430036000 tx=0x7fb430008030 comp rx=0 tx=0).stop 2024-03-27T21:41:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.995+0000 7fb443bf6640 10 monclient: shutdown 2024-03-27T21:41:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.995+0000 7fb443bf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.995+0000 7fb443bf6640 1 -- 172.21.15.60:0/719544222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb43c0b40b0 msgr2=0x7fb43c13e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.995+0000 7fb443bf6640 1 --2- 172.21.15.60:0/719544222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb43c0b40b0 0x7fb43c13e110 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb43800c450 tx=0x7fb43800c910 comp rx=0 tx=0).stop 2024-03-27T21:41:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.995+0000 7fb443bf6640 1 -- 172.21.15.60:0/719544222 shutdown_connections 2024-03-27T21:41:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.995+0000 7fb443bf6640 1 --2- 172.21.15.60:0/719544222 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb43c1416f0 0x7fb43c133230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.995+0000 7fb443bf6640 1 --2- 172.21.15.60:0/719544222 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb42006c590 0x7fb42006ea50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.995+0000 7fb443bf6640 1 --2- 172.21.15.60:0/719544222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb43c13b020 0x7fb43c139d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:12.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.995+0000 7fb443bf6640 1 --2- 172.21.15.60:0/719544222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb43c0b40b0 0x7fb43c13e110 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:12.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.995+0000 7fb443bf6640 1 -- 172.21.15.60:0/719544222 >> 172.21.15.60:0/719544222 conn(0x7fb43c0608e0 msgr2=0x7fb43c0b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:12.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.995+0000 7fb443bf6640 1 -- 172.21.15.60:0/719544222 shutdown_connections 2024-03-27T21:41:12.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.995+0000 7fb443bf6640 1 -- 172.21.15.60:0/719544222 wait complete. 2024-03-27T21:41:12.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:12.995+0000 7fb443bf6640 1 librados: shutdown 2024-03-27T21:41:12.983 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-27T21:41:13.034 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:13.048+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:13.080 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:13.080 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.100+0000 7f277b85f640 10 monclient: get_monmap_and_config 2024-03-27T21:41:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.100+0000 7f277b85f640 10 monclient: build_initial_monmap 2024-03-27T21:41:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.100+0000 7f277b85f640 10 monclient: monmap: 2024-03-27T21:41:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:13.101029+0000 2024-03-27T21:41:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:13.101029+0000 2024-03-27T21:41:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:13.087 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:13.087 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:13.087 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:13.087 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:13.087 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:13.087 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:13.087 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:13.094 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:41:13.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.100+0000 7f277b85f640 1 Processor -- start 2024-03-27T21:41:13.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.100+0000 7f277b85f640 1 -- start start 2024-03-27T21:41:13.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.100+0000 7f277b85f640 10 monclient: init 2024-03-27T21:41:13.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f277b85f640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:13.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f277b85f640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-03-27T21:41:13.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f277b85f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f27740b40b0 0x7f27740a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:13.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f277b85f640 10 monclient(hunting): picked mon.noname-a con 0x7f27740b40b0 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:13.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f277b85f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f277413b020 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:13.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f277b85f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f277413b480 0x7f2774141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:13.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f277b85f640 10 monclient(hunting): picked mon.noname-b con 0x7f277413b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:13.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f277b85f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27740b3830 con 0x7f277413b020 2024-03-27T21:41:13.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f277b85f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:13.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f277b85f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f277413b910 con 0x7f27740b40b0 2024-03-27T21:41:13.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f277b85f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:13.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f277b85f640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27740b4490 con 0x7f277413b480 2024-03-27T21:41:13.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f277b85f640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:13.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f277b85f640 10 monclient(hunting): authenticate will time out at 914.322320s 2024-03-27T21:41:13.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f27795fb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f27740b40b0 0x7f27740a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:13.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f2779dfc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f277413b480 0x7f2774141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:13.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f27795fb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f27740b40b0 0x7f27740a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37240/0 (socket says 172.21.15.60:37240) 2024-03-27T21:41:13.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f2779dfc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f277413b480 0x7f2774141e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:37820/0 (socket says 172.21.15.60:37820) 2024-03-27T21:41:13.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f27795fb640 1 -- 172.21.15.60:0/1810274995 learned_addr learned my addr 172.21.15.60:0/1810274995 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:13.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f27795fb640 10 monclient(hunting): get_auth_request con 0x7f27740b40b0 auth_method 0 2024-03-27T21:41:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f27795fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f27795fb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f27795fb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f2779dfc640 10 monclient(hunting): get_auth_request con 0x7f277413b480 auth_method 0 2024-03-27T21:41:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f2779dfc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f2779dfc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f2779dfc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f2779dfc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f2779dfc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f2779dfc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f27795fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f27795fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f27795fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f2779dfc640 10 monclient(hunting): handle_auth_done global_id 4733 payload 274 2024-03-27T21:41:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f2779dfc640 1 -- 172.21.15.60:0/1810274995 >> v2:172.21.15.60:3300/0 conn(0x7f27740b40b0 msgr2=0x7f27740a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f2779dfc640 1 --2- 172.21.15.60:0/1810274995 >> v2:172.21.15.60:3300/0 conn(0x7f27740b40b0 0x7f27740a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f2779dfc640 1 -- 172.21.15.60:0/1810274995 >> v1:172.21.15.60:6789/0 conn(0x7f277413b020 legacy=0x7f27741416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f2779dfc640 10 monclient: _finish_hunting 0 2024-03-27T21:41:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f2779dfc640 1 monclient: found mon.noname-b 2024-03-27T21:41:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f2779dfc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f2779dfc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f2779dfc640 1 -- 172.21.15.60:0/1810274995 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27740aa520 con 0x7f277413b480 2024-03-27T21:41:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.101+0000 7f27795fb640 1 --2- 172.21.15.60:0/1810274995 >> v2:172.21.15.60:3300/0 conn(0x7f27740b40b0 0x7f27740a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.102+0000 7f2779dfc640 1 --2- 172.21.15.60:0/1810274995 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f277413b480 0x7f2774141e00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f276c009d90 tx=0x7f276c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5ca3b6828debf6f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.102+0000 7f276bfff640 1 -- 172.21.15.60:0/1810274995 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c02cbc0 con 0x7f277413b480 2024-03-27T21:41:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.102+0000 7f276bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.102+0000 7f276bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.102+0000 7f276bfff640 10 monclient: dump: 2024-03-27T21:41:13.101 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:13.101 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:13.101 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:13.101 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:13.101 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:13.101 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:13.101 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:13.101 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:13.101 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:13.101 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:13.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.102+0000 7f276bfff640 10 monclient: _finish_auth 0 2024-03-27T21:41:13.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.102+0000 7f276bfff640 10 monclient: _check_auth_tickets 2024-03-27T21:41:13.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.102+0000 7f276bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:13.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.102+0000 7f276bfff640 1 -- 172.21.15.60:0/1810274995 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f276c02cd20 con 0x7f277413b480 2024-03-27T21:41:13.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.102+0000 7f277b85f640 5 monclient: authenticate success, global_id 4733 2024-03-27T21:41:13.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.102+0000 7f277b85f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:13.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.102+0000 7f276bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:13.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.102+0000 7f277b85f640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:13.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.102+0000 7f277b85f640 4 set_mon_vals no callback set 2024-03-27T21:41:13.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.102+0000 7f276bfff640 1 -- 172.21.15.60:0/1810274995 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c034660 con 0x7f277413b480 2024-03-27T21:41:13.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.102+0000 7f276bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:13.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.102+0000 7f276bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:13.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.102+0000 7f276bfff640 10 monclient: dump: 2024-03-27T21:41:13.102 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:13.102 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:13.102 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:13.102 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:13.103 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.103+0000 7f277b85f640 10 monclient: shutdown 2024-03-27T21:41:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.103+0000 7f277b85f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.103+0000 7f277b85f640 1 -- 172.21.15.60:0/1810274995 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f277413b480 msgr2=0x7f2774141e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.103+0000 7f277b85f640 1 --2- 172.21.15.60:0/1810274995 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f277413b480 0x7f2774141e00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f276c009d90 tx=0x7f276c02be10 comp rx=0 tx=0).stop 2024-03-27T21:41:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.103+0000 7f277b85f640 1 -- 172.21.15.60:0/1810274995 shutdown_connections 2024-03-27T21:41:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.103+0000 7f277b85f640 1 --2- 172.21.15.60:0/1810274995 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f277413b480 0x7f2774141e00 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.103+0000 7f277b85f640 1 --2- 172.21.15.60:0/1810274995 >> v2:172.21.15.60:3300/0 conn(0x7f27740b40b0 0x7f27740a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.103+0000 7f277b85f640 1 -- 172.21.15.60:0/1810274995 >> 172.21.15.60:0/1810274995 conn(0x7f27740608e0 msgr2=0x7f2774060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:13.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.103+0000 7f277b85f640 1 -- 172.21.15.60:0/1810274995 shutdown_connections 2024-03-27T21:41:13.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.103+0000 7f277b85f640 1 -- 172.21.15.60:0/1810274995 wait complete. 2024-03-27T21:41:13.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.103+0000 7f277b85f640 10 monclient: build_initial_monmap 2024-03-27T21:41:13.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.103+0000 7f277b85f640 10 monclient: monmap: 2024-03-27T21:41:13.104 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:13.104 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:13.104 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:13.104 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:13.104 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:13.104 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:13.104 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:13.104 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:13.104 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:13.104 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:13.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 1 librados: starting msgr at 2024-03-27T21:41:13.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 1 librados: starting objecter 2024-03-27T21:41:13.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 1 Processor -- start 2024-03-27T21:41:13.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 1 -- start start 2024-03-27T21:41:13.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 1 librados: setting wanted keys 2024-03-27T21:41:13.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 1 librados: calling monclient init 2024-03-27T21:41:13.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 10 monclient: init 2024-03-27T21:41:13.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:13.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:41:13.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27740b40b0 0x7f27741346b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:13.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 10 monclient(hunting): picked mon.noname-b con 0x7f27740b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:13.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f277413b020 0x7f27741d96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:13.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 10 monclient(hunting): picked mon.noname-a con 0x7f277413b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:13.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f277413b480 0x7f27741d9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:13.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 10 monclient(hunting): picked mon.noname-c con 0x7f277413b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:13.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:13.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277413bf80 con 0x7f277413b020 2024-03-27T21:41:13.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:13.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277413c410 con 0x7f27740b40b0 2024-03-27T21:41:13.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:13.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27741da170 con 0x7f277413b480 2024-03-27T21:41:13.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:13.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f277b85f640 10 monclient(hunting): authenticate will time out at 914.325815s 2024-03-27T21:41:13.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f2778dfa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f277413b020 0x7f27741d96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:13.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.104+0000 7f2779dfc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f277413b480 0x7f27741d9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:13.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2779dfc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f277413b480 0x7f27741d9c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:37836/0 (socket says 172.21.15.60:37836) 2024-03-27T21:41:13.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2779dfc640 1 -- 172.21.15.60:0/2749021005 learned_addr learned my addr 172.21.15.60:0/2749021005 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:13.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2779dfc640 10 monclient(hunting): get_auth_request con 0x7f277413b480 auth_method 0 2024-03-27T21:41:13.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2779dfc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:13.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2779dfc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2779dfc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2778dfa640 1 --2- 172.21.15.60:0/2749021005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f277413b020 0x7f27741d96f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37250/0 (socket says 172.21.15.60:37250) 2024-03-27T21:41:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2778dfa640 10 monclient(hunting): get_auth_request con 0x7f277413b020 auth_method 0 2024-03-27T21:41:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2778dfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2778dfa640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2778dfa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2779dfc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2779dfc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2779dfc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2778dfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2778dfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2778dfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2779dfc640 10 monclient(hunting): handle_auth_done global_id 4736 payload 824 2024-03-27T21:41:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f27795fb640 1 --2- 172.21.15.60:0/2749021005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27740b40b0 0x7f27741346b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2779dfc640 1 -- 172.21.15.60:0/2749021005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27740b40b0 msgr2=0x7f27741346b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2779dfc640 1 --2- 172.21.15.60:0/2749021005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27740b40b0 0x7f27741346b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2779dfc640 1 -- 172.21.15.60:0/2749021005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f277413b020 msgr2=0x7f27741d96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2779dfc640 1 --2- 172.21.15.60:0/2749021005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f277413b020 0x7f27741d96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2779dfc640 10 monclient: _finish_hunting 0 2024-03-27T21:41:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2779dfc640 1 monclient: found mon.noname-c 2024-03-27T21:41:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2779dfc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2779dfc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2779dfc640 1 -- 172.21.15.60:0/2749021005 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27700094d0 con 0x7f277413b480 2024-03-27T21:41:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f27795fb640 10 monclient: get_auth_request con 0x7f27740b40b0 auth_method 0 2024-03-27T21:41:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2778dfa640 1 --2- 172.21.15.60:0/2749021005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f277413b020 0x7f27741d96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f27795fb640 1 --2- 172.21.15.60:0/2749021005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27740b40b0 0x7f27741346b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:41:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2779dfc640 1 --2- 172.21.15.60:0/2749021005 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f277413b480 0x7f27741d9c30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f276c009eb0 tx=0x7f276c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2769ffb640 10 client.?.objecter ms_handle_connect 0x7f277413b480 2024-03-27T21:41:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2769ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:13.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2769ffb640 1 -- 172.21.15.60:0/2749021005 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c040650 con 0x7f277413b480 2024-03-27T21:41:13.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2769ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:13.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2769ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:13.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2769ffb640 10 monclient: dump: 2024-03-27T21:41:13.109 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:13.109 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:13.109 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:13.109 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:13.109 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:13.109 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:13.109 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:13.109 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:13.109 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:13.109 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:13.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2769ffb640 10 monclient: _finish_auth 0 2024-03-27T21:41:13.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2769ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:41:13.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2769ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:13.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2769ffb640 1 -- 172.21.15.60:0/2749021005 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f276c02ce90 con 0x7f277413b480 2024-03-27T21:41:13.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f277b85f640 5 monclient: authenticate success, global_id 4736 2024-03-27T21:41:13.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2769ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:13.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2769ffb640 1 -- 172.21.15.60:0/2749021005 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c0409f0 con 0x7f277413b480 2024-03-27T21:41:13.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f277b85f640 10 monclient: _renew_subs 2024-03-27T21:41:13.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f277b85f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:13.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f277b85f640 1 -- 172.21.15.60:0/2749021005 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f276c009a70 con 0x7f277413b480 2024-03-27T21:41:13.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2769ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:13.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f276b7fe640 4 set_mon_vals no callback set 2024-03-27T21:41:13.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2769ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:13.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f2769ffb640 10 monclient: dump: 2024-03-27T21:41:13.110 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:13.111 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:13.111 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:13.111 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:13.111 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:13.111 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:13.111 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:13.111 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:13.111 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:13.111 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:13.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f277b85f640 10 client.4736.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:13.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f277b85f640 10 monclient: _renew_subs 2024-03-27T21:41:13.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f277b85f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:13.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f277b85f640 1 -- 172.21.15.60:0/2749021005 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f277413ba60 con 0x7f277413b480 2024-03-27T21:41:13.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.105+0000 7f277b85f640 1 librados: init done 2024-03-27T21:41:13.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.106+0000 7f277b85f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:13.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.106+0000 7f277b85f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:13.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.106+0000 7f277b85f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:13.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.106+0000 7f277b85f640 1 -- 172.21.15.60:0/2749021005 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f277413c130 con 0x7f277413b480 2024-03-27T21:41:13.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.107+0000 7f2769ffb640 1 -- 172.21.15.60:0/2749021005 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f276c040b50 con 0x7f277413b480 2024-03-27T21:41:13.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.107+0000 7f2769ffb640 1 --2- 172.21.15.60:0/2749021005 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f275806c590 0x7f275806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:13.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.107+0000 7f2769ffb640 1 -- 172.21.15.60:0/2749021005 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f276c0abd00 con 0x7f277413b480 2024-03-27T21:41:13.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.107+0000 7f2769ffb640 10 client.4736.objecter ms_dispatch 0x7f2774000b90 osd_map(54..54 src has 1..54) v4 2024-03-27T21:41:13.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.107+0000 7f2769ffb640 3 client.4736.objecter handle_osd_map got epochs [54,54] > 0 2024-03-27T21:41:13.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.107+0000 7f2769ffb640 3 client.4736.objecter handle_osd_map decoding full epoch 54 2024-03-27T21:41:13.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.107+0000 7f27795fb640 1 --2- 172.21.15.60:0/2749021005 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f275806c590 0x7f275806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:13.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.107+0000 7f27795fb640 10 monclient: get_auth_request con 0x7f275806c590 auth_method 0 2024-03-27T21:41:13.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.107+0000 7f2769ffb640 20 client.4736.objecter dump_active .. 0 homeless 2024-03-27T21:41:13.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.108+0000 7f2769ffb640 10 client.4736.objecter ms_handle_connect 0x7f275806c590 2024-03-27T21:41:13.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.108+0000 7f27795fb640 1 --2- 172.21.15.60:0/2749021005 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f275806c590 0x7f275806ea50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2770009c50 tx=0x7f2770008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:13.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.111+0000 7f2769ffb640 1 -- 172.21.15.60:0/2749021005 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f277413c130 con 0x7f277413b480 2024-03-27T21:41:13.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.111+0000 7f2769ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:13.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.111+0000 7f2769ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:13.114 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:13.127+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:13.214 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:13.229+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:13.218 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:13.219 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:13.227 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:41:13.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.304+0000 7f277b85f640 10 start_mon_command cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:41:13.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.304+0000 7f277b85f640 10 monclient: _send_command 2 [{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:41:13.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.304+0000 7f277b85f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:13.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.304+0000 7f277b85f640 1 -- 172.21.15.60:0/2749021005 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f277413c130 con 0x7f277413b480 2024-03-27T21:41:13.315 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:13.330+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:13.372 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:13.372 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:13.380 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:41:13.421 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:13.436+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:13.517 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:13.532+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:13.525 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:13.525 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:13.534 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:41:13.618 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:13.633+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:13.679 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:13.679 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:13.688 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:41:13.731 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:13.746+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:13.792 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:13.792 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:13.800 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:41:13.841 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:13.856+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:13.921 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:13.935+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:13.945 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:13.946 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:13.954 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:41:13.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.993+0000 7f2769ffb640 1 -- 172.21.15.60:0/2749021005 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v55) v1 ==== 101+0+0 (secure 0 0 0) 0x7f277413c130 con 0x7f277413b480 2024-03-27T21:41:13.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.993+0000 7f2769ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:41:13.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.993+0000 7f2769ffb640 10 monclient: _finish_command 2 = system:0 nodeep-scrub is unset 2024-03-27T21:41:13.980 INFO:teuthology.orchestra.run.smithi060.stderr:nodeep-scrub is unset 2024-03-27T21:41:13.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.996+0000 7f277b85f640 10 librados: watch_flush enter 2024-03-27T21:41:13.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.996+0000 7f277b85f640 10 librados: watch_flush exit 2024-03-27T21:41:13.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.996+0000 7f277b85f640 20 client.4736.objecter shutdown clearing up homeless session... 2024-03-27T21:41:13.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.996+0000 7f277b85f640 10 client.4736.objecter successfully canceled tick 2024-03-27T21:41:13.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.996+0000 7f277b85f640 1 -- 172.21.15.60:0/2749021005 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f275806c590 msgr2=0x7f275806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:13.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.996+0000 7f277b85f640 1 --2- 172.21.15.60:0/2749021005 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f275806c590 0x7f275806ea50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2770009c50 tx=0x7f2770008040 comp rx=0 tx=0).stop 2024-03-27T21:41:13.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.996+0000 7f277b85f640 10 monclient: shutdown 2024-03-27T21:41:13.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.996+0000 7f277b85f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:13.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.996+0000 7f277b85f640 1 -- 172.21.15.60:0/2749021005 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f277413b480 msgr2=0x7f27741d9c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:13.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.996+0000 7f277b85f640 1 --2- 172.21.15.60:0/2749021005 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f277413b480 0x7f27741d9c30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f276c009eb0 tx=0x7f276c004a40 comp rx=0 tx=0).stop 2024-03-27T21:41:13.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.996+0000 7f277b85f640 1 -- 172.21.15.60:0/2749021005 shutdown_connections 2024-03-27T21:41:13.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.997+0000 7f277b85f640 1 --2- 172.21.15.60:0/2749021005 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f277413b480 0x7f27741d9c30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:13.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.997+0000 7f277b85f640 1 --2- 172.21.15.60:0/2749021005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f277413b020 0x7f27741d96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:13.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.997+0000 7f277b85f640 1 --2- 172.21.15.60:0/2749021005 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f275806c590 0x7f275806ea50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:13.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.997+0000 7f277b85f640 1 --2- 172.21.15.60:0/2749021005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27740b40b0 0x7f27741346b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:13.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.997+0000 7f277b85f640 1 -- 172.21.15.60:0/2749021005 >> 172.21.15.60:0/2749021005 conn(0x7f27740608e0 msgr2=0x7f27740b67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:13.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.997+0000 7f277b85f640 1 -- 172.21.15.60:0/2749021005 shutdown_connections 2024-03-27T21:41:13.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.997+0000 7f277b85f640 1 -- 172.21.15.60:0/2749021005 wait complete. 2024-03-27T21:41:13.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:13.997+0000 7f277b85f640 1 librados: shutdown 2024-03-27T21:41:14.023 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:14.037+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:14.068 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:14.069 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:14.078 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:41:14.124 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:14.139+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:14.184 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:14.185 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:14.193 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:41:14.225 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:14.239+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:14.304 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:14.304 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:14.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:41:14.357 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:14.371+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:14.421 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:14.421 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:14.425 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:14.439+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:14.430 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:41:14.527 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:14.541+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:14.589 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:14.590 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:14.601 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:41:14.627 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:14.642+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:14.654 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:41:14.713 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:14.713 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:14.722 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:41:14.766 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:14.780+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:14.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.803+0000 7fe218577640 10 monclient: get_monmap_and_config 2024-03-27T21:41:14.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.803+0000 7fe218577640 10 monclient: build_initial_monmap 2024-03-27T21:41:14.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.803+0000 7fe218577640 10 monclient: monmap: 2024-03-27T21:41:14.790 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:14.790 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:14.790 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:14.804343+0000 2024-03-27T21:41:14.790 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:14.804343+0000 2024-03-27T21:41:14.791 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:14.791 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:14.791 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:14.791 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:14.791 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:14.791 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:14.791 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:14.792 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:14.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.804+0000 7fe218577640 1 Processor -- start 2024-03-27T21:41:14.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.804+0000 7fe218577640 1 -- start start 2024-03-27T21:41:14.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.804+0000 7fe218577640 10 monclient: init 2024-03-27T21:41:14.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe218577640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:14.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe218577640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-03-27T21:41:14.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe218577640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe2100b40b0 0x7fe2100a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:14.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe218577640 10 monclient(hunting): picked mon.noname-b con 0x7fe2100b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:14.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe218577640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe21013b020 0x7fe21013b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:14.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe218577640 10 monclient(hunting): picked mon.noname-a con 0x7fe21013b020 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:14.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe218577640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2101416f0 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:14.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe218577640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2100b3830 con 0x7fe2101416f0 2024-03-27T21:41:14.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe218577640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:14.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe218577640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2100b4490 con 0x7fe21013b020 2024-03-27T21:41:14.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe218577640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:14.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe218577640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe210142290 con 0x7fe2100b40b0 2024-03-27T21:41:14.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe218577640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:14.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe218577640 10 monclient(hunting): authenticate will time out at 916.026615s 2024-03-27T21:41:14.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe216313640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe2100b40b0 0x7fe2100a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:14.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe215b12640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe21013b020 0x7fe21013b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:14.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe216b14640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe2101416f0 0x7fe210141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40218/0 (socket says 172.21.15.60:40218) 2024-03-27T21:41:14.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe216b14640 1 -- 172.21.15.60:0/2064730797 learned_addr learned my addr 172.21.15.60:0/2064730797 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:14.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe216313640 10 monclient(hunting): get_auth_request con 0x7fe2100b40b0 auth_method 0 2024-03-27T21:41:14.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe216313640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:14.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe216313640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:14.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe216313640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:14.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe215b12640 10 monclient(hunting): get_auth_request con 0x7fe21013b020 auth_method 0 2024-03-27T21:41:14.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe215b12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:14.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe215b12640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:14.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe215b12640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:14.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe216313640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:14.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe216313640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:14.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe216313640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:14.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe215b12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:14.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe215b12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:14.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.805+0000 7fe215b12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:14.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.806+0000 7fe215b12640 10 monclient(hunting): handle_auth_done global_id 4920 payload 274 2024-03-27T21:41:14.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.806+0000 7fe215b12640 1 -- 172.21.15.60:0/2064730797 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe2100b40b0 msgr2=0x7fe2100a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:14.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.806+0000 7fe215b12640 1 --2- 172.21.15.60:0/2064730797 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe2100b40b0 0x7fe2100a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:14.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.806+0000 7fe215311640 1 -- 172.21.15.60:0/2064730797 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 750208952 0 0) 0x7fe2100b3830 con 0x7fe2101416f0 2024-03-27T21:41:14.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.806+0000 7fe215b12640 1 -- 172.21.15.60:0/2064730797 >> v1:172.21.15.60:6789/0 conn(0x7fe2101416f0 legacy=0x7fe210141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:14.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.806+0000 7fe215b12640 10 monclient: _finish_hunting 0 2024-03-27T21:41:14.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.806+0000 7fe215b12640 1 monclient: found mon.noname-a 2024-03-27T21:41:14.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.806+0000 7fe215b12640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:14.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.806+0000 7fe215b12640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:14.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.806+0000 7fe215b12640 1 -- 172.21.15.60:0/2064730797 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2100aa520 con 0x7fe21013b020 2024-03-27T21:41:14.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.806+0000 7fe215311640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:41:14.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.807+0000 7fe215b12640 1 --2- 172.21.15.60:0/2064730797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe21013b020 0x7fe21013b480 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe2000097a0 tx=0x7fe20002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=273e692d47dd198f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:14.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.807+0000 7fe215311640 1 -- 172.21.15.60:0/2064730797 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20002d040 con 0x7fe21013b020 2024-03-27T21:41:14.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.807+0000 7fe215311640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:14.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.807+0000 7fe215311640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:14.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.807+0000 7fe215311640 10 monclient: dump: 2024-03-27T21:41:14.803 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:14.803 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:14.803 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:14.803 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:14.803 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:14.803 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:14.803 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:14.804 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:14.804 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:14.804 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:14.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.807+0000 7fe215311640 10 monclient: _finish_auth 0 2024-03-27T21:41:14.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.807+0000 7fe215311640 10 monclient: _check_auth_tickets 2024-03-27T21:41:14.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.807+0000 7fe215311640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:14.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.807+0000 7fe215311640 1 -- 172.21.15.60:0/2064730797 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe200031810 con 0x7fe21013b020 2024-03-27T21:41:14.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.807+0000 7fe215311640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:14.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.807+0000 7fe215311640 1 -- 172.21.15.60:0/2064730797 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe200031ba0 con 0x7fe21013b020 2024-03-27T21:41:14.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.807+0000 7fe215311640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:14.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.807+0000 7fe215311640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:14.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.807+0000 7fe215311640 10 monclient: dump: 2024-03-27T21:41:14.805 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:14.805 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:14.806 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:14.806 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:14.806 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:14.806 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:14.806 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:14.806 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:14.807 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:14.807 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:14.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.807+0000 7fe218577640 5 monclient: authenticate success, global_id 4920 2024-03-27T21:41:14.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.807+0000 7fe218577640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:14.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.807+0000 7fe218577640 4 set_mon_vals no callback set 2024-03-27T21:41:14.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.808+0000 7fe218577640 10 monclient: shutdown 2024-03-27T21:41:14.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.808+0000 7fe218577640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:14.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.808+0000 7fe218577640 1 -- 172.21.15.60:0/2064730797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe21013b020 msgr2=0x7fe21013b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:14.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.808+0000 7fe218577640 1 --2- 172.21.15.60:0/2064730797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe21013b020 0x7fe21013b480 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe2000097a0 tx=0x7fe20002b8b0 comp rx=0 tx=0).stop 2024-03-27T21:41:14.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.808+0000 7fe218577640 1 -- 172.21.15.60:0/2064730797 shutdown_connections 2024-03-27T21:41:14.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.808+0000 7fe218577640 1 --2- 172.21.15.60:0/2064730797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe21013b020 0x7fe21013b480 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:14.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.808+0000 7fe218577640 1 --2- 172.21.15.60:0/2064730797 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe2100b40b0 0x7fe2100a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:14.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.808+0000 7fe218577640 1 -- 172.21.15.60:0/2064730797 >> 172.21.15.60:0/2064730797 conn(0x7fe2100608e0 msgr2=0x7fe210060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:14.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.808+0000 7fe218577640 1 -- 172.21.15.60:0/2064730797 shutdown_connections 2024-03-27T21:41:14.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.808+0000 7fe218577640 1 -- 172.21.15.60:0/2064730797 wait complete. 2024-03-27T21:41:14.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.809+0000 7fe218577640 10 monclient: build_initial_monmap 2024-03-27T21:41:14.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.809+0000 7fe218577640 10 monclient: monmap: 2024-03-27T21:41:14.809 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:14.809 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:14.809 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:14.810 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:14.810 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:14.810 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:14.810 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:14.810 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:14.810 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:14.810 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:14.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.810+0000 7fe218577640 1 librados: starting msgr at 2024-03-27T21:41:14.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.810+0000 7fe218577640 1 librados: starting objecter 2024-03-27T21:41:14.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.810+0000 7fe218577640 1 Processor -- start 2024-03-27T21:41:14.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.810+0000 7fe218577640 1 -- start start 2024-03-27T21:41:14.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.810+0000 7fe218577640 1 librados: setting wanted keys 2024-03-27T21:41:14.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.810+0000 7fe218577640 1 librados: calling monclient init 2024-03-27T21:41:14.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.810+0000 7fe218577640 10 monclient: init 2024-03-27T21:41:14.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe218577640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:14.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe218577640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:41:14.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe218577640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2100b40b0 0x7fe2100aac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:14.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe218577640 10 monclient(hunting): picked mon.noname-b con 0x7fe2100b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:14.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe218577640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe21013b020 0x7fe2101da0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:14.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe218577640 10 monclient(hunting): picked mon.noname-c con 0x7fe21013b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:14.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe218577640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2101416f0 0x7fe2101da610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:14.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe215b12640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe21013b020 0x7fe2101da0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:14.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe215b12640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe21013b020 0x7fe2101da0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:37852/0 (socket says 172.21.15.60:37852) 2024-03-27T21:41:14.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe215b12640 1 -- 172.21.15.60:0/1148298670 learned_addr learned my addr 172.21.15.60:0/1148298670 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:14.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe216313640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2100b40b0 0x7fe2100aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:14.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe218577640 10 monclient(hunting): picked mon.noname-a con 0x7fe2101416f0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:14.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe218577640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:14.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe218577640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2101dab50 con 0x7fe2101416f0 2024-03-27T21:41:14.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe218577640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:14.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe218577640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2100ac0f0 con 0x7fe2100b40b0 2024-03-27T21:41:14.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe218577640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:14.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe218577640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21013d330 con 0x7fe21013b020 2024-03-27T21:41:14.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe218577640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:14.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe218577640 10 monclient(hunting): authenticate will time out at 916.032561s 2024-03-27T21:41:14.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe216b14640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2101416f0 0x7fe2101da610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:14.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe216b14640 10 monclient(hunting): get_auth_request con 0x7fe2101416f0 auth_method 0 2024-03-27T21:41:14.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe216b14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:14.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe216b14640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:14.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe216b14640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:14.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe216b14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:14.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe216b14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:14.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe216b14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:14.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe215b12640 10 monclient(hunting): get_auth_request con 0x7fe21013b020 auth_method 0 2024-03-27T21:41:14.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe215b12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:14.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe215b12640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:14.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe215b12640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:14.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe216313640 10 monclient(hunting): get_auth_request con 0x7fe2100b40b0 auth_method 0 2024-03-27T21:41:14.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe216313640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:14.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe216313640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:14.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe216313640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:14.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe215b12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:14.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe215b12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:14.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.811+0000 7fe215b12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:14.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe216b14640 10 monclient(hunting): handle_auth_done global_id 4926 payload 824 2024-03-27T21:41:14.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe216b14640 1 -- 172.21.15.60:0/1148298670 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe21013b020 msgr2=0x7fe2101da0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:14.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe216b14640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe21013b020 0x7fe2101da0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:14.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe216b14640 1 -- 172.21.15.60:0/1148298670 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2100b40b0 msgr2=0x7fe2100aac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:14.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe216b14640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2100b40b0 0x7fe2100aac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:14.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe216b14640 10 monclient: _finish_hunting 0 2024-03-27T21:41:14.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe216b14640 1 monclient: found mon.noname-a 2024-03-27T21:41:14.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe216b14640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:14.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe216b14640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:14.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe216b14640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe200009400 con 0x7fe2101416f0 2024-03-27T21:41:14.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe216b14640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2101416f0 0x7fe2101da610 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe208002230 tx=0x7fe20800cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:14.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe206ffd640 10 client.?.objecter ms_handle_connect 0x7fe2101416f0 2024-03-27T21:41:14.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe206ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:14.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe206ffd640 1 -- 172.21.15.60:0/1148298670 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe208004d60 con 0x7fe2101416f0 2024-03-27T21:41:14.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe206ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:14.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe206ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:14.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe206ffd640 10 monclient: dump: 2024-03-27T21:41:14.826 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:14.827 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:14.827 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:14.827 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:14.827 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:14.827 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:14.827 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:14.827 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:14.827 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:14.827 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:14.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe206ffd640 10 monclient: _finish_auth 0 2024-03-27T21:41:14.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe206ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:41:14.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe206ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:14.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe206ffd640 1 -- 172.21.15.60:0/1148298670 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe208015960 con 0x7fe2101416f0 2024-03-27T21:41:14.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe218577640 5 monclient: authenticate success, global_id 4926 2024-03-27T21:41:14.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe218577640 10 monclient: _renew_subs 2024-03-27T21:41:14.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe218577640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:14.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe218577640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2101d9bd0 con 0x7fe2101416f0 2024-03-27T21:41:14.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe218577640 10 client.4926.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:14.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe218577640 10 monclient: _renew_subs 2024-03-27T21:41:14.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe218577640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:14.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe218577640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2100ab180 con 0x7fe2101416f0 2024-03-27T21:41:14.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.812+0000 7fe218577640 1 librados: init done 2024-03-27T21:41:14.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.813+0000 7fe206ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:14.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.813+0000 7fe206ffd640 1 -- 172.21.15.60:0/1148298670 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe208015d10 con 0x7fe2101416f0 2024-03-27T21:41:14.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.813+0000 7fe214b10640 4 set_mon_vals no callback set 2024-03-27T21:41:14.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.813+0000 7fe206ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:14.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.813+0000 7fe206ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:14.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.813+0000 7fe206ffd640 10 monclient: dump: 2024-03-27T21:41:14.829 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:14.829 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:14.829 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:14.829 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:14.829 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:14.829 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:14.829 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:14.829 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:14.829 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:14.829 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:14.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.814+0000 7fe218577640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:41:14.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.814+0000 7fe218577640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:14.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.814+0000 7fe218577640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:14.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.814+0000 7fe218577640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fe1f8005020 con 0x7fe2101416f0 2024-03-27T21:41:14.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.815+0000 7fe206ffd640 1 -- 172.21.15.60:0/1148298670 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe20801b020 con 0x7fe2101416f0 2024-03-27T21:41:14.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.815+0000 7fe206ffd640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe1f406c590 0x7fe1f406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:14.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.815+0000 7fe206ffd640 1 -- 172.21.15.60:0/1148298670 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fe2080910d0 con 0x7fe2101416f0 2024-03-27T21:41:14.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.815+0000 7fe206ffd640 10 client.4926.objecter ms_dispatch 0x7fe210000b90 osd_map(55..55 src has 1..55) v4 2024-03-27T21:41:14.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.815+0000 7fe206ffd640 3 client.4926.objecter handle_osd_map got epochs [55,55] > 0 2024-03-27T21:41:14.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.815+0000 7fe206ffd640 3 client.4926.objecter handle_osd_map decoding full epoch 55 2024-03-27T21:41:14.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.815+0000 7fe206ffd640 20 client.4926.objecter dump_active .. 0 homeless 2024-03-27T21:41:14.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.815+0000 7fe206ffd640 1 -- 172.21.15.60:0/1148298670 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v55) v1 ==== 54+0+16 (secure 0 0 0) 0x7fe208098070 con 0x7fe2101416f0 2024-03-27T21:41:14.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.815+0000 7fe206ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:14.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.815+0000 7fe206ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:14.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.815+0000 7fe216313640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe1f406c590 0x7fe1f406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:14.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.815+0000 7fe216313640 10 monclient: get_auth_request con 0x7fe1f406c590 auth_method 0 2024-03-27T21:41:14.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.816+0000 7fe204ff9640 10 client.4926.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:14.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.816+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign -1 1 2024-03-27T21:41:14.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.816+0000 7fe204ff9640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fe1f80058c0 0x7fe1f8007cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:14.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.816+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f80052c0 osd=0 3 2024-03-27T21:41:14.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.816+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f80052c0 osd=0 3 2024-03-27T21:41:14.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.816+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f80052c0 osd=0 3 2024-03-27T21:41:14.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.816+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_remove -1 1 2024-03-27T21:41:14.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.816+0000 7fe204ff9640 20 client.4926.objecter get_session s=0x7fe1f80052c0 osd=0 3 2024-03-27T21:41:14.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.816+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign 0 1 2024-03-27T21:41:14.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.816+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f80052c0 osd=0 4 2024-03-27T21:41:14.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.816+0000 7fe204ff9640 10 client.4926.objecter _send_command 1 2024-03-27T21:41:14.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.816+0000 7fe204ff9640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1f80022e0 con 0x7fe1f80058c0 2024-03-27T21:41:14.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.816+0000 7fe215b12640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fe1f80058c0 0x7fe1f8007cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:14.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.816+0000 7fe215b12640 10 monclient: get_auth_request con 0x7fe1f80058c0 auth_method 0 2024-03-27T21:41:14.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.816+0000 7fe206ffd640 10 client.4926.objecter ms_handle_connect 0x7fe1f406c590 2024-03-27T21:41:14.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.816+0000 7fe216313640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe1f406c590 0x7fe1f406ea50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe20c009b20 tx=0x7fe20c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:14.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.816+0000 7fe206ffd640 10 client.4926.objecter ms_handle_connect 0x7fe1f80058c0 2024-03-27T21:41:14.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.816+0000 7fe215b12640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fe1f80058c0 0x7fe1f8007cb0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:14.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.817+0000 7fe206ffd640 1 -- 172.21.15.60:0/1148298670 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe1f80022e0 con 0x7fe1f80058c0 2024-03-27T21:41:14.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.817+0000 7fe206ffd640 10 client.4926.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:41:14.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.817+0000 7fe206ffd640 10 client.4926.objecter _finish_command 1 = system:0 2024-03-27T21:41:14.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.817+0000 7fe206ffd640 20 client.4926.objecter put_session s=0x7fe1f80052c0 osd=0 4 2024-03-27T21:41:14.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.817+0000 7fe206ffd640 15 client.4926.objecter _session_command_op_remove 0 1 2024-03-27T21:41:14.833 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:14.848+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:14.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.848+0000 7fe204ff9640 10 client.4926.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:14.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.848+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign -1 2 2024-03-27T21:41:14.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.848+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f80052c0 osd=0 3 2024-03-27T21:41:14.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.848+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f80052c0 osd=0 3 2024-03-27T21:41:14.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.848+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f80052c0 osd=0 3 2024-03-27T21:41:14.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.848+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_remove -1 2 2024-03-27T21:41:14.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.848+0000 7fe204ff9640 20 client.4926.objecter get_session s=0x7fe1f80052c0 osd=0 3 2024-03-27T21:41:14.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.848+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign 0 2 2024-03-27T21:41:14.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.848+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f80052c0 osd=0 4 2024-03-27T21:41:14.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.848+0000 7fe204ff9640 10 client.4926.objecter _send_command 2 2024-03-27T21:41:14.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.848+0000 7fe204ff9640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe1f8001850 con 0x7fe1f80058c0 2024-03-27T21:41:14.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.851+0000 7fe206ffd640 1 -- 172.21.15.60:0/1148298670 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe1f8001850 con 0x7fe1f80058c0 2024-03-27T21:41:14.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.851+0000 7fe206ffd640 10 client.4926.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:14.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.851+0000 7fe206ffd640 10 client.4926.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:14.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.851+0000 7fe206ffd640 20 client.4926.objecter put_session s=0x7fe1f80052c0 osd=0 4 2024-03-27T21:41:14.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.851+0000 7fe206ffd640 15 client.4926.objecter _session_command_op_remove 0 2 2024-03-27T21:41:14.837 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:41:14.839 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:41:14.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.851+0000 7fe204ff9640 10 client.4926.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:14.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.851+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign -1 3 2024-03-27T21:41:14.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.851+0000 7fe204ff9640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fe1f8009370 0x7fe1f800b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:14.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.851+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f8004610 osd=1 3 2024-03-27T21:41:14.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.851+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f8004610 osd=1 3 2024-03-27T21:41:14.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.851+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f8004610 osd=1 3 2024-03-27T21:41:14.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.851+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_remove -1 3 2024-03-27T21:41:14.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.851+0000 7fe204ff9640 20 client.4926.objecter get_session s=0x7fe1f8004610 osd=1 3 2024-03-27T21:41:14.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.851+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign 1 3 2024-03-27T21:41:14.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.851+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f8004610 osd=1 4 2024-03-27T21:41:14.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.851+0000 7fe204ff9640 10 client.4926.objecter _send_command 3 2024-03-27T21:41:14.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.851+0000 7fe204ff9640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1f800e530 con 0x7fe1f8009370 2024-03-27T21:41:14.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.851+0000 7fe216313640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fe1f8009370 0x7fe1f800b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:14.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.851+0000 7fe216313640 10 monclient: get_auth_request con 0x7fe1f8009370 auth_method 0 2024-03-27T21:41:14.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.852+0000 7fe206ffd640 10 client.4926.objecter ms_handle_connect 0x7fe1f8009370 2024-03-27T21:41:14.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.852+0000 7fe216313640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fe1f8009370 0x7fe1f800b760 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:14.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.853+0000 7fe206ffd640 1 -- 172.21.15.60:0/1148298670 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe1f800e530 con 0x7fe1f8009370 2024-03-27T21:41:14.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.853+0000 7fe206ffd640 10 client.4926.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:41:14.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.853+0000 7fe206ffd640 10 client.4926.objecter _finish_command 3 = system:0 2024-03-27T21:41:14.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.853+0000 7fe206ffd640 20 client.4926.objecter put_session s=0x7fe1f8004610 osd=1 4 2024-03-27T21:41:14.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.853+0000 7fe206ffd640 15 client.4926.objecter _session_command_op_remove 1 3 2024-03-27T21:41:14.868 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:14.868 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:14.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.886+0000 7fe204ff9640 10 client.4926.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:14.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.886+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign -1 4 2024-03-27T21:41:14.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.886+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f8004610 osd=1 3 2024-03-27T21:41:14.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.886+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f8004610 osd=1 3 2024-03-27T21:41:14.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.886+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f8004610 osd=1 3 2024-03-27T21:41:14.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.886+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_remove -1 4 2024-03-27T21:41:14.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.886+0000 7fe204ff9640 20 client.4926.objecter get_session s=0x7fe1f8004610 osd=1 3 2024-03-27T21:41:14.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.886+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign 1 4 2024-03-27T21:41:14.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.886+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f8004610 osd=1 4 2024-03-27T21:41:14.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.886+0000 7fe204ff9640 10 client.4926.objecter _send_command 4 2024-03-27T21:41:14.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.886+0000 7fe204ff9640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe1f8004090 con 0x7fe1f8009370 2024-03-27T21:41:14.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.888+0000 7fe206ffd640 1 -- 172.21.15.60:0/1148298670 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe1f8004090 con 0x7fe1f8009370 2024-03-27T21:41:14.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.888+0000 7fe206ffd640 10 client.4926.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:14.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.888+0000 7fe206ffd640 10 client.4926.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:14.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.888+0000 7fe206ffd640 20 client.4926.objecter put_session s=0x7fe1f8004610 osd=1 4 2024-03-27T21:41:14.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.888+0000 7fe206ffd640 15 client.4926.objecter _session_command_op_remove 1 4 2024-03-27T21:41:14.875 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:41:14.877 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:41:14.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.889+0000 7fe204ff9640 10 client.4926.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:14.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.889+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign -1 5 2024-03-27T21:41:14.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.889+0000 7fe204ff9640 20 client.4926.objecter _get_session osd=-1 returning homeless 2024-03-27T21:41:14.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.889+0000 7fe204ff9640 10 client.4926.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:14.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.889+0000 7fe204ff9640 10 monclient: _renew_subs 2024-03-27T21:41:14.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.889+0000 7fe204ff9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:14.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.889+0000 7fe204ff9640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=56}) v3 -- 0x7fe1f80089b0 con 0x7fe2101416f0 2024-03-27T21:41:14.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.889+0000 7fe204ff9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:14.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.889+0000 7fe204ff9640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe1f8008d30 con 0x7fe2101416f0 2024-03-27T21:41:14.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.889+0000 7fe206ffd640 1 -- 172.21.15.60:0/1148298670 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe208015e70 con 0x7fe2101416f0 2024-03-27T21:41:14.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.889+0000 7fe206ffd640 10 monclient: handle_get_version_reply finishing 1 version 55 2024-03-27T21:41:14.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.889+0000 7fe214b10640 10 client.4926.objecter _check_command_map_dne tid 5 current 55 map_dne_bound 55 2024-03-27T21:41:14.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.889+0000 7fe214b10640 10 client.4926.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:41:14.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.889+0000 7fe214b10640 15 client.4926.objecter _session_command_op_remove -1 5 2024-03-27T21:41:14.878 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:41:14.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.889+0000 7fe204ff9640 10 client.4926.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:14.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.889+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign -1 6 2024-03-27T21:41:14.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.889+0000 7fe204ff9640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fe1f800d330 0x7fe1f800d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:14.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.890+0000 7fe215b12640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fe1f800d330 0x7fe1f800d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:14.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.890+0000 7fe215b12640 10 monclient: get_auth_request con 0x7fe1f800d330 auth_method 0 2024-03-27T21:41:14.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.890+0000 7fe206ffd640 10 client.4926.objecter ms_handle_connect 0x7fe1f800d330 2024-03-27T21:41:14.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.890+0000 7fe215b12640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fe1f800d330 0x7fe1f800d7a0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:14.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.890+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f80090e0 osd=3 3 2024-03-27T21:41:14.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.890+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f80090e0 osd=3 3 2024-03-27T21:41:14.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.890+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f80090e0 osd=3 3 2024-03-27T21:41:14.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.890+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_remove -1 6 2024-03-27T21:41:14.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.890+0000 7fe204ff9640 20 client.4926.objecter get_session s=0x7fe1f80090e0 osd=3 3 2024-03-27T21:41:14.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.890+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign 3 6 2024-03-27T21:41:14.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.890+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f80090e0 osd=3 4 2024-03-27T21:41:14.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.890+0000 7fe204ff9640 10 client.4926.objecter _send_command 6 2024-03-27T21:41:14.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.890+0000 7fe204ff9640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1f800dd90 con 0x7fe1f800d330 2024-03-27T21:41:14.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.891+0000 7fe206ffd640 1 -- 172.21.15.60:0/1148298670 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe1f800dd90 con 0x7fe1f800d330 2024-03-27T21:41:14.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.891+0000 7fe206ffd640 10 client.4926.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:41:14.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.891+0000 7fe206ffd640 10 client.4926.objecter _finish_command 6 = system:0 2024-03-27T21:41:14.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.891+0000 7fe206ffd640 20 client.4926.objecter put_session s=0x7fe1f80090e0 osd=3 4 2024-03-27T21:41:14.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.891+0000 7fe206ffd640 15 client.4926.objecter _session_command_op_remove 3 6 2024-03-27T21:41:14.882 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:41:14.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.905+0000 7fe204ff9640 10 client.4926.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:14.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.905+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign -1 7 2024-03-27T21:41:14.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.905+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f80090e0 osd=3 3 2024-03-27T21:41:14.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.905+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f80090e0 osd=3 3 2024-03-27T21:41:14.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.905+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f80090e0 osd=3 3 2024-03-27T21:41:14.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.905+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_remove -1 7 2024-03-27T21:41:14.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.905+0000 7fe204ff9640 20 client.4926.objecter get_session s=0x7fe1f80090e0 osd=3 3 2024-03-27T21:41:14.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.905+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign 3 7 2024-03-27T21:41:14.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.905+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f80090e0 osd=3 4 2024-03-27T21:41:14.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.905+0000 7fe204ff9640 10 client.4926.objecter _send_command 7 2024-03-27T21:41:14.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.905+0000 7fe204ff9640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe1f800e1d0 con 0x7fe1f800d330 2024-03-27T21:41:14.895 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:41:14.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.908+0000 7fe206ffd640 1 -- 172.21.15.60:0/1148298670 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe1f800e1d0 con 0x7fe1f800d330 2024-03-27T21:41:14.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.908+0000 7fe206ffd640 10 client.4926.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:14.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.908+0000 7fe206ffd640 10 client.4926.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:14.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.908+0000 7fe206ffd640 20 client.4926.objecter put_session s=0x7fe1f80090e0 osd=3 4 2024-03-27T21:41:14.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.908+0000 7fe206ffd640 15 client.4926.objecter _session_command_op_remove 3 7 2024-03-27T21:41:14.896 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:41:14.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.908+0000 7fe204ff9640 10 client.4926.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:14.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.908+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign -1 8 2024-03-27T21:41:14.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.908+0000 7fe204ff9640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fe1f80085a0 0x7fe1f8013840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:14.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.908+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f8008430 osd=4 3 2024-03-27T21:41:14.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.908+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f8008430 osd=4 3 2024-03-27T21:41:14.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.908+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f8008430 osd=4 3 2024-03-27T21:41:14.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.908+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_remove -1 8 2024-03-27T21:41:14.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.908+0000 7fe204ff9640 20 client.4926.objecter get_session s=0x7fe1f8008430 osd=4 3 2024-03-27T21:41:14.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.908+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign 4 8 2024-03-27T21:41:14.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.908+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f8008430 osd=4 4 2024-03-27T21:41:14.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.908+0000 7fe204ff9640 10 client.4926.objecter _send_command 8 2024-03-27T21:41:14.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.908+0000 7fe204ff9640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1f800ca20 con 0x7fe1f80085a0 2024-03-27T21:41:14.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.909+0000 7fe216b14640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fe1f80085a0 0x7fe1f8013840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:14.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.909+0000 7fe216b14640 10 monclient: get_auth_request con 0x7fe1f80085a0 auth_method 0 2024-03-27T21:41:14.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.909+0000 7fe206ffd640 10 client.4926.objecter ms_handle_connect 0x7fe1f80085a0 2024-03-27T21:41:14.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.909+0000 7fe216b14640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fe1f80085a0 0x7fe1f8013840 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:14.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.910+0000 7fe206ffd640 1 -- 172.21.15.60:0/1148298670 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe1f800ca20 con 0x7fe1f80085a0 2024-03-27T21:41:14.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.910+0000 7fe206ffd640 10 client.4926.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:41:14.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.910+0000 7fe206ffd640 10 client.4926.objecter _finish_command 8 = system:0 2024-03-27T21:41:14.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.910+0000 7fe206ffd640 20 client.4926.objecter put_session s=0x7fe1f8008430 osd=4 4 2024-03-27T21:41:14.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.910+0000 7fe206ffd640 15 client.4926.objecter _session_command_op_remove 4 8 2024-03-27T21:41:14.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.927+0000 7fe204ff9640 10 client.4926.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:14.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.927+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign -1 9 2024-03-27T21:41:14.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.927+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f8008430 osd=4 3 2024-03-27T21:41:14.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.927+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f8008430 osd=4 3 2024-03-27T21:41:14.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.927+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f8008430 osd=4 3 2024-03-27T21:41:14.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.927+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_remove -1 9 2024-03-27T21:41:14.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.927+0000 7fe204ff9640 20 client.4926.objecter get_session s=0x7fe1f8008430 osd=4 3 2024-03-27T21:41:14.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.927+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign 4 9 2024-03-27T21:41:14.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.927+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f8008430 osd=4 4 2024-03-27T21:41:14.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.927+0000 7fe204ff9640 10 client.4926.objecter _send_command 9 2024-03-27T21:41:14.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.927+0000 7fe204ff9640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe1f800cc20 con 0x7fe1f80085a0 2024-03-27T21:41:14.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.929+0000 7fe206ffd640 1 -- 172.21.15.60:0/1148298670 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe1f800cc20 con 0x7fe1f80085a0 2024-03-27T21:41:14.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.929+0000 7fe206ffd640 10 client.4926.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:14.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.929+0000 7fe206ffd640 10 client.4926.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:14.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.929+0000 7fe206ffd640 20 client.4926.objecter put_session s=0x7fe1f8008430 osd=4 4 2024-03-27T21:41:14.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.929+0000 7fe206ffd640 15 client.4926.objecter _session_command_op_remove 4 9 2024-03-27T21:41:14.916 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:41:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:41:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.930+0000 7fe204ff9640 10 client.4926.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.930+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign -1 10 2024-03-27T21:41:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.930+0000 7fe204ff9640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fe1f800c050 0x7fe1f8016f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.930+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f800bee0 osd=5 3 2024-03-27T21:41:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.930+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f800bee0 osd=5 3 2024-03-27T21:41:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.930+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f800bee0 osd=5 3 2024-03-27T21:41:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.930+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_remove -1 10 2024-03-27T21:41:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.930+0000 7fe204ff9640 20 client.4926.objecter get_session s=0x7fe1f800bee0 osd=5 3 2024-03-27T21:41:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.930+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign 5 10 2024-03-27T21:41:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.930+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f800bee0 osd=5 4 2024-03-27T21:41:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.930+0000 7fe204ff9640 10 client.4926.objecter _send_command 10 2024-03-27T21:41:14.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.930+0000 7fe204ff9640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1f8011540 con 0x7fe1f800c050 2024-03-27T21:41:14.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.930+0000 7fe216313640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fe1f800c050 0x7fe1f8016f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:14.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.930+0000 7fe216313640 10 monclient: get_auth_request con 0x7fe1f800c050 auth_method 0 2024-03-27T21:41:14.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.930+0000 7fe216313640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fe1f800c050 0x7fe1f8016f10 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:14.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.931+0000 7fe206ffd640 10 client.4926.objecter ms_handle_connect 0x7fe1f800c050 2024-03-27T21:41:14.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.931+0000 7fe206ffd640 1 -- 172.21.15.60:0/1148298670 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe1f8011540 con 0x7fe1f800c050 2024-03-27T21:41:14.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.931+0000 7fe206ffd640 10 client.4926.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:41:14.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.931+0000 7fe206ffd640 10 client.4926.objecter _finish_command 10 = system:0 2024-03-27T21:41:14.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.931+0000 7fe206ffd640 20 client.4926.objecter put_session s=0x7fe1f800bee0 osd=5 4 2024-03-27T21:41:14.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.931+0000 7fe206ffd640 15 client.4926.objecter _session_command_op_remove 5 10 2024-03-27T21:41:14.935 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:14.948+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:14.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.951+0000 7fe204ff9640 10 client.4926.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:14.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.951+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign -1 11 2024-03-27T21:41:14.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.951+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f800bee0 osd=5 3 2024-03-27T21:41:14.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.951+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f800bee0 osd=5 3 2024-03-27T21:41:14.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.951+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f800bee0 osd=5 3 2024-03-27T21:41:14.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.951+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_remove -1 11 2024-03-27T21:41:14.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.951+0000 7fe204ff9640 20 client.4926.objecter get_session s=0x7fe1f800bee0 osd=5 3 2024-03-27T21:41:14.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.951+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign 5 11 2024-03-27T21:41:14.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.951+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f800bee0 osd=5 4 2024-03-27T21:41:14.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.951+0000 7fe204ff9640 10 client.4926.objecter _send_command 11 2024-03-27T21:41:14.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.951+0000 7fe204ff9640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe1f8011540 con 0x7fe1f800c050 2024-03-27T21:41:14.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.954+0000 7fe206ffd640 1 -- 172.21.15.60:0/1148298670 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe1f8011540 con 0x7fe1f800c050 2024-03-27T21:41:14.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.954+0000 7fe206ffd640 10 client.4926.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:14.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.954+0000 7fe206ffd640 10 client.4926.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:14.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.954+0000 7fe206ffd640 20 client.4926.objecter put_session s=0x7fe1f800bee0 osd=5 4 2024-03-27T21:41:14.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.954+0000 7fe206ffd640 15 client.4926.objecter _session_command_op_remove 5 11 2024-03-27T21:41:14.942 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:41:14.944 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:41:14.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.955+0000 7fe204ff9640 10 client.4926.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:14.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.955+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign -1 12 2024-03-27T21:41:14.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.955+0000 7fe204ff9640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fe1f8017960 0x7fe1f8019d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:14.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.955+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f8011540 osd=6 3 2024-03-27T21:41:14.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.955+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f8011540 osd=6 3 2024-03-27T21:41:14.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.955+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f8011540 osd=6 3 2024-03-27T21:41:14.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.955+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_remove -1 12 2024-03-27T21:41:14.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.955+0000 7fe204ff9640 20 client.4926.objecter get_session s=0x7fe1f8011540 osd=6 3 2024-03-27T21:41:14.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.955+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign 6 12 2024-03-27T21:41:14.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.955+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f8011540 osd=6 4 2024-03-27T21:41:14.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.955+0000 7fe204ff9640 10 client.4926.objecter _send_command 12 2024-03-27T21:41:14.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.955+0000 7fe204ff9640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1f801cc50 con 0x7fe1f8017960 2024-03-27T21:41:14.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.956+0000 7fe215b12640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fe1f8017960 0x7fe1f8019d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:14.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.956+0000 7fe215b12640 10 monclient: get_auth_request con 0x7fe1f8017960 auth_method 0 2024-03-27T21:41:14.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.956+0000 7fe215b12640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fe1f8017960 0x7fe1f8019d50 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:14.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.956+0000 7fe206ffd640 10 client.4926.objecter ms_handle_connect 0x7fe1f8017960 2024-03-27T21:41:14.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.957+0000 7fe206ffd640 1 -- 172.21.15.60:0/1148298670 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe1f801cc50 con 0x7fe1f8017960 2024-03-27T21:41:14.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.957+0000 7fe206ffd640 10 client.4926.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:41:14.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.957+0000 7fe206ffd640 10 client.4926.objecter _finish_command 12 = system:0 2024-03-27T21:41:14.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.957+0000 7fe206ffd640 20 client.4926.objecter put_session s=0x7fe1f8011540 osd=6 4 2024-03-27T21:41:14.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.957+0000 7fe206ffd640 15 client.4926.objecter _session_command_op_remove 6 12 2024-03-27T21:41:14.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.971+0000 7fe204ff9640 10 client.4926.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:14.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.971+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign -1 13 2024-03-27T21:41:14.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.971+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f8011540 osd=6 3 2024-03-27T21:41:14.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.971+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f8011540 osd=6 3 2024-03-27T21:41:14.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.971+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f8011540 osd=6 3 2024-03-27T21:41:14.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.971+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_remove -1 13 2024-03-27T21:41:14.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.971+0000 7fe204ff9640 20 client.4926.objecter get_session s=0x7fe1f8011540 osd=6 3 2024-03-27T21:41:14.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.971+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign 6 13 2024-03-27T21:41:14.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.971+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f8011540 osd=6 4 2024-03-27T21:41:14.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.971+0000 7fe204ff9640 10 client.4926.objecter _send_command 13 2024-03-27T21:41:14.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.971+0000 7fe204ff9640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe1f801cc50 con 0x7fe1f8017960 2024-03-27T21:41:14.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.973+0000 7fe206ffd640 1 -- 172.21.15.60:0/1148298670 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe1f801cc50 con 0x7fe1f8017960 2024-03-27T21:41:14.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.973+0000 7fe206ffd640 10 client.4926.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:14.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.973+0000 7fe206ffd640 10 client.4926.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:14.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.973+0000 7fe206ffd640 20 client.4926.objecter put_session s=0x7fe1f8011540 osd=6 4 2024-03-27T21:41:14.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.973+0000 7fe206ffd640 15 client.4926.objecter _session_command_op_remove 6 13 2024-03-27T21:41:14.966 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:41:14.968 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:41:14.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.973+0000 7fe204ff9640 10 client.4926.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:14.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.973+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign -1 14 2024-03-27T21:41:14.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.973+0000 7fe204ff9640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fe1f801bf80 0x7fe1f801c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:14.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.974+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f801cc50 osd=7 3 2024-03-27T21:41:14.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.974+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f801cc50 osd=7 3 2024-03-27T21:41:14.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.974+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f801cc50 osd=7 3 2024-03-27T21:41:14.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.974+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_remove -1 14 2024-03-27T21:41:14.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.974+0000 7fe204ff9640 20 client.4926.objecter get_session s=0x7fe1f801cc50 osd=7 3 2024-03-27T21:41:14.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.974+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign 7 14 2024-03-27T21:41:14.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.974+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f801cc50 osd=7 4 2024-03-27T21:41:14.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.974+0000 7fe204ff9640 10 client.4926.objecter _send_command 14 2024-03-27T21:41:14.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.974+0000 7fe204ff9640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1f8020010 con 0x7fe1f801bf80 2024-03-27T21:41:14.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.974+0000 7fe216b14640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fe1f801bf80 0x7fe1f801c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:14.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.974+0000 7fe216b14640 10 monclient: get_auth_request con 0x7fe1f801bf80 auth_method 0 2024-03-27T21:41:14.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.974+0000 7fe206ffd640 10 client.4926.objecter ms_handle_connect 0x7fe1f801bf80 2024-03-27T21:41:14.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.974+0000 7fe216b14640 1 --2- 172.21.15.60:0/1148298670 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fe1f801bf80 0x7fe1f801c3e0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:14.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.975+0000 7fe206ffd640 1 -- 172.21.15.60:0/1148298670 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe1f8020010 con 0x7fe1f801bf80 2024-03-27T21:41:14.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.975+0000 7fe206ffd640 10 client.4926.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:41:14.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.975+0000 7fe206ffd640 10 client.4926.objecter _finish_command 14 = system:0 2024-03-27T21:41:14.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.975+0000 7fe206ffd640 20 client.4926.objecter put_session s=0x7fe1f801cc50 osd=7 4 2024-03-27T21:41:14.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.975+0000 7fe206ffd640 15 client.4926.objecter _session_command_op_remove 7 14 2024-03-27T21:41:14.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.989+0000 7fe204ff9640 10 client.4926.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:14.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.989+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign -1 15 2024-03-27T21:41:14.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.989+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f801cc50 osd=7 3 2024-03-27T21:41:14.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.989+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f801cc50 osd=7 3 2024-03-27T21:41:14.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.989+0000 7fe204ff9640 20 client.4926.objecter _get_session s=0x7fe1f801cc50 osd=7 3 2024-03-27T21:41:14.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.989+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_remove -1 15 2024-03-27T21:41:14.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.989+0000 7fe204ff9640 20 client.4926.objecter get_session s=0x7fe1f801cc50 osd=7 3 2024-03-27T21:41:14.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.989+0000 7fe204ff9640 15 client.4926.objecter _session_command_op_assign 7 15 2024-03-27T21:41:14.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.989+0000 7fe204ff9640 20 client.4926.objecter put_session s=0x7fe1f801cc50 osd=7 4 2024-03-27T21:41:14.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.989+0000 7fe204ff9640 10 client.4926.objecter _send_command 15 2024-03-27T21:41:14.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.989+0000 7fe204ff9640 1 -- 172.21.15.60:0/1148298670 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe1f801a2e0 con 0x7fe1f801bf80 2024-03-27T21:41:14.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.991+0000 7fe206ffd640 1 -- 172.21.15.60:0/1148298670 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe1f801a2e0 con 0x7fe1f801bf80 2024-03-27T21:41:14.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.991+0000 7fe206ffd640 10 client.4926.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:14.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.991+0000 7fe206ffd640 10 client.4926.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:14.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.991+0000 7fe206ffd640 20 client.4926.objecter put_session s=0x7fe1f801cc50 osd=7 4 2024-03-27T21:41:14.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:14.991+0000 7fe206ffd640 15 client.4926.objecter _session_command_op_remove 7 15 2024-03-27T21:41:14.979 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:41:14.979 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:41:14.983 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:41:14.983 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:41:15.036 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:15.050+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:15.061 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:15.061 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:15.062 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:41:15.062 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:41:15.062 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:15.073 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:41:15.137 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:15.151+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:15.181 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:15.181 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:41:15.181 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:41:15.181 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:41:15.181 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:15.190 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:41:15.238 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:15.252+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:15.306 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:15.306 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:15.306 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:15.306 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:15.315 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:41:15.358 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:15.373+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:15.421 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:15.421 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:15.421 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:15.421 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:15.421 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:15.429 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:41:15.439 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:15.453+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:15.540 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:15.554+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:15.574 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:15.574 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:15.574 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:15.574 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:41:15.574 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:15.583 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:41:15.641 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:15.656+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:15.688 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:15.689 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:15.689 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:15.689 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:15.697 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:41:15.742 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:15.756+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:15.842 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:15.842 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:15.843 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:15.843 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:15.843 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:15.843 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:15.857+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:15.851 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:41:15.944 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:15.957+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:15.955 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:15.955 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:15.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:15.956 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:41:15.956 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:15.964 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:41:15.997 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-27T21:41:16.046 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:16.059+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:16.071 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:16.071 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:16.071 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:16.071 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:16.080 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:41:16.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.129+0000 7ff9df167640 10 monclient: get_monmap_and_config 2024-03-27T21:41:16.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.129+0000 7ff9df167640 10 monclient: build_initial_monmap 2024-03-27T21:41:16.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.129+0000 7ff9df167640 10 monclient: monmap: 2024-03-27T21:41:16.116 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:16.116 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:16.116 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:16.130298+0000 2024-03-27T21:41:16.116 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:16.130298+0000 2024-03-27T21:41:16.117 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:16.117 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:16.117 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:16.117 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:16.117 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:16.117 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:16.118 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:16.118 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:16.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.130+0000 7ff9df167640 1 Processor -- start 2024-03-27T21:41:16.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.130+0000 7ff9df167640 1 -- start start 2024-03-27T21:41:16.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.130+0000 7ff9df167640 10 monclient: init 2024-03-27T21:41:16.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.130+0000 7ff9df167640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:16.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.130+0000 7ff9df167640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-03-27T21:41:16.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.130+0000 7ff9df167640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff9d8071a50 0x7ff9d80a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:16.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.130+0000 7ff9df167640 10 monclient(hunting): picked mon.noname-a con 0x7ff9d8071a50 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:16.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.130+0000 7ff9df167640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff9d813e810 0x7ff9d813ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:16.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.130+0000 7ff9df167640 10 monclient(hunting): picked mon.noname-c con 0x7ff9d813e810 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:16.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.130+0000 7ff9df167640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9d8141630 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:16.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.130+0000 7ff9df167640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d813f190 con 0x7ff9d8141630 2024-03-27T21:41:16.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.130+0000 7ff9df167640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:16.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.130+0000 7ff9df167640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d8071e30 con 0x7ff9d8071a50 2024-03-27T21:41:16.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.130+0000 7ff9df167640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:16.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.130+0000 7ff9df167640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d80a8e90 con 0x7ff9d813e810 2024-03-27T21:41:16.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.130+0000 7ff9df167640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:16.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.130+0000 7ff9df167640 10 monclient(hunting): authenticate will time out at 917.351718s 2024-03-27T21:41:16.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.130+0000 7ff9dd704640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff9d8141630 0x7ff9d8141aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40232/0 (socket says 172.21.15.60:40232) 2024-03-27T21:41:16.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.130+0000 7ff9dcf03640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff9d8071a50 0x7ff9d80a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:16.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.130+0000 7ff9dd704640 1 -- 172.21.15.60:0/3251014510 learned_addr learned my addr 172.21.15.60:0/3251014510 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:16.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9dcf03640 10 monclient(hunting): get_auth_request con 0x7ff9d8071a50 auth_method 0 2024-03-27T21:41:16.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9dcf03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:16.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9dcf03640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:16.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9dcf03640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:16.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9dcf03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:16.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9dcf03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:16.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9d7fff640 1 --2- 172.21.15.60:0/3251014510 >> v2:172.21.15.191:3300/0 conn(0x7ff9d813e810 0x7ff9d813ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:16.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9dcf03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:16.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9d7fff640 10 monclient(hunting): get_auth_request con 0x7ff9d813e810 auth_method 0 2024-03-27T21:41:16.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:16.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:16.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:16.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:16.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:16.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:16.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9dcf03640 10 monclient(hunting): handle_auth_done global_id 4929 payload 274 2024-03-27T21:41:16.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9dcf03640 1 -- 172.21.15.60:0/3251014510 >> v2:172.21.15.191:3300/0 conn(0x7ff9d813e810 msgr2=0x7ff9d813ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:16.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9dcf03640 1 --2- 172.21.15.60:0/3251014510 >> v2:172.21.15.191:3300/0 conn(0x7ff9d813e810 0x7ff9d813ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:16.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9dcf03640 1 -- 172.21.15.60:0/3251014510 >> v1:172.21.15.60:6789/0 conn(0x7ff9d8141630 legacy=0x7ff9d8141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:16.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9dcf03640 10 monclient: _finish_hunting 0 2024-03-27T21:41:16.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9dcf03640 1 monclient: found mon.noname-a 2024-03-27T21:41:16.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9dcf03640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:16.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9dcf03640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:16.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.131+0000 7ff9dcf03640 1 -- 172.21.15.60:0/3251014510 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d80aa040 con 0x7ff9d8071a50 2024-03-27T21:41:16.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.132+0000 7ff9dcf03640 1 --2- 172.21.15.60:0/3251014510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9d8071a50 0x7ff9d80a87e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff9d80709f0 tx=0x7ff9cc02b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64c71f6a06a40242 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:16.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.132+0000 7ff9d77fe640 1 -- 172.21.15.60:0/3251014510 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc030710 con 0x7ff9d8071a50 2024-03-27T21:41:16.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.132+0000 7ff9d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:16.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.132+0000 7ff9d77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:16.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.132+0000 7ff9d77fe640 10 monclient: dump: 2024-03-27T21:41:16.145 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:16.145 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:16.145 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:16.145 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:16.145 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:16.145 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:16.145 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:16.145 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:16.145 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:16.145 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:16.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.132+0000 7ff9d77fe640 10 monclient: _finish_auth 0 2024-03-27T21:41:16.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.132+0000 7ff9d77fe640 10 monclient: _check_auth_tickets 2024-03-27T21:41:16.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.132+0000 7ff9d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:16.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.132+0000 7ff9df167640 5 monclient: authenticate success, global_id 4929 2024-03-27T21:41:16.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.132+0000 7ff9df167640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:16.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.132+0000 7ff9d77fe640 1 -- 172.21.15.60:0/3251014510 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9cc02e050 con 0x7ff9d8071a50 2024-03-27T21:41:16.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.132+0000 7ff9d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:16.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.132+0000 7ff9df167640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:16.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.132+0000 7ff9df167640 4 set_mon_vals no callback set 2024-03-27T21:41:16.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.132+0000 7ff9d77fe640 1 -- 172.21.15.60:0/3251014510 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc030a90 con 0x7ff9d8071a50 2024-03-27T21:41:16.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.132+0000 7ff9d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:16.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.132+0000 7ff9d77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:16.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.132+0000 7ff9d77fe640 10 monclient: dump: 2024-03-27T21:41:16.146 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:16.146 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:16.146 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:16.146 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:16.146 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:16.147 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:16.147 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:16.147 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:16.147 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:16.147 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:16.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.133+0000 7ff9df167640 10 monclient: shutdown 2024-03-27T21:41:16.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.133+0000 7ff9df167640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:16.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.133+0000 7ff9df167640 1 -- 172.21.15.60:0/3251014510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9d8071a50 msgr2=0x7ff9d80a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:16.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.133+0000 7ff9df167640 1 --2- 172.21.15.60:0/3251014510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9d8071a50 0x7ff9d80a87e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff9d80709f0 tx=0x7ff9cc02b710 comp rx=0 tx=0).stop 2024-03-27T21:41:16.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.133+0000 7ff9df167640 1 -- 172.21.15.60:0/3251014510 shutdown_connections 2024-03-27T21:41:16.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.133+0000 7ff9df167640 1 --2- 172.21.15.60:0/3251014510 >> v2:172.21.15.191:3300/0 conn(0x7ff9d813e810 0x7ff9d813ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:16.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.133+0000 7ff9df167640 1 --2- 172.21.15.60:0/3251014510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9d8071a50 0x7ff9d80a87e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:16.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.133+0000 7ff9df167640 1 -- 172.21.15.60:0/3251014510 >> 172.21.15.60:0/3251014510 conn(0x7ff9d80b7300 msgr2=0x7ff9d80b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:16.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.133+0000 7ff9df167640 1 -- 172.21.15.60:0/3251014510 shutdown_connections 2024-03-27T21:41:16.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.133+0000 7ff9df167640 1 -- 172.21.15.60:0/3251014510 wait complete. 2024-03-27T21:41:16.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.133+0000 7ff9df167640 10 monclient: build_initial_monmap 2024-03-27T21:41:16.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.133+0000 7ff9df167640 10 monclient: monmap: 2024-03-27T21:41:16.148 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:16.148 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:16.148 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:16.148 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:16.148 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:16.148 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:16.148 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:16.148 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:16.148 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:16.148 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:16.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.134+0000 7ff9df167640 1 librados: starting msgr at 2024-03-27T21:41:16.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.134+0000 7ff9df167640 1 librados: starting objecter 2024-03-27T21:41:16.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.134+0000 7ff9df167640 1 Processor -- start 2024-03-27T21:41:16.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.134+0000 7ff9df167640 1 -- start start 2024-03-27T21:41:16.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.134+0000 7ff9df167640 1 librados: setting wanted keys 2024-03-27T21:41:16.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.134+0000 7ff9df167640 1 librados: calling monclient init 2024-03-27T21:41:16.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.134+0000 7ff9df167640 10 monclient: init 2024-03-27T21:41:16.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9df167640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:16.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9df167640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:41:16.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9df167640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9d8071a50 0x7ff9d81d65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:16.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9df167640 10 monclient(hunting): picked mon.noname-a con 0x7ff9d8071a50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:16.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9df167640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9d813e810 0x7ff9d813ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:16.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9df167640 10 monclient(hunting): picked mon.noname-b con 0x7ff9d813e810 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:16.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9df167640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff9d8141630 0x7ff9d8132f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:16.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9df167640 10 monclient(hunting): picked mon.noname-c con 0x7ff9d8141630 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:16.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9df167640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:16.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9df167640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d8139fb0 con 0x7ff9d8071a50 2024-03-27T21:41:16.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9df167640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:16.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9df167640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d8139290 con 0x7ff9d813e810 2024-03-27T21:41:16.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9df167640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:16.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9df167640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d813a5f0 con 0x7ff9d8141630 2024-03-27T21:41:16.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9df167640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:16.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9df167640 10 monclient(hunting): authenticate will time out at 917.356022s 2024-03-27T21:41:16.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d7fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9d813e810 0x7ff9d813ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:16.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d7fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9d813e810 0x7ff9d813ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:40286/0 (socket says 172.21.15.60:40286) 2024-03-27T21:41:16.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d7fff640 1 -- 172.21.15.60:0/428622537 learned_addr learned my addr 172.21.15.60:0/428622537 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:16.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d7fff640 10 monclient(hunting): get_auth_request con 0x7ff9d813e810 auth_method 0 2024-03-27T21:41:16.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:16.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:16.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:16.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:16.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:16.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:16.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9dd704640 1 --2- 172.21.15.60:0/428622537 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff9d8141630 0x7ff9d8132f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:16.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9dd704640 10 monclient(hunting): get_auth_request con 0x7ff9d8141630 auth_method 0 2024-03-27T21:41:16.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9dd704640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:16.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9dd704640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:16.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9dd704640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:16.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9dcf03640 1 --2- 172.21.15.60:0/428622537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9d8071a50 0x7ff9d81d65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:16.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9dcf03640 10 monclient(hunting): get_auth_request con 0x7ff9d8071a50 auth_method 0 2024-03-27T21:41:16.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9dcf03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:16.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9dcf03640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:16.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9dcf03640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:16.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d7fff640 10 monclient(hunting): handle_auth_done global_id 4864 payload 824 2024-03-27T21:41:16.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d7fff640 1 -- 172.21.15.60:0/428622537 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff9d8141630 msgr2=0x7ff9d8132f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:16.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d7fff640 1 --2- 172.21.15.60:0/428622537 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff9d8141630 0x7ff9d8132f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:16.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d7fff640 1 -- 172.21.15.60:0/428622537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9d8071a50 msgr2=0x7ff9d81d65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:16.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d7fff640 1 --2- 172.21.15.60:0/428622537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9d8071a50 0x7ff9d81d65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:16.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d7fff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:16.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d7fff640 1 monclient: found mon.noname-b 2024-03-27T21:41:16.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:16.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:41:16.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d7fff640 1 -- 172.21.15.60:0/428622537 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9cc009400 con 0x7ff9d813e810 2024-03-27T21:41:16.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d7fff640 1 --2- 172.21.15.60:0/428622537 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9d813e810 0x7ff9d813ab10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400e9e0 tx=0x7ff9c400eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:16.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d57fa640 10 client.?.objecter ms_handle_connect 0x7ff9d813e810 2024-03-27T21:41:16.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d57fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:16.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d57fa640 1 -- 172.21.15.60:0/428622537 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c400c9d0 con 0x7ff9d813e810 2024-03-27T21:41:16.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:16.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:16.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.135+0000 7ff9d57fa640 10 monclient: dump: 2024-03-27T21:41:16.153 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:16.153 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:16.153 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:16.153 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:16.153 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:16.153 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:16.153 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:16.153 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:16.153 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:16.153 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:16.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.136+0000 7ff9d57fa640 10 monclient: _finish_auth 0 2024-03-27T21:41:16.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.136+0000 7ff9d57fa640 10 monclient: _check_auth_tickets 2024-03-27T21:41:16.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.136+0000 7ff9d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:16.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.136+0000 7ff9d57fa640 1 -- 172.21.15.60:0/428622537 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c400cb30 con 0x7ff9d813e810 2024-03-27T21:41:16.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.136+0000 7ff9d57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:16.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.136+0000 7ff9d57fa640 1 -- 172.21.15.60:0/428622537 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c400cee0 con 0x7ff9d813e810 2024-03-27T21:41:16.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.136+0000 7ff9d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:16.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.136+0000 7ff9d57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:16.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.136+0000 7ff9d57fa640 10 monclient: dump: 2024-03-27T21:41:16.154 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:16.154 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:16.154 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:16.154 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:16.154 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:16.154 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:16.155 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.136+0000 7ff9d6ffd640 4 set_mon_vals no callback set 2024-03-27T21:41:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.136+0000 7ff9df167640 5 monclient: authenticate success, global_id 4864 2024-03-27T21:41:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.136+0000 7ff9df167640 10 monclient: _renew_subs 2024-03-27T21:41:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.136+0000 7ff9df167640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.136+0000 7ff9df167640 1 -- 172.21.15.60:0/428622537 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d813a110 con 0x7ff9d813e810 2024-03-27T21:41:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.136+0000 7ff9df167640 10 client.4864.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.136+0000 7ff9df167640 10 monclient: _renew_subs 2024-03-27T21:41:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.136+0000 7ff9df167640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.136+0000 7ff9df167640 1 -- 172.21.15.60:0/428622537 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d813b230 con 0x7ff9d813e810 2024-03-27T21:41:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.136+0000 7ff9df167640 1 librados: init done 2024-03-27T21:41:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.137+0000 7ff9df167640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.137+0000 7ff9df167640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.137+0000 7ff9df167640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.137+0000 7ff9df167640 1 -- 172.21.15.60:0/428622537 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d8069470 con 0x7ff9d813e810 2024-03-27T21:41:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.137+0000 7ff9d57fa640 1 -- 172.21.15.60:0/428622537 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff9c4013470 con 0x7ff9d813e810 2024-03-27T21:41:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.138+0000 7ff9d57fa640 1 --2- 172.21.15.60:0/428622537 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff9bc06c590 0x7ff9bc06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.138+0000 7ff9d57fa640 1 -- 172.21.15.60:0/428622537 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5242+0+0 (secure 0 0 0) 0x7ff9c40893e0 con 0x7ff9d813e810 2024-03-27T21:41:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.138+0000 7ff9d57fa640 10 client.4864.objecter ms_dispatch 0x7ff9d8000b90 osd_map(55..55 src has 1..55) v4 2024-03-27T21:41:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.138+0000 7ff9d57fa640 3 client.4864.objecter handle_osd_map got epochs [55,55] > 0 2024-03-27T21:41:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.138+0000 7ff9d57fa640 3 client.4864.objecter handle_osd_map decoding full epoch 55 2024-03-27T21:41:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.138+0000 7ff9dcf03640 1 --2- 172.21.15.60:0/428622537 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff9bc06c590 0x7ff9bc06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.138+0000 7ff9dcf03640 10 monclient: get_auth_request con 0x7ff9bc06c590 auth_method 0 2024-03-27T21:41:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.138+0000 7ff9d57fa640 20 client.4864.objecter dump_active .. 0 homeless 2024-03-27T21:41:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.138+0000 7ff9d57fa640 10 client.4864.objecter ms_handle_connect 0x7ff9bc06c590 2024-03-27T21:41:16.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.138+0000 7ff9dcf03640 1 --2- 172.21.15.60:0/428622537 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff9bc06c590 0x7ff9bc06ea50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc034000 tx=0x7ff9cc008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:16.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.142+0000 7ff9d57fa640 1 -- 172.21.15.60:0/428622537 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff9c4054790 con 0x7ff9d813e810 2024-03-27T21:41:16.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.142+0000 7ff9d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:16.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.142+0000 7ff9d57fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:16.172 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:16.187+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:16.228 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:16.229 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:16.229 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:16.229 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:16.229 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:16.237 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:41:16.279 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:16.293+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:16.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.333+0000 7ff9df167640 10 start_mon_command cmd=[{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:41:16.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.333+0000 7ff9df167640 10 monclient: _send_command 2 [{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:41:16.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.333+0000 7ff9df167640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:16.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:16.333+0000 7ff9df167640 1 -- 172.21.15.60:0/428622537 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7ff9d80a9260 con 0x7ff9d813e810 2024-03-27T21:41:16.359 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:16.374+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:16.384 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:16.384 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:16.384 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:16.384 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:41:16.384 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:16.394 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:41:16.460 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:16.474+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:16.539 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:16.539 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:16.540 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:16.540 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:16.548 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:41:16.561 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:16.575+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:16.653 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:16.653 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:16.653 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:16.653 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:16.653 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:16.662 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:41:16.662 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:16.675+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:16.763 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:16.776+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:16.809 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:16.809 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:16.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:16.810 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-03-27T21:41:16.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.810 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:16.810 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:16.810 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 97.389827005000001, 2024-03-27T21:41:16.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43537978700000002, 2024-03-27T21:41:16.810 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:16.810 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:16.810 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:16.810 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:16.810 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:16.811 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-03-27T21:41:16.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.811 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:16.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:16.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:16.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:16.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:16.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:16.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432019+0000", 2024-03-27T21:41:16.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-03-27T21:41:16.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:16.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432027+0000", 2024-03-27T21:41:16.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1200000000000002e-06 2024-03-27T21:41:16.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:16.813 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432039+0000", 2024-03-27T21:41:16.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1123e-05 2024-03-27T21:41:16.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.813 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:16.813 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432051+0000", 2024-03-27T21:41:16.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-03-27T21:41:16.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.813 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:16.813 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432157+0000", 2024-03-27T21:41:16.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106846 2024-03-27T21:41:16.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-03-27T21:41:16.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432176+0000", 2024-03-27T21:41:16.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8490000000000001e-05 2024-03-27T21:41:16.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:16.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866170+0000", 2024-03-27T21:41:16.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43399370100000001 2024-03-27T21:41:16.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:16.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866335+0000", 2024-03-27T21:41:16.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016559699999999999 2024-03-27T21:41:16.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:16.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866495+0000", 2024-03-27T21:41:16.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015944200000000001 2024-03-27T21:41:16.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:16.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867012+0000", 2024-03-27T21:41:16.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0005176 2024-03-27T21:41:16.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:16.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867334+0000", 2024-03-27T21:41:16.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032158600000000001 2024-03-27T21:41:16.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:16.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867374+0000", 2024-03-27T21:41:16.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0547000000000003e-05 2024-03-27T21:41:16.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:16.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867400+0000", 2024-03-27T21:41:16.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5958000000000001e-05 2024-03-27T21:41:16.817 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.817 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:16.817 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.817 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:16.817 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:16.817 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 96.892444553999994, 2024-03-27T21:41:16.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064560310000000001, 2024-03-27T21:41:16.818 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:16.818 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:16.818 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:16.818 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:16.818 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:16.818 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 70 2024-03-27T21:41:16.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.818 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:16.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.818 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:16.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:16.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:16.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:16.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:16.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929402+0000", 2024-03-27T21:41:16.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:16.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:16.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929407+0000", 2024-03-27T21:41:16.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-06 2024-03-27T21:41:16.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:16.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929414+0000", 2024-03-27T21:41:16.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5689999999999999e-06 2024-03-27T21:41:16.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:16.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929419+0000", 2024-03-27T21:41:16.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-03-27T21:41:16.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:16.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930245+0000", 2024-03-27T21:41:16.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082521500000000002 2024-03-27T21:41:16.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:16.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930287+0000", 2024-03-27T21:41:16.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2305000000000002e-05 2024-03-27T21:41:16.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:16.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930367+0000", 2024-03-27T21:41:16.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9589e-05 2024-03-27T21:41:16.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.822 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:16.822 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.931214+0000", 2024-03-27T21:41:16.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000847827 2024-03-27T21:41:16.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.822 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:16.822 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935807+0000", 2024-03-27T21:41:16.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045923780000000003 2024-03-27T21:41:16.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:16.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935840+0000", 2024-03-27T21:41:16.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3501999999999999e-05 2024-03-27T21:41:16.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:16.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935859+0000", 2024-03-27T21:41:16.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8845e-05 2024-03-27T21:41:16.824 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.824 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:16.824 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.824 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:16.824 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:16.825 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 96.885184963, 2024-03-27T21:41:16.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071695429999999996, 2024-03-27T21:41:16.825 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:16.825 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:16.825 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:16.825 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:16.825 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:16.825 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-03-27T21:41:16.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.825 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:16.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:16.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:16.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:16.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:16.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:16.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936662+0000", 2024-03-27T21:41:16.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:16.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:16.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936737+0000", 2024-03-27T21:41:16.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5303999999999996e-05 2024-03-27T21:41:16.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:16.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936746+0000", 2024-03-27T21:41:16.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6340000000000007e-06 2024-03-27T21:41:16.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:16.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936753+0000", 2024-03-27T21:41:16.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-03-27T21:41:16.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.828 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:16.828 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937926+0000", 2024-03-27T21:41:16.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011731280000000001 2024-03-27T21:41:16.828 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.828 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.828 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:16.828 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937969+0000", 2024-03-27T21:41:16.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3285999999999999e-05 2024-03-27T21:41:16.828 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.828 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.828 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:16.828 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938040+0000", 2024-03-27T21:41:16.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0358000000000006e-05 2024-03-27T21:41:16.828 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:16.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939324+0000", 2024-03-27T21:41:16.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012844899999999999 2024-03-27T21:41:16.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:16.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943780+0000", 2024-03-27T21:41:16.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044555630000000001 2024-03-27T21:41:16.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:16.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943811+0000", 2024-03-27T21:41:16.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1415999999999999e-05 2024-03-27T21:41:16.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:16.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943832+0000", 2024-03-27T21:41:16.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0981000000000001e-05 2024-03-27T21:41:16.830 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.830 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:16.830 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.830 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:16.830 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:16.830 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 96.885086947000005, 2024-03-27T21:41:16.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071139740000000003, 2024-03-27T21:41:16.831 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:16.831 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:16.831 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:16.831 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:16.831 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:16.831 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-03-27T21:41:16.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.831 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:16.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:16.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:16.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:16.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:16.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:16.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936760+0000", 2024-03-27T21:41:16.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:16.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:16.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936765+0000", 2024-03-27T21:41:16.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6330000000000004e-06 2024-03-27T21:41:16.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:16.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936771+0000", 2024-03-27T21:41:16.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0560000000000001e-06 2024-03-27T21:41:16.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:16.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936776+0000", 2024-03-27T21:41:16.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1510000000000003e-06 2024-03-27T21:41:16.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:16.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938605+0000", 2024-03-27T21:41:16.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001829589 2024-03-27T21:41:16.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.834 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:16.834 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938657+0000", 2024-03-27T21:41:16.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1860000000000002e-05 2024-03-27T21:41:16.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.834 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:16.834 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938734+0000", 2024-03-27T21:41:16.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6639999999999998e-05 2024-03-27T21:41:16.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.834 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:16.834 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939562+0000", 2024-03-27T21:41:16.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082803100000000001 2024-03-27T21:41:16.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.835 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.835 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:16.835 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943838+0000", 2024-03-27T21:41:16.835 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042757050000000003 2024-03-27T21:41:16.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.835 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.835 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:16.835 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943859+0000", 2024-03-27T21:41:16.835 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1175e-05 2024-03-27T21:41:16.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.836 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:16.836 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943875+0000", 2024-03-27T21:41:16.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5744e-05 2024-03-27T21:41:16.836 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.836 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:16.836 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.836 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:16.836 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:16.836 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 96.741572895999994, 2024-03-27T21:41:16.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059712230000000003, 2024-03-27T21:41:16.836 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:16.836 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:16.836 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:16.837 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:16.837 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:16.837 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 112 2024-03-27T21:41:16.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.837 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:16.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:16.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:16.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:16.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:16.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:16.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080274+0000", 2024-03-27T21:41:16.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:16.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:16.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080318+0000", 2024-03-27T21:41:16.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3634999999999998e-05 2024-03-27T21:41:16.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:16.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080329+0000", 2024-03-27T21:41:16.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1232e-05 2024-03-27T21:41:16.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:16.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080333+0000", 2024-03-27T21:41:16.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6009999999999999e-06 2024-03-27T21:41:16.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:16.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083786+0000", 2024-03-27T21:41:16.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034528940000000002 2024-03-27T21:41:16.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:16.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083861+0000", 2024-03-27T21:41:16.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4893000000000003e-05 2024-03-27T21:41:16.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:16.840 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083981+0000", 2024-03-27T21:41:16.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012058299999999999 2024-03-27T21:41:16.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:16.840 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086148+0000", 2024-03-27T21:41:16.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021671469999999999 2024-03-27T21:41:16.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:16.841 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086195+0000", 2024-03-27T21:41:16.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6768999999999999e-05 2024-03-27T21:41:16.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.841 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.841 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:16.841 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086224+0000", 2024-03-27T21:41:16.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8758000000000001e-05 2024-03-27T21:41:16.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.841 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.841 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:16.841 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086246+0000", 2024-03-27T21:41:16.842 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2064000000000001e-05 2024-03-27T21:41:16.842 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.842 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:16.842 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.842 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.842 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.842 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:16.842 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:16.843 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 96.741509949000005, 2024-03-27T21:41:16.843 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066196320000000003, 2024-03-27T21:41:16.843 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:16.843 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:16.843 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:16.843 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:16.843 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:16.843 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113 2024-03-27T21:41:16.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.844 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:16.844 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.844 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:16.844 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:16.844 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.844 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.844 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.844 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:16.844 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:16.845 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.845 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.845 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.845 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:16.845 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080337+0000", 2024-03-27T21:41:16.845 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:16.845 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.845 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.846 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:16.846 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080340+0000", 2024-03-27T21:41:16.846 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6690000000000002e-06 2024-03-27T21:41:16.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.846 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.846 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:16.846 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080344+0000", 2024-03-27T21:41:16.846 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6550000000000002e-06 2024-03-27T21:41:16.847 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.847 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:16.847 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080348+0000", 2024-03-27T21:41:16.847 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9169999999999999e-06 2024-03-27T21:41:16.847 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.847 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:16.847 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084595+0000", 2024-03-27T21:41:16.848 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042471719999999996 2024-03-27T21:41:16.848 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.848 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.848 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:16.848 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084676+0000", 2024-03-27T21:41:16.848 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1147000000000001e-05 2024-03-27T21:41:16.848 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.848 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.848 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:16.849 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084796+0000", 2024-03-27T21:41:16.849 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012032900000000001 2024-03-27T21:41:16.849 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.849 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.849 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:16.849 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086190+0000", 2024-03-27T21:41:16.849 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013934399999999999 2024-03-27T21:41:16.849 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.849 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.849 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:16.850 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086905+0000", 2024-03-27T21:41:16.850 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071536199999999999 2024-03-27T21:41:16.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.850 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.850 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:16.850 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086937+0000", 2024-03-27T21:41:16.850 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2125000000000002e-05 2024-03-27T21:41:16.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.851 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.851 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:16.851 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086957+0000", 2024-03-27T21:41:16.851 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0157999999999999e-05 2024-03-27T21:41:16.851 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.851 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:16.851 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.851 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.852 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.852 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:16.852 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:16.852 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 96.741497233000004, 2024-03-27T21:41:16.852 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007405661, 2024-03-27T21:41:16.852 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:16.852 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:16.852 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:16.852 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:16.853 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:16.853 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 114 2024-03-27T21:41:16.853 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.853 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:16.853 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.853 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:16.853 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:16.853 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.853 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.854 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.854 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:16.854 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:16.854 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.854 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.854 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.854 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:16.854 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:16.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-03-27T21:41:16.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.855 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:16.855 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080353+0000", 2024-03-27T21:41:16.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.441e-06 2024-03-27T21:41:16.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.855 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:16.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080356+0000", 2024-03-27T21:41:16.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0769999999999999e-06 2024-03-27T21:41:16.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:16.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080358+0000", 2024-03-27T21:41:16.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3369999999999998e-06 2024-03-27T21:41:16.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:16.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085118+0000", 2024-03-27T21:41:16.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047603949999999997 2024-03-27T21:41:16.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:16.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085210+0000", 2024-03-27T21:41:16.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1983000000000003e-05 2024-03-27T21:41:16.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:16.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085332+0000", 2024-03-27T21:41:16.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121803 2024-03-27T21:41:16.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:16.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086292+0000", 2024-03-27T21:41:16.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095961400000000002 2024-03-27T21:41:16.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.858 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:16.858 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087710+0000", 2024-03-27T21:41:16.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014181459999999999 2024-03-27T21:41:16.858 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.858 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.858 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:16.858 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087736+0000", 2024-03-27T21:41:16.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6475e-05 2024-03-27T21:41:16.858 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.858 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.858 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:16.858 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087756+0000", 2024-03-27T21:41:16.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9626000000000001e-05 2024-03-27T21:41:16.858 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.859 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:16.859 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.859 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.859 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.859 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:16.859 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:16.859 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 96.741474797999999, 2024-03-27T21:41:16.859 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075011720000000004, 2024-03-27T21:41:16.859 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:16.859 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:16.859 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:16.859 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:16.859 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:16.859 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115 2024-03-27T21:41:16.859 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.860 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:16.860 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.860 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:16.860 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:16.860 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.860 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.860 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.860 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:16.860 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:16.860 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.860 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.860 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.860 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:16.860 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:16.860 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:16.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.861 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.861 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:16.861 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080458+0000", 2024-03-27T21:41:16.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5228000000000005e-05 2024-03-27T21:41:16.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.861 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.861 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:16.861 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080465+0000", 2024-03-27T21:41:16.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.238e-06 2024-03-27T21:41:16.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.861 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.861 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:16.861 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080468+0000", 2024-03-27T21:41:16.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5109999999999999e-06 2024-03-27T21:41:16.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:16.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086326+0000", 2024-03-27T21:41:16.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058579169999999998 2024-03-27T21:41:16.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:16.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086382+0000", 2024-03-27T21:41:16.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5309999999999997e-05 2024-03-27T21:41:16.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:16.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086457+0000", 2024-03-27T21:41:16.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5413000000000005e-05 2024-03-27T21:41:16.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.863 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:16.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087762+0000", 2024-03-27T21:41:16.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013052300000000001 2024-03-27T21:41:16.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.863 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:16.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087798+0000", 2024-03-27T21:41:16.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5614999999999997e-05 2024-03-27T21:41:16.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.863 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:16.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087840+0000", 2024-03-27T21:41:16.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2196000000000001e-05 2024-03-27T21:41:16.864 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.864 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.864 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:16.864 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087874+0000", 2024-03-27T21:41:16.864 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3785999999999998e-05 2024-03-27T21:41:16.864 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.864 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:16.864 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.864 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.864 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.864 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:16.864 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:16.864 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 96.741374715000006, 2024-03-27T21:41:16.864 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075050680000000002, 2024-03-27T21:41:16.864 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:16.865 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:16.865 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:16.865 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:16.865 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:16.865 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 116 2024-03-27T21:41:16.865 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.865 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:16.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:16.865 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:16.865 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.865 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:16.865 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:16.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.866 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.866 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.866 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:16.866 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080472+0000", 2024-03-27T21:41:16.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:16.866 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.866 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.866 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:16.866 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080475+0000", 2024-03-27T21:41:16.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.712e-06 2024-03-27T21:41:16.866 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.866 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.866 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:16.867 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080478+0000", 2024-03-27T21:41:16.867 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1070000000000001e-06 2024-03-27T21:41:16.867 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.867 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.867 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:16.867 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080481+0000", 2024-03-27T21:41:16.867 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4269999999999998e-06 2024-03-27T21:41:16.867 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.867 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.867 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:16.867 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086992+0000", 2024-03-27T21:41:16.867 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065114099999999996 2024-03-27T21:41:16.867 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.867 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.867 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:16.867 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087064+0000", 2024-03-27T21:41:16.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1927000000000004e-05 2024-03-27T21:41:16.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:16.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087188+0000", 2024-03-27T21:41:16.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123585 2024-03-27T21:41:16.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:16.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087766+0000", 2024-03-27T21:41:16.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057814300000000004 2024-03-27T21:41:16.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:16.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087934+0000", 2024-03-27T21:41:16.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000167894 2024-03-27T21:41:16.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.869 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.869 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:16.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087958+0000", 2024-03-27T21:41:16.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4663000000000001e-05 2024-03-27T21:41:16.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.869 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.869 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:16.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087978+0000", 2024-03-27T21:41:16.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9636e-05 2024-03-27T21:41:16.869 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.869 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:16.869 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.869 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.870 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:16.870 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:16.870 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 96.741342592999999, 2024-03-27T21:41:16.870 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076850679999999998, 2024-03-27T21:41:16.870 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:16.870 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:16.870 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:16.870 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:16.870 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:16.870 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 117 2024-03-27T21:41:16.870 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.870 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:16.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:16.871 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:16.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:16.871 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:16.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:16.871 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:16.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:16.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:16.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080508+0000", 2024-03-27T21:41:16.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.298e-06 2024-03-27T21:41:16.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.872 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:16.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080515+0000", 2024-03-27T21:41:16.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0790000000000003e-06 2024-03-27T21:41:16.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.872 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:16.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080518+0000", 2024-03-27T21:41:16.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8509999999999999e-06 2024-03-27T21:41:16.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.872 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:16.873 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087444+0000", 2024-03-27T21:41:16.873 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0069259120000000002 2024-03-27T21:41:16.873 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.873 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.873 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:16.873 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087491+0000", 2024-03-27T21:41:16.873 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7043999999999999e-05 2024-03-27T21:41:16.873 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.873 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.873 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:16.873 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087564+0000", 2024-03-27T21:41:16.873 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3603999999999995e-05 2024-03-27T21:41:16.873 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.873 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.873 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:16.873 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087902+0000", 2024-03-27T21:41:16.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033734100000000001 2024-03-27T21:41:16.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.874 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:16.874 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088117+0000", 2024-03-27T21:41:16.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000215164 2024-03-27T21:41:16.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.874 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:16.874 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088160+0000", 2024-03-27T21:41:16.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2933999999999999e-05 2024-03-27T21:41:16.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.874 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:16.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088190+0000", 2024-03-27T21:41:16.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0139999999999999e-05 2024-03-27T21:41:16.875 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.875 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:16.875 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.875 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.875 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:16.875 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:16.875 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 96.325062204000005, 2024-03-27T21:41:16.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052465439999999997, 2024-03-27T21:41:16.875 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:16.875 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:16.875 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:16.876 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:16.876 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:16.876 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 133 2024-03-27T21:41:16.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.876 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:16.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.876 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:16.876 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:16.876 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.876 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:16.876 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:16.876 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.877 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.877 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:16.877 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:16.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:16.877 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.877 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.877 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:16.877 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496788+0000", 2024-03-27T21:41:16.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-03-27T21:41:16.877 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.877 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.877 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:16.877 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496793+0000", 2024-03-27T21:41:16.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4790000000000004e-06 2024-03-27T21:41:16.877 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.877 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.878 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:16.878 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496796+0000", 2024-03-27T21:41:16.878 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1e-06 2024-03-27T21:41:16.878 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.878 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.878 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:16.878 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500447+0000", 2024-03-27T21:41:16.878 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036504459999999999 2024-03-27T21:41:16.878 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.878 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.878 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:16.878 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500492+0000", 2024-03-27T21:41:16.878 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5765e-05 2024-03-27T21:41:16.878 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:16.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500562+0000", 2024-03-27T21:41:16.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9206000000000001e-05 2024-03-27T21:41:16.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:16.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501959+0000", 2024-03-27T21:41:16.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001397187 2024-03-27T21:41:16.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:16.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501984+0000", 2024-03-27T21:41:16.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.552e-05 2024-03-27T21:41:16.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:16.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502010+0000", 2024-03-27T21:41:16.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5232000000000002e-05 2024-03-27T21:41:16.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:16.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502032+0000", 2024-03-27T21:41:16.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2337999999999999e-05 2024-03-27T21:41:16.880 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.880 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:16.880 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.880 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:16.880 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:16.881 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 96.325003546000005, 2024-03-27T21:41:16.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052892119999999997, 2024-03-27T21:41:16.881 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:16.881 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:16.881 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:16.881 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:16.881 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:16.881 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 134 2024-03-27T21:41:16.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.881 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:16.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:16.881 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:16.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:16.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:16.882 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:16.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:16.882 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:16.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:16.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496929+0000", 2024-03-27T21:41:16.882 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5098000000000004e-05 2024-03-27T21:41:16.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:16.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496937+0000", 2024-03-27T21:41:16.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9559999999999997e-06 2024-03-27T21:41:16.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:16.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496942+0000", 2024-03-27T21:41:16.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6350000000000003e-06 2024-03-27T21:41:16.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:16.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500749+0000", 2024-03-27T21:41:16.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038068529999999998 2024-03-27T21:41:16.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.884 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.884 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:16.884 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500792+0000", 2024-03-27T21:41:16.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2823e-05 2024-03-27T21:41:16.884 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.884 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.884 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:16.884 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500864+0000", 2024-03-27T21:41:16.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1715e-05 2024-03-27T21:41:16.884 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.884 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.884 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:16.884 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502068+0000", 2024-03-27T21:41:16.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012044860000000001 2024-03-27T21:41:16.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:16.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502073+0000", 2024-03-27T21:41:16.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5700000000000003e-06 2024-03-27T21:41:16.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:16.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502098+0000", 2024-03-27T21:41:16.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5120999999999999e-05 2024-03-27T21:41:16.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:16.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502133+0000", 2024-03-27T21:41:16.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5306999999999999e-05 2024-03-27T21:41:16.886 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.886 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:16.886 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.886 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:16.886 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:16.886 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 96.324898239000007, 2024-03-27T21:41:16.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062529040000000001, 2024-03-27T21:41:16.886 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:16.886 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:16.886 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:16.887 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:16.887 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:16.887 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135 2024-03-27T21:41:16.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.887 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:16.887 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.887 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:16.887 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:16.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.887 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.887 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:16.887 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:16.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:16.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:16.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:16.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:16.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496953+0000", 2024-03-27T21:41:16.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0709999999999996e-06 2024-03-27T21:41:16.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:16.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496961+0000", 2024-03-27T21:41:16.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1300000000000001e-06 2024-03-27T21:41:16.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:16.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496968+0000", 2024-03-27T21:41:16.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0999999999999998e-06 2024-03-27T21:41:16.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:16.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501244+0000", 2024-03-27T21:41:16.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042756310000000002 2024-03-27T21:41:16.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:16.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501377+0000", 2024-03-27T21:41:16.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013348799999999999 2024-03-27T21:41:16.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:16.890 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501455+0000", 2024-03-27T21:41:16.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7410000000000006e-05 2024-03-27T21:41:16.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:16.890 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502138+0000", 2024-03-27T21:41:16.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068386599999999999 2024-03-27T21:41:16.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:16.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503152+0000", 2024-03-27T21:41:16.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010139680000000001 2024-03-27T21:41:16.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:16.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503180+0000", 2024-03-27T21:41:16.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7426000000000001e-05 2024-03-27T21:41:16.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:16.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503202+0000", 2024-03-27T21:41:16.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2463000000000002e-05 2024-03-27T21:41:16.891 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.891 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:16.891 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.892 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.892 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:16.892 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:16.892 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 96.324874101000006, 2024-03-27T21:41:16.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066253680000000004, 2024-03-27T21:41:16.892 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:16.892 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:16.892 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:16.892 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:16.892 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:16.892 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 136 2024-03-27T21:41:16.892 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.892 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:16.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:16.893 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:16.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:16.893 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:16.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:16.893 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:16.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:16.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:16.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496977+0000", 2024-03-27T21:41:16.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5049999999999999e-06 2024-03-27T21:41:16.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:16.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496984+0000", 2024-03-27T21:41:16.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1520000000000003e-06 2024-03-27T21:41:16.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:16.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496990+0000", 2024-03-27T21:41:16.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-03-27T21:41:16.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:16.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501669+0000", 2024-03-27T21:41:16.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046793859999999998 2024-03-27T21:41:16.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:16.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501714+0000", 2024-03-27T21:41:16.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5154999999999997e-05 2024-03-27T21:41:16.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:16.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501783+0000", 2024-03-27T21:41:16.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8454999999999994e-05 2024-03-27T21:41:16.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:16.896 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503226+0000", 2024-03-27T21:41:16.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001443441 2024-03-27T21:41:16.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.896 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:16.896 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503540+0000", 2024-03-27T21:41:16.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031347100000000001 2024-03-27T21:41:16.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:16.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503563+0000", 2024-03-27T21:41:16.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3223000000000001e-05 2024-03-27T21:41:16.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:16.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503599+0000", 2024-03-27T21:41:16.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6105000000000001e-05 2024-03-27T21:41:16.897 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.897 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:16.897 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.898 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:16.898 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:16.898 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 96.324851291000002, 2024-03-27T21:41:16.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074261320000000002, 2024-03-27T21:41:16.898 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:16.898 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:16.898 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:16.898 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:16.898 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:16.898 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 137 2024-03-27T21:41:16.898 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.898 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:16.898 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.898 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:16.898 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:16.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:16.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:16.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:16.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:16.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:16.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:16.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497077+0000", 2024-03-27T21:41:16.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1599000000000003e-05 2024-03-27T21:41:16.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:16.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497085+0000", 2024-03-27T21:41:16.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6109999999999996e-06 2024-03-27T21:41:16.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:16.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497092+0000", 2024-03-27T21:41:16.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.742e-06 2024-03-27T21:41:16.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:16.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502159+0000", 2024-03-27T21:41:16.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050666870000000003 2024-03-27T21:41:16.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:16.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502200+0000", 2024-03-27T21:41:16.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1919999999999998e-05 2024-03-27T21:41:16.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:16.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502280+0000", 2024-03-27T21:41:16.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9417000000000002e-05 2024-03-27T21:41:16.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:16.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503605+0000", 2024-03-27T21:41:16.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001325641 2024-03-27T21:41:16.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:16.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504361+0000", 2024-03-27T21:41:16.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075555399999999997 2024-03-27T21:41:16.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:16.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504386+0000", 2024-03-27T21:41:16.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5369999999999999e-05 2024-03-27T21:41:16.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.903 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:16.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504422+0000", 2024-03-27T21:41:16.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6019000000000002e-05 2024-03-27T21:41:16.903 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.903 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:16.903 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.903 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:16.903 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:16.903 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 96.324747493999993, 2024-03-27T21:41:16.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074059750000000004, 2024-03-27T21:41:16.903 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:16.903 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:16.904 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:16.904 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:16.904 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:16.904 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-03-27T21:41:16.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.904 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:16.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:16.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:16.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:16.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:16.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.905 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:16.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:16.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:16.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.905 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:16.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497103+0000", 2024-03-27T21:41:16.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.856e-06 2024-03-27T21:41:16.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.905 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:16.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497121+0000", 2024-03-27T21:41:16.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7190999999999999e-05 2024-03-27T21:41:16.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:16.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497126+0000", 2024-03-27T21:41:16.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3770000000000004e-06 2024-03-27T21:41:16.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:16.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502833+0000", 2024-03-27T21:41:16.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057073510000000003 2024-03-27T21:41:16.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:16.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502881+0000", 2024-03-27T21:41:16.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7961999999999999e-05 2024-03-27T21:41:16.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:16.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502949+0000", 2024-03-27T21:41:16.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7957e-05 2024-03-27T21:41:16.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:16.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504430+0000", 2024-03-27T21:41:16.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001480763 2024-03-27T21:41:16.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:16.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504459+0000", 2024-03-27T21:41:16.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8983999999999999e-05 2024-03-27T21:41:16.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:16.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504486+0000", 2024-03-27T21:41:16.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6883999999999999e-05 2024-03-27T21:41:16.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:16.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504506+0000", 2024-03-27T21:41:16.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0214e-05 2024-03-27T21:41:16.909 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.909 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:16.909 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.909 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:16.909 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:16.909 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 96.324715659999995, 2024-03-27T21:41:16.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074754859999999999, 2024-03-27T21:41:16.909 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:16.909 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:16.909 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:16.910 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:16.910 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:16.910 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-03-27T21:41:16.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.910 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:16.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:16.910 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:16.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:16.910 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:16.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:16.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:16.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:16.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:16.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497137+0000", 2024-03-27T21:41:16.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0730000000000004e-06 2024-03-27T21:41:16.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:16.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497143+0000", 2024-03-27T21:41:16.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3759999999999996e-06 2024-03-27T21:41:16.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:16.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497150+0000", 2024-03-27T21:41:16.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.002e-06 2024-03-27T21:41:16.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:16.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503249+0000", 2024-03-27T21:41:16.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060989750000000004 2024-03-27T21:41:16.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:16.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503291+0000", 2024-03-27T21:41:16.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1563999999999997e-05 2024-03-27T21:41:16.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:16.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503376+0000", 2024-03-27T21:41:16.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5752000000000002e-05 2024-03-27T21:41:16.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:16.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504545+0000", 2024-03-27T21:41:16.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169172 2024-03-27T21:41:16.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.914 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:16.914 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504550+0000", 2024-03-27T21:41:16.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7419999999999997e-06 2024-03-27T21:41:16.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.914 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:16.914 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504574+0000", 2024-03-27T21:41:16.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4124000000000001e-05 2024-03-27T21:41:16.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:16.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504607+0000", 2024-03-27T21:41:16.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3118999999999999e-05 2024-03-27T21:41:16.915 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.915 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:16.915 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.915 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:16.915 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:16.915 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 96.324691994000005, 2024-03-27T21:41:16.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078150589999999992, 2024-03-27T21:41:16.915 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:16.916 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:16.916 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:16.916 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:16.916 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:16.916 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-03-27T21:41:16.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.916 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:16.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:16.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:16.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:16.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:16.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.917 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.917 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:16.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497155+0000", 2024-03-27T21:41:16.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:16.917 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.917 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:16.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497230+0000", 2024-03-27T21:41:16.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4789000000000005e-05 2024-03-27T21:41:16.917 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.917 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:16.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497241+0000", 2024-03-27T21:41:16.918 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1442e-05 2024-03-27T21:41:16.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.918 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:16.918 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497255+0000", 2024-03-27T21:41:16.918 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3087e-05 2024-03-27T21:41:16.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.918 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:16.919 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503630+0000", 2024-03-27T21:41:16.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063755269999999998 2024-03-27T21:41:16.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.919 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:16.919 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503673+0000", 2024-03-27T21:41:16.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2616e-05 2024-03-27T21:41:16.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.919 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:16.919 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503738+0000", 2024-03-27T21:41:16.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5114e-05 2024-03-27T21:41:16.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:16.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504907+0000", 2024-03-27T21:41:16.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169432 2024-03-27T21:41:16.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:16.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504926+0000", 2024-03-27T21:41:16.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9086999999999999e-05 2024-03-27T21:41:16.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:16.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504950+0000", 2024-03-27T21:41:16.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3598999999999999e-05 2024-03-27T21:41:16.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.922 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:16.922 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504971+0000", 2024-03-27T21:41:16.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0772999999999999e-05 2024-03-27T21:41:16.922 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.922 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:16.922 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.922 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:16.922 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:16.922 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 96.324584954000002, 2024-03-27T21:41:16.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082962590000000003, 2024-03-27T21:41:16.923 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:16.923 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:16.923 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:16.923 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:16.923 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:16.923 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-03-27T21:41:16.923 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.923 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:16.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.923 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:16.923 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:16.923 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.923 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.923 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:16.924 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:16.924 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.924 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.924 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.924 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:16.924 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497262+0000", 2024-03-27T21:41:16.924 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:16.924 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.924 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.924 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:16.924 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497267+0000", 2024-03-27T21:41:16.924 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.955e-06 2024-03-27T21:41:16.924 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.924 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.924 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:16.925 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497275+0000", 2024-03-27T21:41:16.925 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.824e-06 2024-03-27T21:41:16.925 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.925 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.925 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:16.925 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497280+0000", 2024-03-27T21:41:16.925 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8230000000000002e-06 2024-03-27T21:41:16.925 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.925 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.925 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:16.925 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504628+0000", 2024-03-27T21:41:16.925 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007348561 2024-03-27T21:41:16.925 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.925 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.926 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:16.926 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504673+0000", 2024-03-27T21:41:16.926 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4313000000000001e-05 2024-03-27T21:41:16.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.926 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.926 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:16.926 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504745+0000", 2024-03-27T21:41:16.926 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2303000000000002e-05 2024-03-27T21:41:16.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.926 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.926 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:16.926 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505358+0000", 2024-03-27T21:41:16.926 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061281300000000003 2024-03-27T21:41:16.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:16.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505503+0000", 2024-03-27T21:41:16.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000145134 2024-03-27T21:41:16.927 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:16.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505526+0000", 2024-03-27T21:41:16.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3312e-05 2024-03-27T21:41:16.927 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:16.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505559+0000", 2024-03-27T21:41:16.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2821999999999998e-05 2024-03-27T21:41:16.927 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.928 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:16.928 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.928 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.928 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.928 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:16.928 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:16.928 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 95.715362714999998, 2024-03-27T21:41:16.928 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053753339999999998, 2024-03-27T21:41:16.928 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:16.928 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:16.928 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:16.928 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:16.929 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:16.929 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159 2024-03-27T21:41:16.929 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.929 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:16.929 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.929 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:16.929 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:16.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.929 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.929 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.929 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:16.929 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:16.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:16.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:16.930 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:16.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:16.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:16.930 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106488+0000", 2024-03-27T21:41:16.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9830000000000001e-06 2024-03-27T21:41:16.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:16.931 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106491+0000", 2024-03-27T21:41:16.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3459999999999998e-06 2024-03-27T21:41:16.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:16.931 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106493+0000", 2024-03-27T21:41:16.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.509e-06 2024-03-27T21:41:16.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:16.931 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110636+0000", 2024-03-27T21:41:16.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041427260000000002 2024-03-27T21:41:16.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:16.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110683+0000", 2024-03-27T21:41:16.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6557000000000002e-05 2024-03-27T21:41:16.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:16.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110755+0000", 2024-03-27T21:41:16.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2119000000000005e-05 2024-03-27T21:41:16.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:16.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111415+0000", 2024-03-27T21:41:16.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066065199999999996 2024-03-27T21:41:16.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:16.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111793+0000", 2024-03-27T21:41:16.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037732100000000001 2024-03-27T21:41:16.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:16.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111829+0000", 2024-03-27T21:41:16.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418999999999998e-05 2024-03-27T21:41:16.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:16.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:16.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:16.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111860+0000", 2024-03-27T21:41:16.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1003000000000001e-05 2024-03-27T21:41:16.934 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.934 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:16.934 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.934 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:16.934 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:16.934 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:16.935 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:16.949+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:16.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:41:16.985 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:41:17.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.013+0000 7ff9d57fa640 1 -- 172.21.15.60:0/428622537 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v56) v1 ==== 87+0+0 (secure 0 0 0) 0x7ff9c4057db0 con 0x7ff9d813e810 2024-03-27T21:41:17.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.013+0000 7ff9d57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:41:17.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.013+0000 7ff9d57fa640 10 monclient: _finish_command 2 = system:0 noscrub is set 2024-03-27T21:41:17.001 INFO:teuthology.orchestra.run.smithi060.stderr:noscrub is set 2024-03-27T21:41:17.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.016+0000 7ff9df167640 10 librados: watch_flush enter 2024-03-27T21:41:17.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.016+0000 7ff9df167640 10 librados: watch_flush exit 2024-03-27T21:41:17.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.016+0000 7ff9df167640 20 client.4864.objecter shutdown clearing up homeless session... 2024-03-27T21:41:17.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.016+0000 7ff9df167640 10 client.4864.objecter successfully canceled tick 2024-03-27T21:41:17.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.016+0000 7ff9df167640 1 -- 172.21.15.60:0/428622537 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff9bc06c590 msgr2=0x7ff9bc06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:17.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.016+0000 7ff9df167640 1 --2- 172.21.15.60:0/428622537 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff9bc06c590 0x7ff9bc06ea50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc034000 tx=0x7ff9cc008030 comp rx=0 tx=0).stop 2024-03-27T21:41:17.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.016+0000 7ff9df167640 10 monclient: shutdown 2024-03-27T21:41:17.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.016+0000 7ff9df167640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:17.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.016+0000 7ff9df167640 1 -- 172.21.15.60:0/428622537 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9d813e810 msgr2=0x7ff9d813ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:17.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.016+0000 7ff9df167640 1 --2- 172.21.15.60:0/428622537 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9d813e810 0x7ff9d813ab10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400e9e0 tx=0x7ff9c400eea0 comp rx=0 tx=0).stop 2024-03-27T21:41:17.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.016+0000 7ff9df167640 1 -- 172.21.15.60:0/428622537 shutdown_connections 2024-03-27T21:41:17.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.016+0000 7ff9df167640 1 --2- 172.21.15.60:0/428622537 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff9d8141630 0x7ff9d8132f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:17.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.016+0000 7ff9df167640 1 --2- 172.21.15.60:0/428622537 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff9bc06c590 0x7ff9bc06ea50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:17.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.016+0000 7ff9df167640 1 --2- 172.21.15.60:0/428622537 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9d813e810 0x7ff9d813ab10 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:17.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.016+0000 7ff9df167640 1 --2- 172.21.15.60:0/428622537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9d8071a50 0x7ff9d81d65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:17.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.016+0000 7ff9df167640 1 -- 172.21.15.60:0/428622537 >> 172.21.15.60:0/428622537 conn(0x7ff9d80b7300 msgr2=0x7ff9d8141aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:17.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.017+0000 7ff9df167640 1 -- 172.21.15.60:0/428622537 shutdown_connections 2024-03-27T21:41:17.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.017+0000 7ff9df167640 1 -- 172.21.15.60:0/428622537 wait complete. 2024-03-27T21:41:17.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.017+0000 7ff9df167640 1 librados: shutdown 2024-03-27T21:41:17.036 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:17.049+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:17.109 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:17.109 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:17.110 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:17.110 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:17.122 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:41:17.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.156+0000 7fee0705c640 10 monclient: get_monmap_and_config 2024-03-27T21:41:17.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.156+0000 7fee0705c640 10 monclient: build_initial_monmap 2024-03-27T21:41:17.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.156+0000 7fee0705c640 10 monclient: monmap: 2024-03-27T21:41:17.143 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:17.143 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:17.143 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:17.157679+0000 2024-03-27T21:41:17.143 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:17.157679+0000 2024-03-27T21:41:17.143 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:17.143 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:17.143 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:17.144 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:17.144 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:17.144 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:17.144 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:17.144 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:17.145 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:17.157+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:17.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.157+0000 7fee0705c640 1 Processor -- start 2024-03-27T21:41:17.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.157+0000 7fee0705c640 1 -- start start 2024-03-27T21:41:17.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.157+0000 7fee0705c640 10 monclient: init 2024-03-27T21:41:17.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fee0705c640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:17.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fee0705c640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-03-27T21:41:17.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fee0705c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fee000b40b0 0x7fee000a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fee0705c640 10 monclient(hunting): picked mon.noname-b con 0x7fee000b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fee0705c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fee0013b020 0x7fee0013b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fee0705c640 10 monclient(hunting): picked mon.noname-a con 0x7fee0013b020 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fee0705c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fee001416f0 0x7fee00141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fee0705c640 10 monclient(hunting): picked mon.noname-c con 0x7fee001416f0 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fee0705c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fee0705c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee000b4490 con 0x7fee0013b020 2024-03-27T21:41:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fee0705c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fee0705c640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee000b3830 con 0x7fee001416f0 2024-03-27T21:41:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fee0705c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fee0705c640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee000aa520 con 0x7fee000b40b0 2024-03-27T21:41:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fee0705c640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fee0705c640 10 monclient(hunting): authenticate will time out at 918.379728s 2024-03-27T21:41:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fedfffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fee0013b020 0x7fee0013b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fedfffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fee0013b020 0x7fee0013b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37302/0 (socket says 172.21.15.60:37302) 2024-03-27T21:41:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fedfffff640 1 -- 172.21.15.60:0/2925447015 learned_addr learned my addr 172.21.15.60:0/2925447015 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fedfffff640 10 monclient(hunting): get_auth_request con 0x7fee0013b020 auth_method 0 2024-03-27T21:41:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fedfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fedfffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.158+0000 7fedfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fedfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fedfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fedfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fee055f9640 1 --2- 172.21.15.60:0/2925447015 >> v2:172.21.15.191:3300/0 conn(0x7fee001416f0 0x7fee00141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fee055f9640 10 monclient(hunting): get_auth_request con 0x7fee001416f0 auth_method 0 2024-03-27T21:41:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fee055f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fee055f9640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:17.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fee055f9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:17.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fee04df8640 1 --2- 172.21.15.60:0/2925447015 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fee000b40b0 0x7fee000a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:17.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fedfffff640 10 monclient(hunting): handle_auth_done global_id 4938 payload 274 2024-03-27T21:41:17.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fedfffff640 1 -- 172.21.15.60:0/2925447015 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fee000b40b0 msgr2=0x7fee000a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:17.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fedfffff640 1 --2- 172.21.15.60:0/2925447015 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fee000b40b0 0x7fee000a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:17.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fedfffff640 1 -- 172.21.15.60:0/2925447015 >> v2:172.21.15.191:3300/0 conn(0x7fee001416f0 msgr2=0x7fee00141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:17.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fedfffff640 1 --2- 172.21.15.60:0/2925447015 >> v2:172.21.15.191:3300/0 conn(0x7fee001416f0 0x7fee00141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:17.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fedfffff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:17.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fedfffff640 1 monclient: found mon.noname-a 2024-03-27T21:41:17.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fedfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:17.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fedfffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:17.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fedfffff640 1 -- 172.21.15.60:0/2925447015 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee000a9470 con 0x7fee0013b020 2024-03-27T21:41:17.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fee04df8640 10 monclient: get_auth_request con 0x7fee000b40b0 auth_method 0 2024-03-27T21:41:17.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fee055f9640 1 --2- 172.21.15.60:0/2925447015 >> v2:172.21.15.191:3300/0 conn(0x7fee001416f0 0x7fee00141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:41:17.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.159+0000 7fee04df8640 1 --2- 172.21.15.60:0/2925447015 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fee000b40b0 0x7fee000a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:41:17.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.160+0000 7fedfffff640 1 --2- 172.21.15.60:0/2925447015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee0013b020 0x7fee0013b480 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fee000b2e20 tx=0x7fedec02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d401bd99ac89c4d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:17.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.160+0000 7fedff7fe640 1 -- 172.21.15.60:0/2925447015 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec031880 con 0x7fee0013b020 2024-03-27T21:41:17.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.160+0000 7fedff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:17.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.160+0000 7fedff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:17.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.160+0000 7fedff7fe640 10 monclient: dump: 2024-03-27T21:41:17.161 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:17.161 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:17.161 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:17.161 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:17.162 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:17.162 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:17.162 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:17.162 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:17.162 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:17.162 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:17.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.160+0000 7fedff7fe640 10 monclient: _finish_auth 0 2024-03-27T21:41:17.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.160+0000 7fedff7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:41:17.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.160+0000 7fedff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:17.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.160+0000 7fedff7fe640 1 -- 172.21.15.60:0/2925447015 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedec0319e0 con 0x7fee0013b020 2024-03-27T21:41:17.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.160+0000 7fedff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:17.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.160+0000 7fedff7fe640 1 -- 172.21.15.60:0/2925447015 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec031d70 con 0x7fee0013b020 2024-03-27T21:41:17.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.160+0000 7fedff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:17.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.160+0000 7fedff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:17.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.160+0000 7fedff7fe640 10 monclient: dump: 2024-03-27T21:41:17.163 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:17.163 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:17.163 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:17.163 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:17.163 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:17.163 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:17.163 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:17.163 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:17.163 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:17.163 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:17.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.160+0000 7fee0705c640 5 monclient: authenticate success, global_id 4938 2024-03-27T21:41:17.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.160+0000 7fee0705c640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:17.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.160+0000 7fee0705c640 4 set_mon_vals no callback set 2024-03-27T21:41:17.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.161+0000 7fee0705c640 10 monclient: shutdown 2024-03-27T21:41:17.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.161+0000 7fee0705c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:17.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.161+0000 7fee0705c640 1 -- 172.21.15.60:0/2925447015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee0013b020 msgr2=0x7fee0013b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:17.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.161+0000 7fee0705c640 1 --2- 172.21.15.60:0/2925447015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee0013b020 0x7fee0013b480 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fee000b2e20 tx=0x7fedec02b920 comp rx=0 tx=0).stop 2024-03-27T21:41:17.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.161+0000 7fee0705c640 1 -- 172.21.15.60:0/2925447015 shutdown_connections 2024-03-27T21:41:17.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.161+0000 7fee0705c640 1 --2- 172.21.15.60:0/2925447015 >> v2:172.21.15.191:3300/0 conn(0x7fee001416f0 0x7fee00141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:17.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.161+0000 7fee0705c640 1 --2- 172.21.15.60:0/2925447015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee0013b020 0x7fee0013b480 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:17.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.161+0000 7fee0705c640 1 --2- 172.21.15.60:0/2925447015 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fee000b40b0 0x7fee000a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:17.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.161+0000 7fee0705c640 1 -- 172.21.15.60:0/2925447015 >> 172.21.15.60:0/2925447015 conn(0x7fee000608e0 msgr2=0x7fee00060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:17.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.161+0000 7fee0705c640 1 -- 172.21.15.60:0/2925447015 shutdown_connections 2024-03-27T21:41:17.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.161+0000 7fee0705c640 1 -- 172.21.15.60:0/2925447015 wait complete. 2024-03-27T21:41:17.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.162+0000 7fee0705c640 10 monclient: build_initial_monmap 2024-03-27T21:41:17.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.162+0000 7fee0705c640 10 monclient: monmap: 2024-03-27T21:41:17.165 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:17.165 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:17.165 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:17.165 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:17.165 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:17.165 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:17.165 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:17.165 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:17.165 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:17.166 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:17.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.163+0000 7fee0705c640 1 librados: starting msgr at 2024-03-27T21:41:17.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.163+0000 7fee0705c640 1 librados: starting objecter 2024-03-27T21:41:17.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.163+0000 7fee0705c640 1 Processor -- start 2024-03-27T21:41:17.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.163+0000 7fee0705c640 1 -- start start 2024-03-27T21:41:17.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.163+0000 7fee0705c640 1 librados: setting wanted keys 2024-03-27T21:41:17.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.163+0000 7fee0705c640 1 librados: calling monclient init 2024-03-27T21:41:17.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.163+0000 7fee0705c640 10 monclient: init 2024-03-27T21:41:17.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee0705c640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:17.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee0705c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:41:17.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee0705c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee000b40b0 0x7fee001dab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee0705c640 10 monclient(hunting): picked mon.noname-b con 0x7fee000b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee0705c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee001416f0 0x7fee001db070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee0705c640 10 monclient(hunting): picked mon.noname-a con 0x7fee001416f0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee0705c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fee0006ed30 0x7fee0006f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee0705c640 10 monclient(hunting): picked mon.noname-c con 0x7fee0006ed30 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee0705c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee0705c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee001d8220 con 0x7fee001416f0 2024-03-27T21:41:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee0705c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee0705c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee000aadc0 con 0x7fee000b40b0 2024-03-27T21:41:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee0705c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee0705c640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee001da630 con 0x7fee0006ed30 2024-03-27T21:41:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee0705c640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee0705c640 10 monclient(hunting): authenticate will time out at 918.385695s 2024-03-27T21:41:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fedfffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee001416f0 0x7fee001db070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee055f9640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fee0006ed30 0x7fee0006f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fedfffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee001416f0 0x7fee001db070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37318/0 (socket says 172.21.15.60:37318) 2024-03-27T21:41:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee055f9640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fee0006ed30 0x7fee0006f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:37882/0 (socket says 172.21.15.60:37882) 2024-03-27T21:41:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fedfffff640 1 -- 172.21.15.60:0/2194545019 learned_addr learned my addr 172.21.15.60:0/2194545019 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fedfffff640 10 monclient(hunting): get_auth_request con 0x7fee001416f0 auth_method 0 2024-03-27T21:41:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fedfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fedfffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fedfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee055f9640 10 monclient(hunting): get_auth_request con 0x7fee0006ed30 auth_method 0 2024-03-27T21:41:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee055f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee04df8640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee000b40b0 0x7fee001dab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee055f9640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee055f9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee04df8640 10 monclient(hunting): get_auth_request con 0x7fee000b40b0 auth_method 0 2024-03-27T21:41:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee04df8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee04df8640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee04df8640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fedfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fedfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fedfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee055f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.164+0000 7fee055f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fee055f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fee04df8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fee04df8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fee04df8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfffff640 10 monclient(hunting): handle_auth_done global_id 4941 payload 824 2024-03-27T21:41:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfffff640 1 -- 172.21.15.60:0/2194545019 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fee0006ed30 msgr2=0x7fee0006f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfffff640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fee0006ed30 0x7fee0006f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfffff640 1 -- 172.21.15.60:0/2194545019 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee000b40b0 msgr2=0x7fee001dab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfffff640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee000b40b0 0x7fee001dab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfffff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfffff640 1 monclient: found mon.noname-a 2024-03-27T21:41:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfffff640 1 -- 172.21.15.60:0/2194545019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedec0094d0 con 0x7fee001416f0 2024-03-27T21:41:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fee04df8640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee000b40b0 0x7fee001dab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fee055f9640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fee0006ed30 0x7fee0006f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfffff640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee001416f0 0x7fee001db070 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fedec000c00 tx=0x7fedec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfd7fa640 10 client.?.objecter ms_handle_connect 0x7fee001416f0 2024-03-27T21:41:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfd7fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfd7fa640 1 -- 172.21.15.60:0/2194545019 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec03ad30 con 0x7fee001416f0 2024-03-27T21:41:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfd7fa640 10 monclient: dump: 2024-03-27T21:41:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:17.171 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfd7fa640 10 monclient: _finish_auth 0 2024-03-27T21:41:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfd7fa640 10 monclient: _check_auth_tickets 2024-03-27T21:41:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:17.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fee0705c640 5 monclient: authenticate success, global_id 4941 2024-03-27T21:41:17.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfd7fa640 1 -- 172.21.15.60:0/2194545019 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedec03ae90 con 0x7fee001416f0 2024-03-27T21:41:17.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:17.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfd7fa640 1 -- 172.21.15.60:0/2194545019 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec044610 con 0x7fee001416f0 2024-03-27T21:41:17.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:17.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:17.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfd7fa640 10 monclient: dump: 2024-03-27T21:41:17.172 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:17.172 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:17.172 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:17.172 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:17.172 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:17.172 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:17.172 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:17.173 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:17.173 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:17.173 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:17.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.165+0000 7fedfeffd640 4 set_mon_vals no callback set 2024-03-27T21:41:17.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.166+0000 7fee0705c640 10 monclient: _renew_subs 2024-03-27T21:41:17.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.166+0000 7fee0705c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:17.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.166+0000 7fee0705c640 1 -- 172.21.15.60:0/2194545019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee0006bef0 con 0x7fee001416f0 2024-03-27T21:41:17.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.166+0000 7fee0705c640 10 client.4941.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:17.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.166+0000 7fee0705c640 10 monclient: _renew_subs 2024-03-27T21:41:17.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.166+0000 7fee0705c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:17.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.166+0000 7fee0705c640 1 -- 172.21.15.60:0/2194545019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee0006f8e0 con 0x7fee001416f0 2024-03-27T21:41:17.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.166+0000 7fee0705c640 1 librados: init done 2024-03-27T21:41:17.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.167+0000 7fee0705c640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:41:17.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.167+0000 7fee0705c640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:17.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.167+0000 7fee0705c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:17.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.167+0000 7fee0705c640 1 -- 172.21.15.60:0/2194545019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fee00069470 con 0x7fee001416f0 2024-03-27T21:41:17.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.168+0000 7fedfd7fa640 1 -- 172.21.15.60:0/2194545019 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fedec04d450 con 0x7fee001416f0 2024-03-27T21:41:17.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fedfd7fa640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fede406c590 0x7fede406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:17.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fedfd7fa640 1 -- 172.21.15.60:0/2194545019 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fedec0abc80 con 0x7fee001416f0 2024-03-27T21:41:17.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fedfd7fa640 10 client.4941.objecter ms_dispatch 0x7fee00000b90 osd_map(56..56 src has 1..56) v4 2024-03-27T21:41:17.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fedfd7fa640 3 client.4941.objecter handle_osd_map got epochs [56,56] > 0 2024-03-27T21:41:17.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fedfd7fa640 3 client.4941.objecter handle_osd_map decoding full epoch 56 2024-03-27T21:41:17.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fee04df8640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fede406c590 0x7fede406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:17.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fee04df8640 10 monclient: get_auth_request con 0x7fede406c590 auth_method 0 2024-03-27T21:41:17.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fedfd7fa640 20 client.4941.objecter dump_active .. 0 homeless 2024-03-27T21:41:17.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fedfd7fa640 1 -- 172.21.15.60:0/2194545019 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v56) v1 ==== 54+0+16 (secure 0 0 0) 0x7fedec0ac0d0 con 0x7fee001416f0 2024-03-27T21:41:17.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fedfd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:17.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fedfd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:17.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fee0705c640 10 client.4941.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:17.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign -1 1 2024-03-27T21:41:17.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fee0705c640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fee001ddfa0 0x7fee001de380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:17.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee0006c0b0 osd=0 3 2024-03-27T21:41:17.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee0006c0b0 osd=0 3 2024-03-27T21:41:17.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee0006c0b0 osd=0 3 2024-03-27T21:41:17.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fee0705c640 15 client.4941.objecter _session_command_op_remove -1 1 2024-03-27T21:41:17.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fee0705c640 20 client.4941.objecter get_session s=0x7fee0006c0b0 osd=0 3 2024-03-27T21:41:17.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign 0 1 2024-03-27T21:41:17.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee0006c0b0 osd=0 4 2024-03-27T21:41:17.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fee0705c640 10 client.4941.objecter _send_command 1 2024-03-27T21:41:17.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fee0705c640 1 -- 172.21.15.60:0/2194545019 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fee00144990 con 0x7fee001ddfa0 2024-03-27T21:41:17.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fee055f9640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fee001ddfa0 0x7fee001de380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:17.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fee055f9640 10 monclient: get_auth_request con 0x7fee001ddfa0 auth_method 0 2024-03-27T21:41:17.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fee04df8640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fede406c590 0x7fede406ea50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fedf4006b10 tx=0x7fedf4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:17.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.169+0000 7fedfd7fa640 10 client.4941.objecter ms_handle_connect 0x7fede406c590 2024-03-27T21:41:17.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.170+0000 7fedfd7fa640 10 client.4941.objecter ms_handle_connect 0x7fee001ddfa0 2024-03-27T21:41:17.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.170+0000 7fee055f9640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fee001ddfa0 0x7fee001de380 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:17.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.171+0000 7fedfd7fa640 1 -- 172.21.15.60:0/2194545019 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fedf800f040 con 0x7fee001ddfa0 2024-03-27T21:41:17.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.171+0000 7fedfd7fa640 10 client.4941.objecter ms_dispatch 0x7fee00000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:41:17.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.171+0000 7fedfd7fa640 10 client.4941.objecter _finish_command 1 = system:0 2024-03-27T21:41:17.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.171+0000 7fedfd7fa640 20 client.4941.objecter put_session s=0x7fee0006c0b0 osd=0 4 2024-03-27T21:41:17.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.171+0000 7fedfd7fa640 15 client.4941.objecter _session_command_op_remove 0 1 2024-03-27T21:41:17.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.186+0000 7fee0705c640 10 client.4941.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:17.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.186+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign -1 2 2024-03-27T21:41:17.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.186+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee0006c0b0 osd=0 3 2024-03-27T21:41:17.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.186+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee0006c0b0 osd=0 3 2024-03-27T21:41:17.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.186+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee0006c0b0 osd=0 3 2024-03-27T21:41:17.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.186+0000 7fee0705c640 15 client.4941.objecter _session_command_op_remove -1 2 2024-03-27T21:41:17.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.186+0000 7fee0705c640 20 client.4941.objecter get_session s=0x7fee0006c0b0 osd=0 3 2024-03-27T21:41:17.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.186+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign 0 2 2024-03-27T21:41:17.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.186+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee0006c0b0 osd=0 4 2024-03-27T21:41:17.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.186+0000 7fee0705c640 10 client.4941.objecter _send_command 2 2024-03-27T21:41:17.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.186+0000 7fee0705c640 1 -- 172.21.15.60:0/2194545019 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fee001de8c0 con 0x7fee001ddfa0 2024-03-27T21:41:17.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.187+0000 7fedfd7fa640 1 -- 172.21.15.60:0/2194545019 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fee001de8c0 con 0x7fee001ddfa0 2024-03-27T21:41:17.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.187+0000 7fedfd7fa640 10 client.4941.objecter ms_dispatch 0x7fee00000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:17.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.187+0000 7fedfd7fa640 10 client.4941.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:17.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.187+0000 7fedfd7fa640 20 client.4941.objecter put_session s=0x7fee0006c0b0 osd=0 4 2024-03-27T21:41:17.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.187+0000 7fedfd7fa640 15 client.4941.objecter _session_command_op_remove 0 2 2024-03-27T21:41:17.181 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:41:17.183 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:41:17.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.187+0000 7fee0705c640 10 client.4941.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:17.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.187+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign -1 3 2024-03-27T21:41:17.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.187+0000 7fee0705c640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fee000b58c0 0x7fee00065a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:17.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.187+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee000b5750 osd=1 3 2024-03-27T21:41:17.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.187+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee000b5750 osd=1 3 2024-03-27T21:41:17.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.187+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee000b5750 osd=1 3 2024-03-27T21:41:17.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.187+0000 7fee0705c640 15 client.4941.objecter _session_command_op_remove -1 3 2024-03-27T21:41:17.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.187+0000 7fee0705c640 20 client.4941.objecter get_session s=0x7fee000b5750 osd=1 3 2024-03-27T21:41:17.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.187+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign 1 3 2024-03-27T21:41:17.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.187+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee000b5750 osd=1 4 2024-03-27T21:41:17.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.187+0000 7fee0705c640 10 client.4941.objecter _send_command 3 2024-03-27T21:41:17.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.187+0000 7fee0705c640 1 -- 172.21.15.60:0/2194545019 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fee0013a230 con 0x7fee000b58c0 2024-03-27T21:41:17.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.188+0000 7fee04df8640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fee000b58c0 0x7fee00065a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:17.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.188+0000 7fee04df8640 10 monclient: get_auth_request con 0x7fee000b58c0 auth_method 0 2024-03-27T21:41:17.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.188+0000 7fedfd7fa640 10 client.4941.objecter ms_handle_connect 0x7fee000b58c0 2024-03-27T21:41:17.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.188+0000 7fee04df8640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fee000b58c0 0x7fee00065a90 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:17.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.189+0000 7fedfd7fa640 1 -- 172.21.15.60:0/2194545019 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fee0013a230 con 0x7fee000b58c0 2024-03-27T21:41:17.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.189+0000 7fedfd7fa640 10 client.4941.objecter ms_dispatch 0x7fee00000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:41:17.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.189+0000 7fedfd7fa640 10 client.4941.objecter _finish_command 3 = system:0 2024-03-27T21:41:17.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.189+0000 7fedfd7fa640 20 client.4941.objecter put_session s=0x7fee000b5750 osd=1 4 2024-03-27T21:41:17.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.189+0000 7fedfd7fa640 15 client.4941.objecter _session_command_op_remove 1 3 2024-03-27T21:41:17.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.203+0000 7fee0705c640 10 client.4941.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:17.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.203+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign -1 4 2024-03-27T21:41:17.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.203+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee000b5750 osd=1 3 2024-03-27T21:41:17.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.203+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee000b5750 osd=1 3 2024-03-27T21:41:17.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.203+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee000b5750 osd=1 3 2024-03-27T21:41:17.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.203+0000 7fee0705c640 15 client.4941.objecter _session_command_op_remove -1 4 2024-03-27T21:41:17.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.203+0000 7fee0705c640 20 client.4941.objecter get_session s=0x7fee000b5750 osd=1 3 2024-03-27T21:41:17.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.203+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign 1 4 2024-03-27T21:41:17.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.203+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee000b5750 osd=1 4 2024-03-27T21:41:17.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.203+0000 7fee0705c640 10 client.4941.objecter _send_command 4 2024-03-27T21:41:17.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.203+0000 7fee0705c640 1 -- 172.21.15.60:0/2194545019 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fee0013a3a0 con 0x7fee000b58c0 2024-03-27T21:41:17.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.205+0000 7fedfd7fa640 1 -- 172.21.15.60:0/2194545019 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fee0013a3a0 con 0x7fee000b58c0 2024-03-27T21:41:17.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.205+0000 7fedfd7fa640 10 client.4941.objecter ms_dispatch 0x7fee00000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:17.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.205+0000 7fedfd7fa640 10 client.4941.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:17.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.205+0000 7fedfd7fa640 20 client.4941.objecter put_session s=0x7fee000b5750 osd=1 4 2024-03-27T21:41:17.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.205+0000 7fedfd7fa640 15 client.4941.objecter _session_command_op_remove 1 4 2024-03-27T21:41:17.192 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:41:17.194 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:41:17.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.205+0000 7fee0705c640 10 client.4941.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:17.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.205+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign -1 5 2024-03-27T21:41:17.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.205+0000 7fee0705c640 20 client.4941.objecter _get_session osd=-1 returning homeless 2024-03-27T21:41:17.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.205+0000 7fee0705c640 10 client.4941.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:17.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.205+0000 7fee0705c640 10 monclient: _renew_subs 2024-03-27T21:41:17.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.205+0000 7fee0705c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:17.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.205+0000 7fee0705c640 1 -- 172.21.15.60:0/2194545019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=57}) v3 -- 0x7fee001399e0 con 0x7fee001416f0 2024-03-27T21:41:17.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.205+0000 7fee0705c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:17.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.205+0000 7fee0705c640 1 -- 172.21.15.60:0/2194545019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fee00066790 con 0x7fee001416f0 2024-03-27T21:41:17.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.206+0000 7fedfd7fa640 1 -- 172.21.15.60:0/2194545019 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedec077170 con 0x7fee001416f0 2024-03-27T21:41:17.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.206+0000 7fedfd7fa640 10 monclient: handle_get_version_reply finishing 1 version 56 2024-03-27T21:41:17.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.206+0000 7fedfeffd640 10 client.4941.objecter _check_command_map_dne tid 5 current 56 map_dne_bound 56 2024-03-27T21:41:17.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.206+0000 7fedfeffd640 10 client.4941.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:41:17.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.206+0000 7fedfeffd640 15 client.4941.objecter _session_command_op_remove -1 5 2024-03-27T21:41:17.196 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:41:17.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.206+0000 7fee0705c640 10 client.4941.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:17.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.206+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign -1 6 2024-03-27T21:41:17.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.206+0000 7fee0705c640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fee0002b7f0 0x7fee0002bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:17.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.206+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee0002b170 osd=3 3 2024-03-27T21:41:17.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.206+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee0002b170 osd=3 3 2024-03-27T21:41:17.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.206+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee0002b170 osd=3 3 2024-03-27T21:41:17.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.206+0000 7fee0705c640 15 client.4941.objecter _session_command_op_remove -1 6 2024-03-27T21:41:17.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.206+0000 7fee0705c640 20 client.4941.objecter get_session s=0x7fee0002b170 osd=3 3 2024-03-27T21:41:17.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.206+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign 3 6 2024-03-27T21:41:17.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.206+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee0002b170 osd=3 4 2024-03-27T21:41:17.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.206+0000 7fee0705c640 10 client.4941.objecter _send_command 6 2024-03-27T21:41:17.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.206+0000 7fee0705c640 1 -- 172.21.15.60:0/2194545019 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fee0002c140 con 0x7fee0002b7f0 2024-03-27T21:41:17.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.206+0000 7fedfffff640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fee0002b7f0 0x7fee0002bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:17.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.206+0000 7fedfffff640 10 monclient: get_auth_request con 0x7fee0002b7f0 auth_method 0 2024-03-27T21:41:17.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.206+0000 7fedfffff640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fee0002b7f0 0x7fee0002bc00 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:17.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.206+0000 7fedfd7fa640 10 client.4941.objecter ms_handle_connect 0x7fee0002b7f0 2024-03-27T21:41:17.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.207+0000 7fedfd7fa640 1 -- 172.21.15.60:0/2194545019 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fee0002c140 con 0x7fee0002b7f0 2024-03-27T21:41:17.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.207+0000 7fedfd7fa640 10 client.4941.objecter ms_dispatch 0x7fee00000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:41:17.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.207+0000 7fedfd7fa640 10 client.4941.objecter _finish_command 6 = system:0 2024-03-27T21:41:17.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.207+0000 7fedfd7fa640 20 client.4941.objecter put_session s=0x7fee0002b170 osd=3 4 2024-03-27T21:41:17.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.207+0000 7fedfd7fa640 15 client.4941.objecter _session_command_op_remove 3 6 2024-03-27T21:41:17.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.233+0000 7fee0705c640 10 client.4941.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:17.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.233+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign -1 7 2024-03-27T21:41:17.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.233+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee0002b170 osd=3 3 2024-03-27T21:41:17.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.233+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee0002b170 osd=3 3 2024-03-27T21:41:17.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.233+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee0002b170 osd=3 3 2024-03-27T21:41:17.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.233+0000 7fee0705c640 15 client.4941.objecter _session_command_op_remove -1 7 2024-03-27T21:41:17.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.233+0000 7fee0705c640 20 client.4941.objecter get_session s=0x7fee0002b170 osd=3 3 2024-03-27T21:41:17.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.233+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign 3 7 2024-03-27T21:41:17.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.233+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee0002b170 osd=3 4 2024-03-27T21:41:17.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.233+0000 7fee0705c640 10 client.4941.objecter _send_command 7 2024-03-27T21:41:17.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.233+0000 7fee0705c640 1 -- 172.21.15.60:0/2194545019 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fee001da7f0 con 0x7fee0002b7f0 2024-03-27T21:41:17.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.234+0000 7fedfd7fa640 1 -- 172.21.15.60:0/2194545019 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fee001da7f0 con 0x7fee0002b7f0 2024-03-27T21:41:17.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.234+0000 7fedfd7fa640 10 client.4941.objecter ms_dispatch 0x7fee00000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:17.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.234+0000 7fedfd7fa640 10 client.4941.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:17.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.234+0000 7fedfd7fa640 20 client.4941.objecter put_session s=0x7fee0002b170 osd=3 4 2024-03-27T21:41:17.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.234+0000 7fedfd7fa640 15 client.4941.objecter _session_command_op_remove 3 7 2024-03-27T21:41:17.223 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:41:17.225 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:41:17.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.235+0000 7fee0705c640 10 client.4941.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:17.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.235+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign -1 8 2024-03-27T21:41:17.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.235+0000 7fee0705c640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fee001eb680 0x7fee001eda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:17.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.235+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee001eb000 osd=4 3 2024-03-27T21:41:17.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.235+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee001eb000 osd=4 3 2024-03-27T21:41:17.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.235+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee001eb000 osd=4 3 2024-03-27T21:41:17.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.235+0000 7fee0705c640 15 client.4941.objecter _session_command_op_remove -1 8 2024-03-27T21:41:17.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.235+0000 7fee0705c640 20 client.4941.objecter get_session s=0x7fee001eb000 osd=4 3 2024-03-27T21:41:17.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.235+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign 4 8 2024-03-27T21:41:17.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.235+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee001eb000 osd=4 4 2024-03-27T21:41:17.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.235+0000 7fee0705c640 10 client.4941.objecter _send_command 8 2024-03-27T21:41:17.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.235+0000 7fee0705c640 1 -- 172.21.15.60:0/2194545019 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7fee001f0840 con 0x7fee001eb680 2024-03-27T21:41:17.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.235+0000 7fee055f9640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fee001eb680 0x7fee001eda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:17.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.235+0000 7fee055f9640 10 monclient: get_auth_request con 0x7fee001eb680 auth_method 0 2024-03-27T21:41:17.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.235+0000 7fee055f9640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fee001eb680 0x7fee001eda70 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:17.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.236+0000 7fedfd7fa640 10 client.4941.objecter ms_handle_connect 0x7fee001eb680 2024-03-27T21:41:17.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.236+0000 7fedfd7fa640 1 -- 172.21.15.60:0/2194545019 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fee001f0840 con 0x7fee001eb680 2024-03-27T21:41:17.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.236+0000 7fedfd7fa640 10 client.4941.objecter ms_dispatch 0x7fee00000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:41:17.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.236+0000 7fedfd7fa640 10 client.4941.objecter _finish_command 8 = system:0 2024-03-27T21:41:17.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.236+0000 7fedfd7fa640 20 client.4941.objecter put_session s=0x7fee001eb000 osd=4 4 2024-03-27T21:41:17.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.236+0000 7fedfd7fa640 15 client.4941.objecter _session_command_op_remove 4 8 2024-03-27T21:41:17.238 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:17.252+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:17.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.264+0000 7fee0705c640 10 client.4941.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:17.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.264+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign -1 9 2024-03-27T21:41:17.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.264+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee001eb000 osd=4 3 2024-03-27T21:41:17.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.264+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee001eb000 osd=4 3 2024-03-27T21:41:17.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.264+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee001eb000 osd=4 3 2024-03-27T21:41:17.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.264+0000 7fee0705c640 15 client.4941.objecter _session_command_op_remove -1 9 2024-03-27T21:41:17.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.264+0000 7fee0705c640 20 client.4941.objecter get_session s=0x7fee001eb000 osd=4 3 2024-03-27T21:41:17.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.264+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign 4 9 2024-03-27T21:41:17.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.264+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee001eb000 osd=4 4 2024-03-27T21:41:17.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.264+0000 7fee0705c640 10 client.4941.objecter _send_command 9 2024-03-27T21:41:17.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.264+0000 7fee0705c640 1 -- 172.21.15.60:0/2194545019 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fee00066210 con 0x7fee001eb680 2024-03-27T21:41:17.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.266+0000 7fedfd7fa640 1 -- 172.21.15.60:0/2194545019 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fee00066210 con 0x7fee001eb680 2024-03-27T21:41:17.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.266+0000 7fedfd7fa640 10 client.4941.objecter ms_dispatch 0x7fee00000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:17.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.266+0000 7fedfd7fa640 10 client.4941.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:17.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.266+0000 7fedfd7fa640 20 client.4941.objecter put_session s=0x7fee001eb000 osd=4 4 2024-03-27T21:41:17.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.266+0000 7fedfd7fa640 15 client.4941.objecter _session_command_op_remove 4 9 2024-03-27T21:41:17.253 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:41:17.254 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:41:17.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.267+0000 7fee0705c640 10 client.4941.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:17.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.267+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign -1 10 2024-03-27T21:41:17.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.267+0000 7fee0705c640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fee00066380 0x7fee001ef130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:17.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.267+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee00066210 osd=5 3 2024-03-27T21:41:17.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.267+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee00066210 osd=5 3 2024-03-27T21:41:17.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.267+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee00066210 osd=5 3 2024-03-27T21:41:17.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.267+0000 7fee0705c640 15 client.4941.objecter _session_command_op_remove -1 10 2024-03-27T21:41:17.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.267+0000 7fee0705c640 20 client.4941.objecter get_session s=0x7fee00066210 osd=5 3 2024-03-27T21:41:17.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.267+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign 5 10 2024-03-27T21:41:17.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.267+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee00066210 osd=5 4 2024-03-27T21:41:17.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.267+0000 7fee0705c640 10 client.4941.objecter _send_command 10 2024-03-27T21:41:17.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.267+0000 7fee0705c640 1 -- 172.21.15.60:0/2194545019 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fee0013c780 con 0x7fee00066380 2024-03-27T21:41:17.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.267+0000 7fee04df8640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fee00066380 0x7fee001ef130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:17.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.267+0000 7fee04df8640 10 monclient: get_auth_request con 0x7fee00066380 auth_method 0 2024-03-27T21:41:17.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.268+0000 7fee04df8640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fee00066380 0x7fee001ef130 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:17.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.268+0000 7fedfd7fa640 10 client.4941.objecter ms_handle_connect 0x7fee00066380 2024-03-27T21:41:17.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.268+0000 7fedfd7fa640 1 -- 172.21.15.60:0/2194545019 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fee0013c780 con 0x7fee00066380 2024-03-27T21:41:17.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.268+0000 7fedfd7fa640 10 client.4941.objecter ms_dispatch 0x7fee00000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:41:17.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.268+0000 7fedfd7fa640 10 client.4941.objecter _finish_command 10 = system:0 2024-03-27T21:41:17.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.268+0000 7fedfd7fa640 20 client.4941.objecter put_session s=0x7fee00066210 osd=5 4 2024-03-27T21:41:17.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.268+0000 7fedfd7fa640 15 client.4941.objecter _session_command_op_remove 5 10 2024-03-27T21:41:17.267 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:17.267 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:17.277 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:41:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.296+0000 7fee0705c640 10 client.4941.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.296+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign -1 11 2024-03-27T21:41:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.296+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee00066210 osd=5 3 2024-03-27T21:41:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.296+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee00066210 osd=5 3 2024-03-27T21:41:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.296+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee00066210 osd=5 3 2024-03-27T21:41:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.296+0000 7fee0705c640 15 client.4941.objecter _session_command_op_remove -1 11 2024-03-27T21:41:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.296+0000 7fee0705c640 20 client.4941.objecter get_session s=0x7fee00066210 osd=5 3 2024-03-27T21:41:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.296+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign 5 11 2024-03-27T21:41:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.296+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee00066210 osd=5 4 2024-03-27T21:41:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.296+0000 7fee0705c640 10 client.4941.objecter _send_command 11 2024-03-27T21:41:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.296+0000 7fee0705c640 1 -- 172.21.15.60:0/2194545019 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fee0013c780 con 0x7fee00066380 2024-03-27T21:41:17.285 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:41:17.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.298+0000 7fedfd7fa640 1 -- 172.21.15.60:0/2194545019 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fee0013c780 con 0x7fee00066380 2024-03-27T21:41:17.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.298+0000 7fedfd7fa640 10 client.4941.objecter ms_dispatch 0x7fee00000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:17.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.298+0000 7fedfd7fa640 10 client.4941.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:17.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.298+0000 7fedfd7fa640 20 client.4941.objecter put_session s=0x7fee00066210 osd=5 4 2024-03-27T21:41:17.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.298+0000 7fedfd7fa640 15 client.4941.objecter _session_command_op_remove 5 11 2024-03-27T21:41:17.286 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:41:17.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.298+0000 7fee0705c640 10 client.4941.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:17.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.298+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign -1 12 2024-03-27T21:41:17.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.298+0000 7fee0705c640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fee001efb80 0x7fee001effe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:17.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.298+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee0013c780 osd=6 3 2024-03-27T21:41:17.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.298+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee0013c780 osd=6 3 2024-03-27T21:41:17.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.298+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee0013c780 osd=6 3 2024-03-27T21:41:17.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.298+0000 7fee0705c640 15 client.4941.objecter _session_command_op_remove -1 12 2024-03-27T21:41:17.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.298+0000 7fee0705c640 20 client.4941.objecter get_session s=0x7fee0013c780 osd=6 3 2024-03-27T21:41:17.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.298+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign 6 12 2024-03-27T21:41:17.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.298+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee0013c780 osd=6 4 2024-03-27T21:41:17.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.298+0000 7fee0705c640 10 client.4941.objecter _send_command 12 2024-03-27T21:41:17.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.298+0000 7fee0705c640 1 -- 172.21.15.60:0/2194545019 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fee001f0520 con 0x7fee001efb80 2024-03-27T21:41:17.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.299+0000 7fedfffff640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fee001efb80 0x7fee001effe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:17.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.299+0000 7fedfffff640 10 monclient: get_auth_request con 0x7fee001efb80 auth_method 0 2024-03-27T21:41:17.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.299+0000 7fedfd7fa640 10 client.4941.objecter ms_handle_connect 0x7fee001efb80 2024-03-27T21:41:17.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.299+0000 7fedfffff640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fee001efb80 0x7fee001effe0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:17.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.300+0000 7fedfd7fa640 1 -- 172.21.15.60:0/2194545019 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fee001f0520 con 0x7fee001efb80 2024-03-27T21:41:17.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.300+0000 7fedfd7fa640 10 client.4941.objecter ms_dispatch 0x7fee00000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:41:17.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.300+0000 7fedfd7fa640 10 client.4941.objecter _finish_command 12 = system:0 2024-03-27T21:41:17.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.300+0000 7fedfd7fa640 20 client.4941.objecter put_session s=0x7fee0013c780 osd=6 4 2024-03-27T21:41:17.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.300+0000 7fedfd7fa640 15 client.4941.objecter _session_command_op_remove 6 12 2024-03-27T21:41:17.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.320+0000 7fee0705c640 10 client.4941.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:17.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.320+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign -1 13 2024-03-27T21:41:17.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.320+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee0013c780 osd=6 3 2024-03-27T21:41:17.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.320+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee0013c780 osd=6 3 2024-03-27T21:41:17.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.320+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee0013c780 osd=6 3 2024-03-27T21:41:17.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.320+0000 7fee0705c640 15 client.4941.objecter _session_command_op_remove -1 13 2024-03-27T21:41:17.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.320+0000 7fee0705c640 20 client.4941.objecter get_session s=0x7fee0013c780 osd=6 3 2024-03-27T21:41:17.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.320+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign 6 13 2024-03-27T21:41:17.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.320+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee0013c780 osd=6 4 2024-03-27T21:41:17.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.320+0000 7fee0705c640 10 client.4941.objecter _send_command 13 2024-03-27T21:41:17.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.320+0000 7fee0705c640 1 -- 172.21.15.60:0/2194545019 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fee001f0520 con 0x7fee001efb80 2024-03-27T21:41:17.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.322+0000 7fedfd7fa640 1 -- 172.21.15.60:0/2194545019 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fee001f0520 con 0x7fee001efb80 2024-03-27T21:41:17.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.322+0000 7fedfd7fa640 10 client.4941.objecter ms_dispatch 0x7fee00000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:17.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.322+0000 7fedfd7fa640 10 client.4941.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:17.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.322+0000 7fedfd7fa640 20 client.4941.objecter put_session s=0x7fee0013c780 osd=6 4 2024-03-27T21:41:17.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.322+0000 7fedfd7fa640 15 client.4941.objecter _session_command_op_remove 6 13 2024-03-27T21:41:17.309 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:41:17.310 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:41:17.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.322+0000 7fee0705c640 10 client.4941.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:17.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.322+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign -1 14 2024-03-27T21:41:17.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.322+0000 7fee0705c640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fee001f4ee0 0x7fee001f72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:17.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.322+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee001f0520 osd=7 3 2024-03-27T21:41:17.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.322+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee001f0520 osd=7 3 2024-03-27T21:41:17.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.322+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee001f0520 osd=7 3 2024-03-27T21:41:17.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.322+0000 7fee0705c640 15 client.4941.objecter _session_command_op_remove -1 14 2024-03-27T21:41:17.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.322+0000 7fee0705c640 20 client.4941.objecter get_session s=0x7fee001f0520 osd=7 3 2024-03-27T21:41:17.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.322+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign 7 14 2024-03-27T21:41:17.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.322+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee001f0520 osd=7 4 2024-03-27T21:41:17.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.322+0000 7fee0705c640 10 client.4941.objecter _send_command 14 2024-03-27T21:41:17.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.322+0000 7fee0705c640 1 -- 172.21.15.60:0/2194545019 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fee001fa1a0 con 0x7fee001f4ee0 2024-03-27T21:41:17.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.323+0000 7fee055f9640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fee001f4ee0 0x7fee001f72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:17.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.323+0000 7fee055f9640 10 monclient: get_auth_request con 0x7fee001f4ee0 auth_method 0 2024-03-27T21:41:17.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.323+0000 7fedfd7fa640 10 client.4941.objecter ms_handle_connect 0x7fee001f4ee0 2024-03-27T21:41:17.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.323+0000 7fee055f9640 1 --2- 172.21.15.60:0/2194545019 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fee001f4ee0 0x7fee001f72d0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:17.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.324+0000 7fedfd7fa640 1 -- 172.21.15.60:0/2194545019 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fee001fa1a0 con 0x7fee001f4ee0 2024-03-27T21:41:17.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.324+0000 7fedfd7fa640 10 client.4941.objecter ms_dispatch 0x7fee00000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:41:17.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.324+0000 7fedfd7fa640 10 client.4941.objecter _finish_command 14 = system:0 2024-03-27T21:41:17.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.324+0000 7fedfd7fa640 20 client.4941.objecter put_session s=0x7fee001f0520 osd=7 4 2024-03-27T21:41:17.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.324+0000 7fedfd7fa640 15 client.4941.objecter _session_command_op_remove 7 14 2024-03-27T21:41:17.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.342+0000 7fee0705c640 10 client.4941.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:17.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.342+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign -1 15 2024-03-27T21:41:17.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.342+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee001f0520 osd=7 3 2024-03-27T21:41:17.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.342+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee001f0520 osd=7 3 2024-03-27T21:41:17.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.342+0000 7fee0705c640 20 client.4941.objecter _get_session s=0x7fee001f0520 osd=7 3 2024-03-27T21:41:17.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.342+0000 7fee0705c640 15 client.4941.objecter _session_command_op_remove -1 15 2024-03-27T21:41:17.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.342+0000 7fee0705c640 20 client.4941.objecter get_session s=0x7fee001f0520 osd=7 3 2024-03-27T21:41:17.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.342+0000 7fee0705c640 15 client.4941.objecter _session_command_op_assign 7 15 2024-03-27T21:41:17.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.342+0000 7fee0705c640 20 client.4941.objecter put_session s=0x7fee001f0520 osd=7 4 2024-03-27T21:41:17.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.342+0000 7fee0705c640 10 client.4941.objecter _send_command 15 2024-03-27T21:41:17.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.342+0000 7fee0705c640 1 -- 172.21.15.60:0/2194545019 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fee001fa1a0 con 0x7fee001f4ee0 2024-03-27T21:41:17.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.343+0000 7fedfd7fa640 1 -- 172.21.15.60:0/2194545019 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fee001fa1a0 con 0x7fee001f4ee0 2024-03-27T21:41:17.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.343+0000 7fedfd7fa640 10 client.4941.objecter ms_dispatch 0x7fee00000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:17.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.343+0000 7fedfd7fa640 10 client.4941.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:17.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.343+0000 7fedfd7fa640 20 client.4941.objecter put_session s=0x7fee001f0520 osd=7 4 2024-03-27T21:41:17.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:17.343+0000 7fedfd7fa640 15 client.4941.objecter _session_command_op_remove 7 15 2024-03-27T21:41:17.332 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:41:17.332 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:41:17.333 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:41:17.334 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:41:17.376 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:17.390+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:17.423 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:17.424 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:17.433 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:41:17.439 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:17.453+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:17.540 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:17.554+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:17.587 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:17.587 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:17.597 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:41:17.640 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:17.654+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:17.706 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:17.706 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:17.716 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:41:17.741 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:17.755+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:17.825 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:17.826 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:17.835 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:41:17.842 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:17.856+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:17.943 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:17.957+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:17.944 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:17.945 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:17.954 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:41:17.993 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-27T21:41:17.993 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T21:41:17.994 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-27T21:41:17.994 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.367981 2024-03-27T21:41:17.994 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 0.36798116591749364 2024-03-27T21:41:18.044 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:18.058+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:18.073 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:18.073 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:18.085 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:41:18.145 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:18.158+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.162+0000 7f3273ce0640 10 monclient: get_monmap_and_config 2024-03-27T21:41:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.162+0000 7f3273ce0640 10 monclient: build_initial_monmap 2024-03-27T21:41:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.162+0000 7f3273ce0640 10 monclient: monmap: 2024-03-27T21:41:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:18.163119+0000 2024-03-27T21:41:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:18.163119+0000 2024-03-27T21:41:18.149 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:18.149 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:18.149 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:18.149 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:18.149 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:18.149 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:18.149 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:18.149 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:18.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.162+0000 7f3273ce0640 1 Processor -- start 2024-03-27T21:41:18.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.163+0000 7f3273ce0640 1 -- start start 2024-03-27T21:41:18.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.163+0000 7f3273ce0640 10 monclient: init 2024-03-27T21:41:18.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.163+0000 7f3273ce0640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:18.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.163+0000 7f3273ce0640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-03-27T21:41:18.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.163+0000 7f3273ce0640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f326c0b40b0 0x7f326c0a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:18.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.163+0000 7f3273ce0640 10 monclient(hunting): picked mon.noname-c con 0x7f326c0b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:18.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.164+0000 7f3273ce0640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f326c13b020 0x7f326c13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:18.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.164+0000 7f327127b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f326c13b020 0x7f326c13b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:18.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.164+0000 7f3271a7c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f326c0b40b0 0x7f326c0a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:18.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.164+0000 7f3271a7c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f326c0b40b0 0x7f326c0a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:40328/0 (socket says 172.21.15.60:40328) 2024-03-27T21:41:18.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.164+0000 7f327127b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f326c13b020 0x7f326c13b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:37888/0 (socket says 172.21.15.60:37888) 2024-03-27T21:41:18.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.164+0000 7f3273ce0640 10 monclient(hunting): picked mon.noname-b con 0x7f326c13b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:18.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.164+0000 7f3271a7c640 1 -- 172.21.15.60:0/1830862373 learned_addr learned my addr 172.21.15.60:0/1830862373 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:18.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.164+0000 7f3273ce0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f326c1416f0 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:18.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.164+0000 7f3273ce0640 1 -- 172.21.15.60:0/1830862373 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326c0b3830 con 0x7f326c1416f0 2024-03-27T21:41:18.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.164+0000 7f3273ce0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:18.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.164+0000 7f3273ce0640 1 -- 172.21.15.60:0/1830862373 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f326c0b4490 con 0x7f326c0b40b0 2024-03-27T21:41:18.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.164+0000 7f3273ce0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:18.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.164+0000 7f3273ce0640 1 -- 172.21.15.60:0/1830862373 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f326c142290 con 0x7f326c13b020 2024-03-27T21:41:18.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.164+0000 7f3273ce0640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:18.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3273ce0640 10 monclient(hunting): authenticate will time out at 919.385972s 2024-03-27T21:41:18.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3271a7c640 10 monclient(hunting): get_auth_request con 0x7f326c0b40b0 auth_method 0 2024-03-27T21:41:18.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3271a7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:18.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3271a7c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:18.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3271a7c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:18.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f327127b640 10 monclient(hunting): get_auth_request con 0x7f326c13b020 auth_method 0 2024-03-27T21:41:18.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f327127b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:18.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f327127b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:18.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f327127b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:18.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3270a7a640 1 -- 172.21.15.60:0/1830862373 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306966710 0 0) 0x7f326c0b3830 con 0x7f326c1416f0 2024-03-27T21:41:18.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3270a7a640 10 monclient(hunting): _negotiate 2024-03-27T21:41:18.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3270a7a640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:18.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3270a7a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:18.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3270a7a640 10 monclient(hunting): my global_id is 4873 2024-03-27T21:41:18.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3270a7a640 1 -- 172.21.15.60:0/1830862373 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3258003520 con 0x7f326c1416f0 2024-03-27T21:41:18.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f327127b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:18.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f327127b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:18.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f327127b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:18.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3271a7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:18.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3271a7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:18.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3271a7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:18.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3271a7c640 10 monclient(hunting): handle_auth_done global_id 4876 payload 274 2024-03-27T21:41:18.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3270a7a640 1 -- 172.21.15.60:0/1830862373 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2073919552 0 0) 0x7f3258003520 con 0x7f326c1416f0 2024-03-27T21:41:18.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3271a7c640 1 -- 172.21.15.60:0/1830862373 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f326c13b020 msgr2=0x7f326c13b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:18.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3271a7c640 1 --2- 172.21.15.60:0/1830862373 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f326c13b020 0x7f326c13b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:18.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3271a7c640 1 -- 172.21.15.60:0/1830862373 >> v1:172.21.15.191:6789/0 conn(0x7f326c1416f0 legacy=0x7f326c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:18.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3271a7c640 10 monclient: _finish_hunting 0 2024-03-27T21:41:18.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3271a7c640 1 monclient: found mon.noname-c 2024-03-27T21:41:18.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3271a7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:18.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3271a7c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:41:18.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3271a7c640 1 -- 172.21.15.60:0/1830862373 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326c0aa520 con 0x7f326c0b40b0 2024-03-27T21:41:18.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f3270a7a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:41:18.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.165+0000 7f327127b640 1 --2- 172.21.15.60:0/1830862373 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f326c13b020 0x7f326c13b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:18.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.166+0000 7f3271a7c640 1 --2- 172.21.15.60:0/1830862373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c0b40b0 0x7f326c0a8b90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3268009870 tx=0x7f326802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27e2e205213e90a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:18.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.167+0000 7f3270a7a640 1 -- 172.21.15.60:0/1830862373 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32640019f0 con 0x7f326c0b40b0 2024-03-27T21:41:18.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.167+0000 7f3270a7a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:18.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.167+0000 7f3270a7a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:18.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.167+0000 7f3270a7a640 10 monclient: dump: 2024-03-27T21:41:18.168 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:18.168 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:18.168 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:18.168 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:18.168 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:18.168 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:18.168 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:18.168 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:18.168 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:18.168 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:18.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.167+0000 7f3270a7a640 10 monclient: _finish_auth 0 2024-03-27T21:41:18.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.167+0000 7f3270a7a640 10 monclient: _check_auth_tickets 2024-03-27T21:41:18.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.167+0000 7f3270a7a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:18.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.167+0000 7f3270a7a640 1 -- 172.21.15.60:0/1830862373 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3268031540 con 0x7f326c0b40b0 2024-03-27T21:41:18.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.167+0000 7f3270a7a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:18.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.167+0000 7f3270a7a640 1 -- 172.21.15.60:0/1830862373 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32640019f0 con 0x7f326c0b40b0 2024-03-27T21:41:18.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.167+0000 7f3270a7a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:18.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.167+0000 7f3270a7a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:18.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.167+0000 7f3270a7a640 10 monclient: dump: 2024-03-27T21:41:18.169 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:18.169 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:18.169 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:18.169 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:18.169 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:18.169 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:18.169 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:18.169 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:18.169 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:18.169 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:18.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.167+0000 7f3273ce0640 5 monclient: authenticate success, global_id 4876 2024-03-27T21:41:18.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.167+0000 7f3273ce0640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:18.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.167+0000 7f3273ce0640 4 set_mon_vals no callback set 2024-03-27T21:41:18.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.168+0000 7f3273ce0640 10 monclient: shutdown 2024-03-27T21:41:18.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.168+0000 7f3273ce0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:18.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.168+0000 7f3273ce0640 1 -- 172.21.15.60:0/1830862373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c0b40b0 msgr2=0x7f326c0a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:18.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.168+0000 7f3273ce0640 1 --2- 172.21.15.60:0/1830862373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c0b40b0 0x7f326c0a8b90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3268009870 tx=0x7f326802b860 comp rx=0 tx=0).stop 2024-03-27T21:41:18.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.168+0000 7f3273ce0640 1 -- 172.21.15.60:0/1830862373 shutdown_connections 2024-03-27T21:41:18.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.168+0000 7f3273ce0640 1 --2- 172.21.15.60:0/1830862373 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f326c13b020 0x7f326c13b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:18.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.168+0000 7f3273ce0640 1 --2- 172.21.15.60:0/1830862373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c0b40b0 0x7f326c0a8b90 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:18.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.168+0000 7f3273ce0640 1 -- 172.21.15.60:0/1830862373 >> 172.21.15.60:0/1830862373 conn(0x7f326c0608e0 msgr2=0x7f326c060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:18.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.168+0000 7f3273ce0640 1 -- 172.21.15.60:0/1830862373 shutdown_connections 2024-03-27T21:41:18.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.168+0000 7f3273ce0640 1 -- 172.21.15.60:0/1830862373 wait complete. 2024-03-27T21:41:18.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.169+0000 7f3273ce0640 10 monclient: build_initial_monmap 2024-03-27T21:41:18.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.169+0000 7f3273ce0640 10 monclient: monmap: 2024-03-27T21:41:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:18.171 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.169+0000 7f3273ce0640 1 librados: starting msgr at 2024-03-27T21:41:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.169+0000 7f3273ce0640 1 librados: starting objecter 2024-03-27T21:41:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.169+0000 7f3273ce0640 1 Processor -- start 2024-03-27T21:41:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.169+0000 7f3273ce0640 1 -- start start 2024-03-27T21:41:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.169+0000 7f3273ce0640 1 librados: setting wanted keys 2024-03-27T21:41:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.169+0000 7f3273ce0640 1 librados: calling monclient init 2024-03-27T21:41:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.169+0000 7f3273ce0640 10 monclient: init 2024-03-27T21:41:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3273ce0640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3273ce0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:41:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3273ce0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f326c0b40b0 0x7f326c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3273ce0640 10 monclient(hunting): picked mon.noname-a con 0x7f326c0b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3273ce0640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f326c13b020 0x7f326c1de530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3273ce0640 10 monclient(hunting): picked mon.noname-c con 0x7f326c13b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3273ce0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c1416f0 0x7f326c13a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f327127b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f326c13b020 0x7f326c1de530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3271a7c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f326c0b40b0 0x7f326c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3271a7c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f326c0b40b0 0x7f326c06d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37326/0 (socket says 172.21.15.60:37326) 2024-03-27T21:41:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f327127b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f326c13b020 0x7f326c1de530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:37894/0 (socket says 172.21.15.60:37894) 2024-03-27T21:41:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3273ce0640 10 monclient(hunting): picked mon.noname-b con 0x7f326c1416f0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3273ce0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3273ce0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c1de0a0 con 0x7f326c0b40b0 2024-03-27T21:41:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3273ce0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3273ce0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c06f7f0 con 0x7f326c1416f0 2024-03-27T21:41:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3273ce0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3273ce0640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f326c1ddd30 con 0x7f326c13b020 2024-03-27T21:41:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3273ce0640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3273ce0640 10 monclient(hunting): authenticate will time out at 919.392632s 2024-03-27T21:41:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3271a7c640 1 -- 172.21.15.60:0/2005254861 learned_addr learned my addr 172.21.15.60:0/2005254861 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3271a7c640 10 monclient(hunting): get_auth_request con 0x7f326c0b40b0 auth_method 0 2024-03-27T21:41:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3271a7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3271a7c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3271a7c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3271a7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3271a7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f3271a7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.171+0000 7f327227d640 1 --2- 172.21.15.60:0/2005254861 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c1416f0 0x7f326c13a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f327227d640 10 monclient(hunting): get_auth_request con 0x7f326c1416f0 auth_method 0 2024-03-27T21:41:18.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f327227d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:18.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f327227d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:18.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f327227d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:18.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f327127b640 10 monclient(hunting): get_auth_request con 0x7f326c13b020 auth_method 0 2024-03-27T21:41:18.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f327127b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:18.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f327127b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:18.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f327127b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:18.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f3271a7c640 10 monclient(hunting): handle_auth_done global_id 4944 payload 824 2024-03-27T21:41:18.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f3271a7c640 1 -- 172.21.15.60:0/2005254861 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f326c13b020 msgr2=0x7f326c1de530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:18.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f3271a7c640 1 --2- 172.21.15.60:0/2005254861 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f326c13b020 0x7f326c1de530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:18.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f3271a7c640 1 -- 172.21.15.60:0/2005254861 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c1416f0 msgr2=0x7f326c13a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:18.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f3271a7c640 1 --2- 172.21.15.60:0/2005254861 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c1416f0 0x7f326c13a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:18.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f3271a7c640 10 monclient: _finish_hunting 0 2024-03-27T21:41:18.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f3271a7c640 1 monclient: found mon.noname-a 2024-03-27T21:41:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f3271a7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f3271a7c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f3271a7c640 1 -- 172.21.15.60:0/2005254861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c009400 con 0x7f326c0b40b0 2024-03-27T21:41:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f327127b640 1 --2- 172.21.15.60:0/2005254861 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f326c13b020 0x7f326c1de530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:41:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f327227d640 1 --2- 172.21.15.60:0/2005254861 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c1416f0 0x7f326c13a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:41:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f3271a7c640 1 --2- 172.21.15.60:0/2005254861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f326c0b40b0 0x7f326c06d9c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f326802b8a0 tx=0x7f32680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f32627fc640 10 client.?.objecter ms_handle_connect 0x7f326c0b40b0 2024-03-27T21:41:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f32627fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f32627fc640 1 -- 172.21.15.60:0/2005254861 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32640019f0 con 0x7f326c0b40b0 2024-03-27T21:41:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f32627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f32627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f32627fc640 10 monclient: dump: 2024-03-27T21:41:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:18.176 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:18.176 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:18.176 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:18.176 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:18.176 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:18.176 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:18.176 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:18.176 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:18.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f32627fc640 10 monclient: _finish_auth 0 2024-03-27T21:41:18.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f32627fc640 10 monclient: _check_auth_tickets 2024-03-27T21:41:18.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f32627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:18.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f32627fc640 1 -- 172.21.15.60:0/2005254861 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32680336c0 con 0x7f326c0b40b0 2024-03-27T21:41:18.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f32627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:18.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f32627fc640 1 -- 172.21.15.60:0/2005254861 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3268033a30 con 0x7f326c0b40b0 2024-03-27T21:41:18.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f32627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:18.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f32627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:18.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f32627fc640 10 monclient: dump: 2024-03-27T21:41:18.177 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:18.177 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:18.177 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:18.177 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:18.177 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:18.177 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:18.177 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:18.177 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:18.177 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:18.177 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:18.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f3273ce0640 5 monclient: authenticate success, global_id 4944 2024-03-27T21:41:18.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f3263fff640 4 set_mon_vals no callback set 2024-03-27T21:41:18.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f3273ce0640 10 monclient: _renew_subs 2024-03-27T21:41:18.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f3273ce0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:18.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f3273ce0640 1 -- 172.21.15.60:0/2005254861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32680094d0 con 0x7f326c0b40b0 2024-03-27T21:41:18.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f3273ce0640 10 client.4944.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:18.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f3273ce0640 10 monclient: _renew_subs 2024-03-27T21:41:18.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f3273ce0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:18.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f3273ce0640 1 -- 172.21.15.60:0/2005254861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f326c1dec50 con 0x7f326c0b40b0 2024-03-27T21:41:18.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.172+0000 7f3273ce0640 1 librados: init done 2024-03-27T21:41:18.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.174+0000 7f3273ce0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:18.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.174+0000 7f3273ce0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:18.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.174+0000 7f3273ce0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:18.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.174+0000 7f3273ce0640 1 -- 172.21.15.60:0/2005254861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f326c1ddef0 con 0x7f326c0b40b0 2024-03-27T21:41:18.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.174+0000 7f32627fc640 1 -- 172.21.15.60:0/2005254861 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f326803a040 con 0x7f326c0b40b0 2024-03-27T21:41:18.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.175+0000 7f32627fc640 1 --2- 172.21.15.60:0/2005254861 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f325006c590 0x7f325006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:18.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.175+0000 7f32627fc640 1 -- 172.21.15.60:0/2005254861 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f32680aa690 con 0x7f326c0b40b0 2024-03-27T21:41:18.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.175+0000 7f32627fc640 10 client.4944.objecter ms_dispatch 0x7f326c000b90 osd_map(56..56 src has 1..56) v4 2024-03-27T21:41:18.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.175+0000 7f32627fc640 3 client.4944.objecter handle_osd_map got epochs [56,56] > 0 2024-03-27T21:41:18.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.175+0000 7f32627fc640 3 client.4944.objecter handle_osd_map decoding full epoch 56 2024-03-27T21:41:18.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.175+0000 7f32627fc640 20 client.4944.objecter dump_active .. 0 homeless 2024-03-27T21:41:18.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.175+0000 7f327127b640 1 --2- 172.21.15.60:0/2005254861 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f325006c590 0x7f325006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:18.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.175+0000 7f327127b640 10 monclient: get_auth_request con 0x7f325006c590 auth_method 0 2024-03-27T21:41:18.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.176+0000 7f32627fc640 10 client.4944.objecter ms_handle_connect 0x7f325006c590 2024-03-27T21:41:18.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.176+0000 7f327127b640 1 --2- 172.21.15.60:0/2005254861 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f325006c590 0x7f325006ea50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f326c06f490 tx=0x7f325c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:18.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.183+0000 7f32627fc640 1 -- 172.21.15.60:0/2005254861 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3268075bf0 con 0x7f326c0b40b0 2024-03-27T21:41:18.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.183+0000 7f32627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:18.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.183+0000 7f32627fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:18.245 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:18.259+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:18.246 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:18.246 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:18.255 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:41:18.346 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:18.360+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:18.373 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:18.373 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:18.382 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:41:18.447 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:18.461+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:18.529 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:18.535 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:18.539 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:41:18.548 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:18.562+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:18.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.575+0000 7f3273ce0640 10 start_mon_command cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.36798116591749364}] 2024-03-27T21:41:18.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.575+0000 7f3273ce0640 10 monclient: _send_command 2 [{"prefix": "osd primary-affinity", "id": 5, "weight": 0.36798116591749364}] 2024-03-27T21:41:18.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.575+0000 7f3273ce0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:18.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:18.575+0000 7f3273ce0640 1 -- 172.21.15.60:0/2005254861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 0.36798116591749364} v 0) v1 -- 0x7f326c0b6c30 con 0x7f326c0b40b0 2024-03-27T21:41:18.644 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:18.644 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:18.649 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:18.663+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:18.652 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:41:18.750 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:18.764+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:18.758 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:18.758 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:18.767 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:41:18.851 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:18.865+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:18.912 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:18.912 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:18.922 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:41:18.952 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:18.966+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:19.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.014+0000 7f32627fc640 1 -- 172.21.15.60:0/2005254861 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 0.36798116591749364}]=0 set osd.5 primary-affinity to 0.367981 (8241162) v57) v1 ==== 156+0+0 (secure 0 0 0) 0x7f32680380b0 con 0x7f326c0b40b0 2024-03-27T21:41:19.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.015+0000 7f32627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd primary-affinity", "id": 5, "weight": 0.36798116591749364}] 2024-03-27T21:41:19.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.015+0000 7f32627fc640 10 monclient: _finish_command 2 = system:0 set osd.5 primary-affinity to 0.367981 (8241162) 2024-03-27T21:41:19.001 INFO:teuthology.orchestra.run.smithi060.stderr:set osd.5 primary-affinity to 0.367981 (8241162) 2024-03-27T21:41:19.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.017+0000 7f3273ce0640 10 librados: watch_flush enter 2024-03-27T21:41:19.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.017+0000 7f3273ce0640 10 librados: watch_flush exit 2024-03-27T21:41:19.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.017+0000 7f3273ce0640 20 client.4944.objecter shutdown clearing up homeless session... 2024-03-27T21:41:19.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.017+0000 7f3273ce0640 10 client.4944.objecter successfully canceled tick 2024-03-27T21:41:19.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.018+0000 7f3273ce0640 1 -- 172.21.15.60:0/2005254861 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f325006c590 msgr2=0x7f325006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:19.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.018+0000 7f3273ce0640 1 --2- 172.21.15.60:0/2005254861 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f325006c590 0x7f325006ea50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f326c06f490 tx=0x7f325c008040 comp rx=0 tx=0).stop 2024-03-27T21:41:19.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.018+0000 7f3273ce0640 10 monclient: shutdown 2024-03-27T21:41:19.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.018+0000 7f3273ce0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:19.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.018+0000 7f3273ce0640 1 -- 172.21.15.60:0/2005254861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f326c0b40b0 msgr2=0x7f326c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:19.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.018+0000 7f3273ce0640 1 --2- 172.21.15.60:0/2005254861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f326c0b40b0 0x7f326c06d9c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f326802b8a0 tx=0x7f32680046c0 comp rx=0 tx=0).stop 2024-03-27T21:41:19.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.018+0000 7f3273ce0640 1 -- 172.21.15.60:0/2005254861 shutdown_connections 2024-03-27T21:41:19.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.018+0000 7f3273ce0640 1 --2- 172.21.15.60:0/2005254861 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f325006c590 0x7f325006ea50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:19.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.018+0000 7f3273ce0640 1 --2- 172.21.15.60:0/2005254861 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c1416f0 0x7f326c13a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:19.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.018+0000 7f3273ce0640 1 --2- 172.21.15.60:0/2005254861 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f326c13b020 0x7f326c1de530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:19.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.018+0000 7f3273ce0640 1 --2- 172.21.15.60:0/2005254861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f326c0b40b0 0x7f326c06d9c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:19.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.018+0000 7f3273ce0640 1 -- 172.21.15.60:0/2005254861 >> 172.21.15.60:0/2005254861 conn(0x7f326c0608e0 msgr2=0x7f326c140820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:19.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.018+0000 7f3273ce0640 1 -- 172.21.15.60:0/2005254861 shutdown_connections 2024-03-27T21:41:19.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.018+0000 7f3273ce0640 1 -- 172.21.15.60:0/2005254861 wait complete. 2024-03-27T21:41:19.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.018+0000 7f3273ce0640 1 librados: shutdown 2024-03-27T21:41:19.020 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-27T21:41:19.028 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:19.028 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:19.037 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:41:19.061 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:19.075+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:19.142 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:19.142 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:19.150 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:41:19.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.178+0000 7f64fe167640 10 monclient: get_monmap_and_config 2024-03-27T21:41:19.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.178+0000 7f64fe167640 10 monclient: build_initial_monmap 2024-03-27T21:41:19.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.178+0000 7f64fe167640 10 monclient: monmap: 2024-03-27T21:41:19.164 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:19.164 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:19.165 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:19.179521+0000 2024-03-27T21:41:19.165 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:19.179521+0000 2024-03-27T21:41:19.165 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:19.165 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:19.165 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:19.165 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:19.165 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:19.165 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:19.165 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:19.165 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.179+0000 7f64fe167640 1 Processor -- start 2024-03-27T21:41:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.179+0000 7f64fe167640 1 -- start start 2024-03-27T21:41:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.179+0000 7f64fe167640 10 monclient: init 2024-03-27T21:41:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.179+0000 7f64fe167640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.179+0000 7f64fe167640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-03-27T21:41:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.179+0000 7f64fe167640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f64f8071a50 0x7f64f80a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64fe167640 10 monclient(hunting): picked mon.noname-a con 0x7f64f8071a50 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64fe167640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f64f813e810 0x7f64f813ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64fe167640 10 monclient(hunting): picked mon.noname-c con 0x7f64f813e810 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64fe167640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f8141630 0x7f64f8141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64fe167640 10 monclient(hunting): picked mon.noname-b con 0x7f64f8141630 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64fe167640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64fe167640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f8071e30 con 0x7f64f8071a50 2024-03-27T21:41:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64fe167640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64fe167640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f813f190 con 0x7f64f813e810 2024-03-27T21:41:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64fe167640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64fe167640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f80a8e90 con 0x7f64f8141630 2024-03-27T21:41:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64fe167640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64fe167640 10 monclient(hunting): authenticate will time out at 920.401053s 2024-03-27T21:41:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f7fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f8141630 0x7f64f8141aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f6ffd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f64f813e810 0x7f64f813ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f7fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f8141630 0x7f64f8141aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:37898/0 (socket says 172.21.15.60:37898) 2024-03-27T21:41:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f7fff640 1 -- 172.21.15.60:0/1947150135 learned_addr learned my addr 172.21.15.60:0/1947150135 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f7fff640 10 monclient(hunting): get_auth_request con 0x7f64f8141630 auth_method 0 2024-03-27T21:41:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f6ffd640 10 monclient(hunting): get_auth_request con 0x7f64f813e810 auth_method 0 2024-03-27T21:41:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f77fe640 1 --2- 172.21.15.60:0/1947150135 >> v2:172.21.15.60:3300/0 conn(0x7f64f8071a50 0x7f64f80a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f7fff640 10 monclient(hunting): handle_auth_done global_id 4757 payload 274 2024-03-27T21:41:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f7fff640 1 -- 172.21.15.60:0/1947150135 >> v2:172.21.15.191:3300/0 conn(0x7f64f813e810 msgr2=0x7f64f813ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f7fff640 1 --2- 172.21.15.60:0/1947150135 >> v2:172.21.15.191:3300/0 conn(0x7f64f813e810 0x7f64f813ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f7fff640 1 -- 172.21.15.60:0/1947150135 >> v2:172.21.15.60:3300/0 conn(0x7f64f8071a50 msgr2=0x7f64f80a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f7fff640 1 --2- 172.21.15.60:0/1947150135 >> v2:172.21.15.60:3300/0 conn(0x7f64f8071a50 0x7f64f80a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f7fff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f7fff640 1 monclient: found mon.noname-b 2024-03-27T21:41:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f7fff640 1 -- 172.21.15.60:0/1947150135 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f80aa040 con 0x7f64f8141630 2024-03-27T21:41:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f77fe640 10 monclient: get_auth_request con 0x7f64f8071a50 auth_method 0 2024-03-27T21:41:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.180+0000 7f64f77fe640 1 --2- 172.21.15.60:0/1947150135 >> v2:172.21.15.60:3300/0 conn(0x7f64f8071a50 0x7f64f80a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:41:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.181+0000 7f64f7fff640 1 --2- 172.21.15.60:0/1947150135 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f8141630 0x7f64f8141aa0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f64f80709f0 tx=0x7f64f002bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e084d248de0ac91 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.181+0000 7f64f67fc640 1 -- 172.21.15.60:0/1947150135 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f002ca80 con 0x7f64f8141630 2024-03-27T21:41:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.182+0000 7f64f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.182+0000 7f64f67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.182+0000 7f64f67fc640 10 monclient: dump: 2024-03-27T21:41:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:19.173 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.182+0000 7f64f67fc640 10 monclient: _finish_auth 0 2024-03-27T21:41:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.182+0000 7f64f67fc640 10 monclient: _check_auth_tickets 2024-03-27T21:41:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.182+0000 7f64f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.182+0000 7f64fe167640 5 monclient: authenticate success, global_id 4757 2024-03-27T21:41:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.182+0000 7f64fe167640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.182+0000 7f64f67fc640 1 -- 172.21.15.60:0/1947150135 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f002cbe0 con 0x7f64f8141630 2024-03-27T21:41:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.182+0000 7f64f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.182+0000 7f64f67fc640 1 -- 172.21.15.60:0/1947150135 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f00343f0 con 0x7f64f8141630 2024-03-27T21:41:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.182+0000 7f64f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.182+0000 7f64f67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.182+0000 7f64f67fc640 10 monclient: dump: 2024-03-27T21:41:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:19.175 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.182+0000 7f64fe167640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.182+0000 7f64fe167640 4 set_mon_vals no callback set 2024-03-27T21:41:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.183+0000 7f64fe167640 10 monclient: shutdown 2024-03-27T21:41:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.183+0000 7f64fe167640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.183+0000 7f64fe167640 1 -- 172.21.15.60:0/1947150135 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f8141630 msgr2=0x7f64f8141aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.183+0000 7f64fe167640 1 --2- 172.21.15.60:0/1947150135 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f8141630 0x7f64f8141aa0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f64f80709f0 tx=0x7f64f002bee0 comp rx=0 tx=0).stop 2024-03-27T21:41:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.183+0000 7f64fe167640 1 -- 172.21.15.60:0/1947150135 shutdown_connections 2024-03-27T21:41:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.183+0000 7f64fe167640 1 --2- 172.21.15.60:0/1947150135 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f8141630 0x7f64f8141aa0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.183+0000 7f64fe167640 1 --2- 172.21.15.60:0/1947150135 >> v2:172.21.15.191:3300/0 conn(0x7f64f813e810 0x7f64f813ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:19.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.183+0000 7f64fe167640 1 --2- 172.21.15.60:0/1947150135 >> v2:172.21.15.60:3300/0 conn(0x7f64f8071a50 0x7f64f80a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:19.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.183+0000 7f64fe167640 1 -- 172.21.15.60:0/1947150135 >> 172.21.15.60:0/1947150135 conn(0x7f64f80b7300 msgr2=0x7f64f80b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:19.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.183+0000 7f64fe167640 1 -- 172.21.15.60:0/1947150135 shutdown_connections 2024-03-27T21:41:19.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.183+0000 7f64fe167640 1 -- 172.21.15.60:0/1947150135 wait complete. 2024-03-27T21:41:19.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.183+0000 7f64fe167640 10 monclient: build_initial_monmap 2024-03-27T21:41:19.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.183+0000 7f64fe167640 10 monclient: monmap: 2024-03-27T21:41:19.185 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:19.186 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:19.186 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:19.186 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:19.186 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:19.186 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:19.186 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:19.186 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:19.186 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:19.186 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:19.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.184+0000 7f64fe167640 1 librados: starting msgr at 2024-03-27T21:41:19.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.184+0000 7f64fe167640 1 librados: starting objecter 2024-03-27T21:41:19.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.184+0000 7f64fe167640 1 Processor -- start 2024-03-27T21:41:19.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.184+0000 7f64fe167640 1 -- start start 2024-03-27T21:41:19.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.184+0000 7f64fe167640 1 librados: setting wanted keys 2024-03-27T21:41:19.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.184+0000 7f64fe167640 1 librados: calling monclient init 2024-03-27T21:41:19.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.184+0000 7f64fe167640 10 monclient: init 2024-03-27T21:41:19.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.184+0000 7f64fe167640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:19.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.184+0000 7f64fe167640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:41:19.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.184+0000 7f64fe167640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64f8071a50 0x7f64f81ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:19.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.184+0000 7f64fe167640 10 monclient(hunting): picked mon.noname-b con 0x7f64f8071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:19.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64fe167640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f813e810 0x7f64f80b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:19.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64fe167640 10 monclient(hunting): picked mon.noname-c con 0x7f64f813e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:19.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64fe167640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f8141630 0x7f64f813f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:19.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64fe167640 10 monclient(hunting): picked mon.noname-a con 0x7f64f8141630 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:19.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64fe167640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:19.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64fe167640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f80b5010 con 0x7f64f8141630 2024-03-27T21:41:19.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64fe167640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:19.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64fe167640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f80b52c0 con 0x7f64f8071a50 2024-03-27T21:41:19.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64fe167640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:19.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64fe167640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f81407c0 con 0x7f64f813e810 2024-03-27T21:41:19.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64fe167640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:19.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64fe167640 10 monclient(hunting): authenticate will time out at 920.406033s 2024-03-27T21:41:19.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f6ffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f813e810 0x7f64f80b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:19.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f6ffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f813e810 0x7f64f80b4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:37906/0 (socket says 172.21.15.60:37906) 2024-03-27T21:41:19.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f6ffd640 1 -- 172.21.15.60:0/1208441076 learned_addr learned my addr 172.21.15.60:0/1208441076 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:19.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f6ffd640 10 monclient(hunting): get_auth_request con 0x7f64f813e810 auth_method 0 2024-03-27T21:41:19.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:19.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:19.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:19.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f77fe640 1 --2- 172.21.15.60:0/1208441076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64f8071a50 0x7f64f81ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:19.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:19.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:19.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:19.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f7fff640 1 --2- 172.21.15.60:0/1208441076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f8141630 0x7f64f813f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:19.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f7fff640 10 monclient(hunting): get_auth_request con 0x7f64f8141630 auth_method 0 2024-03-27T21:41:19.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:19.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:19.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:19.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f77fe640 10 monclient(hunting): get_auth_request con 0x7f64f8071a50 auth_method 0 2024-03-27T21:41:19.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:19.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:19.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:19.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:19.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:19.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:19.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f6ffd640 10 monclient(hunting): handle_auth_done global_id 4760 payload 824 2024-03-27T21:41:19.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f6ffd640 1 -- 172.21.15.60:0/1208441076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64f8071a50 msgr2=0x7f64f81ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:19.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f6ffd640 1 --2- 172.21.15.60:0/1208441076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64f8071a50 0x7f64f81ca4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:19.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f6ffd640 1 -- 172.21.15.60:0/1208441076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f8141630 msgr2=0x7f64f813f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:19.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f6ffd640 1 --2- 172.21.15.60:0/1208441076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f8141630 0x7f64f813f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:19.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f6ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:41:19.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f6ffd640 1 monclient: found mon.noname-c 2024-03-27T21:41:19.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:19.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:19.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f6ffd640 1 -- 172.21.15.60:0/1208441076 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f0009b30 con 0x7f64f813e810 2024-03-27T21:41:19.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.185+0000 7f64f7fff640 1 --2- 172.21.15.60:0/1208441076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f8141630 0x7f64f813f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:19.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.186+0000 7f64f6ffd640 1 --2- 172.21.15.60:0/1208441076 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f813e810 0x7f64f80b4ad0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f64ec009f60 tx=0x7f64ec009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:19.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.186+0000 7f64dffff640 10 client.?.objecter ms_handle_connect 0x7f64f813e810 2024-03-27T21:41:19.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.186+0000 7f64dffff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:19.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.186+0000 7f64dffff640 1 -- 172.21.15.60:0/1208441076 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ec009670 con 0x7f64f813e810 2024-03-27T21:41:19.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.186+0000 7f64dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:19.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.186+0000 7f64dffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:19.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.186+0000 7f64dffff640 10 monclient: dump: 2024-03-27T21:41:19.191 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:19.191 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:19.191 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:19.191 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:19.191 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:19.191 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:19.191 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:19.191 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:19.191 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:19.192 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:19.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.186+0000 7f64dffff640 10 monclient: _finish_auth 0 2024-03-27T21:41:19.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.186+0000 7f64dffff640 10 monclient: _check_auth_tickets 2024-03-27T21:41:19.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.186+0000 7f64dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:19.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.186+0000 7f64fe167640 5 monclient: authenticate success, global_id 4760 2024-03-27T21:41:19.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.186+0000 7f64dffff640 1 -- 172.21.15.60:0/1208441076 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64ec017d20 con 0x7f64f813e810 2024-03-27T21:41:19.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.186+0000 7f64dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:19.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.186+0000 7f64fe167640 10 monclient: _renew_subs 2024-03-27T21:41:19.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.186+0000 7f64dffff640 1 -- 172.21.15.60:0/1208441076 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ec00c970 con 0x7f64f813e810 2024-03-27T21:41:19.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.186+0000 7f64fe167640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:19.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.186+0000 7f64fe167640 1 -- 172.21.15.60:0/1208441076 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f80aaab0 con 0x7f64f813e810 2024-03-27T21:41:19.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.186+0000 7f64f5ffb640 4 set_mon_vals no callback set 2024-03-27T21:41:19.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.187+0000 7f64dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:19.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.187+0000 7f64dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:19.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.187+0000 7f64dffff640 10 monclient: dump: 2024-03-27T21:41:19.193 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:19.193 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:19.193 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:19.193 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:19.193 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:19.193 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:19.193 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:19.193 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:19.193 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:19.193 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:19.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.187+0000 7f64fe167640 10 client.4760.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:19.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.187+0000 7f64fe167640 10 monclient: _renew_subs 2024-03-27T21:41:19.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.187+0000 7f64fe167640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:19.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.187+0000 7f64fe167640 1 -- 172.21.15.60:0/1208441076 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f813f870 con 0x7f64f813e810 2024-03-27T21:41:19.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.187+0000 7f64fe167640 1 librados: init done 2024-03-27T21:41:19.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.188+0000 7f64dffff640 1 -- 172.21.15.60:0/1208441076 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f64ec00cad0 con 0x7f64f813e810 2024-03-27T21:41:19.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.188+0000 7f64fe167640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:19.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.188+0000 7f64fe167640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:19.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.188+0000 7f64fe167640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:19.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.188+0000 7f64fe167640 1 -- 172.21.15.60:0/1208441076 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f8069470 con 0x7f64f813e810 2024-03-27T21:41:19.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.188+0000 7f64dffff640 1 --2- 172.21.15.60:0/1208441076 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f64e406c590 0x7f64e406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:19.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.188+0000 7f64dffff640 1 -- 172.21.15.60:0/1208441076 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f64ec0915e0 con 0x7f64f813e810 2024-03-27T21:41:19.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.188+0000 7f64dffff640 10 client.4760.objecter ms_dispatch 0x7f64f8000b90 osd_map(57..57 src has 1..57) v4 2024-03-27T21:41:19.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.188+0000 7f64dffff640 3 client.4760.objecter handle_osd_map got epochs [57,57] > 0 2024-03-27T21:41:19.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.188+0000 7f64dffff640 3 client.4760.objecter handle_osd_map decoding full epoch 57 2024-03-27T21:41:19.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.188+0000 7f64f77fe640 1 --2- 172.21.15.60:0/1208441076 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f64e406c590 0x7f64e406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:19.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.188+0000 7f64dffff640 20 client.4760.objecter dump_active .. 0 homeless 2024-03-27T21:41:19.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.188+0000 7f64f77fe640 10 monclient: get_auth_request con 0x7f64e406c590 auth_method 0 2024-03-27T21:41:19.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.189+0000 7f64f77fe640 1 --2- 172.21.15.60:0/1208441076 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f64e406c590 0x7f64e406ea50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f64e800a950 tx=0x7f64e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:19.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.189+0000 7f64dffff640 10 client.4760.objecter ms_handle_connect 0x7f64e406c590 2024-03-27T21:41:19.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.193+0000 7f64dffff640 1 -- 172.21.15.60:0/1208441076 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f64ec05ca30 con 0x7f64f813e810 2024-03-27T21:41:19.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.193+0000 7f64dffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:19.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.193+0000 7f64dffff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:19.195 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:19.208+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:19.254 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:19.268+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:19.255 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:19.255 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:19.264 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:41:19.335 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:41:19.355 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:19.368+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:19.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.406+0000 7f64fe167640 10 start_mon_command cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:41:19.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.406+0000 7f64fe167640 10 monclient: _send_command 2 [{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:41:19.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.406+0000 7f64fe167640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:19.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.406+0000 7f64fe167640 1 -- 172.21.15.60:0/1208441076 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f64f80b5400 con 0x7f64f813e810 2024-03-27T21:41:19.409 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:19.409 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:19.418 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:41:19.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.459+0000 7f09201a3640 10 monclient: get_monmap_and_config 2024-03-27T21:41:19.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.459+0000 7f09201a3640 10 monclient: build_initial_monmap 2024-03-27T21:41:19.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.459+0000 7f09201a3640 10 monclient: monmap: 2024-03-27T21:41:19.446 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:19.446 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:19.446 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:19.460560+0000 2024-03-27T21:41:19.446 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:19.460560+0000 2024-03-27T21:41:19.446 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:19.446 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:19.446 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:19.446 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:19.446 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:19.446 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:19.446 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:19.447 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:19.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.460+0000 7f09201a3640 1 Processor -- start 2024-03-27T21:41:19.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.460+0000 7f09201a3640 1 -- start start 2024-03-27T21:41:19.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.460+0000 7f09201a3640 10 monclient: init 2024-03-27T21:41:19.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.460+0000 7f09201a3640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:19.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.460+0000 7f09201a3640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-03-27T21:41:19.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.460+0000 7f09201a3640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0918071a50 0x7f09180a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:19.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.460+0000 7f09201a3640 10 monclient(hunting): picked mon.noname-c con 0x7f0918071a50 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:19.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.460+0000 7f09201a3640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f091813e810 0x7f091813ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:19.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.460+0000 7f09201a3640 10 monclient(hunting): picked mon.noname-a con 0x7f091813e810 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:19.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f09201a3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0918141630 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:19.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f09201a3640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f091813f190 con 0x7f0918141630 2024-03-27T21:41:19.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f09201a3640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:19.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f09201a3640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0918071e30 con 0x7f091813e810 2024-03-27T21:41:19.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f09201a3640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:19.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f09201a3640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09180a8e90 con 0x7f0918071a50 2024-03-27T21:41:19.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f09201a3640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:19.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f09201a3640 10 monclient(hunting): authenticate will time out at 920.682006s 2024-03-27T21:41:19.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091d73e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f091813e810 0x7f091813ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:19.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091d73e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f091813e810 0x7f091813ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37348/0 (socket says 172.21.15.60:37348) 2024-03-27T21:41:19.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091d73e640 1 -- 172.21.15.60:0/1500020774 learned_addr learned my addr 172.21.15.60:0/1500020774 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:19.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091d73e640 10 monclient(hunting): get_auth_request con 0x7f091813e810 auth_method 0 2024-03-27T21:41:19.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091d73e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:19.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091d73e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:19.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091d73e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:19.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091d73e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:19.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091d73e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:19.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091d73e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:19.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091df3f640 1 --2- 172.21.15.60:0/1500020774 >> v2:172.21.15.191:3300/0 conn(0x7f0918071a50 0x7f09180a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:19.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091df3f640 10 monclient(hunting): get_auth_request con 0x7f0918071a50 auth_method 0 2024-03-27T21:41:19.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091df3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:19.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091df3f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:19.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091df3f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:19.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091d73e640 10 monclient(hunting): handle_auth_done global_id 4950 payload 274 2024-03-27T21:41:19.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091d73e640 1 -- 172.21.15.60:0/1500020774 >> v2:172.21.15.191:3300/0 conn(0x7f0918071a50 msgr2=0x7f09180a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:19.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091d73e640 1 --2- 172.21.15.60:0/1500020774 >> v2:172.21.15.191:3300/0 conn(0x7f0918071a50 0x7f09180a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:19.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091d73e640 1 -- 172.21.15.60:0/1500020774 >> v1:172.21.15.60:6789/0 conn(0x7f0918141630 legacy=0x7f0918141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:19.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091d73e640 10 monclient: _finish_hunting 0 2024-03-27T21:41:19.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091d73e640 1 monclient: found mon.noname-a 2024-03-27T21:41:19.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091d73e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:19.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091d73e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:19.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.461+0000 7f091d73e640 1 -- 172.21.15.60:0/1500020774 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09180aa040 con 0x7f091813e810 2024-03-27T21:41:19.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.462+0000 7f091d73e640 1 --2- 172.21.15.60:0/1500020774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f091813e810 0x7f091813ec50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f09180b4a80 tx=0x7f090802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20b25c9d38f3c138 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:19.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f091cf3d640 1 -- 172.21.15.60:0/1500020774 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908004230 con 0x7f091813e810 2024-03-27T21:41:19.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f091cf3d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:19.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f091cf3d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:19.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f091cf3d640 10 monclient: dump: 2024-03-27T21:41:19.464 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:19.464 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:19.465 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:19.465 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:19.465 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:19.465 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:19.465 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:19.465 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:19.465 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:19.465 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:19.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f091cf3d640 10 monclient: _finish_auth 0 2024-03-27T21:41:19.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f091cf3d640 10 monclient: _check_auth_tickets 2024-03-27T21:41:19.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f091cf3d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:19.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f09201a3640 5 monclient: authenticate success, global_id 4950 2024-03-27T21:41:19.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f09201a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:19.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f091cf3d640 1 -- 172.21.15.60:0/1500020774 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090802e050 con 0x7f091813e810 2024-03-27T21:41:19.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f091cf3d640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:19.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f09201a3640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:19.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f09201a3640 4 set_mon_vals no callback set 2024-03-27T21:41:19.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f091cf3d640 1 -- 172.21.15.60:0/1500020774 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908004230 con 0x7f091813e810 2024-03-27T21:41:19.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f091cf3d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:19.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f091cf3d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:19.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f091cf3d640 10 monclient: dump: 2024-03-27T21:41:19.466 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:19.466 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:19.466 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:19.466 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:19.466 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:19.466 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:19.466 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:19.466 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:19.467 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:19.467 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:19.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f09201a3640 10 monclient: shutdown 2024-03-27T21:41:19.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f09201a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:19.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f09201a3640 1 -- 172.21.15.60:0/1500020774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f091813e810 msgr2=0x7f091813ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:19.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f09201a3640 1 --2- 172.21.15.60:0/1500020774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f091813e810 0x7f091813ec50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f09180b4a80 tx=0x7f090802b7e0 comp rx=0 tx=0).stop 2024-03-27T21:41:19.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f09201a3640 1 -- 172.21.15.60:0/1500020774 shutdown_connections 2024-03-27T21:41:19.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f09201a3640 1 --2- 172.21.15.60:0/1500020774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f091813e810 0x7f091813ec50 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:19.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f09201a3640 1 --2- 172.21.15.60:0/1500020774 >> v2:172.21.15.191:3300/0 conn(0x7f0918071a50 0x7f09180a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:19.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f09201a3640 1 -- 172.21.15.60:0/1500020774 >> 172.21.15.60:0/1500020774 conn(0x7f09180b7300 msgr2=0x7f09180b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:19.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f09201a3640 1 -- 172.21.15.60:0/1500020774 shutdown_connections 2024-03-27T21:41:19.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.463+0000 7f09201a3640 1 -- 172.21.15.60:0/1500020774 wait complete. 2024-03-27T21:41:19.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.464+0000 7f09201a3640 10 monclient: build_initial_monmap 2024-03-27T21:41:19.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.464+0000 7f09201a3640 10 monclient: monmap: 2024-03-27T21:41:19.468 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:19.468 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:19.468 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:19.468 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:19.468 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:19.468 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:19.468 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:19.468 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:19.468 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:19.468 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:19.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.464+0000 7f09201a3640 1 librados: starting msgr at 2024-03-27T21:41:19.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.464+0000 7f09201a3640 1 librados: starting objecter 2024-03-27T21:41:19.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.464+0000 7f09201a3640 1 Processor -- start 2024-03-27T21:41:19.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f09201a3640 1 -- start start 2024-03-27T21:41:19.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f09201a3640 1 librados: setting wanted keys 2024-03-27T21:41:19.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f09201a3640 1 librados: calling monclient init 2024-03-27T21:41:19.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f09201a3640 10 monclient: init 2024-03-27T21:41:19.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f09201a3640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:19.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f09201a3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:41:19.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f09201a3640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0918071a50 0x7f09180b5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:19.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f09201a3640 10 monclient(hunting): picked mon.noname-c con 0x7f0918071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:19.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f09201a3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f091813e810 0x7f09181ddc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:19.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f09201a3640 10 monclient(hunting): picked mon.noname-a con 0x7f091813e810 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:19.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f09201a3640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0918141630 0x7f09181390b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:19.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f09201a3640 10 monclient(hunting): picked mon.noname-b con 0x7f0918141630 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:19.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f09201a3640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:19.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f09201a3640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09180b6480 con 0x7f091813e810 2024-03-27T21:41:19.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f09201a3640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:19.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f09201a3640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0918138b90 con 0x7f0918141630 2024-03-27T21:41:19.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f09201a3640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:19.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f09201a3640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09181395f0 con 0x7f0918071a50 2024-03-27T21:41:19.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f09201a3640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:19.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f09201a3640 10 monclient(hunting): authenticate will time out at 920.686480s 2024-03-27T21:41:19.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091d73e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f091813e810 0x7f09181ddc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:19.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091d73e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f091813e810 0x7f09181ddc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37360/0 (socket says 172.21.15.60:37360) 2024-03-27T21:41:19.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091d73e640 1 -- 172.21.15.60:0/2084556267 learned_addr learned my addr 172.21.15.60:0/2084556267 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:19.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091d73e640 10 monclient(hunting): get_auth_request con 0x7f091813e810 auth_method 0 2024-03-27T21:41:19.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091d73e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:19.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091d73e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:19.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091d73e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:19.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091df3f640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0918071a50 0x7f09180b5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:19.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091df3f640 10 monclient(hunting): get_auth_request con 0x7f0918071a50 auth_method 0 2024-03-27T21:41:19.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091df3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:19.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091df3f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:19.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091df3f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:19.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091d73e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:19.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091d73e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:19.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091d73e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:19.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091e740640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0918141630 0x7f09181390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:19.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091e740640 10 monclient(hunting): get_auth_request con 0x7f0918141630 auth_method 0 2024-03-27T21:41:19.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091e740640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:19.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091e740640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:19.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091e740640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:19.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091df3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:19.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091df3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:19.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091df3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:19.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091e740640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:19.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.465+0000 7f091e740640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:19.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f091e740640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:19.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f091d73e640 10 monclient(hunting): handle_auth_done global_id 4956 payload 824 2024-03-27T21:41:19.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f091d73e640 1 -- 172.21.15.60:0/2084556267 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0918071a50 msgr2=0x7f09180b5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:19.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f091d73e640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0918071a50 0x7f09180b5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:19.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f091d73e640 1 -- 172.21.15.60:0/2084556267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0918141630 msgr2=0x7f09181390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f091d73e640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0918141630 0x7f09181390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f091d73e640 10 monclient: _finish_hunting 0 2024-03-27T21:41:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f091d73e640 1 monclient: found mon.noname-a 2024-03-27T21:41:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f091d73e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f091d73e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f091d73e640 1 -- 172.21.15.60:0/2084556267 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09080094d0 con 0x7f091813e810 2024-03-27T21:41:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f091e740640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0918141630 0x7f09181390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f091df3f640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0918071a50 0x7f09180b5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f091d73e640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f091813e810 0x7f09181ddc80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0908000c00 tx=0x7f09080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f090e7fc640 10 client.?.objecter ms_handle_connect 0x7f091813e810 2024-03-27T21:41:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f090e7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f090e7fc640 1 -- 172.21.15.60:0/2084556267 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908004230 con 0x7f091813e810 2024-03-27T21:41:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f090e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f090e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:19.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f090e7fc640 10 monclient: dump: 2024-03-27T21:41:19.475 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:19.475 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:19.475 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:19.475 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:19.475 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:19.475 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:19.475 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:19.475 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:19.475 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:19.475 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:19.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f090e7fc640 10 monclient: _finish_auth 0 2024-03-27T21:41:19.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f090e7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:41:19.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f090e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:19.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f090e7fc640 1 -- 172.21.15.60:0/2084556267 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0908004510 con 0x7f091813e810 2024-03-27T21:41:19.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f090e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:19.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f09201a3640 5 monclient: authenticate success, global_id 4956 2024-03-27T21:41:19.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f090e7fc640 1 -- 172.21.15.60:0/2084556267 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908030960 con 0x7f091813e810 2024-03-27T21:41:19.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f09201a3640 10 monclient: _renew_subs 2024-03-27T21:41:19.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f09201a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:19.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f09201a3640 1 -- 172.21.15.60:0/2084556267 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0918139930 con 0x7f091813e810 2024-03-27T21:41:19.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f090e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:19.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f090e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:19.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f090e7fc640 10 monclient: dump: 2024-03-27T21:41:19.477 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:19.477 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:19.477 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:19.477 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:19.477 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:19.477 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:19.477 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:19.477 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:19.478 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:19.478 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:19.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f090ffff640 4 set_mon_vals no callback set 2024-03-27T21:41:19.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f09201a3640 10 client.4956.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:19.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f09201a3640 10 monclient: _renew_subs 2024-03-27T21:41:19.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f09201a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:19.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f09201a3640 1 -- 172.21.15.60:0/2084556267 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09181388b0 con 0x7f091813e810 2024-03-27T21:41:19.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.466+0000 7f09201a3640 1 librados: init done 2024-03-27T21:41:19.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.467+0000 7f09201a3640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:41:19.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.467+0000 7f09201a3640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:19.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.468+0000 7f09201a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:19.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.468+0000 7f09201a3640 1 -- 172.21.15.60:0/2084556267 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f0918067900 con 0x7f091813e810 2024-03-27T21:41:19.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.468+0000 7f090e7fc640 1 -- 172.21.15.60:0/2084556267 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0908039050 con 0x7f091813e810 2024-03-27T21:41:19.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.469+0000 7f090e7fc640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f08fc06c590 0x7f08fc06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:19.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.469+0000 7f090e7fc640 1 -- 172.21.15.60:0/2084556267 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f09080bbc40 con 0x7f091813e810 2024-03-27T21:41:19.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.469+0000 7f090e7fc640 10 client.4956.objecter ms_dispatch 0x7f0918000b90 osd_map(57..57 src has 1..57) v4 2024-03-27T21:41:19.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.469+0000 7f090e7fc640 3 client.4956.objecter handle_osd_map got epochs [57,57] > 0 2024-03-27T21:41:19.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.469+0000 7f090e7fc640 3 client.4956.objecter handle_osd_map decoding full epoch 57 2024-03-27T21:41:19.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.469+0000 7f090e7fc640 20 client.4956.objecter dump_active .. 0 homeless 2024-03-27T21:41:19.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.469+0000 7f090e7fc640 1 -- 172.21.15.60:0/2084556267 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v57) v1 ==== 54+0+16 (secure 0 0 0) 0x7f09080be280 con 0x7f091813e810 2024-03-27T21:41:19.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.469+0000 7f090e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:19.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.469+0000 7f090e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:19.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.469+0000 7f091df3f640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f08fc06c590 0x7f08fc06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:19.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.469+0000 7f091df3f640 10 monclient: get_auth_request con 0x7f08fc06c590 auth_method 0 2024-03-27T21:41:19.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.469+0000 7f090e7fc640 10 client.4956.objecter ms_handle_connect 0x7f08fc06c590 2024-03-27T21:41:19.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.469+0000 7f091df3f640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f08fc06c590 0x7f08fc06ea50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0914006fd0 tx=0x7f0914008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:19.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.469+0000 7f08fbfff640 10 client.4956.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.469+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign -1 1 2024-03-27T21:41:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.470+0000 7f08fbfff640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f08f4002f70 0x7f08f4005430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.470+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f4002860 osd=0 3 2024-03-27T21:41:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.470+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f4002860 osd=0 3 2024-03-27T21:41:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.470+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f4002860 osd=0 3 2024-03-27T21:41:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.470+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_remove -1 1 2024-03-27T21:41:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.470+0000 7f08fbfff640 20 client.4956.objecter get_session s=0x7f08f4002860 osd=0 3 2024-03-27T21:41:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.470+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign 0 1 2024-03-27T21:41:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.470+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f4002860 osd=0 4 2024-03-27T21:41:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.470+0000 7f08fbfff640 10 client.4956.objecter _send_command 1 2024-03-27T21:41:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.470+0000 7f091e740640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f08f4002f70 0x7f08f4005430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.470+0000 7f091e740640 10 monclient: get_auth_request con 0x7f08f4002f70 auth_method 0 2024-03-27T21:41:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.470+0000 7f08fbfff640 1 -- 172.21.15.60:0/2084556267 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08f4008420 con 0x7f08f4002f70 2024-03-27T21:41:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.470+0000 7f091e740640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f08f4002f70 0x7f08f4005430 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:19.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.470+0000 7f090e7fc640 10 client.4956.objecter ms_handle_connect 0x7f08f4002f70 2024-03-27T21:41:19.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.471+0000 7f090e7fc640 1 -- 172.21.15.60:0/2084556267 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f08f4008420 con 0x7f08f4002f70 2024-03-27T21:41:19.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.471+0000 7f090e7fc640 10 client.4956.objecter ms_dispatch 0x7f0918000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:41:19.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.471+0000 7f090e7fc640 10 client.4956.objecter _finish_command 1 = system:0 2024-03-27T21:41:19.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.471+0000 7f090e7fc640 20 client.4956.objecter put_session s=0x7f08f4002860 osd=0 4 2024-03-27T21:41:19.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.471+0000 7f090e7fc640 15 client.4956.objecter _session_command_op_remove 0 1 2024-03-27T21:41:19.482 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:19.472+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:19.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.500+0000 7f08fbfff640 10 client.4956.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:19.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.500+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign -1 2 2024-03-27T21:41:19.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.500+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f4002860 osd=0 3 2024-03-27T21:41:19.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.500+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f4002860 osd=0 3 2024-03-27T21:41:19.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.500+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f4002860 osd=0 3 2024-03-27T21:41:19.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.500+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_remove -1 2 2024-03-27T21:41:19.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.500+0000 7f08fbfff640 20 client.4956.objecter get_session s=0x7f08f4002860 osd=0 3 2024-03-27T21:41:19.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.500+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign 0 2 2024-03-27T21:41:19.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.500+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f4002860 osd=0 4 2024-03-27T21:41:19.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.500+0000 7f08fbfff640 10 client.4956.objecter _send_command 2 2024-03-27T21:41:19.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.500+0000 7f08fbfff640 1 -- 172.21.15.60:0/2084556267 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f08f4001160 con 0x7f08f4002f70 2024-03-27T21:41:19.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.503+0000 7f090e7fc640 1 -- 172.21.15.60:0/2084556267 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f08f4001160 con 0x7f08f4002f70 2024-03-27T21:41:19.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.503+0000 7f090e7fc640 10 client.4956.objecter ms_dispatch 0x7f0918000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:19.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.503+0000 7f090e7fc640 10 client.4956.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:19.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.503+0000 7f090e7fc640 20 client.4956.objecter put_session s=0x7f08f4002860 osd=0 4 2024-03-27T21:41:19.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.503+0000 7f090e7fc640 15 client.4956.objecter _session_command_op_remove 0 2 2024-03-27T21:41:19.491 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:41:19.493 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:41:19.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.503+0000 7f08fbfff640 10 client.4956.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:19.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.503+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign -1 3 2024-03-27T21:41:19.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.503+0000 7f08fbfff640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f08f4005ab0 0x7f08f4005f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:19.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.503+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f40025f0 osd=1 3 2024-03-27T21:41:19.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.503+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f40025f0 osd=1 3 2024-03-27T21:41:19.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.503+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f40025f0 osd=1 3 2024-03-27T21:41:19.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.503+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_remove -1 3 2024-03-27T21:41:19.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.503+0000 7f08fbfff640 20 client.4956.objecter get_session s=0x7f08f40025f0 osd=1 3 2024-03-27T21:41:19.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.503+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign 1 3 2024-03-27T21:41:19.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.503+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f40025f0 osd=1 4 2024-03-27T21:41:19.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.503+0000 7f08fbfff640 10 client.4956.objecter _send_command 3 2024-03-27T21:41:19.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.503+0000 7f08fbfff640 1 -- 172.21.15.60:0/2084556267 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08f40064a0 con 0x7f08f4005ab0 2024-03-27T21:41:19.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.503+0000 7f091df3f640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f08f4005ab0 0x7f08f4005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:19.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.503+0000 7f091df3f640 10 monclient: get_auth_request con 0x7f08f4005ab0 auth_method 0 2024-03-27T21:41:19.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.504+0000 7f090e7fc640 10 client.4956.objecter ms_handle_connect 0x7f08f4005ab0 2024-03-27T21:41:19.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.504+0000 7f091df3f640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f08f4005ab0 0x7f08f4005f60 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:19.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.505+0000 7f090e7fc640 1 -- 172.21.15.60:0/2084556267 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f08f40064a0 con 0x7f08f4005ab0 2024-03-27T21:41:19.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.505+0000 7f090e7fc640 10 client.4956.objecter ms_dispatch 0x7f0918000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:41:19.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.505+0000 7f090e7fc640 10 client.4956.objecter _finish_command 3 = system:0 2024-03-27T21:41:19.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.505+0000 7f090e7fc640 20 client.4956.objecter put_session s=0x7f08f40025f0 osd=1 4 2024-03-27T21:41:19.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.505+0000 7f090e7fc640 15 client.4956.objecter _session_command_op_remove 1 3 2024-03-27T21:41:19.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.519+0000 7f08fbfff640 10 client.4956.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:19.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.519+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign -1 4 2024-03-27T21:41:19.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.519+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f40025f0 osd=1 3 2024-03-27T21:41:19.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.519+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f40025f0 osd=1 3 2024-03-27T21:41:19.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.519+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f40025f0 osd=1 3 2024-03-27T21:41:19.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.519+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_remove -1 4 2024-03-27T21:41:19.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.519+0000 7f08fbfff640 20 client.4956.objecter get_session s=0x7f08f40025f0 osd=1 3 2024-03-27T21:41:19.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.519+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign 1 4 2024-03-27T21:41:19.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.519+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f40025f0 osd=1 4 2024-03-27T21:41:19.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.519+0000 7f08fbfff640 10 client.4956.objecter _send_command 4 2024-03-27T21:41:19.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.519+0000 7f08fbfff640 1 -- 172.21.15.60:0/2084556267 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f08f4006e60 con 0x7f08f4005ab0 2024-03-27T21:41:19.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.522+0000 7f090e7fc640 1 -- 172.21.15.60:0/2084556267 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f08f4006e60 con 0x7f08f4005ab0 2024-03-27T21:41:19.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.522+0000 7f090e7fc640 10 client.4956.objecter ms_dispatch 0x7f0918000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:19.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.522+0000 7f090e7fc640 10 client.4956.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:19.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.522+0000 7f090e7fc640 20 client.4956.objecter put_session s=0x7f08f40025f0 osd=1 4 2024-03-27T21:41:19.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.522+0000 7f090e7fc640 15 client.4956.objecter _session_command_op_remove 1 4 2024-03-27T21:41:19.509 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:41:19.511 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:41:19.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.522+0000 7f08fbfff640 10 client.4956.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:19.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.522+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign -1 5 2024-03-27T21:41:19.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.522+0000 7f08fbfff640 20 client.4956.objecter _get_session osd=-1 returning homeless 2024-03-27T21:41:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.522+0000 7f08fbfff640 10 client.4956.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.522+0000 7f08fbfff640 10 monclient: _renew_subs 2024-03-27T21:41:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.522+0000 7f08fbfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.522+0000 7f08fbfff640 1 -- 172.21.15.60:0/2084556267 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=58}) v3 -- 0x7f08f40078b0 con 0x7f091813e810 2024-03-27T21:41:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.522+0000 7f08fbfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.522+0000 7f08fbfff640 1 -- 172.21.15.60:0/2084556267 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f08f400cec0 con 0x7f091813e810 2024-03-27T21:41:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.522+0000 7f090e7fc640 1 -- 172.21.15.60:0/2084556267 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09080870a0 con 0x7f091813e810 2024-03-27T21:41:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.522+0000 7f090e7fc640 10 monclient: handle_get_version_reply finishing 1 version 57 2024-03-27T21:41:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.522+0000 7f090ffff640 10 client.4956.objecter _check_command_map_dne tid 5 current 57 map_dne_bound 57 2024-03-27T21:41:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.523+0000 7f090ffff640 10 client.4956.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:41:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.523+0000 7f090ffff640 15 client.4956.objecter _session_command_op_remove -1 5 2024-03-27T21:41:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:41:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.523+0000 7f08fbfff640 10 client.4956.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.523+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign -1 6 2024-03-27T21:41:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.523+0000 7f08fbfff640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f08f4006a50 0x7f08f400b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.523+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f40068e0 osd=3 3 2024-03-27T21:41:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.523+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f40068e0 osd=3 3 2024-03-27T21:41:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.523+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f40068e0 osd=3 3 2024-03-27T21:41:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.523+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_remove -1 6 2024-03-27T21:41:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.523+0000 7f08fbfff640 20 client.4956.objecter get_session s=0x7f08f40068e0 osd=3 3 2024-03-27T21:41:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.523+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign 3 6 2024-03-27T21:41:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.523+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f40068e0 osd=3 4 2024-03-27T21:41:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.523+0000 7f08fbfff640 10 client.4956.objecter _send_command 6 2024-03-27T21:41:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.523+0000 7f08fbfff640 1 -- 172.21.15.60:0/2084556267 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08f40023a0 con 0x7f08f4006a50 2024-03-27T21:41:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.523+0000 7f091d73e640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f08f4006a50 0x7f08f400b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.523+0000 7f091d73e640 10 monclient: get_auth_request con 0x7f08f4006a50 auth_method 0 2024-03-27T21:41:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.523+0000 7f090e7fc640 10 client.4956.objecter ms_handle_connect 0x7f08f4006a50 2024-03-27T21:41:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.523+0000 7f091d73e640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f08f4006a50 0x7f08f400b7b0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:19.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.524+0000 7f090e7fc640 1 -- 172.21.15.60:0/2084556267 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f08f40023a0 con 0x7f08f4006a50 2024-03-27T21:41:19.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.524+0000 7f090e7fc640 10 client.4956.objecter ms_dispatch 0x7f0918000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:41:19.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.524+0000 7f090e7fc640 10 client.4956.objecter _finish_command 6 = system:0 2024-03-27T21:41:19.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.524+0000 7f090e7fc640 20 client.4956.objecter put_session s=0x7f08f40068e0 osd=3 4 2024-03-27T21:41:19.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.524+0000 7f090e7fc640 15 client.4956.objecter _session_command_op_remove 3 6 2024-03-27T21:41:19.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.552+0000 7f08fbfff640 10 client.4956.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:19.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.552+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign -1 7 2024-03-27T21:41:19.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.552+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f40068e0 osd=3 3 2024-03-27T21:41:19.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.552+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f40068e0 osd=3 3 2024-03-27T21:41:19.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.552+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f40068e0 osd=3 3 2024-03-27T21:41:19.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.552+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_remove -1 7 2024-03-27T21:41:19.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.552+0000 7f08fbfff640 20 client.4956.objecter get_session s=0x7f08f40068e0 osd=3 3 2024-03-27T21:41:19.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.552+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign 3 7 2024-03-27T21:41:19.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.552+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f40068e0 osd=3 4 2024-03-27T21:41:19.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.552+0000 7f08fbfff640 10 client.4956.objecter _send_command 7 2024-03-27T21:41:19.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.552+0000 7f08fbfff640 1 -- 172.21.15.60:0/2084556267 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f08f40016d0 con 0x7f08f4006a50 2024-03-27T21:41:19.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.555+0000 7f090e7fc640 1 -- 172.21.15.60:0/2084556267 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f08f40016d0 con 0x7f08f4006a50 2024-03-27T21:41:19.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.555+0000 7f090e7fc640 10 client.4956.objecter ms_dispatch 0x7f0918000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:19.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.555+0000 7f090e7fc640 10 client.4956.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:19.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.555+0000 7f090e7fc640 20 client.4956.objecter put_session s=0x7f08f40068e0 osd=3 4 2024-03-27T21:41:19.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.555+0000 7f090e7fc640 15 client.4956.objecter _session_command_op_remove 3 7 2024-03-27T21:41:19.542 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:41:19.543 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:41:19.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.555+0000 7f08fbfff640 10 client.4956.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:19.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.555+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign -1 8 2024-03-27T21:41:19.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.555+0000 7f08fbfff640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f08f400b300 0x7f08f4011050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:19.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.555+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f4001a80 osd=4 3 2024-03-27T21:41:19.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.555+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f4001a80 osd=4 3 2024-03-27T21:41:19.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.555+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f4001a80 osd=4 3 2024-03-27T21:41:19.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.555+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_remove -1 8 2024-03-27T21:41:19.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.555+0000 7f08fbfff640 20 client.4956.objecter get_session s=0x7f08f4001a80 osd=4 3 2024-03-27T21:41:19.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.555+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign 4 8 2024-03-27T21:41:19.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.555+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f4001a80 osd=4 4 2024-03-27T21:41:19.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.555+0000 7f08fbfff640 10 client.4956.objecter _send_command 8 2024-03-27T21:41:19.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.555+0000 7f08fbfff640 1 -- 172.21.15.60:0/2084556267 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08f4013f80 con 0x7f08f400b300 2024-03-27T21:41:19.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.555+0000 7f091e740640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f08f400b300 0x7f08f4011050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:19.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.555+0000 7f091e740640 10 monclient: get_auth_request con 0x7f08f400b300 auth_method 0 2024-03-27T21:41:19.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.556+0000 7f091e740640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f08f400b300 0x7f08f4011050 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:19.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.556+0000 7f090e7fc640 10 client.4956.objecter ms_handle_connect 0x7f08f400b300 2024-03-27T21:41:19.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.556+0000 7f090e7fc640 1 -- 172.21.15.60:0/2084556267 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f08f4013f80 con 0x7f08f400b300 2024-03-27T21:41:19.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.556+0000 7f090e7fc640 10 client.4956.objecter ms_dispatch 0x7f0918000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:41:19.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.556+0000 7f090e7fc640 10 client.4956.objecter _finish_command 8 = system:0 2024-03-27T21:41:19.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.556+0000 7f090e7fc640 20 client.4956.objecter put_session s=0x7f08f4001a80 osd=4 4 2024-03-27T21:41:19.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.556+0000 7f090e7fc640 15 client.4956.objecter _session_command_op_remove 4 8 2024-03-27T21:41:19.559 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:19.573+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:19.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.585+0000 7f08fbfff640 10 client.4956.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:19.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.585+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign -1 9 2024-03-27T21:41:19.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.585+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f4001a80 osd=4 3 2024-03-27T21:41:19.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.585+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f4001a80 osd=4 3 2024-03-27T21:41:19.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.585+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f4001a80 osd=4 3 2024-03-27T21:41:19.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.585+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_remove -1 9 2024-03-27T21:41:19.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.585+0000 7f08fbfff640 20 client.4956.objecter get_session s=0x7f08f4001a80 osd=4 3 2024-03-27T21:41:19.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.585+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign 4 9 2024-03-27T21:41:19.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.585+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f4001a80 osd=4 4 2024-03-27T21:41:19.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.585+0000 7f08fbfff640 10 client.4956.objecter _send_command 9 2024-03-27T21:41:19.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.585+0000 7f08fbfff640 1 -- 172.21.15.60:0/2084556267 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f08f400a8d0 con 0x7f08f400b300 2024-03-27T21:41:19.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.587+0000 7f090e7fc640 1 -- 172.21.15.60:0/2084556267 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f08f400a8d0 con 0x7f08f400b300 2024-03-27T21:41:19.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.587+0000 7f090e7fc640 10 client.4956.objecter ms_dispatch 0x7f0918000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:19.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.587+0000 7f090e7fc640 10 client.4956.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:19.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.587+0000 7f090e7fc640 20 client.4956.objecter put_session s=0x7f08f4001a80 osd=4 4 2024-03-27T21:41:19.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.587+0000 7f090e7fc640 15 client.4956.objecter _session_command_op_remove 4 9 2024-03-27T21:41:19.573 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:41:19.574 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:41:19.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.587+0000 7f08fbfff640 10 client.4956.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:19.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.587+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign -1 10 2024-03-27T21:41:19.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.587+0000 7f08fbfff640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f08f400c9c0 0x7f08f4011630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:19.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.587+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f400a690 osd=5 3 2024-03-27T21:41:19.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.587+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f400a690 osd=5 3 2024-03-27T21:41:19.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.587+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f400a690 osd=5 3 2024-03-27T21:41:19.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.587+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_remove -1 10 2024-03-27T21:41:19.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.587+0000 7f08fbfff640 20 client.4956.objecter get_session s=0x7f08f400a690 osd=5 3 2024-03-27T21:41:19.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.587+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign 5 10 2024-03-27T21:41:19.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.587+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f400a690 osd=5 4 2024-03-27T21:41:19.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.587+0000 7f08fbfff640 10 client.4956.objecter _send_command 10 2024-03-27T21:41:19.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.587+0000 7f08fbfff640 1 -- 172.21.15.60:0/2084556267 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08f40178a0 con 0x7f08f400c9c0 2024-03-27T21:41:19.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.587+0000 7f091df3f640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f08f400c9c0 0x7f08f4011630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:19.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.587+0000 7f091df3f640 10 monclient: get_auth_request con 0x7f08f400c9c0 auth_method 0 2024-03-27T21:41:19.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.588+0000 7f090e7fc640 10 client.4956.objecter ms_handle_connect 0x7f08f400c9c0 2024-03-27T21:41:19.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.588+0000 7f091df3f640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f08f400c9c0 0x7f08f4011630 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:19.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.589+0000 7f090e7fc640 1 -- 172.21.15.60:0/2084556267 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f08f40178a0 con 0x7f08f400c9c0 2024-03-27T21:41:19.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.589+0000 7f090e7fc640 10 client.4956.objecter ms_dispatch 0x7f0918000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:41:19.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.589+0000 7f090e7fc640 10 client.4956.objecter _finish_command 10 = system:0 2024-03-27T21:41:19.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.589+0000 7f090e7fc640 20 client.4956.objecter put_session s=0x7f08f400a690 osd=5 4 2024-03-27T21:41:19.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.589+0000 7f090e7fc640 15 client.4956.objecter _session_command_op_remove 5 10 2024-03-27T21:41:19.589 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:19.589 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:19.598 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:41:19.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.617+0000 7f08fbfff640 10 client.4956.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:19.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.617+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign -1 11 2024-03-27T21:41:19.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.617+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f400a690 osd=5 3 2024-03-27T21:41:19.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.617+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f400a690 osd=5 3 2024-03-27T21:41:19.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.617+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f400a690 osd=5 3 2024-03-27T21:41:19.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.617+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_remove -1 11 2024-03-27T21:41:19.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.617+0000 7f08fbfff640 20 client.4956.objecter get_session s=0x7f08f400a690 osd=5 3 2024-03-27T21:41:19.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.617+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign 5 11 2024-03-27T21:41:19.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.617+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f400a690 osd=5 4 2024-03-27T21:41:19.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.617+0000 7f08fbfff640 10 client.4956.objecter _send_command 11 2024-03-27T21:41:19.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.617+0000 7f08fbfff640 1 -- 172.21.15.60:0/2084556267 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f08f4011b70 con 0x7f08f400c9c0 2024-03-27T21:41:19.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.619+0000 7f090e7fc640 1 -- 172.21.15.60:0/2084556267 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f08f4011b70 con 0x7f08f400c9c0 2024-03-27T21:41:19.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.619+0000 7f090e7fc640 10 client.4956.objecter ms_dispatch 0x7f0918000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:19.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.619+0000 7f090e7fc640 10 client.4956.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:19.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.619+0000 7f090e7fc640 20 client.4956.objecter put_session s=0x7f08f400a690 osd=5 4 2024-03-27T21:41:19.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.619+0000 7f090e7fc640 15 client.4956.objecter _session_command_op_remove 5 11 2024-03-27T21:41:19.605 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:41:19.607 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:41:19.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.619+0000 7f08fbfff640 10 client.4956.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:19.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.619+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign -1 12 2024-03-27T21:41:19.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.619+0000 7f08fbfff640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f08f40121f0 0x7f08f4012630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:19.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.619+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f4011b70 osd=6 3 2024-03-27T21:41:19.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.619+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f4011b70 osd=6 3 2024-03-27T21:41:19.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.619+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f4011b70 osd=6 3 2024-03-27T21:41:19.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.619+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_remove -1 12 2024-03-27T21:41:19.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.619+0000 7f08fbfff640 20 client.4956.objecter get_session s=0x7f08f4011b70 osd=6 3 2024-03-27T21:41:19.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.619+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign 6 12 2024-03-27T21:41:19.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.619+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f4011b70 osd=6 4 2024-03-27T21:41:19.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.619+0000 7f08fbfff640 10 client.4956.objecter _send_command 12 2024-03-27T21:41:19.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.619+0000 7f08fbfff640 1 -- 172.21.15.60:0/2084556267 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08f401aa70 con 0x7f08f40121f0 2024-03-27T21:41:19.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.619+0000 7f091d73e640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f08f40121f0 0x7f08f4012630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:19.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.619+0000 7f091d73e640 10 monclient: get_auth_request con 0x7f08f40121f0 auth_method 0 2024-03-27T21:41:19.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.620+0000 7f090e7fc640 10 client.4956.objecter ms_handle_connect 0x7f08f40121f0 2024-03-27T21:41:19.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.620+0000 7f091d73e640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f08f40121f0 0x7f08f4012630 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:19.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.620+0000 7f090e7fc640 1 -- 172.21.15.60:0/2084556267 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f08f401aa70 con 0x7f08f40121f0 2024-03-27T21:41:19.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.620+0000 7f090e7fc640 10 client.4956.objecter ms_dispatch 0x7f0918000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:41:19.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.620+0000 7f090e7fc640 10 client.4956.objecter _finish_command 12 = system:0 2024-03-27T21:41:19.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.620+0000 7f090e7fc640 20 client.4956.objecter put_session s=0x7f08f4011b70 osd=6 4 2024-03-27T21:41:19.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.620+0000 7f090e7fc640 15 client.4956.objecter _session_command_op_remove 6 12 2024-03-27T21:41:19.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.649+0000 7f08fbfff640 10 client.4956.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:19.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.649+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign -1 13 2024-03-27T21:41:19.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.649+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f4011b70 osd=6 3 2024-03-27T21:41:19.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.649+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f4011b70 osd=6 3 2024-03-27T21:41:19.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.649+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f4011b70 osd=6 3 2024-03-27T21:41:19.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.649+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_remove -1 13 2024-03-27T21:41:19.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.649+0000 7f08fbfff640 20 client.4956.objecter get_session s=0x7f08f4011b70 osd=6 3 2024-03-27T21:41:19.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.649+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign 6 13 2024-03-27T21:41:19.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.649+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f4011b70 osd=6 4 2024-03-27T21:41:19.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.649+0000 7f08fbfff640 10 client.4956.objecter _send_command 13 2024-03-27T21:41:19.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.649+0000 7f08fbfff640 1 -- 172.21.15.60:0/2084556267 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f08f40181a0 con 0x7f08f40121f0 2024-03-27T21:41:19.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.651+0000 7f090e7fc640 1 -- 172.21.15.60:0/2084556267 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f08f40181a0 con 0x7f08f40121f0 2024-03-27T21:41:19.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.651+0000 7f090e7fc640 10 client.4956.objecter ms_dispatch 0x7f0918000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:19.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.651+0000 7f090e7fc640 10 client.4956.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:19.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.651+0000 7f090e7fc640 20 client.4956.objecter put_session s=0x7f08f4011b70 osd=6 4 2024-03-27T21:41:19.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.651+0000 7f090e7fc640 15 client.4956.objecter _session_command_op_remove 6 13 2024-03-27T21:41:19.638 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:41:19.639 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:41:19.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.651+0000 7f08fbfff640 10 client.4956.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:19.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.651+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign -1 14 2024-03-27T21:41:19.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.652+0000 7f08fbfff640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f08f4018870 0x7f08f4018cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:19.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.652+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f40181a0 osd=7 3 2024-03-27T21:41:19.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.652+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f40181a0 osd=7 3 2024-03-27T21:41:19.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.652+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f40181a0 osd=7 3 2024-03-27T21:41:19.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.652+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_remove -1 14 2024-03-27T21:41:19.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.652+0000 7f08fbfff640 20 client.4956.objecter get_session s=0x7f08f40181a0 osd=7 3 2024-03-27T21:41:19.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.652+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign 7 14 2024-03-27T21:41:19.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.652+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f40181a0 osd=7 4 2024-03-27T21:41:19.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.652+0000 7f08fbfff640 10 client.4956.objecter _send_command 14 2024-03-27T21:41:19.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.652+0000 7f08fbfff640 1 -- 172.21.15.60:0/2084556267 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08f401dda0 con 0x7f08f4018870 2024-03-27T21:41:19.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.652+0000 7f091e740640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f08f4018870 0x7f08f4018cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:19.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.652+0000 7f091e740640 10 monclient: get_auth_request con 0x7f08f4018870 auth_method 0 2024-03-27T21:41:19.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.652+0000 7f090e7fc640 10 client.4956.objecter ms_handle_connect 0x7f08f4018870 2024-03-27T21:41:19.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.652+0000 7f091e740640 1 --2- 172.21.15.60:0/2084556267 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f08f4018870 0x7f08f4018cd0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:19.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.653+0000 7f090e7fc640 1 -- 172.21.15.60:0/2084556267 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f08f401dda0 con 0x7f08f4018870 2024-03-27T21:41:19.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.653+0000 7f090e7fc640 10 client.4956.objecter ms_dispatch 0x7f0918000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:41:19.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.653+0000 7f090e7fc640 10 client.4956.objecter _finish_command 14 = system:0 2024-03-27T21:41:19.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.653+0000 7f090e7fc640 20 client.4956.objecter put_session s=0x7f08f40181a0 osd=7 4 2024-03-27T21:41:19.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.653+0000 7f090e7fc640 15 client.4956.objecter _session_command_op_remove 7 14 2024-03-27T21:41:19.661 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:19.675+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:19.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.681+0000 7f08fbfff640 10 client.4956.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:19.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.681+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign -1 15 2024-03-27T21:41:19.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.681+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f40181a0 osd=7 3 2024-03-27T21:41:19.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.681+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f40181a0 osd=7 3 2024-03-27T21:41:19.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.681+0000 7f08fbfff640 20 client.4956.objecter _get_session s=0x7f08f40181a0 osd=7 3 2024-03-27T21:41:19.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.681+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_remove -1 15 2024-03-27T21:41:19.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.681+0000 7f08fbfff640 20 client.4956.objecter get_session s=0x7f08f40181a0 osd=7 3 2024-03-27T21:41:19.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.681+0000 7f08fbfff640 15 client.4956.objecter _session_command_op_assign 7 15 2024-03-27T21:41:19.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.681+0000 7f08fbfff640 20 client.4956.objecter put_session s=0x7f08f40181a0 osd=7 4 2024-03-27T21:41:19.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.681+0000 7f08fbfff640 10 client.4956.objecter _send_command 15 2024-03-27T21:41:19.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.681+0000 7f08fbfff640 1 -- 172.21.15.60:0/2084556267 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f08f4019320 con 0x7f08f4018870 2024-03-27T21:41:19.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.683+0000 7f090e7fc640 1 -- 172.21.15.60:0/2084556267 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f08f4019320 con 0x7f08f4018870 2024-03-27T21:41:19.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.683+0000 7f090e7fc640 10 client.4956.objecter ms_dispatch 0x7f0918000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:19.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.683+0000 7f090e7fc640 10 client.4956.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:19.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.683+0000 7f090e7fc640 20 client.4956.objecter put_session s=0x7f08f40181a0 osd=7 4 2024-03-27T21:41:19.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:19.683+0000 7f090e7fc640 15 client.4956.objecter _session_command_op_remove 7 15 2024-03-27T21:41:19.669 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:41:19.670 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:41:19.673 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:41:19.674 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:41:19.704 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:19.705 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:19.705 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:19.705 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:19.705 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:19.713 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:41:19.797 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:19.811+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:19.860 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:19.860 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:19.860 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:19.860 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-03-27T21:41:19.860 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.860 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:19.861 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:19.861 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 100.440949687, 2024-03-27T21:41:19.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43537978700000002, 2024-03-27T21:41:19.861 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:19.861 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:19.861 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:19.861 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:19.861 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:19.861 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-03-27T21:41:19.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.861 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:19.861 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.861 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:19.861 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:19.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:19.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:19.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:19.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432019+0000", 2024-03-27T21:41:19.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-03-27T21:41:19.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:19.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432027+0000", 2024-03-27T21:41:19.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1200000000000002e-06 2024-03-27T21:41:19.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.863 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:19.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432039+0000", 2024-03-27T21:41:19.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1123e-05 2024-03-27T21:41:19.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.863 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:19.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432051+0000", 2024-03-27T21:41:19.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-03-27T21:41:19.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.864 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:19.864 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432157+0000", 2024-03-27T21:41:19.864 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106846 2024-03-27T21:41:19.864 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.864 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.864 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-03-27T21:41:19.864 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432176+0000", 2024-03-27T21:41:19.864 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8490000000000001e-05 2024-03-27T21:41:19.864 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.864 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.864 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:19.864 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866170+0000", 2024-03-27T21:41:19.864 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43399370100000001 2024-03-27T21:41:19.864 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:19.865 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866335+0000", 2024-03-27T21:41:19.865 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016559699999999999 2024-03-27T21:41:19.865 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:19.865 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866495+0000", 2024-03-27T21:41:19.865 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015944200000000001 2024-03-27T21:41:19.865 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:19.865 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867012+0000", 2024-03-27T21:41:19.865 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0005176 2024-03-27T21:41:19.865 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.866 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.866 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:19.866 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867334+0000", 2024-03-27T21:41:19.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032158600000000001 2024-03-27T21:41:19.866 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.866 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.866 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:19.866 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867374+0000", 2024-03-27T21:41:19.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0547000000000003e-05 2024-03-27T21:41:19.866 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.866 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.866 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:19.866 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867400+0000", 2024-03-27T21:41:19.867 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5958000000000001e-05 2024-03-27T21:41:19.867 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.867 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:19.867 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.867 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.867 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.867 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:19.867 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:19.867 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 99.943567236000007, 2024-03-27T21:41:19.867 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064560310000000001, 2024-03-27T21:41:19.867 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:19.867 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:19.867 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:19.867 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:19.867 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:19.868 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 70 2024-03-27T21:41:19.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.868 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:19.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:19.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:19.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:19.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:19.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:19.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929402+0000", 2024-03-27T21:41:19.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:19.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.869 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.869 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:19.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929407+0000", 2024-03-27T21:41:19.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-06 2024-03-27T21:41:19.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.869 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.869 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:19.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929414+0000", 2024-03-27T21:41:19.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5689999999999999e-06 2024-03-27T21:41:19.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.869 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:19.870 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929419+0000", 2024-03-27T21:41:19.870 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-03-27T21:41:19.870 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:19.870 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930245+0000", 2024-03-27T21:41:19.870 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082521500000000002 2024-03-27T21:41:19.870 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:19.870 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930287+0000", 2024-03-27T21:41:19.870 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2305000000000002e-05 2024-03-27T21:41:19.870 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:19.871 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930367+0000", 2024-03-27T21:41:19.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9589e-05 2024-03-27T21:41:19.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:19.871 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.931214+0000", 2024-03-27T21:41:19.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000847827 2024-03-27T21:41:19.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:19.871 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935807+0000", 2024-03-27T21:41:19.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045923780000000003 2024-03-27T21:41:19.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.872 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:19.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935840+0000", 2024-03-27T21:41:19.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3501999999999999e-05 2024-03-27T21:41:19.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.872 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:19.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935859+0000", 2024-03-27T21:41:19.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8845e-05 2024-03-27T21:41:19.872 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.872 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:19.872 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.873 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:19.873 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:19.873 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 99.936307644999999, 2024-03-27T21:41:19.873 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071695429999999996, 2024-03-27T21:41:19.873 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:19.873 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:19.873 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:19.873 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:19.873 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:19.873 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-03-27T21:41:19.873 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.873 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:19.873 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.873 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:19.873 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:19.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.874 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:19.874 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:19.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.874 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:19.874 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936662+0000", 2024-03-27T21:41:19.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:19.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.874 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:19.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936737+0000", 2024-03-27T21:41:19.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5303999999999996e-05 2024-03-27T21:41:19.875 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:19.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936746+0000", 2024-03-27T21:41:19.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6340000000000007e-06 2024-03-27T21:41:19.875 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:19.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936753+0000", 2024-03-27T21:41:19.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-03-27T21:41:19.875 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:19.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937926+0000", 2024-03-27T21:41:19.876 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011731280000000001 2024-03-27T21:41:19.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.876 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:19.876 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937969+0000", 2024-03-27T21:41:19.876 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3285999999999999e-05 2024-03-27T21:41:19.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.876 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:19.876 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938040+0000", 2024-03-27T21:41:19.876 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0358000000000006e-05 2024-03-27T21:41:19.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.877 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:19.877 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939324+0000", 2024-03-27T21:41:19.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012844899999999999 2024-03-27T21:41:19.877 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.877 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.877 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:19.877 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943780+0000", 2024-03-27T21:41:19.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044555630000000001 2024-03-27T21:41:19.877 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.877 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.877 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:19.877 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943811+0000", 2024-03-27T21:41:19.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1415999999999999e-05 2024-03-27T21:41:19.877 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.877 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.878 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:19.878 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943832+0000", 2024-03-27T21:41:19.878 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0981000000000001e-05 2024-03-27T21:41:19.878 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.878 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:19.878 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.878 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.878 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.878 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:19.878 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:19.878 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 99.936209629000004, 2024-03-27T21:41:19.878 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071139740000000003, 2024-03-27T21:41:19.878 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:19.878 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:19.879 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:19.879 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:19.879 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:19.879 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-03-27T21:41:19.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.879 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:19.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:19.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:19.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:19.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:19.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:19.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936760+0000", 2024-03-27T21:41:19.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:19.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:19.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936765+0000", 2024-03-27T21:41:19.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6330000000000004e-06 2024-03-27T21:41:19.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:19.881 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936771+0000", 2024-03-27T21:41:19.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0560000000000001e-06 2024-03-27T21:41:19.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:19.881 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936776+0000", 2024-03-27T21:41:19.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1510000000000003e-06 2024-03-27T21:41:19.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.881 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:19.881 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938605+0000", 2024-03-27T21:41:19.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001829589 2024-03-27T21:41:19.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:19.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938657+0000", 2024-03-27T21:41:19.882 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1860000000000002e-05 2024-03-27T21:41:19.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:19.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938734+0000", 2024-03-27T21:41:19.882 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6639999999999998e-05 2024-03-27T21:41:19.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:19.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939562+0000", 2024-03-27T21:41:19.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082803100000000001 2024-03-27T21:41:19.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:19.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943838+0000", 2024-03-27T21:41:19.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042757050000000003 2024-03-27T21:41:19.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:19.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943859+0000", 2024-03-27T21:41:19.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1175e-05 2024-03-27T21:41:19.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.884 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:19.884 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943875+0000", 2024-03-27T21:41:19.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5744e-05 2024-03-27T21:41:19.884 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.884 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:19.884 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.884 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.884 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.884 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:19.885 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:19.885 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 99.792695578000007, 2024-03-27T21:41:19.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059712230000000003, 2024-03-27T21:41:19.885 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:19.885 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:19.885 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:19.885 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:19.885 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:19.885 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 112 2024-03-27T21:41:19.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.885 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:19.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:19.886 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:19.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.886 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:19.886 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:19.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.886 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:19.886 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080274+0000", 2024-03-27T21:41:19.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:19.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.886 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:19.887 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080318+0000", 2024-03-27T21:41:19.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3634999999999998e-05 2024-03-27T21:41:19.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.887 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.887 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:19.887 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080329+0000", 2024-03-27T21:41:19.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1232e-05 2024-03-27T21:41:19.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.887 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.887 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:19.887 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080333+0000", 2024-03-27T21:41:19.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6009999999999999e-06 2024-03-27T21:41:19.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.887 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:19.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083786+0000", 2024-03-27T21:41:19.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034528940000000002 2024-03-27T21:41:19.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:19.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083861+0000", 2024-03-27T21:41:19.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4893000000000003e-05 2024-03-27T21:41:19.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:19.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083981+0000", 2024-03-27T21:41:19.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012058299999999999 2024-03-27T21:41:19.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:19.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086148+0000", 2024-03-27T21:41:19.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021671469999999999 2024-03-27T21:41:19.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:19.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086195+0000", 2024-03-27T21:41:19.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6768999999999999e-05 2024-03-27T21:41:19.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:19.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086224+0000", 2024-03-27T21:41:19.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8758000000000001e-05 2024-03-27T21:41:19.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:19.890 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086246+0000", 2024-03-27T21:41:19.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2064000000000001e-05 2024-03-27T21:41:19.890 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.890 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:19.890 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.890 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:19.890 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:19.890 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 99.792632631000004, 2024-03-27T21:41:19.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066196320000000003, 2024-03-27T21:41:19.891 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:19.891 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:19.891 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:19.891 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:19.891 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:19.891 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113 2024-03-27T21:41:19.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.891 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:19.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:19.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:19.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:19.892 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:19.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.892 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:19.892 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080337+0000", 2024-03-27T21:41:19.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:19.892 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:19.892 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080340+0000", 2024-03-27T21:41:19.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6690000000000002e-06 2024-03-27T21:41:19.892 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:19.893 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080344+0000", 2024-03-27T21:41:19.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6550000000000002e-06 2024-03-27T21:41:19.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:19.893 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080348+0000", 2024-03-27T21:41:19.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9169999999999999e-06 2024-03-27T21:41:19.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:19.893 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084595+0000", 2024-03-27T21:41:19.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042471719999999996 2024-03-27T21:41:19.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:19.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084676+0000", 2024-03-27T21:41:19.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1147000000000001e-05 2024-03-27T21:41:19.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:19.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084796+0000", 2024-03-27T21:41:19.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012032900000000001 2024-03-27T21:41:19.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:19.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086190+0000", 2024-03-27T21:41:19.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013934399999999999 2024-03-27T21:41:19.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:19.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086905+0000", 2024-03-27T21:41:19.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071536199999999999 2024-03-27T21:41:19.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:19.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086937+0000", 2024-03-27T21:41:19.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2125000000000002e-05 2024-03-27T21:41:19.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:19.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086957+0000", 2024-03-27T21:41:19.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0157999999999999e-05 2024-03-27T21:41:19.896 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.896 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:19.896 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.896 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.896 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:19.896 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:19.896 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 99.792619915000003, 2024-03-27T21:41:19.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007405661, 2024-03-27T21:41:19.896 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:19.896 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:19.896 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:19.896 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:19.896 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:19.897 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 114 2024-03-27T21:41:19.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.897 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:19.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:19.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:19.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:19.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:19.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.898 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:19.898 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:19.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-03-27T21:41:19.898 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.898 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.898 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:19.898 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080353+0000", 2024-03-27T21:41:19.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.441e-06 2024-03-27T21:41:19.898 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.898 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.898 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:19.898 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080356+0000", 2024-03-27T21:41:19.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0769999999999999e-06 2024-03-27T21:41:19.898 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:19.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080358+0000", 2024-03-27T21:41:19.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3369999999999998e-06 2024-03-27T21:41:19.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:19.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085118+0000", 2024-03-27T21:41:19.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047603949999999997 2024-03-27T21:41:19.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:19.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085210+0000", 2024-03-27T21:41:19.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1983000000000003e-05 2024-03-27T21:41:19.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:19.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085332+0000", 2024-03-27T21:41:19.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121803 2024-03-27T21:41:19.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:19.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086292+0000", 2024-03-27T21:41:19.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095961400000000002 2024-03-27T21:41:19.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:19.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087710+0000", 2024-03-27T21:41:19.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014181459999999999 2024-03-27T21:41:19.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:19.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087736+0000", 2024-03-27T21:41:19.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6475e-05 2024-03-27T21:41:19.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:19.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087756+0000", 2024-03-27T21:41:19.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9626000000000001e-05 2024-03-27T21:41:19.901 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.901 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:19.901 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.902 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:19.902 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:19.902 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 99.792597479999998, 2024-03-27T21:41:19.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075011720000000004, 2024-03-27T21:41:19.902 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:19.902 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:19.902 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:19.902 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:19.902 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:19.902 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115 2024-03-27T21:41:19.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.903 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:19.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.903 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:19.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:19.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.903 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:19.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:19.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.903 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:19.903 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:19.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:19.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:19.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080458+0000", 2024-03-27T21:41:19.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5228000000000005e-05 2024-03-27T21:41:19.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:19.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080465+0000", 2024-03-27T21:41:19.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.238e-06 2024-03-27T21:41:19.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:19.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080468+0000", 2024-03-27T21:41:19.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5109999999999999e-06 2024-03-27T21:41:19.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.905 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:19.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086326+0000", 2024-03-27T21:41:19.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058579169999999998 2024-03-27T21:41:19.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.905 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:19.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086382+0000", 2024-03-27T21:41:19.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5309999999999997e-05 2024-03-27T21:41:19.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.905 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:19.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086457+0000", 2024-03-27T21:41:19.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5413000000000005e-05 2024-03-27T21:41:19.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:19.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087762+0000", 2024-03-27T21:41:19.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013052300000000001 2024-03-27T21:41:19.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:19.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087798+0000", 2024-03-27T21:41:19.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5614999999999997e-05 2024-03-27T21:41:19.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:19.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087840+0000", 2024-03-27T21:41:19.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2196000000000001e-05 2024-03-27T21:41:19.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:19.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087874+0000", 2024-03-27T21:41:19.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3785999999999998e-05 2024-03-27T21:41:19.907 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.907 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:19.907 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.907 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:19.907 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:19.908 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 99.792497397000005, 2024-03-27T21:41:19.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075050680000000002, 2024-03-27T21:41:19.908 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:19.908 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:19.908 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:19.908 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:19.908 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:19.908 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 116 2024-03-27T21:41:19.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.908 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:19.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:19.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:19.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:19.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:19.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:19.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080472+0000", 2024-03-27T21:41:19.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:19.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:19.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080475+0000", 2024-03-27T21:41:19.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.712e-06 2024-03-27T21:41:19.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:19.910 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080478+0000", 2024-03-27T21:41:19.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1070000000000001e-06 2024-03-27T21:41:19.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:19.910 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080481+0000", 2024-03-27T21:41:19.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4269999999999998e-06 2024-03-27T21:41:19.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:19.910 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086992+0000", 2024-03-27T21:41:19.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065114099999999996 2024-03-27T21:41:19.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:19.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087064+0000", 2024-03-27T21:41:19.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1927000000000004e-05 2024-03-27T21:41:19.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:19.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087188+0000", 2024-03-27T21:41:19.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123585 2024-03-27T21:41:19.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:19.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087766+0000", 2024-03-27T21:41:19.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057814300000000004 2024-03-27T21:41:19.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:19.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087934+0000", 2024-03-27T21:41:19.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000167894 2024-03-27T21:41:19.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:19.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087958+0000", 2024-03-27T21:41:19.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4663000000000001e-05 2024-03-27T21:41:19.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:19.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087978+0000", 2024-03-27T21:41:19.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9636e-05 2024-03-27T21:41:19.913 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.913 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:19.913 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.913 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:19.913 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:19.913 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 99.792465274999998, 2024-03-27T21:41:19.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076850679999999998, 2024-03-27T21:41:19.913 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:19.913 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:19.913 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:19.913 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:19.914 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:19.914 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 117 2024-03-27T21:41:19.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.914 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:19.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.914 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:19.914 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:19.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.914 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:19.914 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:19.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:19.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:19.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:19.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:19.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080508+0000", 2024-03-27T21:41:19.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.298e-06 2024-03-27T21:41:19.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:19.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080515+0000", 2024-03-27T21:41:19.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0790000000000003e-06 2024-03-27T21:41:19.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:19.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080518+0000", 2024-03-27T21:41:19.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8509999999999999e-06 2024-03-27T21:41:19.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:19.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087444+0000", 2024-03-27T21:41:19.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0069259120000000002 2024-03-27T21:41:19.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.916 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:19.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087491+0000", 2024-03-27T21:41:19.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7043999999999999e-05 2024-03-27T21:41:19.917 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.917 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:19.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087564+0000", 2024-03-27T21:41:19.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3603999999999995e-05 2024-03-27T21:41:19.917 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.917 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:19.917 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087902+0000", 2024-03-27T21:41:19.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033734100000000001 2024-03-27T21:41:19.917 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.918 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:19.918 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088117+0000", 2024-03-27T21:41:19.918 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000215164 2024-03-27T21:41:19.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.918 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:19.918 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088160+0000", 2024-03-27T21:41:19.918 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2933999999999999e-05 2024-03-27T21:41:19.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.918 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.918 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:19.918 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088190+0000", 2024-03-27T21:41:19.918 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0139999999999999e-05 2024-03-27T21:41:19.918 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.919 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:19.919 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.919 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:19.919 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:19.919 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 99.376184886000004, 2024-03-27T21:41:19.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052465439999999997, 2024-03-27T21:41:19.919 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:19.919 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:19.919 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:19.919 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:19.919 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:19.920 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 133 2024-03-27T21:41:19.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.920 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:19.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:19.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:19.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:19.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:19.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:19.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:19.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:19.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:19.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496788+0000", 2024-03-27T21:41:19.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-03-27T21:41:19.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:19.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496793+0000", 2024-03-27T21:41:19.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4790000000000004e-06 2024-03-27T21:41:19.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.922 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:19.922 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496796+0000", 2024-03-27T21:41:19.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1e-06 2024-03-27T21:41:19.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.922 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:19.922 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500447+0000", 2024-03-27T21:41:19.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036504459999999999 2024-03-27T21:41:19.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.922 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:19.923 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500492+0000", 2024-03-27T21:41:19.923 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5765e-05 2024-03-27T21:41:19.923 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.923 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:19.923 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500562+0000", 2024-03-27T21:41:19.923 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9206000000000001e-05 2024-03-27T21:41:19.923 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.923 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:19.923 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501959+0000", 2024-03-27T21:41:19.923 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001397187 2024-03-27T21:41:19.923 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.924 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:19.924 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501984+0000", 2024-03-27T21:41:19.924 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.552e-05 2024-03-27T21:41:19.924 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.924 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.924 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:19.924 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502010+0000", 2024-03-27T21:41:19.924 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5232000000000002e-05 2024-03-27T21:41:19.924 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.924 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.924 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:19.924 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502032+0000", 2024-03-27T21:41:19.924 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2337999999999999e-05 2024-03-27T21:41:19.925 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.925 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:19.925 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.925 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.925 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.925 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:19.925 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:19.925 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 99.376126228000004, 2024-03-27T21:41:19.925 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052892119999999997, 2024-03-27T21:41:19.925 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:19.925 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:19.925 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:19.925 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:19.926 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:19.926 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 134 2024-03-27T21:41:19.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.926 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:19.926 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.926 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:19.926 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:19.926 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.926 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.926 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:19.926 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:19.926 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.926 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:19.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:19.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:19.927 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:19.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496929+0000", 2024-03-27T21:41:19.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5098000000000004e-05 2024-03-27T21:41:19.927 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:19.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496937+0000", 2024-03-27T21:41:19.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9559999999999997e-06 2024-03-27T21:41:19.927 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.928 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.928 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:19.928 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496942+0000", 2024-03-27T21:41:19.928 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6350000000000003e-06 2024-03-27T21:41:19.928 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.928 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.928 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:19.928 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500749+0000", 2024-03-27T21:41:19.928 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038068529999999998 2024-03-27T21:41:19.928 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.928 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.928 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:19.929 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500792+0000", 2024-03-27T21:41:19.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2823e-05 2024-03-27T21:41:19.929 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.929 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.929 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:19.929 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500864+0000", 2024-03-27T21:41:19.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1715e-05 2024-03-27T21:41:19.929 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.929 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.929 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:19.929 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502068+0000", 2024-03-27T21:41:19.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012044860000000001 2024-03-27T21:41:19.929 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.929 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.929 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:19.930 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502073+0000", 2024-03-27T21:41:19.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5700000000000003e-06 2024-03-27T21:41:19.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:19.930 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502098+0000", 2024-03-27T21:41:19.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5120999999999999e-05 2024-03-27T21:41:19.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:19.930 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502133+0000", 2024-03-27T21:41:19.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5306999999999999e-05 2024-03-27T21:41:19.930 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.931 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:19.931 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.931 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:19.931 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:19.931 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 99.376020921000006, 2024-03-27T21:41:19.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062529040000000001, 2024-03-27T21:41:19.931 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:19.931 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:19.931 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:19.931 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:19.931 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:19.931 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135 2024-03-27T21:41:19.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.932 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:19.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:19.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:19.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:19.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:19.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:19.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:19.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:19.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:19.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496953+0000", 2024-03-27T21:41:19.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0709999999999996e-06 2024-03-27T21:41:19.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:19.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496961+0000", 2024-03-27T21:41:19.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1300000000000001e-06 2024-03-27T21:41:19.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:19.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496968+0000", 2024-03-27T21:41:19.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0999999999999998e-06 2024-03-27T21:41:19.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.934 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:19.934 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501244+0000", 2024-03-27T21:41:19.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042756310000000002 2024-03-27T21:41:19.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.934 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:19.934 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501377+0000", 2024-03-27T21:41:19.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013348799999999999 2024-03-27T21:41:19.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.934 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:19.934 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501455+0000", 2024-03-27T21:41:19.935 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7410000000000006e-05 2024-03-27T21:41:19.935 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.935 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.935 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:19.935 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502138+0000", 2024-03-27T21:41:19.935 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068386599999999999 2024-03-27T21:41:19.935 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.935 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.935 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:19.935 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503152+0000", 2024-03-27T21:41:19.935 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010139680000000001 2024-03-27T21:41:19.935 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.935 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.935 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:19.936 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503180+0000", 2024-03-27T21:41:19.936 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7426000000000001e-05 2024-03-27T21:41:19.936 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.936 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.936 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:19.936 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503202+0000", 2024-03-27T21:41:19.936 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2463000000000002e-05 2024-03-27T21:41:19.936 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.936 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:19.936 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.936 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.936 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.936 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:19.936 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:19.937 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 99.375996783000005, 2024-03-27T21:41:19.937 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066253680000000004, 2024-03-27T21:41:19.937 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:19.937 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:19.937 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:19.937 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:19.937 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:19.937 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 136 2024-03-27T21:41:19.937 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.937 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:19.937 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.937 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:19.937 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:19.937 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.938 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:19.938 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:19.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.938 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:19.938 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:19.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:19.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.938 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:19.938 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496977+0000", 2024-03-27T21:41:19.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5049999999999999e-06 2024-03-27T21:41:19.939 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.939 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.939 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:19.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496984+0000", 2024-03-27T21:41:19.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1520000000000003e-06 2024-03-27T21:41:19.939 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.939 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.939 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:19.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496990+0000", 2024-03-27T21:41:19.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-03-27T21:41:19.939 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.939 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.939 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:19.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501669+0000", 2024-03-27T21:41:19.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046793859999999998 2024-03-27T21:41:19.940 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.940 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.940 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:19.940 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501714+0000", 2024-03-27T21:41:19.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5154999999999997e-05 2024-03-27T21:41:19.940 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.940 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.940 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:19.940 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501783+0000", 2024-03-27T21:41:19.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8454999999999994e-05 2024-03-27T21:41:19.940 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.940 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.940 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:19.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503226+0000", 2024-03-27T21:41:19.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001443441 2024-03-27T21:41:19.941 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.941 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.941 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:19.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503540+0000", 2024-03-27T21:41:19.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031347100000000001 2024-03-27T21:41:19.941 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.941 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.941 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:19.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503563+0000", 2024-03-27T21:41:19.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3223000000000001e-05 2024-03-27T21:41:19.941 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.942 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.942 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:19.942 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503599+0000", 2024-03-27T21:41:19.942 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6105000000000001e-05 2024-03-27T21:41:19.942 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.942 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:19.942 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.942 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.942 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.942 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:19.942 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:19.942 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 99.375973973000001, 2024-03-27T21:41:19.942 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074261320000000002, 2024-03-27T21:41:19.942 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:19.943 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:19.943 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:19.943 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:19.943 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:19.943 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 137 2024-03-27T21:41:19.943 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.943 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:19.943 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.943 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:19.943 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:19.943 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.943 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.943 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.943 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:19.943 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:19.943 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.944 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.944 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.944 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:19.944 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:19.944 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:19.944 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.944 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.944 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:19.944 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497077+0000", 2024-03-27T21:41:19.944 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1599000000000003e-05 2024-03-27T21:41:19.944 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.944 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.944 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:19.944 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497085+0000", 2024-03-27T21:41:19.944 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6109999999999996e-06 2024-03-27T21:41:19.944 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.945 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:19.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497092+0000", 2024-03-27T21:41:19.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.742e-06 2024-03-27T21:41:19.945 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.945 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:19.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502159+0000", 2024-03-27T21:41:19.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050666870000000003 2024-03-27T21:41:19.945 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.945 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:19.946 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502200+0000", 2024-03-27T21:41:19.946 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1919999999999998e-05 2024-03-27T21:41:19.946 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.946 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.946 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:19.946 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502280+0000", 2024-03-27T21:41:19.946 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9417000000000002e-05 2024-03-27T21:41:19.946 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.946 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.946 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:19.946 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503605+0000", 2024-03-27T21:41:19.946 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001325641 2024-03-27T21:41:19.946 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.946 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.947 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:19.947 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504361+0000", 2024-03-27T21:41:19.947 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075555399999999997 2024-03-27T21:41:19.947 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.947 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.947 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:19.947 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504386+0000", 2024-03-27T21:41:19.947 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5369999999999999e-05 2024-03-27T21:41:19.947 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.947 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.947 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:19.947 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504422+0000", 2024-03-27T21:41:19.947 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6019000000000002e-05 2024-03-27T21:41:19.947 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.947 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:19.948 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.948 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.948 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.948 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:19.948 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:19.948 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 99.375870176000006, 2024-03-27T21:41:19.948 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074059750000000004, 2024-03-27T21:41:19.948 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:19.948 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:19.948 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:19.948 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:19.948 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:19.948 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-03-27T21:41:19.948 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.949 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:19.949 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.949 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:19.949 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:19.949 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.949 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.949 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.949 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:19.949 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:19.949 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.949 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.949 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.949 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:19.950 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:19.950 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:19.950 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.950 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.950 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:19.950 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497103+0000", 2024-03-27T21:41:19.950 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.856e-06 2024-03-27T21:41:19.950 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.950 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.950 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:19.950 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497121+0000", 2024-03-27T21:41:19.950 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7190999999999999e-05 2024-03-27T21:41:19.950 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.951 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.951 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:19.951 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497126+0000", 2024-03-27T21:41:19.951 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3770000000000004e-06 2024-03-27T21:41:19.951 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.951 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.951 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:19.951 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502833+0000", 2024-03-27T21:41:19.951 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057073510000000003 2024-03-27T21:41:19.951 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.951 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.951 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:19.951 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502881+0000", 2024-03-27T21:41:19.952 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7961999999999999e-05 2024-03-27T21:41:19.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.952 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.952 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:19.952 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502949+0000", 2024-03-27T21:41:19.952 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7957e-05 2024-03-27T21:41:19.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.952 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.952 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:19.952 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504430+0000", 2024-03-27T21:41:19.952 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001480763 2024-03-27T21:41:19.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.952 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.952 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:19.952 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504459+0000", 2024-03-27T21:41:19.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8983999999999999e-05 2024-03-27T21:41:19.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:19.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504486+0000", 2024-03-27T21:41:19.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6883999999999999e-05 2024-03-27T21:41:19.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:19.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504506+0000", 2024-03-27T21:41:19.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0214e-05 2024-03-27T21:41:19.953 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.953 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:19.954 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.954 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.954 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.954 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:19.954 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:19.954 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 99.375838341999994, 2024-03-27T21:41:19.954 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074754859999999999, 2024-03-27T21:41:19.954 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:19.954 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:19.954 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:19.954 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:19.954 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:19.954 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-03-27T21:41:19.954 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.955 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:19.955 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.955 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:19.955 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:19.955 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.955 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.955 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.955 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:19.955 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:19.955 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.955 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.955 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.955 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:19.955 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:19.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:19.956 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.956 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.956 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:19.956 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497137+0000", 2024-03-27T21:41:19.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0730000000000004e-06 2024-03-27T21:41:19.956 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.956 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.956 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:19.956 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497143+0000", 2024-03-27T21:41:19.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3759999999999996e-06 2024-03-27T21:41:19.956 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.956 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.956 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:19.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497150+0000", 2024-03-27T21:41:19.957 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.002e-06 2024-03-27T21:41:19.957 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.957 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.957 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:19.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503249+0000", 2024-03-27T21:41:19.957 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060989750000000004 2024-03-27T21:41:19.957 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.957 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.957 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:19.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503291+0000", 2024-03-27T21:41:19.957 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1563999999999997e-05 2024-03-27T21:41:19.957 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.957 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.957 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:19.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503376+0000", 2024-03-27T21:41:19.958 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5752000000000002e-05 2024-03-27T21:41:19.958 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.958 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.958 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:19.958 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504545+0000", 2024-03-27T21:41:19.958 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169172 2024-03-27T21:41:19.958 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.958 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.958 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:19.958 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504550+0000", 2024-03-27T21:41:19.958 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7419999999999997e-06 2024-03-27T21:41:19.958 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.958 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.958 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:19.959 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504574+0000", 2024-03-27T21:41:19.959 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4124000000000001e-05 2024-03-27T21:41:19.959 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.959 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.959 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:19.959 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504607+0000", 2024-03-27T21:41:19.959 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3118999999999999e-05 2024-03-27T21:41:19.959 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.959 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:19.959 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.959 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.959 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.959 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:19.959 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:19.959 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 99.375814676000005, 2024-03-27T21:41:19.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078150589999999992, 2024-03-27T21:41:19.960 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:19.960 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:19.960 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:19.960 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:19.960 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:19.960 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-03-27T21:41:19.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.960 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:19.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.960 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:19.960 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:19.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:19.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:19.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:19.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497155+0000", 2024-03-27T21:41:19.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:19.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:19.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497230+0000", 2024-03-27T21:41:19.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4789000000000005e-05 2024-03-27T21:41:19.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:19.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497241+0000", 2024-03-27T21:41:19.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1442e-05 2024-03-27T21:41:19.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:19.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497255+0000", 2024-03-27T21:41:19.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3087e-05 2024-03-27T21:41:19.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:19.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503630+0000", 2024-03-27T21:41:19.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063755269999999998 2024-03-27T21:41:19.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:19.963 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503673+0000", 2024-03-27T21:41:19.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2616e-05 2024-03-27T21:41:19.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:19.963 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503738+0000", 2024-03-27T21:41:19.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5114e-05 2024-03-27T21:41:19.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:19.963 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504907+0000", 2024-03-27T21:41:19.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169432 2024-03-27T21:41:19.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:19.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504926+0000", 2024-03-27T21:41:19.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9086999999999999e-05 2024-03-27T21:41:19.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:19.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504950+0000", 2024-03-27T21:41:19.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3598999999999999e-05 2024-03-27T21:41:19.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:19.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504971+0000", 2024-03-27T21:41:19.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0772999999999999e-05 2024-03-27T21:41:19.965 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.965 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:19.965 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.965 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:19.965 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:19.965 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 99.375707636000001, 2024-03-27T21:41:19.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082962590000000003, 2024-03-27T21:41:19.965 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:19.965 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:19.965 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:19.965 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:19.965 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:19.966 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-03-27T21:41:19.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.966 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:19.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:19.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:19.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:19.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:19.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:19.967 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497262+0000", 2024-03-27T21:41:19.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:19.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.967 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:19.967 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497267+0000", 2024-03-27T21:41:19.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.955e-06 2024-03-27T21:41:19.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.967 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:19.967 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497275+0000", 2024-03-27T21:41:19.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.824e-06 2024-03-27T21:41:19.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.967 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:19.967 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497280+0000", 2024-03-27T21:41:19.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8230000000000002e-06 2024-03-27T21:41:19.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:19.968 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504628+0000", 2024-03-27T21:41:19.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007348561 2024-03-27T21:41:19.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:19.968 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504673+0000", 2024-03-27T21:41:19.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4313000000000001e-05 2024-03-27T21:41:19.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:19.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504745+0000", 2024-03-27T21:41:19.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2303000000000002e-05 2024-03-27T21:41:19.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:19.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505358+0000", 2024-03-27T21:41:19.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061281300000000003 2024-03-27T21:41:19.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:19.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505503+0000", 2024-03-27T21:41:19.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000145134 2024-03-27T21:41:19.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:19.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505526+0000", 2024-03-27T21:41:19.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3312e-05 2024-03-27T21:41:19.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:19.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505559+0000", 2024-03-27T21:41:19.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2821999999999998e-05 2024-03-27T21:41:19.970 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.970 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:19.970 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.970 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:19.971 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:19.971 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 98.766485396999997, 2024-03-27T21:41:19.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053753339999999998, 2024-03-27T21:41:19.971 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:19.971 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:19.971 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:19.971 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:19.971 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:19.971 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159 2024-03-27T21:41:19.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.971 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:19.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:19.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:19.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:19.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:19.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:19.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:19.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:19.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:19.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:19.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106488+0000", 2024-03-27T21:41:19.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9830000000000001e-06 2024-03-27T21:41:19.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:19.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106491+0000", 2024-03-27T21:41:19.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3459999999999998e-06 2024-03-27T21:41:19.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:19.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106493+0000", 2024-03-27T21:41:19.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.509e-06 2024-03-27T21:41:19.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:19.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110636+0000", 2024-03-27T21:41:19.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041427260000000002 2024-03-27T21:41:19.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:19.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110683+0000", 2024-03-27T21:41:19.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6557000000000002e-05 2024-03-27T21:41:19.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:19.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110755+0000", 2024-03-27T21:41:19.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2119000000000005e-05 2024-03-27T21:41:19.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:19.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111415+0000", 2024-03-27T21:41:19.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066065199999999996 2024-03-27T21:41:19.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.975 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:19.975 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111793+0000", 2024-03-27T21:41:19.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037732100000000001 2024-03-27T21:41:19.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.975 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:19.975 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111829+0000", 2024-03-27T21:41:19.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418999999999998e-05 2024-03-27T21:41:19.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:19.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:19.975 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:19.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111860+0000", 2024-03-27T21:41:19.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1003000000000001e-05 2024-03-27T21:41:19.976 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.976 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:19.976 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.976 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:19.976 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:19.976 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:19.977 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:19.991+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:19.978 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:41:20.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.017+0000 7f64dffff640 1 -- 172.21.15.60:0/1208441076 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v58) v1 ==== 97+0+0 (secure 0 0 0) 0x7f64ec024020 con 0x7f64f813e810 2024-03-27T21:41:20.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.017+0000 7f64dffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:41:20.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.017+0000 7f64dffff640 10 monclient: _finish_command 2 = system:0 nodeep-scrub is set 2024-03-27T21:41:20.003 INFO:teuthology.orchestra.run.smithi060.stderr:nodeep-scrub is set 2024-03-27T21:41:20.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.020+0000 7f64fe167640 10 librados: watch_flush enter 2024-03-27T21:41:20.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.020+0000 7f64fe167640 10 librados: watch_flush exit 2024-03-27T21:41:20.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.020+0000 7f64fe167640 20 client.4760.objecter shutdown clearing up homeless session... 2024-03-27T21:41:20.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.020+0000 7f64fe167640 10 client.4760.objecter successfully canceled tick 2024-03-27T21:41:20.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.020+0000 7f64fe167640 1 -- 172.21.15.60:0/1208441076 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f64e406c590 msgr2=0x7f64e406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:20.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.020+0000 7f64fe167640 1 --2- 172.21.15.60:0/1208441076 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f64e406c590 0x7f64e406ea50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f64e800a950 tx=0x7f64e8008040 comp rx=0 tx=0).stop 2024-03-27T21:41:20.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.020+0000 7f64fe167640 10 monclient: shutdown 2024-03-27T21:41:20.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.020+0000 7f64fe167640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:20.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.020+0000 7f64fe167640 1 -- 172.21.15.60:0/1208441076 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f813e810 msgr2=0x7f64f80b4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:20.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.020+0000 7f64fe167640 1 --2- 172.21.15.60:0/1208441076 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f813e810 0x7f64f80b4ad0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f64ec009f60 tx=0x7f64ec009f90 comp rx=0 tx=0).stop 2024-03-27T21:41:20.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.020+0000 7f64fe167640 1 -- 172.21.15.60:0/1208441076 shutdown_connections 2024-03-27T21:41:20.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.020+0000 7f64fe167640 1 --2- 172.21.15.60:0/1208441076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f8141630 0x7f64f813f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:20.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.020+0000 7f64fe167640 1 --2- 172.21.15.60:0/1208441076 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f64f813e810 0x7f64f80b4ad0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:20.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.020+0000 7f64fe167640 1 --2- 172.21.15.60:0/1208441076 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f64e406c590 0x7f64e406ea50 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:20.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.020+0000 7f64fe167640 1 --2- 172.21.15.60:0/1208441076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64f8071a50 0x7f64f81ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:20.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.020+0000 7f64fe167640 1 -- 172.21.15.60:0/1208441076 >> 172.21.15.60:0/1208441076 conn(0x7f64f80b7300 msgr2=0x7f64f80a8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:20.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.021+0000 7f64fe167640 1 -- 172.21.15.60:0/1208441076 shutdown_connections 2024-03-27T21:41:20.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.021+0000 7f64fe167640 1 -- 172.21.15.60:0/1208441076 wait complete. 2024-03-27T21:41:20.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:20.021+0000 7f64fe167640 1 librados: shutdown 2024-03-27T21:41:20.078 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:20.092+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:20.110 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:20.110 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:20.111 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:20.111 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:20.121 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:41:20.205 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:20.219+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:20.280 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:20.293+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:20.284 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:20.284 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:20.284 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:41:20.284 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:41:20.284 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:20.294 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:41:20.380 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:20.394+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:20.481 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:20.494+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:20.487 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:20.487 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:41:20.487 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:41:20.487 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-03-27T21:41:20.487 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.487 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:20 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2024-03-27T21:41:20.488 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:41:20.488 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 100.611546956, 2024-03-27T21:41:20.488 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015345630000000001, 2024-03-27T21:41:20.488 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:20.488 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:20.488 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:20.488 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.488 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:20.488 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:41:20.488 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.488 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.488 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.488 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:20.488 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:41:20.489 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.489 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.489 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.489 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:20.489 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888562+0000", 2024-03-27T21:41:20.489 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:20.489 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.489 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.489 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:20.489 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888735+0000", 2024-03-27T21:41:20.489 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017254799999999999 2024-03-27T21:41:20.489 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.489 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.489 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:20.490 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888747+0000", 2024-03-27T21:41:20.490 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2469999999999999e-05 2024-03-27T21:41:20.490 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.490 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.490 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:20.490 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888761+0000", 2024-03-27T21:41:20.490 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3254e-05 2024-03-27T21:41:20.490 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.490 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.490 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:20.490 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888804+0000", 2024-03-27T21:41:20.490 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3129e-05 2024-03-27T21:41:20.490 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.490 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.491 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:20.491 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888820+0000", 2024-03-27T21:41:20.491 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6426000000000002e-05 2024-03-27T21:41:20.491 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.491 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.491 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:20.491 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890060+0000", 2024-03-27T21:41:20.491 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012398050000000001 2024-03-27T21:41:20.491 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.491 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.491 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:20.491 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890098+0000", 2024-03-27T21:41:20.491 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7795e-05 2024-03-27T21:41:20.491 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.491 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:20.492 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.492 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2024-03-27T21:41:20.492 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:41:20.492 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 100.610722556, 2024-03-27T21:41:20.492 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001069555, 2024-03-27T21:41:20.492 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:20.492 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:20.492 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:20.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.492 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:20.492 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:41:20.493 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.493 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.493 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:20.493 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:41:20.493 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.493 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.493 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:20.493 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889387+0000", 2024-03-27T21:41:20.493 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:20.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.493 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.493 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:20.494 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889455+0000", 2024-03-27T21:41:20.494 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7930999999999997e-05 2024-03-27T21:41:20.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.494 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.494 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:20.494 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889470+0000", 2024-03-27T21:41:20.494 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5554e-05 2024-03-27T21:41:20.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.494 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.494 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:20.494 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889476+0000", 2024-03-27T21:41:20.494 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0619999999999997e-06 2024-03-27T21:41:20.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.494 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.495 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:20.495 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889513+0000", 2024-03-27T21:41:20.495 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6606999999999997e-05 2024-03-27T21:41:20.495 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.495 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.495 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:20.495 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889528+0000", 2024-03-27T21:41:20.495 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4982e-05 2024-03-27T21:41:20.495 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.495 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.495 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:20.495 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890421+0000", 2024-03-27T21:41:20.495 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00089293700000000001 2024-03-27T21:41:20.495 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.495 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.496 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:20.496 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890457+0000", 2024-03-27T21:41:20.496 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6260000000000002e-05 2024-03-27T21:41:20.496 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.496 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:20.496 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.496 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.496 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.496 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:38 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'24, mlcod=17'22)", 2024-03-27T21:41:20.496 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:41:20.496 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 100.59348659, 2024-03-27T21:41:20.496 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016182080000000001, 2024-03-27T21:41:20.496 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:20.496 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:20.497 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:20.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.497 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:20.497 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:41:20.497 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.497 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.497 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:20.497 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:41:20.497 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.497 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.497 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:20.497 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906623+0000", 2024-03-27T21:41:20.498 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:20.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.498 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.498 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:20.498 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906687+0000", 2024-03-27T21:41:20.498 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3706000000000003e-05 2024-03-27T21:41:20.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.498 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.498 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:20.498 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906698+0000", 2024-03-27T21:41:20.498 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1752e-05 2024-03-27T21:41:20.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.498 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.498 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:20.499 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906704+0000", 2024-03-27T21:41:20.499 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3940000000000004e-06 2024-03-27T21:41:20.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.499 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.499 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:20.499 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906741+0000", 2024-03-27T21:41:20.499 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6944999999999999e-05 2024-03-27T21:41:20.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.499 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.499 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:20.501 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906760+0000", 2024-03-27T21:41:20.501 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9293e-05 2024-03-27T21:41:20.501 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.501 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.501 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:20.501 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908211+0000", 2024-03-27T21:41:20.501 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014511350000000001 2024-03-27T21:41:20.501 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.501 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.501 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:20.502 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908242+0000", 2024-03-27T21:41:20.502 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0712e-05 2024-03-27T21:41:20.502 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.502 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:20.502 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.502 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.502 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.502 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:39 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'25, mlcod=17'22)", 2024-03-27T21:41:20.502 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:20.502 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 100.592641675, 2024-03-27T21:41:20.502 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012060199999999999, 2024-03-27T21:41:20.502 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:20.502 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:20.502 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:20.503 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.503 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:20.503 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:20.503 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.503 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.503 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:20.503 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:20.503 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.503 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.503 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:20.503 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:20.503 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:20.504 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.504 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.504 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:20.504 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907533+0000", 2024-03-27T21:41:20.504 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5675999999999999e-05 2024-03-27T21:41:20.504 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.504 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.504 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:20.504 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907545+0000", 2024-03-27T21:41:20.504 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.115e-05 2024-03-27T21:41:20.504 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.504 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.504 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:20.504 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907550+0000", 2024-03-27T21:41:20.505 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8320000000000002e-06 2024-03-27T21:41:20.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.505 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:20.505 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907592+0000", 2024-03-27T21:41:20.505 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1309e-05 2024-03-27T21:41:20.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.505 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:20.505 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907611+0000", 2024-03-27T21:41:20.505 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9009e-05 2024-03-27T21:41:20.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.505 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:20.506 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908638+0000", 2024-03-27T21:41:20.506 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001027533 2024-03-27T21:41:20.506 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.506 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.506 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:20.506 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908675+0000", 2024-03-27T21:41:20.506 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6329999999999999e-05 2024-03-27T21:41:20.506 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.506 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:20.506 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.506 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.506 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.506 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:57 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'35, mlcod=17'33)", 2024-03-27T21:41:20.506 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:41:20.507 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 100.57855724300001, 2024-03-27T21:41:20.507 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001346578, 2024-03-27T21:41:20.507 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:20.507 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:20.507 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:20.507 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.507 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:20.507 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:41:20.507 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.507 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.507 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:20.507 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:41:20.507 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.508 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.508 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:20.508 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921552+0000", 2024-03-27T21:41:20.508 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:20.508 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.508 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.508 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:20.508 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921615+0000", 2024-03-27T21:41:20.508 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2787999999999996e-05 2024-03-27T21:41:20.508 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.508 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.508 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:20.508 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921626+0000", 2024-03-27T21:41:20.508 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0888e-05 2024-03-27T21:41:20.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.509 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:20.509 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921635+0000", 2024-03-27T21:41:20.509 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4639999999999995e-06 2024-03-27T21:41:20.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.509 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:20.509 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921673+0000", 2024-03-27T21:41:20.509 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7891e-05 2024-03-27T21:41:20.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.509 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:20.510 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921694+0000", 2024-03-27T21:41:20.510 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0426000000000001e-05 2024-03-27T21:41:20.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.510 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:20.510 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922865+0000", 2024-03-27T21:41:20.510 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011709509999999999 2024-03-27T21:41:20.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.510 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:20.510 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922900+0000", 2024-03-27T21:41:20.510 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4819e-05 2024-03-27T21:41:20.510 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.511 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:20.511 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.511 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:71 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'45, mlcod=17'43)", 2024-03-27T21:41:20.511 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:41:20.511 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 100.563258531, 2024-03-27T21:41:20.511 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001460907, 2024-03-27T21:41:20.511 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:20.511 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:20.511 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:20.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.511 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:20.511 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:41:20.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.512 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.512 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:20.512 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:41:20.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.512 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.512 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:20.512 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936851+0000", 2024-03-27T21:41:20.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:20.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.512 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.512 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:20.513 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936903+0000", 2024-03-27T21:41:20.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2302999999999997e-05 2024-03-27T21:41:20.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.513 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:20.513 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936915+0000", 2024-03-27T21:41:20.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1967000000000001e-05 2024-03-27T21:41:20.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.513 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:20.513 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936923+0000", 2024-03-27T21:41:20.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9370000000000004e-06 2024-03-27T21:41:20.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:20.514 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936970+0000", 2024-03-27T21:41:20.514 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6987000000000003e-05 2024-03-27T21:41:20.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.514 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:20.514 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937002+0000", 2024-03-27T21:41:20.514 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2410999999999999e-05 2024-03-27T21:41:20.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.514 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:20.514 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938279+0000", 2024-03-27T21:41:20.514 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001276552 2024-03-27T21:41:20.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.515 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.515 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:20.515 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938313+0000", 2024-03-27T21:41:20.515 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3526999999999999e-05 2024-03-27T21:41:20.515 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.515 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:20.515 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.515 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.515 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:72 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'46, mlcod=17'43)", 2024-03-27T21:41:20.515 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:41:20.515 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 100.56262096899999, 2024-03-27T21:41:20.516 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014239490000000001, 2024-03-27T21:41:20.516 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:20.516 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:20.516 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:20.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.516 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:20.516 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:41:20.516 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.516 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:20.516 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:41:20.516 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.517 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:20.517 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937488+0000", 2024-03-27T21:41:20.517 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:20.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.517 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:20.517 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937547+0000", 2024-03-27T21:41:20.517 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8196999999999999e-05 2024-03-27T21:41:20.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.517 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:20.517 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937559+0000", 2024-03-27T21:41:20.518 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1928e-05 2024-03-27T21:41:20.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.518 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:20.518 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937564+0000", 2024-03-27T21:41:20.518 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4759999999999997e-06 2024-03-27T21:41:20.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.518 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:20.518 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937621+0000", 2024-03-27T21:41:20.518 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6712000000000002e-05 2024-03-27T21:41:20.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.518 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:20.519 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937640+0000", 2024-03-27T21:41:20.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9517e-05 2024-03-27T21:41:20.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.519 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:20.519 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938883+0000", 2024-03-27T21:41:20.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001242567 2024-03-27T21:41:20.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.519 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:20.519 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938913+0000", 2024-03-27T21:41:20.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0287e-05 2024-03-27T21:41:20.519 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.519 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:20.520 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.520 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.520 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:73 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'47, mlcod=17'43)", 2024-03-27T21:41:20.520 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:20.520 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 100.56194487499999, 2024-03-27T21:41:20.520 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010746239999999999, 2024-03-27T21:41:20.520 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:20.520 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:20.520 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:20.520 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.520 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:20.520 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:20.520 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.520 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.521 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:20.521 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:20.521 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.521 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:20.521 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:20.521 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:20.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.521 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:20.521 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938217+0000", 2024-03-27T21:41:20.521 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2743e-05 2024-03-27T21:41:20.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.522 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:20.522 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938229+0000", 2024-03-27T21:41:20.522 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1956999999999999e-05 2024-03-27T21:41:20.522 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.522 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.522 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:20.522 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938235+0000", 2024-03-27T21:41:20.522 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4589999999999997e-06 2024-03-27T21:41:20.522 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.522 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.522 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:20.522 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938341+0000", 2024-03-27T21:41:20.522 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000106054 2024-03-27T21:41:20.522 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.522 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:20.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938369+0000", 2024-03-27T21:41:20.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8291000000000001e-05 2024-03-27T21:41:20.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:20.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939209+0000", 2024-03-27T21:41:20.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084045400000000003 2024-03-27T21:41:20.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:20.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939240+0000", 2024-03-27T21:41:20.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0429999999999998e-05 2024-03-27T21:41:20.523 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.523 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:20.523 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.524 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:75 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'49, mlcod=17'47)", 2024-03-27T21:41:20.524 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:41:20.524 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 100.555507228, 2024-03-27T21:41:20.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001017001, 2024-03-27T21:41:20.524 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:20.524 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:20.524 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:20.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:20.525 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:41:20.525 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.525 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.525 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:20.525 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:41:20.525 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.525 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.525 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:20.525 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944601+0000", 2024-03-27T21:41:20.525 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999986 2024-03-27T21:41:20.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.525 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.526 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:20.526 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944614+0000", 2024-03-27T21:41:20.526 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2496e-05 2024-03-27T21:41:20.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.526 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:20.526 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944640+0000", 2024-03-27T21:41:20.526 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5865999999999999e-05 2024-03-27T21:41:20.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.526 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:20.526 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944651+0000", 2024-03-27T21:41:20.526 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1797000000000001e-05 2024-03-27T21:41:20.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.527 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:20.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944762+0000", 2024-03-27T21:41:20.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110412 2024-03-27T21:41:20.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.527 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:20.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944820+0000", 2024-03-27T21:41:20.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7649999999999999e-05 2024-03-27T21:41:20.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.527 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:20.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945585+0000", 2024-03-27T21:41:20.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076562600000000005 2024-03-27T21:41:20.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.528 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:20.528 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945620+0000", 2024-03-27T21:41:20.528 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4764000000000001e-05 2024-03-27T21:41:20.528 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.528 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:20.528 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.528 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.528 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:86 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'50)", 2024-03-27T21:41:20.528 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:20.528 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 100.547745741, 2024-03-27T21:41:20.528 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011392380000000001, 2024-03-27T21:41:20.528 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:20.528 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:20.528 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:20.528 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.529 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:20.529 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:20.529 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.529 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.529 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:20.529 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:20.529 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.529 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.529 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:20.529 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:20.529 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:20.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.530 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.530 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:20.530 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952373+0000", 2024-03-27T21:41:20.530 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9220000000000003e-06 2024-03-27T21:41:20.530 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.530 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.530 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:20.530 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952381+0000", 2024-03-27T21:41:20.530 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3070000000000003e-06 2024-03-27T21:41:20.530 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.530 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.530 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:20.530 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952385+0000", 2024-03-27T21:41:20.530 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.121e-06 2024-03-27T21:41:20.530 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.531 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.531 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:20.531 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952419+0000", 2024-03-27T21:41:20.531 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4082999999999999e-05 2024-03-27T21:41:20.531 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.531 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.531 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:20.531 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952440+0000", 2024-03-27T21:41:20.531 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0837999999999999e-05 2024-03-27T21:41:20.531 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.531 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.531 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:20.531 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953472+0000", 2024-03-27T21:41:20.532 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001032119 2024-03-27T21:41:20.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.532 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.532 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:20.532 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953504+0000", 2024-03-27T21:41:20.532 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1250000000000001e-05 2024-03-27T21:41:20.532 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.532 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:20.532 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.532 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.532 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:87 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-27T21:41:20.532 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:41:20.533 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 100.54737848400001, 2024-03-27T21:41:20.533 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00135519, 2024-03-27T21:41:20.533 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:20.533 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:20.533 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:20.533 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.533 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:20.533 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:41:20.533 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.533 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.533 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:20.533 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:41:20.533 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.534 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.534 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.534 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:20.534 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952731+0000", 2024-03-27T21:41:20.534 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:20.534 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.534 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.534 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:20.534 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952781+0000", 2024-03-27T21:41:20.534 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0161000000000003e-05 2024-03-27T21:41:20.534 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.534 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.534 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:20.534 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952794+0000", 2024-03-27T21:41:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2853e-05 2024-03-27T21:41:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952799+0000", 2024-03-27T21:41:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5560000000000001e-06 2024-03-27T21:41:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952832+0000", 2024-03-27T21:41:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3834999999999997e-05 2024-03-27T21:41:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952851+0000", 2024-03-27T21:41:20.536 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8289999999999999e-05 2024-03-27T21:41:20.536 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.536 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.536 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:20.536 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954055+0000", 2024-03-27T21:41:20.536 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012037949999999999 2024-03-27T21:41:20.536 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.536 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.536 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:20.536 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954087+0000", 2024-03-27T21:41:20.536 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2292999999999997e-05 2024-03-27T21:41:20.536 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.536 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:20.537 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.537 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.537 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.538 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:89 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'59, mlcod=17'57)", 2024-03-27T21:41:20.538 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:41:20.538 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 100.545104263, 2024-03-27T21:41:20.538 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00138893, 2024-03-27T21:41:20.538 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:20.538 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:20.538 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:20.538 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.538 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:20.538 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:41:20.538 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.538 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.539 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.539 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:20.539 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:41:20.539 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.539 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.539 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.539 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:20.539 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955005+0000", 2024-03-27T21:41:20.539 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:20.539 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.539 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.539 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:20.540 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955071+0000", 2024-03-27T21:41:20.540 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5290999999999995e-05 2024-03-27T21:41:20.540 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.540 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.540 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:20.540 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955081+0000", 2024-03-27T21:41:20.540 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0572e-05 2024-03-27T21:41:20.540 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.540 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.540 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:20.540 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955086+0000", 2024-03-27T21:41:20.540 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8380000000000001e-06 2024-03-27T21:41:20.540 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.540 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.540 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:20.540 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955122+0000", 2024-03-27T21:41:20.541 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5698000000000003e-05 2024-03-27T21:41:20.541 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.541 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.541 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:20.541 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955140+0000", 2024-03-27T21:41:20.541 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7900000000000001e-05 2024-03-27T21:41:20.541 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.541 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.541 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:20.541 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956366+0000", 2024-03-27T21:41:20.541 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001226471 2024-03-27T21:41:20.541 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.541 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.541 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:20.542 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956395+0000", 2024-03-27T21:41:20.542 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8782e-05 2024-03-27T21:41:20.542 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.542 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:20.542 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.542 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.542 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.543 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:90 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'60, mlcod=17'57)", 2024-03-27T21:41:20.543 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:20.543 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 100.544419686, 2024-03-27T21:41:20.543 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010778859999999999, 2024-03-27T21:41:20.543 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:20.543 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:20.543 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:20.543 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.543 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:20.543 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:20.543 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.543 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.543 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.543 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:20.543 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:20.544 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.544 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.544 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.544 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:20.544 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:20.544 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:20.544 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.544 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.544 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:20.544 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955747+0000", 2024-03-27T21:41:20.544 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7130999999999999e-05 2024-03-27T21:41:20.544 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.544 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.544 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:20.544 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955756+0000", 2024-03-27T21:41:20.545 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2750000000000001e-06 2024-03-27T21:41:20.545 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.545 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.545 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:20.545 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955761+0000", 2024-03-27T21:41:20.545 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3900000000000003e-06 2024-03-27T21:41:20.545 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.545 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.545 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:20.545 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955797+0000", 2024-03-27T21:41:20.545 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6650000000000003e-05 2024-03-27T21:41:20.545 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.545 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.545 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:20.545 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955816+0000", 2024-03-27T21:41:20.546 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8195000000000001e-05 2024-03-27T21:41:20.546 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.546 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.546 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:20.546 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956739+0000", 2024-03-27T21:41:20.546 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00092369700000000002 2024-03-27T21:41:20.546 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.546 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.546 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:20.547 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956768+0000", 2024-03-27T21:41:20.547 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9048e-05 2024-03-27T21:41:20.547 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.547 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:20.547 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.547 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.548 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.548 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:118 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'80, mlcod=17'78)", 2024-03-27T21:41:20.548 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:20.548 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 100.411428863, 2024-03-27T21:41:20.548 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014200669999999999, 2024-03-27T21:41:20.548 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:20.549 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:20.549 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:20.549 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.549 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:20.549 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:20.549 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.550 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.550 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.550 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:20.550 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:20.550 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.550 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.551 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.551 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:20.551 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:20.551 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:20.551 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.551 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.552 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:20.552 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088756+0000", 2024-03-27T21:41:20.552 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4902999999999998e-05 2024-03-27T21:41:20.552 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.552 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.552 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:20.553 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088764+0000", 2024-03-27T21:41:20.553 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8169999999999999e-06 2024-03-27T21:41:20.553 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.553 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.553 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:20.553 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088767+0000", 2024-03-27T21:41:20.554 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8310000000000003e-06 2024-03-27T21:41:20.554 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.554 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.554 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:20.554 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088798+0000", 2024-03-27T21:41:20.554 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0800999999999999e-05 2024-03-27T21:41:20.555 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.555 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.555 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:20.555 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088817+0000", 2024-03-27T21:41:20.555 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8467000000000001e-05 2024-03-27T21:41:20.555 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.555 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.556 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:20.556 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090074+0000", 2024-03-27T21:41:20.556 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012574089999999999 2024-03-27T21:41:20.556 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.556 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.556 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:20.557 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090101+0000", 2024-03-27T21:41:20.557 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7223e-05 2024-03-27T21:41:20.557 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.557 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:20.557 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.557 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.558 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.558 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:119 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'81, mlcod=17'78)", 2024-03-27T21:41:20.558 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:20.558 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 100.410694068, 2024-03-27T21:41:20.558 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014990870000000001, 2024-03-27T21:41:20.558 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:20.559 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:20.559 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:20.559 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.559 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:20.559 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:20.559 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.560 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.560 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.560 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:20.560 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:20.560 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.561 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.561 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.561 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:20.561 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:20.561 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:20.561 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.561 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.562 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:20.562 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089491+0000", 2024-03-27T21:41:20.562 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4805e-05 2024-03-27T21:41:20.562 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.562 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.562 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:20.563 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089498+0000", 2024-03-27T21:41:20.563 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0650000000000001e-06 2024-03-27T21:41:20.563 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.563 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.563 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:20.563 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089501+0000", 2024-03-27T21:41:20.564 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.54e-06 2024-03-27T21:41:20.564 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.564 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.564 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:20.564 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089533+0000", 2024-03-27T21:41:20.564 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1838999999999998e-05 2024-03-27T21:41:20.565 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.565 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.565 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:20.565 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089550+0000", 2024-03-27T21:41:20.565 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7465999999999998e-05 2024-03-27T21:41:20.565 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.565 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.565 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:20.566 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090890+0000", 2024-03-27T21:41:20.566 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013397979999999999 2024-03-27T21:41:20.566 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.566 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.566 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:20.566 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090915+0000", 2024-03-27T21:41:20.566 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4905e-05 2024-03-27T21:41:20.566 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.567 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:20.567 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.567 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.567 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.567 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:120 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'82, mlcod=17'78)", 2024-03-27T21:41:20.567 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:20.567 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 100.409894188, 2024-03-27T21:41:20.567 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001527093, 2024-03-27T21:41:20.568 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:20.568 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:20.568 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:20.568 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.568 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:20.568 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:20.568 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.569 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.569 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.569 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:20.569 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:20.569 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.569 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.569 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.569 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:20.570 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:20.570 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:20.570 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.570 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.570 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:20.570 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090267+0000", 2024-03-27T21:41:20.570 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1477000000000002e-05 2024-03-27T21:41:20.570 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.570 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.570 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:20.570 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090273+0000", 2024-03-27T21:41:20.570 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9519999999999999e-06 2024-03-27T21:41:20.570 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.570 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.571 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:20.571 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090276+0000", 2024-03-27T21:41:20.571 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.219e-06 2024-03-27T21:41:20.571 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.571 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.571 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:20.571 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090304+0000", 2024-03-27T21:41:20.571 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8073999999999999e-05 2024-03-27T21:41:20.571 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.571 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.571 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:20.571 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090316+0000", 2024-03-27T21:41:20.571 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2088e-05 2024-03-27T21:41:20.571 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.572 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.572 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:20.572 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091717+0000", 2024-03-27T21:41:20.572 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014003889999999999 2024-03-27T21:41:20.572 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.572 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.572 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:20.572 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091743+0000", 2024-03-27T21:41:20.572 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6234e-05 2024-03-27T21:41:20.572 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.572 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:20.572 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.572 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.572 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.572 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:121 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'83, mlcod=17'78)", 2024-03-27T21:41:20.573 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:20.573 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 100.409083823, 2024-03-27T21:41:20.573 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010226899999999999, 2024-03-27T21:41:20.573 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:20.573 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:20.573 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:20.573 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.573 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:20.573 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:20.573 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.573 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.573 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.573 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:20.573 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:20.573 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.574 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.574 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.574 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:20.574 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:20.574 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:20.574 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.574 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.574 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:20.574 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091142+0000", 2024-03-27T21:41:20.574 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011608300000000001 2024-03-27T21:41:20.574 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.574 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.574 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:20.575 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091151+0000", 2024-03-27T21:41:20.575 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2350000000000005e-06 2024-03-27T21:41:20.575 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.575 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.575 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:20.575 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091156+0000", 2024-03-27T21:41:20.575 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5419999999999999e-06 2024-03-27T21:41:20.575 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.575 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.575 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:20.575 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091191+0000", 2024-03-27T21:41:20.575 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5453000000000001e-05 2024-03-27T21:41:20.575 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.575 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.576 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:20.576 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091208+0000", 2024-03-27T21:41:20.576 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7119999999999999e-05 2024-03-27T21:41:20.576 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.576 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.576 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:20.576 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092024+0000", 2024-03-27T21:41:20.576 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000815539 2024-03-27T21:41:20.576 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.576 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.576 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:20.576 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092049+0000", 2024-03-27T21:41:20.576 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5171e-05 2024-03-27T21:41:20.576 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.576 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:20.577 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.577 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.577 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.577 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:142 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'100, mlcod=18'98)", 2024-03-27T21:41:20.577 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:20.577 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 99.994076868999997, 2024-03-27T21:41:20.577 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001198778, 2024-03-27T21:41:20.577 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:20.577 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:20.577 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:20.577 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.577 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:20.577 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:20.577 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.578 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.578 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.578 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:20.578 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:20.578 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.578 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.578 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.578 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:20.578 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:20.578 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:20.578 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.578 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.578 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:20.578 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506101+0000", 2024-03-27T21:41:20.579 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8202999999999997e-05 2024-03-27T21:41:20.579 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.579 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.579 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:20.579 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506109+0000", 2024-03-27T21:41:20.579 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3059999999999998e-06 2024-03-27T21:41:20.579 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.579 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.579 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:20.579 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506113+0000", 2024-03-27T21:41:20.579 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0119999999999999e-06 2024-03-27T21:41:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506137+0000", 2024-03-27T21:41:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4043999999999999e-05 2024-03-27T21:41:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506148+0000", 2024-03-27T21:41:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0446e-05 2024-03-27T21:41:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507213+0000", 2024-03-27T21:41:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001065215 2024-03-27T21:41:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507232+0000", 2024-03-27T21:41:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9038e-05 2024-03-27T21:41:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:143 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'101, mlcod=18'98)", 2024-03-27T21:41:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 99.993464250000002, 2024-03-27T21:41:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001357488, 2024-03-27T21:41:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506723+0000", 2024-03-27T21:41:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7404000000000006e-05 2024-03-27T21:41:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506730+0000", 2024-03-27T21:41:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5749999999999997e-06 2024-03-27T21:41:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506733+0000", 2024-03-27T21:41:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7979999999999999e-06 2024-03-27T21:41:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506758+0000", 2024-03-27T21:41:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4179e-05 2024-03-27T21:41:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506768+0000", 2024-03-27T21:41:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0740000000000001e-05 2024-03-27T21:41:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507978+0000", 2024-03-27T21:41:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012093900000000001 2024-03-27T21:41:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508003+0000", 2024-03-27T21:41:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5667e-05 2024-03-27T21:41:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:144 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'102, mlcod=18'98)", 2024-03-27T21:41:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:41:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 99.992805352999994, 2024-03-27T21:41:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013453390000000001, 2024-03-27T21:41:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:41:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:41:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507304+0000", 2024-03-27T21:41:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507397+0000", 2024-03-27T21:41:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2995e-05 2024-03-27T21:41:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507403+0000", 2024-03-27T21:41:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8640000000000001e-06 2024-03-27T21:41:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507407+0000", 2024-03-27T21:41:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5710000000000002e-06 2024-03-27T21:41:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507428+0000", 2024-03-27T21:41:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1192e-05 2024-03-27T21:41:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507438+0000", 2024-03-27T21:41:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8309999999999996e-06 2024-03-27T21:41:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508624+0000", 2024-03-27T21:41:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011861210000000001 2024-03-27T21:41:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:20.590 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:20.590 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508650+0000", 2024-03-27T21:41:20.590 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6118e-05 2024-03-27T21:41:20.590 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.590 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:20.590 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.590 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:20.590 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:20.590 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:20.592 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:41:20.592 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:20.605+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:20.692 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:20.706+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:20.701 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:20.701 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:20.701 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:20.701 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:20.714 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:41:20.793 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:20.807+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:20.873 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:20.873 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:20.873 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:41:20.873 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:41:20.873 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:20.882 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:41:20.894 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:20.908+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:20.992 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:20.992 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:41:20.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:41:20.992 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:41:20.992 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:20.995 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:21.009+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:21.001 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:41:21.096 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:21.110+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:21.144 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:21.144 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:21.145 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:21.145 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:21.154 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:41:21.198 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:21.212+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:21.289 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:21.289 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:21.289 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:41:21.289 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:41:21.289 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:21.298 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:41:21.298 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:21.311+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:21.398 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:21.412+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:21.449 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:21.449 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:41:21.449 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:41:21.449 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:41:21.449 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:21.458 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:41:21.502 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:21.516+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:21.575 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:21.575 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:21.575 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:21.575 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:21.584 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:41:21.599 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:21.613+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:21.675 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:41:21.700 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:21.714+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:21.730 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:21.731 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:21.731 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:21.731 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:21.731 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:21.740 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:41:21.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.792+0000 7fe215ddd640 10 monclient: get_monmap_and_config 2024-03-27T21:41:21.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.792+0000 7fe215ddd640 10 monclient: build_initial_monmap 2024-03-27T21:41:21.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.792+0000 7fe215ddd640 10 monclient: monmap: 2024-03-27T21:41:21.778 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:21.778 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:21.778 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:21.793310+0000 2024-03-27T21:41:21.778 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:21.793310+0000 2024-03-27T21:41:21.779 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:21.779 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:21.779 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:21.779 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:21.779 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:21.779 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:21.779 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:21.779 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:21.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.793+0000 7fe215ddd640 1 Processor -- start 2024-03-27T21:41:21.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.793+0000 7fe215ddd640 1 -- start start 2024-03-27T21:41:21.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.793+0000 7fe215ddd640 10 monclient: init 2024-03-27T21:41:21.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.793+0000 7fe215ddd640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:21.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.793+0000 7fe215ddd640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-03-27T21:41:21.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.793+0000 7fe215ddd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe210071a50 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:21.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.793+0000 7fe215ddd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe21013ef20 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:21.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.793+0000 7fe215ddd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe2100a87e0 0x7fe210141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:21.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.793+0000 7fe215ddd640 10 monclient(hunting): picked mon.noname-c con 0x7fe2100a87e0 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:21.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.793+0000 7fe215ddd640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2100a8e90 con 0x7fe210071a50 2024-03-27T21:41:21.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.793+0000 7fe215ddd640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2100aa040 con 0x7fe21013ef20 2024-03-27T21:41:21.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.793+0000 7fe215ddd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:21.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.793+0000 7fe215ddd640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe210071e30 con 0x7fe2100a87e0 2024-03-27T21:41:21.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.793+0000 7fe215ddd640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:21.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.793+0000 7fe215ddd640 10 monclient(hunting): authenticate will time out at 923.014852s 2024-03-27T21:41:21.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.793+0000 7fe20f7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe210071a50 0x7fe21013e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56778/0 (socket says 172.21.15.60:56778) 2024-03-27T21:41:21.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.793+0000 7fe20f7fe640 1 -- 172.21.15.60:0/1650504474 learned_addr learned my addr 172.21.15.60:0/1650504474 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:21.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20ffff640 1 --2- 172.21.15.60:0/1650504474 >> v2:172.21.15.191:3300/0 conn(0x7fe2100a87e0 0x7fe210141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:21.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20ffff640 10 monclient(hunting): get_auth_request con 0x7fe2100a87e0 auth_method 0 2024-03-27T21:41:21.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:21.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:21.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:21.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20e7fc640 1 -- 172.21.15.60:0/1650504474 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608997386 0 0) 0x7fe2100a8e90 con 0x7fe210071a50 2024-03-27T21:41:21.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20e7fc640 10 monclient(hunting): _negotiate 2024-03-27T21:41:21.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:21.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:21.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20e7fc640 10 monclient(hunting): my global_id is 4959 2024-03-27T21:41:21.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20e7fc640 1 -- 172.21.15.60:0/1650504474 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f8003520 con 0x7fe210071a50 2024-03-27T21:41:21.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:21.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:21.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:21.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20ffff640 10 monclient(hunting): handle_auth_done global_id 4894 payload 274 2024-03-27T21:41:21.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20ffff640 1 -- 172.21.15.60:0/1650504474 >> v1:172.21.15.191:6789/0 conn(0x7fe21013ef20 legacy=0x7fe210141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:21.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20ffff640 1 -- 172.21.15.60:0/1650504474 >> v1:172.21.15.60:6789/0 conn(0x7fe210071a50 legacy=0x7fe21013e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:21.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20ffff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:21.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20ffff640 1 monclient: found mon.noname-c 2024-03-27T21:41:21.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:21.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:41:21.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.794+0000 7fe20ffff640 1 -- 172.21.15.60:0/1650504474 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe210142280 con 0x7fe2100a87e0 2024-03-27T21:41:21.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.795+0000 7fe20ffff640 1 --2- 172.21.15.60:0/1650504474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2100a87e0 0x7fe210141d40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe208009d90 tx=0x7fe20802bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bef83b1a3a292f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:21.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.795+0000 7fe20e7fc640 1 -- 172.21.15.60:0/1650504474 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20802ccf0 con 0x7fe2100a87e0 2024-03-27T21:41:21.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.795+0000 7fe20e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:21.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.795+0000 7fe20e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:21.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.795+0000 7fe20e7fc640 10 monclient: dump: 2024-03-27T21:41:21.787 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:21.787 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:21.787 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:21.787 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:21.787 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:21.787 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:21.787 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:21.787 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:21.787 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:21.788 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:21.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.795+0000 7fe20e7fc640 10 monclient: _finish_auth 0 2024-03-27T21:41:21.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.795+0000 7fe20e7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:41:21.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.795+0000 7fe20e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:21.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.795+0000 7fe20e7fc640 1 -- 172.21.15.60:0/1650504474 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe20802ce50 con 0x7fe2100a87e0 2024-03-27T21:41:21.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.795+0000 7fe215ddd640 5 monclient: authenticate success, global_id 4894 2024-03-27T21:41:21.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.795+0000 7fe215ddd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:21.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.795+0000 7fe20e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:21.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.795+0000 7fe215ddd640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:21.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.795+0000 7fe20e7fc640 1 -- 172.21.15.60:0/1650504474 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe208034660 con 0x7fe2100a87e0 2024-03-27T21:41:21.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.795+0000 7fe215ddd640 4 set_mon_vals no callback set 2024-03-27T21:41:21.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.795+0000 7fe20e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:21.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.795+0000 7fe20e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:21.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.795+0000 7fe20e7fc640 10 monclient: dump: 2024-03-27T21:41:21.789 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:21.789 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:21.789 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:21.789 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:21.789 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:21.789 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:21.789 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:21.789 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:21.789 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:21.789 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:21.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.796+0000 7fe215ddd640 10 monclient: shutdown 2024-03-27T21:41:21.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.796+0000 7fe215ddd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:21.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.796+0000 7fe215ddd640 1 -- 172.21.15.60:0/1650504474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2100a87e0 msgr2=0x7fe210141d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:21.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.796+0000 7fe215ddd640 1 --2- 172.21.15.60:0/1650504474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2100a87e0 0x7fe210141d40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe208009d90 tx=0x7fe20802bf40 comp rx=0 tx=0).stop 2024-03-27T21:41:21.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.796+0000 7fe215ddd640 1 -- 172.21.15.60:0/1650504474 shutdown_connections 2024-03-27T21:41:21.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.796+0000 7fe215ddd640 1 --2- 172.21.15.60:0/1650504474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2100a87e0 0x7fe210141d40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:21.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.796+0000 7fe215ddd640 1 -- 172.21.15.60:0/1650504474 >> 172.21.15.60:0/1650504474 conn(0x7fe2100b7300 msgr2=0x7fe2100b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:21.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.796+0000 7fe215ddd640 1 -- 172.21.15.60:0/1650504474 shutdown_connections 2024-03-27T21:41:21.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.796+0000 7fe215ddd640 1 -- 172.21.15.60:0/1650504474 wait complete. 2024-03-27T21:41:21.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.797+0000 7fe215ddd640 10 monclient: build_initial_monmap 2024-03-27T21:41:21.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.797+0000 7fe215ddd640 10 monclient: monmap: 2024-03-27T21:41:21.790 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:21.790 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:21.790 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:21.790 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:21.790 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:21.790 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:21.790 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:21.791 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:21.791 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:21.791 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:21.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.797+0000 7fe215ddd640 1 librados: starting msgr at 2024-03-27T21:41:21.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.797+0000 7fe215ddd640 1 librados: starting objecter 2024-03-27T21:41:21.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.797+0000 7fe215ddd640 1 Processor -- start 2024-03-27T21:41:21.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.797+0000 7fe215ddd640 1 -- start start 2024-03-27T21:41:21.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.797+0000 7fe215ddd640 1 librados: setting wanted keys 2024-03-27T21:41:21.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.797+0000 7fe215ddd640 1 librados: calling monclient init 2024-03-27T21:41:21.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.797+0000 7fe215ddd640 10 monclient: init 2024-03-27T21:41:21.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.798+0000 7fe215ddd640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:21.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.798+0000 7fe215ddd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:41:21.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.798+0000 7fe215ddd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe210071a50 0x7fe210137450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:21.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.798+0000 7fe215ddd640 10 monclient(hunting): picked mon.noname-a con 0x7fe210071a50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:21.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.798+0000 7fe215ddd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2100a87e0 0x7fe210139070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:21.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.798+0000 7fe20f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe210071a50 0x7fe210137450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:21.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.798+0000 7fe20f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe210071a50 0x7fe210137450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36418/0 (socket says 172.21.15.60:36418) 2024-03-27T21:41:21.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.798+0000 7fe215ddd640 10 monclient(hunting): picked mon.noname-b con 0x7fe2100a87e0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:21.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.798+0000 7fe20f7fe640 1 -- 172.21.15.60:0/3375579277 learned_addr learned my addr 172.21.15.60:0/3375579277 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:21.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe215ddd640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe21013ef20 0x7fe2101395b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:21.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe215ddd640 10 monclient(hunting): picked mon.noname-c con 0x7fe21013ef20 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:21.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe215ddd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:21.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe210138b90 con 0x7fe210071a50 2024-03-27T21:41:21.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe215ddd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:21.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21013a5c0 con 0x7fe2100a87e0 2024-03-27T21:41:21.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe215ddd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:21.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2100b0830 con 0x7fe21013ef20 2024-03-27T21:41:21.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe215ddd640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:21.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe215ddd640 10 monclient(hunting): authenticate will time out at 923.019996s 2024-03-27T21:41:21.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20f7fe640 10 monclient(hunting): get_auth_request con 0x7fe210071a50 auth_method 0 2024-03-27T21:41:21.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:21.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:21.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:21.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20effd640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2100a87e0 0x7fe210139070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:21.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:21.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:21.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:21.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20effd640 10 monclient(hunting): get_auth_request con 0x7fe2100a87e0 auth_method 0 2024-03-27T21:41:21.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:21.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:21.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:21.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20ffff640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe21013ef20 0x7fe2101395b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:21.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20ffff640 10 monclient(hunting): get_auth_request con 0x7fe21013ef20 auth_method 0 2024-03-27T21:41:21.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:21.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:21.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:21.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:21.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:21.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:21.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:21.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:21.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:21.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20f7fe640 10 monclient(hunting): handle_auth_done global_id 4962 payload 824 2024-03-27T21:41:21.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20f7fe640 1 -- 172.21.15.60:0/3375579277 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe21013ef20 msgr2=0x7fe2101395b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:21.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20f7fe640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe21013ef20 0x7fe2101395b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:21.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20f7fe640 1 -- 172.21.15.60:0/3375579277 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2100a87e0 msgr2=0x7fe210139070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:21.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20f7fe640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2100a87e0 0x7fe210139070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:21.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20f7fe640 10 monclient: _finish_hunting 0 2024-03-27T21:41:21.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20f7fe640 1 monclient: found mon.noname-a 2024-03-27T21:41:21.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:21.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:21.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20f7fe640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe208009a70 con 0x7fe210071a50 2024-03-27T21:41:21.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe20f7fe640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe210071a50 0x7fe210137450 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe20000c530 tx=0x7fe20000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:21.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe1f7fff640 10 client.?.objecter ms_handle_connect 0x7fe210071a50 2024-03-27T21:41:21.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.799+0000 7fe1f7fff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:21.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe1f7fff640 1 -- 172.21.15.60:0/3375579277 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe200005ce0 con 0x7fe210071a50 2024-03-27T21:41:21.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe1f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:21.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe1f7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:21.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe1f7fff640 10 monclient: dump: 2024-03-27T21:41:21.802 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:21.802 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:21.802 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:21.802 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:21.802 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:21.802 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:21.802 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:21.802 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:21.802 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:21.802 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:21.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe1f7fff640 10 monclient: _finish_auth 0 2024-03-27T21:41:21.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe1f7fff640 10 monclient: _check_auth_tickets 2024-03-27T21:41:21.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe1f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:21.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe1f7fff640 1 -- 172.21.15.60:0/3375579277 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe200005e40 con 0x7fe210071a50 2024-03-27T21:41:21.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe215ddd640 5 monclient: authenticate success, global_id 4962 2024-03-27T21:41:21.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe215ddd640 10 monclient: _renew_subs 2024-03-27T21:41:21.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe215ddd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:21.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21013a830 con 0x7fe210071a50 2024-03-27T21:41:21.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe1f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:21.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe1f7fff640 1 -- 172.21.15.60:0/3375579277 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2000072b0 con 0x7fe210071a50 2024-03-27T21:41:21.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe1f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:21.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe1f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:21.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe1f7fff640 10 monclient: dump: 2024-03-27T21:41:21.803 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:21.803 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:21.803 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:21.803 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:21.803 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:21.804 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:21.804 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:21.804 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:21.804 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:21.804 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:21.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe215ddd640 10 client.4962.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:21.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe215ddd640 10 monclient: _renew_subs 2024-03-27T21:41:21.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe215ddd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:21.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe210138720 con 0x7fe210071a50 2024-03-27T21:41:21.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe215ddd640 1 librados: init done 2024-03-27T21:41:21.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.800+0000 7fe20dffb640 4 set_mon_vals no callback set 2024-03-27T21:41:21.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.801+0000 7fe215ddd640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:41:21.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.801+0000 7fe215ddd640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:21.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.801+0000 7fe215ddd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:21.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.801+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fe2101389a0 con 0x7fe210071a50 2024-03-27T21:41:21.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.801+0000 7fe1f7fff640 1 -- 172.21.15.60:0/3375579277 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe200007410 con 0x7fe210071a50 2024-03-27T21:41:21.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe1f7fff640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe1fc06c590 0x7fe1fc06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:21.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe1f7fff640 1 -- 172.21.15.60:0/3375579277 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fe200088710 con 0x7fe210071a50 2024-03-27T21:41:21.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe1f7fff640 10 client.4962.objecter ms_dispatch 0x7fe210000b90 osd_map(58..58 src has 1..58) v4 2024-03-27T21:41:21.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe1f7fff640 3 client.4962.objecter handle_osd_map got epochs [58,58] > 0 2024-03-27T21:41:21.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe1f7fff640 3 client.4962.objecter handle_osd_map decoding full epoch 58 2024-03-27T21:41:21.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe1f7fff640 20 client.4962.objecter dump_active .. 0 homeless 2024-03-27T21:41:21.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe20effd640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe1fc06c590 0x7fe1fc06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:21.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe1f7fff640 1 -- 172.21.15.60:0/3375579277 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v58) v1 ==== 54+0+16 (secure 0 0 0) 0x7fe2101389a0 con 0x7fe210071a50 2024-03-27T21:41:21.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe1f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:21.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe1f7fff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:21.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe20effd640 10 monclient: get_auth_request con 0x7fe1fc06c590 auth_method 0 2024-03-27T21:41:21.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe215ddd640 10 client.4962.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:21.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign -1 1 2024-03-27T21:41:21.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe215ddd640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fe210139af0 0x7fe2101310f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:21.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe2100aa590 osd=0 3 2024-03-27T21:41:21.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe2100aa590 osd=0 3 2024-03-27T21:41:21.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe2100aa590 osd=0 3 2024-03-27T21:41:21.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_remove -1 1 2024-03-27T21:41:21.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe215ddd640 20 client.4962.objecter get_session s=0x7fe2100aa590 osd=0 3 2024-03-27T21:41:21.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign 0 1 2024-03-27T21:41:21.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe2100aa590 osd=0 4 2024-03-27T21:41:21.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe215ddd640 10 client.4962.objecter _send_command 1 2024-03-27T21:41:21.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.802+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2101389a0 con 0x7fe210139af0 2024-03-27T21:41:21.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.803+0000 7fe20ffff640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fe210139af0 0x7fe2101310f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:21.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.803+0000 7fe20ffff640 10 monclient: get_auth_request con 0x7fe210139af0 auth_method 0 2024-03-27T21:41:21.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.803+0000 7fe20effd640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe1fc06c590 0x7fe1fc06ea50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe2101386b0 tx=0x7fe204009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:21.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.803+0000 7fe1f7fff640 10 client.4962.objecter ms_handle_connect 0x7fe1fc06c590 2024-03-27T21:41:21.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.803+0000 7fe1f7fff640 10 client.4962.objecter ms_handle_connect 0x7fe210139af0 2024-03-27T21:41:21.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.803+0000 7fe20ffff640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fe210139af0 0x7fe2101310f0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:21.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.804+0000 7fe1f7fff640 1 -- 172.21.15.60:0/3375579277 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe2101389a0 con 0x7fe210139af0 2024-03-27T21:41:21.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.804+0000 7fe1f7fff640 10 client.4962.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:41:21.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.804+0000 7fe1f7fff640 10 client.4962.objecter _finish_command 1 = system:0 2024-03-27T21:41:21.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.804+0000 7fe1f7fff640 20 client.4962.objecter put_session s=0x7fe2100aa590 osd=0 4 2024-03-27T21:41:21.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.804+0000 7fe1f7fff640 15 client.4962.objecter _session_command_op_remove 0 1 2024-03-27T21:41:21.808 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:21.822+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:21.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.832+0000 7fe215ddd640 10 client.4962.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:21.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.832+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign -1 2 2024-03-27T21:41:21.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.832+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe2100aa590 osd=0 3 2024-03-27T21:41:21.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.832+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe2100aa590 osd=0 3 2024-03-27T21:41:21.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.832+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe2100aa590 osd=0 3 2024-03-27T21:41:21.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.832+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_remove -1 2 2024-03-27T21:41:21.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.832+0000 7fe215ddd640 20 client.4962.objecter get_session s=0x7fe2100aa590 osd=0 3 2024-03-27T21:41:21.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.832+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign 0 2 2024-03-27T21:41:21.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.832+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe2100aa590 osd=0 4 2024-03-27T21:41:21.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.832+0000 7fe215ddd640 10 client.4962.objecter _send_command 2 2024-03-27T21:41:21.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.832+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe210069470 con 0x7fe210139af0 2024-03-27T21:41:21.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.835+0000 7fe1f7fff640 1 -- 172.21.15.60:0/3375579277 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe210069470 con 0x7fe210139af0 2024-03-27T21:41:21.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.835+0000 7fe1f7fff640 10 client.4962.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:21.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.835+0000 7fe1f7fff640 10 client.4962.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:21.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.835+0000 7fe1f7fff640 20 client.4962.objecter put_session s=0x7fe2100aa590 osd=0 4 2024-03-27T21:41:21.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.835+0000 7fe1f7fff640 15 client.4962.objecter _session_command_op_remove 0 2 2024-03-27T21:41:21.822 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:41:21.823 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:41:21.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.835+0000 7fe215ddd640 10 client.4962.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:21.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.835+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign -1 3 2024-03-27T21:41:21.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.835+0000 7fe215ddd640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fe210136200 0x7fe2101365e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:21.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.835+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe2100a9260 osd=1 3 2024-03-27T21:41:21.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.835+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe2100a9260 osd=1 3 2024-03-27T21:41:21.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.835+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe2100a9260 osd=1 3 2024-03-27T21:41:21.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.835+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_remove -1 3 2024-03-27T21:41:21.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.835+0000 7fe215ddd640 20 client.4962.objecter get_session s=0x7fe2100a9260 osd=1 3 2024-03-27T21:41:21.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.835+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign 1 3 2024-03-27T21:41:21.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.835+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe2100a9260 osd=1 4 2024-03-27T21:41:21.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.835+0000 7fe215ddd640 10 client.4962.objecter _send_command 3 2024-03-27T21:41:21.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.835+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe21013ab10 con 0x7fe210136200 2024-03-27T21:41:21.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.835+0000 7fe20f7fe640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fe210136200 0x7fe2101365e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:21.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.835+0000 7fe20f7fe640 10 monclient: get_auth_request con 0x7fe210136200 auth_method 0 2024-03-27T21:41:21.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.835+0000 7fe1f7fff640 10 client.4962.objecter ms_handle_connect 0x7fe210136200 2024-03-27T21:41:21.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.835+0000 7fe20f7fe640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fe210136200 0x7fe2101365e0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:21.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.836+0000 7fe1f7fff640 1 -- 172.21.15.60:0/3375579277 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe21013ab10 con 0x7fe210136200 2024-03-27T21:41:21.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.836+0000 7fe1f7fff640 10 client.4962.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:41:21.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.836+0000 7fe1f7fff640 10 client.4962.objecter _finish_command 3 = system:0 2024-03-27T21:41:21.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.836+0000 7fe1f7fff640 20 client.4962.objecter put_session s=0x7fe2100a9260 osd=1 4 2024-03-27T21:41:21.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.836+0000 7fe1f7fff640 15 client.4962.objecter _session_command_op_remove 1 3 2024-03-27T21:41:21.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.853+0000 7fe215ddd640 10 client.4962.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:21.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.853+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign -1 4 2024-03-27T21:41:21.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.853+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe2100a9260 osd=1 3 2024-03-27T21:41:21.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.853+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe2100a9260 osd=1 3 2024-03-27T21:41:21.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.853+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe2100a9260 osd=1 3 2024-03-27T21:41:21.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.853+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_remove -1 4 2024-03-27T21:41:21.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.853+0000 7fe215ddd640 20 client.4962.objecter get_session s=0x7fe2100a9260 osd=1 3 2024-03-27T21:41:21.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.853+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign 1 4 2024-03-27T21:41:21.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.853+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe2100a9260 osd=1 4 2024-03-27T21:41:21.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.853+0000 7fe215ddd640 10 client.4962.objecter _send_command 4 2024-03-27T21:41:21.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.853+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe21013ac80 con 0x7fe210136200 2024-03-27T21:41:21.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.855+0000 7fe1f7fff640 1 -- 172.21.15.60:0/3375579277 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe21013ac80 con 0x7fe210136200 2024-03-27T21:41:21.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.855+0000 7fe1f7fff640 10 client.4962.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:21.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.855+0000 7fe1f7fff640 10 client.4962.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:21.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.855+0000 7fe1f7fff640 20 client.4962.objecter put_session s=0x7fe2100a9260 osd=1 4 2024-03-27T21:41:21.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.855+0000 7fe1f7fff640 15 client.4962.objecter _session_command_op_remove 1 4 2024-03-27T21:41:21.842 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:41:21.844 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:41:21.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe215ddd640 10 client.4962.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:21.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign -1 5 2024-03-27T21:41:21.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe215ddd640 20 client.4962.objecter _get_session osd=-1 returning homeless 2024-03-27T21:41:21.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe215ddd640 10 client.4962.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:21.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe215ddd640 10 monclient: _renew_subs 2024-03-27T21:41:21.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe215ddd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:21.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=59}) v3 -- 0x7fe210136d60 con 0x7fe210071a50 2024-03-27T21:41:21.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe215ddd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:21.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe210137060 con 0x7fe210071a50 2024-03-27T21:41:21.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe1f7fff640 1 -- 172.21.15.60:0/3375579277 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_get_version_reply(handle=1 version=58) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe20005b670 con 0x7fe210071a50 2024-03-27T21:41:21.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe1f7fff640 10 monclient: handle_get_version_reply finishing 1 version 58 2024-03-27T21:41:21.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe20dffb640 10 client.4962.objecter _check_command_map_dne tid 5 current 58 map_dne_bound 58 2024-03-27T21:41:21.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe20dffb640 10 client.4962.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:41:21.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe20dffb640 15 client.4962.objecter _session_command_op_remove -1 5 2024-03-27T21:41:21.845 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:41:21.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe215ddd640 10 client.4962.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:21.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign -1 6 2024-03-27T21:41:21.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe215ddd640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fe21013bac0 0x7fe21002b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:21.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe21013b950 osd=3 3 2024-03-27T21:41:21.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe21013b950 osd=3 3 2024-03-27T21:41:21.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe21013b950 osd=3 3 2024-03-27T21:41:21.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_remove -1 6 2024-03-27T21:41:21.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe215ddd640 20 client.4962.objecter get_session s=0x7fe21013b950 osd=3 3 2024-03-27T21:41:21.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign 3 6 2024-03-27T21:41:21.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe21013b950 osd=3 4 2024-03-27T21:41:21.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe215ddd640 10 client.4962.objecter _send_command 6 2024-03-27T21:41:21.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe210133ae0 con 0x7fe21013bac0 2024-03-27T21:41:21.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe20effd640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fe21013bac0 0x7fe21002b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:21.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.856+0000 7fe20effd640 10 monclient: get_auth_request con 0x7fe21013bac0 auth_method 0 2024-03-27T21:41:21.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.857+0000 7fe1f7fff640 10 client.4962.objecter ms_handle_connect 0x7fe21013bac0 2024-03-27T21:41:21.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.857+0000 7fe20effd640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fe21013bac0 0x7fe21002b170 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:21.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.858+0000 7fe1f7fff640 1 -- 172.21.15.60:0/3375579277 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe210133ae0 con 0x7fe21013bac0 2024-03-27T21:41:21.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.858+0000 7fe1f7fff640 10 client.4962.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:41:21.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.858+0000 7fe1f7fff640 10 client.4962.objecter _finish_command 6 = system:0 2024-03-27T21:41:21.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.858+0000 7fe1f7fff640 20 client.4962.objecter put_session s=0x7fe21013b950 osd=3 4 2024-03-27T21:41:21.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.858+0000 7fe1f7fff640 15 client.4962.objecter _session_command_op_remove 3 6 2024-03-27T21:41:21.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.884+0000 7fe215ddd640 10 client.4962.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:21.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.884+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign -1 7 2024-03-27T21:41:21.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.884+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe21013b950 osd=3 3 2024-03-27T21:41:21.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.884+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe21013b950 osd=3 3 2024-03-27T21:41:21.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.884+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe21013b950 osd=3 3 2024-03-27T21:41:21.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.884+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_remove -1 7 2024-03-27T21:41:21.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.884+0000 7fe215ddd640 20 client.4962.objecter get_session s=0x7fe21013b950 osd=3 3 2024-03-27T21:41:21.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.884+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign 3 7 2024-03-27T21:41:21.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.884+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe21013b950 osd=3 4 2024-03-27T21:41:21.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.884+0000 7fe215ddd640 10 client.4962.objecter _send_command 7 2024-03-27T21:41:21.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.884+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe21002b6b0 con 0x7fe21013bac0 2024-03-27T21:41:21.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.886+0000 7fe1f7fff640 1 -- 172.21.15.60:0/3375579277 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe21002b6b0 con 0x7fe21013bac0 2024-03-27T21:41:21.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.886+0000 7fe1f7fff640 10 client.4962.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:21.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.886+0000 7fe1f7fff640 10 client.4962.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:21.872 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:41:21.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.886+0000 7fe1f7fff640 20 client.4962.objecter put_session s=0x7fe21013b950 osd=3 4 2024-03-27T21:41:21.873 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:41:21.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.886+0000 7fe1f7fff640 15 client.4962.objecter _session_command_op_remove 3 7 2024-03-27T21:41:21.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.886+0000 7fe215ddd640 10 client.4962.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:21.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.886+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign -1 8 2024-03-27T21:41:21.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.886+0000 7fe215ddd640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fe2101331c0 0x7fe2101e94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:21.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.886+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe210133050 osd=4 3 2024-03-27T21:41:21.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.886+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe210133050 osd=4 3 2024-03-27T21:41:21.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.886+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe210133050 osd=4 3 2024-03-27T21:41:21.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.886+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_remove -1 8 2024-03-27T21:41:21.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.886+0000 7fe215ddd640 20 client.4962.objecter get_session s=0x7fe210133050 osd=4 3 2024-03-27T21:41:21.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.886+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign 4 8 2024-03-27T21:41:21.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.886+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe210133050 osd=4 4 2024-03-27T21:41:21.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.886+0000 7fe215ddd640 10 client.4962.objecter _send_command 8 2024-03-27T21:41:21.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.886+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe21002c4f0 con 0x7fe2101331c0 2024-03-27T21:41:21.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.886+0000 7fe20ffff640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fe2101331c0 0x7fe2101e94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:21.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.886+0000 7fe20ffff640 10 monclient: get_auth_request con 0x7fe2101331c0 auth_method 0 2024-03-27T21:41:21.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.887+0000 7fe1f7fff640 10 client.4962.objecter ms_handle_connect 0x7fe2101331c0 2024-03-27T21:41:21.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.887+0000 7fe20ffff640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fe2101331c0 0x7fe2101e94b0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:21.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.888+0000 7fe1f7fff640 1 -- 172.21.15.60:0/3375579277 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe21002c4f0 con 0x7fe2101331c0 2024-03-27T21:41:21.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.888+0000 7fe1f7fff640 10 client.4962.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:41:21.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.888+0000 7fe1f7fff640 10 client.4962.objecter _finish_command 8 = system:0 2024-03-27T21:41:21.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.888+0000 7fe1f7fff640 20 client.4962.objecter put_session s=0x7fe210133050 osd=4 4 2024-03-27T21:41:21.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.888+0000 7fe1f7fff640 15 client.4962.objecter _session_command_op_remove 4 8 2024-03-27T21:41:21.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.901+0000 7fe215ddd640 10 client.4962.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:21.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.901+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign -1 9 2024-03-27T21:41:21.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.901+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe210133050 osd=4 3 2024-03-27T21:41:21.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.901+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe210133050 osd=4 3 2024-03-27T21:41:21.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.901+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe210133050 osd=4 3 2024-03-27T21:41:21.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.901+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_remove -1 9 2024-03-27T21:41:21.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.902+0000 7fe215ddd640 20 client.4962.objecter get_session s=0x7fe210133050 osd=4 3 2024-03-27T21:41:21.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.902+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign 4 9 2024-03-27T21:41:21.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.902+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe210133050 osd=4 4 2024-03-27T21:41:21.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.902+0000 7fe215ddd640 10 client.4962.objecter _send_command 9 2024-03-27T21:41:21.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.902+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe21002c690 con 0x7fe2101331c0 2024-03-27T21:41:21.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.904+0000 7fe1f7fff640 1 -- 172.21.15.60:0/3375579277 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe21002c690 con 0x7fe2101331c0 2024-03-27T21:41:21.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.904+0000 7fe1f7fff640 10 client.4962.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:21.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.904+0000 7fe1f7fff640 10 client.4962.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:21.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.904+0000 7fe1f7fff640 20 client.4962.objecter put_session s=0x7fe210133050 osd=4 4 2024-03-27T21:41:21.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.904+0000 7fe1f7fff640 15 client.4962.objecter _session_command_op_remove 4 9 2024-03-27T21:41:21.891 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:41:21.892 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:41:21.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.904+0000 7fe215ddd640 10 client.4962.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:21.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.904+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign -1 10 2024-03-27T21:41:21.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.904+0000 7fe215ddd640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fe210066250 0x7fe210066690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:21.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.904+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe21002c690 osd=5 3 2024-03-27T21:41:21.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.904+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe21002c690 osd=5 3 2024-03-27T21:41:21.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.904+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe21002c690 osd=5 3 2024-03-27T21:41:21.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.904+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_remove -1 10 2024-03-27T21:41:21.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.904+0000 7fe215ddd640 20 client.4962.objecter get_session s=0x7fe21002c690 osd=5 3 2024-03-27T21:41:21.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.904+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign 5 10 2024-03-27T21:41:21.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.904+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe21002c690 osd=5 4 2024-03-27T21:41:21.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.904+0000 7fe215ddd640 10 client.4962.objecter _send_command 10 2024-03-27T21:41:21.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.904+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2101ef410 con 0x7fe210066250 2024-03-27T21:41:21.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.905+0000 7fe20f7fe640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fe210066250 0x7fe210066690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:21.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.905+0000 7fe20f7fe640 10 monclient: get_auth_request con 0x7fe210066250 auth_method 0 2024-03-27T21:41:21.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.905+0000 7fe1f7fff640 10 client.4962.objecter ms_handle_connect 0x7fe210066250 2024-03-27T21:41:21.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.905+0000 7fe20f7fe640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fe210066250 0x7fe210066690 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:21.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.906+0000 7fe1f7fff640 1 -- 172.21.15.60:0/3375579277 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe2101ef410 con 0x7fe210066250 2024-03-27T21:41:21.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.906+0000 7fe1f7fff640 10 client.4962.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:41:21.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.906+0000 7fe1f7fff640 10 client.4962.objecter _finish_command 10 = system:0 2024-03-27T21:41:21.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.906+0000 7fe1f7fff640 20 client.4962.objecter put_session s=0x7fe21002c690 osd=5 4 2024-03-27T21:41:21.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.906+0000 7fe1f7fff640 15 client.4962.objecter _session_command_op_remove 5 10 2024-03-27T21:41:21.898 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:21.898 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.920+0000 7fe215ddd640 10 client.4962.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.920+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign -1 11 2024-03-27T21:41:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.920+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe21002c690 osd=5 3 2024-03-27T21:41:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.920+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe21002c690 osd=5 3 2024-03-27T21:41:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.920+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe21002c690 osd=5 3 2024-03-27T21:41:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.920+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_remove -1 11 2024-03-27T21:41:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.920+0000 7fe215ddd640 20 client.4962.objecter get_session s=0x7fe21002c690 osd=5 3 2024-03-27T21:41:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.920+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign 5 11 2024-03-27T21:41:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.920+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe21002c690 osd=5 4 2024-03-27T21:41:21.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.920+0000 7fe215ddd640 10 client.4962.objecter _send_command 11 2024-03-27T21:41:21.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.920+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe2101ecb80 con 0x7fe210066250 2024-03-27T21:41:21.908 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:41:21.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.921+0000 7fe1f7fff640 1 -- 172.21.15.60:0/3375579277 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe2101ecb80 con 0x7fe210066250 2024-03-27T21:41:21.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.921+0000 7fe1f7fff640 10 client.4962.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:21.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.921+0000 7fe1f7fff640 10 client.4962.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:21.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.921+0000 7fe1f7fff640 20 client.4962.objecter put_session s=0x7fe21002c690 osd=5 4 2024-03-27T21:41:21.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.921+0000 7fe1f7fff640 15 client.4962.objecter _session_command_op_remove 5 11 2024-03-27T21:41:21.909 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:41:21.910 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:41:21.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.921+0000 7fe215ddd640 10 client.4962.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:21.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.921+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign -1 12 2024-03-27T21:41:21.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.921+0000 7fe215ddd640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fe2101ed260 0x7fe2101ef650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:21.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.922+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe2101ecb80 osd=6 3 2024-03-27T21:41:21.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.922+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe2101ecb80 osd=6 3 2024-03-27T21:41:21.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.922+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe2101ecb80 osd=6 3 2024-03-27T21:41:21.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.922+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_remove -1 12 2024-03-27T21:41:21.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.922+0000 7fe215ddd640 20 client.4962.objecter get_session s=0x7fe2101ecb80 osd=6 3 2024-03-27T21:41:21.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.922+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign 6 12 2024-03-27T21:41:21.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.922+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe2101ecb80 osd=6 4 2024-03-27T21:41:21.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.922+0000 7fe215ddd640 10 client.4962.objecter _send_command 12 2024-03-27T21:41:21.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.922+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2101f2420 con 0x7fe2101ed260 2024-03-27T21:41:21.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.922+0000 7fe20effd640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fe2101ed260 0x7fe2101ef650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:21.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.922+0000 7fe20effd640 10 monclient: get_auth_request con 0x7fe2101ed260 auth_method 0 2024-03-27T21:41:21.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.922+0000 7fe1f7fff640 10 client.4962.objecter ms_handle_connect 0x7fe2101ed260 2024-03-27T21:41:21.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.922+0000 7fe20effd640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fe2101ed260 0x7fe2101ef650 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:21.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.923+0000 7fe1f7fff640 1 -- 172.21.15.60:0/3375579277 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe2101f2420 con 0x7fe2101ed260 2024-03-27T21:41:21.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.923+0000 7fe1f7fff640 10 client.4962.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:41:21.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.923+0000 7fe1f7fff640 10 client.4962.objecter _finish_command 12 = system:0 2024-03-27T21:41:21.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.923+0000 7fe1f7fff640 20 client.4962.objecter put_session s=0x7fe2101ecb80 osd=6 4 2024-03-27T21:41:21.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.923+0000 7fe1f7fff640 15 client.4962.objecter _session_command_op_remove 6 12 2024-03-27T21:41:21.912 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:21.922+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:21.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.937+0000 7fe215ddd640 10 client.4962.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:21.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.937+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign -1 13 2024-03-27T21:41:21.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.937+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe2101ecb80 osd=6 3 2024-03-27T21:41:21.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.937+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe2101ecb80 osd=6 3 2024-03-27T21:41:21.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.937+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe2101ecb80 osd=6 3 2024-03-27T21:41:21.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.937+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_remove -1 13 2024-03-27T21:41:21.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.937+0000 7fe215ddd640 20 client.4962.objecter get_session s=0x7fe2101ecb80 osd=6 3 2024-03-27T21:41:21.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.937+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign 6 13 2024-03-27T21:41:21.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.937+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe2101ecb80 osd=6 4 2024-03-27T21:41:21.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.937+0000 7fe215ddd640 10 client.4962.objecter _send_command 13 2024-03-27T21:41:21.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.937+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe2101efb90 con 0x7fe2101ed260 2024-03-27T21:41:21.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.938+0000 7fe1f7fff640 1 -- 172.21.15.60:0/3375579277 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe2101efb90 con 0x7fe2101ed260 2024-03-27T21:41:21.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.938+0000 7fe1f7fff640 10 client.4962.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:21.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.938+0000 7fe1f7fff640 10 client.4962.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:21.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.938+0000 7fe1f7fff640 20 client.4962.objecter put_session s=0x7fe2101ecb80 osd=6 4 2024-03-27T21:41:21.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.938+0000 7fe1f7fff640 15 client.4962.objecter _session_command_op_remove 6 13 2024-03-27T21:41:21.925 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:41:21.926 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:41:21.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.939+0000 7fe215ddd640 10 client.4962.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:21.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.939+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign -1 14 2024-03-27T21:41:21.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.939+0000 7fe215ddd640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fe2101f0210 0x7fe2101f0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:21.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.939+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe2101efb90 osd=7 3 2024-03-27T21:41:21.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.939+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe2101efb90 osd=7 3 2024-03-27T21:41:21.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.939+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe2101efb90 osd=7 3 2024-03-27T21:41:21.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.939+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_remove -1 14 2024-03-27T21:41:21.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.939+0000 7fe215ddd640 20 client.4962.objecter get_session s=0x7fe2101efb90 osd=7 3 2024-03-27T21:41:21.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.939+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign 7 14 2024-03-27T21:41:21.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.939+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe2101efb90 osd=7 4 2024-03-27T21:41:21.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.939+0000 7fe215ddd640 10 client.4962.objecter _send_command 14 2024-03-27T21:41:21.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.939+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2101f57a0 con 0x7fe2101f0210 2024-03-27T21:41:21.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.939+0000 7fe20ffff640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fe2101f0210 0x7fe2101f0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:21.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.939+0000 7fe20ffff640 10 monclient: get_auth_request con 0x7fe2101f0210 auth_method 0 2024-03-27T21:41:21.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.939+0000 7fe1f7fff640 10 client.4962.objecter ms_handle_connect 0x7fe2101f0210 2024-03-27T21:41:21.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.939+0000 7fe20ffff640 1 --2- 172.21.15.60:0/3375579277 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fe2101f0210 0x7fe2101f0650 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:21.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.940+0000 7fe1f7fff640 1 -- 172.21.15.60:0/3375579277 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe2101f57a0 con 0x7fe2101f0210 2024-03-27T21:41:21.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.940+0000 7fe1f7fff640 10 client.4962.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:41:21.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.940+0000 7fe1f7fff640 10 client.4962.objecter _finish_command 14 = system:0 2024-03-27T21:41:21.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.940+0000 7fe1f7fff640 20 client.4962.objecter put_session s=0x7fe2101efb90 osd=7 4 2024-03-27T21:41:21.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.940+0000 7fe1f7fff640 15 client.4962.objecter _session_command_op_remove 7 14 2024-03-27T21:41:21.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.954+0000 7fe215ddd640 10 client.4962.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:21.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.954+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign -1 15 2024-03-27T21:41:21.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.954+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe2101efb90 osd=7 3 2024-03-27T21:41:21.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.954+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe2101efb90 osd=7 3 2024-03-27T21:41:21.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.954+0000 7fe215ddd640 20 client.4962.objecter _get_session s=0x7fe2101efb90 osd=7 3 2024-03-27T21:41:21.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.954+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_remove -1 15 2024-03-27T21:41:21.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.954+0000 7fe215ddd640 20 client.4962.objecter get_session s=0x7fe2101efb90 osd=7 3 2024-03-27T21:41:21.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.954+0000 7fe215ddd640 15 client.4962.objecter _session_command_op_assign 7 15 2024-03-27T21:41:21.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.954+0000 7fe215ddd640 20 client.4962.objecter put_session s=0x7fe2101efb90 osd=7 4 2024-03-27T21:41:21.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.954+0000 7fe215ddd640 10 client.4962.objecter _send_command 15 2024-03-27T21:41:21.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.954+0000 7fe215ddd640 1 -- 172.21.15.60:0/3375579277 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe2101f0ca0 con 0x7fe2101f0210 2024-03-27T21:41:21.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.955+0000 7fe1f7fff640 1 -- 172.21.15.60:0/3375579277 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fe2101f0ca0 con 0x7fe2101f0210 2024-03-27T21:41:21.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.955+0000 7fe1f7fff640 10 client.4962.objecter ms_dispatch 0x7fe210000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:21.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.955+0000 7fe1f7fff640 10 client.4962.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:21.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.955+0000 7fe1f7fff640 20 client.4962.objecter put_session s=0x7fe2101efb90 osd=7 4 2024-03-27T21:41:21.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:21.955+0000 7fe1f7fff640 15 client.4962.objecter _session_command_op_remove 7 15 2024-03-27T21:41:21.943 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:41:21.943 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:41:21.945 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:41:21.946 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:41:22.010 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:22.024+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:22.017 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:22.017 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:22.023 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-27T21:41:22.027 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:41:22.111 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:22.125+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:22.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.150+0000 7f5a51009640 10 monclient: get_monmap_and_config 2024-03-27T21:41:22.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.150+0000 7f5a51009640 10 monclient: build_initial_monmap 2024-03-27T21:41:22.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.150+0000 7f5a51009640 10 monclient: monmap: 2024-03-27T21:41:22.137 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:22.137 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:22.137 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:22.151811+0000 2024-03-27T21:41:22.138 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:22.151811+0000 2024-03-27T21:41:22.138 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:22.138 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:22.138 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:22.138 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:22.138 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:22.139 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:22.139 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:22.139 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:22.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.151+0000 7f5a51009640 1 Processor -- start 2024-03-27T21:41:22.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.151+0000 7f5a51009640 1 -- start start 2024-03-27T21:41:22.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.151+0000 7f5a51009640 10 monclient: init 2024-03-27T21:41:22.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.152+0000 7f5a51009640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:22.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.152+0000 7f5a51009640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-03-27T21:41:22.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.152+0000 7f5a51009640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5a4c0b40b0 0x7f5a4c0a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:22.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.152+0000 7f5a51009640 10 monclient(hunting): picked mon.noname-c con 0x7f5a4c0b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:22.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.152+0000 7f5a51009640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5a4c13b020 0x7f5a4c13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:22.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.152+0000 7f5a51009640 10 monclient(hunting): picked mon.noname-a con 0x7f5a4c13b020 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:22.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.152+0000 7f5a51009640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5a4c1416f0 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:22.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.152+0000 7f5a51009640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4c0b3830 con 0x7f5a4c1416f0 2024-03-27T21:41:22.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.152+0000 7f5a51009640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:22.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.152+0000 7f5a51009640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a4c0b4490 con 0x7f5a4c13b020 2024-03-27T21:41:22.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.152+0000 7f5a51009640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:22.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.152+0000 7f5a51009640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a4c0aa520 con 0x7f5a4c0b40b0 2024-03-27T21:41:22.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.152+0000 7f5a51009640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:22.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.152+0000 7f5a51009640 10 monclient(hunting): authenticate will time out at 923.373690s 2024-03-27T21:41:22.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.152+0000 7f5a4a59c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5a4c13b020 0x7f5a4c13b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:22.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4ad9d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5a4c0b40b0 0x7f5a4c0a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:22.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4a59c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5a4c13b020 0x7f5a4c13b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36422/0 (socket says 172.21.15.60:36422) 2024-03-27T21:41:22.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4ad9d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5a4c0b40b0 0x7f5a4c0a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:59192/0 (socket says 172.21.15.60:59192) 2024-03-27T21:41:22.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4a59c640 1 -- 172.21.15.60:0/2326827780 learned_addr learned my addr 172.21.15.60:0/2326827780 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:22.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4ad9d640 10 monclient(hunting): get_auth_request con 0x7f5a4c0b40b0 auth_method 0 2024-03-27T21:41:22.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:22.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:22.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:22.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4b59e640 1 --1- 172.21.15.60:0/2326827780 >> v1:172.21.15.191:6789/0 conn(0x7f5a4c1416f0 0x7f5a4c141b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.60:44348/0 (socket says 172.21.15.60:44348) 2024-03-27T21:41:22.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4a59c640 10 monclient(hunting): get_auth_request con 0x7f5a4c13b020 auth_method 0 2024-03-27T21:41:22.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:22.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4a59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:22.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:22.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:22.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:22.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:22.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:22.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:22.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:22.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4ad9d640 10 monclient(hunting): handle_auth_done global_id 4903 payload 274 2024-03-27T21:41:22.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4ad9d640 1 -- 172.21.15.60:0/2326827780 >> v2:172.21.15.60:3300/0 conn(0x7f5a4c13b020 msgr2=0x7f5a4c13b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:22.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4ad9d640 1 --2- 172.21.15.60:0/2326827780 >> v2:172.21.15.60:3300/0 conn(0x7f5a4c13b020 0x7f5a4c13b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:22.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4ad9d640 1 -- 172.21.15.60:0/2326827780 >> v1:172.21.15.191:6789/0 conn(0x7f5a4c1416f0 legacy=0x7f5a4c141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:22.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a49d9b640 1 -- 172.21.15.60:0/2326827780 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463779376 0 0) 0x7f5a4c0b3830 con 0x7f5a4c1416f0 2024-03-27T21:41:22.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4ad9d640 10 monclient: _finish_hunting 0 2024-03-27T21:41:22.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4ad9d640 1 monclient: found mon.noname-c 2024-03-27T21:41:22.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:22.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:41:22.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.153+0000 7f5a4ad9d640 1 -- 172.21.15.60:0/2326827780 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4c0a9470 con 0x7f5a4c0b40b0 2024-03-27T21:41:22.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.154+0000 7f5a49d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:41:22.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.154+0000 7f5a4ad9d640 1 --2- 172.21.15.60:0/2326827780 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a4c0b40b0 0x7f5a4c0a8b90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c0b4810 tx=0x7f5a3c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ab5add7107a9861 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:22.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.154+0000 7f5a49d9b640 1 -- 172.21.15.60:0/2326827780 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c031830 con 0x7f5a4c0b40b0 2024-03-27T21:41:22.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.154+0000 7f5a49d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:22.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.154+0000 7f5a49d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:22.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.154+0000 7f5a49d9b640 10 monclient: dump: 2024-03-27T21:41:22.167 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:22.168 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:22.168 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:22.168 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:22.168 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:22.168 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:22.168 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:22.168 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:22.168 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:22.169 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:22.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.154+0000 7f5a49d9b640 10 monclient: _finish_auth 0 2024-03-27T21:41:22.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.154+0000 7f5a49d9b640 10 monclient: _check_auth_tickets 2024-03-27T21:41:22.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.154+0000 7f5a49d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:22.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.154+0000 7f5a49d9b640 1 -- 172.21.15.60:0/2326827780 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3c031990 con 0x7f5a4c0b40b0 2024-03-27T21:41:22.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.154+0000 7f5a51009640 5 monclient: authenticate success, global_id 4903 2024-03-27T21:41:22.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.154+0000 7f5a51009640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:22.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.154+0000 7f5a49d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:22.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.154+0000 7f5a51009640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:22.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.154+0000 7f5a49d9b640 1 -- 172.21.15.60:0/2326827780 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c031d20 con 0x7f5a4c0b40b0 2024-03-27T21:41:22.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.154+0000 7f5a51009640 4 set_mon_vals no callback set 2024-03-27T21:41:22.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.154+0000 7f5a49d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:22.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.154+0000 7f5a49d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:22.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.154+0000 7f5a49d9b640 10 monclient: dump: 2024-03-27T21:41:22.170 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:22.170 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:22.170 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:22.170 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:22.170 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:22.170 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:22.170 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:22.170 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:22.170 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:22.170 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:22.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.155+0000 7f5a51009640 10 monclient: shutdown 2024-03-27T21:41:22.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.155+0000 7f5a51009640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.155+0000 7f5a51009640 1 -- 172.21.15.60:0/2326827780 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a4c0b40b0 msgr2=0x7f5a4c0a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.155+0000 7f5a51009640 1 --2- 172.21.15.60:0/2326827780 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a4c0b40b0 0x7f5a4c0a8b90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c0b4810 tx=0x7f5a3c02b8d0 comp rx=0 tx=0).stop 2024-03-27T21:41:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.155+0000 7f5a51009640 1 -- 172.21.15.60:0/2326827780 shutdown_connections 2024-03-27T21:41:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.155+0000 7f5a51009640 1 --2- 172.21.15.60:0/2326827780 >> v2:172.21.15.60:3300/0 conn(0x7f5a4c13b020 0x7f5a4c13b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.155+0000 7f5a51009640 1 --2- 172.21.15.60:0/2326827780 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a4c0b40b0 0x7f5a4c0a8b90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.155+0000 7f5a51009640 1 -- 172.21.15.60:0/2326827780 >> 172.21.15.60:0/2326827780 conn(0x7f5a4c0608e0 msgr2=0x7f5a4c060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.155+0000 7f5a51009640 1 -- 172.21.15.60:0/2326827780 shutdown_connections 2024-03-27T21:41:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.155+0000 7f5a51009640 1 -- 172.21.15.60:0/2326827780 wait complete. 2024-03-27T21:41:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.156+0000 7f5a51009640 10 monclient: build_initial_monmap 2024-03-27T21:41:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.156+0000 7f5a51009640 10 monclient: monmap: 2024-03-27T21:41:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:22.172 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.157+0000 7f5a51009640 1 librados: starting msgr at 2024-03-27T21:41:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.157+0000 7f5a51009640 1 librados: starting objecter 2024-03-27T21:41:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.157+0000 7f5a51009640 1 Processor -- start 2024-03-27T21:41:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.157+0000 7f5a51009640 1 -- start start 2024-03-27T21:41:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.157+0000 7f5a51009640 1 librados: setting wanted keys 2024-03-27T21:41:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.157+0000 7f5a51009640 1 librados: calling monclient init 2024-03-27T21:41:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.157+0000 7f5a51009640 10 monclient: init 2024-03-27T21:41:22.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a51009640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:22.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a51009640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:41:22.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a51009640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a4c0b40b0 0x7f5a4c13e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:22.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a51009640 10 monclient(hunting): picked mon.noname-b con 0x7f5a4c0b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:22.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a51009640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a4c13b020 0x7f5a4c13ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:22.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a51009640 10 monclient(hunting): picked mon.noname-a con 0x7f5a4c13b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:22.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a51009640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5a4c1416f0 0x7f5a4c1d6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:22.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a51009640 10 monclient(hunting): picked mon.noname-c con 0x7f5a4c1416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:22.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a51009640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:22.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a51009640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c0abc70 con 0x7f5a4c13b020 2024-03-27T21:41:22.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a51009640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:22.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a51009640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c0abf50 con 0x7f5a4c0b40b0 2024-03-27T21:41:22.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a51009640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:22.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a51009640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c0ac230 con 0x7f5a4c1416f0 2024-03-27T21:41:22.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a51009640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:22.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a51009640 10 monclient(hunting): authenticate will time out at 923.379637s 2024-03-27T21:41:22.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a4a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a4c13b020 0x7f5a4c13ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:22.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a4ad9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a4c0b40b0 0x7f5a4c13e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:22.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a4a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a4c13b020 0x7f5a4c13ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36428/0 (socket says 172.21.15.60:36428) 2024-03-27T21:41:22.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a4ad9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a4c0b40b0 0x7f5a4c13e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:59194/0 (socket says 172.21.15.60:59194) 2024-03-27T21:41:22.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a4a59c640 1 -- 172.21.15.60:0/139960226 learned_addr learned my addr 172.21.15.60:0/139960226 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:22.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a4a59c640 10 monclient(hunting): get_auth_request con 0x7f5a4c13b020 auth_method 0 2024-03-27T21:41:22.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:22.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a4a59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:22.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:22.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a4b59e640 1 --2- 172.21.15.60:0/139960226 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5a4c1416f0 0x7f5a4c1d6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:22.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a4ad9d640 10 monclient(hunting): get_auth_request con 0x7f5a4c0b40b0 auth_method 0 2024-03-27T21:41:22.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:22.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a4ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:22.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:22.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a4b59e640 10 monclient(hunting): get_auth_request con 0x7f5a4c1416f0 auth_method 0 2024-03-27T21:41:22.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a4b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:22.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a4b59e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:22.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.158+0000 7f5a4b59e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:22.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a4ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:22.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a4ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:22.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a4ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:22.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:22.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:22.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:22.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a4b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:22.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a4b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:22.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a4b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:22.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a4ad9d640 10 monclient(hunting): handle_auth_done global_id 4909 payload 824 2024-03-27T21:41:22.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a4ad9d640 1 -- 172.21.15.60:0/139960226 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5a4c1416f0 msgr2=0x7f5a4c1d6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:22.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a4ad9d640 1 --2- 172.21.15.60:0/139960226 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5a4c1416f0 0x7f5a4c1d6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:22.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a4ad9d640 1 -- 172.21.15.60:0/139960226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a4c13b020 msgr2=0x7f5a4c13ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:22.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a4ad9d640 1 --2- 172.21.15.60:0/139960226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a4c13b020 0x7f5a4c13ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:22.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a4ad9d640 10 monclient: _finish_hunting 0 2024-03-27T21:41:22.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a4ad9d640 1 monclient: found mon.noname-b 2024-03-27T21:41:22.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a4ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:22.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a4ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:41:22.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a4ad9d640 1 -- 172.21.15.60:0/139960226 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3c009400 con 0x7f5a4c0b40b0 2024-03-27T21:41:22.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a4ad9d640 1 --2- 172.21.15.60:0/139960226 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a4c0b40b0 0x7f5a4c13e1d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c033000 tx=0x7f5a3c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:22.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a337fe640 10 client.?.objecter ms_handle_connect 0x7f5a4c0b40b0 2024-03-27T21:41:22.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a337fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:22.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a337fe640 1 -- 172.21.15.60:0/139960226 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c044d20 con 0x7f5a4c0b40b0 2024-03-27T21:41:22.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:22.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:22.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a337fe640 10 monclient: dump: 2024-03-27T21:41:22.177 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:22.177 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:22.177 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:22.177 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:22.177 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:22.178 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:22.178 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:22.178 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:22.178 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:22.178 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:22.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a337fe640 10 monclient: _finish_auth 0 2024-03-27T21:41:22.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a337fe640 10 monclient: _check_auth_tickets 2024-03-27T21:41:22.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:22.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a51009640 5 monclient: authenticate success, global_id 4909 2024-03-27T21:41:22.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a337fe640 1 -- 172.21.15.60:0/139960226 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3c048070 con 0x7f5a4c0b40b0 2024-03-27T21:41:22.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:22.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.159+0000 7f5a4959a640 4 set_mon_vals no callback set 2024-03-27T21:41:22.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.160+0000 7f5a51009640 10 monclient: _renew_subs 2024-03-27T21:41:22.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.160+0000 7f5a337fe640 1 -- 172.21.15.60:0/139960226 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c044ea0 con 0x7f5a4c0b40b0 2024-03-27T21:41:22.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.160+0000 7f5a51009640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:22.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.160+0000 7f5a51009640 1 -- 172.21.15.60:0/139960226 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a4c1d6100 con 0x7f5a4c0b40b0 2024-03-27T21:41:22.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.160+0000 7f5a337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:22.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.160+0000 7f5a337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:22.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.160+0000 7f5a337fe640 10 monclient: dump: 2024-03-27T21:41:22.179 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:22.179 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:22.179 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:22.179 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:22.179 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:22.179 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:22.179 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:22.179 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:22.179 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:22.179 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:22.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.160+0000 7f5a51009640 10 client.4909.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:22.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.160+0000 7f5a51009640 10 monclient: _renew_subs 2024-03-27T21:41:22.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.160+0000 7f5a51009640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:22.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.160+0000 7f5a51009640 1 -- 172.21.15.60:0/139960226 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4c0aaeb0 con 0x7f5a4c0b40b0 2024-03-27T21:41:22.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.160+0000 7f5a51009640 1 librados: init done 2024-03-27T21:41:22.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.161+0000 7f5a51009640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:22.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.161+0000 7f5a51009640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:22.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.161+0000 7f5a51009640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:22.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.161+0000 7f5a51009640 1 -- 172.21.15.60:0/139960226 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a4c1d6620 con 0x7f5a4c0b40b0 2024-03-27T21:41:22.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.162+0000 7f5a337fe640 1 -- 172.21.15.60:0/139960226 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5a3c03a420 con 0x7f5a4c0b40b0 2024-03-27T21:41:22.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.162+0000 7f5a337fe640 1 --2- 172.21.15.60:0/139960226 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5a3806c590 0x7f5a3806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:22.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.162+0000 7f5a337fe640 1 -- 172.21.15.60:0/139960226 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f5a3c04a070 con 0x7f5a4c0b40b0 2024-03-27T21:41:22.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.162+0000 7f5a337fe640 10 client.4909.objecter ms_dispatch 0x7f5a4c000b90 osd_map(58..58 src has 1..58) v4 2024-03-27T21:41:22.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.162+0000 7f5a337fe640 3 client.4909.objecter handle_osd_map got epochs [58,58] > 0 2024-03-27T21:41:22.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.162+0000 7f5a337fe640 3 client.4909.objecter handle_osd_map decoding full epoch 58 2024-03-27T21:41:22.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.163+0000 7f5a337fe640 20 client.4909.objecter dump_active .. 0 homeless 2024-03-27T21:41:22.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.163+0000 7f5a4a59c640 1 --2- 172.21.15.60:0/139960226 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5a3806c590 0x7f5a3806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:22.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.163+0000 7f5a4a59c640 10 monclient: get_auth_request con 0x7f5a3806c590 auth_method 0 2024-03-27T21:41:22.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.163+0000 7f5a337fe640 10 client.4909.objecter ms_handle_connect 0x7f5a3806c590 2024-03-27T21:41:22.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.163+0000 7f5a4a59c640 1 --2- 172.21.15.60:0/139960226 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5a3806c590 0x7f5a3806ea50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c0b16d0 tx=0x7f5a40009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:22.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.167+0000 7f5a337fe640 1 -- 172.21.15.60:0/139960226 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5a4c1d6620 con 0x7f5a4c0b40b0 2024-03-27T21:41:22.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.167+0000 7f5a337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:22.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.167+0000 7f5a337fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:22.182 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:22.182 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:22.183 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:41:22.224 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:22.238+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:22.312 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:22.326+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:22.328 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:22.328 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:22.337 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:41:22.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.358+0000 7f5a51009640 10 start_mon_command cmd=[{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:41:22.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.358+0000 7f5a51009640 10 monclient: _send_command 2 [{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:41:22.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.358+0000 7f5a51009640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:22.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:22.358+0000 7f5a51009640 1 -- 172.21.15.60:0/139960226 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f5a4c0ac090 con 0x7f5a4c0b40b0 2024-03-27T21:41:22.414 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:22.427+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:22.443 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:22.443 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:22.451 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:41:22.535 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:22.549+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:22.596 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:22.597 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:22.605 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:41:22.646 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:22.660+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:22.716 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:22.730+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:22.750 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:22.750 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:22.758 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:41:22.817 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:22.831+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:22.863 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:22.863 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:22.872 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:41:22.917 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:22.931+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:22.976 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:22.976 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:22.984 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:41:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.033+0000 7f5a337fe640 1 -- 172.21.15.60:0/139960226 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v59) v1 ==== 91+0+0 (secure 0 0 0) 0x7f5a4c0ac090 con 0x7f5a4c0b40b0 2024-03-27T21:41:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.033+0000 7f5a337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:41:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.033+0000 7f5a337fe640 10 monclient: _finish_command 2 = system:0 noscrub is unset 2024-03-27T21:41:23.020 INFO:teuthology.orchestra.run.smithi060.stderr:noscrub is unset 2024-03-27T21:41:23.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.035+0000 7f5a51009640 10 librados: watch_flush enter 2024-03-27T21:41:23.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.035+0000 7f5a51009640 10 librados: watch_flush exit 2024-03-27T21:41:23.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.035+0000 7f5a51009640 20 client.4909.objecter shutdown clearing up homeless session... 2024-03-27T21:41:23.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.035+0000 7f5a51009640 10 client.4909.objecter successfully canceled tick 2024-03-27T21:41:23.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.035+0000 7f5a51009640 1 -- 172.21.15.60:0/139960226 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5a3806c590 msgr2=0x7f5a3806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:23.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.035+0000 7f5a51009640 1 --2- 172.21.15.60:0/139960226 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5a3806c590 0x7f5a3806ea50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c0b16d0 tx=0x7f5a40009240 comp rx=0 tx=0).stop 2024-03-27T21:41:23.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.035+0000 7f5a51009640 10 monclient: shutdown 2024-03-27T21:41:23.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.035+0000 7f5a51009640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:23.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.035+0000 7f5a51009640 1 -- 172.21.15.60:0/139960226 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a4c0b40b0 msgr2=0x7f5a4c13e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.035+0000 7f5a51009640 1 --2- 172.21.15.60:0/139960226 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a4c0b40b0 0x7f5a4c13e1d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c033000 tx=0x7f5a3c0046c0 comp rx=0 tx=0).stop 2024-03-27T21:41:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.035+0000 7f5a51009640 1 -- 172.21.15.60:0/139960226 shutdown_connections 2024-03-27T21:41:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.035+0000 7f5a51009640 1 --2- 172.21.15.60:0/139960226 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5a4c1416f0 0x7f5a4c1d6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.036+0000 7f5a51009640 1 --2- 172.21.15.60:0/139960226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a4c13b020 0x7f5a4c13ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.036+0000 7f5a51009640 1 --2- 172.21.15.60:0/139960226 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5a3806c590 0x7f5a3806ea50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.036+0000 7f5a51009640 1 --2- 172.21.15.60:0/139960226 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a4c0b40b0 0x7f5a4c13e1d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.036+0000 7f5a51009640 1 -- 172.21.15.60:0/139960226 >> 172.21.15.60:0/139960226 conn(0x7f5a4c0608e0 msgr2=0x7f5a4c0b1950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.036+0000 7f5a51009640 1 -- 172.21.15.60:0/139960226 shutdown_connections 2024-03-27T21:41:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.036+0000 7f5a51009640 1 -- 172.21.15.60:0/139960226 wait complete. 2024-03-27T21:41:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:23.036+0000 7f5a51009640 1 librados: shutdown 2024-03-27T21:41:23.025 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:23.039+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:23.119 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:23.133+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:23.130 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:23.130 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:23.139 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:41:23.220 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:23.234+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:23.247 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:23.247 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:23.256 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:41:23.322 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:23.335+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:23.365 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:23.366 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:23.373 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:41:23.423 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:23.437+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:23.484 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:23.484 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:23.493 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:41:23.524 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:23.538+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:23.625 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:23.638+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:23.643 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:23.643 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:23.652 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:41:23.726 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:23.740+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:23.760 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:23.760 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:23.769 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:41:23.827 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:23.840+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:23.927 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:23.941+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:23.930 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:23.930 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:23.943 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:41:23.946 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:41:24.021 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-27T21:41:24.021 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-03-27T21:41:24.021 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-27T21:41:24.021 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-27T21:41:24.028 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:24.042+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:24.032 INFO:tasks.ceph.osd.2:Started 2024-03-27T21:41:24.032 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-27T21:41:24.094 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:24.095 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:24.109 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:41:24.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.138+0000 7fa1d2a37640 10 monclient: get_monmap_and_config 2024-03-27T21:41:24.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.138+0000 7fa1d2a37640 10 monclient: build_initial_monmap 2024-03-27T21:41:24.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.138+0000 7fa1d2a37640 10 monclient: monmap: 2024-03-27T21:41:24.127 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:24.127 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:24.127 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:24.139943+0000 2024-03-27T21:41:24.127 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:24.139943+0000 2024-03-27T21:41:24.127 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:24.127 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:24.127 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:24.127 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:24.127 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:24.128 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:24.128 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:24.128 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:24.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.140+0000 7fa1d2a37640 1 Processor -- start 2024-03-27T21:41:24.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.140+0000 7fa1d2a37640 1 -- start start 2024-03-27T21:41:24.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.140+0000 7fa1d2a37640 10 monclient: init 2024-03-27T21:41:24.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.140+0000 7fa1d2a37640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:24.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d2a37640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-03-27T21:41:24.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d2a37640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa1c402e220 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:24.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d2a37640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa1c402e600 0x7fa1c40327f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:24.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d2a37640 10 monclient(hunting): picked mon.noname-b con 0x7fa1c402e600 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:24.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d2a37640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa1c4032d30 0x7fa1c40d25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:24.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d2a37640 10 monclient(hunting): picked mon.noname-c con 0x7fa1c4032d30 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:24.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d2a37640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1c402c8f0 con 0x7fa1c402e220 2024-03-27T21:41:24.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d2a37640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:24.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d2a37640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1c40c8720 con 0x7fa1c4032d30 2024-03-27T21:41:24.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d2a37640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:24.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d2a37640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c402f8a0 con 0x7fa1c402e600 2024-03-27T21:41:24.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d2a37640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:24.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d2a37640 10 monclient(hunting): authenticate will time out at 925.362022s 2024-03-27T21:41:24.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d1234640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa1c402e600 0x7fa1c40327f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:24.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d1234640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa1c402e600 0x7fa1c40327f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:60262/0 (socket says 172.21.15.60:60262) 2024-03-27T21:41:24.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d1234640 1 -- 172.21.15.60:0/1795914116 learned_addr learned my addr 172.21.15.60:0/1795914116 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:24.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d1234640 10 monclient(hunting): get_auth_request con 0x7fa1c402e600 auth_method 0 2024-03-27T21:41:24.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d1234640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:24.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d1234640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:24.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d1234640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:24.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d1234640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:24.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d1234640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:24.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.141+0000 7fa1d1234640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:24.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.142+0000 7fa1d1234640 10 monclient(hunting): handle_auth_done global_id 4772 payload 274 2024-03-27T21:41:24.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.142+0000 7fa1d1234640 1 -- 172.21.15.60:0/1795914116 >> v2:172.21.15.191:3300/0 conn(0x7fa1c4032d30 msgr2=0x7fa1c40d25d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T21:41:24.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.142+0000 7fa1d1234640 1 --2- 172.21.15.60:0/1795914116 >> v2:172.21.15.191:3300/0 conn(0x7fa1c4032d30 0x7fa1c40d25d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:24.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.142+0000 7fa1d1234640 1 -- 172.21.15.60:0/1795914116 >> v1:172.21.15.60:6789/0 conn(0x7fa1c402e220 legacy=0x7fa1c40c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:24.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.142+0000 7fa1d1234640 10 monclient: _finish_hunting 0 2024-03-27T21:41:24.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.142+0000 7fa1d1234640 1 monclient: found mon.noname-b 2024-03-27T21:41:24.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.142+0000 7fa1d1234640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:24.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.142+0000 7fa1d1234640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:24.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.142+0000 7fa1d1234640 1 -- 172.21.15.60:0/1795914116 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c402fb80 con 0x7fa1c402e600 2024-03-27T21:41:24.133 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:24.143+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:24.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.143+0000 7fa1d1234640 1 --2- 172.21.15.60:0/1795914116 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa1c402e600 0x7fa1c40327f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8011f90 tx=0x7fa1c803c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96ddab8a8adddbd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:24.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.143+0000 7fa1d0a33640 1 -- 172.21.15.60:0/1795914116 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c80114e0 con 0x7fa1c402e600 2024-03-27T21:41:24.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.143+0000 7fa1d0a33640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:24.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.143+0000 7fa1d0a33640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:24.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.143+0000 7fa1d0a33640 10 monclient: dump: 2024-03-27T21:41:24.150 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:24.150 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:24.150 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:24.150 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:24.150 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:24.150 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:24.151 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:24.151 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:24.151 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:24.151 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:24.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.143+0000 7fa1d0a33640 10 monclient: _finish_auth 0 2024-03-27T21:41:24.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.143+0000 7fa1d0a33640 10 monclient: _check_auth_tickets 2024-03-27T21:41:24.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.143+0000 7fa1d0a33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:24.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.143+0000 7fa1d0a33640 1 -- 172.21.15.60:0/1795914116 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1c8011640 con 0x7fa1c402e600 2024-03-27T21:41:24.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.143+0000 7fa1d0a33640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:24.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.143+0000 7fa1d0a33640 1 -- 172.21.15.60:0/1795914116 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c800fa70 con 0x7fa1c402e600 2024-03-27T21:41:24.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.144+0000 7fa1d0a33640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:24.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.144+0000 7fa1d0a33640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.144+0000 7fa1d0a33640 10 monclient: dump: 2024-03-27T21:41:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:24.152 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.144+0000 7fa1d2a37640 5 monclient: authenticate success, global_id 4772 2024-03-27T21:41:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.144+0000 7fa1d2a37640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:24.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.144+0000 7fa1d2a37640 4 set_mon_vals no callback set 2024-03-27T21:41:24.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.144+0000 7fa1d2a37640 10 monclient: shutdown 2024-03-27T21:41:24.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.144+0000 7fa1d2a37640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:24.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.144+0000 7fa1d2a37640 1 -- 172.21.15.60:0/1795914116 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa1c402e600 msgr2=0x7fa1c40327f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:24.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.144+0000 7fa1d2a37640 1 --2- 172.21.15.60:0/1795914116 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa1c402e600 0x7fa1c40327f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8011f90 tx=0x7fa1c803c7e0 comp rx=0 tx=0).stop 2024-03-27T21:41:24.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.145+0000 7fa1d2a37640 1 -- 172.21.15.60:0/1795914116 shutdown_connections 2024-03-27T21:41:24.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.145+0000 7fa1d2a37640 1 --2- 172.21.15.60:0/1795914116 >> v2:172.21.15.191:3300/0 conn(0x7fa1c4032d30 0x7fa1c40d25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:24.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.145+0000 7fa1d2a37640 1 --2- 172.21.15.60:0/1795914116 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa1c402e600 0x7fa1c40327f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:24.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.145+0000 7fa1d2a37640 1 -- 172.21.15.60:0/1795914116 >> 172.21.15.60:0/1795914116 conn(0x7fa1c4036610 msgr2=0x7fa1c4005eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.146+0000 7fa1d2a37640 1 -- 172.21.15.60:0/1795914116 shutdown_connections 2024-03-27T21:41:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.146+0000 7fa1d2a37640 1 -- 172.21.15.60:0/1795914116 wait complete. 2024-03-27T21:41:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.147+0000 7fa1d2a37640 10 monclient: build_initial_monmap 2024-03-27T21:41:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.147+0000 7fa1d2a37640 10 monclient: monmap: 2024-03-27T21:41:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:24.155 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:24.155 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:24.155 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:24.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.147+0000 7fa1d2a37640 1 librados: starting msgr at 2024-03-27T21:41:24.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.147+0000 7fa1d2a37640 1 librados: starting objecter 2024-03-27T21:41:24.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.147+0000 7fa1d2a37640 1 Processor -- start 2024-03-27T21:41:24.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.147+0000 7fa1d2a37640 1 -- start start 2024-03-27T21:41:24.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.147+0000 7fa1d2a37640 1 librados: setting wanted keys 2024-03-27T21:41:24.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.147+0000 7fa1d2a37640 1 librados: calling monclient init 2024-03-27T21:41:24.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.147+0000 7fa1d2a37640 10 monclient: init 2024-03-27T21:41:24.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.148+0000 7fa1d2a37640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.148+0000 7fa1d2a37640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:41:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.148+0000 7fa1d2a37640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1c402e220 0x7fa1c40c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.148+0000 7fa1d2a37640 10 monclient(hunting): picked mon.noname-a con 0x7fa1c402e220 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.148+0000 7fa1d2a37640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1c402e600 0x7fa1c40c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.148+0000 7fa1d2a37640 10 monclient(hunting): picked mon.noname-b con 0x7fa1c402e600 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.148+0000 7fa1d2a37640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa1c4032d30 0x7fa1c40cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.149+0000 7fa1d1a35640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1c402e220 0x7fa1c40c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.149+0000 7fa1d1a35640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1c402e220 0x7fa1c40c7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36444/0 (socket says 172.21.15.60:36444) 2024-03-27T21:41:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.149+0000 7fa1d2236640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa1c4032d30 0x7fa1c40cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.149+0000 7fa1d2236640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa1c4032d30 0x7fa1c40cb6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:60266/0 (socket says 172.21.15.60:60266) 2024-03-27T21:41:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.149+0000 7fa1d1234640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1c402e600 0x7fa1c40c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.149+0000 7fa1d1234640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1c402e600 0x7fa1c40c09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:59200/0 (socket says 172.21.15.60:59200) 2024-03-27T21:41:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.149+0000 7fa1d2a37640 10 monclient(hunting): picked mon.noname-c con 0x7fa1c4032d30 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d2a37640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d2a37640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c40c1130 con 0x7fa1c402e220 2024-03-27T21:41:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d2a37640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d2a37640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c40cb2b0 con 0x7fa1c402e600 2024-03-27T21:41:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d2a37640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d2a37640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c40cac90 con 0x7fa1c4032d30 2024-03-27T21:41:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d2a37640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d2a37640 10 monclient(hunting): authenticate will time out at 925.370946s 2024-03-27T21:41:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d1a35640 1 -- 172.21.15.60:0/4044067427 learned_addr learned my addr 172.21.15.60:0/4044067427 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d1a35640 10 monclient(hunting): get_auth_request con 0x7fa1c402e220 auth_method 0 2024-03-27T21:41:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d1a35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d1a35640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d1a35640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d2236640 10 monclient(hunting): get_auth_request con 0x7fa1c4032d30 auth_method 0 2024-03-27T21:41:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d2236640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d2236640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d2236640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d1a35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d1a35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d1a35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d1234640 10 monclient(hunting): get_auth_request con 0x7fa1c402e600 auth_method 0 2024-03-27T21:41:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d1234640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d1234640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d1234640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d2236640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d2236640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d2236640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d1a35640 10 monclient(hunting): handle_auth_done global_id 4971 payload 824 2024-03-27T21:41:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.150+0000 7fa1d1a35640 1 -- 172.21.15.60:0/4044067427 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa1c4032d30 msgr2=0x7fa1c40cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.151+0000 7fa1d1a35640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa1c4032d30 0x7fa1c40cb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.151+0000 7fa1d1a35640 1 -- 172.21.15.60:0/4044067427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1c402e600 msgr2=0x7fa1c40c09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.151+0000 7fa1d1a35640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1c402e600 0x7fa1c40c09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.151+0000 7fa1d1a35640 10 monclient: _finish_hunting 0 2024-03-27T21:41:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.151+0000 7fa1d1a35640 1 monclient: found mon.noname-a 2024-03-27T21:41:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.151+0000 7fa1d1a35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.151+0000 7fa1d1a35640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.151+0000 7fa1d1a35640 1 -- 172.21.15.60:0/4044067427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c8011d70 con 0x7fa1c402e220 2024-03-27T21:41:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.151+0000 7fa1d1234640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1c402e600 0x7fa1c40c09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:41:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.151+0000 7fa1d2236640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa1c4032d30 0x7fa1c40cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.151+0000 7fa1d1a35640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1c402e220 0x7fa1c40c7d90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc00c410 tx=0x7fa1bc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1c27fc640 10 client.?.objecter ms_handle_connect 0x7fa1c402e220 2024-03-27T21:41:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1c27fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1c27fc640 1 -- 172.21.15.60:0/4044067427 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc0035f0 con 0x7fa1c402e220 2024-03-27T21:41:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1c27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1c27fc640 10 monclient: dump: 2024-03-27T21:41:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:24.160 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1c27fc640 10 monclient: _finish_auth 0 2024-03-27T21:41:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1c27fc640 10 monclient: _check_auth_tickets 2024-03-27T21:41:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:24.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1c27fc640 1 -- 172.21.15.60:0/4044067427 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1bc003750 con 0x7fa1c402e220 2024-03-27T21:41:24.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:24.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1d2a37640 5 monclient: authenticate success, global_id 4971 2024-03-27T21:41:24.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1c27fc640 1 -- 172.21.15.60:0/4044067427 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc003b40 con 0x7fa1c402e220 2024-03-27T21:41:24.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1d2a37640 10 monclient: _renew_subs 2024-03-27T21:41:24.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1d2a37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:24.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1d2a37640 1 -- 172.21.15.60:0/4044067427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c40c6f50 con 0x7fa1c402e220 2024-03-27T21:41:24.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1c3fff640 4 set_mon_vals no callback set 2024-03-27T21:41:24.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1d2a37640 10 client.4971.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:24.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1d2a37640 10 monclient: _renew_subs 2024-03-27T21:41:24.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1d2a37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:24.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1d2a37640 1 -- 172.21.15.60:0/4044067427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c40c9220 con 0x7fa1c402e220 2024-03-27T21:41:24.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.152+0000 7fa1d2a37640 1 librados: init done 2024-03-27T21:41:24.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.153+0000 7fa1c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:24.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.153+0000 7fa1c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:24.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.153+0000 7fa1c27fc640 10 monclient: dump: 2024-03-27T21:41:24.162 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:24.162 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:24.162 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:24.162 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:24.162 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:24.162 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:24.162 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:24.163 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:24.163 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:24.163 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:24.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.154+0000 7fa1d2a37640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:41:24.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.154+0000 7fa1d2a37640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:24.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.154+0000 7fa1d2a37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:24.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.154+0000 7fa1d2a37640 1 -- 172.21.15.60:0/4044067427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fa1c4005290 con 0x7fa1c402e220 2024-03-27T21:41:24.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.156+0000 7fa1c27fc640 1 -- 172.21.15.60:0/4044067427 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa1bc003ca0 con 0x7fa1c402e220 2024-03-27T21:41:24.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.156+0000 7fa1c27fc640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa1b006c660 0x7fa1b006eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:24.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.156+0000 7fa1d1234640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa1b006c660 0x7fa1b006eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:24.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.157+0000 7fa1d1234640 10 monclient: get_auth_request con 0x7fa1b006c660 auth_method 0 2024-03-27T21:41:24.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.157+0000 7fa1c27fc640 1 -- 172.21.15.60:0/4044067427 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fa1bc0887d0 con 0x7fa1c402e220 2024-03-27T21:41:24.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.157+0000 7fa1c27fc640 10 client.4971.objecter ms_dispatch 0x7fa1cc000b90 osd_map(59..59 src has 1..59) v4 2024-03-27T21:41:24.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.157+0000 7fa1c27fc640 3 client.4971.objecter handle_osd_map got epochs [59,59] > 0 2024-03-27T21:41:24.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.157+0000 7fa1c27fc640 3 client.4971.objecter handle_osd_map decoding full epoch 59 2024-03-27T21:41:24.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.157+0000 7fa1c27fc640 20 client.4971.objecter dump_active .. 0 homeless 2024-03-27T21:41:24.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.157+0000 7fa1c27fc640 1 -- 172.21.15.60:0/4044067427 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v59) v1 ==== 54+0+16 (secure 0 0 0) 0x7fa1bc08f070 con 0x7fa1c402e220 2024-03-27T21:41:24.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.157+0000 7fa1d1234640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa1b006c660 0x7fa1b006eb20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8046000 tx=0x7fa1c8011000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:24.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.157+0000 7fa1c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:24.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.157+0000 7fa1c27fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:24.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.157+0000 7fa1c27fc640 10 client.4971.objecter ms_handle_connect 0x7fa1b006c660 2024-03-27T21:41:24.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.158+0000 7fa1affff640 10 client.4971.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:24.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.158+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign -1 1 2024-03-27T21:41:24.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.158+0000 7fa1affff640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fa1a8002f70 0x7fa1a8005430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:24.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.158+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a8002860 osd=0 3 2024-03-27T21:41:24.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.158+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a8002860 osd=0 3 2024-03-27T21:41:24.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.158+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a8002860 osd=0 3 2024-03-27T21:41:24.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.158+0000 7fa1affff640 15 client.4971.objecter _session_command_op_remove -1 1 2024-03-27T21:41:24.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.158+0000 7fa1affff640 20 client.4971.objecter get_session s=0x7fa1a8002860 osd=0 3 2024-03-27T21:41:24.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.158+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign 0 1 2024-03-27T21:41:24.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.158+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a8002860 osd=0 4 2024-03-27T21:41:24.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.158+0000 7fa1affff640 10 client.4971.objecter _send_command 1 2024-03-27T21:41:24.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.158+0000 7fa1affff640 1 -- 172.21.15.60:0/4044067427 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa1a8008420 con 0x7fa1a8002f70 2024-03-27T21:41:24.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.158+0000 7fa1d2236640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fa1a8002f70 0x7fa1a8005430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:24.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.158+0000 7fa1d2236640 10 monclient: get_auth_request con 0x7fa1a8002f70 auth_method 0 2024-03-27T21:41:24.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.159+0000 7fa1d2236640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fa1a8002f70 0x7fa1a8005430 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:24.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.159+0000 7fa1c27fc640 10 client.4971.objecter ms_handle_connect 0x7fa1a8002f70 2024-03-27T21:41:24.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.160+0000 7fa1c27fc640 1 -- 172.21.15.60:0/4044067427 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fa1a8008420 con 0x7fa1a8002f70 2024-03-27T21:41:24.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.160+0000 7fa1c27fc640 10 client.4971.objecter ms_dispatch 0x7fa1cc000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:41:24.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.160+0000 7fa1c27fc640 10 client.4971.objecter _finish_command 1 = system:0 2024-03-27T21:41:24.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.160+0000 7fa1c27fc640 20 client.4971.objecter put_session s=0x7fa1a8002860 osd=0 4 2024-03-27T21:41:24.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.160+0000 7fa1c27fc640 15 client.4971.objecter _session_command_op_remove 0 1 2024-03-27T21:41:24.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.185+0000 7fa1affff640 10 client.4971.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:24.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.185+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign -1 2 2024-03-27T21:41:24.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.185+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a8002860 osd=0 3 2024-03-27T21:41:24.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.185+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a8002860 osd=0 3 2024-03-27T21:41:24.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.185+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a8002860 osd=0 3 2024-03-27T21:41:24.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.185+0000 7fa1affff640 15 client.4971.objecter _session_command_op_remove -1 2 2024-03-27T21:41:24.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.185+0000 7fa1affff640 20 client.4971.objecter get_session s=0x7fa1a8002860 osd=0 3 2024-03-27T21:41:24.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.185+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign 0 2 2024-03-27T21:41:24.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.185+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a8002860 osd=0 4 2024-03-27T21:41:24.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.185+0000 7fa1affff640 10 client.4971.objecter _send_command 2 2024-03-27T21:41:24.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.185+0000 7fa1affff640 1 -- 172.21.15.60:0/4044067427 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa1a8001100 con 0x7fa1a8002f70 2024-03-27T21:41:24.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.188+0000 7fa1c27fc640 1 -- 172.21.15.60:0/4044067427 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa1a8001100 con 0x7fa1a8002f70 2024-03-27T21:41:24.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.188+0000 7fa1c27fc640 10 client.4971.objecter ms_dispatch 0x7fa1cc000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:24.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.188+0000 7fa1c27fc640 10 client.4971.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:24.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.188+0000 7fa1c27fc640 20 client.4971.objecter put_session s=0x7fa1a8002860 osd=0 4 2024-03-27T21:41:24.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.188+0000 7fa1c27fc640 15 client.4971.objecter _session_command_op_remove 0 2 2024-03-27T21:41:24.176 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:41:24.176 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:41:24.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.191+0000 7fa1affff640 10 client.4971.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:24.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.191+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign -1 3 2024-03-27T21:41:24.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.191+0000 7fa1affff640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fa1a8005ab0 0x7fa1a8005f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:24.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.191+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a8002560 osd=1 3 2024-03-27T21:41:24.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.191+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a8002560 osd=1 3 2024-03-27T21:41:24.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.191+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a8002560 osd=1 3 2024-03-27T21:41:24.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.191+0000 7fa1affff640 15 client.4971.objecter _session_command_op_remove -1 3 2024-03-27T21:41:24.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.191+0000 7fa1affff640 20 client.4971.objecter get_session s=0x7fa1a8002560 osd=1 3 2024-03-27T21:41:24.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.191+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign 1 3 2024-03-27T21:41:24.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.191+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a8002560 osd=1 4 2024-03-27T21:41:24.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.191+0000 7fa1affff640 10 client.4971.objecter _send_command 3 2024-03-27T21:41:24.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.191+0000 7fa1affff640 1 -- 172.21.15.60:0/4044067427 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa1a80064a0 con 0x7fa1a8005ab0 2024-03-27T21:41:24.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.191+0000 7fa1d1a35640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fa1a8005ab0 0x7fa1a8005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:24.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.191+0000 7fa1d1a35640 10 monclient: get_auth_request con 0x7fa1a8005ab0 auth_method 0 2024-03-27T21:41:24.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.191+0000 7fa1c27fc640 10 client.4971.objecter ms_handle_connect 0x7fa1a8005ab0 2024-03-27T21:41:24.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.191+0000 7fa1d1a35640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fa1a8005ab0 0x7fa1a8005f60 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:24.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.192+0000 7fa1c27fc640 1 -- 172.21.15.60:0/4044067427 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fa1a80064a0 con 0x7fa1a8005ab0 2024-03-27T21:41:24.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.192+0000 7fa1c27fc640 10 client.4971.objecter ms_dispatch 0x7fa1cc000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:41:24.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.192+0000 7fa1c27fc640 10 client.4971.objecter _finish_command 3 = system:0 2024-03-27T21:41:24.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.192+0000 7fa1c27fc640 20 client.4971.objecter put_session s=0x7fa1a8002560 osd=1 4 2024-03-27T21:41:24.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.192+0000 7fa1c27fc640 15 client.4971.objecter _session_command_op_remove 1 3 2024-03-27T21:41:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:no valid command found; 10 closest matches: 2024-03-27T21:41:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:0 2024-03-27T21:41:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:1 2024-03-27T21:41:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:2 2024-03-27T21:41:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:abort 2024-03-27T21:41:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:assert 2024-03-27T21:41:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:config diff 2024-03-27T21:41:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:config diff get 2024-03-27T21:41:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:config get 2024-03-27T21:41:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:config help [] 2024-03-27T21:41:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:config set ... 2024-03-27T21:41:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:admin_socket: invalid command 2024-03-27T21:41:24.196 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T21:41:24.196 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-03-27T21:41:24.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.223+0000 7fa1affff640 10 client.4971.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:24.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.223+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign -1 4 2024-03-27T21:41:24.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.223+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a8002560 osd=1 3 2024-03-27T21:41:24.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.223+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a8002560 osd=1 3 2024-03-27T21:41:24.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.223+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a8002560 osd=1 3 2024-03-27T21:41:24.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.223+0000 7fa1affff640 15 client.4971.objecter _session_command_op_remove -1 4 2024-03-27T21:41:24.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.223+0000 7fa1affff640 20 client.4971.objecter get_session s=0x7fa1a8002560 osd=1 3 2024-03-27T21:41:24.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.223+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign 1 4 2024-03-27T21:41:24.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.223+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a8002560 osd=1 4 2024-03-27T21:41:24.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.223+0000 7fa1affff640 10 client.4971.objecter _send_command 4 2024-03-27T21:41:24.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.223+0000 7fa1affff640 1 -- 172.21.15.60:0/4044067427 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa1a8006e60 con 0x7fa1a8005ab0 2024-03-27T21:41:24.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.226+0000 7fa1c27fc640 1 -- 172.21.15.60:0/4044067427 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa1a8006e60 con 0x7fa1a8005ab0 2024-03-27T21:41:24.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.226+0000 7fa1c27fc640 10 client.4971.objecter ms_dispatch 0x7fa1cc000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:24.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.226+0000 7fa1c27fc640 10 client.4971.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:24.213 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:41:24.219 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:41:24.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.226+0000 7fa1c27fc640 20 client.4971.objecter put_session s=0x7fa1a8002560 osd=1 4 2024-03-27T21:41:24.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.226+0000 7fa1c27fc640 15 client.4971.objecter _session_command_op_remove 1 4 2024-03-27T21:41:24.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.226+0000 7fa1affff640 10 client.4971.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:24.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.226+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign -1 5 2024-03-27T21:41:24.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.226+0000 7fa1affff640 20 client.4971.objecter _get_session osd=-1 returning homeless 2024-03-27T21:41:24.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.226+0000 7fa1affff640 10 client.4971.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:24.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.226+0000 7fa1affff640 10 monclient: _renew_subs 2024-03-27T21:41:24.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.226+0000 7fa1affff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:24.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.226+0000 7fa1affff640 1 -- 172.21.15.60:0/4044067427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=60}) v3 -- 0x7fa1a8006fd0 con 0x7fa1c402e220 2024-03-27T21:41:24.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.226+0000 7fa1affff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:24.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.226+0000 7fa1affff640 1 -- 172.21.15.60:0/4044067427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa1a80079b0 con 0x7fa1c402e220 2024-03-27T21:41:24.221 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:41:24.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.227+0000 7fa1c27fc640 1 -- 172.21.15.60:0/4044067427 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1bc001140 con 0x7fa1c402e220 2024-03-27T21:41:24.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.227+0000 7fa1c27fc640 10 monclient: handle_get_version_reply finishing 1 version 59 2024-03-27T21:41:24.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.227+0000 7fa1c3fff640 10 client.4971.objecter _check_command_map_dne tid 5 current 59 map_dne_bound 59 2024-03-27T21:41:24.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.227+0000 7fa1c3fff640 10 client.4971.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:41:24.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.227+0000 7fa1c3fff640 15 client.4971.objecter _session_command_op_remove -1 5 2024-03-27T21:41:24.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.227+0000 7fa1affff640 10 client.4971.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:24.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.227+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign -1 6 2024-03-27T21:41:24.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.227+0000 7fa1affff640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fa1a80069c0 0x7fa1a800b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:24.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.227+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a8006850 osd=3 3 2024-03-27T21:41:24.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.227+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a8006850 osd=3 3 2024-03-27T21:41:24.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.227+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a8006850 osd=3 3 2024-03-27T21:41:24.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.227+0000 7fa1affff640 15 client.4971.objecter _session_command_op_remove -1 6 2024-03-27T21:41:24.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.227+0000 7fa1affff640 20 client.4971.objecter get_session s=0x7fa1a8006850 osd=3 3 2024-03-27T21:41:24.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.227+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign 3 6 2024-03-27T21:41:24.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.227+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a8006850 osd=3 4 2024-03-27T21:41:24.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.227+0000 7fa1affff640 10 client.4971.objecter _send_command 6 2024-03-27T21:41:24.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.227+0000 7fa1affff640 1 -- 172.21.15.60:0/4044067427 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa1a8002200 con 0x7fa1a80069c0 2024-03-27T21:41:24.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.227+0000 7fa1d1234640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fa1a80069c0 0x7fa1a800b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:24.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.227+0000 7fa1d1234640 10 monclient: get_auth_request con 0x7fa1a80069c0 auth_method 0 2024-03-27T21:41:24.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.228+0000 7fa1d1234640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fa1a80069c0 0x7fa1a800b780 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:24.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.228+0000 7fa1c27fc640 10 client.4971.objecter ms_handle_connect 0x7fa1a80069c0 2024-03-27T21:41:24.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.229+0000 7fa1c27fc640 1 -- 172.21.15.60:0/4044067427 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fa1a8002200 con 0x7fa1a80069c0 2024-03-27T21:41:24.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.229+0000 7fa1c27fc640 10 client.4971.objecter ms_dispatch 0x7fa1cc000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:41:24.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.229+0000 7fa1c27fc640 10 client.4971.objecter _finish_command 6 = system:0 2024-03-27T21:41:24.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.229+0000 7fa1c27fc640 20 client.4971.objecter put_session s=0x7fa1a8006850 osd=3 4 2024-03-27T21:41:24.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.229+0000 7fa1c27fc640 15 client.4971.objecter _session_command_op_remove 3 6 2024-03-27T21:41:24.230 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:24.243+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:24.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.258+0000 7fa1affff640 10 client.4971.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:24.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.258+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign -1 7 2024-03-27T21:41:24.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.258+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a8006850 osd=3 3 2024-03-27T21:41:24.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.258+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a8006850 osd=3 3 2024-03-27T21:41:24.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.258+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a8006850 osd=3 3 2024-03-27T21:41:24.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.258+0000 7fa1affff640 15 client.4971.objecter _session_command_op_remove -1 7 2024-03-27T21:41:24.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.258+0000 7fa1affff640 20 client.4971.objecter get_session s=0x7fa1a8006850 osd=3 3 2024-03-27T21:41:24.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.258+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign 3 7 2024-03-27T21:41:24.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.258+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a8006850 osd=3 4 2024-03-27T21:41:24.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.258+0000 7fa1affff640 10 client.4971.objecter _send_command 7 2024-03-27T21:41:24.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.258+0000 7fa1affff640 1 -- 172.21.15.60:0/4044067427 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa1a80015e0 con 0x7fa1a80069c0 2024-03-27T21:41:24.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.261+0000 7fa1c27fc640 1 -- 172.21.15.60:0/4044067427 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa1a80015e0 con 0x7fa1a80069c0 2024-03-27T21:41:24.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.261+0000 7fa1c27fc640 10 client.4971.objecter ms_dispatch 0x7fa1cc000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:24.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.261+0000 7fa1c27fc640 10 client.4971.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:24.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.261+0000 7fa1c27fc640 20 client.4971.objecter put_session s=0x7fa1a8006850 osd=3 4 2024-03-27T21:41:24.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.261+0000 7fa1c27fc640 15 client.4971.objecter _session_command_op_remove 3 7 2024-03-27T21:41:24.248 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:41:24.249 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:41:24.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.261+0000 7fa1affff640 10 client.4971.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:24.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.261+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign -1 8 2024-03-27T21:41:24.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.261+0000 7fa1affff640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fa1a800b2d0 0x7fa1a8010ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:24.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.261+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a80023b0 osd=4 3 2024-03-27T21:41:24.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.261+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a80023b0 osd=4 3 2024-03-27T21:41:24.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.261+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a80023b0 osd=4 3 2024-03-27T21:41:24.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.261+0000 7fa1affff640 15 client.4971.objecter _session_command_op_remove -1 8 2024-03-27T21:41:24.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.261+0000 7fa1affff640 20 client.4971.objecter get_session s=0x7fa1a80023b0 osd=4 3 2024-03-27T21:41:24.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.261+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign 4 8 2024-03-27T21:41:24.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.261+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a80023b0 osd=4 4 2024-03-27T21:41:24.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.262+0000 7fa1affff640 10 client.4971.objecter _send_command 8 2024-03-27T21:41:24.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.262+0000 7fa1affff640 1 -- 172.21.15.60:0/4044067427 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa1a8013d60 con 0x7fa1a800b2d0 2024-03-27T21:41:24.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.262+0000 7fa1d2236640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fa1a800b2d0 0x7fa1a8010ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:24.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.262+0000 7fa1d2236640 10 monclient: get_auth_request con 0x7fa1a800b2d0 auth_method 0 2024-03-27T21:41:24.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.262+0000 7fa1d2236640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fa1a800b2d0 0x7fa1a8010ee0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:24.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.262+0000 7fa1c27fc640 10 client.4971.objecter ms_handle_connect 0x7fa1a800b2d0 2024-03-27T21:41:24.251 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:24.251 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:24.251 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:41:24.251 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:41:24.251 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:24.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.263+0000 7fa1c27fc640 1 -- 172.21.15.60:0/4044067427 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fa1a8013d60 con 0x7fa1a800b2d0 2024-03-27T21:41:24.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.263+0000 7fa1c27fc640 10 client.4971.objecter ms_dispatch 0x7fa1cc000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:41:24.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.263+0000 7fa1c27fc640 10 client.4971.objecter _finish_command 8 = system:0 2024-03-27T21:41:24.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.263+0000 7fa1c27fc640 20 client.4971.objecter put_session s=0x7fa1a80023b0 osd=4 4 2024-03-27T21:41:24.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.263+0000 7fa1c27fc640 15 client.4971.objecter _session_command_op_remove 4 8 2024-03-27T21:41:24.259 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:41:24.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.291+0000 7fa1affff640 10 client.4971.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:24.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.291+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign -1 9 2024-03-27T21:41:24.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.291+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a80023b0 osd=4 3 2024-03-27T21:41:24.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.291+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a80023b0 osd=4 3 2024-03-27T21:41:24.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.291+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a80023b0 osd=4 3 2024-03-27T21:41:24.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.291+0000 7fa1affff640 15 client.4971.objecter _session_command_op_remove -1 9 2024-03-27T21:41:24.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.291+0000 7fa1affff640 20 client.4971.objecter get_session s=0x7fa1a80023b0 osd=4 3 2024-03-27T21:41:24.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.291+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign 4 9 2024-03-27T21:41:24.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.291+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a80023b0 osd=4 4 2024-03-27T21:41:24.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.291+0000 7fa1affff640 10 client.4971.objecter _send_command 9 2024-03-27T21:41:24.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.291+0000 7fa1affff640 1 -- 172.21.15.60:0/4044067427 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa1a800a8a0 con 0x7fa1a800b2d0 2024-03-27T21:41:24.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.294+0000 7fa1c27fc640 1 -- 172.21.15.60:0/4044067427 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa1a800a8a0 con 0x7fa1a800b2d0 2024-03-27T21:41:24.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.294+0000 7fa1c27fc640 10 client.4971.objecter ms_dispatch 0x7fa1cc000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:24.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.294+0000 7fa1c27fc640 10 client.4971.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:24.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.294+0000 7fa1c27fc640 20 client.4971.objecter put_session s=0x7fa1a80023b0 osd=4 4 2024-03-27T21:41:24.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.294+0000 7fa1c27fc640 15 client.4971.objecter _session_command_op_remove 4 9 2024-03-27T21:41:24.282 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:41:24.283 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:41:24.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.294+0000 7fa1affff640 10 client.4971.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:24.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.294+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign -1 10 2024-03-27T21:41:24.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.294+0000 7fa1affff640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fa1a800c990 0x7fa1a80125a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:24.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.294+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a800a660 osd=5 3 2024-03-27T21:41:24.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.294+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a800a660 osd=5 3 2024-03-27T21:41:24.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.294+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a800a660 osd=5 3 2024-03-27T21:41:24.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.294+0000 7fa1affff640 15 client.4971.objecter _session_command_op_remove -1 10 2024-03-27T21:41:24.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.294+0000 7fa1affff640 20 client.4971.objecter get_session s=0x7fa1a800a660 osd=5 3 2024-03-27T21:41:24.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.294+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign 5 10 2024-03-27T21:41:24.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.294+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a800a660 osd=5 4 2024-03-27T21:41:24.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.294+0000 7fa1affff640 10 client.4971.objecter _send_command 10 2024-03-27T21:41:24.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.294+0000 7fa1affff640 1 -- 172.21.15.60:0/4044067427 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa1a8017650 con 0x7fa1a800c990 2024-03-27T21:41:24.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.294+0000 7fa1d1a35640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fa1a800c990 0x7fa1a80125a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:24.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.294+0000 7fa1d1a35640 10 monclient: get_auth_request con 0x7fa1a800c990 auth_method 0 2024-03-27T21:41:24.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.295+0000 7fa1d1a35640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fa1a800c990 0x7fa1a80125a0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:24.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.295+0000 7fa1c27fc640 10 client.4971.objecter ms_handle_connect 0x7fa1a800c990 2024-03-27T21:41:24.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.295+0000 7fa1c27fc640 1 -- 172.21.15.60:0/4044067427 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fa1a8017650 con 0x7fa1a800c990 2024-03-27T21:41:24.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.295+0000 7fa1c27fc640 10 client.4971.objecter ms_dispatch 0x7fa1cc000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:41:24.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.295+0000 7fa1c27fc640 10 client.4971.objecter _finish_command 10 = system:0 2024-03-27T21:41:24.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.295+0000 7fa1c27fc640 20 client.4971.objecter put_session s=0x7fa1a800a660 osd=5 4 2024-03-27T21:41:24.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.295+0000 7fa1c27fc640 15 client.4971.objecter _session_command_op_remove 5 10 2024-03-27T21:41:24.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.323+0000 7fa1affff640 10 client.4971.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:24.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.323+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign -1 11 2024-03-27T21:41:24.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.323+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a800a660 osd=5 3 2024-03-27T21:41:24.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.323+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a800a660 osd=5 3 2024-03-27T21:41:24.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.323+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a800a660 osd=5 3 2024-03-27T21:41:24.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.323+0000 7fa1affff640 15 client.4971.objecter _session_command_op_remove -1 11 2024-03-27T21:41:24.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.323+0000 7fa1affff640 20 client.4971.objecter get_session s=0x7fa1a800a660 osd=5 3 2024-03-27T21:41:24.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.323+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign 5 11 2024-03-27T21:41:24.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.324+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a800a660 osd=5 4 2024-03-27T21:41:24.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.324+0000 7fa1affff640 10 client.4971.objecter _send_command 11 2024-03-27T21:41:24.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.324+0000 7fa1affff640 1 -- 172.21.15.60:0/4044067427 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa1a800a8e0 con 0x7fa1a800c990 2024-03-27T21:41:24.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.326+0000 7fa1c27fc640 1 -- 172.21.15.60:0/4044067427 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa1a800a8e0 con 0x7fa1a800c990 2024-03-27T21:41:24.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.326+0000 7fa1c27fc640 10 client.4971.objecter ms_dispatch 0x7fa1cc000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:24.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.326+0000 7fa1c27fc640 10 client.4971.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:24.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.326+0000 7fa1c27fc640 20 client.4971.objecter put_session s=0x7fa1a800a660 osd=5 4 2024-03-27T21:41:24.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.326+0000 7fa1c27fc640 15 client.4971.objecter _session_command_op_remove 5 11 2024-03-27T21:41:24.312 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:41:24.314 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:41:24.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.326+0000 7fa1affff640 10 client.4971.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:24.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.326+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign -1 12 2024-03-27T21:41:24.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.326+0000 7fa1affff640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fa1a80130a0 0x7fa1a8013500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:24.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.326+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a800a8e0 osd=6 3 2024-03-27T21:41:24.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.326+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a800a8e0 osd=6 3 2024-03-27T21:41:24.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.326+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a800a8e0 osd=6 3 2024-03-27T21:41:24.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.326+0000 7fa1affff640 15 client.4971.objecter _session_command_op_remove -1 12 2024-03-27T21:41:24.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.326+0000 7fa1affff640 20 client.4971.objecter get_session s=0x7fa1a800a8e0 osd=6 3 2024-03-27T21:41:24.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.326+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign 6 12 2024-03-27T21:41:24.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.326+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a800a8e0 osd=6 4 2024-03-27T21:41:24.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.326+0000 7fa1affff640 10 client.4971.objecter _send_command 12 2024-03-27T21:41:24.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.326+0000 7fa1affff640 1 -- 172.21.15.60:0/4044067427 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa1a801a7e0 con 0x7fa1a80130a0 2024-03-27T21:41:24.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.326+0000 7fa1d1234640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fa1a80130a0 0x7fa1a8013500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:24.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.326+0000 7fa1d1234640 10 monclient: get_auth_request con 0x7fa1a80130a0 auth_method 0 2024-03-27T21:41:24.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.327+0000 7fa1c27fc640 10 client.4971.objecter ms_handle_connect 0x7fa1a80130a0 2024-03-27T21:41:24.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.327+0000 7fa1d1234640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fa1a80130a0 0x7fa1a8013500 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:24.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.328+0000 7fa1c27fc640 1 -- 172.21.15.60:0/4044067427 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fa1a801a7e0 con 0x7fa1a80130a0 2024-03-27T21:41:24.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.328+0000 7fa1c27fc640 10 client.4971.objecter ms_dispatch 0x7fa1cc000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:41:24.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.328+0000 7fa1c27fc640 10 client.4971.objecter _finish_command 12 = system:0 2024-03-27T21:41:24.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.328+0000 7fa1c27fc640 20 client.4971.objecter put_session s=0x7fa1a800a8e0 osd=6 4 2024-03-27T21:41:24.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.328+0000 7fa1c27fc640 15 client.4971.objecter _session_command_op_remove 6 12 2024-03-27T21:41:24.330 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:24.344+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:24.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.356+0000 7fa1affff640 10 client.4971.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:24.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.356+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign -1 13 2024-03-27T21:41:24.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.356+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a800a8e0 osd=6 3 2024-03-27T21:41:24.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.356+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a800a8e0 osd=6 3 2024-03-27T21:41:24.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.356+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a800a8e0 osd=6 3 2024-03-27T21:41:24.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.356+0000 7fa1affff640 15 client.4971.objecter _session_command_op_remove -1 13 2024-03-27T21:41:24.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.356+0000 7fa1affff640 20 client.4971.objecter get_session s=0x7fa1a800a8e0 osd=6 3 2024-03-27T21:41:24.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.356+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign 6 13 2024-03-27T21:41:24.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.356+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a800a8e0 osd=6 4 2024-03-27T21:41:24.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.356+0000 7fa1affff640 10 client.4971.objecter _send_command 13 2024-03-27T21:41:24.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.356+0000 7fa1affff640 1 -- 172.21.15.60:0/4044067427 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa1a8017f50 con 0x7fa1a80130a0 2024-03-27T21:41:24.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.358+0000 7fa1c27fc640 1 -- 172.21.15.60:0/4044067427 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa1a8017f50 con 0x7fa1a80130a0 2024-03-27T21:41:24.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.358+0000 7fa1c27fc640 10 client.4971.objecter ms_dispatch 0x7fa1cc000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:24.345 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:41:24.347 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:41:24.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.358+0000 7fa1c27fc640 10 client.4971.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:24.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.358+0000 7fa1c27fc640 20 client.4971.objecter put_session s=0x7fa1a800a8e0 osd=6 4 2024-03-27T21:41:24.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.358+0000 7fa1c27fc640 15 client.4971.objecter _session_command_op_remove 6 13 2024-03-27T21:41:24.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.358+0000 7fa1affff640 10 client.4971.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:24.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.358+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign -1 14 2024-03-27T21:41:24.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.358+0000 7fa1affff640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fa1a8018620 0x7fa1a801aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:24.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.358+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a8017f50 osd=7 3 2024-03-27T21:41:24.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.358+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a8017f50 osd=7 3 2024-03-27T21:41:24.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.358+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a8017f50 osd=7 3 2024-03-27T21:41:24.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.358+0000 7fa1affff640 15 client.4971.objecter _session_command_op_remove -1 14 2024-03-27T21:41:24.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.358+0000 7fa1affff640 20 client.4971.objecter get_session s=0x7fa1a8017f50 osd=7 3 2024-03-27T21:41:24.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.358+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign 7 14 2024-03-27T21:41:24.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.358+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a8017f50 osd=7 4 2024-03-27T21:41:24.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.358+0000 7fa1affff640 10 client.4971.objecter _send_command 14 2024-03-27T21:41:24.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.358+0000 7fa1affff640 1 -- 172.21.15.60:0/4044067427 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa1a801da00 con 0x7fa1a8018620 2024-03-27T21:41:24.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.359+0000 7fa1d2236640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fa1a8018620 0x7fa1a801aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:24.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.359+0000 7fa1d2236640 10 monclient: get_auth_request con 0x7fa1a8018620 auth_method 0 2024-03-27T21:41:24.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.359+0000 7fa1c27fc640 10 client.4971.objecter ms_handle_connect 0x7fa1a8018620 2024-03-27T21:41:24.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.359+0000 7fa1d2236640 1 --2- 172.21.15.60:0/4044067427 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fa1a8018620 0x7fa1a801aa90 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:24.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.360+0000 7fa1c27fc640 1 -- 172.21.15.60:0/4044067427 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fa1a801da00 con 0x7fa1a8018620 2024-03-27T21:41:24.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.360+0000 7fa1c27fc640 10 client.4971.objecter ms_dispatch 0x7fa1cc000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:41:24.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.360+0000 7fa1c27fc640 10 client.4971.objecter _finish_command 14 = system:0 2024-03-27T21:41:24.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.360+0000 7fa1c27fc640 20 client.4971.objecter put_session s=0x7fa1a8017f50 osd=7 4 2024-03-27T21:41:24.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.360+0000 7fa1c27fc640 15 client.4971.objecter _session_command_op_remove 7 14 2024-03-27T21:41:24.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.388+0000 7fa1affff640 10 client.4971.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:24.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.388+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign -1 15 2024-03-27T21:41:24.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.388+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a8017f50 osd=7 3 2024-03-27T21:41:24.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.388+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a8017f50 osd=7 3 2024-03-27T21:41:24.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.388+0000 7fa1affff640 20 client.4971.objecter _get_session s=0x7fa1a8017f50 osd=7 3 2024-03-27T21:41:24.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.388+0000 7fa1affff640 15 client.4971.objecter _session_command_op_remove -1 15 2024-03-27T21:41:24.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.388+0000 7fa1affff640 20 client.4971.objecter get_session s=0x7fa1a8017f50 osd=7 3 2024-03-27T21:41:24.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.388+0000 7fa1affff640 15 client.4971.objecter _session_command_op_assign 7 15 2024-03-27T21:41:24.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.388+0000 7fa1affff640 20 client.4971.objecter put_session s=0x7fa1a8017f50 osd=7 4 2024-03-27T21:41:24.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.388+0000 7fa1affff640 10 client.4971.objecter _send_command 15 2024-03-27T21:41:24.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.388+0000 7fa1affff640 1 -- 172.21.15.60:0/4044067427 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa1a801c820 con 0x7fa1a8018620 2024-03-27T21:41:24.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.390+0000 7fa1c27fc640 1 -- 172.21.15.60:0/4044067427 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fa1a801c820 con 0x7fa1a8018620 2024-03-27T21:41:24.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.390+0000 7fa1c27fc640 10 client.4971.objecter ms_dispatch 0x7fa1cc000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:24.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.390+0000 7fa1c27fc640 10 client.4971.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:24.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.390+0000 7fa1c27fc640 20 client.4971.objecter put_session s=0x7fa1a8017f50 osd=7 4 2024-03-27T21:41:24.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:24.390+0000 7fa1c27fc640 15 client.4971.objecter _session_command_op_remove 7 15 2024-03-27T21:41:24.377 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:41:24.377 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:41:24.380 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:41:24.380 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:41:24.401 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:24.401 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:41:24.401 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:41:24.401 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:41:24.401 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:24.410 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:41:24.414 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:24.428+0000 7fb853f8f640 -1 Falling back to public interface 2024-03-27T21:41:24.431 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:24.445+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:24.526 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:24.527 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:24.527 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:24.527 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:24.532 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:24.546+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:24.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:41:24.634 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:24.647+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:24.681 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:24.681 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:24.681 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:24.681 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:24.681 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:24.690 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:41:24.734 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:24.747+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:24.834 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:24.848+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:24.836 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:24.836 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:24.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:24.836 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:41:24.836 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:24.845 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:41:24.935 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:24.949+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:24.990 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:24.990 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:24.990 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:24.990 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:24.998 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:41:25.036 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-27T21:41:25.042 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:25.056+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:25.104 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:25.104 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:25.105 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:25.105 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:25.105 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:25.113 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:41:25.137 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:25.151+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:25.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.175+0000 7f063b02f640 10 monclient: get_monmap_and_config 2024-03-27T21:41:25.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.175+0000 7f063b02f640 10 monclient: build_initial_monmap 2024-03-27T21:41:25.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.175+0000 7f063b02f640 10 monclient: monmap: 2024-03-27T21:41:25.162 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:25.162 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:25.162 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:25.176911+0000 2024-03-27T21:41:25.162 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:25.176911+0000 2024-03-27T21:41:25.162 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:25.163 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:25.163 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:25.163 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:25.163 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:25.163 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:25.163 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:25.163 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:25.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.176+0000 7f063b02f640 1 Processor -- start 2024-03-27T21:41:25.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.176+0000 7f063b02f640 1 -- start start 2024-03-27T21:41:25.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.176+0000 7f063b02f640 10 monclient: init 2024-03-27T21:41:25.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.177+0000 7f063b02f640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:25.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.177+0000 7f063b02f640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-03-27T21:41:25.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.177+0000 7f063b02f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0634071a50 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:25.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.177+0000 7f063b02f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f063413ef20 0x7f06340a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:25.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.177+0000 7f063b02f640 10 monclient(hunting): picked mon.noname-a con 0x7f063413ef20 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:25.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.177+0000 7f063b02f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0634141630 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:25.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.177+0000 7f063b02f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06340a8e90 con 0x7f0634141630 2024-03-27T21:41:25.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f063b02f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06340aa040 con 0x7f0634071a50 2024-03-27T21:41:25.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f063b02f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:25.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f063b02f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0634071e30 con 0x7f063413ef20 2024-03-27T21:41:25.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f063b02f640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:25.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f0633fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f063413ef20 0x7f06340a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:25.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f063b02f640 10 monclient(hunting): authenticate will time out at 926.399049s 2024-03-27T21:41:25.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f0633fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f063413ef20 0x7f06340a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36446/0 (socket says 172.21.15.60:36446) 2024-03-27T21:41:25.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f0633fff640 1 -- 172.21.15.60:0/4100833537 learned_addr learned my addr 172.21.15.60:0/4100833537 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:25.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f0633fff640 10 monclient(hunting): get_auth_request con 0x7f063413ef20 auth_method 0 2024-03-27T21:41:25.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f0633fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:25.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f0633fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:25.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f0633fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:25.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f0633fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:25.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f0633fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:25.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f0633fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:25.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f0633fff640 10 monclient(hunting): handle_auth_done global_id 4974 payload 274 2024-03-27T21:41:25.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f0633fff640 1 -- 172.21.15.60:0/4100833537 >> v1:172.21.15.191:6789/0 conn(0x7f0634071a50 legacy=0x7f063413e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:25.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f0633fff640 1 -- 172.21.15.60:0/4100833537 >> v1:172.21.15.60:6789/0 conn(0x7f0634141630 legacy=0x7f0634141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:25.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f0633fff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:25.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f0633fff640 1 monclient: found mon.noname-a 2024-03-27T21:41:25.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f0633fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:25.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f0633fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:25.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.178+0000 7f0633fff640 1 -- 172.21.15.60:0/4100833537 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06340aa2b0 con 0x7f063413ef20 2024-03-27T21:41:25.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.179+0000 7f0633fff640 1 --2- 172.21.15.60:0/4100833537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f063413ef20 0x7f06340a87e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0620009870 tx=0x7f062002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fbd77ef2125a289 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:25.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.179+0000 7f06337fe640 1 -- 172.21.15.60:0/4100833537 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062002f840 con 0x7f063413ef20 2024-03-27T21:41:25.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.179+0000 7f06337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:25.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.179+0000 7f06337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:25.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.179+0000 7f06337fe640 10 monclient: dump: 2024-03-27T21:41:25.170 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:25.170 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:25.170 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:25.170 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:25.170 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:25.171 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:25.171 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:25.171 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:25.171 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:25.171 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:25.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.179+0000 7f06337fe640 10 monclient: _finish_auth 0 2024-03-27T21:41:25.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.179+0000 7f06337fe640 10 monclient: _check_auth_tickets 2024-03-27T21:41:25.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.179+0000 7f06337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:25.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.179+0000 7f06337fe640 1 -- 172.21.15.60:0/4100833537 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062002f9a0 con 0x7f063413ef20 2024-03-27T21:41:25.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.179+0000 7f063b02f640 5 monclient: authenticate success, global_id 4974 2024-03-27T21:41:25.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.179+0000 7f063b02f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:25.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.179+0000 7f06337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:25.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.179+0000 7f06337fe640 1 -- 172.21.15.60:0/4100833537 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062002fd30 con 0x7f063413ef20 2024-03-27T21:41:25.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.179+0000 7f06337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:25.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.179+0000 7f06337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:25.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.179+0000 7f06337fe640 10 monclient: dump: 2024-03-27T21:41:25.172 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:25.172 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:25.172 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:25.172 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:25.172 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:25.172 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:25.172 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:25.172 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:25.173 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:25.173 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:25.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.179+0000 7f063b02f640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:25.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.179+0000 7f063b02f640 4 set_mon_vals no callback set 2024-03-27T21:41:25.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.180+0000 7f063b02f640 10 monclient: shutdown 2024-03-27T21:41:25.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.180+0000 7f063b02f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:25.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.180+0000 7f063b02f640 1 -- 172.21.15.60:0/4100833537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f063413ef20 msgr2=0x7f06340a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:25.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.180+0000 7f063b02f640 1 --2- 172.21.15.60:0/4100833537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f063413ef20 0x7f06340a87e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0620009870 tx=0x7f062002ba20 comp rx=0 tx=0).stop 2024-03-27T21:41:25.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.180+0000 7f063b02f640 1 -- 172.21.15.60:0/4100833537 shutdown_connections 2024-03-27T21:41:25.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.180+0000 7f063b02f640 1 --2- 172.21.15.60:0/4100833537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f063413ef20 0x7f06340a87e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:25.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.180+0000 7f063b02f640 1 -- 172.21.15.60:0/4100833537 >> 172.21.15.60:0/4100833537 conn(0x7f06340b7300 msgr2=0x7f06340b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:25.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.180+0000 7f063b02f640 1 -- 172.21.15.60:0/4100833537 shutdown_connections 2024-03-27T21:41:25.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.180+0000 7f063b02f640 1 -- 172.21.15.60:0/4100833537 wait complete. 2024-03-27T21:41:25.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.181+0000 7f063b02f640 10 monclient: build_initial_monmap 2024-03-27T21:41:25.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.181+0000 7f063b02f640 10 monclient: monmap: 2024-03-27T21:41:25.181 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:25.181 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:25.181 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:25.181 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:25.181 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:25.181 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:25.181 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:25.181 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:25.181 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:25.181 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:25.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.182+0000 7f063b02f640 1 librados: starting msgr at 2024-03-27T21:41:25.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.182+0000 7f063b02f640 1 librados: starting objecter 2024-03-27T21:41:25.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.182+0000 7f063b02f640 1 Processor -- start 2024-03-27T21:41:25.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.182+0000 7f063b02f640 1 -- start start 2024-03-27T21:41:25.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.182+0000 7f063b02f640 1 librados: setting wanted keys 2024-03-27T21:41:25.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.182+0000 7f063b02f640 1 librados: calling monclient init 2024-03-27T21:41:25.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.182+0000 7f063b02f640 10 monclient: init 2024-03-27T21:41:25.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f063b02f640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:25.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f063b02f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:41:25.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f063b02f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0634071a50 0x7f06340aba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:25.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f063b02f640 10 monclient(hunting): picked mon.noname-b con 0x7f0634071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:25.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f063b02f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f063413ef20 0x7f06340abf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:25.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f063b02f640 10 monclient(hunting): picked mon.noname-c con 0x7f063413ef20 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:25.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f063b02f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0634141630 0x7f06341d6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:25.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f063b02f640 10 monclient(hunting): picked mon.noname-a con 0x7f0634141630 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:25.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f063b02f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:25.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f063b02f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06341d65b0 con 0x7f0634141630 2024-03-27T21:41:25.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f063b02f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:25.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f063b02f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0634138bb0 con 0x7f0634071a50 2024-03-27T21:41:25.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f063b02f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:25.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f063b02f640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f063413a950 con 0x7f063413ef20 2024-03-27T21:41:25.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f063b02f640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:25.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f063b02f640 10 monclient(hunting): authenticate will time out at 926.404147s 2024-03-27T21:41:25.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f06395cc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0634141630 0x7f06341d6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:25.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0633fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f063413ef20 0x7f06340abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:25.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f06395cc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0634141630 0x7f06341d6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36448/0 (socket says 172.21.15.60:36448) 2024-03-27T21:41:25.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f06395cc640 1 -- 172.21.15.60:0/714095734 learned_addr learned my addr 172.21.15.60:0/714095734 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:25.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f06395cc640 10 monclient(hunting): get_auth_request con 0x7f0634141630 auth_method 0 2024-03-27T21:41:25.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f06395cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:25.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f06395cc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:25.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f06395cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:25.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0633fff640 10 monclient(hunting): get_auth_request con 0x7f063413ef20 auth_method 0 2024-03-27T21:41:25.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0633fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:25.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0633fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:25.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0633fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:25.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0638dcb640 1 --2- 172.21.15.60:0/714095734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0634071a50 0x7f06340aba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:25.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0638dcb640 10 monclient(hunting): get_auth_request con 0x7f0634071a50 auth_method 0 2024-03-27T21:41:25.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0638dcb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:25.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0638dcb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:25.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0638dcb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:25.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0633fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:25.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0633fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:25.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0633fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:25.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0638dcb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:25.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0638dcb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:25.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0638dcb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:25.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f06395cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:25.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f06395cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:25.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f06395cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:25.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0638dcb640 10 monclient(hunting): handle_auth_done global_id 4921 payload 824 2024-03-27T21:41:25.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0638dcb640 1 -- 172.21.15.60:0/714095734 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f063413ef20 msgr2=0x7f06340abf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:25.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0638dcb640 1 --2- 172.21.15.60:0/714095734 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f063413ef20 0x7f06340abf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:25.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0638dcb640 1 -- 172.21.15.60:0/714095734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0634141630 msgr2=0x7f06341d6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:25.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0638dcb640 1 --2- 172.21.15.60:0/714095734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0634141630 0x7f06341d6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:25.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0638dcb640 10 monclient: _finish_hunting 0 2024-03-27T21:41:25.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0638dcb640 1 monclient: found mon.noname-b 2024-03-27T21:41:25.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0638dcb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:25.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0638dcb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:41:25.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.183+0000 7f0638dcb640 1 -- 172.21.15.60:0/714095734 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06200094d0 con 0x7f0634071a50 2024-03-27T21:41:25.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f0638dcb640 1 --2- 172.21.15.60:0/714095734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0634071a50 0x7f06340aba10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f062800c390 tx=0x7f062800c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:25.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f06317fa640 10 client.?.objecter ms_handle_connect 0x7f0634071a50 2024-03-27T21:41:25.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f06317fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:25.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f06317fa640 1 -- 172.21.15.60:0/714095734 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06280052d0 con 0x7f0634071a50 2024-03-27T21:41:25.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f06317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:25.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f06317fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:25.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f06317fa640 10 monclient: dump: 2024-03-27T21:41:25.187 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:25.187 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:25.187 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:25.188 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:25.188 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:25.188 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:25.188 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:25.188 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:25.188 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:25.188 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:25.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f06317fa640 10 monclient: _finish_auth 0 2024-03-27T21:41:25.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f06317fa640 10 monclient: _check_auth_tickets 2024-03-27T21:41:25.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f06317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:25.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f06317fa640 1 -- 172.21.15.60:0/714095734 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0628005430 con 0x7f0634071a50 2024-03-27T21:41:25.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f06317fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:25.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f063b02f640 5 monclient: authenticate success, global_id 4921 2024-03-27T21:41:25.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f063b02f640 10 monclient: _renew_subs 2024-03-27T21:41:25.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f063b02f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:25.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f063b02f640 1 -- 172.21.15.60:0/714095734 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06340aaf30 con 0x7f0634071a50 2024-03-27T21:41:25.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f0632ffd640 4 set_mon_vals no callback set 2024-03-27T21:41:25.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f063b02f640 10 client.4921.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:25.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f063b02f640 10 monclient: _renew_subs 2024-03-27T21:41:25.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f063b02f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:25.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f063b02f640 1 -- 172.21.15.60:0/714095734 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06341d7160 con 0x7f0634071a50 2024-03-27T21:41:25.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f063b02f640 1 librados: init done 2024-03-27T21:41:25.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f06317fa640 1 -- 172.21.15.60:0/714095734 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0628003830 con 0x7f0634071a50 2024-03-27T21:41:25.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f06317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:25.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f06317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:25.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.184+0000 7f06317fa640 10 monclient: dump: 2024-03-27T21:41:25.190 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:25.190 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:25.190 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:25.190 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:25.190 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:25.190 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:25.190 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:25.190 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:25.191 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:25.191 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:25.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.185+0000 7f063b02f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:25.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.185+0000 7f063b02f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:25.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.185+0000 7f063b02f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:25.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.185+0000 7f063b02f640 1 -- 172.21.15.60:0/714095734 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06340ad720 con 0x7f0634071a50 2024-03-27T21:41:25.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.186+0000 7f06317fa640 1 -- 172.21.15.60:0/714095734 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0628003990 con 0x7f0634071a50 2024-03-27T21:41:25.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.187+0000 7f06317fa640 1 --2- 172.21.15.60:0/714095734 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f061806c590 0x7f061806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:25.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.187+0000 7f06317fa640 1 -- 172.21.15.60:0/714095734 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f0628088640 con 0x7f0634071a50 2024-03-27T21:41:25.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.187+0000 7f06317fa640 10 client.4921.objecter ms_dispatch 0x7f0634000b90 osd_map(59..59 src has 1..59) v4 2024-03-27T21:41:25.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.187+0000 7f06317fa640 3 client.4921.objecter handle_osd_map got epochs [59,59] > 0 2024-03-27T21:41:25.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.187+0000 7f06317fa640 3 client.4921.objecter handle_osd_map decoding full epoch 59 2024-03-27T21:41:25.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.187+0000 7f06317fa640 20 client.4921.objecter dump_active .. 0 homeless 2024-03-27T21:41:25.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.187+0000 7f0633fff640 1 --2- 172.21.15.60:0/714095734 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f061806c590 0x7f061806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:25.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.187+0000 7f0633fff640 10 monclient: get_auth_request con 0x7f061806c590 auth_method 0 2024-03-27T21:41:25.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.188+0000 7f06317fa640 10 client.4921.objecter ms_handle_connect 0x7f061806c590 2024-03-27T21:41:25.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.188+0000 7f0633fff640 1 --2- 172.21.15.60:0/714095734 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f061806c590 0x7f061806ea50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0620036000 tx=0x7f0620008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:25.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.191+0000 7f06317fa640 1 -- 172.21.15.60:0/714095734 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0628053a90 con 0x7f0634071a50 2024-03-27T21:41:25.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.191+0000 7f06317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:25.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.191+0000 7f06317fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:25.238 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:25.252+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:25.260 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:25.260 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:25.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:25.260 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:41:25.260 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:25.268 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:41:25.339 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:25.353+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:25.374 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:25.374 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:25.374 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:25.374 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:25.383 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:41:25.441 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:25.455+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:25.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.487+0000 7f063b02f640 10 start_mon_command cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:41:25.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.487+0000 7f063b02f640 10 monclient: _send_command 2 [{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:41:25.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.487+0000 7f063b02f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:25.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:25.487+0000 7f063b02f640 1 -- 172.21.15.60:0/714095734 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f0634069470 con 0x7f0634071a50 2024-03-27T21:41:25.529 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:25.529 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:25.529 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:25.530 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:25.530 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:25.538 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:41:25.581 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:25.595+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:25.642 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:25.655+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:25.643 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:25.643 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:25.643 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:25.643 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:41:25.643 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:25.652 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:41:25.743 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:25.757+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:25.798 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:25.799 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:25.799 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:25.799 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:25.807 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:41:25.848 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:25.862+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:25.945 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:25.958+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:25.951 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:25.951 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:25.951 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:25.952 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:25.952 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:25.960 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:41:26.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.048+0000 7f06317fa640 1 -- 172.21.15.60:0/714095734 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v60) v1 ==== 101+0+0 (secure 0 0 0) 0x7f06280570b0 con 0x7f0634071a50 2024-03-27T21:41:26.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.048+0000 7f06317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:41:26.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.048+0000 7f06317fa640 10 monclient: _finish_command 2 = system:0 nodeep-scrub is unset 2024-03-27T21:41:26.035 INFO:teuthology.orchestra.run.smithi060.stderr:nodeep-scrub is unset 2024-03-27T21:41:26.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.051+0000 7f063b02f640 10 librados: watch_flush enter 2024-03-27T21:41:26.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.051+0000 7f063b02f640 10 librados: watch_flush exit 2024-03-27T21:41:26.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.051+0000 7f063b02f640 20 client.4921.objecter shutdown clearing up homeless session... 2024-03-27T21:41:26.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.051+0000 7f063b02f640 10 client.4921.objecter successfully canceled tick 2024-03-27T21:41:26.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.051+0000 7f063b02f640 1 -- 172.21.15.60:0/714095734 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f061806c590 msgr2=0x7f061806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:26.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.051+0000 7f063b02f640 1 --2- 172.21.15.60:0/714095734 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f061806c590 0x7f061806ea50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0620036000 tx=0x7f0620008030 comp rx=0 tx=0).stop 2024-03-27T21:41:26.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.051+0000 7f063b02f640 10 monclient: shutdown 2024-03-27T21:41:26.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.051+0000 7f063b02f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:26.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.051+0000 7f063b02f640 1 -- 172.21.15.60:0/714095734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0634071a50 msgr2=0x7f06340aba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:26.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.051+0000 7f063b02f640 1 --2- 172.21.15.60:0/714095734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0634071a50 0x7f06340aba10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f062800c390 tx=0x7f062800c850 comp rx=0 tx=0).stop 2024-03-27T21:41:26.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.051+0000 7f063b02f640 1 -- 172.21.15.60:0/714095734 shutdown_connections 2024-03-27T21:41:26.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.051+0000 7f063b02f640 1 --2- 172.21.15.60:0/714095734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0634141630 0x7f06341d6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:26.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.051+0000 7f063b02f640 1 --2- 172.21.15.60:0/714095734 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f063413ef20 0x7f06340abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:26.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.051+0000 7f063b02f640 1 --2- 172.21.15.60:0/714095734 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f061806c590 0x7f061806ea50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:26.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.051+0000 7f063b02f640 1 --2- 172.21.15.60:0/714095734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0634071a50 0x7f06340aba10 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:26.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.051+0000 7f063b02f640 1 -- 172.21.15.60:0/714095734 >> 172.21.15.60:0/714095734 conn(0x7f06340b7300 msgr2=0x7f06340b24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:26.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.051+0000 7f063b02f640 1 -- 172.21.15.60:0/714095734 shutdown_connections 2024-03-27T21:41:26.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.051+0000 7f063b02f640 1 -- 172.21.15.60:0/714095734 wait complete. 2024-03-27T21:41:26.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.051+0000 7f063b02f640 1 librados: shutdown 2024-03-27T21:41:26.045 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:26.059+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:26.108 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:26.108 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:26.108 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:26.109 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-03-27T21:41:26.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.109 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:26.109 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:26.109 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 106.689324394, 2024-03-27T21:41:26.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43537978700000002, 2024-03-27T21:41:26.109 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:26.109 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:26.109 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:26.109 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:26.109 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:26.109 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-03-27T21:41:26.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.110 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:26.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:26.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:26.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:26.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:26.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:26.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432019+0000", 2024-03-27T21:41:26.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-03-27T21:41:26.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:26.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432027+0000", 2024-03-27T21:41:26.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1200000000000002e-06 2024-03-27T21:41:26.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:26.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432039+0000", 2024-03-27T21:41:26.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1123e-05 2024-03-27T21:41:26.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:26.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432051+0000", 2024-03-27T21:41:26.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-03-27T21:41:26.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:26.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432157+0000", 2024-03-27T21:41:26.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106846 2024-03-27T21:41:26.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-03-27T21:41:26.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432176+0000", 2024-03-27T21:41:26.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8490000000000001e-05 2024-03-27T21:41:26.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:26.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866170+0000", 2024-03-27T21:41:26.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43399370100000001 2024-03-27T21:41:26.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:26.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866335+0000", 2024-03-27T21:41:26.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016559699999999999 2024-03-27T21:41:26.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:26.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866495+0000", 2024-03-27T21:41:26.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015944200000000001 2024-03-27T21:41:26.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.114 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:26.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867012+0000", 2024-03-27T21:41:26.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0005176 2024-03-27T21:41:26.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.114 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:26.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867334+0000", 2024-03-27T21:41:26.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032158600000000001 2024-03-27T21:41:26.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.114 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:26.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867374+0000", 2024-03-27T21:41:26.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0547000000000003e-05 2024-03-27T21:41:26.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:26.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867400+0000", 2024-03-27T21:41:26.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5958000000000001e-05 2024-03-27T21:41:26.115 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.115 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:26.115 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.115 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:26.115 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:26.115 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 106.191941943, 2024-03-27T21:41:26.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064560310000000001, 2024-03-27T21:41:26.116 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:26.116 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:26.116 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:26.116 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:26.116 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:26.116 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 70 2024-03-27T21:41:26.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.116 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:26.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:26.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:26.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:26.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:26.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:26.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929402+0000", 2024-03-27T21:41:26.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:26.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:26.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929407+0000", 2024-03-27T21:41:26.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-06 2024-03-27T21:41:26.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:26.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929414+0000", 2024-03-27T21:41:26.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5689999999999999e-06 2024-03-27T21:41:26.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:26.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929419+0000", 2024-03-27T21:41:26.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-03-27T21:41:26.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:26.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930245+0000", 2024-03-27T21:41:26.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082521500000000002 2024-03-27T21:41:26.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:26.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930287+0000", 2024-03-27T21:41:26.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2305000000000002e-05 2024-03-27T21:41:26.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:26.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930367+0000", 2024-03-27T21:41:26.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9589e-05 2024-03-27T21:41:26.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:26.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.931214+0000", 2024-03-27T21:41:26.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000847827 2024-03-27T21:41:26.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:26.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935807+0000", 2024-03-27T21:41:26.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045923780000000003 2024-03-27T21:41:26.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:26.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935840+0000", 2024-03-27T21:41:26.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3501999999999999e-05 2024-03-27T21:41:26.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:26.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935859+0000", 2024-03-27T21:41:26.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8845e-05 2024-03-27T21:41:26.121 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.121 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:26.121 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.121 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:26.121 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:26.121 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 106.184682352, 2024-03-27T21:41:26.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071695429999999996, 2024-03-27T21:41:26.121 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:26.121 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:26.121 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:26.122 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:26.122 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:26.122 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-03-27T21:41:26.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.122 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:26.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.122 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:26.122 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:26.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.122 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:26.122 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:26.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:26.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936662+0000", 2024-03-27T21:41:26.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:26.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:26.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936737+0000", 2024-03-27T21:41:26.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5303999999999996e-05 2024-03-27T21:41:26.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:26.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936746+0000", 2024-03-27T21:41:26.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6340000000000007e-06 2024-03-27T21:41:26.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:26.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936753+0000", 2024-03-27T21:41:26.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-03-27T21:41:26.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:26.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937926+0000", 2024-03-27T21:41:26.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011731280000000001 2024-03-27T21:41:26.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:26.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937969+0000", 2024-03-27T21:41:26.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3285999999999999e-05 2024-03-27T21:41:26.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:26.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938040+0000", 2024-03-27T21:41:26.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0358000000000006e-05 2024-03-27T21:41:26.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:26.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939324+0000", 2024-03-27T21:41:26.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012844899999999999 2024-03-27T21:41:26.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:26.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943780+0000", 2024-03-27T21:41:26.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044555630000000001 2024-03-27T21:41:26.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:26.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943811+0000", 2024-03-27T21:41:26.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1415999999999999e-05 2024-03-27T21:41:26.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:26.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943832+0000", 2024-03-27T21:41:26.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0981000000000001e-05 2024-03-27T21:41:26.126 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.126 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:26.127 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.127 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:26.127 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:26.127 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 106.184584336, 2024-03-27T21:41:26.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071139740000000003, 2024-03-27T21:41:26.127 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:26.127 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:26.127 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:26.128 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:26.128 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:26.128 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-03-27T21:41:26.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.128 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:26.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:26.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:26.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:26.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:26.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:26.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936760+0000", 2024-03-27T21:41:26.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:26.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:26.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936765+0000", 2024-03-27T21:41:26.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6330000000000004e-06 2024-03-27T21:41:26.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:26.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936771+0000", 2024-03-27T21:41:26.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0560000000000001e-06 2024-03-27T21:41:26.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:26.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936776+0000", 2024-03-27T21:41:26.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1510000000000003e-06 2024-03-27T21:41:26.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:26.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938605+0000", 2024-03-27T21:41:26.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001829589 2024-03-27T21:41:26.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:26.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938657+0000", 2024-03-27T21:41:26.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1860000000000002e-05 2024-03-27T21:41:26.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:26.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938734+0000", 2024-03-27T21:41:26.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6639999999999998e-05 2024-03-27T21:41:26.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:26.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939562+0000", 2024-03-27T21:41:26.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082803100000000001 2024-03-27T21:41:26.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:26.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943838+0000", 2024-03-27T21:41:26.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042757050000000003 2024-03-27T21:41:26.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:26.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943859+0000", 2024-03-27T21:41:26.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1175e-05 2024-03-27T21:41:26.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:26.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943875+0000", 2024-03-27T21:41:26.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5744e-05 2024-03-27T21:41:26.133 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.133 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:26.133 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.133 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:26.133 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:26.134 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 106.041070285, 2024-03-27T21:41:26.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059712230000000003, 2024-03-27T21:41:26.134 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:26.134 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:26.134 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:26.134 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:26.134 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:26.134 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 112 2024-03-27T21:41:26.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.134 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:26.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:26.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:26.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:26.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:26.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:26.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080274+0000", 2024-03-27T21:41:26.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:26.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:26.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080318+0000", 2024-03-27T21:41:26.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3634999999999998e-05 2024-03-27T21:41:26.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:26.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080329+0000", 2024-03-27T21:41:26.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1232e-05 2024-03-27T21:41:26.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:26.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080333+0000", 2024-03-27T21:41:26.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6009999999999999e-06 2024-03-27T21:41:26.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:26.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083786+0000", 2024-03-27T21:41:26.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034528940000000002 2024-03-27T21:41:26.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:26.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083861+0000", 2024-03-27T21:41:26.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4893000000000003e-05 2024-03-27T21:41:26.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:26.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083981+0000", 2024-03-27T21:41:26.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012058299999999999 2024-03-27T21:41:26.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:26.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086148+0000", 2024-03-27T21:41:26.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021671469999999999 2024-03-27T21:41:26.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:26.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086195+0000", 2024-03-27T21:41:26.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6768999999999999e-05 2024-03-27T21:41:26.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:26.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086224+0000", 2024-03-27T21:41:26.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8758000000000001e-05 2024-03-27T21:41:26.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:26.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086246+0000", 2024-03-27T21:41:26.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2064000000000001e-05 2024-03-27T21:41:26.139 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.139 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:26.139 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.139 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:26.139 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:26.139 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 106.041007338, 2024-03-27T21:41:26.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066196320000000003, 2024-03-27T21:41:26.140 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:26.140 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:26.140 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:26.140 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:26.140 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:26.140 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113 2024-03-27T21:41:26.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.140 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:26.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:26.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:26.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:26.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:26.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:26.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080337+0000", 2024-03-27T21:41:26.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:26.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:26.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080340+0000", 2024-03-27T21:41:26.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6690000000000002e-06 2024-03-27T21:41:26.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:26.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080344+0000", 2024-03-27T21:41:26.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6550000000000002e-06 2024-03-27T21:41:26.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:26.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080348+0000", 2024-03-27T21:41:26.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9169999999999999e-06 2024-03-27T21:41:26.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:26.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084595+0000", 2024-03-27T21:41:26.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042471719999999996 2024-03-27T21:41:26.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:26.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084676+0000", 2024-03-27T21:41:26.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1147000000000001e-05 2024-03-27T21:41:26.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:26.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084796+0000", 2024-03-27T21:41:26.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012032900000000001 2024-03-27T21:41:26.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:26.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086190+0000", 2024-03-27T21:41:26.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013934399999999999 2024-03-27T21:41:26.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:26.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086905+0000", 2024-03-27T21:41:26.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071536199999999999 2024-03-27T21:41:26.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:26.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086937+0000", 2024-03-27T21:41:26.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2125000000000002e-05 2024-03-27T21:41:26.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:26.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086957+0000", 2024-03-27T21:41:26.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0157999999999999e-05 2024-03-27T21:41:26.144 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.145 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:26.145 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.145 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:26.145 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:26.145 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 106.040994622, 2024-03-27T21:41:26.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007405661, 2024-03-27T21:41:26.145 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:26.145 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:26.145 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:26.145 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:26.145 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:26.145 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 114 2024-03-27T21:41:26.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.146 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:26.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:26.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:26.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:26.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:26.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:26.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:26.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-03-27T21:41:26.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:26.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080353+0000", 2024-03-27T21:41:26.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.441e-06 2024-03-27T21:41:26.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:26.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080356+0000", 2024-03-27T21:41:26.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0769999999999999e-06 2024-03-27T21:41:26.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:26.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080358+0000", 2024-03-27T21:41:26.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3369999999999998e-06 2024-03-27T21:41:26.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:26.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085118+0000", 2024-03-27T21:41:26.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047603949999999997 2024-03-27T21:41:26.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:26.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085210+0000", 2024-03-27T21:41:26.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1983000000000003e-05 2024-03-27T21:41:26.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:26.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085332+0000", 2024-03-27T21:41:26.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121803 2024-03-27T21:41:26.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:26.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086292+0000", 2024-03-27T21:41:26.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095961400000000002 2024-03-27T21:41:26.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:26.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087710+0000", 2024-03-27T21:41:26.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014181459999999999 2024-03-27T21:41:26.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:26.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087736+0000", 2024-03-27T21:41:26.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6475e-05 2024-03-27T21:41:26.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:26.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087756+0000", 2024-03-27T21:41:26.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9626000000000001e-05 2024-03-27T21:41:26.150 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.150 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:26.150 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.150 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:26.151 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:26.151 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 106.04097218699999, 2024-03-27T21:41:26.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075011720000000004, 2024-03-27T21:41:26.151 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:26.151 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:26.151 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:26.151 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:26.151 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:26.151 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115 2024-03-27T21:41:26.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.151 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:26.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:26.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:26.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:26.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:26.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:26.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:26.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:26.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:26.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080458+0000", 2024-03-27T21:41:26.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5228000000000005e-05 2024-03-27T21:41:26.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:26.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080465+0000", 2024-03-27T21:41:26.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.238e-06 2024-03-27T21:41:26.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:26.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080468+0000", 2024-03-27T21:41:26.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5109999999999999e-06 2024-03-27T21:41:26.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:26.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086326+0000", 2024-03-27T21:41:26.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058579169999999998 2024-03-27T21:41:26.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:26.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086382+0000", 2024-03-27T21:41:26.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5309999999999997e-05 2024-03-27T21:41:26.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:26.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086457+0000", 2024-03-27T21:41:26.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5413000000000005e-05 2024-03-27T21:41:26.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:26.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087762+0000", 2024-03-27T21:41:26.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013052300000000001 2024-03-27T21:41:26.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:26.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087798+0000", 2024-03-27T21:41:26.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5614999999999997e-05 2024-03-27T21:41:26.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:26.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087840+0000", 2024-03-27T21:41:26.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2196000000000001e-05 2024-03-27T21:41:26.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:26.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087874+0000", 2024-03-27T21:41:26.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3785999999999998e-05 2024-03-27T21:41:26.156 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.156 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:26.156 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.156 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:26.156 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:26.156 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 106.040872104, 2024-03-27T21:41:26.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075050680000000002, 2024-03-27T21:41:26.156 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:26.156 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:26.156 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:26.156 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:26.157 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:26.157 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 116 2024-03-27T21:41:26.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.157 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:26.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:26.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:26.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:26.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:26.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:26.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080472+0000", 2024-03-27T21:41:26.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:26.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:26.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080475+0000", 2024-03-27T21:41:26.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.712e-06 2024-03-27T21:41:26.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:26.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080478+0000", 2024-03-27T21:41:26.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1070000000000001e-06 2024-03-27T21:41:26.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:26.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080481+0000", 2024-03-27T21:41:26.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4269999999999998e-06 2024-03-27T21:41:26.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:26.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086992+0000", 2024-03-27T21:41:26.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065114099999999996 2024-03-27T21:41:26.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:26.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087064+0000", 2024-03-27T21:41:26.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1927000000000004e-05 2024-03-27T21:41:26.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.160 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:26.160 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087188+0000", 2024-03-27T21:41:26.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123585 2024-03-27T21:41:26.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.160 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:26.160 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087766+0000", 2024-03-27T21:41:26.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057814300000000004 2024-03-27T21:41:26.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.160 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:26.160 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087934+0000", 2024-03-27T21:41:26.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000167894 2024-03-27T21:41:26.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:26.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087958+0000", 2024-03-27T21:41:26.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4663000000000001e-05 2024-03-27T21:41:26.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:26.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087978+0000", 2024-03-27T21:41:26.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9636e-05 2024-03-27T21:41:26.161 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.161 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:26.161 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.162 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:26.162 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:26.162 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 106.04083998199999, 2024-03-27T21:41:26.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076850679999999998, 2024-03-27T21:41:26.162 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:26.162 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:26.162 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:26.162 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:26.162 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:26.162 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 117 2024-03-27T21:41:26.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.162 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:26.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:26.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:26.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:26.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:26.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:26.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:26.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:26.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:26.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080508+0000", 2024-03-27T21:41:26.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.298e-06 2024-03-27T21:41:26.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:26.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080515+0000", 2024-03-27T21:41:26.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0790000000000003e-06 2024-03-27T21:41:26.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:26.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080518+0000", 2024-03-27T21:41:26.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8509999999999999e-06 2024-03-27T21:41:26.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:26.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087444+0000", 2024-03-27T21:41:26.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0069259120000000002 2024-03-27T21:41:26.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:26.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087491+0000", 2024-03-27T21:41:26.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7043999999999999e-05 2024-03-27T21:41:26.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:26.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087564+0000", 2024-03-27T21:41:26.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3603999999999995e-05 2024-03-27T21:41:26.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:26.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087902+0000", 2024-03-27T21:41:26.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033734100000000001 2024-03-27T21:41:26.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:26.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088117+0000", 2024-03-27T21:41:26.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000215164 2024-03-27T21:41:26.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:26.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088160+0000", 2024-03-27T21:41:26.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2933999999999999e-05 2024-03-27T21:41:26.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.167 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:26.167 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088190+0000", 2024-03-27T21:41:26.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0139999999999999e-05 2024-03-27T21:41:26.167 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.167 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:26.167 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.167 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:26.167 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:26.167 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 105.624559593, 2024-03-27T21:41:26.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052465439999999997, 2024-03-27T21:41:26.167 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:26.167 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:26.168 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:26.168 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:26.168 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:26.168 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 133 2024-03-27T21:41:26.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.168 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:26.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:26.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:26.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:26.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:26.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:26.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:26.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:26.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:26.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496788+0000", 2024-03-27T21:41:26.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-03-27T21:41:26.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:26.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496793+0000", 2024-03-27T21:41:26.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4790000000000004e-06 2024-03-27T21:41:26.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:26.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496796+0000", 2024-03-27T21:41:26.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1e-06 2024-03-27T21:41:26.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:26.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500447+0000", 2024-03-27T21:41:26.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036504459999999999 2024-03-27T21:41:26.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:26.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500492+0000", 2024-03-27T21:41:26.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5765e-05 2024-03-27T21:41:26.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:26.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500562+0000", 2024-03-27T21:41:26.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9206000000000001e-05 2024-03-27T21:41:26.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:26.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501959+0000", 2024-03-27T21:41:26.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001397187 2024-03-27T21:41:26.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:26.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501984+0000", 2024-03-27T21:41:26.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.552e-05 2024-03-27T21:41:26.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:26.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502010+0000", 2024-03-27T21:41:26.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5232000000000002e-05 2024-03-27T21:41:26.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:26.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502032+0000", 2024-03-27T21:41:26.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2337999999999999e-05 2024-03-27T21:41:26.172 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.172 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:26.172 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.173 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:26.173 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:26.173 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 105.624500935, 2024-03-27T21:41:26.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052892119999999997, 2024-03-27T21:41:26.173 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:26.173 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:26.173 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:26.173 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:26.173 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:26.173 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 134 2024-03-27T21:41:26.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.173 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:26.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:26.174 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:26.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:26.174 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:26.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:26.174 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:26.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:26.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:26.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496929+0000", 2024-03-27T21:41:26.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5098000000000004e-05 2024-03-27T21:41:26.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:26.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496937+0000", 2024-03-27T21:41:26.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9559999999999997e-06 2024-03-27T21:41:26.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:26.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496942+0000", 2024-03-27T21:41:26.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6350000000000003e-06 2024-03-27T21:41:26.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:26.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500749+0000", 2024-03-27T21:41:26.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038068529999999998 2024-03-27T21:41:26.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:26.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500792+0000", 2024-03-27T21:41:26.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2823e-05 2024-03-27T21:41:26.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:26.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500864+0000", 2024-03-27T21:41:26.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1715e-05 2024-03-27T21:41:26.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:26.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502068+0000", 2024-03-27T21:41:26.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012044860000000001 2024-03-27T21:41:26.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:26.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502073+0000", 2024-03-27T21:41:26.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5700000000000003e-06 2024-03-27T21:41:26.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:26.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502098+0000", 2024-03-27T21:41:26.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5120999999999999e-05 2024-03-27T21:41:26.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:26.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502133+0000", 2024-03-27T21:41:26.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5306999999999999e-05 2024-03-27T21:41:26.178 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.178 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:26.178 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.178 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:26.178 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:26.179 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 105.624395628, 2024-03-27T21:41:26.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062529040000000001, 2024-03-27T21:41:26.179 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:26.179 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:26.179 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:26.179 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:26.179 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:26.179 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135 2024-03-27T21:41:26.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.179 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:26.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:26.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:26.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:26.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:26.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:26.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:26.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:26.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:26.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496953+0000", 2024-03-27T21:41:26.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0709999999999996e-06 2024-03-27T21:41:26.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:26.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496961+0000", 2024-03-27T21:41:26.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1300000000000001e-06 2024-03-27T21:41:26.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:26.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496968+0000", 2024-03-27T21:41:26.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0999999999999998e-06 2024-03-27T21:41:26.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:26.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501244+0000", 2024-03-27T21:41:26.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042756310000000002 2024-03-27T21:41:26.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:26.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501377+0000", 2024-03-27T21:41:26.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013348799999999999 2024-03-27T21:41:26.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:26.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501455+0000", 2024-03-27T21:41:26.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7410000000000006e-05 2024-03-27T21:41:26.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:26.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502138+0000", 2024-03-27T21:41:26.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068386599999999999 2024-03-27T21:41:26.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:26.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503152+0000", 2024-03-27T21:41:26.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010139680000000001 2024-03-27T21:41:26.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:26.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503180+0000", 2024-03-27T21:41:26.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7426000000000001e-05 2024-03-27T21:41:26.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:26.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503202+0000", 2024-03-27T21:41:26.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2463000000000002e-05 2024-03-27T21:41:26.184 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.184 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:26.184 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.184 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:26.184 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:26.184 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 105.62437149, 2024-03-27T21:41:26.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066253680000000004, 2024-03-27T21:41:26.184 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:26.184 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:26.184 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:26.185 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:26.185 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:26.185 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 136 2024-03-27T21:41:26.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.185 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:26.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:26.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:26.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:26.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:26.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:26.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:26.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:26.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:26.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496977+0000", 2024-03-27T21:41:26.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5049999999999999e-06 2024-03-27T21:41:26.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:26.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496984+0000", 2024-03-27T21:41:26.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1520000000000003e-06 2024-03-27T21:41:26.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:26.187 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496990+0000", 2024-03-27T21:41:26.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-03-27T21:41:26.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:26.187 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501669+0000", 2024-03-27T21:41:26.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046793859999999998 2024-03-27T21:41:26.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:26.187 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501714+0000", 2024-03-27T21:41:26.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5154999999999997e-05 2024-03-27T21:41:26.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:26.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501783+0000", 2024-03-27T21:41:26.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8454999999999994e-05 2024-03-27T21:41:26.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:26.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503226+0000", 2024-03-27T21:41:26.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001443441 2024-03-27T21:41:26.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:26.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503540+0000", 2024-03-27T21:41:26.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031347100000000001 2024-03-27T21:41:26.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:26.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503563+0000", 2024-03-27T21:41:26.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3223000000000001e-05 2024-03-27T21:41:26.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:26.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503599+0000", 2024-03-27T21:41:26.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6105000000000001e-05 2024-03-27T21:41:26.189 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.189 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:26.189 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.190 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:26.190 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:26.190 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 105.62434868, 2024-03-27T21:41:26.190 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074261320000000002, 2024-03-27T21:41:26.190 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:26.190 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:26.190 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:26.190 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:26.190 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:26.190 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 137 2024-03-27T21:41:26.190 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.190 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:26.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.190 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:26.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:26.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:26.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:26.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:26.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:26.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:26.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:26.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497077+0000", 2024-03-27T21:41:26.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1599000000000003e-05 2024-03-27T21:41:26.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:26.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497085+0000", 2024-03-27T21:41:26.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6109999999999996e-06 2024-03-27T21:41:26.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:26.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497092+0000", 2024-03-27T21:41:26.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.742e-06 2024-03-27T21:41:26.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.193 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:26.193 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502159+0000", 2024-03-27T21:41:26.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050666870000000003 2024-03-27T21:41:26.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.193 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:26.193 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502200+0000", 2024-03-27T21:41:26.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1919999999999998e-05 2024-03-27T21:41:26.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.193 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:26.193 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502280+0000", 2024-03-27T21:41:26.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9417000000000002e-05 2024-03-27T21:41:26.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:26.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503605+0000", 2024-03-27T21:41:26.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001325641 2024-03-27T21:41:26.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:26.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504361+0000", 2024-03-27T21:41:26.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075555399999999997 2024-03-27T21:41:26.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:26.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504386+0000", 2024-03-27T21:41:26.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5369999999999999e-05 2024-03-27T21:41:26.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.195 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:26.195 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504422+0000", 2024-03-27T21:41:26.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6019000000000002e-05 2024-03-27T21:41:26.195 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.195 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:26.195 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.195 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:26.195 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:26.195 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 105.624244883, 2024-03-27T21:41:26.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074059750000000004, 2024-03-27T21:41:26.196 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:26.196 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:26.196 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:26.196 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:26.196 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:26.196 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-03-27T21:41:26.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.196 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:26.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.196 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:26.196 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:26.196 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:26.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:26.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:26.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:26.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:26.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:26.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497103+0000", 2024-03-27T21:41:26.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.856e-06 2024-03-27T21:41:26.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:26.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497121+0000", 2024-03-27T21:41:26.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7190999999999999e-05 2024-03-27T21:41:26.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:26.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497126+0000", 2024-03-27T21:41:26.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3770000000000004e-06 2024-03-27T21:41:26.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:26.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502833+0000", 2024-03-27T21:41:26.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057073510000000003 2024-03-27T21:41:26.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:26.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502881+0000", 2024-03-27T21:41:26.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7961999999999999e-05 2024-03-27T21:41:26.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:26.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502949+0000", 2024-03-27T21:41:26.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7957e-05 2024-03-27T21:41:26.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:26.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504430+0000", 2024-03-27T21:41:26.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001480763 2024-03-27T21:41:26.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:26.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504459+0000", 2024-03-27T21:41:26.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8983999999999999e-05 2024-03-27T21:41:26.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:26.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504486+0000", 2024-03-27T21:41:26.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6883999999999999e-05 2024-03-27T21:41:26.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:26.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504506+0000", 2024-03-27T21:41:26.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0214e-05 2024-03-27T21:41:26.201 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.201 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:26.201 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.201 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.201 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:26.201 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:26.201 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 105.62421304900001, 2024-03-27T21:41:26.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074754859999999999, 2024-03-27T21:41:26.201 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:26.201 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:26.201 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:26.201 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:26.201 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:26.201 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-03-27T21:41:26.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.202 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:26.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:26.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:26.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:26.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:26.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497137+0000", 2024-03-27T21:41:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0730000000000004e-06 2024-03-27T21:41:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497143+0000", 2024-03-27T21:41:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3759999999999996e-06 2024-03-27T21:41:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497150+0000", 2024-03-27T21:41:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.002e-06 2024-03-27T21:41:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503249+0000", 2024-03-27T21:41:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060989750000000004 2024-03-27T21:41:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503291+0000", 2024-03-27T21:41:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1563999999999997e-05 2024-03-27T21:41:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503376+0000", 2024-03-27T21:41:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5752000000000002e-05 2024-03-27T21:41:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504545+0000", 2024-03-27T21:41:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169172 2024-03-27T21:41:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504550+0000", 2024-03-27T21:41:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7419999999999997e-06 2024-03-27T21:41:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504574+0000", 2024-03-27T21:41:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4124000000000001e-05 2024-03-27T21:41:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504607+0000", 2024-03-27T21:41:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3118999999999999e-05 2024-03-27T21:41:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:26.206 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 105.624189383, 2024-03-27T21:41:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078150589999999992, 2024-03-27T21:41:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-03-27T21:41:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.207 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497155+0000", 2024-03-27T21:41:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497230+0000", 2024-03-27T21:41:26.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4789000000000005e-05 2024-03-27T21:41:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497241+0000", 2024-03-27T21:41:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1442e-05 2024-03-27T21:41:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497255+0000", 2024-03-27T21:41:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3087e-05 2024-03-27T21:41:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503630+0000", 2024-03-27T21:41:26.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063755269999999998 2024-03-27T21:41:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503673+0000", 2024-03-27T21:41:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2616e-05 2024-03-27T21:41:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503738+0000", 2024-03-27T21:41:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5114e-05 2024-03-27T21:41:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504907+0000", 2024-03-27T21:41:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169432 2024-03-27T21:41:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504926+0000", 2024-03-27T21:41:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9086999999999999e-05 2024-03-27T21:41:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504950+0000", 2024-03-27T21:41:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3598999999999999e-05 2024-03-27T21:41:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:26.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504971+0000", 2024-03-27T21:41:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0772999999999999e-05 2024-03-27T21:41:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 105.624082343, 2024-03-27T21:41:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082962590000000003, 2024-03-27T21:41:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:26.212 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-03-27T21:41:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497262+0000", 2024-03-27T21:41:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497267+0000", 2024-03-27T21:41:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.955e-06 2024-03-27T21:41:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497275+0000", 2024-03-27T21:41:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.824e-06 2024-03-27T21:41:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497280+0000", 2024-03-27T21:41:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8230000000000002e-06 2024-03-27T21:41:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504628+0000", 2024-03-27T21:41:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007348561 2024-03-27T21:41:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504673+0000", 2024-03-27T21:41:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4313000000000001e-05 2024-03-27T21:41:26.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504745+0000", 2024-03-27T21:41:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2303000000000002e-05 2024-03-27T21:41:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505358+0000", 2024-03-27T21:41:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061281300000000003 2024-03-27T21:41:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505503+0000", 2024-03-27T21:41:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000145134 2024-03-27T21:41:26.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505526+0000", 2024-03-27T21:41:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3312e-05 2024-03-27T21:41:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505559+0000", 2024-03-27T21:41:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2821999999999998e-05 2024-03-27T21:41:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.217 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 105.01486010399999, 2024-03-27T21:41:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053753339999999998, 2024-03-27T21:41:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159 2024-03-27T21:41:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:26.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106488+0000", 2024-03-27T21:41:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9830000000000001e-06 2024-03-27T21:41:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106491+0000", 2024-03-27T21:41:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3459999999999998e-06 2024-03-27T21:41:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106493+0000", 2024-03-27T21:41:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.509e-06 2024-03-27T21:41:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110636+0000", 2024-03-27T21:41:26.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041427260000000002 2024-03-27T21:41:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110683+0000", 2024-03-27T21:41:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6557000000000002e-05 2024-03-27T21:41:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110755+0000", 2024-03-27T21:41:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2119000000000005e-05 2024-03-27T21:41:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111415+0000", 2024-03-27T21:41:26.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066065199999999996 2024-03-27T21:41:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111793+0000", 2024-03-27T21:41:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037732100000000001 2024-03-27T21:41:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111829+0000", 2024-03-27T21:41:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418999999999998e-05 2024-03-27T21:41:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:26.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111860+0000", 2024-03-27T21:41:26.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1003000000000001e-05 2024-03-27T21:41:26.223 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.223 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:26.223 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.223 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:26.223 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:26.223 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:26.224 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:26.238+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:26.225 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:41:26.325 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:26.339+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:26.334 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:26.334 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:26.334 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:26.334 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:26.343 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:41:26.381 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:41:26.427 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:26.441+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:26.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.501+0000 7f2c92e8b640 10 monclient: get_monmap_and_config 2024-03-27T21:41:26.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.501+0000 7f2c92e8b640 10 monclient: build_initial_monmap 2024-03-27T21:41:26.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.501+0000 7f2c92e8b640 10 monclient: monmap: 2024-03-27T21:41:26.488 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:26.488 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:26.488 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:26.502552+0000 2024-03-27T21:41:26.488 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:26.502552+0000 2024-03-27T21:41:26.488 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:26.488 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:26.488 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:26.488 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:26.488 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:26.489 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:26.489 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:26.489 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:26.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.502+0000 7f2c92e8b640 1 Processor -- start 2024-03-27T21:41:26.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.502+0000 7f2c92e8b640 1 -- start start 2024-03-27T21:41:26.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.502+0000 7f2c92e8b640 10 monclient: init 2024-03-27T21:41:26.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.502+0000 7f2c92e8b640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:26.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.502+0000 7f2c92e8b640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-03-27T21:41:26.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.502+0000 7f2c92e8b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2c8c0b40b0 0x7f2c8c0a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:26.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.502+0000 7f2c92e8b640 10 monclient(hunting): picked mon.noname-c con 0x7f2c8c0b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:26.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c92e8b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2c8c13b020 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:26.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c92e8b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2c8c13b480 0x7f2c8c141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:26.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c92e8b640 10 monclient(hunting): picked mon.noname-a con 0x7f2c8c13b480 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:26.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c92e8b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c8c13b8d0 con 0x7f2c8c13b020 2024-03-27T21:41:26.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c92e8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:26.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c92e8b640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c8c0b4490 con 0x7f2c8c13b480 2024-03-27T21:41:26.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c92e8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:26.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c92e8b640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c8c142340 con 0x7f2c8c0b40b0 2024-03-27T21:41:26.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c92e8b640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:26.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c92e8b640 10 monclient(hunting): authenticate will time out at 927.724203s 2024-03-27T21:41:26.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c90c27640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2c8c0b40b0 0x7f2c8c0a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:26.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c91428640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2c8c13b480 0x7f2c8c141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:26.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c8bfff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f2c8c13b020 0x7f2c8c1416f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.60:44376/0 (socket says 172.21.15.60:44376) 2024-03-27T21:41:26.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c8bfff640 1 -- 172.21.15.60:0/3114864973 learned_addr learned my addr 172.21.15.60:0/3114864973 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:26.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c90c27640 10 monclient(hunting): get_auth_request con 0x7f2c8c0b40b0 auth_method 0 2024-03-27T21:41:26.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c90c27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:26.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c90c27640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:26.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c90c27640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:26.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c91428640 10 monclient(hunting): get_auth_request con 0x7f2c8c13b480 auth_method 0 2024-03-27T21:41:26.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c91428640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:26.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c91428640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:26.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c91428640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:26.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c90c27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:26.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c90c27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:26.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c90c27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:26.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c91428640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:26.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c91428640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:26.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c91428640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:26.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c90c27640 10 monclient(hunting): handle_auth_done global_id 4924 payload 274 2024-03-27T21:41:26.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c90c27640 1 -- 172.21.15.60:0/3114864973 >> v2:172.21.15.60:3300/0 conn(0x7f2c8c13b480 msgr2=0x7f2c8c141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:26.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c90c27640 1 --2- 172.21.15.60:0/3114864973 >> v2:172.21.15.60:3300/0 conn(0x7f2c8c13b480 0x7f2c8c141e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:26.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c90c27640 1 -- 172.21.15.60:0/3114864973 >> v1:172.21.15.191:6789/0 conn(0x7f2c8c13b020 legacy=0x7f2c8c1416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:26.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c90c27640 10 monclient: _finish_hunting 0 2024-03-27T21:41:26.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c90c27640 1 monclient: found mon.noname-c 2024-03-27T21:41:26.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c90c27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:26.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c90c27640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:41:26.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c90c27640 1 -- 172.21.15.60:0/3114864973 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8c0b3830 con 0x7f2c8c0b40b0 2024-03-27T21:41:26.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c8b7fe640 1 -- 172.21.15.60:0/3114864973 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880448913 0 0) 0x7f2c8c13b8d0 con 0x7f2c8c13b020 2024-03-27T21:41:26.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.503+0000 7f2c8b7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:41:26.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.504+0000 7f2c90c27640 1 --2- 172.21.15.60:0/3114864973 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c8c0b40b0 0x7f2c8c0a8b90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c0b3a20 tx=0x7f2c8002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae208fd4ff05cf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:26.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.504+0000 7f2c8b7fe640 1 -- 172.21.15.60:0/3114864973 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8002e070 con 0x7f2c8c0b40b0 2024-03-27T21:41:26.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.504+0000 7f2c8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:26.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.504+0000 7f2c8b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:26.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.504+0000 7f2c8b7fe640 10 monclient: dump: 2024-03-27T21:41:26.503 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:26.503 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:26.503 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:26.503 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:26.504 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:26.504 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:26.504 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:26.504 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:26.504 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:26.504 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:26.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.504+0000 7f2c8b7fe640 10 monclient: _finish_auth 0 2024-03-27T21:41:26.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.504+0000 7f2c8b7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:41:26.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.504+0000 7f2c8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:26.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.504+0000 7f2c92e8b640 5 monclient: authenticate success, global_id 4924 2024-03-27T21:41:26.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.504+0000 7f2c8b7fe640 1 -- 172.21.15.60:0/3114864973 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c800315a0 con 0x7f2c8c0b40b0 2024-03-27T21:41:26.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.504+0000 7f2c92e8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:26.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.504+0000 7f2c8b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:26.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.504+0000 7f2c92e8b640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:26.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.504+0000 7f2c8b7fe640 1 -- 172.21.15.60:0/3114864973 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c80031930 con 0x7f2c8c0b40b0 2024-03-27T21:41:26.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.504+0000 7f2c92e8b640 4 set_mon_vals no callback set 2024-03-27T21:41:26.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.504+0000 7f2c8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:26.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.504+0000 7f2c8b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:26.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.504+0000 7f2c8b7fe640 10 monclient: dump: 2024-03-27T21:41:26.505 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:26.505 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:26.505 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:26.505 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:26.505 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:26.505 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:26.505 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:26.505 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:26.505 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:26.505 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:26.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.505+0000 7f2c92e8b640 10 monclient: shutdown 2024-03-27T21:41:26.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.505+0000 7f2c92e8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:26.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.505+0000 7f2c92e8b640 1 -- 172.21.15.60:0/3114864973 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c8c0b40b0 msgr2=0x7f2c8c0a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:26.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.505+0000 7f2c92e8b640 1 --2- 172.21.15.60:0/3114864973 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c8c0b40b0 0x7f2c8c0a8b90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c0b3a20 tx=0x7f2c8002b840 comp rx=0 tx=0).stop 2024-03-27T21:41:26.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.505+0000 7f2c92e8b640 1 -- 172.21.15.60:0/3114864973 shutdown_connections 2024-03-27T21:41:26.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.505+0000 7f2c92e8b640 1 --2- 172.21.15.60:0/3114864973 >> v2:172.21.15.60:3300/0 conn(0x7f2c8c13b480 0x7f2c8c141e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:26.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.505+0000 7f2c92e8b640 1 --2- 172.21.15.60:0/3114864973 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c8c0b40b0 0x7f2c8c0a8b90 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:26.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.505+0000 7f2c92e8b640 1 -- 172.21.15.60:0/3114864973 >> 172.21.15.60:0/3114864973 conn(0x7f2c8c0608e0 msgr2=0x7f2c8c060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:26.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.505+0000 7f2c92e8b640 1 -- 172.21.15.60:0/3114864973 shutdown_connections 2024-03-27T21:41:26.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.505+0000 7f2c92e8b640 1 -- 172.21.15.60:0/3114864973 wait complete. 2024-03-27T21:41:26.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.506+0000 7f2c92e8b640 10 monclient: build_initial_monmap 2024-03-27T21:41:26.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.506+0000 7f2c92e8b640 10 monclient: monmap: 2024-03-27T21:41:26.506 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:26.506 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:26.506 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:26.506 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:26.506 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:26.507 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:26.507 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:26.507 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:26.507 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:26.507 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:26.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.507+0000 7f2c92e8b640 1 librados: starting msgr at 2024-03-27T21:41:26.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.507+0000 7f2c92e8b640 1 librados: starting objecter 2024-03-27T21:41:26.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.507+0000 7f2c92e8b640 1 Processor -- start 2024-03-27T21:41:26.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.507+0000 7f2c92e8b640 1 -- start start 2024-03-27T21:41:26.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.507+0000 7f2c92e8b640 1 librados: setting wanted keys 2024-03-27T21:41:26.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.507+0000 7f2c92e8b640 1 librados: calling monclient init 2024-03-27T21:41:26.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.507+0000 7f2c92e8b640 10 monclient: init 2024-03-27T21:41:26.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c92e8b640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:26.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c92e8b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:41:26.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c92e8b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c8c0b40b0 0x7f2c8c13b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:26.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c92e8b640 10 monclient(hunting): picked mon.noname-b con 0x7f2c8c0b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:26.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c92e8b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c8c13b020 0x7f2c8c1368d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:26.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c92e8b640 10 monclient(hunting): picked mon.noname-a con 0x7f2c8c13b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:26.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c8bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c8c13b020 0x7f2c8c1368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:26.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c8bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c8c13b020 0x7f2c8c1368d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36460/0 (socket says 172.21.15.60:36460) 2024-03-27T21:41:26.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c8bfff640 1 -- 172.21.15.60:0/538744612 learned_addr learned my addr 172.21.15.60:0/538744612 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:26.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c92e8b640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2c8c13b480 0x7f2c8c131110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:26.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c92e8b640 10 monclient(hunting): picked mon.noname-c con 0x7f2c8c13b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:26.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c92e8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:26.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c13c5b0 con 0x7f2c8c13b020 2024-03-27T21:41:26.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c92e8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:26.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c13ae40 con 0x7f2c8c0b40b0 2024-03-27T21:41:26.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c92e8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:26.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c13beb0 con 0x7f2c8c13b480 2024-03-27T21:41:26.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c92e8b640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:26.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c92e8b640 10 monclient(hunting): authenticate will time out at 927.729608s 2024-03-27T21:41:26.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c8bfff640 10 monclient(hunting): get_auth_request con 0x7f2c8c13b020 auth_method 0 2024-03-27T21:41:26.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:26.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:26.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:26.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c91428640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2c8c13b480 0x7f2c8c131110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:26.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:26.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:26.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c90c27640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c8c0b40b0 0x7f2c8c13b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:26.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:26.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c90c27640 10 monclient(hunting): get_auth_request con 0x7f2c8c0b40b0 auth_method 0 2024-03-27T21:41:26.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c90c27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:26.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c90c27640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:26.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c90c27640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:26.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c91428640 10 monclient(hunting): get_auth_request con 0x7f2c8c13b480 auth_method 0 2024-03-27T21:41:26.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c91428640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:26.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c91428640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:26.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c91428640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:26.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c91428640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:26.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.508+0000 7f2c91428640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:26.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c91428640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:26.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c8bfff640 10 monclient(hunting): handle_auth_done global_id 4986 payload 824 2024-03-27T21:41:26.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c8bfff640 1 -- 172.21.15.60:0/538744612 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2c8c13b480 msgr2=0x7f2c8c131110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:26.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c8bfff640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2c8c13b480 0x7f2c8c131110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:26.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c8bfff640 1 -- 172.21.15.60:0/538744612 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c8c0b40b0 msgr2=0x7f2c8c13b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:26.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c8bfff640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c8c0b40b0 0x7f2c8c13b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:26.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c8bfff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:26.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c8bfff640 1 monclient: found mon.noname-a 2024-03-27T21:41:26.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:26.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c8bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:26.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c8bfff640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c80009400 con 0x7f2c8c13b020 2024-03-27T21:41:26.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c91428640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2c8c13b480 0x7f2c8c131110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:26.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c90c27640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c8c0b40b0 0x7f2c8c13b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:41:26.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c8bfff640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c8c13b020 0x7f2c8c1368d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2c7800c500 tx=0x7f2c7800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:26.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c897fa640 10 client.?.objecter ms_handle_connect 0x7f2c8c13b020 2024-03-27T21:41:26.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c897fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:26.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c897fa640 1 -- 172.21.15.60:0/538744612 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c78007560 con 0x7f2c8c13b020 2024-03-27T21:41:26.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:26.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c897fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:26.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c897fa640 10 monclient: dump: 2024-03-27T21:41:26.511 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:26.511 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:26.511 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:26.511 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:26.511 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:26.511 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:26.512 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:26.512 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:26.512 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:26.512 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:26.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c897fa640 10 monclient: _finish_auth 0 2024-03-27T21:41:26.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c897fa640 10 monclient: _check_auth_tickets 2024-03-27T21:41:26.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:26.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c897fa640 1 -- 172.21.15.60:0/538744612 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c780047e0 con 0x7f2c8c13b020 2024-03-27T21:41:26.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c897fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:26.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c897fa640 1 -- 172.21.15.60:0/538744612 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c78004b90 con 0x7f2c8c13b020 2024-03-27T21:41:26.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:26.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c897fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:26.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c897fa640 10 monclient: dump: 2024-03-27T21:41:26.513 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:26.513 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:26.513 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:26.513 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:26.513 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:26.513 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:26.513 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:26.513 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:26.513 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:26.513 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:26.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c92e8b640 5 monclient: authenticate success, global_id 4986 2024-03-27T21:41:26.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c92e8b640 10 monclient: _renew_subs 2024-03-27T21:41:26.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c92e8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:26.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c8c13a8a0 con 0x7f2c8c13b020 2024-03-27T21:41:26.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c8affd640 4 set_mon_vals no callback set 2024-03-27T21:41:26.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c92e8b640 10 client.4986.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:26.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c92e8b640 10 monclient: _renew_subs 2024-03-27T21:41:26.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c92e8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:26.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c8c131650 con 0x7f2c8c13b020 2024-03-27T21:41:26.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.509+0000 7f2c92e8b640 1 librados: init done 2024-03-27T21:41:26.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.510+0000 7f2c92e8b640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:41:26.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.510+0000 7f2c92e8b640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:26.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.510+0000 7f2c92e8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:26.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.510+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f2c8c069450 con 0x7f2c8c13b020 2024-03-27T21:41:26.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.511+0000 7f2c897fa640 1 -- 172.21.15.60:0/538744612 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2c78004cf0 con 0x7f2c8c13b020 2024-03-27T21:41:26.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c897fa640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2c7006c590 0x7f2c7006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:26.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c897fa640 1 -- 172.21.15.60:0/538744612 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f2c78088260 con 0x7f2c8c13b020 2024-03-27T21:41:26.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c897fa640 10 client.4986.objecter ms_dispatch 0x7f2c8c000b90 osd_map(60..60 src has 1..60) v4 2024-03-27T21:41:26.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c897fa640 3 client.4986.objecter handle_osd_map got epochs [60,60] > 0 2024-03-27T21:41:26.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c897fa640 3 client.4986.objecter handle_osd_map decoding full epoch 60 2024-03-27T21:41:26.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c90c27640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2c7006c590 0x7f2c7006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:26.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c897fa640 20 client.4986.objecter dump_active .. 0 homeless 2024-03-27T21:41:26.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c90c27640 10 monclient: get_auth_request con 0x7f2c7006c590 auth_method 0 2024-03-27T21:41:26.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c897fa640 1 -- 172.21.15.60:0/538744612 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v60) v1 ==== 54+0+16 (secure 0 0 0) 0x7f2c7808f070 con 0x7f2c8c13b020 2024-03-27T21:41:26.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:26.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c897fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:26.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c92e8b640 10 client.4986.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:26.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign -1 1 2024-03-27T21:41:26.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c92e8b640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f2c8c1318c0 0x7f2c8c1d6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:26.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c0a90d0 osd=0 3 2024-03-27T21:41:26.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c0a90d0 osd=0 3 2024-03-27T21:41:26.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c0a90d0 osd=0 3 2024-03-27T21:41:26.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_remove -1 1 2024-03-27T21:41:26.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c92e8b640 20 client.4986.objecter get_session s=0x7f2c8c0a90d0 osd=0 3 2024-03-27T21:41:26.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign 0 1 2024-03-27T21:41:26.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c0a90d0 osd=0 4 2024-03-27T21:41:26.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c92e8b640 10 client.4986.objecter _send_command 1 2024-03-27T21:41:26.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.512+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c8c13c080 con 0x7f2c8c1318c0 2024-03-27T21:41:26.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.513+0000 7f2c91428640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f2c8c1318c0 0x7f2c8c1d6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:26.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.513+0000 7f2c91428640 10 monclient: get_auth_request con 0x7f2c8c1318c0 auth_method 0 2024-03-27T21:41:26.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.513+0000 7f2c90c27640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2c7006c590 0x7f2c7006ea50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2c80036000 tx=0x7f2c8003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:26.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.513+0000 7f2c897fa640 10 client.4986.objecter ms_handle_connect 0x7f2c7006c590 2024-03-27T21:41:26.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.513+0000 7f2c897fa640 10 client.4986.objecter ms_handle_connect 0x7f2c8c1318c0 2024-03-27T21:41:26.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.513+0000 7f2c91428640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f2c8c1318c0 0x7f2c8c1d6880 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:26.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.514+0000 7f2c897fa640 1 -- 172.21.15.60:0/538744612 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f2c8400f040 con 0x7f2c8c1318c0 2024-03-27T21:41:26.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.514+0000 7f2c897fa640 10 client.4986.objecter ms_dispatch 0x7f2c8c000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:41:26.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.514+0000 7f2c897fa640 10 client.4986.objecter _finish_command 1 = system:0 2024-03-27T21:41:26.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.514+0000 7f2c897fa640 20 client.4986.objecter put_session s=0x7f2c8c0a90d0 osd=0 4 2024-03-27T21:41:26.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.514+0000 7f2c897fa640 15 client.4986.objecter _session_command_op_remove 0 1 2024-03-27T21:41:26.517 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:26.518 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:26.518 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:41:26.518 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:41:26.518 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.528+0000 7f2c92e8b640 10 client.4986.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.528+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign -1 2 2024-03-27T21:41:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.528+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c0a90d0 osd=0 3 2024-03-27T21:41:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.528+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c0a90d0 osd=0 3 2024-03-27T21:41:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.528+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c0a90d0 osd=0 3 2024-03-27T21:41:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.528+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_remove -1 2 2024-03-27T21:41:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.528+0000 7f2c92e8b640 20 client.4986.objecter get_session s=0x7f2c8c0a90d0 osd=0 3 2024-03-27T21:41:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.528+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign 0 2 2024-03-27T21:41:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.528+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c0a90d0 osd=0 4 2024-03-27T21:41:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.528+0000 7f2c92e8b640 10 client.4986.objecter _send_command 2 2024-03-27T21:41:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.528+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2c8c14bd20 con 0x7f2c8c1318c0 2024-03-27T21:41:26.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.530+0000 7f2c897fa640 1 -- 172.21.15.60:0/538744612 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f2c8c14bd20 con 0x7f2c8c1318c0 2024-03-27T21:41:26.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.530+0000 7f2c897fa640 10 client.4986.objecter ms_dispatch 0x7f2c8c000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:26.520 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:41:26.522 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:41:26.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.530+0000 7f2c897fa640 10 client.4986.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:26.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.530+0000 7f2c897fa640 20 client.4986.objecter put_session s=0x7f2c8c0a90d0 osd=0 4 2024-03-27T21:41:26.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.530+0000 7f2c897fa640 15 client.4986.objecter _session_command_op_remove 0 2 2024-03-27T21:41:26.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.530+0000 7f2c92e8b640 10 client.4986.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:26.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.530+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign -1 3 2024-03-27T21:41:26.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.530+0000 7f2c92e8b640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f2c8c1d9c80 0x7f2c8c1da060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:26.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.530+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c1d7000 osd=1 3 2024-03-27T21:41:26.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.530+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c1d7000 osd=1 3 2024-03-27T21:41:26.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.530+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c1d7000 osd=1 3 2024-03-27T21:41:26.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.530+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_remove -1 3 2024-03-27T21:41:26.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.530+0000 7f2c92e8b640 20 client.4986.objecter get_session s=0x7f2c8c1d7000 osd=1 3 2024-03-27T21:41:26.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.530+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign 1 3 2024-03-27T21:41:26.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.530+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c1d7000 osd=1 4 2024-03-27T21:41:26.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.530+0000 7f2c92e8b640 10 client.4986.objecter _send_command 3 2024-03-27T21:41:26.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.530+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c8c1da5a0 con 0x7f2c8c1d9c80 2024-03-27T21:41:26.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.530+0000 7f2c90c27640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f2c8c1d9c80 0x7f2c8c1da060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:26.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.530+0000 7f2c90c27640 10 monclient: get_auth_request con 0x7f2c8c1d9c80 auth_method 0 2024-03-27T21:41:26.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.531+0000 7f2c897fa640 10 client.4986.objecter ms_handle_connect 0x7f2c8c1d9c80 2024-03-27T21:41:26.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.531+0000 7f2c90c27640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f2c8c1d9c80 0x7f2c8c1da060 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:26.526 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:41:26.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.531+0000 7f2c897fa640 1 -- 172.21.15.60:0/538744612 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f2c8c1da5a0 con 0x7f2c8c1d9c80 2024-03-27T21:41:26.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.531+0000 7f2c897fa640 10 client.4986.objecter ms_dispatch 0x7f2c8c000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:41:26.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.531+0000 7f2c897fa640 10 client.4986.objecter _finish_command 3 = system:0 2024-03-27T21:41:26.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.531+0000 7f2c897fa640 20 client.4986.objecter put_session s=0x7f2c8c1d7000 osd=1 4 2024-03-27T21:41:26.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.531+0000 7f2c897fa640 15 client.4986.objecter _session_command_op_remove 1 3 2024-03-27T21:41:26.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.560+0000 7f2c92e8b640 10 client.4986.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:26.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.560+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign -1 4 2024-03-27T21:41:26.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.560+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c1d7000 osd=1 3 2024-03-27T21:41:26.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.560+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c1d7000 osd=1 3 2024-03-27T21:41:26.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.560+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c1d7000 osd=1 3 2024-03-27T21:41:26.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.560+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_remove -1 4 2024-03-27T21:41:26.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.560+0000 7f2c92e8b640 20 client.4986.objecter get_session s=0x7f2c8c1d7000 osd=1 3 2024-03-27T21:41:26.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.560+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign 1 4 2024-03-27T21:41:26.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.560+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c1d7000 osd=1 4 2024-03-27T21:41:26.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.560+0000 7f2c92e8b640 10 client.4986.objecter _send_command 4 2024-03-27T21:41:26.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.560+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2c8c14be90 con 0x7f2c8c1d9c80 2024-03-27T21:41:26.548 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:26.561+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:26.550 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:41:26.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.563+0000 7f2c897fa640 1 -- 172.21.15.60:0/538744612 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f2c8c14be90 con 0x7f2c8c1d9c80 2024-03-27T21:41:26.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.563+0000 7f2c897fa640 10 client.4986.objecter ms_dispatch 0x7f2c8c000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:26.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.563+0000 7f2c897fa640 10 client.4986.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:26.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.563+0000 7f2c897fa640 20 client.4986.objecter put_session s=0x7f2c8c1d7000 osd=1 4 2024-03-27T21:41:26.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.563+0000 7f2c897fa640 15 client.4986.objecter _session_command_op_remove 1 4 2024-03-27T21:41:26.553 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:41:26.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.563+0000 7f2c92e8b640 10 client.4986.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:26.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.563+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign -1 5 2024-03-27T21:41:26.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.563+0000 7f2c92e8b640 20 client.4986.objecter _get_session osd=-1 returning homeless 2024-03-27T21:41:26.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.563+0000 7f2c92e8b640 10 client.4986.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:26.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.563+0000 7f2c92e8b640 10 monclient: _renew_subs 2024-03-27T21:41:26.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.563+0000 7f2c92e8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:26.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.563+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=61}) v3 -- 0x7f2c8c0b54a0 con 0x7f2c8c13b020 2024-03-27T21:41:26.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.563+0000 7f2c92e8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:26.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.563+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2c8c0b2110 con 0x7f2c8c13b020 2024-03-27T21:41:26.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.563+0000 7f2c897fa640 1 -- 172.21.15.60:0/538744612 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_get_version_reply(handle=1 version=60) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c78053750 con 0x7f2c8c13b020 2024-03-27T21:41:26.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.563+0000 7f2c897fa640 10 monclient: handle_get_version_reply finishing 1 version 60 2024-03-27T21:41:26.554 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:41:26.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.563+0000 7f2c8affd640 10 client.4986.objecter _check_command_map_dne tid 5 current 60 map_dne_bound 60 2024-03-27T21:41:26.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.563+0000 7f2c8affd640 10 client.4986.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:41:26.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.563+0000 7f2c8affd640 15 client.4986.objecter _session_command_op_remove -1 5 2024-03-27T21:41:26.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.564+0000 7f2c92e8b640 10 client.4986.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:26.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.564+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign -1 6 2024-03-27T21:41:26.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.564+0000 7f2c92e8b640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f2c8c02b170 0x7f2c8c02b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:26.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.564+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c0b2110 osd=3 3 2024-03-27T21:41:26.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.564+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c0b2110 osd=3 3 2024-03-27T21:41:26.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.564+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c0b2110 osd=3 3 2024-03-27T21:41:26.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.564+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_remove -1 6 2024-03-27T21:41:26.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.564+0000 7f2c92e8b640 20 client.4986.objecter get_session s=0x7f2c8c0b2110 osd=3 3 2024-03-27T21:41:26.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.564+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign 3 6 2024-03-27T21:41:26.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.564+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c0b2110 osd=3 4 2024-03-27T21:41:26.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.564+0000 7f2c92e8b640 10 client.4986.objecter _send_command 6 2024-03-27T21:41:26.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.564+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c8c02ba90 con 0x7f2c8c02b170 2024-03-27T21:41:26.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.564+0000 7f2c8bfff640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f2c8c02b170 0x7f2c8c02b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:26.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.564+0000 7f2c8bfff640 10 monclient: get_auth_request con 0x7f2c8c02b170 auth_method 0 2024-03-27T21:41:26.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.564+0000 7f2c897fa640 10 client.4986.objecter ms_handle_connect 0x7f2c8c02b170 2024-03-27T21:41:26.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.564+0000 7f2c8bfff640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f2c8c02b170 0x7f2c8c02b550 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:26.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.565+0000 7f2c897fa640 1 -- 172.21.15.60:0/538744612 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f2c8c02ba90 con 0x7f2c8c02b170 2024-03-27T21:41:26.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.565+0000 7f2c897fa640 10 client.4986.objecter ms_dispatch 0x7f2c8c000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:41:26.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.565+0000 7f2c897fa640 10 client.4986.objecter _finish_command 6 = system:0 2024-03-27T21:41:26.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.565+0000 7f2c897fa640 20 client.4986.objecter put_session s=0x7f2c8c0b2110 osd=3 4 2024-03-27T21:41:26.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.565+0000 7f2c897fa640 15 client.4986.objecter _session_command_op_remove 3 6 2024-03-27T21:41:26.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.593+0000 7f2c92e8b640 10 client.4986.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:26.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.593+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign -1 7 2024-03-27T21:41:26.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.593+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c0b2110 osd=3 3 2024-03-27T21:41:26.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.593+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c0b2110 osd=3 3 2024-03-27T21:41:26.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.593+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c0b2110 osd=3 3 2024-03-27T21:41:26.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.593+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_remove -1 7 2024-03-27T21:41:26.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.593+0000 7f2c92e8b640 20 client.4986.objecter get_session s=0x7f2c8c0b2110 osd=3 3 2024-03-27T21:41:26.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.593+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign 3 7 2024-03-27T21:41:26.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.593+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c0b2110 osd=3 4 2024-03-27T21:41:26.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.593+0000 7f2c92e8b640 10 client.4986.objecter _send_command 7 2024-03-27T21:41:26.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.593+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2c8c02c450 con 0x7f2c8c02b170 2024-03-27T21:41:26.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.595+0000 7f2c897fa640 1 -- 172.21.15.60:0/538744612 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f2c8c02c450 con 0x7f2c8c02b170 2024-03-27T21:41:26.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.595+0000 7f2c897fa640 10 client.4986.objecter ms_dispatch 0x7f2c8c000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:26.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.595+0000 7f2c897fa640 10 client.4986.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:26.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.595+0000 7f2c897fa640 20 client.4986.objecter put_session s=0x7f2c8c0b2110 osd=3 4 2024-03-27T21:41:26.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.595+0000 7f2c897fa640 15 client.4986.objecter _session_command_op_remove 3 7 2024-03-27T21:41:26.583 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:41:26.585 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:41:26.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.595+0000 7f2c92e8b640 10 client.4986.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:26.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.595+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign -1 8 2024-03-27T21:41:26.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.595+0000 7f2c92e8b640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f2c8c0b1d00 0x7f2c8c1e8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:26.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.595+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c0b1b90 osd=4 3 2024-03-27T21:41:26.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.595+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c0b1b90 osd=4 3 2024-03-27T21:41:26.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.595+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c0b1b90 osd=4 3 2024-03-27T21:41:26.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.595+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_remove -1 8 2024-03-27T21:41:26.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.595+0000 7f2c92e8b640 20 client.4986.objecter get_session s=0x7f2c8c0b1b90 osd=4 3 2024-03-27T21:41:26.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.595+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign 4 8 2024-03-27T21:41:26.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.595+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c0b1b90 osd=4 4 2024-03-27T21:41:26.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.595+0000 7f2c92e8b640 10 client.4986.objecter _send_command 8 2024-03-27T21:41:26.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.595+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c8c1ebdc0 con 0x7f2c8c0b1d00 2024-03-27T21:41:26.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.596+0000 7f2c91428640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f2c8c0b1d00 0x7f2c8c1e8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:26.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.596+0000 7f2c91428640 10 monclient: get_auth_request con 0x7f2c8c0b1d00 auth_method 0 2024-03-27T21:41:26.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.596+0000 7f2c897fa640 10 client.4986.objecter ms_handle_connect 0x7f2c8c0b1d00 2024-03-27T21:41:26.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.596+0000 7f2c91428640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f2c8c0b1d00 0x7f2c8c1e8ff0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:26.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.597+0000 7f2c897fa640 1 -- 172.21.15.60:0/538744612 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f2c8c1ebdc0 con 0x7f2c8c0b1d00 2024-03-27T21:41:26.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.597+0000 7f2c897fa640 10 client.4986.objecter ms_dispatch 0x7f2c8c000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:41:26.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.597+0000 7f2c897fa640 10 client.4986.objecter _finish_command 8 = system:0 2024-03-27T21:41:26.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.597+0000 7f2c897fa640 20 client.4986.objecter put_session s=0x7f2c8c0b1b90 osd=4 4 2024-03-27T21:41:26.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.597+0000 7f2c897fa640 15 client.4986.objecter _session_command_op_remove 4 8 2024-03-27T21:41:26.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.624+0000 7f2c92e8b640 10 client.4986.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:26.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.624+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign -1 9 2024-03-27T21:41:26.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.624+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c0b1b90 osd=4 3 2024-03-27T21:41:26.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.624+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c0b1b90 osd=4 3 2024-03-27T21:41:26.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.624+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c0b1b90 osd=4 3 2024-03-27T21:41:26.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.624+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_remove -1 9 2024-03-27T21:41:26.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.624+0000 7f2c92e8b640 20 client.4986.objecter get_session s=0x7f2c8c0b1b90 osd=4 3 2024-03-27T21:41:26.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.624+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign 4 9 2024-03-27T21:41:26.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.624+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c0b1b90 osd=4 4 2024-03-27T21:41:26.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.624+0000 7f2c92e8b640 10 client.4986.objecter _send_command 9 2024-03-27T21:41:26.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.624+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2c8c02bed0 con 0x7f2c8c0b1d00 2024-03-27T21:41:26.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.626+0000 7f2c897fa640 1 -- 172.21.15.60:0/538744612 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f2c8c02bed0 con 0x7f2c8c0b1d00 2024-03-27T21:41:26.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.626+0000 7f2c897fa640 10 client.4986.objecter ms_dispatch 0x7f2c8c000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:26.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.626+0000 7f2c897fa640 10 client.4986.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:26.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.626+0000 7f2c897fa640 20 client.4986.objecter put_session s=0x7f2c8c0b1b90 osd=4 4 2024-03-27T21:41:26.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.626+0000 7f2c897fa640 15 client.4986.objecter _session_command_op_remove 4 9 2024-03-27T21:41:26.613 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:41:26.614 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:41:26.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.626+0000 7f2c92e8b640 10 client.4986.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:26.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.626+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign -1 10 2024-03-27T21:41:26.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.626+0000 7f2c92e8b640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f2c8c02c070 0x7f2c8c1ea6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:26.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.626+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c02bed0 osd=5 3 2024-03-27T21:41:26.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.626+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c02bed0 osd=5 3 2024-03-27T21:41:26.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.626+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c02bed0 osd=5 3 2024-03-27T21:41:26.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.627+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_remove -1 10 2024-03-27T21:41:26.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.627+0000 7f2c92e8b640 20 client.4986.objecter get_session s=0x7f2c8c02bed0 osd=5 3 2024-03-27T21:41:26.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.627+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign 5 10 2024-03-27T21:41:26.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.627+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c02bed0 osd=5 4 2024-03-27T21:41:26.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.627+0000 7f2c92e8b640 10 client.4986.objecter _send_command 10 2024-03-27T21:41:26.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.627+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c8c066760 con 0x7f2c8c02c070 2024-03-27T21:41:26.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.627+0000 7f2c90c27640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f2c8c02c070 0x7f2c8c1ea6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:26.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.627+0000 7f2c90c27640 10 monclient: get_auth_request con 0x7f2c8c02c070 auth_method 0 2024-03-27T21:41:26.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.627+0000 7f2c90c27640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f2c8c02c070 0x7f2c8c1ea6b0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:26.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.627+0000 7f2c897fa640 10 client.4986.objecter ms_handle_connect 0x7f2c8c02c070 2024-03-27T21:41:26.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.628+0000 7f2c897fa640 1 -- 172.21.15.60:0/538744612 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f2c8c066760 con 0x7f2c8c02c070 2024-03-27T21:41:26.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.628+0000 7f2c897fa640 10 client.4986.objecter ms_dispatch 0x7f2c8c000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:41:26.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.628+0000 7f2c897fa640 10 client.4986.objecter _finish_command 10 = system:0 2024-03-27T21:41:26.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.628+0000 7f2c897fa640 20 client.4986.objecter put_session s=0x7f2c8c02bed0 osd=5 4 2024-03-27T21:41:26.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.628+0000 7f2c897fa640 15 client.4986.objecter _session_command_op_remove 5 10 2024-03-27T21:41:26.628 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:26.642+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:26.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.657+0000 7f2c92e8b640 10 client.4986.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:26.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.657+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign -1 11 2024-03-27T21:41:26.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.657+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c02bed0 osd=5 3 2024-03-27T21:41:26.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.657+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c02bed0 osd=5 3 2024-03-27T21:41:26.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.657+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c02bed0 osd=5 3 2024-03-27T21:41:26.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.657+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_remove -1 11 2024-03-27T21:41:26.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.657+0000 7f2c92e8b640 20 client.4986.objecter get_session s=0x7f2c8c02bed0 osd=5 3 2024-03-27T21:41:26.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.657+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign 5 11 2024-03-27T21:41:26.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.657+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c02bed0 osd=5 4 2024-03-27T21:41:26.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.657+0000 7f2c92e8b640 10 client.4986.objecter _send_command 11 2024-03-27T21:41:26.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.657+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2c8c066760 con 0x7f2c8c02c070 2024-03-27T21:41:26.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.659+0000 7f2c897fa640 1 -- 172.21.15.60:0/538744612 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f2c8c066760 con 0x7f2c8c02c070 2024-03-27T21:41:26.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.659+0000 7f2c897fa640 10 client.4986.objecter ms_dispatch 0x7f2c8c000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:26.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.659+0000 7f2c897fa640 10 client.4986.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:26.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.659+0000 7f2c897fa640 20 client.4986.objecter put_session s=0x7f2c8c02bed0 osd=5 4 2024-03-27T21:41:26.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.659+0000 7f2c897fa640 15 client.4986.objecter _session_command_op_remove 5 11 2024-03-27T21:41:26.646 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:41:26.647 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:41:26.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.659+0000 7f2c92e8b640 10 client.4986.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:26.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.659+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign -1 12 2024-03-27T21:41:26.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.659+0000 7f2c92e8b640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f2c8c1eb100 0x7f2c8c1eb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:26.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.660+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c066760 osd=6 3 2024-03-27T21:41:26.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.660+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c066760 osd=6 3 2024-03-27T21:41:26.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.660+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c066760 osd=6 3 2024-03-27T21:41:26.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.660+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_remove -1 12 2024-03-27T21:41:26.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.660+0000 7f2c92e8b640 20 client.4986.objecter get_session s=0x7f2c8c066760 osd=6 3 2024-03-27T21:41:26.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.660+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign 6 12 2024-03-27T21:41:26.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.660+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c066760 osd=6 4 2024-03-27T21:41:26.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.660+0000 7f2c92e8b640 10 client.4986.objecter _send_command 12 2024-03-27T21:41:26.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.660+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c8c1ebaf0 con 0x7f2c8c1eb100 2024-03-27T21:41:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.660+0000 7f2c8bfff640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f2c8c1eb100 0x7f2c8c1eb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.660+0000 7f2c8bfff640 10 monclient: get_auth_request con 0x7f2c8c1eb100 auth_method 0 2024-03-27T21:41:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.660+0000 7f2c897fa640 10 client.4986.objecter ms_handle_connect 0x7f2c8c1eb100 2024-03-27T21:41:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.660+0000 7f2c8bfff640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f2c8c1eb100 0x7f2c8c1eb560 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.661+0000 7f2c897fa640 1 -- 172.21.15.60:0/538744612 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f2c8c1ebaf0 con 0x7f2c8c1eb100 2024-03-27T21:41:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.661+0000 7f2c897fa640 10 client.4986.objecter ms_dispatch 0x7f2c8c000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:41:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.661+0000 7f2c897fa640 10 client.4986.objecter _finish_command 12 = system:0 2024-03-27T21:41:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.661+0000 7f2c897fa640 20 client.4986.objecter put_session s=0x7f2c8c066760 osd=6 4 2024-03-27T21:41:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.661+0000 7f2c897fa640 15 client.4986.objecter _session_command_op_remove 6 12 2024-03-27T21:41:26.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.689+0000 7f2c92e8b640 10 client.4986.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:26.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.689+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign -1 13 2024-03-27T21:41:26.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.689+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c066760 osd=6 3 2024-03-27T21:41:26.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.689+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c066760 osd=6 3 2024-03-27T21:41:26.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.689+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c066760 osd=6 3 2024-03-27T21:41:26.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.689+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_remove -1 13 2024-03-27T21:41:26.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.689+0000 7f2c92e8b640 20 client.4986.objecter get_session s=0x7f2c8c066760 osd=6 3 2024-03-27T21:41:26.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.689+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign 6 13 2024-03-27T21:41:26.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.689+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c066760 osd=6 4 2024-03-27T21:41:26.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.689+0000 7f2c92e8b640 10 client.4986.objecter _send_command 13 2024-03-27T21:41:26.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.689+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2c8c1ebaf0 con 0x7f2c8c1eb100 2024-03-27T21:41:26.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.691+0000 7f2c897fa640 1 -- 172.21.15.60:0/538744612 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f2c8c1ebaf0 con 0x7f2c8c1eb100 2024-03-27T21:41:26.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.691+0000 7f2c897fa640 10 client.4986.objecter ms_dispatch 0x7f2c8c000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:26.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.691+0000 7f2c897fa640 10 client.4986.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:26.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.691+0000 7f2c897fa640 20 client.4986.objecter put_session s=0x7f2c8c066760 osd=6 4 2024-03-27T21:41:26.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.691+0000 7f2c897fa640 15 client.4986.objecter _session_command_op_remove 6 13 2024-03-27T21:41:26.678 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:41:26.680 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:41:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.691+0000 7f2c92e8b640 10 client.4986.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.691+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign -1 14 2024-03-27T21:41:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.691+0000 7f2c92e8b640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f2c8c1f0460 0x7f2c8c1f2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.691+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c1ebaf0 osd=7 3 2024-03-27T21:41:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.691+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c1ebaf0 osd=7 3 2024-03-27T21:41:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.691+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c1ebaf0 osd=7 3 2024-03-27T21:41:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.691+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_remove -1 14 2024-03-27T21:41:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.691+0000 7f2c92e8b640 20 client.4986.objecter get_session s=0x7f2c8c1ebaf0 osd=7 3 2024-03-27T21:41:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.691+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign 7 14 2024-03-27T21:41:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.691+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c1ebaf0 osd=7 4 2024-03-27T21:41:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.691+0000 7f2c92e8b640 10 client.4986.objecter _send_command 14 2024-03-27T21:41:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.691+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c8c066aa0 con 0x7f2c8c1f0460 2024-03-27T21:41:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.692+0000 7f2c91428640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f2c8c1f0460 0x7f2c8c1f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.692+0000 7f2c91428640 10 monclient: get_auth_request con 0x7f2c8c1f0460 auth_method 0 2024-03-27T21:41:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.692+0000 7f2c897fa640 10 client.4986.objecter ms_handle_connect 0x7f2c8c1f0460 2024-03-27T21:41:26.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.692+0000 7f2c91428640 1 --2- 172.21.15.60:0/538744612 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f2c8c1f0460 0x7f2c8c1f2850 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:26.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.693+0000 7f2c897fa640 1 -- 172.21.15.60:0/538744612 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f2c8401d670 con 0x7f2c8c1f0460 2024-03-27T21:41:26.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.693+0000 7f2c897fa640 10 client.4986.objecter ms_dispatch 0x7f2c8c000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:41:26.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.693+0000 7f2c897fa640 10 client.4986.objecter _finish_command 14 = system:0 2024-03-27T21:41:26.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.693+0000 7f2c897fa640 20 client.4986.objecter put_session s=0x7f2c8c1ebaf0 osd=7 4 2024-03-27T21:41:26.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.693+0000 7f2c897fa640 15 client.4986.objecter _session_command_op_remove 7 14 2024-03-27T21:41:26.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.721+0000 7f2c92e8b640 10 client.4986.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:26.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.721+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign -1 15 2024-03-27T21:41:26.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.721+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c1ebaf0 osd=7 3 2024-03-27T21:41:26.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.721+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c1ebaf0 osd=7 3 2024-03-27T21:41:26.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.721+0000 7f2c92e8b640 20 client.4986.objecter _get_session s=0x7f2c8c1ebaf0 osd=7 3 2024-03-27T21:41:26.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.721+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_remove -1 15 2024-03-27T21:41:26.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.721+0000 7f2c92e8b640 20 client.4986.objecter get_session s=0x7f2c8c1ebaf0 osd=7 3 2024-03-27T21:41:26.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.721+0000 7f2c92e8b640 15 client.4986.objecter _session_command_op_assign 7 15 2024-03-27T21:41:26.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.721+0000 7f2c92e8b640 20 client.4986.objecter put_session s=0x7f2c8c1ebaf0 osd=7 4 2024-03-27T21:41:26.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.721+0000 7f2c92e8b640 10 client.4986.objecter _send_command 15 2024-03-27T21:41:26.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.721+0000 7f2c92e8b640 1 -- 172.21.15.60:0/538744612 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2c8c1f2d90 con 0x7f2c8c1f0460 2024-03-27T21:41:26.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.723+0000 7f2c897fa640 1 -- 172.21.15.60:0/538744612 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f2c8c1f2d90 con 0x7f2c8c1f0460 2024-03-27T21:41:26.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.723+0000 7f2c897fa640 10 client.4986.objecter ms_dispatch 0x7f2c8c000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:26.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.723+0000 7f2c897fa640 10 client.4986.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:26.710 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:41:26.710 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:41:26.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.723+0000 7f2c897fa640 20 client.4986.objecter put_session s=0x7f2c8c1ebaf0 osd=7 4 2024-03-27T21:41:26.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:26.723+0000 7f2c897fa640 15 client.4986.objecter _session_command_op_remove 7 15 2024-03-27T21:41:26.712 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:26.713 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:26.714 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:41:26.715 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:41:26.726 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:41:26.729 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:26.743+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:26.830 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:26.844+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:26.883 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:26.883 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:26.892 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:41:26.936 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:26.950+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:27.001 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:27.002 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:27.011 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:41:27.032 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:27.045+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:27.133 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:27.147+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:27.221 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:27.221 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:27.233 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:41:27.275 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:27.289+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:27.335 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:27.349+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:27.382 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:27.383 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:27.392 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:41:27.436 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:27.450+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:27.501 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:27.501 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:27.510 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:41:27.554 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:27.568+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:27.626 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:27.626 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:27.635 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:41:27.676 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:27.690+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:27.739 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:27.753+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:27.752 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:27.752 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:27.761 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:41:27.840 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:27.853+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:27.866 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:27.867 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:27.875 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:41:27.877 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:27.891+0000 7fb853f8f640 -1 osd.2 44 log_to_monitors true 2024-03-27T21:41:27.959 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:27.973+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:28.020 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:28.020 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:28.028 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:41:28.047 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:28.061+0000 7fb84bd45640 -1 osd.2 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-27T21:41:28.052 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-27T21:41:28.069 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:28.083+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:28.142 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:28.156+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:28.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.168+0000 7f4344f2e640 10 monclient: get_monmap_and_config 2024-03-27T21:41:28.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.168+0000 7f4344f2e640 10 monclient: build_initial_monmap 2024-03-27T21:41:28.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.168+0000 7f4344f2e640 10 monclient: monmap: 2024-03-27T21:41:28.155 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:28.155 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:28.155 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:28.169515+0000 2024-03-27T21:41:28.155 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:28.169515+0000 2024-03-27T21:41:28.155 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:28.155 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:28.155 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:28.155 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:28.155 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:28.155 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:28.155 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:28.155 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:28.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.169+0000 7f4344f2e640 1 Processor -- start 2024-03-27T21:41:28.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.169+0000 7f4344f2e640 1 -- start start 2024-03-27T21:41:28.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.169+0000 7f4344f2e640 10 monclient: init 2024-03-27T21:41:28.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.169+0000 7f4344f2e640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:28.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.169+0000 7f4344f2e640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-03-27T21:41:28.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.169+0000 7f4344f2e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4340071a50 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:28.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.169+0000 7f4344f2e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f434013ef20 0x7f43400a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:28.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.169+0000 7f4344f2e640 10 monclient(hunting): picked mon.noname-c con 0x7f434013ef20 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:28.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.169+0000 7f4344f2e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4340141630 0x7f4340141ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:28.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.169+0000 7f4344f2e640 10 monclient(hunting): picked mon.noname-a con 0x7f4340141630 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:28.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.169+0000 7f4344f2e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43400a8e90 con 0x7f4340071a50 2024-03-27T21:41:28.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f4344f2e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f4344f2e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4340071e30 con 0x7f4340141630 2024-03-27T21:41:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f4344f2e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f4344f2e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43400aa040 con 0x7f434013ef20 2024-03-27T21:41:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f4344f2e640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f4344f2e640 10 monclient(hunting): authenticate will time out at 929.390997s 2024-03-27T21:41:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433dd9b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f434013ef20 0x7f43400a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433ed9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4340141630 0x7f4340141ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433e59c640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f4340071a50 0x7f434013e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.60:44392/0 (socket says 172.21.15.60:44392) 2024-03-27T21:41:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433e59c640 1 -- 172.21.15.60:0/1199558513 learned_addr learned my addr 172.21.15.60:0/1199558513 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433ed9d640 10 monclient(hunting): get_auth_request con 0x7f4340141630 auth_method 0 2024-03-27T21:41:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433ed9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433dd9b640 10 monclient(hunting): get_auth_request con 0x7f434013ef20 auth_method 0 2024-03-27T21:41:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433ed9d640 10 monclient(hunting): handle_auth_done global_id 4992 payload 274 2024-03-27T21:41:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433ed9d640 1 -- 172.21.15.60:0/1199558513 >> v2:172.21.15.191:3300/0 conn(0x7f434013ef20 msgr2=0x7f43400a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433ed9d640 1 --2- 172.21.15.60:0/1199558513 >> v2:172.21.15.191:3300/0 conn(0x7f434013ef20 0x7f43400a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433ed9d640 1 -- 172.21.15.60:0/1199558513 >> v1:172.21.15.191:6789/0 conn(0x7f4340071a50 legacy=0x7f434013e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433ed9d640 10 monclient: _finish_hunting 0 2024-03-27T21:41:28.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433ed9d640 1 monclient: found mon.noname-a 2024-03-27T21:41:28.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:28.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:28.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433ed9d640 1 -- 172.21.15.60:0/1199558513 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43400aa2b0 con 0x7f4340141630 2024-03-27T21:41:28.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.170+0000 7f433dd9b640 1 --2- 172.21.15.60:0/1199558513 >> v2:172.21.15.191:3300/0 conn(0x7f434013ef20 0x7f43400a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:28.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.171+0000 7f433ed9d640 1 --2- 172.21.15.60:0/1199558513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4340141630 0x7f4340141ac0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f43400709f0 tx=0x7f433802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c183d9fde8ad12c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:28.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.171+0000 7f433d59a640 1 -- 172.21.15.60:0/1199558513 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433802cc80 con 0x7f4340141630 2024-03-27T21:41:28.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.171+0000 7f433d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:28.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.171+0000 7f433d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:28.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.171+0000 7f433d59a640 10 monclient: dump: 2024-03-27T21:41:28.168 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:28.168 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:28.168 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:28.168 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:28.169 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.171+0000 7f433d59a640 10 monclient: _finish_auth 0 2024-03-27T21:41:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.171+0000 7f433d59a640 10 monclient: _check_auth_tickets 2024-03-27T21:41:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.171+0000 7f433d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.171+0000 7f4344f2e640 5 monclient: authenticate success, global_id 4992 2024-03-27T21:41:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.171+0000 7f4344f2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.171+0000 7f433d59a640 1 -- 172.21.15.60:0/1199558513 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433802cde0 con 0x7f4340141630 2024-03-27T21:41:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.171+0000 7f433d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.171+0000 7f433d59a640 1 -- 172.21.15.60:0/1199558513 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4338034660 con 0x7f4340141630 2024-03-27T21:41:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.171+0000 7f433d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:28.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.171+0000 7f433d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:28.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.171+0000 7f433d59a640 10 monclient: dump: 2024-03-27T21:41:28.170 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:28.170 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:28.170 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:28.170 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:28.170 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:28.171 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:28.171 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:28.171 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:28.171 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:28.171 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:28.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.171+0000 7f4344f2e640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:28.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.171+0000 7f4344f2e640 4 set_mon_vals no callback set 2024-03-27T21:41:28.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.172+0000 7f4344f2e640 10 monclient: shutdown 2024-03-27T21:41:28.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.172+0000 7f4344f2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:28.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.172+0000 7f4344f2e640 1 -- 172.21.15.60:0/1199558513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4340141630 msgr2=0x7f4340141ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:28.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.172+0000 7f4344f2e640 1 --2- 172.21.15.60:0/1199558513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4340141630 0x7f4340141ac0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f43400709f0 tx=0x7f433802bed0 comp rx=0 tx=0).stop 2024-03-27T21:41:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.172+0000 7f4344f2e640 1 -- 172.21.15.60:0/1199558513 shutdown_connections 2024-03-27T21:41:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.172+0000 7f4344f2e640 1 --2- 172.21.15.60:0/1199558513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4340141630 0x7f4340141ac0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.172+0000 7f4344f2e640 1 --2- 172.21.15.60:0/1199558513 >> v2:172.21.15.191:3300/0 conn(0x7f434013ef20 0x7f43400a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.172+0000 7f4344f2e640 1 -- 172.21.15.60:0/1199558513 >> 172.21.15.60:0/1199558513 conn(0x7f43400b7300 msgr2=0x7f43400b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.172+0000 7f4344f2e640 1 -- 172.21.15.60:0/1199558513 shutdown_connections 2024-03-27T21:41:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.172+0000 7f4344f2e640 1 -- 172.21.15.60:0/1199558513 wait complete. 2024-03-27T21:41:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.173+0000 7f4344f2e640 10 monclient: build_initial_monmap 2024-03-27T21:41:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.173+0000 7f4344f2e640 10 monclient: monmap: 2024-03-27T21:41:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:28.173 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:28.173 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:28.173 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:28.173 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:28.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.173+0000 7f4344f2e640 1 librados: starting msgr at 2024-03-27T21:41:28.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.173+0000 7f4344f2e640 1 librados: starting objecter 2024-03-27T21:41:28.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.173+0000 7f4344f2e640 1 Processor -- start 2024-03-27T21:41:28.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.173+0000 7f4344f2e640 1 -- start start 2024-03-27T21:41:28.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.173+0000 7f4344f2e640 1 librados: setting wanted keys 2024-03-27T21:41:28.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.173+0000 7f4344f2e640 1 librados: calling monclient init 2024-03-27T21:41:28.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.173+0000 7f4344f2e640 10 monclient: init 2024-03-27T21:41:28.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.173+0000 7f4344f2e640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:28.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.173+0000 7f4344f2e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:41:28.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.173+0000 7f4344f2e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4340071a50 0x7f43400b6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:28.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.173+0000 7f4344f2e640 10 monclient(hunting): picked mon.noname-a con 0x7f4340071a50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:28.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.173+0000 7f4344f2e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f434013ef20 0x7f43401397f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:28.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f4344f2e640 10 monclient(hunting): picked mon.noname-b con 0x7f434013ef20 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:28.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f4344f2e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4340141630 0x7f43401d9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:28.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f4344f2e640 10 monclient(hunting): picked mon.noname-c con 0x7f4340141630 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:28.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f4344f2e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:28.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f4344f2e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43401391b0 con 0x7f4340071a50 2024-03-27T21:41:28.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f4344f2e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:28.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f4344f2e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434013a130 con 0x7f434013ef20 2024-03-27T21:41:28.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f4344f2e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:28.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f4344f2e640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4340138490 con 0x7f4340141630 2024-03-27T21:41:28.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f4344f2e640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:28.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f4344f2e640 10 monclient(hunting): authenticate will time out at 929.395004s 2024-03-27T21:41:28.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4340071a50 0x7f43400b6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:28.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4340071a50 0x7f43400b6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36498/0 (socket says 172.21.15.60:36498) 2024-03-27T21:41:28.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433e59c640 1 -- 172.21.15.60:0/1345437927 learned_addr learned my addr 172.21.15.60:0/1345437927 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:28.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433dd9b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f434013ef20 0x7f43401397f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:28.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433e59c640 10 monclient(hunting): get_auth_request con 0x7f4340071a50 auth_method 0 2024-03-27T21:41:28.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:28.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433e59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:28.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:28.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433dd9b640 10 monclient(hunting): get_auth_request con 0x7f434013ef20 auth_method 0 2024-03-27T21:41:28.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:28.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:28.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:28.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433ed9d640 1 --2- 172.21.15.60:0/1345437927 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4340141630 0x7f43401d9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:28.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433ed9d640 10 monclient(hunting): get_auth_request con 0x7f4340141630 auth_method 0 2024-03-27T21:41:28.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:28.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433ed9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:28.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:28.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:28.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:28.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:28.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:28.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:28.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:28.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:28.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:28.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:28.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433ed9d640 10 monclient(hunting): handle_auth_done global_id 4787 payload 824 2024-03-27T21:41:28.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433ed9d640 1 -- 172.21.15.60:0/1345437927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f434013ef20 msgr2=0x7f43401397f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:28.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433ed9d640 1 --2- 172.21.15.60:0/1345437927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f434013ef20 0x7f43401397f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:28.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433ed9d640 1 -- 172.21.15.60:0/1345437927 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4340071a50 msgr2=0x7f43400b6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:28.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433ed9d640 1 --2- 172.21.15.60:0/1345437927 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4340071a50 0x7f43400b6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:28.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433ed9d640 10 monclient: _finish_hunting 0 2024-03-27T21:41:28.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433ed9d640 1 monclient: found mon.noname-c 2024-03-27T21:41:28.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:28.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:28.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433ed9d640 1 -- 172.21.15.60:0/1345437927 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4334009400 con 0x7f4340141630 2024-03-27T21:41:28.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433dd9b640 1 --2- 172.21.15.60:0/1345437927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f434013ef20 0x7f43401397f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:28.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f433ed9d640 1 --2- 172.21.15.60:0/1345437927 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4340141630 0x7f43401d9820 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4340072190 tx=0x7f4338004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:28.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f432effd640 10 client.?.objecter ms_handle_connect 0x7f4340141630 2024-03-27T21:41:28.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.174+0000 7f432effd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:28.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f432effd640 1 -- 172.21.15.60:0/1345437927 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4338041ea0 con 0x7f4340141630 2024-03-27T21:41:28.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f432effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:28.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f432effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:28.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f432effd640 10 monclient: dump: 2024-03-27T21:41:28.177 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:28.178 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:28.178 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:28.178 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:28.178 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:28.178 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:28.178 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:28.178 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:28.178 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:28.178 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:28.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f432effd640 10 monclient: _finish_auth 0 2024-03-27T21:41:28.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f432effd640 10 monclient: _check_auth_tickets 2024-03-27T21:41:28.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f432effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:28.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f432effd640 1 -- 172.21.15.60:0/1345437927 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4338009450 con 0x7f4340141630 2024-03-27T21:41:28.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f432effd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f432effd640 1 -- 172.21.15.60:0/1345437927 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4338046070 con 0x7f4340141630 2024-03-27T21:41:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f432effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f432effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f432effd640 10 monclient: dump: 2024-03-27T21:41:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:28.179 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f4344f2e640 5 monclient: authenticate success, global_id 4787 2024-03-27T21:41:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f433cd99640 4 set_mon_vals no callback set 2024-03-27T21:41:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f4344f2e640 10 monclient: _renew_subs 2024-03-27T21:41:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f4344f2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f4344f2e640 1 -- 172.21.15.60:0/1345437927 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4338009a70 con 0x7f4340141630 2024-03-27T21:41:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f4344f2e640 10 client.4787.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f4344f2e640 10 monclient: _renew_subs 2024-03-27T21:41:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f4344f2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f4344f2e640 1 -- 172.21.15.60:0/1345437927 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43401d9d60 con 0x7f4340141630 2024-03-27T21:41:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.175+0000 7f4344f2e640 1 librados: init done 2024-03-27T21:41:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.176+0000 7f4344f2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.176+0000 7f4344f2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.176+0000 7f4344f2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.176+0000 7f4344f2e640 1 -- 172.21.15.60:0/1345437927 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4340139f10 con 0x7f4340141630 2024-03-27T21:41:28.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.177+0000 7f432effd640 1 -- 172.21.15.60:0/1345437927 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4338035c50 con 0x7f4340141630 2024-03-27T21:41:28.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.177+0000 7f432effd640 1 --2- 172.21.15.60:0/1345437927 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f432006c590 0x7f432006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:28.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.177+0000 7f432effd640 1 -- 172.21.15.60:0/1345437927 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f43380b35f0 con 0x7f4340141630 2024-03-27T21:41:28.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.177+0000 7f432effd640 10 client.4787.objecter ms_dispatch 0x7f4340000b90 osd_map(61..61 src has 1..61) v4 2024-03-27T21:41:28.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.177+0000 7f432effd640 3 client.4787.objecter handle_osd_map got epochs [61,61] > 0 2024-03-27T21:41:28.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.177+0000 7f432effd640 3 client.4787.objecter handle_osd_map decoding full epoch 61 2024-03-27T21:41:28.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.177+0000 7f432effd640 20 client.4787.objecter dump_active .. 0 homeless 2024-03-27T21:41:28.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.177+0000 7f433e59c640 1 --2- 172.21.15.60:0/1345437927 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f432006c590 0x7f432006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:28.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.177+0000 7f433e59c640 10 monclient: get_auth_request con 0x7f432006c590 auth_method 0 2024-03-27T21:41:28.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.178+0000 7f432effd640 10 client.4787.objecter ms_handle_connect 0x7f432006c590 2024-03-27T21:41:28.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.178+0000 7f433e59c640 1 --2- 172.21.15.60:0/1345437927 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f432006c590 0x7f432006ea50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4330004e40 tx=0x7f4330007f10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:28.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.186+0000 7f432effd640 1 -- 172.21.15.60:0/1345437927 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f433807ea40 con 0x7f4340141630 2024-03-27T21:41:28.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.186+0000 7f432effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:28.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.186+0000 7f432effd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:28.183 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:28.183 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:28.184 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:41:28.243 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:28.257+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:28.288 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:28.288 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:28.297 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:41:28.344 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:28.358+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:28.403 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:28.403 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:28.411 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:41:28.455 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:28.469+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:28.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.486+0000 7f4344f2e640 10 start_mon_command cmd=[{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:41:28.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.487+0000 7f4344f2e640 10 monclient: _send_command 2 [{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:41:28.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.487+0000 7f4344f2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:28.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.487+0000 7f4344f2e640 1 -- 172.21.15.60:0/1345437927 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f43400aa620 con 0x7f4340141630 2024-03-27T21:41:28.517 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:28.518 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:28.526 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:41:28.568 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:28.581+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:28.646 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:28.660+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:28.671 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:28.672 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:28.680 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:41:28.716 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:41:28.764 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:28.778+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:28.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.830+0000 7f664bf92640 10 monclient: get_monmap_and_config 2024-03-27T21:41:28.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.830+0000 7f664bf92640 10 monclient: build_initial_monmap 2024-03-27T21:41:28.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.830+0000 7f664bf92640 10 monclient: monmap: 2024-03-27T21:41:28.816 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:28.817 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:28.817 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:28.831351+0000 2024-03-27T21:41:28.817 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:28.831351+0000 2024-03-27T21:41:28.817 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:28.817 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:28.817 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:28.817 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:28.817 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:28.817 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:28.818 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:28.818 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:28.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.830+0000 7f664bf92640 1 Processor -- start 2024-03-27T21:41:28.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.831+0000 7f664bf92640 1 -- start start 2024-03-27T21:41:28.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.831+0000 7f664bf92640 10 monclient: init 2024-03-27T21:41:28.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f664bf92640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:28.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f664bf92640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-03-27T21:41:28.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f664bf92640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6644071a50 0x7f66440a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:28.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f664bf92640 10 monclient(hunting): picked mon.noname-c con 0x7f6644071a50 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:28.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f664bf92640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f664413e810 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:28.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f664bf92640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f664413ec50 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:28.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f664bf92640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66440a8e90 con 0x7f664413e810 2024-03-27T21:41:28.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f664bf92640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66440aa040 con 0x7f664413ec50 2024-03-27T21:41:28.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f664bf92640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:28.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f664bf92640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6644071e30 con 0x7f6644071a50 2024-03-27T21:41:28.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f664bf92640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:28.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f664bf92640 10 monclient(hunting): authenticate will time out at 930.053127s 2024-03-27T21:41:28.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f664952d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f664413e810 0x7f6644141630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56822/0 (socket says 172.21.15.60:56822) 2024-03-27T21:41:28.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f664952d640 1 -- 172.21.15.60:0/1719162092 learned_addr learned my addr 172.21.15.60:0/1719162092 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:28.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6649d2e640 1 --2- 172.21.15.60:0/1719162092 >> v2:172.21.15.191:3300/0 conn(0x7f6644071a50 0x7f66440a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:28.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6649d2e640 10 monclient(hunting): get_auth_request con 0x7f6644071a50 auth_method 0 2024-03-27T21:41:28.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6649d2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:28.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6649d2e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:28.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6649d2e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:28.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6649d2e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:28.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6649d2e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:28.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6649d2e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:28.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6648d2c640 1 -- 172.21.15.60:0/1719162092 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300151994 0 0) 0x7f66440a8e90 con 0x7f664413e810 2024-03-27T21:41:28.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6648d2c640 10 monclient(hunting): _negotiate 2024-03-27T21:41:28.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6648d2c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:28.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6648d2c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:28.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6648d2c640 10 monclient(hunting): my global_id is 4998 2024-03-27T21:41:28.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6648d2c640 1 -- 172.21.15.60:0/1719162092 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6630003520 con 0x7f664413e810 2024-03-27T21:41:28.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6649d2e640 10 monclient(hunting): handle_auth_done global_id 4945 payload 274 2024-03-27T21:41:28.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6649d2e640 1 -- 172.21.15.60:0/1719162092 >> v1:172.21.15.191:6789/0 conn(0x7f664413ec50 legacy=0x7f6644141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:28.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6649d2e640 1 -- 172.21.15.60:0/1719162092 >> v1:172.21.15.60:6789/0 conn(0x7f664413e810 legacy=0x7f6644141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:28.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6649d2e640 10 monclient: _finish_hunting 0 2024-03-27T21:41:28.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6649d2e640 1 monclient: found mon.noname-c 2024-03-27T21:41:28.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6649d2e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:28.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6649d2e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:41:28.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.832+0000 7f6649d2e640 1 -- 172.21.15.60:0/1719162092 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66440aa2b0 con 0x7f6644071a50 2024-03-27T21:41:28.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.833+0000 7f6649d2e640 1 --2- 172.21.15.60:0/1719162092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6644071a50 0x7f66440a87e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6640009720 tx=0x7f664002b730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b504c101c2269773 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:28.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.833+0000 7f6648d2c640 1 -- 172.21.15.60:0/1719162092 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640030710 con 0x7f6644071a50 2024-03-27T21:41:28.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.833+0000 7f6648d2c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:28.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.833+0000 7f6648d2c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:28.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.833+0000 7f6648d2c640 10 monclient: dump: 2024-03-27T21:41:28.836 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:28.836 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:28.836 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:28.836 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:28.836 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:28.836 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:28.836 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:28.836 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:28.836 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:28.836 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:28.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.833+0000 7f6648d2c640 10 monclient: _finish_auth 0 2024-03-27T21:41:28.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.833+0000 7f6648d2c640 10 monclient: _check_auth_tickets 2024-03-27T21:41:28.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.833+0000 7f6648d2c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.833+0000 7f6648d2c640 1 -- 172.21.15.60:0/1719162092 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664002e050 con 0x7f6644071a50 2024-03-27T21:41:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.833+0000 7f6648d2c640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.833+0000 7f664bf92640 5 monclient: authenticate success, global_id 4945 2024-03-27T21:41:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.833+0000 7f6648d2c640 1 -- 172.21.15.60:0/1719162092 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640030a90 con 0x7f6644071a50 2024-03-27T21:41:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.833+0000 7f6648d2c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.833+0000 7f6648d2c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.833+0000 7f6648d2c640 10 monclient: dump: 2024-03-27T21:41:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:28.838 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:28.838 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:28.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.833+0000 7f664bf92640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:28.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.833+0000 7f664bf92640 4 set_mon_vals no callback set 2024-03-27T21:41:28.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.834+0000 7f664bf92640 10 monclient: shutdown 2024-03-27T21:41:28.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.834+0000 7f664bf92640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:28.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.834+0000 7f664bf92640 1 -- 172.21.15.60:0/1719162092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6644071a50 msgr2=0x7f66440a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:28.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.834+0000 7f664bf92640 1 --2- 172.21.15.60:0/1719162092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6644071a50 0x7f66440a87e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6640009720 tx=0x7f664002b730 comp rx=0 tx=0).stop 2024-03-27T21:41:28.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.834+0000 7f664bf92640 1 -- 172.21.15.60:0/1719162092 shutdown_connections 2024-03-27T21:41:28.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.834+0000 7f664bf92640 1 --2- 172.21.15.60:0/1719162092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6644071a50 0x7f66440a87e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:28.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.834+0000 7f664bf92640 1 -- 172.21.15.60:0/1719162092 >> 172.21.15.60:0/1719162092 conn(0x7f66440b7300 msgr2=0x7f66440b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:28.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.834+0000 7f664bf92640 1 -- 172.21.15.60:0/1719162092 shutdown_connections 2024-03-27T21:41:28.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.834+0000 7f664bf92640 1 -- 172.21.15.60:0/1719162092 wait complete. 2024-03-27T21:41:28.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.835+0000 7f664bf92640 10 monclient: build_initial_monmap 2024-03-27T21:41:28.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.835+0000 7f664bf92640 10 monclient: monmap: 2024-03-27T21:41:28.839 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:28.839 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:28.839 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:28.839 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:28.839 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:28.839 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:28.839 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:28.839 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:28.839 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:28.839 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:28.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.835+0000 7f664bf92640 1 librados: starting msgr at 2024-03-27T21:41:28.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.835+0000 7f664bf92640 1 librados: starting objecter 2024-03-27T21:41:28.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.835+0000 7f664bf92640 1 Processor -- start 2024-03-27T21:41:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.835+0000 7f664bf92640 1 -- start start 2024-03-27T21:41:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.835+0000 7f664bf92640 1 librados: setting wanted keys 2024-03-27T21:41:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.835+0000 7f664bf92640 1 librados: calling monclient init 2024-03-27T21:41:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.835+0000 7f664bf92640 10 monclient: init 2024-03-27T21:41:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664bf92640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664bf92640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:41:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664bf92640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6644071a50 0x7f6644139db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664bf92640 10 monclient(hunting): picked mon.noname-c con 0x7f6644071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664bf92640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f664413e810 0x7f6644138de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664bf92640 10 monclient(hunting): picked mon.noname-b con 0x7f664413e810 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664bf92640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f664413ec50 0x7f66440aae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664bf92640 10 monclient(hunting): picked mon.noname-a con 0x7f664413ec50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f6649d2e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6644071a50 0x7f6644139db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664bf92640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664bf92640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6644139620 con 0x7f664413ec50 2024-03-27T21:41:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664bf92640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f6649d2e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6644071a50 0x7f6644139db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:60306/0 (socket says 172.21.15.60:60306) 2024-03-27T21:41:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664bf92640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664413af60 con 0x7f664413e810 2024-03-27T21:41:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664bf92640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f6649d2e640 1 -- 172.21.15.60:0/2452233279 learned_addr learned my addr 172.21.15.60:0/2452233279 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664bf92640 1 -- 172.21.15.60:0/2452233279 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6644137fa0 con 0x7f6644071a50 2024-03-27T21:41:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664bf92640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664bf92640 10 monclient(hunting): authenticate will time out at 930.057200s 2024-03-27T21:41:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f6649d2e640 10 monclient(hunting): get_auth_request con 0x7f6644071a50 auth_method 0 2024-03-27T21:41:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f6649d2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f6649d2e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f6649d2e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664a52f640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f664413ec50 0x7f66440aae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664952d640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f664413e810 0x7f6644138de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664a52f640 10 monclient(hunting): get_auth_request con 0x7f664413ec50 auth_method 0 2024-03-27T21:41:28.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664a52f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:28.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664a52f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:28.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664a52f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:28.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664952d640 10 monclient(hunting): get_auth_request con 0x7f664413e810 auth_method 0 2024-03-27T21:41:28.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664952d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:28.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664952d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:28.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664952d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:28.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f6649d2e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:28.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f6649d2e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:28.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f6649d2e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:28.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664952d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:28.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664952d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:28.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664952d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:28.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664a52f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:28.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664a52f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:28.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664a52f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:28.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664952d640 10 monclient(hunting): handle_auth_done global_id 4951 payload 824 2024-03-27T21:41:28.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664952d640 1 -- 172.21.15.60:0/2452233279 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6644071a50 msgr2=0x7f6644139db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:28.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664952d640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6644071a50 0x7f6644139db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:28.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664952d640 1 -- 172.21.15.60:0/2452233279 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f664413ec50 msgr2=0x7f66440aae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:28.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664952d640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f664413ec50 0x7f66440aae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:28.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664952d640 10 monclient: _finish_hunting 0 2024-03-27T21:41:28.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664952d640 1 monclient: found mon.noname-b 2024-03-27T21:41:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664952d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664952d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:41:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664952d640 1 -- 172.21.15.60:0/2452233279 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6640009400 con 0x7f664413e810 2024-03-27T21:41:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f6649d2e640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6644071a50 0x7f6644139db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664a52f640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f664413ec50 0x7f66440aae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f664952d640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f664413e810 0x7f6644138de0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f663400c430 tx=0x7f663400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f663a7fc640 10 client.?.objecter ms_handle_connect 0x7f664413e810 2024-03-27T21:41:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.836+0000 7f663a7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f663a7fc640 1 -- 172.21.15.60:0/2452233279 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6634005ce0 con 0x7f664413e810 2024-03-27T21:41:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f663a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f663a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:28.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f663a7fc640 10 monclient: dump: 2024-03-27T21:41:28.845 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:28.845 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:28.845 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:28.845 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:28.845 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:28.845 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:28.845 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:28.845 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:28.845 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:28.845 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:28.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f663a7fc640 10 monclient: _finish_auth 0 2024-03-27T21:41:28.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f663a7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:41:28.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f663a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:28.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f664bf92640 5 monclient: authenticate success, global_id 4951 2024-03-27T21:41:28.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f663a7fc640 1 -- 172.21.15.60:0/2452233279 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6634005e40 con 0x7f664413e810 2024-03-27T21:41:28.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f663a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:28.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f663a7fc640 1 -- 172.21.15.60:0/2452233279 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66340072d0 con 0x7f664413e810 2024-03-27T21:41:28.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f663a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:28.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f663a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:28.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f663a7fc640 10 monclient: dump: 2024-03-27T21:41:28.846 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:28.846 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:28.846 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:28.846 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:28.847 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:28.847 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:28.847 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:28.847 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:28.847 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:28.847 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:28.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f663bfff640 4 set_mon_vals no callback set 2024-03-27T21:41:28.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f664bf92640 10 monclient: _renew_subs 2024-03-27T21:41:28.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f664bf92640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:28.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f664bf92640 1 -- 172.21.15.60:0/2452233279 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6644139890 con 0x7f664413e810 2024-03-27T21:41:28.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f664bf92640 10 client.4951.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:28.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f664bf92640 10 monclient: _renew_subs 2024-03-27T21:41:28.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f664bf92640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:28.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f664bf92640 1 -- 172.21.15.60:0/2452233279 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6644137760 con 0x7f664413e810 2024-03-27T21:41:28.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.837+0000 7f664bf92640 1 librados: init done 2024-03-27T21:41:28.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.838+0000 7f664bf92640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:41:28.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.838+0000 7f664bf92640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:28.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.838+0000 7f664bf92640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:28.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.838+0000 7f664bf92640 1 -- 172.21.15.60:0/2452233279 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6644139ab0 con 0x7f664413e810 2024-03-27T21:41:28.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.838+0000 7f663a7fc640 1 -- 172.21.15.60:0/2452233279 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6634007430 con 0x7f664413e810 2024-03-27T21:41:28.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f663a7fc640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f662806c590 0x7f662806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:28.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f663a7fc640 1 -- 172.21.15.60:0/2452233279 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f66340885d0 con 0x7f664413e810 2024-03-27T21:41:28.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f663a7fc640 10 client.4951.objecter ms_dispatch 0x7f6644000b90 osd_map(61..61 src has 1..61) v4 2024-03-27T21:41:28.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f663a7fc640 3 client.4951.objecter handle_osd_map got epochs [61,61] > 0 2024-03-27T21:41:28.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f663a7fc640 3 client.4951.objecter handle_osd_map decoding full epoch 61 2024-03-27T21:41:28.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f6649d2e640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f662806c590 0x7f662806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:28.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f663a7fc640 20 client.4951.objecter dump_active .. 0 homeless 2024-03-27T21:41:28.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f663a7fc640 1 -- 172.21.15.60:0/2452233279 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v61) v1 ==== 54+0+16 (secure 0 0 0) 0x7f663408f070 con 0x7f664413e810 2024-03-27T21:41:28.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f663a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:28.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f663a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:28.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f6649d2e640 10 monclient: get_auth_request con 0x7f662806c590 auth_method 0 2024-03-27T21:41:28.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f664bf92640 10 client.4951.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:28.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign -1 1 2024-03-27T21:41:28.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f664bf92640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f66441320b0 0x7f6644132490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:28.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f664413f0c0 osd=0 3 2024-03-27T21:41:28.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f664413f0c0 osd=0 3 2024-03-27T21:41:28.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f664413f0c0 osd=0 3 2024-03-27T21:41:28.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f664bf92640 15 client.4951.objecter _session_command_op_remove -1 1 2024-03-27T21:41:28.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f664bf92640 20 client.4951.objecter get_session s=0x7f664413f0c0 osd=0 3 2024-03-27T21:41:28.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign 0 1 2024-03-27T21:41:28.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f664413f0c0 osd=0 4 2024-03-27T21:41:28.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f664bf92640 10 client.4951.objecter _send_command 1 2024-03-27T21:41:28.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f664bf92640 1 -- 172.21.15.60:0/2452233279 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6644069470 con 0x7f66441320b0 2024-03-27T21:41:28.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f664a52f640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f66441320b0 0x7f6644132490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:28.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f663a7fc640 10 client.4951.objecter ms_handle_connect 0x7f662806c590 2024-03-27T21:41:28.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f6649d2e640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f662806c590 0x7f662806ea50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f664002b770 tx=0x7f6640008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:28.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.839+0000 7f664a52f640 10 monclient: get_auth_request con 0x7f66441320b0 auth_method 0 2024-03-27T21:41:28.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.840+0000 7f664a52f640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f66441320b0 0x7f6644132490 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:28.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.840+0000 7f663a7fc640 10 client.4951.objecter ms_handle_connect 0x7f66441320b0 2024-03-27T21:41:28.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.840+0000 7f663a7fc640 1 -- 172.21.15.60:0/2452233279 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6644069470 con 0x7f66441320b0 2024-03-27T21:41:28.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.840+0000 7f663a7fc640 10 client.4951.objecter ms_dispatch 0x7f6644000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:41:28.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.840+0000 7f663a7fc640 10 client.4951.objecter _finish_command 1 = system:0 2024-03-27T21:41:28.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.840+0000 7f663a7fc640 20 client.4951.objecter put_session s=0x7f664413f0c0 osd=0 4 2024-03-27T21:41:28.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.840+0000 7f663a7fc640 15 client.4951.objecter _session_command_op_remove 0 1 2024-03-27T21:41:28.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.854+0000 7f664bf92640 10 client.4951.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:28.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.854+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign -1 2 2024-03-27T21:41:28.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.854+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f664413f0c0 osd=0 3 2024-03-27T21:41:28.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.854+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f664413f0c0 osd=0 3 2024-03-27T21:41:28.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.854+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f664413f0c0 osd=0 3 2024-03-27T21:41:28.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.854+0000 7f664bf92640 15 client.4951.objecter _session_command_op_remove -1 2 2024-03-27T21:41:28.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.854+0000 7f664bf92640 20 client.4951.objecter get_session s=0x7f664413f0c0 osd=0 3 2024-03-27T21:41:28.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.854+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign 0 2 2024-03-27T21:41:28.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.854+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f664413f0c0 osd=0 4 2024-03-27T21:41:28.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.854+0000 7f664bf92640 10 client.4951.objecter _send_command 2 2024-03-27T21:41:28.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.854+0000 7f664bf92640 1 -- 172.21.15.60:0/2452233279 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f66440a9260 con 0x7f66441320b0 2024-03-27T21:41:28.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.856+0000 7f663a7fc640 1 -- 172.21.15.60:0/2452233279 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f66440a9260 con 0x7f66441320b0 2024-03-27T21:41:28.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.856+0000 7f663a7fc640 10 client.4951.objecter ms_dispatch 0x7f6644000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:28.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.856+0000 7f663a7fc640 10 client.4951.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:28.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.856+0000 7f663a7fc640 20 client.4951.objecter put_session s=0x7f664413f0c0 osd=0 4 2024-03-27T21:41:28.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.856+0000 7f663a7fc640 15 client.4951.objecter _session_command_op_remove 0 2 2024-03-27T21:41:28.855 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:41:28.855 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:28.855 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:28.856 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:41:28.857 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:41:28.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.856+0000 7f664bf92640 10 client.4951.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:28.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.856+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign -1 3 2024-03-27T21:41:28.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.856+0000 7f664bf92640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f66441367a0 0x7f6644136b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:28.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.856+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f66441329d0 osd=1 3 2024-03-27T21:41:28.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.856+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f66441329d0 osd=1 3 2024-03-27T21:41:28.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.856+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f66441329d0 osd=1 3 2024-03-27T21:41:28.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.856+0000 7f664bf92640 15 client.4951.objecter _session_command_op_remove -1 3 2024-03-27T21:41:28.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.856+0000 7f664bf92640 20 client.4951.objecter get_session s=0x7f66441329d0 osd=1 3 2024-03-27T21:41:28.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.856+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign 1 3 2024-03-27T21:41:28.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.856+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f66441329d0 osd=1 4 2024-03-27T21:41:28.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.856+0000 7f664bf92640 10 client.4951.objecter _send_command 3 2024-03-27T21:41:28.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.856+0000 7f664bf92640 1 -- 172.21.15.60:0/2452233279 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66441370c0 con 0x7f66441367a0 2024-03-27T21:41:28.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.856+0000 7f6649d2e640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f66441367a0 0x7f6644136b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:28.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.856+0000 7f6649d2e640 10 monclient: get_auth_request con 0x7f66441367a0 auth_method 0 2024-03-27T21:41:28.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.857+0000 7f663a7fc640 10 client.4951.objecter ms_handle_connect 0x7f66441367a0 2024-03-27T21:41:28.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.857+0000 7f6649d2e640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f66441367a0 0x7f6644136b80 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:28.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.857+0000 7f663a7fc640 1 -- 172.21.15.60:0/2452233279 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f66441370c0 con 0x7f66441367a0 2024-03-27T21:41:28.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.857+0000 7f663a7fc640 10 client.4951.objecter ms_dispatch 0x7f6644000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:41:28.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.857+0000 7f663a7fc640 10 client.4951.objecter _finish_command 3 = system:0 2024-03-27T21:41:28.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.857+0000 7f663a7fc640 20 client.4951.objecter put_session s=0x7f66441329d0 osd=1 4 2024-03-27T21:41:28.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.857+0000 7f663a7fc640 15 client.4951.objecter _session_command_op_remove 1 3 2024-03-27T21:41:28.859 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:28.866+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:28.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.885+0000 7f664bf92640 10 client.4951.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:28.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.885+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign -1 4 2024-03-27T21:41:28.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.885+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f66441329d0 osd=1 3 2024-03-27T21:41:28.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.885+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f66441329d0 osd=1 3 2024-03-27T21:41:28.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.885+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f66441329d0 osd=1 3 2024-03-27T21:41:28.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.885+0000 7f664bf92640 15 client.4951.objecter _session_command_op_remove -1 4 2024-03-27T21:41:28.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.885+0000 7f664bf92640 20 client.4951.objecter get_session s=0x7f66441329d0 osd=1 3 2024-03-27T21:41:28.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.885+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign 1 4 2024-03-27T21:41:28.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.885+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f66441329d0 osd=1 4 2024-03-27T21:41:28.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.885+0000 7f664bf92640 10 client.4951.objecter _send_command 4 2024-03-27T21:41:28.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.885+0000 7f664bf92640 1 -- 172.21.15.60:0/2452233279 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f664414cdf0 con 0x7f66441367a0 2024-03-27T21:41:28.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.887+0000 7f663a7fc640 1 -- 172.21.15.60:0/2452233279 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f664414cdf0 con 0x7f66441367a0 2024-03-27T21:41:28.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.887+0000 7f663a7fc640 10 client.4951.objecter ms_dispatch 0x7f6644000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:28.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.887+0000 7f663a7fc640 10 client.4951.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:28.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.887+0000 7f663a7fc640 20 client.4951.objecter put_session s=0x7f66441329d0 osd=1 4 2024-03-27T21:41:28.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.887+0000 7f663a7fc640 15 client.4951.objecter _session_command_op_remove 1 4 2024-03-27T21:41:28.874 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:41:28.876 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:41:28.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.887+0000 7f664bf92640 10 client.4951.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:28.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.887+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign -1 5 2024-03-27T21:41:28.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.887+0000 7f664bf92640 20 client.4951.objecter _get_session osd=-1 returning homeless 2024-03-27T21:41:28.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.887+0000 7f664bf92640 10 client.4951.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:28.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.887+0000 7f664bf92640 10 monclient: _renew_subs 2024-03-27T21:41:28.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.887+0000 7f664bf92640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:28.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.887+0000 7f664bf92640 1 -- 172.21.15.60:0/2452233279 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=62}) v3 -- 0x7f664413b730 con 0x7f664413e810 2024-03-27T21:41:28.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.887+0000 7f664bf92640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:28.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.887+0000 7f664bf92640 1 -- 172.21.15.60:0/2452233279 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f664413b9e0 con 0x7f664413e810 2024-03-27T21:41:28.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.888+0000 7f663a7fc640 1 -- 172.21.15.60:0/2452233279 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6634053ac0 con 0x7f664413e810 2024-03-27T21:41:28.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.888+0000 7f663a7fc640 10 monclient: handle_get_version_reply finishing 1 version 61 2024-03-27T21:41:28.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.888+0000 7f663bfff640 10 client.4951.objecter _check_command_map_dne tid 5 current 61 map_dne_bound 61 2024-03-27T21:41:28.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.888+0000 7f663bfff640 10 client.4951.objecter _finish_command 5 = osd:6 osd down 2024-03-27T21:41:28.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.888+0000 7f663bfff640 15 client.4951.objecter _session_command_op_remove -1 5 2024-03-27T21:41:28.878 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-27T21:41:28.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.888+0000 7f664bf92640 10 client.4951.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:28.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.888+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign -1 6 2024-03-27T21:41:28.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.888+0000 7f664bf92640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f664402b170 0x7f664402b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:28.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.888+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f664413fe30 osd=3 3 2024-03-27T21:41:28.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.888+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f664413fe30 osd=3 3 2024-03-27T21:41:28.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.888+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f664413fe30 osd=3 3 2024-03-27T21:41:28.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.888+0000 7f664bf92640 15 client.4951.objecter _session_command_op_remove -1 6 2024-03-27T21:41:28.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.888+0000 7f664bf92640 20 client.4951.objecter get_session s=0x7f664413fe30 osd=3 3 2024-03-27T21:41:28.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.888+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign 3 6 2024-03-27T21:41:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.888+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f664413fe30 osd=3 4 2024-03-27T21:41:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.888+0000 7f664bf92640 10 client.4951.objecter _send_command 6 2024-03-27T21:41:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.888+0000 7f664bf92640 1 -- 172.21.15.60:0/2452233279 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f664402ba90 con 0x7f664402b170 2024-03-27T21:41:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.888+0000 7f664952d640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f664402b170 0x7f664402b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.888+0000 7f664952d640 10 monclient: get_auth_request con 0x7f664402b170 auth_method 0 2024-03-27T21:41:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.888+0000 7f663a7fc640 10 client.4951.objecter ms_handle_connect 0x7f664402b170 2024-03-27T21:41:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.888+0000 7f664952d640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f664402b170 0x7f664402b550 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.889+0000 7f663a7fc640 1 -- 172.21.15.60:0/2452233279 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f664402ba90 con 0x7f664402b170 2024-03-27T21:41:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.889+0000 7f663a7fc640 10 client.4951.objecter ms_dispatch 0x7f6644000b90 command_reply(tid 6: 0 ) v1 2024-03-27T21:41:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.889+0000 7f663a7fc640 10 client.4951.objecter _finish_command 6 = system:0 2024-03-27T21:41:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.889+0000 7f663a7fc640 20 client.4951.objecter put_session s=0x7f664413fe30 osd=3 4 2024-03-27T21:41:28.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.889+0000 7f663a7fc640 15 client.4951.objecter _session_command_op_remove 3 6 2024-03-27T21:41:28.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.903+0000 7f664bf92640 10 client.4951.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:28.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.903+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign -1 7 2024-03-27T21:41:28.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.903+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f664413fe30 osd=3 3 2024-03-27T21:41:28.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.903+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f664413fe30 osd=3 3 2024-03-27T21:41:28.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.903+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f664413fe30 osd=3 3 2024-03-27T21:41:28.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.903+0000 7f664bf92640 15 client.4951.objecter _session_command_op_remove -1 7 2024-03-27T21:41:28.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.903+0000 7f664bf92640 20 client.4951.objecter get_session s=0x7f664413fe30 osd=3 3 2024-03-27T21:41:28.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.903+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign 3 7 2024-03-27T21:41:28.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.903+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f664413fe30 osd=3 4 2024-03-27T21:41:28.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.903+0000 7f664bf92640 10 client.4951.objecter _send_command 7 2024-03-27T21:41:28.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.903+0000 7f664bf92640 1 -- 172.21.15.60:0/2452233279 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f664402c450 con 0x7f664402b170 2024-03-27T21:41:28.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.905+0000 7f663a7fc640 1 -- 172.21.15.60:0/2452233279 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f664402c450 con 0x7f664402b170 2024-03-27T21:41:28.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.905+0000 7f663a7fc640 10 client.4951.objecter ms_dispatch 0x7f6644000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:28.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.905+0000 7f663a7fc640 10 client.4951.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:28.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.905+0000 7f663a7fc640 20 client.4951.objecter put_session s=0x7f664413fe30 osd=3 4 2024-03-27T21:41:28.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.905+0000 7f663a7fc640 15 client.4951.objecter _session_command_op_remove 3 7 2024-03-27T21:41:28.894 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:41:28.895 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:41:28.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.906+0000 7f664bf92640 10 client.4951.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:28.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.906+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign -1 8 2024-03-27T21:41:28.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.906+0000 7f664bf92640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f664413fa20 0x7f66441e9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:28.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.906+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f664413f8b0 osd=4 3 2024-03-27T21:41:28.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.906+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f664413f8b0 osd=4 3 2024-03-27T21:41:28.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.906+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f664413f8b0 osd=4 3 2024-03-27T21:41:28.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.906+0000 7f664bf92640 15 client.4951.objecter _session_command_op_remove -1 8 2024-03-27T21:41:28.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.906+0000 7f664bf92640 20 client.4951.objecter get_session s=0x7f664413f8b0 osd=4 3 2024-03-27T21:41:28.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.906+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign 4 8 2024-03-27T21:41:28.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.906+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f664413f8b0 osd=4 4 2024-03-27T21:41:28.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.906+0000 7f664bf92640 10 client.4951.objecter _send_command 8 2024-03-27T21:41:28.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.906+0000 7f664bf92640 1 -- 172.21.15.60:0/2452233279 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66441ec510 con 0x7f664413fa20 2024-03-27T21:41:28.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.906+0000 7f664a52f640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f664413fa20 0x7f66441e9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:28.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.906+0000 7f664a52f640 10 monclient: get_auth_request con 0x7f664413fa20 auth_method 0 2024-03-27T21:41:28.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.906+0000 7f663a7fc640 10 client.4951.objecter ms_handle_connect 0x7f664413fa20 2024-03-27T21:41:28.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.906+0000 7f664a52f640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f664413fa20 0x7f66441e9740 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:28.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.907+0000 7f663a7fc640 1 -- 172.21.15.60:0/2452233279 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f66441ec510 con 0x7f664413fa20 2024-03-27T21:41:28.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.907+0000 7f663a7fc640 10 client.4951.objecter ms_dispatch 0x7f6644000b90 command_reply(tid 8: 0 ) v1 2024-03-27T21:41:28.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.907+0000 7f663a7fc640 10 client.4951.objecter _finish_command 8 = system:0 2024-03-27T21:41:28.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.907+0000 7f663a7fc640 20 client.4951.objecter put_session s=0x7f664413f8b0 osd=4 4 2024-03-27T21:41:28.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.907+0000 7f663a7fc640 15 client.4951.objecter _session_command_op_remove 4 8 2024-03-27T21:41:28.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.922+0000 7f664bf92640 10 client.4951.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:28.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.922+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign -1 9 2024-03-27T21:41:28.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.922+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f664413f8b0 osd=4 3 2024-03-27T21:41:28.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.922+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f664413f8b0 osd=4 3 2024-03-27T21:41:28.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.922+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f664413f8b0 osd=4 3 2024-03-27T21:41:28.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.922+0000 7f664bf92640 15 client.4951.objecter _session_command_op_remove -1 9 2024-03-27T21:41:28.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.922+0000 7f664bf92640 20 client.4951.objecter get_session s=0x7f664413f8b0 osd=4 3 2024-03-27T21:41:28.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.922+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign 4 9 2024-03-27T21:41:28.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.922+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f664413f8b0 osd=4 4 2024-03-27T21:41:28.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.922+0000 7f664bf92640 10 client.4951.objecter _send_command 9 2024-03-27T21:41:28.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.922+0000 7f664bf92640 1 -- 172.21.15.60:0/2452233279 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f664402bed0 con 0x7f664413fa20 2024-03-27T21:41:28.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.924+0000 7f663a7fc640 1 -- 172.21.15.60:0/2452233279 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f664402bed0 con 0x7f664413fa20 2024-03-27T21:41:28.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.924+0000 7f663a7fc640 10 client.4951.objecter ms_dispatch 0x7f6644000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:28.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.924+0000 7f663a7fc640 10 client.4951.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:28.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.924+0000 7f663a7fc640 20 client.4951.objecter put_session s=0x7f664413f8b0 osd=4 4 2024-03-27T21:41:28.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.924+0000 7f663a7fc640 15 client.4951.objecter _session_command_op_remove 4 9 2024-03-27T21:41:28.911 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:41:28.912 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:41:28.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.924+0000 7f664bf92640 10 client.4951.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:28.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.924+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign -1 10 2024-03-27T21:41:28.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.924+0000 7f664bf92640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f6644066250 0x7f6644066690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:28.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.924+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f664413b0a0 osd=5 3 2024-03-27T21:41:28.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.924+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f664413b0a0 osd=5 3 2024-03-27T21:41:28.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.924+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f664413b0a0 osd=5 3 2024-03-27T21:41:28.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.924+0000 7f664bf92640 15 client.4951.objecter _session_command_op_remove -1 10 2024-03-27T21:41:28.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.924+0000 7f664bf92640 20 client.4951.objecter get_session s=0x7f664413b0a0 osd=5 3 2024-03-27T21:41:28.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.924+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign 5 10 2024-03-27T21:41:28.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.924+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f664413b0a0 osd=5 4 2024-03-27T21:41:28.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.924+0000 7f664bf92640 10 client.4951.objecter _send_command 10 2024-03-27T21:41:28.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.924+0000 7f664bf92640 1 -- 172.21.15.60:0/2452233279 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66441ef810 con 0x7f6644066250 2024-03-27T21:41:28.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.924+0000 7f6649d2e640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f6644066250 0x7f6644066690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:28.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.924+0000 7f6649d2e640 10 monclient: get_auth_request con 0x7f6644066250 auth_method 0 2024-03-27T21:41:28.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.925+0000 7f663a7fc640 10 client.4951.objecter ms_handle_connect 0x7f6644066250 2024-03-27T21:41:28.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.925+0000 7f6649d2e640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f6644066250 0x7f6644066690 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:28.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.925+0000 7f663a7fc640 1 -- 172.21.15.60:0/2452233279 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f66441ef810 con 0x7f6644066250 2024-03-27T21:41:28.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.925+0000 7f663a7fc640 10 client.4951.objecter ms_dispatch 0x7f6644000b90 command_reply(tid 10: 0 ) v1 2024-03-27T21:41:28.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.925+0000 7f663a7fc640 10 client.4951.objecter _finish_command 10 = system:0 2024-03-27T21:41:28.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.925+0000 7f663a7fc640 20 client.4951.objecter put_session s=0x7f664413b0a0 osd=5 4 2024-03-27T21:41:28.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.925+0000 7f663a7fc640 15 client.4951.objecter _session_command_op_remove 5 10 2024-03-27T21:41:28.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.939+0000 7f664bf92640 10 client.4951.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:28.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.939+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign -1 11 2024-03-27T21:41:28.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.939+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f664413b0a0 osd=5 3 2024-03-27T21:41:28.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.939+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f664413b0a0 osd=5 3 2024-03-27T21:41:28.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.939+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f664413b0a0 osd=5 3 2024-03-27T21:41:28.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.939+0000 7f664bf92640 15 client.4951.objecter _session_command_op_remove -1 11 2024-03-27T21:41:28.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.939+0000 7f664bf92640 20 client.4951.objecter get_session s=0x7f664413b0a0 osd=5 3 2024-03-27T21:41:28.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.939+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign 5 11 2024-03-27T21:41:28.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.939+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f664413b0a0 osd=5 4 2024-03-27T21:41:28.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.939+0000 7f664bf92640 10 client.4951.objecter _send_command 11 2024-03-27T21:41:28.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.939+0000 7f664bf92640 1 -- 172.21.15.60:0/2452233279 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f66441eae00 con 0x7f6644066250 2024-03-27T21:41:28.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.941+0000 7f663a7fc640 1 -- 172.21.15.60:0/2452233279 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f66441eae00 con 0x7f6644066250 2024-03-27T21:41:28.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.941+0000 7f663a7fc640 10 client.4951.objecter ms_dispatch 0x7f6644000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:28.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.941+0000 7f663a7fc640 10 client.4951.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:28.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.941+0000 7f663a7fc640 20 client.4951.objecter put_session s=0x7f664413b0a0 osd=5 4 2024-03-27T21:41:28.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.941+0000 7f663a7fc640 15 client.4951.objecter _session_command_op_remove 5 11 2024-03-27T21:41:28.928 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:41:28.929 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:41:28.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.941+0000 7f664bf92640 10 client.4951.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:28.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.941+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign -1 12 2024-03-27T21:41:28.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.941+0000 7f664bf92640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f66441eb480 0x7f66441eb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:28.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.941+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f66441eae00 osd=6 3 2024-03-27T21:41:28.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.941+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f66441eae00 osd=6 3 2024-03-27T21:41:28.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.941+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f66441eae00 osd=6 3 2024-03-27T21:41:28.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.941+0000 7f664bf92640 15 client.4951.objecter _session_command_op_remove -1 12 2024-03-27T21:41:28.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.941+0000 7f664bf92640 20 client.4951.objecter get_session s=0x7f66441eae00 osd=6 3 2024-03-27T21:41:28.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.941+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign 6 12 2024-03-27T21:41:28.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.941+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f66441eae00 osd=6 4 2024-03-27T21:41:28.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.941+0000 7f664bf92640 10 client.4951.objecter _send_command 12 2024-03-27T21:41:28.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.941+0000 7f664bf92640 1 -- 172.21.15.60:0/2452233279 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66441ebe20 con 0x7f66441eb480 2024-03-27T21:41:28.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.941+0000 7f664952d640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f66441eb480 0x7f66441eb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:28.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.941+0000 7f664952d640 10 monclient: get_auth_request con 0x7f66441eb480 auth_method 0 2024-03-27T21:41:28.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.942+0000 7f663a7fc640 10 client.4951.objecter ms_handle_connect 0x7f66441eb480 2024-03-27T21:41:28.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.942+0000 7f664952d640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f66441eb480 0x7f66441eb8e0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:28.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.942+0000 7f663a7fc640 1 -- 172.21.15.60:0/2452233279 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f66441ebe20 con 0x7f66441eb480 2024-03-27T21:41:28.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.942+0000 7f663a7fc640 10 client.4951.objecter ms_dispatch 0x7f6644000b90 command_reply(tid 12: 0 ) v1 2024-03-27T21:41:28.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.942+0000 7f663a7fc640 10 client.4951.objecter _finish_command 12 = system:0 2024-03-27T21:41:28.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.942+0000 7f663a7fc640 20 client.4951.objecter put_session s=0x7f66441eae00 osd=6 4 2024-03-27T21:41:28.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.942+0000 7f663a7fc640 15 client.4951.objecter _session_command_op_remove 6 12 2024-03-27T21:41:28.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.956+0000 7f664bf92640 10 client.4951.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:28.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.956+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign -1 13 2024-03-27T21:41:28.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.956+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f66441eae00 osd=6 3 2024-03-27T21:41:28.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.956+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f66441eae00 osd=6 3 2024-03-27T21:41:28.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.956+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f66441eae00 osd=6 3 2024-03-27T21:41:28.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.956+0000 7f664bf92640 15 client.4951.objecter _session_command_op_remove -1 13 2024-03-27T21:41:28.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.956+0000 7f664bf92640 20 client.4951.objecter get_session s=0x7f66441eae00 osd=6 3 2024-03-27T21:41:28.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.956+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign 6 13 2024-03-27T21:41:28.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.956+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f66441eae00 osd=6 4 2024-03-27T21:41:28.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.956+0000 7f664bf92640 10 client.4951.objecter _send_command 13 2024-03-27T21:41:28.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.956+0000 7f664bf92640 1 -- 172.21.15.60:0/2452233279 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f66441ebe20 con 0x7f66441eb480 2024-03-27T21:41:28.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.957+0000 7f663a7fc640 1 -- 172.21.15.60:0/2452233279 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f66441ebe20 con 0x7f66441eb480 2024-03-27T21:41:28.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.957+0000 7f663a7fc640 10 client.4951.objecter ms_dispatch 0x7f6644000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:28.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.957+0000 7f663a7fc640 10 client.4951.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:28.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.957+0000 7f663a7fc640 20 client.4951.objecter put_session s=0x7f66441eae00 osd=6 4 2024-03-27T21:41:28.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.957+0000 7f663a7fc640 15 client.4951.objecter _session_command_op_remove 6 13 2024-03-27T21:41:28.944 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:41:28.946 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:41:28.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.957+0000 7f664bf92640 10 client.4951.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:28.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.957+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign -1 14 2024-03-27T21:41:28.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.957+0000 7f664bf92640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f66441f06a0 0x7f66441f2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:28.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.957+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f66441ebe20 osd=7 3 2024-03-27T21:41:28.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.958+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f66441ebe20 osd=7 3 2024-03-27T21:41:28.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.958+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f66441ebe20 osd=7 3 2024-03-27T21:41:28.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.958+0000 7f664bf92640 15 client.4951.objecter _session_command_op_remove -1 14 2024-03-27T21:41:28.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.958+0000 7f664bf92640 20 client.4951.objecter get_session s=0x7f66441ebe20 osd=7 3 2024-03-27T21:41:28.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.958+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign 7 14 2024-03-27T21:41:28.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.958+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f66441ebe20 osd=7 4 2024-03-27T21:41:28.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.958+0000 7f664bf92640 10 client.4951.objecter _send_command 14 2024-03-27T21:41:28.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.958+0000 7f664bf92640 1 -- 172.21.15.60:0/2452233279 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66441f5860 con 0x7f66441f06a0 2024-03-27T21:41:28.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.958+0000 7f664a52f640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f66441f06a0 0x7f66441f2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:28.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.958+0000 7f664a52f640 10 monclient: get_auth_request con 0x7f66441f06a0 auth_method 0 2024-03-27T21:41:28.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.958+0000 7f664a52f640 1 --2- 172.21.15.60:0/2452233279 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f66441f06a0 0x7f66441f2a90 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:28.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.958+0000 7f663a7fc640 10 client.4951.objecter ms_handle_connect 0x7f66441f06a0 2024-03-27T21:41:28.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.959+0000 7f663a7fc640 1 -- 172.21.15.60:0/2452233279 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f66441f5860 con 0x7f66441f06a0 2024-03-27T21:41:28.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.959+0000 7f663a7fc640 10 client.4951.objecter ms_dispatch 0x7f6644000b90 command_reply(tid 14: 0 ) v1 2024-03-27T21:41:28.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.959+0000 7f663a7fc640 10 client.4951.objecter _finish_command 14 = system:0 2024-03-27T21:41:28.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.959+0000 7f663a7fc640 20 client.4951.objecter put_session s=0x7f66441ebe20 osd=7 4 2024-03-27T21:41:28.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.959+0000 7f663a7fc640 15 client.4951.objecter _session_command_op_remove 7 14 2024-03-27T21:41:28.954 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:28.968+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:28.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.974+0000 7f664bf92640 10 client.4951.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.974+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign -1 15 2024-03-27T21:41:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.974+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f66441ebe20 osd=7 3 2024-03-27T21:41:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.974+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f66441ebe20 osd=7 3 2024-03-27T21:41:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.974+0000 7f664bf92640 20 client.4951.objecter _get_session s=0x7f66441ebe20 osd=7 3 2024-03-27T21:41:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.974+0000 7f664bf92640 15 client.4951.objecter _session_command_op_remove -1 15 2024-03-27T21:41:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.974+0000 7f664bf92640 20 client.4951.objecter get_session s=0x7f66441ebe20 osd=7 3 2024-03-27T21:41:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.974+0000 7f664bf92640 15 client.4951.objecter _session_command_op_assign 7 15 2024-03-27T21:41:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.974+0000 7f664bf92640 20 client.4951.objecter put_session s=0x7f66441ebe20 osd=7 4 2024-03-27T21:41:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.974+0000 7f664bf92640 10 client.4951.objecter _send_command 15 2024-03-27T21:41:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.974+0000 7f664bf92640 1 -- 172.21.15.60:0/2452233279 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f66441f2fd0 con 0x7f66441f06a0 2024-03-27T21:41:28.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.976+0000 7f663a7fc640 1 -- 172.21.15.60:0/2452233279 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f66441f2fd0 con 0x7f66441f06a0 2024-03-27T21:41:28.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.976+0000 7f663a7fc640 10 client.4951.objecter ms_dispatch 0x7f6644000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:28.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.976+0000 7f663a7fc640 10 client.4951.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:28.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.976+0000 7f663a7fc640 20 client.4951.objecter put_session s=0x7f66441ebe20 osd=7 4 2024-03-27T21:41:28.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:28.976+0000 7f663a7fc640 15 client.4951.objecter _session_command_op_remove 7 15 2024-03-27T21:41:28.962 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:41:28.963 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:41:28.965 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T21:41:28.965 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:41:29.004 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:29.004 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:29.005 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:29.005 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:29.013 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:41:29.038 INFO:teuthology.orchestra.run.smithi060.stderr:noscrub is set 2024-03-27T21:41:29.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.051+0000 7f432effd640 1 -- 172.21.15.60:0/1345437927 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v62) v1 ==== 87+0+0 (secure 0 0 0) 0x7f4338039070 con 0x7f4340141630 2024-03-27T21:41:29.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.051+0000 7f432effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:41:29.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.051+0000 7f432effd640 10 monclient: _finish_command 2 = system:0 noscrub is set 2024-03-27T21:41:29.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.056+0000 7f4344f2e640 10 librados: watch_flush enter 2024-03-27T21:41:29.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.056+0000 7f4344f2e640 10 librados: watch_flush exit 2024-03-27T21:41:29.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.056+0000 7f4344f2e640 20 client.4787.objecter shutdown clearing up homeless session... 2024-03-27T21:41:29.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.056+0000 7f4344f2e640 10 client.4787.objecter successfully canceled tick 2024-03-27T21:41:29.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.057+0000 7f4344f2e640 1 -- 172.21.15.60:0/1345437927 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f432006c590 msgr2=0x7f432006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:29.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.057+0000 7f4344f2e640 1 --2- 172.21.15.60:0/1345437927 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f432006c590 0x7f432006ea50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4330004e40 tx=0x7f4330007f10 comp rx=0 tx=0).stop 2024-03-27T21:41:29.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.057+0000 7f4344f2e640 10 monclient: shutdown 2024-03-27T21:41:29.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.057+0000 7f4344f2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:29.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.057+0000 7f4344f2e640 1 -- 172.21.15.60:0/1345437927 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4340141630 msgr2=0x7f43401d9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:29.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.057+0000 7f4344f2e640 1 --2- 172.21.15.60:0/1345437927 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4340141630 0x7f43401d9820 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4340072190 tx=0x7f4338004a40 comp rx=0 tx=0).stop 2024-03-27T21:41:29.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.057+0000 7f4344f2e640 1 -- 172.21.15.60:0/1345437927 shutdown_connections 2024-03-27T21:41:29.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.057+0000 7f4344f2e640 1 --2- 172.21.15.60:0/1345437927 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4340141630 0x7f43401d9820 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:29.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.057+0000 7f4344f2e640 1 --2- 172.21.15.60:0/1345437927 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f432006c590 0x7f432006ea50 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:29.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.057+0000 7f4344f2e640 1 --2- 172.21.15.60:0/1345437927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f434013ef20 0x7f43401397f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:29.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.057+0000 7f4344f2e640 1 --2- 172.21.15.60:0/1345437927 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4340071a50 0x7f43400b6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:29.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.057+0000 7f4344f2e640 1 -- 172.21.15.60:0/1345437927 >> 172.21.15.60:0/1345437927 conn(0x7f43400b7300 msgr2=0x7f434013e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:29.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.057+0000 7f4344f2e640 1 -- 172.21.15.60:0/1345437927 shutdown_connections 2024-03-27T21:41:29.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.057+0000 7f4344f2e640 1 -- 172.21.15.60:0/1345437927 wait complete. 2024-03-27T21:41:29.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.057+0000 7f4344f2e640 1 librados: shutdown 2024-03-27T21:41:29.057 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:29.071+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:29.120 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:29.120 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:29.120 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:29.120 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:29.120 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:29.128 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:41:29.169 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:29.183+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:29.197 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-27T21:41:29.256 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:29.270+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:29.275 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:29.275 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:29.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:29.276 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-03-27T21:41:29.276 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.276 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:29.276 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:29.276 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 109.856363254, 2024-03-27T21:41:29.276 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43537978700000002, 2024-03-27T21:41:29.276 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:29.276 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.276 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:29.276 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:29.276 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:29.276 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-03-27T21:41:29.276 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.276 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:29.277 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.277 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:29.277 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:29.277 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.277 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.277 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.277 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:29.277 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:29.277 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.277 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.277 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.277 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:29.277 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432019+0000", 2024-03-27T21:41:29.277 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-03-27T21:41:29.277 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:29.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432027+0000", 2024-03-27T21:41:29.278 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1200000000000002e-06 2024-03-27T21:41:29.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:29.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432039+0000", 2024-03-27T21:41:29.278 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1123e-05 2024-03-27T21:41:29.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432051+0000", 2024-03-27T21:41:29.278 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-03-27T21:41:29.279 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.279 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.279 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:29.279 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432157+0000", 2024-03-27T21:41:29.279 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106846 2024-03-27T21:41:29.279 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.279 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.279 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-03-27T21:41:29.279 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432176+0000", 2024-03-27T21:41:29.279 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8490000000000001e-05 2024-03-27T21:41:29.279 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.279 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.279 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:29.279 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866170+0000", 2024-03-27T21:41:29.279 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43399370100000001 2024-03-27T21:41:29.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.280 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.280 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:29.280 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866335+0000", 2024-03-27T21:41:29.280 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016559699999999999 2024-03-27T21:41:29.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.280 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.280 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:29.280 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866495+0000", 2024-03-27T21:41:29.280 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015944200000000001 2024-03-27T21:41:29.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.280 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.280 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:29.280 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867012+0000", 2024-03-27T21:41:29.280 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0005176 2024-03-27T21:41:29.281 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.281 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.281 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:29.281 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867334+0000", 2024-03-27T21:41:29.281 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032158600000000001 2024-03-27T21:41:29.281 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.281 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.281 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:29.281 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867374+0000", 2024-03-27T21:41:29.281 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0547000000000003e-05 2024-03-27T21:41:29.281 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.281 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.281 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:29.281 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867400+0000", 2024-03-27T21:41:29.281 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5958000000000001e-05 2024-03-27T21:41:29.282 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.282 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:29.282 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.282 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.282 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.282 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:29.282 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:29.282 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 109.35898080299999, 2024-03-27T21:41:29.282 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064560310000000001, 2024-03-27T21:41:29.282 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:29.282 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.282 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:29.282 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:29.282 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:29.283 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 70 2024-03-27T21:41:29.283 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.283 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:29.283 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.283 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:29.283 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:29.283 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.283 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.283 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.283 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:29.283 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:29.283 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.283 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.283 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.283 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:29.284 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929402+0000", 2024-03-27T21:41:29.284 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.284 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.284 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.284 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:29.284 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929407+0000", 2024-03-27T21:41:29.284 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-06 2024-03-27T21:41:29.284 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.284 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.284 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:29.284 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929414+0000", 2024-03-27T21:41:29.284 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5689999999999999e-06 2024-03-27T21:41:29.284 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.284 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.284 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.284 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929419+0000", 2024-03-27T21:41:29.285 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-03-27T21:41:29.285 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.285 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.285 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:29.285 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930245+0000", 2024-03-27T21:41:29.285 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082521500000000002 2024-03-27T21:41:29.285 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.285 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.285 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:29.285 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930287+0000", 2024-03-27T21:41:29.285 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2305000000000002e-05 2024-03-27T21:41:29.285 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.285 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.285 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:29.285 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930367+0000", 2024-03-27T21:41:29.286 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9589e-05 2024-03-27T21:41:29.286 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.286 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.286 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:29.286 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.931214+0000", 2024-03-27T21:41:29.286 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000847827 2024-03-27T21:41:29.286 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.286 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.286 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:29.286 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935807+0000", 2024-03-27T21:41:29.286 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045923780000000003 2024-03-27T21:41:29.286 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.286 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.286 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:29.287 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935840+0000", 2024-03-27T21:41:29.287 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3501999999999999e-05 2024-03-27T21:41:29.287 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.287 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.287 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:29.287 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935859+0000", 2024-03-27T21:41:29.287 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8845e-05 2024-03-27T21:41:29.287 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.287 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:29.287 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.287 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.287 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.287 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:29.287 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:29.287 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 109.351721212, 2024-03-27T21:41:29.288 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071695429999999996, 2024-03-27T21:41:29.288 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:29.288 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.288 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:29.288 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:29.288 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:29.288 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-03-27T21:41:29.288 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.288 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:29.288 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.288 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:29.288 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:29.288 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.288 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.288 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.289 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:29.289 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:29.289 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.289 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.289 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.289 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:29.289 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936662+0000", 2024-03-27T21:41:29.289 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.289 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.289 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.289 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:29.289 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936737+0000", 2024-03-27T21:41:29.289 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5303999999999996e-05 2024-03-27T21:41:29.289 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.289 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.290 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:29.290 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936746+0000", 2024-03-27T21:41:29.290 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6340000000000007e-06 2024-03-27T21:41:29.290 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.290 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.290 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.290 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936753+0000", 2024-03-27T21:41:29.290 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-03-27T21:41:29.290 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.290 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.290 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:29.290 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937926+0000", 2024-03-27T21:41:29.290 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011731280000000001 2024-03-27T21:41:29.290 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.291 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.291 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:29.291 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937969+0000", 2024-03-27T21:41:29.291 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3285999999999999e-05 2024-03-27T21:41:29.291 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.291 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.291 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:29.291 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938040+0000", 2024-03-27T21:41:29.291 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0358000000000006e-05 2024-03-27T21:41:29.291 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.291 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.291 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:29.291 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939324+0000", 2024-03-27T21:41:29.291 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012844899999999999 2024-03-27T21:41:29.291 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.292 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.292 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:29.292 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943780+0000", 2024-03-27T21:41:29.292 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044555630000000001 2024-03-27T21:41:29.292 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.292 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.292 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:29.292 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943811+0000", 2024-03-27T21:41:29.292 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1415999999999999e-05 2024-03-27T21:41:29.292 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.292 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.292 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:29.292 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943832+0000", 2024-03-27T21:41:29.292 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0981000000000001e-05 2024-03-27T21:41:29.292 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.292 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:29.292 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.293 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.293 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.293 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:29.293 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:29.293 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 109.35162319600001, 2024-03-27T21:41:29.293 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071139740000000003, 2024-03-27T21:41:29.293 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:29.293 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.293 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:29.293 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:29.293 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:29.293 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-03-27T21:41:29.293 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.293 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:29.293 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.294 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:29.294 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:29.294 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.294 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.294 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.294 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:29.294 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:29.294 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.294 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.294 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.294 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:29.294 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936760+0000", 2024-03-27T21:41:29.294 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.294 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.294 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.295 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:29.295 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936765+0000", 2024-03-27T21:41:29.295 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6330000000000004e-06 2024-03-27T21:41:29.295 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.295 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.295 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:29.295 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936771+0000", 2024-03-27T21:41:29.295 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0560000000000001e-06 2024-03-27T21:41:29.295 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.295 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.295 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.295 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936776+0000", 2024-03-27T21:41:29.295 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1510000000000003e-06 2024-03-27T21:41:29.295 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.295 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.296 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:29.296 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938605+0000", 2024-03-27T21:41:29.296 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001829589 2024-03-27T21:41:29.296 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.296 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.296 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:29.296 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938657+0000", 2024-03-27T21:41:29.296 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1860000000000002e-05 2024-03-27T21:41:29.296 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.296 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.296 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:29.296 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938734+0000", 2024-03-27T21:41:29.296 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6639999999999998e-05 2024-03-27T21:41:29.296 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.296 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.297 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:29.297 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939562+0000", 2024-03-27T21:41:29.297 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082803100000000001 2024-03-27T21:41:29.297 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.297 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.297 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:29.297 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943838+0000", 2024-03-27T21:41:29.297 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042757050000000003 2024-03-27T21:41:29.297 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.297 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.297 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:29.297 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943859+0000", 2024-03-27T21:41:29.297 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1175e-05 2024-03-27T21:41:29.297 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.297 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.298 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:29.298 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943875+0000", 2024-03-27T21:41:29.298 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5744e-05 2024-03-27T21:41:29.298 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.298 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:29.298 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.298 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.298 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.298 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:29.298 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:29.298 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 109.20810914499999, 2024-03-27T21:41:29.298 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059712230000000003, 2024-03-27T21:41:29.298 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:29.298 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.298 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:29.299 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:29.299 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:29.299 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 112 2024-03-27T21:41:29.299 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.299 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:29.299 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.299 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:29.299 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:29.299 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.299 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.299 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.299 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:29.299 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:29.299 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.299 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.299 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.300 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:29.300 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080274+0000", 2024-03-27T21:41:29.300 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.301 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.301 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.302 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:29.302 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080318+0000", 2024-03-27T21:41:29.302 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3634999999999998e-05 2024-03-27T21:41:29.302 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.302 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.302 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:29.302 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080329+0000", 2024-03-27T21:41:29.302 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1232e-05 2024-03-27T21:41:29.302 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.302 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.302 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.303 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080333+0000", 2024-03-27T21:41:29.303 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6009999999999999e-06 2024-03-27T21:41:29.303 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.303 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.303 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:29.303 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083786+0000", 2024-03-27T21:41:29.303 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034528940000000002 2024-03-27T21:41:29.303 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.303 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.303 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:29.303 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083861+0000", 2024-03-27T21:41:29.303 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4893000000000003e-05 2024-03-27T21:41:29.303 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.303 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.303 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:29.304 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083981+0000", 2024-03-27T21:41:29.304 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012058299999999999 2024-03-27T21:41:29.304 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.304 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.304 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:29.304 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086148+0000", 2024-03-27T21:41:29.304 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021671469999999999 2024-03-27T21:41:29.304 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.304 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.304 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:29.304 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086195+0000", 2024-03-27T21:41:29.304 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6768999999999999e-05 2024-03-27T21:41:29.304 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.304 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.304 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:29.305 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086224+0000", 2024-03-27T21:41:29.305 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8758000000000001e-05 2024-03-27T21:41:29.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.305 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.305 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:29.305 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086246+0000", 2024-03-27T21:41:29.305 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2064000000000001e-05 2024-03-27T21:41:29.305 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.305 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:29.305 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.305 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.305 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:29.305 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:29.305 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 109.20804619800001, 2024-03-27T21:41:29.306 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066196320000000003, 2024-03-27T21:41:29.306 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:29.306 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.306 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:29.306 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:29.306 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:29.306 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113 2024-03-27T21:41:29.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.306 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:29.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.306 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:29.306 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:29.306 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.307 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.307 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:29.307 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:29.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.307 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.307 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:29.307 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080337+0000", 2024-03-27T21:41:29.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.307 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.307 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:29.307 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080340+0000", 2024-03-27T21:41:29.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6690000000000002e-06 2024-03-27T21:41:29.308 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.308 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.308 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:29.308 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080344+0000", 2024-03-27T21:41:29.308 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6550000000000002e-06 2024-03-27T21:41:29.308 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.308 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.308 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.308 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080348+0000", 2024-03-27T21:41:29.308 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9169999999999999e-06 2024-03-27T21:41:29.308 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.308 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.308 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:29.308 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084595+0000", 2024-03-27T21:41:29.308 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042471719999999996 2024-03-27T21:41:29.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.309 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:29.309 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084676+0000", 2024-03-27T21:41:29.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1147000000000001e-05 2024-03-27T21:41:29.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.309 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:29.309 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084796+0000", 2024-03-27T21:41:29.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012032900000000001 2024-03-27T21:41:29.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.309 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:29.309 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086190+0000", 2024-03-27T21:41:29.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013934399999999999 2024-03-27T21:41:29.310 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.310 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:29.310 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086905+0000", 2024-03-27T21:41:29.310 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071536199999999999 2024-03-27T21:41:29.310 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.310 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:29.310 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086937+0000", 2024-03-27T21:41:29.310 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2125000000000002e-05 2024-03-27T21:41:29.310 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.310 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:29.310 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086957+0000", 2024-03-27T21:41:29.310 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0157999999999999e-05 2024-03-27T21:41:29.311 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.311 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:29.311 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.311 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.311 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.311 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:29.311 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:29.311 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 109.208033482, 2024-03-27T21:41:29.311 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007405661, 2024-03-27T21:41:29.311 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:29.311 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.311 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:29.311 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:29.311 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:29.311 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 114 2024-03-27T21:41:29.311 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.312 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:29.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:29.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:29.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:29.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:29.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:29.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:29.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-03-27T21:41:29.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:29.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080353+0000", 2024-03-27T21:41:29.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.441e-06 2024-03-27T21:41:29.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:29.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080356+0000", 2024-03-27T21:41:29.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0769999999999999e-06 2024-03-27T21:41:29.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080358+0000", 2024-03-27T21:41:29.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3369999999999998e-06 2024-03-27T21:41:29.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.314 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.314 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:29.314 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085118+0000", 2024-03-27T21:41:29.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047603949999999997 2024-03-27T21:41:29.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.314 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.314 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:29.314 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085210+0000", 2024-03-27T21:41:29.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1983000000000003e-05 2024-03-27T21:41:29.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.314 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.314 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:29.314 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085332+0000", 2024-03-27T21:41:29.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121803 2024-03-27T21:41:29.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.315 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:29.315 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086292+0000", 2024-03-27T21:41:29.315 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095961400000000002 2024-03-27T21:41:29.315 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.315 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:29.315 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087710+0000", 2024-03-27T21:41:29.315 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014181459999999999 2024-03-27T21:41:29.315 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.315 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:29.315 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087736+0000", 2024-03-27T21:41:29.315 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6475e-05 2024-03-27T21:41:29.315 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.315 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:29.316 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087756+0000", 2024-03-27T21:41:29.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9626000000000001e-05 2024-03-27T21:41:29.316 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.316 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:29.316 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.316 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.316 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.316 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:29.316 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:29.316 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 109.208011047, 2024-03-27T21:41:29.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075011720000000004, 2024-03-27T21:41:29.316 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:29.316 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.316 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:29.316 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:29.317 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:29.317 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115 2024-03-27T21:41:29.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.317 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:29.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:29.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:29.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:29.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:29.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:29.318 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:29.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:29.318 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080458+0000", 2024-03-27T21:41:29.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5228000000000005e-05 2024-03-27T21:41:29.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:29.318 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080465+0000", 2024-03-27T21:41:29.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.238e-06 2024-03-27T21:41:29.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.319 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.319 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080468+0000", 2024-03-27T21:41:29.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5109999999999999e-06 2024-03-27T21:41:29.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.319 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.319 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:29.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086326+0000", 2024-03-27T21:41:29.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058579169999999998 2024-03-27T21:41:29.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.319 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.319 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:29.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086382+0000", 2024-03-27T21:41:29.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5309999999999997e-05 2024-03-27T21:41:29.320 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.320 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.320 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:29.320 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086457+0000", 2024-03-27T21:41:29.320 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5413000000000005e-05 2024-03-27T21:41:29.320 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.320 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.320 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:29.320 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087762+0000", 2024-03-27T21:41:29.320 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013052300000000001 2024-03-27T21:41:29.320 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.320 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.320 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:29.320 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087798+0000", 2024-03-27T21:41:29.320 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5614999999999997e-05 2024-03-27T21:41:29.320 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.321 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:29.321 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087840+0000", 2024-03-27T21:41:29.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2196000000000001e-05 2024-03-27T21:41:29.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.321 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:29.321 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087874+0000", 2024-03-27T21:41:29.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3785999999999998e-05 2024-03-27T21:41:29.321 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.321 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:29.321 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.321 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:29.322 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:29.322 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 109.20791096400001, 2024-03-27T21:41:29.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075050680000000002, 2024-03-27T21:41:29.322 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:29.322 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.322 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:29.322 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:29.322 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:29.322 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 116 2024-03-27T21:41:29.322 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.322 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:29.322 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:29.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:29.323 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.323 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:29.323 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:29.323 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.323 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:29.323 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080472+0000", 2024-03-27T21:41:29.323 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.323 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:29.324 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080475+0000", 2024-03-27T21:41:29.324 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.712e-06 2024-03-27T21:41:29.324 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.324 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.324 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:29.324 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080478+0000", 2024-03-27T21:41:29.324 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1070000000000001e-06 2024-03-27T21:41:29.324 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.324 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.324 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.324 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080481+0000", 2024-03-27T21:41:29.324 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4269999999999998e-06 2024-03-27T21:41:29.324 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.324 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.324 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:29.324 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086992+0000", 2024-03-27T21:41:29.325 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065114099999999996 2024-03-27T21:41:29.325 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.325 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.325 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:29.325 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087064+0000", 2024-03-27T21:41:29.325 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1927000000000004e-05 2024-03-27T21:41:29.325 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.325 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.325 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:29.325 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087188+0000", 2024-03-27T21:41:29.325 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123585 2024-03-27T21:41:29.325 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.325 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.325 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:29.325 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087766+0000", 2024-03-27T21:41:29.326 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057814300000000004 2024-03-27T21:41:29.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.326 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:29.326 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087934+0000", 2024-03-27T21:41:29.326 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000167894 2024-03-27T21:41:29.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.326 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:29.326 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087958+0000", 2024-03-27T21:41:29.326 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4663000000000001e-05 2024-03-27T21:41:29.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.326 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:29.326 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087978+0000", 2024-03-27T21:41:29.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9636e-05 2024-03-27T21:41:29.327 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.327 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:29.327 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.327 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.327 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:29.327 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:29.327 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 109.207878842, 2024-03-27T21:41:29.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076850679999999998, 2024-03-27T21:41:29.327 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:29.327 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.327 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:29.327 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:29.328 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:29.328 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 117 2024-03-27T21:41:29.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.328 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:29.328 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.328 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:29.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:29.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.328 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.328 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:29.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:29.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.329 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.329 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:29.329 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:29.329 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.329 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.329 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.329 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:29.329 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080508+0000", 2024-03-27T21:41:29.329 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.298e-06 2024-03-27T21:41:29.329 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.329 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.329 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:29.329 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080515+0000", 2024-03-27T21:41:29.329 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0790000000000003e-06 2024-03-27T21:41:29.330 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.330 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.330 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.330 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080518+0000", 2024-03-27T21:41:29.330 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8509999999999999e-06 2024-03-27T21:41:29.330 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.330 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.330 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:29.330 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087444+0000", 2024-03-27T21:41:29.330 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0069259120000000002 2024-03-27T21:41:29.330 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.330 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.330 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:29.330 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087491+0000", 2024-03-27T21:41:29.331 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7043999999999999e-05 2024-03-27T21:41:29.331 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.331 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.331 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:29.331 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087564+0000", 2024-03-27T21:41:29.331 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3603999999999995e-05 2024-03-27T21:41:29.331 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.331 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.331 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:29.331 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087902+0000", 2024-03-27T21:41:29.331 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033734100000000001 2024-03-27T21:41:29.331 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.331 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.331 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:29.331 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088117+0000", 2024-03-27T21:41:29.331 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000215164 2024-03-27T21:41:29.332 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.332 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.332 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:29.332 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088160+0000", 2024-03-27T21:41:29.332 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2933999999999999e-05 2024-03-27T21:41:29.332 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.332 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.332 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:29.332 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088190+0000", 2024-03-27T21:41:29.332 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0139999999999999e-05 2024-03-27T21:41:29.332 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.332 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:29.332 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.332 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.333 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.333 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:29.333 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:29.333 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 108.79159845300001, 2024-03-27T21:41:29.333 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052465439999999997, 2024-03-27T21:41:29.333 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:29.333 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.333 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:29.333 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:29.333 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:29.333 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 133 2024-03-27T21:41:29.333 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.333 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:29.333 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.333 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:29.334 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:29.334 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.334 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.334 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.334 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:29.334 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:29.334 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.334 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.334 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.334 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:29.334 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:29.334 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.334 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.334 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.335 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:29.335 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496788+0000", 2024-03-27T21:41:29.335 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-03-27T21:41:29.335 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.335 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.335 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:29.335 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496793+0000", 2024-03-27T21:41:29.335 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4790000000000004e-06 2024-03-27T21:41:29.335 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.335 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.335 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.335 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496796+0000", 2024-03-27T21:41:29.335 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1e-06 2024-03-27T21:41:29.335 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.336 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.336 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:29.336 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500447+0000", 2024-03-27T21:41:29.336 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036504459999999999 2024-03-27T21:41:29.336 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.336 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.336 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:29.336 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500492+0000", 2024-03-27T21:41:29.336 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5765e-05 2024-03-27T21:41:29.336 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.336 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.336 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:29.336 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500562+0000", 2024-03-27T21:41:29.336 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9206000000000001e-05 2024-03-27T21:41:29.336 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.336 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.337 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:29.337 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501959+0000", 2024-03-27T21:41:29.337 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001397187 2024-03-27T21:41:29.337 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.337 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.337 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:29.337 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501984+0000", 2024-03-27T21:41:29.337 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.552e-05 2024-03-27T21:41:29.337 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.337 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.337 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:29.337 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502010+0000", 2024-03-27T21:41:29.337 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5232000000000002e-05 2024-03-27T21:41:29.337 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.337 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.337 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:29.338 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502032+0000", 2024-03-27T21:41:29.338 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2337999999999999e-05 2024-03-27T21:41:29.338 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.338 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:29.338 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.338 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.338 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.338 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:29.338 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:29.338 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 108.79153979500001, 2024-03-27T21:41:29.338 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052892119999999997, 2024-03-27T21:41:29.338 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:29.338 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.338 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:29.339 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:29.339 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:29.339 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 134 2024-03-27T21:41:29.339 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.339 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:29.339 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.339 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:29.339 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:29.339 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.339 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.339 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.339 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:29.339 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:29.339 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.339 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.340 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.340 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:29.340 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:29.340 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.340 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.340 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.340 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:29.340 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496929+0000", 2024-03-27T21:41:29.340 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5098000000000004e-05 2024-03-27T21:41:29.340 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.340 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.340 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:29.340 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496937+0000", 2024-03-27T21:41:29.340 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9559999999999997e-06 2024-03-27T21:41:29.340 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.341 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.341 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.341 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496942+0000", 2024-03-27T21:41:29.341 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6350000000000003e-06 2024-03-27T21:41:29.341 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.341 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.341 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:29.341 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500749+0000", 2024-03-27T21:41:29.341 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038068529999999998 2024-03-27T21:41:29.341 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.341 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.341 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:29.341 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500792+0000", 2024-03-27T21:41:29.341 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2823e-05 2024-03-27T21:41:29.342 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.342 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.342 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:29.342 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500864+0000", 2024-03-27T21:41:29.342 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1715e-05 2024-03-27T21:41:29.342 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.342 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.342 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:29.342 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502068+0000", 2024-03-27T21:41:29.342 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012044860000000001 2024-03-27T21:41:29.342 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.342 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.342 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:29.342 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502073+0000", 2024-03-27T21:41:29.342 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5700000000000003e-06 2024-03-27T21:41:29.343 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.343 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.343 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:29.343 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502098+0000", 2024-03-27T21:41:29.343 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5120999999999999e-05 2024-03-27T21:41:29.343 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.343 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.343 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:29.343 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502133+0000", 2024-03-27T21:41:29.343 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5306999999999999e-05 2024-03-27T21:41:29.343 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.343 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:29.343 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.343 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.344 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.344 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:29.344 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:29.344 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 108.79143448799999, 2024-03-27T21:41:29.344 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062529040000000001, 2024-03-27T21:41:29.344 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:29.344 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.344 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:29.344 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:29.344 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:29.344 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135 2024-03-27T21:41:29.344 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.344 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:29.344 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.344 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:29.345 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:29.345 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.345 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.345 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.345 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:29.345 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:29.345 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.345 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.345 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.345 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:29.345 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:29.345 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.345 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.345 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.345 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:29.345 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496953+0000", 2024-03-27T21:41:29.346 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0709999999999996e-06 2024-03-27T21:41:29.346 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.346 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.346 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:29.346 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496961+0000", 2024-03-27T21:41:29.346 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1300000000000001e-06 2024-03-27T21:41:29.346 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.346 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.346 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.346 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496968+0000", 2024-03-27T21:41:29.346 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0999999999999998e-06 2024-03-27T21:41:29.346 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.346 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.346 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:29.346 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501244+0000", 2024-03-27T21:41:29.347 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042756310000000002 2024-03-27T21:41:29.347 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.347 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.347 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:29.347 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501377+0000", 2024-03-27T21:41:29.347 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013348799999999999 2024-03-27T21:41:29.347 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.347 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.347 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:29.347 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501455+0000", 2024-03-27T21:41:29.347 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7410000000000006e-05 2024-03-27T21:41:29.347 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.347 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.347 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:29.347 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502138+0000", 2024-03-27T21:41:29.348 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068386599999999999 2024-03-27T21:41:29.348 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.348 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.348 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:29.348 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503152+0000", 2024-03-27T21:41:29.348 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010139680000000001 2024-03-27T21:41:29.348 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.348 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.348 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:29.348 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503180+0000", 2024-03-27T21:41:29.348 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7426000000000001e-05 2024-03-27T21:41:29.348 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.348 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.348 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:29.349 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503202+0000", 2024-03-27T21:41:29.349 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2463000000000002e-05 2024-03-27T21:41:29.349 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.349 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:29.349 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.349 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.349 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.349 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:29.349 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:29.349 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 108.79141035000001, 2024-03-27T21:41:29.349 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066253680000000004, 2024-03-27T21:41:29.349 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:29.349 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.349 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:29.349 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:29.350 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:29.350 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 136 2024-03-27T21:41:29.350 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.350 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:29.350 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.350 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:29.350 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:29.350 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.350 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.350 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.350 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:29.350 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:29.350 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.350 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.351 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.351 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:29.351 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:29.351 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.351 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.351 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.351 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:29.351 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496977+0000", 2024-03-27T21:41:29.351 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5049999999999999e-06 2024-03-27T21:41:29.351 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.351 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.351 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:29.351 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496984+0000", 2024-03-27T21:41:29.351 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1520000000000003e-06 2024-03-27T21:41:29.352 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.352 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.352 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.352 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496990+0000", 2024-03-27T21:41:29.352 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-03-27T21:41:29.352 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.352 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.352 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:29.352 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501669+0000", 2024-03-27T21:41:29.352 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046793859999999998 2024-03-27T21:41:29.352 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.352 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.352 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:29.352 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501714+0000", 2024-03-27T21:41:29.352 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5154999999999997e-05 2024-03-27T21:41:29.353 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.353 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.353 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:29.353 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501783+0000", 2024-03-27T21:41:29.353 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8454999999999994e-05 2024-03-27T21:41:29.353 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.353 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.353 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:29.353 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503226+0000", 2024-03-27T21:41:29.353 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001443441 2024-03-27T21:41:29.353 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.353 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.353 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:29.353 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503540+0000", 2024-03-27T21:41:29.353 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031347100000000001 2024-03-27T21:41:29.354 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.354 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.354 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:29.354 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503563+0000", 2024-03-27T21:41:29.354 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3223000000000001e-05 2024-03-27T21:41:29.354 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.354 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.354 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:29.354 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503599+0000", 2024-03-27T21:41:29.354 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6105000000000001e-05 2024-03-27T21:41:29.354 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.354 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:29.354 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.355 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.355 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.355 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:29.355 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:29.355 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 108.79138754, 2024-03-27T21:41:29.355 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074261320000000002, 2024-03-27T21:41:29.355 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:29.355 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.355 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:29.355 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:29.355 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:29.355 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 137 2024-03-27T21:41:29.355 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.355 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:29.355 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:29.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:29.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:29.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:29.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:29.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:29.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:29.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497077+0000", 2024-03-27T21:41:29.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1599000000000003e-05 2024-03-27T21:41:29.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:29.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497085+0000", 2024-03-27T21:41:29.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6109999999999996e-06 2024-03-27T21:41:29.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497092+0000", 2024-03-27T21:41:29.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.742e-06 2024-03-27T21:41:29.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.358 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:29.358 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502159+0000", 2024-03-27T21:41:29.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050666870000000003 2024-03-27T21:41:29.358 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.358 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:29.358 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502200+0000", 2024-03-27T21:41:29.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1919999999999998e-05 2024-03-27T21:41:29.358 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.358 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:29.358 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502280+0000", 2024-03-27T21:41:29.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9417000000000002e-05 2024-03-27T21:41:29.358 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.359 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:29.359 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503605+0000", 2024-03-27T21:41:29.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001325641 2024-03-27T21:41:29.359 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.359 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:29.359 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504361+0000", 2024-03-27T21:41:29.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075555399999999997 2024-03-27T21:41:29.359 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.359 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:29.359 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504386+0000", 2024-03-27T21:41:29.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5369999999999999e-05 2024-03-27T21:41:29.359 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.360 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:29.360 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504422+0000", 2024-03-27T21:41:29.360 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6019000000000002e-05 2024-03-27T21:41:29.360 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.360 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:29.360 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.360 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.360 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.360 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:29.360 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:29.360 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 108.79128374299999, 2024-03-27T21:41:29.360 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074059750000000004, 2024-03-27T21:41:29.360 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:29.360 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.361 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:29.361 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:29.361 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:29.361 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-03-27T21:41:29.361 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.361 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:29.361 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.361 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:29.361 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:29.361 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.361 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.361 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.361 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:29.361 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:29.361 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.361 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.362 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.362 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:29.362 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:29.362 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.362 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.362 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.362 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:29.362 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497103+0000", 2024-03-27T21:41:29.362 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.856e-06 2024-03-27T21:41:29.362 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.362 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.362 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:29.362 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497121+0000", 2024-03-27T21:41:29.362 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7190999999999999e-05 2024-03-27T21:41:29.363 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.363 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.363 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.363 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497126+0000", 2024-03-27T21:41:29.363 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3770000000000004e-06 2024-03-27T21:41:29.363 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.363 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.363 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:29.363 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502833+0000", 2024-03-27T21:41:29.363 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057073510000000003 2024-03-27T21:41:29.363 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.363 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.363 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:29.363 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502881+0000", 2024-03-27T21:41:29.363 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7961999999999999e-05 2024-03-27T21:41:29.363 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.364 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.364 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:29.364 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502949+0000", 2024-03-27T21:41:29.364 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7957e-05 2024-03-27T21:41:29.364 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.364 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.364 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:29.364 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504430+0000", 2024-03-27T21:41:29.364 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001480763 2024-03-27T21:41:29.364 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.364 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.364 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:29.364 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504459+0000", 2024-03-27T21:41:29.364 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8983999999999999e-05 2024-03-27T21:41:29.364 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.365 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.365 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:29.365 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504486+0000", 2024-03-27T21:41:29.365 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6883999999999999e-05 2024-03-27T21:41:29.365 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.365 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.365 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:29.365 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504506+0000", 2024-03-27T21:41:29.365 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0214e-05 2024-03-27T21:41:29.365 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.365 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:29.365 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.365 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.365 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.366 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:29.366 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:29.366 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 108.791251909, 2024-03-27T21:41:29.366 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074754859999999999, 2024-03-27T21:41:29.366 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:29.366 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.366 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:29.366 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:29.366 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:29.366 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-03-27T21:41:29.366 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.366 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:29.366 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.366 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:29.367 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:29.367 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.367 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.367 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.367 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:29.367 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:29.367 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.367 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.367 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.367 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:29.367 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:29.367 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.367 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.367 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.367 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:29.368 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497137+0000", 2024-03-27T21:41:29.368 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0730000000000004e-06 2024-03-27T21:41:29.368 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.368 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.368 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:29.368 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497143+0000", 2024-03-27T21:41:29.368 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3759999999999996e-06 2024-03-27T21:41:29.368 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.368 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.368 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.368 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497150+0000", 2024-03-27T21:41:29.368 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.002e-06 2024-03-27T21:41:29.368 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.368 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.368 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:29.369 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503249+0000", 2024-03-27T21:41:29.369 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060989750000000004 2024-03-27T21:41:29.369 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.369 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.369 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:29.369 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503291+0000", 2024-03-27T21:41:29.369 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1563999999999997e-05 2024-03-27T21:41:29.369 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.369 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.369 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:29.369 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503376+0000", 2024-03-27T21:41:29.369 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5752000000000002e-05 2024-03-27T21:41:29.369 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.369 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.369 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:29.370 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504545+0000", 2024-03-27T21:41:29.370 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169172 2024-03-27T21:41:29.370 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.370 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.370 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:29.370 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504550+0000", 2024-03-27T21:41:29.370 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7419999999999997e-06 2024-03-27T21:41:29.370 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.370 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.370 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:29.370 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504574+0000", 2024-03-27T21:41:29.370 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4124000000000001e-05 2024-03-27T21:41:29.370 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.370 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.370 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:29.371 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504607+0000", 2024-03-27T21:41:29.371 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3118999999999999e-05 2024-03-27T21:41:29.371 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.371 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:29.371 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.371 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.371 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.371 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:29.371 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:29.371 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 108.79122824300001, 2024-03-27T21:41:29.371 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078150589999999992, 2024-03-27T21:41:29.371 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:29.371 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.371 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:29.371 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:29.372 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:29.372 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-03-27T21:41:29.372 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.372 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:29.372 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.372 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:29.372 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:29.372 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.372 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.372 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.372 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:29.372 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:29.372 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.372 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.372 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.372 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:29.372 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497155+0000", 2024-03-27T21:41:29.373 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.373 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.373 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.373 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:29.373 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497230+0000", 2024-03-27T21:41:29.373 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4789000000000005e-05 2024-03-27T21:41:29.373 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.373 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.373 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:29.373 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497241+0000", 2024-03-27T21:41:29.373 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1442e-05 2024-03-27T21:41:29.373 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.373 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.373 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.373 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497255+0000", 2024-03-27T21:41:29.373 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3087e-05 2024-03-27T21:41:29.373 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.374 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.374 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:29.374 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503630+0000", 2024-03-27T21:41:29.374 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063755269999999998 2024-03-27T21:41:29.374 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.374 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.374 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:29.374 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503673+0000", 2024-03-27T21:41:29.374 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2616e-05 2024-03-27T21:41:29.374 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.374 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.374 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:29.374 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503738+0000", 2024-03-27T21:41:29.374 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5114e-05 2024-03-27T21:41:29.375 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.375 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.375 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:29.375 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504907+0000", 2024-03-27T21:41:29.375 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169432 2024-03-27T21:41:29.375 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.375 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.375 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:29.375 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504926+0000", 2024-03-27T21:41:29.375 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9086999999999999e-05 2024-03-27T21:41:29.375 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.375 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.375 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:29.375 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504950+0000", 2024-03-27T21:41:29.375 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3598999999999999e-05 2024-03-27T21:41:29.375 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.376 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.376 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:29.376 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504971+0000", 2024-03-27T21:41:29.376 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0772999999999999e-05 2024-03-27T21:41:29.376 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.376 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:29.376 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.376 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.376 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.376 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:29.376 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:29.376 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 108.791121203, 2024-03-27T21:41:29.376 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082962590000000003, 2024-03-27T21:41:29.376 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:29.376 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.377 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:29.377 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:29.377 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:29.377 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-03-27T21:41:29.377 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.377 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:29.377 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.377 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:29.377 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:29.377 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.377 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.377 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.377 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:29.377 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:29.377 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.378 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.378 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.378 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:29.378 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497262+0000", 2024-03-27T21:41:29.378 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.378 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.378 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.378 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:29.378 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497267+0000", 2024-03-27T21:41:29.378 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.955e-06 2024-03-27T21:41:29.378 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.378 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.378 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:29.378 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497275+0000", 2024-03-27T21:41:29.378 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.824e-06 2024-03-27T21:41:29.379 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.379 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.379 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.379 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497280+0000", 2024-03-27T21:41:29.379 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8230000000000002e-06 2024-03-27T21:41:29.379 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.379 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.379 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:29.379 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504628+0000", 2024-03-27T21:41:29.379 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007348561 2024-03-27T21:41:29.379 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.379 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.379 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:29.379 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504673+0000", 2024-03-27T21:41:29.379 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4313000000000001e-05 2024-03-27T21:41:29.380 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.380 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.380 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:29.380 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504745+0000", 2024-03-27T21:41:29.380 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2303000000000002e-05 2024-03-27T21:41:29.380 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.380 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.380 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:29.380 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505358+0000", 2024-03-27T21:41:29.380 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061281300000000003 2024-03-27T21:41:29.380 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.380 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.380 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:29.380 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505503+0000", 2024-03-27T21:41:29.380 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000145134 2024-03-27T21:41:29.380 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.381 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.381 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:29.381 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505526+0000", 2024-03-27T21:41:29.381 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3312e-05 2024-03-27T21:41:29.381 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.381 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.381 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:29.381 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505559+0000", 2024-03-27T21:41:29.381 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2821999999999998e-05 2024-03-27T21:41:29.381 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.381 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:29.381 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.381 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.381 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.381 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:29.382 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:29.382 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 108.181898964, 2024-03-27T21:41:29.382 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053753339999999998, 2024-03-27T21:41:29.382 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:29.382 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.382 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:29.382 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:29.382 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:29.382 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159 2024-03-27T21:41:29.382 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.382 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:29.382 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.382 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:29.382 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:29.382 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.382 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.383 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:29.383 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:29.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:29.383 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.383 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:29.383 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:29.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.383 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.383 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:29.383 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106488+0000", 2024-03-27T21:41:29.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9830000000000001e-06 2024-03-27T21:41:29.383 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.384 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:29.384 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106491+0000", 2024-03-27T21:41:29.384 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3459999999999998e-06 2024-03-27T21:41:29.384 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.384 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.384 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.384 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106493+0000", 2024-03-27T21:41:29.384 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.509e-06 2024-03-27T21:41:29.384 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.384 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.384 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:29.384 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110636+0000", 2024-03-27T21:41:29.384 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041427260000000002 2024-03-27T21:41:29.384 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.384 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.384 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:29.385 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110683+0000", 2024-03-27T21:41:29.385 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6557000000000002e-05 2024-03-27T21:41:29.385 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.385 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.385 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:29.385 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110755+0000", 2024-03-27T21:41:29.385 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2119000000000005e-05 2024-03-27T21:41:29.385 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.385 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.385 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:29.385 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111415+0000", 2024-03-27T21:41:29.385 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066065199999999996 2024-03-27T21:41:29.385 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.385 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.385 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:29.386 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111793+0000", 2024-03-27T21:41:29.386 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037732100000000001 2024-03-27T21:41:29.386 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.386 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.386 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:29.386 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111829+0000", 2024-03-27T21:41:29.386 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418999999999998e-05 2024-03-27T21:41:29.386 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:29.386 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:29.386 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:29.386 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111860+0000", 2024-03-27T21:41:29.386 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1003000000000001e-05 2024-03-27T21:41:29.386 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.386 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:29.386 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.387 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:29.387 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:29.387 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:29.388 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:29.389 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:29.389 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:29.389 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:29.389 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:29.402+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:29.389 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-27T21:41:29.390 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:41:29.489 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:29.502+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:29.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.602+0000 7f4c3a38b640 10 monclient: get_monmap_and_config 2024-03-27T21:41:29.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.602+0000 7f4c3a38b640 10 monclient: build_initial_monmap 2024-03-27T21:41:29.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.602+0000 7f4c3a38b640 10 monclient: monmap: 2024-03-27T21:41:29.588 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:29.589 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:29.589 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:29.603283+0000 2024-03-27T21:41:29.589 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:29.603283+0000 2024-03-27T21:41:29.589 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:29.589 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:29.589 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:29.589 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:29.589 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:29.589 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:29.589 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:29.589 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:29.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.603+0000 7f4c3a38b640 1 Processor -- start 2024-03-27T21:41:29.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.603+0000 7f4c3a38b640 1 -- start start 2024-03-27T21:41:29.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.603+0000 7f4c3a38b640 10 monclient: init 2024-03-27T21:41:29.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.604+0000 7f4c3a38b640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:29.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.604+0000 7f4c3a38b640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-03-27T21:41:29.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.604+0000 7f4c3a38b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4c340b40b0 0x7f4c340a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:29.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.604+0000 7f4c3a38b640 10 monclient(hunting): picked mon.noname-c con 0x7f4c340b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:29.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.604+0000 7f4c3a38b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4c3413b020 0x7f4c3413b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:29.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.604+0000 7f4c3a38b640 10 monclient(hunting): picked mon.noname-a con 0x7f4c3413b020 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:29.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.604+0000 7f4c3a38b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c341416f0 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:29.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.604+0000 7f4c3a38b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c340b3830 con 0x7f4c341416f0 2024-03-27T21:41:29.592 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:29.604+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:29.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.604+0000 7f4c3a38b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:29.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.604+0000 7f4c3a38b640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c340b4490 con 0x7f4c3413b020 2024-03-27T21:41:29.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.604+0000 7f4c337fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4c3413b020 0x7f4c3413b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:29.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.604+0000 7f4c38928640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f4c341416f0 0x7f4c34141b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.60:44406/0 (socket says 172.21.15.60:44406) 2024-03-27T21:41:29.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.604+0000 7f4c38928640 1 -- 172.21.15.60:0/1919237966 learned_addr learned my addr 172.21.15.60:0/1919237966 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:29.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.604+0000 7f4c33fff640 1 --2- 172.21.15.60:0/1919237966 >> v2:172.21.15.191:3300/0 conn(0x7f4c340b40b0 0x7f4c340a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:29.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c3a38b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:29.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c3a38b640 1 -- 172.21.15.60:0/1919237966 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c340aa520 con 0x7f4c340b40b0 2024-03-27T21:41:29.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c3a38b640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:29.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c3a38b640 10 monclient(hunting): authenticate will time out at 930.825970s 2024-03-27T21:41:29.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c337fe640 10 monclient(hunting): get_auth_request con 0x7f4c3413b020 auth_method 0 2024-03-27T21:41:29.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:29.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c337fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:29.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:29.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c33fff640 10 monclient(hunting): get_auth_request con 0x7f4c340b40b0 auth_method 0 2024-03-27T21:41:29.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:29.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c33fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:29.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:29.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:29.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:29.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:29.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:29.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:29.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:29.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c32ffd640 1 -- 172.21.15.60:0/1919237966 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485883725 0 0) 0x7f4c340b3830 con 0x7f4c341416f0 2024-03-27T21:41:29.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c32ffd640 10 monclient(hunting): _negotiate 2024-03-27T21:41:29.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:29.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:29.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c32ffd640 10 monclient(hunting): my global_id is 4954 2024-03-27T21:41:29.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c32ffd640 1 -- 172.21.15.60:0/1919237966 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c1c003520 con 0x7f4c341416f0 2024-03-27T21:41:29.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c33fff640 10 monclient(hunting): handle_auth_done global_id 4957 payload 274 2024-03-27T21:41:29.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c33fff640 1 -- 172.21.15.60:0/1919237966 >> v2:172.21.15.60:3300/0 conn(0x7f4c3413b020 msgr2=0x7f4c3413b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:29.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c32ffd640 1 -- 172.21.15.60:0/1919237966 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 992376174 0 0) 0x7f4c1c003520 con 0x7f4c341416f0 2024-03-27T21:41:29.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c33fff640 1 --2- 172.21.15.60:0/1919237966 >> v2:172.21.15.60:3300/0 conn(0x7f4c3413b020 0x7f4c3413b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:29.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c33fff640 1 -- 172.21.15.60:0/1919237966 >> v1:172.21.15.191:6789/0 conn(0x7f4c341416f0 legacy=0x7f4c34141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:29.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c33fff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:29.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c33fff640 1 monclient: found mon.noname-c 2024-03-27T21:41:29.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:29.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c33fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:41:29.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.605+0000 7f4c33fff640 1 -- 172.21.15.60:0/1919237966 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c340a9470 con 0x7f4c340b40b0 2024-03-27T21:41:29.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.606+0000 7f4c337fe640 1 --2- 172.21.15.60:0/1919237966 >> v2:172.21.15.60:3300/0 conn(0x7f4c3413b020 0x7f4c3413b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:29.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.606+0000 7f4c32ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:41:29.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.606+0000 7f4c33fff640 1 --2- 172.21.15.60:0/1919237966 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c340b40b0 0x7f4c340a8b90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4c340b4810 tx=0x7f4c2402b7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3690bea3d7674ca server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:29.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.606+0000 7f4c32ffd640 1 -- 172.21.15.60:0/1919237966 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c0019f0 con 0x7f4c340b40b0 2024-03-27T21:41:29.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.606+0000 7f4c32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:29.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.606+0000 7f4c32ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:29.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.606+0000 7f4c32ffd640 10 monclient: dump: 2024-03-27T21:41:29.601 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:29.601 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:29.601 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:29.601 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:29.601 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:29.601 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:29.601 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:29.601 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:29.601 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:29.601 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:29.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.606+0000 7f4c32ffd640 10 monclient: _finish_auth 0 2024-03-27T21:41:29.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.606+0000 7f4c32ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:41:29.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.606+0000 7f4c32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:29.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.606+0000 7f4c32ffd640 1 -- 172.21.15.60:0/1919237966 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2402e050 con 0x7f4c340b40b0 2024-03-27T21:41:29.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.606+0000 7f4c32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:29.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.606+0000 7f4c32ffd640 1 -- 172.21.15.60:0/1919237966 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24030710 con 0x7f4c340b40b0 2024-03-27T21:41:29.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.606+0000 7f4c32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:29.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.606+0000 7f4c32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:29.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.606+0000 7f4c32ffd640 10 monclient: dump: 2024-03-27T21:41:29.602 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:29.602 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:29.602 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:29.602 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:29.602 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:29.602 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:29.602 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:29.602 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:29.603 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:29.603 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:29.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.606+0000 7f4c3a38b640 5 monclient: authenticate success, global_id 4957 2024-03-27T21:41:29.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.606+0000 7f4c3a38b640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:29.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.607+0000 7f4c3a38b640 4 set_mon_vals no callback set 2024-03-27T21:41:29.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.607+0000 7f4c3a38b640 10 monclient: shutdown 2024-03-27T21:41:29.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.607+0000 7f4c3a38b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:29.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.607+0000 7f4c3a38b640 1 -- 172.21.15.60:0/1919237966 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c340b40b0 msgr2=0x7f4c340a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:29.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.607+0000 7f4c3a38b640 1 --2- 172.21.15.60:0/1919237966 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c340b40b0 0x7f4c340a8b90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4c340b4810 tx=0x7f4c2402b7c0 comp rx=0 tx=0).stop 2024-03-27T21:41:29.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.607+0000 7f4c3a38b640 1 -- 172.21.15.60:0/1919237966 shutdown_connections 2024-03-27T21:41:29.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.607+0000 7f4c3a38b640 1 --2- 172.21.15.60:0/1919237966 >> v2:172.21.15.60:3300/0 conn(0x7f4c3413b020 0x7f4c3413b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:29.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.607+0000 7f4c3a38b640 1 --2- 172.21.15.60:0/1919237966 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c340b40b0 0x7f4c340a8b90 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:29.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.607+0000 7f4c3a38b640 1 -- 172.21.15.60:0/1919237966 >> 172.21.15.60:0/1919237966 conn(0x7f4c340608e0 msgr2=0x7f4c34060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:29.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.607+0000 7f4c3a38b640 1 -- 172.21.15.60:0/1919237966 shutdown_connections 2024-03-27T21:41:29.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.607+0000 7f4c3a38b640 1 -- 172.21.15.60:0/1919237966 wait complete. 2024-03-27T21:41:29.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.608+0000 7f4c3a38b640 10 monclient: build_initial_monmap 2024-03-27T21:41:29.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.608+0000 7f4c3a38b640 10 monclient: monmap: 2024-03-27T21:41:29.604 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:29.604 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:29.604 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:29.604 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:29.604 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:29.604 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:29.604 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:29.604 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:29.604 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:29.604 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:29.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.609+0000 7f4c3a38b640 1 librados: starting msgr at 2024-03-27T21:41:29.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.609+0000 7f4c3a38b640 1 librados: starting objecter 2024-03-27T21:41:29.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.609+0000 7f4c3a38b640 1 Processor -- start 2024-03-27T21:41:29.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.609+0000 7f4c3a38b640 1 -- start start 2024-03-27T21:41:29.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.609+0000 7f4c3a38b640 1 librados: setting wanted keys 2024-03-27T21:41:29.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.609+0000 7f4c3a38b640 1 librados: calling monclient init 2024-03-27T21:41:29.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.609+0000 7f4c3a38b640 10 monclient: init 2024-03-27T21:41:29.605 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:29.605 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:29.605 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:29.605 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:29.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.610+0000 7f4c3a38b640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:29.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.610+0000 7f4c3a38b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:41:29.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.610+0000 7f4c3a38b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c340b40b0 0x7f4c340aac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:29.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.610+0000 7f4c33fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c340b40b0 0x7f4c340aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:29.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.610+0000 7f4c33fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c340b40b0 0x7f4c340aac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36534/0 (socket says 172.21.15.60:36534) 2024-03-27T21:41:29.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.610+0000 7f4c33fff640 1 -- 172.21.15.60:0/766032506 learned_addr learned my addr 172.21.15.60:0/766032506 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:29.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.610+0000 7f4c3a38b640 10 monclient(hunting): picked mon.noname-a con 0x7f4c340b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:29.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.610+0000 7f4c3a38b640 1 --2- 172.21.15.60:0/766032506 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4c3413b020 0x7f4c341d68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:29.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c337fe640 1 --2- 172.21.15.60:0/766032506 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4c3413b020 0x7f4c341d68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:29.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c3a38b640 10 monclient(hunting): picked mon.noname-c con 0x7f4c3413b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:29.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c3a38b640 1 --2- 172.21.15.60:0/766032506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c341416f0 0x7f4c341d6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:29.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c3a38b640 10 monclient(hunting): picked mon.noname-b con 0x7f4c341416f0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:29.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c3a38b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:29.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c3a38b640 1 -- 172.21.15.60:0/766032506 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3413ba60 con 0x7f4c340b40b0 2024-03-27T21:41:29.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c3a38b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:29.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c3a38b640 1 -- 172.21.15.60:0/766032506 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3413d750 con 0x7f4c341416f0 2024-03-27T21:41:29.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c3a38b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:29.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c3a38b640 1 -- 172.21.15.60:0/766032506 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c341d7330 con 0x7f4c3413b020 2024-03-27T21:41:29.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c3a38b640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:29.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c3a38b640 10 monclient(hunting): authenticate will time out at 930.832125s 2024-03-27T21:41:29.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c33fff640 10 monclient(hunting): get_auth_request con 0x7f4c340b40b0 auth_method 0 2024-03-27T21:41:29.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:29.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c33fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:29.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:29.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c337fe640 10 monclient(hunting): get_auth_request con 0x7f4c3413b020 auth_method 0 2024-03-27T21:41:29.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:29.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c337fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:29.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:29.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:29.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:29.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:29.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c38928640 1 --2- 172.21.15.60:0/766032506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c341416f0 0x7f4c341d6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:29.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c38928640 10 monclient(hunting): get_auth_request con 0x7f4c341416f0 auth_method 0 2024-03-27T21:41:29.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c38928640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:29.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c38928640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:29.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c38928640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:29.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c38928640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:29.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c38928640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:29.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c38928640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:29.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:29.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:29.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:29.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c38928640 10 monclient(hunting): handle_auth_done global_id 4960 payload 824 2024-03-27T21:41:29.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c38928640 1 -- 172.21.15.60:0/766032506 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4c3413b020 msgr2=0x7f4c341d68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:29.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c38928640 1 --2- 172.21.15.60:0/766032506 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4c3413b020 0x7f4c341d68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:29.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.611+0000 7f4c38928640 1 -- 172.21.15.60:0/766032506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c340b40b0 msgr2=0x7f4c340aac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:29.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c38928640 1 --2- 172.21.15.60:0/766032506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c340b40b0 0x7f4c340aac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:29.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c38928640 10 monclient: _finish_hunting 0 2024-03-27T21:41:29.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c38928640 1 monclient: found mon.noname-b 2024-03-27T21:41:29.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c38928640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:29.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c38928640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:41:29.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c38928640 1 -- 172.21.15.60:0/766032506 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c28009400 con 0x7f4c341416f0 2024-03-27T21:41:29.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c337fe640 1 --2- 172.21.15.60:0/766032506 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4c3413b020 0x7f4c341d68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:29.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c33fff640 1 --2- 172.21.15.60:0/766032506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c340b40b0 0x7f4c340aac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:29.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c38928640 1 --2- 172.21.15.60:0/766032506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c341416f0 0x7f4c341d6df0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c001700 tx=0x7f4c2c00a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:29.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c30ff9640 10 client.?.objecter ms_handle_connect 0x7f4c341416f0 2024-03-27T21:41:29.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c30ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:29.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c30ff9640 1 -- 172.21.15.60:0/766032506 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c00ad40 con 0x7f4c341416f0 2024-03-27T21:41:29.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:29.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c30ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:29.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c30ff9640 10 monclient: dump: 2024-03-27T21:41:29.618 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:29.618 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:29.618 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:29.619 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:29.619 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:29.619 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:29.619 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:29.619 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:29.619 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:29.619 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:29.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c30ff9640 10 monclient: _finish_auth 0 2024-03-27T21:41:29.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c30ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:41:29.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:29.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c3a38b640 5 monclient: authenticate success, global_id 4960 2024-03-27T21:41:29.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c3a38b640 10 monclient: _renew_subs 2024-03-27T21:41:29.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c3a38b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:29.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c3a38b640 1 -- 172.21.15.60:0/766032506 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c24009400 con 0x7f4c341416f0 2024-03-27T21:41:29.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c30ff9640 1 -- 172.21.15.60:0/766032506 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2c012400 con 0x7f4c341416f0 2024-03-27T21:41:29.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c30ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:29.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c30ff9640 1 -- 172.21.15.60:0/766032506 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c00aea0 con 0x7f4c341416f0 2024-03-27T21:41:29.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:29.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c30ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:29.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c30ff9640 10 monclient: dump: 2024-03-27T21:41:29.620 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:29.620 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:29.620 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:29.620 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:29.620 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:29.620 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:29.620 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:29.620 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:29.620 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:29.620 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:29.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c3a38b640 10 client.4960.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:29.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c3a38b640 10 monclient: _renew_subs 2024-03-27T21:41:29.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c3a38b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:29.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c3a38b640 1 -- 172.21.15.60:0/766032506 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c3413e9c0 con 0x7f4c341416f0 2024-03-27T21:41:29.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c3a38b640 1 librados: init done 2024-03-27T21:41:29.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.612+0000 7f4c327fc640 4 set_mon_vals no callback set 2024-03-27T21:41:29.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.614+0000 7f4c30ff9640 1 -- 172.21.15.60:0/766032506 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4c2c012790 con 0x7f4c341416f0 2024-03-27T21:41:29.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.614+0000 7f4c3a38b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:29.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.614+0000 7f4c3a38b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:29.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.614+0000 7f4c3a38b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:29.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.615+0000 7f4c3a38b640 1 -- 172.21.15.60:0/766032506 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c34069470 con 0x7f4c341416f0 2024-03-27T21:41:29.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.615+0000 7f4c30ff9640 1 --2- 172.21.15.60:0/766032506 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4c2006c590 0x7f4c2006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:29.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.615+0000 7f4c33fff640 1 --2- 172.21.15.60:0/766032506 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4c2006c590 0x7f4c2006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:29.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.615+0000 7f4c33fff640 10 monclient: get_auth_request con 0x7f4c2006c590 auth_method 0 2024-03-27T21:41:29.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.616+0000 7f4c30ff9640 1 -- 172.21.15.60:0/766032506 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f4c2c089770 con 0x7f4c341416f0 2024-03-27T21:41:29.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.616+0000 7f4c33fff640 1 --2- 172.21.15.60:0/766032506 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4c2006c590 0x7f4c2006ea50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4c2402b800 tx=0x7f4c24042040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:29.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.616+0000 7f4c30ff9640 10 client.4960.objecter ms_dispatch 0x7f4c34000b90 osd_map(62..62 src has 1..62) v4 2024-03-27T21:41:29.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.616+0000 7f4c30ff9640 3 client.4960.objecter handle_osd_map got epochs [62,62] > 0 2024-03-27T21:41:29.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.616+0000 7f4c30ff9640 3 client.4960.objecter handle_osd_map decoding full epoch 62 2024-03-27T21:41:29.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.616+0000 7f4c30ff9640 20 client.4960.objecter dump_active .. 0 homeless 2024-03-27T21:41:29.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.616+0000 7f4c30ff9640 10 client.4960.objecter ms_handle_connect 0x7f4c2006c590 2024-03-27T21:41:29.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.620+0000 7f4c30ff9640 1 -- 172.21.15.60:0/766032506 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4c2c0585c0 con 0x7f4c341416f0 2024-03-27T21:41:29.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.620+0000 7f4c30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:29.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.620+0000 7f4c30ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:29.623 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:41:29.708 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:29.722+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:29.792 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:29.806+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:29.793 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:29.793 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:29.793 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:41:29.793 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:41:29.793 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:29.802 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:41:29.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.836+0000 7f4c3a38b640 10 start_mon_command cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}] 2024-03-27T21:41:29.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.836+0000 7f4c3a38b640 10 monclient: _send_command 2 [{"prefix": "osd reweight", "id": 0, "weight": 1.0}] 2024-03-27T21:41:29.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.836+0000 7f4c3a38b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:29.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:29.836+0000 7f4c3a38b640 1 -- 172.21.15.60:0/766032506 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) v1 -- 0x7f4c340b6b50 con 0x7f4c341416f0 2024-03-27T21:41:29.893 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:29.907+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:29.931 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:29.931 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:41:29.931 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:41:29.932 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-03-27T21:41:29.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.932 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:20 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2024-03-27T21:41:29.932 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:41:29.932 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 110.056346264, 2024-03-27T21:41:29.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015345630000000001, 2024-03-27T21:41:29.932 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:29.932 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.932 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:29.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:29.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:41:29.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.933 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:29.933 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:41:29.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.933 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:29.933 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888562+0000", 2024-03-27T21:41:29.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:29.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.933 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:29.933 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888735+0000", 2024-03-27T21:41:29.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017254799999999999 2024-03-27T21:41:29.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.934 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:29.934 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888747+0000", 2024-03-27T21:41:29.934 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2469999999999999e-05 2024-03-27T21:41:29.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.934 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.934 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888761+0000", 2024-03-27T21:41:29.934 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3254e-05 2024-03-27T21:41:29.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.934 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:29.935 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888804+0000", 2024-03-27T21:41:29.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3129e-05 2024-03-27T21:41:29.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:29.935 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888820+0000", 2024-03-27T21:41:29.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6426000000000002e-05 2024-03-27T21:41:29.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:29.935 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890060+0000", 2024-03-27T21:41:29.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012398050000000001 2024-03-27T21:41:29.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:29.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890098+0000", 2024-03-27T21:41:29.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7795e-05 2024-03-27T21:41:29.936 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.936 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:29.936 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.936 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2024-03-27T21:41:29.936 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:41:29.936 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 110.055521864, 2024-03-27T21:41:29.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001069555, 2024-03-27T21:41:29.936 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:29.936 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.937 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:29.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.937 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:29.937 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:41:29.937 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.937 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:29.937 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:41:29.937 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.937 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:29.937 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889387+0000", 2024-03-27T21:41:29.938 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:29.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.938 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:29.938 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889455+0000", 2024-03-27T21:41:29.938 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7930999999999997e-05 2024-03-27T21:41:29.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.938 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:29.938 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889470+0000", 2024-03-27T21:41:29.938 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5554e-05 2024-03-27T21:41:29.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.938 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.939 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889476+0000", 2024-03-27T21:41:29.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0619999999999997e-06 2024-03-27T21:41:29.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.939 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:29.939 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889513+0000", 2024-03-27T21:41:29.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6606999999999997e-05 2024-03-27T21:41:29.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.939 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:29.939 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889528+0000", 2024-03-27T21:41:29.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4982e-05 2024-03-27T21:41:29.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.940 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:29.940 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890421+0000", 2024-03-27T21:41:29.940 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00089293700000000001 2024-03-27T21:41:29.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.940 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:29.940 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890457+0000", 2024-03-27T21:41:29.940 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6260000000000002e-05 2024-03-27T21:41:29.940 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.940 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:29.940 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.940 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:38 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'24, mlcod=17'22)", 2024-03-27T21:41:29.940 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:41:29.941 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 110.038285898, 2024-03-27T21:41:29.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016182080000000001, 2024-03-27T21:41:29.941 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:29.941 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.941 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:29.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:29.941 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:41:29.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:29.941 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:41:29.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:29.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906623+0000", 2024-03-27T21:41:29.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:29.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:29.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906687+0000", 2024-03-27T21:41:29.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3706000000000003e-05 2024-03-27T21:41:29.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:29.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906698+0000", 2024-03-27T21:41:29.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1752e-05 2024-03-27T21:41:29.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.943 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906704+0000", 2024-03-27T21:41:29.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3940000000000004e-06 2024-03-27T21:41:29.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.943 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:29.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906741+0000", 2024-03-27T21:41:29.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6944999999999999e-05 2024-03-27T21:41:29.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.943 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:29.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906760+0000", 2024-03-27T21:41:29.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9293e-05 2024-03-27T21:41:29.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.944 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:29.944 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908211+0000", 2024-03-27T21:41:29.944 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014511350000000001 2024-03-27T21:41:29.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.944 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:29.944 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908242+0000", 2024-03-27T21:41:29.944 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0712e-05 2024-03-27T21:41:29.944 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.944 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:29.944 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.944 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:39 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'25, mlcod=17'22)", 2024-03-27T21:41:29.945 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:29.945 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 110.037440983, 2024-03-27T21:41:29.945 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012060199999999999, 2024-03-27T21:41:29.945 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:29.945 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.945 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:29.945 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.945 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:29.945 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:29.945 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.945 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.945 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:29.945 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:29.946 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.946 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:29.946 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:29.946 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:29.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.946 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:29.946 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907533+0000", 2024-03-27T21:41:29.946 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5675999999999999e-05 2024-03-27T21:41:29.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.946 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:29.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907545+0000", 2024-03-27T21:41:29.947 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.115e-05 2024-03-27T21:41:29.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907550+0000", 2024-03-27T21:41:29.947 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8320000000000002e-06 2024-03-27T21:41:29.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:29.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907592+0000", 2024-03-27T21:41:29.947 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1309e-05 2024-03-27T21:41:29.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:29.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907611+0000", 2024-03-27T21:41:29.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9009e-05 2024-03-27T21:41:29.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.948 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:29.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908638+0000", 2024-03-27T21:41:29.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001027533 2024-03-27T21:41:29.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.948 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:29.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908675+0000", 2024-03-27T21:41:29.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6329999999999999e-05 2024-03-27T21:41:29.948 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.948 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:29.949 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.949 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.949 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:57 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'35, mlcod=17'33)", 2024-03-27T21:41:29.949 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:41:29.949 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 110.02335655100001, 2024-03-27T21:41:29.949 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001346578, 2024-03-27T21:41:29.949 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:29.949 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.949 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:29.949 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.949 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:29.949 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:41:29.949 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.950 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.950 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:29.950 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:41:29.950 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.950 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.950 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.950 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:29.950 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921552+0000", 2024-03-27T21:41:29.950 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.950 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.950 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.950 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:29.950 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921615+0000", 2024-03-27T21:41:29.950 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2787999999999996e-05 2024-03-27T21:41:29.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.951 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:29.951 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921626+0000", 2024-03-27T21:41:29.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0888e-05 2024-03-27T21:41:29.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.951 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.951 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921635+0000", 2024-03-27T21:41:29.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4639999999999995e-06 2024-03-27T21:41:29.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.951 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:29.951 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921673+0000", 2024-03-27T21:41:29.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7891e-05 2024-03-27T21:41:29.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.952 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:29.952 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921694+0000", 2024-03-27T21:41:29.952 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0426000000000001e-05 2024-03-27T21:41:29.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.952 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:29.952 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922865+0000", 2024-03-27T21:41:29.952 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011709509999999999 2024-03-27T21:41:29.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.952 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:29.952 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922900+0000", 2024-03-27T21:41:29.952 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4819e-05 2024-03-27T21:41:29.953 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.953 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:29.953 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.953 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:71 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'45, mlcod=17'43)", 2024-03-27T21:41:29.953 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:41:29.953 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 110.008057839, 2024-03-27T21:41:29.953 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001460907, 2024-03-27T21:41:29.953 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:29.953 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.953 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:29.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.953 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:29.953 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:41:29.954 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.954 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:29.954 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:41:29.954 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.954 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:29.954 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936851+0000", 2024-03-27T21:41:29.954 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:29.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.954 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:29.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936903+0000", 2024-03-27T21:41:29.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2302999999999997e-05 2024-03-27T21:41:29.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.955 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:29.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936915+0000", 2024-03-27T21:41:29.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1967000000000001e-05 2024-03-27T21:41:29.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.955 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936923+0000", 2024-03-27T21:41:29.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9370000000000004e-06 2024-03-27T21:41:29.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.955 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:29.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936970+0000", 2024-03-27T21:41:29.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6987000000000003e-05 2024-03-27T21:41:29.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:29.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937002+0000", 2024-03-27T21:41:29.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2410999999999999e-05 2024-03-27T21:41:29.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:29.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938279+0000", 2024-03-27T21:41:29.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001276552 2024-03-27T21:41:29.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:29.957 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938313+0000", 2024-03-27T21:41:29.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3526999999999999e-05 2024-03-27T21:41:29.957 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.957 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:29.957 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.957 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:72 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'46, mlcod=17'43)", 2024-03-27T21:41:29.957 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:41:29.957 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 110.00742027699999, 2024-03-27T21:41:29.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014239490000000001, 2024-03-27T21:41:29.957 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:29.957 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.957 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:29.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.958 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:29.958 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:41:29.958 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.958 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.958 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:29.958 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:41:29.958 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.958 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.958 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:29.958 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937488+0000", 2024-03-27T21:41:29.958 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:29.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:29.959 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937547+0000", 2024-03-27T21:41:29.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8196999999999999e-05 2024-03-27T21:41:29.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:29.959 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937559+0000", 2024-03-27T21:41:29.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1928e-05 2024-03-27T21:41:29.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.959 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937564+0000", 2024-03-27T21:41:29.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4759999999999997e-06 2024-03-27T21:41:29.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:29.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937621+0000", 2024-03-27T21:41:29.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6712000000000002e-05 2024-03-27T21:41:29.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:29.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937640+0000", 2024-03-27T21:41:29.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9517e-05 2024-03-27T21:41:29.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:29.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938883+0000", 2024-03-27T21:41:29.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001242567 2024-03-27T21:41:29.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:29.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938913+0000", 2024-03-27T21:41:29.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0287e-05 2024-03-27T21:41:29.961 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.961 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:29.961 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.961 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:73 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'47, mlcod=17'43)", 2024-03-27T21:41:29.961 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:29.961 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 110.006744183, 2024-03-27T21:41:29.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010746239999999999, 2024-03-27T21:41:29.961 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:29.962 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.962 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:29.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.962 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:29.962 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:29.962 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.962 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:29.962 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:29.962 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.962 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:29.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:29.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:29.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.963 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:29.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938217+0000", 2024-03-27T21:41:29.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2743e-05 2024-03-27T21:41:29.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.963 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:29.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938229+0000", 2024-03-27T21:41:29.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1956999999999999e-05 2024-03-27T21:41:29.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938235+0000", 2024-03-27T21:41:29.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4589999999999997e-06 2024-03-27T21:41:29.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:29.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938341+0000", 2024-03-27T21:41:29.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000106054 2024-03-27T21:41:29.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:29.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938369+0000", 2024-03-27T21:41:29.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8291000000000001e-05 2024-03-27T21:41:29.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:29.965 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939209+0000", 2024-03-27T21:41:29.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084045400000000003 2024-03-27T21:41:29.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:29.965 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939240+0000", 2024-03-27T21:41:29.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0429999999999998e-05 2024-03-27T21:41:29.965 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.965 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:29.965 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.965 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:75 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'49, mlcod=17'47)", 2024-03-27T21:41:29.966 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:41:29.966 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 110.000306536, 2024-03-27T21:41:29.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001017001, 2024-03-27T21:41:29.966 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:29.966 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.966 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:29.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:29.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:41:29.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:29.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:41:29.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:29.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944601+0000", 2024-03-27T21:41:29.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999986 2024-03-27T21:41:29.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:29.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944614+0000", 2024-03-27T21:41:29.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2496e-05 2024-03-27T21:41:29.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:29.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944640+0000", 2024-03-27T21:41:29.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5865999999999999e-05 2024-03-27T21:41:29.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944651+0000", 2024-03-27T21:41:29.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1797000000000001e-05 2024-03-27T21:41:29.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:29.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944762+0000", 2024-03-27T21:41:29.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110412 2024-03-27T21:41:29.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:29.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944820+0000", 2024-03-27T21:41:29.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7649999999999999e-05 2024-03-27T21:41:29.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:29.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945585+0000", 2024-03-27T21:41:29.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076562600000000005 2024-03-27T21:41:29.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:29.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945620+0000", 2024-03-27T21:41:29.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4764000000000001e-05 2024-03-27T21:41:29.969 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.969 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:29.969 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.970 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:86 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'50)", 2024-03-27T21:41:29.970 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:29.970 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 109.992545049, 2024-03-27T21:41:29.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011392380000000001, 2024-03-27T21:41:29.970 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:29.970 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.970 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:29.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:29.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:29.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:29.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:29.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:29.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:29.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:29.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952373+0000", 2024-03-27T21:41:29.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9220000000000003e-06 2024-03-27T21:41:29.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:29.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952381+0000", 2024-03-27T21:41:29.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3070000000000003e-06 2024-03-27T21:41:29.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952385+0000", 2024-03-27T21:41:29.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.121e-06 2024-03-27T21:41:29.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:29.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952419+0000", 2024-03-27T21:41:29.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4082999999999999e-05 2024-03-27T21:41:29.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:29.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952440+0000", 2024-03-27T21:41:29.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0837999999999999e-05 2024-03-27T21:41:29.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:29.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953472+0000", 2024-03-27T21:41:29.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001032119 2024-03-27T21:41:29.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:29.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953504+0000", 2024-03-27T21:41:29.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1250000000000001e-05 2024-03-27T21:41:29.973 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.974 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:29.974 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.974 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:87 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-27T21:41:29.974 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:41:29.974 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 109.99217779200001, 2024-03-27T21:41:29.974 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00135519, 2024-03-27T21:41:29.974 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:29.974 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.974 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:29.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.974 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:29.974 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:41:29.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:29.975 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:41:29.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:29.975 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952731+0000", 2024-03-27T21:41:29.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:29.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952781+0000", 2024-03-27T21:41:29.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0161000000000003e-05 2024-03-27T21:41:29.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:29.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952794+0000", 2024-03-27T21:41:29.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2853e-05 2024-03-27T21:41:29.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952799+0000", 2024-03-27T21:41:29.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5560000000000001e-06 2024-03-27T21:41:29.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:29.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952832+0000", 2024-03-27T21:41:29.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3834999999999997e-05 2024-03-27T21:41:29.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:29.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952851+0000", 2024-03-27T21:41:29.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8289999999999999e-05 2024-03-27T21:41:29.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:29.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954055+0000", 2024-03-27T21:41:29.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012037949999999999 2024-03-27T21:41:29.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:29.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954087+0000", 2024-03-27T21:41:29.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2292999999999997e-05 2024-03-27T21:41:29.978 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.978 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:29.978 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.978 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:89 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'59, mlcod=17'57)", 2024-03-27T21:41:29.978 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:41:29.978 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 109.989903571, 2024-03-27T21:41:29.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00138893, 2024-03-27T21:41:29.978 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:29.978 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.978 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:29.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.979 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:29.979 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:41:29.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.979 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:29.979 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:41:29.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.979 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:29.979 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955005+0000", 2024-03-27T21:41:29.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955071+0000", 2024-03-27T21:41:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5290999999999995e-05 2024-03-27T21:41:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955081+0000", 2024-03-27T21:41:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0572e-05 2024-03-27T21:41:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955086+0000", 2024-03-27T21:41:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8380000000000001e-06 2024-03-27T21:41:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:29.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955122+0000", 2024-03-27T21:41:29.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5698000000000003e-05 2024-03-27T21:41:29.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:29.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955140+0000", 2024-03-27T21:41:29.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7900000000000001e-05 2024-03-27T21:41:29.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:29.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956366+0000", 2024-03-27T21:41:29.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001226471 2024-03-27T21:41:29.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:29.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956395+0000", 2024-03-27T21:41:29.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8782e-05 2024-03-27T21:41:29.982 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.982 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:29.982 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.982 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:90 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'60, mlcod=17'57)", 2024-03-27T21:41:29.982 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:29.982 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 109.989218994, 2024-03-27T21:41:29.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010778859999999999, 2024-03-27T21:41:29.982 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:29.982 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.983 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:29.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:29.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:29.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:29.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:29.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:29.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:29.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:29.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955747+0000", 2024-03-27T21:41:29.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7130999999999999e-05 2024-03-27T21:41:29.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:29.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955756+0000", 2024-03-27T21:41:29.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2750000000000001e-06 2024-03-27T21:41:29.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955761+0000", 2024-03-27T21:41:29.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3900000000000003e-06 2024-03-27T21:41:29.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:29.985 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955797+0000", 2024-03-27T21:41:29.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6650000000000003e-05 2024-03-27T21:41:29.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:29.985 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955816+0000", 2024-03-27T21:41:29.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8195000000000001e-05 2024-03-27T21:41:29.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:29.985 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956739+0000", 2024-03-27T21:41:29.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00092369700000000002 2024-03-27T21:41:29.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.986 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:29.986 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956768+0000", 2024-03-27T21:41:29.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9048e-05 2024-03-27T21:41:29.986 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.986 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:29.986 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.986 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:118 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'80, mlcod=17'78)", 2024-03-27T21:41:29.986 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:29.986 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 109.856228171, 2024-03-27T21:41:29.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014200669999999999, 2024-03-27T21:41:29.987 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:29.987 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.987 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:29.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.987 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:29.987 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:29.987 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.987 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:29.987 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:29.987 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.987 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:29.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:29.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.988 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:29.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088756+0000", 2024-03-27T21:41:29.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4902999999999998e-05 2024-03-27T21:41:29.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.988 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:29.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088764+0000", 2024-03-27T21:41:29.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8169999999999999e-06 2024-03-27T21:41:29.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.988 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088767+0000", 2024-03-27T21:41:29.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8310000000000003e-06 2024-03-27T21:41:29.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:29.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088798+0000", 2024-03-27T21:41:29.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0800999999999999e-05 2024-03-27T21:41:29.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:29.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088817+0000", 2024-03-27T21:41:29.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8467000000000001e-05 2024-03-27T21:41:29.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:29.990 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090074+0000", 2024-03-27T21:41:29.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012574089999999999 2024-03-27T21:41:29.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:29.990 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090101+0000", 2024-03-27T21:41:29.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7223e-05 2024-03-27T21:41:29.990 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.990 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:29.990 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.990 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:119 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'81, mlcod=17'78)", 2024-03-27T21:41:29.990 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:29.991 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 109.855493376, 2024-03-27T21:41:29.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014990870000000001, 2024-03-27T21:41:29.991 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:29.991 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.991 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:29.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.991 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:29.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:29.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.991 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:29.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:29.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.992 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:29.992 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:29.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.992 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:29.992 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089491+0000", 2024-03-27T21:41:29.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4805e-05 2024-03-27T21:41:29.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.992 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:29.992 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089498+0000", 2024-03-27T21:41:29.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0650000000000001e-06 2024-03-27T21:41:29.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.993 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.993 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089501+0000", 2024-03-27T21:41:29.993 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.54e-06 2024-03-27T21:41:29.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.993 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:29.993 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089533+0000", 2024-03-27T21:41:29.993 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1838999999999998e-05 2024-03-27T21:41:29.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.993 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:29.993 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089550+0000", 2024-03-27T21:41:29.993 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7465999999999998e-05 2024-03-27T21:41:29.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:29.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090890+0000", 2024-03-27T21:41:29.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013397979999999999 2024-03-27T21:41:29.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:29.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090915+0000", 2024-03-27T21:41:29.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4905e-05 2024-03-27T21:41:29.994 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.994 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:29.994 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.995 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:120 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'82, mlcod=17'78)", 2024-03-27T21:41:29.995 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:29.995 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 109.854693496, 2024-03-27T21:41:29.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001527093, 2024-03-27T21:41:29.995 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:29.995 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.995 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:29.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:29.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:29.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:29.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:29.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:29.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:29.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:29.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:29.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:29.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090267+0000", 2024-03-27T21:41:29.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1477000000000002e-05 2024-03-27T21:41:29.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:29.997 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090273+0000", 2024-03-27T21:41:29.997 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9519999999999999e-06 2024-03-27T21:41:29.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.997 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:29.997 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090276+0000", 2024-03-27T21:41:29.997 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.219e-06 2024-03-27T21:41:29.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.997 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:29.997 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090304+0000", 2024-03-27T21:41:29.997 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8073999999999999e-05 2024-03-27T21:41:29.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.997 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:29.998 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090316+0000", 2024-03-27T21:41:29.998 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2088e-05 2024-03-27T21:41:29.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.998 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:29.998 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091717+0000", 2024-03-27T21:41:29.998 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014003889999999999 2024-03-27T21:41:29.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.998 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:29.998 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091743+0000", 2024-03-27T21:41:29.998 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6234e-05 2024-03-27T21:41:29.998 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.998 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:29.999 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:29.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:29.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.999 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:121 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'83, mlcod=17'78)", 2024-03-27T21:41:29.999 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:29.999 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 109.853883131, 2024-03-27T21:41:29.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010226899999999999, 2024-03-27T21:41:29.999 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:29.999 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:29.999 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:29.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:29.999 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:29.999 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:29.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:30.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:30.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:30.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:30.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:30.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:30.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:30.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:30.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091142+0000", 2024-03-27T21:41:30.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011608300000000001 2024-03-27T21:41:30.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:30.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091151+0000", 2024-03-27T21:41:30.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2350000000000005e-06 2024-03-27T21:41:30.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:30.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091156+0000", 2024-03-27T21:41:30.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5419999999999999e-06 2024-03-27T21:41:30.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.002 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:30.002 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091191+0000", 2024-03-27T21:41:30.002 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5453000000000001e-05 2024-03-27T21:41:30.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.002 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:30.002 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091208+0000", 2024-03-27T21:41:30.002 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7119999999999999e-05 2024-03-27T21:41:30.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.002 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:30.002 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092024+0000", 2024-03-27T21:41:30.002 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000815539 2024-03-27T21:41:30.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.003 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:30.003 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092049+0000", 2024-03-27T21:41:30.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5171e-05 2024-03-27T21:41:30.003 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:30.003 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:30.003 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:30.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.003 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:142 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'100, mlcod=18'98)", 2024-03-27T21:41:30.003 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:30.003 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 109.438876177, 2024-03-27T21:41:30.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001198778, 2024-03-27T21:41:30.003 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:30.003 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:30.004 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:30.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:30.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:30.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:30.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:30.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:30.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:30.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:30.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:30.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:30.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.005 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:30.005 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506101+0000", 2024-03-27T21:41:30.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8202999999999997e-05 2024-03-27T21:41:30.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.005 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:30.005 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506109+0000", 2024-03-27T21:41:30.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3059999999999998e-06 2024-03-27T21:41:30.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.005 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:30.005 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506113+0000", 2024-03-27T21:41:30.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0119999999999999e-06 2024-03-27T21:41:30.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:30.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506137+0000", 2024-03-27T21:41:30.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4043999999999999e-05 2024-03-27T21:41:30.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:30.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506148+0000", 2024-03-27T21:41:30.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0446e-05 2024-03-27T21:41:30.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:30.007 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507213+0000", 2024-03-27T21:41:30.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001065215 2024-03-27T21:41:30.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.007 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:30.007 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507232+0000", 2024-03-27T21:41:30.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9038e-05 2024-03-27T21:41:30.007 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:30.007 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:30.007 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:30.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.007 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:143 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'101, mlcod=18'98)", 2024-03-27T21:41:30.007 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:30.007 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 109.438263558, 2024-03-27T21:41:30.008 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001357488, 2024-03-27T21:41:30.008 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:30.008 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:30.008 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:30.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.008 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:30.008 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:30.008 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:30.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.008 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:30.008 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:30.008 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:30.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:30.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:30.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:30.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:30.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506723+0000", 2024-03-27T21:41:30.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7404000000000006e-05 2024-03-27T21:41:30.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:30.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506730+0000", 2024-03-27T21:41:30.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5749999999999997e-06 2024-03-27T21:41:30.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.010 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:30.010 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506733+0000", 2024-03-27T21:41:30.010 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7979999999999999e-06 2024-03-27T21:41:30.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.010 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:30.010 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506758+0000", 2024-03-27T21:41:30.010 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4179e-05 2024-03-27T21:41:30.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.010 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:30.010 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506768+0000", 2024-03-27T21:41:30.010 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0740000000000001e-05 2024-03-27T21:41:30.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.011 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:30.011 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507978+0000", 2024-03-27T21:41:30.011 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012093900000000001 2024-03-27T21:41:30.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.011 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:30.011 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508003+0000", 2024-03-27T21:41:30.011 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5667e-05 2024-03-27T21:41:30.011 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:30.011 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:30.011 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:30.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.012 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:144 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'102, mlcod=18'98)", 2024-03-27T21:41:30.012 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:41:30.012 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 109.43760466099999, 2024-03-27T21:41:30.012 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013453390000000001, 2024-03-27T21:41:30.012 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:30.012 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:30.012 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:30.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.012 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:30.012 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:41:30.012 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:30.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.012 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:30.012 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:41:30.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:30.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.013 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:30.013 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507304+0000", 2024-03-27T21:41:30.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:30.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.013 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:30.013 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507397+0000", 2024-03-27T21:41:30.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2995e-05 2024-03-27T21:41:30.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.013 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:30.014 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507403+0000", 2024-03-27T21:41:30.014 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8640000000000001e-06 2024-03-27T21:41:30.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.014 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:30.014 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507407+0000", 2024-03-27T21:41:30.014 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5710000000000002e-06 2024-03-27T21:41:30.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.014 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:30.014 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507428+0000", 2024-03-27T21:41:30.014 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1192e-05 2024-03-27T21:41:30.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:30.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507438+0000", 2024-03-27T21:41:30.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8309999999999996e-06 2024-03-27T21:41:30.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:30.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508624+0000", 2024-03-27T21:41:30.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011861210000000001 2024-03-27T21:41:30.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:30.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:30.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:30.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508650+0000", 2024-03-27T21:41:30.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6118e-05 2024-03-27T21:41:30.015 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:30.015 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:30.016 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:30.016 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:30.016 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:30.016 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:30.017 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:41:30.018 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:30.031+0000 7fb850f34640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 56426) UID: 0 2024-03-27T21:41:30.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.061+0000 7f4c30ff9640 1 -- 172.21.15.60:0/766032506 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v63) v1 ==== 113+0+0 (secure 0 0 0) 0x7f4c2c058150 con 0x7f4c341416f0 2024-03-27T21:41:30.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.061+0000 7f4c30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd reweight", "id": 0, "weight": 1.0}] 2024-03-27T21:41:30.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.061+0000 7f4c30ff9640 10 monclient: _finish_command 2 = system:0 reweighted osd.0 to 1 (10000) 2024-03-27T21:41:30.048 INFO:teuthology.orchestra.run.smithi060.stderr:reweighted osd.0 to 1 (10000) 2024-03-27T21:41:30.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.064+0000 7f4c3a38b640 10 librados: watch_flush enter 2024-03-27T21:41:30.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.064+0000 7f4c3a38b640 10 librados: watch_flush exit 2024-03-27T21:41:30.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.064+0000 7f4c3a38b640 20 client.4960.objecter shutdown clearing up homeless session... 2024-03-27T21:41:30.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.064+0000 7f4c3a38b640 10 client.4960.objecter successfully canceled tick 2024-03-27T21:41:30.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.064+0000 7f4c3a38b640 1 -- 172.21.15.60:0/766032506 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4c2006c590 msgr2=0x7f4c2006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:30.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.064+0000 7f4c3a38b640 1 --2- 172.21.15.60:0/766032506 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4c2006c590 0x7f4c2006ea50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4c2402b800 tx=0x7f4c24042040 comp rx=0 tx=0).stop 2024-03-27T21:41:30.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.064+0000 7f4c3a38b640 10 monclient: shutdown 2024-03-27T21:41:30.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.064+0000 7f4c3a38b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:30.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.064+0000 7f4c3a38b640 1 -- 172.21.15.60:0/766032506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c341416f0 msgr2=0x7f4c341d6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:30.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.064+0000 7f4c3a38b640 1 --2- 172.21.15.60:0/766032506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c341416f0 0x7f4c341d6df0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c001700 tx=0x7f4c2c00a430 comp rx=0 tx=0).stop 2024-03-27T21:41:30.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.064+0000 7f4c3a38b640 1 -- 172.21.15.60:0/766032506 shutdown_connections 2024-03-27T21:41:30.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.064+0000 7f4c3a38b640 1 --2- 172.21.15.60:0/766032506 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4c2006c590 0x7f4c2006ea50 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:30.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.064+0000 7f4c3a38b640 1 --2- 172.21.15.60:0/766032506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c341416f0 0x7f4c341d6df0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:30.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.064+0000 7f4c3a38b640 1 --2- 172.21.15.60:0/766032506 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4c3413b020 0x7f4c341d68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:30.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.064+0000 7f4c3a38b640 1 --2- 172.21.15.60:0/766032506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c340b40b0 0x7f4c340aac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:30.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.065+0000 7f4c3a38b640 1 -- 172.21.15.60:0/766032506 >> 172.21.15.60:0/766032506 conn(0x7f4c340608e0 msgr2=0x7f4c34068be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:30.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.065+0000 7f4c3a38b640 1 -- 172.21.15.60:0/766032506 shutdown_connections 2024-03-27T21:41:30.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.065+0000 7f4c3a38b640 1 -- 172.21.15.60:0/766032506 wait complete. 2024-03-27T21:41:30.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.065+0000 7f4c3a38b640 1 librados: shutdown 2024-03-27T21:41:30.068 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-27T21:41:30.118 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:30.131+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:30.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.229+0000 7fa919a82640 10 monclient: get_monmap_and_config 2024-03-27T21:41:30.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.229+0000 7fa919a82640 10 monclient: build_initial_monmap 2024-03-27T21:41:30.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.229+0000 7fa919a82640 10 monclient: monmap: 2024-03-27T21:41:30.216 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:30.216 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:30.216 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:30.230460+0000 2024-03-27T21:41:30.216 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:30.230460+0000 2024-03-27T21:41:30.216 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:30.216 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:30.216 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:30.216 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:30.216 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:30.216 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:30.216 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:30.217 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:30.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.230+0000 7fa919a82640 1 Processor -- start 2024-03-27T21:41:30.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.230+0000 7fa919a82640 1 -- start start 2024-03-27T21:41:30.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.230+0000 7fa919a82640 10 monclient: init 2024-03-27T21:41:30.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa919a82640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:30.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa919a82640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-03-27T21:41:30.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa919a82640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa914071a50 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:30.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa919a82640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa91413ef20 0x7fa9140a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:30.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa919a82640 10 monclient(hunting): picked mon.noname-a con 0x7fa91413ef20 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:30.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa919a82640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa914141630 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:30.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa919a82640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9140a8e90 con 0x7fa914071a50 2024-03-27T21:41:30.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa919a82640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9140aa040 con 0x7fa914141630 2024-03-27T21:41:30.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa919a82640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:30.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa919a82640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa914071e30 con 0x7fa91413ef20 2024-03-27T21:41:30.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa919a82640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:30.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa919a82640 10 monclient(hunting): authenticate will time out at 931.452107s 2024-03-27T21:41:30.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa912ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa91413ef20 0x7fa9140a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:30.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa9137fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa914071a50 0x7fa91413e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56832/0 (socket says 172.21.15.60:56832) 2024-03-27T21:41:30.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa9137fe640 1 -- 172.21.15.60:0/2429308527 learned_addr learned my addr 172.21.15.60:0/2429308527 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:30.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa912ffd640 10 monclient(hunting): get_auth_request con 0x7fa91413ef20 auth_method 0 2024-03-27T21:41:30.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa912ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:30.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa912ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:30.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:30.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa912ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:30.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa912ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:30.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.231+0000 7fa912ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:30.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.232+0000 7fa912ffd640 10 monclient(hunting): handle_auth_done global_id 5010 payload 274 2024-03-27T21:41:30.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.232+0000 7fa9127fc640 1 -- 172.21.15.60:0/2429308527 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2419688840 0 0) 0x7fa9140aa040 con 0x7fa914141630 2024-03-27T21:41:30.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.232+0000 7fa912ffd640 1 -- 172.21.15.60:0/2429308527 >> v1:172.21.15.191:6789/0 conn(0x7fa914141630 legacy=0x7fa914141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:30.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.232+0000 7fa912ffd640 1 -- 172.21.15.60:0/2429308527 >> v1:172.21.15.60:6789/0 conn(0x7fa914071a50 legacy=0x7fa91413e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:30.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.232+0000 7fa912ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:41:30.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.232+0000 7fa912ffd640 1 monclient: found mon.noname-a 2024-03-27T21:41:30.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.232+0000 7fa912ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:30.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.232+0000 7fa912ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:30.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.232+0000 7fa912ffd640 1 -- 172.21.15.60:0/2429308527 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9140aa2b0 con 0x7fa91413ef20 2024-03-27T21:41:30.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.232+0000 7fa9127fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:41:30.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.233+0000 7fa912ffd640 1 --2- 172.21.15.60:0/2429308527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa91413ef20 0x7fa9140a87e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa9080097a0 tx=0x7fa90802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a094e4a29c2d886 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:30.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.233+0000 7fa9127fc640 1 -- 172.21.15.60:0/2429308527 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa908030840 con 0x7fa91413ef20 2024-03-27T21:41:30.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.233+0000 7fa9127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:30.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.233+0000 7fa9127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:30.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.233+0000 7fa9127fc640 10 monclient: dump: 2024-03-27T21:41:30.224 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:30.224 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:30.224 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:30.224 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:30.224 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:30.224 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:30.224 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:30.224 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:30.224 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:30.224 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:30.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.233+0000 7fa9127fc640 10 monclient: _finish_auth 0 2024-03-27T21:41:30.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.233+0000 7fa9127fc640 10 monclient: _check_auth_tickets 2024-03-27T21:41:30.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.233+0000 7fa9127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:30.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.233+0000 7fa919a82640 5 monclient: authenticate success, global_id 5010 2024-03-27T21:41:30.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.233+0000 7fa919a82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:30.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.233+0000 7fa9127fc640 1 -- 172.21.15.60:0/2429308527 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9080309a0 con 0x7fa91413ef20 2024-03-27T21:41:30.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.233+0000 7fa9127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:30.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.233+0000 7fa9127fc640 1 -- 172.21.15.60:0/2429308527 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa908030d30 con 0x7fa91413ef20 2024-03-27T21:41:30.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.233+0000 7fa9127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:30.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.233+0000 7fa9127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:30.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.233+0000 7fa9127fc640 10 monclient: dump: 2024-03-27T21:41:30.225 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:30.225 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:30.225 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:30.225 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:30.226 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:30.226 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:30.226 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:30.226 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:30.226 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:30.226 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:30.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.233+0000 7fa919a82640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:30.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.233+0000 7fa919a82640 4 set_mon_vals no callback set 2024-03-27T21:41:30.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.234+0000 7fa919a82640 10 monclient: shutdown 2024-03-27T21:41:30.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.234+0000 7fa919a82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:30.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.234+0000 7fa919a82640 1 -- 172.21.15.60:0/2429308527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa91413ef20 msgr2=0x7fa9140a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:30.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.234+0000 7fa919a82640 1 --2- 172.21.15.60:0/2429308527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa91413ef20 0x7fa9140a87e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa9080097a0 tx=0x7fa90802b980 comp rx=0 tx=0).stop 2024-03-27T21:41:30.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.234+0000 7fa919a82640 1 -- 172.21.15.60:0/2429308527 shutdown_connections 2024-03-27T21:41:30.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.234+0000 7fa919a82640 1 --2- 172.21.15.60:0/2429308527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa91413ef20 0x7fa9140a87e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:30.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.234+0000 7fa919a82640 1 -- 172.21.15.60:0/2429308527 >> 172.21.15.60:0/2429308527 conn(0x7fa9140b7300 msgr2=0x7fa9140b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:30.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.234+0000 7fa919a82640 1 -- 172.21.15.60:0/2429308527 shutdown_connections 2024-03-27T21:41:30.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.234+0000 7fa919a82640 1 -- 172.21.15.60:0/2429308527 wait complete. 2024-03-27T21:41:30.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.235+0000 7fa919a82640 10 monclient: build_initial_monmap 2024-03-27T21:41:30.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.235+0000 7fa919a82640 10 monclient: monmap: 2024-03-27T21:41:30.227 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:30.227 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:30.227 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:30.227 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:30.227 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:30.227 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:30.227 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:30.227 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:30.228 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:30.228 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:30.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.235+0000 7fa919a82640 1 librados: starting msgr at 2024-03-27T21:41:30.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.235+0000 7fa919a82640 1 librados: starting objecter 2024-03-27T21:41:30.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.236+0000 7fa919a82640 1 Processor -- start 2024-03-27T21:41:30.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.236+0000 7fa919a82640 1 -- start start 2024-03-27T21:41:30.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.236+0000 7fa919a82640 1 librados: setting wanted keys 2024-03-27T21:41:30.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.236+0000 7fa919a82640 1 librados: calling monclient init 2024-03-27T21:41:30.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.236+0000 7fa919a82640 10 monclient: init 2024-03-27T21:41:30.229 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:30.229 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:30.230 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:30.230 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:30.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa919a82640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:30.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa919a82640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:41:30.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa919a82640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa914071a50 0x7fa914138b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:30.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa919a82640 10 monclient(hunting): picked mon.noname-c con 0x7fa914071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:30.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa919a82640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa914141630 0x7fa914139150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:30.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa919a82640 10 monclient(hunting): picked mon.noname-b con 0x7fa914141630 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:30.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa919a82640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa91413ace0 0x7fa914130a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:30.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa9137fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa914071a50 0x7fa914138b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:30.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa9137fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa914071a50 0x7fa914138b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:60318/0 (socket says 172.21.15.60:60318) 2024-03-27T21:41:30.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa913fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa91413ace0 0x7fa914130a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:30.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa913fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa91413ace0 0x7fa914130a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36552/0 (socket says 172.21.15.60:36552) 2024-03-27T21:41:30.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa912ffd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa914141630 0x7fa914139150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:30.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa919a82640 10 monclient(hunting): picked mon.noname-a con 0x7fa91413ace0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:30.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa919a82640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:30.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa919a82640 1 -- 172.21.15.60:0/3696559475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa914138500 con 0x7fa91413ace0 2024-03-27T21:41:30.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa919a82640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:30.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa919a82640 1 -- 172.21.15.60:0/3696559475 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9141381f0 con 0x7fa914141630 2024-03-27T21:41:30.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa919a82640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:30.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa919a82640 1 -- 172.21.15.60:0/3696559475 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa914131ad0 con 0x7fa914071a50 2024-03-27T21:41:30.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa919a82640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:30.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa919a82640 10 monclient(hunting): authenticate will time out at 931.458583s 2024-03-27T21:41:30.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa9137fe640 1 -- 172.21.15.60:0/3696559475 learned_addr learned my addr 172.21.15.60:0/3696559475 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:30.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa9137fe640 10 monclient(hunting): get_auth_request con 0x7fa914071a50 auth_method 0 2024-03-27T21:41:30.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa9137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:30.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa9137fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:30.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa9137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:30.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa913fff640 10 monclient(hunting): get_auth_request con 0x7fa91413ace0 auth_method 0 2024-03-27T21:41:30.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa913fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:30.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa913fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:30.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa913fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:30.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa9137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:30.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa9137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:30.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa9137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:30.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa913fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:30.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa913fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:30.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.237+0000 7fa913fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:30.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa9137fe640 10 monclient(hunting): handle_auth_done global_id 4796 payload 824 2024-03-27T21:41:30.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa9137fe640 1 -- 172.21.15.60:0/3696559475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa914141630 msgr2=0x7fa914139150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:30.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa9137fe640 1 --2- 172.21.15.60:0/3696559475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa914141630 0x7fa914139150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:30.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa9137fe640 1 -- 172.21.15.60:0/3696559475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa91413ace0 msgr2=0x7fa914130a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:30.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa9137fe640 1 --2- 172.21.15.60:0/3696559475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa91413ace0 0x7fa914130a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:30.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa9137fe640 10 monclient: _finish_hunting 0 2024-03-27T21:41:30.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa9137fe640 1 monclient: found mon.noname-c 2024-03-27T21:41:30.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa9137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:30.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa9137fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:30.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa9137fe640 1 -- 172.21.15.60:0/3696559475 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa908009400 con 0x7fa914071a50 2024-03-27T21:41:30.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa913fff640 1 --2- 172.21.15.60:0/3696559475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa91413ace0 0x7fa914130a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:30.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa9137fe640 1 --2- 172.21.15.60:0/3696559475 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa914071a50 0x7fa914138b60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa90400c4a0 tx=0x7fa90400c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:30.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa8fbfff640 10 client.?.objecter ms_handle_connect 0x7fa914071a50 2024-03-27T21:41:30.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa8fbfff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:30.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa8fbfff640 1 -- 172.21.15.60:0/3696559475 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa904005ce0 con 0x7fa914071a50 2024-03-27T21:41:30.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa8fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:30.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa8fbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:30.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa8fbfff640 10 monclient: dump: 2024-03-27T21:41:30.237 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:30.237 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:30.238 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:30.238 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:30.238 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:30.238 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:30.238 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:30.238 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:30.238 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:30.238 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:30.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa8fbfff640 10 monclient: _finish_auth 0 2024-03-27T21:41:30.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa8fbfff640 10 monclient: _check_auth_tickets 2024-03-27T21:41:30.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa8fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:30.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa8fbfff640 1 -- 172.21.15.60:0/3696559475 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa904005e40 con 0x7fa914071a50 2024-03-27T21:41:30.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa8fbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:30.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa8fbfff640 1 -- 172.21.15.60:0/3696559475 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa904007270 con 0x7fa914071a50 2024-03-27T21:41:30.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa8fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:30.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa8fbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:30.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa8fbfff640 10 monclient: dump: 2024-03-27T21:41:30.239 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:30.239 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:30.239 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:30.239 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:30.239 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:30.239 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:30.239 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:30.239 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:30.239 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:30.239 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:30.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa919a82640 5 monclient: authenticate success, global_id 4796 2024-03-27T21:41:30.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa919a82640 10 monclient: _renew_subs 2024-03-27T21:41:30.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa919a82640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:30.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa919a82640 1 -- 172.21.15.60:0/3696559475 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa91413a5e0 con 0x7fa914071a50 2024-03-27T21:41:30.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa911ffb640 4 set_mon_vals no callback set 2024-03-27T21:41:30.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa919a82640 10 client.4796.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:30.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa919a82640 10 monclient: _renew_subs 2024-03-27T21:41:30.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa919a82640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:30.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa919a82640 1 -- 172.21.15.60:0/3696559475 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9140b6c00 con 0x7fa914071a50 2024-03-27T21:41:30.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.238+0000 7fa919a82640 1 librados: init done 2024-03-27T21:41:30.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.240+0000 7fa8f9ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:30.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.240+0000 7fa8f9ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:30.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.240+0000 7fa8f9ffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:30.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.240+0000 7fa8f9ffb640 1 -- 172.21.15.60:0/3696559475 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8ec005040 con 0x7fa914071a50 2024-03-27T21:41:30.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.240+0000 7fa8fbfff640 1 -- 172.21.15.60:0/3696559475 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa904007490 con 0x7fa914071a50 2024-03-27T21:41:30.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.241+0000 7fa8fbfff640 1 --2- 172.21.15.60:0/3696559475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa90006c590 0x7fa90006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:30.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.241+0000 7fa8fbfff640 1 -- 172.21.15.60:0/3696559475 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fa904088530 con 0x7fa914071a50 2024-03-27T21:41:30.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.241+0000 7fa8fbfff640 10 client.4796.objecter ms_dispatch 0x7fa914000b90 osd_map(63..63 src has 1..63) v4 2024-03-27T21:41:30.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.241+0000 7fa8fbfff640 3 client.4796.objecter handle_osd_map got epochs [63,63] > 0 2024-03-27T21:41:30.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.241+0000 7fa8fbfff640 3 client.4796.objecter handle_osd_map decoding full epoch 63 2024-03-27T21:41:30.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.241+0000 7fa8fbfff640 20 client.4796.objecter dump_active .. 0 homeless 2024-03-27T21:41:30.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.241+0000 7fa912ffd640 1 --2- 172.21.15.60:0/3696559475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa90006c590 0x7fa90006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:30.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.241+0000 7fa912ffd640 10 monclient: get_auth_request con 0x7fa90006c590 auth_method 0 2024-03-27T21:41:30.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.242+0000 7fa8fbfff640 10 client.4796.objecter ms_handle_connect 0x7fa90006c590 2024-03-27T21:41:30.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.242+0000 7fa912ffd640 1 --2- 172.21.15.60:0/3696559475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa90006c590 0x7fa90006ea50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa908036000 tx=0x7fa90803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:30.241 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:30.243+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:30.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.246+0000 7fa8fbfff640 1 -- 172.21.15.60:0/3696559475 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa904053a20 con 0x7fa914071a50 2024-03-27T21:41:30.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.246+0000 7fa8fbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:30.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.246+0000 7fa8fbfff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:30.243 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:41:30.330 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:30.344+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:30.430 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:30.444+0000 7fb850f34640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 56426) UID: 0 2024-03-27T21:41:30.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.452+0000 7fa8f9ffb640 10 start_mon_command cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}] 2024-03-27T21:41:30.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.452+0000 7fa8f9ffb640 10 monclient: _send_command 2 [{"prefix": "osd reweight", "id": 2, "weight": 1.0}] 2024-03-27T21:41:30.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.452+0000 7fa8f9ffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:30.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:30.452+0000 7fa8f9ffb640 1 -- 172.21.15.60:0/3696559475 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) v1 -- 0x7fa8ec0055c0 con 0x7fa914071a50 2024-03-27T21:41:30.453 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:30.453 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:30.453 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:41:30.453 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:41:30.453 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:30.466 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:41:30.532 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:30.546+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:30.613 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:30.613 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:41:30.613 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:41:30.613 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:41:30.613 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:30.622 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:41:30.666 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:30.680+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:30.730 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:30.731 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:30.731 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:30.731 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:30.734 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:30.748+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:30.739 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:41:30.835 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:30.849+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:30.925 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:30.925 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:30.925 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:41:30.926 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:41:30.926 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:30.934 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:41:30.936 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:30.949+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:30.966 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:41:31.037 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:31.051+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:31.047 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:31.047 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:41:31.047 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:41:31.047 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:41:31.047 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:31.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.064+0000 7fa8fbfff640 1 -- 172.21.15.60:0/3696559475 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v64) v1 ==== 113+0+0 (secure 0 0 0) 0x7fa904057040 con 0x7fa914071a50 2024-03-27T21:41:31.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.064+0000 7fa8fbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd reweight", "id": 2, "weight": 1.0}] 2024-03-27T21:41:31.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.064+0000 7fa8fbfff640 10 monclient: _finish_command 2 = system:0 reweighted osd.2 to 1 (10000) 2024-03-27T21:41:31.050 INFO:teuthology.orchestra.run.smithi060.stderr:reweighted osd.2 to 1 (10000) 2024-03-27T21:41:31.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.067+0000 7fa8f9ffb640 10 librados: watch_flush enter 2024-03-27T21:41:31.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.067+0000 7fa8f9ffb640 10 librados: watch_flush exit 2024-03-27T21:41:31.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.067+0000 7fa8f9ffb640 20 client.4796.objecter shutdown clearing up homeless session... 2024-03-27T21:41:31.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.067+0000 7fa8f9ffb640 10 client.4796.objecter successfully canceled tick 2024-03-27T21:41:31.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.067+0000 7fa8f9ffb640 1 -- 172.21.15.60:0/3696559475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa90006c590 msgr2=0x7fa90006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.055 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-27T21:41:31.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.067+0000 7fa8f9ffb640 1 --2- 172.21.15.60:0/3696559475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa90006c590 0x7fa90006ea50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa908036000 tx=0x7fa90803b040 comp rx=0 tx=0).stop 2024-03-27T21:41:31.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.070+0000 7fa8f9ffb640 10 monclient: shutdown 2024-03-27T21:41:31.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.070+0000 7fa8f9ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:31.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.070+0000 7fa8f9ffb640 1 -- 172.21.15.60:0/3696559475 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa914071a50 msgr2=0x7fa914138b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.070+0000 7fa8f9ffb640 1 --2- 172.21.15.60:0/3696559475 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa914071a50 0x7fa914138b60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa90400c4a0 tx=0x7fa90400c960 comp rx=0 tx=0).stop 2024-03-27T21:41:31.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.070+0000 7fa8f9ffb640 1 -- 172.21.15.60:0/3696559475 shutdown_connections 2024-03-27T21:41:31.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.070+0000 7fa8f9ffb640 1 --2- 172.21.15.60:0/3696559475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa91413ace0 0x7fa914130a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.070+0000 7fa8f9ffb640 1 --2- 172.21.15.60:0/3696559475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa90006c590 0x7fa90006ea50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.070+0000 7fa8f9ffb640 1 --2- 172.21.15.60:0/3696559475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa914141630 0x7fa914139150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.070+0000 7fa8f9ffb640 1 --2- 172.21.15.60:0/3696559475 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa914071a50 0x7fa914138b60 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.070+0000 7fa8f9ffb640 1 -- 172.21.15.60:0/3696559475 >> 172.21.15.60:0/3696559475 conn(0x7fa9140b7300 msgr2=0x7fa914135700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:31.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.070+0000 7fa8f9ffb640 1 -- 172.21.15.60:0/3696559475 shutdown_connections 2024-03-27T21:41:31.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.070+0000 7fa8f9ffb640 1 -- 172.21.15.60:0/3696559475 wait complete. 2024-03-27T21:41:31.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.070+0000 7fa8f9ffb640 1 librados: shutdown 2024-03-27T21:41:31.062 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:41:31.071 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-27T21:41:31.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.131+0000 7f73a9a2f640 10 monclient: get_monmap_and_config 2024-03-27T21:41:31.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.131+0000 7f73a9a2f640 10 monclient: build_initial_monmap 2024-03-27T21:41:31.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.131+0000 7f73a9a2f640 10 monclient: monmap: 2024-03-27T21:41:31.118 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:31.118 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:31.118 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:31.132701+0000 2024-03-27T21:41:31.118 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:31.132701+0000 2024-03-27T21:41:31.118 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:31.118 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:31.118 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:31.118 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:31.118 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:31.118 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:31.118 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:31.118 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:31.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.132+0000 7f73a9a2f640 1 Processor -- start 2024-03-27T21:41:31.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.132+0000 7f73a9a2f640 1 -- start start 2024-03-27T21:41:31.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.132+0000 7f73a9a2f640 10 monclient: init 2024-03-27T21:41:31.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.133+0000 7f73a9a2f640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:31.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.133+0000 7f73a9a2f640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-03-27T21:41:31.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.133+0000 7f73a9a2f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f73a40b40b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:31.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.133+0000 7f73a9a2f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f73a413b020 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:31.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.133+0000 7f73a9a2f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f73a413b480 0x7f73a4141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.133+0000 7f73a2ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f73a40b40b0 0x7f73a40a8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56846/0 (socket says 172.21.15.60:56846) 2024-03-27T21:41:31.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.133+0000 7f73a27fc640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f73a413b020 0x7f73a41416f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.60:44432/0 (socket says 172.21.15.60:44432) 2024-03-27T21:41:31.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.133+0000 7f73a37fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f73a413b480 0x7f73a4141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.133+0000 7f73a37fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f73a413b480 0x7f73a4141e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:60328/0 (socket says 172.21.15.60:60328) 2024-03-27T21:41:31.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.133+0000 7f73a9a2f640 10 monclient(hunting): picked mon.noname-b con 0x7f73a413b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:31.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.133+0000 7f73a2ffd640 1 -- 172.21.15.60:0/382230216 learned_addr learned my addr 172.21.15.60:0/382230216 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:31.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.133+0000 7f73a9a2f640 1 -- 172.21.15.60:0/382230216 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a40b3830 con 0x7f73a40b40b0 2024-03-27T21:41:31.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a9a2f640 1 -- 172.21.15.60:0/382230216 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a40aa520 con 0x7f73a413b020 2024-03-27T21:41:31.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a9a2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:31.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a9a2f640 1 -- 172.21.15.60:0/382230216 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73a413b910 con 0x7f73a413b480 2024-03-27T21:41:31.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a9a2f640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:31.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a9a2f640 10 monclient(hunting): authenticate will time out at 932.355104s 2024-03-27T21:41:31.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a37fe640 10 monclient(hunting): get_auth_request con 0x7f73a413b480 auth_method 0 2024-03-27T21:41:31.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:31.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:31.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:31.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:31.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:31.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a1ffb640 1 -- 172.21.15.60:0/382230216 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1210438746 0 0) 0x7f73a40b3830 con 0x7f73a40b40b0 2024-03-27T21:41:31.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:31.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a1ffb640 10 monclient(hunting): _negotiate 2024-03-27T21:41:31.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:31.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:31.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a1ffb640 10 monclient(hunting): my global_id is 5019 2024-03-27T21:41:31.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a1ffb640 1 -- 172.21.15.60:0/382230216 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f738c003520 con 0x7f73a40b40b0 2024-03-27T21:41:31.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a1ffb640 1 -- 172.21.15.60:0/382230216 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88281996 0 0) 0x7f73a40aa520 con 0x7f73a413b020 2024-03-27T21:41:31.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a1ffb640 10 monclient(hunting): _negotiate 2024-03-27T21:41:31.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:31.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:31.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a1ffb640 10 monclient(hunting): my global_id is 4966 2024-03-27T21:41:31.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a1ffb640 1 -- 172.21.15.60:0/382230216 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a40b3830 con 0x7f73a413b020 2024-03-27T21:41:31.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a37fe640 10 monclient(hunting): handle_auth_done global_id 4799 payload 274 2024-03-27T21:41:31.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a37fe640 1 -- 172.21.15.60:0/382230216 >> v1:172.21.15.191:6789/0 conn(0x7f73a413b020 legacy=0x7f73a41416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a37fe640 1 -- 172.21.15.60:0/382230216 >> v1:172.21.15.60:6789/0 conn(0x7f73a40b40b0 legacy=0x7f73a40a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a37fe640 10 monclient: _finish_hunting 0 2024-03-27T21:41:31.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a37fe640 1 monclient: found mon.noname-b 2024-03-27T21:41:31.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:31.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:31.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.134+0000 7f73a37fe640 1 -- 172.21.15.60:0/382230216 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a40a9470 con 0x7f73a413b480 2024-03-27T21:41:31.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.135+0000 7f73a37fe640 1 --2- 172.21.15.60:0/382230216 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f73a413b480 0x7f73a4141e00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f739c009dc0 tx=0x7f739c02bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8e06b17c7af07c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:31.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.135+0000 7f73a1ffb640 1 -- 172.21.15.60:0/382230216 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c02ccc0 con 0x7f73a413b480 2024-03-27T21:41:31.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.135+0000 7f73a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:31.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.135+0000 7f73a1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:31.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.135+0000 7f73a1ffb640 10 monclient: dump: 2024-03-27T21:41:31.134 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:31.134 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:31.134 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.134 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.134 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:31.134 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:31.135 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:31.135 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:31.135 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:31.136 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:31.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.135+0000 7f73a1ffb640 10 monclient: _finish_auth 0 2024-03-27T21:41:31.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.135+0000 7f73a1ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:41:31.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.135+0000 7f73a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:31.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.135+0000 7f73a9a2f640 5 monclient: authenticate success, global_id 4799 2024-03-27T21:41:31.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.135+0000 7f73a9a2f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:31.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.136+0000 7f73a1ffb640 1 -- 172.21.15.60:0/382230216 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739c02ce20 con 0x7f73a413b480 2024-03-27T21:41:31.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.136+0000 7f73a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:31.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.136+0000 7f73a1ffb640 1 -- 172.21.15.60:0/382230216 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c034640 con 0x7f73a413b480 2024-03-27T21:41:31.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.136+0000 7f73a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:31.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.136+0000 7f73a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:31.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.136+0000 7f73a1ffb640 10 monclient: dump: 2024-03-27T21:41:31.137 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:31.137 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:31.137 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.137 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.137 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:31.137 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:31.137 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:31.137 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:31.137 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:31.137 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:31.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.136+0000 7f73a9a2f640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:31.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.136+0000 7f73a9a2f640 4 set_mon_vals no callback set 2024-03-27T21:41:31.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.136+0000 7f73a9a2f640 10 monclient: shutdown 2024-03-27T21:41:31.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.136+0000 7f73a9a2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:31.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.136+0000 7f73a9a2f640 1 -- 172.21.15.60:0/382230216 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f73a413b480 msgr2=0x7f73a4141e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.136+0000 7f73a9a2f640 1 --2- 172.21.15.60:0/382230216 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f73a413b480 0x7f73a4141e00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f739c009dc0 tx=0x7f739c02bf10 comp rx=0 tx=0).stop 2024-03-27T21:41:31.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.136+0000 7f73a9a2f640 1 -- 172.21.15.60:0/382230216 shutdown_connections 2024-03-27T21:41:31.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.136+0000 7f73a9a2f640 1 --2- 172.21.15.60:0/382230216 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f73a413b480 0x7f73a4141e00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.136+0000 7f73a9a2f640 1 -- 172.21.15.60:0/382230216 >> 172.21.15.60:0/382230216 conn(0x7f73a40608e0 msgr2=0x7f73a4060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:31.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.137+0000 7f73a9a2f640 1 -- 172.21.15.60:0/382230216 shutdown_connections 2024-03-27T21:41:31.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.137+0000 7f73a9a2f640 1 -- 172.21.15.60:0/382230216 wait complete. 2024-03-27T21:41:31.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.137+0000 7f73a9a2f640 10 monclient: build_initial_monmap 2024-03-27T21:41:31.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.137+0000 7f73a9a2f640 10 monclient: monmap: 2024-03-27T21:41:31.138 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:31.138 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:31.139 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:31.139 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:31.139 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:31.139 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:31.139 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:31.139 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:31.139 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:31.139 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:31.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.138+0000 7f73a9a2f640 1 librados: starting msgr at 2024-03-27T21:41:31.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.138+0000 7f73a9a2f640 1 librados: starting objecter 2024-03-27T21:41:31.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.138+0000 7f73a9a2f640 1 Processor -- start 2024-03-27T21:41:31.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.138+0000 7f73a9a2f640 1 -- start start 2024-03-27T21:41:31.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.138+0000 7f73a9a2f640 1 librados: setting wanted keys 2024-03-27T21:41:31.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.138+0000 7f73a9a2f640 1 librados: calling monclient init 2024-03-27T21:41:31.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.138+0000 7f73a9a2f640 10 monclient: init 2024-03-27T21:41:31.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.139+0000 7f73a9a2f640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:31.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.139+0000 7f73a9a2f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:41:31.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.139+0000 7f73a9a2f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73a40b40b0 0x7f73a413e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.139+0000 7f73a9a2f640 10 monclient(hunting): picked mon.noname-b con 0x7f73a40b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:31.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a9a2f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f73a413b020 0x7f73a41d9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a9a2f640 10 monclient(hunting): picked mon.noname-c con 0x7f73a413b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:31.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a27fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f73a413b020 0x7f73a41d9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a27fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f73a413b020 0x7f73a41d9800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:60338/0 (socket says 172.21.15.60:60338) 2024-03-27T21:41:31.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a27fc640 1 -- 172.21.15.60:0/3630295248 learned_addr learned my addr 172.21.15.60:0/3630295248 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:31.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a2ffd640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73a40b40b0 0x7f73a413e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a9a2f640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73a413b480 0x7f73a41d9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a9a2f640 10 monclient(hunting): picked mon.noname-a con 0x7f73a413b480 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:31.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a9a2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:31.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a40b0de0 con 0x7f73a413b480 2024-03-27T21:41:31.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a9a2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:31.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a4134fb0 con 0x7f73a40b40b0 2024-03-27T21:41:31.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a9a2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:31.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73a41da280 con 0x7f73a413b020 2024-03-27T21:41:31.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a9a2f640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:31.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a9a2f640 10 monclient(hunting): authenticate will time out at 932.361373s 2024-03-27T21:41:31.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a37fe640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73a413b480 0x7f73a41d9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a37fe640 10 monclient(hunting): get_auth_request con 0x7f73a413b480 auth_method 0 2024-03-27T21:41:31.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:31.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:31.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:31.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a2ffd640 10 monclient(hunting): get_auth_request con 0x7f73a40b40b0 auth_method 0 2024-03-27T21:41:31.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:31.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:31.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:31.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a27fc640 10 monclient(hunting): get_auth_request con 0x7f73a413b020 auth_method 0 2024-03-27T21:41:31.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:31.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:31.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:31.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:31.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:31.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.140+0000 7f73a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:31.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f73a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:31.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f73a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:31.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f73a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:31.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f73a27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:31.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f73a27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:31.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f73a27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:31.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f73a37fe640 10 monclient(hunting): handle_auth_done global_id 5022 payload 824 2024-03-27T21:41:31.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f73a37fe640 1 -- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f73a413b020 msgr2=0x7f73a41d9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f73a37fe640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f73a413b020 0x7f73a41d9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f73a37fe640 1 -- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73a40b40b0 msgr2=0x7f73a413e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f73a37fe640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73a40b40b0 0x7f73a413e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f73a37fe640 10 monclient: _finish_hunting 0 2024-03-27T21:41:31.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f73a37fe640 1 monclient: found mon.noname-a 2024-03-27T21:41:31.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f73a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:31.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f73a37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:31.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f73a37fe640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739c009a70 con 0x7f73a413b480 2024-03-27T21:41:31.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f73a27fc640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f73a413b020 0x7f73a41d9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:31.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f73a37fe640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73a413b480 0x7f73a41d9d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f739c033000 tx=0x7f739c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:31.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f738bfff640 10 client.?.objecter ms_handle_connect 0x7f73a413b480 2024-03-27T21:41:31.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f738bfff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:31.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c044070 con 0x7f73a413b480 2024-03-27T21:41:31.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f738bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:31.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f738bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:31.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f738bfff640 10 monclient: dump: 2024-03-27T21:41:31.145 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:31.145 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:31.145 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.145 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.145 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:31.145 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:31.146 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:31.146 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:31.146 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:31.146 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:31.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f738bfff640 10 monclient: _finish_auth 0 2024-03-27T21:41:31.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f738bfff640 10 monclient: _check_auth_tickets 2024-03-27T21:41:31.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f738bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:31.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739c009450 con 0x7f73a413b480 2024-03-27T21:41:31.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.141+0000 7f738bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:31.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.142+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c003b90 con 0x7f73a413b480 2024-03-27T21:41:31.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.142+0000 7f738bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:31.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.142+0000 7f738bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:31.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.142+0000 7f738bfff640 10 monclient: dump: 2024-03-27T21:41:31.146 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:31.147 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:31.147 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.147 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.147 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:31.147 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:31.147 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:31.147 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:31.147 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:31.147 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:31.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.142+0000 7f73a9a2f640 5 monclient: authenticate success, global_id 5022 2024-03-27T21:41:31.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.142+0000 7f73a17fa640 4 set_mon_vals no callback set 2024-03-27T21:41:31.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.142+0000 7f73a9a2f640 10 monclient: _renew_subs 2024-03-27T21:41:31.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.142+0000 7f73a9a2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:31.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.142+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a4135a10 con 0x7f73a413b480 2024-03-27T21:41:31.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.142+0000 7f73a9a2f640 10 client.5022.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:31.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.142+0000 7f73a9a2f640 10 monclient: _renew_subs 2024-03-27T21:41:31.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.142+0000 7f73a9a2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:31.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.142+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a4134d00 con 0x7f73a413b480 2024-03-27T21:41:31.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.142+0000 7f73a9a2f640 1 librados: init done 2024-03-27T21:41:31.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.143+0000 7f73a9a2f640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:41:31.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.143+0000 7f73a9a2f640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:31.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.143+0000 7f73a9a2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:31.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.143+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f73a4135170 con 0x7f73a413b480 2024-03-27T21:41:31.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.144+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f739c003cf0 con 0x7f73a413b480 2024-03-27T21:41:31.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.144+0000 7f738bfff640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f739006c590 0x7f739006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.144+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f739c0ab4e0 con 0x7f73a413b480 2024-03-27T21:41:31.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.144+0000 7f738bfff640 10 client.5022.objecter ms_dispatch 0x7f73a4000b90 osd_map(64..64 src has 1..64) v4 2024-03-27T21:41:31.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.144+0000 7f738bfff640 3 client.5022.objecter handle_osd_map got epochs [64,64] > 0 2024-03-27T21:41:31.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.144+0000 7f738bfff640 3 client.5022.objecter handle_osd_map decoding full epoch 64 2024-03-27T21:41:31.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.144+0000 7f73a2ffd640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f739006c590 0x7f739006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.144+0000 7f73a2ffd640 10 monclient: get_auth_request con 0x7f739006c590 auth_method 0 2024-03-27T21:41:31.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.144+0000 7f738bfff640 20 client.5022.objecter dump_active .. 0 homeless 2024-03-27T21:41:31.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.144+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v64) v1 ==== 54+0+16 (secure 0 0 0) 0x7f739c0b2070 con 0x7f73a413b480 2024-03-27T21:41:31.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.144+0000 7f738bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:31.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.144+0000 7f738bfff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:31.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.145+0000 7f738bfff640 10 client.5022.objecter ms_handle_connect 0x7f739006c590 2024-03-27T21:41:31.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.145+0000 7f73a2ffd640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f739006c590 0x7f739006ea50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7394000f60 tx=0x7f7394007d70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:31.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.145+0000 7f73a9a2f640 10 client.5022.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:31.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.145+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign -1 1 2024-03-27T21:41:31.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.145+0000 7f73a9a2f640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f73a4134010 0x7f73a412e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.145+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a4142340 osd=0 3 2024-03-27T21:41:31.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.145+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a4142340 osd=0 3 2024-03-27T21:41:31.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.145+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a4142340 osd=0 3 2024-03-27T21:41:31.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.145+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_remove -1 1 2024-03-27T21:41:31.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.145+0000 7f73a9a2f640 20 client.5022.objecter get_session s=0x7f73a4142340 osd=0 3 2024-03-27T21:41:31.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.145+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign 0 1 2024-03-27T21:41:31.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.145+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a4142340 osd=0 4 2024-03-27T21:41:31.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.145+0000 7f73a9a2f640 10 client.5022.objecter _send_command 1 2024-03-27T21:41:31.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.145+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73a40b4490 con 0x7f73a4134010 2024-03-27T21:41:31.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.145+0000 7f73a27fc640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f73a4134010 0x7f73a412e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.145+0000 7f73a27fc640 10 monclient: get_auth_request con 0x7f73a4134010 auth_method 0 2024-03-27T21:41:31.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.146+0000 7f73a27fc640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f73a4134010 0x7f73a412e230 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:31.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.146+0000 7f738bfff640 10 client.5022.objecter ms_handle_connect 0x7f73a4134010 2024-03-27T21:41:31.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.147+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f73980019e0 con 0x7f73a4134010 2024-03-27T21:41:31.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.147+0000 7f738bfff640 10 client.5022.objecter ms_dispatch 0x7f73a4000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:41:31.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.147+0000 7f738bfff640 10 client.5022.objecter _finish_command 1 = system:0 2024-03-27T21:41:31.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.147+0000 7f738bfff640 20 client.5022.objecter put_session s=0x7f73a4142340 osd=0 4 2024-03-27T21:41:31.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.147+0000 7f738bfff640 15 client.5022.objecter _session_command_op_remove 0 1 2024-03-27T21:41:31.152 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:31.166+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:31.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.176+0000 7f73a9a2f640 10 client.5022.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:31.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.176+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign -1 2 2024-03-27T21:41:31.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.176+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a4142340 osd=0 3 2024-03-27T21:41:31.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.176+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a4142340 osd=0 3 2024-03-27T21:41:31.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.176+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a4142340 osd=0 3 2024-03-27T21:41:31.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.176+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_remove -1 2 2024-03-27T21:41:31.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.176+0000 7f73a9a2f640 20 client.5022.objecter get_session s=0x7f73a4142340 osd=0 3 2024-03-27T21:41:31.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.176+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign 0 2 2024-03-27T21:41:31.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.176+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a4142340 osd=0 4 2024-03-27T21:41:31.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.176+0000 7f73a9a2f640 10 client.5022.objecter _send_command 2 2024-03-27T21:41:31.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.176+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f73a414cf40 con 0x7f73a4134010 2024-03-27T21:41:31.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.179+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f73980070c0 con 0x7f73a4134010 2024-03-27T21:41:31.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.179+0000 7f738bfff640 10 client.5022.objecter ms_dispatch 0x7f73a4000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:31.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.179+0000 7f738bfff640 10 client.5022.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:31.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.179+0000 7f738bfff640 20 client.5022.objecter put_session s=0x7f73a4142340 osd=0 4 2024-03-27T21:41:31.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.179+0000 7f738bfff640 15 client.5022.objecter _session_command_op_remove 0 2 2024-03-27T21:41:31.165 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:41:31.167 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:41:31.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.179+0000 7f73a9a2f640 10 client.5022.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:31.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.179+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign -1 3 2024-03-27T21:41:31.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.179+0000 7f73a9a2f640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f73a41327d0 0x7f73a4132bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.179+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a4140af0 osd=1 3 2024-03-27T21:41:31.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.179+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a4140af0 osd=1 3 2024-03-27T21:41:31.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.179+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a4140af0 osd=1 3 2024-03-27T21:41:31.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.179+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_remove -1 3 2024-03-27T21:41:31.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.179+0000 7f73a9a2f640 20 client.5022.objecter get_session s=0x7f73a4140af0 osd=1 3 2024-03-27T21:41:31.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.179+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign 1 3 2024-03-27T21:41:31.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.179+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a4140af0 osd=1 4 2024-03-27T21:41:31.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.179+0000 7f73a9a2f640 10 client.5022.objecter _send_command 3 2024-03-27T21:41:31.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.179+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73a4135ba0 con 0x7f73a41327d0 2024-03-27T21:41:31.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.179+0000 7f73a2ffd640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f73a41327d0 0x7f73a4132bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.179+0000 7f73a2ffd640 10 monclient: get_auth_request con 0x7f73a41327d0 auth_method 0 2024-03-27T21:41:31.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.181+0000 7f738bfff640 10 client.5022.objecter ms_handle_connect 0x7f73a41327d0 2024-03-27T21:41:31.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.181+0000 7f73a2ffd640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f73a41327d0 0x7f73a4132bb0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:31.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.184+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f73a4135ba0 con 0x7f73a41327d0 2024-03-27T21:41:31.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.184+0000 7f738bfff640 10 client.5022.objecter ms_dispatch 0x7f73a4000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:41:31.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.184+0000 7f738bfff640 10 client.5022.objecter _finish_command 3 = system:0 2024-03-27T21:41:31.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.184+0000 7f738bfff640 20 client.5022.objecter put_session s=0x7f73a4140af0 osd=1 4 2024-03-27T21:41:31.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.184+0000 7f738bfff640 15 client.5022.objecter _session_command_op_remove 1 3 2024-03-27T21:41:31.181 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:31.181 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:31.181 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:31.181 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:31.190 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:41:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.204+0000 7f73a9a2f640 10 client.5022.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.204+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign -1 4 2024-03-27T21:41:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.204+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a4140af0 osd=1 3 2024-03-27T21:41:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.204+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a4140af0 osd=1 3 2024-03-27T21:41:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.204+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a4140af0 osd=1 3 2024-03-27T21:41:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.204+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_remove -1 4 2024-03-27T21:41:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.204+0000 7f73a9a2f640 20 client.5022.objecter get_session s=0x7f73a4140af0 osd=1 3 2024-03-27T21:41:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.204+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign 1 4 2024-03-27T21:41:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.204+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a4140af0 osd=1 4 2024-03-27T21:41:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.204+0000 7f73a9a2f640 10 client.5022.objecter _send_command 4 2024-03-27T21:41:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.204+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f73a4069470 con 0x7f73a41327d0 2024-03-27T21:41:31.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.207+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f73a4069470 con 0x7f73a41327d0 2024-03-27T21:41:31.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.207+0000 7f738bfff640 10 client.5022.objecter ms_dispatch 0x7f73a4000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:31.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.207+0000 7f738bfff640 10 client.5022.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:31.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.207+0000 7f738bfff640 20 client.5022.objecter put_session s=0x7f73a4140af0 osd=1 4 2024-03-27T21:41:31.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.207+0000 7f738bfff640 15 client.5022.objecter _session_command_op_remove 1 4 2024-03-27T21:41:31.194 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:41:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:41:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.207+0000 7f73a9a2f640 10 client.5022.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.207+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign -1 5 2024-03-27T21:41:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.207+0000 7f73a9a2f640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f73a40b2620 0x7f73a402b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.207+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a41330f0 osd=2 3 2024-03-27T21:41:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.207+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a41330f0 osd=2 3 2024-03-27T21:41:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.207+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a41330f0 osd=2 3 2024-03-27T21:41:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.207+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_remove -1 5 2024-03-27T21:41:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.207+0000 7f73a9a2f640 20 client.5022.objecter get_session s=0x7f73a41330f0 osd=2 3 2024-03-27T21:41:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.207+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign 2 5 2024-03-27T21:41:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.207+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a41330f0 osd=2 4 2024-03-27T21:41:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.207+0000 7f73a9a2f640 10 client.5022.objecter _send_command 5 2024-03-27T21:41:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.207+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73a40a8950 con 0x7f73a40b2620 2024-03-27T21:41:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.207+0000 7f73a27fc640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f73a40b2620 0x7f73a402b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.207+0000 7f73a27fc640 10 monclient: get_auth_request con 0x7f73a40b2620 auth_method 0 2024-03-27T21:41:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.208+0000 7f73a27fc640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f73a40b2620 0x7f73a402b170 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.208+0000 7f738bfff640 10 client.5022.objecter ms_handle_connect 0x7f73a40b2620 2024-03-27T21:41:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.208+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== osd.2 v2:172.21.15.60:6824/56434 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f73a40a8950 con 0x7f73a40b2620 2024-03-27T21:41:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.208+0000 7f738bfff640 10 client.5022.objecter ms_dispatch 0x7f73a4000b90 command_reply(tid 5: 0 ) v1 2024-03-27T21:41:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.208+0000 7f738bfff640 10 client.5022.objecter _finish_command 5 = system:0 2024-03-27T21:41:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.208+0000 7f738bfff640 20 client.5022.objecter put_session s=0x7f73a41330f0 osd=2 4 2024-03-27T21:41:31.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.208+0000 7f738bfff640 15 client.5022.objecter _session_command_op_remove 2 5 2024-03-27T21:41:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.209+0000 7f1310b53640 10 monclient: get_monmap_and_config 2024-03-27T21:41:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.209+0000 7f1310b53640 10 monclient: build_initial_monmap 2024-03-27T21:41:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.209+0000 7f1310b53640 10 monclient: monmap: 2024-03-27T21:41:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:31.210092+0000 2024-03-27T21:41:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:31.210092+0000 2024-03-27T21:41:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:31.202 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:31.203 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.209+0000 7f1310b53640 1 Processor -- start 2024-03-27T21:41:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.210+0000 7f1310b53640 1 -- start start 2024-03-27T21:41:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.210+0000 7f1310b53640 10 monclient: init 2024-03-27T21:41:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f1310b53640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f1310b53640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-03-27T21:41:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f1310b53640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f130c0b40b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f1310b53640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f130c13b020 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f1310b53640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f130c13b480 0x7f130c141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f1310b53640 10 monclient(hunting): picked mon.noname-b con 0x7f130c13b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:31.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f1310b53640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f130c0b3830 con 0x7f130c0b40b0 2024-03-27T21:41:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f1310b53640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f130c0aa520 con 0x7f130c13b020 2024-03-27T21:41:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f1310b53640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f1310b53640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130c13b910 con 0x7f130c13b480 2024-03-27T21:41:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f1310b53640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f1310b53640 10 monclient(hunting): authenticate will time out at 932.432338s 2024-03-27T21:41:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130a59c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f130c0b40b0 0x7f130c0a8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56848/0 (socket says 172.21.15.60:56848) 2024-03-27T21:41:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130ad9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f130c13b480 0x7f130c141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130a59c640 1 -- 172.21.15.60:0/1424496738 learned_addr learned my addr 172.21.15.60:0/1424496738 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130ad9d640 10 monclient(hunting): get_auth_request con 0x7f130c13b480 auth_method 0 2024-03-27T21:41:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:31.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130959a640 1 -- 172.21.15.60:0/1424496738 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599724342 0 0) 0x7f130c0b3830 con 0x7f130c0b40b0 2024-03-27T21:41:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130959a640 10 monclient(hunting): _negotiate 2024-03-27T21:41:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130959a640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130959a640 10 monclient(hunting): my global_id is 5025 2024-03-27T21:41:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130959a640 1 -- 172.21.15.60:0/1424496738 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12f4003520 con 0x7f130c0b40b0 2024-03-27T21:41:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130959a640 1 -- 172.21.15.60:0/1424496738 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253006039 0 0) 0x7f130c0aa520 con 0x7f130c13b020 2024-03-27T21:41:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130959a640 10 monclient(hunting): _negotiate 2024-03-27T21:41:31.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130959a640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130959a640 10 monclient(hunting): my global_id is 4972 2024-03-27T21:41:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.211+0000 7f130959a640 1 -- 172.21.15.60:0/1424496738 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f130c0b3830 con 0x7f130c13b020 2024-03-27T21:41:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.212+0000 7f130ad9d640 10 monclient(hunting): handle_auth_done global_id 4805 payload 274 2024-03-27T21:41:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.212+0000 7f130ad9d640 1 -- 172.21.15.60:0/1424496738 >> v1:172.21.15.191:6789/0 conn(0x7f130c13b020 legacy=0x7f130c1416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.212+0000 7f130ad9d640 1 -- 172.21.15.60:0/1424496738 >> v1:172.21.15.60:6789/0 conn(0x7f130c0b40b0 legacy=0x7f130c0a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.212+0000 7f130ad9d640 10 monclient: _finish_hunting 0 2024-03-27T21:41:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.212+0000 7f130ad9d640 1 monclient: found mon.noname-b 2024-03-27T21:41:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.212+0000 7f130ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.212+0000 7f130ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.212+0000 7f130ad9d640 1 -- 172.21.15.60:0/1424496738 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130c0a9470 con 0x7f130c13b480 2024-03-27T21:41:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.213+0000 7f130ad9d640 1 --2- 172.21.15.60:0/1424496738 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f130c13b480 0x7f130c141e00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1304009dc0 tx=0x7f1304034730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fc96d458326338f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:31.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.213+0000 7f130959a640 1 -- 172.21.15.60:0/1424496738 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1304007740 con 0x7f130c13b480 2024-03-27T21:41:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.213+0000 7f130959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.213+0000 7f130959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.213+0000 7f130959a640 10 monclient: dump: 2024-03-27T21:41:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:31.207 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.213+0000 7f130959a640 10 monclient: _finish_auth 0 2024-03-27T21:41:31.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.213+0000 7f130959a640 10 monclient: _check_auth_tickets 2024-03-27T21:41:31.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.213+0000 7f130959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:31.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.213+0000 7f1310b53640 5 monclient: authenticate success, global_id 4805 2024-03-27T21:41:31.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.213+0000 7f1310b53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:31.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.213+0000 7f130959a640 1 -- 172.21.15.60:0/1424496738 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1304037050 con 0x7f130c13b480 2024-03-27T21:41:31.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.213+0000 7f130959a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:31.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.213+0000 7f1310b53640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:31.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.213+0000 7f1310b53640 4 set_mon_vals no callback set 2024-03-27T21:41:31.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.213+0000 7f130959a640 1 -- 172.21.15.60:0/1424496738 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1304007aa0 con 0x7f130c13b480 2024-03-27T21:41:31.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.213+0000 7f130959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:31.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.213+0000 7f130959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:31.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.213+0000 7f130959a640 10 monclient: dump: 2024-03-27T21:41:31.209 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:31.209 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:31.209 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.209 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.209 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:31.209 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:31.209 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:31.209 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:31.209 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:31.209 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:31.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.214+0000 7f1310b53640 10 monclient: shutdown 2024-03-27T21:41:31.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.214+0000 7f1310b53640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:31.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.214+0000 7f1310b53640 1 -- 172.21.15.60:0/1424496738 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f130c13b480 msgr2=0x7f130c141e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.214+0000 7f1310b53640 1 --2- 172.21.15.60:0/1424496738 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f130c13b480 0x7f130c141e00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1304009dc0 tx=0x7f1304034730 comp rx=0 tx=0).stop 2024-03-27T21:41:31.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.214+0000 7f1310b53640 1 -- 172.21.15.60:0/1424496738 shutdown_connections 2024-03-27T21:41:31.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.214+0000 7f1310b53640 1 --2- 172.21.15.60:0/1424496738 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f130c13b480 0x7f130c141e00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.215+0000 7f1310b53640 1 -- 172.21.15.60:0/1424496738 >> 172.21.15.60:0/1424496738 conn(0x7f130c0608e0 msgr2=0x7f130c060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:31.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.215+0000 7f1310b53640 1 -- 172.21.15.60:0/1424496738 shutdown_connections 2024-03-27T21:41:31.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.215+0000 7f1310b53640 1 -- 172.21.15.60:0/1424496738 wait complete. 2024-03-27T21:41:31.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.216+0000 7f1310b53640 10 monclient: build_initial_monmap 2024-03-27T21:41:31.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.216+0000 7f1310b53640 10 monclient: monmap: 2024-03-27T21:41:31.210 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:31.210 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:31.210 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:31.210 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:31.211 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:31.211 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:31.211 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:31.211 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:31.211 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:31.211 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:31.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.216+0000 7f1310b53640 1 librados: starting msgr at 2024-03-27T21:41:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.216+0000 7f1310b53640 1 librados: starting objecter 2024-03-27T21:41:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.216+0000 7f1310b53640 1 Processor -- start 2024-03-27T21:41:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.217+0000 7f1310b53640 1 -- start start 2024-03-27T21:41:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.217+0000 7f1310b53640 1 librados: setting wanted keys 2024-03-27T21:41:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.217+0000 7f1310b53640 1 librados: calling monclient init 2024-03-27T21:41:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.217+0000 7f1310b53640 10 monclient: init 2024-03-27T21:41:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.218+0000 7f1310b53640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.218+0000 7f1310b53640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:41:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.218+0000 7f1310b53640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f130c0b40b0 0x7f130c13e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.218+0000 7f1310b53640 10 monclient(hunting): picked mon.noname-b con 0x7f130c0b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.218+0000 7f1310b53640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f130c13b020 0x7f130c13c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.218+0000 7f1310b53640 10 monclient(hunting): picked mon.noname-a con 0x7f130c13b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.218+0000 7f1310b53640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f130c13b480 0x7f130c1360f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.218+0000 7f1310b53640 10 monclient(hunting): picked mon.noname-c con 0x7f130c13b480 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.218+0000 7f1310b53640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.218+0000 7f1310b53640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130c13d280 con 0x7f130c13b020 2024-03-27T21:41:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.218+0000 7f1310b53640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.218+0000 7f1309d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f130c13b020 0x7f130c13c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.218+0000 7f130ad9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f130c13b480 0x7f130c1360f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.218+0000 7f130ad9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f130c13b480 0x7f130c1360f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:60344/0 (socket says 172.21.15.60:60344) 2024-03-27T21:41:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.218+0000 7f130ad9d640 1 -- 172.21.15.60:0/222885876 learned_addr learned my addr 172.21.15.60:0/222885876 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.218+0000 7f130a59c640 1 --2- 172.21.15.60:0/222885876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f130c0b40b0 0x7f130c13e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.218+0000 7f1310b53640 1 -- 172.21.15.60:0/222885876 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130c06efa0 con 0x7f130c0b40b0 2024-03-27T21:41:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.218+0000 7f1310b53640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.218+0000 7f1310b53640 1 -- 172.21.15.60:0/222885876 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130c13bdf0 con 0x7f130c13b480 2024-03-27T21:41:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f1310b53640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f1310b53640 10 monclient(hunting): authenticate will time out at 932.440032s 2024-03-27T21:41:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f130ad9d640 10 monclient(hunting): get_auth_request con 0x7f130c13b480 auth_method 0 2024-03-27T21:41:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f130ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f130ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f130ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f1309d9b640 10 monclient(hunting): get_auth_request con 0x7f130c13b020 auth_method 0 2024-03-27T21:41:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f1309d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f1309d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f1309d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f1309d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f1309d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f1309d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:31.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f130a59c640 10 monclient(hunting): get_auth_request con 0x7f130c0b40b0 auth_method 0 2024-03-27T21:41:31.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f130a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:31.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f130a59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:31.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f130a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:31.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f130a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:31.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f130a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:31.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f130a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:31.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f1309d9b640 10 monclient(hunting): handle_auth_done global_id 5028 payload 824 2024-03-27T21:41:31.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f1309d9b640 1 -- 172.21.15.60:0/222885876 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f130c13b480 msgr2=0x7f130c1360f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f1309d9b640 1 --2- 172.21.15.60:0/222885876 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f130c13b480 0x7f130c1360f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f1309d9b640 1 -- 172.21.15.60:0/222885876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f130c0b40b0 msgr2=0x7f130c13e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f1309d9b640 1 --2- 172.21.15.60:0/222885876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f130c0b40b0 0x7f130c13e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f1309d9b640 10 monclient: _finish_hunting 0 2024-03-27T21:41:31.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f1309d9b640 1 monclient: found mon.noname-a 2024-03-27T21:41:31.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f1309d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:31.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f1309d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:31.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f1309d9b640 1 -- 172.21.15.60:0/222885876 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1304009a70 con 0x7f130c13b020 2024-03-27T21:41:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f130ad9d640 1 --2- 172.21.15.60:0/222885876 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f130c13b480 0x7f130c1360f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:41:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.219+0000 7f1309d9b640 1 --2- 172.21.15.60:0/222885876 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f130c13b020 0x7f130c13c8b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1300000f60 tx=0x7f130000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f12faffd640 10 client.?.objecter ms_handle_connect 0x7f130c13b020 2024-03-27T21:41:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f12faffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f12faffd640 1 -- 172.21.15.60:0/222885876 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13000050d0 con 0x7f130c13b020 2024-03-27T21:41:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f12faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f12faffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f12faffd640 10 monclient: dump: 2024-03-27T21:41:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:31.227 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f12faffd640 10 monclient: _finish_auth 0 2024-03-27T21:41:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f12faffd640 10 monclient: _check_auth_tickets 2024-03-27T21:41:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f12faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:31.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f12faffd640 1 -- 172.21.15.60:0/222885876 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1300005270 con 0x7f130c13b020 2024-03-27T21:41:31.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f12faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:31.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f12faffd640 1 -- 172.21.15.60:0/222885876 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130000a640 con 0x7f130c13b020 2024-03-27T21:41:31.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f12faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:31.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f12faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:31.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f12faffd640 10 monclient: dump: 2024-03-27T21:41:31.228 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:31.228 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:31.228 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.228 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.228 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:31.228 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:31.228 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:31.228 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:31.229 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:31.229 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:31.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f1310b53640 5 monclient: authenticate success, global_id 5028 2024-03-27T21:41:31.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f1310b53640 10 monclient: _renew_subs 2024-03-27T21:41:31.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f1310b53640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:31.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f1308d99640 4 set_mon_vals no callback set 2024-03-27T21:41:31.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f1310b53640 1 -- 172.21.15.60:0/222885876 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f130c13cdf0 con 0x7f130c13b020 2024-03-27T21:41:31.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f1310b53640 10 client.5028.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:31.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f1310b53640 10 monclient: _renew_subs 2024-03-27T21:41:31.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f1310b53640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:31.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f1310b53640 1 -- 172.21.15.60:0/222885876 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130c136810 con 0x7f130c13b020 2024-03-27T21:41:31.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.220+0000 7f1310b53640 1 librados: init done 2024-03-27T21:41:31.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.221+0000 7f1310b53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:31.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.221+0000 7f1310b53640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:31.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.221+0000 7f1310b53640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:31.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.221+0000 7f1310b53640 1 -- 172.21.15.60:0/222885876 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f130c1d1c80 con 0x7f130c13b020 2024-03-27T21:41:31.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.222+0000 7f12faffd640 1 -- 172.21.15.60:0/222885876 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f130000a7a0 con 0x7f130c13b020 2024-03-27T21:41:31.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.223+0000 7f12faffd640 1 --2- 172.21.15.60:0/222885876 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f12ec06c660 0x7f12ec06eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.223+0000 7f12faffd640 1 -- 172.21.15.60:0/222885876 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f1300088b80 con 0x7f130c13b020 2024-03-27T21:41:31.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.223+0000 7f12faffd640 10 client.5028.objecter ms_dispatch 0x7f130c000b90 osd_map(64..64 src has 1..64) v4 2024-03-27T21:41:31.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.223+0000 7f12faffd640 3 client.5028.objecter handle_osd_map got epochs [64,64] > 0 2024-03-27T21:41:31.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.223+0000 7f12faffd640 3 client.5028.objecter handle_osd_map decoding full epoch 64 2024-03-27T21:41:31.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.223+0000 7f12faffd640 20 client.5028.objecter dump_active .. 0 homeless 2024-03-27T21:41:31.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.223+0000 7f130a59c640 1 --2- 172.21.15.60:0/222885876 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f12ec06c660 0x7f12ec06eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.223+0000 7f130a59c640 10 monclient: get_auth_request con 0x7f12ec06c660 auth_method 0 2024-03-27T21:41:31.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.224+0000 7f130a59c640 1 --2- 172.21.15.60:0/222885876 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f12ec06c660 0x7f12ec06eb20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f12fc000f60 tx=0x7f12fc007d70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:31.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.224+0000 7f12faffd640 10 client.5028.objecter ms_handle_connect 0x7f12ec06c660 2024-03-27T21:41:31.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.229+0000 7f73a9a2f640 10 client.5022.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:31.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.229+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign -1 6 2024-03-27T21:41:31.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.229+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a41330f0 osd=2 3 2024-03-27T21:41:31.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.229+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a41330f0 osd=2 3 2024-03-27T21:41:31.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.229+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a41330f0 osd=2 3 2024-03-27T21:41:31.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.229+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_remove -1 6 2024-03-27T21:41:31.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.229+0000 7f73a9a2f640 20 client.5022.objecter get_session s=0x7f73a41330f0 osd=2 3 2024-03-27T21:41:31.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.229+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign 2 6 2024-03-27T21:41:31.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.229+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a41330f0 osd=2 4 2024-03-27T21:41:31.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.229+0000 7f73a9a2f640 10 client.5022.objecter _send_command 6 2024-03-27T21:41:31.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.229+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f73a40a8ac0 con 0x7f73a40b2620 2024-03-27T21:41:31.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.230+0000 7f12faffd640 1 -- 172.21.15.60:0/222885876 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1300053fa0 con 0x7f130c13b020 2024-03-27T21:41:31.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.230+0000 7f12faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:31.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.230+0000 7f12faffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:31.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.232+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== osd.2 v2:172.21.15.60:6824/56434 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f73a40a8ac0 con 0x7f73a40b2620 2024-03-27T21:41:31.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.232+0000 7f738bfff640 10 client.5022.objecter ms_dispatch 0x7f73a4000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:31.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.232+0000 7f738bfff640 10 client.5022.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:31.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.232+0000 7f738bfff640 20 client.5022.objecter put_session s=0x7f73a41330f0 osd=2 4 2024-03-27T21:41:31.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.232+0000 7f738bfff640 15 client.5022.objecter _session_command_op_remove 2 6 2024-03-27T21:41:31.233 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2: {} 2024-03-27T21:41:31.235 INFO:teuthology.orchestra.run.smithi060.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-27T21:41:31.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.232+0000 7f73a9a2f640 10 client.5022.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:31.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.232+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign -1 7 2024-03-27T21:41:31.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.232+0000 7f73a9a2f640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f73a402c380 0x7f73a41e8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.232+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a40b1dd0 osd=3 3 2024-03-27T21:41:31.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.232+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a40b1dd0 osd=3 3 2024-03-27T21:41:31.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.232+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a40b1dd0 osd=3 3 2024-03-27T21:41:31.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.232+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_remove -1 7 2024-03-27T21:41:31.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.232+0000 7f73a9a2f640 20 client.5022.objecter get_session s=0x7f73a40b1dd0 osd=3 3 2024-03-27T21:41:31.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.232+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign 3 7 2024-03-27T21:41:31.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.232+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a40b1dd0 osd=3 4 2024-03-27T21:41:31.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.232+0000 7f73a9a2f640 10 client.5022.objecter _send_command 7 2024-03-27T21:41:31.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.232+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73a40b1f40 con 0x7f73a402c380 2024-03-27T21:41:31.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.233+0000 7f73a37fe640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f73a402c380 0x7f73a41e8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.233+0000 7f73a37fe640 10 monclient: get_auth_request con 0x7f73a402c380 auth_method 0 2024-03-27T21:41:31.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.233+0000 7f73a37fe640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f73a402c380 0x7f73a41e8d80 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:31.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.233+0000 7f738bfff640 10 client.5022.objecter ms_handle_connect 0x7f73a402c380 2024-03-27T21:41:31.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.234+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f73a40b1f40 con 0x7f73a402c380 2024-03-27T21:41:31.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.234+0000 7f738bfff640 10 client.5022.objecter ms_dispatch 0x7f73a4000b90 command_reply(tid 7: 0 ) v1 2024-03-27T21:41:31.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.234+0000 7f738bfff640 10 client.5022.objecter _finish_command 7 = system:0 2024-03-27T21:41:31.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.234+0000 7f738bfff640 20 client.5022.objecter put_session s=0x7f73a40b1dd0 osd=3 4 2024-03-27T21:41:31.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.234+0000 7f738bfff640 15 client.5022.objecter _session_command_op_remove 3 7 2024-03-27T21:41:31.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.262+0000 7f73a9a2f640 10 client.5022.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.262+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign -1 8 2024-03-27T21:41:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.262+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a40b1dd0 osd=3 3 2024-03-27T21:41:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.262+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a40b1dd0 osd=3 3 2024-03-27T21:41:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.262+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a40b1dd0 osd=3 3 2024-03-27T21:41:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.262+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_remove -1 8 2024-03-27T21:41:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.262+0000 7f73a9a2f640 20 client.5022.objecter get_session s=0x7f73a40b1dd0 osd=3 3 2024-03-27T21:41:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.262+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign 3 8 2024-03-27T21:41:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.262+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a40b1dd0 osd=3 4 2024-03-27T21:41:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.262+0000 7f73a9a2f640 10 client.5022.objecter _send_command 8 2024-03-27T21:41:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.262+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f73a402b8f0 con 0x7f73a402c380 2024-03-27T21:41:31.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.264+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f73a402b8f0 con 0x7f73a402c380 2024-03-27T21:41:31.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.264+0000 7f738bfff640 10 client.5022.objecter ms_dispatch 0x7f73a4000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:31.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.264+0000 7f738bfff640 10 client.5022.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:31.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.264+0000 7f738bfff640 20 client.5022.objecter put_session s=0x7f73a40b1dd0 osd=3 4 2024-03-27T21:41:31.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.264+0000 7f738bfff640 15 client.5022.objecter _session_command_op_remove 3 8 2024-03-27T21:41:31.251 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:41:31.252 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:41:31.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.264+0000 7f73a9a2f640 10 client.5022.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:31.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.264+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign -1 9 2024-03-27T21:41:31.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.264+0000 7f73a9a2f640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f73a402ba60 0x7f73a41ec450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.264+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a402b8f0 osd=4 3 2024-03-27T21:41:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.264+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a402b8f0 osd=4 3 2024-03-27T21:41:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.264+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a402b8f0 osd=4 3 2024-03-27T21:41:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.264+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_remove -1 9 2024-03-27T21:41:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.264+0000 7f73a9a2f640 20 client.5022.objecter get_session s=0x7f73a402b8f0 osd=4 3 2024-03-27T21:41:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.264+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign 4 9 2024-03-27T21:41:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.264+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a402b8f0 osd=4 4 2024-03-27T21:41:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.264+0000 7f73a9a2f640 10 client.5022.objecter _send_command 9 2024-03-27T21:41:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.264+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73a4066760 con 0x7f73a402ba60 2024-03-27T21:41:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.264+0000 7f73a2ffd640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f73a402ba60 0x7f73a41ec450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.264+0000 7f73a2ffd640 10 monclient: get_auth_request con 0x7f73a402ba60 auth_method 0 2024-03-27T21:41:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.265+0000 7f738bfff640 10 client.5022.objecter ms_handle_connect 0x7f73a402ba60 2024-03-27T21:41:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.265+0000 7f73a2ffd640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f73a402ba60 0x7f73a41ec450 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.265+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f73a4066760 con 0x7f73a402ba60 2024-03-27T21:41:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.266+0000 7f738bfff640 10 client.5022.objecter ms_dispatch 0x7f73a4000b90 command_reply(tid 9: 0 ) v1 2024-03-27T21:41:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.266+0000 7f738bfff640 10 client.5022.objecter _finish_command 9 = system:0 2024-03-27T21:41:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.266+0000 7f738bfff640 20 client.5022.objecter put_session s=0x7f73a402b8f0 osd=4 4 2024-03-27T21:41:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.266+0000 7f738bfff640 15 client.5022.objecter _session_command_op_remove 4 9 2024-03-27T21:41:31.255 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:31.269+0000 7fb850f34640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 56426) UID: 0 2024-03-27T21:41:31.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.279+0000 7f1003bab640 10 monclient: get_monmap_and_config 2024-03-27T21:41:31.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.279+0000 7f1003bab640 10 monclient: build_initial_monmap 2024-03-27T21:41:31.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.279+0000 7f1003bab640 10 monclient: monmap: 2024-03-27T21:41:31.266 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:31.266 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:31.266 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:31.280498+0000 2024-03-27T21:41:31.266 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:31.280498+0000 2024-03-27T21:41:31.267 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:31.267 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:31.267 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:31.267 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:31.267 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:31.267 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:31.267 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:31.267 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:31.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.280+0000 7f1003bab640 1 Processor -- start 2024-03-27T21:41:31.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.280+0000 7f1003bab640 1 -- start start 2024-03-27T21:41:31.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.280+0000 7f1003bab640 10 monclient: init 2024-03-27T21:41:31.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1003bab640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:31.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1003bab640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-03-27T21:41:31.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1003bab640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0ffc0b40b0 0x7f0ffc0a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1003bab640 10 monclient(hunting): picked mon.noname-c con 0x7f0ffc0b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:31.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1003bab640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0ffc13b020 0x7f0ffc13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1003bab640 10 monclient(hunting): picked mon.noname-b con 0x7f0ffc13b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:31.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1003bab640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0ffc1416f0 0x7f0ffc141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1003bab640 10 monclient(hunting): picked mon.noname-a con 0x7f0ffc1416f0 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:31.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1003bab640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:31.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1003bab640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ffc0b4490 con 0x7f0ffc1416f0 2024-03-27T21:41:31.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1003bab640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:31.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1003bab640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ffc0b3830 con 0x7f0ffc0b40b0 2024-03-27T21:41:31.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1003bab640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:31.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1003bab640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc0aa520 con 0x7f0ffc13b020 2024-03-27T21:41:31.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1002148640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0ffc1416f0 0x7f0ffc141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1002148640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0ffc1416f0 0x7f0ffc141b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36580/0 (socket says 172.21.15.60:36580) 2024-03-27T21:41:31.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1002148640 1 -- 172.21.15.60:0/300452080 learned_addr learned my addr 172.21.15.60:0/300452080 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:31.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1001947640 1 --2- 172.21.15.60:0/300452080 >> v2:172.21.15.191:3300/0 conn(0x7f0ffc0b40b0 0x7f0ffc0a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1001146640 1 --2- 172.21.15.60:0/300452080 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0ffc13b020 0x7f0ffc13b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1003bab640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:31.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1003bab640 10 monclient(hunting): authenticate will time out at 932.502410s 2024-03-27T21:41:31.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1002148640 10 monclient(hunting): get_auth_request con 0x7f0ffc1416f0 auth_method 0 2024-03-27T21:41:31.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1002148640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:31.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1002148640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:31.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1002148640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:31.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1001947640 10 monclient(hunting): get_auth_request con 0x7f0ffc0b40b0 auth_method 0 2024-03-27T21:41:31.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1001947640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:31.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1001947640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:31.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1001947640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:31.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1001146640 10 monclient(hunting): get_auth_request con 0x7f0ffc13b020 auth_method 0 2024-03-27T21:41:31.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1001146640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:31.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1001146640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:31.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1001146640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:31.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1002148640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:31.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1002148640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:31.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1002148640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:31.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1001947640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:31.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1001947640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:31.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1001947640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:31.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1001146640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:31.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1001146640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:31.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1001146640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:31.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1002148640 10 monclient(hunting): handle_auth_done global_id 5031 payload 274 2024-03-27T21:41:31.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1002148640 1 -- 172.21.15.60:0/300452080 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0ffc13b020 msgr2=0x7f0ffc13b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:31.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1002148640 1 --2- 172.21.15.60:0/300452080 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0ffc13b020 0x7f0ffc13b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1002148640 1 -- 172.21.15.60:0/300452080 >> v2:172.21.15.191:3300/0 conn(0x7f0ffc0b40b0 msgr2=0x7f0ffc0a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:31.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1002148640 1 --2- 172.21.15.60:0/300452080 >> v2:172.21.15.191:3300/0 conn(0x7f0ffc0b40b0 0x7f0ffc0a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1002148640 10 monclient: _finish_hunting 0 2024-03-27T21:41:31.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1002148640 1 monclient: found mon.noname-a 2024-03-27T21:41:31.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1002148640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:31.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1002148640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:31.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.281+0000 7f1002148640 1 -- 172.21.15.60:0/300452080 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ffc0a9470 con 0x7f0ffc1416f0 2024-03-27T21:41:31.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.283+0000 7f1002148640 1 --2- 172.21.15.60:0/300452080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ffc1416f0 0x7f0ffc141b80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc0b4810 tx=0x7f0ff402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d5bd6115b182b82 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:31.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.283+0000 7f1000945640 1 -- 172.21.15.60:0/300452080 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff402ca80 con 0x7f0ffc1416f0 2024-03-27T21:41:31.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.283+0000 7f1000945640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:31.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.283+0000 7f1000945640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:31.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.283+0000 7f1000945640 10 monclient: dump: 2024-03-27T21:41:31.281 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:31.281 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:31.281 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.281 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.281 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:31.281 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:31.281 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:31.281 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:31.281 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:31.281 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:31.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.283+0000 7f1000945640 10 monclient: _finish_auth 0 2024-03-27T21:41:31.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.283+0000 7f1000945640 10 monclient: _check_auth_tickets 2024-03-27T21:41:31.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.283+0000 7f1000945640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:31.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.283+0000 7f1000945640 1 -- 172.21.15.60:0/300452080 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff402cbe0 con 0x7f0ffc1416f0 2024-03-27T21:41:31.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.283+0000 7f1000945640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:31.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.283+0000 7f1000945640 1 -- 172.21.15.60:0/300452080 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff40343f0 con 0x7f0ffc1416f0 2024-03-27T21:41:31.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.283+0000 7f1000945640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:31.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.283+0000 7f1000945640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:31.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.283+0000 7f1000945640 10 monclient: dump: 2024-03-27T21:41:31.282 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:31.282 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:31.282 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.282 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.282 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:31.282 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:31.282 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:31.282 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:31.283 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:31.283 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:31.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.283+0000 7f1003bab640 5 monclient: authenticate success, global_id 5031 2024-03-27T21:41:31.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.283+0000 7f1003bab640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:31.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.283+0000 7f1003bab640 4 set_mon_vals no callback set 2024-03-27T21:41:31.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.284+0000 7f1003bab640 10 monclient: shutdown 2024-03-27T21:41:31.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.284+0000 7f1003bab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:31.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.284+0000 7f1003bab640 1 -- 172.21.15.60:0/300452080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ffc1416f0 msgr2=0x7f0ffc141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.284+0000 7f1003bab640 1 --2- 172.21.15.60:0/300452080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ffc1416f0 0x7f0ffc141b80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc0b4810 tx=0x7f0ff402bee0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.284+0000 7f1003bab640 1 -- 172.21.15.60:0/300452080 shutdown_connections 2024-03-27T21:41:31.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.284+0000 7f1003bab640 1 --2- 172.21.15.60:0/300452080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ffc1416f0 0x7f0ffc141b80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.284+0000 7f1003bab640 1 --2- 172.21.15.60:0/300452080 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0ffc13b020 0x7f0ffc13b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.284+0000 7f1003bab640 1 --2- 172.21.15.60:0/300452080 >> v2:172.21.15.191:3300/0 conn(0x7f0ffc0b40b0 0x7f0ffc0a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.284+0000 7f1003bab640 1 -- 172.21.15.60:0/300452080 >> 172.21.15.60:0/300452080 conn(0x7f0ffc0608e0 msgr2=0x7f0ffc060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:31.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.284+0000 7f1003bab640 1 -- 172.21.15.60:0/300452080 shutdown_connections 2024-03-27T21:41:31.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.284+0000 7f1003bab640 1 -- 172.21.15.60:0/300452080 wait complete. 2024-03-27T21:41:31.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.285+0000 7f1003bab640 10 monclient: build_initial_monmap 2024-03-27T21:41:31.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.285+0000 7f1003bab640 10 monclient: monmap: 2024-03-27T21:41:31.284 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:31.284 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:31.284 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:31.284 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:31.284 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:31.284 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:31.284 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:31.284 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:31.284 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:31.284 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:31.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.286+0000 7f1003bab640 1 librados: starting msgr at 2024-03-27T21:41:31.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.286+0000 7f1003bab640 1 librados: starting objecter 2024-03-27T21:41:31.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.286+0000 7f1003bab640 1 Processor -- start 2024-03-27T21:41:31.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.286+0000 7f1003bab640 1 -- start start 2024-03-27T21:41:31.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.286+0000 7f1003bab640 1 librados: setting wanted keys 2024-03-27T21:41:31.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.286+0000 7f1003bab640 1 librados: calling monclient init 2024-03-27T21:41:31.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.286+0000 7f1003bab640 10 monclient: init 2024-03-27T21:41:31.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.287+0000 7f1003bab640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:31.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.287+0000 7f1003bab640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:41:31.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.287+0000 7f1003bab640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ffc0b40b0 0x7f0ffc1ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.287+0000 7f1003bab640 10 monclient(hunting): picked mon.noname-a con 0x7f0ffc0b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:31.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.287+0000 7f1001947640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ffc0b40b0 0x7f0ffc1ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.287+0000 7f1003bab640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ffc13b020 0x7f0ffc1d9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.287+0000 7f1003bab640 10 monclient(hunting): picked mon.noname-b con 0x7f0ffc13b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:31.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.287+0000 7f1003bab640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0ffc1416f0 0x7f0ffc13d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.287+0000 7f1003bab640 10 monclient(hunting): picked mon.noname-c con 0x7f0ffc1416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:31.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.287+0000 7f1003bab640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:31.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.287+0000 7f1003bab640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc0ab530 con 0x7f0ffc0b40b0 2024-03-27T21:41:31.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.287+0000 7f1002148640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0ffc1416f0 0x7f0ffc13d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.287+0000 7f1002148640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0ffc1416f0 0x7f0ffc13d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:60356/0 (socket says 172.21.15.60:60356) 2024-03-27T21:41:31.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.287+0000 7f1002148640 1 -- 172.21.15.60:0/4106246441 learned_addr learned my addr 172.21.15.60:0/4106246441 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:31.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.287+0000 7f1003bab640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:31.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.287+0000 7f1003bab640 1 -- 172.21.15.60:0/4106246441 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc1d9f90 con 0x7f0ffc13b020 2024-03-27T21:41:31.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.287+0000 7f1001146640 1 --2- 172.21.15.60:0/4106246441 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ffc13b020 0x7f0ffc1d9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.287+0000 7f1003bab640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:31.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.287+0000 7f1003bab640 1 -- 172.21.15.60:0/4106246441 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc1da150 con 0x7f0ffc1416f0 2024-03-27T21:41:31.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.287+0000 7f1003bab640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:31.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.287+0000 7f1003bab640 10 monclient(hunting): authenticate will time out at 932.508908s 2024-03-27T21:41:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1002148640 10 monclient(hunting): get_auth_request con 0x7f0ffc1416f0 auth_method 0 2024-03-27T21:41:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1002148640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1002148640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1002148640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001947640 10 monclient(hunting): get_auth_request con 0x7f0ffc0b40b0 auth_method 0 2024-03-27T21:41:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001947640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001947640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001947640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001146640 10 monclient(hunting): get_auth_request con 0x7f0ffc13b020 auth_method 0 2024-03-27T21:41:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001146640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001146640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001146640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1002148640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1002148640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:31.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1002148640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:31.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001146640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:31.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001146640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:31.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001146640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:31.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001947640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:31.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001947640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:31.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001947640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:31.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001146640 10 monclient(hunting): handle_auth_done global_id 4981 payload 824 2024-03-27T21:41:31.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001146640 1 -- 172.21.15.60:0/4106246441 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0ffc1416f0 msgr2=0x7f0ffc13d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001146640 1 --2- 172.21.15.60:0/4106246441 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0ffc1416f0 0x7f0ffc13d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001146640 1 -- 172.21.15.60:0/4106246441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ffc0b40b0 msgr2=0x7f0ffc1ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001146640 1 --2- 172.21.15.60:0/4106246441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ffc0b40b0 0x7f0ffc1ca1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001146640 10 monclient: _finish_hunting 0 2024-03-27T21:41:31.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001146640 1 monclient: found mon.noname-b 2024-03-27T21:41:31.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001146640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:31.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001146640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:41:31.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001146640 1 -- 172.21.15.60:0/4106246441 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff4009b30 con 0x7f0ffc13b020 2024-03-27T21:41:31.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1001146640 1 --2- 172.21.15.60:0/4106246441 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ffc13b020 0x7f0ffc1d9a50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0fec00e850 tx=0x7f0fec00ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:31.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f0ff27fc640 10 client.?.objecter ms_handle_connect 0x7f0ffc13b020 2024-03-27T21:41:31.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f0ff27fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:31.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f0ff27fc640 1 -- 172.21.15.60:0/4106246441 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fec00c940 con 0x7f0ffc13b020 2024-03-27T21:41:31.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f0ff27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:31.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f0ff27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:31.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f0ff27fc640 10 monclient: dump: 2024-03-27T21:41:31.289 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:31.289 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:31.289 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.289 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.289 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:31.289 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:31.290 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:31.290 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:31.290 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:31.290 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:31.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f0ff27fc640 10 monclient: _finish_auth 0 2024-03-27T21:41:31.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f0ff27fc640 10 monclient: _check_auth_tickets 2024-03-27T21:41:31.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f0ff27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:31.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f0ff27fc640 1 -- 172.21.15.60:0/4106246441 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fec00caa0 con 0x7f0ffc13b020 2024-03-27T21:41:31.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f0ff27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:31.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f0ff27fc640 1 -- 172.21.15.60:0/4106246441 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fec00ce10 con 0x7f0ffc13b020 2024-03-27T21:41:31.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f0ff27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:31.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f0ff27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:31.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f0ff27fc640 10 monclient: dump: 2024-03-27T21:41:31.290 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:31.291 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f0ff3fff640 4 set_mon_vals no callback set 2024-03-27T21:41:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.288+0000 7f1003bab640 5 monclient: authenticate success, global_id 4981 2024-03-27T21:41:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.289+0000 7f1003bab640 10 monclient: _renew_subs 2024-03-27T21:41:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.289+0000 7f1003bab640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.289+0000 7f1003bab640 1 -- 172.21.15.60:0/4106246441 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ffc13dcf0 con 0x7f0ffc13b020 2024-03-27T21:41:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.289+0000 7f1003bab640 10 client.4981.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:31.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.289+0000 7f1003bab640 10 monclient: _renew_subs 2024-03-27T21:41:31.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.289+0000 7f1003bab640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:31.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.289+0000 7f1003bab640 1 -- 172.21.15.60:0/4106246441 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ffc13e150 con 0x7f0ffc13b020 2024-03-27T21:41:31.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.289+0000 7f1003bab640 1 librados: init done 2024-03-27T21:41:31.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.290+0000 7f1003bab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:31.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.290+0000 7f1003bab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:31.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.290+0000 7f1003bab640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:31.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.290+0000 7f1003bab640 1 -- 172.21.15.60:0/4106246441 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ffc069470 con 0x7f0ffc13b020 2024-03-27T21:41:31.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.291+0000 7f0ff27fc640 1 -- 172.21.15.60:0/4106246441 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0fec013470 con 0x7f0ffc13b020 2024-03-27T21:41:31.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.291+0000 7f73a9a2f640 10 client.5022.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:31.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.291+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign -1 10 2024-03-27T21:41:31.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.291+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a402b8f0 osd=4 3 2024-03-27T21:41:31.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.291+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a402b8f0 osd=4 3 2024-03-27T21:41:31.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.291+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a402b8f0 osd=4 3 2024-03-27T21:41:31.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.291+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_remove -1 10 2024-03-27T21:41:31.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.291+0000 7f73a9a2f640 20 client.5022.objecter get_session s=0x7f73a402b8f0 osd=4 3 2024-03-27T21:41:31.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.291+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign 4 10 2024-03-27T21:41:31.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.291+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a402b8f0 osd=4 4 2024-03-27T21:41:31.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.291+0000 7f73a9a2f640 10 client.5022.objecter _send_command 10 2024-03-27T21:41:31.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.291+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f73a4066760 con 0x7f73a402ba60 2024-03-27T21:41:31.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.291+0000 7f0ff27fc640 1 --2- 172.21.15.60:0/4106246441 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f0fe006c660 0x7f0fe006eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.291+0000 7f0ff27fc640 1 -- 172.21.15.60:0/4106246441 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f0fec089190 con 0x7f0ffc13b020 2024-03-27T21:41:31.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.291+0000 7f0ff27fc640 10 client.4981.objecter ms_dispatch 0x7f0ffc000b90 osd_map(64..64 src has 1..64) v4 2024-03-27T21:41:31.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.291+0000 7f0ff27fc640 3 client.4981.objecter handle_osd_map got epochs [64,64] > 0 2024-03-27T21:41:31.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.291+0000 7f0ff27fc640 3 client.4981.objecter handle_osd_map decoding full epoch 64 2024-03-27T21:41:31.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.291+0000 7f0ff27fc640 20 client.4981.objecter dump_active .. 0 homeless 2024-03-27T21:41:31.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.292+0000 7f1001947640 1 --2- 172.21.15.60:0/4106246441 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f0fe006c660 0x7f0fe006eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.292+0000 7f1001947640 10 monclient: get_auth_request con 0x7f0fe006c660 auth_method 0 2024-03-27T21:41:31.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.292+0000 7f0ff27fc640 10 client.4981.objecter ms_handle_connect 0x7f0fe006c660 2024-03-27T21:41:31.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.292+0000 7f1001947640 1 --2- 172.21.15.60:0/4106246441 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f0fe006c660 0x7f0fe006eb20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8009b40 tx=0x7f0ff8009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:31.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.294+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f73a4066760 con 0x7f73a402ba60 2024-03-27T21:41:31.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.294+0000 7f738bfff640 10 client.5022.objecter ms_dispatch 0x7f73a4000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:31.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.294+0000 7f738bfff640 10 client.5022.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:31.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.294+0000 7f738bfff640 20 client.5022.objecter put_session s=0x7f73a402b8f0 osd=4 4 2024-03-27T21:41:31.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.294+0000 7f738bfff640 15 client.5022.objecter _session_command_op_remove 4 10 2024-03-27T21:41:31.296 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:41:31.298 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:41:31.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.294+0000 7f73a9a2f640 10 client.5022.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:31.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.294+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign -1 11 2024-03-27T21:41:31.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.294+0000 7f73a9a2f640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f73a41ecea0 0x7f73a41ef290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.294+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a4066760 osd=5 3 2024-03-27T21:41:31.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.294+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a4066760 osd=5 3 2024-03-27T21:41:31.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.294+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a4066760 osd=5 3 2024-03-27T21:41:31.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.294+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_remove -1 11 2024-03-27T21:41:31.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.294+0000 7f73a9a2f640 20 client.5022.objecter get_session s=0x7f73a4066760 osd=5 3 2024-03-27T21:41:31.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.294+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign 5 11 2024-03-27T21:41:31.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.294+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a4066760 osd=5 4 2024-03-27T21:41:31.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.294+0000 7f73a9a2f640 10 client.5022.objecter _send_command 11 2024-03-27T21:41:31.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.294+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73a4066950 con 0x7f73a41ecea0 2024-03-27T21:41:31.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.294+0000 7f73a27fc640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f73a41ecea0 0x7f73a41ef290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.295+0000 7f73a27fc640 10 monclient: get_auth_request con 0x7f73a41ecea0 auth_method 0 2024-03-27T21:41:31.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.295+0000 7f73a27fc640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f73a41ecea0 0x7f73a41ef290 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:31.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.295+0000 7f738bfff640 10 client.5022.objecter ms_handle_connect 0x7f73a41ecea0 2024-03-27T21:41:31.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.296+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f73a4066950 con 0x7f73a41ecea0 2024-03-27T21:41:31.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.296+0000 7f738bfff640 10 client.5022.objecter ms_dispatch 0x7f73a4000b90 command_reply(tid 11: 0 ) v1 2024-03-27T21:41:31.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.296+0000 7f738bfff640 10 client.5022.objecter _finish_command 11 = system:0 2024-03-27T21:41:31.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.296+0000 7f738bfff640 20 client.5022.objecter put_session s=0x7f73a4066760 osd=5 4 2024-03-27T21:41:31.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.296+0000 7f738bfff640 15 client.5022.objecter _session_command_op_remove 5 11 2024-03-27T21:41:31.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.296+0000 7f0ff27fc640 1 -- 172.21.15.60:0/4106246441 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0fec054660 con 0x7f0ffc13b020 2024-03-27T21:41:31.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.296+0000 7f0ff27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:31.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.296+0000 7f0ff27fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:31.300 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:31.301 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:31.307 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:41:31.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.323+0000 7f73a9a2f640 10 client.5022.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:31.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.323+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign -1 12 2024-03-27T21:41:31.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.323+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a4066760 osd=5 3 2024-03-27T21:41:31.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.323+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a4066760 osd=5 3 2024-03-27T21:41:31.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.323+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a4066760 osd=5 3 2024-03-27T21:41:31.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.323+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_remove -1 12 2024-03-27T21:41:31.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.323+0000 7f73a9a2f640 20 client.5022.objecter get_session s=0x7f73a4066760 osd=5 3 2024-03-27T21:41:31.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.323+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign 5 12 2024-03-27T21:41:31.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.323+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a4066760 osd=5 4 2024-03-27T21:41:31.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.323+0000 7f73a9a2f640 10 client.5022.objecter _send_command 12 2024-03-27T21:41:31.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.323+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f73a4066950 con 0x7f73a41ecea0 2024-03-27T21:41:31.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.324+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f73a4066950 con 0x7f73a41ecea0 2024-03-27T21:41:31.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.324+0000 7f738bfff640 10 client.5022.objecter ms_dispatch 0x7f73a4000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:31.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.324+0000 7f738bfff640 10 client.5022.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:31.313 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:41:31.316 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:41:31.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.324+0000 7f738bfff640 20 client.5022.objecter put_session s=0x7f73a4066760 osd=5 4 2024-03-27T21:41:31.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.324+0000 7f738bfff640 15 client.5022.objecter _session_command_op_remove 5 12 2024-03-27T21:41:31.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.325+0000 7f73a9a2f640 10 client.5022.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:31.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.325+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign -1 13 2024-03-27T21:41:31.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.325+0000 7f73a9a2f640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f73a41efce0 0x7f73a41f20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.325+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a4066950 osd=6 3 2024-03-27T21:41:31.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.325+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a4066950 osd=6 3 2024-03-27T21:41:31.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.325+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a4066950 osd=6 3 2024-03-27T21:41:31.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.325+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_remove -1 13 2024-03-27T21:41:31.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.325+0000 7f73a9a2f640 20 client.5022.objecter get_session s=0x7f73a4066950 osd=6 3 2024-03-27T21:41:31.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.325+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign 6 13 2024-03-27T21:41:31.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.325+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a4066950 osd=6 4 2024-03-27T21:41:31.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.325+0000 7f73a9a2f640 10 client.5022.objecter _send_command 13 2024-03-27T21:41:31.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.325+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73a41f5040 con 0x7f73a41efce0 2024-03-27T21:41:31.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.325+0000 7f73a37fe640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f73a41efce0 0x7f73a41f20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.325+0000 7f73a37fe640 10 monclient: get_auth_request con 0x7f73a41efce0 auth_method 0 2024-03-27T21:41:31.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.326+0000 7f73a37fe640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f73a41efce0 0x7f73a41f20d0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:31.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.326+0000 7f738bfff640 10 client.5022.objecter ms_handle_connect 0x7f73a41efce0 2024-03-27T21:41:31.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.326+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f73a41f5040 con 0x7f73a41efce0 2024-03-27T21:41:31.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.326+0000 7f738bfff640 10 client.5022.objecter ms_dispatch 0x7f73a4000b90 command_reply(tid 13: 0 ) v1 2024-03-27T21:41:31.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.326+0000 7f738bfff640 10 client.5022.objecter _finish_command 13 = system:0 2024-03-27T21:41:31.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.326+0000 7f738bfff640 20 client.5022.objecter put_session s=0x7f73a4066950 osd=6 4 2024-03-27T21:41:31.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.326+0000 7f738bfff640 15 client.5022.objecter _session_command_op_remove 6 13 2024-03-27T21:41:31.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.354+0000 7f73a9a2f640 10 client.5022.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:31.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.354+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign -1 14 2024-03-27T21:41:31.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.354+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a4066950 osd=6 3 2024-03-27T21:41:31.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.354+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a4066950 osd=6 3 2024-03-27T21:41:31.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.354+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a4066950 osd=6 3 2024-03-27T21:41:31.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.354+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_remove -1 14 2024-03-27T21:41:31.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.354+0000 7f73a9a2f640 20 client.5022.objecter get_session s=0x7f73a4066950 osd=6 3 2024-03-27T21:41:31.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.354+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign 6 14 2024-03-27T21:41:31.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.354+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a4066950 osd=6 4 2024-03-27T21:41:31.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.354+0000 7f73a9a2f640 10 client.5022.objecter _send_command 14 2024-03-27T21:41:31.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.354+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f73a41f5040 con 0x7f73a41efce0 2024-03-27T21:41:31.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.356+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f73a41f5040 con 0x7f73a41efce0 2024-03-27T21:41:31.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.356+0000 7f738bfff640 10 client.5022.objecter ms_dispatch 0x7f73a4000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:31.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.356+0000 7f738bfff640 10 client.5022.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:31.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.356+0000 7f738bfff640 20 client.5022.objecter put_session s=0x7f73a4066950 osd=6 4 2024-03-27T21:41:31.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.356+0000 7f738bfff640 15 client.5022.objecter _session_command_op_remove 6 14 2024-03-27T21:41:31.343 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:41:31.350 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:41:31.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.357+0000 7f73a9a2f640 10 client.5022.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:31.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.357+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign -1 15 2024-03-27T21:41:31.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.357+0000 7f73a9a2f640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f73a41f2c10 0x7f73a41f3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:31.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.357+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a41f5040 osd=7 3 2024-03-27T21:41:31.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.357+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a41f5040 osd=7 3 2024-03-27T21:41:31.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.357+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a41f5040 osd=7 3 2024-03-27T21:41:31.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.357+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_remove -1 15 2024-03-27T21:41:31.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.357+0000 7f73a9a2f640 20 client.5022.objecter get_session s=0x7f73a41f5040 osd=7 3 2024-03-27T21:41:31.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.357+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign 7 15 2024-03-27T21:41:31.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.357+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a41f5040 osd=7 4 2024-03-27T21:41:31.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.357+0000 7f73a9a2f640 10 client.5022.objecter _send_command 15 2024-03-27T21:41:31.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.357+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73a41f83c0 con 0x7f73a41f2c10 2024-03-27T21:41:31.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.357+0000 7f73a2ffd640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f73a41f2c10 0x7f73a41f3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:31.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.357+0000 7f73a2ffd640 10 monclient: get_auth_request con 0x7f73a41f2c10 auth_method 0 2024-03-27T21:41:31.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.357+0000 7f73a2ffd640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f73a41f2c10 0x7f73a41f3070 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:31.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.357+0000 7f738bfff640 10 client.5022.objecter ms_handle_connect 0x7f73a41f2c10 2024-03-27T21:41:31.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.358+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f73a41f83c0 con 0x7f73a41f2c10 2024-03-27T21:41:31.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.358+0000 7f738bfff640 10 client.5022.objecter ms_dispatch 0x7f73a4000b90 command_reply(tid 15: 0 ) v1 2024-03-27T21:41:31.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.358+0000 7f738bfff640 10 client.5022.objecter _finish_command 15 = system:0 2024-03-27T21:41:31.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.358+0000 7f738bfff640 20 client.5022.objecter put_session s=0x7f73a41f5040 osd=7 4 2024-03-27T21:41:31.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.358+0000 7f738bfff640 15 client.5022.objecter _session_command_op_remove 7 15 2024-03-27T21:41:31.356 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:31.370+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:31.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.372+0000 7f73a9a2f640 10 client.5022.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:31.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.372+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign -1 16 2024-03-27T21:41:31.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.372+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a41f5040 osd=7 3 2024-03-27T21:41:31.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.372+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a41f5040 osd=7 3 2024-03-27T21:41:31.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.372+0000 7f73a9a2f640 20 client.5022.objecter _get_session s=0x7f73a41f5040 osd=7 3 2024-03-27T21:41:31.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.372+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_remove -1 16 2024-03-27T21:41:31.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.372+0000 7f73a9a2f640 20 client.5022.objecter get_session s=0x7f73a41f5040 osd=7 3 2024-03-27T21:41:31.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.372+0000 7f73a9a2f640 15 client.5022.objecter _session_command_op_assign 7 16 2024-03-27T21:41:31.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.372+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a41f5040 osd=7 4 2024-03-27T21:41:31.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.372+0000 7f73a9a2f640 10 client.5022.objecter _send_command 16 2024-03-27T21:41:31.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.372+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f73a41f36c0 con 0x7f73a41f2c10 2024-03-27T21:41:31.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.374+0000 7f738bfff640 1 -- 172.21.15.60:0/3630295248 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f73a41f36c0 con 0x7f73a41f2c10 2024-03-27T21:41:31.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.374+0000 7f738bfff640 10 client.5022.objecter ms_dispatch 0x7f73a4000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:31.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.374+0000 7f738bfff640 10 client.5022.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:31.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.374+0000 7f738bfff640 20 client.5022.objecter put_session s=0x7f73a41f5040 osd=7 4 2024-03-27T21:41:31.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.374+0000 7f738bfff640 15 client.5022.objecter _session_command_op_remove 7 16 2024-03-27T21:41:31.361 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:41:31.362 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:41:31.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.374+0000 7f73a9a2f640 10 librados: watch_flush enter 2024-03-27T21:41:31.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.374+0000 7f73a9a2f640 10 librados: watch_flush exit 2024-03-27T21:41:31.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.374+0000 7f73a9a2f640 10 client.5022.objecter close_session for osd.0 2024-03-27T21:41:31.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.374+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f73a4134010 msgr2=0x7f73a412e230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.374+0000 7f73a9a2f640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f73a4134010 0x7f73a412e230 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.374+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a4142340 osd=0 1 2024-03-27T21:41:31.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.374+0000 7f73a9a2f640 10 client.5022.objecter close_session for osd.1 2024-03-27T21:41:31.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.374+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f73a41327d0 msgr2=0x7f73a4132bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.374+0000 7f73a9a2f640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f73a41327d0 0x7f73a4132bb0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.374+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a4140af0 osd=1 1 2024-03-27T21:41:31.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.374+0000 7f73a9a2f640 10 client.5022.objecter close_session for osd.2 2024-03-27T21:41:31.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.374+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f73a40b2620 msgr2=0x7f73a402b170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.374+0000 7f73a9a2f640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f73a40b2620 0x7f73a402b170 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a41330f0 osd=2 1 2024-03-27T21:41:31.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 10 client.5022.objecter close_session for osd.3 2024-03-27T21:41:31.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f73a402c380 msgr2=0x7f73a41e8d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f73a402c380 0x7f73a41e8d80 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a40b1dd0 osd=3 1 2024-03-27T21:41:31.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 10 client.5022.objecter close_session for osd.4 2024-03-27T21:41:31.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a37fe640 1 -- 172.21.15.60:0/3630295248 reap_dead start 2024-03-27T21:41:31.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f73a402ba60 msgr2=0x7f73a41ec450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f73a402ba60 0x7f73a41ec450 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a37fe640 1 -- 172.21.15.60:0/3630295248 reap_dead start 2024-03-27T21:41:31.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a402b8f0 osd=4 1 2024-03-27T21:41:31.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 10 client.5022.objecter close_session for osd.5 2024-03-27T21:41:31.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f73a41ecea0 msgr2=0x7f73a41ef290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f73a41ecea0 0x7f73a41ef290 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a4066760 osd=5 1 2024-03-27T21:41:31.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 10 client.5022.objecter close_session for osd.6 2024-03-27T21:41:31.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f73a41efce0 msgr2=0x7f73a41f20d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f73a41efce0 0x7f73a41f20d0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a4066950 osd=6 1 2024-03-27T21:41:31.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 10 client.5022.objecter close_session for osd.7 2024-03-27T21:41:31.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f73a41f2c10 msgr2=0x7f73a41f3070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f73a41f2c10 0x7f73a41f3070 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 20 client.5022.objecter put_session s=0x7f73a41f5040 osd=7 1 2024-03-27T21:41:31.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 20 client.5022.objecter shutdown clearing up homeless session... 2024-03-27T21:41:31.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 10 client.5022.objecter successfully canceled tick 2024-03-27T21:41:31.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f739006c590 msgr2=0x7f739006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f739006c590 0x7f739006ea50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7394000f60 tx=0x7f7394007d70 comp rx=0 tx=0).stop 2024-03-27T21:41:31.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 10 monclient: shutdown 2024-03-27T21:41:31.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:31.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73a413b480 msgr2=0x7f73a41d9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:31.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 1 --2- 172.21.15.60:0/3630295248 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73a413b480 0x7f73a41d9d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f739c033000 tx=0x7f739c004ab0 comp rx=0 tx=0).stop 2024-03-27T21:41:31.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a37fe640 1 -- 172.21.15.60:0/3630295248 reap_dead start 2024-03-27T21:41:31.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 shutdown_connections 2024-03-27T21:41:31.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 >> 172.21.15.60:0/3630295248 conn(0x7f73a40608e0 msgr2=0x7f73a40b4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:31.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 shutdown_connections 2024-03-27T21:41:31.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 1 -- 172.21.15.60:0/3630295248 wait complete. 2024-03-27T21:41:31.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.375+0000 7f73a9a2f640 1 librados: shutdown 2024-03-27T21:41:31.413 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:31.413 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:31.422 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:41:31.456 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:31.470+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:31.527 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:31.527 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:31.536 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:41:31.557 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:31.571+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:31.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.612+0000 7f1310b53640 10 start_mon_command cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:41:31.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.612+0000 7f1310b53640 10 monclient: _send_command 2 [{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:41:31.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.612+0000 7f1310b53640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:31.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.612+0000 7f1310b53640 1 -- 172.21.15.60:0/222885876 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f130c069470 con 0x7f130c13b020 2024-03-27T21:41:31.641 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:31.642 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:31.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.656+0000 7f1003bab640 10 start_mon_command cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}] 2024-03-27T21:41:31.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.656+0000 7f1003bab640 10 monclient: _send_command 2 [{"prefix": "osd reweight", "id": 3, "weight": 1.0}] 2024-03-27T21:41:31.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.656+0000 7f1003bab640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:31.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:31.656+0000 7f1003bab640 1 -- 172.21.15.60:0/4106246441 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) v1 -- 0x7f0ffc1da310 con 0x7f0ffc13b020 2024-03-27T21:41:31.650 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:41:31.693 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:31.707+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:31.755 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:31.755 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:31.758 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:31.772+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:31.763 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:41:31.859 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:31.873+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:31.909 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:31.909 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:31.917 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:41:31.961 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:31.974+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:32.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.073+0000 7f12faffd640 1 -- 172.21.15.60:0/222885876 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v65) v1 ==== 97+0+0 (secure 0 0 0) 0x7f13000575c0 con 0x7f130c13b020 2024-03-27T21:41:32.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.073+0000 7f12faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:41:32.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.073+0000 7f12faffd640 10 monclient: _finish_command 2 = system:0 nodeep-scrub is set 2024-03-27T21:41:32.059 INFO:teuthology.orchestra.run.smithi060.stderr:nodeep-scrub is set 2024-03-27T21:41:32.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.073+0000 7f0ff27fc640 1 -- 172.21.15.60:0/4106246441 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v65) v1 ==== 113+0+0 (secure 0 0 0) 0x7f0fec057c80 con 0x7f0ffc13b020 2024-03-27T21:41:32.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.073+0000 7f0ff27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd reweight", "id": 3, "weight": 1.0}] 2024-03-27T21:41:32.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.073+0000 7f0ff27fc640 10 monclient: _finish_command 2 = system:0 reweighted osd.3 to 1 (10000) 2024-03-27T21:41:32.060 INFO:teuthology.orchestra.run.smithi060.stderr:reweighted osd.3 to 1 (10000) 2024-03-27T21:41:32.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1003bab640 10 librados: watch_flush enter 2024-03-27T21:41:32.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1003bab640 10 librados: watch_flush exit 2024-03-27T21:41:32.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1003bab640 20 client.4981.objecter shutdown clearing up homeless session... 2024-03-27T21:41:32.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1003bab640 10 client.4981.objecter successfully canceled tick 2024-03-27T21:41:32.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1003bab640 1 -- 172.21.15.60:0/4106246441 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f0fe006c660 msgr2=0x7f0fe006eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:32.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1003bab640 1 --2- 172.21.15.60:0/4106246441 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f0fe006c660 0x7f0fe006eb20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8009b40 tx=0x7f0ff8009190 comp rx=0 tx=0).stop 2024-03-27T21:41:32.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1003bab640 10 monclient: shutdown 2024-03-27T21:41:32.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1003bab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:32.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1003bab640 1 -- 172.21.15.60:0/4106246441 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ffc13b020 msgr2=0x7f0ffc1d9a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:32.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1003bab640 1 --2- 172.21.15.60:0/4106246441 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ffc13b020 0x7f0ffc1d9a50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0fec00e850 tx=0x7f0fec00ed10 comp rx=0 tx=0).stop 2024-03-27T21:41:32.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1003bab640 1 -- 172.21.15.60:0/4106246441 shutdown_connections 2024-03-27T21:41:32.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1003bab640 1 --2- 172.21.15.60:0/4106246441 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0ffc1416f0 0x7f0ffc13d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:32.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1003bab640 1 --2- 172.21.15.60:0/4106246441 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f0fe006c660 0x7f0fe006eb20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:32.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1003bab640 1 --2- 172.21.15.60:0/4106246441 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ffc13b020 0x7f0ffc1d9a50 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:32.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1003bab640 1 --2- 172.21.15.60:0/4106246441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ffc0b40b0 0x7f0ffc1ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:32.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1003bab640 1 -- 172.21.15.60:0/4106246441 >> 172.21.15.60:0/4106246441 conn(0x7f0ffc0608e0 msgr2=0x7f0ffc0b1950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:32.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1003bab640 1 -- 172.21.15.60:0/4106246441 shutdown_connections 2024-03-27T21:41:32.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1310b53640 10 librados: watch_flush enter 2024-03-27T21:41:32.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1310b53640 10 librados: watch_flush exit 2024-03-27T21:41:32.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1310b53640 20 client.5028.objecter shutdown clearing up homeless session... 2024-03-27T21:41:32.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1310b53640 10 client.5028.objecter successfully canceled tick 2024-03-27T21:41:32.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1003bab640 1 -- 172.21.15.60:0/4106246441 wait complete. 2024-03-27T21:41:32.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1310b53640 1 -- 172.21.15.60:0/222885876 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f12ec06c660 msgr2=0x7f12ec06eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:32.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1310b53640 1 --2- 172.21.15.60:0/222885876 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f12ec06c660 0x7f12ec06eb20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f12fc000f60 tx=0x7f12fc007d70 comp rx=0 tx=0).stop 2024-03-27T21:41:32.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1310b53640 10 monclient: shutdown 2024-03-27T21:41:32.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1310b53640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:32.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1310b53640 1 -- 172.21.15.60:0/222885876 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f130c13b020 msgr2=0x7f130c13c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:32.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.075+0000 7f1310b53640 1 --2- 172.21.15.60:0/222885876 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f130c13b020 0x7f130c13c8b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1300000f60 tx=0x7f130000cbe0 comp rx=0 tx=0).stop 2024-03-27T21:41:32.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.076+0000 7f1003bab640 1 librados: shutdown 2024-03-27T21:41:32.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.076+0000 7f1310b53640 1 -- 172.21.15.60:0/222885876 shutdown_connections 2024-03-27T21:41:32.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.076+0000 7f1310b53640 1 --2- 172.21.15.60:0/222885876 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f130c13b480 0x7f130c1360f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:32.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.076+0000 7f1310b53640 1 --2- 172.21.15.60:0/222885876 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f130c13b020 0x7f130c13c8b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:32.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.076+0000 7f1310b53640 1 --2- 172.21.15.60:0/222885876 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f12ec06c660 0x7f12ec06eb20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:32.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.076+0000 7f1310b53640 1 --2- 172.21.15.60:0/222885876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f130c0b40b0 0x7f130c13e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:32.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.076+0000 7f1310b53640 1 -- 172.21.15.60:0/222885876 >> 172.21.15.60:0/222885876 conn(0x7f130c0608e0 msgr2=0x7f130c13f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:32.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.076+0000 7f1310b53640 1 -- 172.21.15.60:0/222885876 shutdown_connections 2024-03-27T21:41:32.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.076+0000 7f1310b53640 1 -- 172.21.15.60:0/222885876 wait complete. 2024-03-27T21:41:32.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.076+0000 7f1310b53640 1 librados: shutdown 2024-03-27T21:41:32.067 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:32.075+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:32.067 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:32.067 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:32.073 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:41:32.074 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-27T21:41:32.162 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:32.176+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:32.217 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:32.217 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:32.226 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:41:32.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.249+0000 7fbd3fe70640 10 monclient: get_monmap_and_config 2024-03-27T21:41:32.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.249+0000 7fbd3fe70640 10 monclient: build_initial_monmap 2024-03-27T21:41:32.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.249+0000 7fbd3fe70640 10 monclient: monmap: 2024-03-27T21:41:32.236 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:32.236 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:32.236 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:32.250747+0000 2024-03-27T21:41:32.236 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:32.250747+0000 2024-03-27T21:41:32.236 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:32.236 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:32.236 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:32.236 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:32.236 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:32.236 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:32.236 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:32.236 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:32.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.250+0000 7fbd3fe70640 1 Processor -- start 2024-03-27T21:41:32.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.250+0000 7fbd3fe70640 1 -- start start 2024-03-27T21:41:32.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.250+0000 7fbd3fe70640 10 monclient: init 2024-03-27T21:41:32.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.251+0000 7fbd3fe70640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:32.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.251+0000 7fbd3fe70640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-03-27T21:41:32.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.251+0000 7fbd3fe70640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbd38071a50 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:32.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.251+0000 7fbd3fe70640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbd3813ef20 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:32.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.251+0000 7fbd3fe70640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbd380a87e0 0x7fbd38141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:32.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.251+0000 7fbd3fe70640 10 monclient(hunting): picked mon.noname-a con 0x7fbd380a87e0 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:32.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.251+0000 7fbd3fe70640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd380a8e90 con 0x7fbd38071a50 2024-03-27T21:41:32.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.251+0000 7fbd3fe70640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd380aa040 con 0x7fbd3813ef20 2024-03-27T21:41:32.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.251+0000 7fbd3fe70640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:32.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.251+0000 7fbd3fe70640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd38071e30 con 0x7fbd380a87e0 2024-03-27T21:41:32.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.251+0000 7fbd3fe70640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:32.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.251+0000 7fbd3fe70640 10 monclient(hunting): authenticate will time out at 933.472671s 2024-03-27T21:41:32.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.251+0000 7fbd3dc0c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbd38071a50 0x7fbd3813e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52284/0 (socket says 172.21.15.60:52284) 2024-03-27T21:41:32.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.251+0000 7fbd3dc0c640 1 -- 172.21.15.60:0/155763362 learned_addr learned my addr 172.21.15.60:0/155763362 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:32.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.251+0000 7fbd3e40d640 1 --2- 172.21.15.60:0/155763362 >> v2:172.21.15.60:3300/0 conn(0x7fbd380a87e0 0x7fbd38141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:32.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.251+0000 7fbd3e40d640 10 monclient(hunting): get_auth_request con 0x7fbd380a87e0 auth_method 0 2024-03-27T21:41:32.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.251+0000 7fbd3e40d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:32.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.251+0000 7fbd3e40d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:32.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.251+0000 7fbd3e40d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:32.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.251+0000 7fbd3e40d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:32.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.251+0000 7fbd3e40d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:32.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.252+0000 7fbd3e40d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:32.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.252+0000 7fbd3e40d640 10 monclient(hunting): handle_auth_done global_id 5037 payload 274 2024-03-27T21:41:32.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.252+0000 7fbd3e40d640 1 -- 172.21.15.60:0/155763362 >> v1:172.21.15.191:6789/0 conn(0x7fbd3813ef20 legacy=0x7fbd38141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:32.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.252+0000 7fbd3e40d640 1 -- 172.21.15.60:0/155763362 >> v1:172.21.15.60:6789/0 conn(0x7fbd38071a50 legacy=0x7fbd3813e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:32.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.252+0000 7fbd3cc0a640 1 -- 172.21.15.60:0/155763362 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542499770 0 0) 0x7fbd380aa040 con 0x7fbd3813ef20 2024-03-27T21:41:32.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.252+0000 7fbd3e40d640 10 monclient: _finish_hunting 0 2024-03-27T21:41:32.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.252+0000 7fbd3e40d640 1 monclient: found mon.noname-a 2024-03-27T21:41:32.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.252+0000 7fbd3e40d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:32.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.252+0000 7fbd3e40d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:32.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.252+0000 7fbd3e40d640 1 -- 172.21.15.60:0/155763362 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd38142280 con 0x7fbd380a87e0 2024-03-27T21:41:32.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.252+0000 7fbd3cc0a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:41:32.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.253+0000 7fbd3e40d640 1 --2- 172.21.15.60:0/155763362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd380a87e0 0x7fbd38141d40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbd30009d90 tx=0x7fbd3002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5348cdffd476e807 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:32.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.253+0000 7fbd3cc0a640 1 -- 172.21.15.60:0/155763362 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3002cd00 con 0x7fbd380a87e0 2024-03-27T21:41:32.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.253+0000 7fbd3cc0a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:32.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.253+0000 7fbd3cc0a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:32.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.253+0000 7fbd3cc0a640 10 monclient: dump: 2024-03-27T21:41:32.242 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:32.242 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:32.242 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:32.242 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:32.242 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:32.242 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:32.242 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:32.243 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:32.243 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:32.243 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:32.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.253+0000 7fbd3cc0a640 10 monclient: _finish_auth 0 2024-03-27T21:41:32.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.253+0000 7fbd3cc0a640 10 monclient: _check_auth_tickets 2024-03-27T21:41:32.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.253+0000 7fbd3cc0a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:32.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.253+0000 7fbd3fe70640 5 monclient: authenticate success, global_id 5037 2024-03-27T21:41:32.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.253+0000 7fbd3fe70640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:32.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.253+0000 7fbd3cc0a640 1 -- 172.21.15.60:0/155763362 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd3002ce60 con 0x7fbd380a87e0 2024-03-27T21:41:32.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.253+0000 7fbd3cc0a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:32.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.253+0000 7fbd3cc0a640 1 -- 172.21.15.60:0/155763362 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd30034660 con 0x7fbd380a87e0 2024-03-27T21:41:32.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.253+0000 7fbd3cc0a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:32.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.253+0000 7fbd3cc0a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:32.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.253+0000 7fbd3cc0a640 10 monclient: dump: 2024-03-27T21:41:32.243 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:32.244 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:32.244 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:32.244 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:32.244 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:32.244 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:32.244 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:32.244 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:32.244 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:32.244 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:32.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.253+0000 7fbd3fe70640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:32.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.253+0000 7fbd3fe70640 4 set_mon_vals no callback set 2024-03-27T21:41:32.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.254+0000 7fbd3fe70640 10 monclient: shutdown 2024-03-27T21:41:32.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.254+0000 7fbd3fe70640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:32.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.254+0000 7fbd3fe70640 1 -- 172.21.15.60:0/155763362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd380a87e0 msgr2=0x7fbd38141d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:32.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.254+0000 7fbd3fe70640 1 --2- 172.21.15.60:0/155763362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd380a87e0 0x7fbd38141d40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbd30009d90 tx=0x7fbd3002bf50 comp rx=0 tx=0).stop 2024-03-27T21:41:32.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.254+0000 7fbd3fe70640 1 -- 172.21.15.60:0/155763362 shutdown_connections 2024-03-27T21:41:32.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.254+0000 7fbd3fe70640 1 --2- 172.21.15.60:0/155763362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd380a87e0 0x7fbd38141d40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:32.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.254+0000 7fbd3fe70640 1 -- 172.21.15.60:0/155763362 >> 172.21.15.60:0/155763362 conn(0x7fbd380b7300 msgr2=0x7fbd380b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:32.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.254+0000 7fbd3fe70640 1 -- 172.21.15.60:0/155763362 shutdown_connections 2024-03-27T21:41:32.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.254+0000 7fbd3fe70640 1 -- 172.21.15.60:0/155763362 wait complete. 2024-03-27T21:41:32.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.255+0000 7fbd3fe70640 10 monclient: build_initial_monmap 2024-03-27T21:41:32.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.255+0000 7fbd3fe70640 10 monclient: monmap: 2024-03-27T21:41:32.250 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:32.250 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:32.250 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:32.251 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:32.251 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:32.251 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:32.251 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:32.251 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:32.251 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:32.251 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:32.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.255+0000 7fbd3fe70640 1 librados: starting msgr at 2024-03-27T21:41:32.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.255+0000 7fbd3fe70640 1 librados: starting objecter 2024-03-27T21:41:32.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.255+0000 7fbd3fe70640 1 Processor -- start 2024-03-27T21:41:32.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.256+0000 7fbd3fe70640 1 -- start start 2024-03-27T21:41:32.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.256+0000 7fbd3fe70640 1 librados: setting wanted keys 2024-03-27T21:41:32.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.256+0000 7fbd3fe70640 1 librados: calling monclient init 2024-03-27T21:41:32.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.256+0000 7fbd3fe70640 10 monclient: init 2024-03-27T21:41:32.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.256+0000 7fbd3fe70640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:32.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.256+0000 7fbd3fe70640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:41:32.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.256+0000 7fbd3fe70640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd38071a50 0x7fbd38139260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:32.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.256+0000 7fbd3fe70640 10 monclient(hunting): picked mon.noname-b con 0x7fbd38071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:32.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.256+0000 7fbd3fe70640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbd380a87e0 0x7fbd38130c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:32.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.256+0000 7fbd3fe70640 10 monclient(hunting): picked mon.noname-c con 0x7fbd380a87e0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:32.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.256+0000 7fbd3fe70640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd3813ef20 0x7fbd381311b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:32.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.256+0000 7fbd3fe70640 10 monclient(hunting): picked mon.noname-a con 0x7fbd3813ef20 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:32.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.256+0000 7fbd3fe70640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:32.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.256+0000 7fbd3fe70640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd381389c0 con 0x7fbd3813ef20 2024-03-27T21:41:32.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.256+0000 7fbd3fe70640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:32.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.256+0000 7fbd3fe70640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd381397a0 con 0x7fbd38071a50 2024-03-27T21:41:32.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3fe70640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:32.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3fe70640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3813a7d0 con 0x7fbd380a87e0 2024-03-27T21:41:32.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3fe70640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:32.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3fe70640 10 monclient(hunting): authenticate will time out at 933.477935s 2024-03-27T21:41:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3d40b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbd380a87e0 0x7fbd38130c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3d40b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbd380a87e0 0x7fbd38130c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:53206/0 (socket says 172.21.15.60:53206) 2024-03-27T21:41:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3d40b640 1 -- 172.21.15.60:0/3761783804 learned_addr learned my addr 172.21.15.60:0/3761783804 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3d40b640 10 monclient(hunting): get_auth_request con 0x7fbd380a87e0 auth_method 0 2024-03-27T21:41:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3d40b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3d40b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3d40b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3e40d640 1 --2- 172.21.15.60:0/3761783804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd3813ef20 0x7fbd381311b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3e40d640 10 monclient(hunting): get_auth_request con 0x7fbd3813ef20 auth_method 0 2024-03-27T21:41:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3e40d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3e40d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3e40d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3dc0c640 1 --2- 172.21.15.60:0/3761783804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd38071a50 0x7fbd38139260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3dc0c640 10 monclient(hunting): get_auth_request con 0x7fbd38071a50 auth_method 0 2024-03-27T21:41:32.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3dc0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:32.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3dc0c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:32.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3dc0c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:32.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3d40b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:32.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3d40b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:32.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3d40b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:32.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3e40d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:32.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3e40d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:32.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3e40d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:32.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3dc0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:32.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3dc0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:32.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3dc0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:32.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3e40d640 10 monclient(hunting): handle_auth_done global_id 5043 payload 824 2024-03-27T21:41:32.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3e40d640 1 -- 172.21.15.60:0/3761783804 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbd380a87e0 msgr2=0x7fbd38130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:32.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3e40d640 1 --2- 172.21.15.60:0/3761783804 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbd380a87e0 0x7fbd38130c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:32.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3e40d640 1 -- 172.21.15.60:0/3761783804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd38071a50 msgr2=0x7fbd38139260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:32.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3e40d640 1 --2- 172.21.15.60:0/3761783804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd38071a50 0x7fbd38139260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:32.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3e40d640 10 monclient: _finish_hunting 0 2024-03-27T21:41:32.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3e40d640 1 monclient: found mon.noname-a 2024-03-27T21:41:32.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3e40d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:32.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3e40d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:32.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3e40d640 1 -- 172.21.15.60:0/3761783804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd30009a70 con 0x7fbd3813ef20 2024-03-27T21:41:32.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3dc0c640 1 --2- 172.21.15.60:0/3761783804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd38071a50 0x7fbd38139260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:32.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3d40b640 1 --2- 172.21.15.60:0/3761783804 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbd380a87e0 0x7fbd38130c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:32.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd3e40d640 1 --2- 172.21.15.60:0/3761783804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd3813ef20 0x7fbd381311b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbd30033000 tx=0x7fbd30004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:32.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd2e7fc640 10 client.?.objecter ms_handle_connect 0x7fbd3813ef20 2024-03-27T21:41:32.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.257+0000 7fbd2e7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:32.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd2e7fc640 1 -- 172.21.15.60:0/3761783804 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd30003b90 con 0x7fbd3813ef20 2024-03-27T21:41:32.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:32.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd2e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:32.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd2e7fc640 10 monclient: dump: 2024-03-27T21:41:32.256 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:32.256 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:32.256 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:32.256 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:32.256 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:32.256 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:32.256 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:32.256 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:32.256 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:32.256 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:32.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd2e7fc640 10 monclient: _finish_auth 0 2024-03-27T21:41:32.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd2e7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:41:32.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:32.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd2e7fc640 1 -- 172.21.15.60:0/3761783804 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd30003cf0 con 0x7fbd3813ef20 2024-03-27T21:41:32.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd2e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:32.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd2e7fc640 1 -- 172.21.15.60:0/3761783804 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd30004230 con 0x7fbd3813ef20 2024-03-27T21:41:32.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:32.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd2e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:32.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd2e7fc640 10 monclient: dump: 2024-03-27T21:41:32.257 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:32.257 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:32.257 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:32.257 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:32.257 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:32.257 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:32.257 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:32.258 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:32.258 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:32.258 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:32.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd2ffff640 4 set_mon_vals no callback set 2024-03-27T21:41:32.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd3fe70640 5 monclient: authenticate success, global_id 5043 2024-03-27T21:41:32.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd3fe70640 10 monclient: _renew_subs 2024-03-27T21:41:32.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd3fe70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:32.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd3fe70640 1 -- 172.21.15.60:0/3761783804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd3813ad20 con 0x7fbd3813ef20 2024-03-27T21:41:32.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd3fe70640 10 client.5043.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:32.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd3fe70640 10 monclient: _renew_subs 2024-03-27T21:41:32.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd3fe70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:32.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd3fe70640 1 -- 172.21.15.60:0/3761783804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd38137a80 con 0x7fbd3813ef20 2024-03-27T21:41:32.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.258+0000 7fbd3fe70640 1 librados: init done 2024-03-27T21:41:32.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.259+0000 7fbd3fe70640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:32.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.259+0000 7fbd3fe70640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:32.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.259+0000 7fbd3fe70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:32.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.259+0000 7fbd3fe70640 1 -- 172.21.15.60:0/3761783804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd38139930 con 0x7fbd3813ef20 2024-03-27T21:41:32.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.259+0000 7fbd2e7fc640 1 -- 172.21.15.60:0/3761783804 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fbd3003b050 con 0x7fbd3813ef20 2024-03-27T21:41:32.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.260+0000 7fbd2e7fc640 1 --2- 172.21.15.60:0/3761783804 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fbd1c06c590 0x7fbd1c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:32.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.260+0000 7fbd2e7fc640 1 -- 172.21.15.60:0/3761783804 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fbd300bc8c0 con 0x7fbd3813ef20 2024-03-27T21:41:32.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.260+0000 7fbd3dc0c640 1 --2- 172.21.15.60:0/3761783804 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fbd1c06c590 0x7fbd1c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:32.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.260+0000 7fbd2e7fc640 10 client.5043.objecter ms_dispatch 0x7fbd38000b90 osd_map(65..65 src has 1..65) v4 2024-03-27T21:41:32.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.260+0000 7fbd2e7fc640 3 client.5043.objecter handle_osd_map got epochs [65,65] > 0 2024-03-27T21:41:32.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.260+0000 7fbd2e7fc640 3 client.5043.objecter handle_osd_map decoding full epoch 65 2024-03-27T21:41:32.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.260+0000 7fbd3dc0c640 10 monclient: get_auth_request con 0x7fbd1c06c590 auth_method 0 2024-03-27T21:41:32.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.260+0000 7fbd2e7fc640 20 client.5043.objecter dump_active .. 0 homeless 2024-03-27T21:41:32.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.260+0000 7fbd2e7fc640 10 client.5043.objecter ms_handle_connect 0x7fbd1c06c590 2024-03-27T21:41:32.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.260+0000 7fbd3dc0c640 1 --2- 172.21.15.60:0/3761783804 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fbd1c06c590 0x7fbd1c06ea50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbd34007fd0 tx=0x7fbd34007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:32.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.264+0000 7fbd2e7fc640 1 -- 172.21.15.60:0/3761783804 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbd30087ed0 con 0x7fbd3813ef20 2024-03-27T21:41:32.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.264+0000 7fbd2e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:32.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.264+0000 7fbd2e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:32.270 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:32.284+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:32.332 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:32.332 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:32.341 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:41:32.363 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:32.377+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:32.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.470+0000 7fbd3fe70640 10 start_mon_command cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}] 2024-03-27T21:41:32.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.470+0000 7fbd3fe70640 10 monclient: _send_command 2 [{"prefix": "osd reweight", "id": 4, "weight": 1.0}] 2024-03-27T21:41:32.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.470+0000 7fbd3fe70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:32.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:32.470+0000 7fbd3fe70640 1 -- 172.21.15.60:0/3761783804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7fbd38069470 con 0x7fbd3813ef20 2024-03-27T21:41:32.465 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:32.478+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:32.487 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:32.487 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:32.496 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:41:32.565 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:32.579+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:32.643 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:32.643 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:32.652 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-27T21:41:32.666 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:32.680+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:32.763 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:32.763 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:32.766 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:32.780+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:32.773 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-27T21:41:32.868 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:32.882+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:32.956 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:32.956 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:32.969 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-27T21:41:32.969 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:32.982+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:33.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.074+0000 7fbd2e7fc640 1 -- 172.21.15.60:0/3761783804 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v66) v1 ==== 113+0+0 (secure 0 0 0) 0x7fbd30039020 con 0x7fbd3813ef20 2024-03-27T21:41:33.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.074+0000 7fbd2e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd reweight", "id": 4, "weight": 1.0}] 2024-03-27T21:41:33.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.074+0000 7fbd2e7fc640 10 monclient: _finish_command 2 = system:0 reweighted osd.4 to 1 (10000) 2024-03-27T21:41:33.060 INFO:teuthology.orchestra.run.smithi060.stderr:reweighted osd.4 to 1 (10000) 2024-03-27T21:41:33.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.077+0000 7fbd3fe70640 10 librados: watch_flush enter 2024-03-27T21:41:33.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.077+0000 7fbd3fe70640 10 librados: watch_flush exit 2024-03-27T21:41:33.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.077+0000 7fbd3fe70640 20 client.5043.objecter shutdown clearing up homeless session... 2024-03-27T21:41:33.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.077+0000 7fbd3fe70640 10 client.5043.objecter successfully canceled tick 2024-03-27T21:41:33.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.077+0000 7fbd3fe70640 1 -- 172.21.15.60:0/3761783804 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fbd1c06c590 msgr2=0x7fbd1c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:33.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.077+0000 7fbd3fe70640 1 --2- 172.21.15.60:0/3761783804 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fbd1c06c590 0x7fbd1c06ea50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbd34007fd0 tx=0x7fbd34007e80 comp rx=0 tx=0).stop 2024-03-27T21:41:33.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.077+0000 7fbd3fe70640 10 monclient: shutdown 2024-03-27T21:41:33.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.077+0000 7fbd3fe70640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:33.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.077+0000 7fbd3fe70640 1 -- 172.21.15.60:0/3761783804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd3813ef20 msgr2=0x7fbd381311b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:33.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.077+0000 7fbd3fe70640 1 --2- 172.21.15.60:0/3761783804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd3813ef20 0x7fbd381311b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbd30033000 tx=0x7fbd30004a40 comp rx=0 tx=0).stop 2024-03-27T21:41:33.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.077+0000 7fbd3fe70640 1 -- 172.21.15.60:0/3761783804 shutdown_connections 2024-03-27T21:41:33.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.077+0000 7fbd3fe70640 1 --2- 172.21.15.60:0/3761783804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd3813ef20 0x7fbd381311b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.077+0000 7fbd3fe70640 1 --2- 172.21.15.60:0/3761783804 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbd380a87e0 0x7fbd38130c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.077+0000 7fbd3fe70640 1 --2- 172.21.15.60:0/3761783804 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fbd1c06c590 0x7fbd1c06ea50 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.077+0000 7fbd3fe70640 1 --2- 172.21.15.60:0/3761783804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd38071a50 0x7fbd38139260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.077+0000 7fbd3fe70640 1 -- 172.21.15.60:0/3761783804 >> 172.21.15.60:0/3761783804 conn(0x7fbd380b7300 msgr2=0x7fbd38135840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:33.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.077+0000 7fbd3fe70640 1 -- 172.21.15.60:0/3761783804 shutdown_connections 2024-03-27T21:41:33.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.077+0000 7fbd3fe70640 1 -- 172.21.15.60:0/3761783804 wait complete. 2024-03-27T21:41:33.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.078+0000 7fbd3fe70640 1 librados: shutdown 2024-03-27T21:41:33.069 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:33.083+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:33.079 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-27T21:41:33.155 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:33.156 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:33.168 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:41:33.169 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:33.184+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:33.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.220+0000 7f59feb18640 10 monclient: get_monmap_and_config 2024-03-27T21:41:33.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.220+0000 7f59feb18640 10 monclient: build_initial_monmap 2024-03-27T21:41:33.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.220+0000 7f59feb18640 10 monclient: monmap: 2024-03-27T21:41:33.207 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:33.207 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:33.207 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:33.221850+0000 2024-03-27T21:41:33.207 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:33.221850+0000 2024-03-27T21:41:33.207 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:33.208 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:33.208 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:33.208 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:33.208 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:33.208 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:33.208 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:33.208 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:33.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.221+0000 7f59feb18640 1 Processor -- start 2024-03-27T21:41:33.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.221+0000 7f59feb18640 1 -- start start 2024-03-27T21:41:33.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.221+0000 7f59feb18640 10 monclient: init 2024-03-27T21:41:33.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59feb18640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:33.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59feb18640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-03-27T21:41:33.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59feb18640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f59f80b40b0 0x7f59f80a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:33.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59feb18640 10 monclient(hunting): picked mon.noname-c con 0x7f59f80b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:33.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59feb18640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59f813b020 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:33.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59feb18640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59f813b480 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:33.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59feb18640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f813b8d0 con 0x7f59f813b020 2024-03-27T21:41:33.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59f7fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f59f813b020 0x7f59f81416f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52294/0 (socket says 172.21.15.60:52294) 2024-03-27T21:41:33.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59f7fff640 1 -- 172.21.15.60:0/3650823133 learned_addr learned my addr 172.21.15.60:0/3650823133 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:33.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59feb18640 1 -- 172.21.15.60:0/3650823133 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f80b3830 con 0x7f59f813b480 2024-03-27T21:41:33.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59fc8b4640 1 --2- 172.21.15.60:0/3650823133 >> v2:172.21.15.191:3300/0 conn(0x7f59f80b40b0 0x7f59f80a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:33.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59feb18640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:33.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59feb18640 1 -- 172.21.15.60:0/3650823133 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59f80b4490 con 0x7f59f80b40b0 2024-03-27T21:41:33.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59feb18640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:33.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59feb18640 10 monclient(hunting): authenticate will time out at 934.443721s 2024-03-27T21:41:33.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59fc8b4640 10 monclient(hunting): get_auth_request con 0x7f59f80b40b0 auth_method 0 2024-03-27T21:41:33.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59fc8b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:33.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59fc8b4640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:33.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59fc8b4640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:33.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59fc8b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:33.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59fc8b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:33.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.222+0000 7f59fc8b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:33.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.223+0000 7f59f77fe640 1 -- 172.21.15.60:0/3650823133 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150584793 0 0) 0x7f59f813b8d0 con 0x7f59f813b020 2024-03-27T21:41:33.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.223+0000 7f59f77fe640 10 monclient(hunting): _negotiate 2024-03-27T21:41:33.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.223+0000 7f59f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:33.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.223+0000 7f59f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:33.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.223+0000 7f59f77fe640 10 monclient(hunting): my global_id is 5046 2024-03-27T21:41:33.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.223+0000 7f59f77fe640 1 -- 172.21.15.60:0/3650823133 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59e8003520 con 0x7f59f813b020 2024-03-27T21:41:33.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.223+0000 7f59fc8b4640 10 monclient(hunting): handle_auth_done global_id 4990 payload 274 2024-03-27T21:41:33.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.223+0000 7f59fc8b4640 1 -- 172.21.15.60:0/3650823133 >> v1:172.21.15.191:6789/0 conn(0x7f59f813b480 legacy=0x7f59f814bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:33.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.223+0000 7f59fc8b4640 1 -- 172.21.15.60:0/3650823133 >> v1:172.21.15.60:6789/0 conn(0x7f59f813b020 legacy=0x7f59f81416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:33.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.223+0000 7f59fc8b4640 10 monclient: _finish_hunting 0 2024-03-27T21:41:33.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.223+0000 7f59fc8b4640 1 monclient: found mon.noname-c 2024-03-27T21:41:33.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.223+0000 7f59fc8b4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:33.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.223+0000 7f59fc8b4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:41:33.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.223+0000 7f59fc8b4640 1 -- 172.21.15.60:0/3650823133 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f80aa520 con 0x7f59f80b40b0 2024-03-27T21:41:33.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.224+0000 7f59fc8b4640 1 --2- 172.21.15.60:0/3650823133 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59f80b40b0 0x7f59f80a8b90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f59ec0097a0 tx=0x7f59ec02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c280b48515979572 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:33.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.224+0000 7f59f77fe640 1 -- 172.21.15.60:0/3650823133 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ec031830 con 0x7f59f80b40b0 2024-03-27T21:41:33.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.224+0000 7f59f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:33.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.224+0000 7f59f77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:33.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.224+0000 7f59f77fe640 10 monclient: dump: 2024-03-27T21:41:33.214 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:33.214 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:33.214 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:33.214 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:33.215 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:33.215 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:33.215 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:33.215 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:33.215 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:33.215 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:33.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.224+0000 7f59f77fe640 10 monclient: _finish_auth 0 2024-03-27T21:41:33.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.224+0000 7f59f77fe640 10 monclient: _check_auth_tickets 2024-03-27T21:41:33.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.224+0000 7f59f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:33.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.224+0000 7f59f77fe640 1 -- 172.21.15.60:0/3650823133 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59ec031990 con 0x7f59f80b40b0 2024-03-27T21:41:33.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.224+0000 7f59f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:33.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.224+0000 7f59f77fe640 1 -- 172.21.15.60:0/3650823133 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ec031d20 con 0x7f59f80b40b0 2024-03-27T21:41:33.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.224+0000 7f59f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:33.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.224+0000 7f59f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:33.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.224+0000 7f59f77fe640 10 monclient: dump: 2024-03-27T21:41:33.216 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:33.216 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:33.216 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:33.216 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:33.216 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:33.216 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:33.216 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:33.216 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:33.216 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:33.216 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:33.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.224+0000 7f59feb18640 5 monclient: authenticate success, global_id 4990 2024-03-27T21:41:33.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.224+0000 7f59feb18640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:33.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.224+0000 7f59feb18640 4 set_mon_vals no callback set 2024-03-27T21:41:33.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.225+0000 7f59feb18640 10 monclient: shutdown 2024-03-27T21:41:33.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.225+0000 7f59feb18640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:33.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.225+0000 7f59feb18640 1 -- 172.21.15.60:0/3650823133 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59f80b40b0 msgr2=0x7f59f80a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:33.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.225+0000 7f59feb18640 1 --2- 172.21.15.60:0/3650823133 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59f80b40b0 0x7f59f80a8b90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f59ec0097a0 tx=0x7f59ec02b8d0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.225+0000 7f59feb18640 1 -- 172.21.15.60:0/3650823133 shutdown_connections 2024-03-27T21:41:33.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.225+0000 7f59feb18640 1 --2- 172.21.15.60:0/3650823133 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59f80b40b0 0x7f59f80a8b90 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.225+0000 7f59feb18640 1 -- 172.21.15.60:0/3650823133 >> 172.21.15.60:0/3650823133 conn(0x7f59f80608e0 msgr2=0x7f59f8060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:33.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.225+0000 7f59feb18640 1 -- 172.21.15.60:0/3650823133 shutdown_connections 2024-03-27T21:41:33.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.225+0000 7f59feb18640 1 -- 172.21.15.60:0/3650823133 wait complete. 2024-03-27T21:41:33.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.226+0000 7f59feb18640 10 monclient: build_initial_monmap 2024-03-27T21:41:33.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.226+0000 7f59feb18640 10 monclient: monmap: 2024-03-27T21:41:33.224 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:33.225 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:33.225 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:33.225 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:33.225 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:33.225 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:33.225 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:33.225 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:33.225 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:33.225 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:33.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.227+0000 7f59feb18640 1 librados: starting msgr at 2024-03-27T21:41:33.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.227+0000 7f59feb18640 1 librados: starting objecter 2024-03-27T21:41:33.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.227+0000 7f59feb18640 1 Processor -- start 2024-03-27T21:41:33.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.227+0000 7f59feb18640 1 -- start start 2024-03-27T21:41:33.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.227+0000 7f59feb18640 1 librados: setting wanted keys 2024-03-27T21:41:33.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.227+0000 7f59feb18640 1 librados: calling monclient init 2024-03-27T21:41:33.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.227+0000 7f59feb18640 10 monclient: init 2024-03-27T21:41:33.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59feb18640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:33.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59feb18640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:41:33.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59feb18640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f59f80b40b0 0x7f59f81d9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:33.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59fc8b4640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f59f80b40b0 0x7f59f81d9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:33.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59fc8b4640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f59f80b40b0 0x7f59f81d9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:53214/0 (socket says 172.21.15.60:53214) 2024-03-27T21:41:33.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59fc8b4640 1 -- 172.21.15.60:0/2429861475 learned_addr learned my addr 172.21.15.60:0/2429861475 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:33.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59feb18640 10 monclient(hunting): picked mon.noname-c con 0x7f59f80b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:33.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59feb18640 1 --2- 172.21.15.60:0/2429861475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59f813b020 0x7f59f8139270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:33.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59feb18640 10 monclient(hunting): picked mon.noname-b con 0x7f59f813b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:33.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59feb18640 1 --2- 172.21.15.60:0/2429861475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59f813b480 0x7f59f81397b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:33.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59feb18640 10 monclient(hunting): picked mon.noname-a con 0x7f59f813b480 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:33.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59feb18640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:33.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59feb18640 1 -- 172.21.15.60:0/2429861475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f8139cf0 con 0x7f59f813b480 2024-03-27T21:41:33.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59feb18640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:33.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59feb18640 1 -- 172.21.15.60:0/2429861475 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f80ae0a0 con 0x7f59f813b020 2024-03-27T21:41:33.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59feb18640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:33.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59feb18640 1 -- 172.21.15.60:0/2429861475 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59f81d9690 con 0x7f59f80b40b0 2024-03-27T21:41:33.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59feb18640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:33.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59feb18640 10 monclient(hunting): authenticate will time out at 934.449727s 2024-03-27T21:41:33.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59fc8b4640 10 monclient(hunting): get_auth_request con 0x7f59f80b40b0 auth_method 0 2024-03-27T21:41:33.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59fc8b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:33.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59fc8b4640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:33.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59fc8b4640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:33.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59fc8b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:33.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59fc8b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:33.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.228+0000 7f59fc8b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:33.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59fd0b5640 1 --2- 172.21.15.60:0/2429861475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59f813b480 0x7f59f81397b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:33.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59fd0b5640 10 monclient(hunting): get_auth_request con 0x7f59f813b480 auth_method 0 2024-03-27T21:41:33.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59fd0b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:33.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59fd0b5640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:33.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59fd0b5640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:33.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59f7fff640 1 --2- 172.21.15.60:0/2429861475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59f813b020 0x7f59f8139270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:33.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59fc8b4640 10 monclient(hunting): handle_auth_done global_id 4820 payload 824 2024-03-27T21:41:33.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59fc8b4640 1 -- 172.21.15.60:0/2429861475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59f813b020 msgr2=0x7f59f8139270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:33.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59fc8b4640 1 --2- 172.21.15.60:0/2429861475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59f813b020 0x7f59f8139270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59fc8b4640 1 -- 172.21.15.60:0/2429861475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59f813b480 msgr2=0x7f59f81397b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:33.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59fc8b4640 1 --2- 172.21.15.60:0/2429861475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59f813b480 0x7f59f81397b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59fc8b4640 10 monclient: _finish_hunting 0 2024-03-27T21:41:33.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59fc8b4640 1 monclient: found mon.noname-c 2024-03-27T21:41:33.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59fc8b4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:33.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59fc8b4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:33.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59fc8b4640 1 -- 172.21.15.60:0/2429861475 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ec009400 con 0x7f59f80b40b0 2024-03-27T21:41:33.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59f7fff640 10 monclient: get_auth_request con 0x7f59f813b020 auth_method 0 2024-03-27T21:41:33.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59f7fff640 1 --2- 172.21.15.60:0/2429861475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59f813b020 0x7f59f8139270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:41:33.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59fd0b5640 1 --2- 172.21.15.60:0/2429861475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59f813b480 0x7f59f81397b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:41:33.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59fc8b4640 1 --2- 172.21.15.60:0/2429861475 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f59f80b40b0 0x7f59f81d9bb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f59ec02bee0 tx=0x7f59ec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:33.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59f57fa640 10 client.?.objecter ms_handle_connect 0x7f59f80b40b0 2024-03-27T21:41:33.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.229+0000 7f59f57fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:33.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59f57fa640 1 -- 172.21.15.60:0/2429861475 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ec03b8b0 con 0x7f59f80b40b0 2024-03-27T21:41:33.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:33.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59f57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:33.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59f57fa640 10 monclient: dump: 2024-03-27T21:41:33.229 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:33.229 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:33.230 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:33.230 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:33.231 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:33.231 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:33.231 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:33.231 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:33.231 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:33.231 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:33.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59f57fa640 10 monclient: _finish_auth 0 2024-03-27T21:41:33.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59f57fa640 10 monclient: _check_auth_tickets 2024-03-27T21:41:33.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:33.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59f57fa640 1 -- 172.21.15.60:0/2429861475 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59ec03ba10 con 0x7f59f80b40b0 2024-03-27T21:41:33.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59f57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:33.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59f57fa640 1 -- 172.21.15.60:0/2429861475 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ec03bdc0 con 0x7f59f80b40b0 2024-03-27T21:41:33.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59feb18640 5 monclient: authenticate success, global_id 4820 2024-03-27T21:41:33.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:33.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59f57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:33.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59f6ffd640 4 set_mon_vals no callback set 2024-03-27T21:41:33.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59f57fa640 10 monclient: dump: 2024-03-27T21:41:33.232 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:33.232 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:33.232 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:33.232 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:33.232 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:33.232 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:33.232 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:33.232 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:33.232 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:33.232 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:33.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59feb18640 10 monclient: _renew_subs 2024-03-27T21:41:33.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59feb18640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:33.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59feb18640 1 -- 172.21.15.60:0/2429861475 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f80ac470 con 0x7f59f80b40b0 2024-03-27T21:41:33.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59feb18640 10 client.4820.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:33.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59feb18640 10 monclient: _renew_subs 2024-03-27T21:41:33.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59feb18640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:33.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59feb18640 1 -- 172.21.15.60:0/2429861475 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f81da2d0 con 0x7f59f80b40b0 2024-03-27T21:41:33.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.230+0000 7f59feb18640 1 librados: init done 2024-03-27T21:41:33.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.232+0000 7f59f57fa640 1 -- 172.21.15.60:0/2429861475 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f59ec04e450 con 0x7f59f80b40b0 2024-03-27T21:41:33.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.232+0000 7f59f57fa640 1 --2- 172.21.15.60:0/2429861475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f59dc06c590 0x7f59dc06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:33.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.232+0000 7f59feb18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:33.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.232+0000 7f59feb18640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:33.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.232+0000 7f59f57fa640 1 -- 172.21.15.60:0/2429861475 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f59ec0ab240 con 0x7f59f80b40b0 2024-03-27T21:41:33.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.232+0000 7f59f57fa640 10 client.4820.objecter ms_dispatch 0x7f59f8000b90 osd_map(66..66 src has 1..66) v4 2024-03-27T21:41:33.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.232+0000 7f59f57fa640 3 client.4820.objecter handle_osd_map got epochs [66,66] > 0 2024-03-27T21:41:33.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.232+0000 7f59f57fa640 3 client.4820.objecter handle_osd_map decoding full epoch 66 2024-03-27T21:41:33.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.232+0000 7f59f7fff640 1 --2- 172.21.15.60:0/2429861475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f59dc06c590 0x7f59dc06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:33.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.232+0000 7f59f57fa640 20 client.4820.objecter dump_active .. 0 homeless 2024-03-27T21:41:33.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.233+0000 7f59feb18640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:33.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.233+0000 7f59feb18640 1 -- 172.21.15.60:0/2429861475 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f8069470 con 0x7f59f80b40b0 2024-03-27T21:41:33.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.233+0000 7f59f7fff640 10 monclient: get_auth_request con 0x7f59dc06c590 auth_method 0 2024-03-27T21:41:33.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.233+0000 7f59f7fff640 1 --2- 172.21.15.60:0/2429861475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f59dc06c590 0x7f59dc06ea50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f59f80b0720 tx=0x7f59e4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:33.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.233+0000 7f59f57fa640 10 client.4820.objecter ms_handle_connect 0x7f59dc06c590 2024-03-27T21:41:33.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.237+0000 7f59f57fa640 1 -- 172.21.15.60:0/2429861475 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f59ec076680 con 0x7f59f80b40b0 2024-03-27T21:41:33.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.237+0000 7f59f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:33.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.237+0000 7f59f57fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:33.271 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:33.285+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:33.283 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:33.283 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:33.292 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:41:33.372 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:33.386+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:33.373 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:41:33.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.485+0000 7f59feb18640 10 start_mon_command cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}] 2024-03-27T21:41:33.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.485+0000 7f59feb18640 10 monclient: _send_command 2 [{"prefix": "osd reweight", "id": 5, "weight": 1.0}] 2024-03-27T21:41:33.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.485+0000 7f59feb18640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:33.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.485+0000 7f59feb18640 1 -- 172.21.15.60:0/2429861475 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) v1 -- 0x7f59f81d9820 con 0x7f59f80b40b0 2024-03-27T21:41:33.473 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:33.486+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:33.480 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:33.480 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:33.492 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:41:33.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.543+0000 7fb39f9d2640 10 monclient: get_monmap_and_config 2024-03-27T21:41:33.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.543+0000 7fb39f9d2640 10 monclient: build_initial_monmap 2024-03-27T21:41:33.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.543+0000 7fb39f9d2640 10 monclient: monmap: 2024-03-27T21:41:33.530 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:33.530 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:33.530 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:33.544221+0000 2024-03-27T21:41:33.530 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:33.544221+0000 2024-03-27T21:41:33.530 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:33.530 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:33.530 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:33.530 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:33.530 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:33.530 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:33.530 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:33.530 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:33.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.544+0000 7fb39f9d2640 1 Processor -- start 2024-03-27T21:41:33.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.544+0000 7fb39f9d2640 1 -- start start 2024-03-27T21:41:33.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.544+0000 7fb39f9d2640 10 monclient: init 2024-03-27T21:41:33.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39f9d2640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:33.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39f9d2640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-03-27T21:41:33.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39f9d2640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb3980b40b0 0x7fb3980a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:33.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39f9d2640 10 monclient(hunting): picked mon.noname-c con 0x7fb3980b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:33.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39f9d2640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb39813b020 0x7fb39813b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:33.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39f9d2640 10 monclient(hunting): picked mon.noname-b con 0x7fb39813b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:33.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39f9d2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb3981416f0 0x7fb398141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:33.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39f9d2640 10 monclient(hunting): picked mon.noname-a con 0x7fb3981416f0 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:33.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39f9d2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:33.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39f9d2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3980b4490 con 0x7fb3981416f0 2024-03-27T21:41:33.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39f9d2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:33.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39f9d2640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3980b3830 con 0x7fb3980b40b0 2024-03-27T21:41:33.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39f9d2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:33.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39f9d2640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3980aa520 con 0x7fb39813b020 2024-03-27T21:41:33.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39f9d2640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:33.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39f9d2640 10 monclient(hunting): authenticate will time out at 934.766343s 2024-03-27T21:41:33.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39cf6d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb39813b020 0x7fb39813b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:33.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39cf6d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb39813b020 0x7fb39813b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:53220/0 (socket says 172.21.15.60:53220) 2024-03-27T21:41:33.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39cf6d640 1 -- 172.21.15.60:0/1059094440 learned_addr learned my addr 172.21.15.60:0/1059094440 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:33.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39cf6d640 10 monclient(hunting): get_auth_request con 0x7fb39813b020 auth_method 0 2024-03-27T21:41:33.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39cf6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:33.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39cf6d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:33.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39cf6d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:33.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39df6f640 1 --2- 172.21.15.60:0/1059094440 >> v2:172.21.15.60:3300/0 conn(0x7fb3981416f0 0x7fb398141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39df6f640 10 monclient(hunting): get_auth_request con 0x7fb3981416f0 auth_method 0 2024-03-27T21:41:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39df6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39df6f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39df6f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39cf6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39cf6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39cf6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39df6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39df6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39df6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39df6f640 10 monclient(hunting): handle_auth_done global_id 5052 payload 274 2024-03-27T21:41:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.545+0000 7fb39df6f640 1 -- 172.21.15.60:0/1059094440 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb39813b020 msgr2=0x7fb39813b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.546+0000 7fb39df6f640 1 --2- 172.21.15.60:0/1059094440 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb39813b020 0x7fb39813b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.546+0000 7fb39df6f640 1 -- 172.21.15.60:0/1059094440 >> v2:172.21.15.191:3300/0 conn(0x7fb3980b40b0 msgr2=0x7fb3980a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:33.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.546+0000 7fb39d76e640 1 --2- 172.21.15.60:0/1059094440 >> v2:172.21.15.191:3300/0 conn(0x7fb3980b40b0 0x7fb3980a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:33.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.546+0000 7fb39df6f640 1 --2- 172.21.15.60:0/1059094440 >> v2:172.21.15.191:3300/0 conn(0x7fb3980b40b0 0x7fb3980a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.546+0000 7fb39df6f640 10 monclient: _finish_hunting 0 2024-03-27T21:41:33.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.546+0000 7fb39df6f640 1 monclient: found mon.noname-a 2024-03-27T21:41:33.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.546+0000 7fb39df6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:33.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.546+0000 7fb39df6f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:33.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.546+0000 7fb39df6f640 1 -- 172.21.15.60:0/1059094440 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3980a9470 con 0x7fb3981416f0 2024-03-27T21:41:33.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.546+0000 7fb39cf6d640 1 --2- 172.21.15.60:0/1059094440 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb39813b020 0x7fb39813b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:33.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.546+0000 7fb39df6f640 1 --2- 172.21.15.60:0/1059094440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3981416f0 0x7fb398141b80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb3980b4810 tx=0x7fb39002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a7f7a5ce483e38 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:33.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb38bfff640 1 -- 172.21.15.60:0/1059094440 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39002cbd0 con 0x7fb3981416f0 2024-03-27T21:41:33.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb38bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:33.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb38bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:33.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb38bfff640 10 monclient: dump: 2024-03-27T21:41:33.552 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:33.553 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:33.553 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:33.553 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:33.553 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:33.553 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:33.553 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:33.553 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:33.553 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:33.553 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:33.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb38bfff640 10 monclient: _finish_auth 0 2024-03-27T21:41:33.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb38bfff640 10 monclient: _check_auth_tickets 2024-03-27T21:41:33.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb38bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:33.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb38bfff640 1 -- 172.21.15.60:0/1059094440 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39002cd30 con 0x7fb3981416f0 2024-03-27T21:41:33.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb39f9d2640 5 monclient: authenticate success, global_id 5052 2024-03-27T21:41:33.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb39f9d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:33.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb38bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:33.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb39f9d2640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:33.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb38bfff640 1 -- 172.21.15.60:0/1059094440 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39002cea0 con 0x7fb3981416f0 2024-03-27T21:41:33.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb38bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:33.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb39f9d2640 4 set_mon_vals no callback set 2024-03-27T21:41:33.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb38bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:33.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb38bfff640 10 monclient: dump: 2024-03-27T21:41:33.554 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:33.554 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:33.554 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:33.554 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:33.554 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:33.554 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:33.555 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb39f9d2640 10 monclient: shutdown 2024-03-27T21:41:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb39f9d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb39f9d2640 1 -- 172.21.15.60:0/1059094440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3981416f0 msgr2=0x7fb398141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/1059094440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3981416f0 0x7fb398141b80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb3980b4810 tx=0x7fb39002be20 comp rx=0 tx=0).stop 2024-03-27T21:41:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb39f9d2640 1 -- 172.21.15.60:0/1059094440 shutdown_connections 2024-03-27T21:41:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/1059094440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3981416f0 0x7fb398141b80 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/1059094440 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb39813b020 0x7fb39813b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/1059094440 >> v2:172.21.15.191:3300/0 conn(0x7fb3980b40b0 0x7fb3980a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb39f9d2640 1 -- 172.21.15.60:0/1059094440 >> 172.21.15.60:0/1059094440 conn(0x7fb3980608e0 msgr2=0x7fb398060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb39f9d2640 1 -- 172.21.15.60:0/1059094440 shutdown_connections 2024-03-27T21:41:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.547+0000 7fb39f9d2640 1 -- 172.21.15.60:0/1059094440 wait complete. 2024-03-27T21:41:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.548+0000 7fb39f9d2640 10 monclient: build_initial_monmap 2024-03-27T21:41:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.548+0000 7fb39f9d2640 10 monclient: monmap: 2024-03-27T21:41:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:33.556 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:33.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.549+0000 7fb39f9d2640 1 librados: starting msgr at 2024-03-27T21:41:33.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.549+0000 7fb39f9d2640 1 librados: starting objecter 2024-03-27T21:41:33.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.549+0000 7fb39f9d2640 1 Processor -- start 2024-03-27T21:41:33.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.549+0000 7fb39f9d2640 1 -- start start 2024-03-27T21:41:33.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.549+0000 7fb39f9d2640 1 librados: setting wanted keys 2024-03-27T21:41:33.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.549+0000 7fb39f9d2640 1 librados: calling monclient init 2024-03-27T21:41:33.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.549+0000 7fb39f9d2640 10 monclient: init 2024-03-27T21:41:33.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39f9d2640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:33.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39f9d2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:41:33.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39f9d2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3980b40b0 0x7fb3980afdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:33.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39f9d2640 10 monclient(hunting): picked mon.noname-a con 0x7fb3980b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:33.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39f9d2640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb39813b020 0x7fb398143fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:33.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39cf6d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb39813b020 0x7fb398143fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:33.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39cf6d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb39813b020 0x7fb398143fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:53236/0 (socket says 172.21.15.60:53236) 2024-03-27T21:41:33.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39d76e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3980b40b0 0x7fb3980afdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:33.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39d76e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3980b40b0 0x7fb3980afdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50808/0 (socket says 172.21.15.60:50808) 2024-03-27T21:41:33.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39cf6d640 1 -- 172.21.15.60:0/4279277301 learned_addr learned my addr 172.21.15.60:0/4279277301 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:33.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39f9d2640 10 monclient(hunting): picked mon.noname-c con 0x7fb39813b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:33.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3981416f0 0x7fb3981da040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:33.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39f9d2640 10 monclient(hunting): picked mon.noname-b con 0x7fb3981416f0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:33.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39f9d2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:33.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3981d9bb0 con 0x7fb3980b40b0 2024-03-27T21:41:33.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39f9d2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:33.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3981da580 con 0x7fb3981416f0 2024-03-27T21:41:33.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39f9d2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:33.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb39813cb40 con 0x7fb39813b020 2024-03-27T21:41:33.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39f9d2640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:33.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39f9d2640 10 monclient(hunting): authenticate will time out at 934.771825s 2024-03-27T21:41:33.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39cf6d640 10 monclient(hunting): get_auth_request con 0x7fb39813b020 auth_method 0 2024-03-27T21:41:33.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39cf6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:33.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39cf6d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:33.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.550+0000 7fb39cf6d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:33.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39cf6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:33.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39cf6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:33.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39cf6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:33.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39d76e640 10 monclient(hunting): get_auth_request con 0x7fb3980b40b0 auth_method 0 2024-03-27T21:41:33.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39d76e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:33.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39d76e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:33.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39d76e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:33.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39df6f640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3981416f0 0x7fb3981da040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:33.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39df6f640 10 monclient(hunting): get_auth_request con 0x7fb3981416f0 auth_method 0 2024-03-27T21:41:33.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39df6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:33.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39df6f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:33.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39df6f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:33.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39cf6d640 10 monclient(hunting): handle_auth_done global_id 4826 payload 824 2024-03-27T21:41:33.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39cf6d640 1 -- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3981416f0 msgr2=0x7fb3981da040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:33.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39cf6d640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3981416f0 0x7fb3981da040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39cf6d640 1 -- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3980b40b0 msgr2=0x7fb3980afdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:33.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39cf6d640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3980b40b0 0x7fb3980afdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39cf6d640 10 monclient: _finish_hunting 0 2024-03-27T21:41:33.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39cf6d640 1 monclient: found mon.noname-c 2024-03-27T21:41:33.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39cf6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:33.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39cf6d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:33.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39cf6d640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38c0094d0 con 0x7fb39813b020 2024-03-27T21:41:33.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39df6f640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3981416f0 0x7fb3981da040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:41:33.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb39cf6d640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb39813b020 0x7fb398143fc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00e9d0 tx=0x7fb38c00ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:33.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb389ffb640 10 client.?.objecter ms_handle_connect 0x7fb39813b020 2024-03-27T21:41:33.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.551+0000 7fb389ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:33.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.552+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c017d80 con 0x7fb39813b020 2024-03-27T21:41:33.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.552+0000 7fb389ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:33.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.552+0000 7fb389ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:33.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.552+0000 7fb389ffb640 10 monclient: dump: 2024-03-27T21:41:33.561 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:33.561 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:33.561 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:33.562 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:33.562 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:33.562 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:33.562 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:33.562 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:33.562 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:33.562 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:33.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.552+0000 7fb389ffb640 10 monclient: _finish_auth 0 2024-03-27T21:41:33.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.552+0000 7fb389ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:41:33.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.552+0000 7fb389ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:33.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.552+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb38c00c730 con 0x7fb39813b020 2024-03-27T21:41:33.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.552+0000 7fb389ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:33.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.552+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c00caa0 con 0x7fb39813b020 2024-03-27T21:41:33.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.552+0000 7fb389ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:33.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.552+0000 7fb389ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:33.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.552+0000 7fb389ffb640 10 monclient: dump: 2024-03-27T21:41:33.563 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:33.563 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:33.563 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:33.563 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:33.563 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:33.563 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:33.563 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:33.563 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:33.563 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:33.563 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:33.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.552+0000 7fb38b7fe640 4 set_mon_vals no callback set 2024-03-27T21:41:33.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.552+0000 7fb39f9d2640 5 monclient: authenticate success, global_id 4826 2024-03-27T21:41:33.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.552+0000 7fb39f9d2640 10 monclient: _renew_subs 2024-03-27T21:41:33.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.552+0000 7fb39f9d2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:33.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.552+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb390009a70 con 0x7fb39813b020 2024-03-27T21:41:33.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.553+0000 7fb39f9d2640 10 client.4826.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:33.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.553+0000 7fb39f9d2640 10 monclient: _renew_subs 2024-03-27T21:41:33.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.553+0000 7fb39f9d2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:33.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.553+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3981d61d0 con 0x7fb39813b020 2024-03-27T21:41:33.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.553+0000 7fb39f9d2640 1 librados: init done 2024-03-27T21:41:33.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.554+0000 7fb39f9d2640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:41:33.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.554+0000 7fb39f9d2640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:33.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.554+0000 7fb39f9d2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:33.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.554+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fb3981d9d20 con 0x7fb39813b020 2024-03-27T21:41:33.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.554+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb38c00cc00 con 0x7fb39813b020 2024-03-27T21:41:33.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.555+0000 7fb389ffb640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb37c06c5e0 0x7fb37c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:33.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.555+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fb38c027360 con 0x7fb39813b020 2024-03-27T21:41:33.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.555+0000 7fb389ffb640 10 client.4826.objecter ms_dispatch 0x7fb398000b90 osd_map(66..66 src has 1..66) v4 2024-03-27T21:41:33.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.555+0000 7fb389ffb640 3 client.4826.objecter handle_osd_map got epochs [66,66] > 0 2024-03-27T21:41:33.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.555+0000 7fb389ffb640 3 client.4826.objecter handle_osd_map decoding full epoch 66 2024-03-27T21:41:33.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.555+0000 7fb39d76e640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb37c06c5e0 0x7fb37c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:33.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.555+0000 7fb389ffb640 20 client.4826.objecter dump_active .. 0 homeless 2024-03-27T21:41:33.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.555+0000 7fb39d76e640 10 monclient: get_auth_request con 0x7fb37c06c5e0 auth_method 0 2024-03-27T21:41:33.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.555+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v66) v1 ==== 54+0+16 (secure 0 0 0) 0x7fb38c05c860 con 0x7fb39813b020 2024-03-27T21:41:33.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.555+0000 7fb389ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:33.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.555+0000 7fb389ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:33.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.555+0000 7fb39f9d2640 10 client.4826.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:33.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.555+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign -1 1 2024-03-27T21:41:33.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.555+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fb3981d6450 0x7fb39813bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:33.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.556+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb398144990 osd=0 3 2024-03-27T21:41:33.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.556+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb398144990 osd=0 3 2024-03-27T21:41:33.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.556+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb398144990 osd=0 3 2024-03-27T21:41:33.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.556+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_remove -1 1 2024-03-27T21:41:33.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.556+0000 7fb39f9d2640 20 client.4826.objecter get_session s=0x7fb398144990 osd=0 3 2024-03-27T21:41:33.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.556+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign 0 1 2024-03-27T21:41:33.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.556+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb398144990 osd=0 4 2024-03-27T21:41:33.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.556+0000 7fb39f9d2640 10 client.4826.objecter _send_command 1 2024-03-27T21:41:33.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.556+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3981d9910 con 0x7fb3981d6450 2024-03-27T21:41:33.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.556+0000 7fb39df6f640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fb3981d6450 0x7fb39813bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:33.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.556+0000 7fb39df6f640 10 monclient: get_auth_request con 0x7fb3981d6450 auth_method 0 2024-03-27T21:41:33.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.556+0000 7fb389ffb640 10 client.4826.objecter ms_handle_connect 0x7fb37c06c5e0 2024-03-27T21:41:33.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.556+0000 7fb39d76e640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb37c06c5e0 0x7fb37c06eaa0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb39400a970 tx=0x7fb394008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:33.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.556+0000 7fb389ffb640 10 client.4826.objecter ms_handle_connect 0x7fb3981d6450 2024-03-27T21:41:33.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.556+0000 7fb39df6f640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fb3981d6450 0x7fb39813bed0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:33.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.557+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fb3981d9910 con 0x7fb3981d6450 2024-03-27T21:41:33.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.557+0000 7fb389ffb640 10 client.4826.objecter ms_dispatch 0x7fb398000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:41:33.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.557+0000 7fb389ffb640 10 client.4826.objecter _finish_command 1 = system:0 2024-03-27T21:41:33.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.557+0000 7fb389ffb640 20 client.4826.objecter put_session s=0x7fb398144990 osd=0 4 2024-03-27T21:41:33.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.557+0000 7fb389ffb640 15 client.4826.objecter _session_command_op_remove 0 1 2024-03-27T21:41:33.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.585+0000 7fb39f9d2640 10 client.4826.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:33.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.585+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign -1 2 2024-03-27T21:41:33.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.585+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb398144990 osd=0 3 2024-03-27T21:41:33.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.585+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb398144990 osd=0 3 2024-03-27T21:41:33.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.585+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb398144990 osd=0 3 2024-03-27T21:41:33.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.585+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_remove -1 2 2024-03-27T21:41:33.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.585+0000 7fb39f9d2640 20 client.4826.objecter get_session s=0x7fb398144990 osd=0 3 2024-03-27T21:41:33.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.585+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign 0 2 2024-03-27T21:41:33.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.585+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb398144990 osd=0 4 2024-03-27T21:41:33.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.585+0000 7fb39f9d2640 10 client.4826.objecter _send_command 2 2024-03-27T21:41:33.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.585+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb398069470 con 0x7fb3981d6450 2024-03-27T21:41:33.573 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:33.587+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:33.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.588+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb398069470 con 0x7fb3981d6450 2024-03-27T21:41:33.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.588+0000 7fb389ffb640 10 client.4826.objecter ms_dispatch 0x7fb398000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:33.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.588+0000 7fb389ffb640 10 client.4826.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:33.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.588+0000 7fb389ffb640 20 client.4826.objecter put_session s=0x7fb398144990 osd=0 4 2024-03-27T21:41:33.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.588+0000 7fb389ffb640 15 client.4826.objecter _session_command_op_remove 0 2 2024-03-27T21:41:33.575 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:41:33.576 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:41:33.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.588+0000 7fb39f9d2640 10 client.4826.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:33.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.588+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign -1 3 2024-03-27T21:41:33.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.588+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fb39813c580 0x7fb39813a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:33.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.588+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb39813c410 osd=1 3 2024-03-27T21:41:33.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.588+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb39813c410 osd=1 3 2024-03-27T21:41:33.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.588+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb39813c410 osd=1 3 2024-03-27T21:41:33.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.588+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_remove -1 3 2024-03-27T21:41:33.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.588+0000 7fb39f9d2640 20 client.4826.objecter get_session s=0x7fb39813c410 osd=1 3 2024-03-27T21:41:33.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.588+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign 1 3 2024-03-27T21:41:33.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.588+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb39813c410 osd=1 4 2024-03-27T21:41:33.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.588+0000 7fb39f9d2640 10 client.4826.objecter _send_command 3 2024-03-27T21:41:33.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.588+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb39806ff10 con 0x7fb39813c580 2024-03-27T21:41:33.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.588+0000 7fb39d76e640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fb39813c580 0x7fb39813a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:33.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.588+0000 7fb39d76e640 10 monclient: get_auth_request con 0x7fb39813c580 auth_method 0 2024-03-27T21:41:33.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.588+0000 7fb389ffb640 10 client.4826.objecter ms_handle_connect 0x7fb39813c580 2024-03-27T21:41:33.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.588+0000 7fb39d76e640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fb39813c580 0x7fb39813a2f0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:33.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.589+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fb39806ff10 con 0x7fb39813c580 2024-03-27T21:41:33.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.589+0000 7fb389ffb640 10 client.4826.objecter ms_dispatch 0x7fb398000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:41:33.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.589+0000 7fb389ffb640 10 client.4826.objecter _finish_command 3 = system:0 2024-03-27T21:41:33.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.589+0000 7fb389ffb640 20 client.4826.objecter put_session s=0x7fb39813c410 osd=1 4 2024-03-27T21:41:33.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.589+0000 7fb389ffb640 15 client.4826.objecter _session_command_op_remove 1 3 2024-03-27T21:41:33.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.617+0000 7fb39f9d2640 10 client.4826.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:33.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.617+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign -1 4 2024-03-27T21:41:33.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.617+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb39813c410 osd=1 3 2024-03-27T21:41:33.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.617+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb39813c410 osd=1 3 2024-03-27T21:41:33.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.617+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb39813c410 osd=1 3 2024-03-27T21:41:33.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.617+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_remove -1 4 2024-03-27T21:41:33.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.617+0000 7fb39f9d2640 20 client.4826.objecter get_session s=0x7fb39813c410 osd=1 3 2024-03-27T21:41:33.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.617+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign 1 4 2024-03-27T21:41:33.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.617+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb39813c410 osd=1 4 2024-03-27T21:41:33.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.617+0000 7fb39f9d2640 10 client.4826.objecter _send_command 4 2024-03-27T21:41:33.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.617+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb39813a830 con 0x7fb39813c580 2024-03-27T21:41:33.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.620+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb39813a830 con 0x7fb39813c580 2024-03-27T21:41:33.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.620+0000 7fb389ffb640 10 client.4826.objecter ms_dispatch 0x7fb398000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:33.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.620+0000 7fb389ffb640 10 client.4826.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:33.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.620+0000 7fb389ffb640 20 client.4826.objecter put_session s=0x7fb39813c410 osd=1 4 2024-03-27T21:41:33.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.620+0000 7fb389ffb640 15 client.4826.objecter _session_command_op_remove 1 4 2024-03-27T21:41:33.607 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:41:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:41:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.620+0000 7fb39f9d2640 10 client.4826.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.620+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign -1 5 2024-03-27T21:41:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.620+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7fb398065fa0 0x7fb3980663c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.620+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb39813ac40 osd=2 3 2024-03-27T21:41:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.620+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb39813ac40 osd=2 3 2024-03-27T21:41:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.620+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb39813ac40 osd=2 3 2024-03-27T21:41:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.620+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_remove -1 5 2024-03-27T21:41:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.620+0000 7fb39f9d2640 20 client.4826.objecter get_session s=0x7fb39813ac40 osd=2 3 2024-03-27T21:41:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.620+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign 2 5 2024-03-27T21:41:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.620+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb39813ac40 osd=2 4 2024-03-27T21:41:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.620+0000 7fb39f9d2640 10 client.4826.objecter _send_command 5 2024-03-27T21:41:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.620+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb398066900 con 0x7fb398065fa0 2024-03-27T21:41:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.620+0000 7fb39cf6d640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7fb398065fa0 0x7fb3980663c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.620+0000 7fb39cf6d640 10 monclient: get_auth_request con 0x7fb398065fa0 auth_method 0 2024-03-27T21:41:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.621+0000 7fb389ffb640 10 client.4826.objecter ms_handle_connect 0x7fb398065fa0 2024-03-27T21:41:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.621+0000 7fb39cf6d640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7fb398065fa0 0x7fb3980663c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.621+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== osd.2 v2:172.21.15.60:6824/56434 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fb398066900 con 0x7fb398065fa0 2024-03-27T21:41:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.621+0000 7fb389ffb640 10 client.4826.objecter ms_dispatch 0x7fb398000b90 command_reply(tid 5: 0 ) v1 2024-03-27T21:41:33.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.621+0000 7fb389ffb640 10 client.4826.objecter _finish_command 5 = system:0 2024-03-27T21:41:33.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.621+0000 7fb389ffb640 20 client.4826.objecter put_session s=0x7fb39813ac40 osd=2 4 2024-03-27T21:41:33.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.621+0000 7fb389ffb640 15 client.4826.objecter _session_command_op_remove 2 5 2024-03-27T21:41:33.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.640+0000 7fb39f9d2640 10 client.4826.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:33.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.640+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign -1 6 2024-03-27T21:41:33.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.640+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb39813ac40 osd=2 3 2024-03-27T21:41:33.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.640+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb39813ac40 osd=2 3 2024-03-27T21:41:33.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.640+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb39813ac40 osd=2 3 2024-03-27T21:41:33.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.640+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_remove -1 6 2024-03-27T21:41:33.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.640+0000 7fb39f9d2640 20 client.4826.objecter get_session s=0x7fb39813ac40 osd=2 3 2024-03-27T21:41:33.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.640+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign 2 6 2024-03-27T21:41:33.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.640+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb39813ac40 osd=2 4 2024-03-27T21:41:33.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.640+0000 7fb39f9d2640 10 client.4826.objecter _send_command 6 2024-03-27T21:41:33.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.640+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb3980b5500 con 0x7fb398065fa0 2024-03-27T21:41:33.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.643+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== osd.2 v2:172.21.15.60:6824/56434 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb3980b5500 con 0x7fb398065fa0 2024-03-27T21:41:33.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.643+0000 7fb389ffb640 10 client.4826.objecter ms_dispatch 0x7fb398000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:33.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.643+0000 7fb389ffb640 10 client.4826.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:33.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.643+0000 7fb389ffb640 20 client.4826.objecter put_session s=0x7fb39813ac40 osd=2 4 2024-03-27T21:41:33.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.643+0000 7fb389ffb640 15 client.4826.objecter _session_command_op_remove 2 6 2024-03-27T21:41:33.630 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2: {} 2024-03-27T21:41:33.630 INFO:teuthology.orchestra.run.smithi060.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-27T21:41:33.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.643+0000 7fb39f9d2640 10 client.4826.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.643+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign -1 7 2024-03-27T21:41:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.643+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fb398142f50 0x7fb398143370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.643+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb3980b5910 osd=3 3 2024-03-27T21:41:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.643+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb3980b5910 osd=3 3 2024-03-27T21:41:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.643+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb3980b5910 osd=3 3 2024-03-27T21:41:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.643+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_remove -1 7 2024-03-27T21:41:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.643+0000 7fb39f9d2640 20 client.4826.objecter get_session s=0x7fb3980b5910 osd=3 3 2024-03-27T21:41:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.643+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign 3 7 2024-03-27T21:41:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.643+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb3980b5910 osd=3 4 2024-03-27T21:41:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.643+0000 7fb39f9d2640 10 client.4826.objecter _send_command 7 2024-03-27T21:41:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.643+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3981e74c0 con 0x7fb398142f50 2024-03-27T21:41:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.643+0000 7fb39df6f640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fb398142f50 0x7fb398143370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.643+0000 7fb39df6f640 10 monclient: get_auth_request con 0x7fb398142f50 auth_method 0 2024-03-27T21:41:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.644+0000 7fb389ffb640 10 client.4826.objecter ms_handle_connect 0x7fb398142f50 2024-03-27T21:41:33.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.644+0000 7fb39df6f640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fb398142f50 0x7fb398143370 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:33.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.645+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fb3981e74c0 con 0x7fb398142f50 2024-03-27T21:41:33.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.645+0000 7fb389ffb640 10 client.4826.objecter ms_dispatch 0x7fb398000b90 command_reply(tid 7: 0 ) v1 2024-03-27T21:41:33.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.645+0000 7fb389ffb640 10 client.4826.objecter _finish_command 7 = system:0 2024-03-27T21:41:33.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.645+0000 7fb389ffb640 20 client.4826.objecter put_session s=0x7fb3980b5910 osd=3 4 2024-03-27T21:41:33.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.645+0000 7fb389ffb640 15 client.4826.objecter _session_command_op_remove 3 7 2024-03-27T21:41:33.645 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:33.645 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:41:33.645 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:41:33.645 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-03-27T21:41:33.645 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.645 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:20 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2024-03-27T21:41:33.645 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:41:33.645 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 113.769654727, 2024-03-27T21:41:33.645 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015345630000000001, 2024-03-27T21:41:33.645 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:33.645 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:33.645 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:33.646 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.646 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:33.646 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:41:33.646 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.646 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.646 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.646 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:33.646 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:41:33.646 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.646 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.646 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.646 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:33.646 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888562+0000", 2024-03-27T21:41:33.646 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:33.647 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.647 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.647 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:33.647 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888735+0000", 2024-03-27T21:41:33.647 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017254799999999999 2024-03-27T21:41:33.647 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.647 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.647 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:33.647 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888747+0000", 2024-03-27T21:41:33.647 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2469999999999999e-05 2024-03-27T21:41:33.647 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.647 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.647 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:33.648 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888761+0000", 2024-03-27T21:41:33.648 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3254e-05 2024-03-27T21:41:33.648 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.648 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.648 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:33.648 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888804+0000", 2024-03-27T21:41:33.648 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3129e-05 2024-03-27T21:41:33.648 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.648 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.648 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:33.648 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888820+0000", 2024-03-27T21:41:33.648 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6426000000000002e-05 2024-03-27T21:41:33.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.649 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.649 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:33.649 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890060+0000", 2024-03-27T21:41:33.649 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012398050000000001 2024-03-27T21:41:33.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.649 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.649 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:33.649 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890098+0000", 2024-03-27T21:41:33.649 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7795e-05 2024-03-27T21:41:33.649 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.649 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:33.649 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.649 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.650 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2024-03-27T21:41:33.650 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:41:33.650 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 113.768830327, 2024-03-27T21:41:33.650 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001069555, 2024-03-27T21:41:33.650 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:33.650 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:33.650 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:33.650 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.650 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:33.650 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:41:33.650 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.650 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.650 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.650 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:33.651 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:41:33.651 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.651 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.651 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.651 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:33.651 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889387+0000", 2024-03-27T21:41:33.651 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:33.651 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.651 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.651 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:33.651 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889455+0000", 2024-03-27T21:41:33.651 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7930999999999997e-05 2024-03-27T21:41:33.651 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.651 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.651 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:33.652 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889470+0000", 2024-03-27T21:41:33.652 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5554e-05 2024-03-27T21:41:33.652 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.652 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.652 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:33.652 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889476+0000", 2024-03-27T21:41:33.652 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0619999999999997e-06 2024-03-27T21:41:33.652 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.652 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.652 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:33.652 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889513+0000", 2024-03-27T21:41:33.652 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6606999999999997e-05 2024-03-27T21:41:33.652 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.652 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.653 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:33.653 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889528+0000", 2024-03-27T21:41:33.653 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4982e-05 2024-03-27T21:41:33.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.653 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.653 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:33.653 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890421+0000", 2024-03-27T21:41:33.653 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00089293700000000001 2024-03-27T21:41:33.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.653 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.653 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:33.654 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890457+0000", 2024-03-27T21:41:33.654 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6260000000000002e-05 2024-03-27T21:41:33.654 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.654 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:33.654 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.654 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.654 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.654 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:38 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'24, mlcod=17'22)", 2024-03-27T21:41:33.654 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:41:33.654 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 113.751594361, 2024-03-27T21:41:33.654 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016182080000000001, 2024-03-27T21:41:33.654 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:33.654 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:33.654 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:33.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.655 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:33.655 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:41:33.655 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.655 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:33.655 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:41:33.655 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.655 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:33.655 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906623+0000", 2024-03-27T21:41:33.655 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:33.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.656 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.656 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:33.656 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906687+0000", 2024-03-27T21:41:33.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3706000000000003e-05 2024-03-27T21:41:33.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.656 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.656 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:33.656 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906698+0000", 2024-03-27T21:41:33.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1752e-05 2024-03-27T21:41:33.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.656 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.656 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:33.656 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906704+0000", 2024-03-27T21:41:33.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3940000000000004e-06 2024-03-27T21:41:33.657 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.657 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.657 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:33.657 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906741+0000", 2024-03-27T21:41:33.657 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6944999999999999e-05 2024-03-27T21:41:33.657 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.657 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.657 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:33.657 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906760+0000", 2024-03-27T21:41:33.657 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9293e-05 2024-03-27T21:41:33.657 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.657 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.657 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:33.657 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908211+0000", 2024-03-27T21:41:33.657 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014511350000000001 2024-03-27T21:41:33.658 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.658 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.658 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:33.658 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908242+0000", 2024-03-27T21:41:33.658 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0712e-05 2024-03-27T21:41:33.658 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.658 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:33.658 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.658 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.658 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.658 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:39 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'25, mlcod=17'22)", 2024-03-27T21:41:33.658 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:33.658 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 113.750749446, 2024-03-27T21:41:33.658 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012060199999999999, 2024-03-27T21:41:33.658 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:33.659 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:33.659 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:33.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.659 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:33.659 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:33.659 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.659 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.659 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:33.659 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:33.659 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.659 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.660 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.661 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:33.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:33.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:33.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.661 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:33.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907533+0000", 2024-03-27T21:41:33.662 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5675999999999999e-05 2024-03-27T21:41:33.662 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.662 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.662 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:33.662 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907545+0000", 2024-03-27T21:41:33.662 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.115e-05 2024-03-27T21:41:33.662 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.662 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.662 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:33.662 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907550+0000", 2024-03-27T21:41:33.662 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8320000000000002e-06 2024-03-27T21:41:33.662 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.662 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.662 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:33.663 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907592+0000", 2024-03-27T21:41:33.663 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1309e-05 2024-03-27T21:41:33.663 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.663 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:33.663 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907611+0000", 2024-03-27T21:41:33.663 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9009e-05 2024-03-27T21:41:33.663 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.663 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:33.663 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908638+0000", 2024-03-27T21:41:33.663 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001027533 2024-03-27T21:41:33.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.664 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:33.664 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908675+0000", 2024-03-27T21:41:33.664 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6329999999999999e-05 2024-03-27T21:41:33.664 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.664 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:33.664 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.664 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:57 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'35, mlcod=17'33)", 2024-03-27T21:41:33.664 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:41:33.665 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 113.736665014, 2024-03-27T21:41:33.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001346578, 2024-03-27T21:41:33.665 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:33.665 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:33.665 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:33.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.665 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:33.665 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:41:33.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.665 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:33.665 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:41:33.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.666 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:33.666 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921552+0000", 2024-03-27T21:41:33.666 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:33.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.666 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:33.666 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921615+0000", 2024-03-27T21:41:33.666 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2787999999999996e-05 2024-03-27T21:41:33.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.666 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:33.666 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921626+0000", 2024-03-27T21:41:33.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0888e-05 2024-03-27T21:41:33.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:33.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921635+0000", 2024-03-27T21:41:33.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4639999999999995e-06 2024-03-27T21:41:33.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:33.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921673+0000", 2024-03-27T21:41:33.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7891e-05 2024-03-27T21:41:33.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:33.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921694+0000", 2024-03-27T21:41:33.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0426000000000001e-05 2024-03-27T21:41:33.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.668 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:33.668 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922865+0000", 2024-03-27T21:41:33.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011709509999999999 2024-03-27T21:41:33.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.668 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:33.668 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922900+0000", 2024-03-27T21:41:33.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4819e-05 2024-03-27T21:41:33.668 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.668 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:33.669 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.669 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:71 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'45, mlcod=17'43)", 2024-03-27T21:41:33.669 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:41:33.669 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 113.72136630200001, 2024-03-27T21:41:33.669 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001460907, 2024-03-27T21:41:33.669 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:33.669 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:33.669 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:33.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.669 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:33.669 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:41:33.669 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.670 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:33.670 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:41:33.670 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.670 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.670 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:33.670 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936851+0000", 2024-03-27T21:41:33.670 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:33.670 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.670 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:33.670 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936903+0000", 2024-03-27T21:41:33.670 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2302999999999997e-05 2024-03-27T21:41:33.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.671 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:33.671 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936915+0000", 2024-03-27T21:41:33.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1967000000000001e-05 2024-03-27T21:41:33.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.671 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:33.671 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936923+0000", 2024-03-27T21:41:33.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9370000000000004e-06 2024-03-27T21:41:33.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.671 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:33.671 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936970+0000", 2024-03-27T21:41:33.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6987000000000003e-05 2024-03-27T21:41:33.672 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.672 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:33.672 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937002+0000", 2024-03-27T21:41:33.672 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2410999999999999e-05 2024-03-27T21:41:33.672 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.672 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:33.672 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938279+0000", 2024-03-27T21:41:33.672 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001276552 2024-03-27T21:41:33.672 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.672 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:33.672 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938313+0000", 2024-03-27T21:41:33.672 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3526999999999999e-05 2024-03-27T21:41:33.673 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.673 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:33.673 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.673 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:72 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'46, mlcod=17'43)", 2024-03-27T21:41:33.673 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:41:33.673 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 113.72072874, 2024-03-27T21:41:33.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014239490000000001, 2024-03-27T21:41:33.673 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:33.673 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:33.673 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:33.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.673 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:33.674 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:41:33.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.674 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:33.674 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:41:33.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.675 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:33.675 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937488+0000", 2024-03-27T21:41:33.675 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:33.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.675 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:33.675 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937547+0000", 2024-03-27T21:41:33.675 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8196999999999999e-05 2024-03-27T21:41:33.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.675 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:33.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937559+0000", 2024-03-27T21:41:33.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1928e-05 2024-03-27T21:41:33.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.676 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:33.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937564+0000", 2024-03-27T21:41:33.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4759999999999997e-06 2024-03-27T21:41:33.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.676 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:33.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937621+0000", 2024-03-27T21:41:33.677 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6712000000000002e-05 2024-03-27T21:41:33.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.677 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:33.677 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937640+0000", 2024-03-27T21:41:33.677 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9517e-05 2024-03-27T21:41:33.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.677 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:33.677 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938883+0000", 2024-03-27T21:41:33.677 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001242567 2024-03-27T21:41:33.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:33.678 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938913+0000", 2024-03-27T21:41:33.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0287e-05 2024-03-27T21:41:33.678 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.678 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:33.678 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.678 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:73 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'47, mlcod=17'43)", 2024-03-27T21:41:33.678 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:33.678 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 113.720052646, 2024-03-27T21:41:33.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010746239999999999, 2024-03-27T21:41:33.679 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:33.679 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:33.679 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:33.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.679 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:33.679 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:33.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.679 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:33.679 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:33.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:33.680 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:33.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:33.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:33.680 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938217+0000", 2024-03-27T21:41:33.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2743e-05 2024-03-27T21:41:33.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:33.680 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938229+0000", 2024-03-27T21:41:33.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1956999999999999e-05 2024-03-27T21:41:33.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:33.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938235+0000", 2024-03-27T21:41:33.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4589999999999997e-06 2024-03-27T21:41:33.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:33.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938341+0000", 2024-03-27T21:41:33.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000106054 2024-03-27T21:41:33.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:33.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938369+0000", 2024-03-27T21:41:33.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8291000000000001e-05 2024-03-27T21:41:33.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.682 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:33.682 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939209+0000", 2024-03-27T21:41:33.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084045400000000003 2024-03-27T21:41:33.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.682 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:33.682 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939240+0000", 2024-03-27T21:41:33.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0429999999999998e-05 2024-03-27T21:41:33.682 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.682 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:33.682 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.683 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:75 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'49, mlcod=17'47)", 2024-03-27T21:41:33.683 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:41:33.683 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 113.713614999, 2024-03-27T21:41:33.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001017001, 2024-03-27T21:41:33.683 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:33.683 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:33.683 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:33.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.683 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:33.683 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:41:33.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.683 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:33.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:41:33.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:33.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944601+0000", 2024-03-27T21:41:33.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999986 2024-03-27T21:41:33.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:33.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944614+0000", 2024-03-27T21:41:33.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2496e-05 2024-03-27T21:41:33.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:33.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944640+0000", 2024-03-27T21:41:33.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5865999999999999e-05 2024-03-27T21:41:33.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:33.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944651+0000", 2024-03-27T21:41:33.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1797000000000001e-05 2024-03-27T21:41:33.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:33.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944762+0000", 2024-03-27T21:41:33.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110412 2024-03-27T21:41:33.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.686 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:33.686 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944820+0000", 2024-03-27T21:41:33.686 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7649999999999999e-05 2024-03-27T21:41:33.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.686 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:33.686 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945585+0000", 2024-03-27T21:41:33.686 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076562600000000005 2024-03-27T21:41:33.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.687 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:33.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945620+0000", 2024-03-27T21:41:33.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4764000000000001e-05 2024-03-27T21:41:33.687 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.687 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:33.687 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.687 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:86 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'50)", 2024-03-27T21:41:33.687 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:33.687 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 113.705853512, 2024-03-27T21:41:33.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011392380000000001, 2024-03-27T21:41:33.687 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:33.688 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:33.688 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:33.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:33.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:33.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:33.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:33.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:33.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:33.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:33.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:33.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952373+0000", 2024-03-27T21:41:33.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9220000000000003e-06 2024-03-27T21:41:33.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:33.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952381+0000", 2024-03-27T21:41:33.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3070000000000003e-06 2024-03-27T21:41:33.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:33.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952385+0000", 2024-03-27T21:41:33.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.121e-06 2024-03-27T21:41:33.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:33.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952419+0000", 2024-03-27T21:41:33.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4082999999999999e-05 2024-03-27T21:41:33.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:33.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952440+0000", 2024-03-27T21:41:33.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0837999999999999e-05 2024-03-27T21:41:33.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:33.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953472+0000", 2024-03-27T21:41:33.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001032119 2024-03-27T21:41:33.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:33.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953504+0000", 2024-03-27T21:41:33.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1250000000000001e-05 2024-03-27T21:41:33.692 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.692 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:33.692 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.692 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:87 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-27T21:41:33.692 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:41:33.692 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 113.705486255, 2024-03-27T21:41:33.692 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00135519, 2024-03-27T21:41:33.692 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:33.692 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:33.692 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:33.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.693 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:33.693 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:41:33.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.693 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:33.693 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:41:33.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.693 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:33.693 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952731+0000", 2024-03-27T21:41:33.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:33.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:33.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952781+0000", 2024-03-27T21:41:33.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0161000000000003e-05 2024-03-27T21:41:33.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:33.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952794+0000", 2024-03-27T21:41:33.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2853e-05 2024-03-27T21:41:33.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:33.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952799+0000", 2024-03-27T21:41:33.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5560000000000001e-06 2024-03-27T21:41:33.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:33.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952832+0000", 2024-03-27T21:41:33.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3834999999999997e-05 2024-03-27T21:41:33.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:33.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952851+0000", 2024-03-27T21:41:33.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8289999999999999e-05 2024-03-27T21:41:33.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:33.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954055+0000", 2024-03-27T21:41:33.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012037949999999999 2024-03-27T21:41:33.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:33.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954087+0000", 2024-03-27T21:41:33.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2292999999999997e-05 2024-03-27T21:41:33.696 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.696 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:33.696 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.696 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:89 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'59, mlcod=17'57)", 2024-03-27T21:41:33.696 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:41:33.696 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 113.703212034, 2024-03-27T21:41:33.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00138893, 2024-03-27T21:41:33.697 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:33.697 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:33.697 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:33.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.697 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:33.697 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:41:33.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.697 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:33.697 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:41:33.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:33.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955005+0000", 2024-03-27T21:41:33.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:33.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:33.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955071+0000", 2024-03-27T21:41:33.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5290999999999995e-05 2024-03-27T21:41:33.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:33.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955081+0000", 2024-03-27T21:41:33.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0572e-05 2024-03-27T21:41:33.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:33.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955086+0000", 2024-03-27T21:41:33.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8380000000000001e-06 2024-03-27T21:41:33.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:33.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955122+0000", 2024-03-27T21:41:33.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5698000000000003e-05 2024-03-27T21:41:33.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:33.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955140+0000", 2024-03-27T21:41:33.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7900000000000001e-05 2024-03-27T21:41:33.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:33.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956366+0000", 2024-03-27T21:41:33.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001226471 2024-03-27T21:41:33.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.701 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:33.701 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956395+0000", 2024-03-27T21:41:33.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8782e-05 2024-03-27T21:41:33.701 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.701 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:33.701 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.701 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:90 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'60, mlcod=17'57)", 2024-03-27T21:41:33.701 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:33.701 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 113.702527457, 2024-03-27T21:41:33.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010778859999999999, 2024-03-27T21:41:33.702 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:33.702 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:33.702 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:33.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.702 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:33.702 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:33.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.702 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:33.702 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:33.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:33.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:33.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:33.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:33.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955747+0000", 2024-03-27T21:41:33.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7130999999999999e-05 2024-03-27T21:41:33.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:33.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955756+0000", 2024-03-27T21:41:33.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2750000000000001e-06 2024-03-27T21:41:33.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:33.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955761+0000", 2024-03-27T21:41:33.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3900000000000003e-06 2024-03-27T21:41:33.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:33.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955797+0000", 2024-03-27T21:41:33.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6650000000000003e-05 2024-03-27T21:41:33.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:33.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955816+0000", 2024-03-27T21:41:33.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8195000000000001e-05 2024-03-27T21:41:33.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:33.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956739+0000", 2024-03-27T21:41:33.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00092369700000000002 2024-03-27T21:41:33.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:33.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956768+0000", 2024-03-27T21:41:33.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9048e-05 2024-03-27T21:41:33.705 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.705 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:33.705 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.705 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:118 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'80, mlcod=17'78)", 2024-03-27T21:41:33.706 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:33.706 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 113.569536634, 2024-03-27T21:41:33.706 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014200669999999999, 2024-03-27T21:41:33.706 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:33.706 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:33.706 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:33.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.706 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:33.706 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:33.706 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.706 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.706 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:33.706 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:33.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:33.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:33.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:33.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:33.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088756+0000", 2024-03-27T21:41:33.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4902999999999998e-05 2024-03-27T21:41:33.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:33.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088764+0000", 2024-03-27T21:41:33.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8169999999999999e-06 2024-03-27T21:41:33.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.708 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:33.708 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088767+0000", 2024-03-27T21:41:33.708 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8310000000000003e-06 2024-03-27T21:41:33.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.708 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:33.708 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088798+0000", 2024-03-27T21:41:33.708 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0800999999999999e-05 2024-03-27T21:41:33.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.709 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:33.709 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088817+0000", 2024-03-27T21:41:33.709 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8467000000000001e-05 2024-03-27T21:41:33.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.709 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:33.709 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090074+0000", 2024-03-27T21:41:33.709 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012574089999999999 2024-03-27T21:41:33.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.709 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:33.709 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090101+0000", 2024-03-27T21:41:33.709 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7223e-05 2024-03-27T21:41:33.710 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.710 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:33.710 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.710 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.710 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:119 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'81, mlcod=17'78)", 2024-03-27T21:41:33.710 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:33.710 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 113.568801839, 2024-03-27T21:41:33.710 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014990870000000001, 2024-03-27T21:41:33.710 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:33.710 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:33.710 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:33.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.711 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:33.711 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:33.711 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.711 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:33.711 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:33.711 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.711 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:33.711 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:33.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:33.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.712 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:33.712 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089491+0000", 2024-03-27T21:41:33.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4805e-05 2024-03-27T21:41:33.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.712 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:33.712 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089498+0000", 2024-03-27T21:41:33.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0650000000000001e-06 2024-03-27T21:41:33.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:33.713 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089501+0000", 2024-03-27T21:41:33.713 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.54e-06 2024-03-27T21:41:33.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:33.713 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089533+0000", 2024-03-27T21:41:33.713 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1838999999999998e-05 2024-03-27T21:41:33.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:33.713 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089550+0000", 2024-03-27T21:41:33.713 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7465999999999998e-05 2024-03-27T21:41:33.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.714 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:33.714 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090890+0000", 2024-03-27T21:41:33.714 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013397979999999999 2024-03-27T21:41:33.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.714 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:33.714 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090915+0000", 2024-03-27T21:41:33.714 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4905e-05 2024-03-27T21:41:33.714 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.714 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:33.714 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.714 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:120 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'82, mlcod=17'78)", 2024-03-27T21:41:33.715 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:33.715 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 113.568001959, 2024-03-27T21:41:33.715 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001527093, 2024-03-27T21:41:33.715 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:33.715 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:33.715 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:33.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.715 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:33.715 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:33.715 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.716 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:33.716 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:33.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.716 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:33.716 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:33.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:33.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.716 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:33.716 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090267+0000", 2024-03-27T21:41:33.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1477000000000002e-05 2024-03-27T21:41:33.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.717 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:33.717 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090273+0000", 2024-03-27T21:41:33.717 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9519999999999999e-06 2024-03-27T21:41:33.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.717 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:33.717 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090276+0000", 2024-03-27T21:41:33.717 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.219e-06 2024-03-27T21:41:33.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.717 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:33.717 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090304+0000", 2024-03-27T21:41:33.717 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8073999999999999e-05 2024-03-27T21:41:33.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.718 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:33.718 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090316+0000", 2024-03-27T21:41:33.718 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2088e-05 2024-03-27T21:41:33.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.718 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:33.718 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091717+0000", 2024-03-27T21:41:33.718 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014003889999999999 2024-03-27T21:41:33.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.718 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:33.719 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091743+0000", 2024-03-27T21:41:33.719 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6234e-05 2024-03-27T21:41:33.719 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.719 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:33.719 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.719 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.719 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:121 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'83, mlcod=17'78)", 2024-03-27T21:41:33.719 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:33.719 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 113.56719159399999, 2024-03-27T21:41:33.719 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010226899999999999, 2024-03-27T21:41:33.719 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:33.719 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:33.720 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:33.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.720 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:33.720 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:33.720 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.720 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:33.720 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:33.720 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.720 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:33.721 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:33.721 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:33.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.721 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:33.721 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091142+0000", 2024-03-27T21:41:33.721 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011608300000000001 2024-03-27T21:41:33.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.721 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:33.721 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091151+0000", 2024-03-27T21:41:33.721 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2350000000000005e-06 2024-03-27T21:41:33.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.722 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:33.722 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091156+0000", 2024-03-27T21:41:33.722 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5419999999999999e-06 2024-03-27T21:41:33.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.722 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:33.722 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091191+0000", 2024-03-27T21:41:33.722 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5453000000000001e-05 2024-03-27T21:41:33.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.722 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:33.722 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091208+0000", 2024-03-27T21:41:33.723 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7119999999999999e-05 2024-03-27T21:41:33.723 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.723 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.723 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:33.723 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092024+0000", 2024-03-27T21:41:33.723 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000815539 2024-03-27T21:41:33.723 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.723 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.723 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:33.723 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092049+0000", 2024-03-27T21:41:33.723 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5171e-05 2024-03-27T21:41:33.723 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.723 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:33.723 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.723 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.724 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:142 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'100, mlcod=18'98)", 2024-03-27T21:41:33.724 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:33.724 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 113.15218464, 2024-03-27T21:41:33.724 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001198778, 2024-03-27T21:41:33.724 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:33.724 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:33.724 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:33.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.724 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:33.724 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:33.724 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.724 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:33.725 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:33.725 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.725 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:33.725 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:33.725 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:33.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.725 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:33.725 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506101+0000", 2024-03-27T21:41:33.725 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8202999999999997e-05 2024-03-27T21:41:33.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.726 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:33.726 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506109+0000", 2024-03-27T21:41:33.726 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3059999999999998e-06 2024-03-27T21:41:33.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.726 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.726 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:33.726 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506113+0000", 2024-03-27T21:41:33.726 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0119999999999999e-06 2024-03-27T21:41:33.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.726 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.726 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:33.726 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506137+0000", 2024-03-27T21:41:33.726 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4043999999999999e-05 2024-03-27T21:41:33.727 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.727 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:33.727 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506148+0000", 2024-03-27T21:41:33.727 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0446e-05 2024-03-27T21:41:33.727 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.727 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:33.727 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507213+0000", 2024-03-27T21:41:33.727 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001065215 2024-03-27T21:41:33.727 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.727 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:33.728 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507232+0000", 2024-03-27T21:41:33.728 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9038e-05 2024-03-27T21:41:33.728 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.728 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:33.728 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.728 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.728 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.728 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:143 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'101, mlcod=18'98)", 2024-03-27T21:41:33.728 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:33.728 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 113.15157202100001, 2024-03-27T21:41:33.728 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001357488, 2024-03-27T21:41:33.728 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:33.729 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:33.729 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:33.729 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.729 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:33.729 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:33.729 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.729 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.729 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.729 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:33.729 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:33.729 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.729 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.729 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.729 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:33.730 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:33.730 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:33.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.730 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:33.730 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506723+0000", 2024-03-27T21:41:33.730 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7404000000000006e-05 2024-03-27T21:41:33.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.730 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:33.730 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506730+0000", 2024-03-27T21:41:33.730 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5749999999999997e-06 2024-03-27T21:41:33.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.730 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:33.731 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506733+0000", 2024-03-27T21:41:33.731 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7979999999999999e-06 2024-03-27T21:41:33.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.731 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:33.731 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506758+0000", 2024-03-27T21:41:33.731 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4179e-05 2024-03-27T21:41:33.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.731 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:33.731 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506768+0000", 2024-03-27T21:41:33.731 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0740000000000001e-05 2024-03-27T21:41:33.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.732 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:33.732 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507978+0000", 2024-03-27T21:41:33.732 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012093900000000001 2024-03-27T21:41:33.732 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.732 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:33.732 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508003+0000", 2024-03-27T21:41:33.732 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5667e-05 2024-03-27T21:41:33.732 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.732 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:33.732 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.732 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.732 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:144 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'102, mlcod=18'98)", 2024-03-27T21:41:33.733 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:41:33.733 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 113.150913124, 2024-03-27T21:41:33.733 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013453390000000001, 2024-03-27T21:41:33.733 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:33.733 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:33.733 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:33.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.733 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:33.733 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:41:33.733 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.733 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:33.734 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:41:33.734 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:33.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.734 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.734 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:33.734 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507304+0000", 2024-03-27T21:41:33.734 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:33.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.734 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.734 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:33.734 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507397+0000", 2024-03-27T21:41:33.734 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2995e-05 2024-03-27T21:41:33.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.734 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.735 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:33.735 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507403+0000", 2024-03-27T21:41:33.735 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8640000000000001e-06 2024-03-27T21:41:33.735 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.735 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.735 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:33.735 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507407+0000", 2024-03-27T21:41:33.735 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5710000000000002e-06 2024-03-27T21:41:33.735 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.735 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.735 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:33.735 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507428+0000", 2024-03-27T21:41:33.735 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1192e-05 2024-03-27T21:41:33.735 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.736 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:33.736 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507438+0000", 2024-03-27T21:41:33.736 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8309999999999996e-06 2024-03-27T21:41:33.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.736 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:33.736 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508624+0000", 2024-03-27T21:41:33.736 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011861210000000001 2024-03-27T21:41:33.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:33.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:33.736 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:33.736 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508650+0000", 2024-03-27T21:41:33.736 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6118e-05 2024-03-27T21:41:33.737 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.737 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:33.737 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.737 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:33.737 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:33.737 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:33.739 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:41:33.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.672+0000 7fb39f9d2640 10 client.4826.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:33.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.672+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign -1 8 2024-03-27T21:41:33.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.672+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb3980b5910 osd=3 3 2024-03-27T21:41:33.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.672+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb3980b5910 osd=3 3 2024-03-27T21:41:33.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.672+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb3980b5910 osd=3 3 2024-03-27T21:41:33.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.672+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_remove -1 8 2024-03-27T21:41:33.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.672+0000 7fb39f9d2640 20 client.4826.objecter get_session s=0x7fb3980b5910 osd=3 3 2024-03-27T21:41:33.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.672+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign 3 8 2024-03-27T21:41:33.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.672+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb3980b5910 osd=3 4 2024-03-27T21:41:33.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.672+0000 7fb39f9d2640 10 client.4826.objecter _send_command 8 2024-03-27T21:41:33.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.672+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb3981429d0 con 0x7fb398142f50 2024-03-27T21:41:33.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.673+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb3981429d0 con 0x7fb398142f50 2024-03-27T21:41:33.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.674+0000 7fb389ffb640 10 client.4826.objecter ms_dispatch 0x7fb398000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:33.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.674+0000 7fb389ffb640 10 client.4826.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:33.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.674+0000 7fb389ffb640 20 client.4826.objecter put_session s=0x7fb3980b5910 osd=3 4 2024-03-27T21:41:33.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.674+0000 7fb389ffb640 15 client.4826.objecter _session_command_op_remove 3 8 2024-03-27T21:41:33.741 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:41:33.742 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:41:33.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.674+0000 7fb39f9d2640 10 client.4826.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:33.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.674+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign -1 9 2024-03-27T21:41:33.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.674+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fb39802be40 0x7fb3981e5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:33.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.674+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb3981429d0 osd=4 3 2024-03-27T21:41:33.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.674+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb3981429d0 osd=4 3 2024-03-27T21:41:33.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.674+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb3981429d0 osd=4 3 2024-03-27T21:41:33.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.674+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_remove -1 9 2024-03-27T21:41:33.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.674+0000 7fb39f9d2640 20 client.4826.objecter get_session s=0x7fb3981429d0 osd=4 3 2024-03-27T21:41:33.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.674+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign 4 9 2024-03-27T21:41:33.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.674+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb3981429d0 osd=4 4 2024-03-27T21:41:33.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.674+0000 7fb39f9d2640 10 client.4826.objecter _send_command 9 2024-03-27T21:41:33.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.674+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb398142de0 con 0x7fb39802be40 2024-03-27T21:41:33.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.674+0000 7fb39d76e640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fb39802be40 0x7fb3981e5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:33.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.674+0000 7fb39d76e640 10 monclient: get_auth_request con 0x7fb39802be40 auth_method 0 2024-03-27T21:41:33.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.674+0000 7fb389ffb640 10 client.4826.objecter ms_handle_connect 0x7fb39802be40 2024-03-27T21:41:33.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.674+0000 7fb39d76e640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fb39802be40 0x7fb3981e5db0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:33.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.675+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fb398142de0 con 0x7fb39802be40 2024-03-27T21:41:33.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.675+0000 7fb389ffb640 10 client.4826.objecter ms_dispatch 0x7fb398000b90 command_reply(tid 9: 0 ) v1 2024-03-27T21:41:33.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.675+0000 7fb389ffb640 10 client.4826.objecter _finish_command 9 = system:0 2024-03-27T21:41:33.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.675+0000 7fb389ffb640 20 client.4826.objecter put_session s=0x7fb3981429d0 osd=4 4 2024-03-27T21:41:33.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.675+0000 7fb389ffb640 15 client.4826.objecter _session_command_op_remove 4 9 2024-03-27T21:41:33.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.689+0000 7fb39f9d2640 10 client.4826.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:33.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.689+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign -1 10 2024-03-27T21:41:33.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.689+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb3981429d0 osd=4 3 2024-03-27T21:41:33.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.689+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb3981429d0 osd=4 3 2024-03-27T21:41:33.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.689+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb3981429d0 osd=4 3 2024-03-27T21:41:33.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.689+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_remove -1 10 2024-03-27T21:41:33.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.689+0000 7fb39f9d2640 20 client.4826.objecter get_session s=0x7fb3981429d0 osd=4 3 2024-03-27T21:41:33.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.689+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign 4 10 2024-03-27T21:41:33.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.689+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb3981429d0 osd=4 4 2024-03-27T21:41:33.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.689+0000 7fb39f9d2640 10 client.4826.objecter _send_command 10 2024-03-27T21:41:33.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.689+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb398142d80 con 0x7fb39802be40 2024-03-27T21:41:33.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.691+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb398142d80 con 0x7fb39802be40 2024-03-27T21:41:33.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.691+0000 7fb389ffb640 10 client.4826.objecter ms_dispatch 0x7fb398000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:33.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.691+0000 7fb389ffb640 10 client.4826.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:33.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.691+0000 7fb389ffb640 20 client.4826.objecter put_session s=0x7fb3981429d0 osd=4 4 2024-03-27T21:41:33.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.691+0000 7fb389ffb640 15 client.4826.objecter _session_command_op_remove 4 10 2024-03-27T21:41:33.745 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:41:33.747 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:41:33.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.691+0000 7fb39f9d2640 10 client.4826.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:33.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.691+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign -1 11 2024-03-27T21:41:33.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.691+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fb3981e6800 0x7fb3981e6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:33.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.691+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb398142d80 osd=5 3 2024-03-27T21:41:33.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.691+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb398142d80 osd=5 3 2024-03-27T21:41:33.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.691+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb398142d80 osd=5 3 2024-03-27T21:41:33.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.691+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_remove -1 11 2024-03-27T21:41:33.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.691+0000 7fb39f9d2640 20 client.4826.objecter get_session s=0x7fb398142d80 osd=5 3 2024-03-27T21:41:33.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.691+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign 5 11 2024-03-27T21:41:33.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.691+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb398142d80 osd=5 4 2024-03-27T21:41:33.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.691+0000 7fb39f9d2640 10 client.4826.objecter _send_command 11 2024-03-27T21:41:33.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.691+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3981e71a0 con 0x7fb3981e6800 2024-03-27T21:41:33.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.691+0000 7fb39cf6d640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fb3981e6800 0x7fb3981e6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:33.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.691+0000 7fb39cf6d640 10 monclient: get_auth_request con 0x7fb3981e6800 auth_method 0 2024-03-27T21:41:33.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.692+0000 7fb389ffb640 10 client.4826.objecter ms_handle_connect 0x7fb3981e6800 2024-03-27T21:41:33.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.692+0000 7fb39cf6d640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fb3981e6800 0x7fb3981e6c60 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:33.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.692+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fb3981e71a0 con 0x7fb3981e6800 2024-03-27T21:41:33.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.692+0000 7fb389ffb640 10 client.4826.objecter ms_dispatch 0x7fb398000b90 command_reply(tid 11: 0 ) v1 2024-03-27T21:41:33.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.692+0000 7fb389ffb640 10 client.4826.objecter _finish_command 11 = system:0 2024-03-27T21:41:33.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.692+0000 7fb389ffb640 20 client.4826.objecter put_session s=0x7fb398142d80 osd=5 4 2024-03-27T21:41:33.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.692+0000 7fb389ffb640 15 client.4826.objecter _session_command_op_remove 5 11 2024-03-27T21:41:33.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.706+0000 7fb39f9d2640 10 client.4826.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:33.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.706+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign -1 12 2024-03-27T21:41:33.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.706+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb398142d80 osd=5 3 2024-03-27T21:41:33.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.706+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb398142d80 osd=5 3 2024-03-27T21:41:33.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.706+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb398142d80 osd=5 3 2024-03-27T21:41:33.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.706+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_remove -1 12 2024-03-27T21:41:33.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.706+0000 7fb39f9d2640 20 client.4826.objecter get_session s=0x7fb398142d80 osd=5 3 2024-03-27T21:41:33.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.706+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign 5 12 2024-03-27T21:41:33.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.706+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb398142d80 osd=5 4 2024-03-27T21:41:33.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.706+0000 7fb39f9d2640 10 client.4826.objecter _send_command 12 2024-03-27T21:41:33.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.706+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb3981e71a0 con 0x7fb3981e6800 2024-03-27T21:41:33.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.708+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb3981e71a0 con 0x7fb3981e6800 2024-03-27T21:41:33.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.708+0000 7fb389ffb640 10 client.4826.objecter ms_dispatch 0x7fb398000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:33.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.708+0000 7fb389ffb640 10 client.4826.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:33.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.708+0000 7fb389ffb640 20 client.4826.objecter put_session s=0x7fb398142d80 osd=5 4 2024-03-27T21:41:33.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.708+0000 7fb389ffb640 15 client.4826.objecter _session_command_op_remove 5 12 2024-03-27T21:41:33.750 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:41:33.751 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:41:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.708+0000 7fb39f9d2640 10 client.4826.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.708+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign -1 13 2024-03-27T21:41:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.708+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fb3981ebb60 0x7fb3981edf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.708+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb3981e71a0 osd=6 3 2024-03-27T21:41:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.708+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb3981e71a0 osd=6 3 2024-03-27T21:41:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.708+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb3981e71a0 osd=6 3 2024-03-27T21:41:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.708+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_remove -1 13 2024-03-27T21:41:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.708+0000 7fb39f9d2640 20 client.4826.objecter get_session s=0x7fb3981e71a0 osd=6 3 2024-03-27T21:41:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.708+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign 6 13 2024-03-27T21:41:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.708+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb3981e71a0 osd=6 4 2024-03-27T21:41:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.708+0000 7fb39f9d2640 10 client.4826.objecter _send_command 13 2024-03-27T21:41:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.708+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3981f0d20 con 0x7fb3981ebb60 2024-03-27T21:41:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.708+0000 7fb39df6f640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fb3981ebb60 0x7fb3981edf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.708+0000 7fb39df6f640 10 monclient: get_auth_request con 0x7fb3981ebb60 auth_method 0 2024-03-27T21:41:33.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.709+0000 7fb39df6f640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fb3981ebb60 0x7fb3981edf50 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:33.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.709+0000 7fb389ffb640 10 client.4826.objecter ms_handle_connect 0x7fb3981ebb60 2024-03-27T21:41:33.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.709+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fb3981f0d20 con 0x7fb3981ebb60 2024-03-27T21:41:33.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.709+0000 7fb389ffb640 10 client.4826.objecter ms_dispatch 0x7fb398000b90 command_reply(tid 13: 0 ) v1 2024-03-27T21:41:33.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.709+0000 7fb389ffb640 10 client.4826.objecter _finish_command 13 = system:0 2024-03-27T21:41:33.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.709+0000 7fb389ffb640 20 client.4826.objecter put_session s=0x7fb3981e71a0 osd=6 4 2024-03-27T21:41:33.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.709+0000 7fb389ffb640 15 client.4826.objecter _session_command_op_remove 6 13 2024-03-27T21:41:33.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.723+0000 7fb39f9d2640 10 client.4826.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:33.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.723+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign -1 14 2024-03-27T21:41:33.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.723+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb3981e71a0 osd=6 3 2024-03-27T21:41:33.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.723+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb3981e71a0 osd=6 3 2024-03-27T21:41:33.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.723+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb3981e71a0 osd=6 3 2024-03-27T21:41:33.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.723+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_remove -1 14 2024-03-27T21:41:33.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.723+0000 7fb39f9d2640 20 client.4826.objecter get_session s=0x7fb3981e71a0 osd=6 3 2024-03-27T21:41:33.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.723+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign 6 14 2024-03-27T21:41:33.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.723+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb3981e71a0 osd=6 4 2024-03-27T21:41:33.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.723+0000 7fb39f9d2640 10 client.4826.objecter _send_command 14 2024-03-27T21:41:33.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.723+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb3981ee490 con 0x7fb3981ebb60 2024-03-27T21:41:33.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.725+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb3981ee490 con 0x7fb3981ebb60 2024-03-27T21:41:33.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.725+0000 7fb389ffb640 10 client.4826.objecter ms_dispatch 0x7fb398000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:33.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.725+0000 7fb389ffb640 10 client.4826.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:33.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.725+0000 7fb389ffb640 20 client.4826.objecter put_session s=0x7fb3981e71a0 osd=6 4 2024-03-27T21:41:33.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.725+0000 7fb389ffb640 15 client.4826.objecter _session_command_op_remove 6 14 2024-03-27T21:41:33.754 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:41:33.756 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:41:33.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.725+0000 7fb39f9d2640 10 client.4826.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:33.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.725+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign -1 15 2024-03-27T21:41:33.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.725+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fb3981eeb10 0x7fb3981f0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:33.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.725+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb3981ee490 osd=7 3 2024-03-27T21:41:33.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.725+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb3981ee490 osd=7 3 2024-03-27T21:41:33.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.725+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb3981ee490 osd=7 3 2024-03-27T21:41:33.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.725+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_remove -1 15 2024-03-27T21:41:33.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.725+0000 7fb39f9d2640 20 client.4826.objecter get_session s=0x7fb3981ee490 osd=7 3 2024-03-27T21:41:33.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.725+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign 7 15 2024-03-27T21:41:33.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.725+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb3981ee490 osd=7 4 2024-03-27T21:41:33.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.725+0000 7fb39f9d2640 10 client.4826.objecter _send_command 15 2024-03-27T21:41:33.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.725+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3981f3e70 con 0x7fb3981eeb10 2024-03-27T21:41:33.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.725+0000 7fb39d76e640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fb3981eeb10 0x7fb3981f0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:33.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.725+0000 7fb39d76e640 10 monclient: get_auth_request con 0x7fb3981eeb10 auth_method 0 2024-03-27T21:41:33.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.725+0000 7fb389ffb640 10 client.4826.objecter ms_handle_connect 0x7fb3981eeb10 2024-03-27T21:41:33.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.725+0000 7fb39d76e640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fb3981eeb10 0x7fb3981f0f00 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:33.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.726+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fb3981f3e70 con 0x7fb3981eeb10 2024-03-27T21:41:33.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.726+0000 7fb389ffb640 10 client.4826.objecter ms_dispatch 0x7fb398000b90 command_reply(tid 15: 0 ) v1 2024-03-27T21:41:33.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.726+0000 7fb389ffb640 10 client.4826.objecter _finish_command 15 = system:0 2024-03-27T21:41:33.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.726+0000 7fb389ffb640 20 client.4826.objecter put_session s=0x7fb3981ee490 osd=7 4 2024-03-27T21:41:33.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.726+0000 7fb389ffb640 15 client.4826.objecter _session_command_op_remove 7 15 2024-03-27T21:41:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.740+0000 7fb39f9d2640 10 client.4826.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.740+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign -1 16 2024-03-27T21:41:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.740+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb3981ee490 osd=7 3 2024-03-27T21:41:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.740+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb3981ee490 osd=7 3 2024-03-27T21:41:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.740+0000 7fb39f9d2640 20 client.4826.objecter _get_session s=0x7fb3981ee490 osd=7 3 2024-03-27T21:41:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.740+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_remove -1 16 2024-03-27T21:41:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.740+0000 7fb39f9d2640 20 client.4826.objecter get_session s=0x7fb3981ee490 osd=7 3 2024-03-27T21:41:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.740+0000 7fb39f9d2640 15 client.4826.objecter _session_command_op_assign 7 16 2024-03-27T21:41:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.740+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb3981ee490 osd=7 4 2024-03-27T21:41:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.740+0000 7fb39f9d2640 10 client.4826.objecter _send_command 16 2024-03-27T21:41:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.740+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb3981f3e70 con 0x7fb3981eeb10 2024-03-27T21:41:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.741+0000 7fb389ffb640 1 -- 172.21.15.60:0/4279277301 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7fb3981f3e70 con 0x7fb3981eeb10 2024-03-27T21:41:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.741+0000 7fb389ffb640 10 client.4826.objecter ms_dispatch 0x7fb398000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:33.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.741+0000 7fb389ffb640 10 client.4826.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:33.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.741+0000 7fb389ffb640 20 client.4826.objecter put_session s=0x7fb3981ee490 osd=7 4 2024-03-27T21:41:33.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.741+0000 7fb389ffb640 15 client.4826.objecter _session_command_op_remove 7 16 2024-03-27T21:41:33.759 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:41:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:41:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 10 librados: watch_flush enter 2024-03-27T21:41:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 10 librados: watch_flush exit 2024-03-27T21:41:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 10 client.4826.objecter close_session for osd.0 2024-03-27T21:41:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fb3981d6450 msgr2=0x7fb39813bed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fb3981d6450 0x7fb39813bed0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb398144990 osd=0 1 2024-03-27T21:41:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 10 client.4826.objecter close_session for osd.1 2024-03-27T21:41:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fb39813c580 msgr2=0x7fb39813a2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7fb39813c580 0x7fb39813a2f0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb39813c410 osd=1 1 2024-03-27T21:41:33.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 10 client.4826.objecter close_session for osd.2 2024-03-27T21:41:33.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7fb398065fa0 msgr2=0x7fb3980663c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:33.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7fb398065fa0 0x7fb3980663c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb39813ac40 osd=2 1 2024-03-27T21:41:33.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 10 client.4826.objecter close_session for osd.3 2024-03-27T21:41:33.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fb398142f50 msgr2=0x7fb398143370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:33.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fb398142f50 0x7fb398143370 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb3980b5910 osd=3 1 2024-03-27T21:41:33.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 10 client.4826.objecter close_session for osd.4 2024-03-27T21:41:33.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39df6f640 1 -- 172.21.15.60:0/4279277301 reap_dead start 2024-03-27T21:41:33.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fb39802be40 msgr2=0x7fb3981e5db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:33.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fb39802be40 0x7fb3981e5db0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39df6f640 1 -- 172.21.15.60:0/4279277301 reap_dead start 2024-03-27T21:41:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb3981429d0 osd=4 1 2024-03-27T21:41:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 10 client.4826.objecter close_session for osd.5 2024-03-27T21:41:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fb3981e6800 msgr2=0x7fb3981e6c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7fb3981e6800 0x7fb3981e6c60 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb398142d80 osd=5 1 2024-03-27T21:41:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 10 client.4826.objecter close_session for osd.6 2024-03-27T21:41:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fb3981ebb60 msgr2=0x7fb3981edf50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7fb3981ebb60 0x7fb3981edf50 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb3981e71a0 osd=6 1 2024-03-27T21:41:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 10 client.4826.objecter close_session for osd.7 2024-03-27T21:41:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fb3981eeb10 msgr2=0x7fb3981f0f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7fb3981eeb10 0x7fb3981f0f00 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 20 client.4826.objecter put_session s=0x7fb3981ee490 osd=7 1 2024-03-27T21:41:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 20 client.4826.objecter shutdown clearing up homeless session... 2024-03-27T21:41:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 10 client.4826.objecter successfully canceled tick 2024-03-27T21:41:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb37c06c5e0 msgr2=0x7fb37c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb37c06c5e0 0x7fb37c06eaa0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb39400a970 tx=0x7fb394008040 comp rx=0 tx=0).stop 2024-03-27T21:41:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 10 monclient: shutdown 2024-03-27T21:41:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb39813b020 msgr2=0x7fb398143fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 --2- 172.21.15.60:0/4279277301 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb39813b020 0x7fb398143fc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00e9d0 tx=0x7fb38c00ee90 comp rx=0 tx=0).stop 2024-03-27T21:41:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39df6f640 1 -- 172.21.15.60:0/4279277301 reap_dead start 2024-03-27T21:41:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 shutdown_connections 2024-03-27T21:41:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 >> 172.21.15.60:0/4279277301 conn(0x7fb3980608e0 msgr2=0x7fb3980b1950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 shutdown_connections 2024-03-27T21:41:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 -- 172.21.15.60:0/4279277301 wait complete. 2024-03-27T21:41:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:33.742+0000 7fb39f9d2640 1 librados: shutdown 2024-03-27T21:41:33.765 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:33.752+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:33.839 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:33.853+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:33.888 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:33.888 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:33.888 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:33.888 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:33.897 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:41:33.940 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:33.954+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:34.040 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:34.054+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:34.051 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:34.051 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:34.051 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:41:34.051 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:41:34.051 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:34.060 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:41:34.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.082+0000 7f59f57fa640 1 -- 172.21.15.60:0/2429861475 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v67) v1 ==== 113+0+0 (secure 0 0 0) 0x7f59ec035030 con 0x7f59f80b40b0 2024-03-27T21:41:34.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.082+0000 7f59f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd reweight", "id": 5, "weight": 1.0}] 2024-03-27T21:41:34.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.082+0000 7f59f57fa640 10 monclient: _finish_command 2 = system:0 reweighted osd.5 to 1 (10000) 2024-03-27T21:41:34.068 INFO:teuthology.orchestra.run.smithi060.stderr:reweighted osd.5 to 1 (10000) 2024-03-27T21:41:34.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.084+0000 7f59feb18640 10 librados: watch_flush enter 2024-03-27T21:41:34.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.085+0000 7f59feb18640 10 librados: watch_flush exit 2024-03-27T21:41:34.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.085+0000 7f59feb18640 20 client.4820.objecter shutdown clearing up homeless session... 2024-03-27T21:41:34.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.085+0000 7f59feb18640 10 client.4820.objecter successfully canceled tick 2024-03-27T21:41:34.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.085+0000 7f59feb18640 1 -- 172.21.15.60:0/2429861475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f59dc06c590 msgr2=0x7f59dc06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:34.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.085+0000 7f59feb18640 1 --2- 172.21.15.60:0/2429861475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f59dc06c590 0x7f59dc06ea50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f59f80b0720 tx=0x7f59e4009040 comp rx=0 tx=0).stop 2024-03-27T21:41:34.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.085+0000 7f59feb18640 10 monclient: shutdown 2024-03-27T21:41:34.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.085+0000 7f59feb18640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:34.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.085+0000 7f59feb18640 1 -- 172.21.15.60:0/2429861475 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f59f80b40b0 msgr2=0x7f59f81d9bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:34.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.085+0000 7f59feb18640 1 --2- 172.21.15.60:0/2429861475 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f59f80b40b0 0x7f59f81d9bb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f59ec02bee0 tx=0x7f59ec0046c0 comp rx=0 tx=0).stop 2024-03-27T21:41:34.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.085+0000 7f59feb18640 1 -- 172.21.15.60:0/2429861475 shutdown_connections 2024-03-27T21:41:34.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.085+0000 7f59feb18640 1 --2- 172.21.15.60:0/2429861475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59f813b480 0x7f59f81397b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:34.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.085+0000 7f59feb18640 1 --2- 172.21.15.60:0/2429861475 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f59dc06c590 0x7f59dc06ea50 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:34.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.085+0000 7f59feb18640 1 --2- 172.21.15.60:0/2429861475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59f813b020 0x7f59f8139270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:34.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.085+0000 7f59feb18640 1 --2- 172.21.15.60:0/2429861475 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f59f80b40b0 0x7f59f81d9bb0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:34.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.085+0000 7f59feb18640 1 -- 172.21.15.60:0/2429861475 >> 172.21.15.60:0/2429861475 conn(0x7f59f80608e0 msgr2=0x7f59f80b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:34.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.086+0000 7f59feb18640 1 -- 172.21.15.60:0/2429861475 shutdown_connections 2024-03-27T21:41:34.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.086+0000 7f59feb18640 1 -- 172.21.15.60:0/2429861475 wait complete. 2024-03-27T21:41:34.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.086+0000 7f59feb18640 1 librados: shutdown 2024-03-27T21:41:34.077 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-27T21:41:34.090 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-27T21:41:34.142 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:34.155+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:34.202 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:34.202 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:41:34.202 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:41:34.202 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:41:34.202 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:34.216 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:41:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.252+0000 7f3206b38640 10 monclient: get_monmap_and_config 2024-03-27T21:41:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.252+0000 7f3206b38640 10 monclient: build_initial_monmap 2024-03-27T21:41:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.252+0000 7f3206b38640 10 monclient: monmap: 2024-03-27T21:41:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:34.253621+0000 2024-03-27T21:41:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:34.253621+0000 2024-03-27T21:41:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:34.240 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:34.240 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:34.240 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:34.240 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:34.240 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:34.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.253+0000 7f3206b38640 1 Processor -- start 2024-03-27T21:41:34.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.254+0000 7f3206b38640 1 -- start start 2024-03-27T21:41:34.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.254+0000 7f3206b38640 10 monclient: init 2024-03-27T21:41:34.247 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:34.256+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:34.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.256+0000 7f3206b38640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:34.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.256+0000 7f3206b38640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-03-27T21:41:34.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.256+0000 7f3206b38640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f32000b40b0 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:34.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.256+0000 7f3206b38640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f320013b020 0x7f320013b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:34.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.256+0000 7f31fffff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f320013b020 0x7f320013b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:34.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.256+0000 7f32048d4640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f32000b40b0 0x7f32000a8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.60:40484/0 (socket says 172.21.15.60:40484) 2024-03-27T21:41:34.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.256+0000 7f31fffff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f320013b020 0x7f320013b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:53240/0 (socket says 172.21.15.60:53240) 2024-03-27T21:41:34.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.256+0000 7f32048d4640 1 -- 172.21.15.60:0/1553112988 learned_addr learned my addr 172.21.15.60:0/1553112988 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:34.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.256+0000 7f3206b38640 10 monclient(hunting): picked mon.noname-b con 0x7f320013b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:34.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.256+0000 7f3206b38640 1 --2- 172.21.15.60:0/1553112988 >> v2:172.21.15.191:3300/0 conn(0x7f32001416f0 0x7f3200141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:34.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.256+0000 7f3206b38640 10 monclient(hunting): picked mon.noname-c con 0x7f32001416f0 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:34.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.256+0000 7f3206b38640 1 -- 172.21.15.60:0/1553112988 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32000b3830 con 0x7f32000b40b0 2024-03-27T21:41:34.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.256+0000 7f3206b38640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:34.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.256+0000 7f3206b38640 1 -- 172.21.15.60:0/1553112988 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32000b4490 con 0x7f32001416f0 2024-03-27T21:41:34.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.256+0000 7f3206b38640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:34.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.256+0000 7f3206b38640 1 -- 172.21.15.60:0/1553112988 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32000aa520 con 0x7f320013b020 2024-03-27T21:41:34.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.256+0000 7f3206b38640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:34.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.256+0000 7f3206b38640 10 monclient(hunting): authenticate will time out at 935.477914s 2024-03-27T21:41:34.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f32050d5640 1 --2- 172.21.15.60:0/1553112988 >> v2:172.21.15.191:3300/0 conn(0x7f32001416f0 0x7f3200141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:34.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f31ff7fe640 1 -- 172.21.15.60:0/1553112988 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141918266 0 0) 0x7f32000b3830 con 0x7f32000b40b0 2024-03-27T21:41:34.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f31fffff640 10 monclient(hunting): get_auth_request con 0x7f320013b020 auth_method 0 2024-03-27T21:41:34.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f31fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:34.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f31fffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:34.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f31fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:34.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f32050d5640 10 monclient(hunting): get_auth_request con 0x7f32001416f0 auth_method 0 2024-03-27T21:41:34.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f32050d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:34.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f32050d5640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:34.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f32050d5640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:34.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f31ff7fe640 10 monclient(hunting): _negotiate 2024-03-27T21:41:34.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f31ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:34.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f31ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:34.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f31ff7fe640 10 monclient(hunting): my global_id is 4999 2024-03-27T21:41:34.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f31ff7fe640 1 -- 172.21.15.60:0/1553112988 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31f0003520 con 0x7f32000b40b0 2024-03-27T21:41:34.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f31fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:34.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f31fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:34.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f31fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:34.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f32050d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:34.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f32050d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:34.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f32050d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:34.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f32050d5640 10 monclient(hunting): handle_auth_done global_id 5002 payload 274 2024-03-27T21:41:34.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f32050d5640 1 -- 172.21.15.60:0/1553112988 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f320013b020 msgr2=0x7f320013b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:34.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f32050d5640 1 --2- 172.21.15.60:0/1553112988 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f320013b020 0x7f320013b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:34.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f32050d5640 1 -- 172.21.15.60:0/1553112988 >> v1:172.21.15.191:6789/0 conn(0x7f32000b40b0 legacy=0x7f32000a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:34.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f32050d5640 10 monclient: _finish_hunting 0 2024-03-27T21:41:34.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f32050d5640 1 monclient: found mon.noname-c 2024-03-27T21:41:34.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f32050d5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:34.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f32050d5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:41:34.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f32050d5640 1 -- 172.21.15.60:0/1553112988 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32000a9470 con 0x7f32001416f0 2024-03-27T21:41:34.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.257+0000 7f31fffff640 1 --2- 172.21.15.60:0/1553112988 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f320013b020 0x7f320013b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:34.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.258+0000 7f32050d5640 1 --2- 172.21.15.60:0/1553112988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32001416f0 0x7f3200141b80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f31f8009e10 tx=0x7f31f80346d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddc7c64d563f411 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:34.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.259+0000 7f31ff7fe640 1 -- 172.21.15.60:0/1553112988 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f8007740 con 0x7f32001416f0 2024-03-27T21:41:34.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.259+0000 7f31ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:34.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.259+0000 7f31ff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:34.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.259+0000 7f31ff7fe640 10 monclient: dump: 2024-03-27T21:41:34.262 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:34.262 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:34.262 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:34.262 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:34.262 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:34.263 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:34.263 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:34.263 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:34.263 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:34.263 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:34.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.259+0000 7f31ff7fe640 10 monclient: _finish_auth 0 2024-03-27T21:41:34.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.259+0000 7f31ff7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:41:34.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.259+0000 7f31ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:34.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.259+0000 7f31ff7fe640 1 -- 172.21.15.60:0/1553112988 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f8037050 con 0x7f32001416f0 2024-03-27T21:41:34.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.259+0000 7f31ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:34.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.259+0000 7f31ff7fe640 1 -- 172.21.15.60:0/1553112988 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f8007b00 con 0x7f32001416f0 2024-03-27T21:41:34.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.259+0000 7f31ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:34.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.259+0000 7f31ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:34.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.259+0000 7f31ff7fe640 10 monclient: dump: 2024-03-27T21:41:34.264 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:34.264 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:34.264 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:34.264 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:34.264 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:34.264 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:34.264 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:34.264 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:34.264 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:34.264 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:34.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.259+0000 7f3206b38640 5 monclient: authenticate success, global_id 5002 2024-03-27T21:41:34.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.259+0000 7f3206b38640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:34.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.259+0000 7f3206b38640 4 set_mon_vals no callback set 2024-03-27T21:41:34.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.260+0000 7f3206b38640 10 monclient: shutdown 2024-03-27T21:41:34.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.260+0000 7f3206b38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:34.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.260+0000 7f3206b38640 1 -- 172.21.15.60:0/1553112988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32001416f0 msgr2=0x7f3200141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:34.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.260+0000 7f3206b38640 1 --2- 172.21.15.60:0/1553112988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32001416f0 0x7f3200141b80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f31f8009e10 tx=0x7f31f80346d0 comp rx=0 tx=0).stop 2024-03-27T21:41:34.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.260+0000 7f3206b38640 1 -- 172.21.15.60:0/1553112988 shutdown_connections 2024-03-27T21:41:34.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.260+0000 7f3206b38640 1 --2- 172.21.15.60:0/1553112988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32001416f0 0x7f3200141b80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:34.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.260+0000 7f3206b38640 1 --2- 172.21.15.60:0/1553112988 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f320013b020 0x7f320013b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:34.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.260+0000 7f3206b38640 1 -- 172.21.15.60:0/1553112988 >> 172.21.15.60:0/1553112988 conn(0x7f32000608e0 msgr2=0x7f3200060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:34.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.260+0000 7f3206b38640 1 -- 172.21.15.60:0/1553112988 shutdown_connections 2024-03-27T21:41:34.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.260+0000 7f3206b38640 1 -- 172.21.15.60:0/1553112988 wait complete. 2024-03-27T21:41:34.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.261+0000 7f3206b38640 10 monclient: build_initial_monmap 2024-03-27T21:41:34.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.261+0000 7f3206b38640 10 monclient: monmap: 2024-03-27T21:41:34.265 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:34.265 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:34.265 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:34.266 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:34.266 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:34.266 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:34.266 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:34.266 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:34.266 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:34.266 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:34.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.261+0000 7f3206b38640 1 librados: starting msgr at 2024-03-27T21:41:34.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.261+0000 7f3206b38640 1 librados: starting objecter 2024-03-27T21:41:34.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.261+0000 7f3206b38640 1 Processor -- start 2024-03-27T21:41:34.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.261+0000 7f3206b38640 1 -- start start 2024-03-27T21:41:34.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.261+0000 7f3206b38640 1 librados: setting wanted keys 2024-03-27T21:41:34.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.261+0000 7f3206b38640 1 librados: calling monclient init 2024-03-27T21:41:34.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.261+0000 7f3206b38640 10 monclient: init 2024-03-27T21:41:34.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.262+0000 7f3206b38640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:34.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.262+0000 7f3206b38640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:41:34.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f3206b38640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32000b40b0 0x7f32001381d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:34.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f3206b38640 10 monclient(hunting): picked mon.noname-a con 0x7f32000b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:34.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f3206b38640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f320013b020 0x7f320013cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:34.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f3206b38640 10 monclient(hunting): picked mon.noname-c con 0x7f320013b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:34.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f3206b38640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32001416f0 0x7f320013d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:34.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f3206b38640 10 monclient(hunting): picked mon.noname-b con 0x7f32001416f0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:34.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f3206b38640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:34.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f3206b38640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3200138a60 con 0x7f32000b40b0 2024-03-27T21:41:34.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f3206b38640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:34.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f3206b38640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3200138710 con 0x7f32001416f0 2024-03-27T21:41:34.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f3206b38640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:34.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f3206b38640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f320013e4f0 con 0x7f320013b020 2024-03-27T21:41:34.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f3206b38640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:34.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f3206b38640 10 monclient(hunting): authenticate will time out at 935.483996s 2024-03-27T21:41:34.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f31fffff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f320013b020 0x7f320013cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:34.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f32048d4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32000b40b0 0x7f32001381d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:34.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f32048d4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32000b40b0 0x7f32001381d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50814/0 (socket says 172.21.15.60:50814) 2024-03-27T21:41:34.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f32048d4640 1 -- 172.21.15.60:0/267482553 learned_addr learned my addr 172.21.15.60:0/267482553 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:34.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f32048d4640 10 monclient(hunting): get_auth_request con 0x7f32000b40b0 auth_method 0 2024-03-27T21:41:34.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f32048d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:34.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f32048d4640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:34.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f32048d4640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:34.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f32050d5640 1 --2- 172.21.15.60:0/267482553 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32001416f0 0x7f320013d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:34.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f32050d5640 10 monclient(hunting): get_auth_request con 0x7f32001416f0 auth_method 0 2024-03-27T21:41:34.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f32050d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:34.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f32050d5640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:34.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f32050d5640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:34.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f31fffff640 10 monclient(hunting): get_auth_request con 0x7f320013b020 auth_method 0 2024-03-27T21:41:34.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f31fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:34.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f31fffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:34.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f31fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:34.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f32048d4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:34.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f32048d4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:34.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f32048d4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:34.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f31fffff640 1 -- 172.21.15.60:0/267482553 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f320013b020 msgr2=0x7f320013cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-27T21:41:34.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f31fffff640 1 -- 172.21.15.60:0/267482553 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f320013b020 msgr2=0x7f320013cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-27T21:41:34.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.263+0000 7f31fffff640 1 --2- 172.21.15.60:0/267482553 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f320013b020 0x7f320013cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-27T21:41:34.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f31fffff640 1 --2- 172.21.15.60:0/267482553 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f320013b020 0x7f320013cc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-27T21:41:34.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f32050d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:34.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f32050d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:34.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f32050d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:34.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f32048d4640 10 monclient(hunting): handle_auth_done global_id 5058 payload 824 2024-03-27T21:41:34.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f32048d4640 1 -- 172.21.15.60:0/267482553 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f320013b020 msgr2=0x7f320013cc80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T21:41:34.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f32048d4640 1 --2- 172.21.15.60:0/267482553 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f320013b020 0x7f320013cc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:34.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f32048d4640 1 -- 172.21.15.60:0/267482553 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32001416f0 msgr2=0x7f320013d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:34.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f32048d4640 1 --2- 172.21.15.60:0/267482553 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32001416f0 0x7f320013d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:34.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f32048d4640 10 monclient: _finish_hunting 0 2024-03-27T21:41:34.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f32048d4640 1 monclient: found mon.noname-a 2024-03-27T21:41:34.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f32048d4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:34.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f32048d4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:34.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f32048d4640 1 -- 172.21.15.60:0/267482553 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ec009400 con 0x7f32000b40b0 2024-03-27T21:41:34.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f32048d4640 1 --2- 172.21.15.60:0/267482553 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32000b40b0 0x7f32001381d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f31f400c500 tx=0x7f31f400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:34.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f31fd7fa640 10 client.?.objecter ms_handle_connect 0x7f32000b40b0 2024-03-27T21:41:34.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f31fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:34.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f31fd7fa640 1 -- 172.21.15.60:0/267482553 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f4005ce0 con 0x7f32000b40b0 2024-03-27T21:41:34.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f31fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:34.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f31fd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:34.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f31fd7fa640 10 monclient: dump: 2024-03-27T21:41:34.271 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:34.271 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:34.271 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:34.271 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:34.271 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:34.271 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:34.271 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:34.271 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:34.272 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:34.272 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:34.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f31fd7fa640 10 monclient: _finish_auth 0 2024-03-27T21:41:34.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f31fd7fa640 10 monclient: _check_auth_tickets 2024-03-27T21:41:34.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f31fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:34.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f31fd7fa640 1 -- 172.21.15.60:0/267482553 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f4005e40 con 0x7f32000b40b0 2024-03-27T21:41:34.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f31fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:34.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f31fd7fa640 1 -- 172.21.15.60:0/267482553 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f4007330 con 0x7f32000b40b0 2024-03-27T21:41:34.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.264+0000 7f31fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:34.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.265+0000 7f31fd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:34.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.265+0000 7f31fd7fa640 10 monclient: dump: 2024-03-27T21:41:34.272 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:34.272 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:34.272 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:34.272 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:34.273 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:34.273 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:34.273 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:34.273 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:34.273 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:34.273 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:34.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.265+0000 7f31feffd640 4 set_mon_vals no callback set 2024-03-27T21:41:34.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.265+0000 7f3206b38640 5 monclient: authenticate success, global_id 5058 2024-03-27T21:41:34.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.265+0000 7f3206b38640 10 monclient: _renew_subs 2024-03-27T21:41:34.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.265+0000 7f3206b38640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:34.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.265+0000 7f3206b38640 1 -- 172.21.15.60:0/267482553 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f8009a70 con 0x7f32000b40b0 2024-03-27T21:41:34.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.265+0000 7f3206b38640 10 client.5058.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:34.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.265+0000 7f3206b38640 10 monclient: _renew_subs 2024-03-27T21:41:34.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.265+0000 7f3206b38640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:34.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.265+0000 7f3206b38640 1 -- 172.21.15.60:0/267482553 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f320013d700 con 0x7f32000b40b0 2024-03-27T21:41:34.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.265+0000 7f3206b38640 1 librados: init done 2024-03-27T21:41:34.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.267+0000 7f3206b38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:34.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.267+0000 7f3206b38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:34.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.267+0000 7f3206b38640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:34.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.267+0000 7f3206b38640 1 -- 172.21.15.60:0/267482553 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f320013d9e0 con 0x7f32000b40b0 2024-03-27T21:41:34.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.268+0000 7f31fd7fa640 1 -- 172.21.15.60:0/267482553 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f31f4007550 con 0x7f32000b40b0 2024-03-27T21:41:34.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.268+0000 7f31fd7fa640 1 --2- 172.21.15.60:0/267482553 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f31e406c590 0x7f31e406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:34.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.269+0000 7f31fd7fa640 1 -- 172.21.15.60:0/267482553 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f31f4088790 con 0x7f32000b40b0 2024-03-27T21:41:34.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.269+0000 7f31fd7fa640 10 client.5058.objecter ms_dispatch 0x7f3200000b90 osd_map(67..67 src has 1..67) v4 2024-03-27T21:41:34.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.269+0000 7f31fd7fa640 3 client.5058.objecter handle_osd_map got epochs [67,67] > 0 2024-03-27T21:41:34.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.269+0000 7f31fd7fa640 3 client.5058.objecter handle_osd_map decoding full epoch 67 2024-03-27T21:41:34.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.269+0000 7f31fd7fa640 20 client.5058.objecter dump_active .. 0 homeless 2024-03-27T21:41:34.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.269+0000 7f31fffff640 1 --2- 172.21.15.60:0/267482553 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f31e406c590 0x7f31e406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:34.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.269+0000 7f31fffff640 10 monclient: get_auth_request con 0x7f31e406c590 auth_method 0 2024-03-27T21:41:34.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.269+0000 7f31fffff640 1 --2- 172.21.15.60:0/267482553 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f31e406c590 0x7f31e406ea50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f32001d6490 tx=0x7f31ec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:34.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.269+0000 7f31fd7fa640 10 client.5058.objecter ms_handle_connect 0x7f31e406c590 2024-03-27T21:41:34.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.277+0000 7f31fd7fa640 1 -- 172.21.15.60:0/267482553 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f31f4053ca0 con 0x7f32000b40b0 2024-03-27T21:41:34.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.277+0000 7f31fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:34.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.277+0000 7f31fd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:34.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.292+0000 7f1fa938b640 10 monclient: get_monmap_and_config 2024-03-27T21:41:34.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.292+0000 7f1fa938b640 10 monclient: build_initial_monmap 2024-03-27T21:41:34.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.292+0000 7f1fa938b640 10 monclient: monmap: 2024-03-27T21:41:34.279 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:34.279 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:34.279 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:34.293470+0000 2024-03-27T21:41:34.279 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:34.293470+0000 2024-03-27T21:41:34.279 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:34.279 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:34.279 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:34.279 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:34.279 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:34.279 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:34.279 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:34.279 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:34.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.293+0000 7f1fa938b640 1 Processor -- start 2024-03-27T21:41:34.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.293+0000 7f1fa938b640 1 -- start start 2024-03-27T21:41:34.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.293+0000 7f1fa938b640 10 monclient: init 2024-03-27T21:41:34.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.293+0000 7f1fa938b640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:34.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.293+0000 7f1fa938b640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-03-27T21:41:34.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.293+0000 7f1fa938b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1fa40b40b0 0x7f1fa40a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:34.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.293+0000 7f1fa938b640 10 monclient(hunting): picked mon.noname-c con 0x7f1fa40b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:34.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.293+0000 7f1fa938b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1fa413b020 0x7f1fa413b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:34.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa938b640 10 monclient(hunting): picked mon.noname-a con 0x7f1fa413b020 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:34.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa938b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1fa41416f0 0x7f1fa4141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:34.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa938b640 10 monclient(hunting): picked mon.noname-b con 0x7f1fa41416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:34.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa938b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:34.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa938b640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fa40b4490 con 0x7f1fa413b020 2024-03-27T21:41:34.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa938b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:34.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa938b640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fa40b3830 con 0x7f1fa40b40b0 2024-03-27T21:41:34.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa938b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:34.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa938b640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa40aa520 con 0x7f1fa41416f0 2024-03-27T21:41:34.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa938b640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:34.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa938b640 10 monclient(hunting): authenticate will time out at 935.514964s 2024-03-27T21:41:34.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa27fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1fa413b020 0x7f1fa413b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:34.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa27fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1fa413b020 0x7f1fa413b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50826/0 (socket says 172.21.15.60:50826) 2024-03-27T21:41:34.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa27fc640 1 -- 172.21.15.60:0/3440881204 learned_addr learned my addr 172.21.15.60:0/3440881204 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:34.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa27fc640 10 monclient(hunting): get_auth_request con 0x7f1fa413b020 auth_method 0 2024-03-27T21:41:34.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:34.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa27fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:34.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:34.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa37fe640 1 --2- 172.21.15.60:0/3440881204 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1fa41416f0 0x7f1fa4141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:34.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa37fe640 10 monclient(hunting): get_auth_request con 0x7f1fa41416f0 auth_method 0 2024-03-27T21:41:34.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:34.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa37fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:34.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:34.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:34.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:34.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:34.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa2ffd640 1 --2- 172.21.15.60:0/3440881204 >> v2:172.21.15.191:3300/0 conn(0x7f1fa40b40b0 0x7f1fa40a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:34.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa2ffd640 10 monclient(hunting): get_auth_request con 0x7f1fa40b40b0 auth_method 0 2024-03-27T21:41:34.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:34.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:34.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:34.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:34.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:34.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:34.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:34.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:34.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:34.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa37fe640 10 monclient(hunting): handle_auth_done global_id 4835 payload 274 2024-03-27T21:41:34.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa37fe640 1 -- 172.21.15.60:0/3440881204 >> v2:172.21.15.191:3300/0 conn(0x7f1fa40b40b0 msgr2=0x7f1fa40a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:34.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa37fe640 1 --2- 172.21.15.60:0/3440881204 >> v2:172.21.15.191:3300/0 conn(0x7f1fa40b40b0 0x7f1fa40a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:34.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa37fe640 1 -- 172.21.15.60:0/3440881204 >> v2:172.21.15.60:3300/0 conn(0x7f1fa413b020 msgr2=0x7f1fa413b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:34.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa37fe640 1 --2- 172.21.15.60:0/3440881204 >> v2:172.21.15.60:3300/0 conn(0x7f1fa413b020 0x7f1fa413b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:34.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa37fe640 10 monclient: _finish_hunting 0 2024-03-27T21:41:34.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa37fe640 1 monclient: found mon.noname-b 2024-03-27T21:41:34.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:34.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:34.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.294+0000 7f1fa37fe640 1 -- 172.21.15.60:0/3440881204 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa40a9470 con 0x7f1fa41416f0 2024-03-27T21:41:34.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.296+0000 7f1fa37fe640 1 --2- 172.21.15.60:0/3440881204 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1fa41416f0 0x7f1fa4141b80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1fa40b4810 tx=0x7f1f9c02be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17ff3cdf8bee515f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:34.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.296+0000 7f1fa1ffb640 1 -- 172.21.15.60:0/3440881204 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c02cbf0 con 0x7f1fa41416f0 2024-03-27T21:41:34.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.296+0000 7f1fa1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:34.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.296+0000 7f1fa1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:34.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.296+0000 7f1fa1ffb640 10 monclient: dump: 2024-03-27T21:41:34.295 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:34.295 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:34.295 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:34.295 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:34.295 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:34.295 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:34.296 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:34.296 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:34.296 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:34.296 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:34.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.296+0000 7f1fa1ffb640 10 monclient: _finish_auth 0 2024-03-27T21:41:34.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.296+0000 7f1fa1ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:41:34.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.296+0000 7f1fa1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:34.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.296+0000 7f1fa938b640 5 monclient: authenticate success, global_id 4835 2024-03-27T21:41:34.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.296+0000 7f1fa938b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:34.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.296+0000 7f1fa1ffb640 1 -- 172.21.15.60:0/3440881204 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9c02cd50 con 0x7f1fa41416f0 2024-03-27T21:41:34.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.296+0000 7f1fa1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:34.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.296+0000 7f1fa938b640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:34.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.296+0000 7f1fa938b640 4 set_mon_vals no callback set 2024-03-27T21:41:34.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.296+0000 7f1fa1ffb640 1 -- 172.21.15.60:0/3440881204 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c034660 con 0x7f1fa41416f0 2024-03-27T21:41:34.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.296+0000 7f1fa1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:34.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.296+0000 7f1fa1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:34.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.296+0000 7f1fa1ffb640 10 monclient: dump: 2024-03-27T21:41:34.297 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:34.297 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:34.297 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:34.297 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:34.297 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:34.297 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:34.297 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:34.297 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:34.297 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:34.297 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:34.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.297+0000 7f1fa938b640 10 monclient: shutdown 2024-03-27T21:41:34.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.297+0000 7f1fa938b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:34.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.297+0000 7f1fa938b640 1 -- 172.21.15.60:0/3440881204 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1fa41416f0 msgr2=0x7f1fa4141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:34.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.297+0000 7f1fa938b640 1 --2- 172.21.15.60:0/3440881204 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1fa41416f0 0x7f1fa4141b80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1fa40b4810 tx=0x7f1f9c02be40 comp rx=0 tx=0).stop 2024-03-27T21:41:34.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.297+0000 7f1fa938b640 1 -- 172.21.15.60:0/3440881204 shutdown_connections 2024-03-27T21:41:34.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.297+0000 7f1fa938b640 1 --2- 172.21.15.60:0/3440881204 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1fa41416f0 0x7f1fa4141b80 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:34.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.297+0000 7f1fa938b640 1 --2- 172.21.15.60:0/3440881204 >> v2:172.21.15.60:3300/0 conn(0x7f1fa413b020 0x7f1fa413b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:34.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.297+0000 7f1fa938b640 1 --2- 172.21.15.60:0/3440881204 >> v2:172.21.15.191:3300/0 conn(0x7f1fa40b40b0 0x7f1fa40a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:34.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.297+0000 7f1fa938b640 1 -- 172.21.15.60:0/3440881204 >> 172.21.15.60:0/3440881204 conn(0x7f1fa40608e0 msgr2=0x7f1fa4060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:34.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.297+0000 7f1fa938b640 1 -- 172.21.15.60:0/3440881204 shutdown_connections 2024-03-27T21:41:34.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.297+0000 7f1fa938b640 1 -- 172.21.15.60:0/3440881204 wait complete. 2024-03-27T21:41:34.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.298+0000 7f1fa938b640 10 monclient: build_initial_monmap 2024-03-27T21:41:34.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.298+0000 7f1fa938b640 10 monclient: monmap: 2024-03-27T21:41:34.298 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:34.298 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:34.298 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:34.298 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:34.299 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:34.299 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:34.299 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:34.299 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:34.299 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:34.299 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:34.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.298+0000 7f1fa938b640 1 librados: starting msgr at 2024-03-27T21:41:34.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.298+0000 7f1fa938b640 1 librados: starting objecter 2024-03-27T21:41:34.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.298+0000 7f1fa938b640 1 Processor -- start 2024-03-27T21:41:34.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.298+0000 7f1fa938b640 1 -- start start 2024-03-27T21:41:34.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.298+0000 7f1fa938b640 1 librados: setting wanted keys 2024-03-27T21:41:34.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.298+0000 7f1fa938b640 1 librados: calling monclient init 2024-03-27T21:41:34.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.298+0000 7f1fa938b640 10 monclient: init 2024-03-27T21:41:34.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa938b640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:34.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa938b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:41:34.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa938b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1fa40b40b0 0x7f1fa40b6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:34.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa938b640 10 monclient(hunting): picked mon.noname-c con 0x7f1fa40b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:34.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa938b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fa413b020 0x7f1fa41ddd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:34.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa938b640 10 monclient(hunting): picked mon.noname-b con 0x7f1fa413b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:34.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa938b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fa41416f0 0x7f1fa413d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:34.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa938b640 10 monclient(hunting): picked mon.noname-a con 0x7f1fa41416f0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:34.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa938b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:34.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa938b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa413dd50 con 0x7f1fa41416f0 2024-03-27T21:41:34.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa938b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:34.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa938b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa413df20 con 0x7f1fa413b020 2024-03-27T21:41:34.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa938b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:34.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa938b640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa413e200 con 0x7f1fa40b40b0 2024-03-27T21:41:34.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa938b640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:34.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa938b640 10 monclient(hunting): authenticate will time out at 935.520437s 2024-03-27T21:41:34.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa2ffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1fa40b40b0 0x7f1fa40b6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:34.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa2ffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1fa40b40b0 0x7f1fa40b6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:53262/0 (socket says 172.21.15.60:53262) 2024-03-27T21:41:34.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa2ffd640 1 -- 172.21.15.60:0/4275352833 learned_addr learned my addr 172.21.15.60:0/4275352833 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:34.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa2ffd640 10 monclient(hunting): get_auth_request con 0x7f1fa40b40b0 auth_method 0 2024-03-27T21:41:34.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:34.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:34.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:34.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa37fe640 1 --2- 172.21.15.60:0/4275352833 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fa41416f0 0x7f1fa413d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:34.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa37fe640 10 monclient(hunting): get_auth_request con 0x7f1fa41416f0 auth_method 0 2024-03-27T21:41:34.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:34.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa37fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:34.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:34.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:34.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:34.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.299+0000 7f1fa2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:34.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa27fc640 1 --2- 172.21.15.60:0/4275352833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fa413b020 0x7f1fa41ddd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:34.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa27fc640 10 monclient(hunting): get_auth_request con 0x7f1fa413b020 auth_method 0 2024-03-27T21:41:34.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:34.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa27fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:34.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:34.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:34.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:34.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:34.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:34.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:34.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:34.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa2ffd640 10 monclient(hunting): handle_auth_done global_id 4838 payload 824 2024-03-27T21:41:34.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa2ffd640 1 -- 172.21.15.60:0/4275352833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fa413b020 msgr2=0x7f1fa41ddd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:34.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa2ffd640 1 --2- 172.21.15.60:0/4275352833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fa413b020 0x7f1fa41ddd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:34.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa2ffd640 1 -- 172.21.15.60:0/4275352833 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fa41416f0 msgr2=0x7f1fa413d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:34.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa2ffd640 1 --2- 172.21.15.60:0/4275352833 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fa41416f0 0x7f1fa413d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:34.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa2ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:41:34.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa2ffd640 1 monclient: found mon.noname-c 2024-03-27T21:41:34.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:34.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:34.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa2ffd640 1 -- 172.21.15.60:0/4275352833 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9c009a70 con 0x7f1fa40b40b0 2024-03-27T21:41:34.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.300+0000 7f1fa37fe640 1 --2- 172.21.15.60:0/4275352833 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fa41416f0 0x7f1fa413d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:34.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.301+0000 7f1fa2ffd640 1 --2- 172.21.15.60:0/4275352833 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1fa40b40b0 0x7f1fa40b6680 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1f9400e9e0 tx=0x7f1f9400eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:34.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.301+0000 7f1f8b7fe640 10 client.?.objecter ms_handle_connect 0x7f1fa40b40b0 2024-03-27T21:41:34.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.301+0000 7f1f8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:34.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.301+0000 7f1f8b7fe640 1 -- 172.21.15.60:0/4275352833 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f94009670 con 0x7f1fa40b40b0 2024-03-27T21:41:34.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.301+0000 7f1f8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:34.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.301+0000 7f1f8b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:34.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.301+0000 7f1f8b7fe640 10 monclient: dump: 2024-03-27T21:41:34.304 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:34.304 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:34.304 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:34.304 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:34.304 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:34.304 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:34.304 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:34.304 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:34.304 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:34.304 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:34.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.301+0000 7f1f8b7fe640 10 monclient: _finish_auth 0 2024-03-27T21:41:34.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.301+0000 7f1f8b7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:41:34.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.301+0000 7f1f8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:34.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.301+0000 7f1f8b7fe640 1 -- 172.21.15.60:0/4275352833 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9400c9d0 con 0x7f1fa40b40b0 2024-03-27T21:41:34.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.301+0000 7f1f8b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:34.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.301+0000 7f1f8b7fe640 1 -- 172.21.15.60:0/4275352833 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9400cd80 con 0x7f1fa40b40b0 2024-03-27T21:41:34.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.301+0000 7f1f8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:34.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.301+0000 7f1f8b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:34.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.301+0000 7f1f8b7fe640 10 monclient: dump: 2024-03-27T21:41:34.305 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:34.305 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:34.305 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:34.305 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:34.305 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:34.305 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:34.305 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:34.305 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:34.305 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:34.305 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:34.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.301+0000 7f1fa17fa640 4 set_mon_vals no callback set 2024-03-27T21:41:34.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.302+0000 7f1fa938b640 5 monclient: authenticate success, global_id 4838 2024-03-27T21:41:34.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.302+0000 7f1fa938b640 10 monclient: _renew_subs 2024-03-27T21:41:34.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.302+0000 7f1fa938b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:34.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.302+0000 7f1fa938b640 1 -- 172.21.15.60:0/4275352833 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fa413cc60 con 0x7f1fa40b40b0 2024-03-27T21:41:34.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.302+0000 7f1fa938b640 10 client.4838.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:34.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.302+0000 7f1fa938b640 10 monclient: _renew_subs 2024-03-27T21:41:34.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.302+0000 7f1fa938b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:34.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.302+0000 7f1fa938b640 1 -- 172.21.15.60:0/4275352833 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fa413e900 con 0x7f1fa40b40b0 2024-03-27T21:41:34.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.302+0000 7f1fa938b640 1 librados: init done 2024-03-27T21:41:34.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.304+0000 7f1f8b7fe640 1 -- 172.21.15.60:0/4275352833 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1f94013470 con 0x7f1fa40b40b0 2024-03-27T21:41:34.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.304+0000 7f1fa938b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:34.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.304+0000 7f1fa938b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:34.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.304+0000 7f1fa938b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:34.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.304+0000 7f1fa938b640 1 -- 172.21.15.60:0/4275352833 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa413e420 con 0x7f1fa40b40b0 2024-03-27T21:41:34.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.305+0000 7f1f8b7fe640 1 --2- 172.21.15.60:0/4275352833 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1f9006c5c0 0x7f1f9006ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:34.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.305+0000 7f1f8b7fe640 1 -- 172.21.15.60:0/4275352833 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f1f94089380 con 0x7f1fa40b40b0 2024-03-27T21:41:34.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.305+0000 7f1f8b7fe640 10 client.4838.objecter ms_dispatch 0x7f1fa4000b90 osd_map(67..67 src has 1..67) v4 2024-03-27T21:41:34.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.305+0000 7f1f8b7fe640 3 client.4838.objecter handle_osd_map got epochs [67,67] > 0 2024-03-27T21:41:34.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.305+0000 7f1f8b7fe640 3 client.4838.objecter handle_osd_map decoding full epoch 67 2024-03-27T21:41:34.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.305+0000 7f1f8b7fe640 20 client.4838.objecter dump_active .. 0 homeless 2024-03-27T21:41:34.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.305+0000 7f1fa27fc640 1 --2- 172.21.15.60:0/4275352833 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1f9006c5c0 0x7f1f9006ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:34.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.305+0000 7f1fa27fc640 10 monclient: get_auth_request con 0x7f1f9006c5c0 auth_method 0 2024-03-27T21:41:34.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.306+0000 7f1fa27fc640 1 --2- 172.21.15.60:0/4275352833 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1f9006c5c0 0x7f1f9006ea80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1fa413ad20 tx=0x7f1f98009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:34.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.306+0000 7f1f8b7fe640 10 client.4838.objecter ms_handle_connect 0x7f1f9006c5c0 2024-03-27T21:41:34.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.313+0000 7f1f8b7fe640 1 -- 172.21.15.60:0/4275352833 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1f94054740 con 0x7f1fa40b40b0 2024-03-27T21:41:34.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.313+0000 7f1f8b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:34.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.313+0000 7f1f8b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:34.344 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:34.358+0000 7fb850f34640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 56426) UID: 0 2024-03-27T21:41:34.397 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:34.397 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:34.397 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:34.397 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:34.411 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:41:34.452 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:34.466+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:34.546 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:34.560+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:34.619 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:34.620 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:34.620 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:41:34.620 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:41:34.620 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:34.632 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:41:34.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.658+0000 7f1fa938b640 10 start_mon_command cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}] 2024-03-27T21:41:34.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.658+0000 7f1fa938b640 10 monclient: _send_command 2 [{"prefix": "osd reweight", "id": 6, "weight": 1.0}] 2024-03-27T21:41:34.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.658+0000 7f1fa938b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:34.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.658+0000 7f1fa938b640 1 -- 172.21.15.60:0/4275352833 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7f1fa4069470 con 0x7f1fa40b40b0 2024-03-27T21:41:34.648 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:34.662+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:34.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.663+0000 7f3206b38640 10 start_mon_command cmd=[{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:41:34.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.664+0000 7f3206b38640 10 monclient: _send_command 2 [{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:41:34.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.664+0000 7f3206b38640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:34.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:34.664+0000 7f3206b38640 1 -- 172.21.15.60:0/267482553 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f3200069470 con 0x7f32000b40b0 2024-03-27T21:41:34.749 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:34.762+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:34.814 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:34.814 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:41:34.814 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:41:34.815 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:41:34.815 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:34.826 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:41:34.868 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:34.882+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:34.942 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:34.943 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:34.943 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:34.943 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:34.950 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:34.964+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:34.951 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:41:35.051 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:35.065+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:35.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.089+0000 7f31fd7fa640 1 -- 172.21.15.60:0/267482553 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v68) v1 ==== 91+0+0 (secure 0 0 0) 0x7f31f40572c0 con 0x7f32000b40b0 2024-03-27T21:41:35.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.089+0000 7f31fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:41:35.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.089+0000 7f31fd7fa640 10 monclient: _finish_command 2 = system:0 noscrub is unset 2024-03-27T21:41:35.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.089+0000 7f1f8b7fe640 1 -- 172.21.15.60:0/4275352833 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v68) v1 ==== 113+0+0 (secure 0 0 0) 0x7f1f94057d60 con 0x7f1fa40b40b0 2024-03-27T21:41:35.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.089+0000 7f1f8b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd reweight", "id": 6, "weight": 1.0}] 2024-03-27T21:41:35.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.089+0000 7f1f8b7fe640 10 monclient: _finish_command 2 = system:0 reweighted osd.6 to 1 (10000) 2024-03-27T21:41:35.076 INFO:teuthology.orchestra.run.smithi060.stderr:noscrub is unset 2024-03-27T21:41:35.076 INFO:teuthology.orchestra.run.smithi060.stderr:reweighted osd.6 to 1 (10000) 2024-03-27T21:41:35.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f3206b38640 10 librados: watch_flush enter 2024-03-27T21:41:35.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f3206b38640 10 librados: watch_flush exit 2024-03-27T21:41:35.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f3206b38640 20 client.5058.objecter shutdown clearing up homeless session... 2024-03-27T21:41:35.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f3206b38640 10 client.5058.objecter successfully canceled tick 2024-03-27T21:41:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f1fa938b640 10 librados: watch_flush enter 2024-03-27T21:41:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f1fa938b640 10 librados: watch_flush exit 2024-03-27T21:41:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f1fa938b640 20 client.4838.objecter shutdown clearing up homeless session... 2024-03-27T21:41:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f1fa938b640 10 client.4838.objecter successfully canceled tick 2024-03-27T21:41:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f1fa938b640 1 -- 172.21.15.60:0/4275352833 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1f9006c5c0 msgr2=0x7f1f9006ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f1fa938b640 1 --2- 172.21.15.60:0/4275352833 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1f9006c5c0 0x7f1f9006ea80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1fa413ad20 tx=0x7f1f98009240 comp rx=0 tx=0).stop 2024-03-27T21:41:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f1fa938b640 10 monclient: shutdown 2024-03-27T21:41:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f1fa938b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f1fa938b640 1 -- 172.21.15.60:0/4275352833 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1fa40b40b0 msgr2=0x7f1fa40b6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f1fa938b640 1 --2- 172.21.15.60:0/4275352833 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1fa40b40b0 0x7f1fa40b6680 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1f9400e9e0 tx=0x7f1f9400eea0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f3206b38640 1 -- 172.21.15.60:0/267482553 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f31e406c590 msgr2=0x7f31e406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f3206b38640 1 --2- 172.21.15.60:0/267482553 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f31e406c590 0x7f31e406ea50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f32001d6490 tx=0x7f31ec008040 comp rx=0 tx=0).stop 2024-03-27T21:41:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f3206b38640 10 monclient: shutdown 2024-03-27T21:41:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f3206b38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f3206b38640 1 -- 172.21.15.60:0/267482553 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32000b40b0 msgr2=0x7f32001381d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f3206b38640 1 --2- 172.21.15.60:0/267482553 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32000b40b0 0x7f32001381d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f31f400c500 tx=0x7f31f400c9c0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f3206b38640 1 -- 172.21.15.60:0/267482553 shutdown_connections 2024-03-27T21:41:35.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f3206b38640 1 --2- 172.21.15.60:0/267482553 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f31e406c590 0x7f31e406ea50 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f3206b38640 1 --2- 172.21.15.60:0/267482553 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32001416f0 0x7f320013d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f3206b38640 1 --2- 172.21.15.60:0/267482553 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f320013b020 0x7f320013cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f3206b38640 1 --2- 172.21.15.60:0/267482553 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32000b40b0 0x7f32001381d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.092+0000 7f3206b38640 1 -- 172.21.15.60:0/267482553 >> 172.21.15.60:0/267482553 conn(0x7f32000608e0 msgr2=0x7f32000b1950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:35.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.093+0000 7f1fa938b640 1 -- 172.21.15.60:0/4275352833 shutdown_connections 2024-03-27T21:41:35.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.093+0000 7f1fa938b640 1 --2- 172.21.15.60:0/4275352833 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fa41416f0 0x7f1fa413d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.093+0000 7f1fa938b640 1 --2- 172.21.15.60:0/4275352833 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1f9006c5c0 0x7f1f9006ea80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.093+0000 7f1fa938b640 1 --2- 172.21.15.60:0/4275352833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fa413b020 0x7f1fa41ddd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.093+0000 7f1fa938b640 1 --2- 172.21.15.60:0/4275352833 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1fa40b40b0 0x7f1fa40b6680 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1f9400e9e0 tx=0x7f1f9400eea0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.093+0000 7f1fa938b640 1 -- 172.21.15.60:0/4275352833 >> 172.21.15.60:0/4275352833 conn(0x7f1fa40608e0 msgr2=0x7f1fa40b28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:35.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.093+0000 7f1fa938b640 1 -- 172.21.15.60:0/4275352833 shutdown_connections 2024-03-27T21:41:35.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.093+0000 7f1fa938b640 1 -- 172.21.15.60:0/4275352833 wait complete. 2024-03-27T21:41:35.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.093+0000 7f1fa938b640 1 librados: shutdown 2024-03-27T21:41:35.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.093+0000 7f3206b38640 1 -- 172.21.15.60:0/267482553 shutdown_connections 2024-03-27T21:41:35.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.093+0000 7f3206b38640 1 -- 172.21.15.60:0/267482553 wait complete. 2024-03-27T21:41:35.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.093+0000 7f3206b38640 1 librados: shutdown 2024-03-27T21:41:35.096 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-27T21:41:35.098 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:35.098 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:35.099 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:35.099 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:35.099 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:35.107 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:41:35.182 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:35.196+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:35.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.264+0000 7f5c9a8aa640 10 monclient: get_monmap_and_config 2024-03-27T21:41:35.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.264+0000 7f5c9a8aa640 10 monclient: build_initial_monmap 2024-03-27T21:41:35.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.264+0000 7f5c9a8aa640 10 monclient: monmap: 2024-03-27T21:41:35.251 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:35.251 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:35.251 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:35.265346+0000 2024-03-27T21:41:35.251 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:35.265346+0000 2024-03-27T21:41:35.251 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:35.251 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:35.251 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:35.251 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:35.251 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:35.251 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:35.252 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:35.252 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:35.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.265+0000 7f5c9a8aa640 1 Processor -- start 2024-03-27T21:41:35.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.265+0000 7f5c9a8aa640 1 -- start start 2024-03-27T21:41:35.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.265+0000 7f5c9a8aa640 10 monclient: init 2024-03-27T21:41:35.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.265+0000 7f5c9a8aa640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:35.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.265+0000 7f5c9a8aa640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-03-27T21:41:35.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.265+0000 7f5c9a8aa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5c94071a50 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:35.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.265+0000 7f5c9a8aa640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5c9413ef20 0x7f5c940a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:35.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.265+0000 7f5c9a8aa640 10 monclient(hunting): picked mon.noname-a con 0x7f5c9413ef20 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:35.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.265+0000 7f5c9a8aa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5c94141630 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:35.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.265+0000 7f5c937fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5c9413ef20 0x7f5c940a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:35.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.265+0000 7f5c937fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5c9413ef20 0x7f5c940a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50842/0 (socket says 172.21.15.60:50842) 2024-03-27T21:41:35.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.265+0000 7f5c937fe640 1 -- 172.21.15.60:0/430270531 learned_addr learned my addr 172.21.15.60:0/430270531 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:35.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.265+0000 7f5c9a8aa640 1 -- 172.21.15.60:0/430270531 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c940a8e90 con 0x7f5c94071a50 2024-03-27T21:41:35.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.265+0000 7f5c9a8aa640 1 -- 172.21.15.60:0/430270531 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c940aa040 con 0x7f5c94141630 2024-03-27T21:41:35.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.265+0000 7f5c9a8aa640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:35.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.265+0000 7f5c9a8aa640 1 -- 172.21.15.60:0/430270531 --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c94071e30 con 0x7f5c9413ef20 2024-03-27T21:41:35.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.266+0000 7f5c9a8aa640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:35.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.266+0000 7f5c9a8aa640 10 monclient(hunting): authenticate will time out at 936.486930s 2024-03-27T21:41:35.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.266+0000 7f5c937fe640 10 monclient(hunting): get_auth_request con 0x7f5c9413ef20 auth_method 0 2024-03-27T21:41:35.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.266+0000 7f5c937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:35.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.266+0000 7f5c937fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:35.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.266+0000 7f5c937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:35.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.266+0000 7f5c937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:35.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.266+0000 7f5c937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:35.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.266+0000 7f5c937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:35.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.266+0000 7f5c937fe640 10 monclient(hunting): handle_auth_done global_id 5067 payload 274 2024-03-27T21:41:35.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.266+0000 7f5c937fe640 1 -- 172.21.15.60:0/430270531 >> v1:172.21.15.191:6789/0 conn(0x7f5c94141630 legacy=0x7f5c94141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:35.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.266+0000 7f5c937fe640 1 -- 172.21.15.60:0/430270531 >> v1:172.21.15.60:6789/0 conn(0x7f5c94071a50 legacy=0x7f5c9413e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:35.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.266+0000 7f5c937fe640 10 monclient: _finish_hunting 0 2024-03-27T21:41:35.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.266+0000 7f5c937fe640 1 monclient: found mon.noname-a 2024-03-27T21:41:35.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.266+0000 7f5c937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:35.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.266+0000 7f5c937fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:35.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.266+0000 7f5c937fe640 1 -- 172.21.15.60:0/430270531 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c940aa2b0 con 0x7f5c9413ef20 2024-03-27T21:41:35.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c937fe640 1 --2- 172.21.15.60:0/430270531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c9413ef20 0x7f5c940a87e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5c88009870 tx=0x7f5c8802ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a5c7dd0931fc51d server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:35.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c92ffd640 1 -- 172.21.15.60:0/430270531 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8802f860 con 0x7f5c9413ef20 2024-03-27T21:41:35.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:35.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c92ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:35.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c92ffd640 10 monclient: dump: 2024-03-27T21:41:35.259 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:35.259 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:35.259 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:35.259 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:35.259 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:35.259 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:35.259 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:35.259 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:35.259 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:35.259 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:35.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c92ffd640 10 monclient: _finish_auth 0 2024-03-27T21:41:35.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c92ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:41:35.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:35.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c9a8aa640 5 monclient: authenticate success, global_id 5067 2024-03-27T21:41:35.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c9a8aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:35.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c92ffd640 1 -- 172.21.15.60:0/430270531 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8802f9c0 con 0x7f5c9413ef20 2024-03-27T21:41:35.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:35.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c9a8aa640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:35.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c92ffd640 1 -- 172.21.15.60:0/430270531 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8802fd50 con 0x7f5c9413ef20 2024-03-27T21:41:35.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:35.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c9a8aa640 4 set_mon_vals no callback set 2024-03-27T21:41:35.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:35.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c92ffd640 10 monclient: dump: 2024-03-27T21:41:35.260 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:35.260 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:35.260 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:35.261 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:35.261 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:35.261 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:35.261 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:35.261 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:35.261 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:35.261 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:35.261 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:35.267+0000 7fb850f34640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 56426) UID: 0 2024-03-27T21:41:35.263 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:35.263 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:35.263 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:35.263 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:41:35.264 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:35.264 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:41:35.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c9a8aa640 10 monclient: shutdown 2024-03-27T21:41:35.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c9a8aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:35.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c9a8aa640 1 -- 172.21.15.60:0/430270531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c9413ef20 msgr2=0x7f5c940a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:35.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c9a8aa640 1 --2- 172.21.15.60:0/430270531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c9413ef20 0x7f5c940a87e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5c88009870 tx=0x7f5c8802ba40 comp rx=0 tx=0).stop 2024-03-27T21:41:35.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.267+0000 7f5c9a8aa640 1 -- 172.21.15.60:0/430270531 shutdown_connections 2024-03-27T21:41:35.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.268+0000 7f5c9a8aa640 1 --2- 172.21.15.60:0/430270531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c9413ef20 0x7f5c940a87e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.268+0000 7f5c9a8aa640 1 -- 172.21.15.60:0/430270531 >> 172.21.15.60:0/430270531 conn(0x7f5c940b7300 msgr2=0x7f5c940b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:35.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.268+0000 7f5c9a8aa640 1 -- 172.21.15.60:0/430270531 shutdown_connections 2024-03-27T21:41:35.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.268+0000 7f5c9a8aa640 1 -- 172.21.15.60:0/430270531 wait complete. 2024-03-27T21:41:35.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.268+0000 7f5c9a8aa640 10 monclient: build_initial_monmap 2024-03-27T21:41:35.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.268+0000 7f5c9a8aa640 10 monclient: monmap: 2024-03-27T21:41:35.270 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:35.270 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:35.270 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:35.270 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:35.270 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:35.270 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:35.270 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:35.270 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:35.270 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:35.270 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:35.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.269+0000 7f5c9a8aa640 1 librados: starting msgr at 2024-03-27T21:41:35.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.269+0000 7f5c9a8aa640 1 librados: starting objecter 2024-03-27T21:41:35.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.269+0000 7f5c9a8aa640 1 Processor -- start 2024-03-27T21:41:35.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.269+0000 7f5c9a8aa640 1 -- start start 2024-03-27T21:41:35.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.269+0000 7f5c9a8aa640 1 librados: setting wanted keys 2024-03-27T21:41:35.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.269+0000 7f5c9a8aa640 1 librados: calling monclient init 2024-03-27T21:41:35.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.269+0000 7f5c9a8aa640 10 monclient: init 2024-03-27T21:41:35.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c9a8aa640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:35.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c9a8aa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:41:35.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c9a8aa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c94071a50 0x7f5c94140c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:35.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c9a8aa640 10 monclient(hunting): picked mon.noname-a con 0x7f5c94071a50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:35.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c9a8aa640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c9413ef20 0x7f5c9412f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:35.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c9a8aa640 10 monclient(hunting): picked mon.noname-c con 0x7f5c9413ef20 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:35.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c9a8aa640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94141630 0x7f5c9412b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:35.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c9a8aa640 10 monclient(hunting): picked mon.noname-b con 0x7f5c94141630 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c9a8aa640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c9a8aa640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c940b6690 con 0x7f5c94071a50 2024-03-27T21:41:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c9a8aa640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c9a8aa640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c940b6930 con 0x7f5c94141630 2024-03-27T21:41:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c9a8aa640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c9a8aa640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9412fdc0 con 0x7f5c9413ef20 2024-03-27T21:41:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c9a8aa640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c9a8aa640 10 monclient(hunting): authenticate will time out at 936.491657s 2024-03-27T21:41:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c93fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c94071a50 0x7f5c94140c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c937fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c9413ef20 0x7f5c9412f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c98e47640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94141630 0x7f5c9412b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c937fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c9413ef20 0x7f5c9412f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:53274/0 (socket says 172.21.15.60:53274) 2024-03-27T21:41:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c937fe640 1 -- 172.21.15.60:0/3117454145 learned_addr learned my addr 172.21.15.60:0/3117454145 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c937fe640 10 monclient(hunting): get_auth_request con 0x7f5c9413ef20 auth_method 0 2024-03-27T21:41:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:35.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c937fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:35.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:35.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c98e47640 10 monclient(hunting): get_auth_request con 0x7f5c94141630 auth_method 0 2024-03-27T21:41:35.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c98e47640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:35.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c98e47640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:35.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c98e47640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:35.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c937fe640 1 -- 172.21.15.60:0/3117454145 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c9413ef20 msgr2=0x7f5c9412f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-27T21:41:35.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c937fe640 1 -- 172.21.15.60:0/3117454145 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c9413ef20 msgr2=0x7f5c9412f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-27T21:41:35.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.270+0000 7f5c937fe640 1 --2- 172.21.15.60:0/3117454145 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c9413ef20 0x7f5c9412f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-27T21:41:35.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c937fe640 1 --2- 172.21.15.60:0/3117454145 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c9413ef20 0x7f5c9412f290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-27T21:41:35.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c98e47640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:35.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c98e47640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:35.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c98e47640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:35.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c93fff640 10 monclient(hunting): get_auth_request con 0x7f5c94071a50 auth_method 0 2024-03-27T21:41:35.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:35.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c93fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:35.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:35.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:35.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:35.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:35.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c98e47640 10 monclient(hunting): handle_auth_done global_id 5017 payload 824 2024-03-27T21:41:35.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c98e47640 1 -- 172.21.15.60:0/3117454145 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c9413ef20 msgr2=0x7f5c9412f290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T21:41:35.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c98e47640 1 --2- 172.21.15.60:0/3117454145 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c9413ef20 0x7f5c9412f290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c98e47640 1 -- 172.21.15.60:0/3117454145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c94071a50 msgr2=0x7f5c94140c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:35.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c98e47640 1 --2- 172.21.15.60:0/3117454145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c94071a50 0x7f5c94140c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c98e47640 10 monclient: _finish_hunting 0 2024-03-27T21:41:35.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c98e47640 1 monclient: found mon.noname-b 2024-03-27T21:41:35.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c98e47640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:35.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c98e47640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:41:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c98e47640 1 -- 172.21.15.60:0/3117454145 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c880094d0 con 0x7f5c94141630 2024-03-27T21:41:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c93fff640 1 --2- 172.21.15.60:0/3117454145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c94071a50 0x7f5c94140c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c98e47640 1 --2- 172.21.15.60:0/3117454145 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94141630 0x7f5c9412b0d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00c700 tx=0x7f5c8c00cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c90ff9640 10 client.?.objecter ms_handle_connect 0x7f5c94141630 2024-03-27T21:41:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c90ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c90ff9640 1 -- 172.21.15.60:0/3117454145 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c003b70 con 0x7f5c94141630 2024-03-27T21:41:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c90ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c90ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c90ff9640 10 monclient: dump: 2024-03-27T21:41:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:35.276 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c90ff9640 10 monclient: _finish_auth 0 2024-03-27T21:41:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c90ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:41:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c90ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c9a8aa640 5 monclient: authenticate success, global_id 5017 2024-03-27T21:41:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c90ff9640 1 -- 172.21.15.60:0/3117454145 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8c003cd0 con 0x7f5c94141630 2024-03-27T21:41:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c9a8aa640 10 monclient: _renew_subs 2024-03-27T21:41:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c9a8aa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c9a8aa640 1 -- 172.21.15.60:0/3117454145 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c94130020 con 0x7f5c94141630 2024-03-27T21:41:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c90ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c90ff9640 1 -- 172.21.15.60:0/3117454145 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c005310 con 0x7f5c94141630 2024-03-27T21:41:35.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c90ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:35.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c90ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:35.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.271+0000 7f5c90ff9640 10 monclient: dump: 2024-03-27T21:41:35.277 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:35.277 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:35.277 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:35.277 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:35.277 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:35.277 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:35.277 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:35.277 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:35.277 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:35.277 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:35.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.272+0000 7f5c927fc640 4 set_mon_vals no callback set 2024-03-27T21:41:35.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.272+0000 7f5c9a8aa640 10 client.5017.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:35.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.272+0000 7f5c9a8aa640 10 monclient: _renew_subs 2024-03-27T21:41:35.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.272+0000 7f5c9a8aa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:35.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.272+0000 7f5c9a8aa640 1 -- 172.21.15.60:0/3117454145 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c9412b7f0 con 0x7f5c94141630 2024-03-27T21:41:35.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.272+0000 7f5c9a8aa640 1 librados: init done 2024-03-27T21:41:35.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.273+0000 7f5c90ff9640 1 -- 172.21.15.60:0/3117454145 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5c8c005470 con 0x7f5c94141630 2024-03-27T21:41:35.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.274+0000 7f5c9a8aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:35.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.274+0000 7f5c9a8aa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:35.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.274+0000 7f5c9a8aa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:35.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.274+0000 7f5c9a8aa640 1 -- 172.21.15.60:0/3117454145 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c9412b9e0 con 0x7f5c94141630 2024-03-27T21:41:35.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.274+0000 7f5c90ff9640 1 --2- 172.21.15.60:0/3117454145 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5c8006c590 0x7f5c8006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:35.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.274+0000 7f5c90ff9640 1 -- 172.21.15.60:0/3117454145 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f5c8c088560 con 0x7f5c94141630 2024-03-27T21:41:35.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.274+0000 7f5c90ff9640 10 client.5017.objecter ms_dispatch 0x7f5c94000b90 osd_map(68..68 src has 1..68) v4 2024-03-27T21:41:35.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.274+0000 7f5c90ff9640 3 client.5017.objecter handle_osd_map got epochs [68,68] > 0 2024-03-27T21:41:35.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.274+0000 7f5c90ff9640 3 client.5017.objecter handle_osd_map decoding full epoch 68 2024-03-27T21:41:35.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.274+0000 7f5c90ff9640 20 client.5017.objecter dump_active .. 0 homeless 2024-03-27T21:41:35.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.274+0000 7f5c93fff640 1 --2- 172.21.15.60:0/3117454145 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5c8006c590 0x7f5c8006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:35.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.274+0000 7f5c93fff640 10 monclient: get_auth_request con 0x7f5c8006c590 auth_method 0 2024-03-27T21:41:35.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.275+0000 7f5c90ff9640 10 client.5017.objecter ms_handle_connect 0x7f5c8006c590 2024-03-27T21:41:35.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.275+0000 7f5c93fff640 1 --2- 172.21.15.60:0/3117454145 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5c8006c590 0x7f5c8006ea50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5c84000f60 tx=0x7f5c84009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:35.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.279+0000 7f5c90ff9640 1 -- 172.21.15.60:0/3117454145 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5c8c053910 con 0x7f5c94141630 2024-03-27T21:41:35.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.279+0000 7f5c90ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:35.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.279+0000 7f5c90ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:35.354 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:35.368+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:35.372 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:35.372 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:35.372 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:35.373 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:35.381 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:41:35.455 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:35.469+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:35.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.472+0000 7f5c9a8aa640 10 start_mon_command cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}] 2024-03-27T21:41:35.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.472+0000 7f5c9a8aa640 10 monclient: _send_command 2 [{"prefix": "osd reweight", "id": 7, "weight": 1.0}] 2024-03-27T21:41:35.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.472+0000 7f5c9a8aa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:35.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.472+0000 7f5c9a8aa640 1 -- 172.21.15.60:0/3117454145 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f5c94069470 con 0x7f5c94141630 2024-03-27T21:41:35.526 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:35.526 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:35.526 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:35.527 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:35.527 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:35.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:41:35.578 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:35.592+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:35.640 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:35.640 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:35.640 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:35.640 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:41:35.640 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:35.650 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:41:35.656 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:35.670+0000 7fb850f34640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 56426) UID: 0 2024-03-27T21:41:35.758 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:35.772+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:35.766 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:41:35.795 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:35.796 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:35.796 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:35.796 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:35.804 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:41:35.859 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:35.873+0000 7fb850f34640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 56426) UID: 0 2024-03-27T21:41:35.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.890+0000 7f65731b5640 10 monclient: get_monmap_and_config 2024-03-27T21:41:35.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.890+0000 7f65731b5640 10 monclient: build_initial_monmap 2024-03-27T21:41:35.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.890+0000 7f65731b5640 10 monclient: monmap: 2024-03-27T21:41:35.877 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:35.877 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:35.878 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:35.891589+0000 2024-03-27T21:41:35.878 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:35.891589+0000 2024-03-27T21:41:35.878 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:35.878 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:35.878 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:35.878 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:35.878 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:35.878 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:35.878 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:35.878 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:35.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.891+0000 7f65731b5640 1 Processor -- start 2024-03-27T21:41:35.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.891+0000 7f65731b5640 1 -- start start 2024-03-27T21:41:35.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.891+0000 7f65731b5640 10 monclient: init 2024-03-27T21:41:35.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.891+0000 7f65731b5640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:35.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.891+0000 7f65731b5640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-03-27T21:41:35.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f65731b5640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f656c071a50 0x7f656c0a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:35.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f65731b5640 10 monclient(hunting): picked mon.noname-c con 0x7f656c071a50 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:35.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f65731b5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f656c13e810 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:35.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f65731b5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f656c13ec50 0x7f656c141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:35.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f65731b5640 10 monclient(hunting): picked mon.noname-a con 0x7f656c13ec50 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:35.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f65731b5640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656c0a8e90 con 0x7f656c13e810 2024-03-27T21:41:35.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f65731b5640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:35.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f65731b5640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f656c071e30 con 0x7f656c13ec50 2024-03-27T21:41:35.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f65731b5640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:35.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f65731b5640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f656c0aa040 con 0x7f656c071a50 2024-03-27T21:41:35.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f65731b5640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:35.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f65731b5640 10 monclient(hunting): authenticate will time out at 937.113076s 2024-03-27T21:41:35.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6571752640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f656c13ec50 0x7f656c141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:35.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6571752640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f656c13ec50 0x7f656c141d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50852/0 (socket says 172.21.15.60:50852) 2024-03-27T21:41:35.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6571752640 1 -- 172.21.15.60:0/2569524085 learned_addr learned my addr 172.21.15.60:0/2569524085 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:35.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6571752640 10 monclient(hunting): get_auth_request con 0x7f656c13ec50 auth_method 0 2024-03-27T21:41:35.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6571752640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:35.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6571752640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:35.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6571752640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:35.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6570f51640 1 --2- 172.21.15.60:0/2569524085 >> v2:172.21.15.191:3300/0 conn(0x7f656c071a50 0x7f656c0a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:35.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6571752640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:35.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6571752640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:35.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6571752640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:35.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6570f51640 10 monclient(hunting): get_auth_request con 0x7f656c071a50 auth_method 0 2024-03-27T21:41:35.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6570f51640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:35.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6570f51640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:35.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6570f51640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:35.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6570f51640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:35.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6570f51640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:35.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6570f51640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:35.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6571752640 10 monclient(hunting): handle_auth_done global_id 5076 payload 274 2024-03-27T21:41:35.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6571752640 1 -- 172.21.15.60:0/2569524085 >> v2:172.21.15.191:3300/0 conn(0x7f656c071a50 msgr2=0x7f656c0a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:35.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6571752640 1 --2- 172.21.15.60:0/2569524085 >> v2:172.21.15.191:3300/0 conn(0x7f656c071a50 0x7f656c0a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6571752640 1 -- 172.21.15.60:0/2569524085 >> v1:172.21.15.191:6789/0 conn(0x7f656c13e810 legacy=0x7f656c141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:35.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6571752640 10 monclient: _finish_hunting 0 2024-03-27T21:41:35.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6571752640 1 monclient: found mon.noname-a 2024-03-27T21:41:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6571752640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6571752640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.892+0000 7f6571752640 1 -- 172.21.15.60:0/2569524085 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656c142280 con 0x7f656c13ec50 2024-03-27T21:41:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.893+0000 7f6571752640 1 --2- 172.21.15.60:0/2569524085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f656c13ec50 0x7f656c141d40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f656c0b4a80 tx=0x7f656402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f8b075d4317353 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.893+0000 7f656b7fe640 1 -- 172.21.15.60:0/2569524085 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656402cb20 con 0x7f656c13ec50 2024-03-27T21:41:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.893+0000 7f656b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.893+0000 7f656b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.893+0000 7f656b7fe640 10 monclient: dump: 2024-03-27T21:41:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:35.894 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.893+0000 7f656b7fe640 10 monclient: _finish_auth 0 2024-03-27T21:41:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.893+0000 7f656b7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:41:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.893+0000 7f656b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.893+0000 7f656b7fe640 1 -- 172.21.15.60:0/2569524085 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656402cc80 con 0x7f656c13ec50 2024-03-27T21:41:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.893+0000 7f656b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.893+0000 7f65731b5640 5 monclient: authenticate success, global_id 5076 2024-03-27T21:41:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.893+0000 7f65731b5640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.893+0000 7f65731b5640 4 set_mon_vals no callback set 2024-03-27T21:41:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.893+0000 7f656b7fe640 1 -- 172.21.15.60:0/2569524085 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65640343f0 con 0x7f656c13ec50 2024-03-27T21:41:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.893+0000 7f656b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.893+0000 7f656b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.893+0000 7f656b7fe640 10 monclient: dump: 2024-03-27T21:41:35.895 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:35.895 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:35.895 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:35.895 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:35.895 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:35.895 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:35.895 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:35.895 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:35.895 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:35.895 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:35.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.894+0000 7f65731b5640 10 monclient: shutdown 2024-03-27T21:41:35.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.894+0000 7f65731b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:35.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.894+0000 7f65731b5640 1 -- 172.21.15.60:0/2569524085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f656c13ec50 msgr2=0x7f656c141d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:35.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.894+0000 7f65731b5640 1 --2- 172.21.15.60:0/2569524085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f656c13ec50 0x7f656c141d40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f656c0b4a80 tx=0x7f656402bf80 comp rx=0 tx=0).stop 2024-03-27T21:41:35.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.894+0000 7f65731b5640 1 -- 172.21.15.60:0/2569524085 shutdown_connections 2024-03-27T21:41:35.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.894+0000 7f65731b5640 1 --2- 172.21.15.60:0/2569524085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f656c13ec50 0x7f656c141d40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.894+0000 7f65731b5640 1 --2- 172.21.15.60:0/2569524085 >> v2:172.21.15.191:3300/0 conn(0x7f656c071a50 0x7f656c0a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.894+0000 7f65731b5640 1 -- 172.21.15.60:0/2569524085 >> 172.21.15.60:0/2569524085 conn(0x7f656c0b7300 msgr2=0x7f656c0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:35.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.894+0000 7f65731b5640 1 -- 172.21.15.60:0/2569524085 shutdown_connections 2024-03-27T21:41:35.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.894+0000 7f65731b5640 1 -- 172.21.15.60:0/2569524085 wait complete. 2024-03-27T21:41:35.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.895+0000 7f65731b5640 10 monclient: build_initial_monmap 2024-03-27T21:41:35.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.895+0000 7f65731b5640 10 monclient: monmap: 2024-03-27T21:41:35.896 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:35.896 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:35.896 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:35.896 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:35.896 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:35.896 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:35.896 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:35.897 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:35.897 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:35.897 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:35.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.895+0000 7f65731b5640 1 librados: starting msgr at 2024-03-27T21:41:35.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.895+0000 7f65731b5640 1 librados: starting objecter 2024-03-27T21:41:35.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.895+0000 7f65731b5640 1 Processor -- start 2024-03-27T21:41:35.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.895+0000 7f65731b5640 1 -- start start 2024-03-27T21:41:35.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.895+0000 7f65731b5640 1 librados: setting wanted keys 2024-03-27T21:41:35.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.895+0000 7f65731b5640 1 librados: calling monclient init 2024-03-27T21:41:35.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.895+0000 7f65731b5640 10 monclient: init 2024-03-27T21:41:35.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.896+0000 7f65731b5640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:35.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.896+0000 7f65731b5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:41:35.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.896+0000 7f65731b5640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f656c071a50 0x7f656c13a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:35.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.896+0000 7f65731b5640 10 monclient(hunting): picked mon.noname-c con 0x7f656c071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:35.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.896+0000 7f65731b5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f656c13e810 0x7f656c1d6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:35.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f65731b5640 10 monclient(hunting): picked mon.noname-a con 0x7f656c13e810 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:35.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f65731b5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f656c13ec50 0x7f656c1d6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:35.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f65731b5640 10 monclient(hunting): picked mon.noname-b con 0x7f656c13ec50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:35.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f65731b5640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:35.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f65731b5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c13a090 con 0x7f656c13e810 2024-03-27T21:41:35.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f65731b5640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:35.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f656bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f656c13e810 0x7f656c1d6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:35.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6570f51640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f656c071a50 0x7f656c13a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:35.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f656bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f656c13e810 0x7f656c1d6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50862/0 (socket says 172.21.15.60:50862) 2024-03-27T21:41:35.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f656bfff640 1 -- 172.21.15.60:0/3437457775 learned_addr learned my addr 172.21.15.60:0/3437457775 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:35.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f65731b5640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c1374a0 con 0x7f656c13ec50 2024-03-27T21:41:35.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f65731b5640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:35.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f656c0aaab0 con 0x7f656c071a50 2024-03-27T21:41:35.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f65731b5640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:35.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f65731b5640 10 monclient(hunting): authenticate will time out at 937.118067s 2024-03-27T21:41:35.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f656bfff640 10 monclient(hunting): get_auth_request con 0x7f656c13e810 auth_method 0 2024-03-27T21:41:35.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f656bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:35.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f656bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:35.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f656bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:35.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6570f51640 10 monclient(hunting): get_auth_request con 0x7f656c071a50 auth_method 0 2024-03-27T21:41:35.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6570f51640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:35.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6570f51640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:35.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6570f51640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:35.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6571752640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f656c13ec50 0x7f656c1d6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:35.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f656bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:35.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f656bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:35.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f656bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:35.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6571752640 10 monclient(hunting): get_auth_request con 0x7f656c13ec50 auth_method 0 2024-03-27T21:41:35.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6571752640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:35.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6571752640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:35.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6571752640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:35.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6571752640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:35.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6571752640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:35.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6571752640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:35.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6570f51640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:35.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6570f51640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:35.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6570f51640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:35.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6571752640 10 monclient(hunting): handle_auth_done global_id 5026 payload 824 2024-03-27T21:41:35.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6571752640 1 -- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f656c071a50 msgr2=0x7f656c13a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:35.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6571752640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f656c071a50 0x7f656c13a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6571752640 1 -- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f656c13e810 msgr2=0x7f656c1d6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:35.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6571752640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f656c13e810 0x7f656c1d6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6571752640 10 monclient: _finish_hunting 0 2024-03-27T21:41:35.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6571752640 1 monclient: found mon.noname-b 2024-03-27T21:41:35.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6571752640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:35.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6571752640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:41:35.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6571752640 1 -- 172.21.15.60:0/3437457775 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6564009b30 con 0x7f656c13ec50 2024-03-27T21:41:35.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f656bfff640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f656c13e810 0x7f656c1d6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:35.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f6571752640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f656c13ec50 0x7f656c1d6d50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6564032000 tx=0x7f6564004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:35.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f65697fa640 10 client.?.objecter ms_handle_connect 0x7f656c13ec50 2024-03-27T21:41:35.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.897+0000 7f65697fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:35.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6564009510 con 0x7f656c13ec50 2024-03-27T21:41:35.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:35.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65697fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:35.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65697fa640 10 monclient: dump: 2024-03-27T21:41:35.903 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:35.903 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:35.903 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:35.903 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:35.903 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:35.903 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:35.903 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:35.903 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:35.903 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:35.903 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:35.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65697fa640 10 monclient: _finish_auth 0 2024-03-27T21:41:35.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65697fa640 10 monclient: _check_auth_tickets 2024-03-27T21:41:35.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:35.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65731b5640 5 monclient: authenticate success, global_id 5026 2024-03-27T21:41:35.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6564007e60 con 0x7f656c13ec50 2024-03-27T21:41:35.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65697fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:35.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656403e670 con 0x7f656c13ec50 2024-03-27T21:41:35.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:35.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:35.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65697fa640 10 monclient: dump: 2024-03-27T21:41:35.904 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:35.904 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:35.904 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:35.904 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:35.904 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:35.904 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:35.904 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:35.904 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:35.904 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:35.905 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:35.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f656affd640 4 set_mon_vals no callback set 2024-03-27T21:41:35.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65731b5640 10 monclient: _renew_subs 2024-03-27T21:41:35.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65731b5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:35.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656c0aad20 con 0x7f656c13ec50 2024-03-27T21:41:35.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65731b5640 10 client.5026.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:35.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65731b5640 10 monclient: _renew_subs 2024-03-27T21:41:35.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65731b5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:35.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656c1d7350 con 0x7f656c13ec50 2024-03-27T21:41:35.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.898+0000 7f65731b5640 1 librados: init done 2024-03-27T21:41:35.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.899+0000 7f65731b5640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:41:35.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.899+0000 7f65731b5640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:35.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.899+0000 7f65731b5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:35.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.899+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f656c1d64b0 con 0x7f656c13ec50 2024-03-27T21:41:35.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.900+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f656403e7d0 con 0x7f656c13ec50 2024-03-27T21:41:35.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.900+0000 7f65697fa640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f655006c590 0x7f655006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:35.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.900+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f65640abdb0 con 0x7f656c13ec50 2024-03-27T21:41:35.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.900+0000 7f65697fa640 10 client.5026.objecter ms_dispatch 0x7f656c000b90 osd_map(68..68 src has 1..68) v4 2024-03-27T21:41:35.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.900+0000 7f65697fa640 3 client.5026.objecter handle_osd_map got epochs [68,68] > 0 2024-03-27T21:41:35.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.900+0000 7f65697fa640 3 client.5026.objecter handle_osd_map decoding full epoch 68 2024-03-27T21:41:35.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.901+0000 7f65697fa640 20 client.5026.objecter dump_active .. 0 homeless 2024-03-27T21:41:35.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.901+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v68) v1 ==== 54+0+16 (secure 0 0 0) 0x7f65640ae3f0 con 0x7f656c13ec50 2024-03-27T21:41:35.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.901+0000 7f65697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:35.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.901+0000 7f65697fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:35.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.901+0000 7f6570f51640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f655006c590 0x7f655006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:35.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.901+0000 7f6570f51640 10 monclient: get_auth_request con 0x7f655006c590 auth_method 0 2024-03-27T21:41:35.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.901+0000 7f65731b5640 10 client.5026.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:35.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.901+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign -1 1 2024-03-27T21:41:35.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.901+0000 7f65731b5640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f656c1d9930 0x7f656c1d9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:35.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.901+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c13f0c0 osd=0 3 2024-03-27T21:41:35.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.901+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c13f0c0 osd=0 3 2024-03-27T21:41:35.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.901+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c13f0c0 osd=0 3 2024-03-27T21:41:35.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.901+0000 7f65731b5640 15 client.5026.objecter _session_command_op_remove -1 1 2024-03-27T21:41:35.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.901+0000 7f65731b5640 20 client.5026.objecter get_session s=0x7f656c13f0c0 osd=0 3 2024-03-27T21:41:35.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.901+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign 0 1 2024-03-27T21:41:35.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.901+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c13f0c0 osd=0 4 2024-03-27T21:41:35.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.901+0000 7f65731b5640 10 client.5026.objecter _send_command 1 2024-03-27T21:41:35.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.901+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f656c0aa590 con 0x7f656c1d9930 2024-03-27T21:41:35.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.901+0000 7f656bfff640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f656c1d9930 0x7f656c1d9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:35.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.901+0000 7f656bfff640 10 monclient: get_auth_request con 0x7f656c1d9930 auth_method 0 2024-03-27T21:41:35.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.902+0000 7f65697fa640 10 client.5026.objecter ms_handle_connect 0x7f656c1d9930 2024-03-27T21:41:35.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.902+0000 7f656bfff640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f656c1d9930 0x7f656c1d9d10 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:35.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.902+0000 7f65697fa640 10 client.5026.objecter ms_handle_connect 0x7f655006c590 2024-03-27T21:41:35.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.902+0000 7f6570f51640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f655006c590 0x7f655006ea50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6560009b90 tx=0x7f6560009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:35.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.903+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6558001950 con 0x7f656c1d9930 2024-03-27T21:41:35.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.903+0000 7f65697fa640 10 client.5026.objecter ms_dispatch 0x7f656c000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:41:35.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.903+0000 7f65697fa640 10 client.5026.objecter _finish_command 1 = system:0 2024-03-27T21:41:35.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.903+0000 7f65697fa640 20 client.5026.objecter put_session s=0x7f656c13f0c0 osd=0 4 2024-03-27T21:41:35.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.903+0000 7f65697fa640 15 client.5026.objecter _session_command_op_remove 0 1 2024-03-27T21:41:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.917+0000 7f65731b5640 10 client.5026.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.917+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign -1 2 2024-03-27T21:41:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.917+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c13f0c0 osd=0 3 2024-03-27T21:41:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.917+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c13f0c0 osd=0 3 2024-03-27T21:41:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.917+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c13f0c0 osd=0 3 2024-03-27T21:41:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.917+0000 7f65731b5640 15 client.5026.objecter _session_command_op_remove -1 2 2024-03-27T21:41:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.917+0000 7f65731b5640 20 client.5026.objecter get_session s=0x7f656c13f0c0 osd=0 3 2024-03-27T21:41:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.917+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign 0 2 2024-03-27T21:41:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.917+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c13f0c0 osd=0 4 2024-03-27T21:41:35.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.917+0000 7f65731b5640 10 client.5026.objecter _send_command 2 2024-03-27T21:41:35.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.917+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f656c14bde0 con 0x7f656c1d9930 2024-03-27T21:41:35.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.918+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f6558013390 con 0x7f656c1d9930 2024-03-27T21:41:35.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.918+0000 7f65697fa640 10 client.5026.objecter ms_dispatch 0x7f656c000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:35.911 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:41:35.914 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:41:35.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.918+0000 7f65697fa640 10 client.5026.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:35.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.918+0000 7f65697fa640 20 client.5026.objecter put_session s=0x7f656c13f0c0 osd=0 4 2024-03-27T21:41:35.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.918+0000 7f65697fa640 15 client.5026.objecter _session_command_op_remove 0 2 2024-03-27T21:41:35.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.919+0000 7f65731b5640 10 client.5026.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:35.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.919+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign -1 3 2024-03-27T21:41:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.919+0000 7f65731b5640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f656c133110 0x7f656c1334f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.919+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c067900 osd=1 3 2024-03-27T21:41:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.919+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c067900 osd=1 3 2024-03-27T21:41:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.919+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c067900 osd=1 3 2024-03-27T21:41:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.919+0000 7f65731b5640 15 client.5026.objecter _session_command_op_remove -1 3 2024-03-27T21:41:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.919+0000 7f65731b5640 20 client.5026.objecter get_session s=0x7f656c067900 osd=1 3 2024-03-27T21:41:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.919+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign 1 3 2024-03-27T21:41:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.919+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c067900 osd=1 4 2024-03-27T21:41:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.919+0000 7f65731b5640 10 client.5026.objecter _send_command 3 2024-03-27T21:41:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.919+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f656c133a30 con 0x7f656c133110 2024-03-27T21:41:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.919+0000 7f6570f51640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f656c133110 0x7f656c1334f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.919+0000 7f6570f51640 10 monclient: get_auth_request con 0x7f656c133110 auth_method 0 2024-03-27T21:41:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.919+0000 7f65697fa640 10 client.5026.objecter ms_handle_connect 0x7f656c133110 2024-03-27T21:41:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.919+0000 7f6570f51640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f656c133110 0x7f656c1334f0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:35.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.920+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f656c133a30 con 0x7f656c133110 2024-03-27T21:41:35.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.920+0000 7f65697fa640 10 client.5026.objecter ms_dispatch 0x7f656c000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:41:35.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.920+0000 7f65697fa640 10 client.5026.objecter _finish_command 3 = system:0 2024-03-27T21:41:35.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.920+0000 7f65697fa640 20 client.5026.objecter put_session s=0x7f656c067900 osd=1 4 2024-03-27T21:41:35.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.920+0000 7f65697fa640 15 client.5026.objecter _session_command_op_remove 1 3 2024-03-27T21:41:35.917 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:35.917 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:35.917 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:35.917 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:35.917 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:35.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.930+0000 7f5c90ff9640 1 -- 172.21.15.60:0/3117454145 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v69) v1 ==== 113+0+0 (secure 0 0 0) 0x7f5c8c056f30 con 0x7f5c94141630 2024-03-27T21:41:35.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.930+0000 7f5c90ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd reweight", "id": 7, "weight": 1.0}] 2024-03-27T21:41:35.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.930+0000 7f5c90ff9640 10 monclient: _finish_command 2 = system:0 reweighted osd.7 to 1 (10000) 2024-03-27T21:41:35.917 INFO:teuthology.orchestra.run.smithi060.stderr:reweighted osd.7 to 1 (10000) 2024-03-27T21:41:35.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.933+0000 7f5c9a8aa640 10 librados: watch_flush enter 2024-03-27T21:41:35.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.933+0000 7f5c9a8aa640 10 librados: watch_flush exit 2024-03-27T21:41:35.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.933+0000 7f5c9a8aa640 20 client.5017.objecter shutdown clearing up homeless session... 2024-03-27T21:41:35.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.933+0000 7f5c9a8aa640 10 client.5017.objecter successfully canceled tick 2024-03-27T21:41:35.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.933+0000 7f5c9a8aa640 1 -- 172.21.15.60:0/3117454145 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5c8006c590 msgr2=0x7f5c8006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:35.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.933+0000 7f5c9a8aa640 1 --2- 172.21.15.60:0/3117454145 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5c8006c590 0x7f5c8006ea50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5c84000f60 tx=0x7f5c84009040 comp rx=0 tx=0).stop 2024-03-27T21:41:35.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.933+0000 7f5c9a8aa640 10 monclient: shutdown 2024-03-27T21:41:35.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.933+0000 7f5c9a8aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:35.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.933+0000 7f5c9a8aa640 1 -- 172.21.15.60:0/3117454145 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94141630 msgr2=0x7f5c9412b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:35.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.933+0000 7f5c9a8aa640 1 --2- 172.21.15.60:0/3117454145 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94141630 0x7f5c9412b0d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00c700 tx=0x7f5c8c00cbc0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.933+0000 7f5c9a8aa640 1 -- 172.21.15.60:0/3117454145 shutdown_connections 2024-03-27T21:41:35.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.933+0000 7f5c9a8aa640 1 --2- 172.21.15.60:0/3117454145 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5c8006c590 0x7f5c8006ea50 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.933+0000 7f5c9a8aa640 1 --2- 172.21.15.60:0/3117454145 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94141630 0x7f5c9412b0d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.933+0000 7f5c9a8aa640 1 --2- 172.21.15.60:0/3117454145 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c9413ef20 0x7f5c9412f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.933+0000 7f5c9a8aa640 1 --2- 172.21.15.60:0/3117454145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c94071a50 0x7f5c94140c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:35.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.933+0000 7f5c9a8aa640 1 -- 172.21.15.60:0/3117454145 >> 172.21.15.60:0/3117454145 conn(0x7f5c940b7300 msgr2=0x7f5c9413e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:35.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.933+0000 7f5c9a8aa640 1 -- 172.21.15.60:0/3117454145 shutdown_connections 2024-03-27T21:41:35.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.934+0000 7f5c9a8aa640 1 -- 172.21.15.60:0/3117454145 wait complete. 2024-03-27T21:41:35.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.934+0000 7f5c9a8aa640 1 librados: shutdown 2024-03-27T21:41:35.922 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:41:35.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.947+0000 7f65731b5640 10 client.5026.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:35.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.947+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign -1 4 2024-03-27T21:41:35.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.947+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c067900 osd=1 3 2024-03-27T21:41:35.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.947+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c067900 osd=1 3 2024-03-27T21:41:35.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.947+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c067900 osd=1 3 2024-03-27T21:41:35.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.947+0000 7f65731b5640 15 client.5026.objecter _session_command_op_remove -1 4 2024-03-27T21:41:35.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.947+0000 7f65731b5640 20 client.5026.objecter get_session s=0x7f656c067900 osd=1 3 2024-03-27T21:41:35.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.947+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign 1 4 2024-03-27T21:41:35.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.947+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c067900 osd=1 4 2024-03-27T21:41:35.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.947+0000 7f65731b5640 10 client.5026.objecter _send_command 4 2024-03-27T21:41:35.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.947+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f656c069470 con 0x7f656c133110 2024-03-27T21:41:35.936 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-27T21:41:35.937 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T21:41:35.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.950+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f656c069470 con 0x7f656c133110 2024-03-27T21:41:35.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.950+0000 7f65697fa640 10 client.5026.objecter ms_dispatch 0x7f656c000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:35.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.950+0000 7f65697fa640 10 client.5026.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:35.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.950+0000 7f65697fa640 20 client.5026.objecter put_session s=0x7f656c067900 osd=1 4 2024-03-27T21:41:35.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.950+0000 7f65697fa640 15 client.5026.objecter _session_command_op_remove 1 4 2024-03-27T21:41:35.938 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:41:35.939 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:41:35.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.950+0000 7f65731b5640 10 client.5026.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:35.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.950+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign -1 5 2024-03-27T21:41:35.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.950+0000 7f65731b5640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f656c13fbf0 0x7f656c13ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:35.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.950+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c0a8560 osd=2 3 2024-03-27T21:41:35.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.950+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c0a8560 osd=2 3 2024-03-27T21:41:35.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.950+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c0a8560 osd=2 3 2024-03-27T21:41:35.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.950+0000 7f65731b5640 15 client.5026.objecter _session_command_op_remove -1 5 2024-03-27T21:41:35.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.950+0000 7f65731b5640 20 client.5026.objecter get_session s=0x7f656c0a8560 osd=2 3 2024-03-27T21:41:35.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.950+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign 2 5 2024-03-27T21:41:35.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.950+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c0a8560 osd=2 4 2024-03-27T21:41:35.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.950+0000 7f65731b5640 10 client.5026.objecter _send_command 5 2024-03-27T21:41:35.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.950+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 --> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f656c0a9260 con 0x7f656c13fbf0 2024-03-27T21:41:35.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.950+0000 7f656bfff640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f656c13fbf0 0x7f656c13ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:35.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.950+0000 7f656bfff640 10 monclient: get_auth_request con 0x7f656c13fbf0 auth_method 0 2024-03-27T21:41:35.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.951+0000 7f65697fa640 10 client.5026.objecter ms_handle_connect 0x7f656c13fbf0 2024-03-27T21:41:35.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.951+0000 7f656bfff640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f656c13fbf0 0x7f656c13ffd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:35.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.952+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== osd.2 v2:172.21.15.60:6824/56434 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f656c0a9260 con 0x7f656c13fbf0 2024-03-27T21:41:35.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.952+0000 7f65697fa640 10 client.5026.objecter ms_dispatch 0x7f656c000b90 command_reply(tid 5: 0 ) v1 2024-03-27T21:41:35.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.952+0000 7f65697fa640 10 client.5026.objecter _finish_command 5 = system:0 2024-03-27T21:41:35.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.952+0000 7f65697fa640 20 client.5026.objecter put_session s=0x7f656c0a8560 osd=2 4 2024-03-27T21:41:35.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.952+0000 7f65697fa640 15 client.5026.objecter _session_command_op_remove 2 5 2024-03-27T21:41:35.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.979+0000 7f65731b5640 10 client.5026.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:35.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.979+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign -1 6 2024-03-27T21:41:35.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.979+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c0a8560 osd=2 3 2024-03-27T21:41:35.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.979+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c0a8560 osd=2 3 2024-03-27T21:41:35.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.979+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c0a8560 osd=2 3 2024-03-27T21:41:35.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.979+0000 7f65731b5640 15 client.5026.objecter _session_command_op_remove -1 6 2024-03-27T21:41:35.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.979+0000 7f65731b5640 20 client.5026.objecter get_session s=0x7f656c0a8560 osd=2 3 2024-03-27T21:41:35.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.979+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign 2 6 2024-03-27T21:41:35.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.979+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c0a8560 osd=2 4 2024-03-27T21:41:35.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.979+0000 7f65731b5640 10 client.5026.objecter _send_command 6 2024-03-27T21:41:35.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.979+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 --> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f656c13f670 con 0x7f656c13fbf0 2024-03-27T21:41:35.967 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:35.979+0000 7fb850f34640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 56426) UID: 0 2024-03-27T21:41:35.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.981+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== osd.2 v2:172.21.15.60:6824/56434 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f656c13f670 con 0x7f656c13fbf0 2024-03-27T21:41:35.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.981+0000 7f65697fa640 10 client.5026.objecter ms_dispatch 0x7f656c000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:35.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.981+0000 7f65697fa640 10 client.5026.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:35.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.981+0000 7f65697fa640 20 client.5026.objecter put_session s=0x7f656c0a8560 osd=2 4 2024-03-27T21:41:35.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.981+0000 7f65697fa640 15 client.5026.objecter _session_command_op_remove 2 6 2024-03-27T21:41:35.969 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2: {} 2024-03-27T21:41:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-27T21:41:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.982+0000 7f65731b5640 10 client.5026.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.982+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign -1 7 2024-03-27T21:41:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.982+0000 7f65731b5640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f656c066250 0x7f656c066670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.982+0000 7f6571752640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f656c066250 0x7f656c066670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.982+0000 7f6571752640 10 monclient: get_auth_request con 0x7f656c066250 auth_method 0 2024-03-27T21:41:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.982+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c13fa20 osd=3 3 2024-03-27T21:41:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.982+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c13fa20 osd=3 3 2024-03-27T21:41:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.983+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c13fa20 osd=3 3 2024-03-27T21:41:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.983+0000 7f6571752640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f656c066250 0x7f656c066670 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.983+0000 7f65731b5640 15 client.5026.objecter _session_command_op_remove -1 7 2024-03-27T21:41:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.983+0000 7f65731b5640 20 client.5026.objecter get_session s=0x7f656c13fa20 osd=3 3 2024-03-27T21:41:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.983+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign 3 7 2024-03-27T21:41:35.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.983+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c13fa20 osd=3 4 2024-03-27T21:41:35.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.983+0000 7f65697fa640 10 client.5026.objecter ms_handle_connect 0x7f656c066250 2024-03-27T21:41:35.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.983+0000 7f65731b5640 10 client.5026.objecter _send_command 7 2024-03-27T21:41:35.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.983+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f656c1eb740 con 0x7f656c066250 2024-03-27T21:41:35.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.984+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f656c1eb740 con 0x7f656c066250 2024-03-27T21:41:35.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.984+0000 7f65697fa640 10 client.5026.objecter ms_dispatch 0x7f656c000b90 command_reply(tid 7: 0 ) v1 2024-03-27T21:41:35.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.984+0000 7f65697fa640 10 client.5026.objecter _finish_command 7 = system:0 2024-03-27T21:41:35.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.984+0000 7f65697fa640 20 client.5026.objecter put_session s=0x7f656c13fa20 osd=3 4 2024-03-27T21:41:35.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:35.984+0000 7f65697fa640 15 client.5026.objecter _session_command_op_remove 3 7 2024-03-27T21:41:36.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.012+0000 7f65731b5640 10 client.5026.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:36.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.012+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign -1 8 2024-03-27T21:41:36.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.012+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c13fa20 osd=3 3 2024-03-27T21:41:36.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.012+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c13fa20 osd=3 3 2024-03-27T21:41:36.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.012+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c13fa20 osd=3 3 2024-03-27T21:41:36.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.012+0000 7f65731b5640 15 client.5026.objecter _session_command_op_remove -1 8 2024-03-27T21:41:36.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.012+0000 7f65731b5640 20 client.5026.objecter get_session s=0x7f656c13fa20 osd=3 3 2024-03-27T21:41:36.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.012+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign 3 8 2024-03-27T21:41:36.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.012+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c13fa20 osd=3 4 2024-03-27T21:41:36.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.012+0000 7f65731b5640 10 client.5026.objecter _send_command 8 2024-03-27T21:41:36.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.012+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f656c065cd0 con 0x7f656c066250 2024-03-27T21:41:36.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.015+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f656c065cd0 con 0x7f656c066250 2024-03-27T21:41:36.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.015+0000 7f65697fa640 10 client.5026.objecter ms_dispatch 0x7f656c000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:36.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.015+0000 7f65697fa640 10 client.5026.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:36.002 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:41:36.005 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:41:36.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.015+0000 7f65697fa640 20 client.5026.objecter put_session s=0x7f656c13fa20 osd=3 4 2024-03-27T21:41:36.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.015+0000 7f65697fa640 15 client.5026.objecter _session_command_op_remove 3 8 2024-03-27T21:41:36.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.015+0000 7f65731b5640 10 client.5026.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:36.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.015+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign -1 9 2024-03-27T21:41:36.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.015+0000 7f65731b5640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f656c065e40 0x7f656c1ea030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.015+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c065cd0 osd=4 3 2024-03-27T21:41:36.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.015+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c065cd0 osd=4 3 2024-03-27T21:41:36.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.015+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c065cd0 osd=4 3 2024-03-27T21:41:36.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.015+0000 7f65731b5640 15 client.5026.objecter _session_command_op_remove -1 9 2024-03-27T21:41:36.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.015+0000 7f65731b5640 20 client.5026.objecter get_session s=0x7f656c065cd0 osd=4 3 2024-03-27T21:41:36.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.015+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign 4 9 2024-03-27T21:41:36.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.015+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c065cd0 osd=4 4 2024-03-27T21:41:36.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.015+0000 7f65731b5640 10 client.5026.objecter _send_command 9 2024-03-27T21:41:36.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.015+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f656c02be40 con 0x7f656c065e40 2024-03-27T21:41:36.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.015+0000 7f6570f51640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f656c065e40 0x7f656c1ea030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:36.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.015+0000 7f6570f51640 10 monclient: get_auth_request con 0x7f656c065e40 auth_method 0 2024-03-27T21:41:36.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.016+0000 7f6570f51640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f656c065e40 0x7f656c1ea030 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:36.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.016+0000 7f65697fa640 10 client.5026.objecter ms_handle_connect 0x7f656c065e40 2024-03-27T21:41:36.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.017+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f656c02be40 con 0x7f656c065e40 2024-03-27T21:41:36.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.017+0000 7f65697fa640 10 client.5026.objecter ms_dispatch 0x7f656c000b90 command_reply(tid 9: 0 ) v1 2024-03-27T21:41:36.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.017+0000 7f65697fa640 10 client.5026.objecter _finish_command 9 = system:0 2024-03-27T21:41:36.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.017+0000 7f65697fa640 20 client.5026.objecter put_session s=0x7f656c065cd0 osd=4 4 2024-03-27T21:41:36.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.017+0000 7f65697fa640 15 client.5026.objecter _session_command_op_remove 4 9 2024-03-27T21:41:36.027 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:36.027 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:36.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:36.027 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:41:36.027 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:36.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.045+0000 7f65731b5640 10 client.5026.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:36.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.045+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign -1 10 2024-03-27T21:41:36.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.045+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c065cd0 osd=4 3 2024-03-27T21:41:36.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.045+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c065cd0 osd=4 3 2024-03-27T21:41:36.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.045+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c065cd0 osd=4 3 2024-03-27T21:41:36.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.045+0000 7f65731b5640 15 client.5026.objecter _session_command_op_remove -1 10 2024-03-27T21:41:36.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.045+0000 7f65731b5640 20 client.5026.objecter get_session s=0x7f656c065cd0 osd=4 3 2024-03-27T21:41:36.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.045+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign 4 10 2024-03-27T21:41:36.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.046+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c065cd0 osd=4 4 2024-03-27T21:41:36.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.046+0000 7f65731b5640 10 client.5026.objecter _send_command 10 2024-03-27T21:41:36.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.046+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f656c02be40 con 0x7f656c065e40 2024-03-27T21:41:36.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.047+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f656c02be40 con 0x7f656c065e40 2024-03-27T21:41:36.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.048+0000 7f65697fa640 10 client.5026.objecter ms_dispatch 0x7f656c000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:36.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.048+0000 7f65697fa640 10 client.5026.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:36.035 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:41:36.037 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:41:36.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.048+0000 7f65697fa640 20 client.5026.objecter put_session s=0x7f656c065cd0 osd=4 4 2024-03-27T21:41:36.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.048+0000 7f65697fa640 15 client.5026.objecter _session_command_op_remove 4 10 2024-03-27T21:41:36.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.048+0000 7f65731b5640 10 client.5026.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:36.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.048+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign -1 11 2024-03-27T21:41:36.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.048+0000 7f65731b5640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f656c1eaa80 0x7f656c1eaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.048+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c02be40 osd=5 3 2024-03-27T21:41:36.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.048+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c02be40 osd=5 3 2024-03-27T21:41:36.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.048+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c02be40 osd=5 3 2024-03-27T21:41:36.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.048+0000 7f65731b5640 15 client.5026.objecter _session_command_op_remove -1 11 2024-03-27T21:41:36.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.048+0000 7f65731b5640 20 client.5026.objecter get_session s=0x7f656c02be40 osd=5 3 2024-03-27T21:41:36.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.048+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign 5 11 2024-03-27T21:41:36.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.048+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c02be40 osd=5 4 2024-03-27T21:41:36.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.048+0000 7f65731b5640 10 client.5026.objecter _send_command 11 2024-03-27T21:41:36.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.048+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f656c02bfb0 con 0x7f656c1eaa80 2024-03-27T21:41:36.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.048+0000 7f656bfff640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f656c1eaa80 0x7f656c1eaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:36.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.048+0000 7f656bfff640 10 monclient: get_auth_request con 0x7f656c1eaa80 auth_method 0 2024-03-27T21:41:36.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.049+0000 7f656bfff640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f656c1eaa80 0x7f656c1eaee0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:36.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.049+0000 7f65697fa640 10 client.5026.objecter ms_handle_connect 0x7f656c1eaa80 2024-03-27T21:41:36.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.049+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f656c02bfb0 con 0x7f656c1eaa80 2024-03-27T21:41:36.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.049+0000 7f65697fa640 10 client.5026.objecter ms_dispatch 0x7f656c000b90 command_reply(tid 11: 0 ) v1 2024-03-27T21:41:36.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.049+0000 7f65697fa640 10 client.5026.objecter _finish_command 11 = system:0 2024-03-27T21:41:36.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.049+0000 7f65697fa640 20 client.5026.objecter put_session s=0x7f656c02be40 osd=5 4 2024-03-27T21:41:36.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.049+0000 7f65697fa640 15 client.5026.objecter _session_command_op_remove 5 11 2024-03-27T21:41:36.040 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:41:36.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.064+0000 7f036f89d640 10 monclient: get_monmap_and_config 2024-03-27T21:41:36.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.064+0000 7f036f89d640 10 monclient: build_initial_monmap 2024-03-27T21:41:36.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.064+0000 7f036f89d640 10 monclient: monmap: 2024-03-27T21:41:36.051 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:36.051 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:36.051 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:36.066198+0000 2024-03-27T21:41:36.051 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:36.066198+0000 2024-03-27T21:41:36.051 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:36.051 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:36.052 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:36.052 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:36.052 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:36.052 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:36.052 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:36.052 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:36.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.064+0000 7f036f89d640 1 Processor -- start 2024-03-27T21:41:36.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036f89d640 1 -- start start 2024-03-27T21:41:36.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036f89d640 10 monclient: init 2024-03-27T21:41:36.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036f89d640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:36.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036f89d640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-03-27T21:41:36.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036f89d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0368071a50 0x7f03680a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036f89d640 10 monclient(hunting): picked mon.noname-c con 0x7f0368071a50 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:36.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036f89d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f036813e810 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:36.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036f89d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f036813ec50 0x7f0368141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036f89d640 10 monclient(hunting): picked mon.noname-b con 0x7f036813ec50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:36.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036f89d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03680a8e90 con 0x7f036813e810 2024-03-27T21:41:36.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036f89d640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:36.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036f89d640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0368071e30 con 0x7f0368071a50 2024-03-27T21:41:36.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036f89d640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:36.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036f89d640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03680aa040 con 0x7f036813ec50 2024-03-27T21:41:36.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036f89d640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:36.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036f89d640 10 monclient(hunting): authenticate will time out at 937.287620s 2024-03-27T21:41:36.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036de3a640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f036813ec50 0x7f0368141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:36.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036de3a640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f036813ec50 0x7f0368141d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:53304/0 (socket says 172.21.15.60:53304) 2024-03-27T21:41:36.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036de3a640 1 -- 172.21.15.60:0/1703998096 learned_addr learned my addr 172.21.15.60:0/1703998096 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:36.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036de3a640 10 monclient(hunting): get_auth_request con 0x7f036813ec50 auth_method 0 2024-03-27T21:41:36.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036de3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:36.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036de3a640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:36.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.065+0000 7f036de3a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:36.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.066+0000 7f036de3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:36.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.066+0000 7f036de3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:36.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.066+0000 7f036de3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:36.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.066+0000 7f036de3a640 10 monclient(hunting): handle_auth_done global_id 4844 payload 274 2024-03-27T21:41:36.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.066+0000 7f036de3a640 1 -- 172.21.15.60:0/1703998096 >> v2:172.21.15.191:3300/0 conn(0x7f0368071a50 msgr2=0x7f03680a87e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-27T21:41:36.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.066+0000 7f036de3a640 1 --2- 172.21.15.60:0/1703998096 >> v2:172.21.15.191:3300/0 conn(0x7f0368071a50 0x7f03680a87e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.066+0000 7f036de3a640 1 -- 172.21.15.60:0/1703998096 >> v1:172.21.15.191:6789/0 conn(0x7f036813e810 legacy=0x7f0368141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:36.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.066+0000 7f036de3a640 10 monclient: _finish_hunting 0 2024-03-27T21:41:36.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.066+0000 7f036de3a640 1 monclient: found mon.noname-b 2024-03-27T21:41:36.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.066+0000 7f036de3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:36.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.066+0000 7f036de3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:36.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.066+0000 7f036de3a640 1 -- 172.21.15.60:0/1703998096 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0368142280 con 0x7f036813ec50 2024-03-27T21:41:36.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.066+0000 7f036de3a640 1 --2- 172.21.15.60:0/1703998096 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f036813ec50 0x7f0368141d40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0360009e50 tx=0x7f036002bf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=358fc3458c7bbcd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:36.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.067+0000 7f035ffff640 1 -- 172.21.15.60:0/1703998096 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036002caa0 con 0x7f036813ec50 2024-03-27T21:41:36.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.067+0000 7f035ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:36.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.067+0000 7f035ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:36.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.067+0000 7f035ffff640 10 monclient: dump: 2024-03-27T21:41:36.066 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:36.067 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.067+0000 7f035ffff640 10 monclient: _finish_auth 0 2024-03-27T21:41:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.067+0000 7f035ffff640 10 monclient: _check_auth_tickets 2024-03-27T21:41:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.067+0000 7f035ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.067+0000 7f035ffff640 1 -- 172.21.15.60:0/1703998096 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036002cc00 con 0x7f036813ec50 2024-03-27T21:41:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.067+0000 7f035ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.067+0000 7f035ffff640 1 -- 172.21.15.60:0/1703998096 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03600343f0 con 0x7f036813ec50 2024-03-27T21:41:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.067+0000 7f035ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.067+0000 7f035ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.067+0000 7f035ffff640 10 monclient: dump: 2024-03-27T21:41:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:36.069 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:36.069 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:36.069 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:36.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.067+0000 7f036f89d640 5 monclient: authenticate success, global_id 4844 2024-03-27T21:41:36.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.067+0000 7f036f89d640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:36.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.067+0000 7f036f89d640 4 set_mon_vals no callback set 2024-03-27T21:41:36.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.067+0000 7f036f89d640 10 monclient: shutdown 2024-03-27T21:41:36.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.067+0000 7f036f89d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:36.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.067+0000 7f036f89d640 1 -- 172.21.15.60:0/1703998096 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f036813ec50 msgr2=0x7f0368141d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.067+0000 7f036f89d640 1 --2- 172.21.15.60:0/1703998096 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f036813ec50 0x7f0368141d40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0360009e50 tx=0x7f036002bf00 comp rx=0 tx=0).stop 2024-03-27T21:41:36.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.067+0000 7f036f89d640 1 -- 172.21.15.60:0/1703998096 shutdown_connections 2024-03-27T21:41:36.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.068+0000 7f036f89d640 1 --2- 172.21.15.60:0/1703998096 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f036813ec50 0x7f0368141d40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.068+0000 7f036f89d640 1 --2- 172.21.15.60:0/1703998096 >> v2:172.21.15.191:3300/0 conn(0x7f0368071a50 0x7f03680a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.068+0000 7f036f89d640 1 -- 172.21.15.60:0/1703998096 >> 172.21.15.60:0/1703998096 conn(0x7f03680b7300 msgr2=0x7f03680b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:36.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.068+0000 7f036f89d640 1 -- 172.21.15.60:0/1703998096 shutdown_connections 2024-03-27T21:41:36.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.068+0000 7f036f89d640 1 -- 172.21.15.60:0/1703998096 wait complete. 2024-03-27T21:41:36.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.069+0000 7f036f89d640 10 monclient: build_initial_monmap 2024-03-27T21:41:36.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.069+0000 7f036f89d640 10 monclient: monmap: 2024-03-27T21:41:36.070 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:36.070 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:36.070 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:36.070 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:36.070 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:36.070 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:36.070 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:36.070 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:36.071 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:36.071 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:36.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.069+0000 7f036f89d640 1 librados: starting msgr at 2024-03-27T21:41:36.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.069+0000 7f036f89d640 1 librados: starting objecter 2024-03-27T21:41:36.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.070+0000 7f036f89d640 1 Processor -- start 2024-03-27T21:41:36.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.070+0000 7f036f89d640 1 -- start start 2024-03-27T21:41:36.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.070+0000 7f036f89d640 1 librados: setting wanted keys 2024-03-27T21:41:36.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.070+0000 7f036f89d640 1 librados: calling monclient init 2024-03-27T21:41:36.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.070+0000 7f036f89d640 10 monclient: init 2024-03-27T21:41:36.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036f89d640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:36.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036f89d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:41:36.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036f89d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0368071a50 0x7f036813a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036f89d640 10 monclient(hunting): picked mon.noname-b con 0x7f0368071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:36.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036f89d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f036813e810 0x7f0368137430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036f89d640 10 monclient(hunting): picked mon.noname-c con 0x7f036813e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:36.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036ce38640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f036813e810 0x7f0368137430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:36.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036d639640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0368071a50 0x7f036813a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:36.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036ce38640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f036813e810 0x7f0368137430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:53306/0 (socket says 172.21.15.60:53306) 2024-03-27T21:41:36.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036ce38640 1 -- 172.21.15.60:0/2273252980 learned_addr learned my addr 172.21.15.60:0/2273252980 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:36.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036f89d640 1 --2- 172.21.15.60:0/2273252980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f036813ec50 0x7f03680b0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036de3a640 1 --2- 172.21.15.60:0/2273252980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f036813ec50 0x7f03680b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:36.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036f89d640 10 monclient(hunting): picked mon.noname-a con 0x7f036813ec50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:36.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036f89d640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:36.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036f89d640 1 -- 172.21.15.60:0/2273252980 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036813abc0 con 0x7f036813ec50 2024-03-27T21:41:36.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036f89d640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:36.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036f89d640 1 -- 172.21.15.60:0/2273252980 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03680ab7b0 con 0x7f0368071a50 2024-03-27T21:41:36.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036f89d640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:36.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036f89d640 1 -- 172.21.15.60:0/2273252980 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03680aba90 con 0x7f036813e810 2024-03-27T21:41:36.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036f89d640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:36.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036f89d640 10 monclient(hunting): authenticate will time out at 937.293873s 2024-03-27T21:41:36.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036ce38640 10 monclient(hunting): get_auth_request con 0x7f036813e810 auth_method 0 2024-03-27T21:41:36.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036ce38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:36.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036ce38640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:36.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.071+0000 7f036ce38640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:36.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036d639640 10 monclient(hunting): get_auth_request con 0x7f0368071a50 auth_method 0 2024-03-27T21:41:36.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036d639640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:36.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036d639640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:36.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036d639640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:36.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036de3a640 10 monclient(hunting): get_auth_request con 0x7f036813ec50 auth_method 0 2024-03-27T21:41:36.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036de3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:36.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036de3a640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:36.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036de3a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:36.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036d639640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:36.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036d639640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:36.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036d639640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:36.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036de3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:36.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036de3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:36.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036de3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:36.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036ce38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:36.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036ce38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:36.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036ce38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:36.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036d639640 10 monclient(hunting): handle_auth_done global_id 5029 payload 824 2024-03-27T21:41:36.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036d639640 1 -- 172.21.15.60:0/2273252980 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f036813e810 msgr2=0x7f0368137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036d639640 1 --2- 172.21.15.60:0/2273252980 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f036813e810 0x7f0368137430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036d639640 1 -- 172.21.15.60:0/2273252980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f036813ec50 msgr2=0x7f03680b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036d639640 1 --2- 172.21.15.60:0/2273252980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f036813ec50 0x7f03680b0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036d639640 10 monclient: _finish_hunting 0 2024-03-27T21:41:36.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036d639640 1 monclient: found mon.noname-b 2024-03-27T21:41:36.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036d639640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:36.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036d639640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:41:36.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036d639640 1 -- 172.21.15.60:0/2273252980 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0360009b30 con 0x7f0368071a50 2024-03-27T21:41:36.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036ce38640 1 --2- 172.21.15.60:0/2273252980 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f036813e810 0x7f0368137430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:36.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036de3a640 1 --2- 172.21.15.60:0/2273252980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f036813ec50 0x7f03680b0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:36.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f036d639640 1 --2- 172.21.15.60:0/2273252980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0368071a50 0x7f036813a010 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0364009fd0 tx=0x7f036400b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:36.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f035dffb640 10 client.?.objecter ms_handle_connect 0x7f0368071a50 2024-03-27T21:41:36.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f035dffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:36.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f035dffb640 1 -- 172.21.15.60:0/2273252980 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0364011830 con 0x7f0368071a50 2024-03-27T21:41:36.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f035dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:36.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f035dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:36.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f035dffb640 10 monclient: dump: 2024-03-27T21:41:36.076 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:36.076 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:36.076 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.076 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.076 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:36.076 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:36.076 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:36.076 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:36.077 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:36.077 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:36.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f035dffb640 10 monclient: _finish_auth 0 2024-03-27T21:41:36.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f035dffb640 10 monclient: _check_auth_tickets 2024-03-27T21:41:36.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f035dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:36.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f035dffb640 1 -- 172.21.15.60:0/2273252980 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0364011990 con 0x7f0368071a50 2024-03-27T21:41:36.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f035dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:36.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.072+0000 7f035f7fe640 4 set_mon_vals no callback set 2024-03-27T21:41:36.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.073+0000 7f035dffb640 1 -- 172.21.15.60:0/2273252980 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0364011d40 con 0x7f0368071a50 2024-03-27T21:41:36.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.073+0000 7f035dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:36.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.073+0000 7f035dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:36.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.073+0000 7f035dffb640 10 monclient: dump: 2024-03-27T21:41:36.077 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:36.078 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:36.078 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.078 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.078 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:36.078 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:36.078 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:36.078 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:36.078 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:36.078 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:36.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.073+0000 7f036f89d640 5 monclient: authenticate success, global_id 5029 2024-03-27T21:41:36.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.073+0000 7f036f89d640 10 monclient: _renew_subs 2024-03-27T21:41:36.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.073+0000 7f036f89d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:36.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.073+0000 7f036f89d640 1 -- 172.21.15.60:0/2273252980 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03681391d0 con 0x7f0368071a50 2024-03-27T21:41:36.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.073+0000 7f036f89d640 10 client.5029.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:36.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.073+0000 7f036f89d640 10 monclient: _renew_subs 2024-03-27T21:41:36.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.073+0000 7f036f89d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:36.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.073+0000 7f036f89d640 1 -- 172.21.15.60:0/2273252980 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f036813a7f0 con 0x7f0368071a50 2024-03-27T21:41:36.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.073+0000 7f036f89d640 1 librados: init done 2024-03-27T21:41:36.079 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:36.074+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:36.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.075+0000 7f036f89d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:36.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.075+0000 7f036f89d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:36.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.075+0000 7f036f89d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:36.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.075+0000 7f036f89d640 1 -- 172.21.15.60:0/2273252980 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f036814bda0 con 0x7f0368071a50 2024-03-27T21:41:36.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.075+0000 7f035dffb640 1 -- 172.21.15.60:0/2273252980 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f036401a400 con 0x7f0368071a50 2024-03-27T21:41:36.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.076+0000 7f035dffb640 1 --2- 172.21.15.60:0/2273252980 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f034c06c5e0 0x7f034c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.076+0000 7f035dffb640 1 -- 172.21.15.60:0/2273252980 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f0364089130 con 0x7f0368071a50 2024-03-27T21:41:36.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.076+0000 7f035dffb640 10 client.5029.objecter ms_dispatch 0x7f0368000b90 osd_map(69..69 src has 1..69) v4 2024-03-27T21:41:36.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.076+0000 7f035dffb640 3 client.5029.objecter handle_osd_map got epochs [69,69] > 0 2024-03-27T21:41:36.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.076+0000 7f035dffb640 3 client.5029.objecter handle_osd_map decoding full epoch 69 2024-03-27T21:41:36.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.076+0000 7f036ce38640 1 --2- 172.21.15.60:0/2273252980 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f034c06c5e0 0x7f034c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:36.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.076+0000 7f036ce38640 10 monclient: get_auth_request con 0x7f034c06c5e0 auth_method 0 2024-03-27T21:41:36.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.076+0000 7f035dffb640 20 client.5029.objecter dump_active .. 0 homeless 2024-03-27T21:41:36.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.076+0000 7f035dffb640 10 client.5029.objecter ms_handle_connect 0x7f034c06c5e0 2024-03-27T21:41:36.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.076+0000 7f036ce38640 1 --2- 172.21.15.60:0/2273252980 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f034c06c5e0 0x7f034c06eaa0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f03681366c0 tx=0x7f0358009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:36.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.078+0000 7f65731b5640 10 client.5026.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:36.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.078+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign -1 12 2024-03-27T21:41:36.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.078+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c02be40 osd=5 3 2024-03-27T21:41:36.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.078+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c02be40 osd=5 3 2024-03-27T21:41:36.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.078+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c02be40 osd=5 3 2024-03-27T21:41:36.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.078+0000 7f65731b5640 15 client.5026.objecter _session_command_op_remove -1 12 2024-03-27T21:41:36.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.078+0000 7f65731b5640 20 client.5026.objecter get_session s=0x7f656c02be40 osd=5 3 2024-03-27T21:41:36.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.078+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign 5 12 2024-03-27T21:41:36.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.078+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c02be40 osd=5 4 2024-03-27T21:41:36.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.078+0000 7f65731b5640 10 client.5026.objecter _send_command 12 2024-03-27T21:41:36.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.078+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f656c02c180 con 0x7f656c1eaa80 2024-03-27T21:41:36.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.080+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f656c02c180 con 0x7f656c1eaa80 2024-03-27T21:41:36.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.080+0000 7f65697fa640 10 client.5026.objecter ms_dispatch 0x7f656c000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:36.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.080+0000 7f65697fa640 10 client.5026.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:36.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.080+0000 7f65697fa640 20 client.5026.objecter put_session s=0x7f656c02be40 osd=5 4 2024-03-27T21:41:36.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.080+0000 7f65697fa640 15 client.5026.objecter _session_command_op_remove 5 12 2024-03-27T21:41:36.084 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:41:36.085 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:41:36.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.080+0000 7f65731b5640 10 client.5026.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:36.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.080+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign -1 13 2024-03-27T21:41:36.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.080+0000 7f65731b5640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f656c1efde0 0x7f656c1f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.080+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c02c180 osd=6 3 2024-03-27T21:41:36.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.080+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c02c180 osd=6 3 2024-03-27T21:41:36.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.080+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c02c180 osd=6 3 2024-03-27T21:41:36.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.080+0000 7f65731b5640 15 client.5026.objecter _session_command_op_remove -1 13 2024-03-27T21:41:36.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.080+0000 7f65731b5640 20 client.5026.objecter get_session s=0x7f656c02c180 osd=6 3 2024-03-27T21:41:36.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.080+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign 6 13 2024-03-27T21:41:36.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.080+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c02c180 osd=6 4 2024-03-27T21:41:36.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.080+0000 7f65731b5640 10 client.5026.objecter _send_command 13 2024-03-27T21:41:36.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.080+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f656c02bfb0 con 0x7f656c1efde0 2024-03-27T21:41:36.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.081+0000 7f6571752640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f656c1efde0 0x7f656c1f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:36.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.081+0000 7f6571752640 10 monclient: get_auth_request con 0x7f656c1efde0 auth_method 0 2024-03-27T21:41:36.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.081+0000 7f65697fa640 10 client.5026.objecter ms_handle_connect 0x7f656c1efde0 2024-03-27T21:41:36.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.081+0000 7f6571752640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f656c1efde0 0x7f656c1f21f0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:36.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.081+0000 7f035dffb640 1 -- 172.21.15.60:0/2273252980 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0364054580 con 0x7f0368071a50 2024-03-27T21:41:36.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.081+0000 7f035dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:36.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.081+0000 7f035dffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:36.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.082+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f656c02bfb0 con 0x7f656c1efde0 2024-03-27T21:41:36.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.082+0000 7f65697fa640 10 client.5026.objecter ms_dispatch 0x7f656c000b90 command_reply(tid 13: 0 ) v1 2024-03-27T21:41:36.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.082+0000 7f65697fa640 10 client.5026.objecter _finish_command 13 = system:0 2024-03-27T21:41:36.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.082+0000 7f65697fa640 20 client.5026.objecter put_session s=0x7f656c02c180 osd=6 4 2024-03-27T21:41:36.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.082+0000 7f65697fa640 15 client.5026.objecter _session_command_op_remove 6 13 2024-03-27T21:41:36.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.110+0000 7f65731b5640 10 client.5026.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:36.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.110+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign -1 14 2024-03-27T21:41:36.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.110+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c02c180 osd=6 3 2024-03-27T21:41:36.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.110+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c02c180 osd=6 3 2024-03-27T21:41:36.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.110+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c02c180 osd=6 3 2024-03-27T21:41:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.110+0000 7f65731b5640 15 client.5026.objecter _session_command_op_remove -1 14 2024-03-27T21:41:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.110+0000 7f65731b5640 20 client.5026.objecter get_session s=0x7f656c02c180 osd=6 3 2024-03-27T21:41:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.110+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign 6 14 2024-03-27T21:41:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.110+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c02c180 osd=6 4 2024-03-27T21:41:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.110+0000 7f65731b5640 10 client.5026.objecter _send_command 14 2024-03-27T21:41:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.110+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f656c02bfb0 con 0x7f656c1efde0 2024-03-27T21:41:36.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.113+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f656c02bfb0 con 0x7f656c1efde0 2024-03-27T21:41:36.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.113+0000 7f65697fa640 10 client.5026.objecter ms_dispatch 0x7f656c000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:36.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.113+0000 7f65697fa640 10 client.5026.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:36.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.113+0000 7f65697fa640 20 client.5026.objecter put_session s=0x7f656c02c180 osd=6 4 2024-03-27T21:41:36.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.113+0000 7f65697fa640 15 client.5026.objecter _session_command_op_remove 6 14 2024-03-27T21:41:36.100 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:41:36.102 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:41:36.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.113+0000 7f65731b5640 10 client.5026.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:36.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.113+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign -1 15 2024-03-27T21:41:36.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.113+0000 7f65731b5640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f656c1f4520 0x7f656c1f49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.113+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c02bfb0 osd=7 3 2024-03-27T21:41:36.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.113+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c02bfb0 osd=7 3 2024-03-27T21:41:36.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.113+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c02bfb0 osd=7 3 2024-03-27T21:41:36.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.113+0000 7f65731b5640 15 client.5026.objecter _session_command_op_remove -1 15 2024-03-27T21:41:36.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.113+0000 7f65731b5640 20 client.5026.objecter get_session s=0x7f656c02bfb0 osd=7 3 2024-03-27T21:41:36.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.113+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign 7 15 2024-03-27T21:41:36.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.113+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c02bfb0 osd=7 4 2024-03-27T21:41:36.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.113+0000 7f65731b5640 10 client.5026.objecter _send_command 15 2024-03-27T21:41:36.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.113+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f656c1f8370 con 0x7f656c1f4520 2024-03-27T21:41:36.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.113+0000 7f6570f51640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f656c1f4520 0x7f656c1f49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:36.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.113+0000 7f6570f51640 10 monclient: get_auth_request con 0x7f656c1f4520 auth_method 0 2024-03-27T21:41:36.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.114+0000 7f6570f51640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f656c1f4520 0x7f656c1f49a0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:36.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.114+0000 7f65697fa640 10 client.5026.objecter ms_handle_connect 0x7f656c1f4520 2024-03-27T21:41:36.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.114+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f656c1f8370 con 0x7f656c1f4520 2024-03-27T21:41:36.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.114+0000 7f65697fa640 10 client.5026.objecter ms_dispatch 0x7f656c000b90 command_reply(tid 15: 0 ) v1 2024-03-27T21:41:36.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.114+0000 7f65697fa640 10 client.5026.objecter _finish_command 15 = system:0 2024-03-27T21:41:36.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.114+0000 7f65697fa640 20 client.5026.objecter put_session s=0x7f656c02bfb0 osd=7 4 2024-03-27T21:41:36.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.114+0000 7f65697fa640 15 client.5026.objecter _session_command_op_remove 7 15 2024-03-27T21:41:36.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.143+0000 7f65731b5640 10 client.5026.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:36.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.143+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign -1 16 2024-03-27T21:41:36.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.143+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c02bfb0 osd=7 3 2024-03-27T21:41:36.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.143+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c02bfb0 osd=7 3 2024-03-27T21:41:36.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.143+0000 7f65731b5640 20 client.5026.objecter _get_session s=0x7f656c02bfb0 osd=7 3 2024-03-27T21:41:36.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.143+0000 7f65731b5640 15 client.5026.objecter _session_command_op_remove -1 16 2024-03-27T21:41:36.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.143+0000 7f65731b5640 20 client.5026.objecter get_session s=0x7f656c02bfb0 osd=7 3 2024-03-27T21:41:36.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.143+0000 7f65731b5640 15 client.5026.objecter _session_command_op_assign 7 16 2024-03-27T21:41:36.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.143+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c02bfb0 osd=7 4 2024-03-27T21:41:36.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.143+0000 7f65731b5640 10 client.5026.objecter _send_command 16 2024-03-27T21:41:36.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.143+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f656c1f2840 con 0x7f656c1f4520 2024-03-27T21:41:36.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.145+0000 7f65697fa640 1 -- 172.21.15.60:0/3437457775 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f656c1f2840 con 0x7f656c1f4520 2024-03-27T21:41:36.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.145+0000 7f65697fa640 10 client.5026.objecter ms_dispatch 0x7f656c000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:36.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.145+0000 7f65697fa640 10 client.5026.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:36.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.145+0000 7f65697fa640 20 client.5026.objecter put_session s=0x7f656c02bfb0 osd=7 4 2024-03-27T21:41:36.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.145+0000 7f65697fa640 15 client.5026.objecter _session_command_op_remove 7 16 2024-03-27T21:41:36.133 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:41:36.138 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:41:36.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.145+0000 7f65731b5640 10 librados: watch_flush enter 2024-03-27T21:41:36.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.145+0000 7f65731b5640 10 librados: watch_flush exit 2024-03-27T21:41:36.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.145+0000 7f65731b5640 10 client.5026.objecter close_session for osd.0 2024-03-27T21:41:36.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.145+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f656c1d9930 msgr2=0x7f656c1d9d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.145+0000 7f65731b5640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f656c1d9930 0x7f656c1d9d10 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.145+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c13f0c0 osd=0 1 2024-03-27T21:41:36.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.145+0000 7f65731b5640 10 client.5026.objecter close_session for osd.1 2024-03-27T21:41:36.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.145+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f656c133110 msgr2=0x7f656c1334f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.145+0000 7f65731b5640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f656c133110 0x7f656c1334f0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.145+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c067900 osd=1 1 2024-03-27T21:41:36.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 10 client.5026.objecter close_session for osd.2 2024-03-27T21:41:36.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f656c13fbf0 msgr2=0x7f656c13ffd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f656c13fbf0 0x7f656c13ffd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f6571752640 1 -- 172.21.15.60:0/3437457775 reap_dead start 2024-03-27T21:41:36.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c0a8560 osd=2 1 2024-03-27T21:41:36.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 10 client.5026.objecter close_session for osd.3 2024-03-27T21:41:36.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f656c066250 msgr2=0x7f656c066670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f656c066250 0x7f656c066670 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c13fa20 osd=3 1 2024-03-27T21:41:36.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 10 client.5026.objecter close_session for osd.4 2024-03-27T21:41:36.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f656c065e40 msgr2=0x7f656c1ea030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f656c065e40 0x7f656c1ea030 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c065cd0 osd=4 1 2024-03-27T21:41:36.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 10 client.5026.objecter close_session for osd.5 2024-03-27T21:41:36.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f656c1eaa80 msgr2=0x7f656c1eaee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f656c1eaa80 0x7f656c1eaee0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c02be40 osd=5 1 2024-03-27T21:41:36.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 10 client.5026.objecter close_session for osd.6 2024-03-27T21:41:36.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f656c1efde0 msgr2=0x7f656c1f21f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f656c1efde0 0x7f656c1f21f0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c02c180 osd=6 1 2024-03-27T21:41:36.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 10 client.5026.objecter close_session for osd.7 2024-03-27T21:41:36.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f656c1f4520 msgr2=0x7f656c1f49a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f656c1f4520 0x7f656c1f49a0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f6571752640 1 -- 172.21.15.60:0/3437457775 reap_dead start 2024-03-27T21:41:36.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 20 client.5026.objecter put_session s=0x7f656c02bfb0 osd=7 1 2024-03-27T21:41:36.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 20 client.5026.objecter shutdown clearing up homeless session... 2024-03-27T21:41:36.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 10 client.5026.objecter successfully canceled tick 2024-03-27T21:41:36.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f655006c590 msgr2=0x7f655006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f655006c590 0x7f655006ea50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6560009b90 tx=0x7f6560009190 comp rx=0 tx=0).stop 2024-03-27T21:41:36.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 10 monclient: shutdown 2024-03-27T21:41:36.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:36.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.146+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f656c13ec50 msgr2=0x7f656c1d6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.147+0000 7f65731b5640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f656c13ec50 0x7f656c1d6d50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6564032000 tx=0x7f6564004a40 comp rx=0 tx=0).stop 2024-03-27T21:41:36.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.147+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 shutdown_connections 2024-03-27T21:41:36.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.147+0000 7f65731b5640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f655006c590 0x7f655006ea50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.147+0000 7f65731b5640 1 --2- 172.21.15.60:0/3437457775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f656c13ec50 0x7f656c1d6d50 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.147+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 >> 172.21.15.60:0/3437457775 conn(0x7f656c0b7300 msgr2=0x7f656c134040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:36.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.147+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 shutdown_connections 2024-03-27T21:41:36.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.147+0000 7f65731b5640 1 -- 172.21.15.60:0/3437457775 wait complete. 2024-03-27T21:41:36.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.147+0000 7f65731b5640 1 librados: shutdown 2024-03-27T21:41:36.147 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:36.148 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:36.156 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:41:36.191 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:36.204+0000 7fb850f34640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 56426) UID: 0 2024-03-27T21:41:36.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.267+0000 7f036f89d640 1 -- 172.21.15.60:0/2273252980 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f036813f0c0 con 0x7f034c06c5e0 2024-03-27T21:41:36.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.269+0000 7f035dffb640 1 -- 172.21.15.60:0/2273252980 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+62431 (secure 0 0 0) 0x7f03580018d0 con 0x7f034c06c5e0 2024-03-27T21:41:36.257 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:41:36.258 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-03-27T21:41:36.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.271+0000 7f036f89d640 10 librados: watch_flush enter 2024-03-27T21:41:36.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.271+0000 7f036f89d640 10 librados: watch_flush exit 2024-03-27T21:41:36.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.271+0000 7f036f89d640 20 client.5029.objecter shutdown clearing up homeless session... 2024-03-27T21:41:36.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.271+0000 7f036f89d640 10 client.5029.objecter successfully canceled tick 2024-03-27T21:41:36.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.272+0000 7f036f89d640 1 -- 172.21.15.60:0/2273252980 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f034c06c5e0 msgr2=0x7f034c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.272+0000 7f036f89d640 1 --2- 172.21.15.60:0/2273252980 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f034c06c5e0 0x7f034c06eaa0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f03681366c0 tx=0x7f0358009040 comp rx=0 tx=0).stop 2024-03-27T21:41:36.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.272+0000 7f036f89d640 10 monclient: shutdown 2024-03-27T21:41:36.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.272+0000 7f036f89d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:36.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.272+0000 7f036f89d640 1 -- 172.21.15.60:0/2273252980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0368071a50 msgr2=0x7f036813a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.272+0000 7f036f89d640 1 --2- 172.21.15.60:0/2273252980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0368071a50 0x7f036813a010 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0364009fd0 tx=0x7f036400b870 comp rx=0 tx=0).stop 2024-03-27T21:41:36.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.272+0000 7f036f89d640 1 -- 172.21.15.60:0/2273252980 shutdown_connections 2024-03-27T21:41:36.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.272+0000 7f036f89d640 1 --2- 172.21.15.60:0/2273252980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f036813ec50 0x7f03680b0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.272+0000 7f036f89d640 1 --2- 172.21.15.60:0/2273252980 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f036813e810 0x7f0368137430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.272+0000 7f036f89d640 1 --2- 172.21.15.60:0/2273252980 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f034c06c5e0 0x7f034c06eaa0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.272+0000 7f036f89d640 1 --2- 172.21.15.60:0/2273252980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0368071a50 0x7f036813a010 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.272+0000 7f036f89d640 1 -- 172.21.15.60:0/2273252980 >> 172.21.15.60:0/2273252980 conn(0x7f03680b7300 msgr2=0x7f036813b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:36.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.272+0000 7f036f89d640 1 -- 172.21.15.60:0/2273252980 shutdown_connections 2024-03-27T21:41:36.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.272+0000 7f036f89d640 1 -- 172.21.15.60:0/2273252980 wait complete. 2024-03-27T21:41:36.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.272+0000 7f036f89d640 1 librados: shutdown 2024-03-27T21:41:36.262 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:36.262 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:36.262 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:36.275+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:36.271 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:41:36.272 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2024-03-27T21:41:35.928201+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":200084,"kb_used_data":5164,"kb_used_omap":15,"kb_used_meta":194800,"kb_avail":660402796,"statfs":{"total":676457349120,"available":676252463104,"internally_reserved":0,"allocated":5287936,"data_stored":3225366,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":15606,"internal_metadata":199475978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.857068"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":52,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-03-27T21:41:30.076678+0000","last_change":"2024-03-27T21:39:46.470433+0000","last_active":"2024-03-27T21:41:30.076678+0000","last_peered":"2024-03-27T21:41:30.076678+0000","last_clean":"2024-03-27T21:41:30.076678+0000","last_became_active":"2024-03-27T21:39:46.470153+0000","last_became_peered":"2024-03-27T21:39:46.470153+0000","last_unstale":"2024-03-27T21:41:30.076678+0000","last_undegraded":"2024-03-27T21:41:30.076678+0000","last_fullsized":"2024-03-27T21:41:30.076678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T21:39:45.811197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-03-27T21:41:30.076678+0000","last_change":"2024-03-27T21:41:14.289685+0000","last_active":"2024-03-27T21:41:30.076678+0000","last_peered":"2024-03-27T21:41:30.076678+0000","last_clean":"2024-03-27T21:41:30.076678+0000","last_became_active":"2024-03-27T21:41:14.289398+0000","last_became_peered":"2024-03-27T21:41:14.289398+0000","last_unstale":"2024-03-27T21:41:30.076678+0000","last_undegraded":"2024-03-27T21:41:30.076678+0000","last_fullsized":"2024-03-27T21:41:30.076678+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T23:03:57.316429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":53,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-03-27T21:41:31.174378+0000","last_change":"2024-03-27T21:39:46.679788+0000","last_active":"2024-03-27T21:41:31.174378+0000","last_peered":"2024-03-27T21:41:31.174378+0000","last_clean":"2024-03-27T21:41:31.174378+0000","last_became_active":"2024-03-27T21:39:46.679561+0000","last_became_peered":"2024-03-27T21:39:46.679561+0000","last_unstale":"2024-03-27T21:41:31.174378+0000","last_undegraded":"2024-03-27T21:41:31.174378+0000","last_fullsized":"2024-03-27T21:41:31.174378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T23:53:50.373923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-03-27T21:41:31.176020+0000","last_change":"2024-03-27T21:41:13.998050+0000","last_active":"2024-03-27T21:41:31.176020+0000","last_peered":"2024-03-27T21:41:31.176020+0000","last_clean":"2024-03-27T21:41:31.176020+0000","last_became_active":"2024-03-27T21:41:13.997758+0000","last_became_peered":"2024-03-27T21:41:13.997758+0000","last_unstale":"2024-03-27T21:41:31.176020+0000","last_undegraded":"2024-03-27T21:41:31.176020+0000","last_fullsized":"2024-03-27T21:41:31.176020+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:14:50.994666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":61,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-03-27T21:41:30.076663+0000","last_change":"2024-03-27T21:41:20.021508+0000","last_active":"2024-03-27T21:41:30.076663+0000","last_peered":"2024-03-27T21:41:30.076663+0000","last_clean":"2024-03-27T21:41:30.076663+0000","last_became_active":"2024-03-27T21:41:20.021308+0000","last_became_peered":"2024-03-27T21:41:20.021308+0000","last_unstale":"2024-03-27T21:41:30.076663+0000","last_undegraded":"2024-03-27T21:41:30.076663+0000","last_fullsized":"2024-03-27T21:41:30.076663+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T21:56:40.416175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":33,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-03-27T21:41:30.080763+0000","last_change":"2024-03-27T21:41:30.080153+0000","last_active":"2024-03-27T21:41:30.080763+0000","last_peered":"2024-03-27T21:41:30.080763+0000","last_clean":"2024-03-27T21:41:30.080763+0000","last_became_active":"2024-03-27T21:41:30.079829+0000","last_became_peered":"2024-03-27T21:41:30.079829+0000","last_unstale":"2024-03-27T21:41:30.080763+0000","last_undegraded":"2024-03-27T21:41:30.080763+0000","last_fullsized":"2024-03-27T21:41:30.080763+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:21:00.966562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":61,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-03-27T21:41:31.174390+0000","last_change":"2024-03-27T21:41:13.996910+0000","last_active":"2024-03-27T21:41:31.174390+0000","last_peered":"2024-03-27T21:41:31.174390+0000","last_clean":"2024-03-27T21:41:31.174390+0000","last_became_active":"2024-03-27T21:41:13.996648+0000","last_became_peered":"2024-03-27T21:41:13.996648+0000","last_unstale":"2024-03-27T21:41:31.174390+0000","last_undegraded":"2024-03-27T21:41:31.174390+0000","last_fullsized":"2024-03-27T21:41:31.174390+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:00:43.185821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":42,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-03-27T21:41:31.175066+0000","last_change":"2024-03-27T21:41:30.062933+0000","last_active":"2024-03-27T21:41:31.175066+0000","last_peered":"2024-03-27T21:41:31.175066+0000","last_clean":"2024-03-27T21:41:31.175066+0000","last_became_active":"2024-03-27T21:41:30.062614+0000","last_became_peered":"2024-03-27T21:41:30.062614+0000","last_unstale":"2024-03-27T21:41:31.175066+0000","last_undegraded":"2024-03-27T21:41:31.175066+0000","last_fullsized":"2024-03-27T21:41:31.175066+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:55:05.293213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":206,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-03-27T21:41:31.174359+0000","last_change":"2024-03-27T21:39:39.866806+0000","last_active":"2024-03-27T21:41:31.174359+0000","last_peered":"2024-03-27T21:41:31.174359+0000","last_clean":"2024-03-27T21:41:31.174359+0000","last_became_active":"2024-03-27T21:39:39.865948+0000","last_became_peered":"2024-03-27T21:39:39.865948+0000","last_unstale":"2024-03-27T21:41:31.174359+0000","last_undegraded":"2024-03-27T21:41:31.174359+0000","last_fullsized":"2024-03-27T21:41:31.174359+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_clean_scrub_stamp":"2024-03-27T21:39:38.411432+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:36:49.375869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":16,"seq":68719476764,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":757760,"data_stored":317430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.66900000000000004}]},{"osd":1,"last update":"Wed Mar 27 21:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.55000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":3,"last update":"Wed Mar 27 21:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.68400000000000005}]},{"osd":4,"last update":"Wed Mar 27 21:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.52000000000000002}]},{"osd":6,"last update":"Wed Mar 27 21:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.64300000000000002}]},{"osd":7,"last update":"Wed Mar 27 21:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.628}]}]},{"osd":3,"up_from":16,"seq":68719476763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":757760,"data_stored":317430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31848905},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":0.90600000000000003}]},{"osd":1,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.39300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":4,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.57199999999999995}]},{"osd":5,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.47999999999999998}]},{"osd":6,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.60299999999999998}]},{"osd":7,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.454,"5min":1.454,"15min":1.454},"last":0.85799999999999998}]}]},{"osd":2,"up_from":62,"seq":266287972354,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29400,"kb_used_data":716,"kb_used_omap":5,"kb_used_meta":28666,"kb_avail":94342440,"statfs":{"total":96636764160,"available":96606658560,"internally_reserved":0,"allocated":733184,"data_stored":306530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6051,"internal_metadata":29354077},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.877}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]}]},{"osd":7,"up_from":14,"seq":60129542170,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":835458,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:40:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.60499999999999998}]},{"osd":1,"last update":"Wed Mar 27 21:40:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.75700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":3,"last update":"Wed Mar 27 21:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.62},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.64600000000000002}]},{"osd":4,"last update":"Wed Mar 27 21:40:37 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.55900000000000005}]},{"osd":5,"last update":"Wed Mar 27 21:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.436},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.39700000000000002}]},{"osd":6,"last update":"Wed Mar 27 21:40:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.71699999999999997}]}]},{"osd":6,"up_from":14,"seq":60129542170,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":733184,"data_stored":306530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.495},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.44600000000000001}]},{"osd":1,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.43099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":3,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.57399999999999995}]},{"osd":4,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.38400000000000001}]},{"osd":5,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.47099999999999997}]},{"osd":7,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.505}]}]},{"osd":4,"up_from":14,"seq":60129542170,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":733184,"data_stored":306530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:40:38 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.34999999999999998}]},{"osd":1,"last update":"Wed Mar 27 21:40:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.65200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":3,"last update":"Wed Mar 27 21:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.52500000000000002}]},{"osd":5,"last update":"Wed Mar 27 21:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.443},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.45500000000000002}]},{"osd":6,"last update":"Wed Mar 27 21:40:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.46800000000000003}]},{"osd":7,"last update":"Wed Mar 27 21:40:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.50800000000000001}]}]},{"osd":1,"up_from":0,"seq":55834574874,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":14,"seq":60129542170,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":835458,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 27 21:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.28499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":3,"last update":"Wed Mar 27 21:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.36199999999999999}]},{"osd":4,"last update":"Wed Mar 27 21:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.41999999999999998}]},{"osd":5,"last update":"Wed Mar 27 21:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.503},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.51600000000000001}]},{"osd":6,"last update":"Wed Mar 27 21:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.433},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.442}]},{"osd":7,"last update":"Wed Mar 27 21:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.46700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T21:41:36.274 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T21:41:36.363 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:36.376+0000 7fb850f34640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 56426) UID: 0 2024-03-27T21:41:36.378 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:36.378 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:36.386 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-27T21:41:36.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.431+0000 7fb5c6b49640 10 monclient: get_monmap_and_config 2024-03-27T21:41:36.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.431+0000 7fb5c6b49640 10 monclient: build_initial_monmap 2024-03-27T21:41:36.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.431+0000 7fb5c6b49640 10 monclient: monmap: 2024-03-27T21:41:36.419 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:36.419 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:36.419 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:36.433333+0000 2024-03-27T21:41:36.419 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:36.433333+0000 2024-03-27T21:41:36.419 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:36.419 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:36.419 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:36.419 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:36.419 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:36.419 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:36.419 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:36.419 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:36.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.432+0000 7fb5c6b49640 1 Processor -- start 2024-03-27T21:41:36.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.432+0000 7fb5c6b49640 1 -- start start 2024-03-27T21:41:36.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.432+0000 7fb5c6b49640 10 monclient: init 2024-03-27T21:41:36.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.432+0000 7fb5c6b49640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:36.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.432+0000 7fb5c6b49640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-03-27T21:41:36.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.432+0000 7fb5c6b49640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5c00b40b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:36.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.432+0000 7fb5c6b49640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5c013b020 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:36.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c6b49640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb5c013b480 0x7fb5c0141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c6b49640 10 monclient(hunting): picked mon.noname-c con 0x7fb5c013b480 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:36.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c6b49640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c013b8d0 con 0x7fb5c00b40b0 2024-03-27T21:41:36.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c6b49640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c00b3830 con 0x7fb5c013b020 2024-03-27T21:41:36.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c6b49640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:36.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c6b49640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c00b4490 con 0x7fb5c013b480 2024-03-27T21:41:36.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c6b49640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:36.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c6b49640 10 monclient(hunting): authenticate will time out at 937.655068s 2024-03-27T21:41:36.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c48e5640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb5c00b40b0 0x7fb5c00a8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52298/0 (socket says 172.21.15.60:52298) 2024-03-27T21:41:36.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c48e5640 1 -- 172.21.15.60:0/2268863700 learned_addr learned my addr 172.21.15.60:0/2268863700 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:36.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c50e6640 1 --2- 172.21.15.60:0/2268863700 >> v2:172.21.15.191:3300/0 conn(0x7fb5c013b480 0x7fb5c0141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:36.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c50e6640 10 monclient(hunting): get_auth_request con 0x7fb5c013b480 auth_method 0 2024-03-27T21:41:36.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c50e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:36.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c50e6640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:36.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c50e6640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:36.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c50e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:36.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c50e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:36.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c50e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:36.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5bf7fe640 1 -- 172.21.15.60:0/2268863700 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506447069 0 0) 0x7fb5c013b8d0 con 0x7fb5c00b40b0 2024-03-27T21:41:36.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5bf7fe640 10 monclient(hunting): _negotiate 2024-03-27T21:41:36.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:36.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:36.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5bf7fe640 10 monclient(hunting): my global_id is 5085 2024-03-27T21:41:36.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5bf7fe640 1 -- 172.21.15.60:0/2268863700 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5b0003520 con 0x7fb5c00b40b0 2024-03-27T21:41:36.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c50e6640 10 monclient(hunting): handle_auth_done global_id 5032 payload 274 2024-03-27T21:41:36.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c50e6640 1 -- 172.21.15.60:0/2268863700 >> v1:172.21.15.191:6789/0 conn(0x7fb5c013b020 legacy=0x7fb5c01416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c50e6640 1 -- 172.21.15.60:0/2268863700 >> v1:172.21.15.60:6789/0 conn(0x7fb5c00b40b0 legacy=0x7fb5c00a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c50e6640 10 monclient: _finish_hunting 0 2024-03-27T21:41:36.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c50e6640 1 monclient: found mon.noname-c 2024-03-27T21:41:36.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c50e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:36.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c50e6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:41:36.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.433+0000 7fb5c50e6640 1 -- 172.21.15.60:0/2268863700 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c00aa520 con 0x7fb5c013b480 2024-03-27T21:41:36.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.434+0000 7fb5c50e6640 1 --2- 172.21.15.60:0/2268863700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5c013b480 0x7fb5c0141e00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb5b8009d90 tx=0x7fb5b802bdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6354fc09338d72a server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:36.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.435+0000 7fb5bf7fe640 1 -- 172.21.15.60:0/2268863700 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b802cae0 con 0x7fb5c013b480 2024-03-27T21:41:36.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.435+0000 7fb5bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:36.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.435+0000 7fb5bf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:36.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.435+0000 7fb5bf7fe640 10 monclient: dump: 2024-03-27T21:41:36.430 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:36.430 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:36.430 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.430 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.430 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:36.430 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:36.430 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:36.430 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:36.430 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:36.431 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:36.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.435+0000 7fb5bf7fe640 10 monclient: _finish_auth 0 2024-03-27T21:41:36.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.435+0000 7fb5bf7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:41:36.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.435+0000 7fb5bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:36.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.435+0000 7fb5c6b49640 5 monclient: authenticate success, global_id 5032 2024-03-27T21:41:36.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.435+0000 7fb5c6b49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:36.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.435+0000 7fb5bf7fe640 1 -- 172.21.15.60:0/2268863700 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b802cc40 con 0x7fb5c013b480 2024-03-27T21:41:36.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.435+0000 7fb5bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:36.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.435+0000 7fb5bf7fe640 1 -- 172.21.15.60:0/2268863700 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8034440 con 0x7fb5c013b480 2024-03-27T21:41:36.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.435+0000 7fb5bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:36.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.435+0000 7fb5bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:36.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.435+0000 7fb5bf7fe640 10 monclient: dump: 2024-03-27T21:41:36.431 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:36.431 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:36.431 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.432 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.432 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:36.432 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:36.432 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:36.432 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:36.432 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:36.432 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:36.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.435+0000 7fb5c6b49640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:36.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.435+0000 7fb5c6b49640 4 set_mon_vals no callback set 2024-03-27T21:41:36.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.436+0000 7fb5c6b49640 10 monclient: shutdown 2024-03-27T21:41:36.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.436+0000 7fb5c6b49640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:36.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.436+0000 7fb5c6b49640 1 -- 172.21.15.60:0/2268863700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5c013b480 msgr2=0x7fb5c0141e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.436+0000 7fb5c6b49640 1 --2- 172.21.15.60:0/2268863700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5c013b480 0x7fb5c0141e00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb5b8009d90 tx=0x7fb5b802bdb0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.436+0000 7fb5c6b49640 1 -- 172.21.15.60:0/2268863700 shutdown_connections 2024-03-27T21:41:36.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.436+0000 7fb5c6b49640 1 --2- 172.21.15.60:0/2268863700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5c013b480 0x7fb5c0141e00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.436+0000 7fb5c6b49640 1 -- 172.21.15.60:0/2268863700 >> 172.21.15.60:0/2268863700 conn(0x7fb5c00608e0 msgr2=0x7fb5c0060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:36.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.437+0000 7fb5c6b49640 1 -- 172.21.15.60:0/2268863700 shutdown_connections 2024-03-27T21:41:36.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.437+0000 7fb5c6b49640 1 -- 172.21.15.60:0/2268863700 wait complete. 2024-03-27T21:41:36.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.438+0000 7fb5c6b49640 10 monclient: build_initial_monmap 2024-03-27T21:41:36.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.438+0000 7fb5c6b49640 10 monclient: monmap: 2024-03-27T21:41:36.434 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:36.434 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:36.434 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:36.434 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:36.434 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:36.434 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:36.435 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:36.435 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:36.435 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:36.435 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:36.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.438+0000 7fb5c6b49640 1 librados: starting msgr at 2024-03-27T21:41:36.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.438+0000 7fb5c6b49640 1 librados: starting objecter 2024-03-27T21:41:36.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.439+0000 7fb5c6b49640 1 Processor -- start 2024-03-27T21:41:36.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.439+0000 7fb5c6b49640 1 -- start start 2024-03-27T21:41:36.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.439+0000 7fb5c6b49640 1 librados: setting wanted keys 2024-03-27T21:41:36.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.439+0000 7fb5c6b49640 1 librados: calling monclient init 2024-03-27T21:41:36.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.439+0000 7fb5c6b49640 10 monclient: init 2024-03-27T21:41:36.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.440+0000 7fb5c6b49640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:36.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.440+0000 7fb5c6b49640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:41:36.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.440+0000 7fb5c6b49640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb5c00b40b0 0x7fb5c01d6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.441+0000 7fb5c6b49640 10 monclient(hunting): picked mon.noname-c con 0x7fb5c00b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:36.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.441+0000 7fb5c48e5640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb5c00b40b0 0x7fb5c01d6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:36.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.441+0000 7fb5c48e5640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb5c00b40b0 0x7fb5c01d6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:53324/0 (socket says 172.21.15.60:53324) 2024-03-27T21:41:36.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.441+0000 7fb5c48e5640 1 -- 172.21.15.60:0/1071727464 learned_addr learned my addr 172.21.15.60:0/1071727464 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:36.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.441+0000 7fb5c6b49640 1 --2- 172.21.15.60:0/1071727464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5c013b020 0x7fb5c01d6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.441+0000 7fb5c6b49640 10 monclient(hunting): picked mon.noname-b con 0x7fb5c013b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:36.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.441+0000 7fb5c6b49640 1 --2- 172.21.15.60:0/1071727464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5c013b480 0x7fb5c013a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.441+0000 7fb5c6b49640 10 monclient(hunting): picked mon.noname-a con 0x7fb5c013b480 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:36.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.441+0000 7fb5c6b49640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:36.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.441+0000 7fb5c50e6640 1 --2- 172.21.15.60:0/1071727464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5c013b480 0x7fb5c013a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:36.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.441+0000 7fb5bffff640 1 --2- 172.21.15.60:0/1071727464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5c013b020 0x7fb5c01d6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:36.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.441+0000 7fb5c6b49640 1 -- 172.21.15.60:0/1071727464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c01d6580 con 0x7fb5c013b480 2024-03-27T21:41:36.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.441+0000 7fb5c6b49640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:36.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.441+0000 7fb5c6b49640 1 -- 172.21.15.60:0/1071727464 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c0139620 con 0x7fb5c013b020 2024-03-27T21:41:36.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5c6b49640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:36.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5c6b49640 1 -- 172.21.15.60:0/1071727464 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c0137250 con 0x7fb5c00b40b0 2024-03-27T21:41:36.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5c6b49640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:36.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5c6b49640 10 monclient(hunting): authenticate will time out at 937.663937s 2024-03-27T21:41:36.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5c48e5640 10 monclient(hunting): get_auth_request con 0x7fb5c00b40b0 auth_method 0 2024-03-27T21:41:36.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5c48e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:36.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5c48e5640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:36.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5c48e5640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:36.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5c50e6640 10 monclient(hunting): get_auth_request con 0x7fb5c013b480 auth_method 0 2024-03-27T21:41:36.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5c50e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:36.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5c50e6640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:36.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5c50e6640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:36.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5bffff640 10 monclient(hunting): get_auth_request con 0x7fb5c013b020 auth_method 0 2024-03-27T21:41:36.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:36.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5bffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:36.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:36.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5c48e5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:36.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5c48e5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:36.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5c48e5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:36.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5c50e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:36.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5c50e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:36.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5c50e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:36.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:36.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:36.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:36.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5bffff640 10 monclient(hunting): handle_auth_done global_id 5038 payload 824 2024-03-27T21:41:36.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5bffff640 1 -- 172.21.15.60:0/1071727464 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb5c00b40b0 msgr2=0x7fb5c01d6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5bffff640 1 --2- 172.21.15.60:0/1071727464 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb5c00b40b0 0x7fb5c01d6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5bffff640 1 -- 172.21.15.60:0/1071727464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5c013b480 msgr2=0x7fb5c013a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5bffff640 1 --2- 172.21.15.60:0/1071727464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5c013b480 0x7fb5c013a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5bffff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:36.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5bffff640 1 monclient: found mon.noname-b 2024-03-27T21:41:36.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:36.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5bffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:41:36.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5bffff640 1 -- 172.21.15.60:0/1071727464 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b8009a70 con 0x7fb5c013b020 2024-03-27T21:41:36.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5c48e5640 1 --2- 172.21.15.60:0/1071727464 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb5c00b40b0 0x7fb5c01d6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:36.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5bffff640 1 --2- 172.21.15.60:0/1071727464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5c013b020 0x7fb5c01d6fe0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac00c540 tx=0x7fb5ac00ca00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:36.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5bd7fa640 10 client.?.objecter ms_handle_connect 0x7fb5c013b020 2024-03-27T21:41:36.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.442+0000 7fb5bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:36.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.443+0000 7fb5bd7fa640 1 -- 172.21.15.60:0/1071727464 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac005480 con 0x7fb5c013b020 2024-03-27T21:41:36.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.443+0000 7fb5bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:36.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.443+0000 7fb5bd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:36.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.443+0000 7fb5bd7fa640 10 monclient: dump: 2024-03-27T21:41:36.448 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:36.448 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:36.448 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.448 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.449 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:36.449 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:36.449 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:36.449 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:36.449 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:36.449 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:36.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.443+0000 7fb5bd7fa640 10 monclient: _finish_auth 0 2024-03-27T21:41:36.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.443+0000 7fb5bd7fa640 10 monclient: _check_auth_tickets 2024-03-27T21:41:36.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.443+0000 7fb5bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:36.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.443+0000 7fb5bd7fa640 1 -- 172.21.15.60:0/1071727464 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5ac0035f0 con 0x7fb5c013b020 2024-03-27T21:41:36.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.443+0000 7fb5bd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:36.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.443+0000 7fb5bd7fa640 1 -- 172.21.15.60:0/1071727464 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac003980 con 0x7fb5c013b020 2024-03-27T21:41:36.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.443+0000 7fb5bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:36.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.443+0000 7fb5bd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:36.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.443+0000 7fb5bd7fa640 10 monclient: dump: 2024-03-27T21:41:36.450 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:36.450 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:36.450 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.450 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.450 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:36.450 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:36.450 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:36.450 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:36.450 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:36.450 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:36.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.443+0000 7fb5beffd640 4 set_mon_vals no callback set 2024-03-27T21:41:36.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.443+0000 7fb5c6b49640 5 monclient: authenticate success, global_id 5038 2024-03-27T21:41:36.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.443+0000 7fb5c6b49640 10 monclient: _renew_subs 2024-03-27T21:41:36.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.443+0000 7fb5c6b49640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:36.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.443+0000 7fb5c6b49640 1 -- 172.21.15.60:0/1071727464 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c013a590 con 0x7fb5c013b020 2024-03-27T21:41:36.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.443+0000 7fb5c6b49640 10 client.5038.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:36.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.444+0000 7fb5c6b49640 10 monclient: _renew_subs 2024-03-27T21:41:36.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.444+0000 7fb5c6b49640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:36.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.444+0000 7fb5c6b49640 1 -- 172.21.15.60:0/1071727464 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c013aa50 con 0x7fb5c013b020 2024-03-27T21:41:36.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.444+0000 7fb5c6b49640 1 librados: init done 2024-03-27T21:41:36.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.445+0000 7fb5c6b49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:36.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.445+0000 7fb5c6b49640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:36.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.445+0000 7fb5c6b49640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:36.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.445+0000 7fb5c6b49640 1 -- 172.21.15.60:0/1071727464 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c013acd0 con 0x7fb5c013b020 2024-03-27T21:41:36.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.446+0000 7fb5bd7fa640 1 -- 172.21.15.60:0/1071727464 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb5ac003ae0 con 0x7fb5c013b020 2024-03-27T21:41:36.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.446+0000 7fb5bd7fa640 1 --2- 172.21.15.60:0/1071727464 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb5a406c590 0x7fb5a406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.446+0000 7fb5bd7fa640 1 -- 172.21.15.60:0/1071727464 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fb5ac088c20 con 0x7fb5c013b020 2024-03-27T21:41:36.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.446+0000 7fb5bd7fa640 10 client.5038.objecter ms_dispatch 0x7fb5c0000b90 osd_map(69..69 src has 1..69) v4 2024-03-27T21:41:36.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.446+0000 7fb5bd7fa640 3 client.5038.objecter handle_osd_map got epochs [69,69] > 0 2024-03-27T21:41:36.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.446+0000 7fb5bd7fa640 3 client.5038.objecter handle_osd_map decoding full epoch 69 2024-03-27T21:41:36.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.447+0000 7fb5bd7fa640 20 client.5038.objecter dump_active .. 0 homeless 2024-03-27T21:41:36.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.447+0000 7fb5c48e5640 1 --2- 172.21.15.60:0/1071727464 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb5a406c590 0x7fb5a406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:36.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.447+0000 7fb5c48e5640 10 monclient: get_auth_request con 0x7fb5a406c590 auth_method 0 2024-03-27T21:41:36.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.447+0000 7fb5c48e5640 1 --2- 172.21.15.60:0/1071727464 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb5a406c590 0x7fb5a406ea50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb5b40011e0 tx=0x7fb5b4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:36.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.447+0000 7fb5bd7fa640 10 client.5038.objecter ms_handle_connect 0x7fb5a406c590 2024-03-27T21:41:36.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.451+0000 7fb5bd7fa640 1 -- 172.21.15.60:0/1071727464 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb5ac0540b0 con 0x7fb5c013b020 2024-03-27T21:41:36.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.451+0000 7fb5bd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:36.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.451+0000 7fb5bd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:36.465 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:36.478+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:36.559 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:36.559 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:36.564 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:36.577+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:36.572 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-27T21:41:36.666 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:36.679+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:36.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.706+0000 7fb5c6b49640 1 -- 172.21.15.60:0/1071727464 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb5c0142340 con 0x7fb5a406c590 2024-03-27T21:41:36.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.710+0000 7fb5bd7fa640 1 -- 172.21.15.60:0/1071727464 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+62431 (secure 0 0 0) 0x7fb5b400fcb0 con 0x7fb5a406c590 2024-03-27T21:41:36.698 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:41:36.698 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-03-27T21:41:36.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.712+0000 7fb5c6b49640 10 librados: watch_flush enter 2024-03-27T21:41:36.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.712+0000 7fb5c6b49640 10 librados: watch_flush exit 2024-03-27T21:41:36.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.712+0000 7fb5c6b49640 20 client.5038.objecter shutdown clearing up homeless session... 2024-03-27T21:41:36.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.712+0000 7fb5c6b49640 10 client.5038.objecter successfully canceled tick 2024-03-27T21:41:36.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.712+0000 7fb5c6b49640 1 -- 172.21.15.60:0/1071727464 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb5a406c590 msgr2=0x7fb5a406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.712+0000 7fb5c6b49640 1 --2- 172.21.15.60:0/1071727464 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb5a406c590 0x7fb5a406ea50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb5b40011e0 tx=0x7fb5b4009040 comp rx=0 tx=0).stop 2024-03-27T21:41:36.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.712+0000 7fb5c6b49640 10 monclient: shutdown 2024-03-27T21:41:36.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.712+0000 7fb5c6b49640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:36.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.712+0000 7fb5c6b49640 1 -- 172.21.15.60:0/1071727464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5c013b020 msgr2=0x7fb5c01d6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.713+0000 7fb5c6b49640 1 --2- 172.21.15.60:0/1071727464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5c013b020 0x7fb5c01d6fe0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac00c540 tx=0x7fb5ac00ca00 comp rx=0 tx=0).stop 2024-03-27T21:41:36.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.713+0000 7fb5c6b49640 1 -- 172.21.15.60:0/1071727464 shutdown_connections 2024-03-27T21:41:36.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.713+0000 7fb5c6b49640 1 --2- 172.21.15.60:0/1071727464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5c013b480 0x7fb5c013a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.713+0000 7fb5c6b49640 1 --2- 172.21.15.60:0/1071727464 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb5a406c590 0x7fb5a406ea50 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.713+0000 7fb5c6b49640 1 --2- 172.21.15.60:0/1071727464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5c013b020 0x7fb5c01d6fe0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.713+0000 7fb5c6b49640 1 --2- 172.21.15.60:0/1071727464 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb5c00b40b0 0x7fb5c01d6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.713+0000 7fb5c6b49640 1 -- 172.21.15.60:0/1071727464 >> 172.21.15.60:0/1071727464 conn(0x7fb5c00608e0 msgr2=0x7fb5c00b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:36.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.713+0000 7fb5c6b49640 1 -- 172.21.15.60:0/1071727464 shutdown_connections 2024-03-27T21:41:36.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.713+0000 7fb5c6b49640 1 -- 172.21.15.60:0/1071727464 wait complete. 2024-03-27T21:41:36.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.713+0000 7fb5c6b49640 1 librados: shutdown 2024-03-27T21:41:36.711 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2024-03-27T21:41:35.928201+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":200084,"kb_used_data":5164,"kb_used_omap":15,"kb_used_meta":194800,"kb_avail":660402796,"statfs":{"total":676457349120,"available":676252463104,"internally_reserved":0,"allocated":5287936,"data_stored":3225366,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":15606,"internal_metadata":199475978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.857068"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":52,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-03-27T21:41:30.076678+0000","last_change":"2024-03-27T21:39:46.470433+0000","last_active":"2024-03-27T21:41:30.076678+0000","last_peered":"2024-03-27T21:41:30.076678+0000","last_clean":"2024-03-27T21:41:30.076678+0000","last_became_active":"2024-03-27T21:39:46.470153+0000","last_became_peered":"2024-03-27T21:39:46.470153+0000","last_unstale":"2024-03-27T21:41:30.076678+0000","last_undegraded":"2024-03-27T21:41:30.076678+0000","last_fullsized":"2024-03-27T21:41:30.076678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T21:39:45.811197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-03-27T21:41:30.076678+0000","last_change":"2024-03-27T21:41:14.289685+0000","last_active":"2024-03-27T21:41:30.076678+0000","last_peered":"2024-03-27T21:41:30.076678+0000","last_clean":"2024-03-27T21:41:30.076678+0000","last_became_active":"2024-03-27T21:41:14.289398+0000","last_became_peered":"2024-03-27T21:41:14.289398+0000","last_unstale":"2024-03-27T21:41:30.076678+0000","last_undegraded":"2024-03-27T21:41:30.076678+0000","last_fullsized":"2024-03-27T21:41:30.076678+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T23:03:57.316429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":53,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-03-27T21:41:31.174378+0000","last_change":"2024-03-27T21:39:46.679788+0000","last_active":"2024-03-27T21:41:31.174378+0000","last_peered":"2024-03-27T21:41:31.174378+0000","last_clean":"2024-03-27T21:41:31.174378+0000","last_became_active":"2024-03-27T21:39:46.679561+0000","last_became_peered":"2024-03-27T21:39:46.679561+0000","last_unstale":"2024-03-27T21:41:31.174378+0000","last_undegraded":"2024-03-27T21:41:31.174378+0000","last_fullsized":"2024-03-27T21:41:31.174378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T23:53:50.373923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-03-27T21:41:31.176020+0000","last_change":"2024-03-27T21:41:13.998050+0000","last_active":"2024-03-27T21:41:31.176020+0000","last_peered":"2024-03-27T21:41:31.176020+0000","last_clean":"2024-03-27T21:41:31.176020+0000","last_became_active":"2024-03-27T21:41:13.997758+0000","last_became_peered":"2024-03-27T21:41:13.997758+0000","last_unstale":"2024-03-27T21:41:31.176020+0000","last_undegraded":"2024-03-27T21:41:31.176020+0000","last_fullsized":"2024-03-27T21:41:31.176020+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:14:50.994666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":61,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-03-27T21:41:30.076663+0000","last_change":"2024-03-27T21:41:20.021508+0000","last_active":"2024-03-27T21:41:30.076663+0000","last_peered":"2024-03-27T21:41:30.076663+0000","last_clean":"2024-03-27T21:41:30.076663+0000","last_became_active":"2024-03-27T21:41:20.021308+0000","last_became_peered":"2024-03-27T21:41:20.021308+0000","last_unstale":"2024-03-27T21:41:30.076663+0000","last_undegraded":"2024-03-27T21:41:30.076663+0000","last_fullsized":"2024-03-27T21:41:30.076663+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T21:56:40.416175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":33,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-03-27T21:41:30.080763+0000","last_change":"2024-03-27T21:41:30.080153+0000","last_active":"2024-03-27T21:41:30.080763+0000","last_peered":"2024-03-27T21:41:30.080763+0000","last_clean":"2024-03-27T21:41:30.080763+0000","last_became_active":"2024-03-27T21:41:30.079829+0000","last_became_peered":"2024-03-27T21:41:30.079829+0000","last_unstale":"2024-03-27T21:41:30.080763+0000","last_undegraded":"2024-03-27T21:41:30.080763+0000","last_fullsized":"2024-03-27T21:41:30.080763+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:21:00.966562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":61,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-03-27T21:41:31.174390+0000","last_change":"2024-03-27T21:41:13.996910+0000","last_active":"2024-03-27T21:41:31.174390+0000","last_peered":"2024-03-27T21:41:31.174390+0000","last_clean":"2024-03-27T21:41:31.174390+0000","last_became_active":"2024-03-27T21:41:13.996648+0000","last_became_peered":"2024-03-27T21:41:13.996648+0000","last_unstale":"2024-03-27T21:41:31.174390+0000","last_undegraded":"2024-03-27T21:41:31.174390+0000","last_fullsized":"2024-03-27T21:41:31.174390+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:00:43.185821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":42,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-03-27T21:41:31.175066+0000","last_change":"2024-03-27T21:41:30.062933+0000","last_active":"2024-03-27T21:41:31.175066+0000","last_peered":"2024-03-27T21:41:31.175066+0000","last_clean":"2024-03-27T21:41:31.175066+0000","last_became_active":"2024-03-27T21:41:30.062614+0000","last_became_peered":"2024-03-27T21:41:30.062614+0000","last_unstale":"2024-03-27T21:41:31.175066+0000","last_undegraded":"2024-03-27T21:41:31.175066+0000","last_fullsized":"2024-03-27T21:41:31.175066+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:55:05.293213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":206,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-03-27T21:41:31.174359+0000","last_change":"2024-03-27T21:39:39.866806+0000","last_active":"2024-03-27T21:41:31.174359+0000","last_peered":"2024-03-27T21:41:31.174359+0000","last_clean":"2024-03-27T21:41:31.174359+0000","last_became_active":"2024-03-27T21:39:39.865948+0000","last_became_peered":"2024-03-27T21:39:39.865948+0000","last_unstale":"2024-03-27T21:41:31.174359+0000","last_undegraded":"2024-03-27T21:41:31.174359+0000","last_fullsized":"2024-03-27T21:41:31.174359+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_clean_scrub_stamp":"2024-03-27T21:39:38.411432+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:36:49.375869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":16,"seq":68719476764,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":757760,"data_stored":317430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.66900000000000004}]},{"osd":1,"last update":"Wed Mar 27 21:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.55000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":3,"last update":"Wed Mar 27 21:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.68400000000000005}]},{"osd":4,"last update":"Wed Mar 27 21:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.52000000000000002}]},{"osd":6,"last update":"Wed Mar 27 21:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.64300000000000002}]},{"osd":7,"last update":"Wed Mar 27 21:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.628}]}]},{"osd":3,"up_from":16,"seq":68719476763,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":757760,"data_stored":317430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31848905},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":0.90600000000000003}]},{"osd":1,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.39300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":4,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.57199999999999995}]},{"osd":5,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.47999999999999998}]},{"osd":6,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.60299999999999998}]},{"osd":7,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.454,"5min":1.454,"15min":1.454},"last":0.85799999999999998}]}]},{"osd":2,"up_from":62,"seq":266287972354,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29400,"kb_used_data":716,"kb_used_omap":5,"kb_used_meta":28666,"kb_avail":94342440,"statfs":{"total":96636764160,"available":96606658560,"internally_reserved":0,"allocated":733184,"data_stored":306530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6051,"internal_metadata":29354077},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.877}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]}]},{"osd":7,"up_from":14,"seq":60129542170,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":835458,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:40:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.60499999999999998}]},{"osd":1,"last update":"Wed Mar 27 21:40:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.75700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":3,"last update":"Wed Mar 27 21:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.62},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.64600000000000002}]},{"osd":4,"last update":"Wed Mar 27 21:40:37 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.55900000000000005}]},{"osd":5,"last update":"Wed Mar 27 21:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.436},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.39700000000000002}]},{"osd":6,"last update":"Wed Mar 27 21:40:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.71699999999999997}]}]},{"osd":6,"up_from":14,"seq":60129542170,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":733184,"data_stored":306530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.495},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.44600000000000001}]},{"osd":1,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.43099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":3,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.57399999999999995}]},{"osd":4,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.38400000000000001}]},{"osd":5,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.47099999999999997}]},{"osd":7,"last update":"Wed Mar 27 21:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.505}]}]},{"osd":4,"up_from":14,"seq":60129542170,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":733184,"data_stored":306530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:40:38 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.34999999999999998}]},{"osd":1,"last update":"Wed Mar 27 21:40:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.65200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":3,"last update":"Wed Mar 27 21:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.52500000000000002}]},{"osd":5,"last update":"Wed Mar 27 21:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.443},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.45500000000000002}]},{"osd":6,"last update":"Wed Mar 27 21:40:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.46800000000000003}]},{"osd":7,"last update":"Wed Mar 27 21:40:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.50800000000000001}]}]},{"osd":1,"up_from":0,"seq":55834574874,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":14,"seq":60129542170,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":835458,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 27 21:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.28499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":3,"last update":"Wed Mar 27 21:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.36199999999999999}]},{"osd":4,"last update":"Wed Mar 27 21:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.41999999999999998}]},{"osd":5,"last update":"Wed Mar 27 21:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.503},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.51600000000000001}]},{"osd":6,"last update":"Wed Mar 27 21:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.433},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.442}]},{"osd":7,"last update":"Wed Mar 27 21:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.46700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T21:41:36.713 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-27T21:41:36.713 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T21:41:36.713 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-27T21:41:36.713 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.000000 2024-03-27T21:41:36.713 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 0 2024-03-27T21:41:36.783 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:36.784 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:36.784 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:36.796+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:36.796 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-27T21:41:36.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.872+0000 7f83ea942640 10 monclient: get_monmap_and_config 2024-03-27T21:41:36.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.872+0000 7f83ea942640 10 monclient: build_initial_monmap 2024-03-27T21:41:36.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.872+0000 7f83ea942640 10 monclient: monmap: 2024-03-27T21:41:36.860 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:36.860 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:36.860 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:36.874541+0000 2024-03-27T21:41:36.860 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:36.874541+0000 2024-03-27T21:41:36.860 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:36.860 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:36.861 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:36.861 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:36.861 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:36.861 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:36.861 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:36.861 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:36.875 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:36.881+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:36.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.873+0000 7f83ea942640 1 Processor -- start 2024-03-27T21:41:36.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.873+0000 7f83ea942640 1 -- start start 2024-03-27T21:41:36.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.873+0000 7f83ea942640 10 monclient: init 2024-03-27T21:41:36.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.873+0000 7f83ea942640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:36.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.873+0000 7f83ea942640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-03-27T21:41:36.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.873+0000 7f83ea942640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f83e40b40b0 0x7f83e40a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83ea942640 10 monclient(hunting): picked mon.noname-b con 0x7f83e40b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:36.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83ea942640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f83e413b020 0x7f83e413b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83ea942640 10 monclient(hunting): picked mon.noname-c con 0x7f83e413b020 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:36.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83ea942640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f83e41416f0 0x7f83e4141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83ea942640 10 monclient(hunting): picked mon.noname-a con 0x7f83e41416f0 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:36.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83ea942640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:36.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83ea942640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e40b4490 con 0x7f83e41416f0 2024-03-27T21:41:36.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83ea942640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:36.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83ea942640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e40b3830 con 0x7f83e413b020 2024-03-27T21:41:36.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83ea942640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:36.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83ea942640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e40aa520 con 0x7f83e40b40b0 2024-03-27T21:41:36.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83ea942640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:36.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83ea942640 10 monclient(hunting): authenticate will time out at 938.096058s 2024-03-27T21:41:36.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e3fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f83e40b40b0 0x7f83e40a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:36.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e3fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f83e40b40b0 0x7f83e40a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:53326/0 (socket says 172.21.15.60:53326) 2024-03-27T21:41:36.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e3fff640 1 -- 172.21.15.60:0/2560491359 learned_addr learned my addr 172.21.15.60:0/2560491359 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:36.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e3fff640 10 monclient(hunting): get_auth_request con 0x7f83e40b40b0 auth_method 0 2024-03-27T21:41:36.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:36.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:36.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:36.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e8edf640 1 --2- 172.21.15.60:0/2560491359 >> v2:172.21.15.60:3300/0 conn(0x7f83e41416f0 0x7f83e4141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:36.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e8edf640 10 monclient(hunting): get_auth_request con 0x7f83e41416f0 auth_method 0 2024-03-27T21:41:36.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e8edf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:36.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e8edf640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:36.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e8edf640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:36.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:36.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:36.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:36.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e8edf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:36.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e8edf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:36.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e8edf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:36.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e3fff640 10 monclient(hunting): handle_auth_done global_id 4853 payload 274 2024-03-27T21:41:36.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e3fff640 1 -- 172.21.15.60:0/2560491359 >> v2:172.21.15.191:3300/0 conn(0x7f83e413b020 msgr2=0x7f83e413b480 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-27T21:41:36.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e3fff640 1 --2- 172.21.15.60:0/2560491359 >> v2:172.21.15.191:3300/0 conn(0x7f83e413b020 0x7f83e413b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e3fff640 1 -- 172.21.15.60:0/2560491359 >> v2:172.21.15.60:3300/0 conn(0x7f83e41416f0 msgr2=0x7f83e4141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:36.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e3fff640 1 --2- 172.21.15.60:0/2560491359 >> v2:172.21.15.60:3300/0 conn(0x7f83e41416f0 0x7f83e4141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e3fff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:36.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e3fff640 1 monclient: found mon.noname-b 2024-03-27T21:41:36.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:36.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:36.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e3fff640 1 -- 172.21.15.60:0/2560491359 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e40a9470 con 0x7f83e40b40b0 2024-03-27T21:41:36.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.874+0000 7f83e8edf640 1 --2- 172.21.15.60:0/2560491359 >> v2:172.21.15.60:3300/0 conn(0x7f83e41416f0 0x7f83e4141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:36.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.875+0000 7f83e3fff640 1 --2- 172.21.15.60:0/2560491359 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f83e40b40b0 0x7f83e40a8b90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f83e40b8060 tx=0x7f83d402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f67633adbaa0c93e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:36.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.876+0000 7f83e2ffd640 1 -- 172.21.15.60:0/2560491359 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d4031890 con 0x7f83e40b40b0 2024-03-27T21:41:36.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.876+0000 7f83e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:36.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.876+0000 7f83e2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:36.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.876+0000 7f83e2ffd640 10 monclient: dump: 2024-03-27T21:41:36.882 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:36.882 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:36.882 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.882 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.882 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:36.882 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:36.882 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:36.882 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:36.882 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:36.882 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:36.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.876+0000 7f83e2ffd640 10 monclient: _finish_auth 0 2024-03-27T21:41:36.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.876+0000 7f83e2ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:41:36.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.876+0000 7f83e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:36.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.876+0000 7f83e2ffd640 1 -- 172.21.15.60:0/2560491359 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d40319f0 con 0x7f83e40b40b0 2024-03-27T21:41:36.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.876+0000 7f83e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:36.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.876+0000 7f83e2ffd640 1 -- 172.21.15.60:0/2560491359 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d4031d80 con 0x7f83e40b40b0 2024-03-27T21:41:36.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.876+0000 7f83e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:36.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.876+0000 7f83e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:36.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.876+0000 7f83e2ffd640 10 monclient: dump: 2024-03-27T21:41:36.883 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:36.883 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:36.883 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.883 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.883 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:36.883 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:36.883 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:36.883 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:36.883 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:36.884 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:36.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.876+0000 7f83ea942640 5 monclient: authenticate success, global_id 4853 2024-03-27T21:41:36.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.876+0000 7f83ea942640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:36.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.876+0000 7f83ea942640 4 set_mon_vals no callback set 2024-03-27T21:41:36.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.877+0000 7f83ea942640 10 monclient: shutdown 2024-03-27T21:41:36.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.877+0000 7f83ea942640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:36.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.877+0000 7f83ea942640 1 -- 172.21.15.60:0/2560491359 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f83e40b40b0 msgr2=0x7f83e40a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.877+0000 7f83ea942640 1 --2- 172.21.15.60:0/2560491359 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f83e40b40b0 0x7f83e40a8b90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f83e40b8060 tx=0x7f83d402b930 comp rx=0 tx=0).stop 2024-03-27T21:41:36.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.877+0000 7f83ea942640 1 -- 172.21.15.60:0/2560491359 shutdown_connections 2024-03-27T21:41:36.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.877+0000 7f83ea942640 1 --2- 172.21.15.60:0/2560491359 >> v2:172.21.15.60:3300/0 conn(0x7f83e41416f0 0x7f83e4141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.877+0000 7f83ea942640 1 --2- 172.21.15.60:0/2560491359 >> v2:172.21.15.191:3300/0 conn(0x7f83e413b020 0x7f83e413b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.877+0000 7f83ea942640 1 --2- 172.21.15.60:0/2560491359 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f83e40b40b0 0x7f83e40a8b90 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.877+0000 7f83ea942640 1 -- 172.21.15.60:0/2560491359 >> 172.21.15.60:0/2560491359 conn(0x7f83e40608e0 msgr2=0x7f83e4060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:36.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.877+0000 7f83ea942640 1 -- 172.21.15.60:0/2560491359 shutdown_connections 2024-03-27T21:41:36.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.877+0000 7f83ea942640 1 -- 172.21.15.60:0/2560491359 wait complete. 2024-03-27T21:41:36.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.878+0000 7f83ea942640 10 monclient: build_initial_monmap 2024-03-27T21:41:36.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.878+0000 7f83ea942640 10 monclient: monmap: 2024-03-27T21:41:36.885 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:36.885 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:36.885 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:36.885 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:36.885 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:36.885 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:36.885 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:36.885 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:36.885 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:36.885 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:36.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.879+0000 7f83ea942640 1 librados: starting msgr at 2024-03-27T21:41:36.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.879+0000 7f83ea942640 1 librados: starting objecter 2024-03-27T21:41:36.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.879+0000 7f83ea942640 1 Processor -- start 2024-03-27T21:41:36.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.879+0000 7f83ea942640 1 -- start start 2024-03-27T21:41:36.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.879+0000 7f83ea942640 1 librados: setting wanted keys 2024-03-27T21:41:36.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.879+0000 7f83ea942640 1 librados: calling monclient init 2024-03-27T21:41:36.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.879+0000 7f83ea942640 10 monclient: init 2024-03-27T21:41:36.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83ea942640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:36.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83ea942640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:41:36.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83ea942640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f83e40b40b0 0x7f83e406ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83ea942640 10 monclient(hunting): picked mon.noname-c con 0x7f83e40b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:36.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83ea942640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83e413b020 0x7f83e406f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83e3fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f83e40b40b0 0x7f83e406ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:36.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83e3fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f83e40b40b0 0x7f83e406ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:53336/0 (socket says 172.21.15.60:53336) 2024-03-27T21:41:36.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83e37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83e413b020 0x7f83e406f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:36.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83e37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83e413b020 0x7f83e406f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50900/0 (socket says 172.21.15.60:50900) 2024-03-27T21:41:36.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83ea942640 10 monclient(hunting): picked mon.noname-a con 0x7f83e413b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:36.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83ea942640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83e41416f0 0x7f83e406c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83e37fe640 1 -- 172.21.15.60:0/2636099805 learned_addr learned my addr 172.21.15.60:0/2636099805 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:36.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83ea942640 10 monclient(hunting): picked mon.noname-b con 0x7f83e41416f0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:36.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83ea942640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:36.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83ea942640 1 -- 172.21.15.60:0/2636099805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e40ab0a0 con 0x7f83e413b020 2024-03-27T21:41:36.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83ea942640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:36.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83ea942640 1 -- 172.21.15.60:0/2636099805 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e406d760 con 0x7f83e41416f0 2024-03-27T21:41:36.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83ea942640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:36.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83ea942640 1 -- 172.21.15.60:0/2636099805 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e4139c60 con 0x7f83e40b40b0 2024-03-27T21:41:36.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83ea942640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:36.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83ea942640 10 monclient(hunting): authenticate will time out at 938.102835s 2024-03-27T21:41:36.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83e37fe640 10 monclient(hunting): get_auth_request con 0x7f83e413b020 auth_method 0 2024-03-27T21:41:36.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:36.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:36.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:36.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83e3fff640 10 monclient(hunting): get_auth_request con 0x7f83e40b40b0 auth_method 0 2024-03-27T21:41:36.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:36.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:36.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.880+0000 7f83e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:36.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:36.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:36.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:36.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:36.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:36.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:36.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e37fe640 10 monclient(hunting): handle_auth_done global_id 5094 payload 824 2024-03-27T21:41:36.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e8edf640 1 --2- 172.21.15.60:0/2636099805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83e41416f0 0x7f83e406c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:36.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e37fe640 1 -- 172.21.15.60:0/2636099805 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f83e40b40b0 msgr2=0x7f83e406ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e37fe640 1 --2- 172.21.15.60:0/2636099805 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f83e40b40b0 0x7f83e406ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e37fe640 1 -- 172.21.15.60:0/2636099805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83e41416f0 msgr2=0x7f83e406c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:36.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e37fe640 1 --2- 172.21.15.60:0/2636099805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83e41416f0 0x7f83e406c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:36.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e37fe640 10 monclient: _finish_hunting 0 2024-03-27T21:41:36.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e37fe640 1 monclient: found mon.noname-a 2024-03-27T21:41:36.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:36.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:36.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e37fe640 1 -- 172.21.15.60:0/2636099805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83dc009a70 con 0x7f83e413b020 2024-03-27T21:41:36.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e8edf640 10 monclient: get_auth_request con 0x7f83e41416f0 auth_method 0 2024-03-27T21:41:36.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e8edf640 1 --2- 172.21.15.60:0/2636099805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83e41416f0 0x7f83e406c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:41:36.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e3fff640 1 --2- 172.21.15.60:0/2636099805 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f83e40b40b0 0x7f83e406ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:36.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e37fe640 1 --2- 172.21.15.60:0/2636099805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83e413b020 0x7f83e406f3a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f83d800b450 tx=0x7f83d800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:36.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e0ff9640 10 client.?.objecter ms_handle_connect 0x7f83e413b020 2024-03-27T21:41:36.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.881+0000 7f83e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:36.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83e0ff9640 1 -- 172.21.15.60:0/2636099805 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d8011890 con 0x7f83e413b020 2024-03-27T21:41:36.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:36.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83e0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:36.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83e0ff9640 10 monclient: dump: 2024-03-27T21:41:36.890 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:36.890 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:36.890 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.890 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.890 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:36.891 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:36.891 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:36.891 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:36.891 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:36.891 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:36.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83e0ff9640 10 monclient: _finish_auth 0 2024-03-27T21:41:36.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83e0ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:41:36.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:36.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83e0ff9640 1 -- 172.21.15.60:0/2636099805 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d8011a10 con 0x7f83e413b020 2024-03-27T21:41:36.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:36.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83e0ff9640 1 -- 172.21.15.60:0/2636099805 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d8011dc0 con 0x7f83e413b020 2024-03-27T21:41:36.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:36.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83e0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:36.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83e0ff9640 10 monclient: dump: 2024-03-27T21:41:36.892 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:36.892 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:36.892 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.892 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:36.892 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:36.892 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:36.892 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:36.892 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:36.892 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:36.892 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:36.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83ea942640 5 monclient: authenticate success, global_id 5094 2024-03-27T21:41:36.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83ea942640 10 monclient: _renew_subs 2024-03-27T21:41:36.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83ea942640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:36.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83ea942640 1 -- 172.21.15.60:0/2636099805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83d40094d0 con 0x7f83e413b020 2024-03-27T21:41:36.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83ea942640 10 client.5094.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:36.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83ea942640 10 monclient: _renew_subs 2024-03-27T21:41:36.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83ea942640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:36.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83ea942640 1 -- 172.21.15.60:0/2636099805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e406c9f0 con 0x7f83e413b020 2024-03-27T21:41:36.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83ea942640 1 librados: init done 2024-03-27T21:41:36.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.882+0000 7f83e27fc640 4 set_mon_vals no callback set 2024-03-27T21:41:36.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.884+0000 7f83ea942640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:36.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.884+0000 7f83ea942640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:36.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.884+0000 7f83ea942640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:36.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.884+0000 7f83e0ff9640 1 -- 172.21.15.60:0/2636099805 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f83d801a400 con 0x7f83e413b020 2024-03-27T21:41:36.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.884+0000 7f83ea942640 1 -- 172.21.15.60:0/2636099805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e406f900 con 0x7f83e413b020 2024-03-27T21:41:36.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.885+0000 7f83e0ff9640 1 --2- 172.21.15.60:0/2636099805 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f83d006c590 0x7f83d006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:36.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.885+0000 7f83e0ff9640 1 -- 172.21.15.60:0/2636099805 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f83d8089420 con 0x7f83e413b020 2024-03-27T21:41:36.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.885+0000 7f83e0ff9640 10 client.5094.objecter ms_dispatch 0x7f83e4000b90 osd_map(69..69 src has 1..69) v4 2024-03-27T21:41:36.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.885+0000 7f83e0ff9640 3 client.5094.objecter handle_osd_map got epochs [69,69] > 0 2024-03-27T21:41:36.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.885+0000 7f83e0ff9640 3 client.5094.objecter handle_osd_map decoding full epoch 69 2024-03-27T21:41:36.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.885+0000 7f83e0ff9640 20 client.5094.objecter dump_active .. 0 homeless 2024-03-27T21:41:36.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.885+0000 7f83e3fff640 1 --2- 172.21.15.60:0/2636099805 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f83d006c590 0x7f83d006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:36.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.885+0000 7f83e3fff640 10 monclient: get_auth_request con 0x7f83d006c590 auth_method 0 2024-03-27T21:41:36.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.886+0000 7f83e0ff9640 10 client.5094.objecter ms_handle_connect 0x7f83d006c590 2024-03-27T21:41:36.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.886+0000 7f83e3fff640 1 --2- 172.21.15.60:0/2636099805 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f83d006c590 0x7f83d006ea50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f83d4036000 tx=0x7f83d403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:36.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.890+0000 7f83e0ff9640 1 -- 172.21.15.60:0/2636099805 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f83e406f900 con 0x7f83e413b020 2024-03-27T21:41:36.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.890+0000 7f83e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:36.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:36.890+0000 7f83e0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:36.969 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:36.982+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:37.008 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:37.009 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:37.022 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:41:37.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.089+0000 7f83ea942640 10 start_mon_command cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}] 2024-03-27T21:41:37.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.089+0000 7f83ea942640 10 monclient: _send_command 2 [{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}] 2024-03-27T21:41:37.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.089+0000 7f83ea942640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:37.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.089+0000 7f83ea942640 1 -- 172.21.15.60:0/2636099805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 6, "weight": 0.0} v 0) v1 -- 0x7f83e406f900 con 0x7f83e413b020 2024-03-27T21:41:37.078 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:37.089+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:37.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.100+0000 7f83e0ff9640 1 -- 172.21.15.60:0/2636099805 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]=0 set osd.6 primary-affinity to 0 (802) v70) v1 ==== 129+0+0 (secure 0 0 0) 0x7f83e406f900 con 0x7f83e413b020 2024-03-27T21:41:37.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.100+0000 7f83e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}] 2024-03-27T21:41:37.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.100+0000 7f83e0ff9640 10 monclient: _finish_command 2 = system:0 set osd.6 primary-affinity to 0 (802) 2024-03-27T21:41:37.088 INFO:teuthology.orchestra.run.smithi060.stderr:set osd.6 primary-affinity to 0 (802) 2024-03-27T21:41:37.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.104+0000 7f83ea942640 10 librados: watch_flush enter 2024-03-27T21:41:37.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.104+0000 7f83ea942640 10 librados: watch_flush exit 2024-03-27T21:41:37.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.104+0000 7f83ea942640 20 client.5094.objecter shutdown clearing up homeless session... 2024-03-27T21:41:37.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.104+0000 7f83ea942640 10 client.5094.objecter successfully canceled tick 2024-03-27T21:41:37.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.104+0000 7f83ea942640 1 -- 172.21.15.60:0/2636099805 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f83d006c590 msgr2=0x7f83d006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:37.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.104+0000 7f83ea942640 1 --2- 172.21.15.60:0/2636099805 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f83d006c590 0x7f83d006ea50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f83d4036000 tx=0x7f83d403c040 comp rx=0 tx=0).stop 2024-03-27T21:41:37.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.104+0000 7f83ea942640 10 monclient: shutdown 2024-03-27T21:41:37.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.104+0000 7f83ea942640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:37.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.104+0000 7f83ea942640 1 -- 172.21.15.60:0/2636099805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83e413b020 msgr2=0x7f83e406f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:37.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.104+0000 7f83ea942640 1 --2- 172.21.15.60:0/2636099805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83e413b020 0x7f83e406f3a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f83d800b450 tx=0x7f83d800b910 comp rx=0 tx=0).stop 2024-03-27T21:41:37.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.104+0000 7f83ea942640 1 -- 172.21.15.60:0/2636099805 shutdown_connections 2024-03-27T21:41:37.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.104+0000 7f83ea942640 1 --2- 172.21.15.60:0/2636099805 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f83d006c590 0x7f83d006ea50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:37.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.104+0000 7f83ea942640 1 --2- 172.21.15.60:0/2636099805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83e41416f0 0x7f83e406c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:37.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.104+0000 7f83ea942640 1 --2- 172.21.15.60:0/2636099805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83e413b020 0x7f83e406f3a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:37.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.104+0000 7f83ea942640 1 --2- 172.21.15.60:0/2636099805 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f83e40b40b0 0x7f83e406ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:37.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.104+0000 7f83ea942640 1 -- 172.21.15.60:0/2636099805 >> 172.21.15.60:0/2636099805 conn(0x7f83e40608e0 msgr2=0x7f83e4060cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:37.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.104+0000 7f83ea942640 1 -- 172.21.15.60:0/2636099805 shutdown_connections 2024-03-27T21:41:37.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.104+0000 7f83ea942640 1 -- 172.21.15.60:0/2636099805 wait complete. 2024-03-27T21:41:37.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.105+0000 7f83ea942640 1 librados: shutdown 2024-03-27T21:41:37.094 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-27T21:41:37.171 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:37.185+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:37.186 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:37.186 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:37.196 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:41:37.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.215+0000 7fb1ac8c0640 10 monclient: get_monmap_and_config 2024-03-27T21:41:37.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.215+0000 7fb1ac8c0640 10 monclient: build_initial_monmap 2024-03-27T21:41:37.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.215+0000 7fb1ac8c0640 10 monclient: monmap: 2024-03-27T21:41:37.203 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:37.203 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:37.203 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:37.217445+0000 2024-03-27T21:41:37.203 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:37.217445+0000 2024-03-27T21:41:37.203 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:37.203 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:37.203 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:37.203 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:37.203 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:37.203 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:37.203 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:37.203 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:37.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.216+0000 7fb1ac8c0640 1 Processor -- start 2024-03-27T21:41:37.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.216+0000 7fb1ac8c0640 1 -- start start 2024-03-27T21:41:37.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.216+0000 7fb1ac8c0640 10 monclient: init 2024-03-27T21:41:37.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1ac8c0640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:37.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1ac8c0640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-03-27T21:41:37.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1ac8c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb1a80b40b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:37.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1ac8c0640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb1a813b020 0x7fb1a813b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:37.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1ac8c0640 10 monclient(hunting): picked mon.noname-c con 0x7fb1a813b020 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:37.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1ac8c0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb1a81416f0 0x7fb1a8141b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:37.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1ac8c0640 10 monclient(hunting): picked mon.noname-a con 0x7fb1a81416f0 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:37.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1ac8c0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a80b3830 con 0x7fb1a80b40b0 2024-03-27T21:41:37.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1ac8c0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:37.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1ac8c0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1a80b4490 con 0x7fb1a81416f0 2024-03-27T21:41:37.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1ac8c0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:37.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1ac8c0640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1a80aa520 con 0x7fb1a813b020 2024-03-27T21:41:37.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1ac8c0640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:37.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1ac8c0640 10 monclient(hunting): authenticate will time out at 938.439438s 2024-03-27T21:41:37.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1a659c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb1a80b40b0 0x7fb1a80a8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52306/0 (socket says 172.21.15.60:52306) 2024-03-27T21:41:37.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1a659c640 1 -- 172.21.15.60:0/2321149476 learned_addr learned my addr 172.21.15.60:0/2321149476 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:37.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1a6d9d640 1 --2- 172.21.15.60:0/2321149476 >> v2:172.21.15.60:3300/0 conn(0x7fb1a81416f0 0x7fb1a8141b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:37.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1a6d9d640 10 monclient(hunting): get_auth_request con 0x7fb1a81416f0 auth_method 0 2024-03-27T21:41:37.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:37.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1a6d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:37.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:37.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:37.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:37.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:37.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb19ffff640 1 --2- 172.21.15.60:0/2321149476 >> v2:172.21.15.191:3300/0 conn(0x7fb1a813b020 0x7fb1a813b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:37.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1a6d9d640 10 monclient(hunting): handle_auth_done global_id 5097 payload 274 2024-03-27T21:41:37.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1a6d9d640 1 -- 172.21.15.60:0/2321149476 >> v2:172.21.15.191:3300/0 conn(0x7fb1a813b020 msgr2=0x7fb1a813b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:37.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1a6d9d640 1 --2- 172.21.15.60:0/2321149476 >> v2:172.21.15.191:3300/0 conn(0x7fb1a813b020 0x7fb1a813b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:37.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1a6d9d640 1 -- 172.21.15.60:0/2321149476 >> v1:172.21.15.60:6789/0 conn(0x7fb1a80b40b0 legacy=0x7fb1a80a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:37.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1a6d9d640 10 monclient: _finish_hunting 0 2024-03-27T21:41:37.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1a6d9d640 1 monclient: found mon.noname-a 2024-03-27T21:41:37.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1a6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:37.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1a6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:37.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb1a6d9d640 1 -- 172.21.15.60:0/2321149476 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a80a9470 con 0x7fb1a81416f0 2024-03-27T21:41:37.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb19ffff640 10 monclient: get_auth_request con 0x7fb1a813b020 auth_method 0 2024-03-27T21:41:37.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.217+0000 7fb19ffff640 1 --2- 172.21.15.60:0/2321149476 >> v2:172.21.15.191:3300/0 conn(0x7fb1a813b020 0x7fb1a813b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:41:37.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.219+0000 7fb1a6d9d640 1 --2- 172.21.15.60:0/2321149476 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1a81416f0 0x7fb1a8141b40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb1a80b47e0 tx=0x7fb1a002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c121eb59af402af7 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:37.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.219+0000 7fb1a5d9b640 1 -- 172.21.15.60:0/2321149476 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a002cc80 con 0x7fb1a81416f0 2024-03-27T21:41:37.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.219+0000 7fb1a5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:37.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.219+0000 7fb1a5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:37.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.219+0000 7fb1a5d9b640 10 monclient: dump: 2024-03-27T21:41:37.211 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:37.211 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:37.211 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:37.211 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:37.211 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:37.211 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:37.211 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:37.211 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:37.211 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:37.212 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.219+0000 7fb1a5d9b640 10 monclient: _finish_auth 0 2024-03-27T21:41:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.219+0000 7fb1a5d9b640 10 monclient: _check_auth_tickets 2024-03-27T21:41:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.219+0000 7fb1a5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.219+0000 7fb1a5d9b640 1 -- 172.21.15.60:0/2321149476 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1a002cde0 con 0x7fb1a81416f0 2024-03-27T21:41:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.219+0000 7fb1a5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.219+0000 7fb1a5d9b640 1 -- 172.21.15.60:0/2321149476 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0034660 con 0x7fb1a81416f0 2024-03-27T21:41:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.219+0000 7fb1a5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.219+0000 7fb1a5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.219+0000 7fb1a5d9b640 10 monclient: dump: 2024-03-27T21:41:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:37.213 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:37.213 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:37.213 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:37.213 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:37.213 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:37.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.219+0000 7fb1ac8c0640 5 monclient: authenticate success, global_id 5097 2024-03-27T21:41:37.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.219+0000 7fb1ac8c0640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:37.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.219+0000 7fb1ac8c0640 4 set_mon_vals no callback set 2024-03-27T21:41:37.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.220+0000 7fb1ac8c0640 10 monclient: shutdown 2024-03-27T21:41:37.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.220+0000 7fb1ac8c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:37.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.220+0000 7fb1ac8c0640 1 -- 172.21.15.60:0/2321149476 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1a81416f0 msgr2=0x7fb1a8141b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:37.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.220+0000 7fb1ac8c0640 1 --2- 172.21.15.60:0/2321149476 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1a81416f0 0x7fb1a8141b40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb1a80b47e0 tx=0x7fb1a002bed0 comp rx=0 tx=0).stop 2024-03-27T21:41:37.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.220+0000 7fb1ac8c0640 1 -- 172.21.15.60:0/2321149476 shutdown_connections 2024-03-27T21:41:37.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.220+0000 7fb1ac8c0640 1 --2- 172.21.15.60:0/2321149476 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1a81416f0 0x7fb1a8141b40 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:37.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.220+0000 7fb1ac8c0640 1 --2- 172.21.15.60:0/2321149476 >> v2:172.21.15.191:3300/0 conn(0x7fb1a813b020 0x7fb1a813b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:37.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.220+0000 7fb1ac8c0640 1 -- 172.21.15.60:0/2321149476 >> 172.21.15.60:0/2321149476 conn(0x7fb1a80608e0 msgr2=0x7fb1a8060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:37.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.220+0000 7fb1ac8c0640 1 -- 172.21.15.60:0/2321149476 shutdown_connections 2024-03-27T21:41:37.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.220+0000 7fb1ac8c0640 1 -- 172.21.15.60:0/2321149476 wait complete. 2024-03-27T21:41:37.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.221+0000 7fb1ac8c0640 10 monclient: build_initial_monmap 2024-03-27T21:41:37.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.221+0000 7fb1ac8c0640 10 monclient: monmap: 2024-03-27T21:41:37.220 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:37.220 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:37.220 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:37.220 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:37.221 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:37.221 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:37.221 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:37.221 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:37.221 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:37.221 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:37.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.222+0000 7fb1ac8c0640 1 librados: starting msgr at 2024-03-27T21:41:37.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.222+0000 7fb1ac8c0640 1 librados: starting objecter 2024-03-27T21:41:37.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.222+0000 7fb1ac8c0640 1 Processor -- start 2024-03-27T21:41:37.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.222+0000 7fb1ac8c0640 1 -- start start 2024-03-27T21:41:37.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.222+0000 7fb1ac8c0640 1 librados: setting wanted keys 2024-03-27T21:41:37.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.222+0000 7fb1ac8c0640 1 librados: calling monclient init 2024-03-27T21:41:37.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.222+0000 7fb1ac8c0640 10 monclient: init 2024-03-27T21:41:37.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.223+0000 7fb1ac8c0640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:37.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.223+0000 7fb1ac8c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:41:37.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.223+0000 7fb1ac8c0640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb1a80b40b0 0x7fb1a813a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:37.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.223+0000 7fb1a659c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb1a80b40b0 0x7fb1a813a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:37.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.223+0000 7fb1a659c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb1a80b40b0 0x7fb1a813a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:53342/0 (socket says 172.21.15.60:53342) 2024-03-27T21:41:37.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.223+0000 7fb1a659c640 1 -- 172.21.15.60:0/2881349065 learned_addr learned my addr 172.21.15.60:0/2881349065 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:37.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.223+0000 7fb1ac8c0640 10 monclient(hunting): picked mon.noname-c con 0x7fb1a80b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:37.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.223+0000 7fb1ac8c0640 1 --2- 172.21.15.60:0/2881349065 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1a813b020 0x7fb1a80aa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:37.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.223+0000 7fb1ac8c0640 10 monclient(hunting): picked mon.noname-b con 0x7fb1a813b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:37.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.223+0000 7fb1ac8c0640 1 --2- 172.21.15.60:0/2881349065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1a81416f0 0x7fb1a8134160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:37.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.223+0000 7fb1ac8c0640 10 monclient(hunting): picked mon.noname-a con 0x7fb1a81416f0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:37.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.223+0000 7fb1ac8c0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:37.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.223+0000 7fb1ac8c0640 1 -- 172.21.15.60:0/2881349065 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a813e430 con 0x7fb1a81416f0 2024-03-27T21:41:37.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.223+0000 7fb1ac8c0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:37.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.223+0000 7fb1ac8c0640 1 -- 172.21.15.60:0/2881349065 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a813e680 con 0x7fb1a813b020 2024-03-27T21:41:37.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.223+0000 7fb1ac8c0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:37.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.223+0000 7fb1ac8c0640 1 -- 172.21.15.60:0/2881349065 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a813d920 con 0x7fb1a80b40b0 2024-03-27T21:41:37.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.223+0000 7fb1ac8c0640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:37.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.223+0000 7fb1ac8c0640 10 monclient(hunting): authenticate will time out at 938.445890s 2024-03-27T21:41:37.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.223+0000 7fb1a659c640 10 monclient(hunting): get_auth_request con 0x7fb1a80b40b0 auth_method 0 2024-03-27T21:41:37.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb1a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:37.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb1a659c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:37.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb1a659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:37.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb1a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:37.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb1a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:37.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb1a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:37.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb19ffff640 1 --2- 172.21.15.60:0/2881349065 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1a813b020 0x7fb1a80aa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:37.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb19ffff640 10 monclient(hunting): get_auth_request con 0x7fb1a813b020 auth_method 0 2024-03-27T21:41:37.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb19ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:37.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb19ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:37.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb19ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:37.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb19ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:37.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb19ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:37.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb19ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:37.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb1a659c640 10 monclient(hunting): handle_auth_done global_id 4859 payload 824 2024-03-27T21:41:37.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb1a659c640 1 -- 172.21.15.60:0/2881349065 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1a813b020 msgr2=0x7fb1a80aa9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:37.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb1a659c640 1 --2- 172.21.15.60:0/2881349065 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1a813b020 0x7fb1a80aa9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:37.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb1a659c640 1 -- 172.21.15.60:0/2881349065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1a81416f0 msgr2=0x7fb1a8134160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T21:41:37.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb1a6d9d640 1 --2- 172.21.15.60:0/2881349065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1a81416f0 0x7fb1a8134160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:37.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb1a659c640 1 --2- 172.21.15.60:0/2881349065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1a81416f0 0x7fb1a8134160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb1a659c640 10 monclient: _finish_hunting 0 2024-03-27T21:41:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb1a659c640 1 monclient: found mon.noname-c 2024-03-27T21:41:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb1a659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb1a659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb1a659c640 1 -- 172.21.15.60:0/2881349065 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a0009a70 con 0x7fb1a80b40b0 2024-03-27T21:41:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.224+0000 7fb1a659c640 1 --2- 172.21.15.60:0/2881349065 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb1a80b40b0 0x7fb1a813a830 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb19800c700 tx=0x7fb19800cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.225+0000 7fb19effd640 10 client.?.objecter ms_handle_connect 0x7fb1a80b40b0 2024-03-27T21:41:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.225+0000 7fb19effd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.225+0000 7fb1a6d9d640 10 monclient: get_auth_request con 0x7fb1a81416f0 auth_method 0 2024-03-27T21:41:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.225+0000 7fb1a6d9d640 1 --2- 172.21.15.60:0/2881349065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1a81416f0 0x7fb1a8134160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:41:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.225+0000 7fb19effd640 1 -- 172.21.15.60:0/2881349065 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb198005ab0 con 0x7fb1a80b40b0 2024-03-27T21:41:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.225+0000 7fb19effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.225+0000 7fb19effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.225+0000 7fb19effd640 10 monclient: dump: 2024-03-27T21:41:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:37.226 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.225+0000 7fb19effd640 10 monclient: _finish_auth 0 2024-03-27T21:41:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.225+0000 7fb19effd640 10 monclient: _check_auth_tickets 2024-03-27T21:41:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.225+0000 7fb19effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:37.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.225+0000 7fb19effd640 1 -- 172.21.15.60:0/2881349065 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb198005c10 con 0x7fb1a80b40b0 2024-03-27T21:41:37.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.225+0000 7fb1ac8c0640 5 monclient: authenticate success, global_id 4859 2024-03-27T21:41:37.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.225+0000 7fb1ac8c0640 10 monclient: _renew_subs 2024-03-27T21:41:37.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.225+0000 7fb1ac8c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:37.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.225+0000 7fb1ac8c0640 1 -- 172.21.15.60:0/2881349065 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a813db40 con 0x7fb1a80b40b0 2024-03-27T21:41:37.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.226+0000 7fb19effd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:37.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.226+0000 7fb19effd640 1 -- 172.21.15.60:0/2881349065 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19800ce10 con 0x7fb1a80b40b0 2024-03-27T21:41:37.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.226+0000 7fb19effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:37.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.226+0000 7fb19effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:37.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.226+0000 7fb19effd640 10 monclient: dump: 2024-03-27T21:41:37.227 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:37.227 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:37.227 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:37.227 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:37.228 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:37.228 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:37.228 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:37.228 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:37.228 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:37.228 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:37.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.226+0000 7fb1a559a640 4 set_mon_vals no callback set 2024-03-27T21:41:37.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.226+0000 7fb1ac8c0640 10 client.4859.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:37.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.226+0000 7fb1ac8c0640 10 monclient: _renew_subs 2024-03-27T21:41:37.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.226+0000 7fb1ac8c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:37.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.226+0000 7fb1ac8c0640 1 -- 172.21.15.60:0/2881349065 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a8134880 con 0x7fb1a80b40b0 2024-03-27T21:41:37.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.226+0000 7fb1ac8c0640 1 librados: init done 2024-03-27T21:41:37.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.228+0000 7fb1ac8c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:37.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.228+0000 7fb1ac8c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.228+0000 7fb1ac8c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.228+0000 7fb1ac8c0640 1 -- 172.21.15.60:0/2881349065 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1a8069470 con 0x7fb1a80b40b0 2024-03-27T21:41:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.228+0000 7fb19effd640 1 -- 172.21.15.60:0/2881349065 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb198007070 con 0x7fb1a80b40b0 2024-03-27T21:41:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.228+0000 7fb19effd640 1 --2- 172.21.15.60:0/2881349065 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb18806c590 0x7fb18806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.229+0000 7fb19effd640 1 -- 172.21.15.60:0/2881349065 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fb1980883d0 con 0x7fb1a80b40b0 2024-03-27T21:41:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.229+0000 7fb19effd640 10 client.4859.objecter ms_dispatch 0x7fb1a8000b90 osd_map(70..70 src has 1..70) v4 2024-03-27T21:41:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.229+0000 7fb19effd640 3 client.4859.objecter handle_osd_map got epochs [70,70] > 0 2024-03-27T21:41:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.229+0000 7fb19effd640 3 client.4859.objecter handle_osd_map decoding full epoch 70 2024-03-27T21:41:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.229+0000 7fb19ffff640 1 --2- 172.21.15.60:0/2881349065 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb18806c590 0x7fb18806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.229+0000 7fb19effd640 20 client.4859.objecter dump_active .. 0 homeless 2024-03-27T21:41:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.229+0000 7fb19ffff640 10 monclient: get_auth_request con 0x7fb18806c590 auth_method 0 2024-03-27T21:41:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.229+0000 7fb19effd640 10 client.4859.objecter ms_handle_connect 0x7fb18806c590 2024-03-27T21:41:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.229+0000 7fb19ffff640 1 --2- 172.21.15.60:0/2881349065 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb18806c590 0x7fb18806ea50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb1a81382c0 tx=0x7fb190008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:37.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.237+0000 7fb19effd640 1 -- 172.21.15.60:0/2881349065 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb198053830 con 0x7fb1a80b40b0 2024-03-27T21:41:37.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.237+0000 7fb19effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:37.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.237+0000 7fb19effd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:37.272 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:37.286+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:37.362 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:37.363 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:37.373 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:37.386+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:37.376 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:41:37.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.432+0000 7fb1ac8c0640 10 start_mon_command cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:41:37.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.432+0000 7fb1ac8c0640 10 monclient: _send_command 2 [{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:41:37.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.432+0000 7fb1ac8c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:37.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:37.432+0000 7fb1ac8c0640 1 -- 172.21.15.60:0/2881349065 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fb1a8069e40 con 0x7fb1a80b40b0 2024-03-27T21:41:37.474 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:37.487+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:37.526 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:37.527 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:37.536 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:41:37.576 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:37.590+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:37.676 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:37.689+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:37.697 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:37.697 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:37.708 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:41:37.795 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:37.808+0000 7fb850f34640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 56426) UID: 0 2024-03-27T21:41:37.859 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:37.860 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:37.869 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:41:37.878 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:37.892+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:37.979 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:37.993+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:38.027 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:38.027 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:38.036 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:41:38.080 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:38.093+0000 7fb850f34640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 56426) UID: 0 2024-03-27T21:41:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.102+0000 7fb19effd640 1 -- 172.21.15.60:0/2881349065 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v71) v1 ==== 101+0+0 (secure 0 0 0) 0x7fb198056e50 con 0x7fb1a80b40b0 2024-03-27T21:41:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.102+0000 7fb19effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:41:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.102+0000 7fb19effd640 10 monclient: _finish_command 2 = system:0 nodeep-scrub is unset 2024-03-27T21:41:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:nodeep-scrub is unset 2024-03-27T21:41:38.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.105+0000 7fb1ac8c0640 10 librados: watch_flush enter 2024-03-27T21:41:38.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.105+0000 7fb1ac8c0640 10 librados: watch_flush exit 2024-03-27T21:41:38.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.105+0000 7fb1ac8c0640 20 client.4859.objecter shutdown clearing up homeless session... 2024-03-27T21:41:38.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.105+0000 7fb1ac8c0640 10 client.4859.objecter successfully canceled tick 2024-03-27T21:41:38.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.105+0000 7fb1ac8c0640 1 -- 172.21.15.60:0/2881349065 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb18806c590 msgr2=0x7fb18806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:38.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.105+0000 7fb1ac8c0640 1 --2- 172.21.15.60:0/2881349065 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb18806c590 0x7fb18806ea50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb1a81382c0 tx=0x7fb190008040 comp rx=0 tx=0).stop 2024-03-27T21:41:38.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.105+0000 7fb1ac8c0640 10 monclient: shutdown 2024-03-27T21:41:38.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.105+0000 7fb1ac8c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:38.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.105+0000 7fb1ac8c0640 1 -- 172.21.15.60:0/2881349065 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb1a80b40b0 msgr2=0x7fb1a813a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:38.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.105+0000 7fb1ac8c0640 1 --2- 172.21.15.60:0/2881349065 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb1a80b40b0 0x7fb1a813a830 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb19800c700 tx=0x7fb19800cbc0 comp rx=0 tx=0).stop 2024-03-27T21:41:38.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.105+0000 7fb1ac8c0640 1 -- 172.21.15.60:0/2881349065 shutdown_connections 2024-03-27T21:41:38.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.105+0000 7fb1ac8c0640 1 --2- 172.21.15.60:0/2881349065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1a81416f0 0x7fb1a8134160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:38.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.105+0000 7fb1ac8c0640 1 --2- 172.21.15.60:0/2881349065 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb18806c590 0x7fb18806ea50 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:38.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.105+0000 7fb1ac8c0640 1 --2- 172.21.15.60:0/2881349065 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1a813b020 0x7fb1a80aa9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:38.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.105+0000 7fb1ac8c0640 1 --2- 172.21.15.60:0/2881349065 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb1a80b40b0 0x7fb1a813a830 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb19800c700 tx=0x7fb19800cbc0 comp rx=0 tx=0).stop 2024-03-27T21:41:38.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.105+0000 7fb1ac8c0640 1 -- 172.21.15.60:0/2881349065 >> 172.21.15.60:0/2881349065 conn(0x7fb1a80608e0 msgr2=0x7fb1a813f810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:38.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.106+0000 7fb1ac8c0640 1 -- 172.21.15.60:0/2881349065 shutdown_connections 2024-03-27T21:41:38.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.106+0000 7fb1ac8c0640 1 -- 172.21.15.60:0/2881349065 wait complete. 2024-03-27T21:41:38.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.106+0000 7fb1ac8c0640 1 librados: shutdown 2024-03-27T21:41:38.151 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:41:38.181 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:38.195+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:38.187 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:38.187 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:38.200 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:41:38.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.267+0000 7f386c52e640 10 monclient: get_monmap_and_config 2024-03-27T21:41:38.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.267+0000 7f386c52e640 10 monclient: build_initial_monmap 2024-03-27T21:41:38.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.267+0000 7f386c52e640 10 monclient: monmap: 2024-03-27T21:41:38.255 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:38.255 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:38.255 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:38.269461+0000 2024-03-27T21:41:38.255 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:38.269461+0000 2024-03-27T21:41:38.255 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:38.255 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:38.255 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:38.255 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:38.255 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:38.255 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:38.255 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:38.255 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:38.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.268+0000 7f386c52e640 1 Processor -- start 2024-03-27T21:41:38.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.268+0000 7f386c52e640 1 -- start start 2024-03-27T21:41:38.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.268+0000 7f386c52e640 10 monclient: init 2024-03-27T21:41:38.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f386c52e640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:38.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f386c52e640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-03-27T21:41:38.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f386c52e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f38640acac0 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:38.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f386c52e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f386413e630 0x7f386413ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:38.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f3869ac9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f386413e630 0x7f386413ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:38.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f386a2ca640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f38640acac0 0x7f386406cf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.60:40530/0 (socket says 172.21.15.60:40530) 2024-03-27T21:41:38.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f3869ac9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f386413e630 0x7f386413ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50928/0 (socket says 172.21.15.60:50928) 2024-03-27T21:41:38.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f386c52e640 10 monclient(hunting): picked mon.noname-a con 0x7f386413e630 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:38.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f386c52e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3864144890 0x7f3864144d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:38.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f386c52e640 10 monclient(hunting): picked mon.noname-b con 0x7f3864144890 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:38.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f386c52e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f386406d8a0 con 0x7f38640acac0 2024-03-27T21:41:38.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f386c52e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:38.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f386c52e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f386406d6a0 con 0x7f386413e630 2024-03-27T21:41:38.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f386c52e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:38.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f386c52e640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38640acea0 con 0x7f3864144890 2024-03-27T21:41:38.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f386c52e640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:38.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f386aacb640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3864144890 0x7f3864144d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:38.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f386c52e640 10 monclient(hunting): authenticate will time out at 939.491738s 2024-03-27T21:41:38.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f3869ac9640 1 -- 172.21.15.60:0/3861486815 learned_addr learned my addr 172.21.15.60:0/3861486815 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:38.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f386aacb640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3864144890 0x7f3864144d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:53354/0 (socket says 172.21.15.60:53354) 2024-03-27T21:41:38.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f386aacb640 10 monclient(hunting): get_auth_request con 0x7f3864144890 auth_method 0 2024-03-27T21:41:38.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f386aacb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:38.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f386aacb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:38.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f386aacb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:38.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f3869ac9640 10 monclient(hunting): get_auth_request con 0x7f386413e630 auth_method 0 2024-03-27T21:41:38.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f3869ac9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:38.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f3869ac9640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:38.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.269+0000 7f3869ac9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:38.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.270+0000 7f386aacb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:38.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.270+0000 7f386aacb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:38.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.270+0000 7f386aacb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:38.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.270+0000 7f3869ac9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:38.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.270+0000 7f3869ac9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:38.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.270+0000 7f3869ac9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:38.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.270+0000 7f386aacb640 10 monclient(hunting): handle_auth_done global_id 4862 payload 274 2024-03-27T21:41:38.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.270+0000 7f386aacb640 1 -- 172.21.15.60:0/3861486815 >> v2:172.21.15.60:3300/0 conn(0x7f386413e630 msgr2=0x7f386413ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:38.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.270+0000 7f386aacb640 1 --2- 172.21.15.60:0/3861486815 >> v2:172.21.15.60:3300/0 conn(0x7f386413e630 0x7f386413ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:38.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.270+0000 7f386aacb640 1 -- 172.21.15.60:0/3861486815 >> v1:172.21.15.191:6789/0 conn(0x7f38640acac0 legacy=0x7f386406cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:38.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.270+0000 7f386aacb640 10 monclient: _finish_hunting 0 2024-03-27T21:41:38.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.270+0000 7f386aacb640 1 monclient: found mon.noname-b 2024-03-27T21:41:38.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.270+0000 7f386aacb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:38.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.270+0000 7f386aacb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:38.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.270+0000 7f386aacb640 1 -- 172.21.15.60:0/3861486815 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38640af4e0 con 0x7f3864144890 2024-03-27T21:41:38.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.271+0000 7f3869ac9640 1 --2- 172.21.15.60:0/3861486815 >> v2:172.21.15.60:3300/0 conn(0x7f386413e630 0x7f386413ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:38.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.271+0000 7f386aacb640 1 --2- 172.21.15.60:0/3861486815 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3864144890 0x7f3864144d00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f385c009d90 tx=0x7f385c02beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=184c94a636348635 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:38.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.271+0000 7f38692c8640 1 -- 172.21.15.60:0/3861486815 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385c02cc60 con 0x7f3864144890 2024-03-27T21:41:38.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.271+0000 7f38692c8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:38.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.271+0000 7f38692c8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:38.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.271+0000 7f38692c8640 10 monclient: dump: 2024-03-27T21:41:38.266 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:38.266 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:38.267 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:38.267 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:38.267 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:38.267 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:38.267 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:38.267 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:38.267 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:38.267 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:38.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.271+0000 7f38692c8640 10 monclient: _finish_auth 0 2024-03-27T21:41:38.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.271+0000 7f38692c8640 10 monclient: _check_auth_tickets 2024-03-27T21:41:38.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.271+0000 7f38692c8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:38.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.271+0000 7f386c52e640 5 monclient: authenticate success, global_id 4862 2024-03-27T21:41:38.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.271+0000 7f386c52e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:38.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.272+0000 7f38692c8640 1 -- 172.21.15.60:0/3861486815 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f385c02cdc0 con 0x7f3864144890 2024-03-27T21:41:38.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.272+0000 7f38692c8640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:38.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.272+0000 7f38692c8640 1 -- 172.21.15.60:0/3861486815 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385c034660 con 0x7f3864144890 2024-03-27T21:41:38.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.272+0000 7f386c52e640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:38.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.272+0000 7f386c52e640 4 set_mon_vals no callback set 2024-03-27T21:41:38.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.272+0000 7f38692c8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:38.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.272+0000 7f38692c8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:38.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.272+0000 7f38692c8640 10 monclient: dump: 2024-03-27T21:41:38.268 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:38.268 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:38.268 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:38.268 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:38.268 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:38.268 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:38.268 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:38.268 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:38.268 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:38.269 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.272+0000 7f386c52e640 10 monclient: shutdown 2024-03-27T21:41:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.272+0000 7f386c52e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.272+0000 7f386c52e640 1 -- 172.21.15.60:0/3861486815 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3864144890 msgr2=0x7f3864144d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.272+0000 7f386c52e640 1 --2- 172.21.15.60:0/3861486815 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3864144890 0x7f3864144d00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f385c009d90 tx=0x7f385c02beb0 comp rx=0 tx=0).stop 2024-03-27T21:41:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.273+0000 7f386c52e640 1 -- 172.21.15.60:0/3861486815 shutdown_connections 2024-03-27T21:41:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.273+0000 7f386c52e640 1 --2- 172.21.15.60:0/3861486815 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3864144890 0x7f3864144d00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.273+0000 7f386c52e640 1 --2- 172.21.15.60:0/3861486815 >> v2:172.21.15.60:3300/0 conn(0x7f386413e630 0x7f386413ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.273+0000 7f386c52e640 1 -- 172.21.15.60:0/3861486815 >> 172.21.15.60:0/3861486815 conn(0x7f38640b2ea0 msgr2=0x7f38640b32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.273+0000 7f386c52e640 1 -- 172.21.15.60:0/3861486815 shutdown_connections 2024-03-27T21:41:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.273+0000 7f386c52e640 1 -- 172.21.15.60:0/3861486815 wait complete. 2024-03-27T21:41:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.274+0000 7f386c52e640 10 monclient: build_initial_monmap 2024-03-27T21:41:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.274+0000 7f386c52e640 10 monclient: monmap: 2024-03-27T21:41:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:38.269 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:38.270 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:38.270 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:38.270 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:38.270 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:38.270 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:38.270 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:38.270 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:38.270 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:38.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.275+0000 7f386c52e640 1 librados: starting msgr at 2024-03-27T21:41:38.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.275+0000 7f386c52e640 1 librados: starting objecter 2024-03-27T21:41:38.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.275+0000 7f386c52e640 1 Processor -- start 2024-03-27T21:41:38.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.275+0000 7f386c52e640 1 -- start start 2024-03-27T21:41:38.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.275+0000 7f386c52e640 1 librados: setting wanted keys 2024-03-27T21:41:38.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.275+0000 7f386c52e640 1 librados: calling monclient init 2024-03-27T21:41:38.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.275+0000 7f386c52e640 10 monclient: init 2024-03-27T21:41:38.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.276+0000 7f386c52e640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:38.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.276+0000 7f386c52e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:41:38.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.276+0000 7f386c52e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38640acac0 0x7f38641370c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:38.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.276+0000 7f386c52e640 10 monclient(hunting): picked mon.noname-a con 0x7f38640acac0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:38.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.276+0000 7f386c52e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f386413e630 0x7f38641d67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:38.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.276+0000 7f386c52e640 10 monclient(hunting): picked mon.noname-c con 0x7f386413e630 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:38.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.276+0000 7f386c52e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3864144890 0x7f386413a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:38.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.277+0000 7f386c52e640 10 monclient(hunting): picked mon.noname-b con 0x7f3864144890 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:38.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.277+0000 7f3869ac9640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f386413e630 0x7f38641d67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:38.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.277+0000 7f3869ac9640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f386413e630 0x7f38641d67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:53362/0 (socket says 172.21.15.60:53362) 2024-03-27T21:41:38.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.277+0000 7f3869ac9640 1 -- 172.21.15.60:0/2099221919 learned_addr learned my addr 172.21.15.60:0/2099221919 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:38.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.277+0000 7f386a2ca640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38640acac0 0x7f38641370c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:38.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.277+0000 7f386aacb640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3864144890 0x7f386413a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:38.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.277+0000 7f386c52e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:38.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.277+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38641d6060 con 0x7f38640acac0 2024-03-27T21:41:38.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.277+0000 7f386c52e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:38.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.277+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386413f270 con 0x7f3864144890 2024-03-27T21:41:38.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.277+0000 7f386c52e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:38.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.277+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38641d62b0 con 0x7f386413e630 2024-03-27T21:41:38.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.277+0000 7f386c52e640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:38.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f386c52e640 10 monclient(hunting): authenticate will time out at 939.499954s 2024-03-27T21:41:38.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f3869ac9640 10 monclient(hunting): get_auth_request con 0x7f386413e630 auth_method 0 2024-03-27T21:41:38.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f3869ac9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:38.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f3869ac9640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:38.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f3869ac9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:38.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f386a2ca640 10 monclient(hunting): get_auth_request con 0x7f38640acac0 auth_method 0 2024-03-27T21:41:38.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f386a2ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:38.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f386a2ca640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:38.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f386a2ca640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:38.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f386aacb640 10 monclient(hunting): get_auth_request con 0x7f3864144890 auth_method 0 2024-03-27T21:41:38.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f386aacb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:38.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f386aacb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:38.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f386aacb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:38.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f386a2ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:38.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f386a2ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:38.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f386a2ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:38.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f3869ac9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:38.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f3869ac9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:38.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f3869ac9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:38.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f386aacb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:38.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f386aacb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:38.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f386aacb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:38.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f3869ac9640 10 monclient(hunting): handle_auth_done global_id 4865 payload 824 2024-03-27T21:41:38.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f3869ac9640 1 -- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3864144890 msgr2=0x7f386413a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:38.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f3869ac9640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3864144890 0x7f386413a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:38.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f3869ac9640 1 -- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38640acac0 msgr2=0x7f38641370c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:38.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f3869ac9640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38640acac0 0x7f38641370c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:38.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f3869ac9640 10 monclient: _finish_hunting 0 2024-03-27T21:41:38.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f3869ac9640 1 monclient: found mon.noname-c 2024-03-27T21:41:38.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f3869ac9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:38.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f3869ac9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:38.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.278+0000 7f3869ac9640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38540094d0 con 0x7f386413e630 2024-03-27T21:41:38.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.279+0000 7f3869ac9640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f386413e630 0x7f38641d67d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f385400e9d0 tx=0x7f385400ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:38.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.279+0000 7f385affd640 10 client.?.objecter ms_handle_connect 0x7f386413e630 2024-03-27T21:41:38.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.279+0000 7f385affd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:38.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.279+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385400ca60 con 0x7f386413e630 2024-03-27T21:41:38.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.279+0000 7f385affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:38.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.279+0000 7f385affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:38.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.279+0000 7f385affd640 10 monclient: dump: 2024-03-27T21:41:38.285 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:38.285 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:38.285 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:38.285 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:38.285 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:38.285 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:38.285 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:38.285 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:38.285 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:38.286 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:38.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.279+0000 7f385affd640 10 monclient: _finish_auth 0 2024-03-27T21:41:38.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.279+0000 7f385affd640 10 monclient: _check_auth_tickets 2024-03-27T21:41:38.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.279+0000 7f385affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:38.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.279+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f385400cbc0 con 0x7f386413e630 2024-03-27T21:41:38.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.279+0000 7f385affd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:38.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.279+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3854013470 con 0x7f386413e630 2024-03-27T21:41:38.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.279+0000 7f385affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:38.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.279+0000 7f385affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:38.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.279+0000 7f385affd640 10 monclient: dump: 2024-03-27T21:41:38.286 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:38.286 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:38.286 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:38.287 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:38.287 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:38.287 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:38.287 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:38.287 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:38.287 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:38.287 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:38.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.279+0000 7f3868ac7640 4 set_mon_vals no callback set 2024-03-27T21:41:38.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.280+0000 7f386c52e640 5 monclient: authenticate success, global_id 4865 2024-03-27T21:41:38.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.280+0000 7f386c52e640 10 monclient: _renew_subs 2024-03-27T21:41:38.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.280+0000 7f386c52e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:38.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.280+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f385c009a70 con 0x7f386413e630 2024-03-27T21:41:38.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.280+0000 7f386c52e640 10 client.4865.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:38.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.280+0000 7f386c52e640 10 monclient: _renew_subs 2024-03-27T21:41:38.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.280+0000 7f386c52e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:38.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.280+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38641399f0 con 0x7f386413e630 2024-03-27T21:41:38.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.280+0000 7f386c52e640 1 librados: init done 2024-03-27T21:41:38.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.281+0000 7f3858ff9640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:41:38.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.281+0000 7f3858ff9640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:38.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.281+0000 7f3858ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:38.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.281+0000 7f3858ff9640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f3840005020 con 0x7f386413e630 2024-03-27T21:41:38.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.282+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f38540135d0 con 0x7f386413e630 2024-03-27T21:41:38.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.283+0000 7f385affd640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f384806c590 0x7f384806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:38.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.283+0000 7f386a2ca640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f384806c590 0x7f384806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:38.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.283+0000 7f386a2ca640 10 monclient: get_auth_request con 0x7f384806c590 auth_method 0 2024-03-27T21:41:38.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.283+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f385408ad60 con 0x7f386413e630 2024-03-27T21:41:38.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.283+0000 7f385affd640 10 client.4865.objecter ms_dispatch 0x7f3864000b90 osd_map(71..71 src has 1..71) v4 2024-03-27T21:41:38.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.283+0000 7f385affd640 3 client.4865.objecter handle_osd_map got epochs [71,71] > 0 2024-03-27T21:41:38.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.283+0000 7f385affd640 3 client.4865.objecter handle_osd_map decoding full epoch 71 2024-03-27T21:41:38.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.283+0000 7f385affd640 20 client.4865.objecter dump_active .. 0 homeless 2024-03-27T21:41:38.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.283+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v71) v1 ==== 54+0+16 (secure 0 0 0) 0x7f385408e060 con 0x7f386413e630 2024-03-27T21:41:38.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.283+0000 7f385affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:38.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.283+0000 7f385affd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:38.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.284+0000 7f385affd640 10 client.4865.objecter ms_handle_connect 0x7f384806c590 2024-03-27T21:41:38.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.284+0000 7f386a2ca640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f384806c590 0x7f384806ea50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3860007fd0 tx=0x7f3860007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:38.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.284+0000 7f386c52e640 10 client.4865.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:38.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.284+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign -1 1 2024-03-27T21:41:38.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.284+0000 7f386c52e640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f3840005910 0x7f3840007d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:38.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.284+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840001dd0 osd=0 3 2024-03-27T21:41:38.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.284+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840001dd0 osd=0 3 2024-03-27T21:41:38.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.284+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840001dd0 osd=0 3 2024-03-27T21:41:38.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.285+0000 7f386aacb640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f3840005910 0x7f3840007d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:38.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.285+0000 7f386c52e640 15 client.4865.objecter _session_command_op_remove -1 1 2024-03-27T21:41:38.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.285+0000 7f386c52e640 20 client.4865.objecter get_session s=0x7f3840001dd0 osd=0 3 2024-03-27T21:41:38.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.285+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign 0 1 2024-03-27T21:41:38.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.285+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840001dd0 osd=0 4 2024-03-27T21:41:38.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.285+0000 7f386c52e640 10 client.4865.objecter _send_command 1 2024-03-27T21:41:38.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.285+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f384000aad0 con 0x7f3840005910 2024-03-27T21:41:38.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.285+0000 7f386aacb640 10 monclient: get_auth_request con 0x7f3840005910 auth_method 0 2024-03-27T21:41:38.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.285+0000 7f386aacb640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f3840005910 0x7f3840007d00 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:38.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.285+0000 7f385affd640 10 client.4865.objecter ms_handle_connect 0x7f3840005910 2024-03-27T21:41:38.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.286+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f384000aad0 con 0x7f3840005910 2024-03-27T21:41:38.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.286+0000 7f385affd640 10 client.4865.objecter ms_dispatch 0x7f3864000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:41:38.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.286+0000 7f385affd640 10 client.4865.objecter _finish_command 1 = system:0 2024-03-27T21:41:38.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.286+0000 7f385affd640 20 client.4865.objecter put_session s=0x7f3840001dd0 osd=0 4 2024-03-27T21:41:38.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.286+0000 7f385affd640 15 client.4865.objecter _session_command_op_remove 0 1 2024-03-27T21:41:38.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.302+0000 7f386c52e640 10 client.4865.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:38.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.302+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign -1 2 2024-03-27T21:41:38.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.302+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840001dd0 osd=0 3 2024-03-27T21:41:38.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.302+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840001dd0 osd=0 3 2024-03-27T21:41:38.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.302+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840001dd0 osd=0 3 2024-03-27T21:41:38.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.302+0000 7f386c52e640 15 client.4865.objecter _session_command_op_remove -1 2 2024-03-27T21:41:38.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.302+0000 7f386c52e640 20 client.4865.objecter get_session s=0x7f3840001dd0 osd=0 3 2024-03-27T21:41:38.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.302+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign 0 2 2024-03-27T21:41:38.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.302+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840001dd0 osd=0 4 2024-03-27T21:41:38.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.302+0000 7f386c52e640 10 client.4865.objecter _send_command 2 2024-03-27T21:41:38.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.303+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3840005560 con 0x7f3840005910 2024-03-27T21:41:38.292 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:38.305+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:38.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.305+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3840005560 con 0x7f3840005910 2024-03-27T21:41:38.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.305+0000 7f385affd640 10 client.4865.objecter ms_dispatch 0x7f3864000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:38.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.305+0000 7f385affd640 10 client.4865.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:38.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.305+0000 7f385affd640 20 client.4865.objecter put_session s=0x7f3840001dd0 osd=0 4 2024-03-27T21:41:38.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.305+0000 7f385affd640 15 client.4865.objecter _session_command_op_remove 0 2 2024-03-27T21:41:38.293 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:41:38.295 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:41:38.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.305+0000 7f386c52e640 10 client.4865.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:38.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.305+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign -1 3 2024-03-27T21:41:38.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.305+0000 7f386c52e640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f3840004b50 0x7f38400093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:38.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.305+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840001a90 osd=1 3 2024-03-27T21:41:38.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.305+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840001a90 osd=1 3 2024-03-27T21:41:38.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.305+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840001a90 osd=1 3 2024-03-27T21:41:38.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.305+0000 7f386c52e640 15 client.4865.objecter _session_command_op_remove -1 3 2024-03-27T21:41:38.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.305+0000 7f386c52e640 20 client.4865.objecter get_session s=0x7f3840001a90 osd=1 3 2024-03-27T21:41:38.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.305+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign 1 3 2024-03-27T21:41:38.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.305+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840001a90 osd=1 4 2024-03-27T21:41:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.305+0000 7f386c52e640 10 client.4865.objecter _send_command 3 2024-03-27T21:41:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.305+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f384000e360 con 0x7f3840004b50 2024-03-27T21:41:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.306+0000 7f386a2ca640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f3840004b50 0x7f38400093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.306+0000 7f386a2ca640 10 monclient: get_auth_request con 0x7f3840004b50 auth_method 0 2024-03-27T21:41:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.306+0000 7f385affd640 10 client.4865.objecter ms_handle_connect 0x7f3840004b50 2024-03-27T21:41:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.306+0000 7f386a2ca640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f3840004b50 0x7f38400093c0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.307+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f384000e360 con 0x7f3840004b50 2024-03-27T21:41:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.307+0000 7f385affd640 10 client.4865.objecter ms_dispatch 0x7f3864000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:41:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.307+0000 7f385affd640 10 client.4865.objecter _finish_command 3 = system:0 2024-03-27T21:41:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.307+0000 7f385affd640 20 client.4865.objecter put_session s=0x7f3840001a90 osd=1 4 2024-03-27T21:41:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.307+0000 7f385affd640 15 client.4865.objecter _session_command_op_remove 1 3 2024-03-27T21:41:38.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.321+0000 7f386c52e640 10 client.4865.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:38.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.321+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign -1 4 2024-03-27T21:41:38.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.321+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840001a90 osd=1 3 2024-03-27T21:41:38.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.321+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840001a90 osd=1 3 2024-03-27T21:41:38.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.321+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840001a90 osd=1 3 2024-03-27T21:41:38.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.321+0000 7f386c52e640 15 client.4865.objecter _session_command_op_remove -1 4 2024-03-27T21:41:38.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.321+0000 7f386c52e640 20 client.4865.objecter get_session s=0x7f3840001a90 osd=1 3 2024-03-27T21:41:38.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.321+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign 1 4 2024-03-27T21:41:38.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.321+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840001a90 osd=1 4 2024-03-27T21:41:38.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.321+0000 7f386c52e640 10 client.4865.objecter _send_command 4 2024-03-27T21:41:38.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.321+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3840004180 con 0x7f3840004b50 2024-03-27T21:41:38.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.323+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3840004180 con 0x7f3840004b50 2024-03-27T21:41:38.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.323+0000 7f385affd640 10 client.4865.objecter ms_dispatch 0x7f3864000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:38.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.323+0000 7f385affd640 10 client.4865.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:38.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.323+0000 7f385affd640 20 client.4865.objecter put_session s=0x7f3840001a90 osd=1 4 2024-03-27T21:41:38.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.323+0000 7f385affd640 15 client.4865.objecter _session_command_op_remove 1 4 2024-03-27T21:41:38.311 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:41:38.312 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:41:38.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.323+0000 7f386c52e640 10 client.4865.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:38.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.323+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign -1 5 2024-03-27T21:41:38.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.323+0000 7f386c52e640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f384000cc50 0x7f384000d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:38.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.323+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840008a00 osd=2 3 2024-03-27T21:41:38.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.323+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840008a00 osd=2 3 2024-03-27T21:41:38.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.323+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840008a00 osd=2 3 2024-03-27T21:41:38.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.323+0000 7f386c52e640 15 client.4865.objecter _session_command_op_remove -1 5 2024-03-27T21:41:38.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.323+0000 7f386c52e640 20 client.4865.objecter get_session s=0x7f3840008a00 osd=2 3 2024-03-27T21:41:38.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.323+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign 2 5 2024-03-27T21:41:38.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.323+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840008a00 osd=2 4 2024-03-27T21:41:38.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.323+0000 7f386c52e640 10 client.4865.objecter _send_command 5 2024-03-27T21:41:38.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.323+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f384000d5f0 con 0x7f384000cc50 2024-03-27T21:41:38.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.324+0000 7f3869ac9640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f384000cc50 0x7f384000d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:38.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.324+0000 7f3869ac9640 10 monclient: get_auth_request con 0x7f384000cc50 auth_method 0 2024-03-27T21:41:38.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.324+0000 7f385affd640 10 client.4865.objecter ms_handle_connect 0x7f384000cc50 2024-03-27T21:41:38.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.324+0000 7f3869ac9640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f384000cc50 0x7f384000d0b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:38.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.325+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== osd.2 v2:172.21.15.60:6824/56434 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f384000d5f0 con 0x7f384000cc50 2024-03-27T21:41:38.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.325+0000 7f385affd640 10 client.4865.objecter ms_dispatch 0x7f3864000b90 command_reply(tid 5: 0 ) v1 2024-03-27T21:41:38.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.325+0000 7f385affd640 10 client.4865.objecter _finish_command 5 = system:0 2024-03-27T21:41:38.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.325+0000 7f385affd640 20 client.4865.objecter put_session s=0x7f3840008a00 osd=2 4 2024-03-27T21:41:38.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.325+0000 7f385affd640 15 client.4865.objecter _session_command_op_remove 2 5 2024-03-27T21:41:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.339+0000 7f386c52e640 10 client.4865.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.339+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign -1 6 2024-03-27T21:41:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.339+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840008a00 osd=2 3 2024-03-27T21:41:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.339+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840008a00 osd=2 3 2024-03-27T21:41:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.339+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840008a00 osd=2 3 2024-03-27T21:41:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.339+0000 7f386c52e640 15 client.4865.objecter _session_command_op_remove -1 6 2024-03-27T21:41:38.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.339+0000 7f386c52e640 20 client.4865.objecter get_session s=0x7f3840008a00 osd=2 3 2024-03-27T21:41:38.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.339+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign 2 6 2024-03-27T21:41:38.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.339+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840008a00 osd=2 4 2024-03-27T21:41:38.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.339+0000 7f386c52e640 10 client.4865.objecter _send_command 6 2024-03-27T21:41:38.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.339+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f384000dfb0 con 0x7f384000cc50 2024-03-27T21:41:38.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.340+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== osd.2 v2:172.21.15.60:6824/56434 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f384000dfb0 con 0x7f384000cc50 2024-03-27T21:41:38.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.340+0000 7f385affd640 10 client.4865.objecter ms_dispatch 0x7f3864000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:38.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.340+0000 7f385affd640 10 client.4865.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:38.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.340+0000 7f385affd640 20 client.4865.objecter put_session s=0x7f3840008a00 osd=2 4 2024-03-27T21:41:38.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.340+0000 7f385affd640 15 client.4865.objecter _session_command_op_remove 2 6 2024-03-27T21:41:38.328 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2: {} 2024-03-27T21:41:38.330 INFO:teuthology.orchestra.run.smithi060.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-27T21:41:38.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.341+0000 7f386c52e640 10 client.4865.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:38.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.341+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign -1 7 2024-03-27T21:41:38.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.341+0000 7f386c52e640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f38400085f0 0x7f3840012a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:38.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.341+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840008480 osd=3 3 2024-03-27T21:41:38.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.341+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840008480 osd=3 3 2024-03-27T21:41:38.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.341+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840008480 osd=3 3 2024-03-27T21:41:38.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.341+0000 7f386c52e640 15 client.4865.objecter _session_command_op_remove -1 7 2024-03-27T21:41:38.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.341+0000 7f386c52e640 20 client.4865.objecter get_session s=0x7f3840008480 osd=3 3 2024-03-27T21:41:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.341+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign 3 7 2024-03-27T21:41:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.341+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840008480 osd=3 4 2024-03-27T21:41:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.341+0000 7f386c52e640 10 client.4865.objecter _send_command 7 2024-03-27T21:41:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.341+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3840015980 con 0x7f38400085f0 2024-03-27T21:41:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.341+0000 7f386aacb640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f38400085f0 0x7f3840012a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.341+0000 7f386aacb640 10 monclient: get_auth_request con 0x7f38400085f0 auth_method 0 2024-03-27T21:41:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.341+0000 7f385affd640 10 client.4865.objecter ms_handle_connect 0x7f38400085f0 2024-03-27T21:41:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.341+0000 7f386aacb640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f38400085f0 0x7f3840012a30 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.342+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f3840015980 con 0x7f38400085f0 2024-03-27T21:41:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.342+0000 7f385affd640 10 client.4865.objecter ms_dispatch 0x7f3864000b90 command_reply(tid 7: 0 ) v1 2024-03-27T21:41:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.342+0000 7f385affd640 10 client.4865.objecter _finish_command 7 = system:0 2024-03-27T21:41:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.342+0000 7f385affd640 20 client.4865.objecter put_session s=0x7f3840008480 osd=3 4 2024-03-27T21:41:38.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.342+0000 7f385affd640 15 client.4865.objecter _session_command_op_remove 3 7 2024-03-27T21:41:38.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.370+0000 7f386c52e640 10 client.4865.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:38.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.370+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign -1 8 2024-03-27T21:41:38.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.370+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840008480 osd=3 3 2024-03-27T21:41:38.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.370+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840008480 osd=3 3 2024-03-27T21:41:38.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.370+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840008480 osd=3 3 2024-03-27T21:41:38.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.370+0000 7f386c52e640 15 client.4865.objecter _session_command_op_remove -1 8 2024-03-27T21:41:38.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.370+0000 7f386c52e640 20 client.4865.objecter get_session s=0x7f3840008480 osd=3 3 2024-03-27T21:41:38.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.370+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign 3 8 2024-03-27T21:41:38.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.370+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840008480 osd=3 4 2024-03-27T21:41:38.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.370+0000 7f386c52e640 10 client.4865.objecter _send_command 8 2024-03-27T21:41:38.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.370+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f384000d9a0 con 0x7f38400085f0 2024-03-27T21:41:38.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.371+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f384000d9a0 con 0x7f38400085f0 2024-03-27T21:41:38.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.371+0000 7f385affd640 10 client.4865.objecter ms_dispatch 0x7f3864000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:38.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.371+0000 7f385affd640 10 client.4865.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:38.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.371+0000 7f385affd640 20 client.4865.objecter put_session s=0x7f3840008480 osd=3 4 2024-03-27T21:41:38.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.371+0000 7f385affd640 15 client.4865.objecter _session_command_op_remove 3 8 2024-03-27T21:41:38.359 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:41:38.360 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:41:38.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.371+0000 7f386c52e640 10 client.4865.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:38.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.371+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign -1 9 2024-03-27T21:41:38.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.371+0000 7f386c52e640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f384000a5d0 0x7f3840013010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:38.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.371+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f384000d760 osd=4 3 2024-03-27T21:41:38.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.372+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f384000d760 osd=4 3 2024-03-27T21:41:38.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.372+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f384000d760 osd=4 3 2024-03-27T21:41:38.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.372+0000 7f386c52e640 15 client.4865.objecter _session_command_op_remove -1 9 2024-03-27T21:41:38.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.372+0000 7f386c52e640 20 client.4865.objecter get_session s=0x7f384000d760 osd=4 3 2024-03-27T21:41:38.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.372+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign 4 9 2024-03-27T21:41:38.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.372+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f384000d760 osd=4 4 2024-03-27T21:41:38.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.372+0000 7f386c52e640 10 client.4865.objecter _send_command 9 2024-03-27T21:41:38.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.372+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f384000d990 con 0x7f384000a5d0 2024-03-27T21:41:38.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.372+0000 7f386a2ca640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f384000a5d0 0x7f3840013010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:38.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.372+0000 7f386a2ca640 10 monclient: get_auth_request con 0x7f384000a5d0 auth_method 0 2024-03-27T21:41:38.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.372+0000 7f386a2ca640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f384000a5d0 0x7f3840013010 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:38.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.372+0000 7f385affd640 10 client.4865.objecter ms_handle_connect 0x7f384000a5d0 2024-03-27T21:41:38.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.373+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f38600018b0 con 0x7f384000a5d0 2024-03-27T21:41:38.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.373+0000 7f385affd640 10 client.4865.objecter ms_dispatch 0x7f3864000b90 command_reply(tid 9: 0 ) v1 2024-03-27T21:41:38.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.373+0000 7f385affd640 10 client.4865.objecter _finish_command 9 = system:0 2024-03-27T21:41:38.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.373+0000 7f385affd640 20 client.4865.objecter put_session s=0x7f384000d760 osd=4 4 2024-03-27T21:41:38.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.373+0000 7f385affd640 15 client.4865.objecter _session_command_op_remove 4 9 2024-03-27T21:41:38.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.387+0000 7f386c52e640 10 client.4865.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:38.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.387+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign -1 10 2024-03-27T21:41:38.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.387+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f384000d760 osd=4 3 2024-03-27T21:41:38.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.387+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f384000d760 osd=4 3 2024-03-27T21:41:38.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.387+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f384000d760 osd=4 3 2024-03-27T21:41:38.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.387+0000 7f386c52e640 15 client.4865.objecter _session_command_op_remove -1 10 2024-03-27T21:41:38.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.387+0000 7f386c52e640 20 client.4865.objecter get_session s=0x7f384000d760 osd=4 3 2024-03-27T21:41:38.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.387+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign 4 10 2024-03-27T21:41:38.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.387+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f384000d760 osd=4 4 2024-03-27T21:41:38.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.387+0000 7f386c52e640 10 client.4865.objecter _send_command 10 2024-03-27T21:41:38.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.387+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3840013550 con 0x7f384000a5d0 2024-03-27T21:41:38.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.389+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3840013550 con 0x7f384000a5d0 2024-03-27T21:41:38.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.389+0000 7f385affd640 10 client.4865.objecter ms_dispatch 0x7f3864000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:38.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.389+0000 7f385affd640 10 client.4865.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:38.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.389+0000 7f385affd640 20 client.4865.objecter put_session s=0x7f384000d760 osd=4 4 2024-03-27T21:41:38.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.389+0000 7f385affd640 15 client.4865.objecter _session_command_op_remove 4 10 2024-03-27T21:41:38.377 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:41:38.379 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:41:38.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.390+0000 7f386c52e640 10 client.4865.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:38.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.390+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign -1 11 2024-03-27T21:41:38.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.390+0000 7f386c52e640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f3840013c80 0x7f3840014100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:38.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.390+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840013550 osd=5 3 2024-03-27T21:41:38.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.390+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840013550 osd=5 3 2024-03-27T21:41:38.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.390+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840013550 osd=5 3 2024-03-27T21:41:38.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.390+0000 7f386c52e640 15 client.4865.objecter _session_command_op_remove -1 11 2024-03-27T21:41:38.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.390+0000 7f386c52e640 20 client.4865.objecter get_session s=0x7f3840013550 osd=5 3 2024-03-27T21:41:38.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.390+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign 5 11 2024-03-27T21:41:38.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.390+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840013550 osd=5 4 2024-03-27T21:41:38.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.390+0000 7f386c52e640 10 client.4865.objecter _send_command 11 2024-03-27T21:41:38.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.390+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f384001c580 con 0x7f3840013c80 2024-03-27T21:41:38.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.390+0000 7f3869ac9640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f3840013c80 0x7f3840014100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:38.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.390+0000 7f3869ac9640 10 monclient: get_auth_request con 0x7f3840013c80 auth_method 0 2024-03-27T21:41:38.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.390+0000 7f385affd640 10 client.4865.objecter ms_handle_connect 0x7f3840013c80 2024-03-27T21:41:38.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.390+0000 7f3869ac9640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f3840013c80 0x7f3840014100 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:38.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.391+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f384001c580 con 0x7f3840013c80 2024-03-27T21:41:38.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.391+0000 7f385affd640 10 client.4865.objecter ms_dispatch 0x7f3864000b90 command_reply(tid 11: 0 ) v1 2024-03-27T21:41:38.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.391+0000 7f385affd640 10 client.4865.objecter _finish_command 11 = system:0 2024-03-27T21:41:38.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.391+0000 7f385affd640 20 client.4865.objecter put_session s=0x7f3840013550 osd=5 4 2024-03-27T21:41:38.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.391+0000 7f385affd640 15 client.4865.objecter _session_command_op_remove 5 11 2024-03-27T21:41:38.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.405+0000 7f386c52e640 10 client.4865.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:38.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.405+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign -1 12 2024-03-27T21:41:38.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.405+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840013550 osd=5 3 2024-03-27T21:41:38.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.405+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840013550 osd=5 3 2024-03-27T21:41:38.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.405+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840013550 osd=5 3 2024-03-27T21:41:38.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.405+0000 7f386c52e640 15 client.4865.objecter _session_command_op_remove -1 12 2024-03-27T21:41:38.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.405+0000 7f386c52e640 20 client.4865.objecter get_session s=0x7f3840013550 osd=5 3 2024-03-27T21:41:38.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.405+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign 5 12 2024-03-27T21:41:38.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.405+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840013550 osd=5 4 2024-03-27T21:41:38.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.405+0000 7f386c52e640 10 client.4865.objecter _send_command 12 2024-03-27T21:41:38.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.405+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f384001c580 con 0x7f3840013c80 2024-03-27T21:41:38.393 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:38.406+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:38.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.407+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f384001c580 con 0x7f3840013c80 2024-03-27T21:41:38.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.407+0000 7f385affd640 10 client.4865.objecter ms_dispatch 0x7f3864000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:38.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.407+0000 7f385affd640 10 client.4865.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:38.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.407+0000 7f385affd640 20 client.4865.objecter put_session s=0x7f3840013550 osd=5 4 2024-03-27T21:41:38.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.407+0000 7f385affd640 15 client.4865.objecter _session_command_op_remove 5 12 2024-03-27T21:41:38.395 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:41:38.395 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:41:38.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.407+0000 7f386c52e640 10 client.4865.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:38.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.407+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign -1 13 2024-03-27T21:41:38.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.407+0000 7f386c52e640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f384001b8b0 0x7f384001bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:38.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.407+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f384001c580 osd=6 3 2024-03-27T21:41:38.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.407+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f384001c580 osd=6 3 2024-03-27T21:41:38.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.407+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f384001c580 osd=6 3 2024-03-27T21:41:38.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.407+0000 7f386c52e640 15 client.4865.objecter _session_command_op_remove -1 13 2024-03-27T21:41:38.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.407+0000 7f386c52e640 20 client.4865.objecter get_session s=0x7f384001c580 osd=6 3 2024-03-27T21:41:38.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.407+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign 6 13 2024-03-27T21:41:38.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.407+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f384001c580 osd=6 4 2024-03-27T21:41:38.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.407+0000 7f386c52e640 10 client.4865.objecter _send_command 13 2024-03-27T21:41:38.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.407+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f384001f940 con 0x7f384001b8b0 2024-03-27T21:41:38.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.408+0000 7f386aacb640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f384001b8b0 0x7f384001bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:38.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.408+0000 7f386aacb640 10 monclient: get_auth_request con 0x7f384001b8b0 auth_method 0 2024-03-27T21:41:38.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.408+0000 7f385affd640 10 client.4865.objecter ms_handle_connect 0x7f384001b8b0 2024-03-27T21:41:38.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.408+0000 7f386aacb640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f384001b8b0 0x7f384001bd10 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:38.397 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:38.397 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:38.397 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:41:38.397 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:41:38.397 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:38.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.409+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f384001f940 con 0x7f384001b8b0 2024-03-27T21:41:38.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.409+0000 7f385affd640 10 client.4865.objecter ms_dispatch 0x7f3864000b90 command_reply(tid 13: 0 ) v1 2024-03-27T21:41:38.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.409+0000 7f385affd640 10 client.4865.objecter _finish_command 13 = system:0 2024-03-27T21:41:38.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.409+0000 7f385affd640 20 client.4865.objecter put_session s=0x7f384001c580 osd=6 4 2024-03-27T21:41:38.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.409+0000 7f385affd640 15 client.4865.objecter _session_command_op_remove 6 13 2024-03-27T21:41:38.406 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:41:38.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.423+0000 7f386c52e640 10 client.4865.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:38.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.423+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign -1 14 2024-03-27T21:41:38.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.423+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f384001c580 osd=6 3 2024-03-27T21:41:38.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.423+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f384001c580 osd=6 3 2024-03-27T21:41:38.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.423+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f384001c580 osd=6 3 2024-03-27T21:41:38.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.423+0000 7f386c52e640 15 client.4865.objecter _session_command_op_remove -1 14 2024-03-27T21:41:38.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.423+0000 7f386c52e640 20 client.4865.objecter get_session s=0x7f384001c580 osd=6 3 2024-03-27T21:41:38.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.423+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign 6 14 2024-03-27T21:41:38.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.423+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f384001c580 osd=6 4 2024-03-27T21:41:38.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.423+0000 7f386c52e640 10 client.4865.objecter _send_command 14 2024-03-27T21:41:38.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.423+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3840019c10 con 0x7f384001b8b0 2024-03-27T21:41:38.413 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:41:38.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.425+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f3840019c10 con 0x7f384001b8b0 2024-03-27T21:41:38.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.425+0000 7f385affd640 10 client.4865.objecter ms_dispatch 0x7f3864000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:38.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.425+0000 7f385affd640 10 client.4865.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:38.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.425+0000 7f385affd640 20 client.4865.objecter put_session s=0x7f384001c580 osd=6 4 2024-03-27T21:41:38.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.425+0000 7f385affd640 15 client.4865.objecter _session_command_op_remove 6 14 2024-03-27T21:41:38.414 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:41:38.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.425+0000 7f386c52e640 10 client.4865.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:38.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.425+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign -1 15 2024-03-27T21:41:38.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.425+0000 7f386c52e640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f384001a2f0 0x7f384001a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:38.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.425+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840019c10 osd=7 3 2024-03-27T21:41:38.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.425+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840019c10 osd=7 3 2024-03-27T21:41:38.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.425+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840019c10 osd=7 3 2024-03-27T21:41:38.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.425+0000 7f386c52e640 15 client.4865.objecter _session_command_op_remove -1 15 2024-03-27T21:41:38.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.425+0000 7f386c52e640 20 client.4865.objecter get_session s=0x7f3840019c10 osd=7 3 2024-03-27T21:41:38.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.425+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign 7 15 2024-03-27T21:41:38.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.425+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840019c10 osd=7 4 2024-03-27T21:41:38.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.425+0000 7f386c52e640 10 client.4865.objecter _send_command 15 2024-03-27T21:41:38.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.425+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f384001ae10 con 0x7f384001a2f0 2024-03-27T21:41:38.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.425+0000 7f386a2ca640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f384001a2f0 0x7f384001a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:38.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.425+0000 7f386a2ca640 10 monclient: get_auth_request con 0x7f384001a2f0 auth_method 0 2024-03-27T21:41:38.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.426+0000 7f385affd640 10 client.4865.objecter ms_handle_connect 0x7f384001a2f0 2024-03-27T21:41:38.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.426+0000 7f386a2ca640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f384001a2f0 0x7f384001a730 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:38.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.426+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f384001ae10 con 0x7f384001a2f0 2024-03-27T21:41:38.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.426+0000 7f385affd640 10 client.4865.objecter ms_dispatch 0x7f3864000b90 command_reply(tid 15: 0 ) v1 2024-03-27T21:41:38.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.426+0000 7f385affd640 10 client.4865.objecter _finish_command 15 = system:0 2024-03-27T21:41:38.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.426+0000 7f385affd640 20 client.4865.objecter put_session s=0x7f3840019c10 osd=7 4 2024-03-27T21:41:38.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.426+0000 7f385affd640 15 client.4865.objecter _session_command_op_remove 7 15 2024-03-27T21:41:38.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.440+0000 7f386c52e640 10 client.4865.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:38.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.440+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign -1 16 2024-03-27T21:41:38.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.440+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840019c10 osd=7 3 2024-03-27T21:41:38.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.440+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840019c10 osd=7 3 2024-03-27T21:41:38.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.440+0000 7f386c52e640 20 client.4865.objecter _get_session s=0x7f3840019c10 osd=7 3 2024-03-27T21:41:38.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.440+0000 7f386c52e640 15 client.4865.objecter _session_command_op_remove -1 16 2024-03-27T21:41:38.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.440+0000 7f386c52e640 20 client.4865.objecter get_session s=0x7f3840019c10 osd=7 3 2024-03-27T21:41:38.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.440+0000 7f386c52e640 15 client.4865.objecter _session_command_op_assign 7 16 2024-03-27T21:41:38.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.440+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840019c10 osd=7 4 2024-03-27T21:41:38.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.440+0000 7f386c52e640 10 client.4865.objecter _send_command 16 2024-03-27T21:41:38.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.440+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f384001ae10 con 0x7f384001a2f0 2024-03-27T21:41:38.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f385affd640 1 -- 172.21.15.60:0/2099221919 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f384001ae10 con 0x7f384001a2f0 2024-03-27T21:41:38.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f385affd640 10 client.4865.objecter ms_dispatch 0x7f3864000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:38.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f385affd640 10 client.4865.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:38.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f385affd640 20 client.4865.objecter put_session s=0x7f3840019c10 osd=7 4 2024-03-27T21:41:38.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f385affd640 15 client.4865.objecter _session_command_op_remove 7 16 2024-03-27T21:41:38.430 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:41:38.432 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:41:38.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 10 librados: watch_flush enter 2024-03-27T21:41:38.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 10 librados: watch_flush exit 2024-03-27T21:41:38.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 10 client.4865.objecter close_session for osd.0 2024-03-27T21:41:38.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f3840005910 msgr2=0x7f3840007d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:38.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f3840005910 0x7f3840007d00 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:38.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840001dd0 osd=0 1 2024-03-27T21:41:38.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 10 client.4865.objecter close_session for osd.1 2024-03-27T21:41:38.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f3840004b50 msgr2=0x7f38400093c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:38.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f3840004b50 0x7f38400093c0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:38.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840001a90 osd=1 1 2024-03-27T21:41:38.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 10 client.4865.objecter close_session for osd.2 2024-03-27T21:41:38.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f384000cc50 msgr2=0x7f384000d0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:38.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f384000cc50 0x7f384000d0b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:38.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386aacb640 1 -- 172.21.15.60:0/2099221919 reap_dead start 2024-03-27T21:41:38.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840008a00 osd=2 1 2024-03-27T21:41:38.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 10 client.4865.objecter close_session for osd.3 2024-03-27T21:41:38.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f38400085f0 msgr2=0x7f3840012a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:38.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f38400085f0 0x7f3840012a30 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:38.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840008480 osd=3 1 2024-03-27T21:41:38.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 10 client.4865.objecter close_session for osd.4 2024-03-27T21:41:38.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f384000a5d0 msgr2=0x7f3840013010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:38.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f384000a5d0 0x7f3840013010 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:38.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f384000d760 osd=4 1 2024-03-27T21:41:38.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 10 client.4865.objecter close_session for osd.5 2024-03-27T21:41:38.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f3840013c80 msgr2=0x7f3840014100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:38.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f3840013c80 0x7f3840014100 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:38.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840013550 osd=5 1 2024-03-27T21:41:38.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 10 client.4865.objecter close_session for osd.6 2024-03-27T21:41:38.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f384001b8b0 msgr2=0x7f384001bd10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:38.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f384001b8b0 0x7f384001bd10 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:38.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f384001c580 osd=6 1 2024-03-27T21:41:38.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 10 client.4865.objecter close_session for osd.7 2024-03-27T21:41:38.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f384001a2f0 msgr2=0x7f384001a730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:38.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f384001a2f0 0x7f384001a730 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:38.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 20 client.4865.objecter put_session s=0x7f3840019c10 osd=7 1 2024-03-27T21:41:38.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 20 client.4865.objecter shutdown clearing up homeless session... 2024-03-27T21:41:38.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 10 client.4865.objecter successfully canceled tick 2024-03-27T21:41:38.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386aacb640 1 -- 172.21.15.60:0/2099221919 reap_dead start 2024-03-27T21:41:38.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f384806c590 msgr2=0x7f384806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:38.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f384806c590 0x7f384806ea50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3860007fd0 tx=0x7f3860007e80 comp rx=0 tx=0).stop 2024-03-27T21:41:38.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 10 monclient: shutdown 2024-03-27T21:41:38.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:38.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f386413e630 msgr2=0x7f38641d67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:38.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f386413e630 0x7f38641d67d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f385400e9d0 tx=0x7f385400ee90 comp rx=0 tx=0).stop 2024-03-27T21:41:38.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 shutdown_connections 2024-03-27T21:41:38.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f384806c590 0x7f384806ea50 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:38.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 --2- 172.21.15.60:0/2099221919 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f386413e630 0x7f38641d67d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:38.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 >> 172.21.15.60:0/2099221919 conn(0x7f38640b2ea0 msgr2=0x7f38640aff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:38.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 shutdown_connections 2024-03-27T21:41:38.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 -- 172.21.15.60:0/2099221919 wait complete. 2024-03-27T21:41:38.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:38.444+0000 7f386c52e640 1 librados: shutdown 2024-03-27T21:41:38.493 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:38.506+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:38.568 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:38.568 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:41:38.569 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:41:38.569 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:41:38.569 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:38.581 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:41:38.622 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:38.635+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:38.694 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:38.708+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:38.697 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:38.697 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:38.697 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:38.697 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:38.706 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:41:38.795 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:38.809+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:38.811 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:38.811 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:38.811 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:38.812 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:38.812 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:38.820 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:41:38.904 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:38.918+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:38.965 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:38.965 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:38.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:38.965 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:41:38.965 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:38.973 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:41:39.014 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:39.028+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:39.098 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:39.111+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:39.120 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:39.120 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:39.120 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:39.120 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:39.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:41:39.214 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:39.228+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:39.275 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:39.275 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:39.275 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:39.275 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:39.275 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:39.284 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:41:39.325 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:39.339+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:39.401 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:39.415+0000 7fb850f34640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 56426) UID: 0 2024-03-27T21:41:39.429 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:39.429 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:39.429 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:39.430 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:41:39.430 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:39.438 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:41:39.501 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:39.515+0000 7fb850f34640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 56426) UID: 0 2024-03-27T21:41:39.542 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:39.543 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:39.543 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:39.543 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:39.551 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:41:39.603 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:39.617+0000 7fb850f34640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 56426) UID: 0 2024-03-27T21:41:39.656 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:39.656 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:39.657 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:39.657 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:39.657 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:39.666 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:41:39.709 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:39.723+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:39.770 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:39.771 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:39.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:39.771 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:41:39.771 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:39.779 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:41:39.805 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:39.819+0000 7fb850f34640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 56426) UID: 0 2024-03-27T21:41:39.906 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:39.920+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:39.925 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:39.925 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:39.925 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:39.925 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:39.933 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:41:40.017 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:40.031+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:40.079 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:40.080 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:40.080 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:40.080 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:40.080 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:40.088 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:41:40.109 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-27T21:41:40.130 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:40.143+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:40.209 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:40.223+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:40.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.228+0000 7f9c42bf0640 10 monclient: get_monmap_and_config 2024-03-27T21:41:40.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.228+0000 7f9c42bf0640 10 monclient: build_initial_monmap 2024-03-27T21:41:40.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.228+0000 7f9c42bf0640 10 monclient: monmap: 2024-03-27T21:41:40.214 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:40.214 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:40.214 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:40.229349+0000 2024-03-27T21:41:40.214 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:40.229349+0000 2024-03-27T21:41:40.214 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:40.215 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:40.215 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:40.215 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:40.215 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:40.215 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:40.215 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:40.215 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:40.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.229+0000 7f9c42bf0640 1 Processor -- start 2024-03-27T21:41:40.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.229+0000 7f9c42bf0640 1 -- start start 2024-03-27T21:41:40.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.229+0000 7f9c42bf0640 10 monclient: init 2024-03-27T21:41:40.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.229+0000 7f9c42bf0640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:40.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.229+0000 7f9c42bf0640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-03-27T21:41:40.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.229+0000 7f9c42bf0640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9c3c071a50 0x7f9c3c0a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:40.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.229+0000 7f9c42bf0640 10 monclient(hunting): picked mon.noname-c con 0x7f9c3c071a50 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:40.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.229+0000 7f9c42bf0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9c3c13e810 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:40.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.229+0000 7f9c42bf0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9c3c13ec50 0x7f9c3c141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:40.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.229+0000 7f9c42bf0640 10 monclient(hunting): picked mon.noname-a con 0x7f9c3c13ec50 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:40.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.229+0000 7f9c42bf0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c3c0a8e90 con 0x7f9c3c13e810 2024-03-27T21:41:40.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.229+0000 7f9c42bf0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:40.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.229+0000 7f9c42bf0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c3c071e30 con 0x7f9c3c13ec50 2024-03-27T21:41:40.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.229+0000 7f9c42bf0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:40.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.229+0000 7f9c42bf0640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c3c0aa040 con 0x7f9c3c071a50 2024-03-27T21:41:40.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.229+0000 7f9c42bf0640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:40.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.229+0000 7f9c42bf0640 10 monclient(hunting): authenticate will time out at 941.450715s 2024-03-27T21:41:40.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.229+0000 7f9c4118d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9c3c13ec50 0x7f9c3c141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:40.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4118d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9c3c13ec50 0x7f9c3c141d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50940/0 (socket says 172.21.15.60:50940) 2024-03-27T21:41:40.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4118d640 1 -- 172.21.15.60:0/2630484060 learned_addr learned my addr 172.21.15.60:0/2630484060 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:40.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4118d640 10 monclient(hunting): get_auth_request con 0x7f9c3c13ec50 auth_method 0 2024-03-27T21:41:40.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4118d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:40.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4118d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:40.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4118d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:40.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4098c640 1 --2- 172.21.15.60:0/2630484060 >> v2:172.21.15.191:3300/0 conn(0x7f9c3c071a50 0x7f9c3c0a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:40.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4098c640 10 monclient(hunting): get_auth_request con 0x7f9c3c071a50 auth_method 0 2024-03-27T21:41:40.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4098c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:40.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4098c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:40.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4098c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:40.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4118d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:40.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4118d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:40.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4118d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:40.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4098c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:40.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4098c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:40.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4098c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:40.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c3b7fe640 1 -- 172.21.15.60:0/2630484060 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156936510 0 0) 0x7f9c3c0a8e90 con 0x7f9c3c13e810 2024-03-27T21:41:40.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c3b7fe640 10 monclient(hunting): _negotiate 2024-03-27T21:41:40.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:40.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:40.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c3b7fe640 10 monclient(hunting): my global_id is 5053 2024-03-27T21:41:40.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c3b7fe640 1 -- 172.21.15.60:0/2630484060 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c2c003520 con 0x7f9c3c13e810 2024-03-27T21:41:40.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4118d640 10 monclient(hunting): handle_auth_done global_id 5106 payload 274 2024-03-27T21:41:40.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4118d640 1 -- 172.21.15.60:0/2630484060 >> v2:172.21.15.191:3300/0 conn(0x7f9c3c071a50 msgr2=0x7f9c3c0a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:40.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4118d640 1 --2- 172.21.15.60:0/2630484060 >> v2:172.21.15.191:3300/0 conn(0x7f9c3c071a50 0x7f9c3c0a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4118d640 1 -- 172.21.15.60:0/2630484060 >> v1:172.21.15.191:6789/0 conn(0x7f9c3c13e810 legacy=0x7f9c3c141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:40.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4118d640 10 monclient: _finish_hunting 0 2024-03-27T21:41:40.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4118d640 1 monclient: found mon.noname-a 2024-03-27T21:41:40.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4118d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:40.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4118d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:40.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4118d640 1 -- 172.21.15.60:0/2630484060 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c3c142280 con 0x7f9c3c13ec50 2024-03-27T21:41:40.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.230+0000 7f9c4098c640 1 --2- 172.21.15.60:0/2630484060 >> v2:172.21.15.191:3300/0 conn(0x7f9c3c071a50 0x7f9c3c0a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:40.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.231+0000 7f9c4118d640 1 --2- 172.21.15.60:0/2630484060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c3c13ec50 0x7f9c3c141d40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c0b4a80 tx=0x7f9c3402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c31aa7db09a3168 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:40.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.231+0000 7f9c3b7fe640 1 -- 172.21.15.60:0/2630484060 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3402cb20 con 0x7f9c3c13ec50 2024-03-27T21:41:40.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.231+0000 7f9c3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:40.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.231+0000 7f9c3b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:40.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.231+0000 7f9c3b7fe640 10 monclient: dump: 2024-03-27T21:41:40.224 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:40.224 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:40.224 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:40.224 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:40.224 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:40.224 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:40.224 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:40.224 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:40.224 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:40.224 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:40.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.231+0000 7f9c3b7fe640 10 monclient: _finish_auth 0 2024-03-27T21:41:40.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.231+0000 7f9c3b7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:41:40.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.231+0000 7f9c3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:40.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.231+0000 7f9c42bf0640 5 monclient: authenticate success, global_id 5106 2024-03-27T21:41:40.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.231+0000 7f9c42bf0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:40.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.231+0000 7f9c3b7fe640 1 -- 172.21.15.60:0/2630484060 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c3402cc80 con 0x7f9c3c13ec50 2024-03-27T21:41:40.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.231+0000 7f9c3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:40.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.231+0000 7f9c42bf0640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:40.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.231+0000 7f9c3b7fe640 1 -- 172.21.15.60:0/2630484060 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c340343f0 con 0x7f9c3c13ec50 2024-03-27T21:41:40.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.231+0000 7f9c42bf0640 4 set_mon_vals no callback set 2024-03-27T21:41:40.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.231+0000 7f9c3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:40.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.231+0000 7f9c3b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:40.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.231+0000 7f9c3b7fe640 10 monclient: dump: 2024-03-27T21:41:40.225 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:40.225 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:40.225 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:40.225 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:40.225 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:40.225 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:40.226 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:40.226 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:40.226 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:40.226 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:40.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.232+0000 7f9c42bf0640 10 monclient: shutdown 2024-03-27T21:41:40.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.232+0000 7f9c42bf0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:40.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.232+0000 7f9c42bf0640 1 -- 172.21.15.60:0/2630484060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c3c13ec50 msgr2=0x7f9c3c141d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:40.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.232+0000 7f9c42bf0640 1 --2- 172.21.15.60:0/2630484060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c3c13ec50 0x7f9c3c141d40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c0b4a80 tx=0x7f9c3402bf80 comp rx=0 tx=0).stop 2024-03-27T21:41:40.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.232+0000 7f9c42bf0640 1 -- 172.21.15.60:0/2630484060 shutdown_connections 2024-03-27T21:41:40.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.232+0000 7f9c42bf0640 1 --2- 172.21.15.60:0/2630484060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c3c13ec50 0x7f9c3c141d40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.232+0000 7f9c42bf0640 1 --2- 172.21.15.60:0/2630484060 >> v2:172.21.15.191:3300/0 conn(0x7f9c3c071a50 0x7f9c3c0a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.232+0000 7f9c42bf0640 1 -- 172.21.15.60:0/2630484060 >> 172.21.15.60:0/2630484060 conn(0x7f9c3c0b7300 msgr2=0x7f9c3c0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:40.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.232+0000 7f9c42bf0640 1 -- 172.21.15.60:0/2630484060 shutdown_connections 2024-03-27T21:41:40.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.232+0000 7f9c42bf0640 1 -- 172.21.15.60:0/2630484060 wait complete. 2024-03-27T21:41:40.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.233+0000 7f9c42bf0640 10 monclient: build_initial_monmap 2024-03-27T21:41:40.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.233+0000 7f9c42bf0640 10 monclient: monmap: 2024-03-27T21:41:40.227 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:40.227 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:40.227 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:40.227 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:40.227 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:40.227 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:40.227 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:40.227 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:40.227 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:40.227 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:40.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.233+0000 7f9c42bf0640 1 librados: starting msgr at 2024-03-27T21:41:40.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.233+0000 7f9c42bf0640 1 librados: starting objecter 2024-03-27T21:41:40.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.234+0000 7f9c42bf0640 1 Processor -- start 2024-03-27T21:41:40.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.234+0000 7f9c42bf0640 1 -- start start 2024-03-27T21:41:40.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.234+0000 7f9c42bf0640 1 librados: setting wanted keys 2024-03-27T21:41:40.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.234+0000 7f9c42bf0640 1 librados: calling monclient init 2024-03-27T21:41:40.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.234+0000 7f9c42bf0640 10 monclient: init 2024-03-27T21:41:40.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c42bf0640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:40.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c42bf0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:41:40.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c42bf0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c3c071a50 0x7f9c3c0b67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:40.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c42bf0640 10 monclient(hunting): picked mon.noname-a con 0x7f9c3c071a50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:40.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c42bf0640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c3c13e810 0x7f9c3c1396e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:40.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c42bf0640 10 monclient(hunting): picked mon.noname-c con 0x7f9c3c13e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:40.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c42bf0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c3c13ec50 0x7f9c3c133050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:40.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c42bf0640 10 monclient(hunting): picked mon.noname-b con 0x7f9c3c13ec50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:40.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c42bf0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:40.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c42bf0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c0aadc0 con 0x7f9c3c071a50 2024-03-27T21:41:40.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c42bf0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:40.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c42bf0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c13a340 con 0x7f9c3c13ec50 2024-03-27T21:41:40.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c42bf0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:40.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c42bf0640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c139dd0 con 0x7f9c3c13e810 2024-03-27T21:41:40.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c42bf0640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:40.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c42bf0640 10 monclient(hunting): authenticate will time out at 941.456120s 2024-03-27T21:41:40.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4098c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c3c071a50 0x7f9c3c0b67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:40.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4098c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c3c071a50 0x7f9c3c0b67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50952/0 (socket says 172.21.15.60:50952) 2024-03-27T21:41:40.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4098c640 1 -- 172.21.15.60:0/3045954623 learned_addr learned my addr 172.21.15.60:0/3045954623 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:40.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4098c640 10 monclient(hunting): get_auth_request con 0x7f9c3c071a50 auth_method 0 2024-03-27T21:41:40.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4098c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:40.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4098c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:40.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4098c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:40.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c3bfff640 1 --2- 172.21.15.60:0/3045954623 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c3c13e810 0x7f9c3c1396e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:40.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c3bfff640 10 monclient(hunting): get_auth_request con 0x7f9c3c13e810 auth_method 0 2024-03-27T21:41:40.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:40.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:40.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:40.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4118d640 1 --2- 172.21.15.60:0/3045954623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c3c13ec50 0x7f9c3c133050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:40.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4118d640 10 monclient(hunting): get_auth_request con 0x7f9c3c13ec50 auth_method 0 2024-03-27T21:41:40.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4118d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:40.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4118d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:40.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4118d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:40.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4098c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:40.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4098c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:40.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4098c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:40.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:40.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:40.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:40.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4118d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:40.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4118d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:40.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4118d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:40.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4118d640 1 -- 172.21.15.60:0/3045954623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c3c13ec50 msgr2=0x7f9c3c133050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-27T21:41:40.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4118d640 1 -- 172.21.15.60:0/3045954623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c3c13ec50 msgr2=0x7f9c3c133050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-27T21:41:40.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4118d640 1 --2- 172.21.15.60:0/3045954623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c3c13ec50 0x7f9c3c133050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-27T21:41:40.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4118d640 1 --2- 172.21.15.60:0/3045954623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c3c13ec50 0x7f9c3c133050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-27T21:41:40.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4098c640 10 monclient(hunting): handle_auth_done global_id 5109 payload 824 2024-03-27T21:41:40.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4098c640 1 -- 172.21.15.60:0/3045954623 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c3c13e810 msgr2=0x7f9c3c1396e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:40.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4098c640 1 --2- 172.21.15.60:0/3045954623 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c3c13e810 0x7f9c3c1396e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4098c640 1 -- 172.21.15.60:0/3045954623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c3c13ec50 msgr2=0x7f9c3c133050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T21:41:40.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4098c640 1 --2- 172.21.15.60:0/3045954623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c3c13ec50 0x7f9c3c133050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4098c640 10 monclient: _finish_hunting 0 2024-03-27T21:41:40.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4098c640 1 monclient: found mon.noname-a 2024-03-27T21:41:40.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4098c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:40.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4098c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:40.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4098c640 1 -- 172.21.15.60:0/3045954623 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c30009400 con 0x7f9c3c071a50 2024-03-27T21:41:40.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c3bfff640 1 --2- 172.21.15.60:0/3045954623 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c3c13e810 0x7f9c3c1396e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:40.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c4098c640 1 --2- 172.21.15.60:0/3045954623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c3c071a50 0x7f9c3c0b67d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9c3000ebe0 tx=0x7f9c3000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:40.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c397fa640 10 client.?.objecter ms_handle_connect 0x7f9c3c071a50 2024-03-27T21:41:40.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.235+0000 7f9c397fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:40.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c397fa640 1 -- 172.21.15.60:0/3045954623 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c30009760 con 0x7f9c3c071a50 2024-03-27T21:41:40.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:40.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c397fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:40.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c397fa640 10 monclient: dump: 2024-03-27T21:41:40.237 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:40.237 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:40.237 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:40.238 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:40.238 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:40.238 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:40.238 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:40.238 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:40.238 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:40.238 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:40.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c397fa640 10 monclient: _finish_auth 0 2024-03-27T21:41:40.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c397fa640 10 monclient: _check_auth_tickets 2024-03-27T21:41:40.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:40.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c42bf0640 5 monclient: authenticate success, global_id 5109 2024-03-27T21:41:40.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c397fa640 1 -- 172.21.15.60:0/3045954623 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c3000cc30 con 0x7f9c3c071a50 2024-03-27T21:41:40.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c42bf0640 10 monclient: _renew_subs 2024-03-27T21:41:40.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c42bf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:40.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c42bf0640 1 -- 172.21.15.60:0/3045954623 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c34009b30 con 0x7f9c3c071a50 2024-03-27T21:41:40.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c397fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:40.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c397fa640 1 -- 172.21.15.60:0/3045954623 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3000eee0 con 0x7f9c3c071a50 2024-03-27T21:41:40.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:40.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c397fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:40.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c397fa640 10 monclient: dump: 2024-03-27T21:41:40.239 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:40.239 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:40.239 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:40.239 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:40.239 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:40.239 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:40.239 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:40.240 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:40.240 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:40.240 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:40.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c3affd640 4 set_mon_vals no callback set 2024-03-27T21:41:40.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c42bf0640 10 client.5109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:40.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c42bf0640 10 monclient: _renew_subs 2024-03-27T21:41:40.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c42bf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:40.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c42bf0640 1 -- 172.21.15.60:0/3045954623 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c3c133770 con 0x7f9c3c071a50 2024-03-27T21:41:40.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.236+0000 7f9c42bf0640 1 librados: init done 2024-03-27T21:41:40.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.237+0000 7f9c42bf0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:40.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.237+0000 7f9c42bf0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:40.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.237+0000 7f9c42bf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:40.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.237+0000 7f9c42bf0640 1 -- 172.21.15.60:0/3045954623 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c3c13a500 con 0x7f9c3c071a50 2024-03-27T21:41:40.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.238+0000 7f9c397fa640 1 -- 172.21.15.60:0/3045954623 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9c30013470 con 0x7f9c3c071a50 2024-03-27T21:41:40.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.238+0000 7f9c397fa640 1 --2- 172.21.15.60:0/3045954623 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9c2006c590 0x7f9c2006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:40.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.238+0000 7f9c3bfff640 1 --2- 172.21.15.60:0/3045954623 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9c2006c590 0x7f9c2006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:40.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.238+0000 7f9c3bfff640 10 monclient: get_auth_request con 0x7f9c2006c590 auth_method 0 2024-03-27T21:41:40.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.238+0000 7f9c397fa640 1 -- 172.21.15.60:0/3045954623 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f9c30089310 con 0x7f9c3c071a50 2024-03-27T21:41:40.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.238+0000 7f9c397fa640 10 client.5109.objecter ms_dispatch 0x7f9c3c000b90 osd_map(71..71 src has 1..71) v4 2024-03-27T21:41:40.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.239+0000 7f9c397fa640 3 client.5109.objecter handle_osd_map got epochs [71,71] > 0 2024-03-27T21:41:40.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.239+0000 7f9c397fa640 3 client.5109.objecter handle_osd_map decoding full epoch 71 2024-03-27T21:41:40.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.239+0000 7f9c397fa640 20 client.5109.objecter dump_active .. 0 homeless 2024-03-27T21:41:40.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.239+0000 7f9c397fa640 10 client.5109.objecter ms_handle_connect 0x7f9c2006c590 2024-03-27T21:41:40.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.239+0000 7f9c3bfff640 1 --2- 172.21.15.60:0/3045954623 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9c2006c590 0x7f9c2006ea50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9c28007fd0 tx=0x7f9c28009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:40.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.246+0000 7f9c397fa640 1 -- 172.21.15.60:0/3045954623 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9c30054800 con 0x7f9c3c071a50 2024-03-27T21:41:40.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.246+0000 7f9c397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:40.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.246+0000 7f9c397fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:40.243 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:40.243 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:40.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:40.243 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-03-27T21:41:40.243 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.243 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:40.243 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:40.243 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 120.81747604, 2024-03-27T21:41:40.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43537978700000002, 2024-03-27T21:41:40.243 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:40.243 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:40.243 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:40.244 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:40.244 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:40.244 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-03-27T21:41:40.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.244 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:40.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:40.244 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:40.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:40.244 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:40.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:40.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432019+0000", 2024-03-27T21:41:40.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-03-27T21:41:40.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:40.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432027+0000", 2024-03-27T21:41:40.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1200000000000002e-06 2024-03-27T21:41:40.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:40.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432039+0000", 2024-03-27T21:41:40.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1123e-05 2024-03-27T21:41:40.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.246 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:40.246 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432051+0000", 2024-03-27T21:41:40.246 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-03-27T21:41:40.246 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.246 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.246 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:40.246 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432157+0000", 2024-03-27T21:41:40.246 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106846 2024-03-27T21:41:40.246 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.246 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.246 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-03-27T21:41:40.246 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432176+0000", 2024-03-27T21:41:40.246 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8490000000000001e-05 2024-03-27T21:41:40.246 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.247 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.247 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:40.247 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866170+0000", 2024-03-27T21:41:40.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43399370100000001 2024-03-27T21:41:40.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.247 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.247 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:40.247 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866335+0000", 2024-03-27T21:41:40.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016559699999999999 2024-03-27T21:41:40.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.247 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.247 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:40.247 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866495+0000", 2024-03-27T21:41:40.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015944200000000001 2024-03-27T21:41:40.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.247 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.247 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:40.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867012+0000", 2024-03-27T21:41:40.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0005176 2024-03-27T21:41:40.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:40.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867334+0000", 2024-03-27T21:41:40.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032158600000000001 2024-03-27T21:41:40.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:40.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867374+0000", 2024-03-27T21:41:40.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0547000000000003e-05 2024-03-27T21:41:40.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:40.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867400+0000", 2024-03-27T21:41:40.249 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5958000000000001e-05 2024-03-27T21:41:40.249 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.249 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:40.249 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.249 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.249 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.249 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:40.249 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:40.249 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 120.320093589, 2024-03-27T21:41:40.249 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064560310000000001, 2024-03-27T21:41:40.249 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:40.249 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:40.249 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:40.249 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:40.250 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:40.250 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 70 2024-03-27T21:41:40.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.250 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:40.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.250 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:40.250 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:40.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.250 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:40.250 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:40.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.251 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.251 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:40.251 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929402+0000", 2024-03-27T21:41:40.251 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:40.251 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.251 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.251 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:40.251 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929407+0000", 2024-03-27T21:41:40.251 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-06 2024-03-27T21:41:40.251 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.251 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.251 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:40.251 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929414+0000", 2024-03-27T21:41:40.251 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5689999999999999e-06 2024-03-27T21:41:40.252 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.252 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.252 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:40.252 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929419+0000", 2024-03-27T21:41:40.252 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-03-27T21:41:40.252 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.252 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.252 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:40.252 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930245+0000", 2024-03-27T21:41:40.252 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082521500000000002 2024-03-27T21:41:40.252 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.252 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.252 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:40.252 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930287+0000", 2024-03-27T21:41:40.252 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2305000000000002e-05 2024-03-27T21:41:40.252 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.253 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.253 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:40.253 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930367+0000", 2024-03-27T21:41:40.253 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9589e-05 2024-03-27T21:41:40.253 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.253 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.253 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:40.253 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.931214+0000", 2024-03-27T21:41:40.253 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000847827 2024-03-27T21:41:40.253 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.253 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.253 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:40.253 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935807+0000", 2024-03-27T21:41:40.253 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045923780000000003 2024-03-27T21:41:40.253 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.254 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:40.254 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935840+0000", 2024-03-27T21:41:40.254 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3501999999999999e-05 2024-03-27T21:41:40.254 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.254 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:40.254 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935859+0000", 2024-03-27T21:41:40.254 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8845e-05 2024-03-27T21:41:40.254 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.254 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:40.254 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.254 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.255 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:40.255 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:40.255 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 120.312833998, 2024-03-27T21:41:40.255 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071695429999999996, 2024-03-27T21:41:40.255 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:40.255 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:40.255 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:40.255 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:40.255 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:40.257 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-03-27T21:41:40.258 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.258 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:40.258 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.258 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:40.258 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:40.258 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.258 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.258 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.258 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:40.258 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:40.258 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.258 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.258 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.259 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:40.259 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936662+0000", 2024-03-27T21:41:40.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:40.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.259 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.259 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:40.259 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936737+0000", 2024-03-27T21:41:40.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5303999999999996e-05 2024-03-27T21:41:40.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.259 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.259 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:40.259 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936746+0000", 2024-03-27T21:41:40.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6340000000000007e-06 2024-03-27T21:41:40.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.260 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.260 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:40.260 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936753+0000", 2024-03-27T21:41:40.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-03-27T21:41:40.260 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.260 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.260 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:40.260 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937926+0000", 2024-03-27T21:41:40.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011731280000000001 2024-03-27T21:41:40.260 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.260 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.260 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:40.260 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937969+0000", 2024-03-27T21:41:40.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3285999999999999e-05 2024-03-27T21:41:40.261 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.261 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.261 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:40.261 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938040+0000", 2024-03-27T21:41:40.261 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0358000000000006e-05 2024-03-27T21:41:40.261 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.261 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.261 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:40.261 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939324+0000", 2024-03-27T21:41:40.261 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012844899999999999 2024-03-27T21:41:40.261 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.261 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.261 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:40.261 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943780+0000", 2024-03-27T21:41:40.261 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044555630000000001 2024-03-27T21:41:40.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.262 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:40.262 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943811+0000", 2024-03-27T21:41:40.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1415999999999999e-05 2024-03-27T21:41:40.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.262 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:40.262 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943832+0000", 2024-03-27T21:41:40.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0981000000000001e-05 2024-03-27T21:41:40.262 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.262 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:40.262 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.263 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:40.263 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:40.263 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 120.31273598200001, 2024-03-27T21:41:40.263 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071139740000000003, 2024-03-27T21:41:40.263 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:40.263 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:40.263 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:40.263 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:40.263 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:40.263 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-03-27T21:41:40.263 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.263 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:40.264 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.264 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:40.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:40.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.264 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.264 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:40.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:40.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.264 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.264 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:40.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936760+0000", 2024-03-27T21:41:40.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:40.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.265 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:40.265 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936765+0000", 2024-03-27T21:41:40.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6330000000000004e-06 2024-03-27T21:41:40.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.265 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:40.265 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936771+0000", 2024-03-27T21:41:40.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0560000000000001e-06 2024-03-27T21:41:40.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.265 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:40.265 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936776+0000", 2024-03-27T21:41:40.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1510000000000003e-06 2024-03-27T21:41:40.266 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.266 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.266 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:40.266 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938605+0000", 2024-03-27T21:41:40.266 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001829589 2024-03-27T21:41:40.266 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.266 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.266 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:40.266 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938657+0000", 2024-03-27T21:41:40.266 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1860000000000002e-05 2024-03-27T21:41:40.266 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.266 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.266 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:40.266 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938734+0000", 2024-03-27T21:41:40.267 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6639999999999998e-05 2024-03-27T21:41:40.267 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.267 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.267 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:40.267 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939562+0000", 2024-03-27T21:41:40.267 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082803100000000001 2024-03-27T21:41:40.267 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.267 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.267 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:40.267 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943838+0000", 2024-03-27T21:41:40.267 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042757050000000003 2024-03-27T21:41:40.267 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.267 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.267 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:40.267 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943859+0000", 2024-03-27T21:41:40.268 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1175e-05 2024-03-27T21:41:40.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.268 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.268 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:40.268 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943875+0000", 2024-03-27T21:41:40.268 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5744e-05 2024-03-27T21:41:40.268 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.268 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:40.268 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.268 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.268 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:40.268 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:40.268 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 120.169221931, 2024-03-27T21:41:40.269 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059712230000000003, 2024-03-27T21:41:40.269 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:40.269 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:40.269 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:40.269 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:40.269 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:40.269 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 112 2024-03-27T21:41:40.269 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.269 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:40.269 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.269 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:40.269 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:40.269 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.269 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.269 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.270 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:40.270 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:40.270 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.270 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.270 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.270 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:40.270 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080274+0000", 2024-03-27T21:41:40.270 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:40.270 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.270 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.270 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:40.270 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080318+0000", 2024-03-27T21:41:40.270 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3634999999999998e-05 2024-03-27T21:41:40.270 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.271 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.271 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:40.271 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080329+0000", 2024-03-27T21:41:40.271 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1232e-05 2024-03-27T21:41:40.271 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.271 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.271 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:40.271 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080333+0000", 2024-03-27T21:41:40.271 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6009999999999999e-06 2024-03-27T21:41:40.271 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.271 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.271 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:40.271 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083786+0000", 2024-03-27T21:41:40.271 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034528940000000002 2024-03-27T21:41:40.272 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.272 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.272 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:40.272 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083861+0000", 2024-03-27T21:41:40.272 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4893000000000003e-05 2024-03-27T21:41:40.272 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.272 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.272 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:40.272 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083981+0000", 2024-03-27T21:41:40.272 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012058299999999999 2024-03-27T21:41:40.272 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.272 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.272 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:40.272 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086148+0000", 2024-03-27T21:41:40.272 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021671469999999999 2024-03-27T21:41:40.273 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.273 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.273 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:40.273 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086195+0000", 2024-03-27T21:41:40.273 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6768999999999999e-05 2024-03-27T21:41:40.273 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.273 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.273 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:40.273 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086224+0000", 2024-03-27T21:41:40.273 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8758000000000001e-05 2024-03-27T21:41:40.273 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.273 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.273 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:40.273 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086246+0000", 2024-03-27T21:41:40.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2064000000000001e-05 2024-03-27T21:41:40.274 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.274 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:40.274 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.274 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.274 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.274 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:40.274 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:40.274 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 120.16915898400001, 2024-03-27T21:41:40.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066196320000000003, 2024-03-27T21:41:40.274 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:40.274 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:40.274 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:40.274 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:40.275 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:40.275 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113 2024-03-27T21:41:40.275 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.275 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:40.275 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.275 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:40.275 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:40.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.275 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.275 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.275 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:40.275 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:40.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.275 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.275 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.276 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:40.276 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080337+0000", 2024-03-27T21:41:40.276 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:40.276 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.276 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.276 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:40.276 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080340+0000", 2024-03-27T21:41:40.276 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6690000000000002e-06 2024-03-27T21:41:40.276 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.276 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.276 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:40.276 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080344+0000", 2024-03-27T21:41:40.276 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6550000000000002e-06 2024-03-27T21:41:40.276 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.277 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.277 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:40.277 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080348+0000", 2024-03-27T21:41:40.277 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9169999999999999e-06 2024-03-27T21:41:40.277 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.277 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.277 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:40.277 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084595+0000", 2024-03-27T21:41:40.277 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042471719999999996 2024-03-27T21:41:40.277 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.277 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.277 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:40.277 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084676+0000", 2024-03-27T21:41:40.277 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1147000000000001e-05 2024-03-27T21:41:40.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:40.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084796+0000", 2024-03-27T21:41:40.278 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012032900000000001 2024-03-27T21:41:40.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:40.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086190+0000", 2024-03-27T21:41:40.278 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013934399999999999 2024-03-27T21:41:40.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:40.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086905+0000", 2024-03-27T21:41:40.279 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071536199999999999 2024-03-27T21:41:40.279 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.279 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.279 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:40.279 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086937+0000", 2024-03-27T21:41:40.279 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2125000000000002e-05 2024-03-27T21:41:40.279 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.279 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.279 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:40.279 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086957+0000", 2024-03-27T21:41:40.279 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0157999999999999e-05 2024-03-27T21:41:40.279 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.279 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:40.279 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.280 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.280 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:40.280 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:40.280 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 120.16914626800001, 2024-03-27T21:41:40.280 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007405661, 2024-03-27T21:41:40.281 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:40.281 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:40.281 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:40.281 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:40.281 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:40.281 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 114 2024-03-27T21:41:40.281 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.281 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:40.281 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.281 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:40.281 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:40.281 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.282 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.282 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.282 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:40.282 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:40.282 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.282 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.282 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.282 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:40.282 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:40.282 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-03-27T21:41:40.282 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.282 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.282 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:40.282 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080353+0000", 2024-03-27T21:41:40.282 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.441e-06 2024-03-27T21:41:40.283 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.283 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.283 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:40.283 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080356+0000", 2024-03-27T21:41:40.283 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0769999999999999e-06 2024-03-27T21:41:40.283 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.283 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.283 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:40.283 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080358+0000", 2024-03-27T21:41:40.283 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3369999999999998e-06 2024-03-27T21:41:40.283 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.283 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.283 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:40.283 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085118+0000", 2024-03-27T21:41:40.283 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047603949999999997 2024-03-27T21:41:40.284 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.284 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.284 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:40.284 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085210+0000", 2024-03-27T21:41:40.284 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1983000000000003e-05 2024-03-27T21:41:40.284 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.284 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.284 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:40.284 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085332+0000", 2024-03-27T21:41:40.284 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121803 2024-03-27T21:41:40.284 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.284 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.284 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:40.284 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086292+0000", 2024-03-27T21:41:40.285 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095961400000000002 2024-03-27T21:41:40.285 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.285 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.285 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:40.285 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087710+0000", 2024-03-27T21:41:40.285 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014181459999999999 2024-03-27T21:41:40.285 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.285 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.285 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:40.285 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087736+0000", 2024-03-27T21:41:40.285 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6475e-05 2024-03-27T21:41:40.285 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.285 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.285 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:40.285 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087756+0000", 2024-03-27T21:41:40.286 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9626000000000001e-05 2024-03-27T21:41:40.286 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.286 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:40.286 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.286 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.286 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.286 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:40.286 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:40.286 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 120.169123833, 2024-03-27T21:41:40.286 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075011720000000004, 2024-03-27T21:41:40.286 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:40.286 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:40.286 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:40.286 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:40.287 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:40.287 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115 2024-03-27T21:41:40.287 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.287 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:40.287 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.287 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:40.287 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:40.287 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.287 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.287 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.287 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:40.287 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:40.287 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.287 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.288 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.288 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:40.288 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:40.288 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:40.288 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.288 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.288 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:40.288 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080458+0000", 2024-03-27T21:41:40.288 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5228000000000005e-05 2024-03-27T21:41:40.288 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.288 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.288 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:40.288 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080465+0000", 2024-03-27T21:41:40.288 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.238e-06 2024-03-27T21:41:40.288 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.289 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.289 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:40.289 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080468+0000", 2024-03-27T21:41:40.289 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5109999999999999e-06 2024-03-27T21:41:40.289 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.289 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.289 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:40.289 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086326+0000", 2024-03-27T21:41:40.289 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058579169999999998 2024-03-27T21:41:40.289 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.289 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.289 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:40.289 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086382+0000", 2024-03-27T21:41:40.289 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5309999999999997e-05 2024-03-27T21:41:40.290 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.290 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.290 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:40.290 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086457+0000", 2024-03-27T21:41:40.290 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5413000000000005e-05 2024-03-27T21:41:40.290 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.290 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.290 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:40.290 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087762+0000", 2024-03-27T21:41:40.290 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013052300000000001 2024-03-27T21:41:40.290 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.290 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.290 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:40.290 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087798+0000", 2024-03-27T21:41:40.291 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5614999999999997e-05 2024-03-27T21:41:40.291 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.291 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.291 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:40.291 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087840+0000", 2024-03-27T21:41:40.291 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2196000000000001e-05 2024-03-27T21:41:40.291 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.291 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.291 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:40.291 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087874+0000", 2024-03-27T21:41:40.291 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3785999999999998e-05 2024-03-27T21:41:40.291 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.291 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:40.291 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.292 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.292 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.292 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:40.292 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:40.292 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 120.16902374999999, 2024-03-27T21:41:40.292 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075050680000000002, 2024-03-27T21:41:40.292 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:40.292 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:40.292 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:40.292 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:40.292 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:40.292 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 116 2024-03-27T21:41:40.292 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.292 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:40.293 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.293 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:40.293 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:40.293 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.293 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.293 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.293 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:40.293 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:40.293 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.293 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.293 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.293 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:40.293 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080472+0000", 2024-03-27T21:41:40.293 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:40.293 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.294 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.294 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:40.294 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080475+0000", 2024-03-27T21:41:40.294 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.712e-06 2024-03-27T21:41:40.294 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.294 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.294 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:40.294 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080478+0000", 2024-03-27T21:41:40.294 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1070000000000001e-06 2024-03-27T21:41:40.294 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.294 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.294 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:40.294 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080481+0000", 2024-03-27T21:41:40.295 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4269999999999998e-06 2024-03-27T21:41:40.295 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.295 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.295 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:40.295 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086992+0000", 2024-03-27T21:41:40.295 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065114099999999996 2024-03-27T21:41:40.295 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.295 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.295 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:40.295 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087064+0000", 2024-03-27T21:41:40.295 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1927000000000004e-05 2024-03-27T21:41:40.295 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.295 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.295 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:40.295 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087188+0000", 2024-03-27T21:41:40.296 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123585 2024-03-27T21:41:40.296 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.296 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.296 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:40.296 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087766+0000", 2024-03-27T21:41:40.296 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057814300000000004 2024-03-27T21:41:40.296 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.296 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.296 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:40.296 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087934+0000", 2024-03-27T21:41:40.296 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000167894 2024-03-27T21:41:40.296 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.296 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.296 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:40.297 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087958+0000", 2024-03-27T21:41:40.297 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4663000000000001e-05 2024-03-27T21:41:40.297 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.297 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.297 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:40.297 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087978+0000", 2024-03-27T21:41:40.297 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9636e-05 2024-03-27T21:41:40.297 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.297 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:40.297 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.297 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.297 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.297 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:40.297 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:40.297 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 120.168991628, 2024-03-27T21:41:40.297 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076850679999999998, 2024-03-27T21:41:40.298 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:40.298 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:40.298 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:40.298 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:40.298 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:40.298 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 117 2024-03-27T21:41:40.298 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.298 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:40.298 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.298 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:40.298 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:40.298 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.299 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.299 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.299 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:40.299 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:40.299 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.299 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.299 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.299 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:40.299 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:40.299 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:40.299 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.299 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.299 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:40.299 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080508+0000", 2024-03-27T21:41:40.300 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.298e-06 2024-03-27T21:41:40.300 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.300 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.300 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:40.300 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080515+0000", 2024-03-27T21:41:40.300 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0790000000000003e-06 2024-03-27T21:41:40.300 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.300 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.300 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:40.300 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080518+0000", 2024-03-27T21:41:40.300 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8509999999999999e-06 2024-03-27T21:41:40.300 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.300 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.300 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:40.300 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087444+0000", 2024-03-27T21:41:40.301 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0069259120000000002 2024-03-27T21:41:40.301 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.301 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.301 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:40.301 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087491+0000", 2024-03-27T21:41:40.301 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7043999999999999e-05 2024-03-27T21:41:40.301 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.301 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.301 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:40.301 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087564+0000", 2024-03-27T21:41:40.301 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3603999999999995e-05 2024-03-27T21:41:40.301 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.301 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.301 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:40.302 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087902+0000", 2024-03-27T21:41:40.302 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033734100000000001 2024-03-27T21:41:40.302 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.302 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.302 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:40.302 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088117+0000", 2024-03-27T21:41:40.302 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000215164 2024-03-27T21:41:40.302 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.302 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.302 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:40.302 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088160+0000", 2024-03-27T21:41:40.302 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2933999999999999e-05 2024-03-27T21:41:40.302 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.302 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.303 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:40.303 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088190+0000", 2024-03-27T21:41:40.303 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0139999999999999e-05 2024-03-27T21:41:40.303 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.303 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:40.303 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.303 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.303 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.303 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:40.303 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:40.303 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 119.75271123900001, 2024-03-27T21:41:40.303 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052465439999999997, 2024-03-27T21:41:40.303 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:40.303 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:40.303 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:40.304 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:40.304 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:40.304 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 133 2024-03-27T21:41:40.304 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.304 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:40.304 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.304 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:40.304 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:40.304 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.304 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.304 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.304 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:40.304 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:40.304 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.305 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.305 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:40.305 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:40.305 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:40.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.305 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.305 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:40.305 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496788+0000", 2024-03-27T21:41:40.305 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-03-27T21:41:40.305 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.305 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.305 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:40.305 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496793+0000", 2024-03-27T21:41:40.305 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4790000000000004e-06 2024-03-27T21:41:40.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.306 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:40.306 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496796+0000", 2024-03-27T21:41:40.306 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1e-06 2024-03-27T21:41:40.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.306 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:40.306 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500447+0000", 2024-03-27T21:41:40.306 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036504459999999999 2024-03-27T21:41:40.306 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.306 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.306 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:40.306 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500492+0000", 2024-03-27T21:41:40.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5765e-05 2024-03-27T21:41:40.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.307 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.307 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:40.307 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500562+0000", 2024-03-27T21:41:40.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9206000000000001e-05 2024-03-27T21:41:40.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.307 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.307 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:40.307 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501959+0000", 2024-03-27T21:41:40.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001397187 2024-03-27T21:41:40.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.307 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.307 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:40.307 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501984+0000", 2024-03-27T21:41:40.307 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.552e-05 2024-03-27T21:41:40.308 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.308 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.308 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:40.308 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502010+0000", 2024-03-27T21:41:40.308 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5232000000000002e-05 2024-03-27T21:41:40.308 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.308 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.308 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:40.308 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502032+0000", 2024-03-27T21:41:40.308 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2337999999999999e-05 2024-03-27T21:41:40.308 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.308 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:40.308 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.308 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.308 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.309 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:40.309 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:40.309 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 119.75265258100001, 2024-03-27T21:41:40.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052892119999999997, 2024-03-27T21:41:40.309 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:40.309 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:40.309 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:40.309 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:40.309 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:40.309 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 134 2024-03-27T21:41:40.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.309 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:40.309 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.309 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:40.310 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:40.310 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.310 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.310 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:40.310 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:40.310 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.310 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.310 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:40.310 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:40.310 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:40.310 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.311 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:40.311 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496929+0000", 2024-03-27T21:41:40.311 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5098000000000004e-05 2024-03-27T21:41:40.311 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.311 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.311 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:40.311 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496937+0000", 2024-03-27T21:41:40.311 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9559999999999997e-06 2024-03-27T21:41:40.311 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.311 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.311 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:40.311 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496942+0000", 2024-03-27T21:41:40.311 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6350000000000003e-06 2024-03-27T21:41:40.311 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:40.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500749+0000", 2024-03-27T21:41:40.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038068529999999998 2024-03-27T21:41:40.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:40.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500792+0000", 2024-03-27T21:41:40.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2823e-05 2024-03-27T21:41:40.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:40.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500864+0000", 2024-03-27T21:41:40.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1715e-05 2024-03-27T21:41:40.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:40.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502068+0000", 2024-03-27T21:41:40.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012044860000000001 2024-03-27T21:41:40.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:40.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502073+0000", 2024-03-27T21:41:40.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5700000000000003e-06 2024-03-27T21:41:40.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:40.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502098+0000", 2024-03-27T21:41:40.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5120999999999999e-05 2024-03-27T21:41:40.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.314 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.314 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:40.314 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502133+0000", 2024-03-27T21:41:40.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5306999999999999e-05 2024-03-27T21:41:40.314 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.314 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:40.314 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.314 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.314 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:40.314 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:40.314 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 119.75254727399999, 2024-03-27T21:41:40.315 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062529040000000001, 2024-03-27T21:41:40.315 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:40.315 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:40.315 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:40.315 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:40.315 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:40.315 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135 2024-03-27T21:41:40.315 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.315 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:40.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.315 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:40.315 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:40.315 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.315 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.316 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.316 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:40.316 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:40.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.316 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.316 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.316 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:40.316 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:40.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:40.316 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.316 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.316 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:40.316 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496953+0000", 2024-03-27T21:41:40.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0709999999999996e-06 2024-03-27T21:41:40.316 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:40.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496961+0000", 2024-03-27T21:41:40.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1300000000000001e-06 2024-03-27T21:41:40.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:40.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496968+0000", 2024-03-27T21:41:40.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0999999999999998e-06 2024-03-27T21:41:40.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:40.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501244+0000", 2024-03-27T21:41:40.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042756310000000002 2024-03-27T21:41:40.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:40.318 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501377+0000", 2024-03-27T21:41:40.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013348799999999999 2024-03-27T21:41:40.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:40.318 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501455+0000", 2024-03-27T21:41:40.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7410000000000006e-05 2024-03-27T21:41:40.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.318 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:40.318 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502138+0000", 2024-03-27T21:41:40.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068386599999999999 2024-03-27T21:41:40.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.319 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.319 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:40.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503152+0000", 2024-03-27T21:41:40.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010139680000000001 2024-03-27T21:41:40.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.319 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.319 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:40.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503180+0000", 2024-03-27T21:41:40.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7426000000000001e-05 2024-03-27T21:41:40.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.319 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.319 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:40.320 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503202+0000", 2024-03-27T21:41:40.320 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2463000000000002e-05 2024-03-27T21:41:40.320 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.320 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:40.320 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.320 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.320 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.320 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:40.320 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:40.320 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 119.75252313599999, 2024-03-27T21:41:40.320 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066253680000000004, 2024-03-27T21:41:40.320 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:40.320 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:40.320 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:40.320 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:40.321 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:40.321 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 136 2024-03-27T21:41:40.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.321 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:40.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.321 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:40.321 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:40.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.321 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:40.321 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:40.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.322 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:40.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:40.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:40.322 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.322 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:40.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496977+0000", 2024-03-27T21:41:40.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5049999999999999e-06 2024-03-27T21:41:40.322 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.322 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:40.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496984+0000", 2024-03-27T21:41:40.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1520000000000003e-06 2024-03-27T21:41:40.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.323 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:40.323 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496990+0000", 2024-03-27T21:41:40.323 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-03-27T21:41:40.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.323 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:40.323 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501669+0000", 2024-03-27T21:41:40.323 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046793859999999998 2024-03-27T21:41:40.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.323 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:40.323 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501714+0000", 2024-03-27T21:41:40.323 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5154999999999997e-05 2024-03-27T21:41:40.324 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.324 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.324 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:40.324 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501783+0000", 2024-03-27T21:41:40.324 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8454999999999994e-05 2024-03-27T21:41:40.324 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.324 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.324 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:40.324 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503226+0000", 2024-03-27T21:41:40.324 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001443441 2024-03-27T21:41:40.324 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.324 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.324 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:40.324 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503540+0000", 2024-03-27T21:41:40.325 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031347100000000001 2024-03-27T21:41:40.325 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.325 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.325 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:40.325 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503563+0000", 2024-03-27T21:41:40.325 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3223000000000001e-05 2024-03-27T21:41:40.325 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.325 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.325 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:40.325 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503599+0000", 2024-03-27T21:41:40.325 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6105000000000001e-05 2024-03-27T21:41:40.325 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.325 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:40.325 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.325 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.326 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:40.326 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:40.326 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 119.752500326, 2024-03-27T21:41:40.326 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074261320000000002, 2024-03-27T21:41:40.326 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:40.326 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:40.326 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:40.326 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:40.326 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:40.326 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 137 2024-03-27T21:41:40.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.326 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:40.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.327 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:40.327 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:40.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.327 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.327 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:40.327 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:40.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.327 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.327 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:40.327 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:40.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:40.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.327 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.328 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:40.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497077+0000", 2024-03-27T21:41:40.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1599000000000003e-05 2024-03-27T21:41:40.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.328 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.328 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:40.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497085+0000", 2024-03-27T21:41:40.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6109999999999996e-06 2024-03-27T21:41:40.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.328 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.328 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:40.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497092+0000", 2024-03-27T21:41:40.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.742e-06 2024-03-27T21:41:40.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.329 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.329 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:40.329 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502159+0000", 2024-03-27T21:41:40.329 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050666870000000003 2024-03-27T21:41:40.329 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.329 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.329 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:40.329 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502200+0000", 2024-03-27T21:41:40.329 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1919999999999998e-05 2024-03-27T21:41:40.329 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.329 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.329 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:40.329 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502280+0000", 2024-03-27T21:41:40.329 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9417000000000002e-05 2024-03-27T21:41:40.330 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.330 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.330 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:40.330 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503605+0000", 2024-03-27T21:41:40.330 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001325641 2024-03-27T21:41:40.330 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.330 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.330 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:40.330 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504361+0000", 2024-03-27T21:41:40.330 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075555399999999997 2024-03-27T21:41:40.330 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.330 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.330 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:40.330 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504386+0000", 2024-03-27T21:41:40.331 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5369999999999999e-05 2024-03-27T21:41:40.331 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.331 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.331 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:40.331 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504422+0000", 2024-03-27T21:41:40.331 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6019000000000002e-05 2024-03-27T21:41:40.331 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.331 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:40.331 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.331 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.331 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.331 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:40.331 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:40.331 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 119.75239652899999, 2024-03-27T21:41:40.332 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074059750000000004, 2024-03-27T21:41:40.332 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:40.332 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:40.332 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:40.332 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:40.332 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:40.332 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-03-27T21:41:40.332 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.332 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:40.332 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.332 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:40.332 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:40.332 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.332 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.333 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.333 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:40.333 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:40.333 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.333 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.333 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.333 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:40.333 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:40.333 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:40.333 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.333 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.333 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:40.333 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497103+0000", 2024-03-27T21:41:40.333 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.856e-06 2024-03-27T21:41:40.334 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.334 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.334 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:40.334 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497121+0000", 2024-03-27T21:41:40.334 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7190999999999999e-05 2024-03-27T21:41:40.334 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.334 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.334 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:40.334 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497126+0000", 2024-03-27T21:41:40.334 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3770000000000004e-06 2024-03-27T21:41:40.334 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.334 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.334 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:40.334 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502833+0000", 2024-03-27T21:41:40.335 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057073510000000003 2024-03-27T21:41:40.335 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.335 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.335 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:40.335 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502881+0000", 2024-03-27T21:41:40.335 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7961999999999999e-05 2024-03-27T21:41:40.335 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.335 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.335 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:40.335 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502949+0000", 2024-03-27T21:41:40.335 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7957e-05 2024-03-27T21:41:40.335 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.335 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.335 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:40.336 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504430+0000", 2024-03-27T21:41:40.336 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001480763 2024-03-27T21:41:40.336 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.336 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.336 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:40.336 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504459+0000", 2024-03-27T21:41:40.336 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8983999999999999e-05 2024-03-27T21:41:40.336 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.336 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.336 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:40.336 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504486+0000", 2024-03-27T21:41:40.336 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6883999999999999e-05 2024-03-27T21:41:40.336 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.336 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.336 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:40.337 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504506+0000", 2024-03-27T21:41:40.337 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0214e-05 2024-03-27T21:41:40.337 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.337 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:40.337 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.337 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.337 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.337 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:40.337 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:40.337 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 119.752364695, 2024-03-27T21:41:40.337 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074754859999999999, 2024-03-27T21:41:40.337 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:40.337 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:40.337 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:40.338 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:40.338 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:40.338 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-03-27T21:41:40.338 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.338 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:40.338 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.338 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:40.338 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:40.338 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.338 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.338 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.338 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:40.338 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:40.338 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.339 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.339 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.339 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:40.339 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:40.339 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:40.339 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.339 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.339 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:40.339 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497137+0000", 2024-03-27T21:41:40.339 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0730000000000004e-06 2024-03-27T21:41:40.339 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.339 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.339 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:40.339 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497143+0000", 2024-03-27T21:41:40.339 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3759999999999996e-06 2024-03-27T21:41:40.340 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.340 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.340 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:40.340 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497150+0000", 2024-03-27T21:41:40.340 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.002e-06 2024-03-27T21:41:40.340 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.340 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.340 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:40.340 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503249+0000", 2024-03-27T21:41:40.340 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060989750000000004 2024-03-27T21:41:40.340 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.340 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.340 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:40.340 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503291+0000", 2024-03-27T21:41:40.341 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1563999999999997e-05 2024-03-27T21:41:40.341 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.341 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.341 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:40.341 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503376+0000", 2024-03-27T21:41:40.341 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5752000000000002e-05 2024-03-27T21:41:40.341 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.341 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.341 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:40.341 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504545+0000", 2024-03-27T21:41:40.341 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169172 2024-03-27T21:41:40.341 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.341 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.341 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:40.341 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504550+0000", 2024-03-27T21:41:40.342 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7419999999999997e-06 2024-03-27T21:41:40.342 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.342 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.342 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:40.342 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504574+0000", 2024-03-27T21:41:40.342 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4124000000000001e-05 2024-03-27T21:41:40.342 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.342 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.342 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:40.342 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504607+0000", 2024-03-27T21:41:40.342 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3118999999999999e-05 2024-03-27T21:41:40.342 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.342 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:40.342 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.343 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.343 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.343 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:40.343 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:40.343 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 119.75234102899999, 2024-03-27T21:41:40.343 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078150589999999992, 2024-03-27T21:41:40.343 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:40.343 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:40.343 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:40.343 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:40.343 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:40.343 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-03-27T21:41:40.343 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.343 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:40.344 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.344 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:40.344 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:40.344 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.344 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.344 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.344 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:40.344 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:40.344 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.344 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.344 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.344 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:40.344 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497155+0000", 2024-03-27T21:41:40.344 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:40.345 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.345 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.345 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:40.345 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497230+0000", 2024-03-27T21:41:40.345 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4789000000000005e-05 2024-03-27T21:41:40.345 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.345 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.345 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:40.345 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497241+0000", 2024-03-27T21:41:40.345 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1442e-05 2024-03-27T21:41:40.345 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.345 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.345 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:40.345 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497255+0000", 2024-03-27T21:41:40.346 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3087e-05 2024-03-27T21:41:40.346 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.346 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.346 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:40.346 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503630+0000", 2024-03-27T21:41:40.346 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063755269999999998 2024-03-27T21:41:40.346 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.346 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.346 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:40.346 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503673+0000", 2024-03-27T21:41:40.346 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2616e-05 2024-03-27T21:41:40.346 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.346 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.346 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:40.347 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503738+0000", 2024-03-27T21:41:40.347 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5114e-05 2024-03-27T21:41:40.347 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.347 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.347 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:40.347 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504907+0000", 2024-03-27T21:41:40.347 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169432 2024-03-27T21:41:40.347 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.347 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.347 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:40.347 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504926+0000", 2024-03-27T21:41:40.347 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9086999999999999e-05 2024-03-27T21:41:40.347 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.347 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.348 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:40.348 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504950+0000", 2024-03-27T21:41:40.348 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3598999999999999e-05 2024-03-27T21:41:40.348 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.348 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.348 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:40.348 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504971+0000", 2024-03-27T21:41:40.348 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0772999999999999e-05 2024-03-27T21:41:40.348 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.348 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:40.348 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.348 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.348 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.348 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:40.348 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:40.349 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 119.752233989, 2024-03-27T21:41:40.349 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082962590000000003, 2024-03-27T21:41:40.349 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:40.349 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:40.349 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:40.349 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:40.349 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:40.349 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-03-27T21:41:40.349 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.349 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:40.349 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.349 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:40.349 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:40.349 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.349 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.350 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.350 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:40.350 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:40.350 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.350 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.350 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.350 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:40.350 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497262+0000", 2024-03-27T21:41:40.350 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:40.350 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.350 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.350 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:40.350 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497267+0000", 2024-03-27T21:41:40.350 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.955e-06 2024-03-27T21:41:40.350 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.351 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.351 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:40.351 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497275+0000", 2024-03-27T21:41:40.351 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.824e-06 2024-03-27T21:41:40.351 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.351 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.351 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:40.351 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497280+0000", 2024-03-27T21:41:40.351 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8230000000000002e-06 2024-03-27T21:41:40.351 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.351 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.351 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:40.351 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504628+0000", 2024-03-27T21:41:40.351 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007348561 2024-03-27T21:41:40.352 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.352 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.352 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:40.352 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504673+0000", 2024-03-27T21:41:40.352 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4313000000000001e-05 2024-03-27T21:41:40.352 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.352 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.352 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:40.352 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504745+0000", 2024-03-27T21:41:40.352 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2303000000000002e-05 2024-03-27T21:41:40.352 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.352 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.352 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:40.352 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505358+0000", 2024-03-27T21:41:40.353 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061281300000000003 2024-03-27T21:41:40.353 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.353 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.353 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:40.353 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505503+0000", 2024-03-27T21:41:40.353 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000145134 2024-03-27T21:41:40.353 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.353 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.353 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:40.353 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505526+0000", 2024-03-27T21:41:40.353 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3312e-05 2024-03-27T21:41:40.353 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.353 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.353 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:40.353 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505559+0000", 2024-03-27T21:41:40.354 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2821999999999998e-05 2024-03-27T21:41:40.354 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.354 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:40.354 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.354 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.354 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.354 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:40.354 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:40.354 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 119.14301175, 2024-03-27T21:41:40.354 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053753339999999998, 2024-03-27T21:41:40.354 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:40.354 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:40.354 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:40.354 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:40.355 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:40.355 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159 2024-03-27T21:41:40.355 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.355 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:40.355 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.355 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:40.355 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:40.355 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.355 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.355 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.355 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:40.355 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:40.355 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:40.355 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:40.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:40.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:40.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:40.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106488+0000", 2024-03-27T21:41:40.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9830000000000001e-06 2024-03-27T21:41:40.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:40.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106491+0000", 2024-03-27T21:41:40.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3459999999999998e-06 2024-03-27T21:41:40.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:40.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106493+0000", 2024-03-27T21:41:40.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.509e-06 2024-03-27T21:41:40.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:40.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110636+0000", 2024-03-27T21:41:40.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041427260000000002 2024-03-27T21:41:40.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:40.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110683+0000", 2024-03-27T21:41:40.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6557000000000002e-05 2024-03-27T21:41:40.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.358 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:40.358 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110755+0000", 2024-03-27T21:41:40.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2119000000000005e-05 2024-03-27T21:41:40.358 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.358 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:40.358 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111415+0000", 2024-03-27T21:41:40.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066065199999999996 2024-03-27T21:41:40.358 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.358 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:40.358 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111793+0000", 2024-03-27T21:41:40.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037732100000000001 2024-03-27T21:41:40.359 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.359 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:40.359 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111829+0000", 2024-03-27T21:41:40.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418999999999998e-05 2024-03-27T21:41:40.359 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:40.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:40.359 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:40.359 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111860+0000", 2024-03-27T21:41:40.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1003000000000001e-05 2024-03-27T21:41:40.359 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.359 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:40.359 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.359 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:40.360 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:40.360 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:40.361 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-27T21:41:40.362 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:40.375+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:40.443 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:41:40.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.478+0000 7f9c42bf0640 10 start_mon_command cmd=[{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:41:40.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.478+0000 7f9c42bf0640 10 monclient: _send_command 2 [{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:41:40.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.478+0000 7f9c42bf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:40.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.478+0000 7f9c42bf0640 1 -- 172.21.15.60:0/3045954623 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f9c3c069470 con 0x7f9c3c071a50 2024-03-27T21:41:40.465 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:40.479+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:40.522 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:40.522 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:40.522 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:40.522 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:40.536 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-27T21:41:40.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.570+0000 7f04bd4b4640 10 monclient: get_monmap_and_config 2024-03-27T21:41:40.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.570+0000 7f04bd4b4640 10 monclient: build_initial_monmap 2024-03-27T21:41:40.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.570+0000 7f04bd4b4640 10 monclient: monmap: 2024-03-27T21:41:40.557 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:40.557 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:40.557 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:40.571223+0000 2024-03-27T21:41:40.557 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:40.571223+0000 2024-03-27T21:41:40.557 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:40.557 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:40.557 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:40.557 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:40.557 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:40.558 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:40.558 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:40.558 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:40.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.570+0000 7f04bd4b4640 1 Processor -- start 2024-03-27T21:41:40.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.570+0000 7f04bd4b4640 1 -- start start 2024-03-27T21:41:40.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.570+0000 7f04bd4b4640 10 monclient: init 2024-03-27T21:41:40.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04bd4b4640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:40.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04bd4b4640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-03-27T21:41:40.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04bd4b4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f04b80b40b0 0x7f04b80a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:40.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04bd4b4640 10 monclient(hunting): picked mon.noname-c con 0x7f04b80b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:40.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04bd4b4640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f04b813b020 0x7f04b813b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04bd4b4640 10 monclient(hunting): picked mon.noname-b con 0x7f04b813b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04bd4b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f04b81416f0 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04bd4b4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b80b3830 con 0x7f04b81416f0 2024-03-27T21:41:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04bd4b4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04bd4b4640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b80b4490 con 0x7f04b80b40b0 2024-03-27T21:41:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04bd4b4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04bd4b4640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b8142290 con 0x7f04b813b020 2024-03-27T21:41:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04bd4b4640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04bd4b4640 10 monclient(hunting): authenticate will time out at 941.792539s 2024-03-27T21:41:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04b77fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f04b81416f0 0x7f04b8141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52308/0 (socket says 172.21.15.60:52308) 2024-03-27T21:41:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04b77fe640 1 -- 172.21.15.60:0/873396391 learned_addr learned my addr 172.21.15.60:0/873396391 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04b67fc640 1 --2- 172.21.15.60:0/873396391 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f04b813b020 0x7f04b813b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04b67fc640 10 monclient(hunting): get_auth_request con 0x7f04b813b020 auth_method 0 2024-03-27T21:41:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04b67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04b67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04b6ffd640 1 --2- 172.21.15.60:0/873396391 >> v2:172.21.15.191:3300/0 conn(0x7f04b80b40b0 0x7f04b80a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04b67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04b6ffd640 10 monclient(hunting): get_auth_request con 0x7f04b80b40b0 auth_method 0 2024-03-27T21:41:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.571+0000 7f04b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.572+0000 7f04b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.572+0000 7f04b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.572+0000 7f04b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.572+0000 7f04b5ffb640 1 -- 172.21.15.60:0/873396391 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192592254 0 0) 0x7f04b80b3830 con 0x7f04b81416f0 2024-03-27T21:41:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.572+0000 7f04b5ffb640 10 monclient(hunting): _negotiate 2024-03-27T21:41:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.572+0000 7f04b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.572+0000 7f04b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.572+0000 7f04b5ffb640 10 monclient(hunting): my global_id is 5112 2024-03-27T21:41:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.572+0000 7f04b5ffb640 1 -- 172.21.15.60:0/873396391 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a0003520 con 0x7f04b81416f0 2024-03-27T21:41:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.572+0000 7f04b67fc640 10 monclient(hunting): handle_auth_done global_id 4871 payload 274 2024-03-27T21:41:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.572+0000 7f04b67fc640 1 -- 172.21.15.60:0/873396391 >> v2:172.21.15.191:3300/0 conn(0x7f04b80b40b0 msgr2=0x7f04b80a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.572+0000 7f04b67fc640 1 --2- 172.21.15.60:0/873396391 >> v2:172.21.15.191:3300/0 conn(0x7f04b80b40b0 0x7f04b80a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.572+0000 7f04b67fc640 1 -- 172.21.15.60:0/873396391 >> v1:172.21.15.60:6789/0 conn(0x7f04b81416f0 legacy=0x7f04b8141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.572+0000 7f04b67fc640 10 monclient: _finish_hunting 0 2024-03-27T21:41:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.572+0000 7f04b67fc640 1 monclient: found mon.noname-b 2024-03-27T21:41:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.572+0000 7f04b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.572+0000 7f04b67fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.572+0000 7f04b67fc640 1 -- 172.21.15.60:0/873396391 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b80aa520 con 0x7f04b813b020 2024-03-27T21:41:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.573+0000 7f04b67fc640 1 --2- 172.21.15.60:0/873396391 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f04b813b020 0x7f04b813b480 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0097a0 tx=0x7f04ac02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12b9536ebf443ff2 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.573+0000 7f04b5ffb640 1 -- 172.21.15.60:0/873396391 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac02d040 con 0x7f04b813b020 2024-03-27T21:41:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.573+0000 7f04b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.573+0000 7f04b5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.573+0000 7f04b5ffb640 10 monclient: dump: 2024-03-27T21:41:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:40.572 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:40.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.573+0000 7f04b5ffb640 10 monclient: _finish_auth 0 2024-03-27T21:41:40.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.573+0000 7f04b5ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:41:40.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.573+0000 7f04b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:40.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.573+0000 7f04bd4b4640 5 monclient: authenticate success, global_id 4871 2024-03-27T21:41:40.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.573+0000 7f04bd4b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:40.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.573+0000 7f04b5ffb640 1 -- 172.21.15.60:0/873396391 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ac031810 con 0x7f04b813b020 2024-03-27T21:41:40.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.573+0000 7f04b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:40.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.573+0000 7f04b5ffb640 1 -- 172.21.15.60:0/873396391 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac031b80 con 0x7f04b813b020 2024-03-27T21:41:40.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.573+0000 7f04b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:40.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.573+0000 7f04b5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:40.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.573+0000 7f04b5ffb640 10 monclient: dump: 2024-03-27T21:41:40.573 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:40.573 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:40.573 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:40.574 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:40.574 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:40.574 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:40.574 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:40.574 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:40.574 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:40.574 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:40.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.573+0000 7f04bd4b4640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:40.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.573+0000 7f04bd4b4640 4 set_mon_vals no callback set 2024-03-27T21:41:40.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.574+0000 7f04bd4b4640 10 monclient: shutdown 2024-03-27T21:41:40.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.574+0000 7f04bd4b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:40.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.574+0000 7f04bd4b4640 1 -- 172.21.15.60:0/873396391 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f04b813b020 msgr2=0x7f04b813b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:40.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.574+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/873396391 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f04b813b020 0x7f04b813b480 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0097a0 tx=0x7f04ac02b8b0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.574+0000 7f04bd4b4640 1 -- 172.21.15.60:0/873396391 shutdown_connections 2024-03-27T21:41:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.574+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/873396391 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f04b813b020 0x7f04b813b480 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.574+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/873396391 >> v2:172.21.15.191:3300/0 conn(0x7f04b80b40b0 0x7f04b80a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.574+0000 7f04bd4b4640 1 -- 172.21.15.60:0/873396391 >> 172.21.15.60:0/873396391 conn(0x7f04b80608e0 msgr2=0x7f04b8060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.574+0000 7f04bd4b4640 1 -- 172.21.15.60:0/873396391 shutdown_connections 2024-03-27T21:41:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.574+0000 7f04bd4b4640 1 -- 172.21.15.60:0/873396391 wait complete. 2024-03-27T21:41:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.574+0000 7f04bd4b4640 10 monclient: build_initial_monmap 2024-03-27T21:41:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.574+0000 7f04bd4b4640 10 monclient: monmap: 2024-03-27T21:41:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:40.576 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.575+0000 7f04bd4b4640 1 librados: starting msgr at 2024-03-27T21:41:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.575+0000 7f04bd4b4640 1 librados: starting objecter 2024-03-27T21:41:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.575+0000 7f04bd4b4640 1 Processor -- start 2024-03-27T21:41:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.575+0000 7f04bd4b4640 1 -- start start 2024-03-27T21:41:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.575+0000 7f04bd4b4640 1 librados: setting wanted keys 2024-03-27T21:41:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.575+0000 7f04bd4b4640 1 librados: calling monclient init 2024-03-27T21:41:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.575+0000 7f04bd4b4640 10 monclient: init 2024-03-27T21:41:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04bd4b4640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04bd4b4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:41:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04bd4b4640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f04b80b40b0 0x7f04b80aac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04bd4b4640 10 monclient(hunting): picked mon.noname-c con 0x7f04b80b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04bd4b4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04b81416f0 0x7f04b813e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:40.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04bd4b4640 10 monclient(hunting): picked mon.noname-a con 0x7f04b81416f0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:40.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04b6ffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f04b80b40b0 0x7f04b80aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:40.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04b67fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04b81416f0 0x7f04b813e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:40.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04b67fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04b81416f0 0x7f04b813e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50966/0 (socket says 172.21.15.60:50966) 2024-03-27T21:41:40.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04bd4b4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04b813ba60 0x7f04b81d6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:40.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04b67fc640 1 -- 172.21.15.60:0/1915345576 learned_addr learned my addr 172.21.15.60:0/1915345576 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:40.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04bd4b4640 10 monclient(hunting): picked mon.noname-b con 0x7f04b813ba60 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:40.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04bd4b4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:40.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b81d65f0 con 0x7f04b81416f0 2024-03-27T21:41:40.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04bd4b4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:40.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b813e0b0 con 0x7f04b813ba60 2024-03-27T21:41:40.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04bd4b4640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:40.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b813d4e0 con 0x7f04b80b40b0 2024-03-27T21:41:40.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04bd4b4640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:40.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04bd4b4640 10 monclient(hunting): authenticate will time out at 941.797662s 2024-03-27T21:41:40.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04b67fc640 10 monclient(hunting): get_auth_request con 0x7f04b81416f0 auth_method 0 2024-03-27T21:41:40.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:40.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04b6ffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f04b80b40b0 0x7f04b80aac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:53384/0 (socket says 172.21.15.60:53384) 2024-03-27T21:41:40.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:40.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:40.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04b6ffd640 10 monclient(hunting): get_auth_request con 0x7f04b80b40b0 auth_method 0 2024-03-27T21:41:40.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:40.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:40.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:40.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04b67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:40.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04b67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:40.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04b67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:40.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:40.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:40.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.576+0000 7f04b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:40.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04b77fe640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04b813ba60 0x7f04b81d6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:40.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04b77fe640 10 monclient(hunting): get_auth_request con 0x7f04b813ba60 auth_method 0 2024-03-27T21:41:40.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:40.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:40.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:40.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04b67fc640 10 monclient(hunting): handle_auth_done global_id 5115 payload 824 2024-03-27T21:41:40.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04b67fc640 1 -- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f04b80b40b0 msgr2=0x7f04b80aac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:40.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04b67fc640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f04b80b40b0 0x7f04b80aac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04b67fc640 1 -- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04b813ba60 msgr2=0x7f04b81d6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:40.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04b67fc640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04b813ba60 0x7f04b81d6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04b67fc640 10 monclient: _finish_hunting 0 2024-03-27T21:41:40.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04b67fc640 1 monclient: found mon.noname-a 2024-03-27T21:41:40.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:40.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04b67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:40.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04b67fc640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac009400 con 0x7f04b81416f0 2024-03-27T21:41:40.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04b77fe640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04b813ba60 0x7f04b81d6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:41:40.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04b67fc640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04b81416f0 0x7f04b813e890 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f04ac000c00 tx=0x7f04ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:40.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f049f7fe640 10 client.?.objecter ms_handle_connect 0x7f04b81416f0 2024-03-27T21:41:40.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f049f7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:40.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac006e10 con 0x7f04b81416f0 2024-03-27T21:41:40.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f049f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:40.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f049f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:40.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f049f7fe640 10 monclient: dump: 2024-03-27T21:41:40.580 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:40.580 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:40.581 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:40.581 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:40.581 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:40.581 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:40.581 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:40.581 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:40.581 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:40.581 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:40.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f049f7fe640 10 monclient: _finish_auth 0 2024-03-27T21:41:40.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f049f7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:41:40.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f049f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:40.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ac03f070 con 0x7f04b81416f0 2024-03-27T21:41:40.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f049f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:40.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac003a60 con 0x7f04b81416f0 2024-03-27T21:41:40.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f049f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:40.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f049f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:40.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f049f7fe640 10 monclient: dump: 2024-03-27T21:41:40.582 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:40.582 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:40.582 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:40.582 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:40.582 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:40.582 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:40.582 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:40.582 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:40.583 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04bd4b4640 5 monclient: authenticate success, global_id 5115 2024-03-27T21:41:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04bd4b4640 10 monclient: _renew_subs 2024-03-27T21:41:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04bd4b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b81d6fc0 con 0x7f04b81416f0 2024-03-27T21:41:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04b57fa640 4 set_mon_vals no callback set 2024-03-27T21:41:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04bd4b4640 10 client.5115.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04bd4b4640 10 monclient: _renew_subs 2024-03-27T21:41:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04bd4b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b81d98c0 con 0x7f04b81416f0 2024-03-27T21:41:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.577+0000 7f04bd4b4640 1 librados: init done 2024-03-27T21:41:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.578+0000 7f04bd4b4640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:41:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.578+0000 7f04bd4b4640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:40.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.578+0000 7f04bd4b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.578+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f04b81d63c0 con 0x7f04b81416f0 2024-03-27T21:41:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.579+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f04ac003bc0 con 0x7f04b81416f0 2024-03-27T21:41:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.580+0000 7f049f7fe640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f04a406c5c0 0x7f04a406ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.580+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f04ac041070 con 0x7f04b81416f0 2024-03-27T21:41:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.580+0000 7f049f7fe640 10 client.5115.objecter ms_dispatch 0x7f04b8000b90 osd_map(71..71 src has 1..71) v4 2024-03-27T21:41:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.580+0000 7f049f7fe640 3 client.5115.objecter handle_osd_map got epochs [71,71] > 0 2024-03-27T21:41:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.580+0000 7f049f7fe640 3 client.5115.objecter handle_osd_map decoding full epoch 71 2024-03-27T21:41:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.580+0000 7f04b6ffd640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f04a406c5c0 0x7f04a406ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.580+0000 7f04b6ffd640 10 monclient: get_auth_request con 0x7f04a406c5c0 auth_method 0 2024-03-27T21:41:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.580+0000 7f049f7fe640 20 client.5115.objecter dump_active .. 0 homeless 2024-03-27T21:41:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.580+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v71) v1 ==== 54+0+16 (secure 0 0 0) 0x7f04b81d63c0 con 0x7f04b81416f0 2024-03-27T21:41:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.580+0000 7f049f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:40.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.580+0000 7f049f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.580+0000 7f04bd4b4640 10 client.5115.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.580+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign -1 1 2024-03-27T21:41:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.580+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f04b81da010 0x7f04b8133d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.581+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b80a90d0 osd=0 3 2024-03-27T21:41:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.581+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b80a90d0 osd=0 3 2024-03-27T21:41:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.581+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b80a90d0 osd=0 3 2024-03-27T21:41:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.581+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_remove -1 1 2024-03-27T21:41:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.581+0000 7f04bd4b4640 20 client.5115.objecter get_session s=0x7f04b80a90d0 osd=0 3 2024-03-27T21:41:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.581+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign 0 1 2024-03-27T21:41:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.581+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b80a90d0 osd=0 4 2024-03-27T21:41:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.581+0000 7f04bd4b4640 10 client.5115.objecter _send_command 1 2024-03-27T21:41:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.581+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04b81d63c0 con 0x7f04b81da010 2024-03-27T21:41:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.581+0000 7f04b77fe640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f04b81da010 0x7f04b8133d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.581+0000 7f04b77fe640 10 monclient: get_auth_request con 0x7f04b81da010 auth_method 0 2024-03-27T21:41:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.581+0000 7f049f7fe640 10 client.5115.objecter ms_handle_connect 0x7f04a406c5c0 2024-03-27T21:41:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.581+0000 7f04b6ffd640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f04a406c5c0 0x7f04a406ea80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f04a8009b40 tx=0x7f04a8009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.581+0000 7f049f7fe640 10 client.5115.objecter ms_handle_connect 0x7f04b81da010 2024-03-27T21:41:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.581+0000 7f04b77fe640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f04b81da010 0x7f04b8133d80 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.582+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f04b81d63c0 con 0x7f04b81da010 2024-03-27T21:41:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.582+0000 7f049f7fe640 10 client.5115.objecter ms_dispatch 0x7f04b8000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:41:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.582+0000 7f049f7fe640 10 client.5115.objecter _finish_command 1 = system:0 2024-03-27T21:41:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.582+0000 7f049f7fe640 20 client.5115.objecter put_session s=0x7f04b80a90d0 osd=0 4 2024-03-27T21:41:40.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.582+0000 7f049f7fe640 15 client.5115.objecter _session_command_op_remove 0 1 2024-03-27T21:41:40.587 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:40.579+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:40.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.610+0000 7f04bd4b4640 10 client.5115.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:40.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.610+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign -1 2 2024-03-27T21:41:40.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.610+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b80a90d0 osd=0 3 2024-03-27T21:41:40.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.610+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b80a90d0 osd=0 3 2024-03-27T21:41:40.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.610+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b80a90d0 osd=0 3 2024-03-27T21:41:40.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.610+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_remove -1 2 2024-03-27T21:41:40.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.610+0000 7f04bd4b4640 20 client.5115.objecter get_session s=0x7f04b80a90d0 osd=0 3 2024-03-27T21:41:40.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.610+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign 0 2 2024-03-27T21:41:40.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.610+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b80a90d0 osd=0 4 2024-03-27T21:41:40.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.610+0000 7f04bd4b4640 10 client.5115.objecter _send_command 2 2024-03-27T21:41:40.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.610+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f04b8069470 con 0x7f04b81da010 2024-03-27T21:41:40.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.612+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f04b8069470 con 0x7f04b81da010 2024-03-27T21:41:40.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.612+0000 7f049f7fe640 10 client.5115.objecter ms_dispatch 0x7f04b8000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:40.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.612+0000 7f049f7fe640 10 client.5115.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:40.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.612+0000 7f049f7fe640 20 client.5115.objecter put_session s=0x7f04b80a90d0 osd=0 4 2024-03-27T21:41:40.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.612+0000 7f049f7fe640 15 client.5115.objecter _session_command_op_remove 0 2 2024-03-27T21:41:40.598 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:41:40.599 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:41:40.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.612+0000 7f04bd4b4640 10 client.5115.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:40.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.612+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign -1 3 2024-03-27T21:41:40.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.612+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f04b81da3f0 0x7f04b81347d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:40.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.612+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b814bd20 osd=1 3 2024-03-27T21:41:40.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.612+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b814bd20 osd=1 3 2024-03-27T21:41:40.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.612+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b814bd20 osd=1 3 2024-03-27T21:41:40.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.612+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_remove -1 3 2024-03-27T21:41:40.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.612+0000 7f04bd4b4640 20 client.5115.objecter get_session s=0x7f04b814bd20 osd=1 3 2024-03-27T21:41:40.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.612+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign 1 3 2024-03-27T21:41:40.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.612+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b814bd20 osd=1 4 2024-03-27T21:41:40.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.612+0000 7f04bd4b4640 10 client.5115.objecter _send_command 3 2024-03-27T21:41:40.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.612+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04b814be90 con 0x7f04b81da3f0 2024-03-27T21:41:40.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.612+0000 7f04b6ffd640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f04b81da3f0 0x7f04b81347d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:40.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.612+0000 7f04b6ffd640 10 monclient: get_auth_request con 0x7f04b81da3f0 auth_method 0 2024-03-27T21:41:40.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.613+0000 7f049f7fe640 10 client.5115.objecter ms_handle_connect 0x7f04b81da3f0 2024-03-27T21:41:40.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.613+0000 7f04b6ffd640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f04b81da3f0 0x7f04b81347d0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:40.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.614+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f04b814be90 con 0x7f04b81da3f0 2024-03-27T21:41:40.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.614+0000 7f049f7fe640 10 client.5115.objecter ms_dispatch 0x7f04b8000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:41:40.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.614+0000 7f049f7fe640 10 client.5115.objecter _finish_command 3 = system:0 2024-03-27T21:41:40.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.614+0000 7f049f7fe640 20 client.5115.objecter put_session s=0x7f04b814bd20 osd=1 4 2024-03-27T21:41:40.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.614+0000 7f049f7fe640 15 client.5115.objecter _session_command_op_remove 1 3 2024-03-27T21:41:40.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.642+0000 7f04bd4b4640 10 client.5115.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:40.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.642+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign -1 4 2024-03-27T21:41:40.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.642+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b814bd20 osd=1 3 2024-03-27T21:41:40.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.642+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b814bd20 osd=1 3 2024-03-27T21:41:40.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.642+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b814bd20 osd=1 3 2024-03-27T21:41:40.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.642+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_remove -1 4 2024-03-27T21:41:40.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.642+0000 7f04bd4b4640 20 client.5115.objecter get_session s=0x7f04b814bd20 osd=1 3 2024-03-27T21:41:40.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.642+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign 1 4 2024-03-27T21:41:40.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.642+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b814bd20 osd=1 4 2024-03-27T21:41:40.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.642+0000 7f04bd4b4640 10 client.5115.objecter _send_command 4 2024-03-27T21:41:40.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.642+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f04b80b62c0 con 0x7f04b81da3f0 2024-03-27T21:41:40.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.644+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f04b80b62c0 con 0x7f04b81da3f0 2024-03-27T21:41:40.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.644+0000 7f049f7fe640 10 client.5115.objecter ms_dispatch 0x7f04b8000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:40.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.644+0000 7f049f7fe640 10 client.5115.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:40.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.644+0000 7f049f7fe640 20 client.5115.objecter put_session s=0x7f04b814bd20 osd=1 4 2024-03-27T21:41:40.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.644+0000 7f049f7fe640 15 client.5115.objecter _session_command_op_remove 1 4 2024-03-27T21:41:40.631 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:41:40.632 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:41:40.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.644+0000 7f04bd4b4640 10 client.5115.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:40.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.644+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign -1 5 2024-03-27T21:41:40.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.644+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f04b8066760 0x7f04b802b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:40.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.644+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b80b66d0 osd=2 3 2024-03-27T21:41:40.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.644+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b80b66d0 osd=2 3 2024-03-27T21:41:40.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.644+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b80b66d0 osd=2 3 2024-03-27T21:41:40.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.644+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_remove -1 5 2024-03-27T21:41:40.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.644+0000 7f04bd4b4640 20 client.5115.objecter get_session s=0x7f04b80b66d0 osd=2 3 2024-03-27T21:41:40.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.644+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign 2 5 2024-03-27T21:41:40.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.644+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b80b66d0 osd=2 4 2024-03-27T21:41:40.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.644+0000 7f04bd4b4640 10 client.5115.objecter _send_command 5 2024-03-27T21:41:40.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.644+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04b81e85d0 con 0x7f04b8066760 2024-03-27T21:41:40.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.645+0000 7f04b67fc640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f04b8066760 0x7f04b802b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:40.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.645+0000 7f04b67fc640 10 monclient: get_auth_request con 0x7f04b8066760 auth_method 0 2024-03-27T21:41:40.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.645+0000 7f049f7fe640 10 client.5115.objecter ms_handle_connect 0x7f04b8066760 2024-03-27T21:41:40.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.645+0000 7f04b67fc640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f04b8066760 0x7f04b802b170 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:40.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.646+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== osd.2 v2:172.21.15.60:6824/56434 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f04b81e85d0 con 0x7f04b8066760 2024-03-27T21:41:40.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.646+0000 7f049f7fe640 10 client.5115.objecter ms_dispatch 0x7f04b8000b90 command_reply(tid 5: 0 ) v1 2024-03-27T21:41:40.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.646+0000 7f049f7fe640 10 client.5115.objecter _finish_command 5 = system:0 2024-03-27T21:41:40.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.646+0000 7f049f7fe640 20 client.5115.objecter put_session s=0x7f04b80b66d0 osd=2 4 2024-03-27T21:41:40.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.646+0000 7f049f7fe640 15 client.5115.objecter _session_command_op_remove 2 5 2024-03-27T21:41:40.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.660+0000 7f04bd4b4640 10 client.5115.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:40.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.660+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign -1 6 2024-03-27T21:41:40.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.660+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b80b66d0 osd=2 3 2024-03-27T21:41:40.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.660+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b80b66d0 osd=2 3 2024-03-27T21:41:40.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.660+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b80b66d0 osd=2 3 2024-03-27T21:41:40.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.660+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_remove -1 6 2024-03-27T21:41:40.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.660+0000 7f04bd4b4640 20 client.5115.objecter get_session s=0x7f04b80b66d0 osd=2 3 2024-03-27T21:41:40.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.660+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign 2 6 2024-03-27T21:41:40.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.660+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b80b66d0 osd=2 4 2024-03-27T21:41:40.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.660+0000 7f04bd4b4640 10 client.5115.objecter _send_command 6 2024-03-27T21:41:40.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.660+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f04b8065cd0 con 0x7f04b8066760 2024-03-27T21:41:40.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.662+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== osd.2 v2:172.21.15.60:6824/56434 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f04b8065cd0 con 0x7f04b8066760 2024-03-27T21:41:40.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.662+0000 7f049f7fe640 10 client.5115.objecter ms_dispatch 0x7f04b8000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:40.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.662+0000 7f049f7fe640 10 client.5115.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:40.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.662+0000 7f049f7fe640 20 client.5115.objecter put_session s=0x7f04b80b66d0 osd=2 4 2024-03-27T21:41:40.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.662+0000 7f049f7fe640 15 client.5115.objecter _session_command_op_remove 2 6 2024-03-27T21:41:40.649 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2: {} 2024-03-27T21:41:40.650 INFO:teuthology.orchestra.run.smithi060.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-27T21:41:40.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.662+0000 7f04bd4b4640 10 client.5115.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:40.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.662+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign -1 7 2024-03-27T21:41:40.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.662+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f04b80b5890 0x7f04b81e6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:40.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.662+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b80660e0 osd=3 3 2024-03-27T21:41:40.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.662+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b80660e0 osd=3 3 2024-03-27T21:41:40.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.662+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b80660e0 osd=3 3 2024-03-27T21:41:40.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.662+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_remove -1 7 2024-03-27T21:41:40.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.662+0000 7f04bd4b4640 20 client.5115.objecter get_session s=0x7f04b80660e0 osd=3 3 2024-03-27T21:41:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.662+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign 3 7 2024-03-27T21:41:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.662+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b80660e0 osd=3 4 2024-03-27T21:41:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.662+0000 7f04bd4b4640 10 client.5115.objecter _send_command 7 2024-03-27T21:41:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.662+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04b81ebe60 con 0x7f04b80b5890 2024-03-27T21:41:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.662+0000 7f04b77fe640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f04b80b5890 0x7f04b81e6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.663+0000 7f04b77fe640 10 monclient: get_auth_request con 0x7f04b80b5890 auth_method 0 2024-03-27T21:41:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.663+0000 7f04b77fe640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f04b80b5890 0x7f04b81e6ec0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.663+0000 7f049f7fe640 10 client.5115.objecter ms_handle_connect 0x7f04b80b5890 2024-03-27T21:41:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.663+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f04b81ebe60 con 0x7f04b80b5890 2024-03-27T21:41:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.663+0000 7f049f7fe640 10 client.5115.objecter ms_dispatch 0x7f04b8000b90 command_reply(tid 7: 0 ) v1 2024-03-27T21:41:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.663+0000 7f049f7fe640 10 client.5115.objecter _finish_command 7 = system:0 2024-03-27T21:41:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.663+0000 7f049f7fe640 20 client.5115.objecter put_session s=0x7f04b80660e0 osd=3 4 2024-03-27T21:41:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.663+0000 7f049f7fe640 15 client.5115.objecter _session_command_op_remove 3 7 2024-03-27T21:41:40.667 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:40.681+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:40.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.691+0000 7f04bd4b4640 10 client.5115.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:40.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.691+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign -1 8 2024-03-27T21:41:40.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.691+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b80660e0 osd=3 3 2024-03-27T21:41:40.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.691+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b80660e0 osd=3 3 2024-03-27T21:41:40.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.691+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b80660e0 osd=3 3 2024-03-27T21:41:40.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.691+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_remove -1 8 2024-03-27T21:41:40.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.691+0000 7f04bd4b4640 20 client.5115.objecter get_session s=0x7f04b80660e0 osd=3 3 2024-03-27T21:41:40.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.691+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign 3 8 2024-03-27T21:41:40.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.691+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b80660e0 osd=3 4 2024-03-27T21:41:40.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.691+0000 7f04bd4b4640 10 client.5115.objecter _send_command 8 2024-03-27T21:41:40.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.691+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f04b80b4e00 con 0x7f04b80b5890 2024-03-27T21:41:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.692+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f04b80b4e00 con 0x7f04b80b5890 2024-03-27T21:41:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.693+0000 7f049f7fe640 10 client.5115.objecter ms_dispatch 0x7f04b8000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.693+0000 7f049f7fe640 10 client.5115.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.693+0000 7f049f7fe640 20 client.5115.objecter put_session s=0x7f04b80660e0 osd=3 4 2024-03-27T21:41:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.693+0000 7f049f7fe640 15 client.5115.objecter _session_command_op_remove 3 8 2024-03-27T21:41:40.679 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:41:40.680 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:41:40.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.693+0000 7f04bd4b4640 10 client.5115.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:40.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.693+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign -1 9 2024-03-27T21:41:40.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.693+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f04b80b4f70 0x7f04b81ea750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:40.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.693+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b80b4e00 osd=4 3 2024-03-27T21:41:40.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.693+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b80b4e00 osd=4 3 2024-03-27T21:41:40.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.693+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b80b4e00 osd=4 3 2024-03-27T21:41:40.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.693+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_remove -1 9 2024-03-27T21:41:40.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.693+0000 7f04bd4b4640 20 client.5115.objecter get_session s=0x7f04b80b4e00 osd=4 3 2024-03-27T21:41:40.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.693+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign 4 9 2024-03-27T21:41:40.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.693+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b80b4e00 osd=4 4 2024-03-27T21:41:40.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.693+0000 7f04bd4b4640 10 client.5115.objecter _send_command 9 2024-03-27T21:41:40.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.693+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04b802c380 con 0x7f04b80b4f70 2024-03-27T21:41:40.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.693+0000 7f04b6ffd640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f04b80b4f70 0x7f04b81ea750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:40.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.693+0000 7f04b6ffd640 10 monclient: get_auth_request con 0x7f04b80b4f70 auth_method 0 2024-03-27T21:41:40.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.694+0000 7f049f7fe640 10 client.5115.objecter ms_handle_connect 0x7f04b80b4f70 2024-03-27T21:41:40.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.694+0000 7f04b6ffd640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f04b80b4f70 0x7f04b81ea750 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:40.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.694+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f04b802c380 con 0x7f04b80b4f70 2024-03-27T21:41:40.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.694+0000 7f049f7fe640 10 client.5115.objecter ms_dispatch 0x7f04b8000b90 command_reply(tid 9: 0 ) v1 2024-03-27T21:41:40.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.694+0000 7f049f7fe640 10 client.5115.objecter _finish_command 9 = system:0 2024-03-27T21:41:40.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.694+0000 7f049f7fe640 20 client.5115.objecter put_session s=0x7f04b80b4e00 osd=4 4 2024-03-27T21:41:40.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.694+0000 7f049f7fe640 15 client.5115.objecter _session_command_op_remove 4 9 2024-03-27T21:41:40.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.708+0000 7f04bd4b4640 10 client.5115.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:40.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.708+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign -1 10 2024-03-27T21:41:40.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.708+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b80b4e00 osd=4 3 2024-03-27T21:41:40.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.708+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b80b4e00 osd=4 3 2024-03-27T21:41:40.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.708+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b80b4e00 osd=4 3 2024-03-27T21:41:40.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.708+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_remove -1 10 2024-03-27T21:41:40.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.708+0000 7f04bd4b4640 20 client.5115.objecter get_session s=0x7f04b80b4e00 osd=4 3 2024-03-27T21:41:40.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.708+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign 4 10 2024-03-27T21:41:40.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.708+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b80b4e00 osd=4 4 2024-03-27T21:41:40.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.708+0000 7f04bd4b4640 10 client.5115.objecter _send_command 10 2024-03-27T21:41:40.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.708+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f04b802c380 con 0x7f04b80b4f70 2024-03-27T21:41:40.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.710+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f04b802c380 con 0x7f04b80b4f70 2024-03-27T21:41:40.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.710+0000 7f049f7fe640 10 client.5115.objecter ms_dispatch 0x7f04b8000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:40.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.710+0000 7f049f7fe640 10 client.5115.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:40.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.710+0000 7f049f7fe640 20 client.5115.objecter put_session s=0x7f04b80b4e00 osd=4 4 2024-03-27T21:41:40.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.710+0000 7f049f7fe640 15 client.5115.objecter _session_command_op_remove 4 10 2024-03-27T21:41:40.697 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:41:40.698 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:41:40.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.710+0000 7f04bd4b4640 10 client.5115.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:40.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.710+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign -1 11 2024-03-27T21:41:40.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.710+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f04b81eb1a0 0x7f04b81eb600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:40.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.710+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b802c380 osd=5 3 2024-03-27T21:41:40.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.710+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b802c380 osd=5 3 2024-03-27T21:41:40.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.710+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b802c380 osd=5 3 2024-03-27T21:41:40.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.710+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_remove -1 11 2024-03-27T21:41:40.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.710+0000 7f04bd4b4640 20 client.5115.objecter get_session s=0x7f04b802c380 osd=5 3 2024-03-27T21:41:40.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.710+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign 5 11 2024-03-27T21:41:40.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.710+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b802c380 osd=5 4 2024-03-27T21:41:40.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.710+0000 7f04bd4b4640 10 client.5115.objecter _send_command 11 2024-03-27T21:41:40.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.710+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04b81ebb40 con 0x7f04b81eb1a0 2024-03-27T21:41:40.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.711+0000 7f04b67fc640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f04b81eb1a0 0x7f04b81eb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:40.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.711+0000 7f04b67fc640 10 monclient: get_auth_request con 0x7f04b81eb1a0 auth_method 0 2024-03-27T21:41:40.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.711+0000 7f049f7fe640 10 client.5115.objecter ms_handle_connect 0x7f04b81eb1a0 2024-03-27T21:41:40.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.711+0000 7f04b67fc640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f04b81eb1a0 0x7f04b81eb600 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:40.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.712+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f04b81ebb40 con 0x7f04b81eb1a0 2024-03-27T21:41:40.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.712+0000 7f049f7fe640 10 client.5115.objecter ms_dispatch 0x7f04b8000b90 command_reply(tid 11: 0 ) v1 2024-03-27T21:41:40.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.712+0000 7f049f7fe640 10 client.5115.objecter _finish_command 11 = system:0 2024-03-27T21:41:40.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.712+0000 7f049f7fe640 20 client.5115.objecter put_session s=0x7f04b802c380 osd=5 4 2024-03-27T21:41:40.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.712+0000 7f049f7fe640 15 client.5115.objecter _session_command_op_remove 5 11 2024-03-27T21:41:40.718 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:40.719 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:40.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.739+0000 7f04bd4b4640 10 client.5115.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:40.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.739+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign -1 12 2024-03-27T21:41:40.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.739+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b802c380 osd=5 3 2024-03-27T21:41:40.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.739+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b802c380 osd=5 3 2024-03-27T21:41:40.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.739+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b802c380 osd=5 3 2024-03-27T21:41:40.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.739+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_remove -1 12 2024-03-27T21:41:40.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.739+0000 7f04bd4b4640 20 client.5115.objecter get_session s=0x7f04b802c380 osd=5 3 2024-03-27T21:41:40.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.739+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign 5 12 2024-03-27T21:41:40.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.739+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b802c380 osd=5 4 2024-03-27T21:41:40.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.739+0000 7f04bd4b4640 10 client.5115.objecter _send_command 12 2024-03-27T21:41:40.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.739+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f04b81ebb40 con 0x7f04b81eb1a0 2024-03-27T21:41:40.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.741+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f04b81ebb40 con 0x7f04b81eb1a0 2024-03-27T21:41:40.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.741+0000 7f049f7fe640 10 client.5115.objecter ms_dispatch 0x7f04b8000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:40.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.741+0000 7f049f7fe640 10 client.5115.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:40.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.741+0000 7f049f7fe640 20 client.5115.objecter put_session s=0x7f04b802c380 osd=5 4 2024-03-27T21:41:40.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.741+0000 7f049f7fe640 15 client.5115.objecter _session_command_op_remove 5 12 2024-03-27T21:41:40.728 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:41:40.729 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:41:40.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.741+0000 7f04bd4b4640 10 client.5115.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:40.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.741+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign -1 13 2024-03-27T21:41:40.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.741+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f04b81f0500 0x7f04b81f28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:40.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.741+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b81ebb40 osd=6 3 2024-03-27T21:41:40.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.741+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b81ebb40 osd=6 3 2024-03-27T21:41:40.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.741+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b81ebb40 osd=6 3 2024-03-27T21:41:40.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.741+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_remove -1 13 2024-03-27T21:41:40.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.741+0000 7f04bd4b4640 20 client.5115.objecter get_session s=0x7f04b81ebb40 osd=6 3 2024-03-27T21:41:40.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.741+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign 6 13 2024-03-27T21:41:40.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.741+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b81ebb40 osd=6 4 2024-03-27T21:41:40.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.741+0000 7f04bd4b4640 10 client.5115.objecter _send_command 13 2024-03-27T21:41:40.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.741+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04b802c540 con 0x7f04b81f0500 2024-03-27T21:41:40.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.742+0000 7f04b77fe640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f04b81f0500 0x7f04b81f28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:40.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.742+0000 7f04b77fe640 10 monclient: get_auth_request con 0x7f04b81f0500 auth_method 0 2024-03-27T21:41:40.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.742+0000 7f049f7fe640 10 client.5115.objecter ms_handle_connect 0x7f04b81f0500 2024-03-27T21:41:40.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.742+0000 7f04b77fe640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f04b81f0500 0x7f04b81f28f0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:40.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.743+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f04b802c540 con 0x7f04b81f0500 2024-03-27T21:41:40.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.743+0000 7f049f7fe640 10 client.5115.objecter ms_dispatch 0x7f04b8000b90 command_reply(tid 13: 0 ) v1 2024-03-27T21:41:40.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.743+0000 7f049f7fe640 10 client.5115.objecter _finish_command 13 = system:0 2024-03-27T21:41:40.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.743+0000 7f049f7fe640 20 client.5115.objecter put_session s=0x7f04b81ebb40 osd=6 4 2024-03-27T21:41:40.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.743+0000 7f049f7fe640 15 client.5115.objecter _session_command_op_remove 6 13 2024-03-27T21:41:40.731 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-27T21:41:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.768+0000 7f04bd4b4640 10 client.5115.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.768+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign -1 14 2024-03-27T21:41:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.768+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b81ebb40 osd=6 3 2024-03-27T21:41:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.768+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b81ebb40 osd=6 3 2024-03-27T21:41:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.768+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b81ebb40 osd=6 3 2024-03-27T21:41:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.768+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_remove -1 14 2024-03-27T21:41:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.768+0000 7f04bd4b4640 20 client.5115.objecter get_session s=0x7f04b81ebb40 osd=6 3 2024-03-27T21:41:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.768+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign 6 14 2024-03-27T21:41:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.768+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b81ebb40 osd=6 4 2024-03-27T21:41:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.768+0000 7f04bd4b4640 10 client.5115.objecter _send_command 14 2024-03-27T21:41:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.768+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f04b802c540 con 0x7f04b81f0500 2024-03-27T21:41:40.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.770+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f04b802c540 con 0x7f04b81f0500 2024-03-27T21:41:40.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.770+0000 7f049f7fe640 10 client.5115.objecter ms_dispatch 0x7f04b8000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:40.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.770+0000 7f049f7fe640 10 client.5115.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:40.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.770+0000 7f049f7fe640 20 client.5115.objecter put_session s=0x7f04b81ebb40 osd=6 4 2024-03-27T21:41:40.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.770+0000 7f049f7fe640 15 client.5115.objecter _session_command_op_remove 6 14 2024-03-27T21:41:40.757 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:41:40.757 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:41:40.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.770+0000 7f04bd4b4640 10 client.5115.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:40.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.770+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign -1 15 2024-03-27T21:41:40.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.770+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f04b81f3340 0x7f04b81f5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:40.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.770+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b802c540 osd=7 3 2024-03-27T21:41:40.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.770+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b802c540 osd=7 3 2024-03-27T21:41:40.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.770+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b802c540 osd=7 3 2024-03-27T21:41:40.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.770+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_remove -1 15 2024-03-27T21:41:40.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.770+0000 7f04bd4b4640 20 client.5115.objecter get_session s=0x7f04b802c540 osd=7 3 2024-03-27T21:41:40.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.770+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign 7 15 2024-03-27T21:41:40.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.770+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b802c540 osd=7 4 2024-03-27T21:41:40.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.770+0000 7f04bd4b4640 10 client.5115.objecter _send_command 15 2024-03-27T21:41:40.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.770+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04b81f86a0 con 0x7f04b81f3340 2024-03-27T21:41:40.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.770+0000 7f04b6ffd640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f04b81f3340 0x7f04b81f5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:40.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.770+0000 7f04b6ffd640 10 monclient: get_auth_request con 0x7f04b81f3340 auth_method 0 2024-03-27T21:41:40.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.771+0000 7f049f7fe640 10 client.5115.objecter ms_handle_connect 0x7f04b81f3340 2024-03-27T21:41:40.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.771+0000 7f04b6ffd640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f04b81f3340 0x7f04b81f5730 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:40.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.771+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f04b81f86a0 con 0x7f04b81f3340 2024-03-27T21:41:40.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.771+0000 7f049f7fe640 10 client.5115.objecter ms_dispatch 0x7f04b8000b90 command_reply(tid 15: 0 ) v1 2024-03-27T21:41:40.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.771+0000 7f049f7fe640 10 client.5115.objecter _finish_command 15 = system:0 2024-03-27T21:41:40.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.772+0000 7f049f7fe640 20 client.5115.objecter put_session s=0x7f04b802c540 osd=7 4 2024-03-27T21:41:40.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.772+0000 7f049f7fe640 15 client.5115.objecter _session_command_op_remove 7 15 2024-03-27T21:41:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.788+0000 7f04bd4b4640 10 client.5115.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.788+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign -1 16 2024-03-27T21:41:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.788+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b802c540 osd=7 3 2024-03-27T21:41:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.788+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b802c540 osd=7 3 2024-03-27T21:41:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.788+0000 7f04bd4b4640 20 client.5115.objecter _get_session s=0x7f04b802c540 osd=7 3 2024-03-27T21:41:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.788+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_remove -1 16 2024-03-27T21:41:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.788+0000 7f04bd4b4640 20 client.5115.objecter get_session s=0x7f04b802c540 osd=7 3 2024-03-27T21:41:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.788+0000 7f04bd4b4640 15 client.5115.objecter _session_command_op_assign 7 16 2024-03-27T21:41:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.788+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b802c540 osd=7 4 2024-03-27T21:41:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.788+0000 7f04bd4b4640 10 client.5115.objecter _send_command 16 2024-03-27T21:41:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.788+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f04b81f86a0 con 0x7f04b81f3340 2024-03-27T21:41:40.776 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:40.789+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:40.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.790+0000 7f049f7fe640 1 -- 172.21.15.60:0/1915345576 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f04b81f86a0 con 0x7f04b81f3340 2024-03-27T21:41:40.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.790+0000 7f049f7fe640 10 client.5115.objecter ms_dispatch 0x7f04b8000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:40.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.790+0000 7f049f7fe640 10 client.5115.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:40.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.790+0000 7f049f7fe640 20 client.5115.objecter put_session s=0x7f04b802c540 osd=7 4 2024-03-27T21:41:40.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.790+0000 7f049f7fe640 15 client.5115.objecter _session_command_op_remove 7 16 2024-03-27T21:41:40.777 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:41:40.780 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:41:40.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 10 librados: watch_flush enter 2024-03-27T21:41:40.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 10 librados: watch_flush exit 2024-03-27T21:41:40.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 10 client.5115.objecter close_session for osd.0 2024-03-27T21:41:40.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f04b81da010 msgr2=0x7f04b8133d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:40.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f04b81da010 0x7f04b8133d80 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b80a90d0 osd=0 1 2024-03-27T21:41:40.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 10 client.5115.objecter close_session for osd.1 2024-03-27T21:41:40.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f04b81da3f0 msgr2=0x7f04b81347d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:40.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f04b81da3f0 0x7f04b81347d0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b814bd20 osd=1 1 2024-03-27T21:41:40.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 10 client.5115.objecter close_session for osd.2 2024-03-27T21:41:40.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f04b8066760 msgr2=0x7f04b802b170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:40.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f04b8066760 0x7f04b802b170 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b80b66d0 osd=2 1 2024-03-27T21:41:40.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 10 client.5115.objecter close_session for osd.3 2024-03-27T21:41:40.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f04b80b5890 msgr2=0x7f04b81e6ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:40.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f04b80b5890 0x7f04b81e6ec0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b80660e0 osd=3 1 2024-03-27T21:41:40.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 10 client.5115.objecter close_session for osd.4 2024-03-27T21:41:40.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f04b80b4f70 msgr2=0x7f04b81ea750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:40.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f04b80b4f70 0x7f04b81ea750 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b80b4e00 osd=4 1 2024-03-27T21:41:40.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 10 client.5115.objecter close_session for osd.5 2024-03-27T21:41:40.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f04b81eb1a0 msgr2=0x7f04b81eb600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:40.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f04b81eb1a0 0x7f04b81eb600 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b802c380 osd=5 1 2024-03-27T21:41:40.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 10 client.5115.objecter close_session for osd.6 2024-03-27T21:41:40.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f04b81f0500 msgr2=0x7f04b81f28f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:40.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f04b81f0500 0x7f04b81f28f0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b81ebb40 osd=6 1 2024-03-27T21:41:40.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 10 client.5115.objecter close_session for osd.7 2024-03-27T21:41:40.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f04b81f3340 msgr2=0x7f04b81f5730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:40.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f04b81f3340 0x7f04b81f5730 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 20 client.5115.objecter put_session s=0x7f04b802c540 osd=7 1 2024-03-27T21:41:40.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 20 client.5115.objecter shutdown clearing up homeless session... 2024-03-27T21:41:40.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 10 client.5115.objecter successfully canceled tick 2024-03-27T21:41:40.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04b77fe640 1 -- 172.21.15.60:0/1915345576 reap_dead start 2024-03-27T21:41:40.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04b77fe640 1 -- 172.21.15.60:0/1915345576 reap_dead start 2024-03-27T21:41:40.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04b77fe640 1 -- 172.21.15.60:0/1915345576 reap_dead start 2024-03-27T21:41:40.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04b77fe640 1 -- 172.21.15.60:0/1915345576 reap_dead start 2024-03-27T21:41:40.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04b77fe640 1 -- 172.21.15.60:0/1915345576 reap_dead start 2024-03-27T21:41:40.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f04a406c5c0 msgr2=0x7f04a406ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:40.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f04a406c5c0 0x7f04a406ea80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f04a8009b40 tx=0x7f04a8009190 comp rx=0 tx=0).stop 2024-03-27T21:41:40.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 10 monclient: shutdown 2024-03-27T21:41:40.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:40.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04b81416f0 msgr2=0x7f04b813e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:40.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04b81416f0 0x7f04b813e890 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f04ac000c00 tx=0x7f04ac0046c0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 shutdown_connections 2024-03-27T21:41:40.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f04a406c5c0 0x7f04a406ea80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 --2- 172.21.15.60:0/1915345576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04b81416f0 0x7f04b813e890 secure :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f04ac000c00 tx=0x7f04ac0046c0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.791+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 >> 172.21.15.60:0/1915345576 conn(0x7f04b80608e0 msgr2=0x7f04b8141b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:40.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.792+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 shutdown_connections 2024-03-27T21:41:40.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.792+0000 7f04bd4b4640 1 -- 172.21.15.60:0/1915345576 wait complete. 2024-03-27T21:41:40.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.792+0000 7f04bd4b4640 1 librados: shutdown 2024-03-27T21:41:40.869 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:40.883+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:40.870 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:40.870 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:40.879 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:41:40.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.930+0000 7f9c397fa640 1 -- 172.21.15.60:0/3045954623 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v72) v1 ==== 87+0+0 (secure 0 0 0) 0x7f9c30057e20 con 0x7f9c3c071a50 2024-03-27T21:41:40.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.930+0000 7f9c397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set", "key": "noscrub"}] 2024-03-27T21:41:40.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.930+0000 7f9c397fa640 10 monclient: _finish_command 2 = system:0 noscrub is set 2024-03-27T21:41:40.920 INFO:teuthology.orchestra.run.smithi060.stderr:noscrub is set 2024-03-27T21:41:40.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.933+0000 7f9c42bf0640 10 librados: watch_flush enter 2024-03-27T21:41:40.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.933+0000 7f9c42bf0640 10 librados: watch_flush exit 2024-03-27T21:41:40.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.933+0000 7f9c42bf0640 20 client.5109.objecter shutdown clearing up homeless session... 2024-03-27T21:41:40.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.933+0000 7f9c42bf0640 10 client.5109.objecter successfully canceled tick 2024-03-27T21:41:40.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.933+0000 7f9c42bf0640 1 -- 172.21.15.60:0/3045954623 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9c2006c590 msgr2=0x7f9c2006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:40.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.933+0000 7f9c42bf0640 1 --2- 172.21.15.60:0/3045954623 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9c2006c590 0x7f9c2006ea50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9c28007fd0 tx=0x7f9c28009040 comp rx=0 tx=0).stop 2024-03-27T21:41:40.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.933+0000 7f9c42bf0640 10 monclient: shutdown 2024-03-27T21:41:40.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.933+0000 7f9c42bf0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:40.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.933+0000 7f9c42bf0640 1 -- 172.21.15.60:0/3045954623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c3c071a50 msgr2=0x7f9c3c0b67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:40.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.933+0000 7f9c42bf0640 1 --2- 172.21.15.60:0/3045954623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c3c071a50 0x7f9c3c0b67d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9c3000ebe0 tx=0x7f9c3000c520 comp rx=0 tx=0).stop 2024-03-27T21:41:40.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.933+0000 7f9c42bf0640 1 -- 172.21.15.60:0/3045954623 shutdown_connections 2024-03-27T21:41:40.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.933+0000 7f9c42bf0640 1 --2- 172.21.15.60:0/3045954623 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9c2006c590 0x7f9c2006ea50 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.933+0000 7f9c42bf0640 1 --2- 172.21.15.60:0/3045954623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c3c13ec50 0x7f9c3c133050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.933+0000 7f9c42bf0640 1 --2- 172.21.15.60:0/3045954623 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9c3c13e810 0x7f9c3c1396e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.933+0000 7f9c42bf0640 1 --2- 172.21.15.60:0/3045954623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c3c071a50 0x7f9c3c0b67d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:40.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.933+0000 7f9c42bf0640 1 -- 172.21.15.60:0/3045954623 >> 172.21.15.60:0/3045954623 conn(0x7f9c3c0b7300 msgr2=0x7f9c3c13b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:40.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.934+0000 7f9c42bf0640 1 -- 172.21.15.60:0/3045954623 shutdown_connections 2024-03-27T21:41:40.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.934+0000 7f9c42bf0640 1 -- 172.21.15.60:0/3045954623 wait complete. 2024-03-27T21:41:40.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:40.934+0000 7f9c42bf0640 1 librados: shutdown 2024-03-27T21:41:40.978 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:40.992+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:41.025 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:41.025 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:41.037 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:41:41.072 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:41.086+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:41.173 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:41.187+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:41.215 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:41.215 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:41.227 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:41:41.273 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:41.287+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:41.334 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:41.335 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:41.344 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:41:41.374 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:41.389+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:41.476 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:41.490+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:41.546 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:41.546 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:41.559 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:41:41.602 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:41.616+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:41.678 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:41.692+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:41.688 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:41.688 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:41.701 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:41:41.786 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:41.800+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:41.849 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:41.849 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:41.858 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:41:41.899 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:41.913+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:41.980 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:41.994+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:42.019 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:42.020 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:42.031 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:41:42.108 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-03-27T21:41:42.108 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T21:41:42.116 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:42.130+0000 7fb850f34640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 56426) UID: 0 2024-03-27T21:41:42.181 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:42.195+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:42.184 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:42.184 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:42.193 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:41:42.282 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:42.296+0000 7fb850f34640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 56426) UID: 0 2024-03-27T21:41:42.383 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:42.397+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:42.405 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:42.405 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:42.417 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:41:42.483 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:42.497+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:42.534 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:42.534 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:42.543 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:41:42.585 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:42.599+0000 7fb850f34640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 56426) UID: 0 2024-03-27T21:41:42.685 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:42.699+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:42.689 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:42.689 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:42.698 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:41:42.786 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:42.799+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:42.790 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:41:42.843 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:42.843 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:42.852 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:41:42.888 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:42.901+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:42.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.904+0000 7f547328c640 10 monclient: get_monmap_and_config 2024-03-27T21:41:42.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.904+0000 7f547328c640 10 monclient: build_initial_monmap 2024-03-27T21:41:42.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.904+0000 7f547328c640 10 monclient: monmap: 2024-03-27T21:41:42.890 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:42.890 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:42.890 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:42.905197+0000 2024-03-27T21:41:42.890 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:42.905197+0000 2024-03-27T21:41:42.890 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:42.890 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:42.891 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:42.891 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:42.891 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:42.891 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:42.891 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:42.891 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:42.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.904+0000 7f547328c640 1 Processor -- start 2024-03-27T21:41:42.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.905+0000 7f547328c640 1 -- start start 2024-03-27T21:41:42.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.905+0000 7f547328c640 10 monclient: init 2024-03-27T21:41:42.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.905+0000 7f547328c640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:42.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.905+0000 7f547328c640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-03-27T21:41:42.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.905+0000 7f547328c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f546c06f900 0x7f546c13fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:42.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.905+0000 7f547328c640 10 monclient(hunting): picked mon.noname-b con 0x7f546c06f900 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:42.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.905+0000 7f547328c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f546c06fce0 0x7f546c0af850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:42.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.905+0000 7f547328c640 10 monclient(hunting): picked mon.noname-c con 0x7f546c06fce0 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:42.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.905+0000 7f547328c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f546c138bf0 0x7f546c139080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:42.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.905+0000 7f547328c640 10 monclient(hunting): picked mon.noname-a con 0x7f546c138bf0 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:42.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.905+0000 7f547328c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:42.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.905+0000 7f547328c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f546c140100 con 0x7f546c138bf0 2024-03-27T21:41:42.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.905+0000 7f547328c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:42.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.905+0000 7f547328c640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f546c071b50 con 0x7f546c06fce0 2024-03-27T21:41:42.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.905+0000 7f547328c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:42.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.905+0000 7f547328c640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546c1395c0 con 0x7f546c06f900 2024-03-27T21:41:42.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.905+0000 7f547328c640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:42.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.905+0000 7f547328c640 10 monclient(hunting): authenticate will time out at 944.126762s 2024-03-27T21:41:42.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.905+0000 7f5471829640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f546c138bf0 0x7f546c139080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:42.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.905+0000 7f5471028640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f546c06f900 0x7f546c13fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:42.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471829640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f546c138bf0 0x7f546c139080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41736/0 (socket says 172.21.15.60:41736) 2024-03-27T21:41:42.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471829640 1 -- 172.21.15.60:0/1964670278 learned_addr learned my addr 172.21.15.60:0/1964670278 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:42.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471829640 10 monclient(hunting): get_auth_request con 0x7f546c138bf0 auth_method 0 2024-03-27T21:41:42.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471829640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:42.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471829640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:42.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471829640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:42.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471028640 10 monclient(hunting): get_auth_request con 0x7f546c06f900 auth_method 0 2024-03-27T21:41:42.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471028640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:42.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471028640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:42.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471028640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:42.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471829640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:42.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471829640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:42.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471829640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:42.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471028640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:42.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471028640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:42.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471028640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:42.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5470827640 1 --2- 172.21.15.60:0/1964670278 >> v2:172.21.15.191:3300/0 conn(0x7f546c06fce0 0x7f546c0af850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:42.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471028640 10 monclient(hunting): handle_auth_done global_id 4877 payload 274 2024-03-27T21:41:42.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471028640 1 -- 172.21.15.60:0/1964670278 >> v2:172.21.15.191:3300/0 conn(0x7f546c06fce0 msgr2=0x7f546c0af850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:42.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471028640 1 --2- 172.21.15.60:0/1964670278 >> v2:172.21.15.191:3300/0 conn(0x7f546c06fce0 0x7f546c0af850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:42.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471028640 1 -- 172.21.15.60:0/1964670278 >> v2:172.21.15.60:3300/0 conn(0x7f546c138bf0 msgr2=0x7f546c139080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:42.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471028640 1 --2- 172.21.15.60:0/1964670278 >> v2:172.21.15.60:3300/0 conn(0x7f546c138bf0 0x7f546c139080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:42.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471028640 10 monclient: _finish_hunting 0 2024-03-27T21:41:42.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471028640 1 monclient: found mon.noname-b 2024-03-27T21:41:42.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471028640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:42.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471028640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:42.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471028640 1 -- 172.21.15.60:0/1964670278 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546c13f690 con 0x7f546c06f900 2024-03-27T21:41:42.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5470827640 10 monclient: get_auth_request con 0x7f546c06fce0 auth_method 0 2024-03-27T21:41:42.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5470827640 1 --2- 172.21.15.60:0/1964670278 >> v2:172.21.15.191:3300/0 conn(0x7f546c06fce0 0x7f546c0af850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:41:42.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.906+0000 7f5471829640 1 --2- 172.21.15.60:0/1964670278 >> v2:172.21.15.60:3300/0 conn(0x7f546c138bf0 0x7f546c139080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:42.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.907+0000 7f5471028640 1 --2- 172.21.15.60:0/1964670278 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f546c06f900 0x7f546c13fbc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f546c06ce40 tx=0x7f546802be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47f786c1f7838a7f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:42.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.907+0000 7f5463fff640 1 -- 172.21.15.60:0/1964670278 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546802cbd0 con 0x7f546c06f900 2024-03-27T21:41:42.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.907+0000 7f5463fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:42.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.907+0000 7f5463fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:42.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.907+0000 7f5463fff640 10 monclient: dump: 2024-03-27T21:41:42.898 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:42.899 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:42.899 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:42.899 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:42.899 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:42.899 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:42.899 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:42.899 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:42.899 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:42.899 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:42.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.907+0000 7f5463fff640 10 monclient: _finish_auth 0 2024-03-27T21:41:42.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.907+0000 7f5463fff640 10 monclient: _check_auth_tickets 2024-03-27T21:41:42.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.907+0000 7f5463fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:42.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.907+0000 7f547328c640 5 monclient: authenticate success, global_id 4877 2024-03-27T21:41:42.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.907+0000 7f5463fff640 1 -- 172.21.15.60:0/1964670278 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546802cd30 con 0x7f546c06f900 2024-03-27T21:41:42.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.907+0000 7f547328c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:42.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.907+0000 7f5463fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:42.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.907+0000 7f547328c640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:42.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.907+0000 7f547328c640 4 set_mon_vals no callback set 2024-03-27T21:41:42.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.907+0000 7f5463fff640 1 -- 172.21.15.60:0/1964670278 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546802cea0 con 0x7f546c06f900 2024-03-27T21:41:42.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.907+0000 7f5463fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:42.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.907+0000 7f5463fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:42.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.907+0000 7f5463fff640 10 monclient: dump: 2024-03-27T21:41:42.900 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:42.900 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:42.900 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:42.900 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:42.900 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:42.900 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:42.901 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:42.901 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:42.901 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:42.901 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:42.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.908+0000 7f547328c640 10 monclient: shutdown 2024-03-27T21:41:42.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.908+0000 7f547328c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:42.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.908+0000 7f547328c640 1 -- 172.21.15.60:0/1964670278 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f546c06f900 msgr2=0x7f546c13fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:42.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.908+0000 7f547328c640 1 --2- 172.21.15.60:0/1964670278 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f546c06f900 0x7f546c13fbc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f546c06ce40 tx=0x7f546802be20 comp rx=0 tx=0).stop 2024-03-27T21:41:42.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.908+0000 7f547328c640 1 -- 172.21.15.60:0/1964670278 shutdown_connections 2024-03-27T21:41:42.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.908+0000 7f547328c640 1 --2- 172.21.15.60:0/1964670278 >> v2:172.21.15.60:3300/0 conn(0x7f546c138bf0 0x7f546c139080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:42.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.908+0000 7f547328c640 1 --2- 172.21.15.60:0/1964670278 >> v2:172.21.15.191:3300/0 conn(0x7f546c06fce0 0x7f546c0af850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:42.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.908+0000 7f547328c640 1 --2- 172.21.15.60:0/1964670278 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f546c06f900 0x7f546c13fbc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:42.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.908+0000 7f547328c640 1 -- 172.21.15.60:0/1964670278 >> 172.21.15.60:0/1964670278 conn(0x7f546c0b4e50 msgr2=0x7f546c0b5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:42.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.908+0000 7f547328c640 1 -- 172.21.15.60:0/1964670278 shutdown_connections 2024-03-27T21:41:42.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.908+0000 7f547328c640 1 -- 172.21.15.60:0/1964670278 wait complete. 2024-03-27T21:41:42.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.909+0000 7f547328c640 10 monclient: build_initial_monmap 2024-03-27T21:41:42.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.909+0000 7f547328c640 10 monclient: monmap: 2024-03-27T21:41:42.909 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:42.909 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:42.909 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:42.909 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:42.909 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:42.909 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:42.909 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:42.910 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:42.910 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:42.910 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:42.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.909+0000 7f547328c640 1 librados: starting msgr at 2024-03-27T21:41:42.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.909+0000 7f547328c640 1 librados: starting objecter 2024-03-27T21:41:42.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.909+0000 7f547328c640 1 Processor -- start 2024-03-27T21:41:42.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.909+0000 7f547328c640 1 -- start start 2024-03-27T21:41:42.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.909+0000 7f547328c640 1 librados: setting wanted keys 2024-03-27T21:41:42.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.909+0000 7f547328c640 1 librados: calling monclient init 2024-03-27T21:41:42.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.909+0000 7f547328c640 10 monclient: init 2024-03-27T21:41:42.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f547328c640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:42.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f547328c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:41:42.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f547328c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f546c06f900 0x7f546c135330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:42.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f547328c640 10 monclient(hunting): picked mon.noname-c con 0x7f546c06f900 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:42.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f547328c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f546c06fce0 0x7f546c1d9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:42.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f547328c640 10 monclient(hunting): picked mon.noname-b con 0x7f546c06fce0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:42.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f547328c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f546c138bf0 0x7f546c1d9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:42.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f5471028640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f546c06f900 0x7f546c135330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:42.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f547328c640 10 monclient(hunting): picked mon.noname-a con 0x7f546c138bf0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:42.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f547328c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:42.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f547328c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546c1da480 con 0x7f546c138bf0 2024-03-27T21:41:42.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f547328c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:42.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f547328c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546c1d9570 con 0x7f546c06fce0 2024-03-27T21:41:42.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f547328c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:42.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f547328c640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546c0ae520 con 0x7f546c06f900 2024-03-27T21:41:42.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f5471028640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f546c06f900 0x7f546c135330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:32768/0 (socket says 172.21.15.60:32768) 2024-03-27T21:41:42.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f5471829640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f546c138bf0 0x7f546c1d9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:42.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f5471028640 1 -- 172.21.15.60:0/4040627003 learned_addr learned my addr 172.21.15.60:0/4040627003 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:42.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f547328c640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:42.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f547328c640 10 monclient(hunting): authenticate will time out at 944.131794s 2024-03-27T21:41:42.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f5471028640 10 monclient(hunting): get_auth_request con 0x7f546c06f900 auth_method 0 2024-03-27T21:41:42.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f5471028640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:42.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f5471028640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:42.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f5471028640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:42.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f5471829640 10 monclient(hunting): get_auth_request con 0x7f546c138bf0 auth_method 0 2024-03-27T21:41:42.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f5471829640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:42.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f5471829640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:42.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.910+0000 7f5471829640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:42.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5471028640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:42.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5470827640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f546c06fce0 0x7f546c1d9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:42.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5471028640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:42.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5471028640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:42.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5471829640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:42.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5471829640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:42.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5471829640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:42.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5470827640 10 monclient(hunting): get_auth_request con 0x7f546c06fce0 auth_method 0 2024-03-27T21:41:42.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5470827640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:42.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5470827640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:42.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5470827640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:42.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5470827640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:42.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5470827640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:42.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5470827640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:42.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5471028640 10 monclient(hunting): handle_auth_done global_id 4880 payload 824 2024-03-27T21:41:42.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5471028640 1 -- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f546c06fce0 msgr2=0x7f546c1d9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:42.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5471028640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f546c06fce0 0x7f546c1d9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:42.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5471028640 1 -- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f546c138bf0 msgr2=0x7f546c1d9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:42.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5471028640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f546c138bf0 0x7f546c1d9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:42.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5471028640 10 monclient: _finish_hunting 0 2024-03-27T21:41:42.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5471028640 1 monclient: found mon.noname-c 2024-03-27T21:41:42.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5471028640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:42.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5471028640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:42.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5471028640 1 -- 172.21.15.60:0/4040627003 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54640094d0 con 0x7f546c06f900 2024-03-27T21:41:42.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5471829640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f546c138bf0 0x7f546c1d9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:42.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5471028640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f546c06f900 0x7f546c135330 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5468037040 tx=0x7f5468004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:42.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5461ffb640 10 client.?.objecter ms_handle_connect 0x7f546c06f900 2024-03-27T21:41:42.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5461ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:42.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546802cea0 con 0x7f546c06f900 2024-03-27T21:41:42.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5461ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:42.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5461ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:42.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5461ffb640 10 monclient: dump: 2024-03-27T21:41:42.914 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:42.915 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:42.915 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:42.915 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:42.915 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:42.915 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:42.915 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:42.915 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:42.915 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:42.915 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:42.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5461ffb640 10 monclient: _finish_auth 0 2024-03-27T21:41:42.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5461ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:41:42.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5461ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:42.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f547328c640 5 monclient: authenticate success, global_id 4880 2024-03-27T21:41:42.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f547328c640 10 monclient: _renew_subs 2024-03-27T21:41:42.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5468040650 con 0x7f546c06f900 2024-03-27T21:41:42.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f547328c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:42.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5468009a70 con 0x7f546c06f900 2024-03-27T21:41:42.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5461ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:42.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546802cea0 con 0x7f546c06f900 2024-03-27T21:41:42.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5461ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:42.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5461ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:42.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f5461ffb640 10 monclient: dump: 2024-03-27T21:41:42.916 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:42.916 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:42.916 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:42.916 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:42.916 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:42.916 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:42.916 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:42.917 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:42.917 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:42.917 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:42.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.912+0000 7f547328c640 10 client.4880.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:42.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.912+0000 7f547328c640 10 monclient: _renew_subs 2024-03-27T21:41:42.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.912+0000 7f547328c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:42.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.912+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f546c06a840 con 0x7f546c06f900 2024-03-27T21:41:42.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.912+0000 7f547328c640 1 librados: init done 2024-03-27T21:41:42.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.911+0000 7f54637fe640 4 set_mon_vals no callback set 2024-03-27T21:41:42.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.913+0000 7f547328c640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:41:42.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.913+0000 7f547328c640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:42.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.913+0000 7f547328c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:42.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.913+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f546c069470 con 0x7f546c06f900 2024-03-27T21:41:42.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.913+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f54680407c0 con 0x7f546c06f900 2024-03-27T21:41:42.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f5461ffb640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f545006c5f0 0x7f545006eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:42.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f54680ab5e0 con 0x7f546c06f900 2024-03-27T21:41:42.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f5470827640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f545006c5f0 0x7f545006eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:42.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f5461ffb640 10 client.4880.objecter ms_dispatch 0x7f546c000b90 osd_map(72..72 src has 1..72) v4 2024-03-27T21:41:42.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f5461ffb640 3 client.4880.objecter handle_osd_map got epochs [72,72] > 0 2024-03-27T21:41:42.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f5461ffb640 3 client.4880.objecter handle_osd_map decoding full epoch 72 2024-03-27T21:41:42.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f5470827640 10 monclient: get_auth_request con 0x7f545006c5f0 auth_method 0 2024-03-27T21:41:42.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f5461ffb640 20 client.4880.objecter dump_active .. 0 homeless 2024-03-27T21:41:42.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v72) v1 ==== 54+0+16 (secure 0 0 0) 0x7f54680b2070 con 0x7f546c06f900 2024-03-27T21:41:42.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f5461ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:42.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f5461ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:42.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f547328c640 10 client.4880.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:42.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign -1 1 2024-03-27T21:41:42.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f547328c640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f546c06ad70 0x7f546c06b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:42.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c0afd90 osd=0 3 2024-03-27T21:41:42.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c0afd90 osd=0 3 2024-03-27T21:41:42.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c0afd90 osd=0 3 2024-03-27T21:41:42.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f547328c640 15 client.4880.objecter _session_command_op_remove -1 1 2024-03-27T21:41:42.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f547328c640 20 client.4880.objecter get_session s=0x7f546c0afd90 osd=0 3 2024-03-27T21:41:42.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign 0 1 2024-03-27T21:41:42.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c0afd90 osd=0 4 2024-03-27T21:41:42.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f547328c640 10 client.4880.objecter _send_command 1 2024-03-27T21:41:42.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f546c1d96b0 con 0x7f546c06ad70 2024-03-27T21:41:42.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.914+0000 7f5471829640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f546c06ad70 0x7f546c06b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:42.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.915+0000 7f5471829640 10 monclient: get_auth_request con 0x7f546c06ad70 auth_method 0 2024-03-27T21:41:42.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.915+0000 7f5461ffb640 10 client.4880.objecter ms_handle_connect 0x7f545006c5f0 2024-03-27T21:41:42.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.915+0000 7f5470827640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f545006c5f0 0x7f545006eab0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f546c1d70a0 tx=0x7f545c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:42.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.915+0000 7f5461ffb640 10 client.4880.objecter ms_handle_connect 0x7f546c06ad70 2024-03-27T21:41:42.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.915+0000 7f5471829640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f546c06ad70 0x7f546c06b150 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:42.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.916+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f5464009c30 con 0x7f546c06ad70 2024-03-27T21:41:42.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.916+0000 7f5461ffb640 10 client.4880.objecter ms_dispatch 0x7f546c000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:41:42.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.916+0000 7f5461ffb640 10 client.4880.objecter _finish_command 1 = system:0 2024-03-27T21:41:42.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.916+0000 7f5461ffb640 20 client.4880.objecter put_session s=0x7f546c0afd90 osd=0 4 2024-03-27T21:41:42.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.916+0000 7f5461ffb640 15 client.4880.objecter _session_command_op_remove 0 1 2024-03-27T21:41:42.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.944+0000 7f547328c640 10 client.4880.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:42.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.944+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign -1 2 2024-03-27T21:41:42.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.944+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c0afd90 osd=0 3 2024-03-27T21:41:42.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.944+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c0afd90 osd=0 3 2024-03-27T21:41:42.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.944+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c0afd90 osd=0 3 2024-03-27T21:41:42.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.944+0000 7f547328c640 15 client.4880.objecter _session_command_op_remove -1 2 2024-03-27T21:41:42.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.944+0000 7f547328c640 20 client.4880.objecter get_session s=0x7f546c0afd90 osd=0 3 2024-03-27T21:41:42.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.944+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign 0 2 2024-03-27T21:41:42.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.944+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c0afd90 osd=0 4 2024-03-27T21:41:42.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.944+0000 7f547328c640 10 client.4880.objecter _send_command 2 2024-03-27T21:41:42.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.944+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f546c06b690 con 0x7f546c06ad70 2024-03-27T21:41:42.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.946+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f546c06b690 con 0x7f546c06ad70 2024-03-27T21:41:42.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.946+0000 7f5461ffb640 10 client.4880.objecter ms_dispatch 0x7f546c000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:42.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.946+0000 7f5461ffb640 10 client.4880.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:42.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.946+0000 7f5461ffb640 20 client.4880.objecter put_session s=0x7f546c0afd90 osd=0 4 2024-03-27T21:41:42.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.947+0000 7f5461ffb640 15 client.4880.objecter _session_command_op_remove 0 2 2024-03-27T21:41:42.933 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:41:42.934 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:41:42.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.947+0000 7f547328c640 10 client.4880.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:42.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.947+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign -1 3 2024-03-27T21:41:42.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.947+0000 7f547328c640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f546c1d67a0 0x7f546c065a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:42.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.947+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c06b860 osd=1 3 2024-03-27T21:41:42.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.947+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c06b860 osd=1 3 2024-03-27T21:41:42.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.947+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c06b860 osd=1 3 2024-03-27T21:41:42.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.947+0000 7f547328c640 15 client.4880.objecter _session_command_op_remove -1 3 2024-03-27T21:41:42.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.947+0000 7f547328c640 20 client.4880.objecter get_session s=0x7f546c06b860 osd=1 3 2024-03-27T21:41:42.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.947+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign 1 3 2024-03-27T21:41:42.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.947+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c06b860 osd=1 4 2024-03-27T21:41:42.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.947+0000 7f547328c640 10 client.4880.objecter _send_command 3 2024-03-27T21:41:42.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.947+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f546c1d6b80 con 0x7f546c1d67a0 2024-03-27T21:41:42.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.947+0000 7f5471028640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f546c1d67a0 0x7f546c065a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:42.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.947+0000 7f5471028640 10 monclient: get_auth_request con 0x7f546c1d67a0 auth_method 0 2024-03-27T21:41:42.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.947+0000 7f5461ffb640 10 client.4880.objecter ms_handle_connect 0x7f546c1d67a0 2024-03-27T21:41:42.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.947+0000 7f5471028640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f546c1d67a0 0x7f546c065a90 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:42.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.948+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f546c1d6b80 con 0x7f546c1d67a0 2024-03-27T21:41:42.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.948+0000 7f5461ffb640 10 client.4880.objecter ms_dispatch 0x7f546c000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:41:42.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.948+0000 7f5461ffb640 10 client.4880.objecter _finish_command 3 = system:0 2024-03-27T21:41:42.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.948+0000 7f5461ffb640 20 client.4880.objecter put_session s=0x7f546c06b860 osd=1 4 2024-03-27T21:41:42.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.948+0000 7f5461ffb640 15 client.4880.objecter _session_command_op_remove 1 3 2024-03-27T21:41:42.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-27T21:41:42.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.968+0000 7f547328c640 10 client.4880.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:42.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.968+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign -1 4 2024-03-27T21:41:42.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.968+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c06b860 osd=1 3 2024-03-27T21:41:42.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.968+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c06b860 osd=1 3 2024-03-27T21:41:42.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.968+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c06b860 osd=1 3 2024-03-27T21:41:42.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.968+0000 7f547328c640 15 client.4880.objecter _session_command_op_remove -1 4 2024-03-27T21:41:42.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.968+0000 7f547328c640 20 client.4880.objecter get_session s=0x7f546c06b860 osd=1 3 2024-03-27T21:41:42.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.968+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign 1 4 2024-03-27T21:41:42.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.968+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c06b860 osd=1 4 2024-03-27T21:41:42.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.968+0000 7f547328c640 10 client.4880.objecter _send_command 4 2024-03-27T21:41:42.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.968+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f546c065fd0 con 0x7f546c1d67a0 2024-03-27T21:41:42.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.970+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f546c065fd0 con 0x7f546c1d67a0 2024-03-27T21:41:42.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.970+0000 7f5461ffb640 10 client.4880.objecter ms_dispatch 0x7f546c000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:42.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.970+0000 7f5461ffb640 10 client.4880.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:42.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.970+0000 7f5461ffb640 20 client.4880.objecter put_session s=0x7f546c06b860 osd=1 4 2024-03-27T21:41:42.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.970+0000 7f5461ffb640 15 client.4880.objecter _session_command_op_remove 1 4 2024-03-27T21:41:42.957 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:41:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:41:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.970+0000 7f547328c640 10 client.4880.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.970+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign -1 5 2024-03-27T21:41:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.970+0000 7f547328c640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f546c02b170 0x7f546c02b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.970+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c066900 osd=2 3 2024-03-27T21:41:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.970+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c066900 osd=2 3 2024-03-27T21:41:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.970+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c066900 osd=2 3 2024-03-27T21:41:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.970+0000 7f547328c640 15 client.4880.objecter _session_command_op_remove -1 5 2024-03-27T21:41:42.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.970+0000 7f547328c640 20 client.4880.objecter get_session s=0x7f546c066900 osd=2 3 2024-03-27T21:41:42.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.970+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign 2 5 2024-03-27T21:41:42.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.970+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c066900 osd=2 4 2024-03-27T21:41:42.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.970+0000 7f547328c640 10 client.4880.objecter _send_command 5 2024-03-27T21:41:42.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.970+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 --> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f546c066a70 con 0x7f546c02b170 2024-03-27T21:41:42.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.970+0000 7f5470827640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f546c02b170 0x7f546c02b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:42.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.970+0000 7f5470827640 10 monclient: get_auth_request con 0x7f546c02b170 auth_method 0 2024-03-27T21:41:42.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.971+0000 7f5461ffb640 10 client.4880.objecter ms_handle_connect 0x7f546c02b170 2024-03-27T21:41:42.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.971+0000 7f5470827640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f546c02b170 0x7f546c02b550 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:42.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.972+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== osd.2 v2:172.21.15.60:6824/56434 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f546c066a70 con 0x7f546c02b170 2024-03-27T21:41:42.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.972+0000 7f5461ffb640 10 client.4880.objecter ms_dispatch 0x7f546c000b90 command_reply(tid 5: 0 ) v1 2024-03-27T21:41:42.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.972+0000 7f5461ffb640 10 client.4880.objecter _finish_command 5 = system:0 2024-03-27T21:41:42.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.972+0000 7f5461ffb640 20 client.4880.objecter put_session s=0x7f546c066900 osd=2 4 2024-03-27T21:41:42.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.972+0000 7f5461ffb640 15 client.4880.objecter _session_command_op_remove 2 5 2024-03-27T21:41:42.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.986+0000 7f547328c640 10 client.4880.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:42.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.986+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign -1 6 2024-03-27T21:41:42.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.986+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c066900 osd=2 3 2024-03-27T21:41:42.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.986+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c066900 osd=2 3 2024-03-27T21:41:42.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.986+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c066900 osd=2 3 2024-03-27T21:41:42.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.986+0000 7f547328c640 15 client.4880.objecter _session_command_op_remove -1 6 2024-03-27T21:41:42.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.986+0000 7f547328c640 20 client.4880.objecter get_session s=0x7f546c066900 osd=2 3 2024-03-27T21:41:42.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.986+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign 2 6 2024-03-27T21:41:42.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.986+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c066900 osd=2 4 2024-03-27T21:41:42.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.986+0000 7f547328c640 10 client.4880.objecter _send_command 6 2024-03-27T21:41:42.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.986+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 --> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f546c066380 con 0x7f546c02b170 2024-03-27T21:41:42.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.987+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== osd.2 v2:172.21.15.60:6824/56434 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f546c066380 con 0x7f546c02b170 2024-03-27T21:41:42.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.988+0000 7f5461ffb640 10 client.4880.objecter ms_dispatch 0x7f546c000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:42.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.988+0000 7f5461ffb640 10 client.4880.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:42.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.988+0000 7f5461ffb640 20 client.4880.objecter put_session s=0x7f546c066900 osd=2 4 2024-03-27T21:41:42.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.988+0000 7f5461ffb640 15 client.4880.objecter _session_command_op_remove 2 6 2024-03-27T21:41:42.974 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2: {} 2024-03-27T21:41:42.975 INFO:teuthology.orchestra.run.smithi060.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-27T21:41:42.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.988+0000 7f547328c640 10 client.4880.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.988+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign -1 7 2024-03-27T21:41:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.988+0000 7f547328c640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f546c1e68a0 0x7f546c1e8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.988+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c066730 osd=3 3 2024-03-27T21:41:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.988+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c066730 osd=3 3 2024-03-27T21:41:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.988+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c066730 osd=3 3 2024-03-27T21:41:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.988+0000 7f547328c640 15 client.4880.objecter _session_command_op_remove -1 7 2024-03-27T21:41:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.988+0000 7f547328c640 20 client.4880.objecter get_session s=0x7f546c066730 osd=3 3 2024-03-27T21:41:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.988+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign 3 7 2024-03-27T21:41:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.988+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c066730 osd=3 4 2024-03-27T21:41:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.988+0000 7f547328c640 10 client.4880.objecter _send_command 7 2024-03-27T21:41:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.988+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f546c1eba60 con 0x7f546c1e68a0 2024-03-27T21:41:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.988+0000 7f5471829640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f546c1e68a0 0x7f546c1e8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.988+0000 7f5471829640 10 monclient: get_auth_request con 0x7f546c1e68a0 auth_method 0 2024-03-27T21:41:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.988+0000 7f5471829640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f546c1e68a0 0x7f546c1e8c90 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:42.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.988+0000 7f5461ffb640 10 client.4880.objecter ms_handle_connect 0x7f546c1e68a0 2024-03-27T21:41:42.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.989+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f546c1eba60 con 0x7f546c1e68a0 2024-03-27T21:41:42.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.989+0000 7f5461ffb640 10 client.4880.objecter ms_dispatch 0x7f546c000b90 command_reply(tid 7: 0 ) v1 2024-03-27T21:41:42.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.989+0000 7f5461ffb640 10 client.4880.objecter _finish_command 7 = system:0 2024-03-27T21:41:42.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.989+0000 7f5461ffb640 20 client.4880.objecter put_session s=0x7f546c066730 osd=3 4 2024-03-27T21:41:42.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:42.989+0000 7f5461ffb640 15 client.4880.objecter _session_command_op_remove 3 7 2024-03-27T21:41:42.988 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:43.002+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:42.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.003+0000 7f547328c640 10 client.4880.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:42.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.003+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign -1 8 2024-03-27T21:41:42.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.003+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c066730 osd=3 3 2024-03-27T21:41:42.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.003+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c066730 osd=3 3 2024-03-27T21:41:42.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.003+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c066730 osd=3 3 2024-03-27T21:41:42.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.003+0000 7f547328c640 15 client.4880.objecter _session_command_op_remove -1 8 2024-03-27T21:41:42.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.003+0000 7f547328c640 20 client.4880.objecter get_session s=0x7f546c066730 osd=3 3 2024-03-27T21:41:42.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.003+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign 3 8 2024-03-27T21:41:42.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.003+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c066730 osd=3 4 2024-03-27T21:41:42.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.003+0000 7f547328c640 10 client.4880.objecter _send_command 8 2024-03-27T21:41:42.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.003+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f546c02bcd0 con 0x7f546c1e68a0 2024-03-27T21:41:42.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.005+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f546c02bcd0 con 0x7f546c1e68a0 2024-03-27T21:41:42.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.005+0000 7f5461ffb640 10 client.4880.objecter ms_dispatch 0x7f546c000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:42.993 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:41:42.995 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:41:42.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.005+0000 7f5461ffb640 10 client.4880.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.005+0000 7f5461ffb640 20 client.4880.objecter put_session s=0x7f546c066730 osd=3 4 2024-03-27T21:41:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.005+0000 7f5461ffb640 15 client.4880.objecter _session_command_op_remove 3 8 2024-03-27T21:41:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.005+0000 7f547328c640 10 client.4880.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.005+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign -1 9 2024-03-27T21:41:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.005+0000 7f547328c640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f546c02be40 0x7f546c1ea350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.005+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c02bcd0 osd=4 3 2024-03-27T21:41:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.005+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c02bcd0 osd=4 3 2024-03-27T21:41:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.005+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c02bcd0 osd=4 3 2024-03-27T21:41:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.005+0000 7f547328c640 15 client.4880.objecter _session_command_op_remove -1 9 2024-03-27T21:41:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.005+0000 7f547328c640 20 client.4880.objecter get_session s=0x7f546c02bcd0 osd=4 3 2024-03-27T21:41:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.005+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign 4 9 2024-03-27T21:41:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.005+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c02bcd0 osd=4 4 2024-03-27T21:41:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.005+0000 7f547328c640 10 client.4880.objecter _send_command 9 2024-03-27T21:41:42.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.005+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f546c13a450 con 0x7f546c02be40 2024-03-27T21:41:42.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.005+0000 7f5471028640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f546c02be40 0x7f546c1ea350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:42.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.005+0000 7f5471028640 10 monclient: get_auth_request con 0x7f546c02be40 auth_method 0 2024-03-27T21:41:42.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.006+0000 7f5461ffb640 10 client.4880.objecter ms_handle_connect 0x7f546c02be40 2024-03-27T21:41:42.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.006+0000 7f5471028640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f546c02be40 0x7f546c1ea350 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:42.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.007+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f546c13a450 con 0x7f546c02be40 2024-03-27T21:41:42.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.007+0000 7f5461ffb640 10 client.4880.objecter ms_dispatch 0x7f546c000b90 command_reply(tid 9: 0 ) v1 2024-03-27T21:41:42.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.007+0000 7f5461ffb640 10 client.4880.objecter _finish_command 9 = system:0 2024-03-27T21:41:42.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.007+0000 7f5461ffb640 20 client.4880.objecter put_session s=0x7f546c02bcd0 osd=4 4 2024-03-27T21:41:42.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.007+0000 7f5461ffb640 15 client.4880.objecter _session_command_op_remove 4 9 2024-03-27T21:41:43.006 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:43.006 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:43.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.020+0000 7f547328c640 10 client.4880.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:43.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.020+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign -1 10 2024-03-27T21:41:43.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.020+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c02bcd0 osd=4 3 2024-03-27T21:41:43.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.020+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c02bcd0 osd=4 3 2024-03-27T21:41:43.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.020+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c02bcd0 osd=4 3 2024-03-27T21:41:43.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.020+0000 7f547328c640 15 client.4880.objecter _session_command_op_remove -1 10 2024-03-27T21:41:43.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.020+0000 7f547328c640 20 client.4880.objecter get_session s=0x7f546c02bcd0 osd=4 3 2024-03-27T21:41:43.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.020+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign 4 10 2024-03-27T21:41:43.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.020+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c02bcd0 osd=4 4 2024-03-27T21:41:43.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.020+0000 7f547328c640 10 client.4880.objecter _send_command 10 2024-03-27T21:41:43.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.020+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f546c13a620 con 0x7f546c02be40 2024-03-27T21:41:43.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.022+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f546c13a620 con 0x7f546c02be40 2024-03-27T21:41:43.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.022+0000 7f5461ffb640 10 client.4880.objecter ms_dispatch 0x7f546c000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:43.008 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:41:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:41:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.022+0000 7f5461ffb640 10 client.4880.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.022+0000 7f5461ffb640 20 client.4880.objecter put_session s=0x7f546c02bcd0 osd=4 4 2024-03-27T21:41:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.022+0000 7f5461ffb640 15 client.4880.objecter _session_command_op_remove 4 10 2024-03-27T21:41:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.022+0000 7f547328c640 10 client.4880.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.022+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign -1 11 2024-03-27T21:41:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.022+0000 7f547328c640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f546c1eada0 0x7f546c1eb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.022+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c13a620 osd=5 3 2024-03-27T21:41:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.022+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c13a620 osd=5 3 2024-03-27T21:41:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.022+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c13a620 osd=5 3 2024-03-27T21:41:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.022+0000 7f547328c640 15 client.4880.objecter _session_command_op_remove -1 11 2024-03-27T21:41:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.022+0000 7f547328c640 20 client.4880.objecter get_session s=0x7f546c13a620 osd=5 3 2024-03-27T21:41:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.022+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign 5 11 2024-03-27T21:41:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.022+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c13a620 osd=5 4 2024-03-27T21:41:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.022+0000 7f547328c640 10 client.4880.objecter _send_command 11 2024-03-27T21:41:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.022+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f546c13a790 con 0x7f546c1eada0 2024-03-27T21:41:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.023+0000 7f5470827640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f546c1eada0 0x7f546c1eb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.023+0000 7f5470827640 10 monclient: get_auth_request con 0x7f546c1eada0 auth_method 0 2024-03-27T21:41:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.023+0000 7f5461ffb640 10 client.4880.objecter ms_handle_connect 0x7f546c1eada0 2024-03-27T21:41:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.023+0000 7f5470827640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f546c1eada0 0x7f546c1eb1e0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.024+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f546c13a790 con 0x7f546c1eada0 2024-03-27T21:41:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.024+0000 7f5461ffb640 10 client.4880.objecter ms_dispatch 0x7f546c000b90 command_reply(tid 11: 0 ) v1 2024-03-27T21:41:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.024+0000 7f5461ffb640 10 client.4880.objecter _finish_command 11 = system:0 2024-03-27T21:41:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.024+0000 7f5461ffb640 20 client.4880.objecter put_session s=0x7f546c13a620 osd=5 4 2024-03-27T21:41:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.024+0000 7f5461ffb640 15 client.4880.objecter _session_command_op_remove 5 11 2024-03-27T21:41:43.015 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:41:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.038+0000 7f547328c640 10 client.4880.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.038+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign -1 12 2024-03-27T21:41:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.038+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c13a620 osd=5 3 2024-03-27T21:41:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.038+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c13a620 osd=5 3 2024-03-27T21:41:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.038+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c13a620 osd=5 3 2024-03-27T21:41:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.038+0000 7f547328c640 15 client.4880.objecter _session_command_op_remove -1 12 2024-03-27T21:41:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.038+0000 7f547328c640 20 client.4880.objecter get_session s=0x7f546c13a620 osd=5 3 2024-03-27T21:41:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.038+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign 5 12 2024-03-27T21:41:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.038+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c13a620 osd=5 4 2024-03-27T21:41:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.038+0000 7f547328c640 10 client.4880.objecter _send_command 12 2024-03-27T21:41:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.038+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f546c13a790 con 0x7f546c1eada0 2024-03-27T21:41:43.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.039+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f546c13a790 con 0x7f546c1eada0 2024-03-27T21:41:43.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.039+0000 7f5461ffb640 10 client.4880.objecter ms_dispatch 0x7f546c000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:43.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.039+0000 7f5461ffb640 10 client.4880.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:43.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.039+0000 7f5461ffb640 20 client.4880.objecter put_session s=0x7f546c13a620 osd=5 4 2024-03-27T21:41:43.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.039+0000 7f5461ffb640 15 client.4880.objecter _session_command_op_remove 5 12 2024-03-27T21:41:43.026 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:41:43.027 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:41:43.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.040+0000 7f547328c640 10 client.4880.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:43.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.040+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign -1 13 2024-03-27T21:41:43.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.040+0000 7f547328c640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f546c1f0100 0x7f546c1f24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:43.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.040+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c13a790 osd=6 3 2024-03-27T21:41:43.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.040+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c13a790 osd=6 3 2024-03-27T21:41:43.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.040+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c13a790 osd=6 3 2024-03-27T21:41:43.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.040+0000 7f547328c640 15 client.4880.objecter _session_command_op_remove -1 13 2024-03-27T21:41:43.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.040+0000 7f547328c640 20 client.4880.objecter get_session s=0x7f546c13a790 osd=6 3 2024-03-27T21:41:43.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.040+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign 6 13 2024-03-27T21:41:43.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.040+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c13a790 osd=6 4 2024-03-27T21:41:43.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.040+0000 7f547328c640 10 client.4880.objecter _send_command 13 2024-03-27T21:41:43.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.040+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f546c1eb8f0 con 0x7f546c1f0100 2024-03-27T21:41:43.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.040+0000 7f5471829640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f546c1f0100 0x7f546c1f24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:43.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.040+0000 7f5471829640 10 monclient: get_auth_request con 0x7f546c1f0100 auth_method 0 2024-03-27T21:41:43.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.040+0000 7f5461ffb640 10 client.4880.objecter ms_handle_connect 0x7f546c1f0100 2024-03-27T21:41:43.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.040+0000 7f5471829640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f546c1f0100 0x7f546c1f24f0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:43.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.041+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f546c1eb8f0 con 0x7f546c1f0100 2024-03-27T21:41:43.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.041+0000 7f5461ffb640 10 client.4880.objecter ms_dispatch 0x7f546c000b90 command_reply(tid 13: 0 ) v1 2024-03-27T21:41:43.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.041+0000 7f5461ffb640 10 client.4880.objecter _finish_command 13 = system:0 2024-03-27T21:41:43.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.041+0000 7f5461ffb640 20 client.4880.objecter put_session s=0x7f546c13a790 osd=6 4 2024-03-27T21:41:43.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.041+0000 7f5461ffb640 15 client.4880.objecter _session_command_op_remove 6 13 2024-03-27T21:41:43.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.055+0000 7f547328c640 10 client.4880.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:43.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.055+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign -1 14 2024-03-27T21:41:43.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.055+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c13a790 osd=6 3 2024-03-27T21:41:43.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.055+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c13a790 osd=6 3 2024-03-27T21:41:43.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.055+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c13a790 osd=6 3 2024-03-27T21:41:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.055+0000 7f547328c640 15 client.4880.objecter _session_command_op_remove -1 14 2024-03-27T21:41:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.055+0000 7f547328c640 20 client.4880.objecter get_session s=0x7f546c13a790 osd=6 3 2024-03-27T21:41:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.055+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign 6 14 2024-03-27T21:41:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.055+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c13a790 osd=6 4 2024-03-27T21:41:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.055+0000 7f547328c640 10 client.4880.objecter _send_command 14 2024-03-27T21:41:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.055+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f546c1eb8f0 con 0x7f546c1f0100 2024-03-27T21:41:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.055+0000 7f9d4885b640 10 monclient: get_monmap_and_config 2024-03-27T21:41:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.055+0000 7f9d4885b640 10 monclient: build_initial_monmap 2024-03-27T21:41:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.055+0000 7f9d4885b640 10 monclient: monmap: 2024-03-27T21:41:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:43.056681+0000 2024-03-27T21:41:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:43.056681+0000 2024-03-27T21:41:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:43.044 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.056+0000 7f9d4885b640 1 Processor -- start 2024-03-27T21:41:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.056+0000 7f9d4885b640 1 -- start start 2024-03-27T21:41:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.056+0000 7f9d4885b640 10 monclient: init 2024-03-27T21:41:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.056+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f546c1eb8f0 con 0x7f546c1f0100 2024-03-27T21:41:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.056+0000 7f5461ffb640 10 client.4880.objecter ms_dispatch 0x7f546c000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:43.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.056+0000 7f5461ffb640 10 client.4880.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:43.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.056+0000 7f5461ffb640 20 client.4880.objecter put_session s=0x7f546c13a790 osd=6 4 2024-03-27T21:41:43.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.056+0000 7f5461ffb640 15 client.4880.objecter _session_command_op_remove 6 14 2024-03-27T21:41:43.046 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:41:43.046 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:41:43.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f9d4885b640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:43.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f547328c640 10 client.4880.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:43.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign -1 15 2024-03-27T21:41:43.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f547328c640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f546c1f2f90 0x7f546c1f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:43.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c1eb8f0 osd=7 3 2024-03-27T21:41:43.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c1eb8f0 osd=7 3 2024-03-27T21:41:43.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c1eb8f0 osd=7 3 2024-03-27T21:41:43.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f547328c640 15 client.4880.objecter _session_command_op_remove -1 15 2024-03-27T21:41:43.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f547328c640 20 client.4880.objecter get_session s=0x7f546c1eb8f0 osd=7 3 2024-03-27T21:41:43.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign 7 15 2024-03-27T21:41:43.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c1eb8f0 osd=7 4 2024-03-27T21:41:43.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f547328c640 10 client.4880.objecter _send_command 15 2024-03-27T21:41:43.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f546c1f84d0 con 0x7f546c1f2f90 2024-03-27T21:41:43.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f9d4885b640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-03-27T21:41:43.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f9d4885b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d440b40b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:43.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f9d4885b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9d4413b020 0x7f9d4413b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:43.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f9d4885b640 10 monclient(hunting): picked mon.noname-b con 0x7f9d4413b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:43.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f9d4885b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d441416f0 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:43.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f9d4885b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d440b3830 con 0x7f9d440b40b0 2024-03-27T21:41:43.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f9d3bfff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9d4413b020 0x7f9d4413b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:43.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f9d3bfff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9d4413b020 0x7f9d4413b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:32770/0 (socket says 172.21.15.60:32770) 2024-03-27T21:41:43.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f9d3bfff640 1 -- 172.21.15.60:0/605585058 learned_addr learned my addr 172.21.15.60:0/605585058 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:43.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f9d4885b640 1 -- 172.21.15.60:0/605585058 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d44142290 con 0x7f9d441416f0 2024-03-27T21:41:43.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f9d4885b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:43.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f9d4885b640 1 -- 172.21.15.60:0/605585058 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d440b4490 con 0x7f9d4413b020 2024-03-27T21:41:43.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f9d4885b640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:43.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.057+0000 7f9d4885b640 10 monclient(hunting): authenticate will time out at 944.278815s 2024-03-27T21:41:43.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d3bfff640 10 monclient(hunting): get_auth_request con 0x7f9d4413b020 auth_method 0 2024-03-27T21:41:43.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:43.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:43.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:43.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f5471028640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f546c1f2f90 0x7f546c1f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:43.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f5471028640 10 monclient: get_auth_request con 0x7f546c1f2f90 auth_method 0 2024-03-27T21:41:43.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 1 -- 172.21.15.60:0/605585058 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390073481 0 0) 0x7f9d440b3830 con 0x7f9d440b40b0 2024-03-27T21:41:43.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient(hunting): _negotiate 2024-03-27T21:41:43.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:43.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:43.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient(hunting): my global_id is 5124 2024-03-27T21:41:43.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 1 -- 172.21.15.60:0/605585058 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d30003520 con 0x7f9d440b40b0 2024-03-27T21:41:43.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:43.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:43.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 1 -- 172.21.15.60:0/605585058 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885935311 0 0) 0x7f9d44142290 con 0x7f9d441416f0 2024-03-27T21:41:43.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:43.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient(hunting): _negotiate 2024-03-27T21:41:43.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:43.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:43.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient(hunting): my global_id is 5068 2024-03-27T21:41:43.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 1 -- 172.21.15.60:0/605585058 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d440b3830 con 0x7f9d441416f0 2024-03-27T21:41:43.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 1 -- 172.21.15.60:0/605585058 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2906728964 0 0) 0x7f9d30003520 con 0x7f9d440b40b0 2024-03-27T21:41:43.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 1 -- 172.21.15.60:0/605585058 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9d4413b020 msgr2=0x7f9d4413b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:43.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 1 --2- 172.21.15.60:0/605585058 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9d4413b020 0x7f9d4413b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:43.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f5461ffb640 10 client.4880.objecter ms_handle_connect 0x7f546c1f2f90 2024-03-27T21:41:43.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f5471028640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f546c1f2f90 0x7f546c1f33f0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:43.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 1 -- 172.21.15.60:0/605585058 >> v1:172.21.15.191:6789/0 conn(0x7f9d441416f0 legacy=0x7f9d44141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:43.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient: _finish_hunting 0 2024-03-27T21:41:43.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 1 monclient: found mon.noname-a-legacy 2024-03-27T21:41:43.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:43.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.60:6789/0 2024-03-27T21:41:43.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 1 -- 172.21.15.60:0/605585058 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d440aa520 con 0x7f9d440b40b0 2024-03-27T21:41:43.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient: _finish_auth 0 2024-03-27T21:41:43.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient: _check_auth_tickets 2024-03-27T21:41:43.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:43.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d4885b640 5 monclient: authenticate success, global_id 5124 2024-03-27T21:41:43.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d4885b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:43.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 1 -- 172.21.15.60:0/605585058 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2933500084 0 0) 0x7f9d34001670 con 0x7f9d440b40b0 2024-03-27T21:41:43.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:43.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-03-27T21:41:43.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient: dump: 2024-03-27T21:41:43.059 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:43.059 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:43.059 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:43.059 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:43.059 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:43.059 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:43.059 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:43.059 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:43.059 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:43.060 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:43.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] but i'm connected to v1:172.21.15.60:6789/0, reconnecting 2024-03-27T21:41:43.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:43.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 1 -- 172.21.15.60:0/605585058 >> v1:172.21.15.60:6789/0 conn(0x7f9d440b40b0 legacy=0x7f9d440a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:43.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:41:43.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 1 --2- 172.21.15.60:0/605585058 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9d4413b020 0x7f9d30003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:43.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient(hunting): picked mon.c con 0x7f9d4413b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:43.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 1 --2- 172.21.15.60:0/605585058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d30004090 0x7f9d300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:43.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient(hunting): picked mon.b con 0x7f9d30004090 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:43.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 1 --2- 172.21.15.60:0/605585058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d30004d00 0x7f9d30005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:43.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient(hunting): picked mon.a con 0x7f9d30004d00 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:43.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:43.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 1 -- 172.21.15.60:0/605585058 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d440b4490 con 0x7f9d30004d00 2024-03-27T21:41:43.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:43.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 1 -- 172.21.15.60:0/605585058 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3000ba60 con 0x7f9d30004090 2024-03-27T21:41:43.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:43.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 1 -- 172.21.15.60:0/605585058 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3000bc20 con 0x7f9d4413b020 2024-03-27T21:41:43.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:43.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.058+0000 7f9d41d9b640 10 monclient(hunting): _finish_auth 0 2024-03-27T21:41:43.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d4885b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:43.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d3bfff640 1 --2- 172.21.15.60:0/605585058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d30004090 0x7f9d300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:43.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d3bfff640 10 monclient(hunting): get_auth_request con 0x7f9d30004090 auth_method 0 2024-03-27T21:41:43.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:43.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f546c1f84d0 con 0x7f546c1f2f90 2024-03-27T21:41:43.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f5461ffb640 10 client.4880.objecter ms_dispatch 0x7f546c000b90 command_reply(tid 15: 0 ) v1 2024-03-27T21:41:43.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f5461ffb640 10 client.4880.objecter _finish_command 15 = system:0 2024-03-27T21:41:43.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f5461ffb640 20 client.4880.objecter put_session s=0x7f546c1eb8f0 osd=7 4 2024-03-27T21:41:43.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f5461ffb640 15 client.4880.objecter _session_command_op_remove 7 15 2024-03-27T21:41:43.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d4259c640 1 --2- 172.21.15.60:0/605585058 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9d4413b020 0x7f9d30003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:43.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:43.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d3bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:41:43.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d42d9d640 1 --2- 172.21.15.60:0/605585058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d30004d00 0x7f9d30005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:43.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d42d9d640 10 monclient(hunting): get_auth_request con 0x7f9d30004d00 auth_method 0 2024-03-27T21:41:43.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d42d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:43.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d42d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:43.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d42d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:41:43.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:43.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:43.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:41:43.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d42d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:43.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d42d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:43.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d42d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:41:43.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d4259c640 10 monclient(hunting): get_auth_request con 0x7f9d4413b020 auth_method 0 2024-03-27T21:41:43.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:43.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d4259c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:43.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d4259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:41:43.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d4259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:43.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d4259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:43.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d4259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:41:43.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d3bfff640 10 monclient(hunting): handle_auth_done global_id 5124 payload 293 2024-03-27T21:41:43.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d3bfff640 1 -- 172.21.15.60:0/605585058 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9d4413b020 msgr2=0x7f9d30003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:43.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d3bfff640 1 --2- 172.21.15.60:0/605585058 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9d4413b020 0x7f9d30003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:43.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d3bfff640 1 -- 172.21.15.60:0/605585058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d30004d00 msgr2=0x7f9d30005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:43.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d3bfff640 1 --2- 172.21.15.60:0/605585058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d30004d00 0x7f9d30005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:43.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d3bfff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:43.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d3bfff640 1 monclient: found mon.b 2024-03-27T21:41:43.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:43.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d3bfff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:43.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d3bfff640 1 -- 172.21.15.60:0/605585058 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d30003520 con 0x7f9d30004090 2024-03-27T21:41:43.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d3bfff640 10 monclient: _finish_auth 0 2024-03-27T21:41:43.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d3bfff640 10 monclient: _check_auth_tickets 2024-03-27T21:41:43.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:43.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.059+0000 7f9d42d9d640 1 --2- 172.21.15.60:0/605585058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d30004d00 0x7f9d30005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:43.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.060+0000 7f9d3bfff640 1 --2- 172.21.15.60:0/605585058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d30004090 0x7f9d300047c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c009c10 tx=0x7f9d300057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd9dcae62e73bbe2 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.060+0000 7f9d41d9b640 1 -- 172.21.15.60:0/605585058 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c02c7a0 con 0x7f9d30004090 2024-03-27T21:41:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.060+0000 7f9d41d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.060+0000 7f9d41d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.060+0000 7f9d41d9b640 10 monclient: dump: 2024-03-27T21:41:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:43.073 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.060+0000 7f9d41d9b640 1 -- 172.21.15.60:0/605585058 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2c02c900 con 0x7f9d30004090 2024-03-27T21:41:43.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.060+0000 7f9d41d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:43.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.061+0000 7f9d41d9b640 1 -- 172.21.15.60:0/605585058 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c02cc90 con 0x7f9d30004090 2024-03-27T21:41:43.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.061+0000 7f9d4885b640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:43.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.061+0000 7f9d4885b640 4 set_mon_vals no callback set 2024-03-27T21:41:43.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.061+0000 7f9d41d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:43.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.061+0000 7f9d41d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:43.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.061+0000 7f9d41d9b640 10 monclient: dump: 2024-03-27T21:41:43.074 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:43.074 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:43.074 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:43.074 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:43.074 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:43.074 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:43.074 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:43.075 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:43.075 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:43.075 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:43.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.061+0000 7f9d4885b640 10 monclient: shutdown 2024-03-27T21:41:43.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.061+0000 7f9d4885b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:43.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.061+0000 7f9d4885b640 1 -- 172.21.15.60:0/605585058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d30004090 msgr2=0x7f9d300047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:43.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.061+0000 7f9d4885b640 1 --2- 172.21.15.60:0/605585058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d30004090 0x7f9d300047c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c009c10 tx=0x7f9d300057f0 comp rx=0 tx=0).stop 2024-03-27T21:41:43.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.061+0000 7f9d42d9d640 1 -- 172.21.15.60:0/605585058 reap_dead start 2024-03-27T21:41:43.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.061+0000 7f9d4885b640 1 -- 172.21.15.60:0/605585058 shutdown_connections 2024-03-27T21:41:43.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.061+0000 7f9d4885b640 1 -- 172.21.15.60:0/605585058 >> 172.21.15.60:0/605585058 conn(0x7f9d440608e0 msgr2=0x7f9d44060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:43.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.061+0000 7f9d4885b640 1 -- 172.21.15.60:0/605585058 shutdown_connections 2024-03-27T21:41:43.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.061+0000 7f9d4885b640 1 -- 172.21.15.60:0/605585058 wait complete. 2024-03-27T21:41:43.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.062+0000 7f9d4885b640 10 monclient: build_initial_monmap 2024-03-27T21:41:43.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.062+0000 7f9d4885b640 10 monclient: monmap: 2024-03-27T21:41:43.076 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:43.076 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:43.076 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:43.076 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:43.076 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:43.076 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:43.076 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:43.076 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:43.076 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:43.076 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:43.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.062+0000 7f9d4885b640 1 librados: starting msgr at 2024-03-27T21:41:43.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.062+0000 7f9d4885b640 1 librados: starting objecter 2024-03-27T21:41:43.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.063+0000 7f9d4885b640 1 Processor -- start 2024-03-27T21:41:43.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.063+0000 7f9d4885b640 1 -- start start 2024-03-27T21:41:43.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.063+0000 7f9d4885b640 1 librados: setting wanted keys 2024-03-27T21:41:43.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.063+0000 7f9d4885b640 1 librados: calling monclient init 2024-03-27T21:41:43.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.063+0000 7f9d4885b640 10 monclient: init 2024-03-27T21:41:43.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.063+0000 7f9d4885b640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:43.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.063+0000 7f9d4885b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:41:43.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.063+0000 7f9d4885b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9d30004090 0x7f9d4413e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:43.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.063+0000 7f9d4885b640 10 monclient(hunting): picked mon.noname-c con 0x7f9d30004090 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:43.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.063+0000 7f9d4885b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d4413dc90 0x7f9d441d98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:43.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.063+0000 7f9d4885b640 10 monclient(hunting): picked mon.noname-b con 0x7f9d4413dc90 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:43.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d4885b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d441d9de0 0x7f9d4413a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:43.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d4885b640 10 monclient(hunting): picked mon.noname-a con 0x7f9d441d9de0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:43.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d4259c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9d30004090 0x7f9d4413e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:43.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3bfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d4413dc90 0x7f9d441d98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:43.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d4885b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:43.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d4885b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d441c9be0 con 0x7f9d441d9de0 2024-03-27T21:41:43.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d4885b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:43.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d4885b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d441ca2e0 con 0x7f9d4413dc90 2024-03-27T21:41:43.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d4885b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:43.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d4259c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9d30004090 0x7f9d4413e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:32784/0 (socket says 172.21.15.60:32784) 2024-03-27T21:41:43.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3bfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d4413dc90 0x7f9d441d98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:53524/0 (socket says 172.21.15.60:53524) 2024-03-27T21:41:43.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d4885b640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d441c9dc0 con 0x7f9d30004090 2024-03-27T21:41:43.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d4885b640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:43.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d4259c640 1 -- 172.21.15.60:0/2910650472 learned_addr learned my addr 172.21.15.60:0/2910650472 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:43.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d4885b640 10 monclient(hunting): authenticate will time out at 944.285083s 2024-03-27T21:41:43.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d4259c640 10 monclient(hunting): get_auth_request con 0x7f9d30004090 auth_method 0 2024-03-27T21:41:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d4259c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d4259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d42d9d640 1 --2- 172.21.15.60:0/2910650472 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d441d9de0 0x7f9d4413a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3bfff640 10 monclient(hunting): get_auth_request con 0x7f9d4413dc90 auth_method 0 2024-03-27T21:41:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d42d9d640 10 monclient(hunting): get_auth_request con 0x7f9d441d9de0 auth_method 0 2024-03-27T21:41:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d42d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d42d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d42d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d4259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d4259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d4259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d42d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d42d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d42d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3bfff640 10 monclient(hunting): handle_auth_done global_id 5071 payload 824 2024-03-27T21:41:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3bfff640 1 -- 172.21.15.60:0/2910650472 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9d30004090 msgr2=0x7f9d4413e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3bfff640 1 --2- 172.21.15.60:0/2910650472 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9d30004090 0x7f9d4413e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3bfff640 1 -- 172.21.15.60:0/2910650472 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d441d9de0 msgr2=0x7f9d4413a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3bfff640 1 --2- 172.21.15.60:0/2910650472 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d441d9de0 0x7f9d4413a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3bfff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3bfff640 1 monclient: found mon.noname-b 2024-03-27T21:41:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:43.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:41:43.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3bfff640 1 -- 172.21.15.60:0/2910650472 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4413e910 con 0x7f9d4413dc90 2024-03-27T21:41:43.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d4259c640 1 --2- 172.21.15.60:0/2910650472 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9d30004090 0x7f9d4413e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:43.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d42d9d640 1 --2- 172.21.15.60:0/2910650472 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d441d9de0 0x7f9d4413a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:43.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3bfff640 1 --2- 172.21.15.60:0/2910650472 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d4413dc90 0x7f9d441d98a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000bbf0 tx=0x7f9d2c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:43.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3affd640 10 client.?.objecter ms_handle_connect 0x7f9d4413dc90 2024-03-27T21:41:43.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.064+0000 7f9d3affd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:43.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d3affd640 1 -- 172.21.15.60:0/2910650472 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c02c7a0 con 0x7f9d4413dc90 2024-03-27T21:41:43.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:43.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d3affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:43.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d3affd640 10 monclient: dump: 2024-03-27T21:41:43.081 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:43.081 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:43.081 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:43.082 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d3affd640 10 monclient: _finish_auth 0 2024-03-27T21:41:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d3affd640 10 monclient: _check_auth_tickets 2024-03-27T21:41:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d3affd640 1 -- 172.21.15.60:0/2910650472 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2c034ba0 con 0x7f9d4413dc90 2024-03-27T21:41:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d4159a640 4 set_mon_vals no callback set 2024-03-27T21:41:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d3affd640 1 -- 172.21.15.60:0/2910650472 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c03f420 con 0x7f9d4413dc90 2024-03-27T21:41:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d3affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d3affd640 10 monclient: dump: 2024-03-27T21:41:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:43.083 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d4885b640 5 monclient: authenticate success, global_id 5071 2024-03-27T21:41:43.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d4885b640 10 monclient: _renew_subs 2024-03-27T21:41:43.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d4885b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:43.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d4885b640 1 -- 172.21.15.60:0/2910650472 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d4413d420 con 0x7f9d4413dc90 2024-03-27T21:41:43.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d4885b640 10 client.5071.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:43.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d4885b640 10 monclient: _renew_subs 2024-03-27T21:41:43.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d4885b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:43.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d4885b640 1 -- 172.21.15.60:0/2910650472 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d4413a670 con 0x7f9d4413dc90 2024-03-27T21:41:43.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.065+0000 7f9d4885b640 1 librados: init done 2024-03-27T21:41:43.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.066+0000 7f9d4885b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:43.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.066+0000 7f9d4885b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:43.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.066+0000 7f9d4885b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:43.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.066+0000 7f9d4885b640 1 -- 172.21.15.60:0/2910650472 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d440ab710 con 0x7f9d4413dc90 2024-03-27T21:41:43.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.066+0000 7f9d3affd640 1 -- 172.21.15.60:0/2910650472 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9d2c03b040 con 0x7f9d4413dc90 2024-03-27T21:41:43.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.067+0000 7f9d3affd640 1 --2- 172.21.15.60:0/2910650472 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9d2406c590 0x7f9d2406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:43.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.067+0000 7f9d3affd640 1 -- 172.21.15.60:0/2910650472 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f9d2c0aaf00 con 0x7f9d4413dc90 2024-03-27T21:41:43.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.067+0000 7f9d4259c640 1 --2- 172.21.15.60:0/2910650472 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9d2406c590 0x7f9d2406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:43.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.067+0000 7f9d4259c640 10 monclient: get_auth_request con 0x7f9d2406c590 auth_method 0 2024-03-27T21:41:43.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.067+0000 7f9d3affd640 10 client.5071.objecter ms_dispatch 0x7f9d44000b90 osd_map(72..72 src has 1..72) v4 2024-03-27T21:41:43.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.067+0000 7f9d3affd640 3 client.5071.objecter handle_osd_map got epochs [72,72] > 0 2024-03-27T21:41:43.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.067+0000 7f9d3affd640 3 client.5071.objecter handle_osd_map decoding full epoch 72 2024-03-27T21:41:43.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.067+0000 7f9d3affd640 20 client.5071.objecter dump_active .. 0 homeless 2024-03-27T21:41:43.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.068+0000 7f9d4259c640 1 --2- 172.21.15.60:0/2910650472 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9d2406c590 0x7f9d2406ea50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9d34001c70 tx=0x7f9d34009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:43.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.068+0000 7f9d3affd640 10 client.5071.objecter ms_handle_connect 0x7f9d2406c590 2024-03-27T21:41:43.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.071+0000 7f9d3affd640 1 -- 172.21.15.60:0/2910650472 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9d2c076360 con 0x7f9d4413dc90 2024-03-27T21:41:43.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.071+0000 7f9d3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:43.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.071+0000 7f9d3affd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:43.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.083+0000 7f547328c640 10 client.4880.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-27T21:41:43.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.083+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign -1 16 2024-03-27T21:41:43.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.083+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c1eb8f0 osd=7 3 2024-03-27T21:41:43.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.083+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c1eb8f0 osd=7 3 2024-03-27T21:41:43.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.083+0000 7f547328c640 20 client.4880.objecter _get_session s=0x7f546c1eb8f0 osd=7 3 2024-03-27T21:41:43.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.083+0000 7f547328c640 15 client.4880.objecter _session_command_op_remove -1 16 2024-03-27T21:41:43.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.083+0000 7f547328c640 20 client.4880.objecter get_session s=0x7f546c1eb8f0 osd=7 3 2024-03-27T21:41:43.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.083+0000 7f547328c640 15 client.4880.objecter _session_command_op_assign 7 16 2024-03-27T21:41:43.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.083+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c1eb8f0 osd=7 4 2024-03-27T21:41:43.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.083+0000 7f547328c640 10 client.4880.objecter _send_command 16 2024-03-27T21:41:43.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.083+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f546c1f3a40 con 0x7f546c1f2f90 2024-03-27T21:41:43.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.084+0000 7f5461ffb640 1 -- 172.21.15.60:0/4040627003 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f546c1f3a40 con 0x7f546c1f2f90 2024-03-27T21:41:43.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.084+0000 7f5461ffb640 10 client.4880.objecter ms_dispatch 0x7f546c000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:43.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.084+0000 7f5461ffb640 10 client.4880.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:43.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.084+0000 7f5461ffb640 20 client.4880.objecter put_session s=0x7f546c1eb8f0 osd=7 4 2024-03-27T21:41:43.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.084+0000 7f5461ffb640 15 client.4880.objecter _session_command_op_remove 7 16 2024-03-27T21:41:43.088 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:41:43.093 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:41:43.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 10 librados: watch_flush enter 2024-03-27T21:41:43.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 10 librados: watch_flush exit 2024-03-27T21:41:43.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 10 client.4880.objecter close_session for osd.0 2024-03-27T21:41:43.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f546c06ad70 msgr2=0x7f546c06b150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:43.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f546c06ad70 0x7f546c06b150 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:43.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c0afd90 osd=0 1 2024-03-27T21:41:43.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 10 client.4880.objecter close_session for osd.1 2024-03-27T21:41:43.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f546c1d67a0 msgr2=0x7f546c065a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:43.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f546c1d67a0 0x7f546c065a90 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:43.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c06b860 osd=1 1 2024-03-27T21:41:43.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 10 client.4880.objecter close_session for osd.2 2024-03-27T21:41:43.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f546c02b170 msgr2=0x7f546c02b550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:43.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f546c02b170 0x7f546c02b550 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:43.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c066900 osd=2 1 2024-03-27T21:41:43.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 10 client.4880.objecter close_session for osd.3 2024-03-27T21:41:43.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f546c1e68a0 msgr2=0x7f546c1e8c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f546c1e68a0 0x7f546c1e8c90 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c066730 osd=3 1 2024-03-27T21:41:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 10 client.4880.objecter close_session for osd.4 2024-03-27T21:41:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f546c02be40 msgr2=0x7f546c1ea350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f546c02be40 0x7f546c1ea350 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f5471829640 1 -- 172.21.15.60:0/4040627003 reap_dead start 2024-03-27T21:41:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f5471829640 1 -- 172.21.15.60:0/4040627003 reap_dead start 2024-03-27T21:41:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c02bcd0 osd=4 1 2024-03-27T21:41:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 10 client.4880.objecter close_session for osd.5 2024-03-27T21:41:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f546c1eada0 msgr2=0x7f546c1eb1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f546c1eada0 0x7f546c1eb1e0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c13a620 osd=5 1 2024-03-27T21:41:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 10 client.4880.objecter close_session for osd.6 2024-03-27T21:41:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f546c1f0100 msgr2=0x7f546c1f24f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f546c1f0100 0x7f546c1f24f0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c13a790 osd=6 1 2024-03-27T21:41:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 10 client.4880.objecter close_session for osd.7 2024-03-27T21:41:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f546c1f2f90 msgr2=0x7f546c1f33f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f546c1f2f90 0x7f546c1f33f0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 20 client.4880.objecter put_session s=0x7f546c1eb8f0 osd=7 1 2024-03-27T21:41:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 20 client.4880.objecter shutdown clearing up homeless session... 2024-03-27T21:41:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.085+0000 7f547328c640 10 client.4880.objecter successfully canceled tick 2024-03-27T21:41:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.086+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f545006c5f0 msgr2=0x7f545006eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.086+0000 7f547328c640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f545006c5f0 0x7f545006eab0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f546c1d70a0 tx=0x7f545c008040 comp rx=0 tx=0).stop 2024-03-27T21:41:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.086+0000 7f5471829640 1 -- 172.21.15.60:0/4040627003 reap_dead start 2024-03-27T21:41:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.086+0000 7f547328c640 10 monclient: shutdown 2024-03-27T21:41:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.086+0000 7f547328c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.086+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f546c06f900 msgr2=0x7f546c135330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.086+0000 7f547328c640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f546c06f900 0x7f546c135330 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5468037040 tx=0x7f5468004a40 comp rx=0 tx=0).stop 2024-03-27T21:41:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.086+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 shutdown_connections 2024-03-27T21:41:43.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.086+0000 7f547328c640 1 --2- 172.21.15.60:0/4040627003 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f546c06f900 0x7f546c135330 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:43.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.086+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 >> 172.21.15.60:0/4040627003 conn(0x7f546c0b4e50 msgr2=0x7f546c0b0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:43.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.086+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 shutdown_connections 2024-03-27T21:41:43.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.086+0000 7f547328c640 1 -- 172.21.15.60:0/4040627003 wait complete. 2024-03-27T21:41:43.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.087+0000 7f547328c640 1 librados: shutdown 2024-03-27T21:41:43.098 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:43.103+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:43.120 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:43.120 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:43.120 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:43.120 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:43.120 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:43.129 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:41:43.213 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:43.227+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:43.274 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:43.274 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:43.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:43.274 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:41:43.274 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:43.282 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:41:43.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.328+0000 7f9d4885b640 10 start_mon_command cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:41:43.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.328+0000 7f9d4885b640 10 monclient: _send_command 2 [{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:41:43.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.328+0000 7f9d4885b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:41:43.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:43.328+0000 7f9d4885b640 1 -- 172.21.15.60:0/2910650472 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f9d44069470 con 0x7f9d4413dc90 2024-03-27T21:41:43.323 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:43.337+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:43.393 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:43.406+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:43.430 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:43.431 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:43.431 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:43.431 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:43.436 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:41:43.493 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:43.507+0000 7fb850f34640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 56426) UID: 0 2024-03-27T21:41:43.581 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:43.581 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:43.581 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:43.582 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:43.582 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:43.590 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:41:43.634 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:43.648+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:43.695 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:43.708+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:43.695 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:43.695 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:43.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:43.695 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [] 2024-03-27T21:41:43.695 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:43.704 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:41:43.796 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:43.810+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:43.850 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:43.850 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:43.850 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-03-27T21:41:43.850 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:43.859 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:41:43.897 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:43.911+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:43.963 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:43.963 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-03-27T21:41:43.964 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-03-27T21:41:43.964 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-03-27T21:41:43.964 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:43.972 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:41:44.015 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:44.029+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:44.079 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-27T21:41:44.079 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-03-27T21:41:44.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-03-27T21:41:44.080 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-03-27T21:41:44.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.080 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:44.080 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:44.080 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 124.660209722, 2024-03-27T21:41:44.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43537978700000002, 2024-03-27T21:41:44.080 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:44.080 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:44.080 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:44.080 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:44.080 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:44.080 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1 2024-03-27T21:41:44.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.081 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:44.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:44.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:44.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:44.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432021+0000", 2024-03-27T21:41:44.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:44.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432019+0000", 2024-03-27T21:41:44.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-03-27T21:41:44.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.082 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:44.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432027+0000", 2024-03-27T21:41:44.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1200000000000002e-06 2024-03-27T21:41:44.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.082 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:44.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432039+0000", 2024-03-27T21:41:44.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1123e-05 2024-03-27T21:41:44.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:44.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432051+0000", 2024-03-27T21:41:44.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2094e-05 2024-03-27T21:41:44.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:44.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432157+0000", 2024-03-27T21:41:44.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106846 2024-03-27T21:41:44.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-03-27T21:41:44.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.432176+0000", 2024-03-27T21:41:44.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8490000000000001e-05 2024-03-27T21:41:44.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:44.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866170+0000", 2024-03-27T21:41:44.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.43399370100000001 2024-03-27T21:41:44.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:44.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866335+0000", 2024-03-27T21:41:44.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016559699999999999 2024-03-27T21:41:44.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:44.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.866495+0000", 2024-03-27T21:41:44.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015944200000000001 2024-03-27T21:41:44.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:44.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867012+0000", 2024-03-27T21:41:44.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0005176 2024-03-27T21:41:44.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:44.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867334+0000", 2024-03-27T21:41:44.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032158600000000001 2024-03-27T21:41:44.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:44.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867374+0000", 2024-03-27T21:41:44.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0547000000000003e-05 2024-03-27T21:41:44.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:44.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.867400+0000", 2024-03-27T21:41:44.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5958000000000001e-05 2024-03-27T21:41:44.087 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.088 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:44.088 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.088 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.088 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.088 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:44.088 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:44.088 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 124.162827271, 2024-03-27T21:41:44.088 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064560310000000001, 2024-03-27T21:41:44.088 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:44.088 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:44.089 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:44.089 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:44.089 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:44.089 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 70 2024-03-27T21:41:44.089 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.089 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:44.089 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.089 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:44.089 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:44.089 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:44.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929403+0000", 2024-03-27T21:41:44.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:44.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929402+0000", 2024-03-27T21:41:44.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:44.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:44.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929407+0000", 2024-03-27T21:41:44.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-06 2024-03-27T21:41:44.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.093 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.093 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:44.093 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929414+0000", 2024-03-27T21:41:44.093 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5689999999999999e-06 2024-03-27T21:41:44.093 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.093 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.093 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:44.093 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.929419+0000", 2024-03-27T21:41:44.093 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9760000000000003e-06 2024-03-27T21:41:44.094 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.094 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.094 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:44.094 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930245+0000", 2024-03-27T21:41:44.095 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082521500000000002 2024-03-27T21:41:44.095 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.095 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.095 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:44.095 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930287+0000", 2024-03-27T21:41:44.095 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2305000000000002e-05 2024-03-27T21:41:44.095 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.096 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.096 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:44.096 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.930367+0000", 2024-03-27T21:41:44.096 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9589e-05 2024-03-27T21:41:44.096 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.096 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:44.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.931214+0000", 2024-03-27T21:41:44.097 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000847827 2024-03-27T21:41:44.097 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:44.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935807+0000", 2024-03-27T21:41:44.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0045923780000000003 2024-03-27T21:41:44.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.098 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:44.099 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935840+0000", 2024-03-27T21:41:44.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3501999999999999e-05 2024-03-27T21:41:44.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.099 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.099 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:44.099 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.935859+0000", 2024-03-27T21:41:44.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8845e-05 2024-03-27T21:41:44.100 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.100 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:44.100 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.100 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.100 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:44.101 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:44.101 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 124.15556768, 2024-03-27T21:41:44.101 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071695429999999996, 2024-03-27T21:41:44.101 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:44.101 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:44.101 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:44.102 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:44.102 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:44.102 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73 2024-03-27T21:41:44.102 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.102 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:44.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:44.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:44.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:44.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936663+0000", 2024-03-27T21:41:44.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:44.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936662+0000", 2024-03-27T21:41:44.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:44.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:44.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936737+0000", 2024-03-27T21:41:44.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5303999999999996e-05 2024-03-27T21:41:44.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:44.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936746+0000", 2024-03-27T21:41:44.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6340000000000007e-06 2024-03-27T21:41:44.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.107 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:44.107 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936753+0000", 2024-03-27T21:41:44.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0419999999999997e-06 2024-03-27T21:41:44.107 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.107 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:44.108 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937926+0000", 2024-03-27T21:41:44.108 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011731280000000001 2024-03-27T21:41:44.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.108 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.108 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:44.108 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.937969+0000", 2024-03-27T21:41:44.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3285999999999999e-05 2024-03-27T21:41:44.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.109 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:44.109 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938040+0000", 2024-03-27T21:41:44.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0358000000000006e-05 2024-03-27T21:41:44.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:44.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939324+0000", 2024-03-27T21:41:44.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012844899999999999 2024-03-27T21:41:44.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:44.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943780+0000", 2024-03-27T21:41:44.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044555630000000001 2024-03-27T21:41:44.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:44.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943811+0000", 2024-03-27T21:41:44.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1415999999999999e-05 2024-03-27T21:41:44.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:44.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943832+0000", 2024-03-27T21:41:44.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0981000000000001e-05 2024-03-27T21:41:44.113 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.113 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:44.113 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.114 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:44.114 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:44.114 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 124.15546966399999, 2024-03-27T21:41:44.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071139740000000003, 2024-03-27T21:41:44.115 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:44.115 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:44.115 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:44.115 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:44.115 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:44.115 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74 2024-03-27T21:41:44.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.116 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:44.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:44.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:44.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:44.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936761+0000", 2024-03-27T21:41:44.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:44.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936760+0000", 2024-03-27T21:41:44.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:44.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:44.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936765+0000", 2024-03-27T21:41:44.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6330000000000004e-06 2024-03-27T21:41:44.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:44.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936771+0000", 2024-03-27T21:41:44.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0560000000000001e-06 2024-03-27T21:41:44.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:44.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.936776+0000", 2024-03-27T21:41:44.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1510000000000003e-06 2024-03-27T21:41:44.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:44.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938605+0000", 2024-03-27T21:41:44.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001829589 2024-03-27T21:41:44.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.122 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:44.122 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938657+0000", 2024-03-27T21:41:44.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1860000000000002e-05 2024-03-27T21:41:44.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:44.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.938734+0000", 2024-03-27T21:41:44.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6639999999999998e-05 2024-03-27T21:41:44.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:44.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.939562+0000", 2024-03-27T21:41:44.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00082803100000000001 2024-03-27T21:41:44.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:44.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943838+0000", 2024-03-27T21:41:44.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042757050000000003 2024-03-27T21:41:44.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:44.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943859+0000", 2024-03-27T21:41:44.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1175e-05 2024-03-27T21:41:44.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:44.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:39.943875+0000", 2024-03-27T21:41:44.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5744e-05 2024-03-27T21:41:44.126 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.126 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:44.126 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.126 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:44.127 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:44.127 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 124.011955613, 2024-03-27T21:41:44.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059712230000000003, 2024-03-27T21:41:44.127 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:44.127 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:44.127 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:44.127 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:44.127 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:44.127 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 112 2024-03-27T21:41:44.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.128 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:44.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:44.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:44.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:44.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080275+0000", 2024-03-27T21:41:44.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:44.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080274+0000", 2024-03-27T21:41:44.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:44.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:44.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080318+0000", 2024-03-27T21:41:44.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3634999999999998e-05 2024-03-27T21:41:44.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:44.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080329+0000", 2024-03-27T21:41:44.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1232e-05 2024-03-27T21:41:44.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:44.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080333+0000", 2024-03-27T21:41:44.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6009999999999999e-06 2024-03-27T21:41:44.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:44.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083786+0000", 2024-03-27T21:41:44.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034528940000000002 2024-03-27T21:41:44.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:44.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083861+0000", 2024-03-27T21:41:44.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4893000000000003e-05 2024-03-27T21:41:44.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:44.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.083981+0000", 2024-03-27T21:41:44.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012058299999999999 2024-03-27T21:41:44.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:44.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086148+0000", 2024-03-27T21:41:44.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021671469999999999 2024-03-27T21:41:44.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:44.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086195+0000", 2024-03-27T21:41:44.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6768999999999999e-05 2024-03-27T21:41:44.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:44.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086224+0000", 2024-03-27T21:41:44.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8758000000000001e-05 2024-03-27T21:41:44.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:44.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086246+0000", 2024-03-27T21:41:44.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2064000000000001e-05 2024-03-27T21:41:44.133 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.133 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:44.133 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.133 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:44.134 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:44.134 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 124.01189266599999, 2024-03-27T21:41:44.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066196320000000003, 2024-03-27T21:41:44.134 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:44.134 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:44.134 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:44.134 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:44.134 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:44.134 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 113 2024-03-27T21:41:44.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.135 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:44.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:44.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:44.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:44.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080338+0000", 2024-03-27T21:41:44.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:44.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080337+0000", 2024-03-27T21:41:44.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:44.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:44.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080340+0000", 2024-03-27T21:41:44.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6690000000000002e-06 2024-03-27T21:41:44.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:44.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080344+0000", 2024-03-27T21:41:44.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6550000000000002e-06 2024-03-27T21:41:44.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:44.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080348+0000", 2024-03-27T21:41:44.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9169999999999999e-06 2024-03-27T21:41:44.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:44.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084595+0000", 2024-03-27T21:41:44.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042471719999999996 2024-03-27T21:41:44.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:44.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084676+0000", 2024-03-27T21:41:44.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1147000000000001e-05 2024-03-27T21:41:44.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:44.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.084796+0000", 2024-03-27T21:41:44.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012032900000000001 2024-03-27T21:41:44.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:44.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086190+0000", 2024-03-27T21:41:44.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013934399999999999 2024-03-27T21:41:44.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:44.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086905+0000", 2024-03-27T21:41:44.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00071536199999999999 2024-03-27T21:41:44.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:44.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086937+0000", 2024-03-27T21:41:44.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2125000000000002e-05 2024-03-27T21:41:44.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:44.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086957+0000", 2024-03-27T21:41:44.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0157999999999999e-05 2024-03-27T21:41:44.140 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.140 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:44.140 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.141 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:44.141 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:44.141 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 124.01187994999999, 2024-03-27T21:41:44.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007405661, 2024-03-27T21:41:44.141 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:44.141 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:44.142 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:44.142 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:44.142 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:44.142 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 114 2024-03-27T21:41:44.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.142 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:44.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:44.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:44.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:44.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:44.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:44.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080350+0000", 2024-03-27T21:41:44.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-03-27T21:41:44.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:44.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080353+0000", 2024-03-27T21:41:44.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.441e-06 2024-03-27T21:41:44.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:44.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080356+0000", 2024-03-27T21:41:44.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0769999999999999e-06 2024-03-27T21:41:44.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:44.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080358+0000", 2024-03-27T21:41:44.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3369999999999998e-06 2024-03-27T21:41:44.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:44.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085118+0000", 2024-03-27T21:41:44.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047603949999999997 2024-03-27T21:41:44.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:44.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085210+0000", 2024-03-27T21:41:44.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1983000000000003e-05 2024-03-27T21:41:44.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:44.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.085332+0000", 2024-03-27T21:41:44.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121803 2024-03-27T21:41:44.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:44.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086292+0000", 2024-03-27T21:41:44.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095961400000000002 2024-03-27T21:41:44.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:44.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087710+0000", 2024-03-27T21:41:44.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014181459999999999 2024-03-27T21:41:44.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:44.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087736+0000", 2024-03-27T21:41:44.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6475e-05 2024-03-27T21:41:44.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:44.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087756+0000", 2024-03-27T21:41:44.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9626000000000001e-05 2024-03-27T21:41:44.147 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.147 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:44.147 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.147 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:44.148 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:44.148 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 124.011857515, 2024-03-27T21:41:44.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075011720000000004, 2024-03-27T21:41:44.148 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:44.148 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:44.148 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:44.148 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:44.148 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:44.148 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 115 2024-03-27T21:41:44.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.148 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:44.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:44.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:44.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:44.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:44.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:44.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080373+0000", 2024-03-27T21:41:44.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:44.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:44.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080458+0000", 2024-03-27T21:41:44.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5228000000000005e-05 2024-03-27T21:41:44.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:44.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080465+0000", 2024-03-27T21:41:44.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.238e-06 2024-03-27T21:41:44.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:44.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080468+0000", 2024-03-27T21:41:44.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5109999999999999e-06 2024-03-27T21:41:44.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:44.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086326+0000", 2024-03-27T21:41:44.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0058579169999999998 2024-03-27T21:41:44.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:44.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086382+0000", 2024-03-27T21:41:44.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5309999999999997e-05 2024-03-27T21:41:44.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:44.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086457+0000", 2024-03-27T21:41:44.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5413000000000005e-05 2024-03-27T21:41:44.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:44.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087762+0000", 2024-03-27T21:41:44.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013052300000000001 2024-03-27T21:41:44.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:44.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087798+0000", 2024-03-27T21:41:44.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5614999999999997e-05 2024-03-27T21:41:44.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:44.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087840+0000", 2024-03-27T21:41:44.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2196000000000001e-05 2024-03-27T21:41:44.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:44.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087874+0000", 2024-03-27T21:41:44.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3785999999999998e-05 2024-03-27T21:41:44.153 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.153 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:44.153 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.153 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:44.154 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:44.154 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 124.011757432, 2024-03-27T21:41:44.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075050680000000002, 2024-03-27T21:41:44.154 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:44.154 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:44.154 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:44.154 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:44.154 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:44.154 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 116 2024-03-27T21:41:44.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.155 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:44.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:44.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:44.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:44.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080473+0000", 2024-03-27T21:41:44.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:44.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080472+0000", 2024-03-27T21:41:44.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:44.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:44.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080475+0000", 2024-03-27T21:41:44.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.712e-06 2024-03-27T21:41:44.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:44.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080478+0000", 2024-03-27T21:41:44.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1070000000000001e-06 2024-03-27T21:41:44.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:44.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080481+0000", 2024-03-27T21:41:44.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4269999999999998e-06 2024-03-27T21:41:44.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:44.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.086992+0000", 2024-03-27T21:41:44.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0065114099999999996 2024-03-27T21:41:44.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:44.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087064+0000", 2024-03-27T21:41:44.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1927000000000004e-05 2024-03-27T21:41:44.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:44.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087188+0000", 2024-03-27T21:41:44.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123585 2024-03-27T21:41:44.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:44.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087766+0000", 2024-03-27T21:41:44.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057814300000000004 2024-03-27T21:41:44.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:44.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087934+0000", 2024-03-27T21:41:44.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000167894 2024-03-27T21:41:44.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:44.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087958+0000", 2024-03-27T21:41:44.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4663000000000001e-05 2024-03-27T21:41:44.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:44.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087978+0000", 2024-03-27T21:41:44.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9636e-05 2024-03-27T21:41:44.159 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.159 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:44.159 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.160 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-03-27T21:41:44.160 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:44.160 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 124.01172531, 2024-03-27T21:41:44.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076850679999999998, 2024-03-27T21:41:44.160 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:44.160 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:44.160 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:44.160 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:44.160 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:44.160 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 117 2024-03-27T21:41:44.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.160 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:44.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:44.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:44.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:44.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:44.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:44.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080505+0000", 2024-03-27T21:41:44.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:44.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:44.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080508+0000", 2024-03-27T21:41:44.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.298e-06 2024-03-27T21:41:44.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:44.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080515+0000", 2024-03-27T21:41:44.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0790000000000003e-06 2024-03-27T21:41:44.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:44.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.080518+0000", 2024-03-27T21:41:44.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8509999999999999e-06 2024-03-27T21:41:44.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:44.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087444+0000", 2024-03-27T21:41:44.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0069259120000000002 2024-03-27T21:41:44.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:44.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087491+0000", 2024-03-27T21:41:44.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7043999999999999e-05 2024-03-27T21:41:44.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:44.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087564+0000", 2024-03-27T21:41:44.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3603999999999995e-05 2024-03-27T21:41:44.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:44.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.087902+0000", 2024-03-27T21:41:44.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033734100000000001 2024-03-27T21:41:44.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:44.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088117+0000", 2024-03-27T21:41:44.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000215164 2024-03-27T21:41:44.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:44.167 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088160+0000", 2024-03-27T21:41:44.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2933999999999999e-05 2024-03-27T21:41:44.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.167 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:44.167 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.088190+0000", 2024-03-27T21:41:44.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0139999999999999e-05 2024-03-27T21:41:44.167 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.167 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:44.167 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.167 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:44.168 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:44.168 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 123.595444921, 2024-03-27T21:41:44.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052465439999999997, 2024-03-27T21:41:44.168 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:44.168 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:44.168 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:44.168 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:44.168 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:44.168 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 133 2024-03-27T21:41:44.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.168 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:44.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:44.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:44.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:44.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:44.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:44.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496785+0000", 2024-03-27T21:41:44.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:44.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:44.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496788+0000", 2024-03-27T21:41:44.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.615e-06 2024-03-27T21:41:44.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:44.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496793+0000", 2024-03-27T21:41:44.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4790000000000004e-06 2024-03-27T21:41:44.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:44.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496796+0000", 2024-03-27T21:41:44.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1e-06 2024-03-27T21:41:44.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:44.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500447+0000", 2024-03-27T21:41:44.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036504459999999999 2024-03-27T21:41:44.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:44.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500492+0000", 2024-03-27T21:41:44.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5765e-05 2024-03-27T21:41:44.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:44.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500562+0000", 2024-03-27T21:41:44.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9206000000000001e-05 2024-03-27T21:41:44.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:44.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501959+0000", 2024-03-27T21:41:44.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001397187 2024-03-27T21:41:44.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:44.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501984+0000", 2024-03-27T21:41:44.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.552e-05 2024-03-27T21:41:44.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:44.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502010+0000", 2024-03-27T21:41:44.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5232000000000002e-05 2024-03-27T21:41:44.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:44.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502032+0000", 2024-03-27T21:41:44.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2337999999999999e-05 2024-03-27T21:41:44.173 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.173 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:44.173 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.173 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:44.173 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:44.173 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 123.59538626299999, 2024-03-27T21:41:44.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052892119999999997, 2024-03-27T21:41:44.173 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:44.173 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:44.173 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:44.173 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:44.174 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:44.174 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 134 2024-03-27T21:41:44.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.174 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:44.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:44.174 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:44.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.174 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:44.174 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:44.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:44.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496844+0000", 2024-03-27T21:41:44.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:44.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:44.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496929+0000", 2024-03-27T21:41:44.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5098000000000004e-05 2024-03-27T21:41:44.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:44.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496937+0000", 2024-03-27T21:41:44.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9559999999999997e-06 2024-03-27T21:41:44.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:44.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496942+0000", 2024-03-27T21:41:44.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6350000000000003e-06 2024-03-27T21:41:44.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:44.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500749+0000", 2024-03-27T21:41:44.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038068529999999998 2024-03-27T21:41:44.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:44.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500792+0000", 2024-03-27T21:41:44.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2823e-05 2024-03-27T21:41:44.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:44.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.500864+0000", 2024-03-27T21:41:44.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1715e-05 2024-03-27T21:41:44.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:44.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502068+0000", 2024-03-27T21:41:44.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012044860000000001 2024-03-27T21:41:44.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:44.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502073+0000", 2024-03-27T21:41:44.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5700000000000003e-06 2024-03-27T21:41:44.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:44.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502098+0000", 2024-03-27T21:41:44.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5120999999999999e-05 2024-03-27T21:41:44.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:44.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502133+0000", 2024-03-27T21:41:44.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5306999999999999e-05 2024-03-27T21:41:44.178 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.178 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:44.178 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.179 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:44.179 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:44.179 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 123.595280956, 2024-03-27T21:41:44.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062529040000000001, 2024-03-27T21:41:44.179 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:44.179 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:44.179 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:44.179 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:44.179 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:44.179 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 135 2024-03-27T21:41:44.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.179 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:44.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:44.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:44.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:44.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:44.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:44.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496949+0000", 2024-03-27T21:41:44.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:44.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:44.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496953+0000", 2024-03-27T21:41:44.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0709999999999996e-06 2024-03-27T21:41:44.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:44.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496961+0000", 2024-03-27T21:41:44.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1300000000000001e-06 2024-03-27T21:41:44.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:44.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496968+0000", 2024-03-27T21:41:44.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0999999999999998e-06 2024-03-27T21:41:44.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:44.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501244+0000", 2024-03-27T21:41:44.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042756310000000002 2024-03-27T21:41:44.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:44.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501377+0000", 2024-03-27T21:41:44.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013348799999999999 2024-03-27T21:41:44.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:44.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501455+0000", 2024-03-27T21:41:44.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7410000000000006e-05 2024-03-27T21:41:44.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:44.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502138+0000", 2024-03-27T21:41:44.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068386599999999999 2024-03-27T21:41:44.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:44.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503152+0000", 2024-03-27T21:41:44.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010139680000000001 2024-03-27T21:41:44.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:44.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503180+0000", 2024-03-27T21:41:44.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7426000000000001e-05 2024-03-27T21:41:44.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:44.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503202+0000", 2024-03-27T21:41:44.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2463000000000002e-05 2024-03-27T21:41:44.184 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.184 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:44.184 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.184 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:44.184 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:44.184 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 123.595256818, 2024-03-27T21:41:44.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066253680000000004, 2024-03-27T21:41:44.184 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:44.184 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:44.184 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:44.185 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:44.185 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:44.185 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 136 2024-03-27T21:41:44.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.185 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:44.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:44.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:44.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:44.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:44.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:44.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496973+0000", 2024-03-27T21:41:44.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:44.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:44.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496977+0000", 2024-03-27T21:41:44.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5049999999999999e-06 2024-03-27T21:41:44.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:44.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496984+0000", 2024-03-27T21:41:44.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1520000000000003e-06 2024-03-27T21:41:44.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:44.187 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496990+0000", 2024-03-27T21:41:44.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-03-27T21:41:44.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:44.187 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501669+0000", 2024-03-27T21:41:44.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046793859999999998 2024-03-27T21:41:44.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:44.187 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501714+0000", 2024-03-27T21:41:44.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5154999999999997e-05 2024-03-27T21:41:44.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:44.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.501783+0000", 2024-03-27T21:41:44.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8454999999999994e-05 2024-03-27T21:41:44.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:44.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503226+0000", 2024-03-27T21:41:44.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001443441 2024-03-27T21:41:44.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:44.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503540+0000", 2024-03-27T21:41:44.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031347100000000001 2024-03-27T21:41:44.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:44.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503563+0000", 2024-03-27T21:41:44.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3223000000000001e-05 2024-03-27T21:41:44.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:44.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503599+0000", 2024-03-27T21:41:44.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6105000000000001e-05 2024-03-27T21:41:44.189 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.190 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:44.190 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.190 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.190 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:44.190 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:44.190 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 123.59523400800001, 2024-03-27T21:41:44.190 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074261320000000002, 2024-03-27T21:41:44.190 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:44.190 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:44.190 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:44.190 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:44.190 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:44.190 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 137 2024-03-27T21:41:44.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.191 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:44.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:44.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:44.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:44.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:44.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:44.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.496996+0000", 2024-03-27T21:41:44.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:44.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:44.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497077+0000", 2024-03-27T21:41:44.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1599000000000003e-05 2024-03-27T21:41:44.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:44.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497085+0000", 2024-03-27T21:41:44.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6109999999999996e-06 2024-03-27T21:41:44.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:44.193 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497092+0000", 2024-03-27T21:41:44.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.742e-06 2024-03-27T21:41:44.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.193 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:44.193 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502159+0000", 2024-03-27T21:41:44.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050666870000000003 2024-03-27T21:41:44.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.193 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:44.193 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502200+0000", 2024-03-27T21:41:44.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1919999999999998e-05 2024-03-27T21:41:44.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.193 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:44.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502280+0000", 2024-03-27T21:41:44.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9417000000000002e-05 2024-03-27T21:41:44.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:44.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503605+0000", 2024-03-27T21:41:44.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001325641 2024-03-27T21:41:44.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:44.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504361+0000", 2024-03-27T21:41:44.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075555399999999997 2024-03-27T21:41:44.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.195 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:44.195 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504386+0000", 2024-03-27T21:41:44.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5369999999999999e-05 2024-03-27T21:41:44.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.195 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:44.195 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504422+0000", 2024-03-27T21:41:44.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6019000000000002e-05 2024-03-27T21:41:44.195 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.195 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:44.195 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.195 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:44.196 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:44.196 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 123.595130211, 2024-03-27T21:41:44.196 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074059750000000004, 2024-03-27T21:41:44.196 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:44.196 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:44.196 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:44.196 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:44.196 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:44.196 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 138 2024-03-27T21:41:44.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.196 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:44.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.196 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:44.196 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:44.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:44.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:44.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:44.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497100+0000", 2024-03-27T21:41:44.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:44.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:44.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497103+0000", 2024-03-27T21:41:44.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.856e-06 2024-03-27T21:41:44.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:44.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497121+0000", 2024-03-27T21:41:44.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7190999999999999e-05 2024-03-27T21:41:44.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:44.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497126+0000", 2024-03-27T21:41:44.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3770000000000004e-06 2024-03-27T21:41:44.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:44.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502833+0000", 2024-03-27T21:41:44.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057073510000000003 2024-03-27T21:41:44.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:44.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502881+0000", 2024-03-27T21:41:44.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7961999999999999e-05 2024-03-27T21:41:44.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:44.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.502949+0000", 2024-03-27T21:41:44.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7957e-05 2024-03-27T21:41:44.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:44.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504430+0000", 2024-03-27T21:41:44.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001480763 2024-03-27T21:41:44.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:44.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504459+0000", 2024-03-27T21:41:44.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8983999999999999e-05 2024-03-27T21:41:44.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:44.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504486+0000", 2024-03-27T21:41:44.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6883999999999999e-05 2024-03-27T21:41:44.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:44.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504506+0000", 2024-03-27T21:41:44.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0214e-05 2024-03-27T21:41:44.201 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.201 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:44.201 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.201 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.201 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:44.201 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:44.201 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 123.595098377, 2024-03-27T21:41:44.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074754859999999999, 2024-03-27T21:41:44.201 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:44.201 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:44.202 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:44.202 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:44.202 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:44.202 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 139 2024-03-27T21:41:44.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.202 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:44.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:44.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:44.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:44.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:44.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:44.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497132+0000", 2024-03-27T21:41:44.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:44.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:44.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497137+0000", 2024-03-27T21:41:44.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0730000000000004e-06 2024-03-27T21:41:44.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:44.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497143+0000", 2024-03-27T21:41:44.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3759999999999996e-06 2024-03-27T21:41:44.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:44.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497150+0000", 2024-03-27T21:41:44.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.002e-06 2024-03-27T21:41:44.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:44.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503249+0000", 2024-03-27T21:41:44.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060989750000000004 2024-03-27T21:41:44.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:44.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503291+0000", 2024-03-27T21:41:44.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1563999999999997e-05 2024-03-27T21:41:44.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:44.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503376+0000", 2024-03-27T21:41:44.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5752000000000002e-05 2024-03-27T21:41:44.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:44.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504545+0000", 2024-03-27T21:41:44.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169172 2024-03-27T21:41:44.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:44.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504550+0000", 2024-03-27T21:41:44.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7419999999999997e-06 2024-03-27T21:41:44.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:44.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504574+0000", 2024-03-27T21:41:44.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4124000000000001e-05 2024-03-27T21:41:44.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:44.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504607+0000", 2024-03-27T21:41:44.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3118999999999999e-05 2024-03-27T21:41:44.206 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.206 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:44.206 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.207 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.207 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:44.207 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:44.207 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 123.595074711, 2024-03-27T21:41:44.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078150589999999992, 2024-03-27T21:41:44.207 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:44.207 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:44.207 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:44.207 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:44.207 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:44.207 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 140 2024-03-27T21:41:44.207 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.207 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:44.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:44.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:44.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:44.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497156+0000", 2024-03-27T21:41:44.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:44.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497155+0000", 2024-03-27T21:41:44.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:44.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:44.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497230+0000", 2024-03-27T21:41:44.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4789000000000005e-05 2024-03-27T21:41:44.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:44.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497241+0000", 2024-03-27T21:41:44.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1442e-05 2024-03-27T21:41:44.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:44.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497255+0000", 2024-03-27T21:41:44.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3087e-05 2024-03-27T21:41:44.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:44.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503630+0000", 2024-03-27T21:41:44.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063755269999999998 2024-03-27T21:41:44.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:44.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503673+0000", 2024-03-27T21:41:44.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2616e-05 2024-03-27T21:41:44.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:44.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.503738+0000", 2024-03-27T21:41:44.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5114e-05 2024-03-27T21:41:44.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:44.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504907+0000", 2024-03-27T21:41:44.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001169432 2024-03-27T21:41:44.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:44.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504926+0000", 2024-03-27T21:41:44.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9086999999999999e-05 2024-03-27T21:41:44.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:44.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504950+0000", 2024-03-27T21:41:44.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3598999999999999e-05 2024-03-27T21:41:44.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.212 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:44.212 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504971+0000", 2024-03-27T21:41:44.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0772999999999999e-05 2024-03-27T21:41:44.212 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.212 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:44.212 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.212 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.212 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:44.212 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:44.212 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 123.59496767100001, 2024-03-27T21:41:44.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082962590000000003, 2024-03-27T21:41:44.212 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:44.213 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:44.213 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:44.213 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:44.213 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:44.213 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 141 2024-03-27T21:41:44.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.213 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:44.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:44.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:44.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:44.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497263+0000", 2024-03-27T21:41:44.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:44.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497262+0000", 2024-03-27T21:41:44.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:44.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:44.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497267+0000", 2024-03-27T21:41:44.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.955e-06 2024-03-27T21:41:44.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:44.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497275+0000", 2024-03-27T21:41:44.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.824e-06 2024-03-27T21:41:44.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:44.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.497280+0000", 2024-03-27T21:41:44.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8230000000000002e-06 2024-03-27T21:41:44.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:44.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504628+0000", 2024-03-27T21:41:44.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007348561 2024-03-27T21:41:44.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:44.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504673+0000", 2024-03-27T21:41:44.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4313000000000001e-05 2024-03-27T21:41:44.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:44.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.504745+0000", 2024-03-27T21:41:44.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2303000000000002e-05 2024-03-27T21:41:44.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:44.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505358+0000", 2024-03-27T21:41:44.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061281300000000003 2024-03-27T21:41:44.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.217 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:44.217 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505503+0000", 2024-03-27T21:41:44.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000145134 2024-03-27T21:41:44.217 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.217 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:44.217 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505526+0000", 2024-03-27T21:41:44.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3312e-05 2024-03-27T21:41:44.217 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.217 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:44.217 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:40.505559+0000", 2024-03-27T21:41:44.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2821999999999998e-05 2024-03-27T21:41:44.217 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.218 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:44.218 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.218 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.4175.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-03-27T21:41:44.218 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:44.218 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 122.985745432, 2024-03-27T21:41:44.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053753339999999998, 2024-03-27T21:41:44.218 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-03-27T21:41:44.218 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:44.218 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-03-27T21:41:44.218 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.4175", 2024-03-27T21:41:44.218 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.60:0/42544838", 2024-03-27T21:41:44.218 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159 2024-03-27T21:41:44.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.219 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-03-27T21:41:44.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-03-27T21:41:44.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:44.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-03-27T21:41:44.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:44.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-03-27T21:41:44.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-03-27T21:41:44.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106485+0000", 2024-03-27T21:41:44.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:44.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-03-27T21:41:44.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106488+0000", 2024-03-27T21:41:44.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9830000000000001e-06 2024-03-27T21:41:44.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-03-27T21:41:44.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106491+0000", 2024-03-27T21:41:44.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3459999999999998e-06 2024-03-27T21:41:44.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-03-27T21:41:44.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.106493+0000", 2024-03-27T21:41:44.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.509e-06 2024-03-27T21:41:44.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-03-27T21:41:44.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110636+0000", 2024-03-27T21:41:44.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041427260000000002 2024-03-27T21:41:44.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-03-27T21:41:44.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110683+0000", 2024-03-27T21:41:44.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6557000000000002e-05 2024-03-27T21:41:44.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0", 2024-03-27T21:41:44.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.110755+0000", 2024-03-27T21:41:44.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2119000000000005e-05 2024-03-27T21:41:44.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-03-27T21:41:44.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111415+0000", 2024-03-27T21:41:44.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066065199999999996 2024-03-27T21:41:44.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-03-27T21:41:44.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111793+0000", 2024-03-27T21:41:44.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037732100000000001 2024-03-27T21:41:44.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-03-27T21:41:44.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111829+0000", 2024-03-27T21:41:44.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6418999999999998e-05 2024-03-27T21:41:44.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-27T21:41:44.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-03-27T21:41:44.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-03-27T21:41:44.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-03-27T21:39:41.111860+0000", 2024-03-27T21:41:44.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1003000000000001e-05 2024-03-27T21:41:44.223 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.223 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:44.223 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.223 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-27T21:41:44.223 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-27T21:41:44.223 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-27T21:41:44.226 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-27T21:41:44.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.177+0000 7f9d3affd640 1 -- 172.21.15.60:0/2910650472 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v73) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9d2c079980 con 0x7f9d4413dc90 2024-03-27T21:41:44.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.177+0000 7f9d3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set", "key": "nodeep-scrub"}] 2024-03-27T21:41:44.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.177+0000 7f9d3affd640 10 monclient: _finish_command 2 = system:0 nodeep-scrub is set 2024-03-27T21:41:44.227 INFO:teuthology.orchestra.run.smithi060.stderr:nodeep-scrub is set 2024-03-27T21:41:44.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.179+0000 7f9d4885b640 10 librados: watch_flush enter 2024-03-27T21:41:44.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.179+0000 7f9d4885b640 10 librados: watch_flush exit 2024-03-27T21:41:44.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.179+0000 7f9d4885b640 20 client.5071.objecter shutdown clearing up homeless session... 2024-03-27T21:41:44.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.179+0000 7f9d4885b640 10 client.5071.objecter successfully canceled tick 2024-03-27T21:41:44.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.179+0000 7f9d4885b640 1 -- 172.21.15.60:0/2910650472 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9d2406c590 msgr2=0x7f9d2406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:44.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.179+0000 7f9d4885b640 1 --2- 172.21.15.60:0/2910650472 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9d2406c590 0x7f9d2406ea50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9d34001c70 tx=0x7f9d34009040 comp rx=0 tx=0).stop 2024-03-27T21:41:44.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.179+0000 7f9d4885b640 10 monclient: shutdown 2024-03-27T21:41:44.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.179+0000 7f9d4885b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:44.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.179+0000 7f9d4885b640 1 -- 172.21.15.60:0/2910650472 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d4413dc90 msgr2=0x7f9d441d98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:44.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.179+0000 7f9d4885b640 1 --2- 172.21.15.60:0/2910650472 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d4413dc90 0x7f9d441d98a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000bbf0 tx=0x7f9d2c004730 comp rx=0 tx=0).stop 2024-03-27T21:41:44.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.179+0000 7f9d4885b640 1 -- 172.21.15.60:0/2910650472 shutdown_connections 2024-03-27T21:41:44.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.179+0000 7f9d4885b640 1 --2- 172.21.15.60:0/2910650472 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d441d9de0 0x7f9d4413a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:44.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.179+0000 7f9d4885b640 1 --2- 172.21.15.60:0/2910650472 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9d2406c590 0x7f9d2406ea50 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:44.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.179+0000 7f9d4885b640 1 --2- 172.21.15.60:0/2910650472 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d4413dc90 0x7f9d441d98a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:44.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.179+0000 7f9d4885b640 1 --2- 172.21.15.60:0/2910650472 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9d30004090 0x7f9d4413e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:44.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.179+0000 7f9d4885b640 1 -- 172.21.15.60:0/2910650472 >> 172.21.15.60:0/2910650472 conn(0x7f9d440608e0 msgr2=0x7f9d440b6880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:44.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.179+0000 7f9d4885b640 1 -- 172.21.15.60:0/2910650472 shutdown_connections 2024-03-27T21:41:44.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.179+0000 7f9d4885b640 1 -- 172.21.15.60:0/2910650472 wait complete. 2024-03-27T21:41:44.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:44.179+0000 7f9d4885b640 1 librados: shutdown 2024-03-27T21:41:44.229 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:44.238+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:44.325 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:44.339+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:44.375 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:44.375 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:44.375 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:44.375 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:44.383 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-27T21:41:44.426 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:44.439+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:44.527 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:44.541+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:44.531 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:44.531 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:44.531 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:41:44.531 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:41:44.531 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:44.540 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-27T21:41:44.628 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-27T21:41:44.642+0000 7f1291506640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 34326) UID: 0 2024-03-27T21:41:44.690 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:41:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:41:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [] 2024-03-27T21:41:44.690 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:44.700 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-27T21:41:44.728 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-27T21:41:44.743+0000 7f9122a36640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 34324) UID: 0 2024-03-27T21:41:44.808 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:44.809 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:44.809 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:44.809 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:44.818 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-27T21:41:44.829 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-27T21:41:44.844+0000 7f491dd86640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34330) UID: 0 2024-03-27T21:41:44.924 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:44.924 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:44.924 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-03-27T21:41:44.925 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-03-27T21:41:44.925 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:44.930 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-27T21:41:44.944+0000 7fb850f34640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 56426) UID: 0 2024-03-27T21:41:44.933 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-27T21:41:45.031 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:45.045+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:45.083 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:45.083 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-03-27T21:41:45.083 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-03-27T21:41:45.083 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-03-27T21:41:45.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.083 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:20 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'10, mlcod=17'8)", 2024-03-27T21:41:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:41:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 125.20791296, 2024-03-27T21:41:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015345630000000001, 2024-03-27T21:41:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:41:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888563+0000", 2024-03-27T21:41:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888562+0000", 2024-03-27T21:41:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888735+0000", 2024-03-27T21:41:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017254799999999999 2024-03-27T21:41:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888747+0000", 2024-03-27T21:41:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2469999999999999e-05 2024-03-27T21:41:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888761+0000", 2024-03-27T21:41:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3254e-05 2024-03-27T21:41:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888804+0000", 2024-03-27T21:41:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3129e-05 2024-03-27T21:41:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.888820+0000", 2024-03-27T21:41:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6426000000000002e-05 2024-03-27T21:41:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890060+0000", 2024-03-27T21:41:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012398050000000001 2024-03-27T21:41:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890098+0000", 2024-03-27T21:41:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7795e-05 2024-03-27T21:41:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:21 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2024-03-27T21:41:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:41:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 125.20708856, 2024-03-27T21:41:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001069555, 2024-03-27T21:41:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:41:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889388+0000", 2024-03-27T21:41:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889387+0000", 2024-03-27T21:41:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889455+0000", 2024-03-27T21:41:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7930999999999997e-05 2024-03-27T21:41:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:45.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889470+0000", 2024-03-27T21:41:45.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5554e-05 2024-03-27T21:41:45.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:45.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889476+0000", 2024-03-27T21:41:45.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0619999999999997e-06 2024-03-27T21:41:45.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889513+0000", 2024-03-27T21:41:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6606999999999997e-05 2024-03-27T21:41:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.889528+0000", 2024-03-27T21:41:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4982e-05 2024-03-27T21:41:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890421+0000", 2024-03-27T21:41:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00089293700000000001 2024-03-27T21:41:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.890457+0000", 2024-03-27T21:41:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6260000000000002e-05 2024-03-27T21:41:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:38 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'24, mlcod=17'22)", 2024-03-27T21:41:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:41:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 125.189852594, 2024-03-27T21:41:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016182080000000001, 2024-03-27T21:41:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:41:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906624+0000", 2024-03-27T21:41:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906623+0000", 2024-03-27T21:41:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906687+0000", 2024-03-27T21:41:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3706000000000003e-05 2024-03-27T21:41:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906698+0000", 2024-03-27T21:41:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1752e-05 2024-03-27T21:41:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906704+0000", 2024-03-27T21:41:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3940000000000004e-06 2024-03-27T21:41:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906741+0000", 2024-03-27T21:41:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6944999999999999e-05 2024-03-27T21:41:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.906760+0000", 2024-03-27T21:41:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9293e-05 2024-03-27T21:41:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908211+0000", 2024-03-27T21:41:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014511350000000001 2024-03-27T21:41:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908242+0000", 2024-03-27T21:41:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0712e-05 2024-03-27T21:41:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:39 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'25, mlcod=17'22)", 2024-03-27T21:41:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 125.189007679, 2024-03-27T21:41:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012060199999999999, 2024-03-27T21:41:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907468+0000", 2024-03-27T21:41:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:45.098 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.098 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.098 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:45.098 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907533+0000", 2024-03-27T21:41:45.098 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5675999999999999e-05 2024-03-27T21:41:45.098 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.098 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.098 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:45.098 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907545+0000", 2024-03-27T21:41:45.098 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.115e-05 2024-03-27T21:41:45.098 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.098 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.098 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:45.098 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907550+0000", 2024-03-27T21:41:45.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8320000000000002e-06 2024-03-27T21:41:45.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.099 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:45.099 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907592+0000", 2024-03-27T21:41:45.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1309e-05 2024-03-27T21:41:45.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.099 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:45.099 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.907611+0000", 2024-03-27T21:41:45.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9009e-05 2024-03-27T21:41:45.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.099 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:45.099 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908638+0000", 2024-03-27T21:41:45.100 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001027533 2024-03-27T21:41:45.100 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.100 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.100 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:45.100 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.908675+0000", 2024-03-27T21:41:45.100 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6329999999999999e-05 2024-03-27T21:41:45.100 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.100 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:45.100 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.100 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.100 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.100 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:57 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'35, mlcod=17'33)", 2024-03-27T21:41:45.100 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:41:45.100 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 125.174923247, 2024-03-27T21:41:45.100 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001346578, 2024-03-27T21:41:45.101 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:45.101 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:45.101 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:45.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.101 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:45.101 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:41:45.101 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.101 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:45.101 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921553+0000", 2024-03-27T21:41:45.101 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.101 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921552+0000", 2024-03-27T21:41:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921615+0000", 2024-03-27T21:41:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2787999999999996e-05 2024-03-27T21:41:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921626+0000", 2024-03-27T21:41:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0888e-05 2024-03-27T21:41:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:45.103 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921635+0000", 2024-03-27T21:41:45.103 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4639999999999995e-06 2024-03-27T21:41:45.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.103 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.103 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:45.103 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921673+0000", 2024-03-27T21:41:45.103 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7891e-05 2024-03-27T21:41:45.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.103 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.103 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:45.103 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.921694+0000", 2024-03-27T21:41:45.103 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0426000000000001e-05 2024-03-27T21:41:45.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922865+0000", 2024-03-27T21:41:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011709509999999999 2024-03-27T21:41:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.922900+0000", 2024-03-27T21:41:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4819e-05 2024-03-27T21:41:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:71 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'45, mlcod=17'43)", 2024-03-27T21:41:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:41:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 125.15962453500001, 2024-03-27T21:41:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001460907, 2024-03-27T21:41:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:41:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936852+0000", 2024-03-27T21:41:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936851+0000", 2024-03-27T21:41:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936903+0000", 2024-03-27T21:41:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2302999999999997e-05 2024-03-27T21:41:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936915+0000", 2024-03-27T21:41:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1967000000000001e-05 2024-03-27T21:41:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936923+0000", 2024-03-27T21:41:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9370000000000004e-06 2024-03-27T21:41:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.936970+0000", 2024-03-27T21:41:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6987000000000003e-05 2024-03-27T21:41:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937002+0000", 2024-03-27T21:41:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2410999999999999e-05 2024-03-27T21:41:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938279+0000", 2024-03-27T21:41:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001276552 2024-03-27T21:41:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938313+0000", 2024-03-27T21:41:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3526999999999999e-05 2024-03-27T21:41:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:72 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'46, mlcod=17'43)", 2024-03-27T21:41:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:41:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 125.158986973, 2024-03-27T21:41:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014239490000000001, 2024-03-27T21:41:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:41:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.110 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:45.110 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937489+0000", 2024-03-27T21:41:45.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.110 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:45.110 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937488+0000", 2024-03-27T21:41:45.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:45.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.110 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937547+0000", 2024-03-27T21:41:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8196999999999999e-05 2024-03-27T21:41:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937559+0000", 2024-03-27T21:41:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1928e-05 2024-03-27T21:41:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937564+0000", 2024-03-27T21:41:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4759999999999997e-06 2024-03-27T21:41:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937621+0000", 2024-03-27T21:41:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6712000000000002e-05 2024-03-27T21:41:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.937640+0000", 2024-03-27T21:41:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9517e-05 2024-03-27T21:41:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938883+0000", 2024-03-27T21:41:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001242567 2024-03-27T21:41:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938913+0000", 2024-03-27T21:41:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0287e-05 2024-03-27T21:41:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:73 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'47, mlcod=17'43)", 2024-03-27T21:41:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 125.158310879, 2024-03-27T21:41:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010746239999999999, 2024-03-27T21:41:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:45.114 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:45.114 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:45.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.114 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:45.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:45.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.114 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:45.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:45.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938165+0000", 2024-03-27T21:41:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.999999 2024-03-27T21:41:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938217+0000", 2024-03-27T21:41:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2743e-05 2024-03-27T21:41:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938229+0000", 2024-03-27T21:41:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1956999999999999e-05 2024-03-27T21:41:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938235+0000", 2024-03-27T21:41:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4589999999999997e-06 2024-03-27T21:41:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938341+0000", 2024-03-27T21:41:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000106054 2024-03-27T21:41:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.938369+0000", 2024-03-27T21:41:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8291000000000001e-05 2024-03-27T21:41:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939209+0000", 2024-03-27T21:41:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084045400000000003 2024-03-27T21:41:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.939240+0000", 2024-03-27T21:41:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0429999999999998e-05 2024-03-27T21:41:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:75 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'49, mlcod=17'47)", 2024-03-27T21:41:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:41:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 125.151873232, 2024-03-27T21:41:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001017001, 2024-03-27T21:41:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:41:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944603+0000", 2024-03-27T21:41:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944601+0000", 2024-03-27T21:41:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999986 2024-03-27T21:41:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944614+0000", 2024-03-27T21:41:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2496e-05 2024-03-27T21:41:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944640+0000", 2024-03-27T21:41:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5865999999999999e-05 2024-03-27T21:41:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944651+0000", 2024-03-27T21:41:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1797000000000001e-05 2024-03-27T21:41:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944762+0000", 2024-03-27T21:41:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110412 2024-03-27T21:41:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.944820+0000", 2024-03-27T21:41:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7649999999999999e-05 2024-03-27T21:41:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945585+0000", 2024-03-27T21:41:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076562600000000005 2024-03-27T21:41:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.945620+0000", 2024-03-27T21:41:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4764000000000001e-05 2024-03-27T21:41:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:45.122 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.122 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:86 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'50)", 2024-03-27T21:41:45.122 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:45.122 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 125.144111745, 2024-03-27T21:41:45.122 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011392380000000001, 2024-03-27T21:41:45.122 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:45.122 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:45.122 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:45.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.122 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:45.122 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:45.122 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952364+0000", 2024-03-27T21:41:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952373+0000", 2024-03-27T21:41:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9220000000000003e-06 2024-03-27T21:41:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952381+0000", 2024-03-27T21:41:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3070000000000003e-06 2024-03-27T21:41:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952385+0000", 2024-03-27T21:41:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.121e-06 2024-03-27T21:41:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952419+0000", 2024-03-27T21:41:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4082999999999999e-05 2024-03-27T21:41:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952440+0000", 2024-03-27T21:41:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0837999999999999e-05 2024-03-27T21:41:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953472+0000", 2024-03-27T21:41:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001032119 2024-03-27T21:41:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.953504+0000", 2024-03-27T21:41:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1250000000000001e-05 2024-03-27T21:41:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:87 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'51)", 2024-03-27T21:41:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:41:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 125.143744488, 2024-03-27T21:41:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00135519, 2024-03-27T21:41:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:41:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952732+0000", 2024-03-27T21:41:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952731+0000", 2024-03-27T21:41:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952781+0000", 2024-03-27T21:41:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0161000000000003e-05 2024-03-27T21:41:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952794+0000", 2024-03-27T21:41:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2853e-05 2024-03-27T21:41:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952799+0000", 2024-03-27T21:41:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5560000000000001e-06 2024-03-27T21:41:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952832+0000", 2024-03-27T21:41:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3834999999999997e-05 2024-03-27T21:41:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.952851+0000", 2024-03-27T21:41:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8289999999999999e-05 2024-03-27T21:41:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954055+0000", 2024-03-27T21:41:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012037949999999999 2024-03-27T21:41:45.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:45.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.954087+0000", 2024-03-27T21:41:45.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2292999999999997e-05 2024-03-27T21:41:45.130 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.130 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:45.130 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.130 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:89 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'59, mlcod=17'57)", 2024-03-27T21:41:45.130 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:41:45.130 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 125.141470267, 2024-03-27T21:41:45.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00138893, 2024-03-27T21:41:45.131 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:45.131 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:45.131 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:45.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.131 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:45.131 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:41:45.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.131 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:45.131 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955006+0000", 2024-03-27T21:41:45.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955005+0000", 2024-03-27T21:41:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955071+0000", 2024-03-27T21:41:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5290999999999995e-05 2024-03-27T21:41:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955081+0000", 2024-03-27T21:41:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0572e-05 2024-03-27T21:41:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955086+0000", 2024-03-27T21:41:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8380000000000001e-06 2024-03-27T21:41:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955122+0000", 2024-03-27T21:41:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5698000000000003e-05 2024-03-27T21:41:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955140+0000", 2024-03-27T21:41:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7900000000000001e-05 2024-03-27T21:41:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956366+0000", 2024-03-27T21:41:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001226471 2024-03-27T21:41:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956395+0000", 2024-03-27T21:41:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8782e-05 2024-03-27T21:41:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:90 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'60, mlcod=17'57)", 2024-03-27T21:41:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 125.14078569, 2024-03-27T21:41:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010778859999999999, 2024-03-27T21:41:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955690+0000", 2024-03-27T21:41:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955747+0000", 2024-03-27T21:41:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7130999999999999e-05 2024-03-27T21:41:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955756+0000", 2024-03-27T21:41:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2750000000000001e-06 2024-03-27T21:41:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955761+0000", 2024-03-27T21:41:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3900000000000003e-06 2024-03-27T21:41:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955797+0000", 2024-03-27T21:41:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6650000000000003e-05 2024-03-27T21:41:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.955816+0000", 2024-03-27T21:41:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8195000000000001e-05 2024-03-27T21:41:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956739+0000", 2024-03-27T21:41:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00092369700000000002 2024-03-27T21:41:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:39.956768+0000", 2024-03-27T21:41:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9048e-05 2024-03-27T21:41:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:118 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'80, mlcod=17'78)", 2024-03-27T21:41:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 125.007794867, 2024-03-27T21:41:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014200669999999999, 2024-03-27T21:41:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088681+0000", 2024-03-27T21:41:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.141 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:45.141 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088756+0000", 2024-03-27T21:41:45.141 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4902999999999998e-05 2024-03-27T21:41:45.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.141 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:45.141 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088764+0000", 2024-03-27T21:41:45.141 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8169999999999999e-06 2024-03-27T21:41:45.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.141 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:45.142 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088767+0000", 2024-03-27T21:41:45.142 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8310000000000003e-06 2024-03-27T21:41:45.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.142 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:45.142 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088798+0000", 2024-03-27T21:41:45.142 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0800999999999999e-05 2024-03-27T21:41:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.088817+0000", 2024-03-27T21:41:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8467000000000001e-05 2024-03-27T21:41:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090074+0000", 2024-03-27T21:41:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012574089999999999 2024-03-27T21:41:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090101+0000", 2024-03-27T21:41:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7223e-05 2024-03-27T21:41:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:119 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'81, mlcod=17'78)", 2024-03-27T21:41:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 125.007060072, 2024-03-27T21:41:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014990870000000001, 2024-03-27T21:41:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089416+0000", 2024-03-27T21:41:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089491+0000", 2024-03-27T21:41:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4805e-05 2024-03-27T21:41:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089498+0000", 2024-03-27T21:41:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0650000000000001e-06 2024-03-27T21:41:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089501+0000", 2024-03-27T21:41:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.54e-06 2024-03-27T21:41:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:45.147 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089533+0000", 2024-03-27T21:41:45.147 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1838999999999998e-05 2024-03-27T21:41:45.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:45.147 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.089550+0000", 2024-03-27T21:41:45.147 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7465999999999998e-05 2024-03-27T21:41:45.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:45.148 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090890+0000", 2024-03-27T21:41:45.148 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013397979999999999 2024-03-27T21:41:45.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.148 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:45.148 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090915+0000", 2024-03-27T21:41:45.148 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4905e-05 2024-03-27T21:41:45.148 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.148 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:45.148 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:120 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'82, mlcod=17'78)", 2024-03-27T21:41:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 125.006260192, 2024-03-27T21:41:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001527093, 2024-03-27T21:41:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:45.150 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.150 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.150 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:45.150 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090216+0000", 2024-03-27T21:41:45.150 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:45.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.150 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.150 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:45.150 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090267+0000", 2024-03-27T21:41:45.150 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1477000000000002e-05 2024-03-27T21:41:45.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.150 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.150 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090273+0000", 2024-03-27T21:41:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9519999999999999e-06 2024-03-27T21:41:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090276+0000", 2024-03-27T21:41:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.219e-06 2024-03-27T21:41:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090304+0000", 2024-03-27T21:41:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8073999999999999e-05 2024-03-27T21:41:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.090316+0000", 2024-03-27T21:41:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2088e-05 2024-03-27T21:41:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091717+0000", 2024-03-27T21:41:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014003889999999999 2024-03-27T21:41:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091743+0000", 2024-03-27T21:41:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6234e-05 2024-03-27T21:41:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:121 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'83, mlcod=17'78)", 2024-03-27T21:41:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 125.00544982700001, 2024-03-27T21:41:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010226899999999999, 2024-03-27T21:41:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091026+0000", 2024-03-27T21:41:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091142+0000", 2024-03-27T21:41:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011608300000000001 2024-03-27T21:41:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091151+0000", 2024-03-27T21:41:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2350000000000005e-06 2024-03-27T21:41:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091156+0000", 2024-03-27T21:41:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5419999999999999e-06 2024-03-27T21:41:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091191+0000", 2024-03-27T21:41:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5453000000000001e-05 2024-03-27T21:41:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.091208+0000", 2024-03-27T21:41:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7119999999999999e-05 2024-03-27T21:41:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092024+0000", 2024-03-27T21:41:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000815539 2024-03-27T21:41:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.092049+0000", 2024-03-27T21:41:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5171e-05 2024-03-27T21:41:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.157 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:45.157 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.157 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.157 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:142 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'100, mlcod=18'98)", 2024-03-27T21:41:45.157 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:45.157 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 124.590442873, 2024-03-27T21:41:45.157 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001198778, 2024-03-27T21:41:45.157 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:45.157 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:45.157 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:45.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.157 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506033+0000", 2024-03-27T21:41:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506101+0000", 2024-03-27T21:41:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8202999999999997e-05 2024-03-27T21:41:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506109+0000", 2024-03-27T21:41:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3059999999999998e-06 2024-03-27T21:41:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506113+0000", 2024-03-27T21:41:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0119999999999999e-06 2024-03-27T21:41:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506137+0000", 2024-03-27T21:41:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4043999999999999e-05 2024-03-27T21:41:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506148+0000", 2024-03-27T21:41:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0446e-05 2024-03-27T21:41:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507213+0000", 2024-03-27T21:41:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001065215 2024-03-27T21:41:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507232+0000", 2024-03-27T21:41:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9038e-05 2024-03-27T21:41:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:143 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'101, mlcod=18'98)", 2024-03-27T21:41:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 124.58983025400001, 2024-03-27T21:41:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001357488, 2024-03-27T21:41:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506646+0000", 2024-03-27T21:41:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506723+0000", 2024-03-27T21:41:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7404000000000006e-05 2024-03-27T21:41:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506730+0000", 2024-03-27T21:41:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5749999999999997e-06 2024-03-27T21:41:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506733+0000", 2024-03-27T21:41:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7979999999999999e-06 2024-03-27T21:41:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506758+0000", 2024-03-27T21:41:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4179e-05 2024-03-27T21:41:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.506768+0000", 2024-03-27T21:41:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0740000000000001e-05 2024-03-27T21:41:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507978+0000", 2024-03-27T21:41:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012093900000000001 2024-03-27T21:41:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508003+0000", 2024-03-27T21:41:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5667e-05 2024-03-27T21:41:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.4175.0:144 1.0 e18/16 1:30306672:devicehealth::main.db.0000000000000000:head v 18'102, mlcod=18'98)", 2024-03-27T21:41:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:41:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 124.589171357, 2024-03-27T21:41:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013453390000000001, 2024-03-27T21:41:45.166 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-03-27T21:41:45.166 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-27T21:41:45.166 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-03-27T21:41:45.166 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.166 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-03-27T21:41:45.166 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:41:45.166 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.166 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.166 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.166 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-03-27T21:41:45.166 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507305+0000", 2024-03-27T21:41:45.166 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-03-27T21:41:45.166 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-03-27T21:41:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507304+0000", 2024-03-27T21:41:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999995 2024-03-27T21:41:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-03-27T21:41:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507397+0000", 2024-03-27T21:41:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2995e-05 2024-03-27T21:41:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-03-27T21:41:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507403+0000", 2024-03-27T21:41:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8640000000000001e-06 2024-03-27T21:41:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-03-27T21:41:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507407+0000", 2024-03-27T21:41:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5710000000000002e-06 2024-03-27T21:41:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-03-27T21:41:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507428+0000", 2024-03-27T21:41:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1192e-05 2024-03-27T21:41:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-03-27T21:41:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.507438+0000", 2024-03-27T21:41:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8309999999999996e-06 2024-03-27T21:41:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-03-27T21:41:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508624+0000", 2024-03-27T21:41:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011861210000000001 2024-03-27T21:41:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-27T21:41:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-27T21:41:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-03-27T21:41:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-03-27T21:39:40.508650+0000", 2024-03-27T21:41:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6118e-05 2024-03-27T21:41:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-27T21:41:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-27T21:41:45.169 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:45.170 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:41:45.172 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T21:41:45.172 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-27T21:41:45.185+0000 7fa175cc9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 34581) UID: 0 2024-03-27T21:41:45.272 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:45.286+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.327+0000 7f8744111640 10 monclient: get_monmap_and_config 2024-03-27T21:41:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.327+0000 7f8744111640 10 monclient: build_initial_monmap 2024-03-27T21:41:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.327+0000 7f8744111640 10 monclient: monmap: 2024-03-27T21:41:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:45.328278+0000 2024-03-27T21:41:45.314 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:45.328278+0000 2024-03-27T21:41:45.314 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:45.314 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:45.314 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:45.314 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:45.314 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:45.314 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:45.314 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:45.314 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:45.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.327+0000 7f8744111640 1 Processor -- start 2024-03-27T21:41:45.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.328+0000 7f8744111640 1 -- start start 2024-03-27T21:41:45.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.328+0000 7f8744111640 10 monclient: init 2024-03-27T21:41:45.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.328+0000 7f8744111640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:45.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.328+0000 7f8744111640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-03-27T21:41:45.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.328+0000 7f8744111640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f873c071a50 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.328+0000 7f8744111640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f873c13ef20 0x7f873c0a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.328+0000 7f8744111640 10 monclient(hunting): picked mon.noname-c con 0x7f873c13ef20 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.328+0000 7f8744111640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f873c141630 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.328+0000 7f8744111640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873c0a8e90 con 0x7f873c071a50 2024-03-27T21:41:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.328+0000 7f8744111640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873c0aa040 con 0x7f873c141630 2024-03-27T21:41:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.328+0000 7f8744111640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.328+0000 7f8744111640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f873c071e30 con 0x7f873c13ef20 2024-03-27T21:41:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.328+0000 7f8744111640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.328+0000 7f8744111640 10 monclient(hunting): authenticate will time out at 946.549641s 2024-03-27T21:41:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.328+0000 7f8741ead640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f873c071a50 0x7f873c13e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47994/0 (socket says 172.21.15.60:47994) 2024-03-27T21:41:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.328+0000 7f8741ead640 1 -- 172.21.15.60:0/3510877967 learned_addr learned my addr 172.21.15.60:0/3510877967 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f87416ac640 1 --2- 172.21.15.60:0/3510877967 >> v2:172.21.15.191:3300/0 conn(0x7f873c13ef20 0x7f873c0a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f87416ac640 10 monclient(hunting): get_auth_request con 0x7f873c13ef20 auth_method 0 2024-03-27T21:41:45.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 1 -- 172.21.15.60:0/3510877967 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806078767 0 0) 0x7f873c0a8e90 con 0x7f873c071a50 2024-03-27T21:41:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f87416ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f87416ac640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f87416ac640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 10 monclient(hunting): _negotiate 2024-03-27T21:41:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 10 monclient(hunting): my global_id is 5130 2024-03-27T21:41:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 1 -- 172.21.15.60:0/3510877967 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8728003520 con 0x7f873c071a50 2024-03-27T21:41:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f87416ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f87416ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f87416ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 1 -- 172.21.15.60:0/3510877967 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4054108846 0 0) 0x7f8728003520 con 0x7f873c071a50 2024-03-27T21:41:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 1 -- 172.21.15.60:0/3510877967 >> v2:172.21.15.191:3300/0 conn(0x7f873c13ef20 msgr2=0x7f873c0a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 1 --2- 172.21.15.60:0/3510877967 >> v2:172.21.15.191:3300/0 conn(0x7f873c13ef20 0x7f873c0a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:45.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 1 -- 172.21.15.60:0/3510877967 >> v1:172.21.15.191:6789/0 conn(0x7f873c141630 legacy=0x7f873c141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 10 monclient: _finish_hunting 0 2024-03-27T21:41:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 1 monclient: found mon.noname-a-legacy 2024-03-27T21:41:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.60:6789/0 2024-03-27T21:41:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 1 -- 172.21.15.60:0/3510877967 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c0aa2b0 con 0x7f873c071a50 2024-03-27T21:41:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 10 monclient: _finish_auth 0 2024-03-27T21:41:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 10 monclient: _check_auth_tickets 2024-03-27T21:41:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 1 -- 172.21.15.60:0/3510877967 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2933500084 0 0) 0x7f8738001670 con 0x7f873c071a50 2024-03-27T21:41:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8744111640 5 monclient: authenticate success, global_id 5130 2024-03-27T21:41:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8744111640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-03-27T21:41:45.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 10 monclient: dump: 2024-03-27T21:41:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:45.330 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] but i'm connected to v1:172.21.15.60:6789/0, reconnecting 2024-03-27T21:41:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:45.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 1 -- 172.21.15.60:0/3510877967 >> v1:172.21.15.60:6789/0 conn(0x7f873c071a50 legacy=0x7f873c13e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:41:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 1 --2- 172.21.15.60:0/3510877967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8728003970 0x7f8728003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 10 monclient(hunting): picked mon.a con 0x7f8728003970 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 1 --2- 172.21.15.60:0/3510877967 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87280047c0 0x7f8728004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 10 monclient(hunting): picked mon.b con 0x7f87280047c0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 1 --2- 172.21.15.60:0/3510877967 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8728005180 0x7f872800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 10 monclient(hunting): picked mon.c con 0x7f8728005180 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 1 -- 172.21.15.60:0/3510877967 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c071e30 con 0x7f8728003970 2024-03-27T21:41:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 1 -- 172.21.15.60:0/3510877967 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872800c020 con 0x7f87280047c0 2024-03-27T21:41:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 1 -- 172.21.15.60:0/3510877967 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f872800c1e0 con 0x7f8728005180 2024-03-27T21:41:45.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8740eab640 10 monclient(hunting): _finish_auth 0 2024-03-27T21:41:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8744111640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8741ead640 1 --2- 172.21.15.60:0/3510877967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8728003970 0x7f8728003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8741ead640 10 monclient(hunting): get_auth_request con 0x7f8728003970 auth_method 0 2024-03-27T21:41:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8741ead640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8741ead640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8741ead640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:41:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f87416ac640 1 --2- 172.21.15.60:0/3510877967 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87280047c0 0x7f8728004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f87416ac640 10 monclient(hunting): get_auth_request con 0x7f87280047c0 auth_method 0 2024-03-27T21:41:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f87416ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f87416ac640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f87416ac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:41:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8741ead640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8741ead640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8741ead640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:41:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f87416ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f87416ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f87416ac640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:41:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f87426ae640 1 --2- 172.21.15.60:0/3510877967 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8728005180 0x7f872800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f87426ae640 10 monclient(hunting): get_auth_request con 0x7f8728005180 auth_method 0 2024-03-27T21:41:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f87426ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f87426ae640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f87426ae640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:41:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8741ead640 10 monclient(hunting): handle_auth_done global_id 5130 payload 293 2024-03-27T21:41:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.329+0000 7f8741ead640 1 -- 172.21.15.60:0/3510877967 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8728005180 msgr2=0x7f872800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8741ead640 1 --2- 172.21.15.60:0/3510877967 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8728005180 0x7f872800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8741ead640 1 -- 172.21.15.60:0/3510877967 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87280047c0 msgr2=0x7f8728004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8741ead640 1 --2- 172.21.15.60:0/3510877967 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87280047c0 0x7f8728004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8741ead640 10 monclient: _finish_hunting 0 2024-03-27T21:41:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8741ead640 1 monclient: found mon.a 2024-03-27T21:41:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8741ead640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8741ead640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8741ead640 1 -- 172.21.15.60:0/3510877967 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8728003520 con 0x7f8728003970 2024-03-27T21:41:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8741ead640 10 monclient: _finish_auth 0 2024-03-27T21:41:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8741ead640 10 monclient: _check_auth_tickets 2024-03-27T21:41:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8741ead640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f87426ae640 1 --2- 172.21.15.60:0/3510877967 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8728005180 0x7f872800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:41:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f87426ae640 1 -- 172.21.15.60:0/3510877967 reap_dead start 2024-03-27T21:41:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8741ead640 1 --2- 172.21.15.60:0/3510877967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8728003970 0x7f8728003d90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f873800c240 tx=0x7f87280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b44c71e9400bf server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8740eab640 1 -- 172.21.15.60:0/3510877967 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8738005ad0 con 0x7f8728003970 2024-03-27T21:41:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8740eab640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8740eab640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8740eab640 10 monclient: dump: 2024-03-27T21:41:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:45.335 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8740eab640 1 -- 172.21.15.60:0/3510877967 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8738005c30 con 0x7f8728003970 2024-03-27T21:41:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8740eab640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8744111640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8744111640 4 set_mon_vals no callback set 2024-03-27T21:41:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8740eab640 1 -- 172.21.15.60:0/3510877967 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873802caf0 con 0x7f8728003970 2024-03-27T21:41:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8740eab640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8740eab640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.330+0000 7f8740eab640 10 monclient: dump: 2024-03-27T21:41:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:45.336 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:45.337 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.331+0000 7f8744111640 10 monclient: shutdown 2024-03-27T21:41:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.331+0000 7f8744111640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.331+0000 7f8744111640 1 -- 172.21.15.60:0/3510877967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8728003970 msgr2=0x7f8728003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.331+0000 7f8744111640 1 --2- 172.21.15.60:0/3510877967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8728003970 0x7f8728003d90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f873800c240 tx=0x7f87280057f0 comp rx=0 tx=0).stop 2024-03-27T21:41:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.331+0000 7f8744111640 1 -- 172.21.15.60:0/3510877967 shutdown_connections 2024-03-27T21:41:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.331+0000 7f8744111640 1 --2- 172.21.15.60:0/3510877967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8728003970 0x7f8728003d90 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.331+0000 7f8744111640 1 -- 172.21.15.60:0/3510877967 >> 172.21.15.60:0/3510877967 conn(0x7f873c0b7300 msgr2=0x7f873c0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.331+0000 7f8744111640 1 -- 172.21.15.60:0/3510877967 shutdown_connections 2024-03-27T21:41:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.331+0000 7f8744111640 1 -- 172.21.15.60:0/3510877967 wait complete. 2024-03-27T21:41:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.332+0000 7f8744111640 10 monclient: build_initial_monmap 2024-03-27T21:41:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.332+0000 7f8744111640 10 monclient: monmap: 2024-03-27T21:41:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:45.338 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.333+0000 7f8744111640 1 librados: starting msgr at 2024-03-27T21:41:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.333+0000 7f8744111640 1 librados: starting objecter 2024-03-27T21:41:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.333+0000 7f8744111640 1 Processor -- start 2024-03-27T21:41:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.333+0000 7f8744111640 1 -- start start 2024-03-27T21:41:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.333+0000 7f8744111640 1 librados: setting wanted keys 2024-03-27T21:41:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.333+0000 7f8744111640 1 librados: calling monclient init 2024-03-27T21:41:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.333+0000 7f8744111640 10 monclient: init 2024-03-27T21:41:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8744111640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8744111640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:41:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8744111640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8728003970 0x7f873c137f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8744111640 10 monclient(hunting): picked mon.noname-a con 0x7f8728003970 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8744111640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f873c0ac520 0x7f873c1d63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8744111640 10 monclient(hunting): picked mon.noname-c con 0x7f873c0ac520 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8744111640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f873c1d6920 0x7f873c135f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8744111640 10 monclient(hunting): picked mon.noname-b con 0x7f873c1d6920 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8744111640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8744111640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c0accf0 con 0x7f8728003970 2024-03-27T21:41:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8744111640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8744111640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c0ac9a0 con 0x7f873c1d6920 2024-03-27T21:41:45.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8744111640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:45.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8744111640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873c1d6d80 con 0x7f873c0ac520 2024-03-27T21:41:45.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8744111640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:45.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8744111640 10 monclient(hunting): authenticate will time out at 946.555182s 2024-03-27T21:41:45.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8741ead640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8728003970 0x7f873c137f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:45.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87416ac640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f873c0ac520 0x7f873c1d63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:45.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8741ead640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8728003970 0x7f873c137f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41780/0 (socket says 172.21.15.60:41780) 2024-03-27T21:41:45.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8741ead640 1 -- 172.21.15.60:0/4051066867 learned_addr learned my addr 172.21.15.60:0/4051066867 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:45.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8741ead640 10 monclient(hunting): get_auth_request con 0x7f8728003970 auth_method 0 2024-03-27T21:41:45.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8741ead640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:45.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8741ead640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:45.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8741ead640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:45.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87416ac640 10 monclient(hunting): get_auth_request con 0x7f873c0ac520 auth_method 0 2024-03-27T21:41:45.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87416ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:45.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87416ac640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:45.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87416ac640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:45.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87416ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:45.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87416ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:45.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87416ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:45.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8741ead640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:45.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87426ae640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f873c1d6920 0x7f873c135f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:45.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8741ead640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:45.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f8741ead640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:45.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87426ae640 10 monclient(hunting): get_auth_request con 0x7f873c1d6920 auth_method 0 2024-03-27T21:41:45.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87426ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:45.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87426ae640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:45.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87426ae640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:45.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87426ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:45.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87426ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:45.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87426ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:45.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87416ac640 10 monclient(hunting): handle_auth_done global_id 4889 payload 824 2024-03-27T21:41:45.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87416ac640 1 -- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f873c1d6920 msgr2=0x7f873c135f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:45.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87416ac640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f873c1d6920 0x7f873c135f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:45.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87416ac640 1 -- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8728003970 msgr2=0x7f873c137f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:45.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87416ac640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8728003970 0x7f873c137f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:45.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87416ac640 10 monclient: _finish_hunting 0 2024-03-27T21:41:45.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87416ac640 1 monclient: found mon.noname-c 2024-03-27T21:41:45.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87416ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:45.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87416ac640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:45.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87416ac640 1 -- 172.21.15.60:0/4051066867 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c138820 con 0x7f873c0ac520 2024-03-27T21:41:45.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87416ac640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f873c0ac520 0x7f873c1d63e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f872c0118e0 tx=0x7f872c011da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:45.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87327fc640 10 client.?.objecter ms_handle_connect 0x7f873c0ac520 2024-03-27T21:41:45.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.334+0000 7f87327fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:45.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c004300 con 0x7f873c0ac520 2024-03-27T21:41:45.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f87327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:45.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f87327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:45.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f87327fc640 10 monclient: dump: 2024-03-27T21:41:45.343 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:45.343 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:45.343 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:45.343 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:45.343 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:45.343 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:45.343 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:45.343 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:45.344 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:45.344 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:45.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f87327fc640 10 monclient: _finish_auth 0 2024-03-27T21:41:45.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f87327fc640 10 monclient: _check_auth_tickets 2024-03-27T21:41:45.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f87327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:45.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872c004460 con 0x7f873c0ac520 2024-03-27T21:41:45.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f8744111640 5 monclient: authenticate success, global_id 4889 2024-03-27T21:41:45.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f8744111640 10 monclient: _renew_subs 2024-03-27T21:41:45.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f8744111640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:45.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f873c0ad110 con 0x7f873c0ac520 2024-03-27T21:41:45.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f87327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:45.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c00fa20 con 0x7f873c0ac520 2024-03-27T21:41:45.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f87327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:45.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f87327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:45.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f87327fc640 10 monclient: dump: 2024-03-27T21:41:45.345 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:45.345 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:45.345 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:45.345 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:45.345 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:45.345 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:45.345 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:45.345 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:45.345 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:45.345 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:45.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f87337fe640 4 set_mon_vals no callback set 2024-03-27T21:41:45.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f8744111640 10 client.4889.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:45.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f8744111640 10 monclient: _renew_subs 2024-03-27T21:41:45.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f8744111640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:45.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f873c0ac340 con 0x7f873c0ac520 2024-03-27T21:41:45.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.335+0000 7f8744111640 1 librados: init done 2024-03-27T21:41:45.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.336+0000 7f8744111640 10 start_mon_command cmd=[{"prefix": "osd ls"}] 2024-03-27T21:41:45.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.336+0000 7f8744111640 10 monclient: _send_command 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:45.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.336+0000 7f8744111640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:45.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.336+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f873c069470 con 0x7f873c0ac520 2024-03-27T21:41:45.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.336+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f872c00fb80 con 0x7f873c0ac520 2024-03-27T21:41:45.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.336+0000 7f87327fc640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f872006c5c0 0x7f872006ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:45.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f872c089760 con 0x7f873c0ac520 2024-03-27T21:41:45.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f87327fc640 10 client.4889.objecter ms_dispatch 0x7f873c000b90 osd_map(73..73 src has 1..73) v4 2024-03-27T21:41:45.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f87327fc640 3 client.4889.objecter handle_osd_map got epochs [73,73] > 0 2024-03-27T21:41:45.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f87327fc640 3 client.4889.objecter handle_osd_map decoding full epoch 73 2024-03-27T21:41:45.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f8741ead640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f872006c5c0 0x7f872006ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:45.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f8741ead640 10 monclient: get_auth_request con 0x7f872006c5c0 auth_method 0 2024-03-27T21:41:45.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f87327fc640 20 client.4889.objecter dump_active .. 0 homeless 2024-03-27T21:41:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v73) v1 ==== 54+0+16 (secure 0 0 0) 0x7f872c090070 con 0x7f873c0ac520 2024-03-27T21:41:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f87327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd ls"}] 2024-03-27T21:41:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f87327fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f8744111640 10 client.4889.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign -1 1 2024-03-27T21:41:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f8744111640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f873c1d9960 0x7f873c1d9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c0aa5f0 osd=0 3 2024-03-27T21:41:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c0aa5f0 osd=0 3 2024-03-27T21:41:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c0aa5f0 osd=0 3 2024-03-27T21:41:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f8744111640 15 client.4889.objecter _session_command_op_remove -1 1 2024-03-27T21:41:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f8744111640 20 client.4889.objecter get_session s=0x7f873c0aa5f0 osd=0 3 2024-03-27T21:41:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign 0 1 2024-03-27T21:41:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c0aa5f0 osd=0 4 2024-03-27T21:41:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f8744111640 10 client.4889.objecter _send_command 1 2024-03-27T21:41:45.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f873c0ad2a0 con 0x7f873c1d9960 2024-03-27T21:41:45.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f87426ae640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f873c1d9960 0x7f873c1d9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:45.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f87426ae640 10 monclient: get_auth_request con 0x7f873c1d9960 auth_method 0 2024-03-27T21:41:45.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f87327fc640 10 client.4889.objecter ms_handle_connect 0x7f872006c5c0 2024-03-27T21:41:45.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f8741ead640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f872006c5c0 0x7f872006ea80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8738001d10 tx=0x7f8738009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:45.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f87327fc640 10 client.4889.objecter ms_handle_connect 0x7f873c1d9960 2024-03-27T21:41:45.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.337+0000 7f87426ae640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f873c1d9960 0x7f873c1d9d40 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:45.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.338+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f8734001990 con 0x7f873c1d9960 2024-03-27T21:41:45.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.338+0000 7f87327fc640 10 client.4889.objecter ms_dispatch 0x7f873c000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:41:45.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.338+0000 7f87327fc640 10 client.4889.objecter _finish_command 1 = system:0 2024-03-27T21:41:45.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.338+0000 7f87327fc640 20 client.4889.objecter put_session s=0x7f873c0aa5f0 osd=0 4 2024-03-27T21:41:45.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.338+0000 7f87327fc640 15 client.4889.objecter _session_command_op_remove 0 1 2024-03-27T21:41:45.349 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:41:45.349 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-03-27T21:41:45.349 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-03-27T21:41:45.350 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:41:45.350 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-27T21:41:45.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.353+0000 7f8744111640 10 client.4889.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:45.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.353+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign -1 2 2024-03-27T21:41:45.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.353+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c0aa5f0 osd=0 3 2024-03-27T21:41:45.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.353+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c0aa5f0 osd=0 3 2024-03-27T21:41:45.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.353+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c0aa5f0 osd=0 3 2024-03-27T21:41:45.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.353+0000 7f8744111640 15 client.4889.objecter _session_command_op_remove -1 2 2024-03-27T21:41:45.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.353+0000 7f8744111640 20 client.4889.objecter get_session s=0x7f873c0aa5f0 osd=0 3 2024-03-27T21:41:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.353+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign 0 2 2024-03-27T21:41:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.353+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c0aa5f0 osd=0 4 2024-03-27T21:41:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.353+0000 7f8744111640 10 client.4889.objecter _send_command 2 2024-03-27T21:41:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.353+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f873c0a9260 con 0x7f873c1d9960 2024-03-27T21:41:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.355+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f873c0a9260 con 0x7f873c1d9960 2024-03-27T21:41:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.355+0000 7f87327fc640 10 client.4889.objecter ms_dispatch 0x7f873c000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.355+0000 7f87327fc640 10 client.4889.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.355+0000 7f87327fc640 20 client.4889.objecter put_session s=0x7f873c0aa5f0 osd=0 4 2024-03-27T21:41:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.355+0000 7f87327fc640 15 client.4889.objecter _session_command_op_remove 0 2 2024-03-27T21:41:45.352 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0: {} 2024-03-27T21:41:45.353 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-27T21:41:45.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.355+0000 7f8744111640 10 client.4889.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:45.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.355+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign -1 3 2024-03-27T21:41:45.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.355+0000 7f8744111640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f873c065a90 0x7f873c065e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:45.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.355+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c0a89f0 osd=1 3 2024-03-27T21:41:45.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.355+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c0a89f0 osd=1 3 2024-03-27T21:41:45.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.355+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c0a89f0 osd=1 3 2024-03-27T21:41:45.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.355+0000 7f8744111640 15 client.4889.objecter _session_command_op_remove -1 3 2024-03-27T21:41:45.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.355+0000 7f8744111640 20 client.4889.objecter get_session s=0x7f873c0a89f0 osd=1 3 2024-03-27T21:41:45.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.355+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign 1 3 2024-03-27T21:41:45.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.355+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c0a89f0 osd=1 4 2024-03-27T21:41:45.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.355+0000 7f8744111640 10 client.4889.objecter _send_command 3 2024-03-27T21:41:45.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.355+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f873c0a8b60 con 0x7f873c065a90 2024-03-27T21:41:45.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.355+0000 7f8741ead640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f873c065a90 0x7f873c065e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:45.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.355+0000 7f8741ead640 10 monclient: get_auth_request con 0x7f873c065a90 auth_method 0 2024-03-27T21:41:45.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.355+0000 7f87327fc640 10 client.4889.objecter ms_handle_connect 0x7f873c065a90 2024-03-27T21:41:45.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.355+0000 7f8741ead640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f873c065a90 0x7f873c065e70 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:45.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.356+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f873c0a8b60 con 0x7f873c065a90 2024-03-27T21:41:45.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.356+0000 7f87327fc640 10 client.4889.objecter ms_dispatch 0x7f873c000b90 command_reply(tid 3: 0 ) v1 2024-03-27T21:41:45.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.356+0000 7f87327fc640 10 client.4889.objecter _finish_command 3 = system:0 2024-03-27T21:41:45.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.356+0000 7f87327fc640 20 client.4889.objecter put_session s=0x7f873c0a89f0 osd=1 4 2024-03-27T21:41:45.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.356+0000 7f87327fc640 15 client.4889.objecter _session_command_op_remove 1 3 2024-03-27T21:41:45.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.371+0000 7f8744111640 10 client.4889.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:45.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.371+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign -1 4 2024-03-27T21:41:45.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.371+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c0a89f0 osd=1 3 2024-03-27T21:41:45.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.371+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c0a89f0 osd=1 3 2024-03-27T21:41:45.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.371+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c0a89f0 osd=1 3 2024-03-27T21:41:45.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.371+0000 7f8744111640 15 client.4889.objecter _session_command_op_remove -1 4 2024-03-27T21:41:45.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.371+0000 7f8744111640 20 client.4889.objecter get_session s=0x7f873c0a89f0 osd=1 3 2024-03-27T21:41:45.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.371+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign 1 4 2024-03-27T21:41:45.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.371+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c0a89f0 osd=1 4 2024-03-27T21:41:45.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.371+0000 7f8744111640 10 client.4889.objecter _send_command 4 2024-03-27T21:41:45.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.371+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f873c1da790 con 0x7f873c065a90 2024-03-27T21:41:45.359 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1: {} 2024-03-27T21:41:45.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.372+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f873c1da790 con 0x7f873c065a90 2024-03-27T21:41:45.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.372+0000 7f87327fc640 10 client.4889.objecter ms_dispatch 0x7f873c000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:45.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.372+0000 7f87327fc640 10 client.4889.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:45.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.372+0000 7f87327fc640 20 client.4889.objecter put_session s=0x7f873c0a89f0 osd=1 4 2024-03-27T21:41:45.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.372+0000 7f87327fc640 15 client.4889.objecter _session_command_op_remove 1 4 2024-03-27T21:41:45.360 INFO:teuthology.orchestra.run.smithi060.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-27T21:41:45.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.372+0000 7f8744111640 10 client.4889.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:45.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.372+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign -1 5 2024-03-27T21:41:45.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.372+0000 7f8744111640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f873c02b680 0x7f873c02ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:45.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.372+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c13db40 osd=2 3 2024-03-27T21:41:45.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.372+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c13db40 osd=2 3 2024-03-27T21:41:45.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.372+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c13db40 osd=2 3 2024-03-27T21:41:45.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.372+0000 7f8744111640 15 client.4889.objecter _session_command_op_remove -1 5 2024-03-27T21:41:45.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.372+0000 7f8744111640 20 client.4889.objecter get_session s=0x7f873c13db40 osd=2 3 2024-03-27T21:41:45.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.372+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign 2 5 2024-03-27T21:41:45.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.372+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c13db40 osd=2 4 2024-03-27T21:41:45.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.372+0000 7f8744111640 10 client.4889.objecter _send_command 5 2024-03-27T21:41:45.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.372+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 --> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f873c13dcb0 con 0x7f873c02b680 2024-03-27T21:41:45.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.373+0000 7f87416ac640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f873c02b680 0x7f873c02ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:45.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.373+0000 7f87416ac640 10 monclient: get_auth_request con 0x7f873c02b680 auth_method 0 2024-03-27T21:41:45.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.373+0000 7f87327fc640 10 client.4889.objecter ms_handle_connect 0x7f873c02b680 2024-03-27T21:41:45.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.373+0000 7f87416ac640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f873c02b680 0x7f873c02ba60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:45.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.373+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== osd.2 v2:172.21.15.60:6824/56434 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f873c13dcb0 con 0x7f873c02b680 2024-03-27T21:41:45.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.373+0000 7f87327fc640 10 client.4889.objecter ms_dispatch 0x7f873c000b90 command_reply(tid 5: 0 ) v1 2024-03-27T21:41:45.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.373+0000 7f87327fc640 10 client.4889.objecter _finish_command 5 = system:0 2024-03-27T21:41:45.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.373+0000 7f87327fc640 20 client.4889.objecter put_session s=0x7f873c13db40 osd=2 4 2024-03-27T21:41:45.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.373+0000 7f87327fc640 15 client.4889.objecter _session_command_op_remove 2 5 2024-03-27T21:41:45.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.388+0000 7f8744111640 10 client.4889.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:45.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.388+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign -1 6 2024-03-27T21:41:45.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.388+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c13db40 osd=2 3 2024-03-27T21:41:45.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.388+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c13db40 osd=2 3 2024-03-27T21:41:45.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.388+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c13db40 osd=2 3 2024-03-27T21:41:45.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.388+0000 7f8744111640 15 client.4889.objecter _session_command_op_remove -1 6 2024-03-27T21:41:45.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.388+0000 7f8744111640 20 client.4889.objecter get_session s=0x7f873c13db40 osd=2 3 2024-03-27T21:41:45.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.388+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign 2 6 2024-03-27T21:41:45.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.388+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c13db40 osd=2 4 2024-03-27T21:41:45.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.388+0000 7f8744111640 10 client.4889.objecter _send_command 6 2024-03-27T21:41:45.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.388+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 --> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f873c0665f0 con 0x7f873c02b680 2024-03-27T21:41:45.375 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-27T21:41:45.388+0000 7f33e3743640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 34596) UID: 0 2024-03-27T21:41:45.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.389+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== osd.2 v2:172.21.15.60:6824/56434 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f873c0665f0 con 0x7f873c02b680 2024-03-27T21:41:45.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.389+0000 7f87327fc640 10 client.4889.objecter ms_dispatch 0x7f873c000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:45.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.389+0000 7f87327fc640 10 client.4889.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:45.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.389+0000 7f87327fc640 20 client.4889.objecter put_session s=0x7f873c13db40 osd=2 4 2024-03-27T21:41:45.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.389+0000 7f87327fc640 15 client.4889.objecter _session_command_op_remove 2 6 2024-03-27T21:41:45.376 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2: {} 2024-03-27T21:41:45.377 INFO:teuthology.orchestra.run.smithi060.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-27T21:41:45.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.390+0000 7f8744111640 10 client.4889.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:45.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.390+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign -1 7 2024-03-27T21:41:45.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.390+0000 7f8744111640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f873c1e7230 0x7f873c1e9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:45.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.390+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c0669a0 osd=3 3 2024-03-27T21:41:45.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.390+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c0669a0 osd=3 3 2024-03-27T21:41:45.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.390+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c0669a0 osd=3 3 2024-03-27T21:41:45.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.390+0000 7f8744111640 15 client.4889.objecter _session_command_op_remove -1 7 2024-03-27T21:41:45.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.390+0000 7f8744111640 20 client.4889.objecter get_session s=0x7f873c0669a0 osd=3 3 2024-03-27T21:41:45.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.390+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign 3 7 2024-03-27T21:41:45.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.390+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c0669a0 osd=3 4 2024-03-27T21:41:45.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.390+0000 7f8744111640 10 client.4889.objecter _send_command 7 2024-03-27T21:41:45.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.390+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f873c1ec3f0 con 0x7f873c1e7230 2024-03-27T21:41:45.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.390+0000 7f87426ae640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f873c1e7230 0x7f873c1e9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:45.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.390+0000 7f87426ae640 10 monclient: get_auth_request con 0x7f873c1e7230 auth_method 0 2024-03-27T21:41:45.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.390+0000 7f87327fc640 10 client.4889.objecter ms_handle_connect 0x7f873c1e7230 2024-03-27T21:41:45.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.390+0000 7f87426ae640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f873c1e7230 0x7f873c1e9620 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:45.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.391+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f873c1ec3f0 con 0x7f873c1e7230 2024-03-27T21:41:45.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.391+0000 7f87327fc640 10 client.4889.objecter ms_dispatch 0x7f873c000b90 command_reply(tid 7: 0 ) v1 2024-03-27T21:41:45.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.391+0000 7f87327fc640 10 client.4889.objecter _finish_command 7 = system:0 2024-03-27T21:41:45.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.391+0000 7f87327fc640 20 client.4889.objecter put_session s=0x7f873c0669a0 osd=3 4 2024-03-27T21:41:45.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.391+0000 7f87327fc640 15 client.4889.objecter _session_command_op_remove 3 7 2024-03-27T21:41:45.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.405+0000 7f8744111640 10 client.4889.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:45.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.405+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign -1 8 2024-03-27T21:41:45.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.405+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c0669a0 osd=3 3 2024-03-27T21:41:45.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.405+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c0669a0 osd=3 3 2024-03-27T21:41:45.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.405+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c0669a0 osd=3 3 2024-03-27T21:41:45.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.405+0000 7f8744111640 15 client.4889.objecter _session_command_op_remove -1 8 2024-03-27T21:41:45.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.405+0000 7f8744111640 20 client.4889.objecter get_session s=0x7f873c0669a0 osd=3 3 2024-03-27T21:41:45.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.405+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign 3 8 2024-03-27T21:41:45.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.405+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c0669a0 osd=3 4 2024-03-27T21:41:45.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.405+0000 7f8744111640 10 client.4889.objecter _send_command 8 2024-03-27T21:41:45.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.405+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f873c02c1e0 con 0x7f873c1e7230 2024-03-27T21:41:45.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.406+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f873c02c1e0 con 0x7f873c1e7230 2024-03-27T21:41:45.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.406+0000 7f87327fc640 10 client.4889.objecter ms_dispatch 0x7f873c000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:45.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.406+0000 7f87327fc640 10 client.4889.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:45.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.406+0000 7f87327fc640 20 client.4889.objecter put_session s=0x7f873c0669a0 osd=3 4 2024-03-27T21:41:45.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.406+0000 7f87327fc640 15 client.4889.objecter _session_command_op_remove 3 8 2024-03-27T21:41:45.393 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3: {} 2024-03-27T21:41:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-27T21:41:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.407+0000 7f8744111640 10 client.4889.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.407+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign -1 9 2024-03-27T21:41:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.407+0000 7f8744111640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f873c0b5290 0x7f873c0b56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.407+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c02c1e0 osd=4 3 2024-03-27T21:41:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.407+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c02c1e0 osd=4 3 2024-03-27T21:41:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.407+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c02c1e0 osd=4 3 2024-03-27T21:41:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.407+0000 7f8744111640 15 client.4889.objecter _session_command_op_remove -1 9 2024-03-27T21:41:45.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.407+0000 7f8744111640 20 client.4889.objecter get_session s=0x7f873c02c1e0 osd=4 3 2024-03-27T21:41:45.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.407+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign 4 9 2024-03-27T21:41:45.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.407+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c02c1e0 osd=4 4 2024-03-27T21:41:45.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.407+0000 7f8744111640 10 client.4889.objecter _send_command 9 2024-03-27T21:41:45.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.407+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f873c1ef6f0 con 0x7f873c0b5290 2024-03-27T21:41:45.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.407+0000 7f8741ead640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f873c0b5290 0x7f873c0b56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:45.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.407+0000 7f8741ead640 10 monclient: get_auth_request con 0x7f873c0b5290 auth_method 0 2024-03-27T21:41:45.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.407+0000 7f87327fc640 10 client.4889.objecter ms_handle_connect 0x7f873c0b5290 2024-03-27T21:41:45.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.407+0000 7f8741ead640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f873c0b5290 0x7f873c0b56b0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:45.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.408+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f873c1ef6f0 con 0x7f873c0b5290 2024-03-27T21:41:45.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.408+0000 7f87327fc640 10 client.4889.objecter ms_dispatch 0x7f873c000b90 command_reply(tid 9: 0 ) v1 2024-03-27T21:41:45.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.408+0000 7f87327fc640 10 client.4889.objecter _finish_command 9 = system:0 2024-03-27T21:41:45.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.408+0000 7f87327fc640 20 client.4889.objecter put_session s=0x7f873c02c1e0 osd=4 4 2024-03-27T21:41:45.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.408+0000 7f87327fc640 15 client.4889.objecter _session_command_op_remove 4 9 2024-03-27T21:41:45.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.422+0000 7f8744111640 10 client.4889.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:45.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.422+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign -1 10 2024-03-27T21:41:45.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.422+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c02c1e0 osd=4 3 2024-03-27T21:41:45.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.422+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c02c1e0 osd=4 3 2024-03-27T21:41:45.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.422+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c02c1e0 osd=4 3 2024-03-27T21:41:45.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.422+0000 7f8744111640 15 client.4889.objecter _session_command_op_remove -1 10 2024-03-27T21:41:45.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.422+0000 7f8744111640 20 client.4889.objecter get_session s=0x7f873c02c1e0 osd=4 3 2024-03-27T21:41:45.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.422+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign 4 10 2024-03-27T21:41:45.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.422+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c02c1e0 osd=4 4 2024-03-27T21:41:45.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.422+0000 7f8744111640 10 client.4889.objecter _send_command 10 2024-03-27T21:41:45.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.422+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f873c1eace0 con 0x7f873c0b5290 2024-03-27T21:41:45.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.424+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f873c1eace0 con 0x7f873c0b5290 2024-03-27T21:41:45.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.424+0000 7f87327fc640 10 client.4889.objecter ms_dispatch 0x7f873c000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:45.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.424+0000 7f87327fc640 10 client.4889.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:45.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.424+0000 7f87327fc640 20 client.4889.objecter put_session s=0x7f873c02c1e0 osd=4 4 2024-03-27T21:41:45.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.424+0000 7f87327fc640 15 client.4889.objecter _session_command_op_remove 4 10 2024-03-27T21:41:45.411 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4: {} 2024-03-27T21:41:45.412 INFO:teuthology.orchestra.run.smithi060.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-27T21:41:45.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.425+0000 7f8744111640 10 client.4889.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:45.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.425+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign -1 11 2024-03-27T21:41:45.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.425+0000 7f8744111640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f873c1eb360 0x7f873c1eb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:45.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.425+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c1eace0 osd=5 3 2024-03-27T21:41:45.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.425+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c1eace0 osd=5 3 2024-03-27T21:41:45.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.425+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c1eace0 osd=5 3 2024-03-27T21:41:45.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.425+0000 7f8744111640 15 client.4889.objecter _session_command_op_remove -1 11 2024-03-27T21:41:45.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.425+0000 7f8744111640 20 client.4889.objecter get_session s=0x7f873c1eace0 osd=5 3 2024-03-27T21:41:45.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.425+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign 5 11 2024-03-27T21:41:45.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.425+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c1eace0 osd=5 4 2024-03-27T21:41:45.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.425+0000 7f8744111640 10 client.4889.objecter _send_command 11 2024-03-27T21:41:45.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.425+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f873c1ebd00 con 0x7f873c1eb360 2024-03-27T21:41:45.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.425+0000 7f87416ac640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f873c1eb360 0x7f873c1eb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:45.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.425+0000 7f87416ac640 10 monclient: get_auth_request con 0x7f873c1eb360 auth_method 0 2024-03-27T21:41:45.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.425+0000 7f87416ac640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f873c1eb360 0x7f873c1eb7c0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:45.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.425+0000 7f87327fc640 10 client.4889.objecter ms_handle_connect 0x7f873c1eb360 2024-03-27T21:41:45.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.426+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f873c1ebd00 con 0x7f873c1eb360 2024-03-27T21:41:45.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.426+0000 7f87327fc640 10 client.4889.objecter ms_dispatch 0x7f873c000b90 command_reply(tid 11: 0 ) v1 2024-03-27T21:41:45.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.426+0000 7f87327fc640 10 client.4889.objecter _finish_command 11 = system:0 2024-03-27T21:41:45.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.426+0000 7f87327fc640 20 client.4889.objecter put_session s=0x7f873c1eace0 osd=5 4 2024-03-27T21:41:45.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.426+0000 7f87327fc640 15 client.4889.objecter _session_command_op_remove 5 11 2024-03-27T21:41:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.440+0000 7f8744111640 10 client.4889.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.440+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign -1 12 2024-03-27T21:41:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.440+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c1eace0 osd=5 3 2024-03-27T21:41:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.440+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c1eace0 osd=5 3 2024-03-27T21:41:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.440+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c1eace0 osd=5 3 2024-03-27T21:41:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.440+0000 7f8744111640 15 client.4889.objecter _session_command_op_remove -1 12 2024-03-27T21:41:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.440+0000 7f8744111640 20 client.4889.objecter get_session s=0x7f873c1eace0 osd=5 3 2024-03-27T21:41:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.440+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign 5 12 2024-03-27T21:41:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.440+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c1eace0 osd=5 4 2024-03-27T21:41:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.440+0000 7f8744111640 10 client.4889.objecter _send_command 12 2024-03-27T21:41:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.440+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f873c1ebd00 con 0x7f873c1eb360 2024-03-27T21:41:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.442+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f873c1ebd00 con 0x7f873c1eb360 2024-03-27T21:41:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.442+0000 7f87327fc640 10 client.4889.objecter ms_dispatch 0x7f873c000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.442+0000 7f87327fc640 10 client.4889.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.442+0000 7f87327fc640 20 client.4889.objecter put_session s=0x7f873c1eace0 osd=5 4 2024-03-27T21:41:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.442+0000 7f87327fc640 15 client.4889.objecter _session_command_op_remove 5 12 2024-03-27T21:41:45.429 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5: {} 2024-03-27T21:41:45.431 INFO:teuthology.orchestra.run.smithi060.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-27T21:41:45.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.442+0000 7f8744111640 10 client.4889.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:45.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.442+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign -1 13 2024-03-27T21:41:45.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.442+0000 7f8744111640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f873c1f0580 0x7f873c1f29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:45.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.442+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c1ebd00 osd=6 3 2024-03-27T21:41:45.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.442+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c1ebd00 osd=6 3 2024-03-27T21:41:45.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.442+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c1ebd00 osd=6 3 2024-03-27T21:41:45.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.442+0000 7f8744111640 15 client.4889.objecter _session_command_op_remove -1 13 2024-03-27T21:41:45.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.442+0000 7f8744111640 20 client.4889.objecter get_session s=0x7f873c1ebd00 osd=6 3 2024-03-27T21:41:45.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.442+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign 6 13 2024-03-27T21:41:45.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.442+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c1ebd00 osd=6 4 2024-03-27T21:41:45.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.442+0000 7f8744111640 10 client.4889.objecter _send_command 13 2024-03-27T21:41:45.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.442+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f873c1f5970 con 0x7f873c1f0580 2024-03-27T21:41:45.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.442+0000 7f87426ae640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f873c1f0580 0x7f873c1f29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:45.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.442+0000 7f87426ae640 10 monclient: get_auth_request con 0x7f873c1f0580 auth_method 0 2024-03-27T21:41:45.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.443+0000 7f87426ae640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f873c1f0580 0x7f873c1f29b0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:45.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.443+0000 7f87327fc640 10 client.4889.objecter ms_handle_connect 0x7f873c1f0580 2024-03-27T21:41:45.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.443+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f873c1f5970 con 0x7f873c1f0580 2024-03-27T21:41:45.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.443+0000 7f87327fc640 10 client.4889.objecter ms_dispatch 0x7f873c000b90 command_reply(tid 13: 0 ) v1 2024-03-27T21:41:45.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.443+0000 7f87327fc640 10 client.4889.objecter _finish_command 13 = system:0 2024-03-27T21:41:45.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.443+0000 7f87327fc640 20 client.4889.objecter put_session s=0x7f873c1ebd00 osd=6 4 2024-03-27T21:41:45.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.444+0000 7f87327fc640 15 client.4889.objecter _session_command_op_remove 6 13 2024-03-27T21:41:45.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.457+0000 7f8744111640 10 client.4889.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:45.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.457+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign -1 14 2024-03-27T21:41:45.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.457+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c1ebd00 osd=6 3 2024-03-27T21:41:45.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.457+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c1ebd00 osd=6 3 2024-03-27T21:41:45.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.457+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c1ebd00 osd=6 3 2024-03-27T21:41:45.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.457+0000 7f8744111640 15 client.4889.objecter _session_command_op_remove -1 14 2024-03-27T21:41:45.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.457+0000 7f8744111640 20 client.4889.objecter get_session s=0x7f873c1ebd00 osd=6 3 2024-03-27T21:41:45.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.457+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign 6 14 2024-03-27T21:41:45.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.457+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c1ebd00 osd=6 4 2024-03-27T21:41:45.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.457+0000 7f8744111640 10 client.4889.objecter _send_command 14 2024-03-27T21:41:45.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.457+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f873c1f4740 con 0x7f873c1f0580 2024-03-27T21:41:45.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.459+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f873c1f4740 con 0x7f873c1f0580 2024-03-27T21:41:45.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.459+0000 7f87327fc640 10 client.4889.objecter ms_dispatch 0x7f873c000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:45.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.459+0000 7f87327fc640 10 client.4889.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:45.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.459+0000 7f87327fc640 20 client.4889.objecter put_session s=0x7f873c1ebd00 osd=6 4 2024-03-27T21:41:45.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.459+0000 7f87327fc640 15 client.4889.objecter _session_command_op_remove 6 14 2024-03-27T21:41:45.446 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6: {} 2024-03-27T21:41:45.447 INFO:teuthology.orchestra.run.smithi060.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-27T21:41:45.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.460+0000 7f8744111640 10 client.4889.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:45.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.460+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign -1 15 2024-03-27T21:41:45.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.460+0000 7f8744111640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f873c1f4e60 0x7f873c1f52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:45.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.460+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c1f4740 osd=7 3 2024-03-27T21:41:45.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.460+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c1f4740 osd=7 3 2024-03-27T21:41:45.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.460+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c1f4740 osd=7 3 2024-03-27T21:41:45.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.460+0000 7f8744111640 15 client.4889.objecter _session_command_op_remove -1 15 2024-03-27T21:41:45.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.460+0000 7f8744111640 20 client.4889.objecter get_session s=0x7f873c1f4740 osd=7 3 2024-03-27T21:41:45.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.460+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign 7 15 2024-03-27T21:41:45.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.460+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c1f4740 osd=7 4 2024-03-27T21:41:45.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.460+0000 7f8744111640 10 client.4889.objecter _send_command 15 2024-03-27T21:41:45.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.460+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f873c1f4290 con 0x7f873c1f4e60 2024-03-27T21:41:45.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.460+0000 7f8741ead640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f873c1f4e60 0x7f873c1f52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:45.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.460+0000 7f8741ead640 10 monclient: get_auth_request con 0x7f873c1f4e60 auth_method 0 2024-03-27T21:41:45.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.461+0000 7f8741ead640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f873c1f4e60 0x7f873c1f52c0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:45.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.461+0000 7f87327fc640 10 client.4889.objecter ms_handle_connect 0x7f873c1f4e60 2024-03-27T21:41:45.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.461+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f873c1f4290 con 0x7f873c1f4e60 2024-03-27T21:41:45.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.461+0000 7f87327fc640 10 client.4889.objecter ms_dispatch 0x7f873c000b90 command_reply(tid 15: 0 ) v1 2024-03-27T21:41:45.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.461+0000 7f87327fc640 10 client.4889.objecter _finish_command 15 = system:0 2024-03-27T21:41:45.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.461+0000 7f87327fc640 20 client.4889.objecter put_session s=0x7f873c1f4740 osd=7 4 2024-03-27T21:41:45.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.461+0000 7f87327fc640 15 client.4889.objecter _session_command_op_remove 7 15 2024-03-27T21:41:45.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.475+0000 7f8744111640 10 client.4889.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-27T21:41:45.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.475+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign -1 16 2024-03-27T21:41:45.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.475+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c1f4740 osd=7 3 2024-03-27T21:41:45.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.475+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c1f4740 osd=7 3 2024-03-27T21:41:45.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.475+0000 7f8744111640 20 client.4889.objecter _get_session s=0x7f873c1f4740 osd=7 3 2024-03-27T21:41:45.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.475+0000 7f8744111640 15 client.4889.objecter _session_command_op_remove -1 16 2024-03-27T21:41:45.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.475+0000 7f8744111640 20 client.4889.objecter get_session s=0x7f873c1f4740 osd=7 3 2024-03-27T21:41:45.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.475+0000 7f8744111640 15 client.4889.objecter _session_command_op_assign 7 16 2024-03-27T21:41:45.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.475+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c1f4740 osd=7 4 2024-03-27T21:41:45.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.475+0000 7f8744111640 10 client.4889.objecter _send_command 16 2024-03-27T21:41:45.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.475+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f873c1f4290 con 0x7f873c1f4e60 2024-03-27T21:41:45.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f87327fc640 1 -- 172.21.15.60:0/4051066867 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (crc 0 0 0) 0x7f873c1f4290 con 0x7f873c1f4e60 2024-03-27T21:41:45.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f87327fc640 10 client.4889.objecter ms_dispatch 0x7f873c000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-03-27T21:41:45.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f87327fc640 10 client.4889.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-27T21:41:45.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f87327fc640 20 client.4889.objecter put_session s=0x7f873c1f4740 osd=7 4 2024-03-27T21:41:45.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f87327fc640 15 client.4889.objecter _session_command_op_remove 7 16 2024-03-27T21:41:45.463 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7: {} 2024-03-27T21:41:45.465 INFO:teuthology.orchestra.run.smithi060.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-27T21:41:45.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 10 librados: watch_flush enter 2024-03-27T21:41:45.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 10 librados: watch_flush exit 2024-03-27T21:41:45.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 10 client.4889.objecter close_session for osd.0 2024-03-27T21:41:45.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f873c1d9960 msgr2=0x7f873c1d9d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:45.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f873c1d9960 0x7f873c1d9d40 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:45.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c0aa5f0 osd=0 1 2024-03-27T21:41:45.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 10 client.4889.objecter close_session for osd.1 2024-03-27T21:41:45.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f873c065a90 msgr2=0x7f873c065e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:45.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f873c065a90 0x7f873c065e70 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:45.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c0a89f0 osd=1 1 2024-03-27T21:41:45.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 10 client.4889.objecter close_session for osd.2 2024-03-27T21:41:45.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f873c02b680 msgr2=0x7f873c02ba60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:45.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f873c02b680 0x7f873c02ba60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:45.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c13db40 osd=2 1 2024-03-27T21:41:45.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 10 client.4889.objecter close_session for osd.3 2024-03-27T21:41:45.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f873c1e7230 msgr2=0x7f873c1e9620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:45.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f873c1e7230 0x7f873c1e9620 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:45.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c0669a0 osd=3 1 2024-03-27T21:41:45.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 10 client.4889.objecter close_session for osd.4 2024-03-27T21:41:45.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f873c0b5290 msgr2=0x7f873c0b56b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:45.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f873c0b5290 0x7f873c0b56b0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:45.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c02c1e0 osd=4 1 2024-03-27T21:41:45.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 10 client.4889.objecter close_session for osd.5 2024-03-27T21:41:45.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f873c1eb360 msgr2=0x7f873c1eb7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:45.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f873c1eb360 0x7f873c1eb7c0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:45.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c1eace0 osd=5 1 2024-03-27T21:41:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 10 client.4889.objecter close_session for osd.6 2024-03-27T21:41:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f873c1f0580 msgr2=0x7f873c1f29b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f873c1f0580 0x7f873c1f29b0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c1ebd00 osd=6 1 2024-03-27T21:41:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 10 client.4889.objecter close_session for osd.7 2024-03-27T21:41:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f873c1f4e60 msgr2=0x7f873c1f52c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f873c1f4e60 0x7f873c1f52c0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 20 client.4889.objecter put_session s=0x7f873c1f4740 osd=7 1 2024-03-27T21:41:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 20 client.4889.objecter shutdown clearing up homeless session... 2024-03-27T21:41:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 10 client.4889.objecter successfully canceled tick 2024-03-27T21:41:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f87426ae640 1 -- 172.21.15.60:0/4051066867 reap_dead start 2024-03-27T21:41:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f872006c5c0 msgr2=0x7f872006ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f872006c5c0 0x7f872006ea80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8738001d10 tx=0x7f8738009030 comp rx=0 tx=0).stop 2024-03-27T21:41:45.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f87426ae640 1 -- 172.21.15.60:0/4051066867 reap_dead start 2024-03-27T21:41:45.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 10 monclient: shutdown 2024-03-27T21:41:45.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:45.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f873c0ac520 msgr2=0x7f873c1d63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:45.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.477+0000 7f8744111640 1 --2- 172.21.15.60:0/4051066867 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f873c0ac520 0x7f873c1d63e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f872c0118e0 tx=0x7f872c011da0 comp rx=0 tx=0).stop 2024-03-27T21:41:45.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.478+0000 7f87426ae640 1 -- 172.21.15.60:0/4051066867 reap_dead start 2024-03-27T21:41:45.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.478+0000 7f87426ae640 1 -- 172.21.15.60:0/4051066867 reap_dead start 2024-03-27T21:41:45.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.478+0000 7f87426ae640 1 -- 172.21.15.60:0/4051066867 reap_dead start 2024-03-27T21:41:45.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.478+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 shutdown_connections 2024-03-27T21:41:45.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.478+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 >> 172.21.15.60:0/4051066867 conn(0x7f873c0b7300 msgr2=0x7f873c141a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:45.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.478+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 shutdown_connections 2024-03-27T21:41:45.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.478+0000 7f8744111640 1 -- 172.21.15.60:0/4051066867 wait complete. 2024-03-27T21:41:45.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:45.478+0000 7f8744111640 1 librados: shutdown 2024-03-27T21:41:45.470 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:45.470 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:45.474 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-27T21:41:45.488+0000 7f439c549640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34587) UID: 0 2024-03-27T21:41:45.477 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-27T21:41:45.575 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-27T21:41:45.589+0000 7f84b0dad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 34327) UID: 0 2024-03-27T21:41:45.617 DEBUG:teuthology.orchestra.run:got remote process result: 128 2024-03-27T21:41:45.617 INFO:tasks.workunit.client.0.smithi060.stderr:fatal: reference is not a tree: 7c37d80f7a88f4eb561c4fe02558660a650b10ac 2024-03-27T21:41:45.618 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7c37d80f7a88f4eb561c4fe02558660a650b10ac/qa/tasks/workunit.py", line 345, in _run_tests remote.run(logger=log.getChild(role), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7c37d80f7a88f4eb561c4fe02558660a650b10ac' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_7c37d80f7a88f4eb561c4fe02558660a650b10ac/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_7c37d80f7a88f4eb561c4fe02558660a650b10ac/qa/tasks/workunit.py", line 360, in _run_tests remote.run(logger=log.getChild(role), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7c37d80f7a88f4eb561c4fe02558660a650b10ac' 2024-03-27T21:41:45.847 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=fb7e5aa0366943489b588c0fdabf6654 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7c37d80f7a88f4eb561c4fe02558660a650b10ac/qa/tasks/workunit.py", line 345, in _run_tests remote.run(logger=log.getChild(role), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7c37d80f7a88f4eb561c4fe02558660a650b10ac' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_7c37d80f7a88f4eb561c4fe02558660a650b10ac/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_7c37d80f7a88f4eb561c4fe02558660a650b10ac/qa/tasks/workunit.py", line 360, in _run_tests remote.run(logger=log.getChild(role), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7c37d80f7a88f4eb561c4fe02558660a650b10ac' 2024-03-27T21:41:45.850 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-27T21:41:45.860 INFO:tasks.thrashosds:joining thrashosds 2024-03-27T21:41:45.865 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:45.865 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:45.866 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-27T21:41:46.014 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:46.014 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:46.023 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-27T21:41:46.130 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:46.130 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:46.139 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-27T21:41:46.230 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-27T21:41:46.246 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:46.247 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:46.256 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-27T21:41:46.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.346+0000 7fce0c367640 10 monclient: get_monmap_and_config 2024-03-27T21:41:46.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.346+0000 7fce0c367640 10 monclient: build_initial_monmap 2024-03-27T21:41:46.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.346+0000 7fce0c367640 10 monclient: monmap: 2024-03-27T21:41:46.333 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:46.333 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:46.333 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:46.347203+0000 2024-03-27T21:41:46.333 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:46.347203+0000 2024-03-27T21:41:46.333 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:46.333 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:46.334 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:46.334 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:46.334 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:46.334 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:46.334 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:46.334 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:46.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.346+0000 7fce0c367640 1 Processor -- start 2024-03-27T21:41:46.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.346+0000 7fce0c367640 1 -- start start 2024-03-27T21:41:46.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.346+0000 7fce0c367640 10 monclient: init 2024-03-27T21:41:46.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce0c367640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:46.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce0c367640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-03-27T21:41:46.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce0c367640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fce04071a50 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:46.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce0c367640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce0413ef20 0x7fce040a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:46.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce0c367640 10 monclient(hunting): picked mon.noname-b con 0x7fce0413ef20 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:46.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce0c367640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fce04141630 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:46.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce0c367640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce040a8e90 con 0x7fce04141630 2024-03-27T21:41:46.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce0c367640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce040aa040 con 0x7fce04071a50 2024-03-27T21:41:46.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce0c367640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:46.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce0c367640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce04071e30 con 0x7fce0413ef20 2024-03-27T21:41:46.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce0c367640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:46.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce0c367640 10 monclient(hunting): authenticate will time out at 947.568546s 2024-03-27T21:41:46.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce09902640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce0413ef20 0x7fce040a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:46.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce0a904640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fce04141630 0x7fce04141ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47996/0 (socket says 172.21.15.60:47996) 2024-03-27T21:41:46.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce0a904640 1 -- 172.21.15.60:0/1244629651 learned_addr learned my addr 172.21.15.60:0/1244629651 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:46.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce09902640 10 monclient(hunting): get_auth_request con 0x7fce0413ef20 auth_method 0 2024-03-27T21:41:46.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce09902640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:46.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce09902640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:46.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce09902640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:46.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce09902640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:46.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce09902640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:46.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce09902640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:46.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.347+0000 7fce09101640 1 -- 172.21.15.60:0/1244629651 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352296440 0 0) 0x7fce040a8e90 con 0x7fce04141630 2024-03-27T21:41:46.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09101640 10 monclient(hunting): _negotiate 2024-03-27T21:41:46.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09101640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:46.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09101640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:46.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09101640 10 monclient(hunting): my global_id is 5136 2024-03-27T21:41:46.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09101640 1 -- 172.21.15.60:0/1244629651 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdec003520 con 0x7fce04141630 2024-03-27T21:41:46.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09902640 10 monclient(hunting): handle_auth_done global_id 4892 payload 274 2024-03-27T21:41:46.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09902640 1 -- 172.21.15.60:0/1244629651 >> v1:172.21.15.191:6789/0 conn(0x7fce04071a50 legacy=0x7fce0413e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:46.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09902640 1 -- 172.21.15.60:0/1244629651 >> v1:172.21.15.60:6789/0 conn(0x7fce04141630 legacy=0x7fce04141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:46.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09902640 10 monclient: _finish_hunting 0 2024-03-27T21:41:46.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09902640 1 monclient: found mon.noname-b 2024-03-27T21:41:46.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09902640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:46.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09902640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:46.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09902640 1 -- 172.21.15.60:0/1244629651 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce040aa2b0 con 0x7fce0413ef20 2024-03-27T21:41:46.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09902640 1 --2- 172.21.15.60:0/1244629651 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce0413ef20 0x7fce040a87e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcdf80097d0 tx=0x7fcdf802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5d1290c967a0acd server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:46.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09101640 1 -- 172.21.15.60:0/1244629651 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf8031890 con 0x7fce0413ef20 2024-03-27T21:41:46.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09101640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:46.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09101640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:46.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09101640 10 monclient: dump: 2024-03-27T21:41:46.351 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:46.351 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:46.351 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:46.351 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:46.351 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:46.351 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:46.351 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:46.352 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:46.352 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:46.352 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:46.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09101640 10 monclient: _finish_auth 0 2024-03-27T21:41:46.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09101640 10 monclient: _check_auth_tickets 2024-03-27T21:41:46.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09101640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce0c367640 5 monclient: authenticate success, global_id 4892 2024-03-27T21:41:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce0c367640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09101640 1 -- 172.21.15.60:0/1244629651 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf80319f0 con 0x7fce0413ef20 2024-03-27T21:41:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09101640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce0c367640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce0c367640 4 set_mon_vals no callback set 2024-03-27T21:41:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09101640 1 -- 172.21.15.60:0/1244629651 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf8031d60 con 0x7fce0413ef20 2024-03-27T21:41:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09101640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:46.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09101640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:46.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.348+0000 7fce09101640 10 monclient: dump: 2024-03-27T21:41:46.354 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:46.354 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:46.354 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:46.354 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:46.354 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:46.354 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:46.355 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:46.355 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:46.355 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:46.355 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:46.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.349+0000 7fce0c367640 10 monclient: shutdown 2024-03-27T21:41:46.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.349+0000 7fce0c367640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:46.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.349+0000 7fce0c367640 1 -- 172.21.15.60:0/1244629651 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce0413ef20 msgr2=0x7fce040a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:46.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.349+0000 7fce0c367640 1 --2- 172.21.15.60:0/1244629651 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce0413ef20 0x7fce040a87e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcdf80097d0 tx=0x7fcdf802b930 comp rx=0 tx=0).stop 2024-03-27T21:41:46.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.349+0000 7fce0c367640 1 -- 172.21.15.60:0/1244629651 shutdown_connections 2024-03-27T21:41:46.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.349+0000 7fce0c367640 1 --2- 172.21.15.60:0/1244629651 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce0413ef20 0x7fce040a87e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:46.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.349+0000 7fce0c367640 1 -- 172.21.15.60:0/1244629651 >> 172.21.15.60:0/1244629651 conn(0x7fce040b7300 msgr2=0x7fce040b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:46.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.349+0000 7fce0c367640 1 -- 172.21.15.60:0/1244629651 shutdown_connections 2024-03-27T21:41:46.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.349+0000 7fce0c367640 1 -- 172.21.15.60:0/1244629651 wait complete. 2024-03-27T21:41:46.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.350+0000 7fce0c367640 10 monclient: build_initial_monmap 2024-03-27T21:41:46.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.350+0000 7fce0c367640 10 monclient: monmap: 2024-03-27T21:41:46.356 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:46.356 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:46.357 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:46.357 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:46.357 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:46.357 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:46.357 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:46.357 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:46.357 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:46.357 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:46.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.350+0000 7fce0c367640 1 librados: starting msgr at 2024-03-27T21:41:46.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.350+0000 7fce0c367640 1 librados: starting objecter 2024-03-27T21:41:46.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.350+0000 7fce0c367640 1 Processor -- start 2024-03-27T21:41:46.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.350+0000 7fce0c367640 1 -- start start 2024-03-27T21:41:46.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.350+0000 7fce0c367640 1 librados: setting wanted keys 2024-03-27T21:41:46.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.350+0000 7fce0c367640 1 librados: calling monclient init 2024-03-27T21:41:46.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.350+0000 7fce0c367640 10 monclient: init 2024-03-27T21:41:46.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.350+0000 7fce0c367640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:46.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:41:46.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce04071a50 0x7fce0413ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:46.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 10 monclient(hunting): picked mon.noname-b con 0x7fce04071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:46.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce0413ef20 0x7fce040b6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:46.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 10 monclient(hunting): picked mon.noname-c con 0x7fce0413ef20 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:46.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce04141630 0x7fce041d97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:46.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 10 monclient(hunting): picked mon.noname-a con 0x7fce04141630 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:46.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:46.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce041376f0 con 0x7fce04141630 2024-03-27T21:41:46.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:46.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce041d9eb0 con 0x7fce04071a50 2024-03-27T21:41:46.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:46.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce041da3d0 con 0x7fce0413ef20 2024-03-27T21:41:46.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:46.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 10 monclient(hunting): authenticate will time out at 947.571978s 2024-03-27T21:41:46.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce09902640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce0413ef20 0x7fce040b6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:46.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce09902640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce0413ef20 0x7fce040b6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:32806/0 (socket says 172.21.15.60:32806) 2024-03-27T21:41:46.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce09902640 1 -- 172.21.15.60:0/1410679561 learned_addr learned my addr 172.21.15.60:0/1410679561 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:46.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0a904640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce04141630 0x7fce041d97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:46.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce09902640 10 monclient(hunting): get_auth_request con 0x7fce0413ef20 auth_method 0 2024-03-27T21:41:46.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce09902640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:46.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce09902640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:46.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce09902640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:46.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0a904640 10 monclient(hunting): get_auth_request con 0x7fce04141630 auth_method 0 2024-03-27T21:41:46.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0a904640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:46.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0a904640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:46.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0a904640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:46.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce09902640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:46.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce09902640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:46.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0a103640 1 --2- 172.21.15.60:0/1410679561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce04071a50 0x7fce0413ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:46.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce09902640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:46.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0a904640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:46.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0a904640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:46.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0a904640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:46.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0a103640 10 monclient(hunting): get_auth_request con 0x7fce04071a50 auth_method 0 2024-03-27T21:41:46.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0a103640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:46.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0a103640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:46.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0a103640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:46.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0a103640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:46.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0a103640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:46.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0a103640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:46.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce09902640 10 monclient(hunting): handle_auth_done global_id 4895 payload 824 2024-03-27T21:41:46.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce09902640 1 -- 172.21.15.60:0/1410679561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce04071a50 msgr2=0x7fce0413ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:46.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce09902640 1 --2- 172.21.15.60:0/1410679561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce04071a50 0x7fce0413ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:46.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce09902640 1 -- 172.21.15.60:0/1410679561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce04141630 msgr2=0x7fce041d97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:46.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce09902640 1 --2- 172.21.15.60:0/1410679561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce04141630 0x7fce041d97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:46.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce09902640 10 monclient: _finish_hunting 0 2024-03-27T21:41:46.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce09902640 1 monclient: found mon.noname-c 2024-03-27T21:41:46.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce09902640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:46.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce09902640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:46.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce09902640 1 -- 172.21.15.60:0/1410679561 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf8009400 con 0x7fce0413ef20 2024-03-27T21:41:46.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0a904640 1 --2- 172.21.15.60:0/1410679561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce04141630 0x7fce041d97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:46.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce09902640 1 --2- 172.21.15.60:0/1410679561 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce0413ef20 0x7fce040b6730 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8009720 tx=0x7fcdf80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:46.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce02ffd640 10 client.?.objecter ms_handle_connect 0x7fce0413ef20 2024-03-27T21:41:46.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce02ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:46.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce02ffd640 1 -- 172.21.15.60:0/1410679561 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf803cc00 con 0x7fce0413ef20 2024-03-27T21:41:46.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:46.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce02ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:46.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce02ffd640 10 monclient: dump: 2024-03-27T21:41:46.367 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:46.367 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:46.367 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:46.368 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:46.368 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:46.368 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:46.368 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:46.368 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:46.368 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:46.368 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:46.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce02ffd640 10 monclient: _finish_auth 0 2024-03-27T21:41:46.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce02ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:41:46.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:46.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 5 monclient: authenticate success, global_id 4895 2024-03-27T21:41:46.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce02ffd640 1 -- 172.21.15.60:0/1410679561 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf803cd60 con 0x7fce0413ef20 2024-03-27T21:41:46.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 10 monclient: _renew_subs 2024-03-27T21:41:46.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:46.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 1 -- 172.21.15.60:0/1410679561 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce041da5f0 con 0x7fce0413ef20 2024-03-27T21:41:46.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce02ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:46.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce02ffd640 1 -- 172.21.15.60:0/1410679561 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf8004230 con 0x7fce0413ef20 2024-03-27T21:41:46.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:46.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:46.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce02ffd640 10 monclient: dump: 2024-03-27T21:41:46.370 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:46.370 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:46.370 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:46.370 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:46.371 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:46.371 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:46.371 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:46.371 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:46.371 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:46.371 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:46.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce08900640 4 set_mon_vals no callback set 2024-03-27T21:41:46.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 10 client.4895.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:46.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 10 monclient: _renew_subs 2024-03-27T21:41:46.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:46.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 1 -- 172.21.15.60:0/1410679561 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce041d64c0 con 0x7fce0413ef20 2024-03-27T21:41:46.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.351+0000 7fce0c367640 1 librados: init done 2024-03-27T21:41:46.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.352+0000 7fce0c367640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:46.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.352+0000 7fce0c367640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:46.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.352+0000 7fce0c367640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:46.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.352+0000 7fce0c367640 1 -- 172.21.15.60:0/1410679561 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce041d9d20 con 0x7fce0413ef20 2024-03-27T21:41:46.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.353+0000 7fce02ffd640 1 -- 172.21.15.60:0/1410679561 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcdf8004390 con 0x7fce0413ef20 2024-03-27T21:41:46.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.353+0000 7fce02ffd640 1 --2- 172.21.15.60:0/1410679561 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcdf006c590 0x7fcdf006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:46.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.353+0000 7fce0a103640 1 --2- 172.21.15.60:0/1410679561 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcdf006c590 0x7fcdf006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:46.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.353+0000 7fce02ffd640 1 -- 172.21.15.60:0/1410679561 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fcdf80abd40 con 0x7fce0413ef20 2024-03-27T21:41:46.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.353+0000 7fce02ffd640 10 client.4895.objecter ms_dispatch 0x7fce04000b90 osd_map(74..74 src has 1..74) v4 2024-03-27T21:41:46.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.353+0000 7fce02ffd640 3 client.4895.objecter handle_osd_map got epochs [74,74] > 0 2024-03-27T21:41:46.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.353+0000 7fce02ffd640 3 client.4895.objecter handle_osd_map decoding full epoch 74 2024-03-27T21:41:46.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.353+0000 7fce0a103640 10 monclient: get_auth_request con 0x7fcdf006c590 auth_method 0 2024-03-27T21:41:46.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.353+0000 7fce02ffd640 20 client.4895.objecter dump_active .. 0 homeless 2024-03-27T21:41:46.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.354+0000 7fce02ffd640 10 client.4895.objecter ms_handle_connect 0x7fcdf006c590 2024-03-27T21:41:46.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.354+0000 7fce0a103640 1 --2- 172.21.15.60:0/1410679561 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcdf006c590 0x7fcdf006ea50 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc000f60 tx=0x7fcdfc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:46.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.358+0000 7fce02ffd640 1 -- 172.21.15.60:0/1410679561 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcdf8077190 con 0x7fce0413ef20 2024-03-27T21:41:46.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.358+0000 7fce02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:46.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.358+0000 7fce02ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:46.375 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:46.375 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:46.383 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-27T21:41:46.488 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:46.488 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:46.496 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-27T21:41:46.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.547+0000 7fce0c367640 10 start_mon_command cmd=[{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:41:46.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.547+0000 7fce0c367640 10 monclient: _send_command 2 [{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:41:46.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.547+0000 7fce0c367640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:46.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:46.547+0000 7fce0c367640 1 -- 172.21.15.60:0/1410679561 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fce04069470 con 0x7fce0413ef20 2024-03-27T21:41:46.601 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:46.601 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:46.610 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-27T21:41:46.716 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:46.716 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:46.724 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-27T21:41:46.829 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:46.829 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:46.838 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-27T21:41:46.943 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:46.943 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:46.952 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-27T21:41:47.058 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:47.058 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:47.067 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-27T21:41:47.108 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-27T21:41:47.109 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-03-27T21:41:47.109 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2024-03-27T21:41:47.171 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:47.172 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:47.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.187+0000 7fce02ffd640 1 -- 172.21.15.60:0/1410679561 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v75) v1 ==== 91+0+0 (secure 0 0 0) 0x7fcdf8035070 con 0x7fce0413ef20 2024-03-27T21:41:47.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.187+0000 7fce02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd unset", "key": "noscrub"}] 2024-03-27T21:41:47.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.187+0000 7fce02ffd640 10 monclient: _finish_command 2 = system:0 noscrub is unset 2024-03-27T21:41:47.174 INFO:teuthology.orchestra.run.smithi060.stderr:noscrub is unset 2024-03-27T21:41:47.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.189+0000 7fce0c367640 10 librados: watch_flush enter 2024-03-27T21:41:47.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.189+0000 7fce0c367640 10 librados: watch_flush exit 2024-03-27T21:41:47.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.189+0000 7fce0c367640 20 client.4895.objecter shutdown clearing up homeless session... 2024-03-27T21:41:47.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.189+0000 7fce0c367640 10 client.4895.objecter successfully canceled tick 2024-03-27T21:41:47.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.190+0000 7fce0c367640 1 -- 172.21.15.60:0/1410679561 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcdf006c590 msgr2=0x7fcdf006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:47.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.190+0000 7fce0c367640 1 --2- 172.21.15.60:0/1410679561 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcdf006c590 0x7fcdf006ea50 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc000f60 tx=0x7fcdfc009040 comp rx=0 tx=0).stop 2024-03-27T21:41:47.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.190+0000 7fce0c367640 10 monclient: shutdown 2024-03-27T21:41:47.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.190+0000 7fce0c367640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:47.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.190+0000 7fce0c367640 1 -- 172.21.15.60:0/1410679561 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce0413ef20 msgr2=0x7fce040b6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:47.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.190+0000 7fce0c367640 1 --2- 172.21.15.60:0/1410679561 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce0413ef20 0x7fce040b6730 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8009720 tx=0x7fcdf80046c0 comp rx=0 tx=0).stop 2024-03-27T21:41:47.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.190+0000 7fce0c367640 1 -- 172.21.15.60:0/1410679561 shutdown_connections 2024-03-27T21:41:47.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.190+0000 7fce0c367640 1 --2- 172.21.15.60:0/1410679561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce04141630 0x7fce041d97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:47.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.190+0000 7fce0c367640 1 --2- 172.21.15.60:0/1410679561 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce0413ef20 0x7fce040b6730 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:47.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.190+0000 7fce0c367640 1 --2- 172.21.15.60:0/1410679561 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcdf006c590 0x7fcdf006ea50 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:47.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.190+0000 7fce0c367640 1 --2- 172.21.15.60:0/1410679561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce04071a50 0x7fce0413ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:47.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.190+0000 7fce0c367640 1 -- 172.21.15.60:0/1410679561 >> 172.21.15.60:0/1410679561 conn(0x7fce040b7300 msgr2=0x7fce0413e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:47.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.190+0000 7fce0c367640 1 -- 172.21.15.60:0/1410679561 shutdown_connections 2024-03-27T21:41:47.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.190+0000 7fce0c367640 1 -- 172.21.15.60:0/1410679561 wait complete. 2024-03-27T21:41:47.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.190+0000 7fce0c367640 1 librados: shutdown 2024-03-27T21:41:47.181 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-27T21:41:47.189 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-27T21:41:47.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.240+0000 7fa4fd6cf640 10 monclient: get_monmap_and_config 2024-03-27T21:41:47.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.240+0000 7fa4fd6cf640 10 monclient: build_initial_monmap 2024-03-27T21:41:47.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.240+0000 7fa4fd6cf640 10 monclient: monmap: 2024-03-27T21:41:47.227 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:47.227 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:47.227 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:47.241717+0000 2024-03-27T21:41:47.227 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:47.241717+0000 2024-03-27T21:41:47.228 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:47.228 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:47.228 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:47.228 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:47.228 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:47.228 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:47.228 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:47.228 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:47.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.241+0000 7fa4fd6cf640 1 Processor -- start 2024-03-27T21:41:47.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.241+0000 7fa4fd6cf640 1 -- start start 2024-03-27T21:41:47.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.241+0000 7fa4fd6cf640 10 monclient: init 2024-03-27T21:41:47.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4fd6cf640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:47.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4fd6cf640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-03-27T21:41:47.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4fd6cf640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa4f80b40b0 0x7fa4f80a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:47.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4fd6cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa4f80b40b0 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:47.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4fd6cf640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa4f813b020 0x7fa4f813b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:47.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4fd6cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa4f813b020 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:47.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4fd6cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4f81416f0 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:47.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4fd6cf640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f80b3830 con 0x7fa4f81416f0 2024-03-27T21:41:47.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4fd6cf640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:47.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4fd6cf640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4f80b4490 con 0x7fa4f80b40b0 2024-03-27T21:41:47.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4fd6cf640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:47.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4fd6cf640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4f80aa520 con 0x7fa4f813b020 2024-03-27T21:41:47.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4fd6cf640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:47.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4fd6cf640 10 monclient(hunting): authenticate will time out at 948.463131s 2024-03-27T21:41:47.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4f6ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa4f80b40b0 0x7fa4f80a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:47.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4f6ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa4f80b40b0 0x7fa4f80a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41806/0 (socket says 172.21.15.60:41806) 2024-03-27T21:41:47.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4f6ffd640 1 -- 172.21.15.60:0/2903177964 learned_addr learned my addr 172.21.15.60:0/2903177964 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:47.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4f6ffd640 10 monclient(hunting): get_auth_request con 0x7fa4f80b40b0 auth_method 0 2024-03-27T21:41:47.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:47.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4f77fe640 1 --1- 172.21.15.60:0/2903177964 >> v1:172.21.15.60:6789/0 conn(0x7fa4f81416f0 0x7fa4f8141b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:48002/0 (socket says 172.21.15.60:48002) 2024-03-27T21:41:47.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:47.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:47.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4effff640 1 --2- 172.21.15.60:0/2903177964 >> v2:172.21.15.191:3300/0 conn(0x7fa4f813b020 0x7fa4f813b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:47.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:47.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:47.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:47.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4effff640 10 monclient(hunting): get_auth_request con 0x7fa4f813b020 auth_method 0 2024-03-27T21:41:47.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:47.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4effff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:47.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:47.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:47.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:47.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:47.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4f6ffd640 10 monclient(hunting): handle_auth_done global_id 5142 payload 274 2024-03-27T21:41:47.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4f6ffd640 1 -- 172.21.15.60:0/2903177964 >> v2:172.21.15.191:3300/0 conn(0x7fa4f813b020 msgr2=0x7fa4f813b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4f6ffd640 1 --2- 172.21.15.60:0/2903177964 >> v2:172.21.15.191:3300/0 conn(0x7fa4f813b020 0x7fa4f813b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4f6ffd640 1 -- 172.21.15.60:0/2903177964 >> v1:172.21.15.60:6789/0 conn(0x7fa4f81416f0 legacy=0x7fa4f8141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4f6ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:41:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4f6ffd640 1 monclient: found mon.noname-a 2024-03-27T21:41:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4f6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.242+0000 7fa4f6ffd640 1 -- 172.21.15.60:0/2903177964 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f80a9470 con 0x7fa4f80b40b0 2024-03-27T21:41:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.243+0000 7fa4f6ffd640 1 --2- 172.21.15.60:0/2903177964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4f80b40b0 0x7fa4f80a8b90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa4f80b47e0 tx=0x7fa4e802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f42319c7885a56ba server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.243+0000 7fa4f67fc640 1 -- 172.21.15.60:0/2903177964 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8030860 con 0x7fa4f80b40b0 2024-03-27T21:41:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.243+0000 7fa4f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.243+0000 7fa4f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.243+0000 7fa4f67fc640 10 monclient: dump: 2024-03-27T21:41:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:47.246 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.243+0000 7fa4f67fc640 10 monclient: _finish_auth 0 2024-03-27T21:41:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.243+0000 7fa4f67fc640 10 monclient: _check_auth_tickets 2024-03-27T21:41:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.243+0000 7fa4f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.243+0000 7fa4f67fc640 1 -- 172.21.15.60:0/2903177964 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e80309c0 con 0x7fa4f80b40b0 2024-03-27T21:41:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.243+0000 7fa4fd6cf640 5 monclient: authenticate success, global_id 5142 2024-03-27T21:41:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.243+0000 7fa4fd6cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.243+0000 7fa4f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.243+0000 7fa4f67fc640 1 -- 172.21.15.60:0/2903177964 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8030d50 con 0x7fa4f80b40b0 2024-03-27T21:41:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.243+0000 7fa4fd6cf640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.243+0000 7fa4fd6cf640 4 set_mon_vals no callback set 2024-03-27T21:41:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.243+0000 7fa4f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.243+0000 7fa4f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.243+0000 7fa4f67fc640 10 monclient: dump: 2024-03-27T21:41:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:47.249 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.244+0000 7fa4fd6cf640 10 monclient: shutdown 2024-03-27T21:41:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.244+0000 7fa4fd6cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.244+0000 7fa4fd6cf640 1 -- 172.21.15.60:0/2903177964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4f80b40b0 msgr2=0x7fa4f80a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.244+0000 7fa4fd6cf640 1 --2- 172.21.15.60:0/2903177964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4f80b40b0 0x7fa4f80a8b90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa4f80b47e0 tx=0x7fa4e802b9a0 comp rx=0 tx=0).stop 2024-03-27T21:41:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.244+0000 7fa4fd6cf640 1 -- 172.21.15.60:0/2903177964 shutdown_connections 2024-03-27T21:41:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.244+0000 7fa4fd6cf640 1 --2- 172.21.15.60:0/2903177964 >> v2:172.21.15.191:3300/0 conn(0x7fa4f813b020 0x7fa4f813b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.244+0000 7fa4fd6cf640 1 --2- 172.21.15.60:0/2903177964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4f80b40b0 0x7fa4f80a8b90 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.244+0000 7fa4fd6cf640 1 -- 172.21.15.60:0/2903177964 >> 172.21.15.60:0/2903177964 conn(0x7fa4f80608e0 msgr2=0x7fa4f8060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.244+0000 7fa4fd6cf640 1 -- 172.21.15.60:0/2903177964 shutdown_connections 2024-03-27T21:41:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.244+0000 7fa4fd6cf640 1 -- 172.21.15.60:0/2903177964 wait complete. 2024-03-27T21:41:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.245+0000 7fa4fd6cf640 10 monclient: build_initial_monmap 2024-03-27T21:41:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.245+0000 7fa4fd6cf640 10 monclient: monmap: 2024-03-27T21:41:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:47.251 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.245+0000 7fa4fd6cf640 1 librados: starting msgr at 2024-03-27T21:41:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.245+0000 7fa4fd6cf640 1 librados: starting objecter 2024-03-27T21:41:47.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.245+0000 7fa4fd6cf640 1 Processor -- start 2024-03-27T21:41:47.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.245+0000 7fa4fd6cf640 1 -- start start 2024-03-27T21:41:47.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.245+0000 7fa4fd6cf640 1 librados: setting wanted keys 2024-03-27T21:41:47.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.245+0000 7fa4fd6cf640 1 librados: calling monclient init 2024-03-27T21:41:47.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.245+0000 7fa4fd6cf640 10 monclient: init 2024-03-27T21:41:47.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4fd6cf640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4fd6cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:41:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4fd6cf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4f80b40b0 0x7fa4f80aa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4fd6cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa4f80b40b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4fd6cf640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4f813b020 0x7fa4f80aaf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4fd6cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa4f813b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:47.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4fd6cf640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa4f81416f0 0x7fa4f80afc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:47.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4fd6cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa4f81416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:47.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4fd6cf640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:47.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4fd6cf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f80ac500 con 0x7fa4f80b40b0 2024-03-27T21:41:47.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4fd6cf640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:47.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4fd6cf640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f813e460 con 0x7fa4f813b020 2024-03-27T21:41:47.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4fd6cf640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:47.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4fd6cf640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f813e760 con 0x7fa4f81416f0 2024-03-27T21:41:47.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4fd6cf640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4fd6cf640 10 monclient(hunting): authenticate will time out at 948.467028s 2024-03-27T21:41:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4f80b40b0 0x7fa4f80aa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4f80b40b0 0x7fa4f80aa9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41808/0 (socket says 172.21.15.60:41808) 2024-03-27T21:41:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f6ffd640 1 -- 172.21.15.60:0/1215651908 learned_addr learned my addr 172.21.15.60:0/1215651908 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f6ffd640 10 monclient(hunting): get_auth_request con 0x7fa4f80b40b0 auth_method 0 2024-03-27T21:41:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:47.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f77fe640 1 --2- 172.21.15.60:0/1215651908 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa4f81416f0 0x7fa4f80afc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:47.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f77fe640 10 monclient(hunting): get_auth_request con 0x7fa4f81416f0 auth_method 0 2024-03-27T21:41:47.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:47.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:47.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:47.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4effff640 1 --2- 172.21.15.60:0/1215651908 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4f813b020 0x7fa4f80aaf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:47.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:47.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:47.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:47.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4effff640 10 monclient(hunting): get_auth_request con 0x7fa4f813b020 auth_method 0 2024-03-27T21:41:47.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:47.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4effff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:47.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:47.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:47.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:47.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:47.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:47.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:47.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:47.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f6ffd640 10 monclient(hunting): handle_auth_done global_id 5148 payload 824 2024-03-27T21:41:47.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f6ffd640 1 -- 172.21.15.60:0/1215651908 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa4f81416f0 msgr2=0x7fa4f80afc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:47.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f6ffd640 1 --2- 172.21.15.60:0/1215651908 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa4f81416f0 0x7fa4f80afc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:47.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f6ffd640 1 -- 172.21.15.60:0/1215651908 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4f813b020 msgr2=0x7fa4f80aaf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:47.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f6ffd640 1 --2- 172.21.15.60:0/1215651908 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4f813b020 0x7fa4f80aaf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:47.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f6ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:41:47.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f6ffd640 1 monclient: found mon.noname-a 2024-03-27T21:41:47.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:47.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:47.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f6ffd640 1 -- 172.21.15.60:0/1215651908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e80094d0 con 0x7fa4f80b40b0 2024-03-27T21:41:47.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4f6ffd640 1 --2- 172.21.15.60:0/1215651908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4f80b40b0 0x7fa4f80aa9d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa4e802b9e0 tx=0x7fa4e8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:47.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4ef7fe640 10 client.?.objecter ms_handle_connect 0x7fa4f80b40b0 2024-03-27T21:41:47.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:47.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4ef7fe640 1 -- 172.21.15.60:0/1215651908 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8039c90 con 0x7fa4f80b40b0 2024-03-27T21:41:47.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:47.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4ef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:47.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.246+0000 7fa4ef7fe640 10 monclient: dump: 2024-03-27T21:41:47.261 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:47.261 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:47.261 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:47.261 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:47.262 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:47.262 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:47.262 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:47.262 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:47.262 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:47.262 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:47.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.247+0000 7fa4ef7fe640 10 monclient: _finish_auth 0 2024-03-27T21:41:47.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.247+0000 7fa4ef7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:41:47.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.247+0000 7fa4ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:47.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.247+0000 7fa4ef7fe640 1 -- 172.21.15.60:0/1215651908 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e8039e10 con 0x7fa4f80b40b0 2024-03-27T21:41:47.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.247+0000 7fa4fd6cf640 5 monclient: authenticate success, global_id 5148 2024-03-27T21:41:47.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.247+0000 7fa4fd6cf640 10 monclient: _renew_subs 2024-03-27T21:41:47.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.247+0000 7fa4fd6cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:47.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.247+0000 7fa4fd6cf640 1 -- 172.21.15.60:0/1215651908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f813e980 con 0x7fa4f80b40b0 2024-03-27T21:41:47.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.247+0000 7fa4ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:47.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.247+0000 7fa4ef7fe640 1 -- 172.21.15.60:0/1215651908 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8043640 con 0x7fa4f80b40b0 2024-03-27T21:41:47.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.247+0000 7fa4ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:47.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.247+0000 7fa4ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:47.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.247+0000 7fa4ef7fe640 10 monclient: dump: 2024-03-27T21:41:47.264 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:47.264 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:47.264 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:47.264 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:47.264 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:47.264 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:47.264 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:47.265 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:47.265 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:47.265 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:47.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.247+0000 7fa4fd6cf640 10 client.5148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:47.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.247+0000 7fa4f5ffb640 4 set_mon_vals no callback set 2024-03-27T21:41:47.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.247+0000 7fa4fd6cf640 10 monclient: _renew_subs 2024-03-27T21:41:47.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.247+0000 7fa4fd6cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:47.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.247+0000 7fa4fd6cf640 1 -- 172.21.15.60:0/1215651908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f80b0330 con 0x7fa4f80b40b0 2024-03-27T21:41:47.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.247+0000 7fa4fd6cf640 1 librados: init done 2024-03-27T21:41:47.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.248+0000 7fa4fd6cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:47.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.248+0000 7fa4fd6cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:47.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.248+0000 7fa4fd6cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:47.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.248+0000 7fa4fd6cf640 1 -- 172.21.15.60:0/1215651908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f8069470 con 0x7fa4f80b40b0 2024-03-27T21:41:47.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.249+0000 7fa4ef7fe640 1 -- 172.21.15.60:0/1215651908 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa4e80437a0 con 0x7fa4f80b40b0 2024-03-27T21:41:47.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.249+0000 7fa4ef7fe640 1 --2- 172.21.15.60:0/1215651908 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa4d806c590 0x7fa4d806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:47.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.249+0000 7fa4ef7fe640 1 -- 172.21.15.60:0/1215651908 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fa4e803e070 con 0x7fa4f80b40b0 2024-03-27T21:41:47.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.249+0000 7fa4ef7fe640 10 client.5148.objecter ms_dispatch 0x7fa4f8000b90 osd_map(75..75 src has 1..75) v4 2024-03-27T21:41:47.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.249+0000 7fa4ef7fe640 3 client.5148.objecter handle_osd_map got epochs [75,75] > 0 2024-03-27T21:41:47.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.249+0000 7fa4ef7fe640 3 client.5148.objecter handle_osd_map decoding full epoch 75 2024-03-27T21:41:47.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.249+0000 7fa4effff640 1 --2- 172.21.15.60:0/1215651908 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa4d806c590 0x7fa4d806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:47.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.249+0000 7fa4effff640 10 monclient: get_auth_request con 0x7fa4d806c590 auth_method 0 2024-03-27T21:41:47.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.249+0000 7fa4ef7fe640 20 client.5148.objecter dump_active .. 0 homeless 2024-03-27T21:41:47.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.250+0000 7fa4effff640 1 --2- 172.21.15.60:0/1215651908 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa4d806c590 0x7fa4d806ea50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa4f813ae50 tx=0x7fa4e0009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:47.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.250+0000 7fa4ef7fe640 10 client.5148.objecter ms_handle_connect 0x7fa4d806c590 2024-03-27T21:41:47.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.254+0000 7fa4ef7fe640 1 -- 172.21.15.60:0/1215651908 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa4e80776a0 con 0x7fa4f80b40b0 2024-03-27T21:41:47.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.254+0000 7fa4ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:47.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.254+0000 7fa4ef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:47.286 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:47.286 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:47.295 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-27T21:41:47.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.309+0000 7fa182f95640 10 monclient: get_monmap_and_config 2024-03-27T21:41:47.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.309+0000 7fa182f95640 10 monclient: build_initial_monmap 2024-03-27T21:41:47.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.309+0000 7fa182f95640 10 monclient: monmap: 2024-03-27T21:41:47.296 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:47.297 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:47.297 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:47.310700+0000 2024-03-27T21:41:47.297 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:47.310700+0000 2024-03-27T21:41:47.297 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:47.297 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:47.297 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:47.297 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:47.297 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:47.298 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:47.298 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:47.298 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:47.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.310+0000 7fa182f95640 1 Processor -- start 2024-03-27T21:41:47.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.310+0000 7fa182f95640 1 -- start start 2024-03-27T21:41:47.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.310+0000 7fa182f95640 10 monclient: init 2024-03-27T21:41:47.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.310+0000 7fa182f95640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:47.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.310+0000 7fa182f95640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-03-27T21:41:47.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa182f95640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa17c071a50 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:47.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa182f95640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa17c13ef20 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:47.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa182f95640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa17c0a87e0 0x7fa17c141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:47.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa182f95640 10 monclient(hunting): picked mon.noname-a con 0x7fa17c0a87e0 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:47.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa182f95640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c0a8e90 con 0x7fa17c071a50 2024-03-27T21:41:47.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa182f95640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c0aa040 con 0x7fa17c13ef20 2024-03-27T21:41:47.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa182f95640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:47.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa182f95640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa17c071e30 con 0x7fa17c0a87e0 2024-03-27T21:41:47.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa182f95640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:47.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa182f95640 10 monclient(hunting): authenticate will time out at 948.531990s 2024-03-27T21:41:47.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa180d31640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa17c071a50 0x7fa17c13e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:48004/0 (socket says 172.21.15.60:48004) 2024-03-27T21:41:47.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa181532640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa17c0a87e0 0x7fa17c141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:47.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa180d31640 1 -- 172.21.15.60:0/2102797609 learned_addr learned my addr 172.21.15.60:0/2102797609 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:47.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa181532640 10 monclient(hunting): get_auth_request con 0x7fa17c0a87e0 auth_method 0 2024-03-27T21:41:47.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa181532640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:47.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa181532640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:47.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa181532640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:47.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa181532640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:47.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa181532640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:47.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa181532640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:47.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa181532640 10 monclient(hunting): handle_auth_done global_id 5151 payload 274 2024-03-27T21:41:47.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa181532640 1 -- 172.21.15.60:0/2102797609 >> v1:172.21.15.191:6789/0 conn(0x7fa17c13ef20 legacy=0x7fa17c141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:47.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa181532640 1 -- 172.21.15.60:0/2102797609 >> v1:172.21.15.60:6789/0 conn(0x7fa17c071a50 legacy=0x7fa17c13e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:47.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa181532640 10 monclient: _finish_hunting 0 2024-03-27T21:41:47.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa181532640 1 monclient: found mon.noname-a 2024-03-27T21:41:47.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa181532640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:47.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa17b7fe640 1 -- 172.21.15.60:0/2102797609 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789946867 0 0) 0x7fa17c0a8e90 con 0x7fa17c071a50 2024-03-27T21:41:47.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa181532640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:47.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa181532640 1 -- 172.21.15.60:0/2102797609 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c142280 con 0x7fa17c0a87e0 2024-03-27T21:41:47.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.311+0000 7fa17b7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:41:47.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa181532640 1 --2- 172.21.15.60:0/2102797609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa17c0a87e0 0x7fa17c141d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa174009d90 tx=0x7fa17402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1df182fd3e0f5a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:47.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa17b7fe640 1 -- 172.21.15.60:0/2102797609 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17402cc60 con 0x7fa17c0a87e0 2024-03-27T21:41:47.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa17b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:47.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa17b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:47.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa17b7fe640 10 monclient: dump: 2024-03-27T21:41:47.313 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:47.313 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:47.313 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:47.313 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:47.313 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:47.313 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:47.313 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:47.314 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:47.314 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:47.314 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:47.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa17b7fe640 10 monclient: _finish_auth 0 2024-03-27T21:41:47.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa17b7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:41:47.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa17b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:47.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa17b7fe640 1 -- 172.21.15.60:0/2102797609 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa17402cdc0 con 0x7fa17c0a87e0 2024-03-27T21:41:47.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa182f95640 5 monclient: authenticate success, global_id 5151 2024-03-27T21:41:47.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa182f95640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:47.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa17b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:47.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa17b7fe640 1 -- 172.21.15.60:0/2102797609 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa174034660 con 0x7fa17c0a87e0 2024-03-27T21:41:47.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa182f95640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:47.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa182f95640 4 set_mon_vals no callback set 2024-03-27T21:41:47.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa17b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:47.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa17b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:47.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa17b7fe640 10 monclient: dump: 2024-03-27T21:41:47.315 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:47.316 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:47.316 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:47.316 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:47.316 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:47.316 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:47.316 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:47.316 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:47.316 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:47.316 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:47.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa182f95640 10 monclient: shutdown 2024-03-27T21:41:47.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa182f95640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:47.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa182f95640 1 -- 172.21.15.60:0/2102797609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa17c0a87e0 msgr2=0x7fa17c141d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:47.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.312+0000 7fa182f95640 1 --2- 172.21.15.60:0/2102797609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa17c0a87e0 0x7fa17c141d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa174009d90 tx=0x7fa17402beb0 comp rx=0 tx=0).stop 2024-03-27T21:41:47.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.313+0000 7fa182f95640 1 -- 172.21.15.60:0/2102797609 shutdown_connections 2024-03-27T21:41:47.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.313+0000 7fa182f95640 1 --2- 172.21.15.60:0/2102797609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa17c0a87e0 0x7fa17c141d40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:47.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.313+0000 7fa182f95640 1 -- 172.21.15.60:0/2102797609 >> 172.21.15.60:0/2102797609 conn(0x7fa17c0b7300 msgr2=0x7fa17c0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:47.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.313+0000 7fa182f95640 1 -- 172.21.15.60:0/2102797609 shutdown_connections 2024-03-27T21:41:47.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.313+0000 7fa182f95640 1 -- 172.21.15.60:0/2102797609 wait complete. 2024-03-27T21:41:47.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.313+0000 7fa182f95640 10 monclient: build_initial_monmap 2024-03-27T21:41:47.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.313+0000 7fa182f95640 10 monclient: monmap: 2024-03-27T21:41:47.318 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:47.318 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:47.318 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:47.318 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:47.318 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:47.319 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:47.319 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:47.319 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:47.319 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:47.319 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:47.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.313+0000 7fa182f95640 1 librados: starting msgr at 2024-03-27T21:41:47.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.313+0000 7fa182f95640 1 librados: starting objecter 2024-03-27T21:41:47.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa182f95640 1 Processor -- start 2024-03-27T21:41:47.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa182f95640 1 -- start start 2024-03-27T21:41:47.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa182f95640 1 librados: setting wanted keys 2024-03-27T21:41:47.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa182f95640 1 librados: calling monclient init 2024-03-27T21:41:47.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa182f95640 10 monclient: init 2024-03-27T21:41:47.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa182f95640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:47.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa182f95640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:41:47.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa182f95640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa17c071a50 0x7fa17c138d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:47.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa182f95640 10 monclient(hunting): picked mon.noname-a con 0x7fa17c071a50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:47.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa182f95640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa17c0a87e0 0x7fa17c139260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:47.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa182f95640 10 monclient(hunting): picked mon.noname-c con 0x7fa17c0a87e0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:47.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa182f95640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa17c13ef20 0x7fa17c1366e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:47.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa182f95640 10 monclient(hunting): picked mon.noname-b con 0x7fa17c13ef20 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:47.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa182f95640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:47.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa182f95640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c139ab0 con 0x7fa17c071a50 2024-03-27T21:41:47.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa182f95640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:47.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa182f95640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c136290 con 0x7fa17c13ef20 2024-03-27T21:41:47.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa182f95640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:47.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa182f95640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c0b6c70 con 0x7fa17c0a87e0 2024-03-27T21:41:47.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa182f95640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:47.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa182f95640 10 monclient(hunting): authenticate will time out at 948.535745s 2024-03-27T21:41:47.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa180d31640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa17c071a50 0x7fa17c138d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:47.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa180d31640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa17c071a50 0x7fa17c138d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41822/0 (socket says 172.21.15.60:41822) 2024-03-27T21:41:47.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa180d31640 1 -- 172.21.15.60:0/2847044857 learned_addr learned my addr 172.21.15.60:0/2847044857 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:47.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa180d31640 10 monclient(hunting): get_auth_request con 0x7fa17c071a50 auth_method 0 2024-03-27T21:41:47.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa180d31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:47.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa180d31640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:47.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa180d31640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:47.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa17bfff640 1 --2- 172.21.15.60:0/2847044857 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa17c0a87e0 0x7fa17c139260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:47.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.314+0000 7fa17bfff640 10 monclient(hunting): get_auth_request con 0x7fa17c0a87e0 auth_method 0 2024-03-27T21:41:47.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa17bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:47.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa17bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:47.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa17bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:47.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa181532640 1 --2- 172.21.15.60:0/2847044857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa17c13ef20 0x7fa17c1366e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:47.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa180d31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:47.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa180d31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:47.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa180d31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:47.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa181532640 10 monclient(hunting): get_auth_request con 0x7fa17c13ef20 auth_method 0 2024-03-27T21:41:47.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa181532640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:47.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa181532640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:47.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa181532640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:47.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa17bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:47.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa17bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:47.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa17bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:47.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa181532640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:47.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa181532640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:47.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa181532640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:47.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa180d31640 10 monclient(hunting): handle_auth_done global_id 5157 payload 824 2024-03-27T21:41:47.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa180d31640 1 -- 172.21.15.60:0/2847044857 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa17c0a87e0 msgr2=0x7fa17c139260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:47.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa180d31640 1 --2- 172.21.15.60:0/2847044857 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa17c0a87e0 0x7fa17c139260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:47.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa180d31640 1 -- 172.21.15.60:0/2847044857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa17c13ef20 msgr2=0x7fa17c1366e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:47.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa180d31640 1 --2- 172.21.15.60:0/2847044857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa17c13ef20 0x7fa17c1366e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:47.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa180d31640 10 monclient: _finish_hunting 0 2024-03-27T21:41:47.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa180d31640 1 monclient: found mon.noname-a 2024-03-27T21:41:47.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa180d31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:47.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa180d31640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:47.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa180d31640 1 -- 172.21.15.60:0/2847044857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa174009a70 con 0x7fa17c071a50 2024-03-27T21:41:47.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa180d31640 1 --2- 172.21.15.60:0/2847044857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa17c071a50 0x7fa17c138d20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa17000c4d0 tx=0x7fa17000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:47.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa1797fa640 10 client.?.objecter ms_handle_connect 0x7fa17c071a50 2024-03-27T21:41:47.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa1797fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:47.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa1797fa640 1 -- 172.21.15.60:0/2847044857 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1700074e0 con 0x7fa17c071a50 2024-03-27T21:41:47.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa1797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:47.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa1797fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:47.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa1797fa640 10 monclient: dump: 2024-03-27T21:41:47.329 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:47.329 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:47.329 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:47.329 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:47.329 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:47.329 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:47.329 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:47.330 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:47.330 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:47.330 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:47.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa1797fa640 10 monclient: _finish_auth 0 2024-03-27T21:41:47.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa1797fa640 10 monclient: _check_auth_tickets 2024-03-27T21:41:47.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa1797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:47.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa182f95640 5 monclient: authenticate success, global_id 5157 2024-03-27T21:41:47.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa1797fa640 1 -- 172.21.15.60:0/2847044857 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa170007660 con 0x7fa17c071a50 2024-03-27T21:41:47.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa182f95640 10 monclient: _renew_subs 2024-03-27T21:41:47.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa182f95640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:47.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa182f95640 1 -- 172.21.15.60:0/2847044857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17c13af10 con 0x7fa17c071a50 2024-03-27T21:41:47.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa1797fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:47.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa1797fa640 1 -- 172.21.15.60:0/2847044857 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa170004a20 con 0x7fa17c071a50 2024-03-27T21:41:47.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa1797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:47.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa1797fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:47.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa1797fa640 10 monclient: dump: 2024-03-27T21:41:47.331 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:47.332 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:47.332 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:47.332 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:47.332 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:47.332 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:47.332 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:47.332 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:47.332 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:47.332 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:47.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa182f95640 10 client.5157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:47.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa182f95640 10 monclient: _renew_subs 2024-03-27T21:41:47.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa17a7fc640 4 set_mon_vals no callback set 2024-03-27T21:41:47.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa182f95640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:47.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa182f95640 1 -- 172.21.15.60:0/2847044857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17c136e00 con 0x7fa17c071a50 2024-03-27T21:41:47.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.315+0000 7fa182f95640 1 librados: init done 2024-03-27T21:41:47.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.316+0000 7fa182f95640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:47.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.316+0000 7fa182f95640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:47.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.316+0000 7fa182f95640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:47.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.316+0000 7fa182f95640 1 -- 172.21.15.60:0/2847044857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa17c136400 con 0x7fa17c071a50 2024-03-27T21:41:47.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.317+0000 7fa1797fa640 1 -- 172.21.15.60:0/2847044857 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa170004b80 con 0x7fa17c071a50 2024-03-27T21:41:47.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.317+0000 7fa1797fa640 1 --2- 172.21.15.60:0/2847044857 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa16006c590 0x7fa16006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:47.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.317+0000 7fa17bfff640 1 --2- 172.21.15.60:0/2847044857 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa16006c590 0x7fa16006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:47.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.317+0000 7fa1797fa640 1 -- 172.21.15.60:0/2847044857 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fa170088540 con 0x7fa17c071a50 2024-03-27T21:41:47.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.317+0000 7fa1797fa640 10 client.5157.objecter ms_dispatch 0x7fa17c000b90 osd_map(75..75 src has 1..75) v4 2024-03-27T21:41:47.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.317+0000 7fa1797fa640 3 client.5157.objecter handle_osd_map got epochs [75,75] > 0 2024-03-27T21:41:47.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.317+0000 7fa1797fa640 3 client.5157.objecter handle_osd_map decoding full epoch 75 2024-03-27T21:41:47.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.317+0000 7fa17bfff640 10 monclient: get_auth_request con 0x7fa16006c590 auth_method 0 2024-03-27T21:41:47.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.317+0000 7fa1797fa640 20 client.5157.objecter dump_active .. 0 homeless 2024-03-27T21:41:47.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.318+0000 7fa17bfff640 1 --2- 172.21.15.60:0/2847044857 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa16006c590 0x7fa16006ea50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa17c074270 tx=0x7fa168009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:47.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.318+0000 7fa1797fa640 10 client.5157.objecter ms_handle_connect 0x7fa16006c590 2024-03-27T21:41:47.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.321+0000 7fa1797fa640 1 -- 172.21.15.60:0/2847044857 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa17c136400 con 0x7fa17c071a50 2024-03-27T21:41:47.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.321+0000 7fa1797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:47.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.321+0000 7fa1797fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:47.404 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:47.404 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:47.413 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-27T21:41:47.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.445+0000 7fa4fd6cf640 10 start_mon_command cmd=[{"prefix": "osd in", "ids": ["1"]}] 2024-03-27T21:41:47.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.445+0000 7fa4fd6cf640 10 monclient: _send_command 2 [{"prefix": "osd in", "ids": ["1"]}] 2024-03-27T21:41:47.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.445+0000 7fa4fd6cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:47.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.445+0000 7fa4fd6cf640 1 -- 172.21.15.60:0/1215651908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd in", "ids": ["1"]} v 0) v1 -- 0x7fa4f8069e40 con 0x7fa4f80b40b0 2024-03-27T21:41:47.432 INFO:tasks.ceph.mon.a.smithi060.stderr:2024-03-27T21:41:47.446+0000 7ff34bd31640 -1 mon.a@0(leader).osd e75 definitely_dead 0 2024-03-27T21:41:47.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.512+0000 7fa182f95640 10 start_mon_command cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:41:47.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.512+0000 7fa182f95640 10 monclient: _send_command 2 [{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:41:47.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.512+0000 7fa182f95640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:47.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:47.512+0000 7fa182f95640 1 -- 172.21.15.60:0/2847044857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fa17c136400 con 0x7fa17c071a50 2024-03-27T21:41:47.518 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:47.518 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:47.531 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-27T21:41:47.635 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:47.636 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:47.644 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-27T21:41:47.752 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:47.752 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:47.760 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-27T21:41:47.866 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:47.866 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:47.875 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-27T21:41:47.981 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-03-27T21:41:47.981 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-27T21:41:48.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.190+0000 7fa4ef7fe640 1 -- 172.21.15.60:0/1215651908 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd in", "ids": ["1"]}]=0 marked in osd.1. v76) v1 ==== 85+0+0 (secure 0 0 0) 0x7fa4e807acc0 con 0x7fa4f80b40b0 2024-03-27T21:41:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.190+0000 7fa1797fa640 1 -- 172.21.15.60:0/2847044857 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v76) v1 ==== 101+0+0 (secure 0 0 0) 0x7fa17c136400 con 0x7fa17c071a50 2024-03-27T21:41:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:marked in osd.1. 2024-03-27T21:41:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:nodeep-scrub is unset 2024-03-27T21:41:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.190+0000 7fa1797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd unset", "key": "nodeep-scrub"}] 2024-03-27T21:41:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.190+0000 7fa1797fa640 10 monclient: _finish_command 2 = system:0 nodeep-scrub is unset 2024-03-27T21:41:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.190+0000 7fa4ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd in", "ids": ["1"]}] 2024-03-27T21:41:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.190+0000 7fa4ef7fe640 10 monclient: _finish_command 2 = system:0 marked in osd.1. 2024-03-27T21:41:48.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.193+0000 7fa4fd6cf640 10 librados: watch_flush enter 2024-03-27T21:41:48.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.193+0000 7fa4fd6cf640 10 librados: watch_flush exit 2024-03-27T21:41:48.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.193+0000 7fa4fd6cf640 20 client.5148.objecter shutdown clearing up homeless session... 2024-03-27T21:41:48.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.193+0000 7fa4fd6cf640 10 client.5148.objecter successfully canceled tick 2024-03-27T21:41:48.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.193+0000 7fa4fd6cf640 1 -- 172.21.15.60:0/1215651908 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa4d806c590 msgr2=0x7fa4d806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa4fd6cf640 1 --2- 172.21.15.60:0/1215651908 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa4d806c590 0x7fa4d806ea50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa4f813ae50 tx=0x7fa4e0009190 comp rx=0 tx=0).stop 2024-03-27T21:41:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa4fd6cf640 10 monclient: shutdown 2024-03-27T21:41:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa4fd6cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa4fd6cf640 1 -- 172.21.15.60:0/1215651908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4f80b40b0 msgr2=0x7fa4f80aa9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa4fd6cf640 1 --2- 172.21.15.60:0/1215651908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4f80b40b0 0x7fa4f80aa9d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa4e802b9e0 tx=0x7fa4e8004730 comp rx=0 tx=0).stop 2024-03-27T21:41:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa4fd6cf640 1 -- 172.21.15.60:0/1215651908 shutdown_connections 2024-03-27T21:41:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa4fd6cf640 1 --2- 172.21.15.60:0/1215651908 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa4f81416f0 0x7fa4f80afc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa4fd6cf640 1 --2- 172.21.15.60:0/1215651908 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa4d806c590 0x7fa4d806ea50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa4fd6cf640 1 --2- 172.21.15.60:0/1215651908 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4f813b020 0x7fa4f80aaf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa4fd6cf640 1 --2- 172.21.15.60:0/1215651908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4f80b40b0 0x7fa4f80aa9d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa4fd6cf640 1 -- 172.21.15.60:0/1215651908 >> 172.21.15.60:0/1215651908 conn(0x7fa4f80608e0 msgr2=0x7fa4f813f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:48.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa4fd6cf640 1 -- 172.21.15.60:0/1215651908 shutdown_connections 2024-03-27T21:41:48.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa182f95640 10 librados: watch_flush enter 2024-03-27T21:41:48.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa4fd6cf640 1 -- 172.21.15.60:0/1215651908 wait complete. 2024-03-27T21:41:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa182f95640 10 librados: watch_flush exit 2024-03-27T21:41:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa182f95640 20 client.5157.objecter shutdown clearing up homeless session... 2024-03-27T21:41:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa182f95640 10 client.5157.objecter successfully canceled tick 2024-03-27T21:41:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa4fd6cf640 1 librados: shutdown 2024-03-27T21:41:48.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa182f95640 1 -- 172.21.15.60:0/2847044857 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa16006c590 msgr2=0x7fa16006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa182f95640 1 --2- 172.21.15.60:0/2847044857 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa16006c590 0x7fa16006ea50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa17c074270 tx=0x7fa168009060 comp rx=0 tx=0).stop 2024-03-27T21:41:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa182f95640 10 monclient: shutdown 2024-03-27T21:41:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa182f95640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa182f95640 1 -- 172.21.15.60:0/2847044857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa17c071a50 msgr2=0x7fa17c138d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa182f95640 1 --2- 172.21.15.60:0/2847044857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa17c071a50 0x7fa17c138d20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa17000c4d0 tx=0x7fa17000c990 comp rx=0 tx=0).stop 2024-03-27T21:41:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa182f95640 1 -- 172.21.15.60:0/2847044857 shutdown_connections 2024-03-27T21:41:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa182f95640 1 --2- 172.21.15.60:0/2847044857 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa16006c590 0x7fa16006ea50 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa182f95640 1 --2- 172.21.15.60:0/2847044857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa17c13ef20 0x7fa17c1366e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa182f95640 1 --2- 172.21.15.60:0/2847044857 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa17c0a87e0 0x7fa17c139260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa182f95640 1 --2- 172.21.15.60:0/2847044857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa17c071a50 0x7fa17c138d20 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa182f95640 1 -- 172.21.15.60:0/2847044857 >> 172.21.15.60:0/2847044857 conn(0x7fa17c0b7300 msgr2=0x7fa17c135820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa182f95640 1 -- 172.21.15.60:0/2847044857 shutdown_connections 2024-03-27T21:41:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa182f95640 1 -- 172.21.15.60:0/2847044857 wait complete. 2024-03-27T21:41:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.194+0000 7fa182f95640 1 librados: shutdown 2024-03-27T21:41:48.192 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-03-27T21:41:48.192 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-27T21:41:48.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.345+0000 7f7464070640 10 monclient: get_monmap_and_config 2024-03-27T21:41:48.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.345+0000 7f7464070640 10 monclient: build_initial_monmap 2024-03-27T21:41:48.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.345+0000 7f7464070640 10 monclient: monmap: 2024-03-27T21:41:48.332 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:48.332 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:48.332 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:48.346679+0000 2024-03-27T21:41:48.332 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:48.346679+0000 2024-03-27T21:41:48.332 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:48.332 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:48.333 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:48.333 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:48.333 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:48.333 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:48.333 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:48.333 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:48.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.346+0000 7f7464070640 1 Processor -- start 2024-03-27T21:41:48.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.346+0000 7f7464070640 1 -- start start 2024-03-27T21:41:48.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.346+0000 7f7464070640 10 monclient: init 2024-03-27T21:41:48.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.346+0000 7f7464070640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:48.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.346+0000 7f7464070640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-03-27T21:41:48.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.346+0000 7f7464070640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f745c0b40b0 0x7f745c0a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:48.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f7464070640 10 monclient(hunting): picked mon.noname-c con 0x7f745c0b40b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:48.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f7464070640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c13b020 0x7f745c13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:48.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f7464070640 10 monclient(hunting): picked mon.noname-b con 0x7f745c13b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:48.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f7464070640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f745c1416f0 0x7f745c141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:48.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f7464070640 10 monclient(hunting): picked mon.noname-a con 0x7f745c1416f0 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:48.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f7464070640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:48.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f7464070640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f745c0b4490 con 0x7f745c1416f0 2024-03-27T21:41:48.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f7464070640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:48.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f7464070640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f745c0b3830 con 0x7f745c0b40b0 2024-03-27T21:41:48.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f7464070640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:48.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f7464070640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745c0aa520 con 0x7f745c13b020 2024-03-27T21:41:48.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f7464070640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:48.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f7464070640 10 monclient(hunting): authenticate will time out at 949.567966s 2024-03-27T21:41:48.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746160b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c13b020 0x7f745c13b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:48.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746260d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f745c1416f0 0x7f745c141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:48.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746260d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f745c1416f0 0x7f745c141b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41832/0 (socket says 172.21.15.60:41832) 2024-03-27T21:41:48.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746160b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c13b020 0x7f745c13b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:32828/0 (socket says 172.21.15.60:32828) 2024-03-27T21:41:48.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746260d640 1 -- 172.21.15.60:0/1100938829 learned_addr learned my addr 172.21.15.60:0/1100938829 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:48.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746260d640 10 monclient(hunting): get_auth_request con 0x7f745c1416f0 auth_method 0 2024-03-27T21:41:48.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746260d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:48.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746260d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:48.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746260d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:48.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746160b640 10 monclient(hunting): get_auth_request con 0x7f745c13b020 auth_method 0 2024-03-27T21:41:48.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746160b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:48.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746160b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:48.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746160b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:48.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746160b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:48.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746160b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:48.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746160b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:48.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:48.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:48.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:48.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746160b640 10 monclient(hunting): handle_auth_done global_id 4904 payload 274 2024-03-27T21:41:48.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746160b640 1 -- 172.21.15.60:0/1100938829 >> v2:172.21.15.191:3300/0 conn(0x7f745c0b40b0 msgr2=0x7f745c0a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:48.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746160b640 1 --2- 172.21.15.60:0/1100938829 >> v2:172.21.15.191:3300/0 conn(0x7f745c0b40b0 0x7f745c0a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746160b640 1 -- 172.21.15.60:0/1100938829 >> v2:172.21.15.60:3300/0 conn(0x7f745c1416f0 msgr2=0x7f745c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:48.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746160b640 1 --2- 172.21.15.60:0/1100938829 >> v2:172.21.15.60:3300/0 conn(0x7f745c1416f0 0x7f745c141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746160b640 10 monclient: _finish_hunting 0 2024-03-27T21:41:48.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746160b640 1 monclient: found mon.noname-b 2024-03-27T21:41:48.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746160b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:48.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746160b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:48.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746160b640 1 -- 172.21.15.60:0/1100938829 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745c0a9470 con 0x7f745c13b020 2024-03-27T21:41:48.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.347+0000 7f746260d640 1 --2- 172.21.15.60:0/1100938829 >> v2:172.21.15.60:3300/0 conn(0x7f745c1416f0 0x7f745c141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:48.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f746160b640 1 --2- 172.21.15.60:0/1100938829 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c13b020 0x7f745c13b480 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f745c0b4810 tx=0x7f744c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14ef45ed7df31647 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:48.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7460e0a640 1 -- 172.21.15.60:0/1100938829 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c02e070 con 0x7f745c13b020 2024-03-27T21:41:48.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7460e0a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:48.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7460e0a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:48.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7460e0a640 10 monclient: dump: 2024-03-27T21:41:48.351 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:48.351 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:48.351 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.351 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.351 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:48.351 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:48.351 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:48.352 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:48.352 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:48.352 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:48.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7460e0a640 10 monclient: _finish_auth 0 2024-03-27T21:41:48.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7460e0a640 10 monclient: _check_auth_tickets 2024-03-27T21:41:48.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7460e0a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:48.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7464070640 5 monclient: authenticate success, global_id 4904 2024-03-27T21:41:48.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7464070640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:48.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7460e0a640 1 -- 172.21.15.60:0/1100938829 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f744c0315a0 con 0x7f745c13b020 2024-03-27T21:41:48.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7460e0a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:48.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7464070640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:48.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7464070640 4 set_mon_vals no callback set 2024-03-27T21:41:48.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7460e0a640 1 -- 172.21.15.60:0/1100938829 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c031930 con 0x7f745c13b020 2024-03-27T21:41:48.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7460e0a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:48.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7460e0a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:48.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7460e0a640 10 monclient: dump: 2024-03-27T21:41:48.353 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:48.354 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:48.354 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.354 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.354 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:48.354 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:48.354 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:48.354 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:48.354 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:48.355 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:48.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7464070640 10 monclient: shutdown 2024-03-27T21:41:48.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7464070640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:48.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7464070640 1 -- 172.21.15.60:0/1100938829 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c13b020 msgr2=0x7f745c13b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:48.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7464070640 1 --2- 172.21.15.60:0/1100938829 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c13b020 0x7f745c13b480 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f745c0b4810 tx=0x7f744c02b840 comp rx=0 tx=0).stop 2024-03-27T21:41:48.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7464070640 1 -- 172.21.15.60:0/1100938829 shutdown_connections 2024-03-27T21:41:48.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7464070640 1 --2- 172.21.15.60:0/1100938829 >> v2:172.21.15.60:3300/0 conn(0x7f745c1416f0 0x7f745c141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7464070640 1 --2- 172.21.15.60:0/1100938829 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c13b020 0x7f745c13b480 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7464070640 1 --2- 172.21.15.60:0/1100938829 >> v2:172.21.15.191:3300/0 conn(0x7f745c0b40b0 0x7f745c0a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7464070640 1 -- 172.21.15.60:0/1100938829 >> 172.21.15.60:0/1100938829 conn(0x7f745c0608e0 msgr2=0x7f745c060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:48.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7464070640 1 -- 172.21.15.60:0/1100938829 shutdown_connections 2024-03-27T21:41:48.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.348+0000 7f7464070640 1 -- 172.21.15.60:0/1100938829 wait complete. 2024-03-27T21:41:48.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.349+0000 7f7464070640 10 monclient: build_initial_monmap 2024-03-27T21:41:48.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.349+0000 7f7464070640 10 monclient: monmap: 2024-03-27T21:41:48.357 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:48.357 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:48.357 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:48.357 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:48.357 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:48.357 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:48.357 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:48.358 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:48.358 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:48.358 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:48.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.349+0000 7f7464070640 1 librados: starting msgr at 2024-03-27T21:41:48.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.349+0000 7f7464070640 1 librados: starting objecter 2024-03-27T21:41:48.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.349+0000 7f7464070640 1 Processor -- start 2024-03-27T21:41:48.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.349+0000 7f7464070640 1 -- start start 2024-03-27T21:41:48.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.349+0000 7f7464070640 1 librados: setting wanted keys 2024-03-27T21:41:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.349+0000 7f7464070640 1 librados: calling monclient init 2024-03-27T21:41:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.349+0000 7f7464070640 10 monclient: init 2024-03-27T21:41:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7464070640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7464070640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:41:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7464070640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c0b40b0 0x7f745c13d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7464070640 10 monclient(hunting): picked mon.noname-c con 0x7f745c0b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7464070640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f745c13b020 0x7f745c0ac4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7464070640 10 monclient(hunting): picked mon.noname-b con 0x7f745c13b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7464070640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f745c1416f0 0x7f745c1d9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7464070640 10 monclient(hunting): picked mon.noname-a con 0x7f745c1416f0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7464070640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7464070640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c0ad550 con 0x7f745c1416f0 2024-03-27T21:41:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7464070640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7464070640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c0acd80 con 0x7f745c13b020 2024-03-27T21:41:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7464070640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7464070640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745c0acab0 con 0x7f745c0b40b0 2024-03-27T21:41:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7464070640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:48.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7464070640 10 monclient(hunting): authenticate will time out at 949.571450s 2024-03-27T21:41:48.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7461e0c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c0b40b0 0x7f745c13d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:48.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f746260d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f745c1416f0 0x7f745c1d9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:48.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7461e0c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c0b40b0 0x7f745c13d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:32836/0 (socket says 172.21.15.60:32836) 2024-03-27T21:41:48.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7461e0c640 1 -- 172.21.15.60:0/2249519992 learned_addr learned my addr 172.21.15.60:0/2249519992 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:48.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7461e0c640 10 monclient(hunting): get_auth_request con 0x7f745c0b40b0 auth_method 0 2024-03-27T21:41:48.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7461e0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:48.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7461e0c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:48.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7461e0c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:48.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f746260d640 10 monclient(hunting): get_auth_request con 0x7f745c1416f0 auth_method 0 2024-03-27T21:41:48.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f746260d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:48.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f746260d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:48.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f746260d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:48.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7461e0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:48.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7461e0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:48.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7461e0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:48.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:48.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:48.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:48.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f746160b640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f745c13b020 0x7f745c0ac4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:48.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f746160b640 10 monclient(hunting): get_auth_request con 0x7f745c13b020 auth_method 0 2024-03-27T21:41:48.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f746160b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:48.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f746160b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:48.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f746160b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:48.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7461e0c640 10 monclient(hunting): handle_auth_done global_id 4907 payload 824 2024-03-27T21:41:48.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7461e0c640 1 -- 172.21.15.60:0/2249519992 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f745c13b020 msgr2=0x7f745c0ac4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:48.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7461e0c640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f745c13b020 0x7f745c0ac4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7461e0c640 1 -- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f745c1416f0 msgr2=0x7f745c1d9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:48.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7461e0c640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f745c1416f0 0x7f745c1d9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7461e0c640 10 monclient: _finish_hunting 0 2024-03-27T21:41:48.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7461e0c640 1 monclient: found mon.noname-c 2024-03-27T21:41:48.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7461e0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:48.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7461e0c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:48.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f7461e0c640 1 -- 172.21.15.60:0/2249519992 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7454009b30 con 0x7f745c0b40b0 2024-03-27T21:41:48.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.350+0000 7f746260d640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f745c1416f0 0x7f745c1d9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:48.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f7461e0c640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c0b40b0 0x7f745c13d840 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f745800b4a0 tx=0x7f745800b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:48.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f74527fc640 10 client.?.objecter ms_handle_connect 0x7f745c0b40b0 2024-03-27T21:41:48.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f74527fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:48.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7458010610 con 0x7f745c0b40b0 2024-03-27T21:41:48.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f74527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:48.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f74527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:48.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f74527fc640 10 monclient: dump: 2024-03-27T21:41:48.366 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:48.366 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:48.366 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.366 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.366 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:48.367 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:48.367 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:48.367 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:48.367 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:48.367 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:48.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f74527fc640 10 monclient: _finish_auth 0 2024-03-27T21:41:48.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f74527fc640 10 monclient: _check_auth_tickets 2024-03-27T21:41:48.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f74527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:48.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f7464070640 5 monclient: authenticate success, global_id 4907 2024-03-27T21:41:48.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7458010770 con 0x7f745c0b40b0 2024-03-27T21:41:48.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f7464070640 10 monclient: _renew_subs 2024-03-27T21:41:48.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f7464070640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:48.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f7464070640 1 -- 172.21.15.60:0/2249519992 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f744c009400 con 0x7f745c0b40b0 2024-03-27T21:41:48.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f74527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:48.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7458010b20 con 0x7f745c0b40b0 2024-03-27T21:41:48.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f74527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:48.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f74527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:48.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f74527fc640 10 monclient: dump: 2024-03-27T21:41:48.369 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:48.369 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:48.369 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.369 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.369 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:48.369 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:48.369 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:48.369 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:48.370 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:48.370 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:48.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f7464070640 10 client.4907.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:48.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f7453fff640 4 set_mon_vals no callback set 2024-03-27T21:41:48.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f7464070640 10 monclient: _renew_subs 2024-03-27T21:41:48.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f7464070640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:48.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f7464070640 1 -- 172.21.15.60:0/2249519992 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f745c1d9f60 con 0x7f745c0b40b0 2024-03-27T21:41:48.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f7464070640 1 librados: init done 2024-03-27T21:41:48.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f7464070640 10 start_mon_command cmd=[{"prefix": "mon dump", "format": "json"}] 2024-03-27T21:41:48.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f7464070640 10 monclient: _send_command 1 [{"prefix": "mon dump", "format": "json"}] 2024-03-27T21:41:48.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f7464070640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:48.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.351+0000 7f7464070640 1 -- 172.21.15.60:0/2249519992 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f745c069470 con 0x7f745c0b40b0 2024-03-27T21:41:48.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.352+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7458010c80 con 0x7f745c0b40b0 2024-03-27T21:41:48.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.352+0000 7f74527fc640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f744006c5c0 0x7f744006ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:48.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f746160b640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f744006c5c0 0x7f744006ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:48.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f7458088ce0 con 0x7f745c0b40b0 2024-03-27T21:41:48.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f74527fc640 10 client.4907.objecter ms_dispatch 0x7f745c000b90 osd_map(76..76 src has 1..76) v4 2024-03-27T21:41:48.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f74527fc640 3 client.4907.objecter handle_osd_map got epochs [76,76] > 0 2024-03-27T21:41:48.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f74527fc640 3 client.4907.objecter handle_osd_map decoding full epoch 76 2024-03-27T21:41:48.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f746160b640 10 monclient: get_auth_request con 0x7f744006c5c0 auth_method 0 2024-03-27T21:41:48.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f74527fc640 20 client.4907.objecter dump_active .. 0 homeless 2024-03-27T21:41:48.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+1272 (secure 0 0 0) 0x7f745808a5c0 con 0x7f745c0b40b0 2024-03-27T21:41:48.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f74527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "mon dump", "format": "json"}] 2024-03-27T21:41:48.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f74527fc640 10 monclient: _finish_command 1 = system:0 dumped monmap epoch 1 2024-03-27T21:41:48.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f7464070640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:48.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f7464070640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f745c1da1b0 0x7f745c1d6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:48.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f7464070640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:48.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f7464070640 1 -- 172.21.15.60:0/2249519992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c0acec0 con 0x7f745c1da1b0 2024-03-27T21:41:48.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f74527fc640 10 client.4907.objecter ms_handle_connect 0x7f744006c5c0 2024-03-27T21:41:48.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f746160b640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f744006c5c0 0x7f744006ea80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f744c036000 tx=0x7f744c03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:48.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f746260d640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f745c1da1b0 0x7f745c1d6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:48.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f746260d640 10 monclient: get_auth_request con 0x7f745c1da1b0 auth_method 0 2024-03-27T21:41:48.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f746260d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:48.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f746260d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:48.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f746260d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:48.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:48.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:48.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:48.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f746260d640 10 monclient(hunting): handle_auth_done global_id 5166 payload 824 2024-03-27T21:41:48.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f746260d640 1 -- 172.21.15.60:0/2249519992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f745c0abe10 con 0x7f745c1da1b0 2024-03-27T21:41:48.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f746260d640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f745c1da1b0 0x7f745c1d6320 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7454004950 tx=0x7f745400e8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:48.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f74527fc640 10 client.4907.objecter ms_handle_connect 0x7f745c1da1b0 2024-03-27T21:41:48.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.353+0000 7f74527fc640 10 client.4907.objecter resend_mon_ops 2024-03-27T21:41:48.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.354+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7454015650 con 0x7f745c1da1b0 2024-03-27T21:41:48.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.354+0000 7f74527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:48.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.354+0000 7f74527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:48.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.354+0000 7f74527fc640 10 monclient: dump: 2024-03-27T21:41:48.375 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:48.375 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:48.375 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.376 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.376 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:48.376 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:48.376 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:48.376 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:48.376 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:48.376 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:48.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.354+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 <== mon.0 v2:172.21.15.60:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f745400f040 con 0x7f745c1da1b0 2024-03-27T21:41:48.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.354+0000 7f74527fc640 10 monclient: handle_command_reply 2 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:48.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.354+0000 7f74527fc640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:41:48.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.354+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f745c1da1b0 msgr2=0x7f745c1d6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:48.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.354+0000 7f74527fc640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f745c1da1b0 0x7f745c1d6320 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7454004950 tx=0x7f745400e8c0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.354+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f745c1da1b0 msgr2=0x7f745c1d6320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-27T21:41:48.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.354+0000 7f74527fc640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f745c1da1b0 0x7f745c1d6320 secure :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7454004950 tx=0x7f745400e8c0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.360+0000 7f7464070640 10 start_mon_command cmd=[{"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}] 2024-03-27T21:41:48.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.360+0000 7f7464070640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f745c1d6860 0x7f745c1d6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:48.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.360+0000 7f7464070640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:48.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.360+0000 7f7464070640 1 -- 172.21.15.60:0/2249519992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c073e90 con 0x7f745c1d6860 2024-03-27T21:41:48.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.360+0000 7f746260d640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f745c1d6860 0x7f745c1d6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:48.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.360+0000 7f746260d640 10 monclient: get_auth_request con 0x7f745c1d6860 auth_method 0 2024-03-27T21:41:48.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.360+0000 7f746260d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:48.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.360+0000 7f746260d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:48.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.360+0000 7f746260d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:48.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.360+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:48.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.360+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:48.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.360+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:48.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.360+0000 7f746260d640 10 monclient(hunting): handle_auth_done global_id 5169 payload 824 2024-03-27T21:41:48.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.360+0000 7f746260d640 1 -- 172.21.15.60:0/2249519992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7f745c144990 con 0x7f745c1d6860 2024-03-27T21:41:48.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.360+0000 7f746260d640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f745c1d6860 0x7f745c1d6c40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f745400efd0 tx=0x7f7454004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:48.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.360+0000 7f74527fc640 10 client.4907.objecter ms_handle_connect 0x7f745c1d6860 2024-03-27T21:41:48.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.360+0000 7f74527fc640 10 client.4907.objecter resend_mon_ops 2024-03-27T21:41:48.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.361+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7454025070 con 0x7f745c1d6860 2024-03-27T21:41:48.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.361+0000 7f74527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:48.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.361+0000 7f74527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:48.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.361+0000 7f74527fc640 10 monclient: dump: 2024-03-27T21:41:48.381 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:48.382 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:48.382 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.382 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.382 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:48.382 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:48.382 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:48.382 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:48.382 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:48.382 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:48.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.362+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 <== mon.0 v2:172.21.15.60:3300/0 2 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (secure 0 0 0) 0x7f7454020b60 con 0x7f745c1d6860 2024-03-27T21:41:48.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.362+0000 7f74527fc640 10 monclient: handle_command_reply 3 [{"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}] 2024-03-27T21:41:48.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.362+0000 7f74527fc640 10 monclient: _finish_command 3 = system:0 mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-27T21:41:48.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.362+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f745c1d6860 msgr2=0x7f745c1d6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:48.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.362+0000 7f74527fc640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f745c1d6860 0x7f745c1d6c40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f745400efd0 tx=0x7f7454004a40 comp rx=0 tx=0).stop 2024-03-27T21:41:48.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.362+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f745c1d6860 msgr2=0x7f745c1d6c40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-27T21:41:48.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.362+0000 7f74527fc640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f745c1d6860 0x7f745c1d6c40 secure :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f745400efd0 tx=0x7f7454004a40 comp rx=0 tx=0).stop 2024-03-27T21:41:48.384 INFO:teuthology.orchestra.run.smithi060.stdout:mon.a: {} 2024-03-27T21:41:48.387 INFO:teuthology.orchestra.run.smithi060.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-27T21:41:48.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.362+0000 7f7464070640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:48.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.362+0000 7f7464070640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f745c137dc0 0x7f745c1381d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:48.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.362+0000 7f7464070640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:48.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.362+0000 7f7464070640 1 -- 172.21.15.60:0/2249519992 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c1d7180 con 0x7f745c137dc0 2024-03-27T21:41:48.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.362+0000 7f746260d640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f745c137dc0 0x7f745c1381d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:48.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.362+0000 7f746260d640 10 monclient: get_auth_request con 0x7f745c137dc0 auth_method 0 2024-03-27T21:41:48.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.362+0000 7f746260d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:48.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.362+0000 7f746260d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:48.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.362+0000 7f746260d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:48.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.362+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:48.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.362+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:48.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.362+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:48.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.363+0000 7f746260d640 10 monclient(hunting): handle_auth_done global_id 5101 payload 824 2024-03-27T21:41:48.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.363+0000 7f746260d640 1 -- 172.21.15.60:0/2249519992 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f745c069e40 con 0x7f745c137dc0 2024-03-27T21:41:48.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.363+0000 7f746260d640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f745c137dc0 0x7f745c1381d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f745400e900 tx=0x7f745400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:48.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.363+0000 7f74527fc640 10 client.4907.objecter ms_handle_connect 0x7f745c137dc0 2024-03-27T21:41:48.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.363+0000 7f74527fc640 10 client.4907.objecter resend_mon_ops 2024-03-27T21:41:48.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.363+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7454020d90 con 0x7f745c137dc0 2024-03-27T21:41:48.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.363+0000 7f74527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:48.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.363+0000 7f74527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:48.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.363+0000 7f74527fc640 10 monclient: dump: 2024-03-27T21:41:48.390 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:48.390 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:48.390 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.390 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.390 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:48.390 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:48.390 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:48.390 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:48.391 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:48.391 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:48.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.363+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f745403d410 con 0x7f745c137dc0 2024-03-27T21:41:48.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.363+0000 7f74527fc640 10 monclient: handle_command_reply 4 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:48.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.363+0000 7f74527fc640 10 monclient: _finish_command 4 = system:0 2024-03-27T21:41:48.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.363+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f745c137dc0 msgr2=0x7f745c1381d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:48.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.363+0000 7f74527fc640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f745c137dc0 0x7f745c1381d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f745400e900 tx=0x7f745400b040 comp rx=0 tx=0).stop 2024-03-27T21:41:48.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.363+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f745c137dc0 msgr2=0x7f745c1381d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-27T21:41:48.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.363+0000 7f74527fc640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f745c137dc0 0x7f745c1381d0 secure :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f745400e900 tx=0x7f745400b040 comp rx=0 tx=0).stop 2024-03-27T21:41:48.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.363+0000 7f746260d640 1 -- 172.21.15.60:0/2249519992 reap_dead start 2024-03-27T21:41:48.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.369+0000 7f7464070640 10 start_mon_command cmd=[{"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}] 2024-03-27T21:41:48.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.369+0000 7f7464070640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f745c1d9840 0x7f745c0ac4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:48.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.369+0000 7f7464070640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:48.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.369+0000 7f7464070640 1 -- 172.21.15.60:0/2249519992 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c1d9c20 con 0x7f745c1d9840 2024-03-27T21:41:48.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.370+0000 7f746260d640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f745c1d9840 0x7f745c0ac4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:48.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.370+0000 7f746260d640 10 monclient: get_auth_request con 0x7f745c1d9840 auth_method 0 2024-03-27T21:41:48.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.370+0000 7f746260d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:48.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.370+0000 7f746260d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:48.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.370+0000 7f746260d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:48.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.370+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:48.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.370+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:48.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.370+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:48.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.370+0000 7f746260d640 10 monclient(hunting): handle_auth_done global_id 5104 payload 824 2024-03-27T21:41:48.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.370+0000 7f746260d640 1 -- 172.21.15.60:0/2249519992 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7f745c0b8140 con 0x7f745c1d9840 2024-03-27T21:41:48.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.370+0000 7f746260d640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f745c1d9840 0x7f745c0ac4f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f745400f010 tx=0x7f7454026000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:48.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.370+0000 7f74527fc640 10 client.4907.objecter ms_handle_connect 0x7f745c1d9840 2024-03-27T21:41:48.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.370+0000 7f74527fc640 10 client.4907.objecter resend_mon_ops 2024-03-27T21:41:48.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.370+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745403dbc0 con 0x7f745c1d9840 2024-03-27T21:41:48.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.370+0000 7f74527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:48.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.370+0000 7f74527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:41:48.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.370+0000 7f74527fc640 10 monclient: dump: 2024-03-27T21:41:48.394 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:48.395 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:48.395 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.395 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.395 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:48.395 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:48.395 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:48.395 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:48.395 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:48.396 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:48.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (secure 0 0 0) 0x7f745403dde0 con 0x7f745c1d9840 2024-03-27T21:41:48.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f74527fc640 10 monclient: handle_command_reply 5 [{"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}] 2024-03-27T21:41:48.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f74527fc640 10 monclient: _finish_command 5 = system:0 mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-27T21:41:48.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f745c1d9840 msgr2=0x7f745c0ac4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:48.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f74527fc640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f745c1d9840 0x7f745c0ac4f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f745400f010 tx=0x7f7454026000 comp rx=0 tx=0).stop 2024-03-27T21:41:48.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f745c1d9840 msgr2=0x7f745c0ac4f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-27T21:41:48.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f74527fc640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f745c1d9840 0x7f745c0ac4f0 secure :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f745400f010 tx=0x7f7454026000 comp rx=0 tx=0).stop 2024-03-27T21:41:48.397 INFO:teuthology.orchestra.run.smithi060.stdout:mon.b: {} 2024-03-27T21:41:48.400 INFO:teuthology.orchestra.run.smithi060.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-27T21:41:48.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f7464070640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:48.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f7464070640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c138710 0x7f745c065a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:48.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f7464070640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:48.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f7464070640 1 -- 172.21.15.60:0/2249519992 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745c0acbf0 con 0x7f745c138710 2024-03-27T21:41:48.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f746260d640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c138710 0x7f745c065a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:48.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f746260d640 10 monclient: get_auth_request con 0x7f745c138710 auth_method 0 2024-03-27T21:41:48.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f746260d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:48.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f746260d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:48.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f746260d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:48.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:48.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:48.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:48.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f746260d640 10 monclient(hunting): handle_auth_done global_id 4910 payload 824 2024-03-27T21:41:48.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f746260d640 1 -- 172.21.15.60:0/2249519992 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f745c138b20 con 0x7f745c138710 2024-03-27T21:41:48.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f74527fc640 10 client.4907.objecter ms_handle_connect 0x7f745c138710 2024-03-27T21:41:48.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f74527fc640 10 client.4907.objecter resend_mon_ops 2024-03-27T21:41:48.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f746260d640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c138710 0x7f745c065a90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f74540090a0 tx=0x7f745402b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:48.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7454055070 con 0x7f745c138710 2024-03-27T21:41:48.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f74527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:48.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f74527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:48.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.372+0000 7f74527fc640 10 monclient: dump: 2024-03-27T21:41:48.402 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:48.402 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:48.403 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.403 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.403 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:48.403 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:48.403 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:48.403 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:48.403 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:48.403 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:48.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.373+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 <== mon.2 v2:172.21.15.60:3301/0 2 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f745404f650 con 0x7f745c138710 2024-03-27T21:41:48.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.373+0000 7f74527fc640 10 monclient: handle_command_reply 6 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:48.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.373+0000 7f74527fc640 10 monclient: _finish_command 6 = system:0 2024-03-27T21:41:48.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.373+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c138710 msgr2=0x7f745c065a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:48.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.373+0000 7f74527fc640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c138710 0x7f745c065a90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f74540090a0 tx=0x7f745402b000 comp rx=0 tx=0).stop 2024-03-27T21:41:48.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.373+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c138710 msgr2=0x7f745c065a90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-27T21:41:48.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.373+0000 7f74527fc640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c138710 0x7f745c065a90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.378+0000 7f7464070640 10 start_mon_command cmd=[{"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}] 2024-03-27T21:41:48.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.378+0000 7f7464070640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c065fd0 0x7f745c0663b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:48.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.378+0000 7f7464070640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:48.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.378+0000 7f7464070640 1 -- 172.21.15.60:0/2249519992 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745c1420c0 con 0x7f745c065fd0 2024-03-27T21:41:48.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.379+0000 7f746260d640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c065fd0 0x7f745c0663b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:48.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.379+0000 7f746260d640 10 monclient: get_auth_request con 0x7f745c065fd0 auth_method 0 2024-03-27T21:41:48.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.379+0000 7f746260d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:48.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.379+0000 7f746260d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:48.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.379+0000 7f746260d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:48.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.379+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:48.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.379+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:48.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.379+0000 7f746260d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:48.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.379+0000 7f746260d640 10 monclient(hunting): handle_auth_done global_id 4913 payload 824 2024-03-27T21:41:48.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.379+0000 7f746260d640 1 -- 172.21.15.60:0/2249519992 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7f745c142230 con 0x7f745c065fd0 2024-03-27T21:41:48.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.379+0000 7f74527fc640 10 client.4907.objecter ms_handle_connect 0x7f745c065fd0 2024-03-27T21:41:48.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.379+0000 7f74527fc640 10 client.4907.objecter resend_mon_ops 2024-03-27T21:41:48.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.379+0000 7f746260d640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c065fd0 0x7f745c0663b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f745402af40 tx=0x7f7454046000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:48.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.379+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745401e8f0 con 0x7f745c065fd0 2024-03-27T21:41:48.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.379+0000 7f74527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:48.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.379+0000 7f74527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:48.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.379+0000 7f74527fc640 10 monclient: dump: 2024-03-27T21:41:48.407 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:48.407 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:48.407 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.407 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.407 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:48.407 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:48.407 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:48.408 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:48.408 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:48.408 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:48.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.380+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 <== mon.2 v2:172.21.15.60:3301/0 2 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (secure 0 0 0) 0x7f745401e7b0 con 0x7f745c065fd0 2024-03-27T21:41:48.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.380+0000 7f74527fc640 10 monclient: handle_command_reply 7 [{"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}] 2024-03-27T21:41:48.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.380+0000 7f74527fc640 10 monclient: _finish_command 7 = system:0 mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-27T21:41:48.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.380+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c065fd0 msgr2=0x7f745c0663b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:48.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.380+0000 7f74527fc640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c065fd0 0x7f745c0663b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f745402af40 tx=0x7f7454046000 comp rx=0 tx=0).stop 2024-03-27T21:41:48.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.380+0000 7f74527fc640 1 -- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c065fd0 msgr2=0x7f745c0663b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-03-27T21:41:48.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.380+0000 7f74527fc640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c065fd0 0x7f745c0663b0 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f745402af40 tx=0x7f7454046000 comp rx=0 tx=0).stop 2024-03-27T21:41:48.409 INFO:teuthology.orchestra.run.smithi060.stdout:mon.c: {} 2024-03-27T21:41:48.411 INFO:teuthology.orchestra.run.smithi060.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-27T21:41:48.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.380+0000 7f7464070640 10 librados: watch_flush enter 2024-03-27T21:41:48.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.380+0000 7f7464070640 10 librados: watch_flush exit 2024-03-27T21:41:48.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.380+0000 7f7464070640 20 client.4907.objecter shutdown clearing up homeless session... 2024-03-27T21:41:48.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.380+0000 7f7464070640 10 client.4907.objecter successfully canceled tick 2024-03-27T21:41:48.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.381+0000 7f7464070640 1 -- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f744006c5c0 msgr2=0x7f744006ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:48.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.381+0000 7f7464070640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f744006c5c0 0x7f744006ea80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f744c036000 tx=0x7f744c03d040 comp rx=0 tx=0).stop 2024-03-27T21:41:48.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.381+0000 7f7464070640 10 monclient: shutdown 2024-03-27T21:41:48.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.381+0000 7f7464070640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:48.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.381+0000 7f7464070640 1 -- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c0b40b0 msgr2=0x7f745c13d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:48.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.381+0000 7f7464070640 1 --2- 172.21.15.60:0/2249519992 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f745c0b40b0 0x7f745c13d840 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f745800b4a0 tx=0x7f745800b960 comp rx=0 tx=0).stop 2024-03-27T21:41:48.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.381+0000 7f746260d640 1 -- 172.21.15.60:0/2249519992 reap_dead start 2024-03-27T21:41:48.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.381+0000 7f7464070640 1 -- 172.21.15.60:0/2249519992 shutdown_connections 2024-03-27T21:41:48.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.381+0000 7f7464070640 1 -- 172.21.15.60:0/2249519992 >> 172.21.15.60:0/2249519992 conn(0x7f745c0608e0 msgr2=0x7f745c0b6780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:48.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.381+0000 7f7464070640 1 -- 172.21.15.60:0/2249519992 shutdown_connections 2024-03-27T21:41:48.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.381+0000 7f7464070640 1 -- 172.21.15.60:0/2249519992 wait complete. 2024-03-27T21:41:48.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.381+0000 7f7464070640 1 librados: shutdown 2024-03-27T21:41:48.413 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-27T21:41:48.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.529+0000 7f6fc9dec640 10 monclient: get_monmap_and_config 2024-03-27T21:41:48.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.529+0000 7f6fc9dec640 10 monclient: build_initial_monmap 2024-03-27T21:41:48.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.529+0000 7f6fc9dec640 10 monclient: monmap: 2024-03-27T21:41:48.517 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:48.517 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:48.517 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:48.530152+0000 2024-03-27T21:41:48.517 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:48.530152+0000 2024-03-27T21:41:48.517 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:48.518 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:48.518 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:48.518 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:48.518 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:48.518 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:48.518 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:48.519 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:48.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.529+0000 7f6fc9dec640 1 Processor -- start 2024-03-27T21:41:48.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.529+0000 7f6fc9dec640 1 -- start start 2024-03-27T21:41:48.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.529+0000 7f6fc9dec640 10 monclient: init 2024-03-27T21:41:48.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc9dec640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:48.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc9dec640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-03-27T21:41:48.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc9dec640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6fc4071a50 0x7f6fc40a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:48.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc9dec640 10 monclient(hunting): picked mon.noname-c con 0x7f6fc4071a50 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:48.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc9dec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6fc413e810 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:48.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc9dec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6fc413ec50 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:48.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc9dec640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc40a8e90 con 0x7f6fc413ec50 2024-03-27T21:41:48.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc9dec640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc40aa040 con 0x7f6fc413e810 2024-03-27T21:41:48.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc9dec640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:48.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc9dec640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fc4071e30 con 0x7f6fc4071a50 2024-03-27T21:41:48.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc9dec640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:48.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc9dec640 10 monclient(hunting): authenticate will time out at 949.751401s 2024-03-27T21:41:48.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc3fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6fc413ec50 0x7f6fc4141d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:48020/0 (socket says 172.21.15.60:48020) 2024-03-27T21:41:48.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc3fff640 1 -- 172.21.15.60:0/1021034336 learned_addr learned my addr 172.21.15.60:0/1021034336 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:48.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc37fe640 1 --2- 172.21.15.60:0/1021034336 >> v2:172.21.15.191:3300/0 conn(0x7f6fc4071a50 0x7f6fc40a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:48.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc37fe640 10 monclient(hunting): get_auth_request con 0x7f6fc4071a50 auth_method 0 2024-03-27T21:41:48.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:48.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:48.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:48.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc27fc640 1 -- 172.21.15.60:0/1021034336 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120826584 0 0) 0x7f6fc40a8e90 con 0x7f6fc413ec50 2024-03-27T21:41:48.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc27fc640 10 monclient(hunting): _negotiate 2024-03-27T21:41:48.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:48.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:48.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc27fc640 10 monclient(hunting): my global_id is 5172 2024-03-27T21:41:48.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc27fc640 1 -- 172.21.15.60:0/1021034336 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fac003520 con 0x7f6fc413ec50 2024-03-27T21:41:48.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:48.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:48.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.530+0000 7f6fc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:48.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 1 -- 172.21.15.60:0/1021034336 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4114520087 0 0) 0x7f6fac003520 con 0x7f6fc413ec50 2024-03-27T21:41:48.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 1 -- 172.21.15.60:0/1021034336 >> v2:172.21.15.191:3300/0 conn(0x7f6fc4071a50 msgr2=0x7f6fc40a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:48.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 1 --2- 172.21.15.60:0/1021034336 >> v2:172.21.15.191:3300/0 conn(0x7f6fc4071a50 0x7f6fc40a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 1 -- 172.21.15.60:0/1021034336 >> v1:172.21.15.191:6789/0 conn(0x7f6fc413e810 legacy=0x7f6fc4141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:48.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 10 monclient: _finish_hunting 0 2024-03-27T21:41:48.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 1 monclient: found mon.noname-a-legacy 2024-03-27T21:41:48.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:48.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.60:6789/0 2024-03-27T21:41:48.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 1 -- 172.21.15.60:0/1021034336 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc40aa2b0 con 0x7f6fc413ec50 2024-03-27T21:41:48.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 10 monclient: _finish_auth 0 2024-03-27T21:41:48.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 10 monclient: _check_auth_tickets 2024-03-27T21:41:48.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:48.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc9dec640 5 monclient: authenticate success, global_id 5172 2024-03-27T21:41:48.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 1 -- 172.21.15.60:0/1021034336 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2933500084 0 0) 0x7f6fbc0019f0 con 0x7f6fc413ec50 2024-03-27T21:41:48.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:48.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-03-27T21:41:48.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 10 monclient: dump: 2024-03-27T21:41:48.536 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:48.536 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:48.537 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.537 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.537 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:48.537 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:48.537 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:48.537 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:48.537 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:48.537 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:48.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] but i'm connected to v1:172.21.15.60:6789/0, reconnecting 2024-03-27T21:41:48.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:48.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 1 -- 172.21.15.60:0/1021034336 >> v1:172.21.15.60:6789/0 conn(0x7f6fc413ec50 legacy=0x7f6fc4141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:48.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:41:48.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 1 --2- 172.21.15.60:0/1021034336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6fac003970 0x7f6fac003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:48.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 10 monclient(hunting): picked mon.b con 0x7f6fac003970 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:48.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 1 --2- 172.21.15.60:0/1021034336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fac0047c0 0x7f6fac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:48.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 10 monclient(hunting): picked mon.a con 0x7f6fac0047c0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:48.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 1 --2- 172.21.15.60:0/1021034336 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6fac005180 0x7f6fac00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:48.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 10 monclient(hunting): picked mon.c con 0x7f6fac005180 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:48.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:48.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 1 -- 172.21.15.60:0/1021034336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc4071e30 con 0x7f6fac0047c0 2024-03-27T21:41:48.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:48.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 1 -- 172.21.15.60:0/1021034336 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac00c020 con 0x7f6fac003970 2024-03-27T21:41:48.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:48.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 1 -- 172.21.15.60:0/1021034336 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac00c1e0 con 0x7f6fac005180 2024-03-27T21:41:48.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:48.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc27fc640 10 monclient(hunting): _finish_auth 0 2024-03-27T21:41:48.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc9dec640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:48.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc37fe640 1 --2- 172.21.15.60:0/1021034336 >> v2:172.21.15.191:3300/0 conn(0x7f6fc4071a50 0x7f6fc40a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:48.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc2ffd640 1 --2- 172.21.15.60:0/1021034336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fac0047c0 0x7f6fac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:48.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc2ffd640 10 monclient(hunting): get_auth_request con 0x7f6fac0047c0 auth_method 0 2024-03-27T21:41:48.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:48.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:48.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:41:48.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc3fff640 1 --2- 172.21.15.60:0/1021034336 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6fac005180 0x7f6fac00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:48.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc3fff640 10 monclient(hunting): get_auth_request con 0x7f6fac005180 auth_method 0 2024-03-27T21:41:48.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:48.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:48.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:41:48.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:48.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:48.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:41:48.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:48.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:41:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc37fe640 1 --2- 172.21.15.60:0/1021034336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6fac003970 0x7f6fac003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc2ffd640 10 monclient(hunting): handle_auth_done global_id 5172 payload 293 2024-03-27T21:41:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc2ffd640 1 -- 172.21.15.60:0/1021034336 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6fac005180 msgr2=0x7f6fac00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc2ffd640 1 --2- 172.21.15.60:0/1021034336 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6fac005180 0x7f6fac00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc2ffd640 1 -- 172.21.15.60:0/1021034336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6fac003970 msgr2=0x7f6fac003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc2ffd640 1 --2- 172.21.15.60:0/1021034336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6fac003970 0x7f6fac003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc2ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:41:48.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc2ffd640 1 monclient: found mon.a 2024-03-27T21:41:48.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:48.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc2ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:48.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc3fff640 1 -- 172.21.15.60:0/1021034336 reap_dead start 2024-03-27T21:41:48.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc2ffd640 1 -- 172.21.15.60:0/1021034336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fac003520 con 0x7f6fac0047c0 2024-03-27T21:41:48.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc2ffd640 10 monclient: _finish_auth 0 2024-03-27T21:41:48.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc2ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:41:48.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:48.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc37fe640 10 monclient: get_auth_request con 0x7f6fac003970 auth_method 0 2024-03-27T21:41:48.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.531+0000 7f6fc37fe640 1 --2- 172.21.15.60:0/1021034336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6fac003970 0x7f6fac003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:41:48.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.532+0000 7f6fc2ffd640 1 --2- 172.21.15.60:0/1021034336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fac0047c0 0x7f6fac004c40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8001750 tx=0x7f6fac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6effef5f0d7a8890 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:48.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.532+0000 7f6fc27fc640 1 -- 172.21.15.60:0/1021034336 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb80052d0 con 0x7f6fac0047c0 2024-03-27T21:41:48.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.532+0000 7f6fc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:48.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.532+0000 7f6fc27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:48.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.532+0000 7f6fc27fc640 10 monclient: dump: 2024-03-27T21:41:48.544 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:48.545 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:48.545 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.545 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.545 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:48.545 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:48.545 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:48.546 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:48.546 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:48.546 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:48.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.532+0000 7f6fc9dec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:48.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.532+0000 7f6fc27fc640 1 -- 172.21.15.60:0/1021034336 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb802e050 con 0x7f6fac0047c0 2024-03-27T21:41:48.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.532+0000 7f6fc27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:48.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.532+0000 7f6fc9dec640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:48.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.532+0000 7f6fc27fc640 1 -- 172.21.15.60:0/1021034336 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8003830 con 0x7f6fac0047c0 2024-03-27T21:41:48.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.532+0000 7f6fc9dec640 4 set_mon_vals no callback set 2024-03-27T21:41:48.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.532+0000 7f6fc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:48.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.532+0000 7f6fc27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:48.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.532+0000 7f6fc27fc640 10 monclient: dump: 2024-03-27T21:41:48.547 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:48.547 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:48.547 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.548 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.548 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:48.548 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:48.548 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:48.548 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:48.548 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:48.548 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:48.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.532+0000 7f6fc9dec640 10 monclient: shutdown 2024-03-27T21:41:48.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.532+0000 7f6fc9dec640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:48.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.532+0000 7f6fc9dec640 1 -- 172.21.15.60:0/1021034336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fac0047c0 msgr2=0x7f6fac004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:48.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.532+0000 7f6fc9dec640 1 --2- 172.21.15.60:0/1021034336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fac0047c0 0x7f6fac004c40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8001750 tx=0x7f6fac0057f0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.533+0000 7f6fc9dec640 1 -- 172.21.15.60:0/1021034336 shutdown_connections 2024-03-27T21:41:48.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.533+0000 7f6fc9dec640 1 --2- 172.21.15.60:0/1021034336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fac0047c0 0x7f6fac004c40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.533+0000 7f6fc9dec640 1 -- 172.21.15.60:0/1021034336 >> 172.21.15.60:0/1021034336 conn(0x7f6fc40b7300 msgr2=0x7f6fc40b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:48.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.533+0000 7f6fc9dec640 1 -- 172.21.15.60:0/1021034336 shutdown_connections 2024-03-27T21:41:48.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.533+0000 7f6fc9dec640 1 -- 172.21.15.60:0/1021034336 wait complete. 2024-03-27T21:41:48.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.533+0000 7f6fc9dec640 10 monclient: build_initial_monmap 2024-03-27T21:41:48.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.533+0000 7f6fc9dec640 10 monclient: monmap: 2024-03-27T21:41:48.550 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:48.550 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:48.550 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:48.550 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:48.550 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:48.550 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:48.550 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:48.551 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:48.551 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:48.551 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:48.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.533+0000 7f6fc9dec640 1 librados: starting msgr at 2024-03-27T21:41:48.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.533+0000 7f6fc9dec640 1 librados: starting objecter 2024-03-27T21:41:48.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.533+0000 7f6fc9dec640 1 Processor -- start 2024-03-27T21:41:48.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc9dec640 1 -- start start 2024-03-27T21:41:48.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc9dec640 1 librados: setting wanted keys 2024-03-27T21:41:48.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc9dec640 1 librados: calling monclient init 2024-03-27T21:41:48.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc9dec640 10 monclient: init 2024-03-27T21:41:48.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc9dec640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:48.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc9dec640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:41:48.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc9dec640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6fac0047c0 0x7f6fc40b0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:48.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc9dec640 10 monclient(hunting): picked mon.noname-c con 0x7f6fac0047c0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:48.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc9dec640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6fc40af3c0 0x7f6fc41d9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:48.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc9dec640 10 monclient(hunting): picked mon.noname-b con 0x7f6fc40af3c0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:48.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc9dec640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fc40af800 0x7f6fc41da420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:48.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc9dec640 10 monclient(hunting): picked mon.noname-a con 0x7f6fc40af800 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:48.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc9dec640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:48.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc9dec640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc40b06d0 con 0x7f6fc40af800 2024-03-27T21:41:48.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc9dec640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:48.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc9dec640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc41d9a50 con 0x7f6fc40af3c0 2024-03-27T21:41:48.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc9dec640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:48.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc9dec640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc40ae630 con 0x7f6fac0047c0 2024-03-27T21:41:48.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc9dec640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:48.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc9dec640 10 monclient(hunting): authenticate will time out at 949.755646s 2024-03-27T21:41:48.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc37fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6fac0047c0 0x7f6fc40b0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:48.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fc40af800 0x7f6fc41da420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:48.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc37fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6fac0047c0 0x7f6fc40b0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:32850/0 (socket says 172.21.15.60:32850) 2024-03-27T21:41:48.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc37fe640 1 -- 172.21.15.60:0/1867062222 learned_addr learned my addr 172.21.15.60:0/1867062222 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:48.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc37fe640 10 monclient(hunting): get_auth_request con 0x7f6fac0047c0 auth_method 0 2024-03-27T21:41:48.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:48.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:48.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:48.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc3fff640 10 monclient(hunting): get_auth_request con 0x7f6fc40af800 auth_method 0 2024-03-27T21:41:48.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:48.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:48.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:48.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:48.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:48.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:48.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:48.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:48.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:48.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc2ffd640 1 --2- 172.21.15.60:0/1867062222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6fc40af3c0 0x7f6fc41d9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:48.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc2ffd640 10 monclient(hunting): get_auth_request con 0x7f6fc40af3c0 auth_method 0 2024-03-27T21:41:48.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:48.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.534+0000 7f6fc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:48.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:48.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc3fff640 10 monclient(hunting): handle_auth_done global_id 5175 payload 824 2024-03-27T21:41:48.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc3fff640 1 -- 172.21.15.60:0/1867062222 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6fac0047c0 msgr2=0x7f6fc40b0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:48.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc3fff640 1 --2- 172.21.15.60:0/1867062222 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6fac0047c0 0x7f6fc40b0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc3fff640 1 -- 172.21.15.60:0/1867062222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6fc40af3c0 msgr2=0x7f6fc41d9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:48.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc3fff640 1 --2- 172.21.15.60:0/1867062222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6fc40af3c0 0x7f6fc41d9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:48.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc3fff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:48.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc3fff640 1 monclient: found mon.noname-a 2024-03-27T21:41:48.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:48.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:48.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc3fff640 1 -- 172.21.15.60:0/1867062222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc40ac890 con 0x7f6fc40af800 2024-03-27T21:41:48.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc2ffd640 1 --2- 172.21.15.60:0/1867062222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6fc40af3c0 0x7f6fc41d9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:41:48.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc37fe640 1 --2- 172.21.15.60:0/1867062222 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6fac0047c0 0x7f6fc40b0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:48.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc3fff640 1 --2- 172.21.15.60:0/1867062222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fc40af800 0x7f6fc41da420 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6fc40a8e00 tx=0x7f6fbc007b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:48.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fabfff640 10 client.?.objecter ms_handle_connect 0x7f6fc40af800 2024-03-27T21:41:48.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fabfff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:48.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fabfff640 1 -- 172.21.15.60:0/1867062222 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc00ab70 con 0x7f6fc40af800 2024-03-27T21:41:48.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:48.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fabfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:48.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fabfff640 10 monclient: dump: 2024-03-27T21:41:48.561 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:48.561 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:48.561 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.561 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.562 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:48.562 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:48.562 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:48.562 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:48.562 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:48.562 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:48.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fabfff640 10 monclient: _finish_auth 0 2024-03-27T21:41:48.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fabfff640 10 monclient: _check_auth_tickets 2024-03-27T21:41:48.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:48.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fabfff640 1 -- 172.21.15.60:0/1867062222 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc00ad10 con 0x7f6fc40af800 2024-03-27T21:41:48.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc9dec640 5 monclient: authenticate success, global_id 5175 2024-03-27T21:41:48.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc9dec640 10 monclient: _renew_subs 2024-03-27T21:41:48.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc9dec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:48.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc9dec640 1 -- 172.21.15.60:0/1867062222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc41d6460 con 0x7f6fc40af800 2024-03-27T21:41:48.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fabfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:48.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fabfff640 1 -- 172.21.15.60:0/1867062222 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc013630 con 0x7f6fc40af800 2024-03-27T21:41:48.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:48.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fabfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:48.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fabfff640 10 monclient: dump: 2024-03-27T21:41:48.564 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:48.564 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:48.565 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.565 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:48.565 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:48.565 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:48.566 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:48.566 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:48.566 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:48.566 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc9dec640 10 client.5175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc1ffb640 4 set_mon_vals no callback set 2024-03-27T21:41:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc9dec640 10 monclient: _renew_subs 2024-03-27T21:41:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc9dec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc9dec640 1 -- 172.21.15.60:0/1867062222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc41d6890 con 0x7f6fc40af800 2024-03-27T21:41:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.535+0000 7f6fc9dec640 1 librados: init done 2024-03-27T21:41:48.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.536+0000 7f6fc9dec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:48.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.536+0000 7f6fc9dec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:48.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.536+0000 7f6fc9dec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:48.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.536+0000 7f6fc9dec640 1 -- 172.21.15.60:0/1867062222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc40aca20 con 0x7f6fc40af800 2024-03-27T21:41:48.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.536+0000 7f6fabfff640 1 -- 172.21.15.60:0/1867062222 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6fbc013790 con 0x7f6fc40af800 2024-03-27T21:41:48.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.537+0000 7f6fabfff640 1 --2- 172.21.15.60:0/1867062222 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6fb006c590 0x7f6fb006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:48.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.537+0000 7f6fc37fe640 1 --2- 172.21.15.60:0/1867062222 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6fb006c590 0x7f6fb006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:48.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.537+0000 7f6fc37fe640 10 monclient: get_auth_request con 0x7f6fb006c590 auth_method 0 2024-03-27T21:41:48.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.537+0000 7f6fabfff640 1 -- 172.21.15.60:0/1867062222 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f6fbc0896c0 con 0x7f6fc40af800 2024-03-27T21:41:48.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.537+0000 7f6fabfff640 10 client.5175.objecter ms_dispatch 0x7f6fc4000b90 osd_map(76..76 src has 1..76) v4 2024-03-27T21:41:48.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.537+0000 7f6fabfff640 3 client.5175.objecter handle_osd_map got epochs [76,76] > 0 2024-03-27T21:41:48.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.537+0000 7f6fabfff640 3 client.5175.objecter handle_osd_map decoding full epoch 76 2024-03-27T21:41:48.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.537+0000 7f6fabfff640 20 client.5175.objecter dump_active .. 0 homeless 2024-03-27T21:41:48.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.537+0000 7f6fabfff640 10 client.5175.objecter ms_handle_connect 0x7f6fb006c590 2024-03-27T21:41:48.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.537+0000 7f6fc37fe640 1 --2- 172.21.15.60:0/1867062222 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6fb006c590 0x7f6fb006ea50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4006a90 tx=0x7f6fb4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:48.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.541+0000 7f6fabfff640 1 -- 172.21.15.60:0/1867062222 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6fc40aca20 con 0x7f6fc40af800 2024-03-27T21:41:48.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.541+0000 7f6fabfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:48.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.541+0000 7f6fabfff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:48.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.730+0000 7f6fc9dec640 10 start_mon_command cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}] 2024-03-27T21:41:48.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.730+0000 7f6fc9dec640 10 monclient: _send_command 2 [{"prefix": "osd reweight", "id": 0, "weight": 1.0}] 2024-03-27T21:41:48.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.730+0000 7f6fc9dec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:48.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:48.730+0000 7f6fc9dec640 1 -- 172.21.15.60:0/1867062222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) v1 -- 0x7f6fc40aca20 con 0x7f6fc40af800 2024-03-27T21:41:49.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.192+0000 7f6fabfff640 1 -- 172.21.15.60:0/1867062222 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v77) v1 ==== 113+0+0 (secure 0 0 0) 0x7f6fc40aca20 con 0x7f6fc40af800 2024-03-27T21:41:49.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.192+0000 7f6fabfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd reweight", "id": 0, "weight": 1.0}] 2024-03-27T21:41:49.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.192+0000 7f6fabfff640 10 monclient: _finish_command 2 = system:0 reweighted osd.0 to 1 (10000) 2024-03-27T21:41:49.179 INFO:teuthology.orchestra.run.smithi060.stderr:reweighted osd.0 to 1 (10000) 2024-03-27T21:41:49.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.195+0000 7f6fc9dec640 10 librados: watch_flush enter 2024-03-27T21:41:49.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.195+0000 7f6fc9dec640 10 librados: watch_flush exit 2024-03-27T21:41:49.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.195+0000 7f6fc9dec640 20 client.5175.objecter shutdown clearing up homeless session... 2024-03-27T21:41:49.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.195+0000 7f6fc9dec640 10 client.5175.objecter successfully canceled tick 2024-03-27T21:41:49.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.195+0000 7f6fc9dec640 1 -- 172.21.15.60:0/1867062222 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6fb006c590 msgr2=0x7f6fb006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:49.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.195+0000 7f6fc9dec640 1 --2- 172.21.15.60:0/1867062222 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6fb006c590 0x7f6fb006ea50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4006a90 tx=0x7f6fb4008040 comp rx=0 tx=0).stop 2024-03-27T21:41:49.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.195+0000 7f6fc9dec640 10 monclient: shutdown 2024-03-27T21:41:49.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.195+0000 7f6fc9dec640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:49.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.195+0000 7f6fc9dec640 1 -- 172.21.15.60:0/1867062222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fc40af800 msgr2=0x7f6fc41da420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:49.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.195+0000 7f6fc9dec640 1 --2- 172.21.15.60:0/1867062222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fc40af800 0x7f6fc41da420 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6fc40a8e00 tx=0x7f6fbc007b00 comp rx=0 tx=0).stop 2024-03-27T21:41:49.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.195+0000 7f6fc9dec640 1 -- 172.21.15.60:0/1867062222 shutdown_connections 2024-03-27T21:41:49.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.195+0000 7f6fc9dec640 1 --2- 172.21.15.60:0/1867062222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fc40af800 0x7f6fc41da420 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:49.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.195+0000 7f6fc9dec640 1 --2- 172.21.15.60:0/1867062222 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f6fb006c590 0x7f6fb006ea50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:49.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.195+0000 7f6fc9dec640 1 --2- 172.21.15.60:0/1867062222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6fc40af3c0 0x7f6fc41d9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:49.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.195+0000 7f6fc9dec640 1 --2- 172.21.15.60:0/1867062222 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6fac0047c0 0x7f6fc40b0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:49.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.195+0000 7f6fc9dec640 1 -- 172.21.15.60:0/1867062222 >> 172.21.15.60:0/1867062222 conn(0x7f6fc40b7300 msgr2=0x7f6fc40b17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:49.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.195+0000 7f6fc9dec640 1 -- 172.21.15.60:0/1867062222 shutdown_connections 2024-03-27T21:41:49.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.195+0000 7f6fc9dec640 1 -- 172.21.15.60:0/1867062222 wait complete. 2024-03-27T21:41:49.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.195+0000 7f6fc9dec640 1 librados: shutdown 2024-03-27T21:41:49.194 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-27T21:41:49.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.310+0000 7f5b66eb6640 10 monclient: get_monmap_and_config 2024-03-27T21:41:49.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.310+0000 7f5b66eb6640 10 monclient: build_initial_monmap 2024-03-27T21:41:49.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.310+0000 7f5b66eb6640 10 monclient: monmap: 2024-03-27T21:41:49.297 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:49.297 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:49.297 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:49.311448+0000 2024-03-27T21:41:49.297 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:49.311448+0000 2024-03-27T21:41:49.298 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:49.298 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:49.298 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:49.298 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:49.298 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:49.298 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:49.298 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:49.299 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:49.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.311+0000 7f5b66eb6640 1 Processor -- start 2024-03-27T21:41:49.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.311+0000 7f5b66eb6640 1 -- start start 2024-03-27T21:41:49.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.311+0000 7f5b66eb6640 10 monclient: init 2024-03-27T21:41:49.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.311+0000 7f5b66eb6640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:49.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.311+0000 7f5b66eb6640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-03-27T21:41:49.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.311+0000 7f5b66eb6640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5b60071a50 0x7f5b600a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:49.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.311+0000 7f5b66eb6640 10 monclient(hunting): picked mon.noname-b con 0x7f5b60071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:49.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.311+0000 7f5b66eb6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b6013e810 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:49.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.311+0000 7f5b66eb6640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5b6013ec50 0x7f5b60141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:49.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.311+0000 7f5b66eb6640 10 monclient(hunting): picked mon.noname-a con 0x7f5b6013ec50 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:49.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.311+0000 7f5b66eb6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b600a8e90 con 0x7f5b6013e810 2024-03-27T21:41:49.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.311+0000 7f5b66eb6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:49.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.311+0000 7f5b66eb6640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b60071e30 con 0x7f5b6013ec50 2024-03-27T21:41:49.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.311+0000 7f5b66eb6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:49.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.311+0000 7f5b66eb6640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b600aa040 con 0x7f5b60071a50 2024-03-27T21:41:49.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.311+0000 7f5b66eb6640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:49.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.311+0000 7f5b66eb6640 10 monclient(hunting): authenticate will time out at 950.532827s 2024-03-27T21:41:49.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b64c52640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5b60071a50 0x7f5b600a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:49.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b65453640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5b6013ec50 0x7f5b60141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:49.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b65453640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5b6013ec50 0x7f5b60141d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41896/0 (socket says 172.21.15.60:41896) 2024-03-27T21:41:49.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b64c52640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5b60071a50 0x7f5b600a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:32858/0 (socket says 172.21.15.60:32858) 2024-03-27T21:41:49.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b65453640 1 -- 172.21.15.60:0/3308554355 learned_addr learned my addr 172.21.15.60:0/3308554355 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:49.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b65453640 10 monclient(hunting): get_auth_request con 0x7f5b6013ec50 auth_method 0 2024-03-27T21:41:49.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b65453640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:49.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b65453640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:49.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b65453640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:49.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b64c52640 10 monclient(hunting): get_auth_request con 0x7f5b60071a50 auth_method 0 2024-03-27T21:41:49.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b64c52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:49.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b64c52640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:49.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b64c52640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:49.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b65453640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:49.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b65453640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:49.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b65453640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:49.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b64c52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:49.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b64c52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:49.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b64c52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:49.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b64c52640 10 monclient(hunting): handle_auth_done global_id 4919 payload 274 2024-03-27T21:41:49.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b64c52640 1 -- 172.21.15.60:0/3308554355 >> v2:172.21.15.60:3300/0 conn(0x7f5b6013ec50 msgr2=0x7f5b60141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:49.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b64c52640 1 --2- 172.21.15.60:0/3308554355 >> v2:172.21.15.60:3300/0 conn(0x7f5b6013ec50 0x7f5b60141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:49.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b64c52640 1 -- 172.21.15.60:0/3308554355 >> v1:172.21.15.191:6789/0 conn(0x7f5b6013e810 legacy=0x7f5b60141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:49.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b64c52640 10 monclient: _finish_hunting 0 2024-03-27T21:41:49.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b64c52640 1 monclient: found mon.noname-b 2024-03-27T21:41:49.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b64c52640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:49.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b64c52640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:49.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b64c52640 1 -- 172.21.15.60:0/3308554355 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b60142280 con 0x7f5b60071a50 2024-03-27T21:41:49.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.312+0000 7f5b65453640 1 --2- 172.21.15.60:0/3308554355 >> v2:172.21.15.60:3300/0 conn(0x7f5b6013ec50 0x7f5b60141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:49.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b64c52640 1 --2- 172.21.15.60:0/3308554355 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5b60071a50 0x7f5b600a87e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c0097a0 tx=0x7f5b5c02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=464ffec7c2810ae3 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:49.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b577fe640 1 -- 172.21.15.60:0/3308554355 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c031780 con 0x7f5b60071a50 2024-03-27T21:41:49.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:49.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:49.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b577fe640 10 monclient: dump: 2024-03-27T21:41:49.315 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:49.315 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:49.315 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:49.315 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:49.315 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:49.315 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:49.316 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:49.316 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:49.316 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:49.316 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:49.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b577fe640 10 monclient: _finish_auth 0 2024-03-27T21:41:49.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b577fe640 10 monclient: _check_auth_tickets 2024-03-27T21:41:49.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:49.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b577fe640 1 -- 172.21.15.60:0/3308554355 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b5c0318e0 con 0x7f5b60071a50 2024-03-27T21:41:49.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b66eb6640 5 monclient: authenticate success, global_id 4919 2024-03-27T21:41:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b66eb6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b577fe640 1 -- 172.21.15.60:0/3308554355 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c031c70 con 0x7f5b60071a50 2024-03-27T21:41:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b66eb6640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b66eb6640 4 set_mon_vals no callback set 2024-03-27T21:41:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b577fe640 10 monclient: dump: 2024-03-27T21:41:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:49.318 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:49.318 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:49.318 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:49.318 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:49.318 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:49.318 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:49.318 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:49.318 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:49.319 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:49.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b66eb6640 10 monclient: shutdown 2024-03-27T21:41:49.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b66eb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:49.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b66eb6640 1 -- 172.21.15.60:0/3308554355 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5b60071a50 msgr2=0x7f5b600a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:49.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b66eb6640 1 --2- 172.21.15.60:0/3308554355 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5b60071a50 0x7f5b600a87e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c0097a0 tx=0x7f5b5c02b820 comp rx=0 tx=0).stop 2024-03-27T21:41:49.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b66eb6640 1 -- 172.21.15.60:0/3308554355 shutdown_connections 2024-03-27T21:41:49.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b66eb6640 1 --2- 172.21.15.60:0/3308554355 >> v2:172.21.15.60:3300/0 conn(0x7f5b6013ec50 0x7f5b60141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:49.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b66eb6640 1 --2- 172.21.15.60:0/3308554355 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5b60071a50 0x7f5b600a87e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:49.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b66eb6640 1 -- 172.21.15.60:0/3308554355 >> 172.21.15.60:0/3308554355 conn(0x7f5b600b7300 msgr2=0x7f5b600b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:49.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b66eb6640 1 -- 172.21.15.60:0/3308554355 shutdown_connections 2024-03-27T21:41:49.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.313+0000 7f5b66eb6640 1 -- 172.21.15.60:0/3308554355 wait complete. 2024-03-27T21:41:49.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.314+0000 7f5b66eb6640 10 monclient: build_initial_monmap 2024-03-27T21:41:49.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.314+0000 7f5b66eb6640 10 monclient: monmap: 2024-03-27T21:41:49.320 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:49.320 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:49.320 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:49.321 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:49.321 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:49.321 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:49.321 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:49.321 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:49.321 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:49.321 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:49.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.314+0000 7f5b66eb6640 1 librados: starting msgr at 2024-03-27T21:41:49.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.314+0000 7f5b66eb6640 1 librados: starting objecter 2024-03-27T21:41:49.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.314+0000 7f5b66eb6640 1 Processor -- start 2024-03-27T21:41:49.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.314+0000 7f5b66eb6640 1 -- start start 2024-03-27T21:41:49.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.314+0000 7f5b66eb6640 1 librados: setting wanted keys 2024-03-27T21:41:49.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.314+0000 7f5b66eb6640 1 librados: calling monclient init 2024-03-27T21:41:49.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.314+0000 7f5b66eb6640 10 monclient: init 2024-03-27T21:41:49.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b66eb6640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:49.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b66eb6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:41:49.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b66eb6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b60071a50 0x7f5b601378c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:49.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b66eb6640 10 monclient(hunting): picked mon.noname-a con 0x7f5b60071a50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:49.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b66eb6640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5b6013e810 0x7f5b601341f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:49.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b66eb6640 10 monclient(hunting): picked mon.noname-c con 0x7f5b6013e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:49.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b66eb6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b6013ec50 0x7f5b60134730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:49.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b66eb6640 10 monclient(hunting): picked mon.noname-b con 0x7f5b6013ec50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:49.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b66eb6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:49.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b66eb6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b60136550 con 0x7f5b60071a50 2024-03-27T21:41:49.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b66eb6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:49.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b66eb6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b60136970 con 0x7f5b6013ec50 2024-03-27T21:41:49.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b66eb6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:49.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b66eb6640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b60139210 con 0x7f5b6013e810 2024-03-27T21:41:49.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b66eb6640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:49.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b66eb6640 10 monclient(hunting): authenticate will time out at 950.536279s 2024-03-27T21:41:49.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b57fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5b6013e810 0x7f5b601341f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:49.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b64c52640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b60071a50 0x7f5b601378c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:49.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b57fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5b6013e810 0x7f5b601341f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:32868/0 (socket says 172.21.15.60:32868) 2024-03-27T21:41:49.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b64c52640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b60071a50 0x7f5b601378c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41908/0 (socket says 172.21.15.60:41908) 2024-03-27T21:41:49.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b57fff640 1 -- 172.21.15.60:0/2930698349 learned_addr learned my addr 172.21.15.60:0/2930698349 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:49.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b57fff640 10 monclient(hunting): get_auth_request con 0x7f5b6013e810 auth_method 0 2024-03-27T21:41:49.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:49.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b57fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:49.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:49.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b64c52640 10 monclient(hunting): get_auth_request con 0x7f5b60071a50 auth_method 0 2024-03-27T21:41:49.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b64c52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:49.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b64c52640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:49.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b64c52640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:49.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:49.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:49.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:49.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b64c52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:49.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b64c52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:49.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b65453640 1 --2- 172.21.15.60:0/2930698349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b6013ec50 0x7f5b60134730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:49.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b64c52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:49.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b65453640 10 monclient(hunting): get_auth_request con 0x7f5b6013ec50 auth_method 0 2024-03-27T21:41:49.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b65453640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:49.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b65453640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:49.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b65453640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:49.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b65453640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:49.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b65453640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:49.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b65453640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:49.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b57fff640 10 monclient(hunting): handle_auth_done global_id 4922 payload 824 2024-03-27T21:41:49.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b57fff640 1 -- 172.21.15.60:0/2930698349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b6013ec50 msgr2=0x7f5b60134730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:49.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b57fff640 1 --2- 172.21.15.60:0/2930698349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b6013ec50 0x7f5b60134730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:49.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b57fff640 1 -- 172.21.15.60:0/2930698349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b60071a50 msgr2=0x7f5b601378c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:49.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b57fff640 1 --2- 172.21.15.60:0/2930698349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b60071a50 0x7f5b601378c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:49.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b57fff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:49.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b57fff640 1 monclient: found mon.noname-c 2024-03-27T21:41:49.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:49.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b57fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:49.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b57fff640 1 -- 172.21.15.60:0/2930698349 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b58009b30 con 0x7f5b6013e810 2024-03-27T21:41:49.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b64c52640 1 --2- 172.21.15.60:0/2930698349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b60071a50 0x7f5b601378c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:49.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b57fff640 1 --2- 172.21.15.60:0/2930698349 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5b6013e810 0x7f5b601341f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5b5000c460 tx=0x7f5b5000c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:49.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b557fa640 10 client.?.objecter ms_handle_connect 0x7f5b6013e810 2024-03-27T21:41:49.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.315+0000 7f5b557fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:49.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b557fa640 1 -- 172.21.15.60:0/2930698349 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b50011790 con 0x7f5b6013e810 2024-03-27T21:41:49.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:49.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b557fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:49.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b557fa640 10 monclient: dump: 2024-03-27T21:41:49.331 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:49.331 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:49.331 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:49.331 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:49.331 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:49.331 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:49.332 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:49.332 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:49.332 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:49.332 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:49.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b557fa640 10 monclient: _finish_auth 0 2024-03-27T21:41:49.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b557fa640 10 monclient: _check_auth_tickets 2024-03-27T21:41:49.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:49.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b557fa640 1 -- 172.21.15.60:0/2930698349 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b50011930 con 0x7f5b6013e810 2024-03-27T21:41:49.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b66eb6640 5 monclient: authenticate success, global_id 4922 2024-03-27T21:41:49.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b66eb6640 10 monclient: _renew_subs 2024-03-27T21:41:49.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b66eb6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:49.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b66eb6640 1 -- 172.21.15.60:0/2930698349 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b5c009400 con 0x7f5b6013e810 2024-03-27T21:41:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b557fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b557fa640 1 -- 172.21.15.60:0/2930698349 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b50011ce0 con 0x7f5b6013e810 2024-03-27T21:41:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b557fa640 10 monclient: dump: 2024-03-27T21:41:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:49.335 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b66eb6640 10 client.4922.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b66eb6640 10 monclient: _renew_subs 2024-03-27T21:41:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b66eb6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b66eb6640 1 -- 172.21.15.60:0/2930698349 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b601395a0 con 0x7f5b6013e810 2024-03-27T21:41:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b66eb6640 1 librados: init done 2024-03-27T21:41:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.316+0000 7f5b56ffd640 4 set_mon_vals no callback set 2024-03-27T21:41:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.317+0000 7f5b66eb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.317+0000 7f5b66eb6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.317+0000 7f5b66eb6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.317+0000 7f5b66eb6640 1 -- 172.21.15.60:0/2930698349 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b6014bdc0 con 0x7f5b6013e810 2024-03-27T21:41:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.317+0000 7f5b557fa640 1 -- 172.21.15.60:0/2930698349 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5b5001c460 con 0x7f5b6013e810 2024-03-27T21:41:49.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.318+0000 7f5b557fa640 1 --2- 172.21.15.60:0/2930698349 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5b4c06c5c0 0x7f5b4c06ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:49.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.318+0000 7f5b64c52640 1 --2- 172.21.15.60:0/2930698349 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5b4c06c5c0 0x7f5b4c06ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:49.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.318+0000 7f5b64c52640 10 monclient: get_auth_request con 0x7f5b4c06c5c0 auth_method 0 2024-03-27T21:41:49.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.318+0000 7f5b557fa640 1 -- 172.21.15.60:0/2930698349 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f5b50088e70 con 0x7f5b6013e810 2024-03-27T21:41:49.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.318+0000 7f5b557fa640 10 client.4922.objecter ms_dispatch 0x7f5b60000b90 osd_map(77..77 src has 1..77) v4 2024-03-27T21:41:49.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.318+0000 7f5b557fa640 3 client.4922.objecter handle_osd_map got epochs [77,77] > 0 2024-03-27T21:41:49.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.318+0000 7f5b557fa640 3 client.4922.objecter handle_osd_map decoding full epoch 77 2024-03-27T21:41:49.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.318+0000 7f5b557fa640 20 client.4922.objecter dump_active .. 0 homeless 2024-03-27T21:41:49.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.318+0000 7f5b557fa640 10 client.4922.objecter ms_handle_connect 0x7f5b4c06c5c0 2024-03-27T21:41:49.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.318+0000 7f5b64c52640 1 --2- 172.21.15.60:0/2930698349 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5b4c06c5c0 0x7f5b4c06ea80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c036000 tx=0x7f5b5c03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:49.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.322+0000 7f5b557fa640 1 -- 172.21.15.60:0/2930698349 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5b6014bdc0 con 0x7f5b6013e810 2024-03-27T21:41:49.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.322+0000 7f5b557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:49.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.322+0000 7f5b557fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:49.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.514+0000 7f5b66eb6640 10 start_mon_command cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}] 2024-03-27T21:41:49.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.514+0000 7f5b66eb6640 10 monclient: _send_command 2 [{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}] 2024-03-27T21:41:49.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.514+0000 7f5b66eb6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:49.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:49.514+0000 7f5b66eb6640 1 -- 172.21.15.60:0/2930698349 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} v 0) v1 -- 0x7f5b601362b0 con 0x7f5b6013e810 2024-03-27T21:41:50.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.200+0000 7f5b557fa640 1 -- 172.21.15.60:0/2930698349 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]=0 set osd.0 primary-affinity to 1 (8655362) v78) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5b601362b0 con 0x7f5b6013e810 2024-03-27T21:41:50.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.200+0000 7f5b557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}] 2024-03-27T21:41:50.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.200+0000 7f5b557fa640 10 monclient: _finish_command 2 = system:0 set osd.0 primary-affinity to 1 (8655362) 2024-03-27T21:41:50.192 INFO:teuthology.orchestra.run.smithi060.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-27T21:41:50.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.202+0000 7f5b66eb6640 10 librados: watch_flush enter 2024-03-27T21:41:50.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.202+0000 7f5b66eb6640 10 librados: watch_flush exit 2024-03-27T21:41:50.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.202+0000 7f5b66eb6640 20 client.4922.objecter shutdown clearing up homeless session... 2024-03-27T21:41:50.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.202+0000 7f5b66eb6640 10 client.4922.objecter successfully canceled tick 2024-03-27T21:41:50.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.202+0000 7f5b66eb6640 1 -- 172.21.15.60:0/2930698349 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5b4c06c5c0 msgr2=0x7f5b4c06ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:50.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.202+0000 7f5b66eb6640 1 --2- 172.21.15.60:0/2930698349 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5b4c06c5c0 0x7f5b4c06ea80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c036000 tx=0x7f5b5c03d040 comp rx=0 tx=0).stop 2024-03-27T21:41:50.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.202+0000 7f5b66eb6640 10 monclient: shutdown 2024-03-27T21:41:50.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.202+0000 7f5b66eb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:50.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.202+0000 7f5b66eb6640 1 -- 172.21.15.60:0/2930698349 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5b6013e810 msgr2=0x7f5b601341f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:50.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.202+0000 7f5b66eb6640 1 --2- 172.21.15.60:0/2930698349 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5b6013e810 0x7f5b601341f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5b5000c460 tx=0x7f5b5000c920 comp rx=0 tx=0).stop 2024-03-27T21:41:50.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.202+0000 7f5b66eb6640 1 -- 172.21.15.60:0/2930698349 shutdown_connections 2024-03-27T21:41:50.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.202+0000 7f5b66eb6640 1 --2- 172.21.15.60:0/2930698349 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5b4c06c5c0 0x7f5b4c06ea80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:50.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.202+0000 7f5b66eb6640 1 --2- 172.21.15.60:0/2930698349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b6013ec50 0x7f5b60134730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:50.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.202+0000 7f5b66eb6640 1 --2- 172.21.15.60:0/2930698349 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5b6013e810 0x7f5b601341f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:50.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.202+0000 7f5b66eb6640 1 --2- 172.21.15.60:0/2930698349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b60071a50 0x7f5b601378c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:50.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.202+0000 7f5b66eb6640 1 -- 172.21.15.60:0/2930698349 >> 172.21.15.60:0/2930698349 conn(0x7f5b600b7300 msgr2=0x7f5b60132d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:50.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.203+0000 7f5b66eb6640 1 -- 172.21.15.60:0/2930698349 shutdown_connections 2024-03-27T21:41:50.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.203+0000 7f5b66eb6640 1 -- 172.21.15.60:0/2930698349 wait complete. 2024-03-27T21:41:50.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.203+0000 7f5b66eb6640 1 librados: shutdown 2024-03-27T21:41:50.201 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-27T21:41:50.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.320+0000 7fe6ef389640 10 monclient: get_monmap_and_config 2024-03-27T21:41:50.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.320+0000 7fe6ef389640 10 monclient: build_initial_monmap 2024-03-27T21:41:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.321+0000 7fe6ef389640 10 monclient: monmap: 2024-03-27T21:41:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:50.321978+0000 2024-03-27T21:41:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:50.321978+0000 2024-03-27T21:41:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:50.309 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:50.309 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:50.309 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:50.309 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:50.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.321+0000 7fe6ef389640 1 Processor -- start 2024-03-27T21:41:50.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.321+0000 7fe6ef389640 1 -- start start 2024-03-27T21:41:50.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.321+0000 7fe6ef389640 10 monclient: init 2024-03-27T21:41:50.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ef389640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:50.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ef389640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-03-27T21:41:50.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ef389640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe6e8071a50 0x7fe6e80a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:50.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ef389640 10 monclient(hunting): picked mon.noname-b con 0x7fe6e8071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:50.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ef389640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe6e813e810 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:50.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ef389640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe6e813ec50 0x7fe6e8141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:50.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ef389640 10 monclient(hunting): picked mon.noname-a con 0x7fe6e813ec50 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:50.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ef389640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e80a8e90 con 0x7fe6e813e810 2024-03-27T21:41:50.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ef389640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:50.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ef389640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e8071e30 con 0x7fe6e813ec50 2024-03-27T21:41:50.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ef389640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:50.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ef389640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e80aa040 con 0x7fe6e8071a50 2024-03-27T21:41:50.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ef389640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:50.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ef389640 10 monclient(hunting): authenticate will time out at 951.543261s 2024-03-27T21:41:50.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed125640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe6e8071a50 0x7fe6e80a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:50.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed926640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe6e813ec50 0x7fe6e8141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:50.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed125640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe6e8071a50 0x7fe6e80a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:32870/0 (socket says 172.21.15.60:32870) 2024-03-27T21:41:50.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed926640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe6e813ec50 0x7fe6e8141d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41914/0 (socket says 172.21.15.60:41914) 2024-03-27T21:41:50.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed125640 1 -- 172.21.15.60:0/1256044779 learned_addr learned my addr 172.21.15.60:0/1256044779 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:50.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed125640 10 monclient(hunting): get_auth_request con 0x7fe6e8071a50 auth_method 0 2024-03-27T21:41:50.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed125640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:50.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed125640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:50.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed125640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:50.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed926640 10 monclient(hunting): get_auth_request con 0x7fe6e813ec50 auth_method 0 2024-03-27T21:41:50.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed926640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:50.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed926640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:50.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed926640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:50.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed125640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:50.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed125640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:50.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed125640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:50.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed926640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:50.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed926640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:50.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed926640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:50.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed926640 10 monclient(hunting): handle_auth_done global_id 5184 payload 274 2024-03-27T21:41:50.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed926640 1 -- 172.21.15.60:0/1256044779 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe6e8071a50 msgr2=0x7fe6e80a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:50.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed926640 1 --2- 172.21.15.60:0/1256044779 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe6e8071a50 0x7fe6e80a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:50.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed926640 1 -- 172.21.15.60:0/1256044779 >> v1:172.21.15.60:6789/0 conn(0x7fe6e813e810 legacy=0x7fe6e8141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:50.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed926640 10 monclient: _finish_hunting 0 2024-03-27T21:41:50.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed926640 1 monclient: found mon.noname-a 2024-03-27T21:41:50.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed926640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:50.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed926640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:50.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed926640 1 -- 172.21.15.60:0/1256044779 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e8142280 con 0x7fe6e813ec50 2024-03-27T21:41:50.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.322+0000 7fe6ed125640 1 --2- 172.21.15.60:0/1256044779 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe6e8071a50 0x7fe6e80a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:50.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.323+0000 7fe6ed926640 1 --2- 172.21.15.60:0/1256044779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6e813ec50 0x7fe6e8141d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8009d90 tx=0x7fe6d802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b5f61c458c7e3e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:50.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.323+0000 7fe6e7fff640 1 -- 172.21.15.60:0/1256044779 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d802cc50 con 0x7fe6e813ec50 2024-03-27T21:41:50.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.323+0000 7fe6e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:50.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.323+0000 7fe6e7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:50.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.323+0000 7fe6e7fff640 10 monclient: dump: 2024-03-27T21:41:50.323 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:50.323 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:50.323 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:50.323 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:50.323 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:50.323 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:50.323 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:50.323 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:50.323 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:50.324 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:50.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.323+0000 7fe6e7fff640 10 monclient: _finish_auth 0 2024-03-27T21:41:50.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.323+0000 7fe6e7fff640 10 monclient: _check_auth_tickets 2024-03-27T21:41:50.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.323+0000 7fe6e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:50.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.323+0000 7fe6e7fff640 1 -- 172.21.15.60:0/1256044779 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d802cdb0 con 0x7fe6e813ec50 2024-03-27T21:41:50.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.323+0000 7fe6ef389640 5 monclient: authenticate success, global_id 5184 2024-03-27T21:41:50.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.323+0000 7fe6ef389640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:50.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.323+0000 7fe6e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:50.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.323+0000 7fe6e7fff640 1 -- 172.21.15.60:0/1256044779 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8034660 con 0x7fe6e813ec50 2024-03-27T21:41:50.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.323+0000 7fe6e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:50.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.323+0000 7fe6e7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:50.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.323+0000 7fe6e7fff640 10 monclient: dump: 2024-03-27T21:41:50.325 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:50.325 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:50.325 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:50.325 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:50.325 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:50.326 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:50.326 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:50.326 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:50.326 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:50.326 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:50.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.323+0000 7fe6ef389640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:50.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.323+0000 7fe6ef389640 4 set_mon_vals no callback set 2024-03-27T21:41:50.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.324+0000 7fe6ef389640 10 monclient: shutdown 2024-03-27T21:41:50.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.324+0000 7fe6ef389640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:50.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.324+0000 7fe6ef389640 1 -- 172.21.15.60:0/1256044779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6e813ec50 msgr2=0x7fe6e8141d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:50.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.324+0000 7fe6ef389640 1 --2- 172.21.15.60:0/1256044779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6e813ec50 0x7fe6e8141d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8009d90 tx=0x7fe6d802bea0 comp rx=0 tx=0).stop 2024-03-27T21:41:50.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.324+0000 7fe6ef389640 1 -- 172.21.15.60:0/1256044779 shutdown_connections 2024-03-27T21:41:50.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.324+0000 7fe6ef389640 1 --2- 172.21.15.60:0/1256044779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6e813ec50 0x7fe6e8141d40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:50.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.324+0000 7fe6ef389640 1 --2- 172.21.15.60:0/1256044779 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe6e8071a50 0x7fe6e80a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:50.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.324+0000 7fe6ef389640 1 -- 172.21.15.60:0/1256044779 >> 172.21.15.60:0/1256044779 conn(0x7fe6e80b7300 msgr2=0x7fe6e80b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:50.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.324+0000 7fe6ef389640 1 -- 172.21.15.60:0/1256044779 shutdown_connections 2024-03-27T21:41:50.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.324+0000 7fe6ef389640 1 -- 172.21.15.60:0/1256044779 wait complete. 2024-03-27T21:41:50.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.324+0000 7fe6ef389640 10 monclient: build_initial_monmap 2024-03-27T21:41:50.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.324+0000 7fe6ef389640 10 monclient: monmap: 2024-03-27T21:41:50.328 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:50.328 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:50.328 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:50.328 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:50.328 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:50.328 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:50.328 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:50.329 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:50.329 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:50.329 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:50.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 1 librados: starting msgr at 2024-03-27T21:41:50.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 1 librados: starting objecter 2024-03-27T21:41:50.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 1 Processor -- start 2024-03-27T21:41:50.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 1 -- start start 2024-03-27T21:41:50.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 1 librados: setting wanted keys 2024-03-27T21:41:50.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 1 librados: calling monclient init 2024-03-27T21:41:50.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 10 monclient: init 2024-03-27T21:41:50.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:50.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:41:50.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6e8071a50 0x7fe6e8135a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:50.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 10 monclient(hunting): picked mon.noname-a con 0x7fe6e8071a50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:50.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6e813e810 0x7fe6e8139f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:50.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 10 monclient(hunting): picked mon.noname-b con 0x7fe6e813e810 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:50.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe6e813ec50 0x7fe6e8139170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:50.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 10 monclient(hunting): picked mon.noname-c con 0x7fe6e813ec50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:50.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:50.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e81396b0 con 0x7fe6e8071a50 2024-03-27T21:41:50.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:50.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e81365c0 con 0x7fe6e813e810 2024-03-27T21:41:50.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:50.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e8139a00 con 0x7fe6e813ec50 2024-03-27T21:41:50.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:50.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ef389640 10 monclient(hunting): authenticate will time out at 951.546807s 2024-03-27T21:41:50.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ed926640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe6e813ec50 0x7fe6e8139170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:50.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ed125640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6e8071a50 0x7fe6e8135a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:50.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.325+0000 7fe6ed926640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe6e813ec50 0x7fe6e8139170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:32880/0 (socket says 172.21.15.60:32880) 2024-03-27T21:41:50.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed125640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6e8071a50 0x7fe6e8135a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41928/0 (socket says 172.21.15.60:41928) 2024-03-27T21:41:50.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed926640 1 -- 172.21.15.60:0/1371054605 learned_addr learned my addr 172.21.15.60:0/1371054605 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:50.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed926640 10 monclient(hunting): get_auth_request con 0x7fe6e813ec50 auth_method 0 2024-03-27T21:41:50.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed926640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:50.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed926640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:50.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed926640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:50.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed125640 10 monclient(hunting): get_auth_request con 0x7fe6e8071a50 auth_method 0 2024-03-27T21:41:50.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed125640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:50.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed125640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:50.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed125640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:50.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed926640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:50.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed926640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:50.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed926640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:50.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed125640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:50.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed125640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:50.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed125640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:50.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ec924640 1 --2- 172.21.15.60:0/1371054605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6e813e810 0x7fe6e8139f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:50.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ec924640 10 monclient(hunting): get_auth_request con 0x7fe6e813e810 auth_method 0 2024-03-27T21:41:50.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ec924640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:50.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ec924640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:50.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ec924640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:50.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed926640 10 monclient(hunting): handle_auth_done global_id 4928 payload 824 2024-03-27T21:41:50.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed926640 1 -- 172.21.15.60:0/1371054605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6e813e810 msgr2=0x7fe6e8139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:50.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed926640 1 --2- 172.21.15.60:0/1371054605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6e813e810 0x7fe6e8139f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:50.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed926640 1 -- 172.21.15.60:0/1371054605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6e8071a50 msgr2=0x7fe6e8135a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:50.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed926640 1 --2- 172.21.15.60:0/1371054605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6e8071a50 0x7fe6e8135a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:50.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed926640 10 monclient: _finish_hunting 0 2024-03-27T21:41:50.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed926640 1 monclient: found mon.noname-c 2024-03-27T21:41:50.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed926640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:50.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed926640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:50.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed926640 1 -- 172.21.15.60:0/1371054605 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e00094d0 con 0x7fe6e813ec50 2024-03-27T21:41:50.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed125640 1 --2- 172.21.15.60:0/1371054605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6e8071a50 0x7fe6e8135a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:50.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ed926640 1 --2- 172.21.15.60:0/1371054605 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe6e813ec50 0x7fe6e8139170 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8009d90 tx=0x7fe6d8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:50.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6e5ffb640 10 client.?.objecter ms_handle_connect 0x7fe6e813ec50 2024-03-27T21:41:50.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:50.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6e5ffb640 1 -- 172.21.15.60:0/1371054605 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8040650 con 0x7fe6e813ec50 2024-03-27T21:41:50.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:50.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6e5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:50.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6e5ffb640 10 monclient: dump: 2024-03-27T21:41:50.338 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:50.338 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:50.338 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:50.338 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:50.339 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:50.339 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:50.339 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:50.339 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:50.339 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:50.339 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:50.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6e5ffb640 10 monclient: _finish_auth 0 2024-03-27T21:41:50.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6e5ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:41:50.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:50.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6e5ffb640 1 -- 172.21.15.60:0/1371054605 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d80407b0 con 0x7fe6e813ec50 2024-03-27T21:41:50.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ef389640 5 monclient: authenticate success, global_id 4928 2024-03-27T21:41:50.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ef389640 10 monclient: _renew_subs 2024-03-27T21:41:50.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ef389640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:50.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ef389640 1 -- 172.21.15.60:0/1371054605 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d8009a70 con 0x7fe6e813ec50 2024-03-27T21:41:50.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:50.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6e5ffb640 1 -- 172.21.15.60:0/1371054605 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8040b60 con 0x7fe6e813ec50 2024-03-27T21:41:50.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:50.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6e5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:50.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6e5ffb640 10 monclient: dump: 2024-03-27T21:41:50.341 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:50.341 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:50.341 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:50.341 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:50.341 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:50.341 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:50.341 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:50.342 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:50.342 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:50.342 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:50.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ef389640 10 client.4928.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:50.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6e77fe640 4 set_mon_vals no callback set 2024-03-27T21:41:50.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ef389640 10 monclient: _renew_subs 2024-03-27T21:41:50.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ef389640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:50.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ef389640 1 -- 172.21.15.60:0/1371054605 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e81d9c80 con 0x7fe6e813ec50 2024-03-27T21:41:50.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.326+0000 7fe6ef389640 1 librados: init done 2024-03-27T21:41:50.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.327+0000 7fe6ef389640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:50.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.327+0000 7fe6ef389640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:50.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.327+0000 7fe6ef389640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:50.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.327+0000 7fe6ef389640 1 -- 172.21.15.60:0/1371054605 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e8139860 con 0x7fe6e813ec50 2024-03-27T21:41:50.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.328+0000 7fe6e5ffb640 1 -- 172.21.15.60:0/1371054605 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe6d8040cc0 con 0x7fe6e813ec50 2024-03-27T21:41:50.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.328+0000 7fe6e5ffb640 1 --2- 172.21.15.60:0/1371054605 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe6d406c590 0x7fe6d406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:50.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.328+0000 7fe6e5ffb640 1 -- 172.21.15.60:0/1371054605 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fe6d80abdf0 con 0x7fe6e813ec50 2024-03-27T21:41:50.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.328+0000 7fe6e5ffb640 10 client.4928.objecter ms_dispatch 0x7fe6e8000b90 osd_map(78..78 src has 1..78) v4 2024-03-27T21:41:50.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.328+0000 7fe6e5ffb640 3 client.4928.objecter handle_osd_map got epochs [78,78] > 0 2024-03-27T21:41:50.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.328+0000 7fe6e5ffb640 3 client.4928.objecter handle_osd_map decoding full epoch 78 2024-03-27T21:41:50.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.328+0000 7fe6ed125640 1 --2- 172.21.15.60:0/1371054605 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe6d406c590 0x7fe6d406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:50.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.328+0000 7fe6ed125640 10 monclient: get_auth_request con 0x7fe6d406c590 auth_method 0 2024-03-27T21:41:50.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.328+0000 7fe6e5ffb640 20 client.4928.objecter dump_active .. 0 homeless 2024-03-27T21:41:50.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.328+0000 7fe6ed125640 1 --2- 172.21.15.60:0/1371054605 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe6d406c590 0x7fe6d406ea50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0006ca0 tx=0x7fe6e0009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:50.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.328+0000 7fe6e5ffb640 10 client.4928.objecter ms_handle_connect 0x7fe6d406c590 2024-03-27T21:41:50.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.332+0000 7fe6e5ffb640 1 -- 172.21.15.60:0/1371054605 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe6e8139860 con 0x7fe6e813ec50 2024-03-27T21:41:50.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.332+0000 7fe6e5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:50.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.332+0000 7fe6e5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:50.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.525+0000 7fe6ef389640 10 start_mon_command cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}] 2024-03-27T21:41:50.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.525+0000 7fe6ef389640 10 monclient: _send_command 2 [{"prefix": "osd reweight", "id": 1, "weight": 1.0}] 2024-03-27T21:41:50.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.525+0000 7fe6ef389640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:50.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:50.525+0000 7fe6ef389640 1 -- 172.21.15.60:0/1371054605 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7fe6e8139860 con 0x7fe6e813ec50 2024-03-27T21:41:51.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.202+0000 7fe6e5ffb640 1 -- 172.21.15.60:0/1371054605 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v79) v1 ==== 113+0+0 (secure 0 0 0) 0x7fe6e8139860 con 0x7fe6e813ec50 2024-03-27T21:41:51.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.202+0000 7fe6e5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd reweight", "id": 1, "weight": 1.0}] 2024-03-27T21:41:51.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.202+0000 7fe6e5ffb640 10 monclient: _finish_command 2 = system:0 reweighted osd.1 to 1 (10000) 2024-03-27T21:41:51.189 INFO:teuthology.orchestra.run.smithi060.stderr:reweighted osd.1 to 1 (10000) 2024-03-27T21:41:51.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.205+0000 7fe6ef389640 10 librados: watch_flush enter 2024-03-27T21:41:51.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.205+0000 7fe6ef389640 10 librados: watch_flush exit 2024-03-27T21:41:51.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.205+0000 7fe6ef389640 20 client.4928.objecter shutdown clearing up homeless session... 2024-03-27T21:41:51.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.205+0000 7fe6ef389640 10 client.4928.objecter successfully canceled tick 2024-03-27T21:41:51.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.205+0000 7fe6ef389640 1 -- 172.21.15.60:0/1371054605 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe6d406c590 msgr2=0x7fe6d406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:51.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.205+0000 7fe6ef389640 1 --2- 172.21.15.60:0/1371054605 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe6d406c590 0x7fe6d406ea50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0006ca0 tx=0x7fe6e0009240 comp rx=0 tx=0).stop 2024-03-27T21:41:51.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.205+0000 7fe6ef389640 10 monclient: shutdown 2024-03-27T21:41:51.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.205+0000 7fe6ef389640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:51.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.205+0000 7fe6ef389640 1 -- 172.21.15.60:0/1371054605 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe6e813ec50 msgr2=0x7fe6e8139170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:51.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.205+0000 7fe6ef389640 1 --2- 172.21.15.60:0/1371054605 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe6e813ec50 0x7fe6e8139170 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8009d90 tx=0x7fe6d8004a40 comp rx=0 tx=0).stop 2024-03-27T21:41:51.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.205+0000 7fe6ef389640 1 -- 172.21.15.60:0/1371054605 shutdown_connections 2024-03-27T21:41:51.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.205+0000 7fe6ef389640 1 --2- 172.21.15.60:0/1371054605 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe6e813ec50 0x7fe6e8139170 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:51.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.205+0000 7fe6ef389640 1 --2- 172.21.15.60:0/1371054605 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe6d406c590 0x7fe6d406ea50 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:51.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.205+0000 7fe6ef389640 1 --2- 172.21.15.60:0/1371054605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6e813e810 0x7fe6e8139f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:51.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.205+0000 7fe6ef389640 1 --2- 172.21.15.60:0/1371054605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6e8071a50 0x7fe6e8135a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:51.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.205+0000 7fe6ef389640 1 -- 172.21.15.60:0/1371054605 >> 172.21.15.60:0/1371054605 conn(0x7fe6e80b7300 msgr2=0x7fe6e80b17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:51.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.205+0000 7fe6ef389640 1 -- 172.21.15.60:0/1371054605 shutdown_connections 2024-03-27T21:41:51.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.205+0000 7fe6ef389640 1 -- 172.21.15.60:0/1371054605 wait complete. 2024-03-27T21:41:51.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.205+0000 7fe6ef389640 1 librados: shutdown 2024-03-27T21:41:51.204 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-27T21:41:51.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.323+0000 7fcf23e10640 10 monclient: get_monmap_and_config 2024-03-27T21:41:51.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.323+0000 7fcf23e10640 10 monclient: build_initial_monmap 2024-03-27T21:41:51.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.323+0000 7fcf23e10640 10 monclient: monmap: 2024-03-27T21:41:51.310 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:51.310 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:51.310 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:51.324130+0000 2024-03-27T21:41:51.310 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:51.324130+0000 2024-03-27T21:41:51.310 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:51.310 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:51.310 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:51.311 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:51.311 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:51.311 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:51.311 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:51.311 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:51.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.323+0000 7fcf23e10640 1 Processor -- start 2024-03-27T21:41:51.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.323+0000 7fcf23e10640 1 -- start start 2024-03-27T21:41:51.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.323+0000 7fcf23e10640 10 monclient: init 2024-03-27T21:41:51.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf23e10640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:51.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf23e10640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-03-27T21:41:51.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf23e10640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcf1c0b40b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:51.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf23e10640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcf1c13b020 0x7fcf1c13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:51.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf23e10640 10 monclient(hunting): picked mon.noname-c con 0x7fcf1c13b020 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:51.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf23e10640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcf1c1416f0 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:51.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf23e10640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c0b3830 con 0x7fcf1c0b40b0 2024-03-27T21:41:51.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf23e10640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c0aa520 con 0x7fcf1c1416f0 2024-03-27T21:41:51.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf23e10640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:51.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf23e10640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf1c0b4490 con 0x7fcf1c13b020 2024-03-27T21:41:51.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf23e10640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:51.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf23e10640 10 monclient(hunting): authenticate will time out at 952.545398s 2024-03-27T21:41:51.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf21bac640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcf1c0b40b0 0x7fcf1c0a8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:48034/0 (socket says 172.21.15.60:48034) 2024-03-27T21:41:51.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf21bac640 1 -- 172.21.15.60:0/1855353602 learned_addr learned my addr 172.21.15.60:0/1855353602 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:51.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf20baa640 1 -- 172.21.15.60:0/1855353602 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447959825 0 0) 0x7fcf1c0b3830 con 0x7fcf1c0b40b0 2024-03-27T21:41:51.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf20baa640 10 monclient(hunting): _negotiate 2024-03-27T21:41:51.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf20baa640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:51.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf20baa640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:51.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf20baa640 10 monclient(hunting): my global_id is 5193 2024-03-27T21:41:51.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf20baa640 1 -- 172.21.15.60:0/1855353602 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf04003520 con 0x7fcf1c0b40b0 2024-03-27T21:41:51.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf213ab640 1 --2- 172.21.15.60:0/1855353602 >> v2:172.21.15.191:3300/0 conn(0x7fcf1c13b020 0x7fcf1c13b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:51.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf213ab640 10 monclient(hunting): get_auth_request con 0x7fcf1c13b020 auth_method 0 2024-03-27T21:41:51.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf213ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:51.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf213ab640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:51.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.324+0000 7fcf213ab640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:51.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf213ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:51.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf213ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:51.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf213ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:51.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 1 -- 172.21.15.60:0/1855353602 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1658101296 0 0) 0x7fcf04003520 con 0x7fcf1c0b40b0 2024-03-27T21:41:51.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 1 -- 172.21.15.60:0/1855353602 >> v2:172.21.15.191:3300/0 conn(0x7fcf1c13b020 msgr2=0x7fcf1c13b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:51.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 1 --2- 172.21.15.60:0/1855353602 >> v2:172.21.15.191:3300/0 conn(0x7fcf1c13b020 0x7fcf1c13b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:51.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 1 -- 172.21.15.60:0/1855353602 >> v1:172.21.15.191:6789/0 conn(0x7fcf1c1416f0 legacy=0x7fcf1c141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:51.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 10 monclient: _finish_hunting 0 2024-03-27T21:41:51.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 1 monclient: found mon.noname-a-legacy 2024-03-27T21:41:51.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:51.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.60:6789/0 2024-03-27T21:41:51.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 1 -- 172.21.15.60:0/1855353602 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c0a9470 con 0x7fcf1c0b40b0 2024-03-27T21:41:51.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 10 monclient: _finish_auth 0 2024-03-27T21:41:51.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 10 monclient: _check_auth_tickets 2024-03-27T21:41:51.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:51.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 1 -- 172.21.15.60:0/1855353602 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2933500084 0 0) 0x7fcf14001670 con 0x7fcf1c0b40b0 2024-03-27T21:41:51.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf23e10640 5 monclient: authenticate success, global_id 5193 2024-03-27T21:41:51.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf23e10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:51.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:51.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-03-27T21:41:51.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 10 monclient: dump: 2024-03-27T21:41:51.328 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:51.328 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:51.328 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:51.328 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:51.328 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:51.328 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:51.328 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:51.329 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:51.329 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:51.329 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:51.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] but i'm connected to v1:172.21.15.60:6789/0, reconnecting 2024-03-27T21:41:51.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:51.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 1 -- 172.21.15.60:0/1855353602 >> v1:172.21.15.60:6789/0 conn(0x7fcf1c0b40b0 legacy=0x7fcf1c0a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:51.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:41:51.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 1 --2- 172.21.15.60:0/1855353602 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf04003970 0x7fcf04003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:51.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 10 monclient(hunting): picked mon.c con 0x7fcf04003970 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:51.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 1 --2- 172.21.15.60:0/1855353602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf040047c0 0x7fcf04004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:51.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 10 monclient(hunting): picked mon.b con 0x7fcf040047c0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:51.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 1 --2- 172.21.15.60:0/1855353602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf04005180 0x7fcf0400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:51.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 10 monclient(hunting): picked mon.a con 0x7fcf04005180 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:51.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:51.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 1 -- 172.21.15.60:0/1855353602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c0b4490 con 0x7fcf04005180 2024-03-27T21:41:51.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:51.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 1 -- 172.21.15.60:0/1855353602 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0400c020 con 0x7fcf040047c0 2024-03-27T21:41:51.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:51.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 1 -- 172.21.15.60:0/1855353602 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0400c1e0 con 0x7fcf04003970 2024-03-27T21:41:51.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:51.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf20baa640 10 monclient(hunting): _finish_auth 0 2024-03-27T21:41:51.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf23e10640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:51.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf223ad640 1 --2- 172.21.15.60:0/1855353602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf04005180 0x7fcf0400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:51.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf21bac640 1 --2- 172.21.15.60:0/1855353602 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf04003970 0x7fcf04003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:51.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf223ad640 10 monclient(hunting): get_auth_request con 0x7fcf04005180 auth_method 0 2024-03-27T21:41:51.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf223ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:51.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf223ad640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:51.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf223ad640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:41:51.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf213ab640 1 --2- 172.21.15.60:0/1855353602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf040047c0 0x7fcf04004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:51.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf213ab640 10 monclient(hunting): get_auth_request con 0x7fcf040047c0 auth_method 0 2024-03-27T21:41:51.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf213ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:51.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf213ab640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:51.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf213ab640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:41:51.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf21bac640 10 monclient(hunting): get_auth_request con 0x7fcf04003970 auth_method 0 2024-03-27T21:41:51.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf21bac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:51.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf21bac640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:51.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf21bac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:41:51.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf223ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:51.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf223ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:51.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf223ad640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:41:51.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf213ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:51.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf213ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:51.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf213ab640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:41:51.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf21bac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:51.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf21bac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:51.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf21bac640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:41:51.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf223ad640 10 monclient(hunting): handle_auth_done global_id 5193 payload 293 2024-03-27T21:41:51.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf223ad640 1 -- 172.21.15.60:0/1855353602 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf04003970 msgr2=0x7fcf04003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:51.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf223ad640 1 --2- 172.21.15.60:0/1855353602 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf04003970 0x7fcf04003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:51.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf223ad640 1 -- 172.21.15.60:0/1855353602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf040047c0 msgr2=0x7fcf04004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:51.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf223ad640 1 --2- 172.21.15.60:0/1855353602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf040047c0 0x7fcf04004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:51.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf223ad640 10 monclient: _finish_hunting 0 2024-03-27T21:41:51.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf223ad640 1 monclient: found mon.a 2024-03-27T21:41:51.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf223ad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:51.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf223ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:51.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf223ad640 1 -- 172.21.15.60:0/1855353602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf04003520 con 0x7fcf04005180 2024-03-27T21:41:51.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf223ad640 10 monclient: _finish_auth 0 2024-03-27T21:41:51.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf223ad640 10 monclient: _check_auth_tickets 2024-03-27T21:41:51.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.325+0000 7fcf223ad640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:51.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.326+0000 7fcf223ad640 1 -- 172.21.15.60:0/1855353602 reap_dead start 2024-03-27T21:41:51.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.326+0000 7fcf223ad640 1 --2- 172.21.15.60:0/1855353602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf04005180 0x7fcf0400b980 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c001e50 tx=0x7fcf040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fd313a67613acfd server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:51.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.326+0000 7fcf20baa640 1 -- 172.21.15.60:0/1855353602 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c003b70 con 0x7fcf04005180 2024-03-27T21:41:51.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.326+0000 7fcf20baa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:51.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.326+0000 7fcf20baa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:51.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.326+0000 7fcf20baa640 10 monclient: dump: 2024-03-27T21:41:51.336 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:51.336 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:51.337 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:51.337 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:51.337 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:51.337 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:51.337 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:51.337 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:51.337 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:51.337 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:51.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.326+0000 7fcf23e10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:51.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.326+0000 7fcf20baa640 1 -- 172.21.15.60:0/1855353602 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0c003cf0 con 0x7fcf04005180 2024-03-27T21:41:51.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.326+0000 7fcf20baa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:51.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.326+0000 7fcf23e10640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:51.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.326+0000 7fcf23e10640 4 set_mon_vals no callback set 2024-03-27T21:41:51.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.326+0000 7fcf20baa640 1 -- 172.21.15.60:0/1855353602 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c005310 con 0x7fcf04005180 2024-03-27T21:41:51.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.326+0000 7fcf20baa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:51.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.326+0000 7fcf20baa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:51.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.326+0000 7fcf20baa640 10 monclient: dump: 2024-03-27T21:41:51.339 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:51.339 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:51.339 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:51.339 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:51.339 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:51.339 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:51.339 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:51.339 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:51.339 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:51.340 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:51.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.326+0000 7fcf23e10640 10 monclient: shutdown 2024-03-27T21:41:51.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.326+0000 7fcf23e10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:51.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.326+0000 7fcf23e10640 1 -- 172.21.15.60:0/1855353602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf04005180 msgr2=0x7fcf0400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:51.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.326+0000 7fcf23e10640 1 --2- 172.21.15.60:0/1855353602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf04005180 0x7fcf0400b980 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c001e50 tx=0x7fcf040057f0 comp rx=0 tx=0).stop 2024-03-27T21:41:51.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.326+0000 7fcf23e10640 1 -- 172.21.15.60:0/1855353602 shutdown_connections 2024-03-27T21:41:51.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.327+0000 7fcf23e10640 1 --2- 172.21.15.60:0/1855353602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf04005180 0x7fcf0400b980 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:51.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.327+0000 7fcf23e10640 1 -- 172.21.15.60:0/1855353602 >> 172.21.15.60:0/1855353602 conn(0x7fcf1c0608e0 msgr2=0x7fcf1c060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:51.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.327+0000 7fcf23e10640 1 -- 172.21.15.60:0/1855353602 shutdown_connections 2024-03-27T21:41:51.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.327+0000 7fcf23e10640 1 -- 172.21.15.60:0/1855353602 wait complete. 2024-03-27T21:41:51.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.327+0000 7fcf23e10640 10 monclient: build_initial_monmap 2024-03-27T21:41:51.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.327+0000 7fcf23e10640 10 monclient: monmap: 2024-03-27T21:41:51.341 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:51.341 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:51.341 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:51.341 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:51.341 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:51.342 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:51.342 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:51.342 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:51.342 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:51.342 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:51.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.327+0000 7fcf23e10640 1 librados: starting msgr at 2024-03-27T21:41:51.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.327+0000 7fcf23e10640 1 librados: starting objecter 2024-03-27T21:41:51.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf23e10640 1 Processor -- start 2024-03-27T21:41:51.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf23e10640 1 -- start start 2024-03-27T21:41:51.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf23e10640 1 librados: setting wanted keys 2024-03-27T21:41:51.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf23e10640 1 librados: calling monclient init 2024-03-27T21:41:51.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf23e10640 10 monclient: init 2024-03-27T21:41:51.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf23e10640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:51.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf23e10640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:41:51.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf23e10640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf04005180 0x7fcf1c139e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:51.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf23e10640 10 monclient(hunting): picked mon.noname-c con 0x7fcf04005180 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:51.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf23e10640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf1c0b1fe0 0x7fcf1c13d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:51.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf23e10640 10 monclient(hunting): picked mon.noname-b con 0x7fcf1c0b1fe0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf23e10640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf1c1d9da0 0x7fcf1c13b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf23e10640 10 monclient(hunting): picked mon.noname-a con 0x7fcf1c1d9da0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf23e10640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf23e10640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c1d9980 con 0x7fcf1c1d9da0 2024-03-27T21:41:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf23e10640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf23e10640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c13e910 con 0x7fcf1c0b1fe0 2024-03-27T21:41:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf23e10640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf23e10640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c1d95e0 con 0x7fcf04005180 2024-03-27T21:41:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf23e10640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf23e10640 10 monclient(hunting): authenticate will time out at 952.549616s 2024-03-27T21:41:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf21bac640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf04005180 0x7fcf1c139e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf21bac640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf04005180 0x7fcf1c139e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:32892/0 (socket says 172.21.15.60:32892) 2024-03-27T21:41:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf21bac640 1 -- 172.21.15.60:0/1484541513 learned_addr learned my addr 172.21.15.60:0/1484541513 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf21bac640 10 monclient(hunting): get_auth_request con 0x7fcf04005180 auth_method 0 2024-03-27T21:41:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf21bac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf21bac640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf21bac640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf223ad640 1 --2- 172.21.15.60:0/1484541513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf1c1d9da0 0x7fcf1c13b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf223ad640 10 monclient(hunting): get_auth_request con 0x7fcf1c1d9da0 auth_method 0 2024-03-27T21:41:51.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf223ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:51.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf223ad640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:51.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf223ad640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:51.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf21bac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:51.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf21bac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:51.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf21bac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:51.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf223ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:51.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf223ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:51.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.328+0000 7fcf223ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:51.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf21bac640 10 monclient(hunting): handle_auth_done global_id 4931 payload 824 2024-03-27T21:41:51.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf21bac640 1 -- 172.21.15.60:0/1484541513 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf1c0b1fe0 msgr2=0x7fcf1c13d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:51.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf213ab640 1 --2- 172.21.15.60:0/1484541513 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf1c0b1fe0 0x7fcf1c13d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:51.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf21bac640 1 --2- 172.21.15.60:0/1484541513 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf1c0b1fe0 0x7fcf1c13d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:51.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf21bac640 1 -- 172.21.15.60:0/1484541513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf1c1d9da0 msgr2=0x7fcf1c13b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:51.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf21bac640 1 --2- 172.21.15.60:0/1484541513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf1c1d9da0 0x7fcf1c13b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:51.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf21bac640 10 monclient: _finish_hunting 0 2024-03-27T21:41:51.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf21bac640 1 monclient: found mon.noname-c 2024-03-27T21:41:51.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf21bac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:51.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf21bac640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:51.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf21bac640 1 -- 172.21.15.60:0/1484541513 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c0b2a50 con 0x7fcf04005180 2024-03-27T21:41:51.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf223ad640 1 --2- 172.21.15.60:0/1484541513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf1c1d9da0 0x7fcf1c13b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:51.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf21bac640 1 --2- 172.21.15.60:0/1484541513 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf04005180 0x7fcf1c139e70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcf140070e0 tx=0x7fcf140075a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:51.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf1a7fc640 10 client.?.objecter ms_handle_connect 0x7fcf04005180 2024-03-27T21:41:51.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:51.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf1a7fc640 1 -- 172.21.15.60:0/1484541513 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1400f9c0 con 0x7fcf04005180 2024-03-27T21:41:51.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:51.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf1a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:51.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf1a7fc640 10 monclient: dump: 2024-03-27T21:41:51.349 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:51.349 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:51.349 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:51.349 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:51.349 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:51.350 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:51.350 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:51.350 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:51.350 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:51.350 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:51.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf1a7fc640 10 monclient: _finish_auth 0 2024-03-27T21:41:51.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf1a7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:41:51.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:51.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf23e10640 5 monclient: authenticate success, global_id 4931 2024-03-27T21:41:51.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf1a7fc640 1 -- 172.21.15.60:0/1484541513 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1400fb20 con 0x7fcf04005180 2024-03-27T21:41:51.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:51.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf1a7fc640 1 -- 172.21.15.60:0/1484541513 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1400fed0 con 0x7fcf04005180 2024-03-27T21:41:51.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf23e10640 10 monclient: _renew_subs 2024-03-27T21:41:51.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf23e10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:51.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf23e10640 1 -- 172.21.15.60:0/1484541513 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf1c0ab520 con 0x7fcf04005180 2024-03-27T21:41:51.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:51.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf1bfff640 4 set_mon_vals no callback set 2024-03-27T21:41:51.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:51.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf1a7fc640 10 monclient: dump: 2024-03-27T21:41:51.352 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:51.352 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:51.352 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:51.352 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:51.352 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:51.352 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:51.352 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:51.352 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:51.353 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:51.353 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:51.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf23e10640 10 client.4931.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:51.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf23e10640 10 monclient: _renew_subs 2024-03-27T21:41:51.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf23e10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:51.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf23e10640 1 -- 172.21.15.60:0/1484541513 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1c1da400 con 0x7fcf04005180 2024-03-27T21:41:51.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.329+0000 7fcf23e10640 1 librados: init done 2024-03-27T21:41:51.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.330+0000 7fcf23e10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:51.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.330+0000 7fcf23e10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:51.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.330+0000 7fcf23e10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:51.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.330+0000 7fcf23e10640 1 -- 172.21.15.60:0/1484541513 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1c1d9ac0 con 0x7fcf04005180 2024-03-27T21:41:51.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.330+0000 7fcf1a7fc640 1 -- 172.21.15.60:0/1484541513 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcf1401d360 con 0x7fcf04005180 2024-03-27T21:41:51.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.331+0000 7fcf1a7fc640 1 --2- 172.21.15.60:0/1484541513 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcf0806c5c0 0x7fcf0806ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:51.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.331+0000 7fcf213ab640 1 --2- 172.21.15.60:0/1484541513 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcf0806c5c0 0x7fcf0806ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:51.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.331+0000 7fcf213ab640 10 monclient: get_auth_request con 0x7fcf0806c5c0 auth_method 0 2024-03-27T21:41:51.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.331+0000 7fcf1a7fc640 1 -- 172.21.15.60:0/1484541513 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fcf14088960 con 0x7fcf04005180 2024-03-27T21:41:51.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.331+0000 7fcf1a7fc640 10 client.4931.objecter ms_dispatch 0x7fcf1c000b90 osd_map(79..79 src has 1..79) v4 2024-03-27T21:41:51.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.331+0000 7fcf1a7fc640 3 client.4931.objecter handle_osd_map got epochs [79,79] > 0 2024-03-27T21:41:51.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.331+0000 7fcf1a7fc640 3 client.4931.objecter handle_osd_map decoding full epoch 79 2024-03-27T21:41:51.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.331+0000 7fcf1a7fc640 20 client.4931.objecter dump_active .. 0 homeless 2024-03-27T21:41:51.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.331+0000 7fcf1a7fc640 10 client.4931.objecter ms_handle_connect 0x7fcf0806c5c0 2024-03-27T21:41:51.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.331+0000 7fcf213ab640 1 --2- 172.21.15.60:0/1484541513 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcf0806c5c0 0x7fcf0806ea80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0b2e40 tx=0x7fcf10008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:51.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.335+0000 7fcf1a7fc640 1 -- 172.21.15.60:0/1484541513 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcf1c1d9ac0 con 0x7fcf04005180 2024-03-27T21:41:51.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.335+0000 7fcf1a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:51.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.335+0000 7fcf1a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:51.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.526+0000 7fcf23e10640 10 start_mon_command cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}] 2024-03-27T21:41:51.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.526+0000 7fcf23e10640 10 monclient: _send_command 2 [{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}] 2024-03-27T21:41:51.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.526+0000 7fcf23e10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:51.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:51.526+0000 7fcf23e10640 1 -- 172.21.15.60:0/1484541513 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} v 0) v1 -- 0x7fcf1c1d9ac0 con 0x7fcf04005180 2024-03-27T21:41:52.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.211+0000 7fcf1a7fc640 1 -- 172.21.15.60:0/1484541513 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]=0 set osd.1 primary-affinity to 1 (8655362) v80) v1 ==== 133+0+0 (secure 0 0 0) 0x7fcf1c1d9ac0 con 0x7fcf04005180 2024-03-27T21:41:52.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.211+0000 7fcf1a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}] 2024-03-27T21:41:52.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.211+0000 7fcf1a7fc640 10 monclient: _finish_command 2 = system:0 set osd.1 primary-affinity to 1 (8655362) 2024-03-27T21:41:52.198 INFO:teuthology.orchestra.run.smithi060.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-27T21:41:52.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.214+0000 7fcf23e10640 10 librados: watch_flush enter 2024-03-27T21:41:52.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.214+0000 7fcf23e10640 10 librados: watch_flush exit 2024-03-27T21:41:52.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.214+0000 7fcf23e10640 20 client.4931.objecter shutdown clearing up homeless session... 2024-03-27T21:41:52.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.214+0000 7fcf23e10640 10 client.4931.objecter successfully canceled tick 2024-03-27T21:41:52.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.214+0000 7fcf23e10640 1 -- 172.21.15.60:0/1484541513 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcf0806c5c0 msgr2=0x7fcf0806ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:52.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.214+0000 7fcf23e10640 1 --2- 172.21.15.60:0/1484541513 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcf0806c5c0 0x7fcf0806ea80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0b2e40 tx=0x7fcf10008040 comp rx=0 tx=0).stop 2024-03-27T21:41:52.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.214+0000 7fcf23e10640 10 monclient: shutdown 2024-03-27T21:41:52.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.214+0000 7fcf23e10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:52.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.214+0000 7fcf23e10640 1 -- 172.21.15.60:0/1484541513 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf04005180 msgr2=0x7fcf1c139e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:52.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.214+0000 7fcf23e10640 1 --2- 172.21.15.60:0/1484541513 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf04005180 0x7fcf1c139e70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcf140070e0 tx=0x7fcf140075a0 comp rx=0 tx=0).stop 2024-03-27T21:41:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.214+0000 7fcf23e10640 1 -- 172.21.15.60:0/1484541513 shutdown_connections 2024-03-27T21:41:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.214+0000 7fcf23e10640 1 --2- 172.21.15.60:0/1484541513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf1c1d9da0 0x7fcf1c13b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.214+0000 7fcf23e10640 1 --2- 172.21.15.60:0/1484541513 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcf0806c5c0 0x7fcf0806ea80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.214+0000 7fcf23e10640 1 --2- 172.21.15.60:0/1484541513 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf1c0b1fe0 0x7fcf1c13d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.214+0000 7fcf23e10640 1 --2- 172.21.15.60:0/1484541513 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf04005180 0x7fcf1c139e70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.214+0000 7fcf23e10640 1 -- 172.21.15.60:0/1484541513 >> 172.21.15.60:0/1484541513 conn(0x7fcf1c0608e0 msgr2=0x7fcf1c0b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.214+0000 7fcf23e10640 1 -- 172.21.15.60:0/1484541513 shutdown_connections 2024-03-27T21:41:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.214+0000 7fcf23e10640 1 -- 172.21.15.60:0/1484541513 wait complete. 2024-03-27T21:41:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.214+0000 7fcf23e10640 1 librados: shutdown 2024-03-27T21:41:52.214 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-27T21:41:52.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.336+0000 7fb80a1dd640 10 monclient: get_monmap_and_config 2024-03-27T21:41:52.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.336+0000 7fb80a1dd640 10 monclient: build_initial_monmap 2024-03-27T21:41:52.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.336+0000 7fb80a1dd640 10 monclient: monmap: 2024-03-27T21:41:52.323 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:52.323 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:52.323 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:52.337670+0000 2024-03-27T21:41:52.323 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:52.337670+0000 2024-03-27T21:41:52.323 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:52.323 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:52.324 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:52.324 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:52.324 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:52.324 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:52.324 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:52.324 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:52.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.337+0000 7fb80a1dd640 1 Processor -- start 2024-03-27T21:41:52.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.337+0000 7fb80a1dd640 1 -- start start 2024-03-27T21:41:52.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.337+0000 7fb80a1dd640 10 monclient: init 2024-03-27T21:41:52.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.337+0000 7fb80a1dd640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:52.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.337+0000 7fb80a1dd640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-03-27T21:41:52.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.337+0000 7fb80a1dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb804071a50 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:52.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.337+0000 7fb80a1dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb80413ef20 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:52.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb80a1dd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb8040a87e0 0x7fb804141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:52.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb80a1dd640 10 monclient(hunting): picked mon.noname-c con 0x7fb8040a87e0 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb80a1dd640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8040a8e90 con 0x7fb804071a50 2024-03-27T21:41:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb80a1dd640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8040aa040 con 0x7fb80413ef20 2024-03-27T21:41:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb80a1dd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb80a1dd640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb804071e30 con 0x7fb8040a87e0 2024-03-27T21:41:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb80a1dd640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb80a1dd640 10 monclient(hunting): authenticate will time out at 953.558968s 2024-03-27T21:41:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8037fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb804071a50 0x7fb80413e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47248/0 (socket says 172.21.15.60:47248) 2024-03-27T21:41:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8037fe640 1 -- 172.21.15.60:0/2686590428 learned_addr learned my addr 172.21.15.60:0/2686590428 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:52.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb803fff640 1 --2- 172.21.15.60:0/2686590428 >> v2:172.21.15.191:3300/0 conn(0x7fb8040a87e0 0x7fb804141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb803fff640 10 monclient(hunting): get_auth_request con 0x7fb8040a87e0 auth_method 0 2024-03-27T21:41:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb803fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb803fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb803fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 1 -- 172.21.15.60:0/2686590428 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417514051 0 0) 0x7fb8040a8e90 con 0x7fb804071a50 2024-03-27T21:41:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 10 monclient(hunting): _negotiate 2024-03-27T21:41:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:52.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 10 monclient(hunting): my global_id is 5199 2024-03-27T21:41:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 1 -- 172.21.15.60:0/2686590428 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7ec003520 con 0x7fb804071a50 2024-03-27T21:41:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb803fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb803fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb803fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 1 -- 172.21.15.60:0/2686590428 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 10871220 0 0) 0x7fb7ec003520 con 0x7fb804071a50 2024-03-27T21:41:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 1 -- 172.21.15.60:0/2686590428 >> v2:172.21.15.191:3300/0 conn(0x7fb8040a87e0 msgr2=0x7fb804141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:52.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 1 --2- 172.21.15.60:0/2686590428 >> v2:172.21.15.191:3300/0 conn(0x7fb8040a87e0 0x7fb804141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 1 -- 172.21.15.60:0/2686590428 >> v1:172.21.15.191:6789/0 conn(0x7fb80413ef20 legacy=0x7fb804141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 10 monclient: _finish_hunting 0 2024-03-27T21:41:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 1 monclient: found mon.noname-a-legacy 2024-03-27T21:41:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.60:6789/0 2024-03-27T21:41:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 1 -- 172.21.15.60:0/2686590428 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb804142280 con 0x7fb804071a50 2024-03-27T21:41:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 10 monclient: _finish_auth 0 2024-03-27T21:41:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 10 monclient: _check_auth_tickets 2024-03-27T21:41:52.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb80a1dd640 5 monclient: authenticate success, global_id 5199 2024-03-27T21:41:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 1 -- 172.21.15.60:0/2686590428 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2933500084 0 0) 0x7fb7f8001670 con 0x7fb804071a50 2024-03-27T21:41:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-03-27T21:41:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 10 monclient: dump: 2024-03-27T21:41:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:52.340 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:52.341 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] but i'm connected to v1:172.21.15.60:6789/0, reconnecting 2024-03-27T21:41:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 1 -- 172.21.15.60:0/2686590428 >> v1:172.21.15.60:6789/0 conn(0x7fb804071a50 legacy=0x7fb80413e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:52.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:41:52.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 1 --2- 172.21.15.60:0/2686590428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7ec003970 0x7fb7ec003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:52.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 10 monclient(hunting): picked mon.a con 0x7fb7ec003970 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:52.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 1 --2- 172.21.15.60:0/2686590428 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb7ec004370 0x7fb7ec004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:52.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 10 monclient(hunting): picked mon.c con 0x7fb7ec004370 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:52.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 1 --2- 172.21.15.60:0/2686590428 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7ec005040 0x7fb7ec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:52.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 10 monclient(hunting): picked mon.b con 0x7fb7ec005040 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:52.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:52.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 1 -- 172.21.15.60:0/2686590428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb804071e30 con 0x7fb7ec003970 2024-03-27T21:41:52.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:52.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 1 -- 172.21.15.60:0/2686590428 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec00bfa0 con 0x7fb7ec005040 2024-03-27T21:41:52.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:52.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 1 -- 172.21.15.60:0/2686590428 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec00c160 con 0x7fb7ec004370 2024-03-27T21:41:52.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:52.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8027fc640 10 monclient(hunting): _finish_auth 0 2024-03-27T21:41:52.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb80a1dd640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:52.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8037fe640 1 --2- 172.21.15.60:0/2686590428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7ec003970 0x7fb7ec003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:52.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8037fe640 10 monclient(hunting): get_auth_request con 0x7fb7ec003970 auth_method 0 2024-03-27T21:41:52.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:52.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8037fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:52.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb802ffd640 1 --2- 172.21.15.60:0/2686590428 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb7ec004370 0x7fb7ec004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:52.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8037fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:41:52.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb802ffd640 10 monclient(hunting): get_auth_request con 0x7fb7ec004370 auth_method 0 2024-03-27T21:41:52.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb802ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:52.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb802ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:52.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb802ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:41:52.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb803fff640 1 --2- 172.21.15.60:0/2686590428 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7ec005040 0x7fb7ec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:52.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:52.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.338+0000 7fb8037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:52.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb8037fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:41:52.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb803fff640 10 monclient(hunting): get_auth_request con 0x7fb7ec005040 auth_method 0 2024-03-27T21:41:52.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb803fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:52.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb803fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:52.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb803fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:41:52.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb802ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:52.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb802ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:52.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb802ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:41:52.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb803fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:52.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb803fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:52.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb803fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:41:52.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb8037fe640 10 monclient(hunting): handle_auth_done global_id 5199 payload 293 2024-03-27T21:41:52.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb8037fe640 1 -- 172.21.15.60:0/2686590428 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb7ec004370 msgr2=0x7fb7ec004840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:52.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb8037fe640 1 --2- 172.21.15.60:0/2686590428 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb7ec004370 0x7fb7ec004840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:52.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb8037fe640 1 -- 172.21.15.60:0/2686590428 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7ec005040 msgr2=0x7fb7ec00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:52.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb8037fe640 1 --2- 172.21.15.60:0/2686590428 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7ec005040 0x7fb7ec00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:52.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb8037fe640 10 monclient: _finish_hunting 0 2024-03-27T21:41:52.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb8037fe640 1 monclient: found mon.a 2024-03-27T21:41:52.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb8037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:52.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb8037fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:52.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb8037fe640 1 -- 172.21.15.60:0/2686590428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ec003520 con 0x7fb7ec003970 2024-03-27T21:41:52.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb8037fe640 10 monclient: _finish_auth 0 2024-03-27T21:41:52.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb803fff640 1 -- 172.21.15.60:0/2686590428 reap_dead start 2024-03-27T21:41:52.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb8037fe640 10 monclient: _check_auth_tickets 2024-03-27T21:41:52.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb8037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:52.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.339+0000 7fb8037fe640 1 --2- 172.21.15.60:0/2686590428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7ec003970 0x7fb7ec003d50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800c210 tx=0x7fb7ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71ebc27d92ae2de0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:52.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.340+0000 7fb8027fc640 1 -- 172.21.15.60:0/2686590428 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f8005ad0 con 0x7fb7ec003970 2024-03-27T21:41:52.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.340+0000 7fb8027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:52.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.340+0000 7fb8027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:52.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.340+0000 7fb8027fc640 10 monclient: dump: 2024-03-27T21:41:52.348 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:52.348 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:52.348 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:52.348 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:52.349 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:52.349 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:52.349 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:52.349 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:52.349 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:52.349 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:52.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.340+0000 7fb8027fc640 1 -- 172.21.15.60:0/2686590428 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f8005c50 con 0x7fb7ec003970 2024-03-27T21:41:52.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.340+0000 7fb8027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:52.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.340+0000 7fb8027fc640 1 -- 172.21.15.60:0/2686590428 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f802c870 con 0x7fb7ec003970 2024-03-27T21:41:52.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.340+0000 7fb80a1dd640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:52.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.340+0000 7fb80a1dd640 4 set_mon_vals no callback set 2024-03-27T21:41:52.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.340+0000 7fb8027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:52.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.340+0000 7fb8027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:52.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.340+0000 7fb8027fc640 10 monclient: dump: 2024-03-27T21:41:52.353 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:52.353 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:52.354 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:52.354 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:52.354 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:52.354 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:52.354 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:52.354 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:52.354 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:52.354 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:52.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.340+0000 7fb80a1dd640 10 monclient: shutdown 2024-03-27T21:41:52.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.340+0000 7fb80a1dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:52.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.340+0000 7fb80a1dd640 1 -- 172.21.15.60:0/2686590428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7ec003970 msgr2=0x7fb7ec003d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:52.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.340+0000 7fb80a1dd640 1 --2- 172.21.15.60:0/2686590428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7ec003970 0x7fb7ec003d50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800c210 tx=0x7fb7ec0057f0 comp rx=0 tx=0).stop 2024-03-27T21:41:52.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.340+0000 7fb80a1dd640 1 -- 172.21.15.60:0/2686590428 shutdown_connections 2024-03-27T21:41:52.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.340+0000 7fb80a1dd640 1 --2- 172.21.15.60:0/2686590428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7ec003970 0x7fb7ec003d50 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:52.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.340+0000 7fb80a1dd640 1 -- 172.21.15.60:0/2686590428 >> 172.21.15.60:0/2686590428 conn(0x7fb8040b7300 msgr2=0x7fb8040b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:52.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.340+0000 7fb80a1dd640 1 -- 172.21.15.60:0/2686590428 shutdown_connections 2024-03-27T21:41:52.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.340+0000 7fb80a1dd640 1 -- 172.21.15.60:0/2686590428 wait complete. 2024-03-27T21:41:52.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.341+0000 7fb80a1dd640 10 monclient: build_initial_monmap 2024-03-27T21:41:52.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.341+0000 7fb80a1dd640 10 monclient: monmap: 2024-03-27T21:41:52.356 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:52.356 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:52.356 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:52.356 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:52.356 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:52.356 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:52.356 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:52.357 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:52.357 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:52.357 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:52.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.341+0000 7fb80a1dd640 1 librados: starting msgr at 2024-03-27T21:41:52.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.341+0000 7fb80a1dd640 1 librados: starting objecter 2024-03-27T21:41:52.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.341+0000 7fb80a1dd640 1 Processor -- start 2024-03-27T21:41:52.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.341+0000 7fb80a1dd640 1 -- start start 2024-03-27T21:41:52.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.341+0000 7fb80a1dd640 1 librados: setting wanted keys 2024-03-27T21:41:52.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.341+0000 7fb80a1dd640 1 librados: calling monclient init 2024-03-27T21:41:52.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.341+0000 7fb80a1dd640 10 monclient: init 2024-03-27T21:41:52.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb80a1dd640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:52.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb80a1dd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:41:52.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb80a1dd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb7ec003970 0x7fb8040ade50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:52.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb80a1dd640 10 monclient(hunting): picked mon.noname-c con 0x7fb7ec003970 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:52.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb80a1dd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8040b0660 0x7fb8040ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:52.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb80a1dd640 10 monclient(hunting): picked mon.noname-b con 0x7fb8040b0660 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:52.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb80a1dd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb804138030 0x7fb8041384b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:52.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb80a1dd640 10 monclient(hunting): picked mon.noname-a con 0x7fb804138030 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:52.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb80a1dd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:52.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb80a1dd640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8040ad160 con 0x7fb804138030 2024-03-27T21:41:52.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb80a1dd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:52.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb80a1dd640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8040af130 con 0x7fb8040b0660 2024-03-27T21:41:52.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb80a1dd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:52.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb80a1dd640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8040aee70 con 0x7fb7ec003970 2024-03-27T21:41:52.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb80a1dd640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:52.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb80a1dd640 10 monclient(hunting): authenticate will time out at 953.563312s 2024-03-27T21:41:52.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb8037fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb7ec003970 0x7fb8040ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:52.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb803fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb804138030 0x7fb8041384b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:52.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb803fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb804138030 0x7fb8041384b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55930/0 (socket says 172.21.15.60:55930) 2024-03-27T21:41:52.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb8037fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb7ec003970 0x7fb8040ade50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:38882/0 (socket says 172.21.15.60:38882) 2024-03-27T21:41:52.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb803fff640 1 -- 172.21.15.60:0/1906468027 learned_addr learned my addr 172.21.15.60:0/1906468027 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:52.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb803fff640 10 monclient(hunting): get_auth_request con 0x7fb804138030 auth_method 0 2024-03-27T21:41:52.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb803fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:52.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb803fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:52.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb803fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:52.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb8037fe640 10 monclient(hunting): get_auth_request con 0x7fb7ec003970 auth_method 0 2024-03-27T21:41:52.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb8037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:52.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb8037fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:52.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb8037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:52.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb802ffd640 1 --2- 172.21.15.60:0/1906468027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8040b0660 0x7fb8040ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:52.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb803fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:52.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb803fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:52.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb803fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:52.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb802ffd640 10 monclient(hunting): get_auth_request con 0x7fb8040b0660 auth_method 0 2024-03-27T21:41:52.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb802ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:52.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb802ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:52.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb802ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:52.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb8037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:52.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb8037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:52.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb8037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:52.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb802ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:52.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb802ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:52.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb802ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:52.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb803fff640 10 monclient(hunting): handle_auth_done global_id 5202 payload 824 2024-03-27T21:41:52.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb803fff640 1 -- 172.21.15.60:0/1906468027 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb7ec003970 msgr2=0x7fb8040ade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:52.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb803fff640 1 --2- 172.21.15.60:0/1906468027 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb7ec003970 0x7fb8040ade50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:52.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb803fff640 1 -- 172.21.15.60:0/1906468027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8040b0660 msgr2=0x7fb8040ae930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:52.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb803fff640 1 --2- 172.21.15.60:0/1906468027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8040b0660 0x7fb8040ae930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:52.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb803fff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:52.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb803fff640 1 monclient: found mon.noname-a 2024-03-27T21:41:52.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb803fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:52.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb803fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:52.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb803fff640 1 -- 172.21.15.60:0/1906468027 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8041389f0 con 0x7fb804138030 2024-03-27T21:41:52.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.342+0000 7fb8037fe640 1 --2- 172.21.15.60:0/1906468027 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb7ec003970 0x7fb8040ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:52.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb803fff640 1 --2- 172.21.15.60:0/1906468027 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb804138030 0x7fb8041384b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb8040b11d0 tx=0x7fb7fc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:52.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb7ebfff640 10 client.?.objecter ms_handle_connect 0x7fb804138030 2024-03-27T21:41:52.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb7ebfff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:52.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb7ebfff640 1 -- 172.21.15.60:0/1906468027 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc00e040 con 0x7fb804138030 2024-03-27T21:41:52.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb7ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:52.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb7ebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:52.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb7ebfff640 10 monclient: dump: 2024-03-27T21:41:52.365 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:52.365 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:52.365 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:52.365 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:52.365 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:52.365 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:52.365 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:52.365 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:52.365 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:52.366 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:52.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb7ebfff640 10 monclient: _finish_auth 0 2024-03-27T21:41:52.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb7ebfff640 10 monclient: _check_auth_tickets 2024-03-27T21:41:52.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb7ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:52.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb7ebfff640 1 -- 172.21.15.60:0/1906468027 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7fc00f040 con 0x7fb804138030 2024-03-27T21:41:52.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb80a1dd640 5 monclient: authenticate success, global_id 5202 2024-03-27T21:41:52.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb80a1dd640 10 monclient: _renew_subs 2024-03-27T21:41:52.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb80a1dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:52.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb80a1dd640 1 -- 172.21.15.60:0/1906468027 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8041d65c0 con 0x7fb804138030 2024-03-27T21:41:52.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb7ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:52.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb7ebfff640 1 -- 172.21.15.60:0/1906468027 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc015430 con 0x7fb804138030 2024-03-27T21:41:52.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb7ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:52.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb7ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:52.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb7ebfff640 10 monclient: dump: 2024-03-27T21:41:52.367 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:52.367 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:52.367 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:52.367 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:52.368 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:52.368 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:52.368 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:52.368 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:52.368 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:52.368 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:52.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb80a1dd640 10 client.5202.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:52.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb80a1dd640 10 monclient: _renew_subs 2024-03-27T21:41:52.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb80a1dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:52.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb80a1dd640 1 -- 172.21.15.60:0/1906468027 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8041d6ad0 con 0x7fb804138030 2024-03-27T21:41:52.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb80a1dd640 1 librados: init done 2024-03-27T21:41:52.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.343+0000 7fb801ffb640 4 set_mon_vals no callback set 2024-03-27T21:41:52.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.344+0000 7fb80a1dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:52.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.344+0000 7fb80a1dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:52.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.344+0000 7fb80a1dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:52.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.344+0000 7fb80a1dd640 1 -- 172.21.15.60:0/1906468027 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb804069470 con 0x7fb804138030 2024-03-27T21:41:52.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.344+0000 7fb7ebfff640 1 -- 172.21.15.60:0/1906468027 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb7fc015590 con 0x7fb804138030 2024-03-27T21:41:52.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.345+0000 7fb7ebfff640 1 --2- 172.21.15.60:0/1906468027 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb7f006c5c0 0x7fb7f006ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:52.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.345+0000 7fb7ebfff640 1 -- 172.21.15.60:0/1906468027 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fb7fc089750 con 0x7fb804138030 2024-03-27T21:41:52.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.345+0000 7fb7ebfff640 10 client.5202.objecter ms_dispatch 0x7fb804000b90 osd_map(80..80 src has 1..80) v4 2024-03-27T21:41:52.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.345+0000 7fb7ebfff640 3 client.5202.objecter handle_osd_map got epochs [80,80] > 0 2024-03-27T21:41:52.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.345+0000 7fb7ebfff640 3 client.5202.objecter handle_osd_map decoding full epoch 80 2024-03-27T21:41:52.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.345+0000 7fb8037fe640 1 --2- 172.21.15.60:0/1906468027 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb7f006c5c0 0x7fb7f006ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:52.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.345+0000 7fb8037fe640 10 monclient: get_auth_request con 0x7fb7f006c5c0 auth_method 0 2024-03-27T21:41:52.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.345+0000 7fb7ebfff640 20 client.5202.objecter dump_active .. 0 homeless 2024-03-27T21:41:52.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.345+0000 7fb7ebfff640 10 client.5202.objecter ms_handle_connect 0x7fb7f006c5c0 2024-03-27T21:41:52.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.345+0000 7fb8037fe640 1 --2- 172.21.15.60:0/1906468027 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb7f006c5c0 0x7fb7f006ea80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8034000 tx=0x7fb7f8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:52.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.349+0000 7fb7ebfff640 1 -- 172.21.15.60:0/1906468027 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb7fc054bb0 con 0x7fb804138030 2024-03-27T21:41:52.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.349+0000 7fb7ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:52.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.349+0000 7fb7ebfff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:52.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.539+0000 7fb80a1dd640 10 start_mon_command cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}] 2024-03-27T21:41:52.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.539+0000 7fb80a1dd640 10 monclient: _send_command 2 [{"prefix": "osd reweight", "id": 3, "weight": 1.0}] 2024-03-27T21:41:52.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.539+0000 7fb80a1dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:52.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:52.539+0000 7fb80a1dd640 1 -- 172.21.15.60:0/1906468027 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) v1 -- 0x7fb8040a9240 con 0x7fb804138030 2024-03-27T21:41:53.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.212+0000 7fb7ebfff640 1 -- 172.21.15.60:0/1906468027 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v81) v1 ==== 113+0+0 (secure 0 0 0) 0x7fb7fc0581d0 con 0x7fb804138030 2024-03-27T21:41:53.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.213+0000 7fb7ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd reweight", "id": 3, "weight": 1.0}] 2024-03-27T21:41:53.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.213+0000 7fb7ebfff640 10 monclient: _finish_command 2 = system:0 reweighted osd.3 to 1 (10000) 2024-03-27T21:41:53.200 INFO:teuthology.orchestra.run.smithi060.stderr:reweighted osd.3 to 1 (10000) 2024-03-27T21:41:53.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.215+0000 7fb80a1dd640 10 librados: watch_flush enter 2024-03-27T21:41:53.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.215+0000 7fb80a1dd640 10 librados: watch_flush exit 2024-03-27T21:41:53.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.215+0000 7fb80a1dd640 20 client.5202.objecter shutdown clearing up homeless session... 2024-03-27T21:41:53.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.215+0000 7fb80a1dd640 10 client.5202.objecter successfully canceled tick 2024-03-27T21:41:53.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.215+0000 7fb80a1dd640 1 -- 172.21.15.60:0/1906468027 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb7f006c5c0 msgr2=0x7fb7f006ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:53.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.215+0000 7fb80a1dd640 1 --2- 172.21.15.60:0/1906468027 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb7f006c5c0 0x7fb7f006ea80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8034000 tx=0x7fb7f8009030 comp rx=0 tx=0).stop 2024-03-27T21:41:53.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.215+0000 7fb80a1dd640 10 monclient: shutdown 2024-03-27T21:41:53.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.215+0000 7fb80a1dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:53.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.215+0000 7fb80a1dd640 1 -- 172.21.15.60:0/1906468027 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb804138030 msgr2=0x7fb8041384b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:53.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.215+0000 7fb80a1dd640 1 --2- 172.21.15.60:0/1906468027 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb804138030 0x7fb8041384b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb8040b11d0 tx=0x7fb7fc00f580 comp rx=0 tx=0).stop 2024-03-27T21:41:53.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.216+0000 7fb80a1dd640 1 -- 172.21.15.60:0/1906468027 shutdown_connections 2024-03-27T21:41:53.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.216+0000 7fb80a1dd640 1 --2- 172.21.15.60:0/1906468027 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb804138030 0x7fb8041384b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:53.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.216+0000 7fb80a1dd640 1 --2- 172.21.15.60:0/1906468027 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fb7f006c5c0 0x7fb7f006ea80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:53.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.216+0000 7fb80a1dd640 1 --2- 172.21.15.60:0/1906468027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8040b0660 0x7fb8040ae930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:53.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.216+0000 7fb80a1dd640 1 --2- 172.21.15.60:0/1906468027 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fb7ec003970 0x7fb8040ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:53.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.216+0000 7fb80a1dd640 1 -- 172.21.15.60:0/1906468027 >> 172.21.15.60:0/1906468027 conn(0x7fb8040b7300 msgr2=0x7fb80413fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:53.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.216+0000 7fb80a1dd640 1 -- 172.21.15.60:0/1906468027 shutdown_connections 2024-03-27T21:41:53.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.216+0000 7fb80a1dd640 1 -- 172.21.15.60:0/1906468027 wait complete. 2024-03-27T21:41:53.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.216+0000 7fb80a1dd640 1 librados: shutdown 2024-03-27T21:41:53.216 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-27T21:41:53.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.337+0000 7f93d2b1b640 10 monclient: get_monmap_and_config 2024-03-27T21:41:53.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.337+0000 7f93d2b1b640 10 monclient: build_initial_monmap 2024-03-27T21:41:53.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.337+0000 7f93d2b1b640 10 monclient: monmap: 2024-03-27T21:41:53.324 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:53.324 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:53.324 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:53.338279+0000 2024-03-27T21:41:53.324 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:53.338279+0000 2024-03-27T21:41:53.324 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:53.324 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:53.324 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:53.324 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:53.325 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:53.325 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:53.325 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:53.325 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93d2b1b640 1 Processor -- start 2024-03-27T21:41:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93d2b1b640 1 -- start start 2024-03-27T21:41:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93d2b1b640 10 monclient: init 2024-03-27T21:41:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93d2b1b640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93d2b1b640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-03-27T21:41:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93d2b1b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f93cc0717b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93d2b1b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f93cc071b90 0x7f93cc0a8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93d2b1b640 10 monclient(hunting): picked mon.noname-b con 0x7f93cc071b90 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93d2b1b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f93cc0b0fa0 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93d2b1b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93cc0b0880 con 0x7f93cc0717b0 2024-03-27T21:41:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93d2b1b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93cc072300 con 0x7f93cc0b0fa0 2024-03-27T21:41:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93d2b1b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93d2b1b640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc0a8980 con 0x7f93cc071b90 2024-03-27T21:41:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93d2b1b640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93d2b1b640 10 monclient(hunting): authenticate will time out at 954.559632s 2024-03-27T21:41:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93cbfff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f93cc071b90 0x7f93cc0a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93d08b7640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f93cc0717b0 0x7f93cc1414c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47254/0 (socket says 172.21.15.60:47254) 2024-03-27T21:41:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93d08b7640 1 -- 172.21.15.60:0/345032299 learned_addr learned my addr 172.21.15.60:0/345032299 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93cbfff640 10 monclient(hunting): get_auth_request con 0x7f93cc071b90 auth_method 0 2024-03-27T21:41:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.338+0000 7f93cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.339+0000 7f93cbfff640 10 monclient(hunting): handle_auth_done global_id 4937 payload 274 2024-03-27T21:41:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.339+0000 7f93cbfff640 1 -- 172.21.15.60:0/345032299 >> v1:172.21.15.191:6789/0 conn(0x7f93cc0b0fa0 legacy=0x7f93cc0b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.339+0000 7f93cbfff640 1 -- 172.21.15.60:0/345032299 >> v1:172.21.15.60:6789/0 conn(0x7f93cc0717b0 legacy=0x7f93cc1414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.339+0000 7f93cbfff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.339+0000 7f93cbfff640 1 monclient: found mon.noname-b 2024-03-27T21:41:53.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.339+0000 7f93cb7fe640 1 -- 172.21.15.60:0/345032299 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543624559 0 0) 0x7f93cc0b0880 con 0x7f93cc0717b0 2024-03-27T21:41:53.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.339+0000 7f93cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:53.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.339+0000 7f93cbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:53.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.339+0000 7f93cbfff640 1 -- 172.21.15.60:0/345032299 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93cc0b1410 con 0x7f93cc071b90 2024-03-27T21:41:53.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.339+0000 7f93cb7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:41:53.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.339+0000 7f93cbfff640 1 --2- 172.21.15.60:0/345032299 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f93cc071b90 0x7f93cc0a8440 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f93bc0097d0 tx=0x7f93bc02b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=110484c7bb40cb46 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.339+0000 7f93cb7fe640 1 -- 172.21.15.60:0/345032299 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93bc031770 con 0x7f93cc071b90 2024-03-27T21:41:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.339+0000 7f93cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.339+0000 7f93cb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.339+0000 7f93cb7fe640 10 monclient: dump: 2024-03-27T21:41:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:53.342 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93cb7fe640 10 monclient: _finish_auth 0 2024-03-27T21:41:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93cb7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:41:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93d2b1b640 5 monclient: authenticate success, global_id 4937 2024-03-27T21:41:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93cb7fe640 1 -- 172.21.15.60:0/345032299 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93bc0318d0 con 0x7f93cc071b90 2024-03-27T21:41:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93d2b1b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93d2b1b640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93d2b1b640 4 set_mon_vals no callback set 2024-03-27T21:41:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93cb7fe640 1 -- 172.21.15.60:0/345032299 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93bc031c60 con 0x7f93cc071b90 2024-03-27T21:41:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93cb7fe640 10 monclient: dump: 2024-03-27T21:41:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:53.345 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93d2b1b640 10 monclient: shutdown 2024-03-27T21:41:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93d2b1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93d2b1b640 1 -- 172.21.15.60:0/345032299 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f93cc071b90 msgr2=0x7f93cc0a8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93d2b1b640 1 --2- 172.21.15.60:0/345032299 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f93cc071b90 0x7f93cc0a8440 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f93bc0097d0 tx=0x7f93bc02b810 comp rx=0 tx=0).stop 2024-03-27T21:41:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93d2b1b640 1 -- 172.21.15.60:0/345032299 shutdown_connections 2024-03-27T21:41:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93d2b1b640 1 --2- 172.21.15.60:0/345032299 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f93cc071b90 0x7f93cc0a8440 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93d2b1b640 1 -- 172.21.15.60:0/345032299 >> 172.21.15.60:0/345032299 conn(0x7f93cc0b7290 msgr2=0x7f93cc0b76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93d2b1b640 1 -- 172.21.15.60:0/345032299 shutdown_connections 2024-03-27T21:41:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.340+0000 7f93d2b1b640 1 -- 172.21.15.60:0/345032299 wait complete. 2024-03-27T21:41:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.341+0000 7f93d2b1b640 10 monclient: build_initial_monmap 2024-03-27T21:41:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.341+0000 7f93d2b1b640 10 monclient: monmap: 2024-03-27T21:41:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:53.348 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.341+0000 7f93d2b1b640 1 librados: starting msgr at 2024-03-27T21:41:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.341+0000 7f93d2b1b640 1 librados: starting objecter 2024-03-27T21:41:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.341+0000 7f93d2b1b640 1 Processor -- start 2024-03-27T21:41:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.341+0000 7f93d2b1b640 1 -- start start 2024-03-27T21:41:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.341+0000 7f93d2b1b640 1 librados: setting wanted keys 2024-03-27T21:41:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.341+0000 7f93d2b1b640 1 librados: calling monclient init 2024-03-27T21:41:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.341+0000 7f93d2b1b640 10 monclient: init 2024-03-27T21:41:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d2b1b640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d2b1b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:41:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d2b1b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93cc0717b0 0x7f93cc13aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d2b1b640 10 monclient(hunting): picked mon.noname-a con 0x7f93cc0717b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d2b1b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f93cc071b90 0x7f93cc137850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d2b1b640 10 monclient(hunting): picked mon.noname-c con 0x7f93cc071b90 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d2b1b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93cc0b0fa0 0x7f93cc137d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d2b1b640 10 monclient(hunting): picked mon.noname-b con 0x7f93cc0b0fa0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d2b1b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d2b1b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc13b710 con 0x7f93cc0717b0 2024-03-27T21:41:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d2b1b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d2b1b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc13b410 con 0x7f93cc0b0fa0 2024-03-27T21:41:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d2b1b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d2b1b640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc13ca50 con 0x7f93cc071b90 2024-03-27T21:41:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d2b1b640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d2b1b640 10 monclient(hunting): authenticate will time out at 954.563185s 2024-03-27T21:41:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93cbfff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f93cc071b90 0x7f93cc137850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d08b7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93cc0717b0 0x7f93cc13aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93cbfff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f93cc071b90 0x7f93cc137850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:38914/0 (socket says 172.21.15.60:38914) 2024-03-27T21:41:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d08b7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93cc0717b0 0x7f93cc13aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55938/0 (socket says 172.21.15.60:55938) 2024-03-27T21:41:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93cbfff640 1 -- 172.21.15.60:0/1402607774 learned_addr learned my addr 172.21.15.60:0/1402607774 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93cbfff640 10 monclient(hunting): get_auth_request con 0x7f93cc071b90 auth_method 0 2024-03-27T21:41:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d08b7640 10 monclient(hunting): get_auth_request con 0x7f93cc0717b0 auth_method 0 2024-03-27T21:41:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d08b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d08b7640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d08b7640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d08b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d08b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d08b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d10b8640 1 --2- 172.21.15.60:0/1402607774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93cc0b0fa0 0x7f93cc137d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d10b8640 10 monclient(hunting): get_auth_request con 0x7f93cc0b0fa0 auth_method 0 2024-03-27T21:41:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d10b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d10b8640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d10b8640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93cbfff640 10 monclient(hunting): handle_auth_done global_id 4940 payload 824 2024-03-27T21:41:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93cbfff640 1 -- 172.21.15.60:0/1402607774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93cc0b0fa0 msgr2=0x7f93cc137d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93cbfff640 1 --2- 172.21.15.60:0/1402607774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93cc0b0fa0 0x7f93cc137d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93cbfff640 1 -- 172.21.15.60:0/1402607774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93cc0717b0 msgr2=0x7f93cc13aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93cbfff640 1 --2- 172.21.15.60:0/1402607774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93cc0717b0 0x7f93cc13aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93cbfff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93cbfff640 1 monclient: found mon.noname-c 2024-03-27T21:41:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93cbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93cbfff640 1 -- 172.21.15.60:0/1402607774 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93bc009400 con 0x7f93cc071b90 2024-03-27T21:41:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93d08b7640 1 --2- 172.21.15.60:0/1402607774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93cc0717b0 0x7f93cc13aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93cbfff640 1 --2- 172.21.15.60:0/1402607774 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f93cc071b90 0x7f93cc137850 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009720 tx=0x7f93bc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93c97fa640 10 client.?.objecter ms_handle_connect 0x7f93cc071b90 2024-03-27T21:41:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.342+0000 7f93c97fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93c97fa640 1 -- 172.21.15.60:0/1402607774 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93bc0328d0 con 0x7f93cc071b90 2024-03-27T21:41:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93c97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93c97fa640 10 monclient: dump: 2024-03-27T21:41:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:53.358 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93c97fa640 10 monclient: _finish_auth 0 2024-03-27T21:41:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93c97fa640 10 monclient: _check_auth_tickets 2024-03-27T21:41:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93c97fa640 1 -- 172.21.15.60:0/1402607774 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93bc032a30 con 0x7f93cc071b90 2024-03-27T21:41:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93d2b1b640 5 monclient: authenticate success, global_id 4940 2024-03-27T21:41:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93d2b1b640 10 monclient: _renew_subs 2024-03-27T21:41:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93d2b1b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93d2b1b640 1 -- 172.21.15.60:0/1402607774 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93cc1395f0 con 0x7f93cc071b90 2024-03-27T21:41:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93c97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93c97fa640 1 -- 172.21.15.60:0/1402607774 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93bc032de0 con 0x7f93cc071b90 2024-03-27T21:41:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93c97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93c97fa640 10 monclient: dump: 2024-03-27T21:41:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:53.361 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93caffd640 4 set_mon_vals no callback set 2024-03-27T21:41:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93d2b1b640 10 client.4940.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93d2b1b640 10 monclient: _renew_subs 2024-03-27T21:41:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93d2b1b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93d2b1b640 1 -- 172.21.15.60:0/1402607774 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93cc1382d0 con 0x7f93cc071b90 2024-03-27T21:41:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.343+0000 7f93d2b1b640 1 librados: init done 2024-03-27T21:41:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.344+0000 7f93d2b1b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.344+0000 7f93d2b1b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.344+0000 7f93d2b1b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.344+0000 7f93d2b1b640 1 -- 172.21.15.60:0/1402607774 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93cc069470 con 0x7f93cc071b90 2024-03-27T21:41:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.344+0000 7f93c97fa640 1 -- 172.21.15.60:0/1402607774 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f93bc04d3f0 con 0x7f93cc071b90 2024-03-27T21:41:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.344+0000 7f93c97fa640 1 --2- 172.21.15.60:0/1402607774 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f93ac06c590 0x7f93ac06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.344+0000 7f93c97fa640 1 -- 172.21.15.60:0/1402607774 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f93bc0aaca0 con 0x7f93cc071b90 2024-03-27T21:41:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.344+0000 7f93c97fa640 10 client.4940.objecter ms_dispatch 0x7f93cc000b90 osd_map(81..81 src has 1..81) v4 2024-03-27T21:41:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.344+0000 7f93c97fa640 3 client.4940.objecter handle_osd_map got epochs [81,81] > 0 2024-03-27T21:41:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.344+0000 7f93c97fa640 3 client.4940.objecter handle_osd_map decoding full epoch 81 2024-03-27T21:41:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.344+0000 7f93d08b7640 1 --2- 172.21.15.60:0/1402607774 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f93ac06c590 0x7f93ac06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.344+0000 7f93d08b7640 10 monclient: get_auth_request con 0x7f93ac06c590 auth_method 0 2024-03-27T21:41:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.344+0000 7f93c97fa640 20 client.4940.objecter dump_active .. 0 homeless 2024-03-27T21:41:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.345+0000 7f93d08b7640 1 --2- 172.21.15.60:0/1402607774 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f93ac06c590 0x7f93ac06ea50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f93c40012e0 tx=0x7f93c4001700 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.345+0000 7f93c97fa640 10 client.4940.objecter ms_handle_connect 0x7f93ac06c590 2024-03-27T21:41:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.349+0000 7f93c97fa640 1 -- 172.21.15.60:0/1402607774 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f93bc0760e0 con 0x7f93cc071b90 2024-03-27T21:41:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.349+0000 7f93c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.349+0000 7f93c97fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:53.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.539+0000 7f93d2b1b640 10 start_mon_command cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}] 2024-03-27T21:41:53.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.539+0000 7f93d2b1b640 10 monclient: _send_command 2 [{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}] 2024-03-27T21:41:53.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.539+0000 7f93d2b1b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:53.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:53.539+0000 7f93d2b1b640 1 -- 172.21.15.60:0/1402607774 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} v 0) v1 -- 0x7f93cc067700 con 0x7f93cc071b90 2024-03-27T21:41:54.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.217+0000 7f93c97fa640 1 -- 172.21.15.60:0/1402607774 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]=0 set osd.3 primary-affinity to 1 (8655362) v82) v1 ==== 133+0+0 (secure 0 0 0) 0x7f93bc079700 con 0x7f93cc071b90 2024-03-27T21:41:54.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.218+0000 7f93c97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}] 2024-03-27T21:41:54.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.218+0000 7f93c97fa640 10 monclient: _finish_command 2 = system:0 set osd.3 primary-affinity to 1 (8655362) 2024-03-27T21:41:54.205 INFO:teuthology.orchestra.run.smithi060.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-27T21:41:54.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.220+0000 7f93d2b1b640 10 librados: watch_flush enter 2024-03-27T21:41:54.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.220+0000 7f93d2b1b640 10 librados: watch_flush exit 2024-03-27T21:41:54.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.220+0000 7f93d2b1b640 20 client.4940.objecter shutdown clearing up homeless session... 2024-03-27T21:41:54.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.220+0000 7f93d2b1b640 10 client.4940.objecter successfully canceled tick 2024-03-27T21:41:54.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.220+0000 7f93d2b1b640 1 -- 172.21.15.60:0/1402607774 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f93ac06c590 msgr2=0x7f93ac06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:54.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.220+0000 7f93d2b1b640 1 --2- 172.21.15.60:0/1402607774 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f93ac06c590 0x7f93ac06ea50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f93c40012e0 tx=0x7f93c4001700 comp rx=0 tx=0).stop 2024-03-27T21:41:54.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.220+0000 7f93d2b1b640 10 monclient: shutdown 2024-03-27T21:41:54.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.220+0000 7f93d2b1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:54.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.220+0000 7f93d2b1b640 1 -- 172.21.15.60:0/1402607774 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f93cc071b90 msgr2=0x7f93cc137850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:54.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.220+0000 7f93d2b1b640 1 --2- 172.21.15.60:0/1402607774 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f93cc071b90 0x7f93cc137850 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009720 tx=0x7f93bc0046c0 comp rx=0 tx=0).stop 2024-03-27T21:41:54.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.220+0000 7f93d2b1b640 1 -- 172.21.15.60:0/1402607774 shutdown_connections 2024-03-27T21:41:54.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.220+0000 7f93d2b1b640 1 --2- 172.21.15.60:0/1402607774 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f93ac06c590 0x7f93ac06ea50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:54.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.220+0000 7f93d2b1b640 1 --2- 172.21.15.60:0/1402607774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93cc0b0fa0 0x7f93cc137d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:54.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.220+0000 7f93d2b1b640 1 --2- 172.21.15.60:0/1402607774 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f93cc071b90 0x7f93cc137850 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:54.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.220+0000 7f93d2b1b640 1 --2- 172.21.15.60:0/1402607774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93cc0717b0 0x7f93cc13aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:54.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.220+0000 7f93d2b1b640 1 -- 172.21.15.60:0/1402607774 >> 172.21.15.60:0/1402607774 conn(0x7f93cc0b7290 msgr2=0x7f93cc0b4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:54.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.221+0000 7f93d2b1b640 1 -- 172.21.15.60:0/1402607774 shutdown_connections 2024-03-27T21:41:54.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.221+0000 7f93d2b1b640 1 -- 172.21.15.60:0/1402607774 wait complete. 2024-03-27T21:41:54.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.221+0000 7f93d2b1b640 1 librados: shutdown 2024-03-27T21:41:54.220 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-27T21:41:54.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.341+0000 7f22edb5c640 10 monclient: get_monmap_and_config 2024-03-27T21:41:54.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.341+0000 7f22edb5c640 10 monclient: build_initial_monmap 2024-03-27T21:41:54.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.341+0000 7f22edb5c640 10 monclient: monmap: 2024-03-27T21:41:54.328 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:54.328 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:54.329 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:54.342969+0000 2024-03-27T21:41:54.329 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:54.342969+0000 2024-03-27T21:41:54.329 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:54.329 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:54.329 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:54.329 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:54.329 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:54.329 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:54.329 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:54.330 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:54.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.342+0000 7f22edb5c640 1 Processor -- start 2024-03-27T21:41:54.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.342+0000 7f22edb5c640 1 -- start start 2024-03-27T21:41:54.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.342+0000 7f22edb5c640 10 monclient: init 2024-03-27T21:41:54.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22edb5c640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:54.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22edb5c640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-03-27T21:41:54.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22edb5c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f22e8071a50 0x7f22e80a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:54.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22edb5c640 10 monclient(hunting): picked mon.noname-c con 0x7f22e8071a50 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:54.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22edb5c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f22e813e810 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:54.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22edb5c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f22e813ec50 0x7f22e8141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:54.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22edb5c640 10 monclient(hunting): picked mon.noname-a con 0x7f22e813ec50 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:54.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22edb5c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22e80a8e90 con 0x7f22e813e810 2024-03-27T21:41:54.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22edb5c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:54.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22edb5c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22e8071e30 con 0x7f22e813ec50 2024-03-27T21:41:54.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22edb5c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:54.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22edb5c640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22e80aa040 con 0x7f22e8071a50 2024-03-27T21:41:54.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22edb5c640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:54.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22edb5c640 10 monclient(hunting): authenticate will time out at 955.564253s 2024-03-27T21:41:54.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f22e813ec50 0x7f22e8141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:54.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f22e813ec50 0x7f22e8141d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55944/0 (socket says 172.21.15.60:55944) 2024-03-27T21:41:54.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e7fff640 1 -- 172.21.15.60:0/4120792326 learned_addr learned my addr 172.21.15.60:0/4120792326 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:54.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e7fff640 10 monclient(hunting): get_auth_request con 0x7f22e813ec50 auth_method 0 2024-03-27T21:41:54.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:54.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:54.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:54.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:54.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:54.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:54.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e7fff640 10 monclient(hunting): handle_auth_done global_id 5211 payload 274 2024-03-27T21:41:54.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e77fe640 1 --2- 172.21.15.60:0/4120792326 >> v2:172.21.15.191:3300/0 conn(0x7f22e8071a50 0x7f22e80a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:54.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e7fff640 1 -- 172.21.15.60:0/4120792326 >> v2:172.21.15.191:3300/0 conn(0x7f22e8071a50 msgr2=0x7f22e80a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:54.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e7fff640 1 --2- 172.21.15.60:0/4120792326 >> v2:172.21.15.191:3300/0 conn(0x7f22e8071a50 0x7f22e80a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:54.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e7fff640 1 -- 172.21.15.60:0/4120792326 >> v1:172.21.15.191:6789/0 conn(0x7f22e813e810 legacy=0x7f22e8141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:54.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e7fff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:54.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e7fff640 1 monclient: found mon.noname-a 2024-03-27T21:41:54.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:54.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:54.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e7fff640 1 -- 172.21.15.60:0/4120792326 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e8142280 con 0x7f22e813ec50 2024-03-27T21:41:54.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e77fe640 10 monclient: get_auth_request con 0x7f22e8071a50 auth_method 0 2024-03-27T21:41:54.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.343+0000 7f22e77fe640 1 --2- 172.21.15.60:0/4120792326 >> v2:172.21.15.191:3300/0 conn(0x7f22e8071a50 0x7f22e80a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:41:54.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.344+0000 7f22e7fff640 1 --2- 172.21.15.60:0/4120792326 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22e813ec50 0x7f22e8141d40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f22e80b4a80 tx=0x7f22e002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe8c83db7c4c007 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:54.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.344+0000 7f22e67fc640 1 -- 172.21.15.60:0/4120792326 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e002ca80 con 0x7f22e813ec50 2024-03-27T21:41:54.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.344+0000 7f22e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:54.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.344+0000 7f22e67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:54.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.344+0000 7f22e67fc640 10 monclient: dump: 2024-03-27T21:41:54.344 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:54.344 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:54.344 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:54.344 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:54.344 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:54.345 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:54.345 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:54.345 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:54.345 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:54.345 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:54.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.344+0000 7f22e67fc640 10 monclient: _finish_auth 0 2024-03-27T21:41:54.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.344+0000 7f22e67fc640 10 monclient: _check_auth_tickets 2024-03-27T21:41:54.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.344+0000 7f22e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:54.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.344+0000 7f22edb5c640 5 monclient: authenticate success, global_id 5211 2024-03-27T21:41:54.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.344+0000 7f22edb5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:54.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.344+0000 7f22e67fc640 1 -- 172.21.15.60:0/4120792326 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e002cbe0 con 0x7f22e813ec50 2024-03-27T21:41:54.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.344+0000 7f22e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:54.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.344+0000 7f22edb5c640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:54.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.344+0000 7f22edb5c640 4 set_mon_vals no callback set 2024-03-27T21:41:54.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.344+0000 7f22e67fc640 1 -- 172.21.15.60:0/4120792326 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e00343f0 con 0x7f22e813ec50 2024-03-27T21:41:54.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.344+0000 7f22e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:54.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.344+0000 7f22e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:54.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.344+0000 7f22e67fc640 10 monclient: dump: 2024-03-27T21:41:54.347 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:54.347 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:54.347 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:54.347 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:54.347 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:54.347 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:54.347 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:54.348 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:54.348 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:54.348 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:54.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.345+0000 7f22edb5c640 10 monclient: shutdown 2024-03-27T21:41:54.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.345+0000 7f22edb5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:54.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.345+0000 7f22edb5c640 1 -- 172.21.15.60:0/4120792326 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22e813ec50 msgr2=0x7f22e8141d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:54.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.345+0000 7f22edb5c640 1 --2- 172.21.15.60:0/4120792326 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22e813ec50 0x7f22e8141d40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f22e80b4a80 tx=0x7f22e002bee0 comp rx=0 tx=0).stop 2024-03-27T21:41:54.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.345+0000 7f22edb5c640 1 -- 172.21.15.60:0/4120792326 shutdown_connections 2024-03-27T21:41:54.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.345+0000 7f22edb5c640 1 --2- 172.21.15.60:0/4120792326 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22e813ec50 0x7f22e8141d40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:54.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.345+0000 7f22edb5c640 1 --2- 172.21.15.60:0/4120792326 >> v2:172.21.15.191:3300/0 conn(0x7f22e8071a50 0x7f22e80a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:54.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.345+0000 7f22edb5c640 1 -- 172.21.15.60:0/4120792326 >> 172.21.15.60:0/4120792326 conn(0x7f22e80b7300 msgr2=0x7f22e80b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:54.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.345+0000 7f22edb5c640 1 -- 172.21.15.60:0/4120792326 shutdown_connections 2024-03-27T21:41:54.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.345+0000 7f22edb5c640 1 -- 172.21.15.60:0/4120792326 wait complete. 2024-03-27T21:41:54.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.345+0000 7f22edb5c640 10 monclient: build_initial_monmap 2024-03-27T21:41:54.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.345+0000 7f22edb5c640 10 monclient: monmap: 2024-03-27T21:41:54.349 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:54.349 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:54.350 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:54.350 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:54.350 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:54.350 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:54.350 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:54.350 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:54.351 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:54.351 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:54.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 1 librados: starting msgr at 2024-03-27T21:41:54.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 1 librados: starting objecter 2024-03-27T21:41:54.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 1 Processor -- start 2024-03-27T21:41:54.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 1 -- start start 2024-03-27T21:41:54.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 1 librados: setting wanted keys 2024-03-27T21:41:54.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 1 librados: calling monclient init 2024-03-27T21:41:54.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 10 monclient: init 2024-03-27T21:41:54.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:54.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:41:54.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22e8071a50 0x7f22e8139ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:54.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 10 monclient(hunting): picked mon.noname-a con 0x7f22e8071a50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:54.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22e813e810 0x7f22e81376d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:54.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 10 monclient(hunting): picked mon.noname-b con 0x7f22e813e810 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f22e813ec50 0x7f22e8137c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 10 monclient(hunting): picked mon.noname-c con 0x7f22e813ec50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e8138150 con 0x7f22e8071a50 2024-03-27T21:41:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e813ace0 con 0x7f22e813e810 2024-03-27T21:41:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e80ab880 con 0x7f22e813ec50 2024-03-27T21:41:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.346+0000 7f22edb5c640 10 monclient(hunting): authenticate will time out at 955.567913s 2024-03-27T21:41:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22e8071a50 0x7f22e8139ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22e8071a50 0x7f22e8139ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55952/0 (socket says 172.21.15.60:55952) 2024-03-27T21:41:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e77fe640 1 -- 172.21.15.60:0/3179947539 learned_addr learned my addr 172.21.15.60:0/3179947539 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e77fe640 10 monclient(hunting): get_auth_request con 0x7f22e8071a50 auth_method 0 2024-03-27T21:41:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e7fff640 1 --2- 172.21.15.60:0/3179947539 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f22e813ec50 0x7f22e8137c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:54.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e7fff640 10 monclient(hunting): get_auth_request con 0x7f22e813ec50 auth_method 0 2024-03-27T21:41:54.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:54.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:54.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:54.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:54.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:54.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:54.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:54.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:54.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:54.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e6ffd640 1 --2- 172.21.15.60:0/3179947539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22e813e810 0x7f22e81376d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:54.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e6ffd640 10 monclient(hunting): get_auth_request con 0x7f22e813e810 auth_method 0 2024-03-27T21:41:54.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:54.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:54.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:54.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e77fe640 10 monclient(hunting): handle_auth_done global_id 5214 payload 824 2024-03-27T21:41:54.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e77fe640 1 -- 172.21.15.60:0/3179947539 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f22e813ec50 msgr2=0x7f22e8137c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:54.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e77fe640 1 --2- 172.21.15.60:0/3179947539 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f22e813ec50 0x7f22e8137c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:54.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e77fe640 1 -- 172.21.15.60:0/3179947539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22e813e810 msgr2=0x7f22e81376d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:54.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e77fe640 1 --2- 172.21.15.60:0/3179947539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22e813e810 0x7f22e81376d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:54.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e77fe640 10 monclient: _finish_hunting 0 2024-03-27T21:41:54.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e77fe640 1 monclient: found mon.noname-a 2024-03-27T21:41:54.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:54.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:54.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e77fe640 1 -- 172.21.15.60:0/3179947539 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e0009b30 con 0x7f22e8071a50 2024-03-27T21:41:54.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e7fff640 1 --2- 172.21.15.60:0/3179947539 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f22e813ec50 0x7f22e8137c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:54.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e6ffd640 1 --2- 172.21.15.60:0/3179947539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22e813e810 0x7f22e81376d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:41:54.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e77fe640 1 --2- 172.21.15.60:0/3179947539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22e8071a50 0x7f22e8139ec0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f22d800c7f0 tx=0x7f22d800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:54.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22cffff640 10 client.?.objecter ms_handle_connect 0x7f22e8071a50 2024-03-27T21:41:54.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22cffff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:54.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22cffff640 1 -- 172.21.15.60:0/3179947539 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d8007a30 con 0x7f22e8071a50 2024-03-27T21:41:54.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:54.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:54.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22cffff640 10 monclient: dump: 2024-03-27T21:41:54.358 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:54.359 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:54.359 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:54.359 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:54.359 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:54.359 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:54.359 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:54.359 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:54.359 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:54.359 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:54.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22cffff640 10 monclient: _finish_auth 0 2024-03-27T21:41:54.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22cffff640 10 monclient: _check_auth_tickets 2024-03-27T21:41:54.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:54.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22cffff640 1 -- 172.21.15.60:0/3179947539 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d8007b90 con 0x7f22e8071a50 2024-03-27T21:41:54.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22edb5c640 5 monclient: authenticate success, global_id 5214 2024-03-27T21:41:54.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:54.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22edb5c640 10 monclient: _renew_subs 2024-03-27T21:41:54.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22cffff640 1 -- 172.21.15.60:0/3179947539 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d800ce40 con 0x7f22e8071a50 2024-03-27T21:41:54.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22edb5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22edb5c640 1 -- 172.21.15.60:0/3179947539 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e80abbc0 con 0x7f22e8071a50 2024-03-27T21:41:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22cffff640 10 monclient: dump: 2024-03-27T21:41:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:54.362 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:54.362 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:54.362 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:54.362 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:54.362 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:54.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22e5ffb640 4 set_mon_vals no callback set 2024-03-27T21:41:54.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22edb5c640 10 client.5214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:54.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22edb5c640 10 monclient: _renew_subs 2024-03-27T21:41:54.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22edb5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22edb5c640 1 -- 172.21.15.60:0/3179947539 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e813b170 con 0x7f22e8071a50 2024-03-27T21:41:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.347+0000 7f22edb5c640 1 librados: init done 2024-03-27T21:41:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.348+0000 7f22edb5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.348+0000 7f22edb5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.348+0000 7f22edb5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.348+0000 7f22edb5c640 1 -- 172.21.15.60:0/3179947539 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e8139380 con 0x7f22e8071a50 2024-03-27T21:41:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.349+0000 7f22cffff640 1 -- 172.21.15.60:0/3179947539 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f22d80113f0 con 0x7f22e8071a50 2024-03-27T21:41:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.349+0000 7f22cffff640 1 --2- 172.21.15.60:0/3179947539 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f22d406c590 0x7f22d406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.349+0000 7f22cffff640 1 -- 172.21.15.60:0/3179947539 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f22d8089200 con 0x7f22e8071a50 2024-03-27T21:41:54.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.349+0000 7f22e6ffd640 1 --2- 172.21.15.60:0/3179947539 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f22d406c590 0x7f22d406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:54.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.349+0000 7f22cffff640 10 client.5214.objecter ms_dispatch 0x7f22e8000b90 osd_map(82..82 src has 1..82) v4 2024-03-27T21:41:54.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.349+0000 7f22cffff640 3 client.5214.objecter handle_osd_map got epochs [82,82] > 0 2024-03-27T21:41:54.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.349+0000 7f22cffff640 3 client.5214.objecter handle_osd_map decoding full epoch 82 2024-03-27T21:41:54.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.349+0000 7f22e6ffd640 10 monclient: get_auth_request con 0x7f22d406c590 auth_method 0 2024-03-27T21:41:54.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.349+0000 7f22cffff640 20 client.5214.objecter dump_active .. 0 homeless 2024-03-27T21:41:54.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.349+0000 7f22cffff640 10 client.5214.objecter ms_handle_connect 0x7f22d406c590 2024-03-27T21:41:54.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.349+0000 7f22e6ffd640 1 --2- 172.21.15.60:0/3179947539 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f22d406c590 0x7f22d406ea50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f22e80aaf10 tx=0x7f22dc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:54.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.353+0000 7f22cffff640 1 -- 172.21.15.60:0/3179947539 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f22d80546f0 con 0x7f22e8071a50 2024-03-27T21:41:54.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.353+0000 7f22cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:54.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.353+0000 7f22cffff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:54.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.546+0000 7f22edb5c640 10 start_mon_command cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}] 2024-03-27T21:41:54.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.546+0000 7f22edb5c640 10 monclient: _send_command 2 [{"prefix": "osd reweight", "id": 4, "weight": 1.0}] 2024-03-27T21:41:54.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.546+0000 7f22edb5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:54.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:54.546+0000 7f22edb5c640 1 -- 172.21.15.60:0/3179947539 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7f22e80b2560 con 0x7f22e8071a50 2024-03-27T21:41:55.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.223+0000 7f22cffff640 1 -- 172.21.15.60:0/3179947539 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v83) v1 ==== 113+0+0 (secure 0 0 0) 0x7f22d8057d10 con 0x7f22e8071a50 2024-03-27T21:41:55.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.223+0000 7f22cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd reweight", "id": 4, "weight": 1.0}] 2024-03-27T21:41:55.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.223+0000 7f22cffff640 10 monclient: _finish_command 2 = system:0 reweighted osd.4 to 1 (10000) 2024-03-27T21:41:55.210 INFO:teuthology.orchestra.run.smithi060.stderr:reweighted osd.4 to 1 (10000) 2024-03-27T21:41:55.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.227+0000 7f22edb5c640 10 librados: watch_flush enter 2024-03-27T21:41:55.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.227+0000 7f22edb5c640 10 librados: watch_flush exit 2024-03-27T21:41:55.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.227+0000 7f22edb5c640 20 client.5214.objecter shutdown clearing up homeless session... 2024-03-27T21:41:55.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.227+0000 7f22edb5c640 10 client.5214.objecter successfully canceled tick 2024-03-27T21:41:55.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.227+0000 7f22edb5c640 1 -- 172.21.15.60:0/3179947539 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f22d406c590 msgr2=0x7f22d406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:55.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.227+0000 7f22edb5c640 1 --2- 172.21.15.60:0/3179947539 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f22d406c590 0x7f22d406ea50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f22e80aaf10 tx=0x7f22dc009040 comp rx=0 tx=0).stop 2024-03-27T21:41:55.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.227+0000 7f22edb5c640 10 monclient: shutdown 2024-03-27T21:41:55.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.227+0000 7f22edb5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:55.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.227+0000 7f22edb5c640 1 -- 172.21.15.60:0/3179947539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22e8071a50 msgr2=0x7f22e8139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:55.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.227+0000 7f22edb5c640 1 --2- 172.21.15.60:0/3179947539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22e8071a50 0x7f22e8139ec0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f22d800c7f0 tx=0x7f22d800cbf0 comp rx=0 tx=0).stop 2024-03-27T21:41:55.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.227+0000 7f22edb5c640 1 -- 172.21.15.60:0/3179947539 shutdown_connections 2024-03-27T21:41:55.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.227+0000 7f22edb5c640 1 --2- 172.21.15.60:0/3179947539 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f22e813ec50 0x7f22e8137c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:55.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.227+0000 7f22edb5c640 1 --2- 172.21.15.60:0/3179947539 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f22d406c590 0x7f22d406ea50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:55.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.227+0000 7f22edb5c640 1 --2- 172.21.15.60:0/3179947539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22e813e810 0x7f22e81376d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:55.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.227+0000 7f22edb5c640 1 --2- 172.21.15.60:0/3179947539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22e8071a50 0x7f22e8139ec0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:55.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.227+0000 7f22edb5c640 1 -- 172.21.15.60:0/3179947539 >> 172.21.15.60:0/3179947539 conn(0x7f22e80b7300 msgr2=0x7f22e80a8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:55.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.227+0000 7f22edb5c640 1 -- 172.21.15.60:0/3179947539 shutdown_connections 2024-03-27T21:41:55.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.227+0000 7f22edb5c640 1 -- 172.21.15.60:0/3179947539 wait complete. 2024-03-27T21:41:55.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.227+0000 7f22edb5c640 1 librados: shutdown 2024-03-27T21:41:55.227 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-27T21:41:55.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.347+0000 7f5c9a5ab640 10 monclient: get_monmap_and_config 2024-03-27T21:41:55.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.347+0000 7f5c9a5ab640 10 monclient: build_initial_monmap 2024-03-27T21:41:55.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.347+0000 7f5c9a5ab640 10 monclient: monmap: 2024-03-27T21:41:55.334 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:55.334 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:55.334 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:55.348882+0000 2024-03-27T21:41:55.335 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:55.348882+0000 2024-03-27T21:41:55.335 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:55.335 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:55.335 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:55.335 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:55.335 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:55.335 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:55.335 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:55.336 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:55.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.348+0000 7f5c9a5ab640 1 Processor -- start 2024-03-27T21:41:55.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.348+0000 7f5c9a5ab640 1 -- start start 2024-03-27T21:41:55.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.348+0000 7f5c9a5ab640 10 monclient: init 2024-03-27T21:41:55.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c9a5ab640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:55.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c9a5ab640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-03-27T21:41:55.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c9a5ab640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5c94071a50 0x7f5c940a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:55.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c9a5ab640 10 monclient(hunting): picked mon.noname-a con 0x7f5c94071a50 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:55.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c9a5ab640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5c9413e810 0x7f5c9413ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:55.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c9a5ab640 10 monclient(hunting): picked mon.noname-c con 0x7f5c9413e810 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:55.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c9a5ab640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5c94141630 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:55.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c9a5ab640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9413f190 con 0x7f5c94141630 2024-03-27T21:41:55.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c9a5ab640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:55.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c9a5ab640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c94071e30 con 0x7f5c94071a50 2024-03-27T21:41:55.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c9a5ab640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:55.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c9a5ab640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c940a8e90 con 0x7f5c9413e810 2024-03-27T21:41:55.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c9a5ab640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:55.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c9a5ab640 10 monclient(hunting): authenticate will time out at 956.570158s 2024-03-27T21:41:55.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c93fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5c94071a50 0x7f5c940a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:55.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c93fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5c94071a50 0x7f5c940a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55964/0 (socket says 172.21.15.60:55964) 2024-03-27T21:41:55.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c93fff640 1 -- 172.21.15.60:0/311879879 learned_addr learned my addr 172.21.15.60:0/311879879 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:55.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c93fff640 10 monclient(hunting): get_auth_request con 0x7f5c94071a50 auth_method 0 2024-03-27T21:41:55.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:55.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c93fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:55.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:55.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:55.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:55.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:55.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c937fe640 1 --2- 172.21.15.60:0/311879879 >> v2:172.21.15.191:3300/0 conn(0x7f5c9413e810 0x7f5c9413ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:55.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c937fe640 10 monclient(hunting): get_auth_request con 0x7f5c9413e810 auth_method 0 2024-03-27T21:41:55.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:55.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c937fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:55.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:55.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c93fff640 10 monclient(hunting): handle_auth_done global_id 5217 payload 274 2024-03-27T21:41:55.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c93fff640 1 -- 172.21.15.60:0/311879879 >> v2:172.21.15.191:3300/0 conn(0x7f5c9413e810 msgr2=0x7f5c9413ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:55.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c93fff640 1 --2- 172.21.15.60:0/311879879 >> v2:172.21.15.191:3300/0 conn(0x7f5c9413e810 0x7f5c9413ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:55.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c93fff640 1 -- 172.21.15.60:0/311879879 >> v1:172.21.15.60:6789/0 conn(0x7f5c94141630 legacy=0x7f5c94141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:55.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c93fff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:55.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c93fff640 1 monclient: found mon.noname-a 2024-03-27T21:41:55.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:55.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c93fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:55.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.349+0000 7f5c93fff640 1 -- 172.21.15.60:0/311879879 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c940aa040 con 0x7f5c94071a50 2024-03-27T21:41:55.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.350+0000 7f5c93fff640 1 --2- 172.21.15.60:0/311879879 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c94071a50 0x7f5c940a87e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5c940709f0 tx=0x7f5c8402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af0103c41b3b235b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:55.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.350+0000 7f5c92ffd640 1 -- 172.21.15.60:0/311879879 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c84030860 con 0x7f5c94071a50 2024-03-27T21:41:55.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.350+0000 7f5c92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:55.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.350+0000 7f5c92ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:55.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.350+0000 7f5c92ffd640 10 monclient: dump: 2024-03-27T21:41:55.350 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:55.350 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:55.350 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:55.350 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:55.350 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:55.351 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:55.351 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:55.351 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:55.351 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:55.351 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:55.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.350+0000 7f5c92ffd640 10 monclient: _finish_auth 0 2024-03-27T21:41:55.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.350+0000 7f5c92ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:41:55.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.350+0000 7f5c92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:55.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.350+0000 7f5c92ffd640 1 -- 172.21.15.60:0/311879879 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c840309c0 con 0x7f5c94071a50 2024-03-27T21:41:55.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.350+0000 7f5c9a5ab640 5 monclient: authenticate success, global_id 5217 2024-03-27T21:41:55.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.350+0000 7f5c9a5ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:55.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.350+0000 7f5c92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:55.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.350+0000 7f5c9a5ab640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:55.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.350+0000 7f5c92ffd640 1 -- 172.21.15.60:0/311879879 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c84030d50 con 0x7f5c94071a50 2024-03-27T21:41:55.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.350+0000 7f5c92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:55.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.350+0000 7f5c9a5ab640 4 set_mon_vals no callback set 2024-03-27T21:41:55.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.350+0000 7f5c92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:55.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.350+0000 7f5c92ffd640 10 monclient: dump: 2024-03-27T21:41:55.353 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:55.353 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:55.353 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:55.353 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:55.353 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:55.353 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:55.353 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:55.353 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:55.353 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:55.354 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:55.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.351+0000 7f5c9a5ab640 10 monclient: shutdown 2024-03-27T21:41:55.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.351+0000 7f5c9a5ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:55.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.351+0000 7f5c9a5ab640 1 -- 172.21.15.60:0/311879879 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c94071a50 msgr2=0x7f5c940a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:55.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.351+0000 7f5c9a5ab640 1 --2- 172.21.15.60:0/311879879 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c94071a50 0x7f5c940a87e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5c940709f0 tx=0x7f5c8402b9a0 comp rx=0 tx=0).stop 2024-03-27T21:41:55.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.351+0000 7f5c9a5ab640 1 -- 172.21.15.60:0/311879879 shutdown_connections 2024-03-27T21:41:55.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.351+0000 7f5c9a5ab640 1 --2- 172.21.15.60:0/311879879 >> v2:172.21.15.191:3300/0 conn(0x7f5c9413e810 0x7f5c9413ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:55.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.351+0000 7f5c9a5ab640 1 --2- 172.21.15.60:0/311879879 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c94071a50 0x7f5c940a87e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:55.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.351+0000 7f5c9a5ab640 1 -- 172.21.15.60:0/311879879 >> 172.21.15.60:0/311879879 conn(0x7f5c940b7300 msgr2=0x7f5c940b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:55.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.351+0000 7f5c9a5ab640 1 -- 172.21.15.60:0/311879879 shutdown_connections 2024-03-27T21:41:55.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.351+0000 7f5c9a5ab640 1 -- 172.21.15.60:0/311879879 wait complete. 2024-03-27T21:41:55.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.351+0000 7f5c9a5ab640 10 monclient: build_initial_monmap 2024-03-27T21:41:55.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.351+0000 7f5c9a5ab640 10 monclient: monmap: 2024-03-27T21:41:55.355 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:55.356 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:55.356 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:55.356 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:55.356 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:55.356 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:55.356 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:55.356 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:55.356 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:55.356 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:55.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.351+0000 7f5c9a5ab640 1 librados: starting msgr at 2024-03-27T21:41:55.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.351+0000 7f5c9a5ab640 1 librados: starting objecter 2024-03-27T21:41:55.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c9a5ab640 1 Processor -- start 2024-03-27T21:41:55.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c9a5ab640 1 -- start start 2024-03-27T21:41:55.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c9a5ab640 1 librados: setting wanted keys 2024-03-27T21:41:55.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c9a5ab640 1 librados: calling monclient init 2024-03-27T21:41:55.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c9a5ab640 10 monclient: init 2024-03-27T21:41:55.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c9a5ab640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:55.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c9a5ab640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:41:55.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c9a5ab640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94071a50 0x7f5c941d9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:55.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c9a5ab640 10 monclient(hunting): picked mon.noname-b con 0x7f5c94071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:55.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c9a5ab640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c9413e810 0x7f5c940affb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:55.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c9a5ab640 10 monclient(hunting): picked mon.noname-a con 0x7f5c9413e810 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:55.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c9a5ab640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c94141630 0x7f5c940b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:55.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c9a5ab640 10 monclient(hunting): picked mon.noname-c con 0x7f5c94141630 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:55.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c9a5ab640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:55.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c9a5ab640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c940ab5b0 con 0x7f5c9413e810 2024-03-27T21:41:55.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c9a5ab640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:55.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c9a5ab640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9413a1a0 con 0x7f5c94071a50 2024-03-27T21:41:55.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c9a5ab640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:55.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c9a5ab640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c94139180 con 0x7f5c94141630 2024-03-27T21:41:55.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c9a5ab640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:55.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c9a5ab640 10 monclient(hunting): authenticate will time out at 956.573755s 2024-03-27T21:41:55.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c937fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c9413e810 0x7f5c940affb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:55.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c937fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c9413e810 0x7f5c940affb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55968/0 (socket says 172.21.15.60:55968) 2024-03-27T21:41:55.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c937fe640 1 -- 172.21.15.60:0/3559443582 learned_addr learned my addr 172.21.15.60:0/3559443582 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:55.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c937fe640 10 monclient(hunting): get_auth_request con 0x7f5c9413e810 auth_method 0 2024-03-27T21:41:55.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:55.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c937fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:55.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:55.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.352+0000 7f5c98b48640 1 --2- 172.21.15.60:0/3559443582 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c94141630 0x7f5c940b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:55.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c98b48640 10 monclient(hunting): get_auth_request con 0x7f5c94141630 auth_method 0 2024-03-27T21:41:55.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c98b48640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:55.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c98b48640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:55.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c98b48640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:55.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:55.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:55.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:55.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c98b48640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:55.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c98b48640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:55.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c98b48640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:55.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c93fff640 1 --2- 172.21.15.60:0/3559443582 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94071a50 0x7f5c941d9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:55.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c93fff640 10 monclient(hunting): get_auth_request con 0x7f5c94071a50 auth_method 0 2024-03-27T21:41:55.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:55.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c93fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:55.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:55.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c937fe640 10 monclient(hunting): handle_auth_done global_id 5220 payload 824 2024-03-27T21:41:55.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c937fe640 1 -- 172.21.15.60:0/3559443582 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c94141630 msgr2=0x7f5c940b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:55.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c937fe640 1 --2- 172.21.15.60:0/3559443582 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c94141630 0x7f5c940b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:55.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c937fe640 1 -- 172.21.15.60:0/3559443582 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94071a50 msgr2=0x7f5c941d9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:55.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c937fe640 1 --2- 172.21.15.60:0/3559443582 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94071a50 0x7f5c941d9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:55.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c937fe640 10 monclient: _finish_hunting 0 2024-03-27T21:41:55.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c937fe640 1 monclient: found mon.noname-a 2024-03-27T21:41:55.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:55.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c937fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:55.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c937fe640 1 -- 172.21.15.60:0/3559443582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c840094d0 con 0x7f5c9413e810 2024-03-27T21:41:55.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c98b48640 1 --2- 172.21.15.60:0/3559443582 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c94141630 0x7f5c940b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:55.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c937fe640 1 --2- 172.21.15.60:0/3559443582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c9413e810 0x7f5c940affb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c00d800 tx=0x7f5c7c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:55.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c90ff9640 10 client.?.objecter ms_handle_connect 0x7f5c9413e810 2024-03-27T21:41:55.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c90ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:55.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c90ff9640 1 -- 172.21.15.60:0/3559443582 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c00b660 con 0x7f5c9413e810 2024-03-27T21:41:55.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c90ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:55.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c90ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:55.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c90ff9640 10 monclient: dump: 2024-03-27T21:41:55.365 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:55.365 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:55.365 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:55.365 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:55.365 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:55.366 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:55.366 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:55.366 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:55.366 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:55.366 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:55.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c90ff9640 10 monclient: _finish_auth 0 2024-03-27T21:41:55.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c90ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:41:55.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c90ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:55.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c90ff9640 1 -- 172.21.15.60:0/3559443582 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7c00b7c0 con 0x7f5c9413e810 2024-03-27T21:41:55.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c9a5ab640 5 monclient: authenticate success, global_id 5220 2024-03-27T21:41:55.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c9a5ab640 10 monclient: _renew_subs 2024-03-27T21:41:55.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c9a5ab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:55.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c9a5ab640 1 -- 172.21.15.60:0/3559443582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c941da530 con 0x7f5c9413e810 2024-03-27T21:41:55.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c90ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:55.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c90ff9640 1 -- 172.21.15.60:0/3559443582 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c00bb50 con 0x7f5c9413e810 2024-03-27T21:41:55.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c90ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:55.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c90ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:55.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c90ff9640 10 monclient: dump: 2024-03-27T21:41:55.368 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:55.368 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:55.368 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:55.368 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:55.368 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:55.368 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:55.368 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:55.369 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:55.369 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:55.369 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:55.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c927fc640 4 set_mon_vals no callback set 2024-03-27T21:41:55.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c9a5ab640 10 client.5220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:55.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c9a5ab640 10 monclient: _renew_subs 2024-03-27T21:41:55.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c9a5ab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:55.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c9a5ab640 1 -- 172.21.15.60:0/3559443582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c94139f30 con 0x7f5c9413e810 2024-03-27T21:41:55.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.353+0000 7f5c9a5ab640 1 librados: init done 2024-03-27T21:41:55.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.354+0000 7f5c9a5ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:55.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.354+0000 7f5c9a5ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:55.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.354+0000 7f5c9a5ab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:55.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.354+0000 7f5c9a5ab640 1 -- 172.21.15.60:0/3559443582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c941da7a0 con 0x7f5c9413e810 2024-03-27T21:41:55.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.355+0000 7f5c90ff9640 1 -- 172.21.15.60:0/3559443582 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5c7c00bcb0 con 0x7f5c9413e810 2024-03-27T21:41:55.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.355+0000 7f5c90ff9640 1 --2- 172.21.15.60:0/3559443582 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5c7406c590 0x7f5c7406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:55.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.355+0000 7f5c90ff9640 1 -- 172.21.15.60:0/3559443582 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f5c7c089570 con 0x7f5c9413e810 2024-03-27T21:41:55.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.355+0000 7f5c90ff9640 10 client.5220.objecter ms_dispatch 0x7f5c94000b90 osd_map(83..83 src has 1..83) v4 2024-03-27T21:41:55.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.355+0000 7f5c90ff9640 3 client.5220.objecter handle_osd_map got epochs [83,83] > 0 2024-03-27T21:41:55.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.355+0000 7f5c90ff9640 3 client.5220.objecter handle_osd_map decoding full epoch 83 2024-03-27T21:41:55.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.355+0000 7f5c93fff640 1 --2- 172.21.15.60:0/3559443582 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5c7406c590 0x7f5c7406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:55.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.355+0000 7f5c93fff640 10 monclient: get_auth_request con 0x7f5c7406c590 auth_method 0 2024-03-27T21:41:55.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.355+0000 7f5c90ff9640 20 client.5220.objecter dump_active .. 0 homeless 2024-03-27T21:41:55.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.356+0000 7f5c90ff9640 10 client.5220.objecter ms_handle_connect 0x7f5c7406c590 2024-03-27T21:41:55.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.356+0000 7f5c93fff640 1 --2- 172.21.15.60:0/3559443582 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5c7406c590 0x7f5c7406ea50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5c84036000 tx=0x7f5c84008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:55.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.359+0000 7f5c90ff9640 1 -- 172.21.15.60:0/3559443582 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5c941da7a0 con 0x7f5c9413e810 2024-03-27T21:41:55.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.359+0000 7f5c90ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:55.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.359+0000 7f5c90ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:55.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.554+0000 7f5c9a5ab640 10 start_mon_command cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}] 2024-03-27T21:41:55.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.554+0000 7f5c9a5ab640 10 monclient: _send_command 2 [{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}] 2024-03-27T21:41:55.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.554+0000 7f5c9a5ab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:55.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.554+0000 7f5c9a5ab640 1 -- 172.21.15.60:0/3559443582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} v 0) v1 -- 0x7f5c941da770 con 0x7f5c9413e810 2024-03-27T21:41:55.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.932+0000 7f5c90ff9640 1 -- 172.21.15.60:0/3559443582 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]=0 set osd.4 primary-affinity to 1 (8655362) v84) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5c7c0549b0 con 0x7f5c9413e810 2024-03-27T21:41:55.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.932+0000 7f5c90ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}] 2024-03-27T21:41:55.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.932+0000 7f5c90ff9640 10 monclient: _finish_command 2 = system:0 set osd.4 primary-affinity to 1 (8655362) 2024-03-27T21:41:55.919 INFO:teuthology.orchestra.run.smithi060.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-27T21:41:55.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.934+0000 7f5c9a5ab640 10 librados: watch_flush enter 2024-03-27T21:41:55.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.934+0000 7f5c9a5ab640 10 librados: watch_flush exit 2024-03-27T21:41:55.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.934+0000 7f5c9a5ab640 20 client.5220.objecter shutdown clearing up homeless session... 2024-03-27T21:41:55.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.934+0000 7f5c9a5ab640 10 client.5220.objecter successfully canceled tick 2024-03-27T21:41:55.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.934+0000 7f5c9a5ab640 1 -- 172.21.15.60:0/3559443582 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5c7406c590 msgr2=0x7f5c7406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:55.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.934+0000 7f5c9a5ab640 1 --2- 172.21.15.60:0/3559443582 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5c7406c590 0x7f5c7406ea50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5c84036000 tx=0x7f5c84008030 comp rx=0 tx=0).stop 2024-03-27T21:41:55.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.934+0000 7f5c9a5ab640 10 monclient: shutdown 2024-03-27T21:41:55.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.934+0000 7f5c9a5ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:55.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.934+0000 7f5c9a5ab640 1 -- 172.21.15.60:0/3559443582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c9413e810 msgr2=0x7f5c940affb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:55.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.934+0000 7f5c9a5ab640 1 --2- 172.21.15.60:0/3559443582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c9413e810 0x7f5c940affb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c00d800 tx=0x7f5c7c00dcc0 comp rx=0 tx=0).stop 2024-03-27T21:41:55.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.934+0000 7f5c9a5ab640 1 -- 172.21.15.60:0/3559443582 shutdown_connections 2024-03-27T21:41:55.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.934+0000 7f5c9a5ab640 1 --2- 172.21.15.60:0/3559443582 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5c94141630 0x7f5c940b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:55.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.934+0000 7f5c9a5ab640 1 --2- 172.21.15.60:0/3559443582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c9413e810 0x7f5c940affb0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:55.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.934+0000 7f5c9a5ab640 1 --2- 172.21.15.60:0/3559443582 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5c7406c590 0x7f5c7406ea50 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:55.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.934+0000 7f5c9a5ab640 1 --2- 172.21.15.60:0/3559443582 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c94071a50 0x7f5c941d9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:55.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.934+0000 7f5c9a5ab640 1 -- 172.21.15.60:0/3559443582 >> 172.21.15.60:0/3559443582 conn(0x7f5c940b7300 msgr2=0x7f5c94141a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:55.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.934+0000 7f5c9a5ab640 1 -- 172.21.15.60:0/3559443582 shutdown_connections 2024-03-27T21:41:55.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.934+0000 7f5c9a5ab640 1 -- 172.21.15.60:0/3559443582 wait complete. 2024-03-27T21:41:55.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:55.934+0000 7f5c9a5ab640 1 librados: shutdown 2024-03-27T21:41:55.933 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-27T21:41:56.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.051+0000 7f6195ec0640 10 monclient: get_monmap_and_config 2024-03-27T21:41:56.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.051+0000 7f6195ec0640 10 monclient: build_initial_monmap 2024-03-27T21:41:56.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.051+0000 7f6195ec0640 10 monclient: monmap: 2024-03-27T21:41:56.038 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:56.038 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:56.039 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:56.052947+0000 2024-03-27T21:41:56.039 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:56.052947+0000 2024-03-27T21:41:56.039 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:56.039 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:56.039 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:56.039 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:56.039 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:56.040 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:56.040 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:56.040 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:56.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.052+0000 7f6195ec0640 1 Processor -- start 2024-03-27T21:41:56.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.052+0000 7f6195ec0640 1 -- start start 2024-03-27T21:41:56.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.052+0000 7f6195ec0640 10 monclient: init 2024-03-27T21:41:56.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f6195ec0640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:56.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f6195ec0640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-03-27T21:41:56.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f6195ec0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6190071a50 0x7f61900a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:56.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f6195ec0640 10 monclient(hunting): picked mon.noname-a con 0x7f6190071a50 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:56.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f6195ec0640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f619013e810 0x7f619013ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:56.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f6195ec0640 10 monclient(hunting): picked mon.noname-b con 0x7f619013e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:56.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f6195ec0640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6190141630 0x7f6190141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:56.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f6195ec0640 10 monclient(hunting): picked mon.noname-c con 0x7f6190141630 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:56.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f6195ec0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:56.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f6195ec0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6190071e30 con 0x7f6190071a50 2024-03-27T21:41:56.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f6195ec0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:56.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f6195ec0640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f619013f190 con 0x7f6190141630 2024-03-27T21:41:56.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f6195ec0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:56.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f6195ec0640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61900a8e90 con 0x7f619013e810 2024-03-27T21:41:56.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f6195ec0640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:56.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f6195ec0640 10 monclient(hunting): authenticate will time out at 957.274193s 2024-03-27T21:41:56.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6190071a50 0x7f61900a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:56.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618effd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f619013e810 0x7f619013ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:56.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6190071a50 0x7f61900a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55982/0 (socket says 172.21.15.60:55982) 2024-03-27T21:41:56.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618f7fe640 1 -- 172.21.15.60:0/3369156839 learned_addr learned my addr 172.21.15.60:0/3369156839 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:56.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618f7fe640 10 monclient(hunting): get_auth_request con 0x7f6190071a50 auth_method 0 2024-03-27T21:41:56.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618effd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f619013e810 0x7f619013ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:38948/0 (socket says 172.21.15.60:38948) 2024-03-27T21:41:56.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:56.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:56.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:56.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618effd640 10 monclient(hunting): get_auth_request con 0x7f619013e810 auth_method 0 2024-03-27T21:41:56.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:56.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:56.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:56.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:56.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:56.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:56.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:56.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:56.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:56.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618ffff640 1 --2- 172.21.15.60:0/3369156839 >> v2:172.21.15.191:3300/0 conn(0x7f6190141630 0x7f6190141aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:56.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618effd640 10 monclient(hunting): handle_auth_done global_id 4949 payload 274 2024-03-27T21:41:56.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618effd640 1 -- 172.21.15.60:0/3369156839 >> v2:172.21.15.191:3300/0 conn(0x7f6190141630 msgr2=0x7f6190141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:56.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618effd640 1 --2- 172.21.15.60:0/3369156839 >> v2:172.21.15.191:3300/0 conn(0x7f6190141630 0x7f6190141aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:56.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618effd640 1 -- 172.21.15.60:0/3369156839 >> v2:172.21.15.60:3300/0 conn(0x7f6190071a50 msgr2=0x7f61900a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:56.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618effd640 1 --2- 172.21.15.60:0/3369156839 >> v2:172.21.15.60:3300/0 conn(0x7f6190071a50 0x7f61900a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:56.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618effd640 10 monclient: _finish_hunting 0 2024-03-27T21:41:56.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618effd640 1 monclient: found mon.noname-b 2024-03-27T21:41:56.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:56.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:56.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618effd640 1 -- 172.21.15.60:0/3369156839 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61900aa040 con 0x7f619013e810 2024-03-27T21:41:56.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.053+0000 7f618f7fe640 1 --2- 172.21.15.60:0/3369156839 >> v2:172.21.15.60:3300/0 conn(0x7f6190071a50 0x7f61900a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:56.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.054+0000 7f618effd640 1 --2- 172.21.15.60:0/3369156839 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f619013e810 0x7f619013ec50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6190071250 tx=0x7f618402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20855dc5a58a954 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:56.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.054+0000 7f618e7fc640 1 -- 172.21.15.60:0/3369156839 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618402e070 con 0x7f619013e810 2024-03-27T21:41:56.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.054+0000 7f618e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:56.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.054+0000 7f618e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:56.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.054+0000 7f618e7fc640 10 monclient: dump: 2024-03-27T21:41:56.060 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:56.061 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:56.061 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:56.061 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:56.062 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:56.062 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:56.062 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:56.062 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:56.062 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:56.063 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:56.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.054+0000 7f618e7fc640 10 monclient: _finish_auth 0 2024-03-27T21:41:56.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.054+0000 7f618e7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:41:56.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.054+0000 7f618e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:56.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.054+0000 7f618e7fc640 1 -- 172.21.15.60:0/3369156839 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61840315a0 con 0x7f619013e810 2024-03-27T21:41:56.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.054+0000 7f618e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:56.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.054+0000 7f618e7fc640 1 -- 172.21.15.60:0/3369156839 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6184031930 con 0x7f619013e810 2024-03-27T21:41:56.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.054+0000 7f6195ec0640 5 monclient: authenticate success, global_id 4949 2024-03-27T21:41:56.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.054+0000 7f6195ec0640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:56.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.054+0000 7f6195ec0640 4 set_mon_vals no callback set 2024-03-27T21:41:56.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.054+0000 7f618e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:56.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.054+0000 7f618e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:56.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.054+0000 7f618e7fc640 10 monclient: dump: 2024-03-27T21:41:56.064 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:56.064 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:56.064 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:56.065 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:56.065 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:56.065 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:56.065 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:56.065 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:56.065 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:56.065 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:56.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.055+0000 7f6195ec0640 10 monclient: shutdown 2024-03-27T21:41:56.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.055+0000 7f6195ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:56.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.055+0000 7f6195ec0640 1 -- 172.21.15.60:0/3369156839 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f619013e810 msgr2=0x7f619013ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:56.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.055+0000 7f6195ec0640 1 --2- 172.21.15.60:0/3369156839 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f619013e810 0x7f619013ec50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6190071250 tx=0x7f618402b840 comp rx=0 tx=0).stop 2024-03-27T21:41:56.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.055+0000 7f6195ec0640 1 -- 172.21.15.60:0/3369156839 shutdown_connections 2024-03-27T21:41:56.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.055+0000 7f6195ec0640 1 --2- 172.21.15.60:0/3369156839 >> v2:172.21.15.191:3300/0 conn(0x7f6190141630 0x7f6190141aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:56.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.055+0000 7f6195ec0640 1 --2- 172.21.15.60:0/3369156839 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f619013e810 0x7f619013ec50 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:56.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.055+0000 7f6195ec0640 1 --2- 172.21.15.60:0/3369156839 >> v2:172.21.15.60:3300/0 conn(0x7f6190071a50 0x7f61900a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:56.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.055+0000 7f6195ec0640 1 -- 172.21.15.60:0/3369156839 >> 172.21.15.60:0/3369156839 conn(0x7f61900b7300 msgr2=0x7f61900b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:56.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.055+0000 7f6195ec0640 1 -- 172.21.15.60:0/3369156839 shutdown_connections 2024-03-27T21:41:56.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.055+0000 7f6195ec0640 1 -- 172.21.15.60:0/3369156839 wait complete. 2024-03-27T21:41:56.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.055+0000 7f6195ec0640 10 monclient: build_initial_monmap 2024-03-27T21:41:56.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.055+0000 7f6195ec0640 10 monclient: monmap: 2024-03-27T21:41:56.068 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:56.068 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:56.068 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:56.068 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:56.068 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:56.068 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:56.069 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:56.069 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:56.069 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:56.069 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:56.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 1 librados: starting msgr at 2024-03-27T21:41:56.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 1 librados: starting objecter 2024-03-27T21:41:56.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 1 Processor -- start 2024-03-27T21:41:56.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 1 -- start start 2024-03-27T21:41:56.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 1 librados: setting wanted keys 2024-03-27T21:41:56.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 1 librados: calling monclient init 2024-03-27T21:41:56.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 10 monclient: init 2024-03-27T21:41:56.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:56.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:41:56.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6190071a50 0x7f61900aef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:56.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 10 monclient(hunting): picked mon.noname-b con 0x7f6190071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:56.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f619013e810 0x7f61900ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:56.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 10 monclient(hunting): picked mon.noname-c con 0x7f619013e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:56.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6190141630 0x7f61901d6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:56.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 10 monclient(hunting): picked mon.noname-a con 0x7f6190141630 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:56.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:56.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61901d66c0 con 0x7f6190141630 2024-03-27T21:41:56.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:56.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61900ada90 con 0x7f6190071a50 2024-03-27T21:41:56.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:56.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61900abcf0 con 0x7f619013e810 2024-03-27T21:41:56.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:56.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f6195ec0640 10 monclient(hunting): authenticate will time out at 957.277780s 2024-03-27T21:41:56.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f618effd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f619013e810 0x7f61900ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:56.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f618ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6190141630 0x7f61901d6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:56.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f618effd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f619013e810 0x7f61900ae640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:38960/0 (socket says 172.21.15.60:38960) 2024-03-27T21:41:56.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f618ffff640 10 monclient(hunting): get_auth_request con 0x7f6190141630 auth_method 0 2024-03-27T21:41:56.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f618effd640 1 -- 172.21.15.60:0/3417459798 learned_addr learned my addr 172.21.15.60:0/3417459798 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:56.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f618ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:56.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f618ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:56.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.056+0000 7f618ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:56.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618effd640 10 monclient(hunting): get_auth_request con 0x7f619013e810 auth_method 0 2024-03-27T21:41:56.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:56.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:56.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:56.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:56.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:56.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:56.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:56.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:56.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:56.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618f7fe640 1 --2- 172.21.15.60:0/3417459798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6190071a50 0x7f61900aef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:56.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618f7fe640 10 monclient(hunting): get_auth_request con 0x7f6190071a50 auth_method 0 2024-03-27T21:41:56.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:56.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:56.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:56.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:56.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:56.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:56.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618ffff640 10 monclient(hunting): handle_auth_done global_id 5226 payload 824 2024-03-27T21:41:56.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618ffff640 1 -- 172.21.15.60:0/3417459798 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f619013e810 msgr2=0x7f61900ae640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:56.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618ffff640 1 --2- 172.21.15.60:0/3417459798 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f619013e810 0x7f61900ae640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:56.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618ffff640 1 -- 172.21.15.60:0/3417459798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6190071a50 msgr2=0x7f61900aef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:56.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618ffff640 1 --2- 172.21.15.60:0/3417459798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6190071a50 0x7f61900aef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:56.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618ffff640 10 monclient: _finish_hunting 0 2024-03-27T21:41:56.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618ffff640 1 monclient: found mon.noname-a 2024-03-27T21:41:56.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:56.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:56.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618ffff640 1 -- 172.21.15.60:0/3417459798 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61800094d0 con 0x7f6190141630 2024-03-27T21:41:56.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618effd640 1 --2- 172.21.15.60:0/3417459798 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f619013e810 0x7f61900ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:56.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618ffff640 1 --2- 172.21.15.60:0/3417459798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6190141630 0x7f61901d6ae0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f618800cc80 tx=0x7f618800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:56.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6177fff640 10 client.?.objecter ms_handle_connect 0x7f6190141630 2024-03-27T21:41:56.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6177fff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:56.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6177fff640 1 -- 172.21.15.60:0/3417459798 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618800ce30 con 0x7f6190141630 2024-03-27T21:41:56.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6177fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:56.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6177fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:56.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6177fff640 10 monclient: dump: 2024-03-27T21:41:56.078 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:56.078 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:56.078 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:56.078 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:56.078 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:56.078 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:56.078 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:56.078 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:56.078 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:56.079 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:56.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6177fff640 10 monclient: _finish_auth 0 2024-03-27T21:41:56.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6177fff640 10 monclient: _check_auth_tickets 2024-03-27T21:41:56.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6177fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:56.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6177fff640 1 -- 172.21.15.60:0/3417459798 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6188003990 con 0x7f6190141630 2024-03-27T21:41:56.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6177fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:56.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6195ec0640 5 monclient: authenticate success, global_id 5226 2024-03-27T21:41:56.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6177fff640 1 -- 172.21.15.60:0/3417459798 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6188003d20 con 0x7f6190141630 2024-03-27T21:41:56.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6177fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:56.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6177fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:56.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6177fff640 10 monclient: dump: 2024-03-27T21:41:56.080 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:56.081 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:56.081 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:56.081 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:56.081 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:56.081 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:56.081 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:56.081 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:56.082 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:56.082 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:56.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f618d7fa640 4 set_mon_vals no callback set 2024-03-27T21:41:56.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6195ec0640 10 monclient: _renew_subs 2024-03-27T21:41:56.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6195ec0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:56.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6195ec0640 1 -- 172.21.15.60:0/3417459798 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6184009400 con 0x7f6190141630 2024-03-27T21:41:56.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6195ec0640 10 client.5226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:56.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6195ec0640 10 monclient: _renew_subs 2024-03-27T21:41:56.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6195ec0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:56.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6195ec0640 1 -- 172.21.15.60:0/3417459798 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61901d7020 con 0x7f6190141630 2024-03-27T21:41:56.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.057+0000 7f6195ec0640 1 librados: init done 2024-03-27T21:41:56.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.058+0000 7f6195ec0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:56.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.058+0000 7f6195ec0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:56.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.058+0000 7f6195ec0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:56.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.058+0000 7f6195ec0640 1 -- 172.21.15.60:0/3417459798 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61900b0b00 con 0x7f6190141630 2024-03-27T21:41:56.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.059+0000 7f6177fff640 1 -- 172.21.15.60:0/3417459798 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6188003e80 con 0x7f6190141630 2024-03-27T21:41:56.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.059+0000 7f6177fff640 1 --2- 172.21.15.60:0/3417459798 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f617c06c590 0x7f617c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:56.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.059+0000 7f6177fff640 1 -- 172.21.15.60:0/3417459798 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f6188089340 con 0x7f6190141630 2024-03-27T21:41:56.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.059+0000 7f6177fff640 10 client.5226.objecter ms_dispatch 0x7f6190000b90 osd_map(84..84 src has 1..84) v4 2024-03-27T21:41:56.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.059+0000 7f6177fff640 3 client.5226.objecter handle_osd_map got epochs [84,84] > 0 2024-03-27T21:41:56.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.059+0000 7f6177fff640 3 client.5226.objecter handle_osd_map decoding full epoch 84 2024-03-27T21:41:56.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.059+0000 7f618f7fe640 1 --2- 172.21.15.60:0/3417459798 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f617c06c590 0x7f617c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:56.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.059+0000 7f618f7fe640 10 monclient: get_auth_request con 0x7f617c06c590 auth_method 0 2024-03-27T21:41:56.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.059+0000 7f6177fff640 20 client.5226.objecter dump_active .. 0 homeless 2024-03-27T21:41:56.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.059+0000 7f6177fff640 10 client.5226.objecter ms_handle_connect 0x7f617c06c590 2024-03-27T21:41:56.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.059+0000 7f618f7fe640 1 --2- 172.21.15.60:0/3417459798 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f617c06c590 0x7f617c06ea50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6180006ca0 tx=0x7f6180009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:56.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.063+0000 7f6177fff640 1 -- 172.21.15.60:0/3417459798 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f61900b0b00 con 0x7f6190141630 2024-03-27T21:41:56.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.063+0000 7f6177fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:56.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.063+0000 7f6177fff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:56.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.256+0000 7f6195ec0640 10 start_mon_command cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}] 2024-03-27T21:41:56.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.256+0000 7f6195ec0640 10 monclient: _send_command 2 [{"prefix": "osd reweight", "id": 5, "weight": 1.0}] 2024-03-27T21:41:56.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.256+0000 7f6195ec0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:56.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:56.256+0000 7f6195ec0640 1 -- 172.21.15.60:0/3417459798 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) v1 -- 0x7f61901d6800 con 0x7f6190141630 2024-03-27T21:41:57.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.239+0000 7f6177fff640 1 -- 172.21.15.60:0/3417459798 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v85) v1 ==== 113+0+0 (secure 0 0 0) 0x7f61901d6800 con 0x7f6190141630 2024-03-27T21:41:57.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.239+0000 7f6177fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd reweight", "id": 5, "weight": 1.0}] 2024-03-27T21:41:57.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.239+0000 7f6177fff640 10 monclient: _finish_command 2 = system:0 reweighted osd.5 to 1 (10000) 2024-03-27T21:41:57.226 INFO:teuthology.orchestra.run.smithi060.stderr:reweighted osd.5 to 1 (10000) 2024-03-27T21:41:57.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.241+0000 7f6195ec0640 10 librados: watch_flush enter 2024-03-27T21:41:57.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.241+0000 7f6195ec0640 10 librados: watch_flush exit 2024-03-27T21:41:57.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.241+0000 7f6195ec0640 20 client.5226.objecter shutdown clearing up homeless session... 2024-03-27T21:41:57.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.241+0000 7f6195ec0640 10 client.5226.objecter successfully canceled tick 2024-03-27T21:41:57.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.242+0000 7f6195ec0640 1 -- 172.21.15.60:0/3417459798 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f617c06c590 msgr2=0x7f617c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:57.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.242+0000 7f6195ec0640 1 --2- 172.21.15.60:0/3417459798 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f617c06c590 0x7f617c06ea50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6180006ca0 tx=0x7f6180009240 comp rx=0 tx=0).stop 2024-03-27T21:41:57.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.242+0000 7f6195ec0640 10 monclient: shutdown 2024-03-27T21:41:57.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.242+0000 7f6195ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:57.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.242+0000 7f6195ec0640 1 -- 172.21.15.60:0/3417459798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6190141630 msgr2=0x7f61901d6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:57.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.242+0000 7f6195ec0640 1 --2- 172.21.15.60:0/3417459798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6190141630 0x7f61901d6ae0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f618800cc80 tx=0x7f618800a5c0 comp rx=0 tx=0).stop 2024-03-27T21:41:57.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.242+0000 7f6195ec0640 1 -- 172.21.15.60:0/3417459798 shutdown_connections 2024-03-27T21:41:57.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.242+0000 7f6195ec0640 1 --2- 172.21.15.60:0/3417459798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6190141630 0x7f61901d6ae0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:57.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.242+0000 7f6195ec0640 1 --2- 172.21.15.60:0/3417459798 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f619013e810 0x7f61900ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:57.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.242+0000 7f6195ec0640 1 --2- 172.21.15.60:0/3417459798 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f617c06c590 0x7f617c06ea50 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:57.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.242+0000 7f6195ec0640 1 --2- 172.21.15.60:0/3417459798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6190071a50 0x7f61900aef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:57.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.242+0000 7f6195ec0640 1 -- 172.21.15.60:0/3417459798 >> 172.21.15.60:0/3417459798 conn(0x7f61900b7300 msgr2=0x7f61900a8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:57.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.242+0000 7f6195ec0640 1 -- 172.21.15.60:0/3417459798 shutdown_connections 2024-03-27T21:41:57.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.242+0000 7f6195ec0640 1 -- 172.21.15.60:0/3417459798 wait complete. 2024-03-27T21:41:57.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.242+0000 7f6195ec0640 1 librados: shutdown 2024-03-27T21:41:57.241 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-27T21:41:57.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.363+0000 7f1963395640 10 monclient: get_monmap_and_config 2024-03-27T21:41:57.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.363+0000 7f1963395640 10 monclient: build_initial_monmap 2024-03-27T21:41:57.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.363+0000 7f1963395640 10 monclient: monmap: 2024-03-27T21:41:57.350 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:57.350 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:57.351 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:57.364335+0000 2024-03-27T21:41:57.351 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:57.364335+0000 2024-03-27T21:41:57.351 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:57.351 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:57.351 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:57.351 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:57.351 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:57.351 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:57.351 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:57.352 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:57.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.363+0000 7f1963395640 1 Processor -- start 2024-03-27T21:41:57.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1963395640 1 -- start start 2024-03-27T21:41:57.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1963395640 10 monclient: init 2024-03-27T21:41:57.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1963395640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:57.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1963395640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-03-27T21:41:57.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1963395640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f195c071a50 0x7f195c0a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:57.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1963395640 10 monclient(hunting): picked mon.noname-b con 0x7f195c071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:57.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1963395640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f195c13e810 addr v1:172.21.15.191:6789/0 2024-03-27T21:41:57.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1963395640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f195c13ec50 0x7f195c141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:57.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1963395640 10 monclient(hunting): picked mon.noname-a con 0x7f195c13ec50 addr v2:172.21.15.60:3300/0 2024-03-27T21:41:57.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1963395640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195c0a8e90 con 0x7f195c13e810 2024-03-27T21:41:57.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1963395640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:57.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1963395640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f195c071e30 con 0x7f195c13ec50 2024-03-27T21:41:57.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1963395640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:57.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1963395640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f195c0aa040 con 0x7f195c071a50 2024-03-27T21:41:57.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1963395640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:57.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1963395640 10 monclient(hunting): authenticate will time out at 958.585640s 2024-03-27T21:41:57.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1961131640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f195c071a50 0x7f195c0a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:57.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1961932640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f195c13ec50 0x7f195c141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:57.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1961131640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f195c071a50 0x7f195c0a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:38972/0 (socket says 172.21.15.60:38972) 2024-03-27T21:41:57.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1961131640 1 -- 172.21.15.60:0/3336817414 learned_addr learned my addr 172.21.15.60:0/3336817414 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:57.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1961932640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f195c13ec50 0x7f195c141d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55994/0 (socket says 172.21.15.60:55994) 2024-03-27T21:41:57.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1961131640 10 monclient(hunting): get_auth_request con 0x7f195c071a50 auth_method 0 2024-03-27T21:41:57.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1961131640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:57.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1961131640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:57.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1961131640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:57.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1961932640 10 monclient(hunting): get_auth_request con 0x7f195c13ec50 auth_method 0 2024-03-27T21:41:57.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1961932640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:57.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1961932640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:57.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1961932640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:57.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1961932640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:57.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.364+0000 7f1961932640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:57.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.365+0000 7f1961932640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:57.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.365+0000 7f1961131640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:57.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.365+0000 7f1961131640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:57.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.365+0000 7f1961131640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:57.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.365+0000 7f1961131640 10 monclient(hunting): handle_auth_done global_id 4955 payload 274 2024-03-27T21:41:57.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.365+0000 7f1961131640 1 -- 172.21.15.60:0/3336817414 >> v2:172.21.15.60:3300/0 conn(0x7f195c13ec50 msgr2=0x7f195c141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:57.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.365+0000 7f1961131640 1 --2- 172.21.15.60:0/3336817414 >> v2:172.21.15.60:3300/0 conn(0x7f195c13ec50 0x7f195c141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:57.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.365+0000 7f1961131640 1 -- 172.21.15.60:0/3336817414 >> v1:172.21.15.191:6789/0 conn(0x7f195c13e810 legacy=0x7f195c141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:57.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.365+0000 7f1961131640 10 monclient: _finish_hunting 0 2024-03-27T21:41:57.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.365+0000 7f1961131640 1 monclient: found mon.noname-b 2024-03-27T21:41:57.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.365+0000 7f1961131640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:57.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.365+0000 7f1961131640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:57.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.365+0000 7f1961131640 1 -- 172.21.15.60:0/3336817414 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195c142280 con 0x7f195c071a50 2024-03-27T21:41:57.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.365+0000 7f1961932640 1 --2- 172.21.15.60:0/3336817414 >> v2:172.21.15.60:3300/0 conn(0x7f195c13ec50 0x7f195c141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:57.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.365+0000 7f1961131640 1 --2- 172.21.15.60:0/3336817414 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f195c071a50 0x7f195c0a87e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1958009870 tx=0x7f195802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd4c17732802b7d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:57.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1953fff640 1 -- 172.21.15.60:0/3336817414 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1958031640 con 0x7f195c071a50 2024-03-27T21:41:57.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1953fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:57.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1953fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:57.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1953fff640 10 monclient: dump: 2024-03-27T21:41:57.374 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:57.375 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:57.375 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:57.375 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:57.375 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:57.376 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:57.376 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:57.376 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:57.376 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:57.377 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:57.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1953fff640 10 monclient: _finish_auth 0 2024-03-27T21:41:57.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1953fff640 10 monclient: _check_auth_tickets 2024-03-27T21:41:57.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1953fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:57.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1963395640 5 monclient: authenticate success, global_id 4955 2024-03-27T21:41:57.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1953fff640 1 -- 172.21.15.60:0/3336817414 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19580317a0 con 0x7f195c071a50 2024-03-27T21:41:57.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1963395640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:57.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1953fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:57.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1963395640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:57.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1953fff640 1 -- 172.21.15.60:0/3336817414 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1958031b30 con 0x7f195c071a50 2024-03-27T21:41:57.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1953fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:57.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1963395640 4 set_mon_vals no callback set 2024-03-27T21:41:57.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1953fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:57.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1953fff640 10 monclient: dump: 2024-03-27T21:41:57.380 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:57.380 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:57.381 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:57.381 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:57.381 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:57.381 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:57.381 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:57.381 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:57.382 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:57.382 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:57.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1963395640 10 monclient: shutdown 2024-03-27T21:41:57.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1963395640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:57.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1963395640 1 -- 172.21.15.60:0/3336817414 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f195c071a50 msgr2=0x7f195c0a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:57.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1963395640 1 --2- 172.21.15.60:0/3336817414 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f195c071a50 0x7f195c0a87e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1958009870 tx=0x7f195802b8f0 comp rx=0 tx=0).stop 2024-03-27T21:41:57.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1963395640 1 -- 172.21.15.60:0/3336817414 shutdown_connections 2024-03-27T21:41:57.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1963395640 1 --2- 172.21.15.60:0/3336817414 >> v2:172.21.15.60:3300/0 conn(0x7f195c13ec50 0x7f195c141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:57.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1963395640 1 --2- 172.21.15.60:0/3336817414 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f195c071a50 0x7f195c0a87e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:57.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1963395640 1 -- 172.21.15.60:0/3336817414 >> 172.21.15.60:0/3336817414 conn(0x7f195c0b7300 msgr2=0x7f195c0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:57.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1963395640 1 -- 172.21.15.60:0/3336817414 shutdown_connections 2024-03-27T21:41:57.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.366+0000 7f1963395640 1 -- 172.21.15.60:0/3336817414 wait complete. 2024-03-27T21:41:57.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.367+0000 7f1963395640 10 monclient: build_initial_monmap 2024-03-27T21:41:57.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.367+0000 7f1963395640 10 monclient: monmap: 2024-03-27T21:41:57.385 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:57.385 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:57.385 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:57.385 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:57.385 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:57.386 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:57.386 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:57.386 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:57.386 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:57.386 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:57.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.367+0000 7f1963395640 1 librados: starting msgr at 2024-03-27T21:41:57.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.367+0000 7f1963395640 1 librados: starting objecter 2024-03-27T21:41:57.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.367+0000 7f1963395640 1 Processor -- start 2024-03-27T21:41:57.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.367+0000 7f1963395640 1 -- start start 2024-03-27T21:41:57.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.367+0000 7f1963395640 1 librados: setting wanted keys 2024-03-27T21:41:57.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.367+0000 7f1963395640 1 librados: calling monclient init 2024-03-27T21:41:57.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.367+0000 7f1963395640 10 monclient: init 2024-03-27T21:41:57.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1963395640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:57.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1963395640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:41:57.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1963395640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f195c071a50 0x7f195c1d6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:57.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1963395640 10 monclient(hunting): picked mon.noname-b con 0x7f195c071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:57.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1963395640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f195c13e810 0x7f195c13a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:57.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1963395640 10 monclient(hunting): picked mon.noname-c con 0x7f195c13e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:57.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1963395640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f195c13ec50 0x7f195c13aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:57.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1963395640 10 monclient(hunting): picked mon.noname-a con 0x7f195c13ec50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:57.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1963395640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:57.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1963395640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c1d6920 con 0x7f195c13ec50 2024-03-27T21:41:57.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1963395640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:57.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1963395640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c139240 con 0x7f195c071a50 2024-03-27T21:41:57.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1963395640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:57.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1963395640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f195c13b1e0 con 0x7f195c13e810 2024-03-27T21:41:57.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1963395640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:57.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1963395640 10 monclient(hunting): authenticate will time out at 958.589230s 2024-03-27T21:41:57.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961932640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f195c13ec50 0x7f195c13aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:57.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1960930640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f195c13e810 0x7f195c13a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:57.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961932640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f195c13ec50 0x7f195c13aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56004/0 (socket says 172.21.15.60:56004) 2024-03-27T21:41:57.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1960930640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f195c13e810 0x7f195c13a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:38986/0 (socket says 172.21.15.60:38986) 2024-03-27T21:41:57.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961932640 1 -- 172.21.15.60:0/2409009421 learned_addr learned my addr 172.21.15.60:0/2409009421 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:57.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961932640 10 monclient(hunting): get_auth_request con 0x7f195c13ec50 auth_method 0 2024-03-27T21:41:57.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961932640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:57.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961932640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:57.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961932640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:57.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1960930640 10 monclient(hunting): get_auth_request con 0x7f195c13e810 auth_method 0 2024-03-27T21:41:57.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1960930640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:57.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1960930640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:57.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1960930640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:57.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961131640 1 --2- 172.21.15.60:0/2409009421 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f195c071a50 0x7f195c1d6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:57.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961131640 10 monclient(hunting): get_auth_request con 0x7f195c071a50 auth_method 0 2024-03-27T21:41:57.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961131640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:57.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961131640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:57.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961131640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:57.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961932640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:57.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961932640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:57.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961932640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:57.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1960930640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:57.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1960930640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:57.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1960930640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:57.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961131640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:57.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961131640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:57.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961131640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:57.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961932640 10 monclient(hunting): handle_auth_done global_id 5232 payload 824 2024-03-27T21:41:57.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961932640 1 -- 172.21.15.60:0/2409009421 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f195c13e810 msgr2=0x7f195c13a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:57.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961932640 1 --2- 172.21.15.60:0/2409009421 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f195c13e810 0x7f195c13a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:57.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961932640 1 -- 172.21.15.60:0/2409009421 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f195c071a50 msgr2=0x7f195c1d6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:57.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961932640 1 --2- 172.21.15.60:0/2409009421 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f195c071a50 0x7f195c1d6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:57.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961932640 10 monclient: _finish_hunting 0 2024-03-27T21:41:57.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961932640 1 monclient: found mon.noname-a 2024-03-27T21:41:57.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961932640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:57.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961932640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:41:57.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961932640 1 -- 172.21.15.60:0/2409009421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1954009a70 con 0x7f195c13ec50 2024-03-27T21:41:57.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1961932640 1 --2- 172.21.15.60:0/2409009421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f195c13ec50 0x7f195c13aca0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f195400efc0 tx=0x7f1954009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:57.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1951ffb640 10 client.?.objecter ms_handle_connect 0x7f195c13ec50 2024-03-27T21:41:57.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.368+0000 7f1951ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:57.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1951ffb640 1 -- 172.21.15.60:0/2409009421 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954009d70 con 0x7f195c13ec50 2024-03-27T21:41:57.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:57.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1951ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:41:57.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1951ffb640 10 monclient: dump: 2024-03-27T21:41:57.395 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:57.395 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:57.395 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:57.395 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:57.395 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:57.395 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:57.396 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:57.396 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:57.396 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:57.396 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:57.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1951ffb640 10 monclient: _finish_auth 0 2024-03-27T21:41:57.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1951ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:41:57.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1951ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:57.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1951ffb640 1 -- 172.21.15.60:0/2409009421 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195400f040 con 0x7f195c13ec50 2024-03-27T21:41:57.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1951ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:57.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1951ffb640 1 -- 172.21.15.60:0/2409009421 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954003990 con 0x7f195c13ec50 2024-03-27T21:41:57.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:57.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1951ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:41:57.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1951ffb640 10 monclient: dump: 2024-03-27T21:41:57.397 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:57.397 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:57.397 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:57.397 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:57.398 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:57.398 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:57.398 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:57.398 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:57.398 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:57.398 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:57.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1963395640 5 monclient: authenticate success, global_id 5232 2024-03-27T21:41:57.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1963395640 10 monclient: _renew_subs 2024-03-27T21:41:57.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1963395640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:57.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f19537fe640 4 set_mon_vals no callback set 2024-03-27T21:41:57.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1963395640 1 -- 172.21.15.60:0/2409009421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19580094d0 con 0x7f195c13ec50 2024-03-27T21:41:57.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1963395640 10 client.5232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:57.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1963395640 10 monclient: _renew_subs 2024-03-27T21:41:57.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1963395640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:57.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1963395640 1 -- 172.21.15.60:0/2409009421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f195c137680 con 0x7f195c13ec50 2024-03-27T21:41:57.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.369+0000 7f1963395640 1 librados: init done 2024-03-27T21:41:57.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.370+0000 7f1963395640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:57.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.370+0000 7f1963395640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:57.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.370+0000 7f1963395640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:57.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.370+0000 7f1963395640 1 -- 172.21.15.60:0/2409009421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f195c067900 con 0x7f195c13ec50 2024-03-27T21:41:57.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.370+0000 7f1951ffb640 1 -- 172.21.15.60:0/2409009421 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f19540144a0 con 0x7f195c13ec50 2024-03-27T21:41:57.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.371+0000 7f1951ffb640 1 --2- 172.21.15.60:0/2409009421 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f194006c590 0x7f194006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:57.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.371+0000 7f1951ffb640 1 -- 172.21.15.60:0/2409009421 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f1954089c20 con 0x7f195c13ec50 2024-03-27T21:41:57.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.371+0000 7f1951ffb640 10 client.5232.objecter ms_dispatch 0x7f195c000b90 osd_map(85..85 src has 1..85) v4 2024-03-27T21:41:57.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.371+0000 7f1951ffb640 3 client.5232.objecter handle_osd_map got epochs [85,85] > 0 2024-03-27T21:41:57.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.371+0000 7f1951ffb640 3 client.5232.objecter handle_osd_map decoding full epoch 85 2024-03-27T21:41:57.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.371+0000 7f1961131640 1 --2- 172.21.15.60:0/2409009421 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f194006c590 0x7f194006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:57.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.371+0000 7f1961131640 10 monclient: get_auth_request con 0x7f194006c590 auth_method 0 2024-03-27T21:41:57.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.371+0000 7f1951ffb640 20 client.5232.objecter dump_active .. 0 homeless 2024-03-27T21:41:57.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.371+0000 7f1951ffb640 10 client.5232.objecter ms_handle_connect 0x7f194006c590 2024-03-27T21:41:57.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.371+0000 7f1961131640 1 --2- 172.21.15.60:0/2409009421 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f194006c590 0x7f194006ea50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1958036000 tx=0x7f195803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:57.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.375+0000 7f1951ffb640 1 -- 172.21.15.60:0/2409009421 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1954055110 con 0x7f195c13ec50 2024-03-27T21:41:57.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.375+0000 7f1951ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:57.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.375+0000 7f1951ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:57.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.567+0000 7f1963395640 10 start_mon_command cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}] 2024-03-27T21:41:57.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.567+0000 7f1963395640 10 monclient: _send_command 2 [{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}] 2024-03-27T21:41:57.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.567+0000 7f1963395640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:41:57.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:57.567+0000 7f1963395640 1 -- 172.21.15.60:0/2409009421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} v 0) v1 -- 0x7f195c0a85d0 con 0x7f195c13ec50 2024-03-27T21:41:58.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.240+0000 7f1951ffb640 1 -- 172.21.15.60:0/2409009421 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]=0 set osd.5 primary-affinity to 1 (8655362) v86) v1 ==== 133+0+0 (secure 0 0 0) 0x7f19540180a0 con 0x7f195c13ec50 2024-03-27T21:41:58.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.240+0000 7f1951ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}] 2024-03-27T21:41:58.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.240+0000 7f1951ffb640 10 monclient: _finish_command 2 = system:0 set osd.5 primary-affinity to 1 (8655362) 2024-03-27T21:41:58.227 INFO:teuthology.orchestra.run.smithi060.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-27T21:41:58.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.243+0000 7f1963395640 10 librados: watch_flush enter 2024-03-27T21:41:58.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.243+0000 7f1963395640 10 librados: watch_flush exit 2024-03-27T21:41:58.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.243+0000 7f1963395640 20 client.5232.objecter shutdown clearing up homeless session... 2024-03-27T21:41:58.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.243+0000 7f1963395640 10 client.5232.objecter successfully canceled tick 2024-03-27T21:41:58.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.243+0000 7f1963395640 1 -- 172.21.15.60:0/2409009421 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f194006c590 msgr2=0x7f194006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:58.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.243+0000 7f1963395640 1 --2- 172.21.15.60:0/2409009421 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f194006c590 0x7f194006ea50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1958036000 tx=0x7f195803c040 comp rx=0 tx=0).stop 2024-03-27T21:41:58.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.243+0000 7f1963395640 10 monclient: shutdown 2024-03-27T21:41:58.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.243+0000 7f1963395640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:58.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.243+0000 7f1963395640 1 -- 172.21.15.60:0/2409009421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f195c13ec50 msgr2=0x7f195c13aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:58.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.243+0000 7f1963395640 1 --2- 172.21.15.60:0/2409009421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f195c13ec50 0x7f195c13aca0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f195400efc0 tx=0x7f1954009f90 comp rx=0 tx=0).stop 2024-03-27T21:41:58.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.243+0000 7f1963395640 1 -- 172.21.15.60:0/2409009421 shutdown_connections 2024-03-27T21:41:58.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.243+0000 7f1963395640 1 --2- 172.21.15.60:0/2409009421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f195c13ec50 0x7f195c13aca0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:58.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.243+0000 7f1963395640 1 --2- 172.21.15.60:0/2409009421 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f195c13e810 0x7f195c13a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:58.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.243+0000 7f1963395640 1 --2- 172.21.15.60:0/2409009421 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f194006c590 0x7f194006ea50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:58.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.243+0000 7f1963395640 1 --2- 172.21.15.60:0/2409009421 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f195c071a50 0x7f195c1d6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:58.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.243+0000 7f1963395640 1 -- 172.21.15.60:0/2409009421 >> 172.21.15.60:0/2409009421 conn(0x7f195c0b7300 msgr2=0x7f195c13d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:58.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.243+0000 7f1963395640 1 -- 172.21.15.60:0/2409009421 shutdown_connections 2024-03-27T21:41:58.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.243+0000 7f1963395640 1 -- 172.21.15.60:0/2409009421 wait complete. 2024-03-27T21:41:58.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.243+0000 7f1963395640 1 librados: shutdown 2024-03-27T21:41:58.243 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-27T21:41:58.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.363+0000 7fe4d8c1f640 10 monclient: get_monmap_and_config 2024-03-27T21:41:58.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.363+0000 7fe4d8c1f640 10 monclient: build_initial_monmap 2024-03-27T21:41:58.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.363+0000 7fe4d8c1f640 10 monclient: monmap: 2024-03-27T21:41:58.349 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:58.350 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:58.350 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:58.364178+0000 2024-03-27T21:41:58.350 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:58.364178+0000 2024-03-27T21:41:58.350 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:58.350 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:58.350 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:58.350 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:58.350 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:58.350 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:58.351 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:58.351 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:58.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.363+0000 7fe4d8c1f640 1 Processor -- start 2024-03-27T21:41:58.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.363+0000 7fe4d8c1f640 1 -- start start 2024-03-27T21:41:58.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.363+0000 7fe4d8c1f640 10 monclient: init 2024-03-27T21:41:58.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d8c1f640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:58.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d8c1f640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-03-27T21:41:58.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d8c1f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe4d40b40b0 0x7fe4d40a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:58.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d8c1f640 10 monclient(hunting): picked mon.noname-b con 0x7fe4d40b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:58.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d8c1f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe4d413b020 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:58.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d8c1f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe4d413b480 0x7fe4d4141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:58.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d8c1f640 10 monclient(hunting): picked mon.noname-c con 0x7fe4d413b480 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:58.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d8c1f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d40b3830 con 0x7fe4d413b020 2024-03-27T21:41:58.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d8c1f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:58.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d8c1f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4d413b910 con 0x7fe4d413b480 2024-03-27T21:41:58.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d8c1f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:58.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d8c1f640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d40b4490 con 0x7fe4d40b40b0 2024-03-27T21:41:58.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d8c1f640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:58.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d8c1f640 10 monclient(hunting): authenticate will time out at 959.585422s 2024-03-27T21:41:58.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d259c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe4d40b40b0 0x7fe4d40a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:58.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d1d9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe4d413b020 0x7fe4d41416f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47272/0 (socket says 172.21.15.60:47272) 2024-03-27T21:41:58.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d1d9b640 1 -- 172.21.15.60:0/267060325 learned_addr learned my addr 172.21.15.60:0/267060325 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:58.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d259c640 10 monclient(hunting): get_auth_request con 0x7fe4d40b40b0 auth_method 0 2024-03-27T21:41:58.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:58.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d259c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:58.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:58.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:58.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:58.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:58.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d159a640 1 -- 172.21.15.60:0/267060325 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943959305 0 0) 0x7fe4d40b3830 con 0x7fe4d413b020 2024-03-27T21:41:58.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d259c640 10 monclient(hunting): handle_auth_done global_id 4961 payload 274 2024-03-27T21:41:58.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d2d9d640 1 --2- 172.21.15.60:0/267060325 >> v2:172.21.15.191:3300/0 conn(0x7fe4d413b480 0x7fe4d4141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:58.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d259c640 1 -- 172.21.15.60:0/267060325 >> v2:172.21.15.191:3300/0 conn(0x7fe4d413b480 msgr2=0x7fe4d4141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:58.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d259c640 1 --2- 172.21.15.60:0/267060325 >> v2:172.21.15.191:3300/0 conn(0x7fe4d413b480 0x7fe4d4141e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:58.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d259c640 1 -- 172.21.15.60:0/267060325 >> v1:172.21.15.60:6789/0 conn(0x7fe4d413b020 legacy=0x7fe4d41416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:58.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d259c640 10 monclient: _finish_hunting 0 2024-03-27T21:41:58.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d259c640 1 monclient: found mon.noname-b 2024-03-27T21:41:58.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:58.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:58.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d259c640 1 -- 172.21.15.60:0/267060325 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d40aa520 con 0x7fe4d40b40b0 2024-03-27T21:41:58.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d159a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:41:58.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d2d9d640 10 monclient: get_auth_request con 0x7fe4d413b480 auth_method 0 2024-03-27T21:41:58.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.364+0000 7fe4d2d9d640 1 --2- 172.21.15.60:0/267060325 >> v2:172.21.15.191:3300/0 conn(0x7fe4d413b480 0x7fe4d4141e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:41:58.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.365+0000 7fe4d259c640 1 --2- 172.21.15.60:0/267060325 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe4d40b40b0 0x7fe4d40a8b90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc0097a0 tx=0x7fe4cc02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f08a76006283be3 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:58.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.365+0000 7fe4d159a640 1 -- 172.21.15.60:0/267060325 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc031780 con 0x7fe4d40b40b0 2024-03-27T21:41:58.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.365+0000 7fe4d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:58.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.365+0000 7fe4d159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:58.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.365+0000 7fe4d159a640 10 monclient: dump: 2024-03-27T21:41:58.364 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:58.364 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:58.364 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:58.364 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:58.364 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:58.364 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:58.364 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:58.365 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:58.365 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:58.365 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:58.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.365+0000 7fe4d159a640 10 monclient: _finish_auth 0 2024-03-27T21:41:58.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.365+0000 7fe4d159a640 10 monclient: _check_auth_tickets 2024-03-27T21:41:58.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.365+0000 7fe4d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:58.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.365+0000 7fe4d159a640 1 -- 172.21.15.60:0/267060325 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4cc0318e0 con 0x7fe4d40b40b0 2024-03-27T21:41:58.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.365+0000 7fe4d8c1f640 5 monclient: authenticate success, global_id 4961 2024-03-27T21:41:58.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.365+0000 7fe4d8c1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:58.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.365+0000 7fe4d159a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:58.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.365+0000 7fe4d8c1f640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:58.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.365+0000 7fe4d159a640 1 -- 172.21.15.60:0/267060325 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc031c70 con 0x7fe4d40b40b0 2024-03-27T21:41:58.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.365+0000 7fe4d8c1f640 4 set_mon_vals no callback set 2024-03-27T21:41:58.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.365+0000 7fe4d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:58.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.365+0000 7fe4d159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:58.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.365+0000 7fe4d159a640 10 monclient: dump: 2024-03-27T21:41:58.366 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:58.367 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:58.367 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:58.367 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:58.367 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:58.367 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:58.367 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:58.367 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:58.367 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:58.367 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:58.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.366+0000 7fe4d8c1f640 10 monclient: shutdown 2024-03-27T21:41:58.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.366+0000 7fe4d8c1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:58.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.366+0000 7fe4d8c1f640 1 -- 172.21.15.60:0/267060325 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe4d40b40b0 msgr2=0x7fe4d40a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:58.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.366+0000 7fe4d8c1f640 1 --2- 172.21.15.60:0/267060325 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe4d40b40b0 0x7fe4d40a8b90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc0097a0 tx=0x7fe4cc02b820 comp rx=0 tx=0).stop 2024-03-27T21:41:58.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.366+0000 7fe4d8c1f640 1 -- 172.21.15.60:0/267060325 shutdown_connections 2024-03-27T21:41:58.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.366+0000 7fe4d8c1f640 1 --2- 172.21.15.60:0/267060325 >> v2:172.21.15.191:3300/0 conn(0x7fe4d413b480 0x7fe4d4141e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:58.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.366+0000 7fe4d8c1f640 1 --2- 172.21.15.60:0/267060325 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe4d40b40b0 0x7fe4d40a8b90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:58.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.366+0000 7fe4d8c1f640 1 -- 172.21.15.60:0/267060325 >> 172.21.15.60:0/267060325 conn(0x7fe4d40608e0 msgr2=0x7fe4d4060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:58.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.366+0000 7fe4d8c1f640 1 -- 172.21.15.60:0/267060325 shutdown_connections 2024-03-27T21:41:58.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.366+0000 7fe4d8c1f640 1 -- 172.21.15.60:0/267060325 wait complete. 2024-03-27T21:41:58.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.367+0000 7fe4d8c1f640 10 monclient: build_initial_monmap 2024-03-27T21:41:58.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.367+0000 7fe4d8c1f640 10 monclient: monmap: 2024-03-27T21:41:58.369 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:58.369 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:58.369 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:58.369 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:58.369 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:58.369 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:58.370 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:58.370 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:58.370 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:58.370 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:58.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.367+0000 7fe4d8c1f640 1 librados: starting msgr at 2024-03-27T21:41:58.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.367+0000 7fe4d8c1f640 1 librados: starting objecter 2024-03-27T21:41:58.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.367+0000 7fe4d8c1f640 1 Processor -- start 2024-03-27T21:41:58.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.367+0000 7fe4d8c1f640 1 -- start start 2024-03-27T21:41:58.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.367+0000 7fe4d8c1f640 1 librados: setting wanted keys 2024-03-27T21:41:58.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.367+0000 7fe4d8c1f640 1 librados: calling monclient init 2024-03-27T21:41:58.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.367+0000 7fe4d8c1f640 10 monclient: init 2024-03-27T21:41:58.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d8c1f640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:58.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d8c1f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:41:58.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d8c1f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe4d40b40b0 0x7fe4d413a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:58.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d8c1f640 10 monclient(hunting): picked mon.noname-c con 0x7fe4d40b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:58.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d8c1f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4d413b020 0x7fe4d41d93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:58.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d8c1f640 10 monclient(hunting): picked mon.noname-a con 0x7fe4d413b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:58.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d8c1f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4d413b480 0x7fe4d41d9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:58.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d8c1f640 10 monclient(hunting): picked mon.noname-b con 0x7fe4d413b480 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:58.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d8c1f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:58.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d8c1f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d40aaa40 con 0x7fe4d413b020 2024-03-27T21:41:58.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d8c1f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:58.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d8c1f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d413ce90 con 0x7fe4d413b480 2024-03-27T21:41:58.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d8c1f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:58.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d8c1f640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d413da60 con 0x7fe4d40b40b0 2024-03-27T21:41:58.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d8c1f640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:58.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d8c1f640 10 monclient(hunting): authenticate will time out at 959.589112s 2024-03-27T21:41:58.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d259c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe4d40b40b0 0x7fe4d413a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:58.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d1d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4d413b020 0x7fe4d41d93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:58.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d1d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4d413b020 0x7fe4d41d93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56018/0 (socket says 172.21.15.60:56018) 2024-03-27T21:41:58.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d259c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe4d40b40b0 0x7fe4d413a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:39002/0 (socket says 172.21.15.60:39002) 2024-03-27T21:41:58.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d1d9b640 1 -- 172.21.15.60:0/3019200186 learned_addr learned my addr 172.21.15.60:0/3019200186 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:58.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d1d9b640 10 monclient(hunting): get_auth_request con 0x7fe4d413b020 auth_method 0 2024-03-27T21:41:58.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:58.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:58.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:58.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d259c640 10 monclient(hunting): get_auth_request con 0x7fe4d40b40b0 auth_method 0 2024-03-27T21:41:58.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:58.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d259c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:58.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:58.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:58.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:58.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:58.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:58.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:58.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:58.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d259c640 10 monclient(hunting): handle_auth_done global_id 4964 payload 824 2024-03-27T21:41:58.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d2d9d640 1 --2- 172.21.15.60:0/3019200186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4d413b480 0x7fe4d41d9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:58.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d259c640 1 -- 172.21.15.60:0/3019200186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4d413b480 msgr2=0x7fe4d41d9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:58.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d259c640 1 --2- 172.21.15.60:0/3019200186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4d413b480 0x7fe4d41d9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:58.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d259c640 1 -- 172.21.15.60:0/3019200186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4d413b020 msgr2=0x7fe4d41d93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:58.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d259c640 1 --2- 172.21.15.60:0/3019200186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4d413b020 0x7fe4d41d93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:58.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d259c640 10 monclient: _finish_hunting 0 2024-03-27T21:41:58.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d259c640 1 monclient: found mon.noname-c 2024-03-27T21:41:58.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:58.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:58.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d259c640 1 -- 172.21.15.60:0/3019200186 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4cc009400 con 0x7fe4d40b40b0 2024-03-27T21:41:58.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d2d9d640 10 monclient: get_auth_request con 0x7fe4d413b480 auth_method 0 2024-03-27T21:41:58.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d1d9b640 1 --2- 172.21.15.60:0/3019200186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4d413b020 0x7fe4d41d93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:41:58.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d2d9d640 1 --2- 172.21.15.60:0/3019200186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4d413b480 0x7fe4d41d9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:41:58.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d259c640 1 --2- 172.21.15.60:0/3019200186 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe4d40b40b0 0x7fe4d413a9c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe4d40b8060 tx=0x7fe4cc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:58.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4c2ffd640 10 client.?.objecter ms_handle_connect 0x7fe4d40b40b0 2024-03-27T21:41:58.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:58.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4c2ffd640 1 -- 172.21.15.60:0/3019200186 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc0328e0 con 0x7fe4d40b40b0 2024-03-27T21:41:58.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:58.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4c2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:58.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4c2ffd640 10 monclient: dump: 2024-03-27T21:41:58.380 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:58.380 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:58.380 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:58.380 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:58.380 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:58.380 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:58.380 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:58.381 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:58.381 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:58.381 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:58.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4c2ffd640 10 monclient: _finish_auth 0 2024-03-27T21:41:58.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4c2ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:41:58.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:58.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d8c1f640 5 monclient: authenticate success, global_id 4964 2024-03-27T21:41:58.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4c2ffd640 1 -- 172.21.15.60:0/3019200186 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4cc032a40 con 0x7fe4d40b40b0 2024-03-27T21:41:58.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d8c1f640 10 monclient: _renew_subs 2024-03-27T21:41:58.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d8c1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:58.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4d8c1f640 1 -- 172.21.15.60:0/3019200186 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d40b08a0 con 0x7fe4d40b40b0 2024-03-27T21:41:58.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.368+0000 7fe4c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:58.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.369+0000 7fe4c2ffd640 1 -- 172.21.15.60:0/3019200186 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc032df0 con 0x7fe4d40b40b0 2024-03-27T21:41:58.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.369+0000 7fe4c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:58.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.369+0000 7fe4c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:58.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.369+0000 7fe4c2ffd640 10 monclient: dump: 2024-03-27T21:41:58.383 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:58.383 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:58.383 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:58.383 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:58.383 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:58.383 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:58.383 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:58.383 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:58.383 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:58.384 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:58.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.369+0000 7fe4d0d99640 4 set_mon_vals no callback set 2024-03-27T21:41:58.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.369+0000 7fe4d8c1f640 10 client.4964.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:58.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.369+0000 7fe4d8c1f640 10 monclient: _renew_subs 2024-03-27T21:41:58.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.369+0000 7fe4d8c1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:58.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.369+0000 7fe4d8c1f640 1 -- 172.21.15.60:0/3019200186 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d40aac90 con 0x7fe4d40b40b0 2024-03-27T21:41:58.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.369+0000 7fe4d8c1f640 1 librados: init done 2024-03-27T21:41:58.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.369+0000 7fe4d8c1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:58.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.369+0000 7fe4d8c1f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:58.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.369+0000 7fe4d8c1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:58.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.369+0000 7fe4d8c1f640 1 -- 172.21.15.60:0/3019200186 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d414bd00 con 0x7fe4d40b40b0 2024-03-27T21:41:58.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.370+0000 7fe4c2ffd640 1 -- 172.21.15.60:0/3019200186 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe4cc04e440 con 0x7fe4d40b40b0 2024-03-27T21:41:58.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.370+0000 7fe4c2ffd640 1 --2- 172.21.15.60:0/3019200186 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe4b406c590 0x7fe4b406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:58.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.370+0000 7fe4d1d9b640 1 --2- 172.21.15.60:0/3019200186 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe4b406c590 0x7fe4b406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:58.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.370+0000 7fe4c2ffd640 1 -- 172.21.15.60:0/3019200186 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fe4cc0ab250 con 0x7fe4d40b40b0 2024-03-27T21:41:58.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.370+0000 7fe4c2ffd640 10 client.4964.objecter ms_dispatch 0x7fe4d4000b90 osd_map(86..86 src has 1..86) v4 2024-03-27T21:41:58.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.370+0000 7fe4c2ffd640 3 client.4964.objecter handle_osd_map got epochs [86,86] > 0 2024-03-27T21:41:58.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.370+0000 7fe4c2ffd640 3 client.4964.objecter handle_osd_map decoding full epoch 86 2024-03-27T21:41:58.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.370+0000 7fe4d1d9b640 10 monclient: get_auth_request con 0x7fe4b406c590 auth_method 0 2024-03-27T21:41:58.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.370+0000 7fe4c2ffd640 20 client.4964.objecter dump_active .. 0 homeless 2024-03-27T21:41:58.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.371+0000 7fe4c2ffd640 10 client.4964.objecter ms_handle_connect 0x7fe4b406c590 2024-03-27T21:41:58.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.371+0000 7fe4d1d9b640 1 --2- 172.21.15.60:0/3019200186 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe4b406c590 0x7fe4b406ea50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe4d413ed60 tx=0x7fe4c8007e10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:58.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.374+0000 7fe4c2ffd640 1 -- 172.21.15.60:0/3019200186 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe4d414bd00 con 0x7fe4d40b40b0 2024-03-27T21:41:58.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.375+0000 7fe4c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:58.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.375+0000 7fe4c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:58.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.568+0000 7fe4d8c1f640 10 start_mon_command cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}] 2024-03-27T21:41:58.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.568+0000 7fe4d8c1f640 10 monclient: _send_command 2 [{"prefix": "osd reweight", "id": 6, "weight": 1.0}] 2024-03-27T21:41:58.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.568+0000 7fe4d8c1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:58.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:58.568+0000 7fe4d8c1f640 1 -- 172.21.15.60:0/3019200186 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7fe4d413e630 con 0x7fe4d40b40b0 2024-03-27T21:41:59.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.242+0000 7fe4c2ffd640 1 -- 172.21.15.60:0/3019200186 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v87) v1 ==== 113+0+0 (secure 0 0 0) 0x7fe4d413e630 con 0x7fe4d40b40b0 2024-03-27T21:41:59.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.243+0000 7fe4c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd reweight", "id": 6, "weight": 1.0}] 2024-03-27T21:41:59.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.243+0000 7fe4c2ffd640 10 monclient: _finish_command 2 = system:0 reweighted osd.6 to 1 (10000) 2024-03-27T21:41:59.229 INFO:teuthology.orchestra.run.smithi060.stderr:reweighted osd.6 to 1 (10000) 2024-03-27T21:41:59.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.245+0000 7fe4d8c1f640 10 librados: watch_flush enter 2024-03-27T21:41:59.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.245+0000 7fe4d8c1f640 10 librados: watch_flush exit 2024-03-27T21:41:59.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.245+0000 7fe4d8c1f640 20 client.4964.objecter shutdown clearing up homeless session... 2024-03-27T21:41:59.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.245+0000 7fe4d8c1f640 10 client.4964.objecter successfully canceled tick 2024-03-27T21:41:59.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.245+0000 7fe4d8c1f640 1 -- 172.21.15.60:0/3019200186 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe4b406c590 msgr2=0x7fe4b406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:59.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.245+0000 7fe4d8c1f640 1 --2- 172.21.15.60:0/3019200186 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe4b406c590 0x7fe4b406ea50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe4d413ed60 tx=0x7fe4c8007e10 comp rx=0 tx=0).stop 2024-03-27T21:41:59.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.245+0000 7fe4d8c1f640 10 monclient: shutdown 2024-03-27T21:41:59.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.245+0000 7fe4d8c1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:59.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.245+0000 7fe4d8c1f640 1 -- 172.21.15.60:0/3019200186 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe4d40b40b0 msgr2=0x7fe4d413a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:59.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.245+0000 7fe4d8c1f640 1 --2- 172.21.15.60:0/3019200186 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe4d40b40b0 0x7fe4d413a9c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe4d40b8060 tx=0x7fe4cc0046c0 comp rx=0 tx=0).stop 2024-03-27T21:41:59.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.246+0000 7fe4d8c1f640 1 -- 172.21.15.60:0/3019200186 shutdown_connections 2024-03-27T21:41:59.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.246+0000 7fe4d8c1f640 1 --2- 172.21.15.60:0/3019200186 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe4b406c590 0x7fe4b406ea50 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:59.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.246+0000 7fe4d8c1f640 1 --2- 172.21.15.60:0/3019200186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4d413b480 0x7fe4d41d9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:59.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.246+0000 7fe4d8c1f640 1 --2- 172.21.15.60:0/3019200186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4d413b020 0x7fe4d41d93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:59.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.246+0000 7fe4d8c1f640 1 --2- 172.21.15.60:0/3019200186 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe4d40b40b0 0x7fe4d413a9c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:59.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.246+0000 7fe4d8c1f640 1 -- 172.21.15.60:0/3019200186 >> 172.21.15.60:0/3019200186 conn(0x7fe4d40608e0 msgr2=0x7fe4d41416f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:59.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.246+0000 7fe4d8c1f640 1 -- 172.21.15.60:0/3019200186 shutdown_connections 2024-03-27T21:41:59.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.246+0000 7fe4d8c1f640 1 -- 172.21.15.60:0/3019200186 wait complete. 2024-03-27T21:41:59.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.246+0000 7fe4d8c1f640 1 librados: shutdown 2024-03-27T21:41:59.247 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-27T21:41:59.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.368+0000 7f2a4af2e640 10 monclient: get_monmap_and_config 2024-03-27T21:41:59.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.368+0000 7f2a4af2e640 10 monclient: build_initial_monmap 2024-03-27T21:41:59.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.368+0000 7f2a4af2e640 10 monclient: monmap: 2024-03-27T21:41:59.354 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:59.355 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:59.355 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:41:59.369151+0000 2024-03-27T21:41:59.355 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:41:59.369151+0000 2024-03-27T21:41:59.355 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:59.355 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:59.355 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:41:59.355 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:41:59.355 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:41:59.355 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:41:59.356 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:41:59.356 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:59.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.368+0000 7f2a4af2e640 1 Processor -- start 2024-03-27T21:41:59.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.368+0000 7f2a4af2e640 1 -- start start 2024-03-27T21:41:59.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.368+0000 7f2a4af2e640 10 monclient: init 2024-03-27T21:41:59.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a4af2e640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:59.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a4af2e640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-03-27T21:41:59.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a4af2e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a4413e9c0 addr v1:172.21.15.60:6789/0 2024-03-27T21:41:59.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a4af2e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2a4406f820 0x7f2a440712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:59.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a4af2e640 10 monclient(hunting): picked mon.noname-c con 0x7f2a4406f820 addr v2:172.21.15.191:3300/0 2024-03-27T21:41:59.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a4af2e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2a44071800 0x7f2a44148930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:59.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a4af2e640 10 monclient(hunting): picked mon.noname-b con 0x7f2a44071800 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:59.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a4af2e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a4406ed40 con 0x7f2a4413e9c0 2024-03-27T21:41:59.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a4af2e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:59.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a4af2e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a44071c60 con 0x7f2a4406f820 2024-03-27T21:41:59.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a4af2e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:59.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a4af2e640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a440708f0 con 0x7f2a44071800 2024-03-27T21:41:59.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a4af2e640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:59.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a4af2e640 10 monclient(hunting): authenticate will time out at 960.590478s 2024-03-27T21:41:59.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a494cb640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2a44071800 0x7f2a44148930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:59.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a48cca640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2a4413e9c0 0x7f2a440ad6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47288/0 (socket says 172.21.15.60:47288) 2024-03-27T21:41:59.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a48cca640 1 -- 172.21.15.60:0/4163149369 learned_addr learned my addr 172.21.15.60:0/4163149369 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:41:59.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a494cb640 10 monclient(hunting): get_auth_request con 0x7f2a44071800 auth_method 0 2024-03-27T21:41:59.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a494cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:59.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a494cb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:59.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a494cb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:59.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a494cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:59.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a494cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:59.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a494cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:59.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a43fff640 1 --2- 172.21.15.60:0/4163149369 >> v2:172.21.15.191:3300/0 conn(0x7f2a4406f820 0x7f2a440712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:59.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a43fff640 10 monclient(hunting): get_auth_request con 0x7f2a4406f820 auth_method 0 2024-03-27T21:41:59.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:59.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a43fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:59.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:59.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a437fe640 1 -- 172.21.15.60:0/4163149369 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025761793 0 0) 0x7f2a4406ed40 con 0x7f2a4413e9c0 2024-03-27T21:41:59.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.369+0000 7f2a494cb640 10 monclient(hunting): handle_auth_done global_id 4967 payload 274 2024-03-27T21:41:59.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a494cb640 1 -- 172.21.15.60:0/4163149369 >> v2:172.21.15.191:3300/0 conn(0x7f2a4406f820 msgr2=0x7f2a440712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:41:59.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a494cb640 1 --2- 172.21.15.60:0/4163149369 >> v2:172.21.15.191:3300/0 conn(0x7f2a4406f820 0x7f2a440712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:59.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a494cb640 1 -- 172.21.15.60:0/4163149369 >> v1:172.21.15.60:6789/0 conn(0x7f2a4413e9c0 legacy=0x7f2a440ad6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:59.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a494cb640 10 monclient: _finish_hunting 0 2024-03-27T21:41:59.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a494cb640 1 monclient: found mon.noname-b 2024-03-27T21:41:59.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a494cb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:59.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a494cb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:41:59.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a494cb640 1 -- 172.21.15.60:0/4163149369 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a44070b60 con 0x7f2a44071800 2024-03-27T21:41:59.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a437fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:41:59.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a43fff640 1 --2- 172.21.15.60:0/4163149369 >> v2:172.21.15.191:3300/0 conn(0x7f2a4406f820 0x7f2a440712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:41:59.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a494cb640 1 --2- 172.21.15.60:0/4163149369 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2a44071800 0x7f2a44148930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c00bfa0 tx=0x7f2a3c02e010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f54b2e6c60a4a57 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:59.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a437fe640 1 -- 172.21.15.60:0/4163149369 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c02e9b0 con 0x7f2a44071800 2024-03-27T21:41:59.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:59.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a437fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:41:59.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a437fe640 10 monclient: dump: 2024-03-27T21:41:59.370 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:59.370 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:59.370 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:59.370 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:59.370 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:59.370 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:59.370 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:59.371 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:59.371 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:59.371 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:59.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a437fe640 10 monclient: _finish_auth 0 2024-03-27T21:41:59.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a437fe640 10 monclient: _check_auth_tickets 2024-03-27T21:41:59.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:59.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a437fe640 1 -- 172.21.15.60:0/4163149369 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3c02eb10 con 0x7f2a44071800 2024-03-27T21:41:59.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a4af2e640 5 monclient: authenticate success, global_id 4967 2024-03-27T21:41:59.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a4af2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:41:59.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:59.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a4af2e640 10 monclient: get_monmap_and_config success 2024-03-27T21:41:59.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a437fe640 1 -- 172.21.15.60:0/4163149369 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c02eea0 con 0x7f2a44071800 2024-03-27T21:41:59.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:59.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a4af2e640 4 set_mon_vals no callback set 2024-03-27T21:41:59.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:59.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.370+0000 7f2a437fe640 10 monclient: dump: 2024-03-27T21:41:59.372 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:59.372 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:59.373 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:59.373 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:59.373 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:59.373 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:59.373 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:59.373 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:59.373 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:59.373 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:59.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.371+0000 7f2a4af2e640 10 monclient: shutdown 2024-03-27T21:41:59.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.371+0000 7f2a4af2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:41:59.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.371+0000 7f2a4af2e640 1 -- 172.21.15.60:0/4163149369 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2a44071800 msgr2=0x7f2a44148930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:59.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.371+0000 7f2a4af2e640 1 --2- 172.21.15.60:0/4163149369 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2a44071800 0x7f2a44148930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c00bfa0 tx=0x7f2a3c02e010 comp rx=0 tx=0).stop 2024-03-27T21:41:59.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.371+0000 7f2a4af2e640 1 -- 172.21.15.60:0/4163149369 shutdown_connections 2024-03-27T21:41:59.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.371+0000 7f2a4af2e640 1 --2- 172.21.15.60:0/4163149369 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2a44071800 0x7f2a44148930 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:59.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.371+0000 7f2a4af2e640 1 --2- 172.21.15.60:0/4163149369 >> v2:172.21.15.191:3300/0 conn(0x7f2a4406f820 0x7f2a440712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:59.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.371+0000 7f2a4af2e640 1 -- 172.21.15.60:0/4163149369 >> 172.21.15.60:0/4163149369 conn(0x7f2a440b2ea0 msgr2=0x7f2a440b32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:41:59.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.371+0000 7f2a4af2e640 1 -- 172.21.15.60:0/4163149369 shutdown_connections 2024-03-27T21:41:59.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.371+0000 7f2a4af2e640 1 -- 172.21.15.60:0/4163149369 wait complete. 2024-03-27T21:41:59.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.372+0000 7f2a4af2e640 10 monclient: build_initial_monmap 2024-03-27T21:41:59.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.372+0000 7f2a4af2e640 10 monclient: monmap: 2024-03-27T21:41:59.375 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:41:59.375 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:41:59.375 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:41:59.375 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:41:59.375 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:41:59.375 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:41:59.376 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:41:59.376 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:41:59.376 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:41:59.376 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:59.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.372+0000 7f2a4af2e640 1 librados: starting msgr at 2024-03-27T21:41:59.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.372+0000 7f2a4af2e640 1 librados: starting objecter 2024-03-27T21:41:59.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.372+0000 7f2a4af2e640 1 Processor -- start 2024-03-27T21:41:59.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.372+0000 7f2a4af2e640 1 -- start start 2024-03-27T21:41:59.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.372+0000 7f2a4af2e640 1 librados: setting wanted keys 2024-03-27T21:41:59.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.372+0000 7f2a4af2e640 1 librados: calling monclient init 2024-03-27T21:41:59.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.372+0000 7f2a4af2e640 10 monclient: init 2024-03-27T21:41:59.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a4af2e640 10 monclient: _reopen_session rank -1 2024-03-27T21:41:59.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a4af2e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:41:59.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a4af2e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2a4406f820 0x7f2a440a8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:59.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a4af2e640 10 monclient(hunting): picked mon.noname-c con 0x7f2a4406f820 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:41:59.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a4af2e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a44071800 0x7f2a441d56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:59.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a4af2e640 10 monclient(hunting): picked mon.noname-b con 0x7f2a44071800 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:41:59.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a4af2e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a4413e9c0 0x7f2a441d5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:59.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a4af2e640 10 monclient(hunting): picked mon.noname-a con 0x7f2a4413e9c0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:41:59.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a4af2e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:59.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a4af2e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a441d6150 con 0x7f2a4413e9c0 2024-03-27T21:41:59.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a4af2e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:59.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a4af2e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a441d5240 con 0x7f2a44071800 2024-03-27T21:41:59.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a4af2e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:41:59.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a4af2e640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a441403f0 con 0x7f2a4406f820 2024-03-27T21:41:59.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a4af2e640 10 monclient(hunting): _renew_subs 2024-03-27T21:41:59.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a4af2e640 10 monclient(hunting): authenticate will time out at 960.594305s 2024-03-27T21:41:59.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a48cca640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2a4406f820 0x7f2a440a8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:59.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a494cb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a4413e9c0 0x7f2a441d5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:59.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a494cb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a4413e9c0 0x7f2a441d5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56026/0 (socket says 172.21.15.60:56026) 2024-03-27T21:41:59.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a48cca640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2a4406f820 0x7f2a440a8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:39016/0 (socket says 172.21.15.60:39016) 2024-03-27T21:41:59.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a494cb640 1 -- 172.21.15.60:0/519756345 learned_addr learned my addr 172.21.15.60:0/519756345 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:41:59.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a48cca640 10 monclient(hunting): get_auth_request con 0x7f2a4406f820 auth_method 0 2024-03-27T21:41:59.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a48cca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:59.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a48cca640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:59.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a48cca640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:59.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a494cb640 10 monclient(hunting): get_auth_request con 0x7f2a4413e9c0 auth_method 0 2024-03-27T21:41:59.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a494cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:59.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a494cb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:59.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a494cb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:59.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a48cca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:59.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a48cca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:59.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a48cca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:59.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a494cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:41:59.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a494cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:41:59.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a494cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:41:59.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a43fff640 1 --2- 172.21.15.60:0/519756345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a44071800 0x7f2a441d56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:59.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a43fff640 10 monclient(hunting): get_auth_request con 0x7f2a44071800 auth_method 0 2024-03-27T21:41:59.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:41:59.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a43fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:41:59.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:41:59.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a48cca640 10 monclient(hunting): handle_auth_done global_id 4970 payload 824 2024-03-27T21:41:59.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a48cca640 1 -- 172.21.15.60:0/519756345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a44071800 msgr2=0x7f2a441d56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:59.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a48cca640 1 --2- 172.21.15.60:0/519756345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a44071800 0x7f2a441d56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:59.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a48cca640 1 -- 172.21.15.60:0/519756345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a4413e9c0 msgr2=0x7f2a441d5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:41:59.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a48cca640 1 --2- 172.21.15.60:0/519756345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a4413e9c0 0x7f2a441d5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:41:59.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a48cca640 10 monclient: _finish_hunting 0 2024-03-27T21:41:59.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a48cca640 1 monclient: found mon.noname-c 2024-03-27T21:41:59.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a48cca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:41:59.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a48cca640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:41:59.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a48cca640 1 -- 172.21.15.60:0/519756345 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a3c00bc80 con 0x7f2a4406f820 2024-03-27T21:41:59.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a43fff640 1 --2- 172.21.15.60:0/519756345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a44071800 0x7f2a441d56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:41:59.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a48cca640 1 --2- 172.21.15.60:0/519756345 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2a4406f820 0x7f2a440a8ed0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2a3800c440 tx=0x7f2a3800c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:59.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a417fa640 10 client.?.objecter ms_handle_connect 0x7f2a4406f820 2024-03-27T21:41:59.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.373+0000 7f2a417fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:41:59.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a417fa640 1 -- 172.21.15.60:0/519756345 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a380074e0 con 0x7f2a4406f820 2024-03-27T21:41:59.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:59.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a417fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:41:59.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a417fa640 10 monclient: dump: 2024-03-27T21:41:59.384 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:59.384 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:59.384 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:59.384 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:59.384 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:59.384 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:59.384 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:59.384 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:59.385 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:59.385 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:59.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a417fa640 10 monclient: _finish_auth 0 2024-03-27T21:41:59.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a417fa640 10 monclient: _check_auth_tickets 2024-03-27T21:41:59.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:41:59.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a417fa640 1 -- 172.21.15.60:0/519756345 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a38007660 con 0x7f2a4406f820 2024-03-27T21:41:59.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a4af2e640 5 monclient: authenticate success, global_id 4970 2024-03-27T21:41:59.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a4af2e640 10 monclient: _renew_subs 2024-03-27T21:41:59.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a4af2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:59.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a4af2e640 1 -- 172.21.15.60:0/519756345 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a440aac00 con 0x7f2a4406f820 2024-03-27T21:41:59.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a417fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:41:59.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a417fa640 1 -- 172.21.15.60:0/519756345 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a38004a20 con 0x7f2a4406f820 2024-03-27T21:41:59.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:41:59.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a417fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:41:59.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a4af2e640 10 client.4970.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:41:59.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a42ffd640 4 set_mon_vals no callback set 2024-03-27T21:41:59.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a417fa640 10 monclient: dump: 2024-03-27T21:41:59.386 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:41:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:41:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:41:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:41:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:41:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:41:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:41:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:41:59.387 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:41:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a4af2e640 10 monclient: _renew_subs 2024-03-27T21:41:59.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a4af2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:59.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a4af2e640 1 -- 172.21.15.60:0/519756345 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a440a9410 con 0x7f2a4406f820 2024-03-27T21:41:59.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.374+0000 7f2a4af2e640 1 librados: init done 2024-03-27T21:41:59.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.375+0000 7f2a4af2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:59.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.375+0000 7f2a4af2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:59.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.375+0000 7f2a4af2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:59.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.375+0000 7f2a4af2e640 1 -- 172.21.15.60:0/519756345 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a44070090 con 0x7f2a4406f820 2024-03-27T21:41:59.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.375+0000 7f2a417fa640 1 -- 172.21.15.60:0/519756345 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2a38004b80 con 0x7f2a4406f820 2024-03-27T21:41:59.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.376+0000 7f2a417fa640 1 --2- 172.21.15.60:0/519756345 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2a2806c590 0x7f2a2806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:41:59.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.376+0000 7f2a417fa640 1 -- 172.21.15.60:0/519756345 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f2a38088e20 con 0x7f2a4406f820 2024-03-27T21:41:59.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.376+0000 7f2a417fa640 10 client.4970.objecter ms_dispatch 0x7f2a44000b90 osd_map(87..87 src has 1..87) v4 2024-03-27T21:41:59.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.376+0000 7f2a417fa640 3 client.4970.objecter handle_osd_map got epochs [87,87] > 0 2024-03-27T21:41:59.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.376+0000 7f2a417fa640 3 client.4970.objecter handle_osd_map decoding full epoch 87 2024-03-27T21:41:59.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.376+0000 7f2a43fff640 1 --2- 172.21.15.60:0/519756345 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2a2806c590 0x7f2a2806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:41:59.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.376+0000 7f2a43fff640 10 monclient: get_auth_request con 0x7f2a2806c590 auth_method 0 2024-03-27T21:41:59.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.376+0000 7f2a417fa640 20 client.4970.objecter dump_active .. 0 homeless 2024-03-27T21:41:59.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.376+0000 7f2a417fa640 10 client.4970.objecter ms_handle_connect 0x7f2a2806c590 2024-03-27T21:41:59.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.376+0000 7f2a43fff640 1 --2- 172.21.15.60:0/519756345 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2a2806c590 0x7f2a2806ea50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2a4413b290 tx=0x7f2a30008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:41:59.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.380+0000 7f2a417fa640 1 -- 172.21.15.60:0/519756345 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2a38054320 con 0x7f2a4406f820 2024-03-27T21:41:59.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.380+0000 7f2a417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:41:59.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.380+0000 7f2a417fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:41:59.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.573+0000 7f2a4af2e640 10 start_mon_command cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}] 2024-03-27T21:41:59.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.573+0000 7f2a4af2e640 10 monclient: _send_command 2 [{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}] 2024-03-27T21:41:59.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.573+0000 7f2a4af2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:41:59.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:41:59.573+0000 7f2a4af2e640 1 -- 172.21.15.60:0/519756345 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} v 0) v1 -- 0x7f2a44069470 con 0x7f2a4406f820 2024-03-27T21:42:00.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.250+0000 7f2a417fa640 1 -- 172.21.15.60:0/519756345 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]=0 set osd.6 primary-affinity to 1 (8655362) v88) v1 ==== 133+0+0 (secure 0 0 0) 0x7f2a38057940 con 0x7f2a4406f820 2024-03-27T21:42:00.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.250+0000 7f2a417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}] 2024-03-27T21:42:00.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.250+0000 7f2a417fa640 10 monclient: _finish_command 2 = system:0 set osd.6 primary-affinity to 1 (8655362) 2024-03-27T21:42:00.237 INFO:teuthology.orchestra.run.smithi060.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-27T21:42:00.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.252+0000 7f2a4af2e640 10 librados: watch_flush enter 2024-03-27T21:42:00.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.253+0000 7f2a4af2e640 10 librados: watch_flush exit 2024-03-27T21:42:00.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.253+0000 7f2a4af2e640 20 client.4970.objecter shutdown clearing up homeless session... 2024-03-27T21:42:00.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.253+0000 7f2a4af2e640 10 client.4970.objecter successfully canceled tick 2024-03-27T21:42:00.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.253+0000 7f2a4af2e640 1 -- 172.21.15.60:0/519756345 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2a2806c590 msgr2=0x7f2a2806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:00.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.253+0000 7f2a4af2e640 1 --2- 172.21.15.60:0/519756345 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2a2806c590 0x7f2a2806ea50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2a4413b290 tx=0x7f2a30008040 comp rx=0 tx=0).stop 2024-03-27T21:42:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.253+0000 7f2a4af2e640 10 monclient: shutdown 2024-03-27T21:42:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.253+0000 7f2a4af2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.253+0000 7f2a4af2e640 1 -- 172.21.15.60:0/519756345 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2a4406f820 msgr2=0x7f2a440a8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.253+0000 7f2a4af2e640 1 --2- 172.21.15.60:0/519756345 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2a4406f820 0x7f2a440a8ed0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2a3800c440 tx=0x7f2a3800c900 comp rx=0 tx=0).stop 2024-03-27T21:42:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.253+0000 7f2a4af2e640 1 -- 172.21.15.60:0/519756345 shutdown_connections 2024-03-27T21:42:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.253+0000 7f2a4af2e640 1 --2- 172.21.15.60:0/519756345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a4413e9c0 0x7f2a441d5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.253+0000 7f2a4af2e640 1 --2- 172.21.15.60:0/519756345 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2a2806c590 0x7f2a2806ea50 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.253+0000 7f2a4af2e640 1 --2- 172.21.15.60:0/519756345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a44071800 0x7f2a441d56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:00.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.253+0000 7f2a4af2e640 1 --2- 172.21.15.60:0/519756345 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2a4406f820 0x7f2a440a8ed0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:00.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.253+0000 7f2a4af2e640 1 -- 172.21.15.60:0/519756345 >> 172.21.15.60:0/519756345 conn(0x7f2a440b2ea0 msgr2=0x7f2a440b08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:00.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.253+0000 7f2a4af2e640 1 -- 172.21.15.60:0/519756345 shutdown_connections 2024-03-27T21:42:00.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.253+0000 7f2a4af2e640 1 -- 172.21.15.60:0/519756345 wait complete. 2024-03-27T21:42:00.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.253+0000 7f2a4af2e640 1 librados: shutdown 2024-03-27T21:42:00.254 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-27T21:42:00.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.375+0000 7f75fa281640 10 monclient: get_monmap_and_config 2024-03-27T21:42:00.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.375+0000 7f75fa281640 10 monclient: build_initial_monmap 2024-03-27T21:42:00.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.375+0000 7f75fa281640 10 monclient: monmap: 2024-03-27T21:42:00.362 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:00.362 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:00.362 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:00.376392+0000 2024-03-27T21:42:00.362 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:00.376392+0000 2024-03-27T21:42:00.362 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:00.362 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:00.362 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:00.362 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:00.363 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:00.363 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:00.363 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:00.363 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:00.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75fa281640 1 Processor -- start 2024-03-27T21:42:00.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75fa281640 1 -- start start 2024-03-27T21:42:00.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75fa281640 10 monclient: init 2024-03-27T21:42:00.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75fa281640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:00.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75fa281640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-03-27T21:42:00.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75fa281640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f75f4071a50 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:00.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75fa281640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75f413ef20 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:00.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75fa281640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f75f40a87e0 0x7f75f4141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:00.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75fa281640 10 monclient(hunting): picked mon.noname-b con 0x7f75f40a87e0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:00.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75fa281640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f40a8e90 con 0x7f75f413ef20 2024-03-27T21:42:00.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75fa281640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f40aa040 con 0x7f75f4071a50 2024-03-27T21:42:00.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75fa281640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:00.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75fa281640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f4071e30 con 0x7f75f40a87e0 2024-03-27T21:42:00.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75fa281640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:00.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75fa281640 10 monclient(hunting): authenticate will time out at 961.597666s 2024-03-27T21:42:00.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75f37fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f75f413ef20 0x7f75f4141630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47292/0 (socket says 172.21.15.60:47292) 2024-03-27T21:42:00.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75f37fe640 1 -- 172.21.15.60:0/2482542910 learned_addr learned my addr 172.21.15.60:0/2482542910 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:00.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75f881e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f75f40a87e0 0x7f75f4141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:00.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75f881e640 10 monclient(hunting): get_auth_request con 0x7f75f40a87e0 auth_method 0 2024-03-27T21:42:00.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75f881e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:00.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75f881e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:00.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75f881e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:00.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75f881e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:00.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75f881e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:00.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.376+0000 7f75f881e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:00.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.377+0000 7f75f881e640 10 monclient(hunting): handle_auth_done global_id 4973 payload 274 2024-03-27T21:42:00.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.377+0000 7f75f881e640 1 -- 172.21.15.60:0/2482542910 >> v1:172.21.15.191:6789/0 conn(0x7f75f4071a50 legacy=0x7f75f413e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.377+0000 7f75f881e640 1 -- 172.21.15.60:0/2482542910 >> v1:172.21.15.60:6789/0 conn(0x7f75f413ef20 legacy=0x7f75f4141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.377+0000 7f75f881e640 10 monclient: _finish_hunting 0 2024-03-27T21:42:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.377+0000 7f75f881e640 1 monclient: found mon.noname-b 2024-03-27T21:42:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.377+0000 7f75f881e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.377+0000 7f75f881e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:42:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.377+0000 7f75f2ffd640 1 -- 172.21.15.60:0/2482542910 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863235733 0 0) 0x7f75f40a8e90 con 0x7f75f413ef20 2024-03-27T21:42:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.377+0000 7f75f881e640 1 -- 172.21.15.60:0/2482542910 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f4142280 con 0x7f75f40a87e0 2024-03-27T21:42:00.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.377+0000 7f75f2ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:42:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.377+0000 7f75f881e640 1 --2- 172.21.15.60:0/2482542910 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f75f40a87e0 0x7f75f4141d40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009dc0 tx=0x7f75ec02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1099c5917cc861b server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.377+0000 7f75f2ffd640 1 -- 172.21.15.60:0/2482542910 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec02cc40 con 0x7f75f40a87e0 2024-03-27T21:42:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.377+0000 7f75f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.377+0000 7f75f2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75f2ffd640 10 monclient: dump: 2024-03-27T21:42:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:00.376 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:00.376 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:00.376 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:00.376 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:00.376 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:00.376 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:00.376 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:00.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75f2ffd640 10 monclient: _finish_auth 0 2024-03-27T21:42:00.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75f2ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:00.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:00.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75f2ffd640 1 -- 172.21.15.60:0/2482542910 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ec02cda0 con 0x7f75f40a87e0 2024-03-27T21:42:00.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75fa281640 5 monclient: authenticate success, global_id 4973 2024-03-27T21:42:00.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75fa281640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:00.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:00.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75fa281640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:00.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75f2ffd640 1 -- 172.21.15.60:0/2482542910 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec034660 con 0x7f75f40a87e0 2024-03-27T21:42:00.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:00.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75fa281640 4 set_mon_vals no callback set 2024-03-27T21:42:00.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:00.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75f2ffd640 10 monclient: dump: 2024-03-27T21:42:00.378 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:00.378 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:00.378 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:00.378 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:00.378 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:00.379 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:00.379 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:00.379 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:00.379 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:00.379 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:00.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75fa281640 10 monclient: shutdown 2024-03-27T21:42:00.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75fa281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:00.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75fa281640 1 -- 172.21.15.60:0/2482542910 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f75f40a87e0 msgr2=0x7f75f4141d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:00.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75fa281640 1 --2- 172.21.15.60:0/2482542910 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f75f40a87e0 0x7f75f4141d40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009dc0 tx=0x7f75ec02be90 comp rx=0 tx=0).stop 2024-03-27T21:42:00.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75fa281640 1 -- 172.21.15.60:0/2482542910 shutdown_connections 2024-03-27T21:42:00.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75fa281640 1 --2- 172.21.15.60:0/2482542910 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f75f40a87e0 0x7f75f4141d40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:00.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75fa281640 1 -- 172.21.15.60:0/2482542910 >> 172.21.15.60:0/2482542910 conn(0x7f75f40b7300 msgr2=0x7f75f40b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:00.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75fa281640 1 -- 172.21.15.60:0/2482542910 shutdown_connections 2024-03-27T21:42:00.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.378+0000 7f75fa281640 1 -- 172.21.15.60:0/2482542910 wait complete. 2024-03-27T21:42:00.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.379+0000 7f75fa281640 10 monclient: build_initial_monmap 2024-03-27T21:42:00.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.379+0000 7f75fa281640 10 monclient: monmap: 2024-03-27T21:42:00.380 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:00.381 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:00.381 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:00.381 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:00.381 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:00.381 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:00.381 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:00.381 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:00.381 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:00.381 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:00.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.379+0000 7f75fa281640 1 librados: starting msgr at 2024-03-27T21:42:00.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.379+0000 7f75fa281640 1 librados: starting objecter 2024-03-27T21:42:00.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.379+0000 7f75fa281640 1 Processor -- start 2024-03-27T21:42:00.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.379+0000 7f75fa281640 1 -- start start 2024-03-27T21:42:00.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.379+0000 7f75fa281640 1 librados: setting wanted keys 2024-03-27T21:42:00.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.379+0000 7f75fa281640 1 librados: calling monclient init 2024-03-27T21:42:00.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.379+0000 7f75fa281640 10 monclient: init 2024-03-27T21:42:00.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75fa281640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:00.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75fa281640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:42:00.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75fa281640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75f4071a50 0x7f75f4138e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:00.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75fa281640 10 monclient(hunting): picked mon.noname-b con 0x7f75f4071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:00.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75fa281640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f75f40a87e0 0x7f75f41d6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:00.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75fa281640 10 monclient(hunting): picked mon.noname-c con 0x7f75f40a87e0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:00.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75fa281640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f413ef20 0x7f75f41d6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:00.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75fa281640 10 monclient(hunting): picked mon.noname-a con 0x7f75f413ef20 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:00.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75fa281640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:00.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75fa281640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f41d6fc0 con 0x7f75f413ef20 2024-03-27T21:42:00.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75fa281640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:00.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75fa281640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f414cc00 con 0x7f75f4071a50 2024-03-27T21:42:00.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75fa281640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:00.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75fa281640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f4138420 con 0x7f75f40a87e0 2024-03-27T21:42:00.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75fa281640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:00.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75fa281640 10 monclient(hunting): authenticate will time out at 961.601152s 2024-03-27T21:42:00.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f37fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f75f40a87e0 0x7f75f41d6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:00.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f881e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f413ef20 0x7f75f41d6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:00.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f37fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f75f40a87e0 0x7f75f41d6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:39034/0 (socket says 172.21.15.60:39034) 2024-03-27T21:42:00.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f37fe640 1 -- 172.21.15.60:0/1989340572 learned_addr learned my addr 172.21.15.60:0/1989340572 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:00.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f881e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f413ef20 0x7f75f41d6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56030/0 (socket says 172.21.15.60:56030) 2024-03-27T21:42:00.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f37fe640 10 monclient(hunting): get_auth_request con 0x7f75f40a87e0 auth_method 0 2024-03-27T21:42:00.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:00.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:00.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:00.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f881e640 10 monclient(hunting): get_auth_request con 0x7f75f413ef20 auth_method 0 2024-03-27T21:42:00.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f881e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:00.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f881e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:00.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f881e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:00.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:00.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:00.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:00.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f881e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:00.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f881e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:00.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f881e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:00.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f3fff640 1 --2- 172.21.15.60:0/1989340572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75f4071a50 0x7f75f4138e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:00.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f3fff640 10 monclient(hunting): get_auth_request con 0x7f75f4071a50 auth_method 0 2024-03-27T21:42:00.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:00.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:00.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:00.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f37fe640 10 monclient(hunting): handle_auth_done global_id 4976 payload 824 2024-03-27T21:42:00.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f37fe640 1 -- 172.21.15.60:0/1989340572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75f4071a50 msgr2=0x7f75f4138e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:00.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f37fe640 1 --2- 172.21.15.60:0/1989340572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75f4071a50 0x7f75f4138e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:00.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f37fe640 1 -- 172.21.15.60:0/1989340572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f413ef20 msgr2=0x7f75f41d6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:00.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f37fe640 1 --2- 172.21.15.60:0/1989340572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f413ef20 0x7f75f41d6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:00.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f37fe640 10 monclient: _finish_hunting 0 2024-03-27T21:42:00.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f37fe640 1 monclient: found mon.noname-c 2024-03-27T21:42:00.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:00.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:00.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f37fe640 1 -- 172.21.15.60:0/1989340572 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ec009a70 con 0x7f75f40a87e0 2024-03-27T21:42:00.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f881e640 1 --2- 172.21.15.60:0/1989340572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f413ef20 0x7f75f41d6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:00.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f37fe640 1 --2- 172.21.15.60:0/1989340572 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f75f40a87e0 0x7f75f41d6540 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f75e800c3e0 tx=0x7f75e800c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:00.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f0ff9640 10 client.?.objecter ms_handle_connect 0x7f75f40a87e0 2024-03-27T21:42:00.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:00.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f0ff9640 1 -- 172.21.15.60:0/1989340572 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e80074e0 con 0x7f75f40a87e0 2024-03-27T21:42:00.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:00.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:00.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f0ff9640 10 monclient: dump: 2024-03-27T21:42:00.389 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:00.389 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:00.389 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:00.390 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:00.390 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:00.390 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:00.390 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:00.390 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:00.390 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:00.390 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:00.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f0ff9640 10 monclient: _finish_auth 0 2024-03-27T21:42:00.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f0ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:42:00.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.380+0000 7f75f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:00.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.381+0000 7f75fa281640 5 monclient: authenticate success, global_id 4976 2024-03-27T21:42:00.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.381+0000 7f75f0ff9640 1 -- 172.21.15.60:0/1989340572 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e8007660 con 0x7f75f40a87e0 2024-03-27T21:42:00.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.381+0000 7f75f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:00.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.381+0000 7f75f0ff9640 1 -- 172.21.15.60:0/1989340572 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e8004a20 con 0x7f75f40a87e0 2024-03-27T21:42:00.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.381+0000 7f75f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:00.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.381+0000 7f75f0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:00.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.381+0000 7f75f0ff9640 10 monclient: dump: 2024-03-27T21:42:00.392 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:00.392 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:00.392 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:00.392 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:00.392 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:00.392 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:00.392 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:00.392 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:00.393 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:00.393 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:00.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.381+0000 7f75fa281640 10 monclient: _renew_subs 2024-03-27T21:42:00.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.381+0000 7f75fa281640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:00.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.381+0000 7f75fa281640 1 -- 172.21.15.60:0/1989340572 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f40b6b30 con 0x7f75f40a87e0 2024-03-27T21:42:00.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.381+0000 7f75f27fc640 4 set_mon_vals no callback set 2024-03-27T21:42:00.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.381+0000 7f75fa281640 10 client.4976.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:00.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.381+0000 7f75fa281640 10 monclient: _renew_subs 2024-03-27T21:42:00.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.381+0000 7f75fa281640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:00.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.381+0000 7f75fa281640 1 -- 172.21.15.60:0/1989340572 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f41397f0 con 0x7f75f40a87e0 2024-03-27T21:42:00.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.381+0000 7f75fa281640 1 librados: init done 2024-03-27T21:42:00.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.381+0000 7f75fa281640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:00.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.381+0000 7f75fa281640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:00.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.381+0000 7f75fa281640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:00.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.381+0000 7f75fa281640 1 -- 172.21.15.60:0/1989340572 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f4069470 con 0x7f75f40a87e0 2024-03-27T21:42:00.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.382+0000 7f75f0ff9640 1 -- 172.21.15.60:0/1989340572 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f75e8004b80 con 0x7f75f40a87e0 2024-03-27T21:42:00.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.382+0000 7f75f0ff9640 1 --2- 172.21.15.60:0/1989340572 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f75e006c590 0x7f75e006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:00.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.382+0000 7f75f0ff9640 1 -- 172.21.15.60:0/1989340572 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f75e8088e60 con 0x7f75f40a87e0 2024-03-27T21:42:00.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.383+0000 7f75f3fff640 1 --2- 172.21.15.60:0/1989340572 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f75e006c590 0x7f75e006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:00.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.383+0000 7f75f0ff9640 10 client.4976.objecter ms_dispatch 0x7f75f4000b90 osd_map(88..88 src has 1..88) v4 2024-03-27T21:42:00.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.383+0000 7f75f0ff9640 3 client.4976.objecter handle_osd_map got epochs [88,88] > 0 2024-03-27T21:42:00.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.383+0000 7f75f0ff9640 3 client.4976.objecter handle_osd_map decoding full epoch 88 2024-03-27T21:42:00.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.383+0000 7f75f3fff640 10 monclient: get_auth_request con 0x7f75e006c590 auth_method 0 2024-03-27T21:42:00.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.383+0000 7f75f0ff9640 20 client.4976.objecter dump_active .. 0 homeless 2024-03-27T21:42:00.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.383+0000 7f75f3fff640 1 --2- 172.21.15.60:0/1989340572 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f75e006c590 0x7f75e006ea50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f75e4004ea0 tx=0x7f75e4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:00.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.383+0000 7f75f0ff9640 10 client.4976.objecter ms_handle_connect 0x7f75e006c590 2024-03-27T21:42:00.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.387+0000 7f75f0ff9640 1 -- 172.21.15.60:0/1989340572 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f75e8057d30 con 0x7f75f40a87e0 2024-03-27T21:42:00.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.387+0000 7f75f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:00.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.387+0000 7f75f0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:00.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.579+0000 7f75fa281640 10 start_mon_command cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}] 2024-03-27T21:42:00.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.579+0000 7f75fa281640 10 monclient: _send_command 2 [{"prefix": "osd reweight", "id": 7, "weight": 1.0}] 2024-03-27T21:42:00.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.579+0000 7f75fa281640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:00.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:00.579+0000 7f75fa281640 1 -- 172.21.15.60:0/1989340572 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f75f40a9260 con 0x7f75f40a87e0 2024-03-27T21:42:01.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.258+0000 7f75f0ff9640 1 -- 172.21.15.60:0/1989340572 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v89) v1 ==== 113+0+0 (secure 0 0 0) 0x7f75e80578c0 con 0x7f75f40a87e0 2024-03-27T21:42:01.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.258+0000 7f75f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd reweight", "id": 7, "weight": 1.0}] 2024-03-27T21:42:01.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.258+0000 7f75f0ff9640 10 monclient: _finish_command 2 = system:0 reweighted osd.7 to 1 (10000) 2024-03-27T21:42:01.245 INFO:teuthology.orchestra.run.smithi060.stderr:reweighted osd.7 to 1 (10000) 2024-03-27T21:42:01.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.260+0000 7f75fa281640 10 librados: watch_flush enter 2024-03-27T21:42:01.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.260+0000 7f75fa281640 10 librados: watch_flush exit 2024-03-27T21:42:01.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.260+0000 7f75fa281640 20 client.4976.objecter shutdown clearing up homeless session... 2024-03-27T21:42:01.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.260+0000 7f75fa281640 10 client.4976.objecter successfully canceled tick 2024-03-27T21:42:01.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.261+0000 7f75fa281640 1 -- 172.21.15.60:0/1989340572 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f75e006c590 msgr2=0x7f75e006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:01.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.261+0000 7f75fa281640 1 --2- 172.21.15.60:0/1989340572 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f75e006c590 0x7f75e006ea50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f75e4004ea0 tx=0x7f75e4009040 comp rx=0 tx=0).stop 2024-03-27T21:42:01.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.261+0000 7f75fa281640 10 monclient: shutdown 2024-03-27T21:42:01.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.261+0000 7f75fa281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.261+0000 7f75fa281640 1 -- 172.21.15.60:0/1989340572 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f75f40a87e0 msgr2=0x7f75f41d6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.261+0000 7f75fa281640 1 --2- 172.21.15.60:0/1989340572 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f75f40a87e0 0x7f75f41d6540 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f75e800c3e0 tx=0x7f75e800c8a0 comp rx=0 tx=0).stop 2024-03-27T21:42:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.261+0000 7f75fa281640 1 -- 172.21.15.60:0/1989340572 shutdown_connections 2024-03-27T21:42:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.261+0000 7f75fa281640 1 --2- 172.21.15.60:0/1989340572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f413ef20 0x7f75f41d6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.261+0000 7f75fa281640 1 --2- 172.21.15.60:0/1989340572 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f75f40a87e0 0x7f75f41d6540 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.261+0000 7f75fa281640 1 --2- 172.21.15.60:0/1989340572 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f75e006c590 0x7f75e006ea50 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.261+0000 7f75fa281640 1 --2- 172.21.15.60:0/1989340572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75f4071a50 0x7f75f4138e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.261+0000 7f75fa281640 1 -- 172.21.15.60:0/1989340572 >> 172.21.15.60:0/1989340572 conn(0x7f75f40b7300 msgr2=0x7f75f4135790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.261+0000 7f75fa281640 1 -- 172.21.15.60:0/1989340572 shutdown_connections 2024-03-27T21:42:01.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.261+0000 7f75fa281640 1 -- 172.21.15.60:0/1989340572 wait complete. 2024-03-27T21:42:01.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.261+0000 7f75fa281640 1 librados: shutdown 2024-03-27T21:42:01.260 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-27T21:42:01.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.379+0000 7f50e7441640 10 monclient: get_monmap_and_config 2024-03-27T21:42:01.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.379+0000 7f50e7441640 10 monclient: build_initial_monmap 2024-03-27T21:42:01.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.379+0000 7f50e7441640 10 monclient: monmap: 2024-03-27T21:42:01.365 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:01.365 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:01.365 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:01.380094+0000 2024-03-27T21:42:01.366 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:01.380094+0000 2024-03-27T21:42:01.366 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:01.366 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:01.366 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:01.366 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:01.366 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:01.366 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:01.366 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:01.367 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:01.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.379+0000 7f50e7441640 1 Processor -- start 2024-03-27T21:42:01.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.379+0000 7f50e7441640 1 -- start start 2024-03-27T21:42:01.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.379+0000 7f50e7441640 10 monclient: init 2024-03-27T21:42:01.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e7441640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:01.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e7441640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-03-27T21:42:01.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e7441640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f50e00b40b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:01.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e7441640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f50e013b020 0x7f50e013b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:01.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e7441640 10 monclient(hunting): picked mon.noname-b con 0x7f50e013b020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:01.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e7441640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f50e01416f0 0x7f50e0141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:01.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e7441640 10 monclient(hunting): picked mon.noname-c con 0x7f50e01416f0 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:01.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e7441640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e00b3830 con 0x7f50e00b40b0 2024-03-27T21:42:01.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e7441640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:01.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e7441640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50e00b4490 con 0x7f50e01416f0 2024-03-27T21:42:01.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e7441640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:01.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e7441640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e00aa520 con 0x7f50e013b020 2024-03-27T21:42:01.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e7441640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:01.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e7441640 10 monclient(hunting): authenticate will time out at 962.601375s 2024-03-27T21:42:01.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e49dc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f50e013b020 0x7f50e013b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:01.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e51dd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f50e00b40b0 0x7f50e00a8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47296/0 (socket says 172.21.15.60:47296) 2024-03-27T21:42:01.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e51dd640 1 -- 172.21.15.60:0/557089408 learned_addr learned my addr 172.21.15.60:0/557089408 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:01.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e49dc640 10 monclient(hunting): get_auth_request con 0x7f50e013b020 auth_method 0 2024-03-27T21:42:01.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e49dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:01.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e49dc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:01.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e49dc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:01.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e49dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:01.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e49dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:01.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e49dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:01.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e59de640 1 --2- 172.21.15.60:0/557089408 >> v2:172.21.15.191:3300/0 conn(0x7f50e01416f0 0x7f50e0141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:01.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e59de640 10 monclient(hunting): get_auth_request con 0x7f50e01416f0 auth_method 0 2024-03-27T21:42:01.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e59de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:01.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e59de640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:01.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e59de640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:01.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50dffff640 1 -- 172.21.15.60:0/557089408 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369468529 0 0) 0x7f50e00b3830 con 0x7f50e00b40b0 2024-03-27T21:42:01.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50dffff640 10 monclient(hunting): _negotiate 2024-03-27T21:42:01.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50dffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:01.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:01.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50dffff640 10 monclient(hunting): my global_id is 5253 2024-03-27T21:42:01.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50dffff640 1 -- 172.21.15.60:0/557089408 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50c8003520 con 0x7f50e00b40b0 2024-03-27T21:42:01.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e49dc640 10 monclient(hunting): handle_auth_done global_id 4979 payload 274 2024-03-27T21:42:01.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e49dc640 1 -- 172.21.15.60:0/557089408 >> v2:172.21.15.191:3300/0 conn(0x7f50e01416f0 msgr2=0x7f50e0141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:01.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e49dc640 1 --2- 172.21.15.60:0/557089408 >> v2:172.21.15.191:3300/0 conn(0x7f50e01416f0 0x7f50e0141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:01.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e49dc640 1 -- 172.21.15.60:0/557089408 >> v1:172.21.15.60:6789/0 conn(0x7f50e00b40b0 legacy=0x7f50e00a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:01.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e49dc640 10 monclient: _finish_hunting 0 2024-03-27T21:42:01.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e49dc640 1 monclient: found mon.noname-b 2024-03-27T21:42:01.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e49dc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:01.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e49dc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:42:01.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.380+0000 7f50e49dc640 1 -- 172.21.15.60:0/557089408 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e00a9470 con 0x7f50e013b020 2024-03-27T21:42:01.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.381+0000 7f50e49dc640 1 --2- 172.21.15.60:0/557089408 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f50e013b020 0x7f50e013b480 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f50d40097a0 tx=0x7f50d402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e5cc5219fb37dc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:01.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.381+0000 7f50dffff640 1 -- 172.21.15.60:0/557089408 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d4031830 con 0x7f50e013b020 2024-03-27T21:42:01.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.381+0000 7f50dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:01.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.381+0000 7f50dffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:01.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.381+0000 7f50dffff640 10 monclient: dump: 2024-03-27T21:42:01.381 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:01.381 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:01.381 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:01.381 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:01.381 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:01.381 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:01.382 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:01.382 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:01.382 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:01.382 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:01.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.381+0000 7f50dffff640 10 monclient: _finish_auth 0 2024-03-27T21:42:01.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.381+0000 7f50dffff640 10 monclient: _check_auth_tickets 2024-03-27T21:42:01.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.381+0000 7f50dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:01.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.381+0000 7f50e7441640 5 monclient: authenticate success, global_id 4979 2024-03-27T21:42:01.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.381+0000 7f50e7441640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:01.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.381+0000 7f50dffff640 1 -- 172.21.15.60:0/557089408 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d4031990 con 0x7f50e013b020 2024-03-27T21:42:01.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.381+0000 7f50dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:01.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.381+0000 7f50e7441640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:01.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.381+0000 7f50dffff640 1 -- 172.21.15.60:0/557089408 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d4031d00 con 0x7f50e013b020 2024-03-27T21:42:01.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.381+0000 7f50dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:01.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.381+0000 7f50e7441640 4 set_mon_vals no callback set 2024-03-27T21:42:01.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.381+0000 7f50dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:01.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.381+0000 7f50dffff640 10 monclient: dump: 2024-03-27T21:42:01.384 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:01.384 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:01.384 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:01.384 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:01.385 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:01.385 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:01.385 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:01.385 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:01.385 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:01.385 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:01.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.382+0000 7f50e7441640 10 monclient: shutdown 2024-03-27T21:42:01.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.382+0000 7f50e7441640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:01.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.382+0000 7f50e7441640 1 -- 172.21.15.60:0/557089408 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f50e013b020 msgr2=0x7f50e013b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:01.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.382+0000 7f50e7441640 1 --2- 172.21.15.60:0/557089408 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f50e013b020 0x7f50e013b480 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f50d40097a0 tx=0x7f50d402b8d0 comp rx=0 tx=0).stop 2024-03-27T21:42:01.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.382+0000 7f50e7441640 1 -- 172.21.15.60:0/557089408 shutdown_connections 2024-03-27T21:42:01.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.382+0000 7f50e7441640 1 --2- 172.21.15.60:0/557089408 >> v2:172.21.15.191:3300/0 conn(0x7f50e01416f0 0x7f50e0141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:01.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.382+0000 7f50e7441640 1 --2- 172.21.15.60:0/557089408 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f50e013b020 0x7f50e013b480 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:01.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.382+0000 7f50e7441640 1 -- 172.21.15.60:0/557089408 >> 172.21.15.60:0/557089408 conn(0x7f50e00608e0 msgr2=0x7f50e0060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:01.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.382+0000 7f50e7441640 1 -- 172.21.15.60:0/557089408 shutdown_connections 2024-03-27T21:42:01.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.382+0000 7f50e7441640 1 -- 172.21.15.60:0/557089408 wait complete. 2024-03-27T21:42:01.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.383+0000 7f50e7441640 10 monclient: build_initial_monmap 2024-03-27T21:42:01.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.383+0000 7f50e7441640 10 monclient: monmap: 2024-03-27T21:42:01.387 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:01.387 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:01.387 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:01.387 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:01.387 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:01.387 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:01.387 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:01.388 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:01.388 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:01.388 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:01.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.383+0000 7f50e7441640 1 librados: starting msgr at 2024-03-27T21:42:01.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.383+0000 7f50e7441640 1 librados: starting objecter 2024-03-27T21:42:01.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.383+0000 7f50e7441640 1 Processor -- start 2024-03-27T21:42:01.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.383+0000 7f50e7441640 1 -- start start 2024-03-27T21:42:01.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.383+0000 7f50e7441640 1 librados: setting wanted keys 2024-03-27T21:42:01.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.383+0000 7f50e7441640 1 librados: calling monclient init 2024-03-27T21:42:01.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.383+0000 7f50e7441640 10 monclient: init 2024-03-27T21:42:01.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.383+0000 7f50e7441640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:01.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.383+0000 7f50e7441640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:42:01.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e7441640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f50e00b40b0 0x7f50e00aac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:01.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e7441640 10 monclient(hunting): picked mon.noname-c con 0x7f50e00b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:01.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e7441640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50e013b020 0x7f50e013bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:01.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e7441640 10 monclient(hunting): picked mon.noname-b con 0x7f50e013b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:01.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e7441640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50e01416f0 0x7f50e013c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:01.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e7441640 10 monclient(hunting): picked mon.noname-a con 0x7f50e01416f0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:01.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e7441640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:01.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e7441640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e00acc60 con 0x7f50e01416f0 2024-03-27T21:42:01.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e7441640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:01.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e7441640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e00ab760 con 0x7f50e013b020 2024-03-27T21:42:01.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e7441640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:01.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e7441640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e013dbe0 con 0x7f50e00b40b0 2024-03-27T21:42:01.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e7441640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:01.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e7441640 10 monclient(hunting): authenticate will time out at 962.604967s 2024-03-27T21:42:01.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e51dd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f50e00b40b0 0x7f50e00aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:01.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e59de640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50e01416f0 0x7f50e013c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:01.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e51dd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f50e00b40b0 0x7f50e00aac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:39052/0 (socket says 172.21.15.60:39052) 2024-03-27T21:42:01.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e59de640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50e01416f0 0x7f50e013c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56042/0 (socket says 172.21.15.60:56042) 2024-03-27T21:42:01.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e51dd640 1 -- 172.21.15.60:0/2340002097 learned_addr learned my addr 172.21.15.60:0/2340002097 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:01.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e51dd640 10 monclient(hunting): get_auth_request con 0x7f50e00b40b0 auth_method 0 2024-03-27T21:42:01.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e51dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:01.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e51dd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:01.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e51dd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:01.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e59de640 10 monclient(hunting): get_auth_request con 0x7f50e01416f0 auth_method 0 2024-03-27T21:42:01.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e59de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:01.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e59de640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:01.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e59de640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:01.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e51dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:01.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e51dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:01.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e51dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:01.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e59de640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:01.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e59de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:01.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e59de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:01.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e49dc640 1 --2- 172.21.15.60:0/2340002097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50e013b020 0x7f50e013bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:01.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e49dc640 10 monclient(hunting): get_auth_request con 0x7f50e013b020 auth_method 0 2024-03-27T21:42:01.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e49dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:01.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e49dc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:01.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e49dc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:01.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e51dd640 10 monclient(hunting): handle_auth_done global_id 4982 payload 824 2024-03-27T21:42:01.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e51dd640 1 -- 172.21.15.60:0/2340002097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50e013b020 msgr2=0x7f50e013bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:01.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e51dd640 1 --2- 172.21.15.60:0/2340002097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50e013b020 0x7f50e013bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:01.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e51dd640 1 -- 172.21.15.60:0/2340002097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50e01416f0 msgr2=0x7f50e013c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:01.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e51dd640 1 --2- 172.21.15.60:0/2340002097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50e01416f0 0x7f50e013c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:01.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e51dd640 10 monclient: _finish_hunting 0 2024-03-27T21:42:01.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e51dd640 1 monclient: found mon.noname-c 2024-03-27T21:42:01.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e51dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:01.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e51dd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:01.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e51dd640 1 -- 172.21.15.60:0/2340002097 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d4009400 con 0x7f50e00b40b0 2024-03-27T21:42:01.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e49dc640 1 --2- 172.21.15.60:0/2340002097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50e013b020 0x7f50e013bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:42:01.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e51dd640 1 --2- 172.21.15.60:0/2340002097 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f50e00b40b0 0x7f50e00aac40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f50d800c460 tx=0x7f50d800c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:01.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50ddffb640 10 client.?.objecter ms_handle_connect 0x7f50e00b40b0 2024-03-27T21:42:01.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50ddffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:01.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50ddffb640 1 -- 172.21.15.60:0/2340002097 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d8005ce0 con 0x7f50e00b40b0 2024-03-27T21:42:01.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:01.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50ddffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:01.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50ddffb640 10 monclient: dump: 2024-03-27T21:42:01.396 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:01.396 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:01.396 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:01.396 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:01.396 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:01.396 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:01.396 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:01.397 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:01.397 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:01.397 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:01.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50ddffb640 10 monclient: _finish_auth 0 2024-03-27T21:42:01.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50ddffb640 10 monclient: _check_auth_tickets 2024-03-27T21:42:01.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:01.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50ddffb640 1 -- 172.21.15.60:0/2340002097 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d8005e40 con 0x7f50e00b40b0 2024-03-27T21:42:01.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e7441640 5 monclient: authenticate success, global_id 4982 2024-03-27T21:42:01.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:01.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50ddffb640 1 -- 172.21.15.60:0/2340002097 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d8007290 con 0x7f50e00b40b0 2024-03-27T21:42:01.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e7441640 10 monclient: _renew_subs 2024-03-27T21:42:01.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e7441640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:01.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e7441640 1 -- 172.21.15.60:0/2340002097 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e00ab240 con 0x7f50e00b40b0 2024-03-27T21:42:01.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:01.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:01.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50ddffb640 10 monclient: dump: 2024-03-27T21:42:01.398 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:01.399 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:01.399 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:01.399 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:01.399 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:01.399 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:01.399 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:01.399 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:01.399 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:01.399 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:01.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50df7fe640 4 set_mon_vals no callback set 2024-03-27T21:42:01.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e7441640 10 client.4982.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:01.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.384+0000 7f50e7441640 10 monclient: _renew_subs 2024-03-27T21:42:01.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.385+0000 7f50e7441640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:01.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.385+0000 7f50e7441640 1 -- 172.21.15.60:0/2340002097 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e00ab960 con 0x7f50e00b40b0 2024-03-27T21:42:01.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.385+0000 7f50e7441640 1 librados: init done 2024-03-27T21:42:01.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.385+0000 7f50e7441640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:01.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.385+0000 7f50e7441640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:01.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.385+0000 7f50e7441640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:01.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.385+0000 7f50e7441640 1 -- 172.21.15.60:0/2340002097 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e00ab480 con 0x7f50e00b40b0 2024-03-27T21:42:01.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.386+0000 7f50ddffb640 1 -- 172.21.15.60:0/2340002097 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f50d80073f0 con 0x7f50e00b40b0 2024-03-27T21:42:01.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.386+0000 7f50ddffb640 1 --2- 172.21.15.60:0/2340002097 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f50cc06c590 0x7f50cc06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:01.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.386+0000 7f50ddffb640 1 -- 172.21.15.60:0/2340002097 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f50d80885e0 con 0x7f50e00b40b0 2024-03-27T21:42:01.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.386+0000 7f50ddffb640 10 client.4982.objecter ms_dispatch 0x7f50e0000b90 osd_map(89..89 src has 1..89) v4 2024-03-27T21:42:01.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.386+0000 7f50ddffb640 3 client.4982.objecter handle_osd_map got epochs [89,89] > 0 2024-03-27T21:42:01.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.386+0000 7f50ddffb640 3 client.4982.objecter handle_osd_map decoding full epoch 89 2024-03-27T21:42:01.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.386+0000 7f50e49dc640 1 --2- 172.21.15.60:0/2340002097 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f50cc06c590 0x7f50cc06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:01.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.386+0000 7f50e49dc640 10 monclient: get_auth_request con 0x7f50cc06c590 auth_method 0 2024-03-27T21:42:01.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.386+0000 7f50ddffb640 20 client.4982.objecter dump_active .. 0 homeless 2024-03-27T21:42:01.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.387+0000 7f50ddffb640 10 client.4982.objecter ms_handle_connect 0x7f50cc06c590 2024-03-27T21:42:01.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.387+0000 7f50e49dc640 1 --2- 172.21.15.60:0/2340002097 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f50cc06c590 0x7f50cc06ea50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f50d4036000 tx=0x7f50d403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:01.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.391+0000 7f50ddffb640 1 -- 172.21.15.60:0/2340002097 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f50e00ab480 con 0x7f50e00b40b0 2024-03-27T21:42:01.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.391+0000 7f50ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:01.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.391+0000 7f50ddffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:01.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.583+0000 7f50e7441640 10 start_mon_command cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}] 2024-03-27T21:42:01.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.583+0000 7f50e7441640 10 monclient: _send_command 2 [{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}] 2024-03-27T21:42:01.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.583+0000 7f50e7441640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:01.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:01.583+0000 7f50e7441640 1 -- 172.21.15.60:0/2340002097 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f50e00ab480 con 0x7f50e00b40b0 2024-03-27T21:42:02.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.265+0000 7f50ddffb640 1 -- 172.21.15.60:0/2340002097 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]=0 set osd.7 primary-affinity to 1 (8655362) v90) v1 ==== 133+0+0 (secure 0 0 0) 0x7f50e00ab480 con 0x7f50e00b40b0 2024-03-27T21:42:02.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.265+0000 7f50ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}] 2024-03-27T21:42:02.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.265+0000 7f50ddffb640 10 monclient: _finish_command 2 = system:0 set osd.7 primary-affinity to 1 (8655362) 2024-03-27T21:42:02.252 INFO:teuthology.orchestra.run.smithi060.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-27T21:42:02.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.268+0000 7f50e7441640 10 librados: watch_flush enter 2024-03-27T21:42:02.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.269+0000 7f50e7441640 10 librados: watch_flush exit 2024-03-27T21:42:02.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.269+0000 7f50e7441640 20 client.4982.objecter shutdown clearing up homeless session... 2024-03-27T21:42:02.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.269+0000 7f50e7441640 10 client.4982.objecter successfully canceled tick 2024-03-27T21:42:02.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.269+0000 7f50e7441640 1 -- 172.21.15.60:0/2340002097 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f50cc06c590 msgr2=0x7f50cc06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:02.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.269+0000 7f50e7441640 1 --2- 172.21.15.60:0/2340002097 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f50cc06c590 0x7f50cc06ea50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f50d4036000 tx=0x7f50d403c040 comp rx=0 tx=0).stop 2024-03-27T21:42:02.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.269+0000 7f50e7441640 10 monclient: shutdown 2024-03-27T21:42:02.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.269+0000 7f50e7441640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:02.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.269+0000 7f50e7441640 1 -- 172.21.15.60:0/2340002097 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f50e00b40b0 msgr2=0x7f50e00aac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:02.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.269+0000 7f50e7441640 1 --2- 172.21.15.60:0/2340002097 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f50e00b40b0 0x7f50e00aac40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f50d800c460 tx=0x7f50d800c920 comp rx=0 tx=0).stop 2024-03-27T21:42:02.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.269+0000 7f50e7441640 1 -- 172.21.15.60:0/2340002097 shutdown_connections 2024-03-27T21:42:02.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.269+0000 7f50e7441640 1 --2- 172.21.15.60:0/2340002097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50e01416f0 0x7f50e013c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:02.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.269+0000 7f50e7441640 1 --2- 172.21.15.60:0/2340002097 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f50cc06c590 0x7f50cc06ea50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:02.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.269+0000 7f50e7441640 1 --2- 172.21.15.60:0/2340002097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50e013b020 0x7f50e013bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:02.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.269+0000 7f50e7441640 1 --2- 172.21.15.60:0/2340002097 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f50e00b40b0 0x7f50e00aac40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:02.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.269+0000 7f50e7441640 1 -- 172.21.15.60:0/2340002097 >> 172.21.15.60:0/2340002097 conn(0x7f50e00608e0 msgr2=0x7f50e0060cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:02.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.269+0000 7f50e7441640 1 -- 172.21.15.60:0/2340002097 shutdown_connections 2024-03-27T21:42:02.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.269+0000 7f50e7441640 1 -- 172.21.15.60:0/2340002097 wait complete. 2024-03-27T21:42:02.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.269+0000 7f50e7441640 1 librados: shutdown 2024-03-27T21:42:02.269 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-27T21:42:02.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.390+0000 7fc22452e640 10 monclient: get_monmap_and_config 2024-03-27T21:42:02.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.390+0000 7fc22452e640 10 monclient: build_initial_monmap 2024-03-27T21:42:02.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.390+0000 7fc22452e640 10 monclient: monmap: 2024-03-27T21:42:02.377 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:02.378 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:02.378 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:02.391825+0000 2024-03-27T21:42:02.378 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:02.391825+0000 2024-03-27T21:42:02.378 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:02.378 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:02.378 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:02.378 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:02.378 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:02.379 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:02.379 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:02.379 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:02.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.391+0000 7fc22452e640 1 Processor -- start 2024-03-27T21:42:02.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.391+0000 7fc22452e640 1 -- start start 2024-03-27T21:42:02.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.391+0000 7fc22452e640 10 monclient: init 2024-03-27T21:42:02.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc22452e640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:02.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc22452e640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-03-27T21:42:02.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc22452e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc21c071a50 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:02.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc22452e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc21c13ef20 0x7fc21c0a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:02.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc22452e640 10 monclient(hunting): picked mon.noname-a con 0x7fc21c13ef20 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:02.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc22452e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc21c141630 0x7fc21c141ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:02.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc22452e640 10 monclient(hunting): picked mon.noname-b con 0x7fc21c141630 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:02.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc22452e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc21c0a8e90 con 0x7fc21c071a50 2024-03-27T21:42:02.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc22452e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:02.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc22452e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc21c071e30 con 0x7fc21c13ef20 2024-03-27T21:42:02.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc22452e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:02.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc22452e640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c0aa040 con 0x7fc21c141630 2024-03-27T21:42:02.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc22452e640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:02.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc22452e640 10 monclient(hunting): authenticate will time out at 963.613155s 2024-03-27T21:42:02.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc222acb640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc21c141630 0x7fc21c141ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:02.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc221ac9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc21c13ef20 0x7fc21c0a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:02.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc222acb640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc21c141630 0x7fc21c141ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:33990/0 (socket says 172.21.15.60:33990) 2024-03-27T21:42:02.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc221ac9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc21c13ef20 0x7fc21c0a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39126/0 (socket says 172.21.15.60:39126) 2024-03-27T21:42:02.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc222acb640 1 -- 172.21.15.60:0/2389138449 learned_addr learned my addr 172.21.15.60:0/2389138449 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:02.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc222acb640 10 monclient(hunting): get_auth_request con 0x7fc21c141630 auth_method 0 2024-03-27T21:42:02.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc222acb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:02.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc222acb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:02.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc222acb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:02.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc221ac9640 10 monclient(hunting): get_auth_request con 0x7fc21c13ef20 auth_method 0 2024-03-27T21:42:02.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc221ac9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:02.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc221ac9640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:02.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc221ac9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:02.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc221ac9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:02.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc221ac9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:02.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc221ac9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:02.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc222acb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:02.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc222acb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:02.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc222acb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:02.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc221ac9640 10 monclient(hunting): handle_auth_done global_id 5259 payload 274 2024-03-27T21:42:02.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc221ac9640 1 -- 172.21.15.60:0/2389138449 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc21c141630 msgr2=0x7fc21c141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:02.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc221ac9640 1 --2- 172.21.15.60:0/2389138449 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc21c141630 0x7fc21c141ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:02.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc221ac9640 1 -- 172.21.15.60:0/2389138449 >> v1:172.21.15.60:6789/0 conn(0x7fc21c071a50 legacy=0x7fc21c13e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:02.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc221ac9640 10 monclient: _finish_hunting 0 2024-03-27T21:42:02.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc221ac9640 1 monclient: found mon.noname-a 2024-03-27T21:42:02.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc221ac9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:02.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc221ac9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:02.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc221ac9640 1 -- 172.21.15.60:0/2389138449 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc21c0aa2b0 con 0x7fc21c13ef20 2024-03-27T21:42:02.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.392+0000 7fc222acb640 1 --2- 172.21.15.60:0/2389138449 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc21c141630 0x7fc21c141ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:02.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.393+0000 7fc221ac9640 1 --2- 172.21.15.60:0/2389138449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc21c13ef20 0x7fc21c0a87e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc2100097a0 tx=0x7fc21002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74c0ae1a59038a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:02.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.393+0000 7fc2212c8640 1 -- 172.21.15.60:0/2389138449 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21002d040 con 0x7fc21c13ef20 2024-03-27T21:42:02.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.393+0000 7fc2212c8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:02.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.393+0000 7fc2212c8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:02.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.393+0000 7fc2212c8640 10 monclient: dump: 2024-03-27T21:42:02.391 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:02.391 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:02.391 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:02.391 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:02.392 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:02.392 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:02.392 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:02.392 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:02.392 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:02.392 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:02.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.393+0000 7fc2212c8640 10 monclient: _finish_auth 0 2024-03-27T21:42:02.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.393+0000 7fc2212c8640 10 monclient: _check_auth_tickets 2024-03-27T21:42:02.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.393+0000 7fc2212c8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:02.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.393+0000 7fc22452e640 5 monclient: authenticate success, global_id 5259 2024-03-27T21:42:02.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.393+0000 7fc2212c8640 1 -- 172.21.15.60:0/2389138449 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc210031810 con 0x7fc21c13ef20 2024-03-27T21:42:02.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.393+0000 7fc22452e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:02.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.393+0000 7fc2212c8640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:02.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.393+0000 7fc22452e640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:02.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.393+0000 7fc22452e640 4 set_mon_vals no callback set 2024-03-27T21:42:02.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.393+0000 7fc2212c8640 1 -- 172.21.15.60:0/2389138449 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc210031ba0 con 0x7fc21c13ef20 2024-03-27T21:42:02.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.393+0000 7fc2212c8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:02.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.393+0000 7fc2212c8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:02.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.393+0000 7fc2212c8640 10 monclient: dump: 2024-03-27T21:42:02.394 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:02.394 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:02.394 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:02.394 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:02.394 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:02.394 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:02.395 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:02.395 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:02.395 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:02.395 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:02.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.394+0000 7fc22452e640 10 monclient: shutdown 2024-03-27T21:42:02.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.394+0000 7fc22452e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:02.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.394+0000 7fc22452e640 1 -- 172.21.15.60:0/2389138449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc21c13ef20 msgr2=0x7fc21c0a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:02.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.394+0000 7fc22452e640 1 --2- 172.21.15.60:0/2389138449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc21c13ef20 0x7fc21c0a87e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc2100097a0 tx=0x7fc21002b8b0 comp rx=0 tx=0).stop 2024-03-27T21:42:02.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.394+0000 7fc22452e640 1 -- 172.21.15.60:0/2389138449 shutdown_connections 2024-03-27T21:42:02.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.394+0000 7fc22452e640 1 --2- 172.21.15.60:0/2389138449 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc21c141630 0x7fc21c141ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:02.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.394+0000 7fc22452e640 1 --2- 172.21.15.60:0/2389138449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc21c13ef20 0x7fc21c0a87e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:02.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.394+0000 7fc22452e640 1 -- 172.21.15.60:0/2389138449 >> 172.21.15.60:0/2389138449 conn(0x7fc21c0b7300 msgr2=0x7fc21c0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:02.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.394+0000 7fc22452e640 1 -- 172.21.15.60:0/2389138449 shutdown_connections 2024-03-27T21:42:02.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.394+0000 7fc22452e640 1 -- 172.21.15.60:0/2389138449 wait complete. 2024-03-27T21:42:02.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.394+0000 7fc22452e640 10 monclient: build_initial_monmap 2024-03-27T21:42:02.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.394+0000 7fc22452e640 10 monclient: monmap: 2024-03-27T21:42:02.396 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:02.396 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:02.396 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:02.397 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:02.397 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:02.397 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:02.397 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:02.397 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:02.397 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:02.397 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:02.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 1 librados: starting msgr at 2024-03-27T21:42:02.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 1 librados: starting objecter 2024-03-27T21:42:02.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 1 Processor -- start 2024-03-27T21:42:02.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 1 -- start start 2024-03-27T21:42:02.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 1 librados: setting wanted keys 2024-03-27T21:42:02.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 1 librados: calling monclient init 2024-03-27T21:42:02.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 10 monclient: init 2024-03-27T21:42:02.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:02.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:42:02.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc21c071a50 0x7fc21c137530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:02.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 10 monclient(hunting): picked mon.noname-b con 0x7fc21c071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:02.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc21c13ef20 0x7fc21c1364b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:02.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 10 monclient(hunting): picked mon.noname-c con 0x7fc21c13ef20 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:02.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc21c141630 0x7fc21c138fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:02.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 10 monclient(hunting): picked mon.noname-a con 0x7fc21c141630 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:02.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:02.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c13b120 con 0x7fc21c141630 2024-03-27T21:42:02.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:02.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c1398e0 con 0x7fc21c071a50 2024-03-27T21:42:02.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:02.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c0aab50 con 0x7fc21c13ef20 2024-03-27T21:42:02.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:02.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc22452e640 10 monclient(hunting): authenticate will time out at 963.616700s 2024-03-27T21:42:02.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc222acb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc21c141630 0x7fc21c138fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:02.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc221ac9640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc21c13ef20 0x7fc21c1364b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:02.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc222acb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc21c141630 0x7fc21c138fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39130/0 (socket says 172.21.15.60:39130) 2024-03-27T21:42:02.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc222acb640 1 -- 172.21.15.60:0/4180533158 learned_addr learned my addr 172.21.15.60:0/4180533158 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:02.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc222acb640 10 monclient(hunting): get_auth_request con 0x7fc21c141630 auth_method 0 2024-03-27T21:42:02.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc222acb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:02.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc222acb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:02.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc222acb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:02.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc221ac9640 10 monclient(hunting): get_auth_request con 0x7fc21c13ef20 auth_method 0 2024-03-27T21:42:02.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc221ac9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:02.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc221ac9640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:02.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.395+0000 7fc221ac9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:02.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc222acb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:02.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc222acb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:02.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc222acb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:02.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc221ac9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:02.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc221ac9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:02.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc221ac9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:02.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc2222ca640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc21c071a50 0x7fc21c137530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:02.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc2222ca640 10 monclient(hunting): get_auth_request con 0x7fc21c071a50 auth_method 0 2024-03-27T21:42:02.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc2222ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:02.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc2222ca640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:02.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc2222ca640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:02.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc222acb640 10 monclient(hunting): handle_auth_done global_id 5262 payload 824 2024-03-27T21:42:02.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc222acb640 1 -- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc21c13ef20 msgr2=0x7fc21c1364b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:02.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc222acb640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc21c13ef20 0x7fc21c1364b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:02.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc222acb640 1 -- 172.21.15.60:0/4180533158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc21c071a50 msgr2=0x7fc21c137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:02.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc222acb640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc21c071a50 0x7fc21c137530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:02.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc222acb640 10 monclient: _finish_hunting 0 2024-03-27T21:42:02.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc222acb640 1 monclient: found mon.noname-a 2024-03-27T21:42:02.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc222acb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:02.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc222acb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:02.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc222acb640 1 -- 172.21.15.60:0/4180533158 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20c009b30 con 0x7fc21c141630 2024-03-27T21:42:02.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc221ac9640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc21c13ef20 0x7fc21c1364b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:02.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc2222ca640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc21c071a50 0x7fc21c137530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:42:02.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc222acb640 1 -- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc21c141630 msgr2=0x7fc21c138fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2024-03-27T21:42:02.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc222acb640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc21c141630 0x7fc21c138fd0 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7fc20c00efc0 tx=0x7fc20c00c490 comp rx=0 tx=0).write compression request write failed r=-32 ((32) Broken pipe) 2024-03-27T21:42:02.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.396+0000 7fc222acb640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc21c141630 0x7fc21c138fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-27T21:42:02.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc222acb640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc21c141630 0x7fc21c138fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:02.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc222acb640 10 monclient: get_auth_request con 0x7fc21c141630 auth_method 0 2024-03-27T21:42:02.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc222acb640 0 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc21c141630 0x7fc21c138fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-03-27T21:42:02.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc222acb640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc21c141630 0x7fc21c138fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:02.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc21affd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:02.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc21affd640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:02.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc21affd640 1 -- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc21c141630 msgr2=0x7fc21c138fd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-27T21:42:02.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc21affd640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc21c141630 0x7fc21c138fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:02.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc21affd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:42:02.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc21affd640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc21c071a50 0x7fc2080054e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:02.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc21affd640 10 monclient(hunting): picked mon.noname-b con 0x7fc21c071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:02.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc21affd640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc208005f20 0x7fc2080083b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:02.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc21affd640 10 monclient(hunting): picked mon.noname-a con 0x7fc208005f20 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:02.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc21affd640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc21c13ef20 0x7fc20800af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:02.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc21affd640 10 monclient(hunting): picked mon.noname-c con 0x7fc21c13ef20 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:02.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc21affd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:02.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc21affd640 1 -- 172.21.15.60:0/4180533158 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20800b940 con 0x7fc208005f20 2024-03-27T21:42:02.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc21affd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:02.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc21affd640 1 -- 172.21.15.60:0/4180533158 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20800ba80 con 0x7fc21c071a50 2024-03-27T21:42:02.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc21affd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:02.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc21affd640 1 -- 172.21.15.60:0/4180533158 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20800bc40 con 0x7fc21c13ef20 2024-03-27T21:42:02.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc21affd640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:02.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc221ac9640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc208005f20 0x7fc2080083b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:02.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc221ac9640 10 monclient(hunting): get_auth_request con 0x7fc208005f20 auth_method 0 2024-03-27T21:42:02.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc222acb640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc21c13ef20 0x7fc20800af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:02.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc221ac9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:02.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc221ac9640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:02.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc221ac9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:02.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc222acb640 10 monclient(hunting): get_auth_request con 0x7fc21c13ef20 auth_method 0 2024-03-27T21:42:02.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc222acb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:02.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc222acb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:02.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc222acb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:02.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc221ac9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:02.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc221ac9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:02.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc221ac9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:42:02.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc222acb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:02.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc222acb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:02.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc222acb640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:42:02.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc221ac9640 10 monclient(hunting): handle_auth_done global_id 5262 payload 293 2024-03-27T21:42:02.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc2222ca640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc21c071a50 0x7fc2080054e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:02.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc221ac9640 1 -- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc21c13ef20 msgr2=0x7fc20800af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:02.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc221ac9640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc21c13ef20 0x7fc20800af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:02.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc221ac9640 1 -- 172.21.15.60:0/4180533158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc21c071a50 msgr2=0x7fc2080054e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:02.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc221ac9640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc21c071a50 0x7fc2080054e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:02.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc221ac9640 10 monclient: _finish_hunting 0 2024-03-27T21:42:02.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc221ac9640 1 monclient: found mon.noname-a 2024-03-27T21:42:02.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc221ac9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:02.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc221ac9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:02.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc221ac9640 1 -- 172.21.15.60:0/4180533158 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20800bec0 con 0x7fc208005f20 2024-03-27T21:42:02.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc2222ca640 10 monclient: get_auth_request con 0x7fc21c071a50 auth_method 0 2024-03-27T21:42:02.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc2222ca640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc21c071a50 0x7fc2080054e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:42:02.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc221ac9640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc208005f20 0x7fc2080083b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc210033000 tx=0x7fc20c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:02.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc21affd640 10 client.?.objecter ms_handle_connect 0x7fc208005f20 2024-03-27T21:42:02.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.597+0000 7fc21affd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:02.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc21affd640 1 -- 172.21.15.60:0/4180533158 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21003a620 con 0x7fc208005f20 2024-03-27T21:42:02.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc21affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:02.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc21affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:02.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc21affd640 10 monclient: dump: 2024-03-27T21:42:02.595 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:02.595 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:02.595 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:02.595 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:02.595 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:02.595 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:02.595 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:02.595 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:02.596 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:02.596 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:02.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc21affd640 10 monclient: _finish_auth 0 2024-03-27T21:42:02.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc21affd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:02.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc21affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:02.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc22452e640 5 monclient: authenticate success, global_id 5262 2024-03-27T21:42:02.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc21affd640 1 -- 172.21.15.60:0/4180533158 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc21003a780 con 0x7fc208005f20 2024-03-27T21:42:02.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc22452e640 10 monclient: _renew_subs 2024-03-27T21:42:02.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc22452e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:02.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc22452e640 1 -- 172.21.15.60:0/4180533158 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc210009400 con 0x7fc208005f20 2024-03-27T21:42:02.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc21affd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:02.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc21affd640 1 -- 172.21.15.60:0/4180533158 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21003ab10 con 0x7fc208005f20 2024-03-27T21:42:02.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc21affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:02.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc21affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:02.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc21affd640 10 monclient: dump: 2024-03-27T21:42:02.597 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:02.597 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:02.598 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:02.598 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:02.598 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:02.598 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:02.598 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:02.598 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:02.598 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:02.598 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:02.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc220ac7640 4 set_mon_vals no callback set 2024-03-27T21:42:02.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc22452e640 10 client.5262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:02.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc22452e640 10 monclient: _renew_subs 2024-03-27T21:42:02.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc22452e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:02.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc22452e640 1 -- 172.21.15.60:0/4180533158 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc21c1366e0 con 0x7fc208005f20 2024-03-27T21:42:02.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.598+0000 7fc22452e640 1 librados: init done 2024-03-27T21:42:02.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.599+0000 7fc22452e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:02.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.599+0000 7fc22452e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:02.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.599+0000 7fc22452e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:02.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.599+0000 7fc22452e640 1 -- 172.21.15.60:0/4180533158 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc21c067900 con 0x7fc208005f20 2024-03-27T21:42:02.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.599+0000 7fc21affd640 1 -- 172.21.15.60:0/4180533158 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc21003ac70 con 0x7fc208005f20 2024-03-27T21:42:02.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.599+0000 7fc21affd640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc21c141630 0x7fc208076c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:02.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.599+0000 7fc2222ca640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc21c141630 0x7fc208076c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:02.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.599+0000 7fc21affd640 1 -- 172.21.15.60:0/4180533158 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fc21003f070 con 0x7fc208005f20 2024-03-27T21:42:02.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.599+0000 7fc21affd640 10 client.5262.objecter ms_dispatch 0x7fc21c000b90 osd_map(90..90 src has 1..90) v4 2024-03-27T21:42:02.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.600+0000 7fc21affd640 3 client.5262.objecter handle_osd_map got epochs [90,90] > 0 2024-03-27T21:42:02.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.600+0000 7fc21affd640 3 client.5262.objecter handle_osd_map decoding full epoch 90 2024-03-27T21:42:02.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.600+0000 7fc2222ca640 10 monclient: get_auth_request con 0x7fc21c141630 auth_method 0 2024-03-27T21:42:02.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.600+0000 7fc21affd640 20 client.5262.objecter dump_active .. 0 homeless 2024-03-27T21:42:02.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.600+0000 7fc21affd640 10 client.5262.objecter ms_handle_connect 0x7fc21c141630 2024-03-27T21:42:02.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.600+0000 7fc2222ca640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc21c141630 0x7fc208076c60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc214000f60 tx=0x7fc214009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:02.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.604+0000 7fc21affd640 1 -- 172.21.15.60:0/4180533158 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc210075c60 con 0x7fc208005f20 2024-03-27T21:42:02.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.604+0000 7fc21affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:02.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.604+0000 7fc21affd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:02.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.795+0000 7fc22452e640 10 start_mon_command cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}] 2024-03-27T21:42:02.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.795+0000 7fc22452e640 10 monclient: _send_command 2 [{"prefix": "osd reweight", "id": 2, "weight": 1.0}] 2024-03-27T21:42:02.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.795+0000 7fc22452e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:02.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:02.795+0000 7fc22452e640 1 -- 172.21.15.60:0/4180533158 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) v1 -- 0x7fc21c0b6c60 con 0x7fc208005f20 2024-03-27T21:42:03.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.271+0000 7fc21affd640 1 -- 172.21.15.60:0/4180533158 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v91) v1 ==== 113+0+0 (secure 0 0 0) 0x7fc210079280 con 0x7fc208005f20 2024-03-27T21:42:03.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.271+0000 7fc21affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd reweight", "id": 2, "weight": 1.0}] 2024-03-27T21:42:03.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.271+0000 7fc21affd640 10 monclient: _finish_command 2 = system:0 reweighted osd.2 to 1 (10000) 2024-03-27T21:42:03.258 INFO:teuthology.orchestra.run.smithi060.stderr:reweighted osd.2 to 1 (10000) 2024-03-27T21:42:03.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.274+0000 7fc22452e640 10 librados: watch_flush enter 2024-03-27T21:42:03.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.274+0000 7fc22452e640 10 librados: watch_flush exit 2024-03-27T21:42:03.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.274+0000 7fc22452e640 20 client.5262.objecter shutdown clearing up homeless session... 2024-03-27T21:42:03.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.274+0000 7fc22452e640 10 client.5262.objecter successfully canceled tick 2024-03-27T21:42:03.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.274+0000 7fc22452e640 1 -- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc21c141630 msgr2=0x7fc208076c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:03.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.274+0000 7fc22452e640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc21c141630 0x7fc208076c60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc214000f60 tx=0x7fc214009040 comp rx=0 tx=0).stop 2024-03-27T21:42:03.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.274+0000 7fc22452e640 10 monclient: shutdown 2024-03-27T21:42:03.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.274+0000 7fc22452e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:03.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.274+0000 7fc22452e640 1 -- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc208005f20 msgr2=0x7fc2080083b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:03.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.274+0000 7fc22452e640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc208005f20 0x7fc2080083b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc210033000 tx=0x7fc20c00b040 comp rx=0 tx=0).stop 2024-03-27T21:42:03.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.274+0000 7fc22452e640 1 -- 172.21.15.60:0/4180533158 shutdown_connections 2024-03-27T21:42:03.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.274+0000 7fc22452e640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc21c13ef20 0x7fc20800af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:03.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.274+0000 7fc22452e640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc208005f20 0x7fc2080083b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:03.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.274+0000 7fc22452e640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc21c141630 0x7fc208076c60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:03.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.274+0000 7fc22452e640 1 --2- 172.21.15.60:0/4180533158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc21c071a50 0x7fc2080054e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:03.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.274+0000 7fc22452e640 1 -- 172.21.15.60:0/4180533158 >> 172.21.15.60:0/4180533158 conn(0x7fc21c0b7300 msgr2=0x7fc21c13e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:03.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.274+0000 7fc22452e640 1 -- 172.21.15.60:0/4180533158 shutdown_connections 2024-03-27T21:42:03.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.275+0000 7fc22452e640 1 -- 172.21.15.60:0/4180533158 wait complete. 2024-03-27T21:42:03.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.275+0000 7fc22452e640 1 librados: shutdown 2024-03-27T21:42:03.274 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-27T21:42:03.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.396+0000 7fe20ada0640 10 monclient: get_monmap_and_config 2024-03-27T21:42:03.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.396+0000 7fe20ada0640 10 monclient: build_initial_monmap 2024-03-27T21:42:03.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.397+0000 7fe20ada0640 10 monclient: monmap: 2024-03-27T21:42:03.383 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:03.383 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:03.383 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:03.397989+0000 2024-03-27T21:42:03.383 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:03.397989+0000 2024-03-27T21:42:03.384 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:03.384 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:03.384 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:03.384 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:03.384 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:03.384 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:03.384 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:03.384 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:03.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.397+0000 7fe20ada0640 1 Processor -- start 2024-03-27T21:42:03.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.397+0000 7fe20ada0640 1 -- start start 2024-03-27T21:42:03.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.397+0000 7fe20ada0640 10 monclient: init 2024-03-27T21:42:03.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20ada0640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:03.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20ada0640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-03-27T21:42:03.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20ada0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe2040b4120 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:03.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20ada0640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe2040b13b0 0x7fe204144b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:03.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20ada0640 10 monclient(hunting): picked mon.noname-c con 0x7fe2040b13b0 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:03.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20ada0640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe20413f020 0x7fe2041450c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:03.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20ada0640 10 monclient(hunting): picked mon.noname-b con 0x7fe20413f020 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:03.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20ada0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2040a9390 con 0x7fe2040b4120 2024-03-27T21:42:03.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20ada0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:03.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20ada0640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2040b4500 con 0x7fe2040b13b0 2024-03-27T21:42:03.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20ada0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:03.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20ada0640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2040b3610 con 0x7fe20413f020 2024-03-27T21:42:03.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20ada0640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:03.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20ada0640 10 monclient(hunting): authenticate will time out at 964.619323s 2024-03-27T21:42:03.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20a59f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe20413f020 0x7fe2041450c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:03.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20a59f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe20413f020 0x7fe2041450c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34030/0 (socket says 172.21.15.60:34030) 2024-03-27T21:42:03.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20a59f640 1 -- 172.21.15.60:0/2187067898 learned_addr learned my addr 172.21.15.60:0/2187067898 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:03.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20a59f640 10 monclient(hunting): get_auth_request con 0x7fe20413f020 auth_method 0 2024-03-27T21:42:03.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20a59f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:03.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20a59f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:03.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20a59f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:03.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20a59f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:03.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20a59f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:03.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20a59f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:03.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20959d640 1 --2- 172.21.15.60:0/2187067898 >> v2:172.21.15.191:3300/0 conn(0x7fe2040b13b0 0x7fe204144b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:03.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20a59f640 10 monclient(hunting): handle_auth_done global_id 4991 payload 274 2024-03-27T21:42:03.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20a59f640 1 -- 172.21.15.60:0/2187067898 >> v2:172.21.15.191:3300/0 conn(0x7fe2040b13b0 msgr2=0x7fe204144b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:03.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20a59f640 1 --2- 172.21.15.60:0/2187067898 >> v2:172.21.15.191:3300/0 conn(0x7fe2040b13b0 0x7fe204144b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:03.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20a59f640 1 -- 172.21.15.60:0/2187067898 >> v1:172.21.15.191:6789/0 conn(0x7fe2040b4120 legacy=0x7fe20413e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:03.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20a59f640 10 monclient: _finish_hunting 0 2024-03-27T21:42:03.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20a59f640 1 monclient: found mon.noname-b 2024-03-27T21:42:03.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20a59f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:03.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20a59f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:42:03.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20a59f640 1 -- 172.21.15.60:0/2187067898 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2040b3780 con 0x7fe20413f020 2024-03-27T21:42:03.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20959d640 10 monclient: get_auth_request con 0x7fe2040b13b0 auth_method 0 2024-03-27T21:42:03.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.398+0000 7fe20959d640 1 --2- 172.21.15.60:0/2187067898 >> v2:172.21.15.191:3300/0 conn(0x7fe2040b13b0 0x7fe204144b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:42:03.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.399+0000 7fe20a59f640 1 --2- 172.21.15.60:0/2187067898 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe20413f020 0x7fe2041450c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4009870 tx=0x7fe1f402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33e5f4992ca3b7c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:03.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.399+0000 7fe208d9c640 1 -- 172.21.15.60:0/2187067898 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f4031640 con 0x7fe20413f020 2024-03-27T21:42:03.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.399+0000 7fe208d9c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:03.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.399+0000 7fe208d9c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:03.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.399+0000 7fe208d9c640 10 monclient: dump: 2024-03-27T21:42:03.398 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:03.398 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:03.398 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:03.398 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:03.398 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:03.398 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:03.398 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:03.399 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:03.399 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:03.399 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:03.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.399+0000 7fe208d9c640 10 monclient: _finish_auth 0 2024-03-27T21:42:03.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.399+0000 7fe208d9c640 10 monclient: _check_auth_tickets 2024-03-27T21:42:03.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.399+0000 7fe208d9c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:03.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.399+0000 7fe208d9c640 1 -- 172.21.15.60:0/2187067898 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f40317a0 con 0x7fe20413f020 2024-03-27T21:42:03.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.399+0000 7fe20ada0640 5 monclient: authenticate success, global_id 4991 2024-03-27T21:42:03.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.399+0000 7fe20ada0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:03.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.399+0000 7fe208d9c640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:03.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.399+0000 7fe20ada0640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:03.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.399+0000 7fe20ada0640 4 set_mon_vals no callback set 2024-03-27T21:42:03.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.399+0000 7fe208d9c640 1 -- 172.21.15.60:0/2187067898 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f4031b30 con 0x7fe20413f020 2024-03-27T21:42:03.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.399+0000 7fe208d9c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:03.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.399+0000 7fe208d9c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:03.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.399+0000 7fe208d9c640 10 monclient: dump: 2024-03-27T21:42:03.400 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:03.401 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:03.401 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:03.401 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:03.401 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:03.401 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:03.401 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:03.401 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:03.401 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:03.401 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:03.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.400+0000 7fe20ada0640 10 monclient: shutdown 2024-03-27T21:42:03.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.400+0000 7fe20ada0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:03.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.400+0000 7fe20ada0640 1 -- 172.21.15.60:0/2187067898 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe20413f020 msgr2=0x7fe2041450c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:03.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.400+0000 7fe20ada0640 1 --2- 172.21.15.60:0/2187067898 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe20413f020 0x7fe2041450c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4009870 tx=0x7fe1f402b8f0 comp rx=0 tx=0).stop 2024-03-27T21:42:03.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.400+0000 7fe20ada0640 1 -- 172.21.15.60:0/2187067898 shutdown_connections 2024-03-27T21:42:03.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.400+0000 7fe20ada0640 1 --2- 172.21.15.60:0/2187067898 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe20413f020 0x7fe2041450c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:03.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.400+0000 7fe20ada0640 1 --2- 172.21.15.60:0/2187067898 >> v2:172.21.15.191:3300/0 conn(0x7fe2040b13b0 0x7fe204144b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:03.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.400+0000 7fe20ada0640 1 -- 172.21.15.60:0/2187067898 >> 172.21.15.60:0/2187067898 conn(0x7fe204060b20 msgr2=0x7fe204060f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:03.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.400+0000 7fe20ada0640 1 -- 172.21.15.60:0/2187067898 shutdown_connections 2024-03-27T21:42:03.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.400+0000 7fe20ada0640 1 -- 172.21.15.60:0/2187067898 wait complete. 2024-03-27T21:42:03.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.400+0000 7fe20ada0640 10 monclient: build_initial_monmap 2024-03-27T21:42:03.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.400+0000 7fe20ada0640 10 monclient: monmap: 2024-03-27T21:42:03.403 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:03.403 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:03.403 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:03.403 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:03.404 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:03.404 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:03.404 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:03.404 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:03.404 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:03.404 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:03.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 1 librados: starting msgr at 2024-03-27T21:42:03.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 1 librados: starting objecter 2024-03-27T21:42:03.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 1 Processor -- start 2024-03-27T21:42:03.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 1 -- start start 2024-03-27T21:42:03.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 1 librados: setting wanted keys 2024-03-27T21:42:03.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 1 librados: calling monclient init 2024-03-27T21:42:03.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 10 monclient: init 2024-03-27T21:42:03.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:03.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:42:03.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe2040b13b0 0x7fe2041462d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:03.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 10 monclient(hunting): picked mon.noname-c con 0x7fe2040b13b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:03.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2040b4120 0x7fe204146db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:03.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 10 monclient(hunting): picked mon.noname-b con 0x7fe2040b4120 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:03.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe20413f020 0x7fe204138130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:03.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 10 monclient(hunting): picked mon.noname-a con 0x7fe20413f020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:03.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:03.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2041472f0 con 0x7fe20413f020 2024-03-27T21:42:03.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:03.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2041d9750 con 0x7fe2040b4120 2024-03-27T21:42:03.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:03.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2041416b0 con 0x7fe2040b13b0 2024-03-27T21:42:03.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:03.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.401+0000 7fe20ada0640 10 monclient(hunting): authenticate will time out at 964.622876s 2024-03-27T21:42:03.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe209d9e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe2040b13b0 0x7fe2041462d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:03.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe209d9e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe2040b13b0 0x7fe2041462d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34034/0 (socket says 172.21.15.60:34034) 2024-03-27T21:42:03.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe209d9e640 1 -- 172.21.15.60:0/3095526406 learned_addr learned my addr 172.21.15.60:0/3095526406 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:03.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe209d9e640 10 monclient(hunting): get_auth_request con 0x7fe2040b13b0 auth_method 0 2024-03-27T21:42:03.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe209d9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:03.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe209d9e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:03.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe209d9e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:03.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20a59f640 1 --2- 172.21.15.60:0/3095526406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe20413f020 0x7fe204138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:03.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20a59f640 10 monclient(hunting): get_auth_request con 0x7fe20413f020 auth_method 0 2024-03-27T21:42:03.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20a59f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:03.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20a59f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:03.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20a59f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:03.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe209d9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:03.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe209d9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:03.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe209d9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:03.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20959d640 1 --2- 172.21.15.60:0/3095526406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2040b4120 0x7fe204146db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:03.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20959d640 10 monclient(hunting): get_auth_request con 0x7fe2040b4120 auth_method 0 2024-03-27T21:42:03.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20959d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:03.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20959d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:03.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20959d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:03.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20a59f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:03.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20a59f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:03.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20a59f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:03.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20959d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:03.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20959d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:03.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20959d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:03.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe209d9e640 10 monclient(hunting): handle_auth_done global_id 4994 payload 824 2024-03-27T21:42:03.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe209d9e640 1 -- 172.21.15.60:0/3095526406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2040b4120 msgr2=0x7fe204146db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:03.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe209d9e640 1 --2- 172.21.15.60:0/3095526406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2040b4120 0x7fe204146db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:03.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe209d9e640 1 -- 172.21.15.60:0/3095526406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe20413f020 msgr2=0x7fe204138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:03.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe209d9e640 1 --2- 172.21.15.60:0/3095526406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe20413f020 0x7fe204138130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:03.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe209d9e640 10 monclient: _finish_hunting 0 2024-03-27T21:42:03.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe209d9e640 1 monclient: found mon.noname-c 2024-03-27T21:42:03.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe209d9e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:03.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe209d9e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:03.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe209d9e640 1 -- 172.21.15.60:0/3095526406 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f40094d0 con 0x7fe2040b13b0 2024-03-27T21:42:03.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20a59f640 1 --2- 172.21.15.60:0/3095526406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe20413f020 0x7fe204138130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:03.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe209d9e640 1 --2- 172.21.15.60:0/3095526406 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe2040b13b0 0x7fe2041462d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc00c7f0 tx=0x7fe1fc00ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:03.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe2027fc640 10 client.?.objecter ms_handle_connect 0x7fe2040b13b0 2024-03-27T21:42:03.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe2027fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:03.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe2027fc640 1 -- 172.21.15.60:0/3095526406 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc00aa50 con 0x7fe2040b13b0 2024-03-27T21:42:03.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe2027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:03.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe2027fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:03.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe2027fc640 10 monclient: dump: 2024-03-27T21:42:03.412 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:03.413 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:03.413 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:03.413 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:03.413 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:03.413 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:03.413 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:03.413 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:03.413 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:03.413 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:03.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe2027fc640 10 monclient: _finish_auth 0 2024-03-27T21:42:03.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe2027fc640 10 monclient: _check_auth_tickets 2024-03-27T21:42:03.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe2027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:03.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20ada0640 5 monclient: authenticate success, global_id 4994 2024-03-27T21:42:03.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe2027fc640 1 -- 172.21.15.60:0/3095526406 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1fc00abb0 con 0x7fe2040b13b0 2024-03-27T21:42:03.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20ada0640 10 monclient: _renew_subs 2024-03-27T21:42:03.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20ada0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:03.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20ada0640 1 -- 172.21.15.60:0/3095526406 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2041458a0 con 0x7fe2040b13b0 2024-03-27T21:42:03.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe2027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:03.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe2027fc640 1 -- 172.21.15.60:0/3095526406 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc012420 con 0x7fe2040b13b0 2024-03-27T21:42:03.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe2027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:03.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe2027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:03.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe2027fc640 10 monclient: dump: 2024-03-27T21:42:03.415 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:03.415 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:03.415 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:03.415 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:03.416 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:03.416 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:03.416 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:03.416 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:03.416 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:03.416 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:03.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20ada0640 10 client.4994.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:03.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20ada0640 10 monclient: _renew_subs 2024-03-27T21:42:03.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20ada0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:03.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe2037fe640 4 set_mon_vals no callback set 2024-03-27T21:42:03.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20ada0640 1 -- 172.21.15.60:0/3095526406 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe204145dd0 con 0x7fe2040b13b0 2024-03-27T21:42:03.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.402+0000 7fe20ada0640 1 librados: init done 2024-03-27T21:42:03.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.403+0000 7fe20ada0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:03.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.403+0000 7fe20ada0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:03.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.403+0000 7fe20ada0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:03.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.403+0000 7fe20ada0640 1 -- 172.21.15.60:0/3095526406 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe204067280 con 0x7fe2040b13b0 2024-03-27T21:42:03.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.404+0000 7fe2027fc640 1 -- 172.21.15.60:0/3095526406 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe1fc012580 con 0x7fe2040b13b0 2024-03-27T21:42:03.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.404+0000 7fe2027fc640 1 --2- 172.21.15.60:0/3095526406 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe1f006c5c0 0x7fe1f006ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:03.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.404+0000 7fe2027fc640 1 -- 172.21.15.60:0/3095526406 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fe1fc089510 con 0x7fe2040b13b0 2024-03-27T21:42:03.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.404+0000 7fe2027fc640 10 client.4994.objecter ms_dispatch 0x7fe204000b90 osd_map(91..91 src has 1..91) v4 2024-03-27T21:42:03.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.404+0000 7fe2027fc640 3 client.4994.objecter handle_osd_map got epochs [91,91] > 0 2024-03-27T21:42:03.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.404+0000 7fe2027fc640 3 client.4994.objecter handle_osd_map decoding full epoch 91 2024-03-27T21:42:03.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.404+0000 7fe20959d640 1 --2- 172.21.15.60:0/3095526406 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe1f006c5c0 0x7fe1f006ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:03.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.404+0000 7fe20959d640 10 monclient: get_auth_request con 0x7fe1f006c5c0 auth_method 0 2024-03-27T21:42:03.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.404+0000 7fe2027fc640 20 client.4994.objecter dump_active .. 0 homeless 2024-03-27T21:42:03.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.405+0000 7fe20959d640 1 --2- 172.21.15.60:0/3095526406 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe1f006c5c0 0x7fe1f006ea80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe204069440 tx=0x7fe1f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:03.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.405+0000 7fe2027fc640 10 client.4994.objecter ms_handle_connect 0x7fe1f006c5c0 2024-03-27T21:42:03.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.408+0000 7fe2027fc640 1 -- 172.21.15.60:0/3095526406 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe1fc054a00 con 0x7fe2040b13b0 2024-03-27T21:42:03.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.408+0000 7fe2027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:03.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.408+0000 7fe2027fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:03.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.598+0000 7fe20ada0640 10 start_mon_command cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}] 2024-03-27T21:42:03.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.598+0000 7fe20ada0640 10 monclient: _send_command 2 [{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}] 2024-03-27T21:42:03.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.598+0000 7fe20ada0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:03.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:03.598+0000 7fe20ada0640 1 -- 172.21.15.60:0/3095526406 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} v 0) v1 -- 0x7fe204069470 con 0x7fe2040b13b0 2024-03-27T21:42:04.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.280+0000 7fe2027fc640 1 -- 172.21.15.60:0/3095526406 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]=0 set osd.2 primary-affinity to 1 (8655362) v92) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe1fc0160a0 con 0x7fe2040b13b0 2024-03-27T21:42:04.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.280+0000 7fe2027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}] 2024-03-27T21:42:04.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.280+0000 7fe2027fc640 10 monclient: _finish_command 2 = system:0 set osd.2 primary-affinity to 1 (8655362) 2024-03-27T21:42:04.267 INFO:teuthology.orchestra.run.smithi060.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-27T21:42:04.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.282+0000 7fe20ada0640 10 librados: watch_flush enter 2024-03-27T21:42:04.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.282+0000 7fe20ada0640 10 librados: watch_flush exit 2024-03-27T21:42:04.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.283+0000 7fe20ada0640 20 client.4994.objecter shutdown clearing up homeless session... 2024-03-27T21:42:04.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.283+0000 7fe20ada0640 10 client.4994.objecter successfully canceled tick 2024-03-27T21:42:04.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.283+0000 7fe20ada0640 1 -- 172.21.15.60:0/3095526406 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe1f006c5c0 msgr2=0x7fe1f006ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:04.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.283+0000 7fe20ada0640 1 --2- 172.21.15.60:0/3095526406 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe1f006c5c0 0x7fe1f006ea80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe204069440 tx=0x7fe1f8008040 comp rx=0 tx=0).stop 2024-03-27T21:42:04.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.283+0000 7fe20ada0640 10 monclient: shutdown 2024-03-27T21:42:04.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.283+0000 7fe20ada0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:04.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.283+0000 7fe20ada0640 1 -- 172.21.15.60:0/3095526406 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe2040b13b0 msgr2=0x7fe2041462d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:04.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.283+0000 7fe20ada0640 1 --2- 172.21.15.60:0/3095526406 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe2040b13b0 0x7fe2041462d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc00c7f0 tx=0x7fe1fc00ccb0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.283+0000 7fe20ada0640 1 -- 172.21.15.60:0/3095526406 shutdown_connections 2024-03-27T21:42:04.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.283+0000 7fe20ada0640 1 --2- 172.21.15.60:0/3095526406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe20413f020 0x7fe204138130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.283+0000 7fe20ada0640 1 --2- 172.21.15.60:0/3095526406 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe1f006c5c0 0x7fe1f006ea80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.283+0000 7fe20ada0640 1 --2- 172.21.15.60:0/3095526406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2040b4120 0x7fe204146db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.283+0000 7fe20ada0640 1 --2- 172.21.15.60:0/3095526406 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fe2040b13b0 0x7fe2041462d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.283+0000 7fe20ada0640 1 -- 172.21.15.60:0/3095526406 >> 172.21.15.60:0/3095526406 conn(0x7fe204060b20 msgr2=0x7fe20413ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:04.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.283+0000 7fe20ada0640 1 -- 172.21.15.60:0/3095526406 shutdown_connections 2024-03-27T21:42:04.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.283+0000 7fe20ada0640 1 -- 172.21.15.60:0/3095526406 wait complete. 2024-03-27T21:42:04.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.283+0000 7fe20ada0640 1 librados: shutdown 2024-03-27T21:42:04.283 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-27T21:42:04.283 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-27T21:42:04.283 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-27T21:42:04.283 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-27T21:42:04.283 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-27T21:42:04.283 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T21:42:04.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.403+0000 7fd7d0d55640 10 monclient: get_monmap_and_config 2024-03-27T21:42:04.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.403+0000 7fd7d0d55640 10 monclient: build_initial_monmap 2024-03-27T21:42:04.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.403+0000 7fd7d0d55640 10 monclient: monmap: 2024-03-27T21:42:04.390 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:04.390 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:04.390 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:04.404647+0000 2024-03-27T21:42:04.390 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:04.404647+0000 2024-03-27T21:42:04.390 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:04.390 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:04.391 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:04.391 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:04.391 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:04.391 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:04.391 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:04.391 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:04.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.404+0000 7fd7d0d55640 1 Processor -- start 2024-03-27T21:42:04.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.404+0000 7fd7d0d55640 1 -- start start 2024-03-27T21:42:04.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.404+0000 7fd7d0d55640 10 monclient: init 2024-03-27T21:42:04.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.404+0000 7fd7d0d55640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:04.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.404+0000 7fd7d0d55640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-03-27T21:42:04.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.404+0000 7fd7d0d55640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd7cc071a50 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:04.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.404+0000 7fd7d0d55640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd7cc13ef20 0x7fd7cc0a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:04.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.404+0000 7fd7d0d55640 10 monclient(hunting): picked mon.noname-a con 0x7fd7cc13ef20 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:04.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.404+0000 7fd7d0d55640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd7cc141630 0x7fd7cc141ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:04.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.404+0000 7fd7d0d55640 10 monclient(hunting): picked mon.noname-c con 0x7fd7cc141630 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:04.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7d0d55640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7cc0a8e90 con 0x7fd7cc071a50 2024-03-27T21:42:04.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7d0d55640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:04.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7d0d55640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7cc071e30 con 0x7fd7cc13ef20 2024-03-27T21:42:04.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7d0d55640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:04.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7d0d55640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7cc0aa040 con 0x7fd7cc141630 2024-03-27T21:42:04.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7d0d55640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:04.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7d0d55640 10 monclient(hunting): authenticate will time out at 965.625938s 2024-03-27T21:42:04.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7c3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd7cc13ef20 0x7fd7cc0a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:04.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7c3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd7cc13ef20 0x7fd7cc0a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39176/0 (socket says 172.21.15.60:39176) 2024-03-27T21:42:04.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7c3fff640 1 -- 172.21.15.60:0/1216902892 learned_addr learned my addr 172.21.15.60:0/1216902892 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:04.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7c3fff640 10 monclient(hunting): get_auth_request con 0x7fd7cc13ef20 auth_method 0 2024-03-27T21:42:04.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:04.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:04.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:04.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:04.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:04.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:04.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7c3fff640 10 monclient(hunting): handle_auth_done global_id 5268 payload 274 2024-03-27T21:42:04.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7c3fff640 1 -- 172.21.15.60:0/1216902892 >> v2:172.21.15.191:3300/0 conn(0x7fd7cc141630 msgr2=0x7fd7cc141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:04.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7cad9d640 1 --2- 172.21.15.60:0/1216902892 >> v2:172.21.15.191:3300/0 conn(0x7fd7cc141630 0x7fd7cc141ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:04.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7c3fff640 1 --2- 172.21.15.60:0/1216902892 >> v2:172.21.15.191:3300/0 conn(0x7fd7cc141630 0x7fd7cc141ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7c3fff640 1 -- 172.21.15.60:0/1216902892 >> v1:172.21.15.191:6789/0 conn(0x7fd7cc071a50 legacy=0x7fd7cc13e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:04.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7c3fff640 10 monclient: _finish_hunting 0 2024-03-27T21:42:04.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7c3fff640 1 monclient: found mon.noname-a 2024-03-27T21:42:04.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:04.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7c3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:04.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.405+0000 7fd7c3fff640 1 -- 172.21.15.60:0/1216902892 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7cc0aa2b0 con 0x7fd7cc13ef20 2024-03-27T21:42:04.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7c3fff640 1 --2- 172.21.15.60:0/1216902892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7cc13ef20 0x7fd7cc0a87e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc0a91e0 tx=0x7fd7b402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53b74ca87f37bead server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:04.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7c9d9b640 1 -- 172.21.15.60:0/1216902892 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4031870 con 0x7fd7cc13ef20 2024-03-27T21:42:04.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:04.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7c9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:04.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7c9d9b640 10 monclient: dump: 2024-03-27T21:42:04.405 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:04.405 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:04.405 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:04.405 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:04.405 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:04.405 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:04.405 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:04.405 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:04.406 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:04.406 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:04.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7c9d9b640 10 monclient: _finish_auth 0 2024-03-27T21:42:04.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7c9d9b640 10 monclient: _check_auth_tickets 2024-03-27T21:42:04.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7c9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:04.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7c9d9b640 1 -- 172.21.15.60:0/1216902892 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b40319d0 con 0x7fd7cc13ef20 2024-03-27T21:42:04.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7d0d55640 5 monclient: authenticate success, global_id 5268 2024-03-27T21:42:04.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7d0d55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:04.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7c9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:04.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7d0d55640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:04.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7c9d9b640 1 -- 172.21.15.60:0/1216902892 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4031d60 con 0x7fd7cc13ef20 2024-03-27T21:42:04.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7d0d55640 4 set_mon_vals no callback set 2024-03-27T21:42:04.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:04.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7c9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:04.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7c9d9b640 10 monclient: dump: 2024-03-27T21:42:04.407 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:04.407 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:04.408 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:04.408 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:04.408 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:04.408 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:04.408 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:04.408 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:04.408 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:04.408 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:04.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7d0d55640 10 monclient: shutdown 2024-03-27T21:42:04.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7d0d55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:04.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7d0d55640 1 -- 172.21.15.60:0/1216902892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7cc13ef20 msgr2=0x7fd7cc0a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:04.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7d0d55640 1 --2- 172.21.15.60:0/1216902892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7cc13ef20 0x7fd7cc0a87e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc0a91e0 tx=0x7fd7b402b910 comp rx=0 tx=0).stop 2024-03-27T21:42:04.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7d0d55640 1 -- 172.21.15.60:0/1216902892 shutdown_connections 2024-03-27T21:42:04.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7d0d55640 1 --2- 172.21.15.60:0/1216902892 >> v2:172.21.15.191:3300/0 conn(0x7fd7cc141630 0x7fd7cc141ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7d0d55640 1 --2- 172.21.15.60:0/1216902892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7cc13ef20 0x7fd7cc0a87e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7d0d55640 1 -- 172.21.15.60:0/1216902892 >> 172.21.15.60:0/1216902892 conn(0x7fd7cc0b7300 msgr2=0x7fd7cc0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:04.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.406+0000 7fd7d0d55640 1 -- 172.21.15.60:0/1216902892 shutdown_connections 2024-03-27T21:42:04.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.407+0000 7fd7d0d55640 1 -- 172.21.15.60:0/1216902892 wait complete. 2024-03-27T21:42:04.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.407+0000 7fd7d0d55640 10 monclient: build_initial_monmap 2024-03-27T21:42:04.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.407+0000 7fd7d0d55640 10 monclient: monmap: 2024-03-27T21:42:04.410 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:04.410 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:04.410 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:04.410 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:04.410 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:04.410 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:04.411 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:04.411 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:04.411 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:04.411 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:04.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 1 librados: starting msgr at 2024-03-27T21:42:04.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 1 librados: starting objecter 2024-03-27T21:42:04.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 1 Processor -- start 2024-03-27T21:42:04.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 1 -- start start 2024-03-27T21:42:04.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 1 librados: setting wanted keys 2024-03-27T21:42:04.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 1 librados: calling monclient init 2024-03-27T21:42:04.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 10 monclient: init 2024-03-27T21:42:04.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:04.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:42:04.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7cc071a50 0x7fd7cc139d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:04.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 10 monclient(hunting): picked mon.noname-b con 0x7fd7cc071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:04.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd7cc13ef20 0x7fd7cc0abb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:04.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 10 monclient(hunting): picked mon.noname-c con 0x7fd7cc13ef20 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:04.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7cc141630 0x7fd7cc13a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:04.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 10 monclient(hunting): picked mon.noname-a con 0x7fd7cc141630 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:04.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:04.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc0ac050 con 0x7fd7cc141630 2024-03-27T21:42:04.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:04.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc0ab720 con 0x7fd7cc071a50 2024-03-27T21:42:04.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:04.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc138ee0 con 0x7fd7cc13ef20 2024-03-27T21:42:04.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:04.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7d0d55640 10 monclient(hunting): authenticate will time out at 965.629657s 2024-03-27T21:42:04.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7cad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7cc141630 0x7fd7cc13a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:04.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7c3fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd7cc13ef20 0x7fd7cc0abb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:04.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7cad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7cc141630 0x7fd7cc13a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39182/0 (socket says 172.21.15.60:39182) 2024-03-27T21:42:04.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7cad9d640 1 -- 172.21.15.60:0/2076312905 learned_addr learned my addr 172.21.15.60:0/2076312905 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:04.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7cad9d640 10 monclient(hunting): get_auth_request con 0x7fd7cc141630 auth_method 0 2024-03-27T21:42:04.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:04.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7cad9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:04.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:04.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7c3fff640 10 monclient(hunting): get_auth_request con 0x7fd7cc13ef20 auth_method 0 2024-03-27T21:42:04.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:04.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:04.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:04.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:04.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:04.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:04.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:04.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:04.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.408+0000 7fd7c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:04.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7ca59c640 1 --2- 172.21.15.60:0/2076312905 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7cc071a50 0x7fd7cc139d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:04.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7ca59c640 10 monclient(hunting): get_auth_request con 0x7fd7cc071a50 auth_method 0 2024-03-27T21:42:04.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:04.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7ca59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:04.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:04.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:04.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:04.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:04.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7cad9d640 10 monclient(hunting): handle_auth_done global_id 5271 payload 824 2024-03-27T21:42:04.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7cad9d640 1 -- 172.21.15.60:0/2076312905 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd7cc13ef20 msgr2=0x7fd7cc0abb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:04.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7cad9d640 1 --2- 172.21.15.60:0/2076312905 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd7cc13ef20 0x7fd7cc0abb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7cad9d640 1 -- 172.21.15.60:0/2076312905 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7cc071a50 msgr2=0x7fd7cc139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:04.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7cad9d640 1 --2- 172.21.15.60:0/2076312905 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7cc071a50 0x7fd7cc139d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7cad9d640 10 monclient: _finish_hunting 0 2024-03-27T21:42:04.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7cad9d640 1 monclient: found mon.noname-a 2024-03-27T21:42:04.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7cad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:04.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7cad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:04.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7cad9d640 1 -- 172.21.15.60:0/2076312905 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b40094d0 con 0x7fd7cc141630 2024-03-27T21:42:04.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7c3fff640 1 --2- 172.21.15.60:0/2076312905 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd7cc13ef20 0x7fd7cc0abb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:04.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7cad9d640 1 --2- 172.21.15.60:0/2076312905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7cc141630 0x7fd7cc13a8c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd7c400cc30 tx=0x7fd7c400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:04.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7c2ffd640 10 client.?.objecter ms_handle_connect 0x7fd7cc141630 2024-03-27T21:42:04.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:04.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7c2ffd640 1 -- 172.21.15.60:0/2076312905 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c400ce40 con 0x7fd7cc141630 2024-03-27T21:42:04.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:04.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7c2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:04.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7c2ffd640 10 monclient: dump: 2024-03-27T21:42:04.419 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:04.419 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:04.419 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:04.419 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:04.419 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:04.419 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:04.419 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:04.420 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:04.420 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:04.420 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:04.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7c2ffd640 10 monclient: _finish_auth 0 2024-03-27T21:42:04.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7c2ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:04.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:04.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7c2ffd640 1 -- 172.21.15.60:0/2076312905 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c4003990 con 0x7fd7cc141630 2024-03-27T21:42:04.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7d0d55640 5 monclient: authenticate success, global_id 5271 2024-03-27T21:42:04.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7d0d55640 10 monclient: _renew_subs 2024-03-27T21:42:04.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7d0d55640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:04.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7d0d55640 1 -- 172.21.15.60:0/2076312905 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7cc13a3a0 con 0x7fd7cc141630 2024-03-27T21:42:04.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:04.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7c2ffd640 1 -- 172.21.15.60:0/2076312905 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c4003d20 con 0x7fd7cc141630 2024-03-27T21:42:04.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:04.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:04.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7c2ffd640 10 monclient: dump: 2024-03-27T21:42:04.421 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:04.421 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:04.422 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:04.422 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:04.422 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:04.422 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:04.422 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:04.422 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:04.422 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:04.422 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:04.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7d0d55640 10 client.5271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:04.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7d0d55640 10 monclient: _renew_subs 2024-03-27T21:42:04.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7c959a640 4 set_mon_vals no callback set 2024-03-27T21:42:04.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7d0d55640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:04.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7d0d55640 1 -- 172.21.15.60:0/2076312905 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7cc131cf0 con 0x7fd7cc141630 2024-03-27T21:42:04.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.409+0000 7fd7d0d55640 1 librados: init done 2024-03-27T21:42:04.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.410+0000 7fd7d0d55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:04.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.410+0000 7fd7d0d55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:04.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.410+0000 7fd7d0d55640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:04.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.410+0000 7fd7d0d55640 1 -- 172.21.15.60:0/2076312905 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7cc13a5e0 con 0x7fd7cc141630 2024-03-27T21:42:04.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.411+0000 7fd7c2ffd640 1 -- 172.21.15.60:0/2076312905 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd7c4003e80 con 0x7fd7cc141630 2024-03-27T21:42:04.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.411+0000 7fd7c2ffd640 1 --2- 172.21.15.60:0/2076312905 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd7ac06c5c0 0x7fd7ac06ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:04.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.411+0000 7fd7c2ffd640 1 -- 172.21.15.60:0/2076312905 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fd7c40893e0 con 0x7fd7cc141630 2024-03-27T21:42:04.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.411+0000 7fd7c2ffd640 10 client.5271.objecter ms_dispatch 0x7fd7cc000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:04.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.411+0000 7fd7c2ffd640 3 client.5271.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:04.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.411+0000 7fd7c2ffd640 3 client.5271.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:04.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.411+0000 7fd7ca59c640 1 --2- 172.21.15.60:0/2076312905 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd7ac06c5c0 0x7fd7ac06ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:04.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.411+0000 7fd7ca59c640 10 monclient: get_auth_request con 0x7fd7ac06c5c0 auth_method 0 2024-03-27T21:42:04.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.411+0000 7fd7c2ffd640 20 client.5271.objecter dump_active .. 0 homeless 2024-03-27T21:42:04.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.411+0000 7fd7c2ffd640 10 client.5271.objecter ms_handle_connect 0x7fd7ac06c5c0 2024-03-27T21:42:04.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.411+0000 7fd7ca59c640 1 --2- 172.21.15.60:0/2076312905 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd7ac06c5c0 0x7fd7ac06ea80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc004f60 tx=0x7fd7bc007e30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:04.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.415+0000 7fd7c2ffd640 1 -- 172.21.15.60:0/2076312905 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd7cc13a5e0 con 0x7fd7cc141630 2024-03-27T21:42:04.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.415+0000 7fd7c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:04.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.415+0000 7fd7c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:04.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.596+0000 7fd7d0d55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:42:04.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.596+0000 7fd7d0d55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:42:04.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.596+0000 7fd7d0d55640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:04.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.596+0000 7fd7d0d55640 1 -- 172.21.15.60:0/2076312905 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7cc13a5e0 con 0x7fd7cc141630 2024-03-27T21:42:04.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.597+0000 7fd7c2ffd640 1 -- 172.21.15.60:0/2076312905 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+14175 (secure 0 0 0) 0x7fd7cc13a5e0 con 0x7fd7cc141630 2024-03-27T21:42:04.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.597+0000 7fd7c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:42:04.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.597+0000 7fd7c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:42:04.584 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:42:04.584 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":92,"fsid":"dcd19d9a-535c-4b41-9095-d588268e2860","created":"2024-03-27T21:39:27.092191+0000","modified":"2024-03-27T21:42:04.274002+0000","last_up_change":"2024-03-27T21:41:29.048228+0000","last_in_change":"2024-03-27T21:41:48.185818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T21:39:37.438493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-27T21:39:45.042103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"778aaa64-6720-495b-bcdc-86fbcece9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":34603},{"type":"v1","addr":"172.21.15.60:6817","nonce":34603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":34603},{"type":"v1","addr":"172.21.15.60:6819","nonce":34603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":34603},{"type":"v1","addr":"172.21.15.60:6823","nonce":34603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":34603},{"type":"v1","addr":"172.21.15.60:6821","nonce":34603}]},"public_addr":"172.21.15.60:6817/34603","cluster_addr":"172.21.15.60:6819/34603","heartbeat_back_addr":"172.21.15.60:6823/34603","heartbeat_front_addr":"172.21.15.60:6821/34603","state":["exists","up"]},{"osd":1,"uuid":"d960adf1-3d83-4e24-a612-8df53fa43d0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":34602},{"type":"v1","addr":"172.21.15.60:6801","nonce":34602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":34602},{"type":"v1","addr":"172.21.15.60:6803","nonce":34602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":34602},{"type":"v1","addr":"172.21.15.60:6807","nonce":34602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":34602},{"type":"v1","addr":"172.21.15.60:6805","nonce":34602}]},"public_addr":"172.21.15.60:6801/34602","cluster_addr":"172.21.15.60:6803/34602","heartbeat_back_addr":"172.21.15.60:6807/34602","heartbeat_front_addr":"172.21.15.60:6805/34602","state":["exists","up"]},{"osd":2,"uuid":"b8ffc48b-cf6f-4701-a6aa-c25fdb0f7977","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":33,"last_clean_end":44,"up_from":62,"up_thru":62,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":56434},{"type":"v1","addr":"172.21.15.60:6825","nonce":56434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":56434},{"type":"v1","addr":"172.21.15.60:6827","nonce":56434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":56434},{"type":"v1","addr":"172.21.15.60:6831","nonce":56434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":56434},{"type":"v1","addr":"172.21.15.60:6829","nonce":56434}]},"public_addr":"172.21.15.60:6825/56434","cluster_addr":"172.21.15.60:6827/56434","heartbeat_back_addr":"172.21.15.60:6831/56434","heartbeat_front_addr":"172.21.15.60:6829/56434","state":["exists","up"]},{"osd":3,"uuid":"0f14f2d8-b271-48e4-a198-aa5279a01c06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":34613},{"type":"v1","addr":"172.21.15.60:6809","nonce":34613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":34613},{"type":"v1","addr":"172.21.15.60:6811","nonce":34613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":34613},{"type":"v1","addr":"172.21.15.60:6815","nonce":34613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":34613},{"type":"v1","addr":"172.21.15.60:6813","nonce":34613}]},"public_addr":"172.21.15.60:6809/34613","cluster_addr":"172.21.15.60:6811/34613","heartbeat_back_addr":"172.21.15.60:6815/34613","heartbeat_front_addr":"172.21.15.60:6813/34613","state":["exists","up"]},{"osd":4,"uuid":"8cdeb7ae-1c6e-4d0d-ba44-80248eff2fb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":34332},{"type":"v1","addr":"172.21.15.191:6801","nonce":34332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":34332},{"type":"v1","addr":"172.21.15.191:6803","nonce":34332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":34332},{"type":"v1","addr":"172.21.15.191:6807","nonce":34332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":34332},{"type":"v1","addr":"172.21.15.191:6805","nonce":34332}]},"public_addr":"172.21.15.191:6801/34332","cluster_addr":"172.21.15.191:6803/34332","heartbeat_back_addr":"172.21.15.191:6807/34332","heartbeat_front_addr":"172.21.15.191:6805/34332","state":["exists","up"]},{"osd":5,"uuid":"93748cf8-d3fe-46b8-950f-e01e28a5e02c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6809","nonce":34333},{"type":"v1","addr":"172.21.15.191:6811","nonce":34333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6813","nonce":34333},{"type":"v1","addr":"172.21.15.191:6816","nonce":34333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":34333},{"type":"v1","addr":"172.21.15.191:6823","nonce":34333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":34333},{"type":"v1","addr":"172.21.15.191:6820","nonce":34333}]},"public_addr":"172.21.15.191:6811/34333","cluster_addr":"172.21.15.191:6816/34333","heartbeat_back_addr":"172.21.15.191:6823/34333","heartbeat_front_addr":"172.21.15.191:6820/34333","state":["exists","up"]},{"osd":6,"uuid":"1ad21434-cbc5-4de9-8d63-76f894f7e736","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":34361},{"type":"v1","addr":"172.21.15.191:6825","nonce":34361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":34361},{"type":"v1","addr":"172.21.15.191:6827","nonce":34361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":34361},{"type":"v1","addr":"172.21.15.191:6831","nonce":34361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":34361},{"type":"v1","addr":"172.21.15.191:6829","nonce":34361}]},"public_addr":"172.21.15.191:6825/34361","cluster_addr":"172.21.15.191:6827/34361","heartbeat_back_addr":"172.21.15.191:6831/34361","heartbeat_front_addr":"172.21.15.191:6829/34361","state":["exists","up"]},{"osd":7,"uuid":"9fbd0927-ee2f-471b-8df8-c4e0fc40b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":34334},{"type":"v1","addr":"172.21.15.191:6810","nonce":34334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":34334},{"type":"v1","addr":"172.21.15.191:6814","nonce":34334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6819","nonce":34334},{"type":"v1","addr":"172.21.15.191:6821","nonce":34334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6815","nonce":34334},{"type":"v1","addr":"172.21.15.191:6817","nonce":34334}]},"public_addr":"172.21.15.191:6810/34334","cluster_addr":"172.21.15.191:6814/34334","heartbeat_back_addr":"172.21.15.191:6821/34334","heartbeat_front_addr":"172.21.15.191:6817/34334","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.476097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.411243+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.458154+0000","dead_epoch":45},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.415285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.431201+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.456013+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.474448+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.414076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T21:42:04.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.598+0000 7fd7d0d55640 10 librados: watch_flush enter 2024-03-27T21:42:04.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.598+0000 7fd7d0d55640 10 librados: watch_flush exit 2024-03-27T21:42:04.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.598+0000 7fd7d0d55640 20 client.5271.objecter shutdown clearing up homeless session... 2024-03-27T21:42:04.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.598+0000 7fd7d0d55640 10 client.5271.objecter successfully canceled tick 2024-03-27T21:42:04.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.598+0000 7fd7d0d55640 1 -- 172.21.15.60:0/2076312905 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd7ac06c5c0 msgr2=0x7fd7ac06ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:04.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.598+0000 7fd7d0d55640 1 --2- 172.21.15.60:0/2076312905 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd7ac06c5c0 0x7fd7ac06ea80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc004f60 tx=0x7fd7bc007e30 comp rx=0 tx=0).stop 2024-03-27T21:42:04.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.598+0000 7fd7d0d55640 10 monclient: shutdown 2024-03-27T21:42:04.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.598+0000 7fd7d0d55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:04.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.598+0000 7fd7d0d55640 1 -- 172.21.15.60:0/2076312905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7cc141630 msgr2=0x7fd7cc13a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:04.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.598+0000 7fd7d0d55640 1 --2- 172.21.15.60:0/2076312905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7cc141630 0x7fd7cc13a8c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd7c400cc30 tx=0x7fd7c400a5c0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.598+0000 7fd7d0d55640 1 -- 172.21.15.60:0/2076312905 shutdown_connections 2024-03-27T21:42:04.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.598+0000 7fd7d0d55640 1 --2- 172.21.15.60:0/2076312905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7cc141630 0x7fd7cc13a8c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.598+0000 7fd7d0d55640 1 --2- 172.21.15.60:0/2076312905 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd7cc13ef20 0x7fd7cc0abb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.598+0000 7fd7d0d55640 1 --2- 172.21.15.60:0/2076312905 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd7ac06c5c0 0x7fd7ac06ea80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.598+0000 7fd7d0d55640 1 --2- 172.21.15.60:0/2076312905 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7cc071a50 0x7fd7cc139d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.598+0000 7fd7d0d55640 1 -- 172.21.15.60:0/2076312905 >> 172.21.15.60:0/2076312905 conn(0x7fd7cc0b7300 msgr2=0x7fd7cc13e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:04.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.599+0000 7fd7d0d55640 1 -- 172.21.15.60:0/2076312905 shutdown_connections 2024-03-27T21:42:04.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.599+0000 7fd7d0d55640 1 -- 172.21.15.60:0/2076312905 wait complete. 2024-03-27T21:42:04.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.599+0000 7fd7d0d55640 1 librados: shutdown 2024-03-27T21:42:04.596 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-27T21:42:04.596 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T21:42:04.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.711+0000 7f56b0af7640 10 monclient: get_monmap_and_config 2024-03-27T21:42:04.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.711+0000 7f56b0af7640 10 monclient: build_initial_monmap 2024-03-27T21:42:04.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.711+0000 7f56b0af7640 10 monclient: monmap: 2024-03-27T21:42:04.698 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:04.698 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:04.698 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:04.712583+0000 2024-03-27T21:42:04.698 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:04.712583+0000 2024-03-27T21:42:04.699 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:04.699 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:04.699 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:04.699 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:04.699 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:04.700 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:04.700 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:04.700 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:04.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.712+0000 7f56b0af7640 1 Processor -- start 2024-03-27T21:42:04.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.712+0000 7f56b0af7640 1 -- start start 2024-03-27T21:42:04.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.712+0000 7f56b0af7640 10 monclient: init 2024-03-27T21:42:04.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.712+0000 7f56b0af7640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:04.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.712+0000 7f56b0af7640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-03-27T21:42:04.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.712+0000 7f56b0af7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f56ac071a50 0x7f56ac0a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:04.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.712+0000 7f56b0af7640 10 monclient(hunting): picked mon.noname-a con 0x7f56ac071a50 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:04.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.712+0000 7f56b0af7640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f56ac13e810 0x7f56ac13ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:04.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.712+0000 7f56b0af7640 10 monclient(hunting): picked mon.noname-c con 0x7f56ac13e810 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:04.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.712+0000 7f56b0af7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f56ac141630 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:04.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.712+0000 7f56b0af7640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ac13f190 con 0x7f56ac141630 2024-03-27T21:42:04.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.712+0000 7f56b0af7640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:04.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.712+0000 7f56b0af7640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56ac071e30 con 0x7f56ac071a50 2024-03-27T21:42:04.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.712+0000 7f56b0af7640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:04.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.712+0000 7f56b0af7640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56ac0a8e90 con 0x7f56ac13e810 2024-03-27T21:42:04.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.712+0000 7f56b0af7640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:04.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.712+0000 7f56b0af7640 10 monclient(hunting): authenticate will time out at 965.933869s 2024-03-27T21:42:04.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56aa59c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f56ac071a50 0x7f56ac0a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:04.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56aa59c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f56ac071a50 0x7f56ac0a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39190/0 (socket says 172.21.15.60:39190) 2024-03-27T21:42:04.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56aa59c640 1 -- 172.21.15.60:0/3810354939 learned_addr learned my addr 172.21.15.60:0/3810354939 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:04.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56aa59c640 10 monclient(hunting): get_auth_request con 0x7f56ac071a50 auth_method 0 2024-03-27T21:42:04.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:04.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56aa59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:04.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:04.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:04.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:04.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56a3fff640 1 --2- 172.21.15.60:0/3810354939 >> v2:172.21.15.191:3300/0 conn(0x7f56ac13e810 0x7f56ac13ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:04.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:04.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56a3fff640 10 monclient(hunting): get_auth_request con 0x7f56ac13e810 auth_method 0 2024-03-27T21:42:04.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:04.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:04.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:04.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:04.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:04.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:04.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56aa59c640 10 monclient(hunting): handle_auth_done global_id 5274 payload 274 2024-03-27T21:42:04.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56aa59c640 1 -- 172.21.15.60:0/3810354939 >> v2:172.21.15.191:3300/0 conn(0x7f56ac13e810 msgr2=0x7f56ac13ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:04.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56aa59c640 1 --2- 172.21.15.60:0/3810354939 >> v2:172.21.15.191:3300/0 conn(0x7f56ac13e810 0x7f56ac13ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56aa59c640 1 -- 172.21.15.60:0/3810354939 >> v1:172.21.15.191:6789/0 conn(0x7f56ac141630 legacy=0x7f56ac141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:04.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56aa59c640 10 monclient: _finish_hunting 0 2024-03-27T21:42:04.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56aa59c640 1 monclient: found mon.noname-a 2024-03-27T21:42:04.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56aa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:04.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56aa59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:04.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.713+0000 7f56aa59c640 1 -- 172.21.15.60:0/3810354939 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ac0aa040 con 0x7f56ac071a50 2024-03-27T21:42:04.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56aa59c640 1 --2- 172.21.15.60:0/3810354939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56ac071a50 0x7f56ac0a87e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0709f0 tx=0x7f569c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c69d32acda624794 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:04.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56a9d9b640 1 -- 172.21.15.60:0/3810354939 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c030860 con 0x7f56ac071a50 2024-03-27T21:42:04.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:04.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56a9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:04.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56a9d9b640 10 monclient: dump: 2024-03-27T21:42:04.723 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:04.723 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:04.724 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:04.724 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:04.724 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:04.724 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:04.724 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:04.724 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:04.724 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:04.725 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:04.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56a9d9b640 10 monclient: _finish_auth 0 2024-03-27T21:42:04.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56a9d9b640 10 monclient: _check_auth_tickets 2024-03-27T21:42:04.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:04.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56a9d9b640 1 -- 172.21.15.60:0/3810354939 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f569c0309c0 con 0x7f56ac071a50 2024-03-27T21:42:04.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56b0af7640 5 monclient: authenticate success, global_id 5274 2024-03-27T21:42:04.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56b0af7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:04.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56a9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:04.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56b0af7640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:04.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56a9d9b640 1 -- 172.21.15.60:0/3810354939 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c030d50 con 0x7f56ac071a50 2024-03-27T21:42:04.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:04.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56b0af7640 4 set_mon_vals no callback set 2024-03-27T21:42:04.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56a9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:04.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56a9d9b640 10 monclient: dump: 2024-03-27T21:42:04.727 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:04.728 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:04.728 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:04.728 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:04.728 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:04.728 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:04.728 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:04.729 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:04.729 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:04.729 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:04.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56b0af7640 10 monclient: shutdown 2024-03-27T21:42:04.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56b0af7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:04.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56b0af7640 1 -- 172.21.15.60:0/3810354939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56ac071a50 msgr2=0x7f56ac0a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:04.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56b0af7640 1 --2- 172.21.15.60:0/3810354939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56ac071a50 0x7f56ac0a87e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0709f0 tx=0x7f569c02b9a0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56b0af7640 1 -- 172.21.15.60:0/3810354939 shutdown_connections 2024-03-27T21:42:04.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56b0af7640 1 --2- 172.21.15.60:0/3810354939 >> v2:172.21.15.191:3300/0 conn(0x7f56ac13e810 0x7f56ac13ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56b0af7640 1 --2- 172.21.15.60:0/3810354939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56ac071a50 0x7f56ac0a87e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56b0af7640 1 -- 172.21.15.60:0/3810354939 >> 172.21.15.60:0/3810354939 conn(0x7f56ac0b7300 msgr2=0x7f56ac0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:04.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56b0af7640 1 -- 172.21.15.60:0/3810354939 shutdown_connections 2024-03-27T21:42:04.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.714+0000 7f56b0af7640 1 -- 172.21.15.60:0/3810354939 wait complete. 2024-03-27T21:42:04.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.715+0000 7f56b0af7640 10 monclient: build_initial_monmap 2024-03-27T21:42:04.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.715+0000 7f56b0af7640 10 monclient: monmap: 2024-03-27T21:42:04.731 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:04.731 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:04.732 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:04.732 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:04.732 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:04.732 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:04.732 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:04.732 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:04.733 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:04.733 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:04.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.715+0000 7f56b0af7640 1 librados: starting msgr at 2024-03-27T21:42:04.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.715+0000 7f56b0af7640 1 librados: starting objecter 2024-03-27T21:42:04.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56b0af7640 1 Processor -- start 2024-03-27T21:42:04.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56b0af7640 1 -- start start 2024-03-27T21:42:04.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56b0af7640 1 librados: setting wanted keys 2024-03-27T21:42:04.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56b0af7640 1 librados: calling monclient init 2024-03-27T21:42:04.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56b0af7640 10 monclient: init 2024-03-27T21:42:04.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56b0af7640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:04.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56b0af7640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:42:04.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56b0af7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56ac071a50 0x7f56ac137430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:04.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56b0af7640 10 monclient(hunting): picked mon.noname-a con 0x7f56ac071a50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:04.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56b0af7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56ac13e810 0x7f56ac13a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:04.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56b0af7640 10 monclient(hunting): picked mon.noname-b con 0x7f56ac13e810 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:04.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56b0af7640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f56ac141630 0x7f56ac1da220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:04.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56b0af7640 10 monclient(hunting): picked mon.noname-c con 0x7f56ac141630 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:04.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56b0af7640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:04.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56b0af7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ac1d9ab0 con 0x7f56ac071a50 2024-03-27T21:42:04.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56b0af7640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:04.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56b0af7640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ac1d9d00 con 0x7f56ac13e810 2024-03-27T21:42:04.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56b0af7640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:04.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56b0af7640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56ac138d80 con 0x7f56ac141630 2024-03-27T21:42:04.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56b0af7640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:04.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56b0af7640 10 monclient(hunting): authenticate will time out at 965.937582s 2024-03-27T21:42:04.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56aa59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56ac071a50 0x7f56ac137430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:04.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56aa59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56ac071a50 0x7f56ac137430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39202/0 (socket says 172.21.15.60:39202) 2024-03-27T21:42:04.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56aa59c640 1 -- 172.21.15.60:0/2825419769 learned_addr learned my addr 172.21.15.60:0/2825419769 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:04.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56aa59c640 10 monclient(hunting): get_auth_request con 0x7f56ac071a50 auth_method 0 2024-03-27T21:42:04.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:04.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56aa59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:04.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:04.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56aad9d640 1 --2- 172.21.15.60:0/2825419769 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f56ac141630 0x7f56ac1da220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:04.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56aad9d640 10 monclient(hunting): get_auth_request con 0x7f56ac141630 auth_method 0 2024-03-27T21:42:04.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:04.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56aad9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:04.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:04.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56a3fff640 1 --2- 172.21.15.60:0/2825419769 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56ac13e810 0x7f56ac13a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:04.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:04.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:04.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:04.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56a3fff640 10 monclient(hunting): get_auth_request con 0x7f56ac13e810 auth_method 0 2024-03-27T21:42:04.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:04.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:04.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:04.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:04.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:04.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:04.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:04.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:04.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.716+0000 7f56a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:04.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56aa59c640 10 monclient(hunting): handle_auth_done global_id 5277 payload 824 2024-03-27T21:42:04.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56aa59c640 1 -- 172.21.15.60:0/2825419769 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f56ac141630 msgr2=0x7f56ac1da220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:04.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56aa59c640 1 --2- 172.21.15.60:0/2825419769 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f56ac141630 0x7f56ac1da220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56aa59c640 1 -- 172.21.15.60:0/2825419769 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56ac13e810 msgr2=0x7f56ac13a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:04.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56aa59c640 1 --2- 172.21.15.60:0/2825419769 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56ac13e810 0x7f56ac13a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56aa59c640 10 monclient: _finish_hunting 0 2024-03-27T21:42:04.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56aa59c640 1 monclient: found mon.noname-a 2024-03-27T21:42:04.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56aa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:04.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56aa59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:04.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56aa59c640 1 -- 172.21.15.60:0/2825419769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f569c0094d0 con 0x7f56ac071a50 2024-03-27T21:42:04.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56aad9d640 1 --2- 172.21.15.60:0/2825419769 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f56ac141630 0x7f56ac1da220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:04.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56aa59c640 1 --2- 172.21.15.60:0/2825419769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56ac071a50 0x7f56ac137430 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f569c02b9e0 tx=0x7f569c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:04.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56a2ffd640 10 client.?.objecter ms_handle_connect 0x7f56ac071a50 2024-03-27T21:42:04.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:04.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56a2ffd640 1 -- 172.21.15.60:0/2825419769 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c039c90 con 0x7f56ac071a50 2024-03-27T21:42:04.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:04.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56a2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:04.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56a2ffd640 10 monclient: dump: 2024-03-27T21:42:04.746 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:04.746 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:04.746 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:04.746 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:04.747 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:04.747 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:04.747 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:04.747 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:04.748 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:04.748 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:04.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56a2ffd640 10 monclient: _finish_auth 0 2024-03-27T21:42:04.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56a2ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:04.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:04.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56a2ffd640 1 -- 172.21.15.60:0/2825419769 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f569c039e10 con 0x7f56ac071a50 2024-03-27T21:42:04.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56b0af7640 5 monclient: authenticate success, global_id 5277 2024-03-27T21:42:04.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:04.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56a2ffd640 1 -- 172.21.15.60:0/2825419769 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c043640 con 0x7f56ac071a50 2024-03-27T21:42:04.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56b0af7640 10 monclient: _renew_subs 2024-03-27T21:42:04.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56b0af7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:04.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56b0af7640 1 -- 172.21.15.60:0/2825419769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56ac13abc0 con 0x7f56ac071a50 2024-03-27T21:42:04.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56a959a640 4 set_mon_vals no callback set 2024-03-27T21:42:04.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:04.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:04.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56a2ffd640 10 monclient: dump: 2024-03-27T21:42:04.751 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:04.751 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:04.751 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:04.751 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:04.752 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:04.752 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:04.752 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:04.752 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:04.753 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:04.753 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:04.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56b0af7640 10 client.5277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:04.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56b0af7640 10 monclient: _renew_subs 2024-03-27T21:42:04.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56b0af7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:04.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56b0af7640 1 -- 172.21.15.60:0/2825419769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56ac1da940 con 0x7f56ac071a50 2024-03-27T21:42:04.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.717+0000 7f56b0af7640 1 librados: init done 2024-03-27T21:42:04.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.718+0000 7f56b0af7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:04.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.718+0000 7f56b0af7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:04.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.718+0000 7f56b0af7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:04.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.718+0000 7f56b0af7640 1 -- 172.21.15.60:0/2825419769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56ac0ab460 con 0x7f56ac071a50 2024-03-27T21:42:04.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.718+0000 7f56a2ffd640 1 -- 172.21.15.60:0/2825419769 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f569c0437a0 con 0x7f56ac071a50 2024-03-27T21:42:04.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.719+0000 7f56a2ffd640 1 --2- 172.21.15.60:0/2825419769 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f568c06c590 0x7f568c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:04.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.719+0000 7f56a3fff640 1 --2- 172.21.15.60:0/2825419769 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f568c06c590 0x7f568c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:04.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.719+0000 7f56a2ffd640 1 -- 172.21.15.60:0/2825419769 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f569c03e070 con 0x7f56ac071a50 2024-03-27T21:42:04.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.719+0000 7f56a3fff640 10 monclient: get_auth_request con 0x7f568c06c590 auth_method 0 2024-03-27T21:42:04.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.719+0000 7f56a2ffd640 10 client.5277.objecter ms_dispatch 0x7f56ac000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:04.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.719+0000 7f56a2ffd640 3 client.5277.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:04.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.719+0000 7f56a2ffd640 3 client.5277.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:04.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.719+0000 7f56a2ffd640 20 client.5277.objecter dump_active .. 0 homeless 2024-03-27T21:42:04.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.719+0000 7f56a2ffd640 10 client.5277.objecter ms_handle_connect 0x7f568c06c590 2024-03-27T21:42:04.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.719+0000 7f56a3fff640 1 --2- 172.21.15.60:0/2825419769 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f568c06c590 0x7f568c06ea50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f56ac138be0 tx=0x7f5694009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:04.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.723+0000 7f56a2ffd640 1 -- 172.21.15.60:0/2825419769 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f569c0776a0 con 0x7f56ac071a50 2024-03-27T21:42:04.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.723+0000 7f56a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:04.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.723+0000 7f56a2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:04.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.903+0000 7f56b0af7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:42:04.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.903+0000 7f56b0af7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:42:04.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.903+0000 7f56b0af7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:04.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.903+0000 7f56b0af7640 1 -- 172.21.15.60:0/2825419769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56ac1d9f20 con 0x7f56ac071a50 2024-03-27T21:42:04.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.904+0000 7f56a2ffd640 1 -- 172.21.15.60:0/2825419769 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+14175 (secure 0 0 0) 0x7f569c07acc0 con 0x7f56ac071a50 2024-03-27T21:42:04.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.904+0000 7f56a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:42:04.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.904+0000 7f56a2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:42:04.891 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:42:04.891 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":92,"fsid":"dcd19d9a-535c-4b41-9095-d588268e2860","created":"2024-03-27T21:39:27.092191+0000","modified":"2024-03-27T21:42:04.274002+0000","last_up_change":"2024-03-27T21:41:29.048228+0000","last_in_change":"2024-03-27T21:41:48.185818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T21:39:37.438493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-27T21:39:45.042103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"778aaa64-6720-495b-bcdc-86fbcece9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":34603},{"type":"v1","addr":"172.21.15.60:6817","nonce":34603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":34603},{"type":"v1","addr":"172.21.15.60:6819","nonce":34603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":34603},{"type":"v1","addr":"172.21.15.60:6823","nonce":34603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":34603},{"type":"v1","addr":"172.21.15.60:6821","nonce":34603}]},"public_addr":"172.21.15.60:6817/34603","cluster_addr":"172.21.15.60:6819/34603","heartbeat_back_addr":"172.21.15.60:6823/34603","heartbeat_front_addr":"172.21.15.60:6821/34603","state":["exists","up"]},{"osd":1,"uuid":"d960adf1-3d83-4e24-a612-8df53fa43d0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":34602},{"type":"v1","addr":"172.21.15.60:6801","nonce":34602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":34602},{"type":"v1","addr":"172.21.15.60:6803","nonce":34602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":34602},{"type":"v1","addr":"172.21.15.60:6807","nonce":34602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":34602},{"type":"v1","addr":"172.21.15.60:6805","nonce":34602}]},"public_addr":"172.21.15.60:6801/34602","cluster_addr":"172.21.15.60:6803/34602","heartbeat_back_addr":"172.21.15.60:6807/34602","heartbeat_front_addr":"172.21.15.60:6805/34602","state":["exists","up"]},{"osd":2,"uuid":"b8ffc48b-cf6f-4701-a6aa-c25fdb0f7977","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":33,"last_clean_end":44,"up_from":62,"up_thru":62,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":56434},{"type":"v1","addr":"172.21.15.60:6825","nonce":56434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":56434},{"type":"v1","addr":"172.21.15.60:6827","nonce":56434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":56434},{"type":"v1","addr":"172.21.15.60:6831","nonce":56434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":56434},{"type":"v1","addr":"172.21.15.60:6829","nonce":56434}]},"public_addr":"172.21.15.60:6825/56434","cluster_addr":"172.21.15.60:6827/56434","heartbeat_back_addr":"172.21.15.60:6831/56434","heartbeat_front_addr":"172.21.15.60:6829/56434","state":["exists","up"]},{"osd":3,"uuid":"0f14f2d8-b271-48e4-a198-aa5279a01c06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":34613},{"type":"v1","addr":"172.21.15.60:6809","nonce":34613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":34613},{"type":"v1","addr":"172.21.15.60:6811","nonce":34613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":34613},{"type":"v1","addr":"172.21.15.60:6815","nonce":34613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":34613},{"type":"v1","addr":"172.21.15.60:6813","nonce":34613}]},"public_addr":"172.21.15.60:6809/34613","cluster_addr":"172.21.15.60:6811/34613","heartbeat_back_addr":"172.21.15.60:6815/34613","heartbeat_front_addr":"172.21.15.60:6813/34613","state":["exists","up"]},{"osd":4,"uuid":"8cdeb7ae-1c6e-4d0d-ba44-80248eff2fb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":34332},{"type":"v1","addr":"172.21.15.191:6801","nonce":34332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":34332},{"type":"v1","addr":"172.21.15.191:6803","nonce":34332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":34332},{"type":"v1","addr":"172.21.15.191:6807","nonce":34332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":34332},{"type":"v1","addr":"172.21.15.191:6805","nonce":34332}]},"public_addr":"172.21.15.191:6801/34332","cluster_addr":"172.21.15.191:6803/34332","heartbeat_back_addr":"172.21.15.191:6807/34332","heartbeat_front_addr":"172.21.15.191:6805/34332","state":["exists","up"]},{"osd":5,"uuid":"93748cf8-d3fe-46b8-950f-e01e28a5e02c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6809","nonce":34333},{"type":"v1","addr":"172.21.15.191:6811","nonce":34333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6813","nonce":34333},{"type":"v1","addr":"172.21.15.191:6816","nonce":34333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":34333},{"type":"v1","addr":"172.21.15.191:6823","nonce":34333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":34333},{"type":"v1","addr":"172.21.15.191:6820","nonce":34333}]},"public_addr":"172.21.15.191:6811/34333","cluster_addr":"172.21.15.191:6816/34333","heartbeat_back_addr":"172.21.15.191:6823/34333","heartbeat_front_addr":"172.21.15.191:6820/34333","state":["exists","up"]},{"osd":6,"uuid":"1ad21434-cbc5-4de9-8d63-76f894f7e736","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":34361},{"type":"v1","addr":"172.21.15.191:6825","nonce":34361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":34361},{"type":"v1","addr":"172.21.15.191:6827","nonce":34361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":34361},{"type":"v1","addr":"172.21.15.191:6831","nonce":34361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":34361},{"type":"v1","addr":"172.21.15.191:6829","nonce":34361}]},"public_addr":"172.21.15.191:6825/34361","cluster_addr":"172.21.15.191:6827/34361","heartbeat_back_addr":"172.21.15.191:6831/34361","heartbeat_front_addr":"172.21.15.191:6829/34361","state":["exists","up"]},{"osd":7,"uuid":"9fbd0927-ee2f-471b-8df8-c4e0fc40b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":34334},{"type":"v1","addr":"172.21.15.191:6810","nonce":34334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":34334},{"type":"v1","addr":"172.21.15.191:6814","nonce":34334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6819","nonce":34334},{"type":"v1","addr":"172.21.15.191:6821","nonce":34334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6815","nonce":34334},{"type":"v1","addr":"172.21.15.191:6817","nonce":34334}]},"public_addr":"172.21.15.191:6810/34334","cluster_addr":"172.21.15.191:6814/34334","heartbeat_back_addr":"172.21.15.191:6821/34334","heartbeat_front_addr":"172.21.15.191:6817/34334","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.476097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.411243+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.458154+0000","dead_epoch":45},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.415285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.431201+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.456013+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.474448+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.414076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T21:42:04.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.905+0000 7f56b0af7640 10 librados: watch_flush enter 2024-03-27T21:42:04.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.905+0000 7f56b0af7640 10 librados: watch_flush exit 2024-03-27T21:42:04.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.905+0000 7f56b0af7640 20 client.5277.objecter shutdown clearing up homeless session... 2024-03-27T21:42:04.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.905+0000 7f56b0af7640 10 client.5277.objecter successfully canceled tick 2024-03-27T21:42:04.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.906+0000 7f56b0af7640 1 -- 172.21.15.60:0/2825419769 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f568c06c590 msgr2=0x7f568c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:04.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.906+0000 7f56b0af7640 1 --2- 172.21.15.60:0/2825419769 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f568c06c590 0x7f568c06ea50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f56ac138be0 tx=0x7f5694009190 comp rx=0 tx=0).stop 2024-03-27T21:42:04.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.906+0000 7f56b0af7640 10 monclient: shutdown 2024-03-27T21:42:04.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.906+0000 7f56b0af7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:04.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.906+0000 7f56b0af7640 1 -- 172.21.15.60:0/2825419769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56ac071a50 msgr2=0x7f56ac137430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:04.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.906+0000 7f56b0af7640 1 --2- 172.21.15.60:0/2825419769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56ac071a50 0x7f56ac137430 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f569c02b9e0 tx=0x7f569c004730 comp rx=0 tx=0).stop 2024-03-27T21:42:04.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.906+0000 7f56b0af7640 1 -- 172.21.15.60:0/2825419769 shutdown_connections 2024-03-27T21:42:04.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.906+0000 7f56b0af7640 1 --2- 172.21.15.60:0/2825419769 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f56ac141630 0x7f56ac1da220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.906+0000 7f56b0af7640 1 --2- 172.21.15.60:0/2825419769 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f568c06c590 0x7f568c06ea50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.906+0000 7f56b0af7640 1 --2- 172.21.15.60:0/2825419769 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56ac13e810 0x7f56ac13a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.906+0000 7f56b0af7640 1 --2- 172.21.15.60:0/2825419769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56ac071a50 0x7f56ac137430 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:04.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.906+0000 7f56b0af7640 1 -- 172.21.15.60:0/2825419769 >> 172.21.15.60:0/2825419769 conn(0x7f56ac0b7300 msgr2=0x7f56ac0b2170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:04.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.906+0000 7f56b0af7640 1 -- 172.21.15.60:0/2825419769 shutdown_connections 2024-03-27T21:42:04.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.906+0000 7f56b0af7640 1 -- 172.21.15.60:0/2825419769 wait complete. 2024-03-27T21:42:04.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:04.906+0000 7f56b0af7640 1 librados: shutdown 2024-03-27T21:42:04.903 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-27T21:42:04.904 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-27T21:42:04.904 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-27T21:42:04.904 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-27T21:42:04.905 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-27T21:42:04.905 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-27T21:42:04.906 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-27T21:42:04.906 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-27T21:42:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.089+0000 7f32a3f39640 10 monclient: get_monmap_and_config 2024-03-27T21:42:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.089+0000 7f32a3f39640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.089+0000 7f55aee54640 10 monclient: get_monmap_and_config 2024-03-27T21:42:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.089+0000 7f55aee54640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.089+0000 7f55aee54640 10 monclient: monmap: 2024-03-27T21:42:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.077 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.077 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:05.090529+0000 2024-03-27T21:42:05.077 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:05.090529+0000 2024-03-27T21:42:05.077 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.077 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.077 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:05.077 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:05.077 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:05.077 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:05.078 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:05.078 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.089+0000 7f32a3f39640 10 monclient: monmap: 2024-03-27T21:42:05.078 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.078 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.078 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:05.090526+0000 2024-03-27T21:42:05.079 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:05.090526+0000 2024-03-27T21:42:05.079 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.079 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.079 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:05.079 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:05.079 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:05.079 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:05.079 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:05.079 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.089+0000 7fda84ccf640 10 monclient: get_monmap_and_config 2024-03-27T21:42:05.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.089+0000 7fda84ccf640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.089+0000 7fda84ccf640 10 monclient: monmap: 2024-03-27T21:42:05.081 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.081 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.081 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:05.090751+0000 2024-03-27T21:42:05.081 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:05.090751+0000 2024-03-27T21:42:05.081 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.081 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.081 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:05.081 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:05.082 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:05.082 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:05.082 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:05.082 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.090+0000 7f55aee54640 1 Processor -- start 2024-03-27T21:42:05.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.090+0000 7f32a3f39640 1 Processor -- start 2024-03-27T21:42:05.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.090+0000 7f7b2b59e640 10 monclient: get_monmap_and_config 2024-03-27T21:42:05.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.090+0000 7f7b2b59e640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.090+0000 7f7b2b59e640 10 monclient: monmap: 2024-03-27T21:42:05.084 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.084 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.084 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:05.091589+0000 2024-03-27T21:42:05.084 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:05.091589+0000 2024-03-27T21:42:05.084 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.084 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.084 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:05.084 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:05.085 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:05.085 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:05.085 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:05.085 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.090+0000 7f55aee54640 1 -- start start 2024-03-27T21:42:05.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.090+0000 7f55aee54640 10 monclient: init 2024-03-27T21:42:05.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7fda84ccf640 1 Processor -- start 2024-03-27T21:42:05.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f7b2b59e640 1 Processor -- start 2024-03-27T21:42:05.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f7b2b59e640 1 -- start start 2024-03-27T21:42:05.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f7b2b59e640 10 monclient: init 2024-03-27T21:42:05.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.090+0000 7f32a3f39640 1 -- start start 2024-03-27T21:42:05.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.090+0000 7f32a3f39640 10 monclient: init 2024-03-27T21:42:05.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f32a3f39640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f32a3f39640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-03-27T21:42:05.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f32a3f39640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f329c0714e0 0x7f329c13e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f32a3f39640 10 monclient(hunting): picked mon.noname-a con 0x7f329c0714e0 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:05.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f32a3f39640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f329c0b34a0 0x7f329c141450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f32a3f39640 10 monclient(hunting): picked mon.noname-c con 0x7f329c0b34a0 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:05.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f32a3f39640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f329c0a8650 0x7f329c141990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f32a3f39640 10 monclient(hunting): picked mon.noname-b con 0x7f329c0a8650 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f32a3f39640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f32a3f39640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f329c0718c0 con 0x7f329c0714e0 2024-03-27T21:42:05.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f32a3f39640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f32a3f39640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f329c0a8c90 con 0x7f329c0b34a0 2024-03-27T21:42:05.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f32a3f39640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f32a3f39640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f329c0aa3b0 con 0x7f329c0a8650 2024-03-27T21:42:05.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f32a3f39640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f32a3f39640 10 monclient(hunting): authenticate will time out at 966.312664s 2024-03-27T21:42:05.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f55aee54640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f55aee54640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-03-27T21:42:05.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f55aee54640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f55a0039000 0x7f55a00c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f55aee54640 10 monclient(hunting): picked mon.noname-a con 0x7f55a0039000 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:05.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f55aee54640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f55a002ef40 0x7f55a00c5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f55aee54640 10 monclient(hunting): picked mon.noname-c con 0x7f55a002ef40 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:05.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f55aee54640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f55a002f910 0x7f55a00cb100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f55aee54640 10 monclient(hunting): picked mon.noname-b con 0x7f55a002f910 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f55aee54640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f55aee54640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55a00393e0 con 0x7f55a0039000 2024-03-27T21:42:05.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f55aee54640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f55aee54640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55a0030be0 con 0x7f55a002ef40 2024-03-27T21:42:05.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f55aee54640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f55aee54640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55a002ec80 con 0x7f55a002f910 2024-03-27T21:42:05.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f55aee54640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7f55aee54640 10 monclient(hunting): authenticate will time out at 966.312808s 2024-03-27T21:42:05.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7fda84ccf640 1 -- start start 2024-03-27T21:42:05.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.091+0000 7fda84ccf640 10 monclient: init 2024-03-27T21:42:05.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f32a1cd5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f329c0714e0 0x7f329c13e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f55ad651640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f55a002ef40 0x7f55a00c5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f55ad651640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f55a002ef40 0x7f55a00c5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:58646/0 (socket says 172.21.15.60:58646) 2024-03-27T21:42:05.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f55ad651640 1 -- 172.21.15.60:0/2167132569 learned_addr learned my addr 172.21.15.60:0/2167132569 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f55ad651640 10 monclient(hunting): get_auth_request con 0x7f55a002ef40 auth_method 0 2024-03-27T21:42:05.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f55ad651640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f55ad651640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f55ad651640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f32a1cd5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f329c0714e0 0x7f329c13e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39210/0 (socket says 172.21.15.60:39210) 2024-03-27T21:42:05.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f32a1cd5640 1 -- 172.21.15.60:0/754878001 learned_addr learned my addr 172.21.15.60:0/754878001 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f32a1cd5640 10 monclient(hunting): get_auth_request con 0x7f329c0714e0 auth_method 0 2024-03-27T21:42:05.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f32a1cd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f32a1cd5640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f32a1cd5640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f32a1cd5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f32a1cd5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f32a1cd5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f55ad651640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f55ad651640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f55ad651640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f32a1cd5640 10 monclient(hunting): handle_auth_done global_id 5280 payload 274 2024-03-27T21:42:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f32a1cd5640 1 -- 172.21.15.60:0/754878001 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f329c0a8650 msgr2=0x7f329c141990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T21:42:05.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f32a1cd5640 1 --2- 172.21.15.60:0/754878001 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f329c0a8650 0x7f329c141990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f32a1cd5640 1 -- 172.21.15.60:0/754878001 >> v2:172.21.15.191:3300/0 conn(0x7f329c0b34a0 msgr2=0x7f329c141450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T21:42:05.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f32a1cd5640 1 --2- 172.21.15.60:0/754878001 >> v2:172.21.15.191:3300/0 conn(0x7f329c0b34a0 0x7f329c141450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f32a1cd5640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f32a1cd5640 1 monclient: found mon.noname-a 2024-03-27T21:42:05.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f32a1cd5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f32a1cd5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:05.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f32a1cd5640 1 -- 172.21.15.60:0/754878001 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f329c13ede0 con 0x7f329c0714e0 2024-03-27T21:42:05.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f7b2b59e640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f7b2b59e640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-03-27T21:42:05.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f7b2b59e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7b2c0aeda0 0x7f7b2c13dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f889d570640 10 monclient: get_monmap_and_config 2024-03-27T21:42:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f889d570640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f889d570640 10 monclient: monmap: 2024-03-27T21:42:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:05.093059+0000 2024-03-27T21:42:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:05.093059+0000 2024-03-27T21:42:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:05.102 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f889d570640 1 Processor -- start 2024-03-27T21:42:05.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f889d570640 1 -- start start 2024-03-27T21:42:05.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f889d570640 10 monclient: init 2024-03-27T21:42:05.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f55ad651640 10 monclient(hunting): handle_auth_done global_id 5188 payload 274 2024-03-27T21:42:05.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f55ad651640 1 -- 172.21.15.60:0/2167132569 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f55a002f910 msgr2=0x7f55a00cb100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f55ad651640 1 --2- 172.21.15.60:0/2167132569 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f55a002f910 0x7f55a00cb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f55ad651640 1 -- 172.21.15.60:0/2167132569 >> v2:172.21.15.60:3300/0 conn(0x7f55a0039000 msgr2=0x7f55a00c4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f55ad651640 1 --2- 172.21.15.60:0/2167132569 >> v2:172.21.15.60:3300/0 conn(0x7f55a0039000 0x7f55a00c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f55ad651640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f55ad651640 1 monclient: found mon.noname-c 2024-03-27T21:42:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f55ad651640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f55ad651640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f55ad651640 1 -- 172.21.15.60:0/2167132569 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a002f3f0 con 0x7f55a002ef40 2024-03-27T21:42:05.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f7b2b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f7b2c0aeda0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f7b2b59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7b2c13a900 0x7f7b2c13ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f7b2b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f7b2c13a900 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:05.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f7b2b59e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7b2c070d90 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:05.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f7b2b59e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c13e3c0 con 0x7f7b2c070d90 2024-03-27T21:42:05.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f7b2b59e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f7b2b59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b2c070490 con 0x7f7b2c13a900 2024-03-27T21:42:05.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f7b2b59e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f7b2b59e640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c0af180 con 0x7f7b2c0aeda0 2024-03-27T21:42:05.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f7b2b59e640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f7b2b59e640 10 monclient(hunting): authenticate will time out at 966.313719s 2024-03-27T21:42:05.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f7b2ad9d640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f7b2c070d90 0x7f7b2c071220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.60:58306/0 (socket says 172.21.15.60:58306) 2024-03-27T21:42:05.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f7b2ad9d640 1 -- 172.21.15.60:0/1874088258 learned_addr learned my addr 172.21.15.60:0/1874088258 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:05.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f7b29d9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7b2c13a900 0x7f7b2c13ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f7b29d9b640 10 monclient(hunting): get_auth_request con 0x7f7b2c13a900 auth_method 0 2024-03-27T21:42:05.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f7b29d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f7b29d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f7b29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f7b29d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f7b29d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f7b29d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f7b29d9b640 10 monclient(hunting): handle_auth_done global_id 5283 payload 274 2024-03-27T21:42:05.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f7b29d9b640 1 -- 172.21.15.60:0/1874088258 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7b2c0aeda0 msgr2=0x7f7b2c13dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f7b2a59c640 1 --2- 172.21.15.60:0/1874088258 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7b2c0aeda0 0x7f7b2c13dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f0bdf949640 10 monclient: get_monmap_and_config 2024-03-27T21:42:05.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f0bdf949640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7f0bdf949640 10 monclient: monmap: 2024-03-27T21:42:05.107 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.107 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.108 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:05.093292+0000 2024-03-27T21:42:05.108 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:05.093292+0000 2024-03-27T21:42:05.108 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.108 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.108 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:05.108 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:05.108 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:05.108 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:05.109 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:05.109 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f0bdf949640 1 Processor -- start 2024-03-27T21:42:05.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f7b29d9b640 1 --2- 172.21.15.60:0/1874088258 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7b2c0aeda0 0x7f7b2c13dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f7b29d9b640 1 -- 172.21.15.60:0/1874088258 >> v1:172.21.15.191:6789/0 conn(0x7f7b2c070d90 legacy=0x7f7b2c071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f7b29d9b640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f7b29d9b640 1 monclient: found mon.noname-a 2024-03-27T21:42:05.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f7b29d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f7b29d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:05.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f7b29d9b640 1 -- 172.21.15.60:0/1874088258 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c070690 con 0x7f7b2c13a900 2024-03-27T21:42:05.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f889d570640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f889d570640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-03-27T21:42:05.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f889d570640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f88980ae400 0x7f889808a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f889d570640 10 monclient(hunting): picked mon.noname-b con 0x7f88980ae400 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f889d570640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8898151da0 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:05.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f889d570640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f889814dc00 0x7f889814e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f889d570640 10 monclient(hunting): picked mon.noname-a con 0x7f889814dc00 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:05.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f889d570640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88980abdb0 con 0x7f8898151da0 2024-03-27T21:42:05.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f889d570640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f889d570640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88980ac070 con 0x7f889814dc00 2024-03-27T21:42:05.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f889d570640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f889d570640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88980ae7e0 con 0x7f88980ae400 2024-03-27T21:42:05.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f889d570640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f889d570640 10 monclient(hunting): authenticate will time out at 966.315129s 2024-03-27T21:42:05.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f88977fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f8898151da0 0x7f88980ac4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.60:58322/0 (socket says 172.21.15.60:58322) 2024-03-27T21:42:05.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f88977fe640 1 -- 172.21.15.60:0/1855930078 learned_addr learned my addr 172.21.15.60:0/1855930078 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:05.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f8897fff640 1 --2- 172.21.15.60:0/1855930078 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f88980ae400 0x7f889808a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f8897fff640 10 monclient(hunting): get_auth_request con 0x7f88980ae400 auth_method 0 2024-03-27T21:42:05.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7fda84ccf640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7fda84ccf640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-03-27T21:42:05.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7fda84ccf640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fda800714e0 0x7fda8013e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7fda84ccf640 10 monclient(hunting): picked mon.noname-c con 0x7fda800714e0 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:05.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7fda84ccf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fda800b34a0 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:05.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7fda84ccf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fda800a8650 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:05.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7fda84ccf640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda800a8c90 con 0x7fda800a8650 2024-03-27T21:42:05.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7fda84ccf640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda800aa3b0 con 0x7fda800b34a0 2024-03-27T21:42:05.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7fda84ccf640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7fda84ccf640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda800718c0 con 0x7fda800714e0 2024-03-27T21:42:05.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7fda84ccf640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7fda84ccf640 10 monclient(hunting): authenticate will time out at 966.314438s 2024-03-27T21:42:05.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7fda7e59c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fda800714e0 0x7fda8013e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7fda7e59c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fda800714e0 0x7fda8013e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:58656/0 (socket says 172.21.15.60:58656) 2024-03-27T21:42:05.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7fda7e59c640 1 -- 172.21.15.60:0/4148683534 learned_addr learned my addr 172.21.15.60:0/4148683534 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7fda7e59c640 10 monclient(hunting): get_auth_request con 0x7fda800714e0 auth_method 0 2024-03-27T21:42:05.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7fda7e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7fda7e59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7fda7e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7fda7e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7fda7e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7fda7e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7fda7e59c640 10 monclient(hunting): handle_auth_done global_id 5194 payload 274 2024-03-27T21:42:05.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7fda7e59c640 1 -- 172.21.15.60:0/4148683534 >> v1:172.21.15.191:6789/0 conn(0x7fda800b34a0 legacy=0x7fda80141450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T21:42:05.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7fda7e59c640 1 -- 172.21.15.60:0/4148683534 >> v1:172.21.15.60:6789/0 conn(0x7fda800a8650 legacy=0x7fda80141b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7fda7e59c640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7fda7e59c640 1 monclient: found mon.noname-c 2024-03-27T21:42:05.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7fda7e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7fda7e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7fda7e59c640 1 -- 172.21.15.60:0/4148683534 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8013ede0 con 0x7fda800714e0 2024-03-27T21:42:05.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f7b2a59c640 10 monclient: get_auth_request con 0x7f7b2c0aeda0 auth_method 0 2024-03-27T21:42:05.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f7b2a59c640 1 --2- 172.21.15.60:0/1874088258 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7b2c0aeda0 0x7f7b2c13dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:42:05.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f2124edb640 10 monclient: get_monmap_and_config 2024-03-27T21:42:05.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f2124edb640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f2124edb640 10 monclient: monmap: 2024-03-27T21:42:05.118 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.118 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.118 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:05.095068+0000 2024-03-27T21:42:05.119 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:05.095068+0000 2024-03-27T21:42:05.119 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.119 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.119 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:05.119 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:05.119 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:05.119 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:05.119 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:05.119 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f8897fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f8897fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f8897fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f0bdf949640 1 -- start start 2024-03-27T21:42:05.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7f0bdf949640 10 monclient: init 2024-03-27T21:42:05.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f0bdf949640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f0bdf949640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-03-27T21:42:05.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f0bdf949640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0bd806d770 0x7f0bd813fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f0bdf949640 10 monclient(hunting): picked mon.noname-a con 0x7f0bd806d770 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:05.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f0bdf949640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0bd8140390 0x7f0bd80700c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f0bdf949640 10 monclient(hunting): picked mon.noname-b con 0x7f0bd8140390 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f0bdf949640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0bd8070600 0x7f0bd8070a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f0bdf949640 10 monclient(hunting): picked mon.noname-c con 0x7f0bd8070600 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:05.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f0bdf949640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f0bdf949640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd806f500 con 0x7f0bd806d770 2024-03-27T21:42:05.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f0bdf949640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f0bdf949640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd8140800 con 0x7f0bd8070600 2024-03-27T21:42:05.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f0bdf949640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f0bdf949640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd80718e0 con 0x7f0bd8140390 2024-03-27T21:42:05.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f0bdf949640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f0bdf949640 10 monclient(hunting): authenticate will time out at 966.315592s 2024-03-27T21:42:05.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f8896ffd640 1 -- 172.21.15.60:0/1855930078 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993416672 0 0) 0x7f88980abdb0 con 0x7f8898151da0 2024-03-27T21:42:05.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f8896ffd640 10 monclient(hunting): _negotiate 2024-03-27T21:42:05.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f8896ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f8896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f8896ffd640 10 monclient(hunting): my global_id is 5197 2024-03-27T21:42:05.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f8896ffd640 1 -- 172.21.15.60:0/1855930078 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8880003520 con 0x7f8898151da0 2024-03-27T21:42:05.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7ff055671640 10 monclient: get_monmap_and_config 2024-03-27T21:42:05.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7f55ad651640 1 --2- 172.21.15.60:0/2167132569 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55a002ef40 0x7f55a00c5110 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f55a000fd40 tx=0x7f559802b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b994862f259be3c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8897fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8897fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f2124edb640 1 Processor -- start 2024-03-27T21:42:05.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8897fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7ff055671640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.092+0000 7ff055671640 10 monclient: monmap: 2024-03-27T21:42:05.127 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.127 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.127 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:05.093516+0000 2024-03-27T21:42:05.128 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:05.093516+0000 2024-03-27T21:42:05.128 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.128 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.128 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:05.128 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:05.128 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:05.128 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:05.128 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:05.129 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7ff055671640 1 Processor -- start 2024-03-27T21:42:05.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f889cd6f640 1 --2- 172.21.15.60:0/1855930078 >> v2:172.21.15.60:3300/0 conn(0x7f889814dc00 0x7f889814e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f889cd6f640 10 monclient(hunting): get_auth_request con 0x7f889814dc00 auth_method 0 2024-03-27T21:42:05.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f889cd6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f889cd6f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f889cd6f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 1 -- 172.21.15.60:0/1855930078 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3602354634 0 0) 0x7f8880003520 con 0x7f8898151da0 2024-03-27T21:42:05.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 1 -- 172.21.15.60:0/1855930078 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f88980ae400 msgr2=0x7f889808a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 1 --2- 172.21.15.60:0/1855930078 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f88980ae400 0x7f889808a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 1 -- 172.21.15.60:0/1855930078 >> v2:172.21.15.60:3300/0 conn(0x7f889814dc00 msgr2=0x7f889814e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 1 --2- 172.21.15.60:0/1855930078 >> v2:172.21.15.60:3300/0 conn(0x7f889814dc00 0x7f889814e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 1 monclient: found mon.noname-c-legacy 2024-03-27T21:42:05.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.191:6789/0 2024-03-27T21:42:05.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 1 -- 172.21.15.60:0/1855930078 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88980ab440 con 0x7f8898151da0 2024-03-27T21:42:05.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 1 -- 172.21.15.60:0/1855930078 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2933500084 0 0) 0x7f88900019f0 con 0x7f8898151da0 2024-03-27T21:42:05.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-03-27T21:42:05.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 10 monclient: dump: 2024-03-27T21:42:05.132 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.133 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.133 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.133 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.133 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.133 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.133 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.133 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.134 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.134 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] but i'm connected to v1:172.21.15.191:6789/0, reconnecting 2024-03-27T21:42:05.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 1 -- 172.21.15.60:0/1855930078 >> v1:172.21.15.191:6789/0 conn(0x7f8898151da0 legacy=0x7f88980ac4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:42:05.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 1 --2- 172.21.15.60:0/1855930078 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8880003970 0x7f8880003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 10 monclient(hunting): picked mon.b con 0x7f8880003970 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:05.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 1 --2- 172.21.15.60:0/1855930078 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f88800047c0 0x7f8880004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 10 monclient(hunting): picked mon.c con 0x7f88800047c0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 1 --2- 172.21.15.60:0/1855930078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8880005180 0x7f888000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 10 monclient(hunting): picked mon.a con 0x7f8880005180 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:05.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f0bddee6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0bd8070600 0x7f0bd8070a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 1 -- 172.21.15.60:0/1855930078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88980ac070 con 0x7f8880005180 2024-03-27T21:42:05.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 1 -- 172.21.15.60:0/1855930078 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88980ae7e0 con 0x7f8880003970 2024-03-27T21:42:05.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 1 -- 172.21.15.60:0/1855930078 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888000e230 con 0x7f88800047c0 2024-03-27T21:42:05.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f8896ffd640 10 monclient(hunting): _finish_auth 0 2024-03-27T21:42:05.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f2124edb640 1 -- start start 2024-03-27T21:42:05.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f2124edb640 10 monclient: init 2024-03-27T21:42:05.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f32a1cd5640 1 --2- 172.21.15.60:0/754878001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f329c0714e0 0x7f329c13e8a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f329c0b1690 tx=0x7f329402c170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=838b7d9be79b33ec server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f0bdcee4640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0bd8140390 0x7f0bd80700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f55ace50640 1 -- 172.21.15.60:0/2167132569 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55980317e0 con 0x7f55a002ef40 2024-03-27T21:42:05.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f55ace50640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f55ace50640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:05.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f55ace50640 10 monclient: dump: 2024-03-27T21:42:05.139 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.139 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.140 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.140 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.140 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.140 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.140 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.140 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.140 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.140 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f55ace50640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f55ace50640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f55ace50640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f55aee54640 5 monclient: authenticate success, global_id 5188 2024-03-27T21:42:05.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f55aee54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:05.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f55ace50640 1 -- 172.21.15.60:0/2167132569 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5598031940 con 0x7f55a002ef40 2024-03-27T21:42:05.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f55ace50640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f55aee54640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:05.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f55aee54640 4 set_mon_vals no callback set 2024-03-27T21:42:05.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7fda7e59c640 1 --2- 172.21.15.60:0/4148683534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda800714e0 0x7fda8013e8a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fda74009870 tx=0x7fda7402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f63a7049146384c server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f55ace50640 1 -- 172.21.15.60:0/2167132569 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5598031d10 con 0x7f55a002ef40 2024-03-27T21:42:05.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f55ace50640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f55ace50640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:42:05.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7fda7d59a640 1 -- 172.21.15.60:0/4148683534 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda74031870 con 0x7fda800714e0 2024-03-27T21:42:05.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7fda7d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7fda7d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:05.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7fda7d59a640 10 monclient: dump: 2024-03-27T21:42:05.143 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.144 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.144 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.144 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.144 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.144 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.144 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.144 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.144 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.145 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7fda7d59a640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7fda7d59a640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7fda7d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f55ace50640 10 monclient: dump: 2024-03-27T21:42:05.145 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.146 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.146 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.146 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.146 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.146 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.146 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.146 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.147 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.147 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f88977fe640 1 --2- 172.21.15.60:0/1855930078 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f88800047c0 0x7f8880004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7fda84ccf640 5 monclient: authenticate success, global_id 5194 2024-03-27T21:42:05.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7fda84ccf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:05.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7fda7d59a640 1 -- 172.21.15.60:0/4148683534 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda740319d0 con 0x7fda800714e0 2024-03-27T21:42:05.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7fda7d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f88977fe640 10 monclient(hunting): get_auth_request con 0x7f88800047c0 auth_method 0 2024-03-27T21:42:05.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f88977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f88977fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f88977fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:05.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f32a0cd3640 1 -- 172.21.15.60:0/754878001 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329402cb10 con 0x7f329c0714e0 2024-03-27T21:42:05.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f32a0cd3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f32a0cd3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:05.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f32a0cd3640 10 monclient: dump: 2024-03-27T21:42:05.151 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.151 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.151 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.151 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.151 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.151 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.152 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.152 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.152 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.152 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f32a0cd3640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f32a0cd3640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f32a0cd3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f32a0cd3640 1 -- 172.21.15.60:0/754878001 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329402cc70 con 0x7f329c0714e0 2024-03-27T21:42:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f32a0cd3640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f32a0cd3640 1 -- 172.21.15.60:0/754878001 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32940343f0 con 0x7f329c0714e0 2024-03-27T21:42:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f32a0cd3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f32a0cd3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f32a0cd3640 10 monclient: dump: 2024-03-27T21:42:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.154 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.154 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.154 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.154 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.154 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.154 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.154 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.154 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f55aee54640 10 monclient: shutdown 2024-03-27T21:42:05.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f55aee54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f55aee54640 1 -- 172.21.15.60:0/2167132569 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55a002ef40 msgr2=0x7f55a00c5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f55aee54640 1 --2- 172.21.15.60:0/2167132569 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55a002ef40 0x7f55a00c5110 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f55a000fd40 tx=0x7f559802b880 comp rx=0 tx=0).stop 2024-03-27T21:42:05.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f0bdd6e5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0bd806d770 0x7f0bd813fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f0bdd6e5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0bd806d770 0x7f0bd813fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39264/0 (socket says 172.21.15.60:39264) 2024-03-27T21:42:05.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7f0bdd6e5640 1 -- 172.21.15.60:0/1338075049 learned_addr learned my addr 172.21.15.60:0/1338075049 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f0bdcee4640 10 monclient(hunting): get_auth_request con 0x7f0bd8140390 auth_method 0 2024-03-27T21:42:05.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f0bdcee4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f0bdcee4640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f0bdcee4640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f2124edb640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f2124edb640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-03-27T21:42:05.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f2124edb640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2120060e10 0x7f212006d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f2124edb640 10 monclient(hunting): picked mon.noname-b con 0x7f2120060e10 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f2124edb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f212013fe50 0x7f2120140270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f2124edb640 10 monclient(hunting): picked mon.noname-a con 0x7f212013fe50 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:05.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f2124edb640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f21200700c0 0x7f2120070510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f2124edb640 10 monclient(hunting): picked mon.noname-c con 0x7f21200700c0 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:05.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f2124edb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f2124edb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f212006f500 con 0x7f212013fe50 2024-03-27T21:42:05.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f2124edb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f2124edb640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f212006f6e0 con 0x7f21200700c0 2024-03-27T21:42:05.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f2124edb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f2124edb640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f212006f9c0 con 0x7f2120060e10 2024-03-27T21:42:05.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f2124edb640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f2124edb640 10 monclient(hunting): authenticate will time out at 966.317374s 2024-03-27T21:42:05.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f0bddee6640 10 monclient(hunting): get_auth_request con 0x7f0bd8070600 auth_method 0 2024-03-27T21:42:05.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f0bddee6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f0bddee6640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f211dd9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f212013fe50 0x7f2120140270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f211dd9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f212013fe50 0x7f2120140270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39284/0 (socket says 172.21.15.60:39284) 2024-03-27T21:42:05.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f211dd9b640 1 -- 172.21.15.60:0/81290961 learned_addr learned my addr 172.21.15.60:0/81290961 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f211dd9b640 10 monclient(hunting): get_auth_request con 0x7f212013fe50 auth_method 0 2024-03-27T21:42:05.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f211dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f211dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f211dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f0bddee6640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f0bdd6e5640 10 monclient(hunting): get_auth_request con 0x7f0bd806d770 auth_method 0 2024-03-27T21:42:05.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f0bdd6e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f0bdd6e5640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f0bdd6e5640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f55aee54640 1 -- 172.21.15.60:0/2167132569 shutdown_connections 2024-03-27T21:42:05.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7fda84ccf640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:05.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f55aee54640 1 --2- 172.21.15.60:0/2167132569 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f55a002f910 0x7f55a00cb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f55aee54640 1 --2- 172.21.15.60:0/2167132569 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55a002ef40 0x7f55a00c5110 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f55aee54640 1 --2- 172.21.15.60:0/2167132569 >> v2:172.21.15.60:3300/0 conn(0x7f55a0039000 0x7f55a00c4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f55aee54640 1 -- 172.21.15.60:0/2167132569 >> 172.21.15.60:0/2167132569 conn(0x7f55a0006430 msgr2=0x7f55a0006840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7fda84ccf640 4 set_mon_vals no callback set 2024-03-27T21:42:05.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f0bdcee4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f0bdcee4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f0bdcee4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f0bdd6e5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f0bdd6e5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f0bdd6e5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7fda84ccf640 10 monclient: shutdown 2024-03-27T21:42:05.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f0bddee6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f0bddee6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f0bddee6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f55aee54640 1 -- 172.21.15.60:0/2167132569 shutdown_connections 2024-03-27T21:42:05.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7fda84ccf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7fda84ccf640 1 -- 172.21.15.60:0/4148683534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda800714e0 msgr2=0x7fda8013e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211ed9d640 1 --2- 172.21.15.60:0/81290961 >> v2:172.21.15.191:3300/0 conn(0x7f21200700c0 0x7f2120070510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211ed9d640 10 monclient(hunting): get_auth_request con 0x7f21200700c0 auth_method 0 2024-03-27T21:42:05.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211ed9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211e59c640 1 --2- 172.21.15.60:0/81290961 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2120060e10 0x7f212006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211dd9b640 10 monclient(hunting): handle_auth_done global_id 5295 payload 274 2024-03-27T21:42:05.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211dd9b640 1 -- 172.21.15.60:0/81290961 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2120060e10 msgr2=0x7f212006d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7fda84ccf640 1 --2- 172.21.15.60:0/4148683534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda800714e0 0x7fda8013e8a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fda74009870 tx=0x7fda7402b910 comp rx=0 tx=0).stop 2024-03-27T21:42:05.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f55aee54640 1 -- 172.21.15.60:0/2167132569 wait complete. 2024-03-27T21:42:05.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f0bdcee4640 10 monclient(hunting): handle_auth_done global_id 5009 payload 274 2024-03-27T21:42:05.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f0bdcee4640 1 -- 172.21.15.60:0/1338075049 >> v2:172.21.15.191:3300/0 conn(0x7f0bd8070600 msgr2=0x7f0bd8070a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f0bdcee4640 1 --2- 172.21.15.60:0/1338075049 >> v2:172.21.15.191:3300/0 conn(0x7f0bd8070600 0x7f0bd8070a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f0bdcee4640 1 -- 172.21.15.60:0/1338075049 >> v2:172.21.15.60:3300/0 conn(0x7f0bd806d770 msgr2=0x7f0bd813fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f0bdcee4640 1 --2- 172.21.15.60:0/1338075049 >> v2:172.21.15.60:3300/0 conn(0x7f0bd806d770 0x7f0bd813fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f0bdcee4640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f0bdcee4640 1 monclient: found mon.noname-b 2024-03-27T21:42:05.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f0bdcee4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f0bdcee4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:42:05.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f0bdcee4640 1 -- 172.21.15.60:0/1338075049 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd806f6e0 con 0x7f0bd8140390 2024-03-27T21:42:05.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f88977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f88977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f889cd6f640 1 --2- 172.21.15.60:0/1855930078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8880005180 0x7f888000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f88977fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:42:05.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f889cd6f640 10 monclient(hunting): get_auth_request con 0x7f8880005180 auth_method 0 2024-03-27T21:42:05.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f889cd6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f889cd6f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f889cd6f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:05.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f8897fff640 1 --2- 172.21.15.60:0/1855930078 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8880003970 0x7f8880003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f889cd6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f889cd6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f889cd6f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:42:05.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f889cd6f640 10 monclient(hunting): handle_auth_done global_id 5197 payload 293 2024-03-27T21:42:05.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f889cd6f640 1 -- 172.21.15.60:0/1855930078 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f88800047c0 msgr2=0x7f8880004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f889cd6f640 1 --2- 172.21.15.60:0/1855930078 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f88800047c0 0x7f8880004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f889cd6f640 1 -- 172.21.15.60:0/1855930078 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8880003970 msgr2=0x7f8880003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f889cd6f640 1 --2- 172.21.15.60:0/1855930078 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8880003970 0x7f8880003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f889cd6f640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f889cd6f640 1 monclient: found mon.a 2024-03-27T21:42:05.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f889cd6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211dd9b640 1 --2- 172.21.15.60:0/81290961 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2120060e10 0x7f212006d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211dd9b640 1 -- 172.21.15.60:0/81290961 >> v2:172.21.15.191:3300/0 conn(0x7f21200700c0 msgr2=0x7f2120070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211dd9b640 1 --2- 172.21.15.60:0/81290961 >> v2:172.21.15.191:3300/0 conn(0x7f21200700c0 0x7f2120070510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211dd9b640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211dd9b640 1 monclient: found mon.noname-a 2024-03-27T21:42:05.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:05.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211dd9b640 1 -- 172.21.15.60:0/81290961 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212006fca0 con 0x7f212013fe50 2024-03-27T21:42:05.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211e59c640 10 monclient: get_auth_request con 0x7f2120060e10 auth_method 0 2024-03-27T21:42:05.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f889cd6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:05.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f889cd6f640 1 -- 172.21.15.60:0/1855930078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88980ab440 con 0x7f8880005180 2024-03-27T21:42:05.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f889cd6f640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f889cd6f640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f889cd6f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f889d570640 5 monclient: authenticate success, global_id 5197 2024-03-27T21:42:05.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f889d570640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:05.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211e59c640 1 --2- 172.21.15.60:0/81290961 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2120060e10 0x7f212006d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:42:05.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f211ed9d640 1 --2- 172.21.15.60:0/81290961 >> v2:172.21.15.191:3300/0 conn(0x7f21200700c0 0x7f2120070510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:42:05.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f7b29d9b640 1 --2- 172.21.15.60:0/1874088258 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b2c13a900 0x7f7b2c13ad20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c0097d0 tx=0x7f7b1c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de693c57244fac6c server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f7b2959a640 1 -- 172.21.15.60:0/1874088258 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c02d040 con 0x7f7b2c13a900 2024-03-27T21:42:05.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f7b2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f7b2959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:05.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f7b2959a640 10 monclient: dump: 2024-03-27T21:42:05.179 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.179 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.179 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.179 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.179 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.180 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.180 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.180 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.180 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.180 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f7b2959a640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f7b2959a640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f7b2959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f7b2b59e640 5 monclient: authenticate success, global_id 5283 2024-03-27T21:42:05.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f7b2b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:05.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f7b2959a640 1 -- 172.21.15.60:0/1874088258 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1c031810 con 0x7f7b2c13a900 2024-03-27T21:42:05.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f7b2959a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f7b2b59e640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:05.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7f7b2b59e640 4 set_mon_vals no callback set 2024-03-27T21:42:05.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f7b2b59e640 10 monclient: shutdown 2024-03-27T21:42:05.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f7b2b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f7b2b59e640 1 -- 172.21.15.60:0/1874088258 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b2c13a900 msgr2=0x7f7b2c13ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f7b2959a640 1 -- 172.21.15.60:0/1874088258 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c02d040 con 0x7f7b2c13a900 2024-03-27T21:42:05.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f7b2b59e640 1 --2- 172.21.15.60:0/1874088258 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b2c13a900 0x7f7b2c13ad20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c0097d0 tx=0x7f7b1c02b8b0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f7b2959a640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-27T21:42:05.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7fda84ccf640 1 -- 172.21.15.60:0/4148683534 shutdown_connections 2024-03-27T21:42:05.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7fda84ccf640 1 --2- 172.21.15.60:0/4148683534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda800714e0 0x7fda8013e8a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7fda84ccf640 1 -- 172.21.15.60:0/4148683534 >> 172.21.15.60:0/4148683534 conn(0x7fda800b3ae0 msgr2=0x7fda800b3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f889cd6f640 1 --2- 172.21.15.60:0/1855930078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8880005180 0x7f888000db90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f888c00ba10 tx=0x7f88800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c162f9250c1d4a8d server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f55aee54640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f55aee54640 10 monclient: monmap: 2024-03-27T21:42:05.184 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.184 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:05.184 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:05.184 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:05.184 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.184 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:05.184 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:05.185 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:05.185 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:05.185 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f8896ffd640 1 -- 172.21.15.60:0/1855930078 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c032680 con 0x7f8880005180 2024-03-27T21:42:05.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f8896ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f8896ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:05.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f8896ffd640 10 monclient: dump: 2024-03-27T21:42:05.186 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.186 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.186 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.186 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.186 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.186 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.186 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.187 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.187 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.187 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f8896ffd640 1 -- 172.21.15.60:0/1855930078 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888c0327e0 con 0x7f8880005180 2024-03-27T21:42:05.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f8896ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f8896ffd640 1 -- 172.21.15.60:0/1855930078 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c032bd0 con 0x7f8880005180 2024-03-27T21:42:05.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f8896ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f8896ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:05.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f8896ffd640 10 monclient: dump: 2024-03-27T21:42:05.188 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.188 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.188 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.188 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.188 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.188 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.188 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.188 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.189 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.189 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7ff055671640 1 -- start start 2024-03-27T21:42:05.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.093+0000 7ff055671640 10 monclient: init 2024-03-27T21:42:05.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7ff055671640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7ff055671640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-03-27T21:42:05.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7ff055671640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff05006d360 0x7ff05006fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7ff055671640 10 monclient(hunting): picked mon.noname-a con 0x7ff05006d360 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:05.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7ff055671640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff050151880 0x7ff050151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7ff055671640 10 monclient(hunting): picked mon.noname-b con 0x7ff050151880 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7ff055671640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff050152220 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:05.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7ff055671640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff05006d960 con 0x7ff050152220 2024-03-27T21:42:05.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7ff055671640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7ff055671640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff05006f0f0 con 0x7ff05006d360 2024-03-27T21:42:05.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7ff055671640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7ff055671640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05006d740 con 0x7ff050151880 2024-03-27T21:42:05.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7ff055671640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7ff055671640 10 monclient(hunting): authenticate will time out at 966.315526s 2024-03-27T21:42:05.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.094+0000 7ff04e7fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff050151880 0x7ff050151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7ff04e7fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff050151880 0x7ff050151ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34082/0 (socket says 172.21.15.60:34082) 2024-03-27T21:42:05.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7ff04e7fc640 1 -- 172.21.15.60:0/3424702237 learned_addr learned my addr 172.21.15.60:0/3424702237 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7ff04e7fc640 10 monclient(hunting): get_auth_request con 0x7ff050151880 auth_method 0 2024-03-27T21:42:05.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7ff04e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7ff04e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7ff04e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7ff04effd640 1 --2- 172.21.15.60:0/3424702237 >> v2:172.21.15.60:3300/0 conn(0x7ff05006d360 0x7ff05006fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7ff04effd640 10 monclient(hunting): get_auth_request con 0x7ff05006d360 auth_method 0 2024-03-27T21:42:05.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7ff04effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7ff04effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7ff04effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.095+0000 7ff04e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7ff04e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7ff04e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7ff04effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7ff04effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7ff04effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7ff04e7fc640 10 monclient(hunting): handle_auth_done global_id 5006 payload 274 2024-03-27T21:42:05.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7ff04e7fc640 1 -- 172.21.15.60:0/3424702237 >> v2:172.21.15.60:3300/0 conn(0x7ff05006d360 msgr2=0x7ff05006fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7ff04e7fc640 1 --2- 172.21.15.60:0/3424702237 >> v2:172.21.15.60:3300/0 conn(0x7ff05006d360 0x7ff05006fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7ff04e7fc640 1 -- 172.21.15.60:0/3424702237 >> v1:172.21.15.60:6789/0 conn(0x7ff050152220 legacy=0x7ff0500ad540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7ff04e7fc640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7ff04e7fc640 1 monclient: found mon.noname-b 2024-03-27T21:42:05.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7ff04e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7ff04e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:42:05.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.096+0000 7ff04e7fc640 1 -- 172.21.15.60:0/3424702237 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff050152690 con 0x7ff050151880 2024-03-27T21:42:05.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7ff04e7fc640 1 --2- 172.21.15.60:0/3424702237 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff050151880 0x7ff050151ce0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff04800c7e0 tx=0x7ff04802b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3acd169371d6c0c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7ff04dffb640 1 -- 172.21.15.60:0/3424702237 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff048030850 con 0x7ff050151880 2024-03-27T21:42:05.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7ff04dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7ff04dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:05.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f32a3f39640 5 monclient: authenticate success, global_id 5280 2024-03-27T21:42:05.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f32a3f39640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:05.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7f32a3f39640 4 set_mon_vals no callback set 2024-03-27T21:42:05.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7ff04dffb640 10 monclient: dump: 2024-03-27T21:42:05.196 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.196 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.196 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.197 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.197 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.197 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.197 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.197 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.197 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.197 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7ff04dffb640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7ff04dffb640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7ff04dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f55aee54640 1 librados: starting msgr at 2024-03-27T21:42:05.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f55aee54640 1 librados: starting objecter 2024-03-27T21:42:05.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f55aee54640 1 Processor -- start 2024-03-27T21:42:05.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7fda84ccf640 1 -- 172.21.15.60:0/4148683534 shutdown_connections 2024-03-27T21:42:05.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f55aee54640 1 -- start start 2024-03-27T21:42:05.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f55aee54640 1 librados: setting wanted keys 2024-03-27T21:42:05.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f55aee54640 1 librados: calling monclient init 2024-03-27T21:42:05.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.097+0000 7fda84ccf640 1 -- 172.21.15.60:0/4148683534 wait complete. 2024-03-27T21:42:05.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f55aee54640 10 monclient: init 2024-03-27T21:42:05.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f32a3f39640 10 monclient: shutdown 2024-03-27T21:42:05.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f32a3f39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f32a3f39640 1 -- 172.21.15.60:0/754878001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f329c0714e0 msgr2=0x7f329c13e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f32a3f39640 1 --2- 172.21.15.60:0/754878001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f329c0714e0 0x7f329c13e8a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f329c0b1690 tx=0x7f329402c170 comp rx=0 tx=0).stop 2024-03-27T21:42:05.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f32a3f39640 1 -- 172.21.15.60:0/754878001 shutdown_connections 2024-03-27T21:42:05.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f32a3f39640 1 --2- 172.21.15.60:0/754878001 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f329c0a8650 0x7f329c141990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f32a3f39640 1 --2- 172.21.15.60:0/754878001 >> v2:172.21.15.191:3300/0 conn(0x7f329c0b34a0 0x7f329c141450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f32a3f39640 1 --2- 172.21.15.60:0/754878001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f329c0714e0 0x7f329c13e8a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f32a3f39640 1 -- 172.21.15.60:0/754878001 >> 172.21.15.60:0/754878001 conn(0x7f329c0b3ae0 msgr2=0x7f329c0b3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f7b2b59e640 1 -- 172.21.15.60:0/1874088258 shutdown_connections 2024-03-27T21:42:05.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f7b2b59e640 1 --2- 172.21.15.60:0/1874088258 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b2c13a900 0x7f7b2c13ad20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f7b2b59e640 1 --2- 172.21.15.60:0/1874088258 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7b2c0aeda0 0x7f7b2c13dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f7b2b59e640 1 -- 172.21.15.60:0/1874088258 >> 172.21.15.60:0/1874088258 conn(0x7f7b2c0afc20 msgr2=0x7f7b2c0b0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7fda84ccf640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7fda84ccf640 10 monclient: monmap: 2024-03-27T21:42:05.204 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.204 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:05.204 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:05.204 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:05.205 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.205 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:05.205 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:05.205 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:05.205 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:05.205 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f889d570640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:05.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f889d570640 4 set_mon_vals no callback set 2024-03-27T21:42:05.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f211dd9b640 1 --2- 172.21.15.60:0/81290961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f212013fe50 0x7f2120140270 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2120141b00 tx=0x7f211002bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d55767c76562817e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f211d59a640 1 -- 172.21.15.60:0/81290961 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211002c960 con 0x7f212013fe50 2024-03-27T21:42:05.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f211d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f211d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:05.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f211d59a640 10 monclient: dump: 2024-03-27T21:42:05.207 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.207 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.207 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.207 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.207 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.207 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.208 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.208 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.208 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.208 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f211d59a640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f211d59a640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f211d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f211d59a640 1 -- 172.21.15.60:0/81290961 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f211002cac0 con 0x7f212013fe50 2024-03-27T21:42:05.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f211d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f211d59a640 1 -- 172.21.15.60:0/81290961 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211002ce50 con 0x7f212013fe50 2024-03-27T21:42:05.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f211d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f211d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:05.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f211d59a640 10 monclient: dump: 2024-03-27T21:42:05.209 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.209 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.209 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.209 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.210 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.210 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.210 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.210 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.210 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.210 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f32a3f39640 1 -- 172.21.15.60:0/754878001 shutdown_connections 2024-03-27T21:42:05.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f32a3f39640 1 -- 172.21.15.60:0/754878001 wait complete. 2024-03-27T21:42:05.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f889d570640 10 monclient: shutdown 2024-03-27T21:42:05.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f889d570640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f889d570640 1 -- 172.21.15.60:0/1855930078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8880005180 msgr2=0x7f888000db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f889d570640 1 --2- 172.21.15.60:0/1855930078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8880005180 0x7f888000db90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f888c00ba10 tx=0x7f88800057f0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7f0bdcee4640 1 --2- 172.21.15.60:0/1338075049 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0bd8140390 0x7f0bd80700c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0bd806cc50 tx=0x7f0bcc02fb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59a9b6fcc3b3beb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f889cd6f640 1 -- 172.21.15.60:0/1855930078 reap_dead start 2024-03-27T21:42:05.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f889d570640 1 -- 172.21.15.60:0/1855930078 shutdown_connections 2024-03-27T21:42:05.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f889d570640 1 -- 172.21.15.60:0/1855930078 >> 172.21.15.60:0/1855930078 conn(0x7f8898060700 msgr2=0x7f8898060b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f55aee54640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f55aee54640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:42:05.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f55aee54640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f55a002ef40 0x7f55a00b8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f889d570640 1 -- 172.21.15.60:0/1855930078 shutdown_connections 2024-03-27T21:42:05.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f889d570640 1 -- 172.21.15.60:0/1855930078 wait complete. 2024-03-27T21:42:05.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7ff055671640 5 monclient: authenticate success, global_id 5006 2024-03-27T21:42:05.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7ff055671640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:05.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7ff04dffb640 1 -- 172.21.15.60:0/3424702237 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0480309b0 con 0x7ff050151880 2024-03-27T21:42:05.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7ff04dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7ff055671640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:05.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.098+0000 7ff055671640 4 set_mon_vals no callback set 2024-03-27T21:42:05.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7ff055671640 10 monclient: shutdown 2024-03-27T21:42:05.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7ff055671640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7ff055671640 1 -- 172.21.15.60:0/3424702237 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff050151880 msgr2=0x7ff050151ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7ff055671640 1 --2- 172.21.15.60:0/3424702237 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff050151880 0x7ff050151ce0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff04800c7e0 tx=0x7ff04802b990 comp rx=0 tx=0).stop 2024-03-27T21:42:05.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f55ade52640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f55a002ef40 0x7f55a00b8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f0bd7fff640 1 -- 172.21.15.60:0/1338075049 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc0318f0 con 0x7f0bd8140390 2024-03-27T21:42:05.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f0bd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f0bd7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:05.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f0bd7fff640 10 monclient: dump: 2024-03-27T21:42:05.217 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.217 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.217 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.217 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.217 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.217 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.217 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.217 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.218 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.218 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f0bd7fff640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f0bd7fff640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f0bd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f0bd7fff640 1 -- 172.21.15.60:0/1338075049 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bcc031a50 con 0x7f0bd8140390 2024-03-27T21:42:05.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f0bd7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f0bd7fff640 1 -- 172.21.15.60:0/1338075049 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc031de0 con 0x7f0bd8140390 2024-03-27T21:42:05.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f0bd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f0bd7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:05.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f0bd7fff640 10 monclient: dump: 2024-03-27T21:42:05.219 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.219 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.219 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.219 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.219 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.220 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.220 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.220 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.220 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.220 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f0bdf949640 5 monclient: authenticate success, global_id 5009 2024-03-27T21:42:05.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f0bdf949640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:05.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f0bdf949640 4 set_mon_vals no callback set 2024-03-27T21:42:05.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7ff055671640 1 -- 172.21.15.60:0/3424702237 shutdown_connections 2024-03-27T21:42:05.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7ff055671640 1 --2- 172.21.15.60:0/3424702237 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff050151880 0x7ff050151ce0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7ff055671640 1 --2- 172.21.15.60:0/3424702237 >> v2:172.21.15.60:3300/0 conn(0x7ff05006d360 0x7ff05006fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7ff055671640 1 -- 172.21.15.60:0/3424702237 >> 172.21.15.60:0/3424702237 conn(0x7ff0500b3620 msgr2=0x7ff050060700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f55ade52640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f55a002ef40 0x7f55a00b8330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34116/0 (socket says 172.21.15.60:34116) 2024-03-27T21:42:05.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f55ade52640 1 -- 172.21.15.60:0/1084400107 learned_addr learned my addr 172.21.15.60:0/1084400107 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f55aee54640 10 monclient(hunting): picked mon.noname-c con 0x7f55a002ef40 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7ff055671640 1 -- 172.21.15.60:0/3424702237 shutdown_connections 2024-03-27T21:42:05.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f32a3f39640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f32a3f39640 10 monclient: monmap: 2024-03-27T21:42:05.223 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.223 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:05.223 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:05.224 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:05.224 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.224 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:05.224 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:05.224 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:05.224 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:05.224 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f55aee54640 1 --2- 172.21.15.60:0/1084400107 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55a002f910 0x7f55a00c5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f2124edb640 5 monclient: authenticate success, global_id 5295 2024-03-27T21:42:05.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f2124edb640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:05.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7f2124edb640 4 set_mon_vals no callback set 2024-03-27T21:42:05.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f2124edb640 10 monclient: shutdown 2024-03-27T21:42:05.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f2124edb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f2124edb640 1 -- 172.21.15.60:0/81290961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f212013fe50 msgr2=0x7f2120140270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f2124edb640 1 --2- 172.21.15.60:0/81290961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f212013fe50 0x7f2120140270 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2120141b00 tx=0x7f211002bdc0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f0bdf949640 10 monclient: shutdown 2024-03-27T21:42:05.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f0bdf949640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f0bdf949640 1 -- 172.21.15.60:0/1338075049 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0bd8140390 msgr2=0x7f0bd80700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f0bdf949640 1 --2- 172.21.15.60:0/1338075049 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0bd8140390 0x7f0bd80700c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0bd806cc50 tx=0x7f0bcc02fb30 comp rx=0 tx=0).stop 2024-03-27T21:42:05.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7fda84ccf640 1 librados: starting msgr at 2024-03-27T21:42:05.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f7b2b59e640 1 -- 172.21.15.60:0/1874088258 shutdown_connections 2024-03-27T21:42:05.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7fda84ccf640 1 librados: starting objecter 2024-03-27T21:42:05.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7fda84ccf640 1 Processor -- start 2024-03-27T21:42:05.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7fda84ccf640 1 -- start start 2024-03-27T21:42:05.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7fda84ccf640 1 librados: setting wanted keys 2024-03-27T21:42:05.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7fda84ccf640 1 librados: calling monclient init 2024-03-27T21:42:05.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.099+0000 7fda84ccf640 10 monclient: init 2024-03-27T21:42:05.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda84ccf640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda84ccf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:42:05.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda84ccf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda800714e0 0x7fda80140510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda84ccf640 10 monclient(hunting): picked mon.noname-a con 0x7fda800714e0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:05.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda84ccf640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fda800a8650 0x7fda8013b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda84ccf640 10 monclient(hunting): picked mon.noname-c con 0x7fda800a8650 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f889d570640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda84ccf640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda800b34a0 0x7fda801da8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda84ccf640 10 monclient(hunting): picked mon.noname-b con 0x7fda800b34a0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:05.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda84ccf640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda84ccf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda80140d10 con 0x7fda800714e0 2024-03-27T21:42:05.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda84ccf640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda84ccf640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8013ac90 con 0x7fda800b34a0 2024-03-27T21:42:05.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda84ccf640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda84ccf640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8013af30 con 0x7fda800a8650 2024-03-27T21:42:05.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda84ccf640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda84ccf640 10 monclient(hunting): authenticate will time out at 966.321327s 2024-03-27T21:42:05.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f889d570640 10 monclient: monmap: 2024-03-27T21:42:05.232 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.232 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:05.232 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:05.232 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:05.232 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.233 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:05.233 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:05.233 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:05.233 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:05.233 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f32a3f39640 1 librados: starting msgr at 2024-03-27T21:42:05.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f32a3f39640 1 librados: starting objecter 2024-03-27T21:42:05.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f0bdf949640 1 -- 172.21.15.60:0/1338075049 shutdown_connections 2024-03-27T21:42:05.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f0bdf949640 1 --2- 172.21.15.60:0/1338075049 >> v2:172.21.15.191:3300/0 conn(0x7f0bd8070600 0x7f0bd8070a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f0bdf949640 1 --2- 172.21.15.60:0/1338075049 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0bd8140390 0x7f0bd80700c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f0bdf949640 1 --2- 172.21.15.60:0/1338075049 >> v2:172.21.15.60:3300/0 conn(0x7f0bd806d770 0x7f0bd813fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f0bdf949640 1 -- 172.21.15.60:0/1338075049 >> 172.21.15.60:0/1338075049 conn(0x7f0bd80b3620 msgr2=0x7f0bd8060700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f7b2b59e640 1 -- 172.21.15.60:0/1874088258 wait complete. 2024-03-27T21:42:05.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f32a3f39640 1 Processor -- start 2024-03-27T21:42:05.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f2124edb640 1 -- 172.21.15.60:0/81290961 shutdown_connections 2024-03-27T21:42:05.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f2124edb640 1 --2- 172.21.15.60:0/81290961 >> v2:172.21.15.191:3300/0 conn(0x7f21200700c0 0x7f2120070510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f2124edb640 1 --2- 172.21.15.60:0/81290961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f212013fe50 0x7f2120140270 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f2124edb640 1 --2- 172.21.15.60:0/81290961 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2120060e10 0x7f212006d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f2124edb640 1 -- 172.21.15.60:0/81290961 >> 172.21.15.60:0/81290961 conn(0x7f21200b3620 msgr2=0x7f2120060700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f55aee54640 10 monclient(hunting): picked mon.noname-a con 0x7f55a002f910 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:05.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f55aee54640 1 --2- 172.21.15.60:0/1084400107 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55a0039000 0x7f55a00c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda7e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda800714e0 0x7fda80140510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda7e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda800714e0 0x7fda80140510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39296/0 (socket says 172.21.15.60:39296) 2024-03-27T21:42:05.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda7e59c640 1 -- 172.21.15.60:0/2943937655 learned_addr learned my addr 172.21.15.60:0/2943937655 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda7e59c640 10 monclient(hunting): get_auth_request con 0x7fda800714e0 auth_method 0 2024-03-27T21:42:05.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda7e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda7e59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda7e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda7ed9d640 1 --2- 172.21.15.60:0/2943937655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda800b34a0 0x7fda801da8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda7e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda7e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda7e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f2124edb640 1 -- 172.21.15.60:0/81290961 shutdown_connections 2024-03-27T21:42:05.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f55aee54640 10 monclient(hunting): picked mon.noname-b con 0x7f55a0039000 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:05.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda7ed9d640 10 monclient(hunting): get_auth_request con 0x7fda800b34a0 auth_method 0 2024-03-27T21:42:05.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda7ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda7ed9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7fda7ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f55aee54640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55aee54640 1 -- 172.21.15.60:0/1084400107 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a00b22a0 con 0x7f55a002f910 2024-03-27T21:42:05.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55aee54640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7fda7ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7fda7ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7fda7ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55aee54640 1 -- 172.21.15.60:0/1084400107 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a000a510 con 0x7f55a0039000 2024-03-27T21:42:05.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f889d570640 1 librados: starting msgr at 2024-03-27T21:42:05.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f889d570640 1 librados: starting objecter 2024-03-27T21:42:05.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ae653640 1 --2- 172.21.15.60:0/1084400107 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55a0039000 0x7f55a00c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55aee54640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55aee54640 1 -- 172.21.15.60:0/1084400107 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55a000a740 con 0x7f55a002ef40 2024-03-27T21:42:05.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7fda7e59c640 10 monclient(hunting): handle_auth_done global_id 5301 payload 824 2024-03-27T21:42:05.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55aee54640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7fda7e59c640 1 -- 172.21.15.60:0/2943937655 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fda800a8650 msgr2=0x7fda8013b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7fda7e59c640 1 --2- 172.21.15.60:0/2943937655 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fda800a8650 0x7fda8013b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55aee54640 10 monclient(hunting): authenticate will time out at 966.322154s 2024-03-27T21:42:05.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f889d570640 1 Processor -- start 2024-03-27T21:42:05.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ade52640 10 monclient(hunting): get_auth_request con 0x7f55a002ef40 auth_method 0 2024-03-27T21:42:05.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ade52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ade52640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ade52640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ae653640 10 monclient(hunting): get_auth_request con 0x7f55a0039000 auth_method 0 2024-03-27T21:42:05.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ae653640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ae653640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ae653640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7fda7e59c640 1 -- 172.21.15.60:0/2943937655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda800b34a0 msgr2=0x7fda801da8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7fda7e59c640 1 --2- 172.21.15.60:0/2943937655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda800b34a0 0x7fda801da8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7fda7e59c640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7fda7e59c640 1 monclient: found mon.noname-a 2024-03-27T21:42:05.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7fda7e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7fda7e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:05.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7fda7e59c640 1 -- 172.21.15.60:0/2943937655 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda740094d0 con 0x7fda800714e0 2024-03-27T21:42:05.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7fda7ed9d640 1 --2- 172.21.15.60:0/2943937655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda800b34a0 0x7fda801da8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:05.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f2124edb640 1 -- 172.21.15.60:0/81290961 wait complete. 2024-03-27T21:42:05.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ade52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ade52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ade52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ae653640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ae653640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ae653640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f889d570640 1 -- start start 2024-03-27T21:42:05.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ae653640 10 monclient(hunting): handle_auth_done global_id 5209 payload 824 2024-03-27T21:42:05.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ae653640 1 -- 172.21.15.60:0/1084400107 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f55a002ef40 msgr2=0x7f55a00b8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ae653640 1 --2- 172.21.15.60:0/1084400107 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f55a002ef40 0x7f55a00b8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ae653640 1 -- 172.21.15.60:0/1084400107 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55a002f910 msgr2=0x7f55a00c5980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T21:42:05.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f32a3f39640 1 -- start start 2024-03-27T21:42:05.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f32a3f39640 1 librados: setting wanted keys 2024-03-27T21:42:05.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f32a3f39640 1 librados: calling monclient init 2024-03-27T21:42:05.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7f32a3f39640 10 monclient: init 2024-03-27T21:42:05.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f32a3f39640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f32a3f39640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:42:05.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f32a3f39640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f329c0714e0 0x7f329c138eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f32a3f39640 10 monclient(hunting): picked mon.noname-b con 0x7f329c0714e0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:05.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f32a3f39640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f329c0a8650 0x7f329c144090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f32a3f39640 10 monclient(hunting): picked mon.noname-c con 0x7f329c0a8650 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f32a3f39640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f329c0b34a0 0x7f329c1de360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f32a3f39640 10 monclient(hunting): picked mon.noname-a con 0x7f329c0b34a0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:05.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f32a3f39640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f32a3f39640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329c142a00 con 0x7f329c0b34a0 2024-03-27T21:42:05.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f32a3f39640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f32a3f39640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329c1dde80 con 0x7f329c0714e0 2024-03-27T21:42:05.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f32a3f39640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f32a3f39640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f329c1445d0 con 0x7f329c0a8650 2024-03-27T21:42:05.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f32a3f39640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f32a3f39640 10 monclient(hunting): authenticate will time out at 966.322658s 2024-03-27T21:42:05.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f7b2b59e640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f7b2b59e640 10 monclient: monmap: 2024-03-27T21:42:05.255 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.255 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:05.255 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:05.255 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:05.255 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.255 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:05.255 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:05.255 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:05.256 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:05.256 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f0bdf949640 1 -- 172.21.15.60:0/1338075049 shutdown_connections 2024-03-27T21:42:05.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.100+0000 7ff055671640 1 -- 172.21.15.60:0/3424702237 wait complete. 2024-03-27T21:42:05.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7ff055671640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f889d570640 1 librados: setting wanted keys 2024-03-27T21:42:05.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ad651640 1 --2- 172.21.15.60:0/1084400107 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55a002f910 0x7f55a00c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ae653640 1 --2- 172.21.15.60:0/1084400107 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55a002f910 0x7f55a00c5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7ff055671640 10 monclient: monmap: 2024-03-27T21:42:05.258 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.258 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:05.258 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:05.258 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:05.258 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.258 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:05.258 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:05.258 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:05.259 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:05.259 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7ff055671640 1 librados: starting msgr at 2024-03-27T21:42:05.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7ff055671640 1 librados: starting objecter 2024-03-27T21:42:05.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f889d570640 1 librados: calling monclient init 2024-03-27T21:42:05.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f889d570640 10 monclient: init 2024-03-27T21:42:05.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ae653640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ae653640 1 monclient: found mon.noname-b 2024-03-27T21:42:05.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ae653640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ae653640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7f55ae653640 1 -- 172.21.15.60:0/1084400107 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55980094d0 con 0x7f55a0039000 2024-03-27T21:42:05.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7ff055671640 1 Processor -- start 2024-03-27T21:42:05.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f55ad651640 10 monclient: get_auth_request con 0x7f55a002f910 auth_method 0 2024-03-27T21:42:05.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f55ad651640 1 --2- 172.21.15.60:0/1084400107 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55a002f910 0x7f55a00c5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:42:05.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f55ae653640 1 --2- 172.21.15.60:0/1084400107 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55a0039000 0x7f55a00c7df0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f55a8067ae0 tx=0x7f55a8067040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f559e7fc640 10 client.?.objecter ms_handle_connect 0x7f55a0039000 2024-03-27T21:42:05.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f559e7fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:05.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a1cd5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f329c0714e0 0x7f329c138eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a1cd5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f329c0714e0 0x7f329c138eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:58692/0 (socket says 172.21.15.60:58692) 2024-03-27T21:42:05.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a1cd5640 1 -- 172.21.15.60:0/1304584283 learned_addr learned my addr 172.21.15.60:0/1304584283 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a1cd5640 10 monclient(hunting): get_auth_request con 0x7f329c0714e0 auth_method 0 2024-03-27T21:42:05.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a1cd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a1cd5640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a1cd5640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a1cd5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a1cd5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a1cd5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f7b2b59e640 1 librados: starting msgr at 2024-03-27T21:42:05.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f7b2b59e640 1 librados: starting objecter 2024-03-27T21:42:05.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7ff055671640 1 -- start start 2024-03-27T21:42:05.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7ff055671640 1 librados: setting wanted keys 2024-03-27T21:42:05.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7ff055671640 1 librados: calling monclient init 2024-03-27T21:42:05.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7ff055671640 10 monclient: init 2024-03-27T21:42:05.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7fda7e59c640 1 --2- 172.21.15.60:0/2943937655 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda800714e0 0x7fda80140510 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fda74009870 tx=0x7fda740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7fda6effd640 10 client.?.objecter ms_handle_connect 0x7fda800714e0 2024-03-27T21:42:05.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.101+0000 7fda6effd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:05.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a24d6640 1 --2- 172.21.15.60:0/1304584283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f329c0b34a0 0x7f329c1de360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a24d6640 10 monclient(hunting): get_auth_request con 0x7f329c0b34a0 auth_method 0 2024-03-27T21:42:05.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a24d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a24d6640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a24d6640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a1cd5640 10 monclient(hunting): handle_auth_done global_id 5212 payload 824 2024-03-27T21:42:05.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a1cd5640 1 -- 172.21.15.60:0/1304584283 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f329c0a8650 msgr2=0x7f329c144090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T21:42:05.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f7b2b59e640 1 Processor -- start 2024-03-27T21:42:05.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a1cd5640 1 --2- 172.21.15.60:0/1304584283 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f329c0a8650 0x7f329c144090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a1cd5640 1 -- 172.21.15.60:0/1304584283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f329c0b34a0 msgr2=0x7f329c1de360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a1cd5640 1 --2- 172.21.15.60:0/1304584283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f329c0b34a0 0x7f329c1de360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a1cd5640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a1cd5640 1 monclient: found mon.noname-b 2024-03-27T21:42:05.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a1cd5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a1cd5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f32a1cd5640 1 -- 172.21.15.60:0/1304584283 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3294009b30 con 0x7f329c0714e0 2024-03-27T21:42:05.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda6effd640 1 -- 172.21.15.60:0/2943937655 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda740307d0 con 0x7fda800714e0 2024-03-27T21:42:05.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda6effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:05.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda6effd640 10 monclient: dump: 2024-03-27T21:42:05.269 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.269 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.269 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.269 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.269 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.269 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.269 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.269 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.270 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.270 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda6effd640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda6effd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda6effd640 1 -- 172.21.15.60:0/2943937655 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda74030930 con 0x7fda800714e0 2024-03-27T21:42:05.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda6effd640 1 -- 172.21.15.60:0/2943937655 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda74030ce0 con 0x7fda800714e0 2024-03-27T21:42:05.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda6effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:05.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda6effd640 10 monclient: dump: 2024-03-27T21:42:05.271 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.271 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.271 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.271 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.271 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.271 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.272 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.272 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.272 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.272 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f559e7fc640 1 -- 172.21.15.60:0/1084400107 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a80723c0 con 0x7f55a0039000 2024-03-27T21:42:05.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f559e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f559e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:05.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f559e7fc640 10 monclient: dump: 2024-03-27T21:42:05.273 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.273 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.273 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.273 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.273 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.273 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.273 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.273 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.274 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f559e7fc640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f559e7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f559e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f559e7fc640 1 -- 172.21.15.60:0/1084400107 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a8069130 con 0x7f55a0039000 2024-03-27T21:42:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f559e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f559e7fc640 1 -- 172.21.15.60:0/1084400107 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a8092400 con 0x7f55a0039000 2024-03-27T21:42:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f559e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f559e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:42:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f559e7fc640 10 monclient: dump: 2024-03-27T21:42:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.276 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f559ffff640 4 set_mon_vals no callback set 2024-03-27T21:42:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f55aee54640 5 monclient: authenticate success, global_id 5209 2024-03-27T21:42:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f55aee54640 10 monclient: _renew_subs 2024-03-27T21:42:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f55aee54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f55aee54640 1 -- 172.21.15.60:0/1084400107 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a00b8df0 con 0x7f55a0039000 2024-03-27T21:42:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f55aee54640 10 client.5209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f55aee54640 10 monclient: _renew_subs 2024-03-27T21:42:05.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f55aee54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f55aee54640 1 -- 172.21.15.60:0/1084400107 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a00c7340 con 0x7f55a0039000 2024-03-27T21:42:05.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f55aee54640 1 librados: init done 2024-03-27T21:42:05.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f7b2b59e640 1 -- start start 2024-03-27T21:42:05.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f7b2b59e640 1 librados: setting wanted keys 2024-03-27T21:42:05.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f7b2b59e640 1 librados: calling monclient init 2024-03-27T21:42:05.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f7b2b59e640 10 monclient: init 2024-03-27T21:42:05.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda84ccf640 5 monclient: authenticate success, global_id 5301 2024-03-27T21:42:05.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda84ccf640 10 monclient: _renew_subs 2024-03-27T21:42:05.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda84ccf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:05.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda84ccf640 1 -- 172.21.15.60:0/2943937655 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda801dae20 con 0x7fda800714e0 2024-03-27T21:42:05.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda84ccf640 10 client.5301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda84ccf640 10 monclient: _renew_subs 2024-03-27T21:42:05.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda84ccf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:05.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda84ccf640 1 -- 172.21.15.60:0/2943937655 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda8013aa20 con 0x7fda800714e0 2024-03-27T21:42:05.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7fda84ccf640 1 librados: init done 2024-03-27T21:42:05.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f32a1cd5640 1 --2- 172.21.15.60:0/1304584283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f329c0714e0 0x7f329c138eb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3294036040 tx=0x7f3294004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f32927fc640 10 client.?.objecter ms_handle_connect 0x7f329c0714e0 2024-03-27T21:42:05.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f32927fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:05.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7ff055671640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7ff055671640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:42:05.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7ff055671640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff05006d360 0x7ff05014fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7ff055671640 10 monclient(hunting): picked mon.noname-c con 0x7ff05006d360 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7ff055671640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff050151880 0x7ff0501f7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff055671640 10 monclient(hunting): picked mon.noname-a con 0x7ff050151880 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:05.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff055671640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff050152220 0x7ff0501f7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff055671640 10 monclient(hunting): picked mon.noname-b con 0x7ff050152220 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:05.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff055671640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff055671640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0501f7cc0 con 0x7ff050151880 2024-03-27T21:42:05.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff055671640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff055671640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0500ac1d0 con 0x7ff050152220 2024-03-27T21:42:05.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff055671640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff055671640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0500aa7c0 con 0x7ff05006d360 2024-03-27T21:42:05.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff055671640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff055671640 10 monclient(hunting): authenticate will time out at 966.323961s 2024-03-27T21:42:05.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04f7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff050152220 0x7ff0501f7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04f7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff050152220 0x7ff0501f7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:58700/0 (socket says 172.21.15.60:58700) 2024-03-27T21:42:05.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04f7fe640 1 -- 172.21.15.60:0/1944173661 learned_addr learned my addr 172.21.15.60:0/1944173661 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04f7fe640 10 monclient(hunting): get_auth_request con 0x7ff050152220 auth_method 0 2024-03-27T21:42:05.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04effd640 1 --2- 172.21.15.60:0/1944173661 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff05006d360 0x7ff05014fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04effd640 10 monclient(hunting): get_auth_request con 0x7ff05006d360 auth_method 0 2024-03-27T21:42:05.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04e7fc640 1 --2- 172.21.15.60:0/1944173661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff050151880 0x7ff0501f7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04e7fc640 10 monclient(hunting): get_auth_request con 0x7ff050151880 auth_method 0 2024-03-27T21:42:05.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f7b2b59e640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f7b2b59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:42:05.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04f7fe640 10 monclient(hunting): handle_auth_done global_id 5215 payload 824 2024-03-27T21:42:05.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7ff04f7fe640 1 -- 172.21.15.60:0/1944173661 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff05006d360 msgr2=0x7ff05014fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b2b59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b2c070d90 0x7f7b2c13cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f889d570640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f889d570640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:42:05.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f889d570640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8880005180 0x7f88980a9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f889d570640 10 monclient(hunting): picked mon.noname-b con 0x7f8880005180 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:05.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f889d570640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8898071290 0x7f88981e2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f889d570640 10 monclient(hunting): picked mon.noname-a con 0x7f8898071290 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:05.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f889d570640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8898148c50 0x7f88981e3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f889d570640 10 monclient(hunting): picked mon.noname-c con 0x7f8898148c50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f889d570640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f889d570640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8898071c90 con 0x7f8898071290 2024-03-27T21:42:05.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f889d570640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f889d570640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8898070b20 con 0x7f8880005180 2024-03-27T21:42:05.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f889d570640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f889d570640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8898071770 con 0x7f8898148c50 2024-03-27T21:42:05.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f889d570640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f889d570640 10 monclient(hunting): authenticate will time out at 966.324662s 2024-03-27T21:42:05.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b2b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f7b2c070d90 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:05.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b2b59e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7b2c0aeda0 0x7f7b2c1d9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7ff04f7fe640 1 --2- 172.21.15.60:0/1944173661 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff05006d360 0x7ff05014fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7ff04f7fe640 1 -- 172.21.15.60:0/1944173661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff050151880 msgr2=0x7ff0501f7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7ff04f7fe640 1 --2- 172.21.15.60:0/1944173661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff050151880 0x7ff0501f7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7ff04f7fe640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7ff04f7fe640 1 monclient: found mon.noname-b 2024-03-27T21:42:05.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7ff04f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7ff04f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7ff04f7fe640 1 -- 172.21.15.60:0/1944173661 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff04800c4d0 con 0x7ff050152220 2024-03-27T21:42:05.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7ff04effd640 1 --2- 172.21.15.60:0/1944173661 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7ff05006d360 0x7ff05014fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:05.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b2b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f7b2c0aeda0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f2124edb640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f2124edb640 10 monclient: monmap: 2024-03-27T21:42:05.292 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.292 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:05.292 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:05.292 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:05.292 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.292 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:05.292 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:05.292 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:05.292 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:05.293 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b2b59e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b2c13a900 0x7f7b2c1da060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f2124edb640 1 librados: starting msgr at 2024-03-27T21:42:05.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f2124edb640 1 librados: starting objecter 2024-03-27T21:42:05.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.102+0000 7f2124edb640 1 Processor -- start 2024-03-27T21:42:05.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f2124edb640 1 -- start start 2024-03-27T21:42:05.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f2124edb640 1 librados: setting wanted keys 2024-03-27T21:42:05.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f2124edb640 1 librados: calling monclient init 2024-03-27T21:42:05.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f2124edb640 10 monclient: init 2024-03-27T21:42:05.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f2124edb640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f2124edb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:42:05.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f2124edb640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2120060e10 0x7f2120154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f2124edb640 10 monclient(hunting): picked mon.noname-b con 0x7f2120060e10 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:05.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f2124edb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21200700c0 0x7f21200b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f211e59c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2120060e10 0x7f2120154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f211e59c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2120060e10 0x7f2120154b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:58710/0 (socket says 172.21.15.60:58710) 2024-03-27T21:42:05.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f211e59c640 1 -- 172.21.15.60:0/2051018026 learned_addr learned my addr 172.21.15.60:0/2051018026 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f2124edb640 10 monclient(hunting): picked mon.noname-a con 0x7f21200700c0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:05.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f2124edb640 1 --2- 172.21.15.60:0/2051018026 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f212013fe50 0x7f21201358b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f2124edb640 10 monclient(hunting): picked mon.noname-c con 0x7f212013fe50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f2124edb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b2b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f7b2c13a900 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:05.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f2124edb640 1 -- 172.21.15.60:0/2051018026 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2120132b70 con 0x7f21200700c0 2024-03-27T21:42:05.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f2124edb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f2124edb640 1 -- 172.21.15.60:0/2051018026 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2120135140 con 0x7f2120060e10 2024-03-27T21:42:05.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f2124edb640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f2124edb640 1 -- 172.21.15.60:0/2051018026 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2120135420 con 0x7f212013fe50 2024-03-27T21:42:05.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f2124edb640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f2124edb640 10 monclient(hunting): authenticate will time out at 966.325047s 2024-03-27T21:42:05.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f211e59c640 10 monclient(hunting): get_auth_request con 0x7f2120060e10 auth_method 0 2024-03-27T21:42:05.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f211e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f558bfff640 10 client.5209.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f211e59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f211e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f211e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f211e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f211e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f558bfff640 15 client.5209.objecter _session_command_op_assign -1 1 2024-03-27T21:42:05.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b2b59e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f558bfff640 20 client.5209.objecter _get_session osd=-1 returning homeless 2024-03-27T21:42:05.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f0bdf949640 1 -- 172.21.15.60:0/1338075049 wait complete. 2024-03-27T21:42:05.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f558bfff640 10 client.5209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b2b59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c0b2970 con 0x7f7b2c070d90 2024-03-27T21:42:05.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b29d9b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7b2c0aeda0 0x7f7b2c1d9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f0bdf949640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.103+0000 7f0bdf949640 10 monclient: monmap: 2024-03-27T21:42:05.301 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.301 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:05.301 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:05.301 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:05.301 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.302 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:05.302 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:05.302 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:05.302 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:05.302 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f0bdf949640 1 librados: starting msgr at 2024-03-27T21:42:05.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f0bdf949640 1 librados: starting objecter 2024-03-27T21:42:05.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f558bfff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b2b59e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f0bdf949640 1 Processor -- start 2024-03-27T21:42:05.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f558bfff640 1 -- 172.21.15.60:0/1084400107 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f55a002fe30 con 0x7f55a0039000 2024-03-27T21:42:05.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b29d9b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7b2c0aeda0 0x7f7b2c1d9b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34146/0 (socket says 172.21.15.60:34146) 2024-03-27T21:42:05.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b2b59e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c06ea70 con 0x7f7b2c13a900 2024-03-27T21:42:05.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b2b59e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b29d9b640 1 -- 172.21.15.60:0/3770112909 learned_addr learned my addr 172.21.15.60:0/3770112909 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b2b59e640 1 -- 172.21.15.60:0/3770112909 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c06ed50 con 0x7f7b2c0aeda0 2024-03-27T21:42:05.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b2b59e640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b2ad9d640 1 --2- 172.21.15.60:0/3770112909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b2c13a900 0x7f7b2c1da060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b2b59e640 10 monclient(hunting): authenticate will time out at 966.325457s 2024-03-27T21:42:05.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f211e59c640 10 monclient(hunting): handle_auth_done global_id 5218 payload 824 2024-03-27T21:42:05.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f211e59c640 1 -- 172.21.15.60:0/2051018026 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f212013fe50 msgr2=0x7f21201358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f211e59c640 1 --2- 172.21.15.60:0/2051018026 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f212013fe50 0x7f21201358b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f211e59c640 1 -- 172.21.15.60:0/2051018026 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21200700c0 msgr2=0x7f21200b25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f211e59c640 1 --2- 172.21.15.60:0/2051018026 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21200700c0 0x7f21200b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f211e59c640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f211e59c640 1 monclient: found mon.noname-b 2024-03-27T21:42:05.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f211e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f211e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f211e59c640 1 -- 172.21.15.60:0/2051018026 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2110009b30 con 0x7f2120060e10 2024-03-27T21:42:05.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7ff04f7fe640 1 --2- 172.21.15.60:0/1944173661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff050152220 0x7ff0501f7780 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff03800c3c0 tx=0x7ff03800c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7ff0477fe640 10 client.?.objecter ms_handle_connect 0x7ff050152220 2024-03-27T21:42:05.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7ff0477fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:05.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b29d9b640 10 monclient(hunting): get_auth_request con 0x7f7b2c0aeda0 auth_method 0 2024-03-27T21:42:05.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b29d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b29d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f7b29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b29d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b29d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b29d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f88977fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8898071290 0x7f88981e2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f88977fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8898071290 0x7f88981e2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39330/0 (socket says 172.21.15.60:39330) 2024-03-27T21:42:05.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f88977fe640 1 -- 172.21.15.60:0/530395410 learned_addr learned my addr 172.21.15.60:0/530395410 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f88977fe640 10 monclient(hunting): get_auth_request con 0x7f8898071290 auth_method 0 2024-03-27T21:42:05.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f88977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f88977fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f88977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32927fc640 1 -- 172.21.15.60:0/1304584283 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329403e450 con 0x7f329c0714e0 2024-03-27T21:42:05.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:05.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32927fc640 10 monclient: dump: 2024-03-27T21:42:05.310 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.310 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.311 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.311 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.311 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.311 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.311 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.311 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.311 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.311 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32927fc640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32927fc640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32a3f39640 5 monclient: authenticate success, global_id 5212 2024-03-27T21:42:05.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32a3f39640 10 monclient: _renew_subs 2024-03-27T21:42:05.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32a3f39640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32a3f39640 1 -- 172.21.15.60:0/1304584283 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f329c1da7e0 con 0x7f329c0714e0 2024-03-27T21:42:05.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32a3f39640 10 client.5212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32a3f39640 10 monclient: _renew_subs 2024-03-27T21:42:05.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32a3f39640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32a3f39640 1 -- 172.21.15.60:0/1304584283 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f329c143740 con 0x7f329c0714e0 2024-03-27T21:42:05.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32a3f39640 1 librados: init done 2024-03-27T21:42:05.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32927fc640 1 -- 172.21.15.60:0/1304584283 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329403e5d0 con 0x7f329c0714e0 2024-03-27T21:42:05.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32927fc640 1 -- 172.21.15.60:0/1304584283 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329403e980 con 0x7f329c0714e0 2024-03-27T21:42:05.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:42:05.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32927fc640 10 monclient: dump: 2024-03-27T21:42:05.313 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.314 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.314 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.314 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.314 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.314 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.314 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.314 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.314 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.314 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f32937fe640 4 set_mon_vals no callback set 2024-03-27T21:42:05.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f32a3f39640 10 client.5212.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f32a3f39640 15 client.5212.objecter _session_command_op_assign -1 1 2024-03-27T21:42:05.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f32a3f39640 20 client.5212.objecter _get_session osd=-1 returning homeless 2024-03-27T21:42:05.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f32a3f39640 10 client.5212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f32a3f39640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f32a3f39640 1 -- 172.21.15.60:0/1304584283 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f32840028a0 con 0x7f329c0714e0 2024-03-27T21:42:05.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b2a59c640 1 --2- 172.21.15.60:0/3770112909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b2c070d90 0x7f7b2c13cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b2a59c640 10 monclient(hunting): get_auth_request con 0x7f7b2c070d90 auth_method 0 2024-03-27T21:42:05.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b2a59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b29d9b640 10 monclient(hunting): handle_auth_done global_id 5018 payload 824 2024-03-27T21:42:05.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f0bdf949640 1 -- start start 2024-03-27T21:42:05.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f0bdf949640 1 librados: setting wanted keys 2024-03-27T21:42:05.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f0bdf949640 1 librados: calling monclient init 2024-03-27T21:42:05.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7f0bdf949640 10 monclient: init 2024-03-27T21:42:05.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdf949640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdf949640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:42:05.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdf949640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bd806d770 0x7f0bd81231a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdf949640 10 monclient(hunting): picked mon.noname-a con 0x7f0bd806d770 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:05.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdf949640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0bd8070600 0x7f0bd80abcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.104+0000 7fda7cd99640 4 set_mon_vals no callback set 2024-03-27T21:42:05.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdf949640 10 monclient(hunting): picked mon.noname-c con 0x7f0bd8070600 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdf949640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bd8140390 0x7f0bd81de1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f211e59c640 1 --2- 172.21.15.60:0/2051018026 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2120060e10 0x7f2120154b90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f211400c850 tx=0x7f211400cc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f210effd640 10 client.?.objecter ms_handle_connect 0x7f2120060e10 2024-03-27T21:42:05.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f210effd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:05.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdf949640 10 monclient(hunting): picked mon.noname-b con 0x7f0bd8140390 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:05.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdf949640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdf949640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd81ddca0 con 0x7f0bd806d770 2024-03-27T21:42:05.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdf949640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdf949640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd80ab120 con 0x7f0bd8140390 2024-03-27T21:42:05.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdf949640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdcee4640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0bd8070600 0x7f0bd80abcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7fda84ccf640 10 client.5301.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7fda84ccf640 15 client.5301.objecter _session_command_op_assign -1 1 2024-03-27T21:42:05.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7fda84ccf640 20 client.5301.objecter _get_session osd=-1 returning homeless 2024-03-27T21:42:05.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7fda84ccf640 10 client.5301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7fda84ccf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:05.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7fda84ccf640 1 -- 172.21.15.60:0/2943937655 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fda640028a0 con 0x7fda800714e0 2024-03-27T21:42:05.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdf949640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd80a9340 con 0x7f0bd8070600 2024-03-27T21:42:05.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdf949640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdf949640 10 monclient(hunting): authenticate will time out at 966.326702s 2024-03-27T21:42:05.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdcee4640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0bd8070600 0x7f0bd80abcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34164/0 (socket says 172.21.15.60:34164) 2024-03-27T21:42:05.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdcee4640 1 -- 172.21.15.60:0/3694384783 learned_addr learned my addr 172.21.15.60:0/3694384783 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdcee4640 10 monclient(hunting): get_auth_request con 0x7f0bd8070600 auth_method 0 2024-03-27T21:42:05.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdcee4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdcee4640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdcee4640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b29d9b640 1 -- 172.21.15.60:0/3770112909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b2c13a900 msgr2=0x7f7b2c1da060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b29d9b640 1 --2- 172.21.15.60:0/3770112909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b2c13a900 0x7f7b2c1da060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b29d9b640 1 -- 172.21.15.60:0/3770112909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b2c070d90 msgr2=0x7f7b2c13cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b29d9b640 1 --2- 172.21.15.60:0/3770112909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b2c070d90 0x7f7b2c13cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b29d9b640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b29d9b640 1 monclient: found mon.noname-c 2024-03-27T21:42:05.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b29d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b29d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:05.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b29d9b640 1 -- 172.21.15.60:0/3770112909 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b1c009400 con 0x7f7b2c0aeda0 2024-03-27T21:42:05.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b2ad9d640 10 monclient: get_auth_request con 0x7f7b2c13a900 auth_method 0 2024-03-27T21:42:05.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b2ad9d640 1 --2- 172.21.15.60:0/3770112909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b2c13a900 0x7f7b2c1da060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:42:05.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b29d9b640 1 --2- 172.21.15.60:0/3770112909 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7b2c0aeda0 0x7f7b2c1d9b20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c02b8f0 tx=0x7f7b1c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdcee4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdcee4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bdcee4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f0bddee6640 1 --2- 172.21.15.60:0/3694384783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bd8140390 0x7f0bd81de1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f889cd6f640 1 --2- 172.21.15.60:0/530395410 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8898148c50 0x7f88981e3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f889cd6f640 10 monclient(hunting): get_auth_request con 0x7f8898148c50 auth_method 0 2024-03-27T21:42:05.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f889cd6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b1affd640 10 client.?.objecter ms_handle_connect 0x7f7b2c0aeda0 2024-03-27T21:42:05.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f7b1affd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:05.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7ff0477fe640 1 -- 172.21.15.60:0/1944173661 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0380052d0 con 0x7ff050152220 2024-03-27T21:42:05.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7ff0477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7ff0477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:05.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7ff0477fe640 10 monclient: dump: 2024-03-27T21:42:05.326 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.326 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.326 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.327 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.327 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.327 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.327 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.327 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.327 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.327 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7ff0477fe640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7ff0477fe640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7ff0477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7ff0477fe640 1 -- 172.21.15.60:0/1944173661 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff038005430 con 0x7ff050152220 2024-03-27T21:42:05.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7ff0477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7ff0477fe640 1 -- 172.21.15.60:0/1944173661 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff038003850 con 0x7ff050152220 2024-03-27T21:42:05.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7ff0477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7ff0477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:42:05.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7ff0477fe640 10 monclient: dump: 2024-03-27T21:42:05.328 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.328 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.329 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f0bddee6640 10 monclient(hunting): get_auth_request con 0x7f0bd8140390 auth_method 0 2024-03-27T21:42:05.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f0bddee6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f0bddee6640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f0bddee6640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f889cd6f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f889cd6f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.105+0000 7f559e7fc640 1 -- 172.21.15.60:0/1084400107 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f55a8092560 con 0x7f55a0039000 2024-03-27T21:42:05.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f8897fff640 1 --2- 172.21.15.60:0/530395410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8880005180 0x7f88980a9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f889cd6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f889cd6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f889cd6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f8897fff640 10 monclient(hunting): get_auth_request con 0x7f8880005180 auth_method 0 2024-03-27T21:42:05.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f8897fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f8897fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f8897fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7ff055671640 5 monclient: authenticate success, global_id 5215 2024-03-27T21:42:05.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7ff055671640 10 monclient: _renew_subs 2024-03-27T21:42:05.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7ff055671640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7ff055671640 1 -- 172.21.15.60:0/1944173661 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0500aace0 con 0x7ff050152220 2024-03-27T21:42:05.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f0bddee6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f0bddee6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f0bddee6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f0bddee6640 10 monclient(hunting): handle_auth_done global_id 5221 payload 824 2024-03-27T21:42:05.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f559e7fc640 1 --2- 172.21.15.60:0/1084400107 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f558c06c690 0x7f558c06eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f0bddee6640 1 -- 172.21.15.60:0/3694384783 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0bd8070600 msgr2=0x7f0bd80abcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f0bdd6e5640 1 --2- 172.21.15.60:0/3694384783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bd806d770 0x7f0bd81231a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f0bddee6640 1 --2- 172.21.15.60:0/3694384783 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0bd8070600 0x7f0bd80abcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f0bddee6640 1 -- 172.21.15.60:0/3694384783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bd806d770 msgr2=0x7f0bd81231a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f0bddee6640 1 --2- 172.21.15.60:0/3694384783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bd806d770 0x7f0bd81231a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f0bddee6640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f0bddee6640 1 monclient: found mon.noname-b 2024-03-27T21:42:05.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f0bddee6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f0bddee6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f0bddee6640 1 -- 172.21.15.60:0/3694384783 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bcc009400 con 0x7f0bd8140390 2024-03-27T21:42:05.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b1affd640 1 -- 172.21.15.60:0/3770112909 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c003a20 con 0x7f7b2c0aeda0 2024-03-27T21:42:05.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b1affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:05.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b1affd640 10 monclient: dump: 2024-03-27T21:42:05.340 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.340 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.340 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.340 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.340 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.340 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.340 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.340 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.340 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.341 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b1affd640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b1affd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f0bdcee4640 1 --2- 172.21.15.60:0/3694384783 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0bd8070600 0x7f0bd80abcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:05.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f559e7fc640 1 -- 172.21.15.60:0/1084400107 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f55a8102350 con 0x7f55a0039000 2024-03-27T21:42:05.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f559e7fc640 10 client.5209.objecter ms_dispatch 0x7f55a8000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:05.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f559e7fc640 3 client.5209.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:05.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f559e7fc640 3 client.5209.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:05.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f559e7fc640 10 client.5209.objecter checking command 1 2024-03-27T21:42:05.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f55ade52640 1 --2- 172.21.15.60:0/1084400107 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f558c06c690 0x7f558c06eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f0bdd6e5640 10 monclient: get_auth_request con 0x7f0bd806d770 auth_method 0 2024-03-27T21:42:05.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f0bdd6e5640 1 --2- 172.21.15.60:0/3694384783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bd806d770 0x7f0bd81231a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:42:05.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f559e7fc640 1 --2- 172.21.15.60:0/1084400107 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f558c072ad0 0x7f558c074ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f559e7fc640 20 client.5209.objecter _get_session s=0x7f558c072400 osd=1 3 2024-03-27T21:42:05.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f559e7fc640 20 client.5209.objecter put_session s=0x7f558c072400 osd=1 3 2024-03-27T21:42:05.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f559e7fc640 15 client.5209.objecter _session_command_op_remove -1 1 2024-03-27T21:42:05.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f55ade52640 10 monclient: get_auth_request con 0x7f558c06c690 auth_method 0 2024-03-27T21:42:05.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f559e7fc640 20 client.5209.objecter _get_session s=0x7f558c072400 osd=1 3 2024-03-27T21:42:05.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f559e7fc640 20 client.5209.objecter get_session s=0x7f558c072400 osd=1 3 2024-03-27T21:42:05.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f559e7fc640 15 client.5209.objecter _session_command_op_assign 1 1 2024-03-27T21:42:05.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f559e7fc640 20 client.5209.objecter put_session s=0x7f558c072400 osd=1 4 2024-03-27T21:42:05.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f559e7fc640 10 client.5209.objecter _send_command 1 2024-03-27T21:42:05.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f559e7fc640 1 -- 172.21.15.60:0/1084400107 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f558c0755c0 con 0x7f558c072ad0 2024-03-27T21:42:05.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f559e7fc640 20 client.5209.objecter dump_active .. 0 homeless 2024-03-27T21:42:05.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f559e7fc640 1 -- 172.21.15.60:0/1084400107 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f55a8104970 con 0x7f55a0039000 2024-03-27T21:42:05.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b2b59e640 5 monclient: authenticate success, global_id 5018 2024-03-27T21:42:05.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b2b59e640 10 monclient: _renew_subs 2024-03-27T21:42:05.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b2b59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:05.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b2b59e640 1 -- 172.21.15.60:0/3770112909 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b2c06f030 con 0x7f7b2c0aeda0 2024-03-27T21:42:05.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b2b59e640 10 client.5018.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b2b59e640 10 monclient: _renew_subs 2024-03-27T21:42:05.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b2b59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:05.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b2b59e640 1 -- 172.21.15.60:0/3770112909 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b2c06d810 con 0x7f7b2c0aeda0 2024-03-27T21:42:05.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b2b59e640 1 librados: init done 2024-03-27T21:42:05.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f559e7fc640 10 monclient: handle_get_version_reply finishing 1 version 92 2024-03-27T21:42:05.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7ff055671640 10 client.5215.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7ff055671640 10 monclient: _renew_subs 2024-03-27T21:42:05.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7ff055671640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7ff055671640 1 -- 172.21.15.60:0/1944173661 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0500aa3e0 con 0x7ff050152220 2024-03-27T21:42:05.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7ff055671640 1 librados: init done 2024-03-27T21:42:05.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f8897fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f8897fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f8897fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f8897fff640 10 monclient(hunting): handle_auth_done global_id 5224 payload 824 2024-03-27T21:42:05.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f8897fff640 1 -- 172.21.15.60:0/530395410 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8898148c50 msgr2=0x7f88981e3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f8897fff640 1 --2- 172.21.15.60:0/530395410 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8898148c50 0x7f88981e3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f8897fff640 1 -- 172.21.15.60:0/530395410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8898071290 msgr2=0x7f88981e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f8897fff640 1 --2- 172.21.15.60:0/530395410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8898071290 0x7f88981e2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f8897fff640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f8897fff640 1 monclient: found mon.noname-b 2024-03-27T21:42:05.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f8897fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f8897fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f8897fff640 1 -- 172.21.15.60:0/530395410 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88981e3750 con 0x7f8880005180 2024-03-27T21:42:05.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b1affd640 1 -- 172.21.15.60:0/3770112909 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1c03e070 con 0x7f7b2c0aeda0 2024-03-27T21:42:05.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b1affd640 1 -- 172.21.15.60:0/3770112909 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c003d80 con 0x7f7b2c0aeda0 2024-03-27T21:42:05.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:05.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f7b1affd640 10 monclient: dump: 2024-03-27T21:42:05.350 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.350 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.350 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.350 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.350 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.350 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.350 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.350 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.351 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.351 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.106+0000 7f8897fff640 1 --2- 172.21.15.60:0/530395410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8880005180 0x7f88980a9b40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f888800f650 tx=0x7f888800fb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f8894ff9640 10 client.?.objecter ms_handle_connect 0x7f8880005180 2024-03-27T21:42:05.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f8894ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:05.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f7b28d99640 4 set_mon_vals no callback set 2024-03-27T21:42:05.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f55ade52640 1 --2- 172.21.15.60:0/1084400107 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f558c06c690 0x7f558c06eb50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f55a40099a0 tx=0x7f55a4006f90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f559e7fc640 10 client.5209.objecter ms_handle_connect 0x7f558c06c690 2024-03-27T21:42:05.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f55ad651640 1 --2- 172.21.15.60:0/1084400107 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f558c072ad0 0x7f558c074ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f55ad651640 10 monclient: get_auth_request con 0x7f558c072ad0 auth_method 0 2024-03-27T21:42:05.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f210effd640 1 -- 172.21.15.60:0/2051018026 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2114007a30 con 0x7f2120060e10 2024-03-27T21:42:05.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f210effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f210effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:05.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f210effd640 10 monclient: dump: 2024-03-27T21:42:05.353 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.353 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.353 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.354 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.354 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.354 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.354 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.354 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.354 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.354 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f210effd640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f210effd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f210effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f2124edb640 5 monclient: authenticate success, global_id 5218 2024-03-27T21:42:05.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f2124edb640 10 monclient: _renew_subs 2024-03-27T21:42:05.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f2124edb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f2124edb640 1 -- 172.21.15.60:0/2051018026 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f212014e650 con 0x7f2120060e10 2024-03-27T21:42:05.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f2124edb640 10 client.5218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f2124edb640 10 monclient: _renew_subs 2024-03-27T21:42:05.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f2124edb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f2124edb640 1 -- 172.21.15.60:0/2051018026 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21200aa5f0 con 0x7f2120060e10 2024-03-27T21:42:05.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f2124edb640 1 librados: init done 2024-03-27T21:42:05.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f32927fc640 1 -- 172.21.15.60:0/1304584283 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f329403eae0 con 0x7f329c0714e0 2024-03-27T21:42:05.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f0bddee6640 1 --2- 172.21.15.60:0/3694384783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bd8140390 0x7f0bd81de1c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0bd000efc0 tx=0x7f0bd000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f0bd5ffb640 10 client.?.objecter ms_handle_connect 0x7f0bd8140390 2024-03-27T21:42:05.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f0bd5ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:05.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f210effd640 1 -- 172.21.15.60:0/2051018026 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2114007b90 con 0x7f2120060e10 2024-03-27T21:42:05.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f210effd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f210effd640 1 -- 172.21.15.60:0/2051018026 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211400cea0 con 0x7f2120060e10 2024-03-27T21:42:05.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f210effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f210effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:42:05.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f210effd640 10 monclient: dump: 2024-03-27T21:42:05.358 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.358 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.358 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.358 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.358 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.358 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.358 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.359 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.359 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.359 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f210ffff640 4 set_mon_vals no callback set 2024-03-27T21:42:05.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f0bd5ffb640 1 -- 172.21.15.60:0/3694384783 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd0009c00 con 0x7f0bd8140390 2024-03-27T21:42:05.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f0bd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f0bd5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:05.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f0bd5ffb640 10 monclient: dump: 2024-03-27T21:42:05.360 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.361 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.361 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.361 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.361 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.361 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.361 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.361 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.361 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.361 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f0bd5ffb640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f0bd5ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f0bd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f0bd5ffb640 1 -- 172.21.15.60:0/3694384783 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd000f040 con 0x7f0bd8140390 2024-03-27T21:42:05.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f0bdf949640 5 monclient: authenticate success, global_id 5221 2024-03-27T21:42:05.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f32927fc640 1 --2- 172.21.15.60:0/1304584283 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f328006c660 0x7f328006eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f32927fc640 1 -- 172.21.15.60:0/1304584283 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f32940ac540 con 0x7f329c0714e0 2024-03-27T21:42:05.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f32927fc640 10 client.5212.objecter ms_dispatch 0x7f329c000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:05.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f32927fc640 3 client.5212.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:05.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f32927fc640 3 client.5212.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:05.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f32927fc640 10 client.5212.objecter checking command 1 2024-03-27T21:42:05.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f32927fc640 1 --2- 172.21.15.60:0/1304584283 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f3280072aa0 0x7f3280074ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f32927fc640 20 client.5212.objecter _get_session s=0x7f32800723d0 osd=2 3 2024-03-27T21:42:05.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f32927fc640 20 client.5212.objecter put_session s=0x7f32800723d0 osd=2 3 2024-03-27T21:42:05.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f32927fc640 15 client.5212.objecter _session_command_op_remove -1 1 2024-03-27T21:42:05.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f32927fc640 20 client.5212.objecter _get_session s=0x7f32800723d0 osd=2 3 2024-03-27T21:42:05.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f32927fc640 20 client.5212.objecter get_session s=0x7f32800723d0 osd=2 3 2024-03-27T21:42:05.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f32927fc640 15 client.5212.objecter _session_command_op_assign 2 1 2024-03-27T21:42:05.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f32927fc640 20 client.5212.objecter put_session s=0x7f32800723d0 osd=2 4 2024-03-27T21:42:05.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f32927fc640 10 client.5212.objecter _send_command 1 2024-03-27T21:42:05.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f32927fc640 1 -- 172.21.15.60:0/1304584283 --> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3280075590 con 0x7f3280072aa0 2024-03-27T21:42:05.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f32927fc640 20 client.5212.objecter dump_active .. 0 homeless 2024-03-27T21:42:05.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f32927fc640 1 -- 172.21.15.60:0/1304584283 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32940ac8f0 con 0x7f329c0714e0 2024-03-27T21:42:05.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f32927fc640 10 monclient: handle_get_version_reply finishing 1 version 92 2024-03-27T21:42:05.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7ff04d7fa640 4 set_mon_vals no callback set 2024-03-27T21:42:05.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7ff055671640 10 client.5215.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7ff055671640 15 client.5215.objecter _session_command_op_assign -1 1 2024-03-27T21:42:05.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7ff055671640 20 client.5215.objecter _get_session osd=-1 returning homeless 2024-03-27T21:42:05.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7ff055671640 10 client.5215.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7ff055671640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7ff055671640 1 -- 172.21.15.60:0/1944173661 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff05006e100 con 0x7ff050152220 2024-03-27T21:42:05.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7ff0477fe640 1 -- 172.21.15.60:0/1944173661 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff0380039b0 con 0x7ff050152220 2024-03-27T21:42:05.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f0bdf949640 10 monclient: _renew_subs 2024-03-27T21:42:05.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f8894ff9640 1 -- 172.21.15.60:0/530395410 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888800da40 con 0x7f8880005180 2024-03-27T21:42:05.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f8894ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f8894ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:05.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f8894ff9640 10 monclient: dump: 2024-03-27T21:42:05.367 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.367 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.367 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.367 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.367 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.368 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.368 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.368 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.368 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.368 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f8894ff9640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f8894ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f32a14d4640 1 --2- 172.21.15.60:0/1304584283 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f328006c660 0x7f328006eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f8894ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f8894ff9640 1 -- 172.21.15.60:0/530395410 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888800dba0 con 0x7f8880005180 2024-03-27T21:42:05.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f8894ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f8894ff9640 1 -- 172.21.15.60:0/530395410 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8888012460 con 0x7f8880005180 2024-03-27T21:42:05.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f8894ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f8894ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:42:05.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f8894ff9640 10 monclient: dump: 2024-03-27T21:42:05.370 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.370 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.370 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.370 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.370 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.371 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.371 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.371 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.371 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.371 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f889d570640 5 monclient: authenticate success, global_id 5224 2024-03-27T21:42:05.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f889d570640 10 monclient: _renew_subs 2024-03-27T21:42:05.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f889d570640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f889d570640 1 -- 172.21.15.60:0/530395410 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88981e61d0 con 0x7f8880005180 2024-03-27T21:42:05.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f88967fc640 4 set_mon_vals no callback set 2024-03-27T21:42:05.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f889d570640 10 client.5224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f889d570640 10 monclient: _renew_subs 2024-03-27T21:42:05.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f889d570640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f889d570640 1 -- 172.21.15.60:0/530395410 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88981e6630 con 0x7f8880005180 2024-03-27T21:42:05.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f32a14d4640 10 monclient: get_auth_request con 0x7f328006c660 auth_method 0 2024-03-27T21:42:05.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7f889d570640 1 librados: init done 2024-03-27T21:42:05.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f889d570640 10 client.5224.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f889d570640 15 client.5224.objecter _session_command_op_assign -1 1 2024-03-27T21:42:05.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f889d570640 20 client.5224.objecter _get_session osd=-1 returning homeless 2024-03-27T21:42:05.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f889d570640 10 client.5224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f889d570640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f889d570640 1 -- 172.21.15.60:0/530395410 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f889814e5d0 con 0x7f8880005180 2024-03-27T21:42:05.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f0bdf949640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f0bdf949640 1 -- 172.21.15.60:0/3694384783 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd80ac230 con 0x7f0bd8140390 2024-03-27T21:42:05.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7fda6effd640 1 -- 172.21.15.60:0/2943937655 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fda740443f0 con 0x7fda800714e0 2024-03-27T21:42:05.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7fda6effd640 1 --2- 172.21.15.60:0/2943937655 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fda6006c660 0x7fda6006eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7fda6effd640 1 -- 172.21.15.60:0/2943937655 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fda7402d040 con 0x7fda800714e0 2024-03-27T21:42:05.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7fda6effd640 10 client.5301.objecter ms_dispatch 0x7fda80000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:05.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7fda6effd640 3 client.5301.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:05.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7fda6effd640 3 client.5301.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:05.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7fda6effd640 10 client.5301.objecter checking command 1 2024-03-27T21:42:05.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7fda6effd640 1 --2- 172.21.15.60:0/2943937655 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fda60072aa0 0x7fda60074ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7fda6effd640 20 client.5301.objecter _get_session s=0x7fda600723d0 osd=3 3 2024-03-27T21:42:05.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7fda6effd640 20 client.5301.objecter put_session s=0x7fda600723d0 osd=3 3 2024-03-27T21:42:05.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7fda6effd640 15 client.5301.objecter _session_command_op_remove -1 1 2024-03-27T21:42:05.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7fda6effd640 20 client.5301.objecter _get_session s=0x7fda600723d0 osd=3 3 2024-03-27T21:42:05.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7fda6effd640 20 client.5301.objecter get_session s=0x7fda600723d0 osd=3 3 2024-03-27T21:42:05.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7fda6effd640 15 client.5301.objecter _session_command_op_assign 3 1 2024-03-27T21:42:05.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7fda6effd640 20 client.5301.objecter put_session s=0x7fda600723d0 osd=3 4 2024-03-27T21:42:05.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7fda6effd640 10 client.5301.objecter _send_command 1 2024-03-27T21:42:05.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7fda6effd640 1 -- 172.21.15.60:0/2943937655 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda60075590 con 0x7fda60072aa0 2024-03-27T21:42:05.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.107+0000 7fda6effd640 20 client.5301.objecter dump_active .. 0 homeless 2024-03-27T21:42:05.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7fda6effd640 1 -- 172.21.15.60:0/2943937655 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda74031d60 con 0x7fda800714e0 2024-03-27T21:42:05.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7fda6effd640 10 monclient: handle_get_version_reply finishing 1 version 92 2024-03-27T21:42:05.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7fda7ed9d640 1 --2- 172.21.15.60:0/2943937655 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fda60072aa0 0x7fda60074ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7fda7ed9d640 10 monclient: get_auth_request con 0x7fda60072aa0 auth_method 0 2024-03-27T21:42:05.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f0bd5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f55ad651640 1 --2- 172.21.15.60:0/1084400107 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f558c072ad0 0x7f558c074ef0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f7b2b59e640 10 client.5018.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f7b2b59e640 15 client.5018.objecter _session_command_op_assign -1 1 2024-03-27T21:42:05.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f0bd5ffb640 1 -- 172.21.15.60:0/3694384783 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd00146c0 con 0x7f0bd8140390 2024-03-27T21:42:05.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f0bdf949640 10 client.5221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f0bdf949640 10 monclient: _renew_subs 2024-03-27T21:42:05.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f7b2b59e640 20 client.5018.objecter _get_session osd=-1 returning homeless 2024-03-27T21:42:05.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f7b2b59e640 10 client.5018.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f7b2b59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:05.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.108+0000 7f7b2b59e640 1 -- 172.21.15.60:0/3770112909 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7b100028a0 con 0x7f7b2c0aeda0 2024-03-27T21:42:05.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f7b1affd640 1 -- 172.21.15.60:0/3770112909 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7b1c04d450 con 0x7f7b2c0aeda0 2024-03-27T21:42:05.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f7b1affd640 1 --2- 172.21.15.60:0/3770112909 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f7b0c06c5e0 0x7f7b0c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7fda7ed9d640 1 --2- 172.21.15.60:0/2943937655 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fda60072aa0 0x7fda60074ec0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f7b1affd640 1 -- 172.21.15.60:0/3770112909 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f7b1c0aaeb0 con 0x7f7b2c0aeda0 2024-03-27T21:42:05.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f7b1affd640 10 client.5018.objecter ms_dispatch 0x7f7b2c000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:05.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f7b1affd640 3 client.5018.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:05.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f7b1affd640 3 client.5018.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:05.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f32a14d4640 1 --2- 172.21.15.60:0/1304584283 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f328006c660 0x7f328006eb20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f328c006fd0 tx=0x7f328c006b40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f32927fc640 10 client.5212.objecter ms_handle_connect 0x7f328006c660 2024-03-27T21:42:05.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f0bdf949640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f0bdf949640 1 -- 172.21.15.60:0/3694384783 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd81de8e0 con 0x7f0bd8140390 2024-03-27T21:42:05.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f0bdf949640 1 librados: init done 2024-03-27T21:42:05.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f0bd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f0bd5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:42:05.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f0bd5ffb640 10 monclient: dump: 2024-03-27T21:42:05.382 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.382 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.383 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.383 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.383 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.383 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.383 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.383 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.383 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.383 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f559e7fc640 10 client.5209.objecter ms_handle_connect 0x7f558c072ad0 2024-03-27T21:42:05.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f8894ff9640 1 -- 172.21.15.60:0/530395410 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8888005990 con 0x7f8880005180 2024-03-27T21:42:05.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7f0bd77fe640 4 set_mon_vals no callback set 2024-03-27T21:42:05.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f559e7fc640 1 -- 172.21.15.60:0/1084400107 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f558c0755c0 con 0x7f558c072ad0 2024-03-27T21:42:05.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f559e7fc640 10 client.5209.objecter ms_dispatch 0x7f55a8000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:42:05.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f559e7fc640 10 client.5209.objecter _finish_command 1 = system:0 2024-03-27T21:42:05.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f559e7fc640 20 client.5209.objecter put_session s=0x7f558c072400 osd=1 4 2024-03-27T21:42:05.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f559e7fc640 15 client.5209.objecter _session_command_op_remove 1 1 2024-03-27T21:42:05.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f8894ff9640 1 --2- 172.21.15.60:0/530395410 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f888406c610 0x7f888406ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f0bdf949640 10 client.5221.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7fda6effd640 10 client.5301.objecter ms_handle_connect 0x7fda60072aa0 2024-03-27T21:42:05.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f0bdf949640 15 client.5221.objecter _session_command_op_assign -1 1 2024-03-27T21:42:05.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f8894ff9640 1 -- 172.21.15.60:0/530395410 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f88880893d0 con 0x7f8880005180 2024-03-27T21:42:05.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f8894ff9640 10 client.5224.objecter ms_dispatch 0x7f8898000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:05.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f8894ff9640 3 client.5224.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:05.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f8894ff9640 3 client.5224.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:05.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7fda7dd9b640 1 --2- 172.21.15.60:0/2943937655 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fda6006c660 0x7fda6006eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7fda7dd9b640 10 monclient: get_auth_request con 0x7fda6006c660 auth_method 0 2024-03-27T21:42:05.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f0bdf949640 20 client.5221.objecter _get_session osd=-1 returning homeless 2024-03-27T21:42:05.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f8894ff9640 10 client.5224.objecter checking command 1 2024-03-27T21:42:05.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f8894ff9640 1 --2- 172.21.15.60:0/530395410 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f8884072a50 0x7f8884074e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f8894ff9640 20 client.5224.objecter _get_session s=0x7f8884072380 osd=0 3 2024-03-27T21:42:05.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f8894ff9640 20 client.5224.objecter put_session s=0x7f8884072380 osd=0 3 2024-03-27T21:42:05.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f8894ff9640 15 client.5224.objecter _session_command_op_remove -1 1 2024-03-27T21:42:05.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f2124edb640 10 client.5218.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f2124edb640 15 client.5218.objecter _session_command_op_assign -1 1 2024-03-27T21:42:05.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f2124edb640 20 client.5218.objecter _get_session osd=-1 returning homeless 2024-03-27T21:42:05.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f2124edb640 10 client.5218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f2124edb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f2124edb640 1 -- 172.21.15.60:0/2051018026 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f21201407b0 con 0x7f2120060e10 2024-03-27T21:42:05.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f8894ff9640 20 client.5224.objecter _get_session s=0x7f8884072380 osd=0 3 2024-03-27T21:42:05.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f8894ff9640 20 client.5224.objecter get_session s=0x7f8884072380 osd=0 3 2024-03-27T21:42:05.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f8894ff9640 15 client.5224.objecter _session_command_op_assign 0 1 2024-03-27T21:42:05.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f8894ff9640 20 client.5224.objecter put_session s=0x7f8884072380 osd=0 4 2024-03-27T21:42:05.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f8894ff9640 10 client.5224.objecter _send_command 1 2024-03-27T21:42:05.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f8894ff9640 1 -- 172.21.15.60:0/530395410 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8884075540 con 0x7f8884072a50 2024-03-27T21:42:05.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f8894ff9640 20 client.5224.objecter dump_active .. 0 homeless 2024-03-27T21:42:05.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f8894ff9640 1 -- 172.21.15.60:0/530395410 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8888089870 con 0x7f8880005180 2024-03-27T21:42:05.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f8894ff9640 10 monclient: handle_get_version_reply finishing 1 version 92 2024-03-27T21:42:05.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f0bdf949640 10 client.5221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7ff0477fe640 1 --2- 172.21.15.60:0/1944173661 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff03006c660 0x7ff03006eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7ff0477fe640 1 -- 172.21.15.60:0/1944173661 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7ff038088510 con 0x7ff050152220 2024-03-27T21:42:05.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7ff0477fe640 10 client.5215.objecter ms_dispatch 0x7ff050000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:05.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7ff0477fe640 3 client.5215.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:05.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7ff0477fe640 3 client.5215.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:05.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7ff0477fe640 10 client.5215.objecter checking command 1 2024-03-27T21:42:05.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7ff0477fe640 1 --2- 172.21.15.60:0/1944173661 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7ff030072aa0 0x7ff030074ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7ff0477fe640 20 client.5215.objecter _get_session s=0x7ff0300723d0 osd=5 3 2024-03-27T21:42:05.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7ff0477fe640 20 client.5215.objecter put_session s=0x7ff0300723d0 osd=5 3 2024-03-27T21:42:05.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7ff0477fe640 15 client.5215.objecter _session_command_op_remove -1 1 2024-03-27T21:42:05.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7ff0477fe640 20 client.5215.objecter _get_session s=0x7ff0300723d0 osd=5 3 2024-03-27T21:42:05.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7ff0477fe640 20 client.5215.objecter get_session s=0x7ff0300723d0 osd=5 3 2024-03-27T21:42:05.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7ff0477fe640 15 client.5215.objecter _session_command_op_assign 5 1 2024-03-27T21:42:05.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7ff0477fe640 20 client.5215.objecter put_session s=0x7ff0300723d0 osd=5 4 2024-03-27T21:42:05.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7ff0477fe640 10 client.5215.objecter _send_command 1 2024-03-27T21:42:05.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7ff0477fe640 1 -- 172.21.15.60:0/1944173661 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff030075590 con 0x7ff030072aa0 2024-03-27T21:42:05.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7ff0477fe640 20 client.5215.objecter dump_active .. 0 homeless 2024-03-27T21:42:05.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7ff0477fe640 1 -- 172.21.15.60:0/1944173661 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff038053950 con 0x7ff050152220 2024-03-27T21:42:05.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.109+0000 7ff0477fe640 10 monclient: handle_get_version_reply finishing 1 version 92 2024-03-27T21:42:05.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f0bdf949640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f0bdf949640 1 -- 172.21.15.60:0/3694384783 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0bd806e5a0 con 0x7f0bd8140390 2024-03-27T21:42:05.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f7b1affd640 10 client.5018.objecter checking command 1 2024-03-27T21:42:05.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f7b1affd640 1 --2- 172.21.15.60:0/3770112909 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f7b0c072a20 0x7f7b0c074e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f7b1affd640 20 client.5018.objecter _get_session s=0x7f7b0c072350 osd=4 3 2024-03-27T21:42:05.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f7b1affd640 20 client.5018.objecter put_session s=0x7f7b0c072350 osd=4 3 2024-03-27T21:42:05.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f7b1affd640 15 client.5018.objecter _session_command_op_remove -1 1 2024-03-27T21:42:05.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f7b1affd640 20 client.5018.objecter _get_session s=0x7f7b0c072350 osd=4 3 2024-03-27T21:42:05.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f7b1affd640 20 client.5018.objecter get_session s=0x7f7b0c072350 osd=4 3 2024-03-27T21:42:05.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f7b1affd640 15 client.5018.objecter _session_command_op_assign 4 1 2024-03-27T21:42:05.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f7b1affd640 20 client.5018.objecter put_session s=0x7f7b0c072350 osd=4 4 2024-03-27T21:42:05.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f7b1affd640 10 client.5018.objecter _send_command 1 2024-03-27T21:42:05.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f7b1affd640 1 -- 172.21.15.60:0/3770112909 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7b0c075510 con 0x7f7b0c072a20 2024-03-27T21:42:05.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f7b1affd640 20 client.5018.objecter dump_active .. 0 homeless 2024-03-27T21:42:05.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f7b1affd640 1 -- 172.21.15.60:0/3770112909 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b1c0ad5c0 con 0x7f7b2c0aeda0 2024-03-27T21:42:05.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f7b1affd640 10 monclient: handle_get_version_reply finishing 1 version 92 2024-03-27T21:42:05.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f7b2ad9d640 1 --2- 172.21.15.60:0/3770112909 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f7b0c072a20 0x7f7b0c074e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f7b2ad9d640 10 monclient: get_auth_request con 0x7f7b0c072a20 auth_method 0 2024-03-27T21:42:05.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.110+0000 7f7b2ad9d640 1 --2- 172.21.15.60:0/3770112909 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f7b0c072a20 0x7f7b0c074e40 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f88977fe640 1 --2- 172.21.15.60:0/530395410 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f888406c610 0x7f888406ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f889cd6f640 1 --2- 172.21.15.60:0/530395410 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f8884072a50 0x7f8884074e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7ff04effd640 1 --2- 172.21.15.60:0/1944173661 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff03006c660 0x7ff03006eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7ff04effd640 10 monclient: get_auth_request con 0x7ff03006c660 auth_method 0 2024-03-27T21:42:05.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7ff04e7fc640 1 --2- 172.21.15.60:0/1944173661 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7ff030072aa0 0x7ff030074ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7ff04e7fc640 10 monclient: get_auth_request con 0x7ff030072aa0 auth_method 0 2024-03-27T21:42:05.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f88977fe640 10 monclient: get_auth_request con 0x7f888406c610 auth_method 0 2024-03-27T21:42:05.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f32a24d6640 1 --2- 172.21.15.60:0/1304584283 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f3280072aa0 0x7f3280074ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f32a24d6640 10 monclient: get_auth_request con 0x7f3280072aa0 auth_method 0 2024-03-27T21:42:05.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f7b1affd640 10 client.5018.objecter ms_handle_connect 0x7f7b0c072a20 2024-03-27T21:42:05.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f7b1affd640 1 -- 172.21.15.60:0/3770112909 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7b0c075510 con 0x7f7b0c072a20 2024-03-27T21:42:05.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f7b1affd640 10 client.5018.objecter ms_dispatch 0x7f7b2c000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:42:05.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f7b1affd640 10 client.5018.objecter _finish_command 1 = system:0 2024-03-27T21:42:05.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.112+0000 7fda6effd640 1 -- 172.21.15.60:0/2943937655 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fda60075590 con 0x7fda60072aa0 2024-03-27T21:42:05.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.112+0000 7fda6effd640 10 client.5301.objecter ms_dispatch 0x7fda80000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:42:05.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.112+0000 7fda6effd640 10 client.5301.objecter _finish_command 1 = system:0 2024-03-27T21:42:05.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f7b1affd640 20 client.5018.objecter put_session s=0x7f7b0c072350 osd=4 4 2024-03-27T21:42:05.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f7b1affd640 15 client.5018.objecter _session_command_op_remove 4 1 2024-03-27T21:42:05.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f7b2a59c640 1 --2- 172.21.15.60:0/3770112909 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f7b0c06c5e0 0x7f7b0c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.111+0000 7f7b2a59c640 10 monclient: get_auth_request con 0x7f7b0c06c5e0 auth_method 0 2024-03-27T21:42:05.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.112+0000 7fda6effd640 20 client.5301.objecter put_session s=0x7fda600723d0 osd=3 4 2024-03-27T21:42:05.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.112+0000 7fda6effd640 15 client.5301.objecter _session_command_op_remove 3 1 2024-03-27T21:42:05.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.112+0000 7f210effd640 1 -- 172.21.15.60:0/2051018026 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f21140113f0 con 0x7f2120060e10 2024-03-27T21:42:05.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.112+0000 7ff04effd640 1 --2- 172.21.15.60:0/1944173661 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff03006c660 0x7ff03006eb20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff04000d790 tx=0x7ff04000d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.112+0000 7ff04e7fc640 1 --2- 172.21.15.60:0/1944173661 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7ff030072aa0 0x7ff030074ec0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.112+0000 7ff0477fe640 10 client.5215.objecter ms_handle_connect 0x7ff03006c660 2024-03-27T21:42:05.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.112+0000 7ff0477fe640 10 client.5215.objecter ms_handle_connect 0x7ff030072aa0 2024-03-27T21:42:05.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.112+0000 7fda7dd9b640 1 --2- 172.21.15.60:0/2943937655 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fda6006c660 0x7fda6006eb20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fda80141390 tx=0x7fda78007dd0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.112+0000 7fda6effd640 10 client.5301.objecter ms_handle_connect 0x7fda6006c660 2024-03-27T21:42:05.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.112+0000 7f210effd640 1 --2- 172.21.15.60:0/2051018026 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f210006c660 0x7f210006eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.112+0000 7f0bd5ffb640 1 -- 172.21.15.60:0/3694384783 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0bd0014820 con 0x7f0bd8140390 2024-03-27T21:42:05.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f7b2a59c640 1 --2- 172.21.15.60:0/3770112909 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f7b0c06c5e0 0x7f7b0c06eaa0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7b2000a980 tx=0x7f7b20008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f7b1affd640 10 client.5018.objecter ms_handle_connect 0x7f7b0c06c5e0 2024-03-27T21:42:05.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f0bd5ffb640 1 --2- 172.21.15.60:0/3694384783 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f0bc406c5e0 0x7f0bc406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f0bd5ffb640 1 -- 172.21.15.60:0/3694384783 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f0bd0089eb0 con 0x7f0bd8140390 2024-03-27T21:42:05.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f0bd5ffb640 10 client.5221.objecter ms_dispatch 0x7f0bd8000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:05.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7ff0477fe640 1 -- 172.21.15.60:0/1944173661 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7ff030075590 con 0x7ff030072aa0 2024-03-27T21:42:05.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7ff0477fe640 10 client.5215.objecter ms_dispatch 0x7ff050000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:42:05.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7ff0477fe640 10 client.5215.objecter _finish_command 1 = system:0 2024-03-27T21:42:05.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7ff0477fe640 20 client.5215.objecter put_session s=0x7ff0300723d0 osd=5 4 2024-03-27T21:42:05.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7ff0477fe640 15 client.5215.objecter _session_command_op_remove 5 1 2024-03-27T21:42:05.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f0bd5ffb640 3 client.5221.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:05.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f210effd640 1 -- 172.21.15.60:0/2051018026 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f2114089340 con 0x7f2120060e10 2024-03-27T21:42:05.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f210effd640 10 client.5218.objecter ms_dispatch 0x7f2120000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:05.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f210effd640 3 client.5218.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:05.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f210effd640 3 client.5218.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:05.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f210effd640 10 client.5218.objecter checking command 1 2024-03-27T21:42:05.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f210effd640 1 --2- 172.21.15.60:0/2051018026 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f2100072aa0 0x7f2100074ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f210effd640 20 client.5218.objecter _get_session s=0x7f21000723d0 osd=6 3 2024-03-27T21:42:05.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f210effd640 20 client.5218.objecter put_session s=0x7f21000723d0 osd=6 3 2024-03-27T21:42:05.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f210effd640 15 client.5218.objecter _session_command_op_remove -1 1 2024-03-27T21:42:05.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f210effd640 20 client.5218.objecter _get_session s=0x7f21000723d0 osd=6 3 2024-03-27T21:42:05.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f210effd640 20 client.5218.objecter get_session s=0x7f21000723d0 osd=6 3 2024-03-27T21:42:05.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f210effd640 15 client.5218.objecter _session_command_op_assign 6 1 2024-03-27T21:42:05.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f210effd640 20 client.5218.objecter put_session s=0x7f21000723d0 osd=6 4 2024-03-27T21:42:05.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f210effd640 10 client.5218.objecter _send_command 1 2024-03-27T21:42:05.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f211dd9b640 1 --2- 172.21.15.60:0/2051018026 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f210006c660 0x7f210006eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f210effd640 1 -- 172.21.15.60:0/2051018026 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2100075590 con 0x7f2100072aa0 2024-03-27T21:42:05.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f210effd640 20 client.5218.objecter dump_active .. 0 homeless 2024-03-27T21:42:05.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f210effd640 1 -- 172.21.15.60:0/2051018026 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f211408b960 con 0x7f2120060e10 2024-03-27T21:42:05.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f210effd640 10 monclient: handle_get_version_reply finishing 1 version 92 2024-03-27T21:42:05.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f211dd9b640 10 monclient: get_auth_request con 0x7f210006c660 auth_method 0 2024-03-27T21:42:05.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f0bd5ffb640 3 client.5221.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:05.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f0bd5ffb640 10 client.5221.objecter checking command 1 2024-03-27T21:42:05.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f0bd5ffb640 1 --2- 172.21.15.60:0/3694384783 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f0bc4072a20 0x7f0bc4074e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f0bd5ffb640 20 client.5221.objecter _get_session s=0x7f0bc4072350 osd=7 3 2024-03-27T21:42:05.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f32a24d6640 1 --2- 172.21.15.60:0/1304584283 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f3280072aa0 0x7f3280074ec0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f32927fc640 10 client.5212.objecter ms_handle_connect 0x7f3280072aa0 2024-03-27T21:42:05.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f0bd5ffb640 20 client.5221.objecter put_session s=0x7f0bc4072350 osd=7 3 2024-03-27T21:42:05.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f0bd5ffb640 15 client.5221.objecter _session_command_op_remove -1 1 2024-03-27T21:42:05.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f0bd5ffb640 20 client.5221.objecter _get_session s=0x7f0bc4072350 osd=7 3 2024-03-27T21:42:05.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f0bd5ffb640 20 client.5221.objecter get_session s=0x7f0bc4072350 osd=7 3 2024-03-27T21:42:05.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f0bd5ffb640 15 client.5221.objecter _session_command_op_assign 7 1 2024-03-27T21:42:05.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f0bd5ffb640 20 client.5221.objecter put_session s=0x7f0bc4072350 osd=7 4 2024-03-27T21:42:05.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f0bd5ffb640 10 client.5221.objecter _send_command 1 2024-03-27T21:42:05.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f0bd5ffb640 1 -- 172.21.15.60:0/3694384783 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0bc4075510 con 0x7f0bc4072a20 2024-03-27T21:42:05.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f0bd5ffb640 20 client.5221.objecter dump_active .. 0 homeless 2024-03-27T21:42:05.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f0bd5ffb640 1 -- 172.21.15.60:0/3694384783 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0bd008c490 con 0x7f0bd8140390 2024-03-27T21:42:05.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f0bd5ffb640 10 monclient: handle_get_version_reply finishing 1 version 92 2024-03-27T21:42:05.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f0bdcee4640 1 --2- 172.21.15.60:0/3694384783 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f0bc4072a20 0x7f0bc4074e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f0bdcee4640 10 monclient: get_auth_request con 0x7f0bc4072a20 auth_method 0 2024-03-27T21:42:05.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f8894ff9640 10 client.5224.objecter ms_handle_connect 0x7f888406c610 2024-03-27T21:42:05.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f88977fe640 1 --2- 172.21.15.60:0/530395410 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f888406c610 0x7f888406ead0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f88900014a0 tx=0x7f8890006040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f889cd6f640 10 monclient: get_auth_request con 0x7f8884072a50 auth_method 0 2024-03-27T21:42:05.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.114+0000 7f0bd5ffb640 10 client.5221.objecter ms_handle_connect 0x7f0bc4072a20 2024-03-27T21:42:05.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.114+0000 7f0bdcee4640 1 --2- 172.21.15.60:0/3694384783 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f0bc4072a20 0x7f0bc4074e40 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f211ed9d640 1 --2- 172.21.15.60:0/2051018026 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f2100072aa0 0x7f2100074ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.113+0000 7f211ed9d640 10 monclient: get_auth_request con 0x7f2100072aa0 auth_method 0 2024-03-27T21:42:05.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.114+0000 7f211ed9d640 1 --2- 172.21.15.60:0/2051018026 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f2100072aa0 0x7f2100074ec0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.114+0000 7f211dd9b640 1 --2- 172.21.15.60:0/2051018026 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f210006c660 0x7f210006eb20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2110035000 tx=0x7f211002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.114+0000 7f210effd640 10 client.5218.objecter ms_handle_connect 0x7f2100072aa0 2024-03-27T21:42:05.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.114+0000 7f210effd640 10 client.5218.objecter ms_handle_connect 0x7f210006c660 2024-03-27T21:42:05.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.114+0000 7f0bdd6e5640 1 --2- 172.21.15.60:0/3694384783 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f0bc406c5e0 0x7f0bc406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.114+0000 7f0bdd6e5640 10 monclient: get_auth_request con 0x7f0bc406c5e0 auth_method 0 2024-03-27T21:42:05.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.114+0000 7f32927fc640 1 -- 172.21.15.60:0/1304584283 <== osd.2 v2:172.21.15.60:6824/56434 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f3280075590 con 0x7f3280072aa0 2024-03-27T21:42:05.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.114+0000 7f32927fc640 10 client.5212.objecter ms_dispatch 0x7f329c000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:42:05.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.114+0000 7f32927fc640 10 client.5212.objecter _finish_command 1 = system:0 2024-03-27T21:42:05.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.114+0000 7f32927fc640 20 client.5212.objecter put_session s=0x7f32800723d0 osd=2 4 2024-03-27T21:42:05.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.114+0000 7f32927fc640 15 client.5212.objecter _session_command_op_remove 2 1 2024-03-27T21:42:05.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.114+0000 7f0bd5ffb640 1 -- 172.21.15.60:0/3694384783 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f0bc4075510 con 0x7f0bc4072a20 2024-03-27T21:42:05.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.114+0000 7f0bd5ffb640 10 client.5221.objecter ms_dispatch 0x7f0bd8000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:42:05.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.114+0000 7f0bd5ffb640 10 client.5221.objecter _finish_command 1 = system:0 2024-03-27T21:42:05.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.114+0000 7f0bd5ffb640 20 client.5221.objecter put_session s=0x7f0bc4072350 osd=7 4 2024-03-27T21:42:05.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.114+0000 7f0bd5ffb640 15 client.5221.objecter _session_command_op_remove 7 1 2024-03-27T21:42:05.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.115+0000 7f0bd5ffb640 10 client.5221.objecter ms_handle_connect 0x7f0bc406c5e0 2024-03-27T21:42:05.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.115+0000 7f0bdd6e5640 1 --2- 172.21.15.60:0/3694384783 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f0bc406c5e0 0x7f0bc406eaa0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0bc800a4e0 tx=0x7f0bc800c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.115+0000 7f210effd640 1 -- 172.21.15.60:0/2051018026 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f2100075590 con 0x7f2100072aa0 2024-03-27T21:42:05.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.115+0000 7f210effd640 10 client.5218.objecter ms_dispatch 0x7f2120000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:42:05.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.115+0000 7f210effd640 10 client.5218.objecter _finish_command 1 = system:0 2024-03-27T21:42:05.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.115+0000 7f210effd640 20 client.5218.objecter put_session s=0x7f21000723d0 osd=6 4 2024-03-27T21:42:05.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.115+0000 7f210effd640 15 client.5218.objecter _session_command_op_remove 6 1 2024-03-27T21:42:05.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.116+0000 7f889cd6f640 1 --2- 172.21.15.60:0/530395410 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f8884072a50 0x7f8884074e70 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.116+0000 7f8894ff9640 10 client.5224.objecter ms_handle_connect 0x7f8884072a50 2024-03-27T21:42:05.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.117+0000 7f8894ff9640 1 -- 172.21.15.60:0/530395410 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f8884075540 con 0x7f8884072a50 2024-03-27T21:42:05.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.117+0000 7f8894ff9640 10 client.5224.objecter ms_dispatch 0x7f8898000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:42:05.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.117+0000 7f8894ff9640 10 client.5224.objecter _finish_command 1 = system:0 2024-03-27T21:42:05.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.117+0000 7f8894ff9640 20 client.5224.objecter put_session s=0x7f8884072380 osd=0 4 2024-03-27T21:42:05.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.117+0000 7f8894ff9640 15 client.5224.objecter _session_command_op_remove 0 1 2024-03-27T21:42:05.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7f558bfff640 10 client.5209.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-27T21:42:05.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7f558bfff640 15 client.5209.objecter _session_command_op_assign -1 2 2024-03-27T21:42:05.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7f558bfff640 20 client.5209.objecter _get_session s=0x7f558c072400 osd=1 3 2024-03-27T21:42:05.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7f558bfff640 20 client.5209.objecter put_session s=0x7f558c072400 osd=1 3 2024-03-27T21:42:05.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7f558bfff640 20 client.5209.objecter _get_session s=0x7f558c072400 osd=1 3 2024-03-27T21:42:05.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7f558bfff640 15 client.5209.objecter _session_command_op_remove -1 2 2024-03-27T21:42:05.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7f558bfff640 20 client.5209.objecter get_session s=0x7f558c072400 osd=1 3 2024-03-27T21:42:05.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7f558bfff640 15 client.5209.objecter _session_command_op_assign 1 2 2024-03-27T21:42:05.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7f558bfff640 20 client.5209.objecter put_session s=0x7f558c072400 osd=1 4 2024-03-27T21:42:05.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7f558bfff640 10 client.5209.objecter _send_command 2 2024-03-27T21:42:05.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7f558bfff640 1 -- 172.21.15.60:0/1084400107 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f55a00c75f0 con 0x7f558c072ad0 2024-03-27T21:42:05.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7f559e7fc640 1 -- 172.21.15.60:0/1084400107 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f55a00c75f0 con 0x7f558c072ad0 2024-03-27T21:42:05.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7f559e7fc640 10 client.5209.objecter ms_dispatch 0x7f55a8000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:42:05.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7f559e7fc640 10 client.5209.objecter _finish_command 2 = system:0 2024-03-27T21:42:05.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7f559e7fc640 20 client.5209.objecter put_session s=0x7f558c072400 osd=1 4 2024-03-27T21:42:05.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7f559e7fc640 15 client.5209.objecter _session_command_op_remove 1 2 2024-03-27T21:42:05.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7fda84ccf640 10 client.5301.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-27T21:42:05.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7fda84ccf640 15 client.5301.objecter _session_command_op_assign -1 2 2024-03-27T21:42:05.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7fda84ccf640 20 client.5301.objecter _get_session s=0x7fda600723d0 osd=3 3 2024-03-27T21:42:05.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7fda84ccf640 20 client.5301.objecter put_session s=0x7fda600723d0 osd=3 3 2024-03-27T21:42:05.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7fda84ccf640 20 client.5301.objecter _get_session s=0x7fda600723d0 osd=3 3 2024-03-27T21:42:05.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7fda84ccf640 15 client.5301.objecter _session_command_op_remove -1 2 2024-03-27T21:42:05.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7fda84ccf640 20 client.5301.objecter get_session s=0x7fda600723d0 osd=3 3 2024-03-27T21:42:05.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7fda84ccf640 15 client.5301.objecter _session_command_op_assign 3 2 2024-03-27T21:42:05.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7fda84ccf640 20 client.5301.objecter put_session s=0x7fda600723d0 osd=3 4 2024-03-27T21:42:05.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7fda84ccf640 10 client.5301.objecter _send_command 2 2024-03-27T21:42:05.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.142+0000 7fda84ccf640 1 -- 172.21.15.60:0/2943937655 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fda640011a0 con 0x7fda60072aa0 2024-03-27T21:42:05.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7f55aee54640 10 librados: watch_flush enter 2024-03-27T21:42:05.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7f55aee54640 10 librados: watch_flush exit 2024-03-27T21:42:05.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7f55aee54640 10 client.5209.objecter close_session for osd.1 2024-03-27T21:42:05.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7f55aee54640 1 -- 172.21.15.60:0/1084400107 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f558c072ad0 msgr2=0x7f558c074ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7fda6effd640 1 -- 172.21.15.60:0/2943937655 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fda640011a0 con 0x7fda60072aa0 2024-03-27T21:42:05.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7fda6effd640 10 client.5301.objecter ms_dispatch 0x7fda80000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:42:05.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7fda6effd640 10 client.5301.objecter _finish_command 2 = system:0 2024-03-27T21:42:05.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7fda6effd640 20 client.5301.objecter put_session s=0x7fda600723d0 osd=3 4 2024-03-27T21:42:05.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7fda6effd640 15 client.5301.objecter _session_command_op_remove 3 2 2024-03-27T21:42:05.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7f55aee54640 1 --2- 172.21.15.60:0/1084400107 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f558c072ad0 0x7f558c074ef0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7f55aee54640 20 client.5209.objecter put_session s=0x7f558c072400 osd=1 1 2024-03-27T21:42:05.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7f55aee54640 20 client.5209.objecter shutdown clearing up homeless session... 2024-03-27T21:42:05.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7f55aee54640 10 client.5209.objecter successfully canceled tick 2024-03-27T21:42:05.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7f55aee54640 1 -- 172.21.15.60:0/1084400107 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f558c06c690 msgr2=0x7f558c06eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7f55aee54640 1 --2- 172.21.15.60:0/1084400107 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f558c06c690 0x7f558c06eb50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f55a40099a0 tx=0x7f55a4006f90 comp rx=0 tx=0).stop 2024-03-27T21:42:05.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7f55aee54640 10 monclient: shutdown 2024-03-27T21:42:05.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7f55aee54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7f55aee54640 1 -- 172.21.15.60:0/1084400107 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55a0039000 msgr2=0x7f55a00c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7f55aee54640 1 --2- 172.21.15.60:0/1084400107 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55a0039000 0x7f55a00c7df0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f55a8067ae0 tx=0x7f55a8067040 comp rx=0 tx=0).stop 2024-03-27T21:42:05.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f55ae653640 1 -- 172.21.15.60:0/1084400107 reap_dead start 2024-03-27T21:42:05.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7fda84ccf640 10 librados: watch_flush enter 2024-03-27T21:42:05.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f7b2b59e640 10 client.5018.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-27T21:42:05.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7ff055671640 10 client.5215.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-27T21:42:05.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7ff055671640 15 client.5215.objecter _session_command_op_assign -1 2 2024-03-27T21:42:05.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7ff055671640 20 client.5215.objecter _get_session s=0x7ff0300723d0 osd=5 3 2024-03-27T21:42:05.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7ff055671640 20 client.5215.objecter put_session s=0x7ff0300723d0 osd=5 3 2024-03-27T21:42:05.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7ff055671640 20 client.5215.objecter _get_session s=0x7ff0300723d0 osd=5 3 2024-03-27T21:42:05.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7ff055671640 15 client.5215.objecter _session_command_op_remove -1 2 2024-03-27T21:42:05.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7ff055671640 20 client.5215.objecter get_session s=0x7ff0300723d0 osd=5 3 2024-03-27T21:42:05.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7ff055671640 15 client.5215.objecter _session_command_op_assign 5 2 2024-03-27T21:42:05.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7ff055671640 20 client.5215.objecter put_session s=0x7ff0300723d0 osd=5 4 2024-03-27T21:42:05.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7ff055671640 10 client.5215.objecter _send_command 2 2024-03-27T21:42:05.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.143+0000 7ff055671640 1 -- 172.21.15.60:0/1944173661 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff05006f890 con 0x7ff030072aa0 2024-03-27T21:42:05.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f7b2b59e640 15 client.5018.objecter _session_command_op_assign -1 2 2024-03-27T21:42:05.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f7b2b59e640 20 client.5018.objecter _get_session s=0x7f7b0c072350 osd=4 3 2024-03-27T21:42:05.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f7b2b59e640 20 client.5018.objecter put_session s=0x7f7b0c072350 osd=4 3 2024-03-27T21:42:05.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7fda84ccf640 10 librados: watch_flush exit 2024-03-27T21:42:05.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f7b2b59e640 20 client.5018.objecter _get_session s=0x7f7b0c072350 osd=4 3 2024-03-27T21:42:05.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7fda84ccf640 10 client.5301.objecter close_session for osd.3 2024-03-27T21:42:05.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f7b2b59e640 15 client.5018.objecter _session_command_op_remove -1 2 2024-03-27T21:42:05.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f7b2b59e640 20 client.5018.objecter get_session s=0x7f7b0c072350 osd=4 3 2024-03-27T21:42:05.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7fda84ccf640 1 -- 172.21.15.60:0/2943937655 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fda60072aa0 msgr2=0x7fda60074ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7fda84ccf640 1 --2- 172.21.15.60:0/2943937655 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7fda60072aa0 0x7fda60074ec0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f7b2b59e640 15 client.5018.objecter _session_command_op_assign 4 2 2024-03-27T21:42:05.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f7b2b59e640 20 client.5018.objecter put_session s=0x7f7b0c072350 osd=4 4 2024-03-27T21:42:05.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f7b2b59e640 10 client.5018.objecter _send_command 2 2024-03-27T21:42:05.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f7b2b59e640 1 -- 172.21.15.60:0/3770112909 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7b100011a0 con 0x7f7b0c072a20 2024-03-27T21:42:05.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7fda84ccf640 20 client.5301.objecter put_session s=0x7fda600723d0 osd=3 1 2024-03-27T21:42:05.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7fda84ccf640 20 client.5301.objecter shutdown clearing up homeless session... 2024-03-27T21:42:05.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7fda84ccf640 10 client.5301.objecter successfully canceled tick 2024-03-27T21:42:05.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f55aee54640 1 -- 172.21.15.60:0/1084400107 shutdown_connections 2024-03-27T21:42:05.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f55aee54640 1 -- 172.21.15.60:0/1084400107 >> 172.21.15.60:0/1084400107 conn(0x7f55a0006430 msgr2=0x7f55a0037c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f55aee54640 1 -- 172.21.15.60:0/1084400107 shutdown_connections 2024-03-27T21:42:05.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f55aee54640 1 -- 172.21.15.60:0/1084400107 wait complete. 2024-03-27T21:42:05.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7fda84ccf640 1 -- 172.21.15.60:0/2943937655 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fda6006c660 msgr2=0x7fda6006eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7fda84ccf640 1 --2- 172.21.15.60:0/2943937655 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fda6006c660 0x7fda6006eb20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fda80141390 tx=0x7fda78007dd0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7fda84ccf640 10 monclient: shutdown 2024-03-27T21:42:05.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7ff0477fe640 1 -- 172.21.15.60:0/1944173661 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff030075590 con 0x7ff030072aa0 2024-03-27T21:42:05.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7ff0477fe640 10 client.5215.objecter ms_dispatch 0x7ff050000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:42:05.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7ff0477fe640 10 client.5215.objecter _finish_command 2 = system:0 2024-03-27T21:42:05.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7ff0477fe640 20 client.5215.objecter put_session s=0x7ff0300723d0 osd=5 4 2024-03-27T21:42:05.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7ff0477fe640 15 client.5215.objecter _session_command_op_remove 5 2 2024-03-27T21:42:05.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7fda84ccf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7fda84ccf640 1 -- 172.21.15.60:0/2943937655 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda800714e0 msgr2=0x7fda80140510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7fda84ccf640 1 --2- 172.21.15.60:0/2943937655 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda800714e0 0x7fda80140510 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fda74009870 tx=0x7fda740046c0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7fda7ed9d640 1 -- 172.21.15.60:0/2943937655 reap_dead start 2024-03-27T21:42:05.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f55aee54640 1 librados: shutdown 2024-03-27T21:42:05.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7fda84ccf640 1 -- 172.21.15.60:0/2943937655 shutdown_connections 2024-03-27T21:42:05.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7fda84ccf640 1 -- 172.21.15.60:0/2943937655 >> 172.21.15.60:0/2943937655 conn(0x7fda800b3ae0 msgr2=0x7fda8014b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f7b1affd640 1 -- 172.21.15.60:0/3770112909 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7b100011a0 con 0x7f7b0c072a20 2024-03-27T21:42:05.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f7b1affd640 10 client.5018.objecter ms_dispatch 0x7f7b2c000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:42:05.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f7b1affd640 10 client.5018.objecter _finish_command 2 = system:0 2024-03-27T21:42:05.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f7b1affd640 20 client.5018.objecter put_session s=0x7f7b0c072350 osd=4 4 2024-03-27T21:42:05.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.144+0000 7f7b1affd640 15 client.5018.objecter _session_command_op_remove 4 2 2024-03-27T21:42:05.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7ff055671640 10 librados: watch_flush enter 2024-03-27T21:42:05.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7fda84ccf640 1 -- 172.21.15.60:0/2943937655 shutdown_connections 2024-03-27T21:42:05.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7fda84ccf640 1 -- 172.21.15.60:0/2943937655 wait complete. 2024-03-27T21:42:05.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7ff055671640 10 librados: watch_flush exit 2024-03-27T21:42:05.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7ff055671640 10 client.5215.objecter close_session for osd.5 2024-03-27T21:42:05.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7ff055671640 1 -- 172.21.15.60:0/1944173661 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7ff030072aa0 msgr2=0x7ff030074ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7ff055671640 1 --2- 172.21.15.60:0/1944173661 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7ff030072aa0 0x7ff030074ec0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7ff055671640 20 client.5215.objecter put_session s=0x7ff0300723d0 osd=5 1 2024-03-27T21:42:05.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7ff055671640 20 client.5215.objecter shutdown clearing up homeless session... 2024-03-27T21:42:05.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7ff055671640 10 client.5215.objecter successfully canceled tick 2024-03-27T21:42:05.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7fda84ccf640 1 librados: shutdown 2024-03-27T21:42:05.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7f7b2b59e640 10 librados: watch_flush enter 2024-03-27T21:42:05.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7f7b2b59e640 10 librados: watch_flush exit 2024-03-27T21:42:05.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7ff055671640 1 -- 172.21.15.60:0/1944173661 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff03006c660 msgr2=0x7ff03006eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7ff055671640 1 --2- 172.21.15.60:0/1944173661 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7ff03006c660 0x7ff03006eb20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff04000d790 tx=0x7ff04000d040 comp rx=0 tx=0).stop 2024-03-27T21:42:05.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7ff055671640 10 monclient: shutdown 2024-03-27T21:42:05.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7ff055671640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7ff055671640 1 -- 172.21.15.60:0/1944173661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff050152220 msgr2=0x7ff0501f7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7ff055671640 1 --2- 172.21.15.60:0/1944173661 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff050152220 0x7ff0501f7780 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff03800c3c0 tx=0x7ff03800c880 comp rx=0 tx=0).stop 2024-03-27T21:42:05.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7ff04f7fe640 1 -- 172.21.15.60:0/1944173661 reap_dead start 2024-03-27T21:42:05.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7f7b2b59e640 10 client.5018.objecter close_session for osd.4 2024-03-27T21:42:05.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7ff055671640 1 -- 172.21.15.60:0/1944173661 shutdown_connections 2024-03-27T21:42:05.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7ff055671640 1 -- 172.21.15.60:0/1944173661 >> 172.21.15.60:0/1944173661 conn(0x7ff0500b3620 msgr2=0x7ff050070290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7f7b2b59e640 1 -- 172.21.15.60:0/3770112909 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f7b0c072a20 msgr2=0x7f7b0c074e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7f7b2b59e640 1 --2- 172.21.15.60:0/3770112909 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7f7b0c072a20 0x7f7b0c074e40 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7f7b2b59e640 20 client.5018.objecter put_session s=0x7f7b0c072350 osd=4 1 2024-03-27T21:42:05.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7f7b2b59e640 20 client.5018.objecter shutdown clearing up homeless session... 2024-03-27T21:42:05.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7f7b2b59e640 10 client.5018.objecter successfully canceled tick 2024-03-27T21:42:05.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7f7b2b59e640 1 -- 172.21.15.60:0/3770112909 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f7b0c06c5e0 msgr2=0x7f7b0c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7f7b2b59e640 1 --2- 172.21.15.60:0/3770112909 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f7b0c06c5e0 0x7f7b0c06eaa0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7b2000a980 tx=0x7f7b20008040 comp rx=0 tx=0).stop 2024-03-27T21:42:05.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7f7b2b59e640 10 monclient: shutdown 2024-03-27T21:42:05.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7f7b2b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7f7b2b59e640 1 -- 172.21.15.60:0/3770112909 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7b2c0aeda0 msgr2=0x7f7b2c1d9b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7f7b2b59e640 1 --2- 172.21.15.60:0/3770112909 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f7b2c0aeda0 0x7f7b2c1d9b20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c02b8f0 tx=0x7f7b1c0046c0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7f7b2ad9d640 1 -- 172.21.15.60:0/3770112909 reap_dead start 2024-03-27T21:42:05.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7ff055671640 1 -- 172.21.15.60:0/1944173661 shutdown_connections 2024-03-27T21:42:05.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7ff055671640 1 -- 172.21.15.60:0/1944173661 wait complete. 2024-03-27T21:42:05.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.145+0000 7ff055671640 1 librados: shutdown 2024-03-27T21:42:05.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f7b2b59e640 1 -- 172.21.15.60:0/3770112909 shutdown_connections 2024-03-27T21:42:05.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f7b2b59e640 1 -- 172.21.15.60:0/3770112909 >> 172.21.15.60:0/3770112909 conn(0x7f7b2c0afc20 msgr2=0x7f7b2c13d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f7b2b59e640 1 -- 172.21.15.60:0/3770112909 shutdown_connections 2024-03-27T21:42:05.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f2124edb640 10 client.5218.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-27T21:42:05.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f2124edb640 15 client.5218.objecter _session_command_op_assign -1 2 2024-03-27T21:42:05.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f2124edb640 20 client.5218.objecter _get_session s=0x7f21000723d0 osd=6 3 2024-03-27T21:42:05.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f2124edb640 20 client.5218.objecter put_session s=0x7f21000723d0 osd=6 3 2024-03-27T21:42:05.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f2124edb640 20 client.5218.objecter _get_session s=0x7f21000723d0 osd=6 3 2024-03-27T21:42:05.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f2124edb640 15 client.5218.objecter _session_command_op_remove -1 2 2024-03-27T21:42:05.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f2124edb640 20 client.5218.objecter get_session s=0x7f21000723d0 osd=6 3 2024-03-27T21:42:05.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f2124edb640 15 client.5218.objecter _session_command_op_assign 6 2 2024-03-27T21:42:05.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f2124edb640 20 client.5218.objecter put_session s=0x7f21000723d0 osd=6 4 2024-03-27T21:42:05.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f2124edb640 10 client.5218.objecter _send_command 2 2024-03-27T21:42:05.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f2124edb640 1 -- 172.21.15.60:0/2051018026 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2120135560 con 0x7f2100072aa0 2024-03-27T21:42:05.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f0bdf949640 10 client.5221.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-27T21:42:05.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f0bdf949640 15 client.5221.objecter _session_command_op_assign -1 2 2024-03-27T21:42:05.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f0bdf949640 20 client.5221.objecter _get_session s=0x7f0bc4072350 osd=7 3 2024-03-27T21:42:05.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f0bdf949640 20 client.5221.objecter put_session s=0x7f0bc4072350 osd=7 3 2024-03-27T21:42:05.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f0bdf949640 20 client.5221.objecter _get_session s=0x7f0bc4072350 osd=7 3 2024-03-27T21:42:05.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f0bdf949640 15 client.5221.objecter _session_command_op_remove -1 2 2024-03-27T21:42:05.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f0bdf949640 20 client.5221.objecter get_session s=0x7f0bc4072350 osd=7 3 2024-03-27T21:42:05.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f0bdf949640 15 client.5221.objecter _session_command_op_assign 7 2 2024-03-27T21:42:05.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f0bdf949640 20 client.5221.objecter put_session s=0x7f0bc4072350 osd=7 4 2024-03-27T21:42:05.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f0bdf949640 10 client.5221.objecter _send_command 2 2024-03-27T21:42:05.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f0bdf949640 1 -- 172.21.15.60:0/3694384783 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0bd806fca0 con 0x7f0bc4072a20 2024-03-27T21:42:05.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f7b2b59e640 1 -- 172.21.15.60:0/3770112909 wait complete. 2024-03-27T21:42:05.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.146+0000 7f7b2b59e640 1 librados: shutdown 2024-03-27T21:42:05.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f210effd640 1 -- 172.21.15.60:0/2051018026 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f21180097b0 con 0x7f2100072aa0 2024-03-27T21:42:05.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f210effd640 10 client.5218.objecter ms_dispatch 0x7f2120000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:42:05.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f210effd640 10 client.5218.objecter _finish_command 2 = system:0 2024-03-27T21:42:05.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f32a3f39640 10 client.5212.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-27T21:42:05.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f32a3f39640 15 client.5212.objecter _session_command_op_assign -1 2 2024-03-27T21:42:05.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f32a3f39640 20 client.5212.objecter _get_session s=0x7f32800723d0 osd=2 3 2024-03-27T21:42:05.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f32a3f39640 20 client.5212.objecter put_session s=0x7f32800723d0 osd=2 3 2024-03-27T21:42:05.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f32a3f39640 20 client.5212.objecter _get_session s=0x7f32800723d0 osd=2 3 2024-03-27T21:42:05.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f32a3f39640 15 client.5212.objecter _session_command_op_remove -1 2 2024-03-27T21:42:05.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f32a3f39640 20 client.5212.objecter get_session s=0x7f32800723d0 osd=2 3 2024-03-27T21:42:05.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f32a3f39640 15 client.5212.objecter _session_command_op_assign 2 2 2024-03-27T21:42:05.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f32a3f39640 20 client.5212.objecter put_session s=0x7f32800723d0 osd=2 4 2024-03-27T21:42:05.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f32a3f39640 10 client.5212.objecter _send_command 2 2024-03-27T21:42:05.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f32a3f39640 1 -- 172.21.15.60:0/1304584283 --> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f32840011f0 con 0x7f3280072aa0 2024-03-27T21:42:05.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f210effd640 20 client.5218.objecter put_session s=0x7f21000723d0 osd=6 4 2024-03-27T21:42:05.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f210effd640 15 client.5218.objecter _session_command_op_remove 6 2 2024-03-27T21:42:05.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f0bd5ffb640 1 -- 172.21.15.60:0/3694384783 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0bc4075510 con 0x7f0bc4072a20 2024-03-27T21:42:05.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f0bd5ffb640 10 client.5221.objecter ms_dispatch 0x7f0bd8000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:42:05.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f0bd5ffb640 10 client.5221.objecter _finish_command 2 = system:0 2024-03-27T21:42:05.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f0bd5ffb640 20 client.5221.objecter put_session s=0x7f0bc4072350 osd=7 4 2024-03-27T21:42:05.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f0bd5ffb640 15 client.5221.objecter _session_command_op_remove 7 2 2024-03-27T21:42:05.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f32927fc640 1 -- 172.21.15.60:0/1304584283 <== osd.2 v2:172.21.15.60:6824/56434 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f32840011f0 con 0x7f3280072aa0 2024-03-27T21:42:05.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f32927fc640 10 client.5212.objecter ms_dispatch 0x7f329c000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:42:05.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f2124edb640 10 librados: watch_flush enter 2024-03-27T21:42:05.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f32927fc640 10 client.5212.objecter _finish_command 2 = system:0 2024-03-27T21:42:05.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f0bdf949640 10 librados: watch_flush enter 2024-03-27T21:42:05.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f32927fc640 20 client.5212.objecter put_session s=0x7f32800723d0 osd=2 4 2024-03-27T21:42:05.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f32927fc640 15 client.5212.objecter _session_command_op_remove 2 2 2024-03-27T21:42:05.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f2124edb640 10 librados: watch_flush exit 2024-03-27T21:42:05.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f2124edb640 10 client.5218.objecter close_session for osd.6 2024-03-27T21:42:05.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f2124edb640 1 -- 172.21.15.60:0/2051018026 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f2100072aa0 msgr2=0x7f2100074ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.147+0000 7f2124edb640 1 --2- 172.21.15.60:0/2051018026 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f2100072aa0 0x7f2100074ec0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f2124edb640 20 client.5218.objecter put_session s=0x7f21000723d0 osd=6 1 2024-03-27T21:42:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f2124edb640 20 client.5218.objecter shutdown clearing up homeless session... 2024-03-27T21:42:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f2124edb640 10 client.5218.objecter successfully canceled tick 2024-03-27T21:42:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f32a3f39640 10 librados: watch_flush enter 2024-03-27T21:42:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f32a3f39640 10 librados: watch_flush exit 2024-03-27T21:42:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f32a3f39640 10 client.5212.objecter close_session for osd.2 2024-03-27T21:42:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f32a3f39640 1 -- 172.21.15.60:0/1304584283 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f3280072aa0 msgr2=0x7f3280074ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f32a3f39640 1 --2- 172.21.15.60:0/1304584283 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f3280072aa0 0x7f3280074ec0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f2124edb640 1 -- 172.21.15.60:0/2051018026 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f210006c660 msgr2=0x7f210006eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f2124edb640 1 --2- 172.21.15.60:0/2051018026 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f210006c660 0x7f210006eb20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2110035000 tx=0x7f211002e000 comp rx=0 tx=0).stop 2024-03-27T21:42:05.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f2124edb640 10 monclient: shutdown 2024-03-27T21:42:05.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f2124edb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f2124edb640 1 -- 172.21.15.60:0/2051018026 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2120060e10 msgr2=0x7f2120154b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f2124edb640 1 --2- 172.21.15.60:0/2051018026 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2120060e10 0x7f2120154b90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f211400c850 tx=0x7f211400cc50 comp rx=0 tx=0).stop 2024-03-27T21:42:05.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f211ed9d640 1 -- 172.21.15.60:0/2051018026 reap_dead start 2024-03-27T21:42:05.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f0bdf949640 10 librados: watch_flush exit 2024-03-27T21:42:05.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f0bdf949640 10 client.5221.objecter close_session for osd.7 2024-03-27T21:42:05.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f0bdf949640 1 -- 172.21.15.60:0/3694384783 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f0bc4072a20 msgr2=0x7f0bc4074e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f0bdf949640 1 --2- 172.21.15.60:0/3694384783 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f0bc4072a20 0x7f0bc4074e40 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f0bdf949640 20 client.5221.objecter put_session s=0x7f0bc4072350 osd=7 1 2024-03-27T21:42:05.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f0bdf949640 20 client.5221.objecter shutdown clearing up homeless session... 2024-03-27T21:42:05.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f0bdf949640 10 client.5221.objecter successfully canceled tick 2024-03-27T21:42:05.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f2124edb640 1 -- 172.21.15.60:0/2051018026 shutdown_connections 2024-03-27T21:42:05.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f32a3f39640 20 client.5212.objecter put_session s=0x7f32800723d0 osd=2 1 2024-03-27T21:42:05.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f2124edb640 1 -- 172.21.15.60:0/2051018026 >> 172.21.15.60:0/2051018026 conn(0x7f21200b3620 msgr2=0x7f212013dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f32a3f39640 20 client.5212.objecter shutdown clearing up homeless session... 2024-03-27T21:42:05.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f32a3f39640 10 client.5212.objecter successfully canceled tick 2024-03-27T21:42:05.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f2124edb640 1 -- 172.21.15.60:0/2051018026 shutdown_connections 2024-03-27T21:42:05.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f2124edb640 1 -- 172.21.15.60:0/2051018026 wait complete. 2024-03-27T21:42:05.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f2124edb640 1 librados: shutdown 2024-03-27T21:42:05.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f0bdf949640 1 -- 172.21.15.60:0/3694384783 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f0bc406c5e0 msgr2=0x7f0bc406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f0bdf949640 1 --2- 172.21.15.60:0/3694384783 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f0bc406c5e0 0x7f0bc406eaa0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0bc800a4e0 tx=0x7f0bc800c040 comp rx=0 tx=0).stop 2024-03-27T21:42:05.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f0bdf949640 10 monclient: shutdown 2024-03-27T21:42:05.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f0bdf949640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f0bdf949640 1 -- 172.21.15.60:0/3694384783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bd8140390 msgr2=0x7f0bd81de1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.148+0000 7f0bdf949640 1 --2- 172.21.15.60:0/3694384783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bd8140390 0x7f0bd81de1c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0bd000efc0 tx=0x7f0bd000c490 comp rx=0 tx=0).stop 2024-03-27T21:42:05.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f0bddee6640 1 -- 172.21.15.60:0/3694384783 reap_dead start 2024-03-27T21:42:05.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f0bdf949640 1 -- 172.21.15.60:0/3694384783 shutdown_connections 2024-03-27T21:42:05.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f0bdf949640 1 -- 172.21.15.60:0/3694384783 >> 172.21.15.60:0/3694384783 conn(0x7f0bd80b3620 msgr2=0x7f0bd813da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f0bdf949640 1 -- 172.21.15.60:0/3694384783 shutdown_connections 2024-03-27T21:42:05.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f0bdf949640 1 -- 172.21.15.60:0/3694384783 wait complete. 2024-03-27T21:42:05.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f0bdf949640 1 librados: shutdown 2024-03-27T21:42:05.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f889d570640 10 client.5224.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-27T21:42:05.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f889d570640 15 client.5224.objecter _session_command_op_assign -1 2 2024-03-27T21:42:05.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f889d570640 20 client.5224.objecter _get_session s=0x7f8884072380 osd=0 3 2024-03-27T21:42:05.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f889d570640 20 client.5224.objecter put_session s=0x7f8884072380 osd=0 3 2024-03-27T21:42:05.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f889d570640 20 client.5224.objecter _get_session s=0x7f8884072380 osd=0 3 2024-03-27T21:42:05.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f889d570640 15 client.5224.objecter _session_command_op_remove -1 2 2024-03-27T21:42:05.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f889d570640 20 client.5224.objecter get_session s=0x7f8884072380 osd=0 3 2024-03-27T21:42:05.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f889d570640 15 client.5224.objecter _session_command_op_assign 0 2 2024-03-27T21:42:05.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f889d570640 20 client.5224.objecter put_session s=0x7f8884072380 osd=0 4 2024-03-27T21:42:05.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f889d570640 10 client.5224.objecter _send_command 2 2024-03-27T21:42:05.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f889d570640 1 -- 172.21.15.60:0/530395410 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f88980ab990 con 0x7f8884072a50 2024-03-27T21:42:05.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f32a3f39640 1 -- 172.21.15.60:0/1304584283 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f328006c660 msgr2=0x7f328006eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f32a3f39640 1 --2- 172.21.15.60:0/1304584283 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f328006c660 0x7f328006eb20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f328c006fd0 tx=0x7f328c006b40 comp rx=0 tx=0).stop 2024-03-27T21:42:05.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f32a3f39640 10 monclient: shutdown 2024-03-27T21:42:05.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f32a3f39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f32a3f39640 1 -- 172.21.15.60:0/1304584283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f329c0714e0 msgr2=0x7f329c138eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.149+0000 7f32a3f39640 1 --2- 172.21.15.60:0/1304584283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f329c0714e0 0x7f329c138eb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3294036040 tx=0x7f3294004a40 comp rx=0 tx=0).stop 2024-03-27T21:42:05.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.150+0000 7f32a24d6640 1 -- 172.21.15.60:0/1304584283 reap_dead start 2024-03-27T21:42:05.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.150+0000 7f8894ff9640 1 -- 172.21.15.60:0/530395410 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f888c043040 con 0x7f8884072a50 2024-03-27T21:42:05.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.150+0000 7f8894ff9640 10 client.5224.objecter ms_dispatch 0x7f8898000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:42:05.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.150+0000 7f8894ff9640 10 client.5224.objecter _finish_command 2 = system:0 2024-03-27T21:42:05.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.150+0000 7f8894ff9640 20 client.5224.objecter put_session s=0x7f8884072380 osd=0 4 2024-03-27T21:42:05.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.150+0000 7f8894ff9640 15 client.5224.objecter _session_command_op_remove 0 2 2024-03-27T21:42:05.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.150+0000 7f889d570640 10 librados: watch_flush enter 2024-03-27T21:42:05.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.150+0000 7f889d570640 10 librados: watch_flush exit 2024-03-27T21:42:05.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.150+0000 7f32a3f39640 1 -- 172.21.15.60:0/1304584283 shutdown_connections 2024-03-27T21:42:05.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.150+0000 7f32a3f39640 1 -- 172.21.15.60:0/1304584283 >> 172.21.15.60:0/1304584283 conn(0x7f329c0b3ae0 msgr2=0x7f329c0b6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.151+0000 7f889d570640 10 client.5224.objecter close_session for osd.0 2024-03-27T21:42:05.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.151+0000 7f889d570640 1 -- 172.21.15.60:0/530395410 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f8884072a50 msgr2=0x7f8884074e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.151+0000 7f889d570640 1 --2- 172.21.15.60:0/530395410 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7f8884072a50 0x7f8884074e70 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.151+0000 7f889d570640 20 client.5224.objecter put_session s=0x7f8884072380 osd=0 1 2024-03-27T21:42:05.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.151+0000 7f889d570640 20 client.5224.objecter shutdown clearing up homeless session... 2024-03-27T21:42:05.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.151+0000 7f889d570640 10 client.5224.objecter successfully canceled tick 2024-03-27T21:42:05.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.151+0000 7f889d570640 1 -- 172.21.15.60:0/530395410 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f888406c610 msgr2=0x7f888406ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.151+0000 7f889d570640 1 --2- 172.21.15.60:0/530395410 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f888406c610 0x7f888406ead0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f88900014a0 tx=0x7f8890006040 comp rx=0 tx=0).stop 2024-03-27T21:42:05.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.151+0000 7f889d570640 10 monclient: shutdown 2024-03-27T21:42:05.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.151+0000 7f889d570640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.151+0000 7f889d570640 1 -- 172.21.15.60:0/530395410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8880005180 msgr2=0x7f88980a9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.151+0000 7f889d570640 1 --2- 172.21.15.60:0/530395410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8880005180 0x7f88980a9b40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f888800f650 tx=0x7f888800fb10 comp rx=0 tx=0).stop 2024-03-27T21:42:05.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.151+0000 7f889cd6f640 1 -- 172.21.15.60:0/530395410 reap_dead start 2024-03-27T21:42:05.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.151+0000 7f889d570640 1 -- 172.21.15.60:0/530395410 shutdown_connections 2024-03-27T21:42:05.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.151+0000 7f889d570640 1 -- 172.21.15.60:0/530395410 >> 172.21.15.60:0/530395410 conn(0x7f8898060700 msgr2=0x7f88980ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.152+0000 7f889d570640 1 -- 172.21.15.60:0/530395410 shutdown_connections 2024-03-27T21:42:05.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.152+0000 7f889d570640 1 -- 172.21.15.60:0/530395410 wait complete. 2024-03-27T21:42:05.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.152+0000 7f889d570640 1 librados: shutdown 2024-03-27T21:42:05.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.152+0000 7f32a3f39640 1 -- 172.21.15.60:0/1304584283 shutdown_connections 2024-03-27T21:42:05.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.152+0000 7f32a3f39640 1 -- 172.21.15.60:0/1304584283 wait complete. 2024-03-27T21:42:05.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.152+0000 7f32a3f39640 1 librados: shutdown 2024-03-27T21:42:05.475 INFO:teuthology.orchestra.run.smithi060.stdout:55834574881 2024-03-27T21:42:05.475 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-27T21:42:05.476 INFO:teuthology.orchestra.run.smithi060.stdout:68719476771 2024-03-27T21:42:05.477 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-27T21:42:05.477 INFO:teuthology.orchestra.run.smithi060.stdout:68719476770 2024-03-27T21:42:05.478 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-27T21:42:05.478 INFO:teuthology.orchestra.run.smithi060.stdout:60129542177 2024-03-27T21:42:05.479 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-27T21:42:05.479 INFO:teuthology.orchestra.run.smithi060.stdout:60129542177 2024-03-27T21:42:05.480 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-27T21:42:05.480 INFO:teuthology.orchestra.run.smithi060.stdout:60129542177 2024-03-27T21:42:05.481 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-27T21:42:05.481 INFO:teuthology.orchestra.run.smithi060.stdout:266287972361 2024-03-27T21:42:05.482 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-27T21:42:05.483 INFO:teuthology.orchestra.run.smithi060.stdout:60129542177 2024-03-27T21:42:05.483 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-27T21:42:05.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.703+0000 7f3f364f1640 10 monclient: get_monmap_and_config 2024-03-27T21:42:05.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.703+0000 7f3f364f1640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.703+0000 7f3f364f1640 10 monclient: monmap: 2024-03-27T21:42:05.691 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.691 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.691 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:05.704305+0000 2024-03-27T21:42:05.691 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:05.704305+0000 2024-03-27T21:42:05.692 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.692 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.692 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:05.692 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:05.692 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:05.692 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:05.692 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:05.692 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.704+0000 7f3f364f1640 1 Processor -- start 2024-03-27T21:42:05.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.704+0000 7f3f364f1640 1 -- start start 2024-03-27T21:42:05.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.704+0000 7f3f364f1640 10 monclient: init 2024-03-27T21:42:05.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.704+0000 7f2e3359e640 10 monclient: get_monmap_and_config 2024-03-27T21:42:05.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.704+0000 7f2e3359e640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.704+0000 7f2e3359e640 10 monclient: monmap: 2024-03-27T21:42:05.693 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.693 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.694 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:05.705746+0000 2024-03-27T21:42:05.694 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:05.705746+0000 2024-03-27T21:42:05.694 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.694 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.694 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:05.695 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:05.695 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:05.695 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:05.695 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:05.695 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.704+0000 7f051b669640 10 monclient: get_monmap_and_config 2024-03-27T21:42:05.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.704+0000 7f051b669640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.704+0000 7f051b669640 10 monclient: monmap: 2024-03-27T21:42:05.696 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.696 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.697 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:05.705905+0000 2024-03-27T21:42:05.697 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:05.705905+0000 2024-03-27T21:42:05.697 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.697 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.697 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:05.697 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:05.697 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:05.697 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:05.697 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:05.698 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.705+0000 7f18c8f42640 10 monclient: get_monmap_and_config 2024-03-27T21:42:05.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.705+0000 7f18c8f42640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.705+0000 7f18c8f42640 10 monclient: monmap: 2024-03-27T21:42:05.699 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.699 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.699 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:05.706229+0000 2024-03-27T21:42:05.699 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:05.706229+0000 2024-03-27T21:42:05.699 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.699 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.700 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:05.700 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:05.700 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:05.700 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:05.700 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:05.700 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.705+0000 7fd755cb9640 10 monclient: get_monmap_and_config 2024-03-27T21:42:05.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.705+0000 7fd755cb9640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.705+0000 7fd755cb9640 10 monclient: monmap: 2024-03-27T21:42:05.701 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.701 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.701 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:05.706533+0000 2024-03-27T21:42:05.702 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:05.706533+0000 2024-03-27T21:42:05.702 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.702 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.702 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:05.702 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:05.702 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:05.702 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:05.702 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:05.702 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.705+0000 7f3f364f1640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.705+0000 7f3f364f1640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-03-27T21:42:05.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.705+0000 7f3f364f1640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3f3006d0f0 0x7f3f30151f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.705+0000 7f3f364f1640 10 monclient(hunting): picked mon.noname-b con 0x7f3f3006d0f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.705+0000 7f3f364f1640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3f3006d4d0 0x7f3f3006fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.705+0000 7f3f364f1640 10 monclient(hunting): picked mon.noname-c con 0x7f3f3006d4d0 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:05.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.705+0000 7f3f364f1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f30152440 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:05.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.705+0000 7f3f364f1640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3006d960 con 0x7f3f30152440 2024-03-27T21:42:05.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.705+0000 7f3f364f1640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.705+0000 7f3f364f1640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f30154ab0 con 0x7f3f3006d4d0 2024-03-27T21:42:05.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.705+0000 7f3f364f1640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.705+0000 7f3f364f1640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3006f5b0 con 0x7f3f3006d0f0 2024-03-27T21:42:05.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.705+0000 7f3f364f1640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.705+0000 7f3f364f1640 10 monclient(hunting): authenticate will time out at 966.926599s 2024-03-27T21:42:05.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f5f1a931640 10 monclient: get_monmap_and_config 2024-03-27T21:42:05.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f5f1a931640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f5f1a931640 10 monclient: monmap: 2024-03-27T21:42:05.706 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.706 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.706 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:05.707090+0000 2024-03-27T21:42:05.706 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:05.707090+0000 2024-03-27T21:42:05.706 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.707 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.707 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:05.707 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:05.707 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:05.707 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:05.707 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:05.707 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f34a8e640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f3f30152440 0x7f3f300ad540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.60:58328/0 (socket says 172.21.15.60:58328) 2024-03-27T21:42:05.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f34a8e640 1 -- 172.21.15.60:0/1812149803 learned_addr learned my addr 172.21.15.60:0/1812149803 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:05.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f051b669640 1 Processor -- start 2024-03-27T21:42:05.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2f7fe640 1 --2- 172.21.15.60:0/1812149803 >> v2:172.21.15.191:3300/0 conn(0x7f3f3006d4d0 0x7f3f3006fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2f7fe640 10 monclient(hunting): get_auth_request con 0x7f3f3006d4d0 auth_method 0 2024-03-27T21:42:05.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2ffff640 1 --2- 172.21.15.60:0/1812149803 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3f3006d0f0 0x7f3f30151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2ffff640 10 monclient(hunting): get_auth_request con 0x7f3f3006d0f0 auth_method 0 2024-03-27T21:42:05.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7fd755cb9640 1 Processor -- start 2024-03-27T21:42:05.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2f7fe640 10 monclient(hunting): handle_auth_done global_id 5227 payload 274 2024-03-27T21:42:05.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2f7fe640 1 -- 172.21.15.60:0/1812149803 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3f3006d0f0 msgr2=0x7f3f30151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7fd755cb9640 1 -- start start 2024-03-27T21:42:05.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7fd755cb9640 10 monclient: init 2024-03-27T21:42:05.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2f7fe640 1 --2- 172.21.15.60:0/1812149803 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3f3006d0f0 0x7f3f30151f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2f7fe640 1 -- 172.21.15.60:0/1812149803 >> v1:172.21.15.191:6789/0 conn(0x7f3f30152440 legacy=0x7f3f300ad540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2f7fe640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2f7fe640 1 monclient: found mon.noname-c 2024-03-27T21:42:05.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f3f2f7fe640 1 -- 172.21.15.60:0/1812149803 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3006f890 con 0x7f3f3006d4d0 2024-03-27T21:42:05.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f2e3359e640 1 Processor -- start 2024-03-27T21:42:05.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f2e3359e640 1 -- start start 2024-03-27T21:42:05.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f2e3359e640 10 monclient: init 2024-03-27T21:42:05.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e3359e640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e3359e640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-03-27T21:42:05.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e3359e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2e340b0ec0 0x7f2e34142be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e3359e640 10 monclient(hunting): picked mon.noname-a con 0x7f2e340b0ec0 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:05.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e3359e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2e34143120 0x7f2e34145ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e3359e640 10 monclient(hunting): picked mon.noname-c con 0x7f2e34143120 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:05.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e3359e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e34146000 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:05.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e3359e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e340b37e0 con 0x7f2e34146000 2024-03-27T21:42:05.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7fd755cb9640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7fd755cb9640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-03-27T21:42:05.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7fd755cb9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd7500b40c0 0x7fd7500a8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7fd755cb9640 10 monclient(hunting): picked mon.noname-a con 0x7fd7500b40c0 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:05.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7fd755cb9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd75013e830 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:05.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7fd755cb9640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd750141b10 0x7fd75013ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7fd755cb9640 10 monclient(hunting): picked mon.noname-b con 0x7fd750141b10 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7fd755cb9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7500b36f0 con 0x7fd75013e830 2024-03-27T21:42:05.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7fd755cb9640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7fd755cb9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7500b44a0 con 0x7fd7500b40c0 2024-03-27T21:42:05.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7fd755cb9640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7fd755cb9640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7500a8b90 con 0x7fd750141b10 2024-03-27T21:42:05.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7fd755cb9640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7fd755cb9640 10 monclient(hunting): authenticate will time out at 966.928580s 2024-03-27T21:42:05.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7fd74effd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd75013e830 0x7fd750141400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39392/0 (socket says 172.21.15.60:39392) 2024-03-27T21:42:05.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7fd74effd640 1 -- 172.21.15.60:0/303591116 learned_addr learned my addr 172.21.15.60:0/303591116 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:05.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7f18c8f42640 1 Processor -- start 2024-03-27T21:42:05.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7f5f1a931640 1 Processor -- start 2024-03-27T21:42:05.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7f5f1a931640 1 -- start start 2024-03-27T21:42:05.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7f5f1a931640 10 monclient: init 2024-03-27T21:42:05.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5f1a931640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5f1a931640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-03-27T21:42:05.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f051b669640 1 -- start start 2024-03-27T21:42:05.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.706+0000 7f051b669640 10 monclient: init 2024-03-27T21:42:05.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7f051b669640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7f051b669640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-03-27T21:42:05.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7f051b669640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0514060970 0x7f0514145790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7f051b669640 10 monclient(hunting): picked mon.noname-c con 0x7f0514060970 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:05.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7f051b669640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f05140b0270 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:05.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7f051b669640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f05140af8c0 0x7f05140717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7f051b669640 10 monclient(hunting): picked mon.noname-a con 0x7f05140af8c0 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:05.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7f051b669640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051406f660 con 0x7f05140b0270 2024-03-27T21:42:05.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7f051b669640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7f051b669640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f051406f940 con 0x7f05140af8c0 2024-03-27T21:42:05.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7f051b669640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7f051b669640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05140afd50 con 0x7f0514060970 2024-03-27T21:42:05.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7f051b669640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7f051b669640 10 monclient(hunting): authenticate will time out at 966.928500s 2024-03-27T21:42:05.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f0518c04640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f05140b0270 0x7f05140710a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39384/0 (socket says 172.21.15.60:39384) 2024-03-27T21:42:05.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f0518c04640 1 -- 172.21.15.60:0/3759162580 learned_addr learned my addr 172.21.15.60:0/3759162580 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:05.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e3359e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e3359e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e340b12a0 con 0x7f2e340b0ec0 2024-03-27T21:42:05.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e3359e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e3359e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e3406cdb0 con 0x7f2e34143120 2024-03-27T21:42:05.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e3359e640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e3359e640 10 monclient(hunting): authenticate will time out at 966.929127s 2024-03-27T21:42:05.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e31d9b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2e34143120 0x7f2e34145ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e31d9b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2e34143120 0x7f2e34145ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:58766/0 (socket says 172.21.15.60:58766) 2024-03-27T21:42:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e31d9b640 1 -- 172.21.15.60:0/1546772084 learned_addr learned my addr 172.21.15.60:0/1546772084 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e31d9b640 10 monclient(hunting): get_auth_request con 0x7f2e34143120 auth_method 0 2024-03-27T21:42:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e31d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e31d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e31d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e31d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e31d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5f1a931640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5f0c038fe0 0x7f5f0c0c9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5f1a931640 10 monclient(hunting): picked mon.noname-c con 0x7f5f0c038fe0 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5f1a931640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5f0c035640 0x7f5f0c0cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5f1a931640 10 monclient(hunting): picked mon.noname-b con 0x7f5f0c035640 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5f1a931640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5f0c0cc600 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5f1a931640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f0c035bf0 con 0x7f5f0c0cc600 2024-03-27T21:42:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5f1a931640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5f1a931640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f0c0393c0 con 0x7f5f0c038fe0 2024-03-27T21:42:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5f1a931640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5f1a931640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c0373e0 con 0x7f5f0c035640 2024-03-27T21:42:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5f1a931640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5f1a931640 10 monclient(hunting): authenticate will time out at 966.929152s 2024-03-27T21:42:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5f13fff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5f0c038fe0 0x7f5f0c0c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5f13fff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5f0c038fe0 0x7f5f0c0c9640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:58774/0 (socket says 172.21.15.60:58774) 2024-03-27T21:42:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5f13fff640 1 -- 172.21.15.60:0/1600145163 learned_addr learned my addr 172.21.15.60:0/1600145163 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5f13fff640 10 monclient(hunting): get_auth_request con 0x7f5f0c038fe0 auth_method 0 2024-03-27T21:42:05.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5f13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5f13fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5f13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f0519c06640 1 --2- 172.21.15.60:0/3759162580 >> v2:172.21.15.60:3300/0 conn(0x7f05140af8c0 0x7f05140717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7fd74f7fe640 1 --2- 172.21.15.60:0/303591116 >> v2:172.21.15.60:3300/0 conn(0x7fd7500b40c0 0x7fd7500a8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7fd74f7fe640 10 monclient(hunting): get_auth_request con 0x7fd7500b40c0 auth_method 0 2024-03-27T21:42:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7fd74f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7fd74f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7fd74f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f050bfff640 1 -- 172.21.15.60:0/3759162580 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383151046 0 0) 0x7f051406f660 con 0x7f05140b0270 2024-03-27T21:42:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f8919be2640 10 monclient: get_monmap_and_config 2024-03-27T21:42:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f050bfff640 10 monclient(hunting): _negotiate 2024-03-27T21:42:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f050bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f050bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f050bfff640 10 monclient(hunting): my global_id is 5319 2024-03-27T21:42:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f8919be2640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f5aa74e0640 10 monclient: get_monmap_and_config 2024-03-27T21:42:05.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f5aa74e0640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f5aa74e0640 10 monclient: monmap: 2024-03-27T21:42:05.729 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.729 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.729 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:05.710029+0000 2024-03-27T21:42:05.729 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:05.710029+0000 2024-03-27T21:42:05.729 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.729 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.729 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:05.730 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:05.730 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:05.730 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:05.730 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:05.730 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f050bfff640 1 -- 172.21.15.60:0/3759162580 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0500003520 con 0x7f05140b0270 2024-03-27T21:42:05.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f0519405640 1 --2- 172.21.15.60:0/3759162580 >> v2:172.21.15.191:3300/0 conn(0x7f0514060970 0x7f0514145790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f8919be2640 10 monclient: monmap: 2024-03-27T21:42:05.731 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.731 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.731 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:05.709851+0000 2024-03-27T21:42:05.731 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:05.709851+0000 2024-03-27T21:42:05.731 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.732 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.732 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:05.732 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:05.732 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:05.732 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:05.732 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:05.732 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f0519405640 10 monclient(hunting): get_auth_request con 0x7f0514060970 auth_method 0 2024-03-27T21:42:05.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f0519405640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f0519405640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 1 -- 172.21.15.60:0/3759162580 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 781430574 0 0) 0x7f0500003520 con 0x7f05140b0270 2024-03-27T21:42:05.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f0519405640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f0519c06640 10 monclient(hunting): get_auth_request con 0x7f05140af8c0 auth_method 0 2024-03-27T21:42:05.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f0519c06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f0519c06640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f0519c06640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7f18c8f42640 1 -- start start 2024-03-27T21:42:05.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7f18c8f42640 10 monclient: init 2024-03-27T21:42:05.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f18c8f42640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f18c8f42640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-03-27T21:42:05.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f18c8f42640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f18c4148f10 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:05.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f18c8f42640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f18c406ea80 0x7f18c4142840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f18c8f42640 10 monclient(hunting): picked mon.noname-a con 0x7f18c406ea80 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:05.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f18c8f42640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f18c40ad910 0x7f18c4142d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f18c8f42640 10 monclient(hunting): picked mon.noname-c con 0x7f18c40ad910 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:05.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f18c8f42640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c406d520 con 0x7f18c4148f10 2024-03-27T21:42:05.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f18c8f42640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f18c8f42640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c406d770 con 0x7f18c406ea80 2024-03-27T21:42:05.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f18c8f42640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f18c8f42640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c406f420 con 0x7f18c40ad910 2024-03-27T21:42:05.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f18c8f42640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f18c8f42640 10 monclient(hunting): authenticate will time out at 966.929282s 2024-03-27T21:42:05.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f18c2ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f18c406ea80 0x7f18c4142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f18c2ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f18c406ea80 0x7f18c4142840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39384/0 (socket says 172.21.15.60:39384) 2024-03-27T21:42:05.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f18c2ffd640 1 -- 172.21.15.60:0/703225396 learned_addr learned my addr 172.21.15.60:0/703225396 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f18c2ffd640 10 monclient(hunting): get_auth_request con 0x7f18c406ea80 auth_method 0 2024-03-27T21:42:05.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f18c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f18c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f18c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f18c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f18c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f18c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 1 -- 172.21.15.60:0/3759162580 >> v2:172.21.15.191:3300/0 conn(0x7f0514060970 msgr2=0x7f0514145790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 1 --2- 172.21.15.60:0/3759162580 >> v2:172.21.15.191:3300/0 conn(0x7f0514060970 0x7f0514145790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 1 -- 172.21.15.60:0/3759162580 >> v2:172.21.15.60:3300/0 conn(0x7f05140af8c0 msgr2=0x7f05140717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 1 --2- 172.21.15.60:0/3759162580 >> v2:172.21.15.60:3300/0 conn(0x7f05140af8c0 0x7f05140717b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74ffff640 1 --2- 172.21.15.60:0/303591116 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd750141b10 0x7fd75013ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74ffff640 10 monclient(hunting): get_auth_request con 0x7fd750141b10 auth_method 0 2024-03-27T21:42:05.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e31d9b640 10 monclient(hunting): handle_auth_done global_id 5233 payload 274 2024-03-27T21:42:05.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e31d9b640 1 -- 172.21.15.60:0/1546772084 >> v2:172.21.15.60:3300/0 conn(0x7f2e340b0ec0 msgr2=0x7f2e34142be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e31d9b640 1 --2- 172.21.15.60:0/1546772084 >> v2:172.21.15.60:3300/0 conn(0x7f2e340b0ec0 0x7f2e34142be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74e7fc640 1 -- 172.21.15.60:0/303591116 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3492387478 0 0) 0x7fd7500b36f0 con 0x7fd75013e830 2024-03-27T21:42:05.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74e7fc640 10 monclient(hunting): _negotiate 2024-03-27T21:42:05.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74e7fc640 10 monclient(hunting): my global_id is 5316 2024-03-27T21:42:05.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74e7fc640 1 -- 172.21.15.60:0/303591116 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd738003520 con 0x7fd75013e830 2024-03-27T21:42:05.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e31d9b640 1 -- 172.21.15.60:0/1546772084 >> v1:172.21.15.60:6789/0 conn(0x7f2e34146000 legacy=0x7f2e34148d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e31d9b640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e31d9b640 1 monclient: found mon.noname-c 2024-03-27T21:42:05.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e31d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e31d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74ffff640 10 monclient(hunting): handle_auth_done global_id 5030 payload 274 2024-03-27T21:42:05.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74ffff640 1 -- 172.21.15.60:0/303591116 >> v2:172.21.15.60:3300/0 conn(0x7fd7500b40c0 msgr2=0x7fd7500a8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74ffff640 1 --2- 172.21.15.60:0/303591116 >> v2:172.21.15.60:3300/0 conn(0x7fd7500b40c0 0x7fd7500a8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74ffff640 1 -- 172.21.15.60:0/303591116 >> v1:172.21.15.60:6789/0 conn(0x7fd75013e830 legacy=0x7fd750141400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74ffff640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74ffff640 1 monclient: found mon.noname-b 2024-03-27T21:42:05.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:42:05.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7fd74ffff640 1 -- 172.21.15.60:0/303591116 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7500a8db0 con 0x7fd750141b10 2024-03-27T21:42:05.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f2e31d9b640 1 -- 172.21.15.60:0/1546772084 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3406d090 con 0x7f2e34143120 2024-03-27T21:42:05.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 1 monclient: found mon.noname-a-legacy 2024-03-27T21:42:05.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f5f137fe640 1 --2- 172.21.15.60:0/1600145163 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5f0c035640 0x7f5f0c0cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f5f137fe640 10 monclient(hunting): get_auth_request con 0x7f5f0c035640 auth_method 0 2024-03-27T21:42:05.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f5f137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f5f137fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f5f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f5f13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f5f13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f5f13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f5aa74e0640 1 Processor -- start 2024-03-27T21:42:05.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f2e31d9b640 1 --2- 172.21.15.60:0/1546772084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e34143120 0x7f2e34145ac0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2e340b1490 tx=0x7f2e2802b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af8e516712cdfe0a server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f18c2ffd640 10 monclient(hunting): handle_auth_done global_id 5325 payload 274 2024-03-27T21:42:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f18c2ffd640 1 -- 172.21.15.60:0/703225396 >> v2:172.21.15.191:3300/0 conn(0x7f18c40ad910 msgr2=0x7f18c4142d80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T21:42:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f18c2ffd640 1 --2- 172.21.15.60:0/703225396 >> v2:172.21.15.191:3300/0 conn(0x7f18c40ad910 0x7f18c4142d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f18c2ffd640 1 -- 172.21.15.60:0/703225396 >> v1:172.21.15.60:6789/0 conn(0x7f18c4148f10 legacy=0x7f18c41492f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f18c2ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f18c2ffd640 1 monclient: found mon.noname-a 2024-03-27T21:42:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f18c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f18c2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:05.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f18c2ffd640 1 -- 172.21.15.60:0/703225396 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c406f700 con 0x7f18c406ea80 2024-03-27T21:42:05.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f5aa74e0640 1 -- start start 2024-03-27T21:42:05.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f5aa74e0640 10 monclient: init 2024-03-27T21:42:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f5f13fff640 10 monclient(hunting): handle_auth_done global_id 5236 payload 274 2024-03-27T21:42:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f5f13fff640 1 -- 172.21.15.60:0/1600145163 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5f0c035640 msgr2=0x7f5f0c0cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f5f13fff640 1 --2- 172.21.15.60:0/1600145163 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5f0c035640 0x7f5f0c0cc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f5f13fff640 1 -- 172.21.15.60:0/1600145163 >> v1:172.21.15.60:6789/0 conn(0x7f5f0c0cc600 legacy=0x7f5f0c034420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f5f13fff640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f5f13fff640 1 monclient: found mon.noname-c 2024-03-27T21:42:05.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f5f13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f5f13fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f5f13fff640 1 -- 172.21.15.60:0/1600145163 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f0c037650 con 0x7f5f0c038fe0 2024-03-27T21:42:05.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.707+0000 7f3f2f7fe640 1 --2- 172.21.15.60:0/1812149803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f3006d4d0 0x7f3f3006fcb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3f200097a0 tx=0x7f3f2002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30e5639684458275 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f3f2effd640 1 -- 172.21.15.60:0/1812149803 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f200316f0 con 0x7f3f3006d4d0 2024-03-27T21:42:05.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f3f2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f3f2effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:05.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f3f2effd640 10 monclient: dump: 2024-03-27T21:42:05.755 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.755 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.755 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.756 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.756 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.756 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.756 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.756 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.756 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.756 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f3f2effd640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f3f2effd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f3f2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f3f364f1640 5 monclient: authenticate success, global_id 5227 2024-03-27T21:42:05.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f3f364f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:05.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f3f2effd640 1 -- 172.21.15.60:0/1812149803 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f20031850 con 0x7f3f3006d4d0 2024-03-27T21:42:05.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f3f2effd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f3f364f1640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:05.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.708+0000 7f3f364f1640 4 set_mon_vals no callback set 2024-03-27T21:42:05.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f3f364f1640 10 monclient: shutdown 2024-03-27T21:42:05.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f3f364f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f3f364f1640 1 -- 172.21.15.60:0/1812149803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f3006d4d0 msgr2=0x7f3f3006fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f3f364f1640 1 --2- 172.21.15.60:0/1812149803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f3006d4d0 0x7f3f3006fcb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3f200097a0 tx=0x7f3f2002b790 comp rx=0 tx=0).stop 2024-03-27T21:42:05.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f3f364f1640 1 -- 172.21.15.60:0/1812149803 shutdown_connections 2024-03-27T21:42:05.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f3f364f1640 1 --2- 172.21.15.60:0/1812149803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f3006d4d0 0x7f3f3006fcb0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f3f364f1640 1 --2- 172.21.15.60:0/1812149803 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3f3006d0f0 0x7f3f30151f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f3f364f1640 1 -- 172.21.15.60:0/1812149803 >> 172.21.15.60:0/1812149803 conn(0x7f3f300b3620 msgr2=0x7f3f30060700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.60:6789/0 2024-03-27T21:42:05.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 1 -- 172.21.15.60:0/3759162580 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05140ae130 con 0x7f05140b0270 2024-03-27T21:42:05.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f2e3159a640 1 -- 172.21.15.60:0/1546772084 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e280317e0 con 0x7f2e34143120 2024-03-27T21:42:05.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f2e3159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f2e3159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:05.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f2e3159a640 10 monclient: dump: 2024-03-27T21:42:05.760 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.760 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.760 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.760 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.761 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f2e3159a640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f2e3159a640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f2e3159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f2e3159a640 1 -- 172.21.15.60:0/1546772084 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e28031940 con 0x7f2e34143120 2024-03-27T21:42:05.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f2e3159a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f2e3159a640 1 -- 172.21.15.60:0/1546772084 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e28031cd0 con 0x7f2e34143120 2024-03-27T21:42:05.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f2e3159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f2e3159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:42:05.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f2e3159a640 10 monclient: dump: 2024-03-27T21:42:05.762 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.762 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.763 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.763 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.763 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.763 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.763 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.763 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.763 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.763 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 1 -- 172.21.15.60:0/3759162580 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2933500084 0 0) 0x7f050c0019f0 con 0x7f05140b0270 2024-03-27T21:42:05.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-03-27T21:42:05.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 10 monclient: dump: 2024-03-27T21:42:05.766 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.766 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.766 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.766 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.766 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.766 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.767 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.767 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.767 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.767 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] but i'm connected to v1:172.21.15.60:6789/0, reconnecting 2024-03-27T21:42:05.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 1 -- 172.21.15.60:0/3759162580 >> v1:172.21.15.60:6789/0 conn(0x7f05140b0270 legacy=0x7f05140710a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:42:05.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 1 --2- 172.21.15.60:0/3759162580 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0500003970 0x7f0500003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 10 monclient(hunting): picked mon.c con 0x7f0500003970 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 1 --2- 172.21.15.60:0/3759162580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05000047c0 0x7f0500004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 10 monclient(hunting): picked mon.a con 0x7f05000047c0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:05.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 1 --2- 172.21.15.60:0/3759162580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0500005180 0x7f050000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 10 monclient(hunting): picked mon.b con 0x7f0500005180 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:05.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 1 -- 172.21.15.60:0/3759162580 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051406f940 con 0x7f05000047c0 2024-03-27T21:42:05.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f18c2ffd640 1 --2- 172.21.15.60:0/703225396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18c406ea80 0x7f18c4142840 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f18c40af090 tx=0x7f18b402b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f9aa6baa62a5d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f3f364f1640 1 -- 172.21.15.60:0/1812149803 shutdown_connections 2024-03-27T21:42:05.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 1 -- 172.21.15.60:0/3759162580 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05140afd50 con 0x7f0500005180 2024-03-27T21:42:05.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f8919be2640 1 Processor -- start 2024-03-27T21:42:05.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f8919be2640 1 -- start start 2024-03-27T21:42:05.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f8919be2640 10 monclient: init 2024-03-27T21:42:05.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 1 -- 172.21.15.60:0/3759162580 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f050000c080 con 0x7f0500003970 2024-03-27T21:42:05.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f8919be2640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f050bfff640 10 monclient(hunting): _finish_auth 0 2024-03-27T21:42:05.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f0518c04640 1 --2- 172.21.15.60:0/3759162580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05000047c0 0x7f0500004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f0518c04640 10 monclient(hunting): get_auth_request con 0x7f05000047c0 auth_method 0 2024-03-27T21:42:05.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f0518c04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f0518c04640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f0518c04640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:05.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.709+0000 7f0519405640 1 --2- 172.21.15.60:0/3759162580 >> v2:172.21.15.191:3300/0 conn(0x7f0514060970 0x7f0514145790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:42:05.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f0518c04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f0518c04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f0518c04640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:42:05.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f0519405640 1 --2- 172.21.15.60:0/3759162580 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0500003970 0x7f0500003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f0519405640 10 monclient(hunting): get_auth_request con 0x7f0500003970 auth_method 0 2024-03-27T21:42:05.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f0519405640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f0519405640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f0519405640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:05.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f0518c04640 10 monclient(hunting): handle_auth_done global_id 5319 payload 293 2024-03-27T21:42:05.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f0518c04640 1 -- 172.21.15.60:0/3759162580 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0500003970 msgr2=0x7f0500003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:05.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f0518c04640 1 --2- 172.21.15.60:0/3759162580 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f0500003970 0x7f0500003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f0518c04640 1 -- 172.21.15.60:0/3759162580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0500005180 msgr2=0x7f050000b980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T21:42:05.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f0518c04640 1 --2- 172.21.15.60:0/3759162580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0500005180 0x7f050000b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f0518c04640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f0518c04640 1 monclient: found mon.a 2024-03-27T21:42:05.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f0518c04640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f0518c04640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:05.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f0518c04640 1 -- 172.21.15.60:0/3759162580 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05140ae130 con 0x7f05000047c0 2024-03-27T21:42:05.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f0518c04640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f0518c04640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f0518c04640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f051b669640 5 monclient: authenticate success, global_id 5319 2024-03-27T21:42:05.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f051b669640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:05.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f8919be2640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-03-27T21:42:05.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f2e3359e640 5 monclient: authenticate success, global_id 5233 2024-03-27T21:42:05.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f2e3359e640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:05.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.710+0000 7f2e3359e640 4 set_mon_vals no callback set 2024-03-27T21:42:05.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f2e3359e640 10 monclient: shutdown 2024-03-27T21:42:05.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f2e3359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f2e3359e640 1 -- 172.21.15.60:0/1546772084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e34143120 msgr2=0x7f2e34145ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f2e3359e640 1 --2- 172.21.15.60:0/1546772084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e34143120 0x7f2e34145ac0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2e340b1490 tx=0x7f2e2802b880 comp rx=0 tx=0).stop 2024-03-27T21:42:05.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f18c27fc640 1 -- 172.21.15.60:0/703225396 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b4031800 con 0x7f18c406ea80 2024-03-27T21:42:05.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f18c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f18c27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:05.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f18c27fc640 10 monclient: dump: 2024-03-27T21:42:05.777 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.777 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.777 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.777 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.777 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.778 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.778 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.778 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.778 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.778 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f18c27fc640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f18c27fc640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f18c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f18c27fc640 1 -- 172.21.15.60:0/703225396 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b4031960 con 0x7f18c406ea80 2024-03-27T21:42:05.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f18c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f18c27fc640 1 -- 172.21.15.60:0/703225396 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b4031cf0 con 0x7f18c406ea80 2024-03-27T21:42:05.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f18c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f18c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:05.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f18c27fc640 10 monclient: dump: 2024-03-27T21:42:05.779 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.779 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.779 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.779 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.780 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.780 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.780 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.780 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.780 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.780 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f18c8f42640 5 monclient: authenticate success, global_id 5325 2024-03-27T21:42:05.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f18c8f42640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:05.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f18c8f42640 4 set_mon_vals no callback set 2024-03-27T21:42:05.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5f13fff640 1 --2- 172.21.15.60:0/1600145163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f0c038fe0 0x7f5f0c0c9640 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5f08009870 tx=0x7f5f0802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b328b9f5a0faa37 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f8919be2640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f89140b3f60 0x7f89140a8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f8919be2640 10 monclient(hunting): picked mon.noname-c con 0x7f89140b3f60 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:05.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f8919be2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f89140b3580 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:05.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f8919be2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f891413ecf0 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:05.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f89193e1640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f891413ecf0 0x7f8914141390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39430/0 (socket says 172.21.15.60:39430) 2024-03-27T21:42:05.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f89193e1640 1 -- 172.21.15.60:0/1112313188 learned_addr learned my addr 172.21.15.60:0/1112313188 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:05.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f8919be2640 1 -- 172.21.15.60:0/1112313188 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89140a8d00 con 0x7f891413ecf0 2024-03-27T21:42:05.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f8918be0640 1 --2- 172.21.15.60:0/1112313188 >> v2:172.21.15.191:3300/0 conn(0x7f89140b3f60 0x7f89140a8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f0519c06640 1 -- 172.21.15.60:0/3759162580 reap_dead start 2024-03-27T21:42:05.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f0518c04640 1 --2- 172.21.15.60:0/3759162580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05000047c0 0x7f0500004c40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f050c00a880 tx=0x7f05000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c1241ca34f264f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa74e0640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa74e0640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-03-27T21:42:05.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f8919be2640 1 -- 172.21.15.60:0/1112313188 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89140a9220 con 0x7f89140b3580 2024-03-27T21:42:05.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f8919be2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f8919be2640 1 -- 172.21.15.60:0/1112313188 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89140b4340 con 0x7f89140b3f60 2024-03-27T21:42:05.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f8919be2640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f8919be2640 10 monclient(hunting): authenticate will time out at 966.932833s 2024-03-27T21:42:05.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa74e0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5a980c82c0 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:05.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa74e0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5a98007b60 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:05.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa74e0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5a980c76d0 0x7f5a980bdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa74e0640 10 monclient(hunting): picked mon.noname-a con 0x7f5a980c76d0 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:05.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa74e0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a980c7b30 con 0x7f5a980c82c0 2024-03-27T21:42:05.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa74e0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a98033b40 con 0x7f5a98007b60 2024-03-27T21:42:05.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa74e0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa74e0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a980c86a0 con 0x7f5a980c76d0 2024-03-27T21:42:05.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7fd74ffff640 1 --2- 172.21.15.60:0/303591116 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd750141b10 0x7fd75013ec90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd7440097a0 tx=0x7fd74402b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b7d995b37c98b65 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7fd74e7fc640 1 -- 172.21.15.60:0/303591116 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd744030710 con 0x7fd750141b10 2024-03-27T21:42:05.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7fd74e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7fd74e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:05.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7fd74e7fc640 10 monclient: dump: 2024-03-27T21:42:05.786 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.786 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.786 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.786 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.786 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.786 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.787 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.787 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.787 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.787 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa74e0640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa74e0640 10 monclient(hunting): authenticate will time out at 966.932193s 2024-03-27T21:42:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa6cdf640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5a980c76d0 0x7f5a980bdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa6cdf640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5a980c76d0 0x7f5a980bdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39402/0 (socket says 172.21.15.60:39402) 2024-03-27T21:42:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa6cdf640 1 -- 172.21.15.60:0/3336867045 learned_addr learned my addr 172.21.15.60:0/3336867045 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa6cdf640 10 monclient(hunting): get_auth_request con 0x7f5a980c76d0 auth_method 0 2024-03-27T21:42:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa6cdf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa6cdf640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa6cdf640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa6cdf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa6cdf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f5aa6cdf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7fd74e7fc640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7fd74e7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7fd74e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7fd74e7fc640 1 -- 172.21.15.60:0/303591116 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74402e050 con 0x7fd750141b10 2024-03-27T21:42:05.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7fd74e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7fd74e7fc640 1 -- 172.21.15.60:0/303591116 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd744030a90 con 0x7fd750141b10 2024-03-27T21:42:05.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7fd74e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7fd74e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:05.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7fd74e7fc640 10 monclient: dump: 2024-03-27T21:42:05.790 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.790 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.790 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.790 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.790 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.790 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.791 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.791 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.791 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.791 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5f12ffd640 1 -- 172.21.15.60:0/1600145163 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f08031640 con 0x7f5f0c038fe0 2024-03-27T21:42:05.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5f12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5f12ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:05.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5f12ffd640 10 monclient: dump: 2024-03-27T21:42:05.792 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.792 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.792 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.792 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.792 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.792 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.792 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.793 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.793 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.793 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5f12ffd640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5f12ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5aa54dc640 1 -- 172.21.15.60:0/3336867045 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913221962 0 0) 0x7f5a98033b40 con 0x7f5a98007b60 2024-03-27T21:42:05.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5aa54dc640 10 monclient(hunting): _negotiate 2024-03-27T21:42:05.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5aa54dc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5aa54dc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5f12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5f12ffd640 1 -- 172.21.15.60:0/1600145163 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f080317a0 con 0x7f5f0c038fe0 2024-03-27T21:42:05.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5f12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5f12ffd640 1 -- 172.21.15.60:0/1600145163 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f08031b30 con 0x7f5f0c038fe0 2024-03-27T21:42:05.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5f12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5f12ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:42:05.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5aa54dc640 10 monclient(hunting): my global_id is 5242 2024-03-27T21:42:05.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5aa54dc640 1 -- 172.21.15.60:0/3336867045 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a8c003520 con 0x7f5a98007b60 2024-03-27T21:42:05.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5f12ffd640 10 monclient: dump: 2024-03-27T21:42:05.796 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.796 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.796 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.796 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.796 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.796 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.796 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.796 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.797 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.797 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5f1a931640 5 monclient: authenticate success, global_id 5236 2024-03-27T21:42:05.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5f1a931640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:05.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5f1a931640 4 set_mon_vals no callback set 2024-03-27T21:42:05.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7fd755cb9640 5 monclient: authenticate success, global_id 5030 2024-03-27T21:42:05.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7fd755cb9640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:05.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7fd755cb9640 4 set_mon_vals no callback set 2024-03-27T21:42:05.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5aa6cdf640 10 monclient(hunting): handle_auth_done global_id 5334 payload 274 2024-03-27T21:42:05.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5aa6cdf640 1 -- 172.21.15.60:0/3336867045 >> v1:172.21.15.191:6789/0 conn(0x7f5a98007b60 legacy=0x7f5a980c4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.711+0000 7f3f364f1640 1 -- 172.21.15.60:0/1812149803 wait complete. 2024-03-27T21:42:05.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5aa6cdf640 1 -- 172.21.15.60:0/3336867045 >> v1:172.21.15.60:6789/0 conn(0x7f5a980c82c0 legacy=0x7f5a98006240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5aa6cdf640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5aa6cdf640 1 monclient: found mon.noname-a 2024-03-27T21:42:05.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f89137fe640 1 -- 172.21.15.60:0/1112313188 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328648928 0 0) 0x7f89140a9220 con 0x7f89140b3580 2024-03-27T21:42:05.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f89137fe640 10 monclient(hunting): _negotiate 2024-03-27T21:42:05.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f89137fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f89137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f89137fe640 10 monclient(hunting): my global_id is 5245 2024-03-27T21:42:05.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5aa6cdf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f89137fe640 1 -- 172.21.15.60:0/1112313188 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88fc003520 con 0x7f89140b3580 2024-03-27T21:42:05.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5aa6cdf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f8918be0640 10 monclient(hunting): get_auth_request con 0x7f89140b3f60 auth_method 0 2024-03-27T21:42:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f8918be0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f8918be0640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f8918be0640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f89137fe640 1 -- 172.21.15.60:0/1112313188 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692466342 0 0) 0x7f89140a8d00 con 0x7f891413ecf0 2024-03-27T21:42:05.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f5aa6cdf640 1 -- 172.21.15.60:0/3336867045 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9802d100 con 0x7f5a980c76d0 2024-03-27T21:42:05.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f8918be0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f18c8f42640 10 monclient: shutdown 2024-03-27T21:42:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f18c8f42640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f18c8f42640 1 -- 172.21.15.60:0/703225396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18c406ea80 msgr2=0x7f18c4142840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f18c8f42640 1 --2- 172.21.15.60:0/703225396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18c406ea80 0x7f18c4142840 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f18c40af090 tx=0x7f18b402b8a0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f8918be0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f18c8f42640 1 -- 172.21.15.60:0/703225396 shutdown_connections 2024-03-27T21:42:05.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f18c8f42640 1 --2- 172.21.15.60:0/703225396 >> v2:172.21.15.191:3300/0 conn(0x7f18c40ad910 0x7f18c4142d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f18c8f42640 1 --2- 172.21.15.60:0/703225396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18c406ea80 0x7f18c4142840 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f3f364f1640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f3f364f1640 10 monclient: monmap: 2024-03-27T21:42:05.806 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.806 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:05.806 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:05.806 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:05.806 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.806 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:05.806 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:05.806 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:05.807 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:05.807 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f18c8f42640 1 -- 172.21.15.60:0/703225396 >> 172.21.15.60:0/703225396 conn(0x7f18c40b30f0 msgr2=0x7f18c40b3500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f2e3359e640 1 -- 172.21.15.60:0/1546772084 shutdown_connections 2024-03-27T21:42:05.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f2e3359e640 1 --2- 172.21.15.60:0/1546772084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e34143120 0x7f2e34145ac0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f2e3359e640 1 --2- 172.21.15.60:0/1546772084 >> v2:172.21.15.60:3300/0 conn(0x7f2e340b0ec0 0x7f2e34142be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f2e3359e640 1 -- 172.21.15.60:0/1546772084 >> 172.21.15.60:0/1546772084 conn(0x7f2e340608e0 msgr2=0x7f2e34060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f8918be0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f8918be0640 10 monclient(hunting): handle_auth_done global_id 5248 payload 274 2024-03-27T21:42:05.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f8918be0640 1 -- 172.21.15.60:0/1112313188 >> v1:172.21.15.191:6789/0 conn(0x7f89140b3580 legacy=0x7f891413e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f8918be0640 1 -- 172.21.15.60:0/1112313188 >> v1:172.21.15.60:6789/0 conn(0x7f891413ecf0 legacy=0x7f8914141390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f8918be0640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f8918be0640 1 monclient: found mon.noname-c 2024-03-27T21:42:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f8918be0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f8918be0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.712+0000 7f8918be0640 1 -- 172.21.15.60:0/1112313188 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89140a93b0 con 0x7f89140b3f60 2024-03-27T21:42:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f89137fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:42:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f5f1a931640 10 monclient: shutdown 2024-03-27T21:42:05.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f5f1a931640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f5f1a931640 1 -- 172.21.15.60:0/1600145163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f0c038fe0 msgr2=0x7f5f0c0c9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f5f1a931640 1 --2- 172.21.15.60:0/1600145163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f0c038fe0 0x7f5f0c0c9640 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5f08009870 tx=0x7f5f0802b8f0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f2e3359e640 1 -- 172.21.15.60:0/1546772084 shutdown_connections 2024-03-27T21:42:05.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f2e3359e640 1 -- 172.21.15.60:0/1546772084 wait complete. 2024-03-27T21:42:05.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f5f1a931640 1 -- 172.21.15.60:0/1600145163 shutdown_connections 2024-03-27T21:42:05.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f5f1a931640 1 --2- 172.21.15.60:0/1600145163 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5f0c035640 0x7f5f0c0cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f5f1a931640 1 --2- 172.21.15.60:0/1600145163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f0c038fe0 0x7f5f0c0c9640 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f5f1a931640 1 -- 172.21.15.60:0/1600145163 >> 172.21.15.60:0/1600145163 conn(0x7f5f0c0065d0 msgr2=0x7f5f0c0069e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f3f364f1640 1 librados: starting msgr at 2024-03-27T21:42:05.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f3f364f1640 1 librados: starting objecter 2024-03-27T21:42:05.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f5f1a931640 1 -- 172.21.15.60:0/1600145163 shutdown_connections 2024-03-27T21:42:05.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7fd755cb9640 10 monclient: shutdown 2024-03-27T21:42:05.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7fd755cb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7fd755cb9640 1 -- 172.21.15.60:0/303591116 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd750141b10 msgr2=0x7fd75013ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7fd755cb9640 1 --2- 172.21.15.60:0/303591116 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd750141b10 0x7fd75013ec90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd7440097a0 tx=0x7fd74402b710 comp rx=0 tx=0).stop 2024-03-27T21:42:05.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f3f364f1640 1 Processor -- start 2024-03-27T21:42:05.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f5aa6cdf640 1 --2- 172.21.15.60:0/3336867045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a980c76d0 0x7f5a980bdb50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5a90013150 tx=0x7f5a90034120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=accdb106350dfc2d server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7fd755cb9640 1 -- 172.21.15.60:0/303591116 shutdown_connections 2024-03-27T21:42:05.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7fd755cb9640 1 --2- 172.21.15.60:0/303591116 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd750141b10 0x7fd75013ec90 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7fd755cb9640 1 --2- 172.21.15.60:0/303591116 >> v2:172.21.15.60:3300/0 conn(0x7fd7500b40c0 0x7fd7500a8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7fd755cb9640 1 -- 172.21.15.60:0/303591116 >> 172.21.15.60:0/303591116 conn(0x7fd7500608e0 msgr2=0x7fd750060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f050bfff640 1 -- 172.21.15.60:0/3759162580 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c034630 con 0x7f05000047c0 2024-03-27T21:42:05.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f050bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f3f364f1640 1 -- start start 2024-03-27T21:42:05.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f3f364f1640 1 librados: setting wanted keys 2024-03-27T21:42:05.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f3f364f1640 1 librados: calling monclient init 2024-03-27T21:42:05.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f3f364f1640 10 monclient: init 2024-03-27T21:42:05.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f050bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:05.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f050bfff640 10 monclient: dump: 2024-03-27T21:42:05.816 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.816 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.816 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.817 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.817 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.817 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.817 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.817 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.817 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.817 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f050bfff640 1 -- 172.21.15.60:0/3759162580 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050c02f070 con 0x7f05000047c0 2024-03-27T21:42:05.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7fd755cb9640 1 -- 172.21.15.60:0/303591116 shutdown_connections 2024-03-27T21:42:05.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f5aa54dc640 1 -- 172.21.15.60:0/3336867045 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a900110d0 con 0x7f5a980c76d0 2024-03-27T21:42:05.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f5aa54dc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f5aa54dc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:05.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f5aa54dc640 10 monclient: dump: 2024-03-27T21:42:05.819 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.819 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.819 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.819 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.819 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.819 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.819 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.819 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.820 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.820 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f5aa54dc640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f5aa54dc640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f2e3359e640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f2e3359e640 10 monclient: monmap: 2024-03-27T21:42:05.821 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.821 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:05.821 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:05.821 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:05.821 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.821 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:05.821 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:05.821 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:05.822 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:05.822 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f5aa54dc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f5aa74e0640 5 monclient: authenticate success, global_id 5334 2024-03-27T21:42:05.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f5aa74e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:05.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f5aa54dc640 1 -- 172.21.15.60:0/3336867045 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a90011230 con 0x7f5a980c76d0 2024-03-27T21:42:05.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f5aa54dc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f18c8f42640 1 -- 172.21.15.60:0/703225396 shutdown_connections 2024-03-27T21:42:05.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f3f364f1640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f5aa74e0640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:05.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f5aa74e0640 4 set_mon_vals no callback set 2024-03-27T21:42:05.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f5aa54dc640 1 -- 172.21.15.60:0/3336867045 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a900115c0 con 0x7f5a980c76d0 2024-03-27T21:42:05.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f5aa54dc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f5aa54dc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:05.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f5aa54dc640 10 monclient: dump: 2024-03-27T21:42:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.826 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f3f364f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:42:05.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f3f364f1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f3006d0f0 0x7f3f30085ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f2e3359e640 1 librados: starting msgr at 2024-03-27T21:42:05.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f2e3359e640 1 librados: starting objecter 2024-03-27T21:42:05.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f3f364f1640 10 monclient(hunting): picked mon.noname-a con 0x7f3f3006d0f0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:05.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f3f364f1640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f30152440 0x7f3f301ebc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f3f364f1640 10 monclient(hunting): picked mon.noname-b con 0x7f3f30152440 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:05.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f3f364f1640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3f30083230 0x7f3f301ec180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f2e3359e640 1 Processor -- start 2024-03-27T21:42:05.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f18c8f42640 1 -- 172.21.15.60:0/703225396 wait complete. 2024-03-27T21:42:05.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f051b669640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:05.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7f5f1a931640 1 -- 172.21.15.60:0/1600145163 wait complete. 2024-03-27T21:42:05.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f5f1a931640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f5f1a931640 10 monclient: monmap: 2024-03-27T21:42:05.829 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.830 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:05.830 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:05.830 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:05.830 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.830 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:05.830 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:05.830 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:05.830 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:05.831 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f050bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f5f1a931640 1 librados: starting msgr at 2024-03-27T21:42:05.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f5f1a931640 1 librados: starting objecter 2024-03-27T21:42:05.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f5f1a931640 1 Processor -- start 2024-03-27T21:42:05.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f050bfff640 1 -- 172.21.15.60:0/3759162580 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c034a10 con 0x7f05000047c0 2024-03-27T21:42:05.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f051b669640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:05.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f051b669640 4 set_mon_vals no callback set 2024-03-27T21:42:05.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f050bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f050bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:05.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f050bfff640 10 monclient: dump: 2024-03-27T21:42:05.833 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.833 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.833 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.833 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.833 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.833 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.833 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.833 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.834 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.834 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f3f2ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f3006d0f0 0x7f3f30085ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f3f2ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f3006d0f0 0x7f3f30085ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39404/0 (socket says 172.21.15.60:39404) 2024-03-27T21:42:05.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f3f364f1640 10 monclient(hunting): picked mon.noname-c con 0x7f3f30083230 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f3f364f1640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f3f364f1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f301eb7b0 con 0x7f3f3006d0f0 2024-03-27T21:42:05.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f3f364f1640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f3f364f1640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f300873d0 con 0x7f3f30152440 2024-03-27T21:42:05.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f3f364f1640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f3f364f1640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f300ad010 con 0x7f3f30083230 2024-03-27T21:42:05.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f3f364f1640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f3f364f1640 10 monclient(hunting): authenticate will time out at 966.935647s 2024-03-27T21:42:05.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f3f34a8e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3f30083230 0x7f3f301ec180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f34a8e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3f30083230 0x7f3f301ec180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34210/0 (socket says 172.21.15.60:34210) 2024-03-27T21:42:05.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f34a8e640 1 -- 172.21.15.60:0/2588636431 learned_addr learned my addr 172.21.15.60:0/2588636431 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f34a8e640 10 monclient(hunting): get_auth_request con 0x7f3f30083230 auth_method 0 2024-03-27T21:42:05.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f34a8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f34a8e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f34a8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f2ffff640 10 monclient(hunting): get_auth_request con 0x7f3f3006d0f0 auth_method 0 2024-03-27T21:42:05.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f34a8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f34a8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f34a8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f2f7fe640 1 --2- 172.21.15.60:0/2588636431 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f30152440 0x7f3f301ebc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f2f7fe640 10 monclient(hunting): get_auth_request con 0x7f3f30152440 auth_method 0 2024-03-27T21:42:05.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f2e3359e640 1 -- start start 2024-03-27T21:42:05.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f2e3359e640 1 librados: setting wanted keys 2024-03-27T21:42:05.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f2e3359e640 1 librados: calling monclient init 2024-03-27T21:42:05.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f2e3359e640 10 monclient: init 2024-03-27T21:42:05.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.713+0000 7fd755cb9640 1 -- 172.21.15.60:0/303591116 wait complete. 2024-03-27T21:42:05.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7fd755cb9640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7fd755cb9640 10 monclient: monmap: 2024-03-27T21:42:05.839 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.839 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:05.839 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:05.839 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:05.839 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.840 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:05.840 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:05.840 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:05.840 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:05.840 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7fd755cb9640 1 librados: starting msgr at 2024-03-27T21:42:05.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7fd755cb9640 1 librados: starting objecter 2024-03-27T21:42:05.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7fd755cb9640 1 Processor -- start 2024-03-27T21:42:05.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f18c8f42640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f18c8f42640 10 monclient: monmap: 2024-03-27T21:42:05.841 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.841 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:05.841 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:05.841 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:05.842 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.842 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:05.842 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:05.842 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:05.842 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:05.842 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f18c8f42640 1 librados: starting msgr at 2024-03-27T21:42:05.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f18c8f42640 1 librados: starting objecter 2024-03-27T21:42:05.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f5aa74e0640 10 monclient: shutdown 2024-03-27T21:42:05.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f5aa74e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f5aa74e0640 1 -- 172.21.15.60:0/3336867045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a980c76d0 msgr2=0x7f5a980bdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f5aa74e0640 1 --2- 172.21.15.60:0/3336867045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a980c76d0 0x7f5a980bdb50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5a90013150 tx=0x7f5a90034120 comp rx=0 tx=0).stop 2024-03-27T21:42:05.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f5aa74e0640 1 -- 172.21.15.60:0/3336867045 shutdown_connections 2024-03-27T21:42:05.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f5aa74e0640 1 --2- 172.21.15.60:0/3336867045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a980c76d0 0x7f5a980bdb50 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f5aa74e0640 1 -- 172.21.15.60:0/3336867045 >> 172.21.15.60:0/3336867045 conn(0x7f5a98006ad0 msgr2=0x7f5a98006ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f051b669640 10 monclient: shutdown 2024-03-27T21:42:05.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f051b669640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f051b669640 1 -- 172.21.15.60:0/3759162580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05000047c0 msgr2=0x7f0500004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f051b669640 1 --2- 172.21.15.60:0/3759162580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05000047c0 0x7f0500004c40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f050c00a880 tx=0x7f05000057f0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f18c8f42640 1 Processor -- start 2024-03-27T21:42:05.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f051b669640 1 -- 172.21.15.60:0/3759162580 shutdown_connections 2024-03-27T21:42:05.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f051b669640 1 --2- 172.21.15.60:0/3759162580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05000047c0 0x7f0500004c40 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f051b669640 1 -- 172.21.15.60:0/3759162580 >> 172.21.15.60:0/3759162580 conn(0x7f05140b3b50 msgr2=0x7f05140b3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f2e3359e640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f3f2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f3f34a8e640 10 monclient(hunting): handle_auth_done global_id 5036 payload 824 2024-03-27T21:42:05.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f2e3359e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:42:05.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f2e3359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e340b0ec0 0x7f2e3406eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7fd755cb9640 1 -- start start 2024-03-27T21:42:05.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7fd755cb9640 1 librados: setting wanted keys 2024-03-27T21:42:05.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7fd755cb9640 1 librados: calling monclient init 2024-03-27T21:42:05.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7fd755cb9640 10 monclient: init 2024-03-27T21:42:05.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f18c8f42640 1 -- start start 2024-03-27T21:42:05.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f18c8f42640 1 librados: setting wanted keys 2024-03-27T21:42:05.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f18c8f42640 1 librados: calling monclient init 2024-03-27T21:42:05.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f18c8f42640 10 monclient: init 2024-03-27T21:42:05.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f2e3359e640 10 monclient(hunting): picked mon.noname-a con 0x7f2e340b0ec0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:05.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.714+0000 7f8918be0640 1 --2- 172.21.15.60:0/1112313188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89140b3f60 0x7f89140a8650 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8904009720 tx=0x7f890402b680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc5e8e4856280f76 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f89137fe640 1 -- 172.21.15.60:0/1112313188 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908001670 con 0x7f89140b3f60 2024-03-27T21:42:05.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f89137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f89137fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:05.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f89137fe640 10 monclient: dump: 2024-03-27T21:42:05.851 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.851 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.851 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.851 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.851 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.851 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.851 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.851 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.852 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.852 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f89137fe640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f89137fe640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f89137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f89137fe640 1 -- 172.21.15.60:0/1112313188 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f890402e050 con 0x7f89140b3f60 2024-03-27T21:42:05.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f89137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f89137fe640 1 -- 172.21.15.60:0/1112313188 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8904030730 con 0x7f89140b3f60 2024-03-27T21:42:05.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f89137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f89137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:42:05.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f89137fe640 10 monclient: dump: 2024-03-27T21:42:05.853 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.853 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.853 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.853 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.853 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.853 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.853 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.854 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.854 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.854 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f051b669640 1 -- 172.21.15.60:0/3759162580 shutdown_connections 2024-03-27T21:42:05.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f8919be2640 5 monclient: authenticate success, global_id 5248 2024-03-27T21:42:05.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f8919be2640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:05.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f8919be2640 4 set_mon_vals no callback set 2024-03-27T21:42:05.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f051b669640 1 -- 172.21.15.60:0/3759162580 wait complete. 2024-03-27T21:42:05.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f2e3259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e340b0ec0 0x7f2e3406eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f2e3259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e340b0ec0 0x7f2e3406eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39412/0 (socket says 172.21.15.60:39412) 2024-03-27T21:42:05.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f2e3259c640 1 -- 172.21.15.60:0/1344941168 learned_addr learned my addr 172.21.15.60:0/1344941168 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f2e3359e640 1 --2- 172.21.15.60:0/1344941168 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e34143120 0x7f2e341da6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f2e3359e640 10 monclient(hunting): picked mon.noname-b con 0x7f2e34143120 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:05.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f2e3359e640 1 --2- 172.21.15.60:0/1344941168 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2e34146000 0x7f2e341dabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f5f1a931640 1 -- start start 2024-03-27T21:42:05.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f5f1a931640 1 librados: setting wanted keys 2024-03-27T21:42:05.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f5f1a931640 1 librados: calling monclient init 2024-03-27T21:42:05.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f5f1a931640 10 monclient: init 2024-03-27T21:42:05.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f2e31d9b640 1 --2- 172.21.15.60:0/1344941168 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e34143120 0x7f2e341da6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f5f1a931640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f5f1a931640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:42:05.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f5f1a931640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f0c035640 0x7f5f0c02f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f5f1a931640 10 monclient(hunting): picked mon.noname-b con 0x7f5f0c035640 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:05.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.715+0000 7f5f1a931640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5f0c038fe0 0x7f5f0c033a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f5f137fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5f0c038fe0 0x7f5f0c033a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f5f137fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5f0c038fe0 0x7f5f0c033a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34216/0 (socket says 172.21.15.60:34216) 2024-03-27T21:42:05.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f2e3359e640 10 monclient(hunting): picked mon.noname-c con 0x7f2e34146000 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f2e3359e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f2e3359e640 1 -- 172.21.15.60:0/1344941168 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e341435c0 con 0x7f2e340b0ec0 2024-03-27T21:42:05.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f2e3359e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f2e3359e640 1 -- 172.21.15.60:0/1344941168 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e34143810 con 0x7f2e34143120 2024-03-27T21:42:05.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f2e3359e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7fd755cb9640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7fd755cb9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:42:05.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7fd755cb9640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd7500b40c0 0x7fd7501e6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7fd755cb9640 10 monclient(hunting): picked mon.noname-c con 0x7fd7500b40c0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7fd755cb9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd75013e830 0x7fd7500b4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7fd755cb9640 10 monclient(hunting): picked mon.noname-b con 0x7fd75013e830 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:05.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7fd755cb9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd750141b10 0x7fd7500b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f2e3359e640 1 -- 172.21.15.60:0/1344941168 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e34143d30 con 0x7f2e34146000 2024-03-27T21:42:05.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7fd755cb9640 10 monclient(hunting): picked mon.noname-a con 0x7fd750141b10 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:05.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7fd755cb9640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7fd755cb9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7500b6c90 con 0x7fd750141b10 2024-03-27T21:42:05.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7fd755cb9640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7fd755cb9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7500b2250 con 0x7fd75013e830 2024-03-27T21:42:05.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7fd755cb9640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7fd755cb9640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7500b68b0 con 0x7fd7500b40c0 2024-03-27T21:42:05.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7fd755cb9640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7fd755cb9640 10 monclient(hunting): authenticate will time out at 966.937769s 2024-03-27T21:42:05.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f2e3359e640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f2e3359e640 10 monclient(hunting): authenticate will time out at 966.937847s 2024-03-27T21:42:05.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f5aa74e0640 1 -- 172.21.15.60:0/3336867045 shutdown_connections 2024-03-27T21:42:05.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f2e3259c640 10 monclient(hunting): get_auth_request con 0x7f2e340b0ec0 auth_method 0 2024-03-27T21:42:05.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f2e3259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e3259c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e3259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f18c8f42640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e31d9b640 10 monclient(hunting): get_auth_request con 0x7f2e34143120 auth_method 0 2024-03-27T21:42:05.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e31d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f18c8f42640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:42:05.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e31d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f18c8f42640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f18c406ea80 0x7f18c40af9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f18c8f42640 10 monclient(hunting): picked mon.noname-c con 0x7f18c406ea80 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e31d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e31d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e31d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e3259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e3259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e3259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f5aa74e0640 1 -- 172.21.15.60:0/3336867045 wait complete. 2024-03-27T21:42:05.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e32d9d640 1 --2- 172.21.15.60:0/1344941168 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2e34146000 0x7f2e341dabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f18c8f42640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18c40ad910 0x7f18c4141ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e32d9d640 10 monclient(hunting): get_auth_request con 0x7f2e34146000 auth_method 0 2024-03-27T21:42:05.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e32d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f8919be2640 10 monclient: shutdown 2024-03-27T21:42:05.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f8919be2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:05.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f8919be2640 1 -- 172.21.15.60:0/1112313188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89140b3f60 msgr2=0x7f89140a8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f8919be2640 1 --2- 172.21.15.60:0/1112313188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89140b3f60 0x7f89140a8650 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8904009720 tx=0x7f890402b680 comp rx=0 tx=0).stop 2024-03-27T21:42:05.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e32d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e32d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f8919be2640 1 -- 172.21.15.60:0/1112313188 shutdown_connections 2024-03-27T21:42:05.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f8919be2640 1 --2- 172.21.15.60:0/1112313188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89140b3f60 0x7f89140a8650 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f8919be2640 1 -- 172.21.15.60:0/1112313188 >> 172.21.15.60:0/1112313188 conn(0x7f89140608e0 msgr2=0x7f8914060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:05.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f8919be2640 1 -- 172.21.15.60:0/1112313188 shutdown_connections 2024-03-27T21:42:05.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f8919be2640 1 -- 172.21.15.60:0/1112313188 wait complete. 2024-03-27T21:42:05.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f3f34a8e640 1 -- 172.21.15.60:0/2588636431 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f30152440 msgr2=0x7f3f301ebc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f3f34a8e640 1 --2- 172.21.15.60:0/2588636431 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f30152440 0x7f3f301ebc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f3f34a8e640 1 -- 172.21.15.60:0/2588636431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f3006d0f0 msgr2=0x7f3f30085ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f3f34a8e640 1 --2- 172.21.15.60:0/2588636431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f3006d0f0 0x7f3f30085ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f3f34a8e640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f3f34a8e640 1 monclient: found mon.noname-c 2024-03-27T21:42:05.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f3f34a8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f3f34a8e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:05.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f3f34a8e640 1 -- 172.21.15.60:0/2588636431 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f20009400 con 0x7f3f30083230 2024-03-27T21:42:05.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f3f2ffff640 1 --2- 172.21.15.60:0/2588636431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f3006d0f0 0x7f3f30085ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:05.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f3f34a8e640 1 --2- 172.21.15.60:0/2588636431 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3f30083230 0x7f3f301ec180 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3f2800cd40 tx=0x7f3f2800a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f3f2cff9640 10 client.?.objecter ms_handle_connect 0x7f3f30083230 2024-03-27T21:42:05.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.716+0000 7f3f2cff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:05.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f2cff9640 1 -- 172.21.15.60:0/2588636431 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2800ad40 con 0x7f3f30083230 2024-03-27T21:42:05.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f2cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:05.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f2cff9640 10 monclient: dump: 2024-03-27T21:42:05.874 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.874 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.874 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.874 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.874 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.874 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.875 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.875 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.875 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.875 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f2cff9640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f2cff9640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f2cff9640 1 -- 172.21.15.60:0/2588636431 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f28012400 con 0x7f3f30083230 2024-03-27T21:42:05.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f2cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f2cff9640 1 -- 172.21.15.60:0/2588636431 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2800aea0 con 0x7f3f30083230 2024-03-27T21:42:05.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f2cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:05.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f2cff9640 10 monclient: dump: 2024-03-27T21:42:05.876 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.876 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.876 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.876 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.876 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.876 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.877 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.877 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.877 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.877 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f2e7fc640 4 set_mon_vals no callback set 2024-03-27T21:42:05.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e31d9b640 10 monclient(hunting): handle_auth_done global_id 5254 payload 824 2024-03-27T21:42:05.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e31d9b640 1 -- 172.21.15.60:0/1344941168 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2e34146000 msgr2=0x7f2e341dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e31d9b640 1 --2- 172.21.15.60:0/1344941168 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2e34146000 0x7f2e341dabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e31d9b640 1 -- 172.21.15.60:0/1344941168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e340b0ec0 msgr2=0x7f2e3406eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e31d9b640 1 --2- 172.21.15.60:0/1344941168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e340b0ec0 0x7f2e3406eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e31d9b640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e31d9b640 1 monclient: found mon.noname-b 2024-03-27T21:42:05.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e31d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e31d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e31d9b640 1 -- 172.21.15.60:0/1344941168 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e280094d0 con 0x7f2e34143120 2024-03-27T21:42:05.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f051b669640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f051b669640 10 monclient: monmap: 2024-03-27T21:42:05.879 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.879 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:05.879 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:05.879 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:05.879 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.879 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:05.880 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:05.880 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:05.880 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:05.880 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7fd74effd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd75013e830 0x7fd7500b4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f051b669640 1 librados: starting msgr at 2024-03-27T21:42:05.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f051b669640 1 librados: starting objecter 2024-03-27T21:42:05.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f051b669640 1 Processor -- start 2024-03-27T21:42:05.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7fd74f7fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd7500b40c0 0x7fd7501e6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7fd74ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd750141b10 0x7fd7500b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f364f1640 5 monclient: authenticate success, global_id 5036 2024-03-27T21:42:05.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f364f1640 10 monclient: _renew_subs 2024-03-27T21:42:05.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f364f1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:05.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f364f1640 1 -- 172.21.15.60:0/2588636431 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f300854a0 con 0x7f3f30083230 2024-03-27T21:42:05.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f364f1640 10 client.5036.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f364f1640 10 monclient: _renew_subs 2024-03-27T21:42:05.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f364f1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:05.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f364f1640 1 -- 172.21.15.60:0/2588636431 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f301eec00 con 0x7f3f30083230 2024-03-27T21:42:05.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f3f364f1640 1 librados: init done 2024-03-27T21:42:05.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e3259c640 1 --2- 172.21.15.60:0/1344941168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e340b0ec0 0x7f2e3406eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:05.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f5f13fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f0c035640 0x7f5f0c02f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f5f13fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f0c035640 0x7f5f0c02f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:58792/0 (socket says 172.21.15.60:58792) 2024-03-27T21:42:05.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f5f137fe640 1 -- 172.21.15.60:0/2535368581 learned_addr learned my addr 172.21.15.60:0/2535368581 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f5f1a931640 10 monclient(hunting): picked mon.noname-c con 0x7f5f0c038fe0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f5f1a931640 1 --2- 172.21.15.60:0/2535368581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f0c0cc600 0x7f5f0c16a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f5f18ece640 1 --2- 172.21.15.60:0/2535368581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f0c0cc600 0x7f5f0c16a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f5f1a931640 10 monclient(hunting): picked mon.noname-a con 0x7f5f0c0cc600 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:05.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f5f1a931640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f5f1a931640 1 -- 172.21.15.60:0/2535368581 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c169cc0 con 0x7f5f0c0cc600 2024-03-27T21:42:05.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f5f1a931640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f5f1a931640 1 -- 172.21.15.60:0/2535368581 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c02edf0 con 0x7f5f0c035640 2024-03-27T21:42:05.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f5f1a931640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f5f1a931640 1 -- 172.21.15.60:0/2535368581 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c033f80 con 0x7f5f0c038fe0 2024-03-27T21:42:05.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f5f1a931640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f5f1a931640 10 monclient(hunting): authenticate will time out at 966.938790s 2024-03-27T21:42:05.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e32d9d640 1 --2- 172.21.15.60:0/1344941168 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2e34146000 0x7f2e341dabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:42:05.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f2e31d9b640 1 --2- 172.21.15.60:0/1344941168 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e34143120 0x7f2e341da6a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2e28033000 tx=0x7f2e280046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7fd74f7fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd7500b40c0 0x7fd7501e6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34228/0 (socket says 172.21.15.60:34228) 2024-03-27T21:42:05.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7fd74f7fe640 1 -- 172.21.15.60:0/3097219464 learned_addr learned my addr 172.21.15.60:0/3097219464 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f2e22ffd640 10 client.?.objecter ms_handle_connect 0x7f2e34143120 2024-03-27T21:42:05.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f5aa74e0640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f5aa74e0640 10 monclient: monmap: 2024-03-27T21:42:05.887 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.887 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:05.887 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:05.887 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:05.888 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.888 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:05.888 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:05.888 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:05.888 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:05.888 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f2e22ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:05.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7fd74ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd750141b10 0x7fd7500b7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39416/0 (socket says 172.21.15.60:39416) 2024-03-27T21:42:05.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7fd74f7fe640 10 monclient(hunting): get_auth_request con 0x7fd7500b40c0 auth_method 0 2024-03-27T21:42:05.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f5aa74e0640 1 librados: starting msgr at 2024-03-27T21:42:05.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f5aa74e0640 1 librados: starting objecter 2024-03-27T21:42:05.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f5aa74e0640 1 Processor -- start 2024-03-27T21:42:05.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f2e22ffd640 1 -- 172.21.15.60:0/1344941168 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e28032cb0 con 0x7f2e34143120 2024-03-27T21:42:05.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f2e22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7fd74f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7fd74f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7fd74f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f2e22ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:05.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f2e22ffd640 10 monclient: dump: 2024-03-27T21:42:05.891 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.891 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.892 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.892 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.892 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.892 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.892 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.892 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.892 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.892 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f2e22ffd640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7fd74ffff640 10 monclient(hunting): get_auth_request con 0x7fd750141b10 auth_method 0 2024-03-27T21:42:05.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7fd74ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f2e22ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f2e22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7fd74ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f2e22ffd640 1 -- 172.21.15.60:0/1344941168 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e28032e10 con 0x7f2e34143120 2024-03-27T21:42:05.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7fd74ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f2e22ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f2e22ffd640 1 -- 172.21.15.60:0/1344941168 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e28045610 con 0x7f2e34143120 2024-03-27T21:42:05.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f2e22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f2e22ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:42:05.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7fd74ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f2e22ffd640 10 monclient: dump: 2024-03-27T21:42:05.896 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.896 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.897 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.897 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.897 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.897 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.897 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.897 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.897 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.897 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7fd74ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7fd74ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f18c8f42640 10 monclient(hunting): picked mon.noname-b con 0x7f18c40ad910 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:05.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f18c8f42640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18c4148f10 0x7f18c4145d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f18c8f42640 10 monclient(hunting): picked mon.noname-a con 0x7f18c4148f10 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:05.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f18c8f42640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f18c8f42640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c4142420 con 0x7f18c4148f10 2024-03-27T21:42:05.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f18c8f42640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f18c8f42640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c40a8ce0 con 0x7f18c40ad910 2024-03-27T21:42:05.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f18c8f42640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f18c8f42640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c40b1410 con 0x7f18c406ea80 2024-03-27T21:42:05.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f18c8f42640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f18c8f42640 10 monclient(hunting): authenticate will time out at 966.938223s 2024-03-27T21:42:05.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f18c37fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f18c406ea80 0x7f18c40af9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f051b669640 1 -- start start 2024-03-27T21:42:05.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f051b669640 1 librados: setting wanted keys 2024-03-27T21:42:05.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f051b669640 1 librados: calling monclient init 2024-03-27T21:42:05.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f051b669640 10 monclient: init 2024-03-27T21:42:05.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f051b669640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f051b669640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:42:05.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f051b669640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05000047c0 0x7f05140aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f051b669640 10 monclient(hunting): picked mon.noname-a con 0x7f05000047c0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:05.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f051b669640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0514144db0 0x7f05140abd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f051b669640 10 monclient(hunting): picked mon.noname-b con 0x7f0514144db0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:05.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f051b669640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f05140ac280 0x7f0514133f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f051b669640 10 monclient(hunting): picked mon.noname-c con 0x7f05140ac280 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f18c37fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f18c406ea80 0x7f18c40af9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34244/0 (socket says 172.21.15.60:34244) 2024-03-27T21:42:05.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f051b669640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f051b669640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05140ab6b0 con 0x7f05000047c0 2024-03-27T21:42:05.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f051b669640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f051b669640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05140ab920 con 0x7f0514144db0 2024-03-27T21:42:05.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f051b669640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f051b669640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05140a86f0 con 0x7f05140ac280 2024-03-27T21:42:05.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f051b669640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f051b669640 10 monclient(hunting): authenticate will time out at 966.939642s 2024-03-27T21:42:05.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f18c37fe640 1 -- 172.21.15.60:0/782235123 learned_addr learned my addr 172.21.15.60:0/782235123 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f5aa74e0640 1 -- start start 2024-03-27T21:42:05.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f18c37fe640 10 monclient(hunting): get_auth_request con 0x7f18c406ea80 auth_method 0 2024-03-27T21:42:05.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f18c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7fd74f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f18c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7fd74f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f18c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f5aa74e0640 1 librados: setting wanted keys 2024-03-27T21:42:05.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f5aa74e0640 1 librados: calling monclient init 2024-03-27T21:42:05.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f5aa74e0640 10 monclient: init 2024-03-27T21:42:05.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7fd74f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7fd74ffff640 10 monclient(hunting): handle_auth_done global_id 5349 payload 824 2024-03-27T21:42:05.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7fd74ffff640 1 -- 172.21.15.60:0/3097219464 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd7500b40c0 msgr2=0x7fd7501e6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7fd74ffff640 1 --2- 172.21.15.60:0/3097219464 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd7500b40c0 0x7fd7501e6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7fd74ffff640 1 -- 172.21.15.60:0/3097219464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd75013e830 msgr2=0x7fd7500b4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7fd74ffff640 1 --2- 172.21.15.60:0/3097219464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd75013e830 0x7fd7500b4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7fd74ffff640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7fd74ffff640 1 monclient: found mon.noname-a 2024-03-27T21:42:05.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7fd74ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7fd74ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:05.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7fd74ffff640 1 -- 172.21.15.60:0/3097219464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd744009400 con 0x7fd750141b10 2024-03-27T21:42:05.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f2e30d99640 4 set_mon_vals no callback set 2024-03-27T21:42:05.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f2e3359e640 5 monclient: authenticate success, global_id 5254 2024-03-27T21:42:05.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f2e3359e640 10 monclient: _renew_subs 2024-03-27T21:42:05.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f2e3359e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f2e3359e640 1 -- 172.21.15.60:0/1344941168 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e34144060 con 0x7f2e34143120 2024-03-27T21:42:05.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f2e3359e640 10 client.5254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f2e3359e640 10 monclient: _renew_subs 2024-03-27T21:42:05.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f2e3359e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f2e3359e640 1 -- 172.21.15.60:0/1344941168 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e341423d0 con 0x7f2e34143120 2024-03-27T21:42:05.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f2e3359e640 1 librados: init done 2024-03-27T21:42:05.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f0519c06640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f05140ac280 0x7f0514133f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f0519c06640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f05140ac280 0x7f0514133f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34260/0 (socket says 172.21.15.60:34260) 2024-03-27T21:42:05.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7fd737fff640 10 client.?.objecter ms_handle_connect 0x7fd750141b10 2024-03-27T21:42:05.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7fd737fff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:05.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7fd74ffff640 1 --2- 172.21.15.60:0/3097219464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd750141b10 0x7fd7500b7120 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd7500616a0 tx=0x7fd7440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f0518c04640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0514144db0 0x7f05140abd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5aa74e0640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5aa74e0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:42:05.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5aa74e0640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5a98007b60 0x7f5a980c3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5aa74e0640 10 monclient(hunting): picked mon.noname-c con 0x7f5a98007b60 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5aa74e0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a980c76d0 0x7f5a98037640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5aa74e0640 10 monclient(hunting): picked mon.noname-b con 0x7f5a980c76d0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:05.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5aa74e0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a980c82c0 0x7f5a9802c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5aa74e0640 10 monclient(hunting): picked mon.noname-a con 0x7f5a980c82c0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:05.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5aa74e0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5aa74e0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9802b900 con 0x7f5a980c82c0 2024-03-27T21:42:05.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5aa74e0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5aa74e0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a98039600 con 0x7f5a980c76d0 2024-03-27T21:42:05.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5aa74e0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5aa74e0640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9802c640 con 0x7f5a98007b60 2024-03-27T21:42:05.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5aa74e0640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5aa74e0640 10 monclient(hunting): authenticate will time out at 966.940505s 2024-03-27T21:42:05.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f0518c04640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0514144db0 0x7f05140abd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:58818/0 (socket says 172.21.15.60:58818) 2024-03-27T21:42:05.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f0518c04640 1 -- 172.21.15.60:0/906218950 learned_addr learned my addr 172.21.15.60:0/906218950 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f5f137fe640 10 monclient(hunting): get_auth_request con 0x7f5f0c038fe0 auth_method 0 2024-03-27T21:42:05.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f5f137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f5f137fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.717+0000 7f5f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5f137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5aa64de640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5a98007b60 0x7f5a980c3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5f137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5f137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f0518c04640 10 monclient(hunting): get_auth_request con 0x7f0514144db0 auth_method 0 2024-03-27T21:42:05.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5f13fff640 10 monclient(hunting): get_auth_request con 0x7f5f0c035640 auth_method 0 2024-03-27T21:42:05.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5f13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5f13fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f5f13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f8919be2640 10 monclient: build_initial_monmap 2024-03-27T21:42:05.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f8919be2640 10 monclient: monmap: 2024-03-27T21:42:05.918 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:05.919 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:05.919 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:05.919 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:05.919 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:05.919 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:05.919 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:05.919 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:05.919 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:05.919 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f8919be2640 1 librados: starting msgr at 2024-03-27T21:42:05.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f8919be2640 1 librados: starting objecter 2024-03-27T21:42:05.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7fd737fff640 1 -- 172.21.15.60:0/3097219464 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd744032540 con 0x7fd750141b10 2024-03-27T21:42:05.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7fd737fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7fd737fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:05.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f8919be2640 1 Processor -- start 2024-03-27T21:42:05.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f8919be2640 1 -- start start 2024-03-27T21:42:05.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f8919be2640 1 librados: setting wanted keys 2024-03-27T21:42:05.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f8919be2640 1 librados: calling monclient init 2024-03-27T21:42:05.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.718+0000 7f8919be2640 10 monclient: init 2024-03-27T21:42:05.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7fd737fff640 10 monclient: dump: 2024-03-27T21:42:05.922 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.922 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.922 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.922 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.922 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.922 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.922 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.923 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.923 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.923 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7fd737fff640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7fd737fff640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7fd737fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f8919be2640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:05.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f8919be2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:42:05.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f8919be2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89140b3580 0x7f89140aafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f8919be2640 10 monclient(hunting): picked mon.noname-b con 0x7f89140b3580 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:05.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f8919be2640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f89140b3f60 0x7f89140ab520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f8919be2640 10 monclient(hunting): picked mon.noname-c con 0x7f89140b3f60 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:05.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f8919be2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f891413ecf0 0x7f89141436c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8919be2640 10 monclient(hunting): picked mon.noname-a con 0x7f891413ecf0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:05.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8919be2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8919be2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89141455f0 con 0x7f891413ecf0 2024-03-27T21:42:05.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8919be2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8919be2640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8914142910 con 0x7f89140b3580 2024-03-27T21:42:05.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8919be2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:05.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8919be2640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891406ee10 con 0x7f89140b3f60 2024-03-27T21:42:05.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8919be2640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:05.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8919be2640 10 monclient(hunting): authenticate will time out at 966.940965s 2024-03-27T21:42:05.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8913fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f89140b3f60 0x7f89140ab520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8913fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f89140b3f60 0x7f89140ab520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34280/0 (socket says 172.21.15.60:34280) 2024-03-27T21:42:05.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8913fff640 1 -- 172.21.15.60:0/2195797653 learned_addr learned my addr 172.21.15.60:0/2195797653 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8913fff640 10 monclient(hunting): get_auth_request con 0x7f89140b3f60 auth_method 0 2024-03-27T21:42:05.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8913fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8913fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8913fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8918be0640 1 --2- 172.21.15.60:0/2195797653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89140b3580 0x7f89140aafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8918be0640 10 monclient(hunting): get_auth_request con 0x7f89140b3580 auth_method 0 2024-03-27T21:42:05.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8918be0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8918be0640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8918be0640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f0518c04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f0518c04640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f0518c04640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f89193e1640 1 --2- 172.21.15.60:0/2195797653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f891413ecf0 0x7f89141436c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8913fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8913fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8913fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f5aa64de640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5a98007b60 0x7f5a980c3790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34270/0 (socket says 172.21.15.60:34270) 2024-03-27T21:42:05.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f5aa64de640 1 -- 172.21.15.60:0/842667156 learned_addr learned my addr 172.21.15.60:0/842667156 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:05.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8918be0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8918be0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8918be0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c2ffd640 1 --2- 172.21.15.60:0/782235123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18c40ad910 0x7f18c4141ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c3fff640 1 --2- 172.21.15.60:0/782235123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18c4148f10 0x7f18c4145d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f5aa64de640 10 monclient(hunting): get_auth_request con 0x7f5a98007b60 auth_method 0 2024-03-27T21:42:05.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f5aa64de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f5aa64de640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f5aa64de640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f5aa64de640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f5aa64de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f5aa64de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f5aa64de640 10 monclient(hunting): handle_auth_done global_id 5051 payload 824 2024-03-27T21:42:05.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f5aa64de640 1 -- 172.21.15.60:0/842667156 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a980c76d0 msgr2=0x7f5a98037640 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-27T21:42:05.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f5aa64de640 1 --2- 172.21.15.60:0/842667156 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a980c76d0 0x7f5a98037640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c3fff640 10 monclient(hunting): get_auth_request con 0x7f18c4148f10 auth_method 0 2024-03-27T21:42:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c2ffd640 10 monclient(hunting): get_auth_request con 0x7f18c40ad910 auth_method 0 2024-03-27T21:42:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f5aa64de640 1 -- 172.21.15.60:0/842667156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a980c82c0 msgr2=0x7f5a9802c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c2ffd640 10 monclient(hunting): handle_auth_done global_id 5263 payload 824 2024-03-27T21:42:05.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c2ffd640 1 -- 172.21.15.60:0/782235123 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f18c406ea80 msgr2=0x7f18c40af9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c2ffd640 1 --2- 172.21.15.60:0/782235123 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f18c406ea80 0x7f18c40af9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c2ffd640 1 -- 172.21.15.60:0/782235123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18c4148f10 msgr2=0x7f18c4145d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c2ffd640 1 --2- 172.21.15.60:0/782235123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18c4148f10 0x7f18c4145d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c2ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c2ffd640 1 monclient: found mon.noname-b 2024-03-27T21:42:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8913fff640 10 monclient(hunting): handle_auth_done global_id 5054 payload 824 2024-03-27T21:42:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8913fff640 1 -- 172.21.15.60:0/2195797653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89140b3580 msgr2=0x7f89140aafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8913fff640 1 --2- 172.21.15.60:0/2195797653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89140b3580 0x7f89140aafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8913fff640 1 -- 172.21.15.60:0/2195797653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f891413ecf0 msgr2=0x7f89141436c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8913fff640 1 --2- 172.21.15.60:0/2195797653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f891413ecf0 0x7f89141436c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8913fff640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8913fff640 1 monclient: found mon.noname-c 2024-03-27T21:42:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8913fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8913fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f8913fff640 1 -- 172.21.15.60:0/2195797653 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8904009400 con 0x7f89140b3f60 2024-03-27T21:42:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f3f167fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f3f167fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f18c2ffd640 1 -- 172.21.15.60:0/782235123 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b40094d0 con 0x7f18c40ad910 2024-03-27T21:42:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f3f167fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7fd755cb9640 5 monclient: authenticate success, global_id 5349 2024-03-27T21:42:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7fd755cb9640 10 monclient: _renew_subs 2024-03-27T21:42:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7fd755cb9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7fd755cb9640 1 -- 172.21.15.60:0/3097219464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7500b6590 con 0x7fd750141b10 2024-03-27T21:42:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7fd755cb9640 10 client.5349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7fd755cb9640 10 monclient: _renew_subs 2024-03-27T21:42:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7fd755cb9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:05.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7fd755cb9640 1 -- 172.21.15.60:0/3097219464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7500b7840 con 0x7fd750141b10 2024-03-27T21:42:05.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7fd755cb9640 1 librados: init done 2024-03-27T21:42:05.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f18c3fff640 1 --2- 172.21.15.60:0/782235123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18c4148f10 0x7f18c4145d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:05.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f3f167fc640 1 -- 172.21.15.60:0/2588636431 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f30087240 con 0x7f3f30083230 2024-03-27T21:42:05.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f0518c04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:05.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f0518c04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:05.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.719+0000 7f0518c04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:05.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f0518c04640 10 monclient(hunting): handle_auth_done global_id 5257 payload 824 2024-03-27T21:42:05.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f0518c04640 1 -- 172.21.15.60:0/906218950 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f05140ac280 msgr2=0x7f0514133f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f0519405640 1 --2- 172.21.15.60:0/906218950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05000047c0 0x7f05140aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0518c04640 1 --2- 172.21.15.60:0/906218950 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f05140ac280 0x7f0514133f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0518c04640 1 -- 172.21.15.60:0/906218950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05000047c0 msgr2=0x7f05140aa1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0518c04640 1 --2- 172.21.15.60:0/906218950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05000047c0 0x7f05140aa1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0518c04640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0518c04640 1 monclient: found mon.noname-b 2024-03-27T21:42:05.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0518c04640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0518c04640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0518c04640 1 -- 172.21.15.60:0/906218950 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05140a92c0 con 0x7f0514144db0 2024-03-27T21:42:05.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0519405640 10 monclient: get_auth_request con 0x7f05000047c0 auth_method 0 2024-03-27T21:42:05.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0519405640 1 --2- 172.21.15.60:0/906218950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05000047c0 0x7f05140aa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:42:05.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0519c06640 10 monclient: get_auth_request con 0x7f05140ac280 auth_method 0 2024-03-27T21:42:05.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0519c06640 1 --2- 172.21.15.60:0/906218950 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f05140ac280 0x7f0514133f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:42:05.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f8913fff640 1 --2- 172.21.15.60:0/2195797653 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f89140b3f60 0x7f89140ab520 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f890800c7d0 tx=0x7f890800cc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f89193e1640 10 monclient: get_auth_request con 0x7f891413ecf0 auth_method 0 2024-03-27T21:42:05.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f89193e1640 1 --2- 172.21.15.60:0/2195797653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f891413ecf0 0x7f89141436c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:42:05.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f89117fa640 10 client.?.objecter ms_handle_connect 0x7f89140b3f60 2024-03-27T21:42:05.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f89117fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:05.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f18abfff640 10 client.?.objecter ms_handle_connect 0x7f18c40ad910 2024-03-27T21:42:05.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f18abfff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:05.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f18c2ffd640 1 --2- 172.21.15.60:0/782235123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18c40ad910 0x7f18c4141ee0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f18c40677c0 tx=0x7f18b40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.720+0000 7f5aa6cdf640 1 --2- 172.21.15.60:0/842667156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a980c82c0 0x7f5a9802c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0509ffb640 10 client.?.objecter ms_handle_connect 0x7f0514144db0 2024-03-27T21:42:05.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0509ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:05.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0518c04640 1 --2- 172.21.15.60:0/906218950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0514144db0 0x7f05140abd40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f050c00a880 tx=0x7f050c001cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f2e20ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f2e20ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f2e20ff9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f2e20ff9640 1 -- 172.21.15.60:0/1344941168 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e34067280 con 0x7f2e34143120 2024-03-27T21:42:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f3f2cff9640 1 -- 172.21.15.60:0/2588636431 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3f28007070 con 0x7f3f30083230 2024-03-27T21:42:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f2e22ffd640 1 -- 172.21.15.60:0/1344941168 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2e2804e450 con 0x7f2e34143120 2024-03-27T21:42:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7fd737fff640 1 -- 172.21.15.60:0/3097219464 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7440326a0 con 0x7fd750141b10 2024-03-27T21:42:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7fd737fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7fd737fff640 1 -- 172.21.15.60:0/3097219464 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd744032540 con 0x7fd750141b10 2024-03-27T21:42:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7fd737fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7fd737fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7fd737fff640 10 monclient: dump: 2024-03-27T21:42:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.949 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7fd74dffb640 4 set_mon_vals no callback set 2024-03-27T21:42:05.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7fd755cb9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f18abfff640 1 -- 172.21.15.60:0/782235123 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b403adf0 con 0x7f18c40ad910 2024-03-27T21:42:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f18abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f18abfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f18abfff640 10 monclient: dump: 2024-03-27T21:42:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.950 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.951 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f18abfff640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f18abfff640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f18abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f5aa64de640 1 --2- 172.21.15.60:0/842667156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a980c82c0 0x7f5a9802c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f5aa64de640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f5aa64de640 1 monclient: found mon.noname-c 2024-03-27T21:42:05.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f5aa64de640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f5aa64de640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:05.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f5aa64de640 1 -- 172.21.15.60:0/842667156 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a90011d70 con 0x7f5a98007b60 2024-03-27T21:42:05.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f5aa64de640 1 --2- 172.21.15.60:0/842667156 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5a98007b60 0x7f5a980c3790 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c00c400 tx=0x7f5a9c00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f5aa6cdf640 10 monclient: get_auth_request con 0x7f5a980c82c0 auth_method 0 2024-03-27T21:42:05.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f5aa6cdf640 1 --2- 172.21.15.60:0/842667156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a980c82c0 0x7f5a9802c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:42:05.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0509ffb640 1 -- 172.21.15.60:0/906218950 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c0019f0 con 0x7f0514144db0 2024-03-27T21:42:05.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0509ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0509ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:05.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0509ffb640 10 monclient: dump: 2024-03-27T21:42:05.954 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.954 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.954 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.954 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.954 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.954 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.954 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.955 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.955 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.955 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0509ffb640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0509ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0509ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f89117fa640 1 -- 172.21.15.60:0/2195797653 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908005480 con 0x7f89140b3f60 2024-03-27T21:42:05.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f89117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f89117fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:05.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f89117fa640 10 monclient: dump: 2024-03-27T21:42:05.956 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.956 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.956 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.956 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.956 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.956 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.957 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.957 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.957 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.957 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f89117fa640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f89117fa640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f89117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f89117fa640 1 -- 172.21.15.60:0/2195797653 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f890800a640 con 0x7f89140b3f60 2024-03-27T21:42:05.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f89117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f89117fa640 1 -- 172.21.15.60:0/2195797653 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890800a9b0 con 0x7f89140b3f60 2024-03-27T21:42:05.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f89117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f89117fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:05.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f89117fa640 10 monclient: dump: 2024-03-27T21:42:05.958 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.958 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.958 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.958 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.958 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.959 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.959 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.959 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.959 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.959 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f8919be2640 5 monclient: authenticate success, global_id 5054 2024-03-27T21:42:05.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f8919be2640 10 monclient: _renew_subs 2024-03-27T21:42:05.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f8919be2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:05.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f8919be2640 1 -- 172.21.15.60:0/2195797653 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f891406f090 con 0x7f89140b3f60 2024-03-27T21:42:05.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f8919be2640 10 client.5054.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f8919be2640 10 monclient: _renew_subs 2024-03-27T21:42:05.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f8919be2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:05.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f8919be2640 1 -- 172.21.15.60:0/2195797653 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8914143de0 con 0x7f89140b3f60 2024-03-27T21:42:05.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f8919be2640 1 librados: init done 2024-03-27T21:42:05.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f2e22ffd640 1 --2- 172.21.15.60:0/1344941168 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2e1406c660 0x7f2e1406eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0509ffb640 1 -- 172.21.15.60:0/906218950 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050c004d70 con 0x7f0514144db0 2024-03-27T21:42:05.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f0509ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f050b7fe640 4 set_mon_vals no callback set 2024-03-27T21:42:05.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f051b669640 5 monclient: authenticate success, global_id 5257 2024-03-27T21:42:05.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f051b669640 10 monclient: _renew_subs 2024-03-27T21:42:05.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f051b669640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f051b669640 1 -- 172.21.15.60:0/906218950 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f051412f590 con 0x7f0514144db0 2024-03-27T21:42:05.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f18abfff640 1 -- 172.21.15.60:0/782235123 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b40443f0 con 0x7f18c40ad910 2024-03-27T21:42:05.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f18abfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f18c1ffb640 4 set_mon_vals no callback set 2024-03-27T21:42:05.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f18abfff640 1 -- 172.21.15.60:0/782235123 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b403adf0 con 0x7f18c40ad910 2024-03-27T21:42:05.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f18abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f0509ffb640 1 -- 172.21.15.60:0/906218950 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c0019f0 con 0x7f0514144db0 2024-03-27T21:42:05.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f0509ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f0509ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:42:05.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f0509ffb640 10 monclient: dump: 2024-03-27T21:42:05.963 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.963 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.964 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.964 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.964 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.964 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.964 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.964 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.964 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.964 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f3f2cff9640 1 --2- 172.21.15.60:0/2588636431 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3f1c06c660 0x7f3f1c06eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f18abfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:42:05.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f18abfff640 10 monclient: dump: 2024-03-27T21:42:05.966 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.966 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.966 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.966 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.966 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.966 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.966 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.966 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.967 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.967 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f18c8f42640 5 monclient: authenticate success, global_id 5263 2024-03-27T21:42:05.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f18c8f42640 10 monclient: _renew_subs 2024-03-27T21:42:05.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f18c8f42640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7fd755cb9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f18c8f42640 1 -- 172.21.15.60:0/782235123 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c4134f90 con 0x7f18c40ad910 2024-03-27T21:42:05.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7fd755cb9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:05.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f3f2cff9640 1 -- 172.21.15.60:0/2588636431 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f3f280890f0 con 0x7f3f30083230 2024-03-27T21:42:05.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f3f2cff9640 10 client.5036.objecter ms_dispatch 0x7f3f30000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:05.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f3f2cff9640 3 client.5036.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:05.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f3f2cff9640 3 client.5036.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:05.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f2e22ffd640 1 -- 172.21.15.60:0/1344941168 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f2e280ab4f0 con 0x7f2e34143120 2024-03-27T21:42:05.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f2e3259c640 1 --2- 172.21.15.60:0/1344941168 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2e1406c660 0x7f2e1406eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f2e3259c640 10 monclient: get_auth_request con 0x7f2e1406c660 auth_method 0 2024-03-27T21:42:05.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f2e22ffd640 10 client.5254.objecter ms_dispatch 0x7f2e34000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:05.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f2e22ffd640 3 client.5254.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:05.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f2e22ffd640 3 client.5254.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:05.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f18c8f42640 10 client.5263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f18c8f42640 10 monclient: _renew_subs 2024-03-27T21:42:05.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f18c8f42640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f18c8f42640 1 -- 172.21.15.60:0/782235123 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c41317b0 con 0x7f18c40ad910 2024-03-27T21:42:05.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f8912ffd640 4 set_mon_vals no callback set 2024-03-27T21:42:05.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f3f2cff9640 20 client.5036.objecter dump_active .. 0 homeless 2024-03-27T21:42:05.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f18c8f42640 1 librados: init done 2024-03-27T21:42:05.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f5a96ffd640 10 client.?.objecter ms_handle_connect 0x7f5a98007b60 2024-03-27T21:42:05.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f5a96ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:05.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f2e22ffd640 20 client.5254.objecter dump_active .. 0 homeless 2024-03-27T21:42:05.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f3f2ffff640 1 --2- 172.21.15.60:0/2588636431 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3f1c06c660 0x7f3f1c06eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f3f2ffff640 10 monclient: get_auth_request con 0x7f3f1c06c660 auth_method 0 2024-03-27T21:42:05.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f2e3259c640 1 --2- 172.21.15.60:0/1344941168 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2e1406c660 0x7f2e1406eb20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c00c530 tx=0x7f2e2c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f2e22ffd640 10 client.5254.objecter ms_handle_connect 0x7f2e1406c660 2024-03-27T21:42:05.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7fd755cb9640 1 -- 172.21.15.60:0/3097219464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd730005040 con 0x7fd750141b10 2024-03-27T21:42:05.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7fd737fff640 1 -- 172.21.15.60:0/3097219464 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd744039070 con 0x7fd750141b10 2024-03-27T21:42:05.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7fd737fff640 1 --2- 172.21.15.60:0/3097219464 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd73c06c5f0 0x7fd73c06eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7fd737fff640 1 -- 172.21.15.60:0/3097219464 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fd744079ad0 con 0x7fd750141b10 2024-03-27T21:42:05.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7fd737fff640 10 client.5349.objecter ms_dispatch 0x7fd750000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:05.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7fd737fff640 3 client.5349.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:05.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7fd737fff640 3 client.5349.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:05.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7fd737fff640 20 client.5349.objecter dump_active .. 0 homeless 2024-03-27T21:42:05.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7fd74f7fe640 1 --2- 172.21.15.60:0/3097219464 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd73c06c5f0 0x7fd73c06eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7fd74f7fe640 10 monclient: get_auth_request con 0x7fd73c06c5f0 auth_method 0 2024-03-27T21:42:05.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f5a96ffd640 1 -- 172.21.15.60:0/842667156 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c0052d0 con 0x7f5a98007b60 2024-03-27T21:42:05.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f5a96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f5a96ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:05.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f5a96ffd640 10 monclient: dump: 2024-03-27T21:42:05.976 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.976 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.976 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.976 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.976 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.976 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.976 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.976 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.976 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.976 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f5a96ffd640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f5a96ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f5a96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f5f137fe640 10 monclient(hunting): handle_auth_done global_id 5042 payload 824 2024-03-27T21:42:05.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f5f137fe640 1 -- 172.21.15.60:0/2535368581 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f0c035640 msgr2=0x7f5f0c02f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f5f137fe640 1 --2- 172.21.15.60:0/2535368581 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f0c035640 0x7f5f0c02f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f5f137fe640 1 -- 172.21.15.60:0/2535368581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f0c0cc600 msgr2=0x7f5f0c16a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:05.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f5f137fe640 1 --2- 172.21.15.60:0/2535368581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f0c0cc600 0x7f5f0c16a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:05.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f5f137fe640 10 monclient: _finish_hunting 0 2024-03-27T21:42:05.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f5f137fe640 1 monclient: found mon.noname-c 2024-03-27T21:42:05.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f5f137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:05.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f5f137fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:05.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.721+0000 7f5f137fe640 1 -- 172.21.15.60:0/2535368581 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f080094d0 con 0x7f5f0c038fe0 2024-03-27T21:42:05.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f5f18ece640 10 monclient: get_auth_request con 0x7f5f0c0cc600 auth_method 0 2024-03-27T21:42:05.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.722+0000 7f5f18ece640 1 --2- 172.21.15.60:0/2535368581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f0c0cc600 0x7f5f0c16a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:42:05.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f5f137fe640 1 --2- 172.21.15.60:0/2535368581 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5f0c038fe0 0x7f5f0c033a40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5f1402c880 tx=0x7f5f14067900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f5f10ff9640 10 client.?.objecter ms_handle_connect 0x7f5f0c038fe0 2024-03-27T21:42:05.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f5f10ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:05.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f051b669640 10 client.5257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f051b669640 10 monclient: _renew_subs 2024-03-27T21:42:05.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f051b669640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f051b669640 1 -- 172.21.15.60:0/906218950 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05141453f0 con 0x7f0514144db0 2024-03-27T21:42:05.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f051b669640 1 librados: init done 2024-03-27T21:42:05.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.723+0000 7f8919be2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f8919be2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f8919be2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:05.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f8919be2640 1 -- 172.21.15.60:0/2195797653 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f4005040 con 0x7f89140b3f60 2024-03-27T21:42:05.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f18c8f42640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f18c8f42640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f18c8f42640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f051b669640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f051b669640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f18c8f42640 1 -- 172.21.15.60:0/782235123 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c40b1270 con 0x7f18c40ad910 2024-03-27T21:42:05.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f051b669640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:05.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f051b669640 1 -- 172.21.15.60:0/906218950 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05140a9450 con 0x7f0514144db0 2024-03-27T21:42:05.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.725+0000 7f3f2ffff640 1 --2- 172.21.15.60:0/2588636431 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3f1c06c660 0x7f3f1c06eb20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009e90 tx=0x7f3f24009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.725+0000 7fd737fff640 10 client.5349.objecter ms_handle_connect 0x7fd73c06c5f0 2024-03-27T21:42:05.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.725+0000 7fd74f7fe640 1 --2- 172.21.15.60:0/3097219464 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd73c06c5f0 0x7fd73c06eab0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd740009c50 tx=0x7fd740009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f5aa74e0640 5 monclient: authenticate success, global_id 5051 2024-03-27T21:42:05.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f5aa74e0640 10 monclient: _renew_subs 2024-03-27T21:42:05.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f5aa74e0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:05.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f5aa74e0640 1 -- 172.21.15.60:0/842667156 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a980390e0 con 0x7f5a98007b60 2024-03-27T21:42:05.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f5aa74e0640 10 client.5051.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:05.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f5aa74e0640 10 monclient: _renew_subs 2024-03-27T21:42:05.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f5aa74e0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:05.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f5aa74e0640 1 -- 172.21.15.60:0/842667156 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a9802a090 con 0x7f5a98007b60 2024-03-27T21:42:05.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.724+0000 7f5aa74e0640 1 librados: init done 2024-03-27T21:42:05.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.725+0000 7f5a96ffd640 1 -- 172.21.15.60:0/842667156 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9c005430 con 0x7f5a98007b60 2024-03-27T21:42:05.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.725+0000 7f5a96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:05.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.725+0000 7f5aa4cdb640 4 set_mon_vals no callback set 2024-03-27T21:42:05.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.725+0000 7f3f2cff9640 10 client.5036.objecter ms_handle_connect 0x7f3f1c06c660 2024-03-27T21:42:05.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.725+0000 7f5aa74e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.725+0000 7f89117fa640 1 -- 172.21.15.60:0/2195797653 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f890800ab10 con 0x7f89140b3f60 2024-03-27T21:42:05.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.725+0000 7f5a96ffd640 1 -- 172.21.15.60:0/842667156 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c003810 con 0x7f5a98007b60 2024-03-27T21:42:05.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.726+0000 7f89117fa640 1 --2- 172.21.15.60:0/2195797653 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f890006c610 0x7f890006ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.726+0000 7f5aa74e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.726+0000 7f5aa74e0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:05.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.726+0000 7f5aa74e0640 1 -- 172.21.15.60:0/842667156 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a88005040 con 0x7f5a98007b60 2024-03-27T21:42:05.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.726+0000 7f5a96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.726+0000 7f5a96ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:05.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.726+0000 7f5a96ffd640 10 monclient: dump: 2024-03-27T21:42:05.988 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.988 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.988 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.988 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.988 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.988 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.989 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.989 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.989 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.989 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.726+0000 7f8918be0640 1 --2- 172.21.15.60:0/2195797653 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f890006c610 0x7f890006ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.726+0000 7f89117fa640 1 -- 172.21.15.60:0/2195797653 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f8908089160 con 0x7f89140b3f60 2024-03-27T21:42:05.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.726+0000 7f89117fa640 10 client.5054.objecter ms_dispatch 0x7f8914000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:05.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.726+0000 7f89117fa640 3 client.5054.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:05.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.726+0000 7f89117fa640 3 client.5054.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:05.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.726+0000 7f8918be0640 10 monclient: get_auth_request con 0x7f890006c610 auth_method 0 2024-03-27T21:42:05.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.726+0000 7f89117fa640 20 client.5054.objecter dump_active .. 0 homeless 2024-03-27T21:42:05.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.726+0000 7f8918be0640 1 --2- 172.21.15.60:0/2195797653 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f890006c610 0x7f890006ead0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8904034000 tx=0x7f8904008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.726+0000 7f89117fa640 10 client.5054.objecter ms_handle_connect 0x7f890006c610 2024-03-27T21:42:05.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.727+0000 7f2e22ffd640 1 -- 172.21.15.60:0/1344941168 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2e28076880 con 0x7f2e34143120 2024-03-27T21:42:05.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.727+0000 7f2e22ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.727+0000 7f2e22ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:05.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.728+0000 7f18abfff640 1 -- 172.21.15.60:0/782235123 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f18b40433a0 con 0x7f18c40ad910 2024-03-27T21:42:05.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.729+0000 7f18abfff640 1 --2- 172.21.15.60:0/782235123 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f18b006c680 0x7f18b006eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.729+0000 7f18abfff640 1 -- 172.21.15.60:0/782235123 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f18b402e070 con 0x7f18c40ad910 2024-03-27T21:42:05.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.729+0000 7f18abfff640 10 client.5263.objecter ms_dispatch 0x7f18c4000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:05.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.729+0000 7f18abfff640 3 client.5263.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:05.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.729+0000 7f18abfff640 3 client.5263.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:05.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.729+0000 7f18abfff640 20 client.5263.objecter dump_active .. 0 homeless 2024-03-27T21:42:05.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.729+0000 7f18c37fe640 1 --2- 172.21.15.60:0/782235123 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f18b006c680 0x7f18b006eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.729+0000 7f0509ffb640 1 -- 172.21.15.60:0/906218950 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f050c031030 con 0x7f0514144db0 2024-03-27T21:42:05.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.729+0000 7f0509ffb640 1 --2- 172.21.15.60:0/906218950 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f04f806c650 0x7f04f806eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:05.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.729+0000 7f0509ffb640 1 -- 172.21.15.60:0/906218950 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f050c0ab4f0 con 0x7f0514144db0 2024-03-27T21:42:05.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.729+0000 7f0509ffb640 10 client.5257.objecter ms_dispatch 0x7f0514000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:05.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.729+0000 7f0509ffb640 3 client.5257.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:05.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.729+0000 7f0509ffb640 3 client.5257.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:05.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.729+0000 7f18c37fe640 10 monclient: get_auth_request con 0x7f18b006c680 auth_method 0 2024-03-27T21:42:05.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.729+0000 7f0519405640 1 --2- 172.21.15.60:0/906218950 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f04f806c650 0x7f04f806eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:05.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.729+0000 7f0519405640 10 monclient: get_auth_request con 0x7f04f806c650 auth_method 0 2024-03-27T21:42:05.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.729+0000 7f0509ffb640 20 client.5257.objecter dump_active .. 0 homeless 2024-03-27T21:42:05.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.730+0000 7f0509ffb640 10 client.5257.objecter ms_handle_connect 0x7f04f806c650 2024-03-27T21:42:05.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.730+0000 7f0519405640 1 --2- 172.21.15.60:0/906218950 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f04f806c650 0x7f04f806eb10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0504006fd0 tx=0x7f0504008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.730+0000 7f18c37fe640 1 --2- 172.21.15.60:0/782235123 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f18b006c680 0x7f18b006eb40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f18bc00e2a0 tx=0x7f18bc00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.730+0000 7f18abfff640 10 client.5263.objecter ms_handle_connect 0x7f18b006c680 2024-03-27T21:42:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.730+0000 7f3f2cff9640 1 -- 172.21.15.60:0/2588636431 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3f28054530 con 0x7f3f30083230 2024-03-27T21:42:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.730+0000 7f3f2cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.730+0000 7f3f2cff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.730+0000 7f5f10ff9640 1 -- 172.21.15.60:0/2535368581 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f140690b0 con 0x7f5f0c038fe0 2024-03-27T21:42:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.730+0000 7f5f10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.730+0000 7f5f10ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.730+0000 7f5f10ff9640 10 monclient: dump: 2024-03-27T21:42:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:05.998 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:05.998 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.998 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:05.998 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:05.998 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:05.998 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:05.998 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:05.998 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:05.998 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:05.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.730+0000 7f5f10ff9640 10 monclient: _finish_auth 0 2024-03-27T21:42:05.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.730+0000 7f5f10ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:42:05.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.730+0000 7f5f10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:05.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.731+0000 7f5f1a931640 5 monclient: authenticate success, global_id 5042 2024-03-27T21:42:05.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.731+0000 7f5f1a931640 10 monclient: _renew_subs 2024-03-27T21:42:05.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.731+0000 7f5f1a931640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:06.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.731+0000 7fd737fff640 1 -- 172.21.15.60:0/3097219464 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd744075ea0 con 0x7fd750141b10 2024-03-27T21:42:06.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.731+0000 7fd737fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:06.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.731+0000 7fd737fff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:06.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.731+0000 7f5f1a931640 1 -- 172.21.15.60:0/2535368581 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f0c034200 con 0x7f5f0c038fe0 2024-03-27T21:42:06.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.731+0000 7f5f1a931640 10 client.5042.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:06.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.731+0000 7f5f1a931640 10 monclient: _renew_subs 2024-03-27T21:42:06.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.731+0000 7f5f1a931640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:06.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.731+0000 7f5f1a931640 1 -- 172.21.15.60:0/2535368581 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f0c0333c0 con 0x7f5f0c038fe0 2024-03-27T21:42:06.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.731+0000 7f5f1a931640 1 librados: init done 2024-03-27T21:42:06.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.732+0000 7f18abfff640 1 -- 172.21.15.60:0/782235123 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f18b4076d80 con 0x7f18c40ad910 2024-03-27T21:42:06.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.732+0000 7f18abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:06.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.732+0000 7f18abfff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:06.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.732+0000 7f5a96ffd640 1 -- 172.21.15.60:0/842667156 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5a9c003970 con 0x7f5a98007b60 2024-03-27T21:42:06.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.732+0000 7f5f1a931640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:06.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.732+0000 7f5f1a931640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:06.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.732+0000 7f5f1a931640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:06.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.732+0000 7f5f1a931640 1 -- 172.21.15.60:0/2535368581 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f0c0085f0 con 0x7f5f0c038fe0 2024-03-27T21:42:06.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.733+0000 7f5a96ffd640 1 --2- 172.21.15.60:0/842667156 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5a8406c590 0x7f5a8406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:06.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.733+0000 7f5aa5cdd640 1 --2- 172.21.15.60:0/842667156 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5a8406c590 0x7f5a8406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:06.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.733+0000 7f5aa5cdd640 10 monclient: get_auth_request con 0x7f5a8406c590 auth_method 0 2024-03-27T21:42:06.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.733+0000 7f5a96ffd640 10 client.5051.objecter ms_handle_connect 0x7f5a8406c590 2024-03-27T21:42:06.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.733+0000 7f5aa5cdd640 1 --2- 172.21.15.60:0/842667156 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5a8406c590 0x7f5a8406ea50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5aa004f810 tx=0x7f5aa00726a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:06.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.736+0000 7f0509ffb640 1 -- 172.21.15.60:0/906218950 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f050c076930 con 0x7f0514144db0 2024-03-27T21:42:06.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.736+0000 7f0509ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:06.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.736+0000 7f0509ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:06.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.736+0000 7f5a96ffd640 1 -- 172.21.15.60:0/842667156 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f5a9c057fc0 con 0x7f5a98007b60 2024-03-27T21:42:06.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.737+0000 7f5a96ffd640 10 client.5051.objecter ms_dispatch 0x7f5aa0000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:06.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.737+0000 7f89117fa640 1 -- 172.21.15.60:0/2195797653 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8908054610 con 0x7f89140b3f60 2024-03-27T21:42:06.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.737+0000 7f89117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:06.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.737+0000 7f89117fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:06.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.736+0000 7f5f10ff9640 1 -- 172.21.15.60:0/2535368581 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f14067280 con 0x7f5f0c038fe0 2024-03-27T21:42:06.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.736+0000 7f5f10ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:06.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.736+0000 7f5f10ff9640 1 -- 172.21.15.60:0/2535368581 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f14069430 con 0x7f5f0c038fe0 2024-03-27T21:42:06.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.736+0000 7f5f10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:06.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.736+0000 7f5f10ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:06.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.736+0000 7f5f10ff9640 10 monclient: dump: 2024-03-27T21:42:06.008 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:06.008 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:06.008 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:06.008 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:06.008 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:06.008 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:06.009 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:06.009 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:06.009 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:06.009 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:06.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.736+0000 7f5f127fc640 4 set_mon_vals no callback set 2024-03-27T21:42:06.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.737+0000 7f5a96ffd640 3 client.5051.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:06.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.737+0000 7f5a96ffd640 3 client.5051.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:06.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.737+0000 7f5a96ffd640 20 client.5051.objecter dump_active .. 0 homeless 2024-03-27T21:42:06.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.739+0000 7f5f10ff9640 1 -- 172.21.15.60:0/2535368581 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5f140923f0 con 0x7f5f0c038fe0 2024-03-27T21:42:06.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.739+0000 7f5f10ff9640 1 --2- 172.21.15.60:0/2535368581 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5ef806c590 0x7f5ef806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:06.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.740+0000 7f5f13fff640 1 --2- 172.21.15.60:0/2535368581 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5ef806c590 0x7f5ef806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:06.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.740+0000 7f5f13fff640 10 monclient: get_auth_request con 0x7f5ef806c590 auth_method 0 2024-03-27T21:42:06.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.740+0000 7f5f13fff640 1 --2- 172.21.15.60:0/2535368581 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5ef806c590 0x7f5ef806ea50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5f08036000 tx=0x7f5f0803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:06.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.740+0000 7f5f10ff9640 10 client.5042.objecter ms_handle_connect 0x7f5ef806c590 2024-03-27T21:42:06.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.743+0000 7f5a96ffd640 1 -- 172.21.15.60:0/842667156 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5a9c054300 con 0x7f5a98007b60 2024-03-27T21:42:06.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.743+0000 7f5a96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:06.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.743+0000 7f5a96ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:06.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.748+0000 7f5f10ff9640 1 -- 172.21.15.60:0/2535368581 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f5f140d1d60 con 0x7f5f0c038fe0 2024-03-27T21:42:06.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.748+0000 7f5f10ff9640 10 client.5042.objecter ms_dispatch 0x7f5f14000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:06.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.748+0000 7f5f10ff9640 3 client.5042.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:06.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.748+0000 7f5f10ff9640 3 client.5042.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:06.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.748+0000 7f5f10ff9640 20 client.5042.objecter dump_active .. 0 homeless 2024-03-27T21:42:06.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.752+0000 7f5f10ff9640 1 -- 172.21.15.60:0/2535368581 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5f140cd9a0 con 0x7f5f0c038fe0 2024-03-27T21:42:06.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.752+0000 7f5f10ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:06.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:05.752+0000 7f5f10ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:06.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.142+0000 7f3f167fc640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-27T21:42:06.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.142+0000 7f3f167fc640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-27T21:42:06.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.143+0000 7f3f167fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:06.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.143+0000 7f3f167fc640 1 -- 172.21.15.60:0/2588636431 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3f30069470 con 0x7f3f30083230 2024-03-27T21:42:06.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.144+0000 7f3f2cff9640 1 -- 172.21.15.60:0/2588636431 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3f28017070 con 0x7f3f30083230 2024-03-27T21:42:06.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.144+0000 7f3f2cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-27T21:42:06.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.144+0000 7f3f2cff9640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:42:06.130 INFO:teuthology.orchestra.run.smithi060.stdout:55834574880 2024-03-27T21:42:06.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.147+0000 7f3f364f1640 10 librados: watch_flush enter 2024-03-27T21:42:06.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.147+0000 7f3f364f1640 10 librados: watch_flush exit 2024-03-27T21:42:06.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.147+0000 7f3f364f1640 20 client.5036.objecter shutdown clearing up homeless session... 2024-03-27T21:42:06.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.147+0000 7f3f364f1640 10 client.5036.objecter successfully canceled tick 2024-03-27T21:42:06.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.147+0000 7f3f364f1640 1 -- 172.21.15.60:0/2588636431 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3f1c06c660 msgr2=0x7f3f1c06eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:06.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.147+0000 7f3f364f1640 1 --2- 172.21.15.60:0/2588636431 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3f1c06c660 0x7f3f1c06eb20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009e90 tx=0x7f3f24009190 comp rx=0 tx=0).stop 2024-03-27T21:42:06.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.148+0000 7f3f364f1640 10 monclient: shutdown 2024-03-27T21:42:06.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.148+0000 7f3f364f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:06.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.148+0000 7f3f364f1640 1 -- 172.21.15.60:0/2588636431 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3f30083230 msgr2=0x7f3f301ec180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:06.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.148+0000 7f3f364f1640 1 --2- 172.21.15.60:0/2588636431 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3f30083230 0x7f3f301ec180 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3f2800cd40 tx=0x7f3f2800a430 comp rx=0 tx=0).stop 2024-03-27T21:42:06.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.148+0000 7fd755cb9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-27T21:42:06.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.148+0000 7fd755cb9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-27T21:42:06.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.148+0000 7fd755cb9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:06.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.148+0000 7fd755cb9640 1 -- 172.21.15.60:0/3097219464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fd7300055c0 con 0x7fd750141b10 2024-03-27T21:42:06.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.148+0000 7f3f364f1640 1 -- 172.21.15.60:0/2588636431 shutdown_connections 2024-03-27T21:42:06.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.148+0000 7fd737fff640 1 -- 172.21.15.60:0/3097219464 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd744040070 con 0x7fd750141b10 2024-03-27T21:42:06.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.148+0000 7fd737fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-27T21:42:06.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.148+0000 7fd737fff640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:42:06.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.149+0000 7f3f364f1640 1 --2- 172.21.15.60:0/2588636431 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3f30083230 0x7f3f301ec180 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.137 INFO:teuthology.orchestra.run.smithi060.stdout:68719476768 2024-03-27T21:42:06.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.149+0000 7f3f364f1640 1 --2- 172.21.15.60:0/2588636431 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3f1c06c660 0x7f3f1c06eb20 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.149+0000 7f3f364f1640 1 --2- 172.21.15.60:0/2588636431 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f30152440 0x7f3f301ebc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.149+0000 7f3f364f1640 1 --2- 172.21.15.60:0/2588636431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f3006d0f0 0x7f3f30085ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.149+0000 7f3f364f1640 1 -- 172.21.15.60:0/2588636431 >> 172.21.15.60:0/2588636431 conn(0x7f3f300b3620 msgr2=0x7f3f30070310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:06.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.149+0000 7f3f364f1640 1 -- 172.21.15.60:0/2588636431 shutdown_connections 2024-03-27T21:42:06.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.149+0000 7f3f364f1640 1 -- 172.21.15.60:0/2588636431 wait complete. 2024-03-27T21:42:06.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.149+0000 7f3f364f1640 1 librados: shutdown 2024-03-27T21:42:06.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.151+0000 7f2e20ff9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-27T21:42:06.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.151+0000 7f2e20ff9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-27T21:42:06.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.151+0000 7f2e20ff9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:06.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.151+0000 7f2e20ff9640 1 -- 172.21.15.60:0/1344941168 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f2e34143e70 con 0x7f2e34143120 2024-03-27T21:42:06.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7fd755cb9640 10 librados: watch_flush enter 2024-03-27T21:42:06.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7fd755cb9640 10 librados: watch_flush exit 2024-03-27T21:42:06.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7fd755cb9640 20 client.5349.objecter shutdown clearing up homeless session... 2024-03-27T21:42:06.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7fd755cb9640 10 client.5349.objecter successfully canceled tick 2024-03-27T21:42:06.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7fd755cb9640 1 -- 172.21.15.60:0/3097219464 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd73c06c5f0 msgr2=0x7fd73c06eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:06.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7fd755cb9640 1 --2- 172.21.15.60:0/3097219464 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd73c06c5f0 0x7fd73c06eab0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd740009c50 tx=0x7fd740009190 comp rx=0 tx=0).stop 2024-03-27T21:42:06.142 INFO:teuthology.orchestra.run.smithi060.stdout:68719476769 2024-03-27T21:42:06.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7f2e22ffd640 1 -- 172.21.15.60:0/1344941168 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2e28079ea0 con 0x7f2e34143120 2024-03-27T21:42:06.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7f2e22ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-27T21:42:06.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7f2e22ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:42:06.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7fd755cb9640 10 monclient: shutdown 2024-03-27T21:42:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7fd755cb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7fd755cb9640 1 -- 172.21.15.60:0/3097219464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd750141b10 msgr2=0x7fd7500b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7fd755cb9640 1 --2- 172.21.15.60:0/3097219464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd750141b10 0x7fd7500b7120 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd7500616a0 tx=0x7fd7440046c0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7fd755cb9640 1 -- 172.21.15.60:0/3097219464 shutdown_connections 2024-03-27T21:42:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7fd755cb9640 1 --2- 172.21.15.60:0/3097219464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd750141b10 0x7fd7500b7120 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7fd755cb9640 1 --2- 172.21.15.60:0/3097219464 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd73c06c5f0 0x7fd73c06eab0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7fd755cb9640 1 --2- 172.21.15.60:0/3097219464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd75013e830 0x7fd7500b4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7fd755cb9640 1 --2- 172.21.15.60:0/3097219464 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd7500b40c0 0x7fd7501e6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7fd755cb9640 1 -- 172.21.15.60:0/3097219464 >> 172.21.15.60:0/3097219464 conn(0x7fd7500608e0 msgr2=0x7fd7500b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7fd755cb9640 1 -- 172.21.15.60:0/3097219464 shutdown_connections 2024-03-27T21:42:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7fd755cb9640 1 -- 172.21.15.60:0/3097219464 wait complete. 2024-03-27T21:42:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7fd755cb9640 1 librados: shutdown 2024-03-27T21:42:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7f051b669640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-27T21:42:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7f051b669640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-27T21:42:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7f051b669640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.152+0000 7f051b669640 1 -- 172.21.15.60:0/906218950 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f0514069470 con 0x7f0514144db0 2024-03-27T21:42:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.154+0000 7f0509ffb640 1 -- 172.21.15.60:0/906218950 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f050c079f50 con 0x7f0514144db0 2024-03-27T21:42:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.154+0000 7f0509ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-27T21:42:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.154+0000 7f0509ffb640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:42:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.155+0000 7f2e20ff9640 10 librados: watch_flush enter 2024-03-27T21:42:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.155+0000 7f2e20ff9640 10 librados: watch_flush exit 2024-03-27T21:42:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.155+0000 7f2e20ff9640 20 client.5254.objecter shutdown clearing up homeless session... 2024-03-27T21:42:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.155+0000 7f2e20ff9640 10 client.5254.objecter successfully canceled tick 2024-03-27T21:42:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.155+0000 7f2e20ff9640 1 -- 172.21.15.60:0/1344941168 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2e1406c660 msgr2=0x7f2e1406eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.155+0000 7f2e20ff9640 1 --2- 172.21.15.60:0/1344941168 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2e1406c660 0x7f2e1406eb20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c00c530 tx=0x7f2e2c00b040 comp rx=0 tx=0).stop 2024-03-27T21:42:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.155+0000 7f2e20ff9640 10 monclient: shutdown 2024-03-27T21:42:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.155+0000 7f2e20ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.155+0000 7f2e20ff9640 1 -- 172.21.15.60:0/1344941168 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e34143120 msgr2=0x7f2e341da6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.156+0000 7f2e20ff9640 1 --2- 172.21.15.60:0/1344941168 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e34143120 0x7f2e341da6a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2e28033000 tx=0x7f2e280046c0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.156+0000 7f2e20ff9640 1 -- 172.21.15.60:0/1344941168 shutdown_connections 2024-03-27T21:42:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.156+0000 7f2e20ff9640 1 --2- 172.21.15.60:0/1344941168 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2e34146000 0x7f2e341dabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.156+0000 7f2e20ff9640 1 --2- 172.21.15.60:0/1344941168 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2e1406c660 0x7f2e1406eb20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.156+0000 7f2e20ff9640 1 --2- 172.21.15.60:0/1344941168 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e34143120 0x7f2e341da6a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.156+0000 7f2e20ff9640 1 --2- 172.21.15.60:0/1344941168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e340b0ec0 0x7f2e3406eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.156+0000 7f2e20ff9640 1 -- 172.21.15.60:0/1344941168 >> 172.21.15.60:0/1344941168 conn(0x7f2e340608e0 msgr2=0x7f2e340b5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:06.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.156+0000 7f2e20ff9640 1 -- 172.21.15.60:0/1344941168 shutdown_connections 2024-03-27T21:42:06.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.156+0000 7f2e20ff9640 1 -- 172.21.15.60:0/1344941168 wait complete. 2024-03-27T21:42:06.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.156+0000 7f2e20ff9640 1 librados: shutdown 2024-03-27T21:42:06.150 INFO:teuthology.orchestra.run.smithi060.stdout:266287972360 2024-03-27T21:42:06.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.158+0000 7f051b669640 10 librados: watch_flush enter 2024-03-27T21:42:06.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.158+0000 7f051b669640 10 librados: watch_flush exit 2024-03-27T21:42:06.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.158+0000 7f18c8f42640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-27T21:42:06.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.158+0000 7f18c8f42640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-27T21:42:06.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.158+0000 7f18c8f42640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:06.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.158+0000 7f18c8f42640 1 -- 172.21.15.60:0/782235123 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f18c40b1270 con 0x7f18c40ad910 2024-03-27T21:42:06.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.158+0000 7f051b669640 20 client.5257.objecter shutdown clearing up homeless session... 2024-03-27T21:42:06.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.158+0000 7f051b669640 10 client.5257.objecter successfully canceled tick 2024-03-27T21:42:06.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.158+0000 7f051b669640 1 -- 172.21.15.60:0/906218950 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f04f806c650 msgr2=0x7f04f806eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:06.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.158+0000 7f051b669640 1 --2- 172.21.15.60:0/906218950 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f04f806c650 0x7f04f806eb10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0504006fd0 tx=0x7f0504008040 comp rx=0 tx=0).stop 2024-03-27T21:42:06.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.158+0000 7f18abfff640 1 -- 172.21.15.60:0/782235123 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f18c40b1270 con 0x7f18c40ad910 2024-03-27T21:42:06.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.158+0000 7f18abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-27T21:42:06.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.158+0000 7f18abfff640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:42:06.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.158+0000 7f051b669640 10 monclient: shutdown 2024-03-27T21:42:06.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.159+0000 7f051b669640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:06.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.159+0000 7f051b669640 1 -- 172.21.15.60:0/906218950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0514144db0 msgr2=0x7f05140abd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:06.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.159+0000 7f051b669640 1 --2- 172.21.15.60:0/906218950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0514144db0 0x7f05140abd40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f050c00a880 tx=0x7f050c001cb0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.155 INFO:teuthology.orchestra.run.smithi060.stdout:60129542176 2024-03-27T21:42:06.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.160+0000 7f051b669640 1 -- 172.21.15.60:0/906218950 shutdown_connections 2024-03-27T21:42:06.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.160+0000 7f051b669640 1 --2- 172.21.15.60:0/906218950 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f05140ac280 0x7f0514133f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.160+0000 7f051b669640 1 --2- 172.21.15.60:0/906218950 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f04f806c650 0x7f04f806eb10 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.160+0000 7f051b669640 1 --2- 172.21.15.60:0/906218950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0514144db0 0x7f05140abd40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.160+0000 7f051b669640 1 --2- 172.21.15.60:0/906218950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05000047c0 0x7f05140aa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.160+0000 7f051b669640 1 -- 172.21.15.60:0/906218950 >> 172.21.15.60:0/906218950 conn(0x7f05140b3b50 msgr2=0x7f0514071450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:06.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.160+0000 7f051b669640 1 -- 172.21.15.60:0/906218950 shutdown_connections 2024-03-27T21:42:06.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.160+0000 7f051b669640 1 -- 172.21.15.60:0/906218950 wait complete. 2024-03-27T21:42:06.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.160+0000 7f051b669640 1 librados: shutdown 2024-03-27T21:42:06.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.162+0000 7f8919be2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-27T21:42:06.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.162+0000 7f8919be2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-27T21:42:06.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.162+0000 7f8919be2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:06.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.162+0000 7f8919be2640 1 -- 172.21.15.60:0/2195797653 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f88f40055c0 con 0x7f89140b3f60 2024-03-27T21:42:06.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.162+0000 7f18c8f42640 10 librados: watch_flush enter 2024-03-27T21:42:06.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.162+0000 7f18c8f42640 10 librados: watch_flush exit 2024-03-27T21:42:06.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.162+0000 7f18c8f42640 20 client.5263.objecter shutdown clearing up homeless session... 2024-03-27T21:42:06.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.162+0000 7f18c8f42640 10 client.5263.objecter successfully canceled tick 2024-03-27T21:42:06.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.163+0000 7f18c8f42640 1 -- 172.21.15.60:0/782235123 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f18b006c680 msgr2=0x7f18b006eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:06.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.163+0000 7f18c8f42640 1 --2- 172.21.15.60:0/782235123 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f18b006c680 0x7f18b006eb40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f18bc00e2a0 tx=0x7f18bc00b040 comp rx=0 tx=0).stop 2024-03-27T21:42:06.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.163+0000 7f18c8f42640 10 monclient: shutdown 2024-03-27T21:42:06.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.163+0000 7f18c8f42640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:06.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.163+0000 7f18c8f42640 1 -- 172.21.15.60:0/782235123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18c40ad910 msgr2=0x7f18c4141ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:06.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.163+0000 7f18c8f42640 1 --2- 172.21.15.60:0/782235123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18c40ad910 0x7f18c4141ee0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f18c40677c0 tx=0x7f18b40046c0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.163+0000 7f18c8f42640 1 -- 172.21.15.60:0/782235123 shutdown_connections 2024-03-27T21:42:06.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.163+0000 7f18c8f42640 1 --2- 172.21.15.60:0/782235123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18c4148f10 0x7f18c4145d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.163+0000 7f18c8f42640 1 --2- 172.21.15.60:0/782235123 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f18b006c680 0x7f18b006eb40 secure :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f18bc00e2a0 tx=0x7f18bc00b040 comp rx=0 tx=0).stop 2024-03-27T21:42:06.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.163+0000 7f18c8f42640 1 --2- 172.21.15.60:0/782235123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18c40ad910 0x7f18c4141ee0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.163+0000 7f18c8f42640 1 --2- 172.21.15.60:0/782235123 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f18c406ea80 0x7f18c40af9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.163+0000 7f18c8f42640 1 -- 172.21.15.60:0/782235123 >> 172.21.15.60:0/782235123 conn(0x7f18c40b30f0 msgr2=0x7f18c40ae5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:06.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.163+0000 7f89117fa640 1 -- 172.21.15.60:0/2195797653 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8908021020 con 0x7f89140b3f60 2024-03-27T21:42:06.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.163+0000 7f89117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-27T21:42:06.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.163+0000 7f89117fa640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:42:06.161 INFO:teuthology.orchestra.run.smithi060.stdout:60129542176 2024-03-27T21:42:06.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.165+0000 7f5aa74e0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-27T21:42:06.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.165+0000 7f5aa74e0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-27T21:42:06.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.165+0000 7f5aa74e0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:06.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.165+0000 7f5aa74e0640 1 -- 172.21.15.60:0/842667156 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f5a880055c0 con 0x7f5a98007b60 2024-03-27T21:42:06.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.166+0000 7f8919be2640 10 librados: watch_flush enter 2024-03-27T21:42:06.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.166+0000 7f8919be2640 10 librados: watch_flush exit 2024-03-27T21:42:06.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.166+0000 7f8919be2640 20 client.5054.objecter shutdown clearing up homeless session... 2024-03-27T21:42:06.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.166+0000 7f8919be2640 10 client.5054.objecter successfully canceled tick 2024-03-27T21:42:06.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.166+0000 7f18c8f42640 1 -- 172.21.15.60:0/782235123 shutdown_connections 2024-03-27T21:42:06.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.167+0000 7f18c8f42640 1 -- 172.21.15.60:0/782235123 wait complete. 2024-03-27T21:42:06.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.167+0000 7f18c8f42640 1 librados: shutdown 2024-03-27T21:42:06.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.167+0000 7f8919be2640 1 -- 172.21.15.60:0/2195797653 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f890006c610 msgr2=0x7f890006ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:06.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.167+0000 7f8919be2640 1 --2- 172.21.15.60:0/2195797653 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f890006c610 0x7f890006ead0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8904034000 tx=0x7f8904008030 comp rx=0 tx=0).stop 2024-03-27T21:42:06.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.167+0000 7f8919be2640 10 monclient: shutdown 2024-03-27T21:42:06.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.167+0000 7f8919be2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:06.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.167+0000 7f8919be2640 1 -- 172.21.15.60:0/2195797653 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f89140b3f60 msgr2=0x7f89140ab520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:06.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.167+0000 7f8919be2640 1 --2- 172.21.15.60:0/2195797653 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f89140b3f60 0x7f89140ab520 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f890800c7d0 tx=0x7f890800cc90 comp rx=0 tx=0).stop 2024-03-27T21:42:06.165 INFO:teuthology.orchestra.run.smithi060.stdout:60129542176 2024-03-27T21:42:06.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.167+0000 7f5a96ffd640 1 -- 172.21.15.60:0/842667156 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5a9c017020 con 0x7f5a98007b60 2024-03-27T21:42:06.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.167+0000 7f5a96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-27T21:42:06.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.167+0000 7f5a96ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:42:06.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.168+0000 7f8919be2640 1 -- 172.21.15.60:0/2195797653 shutdown_connections 2024-03-27T21:42:06.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.169+0000 7f8919be2640 1 --2- 172.21.15.60:0/2195797653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f891413ecf0 0x7f89141436c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.169+0000 7f8919be2640 1 --2- 172.21.15.60:0/2195797653 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f89140b3f60 0x7f89140ab520 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.169+0000 7f8919be2640 1 --2- 172.21.15.60:0/2195797653 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f890006c610 0x7f890006ead0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.169+0000 7f8919be2640 1 --2- 172.21.15.60:0/2195797653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89140b3580 0x7f89140aafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.169+0000 7f8919be2640 1 -- 172.21.15.60:0/2195797653 >> 172.21.15.60:0/2195797653 conn(0x7f89140608e0 msgr2=0x7f891413e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:06.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.169+0000 7f8919be2640 1 -- 172.21.15.60:0/2195797653 shutdown_connections 2024-03-27T21:42:06.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.169+0000 7f8919be2640 1 -- 172.21.15.60:0/2195797653 wait complete. 2024-03-27T21:42:06.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.169+0000 7f8919be2640 1 librados: shutdown 2024-03-27T21:42:06.169 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476770 got 68719476768 for osd.3 2024-03-27T21:42:06.169 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574881 got 55834574880 for osd.1 2024-03-27T21:42:06.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5aa74e0640 10 librados: watch_flush enter 2024-03-27T21:42:06.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5aa74e0640 10 librados: watch_flush exit 2024-03-27T21:42:06.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5aa74e0640 20 client.5051.objecter shutdown clearing up homeless session... 2024-03-27T21:42:06.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5aa74e0640 10 client.5051.objecter successfully canceled tick 2024-03-27T21:42:06.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5f1a931640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-27T21:42:06.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5f1a931640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-27T21:42:06.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5f1a931640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:06.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5f1a931640 1 -- 172.21.15.60:0/2535368581 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f5f0c0056d0 con 0x7f5f0c038fe0 2024-03-27T21:42:06.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5aa74e0640 1 -- 172.21.15.60:0/842667156 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5a8406c590 msgr2=0x7f5a8406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:06.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5aa74e0640 1 --2- 172.21.15.60:0/842667156 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5a8406c590 0x7f5a8406ea50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5aa004f810 tx=0x7f5aa00726a0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5aa74e0640 10 monclient: shutdown 2024-03-27T21:42:06.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5aa74e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:06.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5aa74e0640 1 -- 172.21.15.60:0/842667156 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5a98007b60 msgr2=0x7f5a980c3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:06.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5aa74e0640 1 --2- 172.21.15.60:0/842667156 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5a98007b60 0x7f5a980c3790 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c00c400 tx=0x7f5a9c00c8c0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5aa74e0640 1 -- 172.21.15.60:0/842667156 shutdown_connections 2024-03-27T21:42:06.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5aa74e0640 1 --2- 172.21.15.60:0/842667156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a980c82c0 0x7f5a9802c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5aa74e0640 1 --2- 172.21.15.60:0/842667156 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5a8406c590 0x7f5a8406ea50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5aa74e0640 1 --2- 172.21.15.60:0/842667156 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a980c76d0 0x7f5a98037640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5aa74e0640 1 --2- 172.21.15.60:0/842667156 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5a98007b60 0x7f5a980c3790 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5aa74e0640 1 -- 172.21.15.60:0/842667156 >> 172.21.15.60:0/842667156 conn(0x7f5a98006ad0 msgr2=0x7f5a98035310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:06.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5aa74e0640 1 -- 172.21.15.60:0/842667156 shutdown_connections 2024-03-27T21:42:06.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5f10ff9640 1 -- 172.21.15.60:0/2535368581 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5f14097030 con 0x7f5f0c038fe0 2024-03-27T21:42:06.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5f10ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-27T21:42:06.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5f10ff9640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:42:06.173 INFO:teuthology.orchestra.run.smithi060.stdout:60129542176 2024-03-27T21:42:06.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5aa74e0640 1 -- 172.21.15.60:0/842667156 wait complete. 2024-03-27T21:42:06.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.171+0000 7f5aa74e0640 1 librados: shutdown 2024-03-27T21:42:06.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.174+0000 7f5f1a931640 10 librados: watch_flush enter 2024-03-27T21:42:06.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.174+0000 7f5f1a931640 10 librados: watch_flush exit 2024-03-27T21:42:06.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.174+0000 7f5f1a931640 20 client.5042.objecter shutdown clearing up homeless session... 2024-03-27T21:42:06.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.174+0000 7f5f1a931640 10 client.5042.objecter successfully canceled tick 2024-03-27T21:42:06.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.174+0000 7f5f1a931640 1 -- 172.21.15.60:0/2535368581 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5ef806c590 msgr2=0x7f5ef806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:06.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.174+0000 7f5f1a931640 1 --2- 172.21.15.60:0/2535368581 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5ef806c590 0x7f5ef806ea50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5f08036000 tx=0x7f5f0803c040 comp rx=0 tx=0).stop 2024-03-27T21:42:06.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.174+0000 7f5f1a931640 10 monclient: shutdown 2024-03-27T21:42:06.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.174+0000 7f5f1a931640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:06.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.174+0000 7f5f1a931640 1 -- 172.21.15.60:0/2535368581 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5f0c038fe0 msgr2=0x7f5f0c033a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:06.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.174+0000 7f5f1a931640 1 --2- 172.21.15.60:0/2535368581 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5f0c038fe0 0x7f5f0c033a40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5f1402c880 tx=0x7f5f14067900 comp rx=0 tx=0).stop 2024-03-27T21:42:06.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.175+0000 7f5f1a931640 1 -- 172.21.15.60:0/2535368581 shutdown_connections 2024-03-27T21:42:06.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.175+0000 7f5f1a931640 1 --2- 172.21.15.60:0/2535368581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f0c0cc600 0x7f5f0c16a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.175+0000 7f5f1a931640 1 --2- 172.21.15.60:0/2535368581 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f5f0c038fe0 0x7f5f0c033a40 secure :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5f1402c880 tx=0x7f5f14067900 comp rx=0 tx=0).stop 2024-03-27T21:42:06.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.175+0000 7f5f1a931640 1 --2- 172.21.15.60:0/2535368581 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f5ef806c590 0x7f5ef806ea50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.175+0000 7f5f1a931640 1 --2- 172.21.15.60:0/2535368581 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f0c035640 0x7f5f0c02f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:06.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.175+0000 7f5f1a931640 1 -- 172.21.15.60:0/2535368581 >> 172.21.15.60:0/2535368581 conn(0x7f5f0c0065d0 msgr2=0x7f5f0c004e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:06.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.175+0000 7f5f1a931640 1 -- 172.21.15.60:0/2535368581 shutdown_connections 2024-03-27T21:42:06.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.175+0000 7f5f1a931640 1 -- 172.21.15.60:0/2535368581 wait complete. 2024-03-27T21:42:06.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:06.175+0000 7f5f1a931640 1 librados: shutdown 2024-03-27T21:42:06.178 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476771 got 68719476769 for osd.5 2024-03-27T21:42:06.179 INFO:tasks.ceph.ceph_manager.ceph:need seq 266287972361 got 266287972360 for osd.2 2024-03-27T21:42:06.180 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542177 got 60129542176 for osd.4 2024-03-27T21:42:06.181 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542177 got 60129542176 for osd.0 2024-03-27T21:42:06.182 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542177 got 60129542176 for osd.7 2024-03-27T21:42:06.182 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542177 got 60129542176 for osd.6 2024-03-27T21:42:07.170 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-27T21:42:07.171 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-27T21:42:07.182 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-27T21:42:07.182 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-27T21:42:07.183 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-27T21:42:07.183 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-27T21:42:07.183 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-27T21:42:07.184 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-27T21:42:07.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.348+0000 7fd5b06f2640 10 monclient: get_monmap_and_config 2024-03-27T21:42:07.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.348+0000 7fd5b06f2640 10 monclient: build_initial_monmap 2024-03-27T21:42:07.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.348+0000 7fd5b06f2640 10 monclient: monmap: 2024-03-27T21:42:07.335 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:07.335 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.335 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:07.349543+0000 2024-03-27T21:42:07.335 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:07.349543+0000 2024-03-27T21:42:07.336 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:07.336 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.336 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:07.336 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:07.336 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:07.336 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:07.336 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:07.336 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.349+0000 7fd5b06f2640 1 Processor -- start 2024-03-27T21:42:07.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.349+0000 7fd5b06f2640 1 -- start start 2024-03-27T21:42:07.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.349+0000 7fd5b06f2640 10 monclient: init 2024-03-27T21:42:07.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.350+0000 7fd5b06f2640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:07.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.350+0000 7fd5b06f2640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-03-27T21:42:07.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.350+0000 7fd5b06f2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd5a0038d70 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:07.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.351+0000 7fd5b06f2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5a0039150 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:07.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.351+0000 7fd5b06f2640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd5a00c8a20 0x7fd5a00d3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.351+0000 7fd5b06f2640 10 monclient(hunting): picked mon.noname-c con 0x7fd5a00c8a20 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:07.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.351+0000 7fd5b06f2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a00c7e60 con 0x7fd5a0039150 2024-03-27T21:42:07.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.351+0000 7fd5b06f2640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a0035430 con 0x7fd5a0038d70 2024-03-27T21:42:07.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.351+0000 7fd5b06f2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.351+0000 7fd5b06f2640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a0037600 con 0x7fd5a00c8a20 2024-03-27T21:42:07.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.351+0000 7fd5b06f2640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:07.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.351+0000 7fd5b06f2640 10 monclient(hunting): authenticate will time out at 968.571998s 2024-03-27T21:42:07.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.352+0000 7fd5adc8d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd5a0039150 0x7fd5a00c8310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39436/0 (socket says 172.21.15.60:39436) 2024-03-27T21:42:07.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.352+0000 7fd5adc8d640 1 -- 172.21.15.60:0/4189807845 learned_addr learned my addr 172.21.15.60:0/4189807845 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:07.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.352+0000 7fd5ad48c640 1 -- 172.21.15.60:0/4189807845 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246360332 0 0) 0x7fd5a00c7e60 con 0x7fd5a0039150 2024-03-27T21:42:07.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.353+0000 7fd5ad48c640 10 monclient(hunting): _negotiate 2024-03-27T21:42:07.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.353+0000 7fd5ad48c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.353+0000 7fd5ad48c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.353+0000 7fd5ad48c640 10 monclient(hunting): my global_id is 5355 2024-03-27T21:42:07.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.353+0000 7fd5ad48c640 1 -- 172.21.15.60:0/4189807845 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd594003520 con 0x7fd5a0039150 2024-03-27T21:42:07.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.353+0000 7efec85dc640 10 monclient: get_monmap_and_config 2024-03-27T21:42:07.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.353+0000 7efec85dc640 10 monclient: build_initial_monmap 2024-03-27T21:42:07.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.353+0000 7efec85dc640 10 monclient: monmap: 2024-03-27T21:42:07.342 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:07.343 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.343 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:07.354443+0000 2024-03-27T21:42:07.343 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:07.354443+0000 2024-03-27T21:42:07.343 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:07.343 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.343 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:07.343 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:07.343 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:07.343 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:07.344 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:07.344 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.353+0000 7fd5ad48c640 1 -- 172.21.15.60:0/4189807845 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4163077305 0 0) 0x7fd594003520 con 0x7fd5a0039150 2024-03-27T21:42:07.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.353+0000 7fd5ad48c640 1 -- 172.21.15.60:0/4189807845 >> v2:172.21.15.191:3300/0 conn(0x7fd5a00c8a20 msgr2=0x7fd5a00d3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:07.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.353+0000 7fd5ad48c640 1 --2- 172.21.15.60:0/4189807845 >> v2:172.21.15.191:3300/0 conn(0x7fd5a00c8a20 0x7fd5a00d3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.354+0000 7fd5ad48c640 1 -- 172.21.15.60:0/4189807845 >> v1:172.21.15.191:6789/0 conn(0x7fd5a0038d70 legacy=0x7fd5a00c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.354+0000 7fd5ad48c640 10 monclient: _finish_hunting 0 2024-03-27T21:42:07.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.354+0000 7fd5ad48c640 1 monclient: found mon.noname-a-legacy 2024-03-27T21:42:07.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.354+0000 7fd5ad48c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:07.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.354+0000 7fd5ad48c640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.60:6789/0 2024-03-27T21:42:07.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.354+0000 7fd5ad48c640 1 -- 172.21.15.60:0/4189807845 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a00c5930 con 0x7fd5a0039150 2024-03-27T21:42:07.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.354+0000 7fd5ad48c640 10 monclient: _finish_auth 0 2024-03-27T21:42:07.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.354+0000 7fd5ad48c640 10 monclient: _check_auth_tickets 2024-03-27T21:42:07.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.354+0000 7fd5ad48c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:07.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.354+0000 7fd5b06f2640 5 monclient: authenticate success, global_id 5355 2024-03-27T21:42:07.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.354+0000 7fd5b06f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:07.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7fd5ad48c640 1 -- 172.21.15.60:0/4189807845 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2933500084 0 0) 0x7fd5a804e550 con 0x7fd5a0039150 2024-03-27T21:42:07.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7fd5ad48c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7fd5ad48c640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-03-27T21:42:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7fd5ad48c640 10 monclient: dump: 2024-03-27T21:42:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.351 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.351 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7fd5ad48c640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] but i'm connected to v1:172.21.15.60:6789/0, reconnecting 2024-03-27T21:42:07.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7fd5ad48c640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:07.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7fd5ad48c640 1 -- 172.21.15.60:0/4189807845 >> v1:172.21.15.60:6789/0 conn(0x7fd5a0039150 legacy=0x7fd5a00c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7fd5ad48c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:42:07.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7fd5ad48c640 1 --2- 172.21.15.60:0/4189807845 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd594003970 0x7fd594003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7fd5ad48c640 10 monclient(hunting): picked mon.c con 0x7fd594003970 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:07.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7fd5ad48c640 1 --2- 172.21.15.60:0/4189807845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5940047c0 0x7fd594004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7fd5ae48e640 1 --2- 172.21.15.60:0/4189807845 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd594003970 0x7fd594003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7fd5adc8d640 1 --2- 172.21.15.60:0/4189807845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5940047c0 0x7fd594004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7fd5ad48c640 10 monclient(hunting): picked mon.a con 0x7fd5940047c0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:07.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7fd5ad48c640 1 --2- 172.21.15.60:0/4189807845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd594005180 0x7fd59400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7fd5ad48c640 10 monclient(hunting): picked mon.b con 0x7fd594005180 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:07.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7fd5ad48c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7fd5ad48c640 1 -- 172.21.15.60:0/4189807845 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a0037600 con 0x7fd5940047c0 2024-03-27T21:42:07.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7fd5ad48c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.357+0000 7fd5ad48c640 1 -- 172.21.15.60:0/4189807845 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59400c0b0 con 0x7fd594005180 2024-03-27T21:42:07.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.357+0000 7fd5ad48c640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.357+0000 7fd5ad48c640 1 -- 172.21.15.60:0/4189807845 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59400c270 con 0x7fd594003970 2024-03-27T21:42:07.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.357+0000 7fd5ad48c640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:07.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.357+0000 7fd5ad48c640 10 monclient(hunting): _finish_auth 0 2024-03-27T21:42:07.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.354+0000 7efec85dc640 1 Processor -- start 2024-03-27T21:42:07.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.354+0000 7efec85dc640 1 -- start start 2024-03-27T21:42:07.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.354+0000 7efec85dc640 10 monclient: init 2024-03-27T21:42:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7efec85dc640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7efec85dc640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-03-27T21:42:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7efec85dc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efeb8038fe0 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7efec85dc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efeb8035640 0x7efeb80cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7efec85dc640 10 monclient(hunting): picked mon.noname-b con 0x7efeb8035640 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7efec85dc640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7efeb80cc600 0x7efeb80d6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7efec85dc640 10 monclient(hunting): picked mon.noname-c con 0x7efeb80cc600 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:07.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7efec85dc640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb8035bf0 con 0x7efeb8038fe0 2024-03-27T21:42:07.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7efec85dc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7efec85dc640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeb80c9d50 con 0x7efeb80cc600 2024-03-27T21:42:07.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7efec85dc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7efec85dc640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb80393c0 con 0x7efeb8035640 2024-03-27T21:42:07.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7efec85dc640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:07.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.355+0000 7efec85dc640 10 monclient(hunting): authenticate will time out at 968.576665s 2024-03-27T21:42:07.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7efec6b79640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7efeb80cc600 0x7efeb80d6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7efec6b79640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7efeb80cc600 0x7efeb80d6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:58880/0 (socket says 172.21.15.60:58880) 2024-03-27T21:42:07.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7efec6b79640 1 -- 172.21.15.60:0/3661183344 learned_addr learned my addr 172.21.15.60:0/3661183344 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:07.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7efec6b79640 10 monclient(hunting): get_auth_request con 0x7efeb80cc600 auth_method 0 2024-03-27T21:42:07.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7efec6b79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7efec6b79640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7efec6b79640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7efec5b77640 1 --2- 172.21.15.60:0/3661183344 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efeb8035640 0x7efeb80cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7efec5b77640 10 monclient(hunting): get_auth_request con 0x7efeb8035640 auth_method 0 2024-03-27T21:42:07.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7efec5b77640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7efec5b77640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7efec5b77640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7efec6b79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7efec6b79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7efec6b79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7efec5b77640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7efec5b77640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.356+0000 7efec5b77640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.357+0000 7efec6b79640 10 monclient(hunting): handle_auth_done global_id 5272 payload 274 2024-03-27T21:42:07.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.357+0000 7efec6b79640 1 -- 172.21.15.60:0/3661183344 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efeb8035640 msgr2=0x7efeb80cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:07.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.357+0000 7efec6b79640 1 --2- 172.21.15.60:0/3661183344 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efeb8035640 0x7efeb80cc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.357+0000 7efec6b79640 1 -- 172.21.15.60:0/3661183344 >> v1:172.21.15.60:6789/0 conn(0x7efeb8038fe0 legacy=0x7efeb80c9640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-27T21:42:07.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.357+0000 7efec6b79640 10 monclient: _finish_hunting 0 2024-03-27T21:42:07.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.357+0000 7efec6b79640 1 monclient: found mon.noname-c 2024-03-27T21:42:07.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.357+0000 7efec6b79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:07.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.357+0000 7efec6b79640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.191:3300/0 2024-03-27T21:42:07.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.357+0000 7efec6b79640 1 -- 172.21.15.60:0/3661183344 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb80373e0 con 0x7efeb80cc600 2024-03-27T21:42:07.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.357+0000 7fd5aec8f640 1 --2- 172.21.15.60:0/4189807845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd594005180 0x7fd59400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.357+0000 7fd5aec8f640 10 monclient(hunting): get_auth_request con 0x7fd594005180 auth_method 0 2024-03-27T21:42:07.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.357+0000 7fd5aec8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5aec8f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5aec8f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:07.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7efec5b77640 1 --2- 172.21.15.60:0/3661183344 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efeb8035640 0x7efeb80cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:07.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5ae48e640 10 monclient(hunting): get_auth_request con 0x7fd594003970 auth_method 0 2024-03-27T21:42:07.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5ae48e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5ae48e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5ae48e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:07.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5adc8d640 10 monclient(hunting): get_auth_request con 0x7fd5940047c0 auth_method 0 2024-03-27T21:42:07.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5adc8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5adc8d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5adc8d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:07.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5b06f2640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:07.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5ae48e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5ae48e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5ae48e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:42:07.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5aec8f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5aec8f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5aec8f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:42:07.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5adc8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5adc8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5adc8d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:42:07.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5ae48e640 10 monclient(hunting): handle_auth_done global_id 5355 payload 293 2024-03-27T21:42:07.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5ae48e640 1 -- 172.21.15.60:0/4189807845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd594005180 msgr2=0x7fd59400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:07.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5ae48e640 1 --2- 172.21.15.60:0/4189807845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd594005180 0x7fd59400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5ae48e640 1 -- 172.21.15.60:0/4189807845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5940047c0 msgr2=0x7fd594004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:07.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5ae48e640 1 --2- 172.21.15.60:0/4189807845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5940047c0 0x7fd594004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5ae48e640 10 monclient: _finish_hunting 0 2024-03-27T21:42:07.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5ae48e640 1 monclient: found mon.c 2024-03-27T21:42:07.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5ae48e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:07.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5ae48e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:07.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5ae48e640 1 -- 172.21.15.60:0/4189807845 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd594003520 con 0x7fd594003970 2024-03-27T21:42:07.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5ae48e640 10 monclient: _finish_auth 0 2024-03-27T21:42:07.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5ae48e640 10 monclient: _check_auth_tickets 2024-03-27T21:42:07.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.358+0000 7fd5ae48e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:07.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.359+0000 7fd5aec8f640 1 -- 172.21.15.60:0/4189807845 reap_dead start 2024-03-27T21:42:07.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.359+0000 7efec6b79640 1 --2- 172.21.15.60:0/3661183344 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efeb80cc600 0x7efeb80d6930 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efec0072960 tx=0x7efec00a5320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9721320dac7f12 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.359+0000 7efec5376640 1 -- 172.21.15.60:0/3661183344 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec00ad640 con 0x7efeb80cc600 2024-03-27T21:42:07.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.359+0000 7efec5376640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.359+0000 7efec5376640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:07.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.359+0000 7efec5376640 10 monclient: dump: 2024-03-27T21:42:07.366 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.367 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.367 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.367 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.367 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.367 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.367 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.367 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.367 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.367 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.359+0000 7efec5376640 10 monclient: _finish_auth 0 2024-03-27T21:42:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.359+0000 7efec5376640 10 monclient: _check_auth_tickets 2024-03-27T21:42:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.359+0000 7efec5376640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.359+0000 7efec5376640 1 -- 172.21.15.60:0/3661183344 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec00ad7a0 con 0x7efeb80cc600 2024-03-27T21:42:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.359+0000 7efec85dc640 5 monclient: authenticate success, global_id 5272 2024-03-27T21:42:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.359+0000 7efec85dc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.359+0000 7efec5376640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.359+0000 7efec85dc640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.359+0000 7efec5376640 1 -- 172.21.15.60:0/3661183344 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec00adb90 con 0x7efeb80cc600 2024-03-27T21:42:07.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.359+0000 7efec85dc640 4 set_mon_vals no callback set 2024-03-27T21:42:07.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.359+0000 7efec5376640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.359+0000 7efec5376640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:42:07.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.359+0000 7efec5376640 10 monclient: dump: 2024-03-27T21:42:07.369 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.369 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.369 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.369 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.369 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.370 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.370 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.370 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.370 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.370 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.360+0000 7fd5ae48e640 1 --2- 172.21.15.60:0/4189807845 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd594003970 0x7fd594003d50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4001810 tx=0x7fd5940057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66b54902f4c55577 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.360+0000 7efec85dc640 10 monclient: shutdown 2024-03-27T21:42:07.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.360+0000 7efec85dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:07.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.360+0000 7efec85dc640 1 -- 172.21.15.60:0/3661183344 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efeb80cc600 msgr2=0x7efeb80d6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.360+0000 7efec85dc640 1 --2- 172.21.15.60:0/3661183344 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efeb80cc600 0x7efeb80d6930 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efec0072960 tx=0x7efec00a5320 comp rx=0 tx=0).stop 2024-03-27T21:42:07.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.361+0000 7efec85dc640 1 -- 172.21.15.60:0/3661183344 shutdown_connections 2024-03-27T21:42:07.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.361+0000 7efec85dc640 1 --2- 172.21.15.60:0/3661183344 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efeb80cc600 0x7efeb80d6930 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.361+0000 7efec85dc640 1 --2- 172.21.15.60:0/3661183344 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efeb8035640 0x7efeb80cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.361+0000 7efec85dc640 1 -- 172.21.15.60:0/3661183344 >> 172.21.15.60:0/3661183344 conn(0x7efeb80065d0 msgr2=0x7efeb80069e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:07.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.361+0000 7efec85dc640 1 -- 172.21.15.60:0/3661183344 shutdown_connections 2024-03-27T21:42:07.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.361+0000 7efec85dc640 1 -- 172.21.15.60:0/3661183344 wait complete. 2024-03-27T21:42:07.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.362+0000 7efec85dc640 10 monclient: build_initial_monmap 2024-03-27T21:42:07.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.362+0000 7efec85dc640 10 monclient: monmap: 2024-03-27T21:42:07.372 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:07.372 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:07.372 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:07.372 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:07.373 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:07.373 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:07.373 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:07.373 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:07.373 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:07.373 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.361+0000 7fd5ad48c640 1 -- 172.21.15.60:0/4189807845 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a4032600 con 0x7fd594003970 2024-03-27T21:42:07.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.361+0000 7fd5ad48c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.361+0000 7fd5ad48c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:07.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.361+0000 7fd5ad48c640 10 monclient: dump: 2024-03-27T21:42:07.375 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.375 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.375 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.375 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.375 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.376 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.376 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.376 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.376 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.376 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.361+0000 7fd5b06f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:07.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.361+0000 7fd5ad48c640 1 -- 172.21.15.60:0/4189807845 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a402e050 con 0x7fd594003970 2024-03-27T21:42:07.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.361+0000 7fd5ad48c640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:07.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.361+0000 7fd5b06f2640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:07.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.361+0000 7fd5b06f2640 4 set_mon_vals no callback set 2024-03-27T21:42:07.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.362+0000 7fd5b06f2640 10 monclient: shutdown 2024-03-27T21:42:07.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.362+0000 7fd5b06f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:07.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.362+0000 7fd5b06f2640 1 -- 172.21.15.60:0/4189807845 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd594003970 msgr2=0x7fd594003d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.362+0000 7fd5b06f2640 1 --2- 172.21.15.60:0/4189807845 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd594003970 0x7fd594003d50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4001810 tx=0x7fd5940057f0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.362+0000 7fd5b06f2640 1 -- 172.21.15.60:0/4189807845 shutdown_connections 2024-03-27T21:42:07.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.362+0000 7fd5b06f2640 1 --2- 172.21.15.60:0/4189807845 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd594003970 0x7fd594003d50 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.362+0000 7fd5b06f2640 1 -- 172.21.15.60:0/4189807845 >> 172.21.15.60:0/4189807845 conn(0x7fd5a00065d0 msgr2=0x7fd5a00069e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:07.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.363+0000 7fd5b06f2640 1 -- 172.21.15.60:0/4189807845 shutdown_connections 2024-03-27T21:42:07.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.363+0000 7fd5b06f2640 1 -- 172.21.15.60:0/4189807845 wait complete. 2024-03-27T21:42:07.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7fd5b06f2640 10 monclient: build_initial_monmap 2024-03-27T21:42:07.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7fd5b06f2640 10 monclient: monmap: 2024-03-27T21:42:07.378 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:07.378 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:07.378 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:07.379 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:07.379 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:07.379 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:07.379 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:07.379 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:07.379 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:07.379 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7fd5b06f2640 1 librados: starting msgr at 2024-03-27T21:42:07.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7fd5b06f2640 1 librados: starting objecter 2024-03-27T21:42:07.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7fd5b06f2640 1 Processor -- start 2024-03-27T21:42:07.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.362+0000 7efec85dc640 1 librados: starting msgr at 2024-03-27T21:42:07.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.362+0000 7efec85dc640 1 librados: starting objecter 2024-03-27T21:42:07.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.362+0000 7efec85dc640 1 Processor -- start 2024-03-27T21:42:07.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.362+0000 7efec85dc640 1 -- start start 2024-03-27T21:42:07.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.362+0000 7efec85dc640 1 librados: setting wanted keys 2024-03-27T21:42:07.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.362+0000 7efec85dc640 1 librados: calling monclient init 2024-03-27T21:42:07.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.362+0000 7efec85dc640 10 monclient: init 2024-03-27T21:42:07.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.363+0000 7efec85dc640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:07.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.363+0000 7efec85dc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:42:07.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.363+0000 7efec85dc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efeb8035640 0x7efeb80320d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.363+0000 7efec85dc640 10 monclient(hunting): picked mon.noname-a con 0x7efeb8035640 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:07.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.363+0000 7efec85dc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efeb8038fe0 0x7efeb8032610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.363+0000 7efec85dc640 10 monclient(hunting): picked mon.noname-c con 0x7efeb8038fe0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:07.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.363+0000 7efec85dc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efeb80cc600 0x7efeb80b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.363+0000 7efec85dc640 10 monclient(hunting): picked mon.noname-b con 0x7efeb80cc600 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:07.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.363+0000 7efec85dc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.365+0000 7fd5b06f2640 1 -- start start 2024-03-27T21:42:07.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.365+0000 7fd5b06f2640 1 librados: setting wanted keys 2024-03-27T21:42:07.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.365+0000 7fd5b06f2640 1 librados: calling monclient init 2024-03-27T21:42:07.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.365+0000 7fd5b06f2640 10 monclient: init 2024-03-27T21:42:07.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.365+0000 7fd5b06f2640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:07.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.365+0000 7fd5b06f2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:42:07.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.365+0000 7fd5b06f2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd594003970 0x7fd5a00cb600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.365+0000 7fd5b06f2640 10 monclient(hunting): picked mon.noname-b con 0x7fd594003970 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:07.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.365+0000 7fd5b06f2640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd5a00cab60 0x7fd5a00326e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.365+0000 7fd5b06f2640 10 monclient(hunting): picked mon.noname-c con 0x7fd5a00cab60 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:07.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.365+0000 7fd5b06f2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a002fdc0 0x7fd5a0030240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.365+0000 7fd5b06f2640 10 monclient(hunting): picked mon.noname-a con 0x7fd5a002fdc0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:07.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.365+0000 7fd5b06f2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.365+0000 7fd5b06f2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a002e0b0 con 0x7fd5a002fdc0 2024-03-27T21:42:07.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.365+0000 7fd5b06f2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.365+0000 7fd5b06f2640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a00cc120 con 0x7fd594003970 2024-03-27T21:42:07.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.365+0000 7fd5b06f2640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.365+0000 7fd5b06f2640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a00cbc00 con 0x7fd5a00cab60 2024-03-27T21:42:07.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.365+0000 7fd5b06f2640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:07.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.365+0000 7fd5b06f2640 10 monclient(hunting): authenticate will time out at 968.586909s 2024-03-27T21:42:07.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.363+0000 7efec85dc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb8033d20 con 0x7efeb8035640 2024-03-27T21:42:07.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.363+0000 7efec85dc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.366+0000 7fd5adc8d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd5a00cab60 0x7fd5a00326e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.363+0000 7efec85dc640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb80b5bd0 con 0x7efeb80cc600 2024-03-27T21:42:07.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.363+0000 7efec85dc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.363+0000 7efec85dc640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb80315c0 con 0x7efeb8038fe0 2024-03-27T21:42:07.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.363+0000 7efec85dc640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:07.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.363+0000 7efec85dc640 10 monclient(hunting): authenticate will time out at 968.584719s 2024-03-27T21:42:07.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.366+0000 7fd5ae48e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd594003970 0x7fd5a00cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6b79640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efeb80cc600 0x7efeb80b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.366+0000 7fd5ae48e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd594003970 0x7fd5a00cb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:58892/0 (socket says 172.21.15.60:58892) 2024-03-27T21:42:07.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.366+0000 7fd5ae48e640 1 -- 172.21.15.60:0/836846180 learned_addr learned my addr 172.21.15.60:0/836846180 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:07.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.366+0000 7fd5ae48e640 10 monclient(hunting): get_auth_request con 0x7fd594003970 auth_method 0 2024-03-27T21:42:07.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.366+0000 7fd5ae48e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.366+0000 7fd5ae48e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.366+0000 7fd5ae48e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6b79640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efeb80cc600 0x7efeb80b4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:58888/0 (socket says 172.21.15.60:58888) 2024-03-27T21:42:07.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6b79640 1 -- 172.21.15.60:0/506350626 learned_addr learned my addr 172.21.15.60:0/506350626 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:07.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6b79640 10 monclient(hunting): get_auth_request con 0x7efeb80cc600 auth_method 0 2024-03-27T21:42:07.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.366+0000 7fd5ae48e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.366+0000 7fd5ae48e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.366+0000 7fd5ae48e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6b79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.366+0000 7fd5aec8f640 1 --2- 172.21.15.60:0/836846180 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a002fdc0 0x7fd5a0030240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6b79640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd5aec8f640 10 monclient(hunting): get_auth_request con 0x7fd5a002fdc0 auth_method 0 2024-03-27T21:42:07.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd5aec8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd5aec8f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd5aec8f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6b79640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6b79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd5ae48e640 10 monclient(hunting): handle_auth_done global_id 5278 payload 824 2024-03-27T21:42:07.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd5ae48e640 1 -- 172.21.15.60:0/836846180 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd5a00cab60 msgr2=0x7fd5a00326e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd5ae48e640 1 --2- 172.21.15.60:0/836846180 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd5a00cab60 0x7fd5a00326e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd5ae48e640 1 -- 172.21.15.60:0/836846180 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a002fdc0 msgr2=0x7fd5a0030240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6b79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6b79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6378640 1 --2- 172.21.15.60:0/506350626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efeb8035640 0x7efeb80320d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd5ae48e640 1 --2- 172.21.15.60:0/836846180 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a002fdc0 0x7fd5a0030240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd5ae48e640 10 monclient: _finish_hunting 0 2024-03-27T21:42:07.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd5ae48e640 1 monclient: found mon.noname-b 2024-03-27T21:42:07.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd5ae48e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:07.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd5ae48e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:42:07.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd5ae48e640 1 -- 172.21.15.60:0/836846180 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a0032c20 con 0x7fd594003970 2024-03-27T21:42:07.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd5aec8f640 1 --2- 172.21.15.60:0/836846180 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a002fdc0 0x7fd5a0030240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:42:07.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6378640 10 monclient(hunting): get_auth_request con 0x7efeb8035640 auth_method 0 2024-03-27T21:42:07.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6378640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6378640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6378640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6b79640 10 monclient(hunting): handle_auth_done global_id 5275 payload 824 2024-03-27T21:42:07.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6b79640 1 -- 172.21.15.60:0/506350626 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efeb8038fe0 msgr2=0x7efeb8032610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-27T21:42:07.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd5ae48e640 1 --2- 172.21.15.60:0/836846180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd594003970 0x7fd5a00cb600 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd5a402bd60 tx=0x7fd5a4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6b79640 1 --2- 172.21.15.60:0/506350626 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efeb8038fe0 0x7efeb8032610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6b79640 1 -- 172.21.15.60:0/506350626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efeb8035640 msgr2=0x7efeb80320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6b79640 1 --2- 172.21.15.60:0/506350626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efeb8035640 0x7efeb80320d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd59effd640 10 client.?.objecter ms_handle_connect 0x7fd594003970 2024-03-27T21:42:07.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd59effd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:07.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6b79640 10 monclient: _finish_hunting 0 2024-03-27T21:42:07.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd59effd640 1 -- 172.21.15.60:0/836846180 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a40067d0 con 0x7fd594003970 2024-03-27T21:42:07.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd59effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd59effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:07.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd59effd640 10 monclient: dump: 2024-03-27T21:42:07.403 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.403 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.403 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.403 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.403 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.403 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.403 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.403 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.404 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.404 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd59effd640 10 monclient: _finish_auth 0 2024-03-27T21:42:07.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd59effd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:07.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7fd59effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:07.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6b79640 1 monclient: found mon.noname-b 2024-03-27T21:42:07.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6b79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:07.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6b79640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:42:07.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6b79640 1 -- 172.21.15.60:0/506350626 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efebc009400 con 0x7efeb80cc600 2024-03-27T21:42:07.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7fd5b06f2640 5 monclient: authenticate success, global_id 5278 2024-03-27T21:42:07.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7fd5b06f2640 10 monclient: _renew_subs 2024-03-27T21:42:07.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7fd5b06f2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:07.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7fd5b06f2640 1 -- 172.21.15.60:0/836846180 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a00c90c0 con 0x7fd594003970 2024-03-27T21:42:07.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7fd5b06f2640 10 client.5278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:07.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7fd5b06f2640 10 monclient: _renew_subs 2024-03-27T21:42:07.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7fd5b06f2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:07.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7fd5b06f2640 1 -- 172.21.15.60:0/836846180 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a00c9620 con 0x7fd594003970 2024-03-27T21:42:07.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7fd5b06f2640 1 librados: init done 2024-03-27T21:42:07.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7fd59effd640 1 -- 172.21.15.60:0/836846180 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a4006930 con 0x7fd594003970 2024-03-27T21:42:07.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7fd59effd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:07.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.364+0000 7efec6378640 1 --2- 172.21.15.60:0/506350626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efeb8035640 0x7efeb80320d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:42:07.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7fd59effd640 1 -- 172.21.15.60:0/836846180 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a4006ca0 con 0x7fd594003970 2024-03-27T21:42:07.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7fd5acc8b640 4 set_mon_vals no callback set 2024-03-27T21:42:07.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.365+0000 7efec6b79640 1 --2- 172.21.15.60:0/506350626 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efeb80cc600 0x7efeb80b4360 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efec0072960 tx=0x7efec00a7070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7efeb6ffd640 10 client.?.objecter ms_handle_connect 0x7efeb80cc600 2024-03-27T21:42:07.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.367+0000 7efeb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:07.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7efeb6ffd640 1 -- 172.21.15.60:0/506350626 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec00d8830 con 0x7efeb80cc600 2024-03-27T21:42:07.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7efeb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7efeb6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:07.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7efeb6ffd640 10 monclient: dump: 2024-03-27T21:42:07.410 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.410 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.410 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.410 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.410 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.411 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.411 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.411 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.411 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.411 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7efeb6ffd640 10 monclient: _finish_auth 0 2024-03-27T21:42:07.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7efeb6ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:07.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.369+0000 7fd59effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.369+0000 7fd59effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:42:07.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.369+0000 7fd59effd640 10 monclient: dump: 2024-03-27T21:42:07.412 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.412 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.412 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.413 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.413 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.413 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.413 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.413 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.413 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.413 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7efeb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:07.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7efec85dc640 5 monclient: authenticate success, global_id 5275 2024-03-27T21:42:07.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7efec85dc640 10 monclient: _renew_subs 2024-03-27T21:42:07.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7efec85dc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:07.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7efec85dc640 1 -- 172.21.15.60:0/506350626 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec006be70 con 0x7efeb80cc600 2024-03-27T21:42:07.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7efec85dc640 10 client.5275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:07.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7efec85dc640 10 monclient: _renew_subs 2024-03-27T21:42:07.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7efec85dc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:07.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7efec85dc640 1 -- 172.21.15.60:0/506350626 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb80b48a0 con 0x7efeb80cc600 2024-03-27T21:42:07.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7efec85dc640 1 librados: init done 2024-03-27T21:42:07.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7efeb6ffd640 1 -- 172.21.15.60:0/506350626 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec00d8990 con 0x7efeb80cc600 2024-03-27T21:42:07.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7efeb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:07.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7efeb6ffd640 1 -- 172.21.15.60:0/506350626 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec00d8d00 con 0x7efeb80cc600 2024-03-27T21:42:07.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.368+0000 7efeb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.369+0000 7efeb6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:42:07.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.369+0000 7efeb6ffd640 10 monclient: dump: 2024-03-27T21:42:07.418 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.418 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.418 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.418 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.418 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.419 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.419 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.419 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.419 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.419 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.369+0000 7efec4b75640 4 set_mon_vals no callback set 2024-03-27T21:42:07.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.371+0000 7efeb6ffd640 1 -- 172.21.15.60:0/506350626 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7efec00e93f0 con 0x7efeb80cc600 2024-03-27T21:42:07.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.371+0000 7efeb6ffd640 1 --2- 172.21.15.60:0/506350626 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7efea406c660 0x7efea406eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.372+0000 7efec6378640 1 --2- 172.21.15.60:0/506350626 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7efea406c660 0x7efea406eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.372+0000 7efec6378640 10 monclient: get_auth_request con 0x7efea406c660 auth_method 0 2024-03-27T21:42:07.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.372+0000 7efec85dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.372+0000 7efec85dc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.372+0000 7efec85dc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:07.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.372+0000 7efec85dc640 1 -- 172.21.15.60:0/506350626 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea8005040 con 0x7efeb80cc600 2024-03-27T21:42:07.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.372+0000 7efec6378640 1 --2- 172.21.15.60:0/506350626 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7efea406c660 0x7efea406eb20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7efeb0006fd0 tx=0x7efeb0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.373+0000 7fd5b06f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.373+0000 7fd5b06f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.373+0000 7fd5b06f2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:07.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.373+0000 7efeb6ffd640 10 client.5275.objecter ms_handle_connect 0x7efea406c660 2024-03-27T21:42:07.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.373+0000 7fd5b06f2640 1 -- 172.21.15.60:0/836846180 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd590005040 con 0x7fd594003970 2024-03-27T21:42:07.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.373+0000 7efeb6ffd640 1 -- 172.21.15.60:0/506350626 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7efec0146ac0 con 0x7efeb80cc600 2024-03-27T21:42:07.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.373+0000 7efeb6ffd640 10 client.5275.objecter ms_dispatch 0x7efec0000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:07.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.373+0000 7efeb6ffd640 3 client.5275.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:07.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.373+0000 7efeb6ffd640 3 client.5275.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:07.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.373+0000 7efeb6ffd640 20 client.5275.objecter dump_active .. 0 homeless 2024-03-27T21:42:07.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.375+0000 7fd59effd640 1 -- 172.21.15.60:0/836846180 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd5a4039050 con 0x7fd594003970 2024-03-27T21:42:07.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.375+0000 7fd59effd640 1 --2- 172.21.15.60:0/836846180 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd59406e140 0x7fd594070600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.376+0000 7fd59effd640 1 -- 172.21.15.60:0/836846180 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fd5a40aa9a0 con 0x7fd594003970 2024-03-27T21:42:07.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.376+0000 7fd59effd640 10 client.5278.objecter ms_dispatch 0x7fd5a8000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:07.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.376+0000 7fd59effd640 3 client.5278.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:07.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.376+0000 7fd59effd640 3 client.5278.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:07.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.376+0000 7fd59effd640 20 client.5278.objecter dump_active .. 0 homeless 2024-03-27T21:42:07.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.376+0000 7fd5adc8d640 1 --2- 172.21.15.60:0/836846180 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd59406e140 0x7fd594070600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.376+0000 7fd5adc8d640 10 monclient: get_auth_request con 0x7fd59406e140 auth_method 0 2024-03-27T21:42:07.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.376+0000 7fd59effd640 10 client.5278.objecter ms_handle_connect 0x7fd59406e140 2024-03-27T21:42:07.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.376+0000 7fd5adc8d640 1 --2- 172.21.15.60:0/836846180 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd59406e140 0x7fd594070600 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd5a804f930 tx=0x7fd5a804ea70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.378+0000 7efeb6ffd640 1 -- 172.21.15.60:0/506350626 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efec0111e60 con 0x7efeb80cc600 2024-03-27T21:42:07.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.378+0000 7efeb6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.378+0000 7efeb6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:07.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.382+0000 7fd59effd640 1 -- 172.21.15.60:0/836846180 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd5a4075d70 con 0x7fd594003970 2024-03-27T21:42:07.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.382+0000 7fd59effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.382+0000 7fd59effd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:07.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.400+0000 7f28f08af640 10 monclient: get_monmap_and_config 2024-03-27T21:42:07.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.400+0000 7f28f08af640 10 monclient: build_initial_monmap 2024-03-27T21:42:07.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.400+0000 7f28f08af640 10 monclient: monmap: 2024-03-27T21:42:07.428 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:07.428 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.428 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:07.401280+0000 2024-03-27T21:42:07.428 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:07.401280+0000 2024-03-27T21:42:07.428 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:07.429 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.429 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:07.429 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:07.429 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:07.429 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:07.429 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:07.429 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.402+0000 7f4b19918640 10 monclient: get_monmap_and_config 2024-03-27T21:42:07.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.402+0000 7f4b19918640 10 monclient: build_initial_monmap 2024-03-27T21:42:07.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.402+0000 7f4b19918640 10 monclient: monmap: 2024-03-27T21:42:07.430 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:07.430 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.430 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:07.403129+0000 2024-03-27T21:42:07.431 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:07.403129+0000 2024-03-27T21:42:07.431 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:07.431 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.431 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:07.431 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:07.431 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:07.431 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:07.431 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:07.431 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f4b19918640 1 Processor -- start 2024-03-27T21:42:07.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f9e9b59e640 10 monclient: get_monmap_and_config 2024-03-27T21:42:07.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f9e9b59e640 10 monclient: build_initial_monmap 2024-03-27T21:42:07.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f9e9b59e640 10 monclient: monmap: 2024-03-27T21:42:07.432 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:07.433 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.433 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:07.404767+0000 2024-03-27T21:42:07.433 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:07.404767+0000 2024-03-27T21:42:07.433 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:07.433 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.433 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:07.433 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:07.433 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:07.433 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:07.434 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:07.434 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.401+0000 7f28f08af640 1 Processor -- start 2024-03-27T21:42:07.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.401+0000 7f28f08af640 1 -- start start 2024-03-27T21:42:07.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.401+0000 7f28f08af640 10 monclient: init 2024-03-27T21:42:07.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.402+0000 7f28f08af640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:07.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.402+0000 7f28f08af640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-03-27T21:42:07.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.402+0000 7f28f08af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f28ec0b0ec0 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:07.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7fa477381640 10 monclient: get_monmap_and_config 2024-03-27T21:42:07.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7fa477381640 10 monclient: build_initial_monmap 2024-03-27T21:42:07.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7fa477381640 10 monclient: monmap: 2024-03-27T21:42:07.436 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:07.436 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.436 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:07.404797+0000 2024-03-27T21:42:07.436 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:07.404797+0000 2024-03-27T21:42:07.437 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:07.437 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.437 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:07.437 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:07.437 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:07.437 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:07.437 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:07.437 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.402+0000 7f28f08af640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f28ec0b32c0 0x7f28ec149000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.402+0000 7f28f08af640 10 monclient(hunting): picked mon.noname-a con 0x7f28ec0b32c0 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:07.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.402+0000 7f28f08af640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28ec149540 0x7f28ec150190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.402+0000 7f28f08af640 10 monclient(hunting): picked mon.noname-b con 0x7f28ec149540 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:07.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.402+0000 7f28f08af640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28ec06d370 con 0x7f28ec0b0ec0 2024-03-27T21:42:07.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.402+0000 7f28f08af640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.402+0000 7f28f08af640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28ec0b3930 con 0x7f28ec0b32c0 2024-03-27T21:42:07.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.402+0000 7f28f08af640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.402+0000 7f28f08af640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28ec0b12a0 con 0x7f28ec149540 2024-03-27T21:42:07.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.404+0000 7f4b19918640 1 -- start start 2024-03-27T21:42:07.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.404+0000 7f4b19918640 10 monclient: init 2024-03-27T21:42:07.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.402+0000 7f28f08af640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:07.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.402+0000 7f28f08af640 10 monclient(hunting): authenticate will time out at 968.623634s 2024-03-27T21:42:07.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f28ead9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28ec149540 0x7f28ec150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f28e9d9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f28ec0b32c0 0x7f28ec149000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f28e9d9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f28ec0b32c0 0x7f28ec149000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39500/0 (socket says 172.21.15.60:39500) 2024-03-27T21:42:07.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f28e9d9b640 1 -- 172.21.15.60:0/1239298856 learned_addr learned my addr 172.21.15.60:0/1239298856 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:07.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f28e9d9b640 10 monclient(hunting): get_auth_request con 0x7f28ec0b32c0 auth_method 0 2024-03-27T21:42:07.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f28e9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f28e9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f28e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f28ead9d640 10 monclient(hunting): get_auth_request con 0x7f28ec149540 auth_method 0 2024-03-27T21:42:07.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f28ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f28ead9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f28ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f28e9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f28e9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f28e9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f28ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f28ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.403+0000 7f28ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.404+0000 7f28ead9d640 10 monclient(hunting): handle_auth_done global_id 5060 payload 274 2024-03-27T21:42:07.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.404+0000 7f28ead9d640 1 -- 172.21.15.60:0/1239298856 >> v2:172.21.15.60:3300/0 conn(0x7f28ec0b32c0 msgr2=0x7f28ec149000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:07.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.404+0000 7f28ead9d640 1 --2- 172.21.15.60:0/1239298856 >> v2:172.21.15.60:3300/0 conn(0x7f28ec0b32c0 0x7f28ec149000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.404+0000 7f28ead9d640 1 -- 172.21.15.60:0/1239298856 >> v1:172.21.15.191:6789/0 conn(0x7f28ec0b0ec0 legacy=0x7f28ec142cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.404+0000 7f28ead9d640 10 monclient: _finish_hunting 0 2024-03-27T21:42:07.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.404+0000 7f28ead9d640 1 monclient: found mon.noname-b 2024-03-27T21:42:07.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.404+0000 7f28ead9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:07.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.404+0000 7f28ead9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:42:07.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.404+0000 7f28ead9d640 1 -- 172.21.15.60:0/1239298856 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ec06c1b0 con 0x7f28ec149540 2024-03-27T21:42:07.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.404+0000 7f28e9d9b640 1 --2- 172.21.15.60:0/1239298856 >> v2:172.21.15.60:3300/0 conn(0x7f28ec0b32c0 0x7f28ec149000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:07.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f9e9b59e640 1 Processor -- start 2024-03-27T21:42:07.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f9e9b59e640 1 -- start start 2024-03-27T21:42:07.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f9e9b59e640 10 monclient: init 2024-03-27T21:42:07.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f28ead9d640 1 --2- 172.21.15.60:0/1239298856 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28ec149540 0x7f28ec150190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f28e4009d90 tx=0x7f28e402bc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=523ae329e7458316 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f4b19918640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:07.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f4b19918640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-03-27T21:42:07.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f4b19918640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4b140ad7b0 0x7f4b1413b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f4b19918640 10 monclient(hunting): picked mon.noname-b con 0x7f4b140ad7b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:07.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f4b19918640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4b1406e1e0 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:07.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f4b19918640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4b14070da0 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:07.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f4b19918640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b14071940 con 0x7f4b14070da0 2024-03-27T21:42:07.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f4b19918640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b1406eb90 con 0x7f4b1406e1e0 2024-03-27T21:42:07.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f4b19918640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f4b19918640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b140adb90 con 0x7f4b140ad7b0 2024-03-27T21:42:07.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f4b19918640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:07.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f4b19918640 10 monclient(hunting): authenticate will time out at 968.626835s 2024-03-27T21:42:07.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7fce2ffff640 10 monclient: get_monmap_and_config 2024-03-27T21:42:07.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7fce2ffff640 10 monclient: build_initial_monmap 2024-03-27T21:42:07.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7fce2ffff640 10 monclient: monmap: 2024-03-27T21:42:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:07.407103+0000 2024-03-27T21:42:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:07.407103+0000 2024-03-27T21:42:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:07.448 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:07.448 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:07.448 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:07.448 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f28e959a640 1 -- 172.21.15.60:0/1239298856 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e402c980 con 0x7f28ec149540 2024-03-27T21:42:07.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f28e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f28e959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:07.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f28e959a640 10 monclient: dump: 2024-03-27T21:42:07.449 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.449 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.449 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.449 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.449 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.449 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.449 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.449 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.449 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.450 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f28e959a640 10 monclient: _finish_auth 0 2024-03-27T21:42:07.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f28e959a640 10 monclient: _check_auth_tickets 2024-03-27T21:42:07.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f28e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:07.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f28f08af640 5 monclient: authenticate success, global_id 5060 2024-03-27T21:42:07.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f28f08af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:07.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f28e959a640 1 -- 172.21.15.60:0/1239298856 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e402cae0 con 0x7f28ec149540 2024-03-27T21:42:07.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f28e959a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:07.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f28f08af640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:07.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.405+0000 7f28f08af640 4 set_mon_vals no callback set 2024-03-27T21:42:07.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b12ffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4b140ad7b0 0x7f4b1413b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b137fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4b14070da0 0x7f4b14071230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39452/0 (socket says 172.21.15.60:39452) 2024-03-27T21:42:07.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b137fe640 1 -- 172.21.15.60:0/191635678 learned_addr learned my addr 172.21.15.60:0/191635678 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:07.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b12ffd640 10 monclient(hunting): get_auth_request con 0x7f4b140ad7b0 auth_method 0 2024-03-27T21:42:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f28e959a640 1 -- 172.21.15.60:0/1239298856 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e402ce70 con 0x7f28ec149540 2024-03-27T21:42:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f28e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f28e959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f28e959a640 10 monclient: dump: 2024-03-27T21:42:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.490 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.490 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.490 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.490 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.490 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.490 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f9e9b59e640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:07.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f9e9b59e640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-03-27T21:42:07.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f9e9b59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9e9c0ae3d0 0x7f9e9c13d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f9e9b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f9e9c0ae3d0 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:07.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f9e9b59e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9e9c0ac530 0x7f9e9c0aa1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f9e9b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f9e9c0ac530 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:07.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f9e9b59e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9e9c0aa730 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:07.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f9e9b59e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9c0abd90 con 0x7f9e9c0aa730 2024-03-27T21:42:07.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f9e9b59e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f9e9b59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e9c0ae7b0 con 0x7f9e9c0ae3d0 2024-03-27T21:42:07.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f9e9b59e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f9e9b59e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e9c0ab970 con 0x7f9e9c0ac530 2024-03-27T21:42:07.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f9e9b59e640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:07.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f9e9b59e640 10 monclient(hunting): authenticate will time out at 968.627517s 2024-03-27T21:42:07.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b12ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7fa477381640 1 Processor -- start 2024-03-27T21:42:07.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b12ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b12ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b11ffb640 1 -- 172.21.15.60:0/191635678 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901045345 0 0) 0x7f4b1406eb90 con 0x7f4b1406e1e0 2024-03-27T21:42:07.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b11ffb640 10 monclient(hunting): _negotiate 2024-03-27T21:42:07.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b11ffb640 10 monclient(hunting): my global_id is 5284 2024-03-27T21:42:07.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b11ffb640 1 -- 172.21.15.60:0/191635678 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4afc005730 con 0x7f4b1406e1e0 2024-03-27T21:42:07.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b11ffb640 1 -- 172.21.15.60:0/191635678 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204789155 0 0) 0x7f4b14071940 con 0x7f4b14070da0 2024-03-27T21:42:07.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b11ffb640 10 monclient(hunting): _negotiate 2024-03-27T21:42:07.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b11ffb640 10 monclient(hunting): my global_id is 5367 2024-03-27T21:42:07.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f4b11ffb640 1 -- 172.21.15.60:0/191635678 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b1406eb90 con 0x7f4b14070da0 2024-03-27T21:42:07.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f28f08af640 10 monclient: shutdown 2024-03-27T21:42:07.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f28f08af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:07.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f28f08af640 1 -- 172.21.15.60:0/1239298856 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28ec149540 msgr2=0x7f28ec150190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.406+0000 7f28f08af640 1 --2- 172.21.15.60:0/1239298856 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28ec149540 0x7f28ec150190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f28e4009d90 tx=0x7f28e402bc50 comp rx=0 tx=0).stop 2024-03-27T21:42:07.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f28f08af640 1 -- 172.21.15.60:0/1239298856 shutdown_connections 2024-03-27T21:42:07.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7fa477381640 1 -- start start 2024-03-27T21:42:07.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7fa477381640 10 monclient: init 2024-03-27T21:42:07.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f28f08af640 1 --2- 172.21.15.60:0/1239298856 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28ec149540 0x7f28ec150190 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f28f08af640 1 --2- 172.21.15.60:0/1239298856 >> v2:172.21.15.60:3300/0 conn(0x7f28ec0b32c0 0x7f28ec149000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f28f08af640 1 -- 172.21.15.60:0/1239298856 >> 172.21.15.60:0/1239298856 conn(0x7f28ec0608e0 msgr2=0x7f28ec060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:07.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f4b12ffd640 10 monclient(hunting): handle_auth_done global_id 5063 payload 274 2024-03-27T21:42:07.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f4b11ffb640 1 -- 172.21.15.60:0/191635678 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2384579649 0 0) 0x7f4afc005730 con 0x7f4b1406e1e0 2024-03-27T21:42:07.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f4b12ffd640 1 -- 172.21.15.60:0/191635678 >> v1:172.21.15.191:6789/0 conn(0x7f4b1406e1e0 legacy=0x7f4b1413b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f4b12ffd640 1 -- 172.21.15.60:0/191635678 >> v1:172.21.15.60:6789/0 conn(0x7f4b14070da0 legacy=0x7f4b14071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f4b12ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:42:07.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f4b12ffd640 1 monclient: found mon.noname-b 2024-03-27T21:42:07.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f4b12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:07.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f4b12ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:42:07.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f4b12ffd640 1 -- 172.21.15.60:0/191635678 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b14070980 con 0x7f4b140ad7b0 2024-03-27T21:42:07.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f4b11ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:42:07.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f9e99d9b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9e9c0ac530 0x7f9e9c0aa1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f9e99d9b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9e9c0ac530 0x7f9e9c0aa1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:58906/0 (socket says 172.21.15.60:58906) 2024-03-27T21:42:07.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f9e99d9b640 1 -- 172.21.15.60:0/3216083965 learned_addr learned my addr 172.21.15.60:0/3216083965 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:07.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f9e99d9b640 10 monclient(hunting): get_auth_request con 0x7f9e9c0ac530 auth_method 0 2024-03-27T21:42:07.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f9e99d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f9e99d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f9e99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f9e9a59c640 1 --2- 172.21.15.60:0/3216083965 >> v2:172.21.15.60:3300/0 conn(0x7f9e9c0ae3d0 0x7f9e9c13d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f9e9a59c640 10 monclient(hunting): get_auth_request con 0x7f9e9c0ae3d0 auth_method 0 2024-03-27T21:42:07.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f9e9a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f9e9a59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f9e9a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f9e99d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f9e99d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7f9e99d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7f9e9a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7f9e9a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7f9e9a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fa477381640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:07.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fa477381640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-03-27T21:42:07.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7fce2ffff640 1 Processor -- start 2024-03-27T21:42:07.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7fce2ffff640 1 -- start start 2024-03-27T21:42:07.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.407+0000 7fce2ffff640 10 monclient: init 2024-03-27T21:42:07.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fa477381640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa468006a00 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:07.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fa477381640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa468031fc0 0x7fa4680c1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fa47511d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa468006a00 0x7fa4680349b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39454/0 (socket says 172.21.15.60:39454) 2024-03-27T21:42:07.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7f4d45c41640 10 monclient: get_monmap_and_config 2024-03-27T21:42:07.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7f4d45c41640 10 monclient: build_initial_monmap 2024-03-27T21:42:07.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7f4d45c41640 10 monclient: monmap: 2024-03-27T21:42:07.505 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:07.505 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.505 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:07.409153+0000 2024-03-27T21:42:07.505 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:07.409153+0000 2024-03-27T21:42:07.505 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:07.505 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.506 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:07.506 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:07.506 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:07.506 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:07.506 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:07.506 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fce2ffff640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:07.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fce2ffff640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-03-27T21:42:07.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7f9e9a59c640 10 monclient(hunting): handle_auth_done global_id 5370 payload 274 2024-03-27T21:42:07.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fce2ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fce280071b0 0x7fce280337d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fce2ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fce280071b0 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:07.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7f9e9a59c640 1 -- 172.21.15.60:0/3216083965 >> v2:172.21.15.191:3300/0 conn(0x7f9e9c0ac530 msgr2=0x7f9e9c0aa1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:07.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7f9e9a59c640 1 --2- 172.21.15.60:0/3216083965 >> v2:172.21.15.191:3300/0 conn(0x7f9e9c0ac530 0x7f9e9c0aa1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7f9e9a59c640 1 -- 172.21.15.60:0/3216083965 >> v1:172.21.15.191:6789/0 conn(0x7f9e9c0aa730 legacy=0x7f9e9c13aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7f9e9a59c640 10 monclient: _finish_hunting 0 2024-03-27T21:42:07.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7f9e9a59c640 1 monclient: found mon.noname-a 2024-03-27T21:42:07.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7f9e9a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:07.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7f9e9a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:07.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7f9e9a59c640 1 -- 172.21.15.60:0/3216083965 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9c0ac010 con 0x7f9e9c0ae3d0 2024-03-27T21:42:07.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fa477381640 10 monclient(hunting): picked mon.noname-b con 0x7fa468031fc0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:07.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fa47511d640 1 -- 172.21.15.60:0/2597202862 learned_addr learned my addr 172.21.15.60:0/2597202862 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:07.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fa477381640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa4680c3c50 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:07.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fce2ffff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce28035870 0x7fce28035f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fce2ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fce28035870 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:07.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fce2ffff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fce28033d10 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:07.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fa47491c640 1 --2- 172.21.15.60:0/2597202862 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa468031fc0 0x7fa4680c1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f4d45c41640 1 Processor -- start 2024-03-27T21:42:07.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f4d45c41640 1 -- start start 2024-03-27T21:42:07.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f4d45c41640 10 monclient: init 2024-03-27T21:42:07.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fce2ffff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce280c9c20 con 0x7fce28033d10 2024-03-27T21:42:07.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fce2ffff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fce2ffff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce28007590 con 0x7fce280071b0 2024-03-27T21:42:07.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fce2ffff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fce2ffff640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce28037370 con 0x7fce28035870 2024-03-27T21:42:07.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fce2ffff640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:07.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fce2ffff640 10 monclient(hunting): authenticate will time out at 968.629609s 2024-03-27T21:42:07.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fce2f7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fce28033d10 0x7fce280cf570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39456/0 (socket says 172.21.15.60:39456) 2024-03-27T21:42:07.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.408+0000 7fce2f7fe640 1 -- 172.21.15.60:0/3856657155 learned_addr learned my addr 172.21.15.60:0/3856657155 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:07.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2e7fc640 1 --2- 172.21.15.60:0/3856657155 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce28035870 0x7fce28035f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2e7fc640 10 monclient(hunting): get_auth_request con 0x7fce28035870 auth_method 0 2024-03-27T21:42:07.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2effd640 1 --2- 172.21.15.60:0/3856657155 >> v2:172.21.15.60:3300/0 conn(0x7fce280071b0 0x7fce280337d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2effd640 10 monclient(hunting): get_auth_request con 0x7fce280071b0 auth_method 0 2024-03-27T21:42:07.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f4b12ffd640 1 --2- 172.21.15.60:0/191635678 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4b140ad7b0 0x7f4b1413b1c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4b04009750 tx=0x7f4b0402b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19a3eb7c4ae32371 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f4b11ffb640 1 -- 172.21.15.60:0/191635678 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c0019f0 con 0x7f4b140ad7b0 2024-03-27T21:42:07.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f4b11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f4b11ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:07.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f4b11ffb640 10 monclient: dump: 2024-03-27T21:42:07.516 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.516 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.516 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.516 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.516 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.517 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.517 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.517 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.517 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.517 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f4b11ffb640 10 monclient: _finish_auth 0 2024-03-27T21:42:07.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f4b11ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:42:07.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f4b11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:07.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f4b11ffb640 1 -- 172.21.15.60:0/191635678 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0402e050 con 0x7f4b140ad7b0 2024-03-27T21:42:07.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f4b11ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:07.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f4b11ffb640 1 -- 172.21.15.60:0/191635678 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b04030730 con 0x7f4b140ad7b0 2024-03-27T21:42:07.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f4b11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f4b11ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:07.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f4b11ffb640 10 monclient: dump: 2024-03-27T21:42:07.518 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.518 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.518 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.519 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.519 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.519 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.519 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.519 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.519 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.519 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f4b19918640 5 monclient: authenticate success, global_id 5063 2024-03-27T21:42:07.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f4b19918640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:07.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f4b19918640 4 set_mon_vals no callback set 2024-03-27T21:42:07.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2e7fc640 10 monclient(hunting): handle_auth_done global_id 5066 payload 274 2024-03-27T21:42:07.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4d45c41640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:07.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4d45c41640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-03-27T21:42:07.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4d45c41640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d4006d560 0x7f4d4006d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4d45c41640 10 monclient(hunting): picked mon.noname-b con 0x7f4d4006d560 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:07.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4d45c41640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4d4006e6e0 0x7f4d40146290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4d45c41640 10 monclient(hunting): picked mon.noname-c con 0x7f4d4006e6e0 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:07.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4d45c41640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4d400ad6c0 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:07.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4d45c41640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d40145e00 con 0x7f4d400ad6c0 2024-03-27T21:42:07.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4d45c41640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4d45c41640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d4006f7b0 con 0x7f4d4006e6e0 2024-03-27T21:42:07.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4d45c41640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4d45c41640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d400aeaf0 con 0x7f4d4006d560 2024-03-27T21:42:07.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4d45c41640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:07.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4d45c41640 10 monclient(hunting): authenticate will time out at 968.631182s 2024-03-27T21:42:07.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4b19918640 10 monclient: shutdown 2024-03-27T21:42:07.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4b19918640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:07.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4d3ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4d400ad6c0 0x7f4d400ae3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39468/0 (socket says 172.21.15.60:39468) 2024-03-27T21:42:07.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4d3ffff640 1 -- 172.21.15.60:0/1991752252 learned_addr learned my addr 172.21.15.60:0/1991752252 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:07.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4b19918640 1 -- 172.21.15.60:0/191635678 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4b140ad7b0 msgr2=0x7f4b1413b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4b19918640 1 --2- 172.21.15.60:0/191635678 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4b140ad7b0 0x7f4b1413b1c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4b04009750 tx=0x7f4b0402b730 comp rx=0 tx=0).stop 2024-03-27T21:42:07.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4d3f7fe640 1 --2- 172.21.15.60:0/1991752252 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d4006d560 0x7f4d4006d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4d3f7fe640 10 monclient(hunting): get_auth_request con 0x7f4d4006d560 auth_method 0 2024-03-27T21:42:07.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4d3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4d3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4d3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fa477381640 1 -- 172.21.15.60:0/2597202862 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa468031800 con 0x7fa468006a00 2024-03-27T21:42:07.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fa477381640 1 -- 172.21.15.60:0/2597202862 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa468031aa0 con 0x7fa4680c3c50 2024-03-27T21:42:07.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fa477381640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fa477381640 1 -- 172.21.15.60:0/2597202862 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4680291f0 con 0x7fa468031fc0 2024-03-27T21:42:07.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fa477381640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:07.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fa477381640 10 monclient(hunting): authenticate will time out at 968.630681s 2024-03-27T21:42:07.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7fa467fff640 1 -- 172.21.15.60:0/2597202862 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732353343 0 0) 0x7fa468031800 con 0x7fa468006a00 2024-03-27T21:42:07.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7fa467fff640 10 monclient(hunting): _negotiate 2024-03-27T21:42:07.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7fa467fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7fa467fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7fa467fff640 10 monclient(hunting): my global_id is 5379 2024-03-27T21:42:07.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7fa467fff640 1 -- 172.21.15.60:0/2597202862 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa45c007930 con 0x7fa468006a00 2024-03-27T21:42:07.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7fa47491c640 10 monclient(hunting): get_auth_request con 0x7fa468031fc0 auth_method 0 2024-03-27T21:42:07.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7fa47491c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4b19918640 1 -- 172.21.15.60:0/191635678 shutdown_connections 2024-03-27T21:42:07.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4b19918640 1 --2- 172.21.15.60:0/191635678 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4b140ad7b0 0x7f4b1413b1c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f4b19918640 1 -- 172.21.15.60:0/191635678 >> 172.21.15.60:0/191635678 conn(0x7f4b140b3620 msgr2=0x7f4b14060700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:07.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7fa47491c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7fa47491c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7fa467fff640 1 -- 172.21.15.60:0/2597202862 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4231057909 0 0) 0x7fa45c007930 con 0x7fa468006a00 2024-03-27T21:42:07.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7fa467fff640 1 -- 172.21.15.60:0/2597202862 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa468031fc0 msgr2=0x7fa4680c1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:07.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7fa467fff640 1 --2- 172.21.15.60:0/2597202862 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa468031fc0 0x7fa4680c1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f9e9a59c640 1 --2- 172.21.15.60:0/3216083965 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e9c0ae3d0 0x7f9e9c13d7b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0b3b00 tx=0x7f9e94034860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80eb397022653a28 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2e7fc640 1 -- 172.21.15.60:0/3856657155 >> v2:172.21.15.60:3300/0 conn(0x7fce280071b0 msgr2=0x7fce280337d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:07.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2e7fc640 1 --2- 172.21.15.60:0/3856657155 >> v2:172.21.15.60:3300/0 conn(0x7fce280071b0 0x7fce280337d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2e7fc640 1 -- 172.21.15.60:0/3856657155 >> v1:172.21.15.60:6789/0 conn(0x7fce28033d10 legacy=0x7fce280cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2e7fc640 10 monclient: _finish_hunting 0 2024-03-27T21:42:07.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2e7fc640 1 monclient: found mon.noname-b 2024-03-27T21:42:07.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:07.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2dffb640 1 -- 172.21.15.60:0/3856657155 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009188998 0 0) 0x7fce280c9c20 con 0x7fce28033d10 2024-03-27T21:42:07.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:42:07.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2e7fc640 1 -- 172.21.15.60:0/3856657155 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce28037650 con 0x7fce28035870 2024-03-27T21:42:07.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7fce2dffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:42:07.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7fa467fff640 1 -- 172.21.15.60:0/2597202862 >> v1:172.21.15.191:6789/0 conn(0x7fa4680c3c50 legacy=0x7fa4680c40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7fa467fff640 10 monclient: _finish_hunting 0 2024-03-27T21:42:07.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7fa467fff640 1 monclient: found mon.noname-a-legacy 2024-03-27T21:42:07.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f9e9959a640 1 -- 172.21.15.60:0/3216083965 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9400d4e0 con 0x7f9e9c0ae3d0 2024-03-27T21:42:07.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f9e9959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f9e9959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:07.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f9e9959a640 10 monclient: dump: 2024-03-27T21:42:07.531 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.531 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.532 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.532 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.532 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.532 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.532 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.532 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.532 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.532 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f9e9959a640 10 monclient: _finish_auth 0 2024-03-27T21:42:07.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f9e9959a640 10 monclient: _check_auth_tickets 2024-03-27T21:42:07.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f9e9959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:07.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f9e9b59e640 5 monclient: authenticate success, global_id 5370 2024-03-27T21:42:07.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f9e9b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:07.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f9e9959a640 1 -- 172.21.15.60:0/3216083965 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9400b740 con 0x7f9e9c0ae3d0 2024-03-27T21:42:07.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f9e9959a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:07.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f9e9b59e640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:07.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f9e9b59e640 4 set_mon_vals no callback set 2024-03-27T21:42:07.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 1 -- 172.21.15.60:0/1991752252 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860347700 0 0) 0x7f4d40145e00 con 0x7f4d400ad6c0 2024-03-27T21:42:07.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 10 monclient(hunting): _negotiate 2024-03-27T21:42:07.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 10 monclient(hunting): my global_id is 5382 2024-03-27T21:42:07.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 1 -- 172.21.15.60:0/1991752252 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d28003520 con 0x7f4d400ad6c0 2024-03-27T21:42:07.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3effd640 1 --2- 172.21.15.60:0/1991752252 >> v2:172.21.15.191:3300/0 conn(0x7f4d4006e6e0 0x7f4d40146290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3effd640 10 monclient(hunting): get_auth_request con 0x7f4d4006e6e0 auth_method 0 2024-03-27T21:42:07.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 1 -- 172.21.15.60:0/1991752252 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3707056296 0 0) 0x7f4d28003520 con 0x7f4d400ad6c0 2024-03-27T21:42:07.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f9e9959a640 1 -- 172.21.15.60:0/3216083965 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9400bad0 con 0x7f9e9c0ae3d0 2024-03-27T21:42:07.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f9e9959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f9e9959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:07.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f9e9959a640 10 monclient: dump: 2024-03-27T21:42:07.536 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.537 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.537 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.537 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.537 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.537 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.537 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.537 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.537 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.537 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f9e9b59e640 10 monclient: shutdown 2024-03-27T21:42:07.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f9e9b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:07.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f9e9b59e640 1 -- 172.21.15.60:0/3216083965 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e9c0ae3d0 msgr2=0x7f9e9c13d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f9e9b59e640 1 --2- 172.21.15.60:0/3216083965 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e9c0ae3d0 0x7f9e9c13d7b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0b3b00 tx=0x7f9e94034860 comp rx=0 tx=0).stop 2024-03-27T21:42:07.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 1 -- 172.21.15.60:0/1991752252 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d4006d560 msgr2=0x7f4d4006d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:07.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 1 --2- 172.21.15.60:0/1991752252 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d4006d560 0x7f4d4006d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 1 -- 172.21.15.60:0/1991752252 >> v2:172.21.15.191:3300/0 conn(0x7f4d4006e6e0 msgr2=0x7f4d40146290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:07.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 1 --2- 172.21.15.60:0/1991752252 >> v2:172.21.15.191:3300/0 conn(0x7f4d4006e6e0 0x7f4d40146290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 10 monclient: _finish_hunting 0 2024-03-27T21:42:07.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 1 monclient: found mon.noname-a-legacy 2024-03-27T21:42:07.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:07.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.60:6789/0 2024-03-27T21:42:07.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 1 -- 172.21.15.60:0/1991752252 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d400aec60 con 0x7f4d400ad6c0 2024-03-27T21:42:07.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 10 monclient: _finish_auth 0 2024-03-27T21:42:07.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:42:07.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:07.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d45c41640 5 monclient: authenticate success, global_id 5382 2024-03-27T21:42:07.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d45c41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:07.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 1 -- 172.21.15.60:0/1991752252 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2933500084 0 0) 0x7f4d380019f0 con 0x7f4d400ad6c0 2024-03-27T21:42:07.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-03-27T21:42:07.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 10 monclient: dump: 2024-03-27T21:42:07.540 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.540 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.541 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.541 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.541 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.541 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.541 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.541 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.541 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.541 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] but i'm connected to v1:172.21.15.60:6789/0, reconnecting 2024-03-27T21:42:07.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:07.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 1 -- 172.21.15.60:0/1991752252 >> v1:172.21.15.60:6789/0 conn(0x7f4d400ad6c0 legacy=0x7f4d400ae3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:42:07.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 1 --2- 172.21.15.60:0/1991752252 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d28003970 0x7f4d28003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 10 monclient(hunting): picked mon.b con 0x7f4d28003970 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:07.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 1 --2- 172.21.15.60:0/1991752252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d280047c0 0x7f4d28004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 10 monclient(hunting): picked mon.a con 0x7f4d280047c0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:07.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 1 --2- 172.21.15.60:0/1991752252 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d28005180 0x7f4d2800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7fce2e7fc640 1 --2- 172.21.15.60:0/3856657155 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce28035870 0x7fce28035f00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fce200097a0 tx=0x7fce2002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72590efca06b4806 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7fa467fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:07.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7fa467fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.60:6789/0 2024-03-27T21:42:07.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7fa467fff640 1 -- 172.21.15.60:0/2597202862 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4680333c0 con 0x7fa468006a00 2024-03-27T21:42:07.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7fa467fff640 10 monclient: _finish_auth 0 2024-03-27T21:42:07.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7fa467fff640 10 monclient: _check_auth_tickets 2024-03-27T21:42:07.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7fa467fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:07.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7fa467fff640 1 -- 172.21.15.60:0/2597202862 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2933500084 0 0) 0x7fa46c001670 con 0x7fa468006a00 2024-03-27T21:42:07.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7fa467fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7fa467fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-03-27T21:42:07.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7fa467fff640 10 monclient: dump: 2024-03-27T21:42:07.544 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.544 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.544 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.545 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.545 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.545 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.545 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.545 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.545 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.545 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7fa467fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] but i'm connected to v1:172.21.15.60:6789/0, reconnecting 2024-03-27T21:42:07.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7fa467fff640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:07.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7fa467fff640 1 -- 172.21.15.60:0/2597202862 >> v1:172.21.15.60:6789/0 conn(0x7fa468006a00 legacy=0x7fa4680349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4b19918640 1 -- 172.21.15.60:0/191635678 shutdown_connections 2024-03-27T21:42:07.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f28f08af640 1 -- 172.21.15.60:0/1239298856 shutdown_connections 2024-03-27T21:42:07.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.409+0000 7f28f08af640 1 -- 172.21.15.60:0/1239298856 wait complete. 2024-03-27T21:42:07.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f28f08af640 10 monclient: build_initial_monmap 2024-03-27T21:42:07.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.410+0000 7f28f08af640 10 monclient: monmap: 2024-03-27T21:42:07.548 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:07.548 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:07.548 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:07.548 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:07.548 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:07.548 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:07.548 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:07.549 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:07.549 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:07.549 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f28f08af640 1 librados: starting msgr at 2024-03-27T21:42:07.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f28f08af640 1 librados: starting objecter 2024-03-27T21:42:07.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f28f08af640 1 Processor -- start 2024-03-27T21:42:07.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f28f08af640 1 -- start start 2024-03-27T21:42:07.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f28f08af640 1 librados: setting wanted keys 2024-03-27T21:42:07.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f28f08af640 1 librados: calling monclient init 2024-03-27T21:42:07.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f28f08af640 10 monclient: init 2024-03-27T21:42:07.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28f08af640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:07.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28f08af640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:42:07.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28f08af640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28ec0b0ec0 0x7f28ec13e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28f08af640 10 monclient(hunting): picked mon.noname-a con 0x7f28ec0b0ec0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:07.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28f08af640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28ec0b32c0 0x7f28ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28f08af640 10 monclient(hunting): picked mon.noname-b con 0x7f28ec0b32c0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:07.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28f08af640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28ec149540 0x7f28ec1393f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28f08af640 10 monclient(hunting): picked mon.noname-c con 0x7f28ec149540 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:07.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28f08af640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28f08af640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ec13d4f0 con 0x7f28ec0b0ec0 2024-03-27T21:42:07.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28f08af640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28f08af640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ec129f10 con 0x7f28ec0b32c0 2024-03-27T21:42:07.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28f08af640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28f08af640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28ec06f070 con 0x7f28ec149540 2024-03-27T21:42:07.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28f08af640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:07.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28f08af640 10 monclient(hunting): authenticate will time out at 968.633337s 2024-03-27T21:42:07.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28e9d9b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28ec0b32c0 0x7f28ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28e9d9b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28ec0b32c0 0x7f28ec06d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:58932/0 (socket says 172.21.15.60:58932) 2024-03-27T21:42:07.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28e9d9b640 1 -- 172.21.15.60:0/1098457953 learned_addr learned my addr 172.21.15.60:0/1098457953 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:07.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28e9d9b640 10 monclient(hunting): get_auth_request con 0x7f28ec0b32c0 auth_method 0 2024-03-27T21:42:07.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28e9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28e9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28ead9d640 1 --2- 172.21.15.60:0/1098457953 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28ec149540 0x7f28ec1393f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28ead9d640 10 monclient(hunting): get_auth_request con 0x7f28ec149540 auth_method 0 2024-03-27T21:42:07.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28ead9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28e9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28e9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f28e9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7fa467fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:42:07.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7fa467fff640 1 --2- 172.21.15.60:0/2597202862 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa468031fc0 0x7fa45c007db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47511d640 1 --2- 172.21.15.60:0/2597202862 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa468031fc0 0x7fa45c007db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7f28ea59c640 1 --2- 172.21.15.60:0/1098457953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28ec0b0ec0 0x7f28ec13e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7f28ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7f28ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7f28ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 10 monclient(hunting): picked mon.c con 0x7f4d28005180 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:07.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 1 -- 172.21.15.60:0/1991752252 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4006f7b0 con 0x7f4d280047c0 2024-03-27T21:42:07.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 1 -- 172.21.15.60:0/1991752252 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d400aeaf0 con 0x7f4d28003970 2024-03-27T21:42:07.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 1 -- 172.21.15.60:0/1991752252 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2801ced0 con 0x7f4d28005180 2024-03-27T21:42:07.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:07.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-27T21:42:07.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d45c41640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:07.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.411+0000 7f4d3f7fe640 1 --2- 172.21.15.60:0/1991752252 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d4006d560 0x7f4d4006d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:07.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3effd640 1 --2- 172.21.15.60:0/1991752252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d280047c0 0x7f4d28004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3ffff640 1 --2- 172.21.15.60:0/1991752252 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d28005180 0x7f4d2800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3effd640 10 monclient(hunting): get_auth_request con 0x7f4d280047c0 auth_method 0 2024-03-27T21:42:07.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:07.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3ffff640 10 monclient(hunting): get_auth_request con 0x7f4d28005180 auth_method 0 2024-03-27T21:42:07.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:07.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:42:07.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3f7fe640 1 --2- 172.21.15.60:0/1991752252 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d28003970 0x7f4d28003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3f7fe640 10 monclient(hunting): get_auth_request con 0x7f4d28003970 auth_method 0 2024-03-27T21:42:07.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:07.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3effd640 10 monclient(hunting): handle_auth_done global_id 5382 payload 293 2024-03-27T21:42:07.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3effd640 1 -- 172.21.15.60:0/1991752252 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d28005180 msgr2=0x7f4d2800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:07.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3effd640 1 --2- 172.21.15.60:0/1991752252 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d28005180 0x7f4d2800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3effd640 1 -- 172.21.15.60:0/1991752252 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d28003970 msgr2=0x7f4d28003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:07.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3effd640 1 --2- 172.21.15.60:0/1991752252 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d28003970 0x7f4d28003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3effd640 10 monclient: _finish_hunting 0 2024-03-27T21:42:07.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3effd640 1 monclient: found mon.a 2024-03-27T21:42:07.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:07.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3effd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:07.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3effd640 1 -- 172.21.15.60:0/1991752252 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d400aec60 con 0x7f4d280047c0 2024-03-27T21:42:07.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3effd640 10 monclient: _finish_auth 0 2024-03-27T21:42:07.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3effd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:07.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4d3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:07.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7f28ea59c640 10 monclient(hunting): get_auth_request con 0x7f28ec0b0ec0 auth_method 0 2024-03-27T21:42:07.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7f28ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7f28ea59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7f28ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7f4d3f7fe640 1 --2- 172.21.15.60:0/1991752252 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d28003970 0x7f4d28003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:42:07.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fce2dffb640 1 -- 172.21.15.60:0/3856657155 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce20030800 con 0x7fce28035870 2024-03-27T21:42:07.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fce2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fce2dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:07.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fce2dffb640 10 monclient: dump: 2024-03-27T21:42:07.562 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.562 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.562 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.562 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.563 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.563 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.563 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.563 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.563 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.563 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fce2dffb640 10 monclient: _finish_auth 0 2024-03-27T21:42:07.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fce2dffb640 10 monclient: _check_auth_tickets 2024-03-27T21:42:07.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fce2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:07.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fce2dffb640 1 -- 172.21.15.60:0/3856657155 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce20030960 con 0x7fce28035870 2024-03-27T21:42:07.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fce2dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:07.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fce2dffb640 1 -- 172.21.15.60:0/3856657155 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce20030cf0 con 0x7fce28035870 2024-03-27T21:42:07.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fce2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fce2dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:07.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fce2dffb640 10 monclient: dump: 2024-03-27T21:42:07.564 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.564 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.565 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.565 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.565 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.565 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.565 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.565 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.565 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.565 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fce2ffff640 5 monclient: authenticate success, global_id 5066 2024-03-27T21:42:07.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fce2ffff640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:07.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fce2ffff640 4 set_mon_vals no callback set 2024-03-27T21:42:07.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa467fff640 10 monclient(hunting): picked mon.c con 0x7fa468031fc0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:07.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa467fff640 1 --2- 172.21.15.60:0/2597202862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa45c0083a0 0x7fa45c008bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa467fff640 10 monclient(hunting): picked mon.b con 0x7fa45c0083a0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:07.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa467fff640 1 --2- 172.21.15.60:0/2597202862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa45c007460 0x7fa45c009110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa467fff640 10 monclient(hunting): picked mon.a con 0x7fa45c007460 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:07.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa467fff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa467fff640 1 -- 172.21.15.60:0/2597202862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4680291f0 con 0x7fa45c007460 2024-03-27T21:42:07.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa467fff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa467fff640 1 -- 172.21.15.60:0/2597202862 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c009860 con 0x7fa45c0083a0 2024-03-27T21:42:07.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa467fff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa467fff640 1 -- 172.21.15.60:0/2597202862 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c009a20 con 0x7fa468031fc0 2024-03-27T21:42:07.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa467fff640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:07.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa467fff640 10 monclient(hunting): _finish_auth 0 2024-03-27T21:42:07.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47511d640 10 monclient(hunting): get_auth_request con 0x7fa468031fc0 auth_method 0 2024-03-27T21:42:07.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47511d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47511d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47511d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:07.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47491c640 1 --2- 172.21.15.60:0/2597202862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa45c0083a0 0x7fa45c008bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47491c640 10 monclient(hunting): get_auth_request con 0x7fa45c0083a0 auth_method 0 2024-03-27T21:42:07.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47491c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47491c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47491c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:07.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47491c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47491c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47491c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:42:07.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47591e640 1 --2- 172.21.15.60:0/2597202862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa45c007460 0x7fa45c009110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47591e640 10 monclient(hunting): get_auth_request con 0x7fa45c007460 auth_method 0 2024-03-27T21:42:07.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47591e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47591e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47591e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:07.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47491c640 10 monclient(hunting): handle_auth_done global_id 5379 payload 293 2024-03-27T21:42:07.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47491c640 1 -- 172.21.15.60:0/2597202862 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa468031fc0 msgr2=0x7fa45c007db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:07.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47491c640 1 --2- 172.21.15.60:0/2597202862 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa468031fc0 0x7fa45c007db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47491c640 1 -- 172.21.15.60:0/2597202862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa45c007460 msgr2=0x7fa45c009110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:07.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47491c640 1 --2- 172.21.15.60:0/2597202862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa45c007460 0x7fa45c009110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47491c640 10 monclient: _finish_hunting 0 2024-03-27T21:42:07.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47491c640 1 monclient: found mon.b 2024-03-27T21:42:07.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47491c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:07.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47491c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:07.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47491c640 1 -- 172.21.15.60:0/2597202862 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa45c007930 con 0x7fa45c0083a0 2024-03-27T21:42:07.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47491c640 10 monclient: _finish_auth 0 2024-03-27T21:42:07.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47491c640 10 monclient: _check_auth_tickets 2024-03-27T21:42:07.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7fa47491c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:07.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7f9e9b59e640 1 -- 172.21.15.60:0/3216083965 shutdown_connections 2024-03-27T21:42:07.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7f9e9b59e640 1 --2- 172.21.15.60:0/3216083965 >> v2:172.21.15.191:3300/0 conn(0x7f9e9c0ac530 0x7f9e9c0aa1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7f9e9b59e640 1 --2- 172.21.15.60:0/3216083965 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e9c0ae3d0 0x7f9e9c13d7b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7f9e9b59e640 1 -- 172.21.15.60:0/3216083965 >> 172.21.15.60:0/3216083965 conn(0x7f9e9c060b20 msgr2=0x7f9e9c060f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:07.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.412+0000 7f4b19918640 1 -- 172.21.15.60:0/191635678 wait complete. 2024-03-27T21:42:07.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7f4b19918640 10 monclient: build_initial_monmap 2024-03-27T21:42:07.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7f4b19918640 10 monclient: monmap: 2024-03-27T21:42:07.573 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:07.573 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:07.573 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:07.573 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:07.573 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:07.573 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:07.573 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:07.574 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:07.574 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:07.574 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f4b19918640 1 librados: starting msgr at 2024-03-27T21:42:07.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f4b19918640 1 librados: starting objecter 2024-03-27T21:42:07.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f4b19918640 1 Processor -- start 2024-03-27T21:42:07.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f28ead9d640 10 monclient(hunting): handle_auth_done global_id 5075 payload 824 2024-03-27T21:42:07.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f28ead9d640 1 -- 172.21.15.60:0/1098457953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28ec0b32c0 msgr2=0x7f28ec06d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f28ead9d640 1 --2- 172.21.15.60:0/1098457953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28ec0b32c0 0x7f28ec06d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f28ead9d640 1 -- 172.21.15.60:0/1098457953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28ec0b0ec0 msgr2=0x7f28ec13e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f28ead9d640 1 --2- 172.21.15.60:0/1098457953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28ec0b0ec0 0x7f28ec13e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f28ead9d640 10 monclient: _finish_hunting 0 2024-03-27T21:42:07.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f28ead9d640 1 monclient: found mon.noname-c 2024-03-27T21:42:07.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f28ead9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:07.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f28ead9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:07.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f28ead9d640 1 -- 172.21.15.60:0/1098457953 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28dc0094d0 con 0x7f28ec149540 2024-03-27T21:42:07.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7fce2ffff640 10 monclient: shutdown 2024-03-27T21:42:07.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7fce2ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:07.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7fce2ffff640 1 -- 172.21.15.60:0/3856657155 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce28035870 msgr2=0x7fce28035f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7fce2ffff640 1 --2- 172.21.15.60:0/3856657155 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce28035870 0x7fce28035f00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fce200097a0 tx=0x7fce2002b940 comp rx=0 tx=0).stop 2024-03-27T21:42:07.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7fa477381640 5 monclient: authenticate success, global_id 5379 2024-03-27T21:42:07.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7fa477381640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:07.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f9e9b59e640 1 -- 172.21.15.60:0/3216083965 shutdown_connections 2024-03-27T21:42:07.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7fce2ffff640 1 -- 172.21.15.60:0/3856657155 shutdown_connections 2024-03-27T21:42:07.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7fce2ffff640 1 --2- 172.21.15.60:0/3856657155 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce28035870 0x7fce28035f00 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7fce2ffff640 1 --2- 172.21.15.60:0/3856657155 >> v2:172.21.15.60:3300/0 conn(0x7fce280071b0 0x7fce280337d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7fce2ffff640 1 -- 172.21.15.60:0/3856657155 >> 172.21.15.60:0/3856657155 conn(0x7fce28006210 msgr2=0x7fce28006620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:07.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f9e9b59e640 1 -- 172.21.15.60:0/3216083965 wait complete. 2024-03-27T21:42:07.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.413+0000 7f4d3effd640 1 --2- 172.21.15.60:0/1991752252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d280047c0 0x7f4d28004c40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4d34009ad0 tx=0x7f4d280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d7d5fdaee5f060 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f4d3e7fc640 1 -- 172.21.15.60:0/1991752252 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3402c6d0 con 0x7f4d280047c0 2024-03-27T21:42:07.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f4d3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f4d3e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:07.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f4d3e7fc640 10 monclient: dump: 2024-03-27T21:42:07.579 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.580 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.580 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.580 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.580 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.580 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.580 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.580 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.580 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.580 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f4d45c41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:07.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f4d3e7fc640 1 -- 172.21.15.60:0/1991752252 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3402c830 con 0x7f4d280047c0 2024-03-27T21:42:07.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f4d3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:07.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f4d3e7fc640 1 -- 172.21.15.60:0/1991752252 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3402cbe0 con 0x7f4d280047c0 2024-03-27T21:42:07.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f4d45c41640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:07.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f4d45c41640 4 set_mon_vals no callback set 2024-03-27T21:42:07.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f4d3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f4d3e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:07.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f4d3e7fc640 10 monclient: dump: 2024-03-27T21:42:07.582 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.582 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.582 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.582 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.582 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.582 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.582 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.583 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.583 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.583 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f4b19918640 1 -- start start 2024-03-27T21:42:07.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f4b19918640 1 librados: setting wanted keys 2024-03-27T21:42:07.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f4b19918640 1 librados: calling monclient init 2024-03-27T21:42:07.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.414+0000 7f4b19918640 10 monclient: init 2024-03-27T21:42:07.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7fa47491c640 1 --2- 172.21.15.60:0/2597202862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa45c0083a0 0x7fa45c008bd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa46000ba10 tx=0x7fa45c009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45dc8df60e67395e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7fce2ffff640 1 -- 172.21.15.60:0/3856657155 shutdown_connections 2024-03-27T21:42:07.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f28ead9d640 1 --2- 172.21.15.60:0/1098457953 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28ec149540 0x7f28ec1393f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f28e4033000 tx=0x7f28e4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f28daffd640 10 client.?.objecter ms_handle_connect 0x7f28ec149540 2024-03-27T21:42:07.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f28daffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:07.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7fce2ffff640 1 -- 172.21.15.60:0/3856657155 wait complete. 2024-03-27T21:42:07.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7fa467fff640 1 -- 172.21.15.60:0/2597202862 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa460032620 con 0x7fa45c0083a0 2024-03-27T21:42:07.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7fa467fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7fa467fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:42:07.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7fa467fff640 10 monclient: dump: 2024-03-27T21:42:07.586 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.586 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.589 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.589 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.589 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.589 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.589 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.589 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.590 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.590 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7fa467fff640 1 -- 172.21.15.60:0/2597202862 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4600327a0 con 0x7fa45c0083a0 2024-03-27T21:42:07.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7fa477381640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:07.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7fa467fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:07.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7fa467fff640 1 -- 172.21.15.60:0/2597202862 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa460032b50 con 0x7fa45c0083a0 2024-03-27T21:42:07.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7fa467fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7fa467fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:42:07.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7fa467fff640 10 monclient: dump: 2024-03-27T21:42:07.591 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.591 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.591 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.591 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.591 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.591 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.591 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.592 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.592 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.592 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7fa477381640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:07.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7fa477381640 4 set_mon_vals no callback set 2024-03-27T21:42:07.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f4b19918640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:07.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f4b19918640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:42:07.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f4b19918640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b14070da0 0x7f4b14138100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f4b19918640 10 monclient(hunting): picked mon.noname-b con 0x7f4b14070da0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:07.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f4b19918640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4b1412b9a0 0x7f4b141e3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f9e9b59e640 10 monclient: build_initial_monmap 2024-03-27T21:42:07.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f4b19918640 10 monclient(hunting): picked mon.noname-c con 0x7f4b1412b9a0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:07.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f4b19918640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b141e3890 0x7f4b14137410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f9e9b59e640 10 monclient: monmap: 2024-03-27T21:42:07.595 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:07.595 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:07.595 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:07.595 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:07.595 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:07.595 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:07.595 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:07.595 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:07.596 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:07.596 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f4b19918640 10 monclient(hunting): picked mon.noname-a con 0x7f4b141e3890 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:07.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f4b19918640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f4b19918640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1412c560 con 0x7f4b141e3890 2024-03-27T21:42:07.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f4b19918640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f4b19918640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b140aba60 con 0x7f4b14070da0 2024-03-27T21:42:07.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f4b19918640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f4b19918640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b141e3d10 con 0x7f4b1412b9a0 2024-03-27T21:42:07.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f4b19918640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:07.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7fa477381640 10 monclient: shutdown 2024-03-27T21:42:07.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7fa477381640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:07.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7fa477381640 1 -- 172.21.15.60:0/2597202862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa45c0083a0 msgr2=0x7fa45c008bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7fa477381640 1 --2- 172.21.15.60:0/2597202862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa45c0083a0 0x7fa45c008bd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa46000ba10 tx=0x7fa45c009c00 comp rx=0 tx=0).stop 2024-03-27T21:42:07.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7fa47591e640 1 -- 172.21.15.60:0/2597202862 reap_dead start 2024-03-27T21:42:07.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f4b19918640 10 monclient(hunting): authenticate will time out at 968.636698s 2024-03-27T21:42:07.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4b127fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4b1412b9a0 0x7f4b141e3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f9e9b59e640 1 librados: starting msgr at 2024-03-27T21:42:07.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f9e9b59e640 1 librados: starting objecter 2024-03-27T21:42:07.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f28daffd640 1 -- 172.21.15.60:0/1098457953 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e4034c00 con 0x7f28ec149540 2024-03-27T21:42:07.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f28daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f28daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:07.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f28daffd640 10 monclient: dump: 2024-03-27T21:42:07.601 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.601 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.601 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.601 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.601 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.601 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.601 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.601 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.602 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.602 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f28daffd640 10 monclient: _finish_auth 0 2024-03-27T21:42:07.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f28daffd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:07.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f28daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:07.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f28daffd640 1 -- 172.21.15.60:0/1098457953 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e4034d60 con 0x7f28ec149540 2024-03-27T21:42:07.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f28daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:07.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.415+0000 7f28dbfff640 4 set_mon_vals no callback set 2024-03-27T21:42:07.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f28daffd640 1 -- 172.21.15.60:0/1098457953 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e4040690 con 0x7f28ec149540 2024-03-27T21:42:07.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f28daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f28daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:07.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f28daffd640 10 monclient: dump: 2024-03-27T21:42:07.603 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.603 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.603 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.603 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.603 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.604 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.604 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.604 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.604 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.604 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7fce2ffff640 10 monclient: build_initial_monmap 2024-03-27T21:42:07.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7fce2ffff640 10 monclient: monmap: 2024-03-27T21:42:07.605 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:07.605 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:07.605 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:07.605 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:07.605 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:07.605 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:07.605 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:07.605 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:07.605 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:07.606 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4b127fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4b1412b9a0 0x7f4b141e3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34384/0 (socket says 172.21.15.60:34384) 2024-03-27T21:42:07.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4b127fc640 1 -- 172.21.15.60:0/947527844 learned_addr learned my addr 172.21.15.60:0/947527844 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:07.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4b127fc640 10 monclient(hunting): get_auth_request con 0x7f4b1412b9a0 auth_method 0 2024-03-27T21:42:07.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4b127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4b127fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4b127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4b127fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4b127fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4b127fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7fa477381640 1 -- 172.21.15.60:0/2597202862 shutdown_connections 2024-03-27T21:42:07.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7fa477381640 1 -- 172.21.15.60:0/2597202862 >> 172.21.15.60:0/2597202862 conn(0x7fa468005ee0 msgr2=0x7fa4680062f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:07.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f9e9b59e640 1 Processor -- start 2024-03-27T21:42:07.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4b12ffd640 1 --2- 172.21.15.60:0/947527844 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b14070da0 0x7f4b14138100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f9e9b59e640 1 -- start start 2024-03-27T21:42:07.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4d45c41640 10 monclient: shutdown 2024-03-27T21:42:07.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4d45c41640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:07.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4d45c41640 1 -- 172.21.15.60:0/1991752252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d280047c0 msgr2=0x7f4d28004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4d45c41640 1 --2- 172.21.15.60:0/1991752252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d280047c0 0x7f4d28004c40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4d34009ad0 tx=0x7f4d280057f0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4d3ffff640 1 -- 172.21.15.60:0/1991752252 reap_dead start 2024-03-27T21:42:07.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4d45c41640 1 -- 172.21.15.60:0/1991752252 shutdown_connections 2024-03-27T21:42:07.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4d45c41640 1 -- 172.21.15.60:0/1991752252 >> 172.21.15.60:0/1991752252 conn(0x7f4d400b2ea0 msgr2=0x7f4d400b32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:07.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4b12ffd640 10 monclient(hunting): get_auth_request con 0x7f4b14070da0 auth_method 0 2024-03-27T21:42:07.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4b12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4b12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.416+0000 7f4b12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f9e9b59e640 1 librados: setting wanted keys 2024-03-27T21:42:07.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f9e9b59e640 1 librados: calling monclient init 2024-03-27T21:42:07.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f9e9b59e640 10 monclient: init 2024-03-27T21:42:07.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7fa477381640 1 -- 172.21.15.60:0/2597202862 shutdown_connections 2024-03-27T21:42:07.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f28f08af640 5 monclient: authenticate success, global_id 5075 2024-03-27T21:42:07.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f28f08af640 10 monclient: _renew_subs 2024-03-27T21:42:07.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f28f08af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:07.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f28f08af640 1 -- 172.21.15.60:0/1098457953 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e4009a70 con 0x7f28ec149540 2024-03-27T21:42:07.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f28f08af640 10 client.5075.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:07.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f28f08af640 10 monclient: _renew_subs 2024-03-27T21:42:07.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f28f08af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:07.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f28f08af640 1 -- 172.21.15.60:0/1098457953 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28ec13cca0 con 0x7f28ec149540 2024-03-27T21:42:07.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f28f08af640 1 librados: init done 2024-03-27T21:42:07.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f4d45c41640 1 -- 172.21.15.60:0/1991752252 shutdown_connections 2024-03-27T21:42:07.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f4d45c41640 1 -- 172.21.15.60:0/1991752252 wait complete. 2024-03-27T21:42:07.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f4b137fe640 1 --2- 172.21.15.60:0/947527844 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b141e3890 0x7f4b14137410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f4b127fc640 10 monclient(hunting): handle_auth_done global_id 5078 payload 824 2024-03-27T21:42:07.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f4b127fc640 1 -- 172.21.15.60:0/947527844 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b14070da0 msgr2=0x7f4b14138100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f4b127fc640 1 --2- 172.21.15.60:0/947527844 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b14070da0 0x7f4b14138100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7fce2ffff640 1 librados: starting msgr at 2024-03-27T21:42:07.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f4b127fc640 1 -- 172.21.15.60:0/947527844 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b141e3890 msgr2=0x7f4b14137410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f4b127fc640 1 --2- 172.21.15.60:0/947527844 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b141e3890 0x7f4b14137410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f4b127fc640 10 monclient: _finish_hunting 0 2024-03-27T21:42:07.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f4b127fc640 1 monclient: found mon.noname-c 2024-03-27T21:42:07.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f4b127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:07.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f4b127fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:07.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f4b127fc640 1 -- 172.21.15.60:0/947527844 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b04009400 con 0x7f4b1412b9a0 2024-03-27T21:42:07.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f4b137fe640 10 monclient: get_auth_request con 0x7f4b141e3890 auth_method 0 2024-03-27T21:42:07.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7f4b137fe640 1 --2- 172.21.15.60:0/947527844 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b141e3890 0x7f4b14137410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:42:07.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7fa477381640 1 -- 172.21.15.60:0/2597202862 wait complete. 2024-03-27T21:42:07.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.417+0000 7fce2ffff640 1 librados: starting objecter 2024-03-27T21:42:07.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7f9e9b59e640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:07.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7f9e9b59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:42:07.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7f9e9b59e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e9c0aa730 0x7f9e9c133670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7fce2ffff640 1 Processor -- start 2024-03-27T21:42:07.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7fce2ffff640 1 -- start start 2024-03-27T21:42:07.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7fce2ffff640 1 librados: setting wanted keys 2024-03-27T21:42:07.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7fce2ffff640 1 librados: calling monclient init 2024-03-27T21:42:07.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7fce2ffff640 10 monclient: init 2024-03-27T21:42:07.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7f9e9a59c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e9c0aa730 0x7f9e9c133670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7f9e9a59c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e9c0aa730 0x7f9e9c133670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:58964/0 (socket says 172.21.15.60:58964) 2024-03-27T21:42:07.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7fa477381640 10 monclient: build_initial_monmap 2024-03-27T21:42:07.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7fa477381640 10 monclient: monmap: 2024-03-27T21:42:07.620 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:07.620 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:07.620 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:07.620 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:07.620 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:07.621 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:07.621 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:07.621 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:07.621 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:07.621 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7fa477381640 1 librados: starting msgr at 2024-03-27T21:42:07.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7fa477381640 1 librados: starting objecter 2024-03-27T21:42:07.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7fa477381640 1 Processor -- start 2024-03-27T21:42:07.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7f9e9a59c640 1 -- 172.21.15.60:0/576484311 learned_addr learned my addr 172.21.15.60:0/576484311 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:07.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7f9e9b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f9e9c0aa730 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:07.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7f9e9b59e640 1 --2- 172.21.15.60:0/576484311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e9c0ac530 0x7f9e9c1d2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7f9e9b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f9e9c0ac530 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:07.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f4b127fc640 1 --2- 172.21.15.60:0/947527844 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4b1412b9a0 0x7f4b141e3350 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c00ce60 tx=0x7f4b0c00a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e9b59e640 1 --2- 172.21.15.60:0/576484311 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9e9c0ae3d0 0x7f9e9c1d5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e9b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f9e9c0ae3d0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:07.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e9b59e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7fa477381640 1 -- start start 2024-03-27T21:42:07.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e9b59e640 1 -- 172.21.15.60:0/576484311 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c13d480 con 0x7f9e9c0ac530 2024-03-27T21:42:07.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e9b59e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e9b59e640 1 -- 172.21.15.60:0/576484311 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c132920 con 0x7f9e9c0aa730 2024-03-27T21:42:07.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2ffff640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:07.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e9b59e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e9b59e640 1 -- 172.21.15.60:0/576484311 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c132c00 con 0x7f9e9c0ae3d0 2024-03-27T21:42:07.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2ffff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:42:07.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e9b59e640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:07.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce280071b0 0x7fce2800a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e9b59e640 10 monclient(hunting): authenticate will time out at 968.640400s 2024-03-27T21:42:07.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e9a59c640 10 monclient(hunting): get_auth_request con 0x7f9e9c0aa730 auth_method 0 2024-03-27T21:42:07.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e9a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e9a59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e9a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7fa477381640 1 librados: setting wanted keys 2024-03-27T21:42:07.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7fa477381640 1 librados: calling monclient init 2024-03-27T21:42:07.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7fa477381640 10 monclient: init 2024-03-27T21:42:07.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce280071b0 0x7fce2800a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fce280071b0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:07.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2ffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce28033d10 0x7fce28164d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f4afbfff640 10 client.?.objecter ms_handle_connect 0x7f4b1412b9a0 2024-03-27T21:42:07.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f4afbfff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:07.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e9a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e9a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e9a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f4afbfff640 1 -- 172.21.15.60:0/947527844 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c00ad40 con 0x7f4b1412b9a0 2024-03-27T21:42:07.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f28f08af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f28f08af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f28f08af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:07.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f28f08af640 1 -- 172.21.15.60:0/1098457953 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28ec069470 con 0x7f28ec149540 2024-03-27T21:42:07.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f4afbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fce28033d10 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:07.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f4afbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:07.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f4afbfff640 10 monclient: dump: 2024-03-27T21:42:07.634 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.634 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.634 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.634 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.634 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.634 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.634 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.634 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.635 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.635 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f4afbfff640 10 monclient: _finish_auth 0 2024-03-27T21:42:07.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f4afbfff640 10 monclient: _check_auth_tickets 2024-03-27T21:42:07.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f4afbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:07.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e99d9b640 1 --2- 172.21.15.60:0/576484311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e9c0ac530 0x7f9e9c1d2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fa477381640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:07.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fa477381640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:42:07.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fa477381640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa45c0083a0 0x7fa4680c6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e99d9b640 10 monclient(hunting): get_auth_request con 0x7f9e9c0ac530 auth_method 0 2024-03-27T21:42:07.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e99d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e99d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f9e9ad9d640 1 --2- 172.21.15.60:0/576484311 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9e9c0ae3d0 0x7f9e9c1d5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f4b19918640 5 monclient: authenticate success, global_id 5078 2024-03-27T21:42:07.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f4b19918640 10 monclient: _renew_subs 2024-03-27T21:42:07.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7fa47511d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa45c0083a0 0x7fa4680c6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f4b19918640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:07.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f4b19918640 1 -- 172.21.15.60:0/947527844 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b1412c9f0 con 0x7f4b1412b9a0 2024-03-27T21:42:07.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4b19918640 10 client.5078.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:07.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4b19918640 10 monclient: _renew_subs 2024-03-27T21:42:07.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4b19918640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:07.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7fa477381640 10 monclient(hunting): picked mon.noname-c con 0x7fa45c0083a0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:07.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4b19918640 1 -- 172.21.15.60:0/947527844 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b1412c260 con 0x7f4b1412b9a0 2024-03-27T21:42:07.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4b19918640 1 librados: init done 2024-03-27T21:42:07.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f9e9ad9d640 10 monclient(hunting): get_auth_request con 0x7f9e9c0ae3d0 auth_method 0 2024-03-27T21:42:07.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f9e9ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7fa477381640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4680c7380 0x7fa468158490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7fa47511d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa45c0083a0 0x7fa4680c6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34390/0 (socket says 172.21.15.60:34390) 2024-03-27T21:42:07.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7f4d45c41640 10 monclient: build_initial_monmap 2024-03-27T21:42:07.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7f4d45c41640 10 monclient: monmap: 2024-03-27T21:42:07.641 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:07.641 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:07.642 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:07.642 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:07.642 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:07.642 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:07.642 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:07.642 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:07.642 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:07.642 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7f4d45c41640 1 librados: starting msgr at 2024-03-27T21:42:07.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7f4d45c41640 1 librados: starting objecter 2024-03-27T21:42:07.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.418+0000 7f4d45c41640 1 Processor -- start 2024-03-27T21:42:07.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f4d45c41640 1 -- start start 2024-03-27T21:42:07.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f4d45c41640 1 librados: setting wanted keys 2024-03-27T21:42:07.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f4d45c41640 1 librados: calling monclient init 2024-03-27T21:42:07.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7f4d45c41640 10 monclient: init 2024-03-27T21:42:07.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d45c41640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:07.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d45c41640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:42:07.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d45c41640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d4013d650 0x7f4d4013dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d45c41640 10 monclient(hunting): picked mon.noname-a con 0x7f4d4013d650 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:07.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d45c41640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d4013e200 0x7f4d401f7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d45c41640 10 monclient(hunting): picked mon.noname-c con 0x7f4d4013e200 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:07.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d45c41640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d401f8050 0x7f4d401f10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d45c41640 10 monclient(hunting): picked mon.noname-b con 0x7f4d401f8050 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:07.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7fa47491c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4680c7380 0x7fa468158490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7fa47491c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4680c7380 0x7fa468158490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39596/0 (socket says 172.21.15.60:39596) 2024-03-27T21:42:07.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d45c41640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d45c41640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d401f76c0 con 0x7f4d4013d650 2024-03-27T21:42:07.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d45c41640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d45c41640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4013da30 con 0x7f4d401f8050 2024-03-27T21:42:07.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d45c41640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d45c41640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4013e8d0 con 0x7f4d4013e200 2024-03-27T21:42:07.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d45c41640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:07.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d45c41640 10 monclient(hunting): authenticate will time out at 968.641334s 2024-03-27T21:42:07.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3ffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d401f8050 0x7f4d401f10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3ffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d401f8050 0x7f4d401f10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.60:58990/0 (socket says 172.21.15.60:58990) 2024-03-27T21:42:07.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7fa47511d640 1 -- 172.21.15.60:0/1795643745 learned_addr learned my addr 172.21.15.60:0/1795643745 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:07.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3ffff640 1 -- 172.21.15.60:0/1179161353 learned_addr learned my addr 172.21.15.60:0/1179161353 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:07.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3ffff640 10 monclient(hunting): get_auth_request con 0x7f4d401f8050 auth_method 0 2024-03-27T21:42:07.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3effd640 1 --2- 172.21.15.60:0/1179161353 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d4013e200 0x7f4d401f7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3f7fe640 1 --2- 172.21.15.60:0/1179161353 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d4013d650 0x7f4d4013dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3f7fe640 10 monclient(hunting): get_auth_request con 0x7f4d4013d650 auth_method 0 2024-03-27T21:42:07.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7fa477381640 10 monclient(hunting): picked mon.noname-a con 0x7fa4680c7380 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:07.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7fa477381640 1 --2- 172.21.15.60:0/1795643745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa46814eeb0 0x7fa46815b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3effd640 10 monclient(hunting): get_auth_request con 0x7f4d4013e200 auth_method 0 2024-03-27T21:42:07.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f4d3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7fa477381640 10 monclient(hunting): picked mon.noname-b con 0x7fa46814eeb0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:07.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7fa477381640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7fa477381640 1 -- 172.21.15.60:0/1795643745 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa46814fdc0 con 0x7fa4680c7380 2024-03-27T21:42:07.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7fa477381640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7fa477381640 1 -- 172.21.15.60:0/1795643745 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4680c6e80 con 0x7fa46814eeb0 2024-03-27T21:42:07.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7fa477381640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7fa477381640 1 -- 172.21.15.60:0/1795643745 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4681589d0 con 0x7fa45c0083a0 2024-03-27T21:42:07.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7fa477381640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:07.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7fa477381640 10 monclient(hunting): authenticate will time out at 968.641870s 2024-03-27T21:42:07.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47511d640 10 monclient(hunting): get_auth_request con 0x7fa45c0083a0 auth_method 0 2024-03-27T21:42:07.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47511d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47511d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47511d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47591e640 1 --2- 172.21.15.60:0/1795643745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa46814eeb0 0x7fa46815b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47591e640 10 monclient(hunting): get_auth_request con 0x7fa46814eeb0 auth_method 0 2024-03-27T21:42:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47591e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47591e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47591e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47511d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47511d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47511d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47491c640 10 monclient(hunting): get_auth_request con 0x7fa4680c7380 auth_method 0 2024-03-27T21:42:07.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47491c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47491c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47491c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47591e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47591e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47591e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7f4d3f7fe640 10 monclient(hunting): handle_auth_done global_id 5394 payload 824 2024-03-27T21:42:07.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7f4d3f7fe640 1 -- 172.21.15.60:0/1179161353 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d4013e200 msgr2=0x7f4d401f7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7f4d3f7fe640 1 --2- 172.21.15.60:0/1179161353 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d4013e200 0x7f4d401f7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7f4d3f7fe640 1 -- 172.21.15.60:0/1179161353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d401f8050 msgr2=0x7f4d401f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7f4d3f7fe640 1 --2- 172.21.15.60:0/1179161353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d401f8050 0x7f4d401f10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7f4d3f7fe640 10 monclient: _finish_hunting 0 2024-03-27T21:42:07.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7f4d3f7fe640 1 monclient: found mon.noname-a 2024-03-27T21:42:07.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7f4d3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:07.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7f4d3f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:07.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7f4d3f7fe640 1 -- 172.21.15.60:0/1179161353 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d40148be0 con 0x7f4d4013d650 2024-03-27T21:42:07.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47511d640 10 monclient(hunting): handle_auth_done global_id 5087 payload 824 2024-03-27T21:42:07.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47511d640 1 -- 172.21.15.60:0/1795643745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa46814eeb0 msgr2=0x7fa46815b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47511d640 1 --2- 172.21.15.60:0/1795643745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa46814eeb0 0x7fa46815b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47511d640 1 -- 172.21.15.60:0/1795643745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4680c7380 msgr2=0x7fa468158490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47511d640 1 --2- 172.21.15.60:0/1795643745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4680c7380 0x7fa468158490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47511d640 10 monclient: _finish_hunting 0 2024-03-27T21:42:07.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47511d640 1 monclient: found mon.noname-c 2024-03-27T21:42:07.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47511d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:07.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47511d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:07.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47511d640 1 -- 172.21.15.60:0/1795643745 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4680c6ba0 con 0x7fa45c0083a0 2024-03-27T21:42:07.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa47511d640 1 --2- 172.21.15.60:0/1795643745 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa45c0083a0 0x7fa4680c6590 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa46c004870 tx=0x7fa46c0087a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa465ffb640 10 client.?.objecter ms_handle_connect 0x7fa45c0083a0 2024-03-27T21:42:07.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7fa465ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:07.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2ffff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce28035870 0x7fce280cd610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fce28035870 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:07.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2ffff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2ffff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce280cfe30 con 0x7fce280071b0 2024-03-27T21:42:07.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2ffff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2ffff640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce280cf2d0 con 0x7fce28033d10 2024-03-27T21:42:07.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2ffff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:07.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2ffff640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce280d0040 con 0x7fce28035870 2024-03-27T21:42:07.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2ffff640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:07.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2ffff640 10 monclient(hunting): authenticate will time out at 968.640811s 2024-03-27T21:42:07.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce280071b0 0x7fce2800a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39582/0 (socket says 172.21.15.60:39582) 2024-03-27T21:42:07.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2effd640 1 -- 172.21.15.60:0/2383427778 learned_addr learned my addr 172.21.15.60:0/2383427778 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:07.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2effd640 10 monclient(hunting): get_auth_request con 0x7fce280071b0 auth_method 0 2024-03-27T21:42:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.419+0000 7fce2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f9e9ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f9e9ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f9e99d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f9e99d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f9e99d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f9e9a59c640 10 monclient(hunting): handle_auth_done global_id 5305 payload 824 2024-03-27T21:42:07.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f9e9a59c640 1 -- 172.21.15.60:0/576484311 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9e9c0ae3d0 msgr2=0x7f9e9c1d5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f9e9a59c640 1 --2- 172.21.15.60:0/576484311 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9e9c0ae3d0 0x7f9e9c1d5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f9e9a59c640 1 -- 172.21.15.60:0/576484311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e9c0ac530 msgr2=0x7f9e9c1d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f9e9a59c640 1 --2- 172.21.15.60:0/576484311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e9c0ac530 0x7f9e9c1d2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f9e9a59c640 10 monclient: _finish_hunting 0 2024-03-27T21:42:07.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f9e9a59c640 1 monclient: found mon.noname-b 2024-03-27T21:42:07.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f9e9a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:07.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f9e9a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.191:3300/0 2024-03-27T21:42:07.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f9e9a59c640 1 -- 172.21.15.60:0/576484311 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e94008200 con 0x7f9e9c0aa730 2024-03-27T21:42:07.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.420+0000 7f9e9ad9d640 1 --2- 172.21.15.60:0/576484311 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9e9c0ae3d0 0x7f9e9c1d5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:42:07.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7f28daffd640 1 -- 172.21.15.60:0/1098457953 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f28e40407f0 con 0x7f28ec149540 2024-03-27T21:42:07.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7f28daffd640 1 --2- 172.21.15.60:0/1098457953 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f28cc06c660 0x7f28cc06eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f28daffd640 1 -- 172.21.15.60:0/1098457953 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f28e40aba70 con 0x7f28ec149540 2024-03-27T21:42:07.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f28daffd640 10 client.5075.objecter ms_dispatch 0x7f28ec000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:07.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f28daffd640 3 client.5075.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:07.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f28daffd640 3 client.5075.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:07.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f28ea59c640 1 --2- 172.21.15.60:0/1098457953 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f28cc06c660 0x7f28cc06eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f28ea59c640 10 monclient: get_auth_request con 0x7f28cc06c660 auth_method 0 2024-03-27T21:42:07.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f28daffd640 20 client.5075.objecter dump_active .. 0 homeless 2024-03-27T21:42:07.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.421+0000 7f4d3f7fe640 1 --2- 172.21.15.60:0/1179161353 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d4013d650 0x7f4d4013dcc0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4d4006acb0 tx=0x7f4d30011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d27fff640 10 client.?.objecter ms_handle_connect 0x7f4d4013d650 2024-03-27T21:42:07.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d27fff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:07.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d27fff640 1 -- 172.21.15.60:0/1179161353 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d30004440 con 0x7f4d4013d650 2024-03-27T21:42:07.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d27fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:07.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d27fff640 10 monclient: dump: 2024-03-27T21:42:07.666 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.666 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.666 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.666 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.666 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.666 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.666 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.667 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.667 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.667 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d27fff640 10 monclient: _finish_auth 0 2024-03-27T21:42:07.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d27fff640 10 monclient: _check_auth_tickets 2024-03-27T21:42:07.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:07.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d45c41640 5 monclient: authenticate success, global_id 5394 2024-03-27T21:42:07.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d45c41640 10 monclient: _renew_subs 2024-03-27T21:42:07.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d45c41640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:07.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d45c41640 1 -- 172.21.15.60:0/1179161353 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d401f0b00 con 0x7f4d4013d650 2024-03-27T21:42:07.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d45c41640 10 client.5394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:07.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d45c41640 10 monclient: _renew_subs 2024-03-27T21:42:07.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d45c41640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:07.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d45c41640 1 -- 172.21.15.60:0/1179161353 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d401f17f0 con 0x7f4d4013d650 2024-03-27T21:42:07.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d45c41640 1 librados: init done 2024-03-27T21:42:07.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2e7fc640 1 --2- 172.21.15.60:0/2383427778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce28033d10 0x7fce28164d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2e7fc640 10 monclient(hunting): get_auth_request con 0x7fce28033d10 auth_method 0 2024-03-27T21:42:07.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d27fff640 1 -- 172.21.15.60:0/1179161353 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d30003610 con 0x7f4d4013d650 2024-03-27T21:42:07.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d27fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:07.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2f7fe640 1 --2- 172.21.15.60:0/2383427778 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce28035870 0x7fce280cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2f7fe640 10 monclient(hunting): get_auth_request con 0x7fce28035870 auth_method 0 2024-03-27T21:42:07.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:07.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:07.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:07.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d27fff640 1 -- 172.21.15.60:0/1179161353 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d300039a0 con 0x7f4d4013d650 2024-03-27T21:42:07.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d3dffb640 4 set_mon_vals no callback set 2024-03-27T21:42:07.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d27fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:07.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4d27fff640 10 monclient: dump: 2024-03-27T21:42:07.672 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.672 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.672 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.672 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.672 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.672 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.672 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.672 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.673 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.673 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f9e9a59c640 1 --2- 172.21.15.60:0/576484311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e9c0aa730 0x7f9e9c133670 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9e94008690 tx=0x7f9e94004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f9e8affd640 10 client.?.objecter ms_handle_connect 0x7f9e9c0aa730 2024-03-27T21:42:07.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f9e8affd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:07.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:07.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:07.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:07.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2effd640 10 monclient(hunting): handle_auth_done global_id 5388 payload 824 2024-03-27T21:42:07.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2effd640 1 -- 172.21.15.60:0/2383427778 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce28035870 msgr2=0x7fce280cd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2effd640 1 --2- 172.21.15.60:0/2383427778 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce28035870 0x7fce280cd610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2effd640 1 -- 172.21.15.60:0/2383427778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce28033d10 msgr2=0x7fce28164d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2effd640 1 --2- 172.21.15.60:0/2383427778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce28033d10 0x7fce28164d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2effd640 10 monclient: _finish_hunting 0 2024-03-27T21:42:07.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2effd640 1 monclient: found mon.noname-a 2024-03-27T21:42:07.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:07.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:07.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7fce2effd640 1 -- 172.21.15.60:0/2383427778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce20009400 con 0x7fce280071b0 2024-03-27T21:42:07.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e8affd640 1 -- 172.21.15.60:0/576484311 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e94045910 con 0x7f9e9c0aa730 2024-03-27T21:42:07.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e8affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:07.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e8affd640 10 monclient: dump: 2024-03-27T21:42:07.678 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.678 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.678 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.678 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.678 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.679 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.679 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.679 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.679 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.679 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e8affd640 10 monclient: _finish_auth 0 2024-03-27T21:42:07.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e8affd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:07.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:07.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e8affd640 1 -- 172.21.15.60:0/576484311 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e94045a70 con 0x7f9e9c0aa730 2024-03-27T21:42:07.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e8affd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:07.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e8affd640 1 -- 172.21.15.60:0/576484311 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e94045e20 con 0x7f9e9c0aa730 2024-03-27T21:42:07.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e8affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-27T21:42:07.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e8affd640 10 monclient: dump: 2024-03-27T21:42:07.680 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.680 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.680 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.680 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.681 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.681 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.681 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.681 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.681 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.681 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e9b59e640 5 monclient: authenticate success, global_id 5305 2024-03-27T21:42:07.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e9b59e640 10 monclient: _renew_subs 2024-03-27T21:42:07.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e9b59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:07.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e9b59e640 1 -- 172.21.15.60:0/576484311 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e9c132ee0 con 0x7f9e9c0aa730 2024-03-27T21:42:07.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e9b59e640 10 client.5305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:07.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e9b59e640 10 monclient: _renew_subs 2024-03-27T21:42:07.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.424+0000 7fce2effd640 1 --2- 172.21.15.60:0/2383427778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce280071b0 0x7fce2800a070 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fce2400e9e0 tx=0x7fce2400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.424+0000 7fce177fe640 10 client.?.objecter ms_handle_connect 0x7fce280071b0 2024-03-27T21:42:07.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.424+0000 7fce177fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:07.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.424+0000 7fce177fe640 1 -- 172.21.15.60:0/2383427778 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce24009670 con 0x7fce280071b0 2024-03-27T21:42:07.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.424+0000 7fce177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.424+0000 7fce177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:07.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.424+0000 7fce177fe640 10 monclient: dump: 2024-03-27T21:42:07.684 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.684 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.684 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.684 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.684 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.684 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.684 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.685 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.685 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.685 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.424+0000 7fce177fe640 10 monclient: _finish_auth 0 2024-03-27T21:42:07.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.424+0000 7fce177fe640 10 monclient: _check_auth_tickets 2024-03-27T21:42:07.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.424+0000 7fce177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:07.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.424+0000 7fce177fe640 1 -- 172.21.15.60:0/2383427778 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce2400c9d0 con 0x7fce280071b0 2024-03-27T21:42:07.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.424+0000 7fce177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:07.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.424+0000 7fce177fe640 1 -- 172.21.15.60:0/2383427778 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2400cd80 con 0x7fce280071b0 2024-03-27T21:42:07.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.424+0000 7fce177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.424+0000 7fce177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:07.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.424+0000 7fce177fe640 10 monclient: dump: 2024-03-27T21:42:07.686 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.686 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.686 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.686 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.686 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.686 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.687 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.687 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.687 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.687 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.424+0000 7fce2d7fa640 4 set_mon_vals no callback set 2024-03-27T21:42:07.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.424+0000 7fce2ffff640 5 monclient: authenticate success, global_id 5388 2024-03-27T21:42:07.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.424+0000 7fce2ffff640 10 monclient: _renew_subs 2024-03-27T21:42:07.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.424+0000 7fce2ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:07.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.424+0000 7fce2ffff640 1 -- 172.21.15.60:0/2383427778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce2800ab10 con 0x7fce280071b0 2024-03-27T21:42:07.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e9b59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:07.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e9b59e640 1 -- 172.21.15.60:0/576484311 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e9c133bb0 con 0x7f9e9c0aa730 2024-03-27T21:42:07.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e9b59e640 1 librados: init done 2024-03-27T21:42:07.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.423+0000 7f9e98d99640 4 set_mon_vals no callback set 2024-03-27T21:42:07.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4b19918640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4b19918640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4b19918640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:07.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4b19918640 1 -- 172.21.15.60:0/947527844 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b14069470 con 0x7f4b1412b9a0 2024-03-27T21:42:07.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4afbfff640 1 -- 172.21.15.60:0/947527844 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0c0020f0 con 0x7f4b1412b9a0 2024-03-27T21:42:07.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4afbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:07.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4afbfff640 1 -- 172.21.15.60:0/947527844 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c003b70 con 0x7f4b1412b9a0 2024-03-27T21:42:07.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.425+0000 7f4d45c41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.425+0000 7f4d45c41640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4afbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4afbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:07.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4afbfff640 10 monclient: dump: 2024-03-27T21:42:07.691 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.691 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.691 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.691 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.692 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.692 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.692 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.692 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.692 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.692 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.422+0000 7f4b117fa640 4 set_mon_vals no callback set 2024-03-27T21:42:07.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.425+0000 7f4afbfff640 1 -- 172.21.15.60:0/947527844 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4b0c003cd0 con 0x7f4b1412b9a0 2024-03-27T21:42:07.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.425+0000 7f4d45c41640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:07.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.425+0000 7f4d45c41640 1 -- 172.21.15.60:0/1179161353 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d20005040 con 0x7f4d4013d650 2024-03-27T21:42:07.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.425+0000 7f4d27fff640 1 -- 172.21.15.60:0/1179161353 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4d30003b00 con 0x7f4d4013d650 2024-03-27T21:42:07.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.425+0000 7f4afbfff640 1 --2- 172.21.15.60:0/947527844 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4b0006c660 0x7f4b0006eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.425+0000 7f4d27fff640 1 --2- 172.21.15.60:0/1179161353 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4d2c06c5e0 0x7f4d2c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.425+0000 7f4b12ffd640 1 --2- 172.21.15.60:0/947527844 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4b0006c660 0x7f4b0006eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.425+0000 7f4b12ffd640 10 monclient: get_auth_request con 0x7f4b0006c660 auth_method 0 2024-03-27T21:42:07.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.426+0000 7f4d27fff640 1 -- 172.21.15.60:0/1179161353 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f4d30058570 con 0x7f4d4013d650 2024-03-27T21:42:07.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.426+0000 7f4d27fff640 10 client.5394.objecter ms_dispatch 0x7f4d40000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:07.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.426+0000 7f4afbfff640 1 -- 172.21.15.60:0/947527844 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f4b0c089c60 con 0x7f4b1412b9a0 2024-03-27T21:42:07.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.426+0000 7f4d27fff640 3 client.5394.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:07.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.426+0000 7f4afbfff640 10 client.5078.objecter ms_dispatch 0x7f4b14000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:07.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.426+0000 7f4b12ffd640 1 --2- 172.21.15.60:0/947527844 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4b0006c660 0x7f4b0006eb20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4b04034000 tx=0x7f4b04008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.426+0000 7f4afbfff640 3 client.5078.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:07.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.426+0000 7f4afbfff640 3 client.5078.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:07.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.426+0000 7f4d27fff640 3 client.5394.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:07.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.426+0000 7f4afbfff640 20 client.5078.objecter dump_active .. 0 homeless 2024-03-27T21:42:07.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.426+0000 7f4afbfff640 10 client.5078.objecter ms_handle_connect 0x7f4b0006c660 2024-03-27T21:42:07.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.426+0000 7f4d27fff640 20 client.5394.objecter dump_active .. 0 homeless 2024-03-27T21:42:07.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.425+0000 7fce2ffff640 10 client.5388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:07.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.425+0000 7fce2ffff640 10 monclient: _renew_subs 2024-03-27T21:42:07.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.425+0000 7fce2ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:07.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.425+0000 7fce2ffff640 1 -- 172.21.15.60:0/2383427778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce28165500 con 0x7fce280071b0 2024-03-27T21:42:07.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.425+0000 7fce2ffff640 1 librados: init done 2024-03-27T21:42:07.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.426+0000 7fce177fe640 1 -- 172.21.15.60:0/2383427778 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fce24013470 con 0x7fce280071b0 2024-03-27T21:42:07.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.427+0000 7fce177fe640 1 --2- 172.21.15.60:0/2383427778 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fce1c06c5e0 0x7fce1c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.427+0000 7f9e9b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.427+0000 7f9e9b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.427+0000 7f9e9b59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:07.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.427+0000 7f9e9b59e640 1 -- 172.21.15.60:0/576484311 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e80005040 con 0x7f9e9c0aa730 2024-03-27T21:42:07.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.427+0000 7fce2e7fc640 1 --2- 172.21.15.60:0/2383427778 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fce1c06c5e0 0x7fce1c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.427+0000 7fce2e7fc640 10 monclient: get_auth_request con 0x7fce1c06c5e0 auth_method 0 2024-03-27T21:42:07.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.427+0000 7fce177fe640 1 -- 172.21.15.60:0/2383427778 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fce2408ad90 con 0x7fce280071b0 2024-03-27T21:42:07.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.427+0000 7fce177fe640 10 client.5388.objecter ms_dispatch 0x7fce30000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:07.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.427+0000 7fce177fe640 3 client.5388.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:07.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.427+0000 7fce177fe640 3 client.5388.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:07.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.427+0000 7f28daffd640 10 client.5075.objecter ms_handle_connect 0x7f28cc06c660 2024-03-27T21:42:07.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.427+0000 7fce177fe640 20 client.5388.objecter dump_active .. 0 homeless 2024-03-27T21:42:07.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.427+0000 7fce2e7fc640 1 --2- 172.21.15.60:0/2383427778 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fce1c06c5e0 0x7fce1c06eaa0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fce20036000 tx=0x7fce2003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.427+0000 7f28ea59c640 1 --2- 172.21.15.60:0/1098457953 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f28cc06c660 0x7f28cc06eb20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f28e0000f60 tx=0x7f28e0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.427+0000 7fce177fe640 10 client.5388.objecter ms_handle_connect 0x7fce1c06c5e0 2024-03-27T21:42:07.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.428+0000 7f4d3effd640 1 --2- 172.21.15.60:0/1179161353 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4d2c06c5e0 0x7f4d2c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.428+0000 7f4d3effd640 10 monclient: get_auth_request con 0x7f4d2c06c5e0 auth_method 0 2024-03-27T21:42:07.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.428+0000 7fce2ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.428+0000 7fce2ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.428+0000 7fce2ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:07.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.428+0000 7fce2ffff640 1 -- 172.21.15.60:0/2383427778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce28000ec0 con 0x7fce280071b0 2024-03-27T21:42:07.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.428+0000 7f4d3effd640 1 --2- 172.21.15.60:0/1179161353 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4d2c06c5e0 0x7f4d2c06eaa0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4d34036000 tx=0x7f4d34030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.428+0000 7f4d27fff640 10 client.5394.objecter ms_handle_connect 0x7f4d2c06c5e0 2024-03-27T21:42:07.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.428+0000 7f9e8affd640 1 -- 172.21.15.60:0/576484311 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9e9403b410 con 0x7f9e9c0aa730 2024-03-27T21:42:07.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.429+0000 7f9e8affd640 1 --2- 172.21.15.60:0/576484311 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9e7c06c660 0x7f9e7c06eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.429+0000 7f9e8affd640 1 -- 172.21.15.60:0/576484311 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f9e940b4500 con 0x7f9e9c0aa730 2024-03-27T21:42:07.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.429+0000 7f9e8affd640 10 client.5305.objecter ms_dispatch 0x7f9e9c000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:07.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.429+0000 7f9e8affd640 3 client.5305.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:07.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.429+0000 7f9e8affd640 3 client.5305.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:07.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.429+0000 7f9e8affd640 20 client.5305.objecter dump_active .. 0 homeless 2024-03-27T21:42:07.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.429+0000 7f9e99d9b640 1 --2- 172.21.15.60:0/576484311 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9e7c06c660 0x7f9e7c06eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.429+0000 7f9e99d9b640 10 monclient: get_auth_request con 0x7f9e7c06c660 auth_method 0 2024-03-27T21:42:07.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.430+0000 7f9e99d9b640 1 --2- 172.21.15.60:0/576484311 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9e7c06c660 0x7f9e7c06eb20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0aa4b0 tx=0x7f9e9000f040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.430+0000 7f9e8affd640 10 client.5305.objecter ms_handle_connect 0x7f9e7c06c660 2024-03-27T21:42:07.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.430+0000 7fa465ffb640 1 -- 172.21.15.60:0/1795643745 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46c00eb10 con 0x7fa45c0083a0 2024-03-27T21:42:07.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.430+0000 7fa465ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.430+0000 7fa465ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:07.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.430+0000 7fa465ffb640 10 monclient: dump: 2024-03-27T21:42:07.707 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.707 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.707 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.708 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.708 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.708 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.708 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.708 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.708 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.708 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.430+0000 7fa465ffb640 10 monclient: _finish_auth 0 2024-03-27T21:42:07.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.430+0000 7fa465ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:42:07.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.430+0000 7fa465ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:07.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.430+0000 7fa477381640 5 monclient: authenticate success, global_id 5087 2024-03-27T21:42:07.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.430+0000 7fa477381640 10 monclient: _renew_subs 2024-03-27T21:42:07.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.430+0000 7fa477381640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:07.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.430+0000 7fa477381640 1 -- 172.21.15.60:0/1795643745 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa46814f330 con 0x7fa45c0083a0 2024-03-27T21:42:07.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.430+0000 7f28daffd640 1 -- 172.21.15.60:0/1098457953 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f28e4076e90 con 0x7f28ec149540 2024-03-27T21:42:07.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.430+0000 7f28daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.430+0000 7f28daffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:07.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.430+0000 7fa477381640 10 client.5087.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:07.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.430+0000 7fa477381640 10 monclient: _renew_subs 2024-03-27T21:42:07.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.430+0000 7fa477381640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:07.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.430+0000 7fa477381640 1 -- 172.21.15.60:0/1795643745 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa46815bed0 con 0x7fa45c0083a0 2024-03-27T21:42:07.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.430+0000 7fa477381640 1 librados: init done 2024-03-27T21:42:07.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.433+0000 7f9e8affd640 1 -- 172.21.15.60:0/576484311 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9e9407f9d0 con 0x7f9e9c0aa730 2024-03-27T21:42:07.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.433+0000 7f9e8affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.433+0000 7f9e8affd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:07.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.432+0000 7fa45b7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.432+0000 7fa45b7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.432+0000 7fa45b7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:07.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.432+0000 7fa45b7fe640 1 -- 172.21.15.60:0/1795643745 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa450005040 con 0x7fa45c0083a0 2024-03-27T21:42:07.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.434+0000 7f4d27fff640 1 -- 172.21.15.60:0/1179161353 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4d30055040 con 0x7f4d4013d650 2024-03-27T21:42:07.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.434+0000 7f4d27fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.434+0000 7f4d27fff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:07.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.439+0000 7f4afbfff640 1 -- 172.21.15.60:0/947527844 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4b0c055080 con 0x7f4b1412b9a0 2024-03-27T21:42:07.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.439+0000 7f4afbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.439+0000 7f4afbfff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:07.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.442+0000 7fce177fe640 1 -- 172.21.15.60:0/2383427778 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fce2408e060 con 0x7fce280071b0 2024-03-27T21:42:07.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.442+0000 7fce177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.442+0000 7fce177fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:07.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.443+0000 7fa465ffb640 1 -- 172.21.15.60:0/1795643745 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa46c00ec70 con 0x7fa45c0083a0 2024-03-27T21:42:07.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.443+0000 7fa465ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:07.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.443+0000 7fa4677fe640 4 set_mon_vals no callback set 2024-03-27T21:42:07.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.444+0000 7fa465ffb640 1 -- 172.21.15.60:0/1795643745 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46c001cf0 con 0x7fa45c0083a0 2024-03-27T21:42:07.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.444+0000 7fa465ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:07.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.444+0000 7fa465ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:07.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.444+0000 7fa465ffb640 10 monclient: dump: 2024-03-27T21:42:07.716 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:07.716 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:07.716 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.716 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:07.716 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:07.716 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:07.716 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:07.717 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:07.717 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:07.717 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:07.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.444+0000 7fa465ffb640 1 -- 172.21.15.60:0/1795643745 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa46c01d460 con 0x7fa45c0083a0 2024-03-27T21:42:07.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.444+0000 7fa465ffb640 1 --2- 172.21.15.60:0/1795643745 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa45406c610 0x7fa45406ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:07.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.445+0000 7fa47491c640 1 --2- 172.21.15.60:0/1795643745 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa45406c610 0x7fa45406ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:07.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.445+0000 7fa47491c640 10 monclient: get_auth_request con 0x7fa45406c610 auth_method 0 2024-03-27T21:42:07.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.445+0000 7fa47491c640 1 --2- 172.21.15.60:0/1795643745 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa45406c610 0x7fa45406ead0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa46000b9e0 tx=0x7fa460030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:07.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.445+0000 7fa465ffb640 1 -- 172.21.15.60:0/1795643745 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fa46c089e70 con 0x7fa45c0083a0 2024-03-27T21:42:07.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.449+0000 7fa465ffb640 10 client.5087.objecter ms_dispatch 0x7fa470000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:07.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.449+0000 7fa465ffb640 3 client.5087.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:07.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.449+0000 7fa465ffb640 3 client.5087.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:07.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.450+0000 7fa465ffb640 20 client.5087.objecter dump_active .. 0 homeless 2024-03-27T21:42:07.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.450+0000 7fa465ffb640 10 client.5087.objecter ms_handle_connect 0x7fa45406c610 2024-03-27T21:42:07.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.450+0000 7fa465ffb640 1 -- 172.21.15.60:0/1795643745 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa46c055290 con 0x7fa45c0083a0 2024-03-27T21:42:07.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.450+0000 7fa465ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:07.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.450+0000 7fa465ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:07.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.773+0000 7efec85dc640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-27T21:42:07.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.774+0000 7efec85dc640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-27T21:42:07.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.774+0000 7efec85dc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:07.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.774+0000 7efec85dc640 1 -- 172.21.15.60:0/506350626 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7efea80052f0 con 0x7efeb80cc600 2024-03-27T21:42:07.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.774+0000 7efeb6ffd640 1 -- 172.21.15.60:0/506350626 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efec00d8e60 con 0x7efeb80cc600 2024-03-27T21:42:07.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.775+0000 7efeb6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-27T21:42:07.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.775+0000 7efeb6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:42:07.761 INFO:teuthology.orchestra.run.smithi060.stdout:68719476770 2024-03-27T21:42:07.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.777+0000 7efeb4ff9640 10 librados: watch_flush enter 2024-03-27T21:42:07.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.777+0000 7efeb4ff9640 10 librados: watch_flush exit 2024-03-27T21:42:07.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.778+0000 7efeb4ff9640 20 client.5275.objecter shutdown clearing up homeless session... 2024-03-27T21:42:07.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.778+0000 7efeb4ff9640 10 client.5275.objecter successfully canceled tick 2024-03-27T21:42:07.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.778+0000 7efeb4ff9640 1 -- 172.21.15.60:0/506350626 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7efea406c660 msgr2=0x7efea406eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.778+0000 7efeb4ff9640 1 --2- 172.21.15.60:0/506350626 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7efea406c660 0x7efea406eb20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7efeb0006fd0 tx=0x7efeb0008040 comp rx=0 tx=0).stop 2024-03-27T21:42:07.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.778+0000 7efeb4ff9640 10 monclient: shutdown 2024-03-27T21:42:07.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.778+0000 7efeb4ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:07.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.778+0000 7efeb4ff9640 1 -- 172.21.15.60:0/506350626 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efeb80cc600 msgr2=0x7efeb80b4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.778+0000 7efeb4ff9640 1 --2- 172.21.15.60:0/506350626 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efeb80cc600 0x7efeb80b4360 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efec0072960 tx=0x7efec00a7070 comp rx=0 tx=0).stop 2024-03-27T21:42:07.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.778+0000 7efeb4ff9640 1 -- 172.21.15.60:0/506350626 shutdown_connections 2024-03-27T21:42:07.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.778+0000 7efeb4ff9640 1 --2- 172.21.15.60:0/506350626 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7efea406c660 0x7efea406eb20 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.778+0000 7efeb4ff9640 1 --2- 172.21.15.60:0/506350626 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efeb80cc600 0x7efeb80b4360 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.779+0000 7efeb4ff9640 1 --2- 172.21.15.60:0/506350626 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7efeb8038fe0 0x7efeb8032610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.779+0000 7efeb4ff9640 1 --2- 172.21.15.60:0/506350626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efeb8035640 0x7efeb80320d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.779+0000 7efeb4ff9640 1 -- 172.21.15.60:0/506350626 >> 172.21.15.60:0/506350626 conn(0x7efeb80065d0 msgr2=0x7efeb8034cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:07.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.779+0000 7efeb4ff9640 1 -- 172.21.15.60:0/506350626 shutdown_connections 2024-03-27T21:42:07.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.779+0000 7efeb4ff9640 1 -- 172.21.15.60:0/506350626 wait complete. 2024-03-27T21:42:07.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.779+0000 7efeb4ff9640 1 librados: shutdown 2024-03-27T21:42:07.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.785+0000 7fd5b06f2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-27T21:42:07.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.785+0000 7fd5b06f2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-27T21:42:07.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.785+0000 7fd5b06f2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:07.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.785+0000 7fd5b06f2640 1 -- 172.21.15.60:0/836846180 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd5900055c0 con 0x7fd594003970 2024-03-27T21:42:07.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.786+0000 7fd59effd640 1 -- 172.21.15.60:0/836846180 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd5a4037020 con 0x7fd594003970 2024-03-27T21:42:07.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.786+0000 7fd59effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-27T21:42:07.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.786+0000 7fd59effd640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:42:07.772 INFO:teuthology.orchestra.run.smithi060.stdout:55834574882 2024-03-27T21:42:07.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.789+0000 7fd59cff9640 10 librados: watch_flush enter 2024-03-27T21:42:07.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.789+0000 7fd59cff9640 10 librados: watch_flush exit 2024-03-27T21:42:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.789+0000 7fd59cff9640 20 client.5278.objecter shutdown clearing up homeless session... 2024-03-27T21:42:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.789+0000 7fd59cff9640 10 client.5278.objecter successfully canceled tick 2024-03-27T21:42:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.789+0000 7fd59cff9640 1 -- 172.21.15.60:0/836846180 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd59406e140 msgr2=0x7fd594070600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.789+0000 7fd59cff9640 1 --2- 172.21.15.60:0/836846180 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd59406e140 0x7fd594070600 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd5a804f930 tx=0x7fd5a804ea70 comp rx=0 tx=0).stop 2024-03-27T21:42:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.789+0000 7fd59cff9640 10 monclient: shutdown 2024-03-27T21:42:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.789+0000 7fd59cff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.790+0000 7fd59cff9640 1 -- 172.21.15.60:0/836846180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd594003970 msgr2=0x7fd5a00cb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.790+0000 7fd59cff9640 1 --2- 172.21.15.60:0/836846180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd594003970 0x7fd5a00cb600 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd5a402bd60 tx=0x7fd5a4000bc0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.790+0000 7fd59cff9640 1 -- 172.21.15.60:0/836846180 shutdown_connections 2024-03-27T21:42:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.790+0000 7fd59cff9640 1 --2- 172.21.15.60:0/836846180 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a002fdc0 0x7fd5a0030240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.790+0000 7fd59cff9640 1 --2- 172.21.15.60:0/836846180 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd5a00cab60 0x7fd5a00326e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.790+0000 7fd59cff9640 1 --2- 172.21.15.60:0/836846180 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd59406e140 0x7fd594070600 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.790+0000 7fd59cff9640 1 --2- 172.21.15.60:0/836846180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd594003970 0x7fd5a00cb600 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.790+0000 7fd59cff9640 1 -- 172.21.15.60:0/836846180 >> 172.21.15.60:0/836846180 conn(0x7fd5a00065d0 msgr2=0x7fd5a0034420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.790+0000 7fd59cff9640 1 -- 172.21.15.60:0/836846180 shutdown_connections 2024-03-27T21:42:07.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.790+0000 7fd59cff9640 1 -- 172.21.15.60:0/836846180 wait complete. 2024-03-27T21:42:07.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.791+0000 7fd59cff9640 1 librados: shutdown 2024-03-27T21:42:07.782 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476770 got 68719476770 for osd.3 2024-03-27T21:42:07.782 DEBUG:teuthology.parallel:result is None 2024-03-27T21:42:07.792 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574881 got 55834574882 for osd.1 2024-03-27T21:42:07.793 DEBUG:teuthology.parallel:result is None 2024-03-27T21:42:07.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.822+0000 7f4b19918640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-27T21:42:07.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.822+0000 7f4b19918640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-27T21:42:07.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.822+0000 7f4b19918640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:07.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.822+0000 7f4b19918640 1 -- 172.21.15.60:0/947527844 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f4b140b1e10 con 0x7f4b1412b9a0 2024-03-27T21:42:07.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.822+0000 7f4afbfff640 1 -- 172.21.15.60:0/947527844 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4b0c0586a0 con 0x7f4b1412b9a0 2024-03-27T21:42:07.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.822+0000 7f4afbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-27T21:42:07.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.822+0000 7f4afbfff640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:42:07.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.822+0000 7f9e9b59e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-27T21:42:07.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.822+0000 7f9e9b59e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-27T21:42:07.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.822+0000 7f9e9b59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.191:3300/0 2024-03-27T21:42:07.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.822+0000 7f9e9b59e640 1 -- 172.21.15.60:0/576484311 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f9e800052f0 con 0x7f9e9c0aa730 2024-03-27T21:42:07.810 INFO:teuthology.orchestra.run.smithi060.stdout:60129542177 2024-03-27T21:42:07.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.823+0000 7f9e8affd640 1 -- 172.21.15.60:0/576484311 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9e94082ff0 con 0x7f9e9c0aa730 2024-03-27T21:42:07.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.823+0000 7f9e8affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-27T21:42:07.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.823+0000 7f9e8affd640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:42:07.811 INFO:teuthology.orchestra.run.smithi060.stdout:68719476771 2024-03-27T21:42:07.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.824+0000 7f4b19918640 10 librados: watch_flush enter 2024-03-27T21:42:07.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.824+0000 7f4b19918640 10 librados: watch_flush exit 2024-03-27T21:42:07.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.824+0000 7f4b19918640 20 client.5078.objecter shutdown clearing up homeless session... 2024-03-27T21:42:07.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.824+0000 7f4b19918640 10 client.5078.objecter successfully canceled tick 2024-03-27T21:42:07.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f4b19918640 1 -- 172.21.15.60:0/947527844 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4b0006c660 msgr2=0x7f4b0006eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f4b19918640 1 --2- 172.21.15.60:0/947527844 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4b0006c660 0x7f4b0006eb20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4b04034000 tx=0x7f4b04008030 comp rx=0 tx=0).stop 2024-03-27T21:42:07.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f4b19918640 10 monclient: shutdown 2024-03-27T21:42:07.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f4b19918640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:07.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f4b19918640 1 -- 172.21.15.60:0/947527844 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4b1412b9a0 msgr2=0x7f4b141e3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f4b19918640 1 --2- 172.21.15.60:0/947527844 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4b1412b9a0 0x7f4b141e3350 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c00ce60 tx=0x7f4b0c00a430 comp rx=0 tx=0).stop 2024-03-27T21:42:07.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f4b19918640 1 -- 172.21.15.60:0/947527844 shutdown_connections 2024-03-27T21:42:07.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f4b19918640 1 --2- 172.21.15.60:0/947527844 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b141e3890 0x7f4b14137410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f4b19918640 1 --2- 172.21.15.60:0/947527844 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4b1412b9a0 0x7f4b141e3350 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f4b19918640 1 --2- 172.21.15.60:0/947527844 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4b0006c660 0x7f4b0006eb20 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f4b19918640 1 --2- 172.21.15.60:0/947527844 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b14070da0 0x7f4b14138100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f4b19918640 1 -- 172.21.15.60:0/947527844 >> 172.21.15.60:0/947527844 conn(0x7f4b140b3620 msgr2=0x7f4b140b0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:07.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f4b19918640 1 -- 172.21.15.60:0/947527844 shutdown_connections 2024-03-27T21:42:07.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f4b19918640 1 -- 172.21.15.60:0/947527844 wait complete. 2024-03-27T21:42:07.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f9e9b59e640 10 librados: watch_flush enter 2024-03-27T21:42:07.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f9e9b59e640 10 librados: watch_flush exit 2024-03-27T21:42:07.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f9e9b59e640 20 client.5305.objecter shutdown clearing up homeless session... 2024-03-27T21:42:07.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f9e9b59e640 10 client.5305.objecter successfully canceled tick 2024-03-27T21:42:07.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f4b19918640 1 librados: shutdown 2024-03-27T21:42:07.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f9e9b59e640 1 -- 172.21.15.60:0/576484311 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9e7c06c660 msgr2=0x7f9e7c06eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f9e9b59e640 1 --2- 172.21.15.60:0/576484311 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9e7c06c660 0x7f9e7c06eb20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0aa4b0 tx=0x7f9e9000f040 comp rx=0 tx=0).stop 2024-03-27T21:42:07.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f9e9b59e640 10 monclient: shutdown 2024-03-27T21:42:07.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f9e9b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:07.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f9e9b59e640 1 -- 172.21.15.60:0/576484311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e9c0aa730 msgr2=0x7f9e9c133670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.825+0000 7f9e9b59e640 1 --2- 172.21.15.60:0/576484311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e9c0aa730 0x7f9e9c133670 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9e94008690 tx=0x7f9e94004a40 comp rx=0 tx=0).stop 2024-03-27T21:42:07.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.826+0000 7f9e9b59e640 1 -- 172.21.15.60:0/576484311 shutdown_connections 2024-03-27T21:42:07.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.826+0000 7f9e9b59e640 1 --2- 172.21.15.60:0/576484311 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9e9c0ae3d0 0x7f9e9c1d5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.826+0000 7f9e9b59e640 1 --2- 172.21.15.60:0/576484311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e9c0ac530 0x7f9e9c1d2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.826+0000 7f9e9b59e640 1 --2- 172.21.15.60:0/576484311 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9e7c06c660 0x7f9e7c06eb20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.826+0000 7f9e9b59e640 1 --2- 172.21.15.60:0/576484311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e9c0aa730 0x7f9e9c133670 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.826+0000 7f9e9b59e640 1 -- 172.21.15.60:0/576484311 >> 172.21.15.60:0/576484311 conn(0x7f9e9c060b20 msgr2=0x7f9e9c13a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:07.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.826+0000 7f9e9b59e640 1 -- 172.21.15.60:0/576484311 shutdown_connections 2024-03-27T21:42:07.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.826+0000 7f9e9b59e640 1 -- 172.21.15.60:0/576484311 wait complete. 2024-03-27T21:42:07.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.826+0000 7f9e9b59e640 1 librados: shutdown 2024-03-27T21:42:07.823 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542177 got 60129542177 for osd.7 2024-03-27T21:42:07.823 DEBUG:teuthology.parallel:result is None 2024-03-27T21:42:07.824 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476771 got 68719476771 for osd.5 2024-03-27T21:42:07.824 DEBUG:teuthology.parallel:result is None 2024-03-27T21:42:07.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.850+0000 7f28f08af640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-27T21:42:07.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.850+0000 7f28f08af640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-27T21:42:07.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.850+0000 7f28f08af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:07.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.850+0000 7f28f08af640 1 -- 172.21.15.60:0/1098457953 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f28ec1514c0 con 0x7f28ec149540 2024-03-27T21:42:07.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.851+0000 7f28daffd640 1 -- 172.21.15.60:0/1098457953 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f28e4044020 con 0x7f28ec149540 2024-03-27T21:42:07.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.851+0000 7f28daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-27T21:42:07.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.851+0000 7f28daffd640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:42:07.838 INFO:teuthology.orchestra.run.smithi060.stdout:266287972361 2024-03-27T21:42:07.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.853+0000 7f28f08af640 10 librados: watch_flush enter 2024-03-27T21:42:07.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.853+0000 7f28f08af640 10 librados: watch_flush exit 2024-03-27T21:42:07.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.853+0000 7f28f08af640 20 client.5075.objecter shutdown clearing up homeless session... 2024-03-27T21:42:07.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.853+0000 7f28f08af640 10 client.5075.objecter successfully canceled tick 2024-03-27T21:42:07.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.853+0000 7f28f08af640 1 -- 172.21.15.60:0/1098457953 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f28cc06c660 msgr2=0x7f28cc06eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.853+0000 7f28f08af640 1 --2- 172.21.15.60:0/1098457953 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f28cc06c660 0x7f28cc06eb20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f28e0000f60 tx=0x7f28e0009040 comp rx=0 tx=0).stop 2024-03-27T21:42:07.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.853+0000 7f28f08af640 10 monclient: shutdown 2024-03-27T21:42:07.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.853+0000 7f28f08af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:07.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.853+0000 7f28f08af640 1 -- 172.21.15.60:0/1098457953 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28ec149540 msgr2=0x7f28ec1393f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.853+0000 7f28f08af640 1 --2- 172.21.15.60:0/1098457953 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28ec149540 0x7f28ec1393f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f28e4033000 tx=0x7f28e4004a40 comp rx=0 tx=0).stop 2024-03-27T21:42:07.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.854+0000 7f28f08af640 1 -- 172.21.15.60:0/1098457953 shutdown_connections 2024-03-27T21:42:07.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.854+0000 7f28f08af640 1 --2- 172.21.15.60:0/1098457953 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28ec149540 0x7f28ec1393f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.854+0000 7f28f08af640 1 --2- 172.21.15.60:0/1098457953 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f28cc06c660 0x7f28cc06eb20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.854+0000 7f28f08af640 1 --2- 172.21.15.60:0/1098457953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28ec0b32c0 0x7f28ec06d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.854+0000 7f28f08af640 1 --2- 172.21.15.60:0/1098457953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28ec0b0ec0 0x7f28ec13e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.854+0000 7f28f08af640 1 -- 172.21.15.60:0/1098457953 >> 172.21.15.60:0/1098457953 conn(0x7f28ec0608e0 msgr2=0x7f28ec12fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:07.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.854+0000 7f28f08af640 1 -- 172.21.15.60:0/1098457953 shutdown_connections 2024-03-27T21:42:07.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.854+0000 7f28f08af640 1 -- 172.21.15.60:0/1098457953 wait complete. 2024-03-27T21:42:07.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.854+0000 7f28f08af640 1 librados: shutdown 2024-03-27T21:42:07.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.857+0000 7f4d45c41640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-27T21:42:07.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.857+0000 7f4d45c41640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-27T21:42:07.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.857+0000 7f4d45c41640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:07.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.857+0000 7f4d45c41640 1 -- 172.21.15.60:0/1179161353 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4d200055c0 con 0x7f4d4013d650 2024-03-27T21:42:07.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.857+0000 7f4d27fff640 1 -- 172.21.15.60:0/1179161353 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4d30055040 con 0x7f4d4013d650 2024-03-27T21:42:07.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.857+0000 7f4d27fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-27T21:42:07.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.857+0000 7f4d27fff640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:42:07.844 INFO:teuthology.orchestra.run.smithi060.stdout:60129542177 2024-03-27T21:42:07.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.859+0000 7fce2ffff640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-27T21:42:07.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.859+0000 7fce2ffff640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-27T21:42:07.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.859+0000 7fce2ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:07.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.859+0000 7f4d45c41640 10 librados: watch_flush enter 2024-03-27T21:42:07.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.859+0000 7fce2ffff640 1 -- 172.21.15.60:0/2383427778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fce280056d0 con 0x7fce280071b0 2024-03-27T21:42:07.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.859+0000 7f4d45c41640 10 librados: watch_flush exit 2024-03-27T21:42:07.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.859+0000 7f4d45c41640 20 client.5394.objecter shutdown clearing up homeless session... 2024-03-27T21:42:07.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.859+0000 7f4d45c41640 10 client.5394.objecter successfully canceled tick 2024-03-27T21:42:07.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.859+0000 7f4d45c41640 1 -- 172.21.15.60:0/1179161353 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4d2c06c5e0 msgr2=0x7f4d2c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.859+0000 7f4d45c41640 1 --2- 172.21.15.60:0/1179161353 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4d2c06c5e0 0x7f4d2c06eaa0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4d34036000 tx=0x7f4d34030000 comp rx=0 tx=0).stop 2024-03-27T21:42:07.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.859+0000 7f4d45c41640 10 monclient: shutdown 2024-03-27T21:42:07.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.859+0000 7f4d45c41640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:07.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.859+0000 7f4d45c41640 1 -- 172.21.15.60:0/1179161353 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d4013d650 msgr2=0x7f4d4013dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.859+0000 7f4d45c41640 1 --2- 172.21.15.60:0/1179161353 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d4013d650 0x7f4d4013dcc0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4d4006acb0 tx=0x7f4d30011b00 comp rx=0 tx=0).stop 2024-03-27T21:42:07.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.859+0000 7f4d45c41640 1 -- 172.21.15.60:0/1179161353 shutdown_connections 2024-03-27T21:42:07.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.859+0000 7f4d45c41640 1 --2- 172.21.15.60:0/1179161353 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4d2c06c5e0 0x7f4d2c06eaa0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.859+0000 7f4d45c41640 1 --2- 172.21.15.60:0/1179161353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d401f8050 0x7f4d401f10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.859+0000 7f4d45c41640 1 --2- 172.21.15.60:0/1179161353 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4d4013e200 0x7f4d401f7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.859+0000 7f4d45c41640 1 --2- 172.21.15.60:0/1179161353 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d4013d650 0x7f4d4013dcc0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.859+0000 7f4d45c41640 1 -- 172.21.15.60:0/1179161353 >> 172.21.15.60:0/1179161353 conn(0x7f4d400b2ea0 msgr2=0x7f4d400b0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:07.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.860+0000 7f4d45c41640 1 -- 172.21.15.60:0/1179161353 shutdown_connections 2024-03-27T21:42:07.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.860+0000 7fce177fe640 1 -- 172.21.15.60:0/2383427778 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fce24017030 con 0x7fce280071b0 2024-03-27T21:42:07.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.860+0000 7fce177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-27T21:42:07.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.860+0000 7f4d45c41640 1 -- 172.21.15.60:0/1179161353 wait complete. 2024-03-27T21:42:07.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.860+0000 7fce177fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:42:07.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.860+0000 7f4d45c41640 1 librados: shutdown 2024-03-27T21:42:07.850 INFO:teuthology.orchestra.run.smithi060.stdout:60129542177 2024-03-27T21:42:07.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.862+0000 7fce2ffff640 10 librados: watch_flush enter 2024-03-27T21:42:07.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.862+0000 7fce2ffff640 10 librados: watch_flush exit 2024-03-27T21:42:07.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.862+0000 7fce2ffff640 20 client.5388.objecter shutdown clearing up homeless session... 2024-03-27T21:42:07.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.862+0000 7fce2ffff640 10 client.5388.objecter successfully canceled tick 2024-03-27T21:42:07.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.862+0000 7fce2ffff640 1 -- 172.21.15.60:0/2383427778 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fce1c06c5e0 msgr2=0x7fce1c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.862+0000 7fce2ffff640 1 --2- 172.21.15.60:0/2383427778 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fce1c06c5e0 0x7fce1c06eaa0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fce20036000 tx=0x7fce2003b040 comp rx=0 tx=0).stop 2024-03-27T21:42:07.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.862+0000 7fce2ffff640 10 monclient: shutdown 2024-03-27T21:42:07.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.862+0000 7fce2ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:07.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.862+0000 7fce2ffff640 1 -- 172.21.15.60:0/2383427778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce280071b0 msgr2=0x7fce2800a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.862+0000 7fce2ffff640 1 --2- 172.21.15.60:0/2383427778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce280071b0 0x7fce2800a070 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fce2400e9e0 tx=0x7fce2400eea0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.862+0000 7fce2ffff640 1 -- 172.21.15.60:0/2383427778 shutdown_connections 2024-03-27T21:42:07.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.862+0000 7fce2ffff640 1 --2- 172.21.15.60:0/2383427778 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fce28035870 0x7fce280cd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.862+0000 7fce2ffff640 1 --2- 172.21.15.60:0/2383427778 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fce1c06c5e0 0x7fce1c06eaa0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.862+0000 7fce2ffff640 1 --2- 172.21.15.60:0/2383427778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce28033d10 0x7fce28164d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.862+0000 7fce2ffff640 1 --2- 172.21.15.60:0/2383427778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce280071b0 0x7fce2800a070 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.862+0000 7fce2ffff640 1 -- 172.21.15.60:0/2383427778 >> 172.21.15.60:0/2383427778 conn(0x7fce28006210 msgr2=0x7fce280c9300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:07.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.862+0000 7fce2ffff640 1 -- 172.21.15.60:0/2383427778 shutdown_connections 2024-03-27T21:42:07.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.862+0000 7fce2ffff640 1 -- 172.21.15.60:0/2383427778 wait complete. 2024-03-27T21:42:07.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.862+0000 7fce2ffff640 1 librados: shutdown 2024-03-27T21:42:07.856 INFO:tasks.ceph.ceph_manager.ceph:need seq 266287972361 got 266287972361 for osd.2 2024-03-27T21:42:07.857 DEBUG:teuthology.parallel:result is None 2024-03-27T21:42:07.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.870+0000 7fa45b7fe640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-27T21:42:07.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.870+0000 7fa45b7fe640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-27T21:42:07.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.870+0000 7fa45b7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:07.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.870+0000 7fa45b7fe640 1 -- 172.21.15.60:0/1795643745 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa4500055c0 con 0x7fa45c0083a0 2024-03-27T21:42:07.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.870+0000 7fa465ffb640 1 -- 172.21.15.60:0/1795643745 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa46c0588b0 con 0x7fa45c0083a0 2024-03-27T21:42:07.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.870+0000 7fa465ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-27T21:42:07.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.870+0000 7fa465ffb640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:42:07.858 INFO:teuthology.orchestra.run.smithi060.stdout:60129542177 2024-03-27T21:42:07.859 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542177 got 60129542177 for osd.4 2024-03-27T21:42:07.860 DEBUG:teuthology.parallel:result is None 2024-03-27T21:42:07.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.872+0000 7fa45b7fe640 10 librados: watch_flush enter 2024-03-27T21:42:07.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.872+0000 7fa45b7fe640 10 librados: watch_flush exit 2024-03-27T21:42:07.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.872+0000 7fa45b7fe640 20 client.5087.objecter shutdown clearing up homeless session... 2024-03-27T21:42:07.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.872+0000 7fa45b7fe640 10 client.5087.objecter successfully canceled tick 2024-03-27T21:42:07.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.873+0000 7fa45b7fe640 1 -- 172.21.15.60:0/1795643745 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa45406c610 msgr2=0x7fa45406ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.873+0000 7fa45b7fe640 1 --2- 172.21.15.60:0/1795643745 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa45406c610 0x7fa45406ead0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa46000b9e0 tx=0x7fa460030000 comp rx=0 tx=0).stop 2024-03-27T21:42:07.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.873+0000 7fa45b7fe640 10 monclient: shutdown 2024-03-27T21:42:07.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.873+0000 7fa45b7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:07.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.873+0000 7fa45b7fe640 1 -- 172.21.15.60:0/1795643745 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa45c0083a0 msgr2=0x7fa4680c6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:07.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.873+0000 7fa45b7fe640 1 --2- 172.21.15.60:0/1795643745 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa45c0083a0 0x7fa4680c6590 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa46c004870 tx=0x7fa46c0087a0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.873+0000 7fa45b7fe640 1 -- 172.21.15.60:0/1795643745 shutdown_connections 2024-03-27T21:42:07.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.873+0000 7fa45b7fe640 1 --2- 172.21.15.60:0/1795643745 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa45406c610 0x7fa45406ead0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.873+0000 7fa45b7fe640 1 --2- 172.21.15.60:0/1795643745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa46814eeb0 0x7fa46815b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.873+0000 7fa45b7fe640 1 --2- 172.21.15.60:0/1795643745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4680c7380 0x7fa468158490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.873+0000 7fa45b7fe640 1 --2- 172.21.15.60:0/1795643745 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa45c0083a0 0x7fa4680c6590 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:07.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.873+0000 7fa45b7fe640 1 -- 172.21.15.60:0/1795643745 >> 172.21.15.60:0/1795643745 conn(0x7fa468005ee0 msgr2=0x7fa4680056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:07.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.873+0000 7fa45b7fe640 1 -- 172.21.15.60:0/1795643745 shutdown_connections 2024-03-27T21:42:07.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.873+0000 7fa45b7fe640 1 -- 172.21.15.60:0/1795643745 wait complete. 2024-03-27T21:42:07.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:07.873+0000 7fa45b7fe640 1 librados: shutdown 2024-03-27T21:42:07.864 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542177 got 60129542177 for osd.0 2024-03-27T21:42:07.864 DEBUG:teuthology.parallel:result is None 2024-03-27T21:42:07.869 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542177 got 60129542177 for osd.6 2024-03-27T21:42:07.870 DEBUG:teuthology.parallel:result is None 2024-03-27T21:42:07.870 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-27T21:42:07.870 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T21:42:08.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.023+0000 7fa56fd55640 10 monclient: get_monmap_and_config 2024-03-27T21:42:08.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.023+0000 7fa56fd55640 10 monclient: build_initial_monmap 2024-03-27T21:42:08.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.023+0000 7fa56fd55640 10 monclient: monmap: 2024-03-27T21:42:08.010 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:08.010 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:08.010 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:08.024958+0000 2024-03-27T21:42:08.011 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:08.024958+0000 2024-03-27T21:42:08.011 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:08.011 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:08.011 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:08.011 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:08.011 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:08.011 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:08.011 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:08.011 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.024+0000 7fa56fd55640 1 Processor -- start 2024-03-27T21:42:08.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.024+0000 7fa56fd55640 1 -- start start 2024-03-27T21:42:08.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.024+0000 7fa56fd55640 10 monclient: init 2024-03-27T21:42:08.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56fd55640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:08.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56fd55640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-03-27T21:42:08.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56fd55640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa568071a50 0x7fa5680a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56fd55640 10 monclient(hunting): picked mon.noname-a con 0x7fa568071a50 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:08.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56fd55640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa56813e810 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:08.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56fd55640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa56813ec50 0x7fa568141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56fd55640 10 monclient(hunting): picked mon.noname-c con 0x7fa56813ec50 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:08.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56fd55640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5680a8e90 con 0x7fa56813e810 2024-03-27T21:42:08.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56fd55640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56fd55640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa568071e30 con 0x7fa568071a50 2024-03-27T21:42:08.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56fd55640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56fd55640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5680aa040 con 0x7fa56813ec50 2024-03-27T21:42:08.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56fd55640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:08.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56fd55640 10 monclient(hunting): authenticate will time out at 969.246250s 2024-03-27T21:42:08.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56daf1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa568071a50 0x7fa5680a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56daf1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa568071a50 0x7fa5680a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39598/0 (socket says 172.21.15.60:39598) 2024-03-27T21:42:08.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56daf1640 1 -- 172.21.15.60:0/2817243283 learned_addr learned my addr 172.21.15.60:0/2817243283 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:08.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56daf1640 10 monclient(hunting): get_auth_request con 0x7fa568071a50 auth_method 0 2024-03-27T21:42:08.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56daf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:08.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56daf1640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:08.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56daf1640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:08.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56daf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:08.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56daf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:08.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56daf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:08.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56e2f2640 1 --2- 172.21.15.60:0/2817243283 >> v2:172.21.15.191:3300/0 conn(0x7fa56813ec50 0x7fa568141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56e2f2640 10 monclient(hunting): get_auth_request con 0x7fa56813ec50 auth_method 0 2024-03-27T21:42:08.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56e2f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:08.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56e2f2640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:08.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56e2f2640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:08.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56e2f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:08.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56e2f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:08.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56e2f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:08.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56daf1640 10 monclient(hunting): handle_auth_done global_id 5400 payload 274 2024-03-27T21:42:08.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56daf1640 1 -- 172.21.15.60:0/2817243283 >> v2:172.21.15.191:3300/0 conn(0x7fa56813ec50 msgr2=0x7fa568141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:08.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56daf1640 1 --2- 172.21.15.60:0/2817243283 >> v2:172.21.15.191:3300/0 conn(0x7fa56813ec50 0x7fa568141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56daf1640 1 -- 172.21.15.60:0/2817243283 >> v1:172.21.15.60:6789/0 conn(0x7fa56813e810 legacy=0x7fa568141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56daf1640 10 monclient: _finish_hunting 0 2024-03-27T21:42:08.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56daf1640 1 monclient: found mon.noname-a 2024-03-27T21:42:08.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56daf1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:08.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56daf1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:08.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.025+0000 7fa56daf1640 1 -- 172.21.15.60:0/2817243283 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa568142280 con 0x7fa568071a50 2024-03-27T21:42:08.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.026+0000 7fa56daf1640 1 --2- 172.21.15.60:0/2817243283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa568071a50 0x7fa5680a87e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa5680b3460 tx=0x7fa55802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6ef4b9f3504969 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:08.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.026+0000 7fa56caef640 1 -- 172.21.15.60:0/2817243283 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55802cb20 con 0x7fa568071a50 2024-03-27T21:42:08.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.026+0000 7fa56caef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:08.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.026+0000 7fa56caef640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:08.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.026+0000 7fa56caef640 10 monclient: dump: 2024-03-27T21:42:08.025 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:08.025 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:08.025 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.026 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.026 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:08.026 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:08.026 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:08.026 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:08.026 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:08.026 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.026+0000 7fa56caef640 10 monclient: _finish_auth 0 2024-03-27T21:42:08.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.026+0000 7fa56caef640 10 monclient: _check_auth_tickets 2024-03-27T21:42:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.026+0000 7fa56caef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.026+0000 7fa56caef640 1 -- 172.21.15.60:0/2817243283 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa55802cc80 con 0x7fa568071a50 2024-03-27T21:42:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.026+0000 7fa56fd55640 5 monclient: authenticate success, global_id 5400 2024-03-27T21:42:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.026+0000 7fa56fd55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.026+0000 7fa56caef640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.026+0000 7fa56fd55640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.026+0000 7fa56caef640 1 -- 172.21.15.60:0/2817243283 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5580343f0 con 0x7fa568071a50 2024-03-27T21:42:08.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.026+0000 7fa56caef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.026+0000 7fa56fd55640 4 set_mon_vals no callback set 2024-03-27T21:42:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.026+0000 7fa56caef640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.026+0000 7fa56caef640 10 monclient: dump: 2024-03-27T21:42:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:08.029 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.027+0000 7fa56fd55640 10 monclient: shutdown 2024-03-27T21:42:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.027+0000 7fa56fd55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.027+0000 7fa56fd55640 1 -- 172.21.15.60:0/2817243283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa568071a50 msgr2=0x7fa5680a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.027+0000 7fa56fd55640 1 --2- 172.21.15.60:0/2817243283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa568071a50 0x7fa5680a87e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa5680b3460 tx=0x7fa55802bf80 comp rx=0 tx=0).stop 2024-03-27T21:42:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.027+0000 7fa56fd55640 1 -- 172.21.15.60:0/2817243283 shutdown_connections 2024-03-27T21:42:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.027+0000 7fa56fd55640 1 --2- 172.21.15.60:0/2817243283 >> v2:172.21.15.191:3300/0 conn(0x7fa56813ec50 0x7fa568141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.027+0000 7fa56fd55640 1 --2- 172.21.15.60:0/2817243283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa568071a50 0x7fa5680a87e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.027+0000 7fa56fd55640 1 -- 172.21.15.60:0/2817243283 >> 172.21.15.60:0/2817243283 conn(0x7fa5680b7300 msgr2=0x7fa5680b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.027+0000 7fa56fd55640 1 -- 172.21.15.60:0/2817243283 shutdown_connections 2024-03-27T21:42:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.027+0000 7fa56fd55640 1 -- 172.21.15.60:0/2817243283 wait complete. 2024-03-27T21:42:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.027+0000 7fa56fd55640 10 monclient: build_initial_monmap 2024-03-27T21:42:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.027+0000 7fa56fd55640 10 monclient: monmap: 2024-03-27T21:42:08.030 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:08.031 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:08.031 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:08.031 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:08.031 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:08.031 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:08.031 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:08.031 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:08.031 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:08.032 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 1 librados: starting msgr at 2024-03-27T21:42:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 1 librados: starting objecter 2024-03-27T21:42:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 1 Processor -- start 2024-03-27T21:42:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 1 -- start start 2024-03-27T21:42:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 1 librados: setting wanted keys 2024-03-27T21:42:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 1 librados: calling monclient init 2024-03-27T21:42:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 10 monclient: init 2024-03-27T21:42:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:08.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:42:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa568071a50 0x7fa568136350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 10 monclient(hunting): picked mon.noname-b con 0x7fa568071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa56813e810 0x7fa568139620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 10 monclient(hunting): picked mon.noname-c con 0x7fa56813e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa56813ec50 0x7fa5681d6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 10 monclient(hunting): picked mon.noname-a con 0x7fa56813ec50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa568137000 con 0x7fa56813ec50 2024-03-27T21:42:08.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa568139f80 con 0x7fa568071a50 2024-03-27T21:42:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa568138e20 con 0x7fa56813e810 2024-03-27T21:42:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56fd55640 10 monclient(hunting): authenticate will time out at 969.249720s 2024-03-27T21:42:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56e2f2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa56813ec50 0x7fa5681d6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56e2f2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa56813ec50 0x7fa5681d6300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39612/0 (socket says 172.21.15.60:39612) 2024-03-27T21:42:08.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56e2f2640 1 -- 172.21.15.60:0/41610172 learned_addr learned my addr 172.21.15.60:0/41610172 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56e2f2640 10 monclient(hunting): get_auth_request con 0x7fa56813ec50 auth_method 0 2024-03-27T21:42:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56e2f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56e2f2640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56e2f2640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56d2f0640 1 --2- 172.21.15.60:0/41610172 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa56813e810 0x7fa568139620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56d2f0640 10 monclient(hunting): get_auth_request con 0x7fa56813e810 auth_method 0 2024-03-27T21:42:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56d2f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56d2f0640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:08.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56d2f0640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56e2f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.028+0000 7fa56e2f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56e2f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56d2f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56d2f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56d2f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56daf1640 1 --2- 172.21.15.60:0/41610172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa568071a50 0x7fa568136350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56e2f2640 10 monclient(hunting): handle_auth_done global_id 5406 payload 824 2024-03-27T21:42:08.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56e2f2640 1 -- 172.21.15.60:0/41610172 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa56813e810 msgr2=0x7fa568139620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56e2f2640 1 --2- 172.21.15.60:0/41610172 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa56813e810 0x7fa568139620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56e2f2640 1 -- 172.21.15.60:0/41610172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa568071a50 msgr2=0x7fa568136350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56e2f2640 1 --2- 172.21.15.60:0/41610172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa568071a50 0x7fa568136350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56e2f2640 10 monclient: _finish_hunting 0 2024-03-27T21:42:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56e2f2640 1 monclient: found mon.noname-a 2024-03-27T21:42:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56e2f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56e2f2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56e2f2640 1 -- 172.21.15.60:0/41610172 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa558009b30 con 0x7fa56813ec50 2024-03-27T21:42:08.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56daf1640 10 monclient: get_auth_request con 0x7fa568071a50 auth_method 0 2024-03-27T21:42:08.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56daf1640 1 --2- 172.21.15.60:0/41610172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa568071a50 0x7fa568136350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:42:08.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56e2f2640 1 --2- 172.21.15.60:0/41610172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa56813ec50 0x7fa5681d6300 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa56000e7f0 tx=0x7fa56000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:08.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa5667fc640 10 client.?.objecter ms_handle_connect 0x7fa56813ec50 2024-03-27T21:42:08.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa5667fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:08.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa5667fc640 1 -- 172.21.15.60:0/41610172 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa560009510 con 0x7fa56813ec50 2024-03-27T21:42:08.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa5667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:08.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa5667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:08.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa5667fc640 10 monclient: dump: 2024-03-27T21:42:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:08.039 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:08.040 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa5667fc640 10 monclient: _finish_auth 0 2024-03-27T21:42:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa5667fc640 10 monclient: _check_auth_tickets 2024-03-27T21:42:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa5667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa5667fc640 1 -- 172.21.15.60:0/41610172 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56000ee40 con 0x7fa56813ec50 2024-03-27T21:42:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56fd55640 5 monclient: authenticate success, global_id 5406 2024-03-27T21:42:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56fd55640 10 monclient: _renew_subs 2024-03-27T21:42:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56fd55640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:08.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56fd55640 1 -- 172.21.15.60:0/41610172 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa568139b60 con 0x7fa56813ec50 2024-03-27T21:42:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa5667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa5667fc640 1 -- 172.21.15.60:0/41610172 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56000cb80 con 0x7fa56813ec50 2024-03-27T21:42:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa5667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa5667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa5667fc640 10 monclient: dump: 2024-03-27T21:42:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.041 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:08.042 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56fd55640 10 client.5406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:08.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa567fff640 4 set_mon_vals no callback set 2024-03-27T21:42:08.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56fd55640 10 monclient: _renew_subs 2024-03-27T21:42:08.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56fd55640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:08.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56fd55640 1 -- 172.21.15.60:0/41610172 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5681d6a20 con 0x7fa56813ec50 2024-03-27T21:42:08.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.029+0000 7fa56fd55640 1 librados: init done 2024-03-27T21:42:08.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.030+0000 7fa56fd55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:08.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.030+0000 7fa56fd55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:08.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.030+0000 7fa56fd55640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:08.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.030+0000 7fa56fd55640 1 -- 172.21.15.60:0/41610172 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa568139d50 con 0x7fa56813ec50 2024-03-27T21:42:08.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.031+0000 7fa5667fc640 1 -- 172.21.15.60:0/41610172 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa56000cce0 con 0x7fa56813ec50 2024-03-27T21:42:08.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.031+0000 7fa5667fc640 1 --2- 172.21.15.60:0/41610172 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa55406c5c0 0x7fa55406ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.031+0000 7fa5667fc640 1 -- 172.21.15.60:0/41610172 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fa560089210 con 0x7fa56813ec50 2024-03-27T21:42:08.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.031+0000 7fa5667fc640 10 client.5406.objecter ms_dispatch 0x7fa568000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:08.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.031+0000 7fa56daf1640 1 --2- 172.21.15.60:0/41610172 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa55406c5c0 0x7fa55406ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.031+0000 7fa5667fc640 3 client.5406.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:08.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.031+0000 7fa5667fc640 3 client.5406.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:08.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.031+0000 7fa56daf1640 10 monclient: get_auth_request con 0x7fa55406c5c0 auth_method 0 2024-03-27T21:42:08.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.031+0000 7fa5667fc640 20 client.5406.objecter dump_active .. 0 homeless 2024-03-27T21:42:08.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.031+0000 7fa56daf1640 1 --2- 172.21.15.60:0/41610172 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa55406c5c0 0x7fa55406ea80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa558035000 tx=0x7fa558030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:08.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.031+0000 7fa5667fc640 10 client.5406.objecter ms_handle_connect 0x7fa55406c5c0 2024-03-27T21:42:08.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.035+0000 7fa5667fc640 1 -- 172.21.15.60:0/41610172 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa568139d50 con 0x7fa56813ec50 2024-03-27T21:42:08.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.035+0000 7fa5667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:08.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.035+0000 7fa5667fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:08.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.211+0000 7fa56fd55640 1 -- 172.21.15.60:0/41610172 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa56813f0c0 con 0x7fa55406c5c0 2024-03-27T21:42:08.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.213+0000 7fa5667fc640 1 -- 172.21.15.60:0/41610172 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+66898 (secure 0 0 0) 0x7fa558009510 con 0x7fa55406c5c0 2024-03-27T21:42:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:42:08.200 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-03-27T21:42:08.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.215+0000 7fa56fd55640 10 librados: watch_flush enter 2024-03-27T21:42:08.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.215+0000 7fa56fd55640 10 librados: watch_flush exit 2024-03-27T21:42:08.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.215+0000 7fa56fd55640 20 client.5406.objecter shutdown clearing up homeless session... 2024-03-27T21:42:08.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.215+0000 7fa56fd55640 10 client.5406.objecter successfully canceled tick 2024-03-27T21:42:08.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.215+0000 7fa56fd55640 1 -- 172.21.15.60:0/41610172 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa55406c5c0 msgr2=0x7fa55406ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.215+0000 7fa56fd55640 1 --2- 172.21.15.60:0/41610172 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa55406c5c0 0x7fa55406ea80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa558035000 tx=0x7fa558030000 comp rx=0 tx=0).stop 2024-03-27T21:42:08.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.215+0000 7fa56fd55640 10 monclient: shutdown 2024-03-27T21:42:08.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.215+0000 7fa56fd55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:08.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.215+0000 7fa56fd55640 1 -- 172.21.15.60:0/41610172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa56813ec50 msgr2=0x7fa5681d6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.215+0000 7fa56fd55640 1 --2- 172.21.15.60:0/41610172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa56813ec50 0x7fa5681d6300 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa56000e7f0 tx=0x7fa56000ebf0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.216+0000 7fa56fd55640 1 -- 172.21.15.60:0/41610172 shutdown_connections 2024-03-27T21:42:08.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.216+0000 7fa56fd55640 1 --2- 172.21.15.60:0/41610172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa56813ec50 0x7fa5681d6300 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.216+0000 7fa56fd55640 1 --2- 172.21.15.60:0/41610172 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fa56813e810 0x7fa568139620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.216+0000 7fa56fd55640 1 --2- 172.21.15.60:0/41610172 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fa55406c5c0 0x7fa55406ea80 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.216+0000 7fa56fd55640 1 --2- 172.21.15.60:0/41610172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa568071a50 0x7fa568136350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.216+0000 7fa56fd55640 1 -- 172.21.15.60:0/41610172 >> 172.21.15.60:0/41610172 conn(0x7fa5680b7300 msgr2=0x7fa56813d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:08.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.216+0000 7fa56fd55640 1 -- 172.21.15.60:0/41610172 shutdown_connections 2024-03-27T21:42:08.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.216+0000 7fa56fd55640 1 -- 172.21.15.60:0/41610172 wait complete. 2024-03-27T21:42:08.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.216+0000 7fa56fd55640 1 librados: shutdown 2024-03-27T21:42:08.213 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-03-27T21:42:07.412709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":230404,"kb_used_data":8472,"kb_used_omap":16,"kb_used_meta":221807,"kb_avail":754744316,"statfs":{"total":773094113280,"available":772858179584,"internally_reserved":0,"allocated":8675328,"data_stored":4710296,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":17199,"internal_metadata":227130577},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000861"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-27T21:42:02.414394+0000","last_change":"2024-03-27T21:42:01.265365+0000","last_active":"2024-03-27T21:42:02.414394+0000","last_peered":"2024-03-27T21:42:02.414394+0000","last_clean":"2024-03-27T21:42:02.414394+0000","last_became_active":"2024-03-27T21:42:01.265033+0000","last_became_peered":"2024-03-27T21:42:01.265033+0000","last_unstale":"2024-03-27T21:42:02.414394+0000","last_undegraded":"2024-03-27T21:42:02.414394+0000","last_fullsized":"2024-03-27T21:42:02.414394+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:07:09.198158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.601878+0000","last_change":"2024-03-27T21:41:49.474318+0000","last_active":"2024-03-27T21:42:03.601878+0000","last_peered":"2024-03-27T21:42:03.601878+0000","last_clean":"2024-03-27T21:42:03.601878+0000","last_became_active":"2024-03-27T21:41:49.474054+0000","last_became_peered":"2024-03-27T21:41:49.474054+0000","last_unstale":"2024-03-27T21:42:03.601878+0000","last_undegraded":"2024-03-27T21:42:03.601878+0000","last_fullsized":"2024-03-27T21:42:03.601878+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:36:26.918691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271058+0000","last_change":"2024-03-27T21:39:46.679788+0000","last_active":"2024-03-27T21:42:03.271058+0000","last_peered":"2024-03-27T21:42:03.271058+0000","last_clean":"2024-03-27T21:42:03.271058+0000","last_became_active":"2024-03-27T21:39:46.679561+0000","last_became_peered":"2024-03-27T21:39:46.679561+0000","last_unstale":"2024-03-27T21:42:03.271058+0000","last_undegraded":"2024-03-27T21:42:03.271058+0000","last_fullsized":"2024-03-27T21:42:03.271058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T23:53:50.373923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.601870+0000","last_change":"2024-03-27T21:41:49.474121+0000","last_active":"2024-03-27T21:42:03.601870+0000","last_peered":"2024-03-27T21:42:03.601870+0000","last_clean":"2024-03-27T21:42:03.601870+0000","last_became_active":"2024-03-27T21:41:49.473822+0000","last_became_peered":"2024-03-27T21:41:49.473822+0000","last_unstale":"2024-03-27T21:42:03.601870+0000","last_undegraded":"2024-03-27T21:42:03.601870+0000","last_fullsized":"2024-03-27T21:42:03.601870+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:39:51.253704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271086+0000","last_change":"2024-03-27T21:41:59.701200+0000","last_active":"2024-03-27T21:42:03.271086+0000","last_peered":"2024-03-27T21:42:03.271086+0000","last_clean":"2024-03-27T21:42:03.271086+0000","last_became_active":"2024-03-27T21:41:59.700966+0000","last_became_peered":"2024-03-27T21:41:59.700966+0000","last_unstale":"2024-03-27T21:42:03.271086+0000","last_undegraded":"2024-03-27T21:42:03.271086+0000","last_fullsized":"2024-03-27T21:42:03.271086+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:17:01.025558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.270838+0000","last_change":"2024-03-27T21:41:30.080153+0000","last_active":"2024-03-27T21:42:03.270838+0000","last_peered":"2024-03-27T21:42:03.270838+0000","last_clean":"2024-03-27T21:42:03.270838+0000","last_became_active":"2024-03-27T21:41:30.079829+0000","last_became_peered":"2024-03-27T21:41:30.079829+0000","last_unstale":"2024-03-27T21:42:03.270838+0000","last_undegraded":"2024-03-27T21:42:03.270838+0000","last_fullsized":"2024-03-27T21:42:03.270838+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:21:00.966562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":97,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271058+0000","last_change":"2024-03-27T21:41:49.197510+0000","last_active":"2024-03-27T21:42:03.271058+0000","last_peered":"2024-03-27T21:42:03.271058+0000","last_clean":"2024-03-27T21:42:03.271058+0000","last_became_active":"2024-03-27T21:41:49.197154+0000","last_became_peered":"2024-03-27T21:41:49.197154+0000","last_unstale":"2024-03-27T21:42:03.271058+0000","last_undegraded":"2024-03-27T21:42:03.271058+0000","last_fullsized":"2024-03-27T21:42:03.271058+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:18:21.622988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271153+0000","last_change":"2024-03-27T21:41:59.247145+0000","last_active":"2024-03-27T21:42:03.271153+0000","last_peered":"2024-03-27T21:42:03.271153+0000","last_clean":"2024-03-27T21:42:03.271153+0000","last_became_active":"2024-03-27T21:41:59.246895+0000","last_became_peered":"2024-03-27T21:41:59.246895+0000","last_unstale":"2024-03-27T21:42:03.271153+0000","last_undegraded":"2024-03-27T21:42:03.271153+0000","last_fullsized":"2024-03-27T21:42:03.271153+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:40:39.749875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":234,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271065+0000","last_change":"2024-03-27T21:39:39.866806+0000","last_active":"2024-03-27T21:42:03.271065+0000","last_peered":"2024-03-27T21:42:03.271065+0000","last_clean":"2024-03-27T21:42:03.271065+0000","last_became_active":"2024-03-27T21:39:39.865948+0000","last_became_peered":"2024-03-27T21:39:39.865948+0000","last_unstale":"2024-03-27T21:42:03.271065+0000","last_undegraded":"2024-03-27T21:42:03.271065+0000","last_fullsized":"2024-03-27T21:42:03.271065+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_clean_scrub_stamp":"2024-03-27T21:39:38.411432+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:36:49.375869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":16,"seq":68719476771,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.41599999999999998,"15min":0.41199999999999998},"min":{"1min":0.34399999999999997,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.621,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.498},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.47499999999999998,"15min":0.47599999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.66900000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.70499999999999996}]},{"osd":1,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45900000000000002,"15min":0.46500000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.53600000000000003,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.35899999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.56599999999999995,"5min":0.745,"15min":0.745},"last":0.371}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":3,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.50600000000000001,"15min":0.51400000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.56899999999999995,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.35999999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68400000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.56000000000000005}]},{"osd":4,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.432,"15min":0.439},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.56399999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.433,"5min":0.46800000000000003,"15min":0.47399999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.78400000000000003}]},{"osd":6,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.45800000000000002,"15min":0.46300000000000002},"min":{"1min":0.318,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58699999999999997,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.29299999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64300000000000002,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.73999999999999999}]},{"osd":7,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.52400000000000002,"15min":0.53200000000000003},"min":{"1min":0.378,"5min":0.318,"15min":0.318},"max":{"1min":0.61299999999999999,"5min":0.745,"15min":0.745},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.50800000000000001,"15min":0.51400000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65000000000000002,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.72399999999999998}]}]},{"osd":3,"up_from":16,"seq":68719476770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31848905},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.49299999999999999,"15min":0.504},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.91700000000000004,"5min":1.248,"15min":1.248},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.44900000000000001,"15min":0.45600000000000002},"min":{"1min":0.22800000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.90600000000000003,"5min":1.407,"15min":1.407},"last":0.36599999999999999}]},{"osd":1,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48199999999999998,"15min":0.47999999999999998},"min":{"1min":0.25600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.26800000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.96899999999999997,"5min":1.347,"15min":1.347},"last":0.42299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":4,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.61399999999999999,"15min":0.626},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83299999999999996,"5min":1.5,"15min":1.5},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.40000000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72799999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.54000000000000004}]},{"osd":5,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56000000000000005,"15min":0.56599999999999995},"min":{"1min":0.34999999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71099999999999997,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.63300000000000001,"15min":0.64700000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.96799999999999997,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.46000000000000002}]},{"osd":6,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.63100000000000001,"15min":0.63900000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88200000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.40999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76700000000000002,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.503}]},{"osd":7,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.64200000000000002,"15min":0.65600000000000003},"min":{"1min":0.36899999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.94399999999999995,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61599999999999999,"15min":0.622},"min":{"1min":0.36699999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85799999999999998,"5min":1.454,"15min":1.454},"last":0.55700000000000005}]}]},{"osd":2,"up_from":62,"seq":266287972361,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29736,"kb_used_data":1052,"kb_used_omap":5,"kb_used_meta":28666,"kb_avail":94342104,"statfs":{"total":96636764160,"available":96606314496,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6051,"internal_metadata":29354077},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.219}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]}]},{"osd":7,"up_from":14,"seq":60129542177,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1118208,"data_stored":982758,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.45200000000000001,"15min":0.45800000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.56299999999999994,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.42299999999999999,"15min":0.42399999999999999},"min":{"1min":0.29799999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60499999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.58299999999999996}]},{"osd":1,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45500000000000002,"15min":0.46100000000000002},"min":{"1min":0.29899999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.55200000000000005,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.44,"5min":0.44800000000000001,"15min":0.45000000000000001},"min":{"1min":0.28499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":3,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83199999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.49299999999999999,"15min":0.497},"min":{"1min":0.37,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.64600000000000002,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.66900000000000004}]},{"osd":4,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.42799999999999999,"15min":0.439},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.53600000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.47599999999999998,"15min":0.48799999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.60099999999999998,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.68799999999999994}]},{"osd":5,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.54400000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.433,"5min":0.51300000000000001,"15min":0.52700000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.65900000000000003,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.70799999999999996}]},{"osd":6,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.46500000000000002,"15min":0.47499999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.59299999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.52400000000000002,"15min":0.54000000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.71699999999999997,"5min":1.054,"15min":1.054},"last":0.65600000000000003}]}]},{"osd":6,"up_from":14,"seq":60129542177,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343776,"statfs":{"total":96636764160,"available":96608026624,"internally_reserved":0,"allocated":1064960,"data_stored":453830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.42699999999999999,"15min":0.42199999999999999},"min":{"1min":0.313,"5min":0.25,"15min":0.25},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.47399999999999998,"15min":0.47899999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.52400000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.69099999999999995}]},{"osd":1,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.57399999999999995,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.495,"15min":0.497},"min":{"1min":0.373,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.624,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.61499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":3,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68799999999999994,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66500000000000004,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.70699999999999996}]},{"osd":4,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.46200000000000002,"15min":0.46999999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57199999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.47899999999999998,"15min":0.48699999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.64900000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.46600000000000003}]},{"osd":5,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.28499999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.68799999999999994,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.5},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.313,"5min":0.308,"15min":0.308},"max":{"1min":0.65600000000000003,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.60399999999999998}]},{"osd":7,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53700000000000003,"15min":0.54100000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.51300000000000001,"15min":0.51700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64900000000000002,"5min":0.872,"15min":0.872},"last":0.57599999999999996}]}]},{"osd":4,"up_from":14,"seq":60129542177,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28068,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343772,"statfs":{"total":96636764160,"available":96608022528,"internally_reserved":0,"allocated":1064960,"data_stored":453830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.40500000000000003,"15min":0.40200000000000002},"min":{"1min":0.308,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.443,"5min":0.435,"15min":0.434},"min":{"1min":0.33600000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.64600000000000002,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66300000000000003,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46999999999999997,"15min":0.47299999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.91500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":3,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.51100000000000001,"15min":0.51900000000000002},"min":{"1min":0.38100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.58399999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.499,"15min":0.501},"min":{"1min":0.31900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.66900000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.59999999999999998}]},{"osd":5,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.44400000000000001,"15min":0.45000000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.64700000000000002,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.433,"5min":0.45100000000000001,"15min":0.45400000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64100000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.65700000000000003}]},{"osd":6,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66500000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.62},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.61199999999999999,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.47199999999999998}]},{"osd":7,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.49399999999999999,"15min":0.49199999999999999},"min":{"1min":0.34100000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71699999999999997,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.58599999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.50700000000000001,"15min":0.52900000000000003},"min":{"1min":0.219,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.127,"5min":3.2959999999999998,"15min":3.2959999999999998},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.56699999999999995,"15min":0.58499999999999996},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":1.161,"5min":3.2799999999999998,"15min":3.2799999999999998},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":3,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":3.3239999999999998,"15min":3.3239999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.47799999999999998,"15min":0.47699999999999998},"min":{"1min":0.25900000000000001,"5min":0.253,"15min":0.253},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.39100000000000001}]},{"osd":4,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.375,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59299999999999997,"15min":0.58899999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":0.63700000000000001}]},{"osd":5,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.39300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.628,"15min":0.624},"min":{"1min":0.376,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.66400000000000003}]},{"osd":6,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.61899999999999999,"15min":0.625},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0920000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.501},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.70399999999999996,"15min":0.71999999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1779999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":0.622}]},{"osd":7,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.667,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63800000000000001,"15min":0.64200000000000002},"min":{"1min":0.41599999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.57199999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542177,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1118208,"data_stored":982758,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.40500000000000003,"15min":0.40899999999999997},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.79800000000000004,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40100000000000002,"15min":0.40899999999999997},"min":{"1min":0.23200000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.64200000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.27700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25900000000000001}]},{"osd":3,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.46999999999999997,"15min":0.48199999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.59199999999999997,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.311},{"interface":"front","average":{"1min":0.441,"5min":0.48599999999999999,"15min":0.49399999999999999},"min":{"1min":0.28799999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.29899999999999999}]},{"osd":4,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.871,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.56599999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82699999999999996,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.46000000000000002}]},{"osd":5,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54200000000000004,"15min":0.53900000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.52600000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.56699999999999995}]},{"osd":6,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.38100000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59199999999999997,"15min":0.59599999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.50900000000000001}]},{"osd":7,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57299999999999995,"15min":0.56799999999999995},"min":{"1min":0.378,"5min":0.377,"15min":0.377},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.56299999999999994,"15min":0.56899999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.70899999999999996,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T21:42:08.215 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T21:42:08.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.329+0000 7fdf0ca9d640 10 monclient: get_monmap_and_config 2024-03-27T21:42:08.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.329+0000 7fdf0ca9d640 10 monclient: build_initial_monmap 2024-03-27T21:42:08.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.329+0000 7fdf0ca9d640 10 monclient: monmap: 2024-03-27T21:42:08.317 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:08.317 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:08.317 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:08.330493+0000 2024-03-27T21:42:08.317 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:08.330493+0000 2024-03-27T21:42:08.317 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:08.317 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:08.317 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:08.317 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:08.317 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:08.318 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:08.318 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:08.318 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.330+0000 7fdf0ca9d640 1 Processor -- start 2024-03-27T21:42:08.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.330+0000 7fdf0ca9d640 1 -- start start 2024-03-27T21:42:08.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.330+0000 7fdf0ca9d640 10 monclient: init 2024-03-27T21:42:08.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.330+0000 7fdf0ca9d640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:08.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.330+0000 7fdf0ca9d640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-03-27T21:42:08.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.330+0000 7fdf0ca9d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fdf08071a50 0x7fdf080a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.330+0000 7fdf0ca9d640 10 monclient(hunting): picked mon.noname-c con 0x7fdf08071a50 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:08.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.330+0000 7fdf0ca9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdf0813e810 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:08.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.330+0000 7fdf0ca9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf0813ec50 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:08.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.330+0000 7fdf0ca9d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf080a8e90 con 0x7fdf0813e810 2024-03-27T21:42:08.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.330+0000 7fdf0ca9d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf080aa040 con 0x7fdf0813ec50 2024-03-27T21:42:08.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.330+0000 7fdf0ca9d640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.330+0000 7fdf0ca9d640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf08071e30 con 0x7fdf08071a50 2024-03-27T21:42:08.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.330+0000 7fdf0ca9d640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:08.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.330+0000 7fdf0ca9d640 10 monclient(hunting): authenticate will time out at 969.551744s 2024-03-27T21:42:08.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.330+0000 7fdf05d9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdf0813e810 0x7fdf08141630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39478/0 (socket says 172.21.15.60:39478) 2024-03-27T21:42:08.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.330+0000 7fdf05d9b640 1 -- 172.21.15.60:0/278775673 learned_addr learned my addr 172.21.15.60:0/278775673 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:08.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 1 -- 172.21.15.60:0/278775673 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918680702 0 0) 0x7fdf080a8e90 con 0x7fdf0813e810 2024-03-27T21:42:08.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 10 monclient(hunting): _negotiate 2024-03-27T21:42:08.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:08.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:08.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 10 monclient(hunting): my global_id is 5409 2024-03-27T21:42:08.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 1 -- 172.21.15.60:0/278775673 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdef0003520 con 0x7fdf0813e810 2024-03-27T21:42:08.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0659c640 1 --2- 172.21.15.60:0/278775673 >> v2:172.21.15.191:3300/0 conn(0x7fdf08071a50 0x7fdf080a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0659c640 10 monclient(hunting): get_auth_request con 0x7fdf08071a50 auth_method 0 2024-03-27T21:42:08.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:08.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0659c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:08.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:08.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 1 -- 172.21.15.60:0/278775673 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2702825416 0 0) 0x7fdef0003520 con 0x7fdf0813e810 2024-03-27T21:42:08.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 1 -- 172.21.15.60:0/278775673 >> v2:172.21.15.191:3300/0 conn(0x7fdf08071a50 msgr2=0x7fdf080a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:08.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 1 --2- 172.21.15.60:0/278775673 >> v2:172.21.15.191:3300/0 conn(0x7fdf08071a50 0x7fdf080a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 1 -- 172.21.15.60:0/278775673 >> v1:172.21.15.191:6789/0 conn(0x7fdf0813ec50 legacy=0x7fdf08141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 10 monclient: _finish_hunting 0 2024-03-27T21:42:08.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 1 monclient: found mon.noname-a-legacy 2024-03-27T21:42:08.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:08.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.60:6789/0 2024-03-27T21:42:08.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 1 -- 172.21.15.60:0/278775673 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf080aa2b0 con 0x7fdf0813e810 2024-03-27T21:42:08.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 10 monclient: _finish_auth 0 2024-03-27T21:42:08.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 10 monclient: _check_auth_tickets 2024-03-27T21:42:08.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:08.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0ca9d640 5 monclient: authenticate success, global_id 5409 2024-03-27T21:42:08.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 1 -- 172.21.15.60:0/278775673 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2933500084 0 0) 0x7fdefc001670 con 0x7fdf0813e810 2024-03-27T21:42:08.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0ca9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:08.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0659c640 1 --2- 172.21.15.60:0/278775673 >> v2:172.21.15.191:3300/0 conn(0x7fdf08071a50 0x7fdf080a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:42:08.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:08.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-03-27T21:42:08.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 10 monclient: dump: 2024-03-27T21:42:08.337 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:08.337 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:08.337 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.337 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.338 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:08.338 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:08.338 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:08.338 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:08.338 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:08.338 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] but i'm connected to v1:172.21.15.60:6789/0, reconnecting 2024-03-27T21:42:08.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:08.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 1 -- 172.21.15.60:0/278775673 >> v1:172.21.15.60:6789/0 conn(0x7fdf0813e810 legacy=0x7fdf08141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:42:08.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 1 --2- 172.21.15.60:0/278775673 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdef0003970 0x7fdef0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 10 monclient(hunting): picked mon.b con 0x7fdef0003970 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:08.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 1 --2- 172.21.15.60:0/278775673 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdef00047c0 0x7fdef0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 10 monclient(hunting): picked mon.c con 0x7fdef00047c0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:08.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 1 --2- 172.21.15.60:0/278775673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef0005180 0x7fdef000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 10 monclient(hunting): picked mon.a con 0x7fdef0005180 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:08.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 1 -- 172.21.15.60:0/278775673 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf08071e30 con 0x7fdef0005180 2024-03-27T21:42:08.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 1 -- 172.21.15.60:0/278775673 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef000c0d0 con 0x7fdef0003970 2024-03-27T21:42:08.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 1 -- 172.21.15.60:0/278775673 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef000c290 con 0x7fdef00047c0 2024-03-27T21:42:08.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:08.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0559a640 10 monclient(hunting): _finish_auth 0 2024-03-27T21:42:08.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0ca9d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:08.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf05d9b640 1 --2- 172.21.15.60:0/278775673 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdef00047c0 0x7fdef0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf06d9d640 1 --2- 172.21.15.60:0/278775673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef0005180 0x7fdef000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf05d9b640 10 monclient(hunting): get_auth_request con 0x7fdef00047c0 auth_method 0 2024-03-27T21:42:08.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf05d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:08.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf05d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:08.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf05d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:08.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf06d9d640 10 monclient(hunting): get_auth_request con 0x7fdef0005180 auth_method 0 2024-03-27T21:42:08.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf06d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:08.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf06d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:08.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf06d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:08.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0659c640 1 --2- 172.21.15.60:0/278775673 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdef0003970 0x7fdef0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0659c640 10 monclient(hunting): get_auth_request con 0x7fdef0003970 auth_method 0 2024-03-27T21:42:08.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:08.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0659c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:08.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:08.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf06d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:08.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf06d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:08.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf06d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:42:08.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf05d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:08.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf05d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:08.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf05d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:42:08.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:08.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:08.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf0659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:42:08.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf06d9d640 10 monclient(hunting): handle_auth_done global_id 5409 payload 293 2024-03-27T21:42:08.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf06d9d640 1 -- 172.21.15.60:0/278775673 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdef00047c0 msgr2=0x7fdef0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:08.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf06d9d640 1 --2- 172.21.15.60:0/278775673 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdef00047c0 0x7fdef0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf06d9d640 1 -- 172.21.15.60:0/278775673 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdef0003970 msgr2=0x7fdef0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:08.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf06d9d640 1 --2- 172.21.15.60:0/278775673 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdef0003970 0x7fdef0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf06d9d640 10 monclient: _finish_hunting 0 2024-03-27T21:42:08.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf06d9d640 1 monclient: found mon.a 2024-03-27T21:42:08.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf06d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:08.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf06d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:08.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf06d9d640 1 -- 172.21.15.60:0/278775673 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef0003520 con 0x7fdef0005180 2024-03-27T21:42:08.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf06d9d640 10 monclient: _finish_auth 0 2024-03-27T21:42:08.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf06d9d640 10 monclient: _check_auth_tickets 2024-03-27T21:42:08.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf06d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:08.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.331+0000 7fdf05d9b640 1 --2- 172.21.15.60:0/278775673 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdef00047c0 0x7fdef0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:08.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.332+0000 7fdf06d9d640 1 -- 172.21.15.60:0/278775673 reap_dead start 2024-03-27T21:42:08.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.332+0000 7fdf06d9d640 1 --2- 172.21.15.60:0/278775673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef0005180 0x7fdef000b980 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdf00001e50 tx=0x7fdef00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc79743521aacdb server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:08.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.332+0000 7fdf0559a640 1 -- 172.21.15.60:0/278775673 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf00003b70 con 0x7fdef0005180 2024-03-27T21:42:08.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.332+0000 7fdf0559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:08.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.332+0000 7fdf0559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:08.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.332+0000 7fdf0559a640 10 monclient: dump: 2024-03-27T21:42:08.349 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:08.350 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:08.350 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.350 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.350 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:08.351 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:08.351 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:08.351 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:08.351 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:08.352 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.332+0000 7fdf0ca9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:08.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.332+0000 7fdf0559a640 1 -- 172.21.15.60:0/278775673 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf00003cf0 con 0x7fdef0005180 2024-03-27T21:42:08.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.332+0000 7fdf0559a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:08.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.332+0000 7fdf0ca9d640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:08.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.332+0000 7fdf0559a640 1 -- 172.21.15.60:0/278775673 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf00005310 con 0x7fdef0005180 2024-03-27T21:42:08.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.332+0000 7fdf0ca9d640 4 set_mon_vals no callback set 2024-03-27T21:42:08.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.332+0000 7fdf0559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:08.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.332+0000 7fdf0559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:08.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.332+0000 7fdf0559a640 10 monclient: dump: 2024-03-27T21:42:08.354 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:08.354 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:08.355 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.355 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.355 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:08.355 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:08.356 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:08.356 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:08.356 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:08.357 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.333+0000 7fdf0ca9d640 10 monclient: shutdown 2024-03-27T21:42:08.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.333+0000 7fdf0ca9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:08.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.333+0000 7fdf0ca9d640 1 -- 172.21.15.60:0/278775673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef0005180 msgr2=0x7fdef000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.333+0000 7fdf0ca9d640 1 --2- 172.21.15.60:0/278775673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef0005180 0x7fdef000b980 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdf00001e50 tx=0x7fdef00057f0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.333+0000 7fdf0ca9d640 1 -- 172.21.15.60:0/278775673 shutdown_connections 2024-03-27T21:42:08.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.333+0000 7fdf0ca9d640 1 --2- 172.21.15.60:0/278775673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef0005180 0x7fdef000b980 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.333+0000 7fdf0ca9d640 1 -- 172.21.15.60:0/278775673 >> 172.21.15.60:0/278775673 conn(0x7fdf080b7300 msgr2=0x7fdf080b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:08.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.333+0000 7fdf0ca9d640 1 -- 172.21.15.60:0/278775673 shutdown_connections 2024-03-27T21:42:08.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.333+0000 7fdf0ca9d640 1 -- 172.21.15.60:0/278775673 wait complete. 2024-03-27T21:42:08.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.334+0000 7fdf0ca9d640 10 monclient: build_initial_monmap 2024-03-27T21:42:08.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.334+0000 7fdf0ca9d640 10 monclient: monmap: 2024-03-27T21:42:08.360 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:08.360 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:08.360 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:08.361 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:08.361 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:08.361 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:08.361 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:08.362 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:08.362 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:08.362 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.334+0000 7fdf0ca9d640 1 librados: starting msgr at 2024-03-27T21:42:08.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.334+0000 7fdf0ca9d640 1 librados: starting objecter 2024-03-27T21:42:08.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.334+0000 7fdf0ca9d640 1 Processor -- start 2024-03-27T21:42:08.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.334+0000 7fdf0ca9d640 1 -- start start 2024-03-27T21:42:08.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.334+0000 7fdf0ca9d640 1 librados: setting wanted keys 2024-03-27T21:42:08.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.334+0000 7fdf0ca9d640 1 librados: calling monclient init 2024-03-27T21:42:08.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.334+0000 7fdf0ca9d640 10 monclient: init 2024-03-27T21:42:08.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0ca9d640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:08.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0ca9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:42:08.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0ca9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdef0005180 0x7fdf080b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0ca9d640 10 monclient(hunting): picked mon.noname-b con 0x7fdef0005180 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:08.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0ca9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf080ae260 0x7fdf080aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0ca9d640 10 monclient(hunting): picked mon.noname-a con 0x7fdf080ae260 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:08.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0ca9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdf081380a0 0x7fdf08138520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0ca9d640 10 monclient(hunting): picked mon.noname-c con 0x7fdf081380a0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:08.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0ca9d640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0ca9d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf080af1f0 con 0x7fdf080ae260 2024-03-27T21:42:08.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0ca9d640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0ca9d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf080ad540 con 0x7fdef0005180 2024-03-27T21:42:08.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0ca9d640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0ca9d640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf08138a60 con 0x7fdf081380a0 2024-03-27T21:42:08.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0ca9d640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:08.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0ca9d640 10 monclient(hunting): authenticate will time out at 969.556099s 2024-03-27T21:42:08.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf05d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf080ae260 0x7fdf080aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf06d9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdf081380a0 0x7fdf08138520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf05d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf080ae260 0x7fdf080aecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39638/0 (socket says 172.21.15.60:39638) 2024-03-27T21:42:08.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf05d9b640 1 -- 172.21.15.60:0/3615654434 learned_addr learned my addr 172.21.15.60:0/3615654434 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:08.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf05d9b640 10 monclient(hunting): get_auth_request con 0x7fdf080ae260 auth_method 0 2024-03-27T21:42:08.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf05d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:08.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf05d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:08.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf05d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:08.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf06d9d640 10 monclient(hunting): get_auth_request con 0x7fdf081380a0 auth_method 0 2024-03-27T21:42:08.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf06d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:08.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf06d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:08.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf06d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:08.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf06d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:08.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf06d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:08.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf06d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:08.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf05d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:08.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf05d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:08.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf05d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:08.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0659c640 1 --2- 172.21.15.60:0/3615654434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdef0005180 0x7fdf080b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0659c640 10 monclient(hunting): get_auth_request con 0x7fdef0005180 auth_method 0 2024-03-27T21:42:08.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:08.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0659c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:08.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:08.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf06d9d640 10 monclient(hunting): handle_auth_done global_id 5096 payload 824 2024-03-27T21:42:08.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf06d9d640 1 -- 172.21.15.60:0/3615654434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdef0005180 msgr2=0x7fdf080b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf06d9d640 1 --2- 172.21.15.60:0/3615654434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdef0005180 0x7fdf080b0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf06d9d640 1 -- 172.21.15.60:0/3615654434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf080ae260 msgr2=0x7fdf080aecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf06d9d640 1 --2- 172.21.15.60:0/3615654434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf080ae260 0x7fdf080aecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf06d9d640 10 monclient: _finish_hunting 0 2024-03-27T21:42:08.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf06d9d640 1 monclient: found mon.noname-c 2024-03-27T21:42:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf06d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf06d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf06d9d640 1 -- 172.21.15.60:0/3615654434 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf081d9770 con 0x7fdf081380a0 2024-03-27T21:42:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf05d9b640 1 --2- 172.21.15.60:0/3615654434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf080ae260 0x7fdf080aecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf06d9d640 1 --2- 172.21.15.60:0/3615654434 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdf081380a0 0x7fdf08138520 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdf080b11d0 tx=0x7fdf000014a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdef6ffd640 10 client.?.objecter ms_handle_connect 0x7fdf081380a0 2024-03-27T21:42:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdef6ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdef6ffd640 1 -- 172.21.15.60:0/3615654434 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf00006d50 con 0x7fdf081380a0 2024-03-27T21:42:08.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdef6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:08.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdef6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:08.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdef6ffd640 10 monclient: dump: 2024-03-27T21:42:08.374 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:08.374 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:08.374 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.374 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.374 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:08.375 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:08.375 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:08.375 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:08.375 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:08.375 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdef6ffd640 10 monclient: _finish_auth 0 2024-03-27T21:42:08.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdef6ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:08.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdef6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:08.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0ca9d640 5 monclient: authenticate success, global_id 5096 2024-03-27T21:42:08.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdef6ffd640 1 -- 172.21.15.60:0/3615654434 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf00039480 con 0x7fdf081380a0 2024-03-27T21:42:08.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0ca9d640 10 monclient: _renew_subs 2024-03-27T21:42:08.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0ca9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:08.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf0ca9d640 1 -- 172.21.15.60:0/3615654434 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf081d99c0 con 0x7fdf081380a0 2024-03-27T21:42:08.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdef6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:08.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdef6ffd640 1 -- 172.21.15.60:0/3615654434 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf00039830 con 0x7fdf081380a0 2024-03-27T21:42:08.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdef6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:08.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdef6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:08.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdef6ffd640 10 monclient: dump: 2024-03-27T21:42:08.377 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:08.377 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:08.377 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.377 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.377 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:08.378 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:08.378 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:08.378 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:08.378 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:08.378 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.336+0000 7fdf0ca9d640 10 client.5096.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:08.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.336+0000 7fdf0ca9d640 10 monclient: _renew_subs 2024-03-27T21:42:08.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.335+0000 7fdf04d99640 4 set_mon_vals no callback set 2024-03-27T21:42:08.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.336+0000 7fdf0ca9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:08.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.336+0000 7fdf0ca9d640 1 -- 172.21.15.60:0/3615654434 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf081d9ea0 con 0x7fdf081380a0 2024-03-27T21:42:08.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.336+0000 7fdf0ca9d640 1 librados: init done 2024-03-27T21:42:08.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.336+0000 7fdf0ca9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:08.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.336+0000 7fdf0ca9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:08.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.336+0000 7fdf0ca9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:08.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.336+0000 7fdf0ca9d640 1 -- 172.21.15.60:0/3615654434 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf080ae640 con 0x7fdf081380a0 2024-03-27T21:42:08.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.337+0000 7fdef6ffd640 1 -- 172.21.15.60:0/3615654434 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fdf00039990 con 0x7fdf081380a0 2024-03-27T21:42:08.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.337+0000 7fdef6ffd640 1 --2- 172.21.15.60:0/3615654434 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdee806c5c0 0x7fdee806ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.337+0000 7fdef6ffd640 1 -- 172.21.15.60:0/3615654434 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fdf000aaa80 con 0x7fdf081380a0 2024-03-27T21:42:08.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.337+0000 7fdef6ffd640 10 client.5096.objecter ms_dispatch 0x7fdf08000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:08.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.337+0000 7fdef6ffd640 3 client.5096.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:08.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.337+0000 7fdef6ffd640 3 client.5096.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:08.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.337+0000 7fdf0659c640 1 --2- 172.21.15.60:0/3615654434 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdee806c5c0 0x7fdee806ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.337+0000 7fdf0659c640 10 monclient: get_auth_request con 0x7fdee806c5c0 auth_method 0 2024-03-27T21:42:08.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.337+0000 7fdef6ffd640 20 client.5096.objecter dump_active .. 0 homeless 2024-03-27T21:42:08.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.338+0000 7fdef6ffd640 10 client.5096.objecter ms_handle_connect 0x7fdee806c5c0 2024-03-27T21:42:08.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.338+0000 7fdf0659c640 1 --2- 172.21.15.60:0/3615654434 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdee806c5c0 0x7fdee806ea80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdef8006fd0 tx=0x7fdef8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:08.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.342+0000 7fdef6ffd640 1 -- 172.21.15.60:0/3615654434 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdf00075ed0 con 0x7fdf081380a0 2024-03-27T21:42:08.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.342+0000 7fdef6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:08.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.342+0000 7fdef6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:08.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.520+0000 7fdf0ca9d640 1 -- 172.21.15.60:0/3615654434 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdf0813f0c0 con 0x7fdee806c5c0 2024-03-27T21:42:08.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.522+0000 7fdef6ffd640 1 -- 172.21.15.60:0/3615654434 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+66898 (secure 0 0 0) 0x7fdef8009dc0 con 0x7fdee806c5c0 2024-03-27T21:42:08.509 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:42:08.510 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-03-27T21:42:08.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.524+0000 7fdf0ca9d640 10 librados: watch_flush enter 2024-03-27T21:42:08.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.524+0000 7fdf0ca9d640 10 librados: watch_flush exit 2024-03-27T21:42:08.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.524+0000 7fdf0ca9d640 20 client.5096.objecter shutdown clearing up homeless session... 2024-03-27T21:42:08.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.524+0000 7fdf0ca9d640 10 client.5096.objecter successfully canceled tick 2024-03-27T21:42:08.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.524+0000 7fdf0ca9d640 1 -- 172.21.15.60:0/3615654434 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdee806c5c0 msgr2=0x7fdee806ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.524+0000 7fdf0ca9d640 1 --2- 172.21.15.60:0/3615654434 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdee806c5c0 0x7fdee806ea80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdef8006fd0 tx=0x7fdef8008040 comp rx=0 tx=0).stop 2024-03-27T21:42:08.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.524+0000 7fdf0ca9d640 10 monclient: shutdown 2024-03-27T21:42:08.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.524+0000 7fdf0ca9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:08.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.524+0000 7fdf0ca9d640 1 -- 172.21.15.60:0/3615654434 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdf081380a0 msgr2=0x7fdf08138520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.524+0000 7fdf0ca9d640 1 --2- 172.21.15.60:0/3615654434 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdf081380a0 0x7fdf08138520 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdf080b11d0 tx=0x7fdf000014a0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.524+0000 7fdf0ca9d640 1 -- 172.21.15.60:0/3615654434 shutdown_connections 2024-03-27T21:42:08.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.524+0000 7fdf0ca9d640 1 --2- 172.21.15.60:0/3615654434 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdf081380a0 0x7fdf08138520 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.524+0000 7fdf0ca9d640 1 --2- 172.21.15.60:0/3615654434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf080ae260 0x7fdf080aecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.524+0000 7fdf0ca9d640 1 --2- 172.21.15.60:0/3615654434 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdee806c5c0 0x7fdee806ea80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.524+0000 7fdf0ca9d640 1 --2- 172.21.15.60:0/3615654434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdef0005180 0x7fdf080b0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.524+0000 7fdf0ca9d640 1 -- 172.21.15.60:0/3615654434 >> 172.21.15.60:0/3615654434 conn(0x7fdf080b7300 msgr2=0x7fdf0813d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:08.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.524+0000 7fdf0ca9d640 1 -- 172.21.15.60:0/3615654434 shutdown_connections 2024-03-27T21:42:08.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.524+0000 7fdf0ca9d640 1 -- 172.21.15.60:0/3615654434 wait complete. 2024-03-27T21:42:08.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.524+0000 7fdf0ca9d640 1 librados: shutdown 2024-03-27T21:42:08.521 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-03-27T21:42:07.412709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":230404,"kb_used_data":8472,"kb_used_omap":16,"kb_used_meta":221807,"kb_avail":754744316,"statfs":{"total":773094113280,"available":772858179584,"internally_reserved":0,"allocated":8675328,"data_stored":4710296,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":17199,"internal_metadata":227130577},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000861"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-27T21:42:02.414394+0000","last_change":"2024-03-27T21:42:01.265365+0000","last_active":"2024-03-27T21:42:02.414394+0000","last_peered":"2024-03-27T21:42:02.414394+0000","last_clean":"2024-03-27T21:42:02.414394+0000","last_became_active":"2024-03-27T21:42:01.265033+0000","last_became_peered":"2024-03-27T21:42:01.265033+0000","last_unstale":"2024-03-27T21:42:02.414394+0000","last_undegraded":"2024-03-27T21:42:02.414394+0000","last_fullsized":"2024-03-27T21:42:02.414394+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:07:09.198158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.601878+0000","last_change":"2024-03-27T21:41:49.474318+0000","last_active":"2024-03-27T21:42:03.601878+0000","last_peered":"2024-03-27T21:42:03.601878+0000","last_clean":"2024-03-27T21:42:03.601878+0000","last_became_active":"2024-03-27T21:41:49.474054+0000","last_became_peered":"2024-03-27T21:41:49.474054+0000","last_unstale":"2024-03-27T21:42:03.601878+0000","last_undegraded":"2024-03-27T21:42:03.601878+0000","last_fullsized":"2024-03-27T21:42:03.601878+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:36:26.918691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271058+0000","last_change":"2024-03-27T21:39:46.679788+0000","last_active":"2024-03-27T21:42:03.271058+0000","last_peered":"2024-03-27T21:42:03.271058+0000","last_clean":"2024-03-27T21:42:03.271058+0000","last_became_active":"2024-03-27T21:39:46.679561+0000","last_became_peered":"2024-03-27T21:39:46.679561+0000","last_unstale":"2024-03-27T21:42:03.271058+0000","last_undegraded":"2024-03-27T21:42:03.271058+0000","last_fullsized":"2024-03-27T21:42:03.271058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T23:53:50.373923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.601870+0000","last_change":"2024-03-27T21:41:49.474121+0000","last_active":"2024-03-27T21:42:03.601870+0000","last_peered":"2024-03-27T21:42:03.601870+0000","last_clean":"2024-03-27T21:42:03.601870+0000","last_became_active":"2024-03-27T21:41:49.473822+0000","last_became_peered":"2024-03-27T21:41:49.473822+0000","last_unstale":"2024-03-27T21:42:03.601870+0000","last_undegraded":"2024-03-27T21:42:03.601870+0000","last_fullsized":"2024-03-27T21:42:03.601870+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:39:51.253704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271086+0000","last_change":"2024-03-27T21:41:59.701200+0000","last_active":"2024-03-27T21:42:03.271086+0000","last_peered":"2024-03-27T21:42:03.271086+0000","last_clean":"2024-03-27T21:42:03.271086+0000","last_became_active":"2024-03-27T21:41:59.700966+0000","last_became_peered":"2024-03-27T21:41:59.700966+0000","last_unstale":"2024-03-27T21:42:03.271086+0000","last_undegraded":"2024-03-27T21:42:03.271086+0000","last_fullsized":"2024-03-27T21:42:03.271086+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:17:01.025558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.270838+0000","last_change":"2024-03-27T21:41:30.080153+0000","last_active":"2024-03-27T21:42:03.270838+0000","last_peered":"2024-03-27T21:42:03.270838+0000","last_clean":"2024-03-27T21:42:03.270838+0000","last_became_active":"2024-03-27T21:41:30.079829+0000","last_became_peered":"2024-03-27T21:41:30.079829+0000","last_unstale":"2024-03-27T21:42:03.270838+0000","last_undegraded":"2024-03-27T21:42:03.270838+0000","last_fullsized":"2024-03-27T21:42:03.270838+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:21:00.966562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":97,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271058+0000","last_change":"2024-03-27T21:41:49.197510+0000","last_active":"2024-03-27T21:42:03.271058+0000","last_peered":"2024-03-27T21:42:03.271058+0000","last_clean":"2024-03-27T21:42:03.271058+0000","last_became_active":"2024-03-27T21:41:49.197154+0000","last_became_peered":"2024-03-27T21:41:49.197154+0000","last_unstale":"2024-03-27T21:42:03.271058+0000","last_undegraded":"2024-03-27T21:42:03.271058+0000","last_fullsized":"2024-03-27T21:42:03.271058+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:18:21.622988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271153+0000","last_change":"2024-03-27T21:41:59.247145+0000","last_active":"2024-03-27T21:42:03.271153+0000","last_peered":"2024-03-27T21:42:03.271153+0000","last_clean":"2024-03-27T21:42:03.271153+0000","last_became_active":"2024-03-27T21:41:59.246895+0000","last_became_peered":"2024-03-27T21:41:59.246895+0000","last_unstale":"2024-03-27T21:42:03.271153+0000","last_undegraded":"2024-03-27T21:42:03.271153+0000","last_fullsized":"2024-03-27T21:42:03.271153+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:40:39.749875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":234,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271065+0000","last_change":"2024-03-27T21:39:39.866806+0000","last_active":"2024-03-27T21:42:03.271065+0000","last_peered":"2024-03-27T21:42:03.271065+0000","last_clean":"2024-03-27T21:42:03.271065+0000","last_became_active":"2024-03-27T21:39:39.865948+0000","last_became_peered":"2024-03-27T21:39:39.865948+0000","last_unstale":"2024-03-27T21:42:03.271065+0000","last_undegraded":"2024-03-27T21:42:03.271065+0000","last_fullsized":"2024-03-27T21:42:03.271065+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_clean_scrub_stamp":"2024-03-27T21:39:38.411432+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:36:49.375869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":16,"seq":68719476771,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.41599999999999998,"15min":0.41199999999999998},"min":{"1min":0.34399999999999997,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.621,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.498},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.47499999999999998,"15min":0.47599999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.66900000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.70499999999999996}]},{"osd":1,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45900000000000002,"15min":0.46500000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.53600000000000003,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.35899999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.56599999999999995,"5min":0.745,"15min":0.745},"last":0.371}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":3,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.50600000000000001,"15min":0.51400000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.56899999999999995,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.35999999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68400000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.56000000000000005}]},{"osd":4,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.432,"15min":0.439},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.56399999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.433,"5min":0.46800000000000003,"15min":0.47399999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.78400000000000003}]},{"osd":6,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.45800000000000002,"15min":0.46300000000000002},"min":{"1min":0.318,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58699999999999997,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.29299999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64300000000000002,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.73999999999999999}]},{"osd":7,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.52400000000000002,"15min":0.53200000000000003},"min":{"1min":0.378,"5min":0.318,"15min":0.318},"max":{"1min":0.61299999999999999,"5min":0.745,"15min":0.745},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.50800000000000001,"15min":0.51400000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65000000000000002,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.72399999999999998}]}]},{"osd":3,"up_from":16,"seq":68719476770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31848905},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.49299999999999999,"15min":0.504},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.91700000000000004,"5min":1.248,"15min":1.248},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.44900000000000001,"15min":0.45600000000000002},"min":{"1min":0.22800000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.90600000000000003,"5min":1.407,"15min":1.407},"last":0.36599999999999999}]},{"osd":1,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48199999999999998,"15min":0.47999999999999998},"min":{"1min":0.25600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.26800000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.96899999999999997,"5min":1.347,"15min":1.347},"last":0.42299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":4,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.61399999999999999,"15min":0.626},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83299999999999996,"5min":1.5,"15min":1.5},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.40000000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72799999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.54000000000000004}]},{"osd":5,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56000000000000005,"15min":0.56599999999999995},"min":{"1min":0.34999999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71099999999999997,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.63300000000000001,"15min":0.64700000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.96799999999999997,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.46000000000000002}]},{"osd":6,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.63100000000000001,"15min":0.63900000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88200000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.40999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76700000000000002,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.503}]},{"osd":7,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.64200000000000002,"15min":0.65600000000000003},"min":{"1min":0.36899999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.94399999999999995,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61599999999999999,"15min":0.622},"min":{"1min":0.36699999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85799999999999998,"5min":1.454,"15min":1.454},"last":0.55700000000000005}]}]},{"osd":2,"up_from":62,"seq":266287972361,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29736,"kb_used_data":1052,"kb_used_omap":5,"kb_used_meta":28666,"kb_avail":94342104,"statfs":{"total":96636764160,"available":96606314496,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6051,"internal_metadata":29354077},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.219}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]}]},{"osd":7,"up_from":14,"seq":60129542177,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1118208,"data_stored":982758,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.45200000000000001,"15min":0.45800000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.56299999999999994,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.42299999999999999,"15min":0.42399999999999999},"min":{"1min":0.29799999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60499999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.58299999999999996}]},{"osd":1,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45500000000000002,"15min":0.46100000000000002},"min":{"1min":0.29899999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.55200000000000005,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.44,"5min":0.44800000000000001,"15min":0.45000000000000001},"min":{"1min":0.28499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":3,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83199999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.49299999999999999,"15min":0.497},"min":{"1min":0.37,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.64600000000000002,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.66900000000000004}]},{"osd":4,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.42799999999999999,"15min":0.439},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.53600000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.47599999999999998,"15min":0.48799999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.60099999999999998,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.68799999999999994}]},{"osd":5,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.54400000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.433,"5min":0.51300000000000001,"15min":0.52700000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.65900000000000003,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.70799999999999996}]},{"osd":6,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.46500000000000002,"15min":0.47499999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.59299999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.52400000000000002,"15min":0.54000000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.71699999999999997,"5min":1.054,"15min":1.054},"last":0.65600000000000003}]}]},{"osd":6,"up_from":14,"seq":60129542177,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343776,"statfs":{"total":96636764160,"available":96608026624,"internally_reserved":0,"allocated":1064960,"data_stored":453830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.42699999999999999,"15min":0.42199999999999999},"min":{"1min":0.313,"5min":0.25,"15min":0.25},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.47399999999999998,"15min":0.47899999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.52400000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.69099999999999995}]},{"osd":1,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.57399999999999995,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.495,"15min":0.497},"min":{"1min":0.373,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.624,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.61499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":3,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68799999999999994,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66500000000000004,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.70699999999999996}]},{"osd":4,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.46200000000000002,"15min":0.46999999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57199999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.47899999999999998,"15min":0.48699999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.64900000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.46600000000000003}]},{"osd":5,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.28499999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.68799999999999994,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.5},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.313,"5min":0.308,"15min":0.308},"max":{"1min":0.65600000000000003,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.60399999999999998}]},{"osd":7,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53700000000000003,"15min":0.54100000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.51300000000000001,"15min":0.51700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64900000000000002,"5min":0.872,"15min":0.872},"last":0.57599999999999996}]}]},{"osd":4,"up_from":14,"seq":60129542177,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28068,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343772,"statfs":{"total":96636764160,"available":96608022528,"internally_reserved":0,"allocated":1064960,"data_stored":453830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.40500000000000003,"15min":0.40200000000000002},"min":{"1min":0.308,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.443,"5min":0.435,"15min":0.434},"min":{"1min":0.33600000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.64600000000000002,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66300000000000003,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46999999999999997,"15min":0.47299999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.91500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":3,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.51100000000000001,"15min":0.51900000000000002},"min":{"1min":0.38100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.58399999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.499,"15min":0.501},"min":{"1min":0.31900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.66900000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.59999999999999998}]},{"osd":5,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.44400000000000001,"15min":0.45000000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.64700000000000002,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.433,"5min":0.45100000000000001,"15min":0.45400000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64100000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.65700000000000003}]},{"osd":6,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66500000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.62},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.61199999999999999,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.47199999999999998}]},{"osd":7,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.49399999999999999,"15min":0.49199999999999999},"min":{"1min":0.34100000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71699999999999997,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.58599999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.50700000000000001,"15min":0.52900000000000003},"min":{"1min":0.219,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.127,"5min":3.2959999999999998,"15min":3.2959999999999998},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.56699999999999995,"15min":0.58499999999999996},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":1.161,"5min":3.2799999999999998,"15min":3.2799999999999998},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":3,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":3.3239999999999998,"15min":3.3239999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.47799999999999998,"15min":0.47699999999999998},"min":{"1min":0.25900000000000001,"5min":0.253,"15min":0.253},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.39100000000000001}]},{"osd":4,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.375,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59299999999999997,"15min":0.58899999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":0.63700000000000001}]},{"osd":5,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.39300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.628,"15min":0.624},"min":{"1min":0.376,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.66400000000000003}]},{"osd":6,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.61899999999999999,"15min":0.625},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0920000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.501},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.70399999999999996,"15min":0.71999999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1779999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":0.622}]},{"osd":7,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.667,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63800000000000001,"15min":0.64200000000000002},"min":{"1min":0.41599999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.57199999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542177,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1118208,"data_stored":982758,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.40500000000000003,"15min":0.40899999999999997},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.79800000000000004,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40100000000000002,"15min":0.40899999999999997},"min":{"1min":0.23200000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.64200000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.27700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25900000000000001}]},{"osd":3,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.46999999999999997,"15min":0.48199999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.59199999999999997,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.311},{"interface":"front","average":{"1min":0.441,"5min":0.48599999999999999,"15min":0.49399999999999999},"min":{"1min":0.28799999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.29899999999999999}]},{"osd":4,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.871,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.56599999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82699999999999996,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.46000000000000002}]},{"osd":5,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54200000000000004,"15min":0.53900000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.52600000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.56699999999999995}]},{"osd":6,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.38100000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59199999999999997,"15min":0.59599999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.50900000000000001}]},{"osd":7,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57299999999999995,"15min":0.56799999999999995},"min":{"1min":0.378,"5min":0.377,"15min":0.377},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.56299999999999994,"15min":0.56899999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.70899999999999996,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T21:42:08.523 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-27T21:42:08.523 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-03-27T21:42:08.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.638+0000 7fc949c2f640 10 monclient: get_monmap_and_config 2024-03-27T21:42:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.638+0000 7fc949c2f640 10 monclient: build_initial_monmap 2024-03-27T21:42:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.638+0000 7fc949c2f640 10 monclient: monmap: 2024-03-27T21:42:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:08.639775+0000 2024-03-27T21:42:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:08.639775+0000 2024-03-27T21:42:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:08.626 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.639+0000 7fc949c2f640 1 Processor -- start 2024-03-27T21:42:08.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.639+0000 7fc949c2f640 1 -- start start 2024-03-27T21:42:08.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.639+0000 7fc949c2f640 10 monclient: init 2024-03-27T21:42:08.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc949c2f640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:08.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc949c2f640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-03-27T21:42:08.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc949c2f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc944071a50 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:08.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc949c2f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc94413ef20 0x7fc9440a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc949c2f640 10 monclient(hunting): picked mon.noname-a con 0x7fc94413ef20 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:08.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc949c2f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc944141630 0x7fc944141ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc949c2f640 10 monclient(hunting): picked mon.noname-b con 0x7fc944141630 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:08.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc949c2f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9440a8e90 con 0x7fc944071a50 2024-03-27T21:42:08.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc949c2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc949c2f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc944071e30 con 0x7fc94413ef20 2024-03-27T21:42:08.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc949c2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc949c2f640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9440aa040 con 0x7fc944141630 2024-03-27T21:42:08.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc949c2f640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:08.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc949c2f640 10 monclient(hunting): authenticate will time out at 969.861046s 2024-03-27T21:42:08.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc943fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc944141630 0x7fc944141ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc942ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc94413ef20 0x7fc9440a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc943fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc944141630 0x7fc944141ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34432/0 (socket says 172.21.15.60:34432) 2024-03-27T21:42:08.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc942ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc94413ef20 0x7fc9440a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39642/0 (socket says 172.21.15.60:39642) 2024-03-27T21:42:08.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc943fff640 1 -- 172.21.15.60:0/1241158787 learned_addr learned my addr 172.21.15.60:0/1241158787 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:08.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc943fff640 10 monclient(hunting): get_auth_request con 0x7fc944141630 auth_method 0 2024-03-27T21:42:08.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc943fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:08.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc943fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:08.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc943fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:08.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc942ffd640 10 monclient(hunting): get_auth_request con 0x7fc94413ef20 auth_method 0 2024-03-27T21:42:08.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc942ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:08.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc942ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:08.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc942ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:08.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc943fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:08.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc943fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:08.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc943fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:08.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc942ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:08.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc942ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:08.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc942ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:08.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc942ffd640 10 monclient(hunting): handle_auth_done global_id 5415 payload 274 2024-03-27T21:42:08.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc942ffd640 1 -- 172.21.15.60:0/1241158787 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc944141630 msgr2=0x7fc944141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:08.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc942ffd640 1 --2- 172.21.15.60:0/1241158787 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc944141630 0x7fc944141ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc942ffd640 1 -- 172.21.15.60:0/1241158787 >> v1:172.21.15.191:6789/0 conn(0x7fc944071a50 legacy=0x7fc94413e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc942ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:42:08.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc942ffd640 1 monclient: found mon.noname-a 2024-03-27T21:42:08.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc942ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:08.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc942ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:08.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc942ffd640 1 -- 172.21.15.60:0/1241158787 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9440aa2b0 con 0x7fc94413ef20 2024-03-27T21:42:08.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.640+0000 7fc943fff640 1 --2- 172.21.15.60:0/1241158787 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc944141630 0x7fc944141ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:08.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc942ffd640 1 --2- 172.21.15.60:0/1241158787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc94413ef20 0x7fc9440a87e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc9340097a0 tx=0x7fc93402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d4e80c1649f409 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:08.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc9427fc640 1 -- 172.21.15.60:0/1241158787 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93402d040 con 0x7fc94413ef20 2024-03-27T21:42:08.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc9427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:08.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc9427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:08.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc9427fc640 10 monclient: dump: 2024-03-27T21:42:08.644 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:08.644 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:08.644 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.645 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.645 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:08.645 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:08.645 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:08.645 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:08.645 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:08.645 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc9427fc640 10 monclient: _finish_auth 0 2024-03-27T21:42:08.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc9427fc640 10 monclient: _check_auth_tickets 2024-03-27T21:42:08.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc9427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:08.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc949c2f640 5 monclient: authenticate success, global_id 5415 2024-03-27T21:42:08.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc949c2f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:08.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc9427fc640 1 -- 172.21.15.60:0/1241158787 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc934031810 con 0x7fc94413ef20 2024-03-27T21:42:08.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc9427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:08.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc949c2f640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:08.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc949c2f640 4 set_mon_vals no callback set 2024-03-27T21:42:08.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc9427fc640 1 -- 172.21.15.60:0/1241158787 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc934031ba0 con 0x7fc94413ef20 2024-03-27T21:42:08.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc9427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:08.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc9427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:08.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc9427fc640 10 monclient: dump: 2024-03-27T21:42:08.647 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:08.647 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:08.647 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.647 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.647 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:08.648 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:08.648 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:08.648 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:08.648 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:08.648 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc949c2f640 10 monclient: shutdown 2024-03-27T21:42:08.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc949c2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:08.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc949c2f640 1 -- 172.21.15.60:0/1241158787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc94413ef20 msgr2=0x7fc9440a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.641+0000 7fc949c2f640 1 --2- 172.21.15.60:0/1241158787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc94413ef20 0x7fc9440a87e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc9340097a0 tx=0x7fc93402b8b0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.642+0000 7fc949c2f640 1 -- 172.21.15.60:0/1241158787 shutdown_connections 2024-03-27T21:42:08.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.642+0000 7fc949c2f640 1 --2- 172.21.15.60:0/1241158787 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc944141630 0x7fc944141ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.642+0000 7fc949c2f640 1 --2- 172.21.15.60:0/1241158787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc94413ef20 0x7fc9440a87e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.642+0000 7fc949c2f640 1 -- 172.21.15.60:0/1241158787 >> 172.21.15.60:0/1241158787 conn(0x7fc9440b7300 msgr2=0x7fc9440b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:08.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.642+0000 7fc949c2f640 1 -- 172.21.15.60:0/1241158787 shutdown_connections 2024-03-27T21:42:08.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.642+0000 7fc949c2f640 1 -- 172.21.15.60:0/1241158787 wait complete. 2024-03-27T21:42:08.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.642+0000 7fc949c2f640 10 monclient: build_initial_monmap 2024-03-27T21:42:08.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.642+0000 7fc949c2f640 10 monclient: monmap: 2024-03-27T21:42:08.650 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:08.650 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:08.650 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:08.650 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:08.650 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:08.650 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:08.651 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:08.651 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:08.651 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:08.651 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.642+0000 7fc949c2f640 1 librados: starting msgr at 2024-03-27T21:42:08.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.642+0000 7fc949c2f640 1 librados: starting objecter 2024-03-27T21:42:08.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.642+0000 7fc949c2f640 1 Processor -- start 2024-03-27T21:42:08.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc949c2f640 1 -- start start 2024-03-27T21:42:08.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc949c2f640 1 librados: setting wanted keys 2024-03-27T21:42:08.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc949c2f640 1 librados: calling monclient init 2024-03-27T21:42:08.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc949c2f640 10 monclient: init 2024-03-27T21:42:08.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc949c2f640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:08.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc949c2f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:42:08.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc949c2f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc944071a50 0x7fc944137530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc949c2f640 10 monclient(hunting): picked mon.noname-b con 0x7fc944071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:08.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc949c2f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc94413ef20 0x7fc9441364b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc949c2f640 10 monclient(hunting): picked mon.noname-a con 0x7fc94413ef20 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:08.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc949c2f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc944141630 0x7fc944138fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc949c2f640 10 monclient(hunting): picked mon.noname-c con 0x7fc944141630 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:08.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc949c2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc949c2f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94413b120 con 0x7fc94413ef20 2024-03-27T21:42:08.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc949c2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc949c2f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9441398e0 con 0x7fc944071a50 2024-03-27T21:42:08.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc949c2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc949c2f640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9440aab50 con 0x7fc944141630 2024-03-27T21:42:08.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc949c2f640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:08.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc949c2f640 10 monclient(hunting): authenticate will time out at 969.864638s 2024-03-27T21:42:08.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc942ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc94413ef20 0x7fc9441364b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc943fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc944141630 0x7fc944138fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc942ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc94413ef20 0x7fc9441364b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39646/0 (socket says 172.21.15.60:39646) 2024-03-27T21:42:08.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc942ffd640 1 -- 172.21.15.60:0/694630280 learned_addr learned my addr 172.21.15.60:0/694630280 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:08.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc942ffd640 10 monclient(hunting): get_auth_request con 0x7fc94413ef20 auth_method 0 2024-03-27T21:42:08.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc942ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:08.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc942ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:08.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc942ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:08.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc943fff640 10 monclient(hunting): get_auth_request con 0x7fc944141630 auth_method 0 2024-03-27T21:42:08.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc943fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:08.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc943fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:08.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc943fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:08.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc942ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:08.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc942ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:08.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc942ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:08.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc943fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:08.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc943fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:08.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.643+0000 7fc943fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:08.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc942ffd640 10 monclient(hunting): handle_auth_done global_id 5418 payload 824 2024-03-27T21:42:08.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc9437fe640 1 --2- 172.21.15.60:0/694630280 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc944071a50 0x7fc944137530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc942ffd640 1 -- 172.21.15.60:0/694630280 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc944141630 msgr2=0x7fc944138fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc942ffd640 1 --2- 172.21.15.60:0/694630280 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc944141630 0x7fc944138fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc942ffd640 1 -- 172.21.15.60:0/694630280 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc944071a50 msgr2=0x7fc944137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc942ffd640 1 --2- 172.21.15.60:0/694630280 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc944071a50 0x7fc944137530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc942ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:42:08.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc942ffd640 1 monclient: found mon.noname-a 2024-03-27T21:42:08.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc942ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:08.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc942ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:08.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc942ffd640 1 -- 172.21.15.60:0/694630280 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc93c009b30 con 0x7fc94413ef20 2024-03-27T21:42:08.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc9437fe640 10 monclient: get_auth_request con 0x7fc944071a50 auth_method 0 2024-03-27T21:42:08.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc943fff640 1 --2- 172.21.15.60:0/694630280 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc944141630 0x7fc944138fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:08.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc9437fe640 1 --2- 172.21.15.60:0/694630280 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc944071a50 0x7fc944137530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:42:08.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc942ffd640 1 --2- 172.21.15.60:0/694630280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc94413ef20 0x7fc9441364b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc9340098c0 tx=0x7fc9340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:08.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc92bfff640 10 client.?.objecter ms_handle_connect 0x7fc94413ef20 2024-03-27T21:42:08.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc92bfff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:08.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc92bfff640 1 -- 172.21.15.60:0/694630280 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc934006e10 con 0x7fc94413ef20 2024-03-27T21:42:08.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc92bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:08.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc92bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:08.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc92bfff640 10 monclient: dump: 2024-03-27T21:42:08.658 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:08.658 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:08.658 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.659 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.659 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:08.659 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:08.659 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:08.659 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:08.659 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:08.659 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc92bfff640 10 monclient: _finish_auth 0 2024-03-27T21:42:08.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc92bfff640 10 monclient: _check_auth_tickets 2024-03-27T21:42:08.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc92bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:08.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc92bfff640 1 -- 172.21.15.60:0/694630280 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93403ad60 con 0x7fc94413ef20 2024-03-27T21:42:08.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc949c2f640 5 monclient: authenticate success, global_id 5418 2024-03-27T21:42:08.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc92bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:08.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc92bfff640 1 -- 172.21.15.60:0/694630280 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc934044620 con 0x7fc94413ef20 2024-03-27T21:42:08.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc949c2f640 10 monclient: _renew_subs 2024-03-27T21:42:08.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc949c2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:08.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc949c2f640 1 -- 172.21.15.60:0/694630280 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc934009400 con 0x7fc94413ef20 2024-03-27T21:42:08.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc92bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:08.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc92bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:08.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc941ffb640 4 set_mon_vals no callback set 2024-03-27T21:42:08.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc92bfff640 10 monclient: dump: 2024-03-27T21:42:08.661 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:08.661 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:08.661 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.662 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.662 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:08.662 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:08.662 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:08.662 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:08.662 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:08.662 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc949c2f640 10 client.5418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:08.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc949c2f640 10 monclient: _renew_subs 2024-03-27T21:42:08.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc949c2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:08.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc949c2f640 1 -- 172.21.15.60:0/694630280 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc944139de0 con 0x7fc94413ef20 2024-03-27T21:42:08.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.644+0000 7fc949c2f640 1 librados: init done 2024-03-27T21:42:08.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.645+0000 7fc949c2f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:08.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.645+0000 7fc949c2f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:08.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.645+0000 7fc949c2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:08.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.645+0000 7fc949c2f640 1 -- 172.21.15.60:0/694630280 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc944067900 con 0x7fc94413ef20 2024-03-27T21:42:08.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.645+0000 7fc92bfff640 1 -- 172.21.15.60:0/694630280 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc93404d440 con 0x7fc94413ef20 2024-03-27T21:42:08.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.646+0000 7fc92bfff640 1 --2- 172.21.15.60:0/694630280 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc93006c5c0 0x7fc93006ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.646+0000 7fc92bfff640 1 -- 172.21.15.60:0/694630280 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fc9340ab350 con 0x7fc94413ef20 2024-03-27T21:42:08.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.646+0000 7fc92bfff640 10 client.5418.objecter ms_dispatch 0x7fc944000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:08.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.646+0000 7fc92bfff640 3 client.5418.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:08.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.646+0000 7fc92bfff640 3 client.5418.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:08.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.646+0000 7fc9437fe640 1 --2- 172.21.15.60:0/694630280 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc93006c5c0 0x7fc93006ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.646+0000 7fc9437fe640 10 monclient: get_auth_request con 0x7fc93006c5c0 auth_method 0 2024-03-27T21:42:08.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.646+0000 7fc92bfff640 20 client.5418.objecter dump_active .. 0 homeless 2024-03-27T21:42:08.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.646+0000 7fc92bfff640 10 client.5418.objecter ms_handle_connect 0x7fc93006c5c0 2024-03-27T21:42:08.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.646+0000 7fc9437fe640 1 --2- 172.21.15.60:0/694630280 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc93006c5c0 0x7fc93006ea80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc938000f60 tx=0x7fc938009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:08.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.650+0000 7fc92bfff640 1 -- 172.21.15.60:0/694630280 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc934076790 con 0x7fc94413ef20 2024-03-27T21:42:08.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.650+0000 7fc92bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:08.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.650+0000 7fc92bfff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:08.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.832+0000 7fc949c2f640 10 start_mon_command cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}] 2024-03-27T21:42:08.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.832+0000 7fc949c2f640 10 monclient: _send_command 2 [{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}] 2024-03-27T21:42:08.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.832+0000 7fc949c2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:08.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.832+0000 7fc949c2f640 1 -- 172.21.15.60:0/694630280 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} v 0) v1 -- 0x7fc9441396f0 con 0x7fc94413ef20 2024-03-27T21:42:08.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.835+0000 7fc92bfff640 1 -- 172.21.15.60:0/694630280 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]=0 v6) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc934079db0 con 0x7fc94413ef20 2024-03-27T21:42:08.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.835+0000 7fc92bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}] 2024-03-27T21:42:08.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.835+0000 7fc92bfff640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:42:08.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.837+0000 7fc949c2f640 10 librados: watch_flush enter 2024-03-27T21:42:08.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.837+0000 7fc949c2f640 10 librados: watch_flush exit 2024-03-27T21:42:08.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.837+0000 7fc949c2f640 20 client.5418.objecter shutdown clearing up homeless session... 2024-03-27T21:42:08.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.837+0000 7fc949c2f640 10 client.5418.objecter successfully canceled tick 2024-03-27T21:42:08.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.837+0000 7fc949c2f640 1 -- 172.21.15.60:0/694630280 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc93006c5c0 msgr2=0x7fc93006ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.837+0000 7fc949c2f640 1 --2- 172.21.15.60:0/694630280 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc93006c5c0 0x7fc93006ea80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc938000f60 tx=0x7fc938009040 comp rx=0 tx=0).stop 2024-03-27T21:42:08.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.837+0000 7fc949c2f640 10 monclient: shutdown 2024-03-27T21:42:08.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.837+0000 7fc949c2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:08.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.837+0000 7fc949c2f640 1 -- 172.21.15.60:0/694630280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc94413ef20 msgr2=0x7fc9441364b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.837+0000 7fc949c2f640 1 --2- 172.21.15.60:0/694630280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc94413ef20 0x7fc9441364b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc9340098c0 tx=0x7fc9340046c0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.837+0000 7fc949c2f640 1 -- 172.21.15.60:0/694630280 shutdown_connections 2024-03-27T21:42:08.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.837+0000 7fc949c2f640 1 --2- 172.21.15.60:0/694630280 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fc944141630 0x7fc944138fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.837+0000 7fc949c2f640 1 --2- 172.21.15.60:0/694630280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc94413ef20 0x7fc9441364b0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.837+0000 7fc949c2f640 1 --2- 172.21.15.60:0/694630280 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fc93006c5c0 0x7fc93006ea80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.837+0000 7fc949c2f640 1 --2- 172.21.15.60:0/694630280 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc944071a50 0x7fc944137530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.837+0000 7fc949c2f640 1 -- 172.21.15.60:0/694630280 >> 172.21.15.60:0/694630280 conn(0x7fc9440b7300 msgr2=0x7fc94413e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:08.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.838+0000 7fc949c2f640 1 -- 172.21.15.60:0/694630280 shutdown_connections 2024-03-27T21:42:08.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.838+0000 7fc949c2f640 1 -- 172.21.15.60:0/694630280 wait complete. 2024-03-27T21:42:08.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.838+0000 7fc949c2f640 1 librados: shutdown 2024-03-27T21:42:08.835 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-27T21:42:08.844 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-27T21:42:08.844 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T21:42:08.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.958+0000 7fcf6d82f640 10 monclient: get_monmap_and_config 2024-03-27T21:42:08.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.958+0000 7fcf6d82f640 10 monclient: build_initial_monmap 2024-03-27T21:42:08.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.958+0000 7fcf6d82f640 10 monclient: monmap: 2024-03-27T21:42:08.945 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:08.945 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:08.945 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:08.959634+0000 2024-03-27T21:42:08.945 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:08.959634+0000 2024-03-27T21:42:08.945 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:08.945 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:08.946 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:08.946 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:08.946 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:08.946 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:08.946 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:08.946 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf6d82f640 1 Processor -- start 2024-03-27T21:42:08.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf6d82f640 1 -- start start 2024-03-27T21:42:08.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf6d82f640 10 monclient: init 2024-03-27T21:42:08.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf6d82f640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:08.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf6d82f640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-03-27T21:42:08.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf6d82f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcf68071a50 0x7fcf680a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf6d82f640 10 monclient(hunting): picked mon.noname-c con 0x7fcf68071a50 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:08.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf6d82f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf6813e810 0x7fcf6813ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf6d82f640 10 monclient(hunting): picked mon.noname-b con 0x7fcf6813e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:08.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf6d82f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcf68141630 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:08.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf6d82f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf6813f190 con 0x7fcf68141630 2024-03-27T21:42:08.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf6d82f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf6d82f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf68071e30 con 0x7fcf68071a50 2024-03-27T21:42:08.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf6d82f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf6d82f640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf680a8e90 con 0x7fcf6813e810 2024-03-27T21:42:08.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf6d82f640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:08.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf6d82f640 10 monclient(hunting): authenticate will time out at 970.180730s 2024-03-27T21:42:08.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf667fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf6813e810 0x7fcf6813ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf667fc640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf6813e810 0x7fcf6813ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34448/0 (socket says 172.21.15.60:34448) 2024-03-27T21:42:08.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf667fc640 1 -- 172.21.15.60:0/2664172862 learned_addr learned my addr 172.21.15.60:0/2664172862 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:08.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf667fc640 10 monclient(hunting): get_auth_request con 0x7fcf6813e810 auth_method 0 2024-03-27T21:42:08.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:08.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf667fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:08.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.959+0000 7fcf667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:08.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.960+0000 7fcf667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:08.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.960+0000 7fcf667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:08.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.960+0000 7fcf667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:08.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.960+0000 7fcf66ffd640 1 --2- 172.21.15.60:0/2664172862 >> v2:172.21.15.191:3300/0 conn(0x7fcf68071a50 0x7fcf680a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.960+0000 7fcf66ffd640 10 monclient(hunting): get_auth_request con 0x7fcf68071a50 auth_method 0 2024-03-27T21:42:08.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.960+0000 7fcf66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:08.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.960+0000 7fcf66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:08.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.960+0000 7fcf66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:08.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.960+0000 7fcf667fc640 10 monclient(hunting): handle_auth_done global_id 5105 payload 274 2024-03-27T21:42:08.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.960+0000 7fcf667fc640 1 -- 172.21.15.60:0/2664172862 >> v2:172.21.15.191:3300/0 conn(0x7fcf68071a50 msgr2=0x7fcf680a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:08.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.960+0000 7fcf667fc640 1 --2- 172.21.15.60:0/2664172862 >> v2:172.21.15.191:3300/0 conn(0x7fcf68071a50 0x7fcf680a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.960+0000 7fcf667fc640 1 -- 172.21.15.60:0/2664172862 >> v1:172.21.15.191:6789/0 conn(0x7fcf68141630 legacy=0x7fcf68141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.960+0000 7fcf667fc640 10 monclient: _finish_hunting 0 2024-03-27T21:42:08.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.960+0000 7fcf667fc640 1 monclient: found mon.noname-b 2024-03-27T21:42:08.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.960+0000 7fcf667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:08.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.960+0000 7fcf667fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:42:08.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.960+0000 7fcf667fc640 1 -- 172.21.15.60:0/2664172862 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf680aa040 con 0x7fcf6813e810 2024-03-27T21:42:08.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.960+0000 7fcf667fc640 1 --2- 172.21.15.60:0/2664172862 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf6813e810 0x7fcf6813ec50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c009870 tx=0x7fcf5c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d792ce6c702f8b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:08.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf65ffb640 1 -- 172.21.15.60:0/2664172862 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c030840 con 0x7fcf6813e810 2024-03-27T21:42:08.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:08.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf65ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:08.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf65ffb640 10 monclient: dump: 2024-03-27T21:42:08.964 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:08.964 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:08.964 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.964 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.964 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:08.964 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:08.964 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:08.965 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:08.965 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:08.965 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf65ffb640 10 monclient: _finish_auth 0 2024-03-27T21:42:08.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf65ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:42:08.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:08.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf65ffb640 1 -- 172.21.15.60:0/2664172862 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf5c0309a0 con 0x7fcf6813e810 2024-03-27T21:42:08.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf6d82f640 5 monclient: authenticate success, global_id 5105 2024-03-27T21:42:08.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf6d82f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:08.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf65ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:08.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf6d82f640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:08.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf65ffb640 1 -- 172.21.15.60:0/2664172862 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c030d30 con 0x7fcf6813e810 2024-03-27T21:42:08.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:08.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf6d82f640 4 set_mon_vals no callback set 2024-03-27T21:42:08.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf65ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:08.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf65ffb640 10 monclient: dump: 2024-03-27T21:42:08.967 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:08.967 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:08.967 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.967 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.967 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:08.968 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:08.968 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:08.968 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:08.968 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:08.968 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf6d82f640 10 monclient: shutdown 2024-03-27T21:42:08.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf6d82f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:08.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf6d82f640 1 -- 172.21.15.60:0/2664172862 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf6813e810 msgr2=0x7fcf6813ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf6d82f640 1 --2- 172.21.15.60:0/2664172862 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf6813e810 0x7fcf6813ec50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c009870 tx=0x7fcf5c02b980 comp rx=0 tx=0).stop 2024-03-27T21:42:08.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf6d82f640 1 -- 172.21.15.60:0/2664172862 shutdown_connections 2024-03-27T21:42:08.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf6d82f640 1 --2- 172.21.15.60:0/2664172862 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf6813e810 0x7fcf6813ec50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf6d82f640 1 --2- 172.21.15.60:0/2664172862 >> v2:172.21.15.191:3300/0 conn(0x7fcf68071a50 0x7fcf680a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf6d82f640 1 -- 172.21.15.60:0/2664172862 >> 172.21.15.60:0/2664172862 conn(0x7fcf680b7300 msgr2=0x7fcf680b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:08.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf6d82f640 1 -- 172.21.15.60:0/2664172862 shutdown_connections 2024-03-27T21:42:08.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.961+0000 7fcf6d82f640 1 -- 172.21.15.60:0/2664172862 wait complete. 2024-03-27T21:42:08.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.962+0000 7fcf6d82f640 10 monclient: build_initial_monmap 2024-03-27T21:42:08.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.962+0000 7fcf6d82f640 10 monclient: monmap: 2024-03-27T21:42:08.970 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:08.970 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:08.970 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:08.970 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:08.970 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:08.971 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:08.971 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:08.971 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:08.971 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:08.971 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.962+0000 7fcf6d82f640 1 librados: starting msgr at 2024-03-27T21:42:08.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.962+0000 7fcf6d82f640 1 librados: starting objecter 2024-03-27T21:42:08.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.962+0000 7fcf6d82f640 1 Processor -- start 2024-03-27T21:42:08.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.962+0000 7fcf6d82f640 1 -- start start 2024-03-27T21:42:08.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.962+0000 7fcf6d82f640 1 librados: setting wanted keys 2024-03-27T21:42:08.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.962+0000 7fcf6d82f640 1 librados: calling monclient init 2024-03-27T21:42:08.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.962+0000 7fcf6d82f640 10 monclient: init 2024-03-27T21:42:08.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf6d82f640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:08.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf6d82f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:42:08.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf6d82f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf68071a50 0x7fcf68139df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf6d82f640 10 monclient(hunting): picked mon.noname-c con 0x7fcf68071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:08.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf6d82f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf6813e810 0x7fcf681330c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf6d82f640 10 monclient(hunting): picked mon.noname-a con 0x7fcf6813e810 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:08.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf6d82f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf68141630 0x7fcf68133600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf6d82f640 10 monclient(hunting): picked mon.noname-b con 0x7fcf68141630 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:08.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf6d82f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf6d82f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf68138ff0 con 0x7fcf6813e810 2024-03-27T21:42:08.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf6d82f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf6d82f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf680ab3d0 con 0x7fcf68141630 2024-03-27T21:42:08.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf6d82f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:08.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf6d82f640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6814ba70 con 0x7fcf68071a50 2024-03-27T21:42:08.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf6d82f640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:08.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf6d82f640 10 monclient(hunting): authenticate will time out at 970.184206s 2024-03-27T21:42:08.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf66ffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf68071a50 0x7fcf68139df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf667fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf6813e810 0x7fcf681330c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf66ffd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf68071a50 0x7fcf68139df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34462/0 (socket says 172.21.15.60:34462) 2024-03-27T21:42:08.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf66ffd640 1 -- 172.21.15.60:0/421549517 learned_addr learned my addr 172.21.15.60:0/421549517 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:08.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf66ffd640 10 monclient(hunting): get_auth_request con 0x7fcf68071a50 auth_method 0 2024-03-27T21:42:08.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:08.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:08.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:08.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf667fc640 10 monclient(hunting): get_auth_request con 0x7fcf6813e810 auth_method 0 2024-03-27T21:42:08.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:08.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf667fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:08.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:08.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:08.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:08.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf677fe640 1 --2- 172.21.15.60:0/421549517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf68141630 0x7fcf68133600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:08.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:08.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:08.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:08.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf677fe640 10 monclient(hunting): get_auth_request con 0x7fcf68141630 auth_method 0 2024-03-27T21:42:08.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:08.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf677fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf66ffd640 10 monclient(hunting): handle_auth_done global_id 5108 payload 824 2024-03-27T21:42:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf66ffd640 1 -- 172.21.15.60:0/421549517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf68141630 msgr2=0x7fcf68133600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf66ffd640 1 --2- 172.21.15.60:0/421549517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf68141630 0x7fcf68133600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf66ffd640 1 -- 172.21.15.60:0/421549517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf6813e810 msgr2=0x7fcf681330c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf66ffd640 1 --2- 172.21.15.60:0/421549517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf6813e810 0x7fcf681330c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:08.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf66ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:42:08.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf66ffd640 1 monclient: found mon.noname-c 2024-03-27T21:42:08.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:08.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf66ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:08.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf66ffd640 1 -- 172.21.15.60:0/421549517 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf5c0094d0 con 0x7fcf68071a50 2024-03-27T21:42:08.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf667fc640 1 --2- 172.21.15.60:0/421549517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf6813e810 0x7fcf681330c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:08.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf66ffd640 1 --2- 172.21.15.60:0/421549517 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf68071a50 0x7fcf68139df0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcf5800d800 tx=0x7fcf5800dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:08.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf4ffff640 10 client.?.objecter ms_handle_connect 0x7fcf68071a50 2024-03-27T21:42:08.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.963+0000 7fcf4ffff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:08.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf4ffff640 1 -- 172.21.15.60:0/421549517 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5800b6c0 con 0x7fcf68071a50 2024-03-27T21:42:08.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:08.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf4ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:08.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf4ffff640 10 monclient: dump: 2024-03-27T21:42:08.981 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:08.981 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:08.982 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.982 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.982 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:08.982 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:08.982 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:08.982 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:08.982 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:08.982 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf4ffff640 10 monclient: _finish_auth 0 2024-03-27T21:42:08.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf4ffff640 10 monclient: _check_auth_tickets 2024-03-27T21:42:08.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:08.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf6d82f640 5 monclient: authenticate success, global_id 5108 2024-03-27T21:42:08.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf4ffff640 1 -- 172.21.15.60:0/421549517 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf5800b820 con 0x7fcf68071a50 2024-03-27T21:42:08.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf4ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:08.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf4ffff640 1 -- 172.21.15.60:0/421549517 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5800bbb0 con 0x7fcf68071a50 2024-03-27T21:42:08.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf6d82f640 10 monclient: _renew_subs 2024-03-27T21:42:08.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf6d82f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:08.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf6d82f640 1 -- 172.21.15.60:0/421549517 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf6813af00 con 0x7fcf68071a50 2024-03-27T21:42:08.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:08.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf657fa640 4 set_mon_vals no callback set 2024-03-27T21:42:08.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:08.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf4ffff640 10 monclient: dump: 2024-03-27T21:42:08.984 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:08.984 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:08.984 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.984 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:08.985 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:08.985 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:08.985 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:08.985 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:08.985 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:08.985 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:08.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf6d82f640 10 client.5108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:08.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf6d82f640 10 monclient: _renew_subs 2024-03-27T21:42:08.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf6d82f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:08.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf6d82f640 1 -- 172.21.15.60:0/421549517 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf6813a330 con 0x7fcf68071a50 2024-03-27T21:42:08.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.964+0000 7fcf6d82f640 1 librados: init done 2024-03-27T21:42:08.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.965+0000 7fcf6d82f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:08.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.965+0000 7fcf6d82f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:08.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.965+0000 7fcf6d82f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:08.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.965+0000 7fcf6d82f640 1 -- 172.21.15.60:0/421549517 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf6813a580 con 0x7fcf68071a50 2024-03-27T21:42:08.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.965+0000 7fcf4ffff640 1 -- 172.21.15.60:0/421549517 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcf5800bd10 con 0x7fcf68071a50 2024-03-27T21:42:08.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.965+0000 7fcf4ffff640 1 --2- 172.21.15.60:0/421549517 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcf5406c5c0 0x7fcf5406ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:08.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.966+0000 7fcf667fc640 1 --2- 172.21.15.60:0/421549517 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcf5406c5c0 0x7fcf5406ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:08.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.966+0000 7fcf667fc640 10 monclient: get_auth_request con 0x7fcf5406c5c0 auth_method 0 2024-03-27T21:42:08.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.966+0000 7fcf4ffff640 1 -- 172.21.15.60:0/421549517 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fcf58089580 con 0x7fcf68071a50 2024-03-27T21:42:08.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.966+0000 7fcf4ffff640 10 client.5108.objecter ms_dispatch 0x7fcf68000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:08.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.966+0000 7fcf4ffff640 3 client.5108.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:08.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.966+0000 7fcf4ffff640 3 client.5108.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:08.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.966+0000 7fcf4ffff640 20 client.5108.objecter dump_active .. 0 homeless 2024-03-27T21:42:08.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.966+0000 7fcf4ffff640 10 client.5108.objecter ms_handle_connect 0x7fcf5406c5c0 2024-03-27T21:42:08.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.966+0000 7fcf667fc640 1 --2- 172.21.15.60:0/421549517 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcf5406c5c0 0x7fcf5406ea80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c036000 tx=0x7fcf5c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:08.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.970+0000 7fcf4ffff640 1 -- 172.21.15.60:0/421549517 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcf580549e0 con 0x7fcf68071a50 2024-03-27T21:42:08.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.970+0000 7fcf4ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:08.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:08.970+0000 7fcf4ffff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:09.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.144+0000 7fcf6d82f640 1 -- 172.21.15.60:0/421549517 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcf681421b0 con 0x7fcf5406c5c0 2024-03-27T21:42:09.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.146+0000 7fcf4ffff640 1 -- 172.21.15.60:0/421549517 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+66898 (secure 0 0 0) 0x7fcf5c042410 con 0x7fcf5406c5c0 2024-03-27T21:42:09.133 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:42:09.134 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-03-27T21:42:09.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.148+0000 7fcf6d82f640 10 librados: watch_flush enter 2024-03-27T21:42:09.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.148+0000 7fcf6d82f640 10 librados: watch_flush exit 2024-03-27T21:42:09.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.148+0000 7fcf6d82f640 20 client.5108.objecter shutdown clearing up homeless session... 2024-03-27T21:42:09.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.148+0000 7fcf6d82f640 10 client.5108.objecter successfully canceled tick 2024-03-27T21:42:09.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.149+0000 7fcf6d82f640 1 -- 172.21.15.60:0/421549517 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcf5406c5c0 msgr2=0x7fcf5406ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:09.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.149+0000 7fcf6d82f640 1 --2- 172.21.15.60:0/421549517 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcf5406c5c0 0x7fcf5406ea80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c036000 tx=0x7fcf5c008030 comp rx=0 tx=0).stop 2024-03-27T21:42:09.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.149+0000 7fcf6d82f640 10 monclient: shutdown 2024-03-27T21:42:09.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.149+0000 7fcf6d82f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:09.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.149+0000 7fcf6d82f640 1 -- 172.21.15.60:0/421549517 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf68071a50 msgr2=0x7fcf68139df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:09.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.149+0000 7fcf6d82f640 1 --2- 172.21.15.60:0/421549517 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf68071a50 0x7fcf68139df0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcf5800d800 tx=0x7fcf5800dcc0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.149+0000 7fcf6d82f640 1 -- 172.21.15.60:0/421549517 shutdown_connections 2024-03-27T21:42:09.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.149+0000 7fcf6d82f640 1 --2- 172.21.15.60:0/421549517 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcf5406c5c0 0x7fcf5406ea80 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.149+0000 7fcf6d82f640 1 --2- 172.21.15.60:0/421549517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf68141630 0x7fcf68133600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.149+0000 7fcf6d82f640 1 --2- 172.21.15.60:0/421549517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf6813e810 0x7fcf681330c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.149+0000 7fcf6d82f640 1 --2- 172.21.15.60:0/421549517 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcf68071a50 0x7fcf68139df0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.149+0000 7fcf6d82f640 1 -- 172.21.15.60:0/421549517 >> 172.21.15.60:0/421549517 conn(0x7fcf680b7300 msgr2=0x7fcf6813dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:09.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.149+0000 7fcf6d82f640 1 -- 172.21.15.60:0/421549517 shutdown_connections 2024-03-27T21:42:09.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.149+0000 7fcf6d82f640 1 -- 172.21.15.60:0/421549517 wait complete. 2024-03-27T21:42:09.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.149+0000 7fcf6d82f640 1 librados: shutdown 2024-03-27T21:42:09.146 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-03-27T21:42:07.412709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":230404,"kb_used_data":8472,"kb_used_omap":16,"kb_used_meta":221807,"kb_avail":754744316,"statfs":{"total":773094113280,"available":772858179584,"internally_reserved":0,"allocated":8675328,"data_stored":4710296,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":17199,"internal_metadata":227130577},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000861"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-27T21:42:02.414394+0000","last_change":"2024-03-27T21:42:01.265365+0000","last_active":"2024-03-27T21:42:02.414394+0000","last_peered":"2024-03-27T21:42:02.414394+0000","last_clean":"2024-03-27T21:42:02.414394+0000","last_became_active":"2024-03-27T21:42:01.265033+0000","last_became_peered":"2024-03-27T21:42:01.265033+0000","last_unstale":"2024-03-27T21:42:02.414394+0000","last_undegraded":"2024-03-27T21:42:02.414394+0000","last_fullsized":"2024-03-27T21:42:02.414394+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:07:09.198158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.601878+0000","last_change":"2024-03-27T21:41:49.474318+0000","last_active":"2024-03-27T21:42:03.601878+0000","last_peered":"2024-03-27T21:42:03.601878+0000","last_clean":"2024-03-27T21:42:03.601878+0000","last_became_active":"2024-03-27T21:41:49.474054+0000","last_became_peered":"2024-03-27T21:41:49.474054+0000","last_unstale":"2024-03-27T21:42:03.601878+0000","last_undegraded":"2024-03-27T21:42:03.601878+0000","last_fullsized":"2024-03-27T21:42:03.601878+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:36:26.918691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271058+0000","last_change":"2024-03-27T21:39:46.679788+0000","last_active":"2024-03-27T21:42:03.271058+0000","last_peered":"2024-03-27T21:42:03.271058+0000","last_clean":"2024-03-27T21:42:03.271058+0000","last_became_active":"2024-03-27T21:39:46.679561+0000","last_became_peered":"2024-03-27T21:39:46.679561+0000","last_unstale":"2024-03-27T21:42:03.271058+0000","last_undegraded":"2024-03-27T21:42:03.271058+0000","last_fullsized":"2024-03-27T21:42:03.271058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T23:53:50.373923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.601870+0000","last_change":"2024-03-27T21:41:49.474121+0000","last_active":"2024-03-27T21:42:03.601870+0000","last_peered":"2024-03-27T21:42:03.601870+0000","last_clean":"2024-03-27T21:42:03.601870+0000","last_became_active":"2024-03-27T21:41:49.473822+0000","last_became_peered":"2024-03-27T21:41:49.473822+0000","last_unstale":"2024-03-27T21:42:03.601870+0000","last_undegraded":"2024-03-27T21:42:03.601870+0000","last_fullsized":"2024-03-27T21:42:03.601870+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:39:51.253704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271086+0000","last_change":"2024-03-27T21:41:59.701200+0000","last_active":"2024-03-27T21:42:03.271086+0000","last_peered":"2024-03-27T21:42:03.271086+0000","last_clean":"2024-03-27T21:42:03.271086+0000","last_became_active":"2024-03-27T21:41:59.700966+0000","last_became_peered":"2024-03-27T21:41:59.700966+0000","last_unstale":"2024-03-27T21:42:03.271086+0000","last_undegraded":"2024-03-27T21:42:03.271086+0000","last_fullsized":"2024-03-27T21:42:03.271086+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:17:01.025558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.270838+0000","last_change":"2024-03-27T21:41:30.080153+0000","last_active":"2024-03-27T21:42:03.270838+0000","last_peered":"2024-03-27T21:42:03.270838+0000","last_clean":"2024-03-27T21:42:03.270838+0000","last_became_active":"2024-03-27T21:41:30.079829+0000","last_became_peered":"2024-03-27T21:41:30.079829+0000","last_unstale":"2024-03-27T21:42:03.270838+0000","last_undegraded":"2024-03-27T21:42:03.270838+0000","last_fullsized":"2024-03-27T21:42:03.270838+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:21:00.966562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":97,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271058+0000","last_change":"2024-03-27T21:41:49.197510+0000","last_active":"2024-03-27T21:42:03.271058+0000","last_peered":"2024-03-27T21:42:03.271058+0000","last_clean":"2024-03-27T21:42:03.271058+0000","last_became_active":"2024-03-27T21:41:49.197154+0000","last_became_peered":"2024-03-27T21:41:49.197154+0000","last_unstale":"2024-03-27T21:42:03.271058+0000","last_undegraded":"2024-03-27T21:42:03.271058+0000","last_fullsized":"2024-03-27T21:42:03.271058+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:18:21.622988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271153+0000","last_change":"2024-03-27T21:41:59.247145+0000","last_active":"2024-03-27T21:42:03.271153+0000","last_peered":"2024-03-27T21:42:03.271153+0000","last_clean":"2024-03-27T21:42:03.271153+0000","last_became_active":"2024-03-27T21:41:59.246895+0000","last_became_peered":"2024-03-27T21:41:59.246895+0000","last_unstale":"2024-03-27T21:42:03.271153+0000","last_undegraded":"2024-03-27T21:42:03.271153+0000","last_fullsized":"2024-03-27T21:42:03.271153+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:40:39.749875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":234,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271065+0000","last_change":"2024-03-27T21:39:39.866806+0000","last_active":"2024-03-27T21:42:03.271065+0000","last_peered":"2024-03-27T21:42:03.271065+0000","last_clean":"2024-03-27T21:42:03.271065+0000","last_became_active":"2024-03-27T21:39:39.865948+0000","last_became_peered":"2024-03-27T21:39:39.865948+0000","last_unstale":"2024-03-27T21:42:03.271065+0000","last_undegraded":"2024-03-27T21:42:03.271065+0000","last_fullsized":"2024-03-27T21:42:03.271065+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_clean_scrub_stamp":"2024-03-27T21:39:38.411432+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:36:49.375869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":16,"seq":68719476771,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.41599999999999998,"15min":0.41199999999999998},"min":{"1min":0.34399999999999997,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.621,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.498},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.47499999999999998,"15min":0.47599999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.66900000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.70499999999999996}]},{"osd":1,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45900000000000002,"15min":0.46500000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.53600000000000003,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.35899999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.56599999999999995,"5min":0.745,"15min":0.745},"last":0.371}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":3,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.50600000000000001,"15min":0.51400000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.56899999999999995,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.35999999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68400000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.56000000000000005}]},{"osd":4,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.432,"15min":0.439},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.56399999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.433,"5min":0.46800000000000003,"15min":0.47399999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.78400000000000003}]},{"osd":6,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.45800000000000002,"15min":0.46300000000000002},"min":{"1min":0.318,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58699999999999997,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.29299999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64300000000000002,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.73999999999999999}]},{"osd":7,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.52400000000000002,"15min":0.53200000000000003},"min":{"1min":0.378,"5min":0.318,"15min":0.318},"max":{"1min":0.61299999999999999,"5min":0.745,"15min":0.745},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.50800000000000001,"15min":0.51400000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65000000000000002,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.72399999999999998}]}]},{"osd":3,"up_from":16,"seq":68719476770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31848905},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.49299999999999999,"15min":0.504},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.91700000000000004,"5min":1.248,"15min":1.248},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.44900000000000001,"15min":0.45600000000000002},"min":{"1min":0.22800000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.90600000000000003,"5min":1.407,"15min":1.407},"last":0.36599999999999999}]},{"osd":1,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48199999999999998,"15min":0.47999999999999998},"min":{"1min":0.25600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.26800000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.96899999999999997,"5min":1.347,"15min":1.347},"last":0.42299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":4,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.61399999999999999,"15min":0.626},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83299999999999996,"5min":1.5,"15min":1.5},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.40000000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72799999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.54000000000000004}]},{"osd":5,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56000000000000005,"15min":0.56599999999999995},"min":{"1min":0.34999999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71099999999999997,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.63300000000000001,"15min":0.64700000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.96799999999999997,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.46000000000000002}]},{"osd":6,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.63100000000000001,"15min":0.63900000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88200000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.40999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76700000000000002,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.503}]},{"osd":7,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.64200000000000002,"15min":0.65600000000000003},"min":{"1min":0.36899999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.94399999999999995,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61599999999999999,"15min":0.622},"min":{"1min":0.36699999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85799999999999998,"5min":1.454,"15min":1.454},"last":0.55700000000000005}]}]},{"osd":2,"up_from":62,"seq":266287972361,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29736,"kb_used_data":1052,"kb_used_omap":5,"kb_used_meta":28666,"kb_avail":94342104,"statfs":{"total":96636764160,"available":96606314496,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6051,"internal_metadata":29354077},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.219}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]}]},{"osd":7,"up_from":14,"seq":60129542177,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1118208,"data_stored":982758,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.45200000000000001,"15min":0.45800000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.56299999999999994,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.42299999999999999,"15min":0.42399999999999999},"min":{"1min":0.29799999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60499999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.58299999999999996}]},{"osd":1,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45500000000000002,"15min":0.46100000000000002},"min":{"1min":0.29899999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.55200000000000005,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.44,"5min":0.44800000000000001,"15min":0.45000000000000001},"min":{"1min":0.28499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":3,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83199999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.49299999999999999,"15min":0.497},"min":{"1min":0.37,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.64600000000000002,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.66900000000000004}]},{"osd":4,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.42799999999999999,"15min":0.439},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.53600000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.47599999999999998,"15min":0.48799999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.60099999999999998,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.68799999999999994}]},{"osd":5,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.54400000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.433,"5min":0.51300000000000001,"15min":0.52700000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.65900000000000003,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.70799999999999996}]},{"osd":6,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.46500000000000002,"15min":0.47499999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.59299999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.52400000000000002,"15min":0.54000000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.71699999999999997,"5min":1.054,"15min":1.054},"last":0.65600000000000003}]}]},{"osd":6,"up_from":14,"seq":60129542177,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343776,"statfs":{"total":96636764160,"available":96608026624,"internally_reserved":0,"allocated":1064960,"data_stored":453830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.42699999999999999,"15min":0.42199999999999999},"min":{"1min":0.313,"5min":0.25,"15min":0.25},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.47399999999999998,"15min":0.47899999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.52400000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.69099999999999995}]},{"osd":1,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.57399999999999995,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.495,"15min":0.497},"min":{"1min":0.373,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.624,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.61499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":3,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68799999999999994,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66500000000000004,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.70699999999999996}]},{"osd":4,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.46200000000000002,"15min":0.46999999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57199999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.47899999999999998,"15min":0.48699999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.64900000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.46600000000000003}]},{"osd":5,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.28499999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.68799999999999994,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.5},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.313,"5min":0.308,"15min":0.308},"max":{"1min":0.65600000000000003,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.60399999999999998}]},{"osd":7,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53700000000000003,"15min":0.54100000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.51300000000000001,"15min":0.51700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64900000000000002,"5min":0.872,"15min":0.872},"last":0.57599999999999996}]}]},{"osd":4,"up_from":14,"seq":60129542177,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28068,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343772,"statfs":{"total":96636764160,"available":96608022528,"internally_reserved":0,"allocated":1064960,"data_stored":453830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.40500000000000003,"15min":0.40200000000000002},"min":{"1min":0.308,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.443,"5min":0.435,"15min":0.434},"min":{"1min":0.33600000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.64600000000000002,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66300000000000003,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46999999999999997,"15min":0.47299999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.91500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":3,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.51100000000000001,"15min":0.51900000000000002},"min":{"1min":0.38100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.58399999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.499,"15min":0.501},"min":{"1min":0.31900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.66900000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.59999999999999998}]},{"osd":5,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.44400000000000001,"15min":0.45000000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.64700000000000002,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.433,"5min":0.45100000000000001,"15min":0.45400000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64100000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.65700000000000003}]},{"osd":6,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66500000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.62},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.61199999999999999,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.47199999999999998}]},{"osd":7,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.49399999999999999,"15min":0.49199999999999999},"min":{"1min":0.34100000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71699999999999997,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.58599999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.50700000000000001,"15min":0.52900000000000003},"min":{"1min":0.219,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.127,"5min":3.2959999999999998,"15min":3.2959999999999998},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.56699999999999995,"15min":0.58499999999999996},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":1.161,"5min":3.2799999999999998,"15min":3.2799999999999998},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":3,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":3.3239999999999998,"15min":3.3239999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.47799999999999998,"15min":0.47699999999999998},"min":{"1min":0.25900000000000001,"5min":0.253,"15min":0.253},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.39100000000000001}]},{"osd":4,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.375,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59299999999999997,"15min":0.58899999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":0.63700000000000001}]},{"osd":5,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.39300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.628,"15min":0.624},"min":{"1min":0.376,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.66400000000000003}]},{"osd":6,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.61899999999999999,"15min":0.625},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0920000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.501},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.70399999999999996,"15min":0.71999999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1779999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":0.622}]},{"osd":7,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.667,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63800000000000001,"15min":0.64200000000000002},"min":{"1min":0.41599999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.57199999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542177,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1118208,"data_stored":982758,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.40500000000000003,"15min":0.40899999999999997},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.79800000000000004,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40100000000000002,"15min":0.40899999999999997},"min":{"1min":0.23200000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.64200000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.27700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25900000000000001}]},{"osd":3,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.46999999999999997,"15min":0.48199999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.59199999999999997,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.311},{"interface":"front","average":{"1min":0.441,"5min":0.48599999999999999,"15min":0.49399999999999999},"min":{"1min":0.28799999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.29899999999999999}]},{"osd":4,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.871,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.56599999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82699999999999996,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.46000000000000002}]},{"osd":5,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54200000000000004,"15min":0.53900000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.52600000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.56699999999999995}]},{"osd":6,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.38100000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59199999999999997,"15min":0.59599999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.50900000000000001}]},{"osd":7,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57299999999999995,"15min":0.56799999999999995},"min":{"1min":0.378,"5min":0.377,"15min":0.377},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.56299999999999994,"15min":0.56899999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.70899999999999996,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T21:42:09.148 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T21:42:09.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.261+0000 7f66f62e5640 10 monclient: get_monmap_and_config 2024-03-27T21:42:09.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.261+0000 7f66f62e5640 10 monclient: build_initial_monmap 2024-03-27T21:42:09.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.261+0000 7f66f62e5640 10 monclient: monmap: 2024-03-27T21:42:09.248 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:09.248 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:09.248 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:09.262725+0000 2024-03-27T21:42:09.248 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:09.262725+0000 2024-03-27T21:42:09.249 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:09.249 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:09.249 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:09.249 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:09.249 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:09.249 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:09.249 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:09.249 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:09.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.262+0000 7f66f62e5640 1 Processor -- start 2024-03-27T21:42:09.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.262+0000 7f66f62e5640 1 -- start start 2024-03-27T21:42:09.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.262+0000 7f66f62e5640 10 monclient: init 2024-03-27T21:42:09.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.262+0000 7f66f62e5640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:09.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.262+0000 7f66f62e5640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-03-27T21:42:09.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66f62e5640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f66f0071a50 0x7f66f00a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:09.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66f62e5640 10 monclient(hunting): picked mon.noname-b con 0x7f66f0071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:09.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66f62e5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f66f013e810 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:09.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66f62e5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f66f013ec50 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:09.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66f62e5640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f00a8e90 con 0x7f66f013e810 2024-03-27T21:42:09.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66f62e5640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f00aa040 con 0x7f66f013ec50 2024-03-27T21:42:09.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66f62e5640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:09.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66f62e5640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f0071e30 con 0x7f66f0071a50 2024-03-27T21:42:09.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66f62e5640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:09.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66f62e5640 10 monclient(hunting): authenticate will time out at 970.483974s 2024-03-27T21:42:09.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66ef7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f66f013e810 0x7f66f0141630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39492/0 (socket says 172.21.15.60:39492) 2024-03-27T21:42:09.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66ef7fe640 1 -- 172.21.15.60:0/419906726 learned_addr learned my addr 172.21.15.60:0/419906726 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:09.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66effff640 1 --2- 172.21.15.60:0/419906726 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f66f0071a50 0x7f66f00a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:09.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66effff640 10 monclient(hunting): get_auth_request con 0x7f66f0071a50 auth_method 0 2024-03-27T21:42:09.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:09.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66effff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:09.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:09.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:09.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:09.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:09.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66effff640 10 monclient(hunting): handle_auth_done global_id 5111 payload 274 2024-03-27T21:42:09.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66eeffd640 1 -- 172.21.15.60:0/419906726 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846329077 0 0) 0x7f66f00a8e90 con 0x7f66f013e810 2024-03-27T21:42:09.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66effff640 1 -- 172.21.15.60:0/419906726 >> v1:172.21.15.191:6789/0 conn(0x7f66f013ec50 legacy=0x7f66f0141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:09.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66effff640 1 -- 172.21.15.60:0/419906726 >> v1:172.21.15.60:6789/0 conn(0x7f66f013e810 legacy=0x7f66f0141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:09.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66effff640 10 monclient: _finish_hunting 0 2024-03-27T21:42:09.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66effff640 1 monclient: found mon.noname-b 2024-03-27T21:42:09.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:09.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66effff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:42:09.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66effff640 1 -- 172.21.15.60:0/419906726 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f00aa2b0 con 0x7f66f0071a50 2024-03-27T21:42:09.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.263+0000 7f66eeffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:42:09.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66effff640 1 --2- 172.21.15.60:0/419906726 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f66f0071a50 0x7f66f00a87e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f66d80097d0 tx=0x7f66d802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaeec3f0174c367e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:09.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66eeffd640 1 -- 172.21.15.60:0/419906726 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8031800 con 0x7f66f0071a50 2024-03-27T21:42:09.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:09.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66eeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:09.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66eeffd640 10 monclient: dump: 2024-03-27T21:42:09.263 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:09.263 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:09.263 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.263 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.264 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:09.264 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:09.264 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:09.264 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:09.264 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:09.264 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:09.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66eeffd640 10 monclient: _finish_auth 0 2024-03-27T21:42:09.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66eeffd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:09.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:09.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66f62e5640 5 monclient: authenticate success, global_id 5111 2024-03-27T21:42:09.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66f62e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:09.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66eeffd640 1 -- 172.21.15.60:0/419906726 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d8031960 con 0x7f66f0071a50 2024-03-27T21:42:09.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:09.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66f62e5640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:09.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66f62e5640 4 set_mon_vals no callback set 2024-03-27T21:42:09.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66eeffd640 1 -- 172.21.15.60:0/419906726 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8031cf0 con 0x7f66f0071a50 2024-03-27T21:42:09.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:09.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:09.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66eeffd640 10 monclient: dump: 2024-03-27T21:42:09.266 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:09.266 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:09.266 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.266 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.267 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:09.267 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:09.267 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:09.267 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:09.267 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:09.267 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:09.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66f62e5640 10 monclient: shutdown 2024-03-27T21:42:09.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66f62e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:09.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66f62e5640 1 -- 172.21.15.60:0/419906726 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f66f0071a50 msgr2=0x7f66f00a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:09.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66f62e5640 1 --2- 172.21.15.60:0/419906726 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f66f0071a50 0x7f66f00a87e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f66d80097d0 tx=0x7f66d802b8a0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66f62e5640 1 -- 172.21.15.60:0/419906726 shutdown_connections 2024-03-27T21:42:09.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66f62e5640 1 --2- 172.21.15.60:0/419906726 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f66f0071a50 0x7f66f00a87e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66f62e5640 1 -- 172.21.15.60:0/419906726 >> 172.21.15.60:0/419906726 conn(0x7f66f00b7300 msgr2=0x7f66f00b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:09.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.264+0000 7f66f62e5640 1 -- 172.21.15.60:0/419906726 shutdown_connections 2024-03-27T21:42:09.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.265+0000 7f66f62e5640 1 -- 172.21.15.60:0/419906726 wait complete. 2024-03-27T21:42:09.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.265+0000 7f66f62e5640 10 monclient: build_initial_monmap 2024-03-27T21:42:09.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.265+0000 7f66f62e5640 10 monclient: monmap: 2024-03-27T21:42:09.269 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:09.269 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:09.269 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:09.269 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:09.269 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:09.269 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:09.270 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:09.270 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:09.270 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:09.270 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:09.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.265+0000 7f66f62e5640 1 librados: starting msgr at 2024-03-27T21:42:09.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.265+0000 7f66f62e5640 1 librados: starting objecter 2024-03-27T21:42:09.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.265+0000 7f66f62e5640 1 Processor -- start 2024-03-27T21:42:09.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66f62e5640 1 -- start start 2024-03-27T21:42:09.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66f62e5640 1 librados: setting wanted keys 2024-03-27T21:42:09.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66f62e5640 1 librados: calling monclient init 2024-03-27T21:42:09.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66f62e5640 10 monclient: init 2024-03-27T21:42:09.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66f62e5640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:09.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66f62e5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:42:09.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66f62e5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66f0071a50 0x7f66f00ad290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:09.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66f62e5640 10 monclient(hunting): picked mon.noname-a con 0x7f66f0071a50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:09.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66f62e5640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f66f013e810 0x7f66f00ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:09.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66f62e5640 10 monclient(hunting): picked mon.noname-c con 0x7f66f013e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:09.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66f62e5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66f013ec50 0x7f66f0133e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:09.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66f62e5640 10 monclient(hunting): picked mon.noname-b con 0x7f66f013ec50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:09.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66f62e5640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:09.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66f62e5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f00ace70 con 0x7f66f0071a50 2024-03-27T21:42:09.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66f62e5640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:09.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66f62e5640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f00aac70 con 0x7f66f013ec50 2024-03-27T21:42:09.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66f62e5640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:09.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66f62e5640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f013ae30 con 0x7f66f013e810 2024-03-27T21:42:09.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66f62e5640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:09.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66f62e5640 10 monclient(hunting): authenticate will time out at 970.487623s 2024-03-27T21:42:09.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66effff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66f0071a50 0x7f66f00ad290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:09.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66effff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66f0071a50 0x7f66f00ad290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39668/0 (socket says 172.21.15.60:39668) 2024-03-27T21:42:09.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66ef7fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f66f013e810 0x7f66f00ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:09.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66effff640 1 -- 172.21.15.60:0/3606554902 learned_addr learned my addr 172.21.15.60:0/3606554902 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:09.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66effff640 10 monclient(hunting): get_auth_request con 0x7f66f0071a50 auth_method 0 2024-03-27T21:42:09.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:09.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66effff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:09.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:09.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66ef7fe640 10 monclient(hunting): get_auth_request con 0x7f66f013e810 auth_method 0 2024-03-27T21:42:09.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:09.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:09.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:09.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:09.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:09.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:09.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:09.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:09.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.266+0000 7f66ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:09.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66effff640 10 monclient(hunting): handle_auth_done global_id 5427 payload 824 2024-03-27T21:42:09.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66effff640 1 -- 172.21.15.60:0/3606554902 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f66f013e810 msgr2=0x7f66f00ad7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:09.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66effff640 1 --2- 172.21.15.60:0/3606554902 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f66f013e810 0x7f66f00ad7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66f4882640 1 --2- 172.21.15.60:0/3606554902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66f013ec50 0x7f66f0133e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:09.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66effff640 1 -- 172.21.15.60:0/3606554902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66f013ec50 msgr2=0x7f66f0133e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:09.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66effff640 1 --2- 172.21.15.60:0/3606554902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66f013ec50 0x7f66f0133e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66effff640 10 monclient: _finish_hunting 0 2024-03-27T21:42:09.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66effff640 1 monclient: found mon.noname-a 2024-03-27T21:42:09.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:09.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66effff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:09.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66effff640 1 -- 172.21.15.60:0/3606554902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d8009400 con 0x7f66f0071a50 2024-03-27T21:42:09.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66ef7fe640 1 --2- 172.21.15.60:0/3606554902 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f66f013e810 0x7f66f00ad7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:09.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66effff640 1 --2- 172.21.15.60:0/3606554902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66f0071a50 0x7f66f00ad290 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f66d80097d0 tx=0x7f66d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:09.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66ecff9640 10 client.?.objecter ms_handle_connect 0x7f66f0071a50 2024-03-27T21:42:09.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66ecff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:09.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66ecff9640 1 -- 172.21.15.60:0/3606554902 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d803bd80 con 0x7f66f0071a50 2024-03-27T21:42:09.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:09.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66ecff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:09.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66ecff9640 10 monclient: dump: 2024-03-27T21:42:09.278 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:09.278 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:09.278 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.278 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.279 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:09.279 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:09.279 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:09.279 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:09.279 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:09.279 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:09.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66ecff9640 10 monclient: _finish_auth 0 2024-03-27T21:42:09.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66ecff9640 10 monclient: _check_auth_tickets 2024-03-27T21:42:09.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:09.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66ecff9640 1 -- 172.21.15.60:0/3606554902 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d80453f0 con 0x7f66f0071a50 2024-03-27T21:42:09.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66f62e5640 5 monclient: authenticate success, global_id 5427 2024-03-27T21:42:09.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66f62e5640 10 monclient: _renew_subs 2024-03-27T21:42:09.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66f62e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:09.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66f62e5640 1 -- 172.21.15.60:0/3606554902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66f013a260 con 0x7f66f0071a50 2024-03-27T21:42:09.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66ecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:09.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66ecff9640 1 -- 172.21.15.60:0/3606554902 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d80457a0 con 0x7f66f0071a50 2024-03-27T21:42:09.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:09.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66ecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:09.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66ecff9640 10 monclient: dump: 2024-03-27T21:42:09.281 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:09.281 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:09.281 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.281 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.282 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:09.282 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:09.282 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:09.282 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:09.282 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:09.282 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:09.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66ee7fc640 4 set_mon_vals no callback set 2024-03-27T21:42:09.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66f62e5640 10 client.5427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:09.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66f62e5640 10 monclient: _renew_subs 2024-03-27T21:42:09.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66f62e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:09.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66f62e5640 1 -- 172.21.15.60:0/3606554902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f00ab390 con 0x7f66f0071a50 2024-03-27T21:42:09.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.267+0000 7f66f62e5640 1 librados: init done 2024-03-27T21:42:09.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.268+0000 7f66f62e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:09.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.268+0000 7f66f62e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:09.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.268+0000 7f66f62e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:09.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.268+0000 7f66f62e5640 1 -- 172.21.15.60:0/3606554902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66f00a9260 con 0x7f66f0071a50 2024-03-27T21:42:09.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.268+0000 7f66ecff9640 1 -- 172.21.15.60:0/3606554902 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f66d804e450 con 0x7f66f0071a50 2024-03-27T21:42:09.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.269+0000 7f66ecff9640 1 --2- 172.21.15.60:0/3606554902 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f66d006c590 0x7f66d006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:09.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.269+0000 7f66ecff9640 1 -- 172.21.15.60:0/3606554902 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f66d802e070 con 0x7f66f0071a50 2024-03-27T21:42:09.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.269+0000 7f66ef7fe640 1 --2- 172.21.15.60:0/3606554902 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f66d006c590 0x7f66d006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:09.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.269+0000 7f66ecff9640 10 client.5427.objecter ms_dispatch 0x7f66f0000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:09.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.269+0000 7f66ecff9640 3 client.5427.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:09.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.269+0000 7f66ecff9640 3 client.5427.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:09.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.269+0000 7f66ef7fe640 10 monclient: get_auth_request con 0x7f66d006c590 auth_method 0 2024-03-27T21:42:09.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.269+0000 7f66ecff9640 20 client.5427.objecter dump_active .. 0 homeless 2024-03-27T21:42:09.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.269+0000 7f66ef7fe640 1 --2- 172.21.15.60:0/3606554902 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f66d006c590 0x7f66d006ea50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f66f0139e60 tx=0x7f66e0007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:09.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.269+0000 7f66ecff9640 10 client.5427.objecter ms_handle_connect 0x7f66d006c590 2024-03-27T21:42:09.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.273+0000 7f66ecff9640 1 -- 172.21.15.60:0/3606554902 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f66d8076bf0 con 0x7f66f0071a50 2024-03-27T21:42:09.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.273+0000 7f66ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:09.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.273+0000 7f66ecff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:09.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.450+0000 7f66f62e5640 1 -- 172.21.15.60:0/3606554902 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f66f013f0c0 con 0x7f66d006c590 2024-03-27T21:42:09.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.452+0000 7f66ecff9640 1 -- 172.21.15.60:0/3606554902 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+66914 (secure 0 0 0) 0x7f66e0001cd0 con 0x7f66d006c590 2024-03-27T21:42:09.439 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:42:09.439 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-03-27T21:42:09.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.454+0000 7f66f62e5640 10 librados: watch_flush enter 2024-03-27T21:42:09.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.454+0000 7f66f62e5640 10 librados: watch_flush exit 2024-03-27T21:42:09.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.454+0000 7f66f62e5640 20 client.5427.objecter shutdown clearing up homeless session... 2024-03-27T21:42:09.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.454+0000 7f66f62e5640 10 client.5427.objecter successfully canceled tick 2024-03-27T21:42:09.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.454+0000 7f66f62e5640 1 -- 172.21.15.60:0/3606554902 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f66d006c590 msgr2=0x7f66d006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:09.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.454+0000 7f66f62e5640 1 --2- 172.21.15.60:0/3606554902 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f66d006c590 0x7f66d006ea50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f66f0139e60 tx=0x7f66e0007d50 comp rx=0 tx=0).stop 2024-03-27T21:42:09.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.454+0000 7f66f62e5640 10 monclient: shutdown 2024-03-27T21:42:09.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.454+0000 7f66f62e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:09.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.454+0000 7f66f62e5640 1 -- 172.21.15.60:0/3606554902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66f0071a50 msgr2=0x7f66f00ad290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:09.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.454+0000 7f66f62e5640 1 --2- 172.21.15.60:0/3606554902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66f0071a50 0x7f66f00ad290 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f66d80097d0 tx=0x7f66d80046c0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.454+0000 7f66f62e5640 1 -- 172.21.15.60:0/3606554902 shutdown_connections 2024-03-27T21:42:09.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.454+0000 7f66f62e5640 1 --2- 172.21.15.60:0/3606554902 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f66d006c590 0x7f66d006ea50 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.454+0000 7f66f62e5640 1 --2- 172.21.15.60:0/3606554902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66f013ec50 0x7f66f0133e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.454+0000 7f66f62e5640 1 --2- 172.21.15.60:0/3606554902 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f66f013e810 0x7f66f00ad7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.454+0000 7f66f62e5640 1 --2- 172.21.15.60:0/3606554902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66f0071a50 0x7f66f00ad290 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.454+0000 7f66f62e5640 1 -- 172.21.15.60:0/3606554902 >> 172.21.15.60:0/3606554902 conn(0x7f66f00b7300 msgr2=0x7f66f0135580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:09.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.454+0000 7f66f62e5640 1 -- 172.21.15.60:0/3606554902 shutdown_connections 2024-03-27T21:42:09.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.454+0000 7f66f62e5640 1 -- 172.21.15.60:0/3606554902 wait complete. 2024-03-27T21:42:09.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.454+0000 7f66f62e5640 1 librados: shutdown 2024-03-27T21:42:09.451 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-03-27T21:42:09.413031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":230448,"kb_used_data":8520,"kb_used_omap":16,"kb_used_meta":221807,"kb_avail":754744272,"statfs":{"total":773094113280,"available":772858134528,"internally_reserved":0,"allocated":8724480,"data_stored":4732096,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":17199,"internal_metadata":227130577},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.148977"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.602437+0000","last_change":"2024-03-27T21:42:01.265365+0000","last_active":"2024-03-27T21:42:03.602437+0000","last_peered":"2024-03-27T21:42:03.602437+0000","last_clean":"2024-03-27T21:42:03.602437+0000","last_became_active":"2024-03-27T21:42:01.265033+0000","last_became_peered":"2024-03-27T21:42:01.265033+0000","last_unstale":"2024-03-27T21:42:03.602437+0000","last_undegraded":"2024-03-27T21:42:03.602437+0000","last_fullsized":"2024-03-27T21:42:03.602437+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:07:09.198158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.601878+0000","last_change":"2024-03-27T21:41:49.474318+0000","last_active":"2024-03-27T21:42:03.601878+0000","last_peered":"2024-03-27T21:42:03.601878+0000","last_clean":"2024-03-27T21:42:03.601878+0000","last_became_active":"2024-03-27T21:41:49.474054+0000","last_became_peered":"2024-03-27T21:41:49.474054+0000","last_unstale":"2024-03-27T21:42:03.601878+0000","last_undegraded":"2024-03-27T21:42:03.601878+0000","last_fullsized":"2024-03-27T21:42:03.601878+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:36:26.918691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271058+0000","last_change":"2024-03-27T21:39:46.679788+0000","last_active":"2024-03-27T21:42:03.271058+0000","last_peered":"2024-03-27T21:42:03.271058+0000","last_clean":"2024-03-27T21:42:03.271058+0000","last_became_active":"2024-03-27T21:39:46.679561+0000","last_became_peered":"2024-03-27T21:39:46.679561+0000","last_unstale":"2024-03-27T21:42:03.271058+0000","last_undegraded":"2024-03-27T21:42:03.271058+0000","last_fullsized":"2024-03-27T21:42:03.271058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T23:53:50.373923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.601870+0000","last_change":"2024-03-27T21:41:49.474121+0000","last_active":"2024-03-27T21:42:03.601870+0000","last_peered":"2024-03-27T21:42:03.601870+0000","last_clean":"2024-03-27T21:42:03.601870+0000","last_became_active":"2024-03-27T21:41:49.473822+0000","last_became_peered":"2024-03-27T21:41:49.473822+0000","last_unstale":"2024-03-27T21:42:03.601870+0000","last_undegraded":"2024-03-27T21:42:03.601870+0000","last_fullsized":"2024-03-27T21:42:03.601870+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:39:51.253704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271086+0000","last_change":"2024-03-27T21:41:59.701200+0000","last_active":"2024-03-27T21:42:03.271086+0000","last_peered":"2024-03-27T21:42:03.271086+0000","last_clean":"2024-03-27T21:42:03.271086+0000","last_became_active":"2024-03-27T21:41:59.700966+0000","last_became_peered":"2024-03-27T21:41:59.700966+0000","last_unstale":"2024-03-27T21:42:03.271086+0000","last_undegraded":"2024-03-27T21:42:03.271086+0000","last_fullsized":"2024-03-27T21:42:03.271086+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:17:01.025558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.270838+0000","last_change":"2024-03-27T21:41:30.080153+0000","last_active":"2024-03-27T21:42:03.270838+0000","last_peered":"2024-03-27T21:42:03.270838+0000","last_clean":"2024-03-27T21:42:03.270838+0000","last_became_active":"2024-03-27T21:41:30.079829+0000","last_became_peered":"2024-03-27T21:41:30.079829+0000","last_unstale":"2024-03-27T21:42:03.270838+0000","last_undegraded":"2024-03-27T21:42:03.270838+0000","last_fullsized":"2024-03-27T21:42:03.270838+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:21:00.966562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":97,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271058+0000","last_change":"2024-03-27T21:41:49.197510+0000","last_active":"2024-03-27T21:42:03.271058+0000","last_peered":"2024-03-27T21:42:03.271058+0000","last_clean":"2024-03-27T21:42:03.271058+0000","last_became_active":"2024-03-27T21:41:49.197154+0000","last_became_peered":"2024-03-27T21:41:49.197154+0000","last_unstale":"2024-03-27T21:42:03.271058+0000","last_undegraded":"2024-03-27T21:42:03.271058+0000","last_fullsized":"2024-03-27T21:42:03.271058+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:18:21.622988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271153+0000","last_change":"2024-03-27T21:41:59.247145+0000","last_active":"2024-03-27T21:42:03.271153+0000","last_peered":"2024-03-27T21:42:03.271153+0000","last_clean":"2024-03-27T21:42:03.271153+0000","last_became_active":"2024-03-27T21:41:59.246895+0000","last_became_peered":"2024-03-27T21:41:59.246895+0000","last_unstale":"2024-03-27T21:42:03.271153+0000","last_undegraded":"2024-03-27T21:42:03.271153+0000","last_fullsized":"2024-03-27T21:42:03.271153+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:40:39.749875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":234,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271065+0000","last_change":"2024-03-27T21:39:39.866806+0000","last_active":"2024-03-27T21:42:03.271065+0000","last_peered":"2024-03-27T21:42:03.271065+0000","last_clean":"2024-03-27T21:42:03.271065+0000","last_became_active":"2024-03-27T21:39:39.865948+0000","last_became_peered":"2024-03-27T21:39:39.865948+0000","last_unstale":"2024-03-27T21:42:03.271065+0000","last_undegraded":"2024-03-27T21:42:03.271065+0000","last_fullsized":"2024-03-27T21:42:03.271065+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_clean_scrub_stamp":"2024-03-27T21:39:38.411432+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:36:49.375869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":16,"seq":68719476771,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.41599999999999998,"15min":0.41199999999999998},"min":{"1min":0.34399999999999997,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.621,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.498},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.47499999999999998,"15min":0.47599999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.66900000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.70499999999999996}]},{"osd":1,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45900000000000002,"15min":0.46500000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.53600000000000003,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.35899999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.56599999999999995,"5min":0.745,"15min":0.745},"last":0.371}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":3,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.50600000000000001,"15min":0.51400000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.56899999999999995,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.35999999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68400000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.56000000000000005}]},{"osd":4,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.432,"15min":0.439},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.56399999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.433,"5min":0.46800000000000003,"15min":0.47399999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.78400000000000003}]},{"osd":6,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.45800000000000002,"15min":0.46300000000000002},"min":{"1min":0.318,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58699999999999997,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.29299999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64300000000000002,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.73999999999999999}]},{"osd":7,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.52400000000000002,"15min":0.53200000000000003},"min":{"1min":0.378,"5min":0.318,"15min":0.318},"max":{"1min":0.61299999999999999,"5min":0.745,"15min":0.745},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.50800000000000001,"15min":0.51400000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65000000000000002,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.72399999999999998}]}]},{"osd":3,"up_from":16,"seq":68719476770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31848905},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.49299999999999999,"15min":0.504},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.91700000000000004,"5min":1.248,"15min":1.248},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.44900000000000001,"15min":0.45600000000000002},"min":{"1min":0.22800000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.90600000000000003,"5min":1.407,"15min":1.407},"last":0.36599999999999999}]},{"osd":1,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48199999999999998,"15min":0.47999999999999998},"min":{"1min":0.25600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.26800000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.96899999999999997,"5min":1.347,"15min":1.347},"last":0.42299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":4,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.61399999999999999,"15min":0.626},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83299999999999996,"5min":1.5,"15min":1.5},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.40000000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72799999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.54000000000000004}]},{"osd":5,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56000000000000005,"15min":0.56599999999999995},"min":{"1min":0.34999999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71099999999999997,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.63300000000000001,"15min":0.64700000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.96799999999999997,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.46000000000000002}]},{"osd":6,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.63100000000000001,"15min":0.63900000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88200000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.40999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76700000000000002,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.503}]},{"osd":7,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.64200000000000002,"15min":0.65600000000000003},"min":{"1min":0.36899999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.94399999999999995,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61599999999999999,"15min":0.622},"min":{"1min":0.36699999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85799999999999998,"5min":1.454,"15min":1.454},"last":0.55700000000000005}]}]},{"osd":2,"up_from":62,"seq":266287972362,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29736,"kb_used_data":1052,"kb_used_omap":5,"kb_used_meta":28666,"kb_avail":94342104,"statfs":{"total":96636764160,"available":96606314496,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6051,"internal_metadata":29354077},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]}]},{"osd":7,"up_from":14,"seq":60129542178,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28128,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343712,"statfs":{"total":96636764160,"available":96607961088,"internally_reserved":0,"allocated":1130496,"data_stored":988208,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.45200000000000001,"15min":0.45800000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.56299999999999994,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.42299999999999999,"15min":0.42399999999999999},"min":{"1min":0.29799999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60499999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.58299999999999996}]},{"osd":1,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45500000000000002,"15min":0.46100000000000002},"min":{"1min":0.29899999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.55200000000000005,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.44,"5min":0.44800000000000001,"15min":0.45000000000000001},"min":{"1min":0.28499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":3,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83199999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.49299999999999999,"15min":0.497},"min":{"1min":0.37,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.64600000000000002,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.66900000000000004}]},{"osd":4,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.42799999999999999,"15min":0.439},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.53600000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.47599999999999998,"15min":0.48799999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.60099999999999998,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.68799999999999994}]},{"osd":5,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.54400000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.433,"5min":0.51300000000000001,"15min":0.52700000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.65900000000000003,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.70799999999999996}]},{"osd":6,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.46500000000000002,"15min":0.47499999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.59299999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.52400000000000002,"15min":0.54000000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.71699999999999997,"5min":1.054,"15min":1.054},"last":0.65600000000000003}]}]},{"osd":6,"up_from":14,"seq":60129542178,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.42699999999999999,"15min":0.42199999999999999},"min":{"1min":0.313,"5min":0.25,"15min":0.25},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.47399999999999998,"15min":0.47899999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.52400000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.69099999999999995}]},{"osd":1,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.57399999999999995,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.495,"15min":0.497},"min":{"1min":0.373,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.624,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.61499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":3,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68799999999999994,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66500000000000004,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.70699999999999996}]},{"osd":4,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.46200000000000002,"15min":0.46999999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57199999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.47899999999999998,"15min":0.48699999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.64900000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.46600000000000003}]},{"osd":5,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.28499999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.68799999999999994,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.5},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.313,"5min":0.308,"15min":0.308},"max":{"1min":0.65600000000000003,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.60399999999999998}]},{"osd":7,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53700000000000003,"15min":0.54100000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.51300000000000001,"15min":0.51700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64900000000000002,"5min":0.872,"15min":0.872},"last":0.57599999999999996}]}]},{"osd":4,"up_from":14,"seq":60129542178,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.40500000000000003,"15min":0.40200000000000002},"min":{"1min":0.308,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.435,"15min":0.434},"min":{"1min":0.33600000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.64600000000000002,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.52700000000000002}]},{"osd":1,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66300000000000003,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46999999999999997,"15min":0.47299999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.58999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":3,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.51100000000000001,"15min":0.51900000000000002},"min":{"1min":0.38100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.58399999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.499,"15min":0.501},"min":{"1min":0.31900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.66900000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.503}]},{"osd":5,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.44400000000000001,"15min":0.45000000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.64700000000000002,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.45100000000000001,"15min":0.45400000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64100000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.34100000000000003}]},{"osd":6,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66500000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.61199999999999999,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.36099999999999999}]},{"osd":7,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.49399999999999999,"15min":0.49199999999999999},"min":{"1min":0.34100000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71699999999999997,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.38100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.50700000000000001,"15min":0.52900000000000003},"min":{"1min":0.219,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.127,"5min":3.2959999999999998,"15min":3.2959999999999998},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.56699999999999995,"15min":0.58499999999999996},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":1.161,"5min":3.2799999999999998,"15min":3.2799999999999998},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":3,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":3.3239999999999998,"15min":3.3239999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.47799999999999998,"15min":0.47699999999999998},"min":{"1min":0.25900000000000001,"5min":0.253,"15min":0.253},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.39100000000000001}]},{"osd":4,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.375,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59299999999999997,"15min":0.58899999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":0.63700000000000001}]},{"osd":5,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.39300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.628,"15min":0.624},"min":{"1min":0.376,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.66400000000000003}]},{"osd":6,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.61899999999999999,"15min":0.625},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0920000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.501},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.70399999999999996,"15min":0.71999999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1779999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":0.622}]},{"osd":7,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.667,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63800000000000001,"15min":0.64200000000000002},"min":{"1min":0.41599999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.57199999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542178,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28128,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343712,"statfs":{"total":96636764160,"available":96607961088,"internally_reserved":0,"allocated":1130496,"data_stored":988208,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.40500000000000003,"15min":0.40899999999999997},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.79800000000000004,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.434},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40100000000000002,"15min":0.40899999999999997},"min":{"1min":0.23200000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.64200000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.46800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":3,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.46999999999999997,"15min":0.48199999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.59199999999999997,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.48599999999999999,"15min":0.49399999999999999},"min":{"1min":0.28799999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.53300000000000003}]},{"osd":4,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.871,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.56599999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82699999999999996,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.72299999999999998}]},{"osd":5,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54200000000000004,"15min":0.53900000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.52600000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.69499999999999995}]},{"osd":6,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.38100000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59199999999999997,"15min":0.59599999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.68000000000000005}]},{"osd":7,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57299999999999995,"15min":0.56799999999999995},"min":{"1min":0.378,"5min":0.377,"15min":0.377},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.56299999999999994,"15min":0.56899999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.70899999999999996,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.74399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T21:42:09.453 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-27T21:42:09.453 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T21:42:09.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.567+0000 7f2e6f409640 10 monclient: get_monmap_and_config 2024-03-27T21:42:09.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.567+0000 7f2e6f409640 10 monclient: build_initial_monmap 2024-03-27T21:42:09.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.567+0000 7f2e6f409640 10 monclient: monmap: 2024-03-27T21:42:09.554 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:09.554 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:09.554 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:09.568028+0000 2024-03-27T21:42:09.554 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:09.568028+0000 2024-03-27T21:42:09.554 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:09.554 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:09.554 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:09.554 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:09.555 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:09.555 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:09.555 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:09.555 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:09.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.567+0000 7f2e6f409640 1 Processor -- start 2024-03-27T21:42:09.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.567+0000 7f2e6f409640 1 -- start start 2024-03-27T21:42:09.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.567+0000 7f2e6f409640 10 monclient: init 2024-03-27T21:42:09.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6f409640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:09.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6f409640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-03-27T21:42:09.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6f409640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2e68071a50 0x7f2e680a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:09.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6f409640 10 monclient(hunting): picked mon.noname-b con 0x7f2e68071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:09.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6f409640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e6813e810 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:09.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6f409640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e6813ec50 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:09.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6f409640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e680a8e90 con 0x7f2e6813e810 2024-03-27T21:42:09.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6f409640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e680aa040 con 0x7f2e6813ec50 2024-03-27T21:42:09.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6f409640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:09.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6f409640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e68071e30 con 0x7f2e68071a50 2024-03-27T21:42:09.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6f409640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:09.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6f409640 10 monclient(hunting): authenticate will time out at 970.789272s 2024-03-27T21:42:09.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6d1a5640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2e68071a50 0x7f2e680a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:09.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6c9a4640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2e6813e810 0x7f2e68141630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39500/0 (socket says 172.21.15.60:39500) 2024-03-27T21:42:09.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6c9a4640 1 -- 172.21.15.60:0/4096576355 learned_addr learned my addr 172.21.15.60:0/4096576355 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:09.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6d1a5640 10 monclient(hunting): get_auth_request con 0x7f2e68071a50 auth_method 0 2024-03-27T21:42:09.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6d1a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:09.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6d1a5640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:09.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6d1a5640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:09.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6d1a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:09.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6d1a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:09.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6d1a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:09.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6d1a5640 10 monclient(hunting): handle_auth_done global_id 5117 payload 274 2024-03-27T21:42:09.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6d1a5640 1 -- 172.21.15.60:0/4096576355 >> v1:172.21.15.191:6789/0 conn(0x7f2e6813ec50 legacy=0x7f2e68141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:09.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e67fff640 1 -- 172.21.15.60:0/4096576355 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031127876 0 0) 0x7f2e680a8e90 con 0x7f2e6813e810 2024-03-27T21:42:09.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6d1a5640 1 -- 172.21.15.60:0/4096576355 >> v1:172.21.15.60:6789/0 conn(0x7f2e6813e810 legacy=0x7f2e68141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:09.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6d1a5640 10 monclient: _finish_hunting 0 2024-03-27T21:42:09.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6d1a5640 1 monclient: found mon.noname-b 2024-03-27T21:42:09.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6d1a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:09.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6d1a5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:42:09.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e6d1a5640 1 -- 172.21.15.60:0/4096576355 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e680aa2b0 con 0x7f2e68071a50 2024-03-27T21:42:09.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.568+0000 7f2e67fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:42:09.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.569+0000 7f2e6d1a5640 1 --2- 172.21.15.60:0/4096576355 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2e68071a50 0x7f2e680a87e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2e600097d0 tx=0x7f2e6002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcee400e87a9c545 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:09.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.569+0000 7f2e67fff640 1 -- 172.21.15.60:0/4096576355 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e60031800 con 0x7f2e68071a50 2024-03-27T21:42:09.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.569+0000 7f2e67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:09.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.569+0000 7f2e67fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:09.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.569+0000 7f2e67fff640 10 monclient: dump: 2024-03-27T21:42:09.567 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:09.567 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:09.567 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.567 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.568 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:09.568 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:09.568 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:09.568 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:09.568 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:09.568 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:09.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.569+0000 7f2e67fff640 10 monclient: _finish_auth 0 2024-03-27T21:42:09.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.569+0000 7f2e67fff640 10 monclient: _check_auth_tickets 2024-03-27T21:42:09.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.569+0000 7f2e67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:09.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.569+0000 7f2e67fff640 1 -- 172.21.15.60:0/4096576355 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e60031960 con 0x7f2e68071a50 2024-03-27T21:42:09.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.569+0000 7f2e6f409640 5 monclient: authenticate success, global_id 5117 2024-03-27T21:42:09.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.569+0000 7f2e6f409640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:09.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.569+0000 7f2e67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:09.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.569+0000 7f2e6f409640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:09.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.569+0000 7f2e67fff640 1 -- 172.21.15.60:0/4096576355 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e60031cf0 con 0x7f2e68071a50 2024-03-27T21:42:09.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.569+0000 7f2e67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:09.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.569+0000 7f2e6f409640 4 set_mon_vals no callback set 2024-03-27T21:42:09.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.569+0000 7f2e67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:09.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.569+0000 7f2e67fff640 10 monclient: dump: 2024-03-27T21:42:09.570 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:09.570 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:09.570 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.570 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.570 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:09.570 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:09.570 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:09.571 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:09.571 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:09.571 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:09.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.570+0000 7f2e6f409640 10 monclient: shutdown 2024-03-27T21:42:09.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.570+0000 7f2e6f409640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:09.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.570+0000 7f2e6f409640 1 -- 172.21.15.60:0/4096576355 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2e68071a50 msgr2=0x7f2e680a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:09.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.570+0000 7f2e6f409640 1 --2- 172.21.15.60:0/4096576355 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2e68071a50 0x7f2e680a87e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2e600097d0 tx=0x7f2e6002b8a0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.570+0000 7f2e6f409640 1 -- 172.21.15.60:0/4096576355 shutdown_connections 2024-03-27T21:42:09.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.570+0000 7f2e6f409640 1 --2- 172.21.15.60:0/4096576355 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2e68071a50 0x7f2e680a87e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.570+0000 7f2e6f409640 1 -- 172.21.15.60:0/4096576355 >> 172.21.15.60:0/4096576355 conn(0x7f2e680b7300 msgr2=0x7f2e680b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:09.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.570+0000 7f2e6f409640 1 -- 172.21.15.60:0/4096576355 shutdown_connections 2024-03-27T21:42:09.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.570+0000 7f2e6f409640 1 -- 172.21.15.60:0/4096576355 wait complete. 2024-03-27T21:42:09.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.570+0000 7f2e6f409640 10 monclient: build_initial_monmap 2024-03-27T21:42:09.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.570+0000 7f2e6f409640 10 monclient: monmap: 2024-03-27T21:42:09.572 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:09.572 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:09.572 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:09.573 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:09.573 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:09.573 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:09.573 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:09.573 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:09.573 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:09.573 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:09.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 1 librados: starting msgr at 2024-03-27T21:42:09.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 1 librados: starting objecter 2024-03-27T21:42:09.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 1 Processor -- start 2024-03-27T21:42:09.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 1 -- start start 2024-03-27T21:42:09.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 1 librados: setting wanted keys 2024-03-27T21:42:09.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 1 librados: calling monclient init 2024-03-27T21:42:09.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 10 monclient: init 2024-03-27T21:42:09.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:09.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:42:09.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e68071a50 0x7f2e68139090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:09.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 10 monclient(hunting): picked mon.noname-b con 0x7f2e68071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:09.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e6813e810 0x7f2e681398d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:09.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 10 monclient(hunting): picked mon.noname-a con 0x7f2e6813e810 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:09.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2e6813ec50 0x7f2e68136200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:09.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 10 monclient(hunting): picked mon.noname-c con 0x7f2e6813ec50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:09.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:09.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e68138c60 con 0x7f2e6813e810 2024-03-27T21:42:09.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:09.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6813aa30 con 0x7f2e68071a50 2024-03-27T21:42:09.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:09.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6813ad10 con 0x7f2e6813ec50 2024-03-27T21:42:09.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:09.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6f409640 10 monclient(hunting): authenticate will time out at 970.792719s 2024-03-27T21:42:09.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6c9a4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e6813e810 0x7f2e681398d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:09.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6d9a6640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2e6813ec50 0x7f2e68136200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:09.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6c9a4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e6813e810 0x7f2e681398d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39678/0 (socket says 172.21.15.60:39678) 2024-03-27T21:42:09.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6c9a4640 1 -- 172.21.15.60:0/3326593805 learned_addr learned my addr 172.21.15.60:0/3326593805 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:09.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6d9a6640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2e6813ec50 0x7f2e68136200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34492/0 (socket says 172.21.15.60:34492) 2024-03-27T21:42:09.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6c9a4640 10 monclient(hunting): get_auth_request con 0x7f2e6813e810 auth_method 0 2024-03-27T21:42:09.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6c9a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:09.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6c9a4640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:09.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6c9a4640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:09.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6d9a6640 10 monclient(hunting): get_auth_request con 0x7f2e6813ec50 auth_method 0 2024-03-27T21:42:09.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6d9a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:09.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6d9a6640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:09.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6d9a6640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:09.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6c9a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:09.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.571+0000 7f2e6c9a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:09.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6c9a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:09.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d9a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:09.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d9a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:09.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d9a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:09.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d1a5640 1 --2- 172.21.15.60:0/3326593805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e68071a50 0x7f2e68139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:09.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d1a5640 10 monclient(hunting): get_auth_request con 0x7f2e68071a50 auth_method 0 2024-03-27T21:42:09.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d1a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:09.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d1a5640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:09.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d1a5640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:09.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d1a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:09.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d1a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:09.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d1a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:09.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d9a6640 10 monclient(hunting): handle_auth_done global_id 5120 payload 824 2024-03-27T21:42:09.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d9a6640 1 -- 172.21.15.60:0/3326593805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e68071a50 msgr2=0x7f2e68139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:09.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d9a6640 1 --2- 172.21.15.60:0/3326593805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e68071a50 0x7f2e68139090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d9a6640 1 -- 172.21.15.60:0/3326593805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e6813e810 msgr2=0x7f2e681398d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:09.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d9a6640 1 --2- 172.21.15.60:0/3326593805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e6813e810 0x7f2e681398d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d9a6640 10 monclient: _finish_hunting 0 2024-03-27T21:42:09.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d9a6640 1 monclient: found mon.noname-c 2024-03-27T21:42:09.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d9a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:09.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d9a6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:09.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d9a6640 1 -- 172.21.15.60:0/3326593805 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e60009400 con 0x7f2e6813ec50 2024-03-27T21:42:09.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6c9a4640 1 --2- 172.21.15.60:0/3326593805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e6813e810 0x7f2e681398d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:09.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6d9a6640 1 --2- 172.21.15.60:0/3326593805 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2e6813ec50 0x7f2e68136200 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2e5800be50 tx=0x7f2e5800bf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:09.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e65ffb640 10 client.?.objecter ms_handle_connect 0x7f2e6813ec50 2024-03-27T21:42:09.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e65ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:09.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e65ffb640 1 -- 172.21.15.60:0/3326593805 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e58014440 con 0x7f2e6813ec50 2024-03-27T21:42:09.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:09.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e65ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:09.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e65ffb640 10 monclient: dump: 2024-03-27T21:42:09.581 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:09.581 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:09.581 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.581 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.582 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:09.582 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:09.582 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:09.582 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:09.582 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:09.582 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:09.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e65ffb640 10 monclient: _finish_auth 0 2024-03-27T21:42:09.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e65ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:42:09.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:09.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e65ffb640 1 -- 172.21.15.60:0/3326593805 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e580145a0 con 0x7f2e6813ec50 2024-03-27T21:42:09.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6f409640 5 monclient: authenticate success, global_id 5120 2024-03-27T21:42:09.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6f409640 10 monclient: _renew_subs 2024-03-27T21:42:09.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6f409640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:09.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6f409640 1 -- 172.21.15.60:0/3326593805 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e68137a70 con 0x7f2e6813ec50 2024-03-27T21:42:09.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e65ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:09.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e65ffb640 1 -- 172.21.15.60:0/3326593805 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e58014950 con 0x7f2e6813ec50 2024-03-27T21:42:09.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:09.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e65ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:09.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e65ffb640 10 monclient: dump: 2024-03-27T21:42:09.584 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:09.584 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:09.584 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.584 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.584 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:09.584 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:09.584 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:09.584 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:09.585 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:09.585 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:09.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e677fe640 4 set_mon_vals no callback set 2024-03-27T21:42:09.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6f409640 10 client.5120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:09.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6f409640 10 monclient: _renew_subs 2024-03-27T21:42:09.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6f409640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:09.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6f409640 1 -- 172.21.15.60:0/3326593805 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e68136920 con 0x7f2e6813ec50 2024-03-27T21:42:09.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.572+0000 7f2e6f409640 1 librados: init done 2024-03-27T21:42:09.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.573+0000 7f2e6f409640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:09.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.573+0000 7f2e6f409640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:09.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.573+0000 7f2e6f409640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:09.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.573+0000 7f2e6f409640 1 -- 172.21.15.60:0/3326593805 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e68067900 con 0x7f2e6813ec50 2024-03-27T21:42:09.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.574+0000 7f2e65ffb640 1 -- 172.21.15.60:0/3326593805 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2e58014ab0 con 0x7f2e6813ec50 2024-03-27T21:42:09.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.574+0000 7f2e65ffb640 1 --2- 172.21.15.60:0/3326593805 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2e5406c590 0x7f2e5406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:09.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.574+0000 7f2e6d1a5640 1 --2- 172.21.15.60:0/3326593805 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2e5406c590 0x7f2e5406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:09.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.574+0000 7f2e6d1a5640 10 monclient: get_auth_request con 0x7f2e5406c590 auth_method 0 2024-03-27T21:42:09.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.574+0000 7f2e65ffb640 1 -- 172.21.15.60:0/3326593805 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f2e58089240 con 0x7f2e6813ec50 2024-03-27T21:42:09.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.574+0000 7f2e65ffb640 10 client.5120.objecter ms_dispatch 0x7f2e68000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:09.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.574+0000 7f2e65ffb640 3 client.5120.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:09.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.574+0000 7f2e65ffb640 3 client.5120.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:09.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.574+0000 7f2e65ffb640 20 client.5120.objecter dump_active .. 0 homeless 2024-03-27T21:42:09.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.575+0000 7f2e65ffb640 10 client.5120.objecter ms_handle_connect 0x7f2e5406c590 2024-03-27T21:42:09.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.575+0000 7f2e6d1a5640 1 --2- 172.21.15.60:0/3326593805 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2e5406c590 0x7f2e5406ea50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2e60036000 tx=0x7f2e6003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:09.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.579+0000 7f2e65ffb640 1 -- 172.21.15.60:0/3326593805 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2e58054730 con 0x7f2e6813ec50 2024-03-27T21:42:09.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.579+0000 7f2e65ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:09.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.579+0000 7f2e65ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:09.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.754+0000 7f2e6f409640 1 -- 172.21.15.60:0/3326593805 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2e6813f0c0 con 0x7f2e5406c590 2024-03-27T21:42:09.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.757+0000 7f2e65ffb640 1 -- 172.21.15.60:0/3326593805 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+66914 (secure 0 0 0) 0x7f2e60045410 con 0x7f2e5406c590 2024-03-27T21:42:09.743 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:42:09.744 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-03-27T21:42:09.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.758+0000 7f2e6f409640 10 librados: watch_flush enter 2024-03-27T21:42:09.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.758+0000 7f2e6f409640 10 librados: watch_flush exit 2024-03-27T21:42:09.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.758+0000 7f2e6f409640 20 client.5120.objecter shutdown clearing up homeless session... 2024-03-27T21:42:09.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.758+0000 7f2e6f409640 10 client.5120.objecter successfully canceled tick 2024-03-27T21:42:09.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.758+0000 7f2e6f409640 1 -- 172.21.15.60:0/3326593805 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2e5406c590 msgr2=0x7f2e5406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:09.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.758+0000 7f2e6f409640 1 --2- 172.21.15.60:0/3326593805 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2e5406c590 0x7f2e5406ea50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2e60036000 tx=0x7f2e6003c040 comp rx=0 tx=0).stop 2024-03-27T21:42:09.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.758+0000 7f2e6f409640 10 monclient: shutdown 2024-03-27T21:42:09.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.759+0000 7f2e6f409640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:09.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.759+0000 7f2e6f409640 1 -- 172.21.15.60:0/3326593805 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2e6813ec50 msgr2=0x7f2e68136200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:09.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.759+0000 7f2e6f409640 1 --2- 172.21.15.60:0/3326593805 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2e6813ec50 0x7f2e68136200 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2e5800be50 tx=0x7f2e5800bf40 comp rx=0 tx=0).stop 2024-03-27T21:42:09.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.759+0000 7f2e6f409640 1 -- 172.21.15.60:0/3326593805 shutdown_connections 2024-03-27T21:42:09.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.759+0000 7f2e6f409640 1 --2- 172.21.15.60:0/3326593805 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f2e6813ec50 0x7f2e68136200 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.759+0000 7f2e6f409640 1 --2- 172.21.15.60:0/3326593805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e6813e810 0x7f2e681398d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.759+0000 7f2e6f409640 1 --2- 172.21.15.60:0/3326593805 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f2e5406c590 0x7f2e5406ea50 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.759+0000 7f2e6f409640 1 --2- 172.21.15.60:0/3326593805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e68071a50 0x7f2e68139090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.759+0000 7f2e6f409640 1 -- 172.21.15.60:0/3326593805 >> 172.21.15.60:0/3326593805 conn(0x7f2e680b7300 msgr2=0x7f2e681349e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:09.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.759+0000 7f2e6f409640 1 -- 172.21.15.60:0/3326593805 shutdown_connections 2024-03-27T21:42:09.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.759+0000 7f2e6f409640 1 -- 172.21.15.60:0/3326593805 wait complete. 2024-03-27T21:42:09.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.759+0000 7f2e6f409640 1 librados: shutdown 2024-03-27T21:42:09.756 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-03-27T21:42:09.413031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":230448,"kb_used_data":8520,"kb_used_omap":16,"kb_used_meta":221807,"kb_avail":754744272,"statfs":{"total":773094113280,"available":772858134528,"internally_reserved":0,"allocated":8724480,"data_stored":4732096,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":17199,"internal_metadata":227130577},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.148977"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.602437+0000","last_change":"2024-03-27T21:42:01.265365+0000","last_active":"2024-03-27T21:42:03.602437+0000","last_peered":"2024-03-27T21:42:03.602437+0000","last_clean":"2024-03-27T21:42:03.602437+0000","last_became_active":"2024-03-27T21:42:01.265033+0000","last_became_peered":"2024-03-27T21:42:01.265033+0000","last_unstale":"2024-03-27T21:42:03.602437+0000","last_undegraded":"2024-03-27T21:42:03.602437+0000","last_fullsized":"2024-03-27T21:42:03.602437+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:07:09.198158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.601878+0000","last_change":"2024-03-27T21:41:49.474318+0000","last_active":"2024-03-27T21:42:03.601878+0000","last_peered":"2024-03-27T21:42:03.601878+0000","last_clean":"2024-03-27T21:42:03.601878+0000","last_became_active":"2024-03-27T21:41:49.474054+0000","last_became_peered":"2024-03-27T21:41:49.474054+0000","last_unstale":"2024-03-27T21:42:03.601878+0000","last_undegraded":"2024-03-27T21:42:03.601878+0000","last_fullsized":"2024-03-27T21:42:03.601878+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:36:26.918691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271058+0000","last_change":"2024-03-27T21:39:46.679788+0000","last_active":"2024-03-27T21:42:03.271058+0000","last_peered":"2024-03-27T21:42:03.271058+0000","last_clean":"2024-03-27T21:42:03.271058+0000","last_became_active":"2024-03-27T21:39:46.679561+0000","last_became_peered":"2024-03-27T21:39:46.679561+0000","last_unstale":"2024-03-27T21:42:03.271058+0000","last_undegraded":"2024-03-27T21:42:03.271058+0000","last_fullsized":"2024-03-27T21:42:03.271058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T23:53:50.373923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.601870+0000","last_change":"2024-03-27T21:41:49.474121+0000","last_active":"2024-03-27T21:42:03.601870+0000","last_peered":"2024-03-27T21:42:03.601870+0000","last_clean":"2024-03-27T21:42:03.601870+0000","last_became_active":"2024-03-27T21:41:49.473822+0000","last_became_peered":"2024-03-27T21:41:49.473822+0000","last_unstale":"2024-03-27T21:42:03.601870+0000","last_undegraded":"2024-03-27T21:42:03.601870+0000","last_fullsized":"2024-03-27T21:42:03.601870+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:39:51.253704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271086+0000","last_change":"2024-03-27T21:41:59.701200+0000","last_active":"2024-03-27T21:42:03.271086+0000","last_peered":"2024-03-27T21:42:03.271086+0000","last_clean":"2024-03-27T21:42:03.271086+0000","last_became_active":"2024-03-27T21:41:59.700966+0000","last_became_peered":"2024-03-27T21:41:59.700966+0000","last_unstale":"2024-03-27T21:42:03.271086+0000","last_undegraded":"2024-03-27T21:42:03.271086+0000","last_fullsized":"2024-03-27T21:42:03.271086+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:17:01.025558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.270838+0000","last_change":"2024-03-27T21:41:30.080153+0000","last_active":"2024-03-27T21:42:03.270838+0000","last_peered":"2024-03-27T21:42:03.270838+0000","last_clean":"2024-03-27T21:42:03.270838+0000","last_became_active":"2024-03-27T21:41:30.079829+0000","last_became_peered":"2024-03-27T21:41:30.079829+0000","last_unstale":"2024-03-27T21:42:03.270838+0000","last_undegraded":"2024-03-27T21:42:03.270838+0000","last_fullsized":"2024-03-27T21:42:03.270838+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:21:00.966562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":97,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271058+0000","last_change":"2024-03-27T21:41:49.197510+0000","last_active":"2024-03-27T21:42:03.271058+0000","last_peered":"2024-03-27T21:42:03.271058+0000","last_clean":"2024-03-27T21:42:03.271058+0000","last_became_active":"2024-03-27T21:41:49.197154+0000","last_became_peered":"2024-03-27T21:41:49.197154+0000","last_unstale":"2024-03-27T21:42:03.271058+0000","last_undegraded":"2024-03-27T21:42:03.271058+0000","last_fullsized":"2024-03-27T21:42:03.271058+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:18:21.622988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271153+0000","last_change":"2024-03-27T21:41:59.247145+0000","last_active":"2024-03-27T21:42:03.271153+0000","last_peered":"2024-03-27T21:42:03.271153+0000","last_clean":"2024-03-27T21:42:03.271153+0000","last_became_active":"2024-03-27T21:41:59.246895+0000","last_became_peered":"2024-03-27T21:41:59.246895+0000","last_unstale":"2024-03-27T21:42:03.271153+0000","last_undegraded":"2024-03-27T21:42:03.271153+0000","last_fullsized":"2024-03-27T21:42:03.271153+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:40:39.749875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":234,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271065+0000","last_change":"2024-03-27T21:39:39.866806+0000","last_active":"2024-03-27T21:42:03.271065+0000","last_peered":"2024-03-27T21:42:03.271065+0000","last_clean":"2024-03-27T21:42:03.271065+0000","last_became_active":"2024-03-27T21:39:39.865948+0000","last_became_peered":"2024-03-27T21:39:39.865948+0000","last_unstale":"2024-03-27T21:42:03.271065+0000","last_undegraded":"2024-03-27T21:42:03.271065+0000","last_fullsized":"2024-03-27T21:42:03.271065+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_clean_scrub_stamp":"2024-03-27T21:39:38.411432+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:36:49.375869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":16,"seq":68719476771,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.41599999999999998,"15min":0.41199999999999998},"min":{"1min":0.34399999999999997,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.621,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.498},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.47499999999999998,"15min":0.47599999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.66900000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.70499999999999996}]},{"osd":1,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45900000000000002,"15min":0.46500000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.53600000000000003,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.35899999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.56599999999999995,"5min":0.745,"15min":0.745},"last":0.371}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":3,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.50600000000000001,"15min":0.51400000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.56899999999999995,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.35999999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68400000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.56000000000000005}]},{"osd":4,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.432,"15min":0.439},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.56399999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.433,"5min":0.46800000000000003,"15min":0.47399999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.78400000000000003}]},{"osd":6,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.45800000000000002,"15min":0.46300000000000002},"min":{"1min":0.318,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58699999999999997,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.29299999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64300000000000002,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.73999999999999999}]},{"osd":7,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.52400000000000002,"15min":0.53200000000000003},"min":{"1min":0.378,"5min":0.318,"15min":0.318},"max":{"1min":0.61299999999999999,"5min":0.745,"15min":0.745},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.50800000000000001,"15min":0.51400000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65000000000000002,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.72399999999999998}]}]},{"osd":3,"up_from":16,"seq":68719476770,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31848905},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.49299999999999999,"15min":0.504},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.91700000000000004,"5min":1.248,"15min":1.248},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.44900000000000001,"15min":0.45600000000000002},"min":{"1min":0.22800000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.90600000000000003,"5min":1.407,"15min":1.407},"last":0.36599999999999999}]},{"osd":1,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48199999999999998,"15min":0.47999999999999998},"min":{"1min":0.25600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.26800000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.96899999999999997,"5min":1.347,"15min":1.347},"last":0.42299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":4,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.61399999999999999,"15min":0.626},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83299999999999996,"5min":1.5,"15min":1.5},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.40000000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72799999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.54000000000000004}]},{"osd":5,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56000000000000005,"15min":0.56599999999999995},"min":{"1min":0.34999999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71099999999999997,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.63300000000000001,"15min":0.64700000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.96799999999999997,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.46000000000000002}]},{"osd":6,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.63100000000000001,"15min":0.63900000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88200000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.40999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76700000000000002,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.503}]},{"osd":7,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.64200000000000002,"15min":0.65600000000000003},"min":{"1min":0.36899999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.94399999999999995,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61599999999999999,"15min":0.622},"min":{"1min":0.36699999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85799999999999998,"5min":1.454,"15min":1.454},"last":0.55700000000000005}]}]},{"osd":2,"up_from":62,"seq":266287972362,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29736,"kb_used_data":1052,"kb_used_omap":5,"kb_used_meta":28666,"kb_avail":94342104,"statfs":{"total":96636764160,"available":96606314496,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6051,"internal_metadata":29354077},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]}]},{"osd":7,"up_from":14,"seq":60129542178,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28128,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343712,"statfs":{"total":96636764160,"available":96607961088,"internally_reserved":0,"allocated":1130496,"data_stored":988208,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.45200000000000001,"15min":0.45800000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.56299999999999994,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.42299999999999999,"15min":0.42399999999999999},"min":{"1min":0.29799999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60499999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.58299999999999996}]},{"osd":1,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45500000000000002,"15min":0.46100000000000002},"min":{"1min":0.29899999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.55200000000000005,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.44,"5min":0.44800000000000001,"15min":0.45000000000000001},"min":{"1min":0.28499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":3,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83199999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.49299999999999999,"15min":0.497},"min":{"1min":0.37,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.64600000000000002,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.66900000000000004}]},{"osd":4,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.42799999999999999,"15min":0.439},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.53600000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.47599999999999998,"15min":0.48799999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.60099999999999998,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.68799999999999994}]},{"osd":5,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.54400000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.433,"5min":0.51300000000000001,"15min":0.52700000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.65900000000000003,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.70799999999999996}]},{"osd":6,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.46500000000000002,"15min":0.47499999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.59299999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.52400000000000002,"15min":0.54000000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.71699999999999997,"5min":1.054,"15min":1.054},"last":0.65600000000000003}]}]},{"osd":6,"up_from":14,"seq":60129542178,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.42699999999999999,"15min":0.42199999999999999},"min":{"1min":0.313,"5min":0.25,"15min":0.25},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.47399999999999998,"15min":0.47899999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.52400000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.69099999999999995}]},{"osd":1,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.57399999999999995,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.495,"15min":0.497},"min":{"1min":0.373,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.624,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.61499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":3,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68799999999999994,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66500000000000004,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.70699999999999996}]},{"osd":4,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.46200000000000002,"15min":0.46999999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57199999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.47899999999999998,"15min":0.48699999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.64900000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.46600000000000003}]},{"osd":5,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.28499999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.68799999999999994,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.5},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.313,"5min":0.308,"15min":0.308},"max":{"1min":0.65600000000000003,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.60399999999999998}]},{"osd":7,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53700000000000003,"15min":0.54100000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.51300000000000001,"15min":0.51700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64900000000000002,"5min":0.872,"15min":0.872},"last":0.57599999999999996}]}]},{"osd":4,"up_from":14,"seq":60129542178,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.40500000000000003,"15min":0.40200000000000002},"min":{"1min":0.308,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.435,"15min":0.434},"min":{"1min":0.33600000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.64600000000000002,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.52700000000000002}]},{"osd":1,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66300000000000003,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46999999999999997,"15min":0.47299999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.58999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":3,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.51100000000000001,"15min":0.51900000000000002},"min":{"1min":0.38100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.58399999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.499,"15min":0.501},"min":{"1min":0.31900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.66900000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.503}]},{"osd":5,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.44400000000000001,"15min":0.45000000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.64700000000000002,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.45100000000000001,"15min":0.45400000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64100000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.34100000000000003}]},{"osd":6,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66500000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.61199999999999999,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.36099999999999999}]},{"osd":7,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.49399999999999999,"15min":0.49199999999999999},"min":{"1min":0.34100000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71699999999999997,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.38100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.50700000000000001,"15min":0.52900000000000003},"min":{"1min":0.219,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.127,"5min":3.2959999999999998,"15min":3.2959999999999998},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.56699999999999995,"15min":0.58499999999999996},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":1.161,"5min":3.2799999999999998,"15min":3.2799999999999998},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":3,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":3.3239999999999998,"15min":3.3239999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.47799999999999998,"15min":0.47699999999999998},"min":{"1min":0.25900000000000001,"5min":0.253,"15min":0.253},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.39100000000000001}]},{"osd":4,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.375,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59299999999999997,"15min":0.58899999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":0.63700000000000001}]},{"osd":5,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.39300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.628,"15min":0.624},"min":{"1min":0.376,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.66400000000000003}]},{"osd":6,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.61899999999999999,"15min":0.625},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0920000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.501},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.70399999999999996,"15min":0.71999999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1779999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":0.622}]},{"osd":7,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.667,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63800000000000001,"15min":0.64200000000000002},"min":{"1min":0.41599999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.57199999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542178,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28128,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343712,"statfs":{"total":96636764160,"available":96607961088,"internally_reserved":0,"allocated":1130496,"data_stored":988208,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.40500000000000003,"15min":0.40899999999999997},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.79800000000000004,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.434},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40100000000000002,"15min":0.40899999999999997},"min":{"1min":0.23200000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.64200000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.46800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":3,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.46999999999999997,"15min":0.48199999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.59199999999999997,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.48599999999999999,"15min":0.49399999999999999},"min":{"1min":0.28799999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.53300000000000003}]},{"osd":4,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.871,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.56599999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82699999999999996,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.72299999999999998}]},{"osd":5,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54200000000000004,"15min":0.53900000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.52600000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.69499999999999995}]},{"osd":6,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.38100000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59199999999999997,"15min":0.59599999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.68000000000000005}]},{"osd":7,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57299999999999995,"15min":0.56799999999999995},"min":{"1min":0.378,"5min":0.377,"15min":0.377},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.56299999999999994,"15min":0.56899999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.70899999999999996,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.74399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T21:42:09.759 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T21:42:09.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.880+0000 7f4fc3337640 10 monclient: get_monmap_and_config 2024-03-27T21:42:09.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.880+0000 7f4fc3337640 10 monclient: build_initial_monmap 2024-03-27T21:42:09.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.880+0000 7f4fc3337640 10 monclient: monmap: 2024-03-27T21:42:09.867 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:09.867 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:09.867 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:09.881307+0000 2024-03-27T21:42:09.867 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:09.881307+0000 2024-03-27T21:42:09.867 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:09.867 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:09.867 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:09.867 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:09.868 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:09.868 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:09.868 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:09.868 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:09.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.880+0000 7f4fc3337640 1 Processor -- start 2024-03-27T21:42:09.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.881+0000 7f4fc3337640 1 -- start start 2024-03-27T21:42:09.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.881+0000 7f4fc3337640 10 monclient: init 2024-03-27T21:42:09.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.881+0000 7f4fc3337640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:09.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.881+0000 7f4fc3337640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-03-27T21:42:09.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.881+0000 7f4fc3337640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4fbc071a50 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:09.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.881+0000 7f4fc3337640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4fbc13ef20 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:09.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.881+0000 7f4fc3337640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4fbc0a87e0 0x7f4fbc141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:09.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.881+0000 7f4fc3337640 10 monclient(hunting): picked mon.noname-c con 0x7f4fbc0a87e0 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:09.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.881+0000 7f4fc3337640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fbc0a8e90 con 0x7f4fbc071a50 2024-03-27T21:42:09.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.881+0000 7f4fc3337640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fbc0aa040 con 0x7f4fbc13ef20 2024-03-27T21:42:09.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.881+0000 7f4fc3337640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:09.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.881+0000 7f4fc3337640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fbc071e30 con 0x7f4fbc0a87e0 2024-03-27T21:42:09.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.881+0000 7f4fc3337640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:09.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.881+0000 7f4fc3337640 10 monclient(hunting): authenticate will time out at 971.102569s 2024-03-27T21:42:09.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.881+0000 7f4fc10d3640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4fbc071a50 0x7f4fbc13e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39504/0 (socket says 172.21.15.60:39504) 2024-03-27T21:42:09.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.881+0000 7f4fc10d3640 1 -- 172.21.15.60:0/4039142858 learned_addr learned my addr 172.21.15.60:0/4039142858 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:09.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.881+0000 7f4fbbfff640 1 -- 172.21.15.60:0/4039142858 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542505061 0 0) 0x7f4fbc0a8e90 con 0x7f4fbc071a50 2024-03-27T21:42:09.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 10 monclient(hunting): _negotiate 2024-03-27T21:42:09.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:09.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:09.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 10 monclient(hunting): my global_id is 5436 2024-03-27T21:42:09.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc18d4640 1 --2- 172.21.15.60:0/4039142858 >> v2:172.21.15.191:3300/0 conn(0x7f4fbc0a87e0 0x7f4fbc141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:09.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 1 -- 172.21.15.60:0/4039142858 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fa4003520 con 0x7f4fbc071a50 2024-03-27T21:42:09.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc18d4640 10 monclient(hunting): get_auth_request con 0x7f4fbc0a87e0 auth_method 0 2024-03-27T21:42:09.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc18d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:09.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc18d4640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:09.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc18d4640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:09.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc18d4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:09.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc18d4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:09.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc18d4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:09.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 1 -- 172.21.15.60:0/4039142858 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2066332877 0 0) 0x7f4fa4003520 con 0x7f4fbc071a50 2024-03-27T21:42:09.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 1 -- 172.21.15.60:0/4039142858 >> v2:172.21.15.191:3300/0 conn(0x7f4fbc0a87e0 msgr2=0x7f4fbc141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:09.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 1 --2- 172.21.15.60:0/4039142858 >> v2:172.21.15.191:3300/0 conn(0x7f4fbc0a87e0 0x7f4fbc141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 1 -- 172.21.15.60:0/4039142858 >> v1:172.21.15.191:6789/0 conn(0x7f4fbc13ef20 legacy=0x7f4fbc141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:09.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 10 monclient: _finish_hunting 0 2024-03-27T21:42:09.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 1 monclient: found mon.noname-a-legacy 2024-03-27T21:42:09.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:09.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.60:6789/0 2024-03-27T21:42:09.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 1 -- 172.21.15.60:0/4039142858 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fbc142280 con 0x7f4fbc071a50 2024-03-27T21:42:09.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 10 monclient: _finish_auth 0 2024-03-27T21:42:09.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 10 monclient: _check_auth_tickets 2024-03-27T21:42:09.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:09.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 1 -- 172.21.15.60:0/4039142858 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2933500084 0 0) 0x7f4fac0019f0 con 0x7f4fbc071a50 2024-03-27T21:42:09.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:09.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-03-27T21:42:09.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 10 monclient: dump: 2024-03-27T21:42:09.885 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:09.885 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:09.885 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.886 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.886 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:09.886 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:09.886 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:09.886 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:09.886 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:09.886 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:09.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] but i'm connected to v1:172.21.15.60:6789/0, reconnecting 2024-03-27T21:42:09.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:09.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 1 -- 172.21.15.60:0/4039142858 >> v1:172.21.15.60:6789/0 conn(0x7f4fbc071a50 legacy=0x7f4fbc13e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:09.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:42:09.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 1 --2- 172.21.15.60:0/4039142858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fa4003970 0x7f4fa4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:09.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 10 monclient(hunting): picked mon.a con 0x7f4fa4003970 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:09.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 1 --2- 172.21.15.60:0/4039142858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa40047c0 0x7f4fa4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:09.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 10 monclient(hunting): picked mon.b con 0x7f4fa40047c0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:09.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 1 --2- 172.21.15.60:0/4039142858 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fa4005180 0x7f4fa400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:09.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 10 monclient(hunting): picked mon.c con 0x7f4fa4005180 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:09.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:09.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 1 -- 172.21.15.60:0/4039142858 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc071e30 con 0x7f4fa4003970 2024-03-27T21:42:09.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:09.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 1 -- 172.21.15.60:0/4039142858 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa400c020 con 0x7f4fa40047c0 2024-03-27T21:42:09.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:09.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 1 -- 172.21.15.60:0/4039142858 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa400c1e0 con 0x7f4fa4005180 2024-03-27T21:42:09.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:09.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fbbfff640 10 monclient(hunting): _finish_auth 0 2024-03-27T21:42:09.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc10d3640 1 --2- 172.21.15.60:0/4039142858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fa4003970 0x7f4fa4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:09.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc10d3640 10 monclient(hunting): get_auth_request con 0x7f4fa4003970 auth_method 0 2024-03-27T21:42:09.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc10d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:09.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc10d3640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:09.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc10d3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:09.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc18d4640 1 --2- 172.21.15.60:0/4039142858 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fa4005180 0x7f4fa400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:09.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc10d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:09.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc10d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:09.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc10d3640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:42:09.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc18d4640 10 monclient(hunting): get_auth_request con 0x7f4fa4005180 auth_method 0 2024-03-27T21:42:09.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc18d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:09.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc18d4640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:09.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc18d4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:09.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc08d2640 1 --2- 172.21.15.60:0/4039142858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa40047c0 0x7f4fa4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:09.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc08d2640 10 monclient(hunting): get_auth_request con 0x7f4fa40047c0 auth_method 0 2024-03-27T21:42:09.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc08d2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:09.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc08d2640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:09.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc08d2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:09.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc18d4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:09.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc18d4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:09.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc18d4640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:42:09.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc10d3640 10 monclient(hunting): handle_auth_done global_id 5436 payload 293 2024-03-27T21:42:09.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc10d3640 1 -- 172.21.15.60:0/4039142858 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fa4005180 msgr2=0x7f4fa400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:09.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc10d3640 1 --2- 172.21.15.60:0/4039142858 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fa4005180 0x7f4fa400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc10d3640 1 -- 172.21.15.60:0/4039142858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa40047c0 msgr2=0x7f4fa4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:09.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc10d3640 1 --2- 172.21.15.60:0/4039142858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa40047c0 0x7f4fa4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc10d3640 10 monclient: _finish_hunting 0 2024-03-27T21:42:09.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc10d3640 1 monclient: found mon.a 2024-03-27T21:42:09.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc10d3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:09.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc10d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:09.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc10d3640 1 -- 172.21.15.60:0/4039142858 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa4003520 con 0x7f4fa4003970 2024-03-27T21:42:09.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc10d3640 10 monclient: _finish_auth 0 2024-03-27T21:42:09.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc10d3640 10 monclient: _check_auth_tickets 2024-03-27T21:42:09.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc18d4640 1 -- 172.21.15.60:0/4039142858 reap_dead start 2024-03-27T21:42:09.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc10d3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:09.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc3337640 5 monclient: authenticate success, global_id 5436 2024-03-27T21:42:09.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc3337640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:09.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.882+0000 7f4fc08d2640 1 --2- 172.21.15.60:0/4039142858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa40047c0 0x7f4fa4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:42:09.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.883+0000 7f4fc10d3640 1 --2- 172.21.15.60:0/4039142858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fa4003970 0x7f4fa4003d90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4fac00a830 tx=0x7f4fa40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccc6eb827c073dc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:09.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.883+0000 7f4fbbfff640 1 -- 172.21.15.60:0/4039142858 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fac0072d0 con 0x7f4fa4003970 2024-03-27T21:42:09.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.883+0000 7f4fbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:09.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.883+0000 7f4fbbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:09.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.883+0000 7f4fbbfff640 10 monclient: dump: 2024-03-27T21:42:09.897 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:09.897 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:09.898 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.898 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.898 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:09.898 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:09.899 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:09.899 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:09.899 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:09.899 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:09.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.883+0000 7f4fc3337640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:09.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.883+0000 7f4fbbfff640 1 -- 172.21.15.60:0/4039142858 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fac02f070 con 0x7f4fa4003970 2024-03-27T21:42:09.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.883+0000 7f4fbbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:09.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.883+0000 7f4fc3337640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.883+0000 7f4fc3337640 4 set_mon_vals no callback set 2024-03-27T21:42:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.883+0000 7f4fbbfff640 1 -- 172.21.15.60:0/4039142858 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fac007690 con 0x7f4fa4003970 2024-03-27T21:42:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.883+0000 7f4fbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.883+0000 7f4fbbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:09.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.883+0000 7f4fbbfff640 10 monclient: dump: 2024-03-27T21:42:09.902 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:09.902 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:09.902 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.902 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.903 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:09.903 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:09.903 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:09.903 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:09.904 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:09.904 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:09.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.884+0000 7f4fc3337640 10 monclient: shutdown 2024-03-27T21:42:09.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.884+0000 7f4fc3337640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:09.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.884+0000 7f4fc3337640 1 -- 172.21.15.60:0/4039142858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fa4003970 msgr2=0x7f4fa4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:09.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.884+0000 7f4fc3337640 1 --2- 172.21.15.60:0/4039142858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fa4003970 0x7f4fa4003d90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4fac00a830 tx=0x7f4fa40057f0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.884+0000 7f4fc3337640 1 -- 172.21.15.60:0/4039142858 shutdown_connections 2024-03-27T21:42:09.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.884+0000 7f4fc3337640 1 --2- 172.21.15.60:0/4039142858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fa4003970 0x7f4fa4003d90 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.884+0000 7f4fc3337640 1 -- 172.21.15.60:0/4039142858 >> 172.21.15.60:0/4039142858 conn(0x7f4fbc0b7300 msgr2=0x7f4fbc0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:09.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.884+0000 7f4fc3337640 1 -- 172.21.15.60:0/4039142858 shutdown_connections 2024-03-27T21:42:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.884+0000 7f4fc3337640 1 -- 172.21.15.60:0/4039142858 wait complete. 2024-03-27T21:42:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.884+0000 7f4fc3337640 10 monclient: build_initial_monmap 2024-03-27T21:42:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.884+0000 7f4fc3337640 10 monclient: monmap: 2024-03-27T21:42:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:09.907 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:09.907 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:09.907 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:09.907 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:09.907 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:09.907 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:09.907 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:09.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 1 librados: starting msgr at 2024-03-27T21:42:09.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 1 librados: starting objecter 2024-03-27T21:42:09.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 1 Processor -- start 2024-03-27T21:42:09.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 1 -- start start 2024-03-27T21:42:09.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 1 librados: setting wanted keys 2024-03-27T21:42:09.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 1 librados: calling monclient init 2024-03-27T21:42:09.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 10 monclient: init 2024-03-27T21:42:09.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:09.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:42:09.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa4003970 0x7f4fbc0aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:09.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 10 monclient(hunting): picked mon.noname-b con 0x7f4fa4003970 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:09.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fbc0b0ac0 0x7f4fbc1d6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:09.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 10 monclient(hunting): picked mon.noname-a con 0x7f4fbc0b0ac0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:09.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fbc0af660 0x7f4fbc1d6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:09.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 10 monclient(hunting): picked mon.noname-c con 0x7f4fbc0af660 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:09.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:09.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc1d64d0 con 0x7f4fbc0b0ac0 2024-03-27T21:42:09.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:09.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc0accb0 con 0x7f4fa4003970 2024-03-27T21:42:09.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:09.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc1d73e0 con 0x7f4fbc0af660 2024-03-27T21:42:09.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:09.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc3337640 10 monclient(hunting): authenticate will time out at 971.106679s 2024-03-27T21:42:09.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc18d4640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fbc0af660 0x7f4fbc1d6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:09.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc08d2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fbc0b0ac0 0x7f4fbc1d6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:09.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc18d4640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fbc0af660 0x7f4fbc1d6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34512/0 (socket says 172.21.15.60:34512) 2024-03-27T21:42:09.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc08d2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fbc0b0ac0 0x7f4fbc1d6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39688/0 (socket says 172.21.15.60:39688) 2024-03-27T21:42:09.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc18d4640 1 -- 172.21.15.60:0/2313323965 learned_addr learned my addr 172.21.15.60:0/2313323965 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:09.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc18d4640 10 monclient(hunting): get_auth_request con 0x7f4fbc0af660 auth_method 0 2024-03-27T21:42:09.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc18d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:09.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc18d4640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:09.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc18d4640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:09.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc08d2640 10 monclient(hunting): get_auth_request con 0x7f4fbc0b0ac0 auth_method 0 2024-03-27T21:42:09.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc08d2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:09.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc08d2640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:09.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc08d2640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:09.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc18d4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:09.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc18d4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:09.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc18d4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:09.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc08d2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:09.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.885+0000 7f4fc08d2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:09.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc08d2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:09.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc10d3640 1 --2- 172.21.15.60:0/2313323965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa4003970 0x7f4fbc0aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:09.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc10d3640 10 monclient(hunting): get_auth_request con 0x7f4fa4003970 auth_method 0 2024-03-27T21:42:09.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc10d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:09.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc10d3640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:09.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc10d3640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:09.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc18d4640 10 monclient(hunting): handle_auth_done global_id 5123 payload 824 2024-03-27T21:42:09.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc18d4640 1 -- 172.21.15.60:0/2313323965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa4003970 msgr2=0x7f4fbc0aff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:09.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc18d4640 1 --2- 172.21.15.60:0/2313323965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa4003970 0x7f4fbc0aff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc18d4640 1 -- 172.21.15.60:0/2313323965 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fbc0b0ac0 msgr2=0x7f4fbc1d6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:09.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc18d4640 1 --2- 172.21.15.60:0/2313323965 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fbc0b0ac0 0x7f4fbc1d6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:09.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc18d4640 10 monclient: _finish_hunting 0 2024-03-27T21:42:09.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc18d4640 1 monclient: found mon.noname-c 2024-03-27T21:42:09.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc18d4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:09.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc18d4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:09.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc18d4640 1 -- 172.21.15.60:0/2313323965 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fbc0aea90 con 0x7f4fbc0af660 2024-03-27T21:42:09.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc08d2640 1 --2- 172.21.15.60:0/2313323965 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fbc0b0ac0 0x7f4fbc1d6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:09.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc18d4640 1 --2- 172.21.15.60:0/2313323965 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fbc0af660 0x7f4fbc1d6ea0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc0a91e0 tx=0x7f4fb4011cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:09.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fb9ffb640 10 client.?.objecter ms_handle_connect 0x7f4fbc0af660 2024-03-27T21:42:09.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:09.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fb9ffb640 1 -- 172.21.15.60:0/2313323965 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4004030 con 0x7f4fbc0af660 2024-03-27T21:42:09.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:09.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fb9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:09.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fb9ffb640 10 monclient: dump: 2024-03-27T21:42:09.919 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:09.919 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:09.919 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.919 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.920 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:09.920 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:09.920 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:09.920 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:09.920 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:09.920 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:09.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fb9ffb640 10 monclient: _finish_auth 0 2024-03-27T21:42:09.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fb9ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:42:09.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:09.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc3337640 5 monclient: authenticate success, global_id 5123 2024-03-27T21:42:09.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fb9ffb640 1 -- 172.21.15.60:0/2313323965 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb4004190 con 0x7f4fbc0af660 2024-03-27T21:42:09.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc3337640 10 monclient: _renew_subs 2024-03-27T21:42:09.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc3337640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:09.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc3337640 1 -- 172.21.15.60:0/2313323965 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fbc1390c0 con 0x7f4fbc0af660 2024-03-27T21:42:09.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fb9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:09.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fb9ffb640 1 -- 172.21.15.60:0/2313323965 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4004520 con 0x7f4fbc0af660 2024-03-27T21:42:09.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:09.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fb9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:09.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fb9ffb640 10 monclient: dump: 2024-03-27T21:42:09.922 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:09.922 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:09.922 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.922 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:09.922 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:09.922 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:09.923 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:09.923 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:09.923 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:09.923 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:09.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc3337640 10 client.5123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:09.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc3337640 10 monclient: _renew_subs 2024-03-27T21:42:09.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc3337640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:09.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc3337640 1 -- 172.21.15.60:0/2313323965 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fbc1395f0 con 0x7f4fbc0af660 2024-03-27T21:42:09.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fbb7fe640 4 set_mon_vals no callback set 2024-03-27T21:42:09.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.886+0000 7f4fc3337640 1 librados: init done 2024-03-27T21:42:09.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.887+0000 7f4fc3337640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:09.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.887+0000 7f4fc3337640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:09.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.887+0000 7f4fc3337640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:09.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.887+0000 7f4fc3337640 1 -- 172.21.15.60:0/2313323965 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fbc11f650 con 0x7f4fbc0af660 2024-03-27T21:42:09.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.888+0000 7f4fb9ffb640 1 -- 172.21.15.60:0/2313323965 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4fb400f710 con 0x7f4fbc0af660 2024-03-27T21:42:09.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.888+0000 7f4fb9ffb640 1 --2- 172.21.15.60:0/2313323965 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4fa806c590 0x7f4fa806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:09.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.888+0000 7f4fb9ffb640 1 -- 172.21.15.60:0/2313323965 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f4fb4089230 con 0x7f4fbc0af660 2024-03-27T21:42:09.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.888+0000 7f4fb9ffb640 10 client.5123.objecter ms_dispatch 0x7f4fbc000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:09.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.888+0000 7f4fb9ffb640 3 client.5123.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:09.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.888+0000 7f4fb9ffb640 3 client.5123.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:09.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.888+0000 7f4fc10d3640 1 --2- 172.21.15.60:0/2313323965 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4fa806c590 0x7f4fa806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:09.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.888+0000 7f4fc10d3640 10 monclient: get_auth_request con 0x7f4fa806c590 auth_method 0 2024-03-27T21:42:09.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.888+0000 7f4fb9ffb640 20 client.5123.objecter dump_active .. 0 homeless 2024-03-27T21:42:09.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.888+0000 7f4fb9ffb640 10 client.5123.objecter ms_handle_connect 0x7f4fa806c590 2024-03-27T21:42:09.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.888+0000 7f4fc10d3640 1 --2- 172.21.15.60:0/2313323965 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4fa806c590 0x7f4fa806ea50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4fac034000 tx=0x7f4fac007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:09.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.892+0000 7f4fb9ffb640 1 -- 172.21.15.60:0/2313323965 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4fb40545e0 con 0x7f4fbc0af660 2024-03-27T21:42:09.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.892+0000 7f4fb9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:09.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:09.892+0000 7f4fb9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:10.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.072+0000 7f4fc3337640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:42:10.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.072+0000 7f4fc3337640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:42:10.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.072+0000 7f4fc3337640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:10.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.072+0000 7f4fc3337640 1 -- 172.21.15.60:0/2313323965 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4fbc067900 con 0x7f4fbc0af660 2024-03-27T21:42:10.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.073+0000 7f4fb9ffb640 1 -- 172.21.15.60:0/2313323965 <== mon.2 v2:172.21.15.60:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+14175 (secure 0 0 0) 0x7f4fb4057c00 con 0x7f4fbc0af660 2024-03-27T21:42:10.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.073+0000 7f4fb9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-27T21:42:10.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.073+0000 7f4fb9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:42:10.062 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:42:10.062 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":92,"fsid":"dcd19d9a-535c-4b41-9095-d588268e2860","created":"2024-03-27T21:39:27.092191+0000","modified":"2024-03-27T21:42:04.274002+0000","last_up_change":"2024-03-27T21:41:29.048228+0000","last_in_change":"2024-03-27T21:41:48.185818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T21:39:37.438493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-27T21:39:45.042103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"778aaa64-6720-495b-bcdc-86fbcece9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":34603},{"type":"v1","addr":"172.21.15.60:6817","nonce":34603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":34603},{"type":"v1","addr":"172.21.15.60:6819","nonce":34603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":34603},{"type":"v1","addr":"172.21.15.60:6823","nonce":34603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":34603},{"type":"v1","addr":"172.21.15.60:6821","nonce":34603}]},"public_addr":"172.21.15.60:6817/34603","cluster_addr":"172.21.15.60:6819/34603","heartbeat_back_addr":"172.21.15.60:6823/34603","heartbeat_front_addr":"172.21.15.60:6821/34603","state":["exists","up"]},{"osd":1,"uuid":"d960adf1-3d83-4e24-a612-8df53fa43d0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":34602},{"type":"v1","addr":"172.21.15.60:6801","nonce":34602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":34602},{"type":"v1","addr":"172.21.15.60:6803","nonce":34602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":34602},{"type":"v1","addr":"172.21.15.60:6807","nonce":34602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":34602},{"type":"v1","addr":"172.21.15.60:6805","nonce":34602}]},"public_addr":"172.21.15.60:6801/34602","cluster_addr":"172.21.15.60:6803/34602","heartbeat_back_addr":"172.21.15.60:6807/34602","heartbeat_front_addr":"172.21.15.60:6805/34602","state":["exists","up"]},{"osd":2,"uuid":"b8ffc48b-cf6f-4701-a6aa-c25fdb0f7977","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":33,"last_clean_end":44,"up_from":62,"up_thru":62,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":56434},{"type":"v1","addr":"172.21.15.60:6825","nonce":56434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":56434},{"type":"v1","addr":"172.21.15.60:6827","nonce":56434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":56434},{"type":"v1","addr":"172.21.15.60:6831","nonce":56434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":56434},{"type":"v1","addr":"172.21.15.60:6829","nonce":56434}]},"public_addr":"172.21.15.60:6825/56434","cluster_addr":"172.21.15.60:6827/56434","heartbeat_back_addr":"172.21.15.60:6831/56434","heartbeat_front_addr":"172.21.15.60:6829/56434","state":["exists","up"]},{"osd":3,"uuid":"0f14f2d8-b271-48e4-a198-aa5279a01c06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":34613},{"type":"v1","addr":"172.21.15.60:6809","nonce":34613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":34613},{"type":"v1","addr":"172.21.15.60:6811","nonce":34613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":34613},{"type":"v1","addr":"172.21.15.60:6815","nonce":34613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":34613},{"type":"v1","addr":"172.21.15.60:6813","nonce":34613}]},"public_addr":"172.21.15.60:6809/34613","cluster_addr":"172.21.15.60:6811/34613","heartbeat_back_addr":"172.21.15.60:6815/34613","heartbeat_front_addr":"172.21.15.60:6813/34613","state":["exists","up"]},{"osd":4,"uuid":"8cdeb7ae-1c6e-4d0d-ba44-80248eff2fb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":34332},{"type":"v1","addr":"172.21.15.191:6801","nonce":34332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":34332},{"type":"v1","addr":"172.21.15.191:6803","nonce":34332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":34332},{"type":"v1","addr":"172.21.15.191:6807","nonce":34332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":34332},{"type":"v1","addr":"172.21.15.191:6805","nonce":34332}]},"public_addr":"172.21.15.191:6801/34332","cluster_addr":"172.21.15.191:6803/34332","heartbeat_back_addr":"172.21.15.191:6807/34332","heartbeat_front_addr":"172.21.15.191:6805/34332","state":["exists","up"]},{"osd":5,"uuid":"93748cf8-d3fe-46b8-950f-e01e28a5e02c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6809","nonce":34333},{"type":"v1","addr":"172.21.15.191:6811","nonce":34333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6813","nonce":34333},{"type":"v1","addr":"172.21.15.191:6816","nonce":34333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":34333},{"type":"v1","addr":"172.21.15.191:6823","nonce":34333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":34333},{"type":"v1","addr":"172.21.15.191:6820","nonce":34333}]},"public_addr":"172.21.15.191:6811/34333","cluster_addr":"172.21.15.191:6816/34333","heartbeat_back_addr":"172.21.15.191:6823/34333","heartbeat_front_addr":"172.21.15.191:6820/34333","state":["exists","up"]},{"osd":6,"uuid":"1ad21434-cbc5-4de9-8d63-76f894f7e736","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":34361},{"type":"v1","addr":"172.21.15.191:6825","nonce":34361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":34361},{"type":"v1","addr":"172.21.15.191:6827","nonce":34361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":34361},{"type":"v1","addr":"172.21.15.191:6831","nonce":34361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":34361},{"type":"v1","addr":"172.21.15.191:6829","nonce":34361}]},"public_addr":"172.21.15.191:6825/34361","cluster_addr":"172.21.15.191:6827/34361","heartbeat_back_addr":"172.21.15.191:6831/34361","heartbeat_front_addr":"172.21.15.191:6829/34361","state":["exists","up"]},{"osd":7,"uuid":"9fbd0927-ee2f-471b-8df8-c4e0fc40b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":34334},{"type":"v1","addr":"172.21.15.191:6810","nonce":34334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":34334},{"type":"v1","addr":"172.21.15.191:6814","nonce":34334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6819","nonce":34334},{"type":"v1","addr":"172.21.15.191:6821","nonce":34334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6815","nonce":34334},{"type":"v1","addr":"172.21.15.191:6817","nonce":34334}]},"public_addr":"172.21.15.191:6810/34334","cluster_addr":"172.21.15.191:6814/34334","heartbeat_back_addr":"172.21.15.191:6821/34334","heartbeat_front_addr":"172.21.15.191:6817/34334","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.476097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.411243+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.458154+0000","dead_epoch":45},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.415285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.431201+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.456013+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.474448+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-27T21:39:34.414076+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T21:42:10.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.075+0000 7f4fc3337640 10 librados: watch_flush enter 2024-03-27T21:42:10.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.075+0000 7f4fc3337640 10 librados: watch_flush exit 2024-03-27T21:42:10.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.075+0000 7f4fc3337640 20 client.5123.objecter shutdown clearing up homeless session... 2024-03-27T21:42:10.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.075+0000 7f4fc3337640 10 client.5123.objecter successfully canceled tick 2024-03-27T21:42:10.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.075+0000 7f4fc3337640 1 -- 172.21.15.60:0/2313323965 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4fa806c590 msgr2=0x7f4fa806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:10.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.075+0000 7f4fc3337640 1 --2- 172.21.15.60:0/2313323965 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4fa806c590 0x7f4fa806ea50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4fac034000 tx=0x7f4fac007000 comp rx=0 tx=0).stop 2024-03-27T21:42:10.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.075+0000 7f4fc3337640 10 monclient: shutdown 2024-03-27T21:42:10.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.075+0000 7f4fc3337640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:10.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.075+0000 7f4fc3337640 1 -- 172.21.15.60:0/2313323965 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fbc0af660 msgr2=0x7f4fbc1d6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:10.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.075+0000 7f4fc3337640 1 --2- 172.21.15.60:0/2313323965 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fbc0af660 0x7f4fbc1d6ea0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc0a91e0 tx=0x7f4fb4011cf0 comp rx=0 tx=0).stop 2024-03-27T21:42:10.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.075+0000 7f4fc3337640 1 -- 172.21.15.60:0/2313323965 shutdown_connections 2024-03-27T21:42:10.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.075+0000 7f4fc3337640 1 --2- 172.21.15.60:0/2313323965 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4fbc0af660 0x7f4fbc1d6ea0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:10.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.075+0000 7f4fc3337640 1 --2- 172.21.15.60:0/2313323965 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fbc0b0ac0 0x7f4fbc1d6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:10.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.075+0000 7f4fc3337640 1 --2- 172.21.15.60:0/2313323965 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4fa806c590 0x7f4fa806ea50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:10.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.075+0000 7f4fc3337640 1 --2- 172.21.15.60:0/2313323965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa4003970 0x7f4fbc0aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:10.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.075+0000 7f4fc3337640 1 -- 172.21.15.60:0/2313323965 >> 172.21.15.60:0/2313323965 conn(0x7f4fbc0b7300 msgr2=0x7f4fbc0b17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:10.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.075+0000 7f4fc3337640 1 -- 172.21.15.60:0/2313323965 shutdown_connections 2024-03-27T21:42:10.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.075+0000 7f4fc3337640 1 -- 172.21.15.60:0/2313323965 wait complete. 2024-03-27T21:42:10.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:10.075+0000 7f4fc3337640 1 librados: shutdown 2024-03-27T21:42:11.074 INFO:tasks.ceph:Scrubbing osd.0 2024-03-27T21:42:11.074 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-27T21:42:11.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.204+0000 7fbaa5b37640 10 monclient: get_monmap_and_config 2024-03-27T21:42:11.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.204+0000 7fbaa5b37640 10 monclient: build_initial_monmap 2024-03-27T21:42:11.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.204+0000 7fbaa5b37640 10 monclient: monmap: 2024-03-27T21:42:11.191 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:11.191 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:11.191 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:11.205540+0000 2024-03-27T21:42:11.191 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:11.205540+0000 2024-03-27T21:42:11.191 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:11.191 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:11.192 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:11.192 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:11.192 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:11.192 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:11.192 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:11.192 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.205+0000 7fbaa5b37640 1 Processor -- start 2024-03-27T21:42:11.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.205+0000 7fbaa5b37640 1 -- start start 2024-03-27T21:42:11.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.205+0000 7fbaa5b37640 10 monclient: init 2024-03-27T21:42:11.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.205+0000 7fbaa5b37640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:11.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.205+0000 7fbaa5b37640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-03-27T21:42:11.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.205+0000 7fbaa5b37640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbaa00b3980 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:11.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.205+0000 7fbaa5b37640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbaa00b3d60 0x7fbaa01449c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.205+0000 7fbaa5b37640 10 monclient(hunting): picked mon.noname-c con 0x7fbaa00b3d60 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:11.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.205+0000 7fbaa5b37640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbaa00a86e0 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:11.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.205+0000 7fbaa5b37640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa0144f00 con 0x7fbaa00b3980 2024-03-27T21:42:11.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.205+0000 7fbaa5b37640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa00b07f0 con 0x7fbaa00a86e0 2024-03-27T21:42:11.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.205+0000 7fbaa5b37640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:11.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.205+0000 7fbaa5b37640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaa00aa1d0 con 0x7fbaa00b3d60 2024-03-27T21:42:11.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.205+0000 7fbaa5b37640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:11.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.205+0000 7fbaa5b37640 10 monclient(hunting): authenticate will time out at 972.426897s 2024-03-27T21:42:11.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9f7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbaa00b3980 0x7fbaa00b0c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39514/0 (socket says 172.21.15.60:39514) 2024-03-27T21:42:11.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9f7fe640 1 -- 172.21.15.60:0/655871904 learned_addr learned my addr 172.21.15.60:0/655871904 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:11.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9effd640 1 --2- 172.21.15.60:0/655871904 >> v2:172.21.15.191:3300/0 conn(0x7fbaa00b3d60 0x7fbaa01449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 1 -- 172.21.15.60:0/655871904 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141554492 0 0) 0x7fbaa0144f00 con 0x7fbaa00b3980 2024-03-27T21:42:11.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 10 monclient(hunting): _negotiate 2024-03-27T21:42:11.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:11.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:11.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 10 monclient(hunting): my global_id is 5442 2024-03-27T21:42:11.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 1 -- 172.21.15.60:0/655871904 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba88003520 con 0x7fbaa00b3980 2024-03-27T21:42:11.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9effd640 10 monclient(hunting): get_auth_request con 0x7fbaa00b3d60 auth_method 0 2024-03-27T21:42:11.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:11.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:11.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:11.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:11.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:11.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:11.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 1 -- 172.21.15.60:0/655871904 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 615238248 0 0) 0x7fba88003520 con 0x7fbaa00b3980 2024-03-27T21:42:11.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 1 -- 172.21.15.60:0/655871904 >> v2:172.21.15.191:3300/0 conn(0x7fbaa00b3d60 msgr2=0x7fbaa01449c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:11.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 1 --2- 172.21.15.60:0/655871904 >> v2:172.21.15.191:3300/0 conn(0x7fbaa00b3d60 0x7fbaa01449c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 1 -- 172.21.15.60:0/655871904 >> v1:172.21.15.191:6789/0 conn(0x7fbaa00a86e0 legacy=0x7fbaa00b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 10 monclient: _finish_hunting 0 2024-03-27T21:42:11.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 1 monclient: found mon.noname-a-legacy 2024-03-27T21:42:11.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:11.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.60:6789/0 2024-03-27T21:42:11.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 1 -- 172.21.15.60:0/655871904 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa00aa4b0 con 0x7fbaa00b3980 2024-03-27T21:42:11.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 10 monclient: _finish_auth 0 2024-03-27T21:42:11.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:42:11.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:11.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 1 -- 172.21.15.60:0/655871904 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2933500084 0 0) 0x7fba94001670 con 0x7fbaa00b3980 2024-03-27T21:42:11.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fbaa5b37640 5 monclient: authenticate success, global_id 5442 2024-03-27T21:42:11.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fbaa5b37640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:11.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:11.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-03-27T21:42:11.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 10 monclient: dump: 2024-03-27T21:42:11.208 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:11.208 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:11.208 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.209 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.209 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:11.209 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:11.209 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:11.209 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:11.209 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:11.209 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] but i'm connected to v1:172.21.15.60:6789/0, reconnecting 2024-03-27T21:42:11.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:11.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 1 -- 172.21.15.60:0/655871904 >> v1:172.21.15.60:6789/0 conn(0x7fbaa00b3980 legacy=0x7fbaa00b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:42:11.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 1 --2- 172.21.15.60:0/655871904 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fba88003970 0x7fba88003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 10 monclient(hunting): picked mon.c con 0x7fba88003970 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:11.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 1 --2- 172.21.15.60:0/655871904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba880047c0 0x7fba88004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 10 monclient(hunting): picked mon.a con 0x7fba880047c0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:11.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 1 --2- 172.21.15.60:0/655871904 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba88005180 0x7fba8800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 10 monclient(hunting): picked mon.b con 0x7fba88005180 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:11.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:11.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 1 -- 172.21.15.60:0/655871904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa00aa1d0 con 0x7fba880047c0 2024-03-27T21:42:11.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:11.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 1 -- 172.21.15.60:0/655871904 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8800c020 con 0x7fba88005180 2024-03-27T21:42:11.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:11.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 1 -- 172.21.15.60:0/655871904 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8800c1e0 con 0x7fba88003970 2024-03-27T21:42:11.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:11.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-27T21:42:11.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fbaa5b37640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:11.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9f7fe640 1 --2- 172.21.15.60:0/655871904 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fba88003970 0x7fba88003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9effd640 1 --2- 172.21.15.60:0/655871904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba880047c0 0x7fba88004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9f7fe640 10 monclient(hunting): get_auth_request con 0x7fba88003970 auth_method 0 2024-03-27T21:42:11.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:11.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:11.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:11.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9effd640 10 monclient(hunting): get_auth_request con 0x7fba880047c0 auth_method 0 2024-03-27T21:42:11.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:11.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:11.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:11.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9ffff640 1 --2- 172.21.15.60:0/655871904 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba88005180 0x7fba8800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:11.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:11.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:42:11.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:11.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:11.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:42:11.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9ffff640 10 monclient(hunting): get_auth_request con 0x7fba88005180 auth_method 0 2024-03-27T21:42:11.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:11.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:11.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.206+0000 7fba9ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:11.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.207+0000 7fba9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:11.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.207+0000 7fba9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:11.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.207+0000 7fba9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:42:11.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.207+0000 7fba9effd640 10 monclient(hunting): handle_auth_done global_id 5442 payload 293 2024-03-27T21:42:11.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.207+0000 7fba9effd640 1 -- 172.21.15.60:0/655871904 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fba88003970 msgr2=0x7fba88003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:11.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.207+0000 7fba9effd640 1 --2- 172.21.15.60:0/655871904 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fba88003970 0x7fba88003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.207+0000 7fba9effd640 1 -- 172.21.15.60:0/655871904 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba88005180 msgr2=0x7fba8800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:11.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.207+0000 7fba9effd640 1 --2- 172.21.15.60:0/655871904 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba88005180 0x7fba8800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.207+0000 7fba9effd640 10 monclient: _finish_hunting 0 2024-03-27T21:42:11.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.207+0000 7fba9effd640 1 monclient: found mon.a 2024-03-27T21:42:11.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.207+0000 7fba9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:11.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.207+0000 7fba9effd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:11.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.207+0000 7fba9effd640 1 -- 172.21.15.60:0/655871904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba88003520 con 0x7fba880047c0 2024-03-27T21:42:11.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.207+0000 7fba9effd640 10 monclient: _finish_auth 0 2024-03-27T21:42:11.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.207+0000 7fba9effd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:11.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.207+0000 7fba9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:11.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.207+0000 7fba9ffff640 1 -- 172.21.15.60:0/655871904 reap_dead start 2024-03-27T21:42:11.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.207+0000 7fba9f7fe640 1 --2- 172.21.15.60:0/655871904 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fba88003970 0x7fba88003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:11.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.207+0000 7fba9effd640 1 --2- 172.21.15.60:0/655871904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba880047c0 0x7fba88004c40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fba90009a90 tx=0x7fba880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8464aa63c88986f server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:11.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.207+0000 7fba9e7fc640 1 -- 172.21.15.60:0/655871904 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9002c680 con 0x7fba880047c0 2024-03-27T21:42:11.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.207+0000 7fba9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:11.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.208+0000 7fba9e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:11.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.208+0000 7fba9e7fc640 10 monclient: dump: 2024-03-27T21:42:11.217 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:11.217 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:11.217 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.217 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.217 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:11.217 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:11.217 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:11.218 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:11.218 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:11.218 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.208+0000 7fba9e7fc640 1 -- 172.21.15.60:0/655871904 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba9002c7e0 con 0x7fba880047c0 2024-03-27T21:42:11.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.208+0000 7fba9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:11.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.208+0000 7fba9e7fc640 1 -- 172.21.15.60:0/655871904 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9002cb90 con 0x7fba880047c0 2024-03-27T21:42:11.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.208+0000 7fbaa5b37640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:11.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.208+0000 7fbaa5b37640 4 set_mon_vals no callback set 2024-03-27T21:42:11.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.208+0000 7fba9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:11.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.208+0000 7fba9e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:11.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.208+0000 7fba9e7fc640 10 monclient: dump: 2024-03-27T21:42:11.219 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:11.219 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:11.219 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.219 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.219 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:11.219 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:11.219 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:11.220 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:11.220 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:11.220 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.208+0000 7fbaa5b37640 10 monclient: shutdown 2024-03-27T21:42:11.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.208+0000 7fbaa5b37640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:11.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.208+0000 7fbaa5b37640 1 -- 172.21.15.60:0/655871904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba880047c0 msgr2=0x7fba88004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.208+0000 7fbaa5b37640 1 --2- 172.21.15.60:0/655871904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba880047c0 0x7fba88004c40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fba90009a90 tx=0x7fba880057f0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.208+0000 7fbaa5b37640 1 -- 172.21.15.60:0/655871904 shutdown_connections 2024-03-27T21:42:11.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.208+0000 7fbaa5b37640 1 --2- 172.21.15.60:0/655871904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba880047c0 0x7fba88004c40 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.208+0000 7fbaa5b37640 1 -- 172.21.15.60:0/655871904 >> 172.21.15.60:0/655871904 conn(0x7fbaa0071410 msgr2=0x7fbaa0071820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:11.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.208+0000 7fbaa5b37640 1 -- 172.21.15.60:0/655871904 shutdown_connections 2024-03-27T21:42:11.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.208+0000 7fbaa5b37640 1 -- 172.21.15.60:0/655871904 wait complete. 2024-03-27T21:42:11.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.209+0000 7fbaa5b37640 10 monclient: build_initial_monmap 2024-03-27T21:42:11.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.209+0000 7fbaa5b37640 10 monclient: monmap: 2024-03-27T21:42:11.221 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:11.221 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:11.221 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:11.221 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:11.222 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:11.222 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:11.222 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:11.222 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:11.222 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:11.222 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.209+0000 7fbaa5b37640 1 librados: starting msgr at 2024-03-27T21:42:11.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.209+0000 7fbaa5b37640 1 librados: starting objecter 2024-03-27T21:42:11.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.209+0000 7fbaa5b37640 1 Processor -- start 2024-03-27T21:42:11.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.209+0000 7fbaa5b37640 1 -- start start 2024-03-27T21:42:11.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.209+0000 7fbaa5b37640 1 librados: setting wanted keys 2024-03-27T21:42:11.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.209+0000 7fbaa5b37640 1 librados: calling monclient init 2024-03-27T21:42:11.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.209+0000 7fbaa5b37640 10 monclient: init 2024-03-27T21:42:11.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fbaa5b37640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:11.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fbaa5b37640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:42:11.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fbaa5b37640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba880047c0 0x7fbaa00b6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fbaa5b37640 10 monclient(hunting): picked mon.noname-b con 0x7fba880047c0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:11.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fbaa5b37640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbaa00ac730 0x7fbaa0139fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fbaa5b37640 10 monclient(hunting): picked mon.noname-a con 0x7fbaa00ac730 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:11.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fbaa5b37640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbaa00ac1e0 0x7fbaa013a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fbaa5b37640 10 monclient(hunting): picked mon.noname-c con 0x7fbaa00ac1e0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:11.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fbaa5b37640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:11.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fbaa5b37640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa00ad1c0 con 0x7fbaa00ac730 2024-03-27T21:42:11.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fbaa5b37640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:11.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fbaa5b37640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa00b7520 con 0x7fba880047c0 2024-03-27T21:42:11.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fbaa5b37640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:11.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fbaa5b37640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa00acca0 con 0x7fbaa00ac1e0 2024-03-27T21:42:11.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fbaa5b37640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:11.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fbaa5b37640 10 monclient(hunting): authenticate will time out at 972.431403s 2024-03-27T21:42:11.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbaa00ac730 0x7fbaa0139fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9ffff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbaa00ac1e0 0x7fbaa013a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbaa00ac730 0x7fbaa0139fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39712/0 (socket says 172.21.15.60:39712) 2024-03-27T21:42:11.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9ffff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbaa00ac1e0 0x7fbaa013a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34532/0 (socket says 172.21.15.60:34532) 2024-03-27T21:42:11.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9effd640 1 -- 172.21.15.60:0/1732929043 learned_addr learned my addr 172.21.15.60:0/1732929043 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:11.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9effd640 10 monclient(hunting): get_auth_request con 0x7fbaa00ac730 auth_method 0 2024-03-27T21:42:11.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:11.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:11.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:11.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9ffff640 10 monclient(hunting): get_auth_request con 0x7fbaa00ac1e0 auth_method 0 2024-03-27T21:42:11.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:11.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:11.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:11.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:11.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:11.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9f7fe640 1 --2- 172.21.15.60:0/1732929043 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba880047c0 0x7fbaa00b6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:11.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:11.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:11.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:11.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9f7fe640 10 monclient(hunting): get_auth_request con 0x7fba880047c0 auth_method 0 2024-03-27T21:42:11.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:11.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:11.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:11.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:11.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:11.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:11.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9effd640 10 monclient(hunting): handle_auth_done global_id 5445 payload 824 2024-03-27T21:42:11.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9effd640 1 -- 172.21.15.60:0/1732929043 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbaa00ac1e0 msgr2=0x7fbaa013a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9effd640 1 --2- 172.21.15.60:0/1732929043 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbaa00ac1e0 0x7fbaa013a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9effd640 1 -- 172.21.15.60:0/1732929043 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba880047c0 msgr2=0x7fbaa00b6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9effd640 1 --2- 172.21.15.60:0/1732929043 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba880047c0 0x7fbaa00b6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9effd640 10 monclient: _finish_hunting 0 2024-03-27T21:42:11.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9effd640 1 monclient: found mon.noname-a 2024-03-27T21:42:11.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:11.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:11.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9effd640 1 -- 172.21.15.60:0/1732929043 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa01d9ac0 con 0x7fbaa00ac730 2024-03-27T21:42:11.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.210+0000 7fba9ffff640 1 --2- 172.21.15.60:0/1732929043 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fbaa00ac1e0 0x7fbaa013a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:11.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fba9effd640 1 --2- 172.21.15.60:0/1732929043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbaa00ac730 0x7fbaa0139fe0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fba90009bb0 tx=0x7fba90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:11.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fba87fff640 10 client.?.objecter ms_handle_connect 0x7fbaa00ac730 2024-03-27T21:42:11.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fba87fff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:11.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fba87fff640 1 -- 172.21.15.60:0/1732929043 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba90034a10 con 0x7fbaa00ac730 2024-03-27T21:42:11.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fba87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:11.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fba87fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:11.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fba87fff640 10 monclient: dump: 2024-03-27T21:42:11.230 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:11.230 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:11.231 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.231 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.231 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:11.231 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:11.231 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:11.231 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:11.231 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:11.231 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fba87fff640 10 monclient: _finish_auth 0 2024-03-27T21:42:11.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fba87fff640 10 monclient: _check_auth_tickets 2024-03-27T21:42:11.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fba87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:11.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fba87fff640 1 -- 172.21.15.60:0/1732929043 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba90034b90 con 0x7fbaa00ac730 2024-03-27T21:42:11.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fbaa5b37640 5 monclient: authenticate success, global_id 5445 2024-03-27T21:42:11.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fbaa5b37640 10 monclient: _renew_subs 2024-03-27T21:42:11.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fbaa5b37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:11.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fbaa5b37640 1 -- 172.21.15.60:0/1732929043 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa01d9c50 con 0x7fbaa00ac730 2024-03-27T21:42:11.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fba87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:11.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fba87fff640 1 -- 172.21.15.60:0/1732929043 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9003e400 con 0x7fbaa00ac730 2024-03-27T21:42:11.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fba87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:11.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fba87fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:11.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fba87fff640 10 monclient: dump: 2024-03-27T21:42:11.233 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:11.233 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:11.233 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.233 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.233 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:11.234 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:11.234 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:11.234 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:11.234 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:11.234 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fba9dffb640 4 set_mon_vals no callback set 2024-03-27T21:42:11.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fbaa5b37640 10 client.5445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:11.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fbaa5b37640 10 monclient: _renew_subs 2024-03-27T21:42:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fbaa5b37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fbaa5b37640 1 -- 172.21.15.60:0/1732929043 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa01da1b0 con 0x7fbaa00ac730 2024-03-27T21:42:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.211+0000 7fbaa5b37640 1 librados: init done 2024-03-27T21:42:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.212+0000 7fbaa5b37640 10 client.5445.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.212+0000 7fbaa5b37640 15 client.5445.objecter _session_command_op_assign -1 1 2024-03-27T21:42:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.212+0000 7fbaa5b37640 20 client.5445.objecter _get_session osd=-1 returning homeless 2024-03-27T21:42:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.212+0000 7fbaa5b37640 10 client.5445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.212+0000 7fbaa5b37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.212+0000 7fbaa5b37640 1 -- 172.21.15.60:0/1732929043 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbaa00ad1c0 con 0x7fbaa00ac730 2024-03-27T21:42:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.212+0000 7fba87fff640 1 -- 172.21.15.60:0/1732929043 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fba9003e560 con 0x7fbaa00ac730 2024-03-27T21:42:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba87fff640 1 --2- 172.21.15.60:0/1732929043 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fba8c06c590 0x7fba8c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba9f7fe640 1 --2- 172.21.15.60:0/1732929043 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fba8c06c590 0x7fba8c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba87fff640 1 -- 172.21.15.60:0/1732929043 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fba900ab020 con 0x7fbaa00ac730 2024-03-27T21:42:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba87fff640 10 client.5445.objecter ms_dispatch 0x7fbaa0000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba87fff640 3 client.5445.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba87fff640 3 client.5445.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba9f7fe640 10 monclient: get_auth_request con 0x7fba8c06c590 auth_method 0 2024-03-27T21:42:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba87fff640 10 client.5445.objecter checking command 1 2024-03-27T21:42:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba87fff640 1 --2- 172.21.15.60:0/1732929043 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fba8c0729d0 0x7fba8c074df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba87fff640 20 client.5445.objecter _get_session s=0x7fba8c072300 osd=0 3 2024-03-27T21:42:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba87fff640 20 client.5445.objecter put_session s=0x7fba8c072300 osd=0 3 2024-03-27T21:42:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba87fff640 15 client.5445.objecter _session_command_op_remove -1 1 2024-03-27T21:42:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba87fff640 20 client.5445.objecter _get_session s=0x7fba8c072300 osd=0 3 2024-03-27T21:42:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba87fff640 20 client.5445.objecter get_session s=0x7fba8c072300 osd=0 3 2024-03-27T21:42:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba87fff640 15 client.5445.objecter _session_command_op_assign 0 1 2024-03-27T21:42:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba87fff640 20 client.5445.objecter put_session s=0x7fba8c072300 osd=0 4 2024-03-27T21:42:11.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba87fff640 10 client.5445.objecter _send_command 1 2024-03-27T21:42:11.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba87fff640 1 -- 172.21.15.60:0/1732929043 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba8c0754c0 con 0x7fba8c0729d0 2024-03-27T21:42:11.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba87fff640 20 client.5445.objecter dump_active .. 0 homeless 2024-03-27T21:42:11.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba87fff640 1 -- 172.21.15.60:0/1732929043 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba900ad6e0 con 0x7fbaa00ac730 2024-03-27T21:42:11.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba87fff640 10 monclient: handle_get_version_reply finishing 1 version 92 2024-03-27T21:42:11.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba9ffff640 1 --2- 172.21.15.60:0/1732929043 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fba8c0729d0 0x7fba8c074df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba9ffff640 10 monclient: get_auth_request con 0x7fba8c0729d0 auth_method 0 2024-03-27T21:42:11.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba87fff640 10 client.5445.objecter ms_handle_connect 0x7fba8c06c590 2024-03-27T21:42:11.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba9f7fe640 1 --2- 172.21.15.60:0/1732929043 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fba8c06c590 0x7fba8c06ea50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fba940017a0 tx=0x7fba94001470 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:11.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba87fff640 10 client.5445.objecter ms_handle_connect 0x7fba8c0729d0 2024-03-27T21:42:11.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.213+0000 7fba9ffff640 1 --2- 172.21.15.60:0/1732929043 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fba8c0729d0 0x7fba8c074df0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:11.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.214+0000 7fba87fff640 1 -- 172.21.15.60:0/1732929043 <== osd.0 v2:172.21.15.60:6816/34603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fba8c0754c0 con 0x7fba8c0729d0 2024-03-27T21:42:11.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.214+0000 7fba87fff640 10 client.5445.objecter ms_dispatch 0x7fbaa0000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:42:11.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.214+0000 7fba87fff640 10 client.5445.objecter _finish_command 1 = system:0 2024-03-27T21:42:11.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.214+0000 7fba87fff640 20 client.5445.objecter put_session s=0x7fba8c072300 osd=0 4 2024-03-27T21:42:11.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.214+0000 7fba87fff640 15 client.5445.objecter _session_command_op_remove 0 1 2024-03-27T21:42:11.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 10 client.5445.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-27T21:42:11.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 15 client.5445.objecter _session_command_op_assign -1 2 2024-03-27T21:42:11.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 20 client.5445.objecter _get_session s=0x7fba8c072300 osd=0 3 2024-03-27T21:42:11.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 20 client.5445.objecter put_session s=0x7fba8c072300 osd=0 3 2024-03-27T21:42:11.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 20 client.5445.objecter _get_session s=0x7fba8c072300 osd=0 3 2024-03-27T21:42:11.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 15 client.5445.objecter _session_command_op_remove -1 2 2024-03-27T21:42:11.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 20 client.5445.objecter get_session s=0x7fba8c072300 osd=0 3 2024-03-27T21:42:11.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 15 client.5445.objecter _session_command_op_assign 0 2 2024-03-27T21:42:11.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 20 client.5445.objecter put_session s=0x7fba8c072300 osd=0 4 2024-03-27T21:42:11.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 10 client.5445.objecter _send_command 2 2024-03-27T21:42:11.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 1 -- 172.21.15.60:0/1732929043 --> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fbaa00ace60 con 0x7fba8c0729d0 2024-03-27T21:42:11.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fba87fff640 1 -- 172.21.15.60:0/1732929043 <== osd.0 v2:172.21.15.60:6816/34603 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fbaa00ace60 con 0x7fba8c0729d0 2024-03-27T21:42:11.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fba87fff640 10 client.5445.objecter ms_dispatch 0x7fbaa0000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:42:11.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fba87fff640 10 client.5445.objecter _finish_command 2 = system:0 2024-03-27T21:42:11.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fba87fff640 20 client.5445.objecter put_session s=0x7fba8c072300 osd=0 4 2024-03-27T21:42:11.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fba87fff640 15 client.5445.objecter _session_command_op_remove 0 2 2024-03-27T21:42:11.243 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:42:11.243 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-27T21:42:11.243 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:42:11.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 10 librados: watch_flush enter 2024-03-27T21:42:11.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 10 librados: watch_flush exit 2024-03-27T21:42:11.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 10 client.5445.objecter close_session for osd.0 2024-03-27T21:42:11.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 1 -- 172.21.15.60:0/1732929043 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fba8c0729d0 msgr2=0x7fba8c074df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 1 --2- 172.21.15.60:0/1732929043 >> [v2:172.21.15.60:6816/34603,v1:172.21.15.60:6817/34603] conn(0x7fba8c0729d0 0x7fba8c074df0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 20 client.5445.objecter put_session s=0x7fba8c072300 osd=0 1 2024-03-27T21:42:11.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 20 client.5445.objecter shutdown clearing up homeless session... 2024-03-27T21:42:11.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 10 client.5445.objecter successfully canceled tick 2024-03-27T21:42:11.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 1 -- 172.21.15.60:0/1732929043 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fba8c06c590 msgr2=0x7fba8c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 1 --2- 172.21.15.60:0/1732929043 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fba8c06c590 0x7fba8c06ea50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fba940017a0 tx=0x7fba94001470 comp rx=0 tx=0).stop 2024-03-27T21:42:11.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 10 monclient: shutdown 2024-03-27T21:42:11.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:11.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 1 -- 172.21.15.60:0/1732929043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbaa00ac730 msgr2=0x7fbaa0139fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 1 --2- 172.21.15.60:0/1732929043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbaa00ac730 0x7fbaa0139fe0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fba90009bb0 tx=0x7fba90004730 comp rx=0 tx=0).stop 2024-03-27T21:42:11.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fba9ffff640 1 -- 172.21.15.60:0/1732929043 reap_dead start 2024-03-27T21:42:11.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 1 -- 172.21.15.60:0/1732929043 shutdown_connections 2024-03-27T21:42:11.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 1 -- 172.21.15.60:0/1732929043 >> 172.21.15.60:0/1732929043 conn(0x7fbaa0071410 msgr2=0x7fbaa013f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:11.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 1 -- 172.21.15.60:0/1732929043 shutdown_connections 2024-03-27T21:42:11.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.227+0000 7fbaa5b37640 1 -- 172.21.15.60:0/1732929043 wait complete. 2024-03-27T21:42:11.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.228+0000 7fbaa5b37640 1 librados: shutdown 2024-03-27T21:42:11.247 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-27T21:42:11.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.363+0000 7fd990ff6640 10 monclient: get_monmap_and_config 2024-03-27T21:42:11.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.363+0000 7fd990ff6640 10 monclient: build_initial_monmap 2024-03-27T21:42:11.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.363+0000 7fd990ff6640 10 monclient: monmap: 2024-03-27T21:42:11.349 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:11.350 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:11.350 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:11.364253+0000 2024-03-27T21:42:11.350 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:11.364253+0000 2024-03-27T21:42:11.350 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:11.350 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:11.350 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:11.350 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:11.350 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:11.351 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:11.351 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:11.351 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.363+0000 7fd990ff6640 1 Processor -- start 2024-03-27T21:42:11.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.363+0000 7fd990ff6640 1 -- start start 2024-03-27T21:42:11.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.363+0000 7fd990ff6640 10 monclient: init 2024-03-27T21:42:11.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd990ff6640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:11.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd990ff6640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-03-27T21:42:11.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd990ff6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd98c0b40b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:11.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd990ff6640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd98c13b020 0x7fd98c13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd990ff6640 10 monclient(hunting): picked mon.noname-a con 0x7fd98c13b020 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:11.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd990ff6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd98c1416f0 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:11.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd990ff6640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd98c0b3830 con 0x7fd98c0b40b0 2024-03-27T21:42:11.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd990ff6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd98c0aa520 con 0x7fd98c1416f0 2024-03-27T21:42:11.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd990ff6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:11.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd990ff6640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd98c0b4490 con 0x7fd98c13b020 2024-03-27T21:42:11.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd990ff6640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:11.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd990ff6640 10 monclient(hunting): authenticate will time out at 972.585590s 2024-03-27T21:42:11.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd989d9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd98c13b020 0x7fd98c13b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd989d9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd98c13b020 0x7fd98c13b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39720/0 (socket says 172.21.15.60:39720) 2024-03-27T21:42:11.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd989d9b640 1 -- 172.21.15.60:0/4098671434 learned_addr learned my addr 172.21.15.60:0/4098671434 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:11.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd989d9b640 10 monclient(hunting): get_auth_request con 0x7fd98c13b020 auth_method 0 2024-03-27T21:42:11.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd989d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:11.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd989d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:11.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd989d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:11.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd989d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:11.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd989d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:11.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.364+0000 7fd989d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:11.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd989d9b640 10 monclient(hunting): handle_auth_done global_id 5448 payload 274 2024-03-27T21:42:11.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd989d9b640 1 -- 172.21.15.60:0/4098671434 >> v1:172.21.15.191:6789/0 conn(0x7fd98c1416f0 legacy=0x7fd98c141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:11.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd989d9b640 1 -- 172.21.15.60:0/4098671434 >> v1:172.21.15.60:6789/0 conn(0x7fd98c0b40b0 legacy=0x7fd98c0a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd989d9b640 10 monclient: _finish_hunting 0 2024-03-27T21:42:11.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd989d9b640 1 monclient: found mon.noname-a 2024-03-27T21:42:11.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd989d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:11.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd989d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:11.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd989d9b640 1 -- 172.21.15.60:0/4098671434 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd98c0a9470 con 0x7fd98c13b020 2024-03-27T21:42:11.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd989d9b640 1 --2- 172.21.15.60:0/4098671434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd98c13b020 0x7fd98c13b480 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd980009870 tx=0x7fd98002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3761b2450077930d server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:11.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd98959a640 1 -- 172.21.15.60:0/4098671434 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd980030860 con 0x7fd98c13b020 2024-03-27T21:42:11.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd98959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:11.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd98959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:11.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd98959a640 10 monclient: dump: 2024-03-27T21:42:11.363 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:11.363 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:11.363 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.364 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.364 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:11.364 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:11.364 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:11.364 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:11.364 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:11.364 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd98959a640 10 monclient: _finish_auth 0 2024-03-27T21:42:11.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd98959a640 10 monclient: _check_auth_tickets 2024-03-27T21:42:11.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd98959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:11.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd98959a640 1 -- 172.21.15.60:0/4098671434 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9800309c0 con 0x7fd98c13b020 2024-03-27T21:42:11.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd990ff6640 5 monclient: authenticate success, global_id 5448 2024-03-27T21:42:11.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd990ff6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:11.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd98959a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:11.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd98959a640 1 -- 172.21.15.60:0/4098671434 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd980030d50 con 0x7fd98c13b020 2024-03-27T21:42:11.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd990ff6640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:11.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd990ff6640 4 set_mon_vals no callback set 2024-03-27T21:42:11.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd98959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:11.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd98959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:11.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.365+0000 7fd98959a640 10 monclient: dump: 2024-03-27T21:42:11.366 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:11.366 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:11.366 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.366 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.366 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:11.366 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:11.367 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:11.367 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:11.367 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:11.367 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.366+0000 7fd990ff6640 10 monclient: shutdown 2024-03-27T21:42:11.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.366+0000 7fd990ff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:11.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.366+0000 7fd990ff6640 1 -- 172.21.15.60:0/4098671434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd98c13b020 msgr2=0x7fd98c13b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.366+0000 7fd990ff6640 1 --2- 172.21.15.60:0/4098671434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd98c13b020 0x7fd98c13b480 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd980009870 tx=0x7fd98002b9a0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.366+0000 7fd990ff6640 1 -- 172.21.15.60:0/4098671434 shutdown_connections 2024-03-27T21:42:11.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.366+0000 7fd990ff6640 1 --2- 172.21.15.60:0/4098671434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd98c13b020 0x7fd98c13b480 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.366+0000 7fd990ff6640 1 -- 172.21.15.60:0/4098671434 >> 172.21.15.60:0/4098671434 conn(0x7fd98c0608e0 msgr2=0x7fd98c060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:11.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.366+0000 7fd990ff6640 1 -- 172.21.15.60:0/4098671434 shutdown_connections 2024-03-27T21:42:11.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.366+0000 7fd990ff6640 1 -- 172.21.15.60:0/4098671434 wait complete. 2024-03-27T21:42:11.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.367+0000 7fd990ff6640 10 monclient: build_initial_monmap 2024-03-27T21:42:11.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.367+0000 7fd990ff6640 10 monclient: monmap: 2024-03-27T21:42:11.368 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:11.369 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.367+0000 7fd990ff6640 1 librados: starting msgr at 2024-03-27T21:42:11.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.367+0000 7fd990ff6640 1 librados: starting objecter 2024-03-27T21:42:11.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.367+0000 7fd990ff6640 1 Processor -- start 2024-03-27T21:42:11.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.367+0000 7fd990ff6640 1 -- start start 2024-03-27T21:42:11.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.367+0000 7fd990ff6640 1 librados: setting wanted keys 2024-03-27T21:42:11.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.367+0000 7fd990ff6640 1 librados: calling monclient init 2024-03-27T21:42:11.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.367+0000 7fd990ff6640 10 monclient: init 2024-03-27T21:42:11.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd990ff6640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd990ff6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:42:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd990ff6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd98c0b40b0 0x7fd98c13e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd990ff6640 10 monclient(hunting): picked mon.noname-b con 0x7fd98c0b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd990ff6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd98c13b020 0x7fd98c13a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd990ff6640 10 monclient(hunting): picked mon.noname-a con 0x7fd98c13b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd990ff6640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd98c1416f0 0x7fd98c0aba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd990ff6640 10 monclient(hunting): picked mon.noname-c con 0x7fd98c1416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd990ff6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd990ff6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98c138a60 con 0x7fd98c13b020 2024-03-27T21:42:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd990ff6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd990ff6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98c0ab310 con 0x7fd98c0b40b0 2024-03-27T21:42:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd990ff6640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd990ff6640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd98c0ab5f0 con 0x7fd98c1416f0 2024-03-27T21:42:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd990ff6640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd990ff6640 10 monclient(hunting): authenticate will time out at 972.589263s 2024-03-27T21:42:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd989d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd98c13b020 0x7fd98c13a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd989d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd98c13b020 0x7fd98c13a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39726/0 (socket says 172.21.15.60:39726) 2024-03-27T21:42:11.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd989d9b640 1 -- 172.21.15.60:0/316733061 learned_addr learned my addr 172.21.15.60:0/316733061 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:11.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd989d9b640 10 monclient(hunting): get_auth_request con 0x7fd98c13b020 auth_method 0 2024-03-27T21:42:11.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd989d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:11.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd989d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:11.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd989d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:11.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd98ad9d640 1 --2- 172.21.15.60:0/316733061 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd98c1416f0 0x7fd98c0aba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd98ad9d640 10 monclient(hunting): get_auth_request con 0x7fd98c1416f0 auth_method 0 2024-03-27T21:42:11.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd98ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd98ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd98ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd98a59c640 1 --2- 172.21.15.60:0/316733061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd98c0b40b0 0x7fd98c13e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd989d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd989d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd989d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd98a59c640 10 monclient(hunting): get_auth_request con 0x7fd98c0b40b0 auth_method 0 2024-03-27T21:42:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd98a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd98a59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd98a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd98ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd98ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd98ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd98a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd98a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd98a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd989d9b640 10 monclient(hunting): handle_auth_done global_id 5454 payload 824 2024-03-27T21:42:11.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd989d9b640 1 -- 172.21.15.60:0/316733061 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd98c1416f0 msgr2=0x7fd98c0aba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd989d9b640 1 --2- 172.21.15.60:0/316733061 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd98c1416f0 0x7fd98c0aba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd989d9b640 1 -- 172.21.15.60:0/316733061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd98c0b40b0 msgr2=0x7fd98c13e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd989d9b640 1 --2- 172.21.15.60:0/316733061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd98c0b40b0 0x7fd98c13e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd989d9b640 10 monclient: _finish_hunting 0 2024-03-27T21:42:11.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd989d9b640 1 monclient: found mon.noname-a 2024-03-27T21:42:11.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd989d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:11.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd989d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:11.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd989d9b640 1 -- 172.21.15.60:0/316733061 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9800094d0 con 0x7fd98c13b020 2024-03-27T21:42:11.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd98ad9d640 1 --2- 172.21.15.60:0/316733061 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd98c1416f0 0x7fd98c0aba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:11.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd989d9b640 1 --2- 172.21.15.60:0/316733061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd98c13b020 0x7fd98c13a5d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd980009870 tx=0x7fd980004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:11.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd97affd640 10 client.?.objecter ms_handle_connect 0x7fd98c13b020 2024-03-27T21:42:11.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.368+0000 7fd97affd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:11.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd97affd640 1 -- 172.21.15.60:0/316733061 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd980030810 con 0x7fd98c13b020 2024-03-27T21:42:11.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd97affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:11.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd97affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:11.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd97affd640 10 monclient: dump: 2024-03-27T21:42:11.378 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:11.378 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:11.378 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.378 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.378 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:11.378 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:11.378 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:11.379 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:11.379 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:11.379 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd97affd640 10 monclient: _finish_auth 0 2024-03-27T21:42:11.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd97affd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:11.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd97affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:11.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd97affd640 1 -- 172.21.15.60:0/316733061 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd980043400 con 0x7fd98c13b020 2024-03-27T21:42:11.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd990ff6640 5 monclient: authenticate success, global_id 5454 2024-03-27T21:42:11.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd990ff6640 10 monclient: _renew_subs 2024-03-27T21:42:11.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd990ff6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:11.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd990ff6640 1 -- 172.21.15.60:0/316733061 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd98c13a0a0 con 0x7fd98c13b020 2024-03-27T21:42:11.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd97affd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:11.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd97affd640 1 -- 172.21.15.60:0/316733061 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9800437b0 con 0x7fd98c13b020 2024-03-27T21:42:11.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd97affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:11.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd97affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:11.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd97affd640 10 monclient: dump: 2024-03-27T21:42:11.380 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:11.380 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:11.381 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.381 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.381 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:11.381 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:11.381 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:11.381 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:11.381 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:11.381 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd988d99640 4 set_mon_vals no callback set 2024-03-27T21:42:11.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd990ff6640 10 client.5454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:11.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd990ff6640 10 monclient: _renew_subs 2024-03-27T21:42:11.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd990ff6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:11.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd990ff6640 1 -- 172.21.15.60:0/316733061 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd98c0abf50 con 0x7fd98c13b020 2024-03-27T21:42:11.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.369+0000 7fd990ff6640 1 librados: init done 2024-03-27T21:42:11.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.370+0000 7fd990ff6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:11.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.370+0000 7fd990ff6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:11.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.370+0000 7fd990ff6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:11.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.370+0000 7fd990ff6640 1 -- 172.21.15.60:0/316733061 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd98c069470 con 0x7fd98c13b020 2024-03-27T21:42:11.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.370+0000 7fd97affd640 1 -- 172.21.15.60:0/316733061 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd980043910 con 0x7fd98c13b020 2024-03-27T21:42:11.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.371+0000 7fd97affd640 1 --2- 172.21.15.60:0/316733061 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd96c06c590 0x7fd96c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.371+0000 7fd98a59c640 1 --2- 172.21.15.60:0/316733061 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd96c06c590 0x7fd96c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.371+0000 7fd98a59c640 10 monclient: get_auth_request con 0x7fd96c06c590 auth_method 0 2024-03-27T21:42:11.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.371+0000 7fd97affd640 1 -- 172.21.15.60:0/316733061 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fd98003e070 con 0x7fd98c13b020 2024-03-27T21:42:11.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.371+0000 7fd97affd640 10 client.5454.objecter ms_dispatch 0x7fd98c000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:11.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.371+0000 7fd97affd640 3 client.5454.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:11.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.371+0000 7fd97affd640 3 client.5454.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:11.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.371+0000 7fd97affd640 20 client.5454.objecter dump_active .. 0 homeless 2024-03-27T21:42:11.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.371+0000 7fd97affd640 10 client.5454.objecter ms_handle_connect 0x7fd96c06c590 2024-03-27T21:42:11.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.371+0000 7fd98a59c640 1 --2- 172.21.15.60:0/316733061 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd96c06c590 0x7fd96c06ea50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd9840012e0 tx=0x7fd984009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:11.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.375+0000 7fd97affd640 1 -- 172.21.15.60:0/316733061 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd9800778f0 con 0x7fd98c13b020 2024-03-27T21:42:11.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.375+0000 7fd97affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:11.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.375+0000 7fd97affd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:11.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.555+0000 7fd990ff6640 1 -- 172.21.15.60:0/316733061 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fd98c0b6be0 con 0x7fd96c06c590 2024-03-27T21:42:11.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.555+0000 7fd97affd640 1 -- 172.21.15.60:0/316733061 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd98c0b6be0 con 0x7fd96c06c590 2024-03-27T21:42:11.542 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 0 to deep-scrub 2024-03-27T21:42:11.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.557+0000 7fd990ff6640 10 librados: watch_flush enter 2024-03-27T21:42:11.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.557+0000 7fd990ff6640 10 librados: watch_flush exit 2024-03-27T21:42:11.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.557+0000 7fd990ff6640 20 client.5454.objecter shutdown clearing up homeless session... 2024-03-27T21:42:11.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.557+0000 7fd990ff6640 10 client.5454.objecter successfully canceled tick 2024-03-27T21:42:11.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.557+0000 7fd990ff6640 1 -- 172.21.15.60:0/316733061 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd96c06c590 msgr2=0x7fd96c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.557+0000 7fd990ff6640 1 --2- 172.21.15.60:0/316733061 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd96c06c590 0x7fd96c06ea50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd9840012e0 tx=0x7fd984009040 comp rx=0 tx=0).stop 2024-03-27T21:42:11.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.557+0000 7fd990ff6640 10 monclient: shutdown 2024-03-27T21:42:11.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.557+0000 7fd990ff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:11.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.557+0000 7fd990ff6640 1 -- 172.21.15.60:0/316733061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd98c13b020 msgr2=0x7fd98c13a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.557+0000 7fd990ff6640 1 --2- 172.21.15.60:0/316733061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd98c13b020 0x7fd98c13a5d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd980009870 tx=0x7fd980004730 comp rx=0 tx=0).stop 2024-03-27T21:42:11.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.558+0000 7fd990ff6640 1 -- 172.21.15.60:0/316733061 shutdown_connections 2024-03-27T21:42:11.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.558+0000 7fd990ff6640 1 --2- 172.21.15.60:0/316733061 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd98c1416f0 0x7fd98c0aba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.558+0000 7fd990ff6640 1 --2- 172.21.15.60:0/316733061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd98c13b020 0x7fd98c13a5d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.558+0000 7fd990ff6640 1 --2- 172.21.15.60:0/316733061 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd96c06c590 0x7fd96c06ea50 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.558+0000 7fd990ff6640 1 --2- 172.21.15.60:0/316733061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd98c0b40b0 0x7fd98c13e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.558+0000 7fd990ff6640 1 -- 172.21.15.60:0/316733061 >> 172.21.15.60:0/316733061 conn(0x7fd98c0608e0 msgr2=0x7fd98c0b5f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:11.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.558+0000 7fd990ff6640 1 -- 172.21.15.60:0/316733061 shutdown_connections 2024-03-27T21:42:11.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.558+0000 7fd990ff6640 1 -- 172.21.15.60:0/316733061 wait complete. 2024-03-27T21:42:11.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.558+0000 7fd990ff6640 1 librados: shutdown 2024-03-27T21:42:11.554 INFO:tasks.ceph:Scrubbing osd.1 2024-03-27T21:42:11.555 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-27T21:42:11.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.672+0000 7f9cc0f42640 10 monclient: get_monmap_and_config 2024-03-27T21:42:11.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.672+0000 7f9cc0f42640 10 monclient: build_initial_monmap 2024-03-27T21:42:11.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.672+0000 7f9cc0f42640 10 monclient: monmap: 2024-03-27T21:42:11.659 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:11.660 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:11.660 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:11.673376+0000 2024-03-27T21:42:11.660 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:11.673376+0000 2024-03-27T21:42:11.660 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:11.661 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:11.661 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:11.661 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:11.661 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:11.662 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:11.662 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:11.662 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.672+0000 7f9cc0f42640 1 Processor -- start 2024-03-27T21:42:11.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.672+0000 7f9cc0f42640 1 -- start start 2024-03-27T21:42:11.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cc0f42640 10 monclient: init 2024-03-27T21:42:11.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cc0f42640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:11.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cc0f42640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-03-27T21:42:11.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cc0f42640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9cbc0b3980 0x7f9cbc1449c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cc0f42640 10 monclient(hunting): picked mon.noname-c con 0x7f9cbc0b3980 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:11.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cc0f42640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9cbc0b3d60 0x7f9cbc0b0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cc0f42640 10 monclient(hunting): picked mon.noname-a con 0x7f9cbc0b3d60 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:11.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cc0f42640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9cbc0a86e0 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:11.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cc0f42640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cbc144f00 con 0x7f9cbc0a86e0 2024-03-27T21:42:11.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cc0f42640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:11.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cc0f42640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cbc0b1140 con 0x7f9cbc0b3d60 2024-03-27T21:42:11.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cc0f42640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:11.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cc0f42640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cbc0b07f0 con 0x7f9cbc0b3980 2024-03-27T21:42:11.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cc0f42640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:11.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cc0f42640 10 monclient(hunting): authenticate will time out at 972.894656s 2024-03-27T21:42:11.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cb9d9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9cbc0b3d60 0x7f9cbc0b0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cb9d9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9cbc0b3d60 0x7f9cbc0b0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59234/0 (socket says 172.21.15.60:59234) 2024-03-27T21:42:11.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cb9d9b640 1 -- 172.21.15.60:0/3068076004 learned_addr learned my addr 172.21.15.60:0/3068076004 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:11.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cb9d9b640 10 monclient(hunting): get_auth_request con 0x7f9cbc0b3d60 auth_method 0 2024-03-27T21:42:11.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cb9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:11.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cb9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:11.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.673+0000 7f9cb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:11.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.674+0000 7f9cb9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:11.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.674+0000 7f9cb9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:11.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.674+0000 7f9cb9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:11.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.674+0000 7f9cba59c640 1 --2- 172.21.15.60:0/3068076004 >> v2:172.21.15.191:3300/0 conn(0x7f9cbc0b3980 0x7f9cbc1449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.674+0000 7f9cba59c640 10 monclient(hunting): get_auth_request con 0x7f9cbc0b3980 auth_method 0 2024-03-27T21:42:11.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.674+0000 7f9cba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:11.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.674+0000 7f9cba59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:11.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.674+0000 7f9cba59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:11.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.674+0000 7f9cb9d9b640 10 monclient(hunting): handle_auth_done global_id 5457 payload 274 2024-03-27T21:42:11.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.674+0000 7f9cb9d9b640 1 -- 172.21.15.60:0/3068076004 >> v2:172.21.15.191:3300/0 conn(0x7f9cbc0b3980 msgr2=0x7f9cbc1449c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:11.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.674+0000 7f9cb9d9b640 1 --2- 172.21.15.60:0/3068076004 >> v2:172.21.15.191:3300/0 conn(0x7f9cbc0b3980 0x7f9cbc1449c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.674+0000 7f9cb9d9b640 1 -- 172.21.15.60:0/3068076004 >> v1:172.21.15.60:6789/0 conn(0x7f9cbc0a86e0 legacy=0x7f9cbc0b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.674+0000 7f9cb9d9b640 10 monclient: _finish_hunting 0 2024-03-27T21:42:11.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.674+0000 7f9cb9d9b640 1 monclient: found mon.noname-a 2024-03-27T21:42:11.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.674+0000 7f9cb9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:11.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.674+0000 7f9cb9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:11.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.674+0000 7f9cb9d9b640 1 -- 172.21.15.60:0/3068076004 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cbc0aa1d0 con 0x7f9cbc0b3d60 2024-03-27T21:42:11.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.674+0000 7f9cb9d9b640 1 --2- 172.21.15.60:0/3068076004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cbc0b3d60 0x7f9cbc0b0c00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc0b42d0 tx=0x7f9ca402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3351e3e33d4a0234 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:11.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cb959a640 1 -- 172.21.15.60:0/3068076004 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4030860 con 0x7f9cbc0b3d60 2024-03-27T21:42:11.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cb959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:11.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cb959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:11.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cb959a640 10 monclient: dump: 2024-03-27T21:42:11.686 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:11.686 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:11.687 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.687 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.687 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:11.687 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:11.687 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:11.687 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:11.688 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:11.688 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cb959a640 10 monclient: _finish_auth 0 2024-03-27T21:42:11.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cb959a640 10 monclient: _check_auth_tickets 2024-03-27T21:42:11.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cb959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:11.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cb959a640 1 -- 172.21.15.60:0/3068076004 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca40309c0 con 0x7f9cbc0b3d60 2024-03-27T21:42:11.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cc0f42640 5 monclient: authenticate success, global_id 5457 2024-03-27T21:42:11.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cc0f42640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:11.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cb959a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:11.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cc0f42640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:11.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cb959a640 1 -- 172.21.15.60:0/3068076004 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4030d50 con 0x7f9cbc0b3d60 2024-03-27T21:42:11.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cb959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:11.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cc0f42640 4 set_mon_vals no callback set 2024-03-27T21:42:11.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cb959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:11.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cb959a640 10 monclient: dump: 2024-03-27T21:42:11.692 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:11.692 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:11.692 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.692 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.693 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:11.693 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:11.693 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:11.693 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:11.694 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:11.694 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cc0f42640 10 monclient: shutdown 2024-03-27T21:42:11.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cc0f42640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:11.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cc0f42640 1 -- 172.21.15.60:0/3068076004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cbc0b3d60 msgr2=0x7f9cbc0b0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cc0f42640 1 --2- 172.21.15.60:0/3068076004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cbc0b3d60 0x7f9cbc0b0c00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc0b42d0 tx=0x7f9ca402b9a0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cc0f42640 1 -- 172.21.15.60:0/3068076004 shutdown_connections 2024-03-27T21:42:11.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cc0f42640 1 --2- 172.21.15.60:0/3068076004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cbc0b3d60 0x7f9cbc0b0c00 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cc0f42640 1 --2- 172.21.15.60:0/3068076004 >> v2:172.21.15.191:3300/0 conn(0x7f9cbc0b3980 0x7f9cbc1449c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cc0f42640 1 -- 172.21.15.60:0/3068076004 >> 172.21.15.60:0/3068076004 conn(0x7f9cbc071410 msgr2=0x7f9cbc071820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:11.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cc0f42640 1 -- 172.21.15.60:0/3068076004 shutdown_connections 2024-03-27T21:42:11.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.675+0000 7f9cc0f42640 1 -- 172.21.15.60:0/3068076004 wait complete. 2024-03-27T21:42:11.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.676+0000 7f9cc0f42640 10 monclient: build_initial_monmap 2024-03-27T21:42:11.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.676+0000 7f9cc0f42640 10 monclient: monmap: 2024-03-27T21:42:11.697 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:11.697 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:11.698 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:11.698 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:11.698 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:11.699 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:11.699 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:11.699 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:11.699 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:11.700 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.676+0000 7f9cc0f42640 1 librados: starting msgr at 2024-03-27T21:42:11.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.676+0000 7f9cc0f42640 1 librados: starting objecter 2024-03-27T21:42:11.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.676+0000 7f9cc0f42640 1 Processor -- start 2024-03-27T21:42:11.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.676+0000 7f9cc0f42640 1 -- start start 2024-03-27T21:42:11.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.676+0000 7f9cc0f42640 1 librados: setting wanted keys 2024-03-27T21:42:11.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.676+0000 7f9cc0f42640 1 librados: calling monclient init 2024-03-27T21:42:11.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.676+0000 7f9cc0f42640 10 monclient: init 2024-03-27T21:42:11.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cc0f42640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:11.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cc0f42640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-27T21:42:11.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cc0f42640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9cbc0a86e0 0x7f9cbc141890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cc0f42640 10 monclient(hunting): picked mon.noname-c con 0x7f9cbc0a86e0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:11.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cc0f42640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cbc0b3980 0x7f9cbc13d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cc0f42640 10 monclient(hunting): picked mon.noname-a con 0x7f9cbc0b3980 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:11.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cc0f42640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cbc13d6b0 0x7f9cbc136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cc0f42640 10 monclient(hunting): picked mon.noname-b con 0x7f9cbc13d6b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:11.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cc0f42640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:11.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cc0f42640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cbc0aae80 con 0x7f9cbc0b3980 2024-03-27T21:42:11.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cc0f42640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:11.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cc0f42640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cbc13dc40 con 0x7f9cbc13d6b0 2024-03-27T21:42:11.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cc0f42640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:11.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cc0f42640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cbc1374f0 con 0x7f9cbc0a86e0 2024-03-27T21:42:11.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cc0f42640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:11.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cc0f42640 10 monclient(hunting): authenticate will time out at 972.898285s 2024-03-27T21:42:11.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cba59c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9cbc0a86e0 0x7f9cbc141890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cb9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cbc0b3980 0x7f9cbc13d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cba59c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9cbc0a86e0 0x7f9cbc141890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:51404/0 (socket says 172.21.15.60:51404) 2024-03-27T21:42:11.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cba59c640 1 -- 172.21.15.60:0/926821516 learned_addr learned my addr 172.21.15.60:0/926821516 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:11.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cba59c640 10 monclient(hunting): get_auth_request con 0x7f9cbc0a86e0 auth_method 0 2024-03-27T21:42:11.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:11.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cba59c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:11.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cba59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:11.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cb9d9b640 10 monclient(hunting): get_auth_request con 0x7f9cbc0b3980 auth_method 0 2024-03-27T21:42:11.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cb9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:11.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cb9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:11.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:11.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cb9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:11.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cb9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:11.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cb9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:11.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:11.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:11.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:11.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cbad9d640 1 --2- 172.21.15.60:0/926821516 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cbc13d6b0 0x7f9cbc136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cbad9d640 10 monclient(hunting): get_auth_request con 0x7f9cbc13d6b0 auth_method 0 2024-03-27T21:42:11.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cbad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:11.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cbad9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:11.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cbad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:11.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cbad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:11.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cbad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:11.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cbad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:11.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cb9d9b640 10 monclient(hunting): handle_auth_done global_id 5463 payload 824 2024-03-27T21:42:11.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cb9d9b640 1 -- 172.21.15.60:0/926821516 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9cbc0a86e0 msgr2=0x7f9cbc141890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cb9d9b640 1 --2- 172.21.15.60:0/926821516 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9cbc0a86e0 0x7f9cbc141890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cb9d9b640 1 -- 172.21.15.60:0/926821516 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cbc13d6b0 msgr2=0x7f9cbc136fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cb9d9b640 1 --2- 172.21.15.60:0/926821516 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cbc13d6b0 0x7f9cbc136fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cb9d9b640 10 monclient: _finish_hunting 0 2024-03-27T21:42:11.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cb9d9b640 1 monclient: found mon.noname-a 2024-03-27T21:42:11.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cb9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:11.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cb9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:11.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cb9d9b640 1 -- 172.21.15.60:0/926821516 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca40094d0 con 0x7f9cbc0b3980 2024-03-27T21:42:11.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cba59c640 1 --2- 172.21.15.60:0/926821516 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f9cbc0a86e0 0x7f9cbc141890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:11.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cb9d9b640 1 --2- 172.21.15.60:0/926821516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cbc0b3980 0x7f9cbc13d090 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4033000 tx=0x7f9ca4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:11.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cb2ffd640 10 client.?.objecter ms_handle_connect 0x7f9cbc0b3980 2024-03-27T21:42:11.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.677+0000 7f9cb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:11.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cb2ffd640 1 -- 172.21.15.60:0/926821516 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4039a90 con 0x7f9cbc0b3980 2024-03-27T21:42:11.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:11.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cb2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:11.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cb2ffd640 10 monclient: dump: 2024-03-27T21:42:11.716 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:11.716 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:11.716 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.716 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.716 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:11.717 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:11.717 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:11.717 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:11.717 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:11.717 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cb2ffd640 10 monclient: _finish_auth 0 2024-03-27T21:42:11.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cb2ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:11.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:11.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cc0f42640 5 monclient: authenticate success, global_id 5463 2024-03-27T21:42:11.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cb2ffd640 1 -- 172.21.15.60:0/926821516 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca4039c10 con 0x7f9cbc0b3980 2024-03-27T21:42:11.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cb2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:11.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cb2ffd640 1 -- 172.21.15.60:0/926821516 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4043410 con 0x7f9cbc0b3980 2024-03-27T21:42:11.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cc0f42640 10 monclient: _renew_subs 2024-03-27T21:42:11.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cc0f42640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:11.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cc0f42640 1 -- 172.21.15.60:0/926821516 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cbc137710 con 0x7f9cbc0b3980 2024-03-27T21:42:11.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cb8d99640 4 set_mon_vals no callback set 2024-03-27T21:42:11.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:11.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cb2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:11.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cb2ffd640 10 monclient: dump: 2024-03-27T21:42:11.719 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:11.719 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:11.719 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.719 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.720 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:11.720 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:11.720 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:11.720 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:11.721 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:11.721 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cc0f42640 10 client.5463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:11.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cc0f42640 10 monclient: _renew_subs 2024-03-27T21:42:11.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cc0f42640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:11.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cc0f42640 1 -- 172.21.15.60:0/926821516 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cbc1d9ad0 con 0x7f9cbc0b3980 2024-03-27T21:42:11.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cc0f42640 1 librados: init done 2024-03-27T21:42:11.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cc0f42640 10 client.5463.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:11.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cc0f42640 15 client.5463.objecter _session_command_op_assign -1 1 2024-03-27T21:42:11.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cc0f42640 20 client.5463.objecter _get_session osd=-1 returning homeless 2024-03-27T21:42:11.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cc0f42640 10 client.5463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:11.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cc0f42640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:11.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.678+0000 7f9cc0f42640 1 -- 172.21.15.60:0/926821516 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9cbc0b13b0 con 0x7f9cbc0b3980 2024-03-27T21:42:11.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.679+0000 7f9cb2ffd640 1 -- 172.21.15.60:0/926821516 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9ca4043570 con 0x7f9cbc0b3980 2024-03-27T21:42:11.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.679+0000 7f9cb2ffd640 1 --2- 172.21.15.60:0/926821516 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9c9c06c5c0 0x7f9c9c06ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cba59c640 1 --2- 172.21.15.60:0/926821516 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9c9c06c5c0 0x7f9c9c06ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cb2ffd640 1 -- 172.21.15.60:0/926821516 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f9ca40aba20 con 0x7f9cbc0b3980 2024-03-27T21:42:11.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cb2ffd640 10 client.5463.objecter ms_dispatch 0x7f9cbc000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:11.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cb2ffd640 3 client.5463.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:11.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cb2ffd640 3 client.5463.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:11.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cba59c640 10 monclient: get_auth_request con 0x7f9c9c06c5c0 auth_method 0 2024-03-27T21:42:11.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cb2ffd640 10 client.5463.objecter checking command 1 2024-03-27T21:42:11.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cb2ffd640 1 --2- 172.21.15.60:0/926821516 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f9c9c072a00 0x7f9c9c074e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cb2ffd640 20 client.5463.objecter _get_session s=0x7f9c9c072330 osd=1 3 2024-03-27T21:42:11.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cb2ffd640 20 client.5463.objecter put_session s=0x7f9c9c072330 osd=1 3 2024-03-27T21:42:11.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cb2ffd640 15 client.5463.objecter _session_command_op_remove -1 1 2024-03-27T21:42:11.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cb2ffd640 20 client.5463.objecter _get_session s=0x7f9c9c072330 osd=1 3 2024-03-27T21:42:11.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cb2ffd640 20 client.5463.objecter get_session s=0x7f9c9c072330 osd=1 3 2024-03-27T21:42:11.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cb2ffd640 15 client.5463.objecter _session_command_op_assign 1 1 2024-03-27T21:42:11.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cb2ffd640 20 client.5463.objecter put_session s=0x7f9c9c072330 osd=1 4 2024-03-27T21:42:11.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cb2ffd640 10 client.5463.objecter _send_command 1 2024-03-27T21:42:11.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cb2ffd640 1 -- 172.21.15.60:0/926821516 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c9c0754f0 con 0x7f9c9c072a00 2024-03-27T21:42:11.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cb2ffd640 20 client.5463.objecter dump_active .. 0 homeless 2024-03-27T21:42:11.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cb2ffd640 1 -- 172.21.15.60:0/926821516 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ca40abd90 con 0x7f9cbc0b3980 2024-03-27T21:42:11.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cb2ffd640 10 monclient: handle_get_version_reply finishing 1 version 92 2024-03-27T21:42:11.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cbad9d640 1 --2- 172.21.15.60:0/926821516 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f9c9c072a00 0x7f9c9c074e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cbad9d640 10 monclient: get_auth_request con 0x7f9c9c072a00 auth_method 0 2024-03-27T21:42:11.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cb2ffd640 10 client.5463.objecter ms_handle_connect 0x7f9c9c06c5c0 2024-03-27T21:42:11.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cba59c640 1 --2- 172.21.15.60:0/926821516 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9c9c06c5c0 0x7f9c9c06ea80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9cac003fc0 tx=0x7f9cac009970 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:11.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cb2ffd640 10 client.5463.objecter ms_handle_connect 0x7f9c9c072a00 2024-03-27T21:42:11.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.680+0000 7f9cbad9d640 1 --2- 172.21.15.60:0/926821516 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f9c9c072a00 0x7f9c9c074e20 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:11.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.681+0000 7f9cb2ffd640 1 -- 172.21.15.60:0/926821516 <== osd.1 v2:172.21.15.60:6800/34602 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f9c9c0754f0 con 0x7f9c9c072a00 2024-03-27T21:42:11.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.681+0000 7f9cb2ffd640 10 client.5463.objecter ms_dispatch 0x7f9cbc000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:42:11.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.681+0000 7f9cb2ffd640 10 client.5463.objecter _finish_command 1 = system:0 2024-03-27T21:42:11.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.681+0000 7f9cb2ffd640 20 client.5463.objecter put_session s=0x7f9c9c072330 osd=1 4 2024-03-27T21:42:11.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.681+0000 7f9cb2ffd640 15 client.5463.objecter _session_command_op_remove 1 1 2024-03-27T21:42:11.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.693+0000 7f9cc0f42640 10 client.5463.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-27T21:42:11.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.693+0000 7f9cc0f42640 15 client.5463.objecter _session_command_op_assign -1 2 2024-03-27T21:42:11.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.693+0000 7f9cc0f42640 20 client.5463.objecter _get_session s=0x7f9c9c072330 osd=1 3 2024-03-27T21:42:11.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.693+0000 7f9cc0f42640 20 client.5463.objecter put_session s=0x7f9c9c072330 osd=1 3 2024-03-27T21:42:11.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.693+0000 7f9cc0f42640 20 client.5463.objecter _get_session s=0x7f9c9c072330 osd=1 3 2024-03-27T21:42:11.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.693+0000 7f9cc0f42640 15 client.5463.objecter _session_command_op_remove -1 2 2024-03-27T21:42:11.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.693+0000 7f9cc0f42640 20 client.5463.objecter get_session s=0x7f9c9c072330 osd=1 3 2024-03-27T21:42:11.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.693+0000 7f9cc0f42640 15 client.5463.objecter _session_command_op_assign 1 2 2024-03-27T21:42:11.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.693+0000 7f9cc0f42640 20 client.5463.objecter put_session s=0x7f9c9c072330 osd=1 4 2024-03-27T21:42:11.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.693+0000 7f9cc0f42640 10 client.5463.objecter _send_command 2 2024-03-27T21:42:11.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.693+0000 7f9cc0f42640 1 -- 172.21.15.60:0/926821516 --> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9cbc0670d0 con 0x7f9c9c072a00 2024-03-27T21:42:11.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.693+0000 7f9cb2ffd640 1 -- 172.21.15.60:0/926821516 <== osd.1 v2:172.21.15.60:6800/34602 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f9cbc0670d0 con 0x7f9c9c072a00 2024-03-27T21:42:11.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.693+0000 7f9cb2ffd640 10 client.5463.objecter ms_dispatch 0x7f9cbc000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:42:11.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.693+0000 7f9cb2ffd640 10 client.5463.objecter _finish_command 2 = system:0 2024-03-27T21:42:11.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.693+0000 7f9cb2ffd640 20 client.5463.objecter put_session s=0x7f9c9c072330 osd=1 4 2024-03-27T21:42:11.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.693+0000 7f9cb2ffd640 15 client.5463.objecter _session_command_op_remove 1 2 2024-03-27T21:42:11.729 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:42:11.729 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-27T21:42:11.729 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:42:11.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.694+0000 7f9cc0f42640 10 librados: watch_flush enter 2024-03-27T21:42:11.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.694+0000 7f9cc0f42640 10 librados: watch_flush exit 2024-03-27T21:42:11.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.694+0000 7f9cc0f42640 10 client.5463.objecter close_session for osd.1 2024-03-27T21:42:11.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.694+0000 7f9cc0f42640 1 -- 172.21.15.60:0/926821516 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f9c9c072a00 msgr2=0x7f9c9c074e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.694+0000 7f9cc0f42640 1 --2- 172.21.15.60:0/926821516 >> [v2:172.21.15.60:6800/34602,v1:172.21.15.60:6801/34602] conn(0x7f9c9c072a00 0x7f9c9c074e20 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.694+0000 7f9cc0f42640 20 client.5463.objecter put_session s=0x7f9c9c072330 osd=1 1 2024-03-27T21:42:11.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.694+0000 7f9cc0f42640 20 client.5463.objecter shutdown clearing up homeless session... 2024-03-27T21:42:11.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.694+0000 7f9cc0f42640 10 client.5463.objecter successfully canceled tick 2024-03-27T21:42:11.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.694+0000 7f9cc0f42640 1 -- 172.21.15.60:0/926821516 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9c9c06c5c0 msgr2=0x7f9c9c06ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.694+0000 7f9cc0f42640 1 --2- 172.21.15.60:0/926821516 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f9c9c06c5c0 0x7f9c9c06ea80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9cac003fc0 tx=0x7f9cac009970 comp rx=0 tx=0).stop 2024-03-27T21:42:11.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.694+0000 7f9cc0f42640 10 monclient: shutdown 2024-03-27T21:42:11.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.694+0000 7f9cc0f42640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:11.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.694+0000 7f9cc0f42640 1 -- 172.21.15.60:0/926821516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cbc0b3980 msgr2=0x7f9cbc13d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.694+0000 7f9cc0f42640 1 --2- 172.21.15.60:0/926821516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cbc0b3980 0x7f9cbc13d090 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4033000 tx=0x7f9ca4004730 comp rx=0 tx=0).stop 2024-03-27T21:42:11.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.694+0000 7f9cbad9d640 1 -- 172.21.15.60:0/926821516 reap_dead start 2024-03-27T21:42:11.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.694+0000 7f9cc0f42640 1 -- 172.21.15.60:0/926821516 shutdown_connections 2024-03-27T21:42:11.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.694+0000 7f9cc0f42640 1 -- 172.21.15.60:0/926821516 >> 172.21.15.60:0/926821516 conn(0x7f9cbc071410 msgr2=0x7f9cbc0b0050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:11.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.694+0000 7f9cc0f42640 1 -- 172.21.15.60:0/926821516 shutdown_connections 2024-03-27T21:42:11.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.694+0000 7f9cc0f42640 1 -- 172.21.15.60:0/926821516 wait complete. 2024-03-27T21:42:11.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.694+0000 7f9cc0f42640 1 librados: shutdown 2024-03-27T21:42:11.737 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-27T21:42:11.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.852+0000 7f3bbbc2f640 10 monclient: get_monmap_and_config 2024-03-27T21:42:11.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.852+0000 7f3bbbc2f640 10 monclient: build_initial_monmap 2024-03-27T21:42:11.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.852+0000 7f3bbbc2f640 10 monclient: monmap: 2024-03-27T21:42:11.839 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:11.840 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:11.840 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:11.853452+0000 2024-03-27T21:42:11.840 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:11.853452+0000 2024-03-27T21:42:11.840 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:11.840 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:11.841 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:11.841 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:11.841 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:11.841 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:11.841 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:11.842 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bbbc2f640 1 Processor -- start 2024-03-27T21:42:11.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bbbc2f640 1 -- start start 2024-03-27T21:42:11.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bbbc2f640 10 monclient: init 2024-03-27T21:42:11.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bbbc2f640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:11.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bbbc2f640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-03-27T21:42:11.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bbbc2f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3bb4071a50 0x7f3bb40a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bbbc2f640 10 monclient(hunting): picked mon.noname-b con 0x7f3bb4071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:11.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bbbc2f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3bb413e810 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:11.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bbbc2f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3bb413ec50 0x7f3bb4141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bbbc2f640 10 monclient(hunting): picked mon.noname-a con 0x7f3bb413ec50 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:11.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bbbc2f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb40a8e90 con 0x7f3bb413e810 2024-03-27T21:42:11.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bbbc2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:11.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bbbc2f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bb4071e30 con 0x7f3bb413ec50 2024-03-27T21:42:11.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bbbc2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:11.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bbbc2f640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb40aa040 con 0x7f3bb4071a50 2024-03-27T21:42:11.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bbbc2f640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:11.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bbbc2f640 10 monclient(hunting): authenticate will time out at 973.074748s 2024-03-27T21:42:11.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bb99cb640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3bb4071a50 0x7f3bb40a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bba1cc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3bb413ec50 0x7f3bb4141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bb99cb640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3bb4071a50 0x7f3bb40a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:51406/0 (socket says 172.21.15.60:51406) 2024-03-27T21:42:11.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bba1cc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3bb413ec50 0x7f3bb4141d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59244/0 (socket says 172.21.15.60:59244) 2024-03-27T21:42:11.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bb99cb640 1 -- 172.21.15.60:0/1914300732 learned_addr learned my addr 172.21.15.60:0/1914300732 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:11.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bb99cb640 10 monclient(hunting): get_auth_request con 0x7f3bb4071a50 auth_method 0 2024-03-27T21:42:11.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bb99cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:11.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bb99cb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:11.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.853+0000 7f3bb99cb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:11.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.854+0000 7f3bba1cc640 10 monclient(hunting): get_auth_request con 0x7f3bb413ec50 auth_method 0 2024-03-27T21:42:11.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.854+0000 7f3bba1cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:11.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.854+0000 7f3bba1cc640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:11.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.854+0000 7f3bba1cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:11.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.854+0000 7f3bb99cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:11.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.854+0000 7f3bb99cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:11.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.854+0000 7f3bb99cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:11.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.854+0000 7f3bba1cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:11.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.854+0000 7f3bba1cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:11.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.854+0000 7f3bba1cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:11.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.854+0000 7f3bb99cb640 10 monclient(hunting): handle_auth_done global_id 5135 payload 274 2024-03-27T21:42:11.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.854+0000 7f3bb99cb640 1 -- 172.21.15.60:0/1914300732 >> v2:172.21.15.60:3300/0 conn(0x7f3bb413ec50 msgr2=0x7f3bb4141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:11.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.854+0000 7f3bb99cb640 1 --2- 172.21.15.60:0/1914300732 >> v2:172.21.15.60:3300/0 conn(0x7f3bb413ec50 0x7f3bb4141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.854+0000 7f3bb99cb640 1 -- 172.21.15.60:0/1914300732 >> v1:172.21.15.60:6789/0 conn(0x7f3bb413e810 legacy=0x7f3bb4141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.854+0000 7f3bb99cb640 10 monclient: _finish_hunting 0 2024-03-27T21:42:11.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.854+0000 7f3bb99cb640 1 monclient: found mon.noname-b 2024-03-27T21:42:11.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.854+0000 7f3bb99cb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:11.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.854+0000 7f3bb99cb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:42:11.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.854+0000 7f3bb99cb640 1 -- 172.21.15.60:0/1914300732 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb4142280 con 0x7f3bb4071a50 2024-03-27T21:42:11.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.854+0000 7f3bba1cc640 1 --2- 172.21.15.60:0/1914300732 >> v2:172.21.15.60:3300/0 conn(0x7f3bb413ec50 0x7f3bb4141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:11.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bb99cb640 1 --2- 172.21.15.60:0/1914300732 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3bb4071a50 0x7f3bb40a87e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3bac009870 tx=0x7f3bac02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c33cc77489ab009 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:11.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bb89c9640 1 -- 172.21.15.60:0/1914300732 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac030840 con 0x7f3bb4071a50 2024-03-27T21:42:11.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bb89c9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:11.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bb89c9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:11.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bb89c9640 10 monclient: dump: 2024-03-27T21:42:11.867 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:11.868 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:11.868 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.868 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.868 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:11.868 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:11.868 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:11.869 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:11.869 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:11.869 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bb89c9640 10 monclient: _finish_auth 0 2024-03-27T21:42:11.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bb89c9640 10 monclient: _check_auth_tickets 2024-03-27T21:42:11.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bb89c9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:11.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bbbc2f640 5 monclient: authenticate success, global_id 5135 2024-03-27T21:42:11.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bbbc2f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:11.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bb89c9640 1 -- 172.21.15.60:0/1914300732 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bac0309a0 con 0x7f3bb4071a50 2024-03-27T21:42:11.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bb89c9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:11.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bbbc2f640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:11.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bbbc2f640 4 set_mon_vals no callback set 2024-03-27T21:42:11.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bb89c9640 1 -- 172.21.15.60:0/1914300732 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac030d30 con 0x7f3bb4071a50 2024-03-27T21:42:11.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bb89c9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:11.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bb89c9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:11.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bb89c9640 10 monclient: dump: 2024-03-27T21:42:11.871 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:11.872 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:11.872 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.872 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.872 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:11.872 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:11.872 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:11.872 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:11.873 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:11.873 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bbbc2f640 10 monclient: shutdown 2024-03-27T21:42:11.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bbbc2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:11.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bbbc2f640 1 -- 172.21.15.60:0/1914300732 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3bb4071a50 msgr2=0x7f3bb40a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bbbc2f640 1 --2- 172.21.15.60:0/1914300732 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3bb4071a50 0x7f3bb40a87e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3bac009870 tx=0x7f3bac02b980 comp rx=0 tx=0).stop 2024-03-27T21:42:11.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bbbc2f640 1 -- 172.21.15.60:0/1914300732 shutdown_connections 2024-03-27T21:42:11.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bbbc2f640 1 --2- 172.21.15.60:0/1914300732 >> v2:172.21.15.60:3300/0 conn(0x7f3bb413ec50 0x7f3bb4141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bbbc2f640 1 --2- 172.21.15.60:0/1914300732 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3bb4071a50 0x7f3bb40a87e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bbbc2f640 1 -- 172.21.15.60:0/1914300732 >> 172.21.15.60:0/1914300732 conn(0x7f3bb40b7300 msgr2=0x7f3bb40b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:11.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bbbc2f640 1 -- 172.21.15.60:0/1914300732 shutdown_connections 2024-03-27T21:42:11.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.855+0000 7f3bbbc2f640 1 -- 172.21.15.60:0/1914300732 wait complete. 2024-03-27T21:42:11.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.856+0000 7f3bbbc2f640 10 monclient: build_initial_monmap 2024-03-27T21:42:11.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.856+0000 7f3bbbc2f640 10 monclient: monmap: 2024-03-27T21:42:11.874 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:11.874 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:11.875 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:11.875 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:11.875 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:11.875 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:11.875 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:11.875 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:11.875 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:11.875 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.856+0000 7f3bbbc2f640 1 librados: starting msgr at 2024-03-27T21:42:11.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.856+0000 7f3bbbc2f640 1 librados: starting objecter 2024-03-27T21:42:11.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.856+0000 7f3bbbc2f640 1 Processor -- start 2024-03-27T21:42:11.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.856+0000 7f3bbbc2f640 1 -- start start 2024-03-27T21:42:11.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.856+0000 7f3bbbc2f640 1 librados: setting wanted keys 2024-03-27T21:42:11.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.856+0000 7f3bbbc2f640 1 librados: calling monclient init 2024-03-27T21:42:11.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.856+0000 7f3bbbc2f640 10 monclient: init 2024-03-27T21:42:11.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bbbc2f640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:11.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bbbc2f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:42:11.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bbbc2f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3bb4071a50 0x7f3bb4139a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bbbc2f640 10 monclient(hunting): picked mon.noname-a con 0x7f3bb4071a50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:11.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bbbc2f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3bb413e810 0x7f3bb4136540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bbbc2f640 10 monclient(hunting): picked mon.noname-c con 0x7f3bb413e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:11.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bbbc2f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bb413ec50 0x7f3bb4132140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bbbc2f640 10 monclient(hunting): picked mon.noname-b con 0x7f3bb413ec50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:11.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bbbc2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:11.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bbbc2f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb4137200 con 0x7f3bb4071a50 2024-03-27T21:42:11.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bbbc2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:11.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bbbc2f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb413a020 con 0x7f3bb413ec50 2024-03-27T21:42:11.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bbbc2f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:11.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bbbc2f640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb41394f0 con 0x7f3bb413e810 2024-03-27T21:42:11.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bbbc2f640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:11.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bbbc2f640 10 monclient(hunting): authenticate will time out at 973.078314s 2024-03-27T21:42:11.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb99cb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3bb4071a50 0x7f3bb4139a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb91ca640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3bb413e810 0x7f3bb4136540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb99cb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3bb4071a50 0x7f3bb4139a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59250/0 (socket says 172.21.15.60:59250) 2024-03-27T21:42:11.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb99cb640 1 -- 172.21.15.60:0/197161979 learned_addr learned my addr 172.21.15.60:0/197161979 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:11.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb99cb640 10 monclient(hunting): get_auth_request con 0x7f3bb4071a50 auth_method 0 2024-03-27T21:42:11.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb99cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:11.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb99cb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:11.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb99cb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:11.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb91ca640 10 monclient(hunting): get_auth_request con 0x7f3bb413e810 auth_method 0 2024-03-27T21:42:11.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb91ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:11.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb91ca640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:11.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb91ca640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:11.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb99cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:11.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb99cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:11.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb99cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:11.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb91ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:11.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb91ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:11.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb91ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:11.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb99cb640 10 monclient(hunting): handle_auth_done global_id 5472 payload 824 2024-03-27T21:42:11.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bba1cc640 1 --2- 172.21.15.60:0/197161979 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bb413ec50 0x7f3bb4132140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb99cb640 1 -- 172.21.15.60:0/197161979 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3bb413e810 msgr2=0x7f3bb4136540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb99cb640 1 --2- 172.21.15.60:0/197161979 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3bb413e810 0x7f3bb4136540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb99cb640 1 -- 172.21.15.60:0/197161979 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bb413ec50 msgr2=0x7f3bb4132140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:11.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb99cb640 1 --2- 172.21.15.60:0/197161979 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bb413ec50 0x7f3bb4132140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:11.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb99cb640 10 monclient: _finish_hunting 0 2024-03-27T21:42:11.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb99cb640 1 monclient: found mon.noname-a 2024-03-27T21:42:11.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb99cb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:11.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb99cb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:11.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb99cb640 1 -- 172.21.15.60:0/197161979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba4009a70 con 0x7f3bb4071a50 2024-03-27T21:42:11.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bba1cc640 10 monclient: get_auth_request con 0x7f3bb413ec50 auth_method 0 2024-03-27T21:42:11.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb91ca640 1 --2- 172.21.15.60:0/197161979 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3bb413e810 0x7f3bb4136540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:11.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bba1cc640 1 --2- 172.21.15.60:0/197161979 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bb413ec50 0x7f3bb4132140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:42:11.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb99cb640 1 --2- 172.21.15.60:0/197161979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3bb4071a50 0x7f3bb4139a40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3bac000c00 tx=0x7f3bac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:11.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb27fc640 10 client.?.objecter ms_handle_connect 0x7f3bb4071a50 2024-03-27T21:42:11.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.857+0000 7f3bb27fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:11.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bb27fc640 1 -- 172.21.15.60:0/197161979 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac039a90 con 0x7f3bb4071a50 2024-03-27T21:42:11.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:11.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bb27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:11.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bb27fc640 10 monclient: dump: 2024-03-27T21:42:11.883 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:11.883 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:11.883 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.883 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.884 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:11.884 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:11.884 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:11.884 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:11.884 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:11.884 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bb27fc640 10 monclient: _finish_auth 0 2024-03-27T21:42:11.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bb27fc640 10 monclient: _check_auth_tickets 2024-03-27T21:42:11.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:11.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bbbc2f640 5 monclient: authenticate success, global_id 5472 2024-03-27T21:42:11.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bb27fc640 1 -- 172.21.15.60:0/197161979 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bac039bf0 con 0x7f3bb4071a50 2024-03-27T21:42:11.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bb27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:11.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bb27fc640 1 -- 172.21.15.60:0/197161979 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac043410 con 0x7f3bb4071a50 2024-03-27T21:42:11.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:11.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bb27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:11.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bb27fc640 10 monclient: dump: 2024-03-27T21:42:11.885 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:11.886 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:11.886 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.886 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:11.886 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:11.886 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:11.886 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:11.887 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bbbc2f640 10 monclient: _renew_subs 2024-03-27T21:42:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bbbc2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bbbc2f640 1 -- 172.21.15.60:0/197161979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bac0094d0 con 0x7f3bb4071a50 2024-03-27T21:42:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bb37fe640 4 set_mon_vals no callback set 2024-03-27T21:42:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bbbc2f640 10 client.5472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:11.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bbbc2f640 10 monclient: _renew_subs 2024-03-27T21:42:11.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bbbc2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:11.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bbbc2f640 1 -- 172.21.15.60:0/197161979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb4132680 con 0x7f3bb4071a50 2024-03-27T21:42:11.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.858+0000 7f3bbbc2f640 1 librados: init done 2024-03-27T21:42:11.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.859+0000 7f3bbbc2f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:11.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.859+0000 7f3bbbc2f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:11.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.859+0000 7f3bbbc2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:11.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.859+0000 7f3bbbc2f640 1 -- 172.21.15.60:0/197161979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb4067900 con 0x7f3bb4071a50 2024-03-27T21:42:11.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.859+0000 7f3bb27fc640 1 -- 172.21.15.60:0/197161979 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3bac02f390 con 0x7f3bb4071a50 2024-03-27T21:42:11.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.859+0000 7f3bb27fc640 1 --2- 172.21.15.60:0/197161979 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3ba006c590 0x7f3ba006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:11.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.859+0000 7f3bb27fc640 1 -- 172.21.15.60:0/197161979 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f3bac0ab450 con 0x7f3bb4071a50 2024-03-27T21:42:11.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.859+0000 7f3bb27fc640 10 client.5472.objecter ms_dispatch 0x7f3bb4000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:11.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.859+0000 7f3bb27fc640 3 client.5472.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:11.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.859+0000 7f3bb27fc640 3 client.5472.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:11.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.859+0000 7f3bb91ca640 1 --2- 172.21.15.60:0/197161979 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3ba006c590 0x7f3ba006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:11.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.860+0000 7f3bb91ca640 10 monclient: get_auth_request con 0x7f3ba006c590 auth_method 0 2024-03-27T21:42:11.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.860+0000 7f3bb27fc640 20 client.5472.objecter dump_active .. 0 homeless 2024-03-27T21:42:11.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.860+0000 7f3bb27fc640 10 client.5472.objecter ms_handle_connect 0x7f3ba006c590 2024-03-27T21:42:11.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.860+0000 7f3bb91ca640 1 --2- 172.21.15.60:0/197161979 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3ba006c590 0x7f3ba006ea50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3bb41394a0 tx=0x7f3ba8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:11.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.864+0000 7f3bb27fc640 1 -- 172.21.15.60:0/197161979 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3bac0769d0 con 0x7f3bb4071a50 2024-03-27T21:42:11.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.864+0000 7f3bb27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:11.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:11.864+0000 7f3bb27fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:12.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.042+0000 7f3bbbc2f640 1 -- 172.21.15.60:0/197161979 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3bb413f0c0 con 0x7f3ba006c590 2024-03-27T21:42:12.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.042+0000 7f3bb27fc640 1 -- 172.21.15.60:0/197161979 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3ba8001d10 con 0x7f3ba006c590 2024-03-27T21:42:12.029 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 1 to deep-scrub 2024-03-27T21:42:12.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.044+0000 7f3bbbc2f640 10 librados: watch_flush enter 2024-03-27T21:42:12.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.044+0000 7f3bbbc2f640 10 librados: watch_flush exit 2024-03-27T21:42:12.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.044+0000 7f3bbbc2f640 20 client.5472.objecter shutdown clearing up homeless session... 2024-03-27T21:42:12.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.044+0000 7f3bbbc2f640 10 client.5472.objecter successfully canceled tick 2024-03-27T21:42:12.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.044+0000 7f3bbbc2f640 1 -- 172.21.15.60:0/197161979 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3ba006c590 msgr2=0x7f3ba006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.044+0000 7f3bbbc2f640 1 --2- 172.21.15.60:0/197161979 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3ba006c590 0x7f3ba006ea50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3bb41394a0 tx=0x7f3ba8009040 comp rx=0 tx=0).stop 2024-03-27T21:42:12.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.044+0000 7f3bbbc2f640 10 monclient: shutdown 2024-03-27T21:42:12.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.044+0000 7f3bbbc2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:12.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.044+0000 7f3bbbc2f640 1 -- 172.21.15.60:0/197161979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3bb4071a50 msgr2=0x7f3bb4139a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.044+0000 7f3bbbc2f640 1 --2- 172.21.15.60:0/197161979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3bb4071a50 0x7f3bb4139a40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3bac000c00 tx=0x7f3bac004730 comp rx=0 tx=0).stop 2024-03-27T21:42:12.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.044+0000 7f3bbbc2f640 1 -- 172.21.15.60:0/197161979 shutdown_connections 2024-03-27T21:42:12.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.044+0000 7f3bbbc2f640 1 --2- 172.21.15.60:0/197161979 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f3ba006c590 0x7f3ba006ea50 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.044+0000 7f3bbbc2f640 1 --2- 172.21.15.60:0/197161979 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bb413ec50 0x7f3bb4132140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.044+0000 7f3bbbc2f640 1 --2- 172.21.15.60:0/197161979 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f3bb413e810 0x7f3bb4136540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.044+0000 7f3bbbc2f640 1 --2- 172.21.15.60:0/197161979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3bb4071a50 0x7f3bb4139a40 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.044+0000 7f3bbbc2f640 1 -- 172.21.15.60:0/197161979 >> 172.21.15.60:0/197161979 conn(0x7f3bb40b7300 msgr2=0x7f3bb413d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:12.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.044+0000 7f3bbbc2f640 1 -- 172.21.15.60:0/197161979 shutdown_connections 2024-03-27T21:42:12.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.044+0000 7f3bbbc2f640 1 -- 172.21.15.60:0/197161979 wait complete. 2024-03-27T21:42:12.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.044+0000 7f3bbbc2f640 1 librados: shutdown 2024-03-27T21:42:12.041 INFO:tasks.ceph:Scrubbing osd.2 2024-03-27T21:42:12.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-27T21:42:12.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.158+0000 7f9113137640 10 monclient: get_monmap_and_config 2024-03-27T21:42:12.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.158+0000 7f9113137640 10 monclient: build_initial_monmap 2024-03-27T21:42:12.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.158+0000 7f9113137640 10 monclient: monmap: 2024-03-27T21:42:12.145 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:12.145 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:12.146 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:12.159858+0000 2024-03-27T21:42:12.146 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:12.159858+0000 2024-03-27T21:42:12.146 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:12.146 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:12.146 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:12.146 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:12.146 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:12.146 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:12.146 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:12.147 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.159+0000 7f9113137640 1 Processor -- start 2024-03-27T21:42:12.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.159+0000 7f9113137640 1 -- start start 2024-03-27T21:42:12.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.159+0000 7f9113137640 10 monclient: init 2024-03-27T21:42:12.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f9113137640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:12.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f9113137640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-03-27T21:42:12.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f9113137640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f910c0b3980 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:12.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f9113137640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f910c0b3d60 0x7f910c1449c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f9113137640 10 monclient(hunting): picked mon.noname-b con 0x7f910c0b3d60 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:12.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f9113137640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f910c0a86e0 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:12.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f9113137640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f910c144f00 con 0x7f910c0b3980 2024-03-27T21:42:12.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f9113137640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f910c0b07f0 con 0x7f910c0a86e0 2024-03-27T21:42:12.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f9113137640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:12.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f9113137640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f910c0aa1d0 con 0x7f910c0b3d60 2024-03-27T21:42:12.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f9113137640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:12.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f9113137640 10 monclient(hunting): authenticate will time out at 973.381102s 2024-03-27T21:42:12.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f9110ed3640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f910c0b3980 0x7f910c0b0c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60276/0 (socket says 172.21.15.60:60276) 2024-03-27T21:42:12.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f9110ed3640 1 -- 172.21.15.60:0/3028652197 learned_addr learned my addr 172.21.15.60:0/3028652197 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:12.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910bfff640 1 --2- 172.21.15.60:0/3028652197 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f910c0b3d60 0x7f910c1449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910bfff640 10 monclient(hunting): get_auth_request con 0x7f910c0b3d60 auth_method 0 2024-03-27T21:42:12.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:12.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:12.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:12.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:12.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:12.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:12.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910b7fe640 1 -- 172.21.15.60:0/3028652197 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422981014 0 0) 0x7f910c144f00 con 0x7f910c0b3980 2024-03-27T21:42:12.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910b7fe640 10 monclient(hunting): _negotiate 2024-03-27T21:42:12.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:12.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:12.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910b7fe640 10 monclient(hunting): my global_id is 5475 2024-03-27T21:42:12.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910b7fe640 1 -- 172.21.15.60:0/3028652197 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90f4003520 con 0x7f910c0b3980 2024-03-27T21:42:12.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910bfff640 10 monclient(hunting): handle_auth_done global_id 5141 payload 274 2024-03-27T21:42:12.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910bfff640 1 -- 172.21.15.60:0/3028652197 >> v1:172.21.15.191:6789/0 conn(0x7f910c0a86e0 legacy=0x7f910c0b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:12.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910bfff640 1 -- 172.21.15.60:0/3028652197 >> v1:172.21.15.60:6789/0 conn(0x7f910c0b3980 legacy=0x7f910c0b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910bfff640 10 monclient: _finish_hunting 0 2024-03-27T21:42:12.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910bfff640 1 monclient: found mon.noname-b 2024-03-27T21:42:12.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:12.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:42:12.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.160+0000 7f910bfff640 1 -- 172.21.15.60:0/3028652197 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910c0aa4b0 con 0x7f910c0b3d60 2024-03-27T21:42:12.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.161+0000 7f910bfff640 1 --2- 172.21.15.60:0/3028652197 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f910c0b3d60 0x7f910c1449c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f91000097d0 tx=0x7f910002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b2ac29fbbf5865d server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:12.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.161+0000 7f910b7fe640 1 -- 172.21.15.60:0/3028652197 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9100031800 con 0x7f910c0b3d60 2024-03-27T21:42:12.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.161+0000 7f910b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:12.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.161+0000 7f910b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:12.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.161+0000 7f910b7fe640 10 monclient: dump: 2024-03-27T21:42:12.161 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:12.161 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:12.161 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.161 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.162 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:12.162 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:12.162 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:12.162 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:12.162 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:12.162 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.161+0000 7f910b7fe640 10 monclient: _finish_auth 0 2024-03-27T21:42:12.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.161+0000 7f910b7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:42:12.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.161+0000 7f910b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:12.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.161+0000 7f910b7fe640 1 -- 172.21.15.60:0/3028652197 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9100031960 con 0x7f910c0b3d60 2024-03-27T21:42:12.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.161+0000 7f9113137640 5 monclient: authenticate success, global_id 5141 2024-03-27T21:42:12.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.161+0000 7f9113137640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:12.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.161+0000 7f910b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:12.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.161+0000 7f910b7fe640 1 -- 172.21.15.60:0/3028652197 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9100031cd0 con 0x7f910c0b3d60 2024-03-27T21:42:12.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.161+0000 7f9113137640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:12.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.161+0000 7f9113137640 4 set_mon_vals no callback set 2024-03-27T21:42:12.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.161+0000 7f910b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:12.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.161+0000 7f910b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:12.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.161+0000 7f910b7fe640 10 monclient: dump: 2024-03-27T21:42:12.164 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:12.164 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:12.164 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.164 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.164 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:12.164 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:12.165 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:12.165 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:12.165 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:12.165 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.162+0000 7f9113137640 10 monclient: shutdown 2024-03-27T21:42:12.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.162+0000 7f9113137640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:12.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.162+0000 7f9113137640 1 -- 172.21.15.60:0/3028652197 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f910c0b3d60 msgr2=0x7f910c1449c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.162+0000 7f9113137640 1 --2- 172.21.15.60:0/3028652197 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f910c0b3d60 0x7f910c1449c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f91000097d0 tx=0x7f910002b8a0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.162+0000 7f9113137640 1 -- 172.21.15.60:0/3028652197 shutdown_connections 2024-03-27T21:42:12.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.162+0000 7f9113137640 1 --2- 172.21.15.60:0/3028652197 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f910c0b3d60 0x7f910c1449c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.162+0000 7f9113137640 1 -- 172.21.15.60:0/3028652197 >> 172.21.15.60:0/3028652197 conn(0x7f910c071410 msgr2=0x7f910c071820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:12.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.162+0000 7f9113137640 1 -- 172.21.15.60:0/3028652197 shutdown_connections 2024-03-27T21:42:12.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.162+0000 7f9113137640 1 -- 172.21.15.60:0/3028652197 wait complete. 2024-03-27T21:42:12.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.162+0000 7f9113137640 10 monclient: build_initial_monmap 2024-03-27T21:42:12.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.162+0000 7f9113137640 10 monclient: monmap: 2024-03-27T21:42:12.166 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:12.166 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:12.167 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:12.167 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:12.167 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:12.167 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:12.167 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:12.167 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:12.167 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:12.167 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 1 librados: starting msgr at 2024-03-27T21:42:12.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 1 librados: starting objecter 2024-03-27T21:42:12.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 1 Processor -- start 2024-03-27T21:42:12.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 1 -- start start 2024-03-27T21:42:12.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 1 librados: setting wanted keys 2024-03-27T21:42:12.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 1 librados: calling monclient init 2024-03-27T21:42:12.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 10 monclient: init 2024-03-27T21:42:12.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:12.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:42:12.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f910c0a86e0 0x7f910c13e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 10 monclient(hunting): picked mon.noname-b con 0x7f910c0a86e0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:12.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f910c0b3980 0x7f910c0ad900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 10 monclient(hunting): picked mon.noname-c con 0x7f910c0b3980 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:12.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f910c0acf30 0x7f910c0af2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 10 monclient(hunting): picked mon.noname-a con 0x7f910c0acf30 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:12.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:12.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c0af830 con 0x7f910c0acf30 2024-03-27T21:42:12.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:12.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c0ad470 con 0x7f910c0a86e0 2024-03-27T21:42:12.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:12.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f910c0abcc0 con 0x7f910c0b3980 2024-03-27T21:42:12.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:12.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f9113137640 10 monclient(hunting): authenticate will time out at 973.384800s 2024-03-27T21:42:12.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f910bfff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f910c0b3980 0x7f910c0ad900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f910bfff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f910c0b3980 0x7f910c0ad900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:51436/0 (socket says 172.21.15.60:51436) 2024-03-27T21:42:12.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f910bfff640 1 -- 172.21.15.60:0/1807414561 learned_addr learned my addr 172.21.15.60:0/1807414561 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:12.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f910bfff640 10 monclient(hunting): get_auth_request con 0x7f910c0b3980 auth_method 0 2024-03-27T21:42:12.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f910bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:12.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f910bfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:12.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f910bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:12.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.163+0000 7f91116d4640 1 --2- 172.21.15.60:0/1807414561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f910c0acf30 0x7f910c0af2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91116d4640 10 monclient(hunting): get_auth_request con 0x7f910c0acf30 auth_method 0 2024-03-27T21:42:12.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91116d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:12.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91116d4640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:12.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91116d4640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:12.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f910bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:12.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f910bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:12.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f910bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:12.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f9110ed3640 1 --2- 172.21.15.60:0/1807414561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f910c0a86e0 0x7f910c13e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f9110ed3640 10 monclient(hunting): get_auth_request con 0x7f910c0a86e0 auth_method 0 2024-03-27T21:42:12.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f9110ed3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:12.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f9110ed3640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:12.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f9110ed3640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:12.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91116d4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:12.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91116d4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:12.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91116d4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:12.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f9110ed3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:12.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f9110ed3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:12.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f9110ed3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:12.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f910bfff640 10 monclient(hunting): handle_auth_done global_id 5144 payload 824 2024-03-27T21:42:12.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f910bfff640 1 -- 172.21.15.60:0/1807414561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f910c0a86e0 msgr2=0x7f910c13e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f910bfff640 1 --2- 172.21.15.60:0/1807414561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f910c0a86e0 0x7f910c13e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f910bfff640 1 -- 172.21.15.60:0/1807414561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f910c0acf30 msgr2=0x7f910c0af2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f910bfff640 1 --2- 172.21.15.60:0/1807414561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f910c0acf30 0x7f910c0af2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f910bfff640 10 monclient: _finish_hunting 0 2024-03-27T21:42:12.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f910bfff640 1 monclient: found mon.noname-c 2024-03-27T21:42:12.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f910bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:12.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f910bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:12.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f910bfff640 1 -- 172.21.15.60:0/1807414561 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9100009400 con 0x7f910c0b3980 2024-03-27T21:42:12.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91116d4640 1 --2- 172.21.15.60:0/1807414561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f910c0acf30 0x7f910c0af2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:12.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f910bfff640 1 --2- 172.21.15.60:0/1807414561 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f910c0b3980 0x7f910c0ad900 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9100000c00 tx=0x7f91000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:12.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91097fa640 10 client.?.objecter ms_handle_connect 0x7f910c0b3980 2024-03-27T21:42:12.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91097fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:12.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91097fa640 1 -- 172.21.15.60:0/1807414561 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910003bc70 con 0x7f910c0b3980 2024-03-27T21:42:12.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:12.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91097fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:12.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91097fa640 10 monclient: dump: 2024-03-27T21:42:12.175 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:12.175 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:12.175 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.176 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.176 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:12.176 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:12.176 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:12.176 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:12.176 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:12.176 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91097fa640 10 monclient: _finish_auth 0 2024-03-27T21:42:12.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91097fa640 10 monclient: _check_auth_tickets 2024-03-27T21:42:12.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:12.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f9113137640 5 monclient: authenticate success, global_id 5144 2024-03-27T21:42:12.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91097fa640 1 -- 172.21.15.60:0/1807414561 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f910003bdd0 con 0x7f910c0b3980 2024-03-27T21:42:12.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f9113137640 10 monclient: _renew_subs 2024-03-27T21:42:12.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f9113137640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:12.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f9113137640 1 -- 172.21.15.60:0/1807414561 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f910c1d9760 con 0x7f910c0b3980 2024-03-27T21:42:12.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91097fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:12.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91097fa640 1 -- 172.21.15.60:0/1807414561 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9100045610 con 0x7f910c0b3980 2024-03-27T21:42:12.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:12.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:12.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f91097fa640 10 monclient: dump: 2024-03-27T21:42:12.178 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:12.178 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:12.178 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.178 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.178 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:12.178 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:12.179 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:12.179 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:12.179 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:12.179 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f910affd640 4 set_mon_vals no callback set 2024-03-27T21:42:12.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f9113137640 10 client.5144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:12.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f9113137640 10 monclient: _renew_subs 2024-03-27T21:42:12.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f9113137640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:12.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f9113137640 1 -- 172.21.15.60:0/1807414561 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f910c1d9bd0 con 0x7f910c0b3980 2024-03-27T21:42:12.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.164+0000 7f9113137640 1 librados: init done 2024-03-27T21:42:12.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.165+0000 7f9113137640 10 client.5144.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:12.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.165+0000 7f9113137640 15 client.5144.objecter _session_command_op_assign -1 1 2024-03-27T21:42:12.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.165+0000 7f9113137640 20 client.5144.objecter _get_session osd=-1 returning homeless 2024-03-27T21:42:12.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.165+0000 7f9113137640 10 client.5144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:12.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.165+0000 7f9113137640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:12.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.165+0000 7f9113137640 1 -- 172.21.15.60:0/1807414561 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f910c14cd60 con 0x7f910c0b3980 2024-03-27T21:42:12.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91097fa640 1 -- 172.21.15.60:0/1807414561 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f910004e450 con 0x7f910c0b3980 2024-03-27T21:42:12.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91097fa640 1 --2- 172.21.15.60:0/1807414561 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f90f806c5c0 0x7f90f806ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f9110ed3640 1 --2- 172.21.15.60:0/1807414561 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f90f806c5c0 0x7f90f806ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f9110ed3640 10 monclient: get_auth_request con 0x7f90f806c5c0 auth_method 0 2024-03-27T21:42:12.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91097fa640 1 -- 172.21.15.60:0/1807414561 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f910002e070 con 0x7f910c0b3980 2024-03-27T21:42:12.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91097fa640 10 client.5144.objecter ms_dispatch 0x7f910c000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:12.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91097fa640 3 client.5144.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:12.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91097fa640 3 client.5144.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:12.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91097fa640 10 client.5144.objecter checking command 1 2024-03-27T21:42:12.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91097fa640 1 --2- 172.21.15.60:0/1807414561 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f90f8072a00 0x7f90f8074e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91097fa640 20 client.5144.objecter _get_session s=0x7f90f8072330 osd=2 3 2024-03-27T21:42:12.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91097fa640 20 client.5144.objecter put_session s=0x7f90f8072330 osd=2 3 2024-03-27T21:42:12.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91097fa640 15 client.5144.objecter _session_command_op_remove -1 1 2024-03-27T21:42:12.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91097fa640 20 client.5144.objecter _get_session s=0x7f90f8072330 osd=2 3 2024-03-27T21:42:12.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91097fa640 20 client.5144.objecter get_session s=0x7f90f8072330 osd=2 3 2024-03-27T21:42:12.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91097fa640 15 client.5144.objecter _session_command_op_assign 2 1 2024-03-27T21:42:12.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91097fa640 20 client.5144.objecter put_session s=0x7f90f8072330 osd=2 4 2024-03-27T21:42:12.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91097fa640 10 client.5144.objecter _send_command 1 2024-03-27T21:42:12.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91097fa640 1 -- 172.21.15.60:0/1807414561 --> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90f80754f0 con 0x7f90f8072a00 2024-03-27T21:42:12.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91097fa640 20 client.5144.objecter dump_active .. 0 homeless 2024-03-27T21:42:12.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91097fa640 1 -- 172.21.15.60:0/1807414561 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91000ad830 con 0x7f910c0b3980 2024-03-27T21:42:12.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91097fa640 10 monclient: handle_get_version_reply finishing 1 version 92 2024-03-27T21:42:12.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91116d4640 1 --2- 172.21.15.60:0/1807414561 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f90f8072a00 0x7f90f8074e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91116d4640 10 monclient: get_auth_request con 0x7f90f8072a00 auth_method 0 2024-03-27T21:42:12.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f91097fa640 10 client.5144.objecter ms_handle_connect 0x7f90f806c5c0 2024-03-27T21:42:12.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.166+0000 7f9110ed3640 1 --2- 172.21.15.60:0/1807414561 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f90f806c5c0 0x7f90f806ea80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9104005c00 tx=0x7f9104009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:12.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.167+0000 7f91097fa640 10 client.5144.objecter ms_handle_connect 0x7f90f8072a00 2024-03-27T21:42:12.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.167+0000 7f91116d4640 1 --2- 172.21.15.60:0/1807414561 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f90f8072a00 0x7f90f8074e20 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:12.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.167+0000 7f91097fa640 1 -- 172.21.15.60:0/1807414561 <== osd.2 v2:172.21.15.60:6824/56434 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f90f80754f0 con 0x7f90f8072a00 2024-03-27T21:42:12.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.167+0000 7f91097fa640 10 client.5144.objecter ms_dispatch 0x7f910c000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:42:12.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.167+0000 7f91097fa640 10 client.5144.objecter _finish_command 1 = system:0 2024-03-27T21:42:12.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.167+0000 7f91097fa640 20 client.5144.objecter put_session s=0x7f90f8072330 osd=2 4 2024-03-27T21:42:12.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.167+0000 7f91097fa640 15 client.5144.objecter _session_command_op_remove 2 1 2024-03-27T21:42:12.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 10 client.5144.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-27T21:42:12.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 15 client.5144.objecter _session_command_op_assign -1 2 2024-03-27T21:42:12.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 20 client.5144.objecter _get_session s=0x7f90f8072330 osd=2 3 2024-03-27T21:42:12.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 20 client.5144.objecter put_session s=0x7f90f8072330 osd=2 3 2024-03-27T21:42:12.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 20 client.5144.objecter _get_session s=0x7f90f8072330 osd=2 3 2024-03-27T21:42:12.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 15 client.5144.objecter _session_command_op_remove -1 2 2024-03-27T21:42:12.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 20 client.5144.objecter get_session s=0x7f90f8072330 osd=2 3 2024-03-27T21:42:12.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 15 client.5144.objecter _session_command_op_assign 2 2 2024-03-27T21:42:12.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 20 client.5144.objecter put_session s=0x7f90f8072330 osd=2 4 2024-03-27T21:42:12.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 10 client.5144.objecter _send_command 2 2024-03-27T21:42:12.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 1 -- 172.21.15.60:0/1807414561 --> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f910c0670d0 con 0x7f90f8072a00 2024-03-27T21:42:12.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f91097fa640 1 -- 172.21.15.60:0/1807414561 <== osd.2 v2:172.21.15.60:6824/56434 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f910c0670d0 con 0x7f90f8072a00 2024-03-27T21:42:12.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f91097fa640 10 client.5144.objecter ms_dispatch 0x7f910c000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:42:12.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f91097fa640 10 client.5144.objecter _finish_command 2 = system:0 2024-03-27T21:42:12.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f91097fa640 20 client.5144.objecter put_session s=0x7f90f8072330 osd=2 4 2024-03-27T21:42:12.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f91097fa640 15 client.5144.objecter _session_command_op_remove 2 2 2024-03-27T21:42:12.187 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:42:12.187 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-27T21:42:12.187 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:42:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 10 librados: watch_flush enter 2024-03-27T21:42:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 10 librados: watch_flush exit 2024-03-27T21:42:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 10 client.5144.objecter close_session for osd.2 2024-03-27T21:42:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 1 -- 172.21.15.60:0/1807414561 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f90f8072a00 msgr2=0x7f90f8074e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 1 --2- 172.21.15.60:0/1807414561 >> [v2:172.21.15.60:6824/56434,v1:172.21.15.60:6825/56434] conn(0x7f90f8072a00 0x7f90f8074e20 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 20 client.5144.objecter put_session s=0x7f90f8072330 osd=2 1 2024-03-27T21:42:12.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 20 client.5144.objecter shutdown clearing up homeless session... 2024-03-27T21:42:12.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 10 client.5144.objecter successfully canceled tick 2024-03-27T21:42:12.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 1 -- 172.21.15.60:0/1807414561 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f90f806c5c0 msgr2=0x7f90f806ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 1 --2- 172.21.15.60:0/1807414561 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f90f806c5c0 0x7f90f806ea80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9104005c00 tx=0x7f9104009040 comp rx=0 tx=0).stop 2024-03-27T21:42:12.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 10 monclient: shutdown 2024-03-27T21:42:12.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:12.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 1 -- 172.21.15.60:0/1807414561 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f910c0b3980 msgr2=0x7f910c0ad900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 1 --2- 172.21.15.60:0/1807414561 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f910c0b3980 0x7f910c0ad900 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9100000c00 tx=0x7f91000046c0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f91116d4640 1 -- 172.21.15.60:0/1807414561 reap_dead start 2024-03-27T21:42:12.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 1 -- 172.21.15.60:0/1807414561 shutdown_connections 2024-03-27T21:42:12.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 1 -- 172.21.15.60:0/1807414561 >> 172.21.15.60:0/1807414561 conn(0x7f910c071410 msgr2=0x7f910c138540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:12.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 1 -- 172.21.15.60:0/1807414561 shutdown_connections 2024-03-27T21:42:12.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 1 -- 172.21.15.60:0/1807414561 wait complete. 2024-03-27T21:42:12.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.180+0000 7f9113137640 1 librados: shutdown 2024-03-27T21:42:12.191 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-27T21:42:12.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.305+0000 7fea17bf0640 10 monclient: get_monmap_and_config 2024-03-27T21:42:12.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.305+0000 7fea17bf0640 10 monclient: build_initial_monmap 2024-03-27T21:42:12.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.305+0000 7fea17bf0640 10 monclient: monmap: 2024-03-27T21:42:12.293 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:12.293 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:12.293 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:12.306954+0000 2024-03-27T21:42:12.293 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:12.306954+0000 2024-03-27T21:42:12.293 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:12.293 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:12.293 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:12.293 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:12.293 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:12.294 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:12.294 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:12.294 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.306+0000 7fea17bf0640 1 Processor -- start 2024-03-27T21:42:12.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.306+0000 7fea17bf0640 1 -- start start 2024-03-27T21:42:12.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.306+0000 7fea17bf0640 10 monclient: init 2024-03-27T21:42:12.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea17bf0640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:12.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea17bf0640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-03-27T21:42:12.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea17bf0640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fea10071a50 0x7fea100a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea17bf0640 10 monclient(hunting): picked mon.noname-c con 0x7fea10071a50 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:12.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea17bf0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fea1013e810 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:12.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea17bf0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fea1013ec50 0x7fea10141d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea17bf0640 10 monclient(hunting): picked mon.noname-a con 0x7fea1013ec50 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:12.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea17bf0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea100a8e90 con 0x7fea1013e810 2024-03-27T21:42:12.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea17bf0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:12.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea17bf0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea10071e30 con 0x7fea1013ec50 2024-03-27T21:42:12.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea17bf0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:12.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea17bf0640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea100aa040 con 0x7fea10071a50 2024-03-27T21:42:12.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea17bf0640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:12.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea17bf0640 10 monclient(hunting): authenticate will time out at 973.528223s 2024-03-27T21:42:12.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1618d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fea1013ec50 0x7fea10141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1618d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fea1013ec50 0x7fea10141d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59268/0 (socket says 172.21.15.60:59268) 2024-03-27T21:42:12.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1618d640 1 -- 172.21.15.60:0/3760535461 learned_addr learned my addr 172.21.15.60:0/3760535461 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:12.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1618d640 10 monclient(hunting): get_auth_request con 0x7fea1013ec50 auth_method 0 2024-03-27T21:42:12.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1618d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:12.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1618d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:12.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1618d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:12.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1618d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:12.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1618d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:12.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1618d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:12.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1598c640 1 --2- 172.21.15.60:0/3760535461 >> v2:172.21.15.191:3300/0 conn(0x7fea10071a50 0x7fea100a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1598c640 10 monclient(hunting): get_auth_request con 0x7fea10071a50 auth_method 0 2024-03-27T21:42:12.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1598c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:12.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1598c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:12.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1598c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:12.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1618d640 10 monclient(hunting): handle_auth_done global_id 5481 payload 274 2024-03-27T21:42:12.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1618d640 1 -- 172.21.15.60:0/3760535461 >> v2:172.21.15.191:3300/0 conn(0x7fea10071a50 msgr2=0x7fea100a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:12.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1618d640 1 --2- 172.21.15.60:0/3760535461 >> v2:172.21.15.191:3300/0 conn(0x7fea10071a50 0x7fea100a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1618d640 1 -- 172.21.15.60:0/3760535461 >> v1:172.21.15.60:6789/0 conn(0x7fea1013e810 legacy=0x7fea10141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1618d640 10 monclient: _finish_hunting 0 2024-03-27T21:42:12.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1618d640 1 monclient: found mon.noname-a 2024-03-27T21:42:12.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1618d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:12.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1618d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:12.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1618d640 1 -- 172.21.15.60:0/3760535461 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea10142280 con 0x7fea1013ec50 2024-03-27T21:42:12.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.307+0000 7fea1598c640 1 --2- 172.21.15.60:0/3760535461 >> v2:172.21.15.191:3300/0 conn(0x7fea10071a50 0x7fea100a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:42:12.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.308+0000 7fea1618d640 1 --2- 172.21.15.60:0/3760535461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea1013ec50 0x7fea10141d40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fea100b4a80 tx=0x7fea0c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca7ac3b9d9a821f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:12.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.308+0000 7fea1498a640 1 -- 172.21.15.60:0/3760535461 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c030860 con 0x7fea1013ec50 2024-03-27T21:42:12.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.308+0000 7fea1498a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:12.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.308+0000 7fea1498a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:12.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.308+0000 7fea1498a640 10 monclient: dump: 2024-03-27T21:42:12.309 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:12.310 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:12.310 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.310 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.310 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:12.310 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:12.310 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:12.310 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:12.310 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:12.310 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.308+0000 7fea1498a640 10 monclient: _finish_auth 0 2024-03-27T21:42:12.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.308+0000 7fea1498a640 10 monclient: _check_auth_tickets 2024-03-27T21:42:12.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.308+0000 7fea1498a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:12.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.308+0000 7fea1498a640 1 -- 172.21.15.60:0/3760535461 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0c0309c0 con 0x7fea1013ec50 2024-03-27T21:42:12.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.308+0000 7fea17bf0640 5 monclient: authenticate success, global_id 5481 2024-03-27T21:42:12.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.308+0000 7fea17bf0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:12.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.308+0000 7fea1498a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:12.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.308+0000 7fea1498a640 1 -- 172.21.15.60:0/3760535461 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c030d50 con 0x7fea1013ec50 2024-03-27T21:42:12.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.308+0000 7fea17bf0640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:12.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.308+0000 7fea17bf0640 4 set_mon_vals no callback set 2024-03-27T21:42:12.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.308+0000 7fea1498a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:12.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.308+0000 7fea1498a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:12.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.308+0000 7fea1498a640 10 monclient: dump: 2024-03-27T21:42:12.312 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:12.312 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:12.312 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.312 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.312 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:12.313 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:12.313 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:12.313 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:12.313 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:12.313 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.309+0000 7fea17bf0640 10 monclient: shutdown 2024-03-27T21:42:12.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.309+0000 7fea17bf0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:12.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.309+0000 7fea17bf0640 1 -- 172.21.15.60:0/3760535461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea1013ec50 msgr2=0x7fea10141d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.309+0000 7fea17bf0640 1 --2- 172.21.15.60:0/3760535461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea1013ec50 0x7fea10141d40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fea100b4a80 tx=0x7fea0c02b9a0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.309+0000 7fea17bf0640 1 -- 172.21.15.60:0/3760535461 shutdown_connections 2024-03-27T21:42:12.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.309+0000 7fea17bf0640 1 --2- 172.21.15.60:0/3760535461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea1013ec50 0x7fea10141d40 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.309+0000 7fea17bf0640 1 --2- 172.21.15.60:0/3760535461 >> v2:172.21.15.191:3300/0 conn(0x7fea10071a50 0x7fea100a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.309+0000 7fea17bf0640 1 -- 172.21.15.60:0/3760535461 >> 172.21.15.60:0/3760535461 conn(0x7fea100b7300 msgr2=0x7fea100b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:12.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.309+0000 7fea17bf0640 1 -- 172.21.15.60:0/3760535461 shutdown_connections 2024-03-27T21:42:12.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.309+0000 7fea17bf0640 1 -- 172.21.15.60:0/3760535461 wait complete. 2024-03-27T21:42:12.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.309+0000 7fea17bf0640 10 monclient: build_initial_monmap 2024-03-27T21:42:12.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.309+0000 7fea17bf0640 10 monclient: monmap: 2024-03-27T21:42:12.314 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:12.315 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:12.315 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:12.315 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:12.315 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:12.315 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:12.315 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:12.315 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:12.315 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:12.315 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 1 librados: starting msgr at 2024-03-27T21:42:12.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 1 librados: starting objecter 2024-03-27T21:42:12.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 1 Processor -- start 2024-03-27T21:42:12.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 1 -- start start 2024-03-27T21:42:12.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 1 librados: setting wanted keys 2024-03-27T21:42:12.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 1 librados: calling monclient init 2024-03-27T21:42:12.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 10 monclient: init 2024-03-27T21:42:12.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:12.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:42:12.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea10071a50 0x7fea101d6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 10 monclient(hunting): picked mon.noname-a con 0x7fea10071a50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:12.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fea1013e810 0x7fea1013a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 10 monclient(hunting): picked mon.noname-c con 0x7fea1013e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:12.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea1013ec50 0x7fea1013a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 10 monclient(hunting): picked mon.noname-b con 0x7fea1013ec50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:12.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:12.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1013ad90 con 0x7fea10071a50 2024-03-27T21:42:12.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:12.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea10139e80 con 0x7fea1013ec50 2024-03-27T21:42:12.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:12.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea10138ca0 con 0x7fea1013e810 2024-03-27T21:42:12.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:12.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea17bf0640 10 monclient(hunting): authenticate will time out at 973.531672s 2024-03-27T21:42:12.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea1598c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea10071a50 0x7fea101d6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea1598c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea10071a50 0x7fea101d6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59270/0 (socket says 172.21.15.60:59270) 2024-03-27T21:42:12.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea1598c640 1 -- 172.21.15.60:0/2793173960 learned_addr learned my addr 172.21.15.60:0/2793173960 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:12.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea1598c640 10 monclient(hunting): get_auth_request con 0x7fea10071a50 auth_method 0 2024-03-27T21:42:12.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea1598c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:12.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea1598c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:12.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea1598c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:12.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea1518b640 1 --2- 172.21.15.60:0/2793173960 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fea1013e810 0x7fea1013a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea1518b640 10 monclient(hunting): get_auth_request con 0x7fea1013e810 auth_method 0 2024-03-27T21:42:12.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea1518b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:12.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea1518b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:12.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea1518b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:12.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea1598c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:12.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea1598c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:12.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea1598c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:12.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea1518b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:12.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea1518b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:12.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.310+0000 7fea1518b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:12.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea1598c640 10 monclient(hunting): handle_auth_done global_id 5487 payload 824 2024-03-27T21:42:12.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea1618d640 1 --2- 172.21.15.60:0/2793173960 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea1013ec50 0x7fea1013a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea1598c640 1 -- 172.21.15.60:0/2793173960 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fea1013e810 msgr2=0x7fea1013a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea1598c640 1 --2- 172.21.15.60:0/2793173960 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fea1013e810 0x7fea1013a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea1598c640 1 -- 172.21.15.60:0/2793173960 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea1013ec50 msgr2=0x7fea1013a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea1598c640 1 --2- 172.21.15.60:0/2793173960 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea1013ec50 0x7fea1013a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea1598c640 10 monclient: _finish_hunting 0 2024-03-27T21:42:12.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea1598c640 1 monclient: found mon.noname-a 2024-03-27T21:42:12.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea1598c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:12.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea1598c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:12.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea1598c640 1 -- 172.21.15.60:0/2793173960 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea0c0094d0 con 0x7fea10071a50 2024-03-27T21:42:12.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea1598c640 1 --2- 172.21.15.60:0/2793173960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea10071a50 0x7fea101d6690 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fea08009fc0 tx=0x7fea0800d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:12.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea067fc640 10 client.?.objecter ms_handle_connect 0x7fea10071a50 2024-03-27T21:42:12.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea067fc640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:12.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea067fc640 1 -- 172.21.15.60:0/2793173960 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea08003b90 con 0x7fea10071a50 2024-03-27T21:42:12.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:12.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:12.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea067fc640 10 monclient: dump: 2024-03-27T21:42:12.322 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:12.322 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:12.323 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.323 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.323 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:12.323 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:12.323 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:12.323 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:12.323 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:12.323 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea067fc640 10 monclient: _finish_auth 0 2024-03-27T21:42:12.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea067fc640 10 monclient: _check_auth_tickets 2024-03-27T21:42:12.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:12.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea17bf0640 5 monclient: authenticate success, global_id 5487 2024-03-27T21:42:12.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea067fc640 1 -- 172.21.15.60:0/2793173960 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0800f040 con 0x7fea10071a50 2024-03-27T21:42:12.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:12.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea067fc640 1 -- 172.21.15.60:0/2793173960 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea08004030 con 0x7fea10071a50 2024-03-27T21:42:12.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea17bf0640 10 monclient: _renew_subs 2024-03-27T21:42:12.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea17bf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:12.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea17bf0640 1 -- 172.21.15.60:0/2793173960 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea10137430 con 0x7fea10071a50 2024-03-27T21:42:12.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea07fff640 4 set_mon_vals no callback set 2024-03-27T21:42:12.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:12.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:12.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea067fc640 10 monclient: dump: 2024-03-27T21:42:12.325 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:12.325 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:12.325 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.325 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.325 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:12.326 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:12.326 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:12.326 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:12.326 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:12.326 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea17bf0640 10 client.5487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:12.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea17bf0640 10 monclient: _renew_subs 2024-03-27T21:42:12.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea17bf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:12.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea17bf0640 1 -- 172.21.15.60:0/2793173960 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea10139a50 con 0x7fea10071a50 2024-03-27T21:42:12.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.311+0000 7fea17bf0640 1 librados: init done 2024-03-27T21:42:12.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.312+0000 7fea17bf0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:12.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.312+0000 7fea17bf0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:12.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.312+0000 7fea17bf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:12.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.312+0000 7fea17bf0640 1 -- 172.21.15.60:0/2793173960 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea10069470 con 0x7fea10071a50 2024-03-27T21:42:12.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.312+0000 7fea067fc640 1 -- 172.21.15.60:0/2793173960 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fea08004190 con 0x7fea10071a50 2024-03-27T21:42:12.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.313+0000 7fea067fc640 1 --2- 172.21.15.60:0/2793173960 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe9f406c590 0x7fe9f406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.313+0000 7fea067fc640 1 -- 172.21.15.60:0/2793173960 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fea08089410 con 0x7fea10071a50 2024-03-27T21:42:12.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.313+0000 7fea067fc640 10 client.5487.objecter ms_dispatch 0x7fea10000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:12.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.313+0000 7fea067fc640 3 client.5487.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:12.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.313+0000 7fea067fc640 3 client.5487.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:12.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.313+0000 7fea1518b640 1 --2- 172.21.15.60:0/2793173960 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe9f406c590 0x7fe9f406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.313+0000 7fea1518b640 10 monclient: get_auth_request con 0x7fe9f406c590 auth_method 0 2024-03-27T21:42:12.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.313+0000 7fea067fc640 20 client.5487.objecter dump_active .. 0 homeless 2024-03-27T21:42:12.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.313+0000 7fea067fc640 10 client.5487.objecter ms_handle_connect 0x7fe9f406c590 2024-03-27T21:42:12.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.313+0000 7fea1518b640 1 --2- 172.21.15.60:0/2793173960 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe9f406c590 0x7fe9f406ea50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fea100ab0e0 tx=0x7fea00009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:12.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.317+0000 7fea067fc640 1 -- 172.21.15.60:0/2793173960 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fea08054900 con 0x7fea10071a50 2024-03-27T21:42:12.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.317+0000 7fea067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:12.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.317+0000 7fea067fc640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:12.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.497+0000 7fea17bf0640 1 -- 172.21.15.60:0/2793173960 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fea1013f0c0 con 0x7fe9f406c590 2024-03-27T21:42:12.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.497+0000 7fea067fc640 1 -- 172.21.15.60:0/2793173960 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fea000014a0 con 0x7fe9f406c590 2024-03-27T21:42:12.484 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 2 to deep-scrub 2024-03-27T21:42:12.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.499+0000 7fea17bf0640 10 librados: watch_flush enter 2024-03-27T21:42:12.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.499+0000 7fea17bf0640 10 librados: watch_flush exit 2024-03-27T21:42:12.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.499+0000 7fea17bf0640 20 client.5487.objecter shutdown clearing up homeless session... 2024-03-27T21:42:12.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.499+0000 7fea17bf0640 10 client.5487.objecter successfully canceled tick 2024-03-27T21:42:12.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.499+0000 7fea17bf0640 1 -- 172.21.15.60:0/2793173960 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe9f406c590 msgr2=0x7fe9f406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.499+0000 7fea17bf0640 1 --2- 172.21.15.60:0/2793173960 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe9f406c590 0x7fe9f406ea50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fea100ab0e0 tx=0x7fea00009040 comp rx=0 tx=0).stop 2024-03-27T21:42:12.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.499+0000 7fea17bf0640 10 monclient: shutdown 2024-03-27T21:42:12.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.499+0000 7fea17bf0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:12.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.499+0000 7fea17bf0640 1 -- 172.21.15.60:0/2793173960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea10071a50 msgr2=0x7fea101d6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.499+0000 7fea17bf0640 1 --2- 172.21.15.60:0/2793173960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea10071a50 0x7fea101d6690 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fea08009fc0 tx=0x7fea0800d560 comp rx=0 tx=0).stop 2024-03-27T21:42:12.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.500+0000 7fea17bf0640 1 -- 172.21.15.60:0/2793173960 shutdown_connections 2024-03-27T21:42:12.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.500+0000 7fea17bf0640 1 --2- 172.21.15.60:0/2793173960 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fe9f406c590 0x7fe9f406ea50 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.500+0000 7fea17bf0640 1 --2- 172.21.15.60:0/2793173960 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea1013ec50 0x7fea1013a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.500+0000 7fea17bf0640 1 --2- 172.21.15.60:0/2793173960 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fea1013e810 0x7fea1013a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.500+0000 7fea17bf0640 1 --2- 172.21.15.60:0/2793173960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea10071a50 0x7fea101d6690 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.500+0000 7fea17bf0640 1 -- 172.21.15.60:0/2793173960 >> 172.21.15.60:0/2793173960 conn(0x7fea100b7300 msgr2=0x7fea1013b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:12.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.500+0000 7fea17bf0640 1 -- 172.21.15.60:0/2793173960 shutdown_connections 2024-03-27T21:42:12.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.500+0000 7fea17bf0640 1 -- 172.21.15.60:0/2793173960 wait complete. 2024-03-27T21:42:12.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.500+0000 7fea17bf0640 1 librados: shutdown 2024-03-27T21:42:12.496 INFO:tasks.ceph:Scrubbing osd.3 2024-03-27T21:42:12.497 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-27T21:42:12.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.614+0000 7f2895f8b640 10 monclient: get_monmap_and_config 2024-03-27T21:42:12.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.614+0000 7f2895f8b640 10 monclient: build_initial_monmap 2024-03-27T21:42:12.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.614+0000 7f2895f8b640 10 monclient: monmap: 2024-03-27T21:42:12.601 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:12.601 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:12.601 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:12.615665+0000 2024-03-27T21:42:12.601 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:12.615665+0000 2024-03-27T21:42:12.601 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:12.601 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:12.601 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:12.602 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:12.602 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:12.602 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:12.602 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:12.602 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.615+0000 7f2895f8b640 1 Processor -- start 2024-03-27T21:42:12.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.615+0000 7f2895f8b640 1 -- start start 2024-03-27T21:42:12.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.615+0000 7f2895f8b640 10 monclient: init 2024-03-27T21:42:12.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.615+0000 7f2895f8b640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:12.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.615+0000 7f2895f8b640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-03-27T21:42:12.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.615+0000 7f2895f8b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f28900b3980 0x7f28901449c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.615+0000 7f2895f8b640 10 monclient(hunting): picked mon.noname-a con 0x7f28900b3980 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:12.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.615+0000 7f2895f8b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f28900b3d60 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:12.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.615+0000 7f2895f8b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28900a86e0 0x7f28900b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.615+0000 7f2895f8b640 10 monclient(hunting): picked mon.noname-b con 0x7f28900a86e0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:12.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.615+0000 7f2895f8b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2890144f00 con 0x7f28900b3d60 2024-03-27T21:42:12.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f2895f8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:12.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f2895f8b640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28900aa1d0 con 0x7f28900b3980 2024-03-27T21:42:12.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f2895f8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:12.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f2895f8b640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28900aa4b0 con 0x7f28900a86e0 2024-03-27T21:42:12.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f2895f8b640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:12.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f2895f8b640 10 monclient(hunting): authenticate will time out at 973.836940s 2024-03-27T21:42:12.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f28900b3980 0x7f28901449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288ffff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28900a86e0 0x7f28900b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f28900b3980 0x7f28901449c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59282/0 (socket says 172.21.15.60:59282) 2024-03-27T21:42:12.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288ffff640 10 monclient(hunting): get_auth_request con 0x7f28900a86e0 auth_method 0 2024-03-27T21:42:12.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288f7fe640 1 -- 172.21.15.60:0/2657858068 learned_addr learned my addr 172.21.15.60:0/2657858068 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:12.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:12.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:12.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:12.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288f7fe640 10 monclient(hunting): get_auth_request con 0x7f28900b3980 auth_method 0 2024-03-27T21:42:12.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:12.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:12.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:12.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:12.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:12.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:12.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:12.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:12.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:12.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288ffff640 10 monclient(hunting): handle_auth_done global_id 5150 payload 274 2024-03-27T21:42:12.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288ffff640 1 -- 172.21.15.60:0/2657858068 >> v2:172.21.15.60:3300/0 conn(0x7f28900b3980 msgr2=0x7f28901449c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:12.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288ffff640 1 --2- 172.21.15.60:0/2657858068 >> v2:172.21.15.60:3300/0 conn(0x7f28900b3980 0x7f28901449c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288ffff640 1 -- 172.21.15.60:0/2657858068 >> v1:172.21.15.191:6789/0 conn(0x7f28900b3d60 legacy=0x7f28900b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288ffff640 10 monclient: _finish_hunting 0 2024-03-27T21:42:12.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288ffff640 1 monclient: found mon.noname-b 2024-03-27T21:42:12.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:12.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:42:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288ffff640 1 -- 172.21.15.60:0/2657858068 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28900b13b0 con 0x7f28900a86e0 2024-03-27T21:42:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.616+0000 7f288f7fe640 1 --2- 172.21.15.60:0/2657858068 >> v2:172.21.15.60:3300/0 conn(0x7f28900b3980 0x7f28901449c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f288ffff640 1 --2- 172.21.15.60:0/2657858068 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28900a86e0 0x7f28900b00e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2888009d90 tx=0x7f288802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42f879f3823c262e server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f288e7fc640 1 -- 172.21.15.60:0/2657858068 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288802cc50 con 0x7f28900a86e0 2024-03-27T21:42:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f288e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f288e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f288e7fc640 10 monclient: dump: 2024-03-27T21:42:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:12.618 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:12.618 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.618 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.618 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:12.618 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:12.618 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:12.618 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:12.619 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:12.619 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f288e7fc640 10 monclient: _finish_auth 0 2024-03-27T21:42:12.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f288e7fc640 10 monclient: _check_auth_tickets 2024-03-27T21:42:12.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f288e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:12.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f288e7fc640 1 -- 172.21.15.60:0/2657858068 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f288802cdb0 con 0x7f28900a86e0 2024-03-27T21:42:12.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f2895f8b640 5 monclient: authenticate success, global_id 5150 2024-03-27T21:42:12.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f2895f8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:12.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f288e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:12.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f2895f8b640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:12.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f288e7fc640 1 -- 172.21.15.60:0/2657858068 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2888034660 con 0x7f28900a86e0 2024-03-27T21:42:12.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f2895f8b640 4 set_mon_vals no callback set 2024-03-27T21:42:12.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f288e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:12.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f288e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:12.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f288e7fc640 10 monclient: dump: 2024-03-27T21:42:12.620 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:12.620 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:12.620 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.621 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.621 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:12.621 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:12.621 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:12.621 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:12.621 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:12.621 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f2895f8b640 10 monclient: shutdown 2024-03-27T21:42:12.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f2895f8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:12.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f2895f8b640 1 -- 172.21.15.60:0/2657858068 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28900a86e0 msgr2=0x7f28900b00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f2895f8b640 1 --2- 172.21.15.60:0/2657858068 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28900a86e0 0x7f28900b00e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2888009d90 tx=0x7f288802bea0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f2895f8b640 1 -- 172.21.15.60:0/2657858068 shutdown_connections 2024-03-27T21:42:12.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f2895f8b640 1 --2- 172.21.15.60:0/2657858068 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28900a86e0 0x7f28900b00e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f2895f8b640 1 --2- 172.21.15.60:0/2657858068 >> v2:172.21.15.60:3300/0 conn(0x7f28900b3980 0x7f28901449c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f2895f8b640 1 -- 172.21.15.60:0/2657858068 >> 172.21.15.60:0/2657858068 conn(0x7f2890071410 msgr2=0x7f2890071820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:12.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f2895f8b640 1 -- 172.21.15.60:0/2657858068 shutdown_connections 2024-03-27T21:42:12.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.617+0000 7f2895f8b640 1 -- 172.21.15.60:0/2657858068 wait complete. 2024-03-27T21:42:12.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.618+0000 7f2895f8b640 10 monclient: build_initial_monmap 2024-03-27T21:42:12.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.618+0000 7f2895f8b640 10 monclient: monmap: 2024-03-27T21:42:12.623 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:12.623 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:12.623 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:12.623 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:12.623 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:12.623 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:12.623 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:12.623 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:12.624 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:12.624 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.618+0000 7f2895f8b640 1 librados: starting msgr at 2024-03-27T21:42:12.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.618+0000 7f2895f8b640 1 librados: starting objecter 2024-03-27T21:42:12.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.618+0000 7f2895f8b640 1 Processor -- start 2024-03-27T21:42:12.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f2895f8b640 1 -- start start 2024-03-27T21:42:12.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f2895f8b640 1 librados: setting wanted keys 2024-03-27T21:42:12.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f2895f8b640 1 librados: calling monclient init 2024-03-27T21:42:12.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f2895f8b640 10 monclient: init 2024-03-27T21:42:12.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f2895f8b640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:12.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f2895f8b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:42:12.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f2895f8b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28900a86e0 0x7f289013e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f2895f8b640 10 monclient(hunting): picked mon.noname-a con 0x7f28900a86e0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:12.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f2895f8b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28900b3980 0x7f28901d9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f2895f8b640 10 monclient(hunting): picked mon.noname-c con 0x7f28900b3980 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:12.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f2895f8b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28901da130 0x7f28900aec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f2895f8b640 10 monclient(hunting): picked mon.noname-b con 0x7f28901da130 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:12.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f2895f8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:12.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f2895f8b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28900b7730 con 0x7f28900a86e0 2024-03-27T21:42:12.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f2895f8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:12.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f2895f8b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28901da5e0 con 0x7f28901da130 2024-03-27T21:42:12.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f2895f8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:12.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f2895f8b640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28900af140 con 0x7f28900b3980 2024-03-27T21:42:12.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f2895f8b640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:12.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f2895f8b640 10 monclient(hunting): authenticate will time out at 973.840468s 2024-03-27T21:42:12.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28900a86e0 0x7f289013e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288effd640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28900b3980 0x7f28901d9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28900a86e0 0x7f289013e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59298/0 (socket says 172.21.15.60:59298) 2024-03-27T21:42:12.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288f7fe640 1 -- 172.21.15.60:0/3190578097 learned_addr learned my addr 172.21.15.60:0/3190578097 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:12.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288f7fe640 10 monclient(hunting): get_auth_request con 0x7f28900a86e0 auth_method 0 2024-03-27T21:42:12.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:12.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:12.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:12.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288effd640 10 monclient(hunting): get_auth_request con 0x7f28900b3980 auth_method 0 2024-03-27T21:42:12.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:12.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288effd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:12.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:12.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:12.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:12.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:12.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:12.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:12.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:12.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288ffff640 1 --2- 172.21.15.60:0/3190578097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28901da130 0x7f28900aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288ffff640 10 monclient(hunting): get_auth_request con 0x7f28901da130 auth_method 0 2024-03-27T21:42:12.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:12.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:12.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:12.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288effd640 10 monclient(hunting): handle_auth_done global_id 5153 payload 824 2024-03-27T21:42:12.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288effd640 1 -- 172.21.15.60:0/3190578097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28901da130 msgr2=0x7f28900aec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288effd640 1 --2- 172.21.15.60:0/3190578097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28901da130 0x7f28900aec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288effd640 1 -- 172.21.15.60:0/3190578097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28900a86e0 msgr2=0x7f289013e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288effd640 1 --2- 172.21.15.60:0/3190578097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28900a86e0 0x7f289013e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288effd640 10 monclient: _finish_hunting 0 2024-03-27T21:42:12.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288effd640 1 monclient: found mon.noname-c 2024-03-27T21:42:12.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:12.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:12.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288effd640 1 -- 172.21.15.60:0/3190578097 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28800094d0 con 0x7f28900b3980 2024-03-27T21:42:12.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.619+0000 7f288f7fe640 1 --2- 172.21.15.60:0/3190578097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28900a86e0 0x7f289013e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:12.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f288effd640 1 --2- 172.21.15.60:0/3190578097 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28900b3980 0x7f28901d9bf0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2890141420 tx=0x7f288400c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:12.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2877fff640 10 client.?.objecter ms_handle_connect 0x7f28900b3980 2024-03-27T21:42:12.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2877fff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:12.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2877fff640 1 -- 172.21.15.60:0/3190578097 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28840052d0 con 0x7f28900b3980 2024-03-27T21:42:12.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2877fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:12.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2877fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:12.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2877fff640 10 monclient: dump: 2024-03-27T21:42:12.631 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:12.631 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:12.631 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.631 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.631 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:12.632 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:12.632 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:12.632 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:12.632 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:12.632 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2877fff640 10 monclient: _finish_auth 0 2024-03-27T21:42:12.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2877fff640 10 monclient: _check_auth_tickets 2024-03-27T21:42:12.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2877fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:12.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2895f8b640 5 monclient: authenticate success, global_id 5153 2024-03-27T21:42:12.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2877fff640 1 -- 172.21.15.60:0/3190578097 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2884005430 con 0x7f28900b3980 2024-03-27T21:42:12.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2877fff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:12.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2877fff640 1 -- 172.21.15.60:0/3190578097 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2884003810 con 0x7f28900b3980 2024-03-27T21:42:12.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2895f8b640 10 monclient: _renew_subs 2024-03-27T21:42:12.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2895f8b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:12.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2895f8b640 1 -- 172.21.15.60:0/3190578097 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2888009a70 con 0x7f28900b3980 2024-03-27T21:42:12.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2877fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:12.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2877fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:12.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f288dffb640 4 set_mon_vals no callback set 2024-03-27T21:42:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2877fff640 10 monclient: dump: 2024-03-27T21:42:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:12.635 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:12.635 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2895f8b640 10 client.5153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:12.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2895f8b640 10 monclient: _renew_subs 2024-03-27T21:42:12.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2895f8b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:12.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2895f8b640 1 -- 172.21.15.60:0/3190578097 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28900af760 con 0x7f28900b3980 2024-03-27T21:42:12.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.620+0000 7f2895f8b640 1 librados: init done 2024-03-27T21:42:12.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.621+0000 7f2895f8b640 10 client.5153.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:12.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.621+0000 7f2895f8b640 15 client.5153.objecter _session_command_op_assign -1 1 2024-03-27T21:42:12.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.621+0000 7f2895f8b640 20 client.5153.objecter _get_session osd=-1 returning homeless 2024-03-27T21:42:12.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.621+0000 7f2895f8b640 10 client.5153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:12.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.621+0000 7f2895f8b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:12.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.621+0000 7f2895f8b640 1 -- 172.21.15.60:0/3190578097 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f289014bd50 con 0x7f28900b3980 2024-03-27T21:42:12.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.621+0000 7f2877fff640 1 -- 172.21.15.60:0/3190578097 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2884003970 con 0x7f28900b3980 2024-03-27T21:42:12.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.621+0000 7f2877fff640 1 --2- 172.21.15.60:0/3190578097 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f287c06c590 0x7f287c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f288f7fe640 1 --2- 172.21.15.60:0/3190578097 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f287c06c590 0x7f287c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f2877fff640 1 -- 172.21.15.60:0/3190578097 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f2884088630 con 0x7f28900b3980 2024-03-27T21:42:12.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f2877fff640 10 client.5153.objecter ms_dispatch 0x7f2890000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:12.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f288f7fe640 10 monclient: get_auth_request con 0x7f287c06c590 auth_method 0 2024-03-27T21:42:12.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f2877fff640 3 client.5153.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:12.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f2877fff640 3 client.5153.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:12.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f2877fff640 10 client.5153.objecter checking command 1 2024-03-27T21:42:12.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f2877fff640 1 --2- 172.21.15.60:0/3190578097 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f287c0729d0 0x7f287c074df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f2877fff640 20 client.5153.objecter _get_session s=0x7f287c072300 osd=3 3 2024-03-27T21:42:12.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f2877fff640 20 client.5153.objecter put_session s=0x7f287c072300 osd=3 3 2024-03-27T21:42:12.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f2877fff640 15 client.5153.objecter _session_command_op_remove -1 1 2024-03-27T21:42:12.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f2877fff640 20 client.5153.objecter _get_session s=0x7f287c072300 osd=3 3 2024-03-27T21:42:12.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f2877fff640 20 client.5153.objecter get_session s=0x7f287c072300 osd=3 3 2024-03-27T21:42:12.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f2877fff640 15 client.5153.objecter _session_command_op_assign 3 1 2024-03-27T21:42:12.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f2877fff640 20 client.5153.objecter put_session s=0x7f287c072300 osd=3 4 2024-03-27T21:42:12.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f2877fff640 10 client.5153.objecter _send_command 1 2024-03-27T21:42:12.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f2877fff640 1 -- 172.21.15.60:0/3190578097 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f287c0754c0 con 0x7f287c0729d0 2024-03-27T21:42:12.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f2877fff640 20 client.5153.objecter dump_active .. 0 homeless 2024-03-27T21:42:12.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f2877fff640 1 -- 172.21.15.60:0/3190578097 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f288408acf0 con 0x7f28900b3980 2024-03-27T21:42:12.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f2877fff640 10 monclient: handle_get_version_reply finishing 1 version 92 2024-03-27T21:42:12.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f288ffff640 1 --2- 172.21.15.60:0/3190578097 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f287c0729d0 0x7f287c074df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f288ffff640 10 monclient: get_auth_request con 0x7f287c0729d0 auth_method 0 2024-03-27T21:42:12.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f2877fff640 10 client.5153.objecter ms_handle_connect 0x7f287c06c590 2024-03-27T21:42:12.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f288f7fe640 1 --2- 172.21.15.60:0/3190578097 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f287c06c590 0x7f287c06ea50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f28800114d0 tx=0x7f2880009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:12.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f288ffff640 1 --2- 172.21.15.60:0/3190578097 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f287c0729d0 0x7f287c074df0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:12.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.622+0000 7f2877fff640 10 client.5153.objecter ms_handle_connect 0x7f287c0729d0 2024-03-27T21:42:12.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.623+0000 7f2877fff640 1 -- 172.21.15.60:0/3190578097 <== osd.3 v2:172.21.15.60:6808/34613 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f287c0754c0 con 0x7f287c0729d0 2024-03-27T21:42:12.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.623+0000 7f2877fff640 10 client.5153.objecter ms_dispatch 0x7f2890000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:42:12.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.623+0000 7f2877fff640 10 client.5153.objecter _finish_command 1 = system:0 2024-03-27T21:42:12.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.623+0000 7f2877fff640 20 client.5153.objecter put_session s=0x7f287c072300 osd=3 4 2024-03-27T21:42:12.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.623+0000 7f2877fff640 15 client.5153.objecter _session_command_op_remove 3 1 2024-03-27T21:42:12.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.635+0000 7f2895f8b640 10 client.5153.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-27T21:42:12.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.635+0000 7f2895f8b640 15 client.5153.objecter _session_command_op_assign -1 2 2024-03-27T21:42:12.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.635+0000 7f2895f8b640 20 client.5153.objecter _get_session s=0x7f287c072300 osd=3 3 2024-03-27T21:42:12.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.635+0000 7f2895f8b640 20 client.5153.objecter put_session s=0x7f287c072300 osd=3 3 2024-03-27T21:42:12.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.635+0000 7f2895f8b640 20 client.5153.objecter _get_session s=0x7f287c072300 osd=3 3 2024-03-27T21:42:12.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.635+0000 7f2895f8b640 15 client.5153.objecter _session_command_op_remove -1 2 2024-03-27T21:42:12.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.635+0000 7f2895f8b640 20 client.5153.objecter get_session s=0x7f287c072300 osd=3 3 2024-03-27T21:42:12.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.635+0000 7f2895f8b640 15 client.5153.objecter _session_command_op_assign 3 2 2024-03-27T21:42:12.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.635+0000 7f2895f8b640 20 client.5153.objecter put_session s=0x7f287c072300 osd=3 4 2024-03-27T21:42:12.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.635+0000 7f2895f8b640 10 client.5153.objecter _send_command 2 2024-03-27T21:42:12.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.635+0000 7f2895f8b640 1 -- 172.21.15.60:0/3190578097 --> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f28900670d0 con 0x7f287c0729d0 2024-03-27T21:42:12.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2877fff640 1 -- 172.21.15.60:0/3190578097 <== osd.3 v2:172.21.15.60:6808/34613 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f28900670d0 con 0x7f287c0729d0 2024-03-27T21:42:12.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2877fff640 10 client.5153.objecter ms_dispatch 0x7f2890000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:42:12.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2877fff640 10 client.5153.objecter _finish_command 2 = system:0 2024-03-27T21:42:12.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2877fff640 20 client.5153.objecter put_session s=0x7f287c072300 osd=3 4 2024-03-27T21:42:12.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2877fff640 15 client.5153.objecter _session_command_op_remove 3 2 2024-03-27T21:42:12.643 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:42:12.643 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-27T21:42:12.643 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:42:12.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2895f8b640 10 librados: watch_flush enter 2024-03-27T21:42:12.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2895f8b640 10 librados: watch_flush exit 2024-03-27T21:42:12.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2895f8b640 10 client.5153.objecter close_session for osd.3 2024-03-27T21:42:12.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2895f8b640 1 -- 172.21.15.60:0/3190578097 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f287c0729d0 msgr2=0x7f287c074df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2895f8b640 1 --2- 172.21.15.60:0/3190578097 >> [v2:172.21.15.60:6808/34613,v1:172.21.15.60:6809/34613] conn(0x7f287c0729d0 0x7f287c074df0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2895f8b640 20 client.5153.objecter put_session s=0x7f287c072300 osd=3 1 2024-03-27T21:42:12.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2895f8b640 20 client.5153.objecter shutdown clearing up homeless session... 2024-03-27T21:42:12.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2895f8b640 10 client.5153.objecter successfully canceled tick 2024-03-27T21:42:12.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2895f8b640 1 -- 172.21.15.60:0/3190578097 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f287c06c590 msgr2=0x7f287c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2895f8b640 1 --2- 172.21.15.60:0/3190578097 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f287c06c590 0x7f287c06ea50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f28800114d0 tx=0x7f2880009240 comp rx=0 tx=0).stop 2024-03-27T21:42:12.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2895f8b640 10 monclient: shutdown 2024-03-27T21:42:12.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2895f8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:12.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2895f8b640 1 -- 172.21.15.60:0/3190578097 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28900b3980 msgr2=0x7f28901d9bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2895f8b640 1 --2- 172.21.15.60:0/3190578097 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f28900b3980 0x7f28901d9bf0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2890141420 tx=0x7f288400c820 comp rx=0 tx=0).stop 2024-03-27T21:42:12.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f288ffff640 1 -- 172.21.15.60:0/3190578097 reap_dead start 2024-03-27T21:42:12.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2895f8b640 1 -- 172.21.15.60:0/3190578097 shutdown_connections 2024-03-27T21:42:12.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2895f8b640 1 -- 172.21.15.60:0/3190578097 >> 172.21.15.60:0/3190578097 conn(0x7f2890071410 msgr2=0x7f28900717f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:12.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2895f8b640 1 -- 172.21.15.60:0/3190578097 shutdown_connections 2024-03-27T21:42:12.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2895f8b640 1 -- 172.21.15.60:0/3190578097 wait complete. 2024-03-27T21:42:12.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.636+0000 7f2895f8b640 1 librados: shutdown 2024-03-27T21:42:12.647 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-27T21:42:12.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.762+0000 7f42e882f640 10 monclient: get_monmap_and_config 2024-03-27T21:42:12.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.762+0000 7f42e882f640 10 monclient: build_initial_monmap 2024-03-27T21:42:12.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.762+0000 7f42e882f640 10 monclient: monmap: 2024-03-27T21:42:12.748 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:12.748 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:12.748 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:12.763033+0000 2024-03-27T21:42:12.749 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:12.763033+0000 2024-03-27T21:42:12.749 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:12.749 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:12.749 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:12.749 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:12.749 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:12.749 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:12.749 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:12.749 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.762+0000 7f42e882f640 1 Processor -- start 2024-03-27T21:42:12.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.762+0000 7f42e882f640 1 -- start start 2024-03-27T21:42:12.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.762+0000 7f42e882f640 10 monclient: init 2024-03-27T21:42:12.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e882f640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:12.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e882f640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-03-27T21:42:12.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e882f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42e40b40b0 0x7f42e40a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e882f640 10 monclient(hunting): picked mon.noname-b con 0x7f42e40b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:12.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e882f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f42e413b020 0x7f42e413b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e882f640 10 monclient(hunting): picked mon.noname-a con 0x7f42e413b020 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:12.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e882f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f42e41416f0 0x7f42e4141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e882f640 10 monclient(hunting): picked mon.noname-c con 0x7f42e41416f0 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:12.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e882f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:12.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e882f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42e40b4490 con 0x7f42e413b020 2024-03-27T21:42:12.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e882f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:12.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e882f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42e40b3830 con 0x7f42e41416f0 2024-03-27T21:42:12.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e882f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:12.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e882f640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e40aa520 con 0x7f42e40b40b0 2024-03-27T21:42:12.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e882f640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:12.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e882f640 10 monclient(hunting): authenticate will time out at 973.984386s 2024-03-27T21:42:12.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e259c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42e40b40b0 0x7f42e40a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e259c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42e40b40b0 0x7f42e40a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:51464/0 (socket says 172.21.15.60:51464) 2024-03-27T21:42:12.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e259c640 1 -- 172.21.15.60:0/2482443730 learned_addr learned my addr 172.21.15.60:0/2482443730 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:12.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e259c640 10 monclient(hunting): get_auth_request con 0x7f42e40b40b0 auth_method 0 2024-03-27T21:42:12.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:12.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e259c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:12.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:12.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e1d9b640 1 --2- 172.21.15.60:0/2482443730 >> v2:172.21.15.60:3300/0 conn(0x7f42e413b020 0x7f42e413b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e1d9b640 10 monclient(hunting): get_auth_request con 0x7f42e413b020 auth_method 0 2024-03-27T21:42:12.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:12.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:12.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:12.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:12.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:12.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:12.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:12.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:12.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:12.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e2d9d640 1 --2- 172.21.15.60:0/2482443730 >> v2:172.21.15.191:3300/0 conn(0x7f42e41416f0 0x7f42e4141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e2d9d640 10 monclient(hunting): get_auth_request con 0x7f42e41416f0 auth_method 0 2024-03-27T21:42:12.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:12.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e2d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:12.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:12.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:12.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:12.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:12.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e1d9b640 10 monclient(hunting): handle_auth_done global_id 5496 payload 274 2024-03-27T21:42:12.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e1d9b640 1 -- 172.21.15.60:0/2482443730 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42e40b40b0 msgr2=0x7f42e40a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:12.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e1d9b640 1 --2- 172.21.15.60:0/2482443730 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42e40b40b0 0x7f42e40a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e1d9b640 1 -- 172.21.15.60:0/2482443730 >> v2:172.21.15.191:3300/0 conn(0x7f42e41416f0 msgr2=0x7f42e4141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:12.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e1d9b640 1 --2- 172.21.15.60:0/2482443730 >> v2:172.21.15.191:3300/0 conn(0x7f42e41416f0 0x7f42e4141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e1d9b640 10 monclient: _finish_hunting 0 2024-03-27T21:42:12.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e1d9b640 1 monclient: found mon.noname-a 2024-03-27T21:42:12.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:12.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:12.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e1d9b640 1 -- 172.21.15.60:0/2482443730 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e40a9470 con 0x7f42e413b020 2024-03-27T21:42:12.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.763+0000 7f42e259c640 1 --2- 172.21.15.60:0/2482443730 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42e40b40b0 0x7f42e40a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:12.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.764+0000 7f42e1d9b640 1 --2- 172.21.15.60:0/2482443730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42e413b020 0x7f42e413b480 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f42e40b2e20 tx=0x7f42cc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9337314d4c314b74 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:12.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.764+0000 7f42e159a640 1 -- 172.21.15.60:0/2482443730 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc02e070 con 0x7f42e413b020 2024-03-27T21:42:12.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.764+0000 7f42e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:12.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.764+0000 7f42e159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:12.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.764+0000 7f42e159a640 10 monclient: dump: 2024-03-27T21:42:12.766 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:12.766 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:12.766 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.766 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.766 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:12.766 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:12.766 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:12.767 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:12.767 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:12.767 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.764+0000 7f42e159a640 10 monclient: _finish_auth 0 2024-03-27T21:42:12.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.764+0000 7f42e159a640 10 monclient: _check_auth_tickets 2024-03-27T21:42:12.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.764+0000 7f42e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:12.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.764+0000 7f42e159a640 1 -- 172.21.15.60:0/2482443730 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42cc0315a0 con 0x7f42e413b020 2024-03-27T21:42:12.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.764+0000 7f42e882f640 5 monclient: authenticate success, global_id 5496 2024-03-27T21:42:12.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.764+0000 7f42e882f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:12.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.764+0000 7f42e159a640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:12.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.764+0000 7f42e882f640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:12.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.764+0000 7f42e159a640 1 -- 172.21.15.60:0/2482443730 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc031930 con 0x7f42e413b020 2024-03-27T21:42:12.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.764+0000 7f42e882f640 4 set_mon_vals no callback set 2024-03-27T21:42:12.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.764+0000 7f42e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:12.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.764+0000 7f42e159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:12.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.764+0000 7f42e159a640 10 monclient: dump: 2024-03-27T21:42:12.769 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:12.770 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:12.770 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.770 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.770 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:12.770 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:12.770 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:12.770 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:12.770 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:12.771 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.765+0000 7f42e882f640 10 monclient: shutdown 2024-03-27T21:42:12.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.765+0000 7f42e882f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:12.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.765+0000 7f42e882f640 1 -- 172.21.15.60:0/2482443730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42e413b020 msgr2=0x7f42e413b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.765+0000 7f42e882f640 1 --2- 172.21.15.60:0/2482443730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42e413b020 0x7f42e413b480 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f42e40b2e20 tx=0x7f42cc02b840 comp rx=0 tx=0).stop 2024-03-27T21:42:12.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.765+0000 7f42e882f640 1 -- 172.21.15.60:0/2482443730 shutdown_connections 2024-03-27T21:42:12.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.765+0000 7f42e882f640 1 --2- 172.21.15.60:0/2482443730 >> v2:172.21.15.191:3300/0 conn(0x7f42e41416f0 0x7f42e4141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.765+0000 7f42e882f640 1 --2- 172.21.15.60:0/2482443730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42e413b020 0x7f42e413b480 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.765+0000 7f42e882f640 1 --2- 172.21.15.60:0/2482443730 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42e40b40b0 0x7f42e40a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.765+0000 7f42e882f640 1 -- 172.21.15.60:0/2482443730 >> 172.21.15.60:0/2482443730 conn(0x7f42e40608e0 msgr2=0x7f42e4060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:12.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.765+0000 7f42e882f640 1 -- 172.21.15.60:0/2482443730 shutdown_connections 2024-03-27T21:42:12.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.765+0000 7f42e882f640 1 -- 172.21.15.60:0/2482443730 wait complete. 2024-03-27T21:42:12.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.766+0000 7f42e882f640 10 monclient: build_initial_monmap 2024-03-27T21:42:12.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.766+0000 7f42e882f640 10 monclient: monmap: 2024-03-27T21:42:12.772 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:12.772 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:12.772 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:12.772 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:12.773 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:12.773 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:12.773 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:12.773 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:12.773 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:12.773 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.766+0000 7f42e882f640 1 librados: starting msgr at 2024-03-27T21:42:12.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.766+0000 7f42e882f640 1 librados: starting objecter 2024-03-27T21:42:12.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.766+0000 7f42e882f640 1 Processor -- start 2024-03-27T21:42:12.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.766+0000 7f42e882f640 1 -- start start 2024-03-27T21:42:12.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.766+0000 7f42e882f640 1 librados: setting wanted keys 2024-03-27T21:42:12.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.766+0000 7f42e882f640 1 librados: calling monclient init 2024-03-27T21:42:12.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.766+0000 7f42e882f640 10 monclient: init 2024-03-27T21:42:12.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e882f640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:12.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e882f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-27T21:42:12.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e882f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42e40b40b0 0x7f42e413ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e882f640 10 monclient(hunting): picked mon.noname-b con 0x7f42e40b40b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:12.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e882f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42e413b020 0x7f42e413f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e882f640 10 monclient(hunting): picked mon.noname-a con 0x7f42e413b020 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:12.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e882f640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42e41416f0 0x7f42e413e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e882f640 10 monclient(hunting): picked mon.noname-c con 0x7f42e41416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:12.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e882f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:12.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e882f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e4144710 con 0x7f42e413b020 2024-03-27T21:42:12.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e882f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:12.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e882f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e413d790 con 0x7f42e40b40b0 2024-03-27T21:42:12.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e882f640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:12.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e882f640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e413fc90 con 0x7f42e41416f0 2024-03-27T21:42:12.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e882f640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:12.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e882f640 10 monclient(hunting): authenticate will time out at 973.988192s 2024-03-27T21:42:12.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e1d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42e413b020 0x7f42e413f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e2d9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42e41416f0 0x7f42e413e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e1d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42e413b020 0x7f42e413f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59308/0 (socket says 172.21.15.60:59308) 2024-03-27T21:42:12.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e2d9d640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42e41416f0 0x7f42e413e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:51466/0 (socket says 172.21.15.60:51466) 2024-03-27T21:42:12.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e1d9b640 1 -- 172.21.15.60:0/2418134302 learned_addr learned my addr 172.21.15.60:0/2418134302 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:12.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e1d9b640 10 monclient(hunting): get_auth_request con 0x7f42e413b020 auth_method 0 2024-03-27T21:42:12.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:12.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:12.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:12.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e2d9d640 10 monclient(hunting): get_auth_request con 0x7f42e41416f0 auth_method 0 2024-03-27T21:42:12.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:12.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e2d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:12.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:12.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:12.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:12.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:12.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:12.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:12.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:12.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e259c640 1 --2- 172.21.15.60:0/2418134302 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42e40b40b0 0x7f42e413ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e259c640 10 monclient(hunting): get_auth_request con 0x7f42e40b40b0 auth_method 0 2024-03-27T21:42:12.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:12.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e259c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:12.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:12.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:12.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:12.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:12.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e1d9b640 10 monclient(hunting): handle_auth_done global_id 5499 payload 824 2024-03-27T21:42:12.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e1d9b640 1 -- 172.21.15.60:0/2418134302 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42e41416f0 msgr2=0x7f42e413e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e1d9b640 1 --2- 172.21.15.60:0/2418134302 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42e41416f0 0x7f42e413e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e1d9b640 1 -- 172.21.15.60:0/2418134302 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42e40b40b0 msgr2=0x7f42e413ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e1d9b640 1 --2- 172.21.15.60:0/2418134302 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42e40b40b0 0x7f42e413ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e1d9b640 10 monclient: _finish_hunting 0 2024-03-27T21:42:12.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e1d9b640 1 monclient: found mon.noname-a 2024-03-27T21:42:12.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:12.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:12.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e1d9b640 1 -- 172.21.15.60:0/2418134302 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42dc0094d0 con 0x7f42e413b020 2024-03-27T21:42:12.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e2d9d640 1 --2- 172.21.15.60:0/2418134302 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42e41416f0 0x7f42e413e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:12.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42e1d9b640 1 --2- 172.21.15.60:0/2418134302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42e413b020 0x7f42e413f750 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f42cc034040 tx=0x7f42cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:12.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42daffd640 10 client.?.objecter ms_handle_connect 0x7f42e413b020 2024-03-27T21:42:12.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42daffd640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:12.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.767+0000 7f42daffd640 1 -- 172.21.15.60:0/2418134302 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc032840 con 0x7f42e413b020 2024-03-27T21:42:12.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:12.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42daffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:12.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42daffd640 10 monclient: dump: 2024-03-27T21:42:12.781 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:12.781 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:12.781 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.781 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.781 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:12.781 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:12.782 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:12.782 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:12.782 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:12.782 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42daffd640 10 monclient: _finish_auth 0 2024-03-27T21:42:12.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42daffd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:12.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:12.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42daffd640 1 -- 172.21.15.60:0/2418134302 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42cc031f00 con 0x7f42e413b020 2024-03-27T21:42:12.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42e882f640 5 monclient: authenticate success, global_id 5499 2024-03-27T21:42:12.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42e882f640 10 monclient: _renew_subs 2024-03-27T21:42:12.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42e882f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:12.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42e882f640 1 -- 172.21.15.60:0/2418134302 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42cc009400 con 0x7f42e413b020 2024-03-27T21:42:12.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:12.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42daffd640 1 -- 172.21.15.60:0/2418134302 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc032be0 con 0x7f42e413b020 2024-03-27T21:42:12.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:12.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:12.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42daffd640 10 monclient: dump: 2024-03-27T21:42:12.784 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:12.784 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:12.784 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.784 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:12.784 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:12.784 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:12.784 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:12.784 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:12.784 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:12.785 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:12.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42e0d99640 4 set_mon_vals no callback set 2024-03-27T21:42:12.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42e882f640 10 client.5499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:12.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42e882f640 10 monclient: _renew_subs 2024-03-27T21:42:12.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42e882f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:12.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42e882f640 1 -- 172.21.15.60:0/2418134302 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e4144230 con 0x7f42e413b020 2024-03-27T21:42:12.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42e882f640 1 librados: init done 2024-03-27T21:42:12.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42e882f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:12.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42e882f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:12.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42e882f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:12.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.768+0000 7f42e882f640 1 -- 172.21.15.60:0/2418134302 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e413df70 con 0x7f42e413b020 2024-03-27T21:42:12.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.769+0000 7f42daffd640 1 -- 172.21.15.60:0/2418134302 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f42cc032d40 con 0x7f42e413b020 2024-03-27T21:42:12.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.769+0000 7f42daffd640 1 --2- 172.21.15.60:0/2418134302 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f42c406c5c0 0x7f42c406ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:12.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.769+0000 7f42daffd640 1 -- 172.21.15.60:0/2418134302 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f42cc0aae10 con 0x7f42e413b020 2024-03-27T21:42:12.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.769+0000 7f42daffd640 10 client.5499.objecter ms_dispatch 0x7f42e4000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:12.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.769+0000 7f42daffd640 3 client.5499.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:12.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.769+0000 7f42e259c640 1 --2- 172.21.15.60:0/2418134302 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f42c406c5c0 0x7f42c406ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:12.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.770+0000 7f42daffd640 3 client.5499.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:12.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.770+0000 7f42e259c640 10 monclient: get_auth_request con 0x7f42c406c5c0 auth_method 0 2024-03-27T21:42:12.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.770+0000 7f42daffd640 20 client.5499.objecter dump_active .. 0 homeless 2024-03-27T21:42:12.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.770+0000 7f42daffd640 10 client.5499.objecter ms_handle_connect 0x7f42c406c5c0 2024-03-27T21:42:12.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.770+0000 7f42e259c640 1 --2- 172.21.15.60:0/2418134302 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f42c406c5c0 0x7f42c406ea80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f42dc006ca0 tx=0x7f42dc009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:12.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.774+0000 7f42daffd640 1 -- 172.21.15.60:0/2418134302 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f42cc076300 con 0x7f42e413b020 2024-03-27T21:42:12.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.774+0000 7f42daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:12.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.774+0000 7f42daffd640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:12.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.952+0000 7f42e882f640 1 -- 172.21.15.60:0/2418134302 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f42e4144960 con 0x7f42c406c5c0 2024-03-27T21:42:12.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.952+0000 7f42daffd640 1 -- 172.21.15.60:0/2418134302 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f42e4144960 con 0x7f42c406c5c0 2024-03-27T21:42:12.939 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 3 to deep-scrub 2024-03-27T21:42:12.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.954+0000 7f42e882f640 10 librados: watch_flush enter 2024-03-27T21:42:12.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.954+0000 7f42e882f640 10 librados: watch_flush exit 2024-03-27T21:42:12.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.954+0000 7f42e882f640 20 client.5499.objecter shutdown clearing up homeless session... 2024-03-27T21:42:12.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.954+0000 7f42e882f640 10 client.5499.objecter successfully canceled tick 2024-03-27T21:42:12.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.954+0000 7f42e882f640 1 -- 172.21.15.60:0/2418134302 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f42c406c5c0 msgr2=0x7f42c406ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.954+0000 7f42e882f640 1 --2- 172.21.15.60:0/2418134302 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f42c406c5c0 0x7f42c406ea80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f42dc006ca0 tx=0x7f42dc009240 comp rx=0 tx=0).stop 2024-03-27T21:42:12.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.954+0000 7f42e882f640 10 monclient: shutdown 2024-03-27T21:42:12.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.954+0000 7f42e882f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:12.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.954+0000 7f42e882f640 1 -- 172.21.15.60:0/2418134302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42e413b020 msgr2=0x7f42e413f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:12.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.954+0000 7f42e882f640 1 --2- 172.21.15.60:0/2418134302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42e413b020 0x7f42e413f750 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f42cc034040 tx=0x7f42cc0046c0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.954+0000 7f42e882f640 1 -- 172.21.15.60:0/2418134302 shutdown_connections 2024-03-27T21:42:12.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.954+0000 7f42e882f640 1 --2- 172.21.15.60:0/2418134302 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f42e41416f0 0x7f42e413e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.954+0000 7f42e882f640 1 --2- 172.21.15.60:0/2418134302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42e413b020 0x7f42e413f750 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.954+0000 7f42e882f640 1 --2- 172.21.15.60:0/2418134302 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f42c406c5c0 0x7f42c406ea80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.954+0000 7f42e882f640 1 --2- 172.21.15.60:0/2418134302 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42e40b40b0 0x7f42e413ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:12.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.954+0000 7f42e882f640 1 -- 172.21.15.60:0/2418134302 >> 172.21.15.60:0/2418134302 conn(0x7f42e40608e0 msgr2=0x7f42e40b6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:12.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.954+0000 7f42e882f640 1 -- 172.21.15.60:0/2418134302 shutdown_connections 2024-03-27T21:42:12.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.954+0000 7f42e882f640 1 -- 172.21.15.60:0/2418134302 wait complete. 2024-03-27T21:42:12.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:12.954+0000 7f42e882f640 1 librados: shutdown 2024-03-27T21:42:12.951 INFO:tasks.ceph:Scrubbing osd.4 2024-03-27T21:42:12.951 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-27T21:42:13.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.069+0000 7fdd5e58b640 10 monclient: get_monmap_and_config 2024-03-27T21:42:13.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.069+0000 7fdd5e58b640 10 monclient: build_initial_monmap 2024-03-27T21:42:13.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.069+0000 7fdd5e58b640 10 monclient: monmap: 2024-03-27T21:42:13.056 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:13.056 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.056 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:13.070885+0000 2024-03-27T21:42:13.056 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:13.070885+0000 2024-03-27T21:42:13.057 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:13.057 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.057 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:13.057 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:13.057 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:13.057 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:13.057 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:13.058 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.070+0000 7fdd5e58b640 1 Processor -- start 2024-03-27T21:42:13.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.070+0000 7fdd5e58b640 1 -- start start 2024-03-27T21:42:13.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.070+0000 7fdd5e58b640 10 monclient: init 2024-03-27T21:42:13.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5e58b640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:13.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5e58b640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-03-27T21:42:13.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5e58b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdd580b3980 0x7fdd581449c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5e58b640 10 monclient(hunting): picked mon.noname-a con 0x7fdd580b3980 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:13.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5e58b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fdd580b3d60 0x7fdd580b0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5e58b640 10 monclient(hunting): picked mon.noname-c con 0x7fdd580b3d60 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:13.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5e58b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdd580a86e0 0x7fdd580b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5e58b640 10 monclient(hunting): picked mon.noname-b con 0x7fdd580a86e0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:13.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5e58b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5e58b640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd58144f00 con 0x7fdd580b3980 2024-03-27T21:42:13.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5e58b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5e58b640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd580b1140 con 0x7fdd580b3d60 2024-03-27T21:42:13.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5e58b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5e58b640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd580aa1d0 con 0x7fdd580a86e0 2024-03-27T21:42:13.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5e58b640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:13.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5e58b640 10 monclient(hunting): authenticate will time out at 974.292150s 2024-03-27T21:42:13.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd57fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdd580b3980 0x7fdd581449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5cb28640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdd580a86e0 0x7fdd580b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd57fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdd580b3980 0x7fdd581449c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59322/0 (socket says 172.21.15.60:59322) 2024-03-27T21:42:13.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd57fff640 1 -- 172.21.15.60:0/3839125294 learned_addr learned my addr 172.21.15.60:0/3839125294 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:13.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5cb28640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdd580a86e0 0x7fdd580b00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:51474/0 (socket says 172.21.15.60:51474) 2024-03-27T21:42:13.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd57fff640 10 monclient(hunting): get_auth_request con 0x7fdd580b3980 auth_method 0 2024-03-27T21:42:13.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:13.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd57fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5cb28640 10 monclient(hunting): get_auth_request con 0x7fdd580a86e0 auth_method 0 2024-03-27T21:42:13.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5cb28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:13.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5cb28640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5cb28640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5cb28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:13.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5cb28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:13.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5cb28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:13.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:13.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:13.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:13.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd577fe640 1 --2- 172.21.15.60:0/3839125294 >> v2:172.21.15.191:3300/0 conn(0x7fdd580b3d60 0x7fdd580b0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5cb28640 10 monclient(hunting): handle_auth_done global_id 5162 payload 274 2024-03-27T21:42:13.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5cb28640 1 -- 172.21.15.60:0/3839125294 >> v2:172.21.15.191:3300/0 conn(0x7fdd580b3d60 msgr2=0x7fdd580b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:13.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5cb28640 1 --2- 172.21.15.60:0/3839125294 >> v2:172.21.15.191:3300/0 conn(0x7fdd580b3d60 0x7fdd580b0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5cb28640 1 -- 172.21.15.60:0/3839125294 >> v2:172.21.15.60:3300/0 conn(0x7fdd580b3980 msgr2=0x7fdd581449c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:13.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5cb28640 1 --2- 172.21.15.60:0/3839125294 >> v2:172.21.15.60:3300/0 conn(0x7fdd580b3980 0x7fdd581449c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5cb28640 10 monclient: _finish_hunting 0 2024-03-27T21:42:13.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5cb28640 1 monclient: found mon.noname-b 2024-03-27T21:42:13.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5cb28640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:13.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5cb28640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:42:13.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd5cb28640 1 -- 172.21.15.60:0/3839125294 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd580aa4b0 con 0x7fdd580a86e0 2024-03-27T21:42:13.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd577fe640 10 monclient: get_auth_request con 0x7fdd580b3d60 auth_method 0 2024-03-27T21:42:13.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd577fe640 1 --2- 172.21.15.60:0/3839125294 >> v2:172.21.15.191:3300/0 conn(0x7fdd580b3d60 0x7fdd580b0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:42:13.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.071+0000 7fdd57fff640 1 --2- 172.21.15.60:0/3839125294 >> v2:172.21.15.60:3300/0 conn(0x7fdd580b3980 0x7fdd581449c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:13.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.072+0000 7fdd5cb28640 1 --2- 172.21.15.60:0/3839125294 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdd580a86e0 0x7fdd580b00e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdd580b4440 tx=0x7fdd5002be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37c4fb59eb93031 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:13.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.072+0000 7fdd56ffd640 1 -- 172.21.15.60:0/3839125294 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5002cbd0 con 0x7fdd580a86e0 2024-03-27T21:42:13.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.072+0000 7fdd56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:13.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.072+0000 7fdd56ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:13.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.072+0000 7fdd56ffd640 10 monclient: dump: 2024-03-27T21:42:13.076 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:13.076 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.076 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.076 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.076 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:13.076 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.077 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:13.077 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:13.077 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:13.077 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.072+0000 7fdd56ffd640 10 monclient: _finish_auth 0 2024-03-27T21:42:13.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.072+0000 7fdd56ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:13.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.072+0000 7fdd56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:13.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.072+0000 7fdd56ffd640 1 -- 172.21.15.60:0/3839125294 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5002cd30 con 0x7fdd580a86e0 2024-03-27T21:42:13.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.072+0000 7fdd5e58b640 5 monclient: authenticate success, global_id 5162 2024-03-27T21:42:13.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.072+0000 7fdd5e58b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:13.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.072+0000 7fdd56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:13.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.072+0000 7fdd56ffd640 1 -- 172.21.15.60:0/3839125294 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5002cea0 con 0x7fdd580a86e0 2024-03-27T21:42:13.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.072+0000 7fdd5e58b640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:13.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.072+0000 7fdd5e58b640 4 set_mon_vals no callback set 2024-03-27T21:42:13.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.072+0000 7fdd56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:13.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.072+0000 7fdd56ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:13.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.072+0000 7fdd56ffd640 10 monclient: dump: 2024-03-27T21:42:13.079 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:13.079 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.079 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.079 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.079 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:13.079 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.080 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:13.080 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:13.080 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:13.080 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.073+0000 7fdd5e58b640 10 monclient: shutdown 2024-03-27T21:42:13.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.073+0000 7fdd5e58b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:13.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.073+0000 7fdd5e58b640 1 -- 172.21.15.60:0/3839125294 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdd580a86e0 msgr2=0x7fdd580b00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.073+0000 7fdd5e58b640 1 --2- 172.21.15.60:0/3839125294 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdd580a86e0 0x7fdd580b00e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdd580b4440 tx=0x7fdd5002be20 comp rx=0 tx=0).stop 2024-03-27T21:42:13.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.073+0000 7fdd5e58b640 1 -- 172.21.15.60:0/3839125294 shutdown_connections 2024-03-27T21:42:13.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.073+0000 7fdd5e58b640 1 --2- 172.21.15.60:0/3839125294 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdd580a86e0 0x7fdd580b00e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.073+0000 7fdd5e58b640 1 --2- 172.21.15.60:0/3839125294 >> v2:172.21.15.191:3300/0 conn(0x7fdd580b3d60 0x7fdd580b0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.073+0000 7fdd5e58b640 1 --2- 172.21.15.60:0/3839125294 >> v2:172.21.15.60:3300/0 conn(0x7fdd580b3980 0x7fdd581449c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.073+0000 7fdd5e58b640 1 -- 172.21.15.60:0/3839125294 >> 172.21.15.60:0/3839125294 conn(0x7fdd58071410 msgr2=0x7fdd58071820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:13.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.073+0000 7fdd5e58b640 1 -- 172.21.15.60:0/3839125294 shutdown_connections 2024-03-27T21:42:13.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.073+0000 7fdd5e58b640 1 -- 172.21.15.60:0/3839125294 wait complete. 2024-03-27T21:42:13.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.073+0000 7fdd5e58b640 10 monclient: build_initial_monmap 2024-03-27T21:42:13.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.073+0000 7fdd5e58b640 10 monclient: monmap: 2024-03-27T21:42:13.082 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:13.082 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:13.082 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:13.082 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:13.082 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:13.082 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:13.082 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:13.083 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:13.083 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:13.083 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.073+0000 7fdd5e58b640 1 librados: starting msgr at 2024-03-27T21:42:13.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 1 librados: starting objecter 2024-03-27T21:42:13.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 1 Processor -- start 2024-03-27T21:42:13.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 1 -- start start 2024-03-27T21:42:13.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 1 librados: setting wanted keys 2024-03-27T21:42:13.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 1 librados: calling monclient init 2024-03-27T21:42:13.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 10 monclient: init 2024-03-27T21:42:13.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:13.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:42:13.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd580a86e0 0x7fdd58138a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 10 monclient(hunting): picked mon.noname-b con 0x7fdd580a86e0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:13.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdd580b3980 0x7fdd581d9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 10 monclient(hunting): picked mon.noname-c con 0x7fdd580b3980 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:13.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd581d9c70 0x7fdd58136ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 10 monclient(hunting): picked mon.noname-a con 0x7fdd581d9c70 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:13.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd581da680 con 0x7fdd581d9c70 2024-03-27T21:42:13.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd581395e0 con 0x7fdd580a86e0 2024-03-27T21:42:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd581da160 con 0x7fdd580b3980 2024-03-27T21:42:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5e58b640 10 monclient(hunting): authenticate will time out at 974.295770s 2024-03-27T21:42:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd577fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdd580b3980 0x7fdd581d9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5cb28640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd581d9c70 0x7fdd58136ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5cb28640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd581d9c70 0x7fdd58136ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59326/0 (socket says 172.21.15.60:59326) 2024-03-27T21:42:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5cb28640 1 -- 172.21.15.60:0/3120628111 learned_addr learned my addr 172.21.15.60:0/3120628111 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:13.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd577fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdd580b3980 0x7fdd581d9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:51478/0 (socket says 172.21.15.60:51478) 2024-03-27T21:42:13.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5cb28640 10 monclient(hunting): get_auth_request con 0x7fdd581d9c70 auth_method 0 2024-03-27T21:42:13.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5cb28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:13.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5cb28640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd5cb28640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd577fe640 10 monclient(hunting): get_auth_request con 0x7fdd580b3980 auth_method 0 2024-03-27T21:42:13.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:13.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd577fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.074+0000 7fdd577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd5cb28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:13.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd5cb28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:13.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd5cb28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:13.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:13.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:13.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:13.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd57fff640 1 --2- 172.21.15.60:0/3120628111 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd580a86e0 0x7fdd58138a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd57fff640 10 monclient(hunting): get_auth_request con 0x7fdd580a86e0 auth_method 0 2024-03-27T21:42:13.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:13.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd57fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:13.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:13.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:13.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd577fe640 10 monclient(hunting): handle_auth_done global_id 5165 payload 824 2024-03-27T21:42:13.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd577fe640 1 -- 172.21.15.60:0/3120628111 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd580a86e0 msgr2=0x7fdd58138a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd577fe640 1 --2- 172.21.15.60:0/3120628111 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd580a86e0 0x7fdd58138a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd577fe640 1 -- 172.21.15.60:0/3120628111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd581d9c70 msgr2=0x7fdd58136ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd577fe640 1 --2- 172.21.15.60:0/3120628111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd581d9c70 0x7fdd58136ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd577fe640 10 monclient: _finish_hunting 0 2024-03-27T21:42:13.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd577fe640 1 monclient: found mon.noname-c 2024-03-27T21:42:13.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:13.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd577fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:13.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd577fe640 1 -- 172.21.15.60:0/3120628111 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd480094d0 con 0x7fdd580b3980 2024-03-27T21:42:13.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd5cb28640 1 --2- 172.21.15.60:0/3120628111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd581d9c70 0x7fdd58136ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:13.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd577fe640 1 --2- 172.21.15.60:0/3120628111 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdd580b3980 0x7fdd581d9730 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c00c7f0 tx=0x7fdd4c00ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:13.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd54ff9640 10 client.?.objecter ms_handle_connect 0x7fdd580b3980 2024-03-27T21:42:13.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd54ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:13.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd54ff9640 1 -- 172.21.15.60:0/3120628111 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c007820 con 0x7fdd580b3980 2024-03-27T21:42:13.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:13.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd54ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:13.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd54ff9640 10 monclient: dump: 2024-03-27T21:42:13.092 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:13.092 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.092 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.093 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.093 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:13.093 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.093 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:13.093 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:13.093 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:13.093 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd54ff9640 10 monclient: _finish_auth 0 2024-03-27T21:42:13.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd54ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:42:13.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:13.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd54ff9640 1 -- 172.21.15.60:0/3120628111 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4c007980 con 0x7fdd580b3980 2024-03-27T21:42:13.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd5e58b640 5 monclient: authenticate success, global_id 5165 2024-03-27T21:42:13.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd5e58b640 10 monclient: _renew_subs 2024-03-27T21:42:13.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd5e58b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:13.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd5e58b640 1 -- 172.21.15.60:0/3120628111 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd50009a70 con 0x7fdd580b3980 2024-03-27T21:42:13.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd54ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:13.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd54ff9640 1 -- 172.21.15.60:0/3120628111 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c007cf0 con 0x7fdd580b3980 2024-03-27T21:42:13.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:13.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd54ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:13.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd54ff9640 10 monclient: dump: 2024-03-27T21:42:13.095 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:13.095 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.095 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.095 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.095 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:13.095 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.096 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:13.096 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:13.096 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:13.096 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd567fc640 4 set_mon_vals no callback set 2024-03-27T21:42:13.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd5e58b640 10 client.5165.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:13.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd5e58b640 10 monclient: _renew_subs 2024-03-27T21:42:13.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd5e58b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:13.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd5e58b640 1 -- 172.21.15.60:0/3120628111 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd58137640 con 0x7fdd580b3980 2024-03-27T21:42:13.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.075+0000 7fdd5e58b640 1 librados: init done 2024-03-27T21:42:13.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.076+0000 7fdd5e58b640 10 client.5165.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:13.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.076+0000 7fdd5e58b640 15 client.5165.objecter _session_command_op_assign -1 1 2024-03-27T21:42:13.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.076+0000 7fdd5e58b640 20 client.5165.objecter _get_session osd=-1 returning homeless 2024-03-27T21:42:13.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.076+0000 7fdd5e58b640 10 client.5165.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:13.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.076+0000 7fdd5e58b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:13.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.076+0000 7fdd5e58b640 1 -- 172.21.15.60:0/3120628111 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdd580b0620 con 0x7fdd580b3980 2024-03-27T21:42:13.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.076+0000 7fdd54ff9640 1 -- 172.21.15.60:0/3120628111 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fdd4c0113f0 con 0x7fdd580b3980 2024-03-27T21:42:13.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd54ff9640 1 --2- 172.21.15.60:0/3120628111 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdd4406c590 0x7fdd4406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd54ff9640 1 -- 172.21.15.60:0/3120628111 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fdd4c0892e0 con 0x7fdd580b3980 2024-03-27T21:42:13.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd54ff9640 10 client.5165.objecter ms_dispatch 0x7fdd58000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:13.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd54ff9640 3 client.5165.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:13.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd54ff9640 3 client.5165.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:13.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd57fff640 1 --2- 172.21.15.60:0/3120628111 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdd4406c590 0x7fdd4406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd57fff640 10 monclient: get_auth_request con 0x7fdd4406c590 auth_method 0 2024-03-27T21:42:13.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd54ff9640 10 client.5165.objecter checking command 1 2024-03-27T21:42:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd54ff9640 1 --2- 172.21.15.60:0/3120628111 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fdd440729d0 0x7fdd44074df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd54ff9640 20 client.5165.objecter _get_session s=0x7fdd44072300 osd=4 3 2024-03-27T21:42:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd54ff9640 20 client.5165.objecter put_session s=0x7fdd44072300 osd=4 3 2024-03-27T21:42:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd54ff9640 15 client.5165.objecter _session_command_op_remove -1 1 2024-03-27T21:42:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd54ff9640 20 client.5165.objecter _get_session s=0x7fdd44072300 osd=4 3 2024-03-27T21:42:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd54ff9640 20 client.5165.objecter get_session s=0x7fdd44072300 osd=4 3 2024-03-27T21:42:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd54ff9640 15 client.5165.objecter _session_command_op_assign 4 1 2024-03-27T21:42:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd54ff9640 20 client.5165.objecter put_session s=0x7fdd44072300 osd=4 4 2024-03-27T21:42:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd54ff9640 10 client.5165.objecter _send_command 1 2024-03-27T21:42:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd54ff9640 1 -- 172.21.15.60:0/3120628111 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdd440754c0 con 0x7fdd440729d0 2024-03-27T21:42:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd54ff9640 20 client.5165.objecter dump_active .. 0 homeless 2024-03-27T21:42:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd54ff9640 1 -- 172.21.15.60:0/3120628111 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd4c08b9a0 con 0x7fdd580b3980 2024-03-27T21:42:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd54ff9640 10 monclient: handle_get_version_reply finishing 1 version 92 2024-03-27T21:42:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd5cb28640 1 --2- 172.21.15.60:0/3120628111 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fdd440729d0 0x7fdd44074df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd5cb28640 10 monclient: get_auth_request con 0x7fdd440729d0 auth_method 0 2024-03-27T21:42:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd54ff9640 10 client.5165.objecter ms_handle_connect 0x7fdd4406c590 2024-03-27T21:42:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.077+0000 7fdd57fff640 1 --2- 172.21.15.60:0/3120628111 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdd4406c590 0x7fdd4406ea50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdd48006ca0 tx=0x7fdd48009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:13.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.078+0000 7fdd54ff9640 10 client.5165.objecter ms_handle_connect 0x7fdd440729d0 2024-03-27T21:42:13.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.078+0000 7fdd5cb28640 1 --2- 172.21.15.60:0/3120628111 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fdd440729d0 0x7fdd44074df0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:13.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.078+0000 7fdd54ff9640 1 -- 172.21.15.60:0/3120628111 <== osd.4 v2:172.21.15.191:6800/34332 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fdd440754c0 con 0x7fdd440729d0 2024-03-27T21:42:13.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.078+0000 7fdd54ff9640 10 client.5165.objecter ms_dispatch 0x7fdd58000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:42:13.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.078+0000 7fdd54ff9640 10 client.5165.objecter _finish_command 1 = system:0 2024-03-27T21:42:13.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.078+0000 7fdd54ff9640 20 client.5165.objecter put_session s=0x7fdd44072300 osd=4 4 2024-03-27T21:42:13.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.078+0000 7fdd54ff9640 15 client.5165.objecter _session_command_op_remove 4 1 2024-03-27T21:42:13.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.091+0000 7fdd5e58b640 10 client.5165.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-27T21:42:13.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.091+0000 7fdd5e58b640 15 client.5165.objecter _session_command_op_assign -1 2 2024-03-27T21:42:13.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.091+0000 7fdd5e58b640 20 client.5165.objecter _get_session s=0x7fdd44072300 osd=4 3 2024-03-27T21:42:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.091+0000 7fdd5e58b640 20 client.5165.objecter put_session s=0x7fdd44072300 osd=4 3 2024-03-27T21:42:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.091+0000 7fdd5e58b640 20 client.5165.objecter _get_session s=0x7fdd44072300 osd=4 3 2024-03-27T21:42:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.091+0000 7fdd5e58b640 15 client.5165.objecter _session_command_op_remove -1 2 2024-03-27T21:42:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.091+0000 7fdd5e58b640 20 client.5165.objecter get_session s=0x7fdd44072300 osd=4 3 2024-03-27T21:42:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.091+0000 7fdd5e58b640 15 client.5165.objecter _session_command_op_assign 4 2 2024-03-27T21:42:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.091+0000 7fdd5e58b640 20 client.5165.objecter put_session s=0x7fdd44072300 osd=4 4 2024-03-27T21:42:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.091+0000 7fdd5e58b640 10 client.5165.objecter _send_command 2 2024-03-27T21:42:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.091+0000 7fdd5e58b640 1 -- 172.21.15.60:0/3120628111 --> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fdd580670d0 con 0x7fdd440729d0 2024-03-27T21:42:13.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.091+0000 7fdd54ff9640 1 -- 172.21.15.60:0/3120628111 <== osd.4 v2:172.21.15.191:6800/34332 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fdd580670d0 con 0x7fdd440729d0 2024-03-27T21:42:13.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.091+0000 7fdd54ff9640 10 client.5165.objecter ms_dispatch 0x7fdd58000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:42:13.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.091+0000 7fdd54ff9640 10 client.5165.objecter _finish_command 2 = system:0 2024-03-27T21:42:13.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.091+0000 7fdd54ff9640 20 client.5165.objecter put_session s=0x7fdd44072300 osd=4 4 2024-03-27T21:42:13.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.091+0000 7fdd54ff9640 15 client.5165.objecter _session_command_op_remove 4 2 2024-03-27T21:42:13.104 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:42:13.105 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-27T21:42:13.105 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:42:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.092+0000 7fdd5e58b640 10 librados: watch_flush enter 2024-03-27T21:42:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.092+0000 7fdd5e58b640 10 librados: watch_flush exit 2024-03-27T21:42:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.092+0000 7fdd5e58b640 10 client.5165.objecter close_session for osd.4 2024-03-27T21:42:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.092+0000 7fdd5e58b640 1 -- 172.21.15.60:0/3120628111 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fdd440729d0 msgr2=0x7fdd44074df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.092+0000 7fdd5e58b640 1 --2- 172.21.15.60:0/3120628111 >> [v2:172.21.15.191:6800/34332,v1:172.21.15.191:6801/34332] conn(0x7fdd440729d0 0x7fdd44074df0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.092+0000 7fdd5e58b640 20 client.5165.objecter put_session s=0x7fdd44072300 osd=4 1 2024-03-27T21:42:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.092+0000 7fdd5e58b640 20 client.5165.objecter shutdown clearing up homeless session... 2024-03-27T21:42:13.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.092+0000 7fdd5e58b640 10 client.5165.objecter successfully canceled tick 2024-03-27T21:42:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.092+0000 7fdd5e58b640 1 -- 172.21.15.60:0/3120628111 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdd4406c590 msgr2=0x7fdd4406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.092+0000 7fdd5e58b640 1 --2- 172.21.15.60:0/3120628111 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fdd4406c590 0x7fdd4406ea50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdd48006ca0 tx=0x7fdd48009240 comp rx=0 tx=0).stop 2024-03-27T21:42:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.092+0000 7fdd5e58b640 10 monclient: shutdown 2024-03-27T21:42:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.092+0000 7fdd5e58b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.092+0000 7fdd5e58b640 1 -- 172.21.15.60:0/3120628111 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdd580b3980 msgr2=0x7fdd581d9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.092+0000 7fdd5e58b640 1 --2- 172.21.15.60:0/3120628111 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fdd580b3980 0x7fdd581d9730 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c00c7f0 tx=0x7fdd4c00ccb0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.092+0000 7fdd5cb28640 1 -- 172.21.15.60:0/3120628111 reap_dead start 2024-03-27T21:42:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.092+0000 7fdd5e58b640 1 -- 172.21.15.60:0/3120628111 shutdown_connections 2024-03-27T21:42:13.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.092+0000 7fdd5e58b640 1 -- 172.21.15.60:0/3120628111 >> 172.21.15.60:0/3120628111 conn(0x7fdd58071410 msgr2=0x7fdd58144760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:13.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.092+0000 7fdd5e58b640 1 -- 172.21.15.60:0/3120628111 shutdown_connections 2024-03-27T21:42:13.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.092+0000 7fdd5e58b640 1 -- 172.21.15.60:0/3120628111 wait complete. 2024-03-27T21:42:13.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.092+0000 7fdd5e58b640 1 librados: shutdown 2024-03-27T21:42:13.109 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-27T21:42:13.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.224+0000 7fcd4f373640 10 monclient: get_monmap_and_config 2024-03-27T21:42:13.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.224+0000 7fcd4f373640 10 monclient: build_initial_monmap 2024-03-27T21:42:13.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.224+0000 7fcd4f373640 10 monclient: monmap: 2024-03-27T21:42:13.212 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:13.212 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.212 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:13.225719+0000 2024-03-27T21:42:13.213 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:13.225719+0000 2024-03-27T21:42:13.213 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:13.213 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.213 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:13.214 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:13.214 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:13.214 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:13.215 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:13.215 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.225+0000 7fcd4f373640 1 Processor -- start 2024-03-27T21:42:13.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.225+0000 7fcd4f373640 1 -- start start 2024-03-27T21:42:13.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.225+0000 7fcd4f373640 10 monclient: init 2024-03-27T21:42:13.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4f373640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:13.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4f373640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-03-27T21:42:13.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4f373640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcd48071a50 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:13.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4f373640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcd4813ef20 0x7fcd480a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4f373640 10 monclient(hunting): picked mon.noname-c con 0x7fcd4813ef20 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:13.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4f373640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcd48141630 0x7fcd48141ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4f373640 10 monclient(hunting): picked mon.noname-b con 0x7fcd48141630 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:13.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4f373640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd480a8e90 con 0x7fcd48071a50 2024-03-27T21:42:13.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4f373640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4f373640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd48071e30 con 0x7fcd4813ef20 2024-03-27T21:42:13.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4f373640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4f373640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd480aa040 con 0x7fcd48141630 2024-03-27T21:42:13.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4f373640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:13.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4f373640 10 monclient(hunting): authenticate will time out at 974.447010s 2024-03-27T21:42:13.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4d10f640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcd48071a50 0x7fcd4813e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60304/0 (socket says 172.21.15.60:60304) 2024-03-27T21:42:13.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4d910640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcd48141630 0x7fcd48141ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4d10f640 1 -- 172.21.15.60:0/3836118924 learned_addr learned my addr 172.21.15.60:0/3836118924 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:13.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4d910640 10 monclient(hunting): get_auth_request con 0x7fcd48141630 auth_method 0 2024-03-27T21:42:13.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4d910640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:13.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4d910640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4d910640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4d910640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:13.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4d910640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:13.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4d910640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:13.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4c90e640 1 --2- 172.21.15.60:0/3836118924 >> v2:172.21.15.191:3300/0 conn(0x7fcd4813ef20 0x7fcd480a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd3ffff640 1 -- 172.21.15.60:0/3836118924 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511498375 0 0) 0x7fcd480a8e90 con 0x7fcd48071a50 2024-03-27T21:42:13.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd3ffff640 10 monclient(hunting): _negotiate 2024-03-27T21:42:13.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd3ffff640 10 monclient(hunting): my global_id is 5508 2024-03-27T21:42:13.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd3ffff640 1 -- 172.21.15.60:0/3836118924 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd34003520 con 0x7fcd48071a50 2024-03-27T21:42:13.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4c90e640 10 monclient(hunting): get_auth_request con 0x7fcd4813ef20 auth_method 0 2024-03-27T21:42:13.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4c90e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:13.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4c90e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4c90e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4d910640 10 monclient(hunting): handle_auth_done global_id 5168 payload 274 2024-03-27T21:42:13.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4d910640 1 -- 172.21.15.60:0/3836118924 >> v2:172.21.15.191:3300/0 conn(0x7fcd4813ef20 msgr2=0x7fcd480a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:13.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4d910640 1 --2- 172.21.15.60:0/3836118924 >> v2:172.21.15.191:3300/0 conn(0x7fcd4813ef20 0x7fcd480a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4d910640 1 -- 172.21.15.60:0/3836118924 >> v1:172.21.15.60:6789/0 conn(0x7fcd48071a50 legacy=0x7fcd4813e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4d910640 10 monclient: _finish_hunting 0 2024-03-27T21:42:13.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4d910640 1 monclient: found mon.noname-b 2024-03-27T21:42:13.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4d910640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:13.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4d910640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:42:13.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.226+0000 7fcd4d910640 1 -- 172.21.15.60:0/3836118924 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd480aa2b0 con 0x7fcd48141630 2024-03-27T21:42:13.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd4d910640 1 --2- 172.21.15.60:0/3836118924 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcd48141630 0x7fcd48141ac0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcd40009d90 tx=0x7fcd4002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=745bb28f8ee2cb94 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:13.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd3ffff640 1 -- 172.21.15.60:0/3836118924 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4002cc50 con 0x7fcd48141630 2024-03-27T21:42:13.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:13.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd3ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:13.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd3ffff640 10 monclient: dump: 2024-03-27T21:42:13.236 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:13.237 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.237 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.237 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.237 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:13.238 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.238 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:13.238 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:13.238 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:13.239 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd3ffff640 10 monclient: _finish_auth 0 2024-03-27T21:42:13.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd3ffff640 10 monclient: _check_auth_tickets 2024-03-27T21:42:13.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:13.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd4f373640 5 monclient: authenticate success, global_id 5168 2024-03-27T21:42:13.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd4f373640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:13.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd3ffff640 1 -- 172.21.15.60:0/3836118924 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd4002cdb0 con 0x7fcd48141630 2024-03-27T21:42:13.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:13.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd4f373640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:13.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd3ffff640 1 -- 172.21.15.60:0/3836118924 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd40034640 con 0x7fcd48141630 2024-03-27T21:42:13.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd4f373640 4 set_mon_vals no callback set 2024-03-27T21:42:13.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:13.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd3ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:13.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd3ffff640 10 monclient: dump: 2024-03-27T21:42:13.242 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:13.242 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.242 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.242 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.243 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:13.243 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.243 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:13.243 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:13.244 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:13.244 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd4f373640 10 monclient: shutdown 2024-03-27T21:42:13.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd4f373640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:13.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd4f373640 1 -- 172.21.15.60:0/3836118924 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcd48141630 msgr2=0x7fcd48141ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd4f373640 1 --2- 172.21.15.60:0/3836118924 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcd48141630 0x7fcd48141ac0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcd40009d90 tx=0x7fcd4002bea0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd4f373640 1 -- 172.21.15.60:0/3836118924 shutdown_connections 2024-03-27T21:42:13.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd4f373640 1 --2- 172.21.15.60:0/3836118924 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcd48141630 0x7fcd48141ac0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.227+0000 7fcd4f373640 1 --2- 172.21.15.60:0/3836118924 >> v2:172.21.15.191:3300/0 conn(0x7fcd4813ef20 0x7fcd480a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.228+0000 7fcd4f373640 1 -- 172.21.15.60:0/3836118924 >> 172.21.15.60:0/3836118924 conn(0x7fcd480b7300 msgr2=0x7fcd480b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:13.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.228+0000 7fcd4f373640 1 -- 172.21.15.60:0/3836118924 shutdown_connections 2024-03-27T21:42:13.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.228+0000 7fcd4f373640 1 -- 172.21.15.60:0/3836118924 wait complete. 2024-03-27T21:42:13.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.228+0000 7fcd4f373640 10 monclient: build_initial_monmap 2024-03-27T21:42:13.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.228+0000 7fcd4f373640 10 monclient: monmap: 2024-03-27T21:42:13.245 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:13.245 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:13.246 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:13.246 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:13.246 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:13.246 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:13.246 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:13.246 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:13.246 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:13.246 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.228+0000 7fcd4f373640 1 librados: starting msgr at 2024-03-27T21:42:13.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.228+0000 7fcd4f373640 1 librados: starting objecter 2024-03-27T21:42:13.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4f373640 1 Processor -- start 2024-03-27T21:42:13.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4f373640 1 -- start start 2024-03-27T21:42:13.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4f373640 1 librados: setting wanted keys 2024-03-27T21:42:13.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4f373640 1 librados: calling monclient init 2024-03-27T21:42:13.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4f373640 10 monclient: init 2024-03-27T21:42:13.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4f373640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:13.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4f373640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:42:13.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4f373640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd48071a50 0x7fcd48132fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4f373640 10 monclient(hunting): picked mon.noname-a con 0x7fcd48071a50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:13.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4f373640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd4813ef20 0x7fcd48133500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4f373640 10 monclient(hunting): picked mon.noname-b con 0x7fcd4813ef20 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:13.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4f373640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcd48141630 0x7fcd4813a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4f373640 10 monclient(hunting): picked mon.noname-c con 0x7fcd48141630 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:13.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4f373640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4f373640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd480ab310 con 0x7fcd48071a50 2024-03-27T21:42:13.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4f373640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4f373640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd48133a40 con 0x7fcd4813ef20 2024-03-27T21:42:13.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4f373640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4f373640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd48137680 con 0x7fcd48141630 2024-03-27T21:42:13.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4f373640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:13.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4f373640 10 monclient(hunting): authenticate will time out at 974.450618s 2024-03-27T21:42:13.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4d10f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd48071a50 0x7fcd48132fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4d910640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcd48141630 0x7fcd4813a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4d10f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd48071a50 0x7fcd48132fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59340/0 (socket says 172.21.15.60:59340) 2024-03-27T21:42:13.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4d910640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcd48141630 0x7fcd4813a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:51498/0 (socket says 172.21.15.60:51498) 2024-03-27T21:42:13.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4d10f640 1 -- 172.21.15.60:0/1425034771 learned_addr learned my addr 172.21.15.60:0/1425034771 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:13.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4d10f640 10 monclient(hunting): get_auth_request con 0x7fcd48071a50 auth_method 0 2024-03-27T21:42:13.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4d10f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:13.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4d10f640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4d10f640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4d910640 10 monclient(hunting): get_auth_request con 0x7fcd48141630 auth_method 0 2024-03-27T21:42:13.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4d910640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:13.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4d910640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4d910640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4c90e640 1 --2- 172.21.15.60:0/1425034771 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd4813ef20 0x7fcd48133500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4d10f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:13.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4d10f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:13.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4d10f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:13.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4d910640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:13.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4d910640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:13.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4d910640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:13.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4c90e640 10 monclient(hunting): get_auth_request con 0x7fcd4813ef20 auth_method 0 2024-03-27T21:42:13.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4c90e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:13.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4c90e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.229+0000 7fcd4c90e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4c90e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:13.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4c90e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:13.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4c90e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:13.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4d10f640 10 monclient(hunting): handle_auth_done global_id 5511 payload 824 2024-03-27T21:42:13.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4d10f640 1 -- 172.21.15.60:0/1425034771 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcd48141630 msgr2=0x7fcd4813a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4d10f640 1 --2- 172.21.15.60:0/1425034771 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcd48141630 0x7fcd4813a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4d10f640 1 -- 172.21.15.60:0/1425034771 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd4813ef20 msgr2=0x7fcd48133500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4d10f640 1 --2- 172.21.15.60:0/1425034771 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd4813ef20 0x7fcd48133500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4d10f640 10 monclient: _finish_hunting 0 2024-03-27T21:42:13.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4d10f640 1 monclient: found mon.noname-a 2024-03-27T21:42:13.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4d10f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:13.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4d10f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:13.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4d10f640 1 -- 172.21.15.60:0/1425034771 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd40009a70 con 0x7fcd48071a50 2024-03-27T21:42:13.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4d910640 1 --2- 172.21.15.60:0/1425034771 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcd48141630 0x7fcd4813a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:13.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4d10f640 1 --2- 172.21.15.60:0/1425034771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd48071a50 0x7fcd48132fc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fcd4400c460 tx=0x7fcd4400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:13.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd3dffb640 10 client.?.objecter ms_handle_connect 0x7fcd48071a50 2024-03-27T21:42:13.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd3dffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:13.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd3dffb640 1 -- 172.21.15.60:0/1425034771 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd44005ce0 con 0x7fcd48071a50 2024-03-27T21:42:13.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:13.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd3dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:13.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd3dffb640 10 monclient: dump: 2024-03-27T21:42:13.255 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:13.255 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.255 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.255 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.255 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:13.255 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.256 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:13.256 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:13.256 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:13.256 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd3dffb640 10 monclient: _finish_auth 0 2024-03-27T21:42:13.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd3dffb640 10 monclient: _check_auth_tickets 2024-03-27T21:42:13.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:13.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4f373640 5 monclient: authenticate success, global_id 5511 2024-03-27T21:42:13.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd3dffb640 1 -- 172.21.15.60:0/1425034771 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd44005e40 con 0x7fcd48071a50 2024-03-27T21:42:13.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4f373640 10 monclient: _renew_subs 2024-03-27T21:42:13.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4f373640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:13.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4f373640 1 -- 172.21.15.60:0/1425034771 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd48137310 con 0x7fcd48071a50 2024-03-27T21:42:13.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd3dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:13.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd3dffb640 1 -- 172.21.15.60:0/1425034771 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd44007290 con 0x7fcd48071a50 2024-03-27T21:42:13.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:13.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd3dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:13.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd3dffb640 10 monclient: dump: 2024-03-27T21:42:13.258 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:13.258 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.258 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.258 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.258 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:13.258 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.258 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:13.258 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:13.259 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:13.259 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd3f7fe640 4 set_mon_vals no callback set 2024-03-27T21:42:13.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4f373640 10 client.5511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:13.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4f373640 10 monclient: _renew_subs 2024-03-27T21:42:13.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4f373640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:13.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4f373640 1 -- 172.21.15.60:0/1425034771 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd4813a750 con 0x7fcd48071a50 2024-03-27T21:42:13.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.230+0000 7fcd4f373640 1 librados: init done 2024-03-27T21:42:13.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.231+0000 7fcd4f373640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:13.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.231+0000 7fcd4f373640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:13.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.231+0000 7fcd4f373640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:13.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.231+0000 7fcd4f373640 1 -- 172.21.15.60:0/1425034771 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd48067900 con 0x7fcd48071a50 2024-03-27T21:42:13.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.232+0000 7fcd3dffb640 1 -- 172.21.15.60:0/1425034771 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcd440073f0 con 0x7fcd48071a50 2024-03-27T21:42:13.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.232+0000 7fcd3dffb640 1 --2- 172.21.15.60:0/1425034771 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcd2c06c590 0x7fcd2c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.232+0000 7fcd3dffb640 1 -- 172.21.15.60:0/1425034771 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fcd440885e0 con 0x7fcd48071a50 2024-03-27T21:42:13.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.232+0000 7fcd3dffb640 10 client.5511.objecter ms_dispatch 0x7fcd48000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:13.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.232+0000 7fcd4c90e640 1 --2- 172.21.15.60:0/1425034771 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcd2c06c590 0x7fcd2c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.232+0000 7fcd3dffb640 3 client.5511.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:13.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.232+0000 7fcd3dffb640 3 client.5511.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:13.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.232+0000 7fcd4c90e640 10 monclient: get_auth_request con 0x7fcd2c06c590 auth_method 0 2024-03-27T21:42:13.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.232+0000 7fcd3dffb640 20 client.5511.objecter dump_active .. 0 homeless 2024-03-27T21:42:13.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.232+0000 7fcd4c90e640 1 --2- 172.21.15.60:0/1425034771 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcd2c06c590 0x7fcd2c06ea50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fcd4813f400 tx=0x7fcd38008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:13.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.232+0000 7fcd3dffb640 10 client.5511.objecter ms_handle_connect 0x7fcd2c06c590 2024-03-27T21:42:13.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.236+0000 7fcd3dffb640 1 -- 172.21.15.60:0/1425034771 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcd44053ad0 con 0x7fcd48071a50 2024-03-27T21:42:13.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.236+0000 7fcd3dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:13.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.236+0000 7fcd3dffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:13.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.415+0000 7fcd4f373640 1 -- 172.21.15.60:0/1425034771 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd480ab710 con 0x7fcd2c06c590 2024-03-27T21:42:13.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.415+0000 7fcd3dffb640 1 -- 172.21.15.60:0/1425034771 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcd480ab710 con 0x7fcd2c06c590 2024-03-27T21:42:13.401 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 4 to deep-scrub 2024-03-27T21:42:13.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.417+0000 7fcd4f373640 10 librados: watch_flush enter 2024-03-27T21:42:13.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.417+0000 7fcd4f373640 10 librados: watch_flush exit 2024-03-27T21:42:13.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.417+0000 7fcd4f373640 20 client.5511.objecter shutdown clearing up homeless session... 2024-03-27T21:42:13.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.417+0000 7fcd4f373640 10 client.5511.objecter successfully canceled tick 2024-03-27T21:42:13.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.417+0000 7fcd4f373640 1 -- 172.21.15.60:0/1425034771 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcd2c06c590 msgr2=0x7fcd2c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.417+0000 7fcd4f373640 1 --2- 172.21.15.60:0/1425034771 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcd2c06c590 0x7fcd2c06ea50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fcd4813f400 tx=0x7fcd38008040 comp rx=0 tx=0).stop 2024-03-27T21:42:13.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.417+0000 7fcd4f373640 10 monclient: shutdown 2024-03-27T21:42:13.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.417+0000 7fcd4f373640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:13.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.417+0000 7fcd4f373640 1 -- 172.21.15.60:0/1425034771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd48071a50 msgr2=0x7fcd48132fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.417+0000 7fcd4f373640 1 --2- 172.21.15.60:0/1425034771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd48071a50 0x7fcd48132fc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fcd4400c460 tx=0x7fcd4400c920 comp rx=0 tx=0).stop 2024-03-27T21:42:13.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.417+0000 7fcd4f373640 1 -- 172.21.15.60:0/1425034771 shutdown_connections 2024-03-27T21:42:13.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.417+0000 7fcd4f373640 1 --2- 172.21.15.60:0/1425034771 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fcd48141630 0x7fcd4813a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.417+0000 7fcd4f373640 1 --2- 172.21.15.60:0/1425034771 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fcd2c06c590 0x7fcd2c06ea50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.417+0000 7fcd4f373640 1 --2- 172.21.15.60:0/1425034771 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd4813ef20 0x7fcd48133500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.417+0000 7fcd4f373640 1 --2- 172.21.15.60:0/1425034771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd48071a50 0x7fcd48132fc0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.417+0000 7fcd4f373640 1 -- 172.21.15.60:0/1425034771 >> 172.21.15.60:0/1425034771 conn(0x7fcd480b7300 msgr2=0x7fcd4813e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:13.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.417+0000 7fcd4f373640 1 -- 172.21.15.60:0/1425034771 shutdown_connections 2024-03-27T21:42:13.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.417+0000 7fcd4f373640 1 -- 172.21.15.60:0/1425034771 wait complete. 2024-03-27T21:42:13.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.417+0000 7fcd4f373640 1 librados: shutdown 2024-03-27T21:42:13.414 INFO:tasks.ceph:Scrubbing osd.5 2024-03-27T21:42:13.414 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-27T21:42:13.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.532+0000 7f30f2f8b640 10 monclient: get_monmap_and_config 2024-03-27T21:42:13.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.532+0000 7f30f2f8b640 10 monclient: build_initial_monmap 2024-03-27T21:42:13.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.532+0000 7f30f2f8b640 10 monclient: monmap: 2024-03-27T21:42:13.518 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:13.519 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.519 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:13.533235+0000 2024-03-27T21:42:13.519 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:13.533235+0000 2024-03-27T21:42:13.519 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:13.519 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.519 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:13.519 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:13.519 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:13.519 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:13.520 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:13.520 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.532+0000 7f30f2f8b640 1 Processor -- start 2024-03-27T21:42:13.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.532+0000 7f30f2f8b640 1 -- start start 2024-03-27T21:42:13.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f2f8b640 10 monclient: init 2024-03-27T21:42:13.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f2f8b640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:13.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f2f8b640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-03-27T21:42:13.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f2f8b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f30ec071a40 0x7f30ec0a8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f2f8b640 10 monclient(hunting): picked mon.noname-b con 0x7f30ec071a40 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:13.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f2f8b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f30ec13e810 0x7f30ec13ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f2f8b640 10 monclient(hunting): picked mon.noname-c con 0x7f30ec13e810 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:13.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f2f8b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f30ec144a30 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:13.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f2f8b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ec13f190 con 0x7f30ec144a30 2024-03-27T21:42:13.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f2f8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f2f8b640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30ec071e20 con 0x7f30ec13e810 2024-03-27T21:42:13.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f2f8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f2f8b640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec0a9060 con 0x7f30ec071a40 2024-03-27T21:42:13.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f2f8b640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:13.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f2f8b640 10 monclient(hunting): authenticate will time out at 974.754501s 2024-03-27T21:42:13.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f0d27640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f30ec071a40 0x7f30ec0a8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f1528640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f30ec144a30 0x7f30ec144ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60318/0 (socket says 172.21.15.60:60318) 2024-03-27T21:42:13.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f1528640 1 -- 172.21.15.60:0/2756144375 learned_addr learned my addr 172.21.15.60:0/2756144375 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:13.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f0d27640 10 monclient(hunting): get_auth_request con 0x7f30ec071a40 auth_method 0 2024-03-27T21:42:13.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f0d27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f0d27640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f0d27640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30ebfff640 1 --2- 172.21.15.60:0/2756144375 >> v2:172.21.15.191:3300/0 conn(0x7f30ec13e810 0x7f30ec13ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30ebfff640 10 monclient(hunting): get_auth_request con 0x7f30ec13e810 auth_method 0 2024-03-27T21:42:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f0d27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f0d27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30f0d27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.533+0000 7f30eb7fe640 1 -- 172.21.15.60:0/2756144375 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409139492 0 0) 0x7f30e4001490 con 0x7f30ec144a30 2024-03-27T21:42:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30eb7fe640 10 monclient(hunting): _negotiate 2024-03-27T21:42:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30eb7fe640 10 monclient(hunting): my global_id is 5514 2024-03-27T21:42:13.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30eb7fe640 1 -- 172.21.15.60:0/2756144375 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30dc003520 con 0x7f30ec144a30 2024-03-27T21:42:13.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30f0d27640 10 monclient(hunting): handle_auth_done global_id 5174 payload 274 2024-03-27T21:42:13.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30f0d27640 1 -- 172.21.15.60:0/2756144375 >> v2:172.21.15.191:3300/0 conn(0x7f30ec13e810 msgr2=0x7f30ec13ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:13.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30f0d27640 1 --2- 172.21.15.60:0/2756144375 >> v2:172.21.15.191:3300/0 conn(0x7f30ec13e810 0x7f30ec13ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30f0d27640 1 -- 172.21.15.60:0/2756144375 >> v1:172.21.15.60:6789/0 conn(0x7f30ec144a30 legacy=0x7f30ec144ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30f0d27640 10 monclient: _finish_hunting 0 2024-03-27T21:42:13.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30f0d27640 1 monclient: found mon.noname-b 2024-03-27T21:42:13.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30f0d27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:13.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30f0d27640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:42:13.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30f0d27640 1 -- 172.21.15.60:0/2756144375 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ec0aa320 con 0x7f30ec071a40 2024-03-27T21:42:13.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30f0d27640 1 --2- 172.21.15.60:0/2756144375 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f30ec071a40 0x7f30ec0a8770 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f30e00097a0 tx=0x7f30e002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcf8e642bdca4b85 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:13.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30eb7fe640 1 -- 172.21.15.60:0/2756144375 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e002f050 con 0x7f30ec071a40 2024-03-27T21:42:13.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:13.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30eb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:13.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30eb7fe640 10 monclient: dump: 2024-03-27T21:42:13.536 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:13.536 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.536 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.537 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.537 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:13.537 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.537 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:13.537 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:13.537 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:13.537 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30eb7fe640 10 monclient: _finish_auth 0 2024-03-27T21:42:13.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30eb7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:42:13.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:13.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30f2f8b640 5 monclient: authenticate success, global_id 5174 2024-03-27T21:42:13.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30f2f8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:13.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30eb7fe640 1 -- 172.21.15.60:0/2756144375 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e0031700 con 0x7f30ec071a40 2024-03-27T21:42:13.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:13.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30eb7fe640 1 -- 172.21.15.60:0/2756144375 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e0031a70 con 0x7f30ec071a40 2024-03-27T21:42:13.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30f2f8b640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:13.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30f2f8b640 4 set_mon_vals no callback set 2024-03-27T21:42:13.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.534+0000 7f30eb7fe640 10 monclient: dump: 2024-03-27T21:42:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:13.540 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:13.540 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:13.540 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.535+0000 7f30f2f8b640 10 monclient: shutdown 2024-03-27T21:42:13.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.535+0000 7f30f2f8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:13.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.535+0000 7f30f2f8b640 1 -- 172.21.15.60:0/2756144375 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f30ec071a40 msgr2=0x7f30ec0a8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.535+0000 7f30f2f8b640 1 --2- 172.21.15.60:0/2756144375 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f30ec071a40 0x7f30ec0a8770 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f30e00097a0 tx=0x7f30e002b7a0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.535+0000 7f30f2f8b640 1 -- 172.21.15.60:0/2756144375 shutdown_connections 2024-03-27T21:42:13.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.535+0000 7f30f2f8b640 1 --2- 172.21.15.60:0/2756144375 >> v2:172.21.15.191:3300/0 conn(0x7f30ec13e810 0x7f30ec13ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.535+0000 7f30f2f8b640 1 --2- 172.21.15.60:0/2756144375 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f30ec071a40 0x7f30ec0a8770 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.535+0000 7f30f2f8b640 1 -- 172.21.15.60:0/2756144375 >> 172.21.15.60:0/2756144375 conn(0x7f30ec0b7300 msgr2=0x7f30ec0b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:13.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.535+0000 7f30f2f8b640 1 -- 172.21.15.60:0/2756144375 shutdown_connections 2024-03-27T21:42:13.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.535+0000 7f30f2f8b640 1 -- 172.21.15.60:0/2756144375 wait complete. 2024-03-27T21:42:13.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.536+0000 7f30f2f8b640 10 monclient: build_initial_monmap 2024-03-27T21:42:13.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.536+0000 7f30f2f8b640 10 monclient: monmap: 2024-03-27T21:42:13.541 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:13.541 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:13.541 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:13.542 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:13.542 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:13.542 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:13.542 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:13.542 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:13.542 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:13.542 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.536+0000 7f30f2f8b640 1 librados: starting msgr at 2024-03-27T21:42:13.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.536+0000 7f30f2f8b640 1 librados: starting objecter 2024-03-27T21:42:13.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.536+0000 7f30f2f8b640 1 Processor -- start 2024-03-27T21:42:13.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.536+0000 7f30f2f8b640 1 -- start start 2024-03-27T21:42:13.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.536+0000 7f30f2f8b640 1 librados: setting wanted keys 2024-03-27T21:42:13.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.536+0000 7f30f2f8b640 1 librados: calling monclient init 2024-03-27T21:42:13.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.536+0000 7f30f2f8b640 10 monclient: init 2024-03-27T21:42:13.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:13.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:42:13.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f30ec071a40 0x7f30ec140c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 10 monclient(hunting): picked mon.noname-c con 0x7f30ec071a40 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:13.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30ec13e810 0x7f30ec13fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 10 monclient(hunting): picked mon.noname-b con 0x7f30ec13e810 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:13.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30ec144a30 0x7f30ec135000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 10 monclient(hunting): picked mon.noname-a con 0x7f30ec144a30 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:13.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec13f3a0 con 0x7f30ec144a30 2024-03-27T21:42:13.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec138c20 con 0x7f30ec13e810 2024-03-27T21:42:13.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec141210 con 0x7f30ec071a40 2024-03-27T21:42:13.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:13.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 10 monclient(hunting): authenticate will time out at 974.758130s 2024-03-27T21:42:13.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f0d27640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f30ec071a40 0x7f30ec140c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f1528640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30ec144a30 0x7f30ec135000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f0d27640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f30ec071a40 0x7f30ec140c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:51504/0 (socket says 172.21.15.60:51504) 2024-03-27T21:42:13.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f0d27640 1 -- 172.21.15.60:0/4100467499 learned_addr learned my addr 172.21.15.60:0/4100467499 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:13.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f0d27640 10 monclient(hunting): get_auth_request con 0x7f30ec071a40 auth_method 0 2024-03-27T21:42:13.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f0d27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:13.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f0d27640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f0d27640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f1528640 10 monclient(hunting): get_auth_request con 0x7f30ec144a30 auth_method 0 2024-03-27T21:42:13.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f1528640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:13.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f1528640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f1528640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f1528640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:13.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f1528640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:13.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f1528640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:13.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f0d27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:13.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f0d27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:13.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f0d27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:13.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30ebfff640 1 --2- 172.21.15.60:0/4100467499 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30ec13e810 0x7f30ec13fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30ebfff640 10 monclient(hunting): get_auth_request con 0x7f30ec13e810 auth_method 0 2024-03-27T21:42:13.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:13.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f1528640 10 monclient(hunting): handle_auth_done global_id 5517 payload 824 2024-03-27T21:42:13.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f1528640 1 -- 172.21.15.60:0/4100467499 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f30ec071a40 msgr2=0x7f30ec140c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f1528640 1 --2- 172.21.15.60:0/4100467499 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f30ec071a40 0x7f30ec140c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f1528640 1 -- 172.21.15.60:0/4100467499 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30ec13e810 msgr2=0x7f30ec13fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f1528640 1 --2- 172.21.15.60:0/4100467499 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30ec13e810 0x7f30ec13fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f1528640 10 monclient: _finish_hunting 0 2024-03-27T21:42:13.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f1528640 1 monclient: found mon.noname-a 2024-03-27T21:42:13.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f1528640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:13.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f1528640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:13.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f1528640 1 -- 172.21.15.60:0/4100467499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e0009400 con 0x7f30ec144a30 2024-03-27T21:42:13.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30ebfff640 1 --2- 172.21.15.60:0/4100467499 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30ec13e810 0x7f30ec13fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:42:13.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f1528640 1 --2- 172.21.15.60:0/4100467499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30ec144a30 0x7f30ec135000 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f30e400caa0 tx=0x7f30e400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:13.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30e97fa640 10 client.?.objecter ms_handle_connect 0x7f30ec144a30 2024-03-27T21:42:13.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30e97fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:13.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30e97fa640 1 -- 172.21.15.60:0/4100467499 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e400ac00 con 0x7f30ec144a30 2024-03-27T21:42:13.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:13.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30e97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:13.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30e97fa640 10 monclient: dump: 2024-03-27T21:42:13.550 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:13.550 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.550 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.550 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.550 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:13.550 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.551 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:13.551 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:13.551 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:13.551 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30e97fa640 10 monclient: _finish_auth 0 2024-03-27T21:42:13.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30e97fa640 10 monclient: _check_auth_tickets 2024-03-27T21:42:13.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:13.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30e97fa640 1 -- 172.21.15.60:0/4100467499 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e400ad60 con 0x7f30ec144a30 2024-03-27T21:42:13.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 5 monclient: authenticate success, global_id 5517 2024-03-27T21:42:13.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 10 monclient: _renew_subs 2024-03-27T21:42:13.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:13.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 1 -- 172.21.15.60:0/4100467499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ec140430 con 0x7f30ec144a30 2024-03-27T21:42:13.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30e97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:13.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30e97fa640 1 -- 172.21.15.60:0/4100467499 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e4012620 con 0x7f30ec144a30 2024-03-27T21:42:13.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:13.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30e97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:13.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30e97fa640 10 monclient: dump: 2024-03-27T21:42:13.552 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:13.553 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.553 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.553 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.553 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:13.553 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.553 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:13.553 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:13.553 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:13.554 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 10 client.5517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:13.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 10 monclient: _renew_subs 2024-03-27T21:42:13.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30eaffd640 4 set_mon_vals no callback set 2024-03-27T21:42:13.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:13.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 1 -- 172.21.15.60:0/4100467499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ec135720 con 0x7f30ec144a30 2024-03-27T21:42:13.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.537+0000 7f30f2f8b640 1 librados: init done 2024-03-27T21:42:13.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.538+0000 7f30f2f8b640 10 client.5517.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:13.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.538+0000 7f30f2f8b640 15 client.5517.objecter _session_command_op_assign -1 1 2024-03-27T21:42:13.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.538+0000 7f30f2f8b640 20 client.5517.objecter _get_session osd=-1 returning homeless 2024-03-27T21:42:13.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.538+0000 7f30f2f8b640 10 client.5517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:13.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.538+0000 7f30f2f8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:13.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.538+0000 7f30f2f8b640 1 -- 172.21.15.60:0/4100467499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f30ec0a8cb0 con 0x7f30ec144a30 2024-03-27T21:42:13.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.539+0000 7f30e97fa640 1 -- 172.21.15.60:0/4100467499 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f30e4007070 con 0x7f30ec144a30 2024-03-27T21:42:13.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.539+0000 7f30e97fa640 1 --2- 172.21.15.60:0/4100467499 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f30d006c590 0x7f30d006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.539+0000 7f30e97fa640 1 -- 172.21.15.60:0/4100467499 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f30e4088ae0 con 0x7f30ec144a30 2024-03-27T21:42:13.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.539+0000 7f30e97fa640 10 client.5517.objecter ms_dispatch 0x7f30ec000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:13.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.539+0000 7f30e97fa640 3 client.5517.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:13.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.539+0000 7f30e97fa640 3 client.5517.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:13.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.539+0000 7f30f0d27640 1 --2- 172.21.15.60:0/4100467499 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f30d006c590 0x7f30d006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.539+0000 7f30f0d27640 10 monclient: get_auth_request con 0x7f30d006c590 auth_method 0 2024-03-27T21:42:13.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.539+0000 7f30e97fa640 10 client.5517.objecter checking command 1 2024-03-27T21:42:13.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.539+0000 7f30e97fa640 1 --2- 172.21.15.60:0/4100467499 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f30d00729d0 0x7f30d0074df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.539+0000 7f30e97fa640 20 client.5517.objecter _get_session s=0x7f30d0072300 osd=5 3 2024-03-27T21:42:13.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.539+0000 7f30e97fa640 20 client.5517.objecter put_session s=0x7f30d0072300 osd=5 3 2024-03-27T21:42:13.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.539+0000 7f30e97fa640 15 client.5517.objecter _session_command_op_remove -1 1 2024-03-27T21:42:13.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.539+0000 7f30e97fa640 20 client.5517.objecter _get_session s=0x7f30d0072300 osd=5 3 2024-03-27T21:42:13.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.539+0000 7f30e97fa640 20 client.5517.objecter get_session s=0x7f30d0072300 osd=5 3 2024-03-27T21:42:13.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.539+0000 7f30e97fa640 15 client.5517.objecter _session_command_op_assign 5 1 2024-03-27T21:42:13.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.539+0000 7f30e97fa640 20 client.5517.objecter put_session s=0x7f30d0072300 osd=5 4 2024-03-27T21:42:13.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.539+0000 7f30e97fa640 10 client.5517.objecter _send_command 1 2024-03-27T21:42:13.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.539+0000 7f30e97fa640 1 -- 172.21.15.60:0/4100467499 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30d00754c0 con 0x7f30d00729d0 2024-03-27T21:42:13.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.539+0000 7f30e97fa640 20 client.5517.objecter dump_active .. 0 homeless 2024-03-27T21:42:13.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.540+0000 7f30e97fa640 1 -- 172.21.15.60:0/4100467499 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30e408d2c0 con 0x7f30ec144a30 2024-03-27T21:42:13.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.540+0000 7f30e97fa640 10 monclient: handle_get_version_reply finishing 1 version 92 2024-03-27T21:42:13.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.540+0000 7f30e97fa640 10 client.5517.objecter ms_handle_connect 0x7f30d006c590 2024-03-27T21:42:13.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.540+0000 7f30f0d27640 1 --2- 172.21.15.60:0/4100467499 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f30d006c590 0x7f30d006ea50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f30e00097a0 tx=0x7f30e0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:13.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.540+0000 7f30ebfff640 1 --2- 172.21.15.60:0/4100467499 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f30d00729d0 0x7f30d0074df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.540+0000 7f30ebfff640 10 monclient: get_auth_request con 0x7f30d00729d0 auth_method 0 2024-03-27T21:42:13.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.540+0000 7f30e97fa640 10 client.5517.objecter ms_handle_connect 0x7f30d00729d0 2024-03-27T21:42:13.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.540+0000 7f30ebfff640 1 --2- 172.21.15.60:0/4100467499 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f30d00729d0 0x7f30d0074df0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:13.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.541+0000 7f30e97fa640 1 -- 172.21.15.60:0/4100467499 <== osd.5 v2:172.21.15.191:6809/34333 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f30d00754c0 con 0x7f30d00729d0 2024-03-27T21:42:13.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.541+0000 7f30e97fa640 10 client.5517.objecter ms_dispatch 0x7f30ec000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:42:13.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.541+0000 7f30e97fa640 10 client.5517.objecter _finish_command 1 = system:0 2024-03-27T21:42:13.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.541+0000 7f30e97fa640 20 client.5517.objecter put_session s=0x7f30d0072300 osd=5 4 2024-03-27T21:42:13.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.541+0000 7f30e97fa640 15 client.5517.objecter _session_command_op_remove 5 1 2024-03-27T21:42:13.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 10 client.5517.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-27T21:42:13.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 15 client.5517.objecter _session_command_op_assign -1 2 2024-03-27T21:42:13.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 20 client.5517.objecter _get_session s=0x7f30d0072300 osd=5 3 2024-03-27T21:42:13.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 20 client.5517.objecter put_session s=0x7f30d0072300 osd=5 3 2024-03-27T21:42:13.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 20 client.5517.objecter _get_session s=0x7f30d0072300 osd=5 3 2024-03-27T21:42:13.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 15 client.5517.objecter _session_command_op_remove -1 2 2024-03-27T21:42:13.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 20 client.5517.objecter get_session s=0x7f30d0072300 osd=5 3 2024-03-27T21:42:13.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 15 client.5517.objecter _session_command_op_assign 5 2 2024-03-27T21:42:13.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 20 client.5517.objecter put_session s=0x7f30d0072300 osd=5 4 2024-03-27T21:42:13.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 10 client.5517.objecter _send_command 2 2024-03-27T21:42:13.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 1 -- 172.21.15.60:0/4100467499 --> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f30ec0a9580 con 0x7f30d00729d0 2024-03-27T21:42:13.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30e97fa640 1 -- 172.21.15.60:0/4100467499 <== osd.5 v2:172.21.15.191:6809/34333 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f30d80098c0 con 0x7f30d00729d0 2024-03-27T21:42:13.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30e97fa640 10 client.5517.objecter ms_dispatch 0x7f30ec000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:42:13.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30e97fa640 10 client.5517.objecter _finish_command 2 = system:0 2024-03-27T21:42:13.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30e97fa640 20 client.5517.objecter put_session s=0x7f30d0072300 osd=5 4 2024-03-27T21:42:13.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30e97fa640 15 client.5517.objecter _session_command_op_remove 5 2 2024-03-27T21:42:13.561 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:42:13.562 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-27T21:42:13.562 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:42:13.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 10 librados: watch_flush enter 2024-03-27T21:42:13.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 10 librados: watch_flush exit 2024-03-27T21:42:13.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 10 client.5517.objecter close_session for osd.5 2024-03-27T21:42:13.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 1 -- 172.21.15.60:0/4100467499 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f30d00729d0 msgr2=0x7f30d0074df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 1 --2- 172.21.15.60:0/4100467499 >> [v2:172.21.15.191:6809/34333,v1:172.21.15.191:6811/34333] conn(0x7f30d00729d0 0x7f30d0074df0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 20 client.5517.objecter put_session s=0x7f30d0072300 osd=5 1 2024-03-27T21:42:13.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 20 client.5517.objecter shutdown clearing up homeless session... 2024-03-27T21:42:13.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 10 client.5517.objecter successfully canceled tick 2024-03-27T21:42:13.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 1 -- 172.21.15.60:0/4100467499 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f30d006c590 msgr2=0x7f30d006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 1 --2- 172.21.15.60:0/4100467499 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f30d006c590 0x7f30d006ea50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f30e00097a0 tx=0x7f30e0008030 comp rx=0 tx=0).stop 2024-03-27T21:42:13.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 10 monclient: shutdown 2024-03-27T21:42:13.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:13.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 1 -- 172.21.15.60:0/4100467499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30ec144a30 msgr2=0x7f30ec135000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 1 --2- 172.21.15.60:0/4100467499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30ec144a30 0x7f30ec135000 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f30e400caa0 tx=0x7f30e400cf60 comp rx=0 tx=0).stop 2024-03-27T21:42:13.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f1528640 1 -- 172.21.15.60:0/4100467499 reap_dead start 2024-03-27T21:42:13.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 1 -- 172.21.15.60:0/4100467499 shutdown_connections 2024-03-27T21:42:13.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 1 -- 172.21.15.60:0/4100467499 >> 172.21.15.60:0/4100467499 conn(0x7f30ec0b7300 msgr2=0x7f30ec144150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:13.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 1 -- 172.21.15.60:0/4100467499 shutdown_connections 2024-03-27T21:42:13.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.554+0000 7f30f2f8b640 1 -- 172.21.15.60:0/4100467499 wait complete. 2024-03-27T21:42:13.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.555+0000 7f30f2f8b640 1 librados: shutdown 2024-03-27T21:42:13.566 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-27T21:42:13.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.681+0000 7f6909e09640 10 monclient: get_monmap_and_config 2024-03-27T21:42:13.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.681+0000 7f6909e09640 10 monclient: build_initial_monmap 2024-03-27T21:42:13.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.681+0000 7f6909e09640 10 monclient: monmap: 2024-03-27T21:42:13.668 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:13.668 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.668 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:13.682319+0000 2024-03-27T21:42:13.668 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:13.682319+0000 2024-03-27T21:42:13.668 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:13.668 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.668 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:13.669 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:13.669 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:13.669 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:13.669 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:13.669 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.681+0000 7f6909e09640 1 Processor -- start 2024-03-27T21:42:13.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f6909e09640 1 -- start start 2024-03-27T21:42:13.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f6909e09640 10 monclient: init 2024-03-27T21:42:13.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f6909e09640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:13.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f6909e09640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-03-27T21:42:13.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f6909e09640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6904071a50 0x7f69040a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f6909e09640 10 monclient(hunting): picked mon.noname-a con 0x7f6904071a50 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:13.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f6909e09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f690413e810 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:13.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f6909e09640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f690413ec50 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:13.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f6909e09640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69040a8e90 con 0x7f690413ec50 2024-03-27T21:42:13.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f6909e09640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69040aa040 con 0x7f690413e810 2024-03-27T21:42:13.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f6909e09640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f6909e09640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6904071e30 con 0x7f6904071a50 2024-03-27T21:42:13.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f6909e09640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:13.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f6909e09640 10 monclient(hunting): authenticate will time out at 974.903546s 2024-03-27T21:42:13.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f69037fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6904071a50 0x7f69040a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f69037fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6904071a50 0x7f69040a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59368/0 (socket says 172.21.15.60:59368) 2024-03-27T21:42:13.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f69037fe640 1 -- 172.21.15.60:0/2852538957 learned_addr learned my addr 172.21.15.60:0/2852538957 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:13.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f69037fe640 10 monclient(hunting): get_auth_request con 0x7f6904071a50 auth_method 0 2024-03-27T21:42:13.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f69037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:13.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f69037fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f69037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f69037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:13.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f69037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:13.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f69037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:13.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.682+0000 7f69037fe640 10 monclient(hunting): handle_auth_done global_id 5520 payload 274 2024-03-27T21:42:13.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f69037fe640 1 -- 172.21.15.60:0/2852538957 >> v1:172.21.15.191:6789/0 conn(0x7f690413e810 legacy=0x7f6904141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:13.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f69037fe640 1 -- 172.21.15.60:0/2852538957 >> v1:172.21.15.60:6789/0 conn(0x7f690413ec50 legacy=0x7f6904141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f69037fe640 10 monclient: _finish_hunting 0 2024-03-27T21:42:13.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f69037fe640 1 monclient: found mon.noname-a 2024-03-27T21:42:13.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f69037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:13.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f69037fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:13.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f69037fe640 1 -- 172.21.15.60:0/2852538957 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69040aa2b0 con 0x7f6904071a50 2024-03-27T21:42:13.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f69037fe640 1 --2- 172.21.15.60:0/2852538957 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6904071a50 0x7f69040a87e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f68f4009870 tx=0x7f68f402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e3e8ad5bf200489 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:13.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f69027fc640 1 -- 172.21.15.60:0/2852538957 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f4030860 con 0x7f6904071a50 2024-03-27T21:42:13.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f69027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:13.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f69027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:13.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f69027fc640 10 monclient: dump: 2024-03-27T21:42:13.681 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:13.681 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.681 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.681 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.681 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:13.682 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.682 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:13.682 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:13.682 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:13.682 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f69027fc640 10 monclient: _finish_auth 0 2024-03-27T21:42:13.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f69027fc640 10 monclient: _check_auth_tickets 2024-03-27T21:42:13.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f69027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:13.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f69027fc640 1 -- 172.21.15.60:0/2852538957 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f40309c0 con 0x7f6904071a50 2024-03-27T21:42:13.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f6909e09640 5 monclient: authenticate success, global_id 5520 2024-03-27T21:42:13.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f6909e09640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:13.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f69027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:13.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f6909e09640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:13.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f69027fc640 1 -- 172.21.15.60:0/2852538957 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f4030d50 con 0x7f6904071a50 2024-03-27T21:42:13.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f69027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:13.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f6909e09640 4 set_mon_vals no callback set 2024-03-27T21:42:13.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f69027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:13.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.683+0000 7f69027fc640 10 monclient: dump: 2024-03-27T21:42:13.684 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:13.684 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.685 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.685 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.685 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:13.685 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.685 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:13.685 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:13.685 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:13.686 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.684+0000 7f6909e09640 10 monclient: shutdown 2024-03-27T21:42:13.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.684+0000 7f6909e09640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:13.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.684+0000 7f6909e09640 1 -- 172.21.15.60:0/2852538957 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6904071a50 msgr2=0x7f69040a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.684+0000 7f6909e09640 1 --2- 172.21.15.60:0/2852538957 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6904071a50 0x7f69040a87e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f68f4009870 tx=0x7f68f402b9a0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.684+0000 7f6909e09640 1 -- 172.21.15.60:0/2852538957 shutdown_connections 2024-03-27T21:42:13.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.684+0000 7f6909e09640 1 --2- 172.21.15.60:0/2852538957 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6904071a50 0x7f69040a87e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.684+0000 7f6909e09640 1 -- 172.21.15.60:0/2852538957 >> 172.21.15.60:0/2852538957 conn(0x7f69040b7300 msgr2=0x7f69040b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:13.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.684+0000 7f6909e09640 1 -- 172.21.15.60:0/2852538957 shutdown_connections 2024-03-27T21:42:13.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.684+0000 7f6909e09640 1 -- 172.21.15.60:0/2852538957 wait complete. 2024-03-27T21:42:13.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.685+0000 7f6909e09640 10 monclient: build_initial_monmap 2024-03-27T21:42:13.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.685+0000 7f6909e09640 10 monclient: monmap: 2024-03-27T21:42:13.687 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:13.687 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:13.687 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:13.687 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:13.688 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:13.688 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:13.688 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:13.688 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:13.688 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:13.688 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.685+0000 7f6909e09640 1 librados: starting msgr at 2024-03-27T21:42:13.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.685+0000 7f6909e09640 1 librados: starting objecter 2024-03-27T21:42:13.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.685+0000 7f6909e09640 1 Processor -- start 2024-03-27T21:42:13.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.685+0000 7f6909e09640 1 -- start start 2024-03-27T21:42:13.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.685+0000 7f6909e09640 1 librados: setting wanted keys 2024-03-27T21:42:13.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.685+0000 7f6909e09640 1 librados: calling monclient init 2024-03-27T21:42:13.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.685+0000 7f6909e09640 10 monclient: init 2024-03-27T21:42:13.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6909e09640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:13.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6909e09640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:42:13.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6909e09640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6904071a50 0x7f69040aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6909e09640 10 monclient(hunting): picked mon.noname-c con 0x7f6904071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:13.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6909e09640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f690413e810 0x7f69041388d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6909e09640 10 monclient(hunting): picked mon.noname-b con 0x7f690413e810 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:13.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6909e09640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f690413ec50 0x7f69041310a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6909e09640 10 monclient(hunting): picked mon.noname-a con 0x7f690413ec50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:13.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6909e09640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6909e09640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690413b010 con 0x7f690413ec50 2024-03-27T21:42:13.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6909e09640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6909e09640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6904138e10 con 0x7f690413e810 2024-03-27T21:42:13.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6909e09640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6909e09640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6904137d00 con 0x7f6904071a50 2024-03-27T21:42:13.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6909e09640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:13.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6909e09640 10 monclient(hunting): authenticate will time out at 974.907118s 2024-03-27T21:42:13.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f69037fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6904071a50 0x7f69040aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6903fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f690413ec50 0x7f69041310a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f69037fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6904071a50 0x7f69040aaa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:51510/0 (socket says 172.21.15.60:51510) 2024-03-27T21:42:13.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f69037fe640 1 -- 172.21.15.60:0/213448119 learned_addr learned my addr 172.21.15.60:0/213448119 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:13.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f69037fe640 10 monclient(hunting): get_auth_request con 0x7f6904071a50 auth_method 0 2024-03-27T21:42:13.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6903fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f690413ec50 0x7f69041310a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59384/0 (socket says 172.21.15.60:59384) 2024-03-27T21:42:13.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f69037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:13.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f69037fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f69037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6903fff640 10 monclient(hunting): get_auth_request con 0x7f690413ec50 auth_method 0 2024-03-27T21:42:13.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6903fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:13.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6903fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6903fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6903fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:13.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6903fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:13.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6903fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:13.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f69037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:13.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f69037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:13.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6902ffd640 1 --2- 172.21.15.60:0/213448119 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f690413e810 0x7f69041388d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f69037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:13.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6902ffd640 10 monclient(hunting): get_auth_request con 0x7f690413e810 auth_method 0 2024-03-27T21:42:13.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6902ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:13.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6902ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6902ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6902ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:13.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6902ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:13.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6902ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:13.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f69037fe640 10 monclient(hunting): handle_auth_done global_id 5180 payload 824 2024-03-27T21:42:13.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f69037fe640 1 -- 172.21.15.60:0/213448119 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f690413e810 msgr2=0x7f69041388d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f69037fe640 1 --2- 172.21.15.60:0/213448119 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f690413e810 0x7f69041388d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f69037fe640 1 -- 172.21.15.60:0/213448119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f690413ec50 msgr2=0x7f69041310a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f69037fe640 1 --2- 172.21.15.60:0/213448119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f690413ec50 0x7f69041310a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f69037fe640 10 monclient: _finish_hunting 0 2024-03-27T21:42:13.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f69037fe640 1 monclient: found mon.noname-c 2024-03-27T21:42:13.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f69037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:13.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f69037fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:13.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f69037fe640 1 -- 172.21.15.60:0/213448119 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f40094d0 con 0x7f6904071a50 2024-03-27T21:42:13.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6903fff640 1 --2- 172.21.15.60:0/213448119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f690413ec50 0x7f69041310a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:13.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f69037fe640 1 --2- 172.21.15.60:0/213448119 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6904071a50 0x7f69040aaa70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f68f4009870 tx=0x7f68f4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:13.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f68ebfff640 10 client.?.objecter ms_handle_connect 0x7f6904071a50 2024-03-27T21:42:13.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f68ebfff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:13.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f68ebfff640 1 -- 172.21.15.60:0/213448119 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f4030810 con 0x7f6904071a50 2024-03-27T21:42:13.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f68ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:13.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f68ebfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:13.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f68ebfff640 10 monclient: dump: 2024-03-27T21:42:13.696 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:13.696 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.697 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.697 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.697 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:13.697 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.697 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:13.697 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:13.697 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:13.698 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f68ebfff640 10 monclient: _finish_auth 0 2024-03-27T21:42:13.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f68ebfff640 10 monclient: _check_auth_tickets 2024-03-27T21:42:13.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f68ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:13.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f68ebfff640 1 -- 172.21.15.60:0/213448119 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f4043400 con 0x7f6904071a50 2024-03-27T21:42:13.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6909e09640 5 monclient: authenticate success, global_id 5180 2024-03-27T21:42:13.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6909e09640 10 monclient: _renew_subs 2024-03-27T21:42:13.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6909e09640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:13.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6909e09640 1 -- 172.21.15.60:0/213448119 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69041374d0 con 0x7f6904071a50 2024-03-27T21:42:13.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f68ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:13.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f68ebfff640 1 -- 172.21.15.60:0/213448119 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f40437b0 con 0x7f6904071a50 2024-03-27T21:42:13.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f68ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:13.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.687+0000 7f68ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:13.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.687+0000 7f68ebfff640 10 monclient: dump: 2024-03-27T21:42:13.699 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:13.699 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.700 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.700 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.700 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:13.700 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.700 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:13.700 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:13.700 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:13.700 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.687+0000 7f6909e09640 10 client.5180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:13.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.687+0000 7f6909e09640 10 monclient: _renew_subs 2024-03-27T21:42:13.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.686+0000 7f6901ffb640 4 set_mon_vals no callback set 2024-03-27T21:42:13.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.687+0000 7f6909e09640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:13.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.687+0000 7f6909e09640 1 -- 172.21.15.60:0/213448119 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69041317c0 con 0x7f6904071a50 2024-03-27T21:42:13.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.687+0000 7f6909e09640 1 librados: init done 2024-03-27T21:42:13.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.687+0000 7f6909e09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:13.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.687+0000 7f6909e09640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:13.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.687+0000 7f6909e09640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:13.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.687+0000 7f6909e09640 1 -- 172.21.15.60:0/213448119 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69040b1dc0 con 0x7f6904071a50 2024-03-27T21:42:13.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.688+0000 7f68ebfff640 1 -- 172.21.15.60:0/213448119 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f68f4043910 con 0x7f6904071a50 2024-03-27T21:42:13.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.688+0000 7f68ebfff640 1 --2- 172.21.15.60:0/213448119 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f68f006c590 0x7f68f006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.688+0000 7f68ebfff640 1 -- 172.21.15.60:0/213448119 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f68f403e070 con 0x7f6904071a50 2024-03-27T21:42:13.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.688+0000 7f68ebfff640 10 client.5180.objecter ms_dispatch 0x7f6904000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:13.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.688+0000 7f68ebfff640 3 client.5180.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:13.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.688+0000 7f6902ffd640 1 --2- 172.21.15.60:0/213448119 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f68f006c590 0x7f68f006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.688+0000 7f68ebfff640 3 client.5180.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:13.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.688+0000 7f6902ffd640 10 monclient: get_auth_request con 0x7f68f006c590 auth_method 0 2024-03-27T21:42:13.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.688+0000 7f68ebfff640 20 client.5180.objecter dump_active .. 0 homeless 2024-03-27T21:42:13.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.689+0000 7f68ebfff640 10 client.5180.objecter ms_handle_connect 0x7f68f006c590 2024-03-27T21:42:13.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.689+0000 7f6902ffd640 1 --2- 172.21.15.60:0/213448119 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f68f006c590 0x7f68f006ea50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f69040aafb0 tx=0x7f68f8007e30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:13.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.693+0000 7f68ebfff640 1 -- 172.21.15.60:0/213448119 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f68f40778f0 con 0x7f6904071a50 2024-03-27T21:42:13.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.693+0000 7f68ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:13.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.693+0000 7f68ebfff640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:13.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.870+0000 7f6909e09640 1 -- 172.21.15.60:0/213448119 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f690413f0c0 con 0x7f68f006c590 2024-03-27T21:42:13.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.870+0000 7f68ebfff640 1 -- 172.21.15.60:0/213448119 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f68f8003940 con 0x7f68f006c590 2024-03-27T21:42:13.857 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 5 to deep-scrub 2024-03-27T21:42:13.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.872+0000 7f6909e09640 10 librados: watch_flush enter 2024-03-27T21:42:13.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.872+0000 7f6909e09640 10 librados: watch_flush exit 2024-03-27T21:42:13.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.872+0000 7f6909e09640 20 client.5180.objecter shutdown clearing up homeless session... 2024-03-27T21:42:13.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.872+0000 7f6909e09640 10 client.5180.objecter successfully canceled tick 2024-03-27T21:42:13.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.872+0000 7f6909e09640 1 -- 172.21.15.60:0/213448119 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f68f006c590 msgr2=0x7f68f006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.872+0000 7f6909e09640 1 --2- 172.21.15.60:0/213448119 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f68f006c590 0x7f68f006ea50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f69040aafb0 tx=0x7f68f8007e30 comp rx=0 tx=0).stop 2024-03-27T21:42:13.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.872+0000 7f6909e09640 10 monclient: shutdown 2024-03-27T21:42:13.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.872+0000 7f6909e09640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:13.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.872+0000 7f6909e09640 1 -- 172.21.15.60:0/213448119 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6904071a50 msgr2=0x7f69040aaa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.872+0000 7f6909e09640 1 --2- 172.21.15.60:0/213448119 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6904071a50 0x7f69040aaa70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f68f4009870 tx=0x7f68f4004730 comp rx=0 tx=0).stop 2024-03-27T21:42:13.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.872+0000 7f6909e09640 1 -- 172.21.15.60:0/213448119 shutdown_connections 2024-03-27T21:42:13.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.872+0000 7f6909e09640 1 --2- 172.21.15.60:0/213448119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f690413ec50 0x7f69041310a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.872+0000 7f6909e09640 1 --2- 172.21.15.60:0/213448119 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f68f006c590 0x7f68f006ea50 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.872+0000 7f6909e09640 1 --2- 172.21.15.60:0/213448119 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f690413e810 0x7f69041388d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.872+0000 7f6909e09640 1 --2- 172.21.15.60:0/213448119 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f6904071a50 0x7f69040aaa70 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.872+0000 7f6909e09640 1 -- 172.21.15.60:0/213448119 >> 172.21.15.60:0/213448119 conn(0x7f69040b7300 msgr2=0x7f690413b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:13.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.872+0000 7f6909e09640 1 -- 172.21.15.60:0/213448119 shutdown_connections 2024-03-27T21:42:13.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.872+0000 7f6909e09640 1 -- 172.21.15.60:0/213448119 wait complete. 2024-03-27T21:42:13.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.872+0000 7f6909e09640 1 librados: shutdown 2024-03-27T21:42:13.869 INFO:tasks.ceph:Scrubbing osd.6 2024-03-27T21:42:13.869 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-27T21:42:13.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.987+0000 7f33be869640 10 monclient: get_monmap_and_config 2024-03-27T21:42:13.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.987+0000 7f33be869640 10 monclient: build_initial_monmap 2024-03-27T21:42:13.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.987+0000 7f33be869640 10 monclient: monmap: 2024-03-27T21:42:13.974 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:13.974 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.974 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:13.988669+0000 2024-03-27T21:42:13.974 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:13.988669+0000 2024-03-27T21:42:13.974 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:13.974 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:13.975 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.988+0000 7f33be869640 1 Processor -- start 2024-03-27T21:42:13.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.988+0000 7f33be869640 1 -- start start 2024-03-27T21:42:13.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.988+0000 7f33be869640 10 monclient: init 2024-03-27T21:42:13.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.988+0000 7f33be869640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:13.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.988+0000 7f33be869640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-03-27T21:42:13.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33be869640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f33b80b3980 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:13.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33be869640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f33b80b3d60 0x7f33b81449c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33be869640 10 monclient(hunting): picked mon.noname-b con 0x7f33b80b3d60 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:13.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33be869640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f33b80a86e0 0x7f33b80b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33be869640 10 monclient(hunting): picked mon.noname-c con 0x7f33b80a86e0 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:13.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33be869640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b8144f00 con 0x7f33b80b3980 2024-03-27T21:42:13.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33be869640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33be869640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b80aa1d0 con 0x7f33b80a86e0 2024-03-27T21:42:13.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33be869640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33be869640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b80aa4b0 con 0x7f33b80b3d60 2024-03-27T21:42:13.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33be869640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:13.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33be869640 10 monclient(hunting): authenticate will time out at 975.210019s 2024-03-27T21:42:13.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33b77fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f33b80b3d60 0x7f33b81449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33b77fe640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f33b80b3d60 0x7f33b81449c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:51520/0 (socket says 172.21.15.60:51520) 2024-03-27T21:42:13.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33b77fe640 1 -- 172.21.15.60:0/741749487 learned_addr learned my addr 172.21.15.60:0/741749487 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:13.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33b77fe640 10 monclient(hunting): get_auth_request con 0x7f33b80b3d60 auth_method 0 2024-03-27T21:42:13.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:13.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:13.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33bce06640 1 --2- 172.21.15.60:0/741749487 >> v2:172.21.15.191:3300/0 conn(0x7f33b80a86e0 0x7f33b80b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:13.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:13.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33bce06640 10 monclient(hunting): get_auth_request con 0x7f33b80a86e0 auth_method 0 2024-03-27T21:42:13.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33bce06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:13.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33bce06640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:13.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33bce06640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:13.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33b77fe640 10 monclient(hunting): handle_auth_done global_id 5183 payload 274 2024-03-27T21:42:13.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33b77fe640 1 -- 172.21.15.60:0/741749487 >> v2:172.21.15.191:3300/0 conn(0x7f33b80a86e0 msgr2=0x7f33b80b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:13.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33b77fe640 1 --2- 172.21.15.60:0/741749487 >> v2:172.21.15.191:3300/0 conn(0x7f33b80a86e0 0x7f33b80b00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33b77fe640 1 -- 172.21.15.60:0/741749487 >> v1:172.21.15.191:6789/0 conn(0x7f33b80b3980 legacy=0x7f33b80b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33b77fe640 10 monclient: _finish_hunting 0 2024-03-27T21:42:13.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33b77fe640 1 monclient: found mon.noname-b 2024-03-27T21:42:13.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:13.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33b77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:42:13.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33b77fe640 1 -- 172.21.15.60:0/741749487 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b80b13b0 con 0x7f33b80b3d60 2024-03-27T21:42:13.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.989+0000 7f33bce06640 1 --2- 172.21.15.60:0/741749487 >> v2:172.21.15.191:3300/0 conn(0x7f33b80a86e0 0x7f33b80b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:42:13.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33b77fe640 1 --2- 172.21.15.60:0/741749487 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f33b80b3d60 0x7f33b81449c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f33a8009870 tx=0x7f33a802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c0a434bdb69c3a server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:13.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33b6ffd640 1 -- 172.21.15.60:0/741749487 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a8030840 con 0x7f33b80b3d60 2024-03-27T21:42:13.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:13.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33b6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:13.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33b6ffd640 10 monclient: dump: 2024-03-27T21:42:13.989 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:13.990 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.990 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.990 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.990 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:13.990 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.990 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:13.990 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:13.990 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:13.991 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33b6ffd640 10 monclient: _finish_auth 0 2024-03-27T21:42:13.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33b6ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:13.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:13.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33b6ffd640 1 -- 172.21.15.60:0/741749487 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a80309a0 con 0x7f33b80b3d60 2024-03-27T21:42:13.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33be869640 5 monclient: authenticate success, global_id 5183 2024-03-27T21:42:13.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33be869640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:13.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:13.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33b6ffd640 1 -- 172.21.15.60:0/741749487 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a8030d30 con 0x7f33b80b3d60 2024-03-27T21:42:13.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33be869640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:13.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:13.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33be869640 4 set_mon_vals no callback set 2024-03-27T21:42:13.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:13.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33b6ffd640 10 monclient: dump: 2024-03-27T21:42:13.992 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:13.992 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:13.992 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.992 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:13.993 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:13.993 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:13.993 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:13.993 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:13.993 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:13.994 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33be869640 10 monclient: shutdown 2024-03-27T21:42:13.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33be869640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:13.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33be869640 1 -- 172.21.15.60:0/741749487 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f33b80b3d60 msgr2=0x7f33b81449c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:13.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.990+0000 7f33be869640 1 --2- 172.21.15.60:0/741749487 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f33b80b3d60 0x7f33b81449c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f33a8009870 tx=0x7f33a802b980 comp rx=0 tx=0).stop 2024-03-27T21:42:13.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.991+0000 7f33be869640 1 -- 172.21.15.60:0/741749487 shutdown_connections 2024-03-27T21:42:13.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.991+0000 7f33be869640 1 --2- 172.21.15.60:0/741749487 >> v2:172.21.15.191:3300/0 conn(0x7f33b80a86e0 0x7f33b80b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.991+0000 7f33be869640 1 --2- 172.21.15.60:0/741749487 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f33b80b3d60 0x7f33b81449c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:13.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.991+0000 7f33be869640 1 -- 172.21.15.60:0/741749487 >> 172.21.15.60:0/741749487 conn(0x7f33b8071410 msgr2=0x7f33b8071820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:13.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.991+0000 7f33be869640 1 -- 172.21.15.60:0/741749487 shutdown_connections 2024-03-27T21:42:13.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.991+0000 7f33be869640 1 -- 172.21.15.60:0/741749487 wait complete. 2024-03-27T21:42:13.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.991+0000 7f33be869640 10 monclient: build_initial_monmap 2024-03-27T21:42:13.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.991+0000 7f33be869640 10 monclient: monmap: 2024-03-27T21:42:13.995 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:13.995 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:13.995 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:13.996 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:13.996 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:13.996 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:13.996 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:13.996 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:13.996 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:13.996 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:13.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.991+0000 7f33be869640 1 librados: starting msgr at 2024-03-27T21:42:13.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.991+0000 7f33be869640 1 librados: starting objecter 2024-03-27T21:42:13.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33be869640 1 Processor -- start 2024-03-27T21:42:13.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33be869640 1 -- start start 2024-03-27T21:42:13.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33be869640 1 librados: setting wanted keys 2024-03-27T21:42:13.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33be869640 1 librados: calling monclient init 2024-03-27T21:42:13.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33be869640 10 monclient: init 2024-03-27T21:42:13.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33be869640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:13.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33be869640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:42:13.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33be869640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f33b80a86e0 0x7f33b8140f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33be869640 10 monclient(hunting): picked mon.noname-c con 0x7f33b80a86e0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:13.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33be869640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33b80b3980 0x7f33b81d9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33be869640 10 monclient(hunting): picked mon.noname-b con 0x7f33b80b3980 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:13.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33be869640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33b8136fb0 0x7f33b8137460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:13.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33be869640 10 monclient(hunting): picked mon.noname-a con 0x7f33b8136fb0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:13.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33be869640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33be869640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b81da570 con 0x7f33b8136fb0 2024-03-27T21:42:13.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33be869640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33be869640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b8141c10 con 0x7f33b80b3980 2024-03-27T21:42:13.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33be869640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:13.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33be869640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b81379a0 con 0x7f33b80a86e0 2024-03-27T21:42:13.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33be869640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:13.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33be869640 10 monclient(hunting): authenticate will time out at 975.213730s 2024-03-27T21:42:13.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33b7fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f33b80a86e0 0x7f33b8140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33bce06640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33b8136fb0 0x7f33b8137460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:13.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33b7fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f33b80a86e0 0x7f33b8140f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:51524/0 (socket says 172.21.15.60:51524) 2024-03-27T21:42:13.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33b7fff640 1 -- 172.21.15.60:0/1671935814 learned_addr learned my addr 172.21.15.60:0/1671935814 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:14.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33b7fff640 10 monclient(hunting): get_auth_request con 0x7f33b80a86e0 auth_method 0 2024-03-27T21:42:14.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:14.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:14.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:14.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33bce06640 10 monclient(hunting): get_auth_request con 0x7f33b8136fb0 auth_method 0 2024-03-27T21:42:14.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33bce06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:14.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33bce06640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:14.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.992+0000 7f33bce06640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:14.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:14.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:14.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:14.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33bce06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:14.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33bce06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:14.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33bce06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:14.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b77fe640 1 --2- 172.21.15.60:0/1671935814 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33b80b3980 0x7f33b81d9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b7fff640 10 monclient(hunting): handle_auth_done global_id 5186 payload 824 2024-03-27T21:42:14.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b7fff640 1 -- 172.21.15.60:0/1671935814 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33b80b3980 msgr2=0x7f33b81d9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b7fff640 1 --2- 172.21.15.60:0/1671935814 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33b80b3980 0x7f33b81d9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b7fff640 1 -- 172.21.15.60:0/1671935814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33b8136fb0 msgr2=0x7f33b8137460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b7fff640 1 --2- 172.21.15.60:0/1671935814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33b8136fb0 0x7f33b8137460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b7fff640 10 monclient: _finish_hunting 0 2024-03-27T21:42:14.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b7fff640 1 monclient: found mon.noname-c 2024-03-27T21:42:14.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:14.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:14.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b7fff640 1 -- 172.21.15.60:0/1671935814 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a80094d0 con 0x7f33b80a86e0 2024-03-27T21:42:14.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b77fe640 10 monclient: get_auth_request con 0x7f33b80b3980 auth_method 0 2024-03-27T21:42:14.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b77fe640 1 --2- 172.21.15.60:0/1671935814 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33b80b3980 0x7f33b81d9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:42:14.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33bce06640 1 --2- 172.21.15.60:0/1671935814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33b8136fb0 0x7f33b8137460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:14.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b7fff640 1 --2- 172.21.15.60:0/1671935814 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f33b80a86e0 0x7f33b8140f60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f33ac00c450 tx=0x7f33ac00c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:14.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b4ff9640 10 client.?.objecter ms_handle_connect 0x7f33b80a86e0 2024-03-27T21:42:14.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:14.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b4ff9640 1 -- 172.21.15.60:0/1671935814 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac0052d0 con 0x7f33b80a86e0 2024-03-27T21:42:14.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:14.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:14.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b4ff9640 10 monclient: dump: 2024-03-27T21:42:14.004 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:14.004 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.004 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.004 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.004 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:14.004 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.004 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:14.004 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:14.004 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:14.005 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b4ff9640 10 monclient: _finish_auth 0 2024-03-27T21:42:14.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b4ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:42:14.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:14.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b4ff9640 1 -- 172.21.15.60:0/1671935814 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ac005430 con 0x7f33b80a86e0 2024-03-27T21:42:14.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33be869640 5 monclient: authenticate success, global_id 5186 2024-03-27T21:42:14.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:14.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b4ff9640 1 -- 172.21.15.60:0/1671935814 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac012870 con 0x7f33b80a86e0 2024-03-27T21:42:14.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33be869640 10 monclient: _renew_subs 2024-03-27T21:42:14.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33be869640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:14.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33be869640 1 -- 172.21.15.60:0/1671935814 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b81da160 con 0x7f33b80a86e0 2024-03-27T21:42:14.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b67fc640 4 set_mon_vals no callback set 2024-03-27T21:42:14.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:14.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:14.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33b4ff9640 10 monclient: dump: 2024-03-27T21:42:14.006 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:14.006 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.007 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.007 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.007 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:14.007 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.007 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:14.007 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:14.007 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:14.007 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33be869640 10 client.5186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:14.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33be869640 10 monclient: _renew_subs 2024-03-27T21:42:14.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33be869640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:14.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33be869640 1 -- 172.21.15.60:0/1671935814 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b81d6370 con 0x7f33b80a86e0 2024-03-27T21:42:14.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.993+0000 7f33be869640 1 librados: init done 2024-03-27T21:42:14.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.994+0000 7f33be869640 10 client.5186.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:14.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.994+0000 7f33be869640 15 client.5186.objecter _session_command_op_assign -1 1 2024-03-27T21:42:14.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.994+0000 7f33be869640 20 client.5186.objecter _get_session osd=-1 returning homeless 2024-03-27T21:42:14.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.994+0000 7f33be869640 10 client.5186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:14.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.994+0000 7f33be869640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:14.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.994+0000 7f33be869640 1 -- 172.21.15.60:0/1671935814 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f33b813c790 con 0x7f33b80a86e0 2024-03-27T21:42:14.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.994+0000 7f33b4ff9640 1 -- 172.21.15.60:0/1671935814 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f33ac0129d0 con 0x7f33b80a86e0 2024-03-27T21:42:14.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b4ff9640 1 --2- 172.21.15.60:0/1671935814 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f33a406c590 0x7f33a406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b4ff9640 1 -- 172.21.15.60:0/1671935814 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f33ac088c30 con 0x7f33b80a86e0 2024-03-27T21:42:14.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b4ff9640 10 client.5186.objecter ms_dispatch 0x7f33b8000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:14.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b77fe640 1 --2- 172.21.15.60:0/1671935814 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f33a406c590 0x7f33a406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b4ff9640 3 client.5186.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:14.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b4ff9640 3 client.5186.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:14.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b77fe640 10 monclient: get_auth_request con 0x7f33a406c590 auth_method 0 2024-03-27T21:42:14.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b4ff9640 10 client.5186.objecter checking command 1 2024-03-27T21:42:14.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b4ff9640 1 --2- 172.21.15.60:0/1671935814 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f33a40729d0 0x7f33a4074df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b4ff9640 20 client.5186.objecter _get_session s=0x7f33a4072300 osd=6 3 2024-03-27T21:42:14.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b4ff9640 20 client.5186.objecter put_session s=0x7f33a4072300 osd=6 3 2024-03-27T21:42:14.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b4ff9640 15 client.5186.objecter _session_command_op_remove -1 1 2024-03-27T21:42:14.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b4ff9640 20 client.5186.objecter _get_session s=0x7f33a4072300 osd=6 3 2024-03-27T21:42:14.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b4ff9640 20 client.5186.objecter get_session s=0x7f33a4072300 osd=6 3 2024-03-27T21:42:14.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b4ff9640 15 client.5186.objecter _session_command_op_assign 6 1 2024-03-27T21:42:14.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b4ff9640 20 client.5186.objecter put_session s=0x7f33a4072300 osd=6 4 2024-03-27T21:42:14.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b4ff9640 10 client.5186.objecter _send_command 1 2024-03-27T21:42:14.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b4ff9640 1 -- 172.21.15.60:0/1671935814 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33a40754c0 con 0x7f33a40729d0 2024-03-27T21:42:14.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b4ff9640 20 client.5186.objecter dump_active .. 0 homeless 2024-03-27T21:42:14.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b4ff9640 1 -- 172.21.15.60:0/1671935814 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33ac08b270 con 0x7f33b80a86e0 2024-03-27T21:42:14.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b4ff9640 10 monclient: handle_get_version_reply finishing 1 version 92 2024-03-27T21:42:14.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b4ff9640 10 client.5186.objecter ms_handle_connect 0x7f33a406c590 2024-03-27T21:42:14.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33b77fe640 1 --2- 172.21.15.60:0/1671935814 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f33a406c590 0x7f33a406ea50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f33a8036000 tx=0x7f33a803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:14.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33bce06640 1 --2- 172.21.15.60:0/1671935814 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f33a40729d0 0x7f33a4074df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.995+0000 7f33bce06640 10 monclient: get_auth_request con 0x7f33a40729d0 auth_method 0 2024-03-27T21:42:14.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.996+0000 7f33bce06640 1 --2- 172.21.15.60:0/1671935814 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f33a40729d0 0x7f33a4074df0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:14.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.996+0000 7f33b4ff9640 10 client.5186.objecter ms_handle_connect 0x7f33a40729d0 2024-03-27T21:42:14.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.996+0000 7f33b4ff9640 1 -- 172.21.15.60:0/1671935814 <== osd.6 v2:172.21.15.191:6824/34361 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f33a40754c0 con 0x7f33a40729d0 2024-03-27T21:42:14.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.996+0000 7f33b4ff9640 10 client.5186.objecter ms_dispatch 0x7f33b8000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:42:14.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.996+0000 7f33b4ff9640 10 client.5186.objecter _finish_command 1 = system:0 2024-03-27T21:42:14.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.996+0000 7f33b4ff9640 20 client.5186.objecter put_session s=0x7f33a4072300 osd=6 4 2024-03-27T21:42:14.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:13.996+0000 7f33b4ff9640 15 client.5186.objecter _session_command_op_remove 6 1 2024-03-27T21:42:14.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.009+0000 7f33be869640 10 client.5186.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-27T21:42:14.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.009+0000 7f33be869640 15 client.5186.objecter _session_command_op_assign -1 2 2024-03-27T21:42:14.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.009+0000 7f33be869640 20 client.5186.objecter _get_session s=0x7f33a4072300 osd=6 3 2024-03-27T21:42:14.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.009+0000 7f33be869640 20 client.5186.objecter put_session s=0x7f33a4072300 osd=6 3 2024-03-27T21:42:14.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.009+0000 7f33be869640 20 client.5186.objecter _get_session s=0x7f33a4072300 osd=6 3 2024-03-27T21:42:14.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.009+0000 7f33be869640 15 client.5186.objecter _session_command_op_remove -1 2 2024-03-27T21:42:14.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.009+0000 7f33be869640 20 client.5186.objecter get_session s=0x7f33a4072300 osd=6 3 2024-03-27T21:42:14.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.009+0000 7f33be869640 15 client.5186.objecter _session_command_op_assign 6 2 2024-03-27T21:42:14.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.009+0000 7f33be869640 20 client.5186.objecter put_session s=0x7f33a4072300 osd=6 4 2024-03-27T21:42:14.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.009+0000 7f33be869640 10 client.5186.objecter _send_command 2 2024-03-27T21:42:14.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.009+0000 7f33be869640 1 -- 172.21.15.60:0/1671935814 --> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f33b814bd50 con 0x7f33a40729d0 2024-03-27T21:42:14.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.009+0000 7f33b4ff9640 1 -- 172.21.15.60:0/1671935814 <== osd.6 v2:172.21.15.191:6824/34361 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f33b000f040 con 0x7f33a40729d0 2024-03-27T21:42:14.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.009+0000 7f33b4ff9640 10 client.5186.objecter ms_dispatch 0x7f33b8000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:42:14.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.009+0000 7f33b4ff9640 10 client.5186.objecter _finish_command 2 = system:0 2024-03-27T21:42:14.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.009+0000 7f33b4ff9640 20 client.5186.objecter put_session s=0x7f33a4072300 osd=6 4 2024-03-27T21:42:14.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.009+0000 7f33b4ff9640 15 client.5186.objecter _session_command_op_remove 6 2 2024-03-27T21:42:14.016 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:42:14.016 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-27T21:42:14.016 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:42:14.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.010+0000 7f33be869640 10 librados: watch_flush enter 2024-03-27T21:42:14.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.010+0000 7f33be869640 10 librados: watch_flush exit 2024-03-27T21:42:14.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.010+0000 7f33be869640 10 client.5186.objecter close_session for osd.6 2024-03-27T21:42:14.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.010+0000 7f33be869640 1 -- 172.21.15.60:0/1671935814 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f33a40729d0 msgr2=0x7f33a4074df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.010+0000 7f33be869640 1 --2- 172.21.15.60:0/1671935814 >> [v2:172.21.15.191:6824/34361,v1:172.21.15.191:6825/34361] conn(0x7f33a40729d0 0x7f33a4074df0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.010+0000 7f33be869640 20 client.5186.objecter put_session s=0x7f33a4072300 osd=6 1 2024-03-27T21:42:14.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.010+0000 7f33be869640 20 client.5186.objecter shutdown clearing up homeless session... 2024-03-27T21:42:14.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.010+0000 7f33be869640 10 client.5186.objecter successfully canceled tick 2024-03-27T21:42:14.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.010+0000 7f33be869640 1 -- 172.21.15.60:0/1671935814 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f33a406c590 msgr2=0x7f33a406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.010+0000 7f33be869640 1 --2- 172.21.15.60:0/1671935814 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f33a406c590 0x7f33a406ea50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f33a8036000 tx=0x7f33a803b040 comp rx=0 tx=0).stop 2024-03-27T21:42:14.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.010+0000 7f33be869640 10 monclient: shutdown 2024-03-27T21:42:14.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.010+0000 7f33be869640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:14.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.010+0000 7f33be869640 1 -- 172.21.15.60:0/1671935814 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f33b80a86e0 msgr2=0x7f33b8140f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.010+0000 7f33be869640 1 --2- 172.21.15.60:0/1671935814 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f33b80a86e0 0x7f33b8140f60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f33ac00c450 tx=0x7f33ac00c910 comp rx=0 tx=0).stop 2024-03-27T21:42:14.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.010+0000 7f33bce06640 1 -- 172.21.15.60:0/1671935814 reap_dead start 2024-03-27T21:42:14.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.010+0000 7f33be869640 1 -- 172.21.15.60:0/1671935814 shutdown_connections 2024-03-27T21:42:14.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.010+0000 7f33be869640 1 -- 172.21.15.60:0/1671935814 >> 172.21.15.60:0/1671935814 conn(0x7f33b8071410 msgr2=0x7f33b8144600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:14.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.010+0000 7f33be869640 1 -- 172.21.15.60:0/1671935814 shutdown_connections 2024-03-27T21:42:14.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.010+0000 7f33be869640 1 -- 172.21.15.60:0/1671935814 wait complete. 2024-03-27T21:42:14.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.010+0000 7f33be869640 1 librados: shutdown 2024-03-27T21:42:14.020 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-27T21:42:14.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.135+0000 7f821a2af640 10 monclient: get_monmap_and_config 2024-03-27T21:42:14.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.135+0000 7f821a2af640 10 monclient: build_initial_monmap 2024-03-27T21:42:14.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.135+0000 7f821a2af640 10 monclient: monmap: 2024-03-27T21:42:14.122 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:14.122 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.122 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:14.136495+0000 2024-03-27T21:42:14.122 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:14.136495+0000 2024-03-27T21:42:14.123 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:14.123 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.123 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:14.123 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:14.123 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:14.123 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:14.123 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:14.124 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.136+0000 7f821a2af640 1 Processor -- start 2024-03-27T21:42:14.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.136+0000 7f821a2af640 1 -- start start 2024-03-27T21:42:14.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.136+0000 7f821a2af640 10 monclient: init 2024-03-27T21:42:14.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.136+0000 7f821a2af640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:14.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.136+0000 7f821a2af640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-03-27T21:42:14.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.136+0000 7f821a2af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8214071a50 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:14.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.136+0000 7f821a2af640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f821413ef20 0x7f82140a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.136+0000 7f821a2af640 10 monclient(hunting): picked mon.noname-c con 0x7f821413ef20 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:14.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.136+0000 7f821a2af640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8214141630 0x7f8214141ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.136+0000 7f821a2af640 10 monclient(hunting): picked mon.noname-b con 0x7f8214141630 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:14.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.136+0000 7f821a2af640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82140a8e90 con 0x7f8214071a50 2024-03-27T21:42:14.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.136+0000 7f821a2af640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:14.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.136+0000 7f821a2af640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8214071e30 con 0x7f821413ef20 2024-03-27T21:42:14.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.136+0000 7f821a2af640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:14.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.136+0000 7f821a2af640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82140aa040 con 0x7f8214141630 2024-03-27T21:42:14.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.136+0000 7f821a2af640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:14.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.136+0000 7f821a2af640 10 monclient(hunting): authenticate will time out at 975.357750s 2024-03-27T21:42:14.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.136+0000 7f821884c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8214141630 0x7f8214141ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.136+0000 7f8213fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8214071a50 0x7f821413e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60334/0 (socket says 172.21.15.60:60334) 2024-03-27T21:42:14.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.136+0000 7f8213fff640 1 -- 172.21.15.60:0/1151986886 learned_addr learned my addr 172.21.15.60:0/1151986886 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:14.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f821884c640 10 monclient(hunting): get_auth_request con 0x7f8214141630 auth_method 0 2024-03-27T21:42:14.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f821884c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:14.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f821884c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:14.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f821884c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:14.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f821884c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:14.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f821884c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:14.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f82137fe640 1 --2- 172.21.15.60:0/1151986886 >> v2:172.21.15.191:3300/0 conn(0x7f821413ef20 0x7f82140a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f821884c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:14.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f82137fe640 10 monclient(hunting): get_auth_request con 0x7f821413ef20 auth_method 0 2024-03-27T21:42:14.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f82137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:14.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f82137fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:14.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f82137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:14.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f82137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:14.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f82137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:14.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f82137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:14.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f821884c640 10 monclient(hunting): handle_auth_done global_id 5189 payload 274 2024-03-27T21:42:14.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f821884c640 1 -- 172.21.15.60:0/1151986886 >> v2:172.21.15.191:3300/0 conn(0x7f821413ef20 msgr2=0x7f82140a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:14.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f821884c640 1 --2- 172.21.15.60:0/1151986886 >> v2:172.21.15.191:3300/0 conn(0x7f821413ef20 0x7f82140a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f821884c640 1 -- 172.21.15.60:0/1151986886 >> v1:172.21.15.60:6789/0 conn(0x7f8214071a50 legacy=0x7f821413e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f821884c640 10 monclient: _finish_hunting 0 2024-03-27T21:42:14.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f821884c640 1 monclient: found mon.noname-b 2024-03-27T21:42:14.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f821884c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:14.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f821884c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:42:14.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f8212ffd640 1 -- 172.21.15.60:0/1151986886 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 816643978 0 0) 0x7f82140a8e90 con 0x7f8214071a50 2024-03-27T21:42:14.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f821884c640 1 -- 172.21.15.60:0/1151986886 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82140aa2b0 con 0x7f8214141630 2024-03-27T21:42:14.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.137+0000 7f8212ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-27T21:42:14.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f821884c640 1 --2- 172.21.15.60:0/1151986886 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8214141630 0x7f8214141ac0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f820c009d90 tx=0x7f820c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85a58c1058890141 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:14.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f8212ffd640 1 -- 172.21.15.60:0/1151986886 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820c02cc50 con 0x7f8214141630 2024-03-27T21:42:14.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f8212ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:14.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f8212ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:14.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f8212ffd640 10 monclient: dump: 2024-03-27T21:42:14.143 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:14.143 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.143 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.143 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.143 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:14.143 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.143 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:14.144 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:14.144 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:14.144 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f8212ffd640 10 monclient: _finish_auth 0 2024-03-27T21:42:14.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f8212ffd640 10 monclient: _check_auth_tickets 2024-03-27T21:42:14.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f8212ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:14.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f8212ffd640 1 -- 172.21.15.60:0/1151986886 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820c02cdb0 con 0x7f8214141630 2024-03-27T21:42:14.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f821a2af640 5 monclient: authenticate success, global_id 5189 2024-03-27T21:42:14.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f8212ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:14.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f8212ffd640 1 -- 172.21.15.60:0/1151986886 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820c034660 con 0x7f8214141630 2024-03-27T21:42:14.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f821a2af640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:14.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f821a2af640 4 set_mon_vals no callback set 2024-03-27T21:42:14.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f8212ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:14.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f8212ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:14.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f8212ffd640 10 monclient: dump: 2024-03-27T21:42:14.145 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:14.145 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.145 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.146 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.146 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:14.146 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.146 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:14.146 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:14.146 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:14.146 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f821a2af640 10 monclient: shutdown 2024-03-27T21:42:14.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f821a2af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:14.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f821a2af640 1 -- 172.21.15.60:0/1151986886 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8214141630 msgr2=0x7f8214141ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f821a2af640 1 --2- 172.21.15.60:0/1151986886 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8214141630 0x7f8214141ac0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f820c009d90 tx=0x7f820c02bea0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f821a2af640 1 -- 172.21.15.60:0/1151986886 shutdown_connections 2024-03-27T21:42:14.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f821a2af640 1 --2- 172.21.15.60:0/1151986886 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8214141630 0x7f8214141ac0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f821a2af640 1 --2- 172.21.15.60:0/1151986886 >> v2:172.21.15.191:3300/0 conn(0x7f821413ef20 0x7f82140a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f821a2af640 1 -- 172.21.15.60:0/1151986886 >> 172.21.15.60:0/1151986886 conn(0x7f82140b7300 msgr2=0x7f82140b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:14.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f821a2af640 1 -- 172.21.15.60:0/1151986886 shutdown_connections 2024-03-27T21:42:14.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.138+0000 7f821a2af640 1 -- 172.21.15.60:0/1151986886 wait complete. 2024-03-27T21:42:14.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.139+0000 7f821a2af640 10 monclient: build_initial_monmap 2024-03-27T21:42:14.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.139+0000 7f821a2af640 10 monclient: monmap: 2024-03-27T21:42:14.148 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:14.148 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:14.148 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:14.148 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:14.148 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:14.149 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:14.149 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:14.149 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:14.149 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:14.149 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.139+0000 7f821a2af640 1 librados: starting msgr at 2024-03-27T21:42:14.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.139+0000 7f821a2af640 1 librados: starting objecter 2024-03-27T21:42:14.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.139+0000 7f821a2af640 1 Processor -- start 2024-03-27T21:42:14.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.139+0000 7f821a2af640 1 -- start start 2024-03-27T21:42:14.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.139+0000 7f821a2af640 1 librados: setting wanted keys 2024-03-27T21:42:14.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.139+0000 7f821a2af640 1 librados: calling monclient init 2024-03-27T21:42:14.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.139+0000 7f821a2af640 10 monclient: init 2024-03-27T21:42:14.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821a2af640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:14.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821a2af640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:42:14.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821a2af640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8214071a50 0x7f82141331f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821a2af640 10 monclient(hunting): picked mon.noname-a con 0x7f8214071a50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:14.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821a2af640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f821413ef20 0x7f821413a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821a2af640 10 monclient(hunting): picked mon.noname-b con 0x7f821413ef20 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:14.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821a2af640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8214141630 0x7f821413aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821a2af640 10 monclient(hunting): picked mon.noname-c con 0x7f8214141630 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:14.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821a2af640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:14.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821a2af640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8214138d30 con 0x7f8214071a50 2024-03-27T21:42:14.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821a2af640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:14.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821a2af640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82141d99d0 con 0x7f821413ef20 2024-03-27T21:42:14.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821a2af640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:14.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821a2af640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82141372a0 con 0x7f8214141630 2024-03-27T21:42:14.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821a2af640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:14.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821a2af640 10 monclient(hunting): authenticate will time out at 975.361307s 2024-03-27T21:42:14.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f8213fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8214071a50 0x7f82141331f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821884c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8214141630 0x7f821413aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f8213fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8214071a50 0x7f82141331f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59394/0 (socket says 172.21.15.60:59394) 2024-03-27T21:42:14.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821884c640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8214141630 0x7f821413aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:51548/0 (socket says 172.21.15.60:51548) 2024-03-27T21:42:14.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f8213fff640 1 -- 172.21.15.60:0/4142126397 learned_addr learned my addr 172.21.15.60:0/4142126397 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:14.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f8213fff640 10 monclient(hunting): get_auth_request con 0x7f8214071a50 auth_method 0 2024-03-27T21:42:14.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f8213fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:14.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f8213fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:14.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f8213fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:14.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821884c640 10 monclient(hunting): get_auth_request con 0x7f8214141630 auth_method 0 2024-03-27T21:42:14.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821884c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:14.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821884c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:14.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821884c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:14.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f82137fe640 1 --2- 172.21.15.60:0/4142126397 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f821413ef20 0x7f821413a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f82137fe640 10 monclient(hunting): get_auth_request con 0x7f821413ef20 auth_method 0 2024-03-27T21:42:14.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f82137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:14.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f82137fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:14.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f82137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:14.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821884c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:14.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821884c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:14.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821884c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:14.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f8213fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:14.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f8213fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:14.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f8213fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:14.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f82137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:14.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f82137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:14.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f82137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:14.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821884c640 10 monclient(hunting): handle_auth_done global_id 5192 payload 824 2024-03-27T21:42:14.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821884c640 1 -- 172.21.15.60:0/4142126397 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f821413ef20 msgr2=0x7f821413a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821884c640 1 --2- 172.21.15.60:0/4142126397 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f821413ef20 0x7f821413a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821884c640 1 -- 172.21.15.60:0/4142126397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8214071a50 msgr2=0x7f82141331f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821884c640 1 --2- 172.21.15.60:0/4142126397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8214071a50 0x7f82141331f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821884c640 10 monclient: _finish_hunting 0 2024-03-27T21:42:14.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821884c640 1 monclient: found mon.noname-c 2024-03-27T21:42:14.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821884c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:14.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821884c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:14.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821884c640 1 -- 172.21.15.60:0/4142126397 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820c009a70 con 0x7f8214141630 2024-03-27T21:42:14.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f821884c640 1 --2- 172.21.15.60:0/4142126397 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8214141630 0x7f821413aa30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f820c033000 tx=0x7f820c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:14.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f8210ff9640 10 client.?.objecter ms_handle_connect 0x7f8214141630 2024-03-27T21:42:14.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.140+0000 7f8210ff9640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:14.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f8210ff9640 1 -- 172.21.15.60:0/4142126397 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820c040650 con 0x7f8214141630 2024-03-27T21:42:14.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f8210ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:14.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f8210ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:14.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f8210ff9640 10 monclient: dump: 2024-03-27T21:42:14.157 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:14.157 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.157 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.158 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.158 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:14.158 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.158 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:14.158 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:14.158 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:14.158 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f8210ff9640 10 monclient: _finish_auth 0 2024-03-27T21:42:14.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f8210ff9640 10 monclient: _check_auth_tickets 2024-03-27T21:42:14.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f8210ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:14.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f8210ff9640 1 -- 172.21.15.60:0/4142126397 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820c0407b0 con 0x7f8214141630 2024-03-27T21:42:14.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f8210ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:14.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f8210ff9640 1 -- 172.21.15.60:0/4142126397 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820c040ba0 con 0x7f8214141630 2024-03-27T21:42:14.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f821a2af640 5 monclient: authenticate success, global_id 5192 2024-03-27T21:42:14.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f8210ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:14.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f8210ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:14.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f8210ff9640 10 monclient: dump: 2024-03-27T21:42:14.160 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:14.160 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.160 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.160 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.160 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:14.160 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.160 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:14.160 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:14.161 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:14.161 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f821a2af640 10 monclient: _renew_subs 2024-03-27T21:42:14.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f821a2af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:14.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f821a2af640 1 -- 172.21.15.60:0/4142126397 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8214139900 con 0x7f8214141630 2024-03-27T21:42:14.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f82127fc640 4 set_mon_vals no callback set 2024-03-27T21:42:14.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f821a2af640 10 client.5192.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:14.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f821a2af640 10 monclient: _renew_subs 2024-03-27T21:42:14.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f821a2af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:14.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f821a2af640 1 -- 172.21.15.60:0/4142126397 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8214133910 con 0x7f8214141630 2024-03-27T21:42:14.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.141+0000 7f821a2af640 1 librados: init done 2024-03-27T21:42:14.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.142+0000 7f821a2af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:14.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.142+0000 7f821a2af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:14.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.142+0000 7f821a2af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:14.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.142+0000 7f821a2af640 1 -- 172.21.15.60:0/4142126397 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8214067900 con 0x7f8214141630 2024-03-27T21:42:14.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.142+0000 7f8210ff9640 1 -- 172.21.15.60:0/4142126397 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f820c040d00 con 0x7f8214141630 2024-03-27T21:42:14.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.143+0000 7f8210ff9640 1 --2- 172.21.15.60:0/4142126397 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f820006c590 0x7f820006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.143+0000 7f8213fff640 1 --2- 172.21.15.60:0/4142126397 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f820006c590 0x7f820006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.143+0000 7f8213fff640 10 monclient: get_auth_request con 0x7f820006c590 auth_method 0 2024-03-27T21:42:14.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.143+0000 7f8210ff9640 1 -- 172.21.15.60:0/4142126397 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f820c0abd00 con 0x7f8214141630 2024-03-27T21:42:14.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.143+0000 7f8210ff9640 10 client.5192.objecter ms_dispatch 0x7f8214000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:14.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.143+0000 7f8210ff9640 3 client.5192.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:14.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.143+0000 7f8210ff9640 3 client.5192.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:14.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.143+0000 7f8210ff9640 20 client.5192.objecter dump_active .. 0 homeless 2024-03-27T21:42:14.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.143+0000 7f8210ff9640 10 client.5192.objecter ms_handle_connect 0x7f820006c590 2024-03-27T21:42:14.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.143+0000 7f8213fff640 1 --2- 172.21.15.60:0/4142126397 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f820006c590 0x7f820006ea50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8204000f60 tx=0x7f8204007e10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:14.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.147+0000 7f8210ff9640 1 -- 172.21.15.60:0/4142126397 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f820c0771f0 con 0x7f8214141630 2024-03-27T21:42:14.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.147+0000 7f8210ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:14.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.147+0000 7f8210ff9640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:14.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.323+0000 7f821a2af640 1 -- 172.21.15.60:0/4142126397 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f8214069470 con 0x7f820006c590 2024-03-27T21:42:14.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.324+0000 7f8210ff9640 1 -- 172.21.15.60:0/4142126397 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8214069470 con 0x7f820006c590 2024-03-27T21:42:14.310 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 6 to deep-scrub 2024-03-27T21:42:14.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.325+0000 7f821a2af640 10 librados: watch_flush enter 2024-03-27T21:42:14.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.325+0000 7f821a2af640 10 librados: watch_flush exit 2024-03-27T21:42:14.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.325+0000 7f821a2af640 20 client.5192.objecter shutdown clearing up homeless session... 2024-03-27T21:42:14.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.325+0000 7f821a2af640 10 client.5192.objecter successfully canceled tick 2024-03-27T21:42:14.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.325+0000 7f821a2af640 1 -- 172.21.15.60:0/4142126397 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f820006c590 msgr2=0x7f820006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.325+0000 7f821a2af640 1 --2- 172.21.15.60:0/4142126397 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f820006c590 0x7f820006ea50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8204000f60 tx=0x7f8204007e10 comp rx=0 tx=0).stop 2024-03-27T21:42:14.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.325+0000 7f821a2af640 10 monclient: shutdown 2024-03-27T21:42:14.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.325+0000 7f821a2af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:14.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.325+0000 7f821a2af640 1 -- 172.21.15.60:0/4142126397 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8214141630 msgr2=0x7f821413aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.325+0000 7f821a2af640 1 --2- 172.21.15.60:0/4142126397 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8214141630 0x7f821413aa30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f820c033000 tx=0x7f820c004a40 comp rx=0 tx=0).stop 2024-03-27T21:42:14.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.326+0000 7f821a2af640 1 -- 172.21.15.60:0/4142126397 shutdown_connections 2024-03-27T21:42:14.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.326+0000 7f821a2af640 1 --2- 172.21.15.60:0/4142126397 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f8214141630 0x7f821413aa30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.326+0000 7f821a2af640 1 --2- 172.21.15.60:0/4142126397 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f820006c590 0x7f820006ea50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.326+0000 7f821a2af640 1 --2- 172.21.15.60:0/4142126397 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f821413ef20 0x7f821413a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.326+0000 7f821a2af640 1 --2- 172.21.15.60:0/4142126397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8214071a50 0x7f82141331f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.326+0000 7f821a2af640 1 -- 172.21.15.60:0/4142126397 >> 172.21.15.60:0/4142126397 conn(0x7f82140b7300 msgr2=0x7f821413e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:14.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.326+0000 7f821a2af640 1 -- 172.21.15.60:0/4142126397 shutdown_connections 2024-03-27T21:42:14.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.326+0000 7f821a2af640 1 -- 172.21.15.60:0/4142126397 wait complete. 2024-03-27T21:42:14.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.326+0000 7f821a2af640 1 librados: shutdown 2024-03-27T21:42:14.322 INFO:tasks.ceph:Scrubbing osd.7 2024-03-27T21:42:14.323 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-27T21:42:14.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.441+0000 7f969dce0640 10 monclient: get_monmap_and_config 2024-03-27T21:42:14.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.441+0000 7f969dce0640 10 monclient: build_initial_monmap 2024-03-27T21:42:14.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.441+0000 7f969dce0640 10 monclient: monmap: 2024-03-27T21:42:14.428 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:14.428 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.428 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:14.442241+0000 2024-03-27T21:42:14.428 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:14.442241+0000 2024-03-27T21:42:14.428 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:14.428 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.428 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:14.429 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:14.429 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:14.429 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:14.429 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:14.429 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.441+0000 7f969dce0640 1 Processor -- start 2024-03-27T21:42:14.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.441+0000 7f969dce0640 1 -- start start 2024-03-27T21:42:14.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.441+0000 7f969dce0640 10 monclient: init 2024-03-27T21:42:14.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f969dce0640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:14.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f969dce0640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-03-27T21:42:14.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f969dce0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9698071a40 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:14.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f969dce0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f969813e810 0x7f969813ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f969dce0640 10 monclient(hunting): picked mon.noname-a con 0x7f969813e810 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:14.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f969dce0640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9698144a30 0x7f9698144e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f969dce0640 10 monclient(hunting): picked mon.noname-c con 0x7f9698144a30 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:14.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f969dce0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969813f1a0 con 0x7f9698071a40 2024-03-27T21:42:14.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f969dce0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:14.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f969dce0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9698071e20 con 0x7f969813e810 2024-03-27T21:42:14.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f969dce0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:14.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f969dce0640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96980a9060 con 0x7f9698144a30 2024-03-27T21:42:14.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f969dce0640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:14.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f969dce0640 10 monclient(hunting): authenticate will time out at 975.663504s 2024-03-27T21:42:14.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9696ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f969813e810 0x7f969813ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9696ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f969813e810 0x7f969813ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59410/0 (socket says 172.21.15.60:59410) 2024-03-27T21:42:14.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9696ffd640 1 -- 172.21.15.60:0/3149335104 learned_addr learned my addr 172.21.15.60:0/3149335104 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:14.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9696ffd640 10 monclient(hunting): get_auth_request con 0x7f969813e810 auth_method 0 2024-03-27T21:42:14.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9696ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:14.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9696ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:14.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9696ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:14.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9696ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:14.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9696ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:14.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9696ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:14.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9697fff640 1 --2- 172.21.15.60:0/3149335104 >> v2:172.21.15.191:3300/0 conn(0x7f9698144a30 0x7f9698144e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9696ffd640 10 monclient(hunting): handle_auth_done global_id 5535 payload 274 2024-03-27T21:42:14.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9696ffd640 1 -- 172.21.15.60:0/3149335104 >> v2:172.21.15.191:3300/0 conn(0x7f9698144a30 msgr2=0x7f9698144e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:14.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9696ffd640 1 --2- 172.21.15.60:0/3149335104 >> v2:172.21.15.191:3300/0 conn(0x7f9698144a30 0x7f9698144e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9696ffd640 1 -- 172.21.15.60:0/3149335104 >> v1:172.21.15.191:6789/0 conn(0x7f9698071a40 legacy=0x7f96980a8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:14.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9696ffd640 10 monclient: _finish_hunting 0 2024-03-27T21:42:14.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9696ffd640 1 monclient: found mon.noname-a 2024-03-27T21:42:14.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9696ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:14.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9696ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:14.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9696ffd640 1 -- 172.21.15.60:0/3149335104 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96980aa320 con 0x7f969813e810 2024-03-27T21:42:14.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9697fff640 10 monclient: get_auth_request con 0x7f9698144a30 auth_method 0 2024-03-27T21:42:14.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.442+0000 7f9697fff640 1 --2- 172.21.15.60:0/3149335104 >> v2:172.21.15.191:3300/0 conn(0x7f9698144a30 0x7f9698144e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:42:14.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.443+0000 7f9696ffd640 1 --2- 172.21.15.60:0/3149335104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f969813e810 0x7f969813ec60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9698071250 tx=0x7f968c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e01f7ff4e798d234 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:14.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.443+0000 7f96967fc640 1 -- 172.21.15.60:0/3149335104 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c031750 con 0x7f969813e810 2024-03-27T21:42:14.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.443+0000 7f96967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:14.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.443+0000 7f96967fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:14.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.443+0000 7f96967fc640 10 monclient: dump: 2024-03-27T21:42:14.445 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:14.445 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.445 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.445 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:14.446 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.443+0000 7f96967fc640 10 monclient: _finish_auth 0 2024-03-27T21:42:14.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.443+0000 7f96967fc640 10 monclient: _check_auth_tickets 2024-03-27T21:42:14.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.443+0000 7f96967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:14.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.443+0000 7f96967fc640 1 -- 172.21.15.60:0/3149335104 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968c0318b0 con 0x7f969813e810 2024-03-27T21:42:14.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.443+0000 7f969dce0640 5 monclient: authenticate success, global_id 5535 2024-03-27T21:42:14.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.443+0000 7f969dce0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:14.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.443+0000 7f96967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:14.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.443+0000 7f969dce0640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:14.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.443+0000 7f96967fc640 1 -- 172.21.15.60:0/3149335104 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c031c40 con 0x7f969813e810 2024-03-27T21:42:14.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.443+0000 7f96967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:14.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.443+0000 7f969dce0640 4 set_mon_vals no callback set 2024-03-27T21:42:14.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.443+0000 7f96967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:14.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.443+0000 7f96967fc640 10 monclient: dump: 2024-03-27T21:42:14.448 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:14.448 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.448 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.449 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.449 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:14.449 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.449 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:14.449 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:14.449 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:14.449 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.444+0000 7f969dce0640 10 monclient: shutdown 2024-03-27T21:42:14.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.444+0000 7f969dce0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:14.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.444+0000 7f969dce0640 1 -- 172.21.15.60:0/3149335104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f969813e810 msgr2=0x7f969813ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.444+0000 7f969dce0640 1 --2- 172.21.15.60:0/3149335104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f969813e810 0x7f969813ec60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9698071250 tx=0x7f968c02b7f0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.444+0000 7f969dce0640 1 -- 172.21.15.60:0/3149335104 shutdown_connections 2024-03-27T21:42:14.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.444+0000 7f969dce0640 1 --2- 172.21.15.60:0/3149335104 >> v2:172.21.15.191:3300/0 conn(0x7f9698144a30 0x7f9698144e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.444+0000 7f969dce0640 1 --2- 172.21.15.60:0/3149335104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f969813e810 0x7f969813ec60 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.444+0000 7f969dce0640 1 -- 172.21.15.60:0/3149335104 >> 172.21.15.60:0/3149335104 conn(0x7f96980b7300 msgr2=0x7f96980b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:14.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.444+0000 7f969dce0640 1 -- 172.21.15.60:0/3149335104 shutdown_connections 2024-03-27T21:42:14.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.444+0000 7f969dce0640 1 -- 172.21.15.60:0/3149335104 wait complete. 2024-03-27T21:42:14.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.444+0000 7f969dce0640 10 monclient: build_initial_monmap 2024-03-27T21:42:14.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.444+0000 7f969dce0640 10 monclient: monmap: 2024-03-27T21:42:14.451 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:14.451 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:14.451 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:14.451 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:14.451 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:14.451 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:14.452 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:14.452 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:14.452 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:14.452 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.445+0000 7f969dce0640 1 librados: starting msgr at 2024-03-27T21:42:14.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.445+0000 7f969dce0640 1 librados: starting objecter 2024-03-27T21:42:14.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.445+0000 7f969dce0640 1 Processor -- start 2024-03-27T21:42:14.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.445+0000 7f969dce0640 1 -- start start 2024-03-27T21:42:14.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.445+0000 7f969dce0640 1 librados: setting wanted keys 2024-03-27T21:42:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.445+0000 7f969dce0640 1 librados: calling monclient init 2024-03-27T21:42:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.445+0000 7f969dce0640 10 monclient: init 2024-03-27T21:42:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:42:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9698071a40 0x7f969813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 10 monclient(hunting): picked mon.noname-a con 0x7f9698071a40 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f969813e810 0x7f96981d6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 10 monclient(hunting): picked mon.noname-c con 0x7f969813e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9698144a30 0x7f96981d68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 10 monclient(hunting): picked mon.noname-b con 0x7f9698144a30 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:14.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:14.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969813b400 con 0x7f9698071a40 2024-03-27T21:42:14.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:14.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9698139b80 con 0x7f9698144a30 2024-03-27T21:42:14.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:14.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9698144660 con 0x7f969813e810 2024-03-27T21:42:14.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:14.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 10 monclient(hunting): authenticate will time out at 975.667039s 2024-03-27T21:42:14.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f96977fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9698071a40 0x7f969813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f96977fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9698071a40 0x7f969813f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59412/0 (socket says 172.21.15.60:59412) 2024-03-27T21:42:14.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f96977fe640 1 -- 172.21.15.60:0/1436261167 learned_addr learned my addr 172.21.15.60:0/1436261167 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:14.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f96977fe640 10 monclient(hunting): get_auth_request con 0x7f9698071a40 auth_method 0 2024-03-27T21:42:14.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f96977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:14.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f96977fe640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:14.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f96977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:14.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f9696ffd640 1 --2- 172.21.15.60:0/1436261167 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f969813e810 0x7f96981d6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f9696ffd640 10 monclient(hunting): get_auth_request con 0x7f969813e810 auth_method 0 2024-03-27T21:42:14.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f9696ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:14.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f9696ffd640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:14.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f9696ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:14.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f96977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:14.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f96977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:14.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f96977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:14.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f9696ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:14.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f9696ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:14.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f9696ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:14.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f9697fff640 1 --2- 172.21.15.60:0/1436261167 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9698144a30 0x7f96981d68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f96977fe640 10 monclient(hunting): handle_auth_done global_id 5538 payload 824 2024-03-27T21:42:14.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f96977fe640 1 -- 172.21.15.60:0/1436261167 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f969813e810 msgr2=0x7f96981d6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f96977fe640 1 --2- 172.21.15.60:0/1436261167 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f969813e810 0x7f96981d6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f96977fe640 1 -- 172.21.15.60:0/1436261167 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9698144a30 msgr2=0x7f96981d68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f96977fe640 1 --2- 172.21.15.60:0/1436261167 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9698144a30 0x7f96981d68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f96977fe640 10 monclient: _finish_hunting 0 2024-03-27T21:42:14.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f96977fe640 1 monclient: found mon.noname-a 2024-03-27T21:42:14.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f96977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:14.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f96977fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:14.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f96977fe640 1 -- 172.21.15.60:0/1436261167 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968c0094d0 con 0x7f9698071a40 2024-03-27T21:42:14.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f9697fff640 10 monclient: get_auth_request con 0x7f9698144a30 auth_method 0 2024-03-27T21:42:14.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f9696ffd640 1 --2- 172.21.15.60:0/1436261167 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f969813e810 0x7f96981d6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:14.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f9697fff640 1 --2- 172.21.15.60:0/1436261167 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9698144a30 0x7f96981d68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:42:14.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f96977fe640 1 --2- 172.21.15.60:0/1436261167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9698071a40 0x7f969813f5a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f968800c430 tx=0x7f968800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:14.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f967ffff640 10 client.?.objecter ms_handle_connect 0x7f9698071a40 2024-03-27T21:42:14.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f967ffff640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:14.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f967ffff640 1 -- 172.21.15.60:0/1436261167 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9688011790 con 0x7f9698071a40 2024-03-27T21:42:14.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f967ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:14.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f967ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:14.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f967ffff640 10 monclient: dump: 2024-03-27T21:42:14.461 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:14.461 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.461 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.462 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.462 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:14.462 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.462 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:14.462 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:14.462 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:14.463 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f967ffff640 10 monclient: _finish_auth 0 2024-03-27T21:42:14.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f967ffff640 10 monclient: _check_auth_tickets 2024-03-27T21:42:14.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f967ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:14.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f967ffff640 1 -- 172.21.15.60:0/1436261167 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96880118f0 con 0x7f9698071a40 2024-03-27T21:42:14.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 5 monclient: authenticate success, global_id 5538 2024-03-27T21:42:14.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 10 monclient: _renew_subs 2024-03-27T21:42:14.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:14.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 1 -- 172.21.15.60:0/1436261167 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96981d6df0 con 0x7f9698071a40 2024-03-27T21:42:14.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f967ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:14.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f967ffff640 1 -- 172.21.15.60:0/1436261167 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9688011ca0 con 0x7f9698071a40 2024-03-27T21:42:14.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f967ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:14.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f967ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:14.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f967ffff640 10 monclient: dump: 2024-03-27T21:42:14.465 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:14.465 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.466 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.466 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.466 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:14.466 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.467 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:14.467 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:14.467 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:14.467 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f9695ffb640 4 set_mon_vals no callback set 2024-03-27T21:42:14.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 10 client.5538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:14.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 10 monclient: _renew_subs 2024-03-27T21:42:14.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:14.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 1 -- 172.21.15.60:0/1436261167 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f969813b030 con 0x7f9698071a40 2024-03-27T21:42:14.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.446+0000 7f969dce0640 1 librados: init done 2024-03-27T21:42:14.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.447+0000 7f969dce0640 10 client.5538.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:14.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.447+0000 7f969dce0640 15 client.5538.objecter _session_command_op_assign -1 1 2024-03-27T21:42:14.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.447+0000 7f969dce0640 20 client.5538.objecter _get_session osd=-1 returning homeless 2024-03-27T21:42:14.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.447+0000 7f969dce0640 10 client.5538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:14.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.447+0000 7f969dce0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:14.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.447+0000 7f969dce0640 1 -- 172.21.15.60:0/1436261167 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f969814bd60 con 0x7f9698071a40 2024-03-27T21:42:14.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f967ffff640 1 -- 172.21.15.60:0/1436261167 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f968801c460 con 0x7f9698071a40 2024-03-27T21:42:14.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f967ffff640 1 --2- 172.21.15.60:0/1436261167 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f968406c590 0x7f968406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f9696ffd640 1 --2- 172.21.15.60:0/1436261167 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f968406c590 0x7f968406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f967ffff640 1 -- 172.21.15.60:0/1436261167 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f9688088f90 con 0x7f9698071a40 2024-03-27T21:42:14.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f967ffff640 10 client.5538.objecter ms_dispatch 0x7f9698000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:14.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f967ffff640 3 client.5538.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:14.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f967ffff640 3 client.5538.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:14.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f9696ffd640 10 monclient: get_auth_request con 0x7f968406c590 auth_method 0 2024-03-27T21:42:14.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f967ffff640 10 client.5538.objecter checking command 1 2024-03-27T21:42:14.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f967ffff640 1 --2- 172.21.15.60:0/1436261167 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f96840729d0 0x7f9684074df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f967ffff640 20 client.5538.objecter _get_session s=0x7f9684072300 osd=7 3 2024-03-27T21:42:14.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f967ffff640 20 client.5538.objecter put_session s=0x7f9684072300 osd=7 3 2024-03-27T21:42:14.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f967ffff640 15 client.5538.objecter _session_command_op_remove -1 1 2024-03-27T21:42:14.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f967ffff640 20 client.5538.objecter _get_session s=0x7f9684072300 osd=7 3 2024-03-27T21:42:14.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f967ffff640 20 client.5538.objecter get_session s=0x7f9684072300 osd=7 3 2024-03-27T21:42:14.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f967ffff640 15 client.5538.objecter _session_command_op_assign 7 1 2024-03-27T21:42:14.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f967ffff640 20 client.5538.objecter put_session s=0x7f9684072300 osd=7 4 2024-03-27T21:42:14.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f967ffff640 10 client.5538.objecter _send_command 1 2024-03-27T21:42:14.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f967ffff640 1 -- 172.21.15.60:0/1436261167 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96840754c0 con 0x7f96840729d0 2024-03-27T21:42:14.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f967ffff640 20 client.5538.objecter dump_active .. 0 homeless 2024-03-27T21:42:14.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f967ffff640 1 -- 172.21.15.60:0/1436261167 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f968808b5f0 con 0x7f9698071a40 2024-03-27T21:42:14.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.448+0000 7f967ffff640 10 monclient: handle_get_version_reply finishing 1 version 92 2024-03-27T21:42:14.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.449+0000 7f967ffff640 10 client.5538.objecter ms_handle_connect 0x7f968406c590 2024-03-27T21:42:14.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.449+0000 7f9696ffd640 1 --2- 172.21.15.60:0/1436261167 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f968406c590 0x7f968406ea50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f968c035000 tx=0x7f968c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:14.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.449+0000 7f9697fff640 1 --2- 172.21.15.60:0/1436261167 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f96840729d0 0x7f9684074df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.449+0000 7f9697fff640 10 monclient: get_auth_request con 0x7f96840729d0 auth_method 0 2024-03-27T21:42:14.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.449+0000 7f967ffff640 10 client.5538.objecter ms_handle_connect 0x7f96840729d0 2024-03-27T21:42:14.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.449+0000 7f9697fff640 1 --2- 172.21.15.60:0/1436261167 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f96840729d0 0x7f9684074df0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:14.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.450+0000 7f967ffff640 1 -- 172.21.15.60:0/1436261167 <== osd.7 v2:172.21.15.191:6808/34334 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f96840754c0 con 0x7f96840729d0 2024-03-27T21:42:14.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.450+0000 7f967ffff640 10 client.5538.objecter ms_dispatch 0x7f9698000b90 command_reply(tid 1: 0 ) v1 2024-03-27T21:42:14.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.450+0000 7f967ffff640 10 client.5538.objecter _finish_command 1 = system:0 2024-03-27T21:42:14.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.450+0000 7f967ffff640 20 client.5538.objecter put_session s=0x7f9684072300 osd=7 4 2024-03-27T21:42:14.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.450+0000 7f967ffff640 15 client.5538.objecter _session_command_op_remove 7 1 2024-03-27T21:42:14.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.462+0000 7f969dce0640 10 client.5538.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-27T21:42:14.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.462+0000 7f969dce0640 15 client.5538.objecter _session_command_op_assign -1 2 2024-03-27T21:42:14.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.462+0000 7f969dce0640 20 client.5538.objecter _get_session s=0x7f9684072300 osd=7 3 2024-03-27T21:42:14.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.462+0000 7f969dce0640 20 client.5538.objecter put_session s=0x7f9684072300 osd=7 3 2024-03-27T21:42:14.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.462+0000 7f969dce0640 20 client.5538.objecter _get_session s=0x7f9684072300 osd=7 3 2024-03-27T21:42:14.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.462+0000 7f969dce0640 15 client.5538.objecter _session_command_op_remove -1 2 2024-03-27T21:42:14.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.462+0000 7f969dce0640 20 client.5538.objecter get_session s=0x7f9684072300 osd=7 3 2024-03-27T21:42:14.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.462+0000 7f969dce0640 15 client.5538.objecter _session_command_op_assign 7 2 2024-03-27T21:42:14.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.462+0000 7f969dce0640 20 client.5538.objecter put_session s=0x7f9684072300 osd=7 4 2024-03-27T21:42:14.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.462+0000 7f969dce0640 10 client.5538.objecter _send_command 2 2024-03-27T21:42:14.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.462+0000 7f969dce0640 1 -- 172.21.15.60:0/1436261167 --> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9698140940 con 0x7f96840729d0 2024-03-27T21:42:14.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f967ffff640 1 -- 172.21.15.60:0/1436261167 <== osd.7 v2:172.21.15.191:6808/34334 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f9698140940 con 0x7f96840729d0 2024-03-27T21:42:14.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f967ffff640 10 client.5538.objecter ms_dispatch 0x7f9698000b90 command_reply(tid 2: 0 ) v1 2024-03-27T21:42:14.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f967ffff640 10 client.5538.objecter _finish_command 2 = system:0 2024-03-27T21:42:14.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f967ffff640 20 client.5538.objecter put_session s=0x7f9684072300 osd=7 4 2024-03-27T21:42:14.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f967ffff640 15 client.5538.objecter _session_command_op_remove 7 2 2024-03-27T21:42:14.477 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-27T21:42:14.477 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-27T21:42:14.477 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-27T21:42:14.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f969dce0640 10 librados: watch_flush enter 2024-03-27T21:42:14.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f969dce0640 10 librados: watch_flush exit 2024-03-27T21:42:14.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f969dce0640 10 client.5538.objecter close_session for osd.7 2024-03-27T21:42:14.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f969dce0640 1 -- 172.21.15.60:0/1436261167 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f96840729d0 msgr2=0x7f9684074df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f969dce0640 1 --2- 172.21.15.60:0/1436261167 >> [v2:172.21.15.191:6808/34334,v1:172.21.15.191:6810/34334] conn(0x7f96840729d0 0x7f9684074df0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f969dce0640 20 client.5538.objecter put_session s=0x7f9684072300 osd=7 1 2024-03-27T21:42:14.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f969dce0640 20 client.5538.objecter shutdown clearing up homeless session... 2024-03-27T21:42:14.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f969dce0640 10 client.5538.objecter successfully canceled tick 2024-03-27T21:42:14.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f969dce0640 1 -- 172.21.15.60:0/1436261167 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f968406c590 msgr2=0x7f968406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f969dce0640 1 --2- 172.21.15.60:0/1436261167 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f968406c590 0x7f968406ea50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f968c035000 tx=0x7f968c008030 comp rx=0 tx=0).stop 2024-03-27T21:42:14.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f969dce0640 10 monclient: shutdown 2024-03-27T21:42:14.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f969dce0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:14.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f969dce0640 1 -- 172.21.15.60:0/1436261167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9698071a40 msgr2=0x7f969813f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f969dce0640 1 --2- 172.21.15.60:0/1436261167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9698071a40 0x7f969813f5a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f968800c430 tx=0x7f968800c8f0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f9697fff640 1 -- 172.21.15.60:0/1436261167 reap_dead start 2024-03-27T21:42:14.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f969dce0640 1 -- 172.21.15.60:0/1436261167 shutdown_connections 2024-03-27T21:42:14.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f969dce0640 1 -- 172.21.15.60:0/1436261167 >> 172.21.15.60:0/1436261167 conn(0x7f96980b7300 msgr2=0x7f96980a8500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:14.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f969dce0640 1 -- 172.21.15.60:0/1436261167 shutdown_connections 2024-03-27T21:42:14.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.463+0000 7f969dce0640 1 -- 172.21.15.60:0/1436261167 wait complete. 2024-03-27T21:42:14.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.464+0000 7f969dce0640 1 librados: shutdown 2024-03-27T21:42:14.481 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-27T21:42:14.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.596+0000 7f4caed81640 10 monclient: get_monmap_and_config 2024-03-27T21:42:14.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.596+0000 7f4caed81640 10 monclient: build_initial_monmap 2024-03-27T21:42:14.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.596+0000 7f4caed81640 10 monclient: monmap: 2024-03-27T21:42:14.583 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:14.583 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.583 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:14.597415+0000 2024-03-27T21:42:14.583 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:14.597415+0000 2024-03-27T21:42:14.583 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:14.583 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.583 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:14.584 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:14.584 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:14.584 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:14.584 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:14.584 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4caed81640 1 Processor -- start 2024-03-27T21:42:14.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4caed81640 1 -- start start 2024-03-27T21:42:14.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4caed81640 10 monclient: init 2024-03-27T21:42:14.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4caed81640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:14.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4caed81640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-03-27T21:42:14.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4caed81640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ca8071a50 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:14.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4caed81640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4ca813ef20 0x7f4ca80a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4caed81640 10 monclient(hunting): picked mon.noname-a con 0x7f4ca813ef20 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:14.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4caed81640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ca8141630 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:14.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4caed81640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca80a8e90 con 0x7f4ca8141630 2024-03-27T21:42:14.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4caed81640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca80aa040 con 0x7f4ca8071a50 2024-03-27T21:42:14.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4caed81640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:14.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4caed81640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca8071e30 con 0x7f4ca813ef20 2024-03-27T21:42:14.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4caed81640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:14.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4caed81640 10 monclient(hunting): authenticate will time out at 975.818664s 2024-03-27T21:42:14.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4ca7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4ca813ef20 0x7f4ca80a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4ca7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4ca813ef20 0x7f4ca80a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59416/0 (socket says 172.21.15.60:59416) 2024-03-27T21:42:14.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4ca7fff640 1 -- 172.21.15.60:0/1490303826 learned_addr learned my addr 172.21.15.60:0/1490303826 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:14.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4ca7fff640 10 monclient(hunting): get_auth_request con 0x7f4ca813ef20 auth_method 0 2024-03-27T21:42:14.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4ca7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:14.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4ca7fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:14.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4ca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:14.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4ca7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:14.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.597+0000 7f4ca7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:14.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.598+0000 7f4ca7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:14.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.598+0000 7f4ca7fff640 10 monclient(hunting): handle_auth_done global_id 5541 payload 274 2024-03-27T21:42:14.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.598+0000 7f4ca7fff640 1 -- 172.21.15.60:0/1490303826 >> v1:172.21.15.191:6789/0 conn(0x7f4ca8071a50 legacy=0x7f4ca813e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.598+0000 7f4ca7fff640 1 -- 172.21.15.60:0/1490303826 >> v1:172.21.15.60:6789/0 conn(0x7f4ca8141630 legacy=0x7f4ca8141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.598+0000 7f4ca7fff640 10 monclient: _finish_hunting 0 2024-03-27T21:42:14.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.598+0000 7f4ca7fff640 1 monclient: found mon.noname-a 2024-03-27T21:42:14.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.598+0000 7f4ca7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:14.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.598+0000 7f4ca7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:14.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.598+0000 7f4ca7fff640 1 -- 172.21.15.60:0/1490303826 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca80aa2b0 con 0x7f4ca813ef20 2024-03-27T21:42:14.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.598+0000 7f4ca7fff640 1 --2- 172.21.15.60:0/1490303826 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ca813ef20 0x7f4ca80a87e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4c94009870 tx=0x7f4c9402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a94bcee93acdc532 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:14.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4ca77fe640 1 -- 172.21.15.60:0/1490303826 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9402f840 con 0x7f4ca813ef20 2024-03-27T21:42:14.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4ca77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:14.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4ca77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:14.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4ca77fe640 10 monclient: dump: 2024-03-27T21:42:14.605 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:14.605 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.606 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.606 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.606 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:14.606 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.607 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:14.607 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:14.607 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:14.608 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4ca77fe640 10 monclient: _finish_auth 0 2024-03-27T21:42:14.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4ca77fe640 10 monclient: _check_auth_tickets 2024-03-27T21:42:14.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4ca77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:14.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4caed81640 5 monclient: authenticate success, global_id 5541 2024-03-27T21:42:14.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4ca77fe640 1 -- 172.21.15.60:0/1490303826 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9402f9a0 con 0x7f4ca813ef20 2024-03-27T21:42:14.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4caed81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:14.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4ca77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:14.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4caed81640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:14.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4ca77fe640 1 -- 172.21.15.60:0/1490303826 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9402fd30 con 0x7f4ca813ef20 2024-03-27T21:42:14.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4caed81640 4 set_mon_vals no callback set 2024-03-27T21:42:14.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4ca77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:14.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4ca77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:14.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4ca77fe640 10 monclient: dump: 2024-03-27T21:42:14.611 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:14.611 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.612 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.612 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.612 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:14.612 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.613 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:14.613 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:14.613 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:14.614 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4caed81640 10 monclient: shutdown 2024-03-27T21:42:14.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4caed81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:14.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4caed81640 1 -- 172.21.15.60:0/1490303826 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ca813ef20 msgr2=0x7f4ca80a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4caed81640 1 --2- 172.21.15.60:0/1490303826 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ca813ef20 0x7f4ca80a87e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4c94009870 tx=0x7f4c9402ba20 comp rx=0 tx=0).stop 2024-03-27T21:42:14.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4caed81640 1 -- 172.21.15.60:0/1490303826 shutdown_connections 2024-03-27T21:42:14.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4caed81640 1 --2- 172.21.15.60:0/1490303826 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ca813ef20 0x7f4ca80a87e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4caed81640 1 -- 172.21.15.60:0/1490303826 >> 172.21.15.60:0/1490303826 conn(0x7f4ca80b7300 msgr2=0x7f4ca80b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:14.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4caed81640 1 -- 172.21.15.60:0/1490303826 shutdown_connections 2024-03-27T21:42:14.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.599+0000 7f4caed81640 1 -- 172.21.15.60:0/1490303826 wait complete. 2024-03-27T21:42:14.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.600+0000 7f4caed81640 10 monclient: build_initial_monmap 2024-03-27T21:42:14.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.600+0000 7f4caed81640 10 monclient: monmap: 2024-03-27T21:42:14.617 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:14.617 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:14.617 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:14.617 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:14.617 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:14.618 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:14.618 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:14.618 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:14.618 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:14.618 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.600+0000 7f4caed81640 1 librados: starting msgr at 2024-03-27T21:42:14.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.600+0000 7f4caed81640 1 librados: starting objecter 2024-03-27T21:42:14.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.600+0000 7f4caed81640 1 Processor -- start 2024-03-27T21:42:14.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.600+0000 7f4caed81640 1 -- start start 2024-03-27T21:42:14.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.600+0000 7f4caed81640 1 librados: setting wanted keys 2024-03-27T21:42:14.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.600+0000 7f4caed81640 1 librados: calling monclient init 2024-03-27T21:42:14.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.600+0000 7f4caed81640 10 monclient: init 2024-03-27T21:42:14.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4caed81640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:14.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4caed81640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-27T21:42:14.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4caed81640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ca8071a50 0x7f4ca81396d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4caed81640 10 monclient(hunting): picked mon.noname-a con 0x7f4ca8071a50 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:14.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4caed81640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4ca813ef20 0x7f4ca80aac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4caed81640 10 monclient(hunting): picked mon.noname-c con 0x7f4ca813ef20 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:14.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4caed81640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ca8141630 0x7f4ca8130d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4caed81640 10 monclient(hunting): picked mon.noname-b con 0x7f4ca8141630 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:14.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4caed81640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:14.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4caed81640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca8131e10 con 0x7f4ca8071a50 2024-03-27T21:42:14.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4caed81640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:14.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4caed81640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca8139cc0 con 0x7f4ca8141630 2024-03-27T21:42:14.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4caed81640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:14.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4caed81640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca80ab190 con 0x7f4ca813ef20 2024-03-27T21:42:14.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4caed81640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:14.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4caed81640 10 monclient(hunting): authenticate will time out at 975.822356s 2024-03-27T21:42:14.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cacb1d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ca8071a50 0x7f4ca81396d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4ca7fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4ca813ef20 0x7f4ca80aac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cacb1d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ca8071a50 0x7f4ca81396d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59424/0 (socket says 172.21.15.60:59424) 2024-03-27T21:42:14.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4ca7fff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4ca813ef20 0x7f4ca80aac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:51568/0 (socket says 172.21.15.60:51568) 2024-03-27T21:42:14.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cacb1d640 1 -- 172.21.15.60:0/695749229 learned_addr learned my addr 172.21.15.60:0/695749229 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:14.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cacb1d640 10 monclient(hunting): get_auth_request con 0x7f4ca8071a50 auth_method 0 2024-03-27T21:42:14.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cacb1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:14.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cacb1d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:14.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cacb1d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:14.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4ca7fff640 10 monclient(hunting): get_auth_request con 0x7f4ca813ef20 auth_method 0 2024-03-27T21:42:14.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4ca7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:14.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4ca7fff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:14.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4ca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:14.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cad31e640 1 --2- 172.21.15.60:0/695749229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ca8141630 0x7f4ca8130d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cad31e640 10 monclient(hunting): get_auth_request con 0x7f4ca8141630 auth_method 0 2024-03-27T21:42:14.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cad31e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:14.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cad31e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:14.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cad31e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:14.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cacb1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:14.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cacb1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:14.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cacb1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:14.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4ca7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:14.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4ca7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:14.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4ca7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:14.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cad31e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:14.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cad31e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:14.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cad31e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:14.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cacb1d640 10 monclient(hunting): handle_auth_done global_id 5547 payload 824 2024-03-27T21:42:14.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cacb1d640 1 -- 172.21.15.60:0/695749229 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4ca813ef20 msgr2=0x7f4ca80aac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cacb1d640 1 --2- 172.21.15.60:0/695749229 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4ca813ef20 0x7f4ca80aac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cacb1d640 1 -- 172.21.15.60:0/695749229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ca8141630 msgr2=0x7f4ca8130d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cacb1d640 1 --2- 172.21.15.60:0/695749229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ca8141630 0x7f4ca8130d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cacb1d640 10 monclient: _finish_hunting 0 2024-03-27T21:42:14.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cacb1d640 1 monclient: found mon.noname-a 2024-03-27T21:42:14.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cacb1d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:14.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cacb1d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:14.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4cacb1d640 1 -- 172.21.15.60:0/695749229 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c940094d0 con 0x7f4ca8071a50 2024-03-27T21:42:14.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.601+0000 7f4ca7fff640 1 --2- 172.21.15.60:0/695749229 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4ca813ef20 0x7f4ca80aac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:14.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4cacb1d640 1 --2- 172.21.15.60:0/695749229 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ca8071a50 0x7f4ca81396d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00c450 tx=0x7f4c9c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:14.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4ca57fa640 10 client.?.objecter ms_handle_connect 0x7f4ca8071a50 2024-03-27T21:42:14.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4ca57fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:14.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4ca57fa640 1 -- 172.21.15.60:0/695749229 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c0052d0 con 0x7f4ca8071a50 2024-03-27T21:42:14.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4ca57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:14.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4ca57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:14.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4ca57fa640 10 monclient: dump: 2024-03-27T21:42:14.628 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:14.628 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.628 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.629 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.629 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:14.629 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.629 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:14.629 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:14.629 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:14.629 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4ca57fa640 10 monclient: _finish_auth 0 2024-03-27T21:42:14.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4ca57fa640 10 monclient: _check_auth_tickets 2024-03-27T21:42:14.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4ca57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:14.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4ca57fa640 1 -- 172.21.15.60:0/695749229 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9c005430 con 0x7f4ca8071a50 2024-03-27T21:42:14.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4caed81640 5 monclient: authenticate success, global_id 5547 2024-03-27T21:42:14.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4caed81640 10 monclient: _renew_subs 2024-03-27T21:42:14.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4caed81640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:14.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4caed81640 1 -- 172.21.15.60:0/695749229 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca813ae20 con 0x7f4ca8071a50 2024-03-27T21:42:14.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4ca57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:14.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4ca57fa640 1 -- 172.21.15.60:0/695749229 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c012870 con 0x7f4ca8071a50 2024-03-27T21:42:14.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4ca57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:14.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4ca57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:14.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4ca6ffd640 4 set_mon_vals no callback set 2024-03-27T21:42:14.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4ca57fa640 10 monclient: dump: 2024-03-27T21:42:14.631 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:14.631 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.631 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.631 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.631 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:14.632 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.632 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:14.632 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:14.632 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:14.632 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4caed81640 10 client.5547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:14.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4caed81640 10 monclient: _renew_subs 2024-03-27T21:42:14.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4caed81640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:14.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4caed81640 1 -- 172.21.15.60:0/695749229 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca8138a30 con 0x7f4ca8071a50 2024-03-27T21:42:14.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.602+0000 7f4caed81640 1 librados: init done 2024-03-27T21:42:14.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.603+0000 7f4caed81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:14.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.603+0000 7f4caed81640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:14.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.603+0000 7f4caed81640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:14.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.603+0000 7f4caed81640 1 -- 172.21.15.60:0/695749229 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca8067900 con 0x7f4ca8071a50 2024-03-27T21:42:14.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.603+0000 7f4ca57fa640 1 -- 172.21.15.60:0/695749229 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4c9c0129d0 con 0x7f4ca8071a50 2024-03-27T21:42:14.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.603+0000 7f4ca57fa640 1 --2- 172.21.15.60:0/695749229 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4c8c06c5c0 0x7f4c8c06ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.604+0000 7f4ca57fa640 1 -- 172.21.15.60:0/695749229 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f4c9c088ab0 con 0x7f4ca8071a50 2024-03-27T21:42:14.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.604+0000 7f4ca57fa640 10 client.5547.objecter ms_dispatch 0x7f4ca8000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:14.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.604+0000 7f4ca7fff640 1 --2- 172.21.15.60:0/695749229 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4c8c06c5c0 0x7f4c8c06ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.604+0000 7f4ca57fa640 3 client.5547.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:14.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.604+0000 7f4ca57fa640 3 client.5547.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:14.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.604+0000 7f4ca7fff640 10 monclient: get_auth_request con 0x7f4c8c06c5c0 auth_method 0 2024-03-27T21:42:14.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.604+0000 7f4ca57fa640 20 client.5547.objecter dump_active .. 0 homeless 2024-03-27T21:42:14.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.604+0000 7f4ca57fa640 10 client.5547.objecter ms_handle_connect 0x7f4c8c06c5c0 2024-03-27T21:42:14.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.604+0000 7f4ca7fff640 1 --2- 172.21.15.60:0/695749229 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4c8c06c5c0 0x7f4c8c06ea80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4c94036000 tx=0x7f4c94008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:14.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.608+0000 7f4ca57fa640 1 -- 172.21.15.60:0/695749229 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4c9c053fa0 con 0x7f4ca8071a50 2024-03-27T21:42:14.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.608+0000 7f4ca57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:14.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.608+0000 7f4ca57fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:14.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.787+0000 7f4caed81640 1 -- 172.21.15.60:0/695749229 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ca8139e00 con 0x7f4c8c06c5c0 2024-03-27T21:42:14.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.788+0000 7f4ca57fa640 1 -- 172.21.15.60:0/695749229 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4c94037480 con 0x7f4c8c06c5c0 2024-03-27T21:42:14.774 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 7 to deep-scrub 2024-03-27T21:42:14.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.789+0000 7f4caed81640 10 librados: watch_flush enter 2024-03-27T21:42:14.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.789+0000 7f4caed81640 10 librados: watch_flush exit 2024-03-27T21:42:14.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.789+0000 7f4caed81640 20 client.5547.objecter shutdown clearing up homeless session... 2024-03-27T21:42:14.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.789+0000 7f4caed81640 10 client.5547.objecter successfully canceled tick 2024-03-27T21:42:14.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.789+0000 7f4caed81640 1 -- 172.21.15.60:0/695749229 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4c8c06c5c0 msgr2=0x7f4c8c06ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.789+0000 7f4caed81640 1 --2- 172.21.15.60:0/695749229 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4c8c06c5c0 0x7f4c8c06ea80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4c94036000 tx=0x7f4c94008030 comp rx=0 tx=0).stop 2024-03-27T21:42:14.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.789+0000 7f4caed81640 10 monclient: shutdown 2024-03-27T21:42:14.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.789+0000 7f4caed81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:14.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.789+0000 7f4caed81640 1 -- 172.21.15.60:0/695749229 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ca8071a50 msgr2=0x7f4ca81396d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.789+0000 7f4caed81640 1 --2- 172.21.15.60:0/695749229 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ca8071a50 0x7f4ca81396d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00c450 tx=0x7f4c9c00c910 comp rx=0 tx=0).stop 2024-03-27T21:42:14.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.790+0000 7f4caed81640 1 -- 172.21.15.60:0/695749229 shutdown_connections 2024-03-27T21:42:14.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.790+0000 7f4caed81640 1 --2- 172.21.15.60:0/695749229 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f4c8c06c5c0 0x7f4c8c06ea80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.790+0000 7f4caed81640 1 --2- 172.21.15.60:0/695749229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ca8141630 0x7f4ca8130d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.790+0000 7f4caed81640 1 --2- 172.21.15.60:0/695749229 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f4ca813ef20 0x7f4ca80aac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.790+0000 7f4caed81640 1 --2- 172.21.15.60:0/695749229 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ca8071a50 0x7f4ca81396d0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.790+0000 7f4caed81640 1 -- 172.21.15.60:0/695749229 >> 172.21.15.60:0/695749229 conn(0x7f4ca80b7300 msgr2=0x7f4ca813e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:14.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.790+0000 7f4caed81640 1 -- 172.21.15.60:0/695749229 shutdown_connections 2024-03-27T21:42:14.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.790+0000 7f4caed81640 1 -- 172.21.15.60:0/695749229 wait complete. 2024-03-27T21:42:14.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.790+0000 7f4caed81640 1 librados: shutdown 2024-03-27T21:42:14.787 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T21:42:14.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.900+0000 7f812b52e640 10 monclient: get_monmap_and_config 2024-03-27T21:42:14.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.900+0000 7f812b52e640 10 monclient: build_initial_monmap 2024-03-27T21:42:14.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.900+0000 7f812b52e640 10 monclient: monmap: 2024-03-27T21:42:14.888 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:14.889 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.889 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:14.901207+0000 2024-03-27T21:42:14.890 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:14.901207+0000 2024-03-27T21:42:14.890 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:14.891 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.891 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:14.891 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:14.891 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:14.892 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:14.892 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:14.892 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.900+0000 7f812b52e640 1 Processor -- start 2024-03-27T21:42:14.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.900+0000 7f812b52e640 1 -- start start 2024-03-27T21:42:14.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.900+0000 7f812b52e640 10 monclient: init 2024-03-27T21:42:14.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f812b52e640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:14.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f812b52e640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-03-27T21:42:14.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f812b52e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81240b40b0 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:14.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f812b52e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f812413b020 0x7f812413b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f812b52e640 10 monclient(hunting): picked mon.noname-a con 0x7f812413b020 addr v2:172.21.15.60:3300/0 2024-03-27T21:42:14.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f812b52e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f81241416f0 0x7f8124141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f812b52e640 10 monclient(hunting): picked mon.noname-b con 0x7f81241416f0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:14.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f812b52e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81240b3830 con 0x7f81240b40b0 2024-03-27T21:42:14.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f812b52e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:14.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f812b52e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81240b4490 con 0x7f812413b020 2024-03-27T21:42:14.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f812b52e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:14.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f812b52e640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81240aa520 con 0x7f81241416f0 2024-03-27T21:42:14.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f812b52e640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:14.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f812b52e640 10 monclient(hunting): authenticate will time out at 976.122471s 2024-03-27T21:42:14.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8128ac9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f812413b020 0x7f812413b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8129acb640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f81241416f0 0x7f8124141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8128ac9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f812413b020 0x7f812413b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59426/0 (socket says 172.21.15.60:59426) 2024-03-27T21:42:14.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8128ac9640 1 -- 172.21.15.60:0/339564411 learned_addr learned my addr 172.21.15.60:0/339564411 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:14.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8128ac9640 10 monclient(hunting): get_auth_request con 0x7f812413b020 auth_method 0 2024-03-27T21:42:14.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8128ac9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:14.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8128ac9640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:14.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8128ac9640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:14.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8129acb640 10 monclient(hunting): get_auth_request con 0x7f81241416f0 auth_method 0 2024-03-27T21:42:14.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8129acb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:14.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8129acb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:14.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8129acb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:14.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8128ac9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:14.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8128ac9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:14.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8128ac9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8129acb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8129acb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8129acb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8128ac9640 10 monclient(hunting): handle_auth_done global_id 5550 payload 274 2024-03-27T21:42:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8128ac9640 1 -- 172.21.15.60:0/339564411 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f81241416f0 msgr2=0x7f8124141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8128ac9640 1 --2- 172.21.15.60:0/339564411 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f81241416f0 0x7f8124141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8128ac9640 1 -- 172.21.15.60:0/339564411 >> v1:172.21.15.60:6789/0 conn(0x7f81240b40b0 legacy=0x7f81240a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8128ac9640 10 monclient: _finish_hunting 0 2024-03-27T21:42:14.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8128ac9640 1 monclient: found mon.noname-a 2024-03-27T21:42:14.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8128ac9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:14.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8128ac9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:14.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8128ac9640 1 -- 172.21.15.60:0/339564411 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81240a9470 con 0x7f812413b020 2024-03-27T21:42:14.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.901+0000 7f8129acb640 1 --2- 172.21.15.60:0/339564411 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f81241416f0 0x7f8124141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:14.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.902+0000 7f8128ac9640 1 --2- 172.21.15.60:0/339564411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f812413b020 0x7f812413b480 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8114009870 tx=0x7f811402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dcb2387afe74df0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:14.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.902+0000 7f811bfff640 1 -- 172.21.15.60:0/339564411 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8114030840 con 0x7f812413b020 2024-03-27T21:42:14.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.902+0000 7f811bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:14.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.902+0000 7f811bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:14.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.902+0000 7f811bfff640 10 monclient: dump: 2024-03-27T21:42:14.920 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:14.920 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.920 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.920 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.920 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:14.920 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.920 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:14.920 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:14.920 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:14.921 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.902+0000 7f811bfff640 10 monclient: _finish_auth 0 2024-03-27T21:42:14.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.902+0000 7f811bfff640 10 monclient: _check_auth_tickets 2024-03-27T21:42:14.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.902+0000 7f811bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:14.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.902+0000 7f812b52e640 5 monclient: authenticate success, global_id 5550 2024-03-27T21:42:14.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.902+0000 7f811bfff640 1 -- 172.21.15.60:0/339564411 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81140309a0 con 0x7f812413b020 2024-03-27T21:42:14.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.902+0000 7f812b52e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:14.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.902+0000 7f811bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:14.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.902+0000 7f812b52e640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:14.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.902+0000 7f812b52e640 4 set_mon_vals no callback set 2024-03-27T21:42:14.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.902+0000 7f811bfff640 1 -- 172.21.15.60:0/339564411 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8114030d30 con 0x7f812413b020 2024-03-27T21:42:14.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.902+0000 7f811bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:14.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.902+0000 7f811bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:14.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.902+0000 7f811bfff640 10 monclient: dump: 2024-03-27T21:42:14.922 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:14.923 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.923 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.923 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.923 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:14.923 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.923 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:14.923 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:14.923 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:14.924 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.903+0000 7f812b52e640 10 monclient: shutdown 2024-03-27T21:42:14.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.903+0000 7f812b52e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:14.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.903+0000 7f812b52e640 1 -- 172.21.15.60:0/339564411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f812413b020 msgr2=0x7f812413b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.903+0000 7f812b52e640 1 --2- 172.21.15.60:0/339564411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f812413b020 0x7f812413b480 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8114009870 tx=0x7f811402b980 comp rx=0 tx=0).stop 2024-03-27T21:42:14.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.903+0000 7f812b52e640 1 -- 172.21.15.60:0/339564411 shutdown_connections 2024-03-27T21:42:14.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.903+0000 7f812b52e640 1 --2- 172.21.15.60:0/339564411 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f81241416f0 0x7f8124141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.903+0000 7f812b52e640 1 --2- 172.21.15.60:0/339564411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f812413b020 0x7f812413b480 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.903+0000 7f812b52e640 1 -- 172.21.15.60:0/339564411 >> 172.21.15.60:0/339564411 conn(0x7f81240608e0 msgr2=0x7f8124060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:14.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.903+0000 7f812b52e640 1 -- 172.21.15.60:0/339564411 shutdown_connections 2024-03-27T21:42:14.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.903+0000 7f812b52e640 1 -- 172.21.15.60:0/339564411 wait complete. 2024-03-27T21:42:14.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.904+0000 7f812b52e640 10 monclient: build_initial_monmap 2024-03-27T21:42:14.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.904+0000 7f812b52e640 10 monclient: monmap: 2024-03-27T21:42:14.925 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:14.925 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:14.925 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:14.925 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:14.926 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:14.926 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:14.926 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:14.926 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:14.926 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:14.926 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.904+0000 7f812b52e640 1 librados: starting msgr at 2024-03-27T21:42:14.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.904+0000 7f812b52e640 1 librados: starting objecter 2024-03-27T21:42:14.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.904+0000 7f812b52e640 1 Processor -- start 2024-03-27T21:42:14.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.904+0000 7f812b52e640 1 -- start start 2024-03-27T21:42:14.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.904+0000 7f812b52e640 1 librados: setting wanted keys 2024-03-27T21:42:14.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.904+0000 7f812b52e640 1 librados: calling monclient init 2024-03-27T21:42:14.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.904+0000 7f812b52e640 10 monclient: init 2024-03-27T21:42:14.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.904+0000 7f812b52e640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:14.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.904+0000 7f812b52e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:42:14.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.904+0000 7f812b52e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f81240b40b0 0x7f81240aac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.904+0000 7f812b52e640 10 monclient(hunting): picked mon.noname-c con 0x7f81240b40b0 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:14.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.904+0000 7f812b52e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f812413b020 0x7f81241d68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.904+0000 7f812b52e640 10 monclient(hunting): picked mon.noname-b con 0x7f812413b020 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:14.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.904+0000 7f812b52e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81241416f0 0x7f81241d6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.904+0000 7f812b52e640 10 monclient(hunting): picked mon.noname-a con 0x7f81241416f0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:14.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f812b52e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:14.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f812b52e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81241d64b0 con 0x7f81241416f0 2024-03-27T21:42:14.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f812b52e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:14.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f812b52e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81240abe80 con 0x7f812413b020 2024-03-27T21:42:14.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f812b52e640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:14.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f812b52e640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f812413b9c0 con 0x7f81240b40b0 2024-03-27T21:42:14.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f812b52e640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:14.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f812b52e640 10 monclient(hunting): authenticate will time out at 976.125944s 2024-03-27T21:42:14.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f81292ca640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f81240b40b0 0x7f81240aac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f81292ca640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f81240b40b0 0x7f81240aac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:51578/0 (socket says 172.21.15.60:51578) 2024-03-27T21:42:14.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f81292ca640 1 -- 172.21.15.60:0/2094563174 learned_addr learned my addr 172.21.15.60:0/2094563174 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:14.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f81292ca640 10 monclient(hunting): get_auth_request con 0x7f81240b40b0 auth_method 0 2024-03-27T21:42:14.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f81292ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:14.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f81292ca640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:14.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f81292ca640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:14.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8129acb640 1 --2- 172.21.15.60:0/2094563174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81241416f0 0x7f81241d6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8129acb640 10 monclient(hunting): get_auth_request con 0x7f81241416f0 auth_method 0 2024-03-27T21:42:14.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8129acb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:14.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8129acb640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:14.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8129acb640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:14.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f81292ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:14.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f81292ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:14.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f81292ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:14.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8129acb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:14.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8129acb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:14.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8129acb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:14.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8128ac9640 1 --2- 172.21.15.60:0/2094563174 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f812413b020 0x7f81241d68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f81292ca640 10 monclient(hunting): handle_auth_done global_id 5204 payload 824 2024-03-27T21:42:14.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f81292ca640 1 -- 172.21.15.60:0/2094563174 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f812413b020 msgr2=0x7f81241d68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f81292ca640 1 --2- 172.21.15.60:0/2094563174 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f812413b020 0x7f81241d68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f81292ca640 1 -- 172.21.15.60:0/2094563174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81241416f0 msgr2=0x7f81241d6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:14.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f81292ca640 1 --2- 172.21.15.60:0/2094563174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81241416f0 0x7f81241d6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:14.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f81292ca640 10 monclient: _finish_hunting 0 2024-03-27T21:42:14.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f81292ca640 1 monclient: found mon.noname-c 2024-03-27T21:42:14.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f81292ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:14.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f81292ca640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.60:3301/0 2024-03-27T21:42:14.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f81292ca640 1 -- 172.21.15.60:0/2094563174 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8120009a70 con 0x7f81240b40b0 2024-03-27T21:42:14.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8129acb640 1 --2- 172.21.15.60:0/2094563174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81241416f0 0x7f81241d6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:14.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f81292ca640 1 --2- 172.21.15.60:0/2094563174 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f81240b40b0 0x7f81240aac90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f811c00c400 tx=0x7f811c00c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:14.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8119ffb640 10 client.?.objecter ms_handle_connect 0x7f81240b40b0 2024-03-27T21:42:14.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8119ffb640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:14.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8119ffb640 1 -- 172.21.15.60:0/2094563174 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c0052d0 con 0x7f81240b40b0 2024-03-27T21:42:14.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8119ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:14.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8119ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-27T21:42:14.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8119ffb640 10 monclient: dump: 2024-03-27T21:42:14.933 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:14.933 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.933 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.933 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.934 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:14.934 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.934 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:14.934 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:14.934 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:14.934 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8119ffb640 10 monclient: _finish_auth 0 2024-03-27T21:42:14.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8119ffb640 10 monclient: _check_auth_tickets 2024-03-27T21:42:14.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8119ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:14.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8119ffb640 1 -- 172.21.15.60:0/2094563174 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811c005430 con 0x7f81240b40b0 2024-03-27T21:42:14.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f812b52e640 5 monclient: authenticate success, global_id 5204 2024-03-27T21:42:14.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f812b52e640 10 monclient: _renew_subs 2024-03-27T21:42:14.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f812b52e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:14.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f812b52e640 1 -- 172.21.15.60:0/2094563174 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81140094d0 con 0x7f81240b40b0 2024-03-27T21:42:14.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8119ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:14.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8119ffb640 1 -- 172.21.15.60:0/2094563174 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c003830 con 0x7f81240b40b0 2024-03-27T21:42:14.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8119ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:14.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8119ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:14.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f8119ffb640 10 monclient: dump: 2024-03-27T21:42:14.936 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:14.936 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:14.936 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.936 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:14.936 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:14.936 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:14.937 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:14.937 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:14.937 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:14.937 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:14.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f812b52e640 10 client.5204.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:14.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f812b52e640 10 monclient: _renew_subs 2024-03-27T21:42:14.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f811b7fe640 4 set_mon_vals no callback set 2024-03-27T21:42:14.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f812b52e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:14.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f812b52e640 1 -- 172.21.15.60:0/2094563174 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f812413e8b0 con 0x7f81240b40b0 2024-03-27T21:42:14.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.905+0000 7f812b52e640 1 librados: init done 2024-03-27T21:42:14.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.906+0000 7f812b52e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:14.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.906+0000 7f812b52e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:14.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.906+0000 7f812b52e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.60:3301/0 2024-03-27T21:42:14.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.906+0000 7f812b52e640 1 -- 172.21.15.60:0/2094563174 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81241d65f0 con 0x7f81240b40b0 2024-03-27T21:42:14.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.907+0000 7f8119ffb640 1 -- 172.21.15.60:0/2094563174 <== mon.2 v2:172.21.15.60:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f811c003990 con 0x7f81240b40b0 2024-03-27T21:42:14.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.907+0000 7f8119ffb640 1 --2- 172.21.15.60:0/2094563174 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f810806c590 0x7f810806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:14.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.907+0000 7f8119ffb640 1 -- 172.21.15.60:0/2094563174 <== mon.2 v2:172.21.15.60:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f811c0885c0 con 0x7f81240b40b0 2024-03-27T21:42:14.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.907+0000 7f8119ffb640 10 client.5204.objecter ms_dispatch 0x7f8124000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:14.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.907+0000 7f8119ffb640 3 client.5204.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:14.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.907+0000 7f8119ffb640 3 client.5204.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:14.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.907+0000 7f8128ac9640 1 --2- 172.21.15.60:0/2094563174 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f810806c590 0x7f810806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:14.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.907+0000 7f8119ffb640 20 client.5204.objecter dump_active .. 0 homeless 2024-03-27T21:42:14.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.907+0000 7f8128ac9640 10 monclient: get_auth_request con 0x7f810806c590 auth_method 0 2024-03-27T21:42:14.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.908+0000 7f8119ffb640 10 client.5204.objecter ms_handle_connect 0x7f810806c590 2024-03-27T21:42:14.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.908+0000 7f8128ac9640 1 --2- 172.21.15.60:0/2094563174 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f810806c590 0x7f810806ea50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8114036000 tx=0x7f811403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:14.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.911+0000 7f8119ffb640 1 -- 172.21.15.60:0/2094563174 <== mon.2 v2:172.21.15.60:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f81241d65f0 con 0x7f81240b40b0 2024-03-27T21:42:14.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.911+0000 7f8119ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:14.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:14.911+0000 7f8119ffb640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:15.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:15.087+0000 7f812b52e640 1 -- 172.21.15.60:0/2094563174 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f81241d65f0 con 0x7f810806c590 2024-03-27T21:42:15.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:15.090+0000 7f8119ffb640 1 -- 172.21.15.60:0/2094563174 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+66767 (secure 0 0 0) 0x7f81241d65f0 con 0x7f810806c590 2024-03-27T21:42:15.077 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:42:15.077 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-03-27T21:42:15.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:15.091+0000 7f812b52e640 10 librados: watch_flush enter 2024-03-27T21:42:15.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:15.091+0000 7f812b52e640 10 librados: watch_flush exit 2024-03-27T21:42:15.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:15.091+0000 7f812b52e640 20 client.5204.objecter shutdown clearing up homeless session... 2024-03-27T21:42:15.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:15.091+0000 7f812b52e640 10 client.5204.objecter successfully canceled tick 2024-03-27T21:42:15.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:15.092+0000 7f812b52e640 1 -- 172.21.15.60:0/2094563174 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f810806c590 msgr2=0x7f810806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:15.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:15.092+0000 7f812b52e640 1 --2- 172.21.15.60:0/2094563174 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f810806c590 0x7f810806ea50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8114036000 tx=0x7f811403b040 comp rx=0 tx=0).stop 2024-03-27T21:42:15.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:15.092+0000 7f812b52e640 10 monclient: shutdown 2024-03-27T21:42:15.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:15.092+0000 7f812b52e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:15.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:15.092+0000 7f812b52e640 1 -- 172.21.15.60:0/2094563174 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f81240b40b0 msgr2=0x7f81240aac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:15.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:15.092+0000 7f812b52e640 1 --2- 172.21.15.60:0/2094563174 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f81240b40b0 0x7f81240aac90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f811c00c400 tx=0x7f811c00c800 comp rx=0 tx=0).stop 2024-03-27T21:42:15.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:15.092+0000 7f812b52e640 1 -- 172.21.15.60:0/2094563174 shutdown_connections 2024-03-27T21:42:15.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:15.092+0000 7f812b52e640 1 --2- 172.21.15.60:0/2094563174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81241416f0 0x7f81241d6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:15.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:15.092+0000 7f812b52e640 1 --2- 172.21.15.60:0/2094563174 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f810806c590 0x7f810806ea50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:15.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:15.092+0000 7f812b52e640 1 --2- 172.21.15.60:0/2094563174 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f812413b020 0x7f81241d68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:15.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:15.092+0000 7f812b52e640 1 --2- 172.21.15.60:0/2094563174 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f81240b40b0 0x7f81240aac90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:15.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:15.092+0000 7f812b52e640 1 -- 172.21.15.60:0/2094563174 >> 172.21.15.60:0/2094563174 conn(0x7f81240608e0 msgr2=0x7f8124140540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:15.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:15.092+0000 7f812b52e640 1 -- 172.21.15.60:0/2094563174 shutdown_connections 2024-03-27T21:42:15.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:15.092+0000 7f812b52e640 1 -- 172.21.15.60:0/2094563174 wait complete. 2024-03-27T21:42:15.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:15.092+0000 7f812b52e640 1 librados: shutdown 2024-03-27T21:42:15.089 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-03-27T21:42:13.413566+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":230448,"kb_used_data":8520,"kb_used_omap":16,"kb_used_meta":221807,"kb_avail":754744272,"statfs":{"total":773094113280,"available":772858134528,"internally_reserved":0,"allocated":8724480,"data_stored":4732096,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":17199,"internal_metadata":227130577},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001435"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.602437+0000","last_change":"2024-03-27T21:42:01.265365+0000","last_active":"2024-03-27T21:42:03.602437+0000","last_peered":"2024-03-27T21:42:03.602437+0000","last_clean":"2024-03-27T21:42:03.602437+0000","last_became_active":"2024-03-27T21:42:01.265033+0000","last_became_peered":"2024-03-27T21:42:01.265033+0000","last_unstale":"2024-03-27T21:42:03.602437+0000","last_undegraded":"2024-03-27T21:42:03.602437+0000","last_fullsized":"2024-03-27T21:42:03.602437+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:07:09.198158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.601878+0000","last_change":"2024-03-27T21:41:49.474318+0000","last_active":"2024-03-27T21:42:03.601878+0000","last_peered":"2024-03-27T21:42:03.601878+0000","last_clean":"2024-03-27T21:42:03.601878+0000","last_became_active":"2024-03-27T21:41:49.474054+0000","last_became_peered":"2024-03-27T21:41:49.474054+0000","last_unstale":"2024-03-27T21:42:03.601878+0000","last_undegraded":"2024-03-27T21:42:03.601878+0000","last_fullsized":"2024-03-27T21:42:03.601878+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:36:26.918691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271058+0000","last_change":"2024-03-27T21:39:46.679788+0000","last_active":"2024-03-27T21:42:03.271058+0000","last_peered":"2024-03-27T21:42:03.271058+0000","last_clean":"2024-03-27T21:42:03.271058+0000","last_became_active":"2024-03-27T21:39:46.679561+0000","last_became_peered":"2024-03-27T21:39:46.679561+0000","last_unstale":"2024-03-27T21:42:03.271058+0000","last_undegraded":"2024-03-27T21:42:03.271058+0000","last_fullsized":"2024-03-27T21:42:03.271058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T23:53:50.373923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.601870+0000","last_change":"2024-03-27T21:41:49.474121+0000","last_active":"2024-03-27T21:42:03.601870+0000","last_peered":"2024-03-27T21:42:03.601870+0000","last_clean":"2024-03-27T21:42:03.601870+0000","last_became_active":"2024-03-27T21:41:49.473822+0000","last_became_peered":"2024-03-27T21:41:49.473822+0000","last_unstale":"2024-03-27T21:42:03.601870+0000","last_undegraded":"2024-03-27T21:42:03.601870+0000","last_fullsized":"2024-03-27T21:42:03.601870+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:39:51.253704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271086+0000","last_change":"2024-03-27T21:41:59.701200+0000","last_active":"2024-03-27T21:42:03.271086+0000","last_peered":"2024-03-27T21:42:03.271086+0000","last_clean":"2024-03-27T21:42:03.271086+0000","last_became_active":"2024-03-27T21:41:59.700966+0000","last_became_peered":"2024-03-27T21:41:59.700966+0000","last_unstale":"2024-03-27T21:42:03.271086+0000","last_undegraded":"2024-03-27T21:42:03.271086+0000","last_fullsized":"2024-03-27T21:42:03.271086+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:17:01.025558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-27T21:42:04.279853+0000","last_change":"2024-03-27T21:41:30.080153+0000","last_active":"2024-03-27T21:42:04.279853+0000","last_peered":"2024-03-27T21:42:04.279853+0000","last_clean":"2024-03-27T21:42:04.279853+0000","last_became_active":"2024-03-27T21:41:30.079829+0000","last_became_peered":"2024-03-27T21:41:30.079829+0000","last_unstale":"2024-03-27T21:42:04.279853+0000","last_undegraded":"2024-03-27T21:42:04.279853+0000","last_fullsized":"2024-03-27T21:42:04.279853+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":97,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271058+0000","last_change":"2024-03-27T21:41:49.197510+0000","last_active":"2024-03-27T21:42:03.271058+0000","last_peered":"2024-03-27T21:42:03.271058+0000","last_clean":"2024-03-27T21:42:03.271058+0000","last_became_active":"2024-03-27T21:41:49.197154+0000","last_became_peered":"2024-03-27T21:41:49.197154+0000","last_unstale":"2024-03-27T21:42:03.271058+0000","last_undegraded":"2024-03-27T21:42:03.271058+0000","last_fullsized":"2024-03-27T21:42:03.271058+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T04:18:21.622988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271153+0000","last_change":"2024-03-27T21:41:59.247145+0000","last_active":"2024-03-27T21:42:03.271153+0000","last_peered":"2024-03-27T21:42:03.271153+0000","last_clean":"2024-03-27T21:42:03.271153+0000","last_became_active":"2024-03-27T21:41:59.246895+0000","last_became_peered":"2024-03-27T21:41:59.246895+0000","last_unstale":"2024-03-27T21:42:03.271153+0000","last_undegraded":"2024-03-27T21:42:03.271153+0000","last_fullsized":"2024-03-27T21:42:03.271153+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:45.446242+0000","last_clean_scrub_stamp":"2024-03-27T21:39:45.446242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T00:40:39.749875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":234,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-27T21:42:03.271065+0000","last_change":"2024-03-27T21:39:39.866806+0000","last_active":"2024-03-27T21:42:03.271065+0000","last_peered":"2024-03-27T21:42:03.271065+0000","last_clean":"2024-03-27T21:42:03.271065+0000","last_became_active":"2024-03-27T21:39:39.865948+0000","last_became_peered":"2024-03-27T21:39:39.865948+0000","last_unstale":"2024-03-27T21:42:03.271065+0000","last_undegraded":"2024-03-27T21:42:03.271065+0000","last_fullsized":"2024-03-27T21:42:03.271065+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:39:38.411432+0000","last_clean_scrub_stamp":"2024-03-27T21:39:38.411432+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:36:49.375869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":16,"seq":68719476772,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.41599999999999998,"15min":0.41199999999999998},"min":{"1min":0.34399999999999997,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.621,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.47499999999999998,"15min":0.47599999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.66900000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.52800000000000002}]},{"osd":1,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45900000000000002,"15min":0.46500000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.53600000000000003,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.35899999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.56599999999999995,"5min":0.745,"15min":0.745},"last":0.61299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":3,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.50600000000000001,"15min":0.51400000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.56899999999999995,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.495,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.35999999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68400000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.629}]},{"osd":4,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.432,"15min":0.439},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.56399999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.46800000000000003,"15min":0.47399999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.44600000000000001}]},{"osd":6,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.45800000000000002,"15min":0.46300000000000002},"min":{"1min":0.318,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58699999999999997,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.29299999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64300000000000002,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.42399999999999999}]},{"osd":7,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.52400000000000002,"15min":0.53200000000000003},"min":{"1min":0.378,"5min":0.318,"15min":0.318},"max":{"1min":0.61299999999999999,"5min":0.745,"15min":0.745},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.50800000000000001,"15min":0.51400000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65000000000000002,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.50700000000000001}]}]},{"osd":3,"up_from":16,"seq":68719476771,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31848905},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.49299999999999999,"15min":0.504},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.91700000000000004,"5min":1.248,"15min":1.248},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.44900000000000001,"15min":0.45600000000000002},"min":{"1min":0.22800000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.90600000000000003,"5min":1.407,"15min":1.407},"last":1.4330000000000001}]},{"osd":1,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48199999999999998,"15min":0.47999999999999998},"min":{"1min":0.25600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":1.167},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.26800000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.96899999999999997,"5min":1.347,"15min":1.347},"last":0.77600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":4,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.61399999999999999,"15min":0.626},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83299999999999996,"5min":1.5,"15min":1.5},"last":1.375},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.40000000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72799999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":1.208}]},{"osd":5,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56000000000000005,"15min":0.56599999999999995},"min":{"1min":0.34999999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71099999999999997,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":1.2410000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.63300000000000001,"15min":0.64700000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.96799999999999997,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.92700000000000005}]},{"osd":6,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.63100000000000001,"15min":0.63900000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88200000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.405},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.40999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76700000000000002,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":1.2709999999999999}]},{"osd":7,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.64200000000000002,"15min":0.65600000000000003},"min":{"1min":0.36899999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.94399999999999995,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61599999999999999,"15min":0.622},"min":{"1min":0.36699999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85799999999999998,"5min":1.454,"15min":1.454},"last":1.341}]}]},{"osd":2,"up_from":62,"seq":266287972363,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29736,"kb_used_data":1052,"kb_used_omap":5,"kb_used_meta":28666,"kb_avail":94342104,"statfs":{"total":96636764160,"available":96606314496,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6051,"internal_metadata":29354077},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542179,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28128,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343712,"statfs":{"total":96636764160,"available":96607961088,"internally_reserved":0,"allocated":1130496,"data_stored":988208,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.45200000000000001,"15min":0.45800000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.56299999999999994,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.42299999999999999,"15min":0.42399999999999999},"min":{"1min":0.29799999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60499999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.57499999999999996}]},{"osd":1,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45500000000000002,"15min":0.46100000000000002},"min":{"1min":0.29899999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.55200000000000005,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.44,"5min":0.44800000000000001,"15min":0.45000000000000001},"min":{"1min":0.28499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.53700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":3,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83199999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.49299999999999999,"15min":0.497},"min":{"1min":0.37,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.64600000000000002,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.64900000000000002}]},{"osd":4,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.42799999999999999,"15min":0.439},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.53600000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.47599999999999998,"15min":0.48799999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.60099999999999998,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.63400000000000001}]},{"osd":5,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.54400000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.51300000000000001,"15min":0.52700000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.65900000000000003,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.47699999999999998}]},{"osd":6,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.46500000000000002,"15min":0.47499999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.59299999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.52400000000000002,"15min":0.54000000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.71699999999999997,"5min":1.054,"15min":1.054},"last":0.59099999999999997}]}]},{"osd":6,"up_from":14,"seq":60129542179,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.42699999999999999,"15min":0.42199999999999999},"min":{"1min":0.313,"5min":0.25,"15min":0.25},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.47399999999999998,"15min":0.47899999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.52400000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.46500000000000002}]},{"osd":1,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.57399999999999995,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.495,"15min":0.497},"min":{"1min":0.373,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.624,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.47699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":3,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68799999999999994,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.496},{"interface":"front","average":{"1min":0.504,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66500000000000004,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53100000000000003}]},{"osd":4,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.46200000000000002,"15min":0.46999999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57199999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.47899999999999998,"15min":0.48699999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.64900000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.33600000000000002}]},{"osd":5,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.28499999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.68799999999999994,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.313,"5min":0.308,"15min":0.308},"max":{"1min":0.65600000000000003,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.38200000000000001}]},{"osd":7,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53700000000000003,"15min":0.54100000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.51300000000000001,"15min":0.51700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64900000000000002,"5min":0.872,"15min":0.872},"last":0.45200000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.40500000000000003,"15min":0.40200000000000002},"min":{"1min":0.308,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.435,"15min":0.434},"min":{"1min":0.33600000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.64600000000000002,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.52700000000000002}]},{"osd":1,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66300000000000003,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46999999999999997,"15min":0.47299999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.58999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":3,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.51100000000000001,"15min":0.51900000000000002},"min":{"1min":0.38100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.58399999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.499,"15min":0.501},"min":{"1min":0.31900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.66900000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.503}]},{"osd":5,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.44400000000000001,"15min":0.45000000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.64700000000000002,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.45100000000000001,"15min":0.45400000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64100000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.34100000000000003}]},{"osd":6,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66500000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.61199999999999999,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.36099999999999999}]},{"osd":7,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.49399999999999999,"15min":0.49199999999999999},"min":{"1min":0.34100000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71699999999999997,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.38100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574883,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.50700000000000001,"15min":0.52900000000000003},"min":{"1min":0.219,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.127,"5min":3.2959999999999998,"15min":3.2959999999999998},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.56699999999999995,"15min":0.58499999999999996},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":1.161,"5min":3.2799999999999998,"15min":3.2799999999999998},"last":0.29199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":3,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":3.3239999999999998,"15min":3.3239999999999998},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.47799999999999998,"15min":0.47699999999999998},"min":{"1min":0.25900000000000001,"5min":0.253,"15min":0.253},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.31}]},{"osd":4,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.375,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59299999999999997,"15min":0.58899999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":0.434}]},{"osd":5,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.39300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.628,"15min":0.624},"min":{"1min":0.376,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.53800000000000003}]},{"osd":6,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.61899999999999999,"15min":0.625},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0920000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.70399999999999996,"15min":0.71999999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1779999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":0.40100000000000002}]},{"osd":7,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.667,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63800000000000001,"15min":0.64200000000000002},"min":{"1min":0.41599999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.45100000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542179,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28128,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343712,"statfs":{"total":96636764160,"available":96607961088,"internally_reserved":0,"allocated":1130496,"data_stored":988208,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.40500000000000003,"15min":0.40899999999999997},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.79800000000000004,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40100000000000002,"15min":0.40899999999999997},"min":{"1min":0.23200000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.64200000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.31900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":3,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.46999999999999997,"15min":0.48199999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.59199999999999997,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.371},{"interface":"front","average":{"1min":0.441,"5min":0.48599999999999999,"15min":0.49399999999999999},"min":{"1min":0.28799999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.35499999999999998}]},{"osd":4,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.871,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.56599999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82699999999999996,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.60299999999999998}]},{"osd":5,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54200000000000004,"15min":0.53900000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.52600000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.58599999999999997}]},{"osd":6,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.38100000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59199999999999997,"15min":0.59599999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.57399999999999995}]},{"osd":7,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57299999999999995,"15min":0.56799999999999995},"min":{"1min":0.378,"5min":0.377,"15min":0.377},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.56299999999999994,"15min":0.56899999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.70899999999999996,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.621}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T21:42:15.093 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-03-27T21:39:45.446242+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=21, tm_min=39, tm_sec=45, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=21, tm_min=42, tm_sec=10, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T21:42:15.094 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-03-27T21:39:45.446242+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=21, tm_min=39, tm_sec=45, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=21, tm_min=42, tm_sec=10, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T21:42:15.095 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-03-27T21:39:45.446242+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=21, tm_min=39, tm_sec=45, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=21, tm_min=42, tm_sec=10, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T21:42:15.095 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-03-27T21:39:45.446242+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=21, tm_min=39, tm_sec=45, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=21, tm_min=42, tm_sec=10, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T21:42:15.095 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-03-27T21:39:45.446242+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=21, tm_min=39, tm_sec=45, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=21, tm_min=42, tm_sec=10, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T21:42:15.095 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-03-27T21:39:45.446242+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=21, tm_min=39, tm_sec=45, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=21, tm_min=42, tm_sec=10, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T21:42:15.095 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-03-27T21:39:45.446242+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=21, tm_min=39, tm_sec=45, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=21, tm_min=42, tm_sec=10, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T21:42:15.096 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-03-27T21:39:45.446242+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=21, tm_min=39, tm_sec=45, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=21, tm_min=42, tm_sec=10, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T21:42:15.096 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-27T21:39:38.411432+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=21, tm_min=39, tm_sec=38, tm_wday=2, tm_yday=87, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=27, tm_hour=21, tm_min=42, tm_sec=10, tm_wday=2, tm_yday=87, tm_isdst=0) 2024-03-27T21:42:15.096 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-27T21:42:35.098 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T21:42:35.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.224+0000 7fd387077640 10 monclient: get_monmap_and_config 2024-03-27T21:42:35.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.224+0000 7fd387077640 10 monclient: build_initial_monmap 2024-03-27T21:42:35.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.224+0000 7fd387077640 10 monclient: monmap: 2024-03-27T21:42:35.210 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:35.211 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:35.211 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:35.225093+0000 2024-03-27T21:42:35.211 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:35.225093+0000 2024-03-27T21:42:35.211 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:35.211 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:35.211 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:35.211 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:35.211 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:35.212 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:35.212 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:35.212 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:35.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.224+0000 7fd387077640 1 Processor -- start 2024-03-27T21:42:35.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.224+0000 7fd387077640 1 -- start start 2024-03-27T21:42:35.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.224+0000 7fd387077640 10 monclient: init 2024-03-27T21:42:35.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd387077640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:35.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd387077640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-03-27T21:42:35.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd387077640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd380071a50 0x7fd3800a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:35.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd387077640 10 monclient(hunting): picked mon.noname-b con 0x7fd380071a50 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:35.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd387077640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd38013e810 0x7fd38013ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:35.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd387077640 10 monclient(hunting): picked mon.noname-c con 0x7fd38013e810 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:35.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd387077640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd380141630 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:35.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd387077640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38013f190 con 0x7fd380141630 2024-03-27T21:42:35.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd387077640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:35.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd387077640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd380071e30 con 0x7fd38013e810 2024-03-27T21:42:35.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd387077640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:35.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd387077640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3800a8e90 con 0x7fd380071a50 2024-03-27T21:42:35.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd387077640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:35.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd387077640 10 monclient(hunting): authenticate will time out at 996.446433s 2024-03-27T21:42:35.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd384e13640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd380071a50 0x7fd3800a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:35.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd384e13640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd380071a50 0x7fd3800a87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34194/0 (socket says 172.21.15.60:34194) 2024-03-27T21:42:35.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd384e13640 1 -- 172.21.15.60:0/4020067769 learned_addr learned my addr 172.21.15.60:0/4020067769 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:35.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd384e13640 10 monclient(hunting): get_auth_request con 0x7fd380071a50 auth_method 0 2024-03-27T21:42:35.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd384e13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:35.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd384e13640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:35.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd384e13640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:35.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd384e13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:35.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd384e13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:35.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd384e13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:35.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd37ffff640 1 --2- 172.21.15.60:0/4020067769 >> v2:172.21.15.191:3300/0 conn(0x7fd38013e810 0x7fd38013ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:35.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd384e13640 10 monclient(hunting): handle_auth_done global_id 5207 payload 274 2024-03-27T21:42:35.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd384e13640 1 -- 172.21.15.60:0/4020067769 >> v2:172.21.15.191:3300/0 conn(0x7fd38013e810 msgr2=0x7fd38013ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:35.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd384e13640 1 --2- 172.21.15.60:0/4020067769 >> v2:172.21.15.191:3300/0 conn(0x7fd38013e810 0x7fd38013ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd384e13640 1 -- 172.21.15.60:0/4020067769 >> v1:172.21.15.191:6789/0 conn(0x7fd380141630 legacy=0x7fd380141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:35.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd384e13640 10 monclient: _finish_hunting 0 2024-03-27T21:42:35.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd384e13640 1 monclient: found mon.noname-b 2024-03-27T21:42:35.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd384e13640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:35.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd384e13640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.60:3301/0 2024-03-27T21:42:35.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd384e13640 1 -- 172.21.15.60:0/4020067769 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3800aa040 con 0x7fd380071a50 2024-03-27T21:42:35.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd37ffff640 10 monclient: get_auth_request con 0x7fd38013e810 auth_method 0 2024-03-27T21:42:35.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.225+0000 7fd37ffff640 1 --2- 172.21.15.60:0/4020067769 >> v2:172.21.15.191:3300/0 conn(0x7fd38013e810 0x7fd38013ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-27T21:42:35.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.226+0000 7fd384e13640 1 --2- 172.21.15.60:0/4020067769 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd380071a50 0x7fd3800a87e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd378009e50 tx=0x7fd37802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb310ac9b4b65030 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:35.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.226+0000 7fd37f7fe640 1 -- 172.21.15.60:0/4020067769 <== mon.2 v2:172.21.15.60:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd37802ca70 con 0x7fd380071a50 2024-03-27T21:42:35.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.226+0000 7fd37f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:35.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.226+0000 7fd37f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-27T21:42:35.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.226+0000 7fd37f7fe640 10 monclient: dump: 2024-03-27T21:42:35.225 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:35.225 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:35.226 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:35.226 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:35.226 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:35.226 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:35.226 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:35.226 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:35.226 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:35.226 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:35.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.226+0000 7fd37f7fe640 10 monclient: _finish_auth 0 2024-03-27T21:42:35.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.226+0000 7fd37f7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:42:35.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.226+0000 7fd37f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:35.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.226+0000 7fd37f7fe640 1 -- 172.21.15.60:0/4020067769 <== mon.2 v2:172.21.15.60:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd37802cbd0 con 0x7fd380071a50 2024-03-27T21:42:35.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.226+0000 7fd387077640 5 monclient: authenticate success, global_id 5207 2024-03-27T21:42:35.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.226+0000 7fd387077640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:35.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.226+0000 7fd37f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:35.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.226+0000 7fd387077640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:35.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.226+0000 7fd37f7fe640 1 -- 172.21.15.60:0/4020067769 <== mon.2 v2:172.21.15.60:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3780343f0 con 0x7fd380071a50 2024-03-27T21:42:35.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.226+0000 7fd387077640 4 set_mon_vals no callback set 2024-03-27T21:42:35.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.226+0000 7fd37f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:35.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.226+0000 7fd37f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-27T21:42:35.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.226+0000 7fd37f7fe640 10 monclient: dump: 2024-03-27T21:42:35.228 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:35.228 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:35.228 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:35.229 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:35.229 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:35.229 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:35.229 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:35.229 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:35.229 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:35.229 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:35.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.227+0000 7fd387077640 10 monclient: shutdown 2024-03-27T21:42:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.227+0000 7fd387077640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.227+0000 7fd387077640 1 -- 172.21.15.60:0/4020067769 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd380071a50 msgr2=0x7fd3800a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.227+0000 7fd387077640 1 --2- 172.21.15.60:0/4020067769 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd380071a50 0x7fd3800a87e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd378009e50 tx=0x7fd37802bed0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.227+0000 7fd387077640 1 -- 172.21.15.60:0/4020067769 shutdown_connections 2024-03-27T21:42:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.227+0000 7fd387077640 1 --2- 172.21.15.60:0/4020067769 >> v2:172.21.15.191:3300/0 conn(0x7fd38013e810 0x7fd38013ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.227+0000 7fd387077640 1 --2- 172.21.15.60:0/4020067769 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd380071a50 0x7fd3800a87e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.227+0000 7fd387077640 1 -- 172.21.15.60:0/4020067769 >> 172.21.15.60:0/4020067769 conn(0x7fd3800b7300 msgr2=0x7fd3800b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.227+0000 7fd387077640 1 -- 172.21.15.60:0/4020067769 shutdown_connections 2024-03-27T21:42:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.227+0000 7fd387077640 1 -- 172.21.15.60:0/4020067769 wait complete. 2024-03-27T21:42:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.228+0000 7fd387077640 10 monclient: build_initial_monmap 2024-03-27T21:42:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.228+0000 7fd387077640 10 monclient: monmap: 2024-03-27T21:42:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:35.232 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:35.232 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:35.233 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:35.234 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:35.234 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:35.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.228+0000 7fd387077640 1 librados: starting msgr at 2024-03-27T21:42:35.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.228+0000 7fd387077640 1 librados: starting objecter 2024-03-27T21:42:35.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.228+0000 7fd387077640 1 Processor -- start 2024-03-27T21:42:35.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.228+0000 7fd387077640 1 -- start start 2024-03-27T21:42:35.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.228+0000 7fd387077640 1 librados: setting wanted keys 2024-03-27T21:42:35.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.228+0000 7fd387077640 1 librados: calling monclient init 2024-03-27T21:42:35.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.228+0000 7fd387077640 10 monclient: init 2024-03-27T21:42:35.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd387077640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:35.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd387077640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-27T21:42:35.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd387077640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd380071a50 0x7fd3800b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:35.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd387077640 10 monclient(hunting): picked mon.noname-b con 0x7fd380071a50 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:35.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd387077640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd38013e810 0x7fd38013a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:35.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd387077640 10 monclient(hunting): picked mon.noname-c con 0x7fd38013e810 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:35.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd387077640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd380141630 0x7fd3801d9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:35.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd387077640 10 monclient(hunting): picked mon.noname-a con 0x7fd380141630 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:35.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd387077640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:35.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd387077640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38013a100 con 0x7fd380141630 2024-03-27T21:42:35.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd387077640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:35.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd387077640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd380138900 con 0x7fd380071a50 2024-03-27T21:42:35.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd387077640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:35.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd387077640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd380139d90 con 0x7fd38013e810 2024-03-27T21:42:35.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd387077640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:35.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd387077640 10 monclient(hunting): authenticate will time out at 996.450247s 2024-03-27T21:42:35.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd37ffff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd38013e810 0x7fd38013a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:35.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd385614640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd380141630 0x7fd3801d9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:35.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd37ffff640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd38013e810 0x7fd38013a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3301/0 says I am v2:172.21.15.60:34196/0 (socket says 172.21.15.60:34196) 2024-03-27T21:42:35.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd37ffff640 1 -- 172.21.15.60:0/3857462239 learned_addr learned my addr 172.21.15.60:0/3857462239 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:35.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd37ffff640 10 monclient(hunting): get_auth_request con 0x7fd38013e810 auth_method 0 2024-03-27T21:42:35.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd37ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:35.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd37ffff640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:35.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd37ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:35.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd385614640 10 monclient(hunting): get_auth_request con 0x7fd380141630 auth_method 0 2024-03-27T21:42:35.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd385614640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:35.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd385614640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:35.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd385614640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:35.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd385614640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:35.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd385614640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:35.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd385614640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:35.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd37ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:35.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd37ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:35.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd37ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:35.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd384e13640 1 --2- 172.21.15.60:0/3857462239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd380071a50 0x7fd3800b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:35.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd384e13640 10 monclient(hunting): get_auth_request con 0x7fd380071a50 auth_method 0 2024-03-27T21:42:35.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd384e13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:35.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd384e13640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:35.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd384e13640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:35.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd385614640 10 monclient(hunting): handle_auth_done global_id 5556 payload 824 2024-03-27T21:42:35.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd385614640 1 -- 172.21.15.60:0/3857462239 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd38013e810 msgr2=0x7fd38013a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:35.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd385614640 1 --2- 172.21.15.60:0/3857462239 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd38013e810 0x7fd38013a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd385614640 1 -- 172.21.15.60:0/3857462239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd380071a50 msgr2=0x7fd3800b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:35.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd385614640 1 --2- 172.21.15.60:0/3857462239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd380071a50 0x7fd3800b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd385614640 10 monclient: _finish_hunting 0 2024-03-27T21:42:35.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd385614640 1 monclient: found mon.noname-a 2024-03-27T21:42:35.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd385614640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:35.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd385614640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:35.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd385614640 1 -- 172.21.15.60:0/3857462239 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd378009b30 con 0x7fd380141630 2024-03-27T21:42:35.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd37ffff640 1 --2- 172.21.15.60:0/3857462239 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd38013e810 0x7fd38013a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:35.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.229+0000 7fd384e13640 1 --2- 172.21.15.60:0/3857462239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd380071a50 0x7fd3800b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-27T21:42:35.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd385614640 1 --2- 172.21.15.60:0/3857462239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd380141630 0x7fd3801d9890 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd36800b410 tx=0x7fd36800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:35.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd37d7fa640 10 client.?.objecter ms_handle_connect 0x7fd380141630 2024-03-27T21:42:35.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd37d7fa640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:35.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd37d7fa640 1 -- 172.21.15.60:0/3857462239 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd368011830 con 0x7fd380141630 2024-03-27T21:42:35.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd37d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:35.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd37d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:35.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd37d7fa640 10 monclient: dump: 2024-03-27T21:42:35.242 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:35.242 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:35.242 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:35.242 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:35.242 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:35.242 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:35.242 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:35.242 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:35.243 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:35.243 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:35.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd37d7fa640 10 monclient: _finish_auth 0 2024-03-27T21:42:35.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd37d7fa640 10 monclient: _check_auth_tickets 2024-03-27T21:42:35.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd37d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:35.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd37d7fa640 1 -- 172.21.15.60:0/3857462239 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3680119b0 con 0x7fd380141630 2024-03-27T21:42:35.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd387077640 5 monclient: authenticate success, global_id 5556 2024-03-27T21:42:35.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd387077640 10 monclient: _renew_subs 2024-03-27T21:42:35.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd387077640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:35.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd387077640 1 -- 172.21.15.60:0/3857462239 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd380139460 con 0x7fd380141630 2024-03-27T21:42:35.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd37d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:35.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd37d7fa640 1 -- 172.21.15.60:0/3857462239 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd368011d60 con 0x7fd380141630 2024-03-27T21:42:35.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd37d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:35.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd37d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:35.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd37d7fa640 10 monclient: dump: 2024-03-27T21:42:35.244 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:35.245 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:35.245 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:35.245 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:35.245 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:35.245 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:35.245 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:35.245 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:35.245 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:35.245 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:35.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd37effd640 4 set_mon_vals no callback set 2024-03-27T21:42:35.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd387077640 10 client.5556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:35.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd387077640 10 monclient: _renew_subs 2024-03-27T21:42:35.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd387077640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:35.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd387077640 1 -- 172.21.15.60:0/3857462239 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3801d9fb0 con 0x7fd380141630 2024-03-27T21:42:35.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.230+0000 7fd387077640 1 librados: init done 2024-03-27T21:42:35.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.231+0000 7fd387077640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:35.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.231+0000 7fd387077640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:35.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.231+0000 7fd387077640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:35.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.231+0000 7fd387077640 1 -- 172.21.15.60:0/3857462239 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd38013a270 con 0x7fd380141630 2024-03-27T21:42:35.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.231+0000 7fd37d7fa640 1 -- 172.21.15.60:0/3857462239 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd36801a400 con 0x7fd380141630 2024-03-27T21:42:35.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.232+0000 7fd37d7fa640 1 --2- 172.21.15.60:0/3857462239 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd36006c590 0x7fd36006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:35.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.232+0000 7fd37d7fa640 1 -- 172.21.15.60:0/3857462239 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fd368088ff0 con 0x7fd380141630 2024-03-27T21:42:35.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.232+0000 7fd37d7fa640 10 client.5556.objecter ms_dispatch 0x7fd380000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:35.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.232+0000 7fd37d7fa640 3 client.5556.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:35.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.232+0000 7fd37d7fa640 3 client.5556.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:35.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.232+0000 7fd384e13640 1 --2- 172.21.15.60:0/3857462239 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd36006c590 0x7fd36006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:35.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.232+0000 7fd384e13640 10 monclient: get_auth_request con 0x7fd36006c590 auth_method 0 2024-03-27T21:42:35.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.232+0000 7fd37d7fa640 20 client.5556.objecter dump_active .. 0 homeless 2024-03-27T21:42:35.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.232+0000 7fd384e13640 1 --2- 172.21.15.60:0/3857462239 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd36006c590 0x7fd36006ea50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd378035000 tx=0x7fd378030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:35.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.232+0000 7fd37d7fa640 10 client.5556.objecter ms_handle_connect 0x7fd36006c590 2024-03-27T21:42:35.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.236+0000 7fd37d7fa640 1 -- 172.21.15.60:0/3857462239 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd368054430 con 0x7fd380141630 2024-03-27T21:42:35.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.236+0000 7fd37d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:35.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.236+0000 7fd37d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:35.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.414+0000 7fd387077640 1 -- 172.21.15.60:0/3857462239 --> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd3801421b0 con 0x7fd36006c590 2024-03-27T21:42:35.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.417+0000 7fd37d7fa640 1 -- 172.21.15.60:0/3857462239 <== mgr.4104 v2:172.21.15.60:6832/34387 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+70579 (secure 0 0 0) 0x7fd378044090 con 0x7fd36006c590 2024-03-27T21:42:35.403 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-27T21:42:35.404 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-03-27T21:42:35.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.419+0000 7fd387077640 10 librados: watch_flush enter 2024-03-27T21:42:35.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.419+0000 7fd387077640 10 librados: watch_flush exit 2024-03-27T21:42:35.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.419+0000 7fd387077640 20 client.5556.objecter shutdown clearing up homeless session... 2024-03-27T21:42:35.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.419+0000 7fd387077640 10 client.5556.objecter successfully canceled tick 2024-03-27T21:42:35.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.419+0000 7fd387077640 1 -- 172.21.15.60:0/3857462239 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd36006c590 msgr2=0x7fd36006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:35.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.419+0000 7fd387077640 1 --2- 172.21.15.60:0/3857462239 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd36006c590 0x7fd36006ea50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd378035000 tx=0x7fd378030000 comp rx=0 tx=0).stop 2024-03-27T21:42:35.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.419+0000 7fd387077640 10 monclient: shutdown 2024-03-27T21:42:35.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.419+0000 7fd387077640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:35.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.419+0000 7fd387077640 1 -- 172.21.15.60:0/3857462239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd380141630 msgr2=0x7fd3801d9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:35.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.419+0000 7fd387077640 1 --2- 172.21.15.60:0/3857462239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd380141630 0x7fd3801d9890 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd36800b410 tx=0x7fd36800b8d0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.419+0000 7fd387077640 1 -- 172.21.15.60:0/3857462239 shutdown_connections 2024-03-27T21:42:35.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.419+0000 7fd387077640 1 --2- 172.21.15.60:0/3857462239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd380141630 0x7fd3801d9890 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.419+0000 7fd387077640 1 --2- 172.21.15.60:0/3857462239 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7fd38013e810 0x7fd38013a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.419+0000 7fd387077640 1 --2- 172.21.15.60:0/3857462239 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7fd36006c590 0x7fd36006ea50 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.419+0000 7fd387077640 1 --2- 172.21.15.60:0/3857462239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd380071a50 0x7fd3800b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.419+0000 7fd387077640 1 -- 172.21.15.60:0/3857462239 >> 172.21.15.60:0/3857462239 conn(0x7fd3800b7300 msgr2=0x7fd380068be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:35.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.419+0000 7fd387077640 1 -- 172.21.15.60:0/3857462239 shutdown_connections 2024-03-27T21:42:35.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.419+0000 7fd387077640 1 -- 172.21.15.60:0/3857462239 wait complete. 2024-03-27T21:42:35.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.419+0000 7fd387077640 1 librados: shutdown 2024-03-27T21:42:35.416 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-03-27T21:42:33.417537+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":230416,"kb_used_data":8520,"kb_used_omap":16,"kb_used_meta":221807,"kb_avail":754744304,"statfs":{"total":773094113280,"available":772858167296,"internally_reserved":0,"allocated":8724480,"data_stored":4732096,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":17199,"internal_metadata":227130577},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002378"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":79,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-27T21:42:15.582304+0000","last_change":"2024-03-27T21:42:15.582421+0000","last_active":"2024-03-27T21:42:15.582304+0000","last_peered":"2024-03-27T21:42:15.582304+0000","last_clean":"2024-03-27T21:42:15.582304+0000","last_became_active":"2024-03-27T21:42:01.265033+0000","last_became_peered":"2024-03-27T21:42:01.265033+0000","last_unstale":"2024-03-27T21:42:15.582304+0000","last_undegraded":"2024-03-27T21:42:15.582304+0000","last_fullsized":"2024-03-27T21:42:15.582304+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:42:15.582276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:42:15.582276+0000","last_clean_scrub_stamp":"2024-03-27T21:42:15.582276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T02:51:12.019441+0000","scrub_duration":0.0012955760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-27T21:42:15.269210+0000","last_change":"2024-03-27T21:42:15.269305+0000","last_active":"2024-03-27T21:42:15.269210+0000","last_peered":"2024-03-27T21:42:15.269210+0000","last_clean":"2024-03-27T21:42:15.269210+0000","last_became_active":"2024-03-27T21:41:49.474054+0000","last_became_peered":"2024-03-27T21:41:49.474054+0000","last_unstale":"2024-03-27T21:42:15.269210+0000","last_undegraded":"2024-03-27T21:42:15.269210+0000","last_fullsized":"2024-03-27T21:42:15.269210+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:42:15.269192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:42:15.269192+0000","last_clean_scrub_stamp":"2024-03-27T21:42:15.269192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T03:57:00.768081+0000","scrub_duration":0.0017866959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":93,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-27T21:42:21.372003+0000","last_change":"2024-03-27T21:42:21.372120+0000","last_active":"2024-03-27T21:42:21.372003+0000","last_peered":"2024-03-27T21:42:21.372003+0000","last_clean":"2024-03-27T21:42:21.372003+0000","last_became_active":"2024-03-27T21:39:46.679561+0000","last_became_peered":"2024-03-27T21:39:46.679561+0000","last_unstale":"2024-03-27T21:42:21.372003+0000","last_undegraded":"2024-03-27T21:42:21.372003+0000","last_fullsized":"2024-03-27T21:42:21.372003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:42:21.371980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:42:21.371980+0000","last_clean_scrub_stamp":"2024-03-27T21:42:21.371980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:48:51.592349+0000","scrub_duration":0.0017971759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-27T21:42:20.370520+0000","last_change":"2024-03-27T21:42:20.370622+0000","last_active":"2024-03-27T21:42:20.370520+0000","last_peered":"2024-03-27T21:42:20.370520+0000","last_clean":"2024-03-27T21:42:20.370520+0000","last_became_active":"2024-03-27T21:41:49.473822+0000","last_became_peered":"2024-03-27T21:41:49.473822+0000","last_unstale":"2024-03-27T21:42:20.370520+0000","last_undegraded":"2024-03-27T21:42:20.370520+0000","last_fullsized":"2024-03-27T21:42:20.370520+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:42:20.370499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:42:20.370499+0000","last_clean_scrub_stamp":"2024-03-27T21:42:20.370499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T01:44:24.214118+0000","scrub_duration":0.0014428290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":93,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-27T21:42:14.728321+0000","last_change":"2024-03-27T21:42:14.728442+0000","last_active":"2024-03-27T21:42:14.728321+0000","last_peered":"2024-03-27T21:42:14.728321+0000","last_clean":"2024-03-27T21:42:14.728321+0000","last_became_active":"2024-03-27T21:41:59.700966+0000","last_became_peered":"2024-03-27T21:41:59.700966+0000","last_unstale":"2024-03-27T21:42:14.728321+0000","last_undegraded":"2024-03-27T21:42:14.728321+0000","last_fullsized":"2024-03-27T21:42:14.728321+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:42:14.728289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:42:14.728289+0000","last_clean_scrub_stamp":"2024-03-27T21:42:14.728289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:18:15.675747+0000","scrub_duration":0.0014361319999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-27T21:42:14.709137+0000","last_change":"2024-03-27T21:42:14.709218+0000","last_active":"2024-03-27T21:42:14.709137+0000","last_peered":"2024-03-27T21:42:14.709137+0000","last_clean":"2024-03-27T21:42:14.709137+0000","last_became_active":"2024-03-27T21:41:30.079829+0000","last_became_peered":"2024-03-27T21:41:30.079829+0000","last_unstale":"2024-03-27T21:42:14.709137+0000","last_undegraded":"2024-03-27T21:42:14.709137+0000","last_fullsized":"2024-03-27T21:42:14.709137+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:42:14.709118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:42:14.709118+0000","last_clean_scrub_stamp":"2024-03-27T21:42:14.709118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T22:13:10.140053+0000","scrub_duration":0.0013436329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-27T21:42:18.459093+0000","last_change":"2024-03-27T21:42:18.459203+0000","last_active":"2024-03-27T21:42:18.459093+0000","last_peered":"2024-03-27T21:42:18.459093+0000","last_clean":"2024-03-27T21:42:18.459093+0000","last_became_active":"2024-03-27T21:41:49.197154+0000","last_became_peered":"2024-03-27T21:41:49.197154+0000","last_unstale":"2024-03-27T21:42:18.459093+0000","last_undegraded":"2024-03-27T21:42:18.459093+0000","last_fullsized":"2024-03-27T21:42:18.459093+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:42:18.459072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:42:18.459072+0000","last_clean_scrub_stamp":"2024-03-27T21:42:18.459072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-28T22:46:17.307252+0000","scrub_duration":0.002168784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":95,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-27T21:42:21.819942+0000","last_change":"2024-03-27T21:42:21.820063+0000","last_active":"2024-03-27T21:42:21.819942+0000","last_peered":"2024-03-27T21:42:21.819942+0000","last_clean":"2024-03-27T21:42:21.819942+0000","last_became_active":"2024-03-27T21:41:59.246895+0000","last_became_peered":"2024-03-27T21:41:59.246895+0000","last_unstale":"2024-03-27T21:42:21.819942+0000","last_undegraded":"2024-03-27T21:42:21.819942+0000","last_fullsized":"2024-03-27T21:42:21.819942+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T21:42:21.819920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T21:42:21.819920+0000","last_clean_scrub_stamp":"2024-03-27T21:42:21.819920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:08:36.968958+0000","scrub_duration":0.00187711,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"92'119","reported_seq":248,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-27T21:42:16.524178+0000","last_change":"2024-03-27T21:42:16.524325+0000","last_active":"2024-03-27T21:42:16.524178+0000","last_peered":"2024-03-27T21:42:16.524178+0000","last_clean":"2024-03-27T21:42:16.524178+0000","last_became_active":"2024-03-27T21:39:39.865948+0000","last_became_peered":"2024-03-27T21:39:39.865948+0000","last_unstale":"2024-03-27T21:42:16.524178+0000","last_undegraded":"2024-03-27T21:42:16.524178+0000","last_fullsized":"2024-03-27T21:42:16.524178+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'119","last_scrub_stamp":"2024-03-27T21:42:16.524152+0000","last_deep_scrub":"92'119","last_deep_scrub_stamp":"2024-03-27T21:42:16.524152+0000","last_clean_scrub_stamp":"2024-03-27T21:42:16.524152+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:22:27.245636+0000","scrub_duration":0.0052898750000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11016,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":16,"seq":68719476776,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343772,"statfs":{"total":96636764160,"available":96608022528,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.41599999999999998,"15min":0.41199999999999998},"min":{"1min":0.34399999999999997,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.621,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.47499999999999998,"15min":0.47599999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.66900000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.54400000000000004}]},{"osd":1,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45900000000000002,"15min":0.46500000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.53600000000000003,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.35899999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.56599999999999995,"5min":0.745,"15min":0.745},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":3,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.50600000000000001,"15min":0.51400000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.56899999999999995,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.501},{"interface":"front","average":{"1min":0.495,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.35999999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68400000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.49399999999999999}]},{"osd":4,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.432,"15min":0.439},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.56399999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.46800000000000003,"15min":0.47399999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.42699999999999999}]},{"osd":6,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.45800000000000002,"15min":0.46300000000000002},"min":{"1min":0.318,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58699999999999997,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.29299999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64300000000000002,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.51200000000000001}]},{"osd":7,"last update":"Wed Mar 27 21:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.52400000000000002,"15min":0.53200000000000003},"min":{"1min":0.378,"5min":0.318,"15min":0.318},"max":{"1min":0.61299999999999999,"5min":0.745,"15min":0.745},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.50800000000000001,"15min":0.51400000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65000000000000002,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.45800000000000002}]}]},{"osd":3,"up_from":16,"seq":68719476775,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31848905},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.49299999999999999,"15min":0.504},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.91700000000000004,"5min":1.248,"15min":1.248},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.44900000000000001,"15min":0.45600000000000002},"min":{"1min":0.22800000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.90600000000000003,"5min":1.407,"15min":1.407},"last":0.35199999999999998}]},{"osd":1,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48199999999999998,"15min":0.47999999999999998},"min":{"1min":0.25600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.309},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.26800000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.96899999999999997,"5min":1.347,"15min":1.347},"last":0.44}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":4,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.61399999999999999,"15min":0.626},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83299999999999996,"5min":1.5,"15min":1.5},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.40000000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72799999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.65500000000000003}]},{"osd":5,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56000000000000005,"15min":0.56599999999999995},"min":{"1min":0.34999999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71099999999999997,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.63300000000000001,"15min":0.64700000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.96799999999999997,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.75}]},{"osd":6,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.63100000000000001,"15min":0.63900000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88200000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.40999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76700000000000002,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.88100000000000001}]},{"osd":7,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.64200000000000002,"15min":0.65600000000000003},"min":{"1min":0.36899999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.94399999999999995,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61599999999999999,"15min":0.622},"min":{"1min":0.36699999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85799999999999998,"5min":1.454,"15min":1.454},"last":0.80000000000000004}]}]},{"osd":2,"up_from":62,"seq":266287972367,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29732,"kb_used_data":1052,"kb_used_omap":5,"kb_used_meta":28666,"kb_avail":94342108,"statfs":{"total":96636764160,"available":96606318592,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6051,"internal_metadata":29354077},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:42:32 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.748,"5min":1.748,"15min":1.748},"last":0.45900000000000002}]},{"osd":1,"last update":"Wed Mar 27 21:42:32 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.436}]},{"osd":3,"last update":"Wed Mar 27 21:42:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.48299999999999998}]},{"osd":4,"last update":"Wed Mar 27 21:42:32 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.624}]},{"osd":5,"last update":"Wed Mar 27 21:42:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.76100000000000001}]},{"osd":6,"last update":"Wed Mar 27 21:42:32 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.66800000000000004}]},{"osd":7,"last update":"Wed Mar 27 21:42:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.73599999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542183,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343720,"statfs":{"total":96636764160,"available":96607969280,"internally_reserved":0,"allocated":1130496,"data_stored":988208,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.45200000000000001,"15min":0.45800000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.56299999999999994,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.42299999999999999,"15min":0.42399999999999999},"min":{"1min":0.29799999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60499999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.64600000000000002}]},{"osd":1,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45500000000000002,"15min":0.46100000000000002},"min":{"1min":0.29899999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.55200000000000005,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.44,"5min":0.44800000000000001,"15min":0.45000000000000001},"min":{"1min":0.28499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.53100000000000003}]},{"osd":2,"last update":"Wed Mar 27 21:42:31 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.54200000000000004}]},{"osd":3,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83199999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.49299999999999999,"15min":0.497},"min":{"1min":0.37,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.64600000000000002,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.67100000000000004}]},{"osd":4,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.42799999999999999,"15min":0.439},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.53600000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.505},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.47599999999999998,"15min":0.48799999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.60099999999999998,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.48599999999999999}]},{"osd":5,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.54400000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.433,"5min":0.51300000000000001,"15min":0.52700000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.65900000000000003,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.56200000000000006}]},{"osd":6,"last update":"Wed Mar 27 21:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.46500000000000002,"15min":0.47499999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.59299999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.622},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.52400000000000002,"15min":0.54000000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.71699999999999997,"5min":1.054,"15min":1.054},"last":0.60699999999999998}]}]},{"osd":6,"up_from":14,"seq":60129542183,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343772,"statfs":{"total":96636764160,"available":96608022528,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.42699999999999999,"15min":0.42199999999999999},"min":{"1min":0.313,"5min":0.25,"15min":0.25},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.47399999999999998,"15min":0.47899999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.52400000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.64700000000000002}]},{"osd":1,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.57399999999999995,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.495,"15min":0.497},"min":{"1min":0.373,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.624,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.51100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":3,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68799999999999994,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66500000000000004,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.56699999999999995}]},{"osd":4,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.46200000000000002,"15min":0.46999999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57199999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.377},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.47899999999999998,"15min":0.48699999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.64900000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.36499999999999999}]},{"osd":5,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.28499999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.68799999999999994,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.313,"5min":0.308,"15min":0.308},"max":{"1min":0.65600000000000003,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.42299999999999999}]},{"osd":7,"last update":"Wed Mar 27 21:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53700000000000003,"15min":0.54100000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.51300000000000001,"15min":0.51700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64900000000000002,"5min":0.872,"15min":0.872},"last":0.44500000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542183,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.40500000000000003,"15min":0.40200000000000002},"min":{"1min":0.308,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.443,"5min":0.435,"15min":0.434},"min":{"1min":0.33600000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.64600000000000002,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.59199999999999997}]},{"osd":1,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66300000000000003,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46999999999999997,"15min":0.47299999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.55000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":3,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.51100000000000001,"15min":0.51900000000000002},"min":{"1min":0.38100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.58399999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.499,"15min":0.501},"min":{"1min":0.31900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.66900000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.501}]},{"osd":5,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.44400000000000001,"15min":0.45000000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.64700000000000002,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.433,"5min":0.45100000000000001,"15min":0.45400000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64100000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.35499999999999998}]},{"osd":6,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66500000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.61199999999999999,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.436}]},{"osd":7,"last update":"Wed Mar 27 21:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.49399999999999999,"15min":0.49199999999999999},"min":{"1min":0.34100000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71699999999999997,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.46000000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343772,"statfs":{"total":96636764160,"available":96608022528,"internally_reserved":0,"allocated":1077248,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.50700000000000001,"15min":0.52900000000000003},"min":{"1min":0.219,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.127,"5min":3.2959999999999998,"15min":3.2959999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.56699999999999995,"15min":0.58499999999999996},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":1.161,"5min":3.2799999999999998,"15min":3.2799999999999998},"last":0.51300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":3,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":3.3239999999999998,"15min":3.3239999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.47799999999999998,"15min":0.47699999999999998},"min":{"1min":0.25900000000000001,"5min":0.253,"15min":0.253},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.44600000000000001}]},{"osd":4,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.375,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59299999999999997,"15min":0.58899999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":0.65000000000000002}]},{"osd":5,"last update":"Wed Mar 27 21:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.39300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.628,"15min":0.624},"min":{"1min":0.376,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.80200000000000005}]},{"osd":6,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.61899999999999999,"15min":0.625},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0920000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.70399999999999996,"15min":0.71999999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1779999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":0.73999999999999999}]},{"osd":7,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.667,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63800000000000001,"15min":0.64200000000000002},"min":{"1min":0.41599999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.66500000000000004}]}]},{"osd":0,"up_from":14,"seq":60129542183,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343720,"statfs":{"total":96636764160,"available":96607969280,"internally_reserved":0,"allocated":1130496,"data_stored":988208,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.40500000000000003,"15min":0.40899999999999997},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.79800000000000004,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40100000000000002,"15min":0.40899999999999997},"min":{"1min":0.23200000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.64200000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.49199999999999999}]},{"osd":2,"last update":"Wed Mar 27 21:42:31 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.65100000000000002}]},{"osd":3,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.46999999999999997,"15min":0.48199999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.59199999999999997,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.441,"5min":0.48599999999999999,"15min":0.49399999999999999},"min":{"1min":0.28799999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.58999999999999997}]},{"osd":4,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.871,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.56599999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82699999999999996,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.63700000000000001}]},{"osd":5,"last update":"Wed Mar 27 21:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54200000000000004,"15min":0.53900000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.52600000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.81799999999999995}]},{"osd":6,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.38100000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59199999999999997,"15min":0.59599999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.76400000000000001}]},{"osd":7,"last update":"Wed Mar 27 21:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57299999999999995,"15min":0.56799999999999995},"min":{"1min":0.378,"5min":0.377,"15min":0.377},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.748},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.56299999999999994,"15min":0.56899999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.70899999999999996,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.72699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5508,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-27T21:42:35.419 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-27T21:42:35.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.532+0000 7f1f1d209640 10 monclient: get_monmap_and_config 2024-03-27T21:42:35.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.532+0000 7f1f1d209640 10 monclient: build_initial_monmap 2024-03-27T21:42:35.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.533+0000 7f1f1d209640 10 monclient: monmap: 2024-03-27T21:42:35.519 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:35.520 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:35.520 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:42:35.533976+0000 2024-03-27T21:42:35.520 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:42:35.533976+0000 2024-03-27T21:42:35.520 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:35.520 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:35.520 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a-legacy 2024-03-27T21:42:35.520 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c-legacy 2024-03-27T21:42:35.520 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-b 2024-03-27T21:42:35.521 INFO:teuthology.orchestra.run.smithi060.stderr:3: v2:172.21.15.60:3300/0 mon.noname-a 2024-03-27T21:42:35.521 INFO:teuthology.orchestra.run.smithi060.stderr:4: v2:172.21.15.191:3300/0 mon.noname-c 2024-03-27T21:42:35.521 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:35.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.533+0000 7f1f1d209640 1 Processor -- start 2024-03-27T21:42:35.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.533+0000 7f1f1d209640 1 -- start start 2024-03-27T21:42:35.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.533+0000 7f1f1d209640 10 monclient: init 2024-03-27T21:42:35.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f1d209640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:35.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f1d209640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-03-27T21:42:35.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f1d209640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1f180b3fc0 0x7f1f18144950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:35.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f1d209640 10 monclient(hunting): picked mon.noname-c con 0x7f1f180b3fc0 addr v2:172.21.15.191:3300/0 2024-03-27T21:42:35.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f1d209640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1f180a9280 addr v1:172.21.15.191:6789/0 2024-03-27T21:42:35.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f1d209640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f180a8440 addr v1:172.21.15.60:6789/0 2024-03-27T21:42:35.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f1d209640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f1813f080 con 0x7f1f180a8440 2024-03-27T21:42:35.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f1d209640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f180b15d0 con 0x7f1f180a9280 2024-03-27T21:42:35.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f1d209640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:35.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f1d209640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f180a9830 con 0x7f1f180b3fc0 2024-03-27T21:42:35.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f1d209640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:35.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f1d209640 10 monclient(hunting): authenticate will time out at 996.755083s 2024-03-27T21:42:35.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f1759e640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1f180a8440 0x7f1f180afee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34844/0 (socket says 172.21.15.60:34844) 2024-03-27T21:42:35.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f1759e640 1 -- 172.21.15.60:0/1186230061 learned_addr learned my addr 172.21.15.60:0/1186230061 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-27T21:42:35.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f16d9d640 1 --2- 172.21.15.60:0/1186230061 >> v2:172.21.15.191:3300/0 conn(0x7f1f180b3fc0 0x7f1f18144950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:35.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 1 -- 172.21.15.60:0/1186230061 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371706301 0 0) 0x7f1f1813f080 con 0x7f1f180a8440 2024-03-27T21:42:35.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 10 monclient(hunting): _negotiate 2024-03-27T21:42:35.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:35.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:35.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 10 monclient(hunting): my global_id is 5559 2024-03-27T21:42:35.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 1 -- 172.21.15.60:0/1186230061 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f00003520 con 0x7f1f180a8440 2024-03-27T21:42:35.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f16d9d640 10 monclient(hunting): get_auth_request con 0x7f1f180b3fc0 auth_method 0 2024-03-27T21:42:35.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f16d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:35.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f16d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:35.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f16d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:35.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f16d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:35.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f16d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:35.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f16d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:35.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 1 -- 172.21.15.60:0/1186230061 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3784253152 0 0) 0x7f1f00003520 con 0x7f1f180a8440 2024-03-27T21:42:35.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 1 -- 172.21.15.60:0/1186230061 >> v2:172.21.15.191:3300/0 conn(0x7f1f180b3fc0 msgr2=0x7f1f18144950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:35.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 1 --2- 172.21.15.60:0/1186230061 >> v2:172.21.15.191:3300/0 conn(0x7f1f180b3fc0 0x7f1f18144950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 1 -- 172.21.15.60:0/1186230061 >> v1:172.21.15.191:6789/0 conn(0x7f1f180a9280 legacy=0x7f1f180b0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:35.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 10 monclient: _finish_hunting 0 2024-03-27T21:42:35.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 1 monclient: found mon.noname-a-legacy 2024-03-27T21:42:35.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:35.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.60:6789/0 2024-03-27T21:42:35.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 1 -- 172.21.15.60:0/1186230061 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f180b36c0 con 0x7f1f180a8440 2024-03-27T21:42:35.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 10 monclient: _finish_auth 0 2024-03-27T21:42:35.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 10 monclient: _check_auth_tickets 2024-03-27T21:42:35.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:35.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 1 -- 172.21.15.60:0/1186230061 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2933500084 0 0) 0x7f1f08001670 con 0x7f1f180a8440 2024-03-27T21:42:35.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f1d209640 5 monclient: authenticate success, global_id 5559 2024-03-27T21:42:35.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f1d209640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:35.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:35.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-03-27T21:42:35.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 10 monclient: dump: 2024-03-27T21:42:35.537 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:35.538 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:35.538 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:35.538 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:35.538 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:35.538 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:35.538 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:35.538 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:35.538 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:35.539 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:35.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] but i'm connected to v1:172.21.15.60:6789/0, reconnecting 2024-03-27T21:42:35.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:35.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 1 -- 172.21.15.60:0/1186230061 >> v1:172.21.15.60:6789/0 conn(0x7f1f180a8440 legacy=0x7f1f180afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:35.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-27T21:42:35.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 1 --2- 172.21.15.60:0/1186230061 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1f00003970 0x7f1f00003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:35.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 10 monclient(hunting): picked mon.c con 0x7f1f00003970 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:35.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 1 --2- 172.21.15.60:0/1186230061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f000047c0 0x7f1f00004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:35.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 10 monclient(hunting): picked mon.b con 0x7f1f000047c0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:35.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 1 --2- 172.21.15.60:0/1186230061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f000052a0 0x7f1f0000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:35.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 10 monclient(hunting): picked mon.a con 0x7f1f000052a0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:35.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:35.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 1 -- 172.21.15.60:0/1186230061 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f180a9830 con 0x7f1f000052a0 2024-03-27T21:42:35.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:35.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 1 -- 172.21.15.60:0/1186230061 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0000c170 con 0x7f1f000047c0 2024-03-27T21:42:35.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:35.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 1 -- 172.21.15.60:0/1186230061 --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0000c330 con 0x7f1f00003970 2024-03-27T21:42:35.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:35.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f15d9b640 10 monclient(hunting): _finish_auth 0 2024-03-27T21:42:35.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.534+0000 7f1f1d209640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:35.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 1 --2- 172.21.15.60:0/1186230061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f000052a0 0x7f1f0000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:35.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 10 monclient(hunting): get_auth_request con 0x7f1f000052a0 auth_method 0 2024-03-27T21:42:35.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:35.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:35.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:35.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f16d9d640 1 --2- 172.21.15.60:0/1186230061 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1f00003970 0x7f1f00003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:35.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1659c640 1 --2- 172.21.15.60:0/1186230061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f000047c0 0x7f1f00004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:35.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f16d9d640 10 monclient(hunting): get_auth_request con 0x7f1f00003970 auth_method 0 2024-03-27T21:42:35.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f16d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:35.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f16d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:35.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f16d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:35.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1659c640 10 monclient(hunting): get_auth_request con 0x7f1f000047c0 auth_method 0 2024-03-27T21:42:35.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:35.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1659c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:35.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-27T21:42:35.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:35.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:35.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:42:35.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f16d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:35.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f16d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:35.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f16d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:42:35.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:35.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:35.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-27T21:42:35.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 10 monclient(hunting): handle_auth_done global_id 5559 payload 293 2024-03-27T21:42:35.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 1 -- 172.21.15.60:0/1186230061 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1f00003970 msgr2=0x7f1f00003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:35.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 1 --2- 172.21.15.60:0/1186230061 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1f00003970 0x7f1f00003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 1 -- 172.21.15.60:0/1186230061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f000047c0 msgr2=0x7f1f00004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-27T21:42:35.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 1 --2- 172.21.15.60:0/1186230061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f000047c0 0x7f1f00004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 10 monclient: _finish_hunting 0 2024-03-27T21:42:35.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 1 monclient: found mon.a 2024-03-27T21:42:35.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:35.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:35.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 1 -- 172.21.15.60:0/1186230061 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f00003520 con 0x7f1f000052a0 2024-03-27T21:42:35.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 10 monclient: _finish_auth 0 2024-03-27T21:42:35.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 10 monclient: _check_auth_tickets 2024-03-27T21:42:35.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:35.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.535+0000 7f1f1759e640 1 -- 172.21.15.60:0/1186230061 reap_dead start 2024-03-27T21:42:35.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f1759e640 1 --2- 172.21.15.60:0/1186230061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f000052a0 0x7f1f0000b980 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1f0800c210 tx=0x7f1f000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91518ad2bb9e7a99 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:35.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f15d9b640 1 -- 172.21.15.60:0/1186230061 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f08005ad0 con 0x7f1f000052a0 2024-03-27T21:42:35.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f15d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:35.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f15d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:35.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f15d9b640 10 monclient: dump: 2024-03-27T21:42:35.546 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:35.546 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:35.546 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:35.546 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:35.547 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:35.547 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:35.547 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:35.547 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:35.547 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:35.547 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:35.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f15d9b640 1 -- 172.21.15.60:0/1186230061 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f08005c30 con 0x7f1f000052a0 2024-03-27T21:42:35.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f1d209640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-27T21:42:35.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f15d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:35.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f1d209640 10 monclient: get_monmap_and_config success 2024-03-27T21:42:35.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f15d9b640 1 -- 172.21.15.60:0/1186230061 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0802c890 con 0x7f1f000052a0 2024-03-27T21:42:35.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f15d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:35.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f1d209640 4 set_mon_vals no callback set 2024-03-27T21:42:35.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f15d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:35.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f15d9b640 10 monclient: dump: 2024-03-27T21:42:35.548 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:35.548 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:35.549 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:35.549 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:35.549 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:35.549 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:35.549 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:35.549 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:35.549 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:35.549 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:35.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f1d209640 10 monclient: shutdown 2024-03-27T21:42:35.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f1d209640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:35.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f1d209640 1 -- 172.21.15.60:0/1186230061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f000052a0 msgr2=0x7f1f0000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:35.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f1d209640 1 --2- 172.21.15.60:0/1186230061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f000052a0 0x7f1f0000b980 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1f0800c210 tx=0x7f1f000057f0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f1d209640 1 -- 172.21.15.60:0/1186230061 shutdown_connections 2024-03-27T21:42:35.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f1d209640 1 --2- 172.21.15.60:0/1186230061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f000052a0 0x7f1f0000b980 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f1d209640 1 -- 172.21.15.60:0/1186230061 >> 172.21.15.60:0/1186230061 conn(0x7f1f180608e0 msgr2=0x7f1f18060cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:35.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f1d209640 1 -- 172.21.15.60:0/1186230061 shutdown_connections 2024-03-27T21:42:35.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.536+0000 7f1f1d209640 1 -- 172.21.15.60:0/1186230061 wait complete. 2024-03-27T21:42:35.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.537+0000 7f1f1d209640 10 monclient: build_initial_monmap 2024-03-27T21:42:35.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.537+0000 7f1f1d209640 10 monclient: monmap: 2024-03-27T21:42:35.551 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-27T21:42:35.551 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-27T21:42:35.551 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-27T21:42:35.551 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-27T21:42:35.551 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-27T21:42:35.551 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-27T21:42:35.551 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.noname-a 2024-03-27T21:42:35.552 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2024-03-27T21:42:35.552 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.noname-c 2024-03-27T21:42:35.552 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:35.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.537+0000 7f1f1d209640 1 librados: starting msgr at 2024-03-27T21:42:35.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.537+0000 7f1f1d209640 1 librados: starting objecter 2024-03-27T21:42:35.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1d209640 1 Processor -- start 2024-03-27T21:42:35.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1d209640 1 -- start start 2024-03-27T21:42:35.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1d209640 1 librados: setting wanted keys 2024-03-27T21:42:35.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1d209640 1 librados: calling monclient init 2024-03-27T21:42:35.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1d209640 10 monclient: init 2024-03-27T21:42:35.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1d209640 10 monclient: _reopen_session rank -1 2024-03-27T21:42:35.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1d209640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-27T21:42:35.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1d209640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f000052a0 0x7f1f180ae070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:35.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1d209640 10 monclient(hunting): picked mon.noname-a con 0x7f1f000052a0 addr [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-03-27T21:42:35.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1d209640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f180aeb30 0x7f1f1813be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:35.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1d209640 10 monclient(hunting): picked mon.noname-b con 0x7f1f180aeb30 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-27T21:42:35.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1d209640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1f1813c340 0x7f1f181d6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:35.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1d209640 10 monclient(hunting): picked mon.noname-c con 0x7f1f1813c340 addr [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] 2024-03-27T21:42:35.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1d209640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:35.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1d209640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f180af3c0 con 0x7f1f000052a0 2024-03-27T21:42:35.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1d209640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:35.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1d209640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f180aba90 con 0x7f1f180aeb30 2024-03-27T21:42:35.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1d209640 10 monclient(hunting): start opening mon connection 2024-03-27T21:42:35.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1d209640 1 -- --> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1813c7c0 con 0x7f1f1813c340 2024-03-27T21:42:35.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1d209640 10 monclient(hunting): _renew_subs 2024-03-27T21:42:35.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1d209640 10 monclient(hunting): authenticate will time out at 996.759518s 2024-03-27T21:42:35.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f16d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f000052a0 0x7f1f180ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:35.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1759e640 1 --2- >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1f1813c340 0x7f1f181d6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:35.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f16d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f000052a0 0x7f1f180ae070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35600/0 (socket says 172.21.15.60:35600) 2024-03-27T21:42:35.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f16d9d640 1 -- 172.21.15.60:0/3996175902 learned_addr learned my addr 172.21.15.60:0/3996175902 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-27T21:42:35.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f16d9d640 10 monclient(hunting): get_auth_request con 0x7f1f000052a0 auth_method 0 2024-03-27T21:42:35.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f16d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:35.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f16d9d640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:35.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f16d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:35.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1759e640 10 monclient(hunting): get_auth_request con 0x7f1f1813c340 auth_method 0 2024-03-27T21:42:35.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:35.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1759e640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:35.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1759e640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:35.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f16d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:35.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f16d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:35.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f16d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:35.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-27T21:42:35.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-27T21:42:35.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-27T21:42:35.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1659c640 1 --2- 172.21.15.60:0/3996175902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f180aeb30 0x7f1f1813be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:35.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1659c640 10 monclient(hunting): get_auth_request con 0x7f1f180aeb30 auth_method 0 2024-03-27T21:42:35.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-27T21:42:35.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1659c640 10 monclient(hunting): _init_auth method 2 2024-03-27T21:42:35.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f1659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-27T21:42:35.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f16d9d640 10 monclient(hunting): handle_auth_done global_id 5562 payload 824 2024-03-27T21:42:35.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f16d9d640 1 -- 172.21.15.60:0/3996175902 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1f1813c340 msgr2=0x7f1f181d6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:35.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.538+0000 7f1f16d9d640 1 --2- 172.21.15.60:0/3996175902 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1f1813c340 0x7f1f181d6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1f16d9d640 1 -- 172.21.15.60:0/3996175902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f180aeb30 msgr2=0x7f1f1813be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:35.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1f16d9d640 1 --2- 172.21.15.60:0/3996175902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f180aeb30 0x7f1f1813be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1f16d9d640 10 monclient: _finish_hunting 0 2024-03-27T21:42:35.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1f16d9d640 1 monclient: found mon.noname-a 2024-03-27T21:42:35.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1f16d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-27T21:42:35.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1f16d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.60:3300/0 2024-03-27T21:42:35.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1f16d9d640 1 -- 172.21.15.60:0/3996175902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f180aef90 con 0x7f1f000052a0 2024-03-27T21:42:35.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1f1759e640 1 --2- 172.21.15.60:0/3996175902 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1f1813c340 0x7f1f181d6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-27T21:42:35.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1f16d9d640 1 --2- 172.21.15.60:0/3996175902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f000052a0 0x7f1f180ae070 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1f10009f90 tx=0x7f1f1000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:35.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1eff7fe640 10 client.?.objecter ms_handle_connect 0x7f1f000052a0 2024-03-27T21:42:35.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1eff7fe640 10 client.?.objecter resend_mon_ops 2024-03-27T21:42:35.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1eff7fe640 1 -- 172.21.15.60:0/3996175902 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1000e040 con 0x7f1f000052a0 2024-03-27T21:42:35.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1eff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:35.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1eff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-27T21:42:35.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1eff7fe640 10 monclient: dump: 2024-03-27T21:42:35.560 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:35.560 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:35.560 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:35.560 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:35.560 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:35.560 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:35.560 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:35.561 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:35.561 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:35.561 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:35.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1eff7fe640 10 monclient: _finish_auth 0 2024-03-27T21:42:35.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1eff7fe640 10 monclient: _check_auth_tickets 2024-03-27T21:42:35.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1eff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-27T21:42:35.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1eff7fe640 1 -- 172.21.15.60:0/3996175902 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1000f040 con 0x7f1f000052a0 2024-03-27T21:42:35.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1f1d209640 5 monclient: authenticate success, global_id 5562 2024-03-27T21:42:35.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1f1d209640 10 monclient: _renew_subs 2024-03-27T21:42:35.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1f1d209640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:35.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1f1d209640 1 -- 172.21.15.60:0/3996175902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f1813caf0 con 0x7f1f000052a0 2024-03-27T21:42:35.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1eff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-27T21:42:35.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1eff7fe640 1 -- 172.21.15.60:0/3996175902 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f10015870 con 0x7f1f000052a0 2024-03-27T21:42:35.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1eff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-27T21:42:35.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1eff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-27T21:42:35.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1eff7fe640 10 monclient: dump: 2024-03-27T21:42:35.563 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-27T21:42:35.563 INFO:teuthology.orchestra.run.smithi060.stderr:fsid dcd19d9a-535c-4b41-9095-d588268e2860 2024-03-27T21:42:35.563 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:35.563 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-27T21:38:49.923026+0000 2024-03-27T21:42:35.563 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 18 (reef) 2024-03-27T21:42:35.563 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-27T21:42:35.563 INFO:teuthology.orchestra.run.smithi060.stderr:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-03-27T21:42:35.563 INFO:teuthology.orchestra.run.smithi060.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-03-27T21:42:35.563 INFO:teuthology.orchestra.run.smithi060.stderr:2: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.c 2024-03-27T21:42:35.564 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-27T21:42:35.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1f1d209640 10 client.5562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-27T21:42:35.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1f1d209640 10 monclient: _renew_subs 2024-03-27T21:42:35.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1f1d209640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:35.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1f1d209640 1 -- 172.21.15.60:0/3996175902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f181d6c90 con 0x7f1f000052a0 2024-03-27T21:42:35.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1f1559a640 4 set_mon_vals no callback set 2024-03-27T21:42:35.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.539+0000 7f1f1d209640 1 librados: init done 2024-03-27T21:42:35.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.540+0000 7f1f1d209640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:35.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.540+0000 7f1f1d209640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:35.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.540+0000 7f1f1d209640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:35.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.540+0000 7f1f1d209640 1 -- 172.21.15.60:0/3996175902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f18069470 con 0x7f1f000052a0 2024-03-27T21:42:35.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.540+0000 7f1eff7fe640 1 -- 172.21.15.60:0/3996175902 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1f100159d0 con 0x7f1f000052a0 2024-03-27T21:42:35.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.541+0000 7f1eff7fe640 1 --2- 172.21.15.60:0/3996175902 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1f0406c5c0 0x7f1f0406ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-27T21:42:35.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.541+0000 7f1eff7fe640 1 -- 172.21.15.60:0/3996175902 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f1f10089dd0 con 0x7f1f000052a0 2024-03-27T21:42:35.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.541+0000 7f1eff7fe640 10 client.5562.objecter ms_dispatch 0x7f1f18000b90 osd_map(92..92 src has 1..92) v4 2024-03-27T21:42:35.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.541+0000 7f1eff7fe640 3 client.5562.objecter handle_osd_map got epochs [92,92] > 0 2024-03-27T21:42:35.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.541+0000 7f1eff7fe640 3 client.5562.objecter handle_osd_map decoding full epoch 92 2024-03-27T21:42:35.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.541+0000 7f1f1659c640 1 --2- 172.21.15.60:0/3996175902 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1f0406c5c0 0x7f1f0406ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-27T21:42:35.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.541+0000 7f1f1659c640 10 monclient: get_auth_request con 0x7f1f0406c5c0 auth_method 0 2024-03-27T21:42:35.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.541+0000 7f1eff7fe640 20 client.5562.objecter dump_active .. 0 homeless 2024-03-27T21:42:35.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.541+0000 7f1eff7fe640 10 client.5562.objecter ms_handle_connect 0x7f1f0406c5c0 2024-03-27T21:42:35.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.541+0000 7f1f1659c640 1 --2- 172.21.15.60:0/3996175902 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1f0406c5c0 0x7f1f0406ea80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c000c00 tx=0x7f1f0c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-27T21:42:35.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.545+0000 7f1eff7fe640 1 -- 172.21.15.60:0/3996175902 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1f10055210 con 0x7f1f000052a0 2024-03-27T21:42:35.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.545+0000 7f1eff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-27T21:42:35.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.545+0000 7f1eff7fe640 10 monclient: _finish_command 1 = system:0 2024-03-27T21:42:35.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.723+0000 7f1f1d209640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-03-27T21:42:35.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.723+0000 7f1f1d209640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-03-27T21:42:35.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.723+0000 7f1f1d209640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.60:3300/0 2024-03-27T21:42:35.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.723+0000 7f1f1d209640 1 -- 172.21.15.60:0/3996175902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f1f1813cc80 con 0x7f1f000052a0 2024-03-27T21:42:35.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.726+0000 7f1eff7fe640 1 -- 172.21.15.60:0/3996175902 <== mon.0 v2:172.21.15.60:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f1f10055210 con 0x7f1f000052a0 2024-03-27T21:42:35.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.726+0000 7f1eff7fe640 10 monclient: handle_config config(1 keys) v1 2024-03-27T21:42:35.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.726+0000 7f1f1559a640 4 set_mon_vals no callback set 2024-03-27T21:42:35.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.727+0000 7f1f1559a640 10 set_mon_vals mon_health_to_clog = false 2024-03-27T21:42:35.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.727+0000 7f1eff7fe640 1 -- 172.21.15.60:0/3996175902 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v7)=0 v7) v1 ==== 123+0+0 (secure 0 0 0) 0x7f1f1813cc80 con 0x7f1f000052a0 2024-03-27T21:42:35.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.727+0000 7f1eff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-03-27T21:42:35.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.727+0000 7f1eff7fe640 10 monclient: _finish_command 2 = system:0 2024-03-27T21:42:35.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.729+0000 7f1f1d209640 10 librados: watch_flush enter 2024-03-27T21:42:35.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.729+0000 7f1f1d209640 10 librados: watch_flush exit 2024-03-27T21:42:35.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.729+0000 7f1f1d209640 20 client.5562.objecter shutdown clearing up homeless session... 2024-03-27T21:42:35.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.729+0000 7f1f1d209640 10 client.5562.objecter successfully canceled tick 2024-03-27T21:42:35.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.729+0000 7f1f1d209640 1 -- 172.21.15.60:0/3996175902 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1f0406c5c0 msgr2=0x7f1f0406ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:35.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.729+0000 7f1f1d209640 1 --2- 172.21.15.60:0/3996175902 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1f0406c5c0 0x7f1f0406ea80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c000c00 tx=0x7f1f0c009040 comp rx=0 tx=0).stop 2024-03-27T21:42:35.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.730+0000 7f1f1d209640 10 monclient: shutdown 2024-03-27T21:42:35.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.730+0000 7f1f1d209640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-27T21:42:35.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.730+0000 7f1f1d209640 1 -- 172.21.15.60:0/3996175902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f000052a0 msgr2=0x7f1f180ae070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-27T21:42:35.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.730+0000 7f1f1d209640 1 --2- 172.21.15.60:0/3996175902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f000052a0 0x7f1f180ae070 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1f10009f90 tx=0x7f1f1000f580 comp rx=0 tx=0).stop 2024-03-27T21:42:35.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.730+0000 7f1f1d209640 1 -- 172.21.15.60:0/3996175902 shutdown_connections 2024-03-27T21:42:35.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.730+0000 7f1f1d209640 1 --2- 172.21.15.60:0/3996175902 >> [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] conn(0x7f1f1813c340 0x7f1f181d6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.730+0000 7f1f1d209640 1 --2- 172.21.15.60:0/3996175902 >> [v2:172.21.15.60:6832/34387,v1:172.21.15.60:6833/34387] conn(0x7f1f0406c5c0 0x7f1f0406ea80 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.730+0000 7f1f1d209640 1 --2- 172.21.15.60:0/3996175902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f180aeb30 0x7f1f1813be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.730+0000 7f1f1d209640 1 --2- 172.21.15.60:0/3996175902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f000052a0 0x7f1f180ae070 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-27T21:42:35.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.730+0000 7f1f1d209640 1 -- 172.21.15.60:0/3996175902 >> 172.21.15.60:0/3996175902 conn(0x7f1f180608e0 msgr2=0x7f1f180afe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-27T21:42:35.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.730+0000 7f1f1d209640 1 -- 172.21.15.60:0/3996175902 shutdown_connections 2024-03-27T21:42:35.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.730+0000 7f1f1d209640 1 -- 172.21.15.60:0/3996175902 wait complete. 2024-03-27T21:42:35.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-27T21:42:35.730+0000 7f1f1d209640 1 librados: shutdown 2024-03-27T21:42:35.727 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-27T21:42:35.727 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-27T21:42:35.727 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-27T21:42:35.727 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T21:42:35.756 INFO:tasks.ceph.osd.0:Stopped 2024-03-27T21:42:35.757 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-27T21:42:35.757 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T21:42:35.824 INFO:tasks.ceph.osd.1:Stopped 2024-03-27T21:42:35.825 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-27T21:42:35.825 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T21:42:35.899 INFO:tasks.ceph.osd.2:Stopped 2024-03-27T21:42:35.900 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-27T21:42:35.900 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T21:42:35.967 INFO:tasks.ceph.osd.3:Stopped 2024-03-27T21:42:35.967 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-27T21:42:35.967 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T21:42:35.996 INFO:tasks.ceph.osd.4:Stopped 2024-03-27T21:42:35.997 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-27T21:42:35.997 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T21:42:36.069 INFO:tasks.ceph.osd.5:Stopped 2024-03-27T21:42:36.070 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-27T21:42:36.070 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T21:42:36.151 INFO:tasks.ceph.osd.6:Stopped 2024-03-27T21:42:36.151 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-27T21:42:36.151 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T21:42:36.225 INFO:tasks.ceph.osd.7:Stopped 2024-03-27T21:42:36.226 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-27T21:42:36.226 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-27T21:42:36.226 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T21:42:36.253 INFO:tasks.ceph.mgr.y:Stopped 2024-03-27T21:42:36.253 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-27T21:42:36.253 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T21:42:36.295 INFO:tasks.ceph.mgr.x:Stopped 2024-03-27T21:42:36.295 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-27T21:42:36.295 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-27T21:42:36.295 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T21:42:36.306 INFO:tasks.ceph.mon.a:Stopped 2024-03-27T21:42:36.307 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-27T21:42:36.307 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T21:42:36.358 INFO:tasks.ceph.mon.c:Stopped 2024-03-27T21:42:36.359 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-27T21:42:36.359 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T21:42:36.371 INFO:tasks.ceph.mon.b:Stopped 2024-03-27T21:42:36.371 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-27T21:42:36.372 DEBUG:teuthology.orchestra.run.smithi060:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-27T21:42:36.436 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi060.front.sepia.ceph.com 2024-03-27T21:42:36.436 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-27T21:42:36.675 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi060.front.sepia.ceph.com 2024-03-27T21:42:36.675 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-27T21:42:36.750 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi060.front.sepia.ceph.com 2024-03-27T21:42:36.751 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-27T21:42:36.816 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi060.front.sepia.ceph.com 2024-03-27T21:42:36.816 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-27T21:42:36.874 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-27T21:42:36.874 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-27T21:42:37.012 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-27T21:42:37.012 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-27T21:42:37.133 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-27T21:42:37.133 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-27T21:42:37.194 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-27T21:42:37.194 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-27T21:42:37.249 INFO:tasks.ceph:Archiving mon data... 2024-03-27T21:42:37.250 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-03-27_14:58:57-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626549/data/mon.a.tgz 2024-03-27T21:42:37.250 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-03-27T21:42:37.266 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.eMhicY7n1q 2024-03-27T21:42:37.266 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.eMhicY7n1q 2024-03-27T21:42:37.403 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.eMhicY7n1q 2024-03-27T21:42:37.488 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.eMhicY7n1q is 318KB 2024-03-27T21:42:37.583 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.eMhicY7n1q 2024-03-27T21:42:37.598 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-03-27_14:58:57-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626549/data/mon.c.tgz 2024-03-27T21:42:37.598 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-03-27T21:42:37.652 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.VKEj0i6rpI 2024-03-27T21:42:37.653 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.VKEj0i6rpI 2024-03-27T21:42:37.774 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.VKEj0i6rpI 2024-03-27T21:42:37.858 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.VKEj0i6rpI is 318KB 2024-03-27T21:42:37.952 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.VKEj0i6rpI 2024-03-27T21:42:37.967 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-03-27_14:58:57-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626549/data/mon.b.tgz 2024-03-27T21:42:37.967 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2024-03-27T21:42:37.984 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.5tHd9EeBXK 2024-03-27T21:42:37.985 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.5tHd9EeBXK 2024-03-27T21:42:38.126 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.5tHd9EeBXK 2024-03-27T21:42:38.212 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.5tHd9EeBXK is 318KB 2024-03-27T21:42:38.308 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.5tHd9EeBXK 2024-03-27T21:42:38.323 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-27T21:42:38.323 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-27T21:42:38.325 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-27T21:42:38.390 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-03-27T21:42:38.391 DEBUG:teuthology.orchestra.run.smithi060:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-03-27T21:42:38.393 DEBUG:teuthology.orchestra.run.smithi191:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-03-27T21:42:38.419 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-03-27T21:42:38.459 INFO:teuthology.orchestra.run.smithi191.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-03-27T21:42:38.463 INFO:tasks.ceph:Archiving crash dumps... 2024-03-27T21:42:38.465 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-03-27_14:58:57-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626549/remote/smithi060/crash 2024-03-27T21:42:38.465 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-27T21:42:38.492 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-03-27_14:58:57-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626549/remote/smithi191/crash 2024-03-27T21:42:38.493 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-27T21:42:38.529 INFO:tasks.ceph:Compressing logs... 2024-03-27T21:42:38.529 DEBUG:teuthology.orchestra.run.smithi060:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-27T21:42:38.537 DEBUG:teuthology.orchestra.run.smithi191:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-27T21:42:40.971 INFO:tasks.ceph:Archiving logs... 2024-03-27T21:42:40.971 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-27_14:58:57-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626549/remote/smithi060/log 2024-03-27T21:42:40.972 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-27T21:42:41.475 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-27_14:58:57-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626549/remote/smithi191/log 2024-03-27T21:42:41.477 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-27T21:42:41.682 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-27T21:42:41.691 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-27T21:42:41.692 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-27T21:42:41.694 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-27T21:42:41.789 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-27T21:42:41.799 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-27T21:42:41.799 DEBUG:teuthology.orchestra.run.smithi060:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-27T21:42:41.802 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-27T21:42:41.813 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-03-27T21:42:41.817 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-03-27T21:42:41.818 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-27T21:42:41.818 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-03-27T21:42:41.819 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 36 +380us[ +386us] +/- 86ms 2024-03-27T21:42:41.819 INFO:teuthology.orchestra.run.smithi060.stdout:^* hv02.front.sepia.ceph.com 3 6 377 35 -190us[ -183us] +/- 53ms 2024-03-27T21:42:41.819 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 -46us[ -40us] +/- 70ms 2024-03-27T21:42:41.819 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T21:42:41.823 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-27T21:42:41.823 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-03-27T21:42:41.823 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 36 +391us[ +391us] +/- 86ms 2024-03-27T21:42:41.823 INFO:teuthology.orchestra.run.smithi191.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 -183us[ -181us] +/- 53ms 2024-03-27T21:42:41.823 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 -34us[ -34us] +/- 70ms 2024-03-27T21:42:41.823 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T21:42:41.824 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-27T21:42:41.833 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-27T21:42:41.834 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-27T21:42:41.855 DEBUG:teuthology.orchestra.run.smithi060:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-27T21:42:41.937 DEBUG:teuthology.orchestra.run.smithi191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-27T21:42:42.015 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-27T21:42:42.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T21:42:42.045 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-27T21:42:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T21:42:42.078 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-27T21:42:42.087 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-27T21:42:42.149 INFO:teuthology.task.internal:Duration was 645.197386 seconds 2024-03-27T21:42:42.150 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-27T21:42:42.180 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-27T21:42:42.180 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-27T21:42:42.182 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-27T21:42:42.223 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-27T21:42:42.226 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-27T21:42:42.696 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-27T21:42:42.697 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi060.front.sepia.ceph.com 2024-03-27T21:42:42.697 DEBUG:teuthology.orchestra.run.smithi060:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-27T21:42:42.716 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2024-03-27T21:42:42.716 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-27T21:42:42.741 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-27T21:42:42.742 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-27T21:42:42.759 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-27T21:42:42.811 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-27T21:42:42.811 DEBUG:teuthology.orchestra.run.smithi060:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-27T21:42:42.829 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-27T21:42:42.962 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-27T21:42:42.972 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-27T21:42:42.973 DEBUG:teuthology.orchestra.run.smithi060:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-27T21:42:42.999 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-27T21:42:43.026 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-27T21:42:43.037 DEBUG:teuthology.orchestra.run.smithi060:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-27T21:42:43.042 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-27T21:42:43.065 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = core 2024-03-27T21:42:43.092 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2024-03-27T21:42:43.149 DEBUG:teuthology.orchestra.run.smithi060:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-27T21:42:43.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T21:42:43.163 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-27T21:42:43.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T21:42:43.203 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-27T21:42:43.213 INFO:teuthology.task.internal:Transferring archived files... 2024-03-27T21:42:43.214 DEBUG:teuthology.misc:Transferring archived files from smithi060:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-27_14:58:57-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626549/remote/smithi060 2024-03-27T21:42:43.214 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-27T21:42:43.252 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-27_14:58:57-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626549/remote/smithi191 2024-03-27T21:42:43.253 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-27T21:42:43.288 INFO:teuthology.task.internal:Removing archive directory... 2024-03-27T21:42:43.289 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-27T21:42:43.291 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-27T21:42:43.338 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-27T21:42:43.349 INFO:teuthology.task.internal:Not uploading archives. 2024-03-27T21:42:43.350 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-27T21:42:43.372 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-27T21:42:43.372 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-27T21:42:43.375 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-27T21:42:43.388 INFO:teuthology.orchestra.run.smithi060.stdout: 262166 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:42 /home/ubuntu/cephtest 2024-03-27T21:42:43.388 INFO:teuthology.orchestra.run.smithi060.stdout: 402825 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:39 /home/ubuntu/cephtest/mnt.0 2024-03-27T21:42:43.389 INFO:teuthology.orchestra.run.smithi060.stdout: 402826 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:39 /home/ubuntu/cephtest/mnt.0/client.0 2024-03-27T21:42:43.389 INFO:teuthology.orchestra.run.smithi060.stdout: 402827 4 drwxr-xr-x 26 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0 2024-03-27T21:42:43.389 INFO:teuthology.orchestra.run.smithi060.stdout: 403997 4 -rwxr-xr-x 1 ubuntu ubuntu 2777 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/make-debs.sh 2024-03-27T21:42:43.389 INFO:teuthology.orchestra.run.smithi060.stdout: 403023 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian 2024-03-27T21:42:43.389 INFO:teuthology.orchestra.run.smithi060.stdout: 403041 4 -rw-r--r-- 1 ubuntu ubuntu 1209 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mds.postinst 2024-03-27T21:42:43.389 INFO:teuthology.orchestra.run.smithi060.stdout: 403040 4 -rw-r--r-- 1 ubuntu ubuntu 76 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mds.install 2024-03-27T21:42:43.389 INFO:teuthology.orchestra.run.smithi060.stdout: 403087 4 -rw-r--r-- 1 ubuntu ubuntu 3 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/compat 2024-03-27T21:42:43.389 INFO:teuthology.orchestra.run.smithi060.stdout: 403117 4 -rw-r--r-- 1 ubuntu ubuntu 260 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/radosgw.prerm 2024-03-27T21:42:43.390 INFO:teuthology.orchestra.run.smithi060.stdout: 403111 4 -rw-r--r-- 1 ubuntu ubuntu 93 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/python3-rbd.install 2024-03-27T21:42:43.390 INFO:teuthology.orchestra.run.smithi060.stdout: 403116 4 -rw-r--r-- 1 ubuntu ubuntu 1626 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/radosgw.postinst 2024-03-27T21:42:43.390 INFO:teuthology.orchestra.run.smithi060.stdout: 403095 4 -rw-r--r-- 1 ubuntu ubuntu 80 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/librados2.install 2024-03-27T21:42:43.390 INFO:teuthology.orchestra.run.smithi060.stdout: 403049 4 -rw-r--r-- 1 ubuntu ubuntu 61 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr-dashboard.prerm 2024-03-27T21:42:43.390 INFO:teuthology.orchestra.run.smithi060.stdout: 403046 4 -rw-r--r-- 1 ubuntu ubuntu 38 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr-cephadm.requires 2024-03-27T21:42:43.390 INFO:teuthology.orchestra.run.smithi060.stdout: 403052 4 -rw-r--r-- 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr-diskprediction-local.install 2024-03-27T21:42:43.390 INFO:teuthology.orchestra.run.smithi060.stdout: 403094 4 -rw-r--r-- 1 ubuntu ubuntu 165 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/librados-dev.install 2024-03-27T21:42:43.390 INFO:teuthology.orchestra.run.smithi060.stdout: 403076 4 -rw-r--r-- 1 ubuntu ubuntu 44 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-prometheus-alerts.install 2024-03-27T21:42:43.391 INFO:teuthology.orchestra.run.smithi060.stdout: 403088 48 -rw-r--r-- 1 ubuntu ubuntu 45190 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/control 2024-03-27T21:42:43.391 INFO:teuthology.orchestra.run.smithi060.stdout: 403043 4 -rw-r--r-- 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr-cephadm.install 2024-03-27T21:42:43.391 INFO:teuthology.orchestra.run.smithi060.stdout: 403073 4 -rwxr-xr-x 1 ubuntu ubuntu 528 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-osd.install 2024-03-27T21:42:43.391 INFO:teuthology.orchestra.run.smithi060.stdout: 403075 4 -rw-r--r-- 1 ubuntu ubuntu 262 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-osd.prerm 2024-03-27T21:42:43.391 INFO:teuthology.orchestra.run.smithi060.stdout: 403098 4 -rw-r--r-- 1 ubuntu ubuntu 29 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/libradosstriper1.install 2024-03-27T21:42:43.391 INFO:teuthology.orchestra.run.smithi060.stdout: 403078 4 -rw-r--r-- 1 ubuntu ubuntu 546 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-test.install 2024-03-27T21:42:43.391 INFO:teuthology.orchestra.run.smithi060.stdout: 403119 4 -rw-r--r-- 1 ubuntu ubuntu 87 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/rbd-mirror.install 2024-03-27T21:42:43.391 INFO:teuthology.orchestra.run.smithi060.stdout: 403044 4 -rw-r--r-- 1 ubuntu ubuntu 997 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr-cephadm.postinst 2024-03-27T21:42:43.391 INFO:teuthology.orchestra.run.smithi060.stdout: 403080 4 -rw-r--r-- 1 ubuntu ubuntu 46 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/cephadm.install 2024-03-27T21:42:43.392 INFO:teuthology.orchestra.run.smithi060.stdout: 403102 4 -rw-r--r-- 1 ubuntu ubuntu 20 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/librgw2.install 2024-03-27T21:42:43.392 INFO:teuthology.orchestra.run.smithi060.stdout: 403037 4 -rw-r--r-- 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-grafana-dashboards.install 2024-03-27T21:42:43.392 INFO:teuthology.orchestra.run.smithi060.stdout: 403051 4 -rw-r--r-- 1 ubuntu ubuntu 61 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr-diskprediction-cloud.prerm 2024-03-27T21:42:43.392 INFO:teuthology.orchestra.run.smithi060.stdout: 403071 4 -rw-r--r-- 1 ubuntu ubuntu 262 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mon.prerm 2024-03-27T21:42:43.392 INFO:teuthology.orchestra.run.smithi060.stdout: 403106 4 -rw-r--r-- 1 ubuntu ubuntu 126 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/py3dist-overrides 2024-03-27T21:42:43.392 INFO:teuthology.orchestra.run.smithi060.stdout: 403054 4 -rw-r--r-- 1 ubuntu ubuntu 61 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr-diskprediction-local.prerm 2024-03-27T21:42:43.392 INFO:teuthology.orchestra.run.smithi060.stdout: 403113 4 -rw-r--r-- 1 ubuntu ubuntu 29 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/rados-objclass-dev.install 2024-03-27T21:42:43.392 INFO:teuthology.orchestra.run.smithi060.stdout: 403074 4 -rw-r--r-- 1 ubuntu ubuntu 1137 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-osd.postinst 2024-03-27T21:42:43.392 INFO:teuthology.orchestra.run.smithi060.stdout: 403039 4 -rw-r--r-- 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mds.dirs 2024-03-27T21:42:43.393 INFO:teuthology.orchestra.run.smithi060.stdout: 403097 4 -rw-r--r-- 1 ubuntu ubuntu 115 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/libradosstriper-dev.install 2024-03-27T21:42:43.393 INFO:teuthology.orchestra.run.smithi060.stdout: 403115 4 -rw-r--r-- 1 ubuntu ubuntu 296 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/radosgw.install 2024-03-27T21:42:43.393 INFO:teuthology.orchestra.run.smithi060.stdout: 403077 4 -rw-r--r-- 1 ubuntu ubuntu 30 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-resource-agents.install 2024-03-27T21:42:43.393 INFO:teuthology.orchestra.run.smithi060.stdout: 403060 4 -rw-r--r-- 1 ubuntu ubuntu 1012 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr-rook.postinst 2024-03-27T21:42:43.393 INFO:teuthology.orchestra.run.smithi060.stdout: 403092 4 -rw-r--r-- 1 ubuntu ubuntu 38 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/libcephfs-jni.install 2024-03-27T21:42:43.393 INFO:teuthology.orchestra.run.smithi060.stdout: 403112 4 -rw-r--r-- 1 ubuntu ubuntu 93 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/python3-rgw.install 2024-03-27T21:42:43.393 INFO:teuthology.orchestra.run.smithi060.stdout: 403065 4 -rw-r--r-- 1 ubuntu ubuntu 1209 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr.postinst 2024-03-27T21:42:43.393 INFO:teuthology.orchestra.run.smithi060.stdout: 403070 4 -rw-r--r-- 1 ubuntu ubuntu 1075 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mon.postinst 2024-03-27T21:42:43.393 INFO:teuthology.orchestra.run.smithi060.stdout: 403089 40 -rw-r--r-- 1 ubuntu ubuntu 36891 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/copyright 2024-03-27T21:42:43.394 INFO:teuthology.orchestra.run.smithi060.stdout: 403025 4 -rw-r--r-- 1 ubuntu ubuntu 231 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-base.dirs 2024-03-27T21:42:43.394 INFO:teuthology.orchestra.run.smithi060.stdout: 403107 4 -rwxr-xr-x 1 ubuntu ubuntu 173 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/python3-ceph-argparse.install 2024-03-27T21:42:43.394 INFO:teuthology.orchestra.run.smithi060.stdout: 403027 4 -rw-r--r-- 1 ubuntu ubuntu 205 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-base.lintian-overrides 2024-03-27T21:42:43.394 INFO:teuthology.orchestra.run.smithi060.stdout: 403096 4 -rw-r--r-- 1 ubuntu ubuntu 244 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/libradospp-dev.install 2024-03-27T21:42:43.394 INFO:teuthology.orchestra.run.smithi060.stdout: 403063 4 -rw-r--r-- 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr.dirs 2024-03-27T21:42:43.394 INFO:teuthology.orchestra.run.smithi060.stdout: 403053 4 -rw-r--r-- 1 ubuntu ubuntu 1012 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr-diskprediction-local.postinst 2024-03-27T21:42:43.394 INFO:teuthology.orchestra.run.smithi060.stdout: 403029 4 -rw-r--r-- 1 ubuntu ubuntu 1702 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-base.postinst 2024-03-27T21:42:43.394 INFO:teuthology.orchestra.run.smithi060.stdout: 403030 4 -rw-r--r-- 1 ubuntu ubuntu 254 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-base.prerm 2024-03-27T21:42:43.395 INFO:teuthology.orchestra.run.smithi060.stdout: 403101 4 -rw-r--r-- 1 ubuntu ubuntu 74 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/librgw-dev.install 2024-03-27T21:42:43.395 INFO:teuthology.orchestra.run.smithi060.stdout: 403045 4 -rw-r--r-- 1 ubuntu ubuntu 61 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr-cephadm.prerm 2024-03-27T21:42:43.395 INFO:teuthology.orchestra.run.smithi060.stdout: 403108 4 -rw-r--r-- 1 ubuntu ubuntu 83 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/python3-ceph-common.install 2024-03-27T21:42:43.395 INFO:teuthology.orchestra.run.smithi060.stdout: 403068 4 -rw-r--r-- 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mon.dirs 2024-03-27T21:42:43.395 INFO:teuthology.orchestra.run.smithi060.stdout: 403109 4 -rw-r--r-- 1 ubuntu ubuntu 99 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/python3-cephfs.install 2024-03-27T21:42:43.395 INFO:teuthology.orchestra.run.smithi060.stdout: 403057 4 -rw-r--r-- 1 ubuntu ubuntu 777 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr-modules-core.install 2024-03-27T21:42:43.395 INFO:teuthology.orchestra.run.smithi060.stdout: 403118 4 -rw-r--r-- 1 ubuntu ubuntu 47 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/rbd-fuse.install 2024-03-27T21:42:43.395 INFO:teuthology.orchestra.run.smithi060.stdout: 403034 4 -rw-r--r-- 1 ubuntu ubuntu 1263 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-common.postrm 2024-03-27T21:42:43.396 INFO:teuthology.orchestra.run.smithi060.stdout: 403124 4 -rw-r--r-- 1 ubuntu ubuntu 66 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/watch 2024-03-27T21:42:43.396 INFO:teuthology.orchestra.run.smithi060.stdout: 403036 4 -rw-r--r-- 1 ubuntu ubuntu 146 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-fuse.install 2024-03-27T21:42:43.396 INFO:teuthology.orchestra.run.smithi060.stdout: 403082 4 -rw-r--r-- 1 ubuntu ubuntu 987 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/cephadm.postrm 2024-03-27T21:42:43.396 INFO:teuthology.orchestra.run.smithi060.stdout: 403085 4 -rw-r--r-- 1 ubuntu ubuntu 72 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/cephfs-top.install 2024-03-27T21:42:43.396 INFO:teuthology.orchestra.run.smithi060.stdout: 403033 4 -rw-r--r-- 1 ubuntu ubuntu 4086 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-common.postinst 2024-03-27T21:42:43.396 INFO:teuthology.orchestra.run.smithi060.stdout: 403056 4 -rw-r--r-- 1 ubuntu ubuntu 18 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr-k8sevents.requires 2024-03-27T21:42:43.396 INFO:teuthology.orchestra.run.smithi060.stdout: 403086 20 -rw-r--r-- 1 ubuntu ubuntu 18214 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/changelog 2024-03-27T21:42:43.396 INFO:teuthology.orchestra.run.smithi060.stdout: 403061 4 -rw-r--r-- 1 ubuntu ubuntu 61 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr-rook.prerm 2024-03-27T21:42:43.396 INFO:teuthology.orchestra.run.smithi060.stdout: 403099 4 -rw-r--r-- 1 ubuntu ubuntu 118 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/librbd-dev.install 2024-03-27T21:42:43.397 INFO:teuthology.orchestra.run.smithi060.stdout: 403090 4 -rw-r--r-- 1 ubuntu ubuntu 150 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/libcephfs-dev.install 2024-03-27T21:42:43.397 INFO:teuthology.orchestra.run.smithi060.stdout: 403038 4 -rw-r--r-- 1 ubuntu ubuntu 133 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-immutable-object-cache.install 2024-03-27T21:42:43.397 INFO:teuthology.orchestra.run.smithi060.stdout: 403100 4 -rw-r--r-- 1 ubuntu ubuntu 65 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/librbd1.install 2024-03-27T21:42:43.397 INFO:teuthology.orchestra.run.smithi060.stdout: 403032 4 -rwxr-xr-x 1 ubuntu ubuntu 1386 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-common.install 2024-03-27T21:42:43.397 INFO:teuthology.orchestra.run.smithi060.stdout: 403026 4 -rw-r--r-- 1 ubuntu ubuntu 632 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-base.install 2024-03-27T21:42:43.397 INFO:teuthology.orchestra.run.smithi060.stdout: 403048 4 -rw-r--r-- 1 ubuntu ubuntu 1001 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr-dashboard.postinst 2024-03-27T21:42:43.397 INFO:teuthology.orchestra.run.smithi060.stdout: 403047 4 -rw-r--r-- 1 ubuntu ubuntu 29 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr-dashboard.install 2024-03-27T21:42:43.397 INFO:teuthology.orchestra.run.smithi060.stdout: 403064 4 -rw-r--r-- 1 ubuntu ubuntu 143 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr.install 2024-03-27T21:42:43.398 INFO:teuthology.orchestra.run.smithi060.stdout: 403114 4 -rw-r--r-- 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/radosgw.dirs 2024-03-27T21:42:43.398 INFO:teuthology.orchestra.run.smithi060.stdout: 403091 4 -rw-r--r-- 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/libcephfs-java.jlibs 2024-03-27T21:42:43.398 INFO:teuthology.orchestra.run.smithi060.stdout: 403042 4 -rw-r--r-- 1 ubuntu ubuntu 259 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mds.prerm 2024-03-27T21:42:43.398 INFO:teuthology.orchestra.run.smithi060.stdout: 403058 4 -rw-r--r-- 1 ubuntu ubuntu 71 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr-modules-core.requires 2024-03-27T21:42:43.398 INFO:teuthology.orchestra.run.smithi060.stdout: 403072 4 -rw-r--r-- 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-osd.dirs 2024-03-27T21:42:43.398 INFO:teuthology.orchestra.run.smithi060.stdout: 403103 4 -rw-r--r-- 1 ubuntu ubuntu 28 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/libsqlite3-mod-ceph-dev.install 2024-03-27T21:42:43.398 INFO:teuthology.orchestra.run.smithi060.stdout: 403110 4 -rw-r--r-- 1 ubuntu ubuntu 97 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/python3-rados.install 2024-03-27T21:42:43.398 INFO:teuthology.orchestra.run.smithi060.stdout: 403079 4 -rw-r--r-- 1 ubuntu ubuntu 252 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-volume.install 2024-03-27T21:42:43.399 INFO:teuthology.orchestra.run.smithi060.stdout: 403121 8 -rwxr-xr-x 1 ubuntu ubuntu 6427 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/rules 2024-03-27T21:42:43.399 INFO:teuthology.orchestra.run.smithi060.stdout: 403059 4 -rw-r--r-- 1 ubuntu ubuntu 24 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr-rook.install 2024-03-27T21:42:43.399 INFO:teuthology.orchestra.run.smithi060.stdout: 403035 4 -rw-r--r-- 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-exporter.install 2024-03-27T21:42:43.399 INFO:teuthology.orchestra.run.smithi060.stdout: 403028 4 -rw-r--r-- 1 ubuntu ubuntu 78 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-base.maintscript 2024-03-27T21:42:43.399 INFO:teuthology.orchestra.run.smithi060.stdout: 403081 4 -rw-r--r-- 1 ubuntu ubuntu 2039 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/cephadm.postinst 2024-03-27T21:42:43.399 INFO:teuthology.orchestra.run.smithi060.stdout: 403024 4 -rw-r--r-- 1 ubuntu ubuntu 536 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/.gitignore 2024-03-27T21:42:43.399 INFO:teuthology.orchestra.run.smithi060.stdout: 403062 4 -rw-r--r-- 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr-rook.requires 2024-03-27T21:42:43.399 INFO:teuthology.orchestra.run.smithi060.stdout: 403083 4 -rw-r--r-- 1 ubuntu ubuntu 91 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/cephfs-mirror.install 2024-03-27T21:42:43.399 INFO:teuthology.orchestra.run.smithi060.stdout: 403084 4 -rw-r--r-- 1 ubuntu ubuntu 110 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/cephfs-shell.install 2024-03-27T21:42:43.400 INFO:teuthology.orchestra.run.smithi060.stdout: 403120 4 -rw-r--r-- 1 ubuntu ubuntu 81 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/rbd-nbd.install 2024-03-27T21:42:43.400 INFO:teuthology.orchestra.run.smithi060.stdout: 403069 4 -rw-r--r-- 1 ubuntu ubuntu 103 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mon.install 2024-03-27T21:42:43.400 INFO:teuthology.orchestra.run.smithi060.stdout: 403031 4 -rw-r--r-- 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-common.dirs 2024-03-27T21:42:43.400 INFO:teuthology.orchestra.run.smithi060.stdout: 403105 4 -rw-r--r-- 1 ubuntu ubuntu 2084 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/libsqlite3-mod-ceph.symbols 2024-03-27T21:42:43.400 INFO:teuthology.orchestra.run.smithi060.stdout: 403067 4 -rw-r--r-- 1 ubuntu ubuntu 55 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr.requires 2024-03-27T21:42:43.400 INFO:teuthology.orchestra.run.smithi060.stdout: 403122 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/source 2024-03-27T21:42:43.400 INFO:teuthology.orchestra.run.smithi060.stdout: 403123 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/source/format 2024-03-27T21:42:43.400 INFO:teuthology.orchestra.run.smithi060.stdout: 403093 4 -rw-r--r-- 1 ubuntu ubuntu 23 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/libcephfs2.install 2024-03-27T21:42:43.400 INFO:teuthology.orchestra.run.smithi060.stdout: 403104 4 -rw-r--r-- 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/libsqlite3-mod-ceph.install 2024-03-27T21:42:43.401 INFO:teuthology.orchestra.run.smithi060.stdout: 403066 4 -rw-r--r-- 1 ubuntu ubuntu 259 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr.prerm 2024-03-27T21:42:43.401 INFO:teuthology.orchestra.run.smithi060.stdout: 403055 4 -rw-r--r-- 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr-diskprediction-local.requires 2024-03-27T21:42:43.401 INFO:teuthology.orchestra.run.smithi060.stdout: 403050 4 -rw-r--r-- 1 ubuntu ubuntu 108 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/debian/ceph-mgr-dashboard.requires 2024-03-27T21:42:43.401 INFO:teuthology.orchestra.run.smithi060.stdout: 402900 8 -rw-r--r-- 1 ubuntu ubuntu 7652 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/COPYING-LGPL3 2024-03-27T21:42:43.401 INFO:teuthology.orchestra.run.smithi060.stdout: 402885 8 -rw-r--r-- 1 ubuntu ubuntu 8116 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.githubmap 2024-03-27T21:42:43.401 INFO:teuthology.orchestra.run.smithi060.stdout: 402905 8 -rw-r--r-- 1 ubuntu ubuntu 8181 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/README.md 2024-03-27T21:42:43.401 INFO:teuthology.orchestra.run.smithi060.stdout: 403901 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/etc 2024-03-27T21:42:43.401 INFO:teuthology.orchestra.run.smithi060.stdout: 403902 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/etc/default 2024-03-27T21:42:43.401 INFO:teuthology.orchestra.run.smithi060.stdout: 403903 4 -rw-r--r-- 1 ubuntu ubuntu 159 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/etc/default/ceph 2024-03-27T21:42:43.402 INFO:teuthology.orchestra.run.smithi060.stdout: 403906 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/etc/sysctl 2024-03-27T21:42:43.402 INFO:teuthology.orchestra.run.smithi060.stdout: 403908 4 -rw-r--r-- 1 ubuntu ubuntu 41 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/etc/sysctl/90-ceph-osd.conf.in 2024-03-27T21:42:43.402 INFO:teuthology.orchestra.run.smithi060.stdout: 403909 4 -rw-r--r-- 1 ubuntu ubuntu 300 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/etc/sysctl/CMakeLists.txt 2024-03-27T21:42:43.402 INFO:teuthology.orchestra.run.smithi060.stdout: 403907 4 -rw-r--r-- 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/etc/sysctl/.gitignore 2024-03-27T21:42:43.402 INFO:teuthology.orchestra.run.smithi060.stdout: 403904 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/etc/sysconfig 2024-03-27T21:42:43.402 INFO:teuthology.orchestra.run.smithi060.stdout: 403905 4 -rw-r--r-- 1 ubuntu ubuntu 559 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/etc/sysconfig/ceph 2024-03-27T21:42:43.402 INFO:teuthology.orchestra.run.smithi060.stdout: 402890 52 -rw-r--r-- 1 ubuntu ubuntu 50723 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.mailmap 2024-03-27T21:42:43.402 INFO:teuthology.orchestra.run.smithi060.stdout: 408965 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/share 2024-03-27T21:42:43.403 INFO:teuthology.orchestra.run.smithi060.stdout: 408968 4 -rw-r--r-- 1 ubuntu ubuntu 395 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/share/known_hosts_drop.ceph.com 2024-03-27T21:42:43.403 INFO:teuthology.orchestra.run.smithi060.stdout: 408967 4 -rw-r--r-- 1 ubuntu ubuntu 413 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/share/id_rsa_drop.ceph.com.pub 2024-03-27T21:42:43.403 INFO:teuthology.orchestra.run.smithi060.stdout: 408966 4 -rw-r--r-- 1 ubuntu ubuntu 1679 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/share/id_rsa_drop.ceph.com 2024-03-27T21:42:43.403 INFO:teuthology.orchestra.run.smithi060.stdout: 408969 4 -rw-r--r-- 1 ubuntu ubuntu 1002 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/share/uncrustify.cfg 2024-03-27T21:42:43.403 INFO:teuthology.orchestra.run.smithi060.stdout: 402921 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/ceph-erasure-code-corpus 2024-03-27T21:42:43.403 INFO:teuthology.orchestra.run.smithi060.stdout: 402896 4 -rw-r--r-- 1 ubuntu ubuntu 631 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/CONTRIBUTING.rst 2024-03-27T21:42:43.403 INFO:teuthology.orchestra.run.smithi060.stdout: 402898 20 -rw-r--r-- 1 ubuntu ubuntu 18092 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/COPYING-GPL2 2024-03-27T21:42:43.403 INFO:teuthology.orchestra.run.smithi060.stdout: 402902 28 -rw-r--r-- 1 ubuntu ubuntu 25726 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/PendingReleaseNotes 2024-03-27T21:42:43.404 INFO:teuthology.orchestra.run.smithi060.stdout: 403998 8 -rwxr-xr-x 1 ubuntu ubuntu 7456 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/make-dist 2024-03-27T21:42:43.404 INFO:teuthology.orchestra.run.smithi060.stdout: 402894 4 -rw-r--r-- 1 ubuntu ubuntu 322 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/AUTHORS 2024-03-27T21:42:43.404 INFO:teuthology.orchestra.run.smithi060.stdout: 403910 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples 2024-03-27T21:42:43.404 INFO:teuthology.orchestra.run.smithi060.stdout: 403920 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rbd-replay 2024-03-27T21:42:43.404 INFO:teuthology.orchestra.run.smithi060.stdout: 403925 4 -rwxr-xr-x 1 ubuntu ubuntu 326 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rbd-replay/trace 2024-03-27T21:42:43.404 INFO:teuthology.orchestra.run.smithi060.stdout: 403923 4 -rwxr-xr-x 1 ubuntu ubuntu 236 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rbd-replay/replay 2024-03-27T21:42:43.404 INFO:teuthology.orchestra.run.smithi060.stdout: 403922 4 -rwxr-xr-x 1 ubuntu ubuntu 245 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rbd-replay/create-image 2024-03-27T21:42:43.404 INFO:teuthology.orchestra.run.smithi060.stdout: 403921 4 -rw-r--r-- 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rbd-replay/.gitignore 2024-03-27T21:42:43.405 INFO:teuthology.orchestra.run.smithi060.stdout: 403924 4 -rwxr-xr-x 1 ubuntu ubuntu 86 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rbd-replay/run-rbd-replay-prep 2024-03-27T21:42:43.405 INFO:teuthology.orchestra.run.smithi060.stdout: 403911 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/librados 2024-03-27T21:42:43.405 INFO:teuthology.orchestra.run.smithi060.stdout: 403916 12 -rw-r--r-- 1 ubuntu ubuntu 9667 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/librados/hello_world_c.c 2024-03-27T21:42:43.405 INFO:teuthology.orchestra.run.smithi060.stdout: 403912 4 -rw-r--r-- 1 ubuntu ubuntu 1133 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/librados/Makefile 2024-03-27T21:42:43.405 INFO:teuthology.orchestra.run.smithi060.stdout: 403913 4 -rw-r--r-- 1 ubuntu ubuntu 2756 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/librados/hello_radosstriper.cc 2024-03-27T21:42:43.405 INFO:teuthology.orchestra.run.smithi060.stdout: 403915 4 -rw-r--r-- 1 ubuntu ubuntu 646 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/librados/hello_world.readme 2024-03-27T21:42:43.405 INFO:teuthology.orchestra.run.smithi060.stdout: 403914 12 -rw-r--r-- 1 ubuntu ubuntu 9293 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/librados/hello_world.cc 2024-03-27T21:42:43.405 INFO:teuthology.orchestra.run.smithi060.stdout: 403926 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw 2024-03-27T21:42:43.405 INFO:teuthology.orchestra.run.smithi060.stdout: 403938 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/golang 2024-03-27T21:42:43.406 INFO:teuthology.orchestra.run.smithi060.stdout: 403947 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/golang/topic-creation 2024-03-27T21:42:43.406 INFO:teuthology.orchestra.run.smithi060.stdout: 403948 4 -rw-r--r-- 1 ubuntu ubuntu 771 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/golang/topic-creation/README.md 2024-03-27T21:42:43.406 INFO:teuthology.orchestra.run.smithi060.stdout: 403949 4 -rw-r--r-- 1 ubuntu ubuntu 2133 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/golang/topic-creation/topic-creation.go 2024-03-27T21:42:43.406 INFO:teuthology.orchestra.run.smithi060.stdout: 403944 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/golang/put-bucket-notification-creation 2024-03-27T21:42:43.406 INFO:teuthology.orchestra.run.smithi060.stdout: 403945 4 -rw-r--r-- 1 ubuntu ubuntu 1524 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/golang/put-bucket-notification-creation/README.md 2024-03-27T21:42:43.406 INFO:teuthology.orchestra.run.smithi060.stdout: 403946 4 -rw-r--r-- 1 ubuntu ubuntu 2465 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/golang/put-bucket-notification-creation/put-bucket-notification-creation.go 2024-03-27T21:42:43.406 INFO:teuthology.orchestra.run.smithi060.stdout: 403940 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/golang/object-upload 2024-03-27T21:42:43.406 INFO:teuthology.orchestra.run.smithi060.stdout: 403941 4 -rw-r--r-- 1 ubuntu ubuntu 1024 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/golang/object-upload/README.md 2024-03-27T21:42:43.407 INFO:teuthology.orchestra.run.smithi060.stdout: 403943 4 -rw-r--r-- 1 ubuntu ubuntu 2187 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/golang/object-upload/object-upload.go 2024-03-27T21:42:43.407 INFO:teuthology.orchestra.run.smithi060.stdout: 403942 4 -rw-r--r-- 1 ubuntu ubuntu 76 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/golang/object-upload/fortuna.txt 2024-03-27T21:42:43.407 INFO:teuthology.orchestra.run.smithi060.stdout: 403939 4 -rw-r--r-- 1 ubuntu ubuntu 14 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/golang/.gitignore 2024-03-27T21:42:43.407 INFO:teuthology.orchestra.run.smithi060.stdout: 403968 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/lua 2024-03-27T21:42:43.407 INFO:teuthology.orchestra.run.smithi060.stdout: 403980 4 -rw-r--r-- 1 ubuntu ubuntu 2307 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/lua/storage_class.md 2024-03-27T21:42:43.407 INFO:teuthology.orchestra.run.smithi060.stdout: 403973 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/lua/img 2024-03-27T21:42:43.407 INFO:teuthology.orchestra.run.smithi060.stdout: 403974 408 -rw-r--r-- 1 ubuntu ubuntu 414931 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/lua/img/prometheus.png 2024-03-27T21:42:43.407 INFO:teuthology.orchestra.run.smithi060.stdout: 403977 4 -rw-r--r-- 1 ubuntu ubuntu 685 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/lua/prometheus_adapter.lua 2024-03-27T21:42:43.408 INFO:teuthology.orchestra.run.smithi060.stdout: 403971 4 -rw-r--r-- 1 ubuntu ubuntu 3276 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/lua/elasticsearch_adapter.lua 2024-03-27T21:42:43.408 INFO:teuthology.orchestra.run.smithi060.stdout: 403979 4 -rw-r--r-- 1 ubuntu ubuntu 792 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/lua/storage_class.lua 2024-03-27T21:42:43.408 INFO:teuthology.orchestra.run.smithi060.stdout: 403976 4 -rw-r--r-- 1 ubuntu ubuntu 3178 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/lua/nats_adapter.md 2024-03-27T21:42:43.408 INFO:teuthology.orchestra.run.smithi060.stdout: 403978 4 -rw-r--r-- 1 ubuntu ubuntu 1850 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/lua/prometheus_adapter.md 2024-03-27T21:42:43.408 INFO:teuthology.orchestra.run.smithi060.stdout: 403969 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/lua/config 2024-03-27T21:42:43.408 INFO:teuthology.orchestra.run.smithi060.stdout: 403970 4 -rw-r--r-- 1 ubuntu ubuntu 669 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/lua/config/prometheus.yml 2024-03-27T21:42:43.408 INFO:teuthology.orchestra.run.smithi060.stdout: 403975 4 -rw-r--r-- 1 ubuntu ubuntu 3874 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/lua/nats_adapter.lua 2024-03-27T21:42:43.408 INFO:teuthology.orchestra.run.smithi060.stdout: 403972 4 -rw-r--r-- 1 ubuntu ubuntu 1831 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/lua/elasticsearch_adapter.md 2024-03-27T21:42:43.408 INFO:teuthology.orchestra.run.smithi060.stdout: 403981 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/rgw-cache 2024-03-27T21:42:43.409 INFO:teuthology.orchestra.run.smithi060.stdout: 403982 8 -rw-r--r-- 1 ubuntu ubuntu 5205 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/rgw-cache/nginx-default.conf 2024-03-27T21:42:43.409 INFO:teuthology.orchestra.run.smithi060.stdout: 403985 8 -rw-r--r-- 1 ubuntu ubuntu 5232 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/rgw-cache/nginx-slicing.conf 2024-03-27T21:42:43.409 INFO:teuthology.orchestra.run.smithi060.stdout: 403984 4 -rw-r--r-- 1 ubuntu ubuntu 4018 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/rgw-cache/nginx-noprefetch.conf 2024-03-27T21:42:43.409 INFO:teuthology.orchestra.run.smithi060.stdout: 403983 4 -rw-r--r-- 1 ubuntu ubuntu 1161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/rgw-cache/nginx-lua-file.lua 2024-03-27T21:42:43.409 INFO:teuthology.orchestra.run.smithi060.stdout: 403986 4 -rw-r--r-- 1 ubuntu ubuntu 1808 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/rgw-cache/nginx.conf 2024-03-27T21:42:43.409 INFO:teuthology.orchestra.run.smithi060.stdout: 403950 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/java 2024-03-27T21:42:43.409 INFO:teuthology.orchestra.run.smithi060.stdout: 403951 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/java/ceph-s3-upload 2024-03-27T21:42:43.409 INFO:teuthology.orchestra.run.smithi060.stdout: 403952 4 -rw-r--r-- 1 ubuntu ubuntu 1040 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/java/ceph-s3-upload/README.md 2024-03-27T21:42:43.410 INFO:teuthology.orchestra.run.smithi060.stdout: 403953 4 -rw-r--r-- 1 ubuntu ubuntu 46 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/java/ceph-s3-upload/ceph-s3-upload.txt 2024-03-27T21:42:43.410 INFO:teuthology.orchestra.run.smithi060.stdout: 403955 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/java/ceph-s3-upload/src 2024-03-27T21:42:43.410 INFO:teuthology.orchestra.run.smithi060.stdout: 403962 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/java/ceph-s3-upload/src/test 2024-03-27T21:42:43.410 INFO:teuthology.orchestra.run.smithi060.stdout: 403963 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/java/ceph-s3-upload/src/test/java 2024-03-27T21:42:43.410 INFO:teuthology.orchestra.run.smithi060.stdout: 403964 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/java/ceph-s3-upload/src/test/java/org 2024-03-27T21:42:43.410 INFO:teuthology.orchestra.run.smithi060.stdout: 403965 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/java/ceph-s3-upload/src/test/java/org/example 2024-03-27T21:42:43.410 INFO:teuthology.orchestra.run.smithi060.stdout: 403966 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/java/ceph-s3-upload/src/test/java/org/example/cephs3upload 2024-03-27T21:42:43.410 INFO:teuthology.orchestra.run.smithi060.stdout: 403967 4 -rw-r--r-- 1 ubuntu ubuntu 652 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/java/ceph-s3-upload/src/test/java/org/example/cephs3upload/AppTest.java 2024-03-27T21:42:43.411 INFO:teuthology.orchestra.run.smithi060.stdout: 403956 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/java/ceph-s3-upload/src/main 2024-03-27T21:42:43.411 INFO:teuthology.orchestra.run.smithi060.stdout: 403957 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/java/ceph-s3-upload/src/main/java 2024-03-27T21:42:43.411 INFO:teuthology.orchestra.run.smithi060.stdout: 403958 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/java/ceph-s3-upload/src/main/java/org 2024-03-27T21:42:43.411 INFO:teuthology.orchestra.run.smithi060.stdout: 403959 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/java/ceph-s3-upload/src/main/java/org/example 2024-03-27T21:42:43.411 INFO:teuthology.orchestra.run.smithi060.stdout: 403960 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/java/ceph-s3-upload/src/main/java/org/example/cephs3upload 2024-03-27T21:42:43.411 INFO:teuthology.orchestra.run.smithi060.stdout: 403961 4 -rw-r--r-- 1 ubuntu ubuntu 2294 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/java/ceph-s3-upload/src/main/java/org/example/cephs3upload/App.java 2024-03-27T21:42:43.411 INFO:teuthology.orchestra.run.smithi060.stdout: 403954 4 -rw-r--r-- 1 ubuntu ubuntu 2197 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/java/ceph-s3-upload/pom.xml 2024-03-27T21:42:43.411 INFO:teuthology.orchestra.run.smithi060.stdout: 403987 4 -rw-r--r-- 1 ubuntu ubuntu 3813 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/rgw_admin_curl.sh 2024-03-27T21:42:43.412 INFO:teuthology.orchestra.run.smithi060.stdout: 403927 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/boto3 2024-03-27T21:42:43.412 INFO:teuthology.orchestra.run.smithi060.stdout: 403928 8 -rw-r--r-- 1 ubuntu ubuntu 4208 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/boto3/README.md 2024-03-27T21:42:43.412 INFO:teuthology.orchestra.run.smithi060.stdout: 403935 12 -rw-r--r-- 1 ubuntu ubuntu 9059 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/boto3/service-2.sdk-extras.json 2024-03-27T21:42:43.412 INFO:teuthology.orchestra.run.smithi060.stdout: 403932 4 -rwxr-xr-x 1 ubuntu ubuntu 449 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/boto3/get_usage_stats.py 2024-03-27T21:42:43.412 INFO:teuthology.orchestra.run.smithi060.stdout: 403930 4 -rwxr-xr-x 1 ubuntu ubuntu 1229 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/boto3/delete_notification.py 2024-03-27T21:42:43.412 INFO:teuthology.orchestra.run.smithi060.stdout: 403929 4 -rwxr-xr-x 1 ubuntu ubuntu 1241 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/boto3/append_object.py 2024-03-27T21:42:43.412 INFO:teuthology.orchestra.run.smithi060.stdout: 403936 4 -rw-r--r-- 1 ubuntu ubuntu 713 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/boto3/topic_attributes.py 2024-03-27T21:42:43.412 INFO:teuthology.orchestra.run.smithi060.stdout: 403937 4 -rw-r--r-- 1 ubuntu ubuntu 970 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/boto3/topic_with_endpoint.py 2024-03-27T21:42:43.412 INFO:teuthology.orchestra.run.smithi060.stdout: 403933 4 -rwxr-xr-x 1 ubuntu ubuntu 640 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/boto3/list_unordered.py 2024-03-27T21:42:43.413 INFO:teuthology.orchestra.run.smithi060.stdout: 403934 4 -rwxr-xr-x 1 ubuntu ubuntu 1775 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/boto3/notification_filters.py 2024-03-27T21:42:43.413 INFO:teuthology.orchestra.run.smithi060.stdout: 403931 4 -rwxr-xr-x 1 ubuntu ubuntu 826 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/rgw/boto3/get_notification.py 2024-03-27T21:42:43.413 INFO:teuthology.orchestra.run.smithi060.stdout: 403917 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/librbd 2024-03-27T21:42:43.413 INFO:teuthology.orchestra.run.smithi060.stdout: 403918 4 -rw-r--r-- 1 ubuntu ubuntu 699 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/librbd/Makefile 2024-03-27T21:42:43.413 INFO:teuthology.orchestra.run.smithi060.stdout: 403919 8 -rw-r--r-- 1 ubuntu ubuntu 6157 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/examples/librbd/hello_world.cc 2024-03-27T21:42:43.413 INFO:teuthology.orchestra.run.smithi060.stdout: 402895 24 -rw-r--r-- 1 ubuntu ubuntu 24500 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/CMakeLists.txt 2024-03-27T21:42:43.413 INFO:teuthology.orchestra.run.smithi060.stdout: 402857 4 -rw-r--r-- 1 ubuntu ubuntu 55 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.gitattributes 2024-03-27T21:42:43.413 INFO:teuthology.orchestra.run.smithi060.stdout: 402889 4 -rw-r--r-- 1 ubuntu ubuntu 2698 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.gitmodules 2024-03-27T21:42:43.414 INFO:teuthology.orchestra.run.smithi060.stdout: 403991 24 -rwxr-xr-x 1 ubuntu ubuntu 22428 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/install-deps.sh 2024-03-27T21:42:43.414 INFO:teuthology.orchestra.run.smithi060.stdout: 402893 4 -rw-r--r-- 1 ubuntu ubuntu 446 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.readthedocs.yml 2024-03-27T21:42:43.414 INFO:teuthology.orchestra.run.smithi060.stdout: 403988 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/fusetrace 2024-03-27T21:42:43.414 INFO:teuthology.orchestra.run.smithi060.stdout: 403989 4 -rw-r--r-- 1 ubuntu ubuntu 180 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/fusetrace/Makefile 2024-03-27T21:42:43.414 INFO:teuthology.orchestra.run.smithi060.stdout: 403990 28 -rw-r--r-- 1 ubuntu ubuntu 28326 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/fusetrace/fusetrace_ll.cc 2024-03-27T21:42:43.414 INFO:teuthology.orchestra.run.smithi060.stdout: 402922 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/ceph-menv 2024-03-27T21:42:43.414 INFO:teuthology.orchestra.run.smithi060.stdout: 402928 4 -rwxr-xr-x 1 ubuntu ubuntu 533 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/ceph-menv/mset.sh 2024-03-27T21:42:43.414 INFO:teuthology.orchestra.run.smithi060.stdout: 402925 4 -rw-r--r-- 1 ubuntu ubuntu 399 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/ceph-menv/README 2024-03-27T21:42:43.415 INFO:teuthology.orchestra.run.smithi060.stdout: 402923 4 -rw-r--r-- 1 ubuntu ubuntu 326 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/ceph-menv/.menvrc 2024-03-27T21:42:43.415 INFO:teuthology.orchestra.run.smithi060.stdout: 402927 4 -rwxr-xr-x 1 ubuntu ubuntu 327 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/ceph-menv/mdo.sh 2024-03-27T21:42:43.415 INFO:teuthology.orchestra.run.smithi060.stdout: 402924 4 -rw-r--r-- 1 ubuntu ubuntu 427 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/ceph-menv/INSTALL 2024-03-27T21:42:43.415 INFO:teuthology.orchestra.run.smithi060.stdout: 402926 4 -rwxr-xr-x 1 ubuntu ubuntu 246 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/ceph-menv/build_links.sh 2024-03-27T21:42:43.415 INFO:teuthology.orchestra.run.smithi060.stdout: 417159 12 -rwxr-xr-x 1 ubuntu ubuntu 10199 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/win32_build.sh 2024-03-27T21:42:43.415 INFO:teuthology.orchestra.run.smithi060.stdout: 403127 4 drwxr-xr-x 27 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc 2024-03-27T21:42:43.415 INFO:teuthology.orchestra.run.smithi060.stdout: 403331 8 -rw-r--r-- 1 ubuntu ubuntu 7640 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/conf.py 2024-03-27T21:42:43.415 INFO:teuthology.orchestra.run.smithi060.stdout: 403130 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/README.md 2024-03-27T21:42:43.415 INFO:teuthology.orchestra.run.smithi060.stdout: 403619 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr 2024-03-27T21:42:43.416 INFO:teuthology.orchestra.run.smithi060.stdout: 403621 4 -rw-r--r-- 1 ubuntu ubuntu 1899 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/alerts.rst 2024-03-27T21:42:43.416 INFO:teuthology.orchestra.run.smithi060.stdout: 403644 32 -rw-r--r-- 1 ubuntu ubuntu 29630 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/modules.rst 2024-03-27T21:42:43.416 INFO:teuthology.orchestra.run.smithi060.stdout: 403656 4 -rw-r--r-- 1 ubuntu ubuntu 3604 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/zabbix.rst 2024-03-27T21:42:43.416 INFO:teuthology.orchestra.run.smithi060.stdout: 403640 24 -rw-r--r-- 1 ubuntu ubuntu 22189 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/inventory-card.png 2024-03-27T21:42:43.416 INFO:teuthology.orchestra.run.smithi060.stdout: 403643 4 -rw-r--r-- 1 ubuntu ubuntu 802 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/mds_autoscaler.rst 2024-03-27T21:42:43.416 INFO:teuthology.orchestra.run.smithi060.stdout: 403635 4 -rw-r--r-- 1 ubuntu ubuntu 1656 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/diskprediction.rst 2024-03-27T21:42:43.416 INFO:teuthology.orchestra.run.smithi060.stdout: 403630 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/dashboard_plugins 2024-03-27T21:42:43.416 INFO:teuthology.orchestra.run.smithi060.stdout: 403633 4 -rw-r--r-- 1 ubuntu ubuntu 1179 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/dashboard_plugins/motd.inc.rst 2024-03-27T21:42:43.416 INFO:teuthology.orchestra.run.smithi060.stdout: 403632 4 -rw-r--r-- 1 ubuntu ubuntu 1632 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/dashboard_plugins/feature_toggles.inc.rst 2024-03-27T21:42:43.417 INFO:teuthology.orchestra.run.smithi060.stdout: 403631 4 -rw-r--r-- 1 ubuntu ubuntu 1004 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/dashboard_plugins/debug.inc.rst 2024-03-27T21:42:43.417 INFO:teuthology.orchestra.run.smithi060.stdout: 403627 4 -rw-r--r-- 1 ubuntu ubuntu 2614 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/crash.rst 2024-03-27T21:42:43.417 INFO:teuthology.orchestra.run.smithi060.stdout: 403647 12 -rw-r--r-- 1 ubuntu ubuntu 9234 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/orchestrator_modules.rst 2024-03-27T21:42:43.417 INFO:teuthology.orchestra.run.smithi060.stdout: 403655 12 -rw-r--r-- 1 ubuntu ubuntu 10676 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/telemetry.rst 2024-03-27T21:42:43.417 INFO:teuthology.orchestra.run.smithi060.stdout: 403646 12 -rw-r--r-- 1 ubuntu ubuntu 8997 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/orchestrator.rst 2024-03-27T21:42:43.417 INFO:teuthology.orchestra.run.smithi060.stdout: 403653 8 -rw-r--r-- 1 ubuntu ubuntu 6515 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/status-card-open.png 2024-03-27T21:42:43.417 INFO:teuthology.orchestra.run.smithi060.stdout: 403645 28 -rw-r--r-- 1 ubuntu ubuntu 24703 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/nfs.rst 2024-03-27T21:42:43.417 INFO:teuthology.orchestra.run.smithi060.stdout: 403629 60 -rw-r--r-- 1 ubuntu ubuntu 57573 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/dashboard.rst 2024-03-27T21:42:43.418 INFO:teuthology.orchestra.run.smithi060.stdout: 403642 4 -rw-r--r-- 1 ubuntu ubuntu 1105 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/localpool.rst 2024-03-27T21:42:43.418 INFO:teuthology.orchestra.run.smithi060.stdout: 403639 4 -rw-r--r-- 1 ubuntu ubuntu 1857 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/insights.rst 2024-03-27T21:42:43.418 INFO:teuthology.orchestra.run.smithi060.stdout: 403652 4 -rw-r--r-- 1 ubuntu ubuntu 989 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/rook.rst 2024-03-27T21:42:43.418 INFO:teuthology.orchestra.run.smithi060.stdout: 403641 4 -rw-r--r-- 1 ubuntu ubuntu 582 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/iostat.rst 2024-03-27T21:42:43.418 INFO:teuthology.orchestra.run.smithi060.stdout: 403636 4 -rw-r--r-- 1 ubuntu ubuntu 730 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/hello.rst 2024-03-27T21:42:43.418 INFO:teuthology.orchestra.run.smithi060.stdout: 403638 12 -rw-r--r-- 1 ubuntu ubuntu 8208 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/influx.rst 2024-03-27T21:42:43.418 INFO:teuthology.orchestra.run.smithi060.stdout: 403620 8 -rw-r--r-- 1 ubuntu ubuntu 5810 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/administrator.rst 2024-03-27T21:42:43.418 INFO:teuthology.orchestra.run.smithi060.stdout: 403628 140 -rw-r--r-- 1 ubuntu ubuntu 142227 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/dashboard-landing-page.png 2024-03-27T21:42:43.419 INFO:teuthology.orchestra.run.smithi060.stdout: 403651 4 -rw-r--r-- 1 ubuntu ubuntu 3676 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/rgw.rst 2024-03-27T21:42:43.419 INFO:teuthology.orchestra.run.smithi060.stdout: 403634 20 -rw-r--r-- 1 ubuntu ubuntu 16497 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/details-card.png 2024-03-27T21:42:43.419 INFO:teuthology.orchestra.run.smithi060.stdout: 403648 4 -rw-r--r-- 1 ubuntu ubuntu 1355 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/progress.rst 2024-03-27T21:42:43.419 INFO:teuthology.orchestra.run.smithi060.stdout: 403650 8 -rw-r--r-- 1 ubuntu ubuntu 6343 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/restful.rst 2024-03-27T21:42:43.419 INFO:teuthology.orchestra.run.smithi060.stdout: 403637 4 -rw-r--r-- 1 ubuntu ubuntu 1810 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/index.rst 2024-03-27T21:42:43.419 INFO:teuthology.orchestra.run.smithi060.stdout: 403622 20 -rw-r--r-- 1 ubuntu ubuntu 17736 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/capacity-card.png 2024-03-27T21:42:43.419 INFO:teuthology.orchestra.run.smithi060.stdout: 403649 16 -rw-r--r-- 1 ubuntu ubuntu 15134 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/prometheus.rst 2024-03-27T21:42:43.419 INFO:teuthology.orchestra.run.smithi060.stdout: 403626 68 -rw-r--r-- 1 ubuntu ubuntu 68915 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/cluster-utilization-card.png 2024-03-27T21:42:43.420 INFO:teuthology.orchestra.run.smithi060.stdout: 403625 4 -rw-r--r-- 1 ubuntu ubuntu 843 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/cli_api.rst 2024-03-27T21:42:43.420 INFO:teuthology.orchestra.run.smithi060.stdout: 403654 4 -rw-r--r-- 1 ubuntu ubuntu 2286 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/telegraf.rst 2024-03-27T21:42:43.420 INFO:teuthology.orchestra.run.smithi060.stdout: 403623 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/ceph_api 2024-03-27T21:42:43.420 INFO:teuthology.orchestra.run.smithi060.stdout: 403624 4 -rw-r--r-- 1 ubuntu ubuntu 3842 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mgr/ceph_api/index.rst 2024-03-27T21:42:43.420 INFO:teuthology.orchestra.run.smithi060.stdout: 403883 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/security 2024-03-27T21:42:43.420 INFO:teuthology.orchestra.run.smithi060.stdout: 403891 4 -rw-r--r-- 1 ubuntu ubuntu 2798 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/security/process.rst 2024-03-27T21:42:43.422 INFO:teuthology.orchestra.run.smithi191.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:42 /home/ubuntu/cephtest 2024-03-27T21:42:43.481 INFO:teuthology.orchestra.run.smithi060.stdout: 403884 8 -rw-r--r-- 1 ubuntu ubuntu 6913 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/security/CVE-2021-20288.rst 2024-03-27T21:42:43.481 INFO:teuthology.orchestra.run.smithi060.stdout: 403888 4 -rw-r--r-- 1 ubuntu ubuntu 1382 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/security/CVE-2022-0670.rst 2024-03-27T21:42:43.482 INFO:teuthology.orchestra.run.smithi060.stdout: 403887 4 -rw-r--r-- 1 ubuntu ubuntu 596 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/security/CVE-2021-3531.rst 2024-03-27T21:42:43.482 INFO:teuthology.orchestra.run.smithi060.stdout: 403885 4 -rw-r--r-- 1 ubuntu ubuntu 576 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/security/CVE-2021-3509.rst 2024-03-27T21:42:43.482 INFO:teuthology.orchestra.run.smithi060.stdout: 403889 12 -rw-r--r-- 1 ubuntu ubuntu 8635 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/security/cves.rst 2024-03-27T21:42:43.482 INFO:teuthology.orchestra.run.smithi060.stdout: 403890 4 -rw-r--r-- 1 ubuntu ubuntu 1288 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/security/index.rst 2024-03-27T21:42:43.482 INFO:teuthology.orchestra.run.smithi060.stdout: 403886 4 -rw-r--r-- 1 ubuntu ubuntu 804 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/security/CVE-2021-3524.rst 2024-03-27T21:42:43.482 INFO:teuthology.orchestra.run.smithi060.stdout: 403205 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs 2024-03-27T21:42:43.482 INFO:teuthology.orchestra.run.smithi060.stdout: 403229 8 -rw-r--r-- 1 ubuntu ubuntu 7532 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/eviction.rst 2024-03-27T21:42:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: 403249 16 -rw-r--r-- 1 ubuntu ubuntu 12400 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/multimds.rst 2024-03-27T21:42:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: 403257 8 -rw-r--r-- 1 ubuntu ubuntu 7272 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/standby.rst 2024-03-27T21:42:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: 403224 8 -rw-r--r-- 1 ubuntu ubuntu 5806 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/createfs.rst 2024-03-27T21:42:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: 403236 4 -rw-r--r-- 1 ubuntu ubuntu 158 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/journaler.rst 2024-03-27T21:42:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: 403255 8 -rw-r--r-- 1 ubuntu ubuntu 5069 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/scrub.rst 2024-03-27T21:42:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: 403217 4 -rw-r--r-- 1 ubuntu ubuntu 2493 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/cephfs-io-path.rst 2024-03-27T21:42:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: 403214 8 -rw-r--r-- 1 ubuntu ubuntu 7770 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/capabilities.rst 2024-03-27T21:42:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: 403259 20 -rw-r--r-- 1 ubuntu ubuntu 16842 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/troubleshooting.rst 2024-03-27T21:42:43.483 INFO:teuthology.orchestra.run.smithi060.stdout: 403256 12 -rw-r--r-- 1 ubuntu ubuntu 9664 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/snap-schedule.rst 2024-03-27T21:42:43.484 INFO:teuthology.orchestra.run.smithi060.stdout: 403219 16 -rw-r--r-- 1 ubuntu ubuntu 16081 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/cephfs-mirroring.rst 2024-03-27T21:42:43.484 INFO:teuthology.orchestra.run.smithi060.stdout: 403247 8 -rw-r--r-- 1 ubuntu ubuntu 6444 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/mount-using-kernel-driver.rst 2024-03-27T21:42:43.484 INFO:teuthology.orchestra.run.smithi060.stdout: 403220 40 -rw-r--r-- 1 ubuntu ubuntu 37292 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/cephfs-top.png 2024-03-27T21:42:43.484 INFO:teuthology.orchestra.run.smithi060.stdout: 403260 4 -rw-r--r-- 1 ubuntu ubuntu 2993 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/upgrading.rst 2024-03-27T21:42:43.484 INFO:teuthology.orchestra.run.smithi060.stdout: 403227 4 -rw-r--r-- 1 ubuntu ubuntu 2222 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/disaster-recovery.rst 2024-03-27T21:42:43.484 INFO:teuthology.orchestra.run.smithi060.stdout: 403228 8 -rw-r--r-- 1 ubuntu ubuntu 4888 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/dynamic-metadata-management.rst 2024-03-27T21:42:43.484 INFO:teuthology.orchestra.run.smithi060.stdout: 403212 4 -rw-r--r-- 1 ubuntu ubuntu 3349 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/app-best-practices.rst 2024-03-27T21:42:43.484 INFO:teuthology.orchestra.run.smithi060.stdout: 403207 16 -rw-r--r-- 1 ubuntu ubuntu 13282 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/administration.rst 2024-03-27T21:42:43.484 INFO:teuthology.orchestra.run.smithi060.stdout: 403239 12 -rw-r--r-- 1 ubuntu ubuntu 11627 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/mantle.rst 2024-03-27T21:42:43.485 INFO:teuthology.orchestra.run.smithi060.stdout: 403245 4 -rw-r--r-- 1 ubuntu ubuntu 3235 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/mount-prerequisites.rst 2024-03-27T21:42:43.485 INFO:teuthology.orchestra.run.smithi060.stdout: 403231 12 -rw-r--r-- 1 ubuntu ubuntu 10525 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/file-layouts.rst 2024-03-27T21:42:43.485 INFO:teuthology.orchestra.run.smithi060.stdout: 403223 4 -rw-r--r-- 1 ubuntu ubuntu 2896 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/client-config-ref.rst 2024-03-27T21:42:43.485 INFO:teuthology.orchestra.run.smithi060.stdout: 403213 12 -rw-r--r-- 1 ubuntu ubuntu 8733 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/cache-configuration.rst 2024-03-27T21:42:43.485 INFO:teuthology.orchestra.run.smithi060.stdout: 403226 12 -rw-r--r-- 1 ubuntu ubuntu 11004 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/disaster-recovery-experts.rst 2024-03-27T21:42:43.485 INFO:teuthology.orchestra.run.smithi060.stdout: 403250 4 -rw-r--r-- 1 ubuntu ubuntu 3700 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/nfs.rst 2024-03-27T21:42:43.485 INFO:teuthology.orchestra.run.smithi060.stdout: 403248 4 -rw-r--r-- 1 ubuntu ubuntu 1954 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/multifs.rst 2024-03-27T21:42:43.485 INFO:teuthology.orchestra.run.smithi060.stdout: 403251 8 -rw-r--r-- 1 ubuntu ubuntu 5310 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/posix.rst 2024-03-27T21:42:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: 403232 48 -rw-r--r-- 1 ubuntu ubuntu 45765 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/fs-volumes.rst 2024-03-27T21:42:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: 403216 108 -rw-r--r-- 1 ubuntu ubuntu 108799 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/cephfs-architecture.svg 2024-03-27T21:42:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: 403233 4 -rw-r--r-- 1 ubuntu ubuntu 2676 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/full.rst 2024-03-27T21:42:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: 403246 4 -rw-r--r-- 1 ubuntu ubuntu 3596 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/mount-using-fuse.rst 2024-03-27T21:42:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: 403253 8 -rw-r--r-- 1 ubuntu ubuntu 5260 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/quota.rst 2024-03-27T21:42:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: 403242 8 -rw-r--r-- 1 ubuntu ubuntu 7040 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/mds-journaling.rst 2024-03-27T21:42:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: 403254 4 -rw-r--r-- 1 ubuntu ubuntu 2270 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/recover-fs-after-mon-store-loss.rst 2024-03-27T21:42:43.486 INFO:teuthology.orchestra.run.smithi060.stdout: 403244 8 -rw-r--r-- 1 ubuntu ubuntu 5508 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/mds-states.rst 2024-03-27T21:42:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: 403258 256 -rw-r--r-- 1 ubuntu ubuntu 261223 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/subtree-partitioning.svg 2024-03-27T21:42:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: 403243 4 -rw-r--r-- 1 ubuntu ubuntu 2215 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/mds-state-diagram.dot 2024-03-27T21:42:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: 403237 4 -rw-r--r-- 1 ubuntu ubuntu 1716 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/kernel-features.rst 2024-03-27T21:42:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: 403230 4 -rw-r--r-- 1 ubuntu ubuntu 1822 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/experimental-features.rst 2024-03-27T21:42:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: 403241 4 -rw-r--r-- 1 ubuntu ubuntu 2191 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/mds-config-ref.rst 2024-03-27T21:42:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: 403225 8 -rw-r--r-- 1 ubuntu ubuntu 4584 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/dirfrags.rst 2024-03-27T21:42:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: 403235 8 -rw-r--r-- 1 ubuntu ubuntu 5943 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/index.rst 2024-03-27T21:42:43.487 INFO:teuthology.orchestra.run.smithi060.stdout: 403252 160 -rw-r--r-- 1 ubuntu ubuntu 159819 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/quiesce-set-states.svg 2024-03-27T21:42:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: 403222 16 -rw-r--r-- 1 ubuntu ubuntu 14852 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/client-auth.rst 2024-03-27T21:42:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: 403206 8 -rw-r--r-- 1 ubuntu ubuntu 5312 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/add-remove-mds.rst 2024-03-27T21:42:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: 403218 8 -rw-r--r-- 1 ubuntu ubuntu 7115 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/cephfs-journal-tool.rst 2024-03-27T21:42:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: 403234 12 -rw-r--r-- 1 ubuntu ubuntu 10456 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/health-messages.rst 2024-03-27T21:42:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: 403221 8 -rw-r--r-- 1 ubuntu ubuntu 5886 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/cephfs-top.rst 2024-03-27T21:42:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: 403240 4 -rw-r--r-- 1 ubuntu ubuntu 3770 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/mdcache.rst 2024-03-27T21:42:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: 403215 4 -rw-r--r-- 1 ubuntu ubuntu 3152 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/ceph-dokan.rst 2024-03-27T21:42:43.488 INFO:teuthology.orchestra.run.smithi060.stdout: 403208 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/api 2024-03-27T21:42:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: 403211 4 -rw-r--r-- 1 ubuntu ubuntu 236 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/api/libcephfs-py.rst 2024-03-27T21:42:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: 403209 4 -rw-r--r-- 1 ubuntu ubuntu 161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/api/index.rst 2024-03-27T21:42:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: 403210 4 -rw-r--r-- 1 ubuntu ubuntu 501 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/api/libcephfs-java.rst 2024-03-27T21:42:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: 403238 4 -rw-r--r-- 1 ubuntu ubuntu 3188 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephfs/lazyio.rst 2024-03-27T21:42:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: 403662 4 drwxr-xr-x 8 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados 2024-03-27T21:42:43.489 INFO:teuthology.orchestra.run.smithi060.stdout: 403671 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/command 2024-03-27T21:42:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: 403673 4 -rw-r--r-- 1 ubuntu ubuntu 2012 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/command/list-inconsistent-snap.json 2024-03-27T21:42:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: 403672 8 -rw-r--r-- 1 ubuntu ubuntu 7173 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/command/list-inconsistent-obj.json 2024-03-27T21:42:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: 403674 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/configuration 2024-03-27T21:42:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: 403679 4 -rw-r--r-- 1 ubuntu ubuntu 1165 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/configuration/demo-ceph.conf 2024-03-27T21:42:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: 403676 28 -rw-r--r-- 1 ubuntu ubuntu 25230 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/configuration/bluestore-config-ref.rst 2024-03-27T21:42:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: 403686 4 -rw-r--r-- 1 ubuntu ubuntu 2142 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/configuration/mon-lookup-dns.rst 2024-03-27T21:42:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: 403693 8 -rw-r--r-- 1 ubuntu ubuntu 4186 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/configuration/storage-devices.rst 2024-03-27T21:42:43.490 INFO:teuthology.orchestra.run.smithi060.stdout: 403687 12 -rw-r--r-- 1 ubuntu ubuntu 10422 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/configuration/mon-osd-interaction.rst 2024-03-27T21:42:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: 403681 4 -rw-r--r-- 1 ubuntu ubuntu 553 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/configuration/general-config-ref.rst 2024-03-27T21:42:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: 403692 4 -rw-r--r-- 1 ubuntu ubuntu 1114 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/configuration/pool-pg.conf 2024-03-27T21:42:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: 403678 8 -rw-r--r-- 1 ubuntu ubuntu 7195 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/configuration/common.rst 2024-03-27T21:42:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: 403675 16 -rw-r--r-- 1 ubuntu ubuntu 12447 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/configuration/auth-config-ref.rst 2024-03-27T21:42:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: 403684 28 -rw-r--r-- 1 ubuntu ubuntu 28540 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/configuration/mclock-config-ref.rst 2024-03-27T21:42:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: 403690 20 -rw-r--r-- 1 ubuntu ubuntu 17092 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/configuration/osd-config-ref.rst 2024-03-27T21:42:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: 403689 16 -rw-r--r-- 1 ubuntu ubuntu 14567 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/configuration/network-config-ref.rst 2024-03-27T21:42:43.491 INFO:teuthology.orchestra.run.smithi060.stdout: 403683 4 -rw-r--r-- 1 ubuntu ubuntu 1955 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/configuration/journal-ref.rst 2024-03-27T21:42:43.492 INFO:teuthology.orchestra.run.smithi060.stdout: 403688 12 -rw-r--r-- 1 ubuntu ubuntu 9603 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/configuration/msgr2.rst 2024-03-27T21:42:43.492 INFO:teuthology.orchestra.run.smithi060.stdout: 403685 28 -rw-r--r-- 1 ubuntu ubuntu 27796 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/configuration/mon-config-ref.rst 2024-03-27T21:42:43.492 INFO:teuthology.orchestra.run.smithi060.stdout: 403682 4 -rw-r--r-- 1 ubuntu ubuntu 1487 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/configuration/index.rst 2024-03-27T21:42:43.492 INFO:teuthology.orchestra.run.smithi060.stdout: 403680 12 -rw-r--r-- 1 ubuntu ubuntu 10261 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/configuration/filestore-config-ref.rst 2024-03-27T21:42:43.492 INFO:teuthology.orchestra.run.smithi060.stdout: 403691 4 -rw-r--r-- 1 ubuntu ubuntu 2529 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/configuration/pool-pg-config-ref.rst 2024-03-27T21:42:43.492 INFO:teuthology.orchestra.run.smithi060.stdout: 403677 28 -rw-r--r-- 1 ubuntu ubuntu 24776 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/configuration/ceph-conf.rst 2024-03-27T21:42:43.492 INFO:teuthology.orchestra.run.smithi060.stdout: 403732 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/troubleshooting 2024-03-27T21:42:43.492 INFO:teuthology.orchestra.run.smithi060.stdout: 403740 28 -rw-r--r-- 1 ubuntu ubuntu 27284 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/troubleshooting/troubleshooting-pg.rst 2024-03-27T21:42:43.492 INFO:teuthology.orchestra.run.smithi060.stdout: 403738 32 -rw-r--r-- 1 ubuntu ubuntu 32396 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/troubleshooting/troubleshooting-mon.rst 2024-03-27T21:42:43.493 INFO:teuthology.orchestra.run.smithi060.stdout: 403739 32 -rw-r--r-- 1 ubuntu ubuntu 31618 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/troubleshooting/troubleshooting-osd.rst 2024-03-27T21:42:43.493 INFO:teuthology.orchestra.run.smithi060.stdout: 403736 20 -rw-r--r-- 1 ubuntu ubuntu 18722 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/troubleshooting/log-and-debug.rst 2024-03-27T21:42:43.493 INFO:teuthology.orchestra.run.smithi060.stdout: 403735 4 -rw-r--r-- 1 ubuntu ubuntu 484 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/troubleshooting/index.rst 2024-03-27T21:42:43.493 INFO:teuthology.orchestra.run.smithi060.stdout: 403737 8 -rw-r--r-- 1 ubuntu ubuntu 6536 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/troubleshooting/memory-profiling.rst 2024-03-27T21:42:43.493 INFO:teuthology.orchestra.run.smithi060.stdout: 403733 4 -rw-r--r-- 1 ubuntu ubuntu 1305 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/troubleshooting/community.rst 2024-03-27T21:42:43.493 INFO:teuthology.orchestra.run.smithi060.stdout: 403734 4 -rw-r--r-- 1 ubuntu ubuntu 1723 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/troubleshooting/cpu-profiling.rst 2024-03-27T21:42:43.493 INFO:teuthology.orchestra.run.smithi060.stdout: 403697 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations 2024-03-27T21:42:43.493 INFO:teuthology.orchestra.run.smithi060.stdout: 403728 4 -rw-r--r-- 1 ubuntu ubuntu 3407 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/read-balancer.rst 2024-03-27T21:42:43.494 INFO:teuthology.orchestra.run.smithi060.stdout: 403703 4 -rw-r--r-- 1 ubuntu ubuntu 3418 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/change-mon-elections.rst 2024-03-27T21:42:43.494 INFO:teuthology.orchestra.run.smithi060.stdout: 403706 48 -rw-r--r-- 1 ubuntu ubuntu 45725 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/crush-map.rst 2024-03-27T21:42:43.494 INFO:teuthology.orchestra.run.smithi060.stdout: 403707 4 -rw-r--r-- 1 ubuntu ubuntu 2490 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/data-placement.rst 2024-03-27T21:42:43.494 INFO:teuthology.orchestra.run.smithi060.stdout: 403714 4 -rw-r--r-- 1 ubuntu ubuntu 4051 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/erasure-code-shec.rst 2024-03-27T21:42:43.494 INFO:teuthology.orchestra.run.smithi060.stdout: 403722 8 -rw-r--r-- 1 ubuntu ubuntu 4678 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/pg-repair.rst 2024-03-27T21:42:43.494 INFO:teuthology.orchestra.run.smithi060.stdout: 403720 8 -rw-r--r-- 1 ubuntu ubuntu 6056 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/operating.rst 2024-03-27T21:42:43.494 INFO:teuthology.orchestra.run.smithi060.stdout: 403719 24 -rw-r--r-- 1 ubuntu ubuntu 22583 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/monitoring.rst 2024-03-27T21:42:43.494 INFO:teuthology.orchestra.run.smithi060.stdout: 403709 12 -rw-r--r-- 1 ubuntu ubuntu 8682 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/erasure-code-clay.rst 2024-03-27T21:42:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: 403699 16 -rw-r--r-- 1 ubuntu ubuntu 15321 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/add-or-rm-osds.rst 2024-03-27T21:42:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: 403711 4 -rw-r--r-- 1 ubuntu ubuntu 3493 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/erasure-code-jerasure.rst 2024-03-27T21:42:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: 403721 4 -rw-r--r-- 1 ubuntu ubuntu 4090 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/pg-concepts.rst 2024-03-27T21:42:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: 403708 8 -rw-r--r-- 1 ubuntu ubuntu 7435 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/devices.rst 2024-03-27T21:42:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: 403698 24 -rw-r--r-- 1 ubuntu ubuntu 20986 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/add-or-rm-mons.rst 2024-03-27T21:42:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: 403715 16 -rw-r--r-- 1 ubuntu ubuntu 13377 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/erasure-code.rst 2024-03-27T21:42:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: 403705 32 -rw-r--r-- 1 ubuntu ubuntu 30560 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/crush-map-edits.rst 2024-03-27T21:42:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: 403716 56 -rw-r--r-- 1 ubuntu ubuntu 56962 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/health-checks.rst 2024-03-27T21:42:43.495 INFO:teuthology.orchestra.run.smithi060.stdout: 403700 12 -rw-r--r-- 1 ubuntu ubuntu 8877 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/balancer.rst 2024-03-27T21:42:43.496 INFO:teuthology.orchestra.run.smithi060.stdout: 403713 8 -rw-r--r-- 1 ubuntu ubuntu 4106 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/erasure-code-profile.rst 2024-03-27T21:42:43.496 INFO:teuthology.orchestra.run.smithi060.stdout: 403704 20 -rw-r--r-- 1 ubuntu ubuntu 18581 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/control.rst 2024-03-27T21:42:43.496 INFO:teuthology.orchestra.run.smithi060.stdout: 403726 40 -rw-r--r-- 1 ubuntu ubuntu 38795 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/placement-groups.rst 2024-03-27T21:42:43.496 INFO:teuthology.orchestra.run.smithi060.stdout: 403729 12 -rw-r--r-- 1 ubuntu ubuntu 11911 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/stretch-mode.rst 2024-03-27T21:42:43.496 INFO:teuthology.orchestra.run.smithi060.stdout: 403712 12 -rw-r--r-- 1 ubuntu ubuntu 11708 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/erasure-code-lrc.rst 2024-03-27T21:42:43.496 INFO:teuthology.orchestra.run.smithi060.stdout: 403727 24 -rw-r--r-- 1 ubuntu ubuntu 22594 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/pools.rst 2024-03-27T21:42:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: 403710 4 -rw-r--r-- 1 ubuntu ubuntu 2723 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/erasure-code-isa.rst 2024-03-27T21:42:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: 403730 4 -rw-r--r-- 1 ubuntu ubuntu 3985 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/upmap.rst 2024-03-27T21:42:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: 403717 4 -rw-r--r-- 1 ubuntu ubuntu 2273 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/index.rst 2024-03-27T21:42:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: 403702 20 -rw-r--r-- 1 ubuntu ubuntu 19025 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/cache-tiering.rst 2024-03-27T21:42:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: 403723 4 -rw-r--r-- 1 ubuntu ubuntu 3746 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/pg-states.rst 2024-03-27T21:42:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: 403731 36 -rw-r--r-- 1 ubuntu ubuntu 34330 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/user-management.rst 2024-03-27T21:42:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: 403718 24 -rw-r--r-- 1 ubuntu ubuntu 23530 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/monitoring-osd-pg.rst 2024-03-27T21:42:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: 403701 12 -rw-r--r-- 1 ubuntu ubuntu 12195 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/bluestore-migration.rst 2024-03-27T21:42:43.497 INFO:teuthology.orchestra.run.smithi060.stdout: 403724 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/pgcalc 2024-03-27T21:42:43.498 INFO:teuthology.orchestra.run.smithi060.stdout: 403725 8 -rw-r--r-- 1 ubuntu ubuntu 6003 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/operations/pgcalc/index.rst 2024-03-27T21:42:43.498 INFO:teuthology.orchestra.run.smithi060.stdout: 403694 4 -rw-r--r-- 1 ubuntu ubuntu 2115 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/index.rst 2024-03-27T21:42:43.498 INFO:teuthology.orchestra.run.smithi060.stdout: 403663 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/api 2024-03-27T21:42:43.498 INFO:teuthology.orchestra.run.smithi060.stdout: 403667 8 -rw-r--r-- 1 ubuntu ubuntu 6132 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/api/librados.rst 2024-03-27T21:42:43.498 INFO:teuthology.orchestra.run.smithi060.stdout: 403668 4 -rw-r--r-- 1 ubuntu ubuntu 287 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/api/libradospp.rst 2024-03-27T21:42:43.498 INFO:teuthology.orchestra.run.smithi060.stdout: 403665 20 -rw-r--r-- 1 ubuntu ubuntu 17727 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/api/libcephsqlite.rst 2024-03-27T21:42:43.498 INFO:teuthology.orchestra.run.smithi060.stdout: 403670 16 -rw-r--r-- 1 ubuntu ubuntu 12985 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/api/python.rst 2024-03-27T21:42:43.498 INFO:teuthology.orchestra.run.smithi060.stdout: 403664 4 -rw-r--r-- 1 ubuntu ubuntu 924 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/api/index.rst 2024-03-27T21:42:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: 403669 4 -rw-r--r-- 1 ubuntu ubuntu 1816 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/api/objclass-sdk.rst 2024-03-27T21:42:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: 403666 32 -rw-r--r-- 1 ubuntu ubuntu 29409 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/api/librados-intro.rst 2024-03-27T21:42:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: 403695 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/man 2024-03-27T21:42:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: 403696 4 -rw-r--r-- 1 ubuntu ubuntu 744 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rados/man/index.rst 2024-03-27T21:42:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: 403618 4 -rw-r--r-- 1 ubuntu ubuntu 1136 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man_index.rst 2024-03-27T21:42:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: 403564 4 -rw-r--r-- 1 ubuntu ubuntu 3898 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/logo.png 2024-03-27T21:42:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: 403129 4 -rw-r--r-- 1 ubuntu ubuntu 61 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/CMakeLists.txt 2024-03-27T21:42:43.499 INFO:teuthology.orchestra.run.smithi060.stdout: 403657 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mon 2024-03-27T21:42:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: 403658 4 -rw-r--r-- 1 ubuntu ubuntu 828 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mon/README.txt 2024-03-27T21:42:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: 403659 4 -rw-r--r-- 1 ubuntu ubuntu 2832 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/mon/paxos-call-chain.dot 2024-03-27T21:42:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: 403660 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/monitoring 2024-03-27T21:42:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: 403661 20 -rw-r--r-- 1 ubuntu ubuntu 17390 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/monitoring/index.rst 2024-03-27T21:42:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: 403135 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/_static 2024-03-27T21:42:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: 403138 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/_static/js 2024-03-27T21:42:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: 403139 24 -rw-r--r-- 1 ubuntu ubuntu 20784 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/_static/js/pgcalc.js 2024-03-27T21:42:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: 403136 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/_static/css 2024-03-27T21:42:43.500 INFO:teuthology.orchestra.run.smithi060.stdout: 403137 4 -rw-r--r-- 1 ubuntu ubuntu 354 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/_static/css/custom.css 2024-03-27T21:42:43.501 INFO:teuthology.orchestra.run.smithi060.stdout: 403858 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases 2024-03-27T21:42:43.501 INFO:teuthology.orchestra.run.smithi060.stdout: 403874 588 -rw-r--r-- 1 ubuntu ubuntu 601946 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases/nautilus.rst 2024-03-27T21:42:43.501 INFO:teuthology.orchestra.run.smithi060.stdout: 403860 24 -rw-r--r-- 1 ubuntu ubuntu 23570 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases/bobtail.rst 2024-03-27T21:42:43.501 INFO:teuthology.orchestra.run.smithi060.stdout: 403864 108 -rw-r--r-- 1 ubuntu ubuntu 109407 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases/firefly.rst 2024-03-27T21:42:43.501 INFO:teuthology.orchestra.run.smithi060.stdout: 403863 32 -rw-r--r-- 1 ubuntu ubuntu 30377 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases/emperor.rst 2024-03-27T21:42:43.501 INFO:teuthology.orchestra.run.smithi060.stdout: 403866 72 -rw-r--r-- 1 ubuntu ubuntu 69725 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases/giant.rst 2024-03-27T21:42:43.501 INFO:teuthology.orchestra.run.smithi060.stdout: 403859 12 -rw-r--r-- 1 ubuntu ubuntu 9331 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases/argonaut.rst 2024-03-27T21:42:43.501 INFO:teuthology.orchestra.run.smithi060.stdout: 403876 296 -rw-r--r-- 1 ubuntu ubuntu 301736 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases/pacific.rst 2024-03-27T21:42:43.502 INFO:teuthology.orchestra.run.smithi060.stdout: 403870 420 -rw-r--r-- 1 ubuntu ubuntu 428192 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases/jewel.rst 2024-03-27T21:42:43.502 INFO:teuthology.orchestra.run.smithi060.stdout: 403861 32 -rw-r--r-- 1 ubuntu ubuntu 31410 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases/cuttlefish.rst 2024-03-27T21:42:43.502 INFO:teuthology.orchestra.run.smithi060.stdout: 403877 200 -rw-r--r-- 1 ubuntu ubuntu 200807 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases/quincy.rst 2024-03-27T21:42:43.502 INFO:teuthology.orchestra.run.smithi060.stdout: 403879 16 -rw-r--r-- 1 ubuntu ubuntu 12859 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases/releases.yml 2024-03-27T21:42:43.502 INFO:teuthology.orchestra.run.smithi060.stdout: 403862 44 -rw-r--r-- 1 ubuntu ubuntu 41568 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases/dumpling.rst 2024-03-27T21:42:43.502 INFO:teuthology.orchestra.run.smithi060.stdout: 403867 180 -rw-r--r-- 1 ubuntu ubuntu 181661 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases/hammer.rst 2024-03-27T21:42:43.502 INFO:teuthology.orchestra.run.smithi060.stdout: 403872 720 -rw-r--r-- 1 ubuntu ubuntu 736301 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases/luminous.rst 2024-03-27T21:42:43.502 INFO:teuthology.orchestra.run.smithi060.stdout: 403871 316 -rw-r--r-- 1 ubuntu ubuntu 322789 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases/kraken.rst 2024-03-27T21:42:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: 403869 84 -rw-r--r-- 1 ubuntu ubuntu 85821 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases/infernalis.rst 2024-03-27T21:42:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: 403878 80 -rw-r--r-- 1 ubuntu ubuntu 81661 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases/reef.rst 2024-03-27T21:42:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: 403875 700 -rw-r--r-- 1 ubuntu ubuntu 713217 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases/octopus.rst 2024-03-27T21:42:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: 403868 8 -rw-r--r-- 1 ubuntu ubuntu 8029 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases/index.rst 2024-03-27T21:42:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: 403865 4 -rw-r--r-- 1 ubuntu ubuntu 3055 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases/general.rst 2024-03-27T21:42:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: 403873 596 -rw-r--r-- 1 ubuntu ubuntu 606974 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/releases/mimic.rst 2024-03-27T21:42:43.503 INFO:teuthology.orchestra.run.smithi060.stdout: 403544 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/install 2024-03-27T21:42:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: 403545 4 -rw-r--r-- 1 ubuntu ubuntu 3307 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/install/build-ceph.rst 2024-03-27T21:42:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: 403555 20 -rw-r--r-- 1 ubuntu ubuntu 17306 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/install/manual-freebsd-deployment.rst 2024-03-27T21:42:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: 403548 16 -rw-r--r-- 1 ubuntu ubuntu 13222 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/install/get-packages.rst 2024-03-27T21:42:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: 403552 4 -rw-r--r-- 1 ubuntu ubuntu 2198 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/install/install-storage-cluster.rst 2024-03-27T21:42:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: 403558 4 -rw-r--r-- 1 ubuntu ubuntu 2750 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/install/windows-install.rst 2024-03-27T21:42:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: 403546 4 -rw-r--r-- 1 ubuntu ubuntu 3455 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/install/clone-source.rst 2024-03-27T21:42:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: 403547 8 -rw-r--r-- 1 ubuntu ubuntu 5180 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/install/containers.rst 2024-03-27T21:42:43.504 INFO:teuthology.orchestra.run.smithi060.stdout: 403549 4 -rw-r--r-- 1 ubuntu ubuntu 522 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/install/get-tarballs.rst 2024-03-27T21:42:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: 403551 4 -rw-r--r-- 1 ubuntu ubuntu 1966 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/install/index_manual.rst 2024-03-27T21:42:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: 403557 4 -rw-r--r-- 1 ubuntu ubuntu 1399 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/install/windows-basic-config.rst 2024-03-27T21:42:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: 403554 20 -rw-r--r-- 1 ubuntu ubuntu 18003 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/install/manual-deployment.rst 2024-03-27T21:42:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: 403550 4 -rw-r--r-- 1 ubuntu ubuntu 2657 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/install/index.rst 2024-03-27T21:42:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: 403556 4 -rw-r--r-- 1 ubuntu ubuntu 1811 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/install/mirrors.rst 2024-03-27T21:42:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: 403553 4 -rw-r--r-- 1 ubuntu ubuntu 3846 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/install/install-vm-cloud.rst 2024-03-27T21:42:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: 403559 4 -rw-r--r-- 1 ubuntu ubuntu 3146 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/install/windows-troubleshooting.rst 2024-03-27T21:42:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: 403488 4 -rw-r--r-- 1 ubuntu ubuntu 1150 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/favicon.ico 2024-03-27T21:42:43.505 INFO:teuthology.orchestra.run.smithi060.stdout: 403489 4 -rw-r--r-- 1 ubuntu ubuntu 3736 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/foundation.rst 2024-03-27T21:42:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: 403131 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/_ext 2024-03-27T21:42:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: 403132 16 -rw-r--r-- 1 ubuntu ubuntu 14400 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/_ext/ceph_commands.py 2024-03-27T21:42:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: 403133 16 -rw-r--r-- 1 ubuntu ubuntu 15171 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/_ext/ceph_confval.py 2024-03-27T21:42:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: 403134 12 -rw-r--r-- 1 ubuntu ubuntu 12230 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/_ext/ceph_releases.py 2024-03-27T21:42:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: 403332 4 drwxr-xr-x 10 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev 2024-03-27T21:42:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: 403409 4 -rw-r--r-- 1 ubuntu ubuntu 1207 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/internals.rst 2024-03-27T21:42:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: 403414 4 -rw-r--r-- 1 ubuntu ubuntu 1616 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/macos.rst 2024-03-27T21:42:43.506 INFO:teuthology.orchestra.run.smithi060.stdout: 403467 8 -rw-r--r-- 1 ubuntu ubuntu 4830 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/rados-client-protocol.rst 2024-03-27T21:42:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: 403363 12 -rw-r--r-- 1 ubuntu ubuntu 10333 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/continuous-integration.rst 2024-03-27T21:42:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: 403358 12 -rw-r--r-- 1 ubuntu ubuntu 12108 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/cephx.rst 2024-03-27T21:42:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: 403380 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/developer_guide 2024-03-27T21:42:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: 403390 8 -rw-r--r-- 1 ubuntu ubuntu 6676 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/developer_guide/running-tests-locally.rst 2024-03-27T21:42:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: 403391 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/developer_guide/testing_integration_tests 2024-03-27T21:42:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: 403396 16 -rw-r--r-- 1 ubuntu ubuntu 13105 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-workflow.rst 2024-03-27T21:42:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: 403393 8 -rw-r--r-- 1 ubuntu ubuntu 6972 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-debugging-tips.rst 2024-03-27T21:42:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: 403394 28 -rw-r--r-- 1 ubuntu ubuntu 27573 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-intro.rst 2024-03-27T21:42:43.507 INFO:teuthology.orchestra.run.smithi060.stdout: 403397 4 -rw-r--r-- 1 ubuntu ubuntu 103 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/developer_guide/testing_integration_tests/tests-sentry-developers-guide.rst 2024-03-27T21:42:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: 403395 4 -rw-r--r-- 1 ubuntu ubuntu 2933 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-kernel.rst 2024-03-27T21:42:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: 403392 4 -rw-r--r-- 1 ubuntu ubuntu 443 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/developer_guide/testing_integration_tests/index.rst 2024-03-27T21:42:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: 403387 4 -rw-r--r-- 1 ubuntu ubuntu 1408 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/developer_guide/issue-tracker.rst 2024-03-27T21:42:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: 403383 4 -rw-r--r-- 1 ubuntu ubuntu 1393 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/developer_guide/debugging-gdb.rst 2024-03-27T21:42:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: 403386 4 -rw-r--r-- 1 ubuntu ubuntu 1205 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/developer_guide/intro.rst 2024-03-27T21:42:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: 403381 24 -rw-r--r-- 1 ubuntu ubuntu 22342 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/developer_guide/basic-workflow.rst 2024-03-27T21:42:43.508 INFO:teuthology.orchestra.run.smithi060.stdout: 403398 8 -rw-r--r-- 1 ubuntu ubuntu 6182 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/developer_guide/tests-unit-tests.rst 2024-03-27T21:42:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: 403384 12 -rw-r--r-- 1 ubuntu ubuntu 11588 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/developer_guide/essentials.rst 2024-03-27T21:42:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: 403388 4 -rw-r--r-- 1 ubuntu ubuntu 2350 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/developer_guide/jaegertracing.rst 2024-03-27T21:42:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: 403385 4 -rw-r--r-- 1 ubuntu ubuntu 915 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/developer_guide/index.rst 2024-03-27T21:42:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: 403389 4 -rw-r--r-- 1 ubuntu ubuntu 3899 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/developer_guide/merging.rst 2024-03-27T21:42:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: 403382 104 -rw-r--r-- 1 ubuntu ubuntu 104195 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/developer_guide/dash-devel.rst 2024-03-27T21:42:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: 403413 4 -rw-r--r-- 1 ubuntu ubuntu 2597 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/logs.rst 2024-03-27T21:42:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: 403401 8 -rw-r--r-- 1 ubuntu ubuntu 4531 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/dpdk.rst 2024-03-27T21:42:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: 403375 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/dashboard 2024-03-27T21:42:43.509 INFO:teuthology.orchestra.run.smithi060.stdout: 403376 4 -rw-r--r-- 1 ubuntu ubuntu 3891 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/dashboard/ui_goals.rst 2024-03-27T21:42:43.510 INFO:teuthology.orchestra.run.smithi060.stdout: 403355 16 -rw-r--r-- 1 ubuntu ubuntu 15695 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/cephfs-mirroring.rst 2024-03-27T21:42:43.510 INFO:teuthology.orchestra.run.smithi060.stdout: 403480 16 -rw-r--r-- 1 ubuntu ubuntu 13060 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/seastore.rst 2024-03-27T21:42:43.510 INFO:teuthology.orchestra.run.smithi060.stdout: 403362 4 -rw-r--r-- 1 ubuntu ubuntu 997 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/context.rst 2024-03-27T21:42:43.510 INFO:teuthology.orchestra.run.smithi060.stdout: 403377 16 -rw-r--r-- 1 ubuntu ubuntu 15346 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/deduplication.rst 2024-03-27T21:42:43.510 INFO:teuthology.orchestra.run.smithi060.stdout: 403479 16 -rw-r--r-- 1 ubuntu ubuntu 12676 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/release-process.rst 2024-03-27T21:42:43.510 INFO:teuthology.orchestra.run.smithi060.stdout: 403359 24 -rw-r--r-- 1 ubuntu ubuntu 22522 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/cephx_protocol.rst 2024-03-27T21:42:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: 403464 20 -rw-r--r-- 1 ubuntu ubuntu 20324 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/perf_histograms.rst 2024-03-27T21:42:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: 403424 8 -rw-r--r-- 1 ubuntu ubuntu 6865 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/mon-elections.rst 2024-03-27T21:42:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: 403406 4 -rw-r--r-- 1 ubuntu ubuntu 2287 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/generatedocs.rst 2024-03-27T21:42:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: 403404 8 -rw-r--r-- 1 ubuntu ubuntu 7889 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/file-striping.rst 2024-03-27T21:42:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: 403463 12 -rw-r--r-- 1 ubuntu ubuntu 8296 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/perf_counters.rst 2024-03-27T21:42:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: 403487 8 -rw-r--r-- 1 ubuntu ubuntu 7256 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/zoned-storage.rst 2024-03-27T21:42:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: 403462 4 -rw-r--r-- 1 ubuntu ubuntu 1450 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/perf.rst 2024-03-27T21:42:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: 403336 4 -rw-r--r-- 1 ubuntu ubuntu 3471 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/bluestore.rst 2024-03-27T21:42:43.511 INFO:teuthology.orchestra.run.smithi060.stdout: 403477 12 -rw-r--r-- 1 ubuntu ubuntu 10506 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/rbd-layering.rst 2024-03-27T21:42:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: 403486 4 -rw-r--r-- 1 ubuntu ubuntu 1373 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/wireshark.rst 2024-03-27T21:42:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: 403400 8 -rw-r--r-- 1 ubuntu ubuntu 4209 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/documenting.rst 2024-03-27T21:42:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: 403356 4 -rw-r--r-- 1 ubuntu ubuntu 3794 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/cephfs-reclaim.rst 2024-03-27T21:42:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: 403430 4 -rw-r--r-- 1 ubuntu ubuntu 1082 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/object-store.rst 2024-03-27T21:42:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: 403405 4 -rw-r--r-- 1 ubuntu ubuntu 1349 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/freebsd.rst 2024-03-27T21:42:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: 403343 48 -rw-r--r-- 1 ubuntu ubuntu 47993 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/ceph_krb_auth.rst 2024-03-27T21:42:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: 403360 4 -rw-r--r-- 1 ubuntu ubuntu 1769 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/config-key.rst 2024-03-27T21:42:43.512 INFO:teuthology.orchestra.run.smithi060.stdout: 403408 4 -rw-r--r-- 1 ubuntu ubuntu 240 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/iana.rst 2024-03-27T21:42:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: 403484 4 -rw-r--r-- 1 ubuntu ubuntu 2184 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/versions.rst 2024-03-27T21:42:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: 403475 4 -rw-r--r-- 1 ubuntu ubuntu 2149 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/rbd-diff.rst 2024-03-27T21:42:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: 403466 8 -rw-r--r-- 1 ubuntu ubuntu 4153 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/quick_guide.rst 2024-03-27T21:42:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: 403334 4 -rw-r--r-- 1 ubuntu ubuntu 2906 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/balancer-design.rst 2024-03-27T21:42:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: 403402 12 -rw-r--r-- 1 ubuntu ubuntu 8623 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/encoding.rst 2024-03-27T21:42:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: 403337 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/ceph-volume 2024-03-27T21:42:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: 403340 4 -rw-r--r-- 1 ubuntu ubuntu 1732 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/ceph-volume/plugins.rst 2024-03-27T21:42:43.513 INFO:teuthology.orchestra.run.smithi060.stdout: 403338 4 -rw-r--r-- 1 ubuntu ubuntu 198 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/ceph-volume/index.rst 2024-03-27T21:42:43.514 INFO:teuthology.orchestra.run.smithi060.stdout: 403339 4 -rw-r--r-- 1 ubuntu ubuntu 3451 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/ceph-volume/lvm.rst 2024-03-27T21:42:43.514 INFO:teuthology.orchestra.run.smithi060.stdout: 403341 4 -rw-r--r-- 1 ubuntu ubuntu 1340 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/ceph-volume/systemd.rst 2024-03-27T21:42:43.514 INFO:teuthology.orchestra.run.smithi060.stdout: 403342 4 -rw-r--r-- 1 ubuntu ubuntu 3619 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/ceph-volume/zfs.rst 2024-03-27T21:42:43.514 INFO:teuthology.orchestra.run.smithi060.stdout: 403478 8 -rw-r--r-- 1 ubuntu ubuntu 6700 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/release-checklists.rst 2024-03-27T21:42:43.514 INFO:teuthology.orchestra.run.smithi060.stdout: 403421 4 -rw-r--r-- 1 ubuntu ubuntu 2707 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/mempool_accounting.rst 2024-03-27T21:42:43.514 INFO:teuthology.orchestra.run.smithi060.stdout: 403411 4 -rw-r--r-- 1 ubuntu ubuntu 640 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/libs.rst 2024-03-27T21:42:43.514 INFO:teuthology.orchestra.run.smithi060.stdout: 403465 8 -rw-r--r-- 1 ubuntu ubuntu 6515 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/placement-group.rst 2024-03-27T21:42:43.514 INFO:teuthology.orchestra.run.smithi060.stdout: 403366 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/crimson 2024-03-27T21:42:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: 403367 20 -rw-r--r-- 1 ubuntu ubuntu 18179 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/crimson/crimson.rst 2024-03-27T21:42:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: 403370 4 -rw-r--r-- 1 ubuntu ubuntu 2106 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/crimson/osd.rst 2024-03-27T21:42:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: 403371 8 -rw-r--r-- 1 ubuntu ubuntu 5727 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/crimson/pipeline.rst 2024-03-27T21:42:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: 403369 4 -rw-r--r-- 1 ubuntu ubuntu 171 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/crimson/index.rst 2024-03-27T21:42:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: 403372 28 -rw-r--r-- 1 ubuntu ubuntu 26409 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/crimson/poseidonstore.rst 2024-03-27T21:42:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: 403368 8 -rw-r--r-- 1 ubuntu ubuntu 7206 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/crimson/error-handling.rst 2024-03-27T21:42:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: 403361 12 -rw-r--r-- 1 ubuntu ubuntu 8582 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/config.rst 2024-03-27T21:42:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: 403461 12 -rw-r--r-- 1 ubuntu ubuntu 11904 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/peering.rst 2024-03-27T21:42:43.515 INFO:teuthology.orchestra.run.smithi060.stdout: 403335 8 -rw-r--r-- 1 ubuntu ubuntu 7308 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/blkin.rst 2024-03-27T21:42:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: 403483 4 -rw-r--r-- 1 ubuntu ubuntu 865 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/testing.rst 2024-03-27T21:42:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: 403415 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/mds_internals 2024-03-27T21:42:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: 403416 4 -rw-r--r-- 1 ubuntu ubuntu 1649 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/mds_internals/data-structures.rst 2024-03-27T21:42:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: 403420 8 -rw-r--r-- 1 ubuntu ubuntu 6240 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/mds_internals/quiesce.rst 2024-03-27T21:42:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: 403417 4 -rw-r--r-- 1 ubuntu ubuntu 3541 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/mds_internals/exports.rst 2024-03-27T21:42:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: 403418 4 -rw-r--r-- 1 ubuntu ubuntu 142 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/mds_internals/index.rst 2024-03-27T21:42:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: 403419 16 -rw-r--r-- 1 ubuntu ubuntu 12372 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/mds_internals/locking.rst 2024-03-27T21:42:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: 403379 4 -rw-r--r-- 1 ubuntu ubuntu 3537 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/dev_cluster_deployment.rst 2024-03-27T21:42:43.516 INFO:teuthology.orchestra.run.smithi060.stdout: 403468 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/radosgw 2024-03-27T21:42:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: 403469 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/radosgw/admin 2024-03-27T21:42:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: 403470 8 -rw-r--r-- 1 ubuntu ubuntu 7623 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/radosgw/admin/adminops_nonimplemented.rst 2024-03-27T21:42:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: 403471 8 -rw-r--r-- 1 ubuntu ubuntu 6415 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/radosgw/bucket_index.rst 2024-03-27T21:42:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: 403472 4 -rw-r--r-- 1 ubuntu ubuntu 284 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/radosgw/index.rst 2024-03-27T21:42:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: 403473 44 -rw-r--r-- 1 ubuntu ubuntu 43503 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/radosgw/s3_compliance.rst 2024-03-27T21:42:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: 403474 4 -rw-r--r-- 1 ubuntu ubuntu 2825 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/radosgw/usage.rst 2024-03-27T21:42:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: 403407 4 -rw-r--r-- 1 ubuntu ubuntu 2825 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/health-reports.rst 2024-03-27T21:42:43.517 INFO:teuthology.orchestra.run.smithi060.stdout: 403422 4 -rw-r--r-- 1 ubuntu ubuntu 1566 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/messenger.rst 2024-03-27T21:42:43.518 INFO:teuthology.orchestra.run.smithi060.stdout: 403427 32 -rw-r--r-- 1 ubuntu ubuntu 32513 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/msgr2.rst 2024-03-27T21:42:43.518 INFO:teuthology.orchestra.run.smithi060.stdout: 403431 4 -rw-r--r-- 1 ubuntu ubuntu 1752 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd-class-path.rst 2024-03-27T21:42:43.518 INFO:teuthology.orchestra.run.smithi060.stdout: 403423 8 -rw-r--r-- 1 ubuntu ubuntu 7828 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/mon-bootstrap.rst 2024-03-27T21:42:43.518 INFO:teuthology.orchestra.run.smithi060.stdout: 403373 8 -rw-r--r-- 1 ubuntu ubuntu 5070 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/crush-msr.rst 2024-03-27T21:42:43.518 INFO:teuthology.orchestra.run.smithi060.stdout: 403364 4 -rw-r--r-- 1 ubuntu ubuntu 2619 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/corpus.rst 2024-03-27T21:42:43.518 INFO:teuthology.orchestra.run.smithi060.stdout: 403428 4 -rw-r--r-- 1 ubuntu ubuntu 4032 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/network-encoding.rst 2024-03-27T21:42:43.518 INFO:teuthology.orchestra.run.smithi060.stdout: 403425 4 -rw-r--r-- 1 ubuntu ubuntu 3013 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/mon-on-disk-formats.rst 2024-03-27T21:42:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: 403378 4 -rw-r--r-- 1 ubuntu ubuntu 606 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/delayed-delete.rst 2024-03-27T21:42:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: 403426 20 -rw-r--r-- 1 ubuntu ubuntu 18562 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/mon-osdmap-prune.rst 2024-03-27T21:42:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: 403333 4 -rw-r--r-- 1 ubuntu ubuntu 3841 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/PlanningImplementation.txt 2024-03-27T21:42:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: 403344 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/cephadm 2024-03-27T21:42:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: 403345 8 -rw-r--r-- 1 ubuntu ubuntu 4544 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/cephadm/compliance-check.rst 2024-03-27T21:42:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: 403352 4 -rw-r--r-- 1 ubuntu ubuntu 3780 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/cephadm/host-maintenance.rst 2024-03-27T21:42:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: 403353 4 -rw-r--r-- 1 ubuntu ubuntu 316 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/cephadm/index.rst 2024-03-27T21:42:43.519 INFO:teuthology.orchestra.run.smithi060.stdout: 403351 20 -rw-r--r-- 1 ubuntu ubuntu 18252 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/cephadm/developing-cephadm.rst 2024-03-27T21:42:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: 403354 4 -rw-r--r-- 1 ubuntu ubuntu 3242 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/cephadm/scalability-notes.rst 2024-03-27T21:42:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: 403346 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/cephadm/design 2024-03-27T21:42:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: 403350 20 -rw-r--r-- 1 ubuntu ubuntu 19095 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/cephadm/design/storage_devices_and_osds.rst 2024-03-27T21:42:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: 403347 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/cephadm/design/mockups 2024-03-27T21:42:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: 403349 152 -rw-r--r-- 1 ubuntu ubuntu 152649 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/cephadm/design/mockups/OSD_Creation_host_mode.svg 2024-03-27T21:42:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: 403348 120 -rw-r--r-- 1 ubuntu ubuntu 122394 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/cephadm/design/mockups/OSD_Creation_device_mode.svg 2024-03-27T21:42:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: 403482 12 -rw-r--r-- 1 ubuntu ubuntu 10132 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/session_authentication.rst 2024-03-27T21:42:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: 403365 4 -rw-r--r-- 1 ubuntu ubuntu 1194 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/cpu-profiler.rst 2024-03-27T21:42:43.520 INFO:teuthology.orchestra.run.smithi060.stdout: 403403 4 -rw-r--r-- 1 ubuntu ubuntu 3740 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/erasure-coded-pool.rst 2024-03-27T21:42:43.521 INFO:teuthology.orchestra.run.smithi060.stdout: 403412 8 -rw-r--r-- 1 ubuntu ubuntu 4814 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/logging.rst 2024-03-27T21:42:43.521 INFO:teuthology.orchestra.run.smithi060.stdout: 403476 4 -rw-r--r-- 1 ubuntu ubuntu 1853 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/rbd-export.rst 2024-03-27T21:42:43.521 INFO:teuthology.orchestra.run.smithi060.stdout: 403485 4 -rw-r--r-- 1 ubuntu ubuntu 1329 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/vstart-ganesha.rst 2024-03-27T21:42:43.521 INFO:teuthology.orchestra.run.smithi060.stdout: 403481 4 -rw-r--r-- 1 ubuntu ubuntu 254 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/sepia.rst 2024-03-27T21:42:43.521 INFO:teuthology.orchestra.run.smithi060.stdout: 403429 8 -rw-r--r-- 1 ubuntu ubuntu 5201 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/network-protocol.rst 2024-03-27T21:42:43.521 INFO:teuthology.orchestra.run.smithi060.stdout: 403357 8 -rw-r--r-- 1 ubuntu ubuntu 7025 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/cephfs-snapshots.rst 2024-03-27T21:42:43.521 INFO:teuthology.orchestra.run.smithi060.stdout: 403374 4 -rw-r--r-- 1 ubuntu ubuntu 1641 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/cxx.rst 2024-03-27T21:42:43.521 INFO:teuthology.orchestra.run.smithi060.stdout: 403410 8 -rw-r--r-- 1 ubuntu ubuntu 7413 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/kubernetes.rst 2024-03-27T21:42:43.522 INFO:teuthology.orchestra.run.smithi060.stdout: 403432 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals 2024-03-27T21:42:43.522 INFO:teuthology.orchestra.run.smithi060.stdout: 403448 12 -rw-r--r-- 1 ubuntu ubuntu 11327 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/osdmap_versions.txt 2024-03-27T21:42:43.522 INFO:teuthology.orchestra.run.smithi060.stdout: 403460 4 -rw-r--r-- 1 ubuntu ubuntu 1228 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/wbthrottle.rst 2024-03-27T21:42:43.522 INFO:teuthology.orchestra.run.smithi060.stdout: 403456 4 -rw-r--r-- 1 ubuntu ubuntu 2295 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/scrub.rst 2024-03-27T21:42:43.522 INFO:teuthology.orchestra.run.smithi060.stdout: 403445 8 -rw-r--r-- 1 ubuntu ubuntu 5800 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/map_message_handling.rst 2024-03-27T21:42:43.522 INFO:teuthology.orchestra.run.smithi060.stdout: 403458 8 -rw-r--r-- 1 ubuntu ubuntu 4548 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/stale_read.rst 2024-03-27T21:42:43.522 INFO:teuthology.orchestra.run.smithi060.stdout: 403433 4 -rw-r--r-- 1 ubuntu ubuntu 2770 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/async_recovery.rst 2024-03-27T21:42:43.522 INFO:teuthology.orchestra.run.smithi060.stdout: 403443 12 -rw-r--r-- 1 ubuntu ubuntu 9093 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/log_based_pg.rst 2024-03-27T21:42:43.523 INFO:teuthology.orchestra.run.smithi060.stdout: 403434 4 -rw-r--r-- 1 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/backfill_reservation.rst 2024-03-27T21:42:43.523 INFO:teuthology.orchestra.run.smithi060.stdout: 403454 4 -rw-r--r-- 1 ubuntu ubuntu 3147 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/recovery_reservation.rst 2024-03-27T21:42:43.523 INFO:teuthology.orchestra.run.smithi060.stdout: 403457 12 -rw-r--r-- 1 ubuntu ubuntu 9117 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/snaps.rst 2024-03-27T21:42:43.523 INFO:teuthology.orchestra.run.smithi060.stdout: 403450 8 -rw-r--r-- 1 ubuntu ubuntu 5413 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/past_intervals.rst 2024-03-27T21:42:43.523 INFO:teuthology.orchestra.run.smithi060.stdout: 403435 4 -rw-r--r-- 1 ubuntu ubuntu 3625 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/erasure_coding.rst 2024-03-27T21:42:43.523 INFO:teuthology.orchestra.run.smithi060.stdout: 403452 4 -rw-r--r-- 1 ubuntu ubuntu 2518 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/pg_removal.rst 2024-03-27T21:42:43.523 INFO:teuthology.orchestra.run.smithi060.stdout: 403444 24 -rw-r--r-- 1 ubuntu ubuntu 21250 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/manifest.rst 2024-03-27T21:42:43.523 INFO:teuthology.orchestra.run.smithi060.stdout: 403459 4 -rw-r--r-- 1 ubuntu ubuntu 3518 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/watch_notify.rst 2024-03-27T21:42:43.523 INFO:teuthology.orchestra.run.smithi060.stdout: 403453 4 -rw-r--r-- 1 ubuntu ubuntu 1025 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/pgpool.rst 2024-03-27T21:42:43.524 INFO:teuthology.orchestra.run.smithi060.stdout: 403447 4 -rw-r--r-- 1 ubuntu ubuntu 3432 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/osd_overview.rst 2024-03-27T21:42:43.524 INFO:teuthology.orchestra.run.smithi060.stdout: 403455 4 -rw-r--r-- 1 ubuntu ubuntu 1218 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/refcount.rst 2024-03-27T21:42:43.524 INFO:teuthology.orchestra.run.smithi060.stdout: 403449 8 -rw-r--r-- 1 ubuntu ubuntu 6593 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/partial_object_recovery.rst 2024-03-27T21:42:43.524 INFO:teuthology.orchestra.run.smithi060.stdout: 403446 24 -rw-r--r-- 1 ubuntu ubuntu 22921 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/mclock_wpq_cmp_study.rst 2024-03-27T21:42:43.524 INFO:teuthology.orchestra.run.smithi060.stdout: 403436 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/erasure_coding 2024-03-27T21:42:43.524 INFO:teuthology.orchestra.run.smithi060.stdout: 403439 4 -rw-r--r-- 1 ubuntu ubuntu 1396 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/erasure_coding/jerasure.rst 2024-03-27T21:42:43.524 INFO:teuthology.orchestra.run.smithi060.stdout: 403437 12 -rw-r--r-- 1 ubuntu ubuntu 9918 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/erasure_coding/developer_notes.rst 2024-03-27T21:42:43.524 INFO:teuthology.orchestra.run.smithi060.stdout: 403438 12 -rw-r--r-- 1 ubuntu ubuntu 8745 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/erasure_coding/ecbackend.rst 2024-03-27T21:42:43.525 INFO:teuthology.orchestra.run.smithi060.stdout: 403440 20 -rw-r--r-- 1 ubuntu ubuntu 20338 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/erasure_coding/proposals.rst 2024-03-27T21:42:43.525 INFO:teuthology.orchestra.run.smithi060.stdout: 403441 4 -rw-r--r-- 1 ubuntu ubuntu 142 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/index.rst 2024-03-27T21:42:43.525 INFO:teuthology.orchestra.run.smithi060.stdout: 403451 4 -rw-r--r-- 1 ubuntu ubuntu 1156 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/pg.rst 2024-03-27T21:42:43.525 INFO:teuthology.orchestra.run.smithi060.stdout: 403442 4 -rw-r--r-- 1 ubuntu ubuntu 3357 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/osd_internals/last_epoch_started.rst 2024-03-27T21:42:43.525 INFO:teuthology.orchestra.run.smithi060.stdout: 403399 12 -rw-r--r-- 1 ubuntu ubuntu 11156 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/dev/development-workflow.rst 2024-03-27T21:42:43.525 INFO:teuthology.orchestra.run.smithi060.stdout: 403880 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/scripts 2024-03-27T21:42:43.525 INFO:teuthology.orchestra.run.smithi060.stdout: 403881 4 -rw-r--r-- 1 ubuntu ubuntu 464 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/scripts/README.md 2024-03-27T21:42:43.526 INFO:teuthology.orchestra.run.smithi060.stdout: 403882 12 -rwxr-xr-x 1 ubuntu ubuntu 8474 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/scripts/gen_state_diagram.py 2024-03-27T21:42:43.526 INFO:teuthology.orchestra.run.smithi060.stdout: 403153 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume 2024-03-27T21:42:43.526 INFO:teuthology.orchestra.run.smithi060.stdout: 403158 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/lvm 2024-03-27T21:42:43.526 INFO:teuthology.orchestra.run.smithi060.stdout: 403161 4 -rw-r--r-- 1 ubuntu ubuntu 866 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/lvm/create.rst 2024-03-27T21:42:43.526 INFO:teuthology.orchestra.run.smithi060.stdout: 403168 16 -rw-r--r-- 1 ubuntu ubuntu 12376 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/lvm/prepare.rst 2024-03-27T21:42:43.526 INFO:teuthology.orchestra.run.smithi060.stdout: 403159 4 -rw-r--r-- 1 ubuntu ubuntu 4078 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/lvm/activate.rst 2024-03-27T21:42:43.526 INFO:teuthology.orchestra.run.smithi060.stdout: 403166 4 -rw-r--r-- 1 ubuntu ubuntu 339 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/lvm/newdb.rst 2024-03-27T21:42:43.526 INFO:teuthology.orchestra.run.smithi060.stdout: 403171 4 -rw-r--r-- 1 ubuntu ubuntu 2214 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/lvm/zap.rst 2024-03-27T21:42:43.527 INFO:teuthology.orchestra.run.smithi060.stdout: 403167 4 -rw-r--r-- 1 ubuntu ubuntu 357 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/lvm/newwal.rst 2024-03-27T21:42:43.527 INFO:teuthology.orchestra.run.smithi060.stdout: 403169 4 -rw-r--r-- 1 ubuntu ubuntu 342 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/lvm/scan.rst 2024-03-27T21:42:43.527 INFO:teuthology.orchestra.run.smithi060.stdout: 403162 4 -rw-r--r-- 1 ubuntu ubuntu 3273 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/lvm/encryption.rst 2024-03-27T21:42:43.527 INFO:teuthology.orchestra.run.smithi060.stdout: 403165 4 -rw-r--r-- 1 ubuntu ubuntu 1747 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/lvm/migrate.rst 2024-03-27T21:42:43.527 INFO:teuthology.orchestra.run.smithi060.stdout: 403164 8 -rw-r--r-- 1 ubuntu ubuntu 7333 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/lvm/list.rst 2024-03-27T21:42:43.527 INFO:teuthology.orchestra.run.smithi060.stdout: 403160 8 -rw-r--r-- 1 ubuntu ubuntu 7306 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/lvm/batch.rst 2024-03-27T21:42:43.527 INFO:teuthology.orchestra.run.smithi060.stdout: 403163 4 -rw-r--r-- 1 ubuntu ubuntu 759 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/lvm/index.rst 2024-03-27T21:42:43.527 INFO:teuthology.orchestra.run.smithi060.stdout: 403170 4 -rw-r--r-- 1 ubuntu ubuntu 872 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/lvm/systemd.rst 2024-03-27T21:42:43.528 INFO:teuthology.orchestra.run.smithi060.stdout: 403178 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/zfs 2024-03-27T21:42:43.528 INFO:teuthology.orchestra.run.smithi060.stdout: 403180 4 -rw-r--r-- 1 ubuntu ubuntu 762 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/zfs/inventory.rst 2024-03-27T21:42:43.528 INFO:teuthology.orchestra.run.smithi060.stdout: 403179 4 -rw-r--r-- 1 ubuntu ubuntu 717 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/zfs/index.rst 2024-03-27T21:42:43.528 INFO:teuthology.orchestra.run.smithi060.stdout: 403156 4 -rw-r--r-- 1 ubuntu ubuntu 4076 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/intro.rst 2024-03-27T21:42:43.528 INFO:teuthology.orchestra.run.smithi060.stdout: 403154 4 -rw-r--r-- 1 ubuntu ubuntu 417 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/drive-group.rst 2024-03-27T21:42:43.528 INFO:teuthology.orchestra.run.smithi060.stdout: 403157 4 -rw-r--r-- 1 ubuntu ubuntu 705 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/inventory.rst 2024-03-27T21:42:43.528 INFO:teuthology.orchestra.run.smithi060.stdout: 403172 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/simple 2024-03-27T21:42:43.528 INFO:teuthology.orchestra.run.smithi060.stdout: 403173 4 -rw-r--r-- 1 ubuntu ubuntu 3138 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/simple/activate.rst 2024-03-27T21:42:43.528 INFO:teuthology.orchestra.run.smithi060.stdout: 403175 8 -rw-r--r-- 1 ubuntu ubuntu 6234 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/simple/scan.rst 2024-03-27T21:42:43.529 INFO:teuthology.orchestra.run.smithi060.stdout: 403174 4 -rw-r--r-- 1 ubuntu ubuntu 1030 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/simple/index.rst 2024-03-27T21:42:43.529 INFO:teuthology.orchestra.run.smithi060.stdout: 403176 4 -rw-r--r-- 1 ubuntu ubuntu 888 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/simple/systemd.rst 2024-03-27T21:42:43.529 INFO:teuthology.orchestra.run.smithi060.stdout: 403155 4 -rw-r--r-- 1 ubuntu ubuntu 2373 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/index.rst 2024-03-27T21:42:43.529 INFO:teuthology.orchestra.run.smithi060.stdout: 403177 4 -rw-r--r-- 1 ubuntu ubuntu 2000 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/ceph-volume/systemd.rst 2024-03-27T21:42:43.529 INFO:teuthology.orchestra.run.smithi060.stdout: 403490 24 -rw-r--r-- 1 ubuntu ubuntu 23608 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/glossary.rst 2024-03-27T21:42:43.529 INFO:teuthology.orchestra.run.smithi060.stdout: 403152 76 -rw-r--r-- 1 ubuntu ubuntu 77772 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/architecture.rst 2024-03-27T21:42:43.529 INFO:teuthology.orchestra.run.smithi060.stdout: 403492 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/hardware-monitoring 2024-03-27T21:42:43.529 INFO:teuthology.orchestra.run.smithi060.stdout: 403493 12 -rw-r--r-- 1 ubuntu ubuntu 8504 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/hardware-monitoring/index.rst 2024-03-27T21:42:43.530 INFO:teuthology.orchestra.run.smithi060.stdout: 403143 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/_themes 2024-03-27T21:42:43.530 INFO:teuthology.orchestra.run.smithi060.stdout: 403144 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/_themes/ceph 2024-03-27T21:42:43.530 INFO:teuthology.orchestra.run.smithi060.stdout: 403148 4 -rw-r--r-- 1 ubuntu ubuntu 80 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/_themes/ceph/theme.conf 2024-03-27T21:42:43.530 INFO:teuthology.orchestra.run.smithi060.stdout: 403145 12 -rw-r--r-- 1 ubuntu ubuntu 9080 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/_themes/ceph/layout.html 2024-03-27T21:42:43.530 INFO:teuthology.orchestra.run.smithi060.stdout: 403146 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/_themes/ceph/static 2024-03-27T21:42:43.530 INFO:teuthology.orchestra.run.smithi060.stdout: 403147 4 -rw-r--r-- 1 ubuntu ubuntu 3694 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/_themes/ceph/static/ceph.css_t 2024-03-27T21:42:43.530 INFO:teuthology.orchestra.run.smithi060.stdout: 403560 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/jaegertracing 2024-03-27T21:42:43.530 INFO:teuthology.orchestra.run.smithi060.stdout: 403563 132 -rw-r--r-- 1 ubuntu ubuntu 131398 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/jaegertracing/rgw_jaeger.png 2024-03-27T21:42:43.531 INFO:teuthology.orchestra.run.smithi060.stdout: 403562 116 -rw-r--r-- 1 ubuntu ubuntu 117123 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/jaegertracing/osd_jaeger.png 2024-03-27T21:42:43.531 INFO:teuthology.orchestra.run.smithi060.stdout: 403561 4 -rw-r--r-- 1 ubuntu ubuntu 3932 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/jaegertracing/index.rst 2024-03-27T21:42:43.531 INFO:teuthology.orchestra.run.smithi060.stdout: 403140 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/_templates 2024-03-27T21:42:43.531 INFO:teuthology.orchestra.run.smithi060.stdout: 403141 4 -rw-r--r-- 1 ubuntu ubuntu 1189 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/_templates/page.html 2024-03-27T21:42:43.531 INFO:teuthology.orchestra.run.smithi060.stdout: 403142 4 -rw-r--r-- 1 ubuntu ubuntu 685 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/_templates/smarttoc.html 2024-03-27T21:42:43.531 INFO:teuthology.orchestra.run.smithi060.stdout: 403261 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog 2024-03-27T21:42:43.531 INFO:teuthology.orchestra.run.smithi060.stdout: 403305 52 -rw-r--r-- 1 ubuntu ubuntu 49420 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.87.2.txt 2024-03-27T21:42:43.531 INFO:teuthology.orchestra.run.smithi060.stdout: 403317 172 -rw-r--r-- 1 ubuntu ubuntu 175571 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v10.2.10.txt 2024-03-27T21:42:43.531 INFO:teuthology.orchestra.run.smithi060.stdout: 403276 48 -rw-r--r-- 1 ubuntu ubuntu 46694 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.61.5.txt 2024-03-27T21:42:43.532 INFO:teuthology.orchestra.run.smithi060.stdout: 403288 48 -rw-r--r-- 1 ubuntu ubuntu 48291 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.67.6.txt 2024-03-27T21:42:43.532 INFO:teuthology.orchestra.run.smithi060.stdout: 403327 132 -rw-r--r-- 1 ubuntu ubuntu 134912 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v12.2.1.txt 2024-03-27T21:42:43.532 INFO:teuthology.orchestra.run.smithi060.stdout: 403294 116 -rw-r--r-- 1 ubuntu ubuntu 115660 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.80.10.txt 2024-03-27T21:42:43.532 INFO:teuthology.orchestra.run.smithi060.stdout: 403269 72 -rw-r--r-- 1 ubuntu ubuntu 72225 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.56.5.txt 2024-03-27T21:42:43.532 INFO:teuthology.orchestra.run.smithi060.stdout: 403309 92 -rw-r--r-- 1 ubuntu ubuntu 91586 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.94.3.txt 2024-03-27T21:42:43.532 INFO:teuthology.orchestra.run.smithi060.stdout: 403284 8 -rw-r--r-- 1 ubuntu ubuntu 7437 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.67.2.txt 2024-03-27T21:42:43.532 INFO:teuthology.orchestra.run.smithi060.stdout: 403307 92 -rw-r--r-- 1 ubuntu ubuntu 91540 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.94.10.txt 2024-03-27T21:42:43.533 INFO:teuthology.orchestra.run.smithi060.stdout: 403328 336 -rw-r--r-- 1 ubuntu ubuntu 340172 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v12.2.2.txt 2024-03-27T21:42:43.533 INFO:teuthology.orchestra.run.smithi060.stdout: 403271 16 -rw-r--r-- 1 ubuntu ubuntu 15992 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.56.7.txt 2024-03-27T21:42:43.533 INFO:teuthology.orchestra.run.smithi060.stdout: 403283 8 -rw-r--r-- 1 ubuntu ubuntu 7643 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.67.11.txt 2024-03-27T21:42:43.533 INFO:teuthology.orchestra.run.smithi060.stdout: 403310 124 -rw-r--r-- 1 ubuntu ubuntu 124131 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.94.4.txt 2024-03-27T21:42:43.533 INFO:teuthology.orchestra.run.smithi060.stdout: 403290 48 -rw-r--r-- 1 ubuntu ubuntu 46313 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.67.8.txt 2024-03-27T21:42:43.533 INFO:teuthology.orchestra.run.smithi060.stdout: 403302 88 -rw-r--r-- 1 ubuntu ubuntu 89673 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.80.8.txt 2024-03-27T21:42:43.533 INFO:teuthology.orchestra.run.smithi060.stdout: 403291 12 -rw-r--r-- 1 ubuntu ubuntu 8524 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.67.9.txt 2024-03-27T21:42:43.533 INFO:teuthology.orchestra.run.smithi060.stdout: 403299 16 -rw-r--r-- 1 ubuntu ubuntu 13270 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.80.5.txt 2024-03-27T21:42:43.534 INFO:teuthology.orchestra.run.smithi060.stdout: 403320 164 -rw-r--r-- 1 ubuntu ubuntu 167893 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v10.2.4.txt 2024-03-27T21:42:43.534 INFO:teuthology.orchestra.run.smithi060.stdout: 403265 16 -rw-r--r-- 1 ubuntu ubuntu 12592 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.56.1.txt 2024-03-27T21:42:43.534 INFO:teuthology.orchestra.run.smithi060.stdout: 403312 140 -rw-r--r-- 1 ubuntu ubuntu 140106 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.94.6.txt 2024-03-27T21:42:43.534 INFO:teuthology.orchestra.run.smithi060.stdout: 403329 364 -rw-r--r-- 1 ubuntu ubuntu 372035 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v12.2.3.txt 2024-03-27T21:42:43.534 INFO:teuthology.orchestra.run.smithi060.stdout: 403266 48 -rw-r--r-- 1 ubuntu ubuntu 47269 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.56.2.txt 2024-03-27T21:42:43.534 INFO:teuthology.orchestra.run.smithi060.stdout: 403319 208 -rw-r--r-- 1 ubuntu ubuntu 212983 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v10.2.3.txt 2024-03-27T21:42:43.534 INFO:teuthology.orchestra.run.smithi060.stdout: 403321 4 -rw-r--r-- 1 ubuntu ubuntu 1325 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v10.2.5.txt 2024-03-27T21:42:43.534 INFO:teuthology.orchestra.run.smithi060.stdout: 403330 60 -rw-r--r-- 1 ubuntu ubuntu 58748 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v9.2.1.txt 2024-03-27T21:42:43.535 INFO:teuthology.orchestra.run.smithi060.stdout: 403289 4 -rw-r--r-- 1 ubuntu ubuntu 3768 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.67.7.txt 2024-03-27T21:42:43.535 INFO:teuthology.orchestra.run.smithi060.stdout: 403323 60 -rw-r--r-- 1 ubuntu ubuntu 57517 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v10.2.7.txt 2024-03-27T21:42:43.535 INFO:teuthology.orchestra.run.smithi060.stdout: 403298 4 -rw-r--r-- 1 ubuntu ubuntu 1371 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.80.4.txt 2024-03-27T21:42:43.535 INFO:teuthology.orchestra.run.smithi060.stdout: 403313 112 -rw-r--r-- 1 ubuntu ubuntu 113251 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.94.7.txt 2024-03-27T21:42:43.535 INFO:teuthology.orchestra.run.smithi060.stdout: 403264 32 -rw-r--r-- 1 ubuntu ubuntu 30913 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.48.3argonaut.txt 2024-03-27T21:42:43.535 INFO:teuthology.orchestra.run.smithi060.stdout: 403274 32 -rw-r--r-- 1 ubuntu ubuntu 30172 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.61.3.txt 2024-03-27T21:42:43.535 INFO:teuthology.orchestra.run.smithi060.stdout: 403268 40 -rw-r--r-- 1 ubuntu ubuntu 40349 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.56.4.txt 2024-03-27T21:42:43.535 INFO:teuthology.orchestra.run.smithi060.stdout: 403267 20 -rw-r--r-- 1 ubuntu ubuntu 19590 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.56.3.txt 2024-03-27T21:42:43.536 INFO:teuthology.orchestra.run.smithi060.stdout: 403275 32 -rw-r--r-- 1 ubuntu ubuntu 30724 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.61.4.txt 2024-03-27T21:42:43.536 INFO:teuthology.orchestra.run.smithi060.stdout: 403280 24 -rw-r--r-- 1 ubuntu ubuntu 20595 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.61.9.txt 2024-03-27T21:42:43.536 INFO:teuthology.orchestra.run.smithi060.stdout: 403303 40 -rw-r--r-- 1 ubuntu ubuntu 40535 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.80.9.txt 2024-03-27T21:42:43.536 INFO:teuthology.orchestra.run.smithi060.stdout: 403262 48 -rw-r--r-- 1 ubuntu ubuntu 47254 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.48.1argonaut.txt 2024-03-27T21:42:43.536 INFO:teuthology.orchestra.run.smithi060.stdout: 403293 8 -rw-r--r-- 1 ubuntu ubuntu 7566 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.80.1.txt 2024-03-27T21:42:43.536 INFO:teuthology.orchestra.run.smithi060.stdout: 403273 4 -rw-r--r-- 1 ubuntu ubuntu 835 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.61.2.txt 2024-03-27T21:42:43.536 INFO:teuthology.orchestra.run.smithi060.stdout: 403314 76 -rw-r--r-- 1 ubuntu ubuntu 75534 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.94.8.txt 2024-03-27T21:42:43.536 INFO:teuthology.orchestra.run.smithi060.stdout: 403295 108 -rw-r--r-- 1 ubuntu ubuntu 110390 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.80.11.txt 2024-03-27T21:42:43.536 INFO:teuthology.orchestra.run.smithi060.stdout: 403272 8 -rw-r--r-- 1 ubuntu ubuntu 4910 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.61.1.txt 2024-03-27T21:42:43.537 INFO:teuthology.orchestra.run.smithi060.stdout: 403306 4 -rw-r--r-- 1 ubuntu ubuntu 2603 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.94.1.txt 2024-03-27T21:42:43.537 INFO:teuthology.orchestra.run.smithi060.stdout: 403292 8 -rw-r--r-- 1 ubuntu ubuntu 7609 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.72.2.txt 2024-03-27T21:42:43.537 INFO:teuthology.orchestra.run.smithi060.stdout: 403285 28 -rw-r--r-- 1 ubuntu ubuntu 25403 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.67.3.txt 2024-03-27T21:42:43.537 INFO:teuthology.orchestra.run.smithi060.stdout: 403263 20 -rw-r--r-- 1 ubuntu ubuntu 16863 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.48.2argonaut.txt 2024-03-27T21:42:43.537 INFO:teuthology.orchestra.run.smithi060.stdout: 403277 4 -rw-r--r-- 1 ubuntu ubuntu 3002 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.61.6.txt 2024-03-27T21:42:43.537 INFO:teuthology.orchestra.run.smithi060.stdout: 403318 108 -rw-r--r-- 1 ubuntu ubuntu 108410 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v10.2.2.txt 2024-03-27T21:42:43.537 INFO:teuthology.orchestra.run.smithi060.stdout: 403282 24 -rw-r--r-- 1 ubuntu ubuntu 24207 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.67.10.txt 2024-03-27T21:42:43.537 INFO:teuthology.orchestra.run.smithi060.stdout: 403301 4 -rw-r--r-- 1 ubuntu ubuntu 2824 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.80.7.txt 2024-03-27T21:42:43.538 INFO:teuthology.orchestra.run.smithi060.stdout: 403322 184 -rw-r--r-- 1 ubuntu ubuntu 185688 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v10.2.6.txt 2024-03-27T21:42:43.538 INFO:teuthology.orchestra.run.smithi060.stdout: 403315 4 -rw-r--r-- 1 ubuntu ubuntu 2385 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.94.9.txt 2024-03-27T21:42:43.538 INFO:teuthology.orchestra.run.smithi060.stdout: 403304 76 -rw-r--r-- 1 ubuntu ubuntu 76441 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.87.1.txt 2024-03-27T21:42:43.538 INFO:teuthology.orchestra.run.smithi060.stdout: 403286 20 -rw-r--r-- 1 ubuntu ubuntu 19555 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.67.4.txt 2024-03-27T21:42:43.538 INFO:teuthology.orchestra.run.smithi060.stdout: 403325 4 -rw-r--r-- 1 ubuntu ubuntu 1988 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v10.2.9.txt 2024-03-27T21:42:43.538 INFO:teuthology.orchestra.run.smithi060.stdout: 403324 192 -rw-r--r-- 1 ubuntu ubuntu 195489 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v10.2.8.txt 2024-03-27T21:42:43.538 INFO:teuthology.orchestra.run.smithi060.stdout: 403279 32 -rw-r--r-- 1 ubuntu ubuntu 29129 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.61.8.txt 2024-03-27T21:42:43.538 INFO:teuthology.orchestra.run.smithi060.stdout: 403296 44 -rw-r--r-- 1 ubuntu ubuntu 43567 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.80.2.txt 2024-03-27T21:42:43.539 INFO:teuthology.orchestra.run.smithi060.stdout: 403281 8 -rw-r--r-- 1 ubuntu ubuntu 5269 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.67.1.txt 2024-03-27T21:42:43.539 INFO:teuthology.orchestra.run.smithi060.stdout: 403300 116 -rw-r--r-- 1 ubuntu ubuntu 115187 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.80.6.txt 2024-03-27T21:42:43.539 INFO:teuthology.orchestra.run.smithi060.stdout: 403316 80 -rw-r--r-- 1 ubuntu ubuntu 80183 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v10.2.1.txt 2024-03-27T21:42:43.539 INFO:teuthology.orchestra.run.smithi060.stdout: 403326 284 -rw-r--r-- 1 ubuntu ubuntu 289580 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v11.2.1.txt 2024-03-27T21:42:43.539 INFO:teuthology.orchestra.run.smithi060.stdout: 403287 28 -rw-r--r-- 1 ubuntu ubuntu 26410 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.67.5.txt 2024-03-27T21:42:43.539 INFO:teuthology.orchestra.run.smithi060.stdout: 403297 4 -rw-r--r-- 1 ubuntu ubuntu 632 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.80.3.txt 2024-03-27T21:42:43.539 INFO:teuthology.orchestra.run.smithi060.stdout: 403270 4 -rw-r--r-- 1 ubuntu ubuntu 1276 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.56.6.txt 2024-03-27T21:42:43.540 INFO:teuthology.orchestra.run.smithi060.stdout: 403308 56 -rw-r--r-- 1 ubuntu ubuntu 55416 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.94.2.txt 2024-03-27T21:42:43.540 INFO:teuthology.orchestra.run.smithi060.stdout: 403278 12 -rw-r--r-- 1 ubuntu ubuntu 8438 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.61.7.txt 2024-03-27T21:42:43.540 INFO:teuthology.orchestra.run.smithi060.stdout: 403311 4 -rw-r--r-- 1 ubuntu ubuntu 2942 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/changelog/v0.94.5.txt 2024-03-27T21:42:43.540 INFO:teuthology.orchestra.run.smithi060.stdout: 403491 4 -rw-r--r-- 1 ubuntu ubuntu 4022 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/governance.rst 2024-03-27T21:42:43.540 INFO:teuthology.orchestra.run.smithi060.stdout: 403741 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw 2024-03-27T21:42:43.540 INFO:teuthology.orchestra.run.smithi060.stdout: 403754 8 -rw-r--r-- 1 ubuntu ubuntu 5439 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/d3n_datacache.rst 2024-03-27T21:42:43.540 INFO:teuthology.orchestra.run.smithi060.stdout: 403773 4 -rw-r--r-- 1 ubuntu ubuntu 2482 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/oidc.rst 2024-03-27T21:42:43.540 INFO:teuthology.orchestra.run.smithi060.stdout: 403767 4 -rw-r--r-- 1 ubuntu ubuntu 2782 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/mfa.rst 2024-03-27T21:42:43.541 INFO:teuthology.orchestra.run.smithi060.stdout: 403755 8 -rw-r--r-- 1 ubuntu ubuntu 7255 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/dynamicresharding.rst 2024-03-27T21:42:43.541 INFO:teuthology.orchestra.run.smithi060.stdout: 403812 4 -rw-r--r-- 1 ubuntu ubuntu 4038 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/zone-features.rst 2024-03-27T21:42:43.541 INFO:teuthology.orchestra.run.smithi060.stdout: 403751 16 -rw-r--r-- 1 ubuntu ubuntu 14214 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/cloud-transition.rst 2024-03-27T21:42:43.541 INFO:teuthology.orchestra.run.smithi060.stdout: 403766 8 -rw-r--r-- 1 ubuntu ubuntu 8186 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/metrics.rst 2024-03-27T21:42:43.541 INFO:teuthology.orchestra.run.smithi060.stdout: 403776 12 -rw-r--r-- 1 ubuntu ubuntu 8463 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/placement.rst 2024-03-27T21:42:43.541 INFO:teuthology.orchestra.run.smithi060.stdout: 403769 56 -rw-r--r-- 1 ubuntu ubuntu 55449 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/multisite.rst 2024-03-27T21:42:43.541 INFO:teuthology.orchestra.run.smithi060.stdout: 403743 8 -rw-r--r-- 1 ubuntu ubuntu 7995 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/STSLite.rst 2024-03-27T21:42:43.541 INFO:teuthology.orchestra.run.smithi060.stdout: 403810 8 -rw-r--r-- 1 ubuntu ubuntu 7388 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/troubleshooting.rst 2024-03-27T21:42:43.541 INFO:teuthology.orchestra.run.smithi060.stdout: 403781 12 -rw-r--r-- 1 ubuntu ubuntu 11326 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/s3-notification-compatibility.rst 2024-03-27T21:42:43.542 INFO:teuthology.orchestra.run.smithi060.stdout: 403770 8 -rw-r--r-- 1 ubuntu ubuntu 7164 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/multitenancy.rst 2024-03-27T21:42:43.542 INFO:teuthology.orchestra.run.smithi060.stdout: 403780 20 -rw-r--r-- 1 ubuntu ubuntu 16990 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/role.rst 2024-03-27T21:42:43.542 INFO:teuthology.orchestra.run.smithi060.stdout: 403799 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/swift 2024-03-27T21:42:43.542 INFO:teuthology.orchestra.run.smithi060.stdout: 403806 4 -rw-r--r-- 1 ubuntu ubuntu 1417 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/swift/serviceops.rst 2024-03-27T21:42:43.542 INFO:teuthology.orchestra.run.smithi060.stdout: 403802 8 -rw-r--r-- 1 ubuntu ubuntu 4728 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/swift/java.rst 2024-03-27T21:42:43.542 INFO:teuthology.orchestra.run.smithi060.stdout: 403800 4 -rw-r--r-- 1 ubuntu ubuntu 2174 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/swift/auth.rst 2024-03-27T21:42:43.542 INFO:teuthology.orchestra.run.smithi060.stdout: 403804 4 -rw-r--r-- 1 ubuntu ubuntu 2285 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/swift/python.rst 2024-03-27T21:42:43.542 INFO:teuthology.orchestra.run.smithi060.stdout: 403801 12 -rw-r--r-- 1 ubuntu ubuntu 9370 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/swift/containerops.rst 2024-03-27T21:42:43.543 INFO:teuthology.orchestra.run.smithi060.stdout: 403805 4 -rw-r--r-- 1 ubuntu ubuntu 2231 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/swift/ruby.rst 2024-03-27T21:42:43.543 INFO:teuthology.orchestra.run.smithi060.stdout: 403807 4 -rw-r--r-- 1 ubuntu ubuntu 3448 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/swift/tempurl.rst 2024-03-27T21:42:43.543 INFO:teuthology.orchestra.run.smithi060.stdout: 403808 4 -rw-r--r-- 1 ubuntu ubuntu 3259 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/swift/tutorial.rst 2024-03-27T21:42:43.543 INFO:teuthology.orchestra.run.smithi060.stdout: 403803 8 -rw-r--r-- 1 ubuntu ubuntu 7054 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/swift/objectops.rst 2024-03-27T21:42:43.543 INFO:teuthology.orchestra.run.smithi060.stdout: 403783 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/s3 2024-03-27T21:42:43.543 INFO:teuthology.orchestra.run.smithi060.stdout: 403788 8 -rw-r--r-- 1 ubuntu ubuntu 5075 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/s3/csharp.rst 2024-03-27T21:42:43.543 INFO:teuthology.orchestra.run.smithi060.stdout: 403795 4 -rw-r--r-- 1 ubuntu ubuntu 4018 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/s3/serviceops.rst 2024-03-27T21:42:43.543 INFO:teuthology.orchestra.run.smithi060.stdout: 403784 16 -rw-r--r-- 1 ubuntu ubuntu 13527 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/s3/authentication.rst 2024-03-27T21:42:43.544 INFO:teuthology.orchestra.run.smithi060.stdout: 403789 8 -rw-r--r-- 1 ubuntu ubuntu 6544 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/s3/java.rst 2024-03-27T21:42:43.544 INFO:teuthology.orchestra.run.smithi060.stdout: 403793 8 -rw-r--r-- 1 ubuntu ubuntu 4891 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/s3/python.rst 2024-03-27T21:42:43.544 INFO:teuthology.orchestra.run.smithi060.stdout: 403791 8 -rw-r--r-- 1 ubuntu ubuntu 5073 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/s3/perl.rst 2024-03-27T21:42:43.544 INFO:teuthology.orchestra.run.smithi060.stdout: 403786 8 -rw-r--r-- 1 ubuntu ubuntu 5466 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/s3/commons.rst 2024-03-27T21:42:43.544 INFO:teuthology.orchestra.run.smithi060.stdout: 403792 8 -rw-r--r-- 1 ubuntu ubuntu 5693 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/s3/php.rst 2024-03-27T21:42:43.544 INFO:teuthology.orchestra.run.smithi060.stdout: 403785 52 -rw-r--r-- 1 ubuntu ubuntu 50144 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/s3/bucketops.rst 2024-03-27T21:42:43.544 INFO:teuthology.orchestra.run.smithi060.stdout: 403794 12 -rw-r--r-- 1 ubuntu ubuntu 8711 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/s3/ruby.rst 2024-03-27T21:42:43.544 INFO:teuthology.orchestra.run.smithi060.stdout: 403787 8 -rw-r--r-- 1 ubuntu ubuntu 8018 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/s3/cpp.rst 2024-03-27T21:42:43.544 INFO:teuthology.orchestra.run.smithi060.stdout: 403790 40 -rw-r--r-- 1 ubuntu ubuntu 38905 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/s3/objectops.rst 2024-03-27T21:42:43.545 INFO:teuthology.orchestra.run.smithi060.stdout: 403778 8 -rw-r--r-- 1 ubuntu ubuntu 5951 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/qat-accel.rst 2024-03-27T21:42:43.545 INFO:teuthology.orchestra.run.smithi060.stdout: 403753 16 -rw-r--r-- 1 ubuntu ubuntu 12640 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/config-ref.rst 2024-03-27T21:42:43.545 INFO:teuthology.orchestra.run.smithi060.stdout: 403768 28 -rw-r--r-- 1 ubuntu ubuntu 28057 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/multisite-sync-policy.rst 2024-03-27T21:42:43.545 INFO:teuthology.orchestra.run.smithi060.stdout: 403796 56 -rw-r--r-- 1 ubuntu ubuntu 54425 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/s3select.rst 2024-03-27T21:42:43.545 INFO:teuthology.orchestra.run.smithi060.stdout: 403758 8 -rw-r--r-- 1 ubuntu ubuntu 4196 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/frontends.rst 2024-03-27T21:42:43.545 INFO:teuthology.orchestra.run.smithi060.stdout: 403771 16 -rw-r--r-- 1 ubuntu ubuntu 13271 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/nfs.rst 2024-03-27T21:42:43.545 INFO:teuthology.orchestra.run.smithi060.stdout: 403797 20 -rw-r--r-- 1 ubuntu ubuntu 17135 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/session-tags.rst 2024-03-27T21:42:43.545 INFO:teuthology.orchestra.run.smithi060.stdout: 403750 8 -rw-r--r-- 1 ubuntu ubuntu 7762 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/cloud-sync-module.rst 2024-03-27T21:42:43.545 INFO:teuthology.orchestra.run.smithi060.stdout: 403752 4 -rw-r--r-- 1 ubuntu ubuntu 2845 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/compression.rst 2024-03-27T21:42:43.546 INFO:teuthology.orchestra.run.smithi060.stdout: 403782 8 -rw-r--r-- 1 ubuntu ubuntu 5985 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/s3.rst 2024-03-27T21:42:43.546 INFO:teuthology.orchestra.run.smithi060.stdout: 403809 4 -rw-r--r-- 1 ubuntu ubuntu 3399 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/sync-modules.rst 2024-03-27T21:42:43.546 INFO:teuthology.orchestra.run.smithi060.stdout: 403772 28 -rw-r--r-- 1 ubuntu ubuntu 24848 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/notifications.rst 2024-03-27T21:42:43.546 INFO:teuthology.orchestra.run.smithi060.stdout: 403757 4 -rw-r--r-- 1 ubuntu ubuntu 3993 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/encryption.rst 2024-03-27T21:42:43.546 INFO:teuthology.orchestra.run.smithi060.stdout: 403760 8 -rw-r--r-- 1 ubuntu ubuntu 4247 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/keycloak.rst 2024-03-27T21:42:43.546 INFO:teuthology.orchestra.run.smithi060.stdout: 403775 8 -rw-r--r-- 1 ubuntu ubuntu 4560 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/orphans.rst 2024-03-27T21:42:43.546 INFO:teuthology.orchestra.run.smithi060.stdout: 403763 8 -rw-r--r-- 1 ubuntu ubuntu 7788 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/layout.rst 2024-03-27T21:42:43.546 INFO:teuthology.orchestra.run.smithi060.stdout: 403746 4 -rw-r--r-- 1 ubuntu ubuntu 234 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/api.rst 2024-03-27T21:42:43.547 INFO:teuthology.orchestra.run.smithi060.stdout: 403764 8 -rw-r--r-- 1 ubuntu ubuntu 6423 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/ldap-auth.rst 2024-03-27T21:42:43.547 INFO:teuthology.orchestra.run.smithi060.stdout: 403744 32 -rw-r--r-- 1 ubuntu ubuntu 29188 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/admin.rst 2024-03-27T21:42:43.547 INFO:teuthology.orchestra.run.smithi060.stdout: 403777 4 -rw-r--r-- 1 ubuntu ubuntu 1707 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/pools.rst 2024-03-27T21:42:43.547 INFO:teuthology.orchestra.run.smithi060.stdout: 403765 44 -rw-r--r-- 1 ubuntu ubuntu 43178 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/lua-scripting.rst 2024-03-27T21:42:43.547 INFO:teuthology.orchestra.run.smithi060.stdout: 403811 16 -rw-r--r-- 1 ubuntu ubuntu 15360 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/vault.rst 2024-03-27T21:42:43.547 INFO:teuthology.orchestra.run.smithi060.stdout: 403742 12 -rw-r--r-- 1 ubuntu ubuntu 12062 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/STS.rst 2024-03-27T21:42:43.547 INFO:teuthology.orchestra.run.smithi060.stdout: 403745 40 -rw-r--r-- 1 ubuntu ubuntu 40923 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/adminops.rst 2024-03-27T21:42:43.547 INFO:teuthology.orchestra.run.smithi060.stdout: 403779 8 -rw-r--r-- 1 ubuntu ubuntu 5954 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/rgw-cache.rst 2024-03-27T21:42:43.548 INFO:teuthology.orchestra.run.smithi060.stdout: 403759 4 -rw-r--r-- 1 ubuntu ubuntu 3444 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/index.rst 2024-03-27T21:42:43.548 INFO:teuthology.orchestra.run.smithi060.stdout: 403798 8 -rw-r--r-- 1 ubuntu ubuntu 5191 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/swift.rst 2024-03-27T21:42:43.548 INFO:teuthology.orchestra.run.smithi060.stdout: 403774 4 -rw-r--r-- 1 ubuntu ubuntu 2467 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/opa.rst 2024-03-27T21:42:43.548 INFO:teuthology.orchestra.run.smithi060.stdout: 403761 12 -rw-r--r-- 1 ubuntu ubuntu 9399 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/keystone.rst 2024-03-27T21:42:43.548 INFO:teuthology.orchestra.run.smithi060.stdout: 403748 4 -rw-r--r-- 1 ubuntu ubuntu 3768 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/barbican.rst 2024-03-27T21:42:43.548 INFO:teuthology.orchestra.run.smithi060.stdout: 403756 8 -rw-r--r-- 1 ubuntu ubuntu 5334 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/elastic-sync-module.rst 2024-03-27T21:42:43.548 INFO:teuthology.orchestra.run.smithi060.stdout: 403762 8 -rw-r--r-- 1 ubuntu ubuntu 8122 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/kmip.rst 2024-03-27T21:42:43.548 INFO:teuthology.orchestra.run.smithi060.stdout: 403747 4 -rw-r--r-- 1 ubuntu ubuntu 1611 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/archive-sync-module.rst 2024-03-27T21:42:43.549 INFO:teuthology.orchestra.run.smithi060.stdout: 403749 12 -rw-r--r-- 1 ubuntu ubuntu 9857 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/radosgw/bucketpolicy.rst 2024-03-27T21:42:43.549 INFO:teuthology.orchestra.run.smithi060.stdout: 403494 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images 2024-03-27T21:42:43.549 INFO:teuthology.orchestra.run.smithi060.stdout: 403502 28 -rw-r--r-- 1 ubuntu ubuntu 26248 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/docreviewprocess.jpg 2024-03-27T21:42:43.549 INFO:teuthology.orchestra.run.smithi060.stdout: 403542 52 -rw-r--r-- 1 ubuntu ubuntu 51158 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/zone-sync2.png 2024-03-27T21:42:43.549 INFO:teuthology.orchestra.run.smithi060.stdout: 403531 20 -rw-r--r-- 1 ubuntu ubuntu 16414 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/stack.png 2024-03-27T21:42:43.549 INFO:teuthology.orchestra.run.smithi060.stdout: 403504 160 -rw-r--r-- 1 ubuntu ubuntu 162254 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/esx_config_iscsi_main.png 2024-03-27T21:42:43.549 INFO:teuthology.orchestra.run.smithi060.stdout: 403496 32 -rw-r--r-- 1 ubuntu ubuntu 29192 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/HAProxy_for_RGW.svg 2024-03-27T21:42:43.549 INFO:teuthology.orchestra.run.smithi060.stdout: 403500 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/dashboard 2024-03-27T21:42:43.550 INFO:teuthology.orchestra.run.smithi060.stdout: 403501 8 -rw-r--r-- 1 ubuntu ubuntu 5787 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/dashboard/invalid-credentials.png 2024-03-27T21:42:43.550 INFO:teuthology.orchestra.run.smithi060.stdout: 403508 124 -rw-r--r-- 1 ubuntu ubuntu 126675 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/keycloak-userclientmapper.png 2024-03-27T21:42:43.550 INFO:teuthology.orchestra.run.smithi060.stdout: 403497 60 -rw-r--r-- 1 ubuntu ubuntu 58212 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/RADOS.jpg 2024-03-27T21:42:43.550 INFO:teuthology.orchestra.run.smithi060.stdout: 403506 116 -rw-r--r-- 1 ubuntu ubuntu 116445 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/esx_web_client_storage_main.png 2024-03-27T21:42:43.550 INFO:teuthology.orchestra.run.smithi060.stdout: 403536 24 -rwxr-xr-x 1 ubuntu ubuntu 20594 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/win2016_iscsi_target_tab.png 2024-03-27T21:42:43.550 INFO:teuthology.orchestra.run.smithi060.stdout: 403509 96 -rw-r--r-- 1 ubuntu ubuntu 97290 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/keycloak-usercredentials.png 2024-03-27T21:42:43.550 INFO:teuthology.orchestra.run.smithi060.stdout: 403537 24 -rwxr-xr-x 1 ubuntu ubuntu 22340 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/win2016_iscsi_target_tab2.png 2024-03-27T21:42:43.550 INFO:teuthology.orchestra.run.smithi060.stdout: 403539 16 -rwxr-xr-x 1 ubuntu ubuntu 13334 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/win2016_mpio_set_failover_only.png 2024-03-27T21:42:43.551 INFO:teuthology.orchestra.run.smithi060.stdout: 403528 68 -rw-r--r-- 1 ubuntu ubuntu 68497 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/region-sync.png 2024-03-27T21:42:43.551 INFO:teuthology.orchestra.run.smithi060.stdout: 403499 44 -rw-r--r-- 1 ubuntu ubuntu 43211 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/RDBSnapshots.jpg 2024-03-27T21:42:43.551 INFO:teuthology.orchestra.run.smithi060.stdout: 403495 92 -rw-r--r-- 1 ubuntu ubuntu 90271 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/CRUSH.jpg 2024-03-27T21:42:43.551 INFO:teuthology.orchestra.run.smithi060.stdout: 403533 8 -rwxr-xr-x 1 ubuntu ubuntu 6956 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/win2016_iscsi_connect_to_target.png 2024-03-27T21:42:43.551 INFO:teuthology.orchestra.run.smithi060.stdout: 403512 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/mclock_wpq_study 2024-03-27T21:42:43.551 INFO:teuthology.orchestra.run.smithi060.stdout: 403519 32 -rw-r--r-- 1 ubuntu ubuntu 31567 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/mclock_wpq_study/Avg_Obj_Rec_Throughput_HDD_NoWALdB_WPQ_vs_mClock.png 2024-03-27T21:42:43.551 INFO:teuthology.orchestra.run.smithi060.stdout: 403518 20 -rw-r--r-- 1 ubuntu ubuntu 19355 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/mclock_wpq_study/Avg_Client_Throughput_NVMe_SSD_WPQ_vs_mClock.png 2024-03-27T21:42:43.551 INFO:teuthology.orchestra.run.smithi060.stdout: 403520 32 -rw-r--r-- 1 ubuntu ubuntu 28836 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/mclock_wpq_study/Avg_Obj_Rec_Throughput_HDD_WALdB_WPQ_vs_mClock.png 2024-03-27T21:42:43.551 INFO:teuthology.orchestra.run.smithi060.stdout: 403525 60 -rw-r--r-- 1 ubuntu ubuntu 58139 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/mclock_wpq_study/Recovery_Rate_Comparison_HDD_NoWALdB_WPQ_vs_mClock.png 2024-03-27T21:42:43.552 INFO:teuthology.orchestra.run.smithi060.stdout: 403513 32 -rw-r--r-- 1 ubuntu ubuntu 29399 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/mclock_wpq_study/Avg_Client_Latency_Percentiles_HDD_NoWALdB_WPQ_vs_mClock.png 2024-03-27T21:42:43.552 INFO:teuthology.orchestra.run.smithi060.stdout: 403515 28 -rw-r--r-- 1 ubuntu ubuntu 27008 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/mclock_wpq_study/Avg_Client_Latency_Percentiles_NVMe_SSD_WPQ_vs_mClock.png 2024-03-27T21:42:43.552 INFO:teuthology.orchestra.run.smithi060.stdout: 403526 52 -rw-r--r-- 1 ubuntu ubuntu 50218 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/mclock_wpq_study/Recovery_Rate_Comparison_HDD_WALdB_WPQ_vs_mClock.png 2024-03-27T21:42:43.552 INFO:teuthology.orchestra.run.smithi060.stdout: 403522 92 -rw-r--r-- 1 ubuntu ubuntu 92308 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/mclock_wpq_study/Clat_Latency_Comparison_HDD_NoWALdB_WPQ_vs_mClock.png 2024-03-27T21:42:43.552 INFO:teuthology.orchestra.run.smithi060.stdout: 403523 124 -rw-r--r-- 1 ubuntu ubuntu 124128 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/mclock_wpq_study/Clat_Latency_Comparison_HDD_WALdB_WPQ_vs_mClock.png 2024-03-27T21:42:43.552 INFO:teuthology.orchestra.run.smithi060.stdout: 403516 24 -rw-r--r-- 1 ubuntu ubuntu 21076 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/mclock_wpq_study/Avg_Client_Throughput_HDD_NoWALdB_WPQ_vs_mClock.png 2024-03-27T21:42:43.552 INFO:teuthology.orchestra.run.smithi060.stdout: 403517 20 -rw-r--r-- 1 ubuntu ubuntu 19387 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/mclock_wpq_study/Avg_Client_Throughput_HDD_WALdB_WPQ_vs_mClock.png 2024-03-27T21:42:43.552 INFO:teuthology.orchestra.run.smithi060.stdout: 403521 32 -rw-r--r-- 1 ubuntu ubuntu 31652 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/mclock_wpq_study/Avg_Obj_Rec_Throughput_NVMe_SSD_WPQ_vs_mClock.png 2024-03-27T21:42:43.553 INFO:teuthology.orchestra.run.smithi060.stdout: 403527 52 -rw-r--r-- 1 ubuntu ubuntu 50603 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/mclock_wpq_study/Recovery_Rate_Comparison_NVMe_SSD_WPQ_vs_mClock.png 2024-03-27T21:42:43.553 INFO:teuthology.orchestra.run.smithi060.stdout: 403524 64 -rw-r--r-- 1 ubuntu ubuntu 62976 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/mclock_wpq_study/Clat_Latency_Comparison_NVMe_SSD_WPQ_vs_mClock.png 2024-03-27T21:42:43.553 INFO:teuthology.orchestra.run.smithi060.stdout: 403514 28 -rw-r--r-- 1 ubuntu ubuntu 28150 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/mclock_wpq_study/Avg_Client_Latency_Percentiles_HDD_WALdB_WPQ_vs_mClock.png 2024-03-27T21:42:43.553 INFO:teuthology.orchestra.run.smithi060.stdout: 403538 20 -rw-r--r-- 1 ubuntu ubuntu 18464 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/win2016_mpclaim_output.png 2024-03-27T21:42:43.553 INFO:teuthology.orchestra.run.smithi060.stdout: 403532 24 -rwxr-xr-x 1 ubuntu ubuntu 21371 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/win2016_iscsi_advanced_window.png 2024-03-27T21:42:43.553 INFO:teuthology.orchestra.run.smithi060.stdout: 403541 732 -rw-r--r-- 1 ubuntu ubuntu 747810 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/zone-sync.svg 2024-03-27T21:42:43.553 INFO:teuthology.orchestra.run.smithi060.stdout: 403529 792 -rw-r--r-- 1 ubuntu ubuntu 809416 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/region-sync.svg 2024-03-27T21:42:43.554 INFO:teuthology.orchestra.run.smithi060.stdout: 403505 140 -rw-r--r-- 1 ubuntu ubuntu 139270 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/esx_iscsi_recov_timeout.png 2024-03-27T21:42:43.554 INFO:teuthology.orchestra.run.smithi060.stdout: 403511 92 -rw-r--r-- 1 ubuntu ubuntu 93042 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/keycloak-usertags.png 2024-03-27T21:42:43.554 INFO:teuthology.orchestra.run.smithi060.stdout: 403540 48 -rw-r--r-- 1 ubuntu ubuntu 47495 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/zone-sync.png 2024-03-27T21:42:43.554 INFO:teuthology.orchestra.run.smithi060.stdout: 403507 80 -rw-r--r-- 1 ubuntu ubuntu 80244 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/keycloak-adduser.png 2024-03-27T21:42:43.554 INFO:teuthology.orchestra.run.smithi060.stdout: 403535 24 -rwxr-xr-x 1 ubuntu ubuntu 21808 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/win2016_iscsi_discovery_tab.png 2024-03-27T21:42:43.554 INFO:teuthology.orchestra.run.smithi060.stdout: 403503 188 -rw-r--r-- 1 ubuntu ubuntu 191689 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/esx_chap.png 2024-03-27T21:42:43.554 INFO:teuthology.orchestra.run.smithi060.stdout: 403510 88 -rw-r--r-- 1 ubuntu ubuntu 89315 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/keycloak-userdetails.png 2024-03-27T21:42:43.554 INFO:teuthology.orchestra.run.smithi060.stdout: 403530 16 -rw-r--r-- 1 ubuntu ubuntu 14440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/rgw-encryption-barbican.png 2024-03-27T21:42:43.555 INFO:teuthology.orchestra.run.smithi060.stdout: 403534 12 -rwxr-xr-x 1 ubuntu ubuntu 10913 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/win2016_iscsi_devices_mpio.png 2024-03-27T21:42:43.555 INFO:teuthology.orchestra.run.smithi060.stdout: 403498 72 -rw-r--r-- 1 ubuntu ubuntu 71953 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/images/RBD.jpg 2024-03-27T21:42:43.555 INFO:teuthology.orchestra.run.smithi060.stdout: 403813 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd 2024-03-27T21:42:43.555 INFO:teuthology.orchestra.run.smithi060.stdout: 403829 4 -rw-r--r-- 1 ubuntu ubuntu 1019 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/iscsi-targets.rst 2024-03-27T21:42:43.555 INFO:teuthology.orchestra.run.smithi060.stdout: 403837 4 -rw-r--r-- 1 ubuntu ubuntu 612 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/nvmeof-requirements.rst 2024-03-27T21:42:43.555 INFO:teuthology.orchestra.run.smithi060.stdout: 403836 4 -rw-r--r-- 1 ubuntu ubuntu 2468 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/nvmeof-overview.rst 2024-03-27T21:42:43.555 INFO:teuthology.orchestra.run.smithi060.stdout: 403817 4 -rw-r--r-- 1 ubuntu ubuntu 248 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/disk.conf 2024-03-27T21:42:43.555 INFO:teuthology.orchestra.run.smithi060.stdout: 403845 4 -rw-r--r-- 1 ubuntu ubuntu 435 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/rbd-integrations.rst 2024-03-27T21:42:43.555 INFO:teuthology.orchestra.run.smithi060.stdout: 403856 16 -rw-r--r-- 1 ubuntu ubuntu 12441 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/rbd-snapshot.rst 2024-03-27T21:42:43.556 INFO:teuthology.orchestra.run.smithi060.stdout: 403840 12 -rw-r--r-- 1 ubuntu ubuntu 8967 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/rados-rbd-cmds.rst 2024-03-27T21:42:43.556 INFO:teuthology.orchestra.run.smithi060.stdout: 403822 4 -rw-r--r-- 1 ubuntu ubuntu 696 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/iscsi-initiators.rst 2024-03-27T21:42:43.556 INFO:teuthology.orchestra.run.smithi060.stdout: 403855 4 -rw-r--r-- 1 ubuntu ubuntu 1440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/rbd-replay.rst 2024-03-27T21:42:43.556 INFO:teuthology.orchestra.run.smithi060.stdout: 403828 12 -rw-r--r-- 1 ubuntu ubuntu 9153 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/iscsi-target-cli.rst 2024-03-27T21:42:43.556 INFO:teuthology.orchestra.run.smithi060.stdout: 403844 8 -rw-r--r-- 1 ubuntu ubuntu 4841 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/rbd-exclusive-locks.rst 2024-03-27T21:42:43.556 INFO:teuthology.orchestra.run.smithi060.stdout: 403835 4 -rw-r--r-- 1 ubuntu ubuntu 390 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/nvmeof-initiators.rst 2024-03-27T21:42:43.556 INFO:teuthology.orchestra.run.smithi060.stdout: 403830 12 -rw-r--r-- 1 ubuntu ubuntu 11485 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/libvirt.rst 2024-03-27T21:42:43.556 INFO:teuthology.orchestra.run.smithi060.stdout: 403834 4 -rw-r--r-- 1 ubuntu ubuntu 1354 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/nvmeof-initiator-linux.rst 2024-03-27T21:42:43.556 INFO:teuthology.orchestra.run.smithi060.stdout: 403854 8 -rw-r--r-- 1 ubuntu ubuntu 5045 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/rbd-persistent-write-log-cache.rst 2024-03-27T21:42:43.557 INFO:teuthology.orchestra.run.smithi060.stdout: 403821 4 -rw-r--r-- 1 ubuntu ubuntu 3000 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/iscsi-initiator-win.rst 2024-03-27T21:42:43.557 INFO:teuthology.orchestra.run.smithi060.stdout: 403819 4 -rw-r--r-- 1 ubuntu ubuntu 3472 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/iscsi-initiator-esx.rst 2024-03-27T21:42:43.557 INFO:teuthology.orchestra.run.smithi060.stdout: 403823 4 -rw-r--r-- 1 ubuntu ubuntu 2971 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/iscsi-monitoring.rst 2024-03-27T21:42:43.557 INFO:teuthology.orchestra.run.smithi060.stdout: 403850 16 -rw-r--r-- 1 ubuntu ubuntu 14208 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/rbd-nomad.rst 2024-03-27T21:42:43.557 INFO:teuthology.orchestra.run.smithi060.stdout: 403824 4 -rw-r--r-- 1 ubuntu ubuntu 2769 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/iscsi-overview.rst 2024-03-27T21:42:43.557 INFO:teuthology.orchestra.run.smithi060.stdout: 403857 12 -rw-r--r-- 1 ubuntu ubuntu 9396 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/rbd-windows.rst 2024-03-27T21:42:43.557 INFO:teuthology.orchestra.run.smithi060.stdout: 403851 16 -rw-r--r-- 1 ubuntu ubuntu 15282 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/rbd-openstack.rst 2024-03-27T21:42:43.557 INFO:teuthology.orchestra.run.smithi060.stdout: 403849 24 -rw-r--r-- 1 ubuntu ubuntu 23965 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/rbd-mirroring.rst 2024-03-27T21:42:43.558 INFO:teuthology.orchestra.run.smithi060.stdout: 403826 12 -rw-r--r-- 1 ubuntu ubuntu 10109 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/iscsi-target-ansible.rst 2024-03-27T21:42:43.558 INFO:teuthology.orchestra.run.smithi060.stdout: 403843 12 -rw-r--r-- 1 ubuntu ubuntu 11301 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/rbd-encryption.rst 2024-03-27T21:42:43.558 INFO:teuthology.orchestra.run.smithi060.stdout: 403853 8 -rw-r--r-- 1 ubuntu ubuntu 6948 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/rbd-persistent-read-only-cache.rst 2024-03-27T21:42:43.558 INFO:teuthology.orchestra.run.smithi060.stdout: 403820 4 -rw-r--r-- 1 ubuntu ubuntu 3254 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/iscsi-initiator-linux.rst 2024-03-27T21:42:43.558 INFO:teuthology.orchestra.run.smithi060.stdout: 403841 8 -rw-r--r-- 1 ubuntu ubuntu 6659 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/rbd-cloudstack.rst 2024-03-27T21:42:43.558 INFO:teuthology.orchestra.run.smithi060.stdout: 403842 12 -rw-r--r-- 1 ubuntu ubuntu 9629 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/rbd-config-ref.rst 2024-03-27T21:42:43.558 INFO:teuthology.orchestra.run.smithi060.stdout: 403839 8 -rw-r--r-- 1 ubuntu ubuntu 8157 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/qemu-rbd.rst 2024-03-27T21:42:43.558 INFO:teuthology.orchestra.run.smithi060.stdout: 403833 4 -rw-r--r-- 1 ubuntu ubuntu 1779 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/nvmeof-initiator-esx.rst 2024-03-27T21:42:43.559 INFO:teuthology.orchestra.run.smithi060.stdout: 403838 4 -rw-r--r-- 1 ubuntu ubuntu 4059 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/nvmeof-target-configure.rst 2024-03-27T21:42:43.559 INFO:teuthology.orchestra.run.smithi060.stdout: 403852 4 -rw-r--r-- 1 ubuntu ubuntu 491 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/rbd-operations.rst 2024-03-27T21:42:43.559 INFO:teuthology.orchestra.run.smithi060.stdout: 403825 4 -rw-r--r-- 1 ubuntu ubuntu 1565 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/iscsi-requirements.rst 2024-03-27T21:42:43.559 INFO:teuthology.orchestra.run.smithi060.stdout: 403818 4 -rw-r--r-- 1 ubuntu ubuntu 2497 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/index.rst 2024-03-27T21:42:43.559 INFO:teuthology.orchestra.run.smithi060.stdout: 403848 16 -rw-r--r-- 1 ubuntu ubuntu 13430 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/rbd-live-migration.rst 2024-03-27T21:42:43.559 INFO:teuthology.orchestra.run.smithi060.stdout: 403846 4 -rw-r--r-- 1 ubuntu ubuntu 1564 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/rbd-ko.rst 2024-03-27T21:42:43.559 INFO:teuthology.orchestra.run.smithi060.stdout: 403847 16 -rw-r--r-- 1 ubuntu ubuntu 13409 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/rbd-kubernetes.rst 2024-03-27T21:42:43.559 INFO:teuthology.orchestra.run.smithi060.stdout: 403814 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/api 2024-03-27T21:42:43.559 INFO:teuthology.orchestra.run.smithi060.stdout: 403816 4 -rw-r--r-- 1 ubuntu ubuntu 2420 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/api/librbdpy.rst 2024-03-27T21:42:43.560 INFO:teuthology.orchestra.run.smithi060.stdout: 403815 4 -rw-r--r-- 1 ubuntu ubuntu 135 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/api/index.rst 2024-03-27T21:42:43.560 INFO:teuthology.orchestra.run.smithi060.stdout: 403827 8 -rw-r--r-- 1 ubuntu ubuntu 4566 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/iscsi-target-cli-manual-install.rst 2024-03-27T21:42:43.560 INFO:teuthology.orchestra.run.smithi060.stdout: 403831 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/man 2024-03-27T21:42:43.560 INFO:teuthology.orchestra.run.smithi060.stdout: 403832 4 -rw-r--r-- 1 ubuntu ubuntu 461 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/rbd/man/index.rst 2024-03-27T21:42:43.560 INFO:teuthology.orchestra.run.smithi060.stdout: 403543 4 -rw-r--r-- 1 ubuntu ubuntu 3057 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/index.rst 2024-03-27T21:42:43.560 INFO:teuthology.orchestra.run.smithi060.stdout: 403181 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm 2024-03-27T21:42:43.560 INFO:teuthology.orchestra.run.smithi060.stdout: 403203 20 -rw-r--r-- 1 ubuntu ubuntu 16770 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/troubleshooting.rst 2024-03-27T21:42:43.560 INFO:teuthology.orchestra.run.smithi060.stdout: 403187 24 -rw-r--r-- 1 ubuntu ubuntu 20820 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/install.rst 2024-03-27T21:42:43.561 INFO:teuthology.orchestra.run.smithi060.stdout: 403184 4 -rw-r--r-- 1 ubuntu ubuntu 2516 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/compatibility.rst 2024-03-27T21:42:43.561 INFO:teuthology.orchestra.run.smithi060.stdout: 403183 4 -rw-r--r-- 1 ubuntu ubuntu 1884 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/client-setup.rst 2024-03-27T21:42:43.561 INFO:teuthology.orchestra.run.smithi060.stdout: 403182 8 -rw-r--r-- 1 ubuntu ubuntu 6408 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/adoption.rst 2024-03-27T21:42:43.561 INFO:teuthology.orchestra.run.smithi060.stdout: 403188 28 -rw-r--r-- 1 ubuntu ubuntu 25492 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/operations.rst 2024-03-27T21:42:43.561 INFO:teuthology.orchestra.run.smithi060.stdout: 403185 20 -rw-r--r-- 1 ubuntu ubuntu 20117 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/host-management.rst 2024-03-27T21:42:43.561 INFO:teuthology.orchestra.run.smithi060.stdout: 403189 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/services 2024-03-27T21:42:43.561 INFO:teuthology.orchestra.run.smithi060.stdout: 403202 4 -rw-r--r-- 1 ubuntu ubuntu 972 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/services/tracing.rst 2024-03-27T21:42:43.561 INFO:teuthology.orchestra.run.smithi060.stdout: 403196 24 -rw-r--r-- 1 ubuntu ubuntu 20988 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/services/monitoring.rst 2024-03-27T21:42:43.562 INFO:teuthology.orchestra.run.smithi060.stdout: 403193 4 -rw-r--r-- 1 ubuntu ubuntu 1263 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/services/mds.rst 2024-03-27T21:42:43.562 INFO:teuthology.orchestra.run.smithi060.stdout: 403192 4 -rw-r--r-- 1 ubuntu ubuntu 2094 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/services/iscsi.rst 2024-03-27T21:42:43.562 INFO:teuthology.orchestra.run.smithi060.stdout: 403198 28 -rw-r--r-- 1 ubuntu ubuntu 27919 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/services/osd.rst 2024-03-27T21:42:43.562 INFO:teuthology.orchestra.run.smithi060.stdout: 403195 8 -rw-r--r-- 1 ubuntu ubuntu 6892 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/services/mon.rst 2024-03-27T21:42:43.562 INFO:teuthology.orchestra.run.smithi060.stdout: 403200 8 -rw-r--r-- 1 ubuntu ubuntu 7550 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/services/smb.rst 2024-03-27T21:42:43.562 INFO:teuthology.orchestra.run.smithi060.stdout: 403197 8 -rw-r--r-- 1 ubuntu ubuntu 6069 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/services/nfs.rst 2024-03-27T21:42:43.562 INFO:teuthology.orchestra.run.smithi060.stdout: 403199 16 -rw-r--r-- 1 ubuntu ubuntu 13382 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/services/rgw.rst 2024-03-27T21:42:43.563 INFO:teuthology.orchestra.run.smithi060.stdout: 403201 8 -rw-r--r-- 1 ubuntu ubuntu 5966 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/services/snmp-gateway.rst 2024-03-27T21:42:43.563 INFO:teuthology.orchestra.run.smithi060.stdout: 403194 4 -rw-r--r-- 1 ubuntu ubuntu 940 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/services/mgr.rst 2024-03-27T21:42:43.563 INFO:teuthology.orchestra.run.smithi060.stdout: 403191 28 -rw-r--r-- 1 ubuntu ubuntu 25496 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/services/index.rst 2024-03-27T21:42:43.563 INFO:teuthology.orchestra.run.smithi060.stdout: 403190 8 -rw-r--r-- 1 ubuntu ubuntu 5236 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/services/custom-container.rst 2024-03-27T21:42:43.563 INFO:teuthology.orchestra.run.smithi060.stdout: 403186 4 -rw-r--r-- 1 ubuntu ubuntu 1727 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/index.rst 2024-03-27T21:42:43.563 INFO:teuthology.orchestra.run.smithi060.stdout: 403204 12 -rw-r--r-- 1 ubuntu ubuntu 10157 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/cephadm/upgrade.rst 2024-03-27T21:42:43.563 INFO:teuthology.orchestra.run.smithi060.stdout: 403128 4 -rw-r--r-- 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/.gitignore 2024-03-27T21:42:43.563 INFO:teuthology.orchestra.run.smithi060.stdout: 403149 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/api 2024-03-27T21:42:43.564 INFO:teuthology.orchestra.run.smithi060.stdout: 403151 4 -rw-r--r-- 1 ubuntu ubuntu 133 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/api/mon_command_api.rst 2024-03-27T21:42:43.564 INFO:teuthology.orchestra.run.smithi060.stdout: 403150 4 -rw-r--r-- 1 ubuntu ubuntu 743 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/api/index.rst 2024-03-27T21:42:43.564 INFO:teuthology.orchestra.run.smithi060.stdout: 403892 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/start 2024-03-27T21:42:43.564 INFO:teuthology.orchestra.run.smithi060.stdout: 403898 8 -rw-r--r-- 1 ubuntu ubuntu 4278 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/start/os-recommendations.rst 2024-03-27T21:42:43.564 INFO:teuthology.orchestra.run.smithi060.stdout: 403897 4 -rw-r--r-- 1 ubuntu ubuntu 3848 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/start/intro.rst 2024-03-27T21:42:43.564 INFO:teuthology.orchestra.run.smithi060.stdout: 403893 4 -rw-r--r-- 1 ubuntu ubuntu 75 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/start/ceph.conf 2024-03-27T21:42:43.564 INFO:teuthology.orchestra.run.smithi060.stdout: 403894 40 -rw-r--r-- 1 ubuntu ubuntu 39179 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/start/documenting-ceph.rst 2024-03-27T21:42:43.564 INFO:teuthology.orchestra.run.smithi060.stdout: 403899 4 -rw-r--r-- 1 ubuntu ubuntu 2312 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/start/quick-rbd.rst 2024-03-27T21:42:43.565 INFO:teuthology.orchestra.run.smithi060.stdout: 403896 32 -rw-r--r-- 1 ubuntu ubuntu 30983 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/start/hardware-recommendations.rst 2024-03-27T21:42:43.565 INFO:teuthology.orchestra.run.smithi060.stdout: 403895 12 -rw-r--r-- 1 ubuntu ubuntu 10406 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/start/get-involved.rst 2024-03-27T21:42:43.565 INFO:teuthology.orchestra.run.smithi060.stdout: 403565 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man 2024-03-27T21:42:43.565 INFO:teuthology.orchestra.run.smithi060.stdout: 403617 4 -rw-r--r-- 1 ubuntu ubuntu 428 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/CMakeLists.txt 2024-03-27T21:42:43.565 INFO:teuthology.orchestra.run.smithi060.stdout: 403566 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8 2024-03-27T21:42:43.565 INFO:teuthology.orchestra.run.smithi060.stdout: 403588 4 -rw-r--r-- 1 ubuntu ubuntu 1409 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-volume-systemd.rst 2024-03-27T21:42:43.565 INFO:teuthology.orchestra.run.smithi060.stdout: 403604 8 -rw-r--r-- 1 ubuntu ubuntu 6300 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/radosgw.rst 2024-03-27T21:42:43.565 INFO:teuthology.orchestra.run.smithi060.stdout: 403572 4 -rw-r--r-- 1 ubuntu ubuntu 1287 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-create-keys.rst 2024-03-27T21:42:43.565 INFO:teuthology.orchestra.run.smithi060.stdout: 403567 4 -rw-r--r-- 1 ubuntu ubuntu 1867 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/CMakeLists.txt 2024-03-27T21:42:43.566 INFO:teuthology.orchestra.run.smithi060.stdout: 403575 4 -rw-r--r-- 1 ubuntu ubuntu 1869 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-diff-sorted.rst 2024-03-27T21:42:43.566 INFO:teuthology.orchestra.run.smithi060.stdout: 403606 4 -rw-r--r-- 1 ubuntu ubuntu 1591 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/rbd-ggate.rst 2024-03-27T21:42:43.566 INFO:teuthology.orchestra.run.smithi060.stdout: 403616 4 -rw-r--r-- 1 ubuntu ubuntu 3603 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/rgw-restore-bucket-index.rst 2024-03-27T21:42:43.566 INFO:teuthology.orchestra.run.smithi060.stdout: 403591 20 -rw-r--r-- 1 ubuntu ubuntu 17612 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/cephadm.rst 2024-03-27T21:42:43.566 INFO:teuthology.orchestra.run.smithi060.stdout: 403611 4 -rw-r--r-- 1 ubuntu ubuntu 1895 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/rbd-replay.rst 2024-03-27T21:42:43.566 INFO:teuthology.orchestra.run.smithi060.stdout: 403581 4 -rw-r--r-- 1 ubuntu ubuntu 2405 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-monstore-tool.rst 2024-03-27T21:42:43.566 INFO:teuthology.orchestra.run.smithi060.stdout: 403598 4 -rw-r--r-- 1 ubuntu ubuntu 3718 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/monmaptool.rst 2024-03-27T21:42:43.566 INFO:teuthology.orchestra.run.smithi060.stdout: 403609 4 -rw-r--r-- 1 ubuntu ubuntu 1986 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/rbd-replay-many.rst 2024-03-27T21:42:43.567 INFO:teuthology.orchestra.run.smithi060.stdout: 403570 4 -rw-r--r-- 1 ubuntu ubuntu 811 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-clsinfo.rst 2024-03-27T21:42:43.567 INFO:teuthology.orchestra.run.smithi060.stdout: 403608 4 -rw-r--r-- 1 ubuntu ubuntu 2712 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/rbd-nbd.rst 2024-03-27T21:42:43.567 INFO:teuthology.orchestra.run.smithi060.stdout: 403569 12 -rw-r--r-- 1 ubuntu ubuntu 9113 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-bluestore-tool.rst 2024-03-27T21:42:43.567 INFO:teuthology.orchestra.run.smithi060.stdout: 403605 4 -rw-r--r-- 1 ubuntu ubuntu 1345 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/rbd-fuse.rst 2024-03-27T21:42:43.567 INFO:teuthology.orchestra.run.smithi060.stdout: 403573 4 -rw-r--r-- 1 ubuntu ubuntu 1224 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-debugpack.rst 2024-03-27T21:42:43.567 INFO:teuthology.orchestra.run.smithi060.stdout: 403577 4 -rw-r--r-- 1 ubuntu ubuntu 2003 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-immutable-object-cache.rst 2024-03-27T21:42:43.567 INFO:teuthology.orchestra.run.smithi060.stdout: 403568 8 -rw-r--r-- 1 ubuntu ubuntu 6017 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-authtool.rst 2024-03-27T21:42:43.567 INFO:teuthology.orchestra.run.smithi060.stdout: 403576 4 -rw-r--r-- 1 ubuntu ubuntu 2464 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-fuse.rst 2024-03-27T21:42:43.568 INFO:teuthology.orchestra.run.smithi060.stdout: 403580 4 -rw-r--r-- 1 ubuntu ubuntu 3160 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-mon.rst 2024-03-27T21:42:43.568 INFO:teuthology.orchestra.run.smithi060.stdout: 403590 40 -rw-r--r-- 1 ubuntu ubuntu 40906 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph.rst 2024-03-27T21:42:43.568 INFO:teuthology.orchestra.run.smithi060.stdout: 403599 12 -rw-r--r-- 1 ubuntu ubuntu 10836 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/mount.ceph.rst 2024-03-27T21:42:43.568 INFO:teuthology.orchestra.run.smithi060.stdout: 403589 16 -rw-r--r-- 1 ubuntu ubuntu 13205 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-volume.rst 2024-03-27T21:42:43.568 INFO:teuthology.orchestra.run.smithi060.stdout: 403607 4 -rw-r--r-- 1 ubuntu ubuntu 1815 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/rbd-mirror.rst 2024-03-27T21:42:43.568 INFO:teuthology.orchestra.run.smithi060.stdout: 403582 16 -rw-r--r-- 1 ubuntu ubuntu 13928 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-objectstore-tool.rst 2024-03-27T21:42:43.568 INFO:teuthology.orchestra.run.smithi060.stdout: 403602 12 -rw-r--r-- 1 ubuntu ubuntu 10124 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/rados.rst 2024-03-27T21:42:43.569 INFO:teuthology.orchestra.run.smithi060.stdout: 403592 4 -rw-r--r-- 1 ubuntu ubuntu 1811 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/cephfs-mirror.rst 2024-03-27T21:42:43.569 INFO:teuthology.orchestra.run.smithi060.stdout: 403574 4 -rw-r--r-- 1 ubuntu ubuntu 3975 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-dencoder.rst 2024-03-27T21:42:43.569 INFO:teuthology.orchestra.run.smithi060.stdout: 403578 4 -rw-r--r-- 1 ubuntu ubuntu 3424 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-kvstore-tool.rst 2024-03-27T21:42:43.569 INFO:teuthology.orchestra.run.smithi060.stdout: 403600 4 -rw-r--r-- 1 ubuntu ubuntu 2032 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/mount.fuse.ceph.rst 2024-03-27T21:42:43.569 INFO:teuthology.orchestra.run.smithi060.stdout: 403603 24 -rw-r--r-- 1 ubuntu ubuntu 23139 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/radosgw-admin.rst 2024-03-27T21:42:43.569 INFO:teuthology.orchestra.run.smithi060.stdout: 403612 44 -rw-r--r-- 1 ubuntu ubuntu 43507 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/rbd.rst 2024-03-27T21:42:43.569 INFO:teuthology.orchestra.run.smithi060.stdout: 403584 4 -rw-r--r-- 1 ubuntu ubuntu 1857 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-post-file.rst 2024-03-27T21:42:43.569 INFO:teuthology.orchestra.run.smithi060.stdout: 403593 16 -rw-r--r-- 1 ubuntu ubuntu 14576 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/cephfs-shell.rst 2024-03-27T21:42:43.570 INFO:teuthology.orchestra.run.smithi060.stdout: 403585 4 -rw-r--r-- 1 ubuntu ubuntu 789 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-rbdnamer.rst 2024-03-27T21:42:43.570 INFO:teuthology.orchestra.run.smithi060.stdout: 403595 4 -rw-r--r-- 1 ubuntu ubuntu 2993 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/crushdiff.rst 2024-03-27T21:42:43.570 INFO:teuthology.orchestra.run.smithi060.stdout: 403596 12 -rw-r--r-- 1 ubuntu ubuntu 9487 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/crushtool.rst 2024-03-27T21:42:43.570 INFO:teuthology.orchestra.run.smithi060.stdout: 403610 4 -rw-r--r-- 1 ubuntu ubuntu 1208 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/rbd-replay-prep.rst 2024-03-27T21:42:43.570 INFO:teuthology.orchestra.run.smithi060.stdout: 403586 4 -rw-r--r-- 1 ubuntu ubuntu 882 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-run.rst 2024-03-27T21:42:43.570 INFO:teuthology.orchestra.run.smithi060.stdout: 403571 8 -rw-r--r-- 1 ubuntu ubuntu 4212 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-conf.rst 2024-03-27T21:42:43.570 INFO:teuthology.orchestra.run.smithi060.stdout: 403587 4 -rw-r--r-- 1 ubuntu ubuntu 2514 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-syn.rst 2024-03-27T21:42:43.570 INFO:teuthology.orchestra.run.smithi060.stdout: 403597 4 -rw-r--r-- 1 ubuntu ubuntu 821 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/librados-config.rst 2024-03-27T21:42:43.571 INFO:teuthology.orchestra.run.smithi060.stdout: 403601 12 -rw-r--r-- 1 ubuntu ubuntu 11135 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/osdmaptool.rst 2024-03-27T21:42:43.571 INFO:teuthology.orchestra.run.smithi060.stdout: 403594 4 -rw-r--r-- 1 ubuntu ubuntu 2379 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/cephfs-top.rst 2024-03-27T21:42:43.571 INFO:teuthology.orchestra.run.smithi060.stdout: 403579 4 -rw-r--r-- 1 ubuntu ubuntu 2326 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-mds.rst 2024-03-27T21:42:43.571 INFO:teuthology.orchestra.run.smithi060.stdout: 403615 4 -rw-r--r-- 1 ubuntu ubuntu 1413 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/rgw-policy-check.rst 2024-03-27T21:42:43.571 INFO:teuthology.orchestra.run.smithi060.stdout: 403613 8 -rw-r--r-- 1 ubuntu ubuntu 4629 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/rbdmap.rst 2024-03-27T21:42:43.571 INFO:teuthology.orchestra.run.smithi060.stdout: 403583 4 -rw-r--r-- 1 ubuntu ubuntu 4050 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/ceph-osd.rst 2024-03-27T21:42:43.571 INFO:teuthology.orchestra.run.smithi060.stdout: 403614 4 -rw-r--r-- 1 ubuntu ubuntu 1925 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc/man/8/rgw-orphan-list.rst 2024-03-27T21:42:43.571 INFO:teuthology.orchestra.run.smithi060.stdout: 402891 108 -rw-r--r-- 1 ubuntu ubuntu 106582 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.organizationmap 2024-03-27T21:42:43.572 INFO:teuthology.orchestra.run.smithi060.stdout: 402872 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.github 2024-03-27T21:42:43.572 INFO:teuthology.orchestra.run.smithi060.stdout: 402875 8 -rw-r--r-- 1 ubuntu ubuntu 6522 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.github/labeler.yml 2024-03-27T21:42:43.572 INFO:teuthology.orchestra.run.smithi060.stdout: 402876 4 -rw-r--r-- 1 ubuntu ubuntu 107 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.github/milestone.yml 2024-03-27T21:42:43.572 INFO:teuthology.orchestra.run.smithi060.stdout: 402873 12 -rw-r--r-- 1 ubuntu ubuntu 10225 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.github/CODEOWNERS 2024-03-27T21:42:43.572 INFO:teuthology.orchestra.run.smithi060.stdout: 402878 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.github/workflows 2024-03-27T21:42:43.572 INFO:teuthology.orchestra.run.smithi060.stdout: 402881 4 -rw-r--r-- 1 ubuntu ubuntu 298 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.github/workflows/pr-check-deps.yml 2024-03-27T21:42:43.572 INFO:teuthology.orchestra.run.smithi060.stdout: 402883 4 -rw-r--r-- 1 ubuntu ubuntu 1160 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.github/workflows/pr-triage.yml 2024-03-27T21:42:43.572 INFO:teuthology.orchestra.run.smithi060.stdout: 402880 4 -rw-r--r-- 1 ubuntu ubuntu 804 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.github/workflows/needs-rebase.yml 2024-03-27T21:42:43.573 INFO:teuthology.orchestra.run.smithi060.stdout: 402884 4 -rw-r--r-- 1 ubuntu ubuntu 2417 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.github/workflows/stale.yml 2024-03-27T21:42:43.573 INFO:teuthology.orchestra.run.smithi060.stdout: 402882 4 -rw-r--r-- 1 ubuntu ubuntu 396 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.github/workflows/pr-checklist.yml 2024-03-27T21:42:43.573 INFO:teuthology.orchestra.run.smithi060.stdout: 402879 4 -rw-r--r-- 1 ubuntu ubuntu 2152 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.github/workflows/create-backport-trackers.yml 2024-03-27T21:42:43.573 INFO:teuthology.orchestra.run.smithi060.stdout: 402877 4 -rw-r--r-- 1 ubuntu ubuntu 2590 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.github/pull_request_template.md 2024-03-27T21:42:43.573 INFO:teuthology.orchestra.run.smithi060.stdout: 402874 4 -rw-r--r-- 1 ubuntu ubuntu 471 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.github/dependabot.yml 2024-03-27T21:42:43.573 INFO:teuthology.orchestra.run.smithi060.stdout: 402912 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/admin 2024-03-27T21:42:43.573 INFO:teuthology.orchestra.run.smithi060.stdout: 402915 4 -rw-r--r-- 1 ubuntu ubuntu 18 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/admin/doc-python-common-requirements.txt 2024-03-27T21:42:43.573 INFO:teuthology.orchestra.run.smithi060.stdout: 402916 4 -rw-r--r-- 1 ubuntu ubuntu 75 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/admin/doc-read-the-docs.txt 2024-03-27T21:42:43.573 INFO:teuthology.orchestra.run.smithi060.stdout: 402914 4 -rw-r--r-- 1 ubuntu ubuntu 65 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/admin/doc-pybind.txt 2024-03-27T21:42:43.574 INFO:teuthology.orchestra.run.smithi060.stdout: 402917 4 -rw-r--r-- 1 ubuntu ubuntu 426 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/admin/doc-requirements.txt 2024-03-27T21:42:43.574 INFO:teuthology.orchestra.run.smithi060.stdout: 402918 4 -rwxr-xr-x 1 ubuntu ubuntu 848 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/admin/serve-doc 2024-03-27T21:42:43.574 INFO:teuthology.orchestra.run.smithi060.stdout: 402913 4 -rwxr-xr-x 1 ubuntu ubuntu 3770 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/admin/build-doc 2024-03-27T21:42:43.574 INFO:teuthology.orchestra.run.smithi060.stdout: 404013 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring 2024-03-27T21:42:43.574 INFO:teuthology.orchestra.run.smithi060.stdout: 404014 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin 2024-03-27T21:42:43.574 INFO:teuthology.orchestra.run.smithi060.stdout: 404019 4 -rw-r--r-- 1 ubuntu ubuntu 3101 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/README.md 2024-03-27T21:42:43.574 INFO:teuthology.orchestra.run.smithi060.stdout: 404020 4 -rw-r--r-- 1 ubuntu ubuntu 112 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/alerts.jsonnet 2024-03-27T21:42:43.574 INFO:teuthology.orchestra.run.smithi060.stdout: 404017 4 -rw-r--r-- 1 ubuntu ubuntu 2345 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/CMakeLists.txt 2024-03-27T21:42:43.575 INFO:teuthology.orchestra.run.smithi060.stdout: 404065 4 -rw-r--r-- 1 ubuntu ubuntu 305 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/requirements-lint.txt 2024-03-27T21:42:43.575 INFO:teuthology.orchestra.run.smithi060.stdout: 404062 52 -rw-r--r-- 1 ubuntu ubuntu 52182 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/prometheus_alerts.yml 2024-03-27T21:42:43.575 INFO:teuthology.orchestra.run.smithi060.stdout: 404064 4 -rw-r--r-- 1 ubuntu ubuntu 9 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/requirements-grafonnet.txt 2024-03-27T21:42:43.575 INFO:teuthology.orchestra.run.smithi060.stdout: 404093 4 -rw-r--r-- 1 ubuntu ubuntu 1905 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tox.ini 2024-03-27T21:42:43.575 INFO:teuthology.orchestra.run.smithi060.stdout: 404024 4 -rw-r--r-- 1 ubuntu ubuntu 471 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards.libsonnet 2024-03-27T21:42:43.575 INFO:teuthology.orchestra.run.smithi060.stdout: 404067 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_alerts 2024-03-27T21:42:43.575 INFO:teuthology.orchestra.run.smithi060.stdout: 404068 8 -rw-r--r-- 1 ubuntu ubuntu 5399 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_alerts/README.md 2024-03-27T21:42:43.575 INFO:teuthology.orchestra.run.smithi060.stdout: 404070 4 -rw-r--r-- 1 ubuntu ubuntu 345 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_alerts/settings.py 2024-03-27T21:42:43.576 INFO:teuthology.orchestra.run.smithi060.stdout: 404075 20 -rwxr-xr-x 1 ubuntu ubuntu 18342 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_alerts/validate_rules.py 2024-03-27T21:42:43.576 INFO:teuthology.orchestra.run.smithi060.stdout: 404072 4 -rwxr-xr-x 1 ubuntu ubuntu 1176 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_alerts/test_syntax.py 2024-03-27T21:42:43.576 INFO:teuthology.orchestra.run.smithi060.stdout: 404071 112 -rw-r--r-- 1 ubuntu ubuntu 111720 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_alerts/test_alerts.yml 2024-03-27T21:42:43.576 INFO:teuthology.orchestra.run.smithi060.stdout: 404069 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_alerts/__init__.py 2024-03-27T21:42:43.576 INFO:teuthology.orchestra.run.smithi060.stdout: 404073 4 -rw-r--r-- 1 ubuntu ubuntu 603 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_alerts/test_unittests.py 2024-03-27T21:42:43.576 INFO:teuthology.orchestra.run.smithi060.stdout: 404074 4 -rw-r--r-- 1 ubuntu ubuntu 238 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_alerts/utils.py 2024-03-27T21:42:43.576 INFO:teuthology.orchestra.run.smithi060.stdout: 404025 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards 2024-03-27T21:42:43.577 INFO:teuthology.orchestra.run.smithi060.stdout: 404035 28 -rw-r--r-- 1 ubuntu ubuntu 25252 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards/rgw.libsonnet 2024-03-27T21:42:43.577 INFO:teuthology.orchestra.run.smithi060.stdout: 404029 32 -rw-r--r-- 1 ubuntu ubuntu 29028 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards/multi-cluster.libsonnet 2024-03-27T21:42:43.577 INFO:teuthology.orchestra.run.smithi060.stdout: 404030 24 -rw-r--r-- 1 ubuntu ubuntu 20788 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards/osd.libsonnet 2024-03-27T21:42:43.577 INFO:teuthology.orchestra.run.smithi060.stdout: 404034 88 -rw-r--r-- 1 ubuntu ubuntu 86723 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards/rgw-s3-analytics.libsonnet 2024-03-27T21:42:43.577 INFO:teuthology.orchestra.run.smithi060.stdout: 404032 16 -rw-r--r-- 1 ubuntu ubuntu 15942 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards/pool.libsonnet 2024-03-27T21:42:43.577 INFO:teuthology.orchestra.run.smithi060.stdout: 404028 28 -rw-r--r-- 1 ubuntu ubuntu 25730 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards/host.libsonnet 2024-03-27T21:42:43.577 INFO:teuthology.orchestra.run.smithi060.stdout: 404036 4 -rw-r--r-- 1 ubuntu ubuntu 4023 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards/timeseries_panel.libsonnet 2024-03-27T21:42:43.577 INFO:teuthology.orchestra.run.smithi060.stdout: 404027 4 -rw-r--r-- 1 ubuntu ubuntu 2321 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards/cephfs.libsonnet 2024-03-27T21:42:43.578 INFO:teuthology.orchestra.run.smithi060.stdout: 404037 24 -rw-r--r-- 1 ubuntu ubuntu 23124 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards/utils.libsonnet 2024-03-27T21:42:43.578 INFO:teuthology.orchestra.run.smithi060.stdout: 404033 12 -rw-r--r-- 1 ubuntu ubuntu 10567 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards/rbd.libsonnet 2024-03-27T21:42:43.578 INFO:teuthology.orchestra.run.smithi060.stdout: 404026 52 -rw-r--r-- 1 ubuntu ubuntu 50255 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards/ceph-cluster.libsonnet 2024-03-27T21:42:43.578 INFO:teuthology.orchestra.run.smithi060.stdout: 404031 4 -rw-r--r-- 1 ubuntu ubuntu 1702 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards/piechart_panel.libsonnet 2024-03-27T21:42:43.578 INFO:teuthology.orchestra.run.smithi060.stdout: 404058 4 -rw-r--r-- 1 ubuntu ubuntu 357 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/jsonnetfile.lock.json 2024-03-27T21:42:43.578 INFO:teuthology.orchestra.run.smithi060.stdout: 404018 4 -rw-r--r-- 1 ubuntu ubuntu 397 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/Makefile 2024-03-27T21:42:43.578 INFO:teuthology.orchestra.run.smithi060.stdout: 404076 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_dashboards 2024-03-27T21:42:43.578 INFO:teuthology.orchestra.run.smithi060.stdout: 404092 8 -rw-r--r-- 1 ubuntu ubuntu 4256 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_dashboards/util.py 2024-03-27T21:42:43.579 INFO:teuthology.orchestra.run.smithi060.stdout: 404091 4 -rw-r--r-- 1 ubuntu ubuntu 217 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_dashboards/requirements.txt 2024-03-27T21:42:43.579 INFO:teuthology.orchestra.run.smithi060.stdout: 404078 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_dashboards/features 2024-03-27T21:42:43.579 INFO:teuthology.orchestra.run.smithi060.stdout: 404087 16 -rw-r--r-- 1 ubuntu ubuntu 13547 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_dashboards/features/radosgw_overview.feature 2024-03-27T21:42:43.579 INFO:teuthology.orchestra.run.smithi060.stdout: 404080 24 -rw-r--r-- 1 ubuntu ubuntu 23049 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_dashboards/features/ceph-cluster.feature 2024-03-27T21:42:43.579 INFO:teuthology.orchestra.run.smithi060.stdout: 404083 4 -rw-r--r-- 1 ubuntu ubuntu 2185 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_dashboards/features/hosts_overview.feature 2024-03-27T21:42:43.579 INFO:teuthology.orchestra.run.smithi060.stdout: 404085 4 -rw-r--r-- 1 ubuntu ubuntu 815 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_dashboards/features/osds-overview.feature 2024-03-27T21:42:43.579 INFO:teuthology.orchestra.run.smithi060.stdout: 404086 8 -rw-r--r-- 1 ubuntu ubuntu 8014 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_dashboards/features/radosgw-detail.feature 2024-03-27T21:42:43.579 INFO:teuthology.orchestra.run.smithi060.stdout: 404089 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_dashboards/features/steps 2024-03-27T21:42:43.580 INFO:teuthology.orchestra.run.smithi060.stdout: 404090 4 -rw-r--r-- 1 ubuntu ubuntu 79 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_dashboards/features/steps/__init__.py 2024-03-27T21:42:43.580 INFO:teuthology.orchestra.run.smithi060.stdout: 404088 4 -rw-r--r-- 1 ubuntu ubuntu 3973 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_dashboards/features/self.feature 2024-03-27T21:42:43.580 INFO:teuthology.orchestra.run.smithi060.stdout: 404081 8 -rw-r--r-- 1 ubuntu ubuntu 4650 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_dashboards/features/environment.py 2024-03-27T21:42:43.580 INFO:teuthology.orchestra.run.smithi060.stdout: 404084 8 -rw-r--r-- 1 ubuntu ubuntu 5429 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_dashboards/features/osd-device-details.feature 2024-03-27T21:42:43.580 INFO:teuthology.orchestra.run.smithi060.stdout: 404079 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_dashboards/features/__init__.py 2024-03-27T21:42:43.580 INFO:teuthology.orchestra.run.smithi060.stdout: 404082 12 -rw-r--r-- 1 ubuntu ubuntu 9064 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_dashboards/features/host-details.feature 2024-03-27T21:42:43.580 INFO:teuthology.orchestra.run.smithi060.stdout: 404077 8 -rw-r--r-- 1 ubuntu ubuntu 5809 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/tests_dashboards/__init__.py 2024-03-27T21:42:43.580 INFO:teuthology.orchestra.run.smithi060.stdout: 404016 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/.pylintrc -> ../../src/pybind/mgr/dashboard/.pylintrc 2024-03-27T21:42:43.580 INFO:teuthology.orchestra.run.smithi060.stdout: 404023 4 -rw-r--r-- 1 ubuntu ubuntu 140 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards.jsonnet 2024-03-27T21:42:43.581 INFO:teuthology.orchestra.run.smithi060.stdout: 404066 4 -rwxr-xr-x 1 ubuntu ubuntu 1022 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/test-jsonnet.sh 2024-03-27T21:42:43.581 INFO:teuthology.orchestra.run.smithi060.stdout: 404061 60 -rw-r--r-- 1 ubuntu ubuntu 60463 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/prometheus_alerts.libsonnet 2024-03-27T21:42:43.581 INFO:teuthology.orchestra.run.smithi060.stdout: 404022 4 -rw-r--r-- 1 ubuntu ubuntu 634 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/config.libsonnet 2024-03-27T21:42:43.581 INFO:teuthology.orchestra.run.smithi060.stdout: 404063 4 -rw-r--r-- 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/requirements-alerts.txt 2024-03-27T21:42:43.581 INFO:teuthology.orchestra.run.smithi060.stdout: 404059 4 -rwxr-xr-x 1 ubuntu ubuntu 474 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/lint-jsonnet.sh 2024-03-27T21:42:43.581 INFO:teuthology.orchestra.run.smithi060.stdout: 404057 4 -rw-r--r-- 1 ubuntu ubuntu 261 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/jsonnetfile.json 2024-03-27T21:42:43.581 INFO:teuthology.orchestra.run.smithi060.stdout: 404060 4 -rw-r--r-- 1 ubuntu ubuntu 92 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/mixin.libsonnet 2024-03-27T21:42:43.581 INFO:teuthology.orchestra.run.smithi060.stdout: 404015 4 -rw-r--r-- 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/.gitignore 2024-03-27T21:42:43.582 INFO:teuthology.orchestra.run.smithi060.stdout: 404038 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards_out 2024-03-27T21:42:43.582 INFO:teuthology.orchestra.run.smithi060.stdout: 404052 16 -rw-r--r-- 1 ubuntu ubuntu 13843 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards_out/radosgw-sync-overview.json 2024-03-27T21:42:43.582 INFO:teuthology.orchestra.run.smithi060.stdout: 404055 164 -rw-r--r-- 1 ubuntu ubuntu 165387 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards_out/rgw-s3-analytics.json 2024-03-27T21:42:43.582 INFO:teuthology.orchestra.run.smithi060.stdout: 404039 4 -rw-r--r-- 1 ubuntu ubuntu 221 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards_out/.lint 2024-03-27T21:42:43.582 INFO:teuthology.orchestra.run.smithi060.stdout: 404047 36 -rw-r--r-- 1 ubuntu ubuntu 33410 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards_out/osds-overview.json 2024-03-27T21:42:43.582 INFO:teuthology.orchestra.run.smithi060.stdout: 404041 36 -rw-r--r-- 1 ubuntu ubuntu 32787 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards_out/ceph-cluster.json 2024-03-27T21:42:43.583 INFO:teuthology.orchestra.run.smithi060.stdout: 404051 40 -rw-r--r-- 1 ubuntu ubuntu 39749 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards_out/radosgw-overview.json 2024-03-27T21:42:43.583 INFO:teuthology.orchestra.run.smithi060.stdout: 404049 48 -rw-r--r-- 1 ubuntu ubuntu 45460 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards_out/pool-overview.json 2024-03-27T21:42:43.583 INFO:teuthology.orchestra.run.smithi060.stdout: 404054 24 -rw-r--r-- 1 ubuntu ubuntu 22044 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards_out/rbd-overview.json 2024-03-27T21:42:43.583 INFO:teuthology.orchestra.run.smithi060.stdout: 404053 16 -rw-r--r-- 1 ubuntu ubuntu 12811 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards_out/rbd-details.json 2024-03-27T21:42:43.583 INFO:teuthology.orchestra.run.smithi060.stdout: 404050 20 -rw-r--r-- 1 ubuntu ubuntu 16980 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards_out/radosgw-detail.json 2024-03-27T21:42:43.583 INFO:teuthology.orchestra.run.smithi060.stdout: 404042 12 -rw-r--r-- 1 ubuntu ubuntu 9556 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards_out/cephfs-overview.json 2024-03-27T21:42:43.583 INFO:teuthology.orchestra.run.smithi060.stdout: 404046 28 -rw-r--r-- 1 ubuntu ubuntu 27084 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards_out/osd-device-details.json 2024-03-27T21:42:43.583 INFO:teuthology.orchestra.run.smithi060.stdout: 404048 20 -rw-r--r-- 1 ubuntu ubuntu 19980 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards_out/pool-detail.json 2024-03-27T21:42:43.583 INFO:teuthology.orchestra.run.smithi060.stdout: 404044 28 -rw-r--r-- 1 ubuntu ubuntu 27723 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards_out/hosts-overview.json 2024-03-27T21:42:43.584 INFO:teuthology.orchestra.run.smithi060.stdout: 404043 44 -rw-r--r-- 1 ubuntu ubuntu 44123 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards_out/host-details.json 2024-03-27T21:42:43.584 INFO:teuthology.orchestra.run.smithi060.stdout: 404040 116 -rw-r--r-- 1 ubuntu ubuntu 115800 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards_out/ceph-cluster-advanced.json 2024-03-27T21:42:43.584 INFO:teuthology.orchestra.run.smithi060.stdout: 404045 64 -rw-r--r-- 1 ubuntu ubuntu 64728 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/dashboards_out/multi-cluster-overview.json 2024-03-27T21:42:43.584 INFO:teuthology.orchestra.run.smithi060.stdout: 404056 4 -rwxr-xr-x 1 ubuntu ubuntu 233 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/jsonnet-bundler-build.sh 2024-03-27T21:42:43.584 INFO:teuthology.orchestra.run.smithi060.stdout: 404021 4 -rw-r--r-- 1 ubuntu ubuntu 114 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/ceph-mixin/alerts.libsonnet 2024-03-27T21:42:43.584 INFO:teuthology.orchestra.run.smithi060.stdout: 404094 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/grafana 2024-03-27T21:42:43.584 INFO:teuthology.orchestra.run.smithi060.stdout: 404098 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/grafana/screenshots 2024-03-27T21:42:43.584 INFO:teuthology.orchestra.run.smithi060.stdout: 404099 200 -rw-r--r-- 1 ubuntu ubuntu 201798 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/grafana/screenshots/ceph-cluster1.png 2024-03-27T21:42:43.584 INFO:teuthology.orchestra.run.smithi060.stdout: 404105 136 -rw-r--r-- 1 ubuntu ubuntu 137137 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/grafana/screenshots/osd-performance.png 2024-03-27T21:42:43.585 INFO:teuthology.orchestra.run.smithi060.stdout: 404108 100 -rw-r--r-- 1 ubuntu ubuntu 102273 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/grafana/screenshots/pool-overview.png 2024-03-27T21:42:43.585 INFO:teuthology.orchestra.run.smithi060.stdout: 404102 816 -rw-r--r-- 1 ubuntu ubuntu 833169 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/grafana/screenshots/host-details.resized.png 2024-03-27T21:42:43.585 INFO:teuthology.orchestra.run.smithi060.stdout: 404106 352 -rw-r--r-- 1 ubuntu ubuntu 358872 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/grafana/screenshots/osd_overview.png 2024-03-27T21:42:43.585 INFO:teuthology.orchestra.run.smithi060.stdout: 404107 124 -rw-r--r-- 1 ubuntu ubuntu 123041 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/grafana/screenshots/pool-details.png 2024-03-27T21:42:43.585 INFO:teuthology.orchestra.run.smithi060.stdout: 404104 52 -rw-r--r-- 1 ubuntu ubuntu 52682 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/grafana/screenshots/mds-performance.png 2024-03-27T21:42:43.585 INFO:teuthology.orchestra.run.smithi060.stdout: 404103 396 -rw-r--r-- 1 ubuntu ubuntu 403994 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/grafana/screenshots/host_overview.png 2024-03-27T21:42:43.585 INFO:teuthology.orchestra.run.smithi060.stdout: 404100 200 -rw-r--r-- 1 ubuntu ubuntu 204800 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/grafana/screenshots/ceph-cluster2.png 2024-03-27T21:42:43.585 INFO:teuthology.orchestra.run.smithi060.stdout: 404101 212 -rw-r--r-- 1 ubuntu ubuntu 214033 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/grafana/screenshots/host-details.png 2024-03-27T21:42:43.585 INFO:teuthology.orchestra.run.smithi060.stdout: 404109 272 -rw-r--r-- 1 ubuntu ubuntu 277302 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/grafana/screenshots/rgw-detail.png 2024-03-27T21:42:43.586 INFO:teuthology.orchestra.run.smithi060.stdout: 404110 292 -rw-r--r-- 1 ubuntu ubuntu 296415 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/grafana/screenshots/rgw-overview.png 2024-03-27T21:42:43.586 INFO:teuthology.orchestra.run.smithi060.stdout: 404095 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/grafana/build 2024-03-27T21:42:43.586 INFO:teuthology.orchestra.run.smithi060.stdout: 404097 4 -rw-r--r-- 1 ubuntu ubuntu 1744 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/grafana/build/README.md 2024-03-27T21:42:43.586 INFO:teuthology.orchestra.run.smithi060.stdout: 404096 4 -rwxr-xr-x 1 ubuntu ubuntu 3554 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/grafana/build/Makefile 2024-03-27T21:42:43.586 INFO:teuthology.orchestra.run.smithi060.stdout: 404111 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/snmp 2024-03-27T21:42:43.586 INFO:teuthology.orchestra.run.smithi060.stdout: 404113 4 -rw-r--r-- 1 ubuntu ubuntu 2837 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/snmp/README.md 2024-03-27T21:42:43.586 INFO:teuthology.orchestra.run.smithi060.stdout: 404112 12 -rw-r--r-- 1 ubuntu ubuntu 11727 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/monitoring/snmp/CEPH-MIB.txt 2024-03-27T21:42:43.586 INFO:teuthology.orchestra.run.smithi060.stdout: 402904 4 -rw-r--r-- 1 ubuntu ubuntu 1457 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/README.aix 2024-03-27T21:42:43.587 INFO:teuthology.orchestra.run.smithi060.stdout: 402908 4 -rw-r--r-- 1 ubuntu ubuntu 3893 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/SECURITY.md 2024-03-27T21:42:43.587 INFO:teuthology.orchestra.run.smithi060.stdout: 417157 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/udev 2024-03-27T21:42:43.587 INFO:teuthology.orchestra.run.smithi060.stdout: 417158 4 -rw-r--r-- 1 ubuntu ubuntu 362 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/udev/50-rbd.rules 2024-03-27T21:42:43.587 INFO:teuthology.orchestra.run.smithi060.stdout: 408959 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/selinux 2024-03-27T21:42:43.587 INFO:teuthology.orchestra.run.smithi060.stdout: 408961 4 -rw-r--r-- 1 ubuntu ubuntu 375 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/selinux/CMakeLists.txt 2024-03-27T21:42:43.587 INFO:teuthology.orchestra.run.smithi060.stdout: 408963 8 -rw-r--r-- 1 ubuntu ubuntu 4964 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/selinux/ceph.if 2024-03-27T21:42:43.587 INFO:teuthology.orchestra.run.smithi060.stdout: 408964 8 -rw-r--r-- 1 ubuntu ubuntu 4822 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/selinux/ceph.te 2024-03-27T21:42:43.587 INFO:teuthology.orchestra.run.smithi060.stdout: 408962 4 -rw-r--r-- 1 ubuntu ubuntu 1147 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/selinux/ceph.fc 2024-03-27T21:42:43.587 INFO:teuthology.orchestra.run.smithi060.stdout: 408960 4 -rw-r--r-- 1 ubuntu ubuntu 14 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/selinux/.gitignore 2024-03-27T21:42:43.588 INFO:teuthology.orchestra.run.smithi060.stdout: 403999 4 -rwxr-xr-x 1 ubuntu ubuntu 259 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/make-srpm.sh 2024-03-27T21:42:43.588 INFO:teuthology.orchestra.run.smithi060.stdout: 402899 28 -rw-r--r-- 1 ubuntu ubuntu 26436 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/COPYING-LGPL2.1 2024-03-27T21:42:43.588 INFO:teuthology.orchestra.run.smithi060.stdout: 408970 4 drwxr-xr-x 77 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src 2024-03-27T21:42:43.588 INFO:teuthology.orchestra.run.smithi060.stdout: 410505 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/dmclock 2024-03-27T21:42:43.588 INFO:teuthology.orchestra.run.smithi060.stdout: 417125 60 -rwxr-xr-x 1 ubuntu ubuntu 60013 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/vstart.sh 2024-03-27T21:42:43.588 INFO:teuthology.orchestra.run.smithi060.stdout: 410580 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/exporter 2024-03-27T21:42:43.588 INFO:teuthology.orchestra.run.smithi060.stdout: 410581 4 -rw-r--r-- 1 ubuntu ubuntu 254 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/exporter/CMakeLists.txt 2024-03-27T21:42:43.588 INFO:teuthology.orchestra.run.smithi060.stdout: 410586 4 -rw-r--r-- 1 ubuntu ubuntu 71 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/exporter/http_server.h 2024-03-27T21:42:43.589 INFO:teuthology.orchestra.run.smithi060.stdout: 410585 8 -rw-r--r-- 1 ubuntu ubuntu 5568 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/exporter/http_server.cc 2024-03-27T21:42:43.589 INFO:teuthology.orchestra.run.smithi060.stdout: 410583 4 -rw-r--r-- 1 ubuntu ubuntu 2722 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/exporter/DaemonMetricCollector.h 2024-03-27T21:42:43.589 INFO:teuthology.orchestra.run.smithi060.stdout: 410584 4 -rw-r--r-- 1 ubuntu ubuntu 2378 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/exporter/ceph_exporter.cc 2024-03-27T21:42:43.589 INFO:teuthology.orchestra.run.smithi060.stdout: 410587 4 -rw-r--r-- 1 ubuntu ubuntu 1572 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/exporter/util.cc 2024-03-27T21:42:43.589 INFO:teuthology.orchestra.run.smithi060.stdout: 410588 4 -rw-r--r-- 1 ubuntu ubuntu 499 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/exporter/util.h 2024-03-27T21:42:43.589 INFO:teuthology.orchestra.run.smithi060.stdout: 410582 20 -rw-r--r-- 1 ubuntu ubuntu 17326 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/exporter/DaemonMetricCollector.cc 2024-03-27T21:42:43.590 INFO:teuthology.orchestra.run.smithi060.stdout: 412196 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind 2024-03-27T21:42:43.590 INFO:teuthology.orchestra.run.smithi060.stdout: 412209 4 drwxr-xr-x 40 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-27T21:42:43.590 INFO:teuthology.orchestra.run.smithi060.stdout: 414506 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/restful 2024-03-27T21:42:43.590 INFO:teuthology.orchestra.run.smithi060.stdout: 414523 20 -rw-r--r-- 1 ubuntu ubuntu 18203 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/restful/module.py 2024-03-27T21:42:43.590 INFO:teuthology.orchestra.run.smithi060.stdout: 414520 4 -rw-r--r-- 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/restful/context.py 2024-03-27T21:42:43.590 INFO:teuthology.orchestra.run.smithi060.stdout: 414522 4 -rw-r--r-- 1 ubuntu ubuntu 204 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/restful/hooks.py 2024-03-27T21:42:43.590 INFO:teuthology.orchestra.run.smithi060.stdout: 414519 8 -rw-r--r-- 1 ubuntu ubuntu 4740 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/restful/common.py 2024-03-27T21:42:43.590 INFO:teuthology.orchestra.run.smithi060.stdout: 414507 4 -rw-r--r-- 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/restful/__init__.py 2024-03-27T21:42:43.591 INFO:teuthology.orchestra.run.smithi060.stdout: 414521 4 -rw-r--r-- 1 ubuntu ubuntu 2270 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/restful/decorators.py 2024-03-27T21:42:43.591 INFO:teuthology.orchestra.run.smithi060.stdout: 414508 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/restful/api 2024-03-27T21:42:43.591 INFO:teuthology.orchestra.run.smithi060.stdout: 414517 4 -rw-r--r-- 1 ubuntu ubuntu 2604 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/restful/api/request.py 2024-03-27T21:42:43.591 INFO:teuthology.orchestra.run.smithi060.stdout: 414510 4 -rw-r--r-- 1 ubuntu ubuntu 1999 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/restful/api/config.py 2024-03-27T21:42:43.591 INFO:teuthology.orchestra.run.smithi060.stdout: 414516 8 -rw-r--r-- 1 ubuntu ubuntu 4111 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/restful/api/pool.py 2024-03-27T21:42:43.591 INFO:teuthology.orchestra.run.smithi060.stdout: 414513 4 -rw-r--r-- 1 ubuntu ubuntu 953 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/restful/api/mon.py 2024-03-27T21:42:43.591 INFO:teuthology.orchestra.run.smithi060.stdout: 414518 4 -rw-r--r-- 1 ubuntu ubuntu 737 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/restful/api/server.py 2024-03-27T21:42:43.591 INFO:teuthology.orchestra.run.smithi060.stdout: 414512 4 -rw-r--r-- 1 ubuntu ubuntu 320 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/restful/api/doc.py 2024-03-27T21:42:43.591 INFO:teuthology.orchestra.run.smithi060.stdout: 414515 4 -rw-r--r-- 1 ubuntu ubuntu 688 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/restful/api/perf.py 2024-03-27T21:42:43.592 INFO:teuthology.orchestra.run.smithi060.stdout: 414511 4 -rw-r--r-- 1 ubuntu ubuntu 561 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/restful/api/crush.py 2024-03-27T21:42:43.592 INFO:teuthology.orchestra.run.smithi060.stdout: 414509 4 -rw-r--r-- 1 ubuntu ubuntu 990 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/restful/api/__init__.py 2024-03-27T21:42:43.592 INFO:teuthology.orchestra.run.smithi060.stdout: 414514 4 -rw-r--r-- 1 ubuntu ubuntu 3542 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/restful/api/osd.py 2024-03-27T21:42:43.592 INFO:teuthology.orchestra.run.smithi060.stdout: 414428 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mds_autoscaler 2024-03-27T21:42:43.592 INFO:teuthology.orchestra.run.smithi060.stdout: 414430 4 -rw-r--r-- 1 ubuntu ubuntu 3748 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mds_autoscaler/module.py 2024-03-27T21:42:43.592 INFO:teuthology.orchestra.run.smithi060.stdout: 414431 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mds_autoscaler/tests 2024-03-27T21:42:43.592 INFO:teuthology.orchestra.run.smithi060.stdout: 414433 4 -rw-r--r-- 1 ubuntu ubuntu 2566 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mds_autoscaler/tests/test_autoscaler.py 2024-03-27T21:42:43.592 INFO:teuthology.orchestra.run.smithi060.stdout: 414432 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mds_autoscaler/tests/__init__.py 2024-03-27T21:42:43.593 INFO:teuthology.orchestra.run.smithi060.stdout: 414429 4 -rw-r--r-- 1 ubuntu ubuntu 92 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mds_autoscaler/__init__.py 2024-03-27T21:42:43.593 INFO:teuthology.orchestra.run.smithi060.stdout: 412307 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cli_api 2024-03-27T21:42:43.593 INFO:teuthology.orchestra.run.smithi060.stdout: 412309 4 -rwxr-xr-x 1 ubuntu ubuntu 3987 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cli_api/module.py 2024-03-27T21:42:43.593 INFO:teuthology.orchestra.run.smithi060.stdout: 412310 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cli_api/tests 2024-03-27T21:42:43.593 INFO:teuthology.orchestra.run.smithi060.stdout: 412311 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cli_api/tests/__init__.py 2024-03-27T21:42:43.593 INFO:teuthology.orchestra.run.smithi060.stdout: 412312 4 -rw-r--r-- 1 ubuntu ubuntu 1718 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cli_api/tests/test_cli_api.py 2024-03-27T21:42:43.593 INFO:teuthology.orchestra.run.smithi060.stdout: 412308 4 -rw-r--r-- 1 ubuntu ubuntu 182 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cli_api/__init__.py 2024-03-27T21:42:43.593 INFO:teuthology.orchestra.run.smithi060.stdout: 414603 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/tests 2024-03-27T21:42:43.593 INFO:teuthology.orchestra.run.smithi060.stdout: 414606 16 -rw-r--r-- 1 ubuntu ubuntu 16229 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/tests/test_object_format.py 2024-03-27T21:42:43.594 INFO:teuthology.orchestra.run.smithi060.stdout: 414607 4 -rw-r--r-- 1 ubuntu ubuntu 3786 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/tests/test_tls.py 2024-03-27T21:42:43.594 INFO:teuthology.orchestra.run.smithi060.stdout: 414604 8 -rw-r--r-- 1 ubuntu ubuntu 8010 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/tests/__init__.py 2024-03-27T21:42:43.594 INFO:teuthology.orchestra.run.smithi060.stdout: 414605 4 -rw-r--r-- 1 ubuntu ubuntu 513 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/tests/test_mgr_util.py 2024-03-27T21:42:43.594 INFO:teuthology.orchestra.run.smithi060.stdout: 414473 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/osd_perf_query 2024-03-27T21:42:43.594 INFO:teuthology.orchestra.run.smithi060.stdout: 414475 8 -rw-r--r-- 1 ubuntu ubuntu 7796 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/osd_perf_query/module.py 2024-03-27T21:42:43.594 INFO:teuthology.orchestra.run.smithi060.stdout: 414474 4 -rw-r--r-- 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/osd_perf_query/__init__.py 2024-03-27T21:42:43.594 INFO:teuthology.orchestra.run.smithi060.stdout: 414559 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/snap_schedule 2024-03-27T21:42:43.594 INFO:teuthology.orchestra.run.smithi060.stdout: 414566 16 -rw-r--r-- 1 ubuntu ubuntu 14744 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/snap_schedule/module.py 2024-03-27T21:42:43.595 INFO:teuthology.orchestra.run.smithi060.stdout: 414568 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/snap_schedule/tests 2024-03-27T21:42:43.595 INFO:teuthology.orchestra.run.smithi060.stdout: 414571 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/snap_schedule/tests/fs 2024-03-27T21:42:43.595 INFO:teuthology.orchestra.run.smithi060.stdout: 414573 12 -rw-r--r-- 1 ubuntu ubuntu 9126 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/snap_schedule/tests/fs/test_schedule.py 2024-03-27T21:42:43.595 INFO:teuthology.orchestra.run.smithi060.stdout: 414574 4 -rw-r--r-- 1 ubuntu ubuntu 1459 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/snap_schedule/tests/fs/test_schedule_client.py 2024-03-27T21:42:43.595 INFO:teuthology.orchestra.run.smithi060.stdout: 414572 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/snap_schedule/tests/fs/__init__.py 2024-03-27T21:42:43.595 INFO:teuthology.orchestra.run.smithi060.stdout: 414570 4 -rw-r--r-- 1 ubuntu ubuntu 1998 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/snap_schedule/tests/conftest.py 2024-03-27T21:42:43.595 INFO:teuthology.orchestra.run.smithi060.stdout: 414569 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/snap_schedule/tests/__init__.py 2024-03-27T21:42:43.595 INFO:teuthology.orchestra.run.smithi060.stdout: 414562 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/snap_schedule/fs 2024-03-27T21:42:43.596 INFO:teuthology.orchestra.run.smithi060.stdout: 414564 20 -rw-r--r-- 1 ubuntu ubuntu 19624 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/snap_schedule/fs/schedule.py 2024-03-27T21:42:43.596 INFO:teuthology.orchestra.run.smithi060.stdout: 414563 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/snap_schedule/fs/__init__.py 2024-03-27T21:42:43.596 INFO:teuthology.orchestra.run.smithi060.stdout: 414565 24 -rw-r--r-- 1 ubuntu ubuntu 20760 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/snap_schedule/fs/schedule_client.py 2024-03-27T21:42:43.596 INFO:teuthology.orchestra.run.smithi060.stdout: 414575 4 -rw-r--r-- 1 ubuntu ubuntu 518 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/snap_schedule/tox.ini 2024-03-27T21:42:43.596 INFO:teuthology.orchestra.run.smithi060.stdout: 414567 4 -rw-r--r-- 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/snap_schedule/requirements.txt 2024-03-27T21:42:43.596 INFO:teuthology.orchestra.run.smithi060.stdout: 414561 4 -rw-r--r-- 1 ubuntu ubuntu 216 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/snap_schedule/__init__.py 2024-03-27T21:42:43.596 INFO:teuthology.orchestra.run.smithi060.stdout: 414560 4 -rw-r--r-- 1 ubuntu ubuntu 5 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/snap_schedule/.gitignore 2024-03-27T21:42:43.597 INFO:teuthology.orchestra.run.smithi060.stdout: 412212 4 -rw-r--r-- 1 ubuntu ubuntu 1350 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/CMakeLists.txt 2024-03-27T21:42:43.597 INFO:teuthology.orchestra.run.smithi060.stdout: 412316 4 drwxr-xr-x 10 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard 2024-03-27T21:42:43.597 INFO:teuthology.orchestra.run.smithi060.stdout: 414212 24 -rw-r--r-- 1 ubuntu ubuntu 23100 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/module.py 2024-03-27T21:42:43.597 INFO:teuthology.orchestra.run.smithi060.stdout: 414305 32 -rw-r--r-- 1 ubuntu ubuntu 32179 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tools.py 2024-03-27T21:42:43.597 INFO:teuthology.orchestra.run.smithi060.stdout: 414208 8 -rw-r--r-- 1 ubuntu ubuntu 4443 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/grafana.py 2024-03-27T21:42:43.597 INFO:teuthology.orchestra.run.smithi060.stdout: 414264 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests 2024-03-27T21:42:43.597 INFO:teuthology.orchestra.run.smithi060.stdout: 414294 8 -rw-r--r-- 1 ubuntu ubuntu 6939 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_rbd_service.py 2024-03-27T21:42:43.597 INFO:teuthology.orchestra.run.smithi060.stdout: 414303 8 -rw-r--r-- 1 ubuntu ubuntu 7442 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_tools.py 2024-03-27T21:42:43.597 INFO:teuthology.orchestra.run.smithi060.stdout: 414281 4 -rw-r--r-- 1 ubuntu ubuntu 2082 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_feature_toggles.py 2024-03-27T21:42:43.598 INFO:teuthology.orchestra.run.smithi060.stdout: 414299 8 -rw-r--r-- 1 ubuntu ubuntu 7147 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_settings.py 2024-03-27T21:42:43.598 INFO:teuthology.orchestra.run.smithi060.stdout: 414270 4 -rw-r--r-- 1 ubuntu ubuntu 1416 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_cache.py 2024-03-27T21:42:43.598 INFO:teuthology.orchestra.run.smithi060.stdout: 414278 12 -rw-r--r-- 1 ubuntu ubuntu 8653 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_docs.py 2024-03-27T21:42:43.598 INFO:teuthology.orchestra.run.smithi060.stdout: 414290 4 -rw-r--r-- 1 ubuntu ubuntu 1049 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_plugin_debug.py 2024-03-27T21:42:43.598 INFO:teuthology.orchestra.run.smithi060.stdout: 414292 8 -rw-r--r-- 1 ubuntu ubuntu 8102 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_prometheus.py 2024-03-27T21:42:43.598 INFO:teuthology.orchestra.run.smithi060.stdout: 414291 8 -rw-r--r-- 1 ubuntu ubuntu 6012 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_pool.py 2024-03-27T21:42:43.598 INFO:teuthology.orchestra.run.smithi060.stdout: 414304 4 -rw-r--r-- 1 ubuntu ubuntu 2450 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_versioning.py 2024-03-27T21:42:43.599 INFO:teuthology.orchestra.run.smithi060.stdout: 414289 20 -rw-r--r-- 1 ubuntu ubuntu 19390 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_osd.py 2024-03-27T21:42:43.599 INFO:teuthology.orchestra.run.smithi060.stdout: 414267 40 -rw-r--r-- 1 ubuntu ubuntu 38042 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_access_control.py 2024-03-27T21:42:43.599 INFO:teuthology.orchestra.run.smithi060.stdout: 414297 12 -rw-r--r-- 1 ubuntu ubuntu 11406 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_rgw.py 2024-03-27T21:42:43.599 INFO:teuthology.orchestra.run.smithi060.stdout: 414273 4 -rw-r--r-- 1 ubuntu ubuntu 1193 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_cephfs.py 2024-03-27T21:42:43.599 INFO:teuthology.orchestra.run.smithi060.stdout: 414295 8 -rw-r--r-- 1 ubuntu ubuntu 4871 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_rest_client.py 2024-03-27T21:42:43.599 INFO:teuthology.orchestra.run.smithi060.stdout: 414300 4 -rw-r--r-- 1 ubuntu ubuntu 1237 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_ssl.py 2024-03-27T21:42:43.599 INFO:teuthology.orchestra.run.smithi060.stdout: 414301 12 -rw-r--r-- 1 ubuntu ubuntu 8867 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_sso.py 2024-03-27T21:42:43.599 INFO:teuthology.orchestra.run.smithi060.stdout: 414276 4 -rw-r--r-- 1 ubuntu ubuntu 2606 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_crud.py 2024-03-27T21:42:43.599 INFO:teuthology.orchestra.run.smithi060.stdout: 414282 8 -rw-r--r-- 1 ubuntu ubuntu 4870 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_grafana.py 2024-03-27T21:42:43.600 INFO:teuthology.orchestra.run.smithi060.stdout: 414302 20 -rw-r--r-- 1 ubuntu ubuntu 16715 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_task.py 2024-03-27T21:42:43.600 INFO:teuthology.orchestra.run.smithi060.stdout: 414269 4 -rw-r--r-- 1 ubuntu ubuntu 2349 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_auth.py 2024-03-27T21:42:43.600 INFO:teuthology.orchestra.run.smithi060.stdout: 414285 56 -rw-r--r-- 1 ubuntu ubuntu 53551 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_iscsi.py 2024-03-27T21:42:43.600 INFO:teuthology.orchestra.run.smithi060.stdout: 414280 8 -rw-r--r-- 1 ubuntu ubuntu 4523 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_exceptions.py 2024-03-27T21:42:43.600 INFO:teuthology.orchestra.run.smithi060.stdout: 414287 8 -rw-r--r-- 1 ubuntu ubuntu 4800 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_notification.py 2024-03-27T21:42:43.600 INFO:teuthology.orchestra.run.smithi060.stdout: 414283 4 -rw-r--r-- 1 ubuntu ubuntu 2839 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_home.py 2024-03-27T21:42:43.600 INFO:teuthology.orchestra.run.smithi060.stdout: 414286 12 -rw-r--r-- 1 ubuntu ubuntu 8762 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_nfs.py 2024-03-27T21:42:43.600 INFO:teuthology.orchestra.run.smithi060.stdout: 414296 4 -rw-r--r-- 1 ubuntu ubuntu 2790 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_rest_tasks.py 2024-03-27T21:42:43.600 INFO:teuthology.orchestra.run.smithi060.stdout: 414265 16 -rw-r--r-- 1 ubuntu ubuntu 13889 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/__init__.py 2024-03-27T21:42:43.601 INFO:teuthology.orchestra.run.smithi060.stdout: 414266 4 -rw-r--r-- 1 ubuntu ubuntu 1534 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/helper.py 2024-03-27T21:42:43.601 INFO:teuthology.orchestra.run.smithi060.stdout: 414284 24 -rw-r--r-- 1 ubuntu ubuntu 21363 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_host.py 2024-03-27T21:42:43.601 INFO:teuthology.orchestra.run.smithi060.stdout: 414288 4 -rw-r--r-- 1 ubuntu ubuntu 1363 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_orchestrator.py 2024-03-27T21:42:43.601 INFO:teuthology.orchestra.run.smithi060.stdout: 414272 4 -rw-r--r-- 1 ubuntu ubuntu 1769 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_ceph_users.py 2024-03-27T21:42:43.601 INFO:teuthology.orchestra.run.smithi060.stdout: 414277 4 -rw-r--r-- 1 ubuntu ubuntu 1488 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_daemon.py 2024-03-27T21:42:43.601 INFO:teuthology.orchestra.run.smithi060.stdout: 414268 4 -rw-r--r-- 1 ubuntu ubuntu 2970 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_api_auditing.py 2024-03-27T21:42:43.601 INFO:teuthology.orchestra.run.smithi060.stdout: 414274 4 -rw-r--r-- 1 ubuntu ubuntu 2307 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_cluster_upgrade.py 2024-03-27T21:42:43.601 INFO:teuthology.orchestra.run.smithi060.stdout: 414279 4 -rw-r--r-- 1 ubuntu ubuntu 837 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_erasure_code_profile.py 2024-03-27T21:42:43.602 INFO:teuthology.orchestra.run.smithi060.stdout: 414293 12 -rw-r--r-- 1 ubuntu ubuntu 10907 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_rbd_mirroring.py 2024-03-27T21:42:43.602 INFO:teuthology.orchestra.run.smithi060.stdout: 414271 8 -rw-r--r-- 1 ubuntu ubuntu 6852 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_ceph_service.py 2024-03-27T21:42:43.602 INFO:teuthology.orchestra.run.smithi060.stdout: 414298 16 -rw-r--r-- 1 ubuntu ubuntu 14697 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_rgw_client.py 2024-03-27T21:42:43.602 INFO:teuthology.orchestra.run.smithi060.stdout: 414275 8 -rw-r--r-- 1 ubuntu ubuntu 6390 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tests/test_controllers.py 2024-03-27T21:42:43.602 INFO:teuthology.orchestra.run.smithi060.stdout: 412321 4 -rw-r--r-- 1 ubuntu ubuntu 859 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/CMakeLists.txt 2024-03-27T21:42:43.602 INFO:teuthology.orchestra.run.smithi060.stdout: 414263 12 -rw-r--r-- 1 ubuntu ubuntu 9205 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/settings.py 2024-03-27T21:42:43.602 INFO:teuthology.orchestra.run.smithi060.stdout: 412328 8 -rw-r--r-- 1 ubuntu ubuntu 6732 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/awsauth.py 2024-03-27T21:42:43.603 INFO:teuthology.orchestra.run.smithi060.stdout: 414225 4 -rw-r--r-- 1 ubuntu ubuntu 236 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/requirements-lint.txt 2024-03-27T21:42:43.603 INFO:teuthology.orchestra.run.smithi060.stdout: 414214 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/plugins 2024-03-27T21:42:43.603 INFO:teuthology.orchestra.run.smithi060.stdout: 414221 4 -rw-r--r-- 1 ubuntu ubuntu 3898 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/plugins/pluggy.py 2024-03-27T21:42:43.603 INFO:teuthology.orchestra.run.smithi060.stdout: 414218 4 -rw-r--r-- 1 ubuntu ubuntu 1580 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/plugins/interfaces.py 2024-03-27T21:42:43.603 INFO:teuthology.orchestra.run.smithi060.stdout: 414223 4 -rw-r--r-- 1 ubuntu ubuntu 3486 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/plugins/ttl_cache.py 2024-03-27T21:42:43.603 INFO:teuthology.orchestra.run.smithi060.stdout: 414215 4 -rw-r--r-- 1 ubuntu ubuntu 2420 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/plugins/__init__.py 2024-03-27T21:42:43.603 INFO:teuthology.orchestra.run.smithi060.stdout: 414217 8 -rw-r--r-- 1 ubuntu ubuntu 5576 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/plugins/feature_toggles.py 2024-03-27T21:42:43.603 INFO:teuthology.orchestra.run.smithi060.stdout: 414216 4 -rw-r--r-- 1 ubuntu ubuntu 2872 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/plugins/debug.py 2024-03-27T21:42:43.604 INFO:teuthology.orchestra.run.smithi060.stdout: 414220 4 -rw-r--r-- 1 ubuntu ubuntu 3212 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/plugins/motd.py 2024-03-27T21:42:43.604 INFO:teuthology.orchestra.run.smithi060.stdout: 414219 4 -rw-r--r-- 1 ubuntu ubuntu 1307 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/plugins/lru_cache.py 2024-03-27T21:42:43.604 INFO:teuthology.orchestra.run.smithi060.stdout: 414222 4 -rw-r--r-- 1 ubuntu ubuntu 1253 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/plugins/plugin.py 2024-03-27T21:42:43.604 INFO:teuthology.orchestra.run.smithi060.stdout: 414233 4 -rwxr-xr-x 1 ubuntu ubuntu 1367 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/run-frontend-unittests.sh 2024-03-27T21:42:43.604 INFO:teuthology.orchestra.run.smithi060.stdout: 414224 4 -rw-r--r-- 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/requirements-extra.txt 2024-03-27T21:42:43.604 INFO:teuthology.orchestra.run.smithi060.stdout: 414234 4 -rw-r--r-- 1 ubuntu ubuntu 1482 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/security.py 2024-03-27T21:42:43.604 INFO:teuthology.orchestra.run.smithi060.stdout: 412337 4 -rw-r--r-- 1 ubuntu ubuntu 93 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/constraints.txt 2024-03-27T21:42:43.604 INFO:teuthology.orchestra.run.smithi060.stdout: 412394 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend 2024-03-27T21:42:43.605 INFO:teuthology.orchestra.run.smithi060.stdout: 412408 4 -rw-r--r-- 1 ubuntu ubuntu 161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/babel.config.js 2024-03-27T21:42:43.605 INFO:teuthology.orchestra.run.smithi060.stdout: 412528 4 -rw-r--r-- 1 ubuntu ubuntu 242 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/ngcc.config.js 2024-03-27T21:42:43.605 INFO:teuthology.orchestra.run.smithi060.stdout: 412403 4 -rw-r--r-- 1 ubuntu ubuntu 101 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/.prettierrc 2024-03-27T21:42:43.605 INFO:teuthology.orchestra.run.smithi060.stdout: 412529 1272 -rw-r--r-- 1 ubuntu ubuntu 1301840 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/package-lock.json 2024-03-27T21:42:43.605 INFO:teuthology.orchestra.run.smithi060.stdout: 412405 8 -rw-r--r-- 1 ubuntu ubuntu 5191 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/CMakeLists.txt 2024-03-27T21:42:43.605 INFO:teuthology.orchestra.run.smithi060.stdout: 412409 8 -rwxr-xr-x 1 ubuntu ubuntu 4576 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cd.js 2024-03-27T21:42:43.605 INFO:teuthology.orchestra.run.smithi060.stdout: 412404 4 -rw-r--r-- 1 ubuntu ubuntu 1137 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/.stylelintrc 2024-03-27T21:42:43.605 INFO:teuthology.orchestra.run.smithi060.stdout: 412532 4 drwxr-xr-x 8 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src 2024-03-27T21:42:43.605 INFO:teuthology.orchestra.run.smithi060.stdout: 414161 4 -rw-r--r-- 1 ubuntu ubuntu 132 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/jestGlobalMocks.ts 2024-03-27T21:42:43.606 INFO:teuthology.orchestra.run.smithi060.stdout: 414162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/locale 2024-03-27T21:42:43.606 INFO:teuthology.orchestra.run.smithi060.stdout: 414163 308 -rw-r--r-- 1 ubuntu ubuntu 313712 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/locale/messages.cs.xlf 2024-03-27T21:42:43.606 INFO:teuthology.orchestra.run.smithi060.stdout: 414165 308 -rw-r--r-- 1 ubuntu ubuntu 313872 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/locale/messages.es-ES.xlf 2024-03-27T21:42:43.606 INFO:teuthology.orchestra.run.smithi060.stdout: 414174 308 -rw-r--r-- 1 ubuntu ubuntu 311502 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/locale/messages.zh-TW.xlf 2024-03-27T21:42:43.606 INFO:teuthology.orchestra.run.smithi060.stdout: 414170 308 -rw-r--r-- 1 ubuntu ubuntu 314024 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/locale/messages.ko-KR.xlf 2024-03-27T21:42:43.606 INFO:teuthology.orchestra.run.smithi060.stdout: 414167 308 -rw-r--r-- 1 ubuntu ubuntu 311974 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/locale/messages.id-ID.xlf 2024-03-27T21:42:43.606 INFO:teuthology.orchestra.run.smithi060.stdout: 414169 312 -rw-r--r-- 1 ubuntu ubuntu 315976 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/locale/messages.ja-JP.xlf 2024-03-27T21:42:43.606 INFO:teuthology.orchestra.run.smithi060.stdout: 414172 308 -rw-r--r-- 1 ubuntu ubuntu 313420 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/locale/messages.pt-BR.xlf 2024-03-27T21:42:43.607 INFO:teuthology.orchestra.run.smithi060.stdout: 414168 308 -rw-r--r-- 1 ubuntu ubuntu 314101 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/locale/messages.it-IT.xlf 2024-03-27T21:42:43.607 INFO:teuthology.orchestra.run.smithi060.stdout: 414166 308 -rw-r--r-- 1 ubuntu ubuntu 314184 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/locale/messages.fr-FR.xlf 2024-03-27T21:42:43.607 INFO:teuthology.orchestra.run.smithi060.stdout: 414164 308 -rw-r--r-- 1 ubuntu ubuntu 313942 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/locale/messages.de-DE.xlf 2024-03-27T21:42:43.607 INFO:teuthology.orchestra.run.smithi060.stdout: 414171 308 -rw-r--r-- 1 ubuntu ubuntu 312904 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/locale/messages.pl-PL.xlf 2024-03-27T21:42:43.607 INFO:teuthology.orchestra.run.smithi060.stdout: 414173 308 -rw-r--r-- 1 ubuntu ubuntu 311581 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/locale/messages.zh-CN.xlf 2024-03-27T21:42:43.607 INFO:teuthology.orchestra.run.smithi060.stdout: 414175 4 -rw-r--r-- 1 ubuntu ubuntu 880 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/main.ts 2024-03-27T21:42:43.607 INFO:teuthology.orchestra.run.smithi060.stdout: 414157 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/environments 2024-03-27T21:42:43.607 INFO:teuthology.orchestra.run.smithi060.stdout: 414158 4 -rw-r--r-- 1 ubuntu ubuntu 458 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/environments/environment.tpl.ts 2024-03-27T21:42:43.608 INFO:teuthology.orchestra.run.smithi060.stdout: 414176 4 -rw-r--r-- 1 ubuntu ubuntu 1773 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/polyfills.ts 2024-03-27T21:42:43.608 INFO:teuthology.orchestra.run.smithi060.stdout: 414160 4 -rw-r--r-- 1 ubuntu ubuntu 734 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/index.html 2024-03-27T21:42:43.608 INFO:teuthology.orchestra.run.smithi060.stdout: 414159 4 -rw-r--r-- 1 ubuntu ubuntu 1150 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/favicon.ico 2024-03-27T21:42:43.608 INFO:teuthology.orchestra.run.smithi060.stdout: 414179 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles 2024-03-27T21:42:43.608 INFO:teuthology.orchestra.run.smithi060.stdout: 414197 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles/vendor 2024-03-27T21:42:43.608 INFO:teuthology.orchestra.run.smithi060.stdout: 414200 4 -rw-r--r-- 1 ubuntu ubuntu 391 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles/vendor/_variables.scss 2024-03-27T21:42:43.608 INFO:teuthology.orchestra.run.smithi060.stdout: 414198 4 -rw-r--r-- 1 ubuntu ubuntu 528 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles/vendor/_index.scss 2024-03-27T21:42:43.608 INFO:teuthology.orchestra.run.smithi060.stdout: 414199 4 -rw-r--r-- 1 ubuntu ubuntu 98 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles/vendor/_style-overrides.scss 2024-03-27T21:42:43.608 INFO:teuthology.orchestra.run.smithi060.stdout: 414182 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles/ceph-custom 2024-03-27T21:42:43.609 INFO:teuthology.orchestra.run.smithi060.stdout: 414186 4 -rw-r--r-- 1 ubuntu ubuntu 1488 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles/ceph-custom/_forms.scss 2024-03-27T21:42:43.609 INFO:teuthology.orchestra.run.smithi060.stdout: 414185 4 -rw-r--r-- 1 ubuntu ubuntu 566 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles/ceph-custom/_dropdown.scss 2024-03-27T21:42:43.609 INFO:teuthology.orchestra.run.smithi060.stdout: 414189 4 -rw-r--r-- 1 ubuntu ubuntu 96 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles/ceph-custom/_index.scss 2024-03-27T21:42:43.609 INFO:teuthology.orchestra.run.smithi060.stdout: 414187 4 -rw-r--r-- 1 ubuntu ubuntu 103 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles/ceph-custom/_grid.scss 2024-03-27T21:42:43.609 INFO:teuthology.orchestra.run.smithi060.stdout: 414190 4 -rw-r--r-- 1 ubuntu ubuntu 94 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles/ceph-custom/_navs.scss 2024-03-27T21:42:43.609 INFO:teuthology.orchestra.run.smithi060.stdout: 414188 4 -rw-r--r-- 1 ubuntu ubuntu 301 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles/ceph-custom/_icons.scss 2024-03-27T21:42:43.609 INFO:teuthology.orchestra.run.smithi060.stdout: 414184 4 -rw-r--r-- 1 ubuntu ubuntu 1896 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles/ceph-custom/_buttons.scss 2024-03-27T21:42:43.610 INFO:teuthology.orchestra.run.smithi060.stdout: 414191 4 -rw-r--r-- 1 ubuntu ubuntu 443 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles/ceph-custom/_toast.scss 2024-03-27T21:42:43.610 INFO:teuthology.orchestra.run.smithi060.stdout: 414183 4 -rw-r--r-- 1 ubuntu ubuntu 1309 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles/ceph-custom/_basics.scss 2024-03-27T21:42:43.610 INFO:teuthology.orchestra.run.smithi060.stdout: 414181 4 -rw-r--r-- 1 ubuntu ubuntu 1717 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles/bootstrap-extends.scss 2024-03-27T21:42:43.610 INFO:teuthology.orchestra.run.smithi060.stdout: 414180 4 -rw-r--r-- 1 ubuntu ubuntu 1049 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles/_chart-tooltip.scss 2024-03-27T21:42:43.610 INFO:teuthology.orchestra.run.smithi060.stdout: 414192 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles/defaults 2024-03-27T21:42:43.610 INFO:teuthology.orchestra.run.smithi060.stdout: 414193 4 -rw-r--r-- 1 ubuntu ubuntu 4069 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles/defaults/_bootstrap-defaults.scss 2024-03-27T21:42:43.610 INFO:teuthology.orchestra.run.smithi060.stdout: 414194 4 -rw-r--r-- 1 ubuntu ubuntu 96 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles/defaults/_functions.scss 2024-03-27T21:42:43.611 INFO:teuthology.orchestra.run.smithi060.stdout: 414195 4 -rw-r--r-- 1 ubuntu ubuntu 64 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles/defaults/_index.scss 2024-03-27T21:42:43.611 INFO:teuthology.orchestra.run.smithi060.stdout: 414196 4 -rw-r--r-- 1 ubuntu ubuntu 871 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles/defaults/_mixins.scss 2024-03-27T21:42:43.611 INFO:teuthology.orchestra.run.smithi060.stdout: 414177 4 -rw-r--r-- 1 ubuntu ubuntu 430 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/setupJest.ts 2024-03-27T21:42:43.611 INFO:teuthology.orchestra.run.smithi060.stdout: 414204 4 -rw-r--r-- 1 ubuntu ubuntu 104 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/typings.d.ts 2024-03-27T21:42:43.611 INFO:teuthology.orchestra.run.smithi060.stdout: 414178 8 -rw-r--r-- 1 ubuntu ubuntu 4184 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/styles.scss 2024-03-27T21:42:43.611 INFO:teuthology.orchestra.run.smithi060.stdout: 414201 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/testing 2024-03-27T21:42:43.611 INFO:teuthology.orchestra.run.smithi060.stdout: 414202 4 -rw-r--r-- 1 ubuntu ubuntu 746 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/testing/activated-route-stub.ts 2024-03-27T21:42:43.611 INFO:teuthology.orchestra.run.smithi060.stdout: 414203 24 -rw-r--r-- 1 ubuntu ubuntu 20809 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/testing/unit-test-helper.ts 2024-03-27T21:42:43.611 INFO:teuthology.orchestra.run.smithi060.stdout: 412533 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app 2024-03-27T21:42:43.612 INFO:teuthology.orchestra.run.smithi060.stdout: 413492 4 drwxr-xr-x 16 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared 2024-03-27T21:42:43.612 INFO:teuthology.orchestra.run.smithi060.stdout: 413796 4 drwxr-xr-x 8 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable 2024-03-27T21:42:43.612 INFO:teuthology.orchestra.run.smithi060.stdout: 413813 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-key-value 2024-03-27T21:42:43.612 INFO:teuthology.orchestra.run.smithi060.stdout: 413814 4 -rw-r--r-- 1 ubuntu ubuntu 383 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-key-value/table-key-value.component.html 2024-03-27T21:42:43.612 INFO:teuthology.orchestra.run.smithi060.stdout: 413816 12 -rw-r--r-- 1 ubuntu ubuntu 11424 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-key-value/table-key-value.component.spec.ts 2024-03-27T21:42:43.612 INFO:teuthology.orchestra.run.smithi060.stdout: 413815 4 -rw-r--r-- 1 ubuntu ubuntu 74 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-key-value/table-key-value.component.scss 2024-03-27T21:42:43.612 INFO:teuthology.orchestra.run.smithi060.stdout: 413817 8 -rw-r--r-- 1 ubuntu ubuntu 6160 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-key-value/table-key-value.component.ts 2024-03-27T21:42:43.612 INFO:teuthology.orchestra.run.smithi060.stdout: 413823 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table 2024-03-27T21:42:43.613 INFO:teuthology.orchestra.run.smithi060.stdout: 413826 28 -rw-r--r-- 1 ubuntu ubuntu 26419 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table/table.component.spec.ts 2024-03-27T21:42:43.613 INFO:teuthology.orchestra.run.smithi060.stdout: 413825 8 -rw-r--r-- 1 ubuntu ubuntu 5744 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table/table.component.scss 2024-03-27T21:42:43.613 INFO:teuthology.orchestra.run.smithi060.stdout: 413824 16 -rw-r--r-- 1 ubuntu ubuntu 13133 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table/table.component.html 2024-03-27T21:42:43.613 INFO:teuthology.orchestra.run.smithi060.stdout: 413827 32 -rw-r--r-- 1 ubuntu ubuntu 28945 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table/table.component.ts 2024-03-27T21:42:43.613 INFO:teuthology.orchestra.run.smithi060.stdout: 413802 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/crud-table 2024-03-27T21:42:43.613 INFO:teuthology.orchestra.run.smithi060.stdout: 413803 4 -rw-r--r-- 1 ubuntu ubuntu 2458 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/crud-table/crud-table.component.html 2024-03-27T21:42:43.613 INFO:teuthology.orchestra.run.smithi060.stdout: 413806 8 -rw-r--r-- 1 ubuntu ubuntu 7234 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/crud-table/crud-table.component.ts 2024-03-27T21:42:43.613 INFO:teuthology.orchestra.run.smithi060.stdout: 413804 4 -rw-r--r-- 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/crud-table/crud-table.component.scss 2024-03-27T21:42:43.613 INFO:teuthology.orchestra.run.smithi060.stdout: 413805 4 -rw-r--r-- 1 ubuntu ubuntu 1841 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/crud-table/crud-table.component.spec.ts 2024-03-27T21:42:43.614 INFO:teuthology.orchestra.run.smithi060.stdout: 413808 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-actions 2024-03-27T21:42:43.614 INFO:teuthology.orchestra.run.smithi060.stdout: 413811 8 -rw-r--r-- 1 ubuntu ubuntu 7396 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-actions/table-actions.component.spec.ts 2024-03-27T21:42:43.614 INFO:teuthology.orchestra.run.smithi060.stdout: 413810 4 -rw-r--r-- 1 ubuntu ubuntu 291 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-actions/table-actions.component.scss 2024-03-27T21:42:43.614 INFO:teuthology.orchestra.run.smithi060.stdout: 413809 4 -rw-r--r-- 1 ubuntu ubuntu 1863 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-actions/table-actions.component.html 2024-03-27T21:42:43.614 INFO:teuthology.orchestra.run.smithi060.stdout: 413812 8 -rw-r--r-- 1 ubuntu ubuntu 4958 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-actions/table-actions.component.ts 2024-03-27T21:42:43.614 INFO:teuthology.orchestra.run.smithi060.stdout: 413807 8 -rw-r--r-- 1 ubuntu ubuntu 4283 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/datatable.module.ts 2024-03-27T21:42:43.615 INFO:teuthology.orchestra.run.smithi060.stdout: 413797 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/checked-table-form 2024-03-27T21:42:43.615 INFO:teuthology.orchestra.run.smithi060.stdout: 413798 4 -rw-r--r-- 1 ubuntu ubuntu 1947 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/checked-table-form/checked-table-form.component.html 2024-03-27T21:42:43.615 INFO:teuthology.orchestra.run.smithi060.stdout: 413801 8 -rw-r--r-- 1 ubuntu ubuntu 6311 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/checked-table-form/checked-table-form.component.ts 2024-03-27T21:42:43.615 INFO:teuthology.orchestra.run.smithi060.stdout: 413799 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/checked-table-form/checked-table-form.component.scss 2024-03-27T21:42:43.615 INFO:teuthology.orchestra.run.smithi060.stdout: 413800 8 -rw-r--r-- 1 ubuntu ubuntu 4756 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/checked-table-form/checked-table-form.component.spec.ts 2024-03-27T21:42:43.615 INFO:teuthology.orchestra.run.smithi060.stdout: 413818 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-pagination 2024-03-27T21:42:43.615 INFO:teuthology.orchestra.run.smithi060.stdout: 413820 4 -rw-r--r-- 1 ubuntu ubuntu 315 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-pagination/table-pagination.component.scss 2024-03-27T21:42:43.615 INFO:teuthology.orchestra.run.smithi060.stdout: 413821 4 -rw-r--r-- 1 ubuntu ubuntu 1595 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-pagination/table-pagination.component.spec.ts 2024-03-27T21:42:43.616 INFO:teuthology.orchestra.run.smithi060.stdout: 413822 4 -rw-r--r-- 1 ubuntu ubuntu 2348 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-pagination/table-pagination.component.ts 2024-03-27T21:42:43.616 INFO:teuthology.orchestra.run.smithi060.stdout: 413819 4 -rw-r--r-- 1 ubuntu ubuntu 1464 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-pagination/table-pagination.component.html 2024-03-27T21:42:43.616 INFO:teuthology.orchestra.run.smithi060.stdout: 413864 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/enum 2024-03-27T21:42:43.616 INFO:teuthology.orchestra.run.smithi060.stdout: 413873 4 -rw-r--r-- 1 ubuntu ubuntu 60 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/enum/notification-type.enum.ts 2024-03-27T21:42:43.616 INFO:teuthology.orchestra.run.smithi060.stdout: 413871 4 -rw-r--r-- 1 ubuntu ubuntu 98 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/enum/health-label.enum.ts 2024-03-27T21:42:43.616 INFO:teuthology.orchestra.run.smithi060.stdout: 413876 4 -rw-r--r-- 1 ubuntu ubuntu 110 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/enum/unix_errno.enum.ts 2024-03-27T21:42:43.616 INFO:teuthology.orchestra.run.smithi060.stdout: 413877 4 -rw-r--r-- 1 ubuntu ubuntu 103 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/enum/view-cache-status.enum.ts 2024-03-27T21:42:43.616 INFO:teuthology.orchestra.run.smithi060.stdout: 413875 4 -rw-r--r-- 1 ubuntu ubuntu 325 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/enum/retention-frequency.enum.ts 2024-03-27T21:42:43.616 INFO:teuthology.orchestra.run.smithi060.stdout: 413867 4 -rw-r--r-- 1 ubuntu ubuntu 3184 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/enum/dashboard-promqls.enum.ts 2024-03-27T21:42:43.617 INFO:teuthology.orchestra.run.smithi060.stdout: 413870 4 -rw-r--r-- 1 ubuntu ubuntu 241 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/enum/health-icon.enum.ts 2024-03-27T21:42:43.617 INFO:teuthology.orchestra.run.smithi060.stdout: 413866 4 -rw-r--r-- 1 ubuntu ubuntu 146 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/enum/components.enum.ts 2024-03-27T21:42:43.617 INFO:teuthology.orchestra.run.smithi060.stdout: 413874 4 -rw-r--r-- 1 ubuntu ubuntu 332 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/enum/repeat-frequency.enum.ts 2024-03-27T21:42:43.617 INFO:teuthology.orchestra.run.smithi060.stdout: 413865 4 -rw-r--r-- 1 ubuntu ubuntu 2007 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/enum/cell-template.enum.ts 2024-03-27T21:42:43.617 INFO:teuthology.orchestra.run.smithi060.stdout: 413872 8 -rw-r--r-- 1 ubuntu ubuntu 4469 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/enum/icons.enum.ts 2024-03-27T21:42:43.617 INFO:teuthology.orchestra.run.smithi060.stdout: 413869 4 -rw-r--r-- 1 ubuntu ubuntu 142 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/enum/health-color.enum.ts 2024-03-27T21:42:43.617 INFO:teuthology.orchestra.run.smithi060.stdout: 413868 4 -rw-r--r-- 1 ubuntu ubuntu 171 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/enum/hardware.enum.ts 2024-03-27T21:42:43.617 INFO:teuthology.orchestra.run.smithi060.stdout: 413986 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes 2024-03-27T21:42:43.618 INFO:teuthology.orchestra.run.smithi060.stdout: 413997 4 -rw-r--r-- 1 ubuntu ubuntu 618 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/ceph-short-version.pipe.spec.ts 2024-03-27T21:42:43.618 INFO:teuthology.orchestra.run.smithi060.stdout: 414000 4 -rw-r--r-- 1 ubuntu ubuntu 1489 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/dimless-binary.pipe.spec.ts 2024-03-27T21:42:43.618 INFO:teuthology.orchestra.run.smithi060.stdout: 414057 4 -rw-r--r-- 1 ubuntu ubuntu 544 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/truncate.pipe.spec.ts 2024-03-27T21:42:43.618 INFO:teuthology.orchestra.run.smithi060.stdout: 414011 4 -rw-r--r-- 1 ubuntu ubuntu 534 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/filter.pipe.ts 2024-03-27T21:42:43.618 INFO:teuthology.orchestra.run.smithi060.stdout: 414022 4 -rw-r--r-- 1 ubuntu ubuntu 277 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/join.pipe.spec.ts 2024-03-27T21:42:43.618 INFO:teuthology.orchestra.run.smithi060.stdout: 414019 4 -rw-r--r-- 1 ubuntu ubuntu 194 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/iops.pipe.ts 2024-03-27T21:42:43.618 INFO:teuthology.orchestra.run.smithi060.stdout: 413996 4 -rw-r--r-- 1 ubuntu ubuntu 679 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/ceph-release-name.pipe.ts 2024-03-27T21:42:43.618 INFO:teuthology.orchestra.run.smithi060.stdout: 414001 4 -rw-r--r-- 1 ubuntu ubuntu 485 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/dimless-binary.pipe.ts 2024-03-27T21:42:43.619 INFO:teuthology.orchestra.run.smithi060.stdout: 414012 4 -rw-r--r-- 1 ubuntu ubuntu 1131 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/health-color.pipe.spec.ts 2024-03-27T21:42:43.619 INFO:teuthology.orchestra.run.smithi060.stdout: 414058 4 -rw-r--r-- 1 ubuntu ubuntu 384 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/truncate.pipe.ts 2024-03-27T21:42:43.619 INFO:teuthology.orchestra.run.smithi060.stdout: 414010 4 -rw-r--r-- 1 ubuntu ubuntu 1282 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/filter.pipe.spec.ts 2024-03-27T21:42:43.619 INFO:teuthology.orchestra.run.smithi060.stdout: 414052 4 -rw-r--r-- 1 ubuntu ubuntu 250 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/round.pipe.ts 2024-03-27T21:42:43.619 INFO:teuthology.orchestra.run.smithi060.stdout: 413994 4 -rw-r--r-- 1 ubuntu ubuntu 723 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/cd-date.pipe.ts 2024-03-27T21:42:43.619 INFO:teuthology.orchestra.run.smithi060.stdout: 414046 4 -rw-r--r-- 1 ubuntu ubuntu 299 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/pluralize.pipe.ts 2024-03-27T21:42:43.619 INFO:teuthology.orchestra.run.smithi060.stdout: 413995 4 -rw-r--r-- 1 ubuntu ubuntu 823 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/ceph-release-name.pipe.spec.ts 2024-03-27T21:42:43.619 INFO:teuthology.orchestra.run.smithi060.stdout: 414039 4 -rw-r--r-- 1 ubuntu ubuntu 476 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/ordinal.pipe.ts 2024-03-27T21:42:43.620 INFO:teuthology.orchestra.run.smithi060.stdout: 414050 4 -rw-r--r-- 1 ubuntu ubuntu 1721 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/relative-date.pipe.ts 2024-03-27T21:42:43.620 INFO:teuthology.orchestra.run.smithi060.stdout: 413989 4 -rw-r--r-- 1 ubuntu ubuntu 835 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/boolean-text.pipe.spec.ts 2024-03-27T21:42:43.620 INFO:teuthology.orchestra.run.smithi060.stdout: 414053 4 -rw-r--r-- 1 ubuntu ubuntu 735 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/sanitize-html.pipe.spec.ts 2024-03-27T21:42:43.620 INFO:teuthology.orchestra.run.smithi060.stdout: 413988 4 -rwxr-xr-x 1 ubuntu ubuntu 688 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/array.pipe.ts 2024-03-27T21:42:43.620 INFO:teuthology.orchestra.run.smithi060.stdout: 414016 4 -rw-r--r-- 1 ubuntu ubuntu 610 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/health-label.pipe.spec.ts 2024-03-27T21:42:43.620 INFO:teuthology.orchestra.run.smithi060.stdout: 414036 4 -rw-r--r-- 1 ubuntu ubuntu 1085 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/octal-to-human-readable.pipe.spec.ts 2024-03-27T21:42:43.620 INFO:teuthology.orchestra.run.smithi060.stdout: 414005 4 -rw-r--r-- 1 ubuntu ubuntu 1239 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/duration.pipe.ts 2024-03-27T21:42:43.621 INFO:teuthology.orchestra.run.smithi060.stdout: 414043 4 -rw-r--r-- 1 ubuntu ubuntu 482 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/path.pipe.ts 2024-03-27T21:42:43.621 INFO:teuthology.orchestra.run.smithi060.stdout: 414029 4 -rw-r--r-- 1 ubuntu ubuntu 1190 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/mds-summary.pipe.ts 2024-03-27T21:42:43.621 INFO:teuthology.orchestra.run.smithi060.stdout: 414023 4 -rw-r--r-- 1 ubuntu ubuntu 205 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/join.pipe.ts 2024-03-27T21:42:43.621 INFO:teuthology.orchestra.run.smithi060.stdout: 414040 4 -rw-r--r-- 1 ubuntu ubuntu 969 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/osd-summary.pipe.spec.ts 2024-03-27T21:42:43.621 INFO:teuthology.orchestra.run.smithi060.stdout: 414003 4 -rw-r--r-- 1 ubuntu ubuntu 456 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/dimless.pipe.ts 2024-03-27T21:42:43.621 INFO:teuthology.orchestra.run.smithi060.stdout: 414031 4 -rw-r--r-- 1 ubuntu ubuntu 858 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/mgr-summary.pipe.ts 2024-03-27T21:42:43.621 INFO:teuthology.orchestra.run.smithi060.stdout: 414045 4 -rw-r--r-- 1 ubuntu ubuntu 199 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/pluralize.pipe.spec.ts 2024-03-27T21:42:43.621 INFO:teuthology.orchestra.run.smithi060.stdout: 414028 4 -rw-r--r-- 1 ubuntu ubuntu 1652 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/mds-summary.pipe.spec.ts 2024-03-27T21:42:43.621 INFO:teuthology.orchestra.run.smithi060.stdout: 414054 4 -rw-r--r-- 1 ubuntu ubuntu 422 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/sanitize-html.pipe.ts 2024-03-27T21:42:43.622 INFO:teuthology.orchestra.run.smithi060.stdout: 414037 4 -rw-r--r-- 1 ubuntu ubuntu 2359 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/octal-to-human-readable.pipe.ts 2024-03-27T21:42:43.622 INFO:teuthology.orchestra.run.smithi060.stdout: 414027 4 -rw-r--r-- 1 ubuntu ubuntu 311 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/map.pipe.ts 2024-03-27T21:42:43.622 INFO:teuthology.orchestra.run.smithi060.stdout: 414013 4 -rw-r--r-- 1 ubuntu ubuntu 506 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/health-color.pipe.ts 2024-03-27T21:42:43.622 INFO:teuthology.orchestra.run.smithi060.stdout: 414048 4 -rw-r--r-- 1 ubuntu ubuntu 315 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/rbd-configuration-source.pipe.ts 2024-03-27T21:42:43.622 INFO:teuthology.orchestra.run.smithi060.stdout: 414049 4 -rw-r--r-- 1 ubuntu ubuntu 1445 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/relative-date.pipe.spec.ts 2024-03-27T21:42:43.622 INFO:teuthology.orchestra.run.smithi060.stdout: 414051 4 -rw-r--r-- 1 ubuntu ubuntu 273 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/round.pipe.spec.ts 2024-03-27T21:42:43.622 INFO:teuthology.orchestra.run.smithi060.stdout: 413992 4 -rwxr-xr-x 1 ubuntu ubuntu 472 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/boolean.pipe.ts 2024-03-27T21:42:43.623 INFO:teuthology.orchestra.run.smithi060.stdout: 414020 4 -rw-r--r-- 1 ubuntu ubuntu 428 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/iscsi-backstore.pipe.spec.ts 2024-03-27T21:42:43.623 INFO:teuthology.orchestra.run.smithi060.stdout: 413991 4 -rwxr-xr-x 1 ubuntu ubuntu 1309 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/boolean.pipe.spec.ts 2024-03-27T21:42:43.623 INFO:teuthology.orchestra.run.smithi060.stdout: 413987 4 -rwxr-xr-x 1 ubuntu ubuntu 523 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/array.pipe.spec.ts 2024-03-27T21:42:43.623 INFO:teuthology.orchestra.run.smithi060.stdout: 414008 4 -rw-r--r-- 1 ubuntu ubuntu 348 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/encode-uri.pipe.spec.ts 2024-03-27T21:42:43.623 INFO:teuthology.orchestra.run.smithi060.stdout: 414018 4 -rw-r--r-- 1 ubuntu ubuntu 179 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/iops.pipe.spec.ts 2024-03-27T21:42:43.623 INFO:teuthology.orchestra.run.smithi060.stdout: 414004 4 -rw-r--r-- 1 ubuntu ubuntu 517 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/duration.pipe.spec.ts 2024-03-27T21:42:43.623 INFO:teuthology.orchestra.run.smithi060.stdout: 414032 4 -rw-r--r-- 1 ubuntu ubuntu 211 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/milliseconds.pipe.spec.ts 2024-03-27T21:42:43.623 INFO:teuthology.orchestra.run.smithi060.stdout: 414007 4 -rw-r--r-- 1 ubuntu ubuntu 339 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/empty.pipe.ts 2024-03-27T21:42:43.623 INFO:teuthology.orchestra.run.smithi060.stdout: 414033 4 -rw-r--r-- 1 ubuntu ubuntu 208 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/milliseconds.pipe.ts 2024-03-27T21:42:43.624 INFO:teuthology.orchestra.run.smithi060.stdout: 414059 4 -rw-r--r-- 1 ubuntu ubuntu 383 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/upper-first.pipe.spec.ts 2024-03-27T21:42:43.624 INFO:teuthology.orchestra.run.smithi060.stdout: 414055 4 -rw-r--r-- 1 ubuntu ubuntu 1247 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/search-highlight.pipe.spec.ts 2024-03-27T21:42:43.624 INFO:teuthology.orchestra.run.smithi060.stdout: 414024 4 -rw-r--r-- 1 ubuntu ubuntu 753 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/log-priority.pipe.spec.ts 2024-03-27T21:42:43.624 INFO:teuthology.orchestra.run.smithi060.stdout: 414021 4 -rw-r--r-- 1 ubuntu ubuntu 314 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/iscsi-backstore.pipe.ts 2024-03-27T21:42:43.624 INFO:teuthology.orchestra.run.smithi060.stdout: 414060 4 -rw-r--r-- 1 ubuntu ubuntu 241 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/upper-first.pipe.ts 2024-03-27T21:42:43.624 INFO:teuthology.orchestra.run.smithi060.stdout: 414025 4 -rw-r--r-- 1 ubuntu ubuntu 448 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/log-priority.pipe.ts 2024-03-27T21:42:43.624 INFO:teuthology.orchestra.run.smithi060.stdout: 414047 4 -rw-r--r-- 1 ubuntu ubuntu 658 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/rbd-configuration-source.pipe.spec.ts 2024-03-27T21:42:43.624 INFO:teuthology.orchestra.run.smithi060.stdout: 413998 4 -rw-r--r-- 1 ubuntu ubuntu 482 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/ceph-short-version.pipe.ts 2024-03-27T21:42:43.625 INFO:teuthology.orchestra.run.smithi060.stdout: 413993 4 -rw-r--r-- 1 ubuntu ubuntu 609 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/cd-date.pipe.spec.ts 2024-03-27T21:42:43.625 INFO:teuthology.orchestra.run.smithi060.stdout: 414038 4 -rw-r--r-- 1 ubuntu ubuntu 191 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/ordinal.pipe.spec.ts 2024-03-27T21:42:43.625 INFO:teuthology.orchestra.run.smithi060.stdout: 414009 4 -rw-r--r-- 1 ubuntu ubuntu 214 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/encode-uri.pipe.ts 2024-03-27T21:42:43.625 INFO:teuthology.orchestra.run.smithi060.stdout: 414044 8 -rwxr-xr-x 1 ubuntu ubuntu 4283 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/pipes.module.ts 2024-03-27T21:42:43.625 INFO:teuthology.orchestra.run.smithi060.stdout: 413999 4 -rw-r--r-- 1 ubuntu ubuntu 521 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/dimless-binary-per-second.pipe.ts 2024-03-27T21:42:43.626 INFO:teuthology.orchestra.run.smithi060.stdout: 414041 4 -rw-r--r-- 1 ubuntu ubuntu 943 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/osd-summary.pipe.ts 2024-03-27T21:42:43.626 INFO:teuthology.orchestra.run.smithi060.stdout: 414002 4 -rw-r--r-- 1 ubuntu ubuntu 1446 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/dimless.pipe.spec.ts 2024-03-27T21:42:43.626 INFO:teuthology.orchestra.run.smithi060.stdout: 413990 4 -rw-r--r-- 1 ubuntu ubuntu 323 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/boolean-text.pipe.ts 2024-03-27T21:42:43.626 INFO:teuthology.orchestra.run.smithi060.stdout: 414030 4 -rw-r--r-- 1 ubuntu ubuntu 865 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/mgr-summary.pipe.spec.ts 2024-03-27T21:42:43.626 INFO:teuthology.orchestra.run.smithi060.stdout: 414042 4 -rw-r--r-- 1 ubuntu ubuntu 482 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/path.pipe.spec.ts 2024-03-27T21:42:43.626 INFO:teuthology.orchestra.run.smithi060.stdout: 414035 4 -rw-r--r-- 1 ubuntu ubuntu 318 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/not-available.pipe.ts 2024-03-27T21:42:43.626 INFO:teuthology.orchestra.run.smithi060.stdout: 414034 4 -rw-r--r-- 1 ubuntu ubuntu 699 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/not-available.pipe.spec.ts 2024-03-27T21:42:43.626 INFO:teuthology.orchestra.run.smithi060.stdout: 414014 4 -rw-r--r-- 1 ubuntu ubuntu 573 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/health-icon.pipe.spec.ts 2024-03-27T21:42:43.627 INFO:teuthology.orchestra.run.smithi060.stdout: 414056 4 -rw-r--r-- 1 ubuntu ubuntu 604 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/search-highlight.pipe.ts 2024-03-27T21:42:43.627 INFO:teuthology.orchestra.run.smithi060.stdout: 414006 4 -rw-r--r-- 1 ubuntu ubuntu 404 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/empty.pipe.spec.ts 2024-03-27T21:42:43.627 INFO:teuthology.orchestra.run.smithi060.stdout: 414026 4 -rw-r--r-- 1 ubuntu ubuntu 628 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/map.pipe.spec.ts 2024-03-27T21:42:43.627 INFO:teuthology.orchestra.run.smithi060.stdout: 414017 4 -rw-r--r-- 1 ubuntu ubuntu 344 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/health-label.pipe.ts 2024-03-27T21:42:43.627 INFO:teuthology.orchestra.run.smithi060.stdout: 414015 4 -rw-r--r-- 1 ubuntu ubuntu 331 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/pipes/health-icon.pipe.ts 2024-03-27T21:42:43.627 INFO:teuthology.orchestra.run.smithi060.stdout: 413589 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/classes 2024-03-27T21:42:43.627 INFO:teuthology.orchestra.run.smithi060.stdout: 413597 4 -rw-r--r-- 1 ubuntu ubuntu 1224 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/classes/table-status-view-cache.ts 2024-03-27T21:42:43.627 INFO:teuthology.orchestra.run.smithi060.stdout: 413594 4 -rw-r--r-- 1 ubuntu ubuntu 161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/classes/css-helper.ts 2024-03-27T21:42:43.627 INFO:teuthology.orchestra.run.smithi060.stdout: 413590 4 -rw-r--r-- 1 ubuntu ubuntu 1461 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/classes/cd-helper.class.spec.ts 2024-03-27T21:42:43.628 INFO:teuthology.orchestra.run.smithi060.stdout: 413595 4 -rw-r--r-- 1 ubuntu ubuntu 729 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/classes/list-with-details.class.ts 2024-03-27T21:42:43.628 INFO:teuthology.orchestra.run.smithi060.stdout: 413592 8 -rw-r--r-- 1 ubuntu ubuntu 7717 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/classes/crush.node.selection.class.spec.ts 2024-03-27T21:42:43.628 INFO:teuthology.orchestra.run.smithi060.stdout: 413591 4 -rw-r--r-- 1 ubuntu ubuntu 924 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/classes/cd-helper.class.ts 2024-03-27T21:42:43.628 INFO:teuthology.orchestra.run.smithi060.stdout: 413593 8 -rw-r--r-- 1 ubuntu ubuntu 6996 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/classes/crush.node.selection.class.ts 2024-03-27T21:42:43.628 INFO:teuthology.orchestra.run.smithi060.stdout: 413598 4 -rw-r--r-- 1 ubuntu ubuntu 433 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/classes/table-status.spec.ts 2024-03-27T21:42:43.628 INFO:teuthology.orchestra.run.smithi060.stdout: 413596 4 -rw-r--r-- 1 ubuntu ubuntu 1582 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/classes/table-status-view-cache.spec.ts 2024-03-27T21:42:43.628 INFO:teuthology.orchestra.run.smithi060.stdout: 413599 4 -rw-r--r-- 1 ubuntu ubuntu 127 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/classes/table-status.ts 2024-03-27T21:42:43.628 INFO:teuthology.orchestra.run.smithi060.stdout: 413929 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/helpers 2024-03-27T21:42:43.629 INFO:teuthology.orchestra.run.smithi060.stdout: 413930 4 -rw-r--r-- 1 ubuntu ubuntu 182 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/helpers/helpers.module.ts 2024-03-27T21:42:43.629 INFO:teuthology.orchestra.run.smithi060.stdout: 413931 4 -rw-r--r-- 1 ubuntu ubuntu 757 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/helpers/prometheus-list-helper.ts 2024-03-27T21:42:43.629 INFO:teuthology.orchestra.run.smithi060.stdout: 413932 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models 2024-03-27T21:42:43.629 INFO:teuthology.orchestra.run.smithi060.stdout: 413965 4 -rw-r--r-- 1 ubuntu ubuntu 774 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/image-spec.ts 2024-03-27T21:42:43.629 INFO:teuthology.orchestra.run.smithi060.stdout: 413981 4 -rw-r--r-- 1 ubuntu ubuntu 382 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/summary.model.ts 2024-03-27T21:42:43.629 INFO:teuthology.orchestra.run.smithi060.stdout: 413980 4 -rw-r--r-- 1 ubuntu ubuntu 761 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/snapshot-schedule.ts 2024-03-27T21:42:43.629 INFO:teuthology.orchestra.run.smithi060.stdout: 413967 4 -rw-r--r-- 1 ubuntu ubuntu 149 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/login-response.ts 2024-03-27T21:42:43.629 INFO:teuthology.orchestra.run.smithi060.stdout: 413957 4 -rw-r--r-- 1 ubuntu ubuntu 463 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/crush-rule.ts 2024-03-27T21:42:43.629 INFO:teuthology.orchestra.run.smithi060.stdout: 413941 4 -rw-r--r-- 1 ubuntu ubuntu 1596 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/cd-table-action.ts 2024-03-27T21:42:43.629 INFO:teuthology.orchestra.run.smithi060.stdout: 413975 4 -rw-r--r-- 1 ubuntu ubuntu 1747 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/permissions.ts 2024-03-27T21:42:43.630 INFO:teuthology.orchestra.run.smithi060.stdout: 413939 4 -rw-r--r-- 1 ubuntu ubuntu 397 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/cd-pwd-expiration-settings.ts 2024-03-27T21:42:43.630 INFO:teuthology.orchestra.run.smithi060.stdout: 413934 4 -rw-r--r-- 1 ubuntu ubuntu 2179 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/breadcrumbs.ts 2024-03-27T21:42:43.630 INFO:teuthology.orchestra.run.smithi060.stdout: 413961 4 -rw-r--r-- 1 ubuntu ubuntu 375 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/erasure-code-profile.ts 2024-03-27T21:42:43.630 INFO:teuthology.orchestra.run.smithi060.stdout: 413943 4 -rw-r--r-- 1 ubuntu ubuntu 357 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/cd-table-column-filters-change.ts 2024-03-27T21:42:43.630 INFO:teuthology.orchestra.run.smithi060.stdout: 413949 4 -rw-r--r-- 1 ubuntu ubuntu 403 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/cephfs-directory-models.ts 2024-03-27T21:42:43.630 INFO:teuthology.orchestra.run.smithi060.stdout: 413958 4 -rw-r--r-- 1 ubuntu ubuntu 113 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/crush-step.ts 2024-03-27T21:42:43.630 INFO:teuthology.orchestra.run.smithi060.stdout: 413933 4 -rw-r--r-- 1 ubuntu ubuntu 565 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/alertmanager-silence.ts 2024-03-27T21:42:43.630 INFO:teuthology.orchestra.run.smithi060.stdout: 413948 4 -rw-r--r-- 1 ubuntu ubuntu 252 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/cd-user-config.ts 2024-03-27T21:42:43.631 INFO:teuthology.orchestra.run.smithi060.stdout: 413969 4 -rw-r--r-- 1 ubuntu ubuntu 210 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/multi-cluster.ts 2024-03-27T21:42:43.631 INFO:teuthology.orchestra.run.smithi060.stdout: 413935 4 -rw-r--r-- 1 ubuntu ubuntu 104 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/cd-details.ts 2024-03-27T21:42:43.631 INFO:teuthology.orchestra.run.smithi060.stdout: 413964 4 -rw-r--r-- 1 ubuntu ubuntu 175 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/flag.ts 2024-03-27T21:42:43.631 INFO:teuthology.orchestra.run.smithi060.stdout: 413971 4 -rw-r--r-- 1 ubuntu ubuntu 158 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/orchestrator.interface.ts 2024-03-27T21:42:43.631 INFO:teuthology.orchestra.run.smithi060.stdout: 413951 4 -rw-r--r-- 1 ubuntu ubuntu 517 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/cephfs-subvolume.model.ts 2024-03-27T21:42:43.631 INFO:teuthology.orchestra.run.smithi060.stdout: 413976 4 -rw-r--r-- 1 ubuntu ubuntu 677 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/pool-form-info.ts 2024-03-27T21:42:43.631 INFO:teuthology.orchestra.run.smithi060.stdout: 413962 4 -rw-r--r-- 1 ubuntu ubuntu 118 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/executing-task.ts 2024-03-27T21:42:43.631 INFO:teuthology.orchestra.run.smithi060.stdout: 413947 4 -rw-r--r-- 1 ubuntu ubuntu 903 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/cd-table-selection.ts 2024-03-27T21:42:43.632 INFO:teuthology.orchestra.run.smithi060.stdout: 413974 4 -rw-r--r-- 1 ubuntu ubuntu 3533 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/permission.spec.ts 2024-03-27T21:42:43.632 INFO:teuthology.orchestra.run.smithi060.stdout: 413945 4 -rw-r--r-- 1 ubuntu ubuntu 1199 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/cd-table-fetch-data-context.ts 2024-03-27T21:42:43.632 INFO:teuthology.orchestra.run.smithi060.stdout: 413944 4 -rw-r--r-- 1 ubuntu ubuntu 1305 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/cd-table-column.ts 2024-03-27T21:42:43.632 INFO:teuthology.orchestra.run.smithi060.stdout: 413984 4 -rw-r--r-- 1 ubuntu ubuntu 301 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/upgrade.interface.ts 2024-03-27T21:42:43.632 INFO:teuthology.orchestra.run.smithi060.stdout: 413983 4 -rw-r--r-- 1 ubuntu ubuntu 188 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/task.ts 2024-03-27T21:42:43.632 INFO:teuthology.orchestra.run.smithi060.stdout: 413963 4 -rw-r--r-- 1 ubuntu ubuntu 317 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/finished-task.ts 2024-03-27T21:42:43.632 INFO:teuthology.orchestra.run.smithi060.stdout: 413954 4 -rw-r--r-- 1 ubuntu ubuntu 69 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/credentials.ts 2024-03-27T21:42:43.632 INFO:teuthology.orchestra.run.smithi060.stdout: 413960 4 -rw-r--r-- 1 ubuntu ubuntu 694 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/devices.ts 2024-03-27T21:42:43.633 INFO:teuthology.orchestra.run.smithi060.stdout: 413968 4 -rw-r--r-- 1 ubuntu ubuntu 95 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/mirroring-summary.ts 2024-03-27T21:42:43.633 INFO:teuthology.orchestra.run.smithi060.stdout: 413979 8 -rw-r--r-- 1 ubuntu ubuntu 6914 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/smart.ts 2024-03-27T21:42:43.633 INFO:teuthology.orchestra.run.smithi060.stdout: 413959 4 -rw-r--r-- 1 ubuntu ubuntu 260 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/daemon.interface.ts 2024-03-27T21:42:43.633 INFO:teuthology.orchestra.run.smithi060.stdout: 413970 4 -rw-r--r-- 1 ubuntu ubuntu 751 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/orchestrator.enum.ts 2024-03-27T21:42:43.633 INFO:teuthology.orchestra.run.smithi060.stdout: 413936 4 -rw-r--r-- 1 ubuntu ubuntu 883 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/cd-form-modal-field-config.ts 2024-03-27T21:42:43.633 INFO:teuthology.orchestra.run.smithi060.stdout: 413982 4 -rw-r--r-- 1 ubuntu ubuntu 150 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/task-exception.ts 2024-03-27T21:42:43.633 INFO:teuthology.orchestra.run.smithi060.stdout: 413966 4 -rw-r--r-- 1 ubuntu ubuntu 259 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/inventory-device-type.model.ts 2024-03-27T21:42:43.633 INFO:teuthology.orchestra.run.smithi060.stdout: 413952 4 -rw-r--r-- 1 ubuntu ubuntu 3630 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/chart-tooltip.ts 2024-03-27T21:42:43.633 INFO:teuthology.orchestra.run.smithi060.stdout: 413940 4 -rw-r--r-- 1 ubuntu ubuntu 1213 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/cd-pwd-policy-settings.ts 2024-03-27T21:42:43.634 INFO:teuthology.orchestra.run.smithi060.stdout: 413977 4 -rw-r--r-- 1 ubuntu ubuntu 1894 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/prometheus-alerts.ts 2024-03-27T21:42:43.634 INFO:teuthology.orchestra.run.smithi060.stdout: 413950 4 -rw-r--r-- 1 ubuntu ubuntu 262 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/cephfs-subvolume-group.model.ts 2024-03-27T21:42:43.634 INFO:teuthology.orchestra.run.smithi060.stdout: 413985 4 -rw-r--r-- 1 ubuntu ubuntu 81 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/wizard-steps.ts 2024-03-27T21:42:43.634 INFO:teuthology.orchestra.run.smithi060.stdout: 413953 4 -rw-r--r-- 1 ubuntu ubuntu 1008 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/configuration.ts 2024-03-27T21:42:43.634 INFO:teuthology.orchestra.run.smithi060.stdout: 413946 4 -rw-r--r-- 1 ubuntu ubuntu 601 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/cd-table-paging.ts 2024-03-27T21:42:43.634 INFO:teuthology.orchestra.run.smithi060.stdout: 413978 4 -rw-r--r-- 1 ubuntu ubuntu 1138 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/service.interface.ts 2024-03-27T21:42:43.634 INFO:teuthology.orchestra.run.smithi060.stdout: 413972 4 -rw-r--r-- 1 ubuntu ubuntu 513 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/osd-deployment-options.ts 2024-03-27T21:42:43.634 INFO:teuthology.orchestra.run.smithi060.stdout: 413955 4 -rw-r--r-- 1 ubuntu ubuntu 418 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/crud-table-metadata.ts 2024-03-27T21:42:43.635 INFO:teuthology.orchestra.run.smithi060.stdout: 413942 4 -rw-r--r-- 1 ubuntu ubuntu 264 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/cd-table-column-filter.ts 2024-03-27T21:42:43.635 INFO:teuthology.orchestra.run.smithi060.stdout: 413956 4 -rw-r--r-- 1 ubuntu ubuntu 394 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/crush-node.ts 2024-03-27T21:42:43.635 INFO:teuthology.orchestra.run.smithi060.stdout: 413938 4 -rw-r--r-- 1 ubuntu ubuntu 1675 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/cd-notification.ts 2024-03-27T21:42:43.635 INFO:teuthology.orchestra.run.smithi060.stdout: 413937 4 -rw-r--r-- 1 ubuntu ubuntu 2771 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/cd-notification.spec.ts 2024-03-27T21:42:43.635 INFO:teuthology.orchestra.run.smithi060.stdout: 413973 4 -rw-r--r-- 1 ubuntu ubuntu 77 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/models/osd-settings.ts 2024-03-27T21:42:43.635 INFO:teuthology.orchestra.run.smithi060.stdout: 413794 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/constants 2024-03-27T21:42:43.635 INFO:teuthology.orchestra.run.smithi060.stdout: 413795 12 -rw-r--r-- 1 ubuntu ubuntu 8547 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/constants/app.constants.ts 2024-03-27T21:42:43.635 INFO:teuthology.orchestra.run.smithi060.stdout: 413831 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives 2024-03-27T21:42:43.635 INFO:teuthology.orchestra.run.smithi060.stdout: 413843 4 -rw-r--r-- 1 ubuntu ubuntu 2814 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/form-loading.directive.spec.ts 2024-03-27T21:42:43.636 INFO:teuthology.orchestra.run.smithi060.stdout: 413849 4 -rw-r--r-- 1 ubuntu ubuntu 258 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/milliseconds.directive.spec.ts 2024-03-27T21:42:43.636 INFO:teuthology.orchestra.run.smithi060.stdout: 413848 4 -rw-r--r-- 1 ubuntu ubuntu 841 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/iops.directive.ts 2024-03-27T21:42:43.636 INFO:teuthology.orchestra.run.smithi060.stdout: 413844 4 -rw-r--r-- 1 ubuntu ubuntu 1432 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/form-loading.directive.ts 2024-03-27T21:42:43.636 INFO:teuthology.orchestra.run.smithi060.stdout: 413858 4 -rw-r--r-- 1 ubuntu ubuntu 267 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/password-button.directive.spec.ts 2024-03-27T21:42:43.636 INFO:teuthology.orchestra.run.smithi060.stdout: 413860 4 -rw-r--r-- 1 ubuntu ubuntu 1489 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/stateful-tab.directive.spec.ts 2024-03-27T21:42:43.636 INFO:teuthology.orchestra.run.smithi060.stdout: 413859 4 -rw-r--r-- 1 ubuntu ubuntu 1440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/password-button.directive.ts 2024-03-27T21:42:43.637 INFO:teuthology.orchestra.run.smithi060.stdout: 413841 4 -rw-r--r-- 1 ubuntu ubuntu 2204 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/form-input-disable.directive.spec.ts 2024-03-27T21:42:43.637 INFO:teuthology.orchestra.run.smithi060.stdout: 413861 4 -rw-r--r-- 1 ubuntu ubuntu 994 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/stateful-tab.directive.ts 2024-03-27T21:42:43.637 INFO:teuthology.orchestra.run.smithi060.stdout: 413838 4 -rw-r--r-- 1 ubuntu ubuntu 345 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/dimless-binary.directive.spec.ts 2024-03-27T21:42:43.637 INFO:teuthology.orchestra.run.smithi060.stdout: 413834 4 -rw-r--r-- 1 ubuntu ubuntu 2565 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/autofocus.directive.spec.ts 2024-03-27T21:42:43.637 INFO:teuthology.orchestra.run.smithi060.stdout: 413840 4 -rw-r--r-- 1 ubuntu ubuntu 2033 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/directives.module.ts 2024-03-27T21:42:43.637 INFO:teuthology.orchestra.run.smithi060.stdout: 413835 4 -rw-r--r-- 1 ubuntu ubuntu 667 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/autofocus.directive.ts 2024-03-27T21:42:43.637 INFO:teuthology.orchestra.run.smithi060.stdout: 413851 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/ng-bootstrap-form-validation 2024-03-27T21:42:43.637 INFO:teuthology.orchestra.run.smithi060.stdout: 413853 4 -rw-r--r-- 1 ubuntu ubuntu 2684 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/ng-bootstrap-form-validation/cd-form-control.directive.ts 2024-03-27T21:42:43.637 INFO:teuthology.orchestra.run.smithi060.stdout: 413854 4 -rw-r--r-- 1 ubuntu ubuntu 1521 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/ng-bootstrap-form-validation/cd-form-group.directive.spec.ts 2024-03-27T21:42:43.638 INFO:teuthology.orchestra.run.smithi060.stdout: 413855 4 -rw-r--r-- 1 ubuntu ubuntu 2346 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/ng-bootstrap-form-validation/cd-form-group.directive.ts 2024-03-27T21:42:43.638 INFO:teuthology.orchestra.run.smithi060.stdout: 413856 4 -rw-r--r-- 1 ubuntu ubuntu 1476 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/ng-bootstrap-form-validation/cd-form-validation.directive.spec.ts 2024-03-27T21:42:43.638 INFO:teuthology.orchestra.run.smithi060.stdout: 413857 4 -rw-r--r-- 1 ubuntu ubuntu 2333 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/ng-bootstrap-form-validation/cd-form-validation.directive.ts 2024-03-27T21:42:43.638 INFO:teuthology.orchestra.run.smithi060.stdout: 413852 4 -rw-r--r-- 1 ubuntu ubuntu 1524 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/ng-bootstrap-form-validation/cd-form-control.directive.spec.ts 2024-03-27T21:42:43.638 INFO:teuthology.orchestra.run.smithi060.stdout: 413847 4 -rw-r--r-- 1 ubuntu ubuntu 226 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/iops.directive.spec.ts 2024-03-27T21:42:43.638 INFO:teuthology.orchestra.run.smithi060.stdout: 413845 4 -rw-r--r-- 1 ubuntu ubuntu 238 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/form-scope.directive.spec.ts 2024-03-27T21:42:43.638 INFO:teuthology.orchestra.run.smithi060.stdout: 413862 4 -rw-r--r-- 1 ubuntu ubuntu 1581 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/trim.directive.spec.ts 2024-03-27T21:42:43.638 INFO:teuthology.orchestra.run.smithi060.stdout: 413842 4 -rw-r--r-- 1 ubuntu ubuntu 964 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/form-input-disable.directive.ts 2024-03-27T21:42:43.639 INFO:teuthology.orchestra.run.smithi060.stdout: 413833 4 -rw-r--r-- 1 ubuntu ubuntu 1500 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/auth-storage.directive.ts 2024-03-27T21:42:43.639 INFO:teuthology.orchestra.run.smithi060.stdout: 413863 4 -rw-r--r-- 1 ubuntu ubuntu 489 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/trim.directive.ts 2024-03-27T21:42:43.639 INFO:teuthology.orchestra.run.smithi060.stdout: 413846 4 -rw-r--r-- 1 ubuntu ubuntu 160 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/form-scope.directive.ts 2024-03-27T21:42:43.639 INFO:teuthology.orchestra.run.smithi060.stdout: 413832 4 -rw-r--r-- 1 ubuntu ubuntu 3535 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/auth-storage.directive.spec.ts 2024-03-27T21:42:43.639 INFO:teuthology.orchestra.run.smithi060.stdout: 413839 4 -rw-r--r-- 1 ubuntu ubuntu 3497 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/dimless-binary.directive.ts 2024-03-27T21:42:43.639 INFO:teuthology.orchestra.run.smithi060.stdout: 413850 4 -rw-r--r-- 1 ubuntu ubuntu 851 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/milliseconds.directive.ts 2024-03-27T21:42:43.639 INFO:teuthology.orchestra.run.smithi060.stdout: 413836 4 -rw-r--r-- 1 ubuntu ubuntu 383 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/dimless-binary-per-second.directive.spec.ts 2024-03-27T21:42:43.639 INFO:teuthology.orchestra.run.smithi060.stdout: 413837 4 -rw-r--r-- 1 ubuntu ubuntu 3647 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/directives/dimless-binary-per-second.directive.ts 2024-03-27T21:42:43.639 INFO:teuthology.orchestra.run.smithi060.stdout: 413600 4 drwxr-xr-x 39 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components 2024-03-27T21:42:43.640 INFO:teuthology.orchestra.run.smithi060.stdout: 413621 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/card 2024-03-27T21:42:43.641 INFO:teuthology.orchestra.run.smithi060.stdout: 413623 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/card/card.component.scss 2024-03-27T21:42:43.641 INFO:teuthology.orchestra.run.smithi060.stdout: 413624 4 -rw-r--r-- 1 ubuntu ubuntu 954 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/card/card.component.spec.ts 2024-03-27T21:42:43.642 INFO:teuthology.orchestra.run.smithi060.stdout: 413625 4 -rw-r--r-- 1 ubuntu ubuntu 552 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/card/card.component.ts 2024-03-27T21:42:43.642 INFO:teuthology.orchestra.run.smithi060.stdout: 413622 4 -rw-r--r-- 1 ubuntu ubuntu 948 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/card/card.component.html 2024-03-27T21:42:43.642 INFO:teuthology.orchestra.run.smithi060.stdout: 413691 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/form-modal 2024-03-27T21:42:43.642 INFO:teuthology.orchestra.run.smithi060.stdout: 413693 0 -rwxr-xr-x 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/form-modal/form-modal.component.scss 2024-03-27T21:42:43.642 INFO:teuthology.orchestra.run.smithi060.stdout: 413692 4 -rwxr-xr-x 1 ubuntu ubuntu 2956 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/form-modal/form-modal.component.html 2024-03-27T21:42:43.642 INFO:teuthology.orchestra.run.smithi060.stdout: 413695 4 -rwxr-xr-x 1 ubuntu ubuntu 3587 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/form-modal/form-modal.component.ts 2024-03-27T21:42:43.642 INFO:teuthology.orchestra.run.smithi060.stdout: 413694 8 -rwxr-xr-x 1 ubuntu ubuntu 4646 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/form-modal/form-modal.component.spec.ts 2024-03-27T21:42:43.642 INFO:teuthology.orchestra.run.smithi060.stdout: 413661 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/custom-login-banner 2024-03-27T21:42:43.643 INFO:teuthology.orchestra.run.smithi060.stdout: 413664 4 -rw-r--r-- 1 ubuntu ubuntu 794 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/custom-login-banner/custom-login-banner.component.spec.ts 2024-03-27T21:42:43.643 INFO:teuthology.orchestra.run.smithi060.stdout: 413663 4 -rw-r--r-- 1 ubuntu ubuntu 81 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/custom-login-banner/custom-login-banner.component.scss 2024-03-27T21:42:43.643 INFO:teuthology.orchestra.run.smithi060.stdout: 413665 4 -rw-r--r-- 1 ubuntu ubuntu 635 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/custom-login-banner/custom-login-banner.component.ts 2024-03-27T21:42:43.643 INFO:teuthology.orchestra.run.smithi060.stdout: 413662 4 -rw-r--r-- 1 ubuntu ubuntu 88 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/custom-login-banner/custom-login-banner.component.html 2024-03-27T21:42:43.643 INFO:teuthology.orchestra.run.smithi060.stdout: 413616 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/card-row 2024-03-27T21:42:43.643 INFO:teuthology.orchestra.run.smithi060.stdout: 413619 4 -rw-r--r-- 1 ubuntu ubuntu 582 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/card-row/card-row.component.spec.ts 2024-03-27T21:42:43.644 INFO:teuthology.orchestra.run.smithi060.stdout: 413620 4 -rw-r--r-- 1 ubuntu ubuntu 1156 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/card-row/card-row.component.ts 2024-03-27T21:42:43.644 INFO:teuthology.orchestra.run.smithi060.stdout: 413617 8 -rw-r--r-- 1 ubuntu ubuntu 6594 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/card-row/card-row.component.html 2024-03-27T21:42:43.644 INFO:teuthology.orchestra.run.smithi060.stdout: 413618 4 -rw-r--r-- 1 ubuntu ubuntu 322 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/card-row/card-row.component.scss 2024-03-27T21:42:43.644 INFO:teuthology.orchestra.run.smithi060.stdout: 413656 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/critical-confirmation-modal 2024-03-27T21:42:43.644 INFO:teuthology.orchestra.run.smithi060.stdout: 413658 4 -rw-r--r-- 1 ubuntu ubuntu 148 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/critical-confirmation-modal/critical-confirmation-modal.component.scss 2024-03-27T21:42:43.644 INFO:teuthology.orchestra.run.smithi060.stdout: 413660 4 -rw-r--r-- 1 ubuntu ubuntu 2245 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/critical-confirmation-modal/critical-confirmation-modal.component.ts 2024-03-27T21:42:43.644 INFO:teuthology.orchestra.run.smithi060.stdout: 413657 4 -rw-r--r-- 1 ubuntu ubuntu 2649 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/critical-confirmation-modal/critical-confirmation-modal.component.html 2024-03-27T21:42:43.644 INFO:teuthology.orchestra.run.smithi060.stdout: 413659 12 -rw-r--r-- 1 ubuntu ubuntu 8300 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/critical-confirmation-modal/critical-confirmation-modal.component.spec.ts 2024-03-27T21:42:43.645 INFO:teuthology.orchestra.run.smithi060.stdout: 413686 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/form-button-panel 2024-03-27T21:42:43.645 INFO:teuthology.orchestra.run.smithi060.stdout: 413690 4 -rw-r--r-- 1 ubuntu ubuntu 1711 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/form-button-panel/form-button-panel.component.ts 2024-03-27T21:42:43.645 INFO:teuthology.orchestra.run.smithi060.stdout: 413687 4 -rw-r--r-- 1 ubuntu ubuntu 494 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/form-button-panel/form-button-panel.component.html 2024-03-27T21:42:43.645 INFO:teuthology.orchestra.run.smithi060.stdout: 413688 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/form-button-panel/form-button-panel.component.scss 2024-03-27T21:42:43.645 INFO:teuthology.orchestra.run.smithi060.stdout: 413689 4 -rw-r--r-- 1 ubuntu ubuntu 751 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/form-button-panel/form-button-panel.component.spec.ts 2024-03-27T21:42:43.645 INFO:teuthology.orchestra.run.smithi060.stdout: 413774 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/telemetry-notification 2024-03-27T21:42:43.645 INFO:teuthology.orchestra.run.smithi060.stdout: 413775 4 -rw-r--r-- 1 ubuntu ubuntu 596 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/telemetry-notification/telemetry-notification.component.html 2024-03-27T21:42:43.645 INFO:teuthology.orchestra.run.smithi060.stdout: 413778 4 -rw-r--r-- 1 ubuntu ubuntu 2264 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/telemetry-notification/telemetry-notification.component.ts 2024-03-27T21:42:43.646 INFO:teuthology.orchestra.run.smithi060.stdout: 413777 8 -rw-r--r-- 1 ubuntu ubuntu 4314 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/telemetry-notification/telemetry-notification.component.spec.ts 2024-03-27T21:42:43.646 INFO:teuthology.orchestra.run.smithi060.stdout: 413776 4 -rw-r--r-- 1 ubuntu ubuntu 380 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/telemetry-notification/telemetry-notification.component.scss 2024-03-27T21:42:43.646 INFO:teuthology.orchestra.run.smithi060.stdout: 413606 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/back-button 2024-03-27T21:42:43.646 INFO:teuthology.orchestra.run.smithi060.stdout: 413608 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/back-button/back-button.component.scss 2024-03-27T21:42:43.646 INFO:teuthology.orchestra.run.smithi060.stdout: 413609 4 -rw-r--r-- 1 ubuntu ubuntu 731 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/back-button/back-button.component.spec.ts 2024-03-27T21:42:43.646 INFO:teuthology.orchestra.run.smithi060.stdout: 413610 4 -rw-r--r-- 1 ubuntu ubuntu 774 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/back-button/back-button.component.ts 2024-03-27T21:42:43.646 INFO:teuthology.orchestra.run.smithi060.stdout: 413607 4 -rw-r--r-- 1 ubuntu ubuntu 141 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/back-button/back-button.component.html 2024-03-27T21:42:43.646 INFO:teuthology.orchestra.run.smithi060.stdout: 413696 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/grafana 2024-03-27T21:42:43.647 INFO:teuthology.orchestra.run.smithi060.stdout: 413700 8 -rw-r--r-- 1 ubuntu ubuntu 5092 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/grafana/grafana.component.ts 2024-03-27T21:42:43.647 INFO:teuthology.orchestra.run.smithi060.stdout: 413699 8 -rw-r--r-- 1 ubuntu ubuntu 4172 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/grafana/grafana.component.spec.ts 2024-03-27T21:42:43.647 INFO:teuthology.orchestra.run.smithi060.stdout: 413697 4 -rw-r--r-- 1 ubuntu ubuntu 2955 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/grafana/grafana.component.html 2024-03-27T21:42:43.647 INFO:teuthology.orchestra.run.smithi060.stdout: 413698 4 -rw-r--r-- 1 ubuntu ubuntu 318 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/grafana/grafana.component.scss 2024-03-27T21:42:43.647 INFO:teuthology.orchestra.run.smithi060.stdout: 413676 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/download-button 2024-03-27T21:42:43.647 INFO:teuthology.orchestra.run.smithi060.stdout: 413680 4 -rw-r--r-- 1 ubuntu ubuntu 975 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/download-button/download-button.component.ts 2024-03-27T21:42:43.647 INFO:teuthology.orchestra.run.smithi060.stdout: 413677 4 -rw-r--r-- 1 ubuntu ubuntu 618 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/download-button/download-button.component.html 2024-03-27T21:42:43.647 INFO:teuthology.orchestra.run.smithi060.stdout: 413679 4 -rw-r--r-- 1 ubuntu ubuntu 1259 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/download-button/download-button.component.spec.ts 2024-03-27T21:42:43.648 INFO:teuthology.orchestra.run.smithi060.stdout: 413678 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/download-button/download-button.component.scss 2024-03-27T21:42:43.648 INFO:teuthology.orchestra.run.smithi060.stdout: 413701 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/help-text 2024-03-27T21:42:43.648 INFO:teuthology.orchestra.run.smithi060.stdout: 413702 4 -rw-r--r-- 1 ubuntu ubuntu 76 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/help-text/help-text.component.html 2024-03-27T21:42:43.648 INFO:teuthology.orchestra.run.smithi060.stdout: 413704 4 -rw-r--r-- 1 ubuntu ubuntu 607 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/help-text/help-text.component.spec.ts 2024-03-27T21:42:43.648 INFO:teuthology.orchestra.run.smithi060.stdout: 413705 4 -rw-r--r-- 1 ubuntu ubuntu 211 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/help-text/help-text.component.ts 2024-03-27T21:42:43.648 INFO:teuthology.orchestra.run.smithi060.stdout: 413703 4 -rw-r--r-- 1 ubuntu ubuntu 53 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/help-text/help-text.component.scss 2024-03-27T21:42:43.648 INFO:teuthology.orchestra.run.smithi060.stdout: 413681 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/form-advanced-fieldset 2024-03-27T21:42:43.649 INFO:teuthology.orchestra.run.smithi060.stdout: 413682 4 -rw-r--r-- 1 ubuntu ubuntu 673 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/form-advanced-fieldset/form-advanced-fieldset.component.html 2024-03-27T21:42:43.649 INFO:teuthology.orchestra.run.smithi060.stdout: 413683 4 -rw-r--r-- 1 ubuntu ubuntu 596 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/form-advanced-fieldset/form-advanced-fieldset.component.scss 2024-03-27T21:42:43.649 INFO:teuthology.orchestra.run.smithi060.stdout: 413684 4 -rw-r--r-- 1 ubuntu ubuntu 692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/form-advanced-fieldset/form-advanced-fieldset.component.spec.ts 2024-03-27T21:42:43.649 INFO:teuthology.orchestra.run.smithi060.stdout: 413685 4 -rw-r--r-- 1 ubuntu ubuntu 296 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/form-advanced-fieldset/form-advanced-fieldset.component.ts 2024-03-27T21:42:43.649 INFO:teuthology.orchestra.run.smithi060.stdout: 413671 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/doc 2024-03-27T21:42:43.649 INFO:teuthology.orchestra.run.smithi060.stdout: 413673 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/doc/doc.component.scss 2024-03-27T21:42:43.649 INFO:teuthology.orchestra.run.smithi060.stdout: 413672 4 -rw-r--r-- 1 ubuntu ubuntu 60 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/doc/doc.component.html 2024-03-27T21:42:43.649 INFO:teuthology.orchestra.run.smithi060.stdout: 413674 4 -rw-r--r-- 1 ubuntu ubuntu 813 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/doc/doc.component.spec.ts 2024-03-27T21:42:43.650 INFO:teuthology.orchestra.run.smithi060.stdout: 413675 4 -rw-r--r-- 1 ubuntu ubuntu 701 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/doc/doc.component.ts 2024-03-27T21:42:43.650 INFO:teuthology.orchestra.run.smithi060.stdout: 413626 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/cd-label 2024-03-27T21:42:43.650 INFO:teuthology.orchestra.run.smithi060.stdout: 413627 4 -rw-r--r-- 1 ubuntu ubuntu 328 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/cd-label/cd-label.component.html 2024-03-27T21:42:43.650 INFO:teuthology.orchestra.run.smithi060.stdout: 413628 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/cd-label/cd-label.component.scss 2024-03-27T21:42:43.650 INFO:teuthology.orchestra.run.smithi060.stdout: 413629 4 -rw-r--r-- 1 ubuntu ubuntu 706 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/cd-label/cd-label.component.spec.ts 2024-03-27T21:42:43.650 INFO:teuthology.orchestra.run.smithi060.stdout: 413630 4 -rw-r--r-- 1 ubuntu ubuntu 297 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/cd-label/cd-label.component.ts 2024-03-27T21:42:43.650 INFO:teuthology.orchestra.run.smithi060.stdout: 413631 4 -rw-r--r-- 1 ubuntu ubuntu 735 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/cd-label/color-class-from-text.pipe.ts 2024-03-27T21:42:43.650 INFO:teuthology.orchestra.run.smithi060.stdout: 413637 8 -rw-r--r-- 1 ubuntu ubuntu 6245 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/components.module.ts 2024-03-27T21:42:43.651 INFO:teuthology.orchestra.run.smithi060.stdout: 413632 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/code-block 2024-03-27T21:42:43.651 INFO:teuthology.orchestra.run.smithi060.stdout: 413634 4 -rw-r--r-- 1 ubuntu ubuntu 70 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/code-block/code-block.component.scss 2024-03-27T21:42:43.651 INFO:teuthology.orchestra.run.smithi060.stdout: 413633 4 -rw-r--r-- 1 ubuntu ubuntu 769 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/code-block/code-block.component.html 2024-03-27T21:42:43.651 INFO:teuthology.orchestra.run.smithi060.stdout: 413636 4 -rw-r--r-- 1 ubuntu ubuntu 333 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/code-block/code-block.component.ts 2024-03-27T21:42:43.651 INFO:teuthology.orchestra.run.smithi060.stdout: 413635 4 -rw-r--r-- 1 ubuntu ubuntu 1263 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/code-block/code-block.component.spec.ts 2024-03-27T21:42:43.651 INFO:teuthology.orchestra.run.smithi060.stdout: 413784 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/vertical-navigation 2024-03-27T21:42:43.651 INFO:teuthology.orchestra.run.smithi060.stdout: 413785 4 -rw-r--r-- 1 ubuntu ubuntu 807 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/vertical-navigation/vertical-navigation.component.html 2024-03-27T21:42:43.651 INFO:teuthology.orchestra.run.smithi060.stdout: 413787 4 -rw-r--r-- 1 ubuntu ubuntu 2067 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/vertical-navigation/vertical-navigation.component.spec.ts 2024-03-27T21:42:43.651 INFO:teuthology.orchestra.run.smithi060.stdout: 413788 4 -rw-r--r-- 1 ubuntu ubuntu 1082 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/vertical-navigation/vertical-navigation.component.ts 2024-03-27T21:42:43.652 INFO:teuthology.orchestra.run.smithi060.stdout: 413786 4 -rw-r--r-- 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/vertical-navigation/vertical-navigation.component.scss 2024-03-27T21:42:43.652 INFO:teuthology.orchestra.run.smithi060.stdout: 413727 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/motd 2024-03-27T21:42:43.652 INFO:teuthology.orchestra.run.smithi060.stdout: 413728 4 -rw-r--r-- 1 ubuntu ubuntu 312 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/motd/motd.component.html 2024-03-27T21:42:43.652 INFO:teuthology.orchestra.run.smithi060.stdout: 413730 4 -rw-r--r-- 1 ubuntu ubuntu 829 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/motd/motd.component.spec.ts 2024-03-27T21:42:43.652 INFO:teuthology.orchestra.run.smithi060.stdout: 413731 4 -rw-r--r-- 1 ubuntu ubuntu 871 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/motd/motd.component.ts 2024-03-27T21:42:43.652 INFO:teuthology.orchestra.run.smithi060.stdout: 413729 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/motd/motd.component.scss 2024-03-27T21:42:43.652 INFO:teuthology.orchestra.run.smithi060.stdout: 413747 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/refresh-selector 2024-03-27T21:42:43.652 INFO:teuthology.orchestra.run.smithi060.stdout: 413750 4 -rw-r--r-- 1 ubuntu ubuntu 871 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/refresh-selector/refresh-selector.component.spec.ts 2024-03-27T21:42:43.653 INFO:teuthology.orchestra.run.smithi060.stdout: 413749 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/refresh-selector/refresh-selector.component.scss 2024-03-27T21:42:43.653 INFO:teuthology.orchestra.run.smithi060.stdout: 413751 4 -rw-r--r-- 1 ubuntu ubuntu 910 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/refresh-selector/refresh-selector.component.ts 2024-03-27T21:42:43.653 INFO:teuthology.orchestra.run.smithi060.stdout: 413748 4 -rw-r--r-- 1 ubuntu ubuntu 653 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/refresh-selector/refresh-selector.component.html 2024-03-27T21:42:43.653 INFO:teuthology.orchestra.run.smithi060.stdout: 413764 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/sparkline 2024-03-27T21:42:43.653 INFO:teuthology.orchestra.run.smithi060.stdout: 413765 4 -rw-r--r-- 1 ubuntu ubuntu 283 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/sparkline/sparkline.component.html 2024-03-27T21:42:43.653 INFO:teuthology.orchestra.run.smithi060.stdout: 413768 4 -rw-r--r-- 1 ubuntu ubuntu 3180 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/sparkline/sparkline.component.ts 2024-03-27T21:42:43.653 INFO:teuthology.orchestra.run.smithi060.stdout: 413767 4 -rw-r--r-- 1 ubuntu ubuntu 2041 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/sparkline/sparkline.component.spec.ts 2024-03-27T21:42:43.654 INFO:teuthology.orchestra.run.smithi060.stdout: 413766 4 -rw-r--r-- 1 ubuntu ubuntu 88 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/sparkline/sparkline.component.scss 2024-03-27T21:42:43.654 INFO:teuthology.orchestra.run.smithi060.stdout: 413732 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/notifications-sidebar 2024-03-27T21:42:43.654 INFO:teuthology.orchestra.run.smithi060.stdout: 413734 4 -rw-r--r-- 1 ubuntu ubuntu 760 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/notifications-sidebar/notifications-sidebar.component.scss 2024-03-27T21:42:43.654 INFO:teuthology.orchestra.run.smithi060.stdout: 413735 8 -rw-r--r-- 1 ubuntu ubuntu 7483 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/notifications-sidebar/notifications-sidebar.component.spec.ts 2024-03-27T21:42:43.654 INFO:teuthology.orchestra.run.smithi060.stdout: 413733 8 -rw-r--r-- 1 ubuntu ubuntu 5251 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/notifications-sidebar/notifications-sidebar.component.html 2024-03-27T21:42:43.654 INFO:teuthology.orchestra.run.smithi060.stdout: 413736 8 -rw-r--r-- 1 ubuntu ubuntu 7056 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/notifications-sidebar/notifications-sidebar.component.ts 2024-03-27T21:42:43.654 INFO:teuthology.orchestra.run.smithi060.stdout: 413666 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker 2024-03-27T21:42:43.654 INFO:teuthology.orchestra.run.smithi060.stdout: 413669 4 -rw-r--r-- 1 ubuntu ubuntu 2002 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.spec.ts 2024-03-27T21:42:43.655 INFO:teuthology.orchestra.run.smithi060.stdout: 413668 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.scss 2024-03-27T21:42:43.655 INFO:teuthology.orchestra.run.smithi060.stdout: 413667 4 -rw-r--r-- 1 ubuntu ubuntu 439 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.html 2024-03-27T21:42:43.655 INFO:teuthology.orchestra.run.smithi060.stdout: 413670 4 -rw-r--r-- 1 ubuntu ubuntu 1661 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.ts 2024-03-27T21:42:43.655 INFO:teuthology.orchestra.run.smithi060.stdout: 413742 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/pwd-expiration-notification 2024-03-27T21:42:43.655 INFO:teuthology.orchestra.run.smithi060.stdout: 413745 8 -rw-r--r-- 1 ubuntu ubuntu 4126 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/pwd-expiration-notification/pwd-expiration-notification.component.spec.ts 2024-03-27T21:42:43.655 INFO:teuthology.orchestra.run.smithi060.stdout: 413746 4 -rw-r--r-- 1 ubuntu ubuntu 2068 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/pwd-expiration-notification/pwd-expiration-notification.component.ts 2024-03-27T21:42:43.655 INFO:teuthology.orchestra.run.smithi060.stdout: 413743 4 -rw-r--r-- 1 ubuntu ubuntu 738 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/pwd-expiration-notification/pwd-expiration-notification.component.html 2024-03-27T21:42:43.655 INFO:teuthology.orchestra.run.smithi060.stdout: 413744 4 -rw-r--r-- 1 ubuntu ubuntu 42 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/pwd-expiration-notification/pwd-expiration-notification.component.scss 2024-03-27T21:42:43.656 INFO:teuthology.orchestra.run.smithi060.stdout: 413717 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/loading-panel 2024-03-27T21:42:43.656 INFO:teuthology.orchestra.run.smithi060.stdout: 413721 4 -rw-r--r-- 1 ubuntu ubuntu 300 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/loading-panel/loading-panel.component.ts 2024-03-27T21:42:43.656 INFO:teuthology.orchestra.run.smithi060.stdout: 413720 4 -rw-r--r-- 1 ubuntu ubuntu 739 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/loading-panel/loading-panel.component.spec.ts 2024-03-27T21:42:43.656 INFO:teuthology.orchestra.run.smithi060.stdout: 413718 4 -rw-r--r-- 1 ubuntu ubuntu 219 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/loading-panel/loading-panel.component.html 2024-03-27T21:42:43.656 INFO:teuthology.orchestra.run.smithi060.stdout: 413719 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/loading-panel/loading-panel.component.scss 2024-03-27T21:42:43.656 INFO:teuthology.orchestra.run.smithi060.stdout: 413722 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/modal 2024-03-27T21:42:43.656 INFO:teuthology.orchestra.run.smithi060.stdout: 413723 4 -rw-r--r-- 1 ubuntu ubuntu 532 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/modal/modal.component.html 2024-03-27T21:42:43.656 INFO:teuthology.orchestra.run.smithi060.stdout: 413726 4 -rw-r--r-- 1 ubuntu ubuntu 728 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/modal/modal.component.ts 2024-03-27T21:42:43.657 INFO:teuthology.orchestra.run.smithi060.stdout: 413724 4 -rw-r--r-- 1 ubuntu ubuntu 378 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/modal/modal.component.scss 2024-03-27T21:42:43.657 INFO:teuthology.orchestra.run.smithi060.stdout: 413725 4 -rw-r--r-- 1 ubuntu ubuntu 1749 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/modal/modal.component.spec.ts 2024-03-27T21:42:43.657 INFO:teuthology.orchestra.run.smithi060.stdout: 413651 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/copy2clipboard-button 2024-03-27T21:42:43.657 INFO:teuthology.orchestra.run.smithi060.stdout: 413654 4 -rw-r--r-- 1 ubuntu ubuntu 1919 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/copy2clipboard-button/copy2clipboard-button.component.spec.ts 2024-03-27T21:42:43.657 INFO:teuthology.orchestra.run.smithi060.stdout: 413653 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/copy2clipboard-button/copy2clipboard-button.component.scss 2024-03-27T21:42:43.657 INFO:teuthology.orchestra.run.smithi060.stdout: 413655 4 -rw-r--r-- 1 ubuntu ubuntu 1801 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/copy2clipboard-button/copy2clipboard-button.component.ts 2024-03-27T21:42:43.657 INFO:teuthology.orchestra.run.smithi060.stdout: 413652 4 -rw-r--r-- 1 ubuntu ubuntu 419 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/copy2clipboard-button/copy2clipboard-button.component.html 2024-03-27T21:42:43.657 INFO:teuthology.orchestra.run.smithi060.stdout: 413601 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/alert-panel 2024-03-27T21:42:43.657 INFO:teuthology.orchestra.run.smithi060.stdout: 413602 4 -rw-r--r-- 1 ubuntu ubuntu 1273 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/alert-panel/alert-panel.component.html 2024-03-27T21:42:43.658 INFO:teuthology.orchestra.run.smithi060.stdout: 413603 4 -rw-r--r-- 1 ubuntu ubuntu 146 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/alert-panel/alert-panel.component.scss 2024-03-27T21:42:43.658 INFO:teuthology.orchestra.run.smithi060.stdout: 413604 4 -rw-r--r-- 1 ubuntu ubuntu 725 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/alert-panel/alert-panel.component.spec.ts 2024-03-27T21:42:43.658 INFO:teuthology.orchestra.run.smithi060.stdout: 413605 4 -rw-r--r-- 1 ubuntu ubuntu 1995 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/alert-panel/alert-panel.component.ts 2024-03-27T21:42:43.658 INFO:teuthology.orchestra.run.smithi060.stdout: 413611 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/card-group 2024-03-27T21:42:43.658 INFO:teuthology.orchestra.run.smithi060.stdout: 413615 4 -rw-r--r-- 1 ubuntu ubuntu 253 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/card-group/card-group.component.ts 2024-03-27T21:42:43.658 INFO:teuthology.orchestra.run.smithi060.stdout: 413613 4 -rw-r--r-- 1 ubuntu ubuntu 65 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/card-group/card-group.component.scss 2024-03-27T21:42:43.659 INFO:teuthology.orchestra.run.smithi060.stdout: 413612 4 -rw-r--r-- 1 ubuntu ubuntu 184 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/card-group/card-group.component.html 2024-03-27T21:42:43.659 INFO:teuthology.orchestra.run.smithi060.stdout: 413614 4 -rw-r--r-- 1 ubuntu ubuntu 614 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/card-group/card-group.component.spec.ts 2024-03-27T21:42:43.659 INFO:teuthology.orchestra.run.smithi060.stdout: 413711 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/language-selector 2024-03-27T21:42:43.659 INFO:teuthology.orchestra.run.smithi060.stdout: 413716 4 -rw-r--r-- 1 ubuntu ubuntu 473 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/language-selector/supported-languages.enum.ts 2024-03-27T21:42:43.659 INFO:teuthology.orchestra.run.smithi060.stdout: 413714 4 -rw-r--r-- 1 ubuntu ubuntu 2176 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/language-selector/language-selector.component.spec.ts 2024-03-27T21:42:43.659 INFO:teuthology.orchestra.run.smithi060.stdout: 413715 4 -rw-r--r-- 1 ubuntu ubuntu 1138 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/language-selector/language-selector.component.ts 2024-03-27T21:42:43.659 INFO:teuthology.orchestra.run.smithi060.stdout: 413713 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/language-selector/language-selector.component.scss 2024-03-27T21:42:43.659 INFO:teuthology.orchestra.run.smithi060.stdout: 413712 4 -rw-r--r-- 1 ubuntu ubuntu 591 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/language-selector/language-selector.component.html 2024-03-27T21:42:43.660 INFO:teuthology.orchestra.run.smithi060.stdout: 413757 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/select 2024-03-27T21:42:43.660 INFO:teuthology.orchestra.run.smithi060.stdout: 413758 4 -rw-r--r-- 1 ubuntu ubuntu 608 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/select/select-messages.model.ts 2024-03-27T21:42:43.660 INFO:teuthology.orchestra.run.smithi060.stdout: 413763 4 -rw-r--r-- 1 ubuntu ubuntu 3869 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/select/select.component.ts 2024-03-27T21:42:43.660 INFO:teuthology.orchestra.run.smithi060.stdout: 413762 12 -rw-r--r-- 1 ubuntu ubuntu 8583 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/select/select.component.spec.ts 2024-03-27T21:42:43.660 INFO:teuthology.orchestra.run.smithi060.stdout: 413759 4 -rw-r--r-- 1 ubuntu ubuntu 317 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/select/select-option.model.ts 2024-03-27T21:42:43.660 INFO:teuthology.orchestra.run.smithi060.stdout: 413761 4 -rw-r--r-- 1 ubuntu ubuntu 395 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/select/select.component.scss 2024-03-27T21:42:43.660 INFO:teuthology.orchestra.run.smithi060.stdout: 413760 4 -rw-r--r-- 1 ubuntu ubuntu 2565 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/select/select.component.html 2024-03-27T21:42:43.660 INFO:teuthology.orchestra.run.smithi060.stdout: 413638 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/config-option 2024-03-27T21:42:43.661 INFO:teuthology.orchestra.run.smithi060.stdout: 413640 4 -rw-r--r-- 1 ubuntu ubuntu 106 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/config-option/config-option.component.scss 2024-03-27T21:42:43.661 INFO:teuthology.orchestra.run.smithi060.stdout: 413645 8 -rw-r--r-- 1 ubuntu ubuntu 4426 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/config-option/config-option.types.ts 2024-03-27T21:42:43.661 INFO:teuthology.orchestra.run.smithi060.stdout: 413643 4 -rw-r--r-- 1 ubuntu ubuntu 213 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/config-option/config-option.model.ts 2024-03-27T21:42:43.661 INFO:teuthology.orchestra.run.smithi060.stdout: 413639 4 -rw-r--r-- 1 ubuntu ubuntu 3046 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/config-option/config-option.component.html 2024-03-27T21:42:43.661 INFO:teuthology.orchestra.run.smithi060.stdout: 413642 4 -rw-r--r-- 1 ubuntu ubuntu 3973 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/config-option/config-option.component.ts 2024-03-27T21:42:43.661 INFO:teuthology.orchestra.run.smithi060.stdout: 413641 12 -rw-r--r-- 1 ubuntu ubuntu 9069 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/config-option/config-option.component.spec.ts 2024-03-27T21:42:43.661 INFO:teuthology.orchestra.run.smithi060.stdout: 413644 12 -rw-r--r-- 1 ubuntu ubuntu 9811 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/config-option/config-option.types.spec.ts 2024-03-27T21:42:43.661 INFO:teuthology.orchestra.run.smithi060.stdout: 413769 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/submit-button 2024-03-27T21:42:43.661 INFO:teuthology.orchestra.run.smithi060.stdout: 413770 4 -rw-r--r-- 1 ubuntu ubuntu 336 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/submit-button/submit-button.component.html 2024-03-27T21:42:43.662 INFO:teuthology.orchestra.run.smithi060.stdout: 413771 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/submit-button/submit-button.component.scss 2024-03-27T21:42:43.662 INFO:teuthology.orchestra.run.smithi060.stdout: 413773 4 -rw-r--r-- 1 ubuntu ubuntu 2569 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/submit-button/submit-button.component.ts 2024-03-27T21:42:43.662 INFO:teuthology.orchestra.run.smithi060.stdout: 413772 4 -rw-r--r-- 1 ubuntu ubuntu 736 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/submit-button/submit-button.component.spec.ts 2024-03-27T21:42:43.662 INFO:teuthology.orchestra.run.smithi060.stdout: 413752 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/select-badges 2024-03-27T21:42:43.662 INFO:teuthology.orchestra.run.smithi060.stdout: 413755 4 -rw-r--r-- 1 ubuntu ubuntu 2136 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/select-badges/select-badges.component.spec.ts 2024-03-27T21:42:43.662 INFO:teuthology.orchestra.run.smithi060.stdout: 413754 4 -rw-r--r-- 1 ubuntu ubuntu 121 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/select-badges/select-badges.component.scss 2024-03-27T21:42:43.662 INFO:teuthology.orchestra.run.smithi060.stdout: 413753 4 -rw-r--r-- 1 ubuntu ubuntu 692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/select-badges/select-badges.component.html 2024-03-27T21:42:43.662 INFO:teuthology.orchestra.run.smithi060.stdout: 413756 4 -rw-r--r-- 1 ubuntu ubuntu 960 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/select-badges/select-badges.component.ts 2024-03-27T21:42:43.663 INFO:teuthology.orchestra.run.smithi060.stdout: 413737 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/orchestrator-doc-panel 2024-03-27T21:42:43.663 INFO:teuthology.orchestra.run.smithi060.stdout: 413739 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/orchestrator-doc-panel/orchestrator-doc-panel.component.scss 2024-03-27T21:42:43.663 INFO:teuthology.orchestra.run.smithi060.stdout: 413741 4 -rw-r--r-- 1 ubuntu ubuntu 401 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/orchestrator-doc-panel/orchestrator-doc-panel.component.ts 2024-03-27T21:42:43.663 INFO:teuthology.orchestra.run.smithi060.stdout: 413738 4 -rw-r--r-- 1 ubuntu ubuntu 432 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/orchestrator-doc-panel/orchestrator-doc-panel.component.html 2024-03-27T21:42:43.663 INFO:teuthology.orchestra.run.smithi060.stdout: 413740 4 -rw-r--r-- 1 ubuntu ubuntu 1130 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/orchestrator-doc-panel/orchestrator-doc-panel.component.spec.ts 2024-03-27T21:42:43.663 INFO:teuthology.orchestra.run.smithi060.stdout: 413789 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/wizard 2024-03-27T21:42:43.663 INFO:teuthology.orchestra.run.smithi060.stdout: 413790 4 -rw-r--r-- 1 ubuntu ubuntu 642 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/wizard/wizard.component.html 2024-03-27T21:42:43.664 INFO:teuthology.orchestra.run.smithi060.stdout: 413792 4 -rw-r--r-- 1 ubuntu ubuntu 706 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/wizard/wizard.component.spec.ts 2024-03-27T21:42:43.664 INFO:teuthology.orchestra.run.smithi060.stdout: 413791 4 -rw-r--r-- 1 ubuntu ubuntu 514 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/wizard/wizard.component.scss 2024-03-27T21:42:43.664 INFO:teuthology.orchestra.run.smithi060.stdout: 413793 4 -rw-r--r-- 1 ubuntu ubuntu 1114 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/wizard/wizard.component.ts 2024-03-27T21:42:43.664 INFO:teuthology.orchestra.run.smithi060.stdout: 413706 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/helper 2024-03-27T21:42:43.664 INFO:teuthology.orchestra.run.smithi060.stdout: 413708 4 -rw-r--r-- 1 ubuntu ubuntu 113 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/helper/helper.component.scss 2024-03-27T21:42:43.664 INFO:teuthology.orchestra.run.smithi060.stdout: 413707 4 -rw-r--r-- 1 ubuntu ubuntu 285 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/helper/helper.component.html 2024-03-27T21:42:43.664 INFO:teuthology.orchestra.run.smithi060.stdout: 413709 4 -rw-r--r-- 1 ubuntu ubuntu 700 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/helper/helper.component.spec.ts 2024-03-27T21:42:43.664 INFO:teuthology.orchestra.run.smithi060.stdout: 413710 4 -rw-r--r-- 1 ubuntu ubuntu 364 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/helper/helper.component.ts 2024-03-27T21:42:43.665 INFO:teuthology.orchestra.run.smithi060.stdout: 413779 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/usage-bar 2024-03-27T21:42:43.665 INFO:teuthology.orchestra.run.smithi060.stdout: 413783 4 -rw-r--r-- 1 ubuntu ubuntu 1184 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/usage-bar/usage-bar.component.ts 2024-03-27T21:42:43.665 INFO:teuthology.orchestra.run.smithi060.stdout: 413780 4 -rw-r--r-- 1 ubuntu ubuntu 1962 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/usage-bar/usage-bar.component.html 2024-03-27T21:42:43.665 INFO:teuthology.orchestra.run.smithi060.stdout: 413782 4 -rw-r--r-- 1 ubuntu ubuntu 791 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/usage-bar/usage-bar.component.spec.ts 2024-03-27T21:42:43.665 INFO:teuthology.orchestra.run.smithi060.stdout: 413781 4 -rw-r--r-- 1 ubuntu ubuntu 529 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/usage-bar/usage-bar.component.scss 2024-03-27T21:42:43.665 INFO:teuthology.orchestra.run.smithi060.stdout: 413646 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/confirmation-modal 2024-03-27T21:42:43.665 INFO:teuthology.orchestra.run.smithi060.stdout: 413647 4 -rw-r--r-- 1 ubuntu ubuntu 1073 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/confirmation-modal/confirmation-modal.component.html 2024-03-27T21:42:43.665 INFO:teuthology.orchestra.run.smithi060.stdout: 413649 8 -rw-r--r-- 1 ubuntu ubuntu 5754 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/confirmation-modal/confirmation-modal.component.spec.ts 2024-03-27T21:42:43.665 INFO:teuthology.orchestra.run.smithi060.stdout: 413650 4 -rw-r--r-- 1 ubuntu ubuntu 1698 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/confirmation-modal/confirmation-modal.component.ts 2024-03-27T21:42:43.666 INFO:teuthology.orchestra.run.smithi060.stdout: 413648 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/components/confirmation-modal/confirmation-modal.component.scss 2024-03-27T21:42:43.666 INFO:teuthology.orchestra.run.smithi060.stdout: 414147 4 -rw-r--r-- 1 ubuntu ubuntu 1971 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/shared.module.ts 2024-03-27T21:42:43.666 INFO:teuthology.orchestra.run.smithi060.stdout: 414064 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services 2024-03-27T21:42:43.666 INFO:teuthology.orchestra.run.smithi060.stdout: 414118 4 -rw-r--r-- 1 ubuntu ubuntu 2264 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/prometheus-silence-matcher.service.ts 2024-03-27T21:42:43.666 INFO:teuthology.orchestra.run.smithi060.stdout: 414134 4 -rw-r--r-- 1 ubuntu ubuntu 346 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/telemetry-notification.service.ts 2024-03-27T21:42:43.666 INFO:teuthology.orchestra.run.smithi060.stdout: 414067 4 -rw-r--r-- 1 ubuntu ubuntu 1828 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/auth-guard.service.spec.ts 2024-03-27T21:42:43.666 INFO:teuthology.orchestra.run.smithi060.stdout: 414097 4 -rw-r--r-- 1 ubuntu ubuntu 620 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/modal.service.ts 2024-03-27T21:42:43.666 INFO:teuthology.orchestra.run.smithi060.stdout: 414146 4 -rw-r--r-- 1 ubuntu ubuntu 1590 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/wizard-steps.service.ts 2024-03-27T21:42:43.667 INFO:teuthology.orchestra.run.smithi060.stdout: 414108 4 -rw-r--r-- 1 ubuntu ubuntu 1786 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/number-formatter.service.ts 2024-03-27T21:42:43.667 INFO:teuthology.orchestra.run.smithi060.stdout: 414080 4 -rw-r--r-- 1 ubuntu ubuntu 3004 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/data-gateway.service.ts 2024-03-27T21:42:43.667 INFO:teuthology.orchestra.run.smithi060.stdout: 414142 4 -rw-r--r-- 1 ubuntu ubuntu 891 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/timer.service.ts 2024-03-27T21:42:43.667 INFO:teuthology.orchestra.run.smithi060.stdout: 414102 4 -rw-r--r-- 1 ubuntu ubuntu 1124 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/ngzone-scheduler.service.ts 2024-03-27T21:42:43.667 INFO:teuthology.orchestra.run.smithi060.stdout: 414135 4 -rw-r--r-- 1 ubuntu ubuntu 435 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/text-area-json-formatter.service.spec.ts 2024-03-27T21:42:43.667 INFO:teuthology.orchestra.run.smithi060.stdout: 414086 4 -rw-r--r-- 1 ubuntu ubuntu 2233 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/favicon.service.ts 2024-03-27T21:42:43.667 INFO:teuthology.orchestra.run.smithi060.stdout: 414127 4 -rw-r--r-- 1 ubuntu ubuntu 2188 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/task-manager.service.spec.ts 2024-03-27T21:42:43.667 INFO:teuthology.orchestra.run.smithi060.stdout: 414116 4 -rw-r--r-- 1 ubuntu ubuntu 1601 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/prometheus-notification.service.ts 2024-03-27T21:42:43.668 INFO:teuthology.orchestra.run.smithi060.stdout: 414099 4 -rw-r--r-- 1 ubuntu ubuntu 3456 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/module-status-guard.service.ts 2024-03-27T21:42:43.668 INFO:teuthology.orchestra.run.smithi060.stdout: 414077 4 -rw-r--r-- 1 ubuntu ubuntu 509 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/crud-form-adapter.service.spec.ts 2024-03-27T21:42:43.668 INFO:teuthology.orchestra.run.smithi060.stdout: 414140 4 -rw-r--r-- 1 ubuntu ubuntu 1732 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/time-diff.service.ts 2024-03-27T21:42:43.668 INFO:teuthology.orchestra.run.smithi060.stdout: 414085 4 -rw-r--r-- 1 ubuntu ubuntu 629 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/favicon.service.spec.ts 2024-03-27T21:42:43.668 INFO:teuthology.orchestra.run.smithi060.stdout: 414107 4 -rw-r--r-- 1 ubuntu ubuntu 396 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/number-formatter.service.spec.ts 2024-03-27T21:42:43.668 INFO:teuthology.orchestra.run.smithi060.stdout: 414092 8 -rw-r--r-- 1 ubuntu ubuntu 4218 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/formatter.service.ts 2024-03-27T21:42:43.668 INFO:teuthology.orchestra.run.smithi060.stdout: 414120 8 -rw-r--r-- 1 ubuntu ubuntu 5021 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/rbd-configuration.service.ts 2024-03-27T21:42:43.669 INFO:teuthology.orchestra.run.smithi060.stdout: 414125 8 -rw-r--r-- 1 ubuntu ubuntu 4288 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/task-list.service.spec.ts 2024-03-27T21:42:43.669 INFO:teuthology.orchestra.run.smithi060.stdout: 414139 4 -rw-r--r-- 1 ubuntu ubuntu 2689 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/time-diff.service.spec.ts 2024-03-27T21:42:43.669 INFO:teuthology.orchestra.run.smithi060.stdout: 414089 4 -rw-r--r-- 1 ubuntu ubuntu 1507 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/feature-toggles.service.spec.ts 2024-03-27T21:42:43.669 INFO:teuthology.orchestra.run.smithi060.stdout: 414114 4 -rw-r--r-- 1 ubuntu ubuntu 3586 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/prometheus-alert.service.ts 2024-03-27T21:42:43.669 INFO:teuthology.orchestra.run.smithi060.stdout: 414109 8 -rw-r--r-- 1 ubuntu ubuntu 7678 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/password-policy.service.spec.ts 2024-03-27T21:42:43.669 INFO:teuthology.orchestra.run.smithi060.stdout: 414121 4 -rw-r--r-- 1 ubuntu ubuntu 1482 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/refresh-interval.service.spec.ts 2024-03-27T21:42:43.669 INFO:teuthology.orchestra.run.smithi060.stdout: 414104 4 -rw-r--r-- 1 ubuntu ubuntu 744 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/no-sso-guard.service.ts 2024-03-27T21:42:43.669 INFO:teuthology.orchestra.run.smithi060.stdout: 414096 4 -rw-r--r-- 1 ubuntu ubuntu 1673 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/modal.service.spec.ts 2024-03-27T21:42:43.670 INFO:teuthology.orchestra.run.smithi060.stdout: 414131 4 -rw-r--r-- 1 ubuntu ubuntu 3677 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/task-wrapper.service.spec.ts 2024-03-27T21:42:43.670 INFO:teuthology.orchestra.run.smithi060.stdout: 414084 4 -rw-r--r-- 1 ubuntu ubuntu 2584 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/doc.service.ts 2024-03-27T21:42:43.670 INFO:teuthology.orchestra.run.smithi060.stdout: 414122 4 -rw-r--r-- 1 ubuntu ubuntu 1320 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/refresh-interval.service.ts 2024-03-27T21:42:43.670 INFO:teuthology.orchestra.run.smithi060.stdout: 414132 4 -rw-r--r-- 1 ubuntu ubuntu 2140 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/task-wrapper.service.ts 2024-03-27T21:42:43.670 INFO:teuthology.orchestra.run.smithi060.stdout: 414103 4 -rw-r--r-- 1 ubuntu ubuntu 1629 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/no-sso-guard.service.spec.ts 2024-03-27T21:42:43.670 INFO:teuthology.orchestra.run.smithi060.stdout: 414128 4 -rw-r--r-- 1 ubuntu ubuntu 1914 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/task-manager.service.ts 2024-03-27T21:42:43.670 INFO:teuthology.orchestra.run.smithi060.stdout: 414110 4 -rw-r--r-- 1 ubuntu ubuntu 2436 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/password-policy.service.ts 2024-03-27T21:42:43.670 INFO:teuthology.orchestra.run.smithi060.stdout: 414071 4 -rw-r--r-- 1 ubuntu ubuntu 415 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/cd-table-server-side.service.spec.ts 2024-03-27T21:42:43.671 INFO:teuthology.orchestra.run.smithi060.stdout: 414113 8 -rw-r--r-- 1 ubuntu ubuntu 7446 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/prometheus-alert.service.spec.ts 2024-03-27T21:42:43.671 INFO:teuthology.orchestra.run.smithi060.stdout: 414088 4 -rw-r--r-- 1 ubuntu ubuntu 941 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/feature-toggles-guard.service.ts 2024-03-27T21:42:43.671 INFO:teuthology.orchestra.run.smithi060.stdout: 414068 4 -rw-r--r-- 1 ubuntu ubuntu 837 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/auth-guard.service.ts 2024-03-27T21:42:43.671 INFO:teuthology.orchestra.run.smithi060.stdout: 414143 4 -rw-r--r-- 1 ubuntu ubuntu 1115 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/url-builder.service.spec.ts 2024-03-27T21:42:43.671 INFO:teuthology.orchestra.run.smithi060.stdout: 414090 4 -rw-r--r-- 1 ubuntu ubuntu 969 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/feature-toggles.service.ts 2024-03-27T21:42:43.671 INFO:teuthology.orchestra.run.smithi060.stdout: 414078 4 -rw-r--r-- 1 ubuntu ubuntu 1645 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/crud-form-adapter.service.ts 2024-03-27T21:42:43.671 INFO:teuthology.orchestra.run.smithi060.stdout: 414072 4 -rw-r--r-- 1 ubuntu ubuntu 342 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/cd-table-server-side.service.ts 2024-03-27T21:42:43.671 INFO:teuthology.orchestra.run.smithi060.stdout: 414136 4 -rw-r--r-- 1 ubuntu ubuntu 647 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/text-area-json-formatter.service.ts 2024-03-27T21:42:43.671 INFO:teuthology.orchestra.run.smithi060.stdout: 414145 4 -rw-r--r-- 1 ubuntu ubuntu 383 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/wizard-steps.service.spec.ts 2024-03-27T21:42:43.672 INFO:teuthology.orchestra.run.smithi060.stdout: 414129 12 -rw-r--r-- 1 ubuntu ubuntu 10908 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/task-message.service.spec.ts 2024-03-27T21:42:43.672 INFO:teuthology.orchestra.run.smithi060.stdout: 414098 4 -rw-r--r-- 1 ubuntu ubuntu 3375 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/module-status-guard.service.spec.ts 2024-03-27T21:42:43.672 INFO:teuthology.orchestra.run.smithi060.stdout: 414119 4 -rw-r--r-- 1 ubuntu ubuntu 1435 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/rbd-configuration.service.spec.ts 2024-03-27T21:42:43.672 INFO:teuthology.orchestra.run.smithi060.stdout: 414073 4 -rw-r--r-- 1 ubuntu ubuntu 2666 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/change-password-guard.service.spec.ts 2024-03-27T21:42:43.672 INFO:teuthology.orchestra.run.smithi060.stdout: 414075 4 -rw-r--r-- 1 ubuntu ubuntu 360 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/cookie.service.spec.ts 2024-03-27T21:42:43.672 INFO:teuthology.orchestra.run.smithi060.stdout: 414130 20 -rw-r--r-- 1 ubuntu ubuntu 18161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/task-message.service.ts 2024-03-27T21:42:43.672 INFO:teuthology.orchestra.run.smithi060.stdout: 414095 4 -rw-r--r-- 1 ubuntu ubuntu 568 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/language.service.ts 2024-03-27T21:42:43.672 INFO:teuthology.orchestra.run.smithi060.stdout: 414082 4 -rw-r--r-- 1 ubuntu ubuntu 1730 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/device.service.ts 2024-03-27T21:42:43.672 INFO:teuthology.orchestra.run.smithi060.stdout: 414091 8 -rw-r--r-- 1 ubuntu ubuntu 4757 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/formatter.service.spec.ts 2024-03-27T21:42:43.673 INFO:teuthology.orchestra.run.smithi060.stdout: 414105 12 -rw-r--r-- 1 ubuntu ubuntu 9478 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/notification.service.spec.ts 2024-03-27T21:42:43.673 INFO:teuthology.orchestra.run.smithi060.stdout: 414070 4 -rw-r--r-- 1 ubuntu ubuntu 1645 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/auth-storage.service.ts 2024-03-27T21:42:43.673 INFO:teuthology.orchestra.run.smithi060.stdout: 414074 4 -rw-r--r-- 1 ubuntu ubuntu 1351 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/change-password-guard.service.ts 2024-03-27T21:42:43.673 INFO:teuthology.orchestra.run.smithi060.stdout: 414101 4 -rw-r--r-- 1 ubuntu ubuntu 2757 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/motd-notification.service.ts 2024-03-27T21:42:43.673 INFO:teuthology.orchestra.run.smithi060.stdout: 414124 4 -rw-r--r-- 1 ubuntu ubuntu 2702 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/summary.service.ts 2024-03-27T21:42:43.673 INFO:teuthology.orchestra.run.smithi060.stdout: 414117 8 -rw-r--r-- 1 ubuntu ubuntu 4134 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/prometheus-silence-matcher.service.spec.ts 2024-03-27T21:42:43.674 INFO:teuthology.orchestra.run.smithi060.stdout: 414100 4 -rw-r--r-- 1 ubuntu ubuntu 3620 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/motd-notification.service.spec.ts 2024-03-27T21:42:43.674 INFO:teuthology.orchestra.run.smithi060.stdout: 414138 4 -rw-r--r-- 1 ubuntu ubuntu 266 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/text-to-download.service.ts 2024-03-27T21:42:43.674 INFO:teuthology.orchestra.run.smithi060.stdout: 414144 4 -rw-r--r-- 1 ubuntu ubuntu 1491 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/url-builder.service.ts 2024-03-27T21:42:43.674 INFO:teuthology.orchestra.run.smithi060.stdout: 414065 8 -rw-r--r-- 1 ubuntu ubuntu 7151 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/api-interceptor.service.spec.ts 2024-03-27T21:42:43.674 INFO:teuthology.orchestra.run.smithi060.stdout: 414137 4 -rw-r--r-- 1 ubuntu ubuntu 489 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/text-to-download.service.spec.ts 2024-03-27T21:42:43.674 INFO:teuthology.orchestra.run.smithi060.stdout: 414123 8 -rw-r--r-- 1 ubuntu ubuntu 4982 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/summary.service.spec.ts 2024-03-27T21:42:43.674 INFO:teuthology.orchestra.run.smithi060.stdout: 414111 4 -rw-r--r-- 1 ubuntu ubuntu 3223 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/prometheus-alert-formatter.spec.ts 2024-03-27T21:42:43.674 INFO:teuthology.orchestra.run.smithi060.stdout: 414106 8 -rw-r--r-- 1 ubuntu ubuntu 7101 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/notification.service.ts 2024-03-27T21:42:43.675 INFO:teuthology.orchestra.run.smithi060.stdout: 414133 4 -rw-r--r-- 1 ubuntu ubuntu 967 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/telemetry-notification.service.spec.ts 2024-03-27T21:42:43.675 INFO:teuthology.orchestra.run.smithi060.stdout: 414066 8 -rw-r--r-- 1 ubuntu ubuntu 5497 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/api-interceptor.service.ts 2024-03-27T21:42:43.675 INFO:teuthology.orchestra.run.smithi060.stdout: 414126 4 -rw-r--r-- 1 ubuntu ubuntu 3835 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/task-list.service.ts 2024-03-27T21:42:43.675 INFO:teuthology.orchestra.run.smithi060.stdout: 414083 4 -rw-r--r-- 1 ubuntu ubuntu 2091 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/doc.service.spec.ts 2024-03-27T21:42:43.675 INFO:teuthology.orchestra.run.smithi060.stdout: 414093 4 -rw-r--r-- 1 ubuntu ubuntu 1014 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/js-error-handler.service.ts 2024-03-27T21:42:43.675 INFO:teuthology.orchestra.run.smithi060.stdout: 414141 4 -rw-r--r-- 1 ubuntu ubuntu 2017 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/timer.service.spec.ts 2024-03-27T21:42:43.675 INFO:teuthology.orchestra.run.smithi060.stdout: 414076 4 -rw-r--r-- 1 ubuntu ubuntu 497 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/cookie.service.ts 2024-03-27T21:42:43.675 INFO:teuthology.orchestra.run.smithi060.stdout: 414087 4 -rw-r--r-- 1 ubuntu ubuntu 2369 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/feature-toggles-guard.service.spec.ts 2024-03-27T21:42:43.675 INFO:teuthology.orchestra.run.smithi060.stdout: 414081 4 -rw-r--r-- 1 ubuntu ubuntu 3065 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/device.service.spec.ts 2024-03-27T21:42:43.676 INFO:teuthology.orchestra.run.smithi060.stdout: 414079 4 -rw-r--r-- 1 ubuntu ubuntu 639 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/data-gateway.service.spec.ts 2024-03-27T21:42:43.676 INFO:teuthology.orchestra.run.smithi060.stdout: 414094 4 -rw-r--r-- 1 ubuntu ubuntu 919 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/language.service.spec.ts 2024-03-27T21:42:43.676 INFO:teuthology.orchestra.run.smithi060.stdout: 414069 4 -rw-r--r-- 1 ubuntu ubuntu 1189 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/auth-storage.service.spec.ts 2024-03-27T21:42:43.676 INFO:teuthology.orchestra.run.smithi060.stdout: 414115 8 -rw-r--r-- 1 ubuntu ubuntu 8087 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/prometheus-notification.service.spec.ts 2024-03-27T21:42:43.676 INFO:teuthology.orchestra.run.smithi060.stdout: 414112 4 -rw-r--r-- 1 ubuntu ubuntu 2418 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/services/prometheus-alert-formatter.ts 2024-03-27T21:42:43.676 INFO:teuthology.orchestra.run.smithi060.stdout: 414061 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/rxjs 2024-03-27T21:42:43.676 INFO:teuthology.orchestra.run.smithi060.stdout: 414062 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/rxjs/operators 2024-03-27T21:42:43.676 INFO:teuthology.orchestra.run.smithi060.stdout: 414063 4 -rw-r--r-- 1 ubuntu ubuntu 573 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/rxjs/operators/page-visibility.operator.ts 2024-03-27T21:42:43.676 INFO:teuthology.orchestra.run.smithi060.stdout: 413878 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms 2024-03-27T21:42:43.677 INFO:teuthology.orchestra.run.smithi060.stdout: 413879 4 -rw-r--r-- 1 ubuntu ubuntu 845 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/cd-form-builder.spec.ts 2024-03-27T21:42:43.677 INFO:teuthology.orchestra.run.smithi060.stdout: 413882 4 -rw-r--r-- 1 ubuntu ubuntu 2042 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/cd-form-group.ts 2024-03-27T21:42:43.677 INFO:teuthology.orchestra.run.smithi060.stdout: 413885 28 -rw-r--r-- 1 ubuntu ubuntu 27740 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/cd-validators.spec.ts 2024-03-27T21:42:43.677 INFO:teuthology.orchestra.run.smithi060.stdout: 413886 24 -rw-r--r-- 1 ubuntu ubuntu 22900 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/cd-validators.ts 2024-03-27T21:42:43.677 INFO:teuthology.orchestra.run.smithi060.stdout: 413887 4 drwxr-xr-x 9 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form 2024-03-27T21:42:43.677 INFO:teuthology.orchestra.run.smithi060.stdout: 413904 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-input-type 2024-03-27T21:42:43.677 INFO:teuthology.orchestra.run.smithi060.stdout: 413905 4 -rw-r--r-- 1 ubuntu ubuntu 114 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-input-type/formly-input-type.component.html 2024-03-27T21:42:43.677 INFO:teuthology.orchestra.run.smithi060.stdout: 413906 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-input-type/formly-input-type.component.scss 2024-03-27T21:42:43.678 INFO:teuthology.orchestra.run.smithi060.stdout: 413908 4 -rw-r--r-- 1 ubuntu ubuntu 340 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-input-type/formly-input-type.component.ts 2024-03-27T21:42:43.678 INFO:teuthology.orchestra.run.smithi060.stdout: 413907 4 -rw-r--r-- 1 ubuntu ubuntu 1251 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-input-type/formly-input-type.component.spec.ts 2024-03-27T21:42:43.678 INFO:teuthology.orchestra.run.smithi060.stdout: 413898 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-file-type 2024-03-27T21:42:43.678 INFO:teuthology.orchestra.run.smithi060.stdout: 413902 4 -rw-r--r-- 1 ubuntu ubuntu 1025 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-file-type/formly-file-type.component.spec.ts 2024-03-27T21:42:43.678 INFO:teuthology.orchestra.run.smithi060.stdout: 413900 4 -rw-r--r-- 1 ubuntu ubuntu 98 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-file-type/formly-file-type.component.html 2024-03-27T21:42:43.678 INFO:teuthology.orchestra.run.smithi060.stdout: 413903 4 -rw-r--r-- 1 ubuntu ubuntu 302 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-file-type/formly-file-type.component.ts 2024-03-27T21:42:43.678 INFO:teuthology.orchestra.run.smithi060.stdout: 413901 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-file-type/formly-file-type.component.scss 2024-03-27T21:42:43.679 INFO:teuthology.orchestra.run.smithi060.stdout: 413899 4 -rw-r--r-- 1 ubuntu ubuntu 826 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-file-type/formly-file-type-accessor.ts 2024-03-27T21:42:43.679 INFO:teuthology.orchestra.run.smithi060.stdout: 413893 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-array-type 2024-03-27T21:42:43.679 INFO:teuthology.orchestra.run.smithi060.stdout: 413897 4 -rw-r--r-- 1 ubuntu ubuntu 1031 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-array-type/formly-array-type.component.ts 2024-03-27T21:42:43.679 INFO:teuthology.orchestra.run.smithi060.stdout: 413896 4 -rw-r--r-- 1 ubuntu ubuntu 1250 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-array-type/formly-array-type.component.spec.ts 2024-03-27T21:42:43.679 INFO:teuthology.orchestra.run.smithi060.stdout: 413894 4 -rw-r--r-- 1 ubuntu ubuntu 1251 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-array-type/formly-array-type.component.html 2024-03-27T21:42:43.679 INFO:teuthology.orchestra.run.smithi060.stdout: 413895 4 -rw-r--r-- 1 ubuntu ubuntu 38 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-array-type/formly-array-type.component.scss 2024-03-27T21:42:43.679 INFO:teuthology.orchestra.run.smithi060.stdout: 413909 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-input-wrapper 2024-03-27T21:42:43.679 INFO:teuthology.orchestra.run.smithi060.stdout: 413913 4 -rw-r--r-- 1 ubuntu ubuntu 483 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-input-wrapper/formly-input-wrapper.component.ts 2024-03-27T21:42:43.679 INFO:teuthology.orchestra.run.smithi060.stdout: 413910 4 -rw-r--r-- 1 ubuntu ubuntu 1209 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-input-wrapper/formly-input-wrapper.component.html 2024-03-27T21:42:43.680 INFO:teuthology.orchestra.run.smithi060.stdout: 413912 4 -rw-r--r-- 1 ubuntu ubuntu 1266 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-input-wrapper/formly-input-wrapper.component.spec.ts 2024-03-27T21:42:43.680 INFO:teuthology.orchestra.run.smithi060.stdout: 413911 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-input-wrapper/formly-input-wrapper.component.scss 2024-03-27T21:42:43.680 INFO:teuthology.orchestra.run.smithi060.stdout: 413925 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/validators 2024-03-27T21:42:43.680 INFO:teuthology.orchestra.run.smithi060.stdout: 413928 4 -rw-r--r-- 1 ubuntu ubuntu 882 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/validators/rgw-role-validator.ts 2024-03-27T21:42:43.680 INFO:teuthology.orchestra.run.smithi060.stdout: 413927 4 -rw-r--r-- 1 ubuntu ubuntu 306 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/validators/json-validator.ts 2024-03-27T21:42:43.680 INFO:teuthology.orchestra.run.smithi060.stdout: 413926 4 -rw-r--r-- 1 ubuntu ubuntu 426 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/validators/file-validator.ts 2024-03-27T21:42:43.680 INFO:teuthology.orchestra.run.smithi060.stdout: 413890 4 -rw-r--r-- 1 ubuntu ubuntu 1244 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/crud-form.component.spec.ts 2024-03-27T21:42:43.680 INFO:teuthology.orchestra.run.smithi060.stdout: 413919 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-textarea-type 2024-03-27T21:42:43.681 INFO:teuthology.orchestra.run.smithi060.stdout: 413923 4 -rw-r--r-- 1 ubuntu ubuntu 723 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-textarea-type/formly-textarea-type.component.ts 2024-03-27T21:42:43.681 INFO:teuthology.orchestra.run.smithi060.stdout: 413922 4 -rw-r--r-- 1 ubuntu ubuntu 1282 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-textarea-type/formly-textarea-type.component.spec.ts 2024-03-27T21:42:43.681 INFO:teuthology.orchestra.run.smithi060.stdout: 413920 4 -rw-r--r-- 1 ubuntu ubuntu 272 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-textarea-type/formly-textarea-type.component.html 2024-03-27T21:42:43.681 INFO:teuthology.orchestra.run.smithi060.stdout: 413921 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-textarea-type/formly-textarea-type.component.scss 2024-03-27T21:42:43.681 INFO:teuthology.orchestra.run.smithi060.stdout: 413924 4 -rw-r--r-- 1 ubuntu ubuntu 1486 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/helpers.ts 2024-03-27T21:42:43.681 INFO:teuthology.orchestra.run.smithi060.stdout: 413888 4 -rw-r--r-- 1 ubuntu ubuntu 1007 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/crud-form.component.html 2024-03-27T21:42:43.681 INFO:teuthology.orchestra.run.smithi060.stdout: 413914 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-object-type 2024-03-27T21:42:43.681 INFO:teuthology.orchestra.run.smithi060.stdout: 413916 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-object-type/formly-object-type.component.scss 2024-03-27T21:42:43.682 INFO:teuthology.orchestra.run.smithi060.stdout: 413915 4 -rw-r--r-- 1 ubuntu ubuntu 567 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-object-type/formly-object-type.component.html 2024-03-27T21:42:43.682 INFO:teuthology.orchestra.run.smithi060.stdout: 413917 4 -rw-r--r-- 1 ubuntu ubuntu 1270 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-object-type/formly-object-type.component.spec.ts 2024-03-27T21:42:43.682 INFO:teuthology.orchestra.run.smithi060.stdout: 413918 4 -rw-r--r-- 1 ubuntu ubuntu 852 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/formly-object-type/formly-object-type.component.ts 2024-03-27T21:42:43.682 INFO:teuthology.orchestra.run.smithi060.stdout: 413889 4 -rw-r--r-- 1 ubuntu ubuntu 384 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/crud-form.component.scss 2024-03-27T21:42:43.682 INFO:teuthology.orchestra.run.smithi060.stdout: 413892 4 -rw-r--r-- 1 ubuntu ubuntu 309 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/crud-form.model.ts 2024-03-27T21:42:43.682 INFO:teuthology.orchestra.run.smithi060.stdout: 413891 4 -rw-r--r-- 1 ubuntu ubuntu 3494 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/crud-form/crud-form.component.ts 2024-03-27T21:42:43.682 INFO:teuthology.orchestra.run.smithi060.stdout: 413881 8 -rw-r--r-- 1 ubuntu ubuntu 6011 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/cd-form-group.spec.ts 2024-03-27T21:42:43.682 INFO:teuthology.orchestra.run.smithi060.stdout: 413880 4 -rw-r--r-- 1 ubuntu ubuntu 604 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/cd-form-builder.ts 2024-03-27T21:42:43.683 INFO:teuthology.orchestra.run.smithi060.stdout: 413884 4 -rw-r--r-- 1 ubuntu ubuntu 382 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/cd-form.ts 2024-03-27T21:42:43.683 INFO:teuthology.orchestra.run.smithi060.stdout: 413883 4 -rw-r--r-- 1 ubuntu ubuntu 863 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/forms/cd-form.spec.ts 2024-03-27T21:42:43.683 INFO:teuthology.orchestra.run.smithi060.stdout: 413493 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api 2024-03-27T21:42:43.683 INFO:teuthology.orchestra.run.smithi060.stdout: 413526 4 -rw-r--r-- 1 ubuntu ubuntu 1119 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/health.service.spec.ts 2024-03-27T21:42:43.683 INFO:teuthology.orchestra.run.smithi060.stdout: 413497 4 -rw-r--r-- 1 ubuntu ubuntu 1547 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/auth.service.ts 2024-03-27T21:42:43.683 INFO:teuthology.orchestra.run.smithi060.stdout: 413566 4 -rw-r--r-- 1 ubuntu ubuntu 1130 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-multisite.service.ts 2024-03-27T21:42:43.683 INFO:teuthology.orchestra.run.smithi060.stdout: 413509 4 -rw-r--r-- 1 ubuntu ubuntu 630 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/cluster.service.ts 2024-03-27T21:42:43.684 INFO:teuthology.orchestra.run.smithi060.stdout: 413508 4 -rw-r--r-- 1 ubuntu ubuntu 1240 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/cluster.service.spec.ts 2024-03-27T21:42:43.684 INFO:teuthology.orchestra.run.smithi060.stdout: 413544 4 -rw-r--r-- 1 ubuntu ubuntu 2392 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/nfs.service.spec.ts 2024-03-27T21:42:43.684 INFO:teuthology.orchestra.run.smithi060.stdout: 413567 4 -rw-r--r-- 1 ubuntu ubuntu 570 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-realm.service.spec.ts 2024-03-27T21:42:43.684 INFO:teuthology.orchestra.run.smithi060.stdout: 413538 4 -rw-r--r-- 1 ubuntu ubuntu 914 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/monitor.service.spec.ts 2024-03-27T21:42:43.684 INFO:teuthology.orchestra.run.smithi060.stdout: 413582 4 -rw-r--r-- 1 ubuntu ubuntu 2010 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/settings.service.ts 2024-03-27T21:42:43.684 INFO:teuthology.orchestra.run.smithi060.stdout: 413588 4 -rw-r--r-- 1 ubuntu ubuntu 1708 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/user.service.ts 2024-03-27T21:42:43.684 INFO:teuthology.orchestra.run.smithi060.stdout: 413516 4 -rw-r--r-- 1 ubuntu ubuntu 1055 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/daemon.service.spec.ts 2024-03-27T21:42:43.684 INFO:teuthology.orchestra.run.smithi060.stdout: 413513 4 -rw-r--r-- 1 ubuntu ubuntu 921 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/crush-rule.service.ts 2024-03-27T21:42:43.684 INFO:teuthology.orchestra.run.smithi060.stdout: 413511 4 -rw-r--r-- 1 ubuntu ubuntu 1623 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/configuration.service.ts 2024-03-27T21:42:43.685 INFO:teuthology.orchestra.run.smithi060.stdout: 413561 8 -rw-r--r-- 1 ubuntu ubuntu 5712 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rbd.service.ts 2024-03-27T21:42:43.685 INFO:teuthology.orchestra.run.smithi060.stdout: 413515 4 -rw-r--r-- 1 ubuntu ubuntu 339 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/custom-login-banner.service.ts 2024-03-27T21:42:43.685 INFO:teuthology.orchestra.run.smithi060.stdout: 413498 4 -rw-r--r-- 1 ubuntu ubuntu 2170 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/ceph-service.service.ts 2024-03-27T21:42:43.685 INFO:teuthology.orchestra.run.smithi060.stdout: 413563 8 -rw-r--r-- 1 ubuntu ubuntu 6118 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-bucket.service.ts 2024-03-27T21:42:43.685 INFO:teuthology.orchestra.run.smithi060.stdout: 413512 4 -rw-r--r-- 1 ubuntu ubuntu 1380 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/crush-rule.service.spec.ts 2024-03-27T21:42:43.685 INFO:teuthology.orchestra.run.smithi060.stdout: 413535 4 -rw-r--r-- 1 ubuntu ubuntu 369 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/logs.service.ts 2024-03-27T21:42:43.685 INFO:teuthology.orchestra.run.smithi060.stdout: 413501 8 -rw-r--r-- 1 ubuntu ubuntu 5626 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/cephfs-snapshot-schedule.service.ts 2024-03-27T21:42:43.685 INFO:teuthology.orchestra.run.smithi060.stdout: 413555 8 -rw-r--r-- 1 ubuntu ubuntu 6817 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/prometheus.service.spec.ts 2024-03-27T21:42:43.686 INFO:teuthology.orchestra.run.smithi060.stdout: 413576 4 -rw-r--r-- 1 ubuntu ubuntu 3038 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-zonegroup.service.ts 2024-03-27T21:42:43.686 INFO:teuthology.orchestra.run.smithi060.stdout: 413533 4 -rw-r--r-- 1 ubuntu ubuntu 419 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/logging.service.ts 2024-03-27T21:42:43.686 INFO:teuthology.orchestra.run.smithi060.stdout: 413530 4 -rw-r--r-- 1 ubuntu ubuntu 2993 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/iscsi.service.spec.ts 2024-03-27T21:42:43.686 INFO:teuthology.orchestra.run.smithi060.stdout: 413541 4 -rw-r--r-- 1 ubuntu ubuntu 550 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/motd.service.ts 2024-03-27T21:42:43.686 INFO:teuthology.orchestra.run.smithi060.stdout: 413568 4 -rw-r--r-- 1 ubuntu ubuntu 2388 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-realm.service.ts 2024-03-27T21:42:43.686 INFO:teuthology.orchestra.run.smithi060.stdout: 413557 8 -rw-r--r-- 1 ubuntu ubuntu 4861 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rbd-mirroring.service.spec.ts 2024-03-27T21:42:43.686 INFO:teuthology.orchestra.run.smithi060.stdout: 413505 8 -rw-r--r-- 1 ubuntu ubuntu 4938 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/cephfs-subvolume.service.ts 2024-03-27T21:42:43.686 INFO:teuthology.orchestra.run.smithi060.stdout: 413502 4 -rw-r--r-- 1 ubuntu ubuntu 677 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/cephfs-subvolume-group.service.spec.ts 2024-03-27T21:42:43.687 INFO:teuthology.orchestra.run.smithi060.stdout: 413581 8 -rw-r--r-- 1 ubuntu ubuntu 4884 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/settings.service.spec.ts 2024-03-27T21:42:43.687 INFO:teuthology.orchestra.run.smithi060.stdout: 413504 4 -rw-r--r-- 1 ubuntu ubuntu 1832 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/cephfs-subvolume.service.spec.ts 2024-03-27T21:42:43.687 INFO:teuthology.orchestra.run.smithi060.stdout: 413586 4 -rw-r--r-- 1 ubuntu ubuntu 2170 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/upgrade.service.ts 2024-03-27T21:42:43.687 INFO:teuthology.orchestra.run.smithi060.stdout: 413519 4 -rw-r--r-- 1 ubuntu ubuntu 1859 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/directory-store.service.ts 2024-03-27T21:42:43.687 INFO:teuthology.orchestra.run.smithi060.stdout: 413506 4 -rw-r--r-- 1 ubuntu ubuntu 4075 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/cephfs.service.spec.ts 2024-03-27T21:42:43.687 INFO:teuthology.orchestra.run.smithi060.stdout: 413556 12 -rw-r--r-- 1 ubuntu ubuntu 8855 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/prometheus.service.ts 2024-03-27T21:42:43.687 INFO:teuthology.orchestra.run.smithi060.stdout: 413534 4 -rw-r--r-- 1 ubuntu ubuntu 891 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/logs.service.spec.ts 2024-03-27T21:42:43.687 INFO:teuthology.orchestra.run.smithi060.stdout: 413564 4 -rw-r--r-- 1 ubuntu ubuntu 3365 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-daemon.service.spec.ts 2024-03-27T21:42:43.688 INFO:teuthology.orchestra.run.smithi060.stdout: 413584 4 -rw-r--r-- 1 ubuntu ubuntu 506 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/telemetry.service.ts 2024-03-27T21:42:43.688 INFO:teuthology.orchestra.run.smithi060.stdout: 413579 4 -rw-r--r-- 1 ubuntu ubuntu 891 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/scope.service.spec.ts 2024-03-27T21:42:43.688 INFO:teuthology.orchestra.run.smithi060.stdout: 413523 4 -rw-r--r-- 1 ubuntu ubuntu 775 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/feedback.service.ts 2024-03-27T21:42:43.688 INFO:teuthology.orchestra.run.smithi060.stdout: 413585 4 -rw-r--r-- 1 ubuntu ubuntu 2105 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/upgrade.service.spec.ts 2024-03-27T21:42:43.688 INFO:teuthology.orchestra.run.smithi060.stdout: 413507 4 -rw-r--r-- 1 ubuntu ubuntu 3236 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/cephfs.service.ts 2024-03-27T21:42:43.688 INFO:teuthology.orchestra.run.smithi060.stdout: 413525 4 -rw-r--r-- 1 ubuntu ubuntu 460 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/hardware.service.ts 2024-03-27T21:42:43.688 INFO:teuthology.orchestra.run.smithi060.stdout: 413536 4 -rw-r--r-- 1 ubuntu ubuntu 1997 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/mgr-module.service.spec.ts 2024-03-27T21:42:43.688 INFO:teuthology.orchestra.run.smithi060.stdout: 413503 4 -rw-r--r-- 1 ubuntu ubuntu 1934 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/cephfs-subvolume-group.service.ts 2024-03-27T21:42:43.689 INFO:teuthology.orchestra.run.smithi060.stdout: 413528 4 -rw-r--r-- 1 ubuntu ubuntu 3148 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/host.service.spec.ts 2024-03-27T21:42:43.689 INFO:teuthology.orchestra.run.smithi060.stdout: 413500 4 -rw-r--r-- 1 ubuntu ubuntu 649 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/cephfs-snapshot-schedule.service.spec.ts 2024-03-27T21:42:43.689 INFO:teuthology.orchestra.run.smithi060.stdout: 413551 4 -rw-r--r-- 1 ubuntu ubuntu 1312 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/performance-counter.service.spec.ts 2024-03-27T21:42:43.689 INFO:teuthology.orchestra.run.smithi060.stdout: 413552 4 -rw-r--r-- 1 ubuntu ubuntu 686 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/performance-counter.service.ts 2024-03-27T21:42:43.689 INFO:teuthology.orchestra.run.smithi060.stdout: 413571 8 -rw-r--r-- 1 ubuntu ubuntu 5579 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-user.service.spec.ts 2024-03-27T21:42:43.689 INFO:teuthology.orchestra.run.smithi060.stdout: 413499 4 -rw-r--r-- 1 ubuntu ubuntu 324 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/ceph-user.service.ts 2024-03-27T21:42:43.689 INFO:teuthology.orchestra.run.smithi060.stdout: 413494 4 -rw-r--r-- 1 ubuntu ubuntu 324 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/api-client.spec.ts 2024-03-27T21:42:43.689 INFO:teuthology.orchestra.run.smithi060.stdout: 413560 8 -rw-r--r-- 1 ubuntu ubuntu 6761 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rbd.service.spec.ts 2024-03-27T21:42:43.690 INFO:teuthology.orchestra.run.smithi060.stdout: 413550 4 -rw-r--r-- 1 ubuntu ubuntu 390 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/paginate.model.ts 2024-03-27T21:42:43.690 INFO:teuthology.orchestra.run.smithi060.stdout: 413577 4 -rw-r--r-- 1 ubuntu ubuntu 2204 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/role.service.spec.ts 2024-03-27T21:42:43.690 INFO:teuthology.orchestra.run.smithi060.stdout: 413543 8 -rw-r--r-- 1 ubuntu ubuntu 6229 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/multi-cluster.service.ts 2024-03-27T21:42:43.690 INFO:teuthology.orchestra.run.smithi060.stdout: 413570 4 -rw-r--r-- 1 ubuntu ubuntu 1123 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-site.service.ts 2024-03-27T21:42:43.690 INFO:teuthology.orchestra.run.smithi060.stdout: 413554 4 -rw-r--r-- 1 ubuntu ubuntu 1976 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/pool.service.ts 2024-03-27T21:42:43.690 INFO:teuthology.orchestra.run.smithi060.stdout: 413574 8 -rw-r--r-- 1 ubuntu ubuntu 4990 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-zone.service.ts 2024-03-27T21:42:43.690 INFO:teuthology.orchestra.run.smithi060.stdout: 413532 4 -rw-r--r-- 1 ubuntu ubuntu 1046 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/logging.service.spec.ts 2024-03-27T21:42:43.690 INFO:teuthology.orchestra.run.smithi060.stdout: 413572 8 -rw-r--r-- 1 ubuntu ubuntu 5621 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-user.service.ts 2024-03-27T21:42:43.691 INFO:teuthology.orchestra.run.smithi060.stdout: 413580 4 -rw-r--r-- 1 ubuntu ubuntu 265 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/scope.service.ts 2024-03-27T21:42:43.691 INFO:teuthology.orchestra.run.smithi060.stdout: 413539 4 -rw-r--r-- 1 ubuntu ubuntu 272 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/monitor.service.ts 2024-03-27T21:42:43.691 INFO:teuthology.orchestra.run.smithi060.stdout: 413583 4 -rw-r--r-- 1 ubuntu ubuntu 1863 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/telemetry.service.spec.ts 2024-03-27T21:42:43.691 INFO:teuthology.orchestra.run.smithi060.stdout: 413521 8 -rw-r--r-- 1 ubuntu ubuntu 5663 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/erasure-code-profile.service.ts 2024-03-27T21:42:43.691 INFO:teuthology.orchestra.run.smithi060.stdout: 413575 4 -rw-r--r-- 1 ubuntu ubuntu 590 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-zonegroup.service.spec.ts 2024-03-27T21:42:43.691 INFO:teuthology.orchestra.run.smithi060.stdout: 413517 4 -rw-r--r-- 1 ubuntu ubuntu 868 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/daemon.service.ts 2024-03-27T21:42:43.691 INFO:teuthology.orchestra.run.smithi060.stdout: 413573 4 -rw-r--r-- 1 ubuntu ubuntu 565 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-zone.service.spec.ts 2024-03-27T21:42:43.691 INFO:teuthology.orchestra.run.smithi060.stdout: 413520 4 -rw-r--r-- 1 ubuntu ubuntu 1721 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/erasure-code-profile.service.spec.ts 2024-03-27T21:42:43.692 INFO:teuthology.orchestra.run.smithi060.stdout: 413569 4 -rw-r--r-- 1 ubuntu ubuntu 1260 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-site.service.spec.ts 2024-03-27T21:42:43.692 INFO:teuthology.orchestra.run.smithi060.stdout: 413546 4 -rw-r--r-- 1 ubuntu ubuntu 987 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/orchestrator.service.spec.ts 2024-03-27T21:42:43.692 INFO:teuthology.orchestra.run.smithi060.stdout: 413527 4 -rw-r--r-- 1 ubuntu ubuntu 621 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/health.service.ts 2024-03-27T21:42:43.692 INFO:teuthology.orchestra.run.smithi060.stdout: 413548 8 -rw-r--r-- 1 ubuntu ubuntu 5832 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/osd.service.spec.ts 2024-03-27T21:42:43.692 INFO:teuthology.orchestra.run.smithi060.stdout: 413531 4 -rw-r--r-- 1 ubuntu ubuntu 1350 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/iscsi.service.ts 2024-03-27T21:42:43.692 INFO:teuthology.orchestra.run.smithi060.stdout: 413495 4 -rw-r--r-- 1 ubuntu ubuntu 159 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/api-client.ts 2024-03-27T21:42:43.692 INFO:teuthology.orchestra.run.smithi060.stdout: 413545 4 -rw-r--r-- 1 ubuntu ubuntu 2750 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/nfs.service.ts 2024-03-27T21:42:43.692 INFO:teuthology.orchestra.run.smithi060.stdout: 413562 8 -rw-r--r-- 1 ubuntu ubuntu 4107 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-bucket.service.spec.ts 2024-03-27T21:42:43.692 INFO:teuthology.orchestra.run.smithi060.stdout: 413547 4 -rw-r--r-- 1 ubuntu ubuntu 1406 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/orchestrator.service.ts 2024-03-27T21:42:43.693 INFO:teuthology.orchestra.run.smithi060.stdout: 413518 4 -rw-r--r-- 1 ubuntu ubuntu 682 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/directory-store.service.spec.ts 2024-03-27T21:42:43.693 INFO:teuthology.orchestra.run.smithi060.stdout: 413510 4 -rw-r--r-- 1 ubuntu ubuntu 3396 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/configuration.service.spec.ts 2024-03-27T21:42:43.693 INFO:teuthology.orchestra.run.smithi060.stdout: 413540 4 -rw-r--r-- 1 ubuntu ubuntu 897 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/motd.service.spec.ts 2024-03-27T21:42:43.693 INFO:teuthology.orchestra.run.smithi060.stdout: 413578 4 -rw-r--r-- 1 ubuntu ubuntu 1192 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/role.service.ts 2024-03-27T21:42:43.693 INFO:teuthology.orchestra.run.smithi060.stdout: 413537 4 -rw-r--r-- 1 ubuntu ubuntu 1740 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/mgr-module.service.ts 2024-03-27T21:42:43.693 INFO:teuthology.orchestra.run.smithi060.stdout: 413542 4 -rw-r--r-- 1 ubuntu ubuntu 506 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/multi-cluster.service.spec.ts 2024-03-27T21:42:43.693 INFO:teuthology.orchestra.run.smithi060.stdout: 413514 4 -rw-r--r-- 1 ubuntu ubuntu 1028 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/custom-login-banner.service.spec.ts 2024-03-27T21:42:43.693 INFO:teuthology.orchestra.run.smithi060.stdout: 413565 4 -rw-r--r-- 1 ubuntu ubuntu 2814 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-daemon.service.ts 2024-03-27T21:42:43.694 INFO:teuthology.orchestra.run.smithi060.stdout: 413587 4 -rw-r--r-- 1 ubuntu ubuntu 3479 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/user.service.spec.ts 2024-03-27T21:42:43.694 INFO:teuthology.orchestra.run.smithi060.stdout: 413522 4 -rw-r--r-- 1 ubuntu ubuntu 1352 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/feedback.service.spec.ts 2024-03-27T21:42:43.694 INFO:teuthology.orchestra.run.smithi060.stdout: 413524 4 -rw-r--r-- 1 ubuntu ubuntu 603 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/hardware.service.spec.ts 2024-03-27T21:42:43.694 INFO:teuthology.orchestra.run.smithi060.stdout: 413558 4 -rw-r--r-- 1 ubuntu ubuntu 3463 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rbd-mirroring.service.ts 2024-03-27T21:42:43.694 INFO:teuthology.orchestra.run.smithi060.stdout: 413559 4 -rw-r--r-- 1 ubuntu ubuntu 618 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rbd.model.ts 2024-03-27T21:42:43.694 INFO:teuthology.orchestra.run.smithi060.stdout: 413496 4 -rw-r--r-- 1 ubuntu ubuntu 1948 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/auth.service.spec.ts 2024-03-27T21:42:43.695 INFO:teuthology.orchestra.run.smithi060.stdout: 413549 8 -rw-r--r-- 1 ubuntu ubuntu 5236 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/osd.service.ts 2024-03-27T21:42:43.695 INFO:teuthology.orchestra.run.smithi060.stdout: 413553 4 -rw-r--r-- 1 ubuntu ubuntu 3838 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/pool.service.spec.ts 2024-03-27T21:42:43.695 INFO:teuthology.orchestra.run.smithi060.stdout: 413529 8 -rw-r--r-- 1 ubuntu ubuntu 5182 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/api/host.service.ts 2024-03-27T21:42:43.695 INFO:teuthology.orchestra.run.smithi060.stdout: 413828 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/decorators 2024-03-27T21:42:43.695 INFO:teuthology.orchestra.run.smithi060.stdout: 413829 4 -rw-r--r-- 1 ubuntu ubuntu 848 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/decorators/cd-encode.spec.ts 2024-03-27T21:42:43.695 INFO:teuthology.orchestra.run.smithi060.stdout: 413830 4 -rw-r--r-- 1 ubuntu ubuntu 2320 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/shared/decorators/cd-encode.ts 2024-03-27T21:42:43.695 INFO:teuthology.orchestra.run.smithi060.stdout: 412534 20 -rw-r--r-- 1 ubuntu ubuntu 16689 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/app-routing.module.ts 2024-03-27T21:42:43.695 INFO:teuthology.orchestra.run.smithi060.stdout: 412535 4 -rw-r--r-- 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/app.component.html 2024-03-27T21:42:43.696 INFO:teuthology.orchestra.run.smithi060.stdout: 412536 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/app.component.scss 2024-03-27T21:42:43.696 INFO:teuthology.orchestra.run.smithi060.stdout: 412540 4 drwxr-xr-x 12 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph 2024-03-27T21:42:43.696 INFO:teuthology.orchestra.run.smithi060.stdout: 413055 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3 2024-03-27T21:42:43.696 INFO:teuthology.orchestra.run.smithi060.stdout: 413078 4 -rw-r--r-- 1 ubuntu ubuntu 806 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/pg-summary.pipe.ts 2024-03-27T21:42:43.696 INFO:teuthology.orchestra.run.smithi060.stdout: 413072 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/dashboard 2024-03-27T21:42:43.696 INFO:teuthology.orchestra.run.smithi060.stdout: 413076 8 -rw-r--r-- 1 ubuntu ubuntu 6536 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/dashboard/dashboard-v3.component.ts 2024-03-27T21:42:43.696 INFO:teuthology.orchestra.run.smithi060.stdout: 413074 4 -rw-r--r-- 1 ubuntu ubuntu 523 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/dashboard/dashboard-v3.component.scss 2024-03-27T21:42:43.696 INFO:teuthology.orchestra.run.smithi060.stdout: 413073 16 -rw-r--r-- 1 ubuntu ubuntu 13734 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/dashboard/dashboard-v3.component.html 2024-03-27T21:42:43.696 INFO:teuthology.orchestra.run.smithi060.stdout: 413075 12 -rw-r--r-- 1 ubuntu ubuntu 11668 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/dashboard/dashboard-v3.component.spec.ts 2024-03-27T21:42:43.697 INFO:teuthology.orchestra.run.smithi060.stdout: 413066 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/dashboard-time-selector 2024-03-27T21:42:43.697 INFO:teuthology.orchestra.run.smithi060.stdout: 413067 4 -rw-r--r-- 1 ubuntu ubuntu 300 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/dashboard-time-selector/dashboard-time-selector.component.html 2024-03-27T21:42:43.697 INFO:teuthology.orchestra.run.smithi060.stdout: 413068 4 -rw-r--r-- 1 ubuntu ubuntu 84 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/dashboard-time-selector/dashboard-time-selector.component.scss 2024-03-27T21:42:43.697 INFO:teuthology.orchestra.run.smithi060.stdout: 413069 4 -rw-r--r-- 1 ubuntu ubuntu 764 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/dashboard-time-selector/dashboard-time-selector.component.spec.ts 2024-03-27T21:42:43.697 INFO:teuthology.orchestra.run.smithi060.stdout: 413070 4 -rw-r--r-- 1 ubuntu ubuntu 1698 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/dashboard-time-selector/dashboard-time-selector.component.ts 2024-03-27T21:42:43.697 INFO:teuthology.orchestra.run.smithi060.stdout: 413061 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/dashboard-pie 2024-03-27T21:42:43.697 INFO:teuthology.orchestra.run.smithi060.stdout: 413063 4 -rw-r--r-- 1 ubuntu ubuntu 402 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/dashboard-pie/dashboard-pie.component.scss 2024-03-27T21:42:43.697 INFO:teuthology.orchestra.run.smithi060.stdout: 413065 8 -rw-r--r-- 1 ubuntu ubuntu 6782 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/dashboard-pie/dashboard-pie.component.ts 2024-03-27T21:42:43.698 INFO:teuthology.orchestra.run.smithi060.stdout: 413062 4 -rw-r--r-- 1 ubuntu ubuntu 464 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/dashboard-pie/dashboard-pie.component.html 2024-03-27T21:42:43.698 INFO:teuthology.orchestra.run.smithi060.stdout: 413064 4 -rw-r--r-- 1 ubuntu ubuntu 884 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/dashboard-pie/dashboard-pie.component.spec.ts 2024-03-27T21:42:43.698 INFO:teuthology.orchestra.run.smithi060.stdout: 413071 4 -rw-r--r-- 1 ubuntu ubuntu 1547 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/dashboard-v3.module.ts 2024-03-27T21:42:43.698 INFO:teuthology.orchestra.run.smithi060.stdout: 413077 4 -rw-r--r-- 1 ubuntu ubuntu 786 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/pg-summary.pipe.spec.ts 2024-03-27T21:42:43.698 INFO:teuthology.orchestra.run.smithi060.stdout: 413056 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/dashboard-area-chart 2024-03-27T21:42:43.698 INFO:teuthology.orchestra.run.smithi060.stdout: 413058 4 -rw-r--r-- 1 ubuntu ubuntu 210 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/dashboard-area-chart/dashboard-area-chart.component.scss 2024-03-27T21:42:43.698 INFO:teuthology.orchestra.run.smithi060.stdout: 413060 12 -rw-r--r-- 1 ubuntu ubuntu 9908 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/dashboard-area-chart/dashboard-area-chart.component.ts 2024-03-27T21:42:43.698 INFO:teuthology.orchestra.run.smithi060.stdout: 413059 4 -rw-r--r-- 1 ubuntu ubuntu 3273 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/dashboard-area-chart/dashboard-area-chart.component.spec.ts 2024-03-27T21:42:43.699 INFO:teuthology.orchestra.run.smithi060.stdout: 413057 4 -rw-r--r-- 1 ubuntu ubuntu 1329 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard-v3/dashboard-area-chart/dashboard-area-chart.component.html 2024-03-27T21:42:43.699 INFO:teuthology.orchestra.run.smithi060.stdout: 412722 4 drwxr-xr-x 18 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs 2024-03-27T21:42:43.699 INFO:teuthology.orchestra.run.smithi060.stdout: 412768 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-snapshotschedule-list 2024-03-27T21:42:43.699 INFO:teuthology.orchestra.run.smithi060.stdout: 412772 12 -rw-r--r-- 1 ubuntu ubuntu 11014 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-snapshotschedule-list/cephfs-snapshotschedule-list.component.ts 2024-03-27T21:42:43.699 INFO:teuthology.orchestra.run.smithi060.stdout: 412770 4 -rw-r--r-- 1 ubuntu ubuntu 41 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-snapshotschedule-list/cephfs-snapshotschedule-list.component.scss 2024-03-27T21:42:43.699 INFO:teuthology.orchestra.run.smithi060.stdout: 412771 4 -rw-r--r-- 1 ubuntu ubuntu 1175 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-snapshotschedule-list/cephfs-snapshotschedule-list.component.spec.ts 2024-03-27T21:42:43.699 INFO:teuthology.orchestra.run.smithi060.stdout: 412769 4 -rw-r--r-- 1 ubuntu ubuntu 2382 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-snapshotschedule-list/cephfs-snapshotschedule-list.component.html 2024-03-27T21:42:43.700 INFO:teuthology.orchestra.run.smithi060.stdout: 412758 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-mount-details 2024-03-27T21:42:43.700 INFO:teuthology.orchestra.run.smithi060.stdout: 412761 4 -rw-r--r-- 1 ubuntu ubuntu 1127 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-mount-details/cephfs-mount-details.component.spec.ts 2024-03-27T21:42:43.700 INFO:teuthology.orchestra.run.smithi060.stdout: 412759 4 -rw-r--r-- 1 ubuntu ubuntu 919 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-mount-details/cephfs-mount-details.component.html 2024-03-27T21:42:43.700 INFO:teuthology.orchestra.run.smithi060.stdout: 412762 4 -rw-r--r-- 1 ubuntu ubuntu 1297 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-mount-details/cephfs-mount-details.component.ts 2024-03-27T21:42:43.700 INFO:teuthology.orchestra.run.smithi060.stdout: 412760 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-mount-details/cephfs-mount-details.component.scss 2024-03-27T21:42:43.700 INFO:teuthology.orchestra.run.smithi060.stdout: 412748 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-form 2024-03-27T21:42:43.700 INFO:teuthology.orchestra.run.smithi060.stdout: 412750 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-form/cephfs-form.component.scss 2024-03-27T21:42:43.700 INFO:teuthology.orchestra.run.smithi060.stdout: 412751 8 -rw-r--r-- 1 ubuntu ubuntu 4438 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-form/cephfs-form.component.spec.ts 2024-03-27T21:42:43.701 INFO:teuthology.orchestra.run.smithi060.stdout: 412749 8 -rw-r--r-- 1 ubuntu ubuntu 4814 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-form/cephfs-form.component.html 2024-03-27T21:42:43.701 INFO:teuthology.orchestra.run.smithi060.stdout: 412752 8 -rw-r--r-- 1 ubuntu ubuntu 7259 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-form/cephfs-form.component.ts 2024-03-27T21:42:43.701 INFO:teuthology.orchestra.run.smithi060.stdout: 412778 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-group 2024-03-27T21:42:43.701 INFO:teuthology.orchestra.run.smithi060.stdout: 412779 4 -rw-r--r-- 1 ubuntu ubuntu 1812 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-group/cephfs-subvolume-group.component.html 2024-03-27T21:42:43.701 INFO:teuthology.orchestra.run.smithi060.stdout: 412780 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-group/cephfs-subvolume-group.component.scss 2024-03-27T21:42:43.701 INFO:teuthology.orchestra.run.smithi060.stdout: 412782 8 -rw-r--r-- 1 ubuntu ubuntu 5618 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-group/cephfs-subvolume-group.component.ts 2024-03-27T21:42:43.701 INFO:teuthology.orchestra.run.smithi060.stdout: 412781 4 -rw-r--r-- 1 ubuntu ubuntu 1039 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-group/cephfs-subvolume-group.component.spec.ts 2024-03-27T21:42:43.701 INFO:teuthology.orchestra.run.smithi060.stdout: 412728 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-chart 2024-03-27T21:42:43.701 INFO:teuthology.orchestra.run.smithi060.stdout: 412730 4 -rw-r--r-- 1 ubuntu ubuntu 134 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-chart/cephfs-chart.component.scss 2024-03-27T21:42:43.702 INFO:teuthology.orchestra.run.smithi060.stdout: 412732 8 -rw-r--r-- 1 ubuntu ubuntu 4469 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-chart/cephfs-chart.component.ts 2024-03-27T21:42:43.702 INFO:teuthology.orchestra.run.smithi060.stdout: 412731 4 -rw-r--r-- 1 ubuntu ubuntu 2009 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-chart/cephfs-chart.component.spec.ts 2024-03-27T21:42:43.702 INFO:teuthology.orchestra.run.smithi060.stdout: 412729 4 -rw-r--r-- 1 ubuntu ubuntu 284 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-chart/cephfs-chart.component.html 2024-03-27T21:42:43.702 INFO:teuthology.orchestra.run.smithi060.stdout: 412738 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-detail 2024-03-27T21:42:43.702 INFO:teuthology.orchestra.run.smithi060.stdout: 412739 4 -rw-r--r-- 1 ubuntu ubuntu 1172 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-detail/cephfs-detail.component.html 2024-03-27T21:42:43.702 INFO:teuthology.orchestra.run.smithi060.stdout: 412742 4 -rw-r--r-- 1 ubuntu ubuntu 2569 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-detail/cephfs-detail.component.ts 2024-03-27T21:42:43.702 INFO:teuthology.orchestra.run.smithi060.stdout: 412740 4 -rw-r--r-- 1 ubuntu ubuntu 34 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-detail/cephfs-detail.component.scss 2024-03-27T21:42:43.702 INFO:teuthology.orchestra.run.smithi060.stdout: 412741 4 -rw-r--r-- 1 ubuntu ubuntu 1450 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-detail/cephfs-detail.component.spec.ts 2024-03-27T21:42:43.703 INFO:teuthology.orchestra.run.smithi060.stdout: 412773 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-form 2024-03-27T21:42:43.703 INFO:teuthology.orchestra.run.smithi060.stdout: 412777 8 -rw-r--r-- 1 ubuntu ubuntu 7556 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-form/cephfs-subvolume-form.component.ts 2024-03-27T21:42:43.703 INFO:teuthology.orchestra.run.smithi060.stdout: 412774 8 -rw-r--r-- 1 ubuntu ubuntu 7206 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-form/cephfs-subvolume-form.component.html 2024-03-27T21:42:43.703 INFO:teuthology.orchestra.run.smithi060.stdout: 412775 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-form/cephfs-subvolume-form.component.scss 2024-03-27T21:42:43.703 INFO:teuthology.orchestra.run.smithi060.stdout: 412776 4 -rw-r--r-- 1 ubuntu ubuntu 2717 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-form/cephfs-subvolume-form.component.spec.ts 2024-03-27T21:42:43.703 INFO:teuthology.orchestra.run.smithi060.stdout: 412798 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolumegroup-form 2024-03-27T21:42:43.703 INFO:teuthology.orchestra.run.smithi060.stdout: 412801 4 -rw-r--r-- 1 ubuntu ubuntu 1298 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolumegroup-form/cephfs-subvolumegroup-form.component.spec.ts 2024-03-27T21:42:43.703 INFO:teuthology.orchestra.run.smithi060.stdout: 412802 8 -rw-r--r-- 1 ubuntu ubuntu 6377 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolumegroup-form/cephfs-subvolumegroup-form.component.ts 2024-03-27T21:42:43.704 INFO:teuthology.orchestra.run.smithi060.stdout: 412800 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolumegroup-form/cephfs-subvolumegroup-form.component.scss 2024-03-27T21:42:43.704 INFO:teuthology.orchestra.run.smithi060.stdout: 412799 8 -rw-r--r-- 1 ubuntu ubuntu 5693 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolumegroup-form/cephfs-subvolumegroup-form.component.html 2024-03-27T21:42:43.704 INFO:teuthology.orchestra.run.smithi060.stdout: 412733 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-clients 2024-03-27T21:42:43.704 INFO:teuthology.orchestra.run.smithi060.stdout: 412736 4 -rw-r--r-- 1 ubuntu ubuntu 2859 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-clients/cephfs-clients.component.spec.ts 2024-03-27T21:42:43.704 INFO:teuthology.orchestra.run.smithi060.stdout: 412734 4 -rw-r--r-- 1 ubuntu ubuntu 479 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-clients/cephfs-clients.component.html 2024-03-27T21:42:43.704 INFO:teuthology.orchestra.run.smithi060.stdout: 412737 4 -rw-r--r-- 1 ubuntu ubuntu 3374 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-clients/cephfs-clients.component.ts 2024-03-27T21:42:43.704 INFO:teuthology.orchestra.run.smithi060.stdout: 412735 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-clients/cephfs-clients.component.scss 2024-03-27T21:42:43.705 INFO:teuthology.orchestra.run.smithi060.stdout: 412763 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-snapshotschedule-form 2024-03-27T21:42:43.705 INFO:teuthology.orchestra.run.smithi060.stdout: 412767 16 -rw-r--r-- 1 ubuntu ubuntu 15511 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-snapshotschedule-form/cephfs-snapshotschedule-form.component.ts 2024-03-27T21:42:43.705 INFO:teuthology.orchestra.run.smithi060.stdout: 412766 4 -rw-r--r-- 1 ubuntu ubuntu 2440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-snapshotschedule-form/cephfs-snapshotschedule-form.component.spec.ts 2024-03-27T21:42:43.705 INFO:teuthology.orchestra.run.smithi060.stdout: 412765 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-snapshotschedule-form/cephfs-snapshotschedule-form.component.scss 2024-03-27T21:42:43.705 INFO:teuthology.orchestra.run.smithi060.stdout: 412764 8 -rw-r--r-- 1 ubuntu ubuntu 7828 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-snapshotschedule-form/cephfs-snapshotschedule-form.component.html 2024-03-27T21:42:43.705 INFO:teuthology.orchestra.run.smithi060.stdout: 412803 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-tabs 2024-03-27T21:42:43.705 INFO:teuthology.orchestra.run.smithi060.stdout: 412806 8 -rw-r--r-- 1 ubuntu ubuntu 6457 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-tabs/cephfs-tabs.component.spec.ts 2024-03-27T21:42:43.705 INFO:teuthology.orchestra.run.smithi060.stdout: 412805 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-tabs/cephfs-tabs.component.scss 2024-03-27T21:42:43.705 INFO:teuthology.orchestra.run.smithi060.stdout: 412804 4 -rw-r--r-- 1 ubuntu ubuntu 2879 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-tabs/cephfs-tabs.component.html 2024-03-27T21:42:43.706 INFO:teuthology.orchestra.run.smithi060.stdout: 412807 4 -rw-r--r-- 1 ubuntu ubuntu 3252 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-tabs/cephfs-tabs.component.ts 2024-03-27T21:42:43.706 INFO:teuthology.orchestra.run.smithi060.stdout: 412743 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-directories 2024-03-27T21:42:43.706 INFO:teuthology.orchestra.run.smithi060.stdout: 412745 4 -rw-r--r-- 1 ubuntu ubuntu 298 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-directories/cephfs-directories.component.scss 2024-03-27T21:42:43.706 INFO:teuthology.orchestra.run.smithi060.stdout: 412746 40 -rw-r--r-- 1 ubuntu ubuntu 38629 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-directories/cephfs-directories.component.spec.ts 2024-03-27T21:42:43.706 INFO:teuthology.orchestra.run.smithi060.stdout: 412747 24 -rw-r--r-- 1 ubuntu ubuntu 23828 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-directories/cephfs-directories.component.ts 2024-03-27T21:42:43.706 INFO:teuthology.orchestra.run.smithi060.stdout: 412744 4 -rw-r--r-- 1 ubuntu ubuntu 2662 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-directories/cephfs-directories.component.html 2024-03-27T21:42:43.706 INFO:teuthology.orchestra.run.smithi060.stdout: 412788 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-snapshots-list 2024-03-27T21:42:43.706 INFO:teuthology.orchestra.run.smithi060.stdout: 412795 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-snapshots-list/cephfs-subvolume-snapshots-list.component.scss 2024-03-27T21:42:43.707 INFO:teuthology.orchestra.run.smithi060.stdout: 412794 4 -rw-r--r-- 1 ubuntu ubuntu 1802 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-snapshots-list/cephfs-subvolume-snapshots-list.component.html 2024-03-27T21:42:43.707 INFO:teuthology.orchestra.run.smithi060.stdout: 412797 12 -rw-r--r-- 1 ubuntu ubuntu 10184 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-snapshots-list/cephfs-subvolume-snapshots-list.component.ts 2024-03-27T21:42:43.707 INFO:teuthology.orchestra.run.smithi060.stdout: 412789 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-snapshots-list/cephfs-subvolume-snapshots-form 2024-03-27T21:42:43.707 INFO:teuthology.orchestra.run.smithi060.stdout: 412791 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-snapshots-list/cephfs-subvolume-snapshots-form/cephfs-subvolume-snapshots-form.component.scss 2024-03-27T21:42:43.707 INFO:teuthology.orchestra.run.smithi060.stdout: 412793 8 -rw-r--r-- 1 ubuntu ubuntu 4224 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-snapshots-list/cephfs-subvolume-snapshots-form/cephfs-subvolume-snapshots-form.component.ts 2024-03-27T21:42:43.707 INFO:teuthology.orchestra.run.smithi060.stdout: 412792 4 -rw-r--r-- 1 ubuntu ubuntu 1472 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-snapshots-list/cephfs-subvolume-snapshots-form/cephfs-subvolume-snapshots-form.component.spec.ts 2024-03-27T21:42:43.707 INFO:teuthology.orchestra.run.smithi060.stdout: 412790 4 -rw-r--r-- 1 ubuntu ubuntu 3791 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-snapshots-list/cephfs-subvolume-snapshots-form/cephfs-subvolume-snapshots-form.component.html 2024-03-27T21:42:43.707 INFO:teuthology.orchestra.run.smithi060.stdout: 412796 4 -rw-r--r-- 1 ubuntu ubuntu 1477 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-snapshots-list/cephfs-subvolume-snapshots-list.component.spec.ts 2024-03-27T21:42:43.708 INFO:teuthology.orchestra.run.smithi060.stdout: 412783 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-list 2024-03-27T21:42:43.708 INFO:teuthology.orchestra.run.smithi060.stdout: 412787 12 -rw-r--r-- 1 ubuntu ubuntu 8775 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-list/cephfs-subvolume-list.component.ts 2024-03-27T21:42:43.708 INFO:teuthology.orchestra.run.smithi060.stdout: 412786 4 -rw-r--r-- 1 ubuntu ubuntu 1126 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-list/cephfs-subvolume-list.component.spec.ts 2024-03-27T21:42:43.708 INFO:teuthology.orchestra.run.smithi060.stdout: 412785 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-list/cephfs-subvolume-list.component.scss 2024-03-27T21:42:43.708 INFO:teuthology.orchestra.run.smithi060.stdout: 412784 4 -rw-r--r-- 1 ubuntu ubuntu 3983 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-list/cephfs-subvolume-list.component.html 2024-03-27T21:42:43.708 INFO:teuthology.orchestra.run.smithi060.stdout: 412753 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-list 2024-03-27T21:42:43.708 INFO:teuthology.orchestra.run.smithi060.stdout: 412754 4 -rw-r--r-- 1 ubuntu ubuntu 787 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-list/cephfs-list.component.html 2024-03-27T21:42:43.708 INFO:teuthology.orchestra.run.smithi060.stdout: 412756 4 -rw-r--r-- 1 ubuntu ubuntu 3415 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-list/cephfs-list.component.spec.ts 2024-03-27T21:42:43.709 INFO:teuthology.orchestra.run.smithi060.stdout: 412757 8 -rw-r--r-- 1 ubuntu ubuntu 7022 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-list/cephfs-list.component.ts 2024-03-27T21:42:43.709 INFO:teuthology.orchestra.run.smithi060.stdout: 412755 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-list/cephfs-list.component.scss 2024-03-27T21:42:43.709 INFO:teuthology.orchestra.run.smithi060.stdout: 412723 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-auth-modal 2024-03-27T21:42:43.709 INFO:teuthology.orchestra.run.smithi060.stdout: 412726 4 -rw-r--r-- 1 ubuntu ubuntu 1082 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-auth-modal/cephfs-auth-modal.component.spec.ts 2024-03-27T21:42:43.709 INFO:teuthology.orchestra.run.smithi060.stdout: 412725 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-auth-modal/cephfs-auth-modal.component.scss 2024-03-27T21:42:43.709 INFO:teuthology.orchestra.run.smithi060.stdout: 412724 8 -rw-r--r-- 1 ubuntu ubuntu 6169 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-auth-modal/cephfs-auth-modal.component.html 2024-03-27T21:42:43.709 INFO:teuthology.orchestra.run.smithi060.stdout: 412727 8 -rw-r--r-- 1 ubuntu ubuntu 4128 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-auth-modal/cephfs-auth-modal.component.ts 2024-03-27T21:42:43.710 INFO:teuthology.orchestra.run.smithi060.stdout: 412808 4 -rw-r--r-- 1 ubuntu ubuntu 3332 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs.module.ts 2024-03-27T21:42:43.710 INFO:teuthology.orchestra.run.smithi060.stdout: 413079 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard 2024-03-27T21:42:43.710 INFO:teuthology.orchestra.run.smithi060.stdout: 413112 4 -rw-r--r-- 1 ubuntu ubuntu 348 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/mon-summary.pipe.ts 2024-03-27T21:42:43.710 INFO:teuthology.orchestra.run.smithi060.stdout: 413081 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/dashboard 2024-03-27T21:42:43.710 INFO:teuthology.orchestra.run.smithi060.stdout: 413085 4 -rw-r--r-- 1 ubuntu ubuntu 496 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/dashboard/dashboard.component.ts 2024-03-27T21:42:43.710 INFO:teuthology.orchestra.run.smithi060.stdout: 413082 4 -rw-r--r-- 1 ubuntu ubuntu 417 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/dashboard/dashboard.component.html 2024-03-27T21:42:43.710 INFO:teuthology.orchestra.run.smithi060.stdout: 413084 4 -rw-r--r-- 1 ubuntu ubuntu 1020 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/dashboard/dashboard.component.spec.ts 2024-03-27T21:42:43.710 INFO:teuthology.orchestra.run.smithi060.stdout: 413083 4 -rw-r--r-- 1 ubuntu ubuntu 45 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/dashboard/dashboard.component.scss 2024-03-27T21:42:43.711 INFO:teuthology.orchestra.run.smithi060.stdout: 413086 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/health-pie 2024-03-27T21:42:43.711 INFO:teuthology.orchestra.run.smithi060.stdout: 413089 4 -rw-r--r-- 1 ubuntu ubuntu 2695 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/health-pie/health-pie.component.spec.ts 2024-03-27T21:42:43.711 INFO:teuthology.orchestra.run.smithi060.stdout: 413087 4 -rw-r--r-- 1 ubuntu ubuntu 395 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/health-pie/health-pie.component.html 2024-03-27T21:42:43.711 INFO:teuthology.orchestra.run.smithi060.stdout: 413088 4 -rw-r--r-- 1 ubuntu ubuntu 402 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/health-pie/health-pie.component.scss 2024-03-27T21:42:43.711 INFO:teuthology.orchestra.run.smithi060.stdout: 413090 8 -rw-r--r-- 1 ubuntu ubuntu 5416 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/health-pie/health-pie.component.ts 2024-03-27T21:42:43.711 INFO:teuthology.orchestra.run.smithi060.stdout: 413108 4 -rw-r--r-- 1 ubuntu ubuntu 2297 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/mds-summary.pipe.ts 2024-03-27T21:42:43.711 INFO:teuthology.orchestra.run.smithi060.stdout: 413113 8 -rw-r--r-- 1 ubuntu ubuntu 4206 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/osd-summary.pipe.spec.ts 2024-03-27T21:42:43.711 INFO:teuthology.orchestra.run.smithi060.stdout: 413110 4 -rw-r--r-- 1 ubuntu ubuntu 1280 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/mgr-summary.pipe.ts 2024-03-27T21:42:43.712 INFO:teuthology.orchestra.run.smithi060.stdout: 413107 4 -rw-r--r-- 1 ubuntu ubuntu 2297 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/mds-summary.pipe.spec.ts 2024-03-27T21:42:43.712 INFO:teuthology.orchestra.run.smithi060.stdout: 413080 4 -rw-r--r-- 1 ubuntu ubuntu 1664 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/dashboard.module.ts 2024-03-27T21:42:43.712 INFO:teuthology.orchestra.run.smithi060.stdout: 413102 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/info-group 2024-03-27T21:42:43.712 INFO:teuthology.orchestra.run.smithi060.stdout: 413103 4 -rw-r--r-- 1 ubuntu ubuntu 491 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/info-group/info-group.component.html 2024-03-27T21:42:43.712 INFO:teuthology.orchestra.run.smithi060.stdout: 413106 4 -rw-r--r-- 1 ubuntu ubuntu 328 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/info-group/info-group.component.ts 2024-03-27T21:42:43.712 INFO:teuthology.orchestra.run.smithi060.stdout: 413104 4 -rw-r--r-- 1 ubuntu ubuntu 196 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/info-group/info-group.component.scss 2024-03-27T21:42:43.712 INFO:teuthology.orchestra.run.smithi060.stdout: 413105 4 -rw-r--r-- 1 ubuntu ubuntu 1180 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/info-group/info-group.component.spec.ts 2024-03-27T21:42:43.712 INFO:teuthology.orchestra.run.smithi060.stdout: 413114 4 -rw-r--r-- 1 ubuntu ubuntu 2067 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/osd-summary.pipe.ts 2024-03-27T21:42:43.713 INFO:teuthology.orchestra.run.smithi060.stdout: 413096 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/info-card 2024-03-27T21:42:43.713 INFO:teuthology.orchestra.run.smithi060.stdout: 413100 4 -rw-r--r-- 1 ubuntu ubuntu 2036 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/info-card/info-card.component.spec.ts 2024-03-27T21:42:43.713 INFO:teuthology.orchestra.run.smithi060.stdout: 413098 4 -rw-r--r-- 1 ubuntu ubuntu 475 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/info-card/info-card.component.html 2024-03-27T21:42:43.713 INFO:teuthology.orchestra.run.smithi060.stdout: 413099 4 -rw-r--r-- 1 ubuntu ubuntu 751 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/info-card/info-card.component.scss 2024-03-27T21:42:43.713 INFO:teuthology.orchestra.run.smithi060.stdout: 413101 4 -rw-r--r-- 1 ubuntu ubuntu 342 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/info-card/info-card.component.ts 2024-03-27T21:42:43.713 INFO:teuthology.orchestra.run.smithi060.stdout: 413097 4 -rw-r--r-- 1 ubuntu ubuntu 988 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/info-card/info-card-popover.scss 2024-03-27T21:42:43.713 INFO:teuthology.orchestra.run.smithi060.stdout: 413109 4 -rw-r--r-- 1 ubuntu ubuntu 1482 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/mgr-summary.pipe.spec.ts 2024-03-27T21:42:43.713 INFO:teuthology.orchestra.run.smithi060.stdout: 413091 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/health 2024-03-27T21:42:43.713 INFO:teuthology.orchestra.run.smithi060.stdout: 413094 12 -rw-r--r-- 1 ubuntu ubuntu 11713 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/health/health.component.spec.ts 2024-03-27T21:42:43.714 INFO:teuthology.orchestra.run.smithi060.stdout: 413093 4 -rw-r--r-- 1 ubuntu ubuntu 656 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/health/health.component.scss 2024-03-27T21:42:43.714 INFO:teuthology.orchestra.run.smithi060.stdout: 413095 12 -rw-r--r-- 1 ubuntu ubuntu 9308 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/health/health.component.ts 2024-03-27T21:42:43.714 INFO:teuthology.orchestra.run.smithi060.stdout: 413092 12 -rw-r--r-- 1 ubuntu ubuntu 9536 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/health/health.component.html 2024-03-27T21:42:43.714 INFO:teuthology.orchestra.run.smithi060.stdout: 413111 4 -rw-r--r-- 1 ubuntu ubuntu 938 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/mon-summary.pipe.spec.ts 2024-03-27T21:42:43.714 INFO:teuthology.orchestra.run.smithi060.stdout: 412541 4 drwxr-xr-x 27 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block 2024-03-27T21:42:43.714 INFO:teuthology.orchestra.run.smithi060.stdout: 412568 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-image-settings-modal 2024-03-27T21:42:43.715 INFO:teuthology.orchestra.run.smithi060.stdout: 412571 4 -rw-r--r-- 1 ubuntu ubuntu 2806 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-image-settings-modal/iscsi-target-image-settings-modal.component.spec.ts 2024-03-27T21:42:43.715 INFO:teuthology.orchestra.run.smithi060.stdout: 412572 4 -rw-r--r-- 1 ubuntu ubuntu 3194 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-image-settings-modal/iscsi-target-image-settings-modal.component.ts 2024-03-27T21:42:43.715 INFO:teuthology.orchestra.run.smithi060.stdout: 412570 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-image-settings-modal/iscsi-target-image-settings-modal.component.scss 2024-03-27T21:42:43.715 INFO:teuthology.orchestra.run.smithi060.stdout: 412569 4 -rw-r--r-- 1 ubuntu ubuntu 3275 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-image-settings-modal/iscsi-target-image-settings-modal.component.html 2024-03-27T21:42:43.715 INFO:teuthology.orchestra.run.smithi060.stdout: 412701 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-list 2024-03-27T21:42:43.715 INFO:teuthology.orchestra.run.smithi060.stdout: 412702 4 -rw-r--r-- 1 ubuntu ubuntu 1558 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-list/rbd-trash-list.component.html 2024-03-27T21:42:43.715 INFO:teuthology.orchestra.run.smithi060.stdout: 412704 8 -rw-r--r-- 1 ubuntu ubuntu 5541 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-list/rbd-trash-list.component.spec.ts 2024-03-27T21:42:43.715 INFO:teuthology.orchestra.run.smithi060.stdout: 412705 8 -rw-r--r-- 1 ubuntu ubuntu 7443 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-list/rbd-trash-list.component.ts 2024-03-27T21:42:43.716 INFO:teuthology.orchestra.run.smithi060.stdout: 412703 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-list/rbd-trash-list.component.scss 2024-03-27T21:42:43.716 INFO:teuthology.orchestra.run.smithi060.stdout: 412644 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-details 2024-03-27T21:42:43.716 INFO:teuthology.orchestra.run.smithi060.stdout: 412647 4 -rw-r--r-- 1 ubuntu ubuntu 1172 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-details/rbd-details.component.spec.ts 2024-03-27T21:42:43.716 INFO:teuthology.orchestra.run.smithi060.stdout: 412646 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-details/rbd-details.component.scss 2024-03-27T21:42:43.716 INFO:teuthology.orchestra.run.smithi060.stdout: 412648 4 -rw-r--r-- 1 ubuntu ubuntu 829 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-details/rbd-details.component.ts 2024-03-27T21:42:43.716 INFO:teuthology.orchestra.run.smithi060.stdout: 412645 8 -rw-r--r-- 1 ubuntu ubuntu 6689 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-details/rbd-details.component.html 2024-03-27T21:42:43.716 INFO:teuthology.orchestra.run.smithi060.stdout: 412543 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-setting 2024-03-27T21:42:43.716 INFO:teuthology.orchestra.run.smithi060.stdout: 412547 4 -rw-r--r-- 1 ubuntu ubuntu 860 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-setting/iscsi-setting.component.ts 2024-03-27T21:42:43.717 INFO:teuthology.orchestra.run.smithi060.stdout: 412544 4 -rw-r--r-- 1 ubuntu ubuntu 2016 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-setting/iscsi-setting.component.html 2024-03-27T21:42:43.717 INFO:teuthology.orchestra.run.smithi060.stdout: 412545 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-setting/iscsi-setting.component.scss 2024-03-27T21:42:43.717 INFO:teuthology.orchestra.run.smithi060.stdout: 412546 4 -rw-r--r-- 1 ubuntu ubuntu 1160 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-setting/iscsi-setting.component.spec.ts 2024-03-27T21:42:43.717 INFO:teuthology.orchestra.run.smithi060.stdout: 412583 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi 2024-03-27T21:42:43.717 INFO:teuthology.orchestra.run.smithi060.stdout: 412585 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi/iscsi.component.scss 2024-03-27T21:42:43.717 INFO:teuthology.orchestra.run.smithi060.stdout: 412584 4 -rw-r--r-- 1 ubuntu ubuntu 1247 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi/iscsi.component.html 2024-03-27T21:42:43.717 INFO:teuthology.orchestra.run.smithi060.stdout: 412586 4 -rw-r--r-- 1 ubuntu ubuntu 2519 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi/iscsi.component.spec.ts 2024-03-27T21:42:43.717 INFO:teuthology.orchestra.run.smithi060.stdout: 412587 4 -rw-r--r-- 1 ubuntu ubuntu 3157 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi/iscsi.component.ts 2024-03-27T21:42:43.718 INFO:teuthology.orchestra.run.smithi060.stdout: 412558 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-discovery-modal 2024-03-27T21:42:43.718 INFO:teuthology.orchestra.run.smithi060.stdout: 412560 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-discovery-modal/iscsi-target-discovery-modal.component.scss 2024-03-27T21:42:43.718 INFO:teuthology.orchestra.run.smithi060.stdout: 412562 8 -rw-r--r-- 1 ubuntu ubuntu 4196 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-discovery-modal/iscsi-target-discovery-modal.component.ts 2024-03-27T21:42:43.718 INFO:teuthology.orchestra.run.smithi060.stdout: 412559 8 -rw-r--r-- 1 ubuntu ubuntu 5101 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-discovery-modal/iscsi-target-discovery-modal.component.html 2024-03-27T21:42:43.718 INFO:teuthology.orchestra.run.smithi060.stdout: 412561 8 -rw-r--r-- 1 ubuntu ubuntu 4545 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-discovery-modal/iscsi-target-discovery-modal.component.spec.ts 2024-03-27T21:42:43.718 INFO:teuthology.orchestra.run.smithi060.stdout: 412649 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-form 2024-03-27T21:42:43.718 INFO:teuthology.orchestra.run.smithi060.stdout: 412653 4 -rw-r--r-- 1 ubuntu ubuntu 146 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-form/rbd-form-create-request.model.ts 2024-03-27T21:42:43.718 INFO:teuthology.orchestra.run.smithi060.stdout: 412654 4 -rw-r--r-- 1 ubuntu ubuntu 374 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-form/rbd-form-edit-request.model.ts 2024-03-27T21:42:43.719 INFO:teuthology.orchestra.run.smithi060.stdout: 412651 4 -rw-r--r-- 1 ubuntu ubuntu 363 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-form/rbd-form-clone-request.model.ts 2024-03-27T21:42:43.719 INFO:teuthology.orchestra.run.smithi060.stdout: 412657 20 -rw-r--r-- 1 ubuntu ubuntu 18092 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-form/rbd-form.component.html 2024-03-27T21:42:43.719 INFO:teuthology.orchestra.run.smithi060.stdout: 412652 4 -rw-r--r-- 1 ubuntu ubuntu 383 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-form/rbd-form-copy-request.model.ts 2024-03-27T21:42:43.719 INFO:teuthology.orchestra.run.smithi060.stdout: 412659 20 -rw-r--r-- 1 ubuntu ubuntu 17938 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-form/rbd-form.component.spec.ts 2024-03-27T21:42:43.719 INFO:teuthology.orchestra.run.smithi060.stdout: 412656 4 -rw-r--r-- 1 ubuntu ubuntu 215 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-form/rbd-form-response.model.ts 2024-03-27T21:42:43.719 INFO:teuthology.orchestra.run.smithi060.stdout: 412650 4 -rw-r--r-- 1 ubuntu ubuntu 216 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-form/rbd-feature.interface.ts 2024-03-27T21:42:43.719 INFO:teuthology.orchestra.run.smithi060.stdout: 412660 32 -rw-r--r-- 1 ubuntu ubuntu 29143 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-form/rbd-form.component.ts 2024-03-27T21:42:43.720 INFO:teuthology.orchestra.run.smithi060.stdout: 412658 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-form/rbd-form.component.scss 2024-03-27T21:42:43.720 INFO:teuthology.orchestra.run.smithi060.stdout: 412655 4 -rw-r--r-- 1 ubuntu ubuntu 96 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-form/rbd-form-mode.enum.ts 2024-03-27T21:42:43.720 INFO:teuthology.orchestra.run.smithi060.stdout: 412662 4 -rw-r--r-- 1 ubuntu ubuntu 122 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-form/rbd-parent.model.ts 2024-03-27T21:42:43.720 INFO:teuthology.orchestra.run.smithi060.stdout: 412661 4 -rw-r--r-- 1 ubuntu ubuntu 500 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-form/rbd-form.model.ts 2024-03-27T21:42:43.720 INFO:teuthology.orchestra.run.smithi060.stdout: 412553 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-details 2024-03-27T21:42:43.720 INFO:teuthology.orchestra.run.smithi060.stdout: 412554 4 -rw-r--r-- 1 ubuntu ubuntu 1248 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-details/iscsi-target-details.component.html 2024-03-27T21:42:43.720 INFO:teuthology.orchestra.run.smithi060.stdout: 412556 8 -rw-r--r-- 1 ubuntu ubuntu 6432 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-details/iscsi-target-details.component.spec.ts 2024-03-27T21:42:43.720 INFO:teuthology.orchestra.run.smithi060.stdout: 412557 12 -rw-r--r-- 1 ubuntu ubuntu 9658 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-details/iscsi-target-details.component.ts 2024-03-27T21:42:43.720 INFO:teuthology.orchestra.run.smithi060.stdout: 412555 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-details/iscsi-target-details.component.scss 2024-03-27T21:42:43.721 INFO:teuthology.orchestra.run.smithi060.stdout: 412548 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-tabs 2024-03-27T21:42:43.721 INFO:teuthology.orchestra.run.smithi060.stdout: 412549 4 -rw-r--r-- 1 ubuntu ubuntu 414 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-tabs/iscsi-tabs.component.html 2024-03-27T21:42:43.721 INFO:teuthology.orchestra.run.smithi060.stdout: 412550 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-tabs/iscsi-tabs.component.scss 2024-03-27T21:42:43.721 INFO:teuthology.orchestra.run.smithi060.stdout: 412551 4 -rw-r--r-- 1 ubuntu ubuntu 871 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-tabs/iscsi-tabs.component.spec.ts 2024-03-27T21:42:43.721 INFO:teuthology.orchestra.run.smithi060.stdout: 412552 4 -rw-r--r-- 1 ubuntu ubuntu 215 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-tabs/iscsi-tabs.component.ts 2024-03-27T21:42:43.721 INFO:teuthology.orchestra.run.smithi060.stdout: 412679 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-performance 2024-03-27T21:42:43.721 INFO:teuthology.orchestra.run.smithi060.stdout: 412681 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-performance/rbd-performance.component.scss 2024-03-27T21:42:43.721 INFO:teuthology.orchestra.run.smithi060.stdout: 412683 4 -rw-r--r-- 1 ubuntu ubuntu 235 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-performance/rbd-performance.component.ts 2024-03-27T21:42:43.722 INFO:teuthology.orchestra.run.smithi060.stdout: 412682 4 -rw-r--r-- 1 ubuntu ubuntu 1088 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-performance/rbd-performance.component.spec.ts 2024-03-27T21:42:43.722 INFO:teuthology.orchestra.run.smithi060.stdout: 412680 4 -rw-r--r-- 1 ubuntu ubuntu 234 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-performance/rbd-performance.component.html 2024-03-27T21:42:43.722 INFO:teuthology.orchestra.run.smithi060.stdout: 412716 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-restore-modal 2024-03-27T21:42:43.722 INFO:teuthology.orchestra.run.smithi060.stdout: 412719 4 -rw-r--r-- 1 ubuntu ubuntu 2646 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-restore-modal/rbd-trash-restore-modal.component.spec.ts 2024-03-27T21:42:43.722 INFO:teuthology.orchestra.run.smithi060.stdout: 412720 4 -rw-r--r-- 1 ubuntu ubuntu 2080 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-restore-modal/rbd-trash-restore-modal.component.ts 2024-03-27T21:42:43.722 INFO:teuthology.orchestra.run.smithi060.stdout: 412717 4 -rw-r--r-- 1 ubuntu ubuntu 1364 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-restore-modal/rbd-trash-restore-modal.component.html 2024-03-27T21:42:43.722 INFO:teuthology.orchestra.run.smithi060.stdout: 412718 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-restore-modal/rbd-trash-restore-modal.component.scss 2024-03-27T21:42:43.722 INFO:teuthology.orchestra.run.smithi060.stdout: 412639 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-configuration-list 2024-03-27T21:42:43.723 INFO:teuthology.orchestra.run.smithi060.stdout: 412643 4 -rw-r--r-- 1 ubuntu ubuntu 2288 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-configuration-list/rbd-configuration-list.component.ts 2024-03-27T21:42:43.723 INFO:teuthology.orchestra.run.smithi060.stdout: 412640 4 -rw-r--r-- 1 ubuntu ubuntu 887 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-configuration-list/rbd-configuration-list.component.html 2024-03-27T21:42:43.723 INFO:teuthology.orchestra.run.smithi060.stdout: 412642 4 -rw-r--r-- 1 ubuntu ubuntu 3217 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-configuration-list/rbd-configuration-list.component.spec.ts 2024-03-27T21:42:43.723 INFO:teuthology.orchestra.run.smithi060.stdout: 412641 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-configuration-list/rbd-configuration-list.component.scss 2024-03-27T21:42:43.723 INFO:teuthology.orchestra.run.smithi060.stdout: 412711 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-purge-modal 2024-03-27T21:42:43.723 INFO:teuthology.orchestra.run.smithi060.stdout: 412713 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-purge-modal/rbd-trash-purge-modal.component.scss 2024-03-27T21:42:43.723 INFO:teuthology.orchestra.run.smithi060.stdout: 412712 4 -rw-r--r-- 1 ubuntu ubuntu 1525 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-purge-modal/rbd-trash-purge-modal.component.html 2024-03-27T21:42:43.723 INFO:teuthology.orchestra.run.smithi060.stdout: 412715 4 -rw-r--r-- 1 ubuntu ubuntu 2360 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-purge-modal/rbd-trash-purge-modal.component.ts 2024-03-27T21:42:43.723 INFO:teuthology.orchestra.run.smithi060.stdout: 412714 4 -rw-r--r-- 1 ubuntu ubuntu 3364 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-purge-modal/rbd-trash-purge-modal.component.spec.ts 2024-03-27T21:42:43.724 INFO:teuthology.orchestra.run.smithi060.stdout: 412674 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-namespace-list 2024-03-27T21:42:43.724 INFO:teuthology.orchestra.run.smithi060.stdout: 412677 4 -rw-r--r-- 1 ubuntu ubuntu 1426 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-namespace-list/rbd-namespace-list.component.spec.ts 2024-03-27T21:42:43.724 INFO:teuthology.orchestra.run.smithi060.stdout: 412675 4 -rw-r--r-- 1 ubuntu ubuntu 572 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-namespace-list/rbd-namespace-list.component.html 2024-03-27T21:42:43.724 INFO:teuthology.orchestra.run.smithi060.stdout: 412678 8 -rw-r--r-- 1 ubuntu ubuntu 5279 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-namespace-list/rbd-namespace-list.component.ts 2024-03-27T21:42:43.724 INFO:teuthology.orchestra.run.smithi060.stdout: 412676 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-namespace-list/rbd-namespace-list.component.scss 2024-03-27T21:42:43.724 INFO:teuthology.orchestra.run.smithi060.stdout: 412696 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-tabs 2024-03-27T21:42:43.725 INFO:teuthology.orchestra.run.smithi060.stdout: 412697 4 -rw-r--r-- 1 ubuntu ubuntu 1026 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-tabs/rbd-tabs.component.html 2024-03-27T21:42:43.725 INFO:teuthology.orchestra.run.smithi060.stdout: 412698 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-tabs/rbd-tabs.component.scss 2024-03-27T21:42:43.725 INFO:teuthology.orchestra.run.smithi060.stdout: 412699 4 -rw-r--r-- 1 ubuntu ubuntu 784 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-tabs/rbd-tabs.component.spec.ts 2024-03-27T21:42:43.725 INFO:teuthology.orchestra.run.smithi060.stdout: 412700 4 -rw-r--r-- 1 ubuntu ubuntu 548 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-tabs/rbd-tabs.component.ts 2024-03-27T21:42:43.725 INFO:teuthology.orchestra.run.smithi060.stdout: 412588 4 drwxr-xr-x 10 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring 2024-03-27T21:42:43.725 INFO:teuthology.orchestra.run.smithi060.stdout: 412589 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/bootstrap-create-modal 2024-03-27T21:42:43.725 INFO:teuthology.orchestra.run.smithi060.stdout: 412591 4 -rw-r--r-- 1 ubuntu ubuntu 63 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/bootstrap-create-modal/bootstrap-create-modal.component.scss 2024-03-27T21:42:43.725 INFO:teuthology.orchestra.run.smithi060.stdout: 412593 8 -rw-r--r-- 1 ubuntu ubuntu 4709 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/bootstrap-create-modal/bootstrap-create-modal.component.ts 2024-03-27T21:42:43.725 INFO:teuthology.orchestra.run.smithi060.stdout: 412590 4 -rwxr-xr-x 1 ubuntu ubuntu 3171 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/bootstrap-create-modal/bootstrap-create-modal.component.html 2024-03-27T21:42:43.726 INFO:teuthology.orchestra.run.smithi060.stdout: 412592 4 -rw-r--r-- 1 ubuntu ubuntu 4091 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/bootstrap-create-modal/bootstrap-create-modal.component.spec.ts 2024-03-27T21:42:43.726 INFO:teuthology.orchestra.run.smithi060.stdout: 412617 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/pool-edit-mode-modal 2024-03-27T21:42:43.726 INFO:teuthology.orchestra.run.smithi060.stdout: 412621 4 -rw-r--r-- 1 ubuntu ubuntu 3496 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/pool-edit-mode-modal/pool-edit-mode-modal.component.ts 2024-03-27T21:42:43.726 INFO:teuthology.orchestra.run.smithi060.stdout: 412619 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/pool-edit-mode-modal/pool-edit-mode-modal.component.scss 2024-03-27T21:42:43.726 INFO:teuthology.orchestra.run.smithi060.stdout: 412620 4 -rw-r--r-- 1 ubuntu ubuntu 2965 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/pool-edit-mode-modal/pool-edit-mode-modal.component.spec.ts 2024-03-27T21:42:43.726 INFO:teuthology.orchestra.run.smithi060.stdout: 412622 4 -rw-r--r-- 1 ubuntu ubuntu 66 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/pool-edit-mode-modal/pool-edit-mode-response.model.ts 2024-03-27T21:42:43.726 INFO:teuthology.orchestra.run.smithi060.stdout: 412618 4 -rw-r--r-- 1 ubuntu ubuntu 1598 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/pool-edit-mode-modal/pool-edit-mode-modal.component.html 2024-03-27T21:42:43.727 INFO:teuthology.orchestra.run.smithi060.stdout: 412594 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/bootstrap-import-modal 2024-03-27T21:42:43.727 INFO:teuthology.orchestra.run.smithi060.stdout: 412595 4 -rw-r--r-- 1 ubuntu ubuntu 3686 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/bootstrap-import-modal/bootstrap-import-modal.component.html 2024-03-27T21:42:43.727 INFO:teuthology.orchestra.run.smithi060.stdout: 412596 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/bootstrap-import-modal/bootstrap-import-modal.component.scss 2024-03-27T21:42:43.727 INFO:teuthology.orchestra.run.smithi060.stdout: 412597 8 -rw-r--r-- 1 ubuntu ubuntu 4552 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/bootstrap-import-modal/bootstrap-import-modal.component.spec.ts 2024-03-27T21:42:43.727 INFO:teuthology.orchestra.run.smithi060.stdout: 412598 8 -rw-r--r-- 1 ubuntu ubuntu 5524 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/bootstrap-import-modal/bootstrap-import-modal.component.ts 2024-03-27T21:42:43.727 INFO:teuthology.orchestra.run.smithi060.stdout: 412599 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/daemon-list 2024-03-27T21:42:43.727 INFO:teuthology.orchestra.run.smithi060.stdout: 412600 4 -rw-r--r-- 1 ubuntu ubuntu 366 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/daemon-list/daemon-list.component.html 2024-03-27T21:42:43.727 INFO:teuthology.orchestra.run.smithi060.stdout: 412602 4 -rw-r--r-- 1 ubuntu ubuntu 1014 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/daemon-list/daemon-list.component.spec.ts 2024-03-27T21:42:43.728 INFO:teuthology.orchestra.run.smithi060.stdout: 412601 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/daemon-list/daemon-list.component.scss 2024-03-27T21:42:43.728 INFO:teuthology.orchestra.run.smithi060.stdout: 412603 4 -rw-r--r-- 1 ubuntu ubuntu 1712 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/daemon-list/daemon-list.component.ts 2024-03-27T21:42:43.728 INFO:teuthology.orchestra.run.smithi060.stdout: 412610 4 -rw-r--r-- 1 ubuntu ubuntu 441 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/mirror-health-color.pipe.ts 2024-03-27T21:42:43.728 INFO:teuthology.orchestra.run.smithi060.stdout: 412629 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/pool-list 2024-03-27T21:42:43.728 INFO:teuthology.orchestra.run.smithi060.stdout: 412633 8 -rw-r--r-- 1 ubuntu ubuntu 6112 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/pool-list/pool-list.component.ts 2024-03-27T21:42:43.728 INFO:teuthology.orchestra.run.smithi060.stdout: 412631 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/pool-list/pool-list.component.scss 2024-03-27T21:42:43.728 INFO:teuthology.orchestra.run.smithi060.stdout: 412632 4 -rw-r--r-- 1 ubuntu ubuntu 1188 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/pool-list/pool-list.component.spec.ts 2024-03-27T21:42:43.728 INFO:teuthology.orchestra.run.smithi060.stdout: 412630 4 -rw-r--r-- 1 ubuntu ubuntu 1029 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/pool-list/pool-list.component.html 2024-03-27T21:42:43.729 INFO:teuthology.orchestra.run.smithi060.stdout: 412609 4 -rw-r--r-- 1 ubuntu ubuntu 661 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/mirror-health-color.pipe.spec.ts 2024-03-27T21:42:43.729 INFO:teuthology.orchestra.run.smithi060.stdout: 412611 4 -rw-r--r-- 1 ubuntu ubuntu 1666 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/mirroring.module.ts 2024-03-27T21:42:43.729 INFO:teuthology.orchestra.run.smithi060.stdout: 412623 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/pool-edit-peer-modal 2024-03-27T21:42:43.729 INFO:teuthology.orchestra.run.smithi060.stdout: 412627 8 -rw-r--r-- 1 ubuntu ubuntu 4378 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/pool-edit-peer-modal/pool-edit-peer-modal.component.ts 2024-03-27T21:42:43.729 INFO:teuthology.orchestra.run.smithi060.stdout: 412626 8 -rw-r--r-- 1 ubuntu ubuntu 4848 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/pool-edit-peer-modal/pool-edit-peer-modal.component.spec.ts 2024-03-27T21:42:43.729 INFO:teuthology.orchestra.run.smithi060.stdout: 412624 4 -rw-r--r-- 1 ubuntu ubuntu 3722 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/pool-edit-peer-modal/pool-edit-peer-modal.component.html 2024-03-27T21:42:43.729 INFO:teuthology.orchestra.run.smithi060.stdout: 412625 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/pool-edit-peer-modal/pool-edit-peer-modal.component.scss 2024-03-27T21:42:43.729 INFO:teuthology.orchestra.run.smithi060.stdout: 412628 4 -rw-r--r-- 1 ubuntu ubuntu 139 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/pool-edit-peer-modal/pool-edit-peer-response.model.ts 2024-03-27T21:42:43.730 INFO:teuthology.orchestra.run.smithi060.stdout: 412612 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/overview 2024-03-27T21:42:43.730 INFO:teuthology.orchestra.run.smithi060.stdout: 412613 4 -rw-r--r-- 1 ubuntu ubuntu 1756 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/overview/overview.component.html 2024-03-27T21:42:43.730 INFO:teuthology.orchestra.run.smithi060.stdout: 412616 8 -rw-r--r-- 1 ubuntu ubuntu 4165 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/overview/overview.component.ts 2024-03-27T21:42:43.730 INFO:teuthology.orchestra.run.smithi060.stdout: 412614 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/overview/overview.component.scss 2024-03-27T21:42:43.730 INFO:teuthology.orchestra.run.smithi060.stdout: 412615 4 -rw-r--r-- 1 ubuntu ubuntu 2709 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/overview/overview.component.spec.ts 2024-03-27T21:42:43.730 INFO:teuthology.orchestra.run.smithi060.stdout: 412604 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/image-list 2024-03-27T21:42:43.730 INFO:teuthology.orchestra.run.smithi060.stdout: 412607 4 -rw-r--r-- 1 ubuntu ubuntu 1161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/image-list/image-list.component.spec.ts 2024-03-27T21:42:43.731 INFO:teuthology.orchestra.run.smithi060.stdout: 412606 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/image-list/image-list.component.scss 2024-03-27T21:42:43.731 INFO:teuthology.orchestra.run.smithi060.stdout: 412608 4 -rw-r--r-- 1 ubuntu ubuntu 3095 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/image-list/image-list.component.ts 2024-03-27T21:42:43.731 INFO:teuthology.orchestra.run.smithi060.stdout: 412605 4 -rw-r--r-- 1 ubuntu ubuntu 2376 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/mirroring/image-list/image-list.component.html 2024-03-27T21:42:43.731 INFO:teuthology.orchestra.run.smithi060.stdout: 412684 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-snapshot-form 2024-03-27T21:42:43.731 INFO:teuthology.orchestra.run.smithi060.stdout: 412687 4 -rw-r--r-- 1 ubuntu ubuntu 3354 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-snapshot-form/rbd-snapshot-form-modal.component.spec.ts 2024-03-27T21:42:43.731 INFO:teuthology.orchestra.run.smithi060.stdout: 412685 4 -rw-r--r-- 1 ubuntu ubuntu 2896 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-snapshot-form/rbd-snapshot-form-modal.component.html 2024-03-27T21:42:43.731 INFO:teuthology.orchestra.run.smithi060.stdout: 412686 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-snapshot-form/rbd-snapshot-form-modal.component.scss 2024-03-27T21:42:43.731 INFO:teuthology.orchestra.run.smithi060.stdout: 412688 8 -rw-r--r-- 1 ubuntu ubuntu 5014 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-snapshot-form/rbd-snapshot-form-modal.component.ts 2024-03-27T21:42:43.731 INFO:teuthology.orchestra.run.smithi060.stdout: 412573 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-iqn-settings-modal 2024-03-27T21:42:43.732 INFO:teuthology.orchestra.run.smithi060.stdout: 412576 4 -rw-r--r-- 1 ubuntu ubuntu 2317 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-iqn-settings-modal/iscsi-target-iqn-settings-modal.component.spec.ts 2024-03-27T21:42:43.732 INFO:teuthology.orchestra.run.smithi060.stdout: 412574 4 -rw-r--r-- 1 ubuntu ubuntu 1182 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-iqn-settings-modal/iscsi-target-iqn-settings-modal.component.html 2024-03-27T21:42:43.732 INFO:teuthology.orchestra.run.smithi060.stdout: 412575 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-iqn-settings-modal/iscsi-target-iqn-settings-modal.component.scss 2024-03-27T21:42:43.732 INFO:teuthology.orchestra.run.smithi060.stdout: 412577 4 -rw-r--r-- 1 ubuntu ubuntu 1810 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-iqn-settings-modal/iscsi-target-iqn-settings-modal.component.ts 2024-03-27T21:42:43.732 INFO:teuthology.orchestra.run.smithi060.stdout: 412542 8 -rw-r--r-- 1 ubuntu ubuntu 7991 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/block.module.ts 2024-03-27T21:42:43.732 INFO:teuthology.orchestra.run.smithi060.stdout: 412563 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-form 2024-03-27T21:42:43.732 INFO:teuthology.orchestra.run.smithi060.stdout: 412564 28 -rw-r--r-- 1 ubuntu ubuntu 28650 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-form/iscsi-target-form.component.html 2024-03-27T21:42:43.732 INFO:teuthology.orchestra.run.smithi060.stdout: 412566 20 -rw-r--r-- 1 ubuntu ubuntu 18776 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-form/iscsi-target-form.component.spec.ts 2024-03-27T21:42:43.733 INFO:teuthology.orchestra.run.smithi060.stdout: 412565 4 -rw-r--r-- 1 ubuntu ubuntu 34 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-form/iscsi-target-form.component.scss 2024-03-27T21:42:43.733 INFO:teuthology.orchestra.run.smithi060.stdout: 412567 28 -rw-r--r-- 1 ubuntu ubuntu 27729 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-form/iscsi-target-form.component.ts 2024-03-27T21:42:43.733 INFO:teuthology.orchestra.run.smithi060.stdout: 412706 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-move-modal 2024-03-27T21:42:43.733 INFO:teuthology.orchestra.run.smithi060.stdout: 412708 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-move-modal/rbd-trash-move-modal.component.scss 2024-03-27T21:42:43.733 INFO:teuthology.orchestra.run.smithi060.stdout: 412709 4 -rw-r--r-- 1 ubuntu ubuntu 3140 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-move-modal/rbd-trash-move-modal.component.spec.ts 2024-03-27T21:42:43.733 INFO:teuthology.orchestra.run.smithi060.stdout: 412707 4 -rw-r--r-- 1 ubuntu ubuntu 2134 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-move-modal/rbd-trash-move-modal.component.html 2024-03-27T21:42:43.733 INFO:teuthology.orchestra.run.smithi060.stdout: 412710 4 -rw-r--r-- 1 ubuntu ubuntu 2744 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-move-modal/rbd-trash-move-modal.component.ts 2024-03-27T21:42:43.733 INFO:teuthology.orchestra.run.smithi060.stdout: 412663 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-list 2024-03-27T21:42:43.733 INFO:teuthology.orchestra.run.smithi060.stdout: 412667 24 -rw-r--r-- 1 ubuntu ubuntu 21959 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-list/rbd-list.component.ts 2024-03-27T21:42:43.734 INFO:teuthology.orchestra.run.smithi060.stdout: 412668 4 -rw-r--r-- 1 ubuntu ubuntu 226 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-list/rbd-model.ts 2024-03-27T21:42:43.734 INFO:teuthology.orchestra.run.smithi060.stdout: 412666 12 -rw-r--r-- 1 ubuntu ubuntu 12089 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-list/rbd-list.component.spec.ts 2024-03-27T21:42:43.734 INFO:teuthology.orchestra.run.smithi060.stdout: 412664 8 -rw-r--r-- 1 ubuntu ubuntu 4422 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-list/rbd-list.component.html 2024-03-27T21:42:43.734 INFO:teuthology.orchestra.run.smithi060.stdout: 412665 4 -rw-r--r-- 1 ubuntu ubuntu 77 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-list/rbd-list.component.scss 2024-03-27T21:42:43.734 INFO:teuthology.orchestra.run.smithi060.stdout: 412578 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-list 2024-03-27T21:42:43.734 INFO:teuthology.orchestra.run.smithi060.stdout: 412580 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-list/iscsi-target-list.component.scss 2024-03-27T21:42:43.735 INFO:teuthology.orchestra.run.smithi060.stdout: 412582 8 -rw-r--r-- 1 ubuntu ubuntu 7330 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-list/iscsi-target-list.component.ts 2024-03-27T21:42:43.735 INFO:teuthology.orchestra.run.smithi060.stdout: 412579 4 -rw-r--r-- 1 ubuntu ubuntu 1812 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-list/iscsi-target-list.component.html 2024-03-27T21:42:43.735 INFO:teuthology.orchestra.run.smithi060.stdout: 412581 12 -rw-r--r-- 1 ubuntu ubuntu 9946 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-list/iscsi-target-list.component.spec.ts 2024-03-27T21:42:43.735 INFO:teuthology.orchestra.run.smithi060.stdout: 412669 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-namespace-form 2024-03-27T21:42:43.735 INFO:teuthology.orchestra.run.smithi060.stdout: 412670 4 -rw-r--r-- 1 ubuntu ubuntu 3012 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-namespace-form/rbd-namespace-form-modal.component.html 2024-03-27T21:42:43.735 INFO:teuthology.orchestra.run.smithi060.stdout: 412673 8 -rw-r--r-- 1 ubuntu ubuntu 4463 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-namespace-form/rbd-namespace-form-modal.component.ts 2024-03-27T21:42:43.735 INFO:teuthology.orchestra.run.smithi060.stdout: 412672 4 -rw-r--r-- 1 ubuntu ubuntu 1377 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-namespace-form/rbd-namespace-form-modal.component.spec.ts 2024-03-27T21:42:43.735 INFO:teuthology.orchestra.run.smithi060.stdout: 412671 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-namespace-form/rbd-namespace-form-modal.component.scss 2024-03-27T21:42:43.735 INFO:teuthology.orchestra.run.smithi060.stdout: 412634 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-configuration-form 2024-03-27T21:42:43.736 INFO:teuthology.orchestra.run.smithi060.stdout: 412636 4 -rw-r--r-- 1 ubuntu ubuntu 57 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-configuration-form/rbd-configuration-form.component.scss 2024-03-27T21:42:43.736 INFO:teuthology.orchestra.run.smithi060.stdout: 412638 8 -rw-r--r-- 1 ubuntu ubuntu 5692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-configuration-form/rbd-configuration-form.component.ts 2024-03-27T21:42:43.736 INFO:teuthology.orchestra.run.smithi060.stdout: 412635 4 -rw-r--r-- 1 ubuntu ubuntu 3075 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-configuration-form/rbd-configuration-form.component.html 2024-03-27T21:42:43.736 INFO:teuthology.orchestra.run.smithi060.stdout: 412637 12 -rw-r--r-- 1 ubuntu ubuntu 10069 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-configuration-form/rbd-configuration-form.component.spec.ts 2024-03-27T21:42:43.736 INFO:teuthology.orchestra.run.smithi060.stdout: 412689 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-snapshot-list 2024-03-27T21:42:43.736 INFO:teuthology.orchestra.run.smithi060.stdout: 412693 12 -rw-r--r-- 1 ubuntu ubuntu 11963 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-snapshot-list/rbd-snapshot-list.component.spec.ts 2024-03-27T21:42:43.736 INFO:teuthology.orchestra.run.smithi060.stdout: 412694 12 -rw-r--r-- 1 ubuntu ubuntu 11644 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-snapshot-list/rbd-snapshot-list.component.ts 2024-03-27T21:42:43.736 INFO:teuthology.orchestra.run.smithi060.stdout: 412691 4 -rw-r--r-- 1 ubuntu ubuntu 557 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-snapshot-list/rbd-snapshot-list.component.html 2024-03-27T21:42:43.737 INFO:teuthology.orchestra.run.smithi060.stdout: 412695 4 -rw-r--r-- 1 ubuntu ubuntu 150 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-snapshot-list/rbd-snapshot.model.ts 2024-03-27T21:42:43.737 INFO:teuthology.orchestra.run.smithi060.stdout: 412692 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-snapshot-list/rbd-snapshot-list.component.scss 2024-03-27T21:42:43.737 INFO:teuthology.orchestra.run.smithi060.stdout: 412690 8 -rw-r--r-- 1 ubuntu ubuntu 4433 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-snapshot-list/rbd-snapshot-actions.model.ts 2024-03-27T21:42:43.737 INFO:teuthology.orchestra.run.smithi060.stdout: 413181 4 drwxr-xr-x 31 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw 2024-03-27T21:42:43.737 INFO:teuthology.orchestra.run.smithi060.stdout: 413234 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-daemon-details 2024-03-27T21:42:43.737 INFO:teuthology.orchestra.run.smithi060.stdout: 413238 4 -rw-r--r-- 1 ubuntu ubuntu 1277 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-daemon-details/rgw-daemon-details.component.ts 2024-03-27T21:42:43.737 INFO:teuthology.orchestra.run.smithi060.stdout: 413236 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-daemon-details/rgw-daemon-details.component.scss 2024-03-27T21:42:43.737 INFO:teuthology.orchestra.run.smithi060.stdout: 413237 4 -rw-r--r-- 1 ubuntu ubuntu 1512 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-daemon-details/rgw-daemon-details.component.spec.ts 2024-03-27T21:42:43.738 INFO:teuthology.orchestra.run.smithi060.stdout: 413235 4 -rw-r--r-- 1 ubuntu ubuntu 1386 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-daemon-details/rgw-daemon-details.component.html 2024-03-27T21:42:43.738 INFO:teuthology.orchestra.run.smithi060.stdout: 413254 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-import 2024-03-27T21:42:43.738 INFO:teuthology.orchestra.run.smithi060.stdout: 413255 8 -rw-r--r-- 1 ubuntu ubuntu 7539 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-import/rgw-multisite-import.component.html 2024-03-27T21:42:43.738 INFO:teuthology.orchestra.run.smithi060.stdout: 413256 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-import/rgw-multisite-import.component.scss 2024-03-27T21:42:43.738 INFO:teuthology.orchestra.run.smithi060.stdout: 413258 8 -rw-r--r-- 1 ubuntu ubuntu 5692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-import/rgw-multisite-import.component.ts 2024-03-27T21:42:43.738 INFO:teuthology.orchestra.run.smithi060.stdout: 413257 4 -rw-r--r-- 1 ubuntu ubuntu 1230 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-import/rgw-multisite-import.component.spec.ts 2024-03-27T21:42:43.738 INFO:teuthology.orchestra.run.smithi060.stdout: 413290 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-sync-metadata-info 2024-03-27T21:42:43.738 INFO:teuthology.orchestra.run.smithi060.stdout: 413291 4 -rw-r--r-- 1 ubuntu ubuntu 2356 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-sync-metadata-info/rgw-sync-metadata-info.component.html 2024-03-27T21:42:43.739 INFO:teuthology.orchestra.run.smithi060.stdout: 413292 4 -rw-r--r-- 1 ubuntu ubuntu 143 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-sync-metadata-info/rgw-sync-metadata-info.component.scss 2024-03-27T21:42:43.739 INFO:teuthology.orchestra.run.smithi060.stdout: 413293 4 -rw-r--r-- 1 ubuntu ubuntu 3014 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-sync-metadata-info/rgw-sync-metadata-info.component.spec.ts 2024-03-27T21:42:43.739 INFO:teuthology.orchestra.run.smithi060.stdout: 413294 4 -rw-r--r-- 1 ubuntu ubuntu 402 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-sync-metadata-info/rgw-sync-metadata-info.component.ts 2024-03-27T21:42:43.739 INFO:teuthology.orchestra.run.smithi060.stdout: 413249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-export 2024-03-27T21:42:43.739 INFO:teuthology.orchestra.run.smithi060.stdout: 413252 4 -rw-r--r-- 1 ubuntu ubuntu 1230 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-export/rgw-multisite-export.component.spec.ts 2024-03-27T21:42:43.739 INFO:teuthology.orchestra.run.smithi060.stdout: 413250 4 -rw-r--r-- 1 ubuntu ubuntu 2175 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-export/rgw-multisite-export.component.html 2024-03-27T21:42:43.739 INFO:teuthology.orchestra.run.smithi060.stdout: 413251 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-export/rgw-multisite-export.component.scss 2024-03-27T21:42:43.740 INFO:teuthology.orchestra.run.smithi060.stdout: 413253 4 -rw-r--r-- 1 ubuntu ubuntu 1936 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-export/rgw-multisite-export.component.ts 2024-03-27T21:42:43.740 INFO:teuthology.orchestra.run.smithi060.stdout: 413330 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-subuser-modal 2024-03-27T21:42:43.740 INFO:teuthology.orchestra.run.smithi060.stdout: 413333 4 -rw-r--r-- 1 ubuntu ubuntu 2381 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-subuser-modal/rgw-user-subuser-modal.component.spec.ts 2024-03-27T21:42:43.740 INFO:teuthology.orchestra.run.smithi060.stdout: 413332 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-subuser-modal/rgw-user-subuser-modal.component.scss 2024-03-27T21:42:43.740 INFO:teuthology.orchestra.run.smithi060.stdout: 413331 8 -rw-r--r-- 1 ubuntu ubuntu 4714 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-subuser-modal/rgw-user-subuser-modal.component.html 2024-03-27T21:42:43.740 INFO:teuthology.orchestra.run.smithi060.stdout: 413334 4 -rw-r--r-- 1 ubuntu ubuntu 3886 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-subuser-modal/rgw-user-subuser-modal.component.ts 2024-03-27T21:42:43.740 INFO:teuthology.orchestra.run.smithi060.stdout: 413274 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-zonegroup-form 2024-03-27T21:42:43.741 INFO:teuthology.orchestra.run.smithi060.stdout: 413277 4 -rw-r--r-- 1 ubuntu ubuntu 3835 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-zonegroup-form/rgw-multisite-zonegroup-form.component.spec.ts 2024-03-27T21:42:43.741 INFO:teuthology.orchestra.run.smithi060.stdout: 413278 12 -rw-r--r-- 1 ubuntu ubuntu 11613 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-zonegroup-form/rgw-multisite-zonegroup-form.component.ts 2024-03-27T21:42:43.741 INFO:teuthology.orchestra.run.smithi060.stdout: 413275 12 -rw-r--r-- 1 ubuntu ubuntu 9034 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-zonegroup-form/rgw-multisite-zonegroup-form.component.html 2024-03-27T21:42:43.741 INFO:teuthology.orchestra.run.smithi060.stdout: 413276 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-zonegroup-form/rgw-multisite-zonegroup-form.component.scss 2024-03-27T21:42:43.741 INFO:teuthology.orchestra.run.smithi060.stdout: 413218 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-form 2024-03-27T21:42:43.741 INFO:teuthology.orchestra.run.smithi060.stdout: 413221 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-form/rgw-bucket-form.component.scss 2024-03-27T21:42:43.741 INFO:teuthology.orchestra.run.smithi060.stdout: 413219 4 -rw-r--r-- 1 ubuntu ubuntu 349 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-form/rgw-bucket-acl-permissions.enum.ts 2024-03-27T21:42:43.741 INFO:teuthology.orchestra.run.smithi060.stdout: 413223 20 -rw-r--r-- 1 ubuntu ubuntu 18767 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-form/rgw-bucket-form.component.ts 2024-03-27T21:42:43.742 INFO:teuthology.orchestra.run.smithi060.stdout: 413220 28 -rw-r--r-- 1 ubuntu ubuntu 25804 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-form/rgw-bucket-form.component.html 2024-03-27T21:42:43.742 INFO:teuthology.orchestra.run.smithi060.stdout: 413222 12 -rw-r--r-- 1 ubuntu ubuntu 10267 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-form/rgw-bucket-form.component.spec.ts 2024-03-27T21:42:43.742 INFO:teuthology.orchestra.run.smithi060.stdout: 413320 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-list 2024-03-27T21:42:43.742 INFO:teuthology.orchestra.run.smithi060.stdout: 413323 8 -rw-r--r-- 1 ubuntu ubuntu 5277 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-list/rgw-user-list.component.spec.ts 2024-03-27T21:42:43.742 INFO:teuthology.orchestra.run.smithi060.stdout: 413321 4 -rw-r--r-- 1 ubuntu ubuntu 1532 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-list/rgw-user-list.component.html 2024-03-27T21:42:43.742 INFO:teuthology.orchestra.run.smithi060.stdout: 413322 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-list/rgw-user-list.component.scss 2024-03-27T21:42:43.742 INFO:teuthology.orchestra.run.smithi060.stdout: 413324 8 -rw-r--r-- 1 ubuntu ubuntu 6179 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-list/rgw-user-list.component.ts 2024-03-27T21:42:43.742 INFO:teuthology.orchestra.run.smithi060.stdout: 413300 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-system-user 2024-03-27T21:42:43.742 INFO:teuthology.orchestra.run.smithi060.stdout: 413301 4 -rw-r--r-- 1 ubuntu ubuntu 1465 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-system-user/rgw-system-user.component.html 2024-03-27T21:42:43.743 INFO:teuthology.orchestra.run.smithi060.stdout: 413302 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-system-user/rgw-system-user.component.scss 2024-03-27T21:42:43.743 INFO:teuthology.orchestra.run.smithi060.stdout: 413303 4 -rw-r--r-- 1 ubuntu ubuntu 1195 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-system-user/rgw-system-user.component.spec.ts 2024-03-27T21:42:43.743 INFO:teuthology.orchestra.run.smithi060.stdout: 413304 4 -rw-r--r-- 1 ubuntu ubuntu 1692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-system-user/rgw-system-user.component.ts 2024-03-27T21:42:43.743 INFO:teuthology.orchestra.run.smithi060.stdout: 413259 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-migrate 2024-03-27T21:42:43.743 INFO:teuthology.orchestra.run.smithi060.stdout: 413261 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-migrate/rgw-multisite-migrate.component.scss 2024-03-27T21:42:43.743 INFO:teuthology.orchestra.run.smithi060.stdout: 413263 8 -rw-r--r-- 1 ubuntu ubuntu 7025 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-migrate/rgw-multisite-migrate.component.ts 2024-03-27T21:42:43.743 INFO:teuthology.orchestra.run.smithi060.stdout: 413260 8 -rw-r--r-- 1 ubuntu ubuntu 6672 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-migrate/rgw-multisite-migrate.component.html 2024-03-27T21:42:43.743 INFO:teuthology.orchestra.run.smithi060.stdout: 413262 4 -rw-r--r-- 1 ubuntu ubuntu 1237 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-migrate/rgw-multisite-migrate.component.spec.ts 2024-03-27T21:42:43.744 INFO:teuthology.orchestra.run.smithi060.stdout: 413264 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-realm-form 2024-03-27T21:42:43.744 INFO:teuthology.orchestra.run.smithi060.stdout: 413266 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-realm-form/rgw-multisite-realm-form.component.scss 2024-03-27T21:42:43.744 INFO:teuthology.orchestra.run.smithi060.stdout: 413265 4 -rw-r--r-- 1 ubuntu ubuntu 2554 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-realm-form/rgw-multisite-realm-form.component.html 2024-03-27T21:42:43.744 INFO:teuthology.orchestra.run.smithi060.stdout: 413268 8 -rw-r--r-- 1 ubuntu ubuntu 4621 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-realm-form/rgw-multisite-realm-form.component.ts 2024-03-27T21:42:43.744 INFO:teuthology.orchestra.run.smithi060.stdout: 413267 4 -rw-r--r-- 1 ubuntu ubuntu 3305 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-realm-form/rgw-multisite-realm-form.component.spec.ts 2024-03-27T21:42:43.744 INFO:teuthology.orchestra.run.smithi060.stdout: 413310 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-details 2024-03-27T21:42:43.744 INFO:teuthology.orchestra.run.smithi060.stdout: 413314 4 -rw-r--r-- 1 ubuntu ubuntu 3573 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-details/rgw-user-details.component.ts 2024-03-27T21:42:43.745 INFO:teuthology.orchestra.run.smithi060.stdout: 413312 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-details/rgw-user-details.component.scss 2024-03-27T21:42:43.745 INFO:teuthology.orchestra.run.smithi060.stdout: 413313 4 -rw-r--r-- 1 ubuntu ubuntu 2228 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-details/rgw-user-details.component.spec.ts 2024-03-27T21:42:43.745 INFO:teuthology.orchestra.run.smithi060.stdout: 413311 8 -rw-r--r-- 1 ubuntu ubuntu 5483 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-details/rgw-user-details.component.html 2024-03-27T21:42:43.745 INFO:teuthology.orchestra.run.smithi060.stdout: 413269 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-zone-form 2024-03-27T21:42:43.745 INFO:teuthology.orchestra.run.smithi060.stdout: 413270 16 -rw-r--r-- 1 ubuntu ubuntu 12770 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-zone-form/rgw-multisite-zone-form.component.html 2024-03-27T21:42:43.745 INFO:teuthology.orchestra.run.smithi060.stdout: 413271 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-zone-form/rgw-multisite-zone-form.component.scss 2024-03-27T21:42:43.745 INFO:teuthology.orchestra.run.smithi060.stdout: 413273 16 -rw-r--r-- 1 ubuntu ubuntu 12984 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-zone-form/rgw-multisite-zone-form.component.ts 2024-03-27T21:42:43.745 INFO:teuthology.orchestra.run.smithi060.stdout: 413272 4 -rw-r--r-- 1 ubuntu ubuntu 1245 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-zone-form/rgw-multisite-zone-form.component.spec.ts 2024-03-27T21:42:43.745 INFO:teuthology.orchestra.run.smithi060.stdout: 413244 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-details 2024-03-27T21:42:43.746 INFO:teuthology.orchestra.run.smithi060.stdout: 413248 24 -rw-r--r-- 1 ubuntu ubuntu 20624 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-details/rgw-multisite-details.component.ts 2024-03-27T21:42:43.746 INFO:teuthology.orchestra.run.smithi060.stdout: 413246 4 -rw-r--r-- 1 ubuntu ubuntu 203 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-details/rgw-multisite-details.component.scss 2024-03-27T21:42:43.746 INFO:teuthology.orchestra.run.smithi060.stdout: 413247 4 -rw-r--r-- 1 ubuntu ubuntu 1442 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-details/rgw-multisite-details.component.spec.ts 2024-03-27T21:42:43.746 INFO:teuthology.orchestra.run.smithi060.stdout: 413245 8 -rw-r--r-- 1 ubuntu ubuntu 5108 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-details/rgw-multisite-details.component.html 2024-03-27T21:42:43.746 INFO:teuthology.orchestra.run.smithi060.stdout: 413335 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-swift-key-modal 2024-03-27T21:42:43.746 INFO:teuthology.orchestra.run.smithi060.stdout: 413339 4 -rw-r--r-- 1 ubuntu ubuntu 827 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-swift-key-modal/rgw-user-swift-key-modal.component.ts 2024-03-27T21:42:43.746 INFO:teuthology.orchestra.run.smithi060.stdout: 413338 4 -rw-r--r-- 1 ubuntu ubuntu 1098 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-swift-key-modal/rgw-user-swift-key-modal.component.spec.ts 2024-03-27T21:42:43.746 INFO:teuthology.orchestra.run.smithi060.stdout: 413337 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-swift-key-modal/rgw-user-swift-key-modal.component.scss 2024-03-27T21:42:43.747 INFO:teuthology.orchestra.run.smithi060.stdout: 413336 4 -rw-r--r-- 1 ubuntu ubuntu 1714 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-swift-key-modal/rgw-user-swift-key-modal.component.html 2024-03-27T21:42:43.747 INFO:teuthology.orchestra.run.smithi060.stdout: 413229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-config-modal 2024-03-27T21:42:43.747 INFO:teuthology.orchestra.run.smithi060.stdout: 413232 4 -rw-r--r-- 1 ubuntu ubuntu 1203 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-config-modal/rgw-config-modal.component.spec.ts 2024-03-27T21:42:43.747 INFO:teuthology.orchestra.run.smithi060.stdout: 413231 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-config-modal/rgw-config-modal.component.scss 2024-03-27T21:42:43.747 INFO:teuthology.orchestra.run.smithi060.stdout: 413230 12 -rw-r--r-- 1 ubuntu ubuntu 9675 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-config-modal/rgw-config-modal.component.html 2024-03-27T21:42:43.747 INFO:teuthology.orchestra.run.smithi060.stdout: 413233 8 -rw-r--r-- 1 ubuntu ubuntu 4432 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-config-modal/rgw-config-modal.component.ts 2024-03-27T21:42:43.747 INFO:teuthology.orchestra.run.smithi060.stdout: 413340 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-tabs 2024-03-27T21:42:43.747 INFO:teuthology.orchestra.run.smithi060.stdout: 413341 4 -rw-r--r-- 1 ubuntu ubuntu 485 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-tabs/rgw-user-tabs.component.html 2024-03-27T21:42:43.748 INFO:teuthology.orchestra.run.smithi060.stdout: 413343 4 -rw-r--r-- 1 ubuntu ubuntu 640 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-tabs/rgw-user-tabs.component.spec.ts 2024-03-27T21:42:43.748 INFO:teuthology.orchestra.run.smithi060.stdout: 413342 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-tabs/rgw-user-tabs.component.scss 2024-03-27T21:42:43.748 INFO:teuthology.orchestra.run.smithi060.stdout: 413344 4 -rw-r--r-- 1 ubuntu ubuntu 226 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-tabs/rgw-user-tabs.component.ts 2024-03-27T21:42:43.748 INFO:teuthology.orchestra.run.smithi060.stdout: 413182 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/bucket-tag-modal 2024-03-27T21:42:43.748 INFO:teuthology.orchestra.run.smithi060.stdout: 413184 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/bucket-tag-modal/bucket-tag-modal.component.scss 2024-03-27T21:42:43.748 INFO:teuthology.orchestra.run.smithi060.stdout: 413183 4 -rw-r--r-- 1 ubuntu ubuntu 2328 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/bucket-tag-modal/bucket-tag-modal.component.html 2024-03-27T21:42:43.748 INFO:teuthology.orchestra.run.smithi060.stdout: 413185 4 -rw-r--r-- 1 ubuntu ubuntu 935 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/bucket-tag-modal/bucket-tag-modal.component.spec.ts 2024-03-27T21:42:43.748 INFO:teuthology.orchestra.run.smithi060.stdout: 413186 4 -rw-r--r-- 1 ubuntu ubuntu 2080 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/bucket-tag-modal/bucket-tag-modal.component.ts 2024-03-27T21:42:43.749 INFO:teuthology.orchestra.run.smithi060.stdout: 413295 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-sync-primary-zone 2024-03-27T21:42:43.749 INFO:teuthology.orchestra.run.smithi060.stdout: 413299 4 -rw-r--r-- 1 ubuntu ubuntu 446 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-sync-primary-zone/rgw-sync-primary-zone.component.ts 2024-03-27T21:42:43.749 INFO:teuthology.orchestra.run.smithi060.stdout: 413298 4 -rw-r--r-- 1 ubuntu ubuntu 1380 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-sync-primary-zone/rgw-sync-primary-zone.component.spec.ts 2024-03-27T21:42:43.749 INFO:teuthology.orchestra.run.smithi060.stdout: 413297 4 -rw-r--r-- 1 ubuntu ubuntu 189 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-sync-primary-zone/rgw-sync-primary-zone.component.scss 2024-03-27T21:42:43.749 INFO:teuthology.orchestra.run.smithi060.stdout: 413296 4 -rw-r--r-- 1 ubuntu ubuntu 590 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-sync-primary-zone/rgw-sync-primary-zone.component.html 2024-03-27T21:42:43.749 INFO:teuthology.orchestra.run.smithi060.stdout: 413192 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models 2024-03-27T21:42:43.749 INFO:teuthology.orchestra.run.smithi060.stdout: 413211 4 -rw-r--r-- 1 ubuntu ubuntu 121 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-user-subuser.ts 2024-03-27T21:42:43.749 INFO:teuthology.orchestra.run.smithi060.stdout: 413196 4 -rw-r--r-- 1 ubuntu ubuntu 218 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-daemon.ts 2024-03-27T21:42:43.750 INFO:teuthology.orchestra.run.smithi060.stdout: 413208 4 -rw-r--r-- 1 ubuntu ubuntu 339 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-user-capabilities.ts 2024-03-27T21:42:43.750 INFO:teuthology.orchestra.run.smithi060.stdout: 413210 4 -rw-r--r-- 1 ubuntu ubuntu 116 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-user-s3-key.ts 2024-03-27T21:42:43.750 INFO:teuthology.orchestra.run.smithi060.stdout: 413193 4 -rw-r--r-- 1 ubuntu ubuntu 187 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-bucket-encryption.ts 2024-03-27T21:42:43.750 INFO:teuthology.orchestra.run.smithi060.stdout: 413212 4 -rw-r--r-- 1 ubuntu ubuntu 71 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-user-swift-key.ts 2024-03-27T21:42:43.750 INFO:teuthology.orchestra.run.smithi060.stdout: 413197 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-multisite-zone-deletion-form 2024-03-27T21:42:43.750 INFO:teuthology.orchestra.run.smithi060.stdout: 413199 4 -rw-r--r-- 1 ubuntu ubuntu 97 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-multisite-zone-deletion-form/rgw-multisite-zone-deletion-form.component.scss 2024-03-27T21:42:43.750 INFO:teuthology.orchestra.run.smithi060.stdout: 413200 4 -rw-r--r-- 1 ubuntu ubuntu 1271 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-multisite-zone-deletion-form/rgw-multisite-zone-deletion-form.component.spec.ts 2024-03-27T21:42:43.751 INFO:teuthology.orchestra.run.smithi060.stdout: 413198 4 -rw-r--r-- 1 ubuntu ubuntu 2032 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-multisite-zone-deletion-form/rgw-multisite-zone-deletion-form.component.html 2024-03-27T21:42:43.751 INFO:teuthology.orchestra.run.smithi060.stdout: 413201 4 -rw-r--r-- 1 ubuntu ubuntu 3296 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-multisite-zone-deletion-form/rgw-multisite-zone-deletion-form.component.ts 2024-03-27T21:42:43.751 INFO:teuthology.orchestra.run.smithi060.stdout: 413209 4 -rw-r--r-- 1 ubuntu ubuntu 67 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-user-capability.ts 2024-03-27T21:42:43.751 INFO:teuthology.orchestra.run.smithi060.stdout: 413195 4 -rw-r--r-- 1 ubuntu ubuntu 85 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-bucket-versioning.ts 2024-03-27T21:42:43.751 INFO:teuthology.orchestra.run.smithi060.stdout: 413202 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-multisite-zonegroup-deletion-form 2024-03-27T21:42:43.751 INFO:teuthology.orchestra.run.smithi060.stdout: 413204 4 -rw-r--r-- 1 ubuntu ubuntu 97 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-multisite-zonegroup-deletion-form/rgw-multisite-zonegroup-deletion-form.component.scss 2024-03-27T21:42:43.751 INFO:teuthology.orchestra.run.smithi060.stdout: 413203 4 -rw-r--r-- 1 ubuntu ubuntu 2847 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-multisite-zonegroup-deletion-form/rgw-multisite-zonegroup-deletion-form.component.html 2024-03-27T21:42:43.751 INFO:teuthology.orchestra.run.smithi060.stdout: 413206 4 -rw-r--r-- 1 ubuntu ubuntu 3756 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-multisite-zonegroup-deletion-form/rgw-multisite-zonegroup-deletion-form.component.ts 2024-03-27T21:42:43.751 INFO:teuthology.orchestra.run.smithi060.stdout: 413205 4 -rw-r--r-- 1 ubuntu ubuntu 1321 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-multisite-zonegroup-deletion-form/rgw-multisite-zonegroup-deletion-form.component.spec.ts 2024-03-27T21:42:43.752 INFO:teuthology.orchestra.run.smithi060.stdout: 413194 4 -rw-r--r-- 1 ubuntu ubuntu 82 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-bucket-mfa-delete.ts 2024-03-27T21:42:43.752 INFO:teuthology.orchestra.run.smithi060.stdout: 413207 4 -rw-r--r-- 1 ubuntu ubuntu 1023 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-multisite.ts 2024-03-27T21:42:43.752 INFO:teuthology.orchestra.run.smithi060.stdout: 413239 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-daemon-list 2024-03-27T21:42:43.752 INFO:teuthology.orchestra.run.smithi060.stdout: 413243 4 -rw-r--r-- 1 ubuntu ubuntu 2531 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-daemon-list/rgw-daemon-list.component.ts 2024-03-27T21:42:43.752 INFO:teuthology.orchestra.run.smithi060.stdout: 413241 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-daemon-list/rgw-daemon-list.component.scss 2024-03-27T21:42:43.752 INFO:teuthology.orchestra.run.smithi060.stdout: 413242 4 -rw-r--r-- 1 ubuntu ubuntu 3791 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-daemon-list/rgw-daemon-list.component.spec.ts 2024-03-27T21:42:43.752 INFO:teuthology.orchestra.run.smithi060.stdout: 413240 4 -rw-r--r-- 1 ubuntu ubuntu 1574 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-daemon-list/rgw-daemon-list.component.html 2024-03-27T21:42:43.752 INFO:teuthology.orchestra.run.smithi060.stdout: 413285 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-sync-data-info 2024-03-27T21:42:43.753 INFO:teuthology.orchestra.run.smithi060.stdout: 413289 4 -rw-r--r-- 1 ubuntu ubuntu 374 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-sync-data-info/rgw-sync-data-info.component.ts 2024-03-27T21:42:43.753 INFO:teuthology.orchestra.run.smithi060.stdout: 413287 4 -rw-r--r-- 1 ubuntu ubuntu 143 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-sync-data-info/rgw-sync-data-info.component.scss 2024-03-27T21:42:43.753 INFO:teuthology.orchestra.run.smithi060.stdout: 413288 4 -rw-r--r-- 1 ubuntu ubuntu 2949 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-sync-data-info/rgw-sync-data-info.component.spec.ts 2024-03-27T21:42:43.753 INFO:teuthology.orchestra.run.smithi060.stdout: 413286 4 -rw-r--r-- 1 ubuntu ubuntu 1902 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-sync-data-info/rgw-sync-data-info.component.html 2024-03-27T21:42:43.753 INFO:teuthology.orchestra.run.smithi060.stdout: 413279 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard 2024-03-27T21:42:43.753 INFO:teuthology.orchestra.run.smithi060.stdout: 413280 4 -rw-r--r-- 1 ubuntu ubuntu 314 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-card-popover.scss 2024-03-27T21:42:43.753 INFO:teuthology.orchestra.run.smithi060.stdout: 413283 8 -rw-r--r-- 1 ubuntu ubuntu 4827 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.spec.ts 2024-03-27T21:42:43.753 INFO:teuthology.orchestra.run.smithi060.stdout: 413281 8 -rw-r--r-- 1 ubuntu ubuntu 7575 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.html 2024-03-27T21:42:43.753 INFO:teuthology.orchestra.run.smithi060.stdout: 413282 4 -rw-r--r-- 1 ubuntu ubuntu 452 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.scss 2024-03-27T21:42:43.754 INFO:teuthology.orchestra.run.smithi060.stdout: 413284 8 -rw-r--r-- 1 ubuntu ubuntu 5755 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.ts 2024-03-27T21:42:43.754 INFO:teuthology.orchestra.run.smithi060.stdout: 413315 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-form 2024-03-27T21:42:43.754 INFO:teuthology.orchestra.run.smithi060.stdout: 413318 12 -rw-r--r-- 1 ubuntu ubuntu 11388 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-form/rgw-user-form.component.spec.ts 2024-03-27T21:42:43.754 INFO:teuthology.orchestra.run.smithi060.stdout: 413316 28 -rw-r--r-- 1 ubuntu ubuntu 27954 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-form/rgw-user-form.component.html 2024-03-27T21:42:43.754 INFO:teuthology.orchestra.run.smithi060.stdout: 413319 28 -rw-r--r-- 1 ubuntu ubuntu 26346 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-form/rgw-user-form.component.ts 2024-03-27T21:42:43.754 INFO:teuthology.orchestra.run.smithi060.stdout: 413317 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-form/rgw-user-form.component.scss 2024-03-27T21:42:43.755 INFO:teuthology.orchestra.run.smithi060.stdout: 413224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-list 2024-03-27T21:42:43.755 INFO:teuthology.orchestra.run.smithi060.stdout: 413227 8 -rw-r--r-- 1 ubuntu ubuntu 5797 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-list/rgw-bucket-list.component.spec.ts 2024-03-27T21:42:43.755 INFO:teuthology.orchestra.run.smithi060.stdout: 413225 4 -rw-r--r-- 1 ubuntu ubuntu 1708 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-list/rgw-bucket-list.component.html 2024-03-27T21:42:43.755 INFO:teuthology.orchestra.run.smithi060.stdout: 413228 8 -rw-r--r-- 1 ubuntu ubuntu 7363 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-list/rgw-bucket-list.component.ts 2024-03-27T21:42:43.755 INFO:teuthology.orchestra.run.smithi060.stdout: 413226 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-list/rgw-bucket-list.component.scss 2024-03-27T21:42:43.755 INFO:teuthology.orchestra.run.smithi060.stdout: 413213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-details 2024-03-27T21:42:43.755 INFO:teuthology.orchestra.run.smithi060.stdout: 413217 4 -rw-r--r-- 1 ubuntu ubuntu 2150 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-details/rgw-bucket-details.component.ts 2024-03-27T21:42:43.755 INFO:teuthology.orchestra.run.smithi060.stdout: 413214 8 -rw-r--r-- 1 ubuntu ubuntu 5441 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-details/rgw-bucket-details.component.html 2024-03-27T21:42:43.756 INFO:teuthology.orchestra.run.smithi060.stdout: 413215 4 -rw-r--r-- 1 ubuntu ubuntu 72 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-details/rgw-bucket-details.component.scss 2024-03-27T21:42:43.756 INFO:teuthology.orchestra.run.smithi060.stdout: 413216 4 -rw-r--r-- 1 ubuntu ubuntu 1692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-details/rgw-bucket-details.component.spec.ts 2024-03-27T21:42:43.756 INFO:teuthology.orchestra.run.smithi060.stdout: 413187 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/create-rgw-service-entities 2024-03-27T21:42:43.756 INFO:teuthology.orchestra.run.smithi060.stdout: 413188 4 -rw-r--r-- 1 ubuntu ubuntu 2712 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/create-rgw-service-entities/create-rgw-service-entities.component.html 2024-03-27T21:42:43.756 INFO:teuthology.orchestra.run.smithi060.stdout: 413189 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/create-rgw-service-entities/create-rgw-service-entities.component.scss 2024-03-27T21:42:43.756 INFO:teuthology.orchestra.run.smithi060.stdout: 413190 4 -rw-r--r-- 1 ubuntu ubuntu 1273 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/create-rgw-service-entities/create-rgw-service-entities.component.spec.ts 2024-03-27T21:42:43.756 INFO:teuthology.orchestra.run.smithi060.stdout: 413191 4 -rw-r--r-- 1 ubuntu ubuntu 3621 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/create-rgw-service-entities/create-rgw-service-entities.component.ts 2024-03-27T21:42:43.756 INFO:teuthology.orchestra.run.smithi060.stdout: 413305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-capability-modal 2024-03-27T21:42:43.757 INFO:teuthology.orchestra.run.smithi060.stdout: 413306 4 -rw-r--r-- 1 ubuntu ubuntu 2605 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-capability-modal/rgw-user-capability-modal.component.html 2024-03-27T21:42:43.757 INFO:teuthology.orchestra.run.smithi060.stdout: 413309 4 -rw-r--r-- 1 ubuntu ubuntu 2625 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-capability-modal/rgw-user-capability-modal.component.ts 2024-03-27T21:42:43.757 INFO:teuthology.orchestra.run.smithi060.stdout: 413308 4 -rw-r--r-- 1 ubuntu ubuntu 1060 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-capability-modal/rgw-user-capability-modal.component.spec.ts 2024-03-27T21:42:43.757 INFO:teuthology.orchestra.run.smithi060.stdout: 413307 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-capability-modal/rgw-user-capability-modal.component.scss 2024-03-27T21:42:43.757 INFO:teuthology.orchestra.run.smithi060.stdout: 413345 8 -rw-r--r-- 1 ubuntu ubuntu 7798 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw.module.ts 2024-03-27T21:42:43.757 INFO:teuthology.orchestra.run.smithi060.stdout: 413325 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-s3-key-modal 2024-03-27T21:42:43.757 INFO:teuthology.orchestra.run.smithi060.stdout: 413327 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-s3-key-modal/rgw-user-s3-key-modal.component.scss 2024-03-27T21:42:43.757 INFO:teuthology.orchestra.run.smithi060.stdout: 413328 4 -rw-r--r-- 1 ubuntu ubuntu 1026 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-s3-key-modal/rgw-user-s3-key-modal.component.spec.ts 2024-03-27T21:42:43.757 INFO:teuthology.orchestra.run.smithi060.stdout: 413329 4 -rw-r--r-- 1 ubuntu ubuntu 2476 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-s3-key-modal/rgw-user-s3-key-modal.component.ts 2024-03-27T21:42:43.758 INFO:teuthology.orchestra.run.smithi060.stdout: 413326 8 -rw-r--r-- 1 ubuntu ubuntu 4615 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-s3-key-modal/rgw-user-s3-key-modal.component.html 2024-03-27T21:42:43.758 INFO:teuthology.orchestra.run.smithi060.stdout: 413346 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared 2024-03-27T21:42:43.758 INFO:teuthology.orchestra.run.smithi060.stdout: 413358 4 -rw-r--r-- 1 ubuntu ubuntu 1709 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/pg-category.model.ts 2024-03-27T21:42:43.758 INFO:teuthology.orchestra.run.smithi060.stdout: 413353 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/feedback 2024-03-27T21:42:43.758 INFO:teuthology.orchestra.run.smithi060.stdout: 413356 4 -rw-r--r-- 1 ubuntu ubuntu 2454 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/feedback/feedback.component.spec.ts 2024-03-27T21:42:43.758 INFO:teuthology.orchestra.run.smithi060.stdout: 413355 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/feedback/feedback.component.scss 2024-03-27T21:42:43.758 INFO:teuthology.orchestra.run.smithi060.stdout: 413354 8 -rw-r--r-- 1 ubuntu ubuntu 4717 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/feedback/feedback.component.html 2024-03-27T21:42:43.758 INFO:teuthology.orchestra.run.smithi060.stdout: 413357 4 -rw-r--r-- 1 ubuntu ubuntu 3404 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/feedback/feedback.component.ts 2024-03-27T21:42:43.759 INFO:teuthology.orchestra.run.smithi060.stdout: 413348 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/device-list 2024-03-27T21:42:43.759 INFO:teuthology.orchestra.run.smithi060.stdout: 413351 4 -rw-r--r-- 1 ubuntu ubuntu 817 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/device-list/device-list.component.spec.ts 2024-03-27T21:42:43.759 INFO:teuthology.orchestra.run.smithi060.stdout: 413352 4 -rw-r--r-- 1 ubuntu ubuntu 2974 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/device-list/device-list.component.ts 2024-03-27T21:42:43.759 INFO:teuthology.orchestra.run.smithi060.stdout: 413350 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/device-list/device-list.component.scss 2024-03-27T21:42:43.759 INFO:teuthology.orchestra.run.smithi060.stdout: 413349 4 -rw-r--r-- 1 ubuntu ubuntu 1779 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/device-list/device-list.component.html 2024-03-27T21:42:43.760 INFO:teuthology.orchestra.run.smithi060.stdout: 413360 4 -rw-r--r-- 1 ubuntu ubuntu 1611 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/pg-category.service.ts 2024-03-27T21:42:43.760 INFO:teuthology.orchestra.run.smithi060.stdout: 413347 4 -rw-r--r-- 1 ubuntu ubuntu 747 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/ceph-shared.module.ts 2024-03-27T21:42:43.760 INFO:teuthology.orchestra.run.smithi060.stdout: 413361 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/smart-list 2024-03-27T21:42:43.760 INFO:teuthology.orchestra.run.smithi060.stdout: 413369 8 -rw-r--r-- 1 ubuntu ubuntu 6774 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/smart-list/smart-list.component.ts 2024-03-27T21:42:43.760 INFO:teuthology.orchestra.run.smithi060.stdout: 413367 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/smart-list/smart-list.component.scss 2024-03-27T21:42:43.760 INFO:teuthology.orchestra.run.smithi060.stdout: 413362 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/smart-list/fixtures 2024-03-27T21:42:43.760 INFO:teuthology.orchestra.run.smithi060.stdout: 413364 8 -rw-r--r-- 1 ubuntu ubuntu 5866 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/smart-list/fixtures/smart_data_version_1_0_nvme_response.json 2024-03-27T21:42:43.761 INFO:teuthology.orchestra.run.smithi060.stdout: 413363 16 -rw-r--r-- 1 ubuntu ubuntu 13993 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/smart-list/fixtures/smart_data_version_1_0_ata_response.json 2024-03-27T21:42:43.761 INFO:teuthology.orchestra.run.smithi060.stdout: 413365 8 -rw-r--r-- 1 ubuntu ubuntu 8161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/smart-list/fixtures/smart_data_version_1_0_scsi_response.json 2024-03-27T21:42:43.761 INFO:teuthology.orchestra.run.smithi060.stdout: 413366 8 -rw-r--r-- 1 ubuntu ubuntu 5596 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/smart-list/smart-list.component.html 2024-03-27T21:42:43.761 INFO:teuthology.orchestra.run.smithi060.stdout: 413368 12 -rw-r--r-- 1 ubuntu ubuntu 8773 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/smart-list/smart-list.component.spec.ts 2024-03-27T21:42:43.761 INFO:teuthology.orchestra.run.smithi060.stdout: 413359 4 -rw-r--r-- 1 ubuntu ubuntu 1734 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/pg-category.service.spec.ts 2024-03-27T21:42:43.761 INFO:teuthology.orchestra.run.smithi060.stdout: 413151 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool 2024-03-27T21:42:43.761 INFO:teuthology.orchestra.run.smithi060.stdout: 413162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-details 2024-03-27T21:42:43.761 INFO:teuthology.orchestra.run.smithi060.stdout: 413166 4 -rw-r--r-- 1 ubuntu ubuntu 2330 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-details/pool-details.component.ts 2024-03-27T21:42:43.761 INFO:teuthology.orchestra.run.smithi060.stdout: 413165 8 -rw-r--r-- 1 ubuntu ubuntu 5809 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-details/pool-details.component.spec.ts 2024-03-27T21:42:43.762 INFO:teuthology.orchestra.run.smithi060.stdout: 413164 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-details/pool-details.component.scss 2024-03-27T21:42:43.762 INFO:teuthology.orchestra.run.smithi060.stdout: 413163 4 -rw-r--r-- 1 ubuntu ubuntu 1848 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-details/pool-details.component.html 2024-03-27T21:42:43.762 INFO:teuthology.orchestra.run.smithi060.stdout: 413157 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/erasure-code-profile-form 2024-03-27T21:42:43.762 INFO:teuthology.orchestra.run.smithi060.stdout: 413160 24 -rw-r--r-- 1 ubuntu ubuntu 23142 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/erasure-code-profile-form/erasure-code-profile-form-modal.component.spec.ts 2024-03-27T21:42:43.762 INFO:teuthology.orchestra.run.smithi060.stdout: 413159 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/erasure-code-profile-form/erasure-code-profile-form-modal.component.scss 2024-03-27T21:42:43.762 INFO:teuthology.orchestra.run.smithi060.stdout: 413158 20 -rw-r--r-- 1 ubuntu ubuntu 16589 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/erasure-code-profile-form/erasure-code-profile-form-modal.component.html 2024-03-27T21:42:43.762 INFO:teuthology.orchestra.run.smithi060.stdout: 413161 16 -rw-r--r-- 1 ubuntu ubuntu 14256 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/erasure-code-profile-form/erasure-code-profile-form-modal.component.ts 2024-03-27T21:42:43.763 INFO:teuthology.orchestra.run.smithi060.stdout: 413180 4 -rw-r--r-- 1 ubuntu ubuntu 1931 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool.ts 2024-03-27T21:42:43.763 INFO:teuthology.orchestra.run.smithi060.stdout: 413179 4 -rw-r--r-- 1 ubuntu ubuntu 1836 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool.module.ts 2024-03-27T21:42:43.763 INFO:teuthology.orchestra.run.smithi060.stdout: 413173 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-list 2024-03-27T21:42:43.763 INFO:teuthology.orchestra.run.smithi060.stdout: 413177 12 -rw-r--r-- 1 ubuntu ubuntu 11017 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-list/pool-list.component.ts 2024-03-27T21:42:43.763 INFO:teuthology.orchestra.run.smithi060.stdout: 413175 4 -rw-r--r-- 1 ubuntu ubuntu 248 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-list/pool-list.component.scss 2024-03-27T21:42:43.763 INFO:teuthology.orchestra.run.smithi060.stdout: 413176 20 -rw-r--r-- 1 ubuntu ubuntu 17495 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-list/pool-list.component.spec.ts 2024-03-27T21:42:43.763 INFO:teuthology.orchestra.run.smithi060.stdout: 413174 4 -rw-r--r-- 1 ubuntu ubuntu 2055 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-list/pool-list.component.html 2024-03-27T21:42:43.763 INFO:teuthology.orchestra.run.smithi060.stdout: 413167 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-form 2024-03-27T21:42:43.763 INFO:teuthology.orchestra.run.smithi060.stdout: 413169 28 -rw-r--r-- 1 ubuntu ubuntu 26660 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-form/pool-form.component.html 2024-03-27T21:42:43.764 INFO:teuthology.orchestra.run.smithi060.stdout: 413168 4 -rw-r--r-- 1 ubuntu ubuntu 1179 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-form/pool-form-data.ts 2024-03-27T21:42:43.764 INFO:teuthology.orchestra.run.smithi060.stdout: 413171 48 -rw-r--r-- 1 ubuntu ubuntu 48474 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-form/pool-form.component.spec.ts 2024-03-27T21:42:43.764 INFO:teuthology.orchestra.run.smithi060.stdout: 413170 4 -rw-r--r-- 1 ubuntu ubuntu 44 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-form/pool-form.component.scss 2024-03-27T21:42:43.764 INFO:teuthology.orchestra.run.smithi060.stdout: 413172 32 -rw-r--r-- 1 ubuntu ubuntu 29503 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-form/pool-form.component.ts 2024-03-27T21:42:43.764 INFO:teuthology.orchestra.run.smithi060.stdout: 413178 4 -rw-r--r-- 1 ubuntu ubuntu 287 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-stat.ts 2024-03-27T21:42:43.764 INFO:teuthology.orchestra.run.smithi060.stdout: 413152 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/crush-rule-form-modal 2024-03-27T21:42:43.764 INFO:teuthology.orchestra.run.smithi060.stdout: 413153 8 -rw-r--r-- 1 ubuntu ubuntu 4708 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/crush-rule-form-modal/crush-rule-form-modal.component.html 2024-03-27T21:42:43.764 INFO:teuthology.orchestra.run.smithi060.stdout: 413154 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/crush-rule-form-modal/crush-rule-form-modal.component.scss 2024-03-27T21:42:43.765 INFO:teuthology.orchestra.run.smithi060.stdout: 413156 4 -rw-r--r-- 1 ubuntu ubuntu 3338 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/crush-rule-form-modal/crush-rule-form-modal.component.ts 2024-03-27T21:42:43.765 INFO:teuthology.orchestra.run.smithi060.stdout: 413155 8 -rw-r--r-- 1 ubuntu ubuntu 7849 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/crush-rule-form-modal/crush-rule-form-modal.component.spec.ts 2024-03-27T21:42:43.765 INFO:teuthology.orchestra.run.smithi060.stdout: 412721 4 -rw-r--r-- 1 ubuntu ubuntu 681 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/ceph.module.ts 2024-03-27T21:42:43.765 INFO:teuthology.orchestra.run.smithi060.stdout: 412809 4 drwxr-xr-x 16 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster 2024-03-27T21:42:43.765 INFO:teuthology.orchestra.run.smithi060.stdout: 412894 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/monitor 2024-03-27T21:42:43.765 INFO:teuthology.orchestra.run.smithi060.stdout: 412895 4 -rw-r--r-- 1 ubuntu ubuntu 1837 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/monitor/monitor.component.html 2024-03-27T21:42:43.765 INFO:teuthology.orchestra.run.smithi060.stdout: 412898 4 -rw-r--r-- 1 ubuntu ubuntu 2232 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/monitor/monitor.component.ts 2024-03-27T21:42:43.765 INFO:teuthology.orchestra.run.smithi060.stdout: 412897 4 -rw-r--r-- 1 ubuntu ubuntu 2917 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/monitor/monitor.component.spec.ts 2024-03-27T21:42:43.766 INFO:teuthology.orchestra.run.smithi060.stdout: 412896 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/monitor/monitor.component.scss 2024-03-27T21:42:43.766 INFO:teuthology.orchestra.run.smithi060.stdout: 412810 8 -rw-r--r-- 1 ubuntu ubuntu 6569 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/cluster.module.ts 2024-03-27T21:42:43.766 INFO:teuthology.orchestra.run.smithi060.stdout: 412836 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/crushmap 2024-03-27T21:42:43.766 INFO:teuthology.orchestra.run.smithi060.stdout: 412839 8 -rw-r--r-- 1 ubuntu ubuntu 4440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/crushmap/crushmap.component.spec.ts 2024-03-27T21:42:43.766 INFO:teuthology.orchestra.run.smithi060.stdout: 412837 4 -rw-r--r-- 1 ubuntu ubuntu 1534 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/crushmap/crushmap.component.html 2024-03-27T21:42:43.766 INFO:teuthology.orchestra.run.smithi060.stdout: 412838 4 -rw-r--r-- 1 ubuntu ubuntu 51 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/crushmap/crushmap.component.scss 2024-03-27T21:42:43.766 INFO:teuthology.orchestra.run.smithi060.stdout: 412840 4 -rw-r--r-- 1 ubuntu ubuntu 3080 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/crushmap/crushmap.component.ts 2024-03-27T21:42:43.766 INFO:teuthology.orchestra.run.smithi060.stdout: 412827 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/create-cluster 2024-03-27T21:42:43.767 INFO:teuthology.orchestra.run.smithi060.stdout: 412833 4 -rw-r--r-- 1 ubuntu ubuntu 267 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/create-cluster/create-cluster.component.scss 2024-03-27T21:42:43.767 INFO:teuthology.orchestra.run.smithi060.stdout: 412830 4 -rw-r--r-- 1 ubuntu ubuntu 1024 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/create-cluster/create-cluster-review.component.spec.ts 2024-03-27T21:42:43.767 INFO:teuthology.orchestra.run.smithi060.stdout: 412828 4 -rw-r--r-- 1 ubuntu ubuntu 1314 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/create-cluster/create-cluster-review.component.html 2024-03-27T21:42:43.767 INFO:teuthology.orchestra.run.smithi060.stdout: 412832 4 -rw-r--r-- 1 ubuntu ubuntu 3801 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/create-cluster/create-cluster.component.html 2024-03-27T21:42:43.767 INFO:teuthology.orchestra.run.smithi060.stdout: 412834 8 -rw-r--r-- 1 ubuntu ubuntu 6659 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/create-cluster/create-cluster.component.spec.ts 2024-03-27T21:42:43.767 INFO:teuthology.orchestra.run.smithi060.stdout: 412831 4 -rw-r--r-- 1 ubuntu ubuntu 2736 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/create-cluster/create-cluster-review.component.ts 2024-03-27T21:42:43.767 INFO:teuthology.orchestra.run.smithi060.stdout: 412829 4 -rw-r--r-- 1 ubuntu ubuntu 55 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/create-cluster/create-cluster-review.component.scss 2024-03-27T21:42:43.767 INFO:teuthology.orchestra.run.smithi060.stdout: 412835 12 -rw-r--r-- 1 ubuntu ubuntu 8323 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/create-cluster/create-cluster.component.ts 2024-03-27T21:42:43.768 INFO:teuthology.orchestra.run.smithi060.stdout: 412899 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster 2024-03-27T21:42:43.768 INFO:teuthology.orchestra.run.smithi060.stdout: 412913 16 -rw-r--r-- 1 ubuntu ubuntu 14062 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster.component.ts 2024-03-27T21:42:43.768 INFO:teuthology.orchestra.run.smithi060.stdout: 412911 4 -rw-r--r-- 1 ubuntu ubuntu 124 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster.component.scss 2024-03-27T21:42:43.768 INFO:teuthology.orchestra.run.smithi060.stdout: 412900 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-form 2024-03-27T21:42:43.768 INFO:teuthology.orchestra.run.smithi060.stdout: 412903 4 -rw-r--r-- 1 ubuntu ubuntu 1452 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-form/multi-cluster-form.component.spec.ts 2024-03-27T21:42:43.768 INFO:teuthology.orchestra.run.smithi060.stdout: 412902 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-form/multi-cluster-form.component.scss 2024-03-27T21:42:43.768 INFO:teuthology.orchestra.run.smithi060.stdout: 412904 12 -rw-r--r-- 1 ubuntu ubuntu 11018 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-form/multi-cluster-form.component.ts 2024-03-27T21:42:43.768 INFO:teuthology.orchestra.run.smithi060.stdout: 412901 12 -rw-r--r-- 1 ubuntu ubuntu 11739 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-form/multi-cluster-form.component.html 2024-03-27T21:42:43.769 INFO:teuthology.orchestra.run.smithi060.stdout: 412910 12 -rw-r--r-- 1 ubuntu ubuntu 10703 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster.component.html 2024-03-27T21:42:43.769 INFO:teuthology.orchestra.run.smithi060.stdout: 412905 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-list 2024-03-27T21:42:43.769 INFO:teuthology.orchestra.run.smithi060.stdout: 412907 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-list/multi-cluster-list.component.scss 2024-03-27T21:42:43.769 INFO:teuthology.orchestra.run.smithi060.stdout: 412906 4 -rw-r--r-- 1 ubuntu ubuntu 1099 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-list/multi-cluster-list.component.html 2024-03-27T21:42:43.769 INFO:teuthology.orchestra.run.smithi060.stdout: 412908 4 -rw-r--r-- 1 ubuntu ubuntu 1209 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-list/multi-cluster-list.component.spec.ts 2024-03-27T21:42:43.769 INFO:teuthology.orchestra.run.smithi060.stdout: 412909 8 -rw-r--r-- 1 ubuntu ubuntu 8041 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-list/multi-cluster-list.component.ts 2024-03-27T21:42:43.769 INFO:teuthology.orchestra.run.smithi060.stdout: 412912 4 -rw-r--r-- 1 ubuntu ubuntu 1012 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster.component.spec.ts 2024-03-27T21:42:43.770 INFO:teuthology.orchestra.run.smithi060.stdout: 412811 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration 2024-03-27T21:42:43.770 INFO:teuthology.orchestra.run.smithi060.stdout: 412812 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-details 2024-03-27T21:42:43.770 INFO:teuthology.orchestra.run.smithi060.stdout: 412813 4 -rwxr-xr-x 1 ubuntu ubuntu 2727 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-details/configuration-details.component.html 2024-03-27T21:42:43.770 INFO:teuthology.orchestra.run.smithi060.stdout: 412814 0 -rwxr-xr-x 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-details/configuration-details.component.scss 2024-03-27T21:42:43.770 INFO:teuthology.orchestra.run.smithi060.stdout: 412815 4 -rwxr-xr-x 1 ubuntu ubuntu 882 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-details/configuration-details.component.spec.ts 2024-03-27T21:42:43.770 INFO:teuthology.orchestra.run.smithi060.stdout: 412816 4 -rwxr-xr-x 1 ubuntu ubuntu 1014 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-details/configuration-details.component.ts 2024-03-27T21:42:43.770 INFO:teuthology.orchestra.run.smithi060.stdout: 412826 8 -rw-r--r-- 1 ubuntu ubuntu 4498 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration.component.ts 2024-03-27T21:42:43.770 INFO:teuthology.orchestra.run.smithi060.stdout: 412817 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form 2024-03-27T21:42:43.771 INFO:teuthology.orchestra.run.smithi060.stdout: 412821 4 -rw-r--r-- 1 ubuntu ubuntu 3194 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.spec.ts 2024-03-27T21:42:43.771 INFO:teuthology.orchestra.run.smithi060.stdout: 412822 8 -rw-r--r-- 1 ubuntu ubuntu 5785 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.ts 2024-03-27T21:42:43.771 INFO:teuthology.orchestra.run.smithi060.stdout: 412819 8 -rw-r--r-- 1 ubuntu ubuntu 5920 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.html 2024-03-27T21:42:43.771 INFO:teuthology.orchestra.run.smithi060.stdout: 412818 4 -rw-r--r-- 1 ubuntu ubuntu 88 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form-create-request.model.ts 2024-03-27T21:42:43.771 INFO:teuthology.orchestra.run.smithi060.stdout: 412820 4 -rw-r--r-- 1 ubuntu ubuntu 137 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.scss 2024-03-27T21:42:43.771 INFO:teuthology.orchestra.run.smithi060.stdout: 412825 4 -rw-r--r-- 1 ubuntu ubuntu 1619 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration.component.spec.ts 2024-03-27T21:42:43.771 INFO:teuthology.orchestra.run.smithi060.stdout: 412823 4 -rw-r--r-- 1 ubuntu ubuntu 911 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration.component.html 2024-03-27T21:42:43.771 INFO:teuthology.orchestra.run.smithi060.stdout: 412824 4 -rw-r--r-- 1 ubuntu ubuntu 218 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration.component.scss 2024-03-27T21:42:43.772 INFO:teuthology.orchestra.run.smithi060.stdout: 413040 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/upgrade 2024-03-27T21:42:43.772 INFO:teuthology.orchestra.run.smithi060.stdout: 413052 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/upgrade/upgrade.component.scss 2024-03-27T21:42:43.772 INFO:teuthology.orchestra.run.smithi060.stdout: 413054 8 -rw-r--r-- 1 ubuntu ubuntu 4775 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/upgrade/upgrade.component.ts 2024-03-27T21:42:43.772 INFO:teuthology.orchestra.run.smithi060.stdout: 413041 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/upgrade/upgrade-form 2024-03-27T21:42:43.772 INFO:teuthology.orchestra.run.smithi060.stdout: 413045 4 -rw-r--r-- 1 ubuntu ubuntu 3623 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/upgrade/upgrade-form/upgrade-start-modal.component.ts 2024-03-27T21:42:43.772 INFO:teuthology.orchestra.run.smithi060.stdout: 413042 4 -rw-r--r-- 1 ubuntu ubuntu 3594 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/upgrade/upgrade-form/upgrade-start-modal.component.html 2024-03-27T21:42:43.772 INFO:teuthology.orchestra.run.smithi060.stdout: 413044 4 -rw-r--r-- 1 ubuntu ubuntu 1130 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/upgrade/upgrade-form/upgrade-start-modal.component.spec.ts 2024-03-27T21:42:43.772 INFO:teuthology.orchestra.run.smithi060.stdout: 413043 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/upgrade/upgrade-form/upgrade-start-modal.component.scss 2024-03-27T21:42:43.773 INFO:teuthology.orchestra.run.smithi060.stdout: 413046 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/upgrade/upgrade-progress 2024-03-27T21:42:43.773 INFO:teuthology.orchestra.run.smithi060.stdout: 413050 8 -rw-r--r-- 1 ubuntu ubuntu 4593 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/upgrade/upgrade-progress/upgrade-progress.component.ts 2024-03-27T21:42:43.773 INFO:teuthology.orchestra.run.smithi060.stdout: 413047 4 -rw-r--r-- 1 ubuntu ubuntu 2890 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/upgrade/upgrade-progress/upgrade-progress.component.html 2024-03-27T21:42:43.773 INFO:teuthology.orchestra.run.smithi060.stdout: 413049 4 -rw-r--r-- 1 ubuntu ubuntu 1077 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/upgrade/upgrade-progress/upgrade-progress.component.spec.ts 2024-03-27T21:42:43.773 INFO:teuthology.orchestra.run.smithi060.stdout: 413048 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/upgrade/upgrade-progress/upgrade-progress.component.scss 2024-03-27T21:42:43.773 INFO:teuthology.orchestra.run.smithi060.stdout: 413051 12 -rw-r--r-- 1 ubuntu ubuntu 8218 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/upgrade/upgrade.component.html 2024-03-27T21:42:43.773 INFO:teuthology.orchestra.run.smithi060.stdout: 413053 8 -rw-r--r-- 1 ubuntu ubuntu 8110 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/upgrade/upgrade.component.spec.ts 2024-03-27T21:42:43.773 INFO:teuthology.orchestra.run.smithi060.stdout: 412877 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/mgr-modules 2024-03-27T21:42:43.774 INFO:teuthology.orchestra.run.smithi060.stdout: 412883 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/mgr-modules/mgr-module-form 2024-03-27T21:42:43.774 INFO:teuthology.orchestra.run.smithi060.stdout: 412886 4 -rw-r--r-- 1 ubuntu ubuntu 2660 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/mgr-modules/mgr-module-form/mgr-module-form.component.spec.ts 2024-03-27T21:42:43.774 INFO:teuthology.orchestra.run.smithi060.stdout: 412884 8 -rw-r--r-- 1 ubuntu ubuntu 5185 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/mgr-modules/mgr-module-form/mgr-module-form.component.html 2024-03-27T21:42:43.774 INFO:teuthology.orchestra.run.smithi060.stdout: 412887 8 -rw-r--r-- 1 ubuntu ubuntu 4162 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/mgr-modules/mgr-module-form/mgr-module-form.component.ts 2024-03-27T21:42:43.774 INFO:teuthology.orchestra.run.smithi060.stdout: 412885 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/mgr-modules/mgr-module-form/mgr-module-form.component.scss 2024-03-27T21:42:43.774 INFO:teuthology.orchestra.run.smithi060.stdout: 412888 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/mgr-modules/mgr-module-list 2024-03-27T21:42:43.774 INFO:teuthology.orchestra.run.smithi060.stdout: 412892 8 -rw-r--r-- 1 ubuntu ubuntu 6249 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/mgr-modules/mgr-module-list/mgr-module-list.component.ts 2024-03-27T21:42:43.775 INFO:teuthology.orchestra.run.smithi060.stdout: 412891 8 -rw-r--r-- 1 ubuntu ubuntu 6020 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/mgr-modules/mgr-module-list/mgr-module-list.component.spec.ts 2024-03-27T21:42:43.775 INFO:teuthology.orchestra.run.smithi060.stdout: 412890 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/mgr-modules/mgr-module-list/mgr-module-list.component.scss 2024-03-27T21:42:43.775 INFO:teuthology.orchestra.run.smithi060.stdout: 412889 4 -rw-r--r-- 1 ubuntu ubuntu 714 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/mgr-modules/mgr-module-list/mgr-module-list.component.html 2024-03-27T21:42:43.775 INFO:teuthology.orchestra.run.smithi060.stdout: 412878 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/mgr-modules/mgr-module-details 2024-03-27T21:42:43.775 INFO:teuthology.orchestra.run.smithi060.stdout: 412880 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/mgr-modules/mgr-module-details/mgr-module-details.component.scss 2024-03-27T21:42:43.775 INFO:teuthology.orchestra.run.smithi060.stdout: 412882 4 -rw-r--r-- 1 ubuntu ubuntu 659 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/mgr-modules/mgr-module-details/mgr-module-details.component.ts 2024-03-27T21:42:43.775 INFO:teuthology.orchestra.run.smithi060.stdout: 412881 4 -rw-r--r-- 1 ubuntu ubuntu 897 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/mgr-modules/mgr-module-details/mgr-module-details.component.spec.ts 2024-03-27T21:42:43.775 INFO:teuthology.orchestra.run.smithi060.stdout: 412879 4 -rw-r--r-- 1 ubuntu ubuntu 119 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/mgr-modules/mgr-module-details/mgr-module-details.component.html 2024-03-27T21:42:43.776 INFO:teuthology.orchestra.run.smithi060.stdout: 412893 4 -rw-r--r-- 1 ubuntu ubuntu 827 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/mgr-modules/mgr-modules.module.ts 2024-03-27T21:42:43.776 INFO:teuthology.orchestra.run.smithi060.stdout: 412914 4 drwxr-xr-x 14 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd 2024-03-27T21:42:43.776 INFO:teuthology.orchestra.run.smithi060.stdout: 412932 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-devices-selection-modal 2024-03-27T21:42:43.776 INFO:teuthology.orchestra.run.smithi060.stdout: 412935 4 -rw-r--r-- 1 ubuntu ubuntu 3638 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-devices-selection-modal/osd-devices-selection-modal.component.spec.ts 2024-03-27T21:42:43.776 INFO:teuthology.orchestra.run.smithi060.stdout: 412936 4 -rw-r--r-- 1 ubuntu ubuntu 3243 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-devices-selection-modal/osd-devices-selection-modal.component.ts 2024-03-27T21:42:43.776 INFO:teuthology.orchestra.run.smithi060.stdout: 412934 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-devices-selection-modal/osd-devices-selection-modal.component.scss 2024-03-27T21:42:43.776 INFO:teuthology.orchestra.run.smithi060.stdout: 412933 4 -rw-r--r-- 1 ubuntu ubuntu 1791 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-devices-selection-modal/osd-devices-selection-modal.component.html 2024-03-27T21:42:43.776 INFO:teuthology.orchestra.run.smithi060.stdout: 412920 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-details 2024-03-27T21:42:43.777 INFO:teuthology.orchestra.run.smithi060.stdout: 412924 4 -rw-r--r-- 1 ubuntu ubuntu 1094 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-details/osd-details.component.ts 2024-03-27T21:42:43.777 INFO:teuthology.orchestra.run.smithi060.stdout: 412923 4 -rw-r--r-- 1 ubuntu ubuntu 1198 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-details/osd-details.component.spec.ts 2024-03-27T21:42:43.777 INFO:teuthology.orchestra.run.smithi060.stdout: 412922 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-details/osd-details.component.scss 2024-03-27T21:42:43.777 INFO:teuthology.orchestra.run.smithi060.stdout: 412921 4 -rw-r--r-- 1 ubuntu ubuntu 2524 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-details/osd-details.component.html 2024-03-27T21:42:43.777 INFO:teuthology.orchestra.run.smithi060.stdout: 412925 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-devices-selection-groups 2024-03-27T21:42:43.777 INFO:teuthology.orchestra.run.smithi060.stdout: 412927 4 -rw-r--r-- 1 ubuntu ubuntu 207 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-devices-selection-groups/devices-selection-clear-event.interface.ts 2024-03-27T21:42:43.777 INFO:teuthology.orchestra.run.smithi060.stdout: 412929 4 -rw-r--r-- 1 ubuntu ubuntu 49 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-devices-selection-groups/osd-devices-selection-groups.component.scss 2024-03-27T21:42:43.777 INFO:teuthology.orchestra.run.smithi060.stdout: 412926 4 -rw-r--r-- 1 ubuntu ubuntu 198 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-devices-selection-groups/devices-selection-change-event.interface.ts 2024-03-27T21:42:43.777 INFO:teuthology.orchestra.run.smithi060.stdout: 412928 4 -rw-r--r-- 1 ubuntu ubuntu 2364 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-devices-selection-groups/osd-devices-selection-groups.component.html 2024-03-27T21:42:43.778 INFO:teuthology.orchestra.run.smithi060.stdout: 412930 8 -rw-r--r-- 1 ubuntu ubuntu 4189 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-devices-selection-groups/osd-devices-selection-groups.component.spec.ts 2024-03-27T21:42:43.778 INFO:teuthology.orchestra.run.smithi060.stdout: 412931 8 -rw-r--r-- 1 ubuntu ubuntu 4575 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-devices-selection-groups/osd-devices-selection-groups.component.ts 2024-03-27T21:42:43.778 INFO:teuthology.orchestra.run.smithi060.stdout: 412972 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-reweight-modal 2024-03-27T21:42:43.778 INFO:teuthology.orchestra.run.smithi060.stdout: 412975 4 -rw-r--r-- 1 ubuntu ubuntu 2078 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-reweight-modal/osd-reweight-modal.component.spec.ts 2024-03-27T21:42:43.778 INFO:teuthology.orchestra.run.smithi060.stdout: 412973 4 -rw-r--r-- 1 ubuntu ubuntu 1367 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-reweight-modal/osd-reweight-modal.component.html 2024-03-27T21:42:43.778 INFO:teuthology.orchestra.run.smithi060.stdout: 412974 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-reweight-modal/osd-reweight-modal.component.scss 2024-03-27T21:42:43.778 INFO:teuthology.orchestra.run.smithi060.stdout: 412976 4 -rw-r--r-- 1 ubuntu ubuntu 1241 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-reweight-modal/osd-reweight-modal.component.ts 2024-03-27T21:42:43.778 INFO:teuthology.orchestra.run.smithi060.stdout: 412961 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-pg-scrub-modal 2024-03-27T21:42:43.779 INFO:teuthology.orchestra.run.smithi060.stdout: 412964 4 -rw-r--r-- 1 ubuntu ubuntu 2254 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-pg-scrub-modal/osd-pg-scrub-modal.component.spec.ts 2024-03-27T21:42:43.779 INFO:teuthology.orchestra.run.smithi060.stdout: 412963 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-pg-scrub-modal/osd-pg-scrub-modal.component.scss 2024-03-27T21:42:43.779 INFO:teuthology.orchestra.run.smithi060.stdout: 412965 4 -rw-r--r-- 1 ubuntu ubuntu 2359 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-pg-scrub-modal/osd-pg-scrub-modal.component.ts 2024-03-27T21:42:43.779 INFO:teuthology.orchestra.run.smithi060.stdout: 412962 4 -rw-r--r-- 1 ubuntu ubuntu 1873 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-pg-scrub-modal/osd-pg-scrub-modal.component.html 2024-03-27T21:42:43.779 INFO:teuthology.orchestra.run.smithi060.stdout: 412966 4 -rw-r--r-- 1 ubuntu ubuntu 1165 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-pg-scrub-modal/osd-pg-scrub-modal.options.ts 2024-03-27T21:42:43.779 INFO:teuthology.orchestra.run.smithi060.stdout: 412915 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-creation-preview-modal 2024-03-27T21:42:43.779 INFO:teuthology.orchestra.run.smithi060.stdout: 412916 4 -rw-r--r-- 1 ubuntu ubuntu 658 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-creation-preview-modal/osd-creation-preview-modal.component.html 2024-03-27T21:42:43.780 INFO:teuthology.orchestra.run.smithi060.stdout: 412917 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-creation-preview-modal/osd-creation-preview-modal.component.scss 2024-03-27T21:42:43.780 INFO:teuthology.orchestra.run.smithi060.stdout: 412918 4 -rw-r--r-- 1 ubuntu ubuntu 1267 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-creation-preview-modal/osd-creation-preview-modal.component.spec.ts 2024-03-27T21:42:43.780 INFO:teuthology.orchestra.run.smithi060.stdout: 412919 4 -rw-r--r-- 1 ubuntu ubuntu 1872 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-creation-preview-modal/osd-creation-preview-modal.component.ts 2024-03-27T21:42:43.780 INFO:teuthology.orchestra.run.smithi060.stdout: 412977 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-scrub-modal 2024-03-27T21:42:43.780 INFO:teuthology.orchestra.run.smithi060.stdout: 412979 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-scrub-modal/osd-scrub-modal.component.scss 2024-03-27T21:42:43.780 INFO:teuthology.orchestra.run.smithi060.stdout: 412981 4 -rw-r--r-- 1 ubuntu ubuntu 1625 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-scrub-modal/osd-scrub-modal.component.ts 2024-03-27T21:42:43.780 INFO:teuthology.orchestra.run.smithi060.stdout: 412978 4 -rw-r--r-- 1 ubuntu ubuntu 767 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-scrub-modal/osd-scrub-modal.component.html 2024-03-27T21:42:43.780 INFO:teuthology.orchestra.run.smithi060.stdout: 412980 4 -rw-r--r-- 1 ubuntu ubuntu 1515 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-scrub-modal/osd-scrub-modal.component.spec.ts 2024-03-27T21:42:43.780 INFO:teuthology.orchestra.run.smithi060.stdout: 412937 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-flags-indiv-modal 2024-03-27T21:42:43.781 INFO:teuthology.orchestra.run.smithi060.stdout: 412940 16 -rw-r--r-- 1 ubuntu ubuntu 13627 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-flags-indiv-modal/osd-flags-indiv-modal.component.spec.ts 2024-03-27T21:42:43.781 INFO:teuthology.orchestra.run.smithi060.stdout: 412938 4 -rw-r--r-- 1 ubuntu ubuntu 1810 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-flags-indiv-modal/osd-flags-indiv-modal.component.html 2024-03-27T21:42:43.781 INFO:teuthology.orchestra.run.smithi060.stdout: 412939 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-flags-indiv-modal/osd-flags-indiv-modal.component.scss 2024-03-27T21:42:43.781 INFO:teuthology.orchestra.run.smithi060.stdout: 412941 8 -rw-r--r-- 1 ubuntu ubuntu 4220 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-flags-indiv-modal/osd-flags-indiv-modal.component.ts 2024-03-27T21:42:43.781 INFO:teuthology.orchestra.run.smithi060.stdout: 412947 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-form 2024-03-27T21:42:43.781 INFO:teuthology.orchestra.run.smithi060.stdout: 412953 12 -rw-r--r-- 1 ubuntu ubuntu 9870 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-form/osd-form.component.ts 2024-03-27T21:42:43.781 INFO:teuthology.orchestra.run.smithi060.stdout: 412949 4 -rw-r--r-- 1 ubuntu ubuntu 64 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-form/osd-feature.interface.ts 2024-03-27T21:42:43.781 INFO:teuthology.orchestra.run.smithi060.stdout: 412948 4 -rw-r--r-- 1 ubuntu ubuntu 2320 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-form/drive-group.model.ts 2024-03-27T21:42:43.782 INFO:teuthology.orchestra.run.smithi060.stdout: 412951 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-form/osd-form.component.scss 2024-03-27T21:42:43.782 INFO:teuthology.orchestra.run.smithi060.stdout: 412952 12 -rw-r--r-- 1 ubuntu ubuntu 10853 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-form/osd-form.component.spec.ts 2024-03-27T21:42:43.782 INFO:teuthology.orchestra.run.smithi060.stdout: 412950 12 -rw-r--r-- 1 ubuntu ubuntu 10052 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-form/osd-form.component.html 2024-03-27T21:42:43.782 INFO:teuthology.orchestra.run.smithi060.stdout: 412954 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-list 2024-03-27T21:42:43.782 INFO:teuthology.orchestra.run.smithi060.stdout: 412957 8 -rw-r--r-- 1 ubuntu ubuntu 5469 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-list/osd-list.component.html 2024-03-27T21:42:43.782 INFO:teuthology.orchestra.run.smithi060.stdout: 412960 24 -rw-r--r-- 1 ubuntu ubuntu 21481 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-list/osd-list.component.ts 2024-03-27T21:42:43.782 INFO:teuthology.orchestra.run.smithi060.stdout: 412955 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-list/fixtures 2024-03-27T21:42:43.782 INFO:teuthology.orchestra.run.smithi060.stdout: 412956 20 -rw-r--r-- 1 ubuntu ubuntu 17788 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-list/fixtures/osd_list_response.json 2024-03-27T21:42:43.783 INFO:teuthology.orchestra.run.smithi060.stdout: 412958 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-list/osd-list.component.scss 2024-03-27T21:42:43.783 INFO:teuthology.orchestra.run.smithi060.stdout: 412959 24 -rw-r--r-- 1 ubuntu ubuntu 21454 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-list/osd-list.component.spec.ts 2024-03-27T21:42:43.783 INFO:teuthology.orchestra.run.smithi060.stdout: 412942 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-flags-modal 2024-03-27T21:42:43.783 INFO:teuthology.orchestra.run.smithi060.stdout: 412944 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-flags-modal/osd-flags-modal.component.scss 2024-03-27T21:42:43.783 INFO:teuthology.orchestra.run.smithi060.stdout: 412943 4 -rw-r--r-- 1 ubuntu ubuntu 1484 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-flags-modal/osd-flags-modal.component.html 2024-03-27T21:42:43.783 INFO:teuthology.orchestra.run.smithi060.stdout: 412946 8 -rw-r--r-- 1 ubuntu ubuntu 4611 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-flags-modal/osd-flags-modal.component.ts 2024-03-27T21:42:43.783 INFO:teuthology.orchestra.run.smithi060.stdout: 412945 4 -rw-r--r-- 1 ubuntu ubuntu 3403 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-flags-modal/osd-flags-modal.component.spec.ts 2024-03-27T21:42:43.783 INFO:teuthology.orchestra.run.smithi060.stdout: 412967 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-recv-speed-modal 2024-03-27T21:42:43.784 INFO:teuthology.orchestra.run.smithi060.stdout: 412969 0 -rwxr-xr-x 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-recv-speed-modal/osd-recv-speed-modal.component.scss 2024-03-27T21:42:43.784 INFO:teuthology.orchestra.run.smithi060.stdout: 412971 8 -rwxr-xr-x 1 ubuntu ubuntu 7939 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-recv-speed-modal/osd-recv-speed-modal.component.ts 2024-03-27T21:42:43.784 INFO:teuthology.orchestra.run.smithi060.stdout: 412970 12 -rwxr-xr-x 1 ubuntu ubuntu 11202 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-recv-speed-modal/osd-recv-speed-modal.component.spec.ts 2024-03-27T21:42:43.784 INFO:teuthology.orchestra.run.smithi060.stdout: 412968 4 -rwxr-xr-x 1 ubuntu ubuntu 4085 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-recv-speed-modal/osd-recv-speed-modal.component.html 2024-03-27T21:42:43.784 INFO:teuthology.orchestra.run.smithi060.stdout: 412872 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/logs 2024-03-27T21:42:43.784 INFO:teuthology.orchestra.run.smithi060.stdout: 412873 8 -rw-r--r-- 1 ubuntu ubuntu 6755 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/logs/logs.component.html 2024-03-27T21:42:43.784 INFO:teuthology.orchestra.run.smithi060.stdout: 412876 8 -rw-r--r-- 1 ubuntu ubuntu 5303 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/logs/logs.component.ts 2024-03-27T21:42:43.785 INFO:teuthology.orchestra.run.smithi060.stdout: 412875 8 -rw-r--r-- 1 ubuntu ubuntu 5192 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/logs/logs.component.spec.ts 2024-03-27T21:42:43.785 INFO:teuthology.orchestra.run.smithi060.stdout: 412874 4 -rw-r--r-- 1 ubuntu ubuntu 684 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/logs/logs.component.scss 2024-03-27T21:42:43.785 INFO:teuthology.orchestra.run.smithi060.stdout: 413013 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services 2024-03-27T21:42:43.785 INFO:teuthology.orchestra.run.smithi060.stdout: 413021 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-details 2024-03-27T21:42:43.785 INFO:teuthology.orchestra.run.smithi060.stdout: 413023 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-details/service-details.component.scss 2024-03-27T21:42:43.785 INFO:teuthology.orchestra.run.smithi060.stdout: 413022 4 -rw-r--r-- 1 ubuntu ubuntu 146 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-details/service-details.component.html 2024-03-27T21:42:43.785 INFO:teuthology.orchestra.run.smithi060.stdout: 413025 4 -rw-r--r-- 1 ubuntu ubuntu 464 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-details/service-details.component.ts 2024-03-27T21:42:43.785 INFO:teuthology.orchestra.run.smithi060.stdout: 413024 4 -rw-r--r-- 1 ubuntu ubuntu 1604 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-details/service-details.component.spec.ts 2024-03-27T21:42:43.785 INFO:teuthology.orchestra.run.smithi060.stdout: 413034 12 -rw-r--r-- 1 ubuntu ubuntu 8692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/services.component.ts 2024-03-27T21:42:43.786 INFO:teuthology.orchestra.run.smithi060.stdout: 413032 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/services.component.scss 2024-03-27T21:42:43.786 INFO:teuthology.orchestra.run.smithi060.stdout: 413014 4 -rw-r--r-- 1 ubuntu ubuntu 1507 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/placement.pipe.spec.ts 2024-03-27T21:42:43.786 INFO:teuthology.orchestra.run.smithi060.stdout: 413015 4 -rw-r--r-- 1 ubuntu ubuntu 1289 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/placement.pipe.ts 2024-03-27T21:42:43.786 INFO:teuthology.orchestra.run.smithi060.stdout: 413026 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-form 2024-03-27T21:42:43.786 INFO:teuthology.orchestra.run.smithi060.stdout: 413028 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-form/service-form.component.scss 2024-03-27T21:42:43.786 INFO:teuthology.orchestra.run.smithi060.stdout: 413029 20 -rw-r--r-- 1 ubuntu ubuntu 20236 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-form/service-form.component.spec.ts 2024-03-27T21:42:43.786 INFO:teuthology.orchestra.run.smithi060.stdout: 413027 36 -rw-r--r-- 1 ubuntu ubuntu 36863 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-form/service-form.component.html 2024-03-27T21:42:43.786 INFO:teuthology.orchestra.run.smithi060.stdout: 413030 32 -rw-r--r-- 1 ubuntu ubuntu 30773 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-form/service-form.component.ts 2024-03-27T21:42:43.787 INFO:teuthology.orchestra.run.smithi060.stdout: 413033 4 -rw-r--r-- 1 ubuntu ubuntu 3549 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/services.component.spec.ts 2024-03-27T21:42:43.787 INFO:teuthology.orchestra.run.smithi060.stdout: 413031 4 -rw-r--r-- 1 ubuntu ubuntu 1419 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/services.component.html 2024-03-27T21:42:43.787 INFO:teuthology.orchestra.run.smithi060.stdout: 413016 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-daemon-list 2024-03-27T21:42:43.787 INFO:teuthology.orchestra.run.smithi060.stdout: 413020 12 -rw-r--r-- 1 ubuntu ubuntu 9692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-daemon-list/service-daemon-list.component.ts 2024-03-27T21:42:43.787 INFO:teuthology.orchestra.run.smithi060.stdout: 413017 4 -rw-r--r-- 1 ubuntu ubuntu 3122 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-daemon-list/service-daemon-list.component.html 2024-03-27T21:42:43.787 INFO:teuthology.orchestra.run.smithi060.stdout: 413019 12 -rw-r--r-- 1 ubuntu ubuntu 8949 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-daemon-list/service-daemon-list.component.spec.ts 2024-03-27T21:42:43.787 INFO:teuthology.orchestra.run.smithi060.stdout: 413018 4 -rw-r--r-- 1 ubuntu ubuntu 209 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-daemon-list/service-daemon-list.component.scss 2024-03-27T21:42:43.787 INFO:teuthology.orchestra.run.smithi060.stdout: 412858 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/inventory 2024-03-27T21:42:43.788 INFO:teuthology.orchestra.run.smithi060.stdout: 412869 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/inventory/inventory.component.scss 2024-03-27T21:42:43.788 INFO:teuthology.orchestra.run.smithi060.stdout: 412868 4 -rw-r--r-- 1 ubuntu ubuntu 575 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/inventory/inventory.component.html 2024-03-27T21:42:43.788 INFO:teuthology.orchestra.run.smithi060.stdout: 412871 4 -rw-r--r-- 1 ubuntu ubuntu 2691 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/inventory/inventory.component.ts 2024-03-27T21:42:43.788 INFO:teuthology.orchestra.run.smithi060.stdout: 412870 4 -rw-r--r-- 1 ubuntu ubuntu 2490 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/inventory/inventory.component.spec.ts 2024-03-27T21:42:43.788 INFO:teuthology.orchestra.run.smithi060.stdout: 412867 4 -rw-r--r-- 1 ubuntu ubuntu 156 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/inventory/inventory-host.model.ts 2024-03-27T21:42:43.788 INFO:teuthology.orchestra.run.smithi060.stdout: 412859 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/inventory/inventory-devices 2024-03-27T21:42:43.788 INFO:teuthology.orchestra.run.smithi060.stdout: 412864 4 -rw-r--r-- 1 ubuntu ubuntu 136 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/inventory/inventory-devices/inventory-devices.component.scss 2024-03-27T21:42:43.788 INFO:teuthology.orchestra.run.smithi060.stdout: 412862 4 -rw-r--r-- 1 ubuntu ubuntu 358 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/inventory/inventory-devices/inventory-device.model.ts 2024-03-27T21:42:43.788 INFO:teuthology.orchestra.run.smithi060.stdout: 412860 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/inventory/inventory-devices/fixtures 2024-03-27T21:42:43.789 INFO:teuthology.orchestra.run.smithi060.stdout: 412861 12 -rw-r--r-- 1 ubuntu ubuntu 8815 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/inventory/inventory-devices/fixtures/inventory_list_response.json 2024-03-27T21:42:43.789 INFO:teuthology.orchestra.run.smithi060.stdout: 412866 8 -rw-r--r-- 1 ubuntu ubuntu 7928 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/inventory/inventory-devices/inventory-devices.component.ts 2024-03-27T21:42:43.789 INFO:teuthology.orchestra.run.smithi060.stdout: 412865 8 -rw-r--r-- 1 ubuntu ubuntu 6518 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/inventory/inventory-devices/inventory-devices.component.spec.ts 2024-03-27T21:42:43.789 INFO:teuthology.orchestra.run.smithi060.stdout: 412863 4 -rw-r--r-- 1 ubuntu ubuntu 596 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/inventory/inventory-devices/inventory-devices.component.html 2024-03-27T21:42:43.789 INFO:teuthology.orchestra.run.smithi060.stdout: 413035 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/telemetry 2024-03-27T21:42:43.789 INFO:teuthology.orchestra.run.smithi060.stdout: 413037 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/telemetry/telemetry.component.scss 2024-03-27T21:42:43.790 INFO:teuthology.orchestra.run.smithi060.stdout: 413036 16 -rw-r--r-- 1 ubuntu ubuntu 15628 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/telemetry/telemetry.component.html 2024-03-27T21:42:43.790 INFO:teuthology.orchestra.run.smithi060.stdout: 413038 12 -rw-r--r-- 1 ubuntu ubuntu 9011 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/telemetry/telemetry.component.spec.ts 2024-03-27T21:42:43.790 INFO:teuthology.orchestra.run.smithi060.stdout: 413039 12 -rw-r--r-- 1 ubuntu ubuntu 9037 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/telemetry/telemetry.component.ts 2024-03-27T21:42:43.790 INFO:teuthology.orchestra.run.smithi060.stdout: 412982 4 drwxr-xr-x 8 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus 2024-03-27T21:42:43.790 INFO:teuthology.orchestra.run.smithi060.stdout: 413008 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/silence-matcher-modal 2024-03-27T21:42:43.790 INFO:teuthology.orchestra.run.smithi060.stdout: 413012 4 -rw-r--r-- 1 ubuntu ubuntu 3503 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/silence-matcher-modal/silence-matcher-modal.component.ts 2024-03-27T21:42:43.790 INFO:teuthology.orchestra.run.smithi060.stdout: 413009 4 -rw-r--r-- 1 ubuntu ubuntu 3139 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/silence-matcher-modal/silence-matcher-modal.component.html 2024-03-27T21:42:43.790 INFO:teuthology.orchestra.run.smithi060.stdout: 413011 8 -rw-r--r-- 1 ubuntu ubuntu 6262 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/silence-matcher-modal/silence-matcher-modal.component.spec.ts 2024-03-27T21:42:43.791 INFO:teuthology.orchestra.run.smithi060.stdout: 413010 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/silence-matcher-modal/silence-matcher-modal.component.scss 2024-03-27T21:42:43.791 INFO:teuthology.orchestra.run.smithi060.stdout: 412988 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/prometheus-tabs 2024-03-27T21:42:43.791 INFO:teuthology.orchestra.run.smithi060.stdout: 412990 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/prometheus-tabs/prometheus-tabs.component.scss 2024-03-27T21:42:43.791 INFO:teuthology.orchestra.run.smithi060.stdout: 412989 4 -rw-r--r-- 1 ubuntu ubuntu 1087 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/prometheus-tabs/prometheus-tabs.component.html 2024-03-27T21:42:43.791 INFO:teuthology.orchestra.run.smithi060.stdout: 412992 4 -rw-r--r-- 1 ubuntu ubuntu 398 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/prometheus-tabs/prometheus-tabs.component.ts 2024-03-27T21:42:43.791 INFO:teuthology.orchestra.run.smithi060.stdout: 412991 4 -rw-r--r-- 1 ubuntu ubuntu 1002 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/prometheus-tabs/prometheus-tabs.component.spec.ts 2024-03-27T21:42:43.791 INFO:teuthology.orchestra.run.smithi060.stdout: 412998 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/silence-form 2024-03-27T21:42:43.791 INFO:teuthology.orchestra.run.smithi060.stdout: 413002 12 -rw-r--r-- 1 ubuntu ubuntu 11005 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/silence-form/silence-form.component.ts 2024-03-27T21:42:43.792 INFO:teuthology.orchestra.run.smithi060.stdout: 412999 8 -rw-r--r-- 1 ubuntu ubuntu 7499 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/silence-form/silence-form.component.html 2024-03-27T21:42:43.792 INFO:teuthology.orchestra.run.smithi060.stdout: 413001 24 -rw-r--r-- 1 ubuntu ubuntu 21066 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/silence-form/silence-form.component.spec.ts 2024-03-27T21:42:43.792 INFO:teuthology.orchestra.run.smithi060.stdout: 413000 4 -rw-r--r-- 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/silence-form/silence-form.component.scss 2024-03-27T21:42:43.792 INFO:teuthology.orchestra.run.smithi060.stdout: 413003 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/silence-list 2024-03-27T21:42:43.792 INFO:teuthology.orchestra.run.smithi060.stdout: 413004 4 -rw-r--r-- 1 ubuntu ubuntu 1296 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/silence-list/silence-list.component.html 2024-03-27T21:42:43.792 INFO:teuthology.orchestra.run.smithi060.stdout: 413005 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/silence-list/silence-list.component.scss 2024-03-27T21:42:43.792 INFO:teuthology.orchestra.run.smithi060.stdout: 413007 12 -rw-r--r-- 1 ubuntu ubuntu 8253 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/silence-list/silence-list.component.ts 2024-03-27T21:42:43.792 INFO:teuthology.orchestra.run.smithi060.stdout: 413006 8 -rw-r--r-- 1 ubuntu ubuntu 5848 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/silence-list/silence-list.component.spec.ts 2024-03-27T21:42:43.792 INFO:teuthology.orchestra.run.smithi060.stdout: 412983 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/active-alert-list 2024-03-27T21:42:43.793 INFO:teuthology.orchestra.run.smithi060.stdout: 412986 4 -rw-r--r-- 1 ubuntu ubuntu 3389 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/active-alert-list/active-alert-list.component.spec.ts 2024-03-27T21:42:43.793 INFO:teuthology.orchestra.run.smithi060.stdout: 412984 4 -rw-r--r-- 1 ubuntu ubuntu 1501 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/active-alert-list/active-alert-list.component.html 2024-03-27T21:42:43.793 INFO:teuthology.orchestra.run.smithi060.stdout: 412985 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/active-alert-list/active-alert-list.component.scss 2024-03-27T21:42:43.793 INFO:teuthology.orchestra.run.smithi060.stdout: 412987 4 -rw-r--r-- 1 ubuntu ubuntu 3838 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/active-alert-list/active-alert-list.component.ts 2024-03-27T21:42:43.793 INFO:teuthology.orchestra.run.smithi060.stdout: 412993 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/rules-list 2024-03-27T21:42:43.793 INFO:teuthology.orchestra.run.smithi060.stdout: 412996 4 -rw-r--r-- 1 ubuntu ubuntu 1379 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/rules-list/rules-list.component.spec.ts 2024-03-27T21:42:43.793 INFO:teuthology.orchestra.run.smithi060.stdout: 412995 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/rules-list/rules-list.component.scss 2024-03-27T21:42:43.793 INFO:teuthology.orchestra.run.smithi060.stdout: 412997 4 -rw-r--r-- 1 ubuntu ubuntu 2380 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/rules-list/rules-list.component.ts 2024-03-27T21:42:43.794 INFO:teuthology.orchestra.run.smithi060.stdout: 412994 4 -rw-r--r-- 1 ubuntu ubuntu 851 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/rules-list/rules-list.component.html 2024-03-27T21:42:43.794 INFO:teuthology.orchestra.run.smithi060.stdout: 412841 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/hosts 2024-03-27T21:42:43.794 INFO:teuthology.orchestra.run.smithi060.stdout: 412849 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/hosts/host-form 2024-03-27T21:42:43.794 INFO:teuthology.orchestra.run.smithi060.stdout: 412853 8 -rw-r--r-- 1 ubuntu ubuntu 6160 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/hosts/host-form/host-form.component.ts 2024-03-27T21:42:43.794 INFO:teuthology.orchestra.run.smithi060.stdout: 412852 8 -rw-r--r-- 1 ubuntu ubuntu 5259 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/hosts/host-form/host-form.component.spec.ts 2024-03-27T21:42:43.794 INFO:teuthology.orchestra.run.smithi060.stdout: 412851 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/hosts/host-form/host-form.component.scss 2024-03-27T21:42:43.794 INFO:teuthology.orchestra.run.smithi060.stdout: 412850 8 -rw-r--r-- 1 ubuntu ubuntu 4270 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/hosts/host-form/host-form.component.html 2024-03-27T21:42:43.795 INFO:teuthology.orchestra.run.smithi060.stdout: 412842 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/hosts/fixtures 2024-03-27T21:42:43.795 INFO:teuthology.orchestra.run.smithi060.stdout: 412843 4 -rw-r--r-- 1 ubuntu ubuntu 821 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/hosts/fixtures/host_list_response.json 2024-03-27T21:42:43.795 INFO:teuthology.orchestra.run.smithi060.stdout: 412856 16 -rw-r--r-- 1 ubuntu ubuntu 14893 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/hosts/hosts.component.spec.ts 2024-03-27T21:42:43.795 INFO:teuthology.orchestra.run.smithi060.stdout: 412855 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/hosts/hosts.component.scss 2024-03-27T21:42:43.795 INFO:teuthology.orchestra.run.smithi060.stdout: 412844 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/hosts/host-details 2024-03-27T21:42:43.795 INFO:teuthology.orchestra.run.smithi060.stdout: 412846 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/hosts/host-details/host-details.component.scss 2024-03-27T21:42:43.795 INFO:teuthology.orchestra.run.smithi060.stdout: 412848 4 -rw-r--r-- 1 ubuntu ubuntu 481 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/hosts/host-details/host-details.component.ts 2024-03-27T21:42:43.795 INFO:teuthology.orchestra.run.smithi060.stdout: 412847 4 -rw-r--r-- 1 ubuntu ubuntu 2051 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/hosts/host-details/host-details.component.spec.ts 2024-03-27T21:42:43.796 INFO:teuthology.orchestra.run.smithi060.stdout: 412845 4 -rw-r--r-- 1 ubuntu ubuntu 2108 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/hosts/host-details/host-details.component.html 2024-03-27T21:42:43.796 INFO:teuthology.orchestra.run.smithi060.stdout: 412854 4 -rw-r--r-- 1 ubuntu ubuntu 3588 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/hosts/hosts.component.html 2024-03-27T21:42:43.796 INFO:teuthology.orchestra.run.smithi060.stdout: 412857 20 -rw-r--r-- 1 ubuntu ubuntu 18127 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/hosts/hosts.component.ts 2024-03-27T21:42:43.796 INFO:teuthology.orchestra.run.smithi060.stdout: 413139 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/performance-counter 2024-03-27T21:42:43.796 INFO:teuthology.orchestra.run.smithi060.stdout: 413146 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/performance-counter/table-performance-counter 2024-03-27T21:42:43.796 INFO:teuthology.orchestra.run.smithi060.stdout: 413149 4 -rw-r--r-- 1 ubuntu ubuntu 2150 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/performance-counter/table-performance-counter/table-performance-counter.component.spec.ts 2024-03-27T21:42:43.796 INFO:teuthology.orchestra.run.smithi060.stdout: 413150 4 -rw-r--r-- 1 ubuntu ubuntu 1813 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/performance-counter/table-performance-counter/table-performance-counter.component.ts 2024-03-27T21:42:43.796 INFO:teuthology.orchestra.run.smithi060.stdout: 413148 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/performance-counter/table-performance-counter/table-performance-counter.component.scss 2024-03-27T21:42:43.796 INFO:teuthology.orchestra.run.smithi060.stdout: 413147 4 -rw-r--r-- 1 ubuntu ubuntu 476 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/performance-counter/table-performance-counter/table-performance-counter.component.html 2024-03-27T21:42:43.797 INFO:teuthology.orchestra.run.smithi060.stdout: 413141 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/performance-counter/performance-counter 2024-03-27T21:42:43.797 INFO:teuthology.orchestra.run.smithi060.stdout: 413145 4 -rw-r--r-- 1 ubuntu ubuntu 799 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/performance-counter/performance-counter/performance-counter.component.ts 2024-03-27T21:42:43.797 INFO:teuthology.orchestra.run.smithi060.stdout: 413142 4 -rw-r--r-- 1 ubuntu ubuntu 196 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/performance-counter/performance-counter/performance-counter.component.html 2024-03-27T21:42:43.797 INFO:teuthology.orchestra.run.smithi060.stdout: 413143 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/performance-counter/performance-counter/performance-counter.component.scss 2024-03-27T21:42:43.797 INFO:teuthology.orchestra.run.smithi060.stdout: 413144 4 -rw-r--r-- 1 ubuntu ubuntu 1213 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/performance-counter/performance-counter/performance-counter.component.spec.ts 2024-03-27T21:42:43.797 INFO:teuthology.orchestra.run.smithi060.stdout: 413140 4 -rw-r--r-- 1 ubuntu ubuntu 652 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/performance-counter/performance-counter.module.ts 2024-03-27T21:42:43.797 INFO:teuthology.orchestra.run.smithi060.stdout: 413115 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs 2024-03-27T21:42:43.798 INFO:teuthology.orchestra.run.smithi060.stdout: 413138 4 -rw-r--r-- 1 ubuntu ubuntu 943 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs.module.ts 2024-03-27T21:42:43.798 INFO:teuthology.orchestra.run.smithi060.stdout: 413123 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-form-client 2024-03-27T21:42:43.798 INFO:teuthology.orchestra.run.smithi060.stdout: 413126 4 -rw-r--r-- 1 ubuntu ubuntu 2410 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-form-client/nfs-form-client.component.spec.ts 2024-03-27T21:42:43.798 INFO:teuthology.orchestra.run.smithi060.stdout: 413124 8 -rw-r--r-- 1 ubuntu ubuntu 4119 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-form-client/nfs-form-client.component.html 2024-03-27T21:42:43.798 INFO:teuthology.orchestra.run.smithi060.stdout: 413127 4 -rw-r--r-- 1 ubuntu ubuntu 3174 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-form-client/nfs-form-client.component.ts 2024-03-27T21:42:43.798 INFO:teuthology.orchestra.run.smithi060.stdout: 413125 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-form-client/nfs-form-client.component.scss 2024-03-27T21:42:43.798 INFO:teuthology.orchestra.run.smithi060.stdout: 413116 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/models 2024-03-27T21:42:43.798 INFO:teuthology.orchestra.run.smithi060.stdout: 413117 4 -rw-r--r-- 1 ubuntu ubuntu 98 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/models/nfs.fsal.ts 2024-03-27T21:42:43.799 INFO:teuthology.orchestra.run.smithi060.stdout: 413118 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-details 2024-03-27T21:42:43.799 INFO:teuthology.orchestra.run.smithi060.stdout: 413121 4 -rw-r--r-- 1 ubuntu ubuntu 2933 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-details/nfs-details.component.spec.ts 2024-03-27T21:42:43.799 INFO:teuthology.orchestra.run.smithi060.stdout: 413122 4 -rw-r--r-- 1 ubuntu ubuntu 2052 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-details/nfs-details.component.ts 2024-03-27T21:42:43.799 INFO:teuthology.orchestra.run.smithi060.stdout: 413120 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-details/nfs-details.component.scss 2024-03-27T21:42:43.799 INFO:teuthology.orchestra.run.smithi060.stdout: 413119 4 -rw-r--r-- 1 ubuntu ubuntu 887 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-details/nfs-details.component.html 2024-03-27T21:42:43.799 INFO:teuthology.orchestra.run.smithi060.stdout: 413128 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-form 2024-03-27T21:42:43.799 INFO:teuthology.orchestra.run.smithi060.stdout: 413132 20 -rw-r--r-- 1 ubuntu ubuntu 16783 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-form/nfs-form.component.ts 2024-03-27T21:42:43.799 INFO:teuthology.orchestra.run.smithi060.stdout: 413130 4 -rw-r--r-- 1 ubuntu ubuntu 128 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-form/nfs-form.component.scss 2024-03-27T21:42:43.800 INFO:teuthology.orchestra.run.smithi060.stdout: 413131 12 -rw-r--r-- 1 ubuntu ubuntu 8832 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-form/nfs-form.component.spec.ts 2024-03-27T21:42:43.800 INFO:teuthology.orchestra.run.smithi060.stdout: 413129 20 -rw-r--r-- 1 ubuntu ubuntu 17746 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-form/nfs-form.component.html 2024-03-27T21:42:43.800 INFO:teuthology.orchestra.run.smithi060.stdout: 413133 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-list 2024-03-27T21:42:43.800 INFO:teuthology.orchestra.run.smithi060.stdout: 413137 8 -rw-r--r-- 1 ubuntu ubuntu 6066 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-list/nfs-list.component.ts 2024-03-27T21:42:43.800 INFO:teuthology.orchestra.run.smithi060.stdout: 413134 4 -rw-r--r-- 1 ubuntu ubuntu 954 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-list/nfs-list.component.html 2024-03-27T21:42:43.800 INFO:teuthology.orchestra.run.smithi060.stdout: 413135 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-list/nfs-list.component.scss 2024-03-27T21:42:43.800 INFO:teuthology.orchestra.run.smithi060.stdout: 413136 8 -rw-r--r-- 1 ubuntu ubuntu 6608 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-list/nfs-list.component.spec.ts 2024-03-27T21:42:43.800 INFO:teuthology.orchestra.run.smithi060.stdout: 413370 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core 2024-03-27T21:42:43.801 INFO:teuthology.orchestra.run.smithi060.stdout: 413435 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/layouts 2024-03-27T21:42:43.801 INFO:teuthology.orchestra.run.smithi060.stdout: 413441 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/layouts/login-layout 2024-03-27T21:42:43.801 INFO:teuthology.orchestra.run.smithi060.stdout: 413442 4 -rw-r--r-- 1 ubuntu ubuntu 1095 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/layouts/login-layout/login-layout.component.html 2024-03-27T21:42:43.801 INFO:teuthology.orchestra.run.smithi060.stdout: 413443 4 -rw-r--r-- 1 ubuntu ubuntu 1073 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/layouts/login-layout/login-layout.component.scss 2024-03-27T21:42:43.801 INFO:teuthology.orchestra.run.smithi060.stdout: 413445 4 -rw-r--r-- 1 ubuntu ubuntu 414 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/layouts/login-layout/login-layout.component.ts 2024-03-27T21:42:43.801 INFO:teuthology.orchestra.run.smithi060.stdout: 413444 4 -rw-r--r-- 1 ubuntu ubuntu 1013 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/layouts/login-layout/login-layout.component.spec.ts 2024-03-27T21:42:43.801 INFO:teuthology.orchestra.run.smithi060.stdout: 413446 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/layouts/workbench-layout 2024-03-27T21:42:43.801 INFO:teuthology.orchestra.run.smithi060.stdout: 413447 4 -rw-r--r-- 1 ubuntu ubuntu 409 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/layouts/workbench-layout/workbench-layout.component.html 2024-03-27T21:42:43.801 INFO:teuthology.orchestra.run.smithi060.stdout: 413450 4 -rw-r--r-- 1 ubuntu ubuntu 1350 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/layouts/workbench-layout/workbench-layout.component.ts 2024-03-27T21:42:43.802 INFO:teuthology.orchestra.run.smithi060.stdout: 413449 4 -rw-r--r-- 1 ubuntu ubuntu 1351 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/layouts/workbench-layout/workbench-layout.component.spec.ts 2024-03-27T21:42:43.802 INFO:teuthology.orchestra.run.smithi060.stdout: 413448 4 -rw-r--r-- 1 ubuntu ubuntu 243 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/layouts/workbench-layout/workbench-layout.component.scss 2024-03-27T21:42:43.802 INFO:teuthology.orchestra.run.smithi060.stdout: 413436 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/layouts/blank-layout 2024-03-27T21:42:43.802 INFO:teuthology.orchestra.run.smithi060.stdout: 413439 4 -rw-r--r-- 1 ubuntu ubuntu 740 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/layouts/blank-layout/blank-layout.component.spec.ts 2024-03-27T21:42:43.802 INFO:teuthology.orchestra.run.smithi060.stdout: 413437 4 -rw-r--r-- 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/layouts/blank-layout/blank-layout.component.html 2024-03-27T21:42:43.802 INFO:teuthology.orchestra.run.smithi060.stdout: 413438 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/layouts/blank-layout/blank-layout.component.scss 2024-03-27T21:42:43.802 INFO:teuthology.orchestra.run.smithi060.stdout: 413440 4 -rw-r--r-- 1 ubuntu ubuntu 223 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/layouts/blank-layout/blank-layout.component.ts 2024-03-27T21:42:43.803 INFO:teuthology.orchestra.run.smithi060.stdout: 413428 4 -rw-r--r-- 1 ubuntu ubuntu 1140 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/core.module.ts 2024-03-27T21:42:43.803 INFO:teuthology.orchestra.run.smithi060.stdout: 413429 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/error 2024-03-27T21:42:43.803 INFO:teuthology.orchestra.run.smithi060.stdout: 413433 4 -rw-r--r-- 1 ubuntu ubuntu 3252 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/error/error.component.ts 2024-03-27T21:42:43.803 INFO:teuthology.orchestra.run.smithi060.stdout: 413432 4 -rw-r--r-- 1 ubuntu ubuntu 1929 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/error/error.component.spec.ts 2024-03-27T21:42:43.803 INFO:teuthology.orchestra.run.smithi060.stdout: 413431 4 -rw-r--r-- 1 ubuntu ubuntu 191 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/error/error.component.scss 2024-03-27T21:42:43.803 INFO:teuthology.orchestra.run.smithi060.stdout: 413434 4 -rw-r--r-- 1 ubuntu ubuntu 893 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/error/error.ts 2024-03-27T21:42:43.803 INFO:teuthology.orchestra.run.smithi060.stdout: 413430 4 -rw-r--r-- 1 ubuntu ubuntu 2149 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/error/error.component.html 2024-03-27T21:42:43.803 INFO:teuthology.orchestra.run.smithi060.stdout: 413371 4 drwxr-xr-x 11 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth 2024-03-27T21:42:43.803 INFO:teuthology.orchestra.run.smithi060.stdout: 413395 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/role-list 2024-03-27T21:42:43.804 INFO:teuthology.orchestra.run.smithi060.stdout: 413397 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/role-list/role-list.component.scss 2024-03-27T21:42:43.804 INFO:teuthology.orchestra.run.smithi060.stdout: 413398 4 -rw-r--r-- 1 ubuntu ubuntu 3065 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/role-list/role-list.component.spec.ts 2024-03-27T21:42:43.804 INFO:teuthology.orchestra.run.smithi060.stdout: 413396 4 -rw-r--r-- 1 ubuntu ubuntu 701 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/role-list/role-list.component.html 2024-03-27T21:42:43.804 INFO:teuthology.orchestra.run.smithi060.stdout: 413399 8 -rw-r--r-- 1 ubuntu ubuntu 5720 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/role-list/role-list.component.ts 2024-03-27T21:42:43.804 INFO:teuthology.orchestra.run.smithi060.stdout: 413388 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/role-form 2024-03-27T21:42:43.804 INFO:teuthology.orchestra.run.smithi060.stdout: 413394 4 -rw-r--r-- 1 ubuntu ubuntu 97 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/role-form/role-form.model.ts 2024-03-27T21:42:43.804 INFO:teuthology.orchestra.run.smithi060.stdout: 413392 8 -rw-r--r-- 1 ubuntu ubuntu 5351 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/role-form/role-form.component.spec.ts 2024-03-27T21:42:43.804 INFO:teuthology.orchestra.run.smithi060.stdout: 413390 4 -rw-r--r-- 1 ubuntu ubuntu 2798 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/role-form/role-form.component.html 2024-03-27T21:42:43.805 INFO:teuthology.orchestra.run.smithi060.stdout: 413389 4 -rw-r--r-- 1 ubuntu ubuntu 51 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/role-form/role-form-mode.enum.ts 2024-03-27T21:42:43.805 INFO:teuthology.orchestra.run.smithi060.stdout: 413391 4 -rw-r--r-- 1 ubuntu ubuntu 107 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/role-form/role-form.component.scss 2024-03-27T21:42:43.805 INFO:teuthology.orchestra.run.smithi060.stdout: 413393 8 -rw-r--r-- 1 ubuntu ubuntu 5308 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/role-form/role-form.component.ts 2024-03-27T21:42:43.805 INFO:teuthology.orchestra.run.smithi060.stdout: 413418 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-tabs 2024-03-27T21:42:43.805 INFO:teuthology.orchestra.run.smithi060.stdout: 413419 4 -rw-r--r-- 1 ubuntu ubuntu 510 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-tabs/user-tabs.component.html 2024-03-27T21:42:43.805 INFO:teuthology.orchestra.run.smithi060.stdout: 413421 4 -rw-r--r-- 1 ubuntu ubuntu 961 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-tabs/user-tabs.component.spec.ts 2024-03-27T21:42:43.805 INFO:teuthology.orchestra.run.smithi060.stdout: 413422 4 -rw-r--r-- 1 ubuntu ubuntu 310 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-tabs/user-tabs.component.ts 2024-03-27T21:42:43.805 INFO:teuthology.orchestra.run.smithi060.stdout: 413420 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-tabs/user-tabs.component.scss 2024-03-27T21:42:43.806 INFO:teuthology.orchestra.run.smithi060.stdout: 413372 4 -rw-r--r-- 1 ubuntu ubuntu 2681 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/auth.module.ts 2024-03-27T21:42:43.806 INFO:teuthology.orchestra.run.smithi060.stdout: 413408 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-list 2024-03-27T21:42:43.806 INFO:teuthology.orchestra.run.smithi060.stdout: 413411 4 -rw-r--r-- 1 ubuntu ubuntu 3610 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-list/user-list.component.spec.ts 2024-03-27T21:42:43.806 INFO:teuthology.orchestra.run.smithi060.stdout: 413409 4 -rwxr-xr-x 1 ubuntu ubuntu 1707 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-list/user-list.component.html 2024-03-27T21:42:43.806 INFO:teuthology.orchestra.run.smithi060.stdout: 413410 4 -rwxr-xr-x 1 ubuntu ubuntu 297 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-list/user-list.component.scss 2024-03-27T21:42:43.806 INFO:teuthology.orchestra.run.smithi060.stdout: 413412 8 -rwxr-xr-x 1 ubuntu ubuntu 7360 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-list/user-list.component.ts 2024-03-27T21:42:43.806 INFO:teuthology.orchestra.run.smithi060.stdout: 413383 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/role-details 2024-03-27T21:42:43.806 INFO:teuthology.orchestra.run.smithi060.stdout: 413386 4 -rw-r--r-- 1 ubuntu ubuntu 2304 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/role-details/role-details.component.spec.ts 2024-03-27T21:42:43.807 INFO:teuthology.orchestra.run.smithi060.stdout: 413384 4 -rw-r--r-- 1 ubuntu ubuntu 316 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/role-details/role-details.component.html 2024-03-27T21:42:43.807 INFO:teuthology.orchestra.run.smithi060.stdout: 413385 4 -rw-r--r-- 1 ubuntu ubuntu 121 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/role-details/role-details.component.scss 2024-03-27T21:42:43.807 INFO:teuthology.orchestra.run.smithi060.stdout: 413387 4 -rw-r--r-- 1 ubuntu ubuntu 2175 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/role-details/role-details.component.ts 2024-03-27T21:42:43.807 INFO:teuthology.orchestra.run.smithi060.stdout: 413378 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/login 2024-03-27T21:42:43.807 INFO:teuthology.orchestra.run.smithi060.stdout: 413382 4 -rw-r--r-- 1 ubuntu ubuntu 2570 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/login/login.component.ts 2024-03-27T21:42:43.807 INFO:teuthology.orchestra.run.smithi060.stdout: 413381 4 -rw-r--r-- 1 ubuntu ubuntu 2013 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/login/login.component.spec.ts 2024-03-27T21:42:43.807 INFO:teuthology.orchestra.run.smithi060.stdout: 413379 4 -rw-r--r-- 1 ubuntu ubuntu 2195 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/login/login.component.html 2024-03-27T21:42:43.807 INFO:teuthology.orchestra.run.smithi060.stdout: 413380 4 -rw-r--r-- 1 ubuntu ubuntu 1092 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/login/login.component.scss 2024-03-27T21:42:43.808 INFO:teuthology.orchestra.run.smithi060.stdout: 413373 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/login-password-form 2024-03-27T21:42:43.808 INFO:teuthology.orchestra.run.smithi060.stdout: 413377 4 -rwxr-xr-x 1 ubuntu ubuntu 1840 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/login-password-form/login-password-form.component.ts 2024-03-27T21:42:43.808 INFO:teuthology.orchestra.run.smithi060.stdout: 413376 4 -rwxr-xr-x 1 ubuntu ubuntu 2755 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/login-password-form/login-password-form.component.spec.ts 2024-03-27T21:42:43.808 INFO:teuthology.orchestra.run.smithi060.stdout: 413374 4 -rwxr-xr-x 1 ubuntu ubuntu 3523 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/login-password-form/login-password-form.component.html 2024-03-27T21:42:43.808 INFO:teuthology.orchestra.run.smithi060.stdout: 413375 4 -rwxr-xr-x 1 ubuntu ubuntu 1367 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/login-password-form/login-password-form.component.scss 2024-03-27T21:42:43.808 INFO:teuthology.orchestra.run.smithi060.stdout: 413413 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-password-form 2024-03-27T21:42:43.808 INFO:teuthology.orchestra.run.smithi060.stdout: 413416 4 -rw-r--r-- 1 ubuntu ubuntu 3004 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-password-form/user-password-form.component.spec.ts 2024-03-27T21:42:43.808 INFO:teuthology.orchestra.run.smithi060.stdout: 413415 4 -rw-r--r-- 1 ubuntu ubuntu 128 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-password-form/user-password-form.component.scss 2024-03-27T21:42:43.809 INFO:teuthology.orchestra.run.smithi060.stdout: 413414 8 -rw-r--r-- 1 ubuntu ubuntu 4719 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-password-form/user-password-form.component.html 2024-03-27T21:42:43.809 INFO:teuthology.orchestra.run.smithi060.stdout: 413417 4 -rw-r--r-- 1 ubuntu ubuntu 3999 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-password-form/user-password-form.component.ts 2024-03-27T21:42:43.809 INFO:teuthology.orchestra.run.smithi060.stdout: 413400 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form 2024-03-27T21:42:43.809 INFO:teuthology.orchestra.run.smithi060.stdout: 413407 4 -rw-r--r-- 1 ubuntu ubuntu 207 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.model.ts 2024-03-27T21:42:43.809 INFO:teuthology.orchestra.run.smithi060.stdout: 413401 4 -rw-r--r-- 1 ubuntu ubuntu 51 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form-mode.enum.ts 2024-03-27T21:42:43.809 INFO:teuthology.orchestra.run.smithi060.stdout: 413402 4 -rw-r--r-- 1 ubuntu ubuntu 364 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form-role.model.ts 2024-03-27T21:42:43.809 INFO:teuthology.orchestra.run.smithi060.stdout: 413406 12 -rw-r--r-- 1 ubuntu ubuntu 10530 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.ts 2024-03-27T21:42:43.809 INFO:teuthology.orchestra.run.smithi060.stdout: 413404 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.scss 2024-03-27T21:42:43.810 INFO:teuthology.orchestra.run.smithi060.stdout: 413403 12 -rw-r--r-- 1 ubuntu ubuntu 10282 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.html 2024-03-27T21:42:43.810 INFO:teuthology.orchestra.run.smithi060.stdout: 413405 12 -rw-r--r-- 1 ubuntu ubuntu 8831 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.spec.ts 2024-03-27T21:42:43.810 INFO:teuthology.orchestra.run.smithi060.stdout: 413451 4 drwxr-xr-x 10 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation 2024-03-27T21:42:43.810 INFO:teuthology.orchestra.run.smithi060.stdout: 413462 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/api-docs 2024-03-27T21:42:43.810 INFO:teuthology.orchestra.run.smithi060.stdout: 413465 4 -rw-r--r-- 1 ubuntu ubuntu 434 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/api-docs/api-docs.component.ts 2024-03-27T21:42:43.810 INFO:teuthology.orchestra.run.smithi060.stdout: 413463 4 -rw-r--r-- 1 ubuntu ubuntu 50 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/api-docs/api-docs.component.html 2024-03-27T21:42:43.810 INFO:teuthology.orchestra.run.smithi060.stdout: 413464 4 -rw-r--r-- 1 ubuntu ubuntu 148 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/api-docs/api-docs.component.scss 2024-03-27T21:42:43.811 INFO:teuthology.orchestra.run.smithi060.stdout: 413476 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/identity 2024-03-27T21:42:43.811 INFO:teuthology.orchestra.run.smithi060.stdout: 413478 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/identity/identity.component.scss 2024-03-27T21:42:43.811 INFO:teuthology.orchestra.run.smithi060.stdout: 413477 4 -rw-r--r-- 1 ubuntu ubuntu 780 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/identity/identity.component.html 2024-03-27T21:42:43.811 INFO:teuthology.orchestra.run.smithi060.stdout: 413479 4 -rw-r--r-- 1 ubuntu ubuntu 886 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/identity/identity.component.spec.ts 2024-03-27T21:42:43.811 INFO:teuthology.orchestra.run.smithi060.stdout: 413480 4 -rw-r--r-- 1 ubuntu ubuntu 761 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/identity/identity.component.ts 2024-03-27T21:42:43.811 INFO:teuthology.orchestra.run.smithi060.stdout: 413471 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/dashboard-help 2024-03-27T21:42:43.811 INFO:teuthology.orchestra.run.smithi060.stdout: 413472 4 -rw-r--r-- 1 ubuntu ubuntu 824 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/dashboard-help/dashboard-help.component.html 2024-03-27T21:42:43.811 INFO:teuthology.orchestra.run.smithi060.stdout: 413475 4 -rw-r--r-- 1 ubuntu ubuntu 1164 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/dashboard-help/dashboard-help.component.ts 2024-03-27T21:42:43.812 INFO:teuthology.orchestra.run.smithi060.stdout: 413473 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/dashboard-help/dashboard-help.component.scss 2024-03-27T21:42:43.812 INFO:teuthology.orchestra.run.smithi060.stdout: 413474 4 -rw-r--r-- 1 ubuntu ubuntu 922 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/dashboard-help/dashboard-help.component.spec.ts 2024-03-27T21:42:43.812 INFO:teuthology.orchestra.run.smithi060.stdout: 413457 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/administration 2024-03-27T21:42:43.812 INFO:teuthology.orchestra.run.smithi060.stdout: 413458 4 -rw-r--r-- 1 ubuntu ubuntu 670 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/administration/administration.component.html 2024-03-27T21:42:43.812 INFO:teuthology.orchestra.run.smithi060.stdout: 413461 4 -rw-r--r-- 1 ubuntu ubuntu 746 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/administration/administration.component.ts 2024-03-27T21:42:43.812 INFO:teuthology.orchestra.run.smithi060.stdout: 413460 4 -rw-r--r-- 1 ubuntu ubuntu 747 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/administration/administration.component.spec.ts 2024-03-27T21:42:43.812 INFO:teuthology.orchestra.run.smithi060.stdout: 413459 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/administration/administration.component.scss 2024-03-27T21:42:43.812 INFO:teuthology.orchestra.run.smithi060.stdout: 413466 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/breadcrumbs 2024-03-27T21:42:43.813 INFO:teuthology.orchestra.run.smithi060.stdout: 413467 4 -rw-r--r-- 1 ubuntu ubuntu 388 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/breadcrumbs/breadcrumbs.component.html 2024-03-27T21:42:43.813 INFO:teuthology.orchestra.run.smithi060.stdout: 413469 8 -rw-r--r-- 1 ubuntu ubuntu 5348 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/breadcrumbs/breadcrumbs.component.spec.ts 2024-03-27T21:42:43.813 INFO:teuthology.orchestra.run.smithi060.stdout: 413470 8 -rw-r--r-- 1 ubuntu ubuntu 5134 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/breadcrumbs/breadcrumbs.component.ts 2024-03-27T21:42:43.813 INFO:teuthology.orchestra.run.smithi060.stdout: 413468 4 -rw-r--r-- 1 ubuntu ubuntu 215 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/breadcrumbs/breadcrumbs.component.scss 2024-03-27T21:42:43.813 INFO:teuthology.orchestra.run.smithi060.stdout: 413481 4 -rw-r--r-- 1 ubuntu ubuntu 1554 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/navigation.module.ts 2024-03-27T21:42:43.813 INFO:teuthology.orchestra.run.smithi060.stdout: 413452 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/about 2024-03-27T21:42:43.813 INFO:teuthology.orchestra.run.smithi060.stdout: 413456 4 -rw-r--r-- 1 ubuntu ubuntu 2427 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/about/about.component.ts 2024-03-27T21:42:43.813 INFO:teuthology.orchestra.run.smithi060.stdout: 413453 4 -rw-r--r-- 1 ubuntu ubuntu 1246 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/about/about.component.html 2024-03-27T21:42:43.813 INFO:teuthology.orchestra.run.smithi060.stdout: 413454 4 -rw-r--r-- 1 ubuntu ubuntu 689 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/about/about.component.scss 2024-03-27T21:42:43.814 INFO:teuthology.orchestra.run.smithi060.stdout: 413455 4 -rw-r--r-- 1 ubuntu ubuntu 1968 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/about/about.component.spec.ts 2024-03-27T21:42:43.814 INFO:teuthology.orchestra.run.smithi060.stdout: 413487 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/notifications 2024-03-27T21:42:43.814 INFO:teuthology.orchestra.run.smithi060.stdout: 413491 4 -rw-r--r-- 1 ubuntu ubuntu 1304 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/notifications/notifications.component.ts 2024-03-27T21:42:43.814 INFO:teuthology.orchestra.run.smithi060.stdout: 413489 4 -rw-r--r-- 1 ubuntu ubuntu 425 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/notifications/notifications.component.scss 2024-03-27T21:42:43.814 INFO:teuthology.orchestra.run.smithi060.stdout: 413490 4 -rw-r--r-- 1 ubuntu ubuntu 2310 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/notifications/notifications.component.spec.ts 2024-03-27T21:42:43.814 INFO:teuthology.orchestra.run.smithi060.stdout: 413488 4 -rw-r--r-- 1 ubuntu ubuntu 299 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/notifications/notifications.component.html 2024-03-27T21:42:43.814 INFO:teuthology.orchestra.run.smithi060.stdout: 413482 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/navigation 2024-03-27T21:42:43.814 INFO:teuthology.orchestra.run.smithi060.stdout: 413483 16 -rw-r--r-- 1 ubuntu ubuntu 16199 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/navigation/navigation.component.html 2024-03-27T21:42:43.815 INFO:teuthology.orchestra.run.smithi060.stdout: 413485 12 -rw-r--r-- 1 ubuntu ubuntu 9997 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/navigation/navigation.component.spec.ts 2024-03-27T21:42:43.815 INFO:teuthology.orchestra.run.smithi060.stdout: 413484 8 -rw-r--r-- 1 ubuntu ubuntu 5886 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/navigation/navigation.component.scss 2024-03-27T21:42:43.815 INFO:teuthology.orchestra.run.smithi060.stdout: 413486 8 -rw-r--r-- 1 ubuntu ubuntu 8031 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/navigation/navigation.component.ts 2024-03-27T21:42:43.815 INFO:teuthology.orchestra.run.smithi060.stdout: 413423 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/context 2024-03-27T21:42:43.815 INFO:teuthology.orchestra.run.smithi060.stdout: 413425 4 -rw-r--r-- 1 ubuntu ubuntu 106 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/context/context.component.scss 2024-03-27T21:42:43.815 INFO:teuthology.orchestra.run.smithi060.stdout: 413426 4 -rw-r--r-- 1 ubuntu ubuntu 3741 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/context/context.component.spec.ts 2024-03-27T21:42:43.815 INFO:teuthology.orchestra.run.smithi060.stdout: 413424 4 -rw-r--r-- 1 ubuntu ubuntu 1211 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/context/context.component.html 2024-03-27T21:42:43.816 INFO:teuthology.orchestra.run.smithi060.stdout: 413427 4 -rw-r--r-- 1 ubuntu ubuntu 2785 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/core/context/context.component.ts 2024-03-27T21:42:43.816 INFO:teuthology.orchestra.run.smithi060.stdout: 412539 4 -rw-r--r-- 1 ubuntu ubuntu 1511 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/app.module.ts 2024-03-27T21:42:43.816 INFO:teuthology.orchestra.run.smithi060.stdout: 412538 4 -rw-r--r-- 1 ubuntu ubuntu 516 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/app.component.ts 2024-03-27T21:42:43.816 INFO:teuthology.orchestra.run.smithi060.stdout: 412537 4 -rw-r--r-- 1 ubuntu ubuntu 681 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/app/app.component.spec.ts 2024-03-27T21:42:43.816 INFO:teuthology.orchestra.run.smithi060.stdout: 414148 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/assets 2024-03-27T21:42:43.816 INFO:teuthology.orchestra.run.smithi060.stdout: 414155 4 -rw-r--r-- 1 ubuntu ubuntu 1811 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/assets/logo-mini.png 2024-03-27T21:42:43.816 INFO:teuthology.orchestra.run.smithi060.stdout: 414150 12 -rw-r--r-- 1 ubuntu ubuntu 9901 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/assets/Ceph_Ceph_Logo_with_text_red_white.svg 2024-03-27T21:42:43.816 INFO:teuthology.orchestra.run.smithi060.stdout: 414156 4 -rw-r--r-- 1 ubuntu ubuntu 2777 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/assets/prometheus_logo.svg 2024-03-27T21:42:43.816 INFO:teuthology.orchestra.run.smithi060.stdout: 414153 96 -rw-r--r-- 1 ubuntu ubuntu 98115 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/assets/ceph_background.gif 2024-03-27T21:42:43.817 INFO:teuthology.orchestra.run.smithi060.stdout: 414152 8 -rw-r--r-- 1 ubuntu ubuntu 6130 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/assets/Ceph_Logo.svg 2024-03-27T21:42:43.817 INFO:teuthology.orchestra.run.smithi060.stdout: 414154 36 -rwxr-xr-x 1 ubuntu ubuntu 35386 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/assets/loading.gif 2024-03-27T21:42:43.817 INFO:teuthology.orchestra.run.smithi060.stdout: 414151 12 -rw-r--r-- 1 ubuntu ubuntu 9897 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/assets/Ceph_Ceph_Logo_with_text_white.svg 2024-03-27T21:42:43.817 INFO:teuthology.orchestra.run.smithi060.stdout: 414149 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/src/assets/.gitkeep 2024-03-27T21:42:43.817 INFO:teuthology.orchestra.run.smithi060.stdout: 412396 4 -rw-r--r-- 1 ubuntu ubuntu 245 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/.editorconfig 2024-03-27T21:42:43.817 INFO:teuthology.orchestra.run.smithi060.stdout: 414206 4 -rw-r--r-- 1 ubuntu ubuntu 1004 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/tsconfig.json 2024-03-27T21:42:43.817 INFO:teuthology.orchestra.run.smithi060.stdout: 412407 4 -rw-r--r-- 1 ubuntu ubuntu 625 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/applitools.config.js 2024-03-27T21:42:43.817 INFO:teuthology.orchestra.run.smithi060.stdout: 412526 4 -rw-r--r-- 1 ubuntu ubuntu 357 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/i18n.config.json 2024-03-27T21:42:43.817 INFO:teuthology.orchestra.run.smithi060.stdout: 412530 8 -rw-r--r-- 1 ubuntu ubuntu 5569 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/package.json 2024-03-27T21:42:43.818 INFO:teuthology.orchestra.run.smithi060.stdout: 412398 4 -rw-r--r-- 1 ubuntu ubuntu 1014 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/.gherkin-lintrc 2024-03-27T21:42:43.818 INFO:teuthology.orchestra.run.smithi060.stdout: 412397 4 -rw-r--r-- 1 ubuntu ubuntu 1960 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/.eslintrc.json 2024-03-27T21:42:43.818 INFO:teuthology.orchestra.run.smithi060.stdout: 414207 4 -rw-r--r-- 1 ubuntu ubuntu 342 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/tsconfig.spec.json 2024-03-27T21:42:43.818 INFO:teuthology.orchestra.run.smithi060.stdout: 412401 4 -rw-r--r-- 1 ubuntu ubuntu 50 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/.npmrc 2024-03-27T21:42:43.818 INFO:teuthology.orchestra.run.smithi060.stdout: 412406 8 -rw-r--r-- 1 ubuntu ubuntu 7779 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/angular.json 2024-03-27T21:42:43.818 INFO:teuthology.orchestra.run.smithi060.stdout: 412531 4 -rw-r--r-- 1 ubuntu ubuntu 316 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/proxy.conf.json.sample 2024-03-27T21:42:43.818 INFO:teuthology.orchestra.run.smithi060.stdout: 412399 4 -rw-r--r-- 1 ubuntu ubuntu 612 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/.gitignore 2024-03-27T21:42:43.819 INFO:teuthology.orchestra.run.smithi060.stdout: 412410 4 -rw-r--r-- 1 ubuntu ubuntu 1346 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress.config.ts 2024-03-27T21:42:43.819 INFO:teuthology.orchestra.run.smithi060.stdout: 412395 4 -rw-r--r-- 1 ubuntu ubuntu 405 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/.browserslistrc 2024-03-27T21:42:43.819 INFO:teuthology.orchestra.run.smithi060.stdout: 414205 4 -rw-r--r-- 1 ubuntu ubuntu 210 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/tsconfig.app.json 2024-03-27T21:42:43.819 INFO:teuthology.orchestra.run.smithi060.stdout: 412402 4 -rw-r--r-- 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/.prettierignore 2024-03-27T21:42:43.819 INFO:teuthology.orchestra.run.smithi060.stdout: 412525 4 -rw-r--r-- 1 ubuntu ubuntu 189 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/html-linter.config.json 2024-03-27T21:42:43.819 INFO:teuthology.orchestra.run.smithi060.stdout: 412400 4 -rw-r--r-- 1 ubuntu ubuntu 1830 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/.htmllintrc 2024-03-27T21:42:43.819 INFO:teuthology.orchestra.run.smithi060.stdout: 412527 4 -rw-r--r-- 1 ubuntu ubuntu 1140 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/jest.config.cjs 2024-03-27T21:42:43.819 INFO:teuthology.orchestra.run.smithi060.stdout: 412411 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress 2024-03-27T21:42:43.819 INFO:teuthology.orchestra.run.smithi060.stdout: 412518 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/plugins 2024-03-27T21:42:43.820 INFO:teuthology.orchestra.run.smithi060.stdout: 412519 4 -rw-r--r-- 1 ubuntu ubuntu 729 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/plugins/index.js 2024-03-27T21:42:43.820 INFO:teuthology.orchestra.run.smithi060.stdout: 412511 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/fixtures 2024-03-27T21:42:43.820 INFO:teuthology.orchestra.run.smithi060.stdout: 412512 4 -rw-r--r-- 1 ubuntu ubuntu 122 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/fixtures/block-rbd-status.json 2024-03-27T21:42:43.820 INFO:teuthology.orchestra.run.smithi060.stdout: 412517 4 -rw-r--r-- 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/fixtures/rgw-status.json 2024-03-27T21:42:43.820 INFO:teuthology.orchestra.run.smithi060.stdout: 412514 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/fixtures/orchestrator 2024-03-27T21:42:43.820 INFO:teuthology.orchestra.run.smithi060.stdout: 412515 12 -rw-r--r-- 1 ubuntu ubuntu 11385 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/fixtures/orchestrator/inventory.json 2024-03-27T21:42:43.820 INFO:teuthology.orchestra.run.smithi060.stdout: 412516 24 -rw-r--r-- 1 ubuntu ubuntu 24214 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/fixtures/orchestrator/services.json 2024-03-27T21:42:43.820 INFO:teuthology.orchestra.run.smithi060.stdout: 412513 4 -rw-r--r-- 1 ubuntu ubuntu 137 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/fixtures/nfs-ganesha-status.json 2024-03-27T21:42:43.820 INFO:teuthology.orchestra.run.smithi060.stdout: 412520 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/support 2024-03-27T21:42:43.821 INFO:teuthology.orchestra.run.smithi060.stdout: 412523 4 -rw-r--r-- 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/support/eyes-index.d.ts 2024-03-27T21:42:43.821 INFO:teuthology.orchestra.run.smithi060.stdout: 412522 4 -rw-r--r-- 1 ubuntu ubuntu 436 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/support/e2e.ts 2024-03-27T21:42:43.821 INFO:teuthology.orchestra.run.smithi060.stdout: 412521 8 -rw-r--r-- 1 ubuntu ubuntu 4533 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/support/commands.ts 2024-03-27T21:42:43.821 INFO:teuthology.orchestra.run.smithi060.stdout: 412524 4 -rw-r--r-- 1 ubuntu ubuntu 276 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/tsconfig.json 2024-03-27T21:42:43.821 INFO:teuthology.orchestra.run.smithi060.stdout: 412412 4 drwxr-xr-x 12 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e 2024-03-27T21:42:43.821 INFO:teuthology.orchestra.run.smithi060.stdout: 412476 12 -rw-r--r-- 1 ubuntu ubuntu 8772 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/page-helper.po.ts 2024-03-27T21:42:43.821 INFO:teuthology.orchestra.run.smithi060.stdout: 412508 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/visualTests 2024-03-27T21:42:43.822 INFO:teuthology.orchestra.run.smithi060.stdout: 412509 4 -rw-r--r-- 1 ubuntu ubuntu 662 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/visualTests/dashboard.vrt-spec.ts 2024-03-27T21:42:43.822 INFO:teuthology.orchestra.run.smithi060.stdout: 412510 4 -rw-r--r-- 1 ubuntu ubuntu 452 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/visualTests/login.vrt-spec.ts 2024-03-27T21:42:43.822 INFO:teuthology.orchestra.run.smithi060.stdout: 412416 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/block 2024-03-27T21:42:43.822 INFO:teuthology.orchestra.run.smithi060.stdout: 412418 4 -rw-r--r-- 1 ubuntu ubuntu 3746 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/block/images.po.ts 2024-03-27T21:42:43.822 INFO:teuthology.orchestra.run.smithi060.stdout: 412417 4 -rw-r--r-- 1 ubuntu ubuntu 2604 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/block/images.e2e-spec.ts 2024-03-27T21:42:43.822 INFO:teuthology.orchestra.run.smithi060.stdout: 412420 4 -rw-r--r-- 1 ubuntu ubuntu 179 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/block/iscsi.po.ts 2024-03-27T21:42:43.822 INFO:teuthology.orchestra.run.smithi060.stdout: 412422 4 -rw-r--r-- 1 ubuntu ubuntu 2087 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/block/mirroring.po.ts 2024-03-27T21:42:43.822 INFO:teuthology.orchestra.run.smithi060.stdout: 412421 8 -rw-r--r-- 1 ubuntu ubuntu 4180 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/block/mirroring.e2e-spec.ts 2024-03-27T21:42:43.823 INFO:teuthology.orchestra.run.smithi060.stdout: 412419 4 -rw-r--r-- 1 ubuntu ubuntu 687 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/block/iscsi.e2e-spec.ts 2024-03-27T21:42:43.823 INFO:teuthology.orchestra.run.smithi060.stdout: 412480 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/rgw 2024-03-27T21:42:43.823 INFO:teuthology.orchestra.run.smithi060.stdout: 412488 8 -rw-r--r-- 1 ubuntu ubuntu 5270 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/rgw/users.po.ts 2024-03-27T21:42:43.823 INFO:teuthology.orchestra.run.smithi060.stdout: 412483 4 -rw-r--r-- 1 ubuntu ubuntu 933 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/rgw/daemons.e2e-spec.ts 2024-03-27T21:42:43.823 INFO:teuthology.orchestra.run.smithi060.stdout: 412485 4 -rw-r--r-- 1 ubuntu ubuntu 638 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/rgw/roles.e2e-spec.ts 2024-03-27T21:42:43.823 INFO:teuthology.orchestra.run.smithi060.stdout: 412486 4 -rw-r--r-- 1 ubuntu ubuntu 1773 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/rgw/roles.po.ts 2024-03-27T21:42:43.823 INFO:teuthology.orchestra.run.smithi060.stdout: 412482 8 -rw-r--r-- 1 ubuntu ubuntu 6908 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/rgw/buckets.po.ts 2024-03-27T21:42:43.823 INFO:teuthology.orchestra.run.smithi060.stdout: 412481 4 -rw-r--r-- 1 ubuntu ubuntu 2055 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/rgw/buckets.e2e-spec.ts 2024-03-27T21:42:43.823 INFO:teuthology.orchestra.run.smithi060.stdout: 412484 4 -rw-r--r-- 1 ubuntu ubuntu 1047 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/rgw/daemons.po.ts 2024-03-27T21:42:43.824 INFO:teuthology.orchestra.run.smithi060.stdout: 412487 4 -rw-r--r-- 1 ubuntu ubuntu 1310 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/rgw/users.e2e-spec.ts 2024-03-27T21:42:43.824 INFO:teuthology.orchestra.run.smithi060.stdout: 412451 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/filesystems 2024-03-27T21:42:43.824 INFO:teuthology.orchestra.run.smithi060.stdout: 412455 4 -rw-r--r-- 1 ubuntu ubuntu 2160 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/filesystems/subvolumes.e2e-spec.feature 2024-03-27T21:42:43.824 INFO:teuthology.orchestra.run.smithi060.stdout: 412453 4 -rw-r--r-- 1 ubuntu ubuntu 2148 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/filesystems/snapshots.e2e-spec.feature 2024-03-27T21:42:43.824 INFO:teuthology.orchestra.run.smithi060.stdout: 412454 4 -rw-r--r-- 1 ubuntu ubuntu 2267 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/filesystems/subvolume-groups.e2e-spec.feature 2024-03-27T21:42:43.824 INFO:teuthology.orchestra.run.smithi060.stdout: 412452 4 -rw-r--r-- 1 ubuntu ubuntu 1129 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/filesystems/filesystems.e2e-spec.feature 2024-03-27T21:42:43.824 INFO:teuthology.orchestra.run.smithi060.stdout: 412413 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/a11y 2024-03-27T21:42:43.824 INFO:teuthology.orchestra.run.smithi060.stdout: 412415 4 -rw-r--r-- 1 ubuntu ubuntu 513 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/a11y/navigation.e2e-spec.ts 2024-03-27T21:42:43.825 INFO:teuthology.orchestra.run.smithi060.stdout: 412414 4 -rw-r--r-- 1 ubuntu ubuntu 585 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/a11y/dashboard.e2e-spec.ts 2024-03-27T21:42:43.825 INFO:teuthology.orchestra.run.smithi060.stdout: 412489 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui 2024-03-27T21:42:43.825 INFO:teuthology.orchestra.run.smithi060.stdout: 412501 4 -rw-r--r-- 1 ubuntu ubuntu 3418 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/navigation.po.ts 2024-03-27T21:42:43.825 INFO:teuthology.orchestra.run.smithi060.stdout: 412492 4 -rw-r--r-- 1 ubuntu ubuntu 1716 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/dashboard-v3.e2e-spec.ts 2024-03-27T21:42:43.825 INFO:teuthology.orchestra.run.smithi060.stdout: 412499 4 -rw-r--r-- 1 ubuntu ubuntu 610 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/login.po.ts 2024-03-27T21:42:43.825 INFO:teuthology.orchestra.run.smithi060.stdout: 412506 4 -rw-r--r-- 1 ubuntu ubuntu 991 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/user-mgmt.e2e-spec.ts 2024-03-27T21:42:43.825 INFO:teuthology.orchestra.run.smithi060.stdout: 412497 4 -rw-r--r-- 1 ubuntu ubuntu 331 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/language.po.ts 2024-03-27T21:42:43.825 INFO:teuthology.orchestra.run.smithi060.stdout: 412490 4 -rw-r--r-- 1 ubuntu ubuntu 355 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/api-docs.e2e-spec.ts 2024-03-27T21:42:43.825 INFO:teuthology.orchestra.run.smithi060.stdout: 412502 4 -rw-r--r-- 1 ubuntu ubuntu 1586 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/notification.e2e-spec.ts 2024-03-27T21:42:43.826 INFO:teuthology.orchestra.run.smithi060.stdout: 412503 4 -rw-r--r-- 1 ubuntu ubuntu 1127 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/notification.po.ts 2024-03-27T21:42:43.826 INFO:teuthology.orchestra.run.smithi060.stdout: 412496 4 -rw-r--r-- 1 ubuntu ubuntu 515 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/language.e2e-spec.ts 2024-03-27T21:42:43.826 INFO:teuthology.orchestra.run.smithi060.stdout: 412500 4 -rw-r--r-- 1 ubuntu ubuntu 599 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/navigation.e2e-spec.ts 2024-03-27T21:42:43.826 INFO:teuthology.orchestra.run.smithi060.stdout: 412495 4 -rw-r--r-- 1 ubuntu ubuntu 902 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/dashboard.po.ts 2024-03-27T21:42:43.826 INFO:teuthology.orchestra.run.smithi060.stdout: 412507 4 -rw-r--r-- 1 ubuntu ubuntu 1335 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/user-mgmt.po.ts 2024-03-27T21:42:43.826 INFO:teuthology.orchestra.run.smithi060.stdout: 412505 4 -rw-r--r-- 1 ubuntu ubuntu 1254 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/role-mgmt.po.ts 2024-03-27T21:42:43.826 INFO:teuthology.orchestra.run.smithi060.stdout: 412493 4 -rw-r--r-- 1 ubuntu ubuntu 523 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/dashboard-v3.po.ts 2024-03-27T21:42:43.827 INFO:teuthology.orchestra.run.smithi060.stdout: 412504 4 -rw-r--r-- 1 ubuntu ubuntu 963 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/role-mgmt.e2e-spec.ts 2024-03-27T21:42:43.827 INFO:teuthology.orchestra.run.smithi060.stdout: 412491 4 -rw-r--r-- 1 ubuntu ubuntu 166 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/api-docs.po.ts 2024-03-27T21:42:43.827 INFO:teuthology.orchestra.run.smithi060.stdout: 412498 4 -rw-r--r-- 1 ubuntu ubuntu 490 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/login.e2e-spec.ts 2024-03-27T21:42:43.827 INFO:teuthology.orchestra.run.smithi060.stdout: 412494 8 -rw-r--r-- 1 ubuntu ubuntu 4959 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/dashboard.e2e-spec.ts 2024-03-27T21:42:43.827 INFO:teuthology.orchestra.run.smithi060.stdout: 412456 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator 2024-03-27T21:42:43.827 INFO:teuthology.orchestra.run.smithi060.stdout: 412461 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/grafana 2024-03-27T21:42:43.827 INFO:teuthology.orchestra.run.smithi060.stdout: 412462 4 -rw-r--r-- 1 ubuntu ubuntu 2594 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/grafana/grafana.feature 2024-03-27T21:42:43.827 INFO:teuthology.orchestra.run.smithi060.stdout: 412459 4 -rw-r--r-- 1 ubuntu ubuntu 2011 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/04-osds.e2e-spec.ts 2024-03-27T21:42:43.827 INFO:teuthology.orchestra.run.smithi060.stdout: 412457 4 -rw-r--r-- 1 ubuntu ubuntu 1823 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/01-hosts.e2e-spec.ts 2024-03-27T21:42:43.828 INFO:teuthology.orchestra.run.smithi060.stdout: 412460 4 -rw-r--r-- 1 ubuntu ubuntu 910 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/05-services.e2e-spec.ts 2024-03-27T21:42:43.828 INFO:teuthology.orchestra.run.smithi060.stdout: 412463 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/workflow 2024-03-27T21:42:43.828 INFO:teuthology.orchestra.run.smithi060.stdout: 412473 4 -rw-r--r-- 1 ubuntu ubuntu 2656 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/workflow/10-nfs-exports.e2e-spec.ts 2024-03-27T21:42:43.828 INFO:teuthology.orchestra.run.smithi060.stdout: 412467 4 -rw-r--r-- 1 ubuntu ubuntu 1336 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/workflow/04-create-cluster-create-osds.e2e-spec.ts 2024-03-27T21:42:43.828 INFO:teuthology.orchestra.run.smithi060.stdout: 412466 4 -rw-r--r-- 1 ubuntu ubuntu 1450 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/workflow/03-create-cluster-create-services.e2e-spec.ts 2024-03-27T21:42:43.828 INFO:teuthology.orchestra.run.smithi060.stdout: 412465 4 -rw-r--r-- 1 ubuntu ubuntu 2851 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/workflow/02-create-cluster-add-host.feature 2024-03-27T21:42:43.828 INFO:teuthology.orchestra.run.smithi060.stdout: 412468 4 -rw-r--r-- 1 ubuntu ubuntu 2191 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/workflow/05-create-cluster-review.e2e-spec.ts 2024-03-27T21:42:43.828 INFO:teuthology.orchestra.run.smithi060.stdout: 412469 4 -rw-r--r-- 1 ubuntu ubuntu 2740 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/workflow/06-cluster-check.e2e-spec.ts 2024-03-27T21:42:43.829 INFO:teuthology.orchestra.run.smithi060.stdout: 412472 8 -rw-r--r-- 1 ubuntu ubuntu 4296 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/workflow/09-services.e2e-spec.ts 2024-03-27T21:42:43.829 INFO:teuthology.orchestra.run.smithi060.stdout: 412471 4 -rw-r--r-- 1 ubuntu ubuntu 1418 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/workflow/08-hosts.e2e-spec.ts 2024-03-27T21:42:43.829 INFO:teuthology.orchestra.run.smithi060.stdout: 412464 4 -rw-r--r-- 1 ubuntu ubuntu 956 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/workflow/01-create-cluster-welcome.feature 2024-03-27T21:42:43.829 INFO:teuthology.orchestra.run.smithi060.stdout: 412470 4 -rw-r--r-- 1 ubuntu ubuntu 628 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/workflow/07-osds.e2e-spec.ts 2024-03-27T21:42:43.829 INFO:teuthology.orchestra.run.smithi060.stdout: 412474 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/workflow/nfs 2024-03-27T21:42:43.829 INFO:teuthology.orchestra.run.smithi060.stdout: 412475 4 -rw-r--r-- 1 ubuntu ubuntu 1655 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/workflow/nfs/nfs-export.po.ts 2024-03-27T21:42:43.829 INFO:teuthology.orchestra.run.smithi060.stdout: 412458 4 -rw-r--r-- 1 ubuntu ubuntu 730 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/03-inventory.e2e-spec.ts 2024-03-27T21:42:43.829 INFO:teuthology.orchestra.run.smithi060.stdout: 412477 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/pools 2024-03-27T21:42:43.829 INFO:teuthology.orchestra.run.smithi060.stdout: 412479 4 -rw-r--r-- 1 ubuntu ubuntu 2095 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/pools/pools.po.ts 2024-03-27T21:42:43.830 INFO:teuthology.orchestra.run.smithi060.stdout: 412478 4 -rw-r--r-- 1 ubuntu ubuntu 1413 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/pools/pools.e2e-spec.ts 2024-03-27T21:42:43.830 INFO:teuthology.orchestra.run.smithi060.stdout: 412423 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster 2024-03-27T21:42:43.830 INFO:teuthology.orchestra.run.smithi060.stdout: 412432 4 -rw-r--r-- 1 ubuntu ubuntu 1664 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/logs.e2e-spec.ts 2024-03-27T21:42:43.830 INFO:teuthology.orchestra.run.smithi060.stdout: 412442 4 -rw-r--r-- 1 ubuntu ubuntu 1829 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/users.po.ts 2024-03-27T21:42:43.830 INFO:teuthology.orchestra.run.smithi060.stdout: 412427 4 -rw-r--r-- 1 ubuntu ubuntu 959 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/crush-map.e2e-spec.ts 2024-03-27T21:42:43.830 INFO:teuthology.orchestra.run.smithi060.stdout: 412428 4 -rw-r--r-- 1 ubuntu ubuntu 331 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/crush-map.po.ts 2024-03-27T21:42:43.830 INFO:teuthology.orchestra.run.smithi060.stdout: 412435 4 -rw-r--r-- 1 ubuntu ubuntu 1544 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/mgr-modules.po.ts 2024-03-27T21:42:43.830 INFO:teuthology.orchestra.run.smithi060.stdout: 412440 8 -rw-r--r-- 1 ubuntu ubuntu 6410 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/services.po.ts 2024-03-27T21:42:43.830 INFO:teuthology.orchestra.run.smithi060.stdout: 412425 4 -rw-r--r-- 1 ubuntu ubuntu 2639 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/configuration.po.ts 2024-03-27T21:42:43.831 INFO:teuthology.orchestra.run.smithi060.stdout: 412438 4 -rw-r--r-- 1 ubuntu ubuntu 1587 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/osds.e2e-spec.ts 2024-03-27T21:42:43.831 INFO:teuthology.orchestra.run.smithi060.stdout: 412424 4 -rw-r--r-- 1 ubuntu ubuntu 2372 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/configuration.e2e-spec.ts 2024-03-27T21:42:43.831 INFO:teuthology.orchestra.run.smithi060.stdout: 412426 4 -rw-r--r-- 1 ubuntu ubuntu 1512 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/create-cluster.po.ts 2024-03-27T21:42:43.831 INFO:teuthology.orchestra.run.smithi060.stdout: 412439 4 -rw-r--r-- 1 ubuntu ubuntu 2614 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/osds.po.ts 2024-03-27T21:42:43.831 INFO:teuthology.orchestra.run.smithi060.stdout: 412429 4 -rw-r--r-- 1 ubuntu ubuntu 837 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/hosts.e2e-spec.ts 2024-03-27T21:42:43.831 INFO:teuthology.orchestra.run.smithi060.stdout: 412434 4 -rw-r--r-- 1 ubuntu ubuntu 1889 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/mgr-modules.e2e-spec.ts 2024-03-27T21:42:43.831 INFO:teuthology.orchestra.run.smithi060.stdout: 412433 4 -rw-r--r-- 1 ubuntu ubuntu 2186 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/logs.po.ts 2024-03-27T21:42:43.832 INFO:teuthology.orchestra.run.smithi060.stdout: 412431 4 -rw-r--r-- 1 ubuntu ubuntu 636 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/inventory.po.ts 2024-03-27T21:42:43.832 INFO:teuthology.orchestra.run.smithi060.stdout: 412430 8 -rw-r--r-- 1 ubuntu ubuntu 6100 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/hosts.po.ts 2024-03-27T21:42:43.832 INFO:teuthology.orchestra.run.smithi060.stdout: 412441 4 -rw-r--r-- 1 ubuntu ubuntu 1172 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/users.e2e-spec.ts 2024-03-27T21:42:43.832 INFO:teuthology.orchestra.run.smithi060.stdout: 412436 4 -rw-r--r-- 1 ubuntu ubuntu 1921 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/monitors.e2e-spec.ts 2024-03-27T21:42:43.832 INFO:teuthology.orchestra.run.smithi060.stdout: 412437 4 -rw-r--r-- 1 ubuntu ubuntu 171 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/monitors.po.ts 2024-03-27T21:42:43.832 INFO:teuthology.orchestra.run.smithi060.stdout: 412443 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/common 2024-03-27T21:42:43.832 INFO:teuthology.orchestra.run.smithi060.stdout: 412446 4 -rw-r--r-- 1 ubuntu ubuntu 2385 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/common/forms-helper.feature.po.ts 2024-03-27T21:42:43.832 INFO:teuthology.orchestra.run.smithi060.stdout: 412444 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/common/create-cluster 2024-03-27T21:42:43.833 INFO:teuthology.orchestra.run.smithi060.stdout: 412445 4 -rw-r--r-- 1 ubuntu ubuntu 447 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/common/create-cluster/create-cluster.feature.po.ts 2024-03-27T21:42:43.833 INFO:teuthology.orchestra.run.smithi060.stdout: 412449 8 -rw-r--r-- 1 ubuntu ubuntu 5306 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/common/table-helper.feature.po.ts 2024-03-27T21:42:43.833 INFO:teuthology.orchestra.run.smithi060.stdout: 412450 4 -rw-r--r-- 1 ubuntu ubuntu 1408 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/common/urls.po.ts 2024-03-27T21:42:43.833 INFO:teuthology.orchestra.run.smithi060.stdout: 412447 4 -rw-r--r-- 1 ubuntu ubuntu 1073 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/common/global.feature.po.ts 2024-03-27T21:42:43.833 INFO:teuthology.orchestra.run.smithi060.stdout: 412448 4 -rw-r--r-- 1 ubuntu ubuntu 2391 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/e2e/common/grafana.feature.po.ts 2024-03-27T21:42:43.833 INFO:teuthology.orchestra.run.smithi060.stdout: 414306 4 -rw-r--r-- 1 ubuntu ubuntu 3969 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/tox.ini 2024-03-27T21:42:43.833 INFO:teuthology.orchestra.run.smithi060.stdout: 414230 8 -rwxr-xr-x 1 ubuntu ubuntu 5466 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/run-backend-api-tests.sh 2024-03-27T21:42:43.833 INFO:teuthology.orchestra.run.smithi060.stdout: 414226 4 -rw-r--r-- 1 ubuntu ubuntu 71 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/requirements-test.txt 2024-03-27T21:42:43.834 INFO:teuthology.orchestra.run.smithi060.stdout: 412318 4 -rw-r--r-- 1 ubuntu ubuntu 654 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/.editorconfig 2024-03-27T21:42:43.834 INFO:teuthology.orchestra.run.smithi060.stdout: 412323 4 -rw-r--r-- 1 ubuntu ubuntu 1403 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/README.rst 2024-03-27T21:42:43.834 INFO:teuthology.orchestra.run.smithi060.stdout: 414227 4 -rw-r--r-- 1 ubuntu ubuntu 181 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/requirements.txt 2024-03-27T21:42:43.834 INFO:teuthology.orchestra.run.smithi060.stdout: 412338 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers 2024-03-27T21:42:43.834 INFO:teuthology.orchestra.run.smithi060.stdout: 412368 8 -rw-r--r-- 1 ubuntu ubuntu 5379 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/home.py 2024-03-27T21:42:43.834 INFO:teuthology.orchestra.run.smithi060.stdout: 412347 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/_paginate.py 2024-03-27T21:42:43.834 INFO:teuthology.orchestra.run.smithi060.stdout: 412391 12 -rw-r--r-- 1 ubuntu ubuntu 8363 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/telemetry.py 2024-03-27T21:42:43.834 INFO:teuthology.orchestra.run.smithi060.stdout: 412341 4 -rw-r--r-- 1 ubuntu ubuntu 658 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/_auth.py 2024-03-27T21:42:43.835 INFO:teuthology.orchestra.run.smithi060.stdout: 412358 8 -rw-r--r-- 1 ubuntu ubuntu 5085 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/cluster_configuration.py 2024-03-27T21:42:43.835 INFO:teuthology.orchestra.run.smithi060.stdout: 412365 4 -rw-r--r-- 1 ubuntu ubuntu 1636 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/grafana.py 2024-03-27T21:42:43.835 INFO:teuthology.orchestra.run.smithi060.stdout: 412348 4 -rw-r--r-- 1 ubuntu ubuntu 1618 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/_permissions.py 2024-03-27T21:42:43.835 INFO:teuthology.orchestra.run.smithi060.stdout: 412392 12 -rw-r--r-- 1 ubuntu ubuntu 8616 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/user.py 2024-03-27T21:42:43.835 INFO:teuthology.orchestra.run.smithi060.stdout: 412382 20 -rw-r--r-- 1 ubuntu ubuntu 18812 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/rbd.py 2024-03-27T21:42:43.835 INFO:teuthology.orchestra.run.smithi060.stdout: 412349 12 -rw-r--r-- 1 ubuntu ubuntu 10522 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/_rest_controller.py 2024-03-27T21:42:43.835 INFO:teuthology.orchestra.run.smithi060.stdout: 412369 20 -rw-r--r-- 1 ubuntu ubuntu 19736 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/host.py 2024-03-27T21:42:43.835 INFO:teuthology.orchestra.run.smithi060.stdout: 412388 8 -rw-r--r-- 1 ubuntu ubuntu 4271 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/settings.py 2024-03-27T21:42:43.835 INFO:teuthology.orchestra.run.smithi060.stdout: 412352 4 -rw-r--r-- 1 ubuntu ubuntu 384 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/_ui_router.py 2024-03-27T21:42:43.836 INFO:teuthology.orchestra.run.smithi060.stdout: 412380 16 -rw-r--r-- 1 ubuntu ubuntu 14350 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/pool.py 2024-03-27T21:42:43.836 INFO:teuthology.orchestra.run.smithi060.stdout: 412386 8 -rw-r--r-- 1 ubuntu ubuntu 4462 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/saml2.py 2024-03-27T21:42:43.836 INFO:teuthology.orchestra.run.smithi060.stdout: 412372 8 -rw-r--r-- 1 ubuntu ubuntu 7733 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/mgr_modules.py 2024-03-27T21:42:43.836 INFO:teuthology.orchestra.run.smithi060.stdout: 412359 4 -rw-r--r-- 1 ubuntu ubuntu 2213 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/crush_rule.py 2024-03-27T21:42:43.836 INFO:teuthology.orchestra.run.smithi060.stdout: 412353 4 -rw-r--r-- 1 ubuntu ubuntu 2038 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/_version.py 2024-03-27T21:42:43.836 INFO:teuthology.orchestra.run.smithi060.stdout: 412370 56 -rw-r--r-- 1 ubuntu ubuntu 55574 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/iscsi.py 2024-03-27T21:42:43.836 INFO:teuthology.orchestra.run.smithi060.stdout: 412343 20 -rw-r--r-- 1 ubuntu ubuntu 17507 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/_crud.py 2024-03-27T21:42:43.836 INFO:teuthology.orchestra.run.smithi060.stdout: 412374 16 -rw-r--r-- 1 ubuntu ubuntu 13039 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/multi_cluster.py 2024-03-27T21:42:43.836 INFO:teuthology.orchestra.run.smithi060.stdout: 412377 4 -rw-r--r-- 1 ubuntu ubuntu 1906 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/orchestrator.py 2024-03-27T21:42:43.837 INFO:teuthology.orchestra.run.smithi060.stdout: 412361 20 -rw-r--r-- 1 ubuntu ubuntu 16781 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/docs.py 2024-03-27T21:42:43.837 INFO:teuthology.orchestra.run.smithi060.stdout: 412389 8 -rw-r--r-- 1 ubuntu ubuntu 4446 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/summary.py 2024-03-27T21:42:43.837 INFO:teuthology.orchestra.run.smithi060.stdout: 412381 8 -rw-r--r-- 1 ubuntu ubuntu 7691 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/prometheus.py 2024-03-27T21:42:43.837 INFO:teuthology.orchestra.run.smithi060.stdout: 412342 12 -rw-r--r-- 1 ubuntu ubuntu 10918 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/_base_controller.py 2024-03-27T21:42:43.837 INFO:teuthology.orchestra.run.smithi060.stdout: 412379 4 -rw-r--r-- 1 ubuntu ubuntu 2362 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/perf_counters.py 2024-03-27T21:42:43.837 INFO:teuthology.orchestra.run.smithi060.stdout: 412384 48 -rw-r--r-- 1 ubuntu ubuntu 45179 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/rgw.py 2024-03-27T21:42:43.837 INFO:teuthology.orchestra.run.smithi060.stdout: 412357 4 -rw-r--r-- 1 ubuntu ubuntu 3916 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/cluster.py 2024-03-27T21:42:43.837 INFO:teuthology.orchestra.run.smithi060.stdout: 412375 12 -rw-r--r-- 1 ubuntu ubuntu 10503 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/nfs.py 2024-03-27T21:42:43.838 INFO:teuthology.orchestra.run.smithi060.stdout: 412363 8 -rw-r--r-- 1 ubuntu ubuntu 4560 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/feedback.py 2024-03-27T21:42:43.838 INFO:teuthology.orchestra.run.smithi060.stdout: 412376 16 -rw-r--r-- 1 ubuntu ubuntu 14483 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/nvmeof.py 2024-03-27T21:42:43.838 INFO:teuthology.orchestra.run.smithi060.stdout: 412339 4 -rwxr-xr-x 1 ubuntu ubuntu 1170 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/__init__.py 2024-03-27T21:42:43.838 INFO:teuthology.orchestra.run.smithi060.stdout: 412367 12 -rw-r--r-- 1 ubuntu ubuntu 9694 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/health.py 2024-03-27T21:42:43.838 INFO:teuthology.orchestra.run.smithi060.stdout: 412390 4 -rw-r--r-- 1 ubuntu ubuntu 1372 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/task.py 2024-03-27T21:42:43.838 INFO:teuthology.orchestra.run.smithi060.stdout: 412350 4 -rw-r--r-- 1 ubuntu ubuntu 2158 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/_router.py 2024-03-27T21:42:43.838 INFO:teuthology.orchestra.run.smithi060.stdout: 412362 4 -rw-r--r-- 1 ubuntu ubuntu 2324 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/erasure_code_profile.py 2024-03-27T21:42:43.838 INFO:teuthology.orchestra.run.smithi060.stdout: 412383 24 -rw-r--r-- 1 ubuntu ubuntu 23832 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/rbd_mirroring.py 2024-03-27T21:42:43.839 INFO:teuthology.orchestra.run.smithi060.stdout: 412360 4 -rw-r--r-- 1 ubuntu ubuntu 1995 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/daemon.py 2024-03-27T21:42:43.839 INFO:teuthology.orchestra.run.smithi060.stdout: 412366 4 -rw-r--r-- 1 ubuntu ubuntu 852 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/hardware.py 2024-03-27T21:42:43.839 INFO:teuthology.orchestra.run.smithi060.stdout: 412373 4 -rw-r--r-- 1 ubuntu ubuntu 4011 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/monitor.py 2024-03-27T21:42:43.839 INFO:teuthology.orchestra.run.smithi060.stdout: 412345 4 -rw-r--r-- 1 ubuntu ubuntu 2736 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/_endpoint.py 2024-03-27T21:42:43.839 INFO:teuthology.orchestra.run.smithi060.stdout: 412378 28 -rw-r--r-- 1 ubuntu ubuntu 24994 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/osd.py 2024-03-27T21:42:43.839 INFO:teuthology.orchestra.run.smithi060.stdout: 412346 4 -rw-r--r-- 1 ubuntu ubuntu 3990 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/_helpers.py 2024-03-27T21:42:43.839 INFO:teuthology.orchestra.run.smithi060.stdout: 412364 4 -rw-r--r-- 1 ubuntu ubuntu 352 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/frontend_logging.py 2024-03-27T21:42:43.839 INFO:teuthology.orchestra.run.smithi060.stdout: 412356 48 -rw-r--r-- 1 ubuntu ubuntu 47353 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/cephfs.py 2024-03-27T21:42:43.840 INFO:teuthology.orchestra.run.smithi060.stdout: 412385 8 -rw-r--r-- 1 ubuntu ubuntu 5530 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/role.py 2024-03-27T21:42:43.840 INFO:teuthology.orchestra.run.smithi060.stdout: 412371 4 -rw-r--r-- 1 ubuntu ubuntu 2117 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/logs.py 2024-03-27T21:42:43.840 INFO:teuthology.orchestra.run.smithi060.stdout: 412344 8 -rw-r--r-- 1 ubuntu ubuntu 5748 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/_docs.py 2024-03-27T21:42:43.840 INFO:teuthology.orchestra.run.smithi060.stdout: 412351 4 -rw-r--r-- 1 ubuntu ubuntu 3121 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/_task.py 2024-03-27T21:42:43.840 INFO:teuthology.orchestra.run.smithi060.stdout: 412340 4 -rw-r--r-- 1 ubuntu ubuntu 381 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/_api_router.py 2024-03-27T21:42:43.840 INFO:teuthology.orchestra.run.smithi060.stdout: 412355 12 -rw-r--r-- 1 ubuntu ubuntu 8193 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/ceph_users.py 2024-03-27T21:42:43.840 INFO:teuthology.orchestra.run.smithi060.stdout: 412354 8 -rw-r--r-- 1 ubuntu ubuntu 7153 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/auth.py 2024-03-27T21:42:43.840 INFO:teuthology.orchestra.run.smithi060.stdout: 412387 4 -rw-r--r-- 1 ubuntu ubuntu 3866 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/controllers/service.py 2024-03-27T21:42:43.840 INFO:teuthology.orchestra.run.smithi060.stdout: 414231 4 -rwxr-xr-x 1 ubuntu ubuntu 1350 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/run-backend-rook-api-request.sh 2024-03-27T21:42:43.841 INFO:teuthology.orchestra.run.smithi060.stdout: 412317 4 -rw-r--r-- 1 ubuntu ubuntu 110 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/.coveragerc 2024-03-27T21:42:43.841 INFO:teuthology.orchestra.run.smithi060.stdout: 412329 12 -rw-r--r-- 1 ubuntu ubuntu 8332 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/cherrypy_backports.py 2024-03-27T21:42:43.841 INFO:teuthology.orchestra.run.smithi060.stdout: 412320 20 -rw-r--r-- 1 ubuntu ubuntu 16523 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/.pylintrc 2024-03-27T21:42:43.841 INFO:teuthology.orchestra.run.smithi060.stdout: 412330 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/ci 2024-03-27T21:42:43.841 INFO:teuthology.orchestra.run.smithi060.stdout: 412336 8 -rw-r--r-- 1 ubuntu ubuntu 6136 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/ci/check_grafana_dashboards.py 2024-03-27T21:42:43.841 INFO:teuthology.orchestra.run.smithi060.stdout: 412331 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/ci/cephadm 2024-03-27T21:42:43.841 INFO:teuthology.orchestra.run.smithi060.stdout: 412332 4 -rwxr-xr-x 1 ubuntu ubuntu 1396 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/ci/cephadm/bootstrap-cluster.sh 2024-03-27T21:42:43.841 INFO:teuthology.orchestra.run.smithi060.stdout: 412335 4 -rwxr-xr-x 1 ubuntu ubuntu 2983 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/ci/cephadm/start-cluster.sh 2024-03-27T21:42:43.841 INFO:teuthology.orchestra.run.smithi060.stdout: 412333 4 -rwxr-xr-x 1 ubuntu ubuntu 937 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/ci/cephadm/ceph_cluster.yml 2024-03-27T21:42:43.842 INFO:teuthology.orchestra.run.smithi060.stdout: 412334 4 -rwxr-xr-x 1 ubuntu ubuntu 2417 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/ci/cephadm/run-cephadm-e2e-tests.sh 2024-03-27T21:42:43.842 INFO:teuthology.orchestra.run.smithi060.stdout: 412324 4 -rw-r--r-- 1 ubuntu ubuntu 1814 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/__init__.py 2024-03-27T21:42:43.842 INFO:teuthology.orchestra.run.smithi060.stdout: 414213 464 -rw-r--r-- 1 ubuntu ubuntu 472896 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/openapi.yaml 2024-03-27T21:42:43.842 INFO:teuthology.orchestra.run.smithi060.stdout: 414235 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services 2024-03-27T21:42:43.842 INFO:teuthology.orchestra.run.smithi060.stdout: 414237 4 -rw-r--r-- 1 ubuntu ubuntu 2314 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/_paginate.py 2024-03-27T21:42:43.842 INFO:teuthology.orchestra.run.smithi060.stdout: 414261 12 -rw-r--r-- 1 ubuntu ubuntu 9765 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/sso.py 2024-03-27T21:42:43.842 INFO:teuthology.orchestra.run.smithi060.stdout: 414258 32 -rw-r--r-- 1 ubuntu ubuntu 30452 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/rbd.py 2024-03-27T21:42:43.842 INFO:teuthology.orchestra.run.smithi060.stdout: 414245 4 -rw-r--r-- 1 ubuntu ubuntu 1895 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/iscsi_cli.py 2024-03-27T21:42:43.843 INFO:teuthology.orchestra.run.smithi060.stdout: 414249 8 -rw-r--r-- 1 ubuntu ubuntu 4485 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/nvmeof_client.py 2024-03-27T21:42:43.843 INFO:teuthology.orchestra.run.smithi060.stdout: 414262 8 -rw-r--r-- 1 ubuntu ubuntu 4535 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/tcmu_service.py 2024-03-27T21:42:43.843 INFO:teuthology.orchestra.run.smithi060.stdout: 414260 4 -rw-r--r-- 1 ubuntu ubuntu 782 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/settings.py 2024-03-27T21:42:43.843 INFO:teuthology.orchestra.run.smithi060.stdout: 414246 12 -rw-r--r-- 1 ubuntu ubuntu 10116 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/iscsi_client.py 2024-03-27T21:42:43.843 INFO:teuthology.orchestra.run.smithi060.stdout: 414240 24 -rw-r--r-- 1 ubuntu ubuntu 24337 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/ceph_service.py 2024-03-27T21:42:43.843 INFO:teuthology.orchestra.run.smithi060.stdout: 414251 12 -rw-r--r-- 1 ubuntu ubuntu 9941 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/orchestrator.py 2024-03-27T21:42:43.843 INFO:teuthology.orchestra.run.smithi060.stdout: 414259 84 -rw-r--r-- 1 ubuntu ubuntu 82452 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/rgw_client.py 2024-03-27T21:42:43.843 INFO:teuthology.orchestra.run.smithi060.stdout: 414247 4 -rw-r--r-- 1 ubuntu ubuntu 3920 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/iscsi_config.py 2024-03-27T21:42:43.843 INFO:teuthology.orchestra.run.smithi060.stdout: 414253 4 -rw-r--r-- 1 ubuntu ubuntu 3309 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/progress.py 2024-03-27T21:42:43.844 INFO:teuthology.orchestra.run.smithi060.stdout: 414242 4 -rw-r--r-- 1 ubuntu ubuntu 2846 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/cluster.py 2024-03-27T21:42:43.844 INFO:teuthology.orchestra.run.smithi060.stdout: 414236 4 -rw-r--r-- 1 ubuntu ubuntu 24 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/__init__.py 2024-03-27T21:42:43.844 INFO:teuthology.orchestra.run.smithi060.stdout: 414244 4 -rw-r--r-- 1 ubuntu ubuntu 2819 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/hardware.py 2024-03-27T21:42:43.844 INFO:teuthology.orchestra.run.smithi060.stdout: 414243 4 -rw-r--r-- 1 ubuntu ubuntu 4054 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/exception.py 2024-03-27T21:42:43.844 INFO:teuthology.orchestra.run.smithi060.stdout: 414250 4 -rw-r--r-- 1 ubuntu ubuntu 1760 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/nvmeof_conf.py 2024-03-27T21:42:43.844 INFO:teuthology.orchestra.run.smithi060.stdout: 414252 4 -rw-r--r-- 1 ubuntu ubuntu 692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/osd.py 2024-03-27T21:42:43.844 INFO:teuthology.orchestra.run.smithi060.stdout: 414254 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/proto 2024-03-27T21:42:43.845 INFO:teuthology.orchestra.run.smithi060.stdout: 414257 40 -rw-r--r-- 1 ubuntu ubuntu 40598 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/proto/gateway_pb2_grpc.py 2024-03-27T21:42:43.845 INFO:teuthology.orchestra.run.smithi060.stdout: 414256 180 -rw-r--r-- 1 ubuntu ubuntu 181772 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/proto/gateway_pb2.py 2024-03-27T21:42:43.845 INFO:teuthology.orchestra.run.smithi060.stdout: 414255 12 -rw-r--r-- 1 ubuntu ubuntu 10533 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/proto/gateway.proto 2024-03-27T21:42:43.845 INFO:teuthology.orchestra.run.smithi060.stdout: 414241 12 -rw-r--r-- 1 ubuntu ubuntu 10442 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/cephfs.py 2024-03-27T21:42:43.845 INFO:teuthology.orchestra.run.smithi060.stdout: 414238 36 -rw-r--r-- 1 ubuntu ubuntu 33237 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/access_control.py 2024-03-27T21:42:43.845 INFO:teuthology.orchestra.run.smithi060.stdout: 414248 4 -rw-r--r-- 1 ubuntu ubuntu 1441 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/nvmeof_cli.py 2024-03-27T21:42:43.845 INFO:teuthology.orchestra.run.smithi060.stdout: 414239 12 -rw-r--r-- 1 ubuntu ubuntu 9730 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/services/auth.py 2024-03-27T21:42:43.845 INFO:teuthology.orchestra.run.smithi060.stdout: 412322 4 -rw-r--r-- 1 ubuntu ubuntu 473 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/HACKING.rst 2024-03-27T21:42:43.846 INFO:teuthology.orchestra.run.smithi060.stdout: 414209 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/model 2024-03-27T21:42:43.846 INFO:teuthology.orchestra.run.smithi060.stdout: 414211 4 -rw-r--r-- 1 ubuntu ubuntu 1845 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/model/nvmeof.py 2024-03-27T21:42:43.846 INFO:teuthology.orchestra.run.smithi060.stdout: 414210 4 -rw-r--r-- 1 ubuntu ubuntu 24 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/model/__init__.py 2024-03-27T21:42:43.846 INFO:teuthology.orchestra.run.smithi060.stdout: 414232 4 -rwxr-xr-x 1 ubuntu ubuntu 3901 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/run-frontend-e2e-tests.sh 2024-03-27T21:42:43.846 INFO:teuthology.orchestra.run.smithi060.stdout: 412319 4 -rw-r--r-- 1 ubuntu ubuntu 109 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/.gitignore 2024-03-27T21:42:43.846 INFO:teuthology.orchestra.run.smithi060.stdout: 414228 24 -rw-r--r-- 1 ubuntu ubuntu 21421 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/rest_client.py 2024-03-27T21:42:43.846 INFO:teuthology.orchestra.run.smithi060.stdout: 412393 4 -rw-r--r-- 1 ubuntu ubuntu 3670 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/exceptions.py 2024-03-27T21:42:43.846 INFO:teuthology.orchestra.run.smithi060.stdout: 412325 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/api 2024-03-27T21:42:43.847 INFO:teuthology.orchestra.run.smithi060.stdout: 412327 4 -rw-r--r-- 1 ubuntu ubuntu 1426 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/api/doc.py 2024-03-27T21:42:43.847 INFO:teuthology.orchestra.run.smithi060.stdout: 412326 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/api/__init__.py 2024-03-27T21:42:43.847 INFO:teuthology.orchestra.run.smithi060.stdout: 414229 4 -rwxr-xr-x 1 ubuntu ubuntu 700 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/run-backend-api-request.sh 2024-03-27T21:42:43.847 INFO:teuthology.orchestra.run.smithi060.stdout: 414645 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/zabbix 2024-03-27T21:42:43.847 INFO:teuthology.orchestra.run.smithi060.stdout: 414647 20 -rw-r--r-- 1 ubuntu ubuntu 17629 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/zabbix/module.py 2024-03-27T21:42:43.850 INFO:teuthology.orchestra.run.smithi060.stdout: 414648 140 -rw-r--r-- 1 ubuntu ubuntu 139526 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/zabbix/zabbix_template.xml 2024-03-27T21:42:43.850 INFO:teuthology.orchestra.run.smithi060.stdout: 414646 4 -rw-r--r-- 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/zabbix/__init__.py 2024-03-27T21:42:43.850 INFO:teuthology.orchestra.run.smithi060.stdout: 412216 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/balancer 2024-03-27T21:42:43.850 INFO:teuthology.orchestra.run.smithi060.stdout: 412218 72 -rw-r--r-- 1 ubuntu ubuntu 71316 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/balancer/module.py 2024-03-27T21:42:43.850 INFO:teuthology.orchestra.run.smithi060.stdout: 412217 4 -rw-r--r-- 1 ubuntu ubuntu 42 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/balancer/__init__.py 2024-03-27T21:42:43.850 INFO:teuthology.orchestra.run.smithi060.stdout: 414407 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/influx 2024-03-27T21:42:43.851 INFO:teuthology.orchestra.run.smithi060.stdout: 414409 20 -rw-r--r-- 1 ubuntu ubuntu 18317 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/influx/module.py 2024-03-27T21:42:43.851 INFO:teuthology.orchestra.run.smithi060.stdout: 414408 4 -rw-r--r-- 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/influx/__init__.py 2024-03-27T21:42:43.851 INFO:teuthology.orchestra.run.smithi060.stdout: 412213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/alerts 2024-03-27T21:42:43.851 INFO:teuthology.orchestra.run.smithi060.stdout: 412215 12 -rw-r--r-- 1 ubuntu ubuntu 9169 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/alerts/module.py 2024-03-27T21:42:43.851 INFO:teuthology.orchestra.run.smithi060.stdout: 412214 4 -rw-r--r-- 1 ubuntu ubuntu 42 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/alerts/__init__.py 2024-03-27T21:42:43.851 INFO:teuthology.orchestra.run.smithi060.stdout: 414417 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/iostat 2024-03-27T21:42:43.851 INFO:teuthology.orchestra.run.smithi060.stdout: 414419 4 -rw-r--r-- 1 ubuntu ubuntu 2145 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/iostat/module.py 2024-03-27T21:42:43.851 INFO:teuthology.orchestra.run.smithi060.stdout: 414418 4 -rw-r--r-- 1 ubuntu ubuntu 42 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/iostat/__init__.py 2024-03-27T21:42:43.852 INFO:teuthology.orchestra.run.smithi060.stdout: 414464 28 -rw-r--r-- 1 ubuntu ubuntu 24946 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/object_format.py 2024-03-27T21:42:43.852 INFO:teuthology.orchestra.run.smithi060.stdout: 414404 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/hello 2024-03-27T21:42:43.852 INFO:teuthology.orchestra.run.smithi060.stdout: 414406 8 -rw-r--r-- 1 ubuntu ubuntu 4515 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/hello/module.py 2024-03-27T21:42:43.852 INFO:teuthology.orchestra.run.smithi060.stdout: 414405 4 -rw-r--r-- 1 ubuntu ubuntu 41 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/hello/__init__.py 2024-03-27T21:42:43.852 INFO:teuthology.orchestra.run.smithi060.stdout: 414524 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rgw 2024-03-27T21:42:43.852 INFO:teuthology.orchestra.run.smithi060.stdout: 414526 20 -rw-r--r-- 1 ubuntu ubuntu 18831 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rgw/module.py 2024-03-27T21:42:43.852 INFO:teuthology.orchestra.run.smithi060.stdout: 414525 4 -rw-r--r-- 1 ubuntu ubuntu 42 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rgw/__init__.py 2024-03-27T21:42:43.852 INFO:teuthology.orchestra.run.smithi060.stdout: 414495 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rbd_support 2024-03-27T21:42:43.853 INFO:teuthology.orchestra.run.smithi060.stdout: 414499 16 -rw-r--r-- 1 ubuntu ubuntu 12491 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rbd_support/module.py 2024-03-27T21:42:43.853 INFO:teuthology.orchestra.run.smithi060.stdout: 414503 12 -rw-r--r-- 1 ubuntu ubuntu 10674 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rbd_support/trash_purge_schedule.py 2024-03-27T21:42:43.853 INFO:teuthology.orchestra.run.smithi060.stdout: 414501 24 -rw-r--r-- 1 ubuntu ubuntu 22757 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rbd_support/schedule.py 2024-03-27T21:42:43.853 INFO:teuthology.orchestra.run.smithi060.stdout: 414497 4 -rw-r--r-- 1 ubuntu ubuntu 1364 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rbd_support/common.py 2024-03-27T21:42:43.853 INFO:teuthology.orchestra.run.smithi060.stdout: 414500 24 -rw-r--r-- 1 ubuntu ubuntu 22015 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rbd_support/perf.py 2024-03-27T21:42:43.853 INFO:teuthology.orchestra.run.smithi060.stdout: 414496 4 -rw-r--r-- 1 ubuntu ubuntu 42 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rbd_support/__init__.py 2024-03-27T21:42:43.853 INFO:teuthology.orchestra.run.smithi060.stdout: 414502 36 -rw-r--r-- 1 ubuntu ubuntu 33972 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rbd_support/task.py 2024-03-27T21:42:43.853 INFO:teuthology.orchestra.run.smithi060.stdout: 414498 24 -rw-r--r-- 1 ubuntu ubuntu 24341 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rbd_support/mirror_snapshot_schedule.py 2024-03-27T21:42:43.854 INFO:teuthology.orchestra.run.smithi060.stdout: 414487 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/progress 2024-03-27T21:42:43.854 INFO:teuthology.orchestra.run.smithi060.stdout: 414489 32 -rw-r--r-- 1 ubuntu ubuntu 30853 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/progress/module.py 2024-03-27T21:42:43.854 INFO:teuthology.orchestra.run.smithi060.stdout: 414490 8 -rw-r--r-- 1 ubuntu ubuntu 5364 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/progress/test_progress.py 2024-03-27T21:42:43.854 INFO:teuthology.orchestra.run.smithi060.stdout: 414488 4 -rw-r--r-- 1 ubuntu ubuntu 86 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/progress/__init__.py 2024-03-27T21:42:43.854 INFO:teuthology.orchestra.run.smithi060.stdout: 414479 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/pg_autoscaler 2024-03-27T21:42:43.854 INFO:teuthology.orchestra.run.smithi060.stdout: 414481 36 -rw-r--r-- 1 ubuntu ubuntu 34807 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/pg_autoscaler/module.py 2024-03-27T21:42:43.854 INFO:teuthology.orchestra.run.smithi060.stdout: 414482 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/pg_autoscaler/tests 2024-03-27T21:42:43.855 INFO:teuthology.orchestra.run.smithi060.stdout: 414486 20 -rw-r--r-- 1 ubuntu ubuntu 16543 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/pg_autoscaler/tests/test_overlapping_roots.py 2024-03-27T21:42:43.855 INFO:teuthology.orchestra.run.smithi060.stdout: 414485 4 -rw-r--r-- 1 ubuntu ubuntu 1123 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/pg_autoscaler/tests/test_cal_ratio.py 2024-03-27T21:42:43.855 INFO:teuthology.orchestra.run.smithi060.stdout: 414483 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/pg_autoscaler/tests/__init__.py 2024-03-27T21:42:43.855 INFO:teuthology.orchestra.run.smithi060.stdout: 414484 20 -rw-r--r-- 1 ubuntu ubuntu 19491 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/pg_autoscaler/tests/test_cal_final_pg_target.py 2024-03-27T21:42:43.855 INFO:teuthology.orchestra.run.smithi060.stdout: 414480 4 -rw-r--r-- 1 ubuntu ubuntu 115 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/pg_autoscaler/__init__.py 2024-03-27T21:42:43.855 INFO:teuthology.orchestra.run.smithi060.stdout: 414307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/devicehealth 2024-03-27T21:42:43.855 INFO:teuthology.orchestra.run.smithi060.stdout: 414309 32 -rw-r--r-- 1 ubuntu ubuntu 29526 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/devicehealth/module.py 2024-03-27T21:42:43.855 INFO:teuthology.orchestra.run.smithi060.stdout: 414308 4 -rw-r--r-- 1 ubuntu ubuntu 42 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/devicehealth/__init__.py 2024-03-27T21:42:43.855 INFO:teuthology.orchestra.run.smithi060.stdout: 414608 4 -rw-r--r-- 1 ubuntu ubuntu 3564 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/tox.ini 2024-03-27T21:42:43.856 INFO:teuthology.orchestra.run.smithi060.stdout: 414420 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/k8sevents 2024-03-27T21:42:43.856 INFO:teuthology.orchestra.run.smithi060.stdout: 414423 56 -rw-r--r-- 1 ubuntu ubuntu 55816 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/k8sevents/module.py 2024-03-27T21:42:43.856 INFO:teuthology.orchestra.run.smithi060.stdout: 414421 8 -rw-r--r-- 1 ubuntu ubuntu 4591 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/k8sevents/README.md 2024-03-27T21:42:43.856 INFO:teuthology.orchestra.run.smithi060.stdout: 414424 4 -rw-r--r-- 1 ubuntu ubuntu 871 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/k8sevents/rbac_sample.yaml 2024-03-27T21:42:43.856 INFO:teuthology.orchestra.run.smithi060.stdout: 414422 4 -rw-r--r-- 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/k8sevents/__init__.py 2024-03-27T21:42:43.856 INFO:teuthology.orchestra.run.smithi060.stdout: 414435 32 -rw-r--r-- 1 ubuntu ubuntu 31403 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mgr_util.py 2024-03-27T21:42:43.856 INFO:teuthology.orchestra.run.smithi060.stdout: 414527 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook 2024-03-27T21:42:43.856 INFO:teuthology.orchestra.run.smithi060.stdout: 414547 28 -rw-r--r-- 1 ubuntu ubuntu 26252 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/module.py 2024-03-27T21:42:43.857 INFO:teuthology.orchestra.run.smithi060.stdout: 414551 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/tests 2024-03-27T21:42:43.857 INFO:teuthology.orchestra.run.smithi060.stdout: 414552 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/tests/__init__.py 2024-03-27T21:42:43.857 INFO:teuthology.orchestra.run.smithi060.stdout: 414555 8 -rw-r--r-- 1 ubuntu ubuntu 5298 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/tests/test_rook.py 2024-03-27T21:42:43.857 INFO:teuthology.orchestra.run.smithi060.stdout: 414553 4 -rw-r--r-- 1 ubuntu ubuntu 247 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/tests/fixtures.py 2024-03-27T21:42:43.857 INFO:teuthology.orchestra.run.smithi060.stdout: 414554 4 -rw-r--r-- 1 ubuntu ubuntu 3800 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/tests/test_placement.py 2024-03-27T21:42:43.857 INFO:teuthology.orchestra.run.smithi060.stdout: 414529 4 -rw-r--r-- 1 ubuntu ubuntu 842 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/CMakeLists.txt 2024-03-27T21:42:43.857 INFO:teuthology.orchestra.run.smithi060.stdout: 414550 56 -rw-r--r-- 1 ubuntu ubuntu 55862 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/rook_cluster.py 2024-03-27T21:42:43.857 INFO:teuthology.orchestra.run.smithi060.stdout: 414548 4 -rw-r--r-- 1 ubuntu ubuntu 20 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/requirements.txt 2024-03-27T21:42:43.857 INFO:teuthology.orchestra.run.smithi060.stdout: 414531 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/ci 2024-03-27T21:42:43.858 INFO:teuthology.orchestra.run.smithi060.stdout: 414539 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/ci/tests 2024-03-27T21:42:43.858 INFO:teuthology.orchestra.run.smithi060.stdout: 414540 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/ci/tests/features 2024-03-27T21:42:43.858 INFO:teuthology.orchestra.run.smithi060.stdout: 414541 4 -rw-r--r-- 1 ubuntu ubuntu 938 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/ci/tests/features/cluster-prometheus-monitoring.feature 2024-03-27T21:42:43.858 INFO:teuthology.orchestra.run.smithi060.stdout: 414542 4 -rw-r--r-- 1 ubuntu ubuntu 2128 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/ci/tests/features/rook.feature 2024-03-27T21:42:43.858 INFO:teuthology.orchestra.run.smithi060.stdout: 414543 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/ci/tests/features/steps 2024-03-27T21:42:43.858 INFO:teuthology.orchestra.run.smithi060.stdout: 414544 8 -rw-r--r-- 1 ubuntu ubuntu 6121 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/ci/tests/features/steps/implementation.py 2024-03-27T21:42:43.858 INFO:teuthology.orchestra.run.smithi060.stdout: 414545 4 -rw-r--r-- 1 ubuntu ubuntu 1460 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/ci/tests/features/steps/utils.py 2024-03-27T21:42:43.858 INFO:teuthology.orchestra.run.smithi060.stdout: 414533 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/ci/cluster-specs 2024-03-27T21:42:43.859 INFO:teuthology.orchestra.run.smithi060.stdout: 414534 8 -rw-r--r-- 1 ubuntu ubuntu 5568 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/ci/cluster-specs/cluster-on-pvc-minikube.yaml 2024-03-27T21:42:43.859 INFO:teuthology.orchestra.run.smithi060.stdout: 414537 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/ci/scripts 2024-03-27T21:42:43.859 INFO:teuthology.orchestra.run.smithi060.stdout: 414538 8 -rwxr-xr-x 1 ubuntu ubuntu 6904 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/ci/scripts/bootstrap-rook-cluster.sh 2024-03-27T21:42:43.859 INFO:teuthology.orchestra.run.smithi060.stdout: 414535 4 -rw-r--r-- 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/ci/requirements.txt 2024-03-27T21:42:43.859 INFO:teuthology.orchestra.run.smithi060.stdout: 414536 4 -rwxr-xr-x 1 ubuntu ubuntu 344 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/ci/run-rook-e2e-tests.sh 2024-03-27T21:42:43.859 INFO:teuthology.orchestra.run.smithi060.stdout: 414532 4 -rw-r--r-- 1 ubuntu ubuntu 214 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/ci/Dockerfile 2024-03-27T21:42:43.859 INFO:teuthology.orchestra.run.smithi060.stdout: 414530 4 -rw-r--r-- 1 ubuntu ubuntu 94 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/__init__.py 2024-03-27T21:42:43.859 INFO:teuthology.orchestra.run.smithi060.stdout: 414546 4 -rwxr-xr-x 1 ubuntu ubuntu 235 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/generate_rook_ceph_client.sh 2024-03-27T21:42:43.859 INFO:teuthology.orchestra.run.smithi060.stdout: 414528 4 -rw-r--r-- 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/.gitignore 2024-03-27T21:42:43.860 INFO:teuthology.orchestra.run.smithi060.stdout: 414549 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/rook/rook-client-python 2024-03-27T21:42:43.860 INFO:teuthology.orchestra.run.smithi060.stdout: 414505 4 -rw-r--r-- 1 ubuntu ubuntu 92 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/requirements.txt 2024-03-27T21:42:43.860 INFO:teuthology.orchestra.run.smithi060.stdout: 414310 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local 2024-03-27T21:42:43.860 INFO:teuthology.orchestra.run.smithi060.stdout: 414396 16 -rw-r--r-- 1 ubuntu ubuntu 12787 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/module.py 2024-03-27T21:42:43.860 INFO:teuthology.orchestra.run.smithi060.stdout: 414398 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/requirements.txt 2024-03-27T21:42:43.860 INFO:teuthology.orchestra.run.smithi060.stdout: 414312 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models 2024-03-27T21:42:43.860 INFO:teuthology.orchestra.run.smithi060.stdout: 414390 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/redhat 2024-03-27T21:42:43.861 INFO:teuthology.orchestra.run.smithi060.stdout: 414393 4 -rw-r--r-- 1 ubuntu ubuntu 1865 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/redhat/hgst_scaler.pkl 2024-03-27T21:42:43.861 INFO:teuthology.orchestra.run.smithi060.stdout: 414391 4 -rw-r--r-- 1 ubuntu ubuntu 1065 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/redhat/config.json 2024-03-27T21:42:43.861 INFO:teuthology.orchestra.run.smithi060.stdout: 414395 4 -rw-r--r-- 1 ubuntu ubuntu 1481 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/redhat/seagate_scaler.pkl 2024-03-27T21:42:43.861 INFO:teuthology.orchestra.run.smithi060.stdout: 414394 36196 -rw-r--r-- 1 ubuntu ubuntu 37062936 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/redhat/seagate_predictor.pkl 2024-03-27T21:42:43.861 INFO:teuthology.orchestra.run.smithi060.stdout: 414392 2796 -rw-r--r-- 1 ubuntu ubuntu 2860606 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/redhat/hgst_predictor.pkl 2024-03-27T21:42:43.861 INFO:teuthology.orchestra.run.smithi060.stdout: 414313 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor 2024-03-27T21:42:43.861 INFO:teuthology.orchestra.run.smithi060.stdout: 414361 528 -rw-r--r-- 1 ubuntu ubuntu 572380 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_229.pkl 2024-03-27T21:42:43.861 INFO:teuthology.orchestra.run.smithi060.stdout: 414358 336 -rw-r--r-- 1 ubuntu ubuntu 342124 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_223.pkl 2024-03-27T21:42:43.861 INFO:teuthology.orchestra.run.smithi060.stdout: 414329 444 -rw-r--r-- 1 ubuntu ubuntu 452492 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_125.pkl 2024-03-27T21:42:43.862 INFO:teuthology.orchestra.run.smithi060.stdout: 414357 308 -rw-r--r-- 1 ubuntu ubuntu 314800 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_222.pkl 2024-03-27T21:42:43.862 INFO:teuthology.orchestra.run.smithi060.stdout: 414374 324 -rw-r--r-- 1 ubuntu ubuntu 328292 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_57.pkl 2024-03-27T21:42:43.862 INFO:teuthology.orchestra.run.smithi060.stdout: 414334 192 -rw-r--r-- 1 ubuntu ubuntu 244892 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_14.pkl 2024-03-27T21:42:43.862 INFO:teuthology.orchestra.run.smithi060.stdout: 414342 320 -rw-r--r-- 1 ubuntu ubuntu 323764 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_174.pkl 2024-03-27T21:42:43.862 INFO:teuthology.orchestra.run.smithi060.stdout: 414384 280 -rw-r--r-- 1 ubuntu ubuntu 284992 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_79.pkl 2024-03-27T21:42:43.862 INFO:teuthology.orchestra.run.smithi060.stdout: 414325 272 -rw-r--r-- 1 ubuntu ubuntu 275692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_12.pkl 2024-03-27T21:42:43.862 INFO:teuthology.orchestra.run.smithi060.stdout: 414335 416 -rw-r--r-- 1 ubuntu ubuntu 422368 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_141.pkl 2024-03-27T21:42:43.862 INFO:teuthology.orchestra.run.smithi060.stdout: 414323 308 -rw-r--r-- 1 ubuntu ubuntu 315192 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_118.pkl 2024-03-27T21:42:43.863 INFO:teuthology.orchestra.run.smithi060.stdout: 414378 428 -rw-r--r-- 1 ubuntu ubuntu 483492 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_62.pkl 2024-03-27T21:42:43.863 INFO:teuthology.orchestra.run.smithi060.stdout: 414327 244 -rw-r--r-- 1 ubuntu ubuntu 246792 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_123.pkl 2024-03-27T21:42:43.863 INFO:teuthology.orchestra.run.smithi060.stdout: 414352 588 -rw-r--r-- 1 ubuntu ubuntu 617200 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_210.pkl 2024-03-27T21:42:43.863 INFO:teuthology.orchestra.run.smithi060.stdout: 414385 220 -rw-r--r-- 1 ubuntu ubuntu 255292 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_82.pkl 2024-03-27T21:42:43.863 INFO:teuthology.orchestra.run.smithi060.stdout: 414367 524 -rw-r--r-- 1 ubuntu ubuntu 534148 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_243.pkl 2024-03-27T21:42:43.863 INFO:teuthology.orchestra.run.smithi060.stdout: 414379 484 -rw-r--r-- 1 ubuntu ubuntu 492592 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_67.pkl 2024-03-27T21:42:43.863 INFO:teuthology.orchestra.run.smithi060.stdout: 414333 416 -rw-r--r-- 1 ubuntu ubuntu 488292 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_138.pkl 2024-03-27T21:42:43.863 INFO:teuthology.orchestra.run.smithi060.stdout: 414354 352 -rw-r--r-- 1 ubuntu ubuntu 357568 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_213.pkl 2024-03-27T21:42:43.864 INFO:teuthology.orchestra.run.smithi060.stdout: 414373 440 -rw-r--r-- 1 ubuntu ubuntu 448292 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_50.pkl 2024-03-27T21:42:43.864 INFO:teuthology.orchestra.run.smithi060.stdout: 414331 484 -rw-r--r-- 1 ubuntu ubuntu 493192 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_131.pkl 2024-03-27T21:42:43.864 INFO:teuthology.orchestra.run.smithi060.stdout: 414388 296 -rw-r--r-- 1 ubuntu ubuntu 302592 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_93.pkl 2024-03-27T21:42:43.864 INFO:teuthology.orchestra.run.smithi060.stdout: 414363 272 -rw-r--r-- 1 ubuntu ubuntu 277972 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_234.pkl 2024-03-27T21:42:43.864 INFO:teuthology.orchestra.run.smithi060.stdout: 414341 356 -rw-r--r-- 1 ubuntu ubuntu 363400 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_169.pkl 2024-03-27T21:42:43.864 INFO:teuthology.orchestra.run.smithi060.stdout: 414364 192 -rw-r--r-- 1 ubuntu ubuntu 243736 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_235.pkl 2024-03-27T21:42:43.864 INFO:teuthology.orchestra.run.smithi060.stdout: 414328 304 -rw-r--r-- 1 ubuntu ubuntu 310292 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_124.pkl 2024-03-27T21:42:43.864 INFO:teuthology.orchestra.run.smithi060.stdout: 414389 268 -rw-r--r-- 1 ubuntu ubuntu 272392 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_97.pkl 2024-03-27T21:42:43.864 INFO:teuthology.orchestra.run.smithi060.stdout: 414382 480 -rw-r--r-- 1 ubuntu ubuntu 489492 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_72.pkl 2024-03-27T21:42:43.865 INFO:teuthology.orchestra.run.smithi060.stdout: 414365 372 -rw-r--r-- 1 ubuntu ubuntu 377872 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_236.pkl 2024-03-27T21:42:43.865 INFO:teuthology.orchestra.run.smithi060.stdout: 414359 324 -rw-r--r-- 1 ubuntu ubuntu 329812 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_225.pkl 2024-03-27T21:42:43.865 INFO:teuthology.orchestra.run.smithi060.stdout: 414356 360 -rw-r--r-- 1 ubuntu ubuntu 365128 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_221.pkl 2024-03-27T21:42:43.865 INFO:teuthology.orchestra.run.smithi060.stdout: 414345 312 -rw-r--r-- 1 ubuntu ubuntu 317176 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_185.pkl 2024-03-27T21:42:43.865 INFO:teuthology.orchestra.run.smithi060.stdout: 414355 336 -rw-r--r-- 1 ubuntu ubuntu 342232 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_219.pkl 2024-03-27T21:42:43.865 INFO:teuthology.orchestra.run.smithi060.stdout: 414314 16 -rw-r--r-- 1 ubuntu ubuntu 14068 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/config.json 2024-03-27T21:42:43.865 INFO:teuthology.orchestra.run.smithi060.stdout: 414387 492 -rw-r--r-- 1 ubuntu ubuntu 502392 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_88.pkl 2024-03-27T21:42:43.865 INFO:teuthology.orchestra.run.smithi060.stdout: 414347 480 -rw-r--r-- 1 ubuntu ubuntu 489544 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_195.pkl 2024-03-27T21:42:43.866 INFO:teuthology.orchestra.run.smithi060.stdout: 414318 168 -rw-r--r-- 1 ubuntu ubuntu 217192 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_105.pkl 2024-03-27T21:42:43.866 INFO:teuthology.orchestra.run.smithi060.stdout: 414338 304 -rw-r--r-- 1 ubuntu ubuntu 308192 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_16.pkl 2024-03-27T21:42:43.866 INFO:teuthology.orchestra.run.smithi060.stdout: 414386 512 -rw-r--r-- 1 ubuntu ubuntu 522092 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_85.pkl 2024-03-27T21:42:43.866 INFO:teuthology.orchestra.run.smithi060.stdout: 414343 308 -rw-r--r-- 1 ubuntu ubuntu 312692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_18.pkl 2024-03-27T21:42:43.866 INFO:teuthology.orchestra.run.smithi060.stdout: 414346 272 -rw-r--r-- 1 ubuntu ubuntu 276352 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_186.pkl 2024-03-27T21:42:43.866 INFO:teuthology.orchestra.run.smithi060.stdout: 414351 280 -rw-r--r-- 1 ubuntu ubuntu 283588 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_208.pkl 2024-03-27T21:42:43.866 INFO:teuthology.orchestra.run.smithi060.stdout: 414377 488 -rw-r--r-- 1 ubuntu ubuntu 499492 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_61.pkl 2024-03-27T21:42:43.867 INFO:teuthology.orchestra.run.smithi060.stdout: 414340 296 -rw-r--r-- 1 ubuntu ubuntu 301516 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_168.pkl 2024-03-27T21:42:43.867 INFO:teuthology.orchestra.run.smithi060.stdout: 414368 496 -rw-r--r-- 1 ubuntu ubuntu 504592 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_27.pkl 2024-03-27T21:42:43.867 INFO:teuthology.orchestra.run.smithi060.stdout: 414315 276 -rw-r--r-- 1 ubuntu ubuntu 281292 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_1.pkl 2024-03-27T21:42:43.867 INFO:teuthology.orchestra.run.smithi060.stdout: 414375 484 -rw-r--r-- 1 ubuntu ubuntu 494292 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_59.pkl 2024-03-27T21:42:43.867 INFO:teuthology.orchestra.run.smithi060.stdout: 414324 444 -rw-r--r-- 1 ubuntu ubuntu 485992 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_119.pkl 2024-03-27T21:42:43.867 INFO:teuthology.orchestra.run.smithi060.stdout: 414337 268 -rw-r--r-- 1 ubuntu ubuntu 305944 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_151.pkl 2024-03-27T21:42:43.867 INFO:teuthology.orchestra.run.smithi060.stdout: 414321 272 -rw-r--r-- 1 ubuntu ubuntu 276492 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_114.pkl 2024-03-27T21:42:43.867 INFO:teuthology.orchestra.run.smithi060.stdout: 414330 492 -rw-r--r-- 1 ubuntu ubuntu 550492 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_128.pkl 2024-03-27T21:42:43.867 INFO:teuthology.orchestra.run.smithi060.stdout: 414336 352 -rw-r--r-- 1 ubuntu ubuntu 359512 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_145.pkl 2024-03-27T21:42:43.868 INFO:teuthology.orchestra.run.smithi060.stdout: 414348 304 -rw-r--r-- 1 ubuntu ubuntu 307888 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_201.pkl 2024-03-27T21:42:43.868 INFO:teuthology.orchestra.run.smithi060.stdout: 414372 536 -rw-r--r-- 1 ubuntu ubuntu 546592 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_44.pkl 2024-03-27T21:42:43.868 INFO:teuthology.orchestra.run.smithi060.stdout: 414369 548 -rw-r--r-- 1 ubuntu ubuntu 557192 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_3.pkl 2024-03-27T21:42:43.868 INFO:teuthology.orchestra.run.smithi060.stdout: 414317 484 -rw-r--r-- 1 ubuntu ubuntu 492492 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_104.pkl 2024-03-27T21:42:43.868 INFO:teuthology.orchestra.run.smithi060.stdout: 414383 448 -rw-r--r-- 1 ubuntu ubuntu 491392 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_78.pkl 2024-03-27T21:42:43.868 INFO:teuthology.orchestra.run.smithi060.stdout: 414366 560 -rw-r--r-- 1 ubuntu ubuntu 571732 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_239.pkl 2024-03-27T21:42:43.868 INFO:teuthology.orchestra.run.smithi060.stdout: 414371 508 -rw-r--r-- 1 ubuntu ubuntu 516692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_36.pkl 2024-03-27T21:42:43.868 INFO:teuthology.orchestra.run.smithi060.stdout: 414344 348 -rw-r--r-- 1 ubuntu ubuntu 354652 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_182.pkl 2024-03-27T21:42:43.869 INFO:teuthology.orchestra.run.smithi060.stdout: 414322 500 -rw-r--r-- 1 ubuntu ubuntu 509592 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_115.pkl 2024-03-27T21:42:43.869 INFO:teuthology.orchestra.run.smithi060.stdout: 414381 224 -rw-r--r-- 1 ubuntu ubuntu 228792 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_71.pkl 2024-03-27T21:42:43.869 INFO:teuthology.orchestra.run.smithi060.stdout: 414326 304 -rw-r--r-- 1 ubuntu ubuntu 307592 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_120.pkl 2024-03-27T21:42:43.869 INFO:teuthology.orchestra.run.smithi060.stdout: 414376 308 -rw-r--r-- 1 ubuntu ubuntu 314092 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_6.pkl 2024-03-27T21:42:43.869 INFO:teuthology.orchestra.run.smithi060.stdout: 414362 200 -rw-r--r-- 1 ubuntu ubuntu 251188 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_230.pkl 2024-03-27T21:42:43.869 INFO:teuthology.orchestra.run.smithi060.stdout: 414349 524 -rw-r--r-- 1 ubuntu ubuntu 567088 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_204.pkl 2024-03-27T21:42:43.869 INFO:teuthology.orchestra.run.smithi060.stdout: 414350 464 -rw-r--r-- 1 ubuntu ubuntu 474856 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_206.pkl 2024-03-27T21:42:43.869 INFO:teuthology.orchestra.run.smithi060.stdout: 414353 276 -rw-r--r-- 1 ubuntu ubuntu 345148 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_212.pkl 2024-03-27T21:42:43.869 INFO:teuthology.orchestra.run.smithi060.stdout: 414316 168 -rw-r--r-- 1 ubuntu ubuntu 217792 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_10.pkl 2024-03-27T21:42:43.870 INFO:teuthology.orchestra.run.smithi060.stdout: 414319 252 -rw-r--r-- 1 ubuntu ubuntu 256392 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_109.pkl 2024-03-27T21:42:43.870 INFO:teuthology.orchestra.run.smithi060.stdout: 414339 268 -rw-r--r-- 1 ubuntu ubuntu 305188 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_161.pkl 2024-03-27T21:42:43.870 INFO:teuthology.orchestra.run.smithi060.stdout: 414380 284 -rw-r--r-- 1 ubuntu ubuntu 288292 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_69.pkl 2024-03-27T21:42:43.870 INFO:teuthology.orchestra.run.smithi060.stdout: 414360 244 -rw-r--r-- 1 ubuntu ubuntu 296440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_227.pkl 2024-03-27T21:42:43.870 INFO:teuthology.orchestra.run.smithi060.stdout: 414320 488 -rw-r--r-- 1 ubuntu ubuntu 499492 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_112.pkl 2024-03-27T21:42:43.870 INFO:teuthology.orchestra.run.smithi060.stdout: 414370 536 -rw-r--r-- 1 ubuntu ubuntu 547392 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_33.pkl 2024-03-27T21:42:43.870 INFO:teuthology.orchestra.run.smithi060.stdout: 414332 264 -rw-r--r-- 1 ubuntu ubuntu 266692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/models/prophetstor/svm_134.pkl 2024-03-27T21:42:43.870 INFO:teuthology.orchestra.run.smithi060.stdout: 414397 20 -rw-r--r-- 1 ubuntu ubuntu 17526 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/predictor.py 2024-03-27T21:42:43.871 INFO:teuthology.orchestra.run.smithi060.stdout: 414311 4 -rw-r--r-- 1 ubuntu ubuntu 42 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/diskprediction_local/__init__.py 2024-03-27T21:42:43.871 INFO:teuthology.orchestra.run.smithi060.stdout: 414399 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/feedback 2024-03-27T21:42:43.871 INFO:teuthology.orchestra.run.smithi060.stdout: 414402 8 -rw-r--r-- 1 ubuntu ubuntu 5337 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/feedback/module.py 2024-03-27T21:42:43.871 INFO:teuthology.orchestra.run.smithi060.stdout: 414400 4 -rw-r--r-- 1 ubuntu ubuntu 49 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/feedback/__init__.py 2024-03-27T21:42:43.871 INFO:teuthology.orchestra.run.smithi060.stdout: 414401 4 -rw-r--r-- 1 ubuntu ubuntu 1138 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/feedback/model.py 2024-03-27T21:42:43.871 INFO:teuthology.orchestra.run.smithi060.stdout: 414403 4 -rw-r--r-- 1 ubuntu ubuntu 1536 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/feedback/service.py 2024-03-27T21:42:43.871 INFO:teuthology.orchestra.run.smithi060.stdout: 414436 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mirroring 2024-03-27T21:42:43.871 INFO:teuthology.orchestra.run.smithi060.stdout: 414452 8 -rw-r--r-- 1 ubuntu ubuntu 4530 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mirroring/module.py 2024-03-27T21:42:43.872 INFO:teuthology.orchestra.run.smithi060.stdout: 414438 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mirroring/fs 2024-03-27T21:42:43.872 INFO:teuthology.orchestra.run.smithi060.stdout: 414450 36 -rw-r--r-- 1 ubuntu ubuntu 36647 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mirroring/fs/snapshot_mirror.py 2024-03-27T21:42:43.872 INFO:teuthology.orchestra.run.smithi060.stdout: 414449 8 -rw-r--r-- 1 ubuntu ubuntu 4607 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mirroring/fs/notify.py 2024-03-27T21:42:43.872 INFO:teuthology.orchestra.run.smithi060.stdout: 414440 4 -rw-r--r-- 1 ubuntu ubuntu 269 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mirroring/fs/blocklist.py 2024-03-27T21:42:43.872 INFO:teuthology.orchestra.run.smithi060.stdout: 414439 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mirroring/fs/__init__.py 2024-03-27T21:42:43.872 INFO:teuthology.orchestra.run.smithi060.stdout: 414448 4 -rw-r--r-- 1 ubuntu ubuntu 132 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mirroring/fs/exception.py 2024-03-27T21:42:43.872 INFO:teuthology.orchestra.run.smithi060.stdout: 414441 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mirroring/fs/dir_map 2024-03-27T21:42:43.872 INFO:teuthology.orchestra.run.smithi060.stdout: 414446 4 -rw-r--r-- 1 ubuntu ubuntu 3884 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mirroring/fs/dir_map/state_transition.py 2024-03-27T21:42:43.873 INFO:teuthology.orchestra.run.smithi060.stdout: 414443 4 -rw-r--r-- 1 ubuntu ubuntu 724 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mirroring/fs/dir_map/create.py 2024-03-27T21:42:43.873 INFO:teuthology.orchestra.run.smithi060.stdout: 414445 20 -rw-r--r-- 1 ubuntu ubuntu 17256 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mirroring/fs/dir_map/policy.py 2024-03-27T21:42:43.873 INFO:teuthology.orchestra.run.smithi060.stdout: 414442 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mirroring/fs/dir_map/__init__.py 2024-03-27T21:42:43.873 INFO:teuthology.orchestra.run.smithi060.stdout: 414447 8 -rw-r--r-- 1 ubuntu ubuntu 5996 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mirroring/fs/dir_map/update.py 2024-03-27T21:42:43.873 INFO:teuthology.orchestra.run.smithi060.stdout: 414444 4 -rw-r--r-- 1 ubuntu ubuntu 2809 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mirroring/fs/dir_map/load.py 2024-03-27T21:42:43.873 INFO:teuthology.orchestra.run.smithi060.stdout: 414451 8 -rw-r--r-- 1 ubuntu ubuntu 5418 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mirroring/fs/utils.py 2024-03-27T21:42:43.873 INFO:teuthology.orchestra.run.smithi060.stdout: 414437 4 -rw-r--r-- 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mirroring/__init__.py 2024-03-27T21:42:43.873 INFO:teuthology.orchestra.run.smithi060.stdout: 414556 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/selftest 2024-03-27T21:42:43.874 INFO:teuthology.orchestra.run.smithi060.stdout: 414558 20 -rw-r--r-- 1 ubuntu ubuntu 17529 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/selftest/module.py 2024-03-27T21:42:43.874 INFO:teuthology.orchestra.run.smithi060.stdout: 414557 4 -rw-r--r-- 1 ubuntu ubuntu 42 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/selftest/__init__.py 2024-03-27T21:42:43.874 INFO:teuthology.orchestra.run.smithi060.stdout: 414585 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/telegraf 2024-03-27T21:42:43.874 INFO:teuthology.orchestra.run.smithi060.stdout: 414588 12 -rw-r--r-- 1 ubuntu ubuntu 9814 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/telegraf/module.py 2024-03-27T21:42:43.874 INFO:teuthology.orchestra.run.smithi060.stdout: 414587 4 -rw-r--r-- 1 ubuntu ubuntu 1518 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/telegraf/basesocket.py 2024-03-27T21:42:43.874 INFO:teuthology.orchestra.run.smithi060.stdout: 414586 4 -rw-r--r-- 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/telegraf/__init__.py 2024-03-27T21:42:43.874 INFO:teuthology.orchestra.run.smithi060.stdout: 414589 4 -rw-r--r-- 1 ubuntu ubuntu 1674 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/telegraf/protocol.py 2024-03-27T21:42:43.874 INFO:teuthology.orchestra.run.smithi060.stdout: 414590 4 -rw-r--r-- 1 ubuntu ubuntu 658 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/telegraf/utils.py 2024-03-27T21:42:43.875 INFO:teuthology.orchestra.run.smithi060.stdout: 412219 8 -rw-r--r-- 1 ubuntu ubuntu 6088 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/ceph_module.pyi 2024-03-27T21:42:43.875 INFO:teuthology.orchestra.run.smithi060.stdout: 414598 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/test_orchestrator 2024-03-27T21:42:43.875 INFO:teuthology.orchestra.run.smithi060.stdout: 414602 12 -rw-r--r-- 1 ubuntu ubuntu 11182 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/test_orchestrator/module.py 2024-03-27T21:42:43.875 INFO:teuthology.orchestra.run.smithi060.stdout: 414599 4 -rw-r--r-- 1 ubuntu ubuntu 290 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/test_orchestrator/README.md 2024-03-27T21:42:43.875 INFO:teuthology.orchestra.run.smithi060.stdout: 414600 4 -rw-r--r-- 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/test_orchestrator/__init__.py 2024-03-27T21:42:43.875 INFO:teuthology.orchestra.run.smithi060.stdout: 414601 16 -rw-r--r-- 1 ubuntu ubuntu 13565 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/test_orchestrator/dummy_data.json 2024-03-27T21:42:43.875 INFO:teuthology.orchestra.run.smithi060.stdout: 412211 20 -rw-r--r-- 1 ubuntu ubuntu 18450 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/.pylintrc 2024-03-27T21:42:43.875 INFO:teuthology.orchestra.run.smithi060.stdout: 414434 88 -rw-r--r-- 1 ubuntu ubuntu 86092 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/mgr_module.py 2024-03-27T21:42:43.875 INFO:teuthology.orchestra.run.smithi060.stdout: 414425 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/localpool 2024-03-27T21:42:43.876 INFO:teuthology.orchestra.run.smithi060.stdout: 414427 8 -rw-r--r-- 1 ubuntu ubuntu 5082 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/localpool/module.py 2024-03-27T21:42:43.876 INFO:teuthology.orchestra.run.smithi060.stdout: 414426 4 -rw-r--r-- 1 ubuntu ubuntu 42 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/localpool/__init__.py 2024-03-27T21:42:43.876 INFO:teuthology.orchestra.run.smithi060.stdout: 412313 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/crash 2024-03-27T21:42:43.876 INFO:teuthology.orchestra.run.smithi060.stdout: 412315 16 -rw-r--r-- 1 ubuntu ubuntu 16376 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/crash/module.py 2024-03-27T21:42:43.876 INFO:teuthology.orchestra.run.smithi060.stdout: 412314 4 -rw-r--r-- 1 ubuntu ubuntu 42 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/crash/__init__.py 2024-03-27T21:42:43.876 INFO:teuthology.orchestra.run.smithi060.stdout: 414410 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/insights 2024-03-27T21:42:43.876 INFO:teuthology.orchestra.run.smithi060.stdout: 414413 12 -rw-r--r-- 1 ubuntu ubuntu 11800 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/insights/module.py 2024-03-27T21:42:43.876 INFO:teuthology.orchestra.run.smithi060.stdout: 414414 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/insights/tests 2024-03-27T21:42:43.877 INFO:teuthology.orchestra.run.smithi060.stdout: 414416 8 -rw-r--r-- 1 ubuntu ubuntu 8146 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/insights/tests/test_health.py 2024-03-27T21:42:43.877 INFO:teuthology.orchestra.run.smithi060.stdout: 414415 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/insights/tests/__init__.py 2024-03-27T21:42:43.877 INFO:teuthology.orchestra.run.smithi060.stdout: 414411 4 -rw-r--r-- 1 ubuntu ubuntu 85 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/insights/__init__.py 2024-03-27T21:42:43.877 INFO:teuthology.orchestra.run.smithi060.stdout: 414412 8 -rw-r--r-- 1 ubuntu ubuntu 6282 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/insights/health.py 2024-03-27T21:42:43.877 INFO:teuthology.orchestra.run.smithi060.stdout: 414465 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/orchestrator 2024-03-27T21:42:43.877 INFO:teuthology.orchestra.run.smithi060.stdout: 414469 92 -rw-r--r-- 1 ubuntu ubuntu 90496 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/orchestrator/module.py 2024-03-27T21:42:43.877 INFO:teuthology.orchestra.run.smithi060.stdout: 414466 4 -rw-r--r-- 1 ubuntu ubuntu 467 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/orchestrator/README.md 2024-03-27T21:42:43.877 INFO:teuthology.orchestra.run.smithi060.stdout: 414470 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/orchestrator/tests 2024-03-27T21:42:43.877 INFO:teuthology.orchestra.run.smithi060.stdout: 414471 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/orchestrator/tests/__init__.py 2024-03-27T21:42:43.878 INFO:teuthology.orchestra.run.smithi060.stdout: 414472 12 -rw-r--r-- 1 ubuntu ubuntu 10480 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/orchestrator/tests/test_orchestrator.py 2024-03-27T21:42:43.878 INFO:teuthology.orchestra.run.smithi060.stdout: 414467 4 -rw-r--r-- 1 ubuntu ubuntu 723 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/orchestrator/__init__.py 2024-03-27T21:42:43.878 INFO:teuthology.orchestra.run.smithi060.stdout: 414468 64 -rw-r--r-- 1 ubuntu ubuntu 64923 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/orchestrator/_interface.py 2024-03-27T21:42:43.878 INFO:teuthology.orchestra.run.smithi060.stdout: 414476 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/osd_support 2024-03-27T21:42:43.878 INFO:teuthology.orchestra.run.smithi060.stdout: 414478 4 -rw-r--r-- 1 ubuntu ubuntu 495 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/osd_support/module.py 2024-03-27T21:42:43.878 INFO:teuthology.orchestra.run.smithi060.stdout: 414477 4 -rw-r--r-- 1 ubuntu ubuntu 31 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/osd_support/__init__.py 2024-03-27T21:42:43.878 INFO:teuthology.orchestra.run.smithi060.stdout: 414609 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes 2024-03-27T21:42:43.879 INFO:teuthology.orchestra.run.smithi060.stdout: 414644 44 -rw-r--r-- 1 ubuntu ubuntu 41386 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/module.py 2024-03-27T21:42:43.879 INFO:teuthology.orchestra.run.smithi060.stdout: 414611 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs 2024-03-27T21:42:43.879 INFO:teuthology.orchestra.run.smithi060.stdout: 414613 24 -rw-r--r-- 1 ubuntu ubuntu 24026 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/async_cloner.py 2024-03-27T21:42:43.879 INFO:teuthology.orchestra.run.smithi060.stdout: 414642 4 -rw-r--r-- 1 ubuntu ubuntu 1511 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/vol_spec.py 2024-03-27T21:42:43.879 INFO:teuthology.orchestra.run.smithi060.stdout: 414617 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations 2024-03-27T21:42:43.879 INFO:teuthology.orchestra.run.smithi060.stdout: 414627 4 -rw-r--r-- 1 ubuntu ubuntu 779 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/snapshot_util.py 2024-03-27T21:42:43.879 INFO:teuthology.orchestra.run.smithi060.stdout: 414624 4 -rw-r--r-- 1 ubuntu ubuntu 844 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/pin_util.py 2024-03-27T21:42:43.880 INFO:teuthology.orchestra.run.smithi060.stdout: 414619 8 -rw-r--r-- 1 ubuntu ubuntu 4544 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/access.py 2024-03-27T21:42:43.880 INFO:teuthology.orchestra.run.smithi060.stdout: 414621 12 -rw-r--r-- 1 ubuntu ubuntu 11047 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/group.py 2024-03-27T21:42:43.880 INFO:teuthology.orchestra.run.smithi060.stdout: 414622 4 -rw-r--r-- 1 ubuntu ubuntu 637 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/index.py 2024-03-27T21:42:43.880 INFO:teuthology.orchestra.run.smithi060.stdout: 414625 4 -rw-r--r-- 1 ubuntu ubuntu 3879 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/rankevicter.py 2024-03-27T21:42:43.880 INFO:teuthology.orchestra.run.smithi060.stdout: 414629 8 -rw-r--r-- 1 ubuntu ubuntu 6234 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/template.py 2024-03-27T21:42:43.880 INFO:teuthology.orchestra.run.smithi060.stdout: 414623 4 -rw-r--r-- 1 ubuntu ubuntu 1517 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/lock.py 2024-03-27T21:42:43.880 INFO:teuthology.orchestra.run.smithi060.stdout: 414626 4 -rw-r--r-- 1 ubuntu ubuntu 797 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/resolver.py 2024-03-27T21:42:43.880 INFO:teuthology.orchestra.run.smithi060.stdout: 414628 4 -rw-r--r-- 1 ubuntu ubuntu 2905 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/subvolume.py 2024-03-27T21:42:43.881 INFO:teuthology.orchestra.run.smithi060.stdout: 414620 4 -rw-r--r-- 1 ubuntu ubuntu 3541 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/clone_index.py 2024-03-27T21:42:43.881 INFO:teuthology.orchestra.run.smithi060.stdout: 414618 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/__init__.py 2024-03-27T21:42:43.881 INFO:teuthology.orchestra.run.smithi060.stdout: 414631 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/versions 2024-03-27T21:42:43.881 INFO:teuthology.orchestra.run.smithi060.stdout: 414637 24 -rw-r--r-- 1 ubuntu ubuntu 20750 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/versions/subvolume_base.py 2024-03-27T21:42:43.881 INFO:teuthology.orchestra.run.smithi060.stdout: 414635 8 -rw-r--r-- 1 ubuntu ubuntu 4952 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/versions/op_sm.py 2024-03-27T21:42:43.881 INFO:teuthology.orchestra.run.smithi060.stdout: 414639 20 -rw-r--r-- 1 ubuntu ubuntu 18640 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/versions/subvolume_v2.py 2024-03-27T21:42:43.881 INFO:teuthology.orchestra.run.smithi060.stdout: 414632 8 -rw-r--r-- 1 ubuntu ubuntu 5477 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/versions/__init__.py 2024-03-27T21:42:43.881 INFO:teuthology.orchestra.run.smithi060.stdout: 414638 40 -rw-r--r-- 1 ubuntu ubuntu 40869 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/versions/subvolume_v1.py 2024-03-27T21:42:43.881 INFO:teuthology.orchestra.run.smithi060.stdout: 414636 4 -rw-r--r-- 1 ubuntu ubuntu 1788 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/versions/subvolume_attrs.py 2024-03-27T21:42:43.882 INFO:teuthology.orchestra.run.smithi060.stdout: 414634 8 -rw-r--r-- 1 ubuntu ubuntu 7617 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/versions/metadata_manager.py 2024-03-27T21:42:43.882 INFO:teuthology.orchestra.run.smithi060.stdout: 414633 8 -rw-r--r-- 1 ubuntu ubuntu 7199 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/versions/auth_metadata.py 2024-03-27T21:42:43.882 INFO:teuthology.orchestra.run.smithi060.stdout: 414640 12 -rw-r--r-- 1 ubuntu ubuntu 11794 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/volume.py 2024-03-27T21:42:43.882 INFO:teuthology.orchestra.run.smithi060.stdout: 414630 8 -rw-r--r-- 1 ubuntu ubuntu 4482 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/operations/trash.py 2024-03-27T21:42:43.882 INFO:teuthology.orchestra.run.smithi060.stdout: 414614 16 -rw-r--r-- 1 ubuntu ubuntu 13365 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/async_job.py 2024-03-27T21:42:43.882 INFO:teuthology.orchestra.run.smithi060.stdout: 414612 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/__init__.py 2024-03-27T21:42:43.882 INFO:teuthology.orchestra.run.smithi060.stdout: 414616 8 -rw-r--r-- 1 ubuntu ubuntu 7377 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/fs_util.py 2024-03-27T21:42:43.882 INFO:teuthology.orchestra.run.smithi060.stdout: 414615 4 -rw-r--r-- 1 ubuntu ubuntu 1921 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/exception.py 2024-03-27T21:42:43.883 INFO:teuthology.orchestra.run.smithi060.stdout: 414643 48 -rw-r--r-- 1 ubuntu ubuntu 48278 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/volume.py 2024-03-27T21:42:43.883 INFO:teuthology.orchestra.run.smithi060.stdout: 414641 8 -rw-r--r-- 1 ubuntu ubuntu 5253 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/fs/purge_queue.py 2024-03-27T21:42:43.883 INFO:teuthology.orchestra.run.smithi060.stdout: 414610 4 -rw-r--r-- 1 ubuntu ubuntu 28 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/volumes/__init__.py 2024-03-27T21:42:43.883 INFO:teuthology.orchestra.run.smithi060.stdout: 414576 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/stats 2024-03-27T21:42:43.883 INFO:teuthology.orchestra.run.smithi060.stdout: 414581 4 -rw-r--r-- 1 ubuntu ubuntu 1353 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/stats/module.py 2024-03-27T21:42:43.883 INFO:teuthology.orchestra.run.smithi060.stdout: 414578 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/stats/fs 2024-03-27T21:42:43.883 INFO:teuthology.orchestra.run.smithi060.stdout: 414579 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/stats/fs/__init__.py 2024-03-27T21:42:43.883 INFO:teuthology.orchestra.run.smithi060.stdout: 414580 28 -rw-r--r-- 1 ubuntu ubuntu 25764 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/stats/fs/perf_stats.py 2024-03-27T21:42:43.884 INFO:teuthology.orchestra.run.smithi060.stdout: 414577 4 -rw-r--r-- 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/stats/__init__.py 2024-03-27T21:42:43.884 INFO:teuthology.orchestra.run.smithi060.stdout: 412220 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm 2024-03-27T21:42:43.884 INFO:teuthology.orchestra.run.smithi060.stdout: 412252 20 -rw-r--r-- 1 ubuntu ubuntu 18286 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/ssh.py 2024-03-27T21:42:43.884 INFO:teuthology.orchestra.run.smithi060.stdout: 412233 156 -rw-r--r-- 1 ubuntu ubuntu 157921 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/module.py 2024-03-27T21:42:43.884 INFO:teuthology.orchestra.run.smithi060.stdout: 412281 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests 2024-03-27T21:42:43.884 INFO:teuthology.orchestra.run.smithi060.stdout: 412286 4 -rw-r--r-- 1 ubuntu ubuntu 2230 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests/test_autotune.py 2024-03-27T21:42:43.884 INFO:teuthology.orchestra.run.smithi060.stdout: 412287 136 -rw-r--r-- 1 ubuntu ubuntu 135862 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests/test_cephadm.py 2024-03-27T21:42:43.884 INFO:teuthology.orchestra.run.smithi060.stdout: 412297 140 -rw-r--r-- 1 ubuntu ubuntu 141851 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests/test_services.py 2024-03-27T21:42:43.884 INFO:teuthology.orchestra.run.smithi060.stdout: 412283 4 -rw-r--r-- 1 ubuntu ubuntu 526 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests/conftest.py 2024-03-27T21:42:43.885 INFO:teuthology.orchestra.run.smithi060.stdout: 412298 16 -rw-r--r-- 1 ubuntu ubuntu 15705 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests/test_spec.py 2024-03-27T21:42:43.885 INFO:teuthology.orchestra.run.smithi060.stdout: 412293 12 -rw-r--r-- 1 ubuntu ubuntu 11377 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests/test_osd_removal.py 2024-03-27T21:42:43.885 INFO:teuthology.orchestra.run.smithi060.stdout: 412289 24 -rw-r--r-- 1 ubuntu ubuntu 24128 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests/test_configchecks.py 2024-03-27T21:42:43.885 INFO:teuthology.orchestra.run.smithi060.stdout: 412296 8 -rw-r--r-- 1 ubuntu ubuntu 5832 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests/test_service_discovery.py 2024-03-27T21:42:43.885 INFO:teuthology.orchestra.run.smithi060.stdout: 412300 4 -rw-r--r-- 1 ubuntu ubuntu 1056 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests/test_template.py 2024-03-27T21:42:43.885 INFO:teuthology.orchestra.run.smithi060.stdout: 412294 12 -rw-r--r-- 1 ubuntu ubuntu 8408 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests/test_remote_executables.py 2024-03-27T21:42:43.885 INFO:teuthology.orchestra.run.smithi060.stdout: 412290 4 -rw-r--r-- 1 ubuntu ubuntu 1273 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests/test_facts.py 2024-03-27T21:42:43.885 INFO:teuthology.orchestra.run.smithi060.stdout: 412288 4 -rw-r--r-- 1 ubuntu ubuntu 1185 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests/test_completion.py 2024-03-27T21:42:43.886 INFO:teuthology.orchestra.run.smithi060.stdout: 412295 68 -rw-r--r-- 1 ubuntu ubuntu 66119 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests/test_scheduling.py 2024-03-27T21:42:43.886 INFO:teuthology.orchestra.run.smithi060.stdout: 412291 16 -rw-r--r-- 1 ubuntu ubuntu 13871 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests/test_migration.py 2024-03-27T21:42:43.886 INFO:teuthology.orchestra.run.smithi060.stdout: 412299 8 -rw-r--r-- 1 ubuntu ubuntu 4874 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests/test_ssh.py 2024-03-27T21:42:43.886 INFO:teuthology.orchestra.run.smithi060.stdout: 412292 16 -rw-r--r-- 1 ubuntu ubuntu 13595 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests/test_node_proxy.py 2024-03-27T21:42:43.886 INFO:teuthology.orchestra.run.smithi060.stdout: 412282 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests/__init__.py 2024-03-27T21:42:43.886 INFO:teuthology.orchestra.run.smithi060.stdout: 412285 8 -rw-r--r-- 1 ubuntu ubuntu 6183 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests/node_proxy_data.py 2024-03-27T21:42:43.886 INFO:teuthology.orchestra.run.smithi060.stdout: 412284 8 -rw-r--r-- 1 ubuntu ubuntu 7397 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests/fixtures.py 2024-03-27T21:42:43.886 INFO:teuthology.orchestra.run.smithi060.stdout: 412301 16 -rw-r--r-- 1 ubuntu ubuntu 12362 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests/test_tuned_profiles.py 2024-03-27T21:42:43.886 INFO:teuthology.orchestra.run.smithi060.stdout: 412302 24 -rw-r--r-- 1 ubuntu ubuntu 21701 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tests/test_upgrade.py 2024-03-27T21:42:43.887 INFO:teuthology.orchestra.run.smithi060.stdout: 412230 4 -rw-r--r-- 1 ubuntu ubuntu 3762 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/http_server.py 2024-03-27T21:42:43.887 INFO:teuthology.orchestra.run.smithi060.stdout: 412228 32 -rw-r--r-- 1 ubuntu ubuntu 30105 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/configchecks.py 2024-03-27T21:42:43.887 INFO:teuthology.orchestra.run.smithi060.stdout: 412227 4 -rw-r--r-- 1 ubuntu ubuntu 535 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/ceph.repo 2024-03-27T21:42:43.887 INFO:teuthology.orchestra.run.smithi060.stdout: 412236 20 -rw-r--r-- 1 ubuntu ubuntu 20018 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/schedule.py 2024-03-27T21:42:43.887 INFO:teuthology.orchestra.run.smithi060.stdout: 412253 8 -rw-r--r-- 1 ubuntu ubuntu 6244 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/ssl_cert_utils.py 2024-03-27T21:42:43.887 INFO:teuthology.orchestra.run.smithi060.stdout: 412232 20 -rw-r--r-- 1 ubuntu ubuntu 18059 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/migrations.py 2024-03-27T21:42:43.887 INFO:teuthology.orchestra.run.smithi060.stdout: 412234 4 -rw-r--r-- 1 ubuntu ubuntu 2260 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/offline_watcher.py 2024-03-27T21:42:43.887 INFO:teuthology.orchestra.run.smithi060.stdout: 412254 4 -rw-r--r-- 1 ubuntu ubuntu 3731 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/template.py 2024-03-27T21:42:43.888 INFO:teuthology.orchestra.run.smithi060.stdout: 412306 4 -rw-r--r-- 1 ubuntu ubuntu 229 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/vagrant.config.example.json 2024-03-27T21:42:43.888 INFO:teuthology.orchestra.run.smithi060.stdout: 412238 12 -rw-r--r-- 1 ubuntu ubuntu 11027 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/service_discovery.py 2024-03-27T21:42:43.888 INFO:teuthology.orchestra.run.smithi060.stdout: 412224 4 -rw-r--r-- 1 ubuntu ubuntu 174 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/__init__.py 2024-03-27T21:42:43.888 INFO:teuthology.orchestra.run.smithi060.stdout: 412303 8 -rw-r--r-- 1 ubuntu ubuntu 5026 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/tuned_profiles.py 2024-03-27T21:42:43.888 INFO:teuthology.orchestra.run.smithi060.stdout: 412223 4 -rw-r--r-- 1 ubuntu ubuntu 2400 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/Vagrantfile 2024-03-27T21:42:43.888 INFO:teuthology.orchestra.run.smithi060.stdout: 412237 80 -rw-r--r-- 1 ubuntu ubuntu 80849 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/serve.py 2024-03-27T21:42:43.888 INFO:teuthology.orchestra.run.smithi060.stdout: 412239 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/services 2024-03-27T21:42:43.888 INFO:teuthology.orchestra.run.smithi060.stdout: 412241 56 -rw-r--r-- 1 ubuntu ubuntu 53474 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/services/cephadmservice.py 2024-03-27T21:42:43.888 INFO:teuthology.orchestra.run.smithi060.stdout: 412246 36 -rw-r--r-- 1 ubuntu ubuntu 33338 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/services/monitoring.py 2024-03-27T21:42:43.889 INFO:teuthology.orchestra.run.smithi060.stdout: 412243 16 -rw-r--r-- 1 ubuntu ubuntu 16080 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/services/ingress.py 2024-03-27T21:42:43.889 INFO:teuthology.orchestra.run.smithi060.stdout: 412244 12 -rw-r--r-- 1 ubuntu ubuntu 9381 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/services/iscsi.py 2024-03-27T21:42:43.889 INFO:teuthology.orchestra.run.smithi060.stdout: 412245 4 -rw-r--r-- 1 ubuntu ubuntu 3150 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/services/jaeger.py 2024-03-27T21:42:43.889 INFO:teuthology.orchestra.run.smithi060.stdout: 412248 8 -rw-r--r-- 1 ubuntu ubuntu 7521 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/services/node_proxy.py 2024-03-27T21:42:43.889 INFO:teuthology.orchestra.run.smithi060.stdout: 412247 16 -rw-r--r-- 1 ubuntu ubuntu 14272 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/services/nfs.py 2024-03-27T21:42:43.889 INFO:teuthology.orchestra.run.smithi060.stdout: 412242 4 -rw-r--r-- 1 ubuntu ubuntu 1106 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/services/container.py 2024-03-27T21:42:43.889 INFO:teuthology.orchestra.run.smithi060.stdout: 412249 8 -rw-r--r-- 1 ubuntu ubuntu 5674 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/services/nvmeof.py 2024-03-27T21:42:43.889 INFO:teuthology.orchestra.run.smithi060.stdout: 412240 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/services/__init__.py 2024-03-27T21:42:43.890 INFO:teuthology.orchestra.run.smithi060.stdout: 412251 8 -rw-r--r-- 1 ubuntu ubuntu 4513 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/services/smb.py 2024-03-27T21:42:43.890 INFO:teuthology.orchestra.run.smithi060.stdout: 412250 40 -rw-r--r-- 1 ubuntu ubuntu 40167 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/services/osd.py 2024-03-27T21:42:43.890 INFO:teuthology.orchestra.run.smithi060.stdout: 412222 8 -rw-r--r-- 1 ubuntu ubuntu 6757 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/HACKING.rst 2024-03-27T21:42:43.890 INFO:teuthology.orchestra.run.smithi060.stdout: 412235 4 -rw-r--r-- 1 ubuntu ubuntu 2256 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/registry.py 2024-03-27T21:42:43.890 INFO:teuthology.orchestra.run.smithi060.stdout: 412221 4 -rw-r--r-- 1 ubuntu ubuntu 20 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/.gitignore 2024-03-27T21:42:43.890 INFO:teuthology.orchestra.run.smithi060.stdout: 412255 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates 2024-03-27T21:42:43.890 INFO:teuthology.orchestra.run.smithi060.stdout: 412256 4 -rw-r--r-- 1 ubuntu ubuntu 93 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/blink_device_light_cmd.j2 2024-03-27T21:42:43.890 INFO:teuthology.orchestra.run.smithi060.stdout: 412257 4 drwxr-xr-x 10 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services 2024-03-27T21:42:43.891 INFO:teuthology.orchestra.run.smithi060.stdout: 412268 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/iscsi 2024-03-27T21:42:43.891 INFO:teuthology.orchestra.run.smithi060.stdout: 412269 4 -rw-r--r-- 1 ubuntu ubuntu 466 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/iscsi/iscsi-gateway.cfg.j2 2024-03-27T21:42:43.891 INFO:teuthology.orchestra.run.smithi060.stdout: 412261 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/grafana 2024-03-27T21:42:43.891 INFO:teuthology.orchestra.run.smithi060.stdout: 412263 4 -rw-r--r-- 1 ubuntu ubuntu 682 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/grafana/grafana.ini.j2 2024-03-27T21:42:43.891 INFO:teuthology.orchestra.run.smithi060.stdout: 412264 4 -rw-r--r-- 1 ubuntu ubuntu 260 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/grafana/providers.yml.j2 2024-03-27T21:42:43.891 INFO:teuthology.orchestra.run.smithi060.stdout: 412262 4 -rw-r--r-- 1 ubuntu ubuntu 909 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/grafana/ceph-dashboard.yml.j2 2024-03-27T21:42:43.891 INFO:teuthology.orchestra.run.smithi060.stdout: 412280 4 -rw-r--r-- 1 ubuntu ubuntu 321 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/promtail.yml.j2 2024-03-27T21:42:43.891 INFO:teuthology.orchestra.run.smithi060.stdout: 412265 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/ingress 2024-03-27T21:42:43.891 INFO:teuthology.orchestra.run.smithi060.stdout: 412267 4 -rw-r--r-- 1 ubuntu ubuntu 742 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/ingress/keepalived.conf.j2 2024-03-27T21:42:43.892 INFO:teuthology.orchestra.run.smithi060.stdout: 412266 4 -rw-r--r-- 1 ubuntu ubuntu 2566 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/ingress/haproxy.cfg.j2 2024-03-27T21:42:43.892 INFO:teuthology.orchestra.run.smithi060.stdout: 412273 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/node-exporter 2024-03-27T21:42:43.892 INFO:teuthology.orchestra.run.smithi060.stdout: 412274 4 -rw-r--r-- 1 ubuntu ubuntu 80 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/node-exporter/web.yml.j2 2024-03-27T21:42:43.892 INFO:teuthology.orchestra.run.smithi060.stdout: 412270 4 -rw-r--r-- 1 ubuntu ubuntu 523 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/loki.yml.j2 2024-03-27T21:42:43.892 INFO:teuthology.orchestra.run.smithi060.stdout: 412277 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/prometheus 2024-03-27T21:42:43.892 INFO:teuthology.orchestra.run.smithi060.stdout: 412279 4 -rw-r--r-- 1 ubuntu ubuntu 153 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/prometheus/web.yml.j2 2024-03-27T21:42:43.892 INFO:teuthology.orchestra.run.smithi060.stdout: 412278 8 -rw-r--r-- 1 ubuntu ubuntu 4148 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/prometheus/prometheus.yml.j2 2024-03-27T21:42:43.893 INFO:teuthology.orchestra.run.smithi060.stdout: 412275 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/nvmeof 2024-03-27T21:42:43.893 INFO:teuthology.orchestra.run.smithi060.stdout: 412276 4 -rw-r--r-- 1 ubuntu ubuntu 2117 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/nvmeof/ceph-nvmeof.conf.j2 2024-03-27T21:42:43.893 INFO:teuthology.orchestra.run.smithi060.stdout: 412258 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/alertmanager 2024-03-27T21:42:43.893 INFO:teuthology.orchestra.run.smithi060.stdout: 412260 4 -rw-r--r-- 1 ubuntu ubuntu 161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/alertmanager/web.yml.j2 2024-03-27T21:42:43.893 INFO:teuthology.orchestra.run.smithi060.stdout: 412259 4 -rw-r--r-- 1 ubuntu ubuntu 1121 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/alertmanager/alertmanager.yml.j2 2024-03-27T21:42:43.893 INFO:teuthology.orchestra.run.smithi060.stdout: 412271 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/nfs 2024-03-27T21:42:43.893 INFO:teuthology.orchestra.run.smithi060.stdout: 412272 4 -rw-r--r-- 1 ubuntu ubuntu 829 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/templates/services/nfs/ganesha.conf.j2 2024-03-27T21:42:43.893 INFO:teuthology.orchestra.run.smithi060.stdout: 412231 72 -rw-r--r-- 1 ubuntu ubuntu 73475 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/inventory.py 2024-03-27T21:42:43.893 INFO:teuthology.orchestra.run.smithi060.stdout: 412229 8 -rw-r--r-- 1 ubuntu ubuntu 4960 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/exchange.py 2024-03-27T21:42:43.894 INFO:teuthology.orchestra.run.smithi060.stdout: 412226 4 -rw-r--r-- 1 ubuntu ubuntu 1722 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/autotune.py 2024-03-27T21:42:43.894 INFO:teuthology.orchestra.run.smithi060.stdout: 412305 8 -rw-r--r-- 1 ubuntu ubuntu 4811 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/utils.py 2024-03-27T21:42:43.894 INFO:teuthology.orchestra.run.smithi060.stdout: 412225 44 -rw-r--r-- 1 ubuntu ubuntu 42944 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/agent.py 2024-03-27T21:42:43.894 INFO:teuthology.orchestra.run.smithi060.stdout: 412304 60 -rw-r--r-- 1 ubuntu ubuntu 60491 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/cephadm/upgrade.py 2024-03-27T21:42:43.894 INFO:teuthology.orchestra.run.smithi060.stdout: 412210 4 -rw-r--r-- 1 ubuntu ubuntu 130 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/.gitignore 2024-03-27T21:42:43.894 INFO:teuthology.orchestra.run.smithi060.stdout: 414591 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/telemetry 2024-03-27T21:42:43.894 INFO:teuthology.orchestra.run.smithi060.stdout: 414593 88 -rw-r--r-- 1 ubuntu ubuntu 89012 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/telemetry/module.py 2024-03-27T21:42:43.894 INFO:teuthology.orchestra.run.smithi060.stdout: 414594 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/telemetry/tests 2024-03-27T21:42:43.895 INFO:teuthology.orchestra.run.smithi060.stdout: 414595 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/telemetry/tests/__init__.py 2024-03-27T21:42:43.895 INFO:teuthology.orchestra.run.smithi060.stdout: 414596 8 -rw-r--r-- 1 ubuntu ubuntu 4223 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/telemetry/tests/test_telemetry.py 2024-03-27T21:42:43.895 INFO:teuthology.orchestra.run.smithi060.stdout: 414597 4 -rw-r--r-- 1 ubuntu ubuntu 120 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/telemetry/tox.ini 2024-03-27T21:42:43.895 INFO:teuthology.orchestra.run.smithi060.stdout: 414592 4 -rw-r--r-- 1 ubuntu ubuntu 123 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/telemetry/__init__.py 2024-03-27T21:42:43.895 INFO:teuthology.orchestra.run.smithi060.stdout: 414491 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/prometheus 2024-03-27T21:42:43.895 INFO:teuthology.orchestra.run.smithi060.stdout: 414493 80 -rw-r--r-- 1 ubuntu ubuntu 79416 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/prometheus/module.py 2024-03-27T21:42:43.895 INFO:teuthology.orchestra.run.smithi060.stdout: 414492 4 -rw-r--r-- 1 ubuntu ubuntu 57 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/prometheus/__init__.py 2024-03-27T21:42:43.895 INFO:teuthology.orchestra.run.smithi060.stdout: 414494 4 -rw-r--r-- 1 ubuntu ubuntu 3979 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/prometheus/test_module.py 2024-03-27T21:42:43.895 INFO:teuthology.orchestra.run.smithi060.stdout: 414504 4 -rw-r--r-- 1 ubuntu ubuntu 195 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/requirements-required.txt 2024-03-27T21:42:43.896 INFO:teuthology.orchestra.run.smithi060.stdout: 414582 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/status 2024-03-27T21:42:43.896 INFO:teuthology.orchestra.run.smithi060.stdout: 414584 20 -rw-r--r-- 1 ubuntu ubuntu 16519 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/status/module.py 2024-03-27T21:42:43.896 INFO:teuthology.orchestra.run.smithi060.stdout: 414583 4 -rw-r--r-- 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/status/__init__.py 2024-03-27T21:42:43.896 INFO:teuthology.orchestra.run.smithi060.stdout: 414453 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/nfs 2024-03-27T21:42:43.896 INFO:teuthology.orchestra.run.smithi060.stdout: 414459 8 -rw-r--r-- 1 ubuntu ubuntu 7831 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/nfs/module.py 2024-03-27T21:42:43.896 INFO:teuthology.orchestra.run.smithi060.stdout: 414460 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/nfs/tests 2024-03-27T21:42:43.896 INFO:teuthology.orchestra.run.smithi060.stdout: 414462 44 -rw-r--r-- 1 ubuntu ubuntu 41758 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/nfs/tests/test_nfs.py 2024-03-27T21:42:43.896 INFO:teuthology.orchestra.run.smithi060.stdout: 414461 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/nfs/tests/__init__.py 2024-03-27T21:42:43.897 INFO:teuthology.orchestra.run.smithi060.stdout: 414457 36 -rw-r--r-- 1 ubuntu ubuntu 36026 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/nfs/export.py 2024-03-27T21:42:43.897 INFO:teuthology.orchestra.run.smithi060.stdout: 414455 16 -rw-r--r-- 1 ubuntu ubuntu 13463 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/nfs/cluster.py 2024-03-27T21:42:43.897 INFO:teuthology.orchestra.run.smithi060.stdout: 414454 4 -rw-r--r-- 1 ubuntu ubuntu 100 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/nfs/__init__.py 2024-03-27T21:42:43.897 INFO:teuthology.orchestra.run.smithi060.stdout: 414456 4 -rw-r--r-- 1 ubuntu ubuntu 954 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/nfs/exception.py 2024-03-27T21:42:43.897 INFO:teuthology.orchestra.run.smithi060.stdout: 414458 20 -rw-r--r-- 1 ubuntu ubuntu 19920 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/nfs/ganesha_conf.py 2024-03-27T21:42:43.897 INFO:teuthology.orchestra.run.smithi060.stdout: 414463 4 -rw-r--r-- 1 ubuntu ubuntu 3382 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/nfs/utils.py 2024-03-27T21:42:43.897 INFO:teuthology.orchestra.run.smithi060.stdout: 412201 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/cephfs 2024-03-27T21:42:43.897 INFO:teuthology.orchestra.run.smithi060.stdout: 412208 4 -rw-r--r-- 1 ubuntu ubuntu 1499 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/cephfs/types.pxd 2024-03-27T21:42:43.897 INFO:teuthology.orchestra.run.smithi060.stdout: 412202 4 -rw-r--r-- 1 ubuntu ubuntu 179 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/cephfs/CMakeLists.txt 2024-03-27T21:42:43.898 INFO:teuthology.orchestra.run.smithi060.stdout: 412206 12 -rw-r--r-- 1 ubuntu ubuntu 10698 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/cephfs/mock_cephfs.pxi 2024-03-27T21:42:43.898 INFO:teuthology.orchestra.run.smithi060.stdout: 412203 4 -rw-r--r-- 1 ubuntu ubuntu 19 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/cephfs/MANIFEST.in 2024-03-27T21:42:43.898 INFO:teuthology.orchestra.run.smithi060.stdout: 412205 92 -rw-r--r-- 1 ubuntu ubuntu 91191 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/cephfs/cephfs.pyx 2024-03-27T21:42:43.898 INFO:teuthology.orchestra.run.smithi060.stdout: 412204 12 -rw-r--r-- 1 ubuntu ubuntu 9488 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/cephfs/c_cephfs.pxd 2024-03-27T21:42:43.898 INFO:teuthology.orchestra.run.smithi060.stdout: 412207 8 -rwxr-xr-x 1 ubuntu ubuntu 7227 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/cephfs/setup.py 2024-03-27T21:42:43.898 INFO:teuthology.orchestra.run.smithi060.stdout: 414649 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rados 2024-03-27T21:42:43.898 INFO:teuthology.orchestra.run.smithi060.stdout: 414650 4 -rw-r--r-- 1 ubuntu ubuntu 201 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rados/CMakeLists.txt 2024-03-27T21:42:43.899 INFO:teuthology.orchestra.run.smithi060.stdout: 414653 4 -rw-r--r-- 1 ubuntu ubuntu 229 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rados/ctime.pxd 2024-03-27T21:42:43.899 INFO:teuthology.orchestra.run.smithi060.stdout: 414654 24 -rw-r--r-- 1 ubuntu ubuntu 22280 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rados/mock_rados.pxi 2024-03-27T21:42:43.899 INFO:teuthology.orchestra.run.smithi060.stdout: 414651 4 -rw-r--r-- 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rados/MANIFEST.in 2024-03-27T21:42:43.899 INFO:teuthology.orchestra.run.smithi060.stdout: 414656 144 -rw-r--r-- 1 ubuntu ubuntu 146118 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rados/rados.pyx 2024-03-27T21:42:43.899 INFO:teuthology.orchestra.run.smithi060.stdout: 414652 20 -rw-r--r-- 1 ubuntu ubuntu 20374 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rados/c_rados.pxd 2024-03-27T21:42:43.899 INFO:teuthology.orchestra.run.smithi060.stdout: 414657 8 -rwxr-xr-x 1 ubuntu ubuntu 6854 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rados/setup.py 2024-03-27T21:42:43.899 INFO:teuthology.orchestra.run.smithi060.stdout: 414655 4 -rw-r--r-- 1 ubuntu ubuntu 1187 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rados/rados.pxd 2024-03-27T21:42:43.900 INFO:teuthology.orchestra.run.smithi060.stdout: 412197 4 -rw-r--r-- 1 ubuntu ubuntu 2635 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/CMakeLists.txt 2024-03-27T21:42:43.900 INFO:teuthology.orchestra.run.smithi060.stdout: 414666 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rgw 2024-03-27T21:42:43.900 INFO:teuthology.orchestra.run.smithi060.stdout: 414672 20 -rw-r--r-- 1 ubuntu ubuntu 17091 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rgw/rgw.pyx 2024-03-27T21:42:43.900 INFO:teuthology.orchestra.run.smithi060.stdout: 414667 4 -rw-r--r-- 1 ubuntu ubuntu 232 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rgw/CMakeLists.txt 2024-03-27T21:42:43.900 INFO:teuthology.orchestra.run.smithi060.stdout: 414671 8 -rw-r--r-- 1 ubuntu ubuntu 4960 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rgw/mock_rgw.pxi 2024-03-27T21:42:43.900 INFO:teuthology.orchestra.run.smithi060.stdout: 414670 4 -rw-r--r-- 1 ubuntu ubuntu 486 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rgw/cstat.pxd 2024-03-27T21:42:43.900 INFO:teuthology.orchestra.run.smithi060.stdout: 414668 4 -rw-r--r-- 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rgw/MANIFEST.in 2024-03-27T21:42:43.900 INFO:teuthology.orchestra.run.smithi060.stdout: 414673 8 -rwxr-xr-x 1 ubuntu ubuntu 7007 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rgw/setup.py 2024-03-27T21:42:43.900 INFO:teuthology.orchestra.run.smithi060.stdout: 414669 8 -rw-r--r-- 1 ubuntu ubuntu 4583 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rgw/c_rgw.pxd 2024-03-27T21:42:43.901 INFO:teuthology.orchestra.run.smithi060.stdout: 412200 8 -rwxr-xr-x 1 ubuntu ubuntu 4538 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/ceph_mgr_repl.py 2024-03-27T21:42:43.901 INFO:teuthology.orchestra.run.smithi060.stdout: 414674 4 -rw-r--r-- 1 ubuntu ubuntu 269 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/tox.ini 2024-03-27T21:42:43.901 INFO:teuthology.orchestra.run.smithi060.stdout: 414658 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rbd 2024-03-27T21:42:43.901 INFO:teuthology.orchestra.run.smithi060.stdout: 414659 4 -rw-r--r-- 1 ubuntu ubuntu 164 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rbd/CMakeLists.txt 2024-03-27T21:42:43.901 INFO:teuthology.orchestra.run.smithi060.stdout: 414662 4 -rw-r--r-- 1 ubuntu ubuntu 156 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rbd/ctime.pxd 2024-03-27T21:42:43.901 INFO:teuthology.orchestra.run.smithi060.stdout: 414660 4 -rw-r--r-- 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rbd/MANIFEST.in 2024-03-27T21:42:43.901 INFO:teuthology.orchestra.run.smithi060.stdout: 414663 40 -rw-r--r-- 1 ubuntu ubuntu 39601 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rbd/mock_rbd.pxi 2024-03-27T21:42:43.901 INFO:teuthology.orchestra.run.smithi060.stdout: 414664 208 -rw-r--r-- 1 ubuntu ubuntu 210518 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rbd/rbd.pyx 2024-03-27T21:42:43.902 INFO:teuthology.orchestra.run.smithi060.stdout: 414661 36 -rw-r--r-- 1 ubuntu ubuntu 36744 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rbd/c_rbd.pxd 2024-03-27T21:42:43.902 INFO:teuthology.orchestra.run.smithi060.stdout: 414665 8 -rwxr-xr-x 1 ubuntu ubuntu 7840 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/rbd/setup.py 2024-03-27T21:42:43.902 INFO:teuthology.orchestra.run.smithi060.stdout: 412199 16 -rw-r--r-- 1 ubuntu ubuntu 15462 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/ceph_daemon.py 2024-03-27T21:42:43.902 INFO:teuthology.orchestra.run.smithi060.stdout: 412198 60 -rw-r--r-- 1 ubuntu ubuntu 58341 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/pybind/ceph_argparse.py 2024-03-27T21:42:43.902 INFO:teuthology.orchestra.run.smithi060.stdout: 411737 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr 2024-03-27T21:42:43.902 INFO:teuthology.orchestra.run.smithi060.stdout: 411776 16 -rw-r--r-- 1 ubuntu ubuntu 13669 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/MgrStandby.cc 2024-03-27T21:42:43.902 INFO:teuthology.orchestra.run.smithi060.stdout: 411750 4 -rw-r--r-- 1 ubuntu ubuntu 3003 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/DaemonHealthMetricCollector.cc 2024-03-27T21:42:43.902 INFO:teuthology.orchestra.run.smithi060.stdout: 411796 8 -rw-r--r-- 1 ubuntu ubuntu 5657 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/StandbyPyModules.cc 2024-03-27T21:42:43.903 INFO:teuthology.orchestra.run.smithi060.stdout: 411760 4 -rw-r--r-- 1 ubuntu ubuntu 1965 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/MDSPerfMetricCollector.cc 2024-03-27T21:42:43.903 INFO:teuthology.orchestra.run.smithi060.stdout: 411766 8 -rw-r--r-- 1 ubuntu ubuntu 7184 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/MetricTypes.h 2024-03-27T21:42:43.903 INFO:teuthology.orchestra.run.smithi060.stdout: 411739 4 -rw-r--r-- 1 ubuntu ubuntu 2802 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/ActivePyModule.h 2024-03-27T21:42:43.903 INFO:teuthology.orchestra.run.smithi060.stdout: 411748 8 -rw-r--r-- 1 ubuntu ubuntu 4461 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/ClusterState.h 2024-03-27T21:42:43.903 INFO:teuthology.orchestra.run.smithi060.stdout: 411753 4 -rw-r--r-- 1 ubuntu ubuntu 612 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/DaemonKey.h 2024-03-27T21:42:43.903 INFO:teuthology.orchestra.run.smithi060.stdout: 411775 4 -rw-r--r-- 1 ubuntu ubuntu 832 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/MgrSession.h 2024-03-27T21:42:43.903 INFO:teuthology.orchestra.run.smithi060.stdout: 411746 4 -rw-r--r-- 1 ubuntu ubuntu 1511 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/CMakeLists.txt 2024-03-27T21:42:43.903 INFO:teuthology.orchestra.run.smithi060.stdout: 411790 24 -rw-r--r-- 1 ubuntu ubuntu 23099 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/PyOSDMap.cc 2024-03-27T21:42:43.904 INFO:teuthology.orchestra.run.smithi060.stdout: 411757 12 -rw-r--r-- 1 ubuntu ubuntu 10143 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/DaemonState.h 2024-03-27T21:42:43.904 INFO:teuthology.orchestra.run.smithi060.stdout: 411793 4 -rw-r--r-- 1 ubuntu ubuntu 275 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/PyUtil.h 2024-03-27T21:42:43.904 INFO:teuthology.orchestra.run.smithi060.stdout: 411782 4 -rw-r--r-- 1 ubuntu ubuntu 3332 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/PyFormatter.cc 2024-03-27T21:42:43.904 INFO:teuthology.orchestra.run.smithi060.stdout: 411799 4 -rw-r--r-- 1 ubuntu ubuntu 3154 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/TTLCache.h 2024-03-27T21:42:43.904 INFO:teuthology.orchestra.run.smithi060.stdout: 411783 8 -rw-r--r-- 1 ubuntu ubuntu 4151 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/PyFormatter.h 2024-03-27T21:42:43.904 INFO:teuthology.orchestra.run.smithi060.stdout: 411787 8 -rw-r--r-- 1 ubuntu ubuntu 6308 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/PyModuleRegistry.h 2024-03-27T21:42:43.904 INFO:teuthology.orchestra.run.smithi060.stdout: 411788 4 -rw-r--r-- 1 ubuntu ubuntu 2846 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/PyModuleRunner.cc 2024-03-27T21:42:43.904 INFO:teuthology.orchestra.run.smithi060.stdout: 411773 12 -rw-r--r-- 1 ubuntu ubuntu 8339 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/MgrCommands.h 2024-03-27T21:42:43.904 INFO:teuthology.orchestra.run.smithi060.stdout: 411738 12 -rw-r--r-- 1 ubuntu ubuntu 8359 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/ActivePyModule.cc 2024-03-27T21:42:43.905 INFO:teuthology.orchestra.run.smithi060.stdout: 411774 4 -rw-r--r-- 1 ubuntu ubuntu 1379 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/MgrContext.h 2024-03-27T21:42:43.905 INFO:teuthology.orchestra.run.smithi060.stdout: 411741 8 -rw-r--r-- 1 ubuntu ubuntu 7507 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/ActivePyModules.h 2024-03-27T21:42:43.905 INFO:teuthology.orchestra.run.smithi060.stdout: 411770 8 -rw-r--r-- 1 ubuntu ubuntu 6053 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/MgrCap.h 2024-03-27T21:42:43.905 INFO:teuthology.orchestra.run.smithi060.stdout: 411763 12 -rw-r--r-- 1 ubuntu ubuntu 10607 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/MDSPerfMetricTypes.h 2024-03-27T21:42:43.905 INFO:teuthology.orchestra.run.smithi060.stdout: 411772 8 -rw-r--r-- 1 ubuntu ubuntu 5788 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/MgrClient.h 2024-03-27T21:42:43.905 INFO:teuthology.orchestra.run.smithi060.stdout: 411781 16 -rw-r--r-- 1 ubuntu ubuntu 13738 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/OSDPerfMetricTypes.h 2024-03-27T21:42:43.905 INFO:teuthology.orchestra.run.smithi060.stdout: 411755 12 -rw-r--r-- 1 ubuntu ubuntu 9445 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/DaemonServer.h 2024-03-27T21:42:43.905 INFO:teuthology.orchestra.run.smithi060.stdout: 411797 4 -rw-r--r-- 1 ubuntu ubuntu 2931 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/StandbyPyModules.h 2024-03-27T21:42:43.906 INFO:teuthology.orchestra.run.smithi060.stdout: 411754 100 -rw-r--r-- 1 ubuntu ubuntu 99677 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/DaemonServer.cc 2024-03-27T21:42:43.906 INFO:teuthology.orchestra.run.smithi060.stdout: 411742 52 -rw-r--r-- 1 ubuntu ubuntu 53028 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/BaseMgrModule.cc 2024-03-27T21:42:43.906 INFO:teuthology.orchestra.run.smithi060.stdout: 411744 8 -rw-r--r-- 1 ubuntu ubuntu 8136 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/BaseMgrStandbyModule.cc 2024-03-27T21:42:43.906 INFO:teuthology.orchestra.run.smithi060.stdout: 411791 4 -rw-r--r-- 1 ubuntu ubuntu 396 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/PyOSDMap.h 2024-03-27T21:42:43.906 INFO:teuthology.orchestra.run.smithi060.stdout: 411768 4 -rw-r--r-- 1 ubuntu ubuntu 3271 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/Mgr.h 2024-03-27T21:42:43.906 INFO:teuthology.orchestra.run.smithi060.stdout: 411740 48 -rw-r--r-- 1 ubuntu ubuntu 45455 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/ActivePyModules.cc 2024-03-27T21:42:43.906 INFO:teuthology.orchestra.run.smithi060.stdout: 411758 4 -rw-r--r-- 1 ubuntu ubuntu 3010 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/Gil.cc 2024-03-27T21:42:43.907 INFO:teuthology.orchestra.run.smithi060.stdout: 411800 4 -rw-r--r-- 1 ubuntu ubuntu 554 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/Types.h 2024-03-27T21:42:43.907 INFO:teuthology.orchestra.run.smithi060.stdout: 411752 4 -rw-r--r-- 1 ubuntu ubuntu 685 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/DaemonKey.cc 2024-03-27T21:42:43.907 INFO:teuthology.orchestra.run.smithi060.stdout: 411779 4 -rw-r--r-- 1 ubuntu ubuntu 670 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/OSDPerfMetricCollector.h 2024-03-27T21:42:43.907 INFO:teuthology.orchestra.run.smithi060.stdout: 411756 12 -rw-r--r-- 1 ubuntu ubuntu 11285 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/DaemonState.cc 2024-03-27T21:42:43.907 INFO:teuthology.orchestra.run.smithi060.stdout: 411747 12 -rw-r--r-- 1 ubuntu ubuntu 12251 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/ClusterState.cc 2024-03-27T21:42:43.907 INFO:teuthology.orchestra.run.smithi060.stdout: 411804 4 -rw-r--r-- 1 ubuntu ubuntu 359 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/mgr_perf_counters.h 2024-03-27T21:42:43.907 INFO:teuthology.orchestra.run.smithi060.stdout: 411792 4 -rw-r--r-- 1 ubuntu ubuntu 1037 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/PyUtil.cc 2024-03-27T21:42:43.907 INFO:teuthology.orchestra.run.smithi060.stdout: 411785 8 -rw-r--r-- 1 ubuntu ubuntu 4863 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/PyModule.h 2024-03-27T21:42:43.907 INFO:teuthology.orchestra.run.smithi060.stdout: 411764 8 -rw-r--r-- 1 ubuntu ubuntu 5334 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/MetricCollector.cc 2024-03-27T21:42:43.908 INFO:teuthology.orchestra.run.smithi060.stdout: 411801 4 -rw-r--r-- 1 ubuntu ubuntu 457 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/mgr_commands.cc 2024-03-27T21:42:43.908 INFO:teuthology.orchestra.run.smithi060.stdout: 411778 4 -rw-r--r-- 1 ubuntu ubuntu 1299 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/OSDPerfMetricCollector.cc 2024-03-27T21:42:43.908 INFO:teuthology.orchestra.run.smithi060.stdout: 411780 4 -rw-r--r-- 1 ubuntu ubuntu 4070 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/OSDPerfMetricTypes.cc 2024-03-27T21:42:43.908 INFO:teuthology.orchestra.run.smithi060.stdout: 411743 4 -rw-r--r-- 1 ubuntu ubuntu 76 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/BaseMgrModule.h 2024-03-27T21:42:43.908 INFO:teuthology.orchestra.run.smithi060.stdout: 411767 24 -rw-r--r-- 1 ubuntu ubuntu 23482 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/Mgr.cc 2024-03-27T21:42:43.908 INFO:teuthology.orchestra.run.smithi060.stdout: 411784 24 -rw-r--r-- 1 ubuntu ubuntu 23250 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/PyModule.cc 2024-03-27T21:42:43.908 INFO:teuthology.orchestra.run.smithi060.stdout: 411749 4 -rw-r--r-- 1 ubuntu ubuntu 2223 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/DaemonHealthMetric.h 2024-03-27T21:42:43.908 INFO:teuthology.orchestra.run.smithi060.stdout: 411803 4 -rw-r--r-- 1 ubuntu ubuntu 804 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/mgr_perf_counters.cc 2024-03-27T21:42:43.909 INFO:teuthology.orchestra.run.smithi060.stdout: 411759 4 -rw-r--r-- 1 ubuntu ubuntu 2834 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/Gil.h 2024-03-27T21:42:43.909 INFO:teuthology.orchestra.run.smithi060.stdout: 411751 4 -rw-r--r-- 1 ubuntu ubuntu 933 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/DaemonHealthMetricCollector.h 2024-03-27T21:42:43.909 INFO:teuthology.orchestra.run.smithi060.stdout: 411786 16 -rw-r--r-- 1 ubuntu ubuntu 12757 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/PyModuleRegistry.cc 2024-03-27T21:42:43.909 INFO:teuthology.orchestra.run.smithi060.stdout: 411802 4 -rw-r--r-- 1 ubuntu ubuntu 211 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/mgr_commands.h 2024-03-27T21:42:43.909 INFO:teuthology.orchestra.run.smithi060.stdout: 411762 8 -rw-r--r-- 1 ubuntu ubuntu 5513 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/MDSPerfMetricTypes.cc 2024-03-27T21:42:43.909 INFO:teuthology.orchestra.run.smithi060.stdout: 411745 4 -rw-r--r-- 1 ubuntu ubuntu 82 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/BaseMgrStandbyModule.h 2024-03-27T21:42:43.909 INFO:teuthology.orchestra.run.smithi060.stdout: 411795 4 -rw-r--r-- 1 ubuntu ubuntu 2838 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/ServiceMap.h 2024-03-27T21:42:43.909 INFO:teuthology.orchestra.run.smithi060.stdout: 411794 8 -rw-r--r-- 1 ubuntu ubuntu 6178 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/ServiceMap.cc 2024-03-27T21:42:43.910 INFO:teuthology.orchestra.run.smithi060.stdout: 411761 4 -rw-r--r-- 1 ubuntu ubuntu 838 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/MDSPerfMetricCollector.h 2024-03-27T21:42:43.910 INFO:teuthology.orchestra.run.smithi060.stdout: 411769 20 -rw-r--r-- 1 ubuntu ubuntu 17285 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/MgrCap.cc 2024-03-27T21:42:43.910 INFO:teuthology.orchestra.run.smithi060.stdout: 411771 20 -rw-r--r-- 1 ubuntu ubuntu 17653 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/MgrClient.cc 2024-03-27T21:42:43.910 INFO:teuthology.orchestra.run.smithi060.stdout: 411777 4 -rw-r--r-- 1 ubuntu ubuntu 2092 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/MgrStandby.h 2024-03-27T21:42:43.910 INFO:teuthology.orchestra.run.smithi060.stdout: 411789 4 -rw-r--r-- 1 ubuntu ubuntu 1807 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/PyModuleRunner.h 2024-03-27T21:42:43.910 INFO:teuthology.orchestra.run.smithi060.stdout: 411765 4 -rw-r--r-- 1 ubuntu ubuntu 2367 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/MetricCollector.h 2024-03-27T21:42:43.910 INFO:teuthology.orchestra.run.smithi060.stdout: 411798 4 -rw-r--r-- 1 ubuntu ubuntu 2639 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mgr/TTLCache.cc 2024-03-27T21:42:43.910 INFO:teuthology.orchestra.run.smithi060.stdout: 409074 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-node-proxy 2024-03-27T21:42:43.911 INFO:teuthology.orchestra.run.smithi060.stdout: 409075 4 -rw-r--r-- 1 ubuntu ubuntu 1043 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-node-proxy/CMakeLists.txt 2024-03-27T21:42:43.911 INFO:teuthology.orchestra.run.smithi060.stdout: 409076 4 -rw-r--r-- 1 ubuntu ubuntu 29 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-node-proxy/MANIFEST.in 2024-03-27T21:42:43.911 INFO:teuthology.orchestra.run.smithi060.stdout: 409077 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-node-proxy/ceph_node_proxy 2024-03-27T21:42:43.911 INFO:teuthology.orchestra.run.smithi060.stdout: 409080 4 -rw-r--r-- 1 ubuntu ubuntu 487 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-node-proxy/ceph_node_proxy/baseclient.py 2024-03-27T21:42:43.911 INFO:teuthology.orchestra.run.smithi060.stdout: 409079 12 -rw-r--r-- 1 ubuntu ubuntu 10156 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-node-proxy/ceph_node_proxy/api.py 2024-03-27T21:42:43.911 INFO:teuthology.orchestra.run.smithi060.stdout: 409087 8 -rw-r--r-- 1 ubuntu ubuntu 6028 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-node-proxy/ceph_node_proxy/util.py 2024-03-27T21:42:43.911 INFO:teuthology.orchestra.run.smithi060.stdout: 409083 8 -rw-r--r-- 1 ubuntu ubuntu 7642 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-node-proxy/ceph_node_proxy/main.py 2024-03-27T21:42:43.911 INFO:teuthology.orchestra.run.smithi060.stdout: 409086 4 -rw-r--r-- 1 ubuntu ubuntu 3621 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-node-proxy/ceph_node_proxy/reporter.py 2024-03-27T21:42:43.912 INFO:teuthology.orchestra.run.smithi060.stdout: 409081 12 -rw-r--r-- 1 ubuntu ubuntu 11452 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-node-proxy/ceph_node_proxy/baseredfishsystem.py 2024-03-27T21:42:43.912 INFO:teuthology.orchestra.run.smithi060.stdout: 409085 8 -rw-r--r-- 1 ubuntu ubuntu 6624 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-node-proxy/ceph_node_proxy/redfishdellsystem.py 2024-03-27T21:42:43.912 INFO:teuthology.orchestra.run.smithi060.stdout: 409078 4 -rw-r--r-- 1 ubuntu ubuntu 44 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-node-proxy/ceph_node_proxy/__init__.py 2024-03-27T21:42:43.912 INFO:teuthology.orchestra.run.smithi060.stdout: 409084 8 -rw-r--r-- 1 ubuntu ubuntu 5183 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-node-proxy/ceph_node_proxy/redfish_client.py 2024-03-27T21:42:43.912 INFO:teuthology.orchestra.run.smithi060.stdout: 409082 4 -rw-r--r-- 1 ubuntu ubuntu 2786 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-node-proxy/ceph_node_proxy/basesystem.py 2024-03-27T21:42:43.912 INFO:teuthology.orchestra.run.smithi060.stdout: 409088 4 -rw-r--r-- 1 ubuntu ubuntu 1213 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-node-proxy/setup.py 2024-03-27T21:42:43.912 INFO:teuthology.orchestra.run.smithi060.stdout: 408974 24 -rw-r--r-- 1 ubuntu ubuntu 20980 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/SimpleRADOSStriper.cc 2024-03-27T21:42:43.912 INFO:teuthology.orchestra.run.smithi060.stdout: 417126 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/xxHash 2024-03-27T21:42:43.912 INFO:teuthology.orchestra.run.smithi060.stdout: 409390 4 -rw-r--r-- 1 ubuntu ubuntu 271 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph_ver.h.in.cmake 2024-03-27T21:42:43.913 INFO:teuthology.orchestra.run.smithi060.stdout: 409389 4 -rw-r--r-- 1 ubuntu ubuntu 159 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph_ver.c 2024-03-27T21:42:43.913 INFO:teuthology.orchestra.run.smithi060.stdout: 408977 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/arch 2024-03-27T21:42:43.913 INFO:teuthology.orchestra.run.smithi060.stdout: 408978 4 -rw-r--r-- 1 ubuntu ubuntu 248 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/arch/CMakeLists.txt 2024-03-27T21:42:43.913 INFO:teuthology.orchestra.run.smithi060.stdout: 408980 4 -rw-r--r-- 1 ubuntu ubuntu 416 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/arch/arm.h 2024-03-27T21:42:43.913 INFO:teuthology.orchestra.run.smithi060.stdout: 408982 4 -rw-r--r-- 1 ubuntu ubuntu 681 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/arch/intel.h 2024-03-27T21:42:43.913 INFO:teuthology.orchestra.run.smithi060.stdout: 408983 4 -rw-r--r-- 1 ubuntu ubuntu 889 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/arch/ppc.c 2024-03-27T21:42:43.914 INFO:teuthology.orchestra.run.smithi060.stdout: 408981 4 -rw-r--r-- 1 ubuntu ubuntu 1883 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/arch/intel.c 2024-03-27T21:42:43.914 INFO:teuthology.orchestra.run.smithi060.stdout: 408984 4 -rw-r--r-- 1 ubuntu ubuntu 540 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/arch/ppc.h 2024-03-27T21:42:43.914 INFO:teuthology.orchestra.run.smithi060.stdout: 408986 4 -rw-r--r-- 1 ubuntu ubuntu 235 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/arch/probe.h 2024-03-27T21:42:43.914 INFO:teuthology.orchestra.run.smithi060.stdout: 408985 4 -rw-r--r-- 1 ubuntu ubuntu 603 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/arch/probe.cc 2024-03-27T21:42:43.914 INFO:teuthology.orchestra.run.smithi060.stdout: 408979 4 -rw-r--r-- 1 ubuntu ubuntu 758 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/arch/arm.c 2024-03-27T21:42:43.914 INFO:teuthology.orchestra.run.smithi060.stdout: 409505 4 -rwxr-xr-x 1 ubuntu ubuntu 329 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ckill.sh 2024-03-27T21:42:43.914 INFO:teuthology.orchestra.run.smithi060.stdout: 412182 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/perfglue 2024-03-27T21:42:43.914 INFO:teuthology.orchestra.run.smithi060.stdout: 412185 4 -rw-r--r-- 1 ubuntu ubuntu 642 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/perfglue/cpu_profiler.h 2024-03-27T21:42:43.915 INFO:teuthology.orchestra.run.smithi060.stdout: 412183 4 -rw-r--r-- 1 ubuntu ubuntu 555 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/perfglue/CMakeLists.txt 2024-03-27T21:42:43.915 INFO:teuthology.orchestra.run.smithi060.stdout: 412184 4 -rw-r--r-- 1 ubuntu ubuntu 1143 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/perfglue/cpu_profiler.cc 2024-03-27T21:42:43.915 INFO:teuthology.orchestra.run.smithi060.stdout: 412186 4 -rw-r--r-- 1 ubuntu ubuntu 1322 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/perfglue/disabled_heap_profiler.cc 2024-03-27T21:42:43.915 INFO:teuthology.orchestra.run.smithi060.stdout: 412188 8 -rw-r--r-- 1 ubuntu ubuntu 5121 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/perfglue/heap_profiler.cc 2024-03-27T21:42:43.915 INFO:teuthology.orchestra.run.smithi060.stdout: 412187 4 -rw-r--r-- 1 ubuntu ubuntu 657 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/perfglue/disabled_stubs.cc 2024-03-27T21:42:43.915 INFO:teuthology.orchestra.run.smithi060.stdout: 412189 4 -rw-r--r-- 1 ubuntu ubuntu 1468 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/perfglue/heap_profiler.h 2024-03-27T21:42:43.915 INFO:teuthology.orchestra.run.smithi060.stdout: 410454 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush 2024-03-27T21:42:43.915 INFO:teuthology.orchestra.run.smithi060.stdout: 410468 4 -rw-r--r-- 1 ubuntu ubuntu 2978 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/crush.c 2024-03-27T21:42:43.916 INFO:teuthology.orchestra.run.smithi060.stdout: 410463 124 -rw-r--r-- 1 ubuntu ubuntu 123513 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/CrushWrapper.cc 2024-03-27T21:42:43.916 INFO:teuthology.orchestra.run.smithi060.stdout: 410458 4 -rw-r--r-- 1 ubuntu ubuntu 3894 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/CrushLocation.cc 2024-03-27T21:42:43.916 INFO:teuthology.orchestra.run.smithi060.stdout: 410455 4 -rw-r--r-- 1 ubuntu ubuntu 177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/CMakeLists.txt 2024-03-27T21:42:43.916 INFO:teuthology.orchestra.run.smithi060.stdout: 410474 4 -rw-r--r-- 1 ubuntu ubuntu 611 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/hash.h 2024-03-27T21:42:43.916 INFO:teuthology.orchestra.run.smithi060.stdout: 410478 4 -rw-r--r-- 1 ubuntu ubuntu 250 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/types.h 2024-03-27T21:42:43.916 INFO:teuthology.orchestra.run.smithi060.stdout: 410472 12 -rw-r--r-- 1 ubuntu ubuntu 9922 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/grammar.h 2024-03-27T21:42:43.916 INFO:teuthology.orchestra.run.smithi060.stdout: 410462 8 -rw-r--r-- 1 ubuntu ubuntu 7915 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/CrushTreeDumper.h 2024-03-27T21:42:43.916 INFO:teuthology.orchestra.run.smithi060.stdout: 410477 4 -rw-r--r-- 1 ubuntu ubuntu 941 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/old_sample.txt 2024-03-27T21:42:43.917 INFO:teuthology.orchestra.run.smithi060.stdout: 410464 52 -rw-r--r-- 1 ubuntu ubuntu 52372 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/CrushWrapper.h 2024-03-27T21:42:43.917 INFO:teuthology.orchestra.run.smithi060.stdout: 410466 40 -rw-r--r-- 1 ubuntu ubuntu 38355 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/builder.c 2024-03-27T21:42:43.917 INFO:teuthology.orchestra.run.smithi060.stdout: 410465 4 -rw-r--r-- 1 ubuntu ubuntu 951 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/CrushWrapper.i 2024-03-27T21:42:43.917 INFO:teuthology.orchestra.run.smithi060.stdout: 410467 16 -rw-r--r-- 1 ubuntu ubuntu 13701 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/builder.h 2024-03-27T21:42:43.917 INFO:teuthology.orchestra.run.smithi060.stdout: 410460 24 -rw-r--r-- 1 ubuntu ubuntu 24167 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/CrushTester.cc 2024-03-27T21:42:43.917 INFO:teuthology.orchestra.run.smithi060.stdout: 410456 40 -rw-r--r-- 1 ubuntu ubuntu 39691 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/CrushCompiler.cc 2024-03-27T21:42:43.917 INFO:teuthology.orchestra.run.smithi060.stdout: 410461 12 -rw-r--r-- 1 ubuntu ubuntu 10880 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/CrushTester.h 2024-03-27T21:42:43.917 INFO:teuthology.orchestra.run.smithi060.stdout: 410473 4 -rw-r--r-- 1 ubuntu ubuntu 3206 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/hash.c 2024-03-27T21:42:43.917 INFO:teuthology.orchestra.run.smithi060.stdout: 410459 4 -rw-r--r-- 1 ubuntu ubuntu 1047 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/CrushLocation.h 2024-03-27T21:42:43.918 INFO:teuthology.orchestra.run.smithi060.stdout: 410475 56 -rw-r--r-- 1 ubuntu ubuntu 56791 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/mapper.c 2024-03-27T21:42:43.918 INFO:teuthology.orchestra.run.smithi060.stdout: 410476 8 -rw-r--r-- 1 ubuntu ubuntu 4237 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/mapper.h 2024-03-27T21:42:43.918 INFO:teuthology.orchestra.run.smithi060.stdout: 410471 16 -rw-r--r-- 1 ubuntu ubuntu 12506 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/crush_ln_table.h 2024-03-27T21:42:43.918 INFO:teuthology.orchestra.run.smithi060.stdout: 410470 4 -rw-r--r-- 1 ubuntu ubuntu 914 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/crush_compat.h 2024-03-27T21:42:43.918 INFO:teuthology.orchestra.run.smithi060.stdout: 410469 20 -rw-r--r-- 1 ubuntu ubuntu 20470 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/crush.h 2024-03-27T21:42:43.918 INFO:teuthology.orchestra.run.smithi060.stdout: 410457 4 -rw-r--r-- 1 ubuntu ubuntu 2949 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crush/CrushCompiler.h 2024-03-27T21:42:43.918 INFO:teuthology.orchestra.run.smithi060.stdout: 415334 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/spdk 2024-03-27T21:42:43.918 INFO:teuthology.orchestra.run.smithi060.stdout: 411422 4 -rw-r--r-- 1 ubuntu ubuntu 358 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/logrotate.conf 2024-03-27T21:42:43.919 INFO:teuthology.orchestra.run.smithi060.stdout: 408972 32 -rw-r--r-- 1 ubuntu ubuntu 30473 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/CMakeLists.txt 2024-03-27T21:42:43.919 INFO:teuthology.orchestra.run.smithi060.stdout: 414719 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/qatzip 2024-03-27T21:42:43.919 INFO:teuthology.orchestra.run.smithi060.stdout: 410760 4 -rw-r--r-- 1 ubuntu ubuntu 4058 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/init-radosgw 2024-03-27T21:42:43.919 INFO:teuthology.orchestra.run.smithi060.stdout: 411994 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ocf 2024-03-27T21:42:43.919 INFO:teuthology.orchestra.run.smithi060.stdout: 411996 4 -rw-r--r-- 1 ubuntu ubuntu 364 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ocf/CMakeLists.txt 2024-03-27T21:42:43.919 INFO:teuthology.orchestra.run.smithi060.stdout: 411997 12 -rw-r--r-- 1 ubuntu ubuntu 9725 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ocf/rbd.in 2024-03-27T21:42:43.919 INFO:teuthology.orchestra.run.smithi060.stdout: 411995 4 -rw-r--r-- 1 ubuntu ubuntu 11 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ocf/.gitignore 2024-03-27T21:42:43.919 INFO:teuthology.orchestra.run.smithi060.stdout: 410845 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/key_value_store 2024-03-27T21:42:43.919 INFO:teuthology.orchestra.run.smithi060.stdout: 410846 4 -rw-r--r-- 1 ubuntu ubuntu 235 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/key_value_store/CMakeLists.txt 2024-03-27T21:42:43.920 INFO:teuthology.orchestra.run.smithi060.stdout: 410848 4 -rw-r--r-- 1 ubuntu ubuntu 3898 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/key_value_store/key_value_structure.h 2024-03-27T21:42:43.920 INFO:teuthology.orchestra.run.smithi060.stdout: 410849 76 -rw-r--r-- 1 ubuntu ubuntu 74925 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/key_value_store/kv_flat_btree_async.cc 2024-03-27T21:42:43.920 INFO:teuthology.orchestra.run.smithi060.stdout: 410851 4 -rw-r--r-- 1 ubuntu ubuntu 3030 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/key_value_store/kvs_arg_types.h 2024-03-27T21:42:43.920 INFO:teuthology.orchestra.run.smithi060.stdout: 410847 20 -rw-r--r-- 1 ubuntu ubuntu 19454 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/key_value_store/cls_kvs.cc 2024-03-27T21:42:43.920 INFO:teuthology.orchestra.run.smithi060.stdout: 410850 28 -rw-r--r-- 1 ubuntu ubuntu 25031 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/key_value_store/kv_flat_btree_async.h 2024-03-27T21:42:43.920 INFO:teuthology.orchestra.run.smithi060.stdout: 409385 28 -rw-r--r-- 1 ubuntu ubuntu 28641 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph_mon.cc 2024-03-27T21:42:43.920 INFO:teuthology.orchestra.run.smithi060.stdout: 410613 4 drwxr-xr-x 9 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include 2024-03-27T21:42:43.920 INFO:teuthology.orchestra.run.smithi060.stdout: 410646 4 -rw-r--r-- 1 ubuntu ubuntu 2187 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/coredumpctl.h 2024-03-27T21:42:43.921 INFO:teuthology.orchestra.run.smithi060.stdout: 410732 4 -rw-r--r-- 1 ubuntu ubuntu 1162 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/uuid.cc 2024-03-27T21:42:43.921 INFO:teuthology.orchestra.run.smithi060.stdout: 410708 8 -rw-r--r-- 1 ubuntu ubuntu 5218 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rangeset.h 2024-03-27T21:42:43.921 INFO:teuthology.orchestra.run.smithi060.stdout: 410661 4 -rw-r--r-- 1 ubuntu ubuntu 1034 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/error.h 2024-03-27T21:42:43.921 INFO:teuthology.orchestra.run.smithi060.stdout: 410758 8 -rw-r--r-- 1 ubuntu ubuntu 5427 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/xlist.h 2024-03-27T21:42:43.921 INFO:teuthology.orchestra.run.smithi060.stdout: 410723 4 -rw-r--r-- 1 ubuntu ubuntu 2229 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/timegm.h 2024-03-27T21:42:43.921 INFO:teuthology.orchestra.run.smithi060.stdout: 410719 4 -rw-r--r-- 1 ubuntu ubuntu 2750 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/statlite.h 2024-03-27T21:42:43.921 INFO:teuthology.orchestra.run.smithi060.stdout: 410707 12 -rw-r--r-- 1 ubuntu ubuntu 8256 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/random.h 2024-03-27T21:42:43.921 INFO:teuthology.orchestra.run.smithi060.stdout: 410669 4 -rw-r--r-- 1 ubuntu ubuntu 1930 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/health.h 2024-03-27T21:42:43.922 INFO:teuthology.orchestra.run.smithi060.stdout: 410659 52 -rw-r--r-- 1 ubuntu ubuntu 52526 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/encoding.h 2024-03-27T21:42:43.922 INFO:teuthology.orchestra.run.smithi060.stdout: 410641 4 -rw-r--r-- 1 ubuntu ubuntu 898 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/common_fwd.h 2024-03-27T21:42:43.922 INFO:teuthology.orchestra.run.smithi060.stdout: 410648 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/cpp-btree 2024-03-27T21:42:43.922 INFO:teuthology.orchestra.run.smithi060.stdout: 410651 8 -rw-r--r-- 1 ubuntu ubuntu 6052 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/cpp-btree/btree_map.h 2024-03-27T21:42:43.922 INFO:teuthology.orchestra.run.smithi060.stdout: 410652 24 -rw-r--r-- 1 ubuntu ubuntu 21678 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/cpp-btree/btree_set.h 2024-03-27T21:42:43.922 INFO:teuthology.orchestra.run.smithi060.stdout: 410649 92 -rw-r--r-- 1 ubuntu ubuntu 90900 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/cpp-btree/btree.h 2024-03-27T21:42:43.922 INFO:teuthology.orchestra.run.smithi060.stdout: 410650 20 -rw-r--r-- 1 ubuntu ubuntu 19109 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/cpp-btree/btree_container.h 2024-03-27T21:42:43.922 INFO:teuthology.orchestra.run.smithi060.stdout: 410676 4 -rw-r--r-- 1 ubuntu ubuntu 2424 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/libcephsqlite.h 2024-03-27T21:42:43.923 INFO:teuthology.orchestra.run.smithi060.stdout: 410720 4 -rw-r--r-- 1 ubuntu ubuntu 3179 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/str_list.h 2024-03-27T21:42:43.923 INFO:teuthology.orchestra.run.smithi060.stdout: 410620 24 -rw-r--r-- 1 ubuntu ubuntu 21752 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/any.h 2024-03-27T21:42:43.923 INFO:teuthology.orchestra.run.smithi060.stdout: 410634 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/cephfs 2024-03-27T21:42:43.923 INFO:teuthology.orchestra.run.smithi060.stdout: 410635 8 -rw-r--r-- 1 ubuntu ubuntu 6463 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/cephfs/ceph_ll_client.h 2024-03-27T21:42:43.923 INFO:teuthology.orchestra.run.smithi060.stdout: 410636 88 -rw-r--r-- 1 ubuntu ubuntu 88278 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/cephfs/libcephfs.h 2024-03-27T21:42:43.923 INFO:teuthology.orchestra.run.smithi060.stdout: 410637 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/cephfs/metrics 2024-03-27T21:42:43.923 INFO:teuthology.orchestra.run.smithi060.stdout: 410638 20 -rw-r--r-- 1 ubuntu ubuntu 19934 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/cephfs/metrics/Types.h 2024-03-27T21:42:43.923 INFO:teuthology.orchestra.run.smithi060.stdout: 410639 32 -rw-r--r-- 1 ubuntu ubuntu 30215 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/cephfs/types.h 2024-03-27T21:42:43.923 INFO:teuthology.orchestra.run.smithi060.stdout: 410660 4 -rw-r--r-- 1 ubuntu ubuntu 548 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/err.h 2024-03-27T21:42:43.924 INFO:teuthology.orchestra.run.smithi060.stdout: 410690 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rados 2024-03-27T21:42:43.924 INFO:teuthology.orchestra.run.smithi060.stdout: 410698 4 -rw-r--r-- 1 ubuntu ubuntu 922 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rados/librgw.h 2024-03-27T21:42:43.924 INFO:teuthology.orchestra.run.smithi060.stdout: 410696 60 -rw-r--r-- 1 ubuntu ubuntu 58501 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rados/librados.hpp 2024-03-27T21:42:43.924 INFO:teuthology.orchestra.run.smithi060.stdout: 410699 4 -rw-r--r-- 1 ubuntu ubuntu 3860 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rados/objclass.h 2024-03-27T21:42:43.924 INFO:teuthology.orchestra.run.smithi060.stdout: 410695 152 -rw-r--r-- 1 ubuntu ubuntu 151570 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rados/librados.h 2024-03-27T21:42:43.924 INFO:teuthology.orchestra.run.smithi060.stdout: 410701 4 -rw-r--r-- 1 ubuntu ubuntu 720 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rados/rados_types.h 2024-03-27T21:42:43.924 INFO:teuthology.orchestra.run.smithi060.stdout: 410702 12 -rw-r--r-- 1 ubuntu ubuntu 9611 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rados/rados_types.hpp 2024-03-27T21:42:43.924 INFO:teuthology.orchestra.run.smithi060.stdout: 410694 0 lrwxrwxrwx 1 ubuntu ubuntu 18 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rados/inline_memory.h -> ../inline_memory.h 2024-03-27T21:42:43.925 INFO:teuthology.orchestra.run.smithi060.stdout: 410700 0 lrwxrwxrwx 1 ubuntu ubuntu 9 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rados/page.h -> ../page.h 2024-03-27T21:42:43.925 INFO:teuthology.orchestra.run.smithi060.stdout: 410693 0 lrwxrwxrwx 1 ubuntu ubuntu 11 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rados/crc32c.h -> ../crc32c.h 2024-03-27T21:42:43.925 INFO:teuthology.orchestra.run.smithi060.stdout: 410692 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rados/buffer_fwd.h -> ../buffer_fwd.h 2024-03-27T21:42:43.925 INFO:teuthology.orchestra.run.smithi060.stdout: 410703 12 -rw-r--r-- 1 ubuntu ubuntu 10819 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rados/rgw_file.h 2024-03-27T21:42:43.925 INFO:teuthology.orchestra.run.smithi060.stdout: 410697 4 -rw-r--r-- 1 ubuntu ubuntu 841 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rados/librados_fwd.hpp 2024-03-27T21:42:43.925 INFO:teuthology.orchestra.run.smithi060.stdout: 410691 0 lrwxrwxrwx 1 ubuntu ubuntu 11 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rados/buffer.h -> ../buffer.h 2024-03-27T21:42:43.925 INFO:teuthology.orchestra.run.smithi060.stdout: 410623 4 -rw-r--r-- 1 ubuntu ubuntu 1801 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/btree_map.h 2024-03-27T21:42:43.925 INFO:teuthology.orchestra.run.smithi060.stdout: 410614 4 -rw-r--r-- 1 ubuntu ubuntu 953 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/CMakeLists.txt 2024-03-27T21:42:43.925 INFO:teuthology.orchestra.run.smithi060.stdout: 410615 12 -rw-r--r-- 1 ubuntu ubuntu 8745 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/CompatSet.h 2024-03-27T21:42:43.926 INFO:teuthology.orchestra.run.smithi060.stdout: 410715 4 -rw-r--r-- 1 ubuntu ubuntu 1427 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/scope_guard.h 2024-03-27T21:42:43.926 INFO:teuthology.orchestra.run.smithi060.stdout: 410674 4 -rw-r--r-- 1 ubuntu ubuntu 1325 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/ipaddr.h 2024-03-27T21:42:43.926 INFO:teuthology.orchestra.run.smithi060.stdout: 410679 20 -rw-r--r-- 1 ubuntu ubuntu 18065 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/mempool.h 2024-03-27T21:42:43.926 INFO:teuthology.orchestra.run.smithi060.stdout: 410729 4 -rw-r--r-- 1 ubuntu ubuntu 855 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/utime.cc 2024-03-27T21:42:43.926 INFO:teuthology.orchestra.run.smithi060.stdout: 410656 52 -rw-r--r-- 1 ubuntu ubuntu 52631 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/denc.h 2024-03-27T21:42:43.926 INFO:teuthology.orchestra.run.smithi060.stdout: 410663 92 -rw-r--r-- 1 ubuntu ubuntu 92778 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/expected.hpp 2024-03-27T21:42:43.926 INFO:teuthology.orchestra.run.smithi060.stdout: 410668 4 -rw-r--r-- 1 ubuntu ubuntu 1422 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/hash.h 2024-03-27T21:42:43.926 INFO:teuthology.orchestra.run.smithi060.stdout: 410724 20 -rw-r--r-- 1 ubuntu ubuntu 18703 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/types.h 2024-03-27T21:42:43.927 INFO:teuthology.orchestra.run.smithi060.stdout: 410653 4 -rw-r--r-- 1 ubuntu ubuntu 685 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/cpp_lib_backport.h 2024-03-27T21:42:43.927 INFO:teuthology.orchestra.run.smithi060.stdout: 410731 4 -rw-r--r-- 1 ubuntu ubuntu 1245 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/utime_fmt.h 2024-03-27T21:42:43.927 INFO:teuthology.orchestra.run.smithi060.stdout: 410645 12 -rw-r--r-- 1 ubuntu ubuntu 10533 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/config-h.in.cmake 2024-03-27T21:42:43.927 INFO:teuthology.orchestra.run.smithi060.stdout: 410727 12 -rw-r--r-- 1 ubuntu ubuntu 11402 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/uses_allocator.h 2024-03-27T21:42:43.927 INFO:teuthology.orchestra.run.smithi060.stdout: 410670 8 -rw-r--r-- 1 ubuntu ubuntu 4784 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/inline_memory.h 2024-03-27T21:42:43.927 INFO:teuthology.orchestra.run.smithi060.stdout: 410726 4 -rw-r--r-- 1 ubuntu ubuntu 140 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/unordered_set.h 2024-03-27T21:42:43.927 INFO:teuthology.orchestra.run.smithi060.stdout: 410689 28 -rw-r--r-- 1 ubuntu ubuntu 24693 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rados.h 2024-03-27T21:42:43.927 INFO:teuthology.orchestra.run.smithi060.stdout: 410718 4 -rw-r--r-- 1 ubuntu ubuntu 2716 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/stat.h 2024-03-27T21:42:43.928 INFO:teuthology.orchestra.run.smithi060.stdout: 410655 4 -rw-r--r-- 1 ubuntu ubuntu 1146 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/demangle.h 2024-03-27T21:42:43.928 INFO:teuthology.orchestra.run.smithi060.stdout: 410688 4 -rw-r--r-- 1 ubuntu ubuntu 323 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/page.h 2024-03-27T21:42:43.928 INFO:teuthology.orchestra.run.smithi060.stdout: 410616 16 -rw-r--r-- 1 ubuntu ubuntu 13036 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/Context.h 2024-03-27T21:42:43.928 INFO:teuthology.orchestra.run.smithi060.stdout: 410647 4 -rw-r--r-- 1 ubuntu ubuntu 1114 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/counter.h 2024-03-27T21:42:43.928 INFO:teuthology.orchestra.run.smithi060.stdout: 410730 16 -rw-r--r-- 1 ubuntu ubuntu 16154 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/utime.h 2024-03-27T21:42:43.928 INFO:teuthology.orchestra.run.smithi060.stdout: 410671 4 -rw-r--r-- 1 ubuntu ubuntu 883 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/int_types.h 2024-03-27T21:42:43.928 INFO:teuthology.orchestra.run.smithi060.stdout: 410677 4 -rw-r--r-- 1 ubuntu ubuntu 2748 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/linux_fiemap.h 2024-03-27T21:42:43.928 INFO:teuthology.orchestra.run.smithi060.stdout: 410658 8 -rw-r--r-- 1 ubuntu ubuntu 4262 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/elist.h 2024-03-27T21:42:43.929 INFO:teuthology.orchestra.run.smithi060.stdout: 410733 4 -rw-r--r-- 1 ubuntu ubuntu 2326 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/uuid.h 2024-03-27T21:42:43.929 INFO:teuthology.orchestra.run.smithi060.stdout: 410626 4 -rw-r--r-- 1 ubuntu ubuntu 3507 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/buffer_raw.h 2024-03-27T21:42:43.929 INFO:teuthology.orchestra.run.smithi060.stdout: 410680 8 -rw-r--r-- 1 ubuntu ubuntu 7552 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/msgr.h 2024-03-27T21:42:43.929 INFO:teuthology.orchestra.run.smithi060.stdout: 410642 12 -rw-r--r-- 1 ubuntu ubuntu 11062 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/compact_map.h 2024-03-27T21:42:43.929 INFO:teuthology.orchestra.run.smithi060.stdout: 410643 12 -rw-r--r-- 1 ubuntu ubuntu 8630 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/compact_set.h 2024-03-27T21:42:43.929 INFO:teuthology.orchestra.run.smithi060.stdout: 410633 4 -rw-r--r-- 1 ubuntu ubuntu 476 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/ceph_hash.h 2024-03-27T21:42:43.929 INFO:teuthology.orchestra.run.smithi060.stdout: 410654 4 -rw-r--r-- 1 ubuntu ubuntu 1356 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/crc32c.h 2024-03-27T21:42:43.929 INFO:teuthology.orchestra.run.smithi060.stdout: 410621 4 -rw-r--r-- 1 ubuntu ubuntu 1099 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/bitmapper.h 2024-03-27T21:42:43.930 INFO:teuthology.orchestra.run.smithi060.stdout: 410665 20 -rw-r--r-- 1 ubuntu ubuntu 16625 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/frag.h 2024-03-27T21:42:43.930 INFO:teuthology.orchestra.run.smithi060.stdout: 410622 4 -rw-r--r-- 1 ubuntu ubuntu 1272 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/blobhash.h 2024-03-27T21:42:43.930 INFO:teuthology.orchestra.run.smithi060.stdout: 410662 4 -rw-r--r-- 1 ubuntu ubuntu 640 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/event_type.h 2024-03-27T21:42:43.930 INFO:teuthology.orchestra.run.smithi060.stdout: 410704 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/radosstriper 2024-03-27T21:42:43.930 INFO:teuthology.orchestra.run.smithi060.stdout: 410706 8 -rw-r--r-- 1 ubuntu ubuntu 7815 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/radosstriper/libradosstriper.hpp 2024-03-27T21:42:43.930 INFO:teuthology.orchestra.run.smithi060.stdout: 410705 24 -rw-r--r-- 1 ubuntu ubuntu 21367 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/radosstriper/libradosstriper.h 2024-03-27T21:42:43.930 INFO:teuthology.orchestra.run.smithi060.stdout: 410716 4 -rw-r--r-- 1 ubuntu ubuntu 1133 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/sock_compat.h 2024-03-27T21:42:43.930 INFO:teuthology.orchestra.run.smithi060.stdout: 410625 4 -rw-r--r-- 1 ubuntu ubuntu 288 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/buffer_fwd.h 2024-03-27T21:42:43.931 INFO:teuthology.orchestra.run.smithi060.stdout: 410717 4 -rw-r--r-- 1 ubuntu ubuntu 1725 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/spinlock.h 2024-03-27T21:42:43.931 INFO:teuthology.orchestra.run.smithi060.stdout: 410675 4 -rw-r--r-- 1 ubuntu ubuntu 3294 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/krbd.h 2024-03-27T21:42:43.931 INFO:teuthology.orchestra.run.smithi060.stdout: 410673 20 -rw-r--r-- 1 ubuntu ubuntu 19553 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/interval_set.h 2024-03-27T21:42:43.931 INFO:teuthology.orchestra.run.smithi060.stdout: 410734 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32 2024-03-27T21:42:43.931 INFO:teuthology.orchestra.run.smithi060.stdout: 410735 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/arpa 2024-03-27T21:42:43.931 INFO:teuthology.orchestra.run.smithi060.stdout: 410736 4 -rw-r--r-- 1 ubuntu ubuntu 28 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/arpa/inet.h 2024-03-27T21:42:43.931 INFO:teuthology.orchestra.run.smithi060.stdout: 410740 4 -rw-r--r-- 1 ubuntu ubuntu 28 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/netdb.h 2024-03-27T21:42:43.931 INFO:teuthology.orchestra.run.smithi060.stdout: 410757 4 -rw-r--r-- 1 ubuntu ubuntu 683 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/winsock_wrapper.h 2024-03-27T21:42:43.931 INFO:teuthology.orchestra.run.smithi060.stdout: 410755 4 -rw-r--r-- 1 ubuntu ubuntu 1204 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/winioctl_compat.h 2024-03-27T21:42:43.932 INFO:teuthology.orchestra.run.smithi060.stdout: 410737 4 -rw-r--r-- 1 ubuntu ubuntu 29 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/dlfcn.h 2024-03-27T21:42:43.932 INFO:teuthology.orchestra.run.smithi060.stdout: 410756 4 -rw-r--r-- 1 ubuntu ubuntu 1000 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/winsock_compat.h 2024-03-27T21:42:43.932 INFO:teuthology.orchestra.run.smithi060.stdout: 410753 4 -rw-r--r-- 1 ubuntu ubuntu 2597 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/syslog.h 2024-03-27T21:42:43.932 INFO:teuthology.orchestra.run.smithi060.stdout: 410741 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/netinet 2024-03-27T21:42:43.932 INFO:teuthology.orchestra.run.smithi060.stdout: 410742 4 -rw-r--r-- 1 ubuntu ubuntu 28 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/netinet/in.h 2024-03-27T21:42:43.932 INFO:teuthology.orchestra.run.smithi060.stdout: 410743 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/netinet/ip.h 2024-03-27T21:42:43.932 INFO:teuthology.orchestra.run.smithi060.stdout: 410744 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/netinet/tcp.h 2024-03-27T21:42:43.932 INFO:teuthology.orchestra.run.smithi060.stdout: 410754 4 -rw-r--r-- 1 ubuntu ubuntu 2671 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/win32_errno.h 2024-03-27T21:42:43.932 INFO:teuthology.orchestra.run.smithi060.stdout: 410738 4 -rw-r--r-- 1 ubuntu ubuntu 1069 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/fs_compat.h 2024-03-27T21:42:43.933 INFO:teuthology.orchestra.run.smithi060.stdout: 410745 4 -rw-r--r-- 1 ubuntu ubuntu 28 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/poll.h 2024-03-27T21:42:43.933 INFO:teuthology.orchestra.run.smithi060.stdout: 410739 4 -rw-r--r-- 1 ubuntu ubuntu 1100 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/ifaddrs.h 2024-03-27T21:42:43.933 INFO:teuthology.orchestra.run.smithi060.stdout: 410746 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/sys 2024-03-27T21:42:43.933 INFO:teuthology.orchestra.run.smithi060.stdout: 410749 4 -rw-r--r-- 1 ubuntu ubuntu 28 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/sys/socket.h 2024-03-27T21:42:43.933 INFO:teuthology.orchestra.run.smithi060.stdout: 410748 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/sys/select.h 2024-03-27T21:42:43.933 INFO:teuthology.orchestra.run.smithi060.stdout: 410747 4 -rw-r--r-- 1 ubuntu ubuntu 19 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/sys/errno.h 2024-03-27T21:42:43.933 INFO:teuthology.orchestra.run.smithi060.stdout: 410751 4 -rw-r--r-- 1 ubuntu ubuntu 28 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/sys/uio.h 2024-03-27T21:42:43.933 INFO:teuthology.orchestra.run.smithi060.stdout: 410752 4 -rw-r--r-- 1 ubuntu ubuntu 42 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/sys/un.h 2024-03-27T21:42:43.934 INFO:teuthology.orchestra.run.smithi060.stdout: 410750 4 -rw-r--r-- 1 ubuntu ubuntu 734 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/win32/sys/statvfs.h 2024-03-27T21:42:43.934 INFO:teuthology.orchestra.run.smithi060.stdout: 410629 12 -rw-r--r-- 1 ubuntu ubuntu 11741 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/ceph_features.h 2024-03-27T21:42:43.934 INFO:teuthology.orchestra.run.smithi060.stdout: 410618 4 -rw-r--r-- 1 ubuntu ubuntu 525 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/addr_parsing.h 2024-03-27T21:42:43.934 INFO:teuthology.orchestra.run.smithi060.stdout: 410644 12 -rw-r--r-- 1 ubuntu ubuntu 11144 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/compat.h 2024-03-27T21:42:43.934 INFO:teuthology.orchestra.run.smithi060.stdout: 410617 4 -rw-r--r-- 1 ubuntu ubuntu 1418 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/Distribution.h 2024-03-27T21:42:43.934 INFO:teuthology.orchestra.run.smithi060.stdout: 410630 4 -rw-r--r-- 1 ubuntu ubuntu 3114 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/ceph_frag.h 2024-03-27T21:42:43.934 INFO:teuthology.orchestra.run.smithi060.stdout: 410721 8 -rw-r--r-- 1 ubuntu ubuntu 6044 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/str_map.h 2024-03-27T21:42:43.934 INFO:teuthology.orchestra.run.smithi060.stdout: 410672 4 -rw-r--r-- 1 ubuntu ubuntu 2630 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/intarith.h 2024-03-27T21:42:43.935 INFO:teuthology.orchestra.run.smithi060.stdout: 410686 8 -rw-r--r-- 1 ubuntu ubuntu 5506 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/object.h 2024-03-27T21:42:43.935 INFO:teuthology.orchestra.run.smithi060.stdout: 410709 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rbd 2024-03-27T21:42:43.935 INFO:teuthology.orchestra.run.smithi060.stdout: 410711 68 -rw-r--r-- 1 ubuntu ubuntu 67649 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rbd/librbd.h 2024-03-27T21:42:43.935 INFO:teuthology.orchestra.run.smithi060.stdout: 410710 8 -rw-r--r-- 1 ubuntu ubuntu 6383 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rbd/features.h 2024-03-27T21:42:43.935 INFO:teuthology.orchestra.run.smithi060.stdout: 410712 36 -rw-r--r-- 1 ubuntu ubuntu 32889 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rbd/librbd.hpp 2024-03-27T21:42:43.935 INFO:teuthology.orchestra.run.smithi060.stdout: 410713 4 -rw-r--r-- 1 ubuntu ubuntu 426 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rbd/object_map_types.h 2024-03-27T21:42:43.935 INFO:teuthology.orchestra.run.smithi060.stdout: 410722 4 -rw-r--r-- 1 ubuntu ubuntu 625 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/stringify.h 2024-03-27T21:42:43.935 INFO:teuthology.orchestra.run.smithi060.stdout: 410687 4 -rw-r--r-- 1 ubuntu ubuntu 1043 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/on_exit.h 2024-03-27T21:42:43.935 INFO:teuthology.orchestra.run.smithi060.stdout: 410664 8 -rw-r--r-- 1 ubuntu ubuntu 6138 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/filepath.h 2024-03-27T21:42:43.936 INFO:teuthology.orchestra.run.smithi060.stdout: 410678 8 -rw-r--r-- 1 ubuntu ubuntu 5647 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/lru.h 2024-03-27T21:42:43.936 INFO:teuthology.orchestra.run.smithi060.stdout: 410657 4 -rw-r--r-- 1 ubuntu ubuntu 1234 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/dlfcn_compat.h 2024-03-27T21:42:43.936 INFO:teuthology.orchestra.run.smithi060.stdout: 410619 4 -rw-r--r-- 1 ubuntu ubuntu 2562 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/alloc_ptr.h 2024-03-27T21:42:43.936 INFO:teuthology.orchestra.run.smithi060.stdout: 410627 4 -rw-r--r-- 1 ubuntu ubuntu 1137 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/byteorder.h 2024-03-27T21:42:43.936 INFO:teuthology.orchestra.run.smithi060.stdout: 410628 8 -rw-r--r-- 1 ubuntu ubuntu 5066 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/ceph_assert.h 2024-03-27T21:42:43.936 INFO:teuthology.orchestra.run.smithi060.stdout: 410681 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/neorados 2024-03-27T21:42:43.936 INFO:teuthology.orchestra.run.smithi060.stdout: 410682 64 -rw-r--r-- 1 ubuntu ubuntu 61805 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/neorados/RADOS.hpp 2024-03-27T21:42:43.937 INFO:teuthology.orchestra.run.smithi060.stdout: 410683 4 -rw-r--r-- 1 ubuntu ubuntu 2073 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/neorados/RADOS_Decodable.hpp 2024-03-27T21:42:43.937 INFO:teuthology.orchestra.run.smithi060.stdout: 410684 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/neorados/buffer_fwd.h -> ../buffer_fwd.h 2024-03-27T21:42:43.937 INFO:teuthology.orchestra.run.smithi060.stdout: 410685 0 lrwxrwxrwx 1 ubuntu ubuntu 31 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/neorados/completion.h -> ../../common/async/completion.h 2024-03-27T21:42:43.937 INFO:teuthology.orchestra.run.smithi060.stdout: 410714 8 -rw-r--r-- 1 ubuntu ubuntu 4766 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/rbd_types.h 2024-03-27T21:42:43.937 INFO:teuthology.orchestra.run.smithi060.stdout: 410666 8 -rw-r--r-- 1 ubuntu ubuntu 5176 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/fs_types.h 2024-03-27T21:42:43.937 INFO:teuthology.orchestra.run.smithi060.stdout: 410631 40 -rw-r--r-- 1 ubuntu ubuntu 37413 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/ceph_fs.h 2024-03-27T21:42:43.937 INFO:teuthology.orchestra.run.smithi060.stdout: 410632 4 -rw-r--r-- 1 ubuntu ubuntu 1529 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/ceph_fuse.h 2024-03-27T21:42:43.937 INFO:teuthology.orchestra.run.smithi060.stdout: 410728 4 -rw-r--r-- 1 ubuntu ubuntu 3494 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/util.h 2024-03-27T21:42:43.938 INFO:teuthology.orchestra.run.smithi060.stdout: 410667 64 -rw-r--r-- 1 ubuntu ubuntu 62171 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/function2.hpp 2024-03-27T21:42:43.938 INFO:teuthology.orchestra.run.smithi060.stdout: 410640 4 -rw-r--r-- 1 ubuntu ubuntu 309 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/color.h 2024-03-27T21:42:43.938 INFO:teuthology.orchestra.run.smithi060.stdout: 410624 36 -rw-r--r-- 1 ubuntu ubuntu 36350 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/buffer.h 2024-03-27T21:42:43.938 INFO:teuthology.orchestra.run.smithi060.stdout: 410725 4 -rw-r--r-- 1 ubuntu ubuntu 173 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/include/unordered_map.h 2024-03-27T21:42:43.938 INFO:teuthology.orchestra.run.smithi060.stdout: 409090 8 -rwxr-xr-x 1 ubuntu ubuntu 4160 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-post-file.in 2024-03-27T21:42:43.938 INFO:teuthology.orchestra.run.smithi060.stdout: 409383 8 -rw-r--r-- 1 ubuntu ubuntu 7183 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph_mds.cc 2024-03-27T21:42:43.938 INFO:teuthology.orchestra.run.smithi060.stdout: 411805 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon 2024-03-27T21:42:43.938 INFO:teuthology.orchestra.run.smithi060.stdout: 411854 32 -rw-r--r-- 1 ubuntu ubuntu 31002 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/OSDMonitor.h 2024-03-27T21:42:43.939 INFO:teuthology.orchestra.run.smithi060.stdout: 411811 12 -rw-r--r-- 1 ubuntu ubuntu 8929 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/ConfigMap.cc 2024-03-27T21:42:43.939 INFO:teuthology.orchestra.run.smithi060.stdout: 411838 8 -rw-r--r-- 1 ubuntu ubuntu 6644 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/MonCap.h 2024-03-27T21:42:43.939 INFO:teuthology.orchestra.run.smithi060.stdout: 411815 12 -rw-r--r-- 1 ubuntu ubuntu 12084 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/ConnectionTracker.cc 2024-03-27T21:42:43.939 INFO:teuthology.orchestra.run.smithi060.stdout: 411847 4 -rw-r--r-- 1 ubuntu ubuntu 1466 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/MonSub.h 2024-03-27T21:42:43.939 INFO:teuthology.orchestra.run.smithi060.stdout: 411852 4 -rw-r--r-- 1 ubuntu ubuntu 3239 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/MonmapMonitor.h 2024-03-27T21:42:43.939 INFO:teuthology.orchestra.run.smithi060.stdout: 411832 20 -rw-r--r-- 1 ubuntu ubuntu 19485 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/MgrMap.h 2024-03-27T21:42:43.939 INFO:teuthology.orchestra.run.smithi060.stdout: 411808 4 -rw-r--r-- 1 ubuntu ubuntu 965 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/CMakeLists.txt 2024-03-27T21:42:43.939 INFO:teuthology.orchestra.run.smithi060.stdout: 411807 12 -rw-r--r-- 1 ubuntu ubuntu 8826 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/AuthMonitor.h 2024-03-27T21:42:43.939 INFO:teuthology.orchestra.run.smithi060.stdout: 411831 8 -rw-r--r-- 1 ubuntu ubuntu 5035 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/MDSMonitor.h 2024-03-27T21:42:43.939 INFO:teuthology.orchestra.run.smithi060.stdout: 411860 16 -rw-r--r-- 1 ubuntu ubuntu 14075 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/PaxosService.cc 2024-03-27T21:42:43.940 INFO:teuthology.orchestra.run.smithi060.stdout: 411840 24 -rw-r--r-- 1 ubuntu ubuntu 22712 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/MonClient.h 2024-03-27T21:42:43.940 INFO:teuthology.orchestra.run.smithi060.stdout: 411824 28 -rw-r--r-- 1 ubuntu ubuntu 24732 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/HealthMonitor.cc 2024-03-27T21:42:43.940 INFO:teuthology.orchestra.run.smithi060.stdout: 411862 8 -rw-r--r-- 1 ubuntu ubuntu 8043 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/Session.h 2024-03-27T21:42:43.940 INFO:teuthology.orchestra.run.smithi060.stdout: 411856 20 -rw-r--r-- 1 ubuntu ubuntu 18840 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/PGMap.h 2024-03-27T21:42:43.940 INFO:teuthology.orchestra.run.smithi060.stdout: 411830 76 -rw-r--r-- 1 ubuntu ubuntu 74317 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/MDSMonitor.cc 2024-03-27T21:42:43.940 INFO:teuthology.orchestra.run.smithi060.stdout: 411820 24 -rw-r--r-- 1 ubuntu ubuntu 24473 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/Elector.cc 2024-03-27T21:42:43.940 INFO:teuthology.orchestra.run.smithi060.stdout: 411861 28 -rw-r--r-- 1 ubuntu ubuntu 26265 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/PaxosService.h 2024-03-27T21:42:43.940 INFO:teuthology.orchestra.run.smithi060.stdout: 411817 8 -rw-r--r-- 1 ubuntu ubuntu 7095 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/CreatingPGs.h 2024-03-27T21:42:43.940 INFO:teuthology.orchestra.run.smithi060.stdout: 411841 8 -rw-r--r-- 1 ubuntu ubuntu 5427 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/MonCommand.h 2024-03-27T21:42:43.941 INFO:teuthology.orchestra.run.smithi060.stdout: 411849 36 -rw-r--r-- 1 ubuntu ubuntu 36380 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/Monitor.h 2024-03-27T21:42:43.941 INFO:teuthology.orchestra.run.smithi060.stdout: 411828 36 -rw-r--r-- 1 ubuntu ubuntu 35966 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/LogMonitor.cc 2024-03-27T21:42:43.941 INFO:teuthology.orchestra.run.smithi060.stdout: 411806 64 -rw-r--r-- 1 ubuntu ubuntu 62848 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/AuthMonitor.cc 2024-03-27T21:42:43.941 INFO:teuthology.orchestra.run.smithi060.stdout: 411839 60 -rw-r--r-- 1 ubuntu ubuntu 58331 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/MonClient.cc 2024-03-27T21:42:43.941 INFO:teuthology.orchestra.run.smithi060.stdout: 411857 48 -rw-r--r-- 1 ubuntu ubuntu 47542 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/Paxos.cc 2024-03-27T21:42:43.941 INFO:teuthology.orchestra.run.smithi060.stdout: 411834 8 -rw-r--r-- 1 ubuntu ubuntu 4401 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/MgrMonitor.h 2024-03-27T21:42:43.941 INFO:teuthology.orchestra.run.smithi060.stdout: 411829 8 -rw-r--r-- 1 ubuntu ubuntu 6236 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/LogMonitor.h 2024-03-27T21:42:43.941 INFO:teuthology.orchestra.run.smithi060.stdout: 411865 8 -rw-r--r-- 1 ubuntu ubuntu 4914 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/health_check.h 2024-03-27T21:42:43.941 INFO:teuthology.orchestra.run.smithi060.stdout: 411825 4 -rw-r--r-- 1 ubuntu ubuntu 2202 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/HealthMonitor.h 2024-03-27T21:42:43.942 INFO:teuthology.orchestra.run.smithi060.stdout: 411822 60 -rw-r--r-- 1 ubuntu ubuntu 59504 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/FSCommands.cc 2024-03-27T21:42:43.942 INFO:teuthology.orchestra.run.smithi060.stdout: 411819 20 -rw-r--r-- 1 ubuntu ubuntu 16840 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/ElectionLogic.h 2024-03-27T21:42:43.942 INFO:teuthology.orchestra.run.smithi060.stdout: 411812 8 -rw-r--r-- 1 ubuntu ubuntu 4504 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/ConfigMap.h 2024-03-27T21:42:43.942 INFO:teuthology.orchestra.run.smithi060.stdout: 411850 24 -rw-r--r-- 1 ubuntu ubuntu 22848 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/MonitorDBStore.h 2024-03-27T21:42:43.942 INFO:teuthology.orchestra.run.smithi060.stdout: 411827 4 -rw-r--r-- 1 ubuntu ubuntu 1945 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/KVMonitor.h 2024-03-27T21:42:43.942 INFO:teuthology.orchestra.run.smithi060.stdout: 411809 4 -rw-r--r-- 1 ubuntu ubuntu 1083 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/CommandHandler.cc 2024-03-27T21:42:43.942 INFO:teuthology.orchestra.run.smithi060.stdout: 411859 4 -rw-r--r-- 1 ubuntu ubuntu 1354 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/PaxosFSMap.h 2024-03-27T21:42:43.942 INFO:teuthology.orchestra.run.smithi060.stdout: 411821 16 -rw-r--r-- 1 ubuntu ubuntu 12433 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/Elector.h 2024-03-27T21:42:43.943 INFO:teuthology.orchestra.run.smithi060.stdout: 411836 4 -rw-r--r-- 1 ubuntu ubuntu 3045 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/MgrStatMonitor.h 2024-03-27T21:42:43.943 INFO:teuthology.orchestra.run.smithi060.stdout: 411814 4 -rw-r--r-- 1 ubuntu ubuntu 1565 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/ConfigMonitor.h 2024-03-27T21:42:43.943 INFO:teuthology.orchestra.run.smithi060.stdout: 411837 28 -rw-r--r-- 1 ubuntu ubuntu 25669 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/MonCap.cc 2024-03-27T21:42:43.943 INFO:teuthology.orchestra.run.smithi060.stdout: 411845 8 -rw-r--r-- 1 ubuntu ubuntu 5816 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/MonOpRequest.h 2024-03-27T21:42:43.943 INFO:teuthology.orchestra.run.smithi060.stdout: 411835 12 -rw-r--r-- 1 ubuntu ubuntu 10479 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/MgrStatMonitor.cc 2024-03-27T21:42:43.943 INFO:teuthology.orchestra.run.smithi060.stdout: 411851 48 -rw-r--r-- 1 ubuntu ubuntu 45896 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/MonmapMonitor.cc 2024-03-27T21:42:43.943 INFO:teuthology.orchestra.run.smithi060.stdout: 411863 4 -rw-r--r-- 1 ubuntu ubuntu 2260 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/error_code.cc 2024-03-27T21:42:43.944 INFO:teuthology.orchestra.run.smithi060.stdout: 411844 16 -rw-r--r-- 1 ubuntu ubuntu 15785 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/MonMap.h 2024-03-27T21:42:43.944 INFO:teuthology.orchestra.run.smithi060.stdout: 411843 32 -rw-r--r-- 1 ubuntu ubuntu 31027 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/MonMap.cc 2024-03-27T21:42:43.944 INFO:teuthology.orchestra.run.smithi060.stdout: 411842 60 -rw-r--r-- 1 ubuntu ubuntu 58335 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/MonCommands.h 2024-03-27T21:42:43.944 INFO:teuthology.orchestra.run.smithi060.stdout: 411855 132 -rw-r--r-- 1 ubuntu ubuntu 132075 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/PGMap.cc 2024-03-27T21:42:43.944 INFO:teuthology.orchestra.run.smithi060.stdout: 411816 8 -rw-r--r-- 1 ubuntu ubuntu 6799 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/ConnectionTracker.h 2024-03-27T21:42:43.944 INFO:teuthology.orchestra.run.smithi060.stdout: 411846 4 -rw-r--r-- 1 ubuntu ubuntu 2688 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/MonSub.cc 2024-03-27T21:42:43.944 INFO:teuthology.orchestra.run.smithi060.stdout: 411823 4 -rw-r--r-- 1 ubuntu ubuntu 2100 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/FSCommands.h 2024-03-27T21:42:43.944 INFO:teuthology.orchestra.run.smithi060.stdout: 411848 208 -rw-r--r-- 1 ubuntu ubuntu 210520 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/Monitor.cc 2024-03-27T21:42:43.945 INFO:teuthology.orchestra.run.smithi060.stdout: 411853 476 -rw-r--r-- 1 ubuntu ubuntu 483479 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/OSDMonitor.cc 2024-03-27T21:42:43.945 INFO:teuthology.orchestra.run.smithi060.stdout: 411813 28 -rw-r--r-- 1 ubuntu ubuntu 26529 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/ConfigMonitor.cc 2024-03-27T21:42:43.945 INFO:teuthology.orchestra.run.smithi060.stdout: 411810 4 -rw-r--r-- 1 ubuntu ubuntu 823 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/CommandHandler.h 2024-03-27T21:42:43.945 INFO:teuthology.orchestra.run.smithi060.stdout: 411858 44 -rw-r--r-- 1 ubuntu ubuntu 43642 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/Paxos.h 2024-03-27T21:42:43.945 INFO:teuthology.orchestra.run.smithi060.stdout: 411833 44 -rw-r--r-- 1 ubuntu ubuntu 43269 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/MgrMonitor.cc 2024-03-27T21:42:43.945 INFO:teuthology.orchestra.run.smithi060.stdout: 411864 4 -rw-r--r-- 1 ubuntu ubuntu 1230 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/error_code.h 2024-03-27T21:42:43.945 INFO:teuthology.orchestra.run.smithi060.stdout: 411818 20 -rw-r--r-- 1 ubuntu ubuntu 17879 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/ElectionLogic.cc 2024-03-27T21:42:43.945 INFO:teuthology.orchestra.run.smithi060.stdout: 411826 16 -rw-r--r-- 1 ubuntu ubuntu 12341 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/KVMonitor.cc 2024-03-27T21:42:43.945 INFO:teuthology.orchestra.run.smithi060.stdout: 411866 20 -rw-r--r-- 1 ubuntu ubuntu 20114 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mon/mon_types.h 2024-03-27T21:42:43.946 INFO:teuthology.orchestra.run.smithi060.stdout: 411877 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg 2024-03-27T21:42:43.946 INFO:teuthology.orchestra.run.smithi060.stdout: 411878 4 -rw-r--r-- 1 ubuntu ubuntu 1849 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/CMakeLists.txt 2024-03-27T21:42:43.946 INFO:teuthology.orchestra.run.smithi060.stdout: 411975 4 -rw-r--r-- 1 ubuntu ubuntu 654 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/msg_fmt.h 2024-03-27T21:42:43.946 INFO:teuthology.orchestra.run.smithi060.stdout: 411976 12 -rw-r--r-- 1 ubuntu ubuntu 10069 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/msg_types.cc 2024-03-27T21:42:43.946 INFO:teuthology.orchestra.run.smithi060.stdout: 411881 8 -rw-r--r-- 1 ubuntu ubuntu 6999 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/DispatchQueue.cc 2024-03-27T21:42:43.946 INFO:teuthology.orchestra.run.smithi060.stdout: 411890 4 -rw-r--r-- 1 ubuntu ubuntu 2806 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/SimplePolicyMessenger.h 2024-03-27T21:42:43.946 INFO:teuthology.orchestra.run.smithi060.stdout: 411973 4 -rw-r--r-- 1 ubuntu ubuntu 3797 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/compressor_registry.cc 2024-03-27T21:42:43.946 INFO:teuthology.orchestra.run.smithi060.stdout: 411887 4 -rw-r--r-- 1 ubuntu ubuntu 3152 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/Messenger.cc 2024-03-27T21:42:43.947 INFO:teuthology.orchestra.run.smithi060.stdout: 411889 8 -rw-r--r-- 1 ubuntu ubuntu 4232 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/Policy.h 2024-03-27T21:42:43.947 INFO:teuthology.orchestra.run.smithi060.stdout: 411891 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async 2024-03-27T21:42:43.947 INFO:teuthology.orchestra.run.smithi060.stdout: 411909 4 -rw-r--r-- 1 ubuntu ubuntu 3665 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/Protocol.h 2024-03-27T21:42:43.947 INFO:teuthology.orchestra.run.smithi060.stdout: 411920 8 -rw-r--r-- 1 ubuntu ubuntu 4123 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/crypto_onwire.h 2024-03-27T21:42:43.947 INFO:teuthology.orchestra.run.smithi060.stdout: 411901 4 -rw-r--r-- 1 ubuntu ubuntu 1614 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/EventKqueue.h 2024-03-27T21:42:43.947 INFO:teuthology.orchestra.run.smithi060.stdout: 411892 28 -rw-r--r-- 1 ubuntu ubuntu 24932 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/AsyncConnection.cc 2024-03-27T21:42:43.947 INFO:teuthology.orchestra.run.smithi060.stdout: 411908 4 -rw-r--r-- 1 ubuntu ubuntu 340 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/Protocol.cc 2024-03-27T21:42:43.947 INFO:teuthology.orchestra.run.smithi060.stdout: 411904 4 -rw-r--r-- 1 ubuntu ubuntu 2356 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/EventSelect.cc 2024-03-27T21:42:43.948 INFO:teuthology.orchestra.run.smithi060.stdout: 411910 88 -rw-r--r-- 1 ubuntu ubuntu 88685 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/ProtocolV1.cc 2024-03-27T21:42:43.948 INFO:teuthology.orchestra.run.smithi060.stdout: 411894 28 -rw-r--r-- 1 ubuntu ubuntu 26848 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/AsyncMessenger.cc 2024-03-27T21:42:43.948 INFO:teuthology.orchestra.run.smithi060.stdout: 411961 32 -rw-r--r-- 1 ubuntu ubuntu 30923 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/frames_v2.h 2024-03-27T21:42:43.948 INFO:teuthology.orchestra.run.smithi060.stdout: 411919 12 -rw-r--r-- 1 ubuntu ubuntu 9742 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/crypto_onwire.cc 2024-03-27T21:42:43.948 INFO:teuthology.orchestra.run.smithi060.stdout: 411900 8 -rw-r--r-- 1 ubuntu ubuntu 8181 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/EventKqueue.cc 2024-03-27T21:42:43.948 INFO:teuthology.orchestra.run.smithi060.stdout: 411898 8 -rw-r--r-- 1 ubuntu ubuntu 4238 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/EventEpoll.cc 2024-03-27T21:42:43.948 INFO:teuthology.orchestra.run.smithi060.stdout: 411895 12 -rw-r--r-- 1 ubuntu ubuntu 11699 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/AsyncMessenger.h 2024-03-27T21:42:43.948 INFO:teuthology.orchestra.run.smithi060.stdout: 411897 8 -rw-r--r-- 1 ubuntu ubuntu 7469 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/Event.h 2024-03-27T21:42:43.948 INFO:teuthology.orchestra.run.smithi060.stdout: 411917 4 -rw-r--r-- 1 ubuntu ubuntu 2397 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/compression_onwire.cc 2024-03-27T21:42:43.949 INFO:teuthology.orchestra.run.smithi060.stdout: 411907 4 -rw-r--r-- 1 ubuntu ubuntu 1594 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/PosixStack.h 2024-03-27T21:42:43.949 INFO:teuthology.orchestra.run.smithi060.stdout: 411921 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk 2024-03-27T21:42:43.949 INFO:teuthology.orchestra.run.smithi060.stdout: 411936 8 -rw-r--r-- 1 ubuntu ubuntu 4892 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/PacketUtil.h 2024-03-27T21:42:43.949 INFO:teuthology.orchestra.run.smithi060.stdout: 411931 12 -rw-r--r-- 1 ubuntu ubuntu 11881 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/IP.h 2024-03-27T21:42:43.949 INFO:teuthology.orchestra.run.smithi060.stdout: 411952 4 -rw-r--r-- 1 ubuntu ubuntu 2941 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/ip_types.h 2024-03-27T21:42:43.949 INFO:teuthology.orchestra.run.smithi060.stdout: 411955 4 -rw-r--r-- 1 ubuntu ubuntu 2210 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/queue.h 2024-03-27T21:42:43.949 INFO:teuthology.orchestra.run.smithi060.stdout: 411959 4 -rw-r--r-- 1 ubuntu ubuntu 2478 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/transfer.h 2024-03-27T21:42:43.949 INFO:teuthology.orchestra.run.smithi060.stdout: 411943 4 -rw-r--r-- 1 ubuntu ubuntu 1518 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/array_map.h 2024-03-27T21:42:43.950 INFO:teuthology.orchestra.run.smithi060.stdout: 411934 8 -rw-r--r-- 1 ubuntu ubuntu 4497 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/Packet.cc 2024-03-27T21:42:43.950 INFO:teuthology.orchestra.run.smithi060.stdout: 411938 32 -rw-r--r-- 1 ubuntu ubuntu 29016 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/TCP.cc 2024-03-27T21:42:43.950 INFO:teuthology.orchestra.run.smithi060.stdout: 411940 4 -rw-r--r-- 1 ubuntu ubuntu 3333 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/UserspaceEvent.cc 2024-03-27T21:42:43.950 INFO:teuthology.orchestra.run.smithi060.stdout: 411941 4 -rw-r--r-- 1 ubuntu ubuntu 2557 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/UserspaceEvent.h 2024-03-27T21:42:43.950 INFO:teuthology.orchestra.run.smithi060.stdout: 411922 4 -rw-r--r-- 1 ubuntu ubuntu 2433 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/ARP.cc 2024-03-27T21:42:43.950 INFO:teuthology.orchestra.run.smithi060.stdout: 411928 4 -rw-r--r-- 1 ubuntu ubuntu 2059 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/EventDPDK.cc 2024-03-27T21:42:43.950 INFO:teuthology.orchestra.run.smithi060.stdout: 411929 4 -rw-r--r-- 1 ubuntu ubuntu 1152 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/EventDPDK.h 2024-03-27T21:42:43.951 INFO:teuthology.orchestra.run.smithi060.stdout: 411925 28 -rw-r--r-- 1 ubuntu ubuntu 27139 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/DPDK.h 2024-03-27T21:42:43.951 INFO:teuthology.orchestra.run.smithi060.stdout: 411924 48 -rw-r--r-- 1 ubuntu ubuntu 45791 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/DPDK.cc 2024-03-27T21:42:43.951 INFO:teuthology.orchestra.run.smithi060.stdout: 411930 16 -rw-r--r-- 1 ubuntu ubuntu 14372 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/IP.cc 2024-03-27T21:42:43.951 INFO:teuthology.orchestra.run.smithi060.stdout: 411937 4 -rw-r--r-- 1 ubuntu ubuntu 1266 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/TCP-Stack.h 2024-03-27T21:42:43.951 INFO:teuthology.orchestra.run.smithi060.stdout: 411953 8 -rw-r--r-- 1 ubuntu ubuntu 6704 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/net.cc 2024-03-27T21:42:43.951 INFO:teuthology.orchestra.run.smithi060.stdout: 411946 12 -rw-r--r-- 1 ubuntu ubuntu 10157 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/circular_buffer.h 2024-03-27T21:42:43.951 INFO:teuthology.orchestra.run.smithi060.stdout: 411958 4 -rw-r--r-- 1 ubuntu ubuntu 3507 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/toeplitz.h 2024-03-27T21:42:43.951 INFO:teuthology.orchestra.run.smithi060.stdout: 411939 48 -rw-r--r-- 1 ubuntu ubuntu 46417 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/TCP.h 2024-03-27T21:42:43.951 INFO:teuthology.orchestra.run.smithi060.stdout: 411957 8 -rw-r--r-- 1 ubuntu ubuntu 4223 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/stream.h 2024-03-27T21:42:43.952 INFO:teuthology.orchestra.run.smithi060.stdout: 411949 4 -rw-r--r-- 1 ubuntu ubuntu 2314 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/dpdk_rte.h 2024-03-27T21:42:43.952 INFO:teuthology.orchestra.run.smithi060.stdout: 411945 4 -rw-r--r-- 1 ubuntu ubuntu 1259 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/capture.h 2024-03-27T21:42:43.952 INFO:teuthology.orchestra.run.smithi060.stdout: 411942 4 -rw-r--r-- 1 ubuntu ubuntu 1575 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/align.h 2024-03-27T21:42:43.952 INFO:teuthology.orchestra.run.smithi060.stdout: 411954 8 -rw-r--r-- 1 ubuntu ubuntu 4196 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/net.h 2024-03-27T21:42:43.952 INFO:teuthology.orchestra.run.smithi060.stdout: 411926 12 -rw-r--r-- 1 ubuntu ubuntu 9216 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/DPDKStack.cc 2024-03-27T21:42:43.952 INFO:teuthology.orchestra.run.smithi060.stdout: 411933 4 -rw-r--r-- 1 ubuntu ubuntu 1807 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/IPChecksum.h 2024-03-27T21:42:43.952 INFO:teuthology.orchestra.run.smithi060.stdout: 411923 12 -rw-r--r-- 1 ubuntu ubuntu 8248 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/ARP.h 2024-03-27T21:42:43.952 INFO:teuthology.orchestra.run.smithi060.stdout: 411956 12 -rw-r--r-- 1 ubuntu ubuntu 11638 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/shared_ptr.h 2024-03-27T21:42:43.953 INFO:teuthology.orchestra.run.smithi060.stdout: 411935 20 -rw-r--r-- 1 ubuntu ubuntu 17562 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/Packet.h 2024-03-27T21:42:43.953 INFO:teuthology.orchestra.run.smithi060.stdout: 411950 4 -rw-r--r-- 1 ubuntu ubuntu 338 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/ethernet.cc 2024-03-27T21:42:43.953 INFO:teuthology.orchestra.run.smithi060.stdout: 411948 8 -rw-r--r-- 1 ubuntu ubuntu 5671 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/dpdk_rte.cc 2024-03-27T21:42:43.953 INFO:teuthology.orchestra.run.smithi060.stdout: 411944 4 -rw-r--r-- 1 ubuntu ubuntu 2043 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/byteorder.h 2024-03-27T21:42:43.953 INFO:teuthology.orchestra.run.smithi060.stdout: 411951 4 -rw-r--r-- 1 ubuntu ubuntu 2297 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/ethernet.h 2024-03-27T21:42:43.953 INFO:teuthology.orchestra.run.smithi060.stdout: 411932 4 -rw-r--r-- 1 ubuntu ubuntu 2055 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/IPChecksum.cc 2024-03-27T21:42:43.953 INFO:teuthology.orchestra.run.smithi060.stdout: 411927 8 -rw-r--r-- 1 ubuntu ubuntu 7675 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/DPDKStack.h 2024-03-27T21:42:43.953 INFO:teuthology.orchestra.run.smithi060.stdout: 411947 4 -rw-r--r-- 1 ubuntu ubuntu 1293 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/dpdk/const.h 2024-03-27T21:42:43.954 INFO:teuthology.orchestra.run.smithi060.stdout: 411899 4 -rw-r--r-- 1 ubuntu ubuntu 1244 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/EventEpoll.h 2024-03-27T21:42:43.954 INFO:teuthology.orchestra.run.smithi060.stdout: 411902 8 -rw-r--r-- 1 ubuntu ubuntu 4904 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/EventPoll.cc 2024-03-27T21:42:43.954 INFO:teuthology.orchestra.run.smithi060.stdout: 411913 12 -rw-r--r-- 1 ubuntu ubuntu 10197 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/ProtocolV2.h 2024-03-27T21:42:43.954 INFO:teuthology.orchestra.run.smithi060.stdout: 411912 100 -rw-r--r-- 1 ubuntu ubuntu 101663 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/ProtocolV2.cc 2024-03-27T21:42:43.954 INFO:teuthology.orchestra.run.smithi060.stdout: 411911 16 -rw-r--r-- 1 ubuntu ubuntu 12449 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/ProtocolV1.h 2024-03-27T21:42:43.954 INFO:teuthology.orchestra.run.smithi060.stdout: 411914 8 -rw-r--r-- 1 ubuntu ubuntu 5235 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/Stack.cc 2024-03-27T21:42:43.954 INFO:teuthology.orchestra.run.smithi060.stdout: 411916 4 -rw-r--r-- 1 ubuntu ubuntu 654 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/compression_meta.h 2024-03-27T21:42:43.954 INFO:teuthology.orchestra.run.smithi060.stdout: 411918 4 -rw-r--r-- 1 ubuntu ubuntu 2561 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/compression_onwire.h 2024-03-27T21:42:43.955 INFO:teuthology.orchestra.run.smithi060.stdout: 411915 12 -rw-r--r-- 1 ubuntu ubuntu 12199 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/Stack.h 2024-03-27T21:42:43.955 INFO:teuthology.orchestra.run.smithi060.stdout: 411962 8 -rw-r--r-- 1 ubuntu ubuntu 6550 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/net_handler.cc 2024-03-27T21:42:43.955 INFO:teuthology.orchestra.run.smithi060.stdout: 411903 4 -rw-r--r-- 1 ubuntu ubuntu 1177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/EventPoll.h 2024-03-27T21:42:43.955 INFO:teuthology.orchestra.run.smithi060.stdout: 411960 20 -rw-r--r-- 1 ubuntu ubuntu 20453 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/frames_v2.cc 2024-03-27T21:42:43.955 INFO:teuthology.orchestra.run.smithi060.stdout: 411963 4 -rw-r--r-- 1 ubuntu ubuntu 1432 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/net_handler.h 2024-03-27T21:42:43.955 INFO:teuthology.orchestra.run.smithi060.stdout: 411893 8 -rw-r--r-- 1 ubuntu ubuntu 7342 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/AsyncConnection.h 2024-03-27T21:42:43.955 INFO:teuthology.orchestra.run.smithi060.stdout: 411896 16 -rw-r--r-- 1 ubuntu ubuntu 13688 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/Event.cc 2024-03-27T21:42:43.955 INFO:teuthology.orchestra.run.smithi060.stdout: 411905 4 -rw-r--r-- 1 ubuntu ubuntu 1205 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/EventSelect.h 2024-03-27T21:42:43.955 INFO:teuthology.orchestra.run.smithi060.stdout: 411906 12 -rw-r--r-- 1 ubuntu ubuntu 8822 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/PosixStack.cc 2024-03-27T21:42:43.956 INFO:teuthology.orchestra.run.smithi060.stdout: 411964 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/rdma 2024-03-27T21:42:43.956 INFO:teuthology.orchestra.run.smithi060.stdout: 411968 8 -rw-r--r-- 1 ubuntu ubuntu 5671 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/rdma/RDMAIWARPConnectedSocketImpl.cc 2024-03-27T21:42:43.956 INFO:teuthology.orchestra.run.smithi060.stdout: 411966 20 -rw-r--r-- 1 ubuntu ubuntu 17645 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/rdma/Infiniband.h 2024-03-27T21:42:43.956 INFO:teuthology.orchestra.run.smithi060.stdout: 411972 12 -rw-r--r-- 1 ubuntu ubuntu 11697 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/rdma/RDMAStack.h 2024-03-27T21:42:43.956 INFO:teuthology.orchestra.run.smithi060.stdout: 411970 4 -rw-r--r-- 1 ubuntu ubuntu 3787 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/rdma/RDMAServerSocketImpl.cc 2024-03-27T21:42:43.956 INFO:teuthology.orchestra.run.smithi060.stdout: 411965 44 -rw-r--r-- 1 ubuntu ubuntu 41160 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/rdma/Infiniband.cc 2024-03-27T21:42:43.956 INFO:teuthology.orchestra.run.smithi060.stdout: 411971 28 -rw-r--r-- 1 ubuntu ubuntu 28620 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/rdma/RDMAStack.cc 2024-03-27T21:42:43.957 INFO:teuthology.orchestra.run.smithi060.stdout: 411967 20 -rw-r--r-- 1 ubuntu ubuntu 18298 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/rdma/RDMAConnectedSocketImpl.cc 2024-03-27T21:42:43.957 INFO:teuthology.orchestra.run.smithi060.stdout: 411969 4 -rw-r--r-- 1 ubuntu ubuntu 3772 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/async/rdma/RDMAIWARPServerSocketImpl.cc 2024-03-27T21:42:43.957 INFO:teuthology.orchestra.run.smithi060.stdout: 411882 8 -rw-r--r-- 1 ubuntu ubuntu 6715 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/DispatchQueue.h 2024-03-27T21:42:43.957 INFO:teuthology.orchestra.run.smithi060.stdout: 411888 24 -rw-r--r-- 1 ubuntu ubuntu 23947 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/Messenger.h 2024-03-27T21:42:43.957 INFO:teuthology.orchestra.run.smithi060.stdout: 411884 28 -rw-r--r-- 1 ubuntu ubuntu 28330 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/Message.cc 2024-03-27T21:42:43.957 INFO:teuthology.orchestra.run.smithi060.stdout: 411879 4 -rw-r--r-- 1 ubuntu ubuntu 550 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/Connection.cc 2024-03-27T21:42:43.957 INFO:teuthology.orchestra.run.smithi060.stdout: 411885 20 -rw-r--r-- 1 ubuntu ubuntu 19531 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/Message.h 2024-03-27T21:42:43.957 INFO:teuthology.orchestra.run.smithi060.stdout: 411974 4 -rw-r--r-- 1 ubuntu ubuntu 1838 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/compressor_registry.h 2024-03-27T21:42:43.957 INFO:teuthology.orchestra.run.smithi060.stdout: 411886 8 -rw-r--r-- 1 ubuntu ubuntu 4134 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/MessageRef.h 2024-03-27T21:42:43.957 INFO:teuthology.orchestra.run.smithi060.stdout: 411883 8 -rw-r--r-- 1 ubuntu ubuntu 8187 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/Dispatcher.h 2024-03-27T21:42:43.958 INFO:teuthology.orchestra.run.smithi060.stdout: 411977 24 -rw-r--r-- 1 ubuntu ubuntu 23101 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/msg_types.h 2024-03-27T21:42:43.958 INFO:teuthology.orchestra.run.smithi060.stdout: 411880 8 -rw-r--r-- 1 ubuntu ubuntu 7256 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/msg/Connection.h 2024-03-27T21:42:43.958 INFO:teuthology.orchestra.run.smithi060.stdout: 409680 4 -rw-r--r-- 1 ubuntu ubuntu 1223 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls_acl.cc 2024-03-27T21:42:43.958 INFO:teuthology.orchestra.run.smithi060.stdout: 411414 4 -rwxr-xr-x 1 ubuntu ubuntu 624 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/loadclass.sh 2024-03-27T21:42:43.958 INFO:teuthology.orchestra.run.smithi060.stdout: 409682 4 -rwxr-xr-x 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cmonctl 2024-03-27T21:42:43.958 INFO:teuthology.orchestra.run.smithi060.stdout: 414721 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_fuse 2024-03-27T21:42:43.959 INFO:teuthology.orchestra.run.smithi060.stdout: 414722 4 -rw-r--r-- 1 ubuntu ubuntu 161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_fuse/CMakeLists.txt 2024-03-27T21:42:43.959 INFO:teuthology.orchestra.run.smithi060.stdout: 414723 24 -rw-r--r-- 1 ubuntu ubuntu 21241 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_fuse/rbd-fuse.cc 2024-03-27T21:42:43.959 INFO:teuthology.orchestra.run.smithi060.stdout: 411980 4 -rwxr-xr-x 1 ubuntu ubuntu 2364 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/multi-dump.sh 2024-03-27T21:42:43.959 INFO:teuthology.orchestra.run.smithi060.stdout: 414718 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/qatlib 2024-03-27T21:42:43.959 INFO:teuthology.orchestra.run.smithi060.stdout: 417123 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/utf8proc 2024-03-27T21:42:43.959 INFO:teuthology.orchestra.run.smithi060.stdout: 414720 4 -rwxr-xr-x 1 ubuntu ubuntu 2070 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd-replay-many 2024-03-27T21:42:43.959 INFO:teuthology.orchestra.run.smithi060.stdout: 410506 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/doc 2024-03-27T21:42:43.959 INFO:teuthology.orchestra.run.smithi060.stdout: 410517 4 -rw-r--r-- 1 ubuntu ubuntu 1806 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/doc/mon-wishlist.txt 2024-03-27T21:42:43.959 INFO:teuthology.orchestra.run.smithi060.stdout: 410511 4 -rw-r--r-- 1 ubuntu ubuntu 296 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/doc/inos.txt 2024-03-27T21:42:43.960 INFO:teuthology.orchestra.run.smithi060.stdout: 410508 16 -rw-r--r-- 1 ubuntu ubuntu 13872 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/doc/caching.txt 2024-03-27T21:42:43.960 INFO:teuthology.orchestra.run.smithi060.stdout: 410518 4 -rw-r--r-- 1 ubuntu ubuntu 1486 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/doc/rgw.txt 2024-03-27T21:42:43.960 INFO:teuthology.orchestra.run.smithi060.stdout: 410509 8 -rw-r--r-- 1 ubuntu ubuntu 5685 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/doc/dynamic-throttle.txt 2024-03-27T21:42:43.960 INFO:teuthology.orchestra.run.smithi060.stdout: 410519 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/doc/rgw 2024-03-27T21:42:43.960 INFO:teuthology.orchestra.run.smithi060.stdout: 410520 12 -rw-r--r-- 1 ubuntu ubuntu 9096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/doc/rgw/multisite-reshard.md 2024-03-27T21:42:43.960 INFO:teuthology.orchestra.run.smithi060.stdout: 410507 4 -rw-r--r-- 1 ubuntu ubuntu 788 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/doc/Commitdir.txt 2024-03-27T21:42:43.960 INFO:teuthology.orchestra.run.smithi060.stdout: 410513 4 -rw-r--r-- 1 ubuntu ubuntu 1537 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/doc/lazy_posix.txt 2024-03-27T21:42:43.960 INFO:teuthology.orchestra.run.smithi060.stdout: 410516 4 -rw-r--r-- 1 ubuntu ubuntu 1373 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/doc/mon-janitorial-queue.txt 2024-03-27T21:42:43.960 INFO:teuthology.orchestra.run.smithi060.stdout: 410515 4 -rw-r--r-- 1 ubuntu ubuntu 99 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/doc/modeline.txt 2024-03-27T21:42:43.960 INFO:teuthology.orchestra.run.smithi060.stdout: 410510 4 -rw-r--r-- 1 ubuntu ubuntu 428 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/doc/header.txt 2024-03-27T21:42:43.961 INFO:teuthology.orchestra.run.smithi060.stdout: 410514 4 -rw-r--r-- 1 ubuntu ubuntu 2997 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/doc/mds_locks.txt 2024-03-27T21:42:43.961 INFO:teuthology.orchestra.run.smithi060.stdout: 410512 4 -rw-r--r-- 1 ubuntu ubuntu 1711 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/doc/killpoints.txt 2024-03-27T21:42:43.961 INFO:teuthology.orchestra.run.smithi060.stdout: 408973 4 -rw-r--r-- 1 ubuntu ubuntu 73 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/README 2024-03-27T21:42:43.961 INFO:teuthology.orchestra.run.smithi060.stdout: 410870 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados 2024-03-27T21:42:43.961 INFO:teuthology.orchestra.run.smithi060.stdout: 410886 84 -rw-r--r-- 1 ubuntu ubuntu 83039 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados/librados_cxx.cc 2024-03-27T21:42:43.961 INFO:teuthology.orchestra.run.smithi060.stdout: 410875 4 -rw-r--r-- 1 ubuntu ubuntu 2482 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados/ListObjectImpl.h 2024-03-27T21:42:43.961 INFO:teuthology.orchestra.run.smithi060.stdout: 410872 4 -rw-r--r-- 1 ubuntu ubuntu 1271 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados/CMakeLists.txt 2024-03-27T21:42:43.961 INFO:teuthology.orchestra.run.smithi060.stdout: 410887 4 -rw-r--r-- 1 ubuntu ubuntu 210 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados/librados_tp.cc 2024-03-27T21:42:43.962 INFO:teuthology.orchestra.run.smithi060.stdout: 410889 4 -rw-r--r-- 1 ubuntu ubuntu 780 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados/librados_util.h 2024-03-27T21:42:43.962 INFO:teuthology.orchestra.run.smithi060.stdout: 410873 60 -rw-r--r-- 1 ubuntu ubuntu 60127 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados/IoCtxImpl.cc 2024-03-27T21:42:43.962 INFO:teuthology.orchestra.run.smithi060.stdout: 410881 4 -rw-r--r-- 1 ubuntu ubuntu 933 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados/RadosXattrIter.h 2024-03-27T21:42:43.962 INFO:teuthology.orchestra.run.smithi060.stdout: 410879 8 -rw-r--r-- 1 ubuntu ubuntu 6555 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados/RadosClient.h 2024-03-27T21:42:43.962 INFO:teuthology.orchestra.run.smithi060.stdout: 410888 4 -rw-r--r-- 1 ubuntu ubuntu 2228 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados/librados_util.cc 2024-03-27T21:42:43.962 INFO:teuthology.orchestra.run.smithi060.stdout: 410890 8 -rw-r--r-- 1 ubuntu ubuntu 4112 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados/snap_set_diff.cc 2024-03-27T21:42:43.962 INFO:teuthology.orchestra.run.smithi060.stdout: 410874 12 -rw-r--r-- 1 ubuntu ubuntu 12196 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados/IoCtxImpl.h 2024-03-27T21:42:43.962 INFO:teuthology.orchestra.run.smithi060.stdout: 410882 4 -rw-r--r-- 1 ubuntu ubuntu 2108 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados/librados.map 2024-03-27T21:42:43.963 INFO:teuthology.orchestra.run.smithi060.stdout: 410878 32 -rw-r--r-- 1 ubuntu ubuntu 28914 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados/RadosClient.cc 2024-03-27T21:42:43.963 INFO:teuthology.orchestra.run.smithi060.stdout: 410884 148 -rw-r--r-- 1 ubuntu ubuntu 150658 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados/librados_c.cc 2024-03-27T21:42:43.963 INFO:teuthology.orchestra.run.smithi060.stdout: 410880 4 -rw-r--r-- 1 ubuntu ubuntu 645 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados/RadosXattrIter.cc 2024-03-27T21:42:43.963 INFO:teuthology.orchestra.run.smithi060.stdout: 410885 4 -rw-r--r-- 1 ubuntu ubuntu 636 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados/librados_c.h 2024-03-27T21:42:43.963 INFO:teuthology.orchestra.run.smithi060.stdout: 410891 4 -rw-r--r-- 1 ubuntu ubuntu 555 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados/snap_set_diff.h 2024-03-27T21:42:43.963 INFO:teuthology.orchestra.run.smithi060.stdout: 410883 12 -rw-r--r-- 1 ubuntu ubuntu 8556 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados/librados_asio.h 2024-03-27T21:42:43.963 INFO:teuthology.orchestra.run.smithi060.stdout: 410876 4 -rw-r--r-- 1 ubuntu ubuntu 677 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados/ObjectOperationImpl.h 2024-03-27T21:42:43.963 INFO:teuthology.orchestra.run.smithi060.stdout: 410877 4 -rw-r--r-- 1 ubuntu ubuntu 2529 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados/PoolAsyncCompletionImpl.h 2024-03-27T21:42:43.963 INFO:teuthology.orchestra.run.smithi060.stdout: 410871 8 -rw-r--r-- 1 ubuntu ubuntu 5013 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados/AioCompletionImpl.h 2024-03-27T21:42:43.964 INFO:teuthology.orchestra.run.smithi060.stdout: 410853 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/kv 2024-03-27T21:42:43.964 INFO:teuthology.orchestra.run.smithi060.stdout: 410854 4 -rw-r--r-- 1 ubuntu ubuntu 278 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/kv/CMakeLists.txt 2024-03-27T21:42:43.964 INFO:teuthology.orchestra.run.smithi060.stdout: 410861 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/kv/rocksdb_cache 2024-03-27T21:42:43.964 INFO:teuthology.orchestra.run.smithi060.stdout: 410863 16 -rw-r--r-- 1 ubuntu ubuntu 12504 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/kv/rocksdb_cache/BinnedLRUCache.h 2024-03-27T21:42:43.964 INFO:teuthology.orchestra.run.smithi060.stdout: 410864 8 -rw-r--r-- 1 ubuntu ubuntu 5696 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/kv/rocksdb_cache/ShardedCache.cc 2024-03-27T21:42:43.964 INFO:teuthology.orchestra.run.smithi060.stdout: 410862 24 -rw-r--r-- 1 ubuntu ubuntu 20729 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/kv/rocksdb_cache/BinnedLRUCache.cc 2024-03-27T21:42:43.964 INFO:teuthology.orchestra.run.smithi060.stdout: 410865 8 -rw-r--r-- 1 ubuntu ubuntu 7596 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/kv/rocksdb_cache/ShardedCache.h 2024-03-27T21:42:43.964 INFO:teuthology.orchestra.run.smithi060.stdout: 410858 4 -rw-r--r-- 1 ubuntu ubuntu 999 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/kv/KeyValueHistogram.h 2024-03-27T21:42:43.965 INFO:teuthology.orchestra.run.smithi060.stdout: 410857 4 -rw-r--r-- 1 ubuntu ubuntu 2392 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/kv/KeyValueHistogram.cc 2024-03-27T21:42:43.965 INFO:teuthology.orchestra.run.smithi060.stdout: 410855 4 -rw-r--r-- 1 ubuntu ubuntu 603 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/kv/KeyValueDB.cc 2024-03-27T21:42:43.965 INFO:teuthology.orchestra.run.smithi060.stdout: 410860 20 -rw-r--r-- 1 ubuntu ubuntu 18549 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/kv/RocksDBStore.h 2024-03-27T21:42:43.965 INFO:teuthology.orchestra.run.smithi060.stdout: 410856 16 -rw-r--r-- 1 ubuntu ubuntu 14552 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/kv/KeyValueDB.h 2024-03-27T21:42:43.965 INFO:teuthology.orchestra.run.smithi060.stdout: 410859 104 -rw-r--r-- 1 ubuntu ubuntu 105857 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/kv/RocksDBStore.cc 2024-03-27T21:42:43.965 INFO:teuthology.orchestra.run.smithi060.stdout: 411876 4 -rwxr-xr-x 1 ubuntu ubuntu 874 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mrun 2024-03-27T21:42:43.965 INFO:teuthology.orchestra.run.smithi060.stdout: 409072 4 -rwxr-xr-x 1 ubuntu ubuntu 231 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-create-keys 2024-03-27T21:42:43.966 INFO:teuthology.orchestra.run.smithi060.stdout: 415289 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script 2024-03-27T21:42:43.966 INFO:teuthology.orchestra.run.smithi060.stdout: 415324 4 -rwxr-xr-x 1 ubuntu ubuntu 3805 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/run_tox.sh 2024-03-27T21:42:43.966 INFO:teuthology.orchestra.run.smithi060.stdout: 415301 4 -rwxr-xr-x 1 ubuntu ubuntu 495 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/check_commands.sh 2024-03-27T21:42:43.966 INFO:teuthology.orchestra.run.smithi060.stdout: 415306 4 -rwxr-xr-x 1 ubuntu ubuntu 2394 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/credits.sh 2024-03-27T21:42:43.966 INFO:teuthology.orchestra.run.smithi060.stdout: 415308 4 -rwxr-xr-x 1 ubuntu ubuntu 690 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/find_dups_in_pg_log.sh 2024-03-27T21:42:43.966 INFO:teuthology.orchestra.run.smithi060.stdout: 415290 4 -rw-r--r-- 1 ubuntu ubuntu 264 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/CMakeLists.txt 2024-03-27T21:42:43.966 INFO:teuthology.orchestra.run.smithi060.stdout: 415321 4 -rwxr-xr-x 1 ubuntu ubuntu 789 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/run-coverity 2024-03-27T21:42:43.966 INFO:teuthology.orchestra.run.smithi060.stdout: 415309 4 -rwxr-xr-x 1 ubuntu ubuntu 668 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/fix_modeline.pl 2024-03-27T21:42:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: 415326 4 -rwxr-xr-x 1 ubuntu ubuntu 1613 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/set_up_stretch_mode.sh 2024-03-27T21:42:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: 415296 8 -rwxr-xr-x 1 ubuntu ubuntu 4141 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/build-integration-branch 2024-03-27T21:42:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: 415292 4 -rwxr-xr-x 1 ubuntu ubuntu 896 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/add_osd.sh 2024-03-27T21:42:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: 415331 4 -rwxr-xr-x 1 ubuntu ubuntu 396 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/unhexdump-C 2024-03-27T21:42:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: 415313 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/kubejacker 2024-03-27T21:42:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: 415316 4 -rwxr-xr-x 1 ubuntu ubuntu 2665 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/kubejacker/kubejacker.sh 2024-03-27T21:42:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: 415315 4 -rw-r--r-- 1 ubuntu ubuntu 452 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/kubejacker/README.rst 2024-03-27T21:42:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: 415314 4 -rw-r--r-- 1 ubuntu ubuntu 520 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/kubejacker/Dockerfile 2024-03-27T21:42:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: 415297 64 -rwxr-xr-x 1 ubuntu ubuntu 63497 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/ceph-backport.sh 2024-03-27T21:42:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: 415311 4 -rwxr-xr-x 1 ubuntu ubuntu 164 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/kcon_all.sh 2024-03-27T21:42:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: 415307 4 -rwxr-xr-x 1 ubuntu ubuntu 318 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/extend_stretch_cluster.sh 2024-03-27T21:42:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: 415327 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/smr_benchmark 2024-03-27T21:42:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: 415329 4 -rwxr-xr-x 1 ubuntu ubuntu 1789 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/smr_benchmark/linearSMRCopy.sh 2024-03-27T21:42:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: 415328 4 -rwxr-xr-x 1 ubuntu ubuntu 1898 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/smr_benchmark/linearCopy.sh 2024-03-27T21:42:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: 415312 4 -rwxr-xr-x 1 ubuntu ubuntu 435 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/kcon_most.sh 2024-03-27T21:42:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: 415330 4 -rwxr-xr-x 1 ubuntu ubuntu 63 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/strip_trailing_whitespace.sh 2024-03-27T21:42:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: 415303 8 -rwxr-xr-x 1 ubuntu ubuntu 6172 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/cpatch 2024-03-27T21:42:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: 415298 8 -rwxr-xr-x 1 ubuntu ubuntu 6387 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/ceph-debug-docker.sh 2024-03-27T21:42:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: 415320 8 -rwxr-xr-x 1 ubuntu ubuntu 4207 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/run-cbt.sh 2024-03-27T21:42:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: 415318 16 -rwxr-xr-x 1 ubuntu ubuntu 15763 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/ptl-tool.py 2024-03-27T21:42:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: 415294 32 -rwxr-xr-x 1 ubuntu ubuntu 29767 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/backport-resolve-issue 2024-03-27T21:42:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: 415291 4 -rwxr-xr-x 1 ubuntu ubuntu 446 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/add_header.pl 2024-03-27T21:42:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: 415317 4 -rwxr-xr-x 1 ubuntu ubuntu 3031 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/lib-build.sh 2024-03-27T21:42:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: 415295 4 -rwxr-xr-x 1 ubuntu ubuntu 445 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/bdev_grep.pl 2024-03-27T21:42:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: 415323 4 -rwxr-xr-x 1 ubuntu ubuntu 2880 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/run_mypy.sh 2024-03-27T21:42:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: 415300 4 -rw-r--r-- 1 ubuntu ubuntu 3710 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/ceph_dump_log.py 2024-03-27T21:42:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: 415310 4 -rwxr-xr-x 1 ubuntu ubuntu 2323 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/gen-corpus.sh 2024-03-27T21:42:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: 415325 8 -rwxr-xr-x 1 ubuntu ubuntu 6622 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/run_uml.sh 2024-03-27T21:42:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: 415322 8 -rwxr-xr-x 1 ubuntu ubuntu 5333 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/run-make.sh 2024-03-27T21:42:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: 415319 4 -rw-r--r-- 1 ubuntu ubuntu 24 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/requirements.backport-create-issue.txt 2024-03-27T21:42:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: 415302 4 -rw-r--r-- 1 ubuntu ubuntu 1003 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/cmake_uninstall.cmake.in 2024-03-27T21:42:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: 415304 24 -rwxr-xr-x 1 ubuntu ubuntu 22900 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/cpatch.py 2024-03-27T21:42:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: 415299 16 -rwxr-xr-x 1 ubuntu ubuntu 15096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/ceph-release-notes 2024-03-27T21:42:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: 415293 20 -rwxr-xr-x 1 ubuntu ubuntu 17007 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/backport-create-issue 2024-03-27T21:42:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: 415305 4 -rwxr-xr-x 1 ubuntu ubuntu 223 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/script/crash_bdev.sh 2024-03-27T21:42:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: 411998 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os 2024-03-27T21:42:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: 412007 4 -rw-r--r-- 1 ubuntu ubuntu 1526 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/SequencerPosition.h 2024-03-27T21:42:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: 412006 28 -rw-r--r-- 1 ubuntu ubuntu 25191 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/ObjectStore.h 2024-03-27T21:42:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: 412002 32 -rw-r--r-- 1 ubuntu ubuntu 29220 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/FuseStore.cc 2024-03-27T21:42:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: 412010 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore 2024-03-27T21:42:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: 412032 4 -rw-r--r-- 1 ubuntu ubuntu 2001 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/StupidAllocator.h 2024-03-27T21:42:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: 412014 12 -rw-r--r-- 1 ubuntu ubuntu 8339 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/AvlAllocator.h 2024-03-27T21:42:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: 412015 4 -rw-r--r-- 1 ubuntu ubuntu 3397 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/BitmapAllocator.cc 2024-03-27T21:42:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: 412038 44 -rw-r--r-- 1 ubuntu ubuntu 40975 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/bluestore_types.h 2024-03-27T21:42:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: 412012 4 -rw-r--r-- 1 ubuntu ubuntu 3806 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/Allocator.h 2024-03-27T21:42:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: 412031 12 -rw-r--r-- 1 ubuntu ubuntu 9644 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/StupidAllocator.cc 2024-03-27T21:42:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: 412019 152 -rw-r--r-- 1 ubuntu ubuntu 153986 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/BlueFS.cc 2024-03-27T21:42:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: 412033 8 -rw-r--r-- 1 ubuntu ubuntu 7692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/bluefs_types.cc 2024-03-27T21:42:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: 412022 8 -rw-r--r-- 1 ubuntu ubuntu 6431 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/BlueRocksEnv.h 2024-03-27T21:42:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: 412029 8 -rw-r--r-- 1 ubuntu ubuntu 5593 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/HybridAllocator.cc 2024-03-27T21:42:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: 412024 140 -rw-r--r-- 1 ubuntu ubuntu 140284 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/BlueStore.h 2024-03-27T21:42:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: 412035 4 -rw-r--r-- 1 ubuntu ubuntu 2198 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/bluestore_common.h 2024-03-27T21:42:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: 412018 4 -rw-r--r-- 1 ubuntu ubuntu 3122 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/BitmapFreelistManager.h 2024-03-27T21:42:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: 412021 20 -rw-r--r-- 1 ubuntu ubuntu 17782 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/BlueRocksEnv.cc 2024-03-27T21:42:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: 412041 8 -rw-r--r-- 1 ubuntu ubuntu 7889 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/simple_bitmap.cc 2024-03-27T21:42:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: 412040 24 -rw-r--r-- 1 ubuntu ubuntu 23826 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/fastbmap_allocator_impl.h 2024-03-27T21:42:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: 412017 20 -rw-r--r-- 1 ubuntu ubuntu 18100 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/BitmapFreelistManager.cc 2024-03-27T21:42:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: 412036 36 -rw-r--r-- 1 ubuntu ubuntu 35085 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/bluestore_tool.cc 2024-03-27T21:42:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: 412030 4 -rw-r--r-- 1 ubuntu ubuntu 1427 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/HybridAllocator.h 2024-03-27T21:42:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: 412011 12 -rw-r--r-- 1 ubuntu ubuntu 10357 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/Allocator.cc 2024-03-27T21:42:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: 412023 596 -rw-r--r-- 1 ubuntu ubuntu 609217 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/BlueStore.cc 2024-03-27T21:42:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: 412042 8 -rw-r--r-- 1 ubuntu ubuntu 4928 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/simple_bitmap.h 2024-03-27T21:42:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: 412039 24 -rw-r--r-- 1 ubuntu ubuntu 21076 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/fastbmap_allocator_impl.cc 2024-03-27T21:42:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: 412016 4 -rw-r--r-- 1 ubuntu ubuntu 1428 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/BitmapAllocator.h 2024-03-27T21:42:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: 412034 12 -rw-r--r-- 1 ubuntu ubuntu 10011 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/bluefs_types.h 2024-03-27T21:42:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: 412025 16 -rw-r--r-- 1 ubuntu ubuntu 13970 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/BtreeAllocator.cc 2024-03-27T21:42:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: 412026 8 -rw-r--r-- 1 ubuntu ubuntu 5779 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/BtreeAllocator.h 2024-03-27T21:42:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: 412020 28 -rw-r--r-- 1 ubuntu ubuntu 27089 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/BlueFS.h 2024-03-27T21:42:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: 412027 4 -rw-r--r-- 1 ubuntu ubuntu 1035 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/FreelistManager.cc 2024-03-27T21:42:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: 412037 40 -rw-r--r-- 1 ubuntu ubuntu 37631 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/bluestore_types.cc 2024-03-27T21:42:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: 412028 4 -rw-r--r-- 1 ubuntu ubuntu 1803 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/FreelistManager.h 2024-03-27T21:42:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: 412013 16 -rw-r--r-- 1 ubuntu ubuntu 14323 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/bluestore/AvlAllocator.cc 2024-03-27T21:42:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: 412043 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/fs 2024-03-27T21:42:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: 412048 4 -rw-r--r-- 1 ubuntu ubuntu 1942 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/fs/ZFS.cc 2024-03-27T21:42:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: 412047 4 -rw-r--r-- 1 ubuntu ubuntu 677 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/fs/XFS.h 2024-03-27T21:42:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: 412044 8 -rw-r--r-- 1 ubuntu ubuntu 4160 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/fs/FS.cc 2024-03-27T21:42:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: 412050 8 -rw-r--r-- 1 ubuntu ubuntu 5542 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/fs/btrfs_ioctl.h 2024-03-27T21:42:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: 412046 4 -rw-r--r-- 1 ubuntu ubuntu 1125 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/fs/XFS.cc 2024-03-27T21:42:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: 412049 4 -rw-r--r-- 1 ubuntu ubuntu 1039 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/fs/ZFS.h 2024-03-27T21:42:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: 412045 4 -rw-r--r-- 1 ubuntu ubuntu 1130 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/fs/FS.h 2024-03-27T21:42:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: 411999 4 -rw-r--r-- 1 ubuntu ubuntu 2143 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/CMakeLists.txt 2024-03-27T21:42:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: 412056 4 -rw-r--r-- 1 ubuntu ubuntu 1461 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/kv.h 2024-03-27T21:42:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: 412001 20 -rw-r--r-- 1 ubuntu ubuntu 16954 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/DBObjectMap.h 2024-03-27T21:42:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: 412009 44 -rw-r--r-- 1 ubuntu ubuntu 42229 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/Transaction.h 2024-03-27T21:42:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: 412008 16 -rw-r--r-- 1 ubuntu ubuntu 15859 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/Transaction.cc 2024-03-27T21:42:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: 412051 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/kstore 2024-03-27T21:42:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: 412053 20 -rw-r--r-- 1 ubuntu ubuntu 19855 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/kstore/KStore.h 2024-03-27T21:42:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: 412052 88 -rw-r--r-- 1 ubuntu ubuntu 86404 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/kstore/KStore.cc 2024-03-27T21:42:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: 412055 4 -rw-r--r-- 1 ubuntu ubuntu 1969 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/kstore/kstore_types.h 2024-03-27T21:42:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: 412054 4 -rw-r--r-- 1 ubuntu ubuntu 2522 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/kstore/kstore_types.cc 2024-03-27T21:42:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: 412004 8 -rw-r--r-- 1 ubuntu ubuntu 6174 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/ObjectMap.h 2024-03-27T21:42:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: 412005 4 -rw-r--r-- 1 ubuntu ubuntu 2619 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/ObjectStore.cc 2024-03-27T21:42:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: 412057 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/memstore 2024-03-27T21:42:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: 412060 8 -rw-r--r-- 1 ubuntu ubuntu 7090 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/memstore/PageSet.h 2024-03-27T21:42:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: 412059 16 -rw-r--r-- 1 ubuntu ubuntu 13614 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/memstore/MemStore.h 2024-03-27T21:42:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: 412058 48 -rw-r--r-- 1 ubuntu ubuntu 48372 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/memstore/MemStore.cc 2024-03-27T21:42:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: 412000 40 -rw-r--r-- 1 ubuntu ubuntu 37442 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/DBObjectMap.cc 2024-03-27T21:42:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: 412003 4 -rw-r--r-- 1 ubuntu ubuntu 1028 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/os/FuseStore.h 2024-03-27T21:42:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: 410830 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/json_spirit 2024-03-27T21:42:43.979 INFO:teuthology.orchestra.run.smithi060.stdout: 410835 4 -rw-r--r-- 1 ubuntu ubuntu 2571 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/json_spirit/json_spirit_reader.h 2024-03-27T21:42:43.979 INFO:teuthology.orchestra.run.smithi060.stdout: 410831 4 -rw-r--r-- 1 ubuntu ubuntu 129 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/json_spirit/CMakeLists.txt 2024-03-27T21:42:43.979 INFO:teuthology.orchestra.run.smithi060.stdout: 410842 4 -rw-r--r-- 1 ubuntu ubuntu 2364 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/json_spirit/json_spirit_writer.h 2024-03-27T21:42:43.979 INFO:teuthology.orchestra.run.smithi060.stdout: 410833 4 -rw-r--r-- 1 ubuntu ubuntu 1515 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/json_spirit/json_spirit_error_position.h 2024-03-27T21:42:43.979 INFO:teuthology.orchestra.run.smithi060.stdout: 410841 4 -rw-r--r-- 1 ubuntu ubuntu 2860 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/json_spirit/json_spirit_writer.cpp 2024-03-27T21:42:43.979 INFO:teuthology.orchestra.run.smithi060.stdout: 410837 4 -rw-r--r-- 1 ubuntu ubuntu 1794 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/json_spirit/json_spirit_stream_reader.h 2024-03-27T21:42:43.979 INFO:teuthology.orchestra.run.smithi060.stdout: 410839 4 -rw-r--r-- 1 ubuntu ubuntu 200 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/json_spirit/json_spirit_value.cpp 2024-03-27T21:42:43.979 INFO:teuthology.orchestra.run.smithi060.stdout: 410836 24 -rw-r--r-- 1 ubuntu ubuntu 22044 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/json_spirit/json_spirit_reader_template.h 2024-03-27T21:42:43.980 INFO:teuthology.orchestra.run.smithi060.stdout: 410844 12 -rw-r--r-- 1 ubuntu ubuntu 11217 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/json_spirit/json_spirit_writer_template.h 2024-03-27T21:42:43.980 INFO:teuthology.orchestra.run.smithi060.stdout: 410843 4 -rw-r--r-- 1 ubuntu ubuntu 1371 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/json_spirit/json_spirit_writer_options.h 2024-03-27T21:42:43.980 INFO:teuthology.orchestra.run.smithi060.stdout: 410840 16 -rw-r--r-- 1 ubuntu ubuntu 16315 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/json_spirit/json_spirit_value.h 2024-03-27T21:42:43.980 INFO:teuthology.orchestra.run.smithi060.stdout: 410832 4 -rw-r--r-- 1 ubuntu ubuntu 413 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/json_spirit/json_spirit.h 2024-03-27T21:42:43.980 INFO:teuthology.orchestra.run.smithi060.stdout: 410838 4 -rw-r--r-- 1 ubuntu ubuntu 1707 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/json_spirit/json_spirit_utils.h 2024-03-27T21:42:43.980 INFO:teuthology.orchestra.run.smithi060.stdout: 410834 8 -rw-r--r-- 1 ubuntu ubuntu 4106 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/json_spirit/json_spirit_reader.cpp 2024-03-27T21:42:43.980 INFO:teuthology.orchestra.run.smithi060.stdout: 415338 12 drwxr-xr-x 64 ubuntu ubuntu 12288 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test 2024-03-27T21:42:43.980 INFO:teuthology.orchestra.run.smithi060.stdout: 416650 4 -rw-r--r-- 1 ubuntu ubuntu 1144 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_rewrite_latency.cc 2024-03-27T21:42:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: 416621 4 -rw-r--r-- 1 ubuntu ubuntu 3766 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_admin_socket_output.cc 2024-03-27T21:42:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: 415387 12 -rw-r--r-- 1 ubuntu ubuntu 9505 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/ceph_crypto.cc 2024-03-27T21:42:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: 416634 4 -rw-r--r-- 1 ubuntu ubuntu 1433 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_features.cc 2024-03-27T21:42:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: 415814 16 -rw-r--r-- 1 ubuntu ubuntu 12434 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/confutils.cc 2024-03-27T21:42:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: 416679 4 -rw-r--r-- 1 ubuntu ubuntu 2326 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/utf8.cc 2024-03-27T21:42:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: 416681 8 -rw-r--r-- 1 ubuntu ubuntu 5204 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/xattr_bench.cc 2024-03-27T21:42:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: 415938 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/exporter 2024-03-27T21:42:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: 415939 4 -rw-r--r-- 1 ubuntu ubuntu 274 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/exporter/CMakeLists.txt 2024-03-27T21:42:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: 415940 48 -rw-r--r-- 1 ubuntu ubuntu 45082 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/exporter/test_exporter.cc 2024-03-27T21:42:43.982 INFO:teuthology.orchestra.run.smithi060.stdout: 415587 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_numops 2024-03-27T21:42:43.982 INFO:teuthology.orchestra.run.smithi060.stdout: 415588 4 -rw-r--r-- 1 ubuntu ubuntu 340 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_numops/CMakeLists.txt 2024-03-27T21:42:43.982 INFO:teuthology.orchestra.run.smithi060.stdout: 415589 12 -rw-r--r-- 1 ubuntu ubuntu 9859 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_numops/test_cls_numops.cc 2024-03-27T21:42:43.982 INFO:teuthology.orchestra.run.smithi060.stdout: 415378 4 -rw-r--r-- 1 ubuntu ubuntu 360 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/buildtest_skeleton.cc 2024-03-27T21:42:43.982 INFO:teuthology.orchestra.run.smithi060.stdout: 416437 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/pybind 2024-03-27T21:42:43.982 INFO:teuthology.orchestra.run.smithi060.stdout: 416438 4 -rw-r--r-- 1 ubuntu ubuntu 218 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/pybind/CMakeLists.txt 2024-03-27T21:42:43.982 INFO:teuthology.orchestra.run.smithi060.stdout: 416443 36 -rw-r--r-- 1 ubuntu ubuntu 34339 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_cephfs.py 2024-03-27T21:42:43.982 INFO:teuthology.orchestra.run.smithi060.stdout: 416441 68 -rwxr-xr-x 1 ubuntu ubuntu 65789 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_ceph_argparse.py 2024-03-27T21:42:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: 416446 4 -rw-r--r-- 1 ubuntu ubuntu 3577 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rgwfs.py 2024-03-27T21:42:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: 416442 4 -rwxr-xr-x 1 ubuntu ubuntu 1401 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_ceph_daemon.py 2024-03-27T21:42:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: 416444 60 -rw-r--r-- 1 ubuntu ubuntu 57769 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rados.py 2024-03-27T21:42:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: 416445 104 -rw-r--r-- 1 ubuntu ubuntu 105150 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rbd.py 2024-03-27T21:42:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: 416439 4 -rw-r--r-- 1 ubuntu ubuntu 615 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/pybind/assertions.py 2024-03-27T21:42:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: 416440 4 -rw-r--r-- 1 ubuntu ubuntu 98 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/pybind/pytest.ini 2024-03-27T21:42:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: 416322 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/msgr 2024-03-27T21:42:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: 416331 8 -rw-r--r-- 1 ubuntu ubuntu 5005 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/msgr/test_userspace_event.cc 2024-03-27T21:42:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: 416326 12 -rw-r--r-- 1 ubuntu ubuntu 8636 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/msgr/test_async_driver.cc 2024-03-27T21:42:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: 416325 8 -rw-r--r-- 1 ubuntu ubuntu 5740 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/msgr/perf_msgr_server.cc 2024-03-27T21:42:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: 416328 4 -rw-r--r-- 1 ubuntu ubuntu 3565 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/msgr/test_comp_registry.cc 2024-03-27T21:42:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: 416323 4 -rw-r--r-- 1 ubuntu ubuntu 1786 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/msgr/CMakeLists.txt 2024-03-27T21:42:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: 416327 32 -rw-r--r-- 1 ubuntu ubuntu 31246 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/msgr/test_async_networkstack.cc 2024-03-27T21:42:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: 416324 8 -rw-r--r-- 1 ubuntu ubuntu 7108 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/msgr/perf_msgr_client.cc 2024-03-27T21:42:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: 416330 80 -rw-r--r-- 1 ubuntu ubuntu 79949 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/msgr/test_msgr.cc 2024-03-27T21:42:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: 416329 20 -rw-r--r-- 1 ubuntu ubuntu 19530 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/msgr/test_frames_v2.cc 2024-03-27T21:42:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: 415359 4 -rw-r--r-- 1 ubuntu ubuntu 2671 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/base64.cc 2024-03-27T21:42:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: 416282 8 -rw-r--r-- 1 ubuntu ubuntu 7245 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librgw_file.cc 2024-03-27T21:42:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: 415358 4 -rw-r--r-- 1 ubuntu ubuntu 817 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/barclass.cc 2024-03-27T21:42:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: 416296 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/memuse 2024-03-27T21:42:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: 416299 4 -rwxr-xr-x 1 ubuntu ubuntu 185 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/memuse/test_written_pool_memuse.sh 2024-03-27T21:42:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: 416298 4 -rwxr-xr-x 1 ubuntu ubuntu 465 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/memuse/test_pool_memuse_tcmalloc.sh 2024-03-27T21:42:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: 416300 4 -rwxr-xr-x 1 ubuntu ubuntu 969 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/memuse/test_written_pool_memuse_tcmalloc.sh 2024-03-27T21:42:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: 416297 4 -rwxr-xr-x 1 ubuntu ubuntu 371 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/memuse/test_pool_memuse.sh 2024-03-27T21:42:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: 416301 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mgr 2024-03-27T21:42:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: 416303 4 -rwxr-xr-x 1 ubuntu ubuntu 2492 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mgr/mgr-dashboard-smoke.sh 2024-03-27T21:42:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: 416304 12 -rw-r--r-- 1 ubuntu ubuntu 9787 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mgr/test_mgrcap.cc 2024-03-27T21:42:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: 416302 4 -rw-r--r-- 1 ubuntu ubuntu 846 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mgr/CMakeLists.txt 2024-03-27T21:42:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: 416305 4 -rw-r--r-- 1 ubuntu ubuntu 1324 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mgr/test_ttlcache.cc 2024-03-27T21:42:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: 416662 4 -rw-r--r-- 1 ubuntu ubuntu 2036 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_texttable.cc 2024-03-27T21:42:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: 415566 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_cas 2024-03-27T21:42:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: 415567 4 -rw-r--r-- 1 ubuntu ubuntu 334 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_cas/CMakeLists.txt 2024-03-27T21:42:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: 415568 12 -rw-r--r-- 1 ubuntu ubuntu 8706 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_cas/test_cls_cas.cc 2024-03-27T21:42:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: 415946 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fio 2024-03-27T21:42:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: 415950 4 -rw-r--r-- 1 ubuntu ubuntu 1469 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fio/ceph-bluestore.fio 2024-03-27T21:42:43.987 INFO:teuthology.orchestra.run.smithi060.stdout: 415948 4 -rw-r--r-- 1 ubuntu ubuntu 3284 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fio/README.md 2024-03-27T21:42:43.987 INFO:teuthology.orchestra.run.smithi060.stdout: 415953 4 -rw-r--r-- 1 ubuntu ubuntu 549 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fio/ceph-librgw.fio 2024-03-27T21:42:43.987 INFO:teuthology.orchestra.run.smithi060.stdout: 415947 4 -rw-r--r-- 1 ubuntu ubuntu 652 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fio/CMakeLists.txt 2024-03-27T21:42:43.987 INFO:teuthology.orchestra.run.smithi060.stdout: 415954 4 -rw-r--r-- 1 ubuntu ubuntu 389 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fio/ceph-memstore.conf 2024-03-27T21:42:43.987 INFO:teuthology.orchestra.run.smithi060.stdout: 415956 4 -rw-r--r-- 1 ubuntu ubuntu 110 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fio/ceph-messenger.conf 2024-03-27T21:42:43.987 INFO:teuthology.orchestra.run.smithi060.stdout: 415951 4 -rw-r--r-- 1 ubuntu ubuntu 554 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fio/ceph-filestore.conf 2024-03-27T21:42:43.987 INFO:teuthology.orchestra.run.smithi060.stdout: 415949 4 -rw-r--r-- 1 ubuntu ubuntu 451 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fio/ceph-bluestore.conf 2024-03-27T21:42:43.987 INFO:teuthology.orchestra.run.smithi060.stdout: 415955 4 -rw-r--r-- 1 ubuntu ubuntu 361 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fio/ceph-memstore.fio 2024-03-27T21:42:43.988 INFO:teuthology.orchestra.run.smithi060.stdout: 415957 4 -rw-r--r-- 1 ubuntu ubuntu 339 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fio/ceph-messenger.fio 2024-03-27T21:42:43.988 INFO:teuthology.orchestra.run.smithi060.stdout: 415961 4 -rw-r--r-- 1 ubuntu ubuntu 2420 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fio/ring_buffer.h 2024-03-27T21:42:43.988 INFO:teuthology.orchestra.run.smithi060.stdout: 415959 32 -rw-r--r-- 1 ubuntu ubuntu 29440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fio/fio_ceph_objectstore.cc 2024-03-27T21:42:43.988 INFO:teuthology.orchestra.run.smithi060.stdout: 415958 20 -rw-r--r-- 1 ubuntu ubuntu 18125 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fio/fio_ceph_messenger.cc 2024-03-27T21:42:43.988 INFO:teuthology.orchestra.run.smithi060.stdout: 415960 16 -rw-r--r-- 1 ubuntu ubuntu 15229 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fio/fio_librgw.cc 2024-03-27T21:42:43.988 INFO:teuthology.orchestra.run.smithi060.stdout: 415952 4 -rw-r--r-- 1 ubuntu ubuntu 365 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fio/ceph-filestore.fio 2024-03-27T21:42:43.988 INFO:teuthology.orchestra.run.smithi060.stdout: 416286 16 -rw-r--r-- 1 ubuntu ubuntu 12725 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librgw_file_marker.cc 2024-03-27T21:42:43.988 INFO:teuthology.orchestra.run.smithi060.stdout: 416624 12 -rw-r--r-- 1 ubuntu ubuntu 10284 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_auth.cc 2024-03-27T21:42:43.988 INFO:teuthology.orchestra.run.smithi060.stdout: 416436 32 -rw-r--r-- 1 ubuntu ubuntu 29610 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/perf_local.cc 2024-03-27T21:42:43.989 INFO:teuthology.orchestra.run.smithi060.stdout: 416653 4 -rw-r--r-- 1 ubuntu ubuntu 2885 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_rgw_ldap.cc 2024-03-27T21:42:43.989 INFO:teuthology.orchestra.run.smithi060.stdout: 415937 8 -rw-r--r-- 1 ubuntu ubuntu 4786 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/escape.cc 2024-03-27T21:42:43.989 INFO:teuthology.orchestra.run.smithi060.stdout: 415349 8 -rw-r--r-- 1 ubuntu ubuntu 8077 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/TestTimers.cc 2024-03-27T21:42:43.989 INFO:teuthology.orchestra.run.smithi060.stdout: 415880 12 -rw-r--r-- 1 ubuntu ubuntu 12034 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/daemon_config.cc 2024-03-27T21:42:43.989 INFO:teuthology.orchestra.run.smithi060.stdout: 415356 4 -rw-r--r-- 1 ubuntu ubuntu 3504 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/admin_socket_output_tests.cc 2024-03-27T21:42:43.989 INFO:teuthology.orchestra.run.smithi060.stdout: 416355 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore 2024-03-27T21:42:43.989 INFO:teuthology.orchestra.run.smithi060.stdout: 416378 4 -rw-r--r-- 1 ubuntu ubuntu 1028 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/store_test_fixture.h 2024-03-27T21:42:43.989 INFO:teuthology.orchestra.run.smithi060.stdout: 416375 4 -rwxr-xr-x 1 ubuntu ubuntu 2479 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/run_test_deferred.sh 2024-03-27T21:42:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: 416377 4 -rw-r--r-- 1 ubuntu ubuntu 3519 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/store_test_fixture.cc 2024-03-27T21:42:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: 416367 28 -rw-r--r-- 1 ubuntu ubuntu 24785 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/allocator_replay_test.cc 2024-03-27T21:42:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: 416376 324 -rw-r--r-- 1 ubuntu ubuntu 329085 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/store_test.cc 2024-03-27T21:42:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: 416366 4 -rw-r--r-- 1 ubuntu ubuntu 3007 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/TestRocksdbOptionParse.cc 2024-03-27T21:42:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: 416358 20 -rw-r--r-- 1 ubuntu ubuntu 18859 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/Allocator_test.cc 2024-03-27T21:42:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: 416385 8 -rw-r--r-- 1 ubuntu ubuntu 4650 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/test_transaction.cc 2024-03-27T21:42:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: 416359 8 -rw-r--r-- 1 ubuntu ubuntu 4286 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/CMakeLists.txt 2024-03-27T21:42:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: 416372 12 -rwxr-xr-x 1 ubuntu ubuntu 8833 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/hybrid_allocator_test.cc 2024-03-27T21:42:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: 416380 52 -rw-r--r-- 1 ubuntu ubuntu 49969 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/test_bluefs.cc 2024-03-27T21:42:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: 416363 12 -rw-r--r-- 1 ubuntu ubuntu 9053 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/ObjectStoreTransactionBenchmark.cc 2024-03-27T21:42:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: 416373 8 -rwxr-xr-x 1 ubuntu ubuntu 7659 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/run_seed_to.sh 2024-03-27T21:42:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: 416357 12 -rw-r--r-- 1 ubuntu ubuntu 9803 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/Allocator_bench.cc 2024-03-27T21:42:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: 416356 16 -rwxr-xr-x 1 ubuntu ubuntu 13776 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/Allocator_aging_fragmentation.cc 2024-03-27T21:42:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: 416383 36 -rw-r--r-- 1 ubuntu ubuntu 36716 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/test_kv.cc 2024-03-27T21:42:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: 416368 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/allocsim 2024-03-27T21:42:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: 416369 12 -rw-r--r-- 1 ubuntu ubuntu 11520 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/allocsim/ops_scraper.py 2024-03-27T21:42:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: 416370 16 -rwxr-xr-x 1 ubuntu ubuntu 13425 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/allocsim/variable_load.py 2024-03-27T21:42:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: 416361 32 -rw-r--r-- 1 ubuntu ubuntu 31188 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/ObjectStoreImitator.cc 2024-03-27T21:42:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: 416364 8 -rw-r--r-- 1 ubuntu ubuntu 7909 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/TestObjectStoreState.cc 2024-03-27T21:42:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: 416360 16 -rw-r--r-- 1 ubuntu ubuntu 12368 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/Fragmentation_simulator.cc 2024-03-27T21:42:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: 416362 16 -rw-r--r-- 1 ubuntu ubuntu 14029 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/ObjectStoreImitator.h 2024-03-27T21:42:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: 416365 8 -rw-r--r-- 1 ubuntu ubuntu 4213 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/TestObjectStoreState.h 2024-03-27T21:42:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: 416384 8 -rw-r--r-- 1 ubuntu ubuntu 5408 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/test_memstore_clone.cc 2024-03-27T21:42:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: 416381 100 -rw-r--r-- 1 ubuntu ubuntu 99986 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/test_bluestore_types.cc 2024-03-27T21:42:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: 416382 8 -rw-r--r-- 1 ubuntu ubuntu 4206 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/test_deferred.cc 2024-03-27T21:42:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: 416371 40 -rw-r--r-- 1 ubuntu ubuntu 37616 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/fastbmap_allocator_test.cc 2024-03-27T21:42:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: 416374 4 -rwxr-xr-x 1 ubuntu ubuntu 314 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/run_seed_to_range.sh 2024-03-27T21:42:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: 416379 4 -rwxr-xr-x 1 ubuntu ubuntu 2725 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore/test_bdev.cc 2024-03-27T21:42:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: 415870 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crush 2024-03-27T21:42:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: 415874 4 -rw-r--r-- 1 ubuntu ubuntu 1152 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crush/crush-choose-args-expected-one-more-3.txt 2024-03-27T21:42:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: 415872 44 -rw-r--r-- 1 ubuntu ubuntu 42266 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crush/CrushWrapper.cc 2024-03-27T21:42:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: 415871 4 -rw-r--r-- 1 ubuntu ubuntu 413 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crush/CMakeLists.txt 2024-03-27T21:42:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: 415876 4 -rwxr-xr-x 1 ubuntu ubuntu 1507 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crush/crush_weights.sh 2024-03-27T21:42:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: 415873 4 -rw-r--r-- 1 ubuntu ubuntu 1152 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crush/crush-choose-args-expected-one-more-0.txt 2024-03-27T21:42:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: 415875 44 -rw-r--r-- 1 ubuntu ubuntu 44543 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crush/crush.cc 2024-03-27T21:42:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: 415383 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/ceph-erasure-code-tool 2024-03-27T21:42:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: 415384 4 -rwxr-xr-x 1 ubuntu ubuntu 1327 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/ceph-erasure-code-tool/test_ceph-erasure-code-tool.sh 2024-03-27T21:42:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: 415386 8 -rw-r--r-- 1 ubuntu ubuntu 5161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/ceph_compatset.cc 2024-03-27T21:42:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: 415351 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/admin_socket 2024-03-27T21:42:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: 415353 4 -rw-r--r-- 1 ubuntu ubuntu 787 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/admin_socket/osd_requests 2024-03-27T21:42:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: 415352 4 -rwxr-xr-x 1 ubuntu ubuntu 1639 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/admin_socket/objecter_requests 2024-03-27T21:42:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: 415964 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fs 2024-03-27T21:42:43.995 INFO:teuthology.orchestra.run.smithi060.stdout: 415966 8 -rw-r--r-- 1 ubuntu ubuntu 5344 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fs/mds_types.cc 2024-03-27T21:42:43.995 INFO:teuthology.orchestra.run.smithi060.stdout: 415965 4 -rw-r--r-- 1 ubuntu ubuntu 640 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fs/CMakeLists.txt 2024-03-27T21:42:43.995 INFO:teuthology.orchestra.run.smithi060.stdout: 415967 4 -rw-r--r-- 1 ubuntu ubuntu 1785 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fs/test_ino_release_cb.cc 2024-03-27T21:42:43.995 INFO:teuthology.orchestra.run.smithi060.stdout: 415968 4 -rw-r--r-- 1 ubuntu ubuntu 1794 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fs/test_trim_caps.cc 2024-03-27T21:42:43.995 INFO:teuthology.orchestra.run.smithi060.stdout: 416665 4 -rw-r--r-- 1 ubuntu ubuntu 1839 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_utime.cc 2024-03-27T21:42:43.995 INFO:teuthology.orchestra.run.smithi060.stdout: 416633 16 -rw-r--r-- 1 ubuntu ubuntu 16370 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_denc.cc 2024-03-27T21:42:43.995 INFO:teuthology.orchestra.run.smithi060.stdout: 415340 28 -rw-r--r-- 1 ubuntu ubuntu 25487 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/CMakeLists.txt 2024-03-27T21:42:43.995 INFO:teuthology.orchestra.run.smithi060.stdout: 415379 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/centos-8 2024-03-27T21:42:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: 415382 0 lrwxrwxrwx 1 ubuntu ubuntu 24 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/centos-8/install-deps.sh -> ../../../install-deps.sh 2024-03-27T21:42:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: 415380 4 -rw-r--r-- 1 ubuntu ubuntu 1279 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/centos-8/Dockerfile.in 2024-03-27T21:42:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: 415381 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/centos-8/ceph.spec.in -> ../../../ceph.spec.in 2024-03-27T21:42:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: 416646 4 -rwxr-xr-x 1 ubuntu ubuntu 898 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_pools.sh 2024-03-27T21:42:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: 416663 4 -rw-r--r-- 1 ubuntu ubuntu 2052 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_trans.cc 2024-03-27T21:42:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: 416669 4 -rw-r--r-- 1 ubuntu ubuntu 1032 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/testclass.cc 2024-03-27T21:42:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: 416651 56 -rw-r--r-- 1 ubuntu ubuntu 53788 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_rgw_admin_log.cc 2024-03-27T21:42:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: 416635 4 -rw-r--r-- 1 ubuntu ubuntu 2018 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_get_blkdev_props.cc 2024-03-27T21:42:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: 415348 4 -rw-r--r-- 1 ubuntu ubuntu 2911 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/TestSignalHandlers.cc 2024-03-27T21:42:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: 416307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mon 2024-03-27T21:42:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: 416317 4 -rw-r--r-- 1 ubuntu ubuntu 2348 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mon/test_log_rss_usage.cc 2024-03-27T21:42:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: 416319 4 -rw-r--r-- 1 ubuntu ubuntu 2132 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mon/test_mon_rss_usage.cc 2024-03-27T21:42:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: 416308 4 -rw-r--r-- 1 ubuntu ubuntu 2401 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mon/CMakeLists.txt 2024-03-27T21:42:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: 416320 4 -rw-r--r-- 1 ubuntu ubuntu 3969 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mon/test_mon_types.cc 2024-03-27T21:42:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: 416312 4 -rwxr-xr-x 1 ubuntu ubuntu 662 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mon/bench_commit.py 2024-03-27T21:42:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: 416316 36 -rw-r--r-- 1 ubuntu ubuntu 33536 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mon/test_election.cc 2024-03-27T21:42:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: 416314 12 -rw-r--r-- 1 ubuntu ubuntu 8231 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mon/test-mon-msg.cc 2024-03-27T21:42:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: 416311 4 -rwxr-xr-x 1 ubuntu ubuntu 3432 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mon/bench_auth.py 2024-03-27T21:42:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: 416315 4 -rw-r--r-- 1 ubuntu ubuntu 3374 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mon/test_config_map.cc 2024-03-27T21:42:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: 416309 12 -rw-r--r-- 1 ubuntu ubuntu 8513 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mon/MonMap.cc 2024-03-27T21:42:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: 416310 8 -rw-r--r-- 1 ubuntu ubuntu 6264 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mon/PGMap.cc 2024-03-27T21:42:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: 416318 4 -rw-r--r-- 1 ubuntu ubuntu 2276 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mon/test_mon_memory_target.cc 2024-03-27T21:42:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: 416321 28 -rw-r--r-- 1 ubuntu ubuntu 28304 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mon/test_mon_workloadgen.cc 2024-03-27T21:42:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: 416313 16 -rw-r--r-- 1 ubuntu ubuntu 13171 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mon/moncap.cc 2024-03-27T21:42:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: 416666 4 -rw-r--r-- 1 ubuntu ubuntu 2992 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_weighted_shuffle.cc 2024-03-27T21:42:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: 416648 4 -rw-r--r-- 1 ubuntu ubuntu 3587 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_random_string.cc 2024-03-27T21:42:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: 416656 36 -rw-r--r-- 1 ubuntu ubuntu 34961 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_snap_mapper.cc 2024-03-27T21:42:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: 415350 20 -rw-r--r-- 1 ubuntu ubuntu 16477 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/admin_socket.cc 2024-03-27T21:42:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: 416031 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados 2024-03-27T21:42:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: 416065 8 -rw-r--r-- 1 ubuntu ubuntu 4718 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/stat_cxx.cc 2024-03-27T21:42:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: 416041 4 -rw-r--r-- 1 ubuntu ubuntu 1812 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/cls_remote_reads.cc 2024-03-27T21:42:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: 416038 32 -rw-r--r-- 1 ubuntu ubuntu 29256 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/c_read_operations.cc 2024-03-27T21:42:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: 416061 32 -rw-r--r-- 1 ubuntu ubuntu 30853 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/snapshots_cxx.cc 2024-03-27T21:42:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: 416052 12 -rw-r--r-- 1 ubuntu ubuntu 9180 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/lock.cc 2024-03-27T21:42:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: 416032 8 -rw-r--r-- 1 ubuntu ubuntu 6969 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/CMakeLists.txt 2024-03-27T21:42:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: 416072 4 -rw-r--r-- 1 ubuntu ubuntu 1097 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/test_shared.cc 2024-03-27T21:42:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: 416071 4 -rw-r--r-- 1 ubuntu ubuntu 882 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/test_cxx.h 2024-03-27T21:42:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: 416062 12 -rw-r--r-- 1 ubuntu ubuntu 11177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/snapshots_stats.cc 2024-03-27T21:42:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: 416033 8 -rw-r--r-- 1 ubuntu ubuntu 5553 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/TestCase.cc 2024-03-27T21:42:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: 416073 4 -rw-r--r-- 1 ubuntu ubuntu 1456 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/test_shared.h 2024-03-27T21:42:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: 416068 8 -rw-r--r-- 1 ubuntu ubuntu 4246 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/test_common.cc 2024-03-27T21:42:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: 416056 4 -rw-r--r-- 1 ubuntu ubuntu 637 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/op_speed.cc 2024-03-27T21:42:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: 416034 4 -rw-r--r-- 1 ubuntu ubuntu 3255 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/TestCase.h 2024-03-27T21:42:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: 416060 16 -rw-r--r-- 1 ubuntu ubuntu 14249 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/snapshots.cc 2024-03-27T21:42:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: 416075 4 -rw-r--r-- 1 ubuntu ubuntu 3580 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/testcase_cxx.h 2024-03-27T21:42:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: 416054 12 -rw-r--r-- 1 ubuntu ubuntu 11506 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/misc.cc 2024-03-27T21:42:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: 416058 8 -rw-r--r-- 1 ubuntu ubuntu 6615 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/service.cc 2024-03-27T21:42:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: 416051 24 -rw-r--r-- 1 ubuntu ubuntu 23354 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/list_cxx.cc 2024-03-27T21:42:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: 416063 12 -rw-r--r-- 1 ubuntu ubuntu 10251 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/snapshots_stats_cxx.cc 2024-03-27T21:42:44.002 INFO:teuthology.orchestra.run.smithi060.stdout: 416045 4 -rw-r--r-- 1 ubuntu ubuntu 371 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/crimson_utils.h 2024-03-27T21:42:44.002 INFO:teuthology.orchestra.run.smithi060.stdout: 416044 4 -rw-r--r-- 1 ubuntu ubuntu 1075 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/completion_speed.cc 2024-03-27T21:42:44.002 INFO:teuthology.orchestra.run.smithi060.stdout: 416050 20 -rw-r--r-- 1 ubuntu ubuntu 18654 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/list.cc 2024-03-27T21:42:44.002 INFO:teuthology.orchestra.run.smithi060.stdout: 416042 8 -rw-r--r-- 1 ubuntu ubuntu 7556 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/cmd.cc 2024-03-27T21:42:44.002 INFO:teuthology.orchestra.run.smithi060.stdout: 416040 4 -rw-r--r-- 1 ubuntu ubuntu 907 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/cls.cc 2024-03-27T21:42:44.002 INFO:teuthology.orchestra.run.smithi060.stdout: 416064 8 -rw-r--r-- 1 ubuntu ubuntu 4992 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/stat.cc 2024-03-27T21:42:44.002 INFO:teuthology.orchestra.run.smithi060.stdout: 416078 16 -rw-r--r-- 1 ubuntu ubuntu 12835 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/watch_notify_cxx.cc 2024-03-27T21:42:44.002 INFO:teuthology.orchestra.run.smithi060.stdout: 416039 12 -rw-r--r-- 1 ubuntu ubuntu 10088 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/c_write_operations.cc 2024-03-27T21:42:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: 416070 8 -rw-r--r-- 1 ubuntu ubuntu 5801 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/test_cxx.cc 2024-03-27T21:42:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: 416059 4 -rw-r--r-- 1 ubuntu ubuntu 3208 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/service_cxx.cc 2024-03-27T21:42:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: 416076 260 -rw-r--r-- 1 ubuntu ubuntu 263965 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/tier_cxx.cc 2024-03-27T21:42:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: 416037 12 -rw-r--r-- 1 ubuntu ubuntu 10279 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/asio.cc 2024-03-27T21:42:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: 416046 16 -rw-r--r-- 1 ubuntu ubuntu 15525 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/io.cc 2024-03-27T21:42:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: 416069 4 -rw-r--r-- 1 ubuntu ubuntu 325 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/test_common.h 2024-03-27T21:42:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: 416066 8 -rw-r--r-- 1 ubuntu ubuntu 5555 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/test.cc 2024-03-27T21:42:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: 416067 4 -rw-r--r-- 1 ubuntu ubuntu 988 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/test.h 2024-03-27T21:42:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: 416048 4 -rw-r--r-- 1 ubuntu ubuntu 248 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/librados.cc 2024-03-27T21:42:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: 416043 4 -rw-r--r-- 1 ubuntu ubuntu 2618 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/cmd_cxx.cc 2024-03-27T21:42:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: 416053 8 -rw-r--r-- 1 ubuntu ubuntu 7369 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/lock_cxx.cc 2024-03-27T21:42:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: 416055 28 -rw-r--r-- 1 ubuntu ubuntu 26649 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/misc_cxx.cc 2024-03-27T21:42:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: 416074 12 -rw-r--r-- 1 ubuntu ubuntu 12049 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/testcase_cxx.cc 2024-03-27T21:42:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: 416035 60 -rw-r--r-- 1 ubuntu ubuntu 58346 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/aio.cc 2024-03-27T21:42:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: 416036 84 -rw-r--r-- 1 ubuntu ubuntu 82609 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/aio_cxx.cc 2024-03-27T21:42:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: 416047 32 -rw-r--r-- 1 ubuntu ubuntu 30243 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/io_cxx.cc 2024-03-27T21:42:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: 416057 8 -rw-r--r-- 1 ubuntu ubuntu 6540 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/pool.cc 2024-03-27T21:42:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: 416049 4 -rw-r--r-- 1 ubuntu ubuntu 2427 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/librados_config.cc 2024-03-27T21:42:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: 416077 24 -rw-r--r-- 1 ubuntu ubuntu 22602 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados/watch_notify.cc 2024-03-27T21:42:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: 416595 4 -rwxr-xr-x 1 ubuntu ubuntu 1310 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/smoke.sh 2024-03-27T21:42:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: 415402 4 drwxr-xr-x 10 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli 2024-03-27T21:42:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: 415518 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/monmaptool 2024-03-27T21:42:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: 415524 8 -rw-r--r-- 1 ubuntu ubuntu 4795 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/monmaptool/feature-set-unset-list.t 2024-03-27T21:42:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: 415528 4 -rw-r--r-- 1 ubuntu ubuntu 992 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/monmaptool/rm-nonexistent.t 2024-03-27T21:42:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: 415520 4 -rw-r--r-- 1 ubuntu ubuntu 1480 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/monmaptool/add-many.t 2024-03-27T21:42:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: 415522 4 -rw-r--r-- 1 ubuntu ubuntu 934 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/monmaptool/create-print.t 2024-03-27T21:42:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: 415521 4 -rw-r--r-- 1 ubuntu ubuntu 1647 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/monmaptool/clobber.t 2024-03-27T21:42:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: 415519 4 -rw-r--r-- 1 ubuntu ubuntu 1070 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/monmaptool/add-exists.t 2024-03-27T21:42:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: 415529 4 -rw-r--r-- 1 ubuntu ubuntu 918 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/monmaptool/rm.t 2024-03-27T21:42:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: 415523 4 -rw-r--r-- 1 ubuntu ubuntu 646 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/monmaptool/create-with-add.t 2024-03-27T21:42:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: 415530 4 -rw-r--r-- 1 ubuntu ubuntu 58 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/monmaptool/simple.t 2024-03-27T21:42:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: 415525 4 -rw-r--r-- 1 ubuntu ubuntu 521 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/monmaptool/help.t 2024-03-27T21:42:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: 415527 4 -rw-r--r-- 1 ubuntu ubuntu 152 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/monmaptool/print-nonexistent.t 2024-03-27T21:42:44.007 INFO:teuthology.orchestra.run.smithi060.stdout: 415526 4 -rw-r--r-- 1 ubuntu ubuntu 126 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/monmaptool/print-empty.t 2024-03-27T21:42:44.007 INFO:teuthology.orchestra.run.smithi060.stdout: 415546 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/radosgw-admin 2024-03-27T21:42:44.007 INFO:teuthology.orchestra.run.smithi060.stdout: 415547 28 -rw-r--r-- 1 ubuntu ubuntu 26272 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/radosgw-admin/help.t 2024-03-27T21:42:44.007 INFO:teuthology.orchestra.run.smithi060.stdout: 415404 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-authtool 2024-03-27T21:42:44.007 INFO:teuthology.orchestra.run.smithi060.stdout: 415411 4 -rw-r--r-- 1 ubuntu ubuntu 393 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-authtool/create-gen-list-bin.t 2024-03-27T21:42:44.007 INFO:teuthology.orchestra.run.smithi060.stdout: 415408 4 -rw-r--r-- 1 ubuntu ubuntu 392 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-authtool/cap-invalid.t 2024-03-27T21:42:44.007 INFO:teuthology.orchestra.run.smithi060.stdout: 415409 4 -rw-r--r-- 1 ubuntu ubuntu 444 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-authtool/cap-overwrite.t 2024-03-27T21:42:44.007 INFO:teuthology.orchestra.run.smithi060.stdout: 415406 4 -rw-r--r-- 1 ubuntu ubuntu 1034 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-authtool/add-key.t 2024-03-27T21:42:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: 415414 4 -rw-r--r-- 1 ubuntu ubuntu 76 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-authtool/list-empty-bin.t 2024-03-27T21:42:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: 415405 4 -rw-r--r-- 1 ubuntu ubuntu 164 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-authtool/add-key-segv.t 2024-03-27T21:42:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: 415418 4 -rw-r--r-- 1 ubuntu ubuntu 944 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-authtool/manpage.t 2024-03-27T21:42:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: 415416 4 -rw-r--r-- 1 ubuntu ubuntu 243 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-authtool/list-nonexistent-bin.t 2024-03-27T21:42:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: 415417 4 -rw-r--r-- 1 ubuntu ubuntu 243 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-authtool/list-nonexistent.t 2024-03-27T21:42:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: 415419 4 -rw-r--r-- 1 ubuntu ubuntu 64 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-authtool/simple.t 2024-03-27T21:42:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: 415413 4 -rw-r--r-- 1 ubuntu ubuntu 1540 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-authtool/help.t 2024-03-27T21:42:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: 415410 4 -rw-r--r-- 1 ubuntu ubuntu 364 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-authtool/cap.t 2024-03-27T21:42:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: 415407 4 -rw-r--r-- 1 ubuntu ubuntu 245 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-authtool/cap-bin.t 2024-03-27T21:42:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: 415412 4 -rw-r--r-- 1 ubuntu ubuntu 466 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-authtool/create-gen-list.t 2024-03-27T21:42:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: 415415 4 -rw-r--r-- 1 ubuntu ubuntu 76 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-authtool/list-empty.t 2024-03-27T21:42:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: 415531 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/osdmaptool 2024-03-27T21:42:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: 415536 4 -rw-r--r-- 1 ubuntu ubuntu 721 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/osdmaptool/crush.t 2024-03-27T21:42:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: 415542 4 -rw-r--r-- 1 ubuntu ubuntu 1523 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/osdmaptool/test-map-pgs.t 2024-03-27T21:42:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: 415538 4 -rw-r--r-- 1 ubuntu ubuntu 58 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/osdmaptool/missing-argument.t 2024-03-27T21:42:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: 415534 4 -rw-r--r-- 1 ubuntu ubuntu 2734 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/osdmaptool/create-print.t 2024-03-27T21:42:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: 415533 4 -rw-r--r-- 1 ubuntu ubuntu 2156 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/osdmaptool/clobber.t 2024-03-27T21:42:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: 415539 4 -rw-r--r-- 1 ubuntu ubuntu 1389 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/osdmaptool/pool.t 2024-03-27T21:42:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: 415535 24 -rw-r--r-- 1 ubuntu ubuntu 23201 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/osdmaptool/create-racks.t 2024-03-27T21:42:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: 415532 32 -rw-r--r-- 1 ubuntu ubuntu 32627 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/osdmaptool/ceph.conf.withracks 2024-03-27T21:42:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: 415544 4 -rw-r--r-- 1 ubuntu ubuntu 917 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/osdmaptool/upmap-out.t 2024-03-27T21:42:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: 415537 4 -rw-r--r-- 1 ubuntu ubuntu 3137 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/osdmaptool/help.t 2024-03-27T21:42:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: 415541 4 -rw-r--r-- 1 ubuntu ubuntu 178 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/osdmaptool/print-nonexistent.t 2024-03-27T21:42:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: 415543 4 -rw-r--r-- 1 ubuntu ubuntu 2615 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/osdmaptool/tree.t 2024-03-27T21:42:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: 415540 4 -rw-r--r-- 1 ubuntu ubuntu 131 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/osdmaptool/print-empty.t 2024-03-27T21:42:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: 415545 4 -rw-r--r-- 1 ubuntu ubuntu 1326 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/osdmaptool/upmap.t 2024-03-27T21:42:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: 415420 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-conf 2024-03-27T21:42:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: 415427 4 -rw-r--r-- 1 ubuntu ubuntu 1333 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-conf/show-config-value.t 2024-03-27T21:42:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: 415426 4 -rw-r--r-- 1 ubuntu ubuntu 226 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-conf/sections.t 2024-03-27T21:42:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: 415423 4 -rw-r--r-- 1 ubuntu ubuntu 436 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-conf/invalid-args.t 2024-03-27T21:42:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: 415424 4 -rw-r--r-- 1 ubuntu ubuntu 731 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-conf/manpage.t 2024-03-27T21:42:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: 415429 4 -rw-r--r-- 1 ubuntu ubuntu 117 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-conf/simple.t 2024-03-27T21:42:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: 415422 4 -rw-r--r-- 1 ubuntu ubuntu 1936 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-conf/help.t 2024-03-27T21:42:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: 415425 4 -rw-r--r-- 1 ubuntu ubuntu 992 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-conf/option.t 2024-03-27T21:42:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: 415421 4 -rw-r--r-- 1 ubuntu ubuntu 591 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-conf/env-vs-args.t 2024-03-27T21:42:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: 415428 4 -rw-r--r-- 1 ubuntu ubuntu 441 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-conf/show-config.t 2024-03-27T21:42:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: 415432 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool 2024-03-27T21:42:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: 415493 4 -rw-r--r-- 1 ubuntu ubuntu 165 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/straw2.t 2024-03-27T21:42:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: 415503 32 -rw-r--r-- 1 ubuntu ubuntu 31995 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/test-map-indep.crushmap 2024-03-27T21:42:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: 415434 4 -rw-r--r-- 1 ubuntu ubuntu 967 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/add-item-in-tree.t 2024-03-27T21:42:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: 415504 448 -rw-r--r-- 1 ubuntu ubuntu 454875 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/test-map-indep.t 2024-03-27T21:42:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: 415487 4 -rw-r--r-- 1 ubuntu ubuntu 714 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/simple.template.four 2024-03-27T21:42:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: 415468 4 -rw-r--r-- 1 ubuntu ubuntu 550 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/missing-bucket.crushmap.txt 2024-03-27T21:42:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: 415440 4 -rw-r--r-- 1 ubuntu ubuntu 1950 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/build.t 2024-03-27T21:42:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: 415473 24 -rw-r--r-- 1 ubuntu ubuntu 23207 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/reclassify.t 2024-03-27T21:42:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: 415441 4 -rw-r--r-- 1 ubuntu ubuntu 75 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/check-invalid-map.t 2024-03-27T21:42:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: 415489 4 -rw-r--r-- 1 ubuntu ubuntu 1045 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/simple.template.multitree.reweighted 2024-03-27T21:42:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: 415480 4 -rw-r--r-- 1 ubuntu ubuntu 2127 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/show-choose-tries.t 2024-03-27T21:42:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: 415496 12 -rw-r--r-- 1 ubuntu ubuntu 8958 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/test-map-big-1.crushmap 2024-03-27T21:42:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: 415510 88 -rw-r--r-- 1 ubuntu ubuntu 90014 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/test-map-vary-r-0.t 2024-03-27T21:42:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: 415515 4 -rw-r--r-- 1 ubuntu ubuntu 3892 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/test-map-vary-r.crushmap 2024-03-27T21:42:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: 415476 4 -rw-r--r-- 1 ubuntu ubuntu 3854 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/rules.t 2024-03-27T21:42:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: 415511 88 -rw-r--r-- 1 ubuntu ubuntu 90053 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/test-map-vary-r-1.t 2024-03-27T21:42:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: 415501 4 -rw-r--r-- 1 ubuntu ubuntu 3386 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/test-map-hammer-tunables.crushmap 2024-03-27T21:42:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: 415514 88 -rw-r--r-- 1 ubuntu ubuntu 90074 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/test-map-vary-r-4.t 2024-03-27T21:42:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: 415502 312 -rw-r--r-- 1 ubuntu ubuntu 317387 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/test-map-hammer-tunables.t 2024-03-27T21:42:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: 415447 8 -rw-r--r-- 1 ubuntu ubuntu 7191 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/choose-args.t 2024-03-27T21:42:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: 415507 444 -rw-r--r-- 1 ubuntu ubuntu 454632 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/test-map-legacy-tunables.t 2024-03-27T21:42:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: 415463 4 -rw-r--r-- 1 ubuntu ubuntu 715 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/empty-default.cushmap.txt 2024-03-27T21:42:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: 415491 4 -rw-r--r-- 1 ubuntu ubuntu 760 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/simple.template.three 2024-03-27T21:42:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: 415500 12 -rw-r--r-- 1 ubuntu ubuntu 9172 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/test-map-firstn-indep.txt 2024-03-27T21:42:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: 415465 4 -rw-r--r-- 1 ubuntu ubuntu 368 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/five-devices.crushmap 2024-03-27T21:42:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: 415482 4 -rw-r--r-- 1 ubuntu ubuntu 316 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/simple.template 2024-03-27T21:42:44.016 INFO:teuthology.orchestra.run.smithi060.stdout: 415516 4 -rw-r--r-- 1 ubuntu ubuntu 376 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/tree.template 2024-03-27T21:42:44.016 INFO:teuthology.orchestra.run.smithi060.stdout: 415495 32 -rw-r--r-- 1 ubuntu ubuntu 31995 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/test-map-a.crushmap 2024-03-27T21:42:44.016 INFO:teuthology.orchestra.run.smithi060.stdout: 415490 4 -rw-r--r-- 1 ubuntu ubuntu 760 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/simple.template.one 2024-03-27T21:42:44.016 INFO:teuthology.orchestra.run.smithi060.stdout: 415505 4 -rw-r--r-- 1 ubuntu ubuntu 3387 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/test-map-jewel-tunables.crushmap 2024-03-27T21:42:44.016 INFO:teuthology.orchestra.run.smithi060.stdout: 415481 4 -rw-r--r-- 1 ubuntu ubuntu 699 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/show-choose-tries.txt 2024-03-27T21:42:44.016 INFO:teuthology.orchestra.run.smithi060.stdout: 415435 4 -rw-r--r-- 1 ubuntu ubuntu 3507 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/add-item.t 2024-03-27T21:42:44.016 INFO:teuthology.orchestra.run.smithi060.stdout: 415513 88 -rw-r--r-- 1 ubuntu ubuntu 90020 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/test-map-vary-r-3.t 2024-03-27T21:42:44.016 INFO:teuthology.orchestra.run.smithi060.stdout: 415485 4 -rw-r--r-- 1 ubuntu ubuntu 870 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/simple.template.adj.two 2024-03-27T21:42:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: 415446 4 -rw-r--r-- 1 ubuntu ubuntu 1688 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/choose-args.crush 2024-03-27T21:42:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: 415477 4 -rw-r--r-- 1 ubuntu ubuntu 794 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/rules.txt 2024-03-27T21:42:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: 415486 4 -rw-r--r-- 1 ubuntu ubuntu 883 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/simple.template.five 2024-03-27T21:42:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: 415479 1004 -rw-r--r-- 1 ubuntu ubuntu 1024301 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/set-choose.t 2024-03-27T21:42:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: 415474 4 -rw-r--r-- 1 ubuntu ubuntu 401 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/reweight.t 2024-03-27T21:42:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: 415492 4 -rw-r--r-- 1 ubuntu ubuntu 760 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/simple.template.two 2024-03-27T21:42:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: 415461 4 -rw-r--r-- 1 ubuntu ubuntu 1724 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/device-class.crush 2024-03-27T21:42:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: 415475 4 -rw-r--r-- 1 ubuntu ubuntu 246 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/reweight_multiple.t 2024-03-27T21:42:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: 415497 448 -rw-r--r-- 1 ubuntu ubuntu 454828 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/test-map-bobtail-tunables.t 2024-03-27T21:42:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: 415498 312 -rw-r--r-- 1 ubuntu ubuntu 317913 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/test-map-firefly-tunables.t 2024-03-27T21:42:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: 415433 4 -rw-r--r-- 1 ubuntu ubuntu 1617 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/add-bucket.t 2024-03-27T21:42:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: 415478 4 -rw-r--r-- 1 ubuntu ubuntu 2370 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/set-choose.crushmap.txt 2024-03-27T21:42:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: 415448 4 -rw-r--r-- 1 ubuntu ubuntu 689 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/compile-decompile-recompile.t 2024-03-27T21:42:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: 415472 4 -rw-r--r-- 1 ubuntu ubuntu 3597 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/output-csv.t 2024-03-27T21:42:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: 415443 4 -rw-r--r-- 1 ubuntu ubuntu 247 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/check-names.empty.t 2024-03-27T21:42:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: 415470 4 -rw-r--r-- 1 ubuntu ubuntu 1295 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/multitype.before 2024-03-27T21:42:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: 415469 4 -rw-r--r-- 1 ubuntu ubuntu 1441 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/multitype.after 2024-03-27T21:42:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: 415483 4 -rw-r--r-- 1 ubuntu ubuntu 714 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/simple.template.adj.one 2024-03-27T21:42:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: 415449 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/crush-classes 2024-03-27T21:42:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: 415450 4 -rw-r--r-- 1 ubuntu ubuntu 2358 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/crush-classes/a 2024-03-27T21:42:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: 415451 24 -rw-r--r-- 1 ubuntu ubuntu 20656 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/crush-classes/b 2024-03-27T21:42:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: 415456 60 -rw-r--r-- 1 ubuntu ubuntu 61002 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/crush-classes/f 2024-03-27T21:42:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: 415458 44 -rw-r--r-- 1 ubuntu ubuntu 43071 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/crush-classes/g 2024-03-27T21:42:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: 415453 12 -rw-r--r-- 1 ubuntu ubuntu 8801 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/crush-classes/c 2024-03-27T21:42:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: 415454 4 -rw-r--r-- 1 ubuntu ubuntu 3657 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/crush-classes/d 2024-03-27T21:42:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: 415460 60 -rw-r--r-- 1 ubuntu ubuntu 61002 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/crush-classes/gabe2 2024-03-27T21:42:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: 415457 8 -rw-r--r-- 1 ubuntu ubuntu 8184 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/crush-classes/flax 2024-03-27T21:42:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: 415455 8 -rw-r--r-- 1 ubuntu ubuntu 7094 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/crush-classes/e 2024-03-27T21:42:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: 415452 64 -rw-r--r-- 1 ubuntu ubuntu 64806 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/crush-classes/beesly 2024-03-27T21:42:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: 415459 60 -rw-r--r-- 1 ubuntu ubuntu 61114 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/crush-classes/gabe 2024-03-27T21:42:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: 415442 4 -rw-r--r-- 1 ubuntu ubuntu 75 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/check-names.empty.crushmap.txt 2024-03-27T21:42:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: 415464 4 -rw-r--r-- 1 ubuntu ubuntu 242 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/empty-default.t 2024-03-27T21:42:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: 415499 4 -rw-r--r-- 1 ubuntu ubuntu 941 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/test-map-firstn-indep.t 2024-03-27T21:42:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: 415488 4 -rw-r--r-- 1 ubuntu ubuntu 894 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/simple.template.multitree 2024-03-27T21:42:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: 415467 4 -rw-r--r-- 1 ubuntu ubuntu 607 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/location.t 2024-03-27T21:42:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: 415466 8 -rw-r--r-- 1 ubuntu ubuntu 7894 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/help.t 2024-03-27T21:42:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: 415444 4 -rw-r--r-- 1 ubuntu ubuntu 611 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/check-names.max-id.t 2024-03-27T21:42:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: 415436 4 -rw-r--r-- 1 ubuntu ubuntu 734 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/adjust-item-weight.t 2024-03-27T21:42:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: 415506 312 -rw-r--r-- 1 ubuntu ubuntu 317505 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/test-map-jewel-tunables.t 2024-03-27T21:42:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: 415494 4 -rw-r--r-- 1 ubuntu ubuntu 674 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/straw2.txt 2024-03-27T21:42:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: 415471 4 -rw-r--r-- 1 ubuntu ubuntu 980 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/need_tree_order.crush 2024-03-27T21:42:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: 415437 36 -rw-r--r-- 1 ubuntu ubuntu 34419 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/arg-order-checks.t 2024-03-27T21:42:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: 415462 4 -rw-r--r-- 1 ubuntu ubuntu 321 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/device-class.t 2024-03-27T21:42:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: 415439 4 -rw-r--r-- 1 ubuntu ubuntu 508 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/bad-mappings.t 2024-03-27T21:42:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: 415512 88 -rw-r--r-- 1 ubuntu ubuntu 90008 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/test-map-vary-r-2.t 2024-03-27T21:42:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: 415508 4 -rw-r--r-- 1 ubuntu ubuntu 808 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/test-map-tries-vs-retries.crushmap 2024-03-27T21:42:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: 415509 356 -rw-r--r-- 1 ubuntu ubuntu 361412 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/test-map-tries-vs-retries.t 2024-03-27T21:42:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: 415484 4 -rw-r--r-- 1 ubuntu ubuntu 870 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/simple.template.adj.three 2024-03-27T21:42:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: 415445 4 -rw-r--r-- 1 ubuntu ubuntu 986 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/check-overlapped-rules.crushmap.txt 2024-03-27T21:42:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: 415517 4 -rw-r--r-- 1 ubuntu ubuntu 1194 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/tree.template.final 2024-03-27T21:42:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: 415438 4 -rw-r--r-- 1 ubuntu ubuntu 500 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/crushtool/bad-mappings.crushmap.txt 2024-03-27T21:42:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: 415548 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/rbd 2024-03-27T21:42:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: 415550 4 -rw-r--r-- 1 ubuntu ubuntu 3986 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/rbd/invalid-snap-usage.t 2024-03-27T21:42:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: 415552 4 -rw-r--r-- 1 ubuntu ubuntu 718 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/rbd/too-many-args.t 2024-03-27T21:42:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: 415551 8 -rw-r--r-- 1 ubuntu ubuntu 4951 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/rbd/not-enough-args.t 2024-03-27T21:42:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: 415549 108 -rw-r--r-- 1 ubuntu ubuntu 109344 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/rbd/help.t 2024-03-27T21:42:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: 415430 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-kvstore-tool 2024-03-27T21:42:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: 415431 4 -rw-r--r-- 1 ubuntu ubuntu 655 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/ceph-kvstore-tool/help.t 2024-03-27T21:42:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: 415403 4 -rw-r--r-- 1 ubuntu ubuntu 8 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli/.gitignore 2024-03-27T21:42:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: 416306 8 -rw-r--r-- 1 ubuntu ubuntu 4248 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mime.cc 2024-03-27T21:42:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: 416593 4 -rw-r--r-- 1 ubuntu ubuntu 3064 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/signals.cc 2024-03-27T21:42:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: 415962 4 -rw-r--r-- 1 ubuntu ubuntu 817 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fooclass.cc 2024-03-27T21:42:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: 416394 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/opensuse-13.2 2024-03-27T21:42:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: 416397 0 lrwxrwxrwx 1 ubuntu ubuntu 24 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/opensuse-13.2/install-deps.sh -> ../../../install-deps.sh 2024-03-27T21:42:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: 416395 4 -rw-r--r-- 1 ubuntu ubuntu 1192 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/opensuse-13.2/Dockerfile.in 2024-03-27T21:42:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: 416396 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/opensuse-13.2/ceph.spec.in -> ../../../ceph.spec.in 2024-03-27T21:42:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: 416636 4 -rw-r--r-- 1 ubuntu ubuntu 1128 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_intarith.cc 2024-03-27T21:42:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: 415385 16 -rw-r--r-- 1 ubuntu ubuntu 15254 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/ceph_argparse.cc 2024-03-27T21:42:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: 415377 80 -rw-r--r-- 1 ubuntu ubuntu 80982 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/bufferlist.cc 2024-03-27T21:42:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: 415878 4 -rw-r--r-- 1 ubuntu ubuntu 471 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crypto_init.cc 2024-03-27T21:42:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: 415357 4 -rw-r--r-- 1 ubuntu ubuntu 744 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/admin_socket_output_tests.h 2024-03-27T21:42:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: 416644 36 -rw-r--r-- 1 ubuntu ubuntu 33139 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_perf_counters_cache.cc 2024-03-27T21:42:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: 415963 12 -rw-r--r-- 1 ubuntu ubuntu 9231 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/formatter.cc 2024-03-27T21:42:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: 416622 24 -rw-r--r-- 1 ubuntu ubuntu 21692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_any.cc 2024-03-27T21:42:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: 416285 16 -rw-r--r-- 1 ubuntu ubuntu 13117 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librgw_file_gp.cc 2024-03-27T21:42:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: 416508 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw 2024-03-27T21:42:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: 416559 4 -rw-r--r-- 1 ubuntu ubuntu 3831 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_multi.md 2024-03-27T21:42:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: 416562 20 -rw-r--r-- 1 ubuntu ubuntu 17567 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_amqp.cc 2024-03-27T21:42:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: 416570 16 -rw-r--r-- 1 ubuntu ubuntu 15466 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_dmclock_scheduler.cc 2024-03-27T21:42:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: 416550 8 -rw-r--r-- 1 ubuntu ubuntu 4645 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test-rgw-common.sh 2024-03-27T21:42:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: 416578 12 -rw-r--r-- 1 ubuntu ubuntu 10006 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_period_history.cc 2024-03-27T21:42:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: 416514 4 -rw-r--r-- 1 ubuntu ubuntu 1655 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/bench_rgw_ratelimit_gc.cc 2024-03-27T21:42:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: 416558 16 -rw-r--r-- 1 ubuntu ubuntu 12904 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_multen.py 2024-03-27T21:42:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: 416561 12 -rw-r--r-- 1 ubuntu ubuntu 10693 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_posix_bucket_cache.cc 2024-03-27T21:42:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: 416509 12 -rw-r--r-- 1 ubuntu ubuntu 10074 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/CMakeLists.txt 2024-03-27T21:42:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: 416581 4 -rw-r--r-- 1 ubuntu ubuntu 2939 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_reshard.cc 2024-03-27T21:42:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: 416510 12 -rw-r--r-- 1 ubuntu ubuntu 11183 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/amqp_mock.cc 2024-03-27T21:42:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: 416582 8 -rw-r--r-- 1 ubuntu ubuntu 4607 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_reshard_wait.cc 2024-03-27T21:42:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: 416567 16 -rw-r--r-- 1 ubuntu ubuntu 13860 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_common.h 2024-03-27T21:42:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: 416512 8 -rw-r--r-- 1 ubuntu ubuntu 5304 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/amqp_url.c 2024-03-27T21:42:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: 416572 48 -rw-r--r-- 1 ubuntu ubuntu 48138 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_iam_policy.cc 2024-03-27T21:42:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: 416513 12 -rw-r--r-- 1 ubuntu ubuntu 9932 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/bench_rgw_ratelimit.cc 2024-03-27T21:42:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: 416548 4 -rwxr-xr-x 1 ubuntu ubuntu 1962 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test-ceph-diff-sorted.sh 2024-03-27T21:42:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: 416551 4 -rwxr-xr-x 1 ubuntu ubuntu 1679 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test-rgw-meta-sync.sh 2024-03-27T21:42:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: 416554 8 -rw-r--r-- 1 ubuntu ubuntu 5176 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_d4n_directory.cc 2024-03-27T21:42:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: 416574 16 -rw-r--r-- 1 ubuntu ubuntu 13679 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_lc.cc 2024-03-27T21:42:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: 416557 12 -rw-r--r-- 1 ubuntu ubuntu 9825 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_log_backing.cc 2024-03-27T21:42:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: 416526 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/lua 2024-03-27T21:42:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: 416529 4 -rw-r--r-- 1 ubuntu ubuntu 220 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/lua/luatests.conf.SAMPLE 2024-03-27T21:42:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: 416533 4 -rw-r--r-- 1 ubuntu ubuntu 127 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/lua/tox.ini 2024-03-27T21:42:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: 416527 4 -rw-r--r-- 1 ubuntu ubuntu 288 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/lua/README.rst 2024-03-27T21:42:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: 416531 4 -rw-r--r-- 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/lua/requirements.txt 2024-03-27T21:42:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: 416528 4 -rw-r--r-- 1 ubuntu ubuntu 1341 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/lua/__init__.py 2024-03-27T21:42:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: 416532 16 -rw-r--r-- 1 ubuntu ubuntu 15094 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/lua/test_lua.py 2024-03-27T21:42:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: 416530 4 -rw-r--r-- 1 ubuntu ubuntu 62 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/lua/pytest.ini 2024-03-27T21:42:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: 416579 8 -rw-r--r-- 1 ubuntu ubuntu 5311 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_putobj.cc 2024-03-27T21:42:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: 416549 4 -rwxr-xr-x 1 ubuntu ubuntu 314 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test-rgw-call.sh 2024-03-27T21:42:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: 416573 12 -rw-r--r-- 1 ubuntu ubuntu 10511 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_kms.cc 2024-03-27T21:42:44.030 INFO:teuthology.orchestra.run.smithi060.stdout: 416525 4 -rw-r--r-- 1 ubuntu ubuntu 1885 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/kafka_stub.cc 2024-03-27T21:42:44.030 INFO:teuthology.orchestra.run.smithi060.stdout: 416535 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/rgw_multi 2024-03-27T21:42:44.030 INFO:teuthology.orchestra.run.smithi060.stdout: 416542 4 -rw-r--r-- 1 ubuntu ubuntu 2842 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/rgw_multi/tools.py 2024-03-27T21:42:44.030 INFO:teuthology.orchestra.run.smithi060.stdout: 416545 12 -rw-r--r-- 1 ubuntu ubuntu 8780 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/rgw_multi/zone_es.py 2024-03-27T21:42:44.030 INFO:teuthology.orchestra.run.smithi060.stdout: 416540 24 -rw-r--r-- 1 ubuntu ubuntu 21099 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/rgw_multi/tests_az.py 2024-03-27T21:42:44.030 INFO:teuthology.orchestra.run.smithi060.stdout: 416543 4 -rw-r--r-- 1 ubuntu ubuntu 1115 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/rgw_multi/zone_az.py 2024-03-27T21:42:44.030 INFO:teuthology.orchestra.run.smithi060.stdout: 416538 16 -rw-r--r-- 1 ubuntu ubuntu 14657 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/rgw_multi/multisite.py 2024-03-27T21:42:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: 416537 4 -rw-r--r-- 1 ubuntu ubuntu 2767 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/rgw_multi/conn.py 2024-03-27T21:42:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: 416546 8 -rw-r--r-- 1 ubuntu ubuntu 7582 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/rgw_multi/zone_rados.py 2024-03-27T21:42:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: 416536 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/rgw_multi/__init__.py 2024-03-27T21:42:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: 416539 116 -rw-r--r-- 1 ubuntu ubuntu 117010 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/rgw_multi/tests.py 2024-03-27T21:42:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: 416544 12 -rw-r--r-- 1 ubuntu ubuntu 11600 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/rgw_multi/zone_cloud.py 2024-03-27T21:42:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: 416541 16 -rw-r--r-- 1 ubuntu ubuntu 12618 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/rgw_multi/tests_es.py 2024-03-27T21:42:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: 416586 20 -rw-r--r-- 1 ubuntu ubuntu 17125 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_xml.cc 2024-03-27T21:42:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: 416515 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/bucket_notification 2024-03-27T21:42:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: 416518 12 -rw-r--r-- 1 ubuntu ubuntu 10986 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/bucket_notification/api.py 2024-03-27T21:42:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: 416520 4 -rwxr-xr-x 1 ubuntu ubuntu 1707 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/bucket_notification/bootstrap 2024-03-27T21:42:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: 416521 4 -rwxr-xr-x 1 ubuntu ubuntu 1607 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/bucket_notification/kafka-security.sh 2024-03-27T21:42:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: 416519 4 -rw-r--r-- 1 ubuntu ubuntu 270 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/bucket_notification/bntests.conf.SAMPLE 2024-03-27T21:42:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: 416516 8 -rw-r--r-- 1 ubuntu ubuntu 6934 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/bucket_notification/README.rst 2024-03-27T21:42:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: 416522 4 -rw-r--r-- 1 ubuntu ubuntu 109 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/bucket_notification/requirements.txt 2024-03-27T21:42:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: 416517 4 -rw-r--r-- 1 ubuntu ubuntu 2102 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/bucket_notification/__init__.py 2024-03-27T21:42:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: 416524 176 -rw-r--r-- 1 ubuntu ubuntu 179671 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/bucket_notification/test_bn.py 2024-03-27T21:42:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: 416523 4 -rw-r--r-- 1 ubuntu ubuntu 410 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/bucket_notification/setup.py 2024-03-27T21:42:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: 416553 36 -rw-r--r-- 1 ubuntu ubuntu 34352 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_cls_fifo_legacy.cc 2024-03-27T21:42:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: 416565 8 -rw-r--r-- 1 ubuntu ubuntu 5163 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_bucket_sync_cache.cc 2024-03-27T21:42:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: 416566 4 -rw-r--r-- 1 ubuntu ubuntu 2572 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_common.cc 2024-03-27T21:42:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: 416563 8 -rw-r--r-- 1 ubuntu ubuntu 4304 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_arn.cc 2024-03-27T21:42:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: 416534 12 -rw-r--r-- 1 ubuntu ubuntu 10101 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/rgw_cr_test.cc 2024-03-27T21:42:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: 416547 4 -rwxr-xr-x 1 ubuntu ubuntu 671 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/run-d4n-unit-tests.sh 2024-03-27T21:42:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: 416577 8 -rw-r--r-- 1 ubuntu ubuntu 6569 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_obj.cc 2024-03-27T21:42:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: 416585 4 -rw-r--r-- 1 ubuntu ubuntu 3125 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_url.cc 2024-03-27T21:42:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: 416564 4 -rw-r--r-- 1 ubuntu ubuntu 1001 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_bencode.cc 2024-03-27T21:42:44.034 INFO:teuthology.orchestra.run.smithi060.stdout: 416555 60 -rw-r--r-- 1 ubuntu ubuntu 61166 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_d4n_filter.cc 2024-03-27T21:42:44.034 INFO:teuthology.orchestra.run.smithi060.stdout: 416552 4 -rwxr-xr-x 1 ubuntu ubuntu 1605 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test-rgw-multisite.sh 2024-03-27T21:42:44.034 INFO:teuthology.orchestra.run.smithi060.stdout: 416569 28 -rw-r--r-- 1 ubuntu ubuntu 26170 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_crypto.cc 2024-03-27T21:42:44.034 INFO:teuthology.orchestra.run.smithi060.stdout: 416575 44 -rw-r--r-- 1 ubuntu ubuntu 44562 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_lua.cc 2024-03-27T21:42:44.034 INFO:teuthology.orchestra.run.smithi060.stdout: 416580 16 -rw-r--r-- 1 ubuntu ubuntu 13171 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_ratelimit.cc 2024-03-27T21:42:44.034 INFO:teuthology.orchestra.run.smithi060.stdout: 416511 4 -rw-r--r-- 1 ubuntu ubuntu 579 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/amqp_mock.h 2024-03-27T21:42:44.034 INFO:teuthology.orchestra.run.smithi060.stdout: 416583 4 -rw-r--r-- 1 ubuntu ubuntu 2251 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_string.cc 2024-03-27T21:42:44.034 INFO:teuthology.orchestra.run.smithi060.stdout: 416568 8 -rw-r--r-- 1 ubuntu ubuntu 5202 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_compression.cc 2024-03-27T21:42:44.035 INFO:teuthology.orchestra.run.smithi060.stdout: 416556 8 -rw-r--r-- 1 ubuntu ubuntu 4109 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_http_manager.cc 2024-03-27T21:42:44.035 INFO:teuthology.orchestra.run.smithi060.stdout: 416560 16 -rw-r--r-- 1 ubuntu ubuntu 15346 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_multi.py 2024-03-27T21:42:44.035 INFO:teuthology.orchestra.run.smithi060.stdout: 416576 12 -rw-r--r-- 1 ubuntu ubuntu 11568 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_manifest.cc 2024-03-27T21:42:44.035 INFO:teuthology.orchestra.run.smithi060.stdout: 416571 4 -rw-r--r-- 1 ubuntu ubuntu 3599 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_gc_log.cc 2024-03-27T21:42:44.035 INFO:teuthology.orchestra.run.smithi060.stdout: 416584 8 -rw-r--r-- 1 ubuntu ubuntu 5300 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rgw/test_rgw_throttle.cc 2024-03-27T21:42:44.035 INFO:teuthology.orchestra.run.smithi060.stdout: 416109 4 drwxr-xr-x 17 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd 2024-03-27T21:42:44.035 INFO:teuthology.orchestra.run.smithi060.stdout: 416251 16 -rw-r--r-- 1 ubuntu ubuntu 15721 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_Groups.cc 2024-03-27T21:42:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: 416259 4 -rw-r--r-- 1 ubuntu ubuntu 1585 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_fixture.h 2024-03-27T21:42:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: 416271 12 -rw-r--r-- 1 ubuntu ubuntu 11873 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_mock_Watcher.cc 2024-03-27T21:42:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: 416256 4 -rw-r--r-- 1 ubuntu ubuntu 689 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_Operations.cc 2024-03-27T21:42:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: 416255 8 -rw-r--r-- 1 ubuntu ubuntu 7711 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_ObjectMap.cc 2024-03-27T21:42:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: 416131 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/deep_copy 2024-03-27T21:42:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: 416134 40 -rw-r--r-- 1 ubuntu ubuntu 38224 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/deep_copy/test_mock_ObjectCopyRequest.cc 2024-03-27T21:42:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: 416135 12 -rw-r--r-- 1 ubuntu ubuntu 9579 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/deep_copy/test_mock_SetHeadRequest.cc 2024-03-27T21:42:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: 416137 12 -rw-r--r-- 1 ubuntu ubuntu 9814 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/deep_copy/test_mock_SnapshotCreateRequest.cc 2024-03-27T21:42:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: 416132 32 -rw-r--r-- 1 ubuntu ubuntu 29392 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/deep_copy/test_mock_ImageCopyRequest.cc 2024-03-27T21:42:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: 416136 36 -rw-r--r-- 1 ubuntu ubuntu 35850 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/deep_copy/test_mock_SnapshotCopyRequest.cc 2024-03-27T21:42:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: 416133 8 -rw-r--r-- 1 ubuntu ubuntu 6885 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/deep_copy/test_mock_MetadataCopyRequest.cc 2024-03-27T21:42:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: 416237 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/operation 2024-03-27T21:42:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: 416239 24 -rw-r--r-- 1 ubuntu ubuntu 21064 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/operation/test_mock_EnableFeaturesRequest.cc 2024-03-27T21:42:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: 416245 16 -rw-r--r-- 1 ubuntu ubuntu 13800 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/operation/test_mock_SnapshotRollbackRequest.cc 2024-03-27T21:42:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: 416247 20 -rw-r--r-- 1 ubuntu ubuntu 16781 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/operation/test_mock_TrimRequest.cc 2024-03-27T21:42:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: 416246 12 -rw-r--r-- 1 ubuntu ubuntu 9303 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/operation/test_mock_SnapshotUnprotectRequest.cc 2024-03-27T21:42:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: 416244 32 -rw-r--r-- 1 ubuntu ubuntu 30634 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/operation/test_mock_SnapshotRemoveRequest.cc 2024-03-27T21:42:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: 416243 8 -rw-r--r-- 1 ubuntu ubuntu 6075 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/operation/test_mock_SnapshotProtectRequest.cc 2024-03-27T21:42:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: 416240 8 -rw-r--r-- 1 ubuntu ubuntu 4750 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/operation/test_mock_Request.cc 2024-03-27T21:42:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: 416238 20 -rw-r--r-- 1 ubuntu ubuntu 17530 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/operation/test_mock_DisableFeaturesRequest.cc 2024-03-27T21:42:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: 416242 16 -rw-r--r-- 1 ubuntu ubuntu 16216 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/operation/test_mock_SnapshotCreateRequest.cc 2024-03-27T21:42:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: 416241 16 -rw-r--r-- 1 ubuntu ubuntu 15166 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/operation/test_mock_ResizeRequest.cc 2024-03-27T21:42:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: 416110 8 -rw-r--r-- 1 ubuntu ubuntu 7007 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/CMakeLists.txt 2024-03-27T21:42:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: 416276 4 -rw-r--r-- 1 ubuntu ubuntu 1582 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_support.h 2024-03-27T21:42:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: 416270 4 -rw-r--r-- 1 ubuntu ubuntu 2436 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_mock_TrashWatcher.cc 2024-03-27T21:42:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: 416277 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/trash 2024-03-27T21:42:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: 416279 8 -rw-r--r-- 1 ubuntu ubuntu 7328 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/trash/test_mock_RemoveRequest.cc 2024-03-27T21:42:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: 416278 8 -rw-r--r-- 1 ubuntu ubuntu 7803 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/trash/test_mock_MoveRequest.cc 2024-03-27T21:42:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: 416260 64 -rw-r--r-- 1 ubuntu ubuntu 62358 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_internal.cc 2024-03-27T21:42:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: 416252 28 -rw-r--r-- 1 ubuntu ubuntu 27184 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_ImageWatcher.cc 2024-03-27T21:42:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: 416154 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/io 2024-03-27T21:42:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: 416156 28 -rw-r--r-- 1 ubuntu ubuntu 25960 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/io/test_mock_ImageRequest.cc 2024-03-27T21:42:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: 416155 48 -rw-r--r-- 1 ubuntu ubuntu 48591 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/io/test_mock_CopyupRequest.cc 2024-03-27T21:42:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: 416157 92 -rw-r--r-- 1 ubuntu ubuntu 90983 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/io/test_mock_ObjectRequest.cc 2024-03-27T21:42:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: 416158 4 -rw-r--r-- 1 ubuntu ubuntu 3296 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/io/test_mock_QosImageDispatch.cc 2024-03-27T21:42:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: 416159 28 -rw-r--r-- 1 ubuntu ubuntu 27984 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/io/test_mock_SimpleSchedulerObjectDispatch.cc 2024-03-27T21:42:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: 416261 444 -rw-r--r-- 1 ubuntu ubuntu 453820 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_librbd.cc 2024-03-27T21:42:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: 416258 8 -rw-r--r-- 1 ubuntu ubuntu 4989 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_fixture.cc 2024-03-27T21:42:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: 416168 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/managed_lock 2024-03-27T21:42:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: 416169 12 -rw-r--r-- 1 ubuntu ubuntu 8851 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/managed_lock/test_mock_AcquireRequest.cc 2024-03-27T21:42:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: 416170 16 -rw-r--r-- 1 ubuntu ubuntu 15949 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/managed_lock/test_mock_BreakRequest.cc 2024-03-27T21:42:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: 416173 4 -rw-r--r-- 1 ubuntu ubuntu 2406 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/managed_lock/test_mock_ReleaseRequest.cc 2024-03-27T21:42:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: 416172 4 -rw-r--r-- 1 ubuntu ubuntu 3474 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/managed_lock/test_mock_ReacquireRequest.cc 2024-03-27T21:42:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: 416171 12 -rw-r--r-- 1 ubuntu ubuntu 9888 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/managed_lock/test_mock_GetLockerRequest.cc 2024-03-27T21:42:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: 416280 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/watcher 2024-03-27T21:42:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: 416281 8 -rw-r--r-- 1 ubuntu ubuntu 8036 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/watcher/test_mock_RewatchRequest.cc 2024-03-27T21:42:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: 416142 84 -rw-r--r-- 1 ubuntu ubuntu 84985 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/fsx.cc 2024-03-27T21:42:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: 416265 20 -rw-r--r-- 1 ubuntu ubuntu 16440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_mock_DeepCopyRequest.cc 2024-03-27T21:42:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: 416268 28 -rw-r--r-- 1 ubuntu ubuntu 26414 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_mock_ManagedLock.cc 2024-03-27T21:42:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: 416118 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/crypto 2024-03-27T21:42:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: 416128 8 -rw-r--r-- 1 ubuntu ubuntu 7266 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/crypto/test_mock_FormatRequest.cc 2024-03-27T21:42:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: 416129 16 -rw-r--r-- 1 ubuntu ubuntu 14294 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/crypto/test_mock_LoadRequest.cc 2024-03-27T21:42:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: 416119 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/crypto/luks 2024-03-27T21:42:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: 416121 8 -rw-r--r-- 1 ubuntu ubuntu 7788 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/crypto/luks/test_mock_FormatRequest.cc 2024-03-27T21:42:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: 416122 16 -rw-r--r-- 1 ubuntu ubuntu 12322 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/crypto/luks/test_mock_LoadRequest.cc 2024-03-27T21:42:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: 416120 12 -rw-r--r-- 1 ubuntu ubuntu 9362 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/crypto/luks/test_mock_FlattenRequest.cc 2024-03-27T21:42:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: 416127 32 -rw-r--r-- 1 ubuntu ubuntu 31064 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/crypto/test_mock_CryptoObjectDispatch.cc 2024-03-27T21:42:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: 416130 8 -rw-r--r-- 1 ubuntu ubuntu 4318 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/crypto/test_mock_ShutDownCryptoRequest.cc 2024-03-27T21:42:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: 416126 4 -rw-r--r-- 1 ubuntu ubuntu 1850 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/crypto/test_mock_CryptoContextPool.cc 2024-03-27T21:42:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: 416123 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/crypto/openssl 2024-03-27T21:42:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: 416124 8 -rw-r--r-- 1 ubuntu ubuntu 4279 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/crypto/openssl/test_DataCryptor.cc 2024-03-27T21:42:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: 416125 8 -rw-r--r-- 1 ubuntu ubuntu 5097 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/crypto/test_mock_BlockCrypto.cc 2024-03-27T21:42:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: 416143 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/image 2024-03-27T21:42:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: 416152 16 -rw-r--r-- 1 ubuntu ubuntu 14182 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/image/test_mock_RemoveRequest.cc 2024-03-27T21:42:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: 416145 8 -rw-r--r-- 1 ubuntu ubuntu 4373 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/image/test_mock_AttachParentRequest.cc 2024-03-27T21:42:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: 416149 8 -rw-r--r-- 1 ubuntu ubuntu 6221 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/image/test_mock_ListWatchersRequest.cc 2024-03-27T21:42:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: 416153 8 -rw-r--r-- 1 ubuntu ubuntu 6527 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/image/test_mock_ValidatePoolRequest.cc 2024-03-27T21:42:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: 416148 4 -rw-r--r-- 1 ubuntu ubuntu 3602 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/image/test_mock_DetachParentRequest.cc 2024-03-27T21:42:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: 416144 12 -rw-r--r-- 1 ubuntu ubuntu 8380 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/image/test_mock_AttachChildRequest.cc 2024-03-27T21:42:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: 416151 68 -rw-r--r-- 1 ubuntu ubuntu 66295 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/image/test_mock_RefreshRequest.cc 2024-03-27T21:42:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: 416147 16 -rw-r--r-- 1 ubuntu ubuntu 15583 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/image/test_mock_DetachChildRequest.cc 2024-03-27T21:42:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: 416150 16 -rw-r--r-- 1 ubuntu ubuntu 14514 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/image/test_mock_PreRemoveRequest.cc 2024-03-27T21:42:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: 416146 32 -rw-r--r-- 1 ubuntu ubuntu 32156 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/image/test_mock_CloneRequest.cc 2024-03-27T21:42:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: 416249 4 -rw-r--r-- 1 ubuntu ubuntu 2343 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_BlockGuard.cc 2024-03-27T21:42:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: 416262 4 -rw-r--r-- 1 ubuntu ubuntu 2059 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_main.cc 2024-03-27T21:42:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: 416183 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mirror 2024-03-27T21:42:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: 416184 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mirror/snapshot 2024-03-27T21:42:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: 416188 16 -rw-r--r-- 1 ubuntu ubuntu 13444 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mirror/snapshot/test_mock_PromoteRequest.cc 2024-03-27T21:42:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: 416189 16 -rw-r--r-- 1 ubuntu ubuntu 16314 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mirror/snapshot/test_mock_UnlinkPeerRequest.cc 2024-03-27T21:42:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: 416185 16 -rw-r--r-- 1 ubuntu ubuntu 12833 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mirror/snapshot/test_mock_CreateNonPrimaryRequest.cc 2024-03-27T21:42:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: 416187 8 -rw-r--r-- 1 ubuntu ubuntu 4816 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mirror/snapshot/test_mock_ImageMeta.cc 2024-03-27T21:42:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: 416190 8 -rw-r--r-- 1 ubuntu ubuntu 6376 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mirror/snapshot/test_mock_Utils.cc 2024-03-27T21:42:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: 416186 32 -rw-r--r-- 1 ubuntu ubuntu 29101 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mirror/snapshot/test_mock_CreatePrimaryRequest.cc 2024-03-27T21:42:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: 416191 24 -rw-r--r-- 1 ubuntu ubuntu 23373 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mirror/test_mock_DisableRequest.cc 2024-03-27T21:42:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: 416248 4 -rw-r--r-- 1 ubuntu ubuntu 1145 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/rbdrw.py 2024-03-27T21:42:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: 416160 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/journal 2024-03-27T21:42:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: 416162 32 -rw-r--r-- 1 ubuntu ubuntu 28942 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/journal/test_Replay.cc 2024-03-27T21:42:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: 416165 12 -rw-r--r-- 1 ubuntu ubuntu 11216 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/journal/test_mock_PromoteRequest.cc 2024-03-27T21:42:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: 416163 4 -rw-r--r-- 1 ubuntu ubuntu 4073 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/journal/test_Stress.cc 2024-03-27T21:42:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: 416167 12 -rw-r--r-- 1 ubuntu ubuntu 8638 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/journal/test_mock_ResetRequest.cc 2024-03-27T21:42:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: 416166 68 -rw-r--r-- 1 ubuntu ubuntu 67341 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/journal/test_mock_Replay.cc 2024-03-27T21:42:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: 416164 8 -rw-r--r-- 1 ubuntu ubuntu 5829 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/journal/test_mock_OpenRequest.cc 2024-03-27T21:42:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: 416161 12 -rw-r--r-- 1 ubuntu ubuntu 8827 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/journal/test_Entries.cc 2024-03-27T21:42:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: 416257 4 -rw-r--r-- 1 ubuntu ubuntu 3253 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_Trash.cc 2024-03-27T21:42:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: 416272 8 -rw-r--r-- 1 ubuntu ubuntu 5399 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_mock_fixture.cc 2024-03-27T21:42:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: 416192 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock 2024-03-27T21:42:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: 416198 4 -rw-r--r-- 1 ubuntu ubuntu 914 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/MockImageWatcher.h 2024-03-27T21:42:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: 416193 4 -rw-r--r-- 1 ubuntu ubuntu 391 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/MockContextWQ.h 2024-03-27T21:42:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: 416200 4 -rw-r--r-- 1 ubuntu ubuntu 2977 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/MockJournal.h 2024-03-27T21:42:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: 416215 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/io 2024-03-27T21:42:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: 416216 4 -rw-r--r-- 1 ubuntu ubuntu 3209 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/io/MockImageDispatch.h 2024-03-27T21:42:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: 416219 4 -rw-r--r-- 1 ubuntu ubuntu 1281 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/io/MockObjectDispatcher.h 2024-03-27T21:42:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: 416218 8 -rw-r--r-- 1 ubuntu ubuntu 6074 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/io/MockObjectDispatch.h 2024-03-27T21:42:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: 416220 4 -rw-r--r-- 1 ubuntu ubuntu 594 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/io/MockQosImageDispatch.h 2024-03-27T21:42:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: 416217 4 -rw-r--r-- 1 ubuntu ubuntu 1445 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/io/MockImageDispatcher.h 2024-03-27T21:42:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: 416196 12 -rw-r--r-- 1 ubuntu ubuntu 8680 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/MockImageCtx.h 2024-03-27T21:42:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: 416202 4 -rw-r--r-- 1 ubuntu ubuntu 2964 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/MockObjectMap.h 2024-03-27T21:42:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: 416209 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/crypto 2024-03-27T21:42:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: 416211 4 -rw-r--r-- 1 ubuntu ubuntu 1279 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/crypto/MockDataCryptor.h 2024-03-27T21:42:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: 416212 4 -rw-r--r-- 1 ubuntu ubuntu 876 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/crypto/MockEncryptionFormat.h 2024-03-27T21:42:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: 416210 4 -rw-r--r-- 1 ubuntu ubuntu 983 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/crypto/MockCryptoInterface.h 2024-03-27T21:42:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: 416197 4 -rw-r--r-- 1 ubuntu ubuntu 975 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/MockImageState.h 2024-03-27T21:42:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: 416204 4 -rw-r--r-- 1 ubuntu ubuntu 536 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/MockPluginRegistry.h 2024-03-27T21:42:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: 416203 4 -rw-r--r-- 1 ubuntu ubuntu 3640 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/MockOperations.h 2024-03-27T21:42:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: 416206 4 -rw-r--r-- 1 ubuntu ubuntu 489 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/MockSafeTimer.h 2024-03-27T21:42:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: 416205 4 -rw-r--r-- 1 ubuntu ubuntu 478 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/MockReadahead.h 2024-03-27T21:42:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: 416213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/exclusive_lock 2024-03-27T21:42:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: 416214 4 -rw-r--r-- 1 ubuntu ubuntu 572 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/exclusive_lock/MockPolicy.h 2024-03-27T21:42:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: 416195 8 -rw-r--r-- 1 ubuntu ubuntu 5109 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/MockImageCtx.cc 2024-03-27T21:42:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: 416199 4 -rw-r--r-- 1 ubuntu ubuntu 233 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/MockJournal.cc 2024-03-27T21:42:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: 416201 4 -rw-r--r-- 1 ubuntu ubuntu 554 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/MockJournalPolicy.h 2024-03-27T21:42:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: 416194 4 -rw-r--r-- 1 ubuntu ubuntu 1418 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/MockExclusiveLock.h 2024-03-27T21:42:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: 416207 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/cache 2024-03-27T21:42:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: 416208 4 -rw-r--r-- 1 ubuntu ubuntu 2329 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/cache/MockImageCache.h 2024-03-27T21:42:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: 416221 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/migration 2024-03-27T21:42:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: 416223 4 -rw-r--r-- 1 ubuntu ubuntu 866 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/migration/MockStreamInterface.h 2024-03-27T21:42:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: 416222 4 -rw-r--r-- 1 ubuntu ubuntu 1570 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/mock/migration/MockSnapshotInterface.h 2024-03-27T21:42:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: 416138 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/exclusive_lock 2024-03-27T21:42:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: 416140 4 -rw-r--r-- 1 ubuntu ubuntu 2677 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/exclusive_lock/test_mock_PreAcquireRequest.cc 2024-03-27T21:42:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: 416141 16 -rw-r--r-- 1 ubuntu ubuntu 12655 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/exclusive_lock/test_mock_PreReleaseRequest.cc 2024-03-27T21:42:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: 416139 24 -rw-r--r-- 1 ubuntu ubuntu 21606 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/exclusive_lock/test_mock_PostAcquireRequest.cc 2024-03-27T21:42:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: 416263 56 -rw-r--r-- 1 ubuntu ubuntu 57302 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_mirroring.cc 2024-03-27T21:42:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: 416273 4 -rw-r--r-- 1 ubuntu ubuntu 2684 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_mock_fixture.h 2024-03-27T21:42:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: 416266 32 -rw-r--r-- 1 ubuntu ubuntu 29663 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_mock_ExclusiveLock.cc 2024-03-27T21:42:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: 416224 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/object_map 2024-03-27T21:42:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: 416234 8 -rw-r--r-- 1 ubuntu ubuntu 4797 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/object_map/test_mock_SnapshotRollbackRequest.cc 2024-03-27T21:42:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: 416228 8 -rw-r--r-- 1 ubuntu ubuntu 4766 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/object_map/test_mock_InvalidateRequest.cc 2024-03-27T21:42:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: 416233 12 -rw-r--r-- 1 ubuntu ubuntu 11512 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/object_map/test_mock_SnapshotRemoveRequest.cc 2024-03-27T21:42:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: 416229 8 -rw-r--r-- 1 ubuntu ubuntu 6268 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/object_map/test_mock_LockRequest.cc 2024-03-27T21:42:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: 416230 16 -rw-r--r-- 1 ubuntu ubuntu 15673 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/object_map/test_mock_RefreshRequest.cc 2024-03-27T21:42:44.053 INFO:teuthology.orchestra.run.smithi060.stdout: 416232 8 -rw-r--r-- 1 ubuntu ubuntu 7270 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/object_map/test_mock_SnapshotCreateRequest.cc 2024-03-27T21:42:44.053 INFO:teuthology.orchestra.run.smithi060.stdout: 416227 92 -rw-r--r-- 1 ubuntu ubuntu 93857 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/object_map/test_mock_DiffRequest.cc 2024-03-27T21:42:44.053 INFO:teuthology.orchestra.run.smithi060.stdout: 416225 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/object_map/mock 2024-03-27T21:42:44.054 INFO:teuthology.orchestra.run.smithi060.stdout: 416226 4 -rw-r--r-- 1 ubuntu ubuntu 1107 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/object_map/mock/MockInvalidateRequest.h 2024-03-27T21:42:44.054 INFO:teuthology.orchestra.run.smithi060.stdout: 416231 8 -rw-r--r-- 1 ubuntu ubuntu 4695 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/object_map/test_mock_ResizeRequest.cc 2024-03-27T21:42:44.054 INFO:teuthology.orchestra.run.smithi060.stdout: 416235 4 -rw-r--r-- 1 ubuntu ubuntu 1879 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/object_map/test_mock_UnlockRequest.cc 2024-03-27T21:42:44.054 INFO:teuthology.orchestra.run.smithi060.stdout: 416236 12 -rw-r--r-- 1 ubuntu ubuntu 9522 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/object_map/test_mock_UpdateRequest.cc 2024-03-27T21:42:44.054 INFO:teuthology.orchestra.run.smithi060.stdout: 416253 44 -rw-r--r-- 1 ubuntu ubuntu 41554 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_Migration.cc 2024-03-27T21:42:44.054 INFO:teuthology.orchestra.run.smithi060.stdout: 416269 12 -rw-r--r-- 1 ubuntu ubuntu 9780 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_mock_ObjectMap.cc 2024-03-27T21:42:44.054 INFO:teuthology.orchestra.run.smithi060.stdout: 416250 28 -rw-r--r-- 1 ubuntu ubuntu 25853 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_DeepCopy.cc 2024-03-27T21:42:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: 416264 4 -rw-r--r-- 1 ubuntu ubuntu 2622 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_mock_ConfigWatcher.cc 2024-03-27T21:42:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: 416111 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/cache 2024-03-27T21:42:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: 416112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/cache/pwl 2024-03-27T21:42:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: 416113 12 -rw-r--r-- 1 ubuntu ubuntu 10309 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/cache/pwl/test_WriteLogMap.cc 2024-03-27T21:42:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: 416114 24 -rw-r--r-- 1 ubuntu ubuntu 24205 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/cache/pwl/test_mock_ReplicatedWriteLog.cc 2024-03-27T21:42:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: 416115 24 -rw-r--r-- 1 ubuntu ubuntu 24435 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/cache/pwl/test_mock_SSDWriteLog.cc 2024-03-27T21:42:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: 416117 28 -rw-r--r-- 1 ubuntu ubuntu 24920 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/cache/test_mock_WriteAroundObjectDispatch.cc 2024-03-27T21:42:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: 416116 16 -rw-r--r-- 1 ubuntu ubuntu 16108 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/cache/test_mock_ParentCacheObjectDispatch.cc 2024-03-27T21:42:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: 416275 4 -rw-r--r-- 1 ubuntu ubuntu 3890 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_support.cc 2024-03-27T21:42:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: 416267 56 -rw-r--r-- 1 ubuntu ubuntu 56284 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_mock_Journal.cc 2024-03-27T21:42:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: 416174 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/migration 2024-03-27T21:42:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: 416180 8 -rw-r--r-- 1 ubuntu ubuntu 7706 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/migration/test_mock_RawSnapshot.cc 2024-03-27T21:42:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: 416176 28 -rw-r--r-- 1 ubuntu ubuntu 28130 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/migration/test_mock_HttpClient.cc 2024-03-27T21:42:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: 416178 40 -rw-r--r-- 1 ubuntu ubuntu 40351 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/migration/test_mock_QCOWFormat.cc 2024-03-27T21:42:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: 416182 4 -rw-r--r-- 1 ubuntu ubuntu 1697 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/migration/test_mock_Utils.cc 2024-03-27T21:42:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: 416175 8 -rw-r--r-- 1 ubuntu ubuntu 5237 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/migration/test_mock_FileStream.cc 2024-03-27T21:42:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: 416179 20 -rw-r--r-- 1 ubuntu ubuntu 17229 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/migration/test_mock_RawFormat.cc 2024-03-27T21:42:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: 416177 8 -rw-r--r-- 1 ubuntu ubuntu 5003 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/migration/test_mock_HttpStream.cc 2024-03-27T21:42:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: 416181 8 -rw-r--r-- 1 ubuntu ubuntu 6454 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/migration/test_mock_S3Stream.cc 2024-03-27T21:42:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: 416274 8 -rwxr-xr-x 1 ubuntu ubuntu 5937 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_notify.py 2024-03-27T21:42:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: 416254 4 -rw-r--r-- 1 ubuntu ubuntu 2644 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librbd/test_MirroringWatcher.cc 2024-03-27T21:42:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: 416332 8 -rw-r--r-- 1 ubuntu ubuntu 4493 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/multi_stress_watch.cc 2024-03-27T21:42:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: 416392 8 -rw-r--r-- 1 ubuntu ubuntu 5375 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/omap_bench.h 2024-03-27T21:42:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: 415599 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_rgw 2024-03-27T21:42:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: 415601 40 -rw-r--r-- 1 ubuntu ubuntu 40916 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_rgw/test_cls_rgw.cc 2024-03-27T21:42:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: 415600 4 -rw-r--r-- 1 ubuntu ubuntu 661 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_rgw/CMakeLists.txt 2024-03-27T21:42:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: 415602 24 -rw-r--r-- 1 ubuntu ubuntu 20799 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_rgw/test_cls_rgw_stats.cc 2024-03-27T21:42:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: 416028 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephsqlite 2024-03-27T21:42:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: 416029 4 -rw-r--r-- 1 ubuntu ubuntu 363 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephsqlite/CMakeLists.txt 2024-03-27T21:42:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: 416030 36 -rw-r--r-- 1 ubuntu ubuntu 33382 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephsqlite/main.cc 2024-03-27T21:42:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: 415603 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_rgw_gc 2024-03-27T21:42:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: 415604 4 -rw-r--r-- 1 ubuntu ubuntu 387 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_rgw_gc/CMakeLists.txt 2024-03-27T21:42:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: 415605 20 -rw-r--r-- 1 ubuntu ubuntu 19975 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_rgw_gc/test_cls_rgw_gc.cc 2024-03-27T21:42:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: 415376 4 -rw-r--r-- 1 ubuntu ubuntu 1880 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/bench_log.cc 2024-03-27T21:42:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: 416626 4 -rw-r--r-- 1 ubuntu ubuntu 1919 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_c2c.cc 2024-03-27T21:42:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: 416645 4 -rwxr-xr-x 1 ubuntu ubuntu 2479 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_pidfile.sh 2024-03-27T21:42:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: 416284 8 -rw-r--r-- 1 ubuntu ubuntu 4764 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librgw_file_cd.cc 2024-03-27T21:42:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: 416664 4 -rwxr-xr-x 1 ubuntu ubuntu 3137 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_unfound.sh 2024-03-27T21:42:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: 416597 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system 2024-03-27T21:42:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: 416608 4 -rw-r--r-- 1 ubuntu ubuntu 1910 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/st_rados_delete_objs.cc 2024-03-27T21:42:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: 416617 4 -rw-r--r-- 1 ubuntu ubuntu 2326 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/systest_runnable.h 2024-03-27T21:42:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: 416615 4 -rw-r--r-- 1 ubuntu ubuntu 1372 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/st_rados_watch.h 2024-03-27T21:42:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: 416598 4 -rw-r--r-- 1 ubuntu ubuntu 1452 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/CMakeLists.txt 2024-03-27T21:42:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: 416606 4 -rw-r--r-- 1 ubuntu ubuntu 3445 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/st_rados_create_pool.cc 2024-03-27T21:42:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: 416601 4 -rw-r--r-- 1 ubuntu ubuntu 3226 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/rados_delete_pools_parallel.cc 2024-03-27T21:42:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: 416605 4 -rwxr-xr-x 1 ubuntu ubuntu 462 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/rerun.sh 2024-03-27T21:42:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: 416609 4 -rw-r--r-- 1 ubuntu ubuntu 1161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/st_rados_delete_objs.h 2024-03-27T21:42:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: 416600 4 -rw-r--r-- 1 ubuntu ubuntu 971 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/cross_process_sem.h 2024-03-27T21:42:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: 416611 4 -rw-r--r-- 1 ubuntu ubuntu 1066 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/st_rados_delete_pool.h 2024-03-27T21:42:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: 416610 4 -rw-r--r-- 1 ubuntu ubuntu 1696 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/st_rados_delete_pool.cc 2024-03-27T21:42:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: 416599 4 -rw-r--r-- 1 ubuntu ubuntu 2803 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/cross_process_sem.cc 2024-03-27T21:42:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: 416618 4 -rw-r--r-- 1 ubuntu ubuntu 1578 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/systest_settings.cc 2024-03-27T21:42:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: 416613 4 -rw-r--r-- 1 ubuntu ubuntu 1356 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/st_rados_list_objects.h 2024-03-27T21:42:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: 416616 8 -rw-r--r-- 1 ubuntu ubuntu 5575 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/systest_runnable.cc 2024-03-27T21:42:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: 416607 4 -rw-r--r-- 1 ubuntu ubuntu 1376 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/st_rados_create_pool.h 2024-03-27T21:42:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: 416602 12 -rw-r--r-- 1 ubuntu ubuntu 9720 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/rados_list_parallel.cc 2024-03-27T21:42:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: 416614 4 -rw-r--r-- 1 ubuntu ubuntu 1266 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/st_rados_notify.h 2024-03-27T21:42:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: 416603 8 -rw-r--r-- 1 ubuntu ubuntu 4223 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/rados_open_pools_parallel.cc 2024-03-27T21:42:44.062 INFO:teuthology.orchestra.run.smithi060.stdout: 416612 4 -rw-r--r-- 1 ubuntu ubuntu 2727 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/st_rados_list_objects.cc 2024-03-27T21:42:44.062 INFO:teuthology.orchestra.run.smithi060.stdout: 416619 4 -rw-r--r-- 1 ubuntu ubuntu 872 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/systest_settings.h 2024-03-27T21:42:44.062 INFO:teuthology.orchestra.run.smithi060.stdout: 416604 8 -rw-r--r-- 1 ubuntu ubuntu 6442 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/system/rados_watch_notify.cc 2024-03-27T21:42:44.062 INFO:teuthology.orchestra.run.smithi060.stdout: 416433 24 -rw-r--r-- 1 ubuntu ubuntu 23801 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/perf_counters.cc 2024-03-27T21:42:44.062 INFO:teuthology.orchestra.run.smithi060.stdout: 416391 16 -rw-r--r-- 1 ubuntu ubuntu 12407 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/omap_bench.cc 2024-03-27T21:42:44.062 INFO:teuthology.orchestra.run.smithi060.stdout: 416672 4 -rw-r--r-- 1 ubuntu ubuntu 3699 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/testmsgr.cc 2024-03-27T21:42:44.062 INFO:teuthology.orchestra.run.smithi060.stdout: 415900 16 -rw-r--r-- 1 ubuntu ubuntu 13939 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/encoding.cc 2024-03-27T21:42:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: 415609 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_version 2024-03-27T21:42:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: 415610 4 -rw-r--r-- 1 ubuntu ubuntu 284 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_version/CMakeLists.txt 2024-03-27T21:42:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: 415611 8 -rw-r--r-- 1 ubuntu ubuntu 7728 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_version/test_cls_version.cc 2024-03-27T21:42:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: 416434 4 -rw-r--r-- 1 ubuntu ubuntu 1477 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/perf_helper.cc 2024-03-27T21:42:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: 416657 4 -rwxr-xr-x 1 ubuntu ubuntu 1299 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_split.sh 2024-03-27T21:42:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: 415881 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/debian-strech 2024-03-27T21:42:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: 415883 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/debian-strech/debian -> ../../../debian 2024-03-27T21:42:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: 415884 0 lrwxrwxrwx 1 ubuntu ubuntu 24 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/debian-strech/install-deps.sh -> ../../../install-deps.sh 2024-03-27T21:42:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: 415882 4 -rw-r--r-- 1 ubuntu ubuntu 1125 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/debian-strech/Dockerfile.in 2024-03-27T21:42:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: 416001 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/lazy-omap-stats 2024-03-27T21:42:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: 416002 4 -rw-r--r-- 1 ubuntu ubuntu 299 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/lazy-omap-stats/CMakeLists.txt 2024-03-27T21:42:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: 416005 4 -rw-r--r-- 1 ubuntu ubuntu 537 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/lazy-omap-stats/main.cc 2024-03-27T21:42:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: 416003 20 -rw-r--r-- 1 ubuntu ubuntu 17314 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/lazy-omap-stats/lazy_omap_stats_test.cc 2024-03-27T21:42:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: 416004 4 -rw-r--r-- 1 ubuntu ubuntu 2632 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/lazy-omap-stats/lazy_omap_stats_test.h 2024-03-27T21:42:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: 415969 4 -rw-r--r-- 1 ubuntu ubuntu 2887 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/gather.cc 2024-03-27T21:42:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: 416658 4 -rw-r--r-- 1 ubuntu ubuntu 1553 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_str_list.cc 2024-03-27T21:42:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: 416654 4 -rw-r--r-- 1 ubuntu ubuntu 3222 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_rgw_token.cc 2024-03-27T21:42:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: 416631 4 -rwxr-xr-x 1 ubuntu ubuntu 1516 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_crush_bucket.sh 2024-03-27T21:42:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: 416596 16 -rw-r--r-- 1 ubuntu ubuntu 14177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/strtol.cc 2024-03-27T21:42:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: 416594 4 -rw-r--r-- 1 ubuntu ubuntu 2783 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/simple_spin.cc 2024-03-27T21:42:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: 416677 0 lrwxrwxrwx 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/ubuntu-20.04 -> ubuntu-18.04 2024-03-27T21:42:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: 415879 4 -rw-r--r-- 1 ubuntu ubuntu 158 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cxx11_client.cc 2024-03-27T21:42:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: 415816 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson 2024-03-27T21:42:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: 415869 20 -rw-r--r-- 1 ubuntu ubuntu 19315 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/test_socket.cc 2024-03-27T21:42:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: 415854 8 -rw-r--r-- 1 ubuntu ubuntu 7753 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/test_calc_subsets.cc 2024-03-27T21:42:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: 415866 24 -rw-r--r-- 1 ubuntu ubuntu 21203 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/test_messenger_thrash.cc 2024-03-27T21:42:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: 415862 8 -rw-r--r-- 1 ubuntu ubuntu 5213 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/test_lru.cc 2024-03-27T21:42:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: 415855 4 -rw-r--r-- 1 ubuntu ubuntu 3501 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/test_config.cc 2024-03-27T21:42:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: 415858 4 -rw-r--r-- 1 ubuntu ubuntu 1127 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/test_denc.cc 2024-03-27T21:42:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: 415817 8 -rw-r--r-- 1 ubuntu ubuntu 4302 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/CMakeLists.txt 2024-03-27T21:42:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: 415824 4 -rw-r--r-- 1 ubuntu ubuntu 1568 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/gtest_seastar.h 2024-03-27T21:42:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: 415865 16 -rw-r--r-- 1 ubuntu ubuntu 15161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/test_messenger_peer.cc 2024-03-27T21:42:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: 415851 8 -rw-r--r-- 1 ubuntu ubuntu 7051 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/test_async_echo.cc 2024-03-27T21:42:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: 415857 40 -rw-r--r-- 1 ubuntu ubuntu 38592 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/test_crimson_scrub.cc 2024-03-27T21:42:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: 415818 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/cbt 2024-03-27T21:42:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: 415820 4 -rw-r--r-- 1 ubuntu ubuntu 868 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/cbt/radosbench_4K_write.yaml 2024-03-27T21:42:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: 415819 4 -rw-r--r-- 1 ubuntu ubuntu 940 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/cbt/radosbench_4K_read.yaml 2024-03-27T21:42:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: 415821 4 -rwxr-xr-x 1 ubuntu ubuntu 2843 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/cbt/t2c.py 2024-03-27T21:42:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: 415849 12 -rw-r--r-- 1 ubuntu ubuntu 9871 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/test_alien_echo.cc 2024-03-27T21:42:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: 415826 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore 2024-03-27T21:42:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: 415838 20 -rw-r--r-- 1 ubuntu ubuntu 16775 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/test_cbjournal.cc 2024-03-27T21:42:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: 415847 64 -rw-r--r-- 1 ubuntu ubuntu 65061 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/test_transaction_manager.cc 2024-03-27T21:42:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: 415846 12 -rw-r--r-- 1 ubuntu ubuntu 9398 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/test_seastore_journal.cc 2024-03-27T21:42:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: 415827 4 -rw-r--r-- 1 ubuntu ubuntu 3400 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/CMakeLists.txt 2024-03-27T21:42:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: 415843 8 -rw-r--r-- 1 ubuntu ubuntu 4355 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/test_randomblock_manager.cc 2024-03-27T21:42:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: 415845 8 -rw-r--r-- 1 ubuntu ubuntu 7416 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/test_seastore_cache.cc 2024-03-27T21:42:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: 415840 8 -rw-r--r-- 1 ubuntu ubuntu 5270 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/test_extent_allocator.cc 2024-03-27T21:42:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: 415836 8 -rw-r--r-- 1 ubuntu ubuntu 4962 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/test_block.h 2024-03-27T21:42:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: 415835 4 -rw-r--r-- 1 ubuntu ubuntu 911 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/test_block.cc 2024-03-27T21:42:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: 415830 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/onode_tree 2024-03-27T21:42:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: 415834 8 -rw-r--r-- 1 ubuntu ubuntu 7324 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/onode_tree/test_value.h 2024-03-27T21:42:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: 415831 4 -rw-r--r-- 1 ubuntu ubuntu 467 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/onode_tree/CMakeLists.txt 2024-03-27T21:42:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: 415832 12 -rw-r--r-- 1 ubuntu ubuntu 9330 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/onode_tree/test_fltree_onode_manager.cc 2024-03-27T21:42:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: 415833 76 -rw-r--r-- 1 ubuntu ubuntu 74651 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/onode_tree/test_staged_fltree.cc 2024-03-27T21:42:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: 415844 36 -rw-r--r-- 1 ubuntu ubuntu 35483 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/test_seastore.cc 2024-03-27T21:42:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: 415842 20 -rw-r--r-- 1 ubuntu ubuntu 18590 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/test_omap_manager.cc 2024-03-27T21:42:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: 415841 20 -rw-r--r-- 1 ubuntu ubuntu 17176 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/test_object_data_handler.cc 2024-03-27T21:42:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: 415848 16 -rw-r--r-- 1 ubuntu ubuntu 12946 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/transaction_manager_test_state.h 2024-03-27T21:42:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: 415828 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/nvmedevice 2024-03-27T21:42:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: 415829 4 -rw-r--r-- 1 ubuntu ubuntu 2936 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/nvmedevice/test_nvmedevice.cc 2024-03-27T21:42:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: 415839 8 -rw-r--r-- 1 ubuntu ubuntu 5444 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/test_collection_manager.cc 2024-03-27T21:42:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: 415837 20 -rw-r--r-- 1 ubuntu ubuntu 19978 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastore/test_btree_lba_manager.cc 2024-03-27T21:42:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: 415856 12 -rw-r--r-- 1 ubuntu ubuntu 8575 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/test_crimson_coroutine.cc 2024-03-27T21:42:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: 415860 12 -rw-r--r-- 1 ubuntu ubuntu 9711 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/test_fixed_kv_node_layout.cc 2024-03-27T21:42:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: 415852 16 -rw-r--r-- 1 ubuntu ubuntu 14609 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/test_backfill.cc 2024-03-27T21:42:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: 415864 4 -rw-r--r-- 1 ubuntu ubuntu 2397 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/test_messenger.h 2024-03-27T21:42:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: 415853 4 -rw-r--r-- 1 ubuntu ubuntu 1599 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/test_buffer.cc 2024-03-27T21:42:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: 415822 4 -rw-r--r-- 1 ubuntu ubuntu 2566 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/ctest_utils.h 2024-03-27T21:42:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: 415825 4 -rw-r--r-- 1 ubuntu ubuntu 2469 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/seastar_runner.h 2024-03-27T21:42:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: 415823 4 -rw-r--r-- 1 ubuntu ubuntu 1903 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/gtest_seastar.cc 2024-03-27T21:42:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: 415868 4 -rw-r--r-- 1 ubuntu ubuntu 1901 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/test_perfcounters.cc 2024-03-27T21:42:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: 415859 4 -rw-r--r-- 1 ubuntu ubuntu 2414 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/test_errorator.cc 2024-03-27T21:42:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: 415867 4 -rw-r--r-- 1 ubuntu ubuntu 2539 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/test_monc.cc 2024-03-27T21:42:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: 415861 12 -rw-r--r-- 1 ubuntu ubuntu 9561 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/test_interruptible_future.cc 2024-03-27T21:42:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: 415863 136 -rw-r--r-- 1 ubuntu ubuntu 135398 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/test_messenger.cc 2024-03-27T21:42:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: 415850 4 -rw-r--r-- 1 ubuntu ubuntu 2438 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crimson/test_alienstore_thread_pool.cc 2024-03-27T21:42:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: 416668 4 -rw-r--r-- 1 ubuntu ubuntu 2540 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_xlist.cc 2024-03-27T21:42:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: 416398 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd 2024-03-27T21:42:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: 416425 64 -rw-r--r-- 1 ubuntu ubuntu 63090 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/types.cc 2024-03-27T21:42:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: 416405 8 -rw-r--r-- 1 ubuntu ubuntu 6845 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/TestMClockScheduler.cc 2024-03-27T21:42:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: 416399 4 -rw-r--r-- 1 ubuntu ubuntu 3575 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/CMakeLists.txt 2024-03-27T21:42:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: 416406 108 -rw-r--r-- 1 ubuntu ubuntu 110450 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/TestOSDMap.cc 2024-03-27T21:42:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: 416413 8 -rw-r--r-- 1 ubuntu ubuntu 5183 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/hitset.cc 2024-03-27T21:42:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: 416403 100 -rw-r--r-- 1 ubuntu ubuntu 98817 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/RadosModel.h 2024-03-27T21:42:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: 416401 16 -rw-r--r-- 1 ubuntu ubuntu 13817 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/Object.h 2024-03-27T21:42:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: 416408 4 -rw-r--r-- 1 ubuntu ubuntu 1405 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/TestOpStat.cc 2024-03-27T21:42:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: 416422 4 -rw-r--r-- 1 ubuntu ubuntu 3250 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/test_pg_transaction.cc 2024-03-27T21:42:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: 416412 8 -rw-r--r-- 1 ubuntu ubuntu 4536 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/ceph_test_osd_stale_read.cc 2024-03-27T21:42:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: 416411 24 -rw-r--r-- 1 ubuntu ubuntu 22438 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/TestRados.cc 2024-03-27T21:42:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: 416424 20 -rw-r--r-- 1 ubuntu ubuntu 19420 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/test_scrubber_be.cc 2024-03-27T21:42:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: 416414 100 -rw-r--r-- 1 ubuntu ubuntu 98791 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/osdcap.cc 2024-03-27T21:42:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: 416420 4 -rw-r--r-- 1 ubuntu ubuntu 3326 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/test_ec_transaction.cc 2024-03-27T21:42:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: 416416 8 -rw-r--r-- 1 ubuntu ubuntu 4731 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/scrubber_generators.cc 2024-03-27T21:42:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: 416418 4 -rw-r--r-- 1 ubuntu ubuntu 2725 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/scrubber_test_datasets.cc 2024-03-27T21:42:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: 416421 8 -rw-r--r-- 1 ubuntu ubuntu 5929 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/test_extent_cache.cc 2024-03-27T21:42:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: 416419 4 -rw-r--r-- 1 ubuntu ubuntu 600 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/scrubber_test_datasets.h 2024-03-27T21:42:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: 416400 8 -rw-r--r-- 1 ubuntu ubuntu 5268 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/Object.cc 2024-03-27T21:42:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: 416410 100 -rw-r--r-- 1 ubuntu ubuntu 98502 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/TestPGLog.cc 2024-03-27T21:42:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: 416415 4 -rwxr-xr-x 1 ubuntu ubuntu 2262 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/safe-to-destroy.sh 2024-03-27T21:42:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: 416417 8 -rw-r--r-- 1 ubuntu ubuntu 6749 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/scrubber_generators.h 2024-03-27T21:42:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: 416404 4 -rw-r--r-- 1 ubuntu ubuntu 2005 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/TestECBackend.cc 2024-03-27T21:42:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: 416409 4 -rw-r--r-- 1 ubuntu ubuntu 1199 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/TestOpStat.h 2024-03-27T21:42:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: 416402 4 -rw-r--r-- 1 ubuntu ubuntu 833 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/RadosModel.cc 2024-03-27T21:42:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: 416407 8 -rw-r--r-- 1 ubuntu ubuntu 8055 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/TestOSDScrub.cc 2024-03-27T21:42:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: 416423 16 -rw-r--r-- 1 ubuntu ubuntu 13686 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osd/test_scrub_sched.cc 2024-03-27T21:42:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: 416102 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libradosstriper 2024-03-27T21:42:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: 416103 4 -rw-r--r-- 1 ubuntu ubuntu 1005 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libradosstriper/CMakeLists.txt 2024-03-27T21:42:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: 416104 4 -rw-r--r-- 1 ubuntu ubuntu 2025 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libradosstriper/TestCase.cc 2024-03-27T21:42:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: 416105 4 -rw-r--r-- 1 ubuntu ubuntu 2186 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libradosstriper/TestCase.h 2024-03-27T21:42:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: 416107 16 -rw-r--r-- 1 ubuntu ubuntu 16023 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libradosstriper/io.cc 2024-03-27T21:42:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: 416108 20 -rw-r--r-- 1 ubuntu ubuntu 16848 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libradosstriper/striping.cc 2024-03-27T21:42:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: 416106 20 -rw-r--r-- 1 ubuntu ubuntu 20187 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libradosstriper/aio.cc 2024-03-27T21:42:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: 416591 4 -rwxr-xr-x 1 ubuntu ubuntu 346 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/run-rbd-valgrind-unit-tests.sh 2024-03-27T21:42:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: 415901 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/encoding 2024-03-27T21:42:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: 415906 4 -rwxr-xr-x 1 ubuntu ubuntu 716 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/encoding/import-generated.sh 2024-03-27T21:42:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: 415904 4 -rwxr-xr-x 1 ubuntu ubuntu 1105 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/encoding/generate-corpus-objects.sh 2024-03-27T21:42:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: 415908 8 -rwxr-xr-x 1 ubuntu ubuntu 7056 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/encoding/readable.sh 2024-03-27T21:42:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: 415902 4 -rw-r--r-- 1 ubuntu ubuntu 158 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/encoding/CMakeLists.txt 2024-03-27T21:42:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: 415905 4 -rwxr-xr-x 1 ubuntu ubuntu 664 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/encoding/identity.sh 2024-03-27T21:42:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: 415903 8 -rwxr-xr-x 1 ubuntu ubuntu 4867 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/encoding/check-generated.sh 2024-03-27T21:42:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: 415907 4 -rwxr-xr-x 1 ubuntu ubuntu 497 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/encoding/import.sh 2024-03-27T21:42:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: 416000 8 -rw-r--r-- 1 ubuntu ubuntu 5252 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/kv_store_bench.h 2024-03-27T21:42:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: 415590 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_queue 2024-03-27T21:42:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: 415591 4 -rw-r--r-- 1 ubuntu ubuntu 308 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_queue/CMakeLists.txt 2024-03-27T21:42:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: 415592 24 -rw-r--r-- 1 ubuntu ubuntu 21234 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_queue/test_cls_queue.cc 2024-03-27T21:42:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: 416592 4 -rw-r--r-- 1 ubuntu ubuntu 551 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/run_cmd.cc 2024-03-27T21:42:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: 416447 4 -rwxr-xr-x 1 ubuntu ubuntu 901 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd-ggate.sh 2024-03-27T21:42:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: 415897 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/dokan 2024-03-27T21:42:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: 415898 4 -rw-r--r-- 1 ubuntu ubuntu 279 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/dokan/CMakeLists.txt 2024-03-27T21:42:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: 415899 28 -rw-r--r-- 1 ubuntu ubuntu 26820 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/dokan/dokan.cc 2024-03-27T21:42:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: 416632 4 -rw-r--r-- 1 ubuntu ubuntu 633 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_csyn.sh 2024-03-27T21:42:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: 415388 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli-integration 2024-03-27T21:42:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: 415389 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli-integration/balancer 2024-03-27T21:42:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: 415390 4 -rw-r--r-- 1 ubuntu ubuntu 1223 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli-integration/balancer/misplaced.t 2024-03-27T21:42:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: 415391 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli-integration/rbd 2024-03-27T21:42:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: 415401 16 -rw-r--r-- 1 ubuntu ubuntu 14182 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli-integration/rbd/unmap.t 2024-03-27T21:42:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: 415395 4 -rw-r--r-- 1 ubuntu ubuntu 1534 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli-integration/rbd/gwcli_delete.t 2024-03-27T21:42:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: 415398 4 -rw-r--r-- 1 ubuntu ubuntu 2255 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli-integration/rbd/rest_api_create.t 2024-03-27T21:42:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: 415400 4 -rw-r--r-- 1 ubuntu ubuntu 2120 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli-integration/rbd/snap-diff.t 2024-03-27T21:42:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: 415397 4 -rw-r--r-- 1 ubuntu ubuntu 2651 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli-integration/rbd/mon-command-help.t 2024-03-27T21:42:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: 415399 4 -rw-r--r-- 1 ubuntu ubuntu 854 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli-integration/rbd/rest_api_delete.t 2024-03-27T21:42:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: 415392 12 -rw-r--r-- 1 ubuntu ubuntu 8760 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli-integration/rbd/defaults.t 2024-03-27T21:42:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: 415393 36 -rw-r--r-- 1 ubuntu ubuntu 33673 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli-integration/rbd/formatted-output.t 2024-03-27T21:42:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: 415394 8 -rw-r--r-- 1 ubuntu ubuntu 4524 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli-integration/rbd/gwcli_create.t 2024-03-27T21:42:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: 415396 4 -rw-r--r-- 1 ubuntu ubuntu 1444 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cli-integration/rbd/iscsi_client.t 2024-03-27T21:42:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: 416620 12 -rw-r--r-- 1 ubuntu ubuntu 9731 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_addrs.cc 2024-03-27T21:42:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: 416678 4 -rw-r--r-- 1 ubuntu ubuntu 1407 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/unit.cc 2024-03-27T21:42:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: 415982 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/journal 2024-03-27T21:42:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: 415984 8 -rw-r--r-- 1 ubuntu ubuntu 4199 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/journal/RadosTestFixture.cc 2024-03-27T21:42:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: 415983 4 -rw-r--r-- 1 ubuntu ubuntu 661 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/journal/CMakeLists.txt 2024-03-27T21:42:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: 415993 8 -rw-r--r-- 1 ubuntu ubuntu 5450 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/journal/test_JournalRecorder.cc 2024-03-27T21:42:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: 415985 4 -rw-r--r-- 1 ubuntu ubuntu 2386 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/journal/RadosTestFixture.h 2024-03-27T21:42:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: 415997 16 -rw-r--r-- 1 ubuntu ubuntu 15613 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/journal/test_ObjectRecorder.cc 2024-03-27T21:42:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: 415996 8 -rw-r--r-- 1 ubuntu ubuntu 8088 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/journal/test_ObjectPlayer.cc 2024-03-27T21:42:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: 415992 32 -rw-r--r-- 1 ubuntu ubuntu 31562 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/journal/test_JournalPlayer.cc 2024-03-27T21:42:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: 415998 4 -rw-r--r-- 1 ubuntu ubuntu 668 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/journal/test_main.cc 2024-03-27T21:42:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: 415989 4 -rw-r--r-- 1 ubuntu ubuntu 2392 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/journal/test_Entry.cc 2024-03-27T21:42:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: 415994 8 -rw-r--r-- 1 ubuntu ubuntu 6515 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/journal/test_JournalTrimmer.cc 2024-03-27T21:42:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: 415986 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/journal/mock 2024-03-27T21:42:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: 415988 12 -rw-r--r-- 1 ubuntu ubuntu 10179 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/journal/mock/MockJournaler.h 2024-03-27T21:42:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: 415987 4 -rw-r--r-- 1 ubuntu ubuntu 415 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/journal/mock/MockJournaler.cc 2024-03-27T21:42:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: 415991 8 -rw-r--r-- 1 ubuntu ubuntu 6965 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/journal/test_JournalMetadata.cc 2024-03-27T21:42:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: 415995 8 -rw-r--r-- 1 ubuntu ubuntu 5506 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/journal/test_Journaler.cc 2024-03-27T21:42:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: 415990 8 -rw-r--r-- 1 ubuntu ubuntu 7631 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/journal/test_FutureImpl.cc 2024-03-27T21:42:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: 415815 4 -rwxr-xr-x 1 ubuntu ubuntu 900 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/coverage.sh 2024-03-27T21:42:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: 415877 12 -rw-r--r-- 1 ubuntu ubuntu 9684 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/crypto.cc 2024-03-27T21:42:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: 415355 4 -rw-r--r-- 1 ubuntu ubuntu 2157 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/admin_socket_output.h 2024-03-27T21:42:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: 416670 4 -rw-r--r-- 1 ubuntu ubuntu 1438 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/testcrypto.cc 2024-03-27T21:42:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: 415806 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/compressor 2024-03-27T21:42:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: 415808 4 -rw-r--r-- 1 ubuntu ubuntu 1442 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/compressor/compressor_example.h 2024-03-27T21:42:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: 415807 4 -rw-r--r-- 1 ubuntu ubuntu 381 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/compressor/CMakeLists.txt 2024-03-27T21:42:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: 415813 20 -rw-r--r-- 1 ubuntu ubuntu 17393 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/compressor/test_compression.cc 2024-03-27T21:42:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: 415810 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/compressor/osdmaps 2024-03-27T21:42:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: 415811 600 -rw-r--r-- 1 ubuntu ubuntu 612069 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/compressor/osdmaps/osdmap.2982809 2024-03-27T21:42:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: 415812 2632 -rw-r--r-- 1 ubuntu ubuntu 2692799 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/compressor/osdmaps/osdmap.2982809.h 2024-03-27T21:42:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: 415809 4 -rw-r--r-- 1 ubuntu ubuntu 1555 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/compressor/compressor_plugin_example.cc 2024-03-27T21:42:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: 415606 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_sdk 2024-03-27T21:42:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: 415607 4 -rw-r--r-- 1 ubuntu ubuntu 284 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_sdk/CMakeLists.txt 2024-03-27T21:42:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: 415608 4 -rw-r--r-- 1 ubuntu ubuntu 984 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_sdk/test_cls_sdk.cc 2024-03-27T21:42:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: 415895 12 -rwxr-xr-x 1 ubuntu ubuntu 10592 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/docker-test-helper.sh 2024-03-27T21:42:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: 416638 8 -rwxr-xr-x 1 ubuntu ubuntu 6419 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_lost.sh 2024-03-27T21:42:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: 415596 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_refcount 2024-03-27T21:42:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: 415597 4 -rw-r--r-- 1 ubuntu ubuntu 368 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_refcount/CMakeLists.txt 2024-03-27T21:42:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: 415598 20 -rw-r--r-- 1 ubuntu ubuntu 18789 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_refcount/test_cls_refcount.cc 2024-03-27T21:42:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: 416661 12 -rw-r--r-- 1 ubuntu ubuntu 9150 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_subprocess.cc 2024-03-27T21:42:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: 415360 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/behave_tests 2024-03-27T21:42:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: 415361 4 -rw-r--r-- 1 ubuntu ubuntu 1675 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/behave_tests/README.md 2024-03-27T21:42:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: 415371 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/behave_tests/template 2024-03-27T21:42:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: 415372 4 -rw-r--r-- 1 ubuntu ubuntu 1440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/behave_tests/template/bootstrap_script_template 2024-03-27T21:42:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: 415373 4 -rw-r--r-- 1 ubuntu ubuntu 878 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/behave_tests/template/kcli_plan_template 2024-03-27T21:42:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: 415374 4 -rw-r--r-- 1 ubuntu ubuntu 302 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/behave_tests/tox.ini 2024-03-27T21:42:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: 415362 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/behave_tests/features 2024-03-27T21:42:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: 415364 4 -rw-r--r-- 1 ubuntu ubuntu 1665 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/behave_tests/features/ceph_shell_test.feature 2024-03-27T21:42:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: 415370 4 -rw-r--r-- 1 ubuntu ubuntu 454 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/behave_tests/features/validation_util.py 2024-03-27T21:42:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: 415367 4 -rw-r--r-- 1 ubuntu ubuntu 2673 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/behave_tests/features/kcli_handler.py 2024-03-27T21:42:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: 415363 4 -rw-r--r-- 1 ubuntu ubuntu 1744 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/behave_tests/features/ceph_osd_test.feature 2024-03-27T21:42:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: 415368 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/behave_tests/features/steps 2024-03-27T21:42:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: 415369 4 -rw-r--r-- 1 ubuntu ubuntu 3754 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/behave_tests/features/steps/ceph_steps.py 2024-03-27T21:42:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: 415366 8 -rw-r--r-- 1 ubuntu ubuntu 5810 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/behave_tests/features/environment.py 2024-03-27T21:42:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: 415365 4 -rw-r--r-- 1 ubuntu ubuntu 767 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/behave_tests/features/cephadm_test.feature 2024-03-27T21:42:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: 416647 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_rados_tool.sh -> ../../qa/workunits/rados/test_rados_tool.sh 2024-03-27T21:42:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: 415941 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fedora-33 2024-03-27T21:42:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: 415944 0 lrwxrwxrwx 1 ubuntu ubuntu 24 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fedora-33/install-deps.sh -> ../../../install-deps.sh 2024-03-27T21:42:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: 415942 4 -rw-r--r-- 1 ubuntu ubuntu 1003 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fedora-33/Dockerfile.in 2024-03-27T21:42:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: 415943 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fedora-33/ceph.spec.in -> ../../../ceph.spec.in 2024-03-27T21:42:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: 416006 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs 2024-03-27T21:42:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: 416016 4 -rw-r--r-- 1 ubuntu ubuntu 2854 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs/monconfig.cc 2024-03-27T21:42:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: 416021 4 -rw-r--r-- 1 ubuntu ubuntu 3627 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs/reclaim.cc 2024-03-27T21:42:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: 416010 4 -rw-r--r-- 1 ubuntu ubuntu 2644 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs/caps.cc 2024-03-27T21:42:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: 416007 4 -rw-r--r-- 1 ubuntu ubuntu 2391 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs/CMakeLists.txt 2024-03-27T21:42:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: 416009 12 -rw-r--r-- 1 ubuntu ubuntu 11920 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs/acl.cc 2024-03-27T21:42:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: 416015 4 -rw-r--r-- 1 ubuntu ubuntu 1108 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs/main.cc 2024-03-27T21:42:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: 416017 8 -rw-r--r-- 1 ubuntu ubuntu 5089 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs/multiclient.cc 2024-03-27T21:42:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: 416013 20 -rw-r--r-- 1 ubuntu ubuntu 19512 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs/flock.cc 2024-03-27T21:42:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: 416008 16 -rw-r--r-- 1 ubuntu ubuntu 14953 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs/access.cc 2024-03-27T21:42:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: 416018 4 -rw-r--r-- 1 ubuntu ubuntu 2418 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs/newops.cc 2024-03-27T21:42:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: 416012 16 -rw-r--r-- 1 ubuntu ubuntu 13820 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs/deleg.cc 2024-03-27T21:42:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: 416024 12 -rw-r--r-- 1 ubuntu ubuntu 10664 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs/suidsgid.cc 2024-03-27T21:42:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: 416019 8 -rw-r--r-- 1 ubuntu ubuntu 6023 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs/quota.cc 2024-03-27T21:42:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: 416011 4 -rw-r--r-- 1 ubuntu ubuntu 958 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs/ceph_pthread_self.h 2024-03-27T21:42:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: 416023 60 -rw-r--r-- 1 ubuntu ubuntu 60828 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs/snapdiff.cc 2024-03-27T21:42:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: 416025 128 -rw-r--r-- 1 ubuntu ubuntu 127923 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs/test.cc 2024-03-27T21:42:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: 416026 12 -rw-r--r-- 1 ubuntu ubuntu 11147 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs/vxattr.cc 2024-03-27T21:42:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: 416014 16 -rw-r--r-- 1 ubuntu ubuntu 12730 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs/lazyio.cc 2024-03-27T21:42:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: 416020 4 -rw-r--r-- 1 ubuntu ubuntu 2223 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs/readdir_r_cb.cc 2024-03-27T21:42:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: 416022 32 -rw-r--r-- 1 ubuntu ubuntu 31209 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs/recordlock.cc 2024-03-27T21:42:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: 416667 4 -rw-r--r-- 1 ubuntu ubuntu 2428 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_workqueue.cc 2024-03-27T21:42:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: 416630 36 -rw-r--r-- 1 ubuntu ubuntu 34226 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_cors.cc 2024-03-27T21:42:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: 415572 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_hello 2024-03-27T21:42:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: 415573 4 -rw-r--r-- 1 ubuntu ubuntu 292 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_hello/CMakeLists.txt 2024-03-27T21:42:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: 415574 12 -rw-r--r-- 1 ubuntu ubuntu 9493 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_hello/test_cls_hello.cc 2024-03-27T21:42:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: 415999 20 -rw-r--r-- 1 ubuntu ubuntu 16597 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/kv_store_bench.cc 2024-03-27T21:42:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: 416660 4 -rw-r--r-- 1 ubuntu ubuntu 1945 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_striper.cc 2024-03-27T21:42:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: 416629 8 -rwxr-xr-x 1 ubuntu ubuntu 4371 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_common.sh 2024-03-27T21:42:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: 415971 4 -rw-r--r-- 1 ubuntu ubuntu 1172 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/heartbeat_map.cc 2024-03-27T21:42:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: 416589 4 -rwxr-xr-x 1 ubuntu ubuntu 1550 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/run-rbd-tests 2024-03-27T21:42:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: 416642 4 -rwxr-xr-x 1 ubuntu ubuntu 96 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_objectstore_memstore.sh 2024-03-27T21:42:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: 416027 4 -rw-r--r-- 1 ubuntu ubuntu 1645 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/libcephfs_config.cc 2024-03-27T21:42:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: 416671 4 -rw-r--r-- 1 ubuntu ubuntu 1647 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/testkeys.cc 2024-03-27T21:42:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: 416435 4 -rw-r--r-- 1 ubuntu ubuntu 1098 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/perf_helper.h 2024-03-27T21:42:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: 416639 12 -rw-r--r-- 1 ubuntu ubuntu 11250 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_mempool.cc 2024-03-27T21:42:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: 416673 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/ubuntu-18.04 2024-03-27T21:42:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: 416675 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/ubuntu-18.04/debian -> ../../../debian/ 2024-03-27T21:42:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: 416676 0 lrwxrwxrwx 1 ubuntu ubuntu 24 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/ubuntu-18.04/install-deps.sh -> ../../../install-deps.sh 2024-03-27T21:42:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: 416674 4 -rw-r--r-- 1 ubuntu ubuntu 1138 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/ubuntu-18.04/Dockerfile.in 2024-03-27T21:42:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: 415584 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_lua 2024-03-27T21:42:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: 415585 4 -rw-r--r-- 1 ubuntu ubuntu 369 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_lua/CMakeLists.txt 2024-03-27T21:42:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: 415586 32 -rw-r--r-- 1 ubuntu ubuntu 29635 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_lua/test_cls_lua.cc 2024-03-27T21:42:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: 416448 4 drwxr-xr-x 8 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror 2024-03-27T21:42:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: 416485 12 -rw-r--r-- 1 ubuntu ubuntu 10210 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_ImageDeleter.cc 2024-03-27T21:42:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: 416493 4 -rw-r--r-- 1 ubuntu ubuntu 1623 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_fixture.h 2024-03-27T21:42:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: 416504 28 -rw-r--r-- 1 ubuntu ubuntu 25382 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_mock_PoolWatcher.cc 2024-03-27T21:42:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: 416449 4 -rw-r--r-- 1 ubuntu ubuntu 2769 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/CMakeLists.txt 2024-03-27T21:42:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: 416486 52 -rw-r--r-- 1 ubuntu ubuntu 52219 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_ImageReplayer.cc 2024-03-27T21:42:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: 416499 36 -rw-r--r-- 1 ubuntu ubuntu 34516 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_mock_InstanceWatcher.cc 2024-03-27T21:42:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: 416492 8 -rw-r--r-- 1 ubuntu ubuntu 4486 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_fixture.cc 2024-03-27T21:42:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: 416503 36 -rw-r--r-- 1 ubuntu ubuntu 34094 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_mock_PoolReplayer.cc 2024-03-27T21:42:44.094 INFO:teuthology.orchestra.run.smithi060.stdout: 416487 16 -rw-r--r-- 1 ubuntu ubuntu 12634 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_ImageSync.cc 2024-03-27T21:42:44.094 INFO:teuthology.orchestra.run.smithi060.stdout: 416457 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_replayer 2024-03-27T21:42:44.094 INFO:teuthology.orchestra.run.smithi060.stdout: 416470 20 -rw-r--r-- 1 ubuntu ubuntu 19562 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_replayer/test_mock_PrepareLocalImageRequest.cc 2024-03-27T21:42:44.094 INFO:teuthology.orchestra.run.smithi060.stdout: 416469 4 -rw-r--r-- 1 ubuntu ubuntu 3278 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_replayer/test_mock_GetMirrorImageIdRequest.cc 2024-03-27T21:42:44.094 INFO:teuthology.orchestra.run.smithi060.stdout: 416468 24 -rw-r--r-- 1 ubuntu ubuntu 24010 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_replayer/test_mock_CreateImageRequest.cc 2024-03-27T21:42:44.094 INFO:teuthology.orchestra.run.smithi060.stdout: 416458 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_replayer/journal 2024-03-27T21:42:44.094 INFO:teuthology.orchestra.run.smithi060.stdout: 416462 92 -rw-r--r-- 1 ubuntu ubuntu 90150 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_replayer/journal/test_mock_Replayer.cc 2024-03-27T21:42:44.095 INFO:teuthology.orchestra.run.smithi060.stdout: 416460 12 -rw-r--r-- 1 ubuntu ubuntu 10436 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_replayer/journal/test_mock_EventPreprocessor.cc 2024-03-27T21:42:44.095 INFO:teuthology.orchestra.run.smithi060.stdout: 416459 12 -rw-r--r-- 1 ubuntu ubuntu 11447 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_replayer/journal/test_mock_CreateLocalImageRequest.cc 2024-03-27T21:42:44.095 INFO:teuthology.orchestra.run.smithi060.stdout: 416461 32 -rw-r--r-- 1 ubuntu ubuntu 30401 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_replayer/journal/test_mock_PrepareReplayRequest.cc 2024-03-27T21:42:44.095 INFO:teuthology.orchestra.run.smithi060.stdout: 416467 48 -rw-r--r-- 1 ubuntu ubuntu 48271 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_replayer/test_mock_BootstrapRequest.cc 2024-03-27T21:42:44.095 INFO:teuthology.orchestra.run.smithi060.stdout: 416471 36 -rw-r--r-- 1 ubuntu ubuntu 34554 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_replayer/test_mock_PrepareRemoteImageRequest.cc 2024-03-27T21:42:44.095 INFO:teuthology.orchestra.run.smithi060.stdout: 416463 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_replayer/snapshot 2024-03-27T21:42:44.095 INFO:teuthology.orchestra.run.smithi060.stdout: 416466 140 -rw-r--r-- 1 ubuntu ubuntu 140167 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_replayer/snapshot/test_mock_Replayer.cc 2024-03-27T21:42:44.095 INFO:teuthology.orchestra.run.smithi060.stdout: 416465 12 -rw-r--r-- 1 ubuntu ubuntu 12055 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_replayer/snapshot/test_mock_CreateLocalImageRequest.cc 2024-03-27T21:42:44.095 INFO:teuthology.orchestra.run.smithi060.stdout: 416464 24 -rw-r--r-- 1 ubuntu ubuntu 22802 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_replayer/snapshot/test_mock_ApplyImageStateRequest.cc 2024-03-27T21:42:44.096 INFO:teuthology.orchestra.run.smithi060.stdout: 416455 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_map 2024-03-27T21:42:44.096 INFO:teuthology.orchestra.run.smithi060.stdout: 416456 16 -rw-r--r-- 1 ubuntu ubuntu 13298 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_map/test_Policy.cc 2024-03-27T21:42:44.096 INFO:teuthology.orchestra.run.smithi060.stdout: 416496 32 -rw-r--r-- 1 ubuntu ubuntu 30818 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_mock_ImageReplayer.cc 2024-03-27T21:42:44.096 INFO:teuthology.orchestra.run.smithi060.stdout: 416481 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/pool_watcher 2024-03-27T21:42:44.096 INFO:teuthology.orchestra.run.smithi060.stdout: 416482 4 -rw-r--r-- 1 ubuntu ubuntu 3595 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/pool_watcher/test_mock_RefreshImagesRequest.cc 2024-03-27T21:42:44.096 INFO:teuthology.orchestra.run.smithi060.stdout: 416484 8 -rw-r--r-- 1 ubuntu ubuntu 8113 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_ClusterWatcher.cc 2024-03-27T21:42:44.096 INFO:teuthology.orchestra.run.smithi060.stdout: 416501 24 -rw-r--r-- 1 ubuntu ubuntu 24494 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_mock_MirrorStatusUpdater.cc 2024-03-27T21:42:44.097 INFO:teuthology.orchestra.run.smithi060.stdout: 416489 8 -rw-r--r-- 1 ubuntu ubuntu 4603 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_Instances.cc 2024-03-27T21:42:44.097 INFO:teuthology.orchestra.run.smithi060.stdout: 416494 4 -rw-r--r-- 1 ubuntu ubuntu 1558 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_main.cc 2024-03-27T21:42:44.097 INFO:teuthology.orchestra.run.smithi060.stdout: 416483 8 -rw-r--r-- 1 ubuntu ubuntu 5727 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/random_write.cc 2024-03-27T21:42:44.097 INFO:teuthology.orchestra.run.smithi060.stdout: 416472 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_sync 2024-03-27T21:42:44.097 INFO:teuthology.orchestra.run.smithi060.stdout: 416474 16 -rw-r--r-- 1 ubuntu ubuntu 13207 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_sync/test_mock_SyncPointPruneRequest.cc 2024-03-27T21:42:44.097 INFO:teuthology.orchestra.run.smithi060.stdout: 416473 8 -rw-r--r-- 1 ubuntu ubuntu 6891 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_sync/test_mock_SyncPointCreateRequest.cc 2024-03-27T21:42:44.098 INFO:teuthology.orchestra.run.smithi060.stdout: 416506 4 -rw-r--r-- 1 ubuntu ubuntu 2136 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_mock_fixture.cc 2024-03-27T21:42:44.098 INFO:teuthology.orchestra.run.smithi060.stdout: 416475 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/mock 2024-03-27T21:42:44.098 INFO:teuthology.orchestra.run.smithi060.stdout: 416477 4 -rw-r--r-- 1 ubuntu ubuntu 362 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/mock/MockContextWQ.h 2024-03-27T21:42:44.098 INFO:teuthology.orchestra.run.smithi060.stdout: 416479 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/mock/image_sync 2024-03-27T21:42:44.098 INFO:teuthology.orchestra.run.smithi060.stdout: 416480 4 -rw-r--r-- 1 ubuntu ubuntu 821 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/mock/image_sync/MockSyncPointHandler.h 2024-03-27T21:42:44.098 INFO:teuthology.orchestra.run.smithi060.stdout: 416476 4 -rw-r--r-- 1 ubuntu ubuntu 524 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/mock/MockBaseRequest.h 2024-03-27T21:42:44.098 INFO:teuthology.orchestra.run.smithi060.stdout: 416478 4 -rw-r--r-- 1 ubuntu ubuntu 373 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/mock/MockSafeTimer.h 2024-03-27T21:42:44.098 INFO:teuthology.orchestra.run.smithi060.stdout: 416450 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_deleter 2024-03-27T21:42:44.099 INFO:teuthology.orchestra.run.smithi060.stdout: 416454 20 -rw-r--r-- 1 ubuntu ubuntu 16868 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_deleter/test_mock_TrashWatcher.cc 2024-03-27T21:42:44.099 INFO:teuthology.orchestra.run.smithi060.stdout: 416453 16 -rw-r--r-- 1 ubuntu ubuntu 15149 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_deleter/test_mock_TrashRemoveRequest.cc 2024-03-27T21:42:44.099 INFO:teuthology.orchestra.run.smithi060.stdout: 416451 16 -rw-r--r-- 1 ubuntu ubuntu 16334 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_deleter/test_mock_SnapshotPurgeRequest.cc 2024-03-27T21:42:44.099 INFO:teuthology.orchestra.run.smithi060.stdout: 416452 36 -rw-r--r-- 1 ubuntu ubuntu 34172 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/image_deleter/test_mock_TrashMoveRequest.cc 2024-03-27T21:42:44.099 INFO:teuthology.orchestra.run.smithi060.stdout: 416497 20 -rw-r--r-- 1 ubuntu ubuntu 18060 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_mock_ImageSync.cc 2024-03-27T21:42:44.099 INFO:teuthology.orchestra.run.smithi060.stdout: 416491 8 -rw-r--r-- 1 ubuntu ubuntu 8152 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_PoolWatcher.cc 2024-03-27T21:42:44.099 INFO:teuthology.orchestra.run.smithi060.stdout: 416507 4 -rw-r--r-- 1 ubuntu ubuntu 1628 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_mock_fixture.h 2024-03-27T21:42:44.099 INFO:teuthology.orchestra.run.smithi060.stdout: 416490 12 -rw-r--r-- 1 ubuntu ubuntu 9354 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_LeaderWatcher.cc 2024-03-27T21:42:44.100 INFO:teuthology.orchestra.run.smithi060.stdout: 416498 16 -rw-r--r-- 1 ubuntu ubuntu 12869 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_mock_InstanceReplayer.cc 2024-03-27T21:42:44.100 INFO:teuthology.orchestra.run.smithi060.stdout: 416500 20 -rw-r--r-- 1 ubuntu ubuntu 20057 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_mock_LeaderWatcher.cc 2024-03-27T21:42:44.100 INFO:teuthology.orchestra.run.smithi060.stdout: 416488 8 -rw-r--r-- 1 ubuntu ubuntu 4097 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_InstanceWatcher.cc 2024-03-27T21:42:44.100 INFO:teuthology.orchestra.run.smithi060.stdout: 416502 24 -rw-r--r-- 1 ubuntu ubuntu 21510 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_mock_NamespaceReplayer.cc 2024-03-27T21:42:44.100 INFO:teuthology.orchestra.run.smithi060.stdout: 416495 60 -rw-r--r-- 1 ubuntu ubuntu 57432 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_mock_ImageMap.cc 2024-03-27T21:42:44.100 INFO:teuthology.orchestra.run.smithi060.stdout: 416505 8 -rw-r--r-- 1 ubuntu ubuntu 7264 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/rbd_mirror/test_mock_Throttler.cc 2024-03-27T21:42:44.100 INFO:teuthology.orchestra.run.smithi060.stdout: 416290 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mds 2024-03-27T21:42:44.100 INFO:teuthology.orchestra.run.smithi060.stdout: 416291 4 -rw-r--r-- 1 ubuntu ubuntu 1076 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mds/CMakeLists.txt 2024-03-27T21:42:44.100 INFO:teuthology.orchestra.run.smithi060.stdout: 416294 56 -rw-r--r-- 1 ubuntu ubuntu 55287 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mds/TestQuiesceDb.cc 2024-03-27T21:42:44.101 INFO:teuthology.orchestra.run.smithi060.stdout: 416292 16 -rw-r--r-- 1 ubuntu ubuntu 15600 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mds/TestMDSAuthCaps.cc 2024-03-27T21:42:44.101 INFO:teuthology.orchestra.run.smithi060.stdout: 416295 4 -rw-r--r-- 1 ubuntu ubuntu 3870 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mds/TestSessionFilter.cc 2024-03-27T21:42:44.101 INFO:teuthology.orchestra.run.smithi060.stdout: 416293 20 -rw-r--r-- 1 ubuntu ubuntu 16904 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/mds/TestQuiesceAgent.cc 2024-03-27T21:42:44.101 INFO:teuthology.orchestra.run.smithi060.stdout: 415945 0 lrwxrwxrwx 1 ubuntu ubuntu 9 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/fedora-34 -> fedora-33 2024-03-27T21:42:44.101 INFO:teuthology.orchestra.run.smithi060.stdout: 416628 4 -rw-r--r-- 1 ubuntu ubuntu 1005 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_cfuse_cache_invalidate.cc 2024-03-27T21:42:44.101 INFO:teuthology.orchestra.run.smithi060.stdout: 415578 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_lock 2024-03-27T21:42:44.101 INFO:teuthology.orchestra.run.smithi060.stdout: 415579 4 -rw-r--r-- 1 ubuntu ubuntu 323 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_lock/CMakeLists.txt 2024-03-27T21:42:44.101 INFO:teuthology.orchestra.run.smithi060.stdout: 415580 16 -rw-r--r-- 1 ubuntu ubuntu 16127 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_lock/test_cls_lock.cc 2024-03-27T21:42:44.102 INFO:teuthology.orchestra.run.smithi060.stdout: 416588 4 -rwxr-xr-x 1 ubuntu ubuntu 326 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/run-cli-tests-maybe-unset-ccache 2024-03-27T21:42:44.102 INFO:teuthology.orchestra.run.smithi060.stdout: 415553 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/client 2024-03-27T21:42:44.102 INFO:teuthology.orchestra.run.smithi060.stdout: 415557 4 -rw-r--r-- 1 ubuntu ubuntu 1216 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/client/commands.cc 2024-03-27T21:42:44.102 INFO:teuthology.orchestra.run.smithi060.stdout: 415554 4 -rw-r--r-- 1 ubuntu ubuntu 392 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/client/CMakeLists.txt 2024-03-27T21:42:44.102 INFO:teuthology.orchestra.run.smithi060.stdout: 415560 4 -rw-r--r-- 1 ubuntu ubuntu 820 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/client/main.cc 2024-03-27T21:42:44.102 INFO:teuthology.orchestra.run.smithi060.stdout: 415561 20 -rw-r--r-- 1 ubuntu ubuntu 17603 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/client/nonblocking.cc 2024-03-27T21:42:44.102 INFO:teuthology.orchestra.run.smithi060.stdout: 415562 4 -rw-r--r-- 1 ubuntu ubuntu 1342 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/client/ops.cc 2024-03-27T21:42:44.102 INFO:teuthology.orchestra.run.smithi060.stdout: 415556 8 -rw-r--r-- 1 ubuntu ubuntu 6354 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/client/alternate_name.cc 2024-03-27T21:42:44.102 INFO:teuthology.orchestra.run.smithi060.stdout: 415559 4 -rwxr-xr-x 1 ubuntu ubuntu 153 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/client/kernel_untar_build.sh 2024-03-27T21:42:44.103 INFO:teuthology.orchestra.run.smithi060.stdout: 415555 8 -rw-r--r-- 1 ubuntu ubuntu 4990 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/client/TestClient.h 2024-03-27T21:42:44.103 INFO:teuthology.orchestra.run.smithi060.stdout: 415558 4 -rwxr-xr-x 1 ubuntu ubuntu 238 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/client/iozone.sh 2024-03-27T21:42:44.103 INFO:teuthology.orchestra.run.smithi060.stdout: 416659 4 -rw-r--r-- 1 ubuntu ubuntu 2922 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_stress_watch.cc 2024-03-27T21:42:44.103 INFO:teuthology.orchestra.run.smithi060.stdout: 416680 4 -rwxr-xr-x 1 ubuntu ubuntu 2231 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/vstart_wrapper.sh 2024-03-27T21:42:44.103 INFO:teuthology.orchestra.run.smithi060.stdout: 416655 4 -rwxr-xr-x 1 ubuntu ubuntu 854 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_rw.sh 2024-03-27T21:42:44.103 INFO:teuthology.orchestra.run.smithi060.stdout: 416637 28 -rw-r--r-- 1 ubuntu ubuntu 27294 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_ipaddr.cc 2024-03-27T21:42:44.103 INFO:teuthology.orchestra.run.smithi060.stdout: 415885 4 -rw-r--r-- 1 ubuntu ubuntu 639 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/detect-build-env-vars.sh 2024-03-27T21:42:44.103 INFO:teuthology.orchestra.run.smithi060.stdout: 415339 4 -rw-r--r-- 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/.gitignore 2024-03-27T21:42:44.104 INFO:teuthology.orchestra.run.smithi060.stdout: 415593 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_rbd 2024-03-27T21:42:44.104 INFO:teuthology.orchestra.run.smithi060.stdout: 415594 4 -rw-r--r-- 1 ubuntu ubuntu 383 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_rbd/CMakeLists.txt 2024-03-27T21:42:44.104 INFO:teuthology.orchestra.run.smithi060.stdout: 415595 124 -rw-r--r-- 1 ubuntu ubuntu 125038 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_rbd/test_cls_rbd.cc 2024-03-27T21:42:44.104 INFO:teuthology.orchestra.run.smithi060.stdout: 416386 12 -rw-r--r-- 1 ubuntu ubuntu 9291 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/objectstore_bench.cc 2024-03-27T21:42:44.104 INFO:teuthology.orchestra.run.smithi060.stdout: 416426 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osdc 2024-03-27T21:42:44.104 INFO:teuthology.orchestra.run.smithi060.stdout: 416427 4 -rw-r--r-- 1 ubuntu ubuntu 303 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osdc/CMakeLists.txt 2024-03-27T21:42:44.104 INFO:teuthology.orchestra.run.smithi060.stdout: 416428 4 -rw-r--r-- 1 ubuntu ubuntu 2576 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osdc/FakeWriteback.cc 2024-03-27T21:42:44.105 INFO:teuthology.orchestra.run.smithi060.stdout: 416432 16 -rw-r--r-- 1 ubuntu ubuntu 13621 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osdc/object_cacher_stress.cc 2024-03-27T21:42:44.105 INFO:teuthology.orchestra.run.smithi060.stdout: 416431 4 -rw-r--r-- 1 ubuntu ubuntu 1688 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osdc/MemWriteback.h 2024-03-27T21:42:44.105 INFO:teuthology.orchestra.run.smithi060.stdout: 416429 4 -rw-r--r-- 1 ubuntu ubuntu 1424 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osdc/FakeWriteback.h 2024-03-27T21:42:44.105 INFO:teuthology.orchestra.run.smithi060.stdout: 416430 8 -rw-r--r-- 1 ubuntu ubuntu 4928 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/osdc/MemWriteback.cc 2024-03-27T21:42:44.105 INFO:teuthology.orchestra.run.smithi060.stdout: 415375 4 -rw-r--r-- 1 ubuntu ubuntu 535 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/bench_journald_logger.cc 2024-03-27T21:42:44.105 INFO:teuthology.orchestra.run.smithi060.stdout: 416287 32 -rw-r--r-- 1 ubuntu ubuntu 32390 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librgw_file_nfsns.cc 2024-03-27T21:42:44.105 INFO:teuthology.orchestra.run.smithi060.stdout: 416393 4 -rw-r--r-- 1 ubuntu ubuntu 2748 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/on_exit.cc 2024-03-27T21:42:44.105 INFO:teuthology.orchestra.run.smithi060.stdout: 416387 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/old 2024-03-27T21:42:44.105 INFO:teuthology.orchestra.run.smithi060.stdout: 416390 4 -rw-r--r-- 1 ubuntu ubuntu 444 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/old/testfilepath.cc 2024-03-27T21:42:44.106 INFO:teuthology.orchestra.run.smithi060.stdout: 416388 4 -rw-r--r-- 1 ubuntu ubuntu 1280 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/old/test_disk_bw.cc 2024-03-27T21:42:44.106 INFO:teuthology.orchestra.run.smithi060.stdout: 416389 4 -rw-r--r-- 1 ubuntu ubuntu 498 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/old/test_setlayout.c 2024-03-27T21:42:44.106 INFO:teuthology.orchestra.run.smithi060.stdout: 416283 12 -rw-r--r-- 1 ubuntu ubuntu 10914 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librgw_file_aw.cc 2024-03-27T21:42:44.106 INFO:teuthology.orchestra.run.smithi060.stdout: 415341 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/ObjectMap 2024-03-27T21:42:44.106 INFO:teuthology.orchestra.run.smithi060.stdout: 415342 4 -rw-r--r-- 1 ubuntu ubuntu 803 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/ObjectMap/CMakeLists.txt 2024-03-27T21:42:44.106 INFO:teuthology.orchestra.run.smithi060.stdout: 415347 32 -rw-r--r-- 1 ubuntu ubuntu 31252 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/ObjectMap/test_object_map.cc 2024-03-27T21:42:44.106 INFO:teuthology.orchestra.run.smithi060.stdout: 415345 4 -rw-r--r-- 1 ubuntu ubuntu 3191 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/ObjectMap/test_keyvaluedb_atomicity.cc 2024-03-27T21:42:44.106 INFO:teuthology.orchestra.run.smithi060.stdout: 415343 8 -rw-r--r-- 1 ubuntu ubuntu 5982 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/ObjectMap/KeyValueDBMemory.cc 2024-03-27T21:42:44.107 INFO:teuthology.orchestra.run.smithi060.stdout: 415344 8 -rw-r--r-- 1 ubuntu ubuntu 4938 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/ObjectMap/KeyValueDBMemory.h 2024-03-27T21:42:44.107 INFO:teuthology.orchestra.run.smithi060.stdout: 415346 56 -rw-r--r-- 1 ubuntu ubuntu 55602 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/ObjectMap/test_keyvaluedb_iterators.cc 2024-03-27T21:42:44.107 INFO:teuthology.orchestra.run.smithi060.stdout: 415972 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/immutable_object_cache 2024-03-27T21:42:44.107 INFO:teuthology.orchestra.run.smithi060.stdout: 415974 4 -rw-r--r-- 1 ubuntu ubuntu 1237 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/immutable_object_cache/MockCacheDaemon.h 2024-03-27T21:42:44.107 INFO:teuthology.orchestra.run.smithi060.stdout: 415975 8 -rw-r--r-- 1 ubuntu ubuntu 5323 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/immutable_object_cache/test_DomainSocket.cc 2024-03-27T21:42:44.107 INFO:teuthology.orchestra.run.smithi060.stdout: 415973 4 -rw-r--r-- 1 ubuntu ubuntu 684 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/immutable_object_cache/CMakeLists.txt 2024-03-27T21:42:44.107 INFO:teuthology.orchestra.run.smithi060.stdout: 415980 8 -rw-r--r-- 1 ubuntu ubuntu 4948 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/immutable_object_cache/test_multi_session.cc 2024-03-27T21:42:44.107 INFO:teuthology.orchestra.run.smithi060.stdout: 415981 4 -rw-r--r-- 1 ubuntu ubuntu 3068 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/immutable_object_cache/test_object_store.cc 2024-03-27T21:42:44.107 INFO:teuthology.orchestra.run.smithi060.stdout: 415978 4 -rw-r--r-- 1 ubuntu ubuntu 728 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/immutable_object_cache/test_main.cc 2024-03-27T21:42:44.108 INFO:teuthology.orchestra.run.smithi060.stdout: 415979 4 -rw-r--r-- 1 ubuntu ubuntu 1963 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/immutable_object_cache/test_message.cc 2024-03-27T21:42:44.108 INFO:teuthology.orchestra.run.smithi060.stdout: 415976 12 -rw-r--r-- 1 ubuntu ubuntu 12207 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/immutable_object_cache/test_SimplePolicy.cc 2024-03-27T21:42:44.108 INFO:teuthology.orchestra.run.smithi060.stdout: 415977 4 -rw-r--r-- 1 ubuntu ubuntu 846 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/immutable_object_cache/test_common.h 2024-03-27T21:42:44.108 INFO:teuthology.orchestra.run.smithi060.stdout: 416288 8 -rw-r--r-- 1 ubuntu ubuntu 7875 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librgw_file_rename.cc 2024-03-27T21:42:44.108 INFO:teuthology.orchestra.run.smithi060.stdout: 416333 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados 2024-03-27T21:42:44.108 INFO:teuthology.orchestra.run.smithi060.stdout: 416354 8 -rw-r--r-- 1 ubuntu ubuntu 5453 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados/write_operations.cc 2024-03-27T21:42:44.108 INFO:teuthology.orchestra.run.smithi060.stdout: 416351 8 -rw-r--r-- 1 ubuntu ubuntu 4876 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados/start_stop.cc 2024-03-27T21:42:44.108 INFO:teuthology.orchestra.run.smithi060.stdout: 416334 8 -rw-r--r-- 1 ubuntu ubuntu 5216 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados/CMakeLists.txt 2024-03-27T21:42:44.109 INFO:teuthology.orchestra.run.smithi060.stdout: 416340 4 -rw-r--r-- 1 ubuntu ubuntu 3315 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados/ec_io.cc 2024-03-27T21:42:44.109 INFO:teuthology.orchestra.run.smithi060.stdout: 416339 4 -rw-r--r-- 1 ubuntu ubuntu 317 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados/completions.cc 2024-03-27T21:42:44.109 INFO:teuthology.orchestra.run.smithi060.stdout: 416347 4 -rw-r--r-- 1 ubuntu ubuntu 1000 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados/op_speed.cc 2024-03-27T21:42:44.109 INFO:teuthology.orchestra.run.smithi060.stdout: 416352 4 -rw-r--r-- 1 ubuntu ubuntu 1143 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados/test_neorados.cc 2024-03-27T21:42:44.109 INFO:teuthology.orchestra.run.smithi060.stdout: 416350 16 -rw-r--r-- 1 ubuntu ubuntu 14965 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados/snapshots.cc 2024-03-27T21:42:44.109 INFO:teuthology.orchestra.run.smithi060.stdout: 416349 24 -rw-r--r-- 1 ubuntu ubuntu 22359 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados/read_operations.cc 2024-03-27T21:42:44.109 INFO:teuthology.orchestra.run.smithi060.stdout: 416346 12 -rw-r--r-- 1 ubuntu ubuntu 8880 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados/misc.cc 2024-03-27T21:42:44.109 INFO:teuthology.orchestra.run.smithi060.stdout: 416341 4 -rw-r--r-- 1 ubuntu ubuntu 3818 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados/ec_list.cc 2024-03-27T21:42:44.109 INFO:teuthology.orchestra.run.smithi060.stdout: 416342 4 -rw-r--r-- 1 ubuntu ubuntu 1497 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados/handler_error.cc 2024-03-27T21:42:44.110 INFO:teuthology.orchestra.run.smithi060.stdout: 416345 8 -rw-r--r-- 1 ubuntu ubuntu 4278 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados/list_pool.cc 2024-03-27T21:42:44.110 INFO:teuthology.orchestra.run.smithi060.stdout: 416338 20 -rw-r--r-- 1 ubuntu ubuntu 17571 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados/common_tests.h 2024-03-27T21:42:44.110 INFO:teuthology.orchestra.run.smithi060.stdout: 416344 4 -rw-r--r-- 1 ubuntu ubuntu 3892 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados/list.cc 2024-03-27T21:42:44.110 INFO:teuthology.orchestra.run.smithi060.stdout: 416336 4 -rw-r--r-- 1 ubuntu ubuntu 3091 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados/cmd.cc 2024-03-27T21:42:44.110 INFO:teuthology.orchestra.run.smithi060.stdout: 416335 4 -rw-r--r-- 1 ubuntu ubuntu 2559 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados/cls.cc 2024-03-27T21:42:44.110 INFO:teuthology.orchestra.run.smithi060.stdout: 416337 4 -rw-r--r-- 1 ubuntu ubuntu 2453 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados/common_tests.cc 2024-03-27T21:42:44.110 INFO:teuthology.orchestra.run.smithi060.stdout: 416343 12 -rw-r--r-- 1 ubuntu ubuntu 10931 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados/io.cc 2024-03-27T21:42:44.111 INFO:teuthology.orchestra.run.smithi060.stdout: 416348 8 -rw-r--r-- 1 ubuntu ubuntu 4956 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados/pool.cc 2024-03-27T21:42:44.111 INFO:teuthology.orchestra.run.smithi060.stdout: 416353 8 -rw-r--r-- 1 ubuntu ubuntu 5656 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/neorados/watch_notify.cc 2024-03-27T21:42:44.111 INFO:teuthology.orchestra.run.smithi060.stdout: 415909 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code 2024-03-27T21:42:44.111 INFO:teuthology.orchestra.run.smithi060.stdout: 415924 8 -rw-r--r-- 1 ubuntu ubuntu 4366 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/TestErasureCodePlugin.cc 2024-03-27T21:42:44.111 INFO:teuthology.orchestra.run.smithi060.stdout: 415922 12 -rw-r--r-- 1 ubuntu ubuntu 11003 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/TestErasureCodeJerasure.cc 2024-03-27T21:42:44.111 INFO:teuthology.orchestra.run.smithi060.stdout: 415916 4 -rw-r--r-- 1 ubuntu ubuntu 177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/ErasureCodePluginMissingEntryPoint.cc 2024-03-27T21:42:44.111 INFO:teuthology.orchestra.run.smithi060.stdout: 415926 4 -rw-r--r-- 1 ubuntu ubuntu 1815 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/TestErasureCodePluginIsa.cc 2024-03-27T21:42:44.111 INFO:teuthology.orchestra.run.smithi060.stdout: 415910 8 -rw-r--r-- 1 ubuntu ubuntu 7183 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/CMakeLists.txt 2024-03-27T21:42:44.111 INFO:teuthology.orchestra.run.smithi060.stdout: 415932 12 -rw-r--r-- 1 ubuntu ubuntu 11876 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/TestErasureCodeShec_arguments.cc 2024-03-27T21:42:44.112 INFO:teuthology.orchestra.run.smithi060.stdout: 415928 4 -rw-r--r-- 1 ubuntu ubuntu 1539 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/TestErasureCodePluginLrc.cc 2024-03-27T21:42:44.112 INFO:teuthology.orchestra.run.smithi060.stdout: 415913 4 -rw-r--r-- 1 ubuntu ubuntu 791 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/ErasureCodePluginFailToInitialize.cc 2024-03-27T21:42:44.112 INFO:teuthology.orchestra.run.smithi060.stdout: 415934 12 -rw-r--r-- 1 ubuntu ubuntu 10659 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/ceph_erasure_code_benchmark.cc 2024-03-27T21:42:44.112 INFO:teuthology.orchestra.run.smithi060.stdout: 415917 4 -rw-r--r-- 1 ubuntu ubuntu 85 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/ErasureCodePluginMissingVersion.cc 2024-03-27T21:42:44.112 INFO:teuthology.orchestra.run.smithi060.stdout: 415918 8 -rw-r--r-- 1 ubuntu ubuntu 5710 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/TestErasureCode.cc 2024-03-27T21:42:44.112 INFO:teuthology.orchestra.run.smithi060.stdout: 415911 8 -rw-r--r-- 1 ubuntu ubuntu 5876 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/ErasureCodeExample.h 2024-03-27T21:42:44.112 INFO:teuthology.orchestra.run.smithi060.stdout: 415935 4 -rw-r--r-- 1 ubuntu ubuntu 1482 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/ceph_erasure_code_benchmark.h 2024-03-27T21:42:44.112 INFO:teuthology.orchestra.run.smithi060.stdout: 415912 4 -rw-r--r-- 1 ubuntu ubuntu 1376 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/ErasureCodePluginExample.cc 2024-03-27T21:42:44.113 INFO:teuthology.orchestra.run.smithi060.stdout: 415930 76 -rw-r--r-- 1 ubuntu ubuntu 76300 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/TestErasureCodeShec.cc 2024-03-27T21:42:44.113 INFO:teuthology.orchestra.run.smithi060.stdout: 415929 4 -rw-r--r-- 1 ubuntu ubuntu 1977 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/TestErasureCodePluginShec.cc 2024-03-27T21:42:44.113 INFO:teuthology.orchestra.run.smithi060.stdout: 415936 12 -rw-r--r-- 1 ubuntu ubuntu 9521 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/ceph_erasure_code_non_regression.cc 2024-03-27T21:42:44.113 INFO:teuthology.orchestra.run.smithi060.stdout: 415921 32 -rw-r--r-- 1 ubuntu ubuntu 31936 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/TestErasureCodeIsa.cc 2024-03-27T21:42:44.113 INFO:teuthology.orchestra.run.smithi060.stdout: 415933 8 -rw-r--r-- 1 ubuntu ubuntu 5809 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/TestErasureCodeShec_thread.cc 2024-03-27T21:42:44.113 INFO:teuthology.orchestra.run.smithi060.stdout: 415920 8 -rw-r--r-- 1 ubuntu ubuntu 7018 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/TestErasureCodeExample.cc 2024-03-27T21:42:44.113 INFO:teuthology.orchestra.run.smithi060.stdout: 415931 12 -rw-r--r-- 1 ubuntu ubuntu 9076 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/TestErasureCodeShec_all.cc 2024-03-27T21:42:44.113 INFO:teuthology.orchestra.run.smithi060.stdout: 415919 20 -rw-r--r-- 1 ubuntu ubuntu 20469 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/TestErasureCodeClay.cc 2024-03-27T21:42:44.113 INFO:teuthology.orchestra.run.smithi060.stdout: 415925 4 -rw-r--r-- 1 ubuntu ubuntu 3539 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/TestErasureCodePluginClay.cc 2024-03-27T21:42:44.114 INFO:teuthology.orchestra.run.smithi060.stdout: 415915 4 -rw-r--r-- 1 ubuntu ubuntu 800 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/ErasureCodePluginHangs.cc 2024-03-27T21:42:44.114 INFO:teuthology.orchestra.run.smithi060.stdout: 415914 4 -rw-r--r-- 1 ubuntu ubuntu 767 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/ErasureCodePluginFailToRegister.cc 2024-03-27T21:42:44.114 INFO:teuthology.orchestra.run.smithi060.stdout: 415927 4 -rw-r--r-- 1 ubuntu ubuntu 2122 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/TestErasureCodePluginJerasure.cc 2024-03-27T21:42:44.114 INFO:teuthology.orchestra.run.smithi060.stdout: 415923 32 -rw-r--r-- 1 ubuntu ubuntu 29144 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/erasure-code/TestErasureCodeLrc.cc 2024-03-27T21:42:44.114 INFO:teuthology.orchestra.run.smithi060.stdout: 416640 4 -rwxr-xr-x 1 ubuntu ubuntu 512 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_missing_unfound.sh 2024-03-27T21:42:44.114 INFO:teuthology.orchestra.run.smithi060.stdout: 415354 8 -rw-r--r-- 1 ubuntu ubuntu 7700 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/admin_socket_output.cc 2024-03-27T21:42:44.114 INFO:teuthology.orchestra.run.smithi060.stdout: 415896 4 -rwxr-xr-x 1 ubuntu ubuntu 649 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/docker-test.sh 2024-03-27T21:42:44.114 INFO:teuthology.orchestra.run.smithi060.stdout: 415886 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/direct_messenger 2024-03-27T21:42:44.115 INFO:teuthology.orchestra.run.smithi060.stdout: 415890 4 -rw-r--r-- 1 ubuntu ubuntu 906 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/direct_messenger/DispatchStrategy.h 2024-03-27T21:42:44.115 INFO:teuthology.orchestra.run.smithi060.stdout: 415887 4 -rw-r--r-- 1 ubuntu ubuntu 302 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/direct_messenger/CMakeLists.txt 2024-03-27T21:42:44.115 INFO:teuthology.orchestra.run.smithi060.stdout: 415892 4 -rw-r--r-- 1 ubuntu ubuntu 2295 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/direct_messenger/QueueStrategy.cc 2024-03-27T21:42:44.115 INFO:teuthology.orchestra.run.smithi060.stdout: 415891 4 -rw-r--r-- 1 ubuntu ubuntu 900 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/direct_messenger/FastStrategy.h 2024-03-27T21:42:44.115 INFO:teuthology.orchestra.run.smithi060.stdout: 415894 16 -rw-r--r-- 1 ubuntu ubuntu 13475 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/direct_messenger/test_direct_messenger.cc 2024-03-27T21:42:44.115 INFO:teuthology.orchestra.run.smithi060.stdout: 415889 4 -rw-r--r-- 1 ubuntu ubuntu 3186 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/direct_messenger/DirectMessenger.h 2024-03-27T21:42:44.115 INFO:teuthology.orchestra.run.smithi060.stdout: 415888 8 -rw-r--r-- 1 ubuntu ubuntu 6907 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/direct_messenger/DirectMessenger.cc 2024-03-27T21:42:44.115 INFO:teuthology.orchestra.run.smithi060.stdout: 415893 4 -rw-r--r-- 1 ubuntu ubuntu 1597 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/direct_messenger/QueueStrategy.h 2024-03-27T21:42:44.115 INFO:teuthology.orchestra.run.smithi060.stdout: 416079 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub 2024-03-27T21:42:44.116 INFO:teuthology.orchestra.run.smithi060.stdout: 416097 4 -rw-r--r-- 1 ubuntu ubuntu 2667 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/TestMemRadosClient.h 2024-03-27T21:42:44.116 INFO:teuthology.orchestra.run.smithi060.stdout: 416090 12 -rw-r--r-- 1 ubuntu ubuntu 11565 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/TestIoCtxImpl.cc 2024-03-27T21:42:44.116 INFO:teuthology.orchestra.run.smithi060.stdout: 416080 4 -rw-r--r-- 1 ubuntu ubuntu 286 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/CMakeLists.txt 2024-03-27T21:42:44.116 INFO:teuthology.orchestra.run.smithi060.stdout: 416095 8 -rw-r--r-- 1 ubuntu ubuntu 4396 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/TestMemIoCtxImpl.h 2024-03-27T21:42:44.116 INFO:teuthology.orchestra.run.smithi060.stdout: 416088 4 -rw-r--r-- 1 ubuntu ubuntu 2063 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/TestClassHandler.h 2024-03-27T21:42:44.116 INFO:teuthology.orchestra.run.smithi060.stdout: 416081 52 -rw-r--r-- 1 ubuntu ubuntu 51065 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/LibradosTestStub.cc 2024-03-27T21:42:44.116 INFO:teuthology.orchestra.run.smithi060.stdout: 416101 8 -rw-r--r-- 1 ubuntu ubuntu 5364 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/TestWatchNotify.h 2024-03-27T21:42:44.116 INFO:teuthology.orchestra.run.smithi060.stdout: 416085 8 -rw-r--r-- 1 ubuntu ubuntu 4561 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/MockTestMemRadosClient.h 2024-03-27T21:42:44.117 INFO:teuthology.orchestra.run.smithi060.stdout: 416082 4 -rw-r--r-- 1 ubuntu ubuntu 1008 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/LibradosTestStub.h 2024-03-27T21:42:44.117 INFO:teuthology.orchestra.run.smithi060.stdout: 416094 28 -rw-r--r-- 1 ubuntu ubuntu 25213 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/TestMemIoCtxImpl.cc 2024-03-27T21:42:44.117 INFO:teuthology.orchestra.run.smithi060.stdout: 416089 4 -rw-r--r-- 1 ubuntu ubuntu 1565 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/TestCluster.h 2024-03-27T21:42:44.117 INFO:teuthology.orchestra.run.smithi060.stdout: 416096 4 -rw-r--r-- 1 ubuntu ubuntu 3360 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/TestMemRadosClient.cc 2024-03-27T21:42:44.117 INFO:teuthology.orchestra.run.smithi060.stdout: 416083 4 -rw-r--r-- 1 ubuntu ubuntu 984 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/MockTestMemCluster.h 2024-03-27T21:42:44.117 INFO:teuthology.orchestra.run.smithi060.stdout: 416099 8 -rw-r--r-- 1 ubuntu ubuntu 4728 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/TestRadosClient.h 2024-03-27T21:42:44.117 INFO:teuthology.orchestra.run.smithi060.stdout: 416086 20 -rw-r--r-- 1 ubuntu ubuntu 19985 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/NeoradosTestStub.cc 2024-03-27T21:42:44.117 INFO:teuthology.orchestra.run.smithi060.stdout: 416098 8 -rw-r--r-- 1 ubuntu ubuntu 8123 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/TestRadosClient.cc 2024-03-27T21:42:44.118 INFO:teuthology.orchestra.run.smithi060.stdout: 416084 12 -rw-r--r-- 1 ubuntu ubuntu 12014 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/MockTestMemIoCtxImpl.h 2024-03-27T21:42:44.118 INFO:teuthology.orchestra.run.smithi060.stdout: 416092 8 -rw-r--r-- 1 ubuntu ubuntu 5538 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/TestMemCluster.cc 2024-03-27T21:42:44.118 INFO:teuthology.orchestra.run.smithi060.stdout: 416093 4 -rw-r--r-- 1 ubuntu ubuntu 3468 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/TestMemCluster.h 2024-03-27T21:42:44.118 INFO:teuthology.orchestra.run.smithi060.stdout: 416091 12 -rw-r--r-- 1 ubuntu ubuntu 8907 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/TestIoCtxImpl.h 2024-03-27T21:42:44.118 INFO:teuthology.orchestra.run.smithi060.stdout: 416087 8 -rw-r--r-- 1 ubuntu ubuntu 4570 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/TestClassHandler.cc 2024-03-27T21:42:44.118 INFO:teuthology.orchestra.run.smithi060.stdout: 416100 20 -rw-r--r-- 1 ubuntu ubuntu 16552 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librados_test_stub/TestWatchNotify.cc 2024-03-27T21:42:44.118 INFO:teuthology.orchestra.run.smithi060.stdout: 415970 4 -rw-r--r-- 1 ubuntu ubuntu 3605 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/gprof-helper.c 2024-03-27T21:42:44.118 INFO:teuthology.orchestra.run.smithi060.stdout: 416641 4 -rw-r--r-- 1 ubuntu ubuntu 2870 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_mutate.cc 2024-03-27T21:42:44.119 INFO:teuthology.orchestra.run.smithi060.stdout: 415563 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_2pc_queue 2024-03-27T21:42:44.119 INFO:teuthology.orchestra.run.smithi060.stdout: 415564 4 -rw-r--r-- 1 ubuntu ubuntu 348 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_2pc_queue/CMakeLists.txt 2024-03-27T21:42:44.119 INFO:teuthology.orchestra.run.smithi060.stdout: 415565 40 -rw-r--r-- 1 ubuntu ubuntu 40846 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_2pc_queue/test_cls_2pc_queue.cc 2024-03-27T21:42:44.119 INFO:teuthology.orchestra.run.smithi060.stdout: 416649 4 -rw-r--r-- 1 ubuntu ubuntu 4039 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_rbd_replay.cc 2024-03-27T21:42:44.119 INFO:teuthology.orchestra.run.smithi060.stdout: 415612 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common 2024-03-27T21:42:44.119 INFO:teuthology.orchestra.run.smithi060.stdout: 415770 4 -rw-r--r-- 1 ubuntu ubuntu 1907 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_hostname.cc 2024-03-27T21:42:44.119 INFO:teuthology.orchestra.run.smithi060.stdout: 415790 8 -rw-r--r-- 1 ubuntu ubuntu 6051 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_random.cc 2024-03-27T21:42:44.119 INFO:teuthology.orchestra.run.smithi060.stdout: 415756 8 -rw-r--r-- 1 ubuntu ubuntu 6692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blocked_completion.cc 2024-03-27T21:42:44.120 INFO:teuthology.orchestra.run.smithi060.stdout: 415646 4 -rw-r--r-- 1 ubuntu ubuntu 2523 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_allocate_unique.cc 2024-03-27T21:42:44.120 INFO:teuthology.orchestra.run.smithi060.stdout: 415794 16 -rw-r--r-- 1 ubuntu ubuntu 13996 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_shunique_lock.cc 2024-03-27T21:42:44.120 INFO:teuthology.orchestra.run.smithi060.stdout: 415789 4 -rw-r--r-- 1 ubuntu ubuntu 3706 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_rabin_chunk.cc 2024-03-27T21:42:44.120 INFO:teuthology.orchestra.run.smithi060.stdout: 415779 4 -rw-r--r-- 1 ubuntu ubuntu 4085 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_lru.cc 2024-03-27T21:42:44.120 INFO:teuthology.orchestra.run.smithi060.stdout: 415615 4 -rw-r--r-- 1 ubuntu ubuntu 2837 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/ObjectContents.h 2024-03-27T21:42:44.120 INFO:teuthology.orchestra.run.smithi060.stdout: 415775 4 -rw-r--r-- 1 ubuntu ubuntu 1007 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_journald_logger.cc 2024-03-27T21:42:44.120 INFO:teuthology.orchestra.run.smithi060.stdout: 415798 4 -rw-r--r-- 1 ubuntu ubuntu 2485 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_str_map.cc 2024-03-27T21:42:44.120 INFO:teuthology.orchestra.run.smithi060.stdout: 415761 12 -rw-r--r-- 1 ubuntu ubuntu 10802 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_config.cc 2024-03-27T21:42:44.120 INFO:teuthology.orchestra.run.smithi060.stdout: 415613 12 -rw-r--r-- 1 ubuntu ubuntu 11554 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/CMakeLists.txt 2024-03-27T21:42:44.121 INFO:teuthology.orchestra.run.smithi060.stdout: 415804 8 -rw-r--r-- 1 ubuntu ubuntu 7795 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_weighted_priority_queue.cc 2024-03-27T21:42:44.121 INFO:teuthology.orchestra.run.smithi060.stdout: 415642 8 -rw-r--r-- 1 ubuntu ubuntu 5042 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/dns_messages.h 2024-03-27T21:42:44.121 INFO:teuthology.orchestra.run.smithi060.stdout: 415778 4 -rw-r--r-- 1 ubuntu ubuntu 1749 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_lockdep.cc 2024-03-27T21:42:44.121 INFO:teuthology.orchestra.run.smithi060.stdout: 415759 8 -rw-r--r-- 1 ubuntu ubuntu 4405 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_cdc.cc 2024-03-27T21:42:44.121 INFO:teuthology.orchestra.run.smithi060.stdout: 415795 4 -rw-r--r-- 1 ubuntu ubuntu 2761 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_sloppy_crc_map.cc 2024-03-27T21:42:44.121 INFO:teuthology.orchestra.run.smithi060.stdout: 415650 12 -rw-r--r-- 1 ubuntu ubuntu 9607 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_bit_vector.cc 2024-03-27T21:42:44.121 INFO:teuthology.orchestra.run.smithi060.stdout: 415648 12 -rw-r--r-- 1 ubuntu ubuntu 10824 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_async_shared_mutex.cc 2024-03-27T21:42:44.121 INFO:teuthology.orchestra.run.smithi060.stdout: 415802 4 -rw-r--r-- 1 ubuntu ubuntu 1036 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_util.cc 2024-03-27T21:42:44.122 INFO:teuthology.orchestra.run.smithi060.stdout: 415784 4 -rw-r--r-- 1 ubuntu ubuntu 1707 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_option.cc 2024-03-27T21:42:44.122 INFO:teuthology.orchestra.run.smithi060.stdout: 415781 4 -rw-r--r-- 1 ubuntu ubuntu 2114 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_memory.cc 2024-03-27T21:42:44.122 INFO:teuthology.orchestra.run.smithi060.stdout: 415614 4 -rw-r--r-- 1 ubuntu ubuntu 3150 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/ObjectContents.cc 2024-03-27T21:42:44.122 INFO:teuthology.orchestra.run.smithi060.stdout: 415782 4 -rw-r--r-- 1 ubuntu ubuntu 2399 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_mutex_debug.cc 2024-03-27T21:42:44.122 INFO:teuthology.orchestra.run.smithi060.stdout: 415765 20 -rw-r--r-- 1 ubuntu ubuntu 19624 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_crc32c.cc 2024-03-27T21:42:44.122 INFO:teuthology.orchestra.run.smithi060.stdout: 415651 4 -rw-r--r-- 1 ubuntu ubuntu 2842 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev.cc 2024-03-27T21:42:44.122 INFO:teuthology.orchestra.run.smithi060.stdout: 415786 12 -rw-r--r-- 1 ubuntu ubuntu 8897 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_perf_histogram.cc 2024-03-27T21:42:44.122 INFO:teuthology.orchestra.run.smithi060.stdout: 415649 4 -rw-r--r-- 1 ubuntu ubuntu 1219 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_back_trace.cc 2024-03-27T21:42:44.123 INFO:teuthology.orchestra.run.smithi060.stdout: 415801 4 -rw-r--r-- 1 ubuntu ubuntu 1134 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_url_escape.cc 2024-03-27T21:42:44.123 INFO:teuthology.orchestra.run.smithi060.stdout: 415788 8 -rw-r--r-- 1 ubuntu ubuntu 6136 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_prioritized_queue.cc 2024-03-27T21:42:44.123 INFO:teuthology.orchestra.run.smithi060.stdout: 415777 8 -rw-r--r-- 1 ubuntu ubuntu 4218 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_json_formatter.cc 2024-03-27T21:42:44.123 INFO:teuthology.orchestra.run.smithi060.stdout: 415800 8 -rw-r--r-- 1 ubuntu ubuntu 7094 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_time.cc 2024-03-27T21:42:44.123 INFO:teuthology.orchestra.run.smithi060.stdout: 415645 4 -rw-r--r-- 1 ubuntu ubuntu 2478 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/histogram.cc 2024-03-27T21:42:44.123 INFO:teuthology.orchestra.run.smithi060.stdout: 415787 4 -rw-r--r-- 1 ubuntu ubuntu 2046 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_pretty_binary.cc 2024-03-27T21:42:44.123 INFO:teuthology.orchestra.run.smithi060.stdout: 415652 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block 2024-03-27T21:42:44.123 INFO:teuthology.orchestra.run.smithi060.stdout: 415653 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys 2024-03-27T21:42:44.124 INFO:teuthology.orchestra.run.smithi060.stdout: 415654 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block 2024-03-27T21:42:44.124 INFO:teuthology.orchestra.run.smithi060.stdout: 415655 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1 2024-03-27T21:42:44.124 INFO:teuthology.orchestra.run.smithi060.stdout: 415656 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue 2024-03-27T21:42:44.124 INFO:teuthology.orchestra.run.smithi060.stdout: 415662 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/iosched 2024-03-27T21:42:44.124 INFO:teuthology.orchestra.run.smithi060.stdout: 415666 4 -rw-r--r-- 1 ubuntu ubuntu 5 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/iosched/write_expire 2024-03-27T21:42:44.124 INFO:teuthology.orchestra.run.smithi060.stdout: 415664 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/iosched/front_merges 2024-03-27T21:42:44.124 INFO:teuthology.orchestra.run.smithi060.stdout: 415663 4 -rw-r--r-- 1 ubuntu ubuntu 3 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/iosched/fifo_batch 2024-03-27T21:42:44.125 INFO:teuthology.orchestra.run.smithi060.stdout: 415667 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/iosched/writes_starved 2024-03-27T21:42:44.125 INFO:teuthology.orchestra.run.smithi060.stdout: 415665 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/iosched/read_expire 2024-03-27T21:42:44.125 INFO:teuthology.orchestra.run.smithi060.stdout: 415672 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/max_sectors_kb 2024-03-27T21:42:44.125 INFO:teuthology.orchestra.run.smithi060.stdout: 415659 4 -rw-r--r-- 1 ubuntu ubuntu 11 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/discard_max_bytes 2024-03-27T21:42:44.125 INFO:teuthology.orchestra.run.smithi060.stdout: 415678 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/optimal_io_size 2024-03-27T21:42:44.125 INFO:teuthology.orchestra.run.smithi060.stdout: 415673 4 -rw-r--r-- 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/max_segment_size 2024-03-27T21:42:44.125 INFO:teuthology.orchestra.run.smithi060.stdout: 415658 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/discard_granularity 2024-03-27T21:42:44.125 INFO:teuthology.orchestra.run.smithi060.stdout: 415684 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/write_same_max_bytes 2024-03-27T21:42:44.125 INFO:teuthology.orchestra.run.smithi060.stdout: 415657 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/add_random 2024-03-27T21:42:44.126 INFO:teuthology.orchestra.run.smithi060.stdout: 415660 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/discard_zeroes_data 2024-03-27T21:42:44.126 INFO:teuthology.orchestra.run.smithi060.stdout: 415680 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/read_ahead_kb 2024-03-27T21:42:44.126 INFO:teuthology.orchestra.run.smithi060.stdout: 415683 4 -rw-r--r-- 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/scheduler 2024-03-27T21:42:44.126 INFO:teuthology.orchestra.run.smithi060.stdout: 415670 4 -rw-r--r-- 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/max_hw_sectors_kb 2024-03-27T21:42:44.126 INFO:teuthology.orchestra.run.smithi060.stdout: 415681 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/rotational 2024-03-27T21:42:44.126 INFO:teuthology.orchestra.run.smithi060.stdout: 415668 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/iostats 2024-03-27T21:42:44.126 INFO:teuthology.orchestra.run.smithi060.stdout: 415682 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/rq_affinity 2024-03-27T21:42:44.127 INFO:teuthology.orchestra.run.smithi060.stdout: 415674 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/max_segments 2024-03-27T21:42:44.127 INFO:teuthology.orchestra.run.smithi060.stdout: 415676 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/nomerges 2024-03-27T21:42:44.127 INFO:teuthology.orchestra.run.smithi060.stdout: 415677 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/nr_requests 2024-03-27T21:42:44.127 INFO:teuthology.orchestra.run.smithi060.stdout: 415671 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/max_integrity_segments 2024-03-27T21:42:44.127 INFO:teuthology.orchestra.run.smithi060.stdout: 415661 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/hw_sector_size 2024-03-27T21:42:44.127 INFO:teuthology.orchestra.run.smithi060.stdout: 415675 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/minimum_io_size 2024-03-27T21:42:44.127 INFO:teuthology.orchestra.run.smithi060.stdout: 415679 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/physical_block_size 2024-03-27T21:42:44.127 INFO:teuthology.orchestra.run.smithi060.stdout: 415669 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/cciss!c0d1/queue/logical_block_size 2024-03-27T21:42:44.127 INFO:teuthology.orchestra.run.smithi060.stdout: 415720 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb 2024-03-27T21:42:44.128 INFO:teuthology.orchestra.run.smithi060.stdout: 415723 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/device 2024-03-27T21:42:44.128 INFO:teuthology.orchestra.run.smithi060.stdout: 415724 4 -rw-r--r-- 1 ubuntu ubuntu 13 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/device/model 2024-03-27T21:42:44.128 INFO:teuthology.orchestra.run.smithi060.stdout: 415722 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/dev 2024-03-27T21:42:44.128 INFO:teuthology.orchestra.run.smithi060.stdout: 415726 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue 2024-03-27T21:42:44.128 INFO:teuthology.orchestra.run.smithi060.stdout: 415732 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/iosched 2024-03-27T21:42:44.128 INFO:teuthology.orchestra.run.smithi060.stdout: 415736 4 -rw-r--r-- 1 ubuntu ubuntu 5 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/iosched/write_expire 2024-03-27T21:42:44.128 INFO:teuthology.orchestra.run.smithi060.stdout: 415734 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/iosched/front_merges 2024-03-27T21:42:44.128 INFO:teuthology.orchestra.run.smithi060.stdout: 415733 4 -rw-r--r-- 1 ubuntu ubuntu 3 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/iosched/fifo_batch 2024-03-27T21:42:44.129 INFO:teuthology.orchestra.run.smithi060.stdout: 415737 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/iosched/writes_starved 2024-03-27T21:42:44.129 INFO:teuthology.orchestra.run.smithi060.stdout: 415735 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/iosched/read_expire 2024-03-27T21:42:44.129 INFO:teuthology.orchestra.run.smithi060.stdout: 415742 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/max_sectors_kb 2024-03-27T21:42:44.129 INFO:teuthology.orchestra.run.smithi060.stdout: 415729 4 -rw-r--r-- 1 ubuntu ubuntu 11 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/discard_max_bytes 2024-03-27T21:42:44.129 INFO:teuthology.orchestra.run.smithi060.stdout: 415748 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/optimal_io_size 2024-03-27T21:42:44.129 INFO:teuthology.orchestra.run.smithi060.stdout: 415743 4 -rw-r--r-- 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/max_segment_size 2024-03-27T21:42:44.129 INFO:teuthology.orchestra.run.smithi060.stdout: 415728 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/discard_granularity 2024-03-27T21:42:44.130 INFO:teuthology.orchestra.run.smithi060.stdout: 415755 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/write_same_max_bytes 2024-03-27T21:42:44.130 INFO:teuthology.orchestra.run.smithi060.stdout: 415727 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/add_random 2024-03-27T21:42:44.130 INFO:teuthology.orchestra.run.smithi060.stdout: 415730 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/discard_zeroes_data 2024-03-27T21:42:44.130 INFO:teuthology.orchestra.run.smithi060.stdout: 415751 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/read_ahead_kb 2024-03-27T21:42:44.130 INFO:teuthology.orchestra.run.smithi060.stdout: 415754 4 -rw-r--r-- 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/scheduler 2024-03-27T21:42:44.130 INFO:teuthology.orchestra.run.smithi060.stdout: 415740 4 -rw-r--r-- 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/max_hw_sectors_kb 2024-03-27T21:42:44.130 INFO:teuthology.orchestra.run.smithi060.stdout: 415750 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/queue 2024-03-27T21:42:44.130 INFO:teuthology.orchestra.run.smithi060.stdout: 415752 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/rotational 2024-03-27T21:42:44.131 INFO:teuthology.orchestra.run.smithi060.stdout: 415738 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/iostats 2024-03-27T21:42:44.131 INFO:teuthology.orchestra.run.smithi060.stdout: 415753 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/rq_affinity 2024-03-27T21:42:44.131 INFO:teuthology.orchestra.run.smithi060.stdout: 415744 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/max_segments 2024-03-27T21:42:44.131 INFO:teuthology.orchestra.run.smithi060.stdout: 415746 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/nomerges 2024-03-27T21:42:44.131 INFO:teuthology.orchestra.run.smithi060.stdout: 415747 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/nr_requests 2024-03-27T21:42:44.131 INFO:teuthology.orchestra.run.smithi060.stdout: 415741 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/max_integrity_segments 2024-03-27T21:42:44.131 INFO:teuthology.orchestra.run.smithi060.stdout: 415731 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/hw_sector_size 2024-03-27T21:42:44.131 INFO:teuthology.orchestra.run.smithi060.stdout: 415745 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/minimum_io_size 2024-03-27T21:42:44.131 INFO:teuthology.orchestra.run.smithi060.stdout: 415749 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/physical_block_size 2024-03-27T21:42:44.131 INFO:teuthology.orchestra.run.smithi060.stdout: 415739 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/queue/logical_block_size 2024-03-27T21:42:44.132 INFO:teuthology.orchestra.run.smithi060.stdout: 415725 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/foo 2024-03-27T21:42:44.132 INFO:teuthology.orchestra.run.smithi060.stdout: 415721 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sdb/bar 2024-03-27T21:42:44.132 INFO:teuthology.orchestra.run.smithi060.stdout: 415685 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda 2024-03-27T21:42:44.132 INFO:teuthology.orchestra.run.smithi060.stdout: 415688 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/device 2024-03-27T21:42:44.132 INFO:teuthology.orchestra.run.smithi060.stdout: 415689 4 -rw-r--r-- 1 ubuntu ubuntu 13 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/device/model 2024-03-27T21:42:44.132 INFO:teuthology.orchestra.run.smithi060.stdout: 415687 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/dev 2024-03-27T21:42:44.132 INFO:teuthology.orchestra.run.smithi060.stdout: 415691 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue 2024-03-27T21:42:44.132 INFO:teuthology.orchestra.run.smithi060.stdout: 415697 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/iosched 2024-03-27T21:42:44.133 INFO:teuthology.orchestra.run.smithi060.stdout: 415701 4 -rw-r--r-- 1 ubuntu ubuntu 5 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/iosched/write_expire 2024-03-27T21:42:44.133 INFO:teuthology.orchestra.run.smithi060.stdout: 415699 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/iosched/front_merges 2024-03-27T21:42:44.133 INFO:teuthology.orchestra.run.smithi060.stdout: 415698 4 -rw-r--r-- 1 ubuntu ubuntu 3 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/iosched/fifo_batch 2024-03-27T21:42:44.133 INFO:teuthology.orchestra.run.smithi060.stdout: 415702 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/iosched/writes_starved 2024-03-27T21:42:44.133 INFO:teuthology.orchestra.run.smithi060.stdout: 415700 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/iosched/read_expire 2024-03-27T21:42:44.133 INFO:teuthology.orchestra.run.smithi060.stdout: 415707 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/max_sectors_kb 2024-03-27T21:42:44.133 INFO:teuthology.orchestra.run.smithi060.stdout: 415694 4 -rw-r--r-- 1 ubuntu ubuntu 11 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/discard_max_bytes 2024-03-27T21:42:44.133 INFO:teuthology.orchestra.run.smithi060.stdout: 415713 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/optimal_io_size 2024-03-27T21:42:44.133 INFO:teuthology.orchestra.run.smithi060.stdout: 415708 4 -rw-r--r-- 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/max_segment_size 2024-03-27T21:42:44.134 INFO:teuthology.orchestra.run.smithi060.stdout: 415693 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/discard_granularity 2024-03-27T21:42:44.134 INFO:teuthology.orchestra.run.smithi060.stdout: 415719 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/write_same_max_bytes 2024-03-27T21:42:44.134 INFO:teuthology.orchestra.run.smithi060.stdout: 415692 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/add_random 2024-03-27T21:42:44.134 INFO:teuthology.orchestra.run.smithi060.stdout: 415695 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/discard_zeroes_data 2024-03-27T21:42:44.134 INFO:teuthology.orchestra.run.smithi060.stdout: 415715 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/read_ahead_kb 2024-03-27T21:42:44.134 INFO:teuthology.orchestra.run.smithi060.stdout: 415718 4 -rw-r--r-- 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/scheduler 2024-03-27T21:42:44.134 INFO:teuthology.orchestra.run.smithi060.stdout: 415705 4 -rw-r--r-- 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/max_hw_sectors_kb 2024-03-27T21:42:44.134 INFO:teuthology.orchestra.run.smithi060.stdout: 415716 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/rotational 2024-03-27T21:42:44.135 INFO:teuthology.orchestra.run.smithi060.stdout: 415703 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/iostats 2024-03-27T21:42:44.135 INFO:teuthology.orchestra.run.smithi060.stdout: 415717 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/rq_affinity 2024-03-27T21:42:44.135 INFO:teuthology.orchestra.run.smithi060.stdout: 415709 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/max_segments 2024-03-27T21:42:44.135 INFO:teuthology.orchestra.run.smithi060.stdout: 415711 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/nomerges 2024-03-27T21:42:44.135 INFO:teuthology.orchestra.run.smithi060.stdout: 415712 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/nr_requests 2024-03-27T21:42:44.135 INFO:teuthology.orchestra.run.smithi060.stdout: 415706 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/max_integrity_segments 2024-03-27T21:42:44.135 INFO:teuthology.orchestra.run.smithi060.stdout: 415696 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/hw_sector_size 2024-03-27T21:42:44.135 INFO:teuthology.orchestra.run.smithi060.stdout: 415710 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/minimum_io_size 2024-03-27T21:42:44.136 INFO:teuthology.orchestra.run.smithi060.stdout: 415714 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/physical_block_size 2024-03-27T21:42:44.136 INFO:teuthology.orchestra.run.smithi060.stdout: 415704 4 -rw-r--r-- 1 ubuntu ubuntu 4 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/queue/logical_block_size 2024-03-27T21:42:44.136 INFO:teuthology.orchestra.run.smithi060.stdout: 415690 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/foo 2024-03-27T21:42:44.136 INFO:teuthology.orchestra.run.smithi060.stdout: 415686 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_blkdev_sys_block/sys/block/sda/bar 2024-03-27T21:42:44.136 INFO:teuthology.orchestra.run.smithi060.stdout: 415776 12 -rw-r--r-- 1 ubuntu ubuntu 10578 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_json_formattable.cc 2024-03-27T21:42:44.136 INFO:teuthology.orchestra.run.smithi060.stdout: 415757 12 -rw-r--r-- 1 ubuntu ubuntu 8924 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_bloom_filter.cc 2024-03-27T21:42:44.136 INFO:teuthology.orchestra.run.smithi060.stdout: 415647 8 -rw-r--r-- 1 ubuntu ubuntu 6731 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_async_completion.cc 2024-03-27T21:42:44.136 INFO:teuthology.orchestra.run.smithi060.stdout: 415616 8 -rw-r--r-- 1 ubuntu ubuntu 5616 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/Readahead.cc 2024-03-27T21:42:44.137 INFO:teuthology.orchestra.run.smithi060.stdout: 415617 12 -rw-r--r-- 1 ubuntu ubuntu 9135 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/Throttle.cc 2024-03-27T21:42:44.137 INFO:teuthology.orchestra.run.smithi060.stdout: 415791 4 -rw-r--r-- 1 ubuntu ubuntu 961 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_safe_io.cc 2024-03-27T21:42:44.137 INFO:teuthology.orchestra.run.smithi060.stdout: 415763 4 -rw-r--r-- 1 ubuntu ubuntu 1756 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_convenience.cc 2024-03-27T21:42:44.137 INFO:teuthology.orchestra.run.smithi060.stdout: 415771 8 -rw-r--r-- 1 ubuntu ubuntu 8143 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_interval_map.cc 2024-03-27T21:42:44.137 INFO:teuthology.orchestra.run.smithi060.stdout: 415767 8 -rw-r--r-- 1 ubuntu ubuntu 6479 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_fault_injector.cc 2024-03-27T21:42:44.137 INFO:teuthology.orchestra.run.smithi060.stdout: 415618 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples 2024-03-27T21:42:44.137 INFO:teuthology.orchestra.run.smithi060.stdout: 415623 4 -rw-r--r-- 1 ubuntu ubuntu 1180 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/erwan.v1.sda 2024-03-27T21:42:44.137 INFO:teuthology.orchestra.run.smithi060.stdout: 415639 4 -rw-r--r-- 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/reesi001.nvme0n1.devid 2024-03-27T21:42:44.138 INFO:teuthology.orchestra.run.smithi060.stdout: 415632 4 -rw-r--r-- 1 ubuntu ubuntu 911 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/mira055.sda 2024-03-27T21:42:44.138 INFO:teuthology.orchestra.run.smithi060.stdout: 415627 4 -rw-r--r-- 1 ubuntu ubuntu 1201 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/erwan1 2024-03-27T21:42:44.138 INFO:teuthology.orchestra.run.smithi060.stdout: 415633 4 -rw-r--r-- 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/mira055.sda.devid 2024-03-27T21:42:44.138 INFO:teuthology.orchestra.run.smithi060.stdout: 415622 4 -rw-r--r-- 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/cpach.sdn.devid 2024-03-27T21:42:44.138 INFO:teuthology.orchestra.run.smithi060.stdout: 415626 4 -rw-r--r-- 1 ubuntu ubuntu 51 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/erwan.v1.sdb.devid 2024-03-27T21:42:44.138 INFO:teuthology.orchestra.run.smithi060.stdout: 415635 4 -rw-r--r-- 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/mira055.sdb.devid 2024-03-27T21:42:44.138 INFO:teuthology.orchestra.run.smithi060.stdout: 415641 4 -rw-r--r-- 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/stud.nvme0n1.devid 2024-03-27T21:42:44.138 INFO:teuthology.orchestra.run.smithi060.stdout: 415621 4 -rw-r--r-- 1 ubuntu ubuntu 2344 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/cpach.sdn 2024-03-27T21:42:44.139 INFO:teuthology.orchestra.run.smithi060.stdout: 415624 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/erwan.v1.sda.devid 2024-03-27T21:42:44.139 INFO:teuthology.orchestra.run.smithi060.stdout: 415631 4 -rw-r--r-- 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/gnit.sda.devid 2024-03-27T21:42:44.139 INFO:teuthology.orchestra.run.smithi060.stdout: 415634 4 -rw-r--r-- 1 ubuntu ubuntu 946 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/mira055.sdb 2024-03-27T21:42:44.139 INFO:teuthology.orchestra.run.smithi060.stdout: 415630 4 -rw-r--r-- 1 ubuntu ubuntu 1704 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/gnit.sda 2024-03-27T21:42:44.139 INFO:teuthology.orchestra.run.smithi060.stdout: 415636 4 -rw-r--r-- 1 ubuntu ubuntu 947 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/mira055.sde 2024-03-27T21:42:44.139 INFO:teuthology.orchestra.run.smithi060.stdout: 415625 4 -rw-r--r-- 1 ubuntu ubuntu 1345 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/erwan.v1.sdb 2024-03-27T21:42:44.139 INFO:teuthology.orchestra.run.smithi060.stdout: 415638 4 -rw-r--r-- 1 ubuntu ubuntu 561 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/reesi001.nvme0n1 2024-03-27T21:42:44.139 INFO:teuthology.orchestra.run.smithi060.stdout: 415619 4 -rw-r--r-- 1 ubuntu ubuntu 677 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/autriche.nvme0n1 2024-03-27T21:42:44.139 INFO:teuthology.orchestra.run.smithi060.stdout: 415620 4 -rw-r--r-- 1 ubuntu ubuntu 42 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/autriche.nvme0n1.devid 2024-03-27T21:42:44.140 INFO:teuthology.orchestra.run.smithi060.stdout: 415629 4 -rw-r--r-- 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/gnit.nvme0n1.devid 2024-03-27T21:42:44.140 INFO:teuthology.orchestra.run.smithi060.stdout: 415628 4 -rw-r--r-- 1 ubuntu ubuntu 1285 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/gnit.nvme0n1 2024-03-27T21:42:44.140 INFO:teuthology.orchestra.run.smithi060.stdout: 415640 4 -rw-r--r-- 1 ubuntu ubuntu 1480 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/stud.nvme0n1 2024-03-27T21:42:44.140 INFO:teuthology.orchestra.run.smithi060.stdout: 415637 4 -rw-r--r-- 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/blkdev-udevadm-info-samples/mira055.sde.devid 2024-03-27T21:42:44.140 INFO:teuthology.orchestra.run.smithi060.stdout: 415773 8 -rw-r--r-- 1 ubuntu ubuntu 6407 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_intrusive_lru.cc 2024-03-27T21:42:44.140 INFO:teuthology.orchestra.run.smithi060.stdout: 415772 20 -rw-r--r-- 1 ubuntu ubuntu 17188 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_interval_set.cc 2024-03-27T21:42:44.140 INFO:teuthology.orchestra.run.smithi060.stdout: 415766 4 -rw-r--r-- 1 ubuntu ubuntu 2243 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_fair_mutex.cc 2024-03-27T21:42:44.140 INFO:teuthology.orchestra.run.smithi060.stdout: 415793 12 -rw-r--r-- 1 ubuntu ubuntu 9518 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_sharedptr_registry.cc 2024-03-27T21:42:44.141 INFO:teuthology.orchestra.run.smithi060.stdout: 415760 8 -rw-r--r-- 1 ubuntu ubuntu 5180 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_ceph_timer.cc 2024-03-27T21:42:44.141 INFO:teuthology.orchestra.run.smithi060.stdout: 415805 8 -rw-r--r-- 1 ubuntu ubuntu 4221 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_xmlformatter.cc 2024-03-27T21:42:44.141 INFO:teuthology.orchestra.run.smithi060.stdout: 415644 4 -rw-r--r-- 1 ubuntu ubuntu 4018 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/get_command_descriptions.cc 2024-03-27T21:42:44.141 INFO:teuthology.orchestra.run.smithi060.stdout: 415769 12 -rw-r--r-- 1 ubuntu ubuntu 8971 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_hobject.cc 2024-03-27T21:42:44.141 INFO:teuthology.orchestra.run.smithi060.stdout: 415792 12 -rw-r--r-- 1 ubuntu ubuntu 10749 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_shared_cache.cc 2024-03-27T21:42:44.141 INFO:teuthology.orchestra.run.smithi060.stdout: 415785 4 -rw-r--r-- 1 ubuntu ubuntu 3984 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_perf_counters_key.cc 2024-03-27T21:42:44.141 INFO:teuthology.orchestra.run.smithi060.stdout: 415780 4 -rw-r--r-- 1 ubuntu ubuntu 1998 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_lruset.cc 2024-03-27T21:42:44.141 INFO:teuthology.orchestra.run.smithi060.stdout: 415783 4 -rw-r--r-- 1 ubuntu ubuntu 1942 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_numa.cc 2024-03-27T21:42:44.142 INFO:teuthology.orchestra.run.smithi060.stdout: 415803 8 -rw-r--r-- 1 ubuntu ubuntu 7337 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_versioned_variant.cc 2024-03-27T21:42:44.142 INFO:teuthology.orchestra.run.smithi060.stdout: 415764 4 -rw-r--r-- 1 ubuntu ubuntu 1001 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_counter.cc 2024-03-27T21:42:44.142 INFO:teuthology.orchestra.run.smithi060.stdout: 415796 4 -rw-r--r-- 1 ubuntu ubuntu 2923 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_split.cc 2024-03-27T21:42:44.142 INFO:teuthology.orchestra.run.smithi060.stdout: 415758 8 -rw-r--r-- 1 ubuntu ubuntu 6744 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_bounded_key_counter.cc 2024-03-27T21:42:44.142 INFO:teuthology.orchestra.run.smithi060.stdout: 415768 4 -rw-r--r-- 1 ubuntu ubuntu 863 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_global_doublefree.cc 2024-03-27T21:42:44.142 INFO:teuthology.orchestra.run.smithi060.stdout: 415762 8 -rw-r--r-- 1 ubuntu ubuntu 4944 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_context.cc 2024-03-27T21:42:44.142 INFO:teuthology.orchestra.run.smithi060.stdout: 415799 8 -rw-r--r-- 1 ubuntu ubuntu 7420 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_tableformatter.cc 2024-03-27T21:42:44.142 INFO:teuthology.orchestra.run.smithi060.stdout: 415774 4 -rw-r--r-- 1 ubuntu ubuntu 1913 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_iso_8601.cc 2024-03-27T21:42:44.143 INFO:teuthology.orchestra.run.smithi060.stdout: 415797 8 -rw-r--r-- 1 ubuntu ubuntu 5651 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/test_static_ptr.cc 2024-03-27T21:42:44.143 INFO:teuthology.orchestra.run.smithi060.stdout: 415643 12 -rw-r--r-- 1 ubuntu ubuntu 8965 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/common/dns_resolve.cc 2024-03-27T21:42:44.143 INFO:teuthology.orchestra.run.smithi060.stdout: 416587 4 -rwxr-xr-x 1 ubuntu ubuntu 1699 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/run-cli-tests 2024-03-27T21:42:44.143 INFO:teuthology.orchestra.run.smithi060.stdout: 416289 12 -rw-r--r-- 1 ubuntu ubuntu 10586 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/librgw_file_xattr.cc 2024-03-27T21:42:44.143 INFO:teuthology.orchestra.run.smithi060.stdout: 415581 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_log 2024-03-27T21:42:44.143 INFO:teuthology.orchestra.run.smithi060.stdout: 415582 4 -rw-r--r-- 1 ubuntu ubuntu 315 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_log/CMakeLists.txt 2024-03-27T21:42:44.143 INFO:teuthology.orchestra.run.smithi060.stdout: 415583 12 -rw-r--r-- 1 ubuntu ubuntu 10002 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_log/test_cls_log.cc 2024-03-27T21:42:44.143 INFO:teuthology.orchestra.run.smithi060.stdout: 415575 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_journal 2024-03-27T21:42:44.144 INFO:teuthology.orchestra.run.smithi060.stdout: 415576 4 -rw-r--r-- 1 ubuntu ubuntu 375 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_journal/CMakeLists.txt 2024-03-27T21:42:44.144 INFO:teuthology.orchestra.run.smithi060.stdout: 415577 24 -rw-r--r-- 1 ubuntu ubuntu 22225 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_journal/test_cls_journal.cc 2024-03-27T21:42:44.144 INFO:teuthology.orchestra.run.smithi060.stdout: 415569 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_cmpomap 2024-03-27T21:42:44.144 INFO:teuthology.orchestra.run.smithi060.stdout: 415571 28 -rw-r--r-- 1 ubuntu ubuntu 27120 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_cmpomap/test_cls_cmpomap.cc 2024-03-27T21:42:44.144 INFO:teuthology.orchestra.run.smithi060.stdout: 415570 4 -rw-r--r-- 1 ubuntu ubuntu 237 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/cls_cmpomap/CMakeLists.txt 2024-03-27T21:42:44.144 INFO:teuthology.orchestra.run.smithi060.stdout: 416652 32 -rw-r--r-- 1 ubuntu ubuntu 28960 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_rgw_admin_meta.cc 2024-03-27T21:42:44.144 INFO:teuthology.orchestra.run.smithi060.stdout: 416625 4 -rwxr-xr-x 1 ubuntu ubuntu 255 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_backfill.sh 2024-03-27T21:42:44.144 INFO:teuthology.orchestra.run.smithi060.stdout: 416627 4 -rw-r--r-- 1 ubuntu ubuntu 563 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_c_headers.c 2024-03-27T21:42:44.145 INFO:teuthology.orchestra.run.smithi060.stdout: 416590 4 -rwxr-xr-x 1 ubuntu ubuntu 473 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/run-rbd-unit-tests.sh 2024-03-27T21:42:44.145 INFO:teuthology.orchestra.run.smithi060.stdout: 416643 8 -rw-r--r-- 1 ubuntu ubuntu 6808 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_pageset.cc 2024-03-27T21:42:44.145 INFO:teuthology.orchestra.run.smithi060.stdout: 416623 4 -rw-r--r-- 1 ubuntu ubuntu 2353 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/test/test_arch.cc 2024-03-27T21:42:44.145 INFO:teuthology.orchestra.run.smithi060.stdout: 410612 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/googletest 2024-03-27T21:42:44.145 INFO:teuthology.orchestra.run.smithi060.stdout: 415333 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/spawn 2024-03-27T21:42:44.145 INFO:teuthology.orchestra.run.smithi060.stdout: 410601 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/global 2024-03-27T21:42:44.145 INFO:teuthology.orchestra.run.smithi060.stdout: 410609 20 -rw-r--r-- 1 ubuntu ubuntu 19144 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/global/signal_handler.cc 2024-03-27T21:42:44.145 INFO:teuthology.orchestra.run.smithi060.stdout: 410604 4 -rw-r--r-- 1 ubuntu ubuntu 1395 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/global/global_context.h 2024-03-27T21:42:44.145 INFO:teuthology.orchestra.run.smithi060.stdout: 410602 4 -rw-r--r-- 1 ubuntu ubuntu 519 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/global/CMakeLists.txt 2024-03-27T21:42:44.146 INFO:teuthology.orchestra.run.smithi060.stdout: 410605 20 -rw-r--r-- 1 ubuntu ubuntu 18241 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/global/global_init.cc 2024-03-27T21:42:44.146 INFO:teuthology.orchestra.run.smithi060.stdout: 410606 4 -rw-r--r-- 1 ubuntu ubuntu 2952 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/global/global_init.h 2024-03-27T21:42:44.146 INFO:teuthology.orchestra.run.smithi060.stdout: 410607 8 -rw-r--r-- 1 ubuntu ubuntu 5427 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/global/pidfile.cc 2024-03-27T21:42:44.146 INFO:teuthology.orchestra.run.smithi060.stdout: 410610 4 -rw-r--r-- 1 ubuntu ubuntu 1758 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/global/signal_handler.h 2024-03-27T21:42:44.146 INFO:teuthology.orchestra.run.smithi060.stdout: 410603 4 -rw-r--r-- 1 ubuntu ubuntu 2549 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/global/global_context.cc 2024-03-27T21:42:44.146 INFO:teuthology.orchestra.run.smithi060.stdout: 410608 4 -rw-r--r-- 1 ubuntu ubuntu 808 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/global/pidfile.h 2024-03-27T21:42:44.146 INFO:teuthology.orchestra.run.smithi060.stdout: 410611 4 -rw-r--r-- 1 ubuntu ubuntu 1210 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/global/signal_handler_win32.cc 2024-03-27T21:42:44.146 INFO:teuthology.orchestra.run.smithi060.stdout: 410866 68 -rw-r--r-- 1 ubuntu ubuntu 65791 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/libcephfs.cc 2024-03-27T21:42:44.147 INFO:teuthology.orchestra.run.smithi060.stdout: 411868 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mount 2024-03-27T21:42:44.147 INFO:teuthology.orchestra.run.smithi060.stdout: 411869 4 -rw-r--r-- 1 ubuntu ubuntu 412 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mount/CMakeLists.txt 2024-03-27T21:42:44.147 INFO:teuthology.orchestra.run.smithi060.stdout: 411871 4 -rw-r--r-- 1 ubuntu ubuntu 2638 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mount/conf.cc 2024-03-27T21:42:44.147 INFO:teuthology.orchestra.run.smithi060.stdout: 411874 12 -rw-r--r-- 1 ubuntu ubuntu 8269 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mount/mtab.c 2024-03-27T21:42:44.147 INFO:teuthology.orchestra.run.smithi060.stdout: 411873 4 -rw-r--r-- 1 ubuntu ubuntu 1049 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mount/mount.ceph.h 2024-03-27T21:42:44.147 INFO:teuthology.orchestra.run.smithi060.stdout: 411870 8 -rw-r--r-- 1 ubuntu ubuntu 4505 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mount/canonicalize.c 2024-03-27T21:42:44.147 INFO:teuthology.orchestra.run.smithi060.stdout: 411872 28 -rw-r--r-- 1 ubuntu ubuntu 25575 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mount/mount.ceph.c 2024-03-27T21:42:44.147 INFO:teuthology.orchestra.run.smithi060.stdout: 414747 12 drwxr-xr-x 6 ubuntu ubuntu 12288 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw 2024-03-27T21:42:44.147 INFO:teuthology.orchestra.run.smithi060.stdout: 415135 32 -rw-r--r-- 1 ubuntu ubuntu 30802 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_client.cc 2024-03-27T21:42:44.148 INFO:teuthology.orchestra.run.smithi060.stdout: 415189 20 -rw-r--r-- 1 ubuntu ubuntu 17126 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_swift_auth.h 2024-03-27T21:42:44.148 INFO:teuthology.orchestra.run.smithi060.stdout: 415036 4 -rw-r--r-- 1 ubuntu ubuntu 1404 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_kmip_client.h 2024-03-27T21:42:44.148 INFO:teuthology.orchestra.run.smithi060.stdout: 415075 4 -rw-r--r-- 1 ubuntu ubuntu 1262 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_multi_del.h 2024-03-27T21:42:44.148 INFO:teuthology.orchestra.run.smithi060.stdout: 415112 4 -rw-r--r-- 1 ubuntu ubuntu 1178 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_process_env.h 2024-03-27T21:42:44.148 INFO:teuthology.orchestra.run.smithi060.stdout: 414930 16 -rw-r--r-- 1 ubuntu ubuntu 12802 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_acl.h 2024-03-27T21:42:44.148 INFO:teuthology.orchestra.run.smithi060.stdout: 415059 4 -rw-r--r-- 1 ubuntu ubuntu 3823 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_lua_data_filter.cc 2024-03-27T21:42:44.148 INFO:teuthology.orchestra.run.smithi060.stdout: 415060 4 -rw-r--r-- 1 ubuntu ubuntu 1104 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_lua_data_filter.h 2024-03-27T21:42:44.149 INFO:teuthology.orchestra.run.smithi060.stdout: 414935 8 -rw-r--r-- 1 ubuntu ubuntu 6064 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_acl_types.h 2024-03-27T21:42:44.149 INFO:teuthology.orchestra.run.smithi060.stdout: 415053 20 -rw-r--r-- 1 ubuntu ubuntu 19242 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_log.cc 2024-03-27T21:42:44.149 INFO:teuthology.orchestra.run.smithi060.stdout: 415158 28 -rw-r--r-- 1 ubuntu ubuntu 26563 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_sts.cc 2024-03-27T21:42:44.149 INFO:teuthology.orchestra.run.smithi060.stdout: 415147 8 -rw-r--r-- 1 ubuntu ubuntu 6376 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_oidc_provider.cc 2024-03-27T21:42:44.149 INFO:teuthology.orchestra.run.smithi060.stdout: 415012 8 -rw-r--r-- 1 ubuntu ubuntu 5924 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_flight.h 2024-03-27T21:42:44.149 INFO:teuthology.orchestra.run.smithi060.stdout: 415167 4 -rw-r--r-- 1 ubuntu ubuntu 947 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_zero.h 2024-03-27T21:42:44.149 INFO:teuthology.orchestra.run.smithi060.stdout: 415078 8 -rw-r--r-- 1 ubuntu ubuntu 5534 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_notify_event_type.cc 2024-03-27T21:42:44.149 INFO:teuthology.orchestra.run.smithi060.stdout: 414945 4 -rw-r--r-- 1 ubuntu ubuntu 3948 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_arn.h 2024-03-27T21:42:44.150 INFO:teuthology.orchestra.run.smithi060.stdout: 415089 4 -rw-r--r-- 1 ubuntu ubuntu 3506 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_op_type.h 2024-03-27T21:42:44.150 INFO:teuthology.orchestra.run.smithi060.stdout: 415162 4 -rw-r--r-- 1 ubuntu ubuntu 3482 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_usage.cc 2024-03-27T21:42:44.150 INFO:teuthology.orchestra.run.smithi060.stdout: 415070 4 -rw-r--r-- 1 ubuntu ubuntu 2955 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_meta_sync_status.h 2024-03-27T21:42:44.150 INFO:teuthology.orchestra.run.smithi060.stdout: 415145 12 -rw-r--r-- 1 ubuntu ubuntu 8230 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_metadata.cc 2024-03-27T21:42:44.150 INFO:teuthology.orchestra.run.smithi060.stdout: 414955 8 -rw-r--r-- 1 ubuntu ubuntu 7616 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_auth_keystone.h 2024-03-27T21:42:44.150 INFO:teuthology.orchestra.run.smithi060.stdout: 415176 56 -rw-r--r-- 1 ubuntu ubuntu 55411 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_sal_dbstore.cc 2024-03-27T21:42:44.150 INFO:teuthology.orchestra.run.smithi060.stdout: 415129 4 -rw-r--r-- 1 ubuntu ubuntu 851 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_request.h 2024-03-27T21:42:44.150 INFO:teuthology.orchestra.run.smithi060.stdout: 415038 4 -rw-r--r-- 1 ubuntu ubuntu 874 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_kmip_client_impl.h 2024-03-27T21:42:44.150 INFO:teuthology.orchestra.run.smithi060.stdout: 415054 8 -rw-r--r-- 1 ubuntu ubuntu 7535 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_log.h 2024-03-27T21:42:44.151 INFO:teuthology.orchestra.run.smithi060.stdout: 415164 12 -rw-r--r-- 1 ubuntu ubuntu 11614 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_user_policy.cc 2024-03-27T21:42:44.151 INFO:teuthology.orchestra.run.smithi060.stdout: 415047 20 -rw-r--r-- 1 ubuntu ubuntu 16773 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_lib.cc 2024-03-27T21:42:44.151 INFO:teuthology.orchestra.run.smithi060.stdout: 415175 16 -rw-r--r-- 1 ubuntu ubuntu 13345 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_sal_config.h 2024-03-27T21:42:44.151 INFO:teuthology.orchestra.run.smithi060.stdout: 415108 4 -rw-r--r-- 1 ubuntu ubuntu 2599 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_polparser.cc 2024-03-27T21:42:44.151 INFO:teuthology.orchestra.run.smithi060.stdout: 414977 8 -rw-r--r-- 1 ubuntu ubuntu 8148 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_compression.cc 2024-03-27T21:42:44.151 INFO:teuthology.orchestra.run.smithi060.stdout: 415003 4 -rw-r--r-- 1 ubuntu ubuntu 3340 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_dmclock_sync_scheduler.cc 2024-03-27T21:42:44.151 INFO:teuthology.orchestra.run.smithi060.stdout: 415104 4 -rw-r--r-- 1 ubuntu ubuntu 1796 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_period_pusher.h 2024-03-27T21:42:44.151 INFO:teuthology.orchestra.run.smithi060.stdout: 415194 20 -rw-r--r-- 1 ubuntu ubuntu 17626 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_sync_policy.h 2024-03-27T21:42:44.152 INFO:teuthology.orchestra.run.smithi060.stdout: 415077 4 -rw-r--r-- 1 ubuntu ubuntu 781 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_multipart_meta_filter.cc 2024-03-27T21:42:44.152 INFO:teuthology.orchestra.run.smithi060.stdout: 415106 12 -rw-r--r-- 1 ubuntu ubuntu 8222 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_policy_s3.cc 2024-03-27T21:42:44.152 INFO:teuthology.orchestra.run.smithi060.stdout: 414943 24 -rw-r--r-- 1 ubuntu ubuntu 20749 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_appmain.cc 2024-03-27T21:42:44.152 INFO:teuthology.orchestra.run.smithi060.stdout: 415013 8 -rw-r--r-- 1 ubuntu ubuntu 7224 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_flight_frontend.cc 2024-03-27T21:42:44.152 INFO:teuthology.orchestra.run.smithi060.stdout: 415063 8 -rw-r--r-- 1 ubuntu ubuntu 5048 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_lua_utils.cc 2024-03-27T21:42:44.152 INFO:teuthology.orchestra.run.smithi060.stdout: 414940 4 -rw-r--r-- 1 ubuntu ubuntu 3544 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_aio_throttle.h 2024-03-27T21:42:44.152 INFO:teuthology.orchestra.run.smithi060.stdout: 415203 8 -rw-r--r-- 1 ubuntu ubuntu 5066 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_torrent.cc 2024-03-27T21:42:44.152 INFO:teuthology.orchestra.run.smithi060.stdout: 415058 8 -rw-r--r-- 1 ubuntu ubuntu 5503 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_lua_background.h 2024-03-27T21:42:44.152 INFO:teuthology.orchestra.run.smithi060.stdout: 415159 12 -rw-r--r-- 1 ubuntu ubuntu 8472 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_sts.h 2024-03-27T21:42:44.153 INFO:teuthology.orchestra.run.smithi060.stdout: 414937 8 -rw-r--r-- 1 ubuntu ubuntu 5392 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_aio.cc 2024-03-27T21:42:44.153 INFO:teuthology.orchestra.run.smithi060.stdout: 415178 36 -rw-r--r-- 1 ubuntu ubuntu 36043 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_sal_filter.cc 2024-03-27T21:42:44.153 INFO:teuthology.orchestra.run.smithi060.stdout: 415105 4 -rw-r--r-- 1 ubuntu ubuntu 3057 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_placement_types.h 2024-03-27T21:42:44.153 INFO:teuthology.orchestra.run.smithi060.stdout: 415119 36 -rw-r--r-- 1 ubuntu ubuntu 34011 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_quota.cc 2024-03-27T21:42:44.153 INFO:teuthology.orchestra.run.smithi060.stdout: 415050 4 -rw-r--r-- 1 ubuntu ubuntu 3240 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_loadgen.cc 2024-03-27T21:42:44.153 INFO:teuthology.orchestra.run.smithi060.stdout: 415155 204 -rw-r--r-- 1 ubuntu ubuntu 204804 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_s3.cc 2024-03-27T21:42:44.153 INFO:teuthology.orchestra.run.smithi060.stdout: 415161 20 -rw-r--r-- 1 ubuntu ubuntu 18598 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_swift.h 2024-03-27T21:42:44.153 INFO:teuthology.orchestra.run.smithi060.stdout: 415179 44 -rw-r--r-- 1 ubuntu ubuntu 43447 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_sal_filter.h 2024-03-27T21:42:44.154 INFO:teuthology.orchestra.run.smithi060.stdout: 414748 16 -rw-r--r-- 1 ubuntu ubuntu 16234 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/CMakeLists.txt 2024-03-27T21:42:44.154 INFO:teuthology.orchestra.run.smithi060.stdout: 415034 12 -rw-r--r-- 1 ubuntu ubuntu 9783 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_keystone.h 2024-03-27T21:42:44.154 INFO:teuthology.orchestra.run.smithi060.stdout: 414988 20 -rw-r--r-- 1 ubuntu ubuntu 20001 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_cr_rest.h 2024-03-27T21:42:44.154 INFO:teuthology.orchestra.run.smithi060.stdout: 415065 4 -rw-r--r-- 1 ubuntu ubuntu 155 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_lua_version.h 2024-03-27T21:42:44.154 INFO:teuthology.orchestra.run.smithi060.stdout: 414966 12 -rw-r--r-- 1 ubuntu ubuntu 9706 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_bucket_layout.cc 2024-03-27T21:42:44.154 INFO:teuthology.orchestra.run.smithi060.stdout: 415115 36 -rw-r--r-- 1 ubuntu ubuntu 36773 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_pubsub.cc 2024-03-27T21:42:44.154 INFO:teuthology.orchestra.run.smithi060.stdout: 414750 4 drwxr-xr-x 10 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver 2024-03-27T21:42:44.154 INFO:teuthology.orchestra.run.smithi060.stdout: 414802 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/posix 2024-03-27T21:42:44.155 INFO:teuthology.orchestra.run.smithi060.stdout: 414803 4 -rw-r--r-- 1 ubuntu ubuntu 1414 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/posix/README.md 2024-03-27T21:42:44.155 INFO:teuthology.orchestra.run.smithi060.stdout: 414804 4 -rw-r--r-- 1 ubuntu ubuntu 132 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/posix/bucket_cache.cpp 2024-03-27T21:42:44.155 INFO:teuthology.orchestra.run.smithi060.stdout: 414807 12 -rw-r--r-- 1 ubuntu ubuntu 11160 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/posix/lmdb-safe.cc 2024-03-27T21:42:44.155 INFO:teuthology.orchestra.run.smithi060.stdout: 414814 260 -rw-r--r-- 1 ubuntu ubuntu 263287 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/posix/zpp_bits.h 2024-03-27T21:42:44.155 INFO:teuthology.orchestra.run.smithi060.stdout: 414805 20 -rw-r--r-- 1 ubuntu ubuntu 17046 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/posix/bucket_cache.h 2024-03-27T21:42:44.155 INFO:teuthology.orchestra.run.smithi060.stdout: 414810 8 -rw-r--r-- 1 ubuntu ubuntu 6644 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/posix/notify.h 2024-03-27T21:42:44.155 INFO:teuthology.orchestra.run.smithi060.stdout: 414812 28 -rw-r--r-- 1 ubuntu ubuntu 26417 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/posix/rgw_sal_posix.h 2024-03-27T21:42:44.155 INFO:teuthology.orchestra.run.smithi060.stdout: 414809 4 -rw-r--r-- 1 ubuntu ubuntu 527 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/posix/notify.cpp 2024-03-27T21:42:44.156 INFO:teuthology.orchestra.run.smithi060.stdout: 414806 4 -rw-r--r-- 1 ubuntu ubuntu 761 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/posix/lmdb-safe-global.h 2024-03-27T21:42:44.156 INFO:teuthology.orchestra.run.smithi060.stdout: 414811 80 -rw-r--r-- 1 ubuntu ubuntu 79555 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/posix/rgw_sal_posix.cc 2024-03-27T21:42:44.156 INFO:teuthology.orchestra.run.smithi060.stdout: 414808 20 -rw-r--r-- 1 ubuntu ubuntu 17745 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/posix/lmdb-safe.hh 2024-03-27T21:42:44.156 INFO:teuthology.orchestra.run.smithi060.stdout: 414813 64 -rw-r--r-- 1 ubuntu ubuntu 62486 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/posix/unordered_dense.h 2024-03-27T21:42:44.156 INFO:teuthology.orchestra.run.smithi060.stdout: 414751 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/d4n 2024-03-27T21:42:44.156 INFO:teuthology.orchestra.run.smithi060.stdout: 414757 8 -rw-r--r-- 1 ubuntu ubuntu 7205 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/d4n/rgw_sal_d4n.h 2024-03-27T21:42:44.156 INFO:teuthology.orchestra.run.smithi060.stdout: 414753 4 -rw-r--r-- 1 ubuntu ubuntu 1308 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/d4n/d4n_datacache.h 2024-03-27T21:42:44.156 INFO:teuthology.orchestra.run.smithi060.stdout: 414752 12 -rw-r--r-- 1 ubuntu ubuntu 11770 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/d4n/d4n_datacache.cc 2024-03-27T21:42:44.157 INFO:teuthology.orchestra.run.smithi060.stdout: 414755 4 -rw-r--r-- 1 ubuntu ubuntu 1294 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/d4n/d4n_directory.h 2024-03-27T21:42:44.157 INFO:teuthology.orchestra.run.smithi060.stdout: 414754 8 -rw-r--r-- 1 ubuntu ubuntu 4354 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/d4n/d4n_directory.cc 2024-03-27T21:42:44.157 INFO:teuthology.orchestra.run.smithi060.stdout: 414756 20 -rw-r--r-- 1 ubuntu ubuntu 17346 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/d4n/rgw_sal_d4n.cc 2024-03-27T21:42:44.157 INFO:teuthology.orchestra.run.smithi060.stdout: 414815 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados 2024-03-27T21:42:44.157 INFO:teuthology.orchestra.run.smithi060.stdout: 414848 4 -rw-r--r-- 1 ubuntu ubuntu 1704 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_gc_log.cc 2024-03-27T21:42:44.157 INFO:teuthology.orchestra.run.smithi060.stdout: 414840 36 -rw-r--r-- 1 ubuntu ubuntu 32800 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_datalog.cc 2024-03-27T21:42:44.157 INFO:teuthology.orchestra.run.smithi060.stdout: 414873 12 -rw-r--r-- 1 ubuntu ubuntu 11034 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_rest_bucket.cc 2024-03-27T21:42:44.157 INFO:teuthology.orchestra.run.smithi060.stdout: 414915 12 -rw-r--r-- 1 ubuntu ubuntu 8720 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/sync_fairness.cc 2024-03-27T21:42:44.158 INFO:teuthology.orchestra.run.smithi060.stdout: 414850 4 -rw-r--r-- 1 ubuntu ubuntu 1521 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_lc_tier.h 2024-03-27T21:42:44.158 INFO:teuthology.orchestra.run.smithi060.stdout: 414914 40 -rw-r--r-- 1 ubuntu ubuntu 38079 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_zone.h 2024-03-27T21:42:44.158 INFO:teuthology.orchestra.run.smithi060.stdout: 414899 4 -rw-r--r-- 1 ubuntu ubuntu 3586 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_sync_module_log.cc 2024-03-27T21:42:44.158 INFO:teuthology.orchestra.run.smithi060.stdout: 414888 4 -rw-r--r-- 1 ubuntu ubuntu 407 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_sync_counters.h 2024-03-27T21:42:44.158 INFO:teuthology.orchestra.run.smithi060.stdout: 414904 8 -rw-r--r-- 1 ubuntu ubuntu 7140 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_tools.h 2024-03-27T21:42:44.158 INFO:teuthology.orchestra.run.smithi060.stdout: 414867 28 -rw-r--r-- 1 ubuntu ubuntu 24779 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_putobj_processor.cc 2024-03-27T21:42:44.158 INFO:teuthology.orchestra.run.smithi060.stdout: 414884 8 -rw-r--r-- 1 ubuntu ubuntu 6638 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_service.h 2024-03-27T21:42:44.158 INFO:teuthology.orchestra.run.smithi060.stdout: 414844 8 -rw-r--r-- 1 ubuntu ubuntu 6148 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_etag_verifier.cc 2024-03-27T21:42:44.159 INFO:teuthology.orchestra.run.smithi060.stdout: 414830 36 -rw-r--r-- 1 ubuntu ubuntu 34004 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_bucket_sync.cc 2024-03-27T21:42:44.159 INFO:teuthology.orchestra.run.smithi060.stdout: 414897 16 -rw-r--r-- 1 ubuntu ubuntu 14395 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_sync_module_es_rest.cc 2024-03-27T21:42:44.159 INFO:teuthology.orchestra.run.smithi060.stdout: 414870 72 -rw-r--r-- 1 ubuntu ubuntu 70026 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_rados.h 2024-03-27T21:42:44.159 INFO:teuthology.orchestra.run.smithi060.stdout: 414871 48 -rw-r--r-- 1 ubuntu ubuntu 47419 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_reshard.cc 2024-03-27T21:42:44.159 INFO:teuthology.orchestra.run.smithi060.stdout: 414856 52 -rw-r--r-- 1 ubuntu ubuntu 53231 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_notify.cc 2024-03-27T21:42:44.159 INFO:teuthology.orchestra.run.smithi060.stdout: 414916 4 -rw-r--r-- 1 ubuntu ubuntu 1584 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/sync_fairness.h 2024-03-27T21:42:44.159 INFO:teuthology.orchestra.run.smithi060.stdout: 414841 12 -rw-r--r-- 1 ubuntu ubuntu 11839 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_datalog.h 2024-03-27T21:42:44.159 INFO:teuthology.orchestra.run.smithi060.stdout: 414918 4 -rw-r--r-- 1 ubuntu ubuntu 3266 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/topic.h 2024-03-27T21:42:44.159 INFO:teuthology.orchestra.run.smithi060.stdout: 414817 12 -rw-r--r-- 1 ubuntu ubuntu 12040 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/cls_fifo_legacy.h 2024-03-27T21:42:44.160 INFO:teuthology.orchestra.run.smithi060.stdout: 414829 28 -rw-r--r-- 1 ubuntu ubuntu 27000 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_bucket.h 2024-03-27T21:42:44.160 INFO:teuthology.orchestra.run.smithi060.stdout: 414903 12 -rw-r--r-- 1 ubuntu ubuntu 11191 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_tools.cc 2024-03-27T21:42:44.160 INFO:teuthology.orchestra.run.smithi060.stdout: 414832 40 -rw-r--r-- 1 ubuntu ubuntu 38932 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_cr_rados.cc 2024-03-27T21:42:44.160 INFO:teuthology.orchestra.run.smithi060.stdout: 414883 16 -rw-r--r-- 1 ubuntu ubuntu 15417 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_service.cc 2024-03-27T21:42:44.160 INFO:teuthology.orchestra.run.smithi060.stdout: 414885 92 -rw-r--r-- 1 ubuntu ubuntu 92732 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_sync.cc 2024-03-27T21:42:44.160 INFO:teuthology.orchestra.run.smithi060.stdout: 414905 48 -rw-r--r-- 1 ubuntu ubuntu 46105 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_trim_bilog.cc 2024-03-27T21:42:44.160 INFO:teuthology.orchestra.run.smithi060.stdout: 414863 4 -rw-r--r-- 1 ubuntu ubuntu 2559 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_otp.h 2024-03-27T21:42:44.160 INFO:teuthology.orchestra.run.smithi060.stdout: 414911 72 -rw-r--r-- 1 ubuntu ubuntu 72829 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_user.cc 2024-03-27T21:42:44.161 INFO:teuthology.orchestra.run.smithi060.stdout: 414860 16 -rw-r--r-- 1 ubuntu ubuntu 13662 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_object_expirer_core.cc 2024-03-27T21:42:44.161 INFO:teuthology.orchestra.run.smithi060.stdout: 414869 340 -rw-r--r-- 1 ubuntu ubuntu 344332 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_rados.cc 2024-03-27T21:42:44.161 INFO:teuthology.orchestra.run.smithi060.stdout: 414895 36 -rw-r--r-- 1 ubuntu ubuntu 33048 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_sync_module_es.cc 2024-03-27T21:42:44.161 INFO:teuthology.orchestra.run.smithi060.stdout: 414880 4 -rw-r--r-- 1 ubuntu ubuntu 927 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_rest_user.h 2024-03-27T21:42:44.161 INFO:teuthology.orchestra.run.smithi060.stdout: 414902 4 -rw-r--r-- 1 ubuntu ubuntu 3267 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_sync_trace.h 2024-03-27T21:42:44.161 INFO:teuthology.orchestra.run.smithi060.stdout: 414847 4 -rw-r--r-- 1 ubuntu ubuntu 2650 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_gc.h 2024-03-27T21:42:44.161 INFO:teuthology.orchestra.run.smithi060.stdout: 414906 4 -rw-r--r-- 1 ubuntu ubuntu 3920 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_trim_bilog.h 2024-03-27T21:42:44.161 INFO:teuthology.orchestra.run.smithi060.stdout: 414849 40 -rw-r--r-- 1 ubuntu ubuntu 39672 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_lc_tier.cc 2024-03-27T21:42:44.161 INFO:teuthology.orchestra.run.smithi060.stdout: 414852 12 -rw-r--r-- 1 ubuntu ubuntu 11540 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_log_backing.h 2024-03-27T21:42:44.162 INFO:teuthology.orchestra.run.smithi060.stdout: 414898 4 -rw-r--r-- 1 ubuntu ubuntu 521 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_sync_module_es_rest.h 2024-03-27T21:42:44.162 INFO:teuthology.orchestra.run.smithi060.stdout: 414843 4 -rw-r--r-- 1 ubuntu ubuntu 845 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_datalog_notify.h 2024-03-27T21:42:44.162 INFO:teuthology.orchestra.run.smithi060.stdout: 414872 12 -rw-r--r-- 1 ubuntu ubuntu 9277 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_reshard.h 2024-03-27T21:42:44.162 INFO:teuthology.orchestra.run.smithi060.stdout: 414834 8 -rw-r--r-- 1 ubuntu ubuntu 4143 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_cr_tools.cc 2024-03-27T21:42:44.162 INFO:teuthology.orchestra.run.smithi060.stdout: 414865 16 -rw-r--r-- 1 ubuntu ubuntu 12561 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_pubsub_push.cc 2024-03-27T21:42:44.162 INFO:teuthology.orchestra.run.smithi060.stdout: 414896 4 -rw-r--r-- 1 ubuntu ubuntu 1508 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_sync_module_es.h 2024-03-27T21:42:44.162 INFO:teuthology.orchestra.run.smithi060.stdout: 414831 16 -rw-r--r-- 1 ubuntu ubuntu 14707 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_bucket_sync.h 2024-03-27T21:42:44.163 INFO:teuthology.orchestra.run.smithi060.stdout: 414890 4 -rw-r--r-- 1 ubuntu ubuntu 1881 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_sync_error_repo.h 2024-03-27T21:42:44.163 INFO:teuthology.orchestra.run.smithi060.stdout: 414893 64 -rw-r--r-- 1 ubuntu ubuntu 62634 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_sync_module_aws.cc 2024-03-27T21:42:44.163 INFO:teuthology.orchestra.run.smithi060.stdout: 414900 4 -rw-r--r-- 1 ubuntu ubuntu 444 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_sync_module_log.h 2024-03-27T21:42:44.163 INFO:teuthology.orchestra.run.smithi060.stdout: 414889 8 -rw-r--r-- 1 ubuntu ubuntu 5633 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_sync_error_repo.cc 2024-03-27T21:42:44.163 INFO:teuthology.orchestra.run.smithi060.stdout: 414917 12 -rw-r--r-- 1 ubuntu ubuntu 12086 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/topic.cc 2024-03-27T21:42:44.163 INFO:teuthology.orchestra.run.smithi060.stdout: 414858 16 -rw-r--r-- 1 ubuntu ubuntu 12936 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_obj_manifest.cc 2024-03-27T21:42:44.163 INFO:teuthology.orchestra.run.smithi060.stdout: 414828 116 -rw-r--r-- 1 ubuntu ubuntu 116710 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_bucket.cc 2024-03-27T21:42:44.163 INFO:teuthology.orchestra.run.smithi060.stdout: 414864 12 -rw-r--r-- 1 ubuntu ubuntu 10798 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_period.cc 2024-03-27T21:42:44.163 INFO:teuthology.orchestra.run.smithi060.stdout: 414846 28 -rw-r--r-- 1 ubuntu ubuntu 24929 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_gc.cc 2024-03-27T21:42:44.164 INFO:teuthology.orchestra.run.smithi060.stdout: 414833 52 -rw-r--r-- 1 ubuntu ubuntu 51145 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_cr_rados.h 2024-03-27T21:42:44.164 INFO:teuthology.orchestra.run.smithi060.stdout: 414854 12 -rw-r--r-- 1 ubuntu ubuntu 10364 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_metadata.h 2024-03-27T21:42:44.164 INFO:teuthology.orchestra.run.smithi060.stdout: 414816 76 -rw-r--r-- 1 ubuntu ubuntu 76710 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/cls_fifo_legacy.cc 2024-03-27T21:42:44.164 INFO:teuthology.orchestra.run.smithi060.stdout: 414861 8 -rw-r--r-- 1 ubuntu ubuntu 4555 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_object_expirer_core.h 2024-03-27T21:42:44.164 INFO:teuthology.orchestra.run.smithi060.stdout: 414910 4 -rw-r--r-- 1 ubuntu ubuntu 908 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_trim_mdlog.h 2024-03-27T21:42:44.164 INFO:teuthology.orchestra.run.smithi060.stdout: 414868 12 -rw-r--r-- 1 ubuntu ubuntu 10991 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_putobj_processor.h 2024-03-27T21:42:44.164 INFO:teuthology.orchestra.run.smithi060.stdout: 414874 4 -rw-r--r-- 1 ubuntu ubuntu 941 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_rest_bucket.h 2024-03-27T21:42:44.164 INFO:teuthology.orchestra.run.smithi060.stdout: 414877 16 -rw-r--r-- 1 ubuntu ubuntu 12404 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_rest_realm.cc 2024-03-27T21:42:44.165 INFO:teuthology.orchestra.run.smithi060.stdout: 414887 4 -rw-r--r-- 1 ubuntu ubuntu 1043 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_sync_counters.cc 2024-03-27T21:42:44.165 INFO:teuthology.orchestra.run.smithi060.stdout: 414859 20 -rw-r--r-- 1 ubuntu ubuntu 18216 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_obj_manifest.h 2024-03-27T21:42:44.165 INFO:teuthology.orchestra.run.smithi060.stdout: 414913 48 -rw-r--r-- 1 ubuntu ubuntu 48920 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_zone.cc 2024-03-27T21:42:44.165 INFO:teuthology.orchestra.run.smithi060.stdout: 414851 24 -rw-r--r-- 1 ubuntu ubuntu 21319 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_log_backing.cc 2024-03-27T21:42:44.165 INFO:teuthology.orchestra.run.smithi060.stdout: 414886 16 -rw-r--r-- 1 ubuntu ubuntu 15552 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_sync.h 2024-03-27T21:42:44.165 INFO:teuthology.orchestra.run.smithi060.stdout: 414857 8 -rw-r--r-- 1 ubuntu ubuntu 4675 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_notify.h 2024-03-27T21:42:44.165 INFO:teuthology.orchestra.run.smithi060.stdout: 414875 40 -rw-r--r-- 1 ubuntu ubuntu 37211 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_rest_log.cc 2024-03-27T21:42:44.165 INFO:teuthology.orchestra.run.smithi060.stdout: 414908 4 -rw-r--r-- 1 ubuntu ubuntu 965 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_trim_datalog.h 2024-03-27T21:42:44.165 INFO:teuthology.orchestra.run.smithi060.stdout: 414894 4 -rw-r--r-- 1 ubuntu ubuntu 2651 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_sync_module_aws.h 2024-03-27T21:42:44.166 INFO:teuthology.orchestra.run.smithi060.stdout: 414855 4 -rw-r--r-- 1 ubuntu ubuntu 1744 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_metadata_lister.h 2024-03-27T21:42:44.166 INFO:teuthology.orchestra.run.smithi060.stdout: 414881 124 -rw-r--r-- 1 ubuntu ubuntu 124622 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_sal_rados.cc 2024-03-27T21:42:44.166 INFO:teuthology.orchestra.run.smithi060.stdout: 414837 12 -rw-r--r-- 1 ubuntu ubuntu 8487 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_d3n_datacache.h 2024-03-27T21:42:44.166 INFO:teuthology.orchestra.run.smithi060.stdout: 414866 4 -rw-r--r-- 1 ubuntu ubuntu 1776 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_pubsub_push.h 2024-03-27T21:42:44.166 INFO:teuthology.orchestra.run.smithi060.stdout: 414818 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/config 2024-03-27T21:42:44.166 INFO:teuthology.orchestra.run.smithi060.stdout: 414821 12 -rw-r--r-- 1 ubuntu ubuntu 8201 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/config/period.cc 2024-03-27T21:42:44.166 INFO:teuthology.orchestra.run.smithi060.stdout: 414819 4 -rw-r--r-- 1 ubuntu ubuntu 3758 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/config/impl.cc 2024-03-27T21:42:44.166 INFO:teuthology.orchestra.run.smithi060.stdout: 414826 12 -rw-r--r-- 1 ubuntu ubuntu 9991 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/config/zone.cc 2024-03-27T21:42:44.167 INFO:teuthology.orchestra.run.smithi060.stdout: 414825 12 -rw-r--r-- 1 ubuntu ubuntu 9218 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/config/store.h 2024-03-27T21:42:44.167 INFO:teuthology.orchestra.run.smithi060.stdout: 414823 12 -rw-r--r-- 1 ubuntu ubuntu 11737 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/config/realm.cc 2024-03-27T21:42:44.167 INFO:teuthology.orchestra.run.smithi060.stdout: 414820 4 -rw-r--r-- 1 ubuntu ubuntu 4066 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/config/impl.h 2024-03-27T21:42:44.167 INFO:teuthology.orchestra.run.smithi060.stdout: 414827 12 -rw-r--r-- 1 ubuntu ubuntu 10804 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/config/zonegroup.cc 2024-03-27T21:42:44.167 INFO:teuthology.orchestra.run.smithi060.stdout: 414822 4 -rw-r--r-- 1 ubuntu ubuntu 1760 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/config/period_config.cc 2024-03-27T21:42:44.167 INFO:teuthology.orchestra.run.smithi060.stdout: 414824 4 -rw-r--r-- 1 ubuntu ubuntu 1282 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/config/store.cc 2024-03-27T21:42:44.167 INFO:teuthology.orchestra.run.smithi060.stdout: 414907 12 -rw-r--r-- 1 ubuntu ubuntu 8585 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_trim_datalog.cc 2024-03-27T21:42:44.167 INFO:teuthology.orchestra.run.smithi060.stdout: 414862 8 -rw-r--r-- 1 ubuntu ubuntu 5638 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_otp.cc 2024-03-27T21:42:44.167 INFO:teuthology.orchestra.run.smithi060.stdout: 414909 28 -rw-r--r-- 1 ubuntu ubuntu 27546 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_trim_mdlog.cc 2024-03-27T21:42:44.168 INFO:teuthology.orchestra.run.smithi060.stdout: 414835 4 -rw-r--r-- 1 ubuntu ubuntu 1688 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_cr_tools.h 2024-03-27T21:42:44.168 INFO:teuthology.orchestra.run.smithi060.stdout: 414901 12 -rw-r--r-- 1 ubuntu ubuntu 8271 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_sync_trace.cc 2024-03-27T21:42:44.168 INFO:teuthology.orchestra.run.smithi060.stdout: 414892 8 -rw-r--r-- 1 ubuntu ubuntu 5993 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_sync_module.h 2024-03-27T21:42:44.168 INFO:teuthology.orchestra.run.smithi060.stdout: 414891 4 -rw-r--r-- 1 ubuntu ubuntu 3371 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_sync_module.cc 2024-03-27T21:42:44.168 INFO:teuthology.orchestra.run.smithi060.stdout: 414912 28 -rw-r--r-- 1 ubuntu ubuntu 26879 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_user.h 2024-03-27T21:42:44.168 INFO:teuthology.orchestra.run.smithi060.stdout: 414878 4 -rw-r--r-- 1 ubuntu ubuntu 443 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_rest_realm.h 2024-03-27T21:42:44.168 INFO:teuthology.orchestra.run.smithi060.stdout: 414838 248 -rw-r--r-- 1 ubuntu ubuntu 251333 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_data_sync.cc 2024-03-27T21:42:44.168 INFO:teuthology.orchestra.run.smithi060.stdout: 414839 28 -rw-r--r-- 1 ubuntu ubuntu 26605 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_data_sync.h 2024-03-27T21:42:44.169 INFO:teuthology.orchestra.run.smithi060.stdout: 414853 8 -rw-r--r-- 1 ubuntu ubuntu 6718 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_metadata.cc 2024-03-27T21:42:44.169 INFO:teuthology.orchestra.run.smithi060.stdout: 414845 4 -rw-r--r-- 1 ubuntu ubuntu 2817 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_etag_verifier.h 2024-03-27T21:42:44.169 INFO:teuthology.orchestra.run.smithi060.stdout: 414836 16 -rw-r--r-- 1 ubuntu ubuntu 13122 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_d3n_datacache.cc 2024-03-27T21:42:44.169 INFO:teuthology.orchestra.run.smithi060.stdout: 414879 32 -rw-r--r-- 1 ubuntu ubuntu 31076 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_rest_user.cc 2024-03-27T21:42:44.169 INFO:teuthology.orchestra.run.smithi060.stdout: 414876 12 -rw-r--r-- 1 ubuntu ubuntu 9230 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_rest_log.h 2024-03-27T21:42:44.169 INFO:teuthology.orchestra.run.smithi060.stdout: 414842 4 -rw-r--r-- 1 ubuntu ubuntu 2246 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_datalog_notify.cc 2024-03-27T21:42:44.169 INFO:teuthology.orchestra.run.smithi060.stdout: 414882 48 -rw-r--r-- 1 ubuntu ubuntu 47274 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/rados/rgw_sal_rados.h 2024-03-27T21:42:44.169 INFO:teuthology.orchestra.run.smithi060.stdout: 414758 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/daos 2024-03-27T21:42:44.170 INFO:teuthology.orchestra.run.smithi060.stdout: 414759 4 -rw-r--r-- 1 ubuntu ubuntu 879 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/daos/README.md 2024-03-27T21:42:44.170 INFO:teuthology.orchestra.run.smithi060.stdout: 414761 44 -rw-r--r-- 1 ubuntu ubuntu 42602 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/daos/rgw_sal_daos.h 2024-03-27T21:42:44.170 INFO:teuthology.orchestra.run.smithi060.stdout: 414760 80 -rw-r--r-- 1 ubuntu ubuntu 78878 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/daos/rgw_sal_daos.cc 2024-03-27T21:42:44.170 INFO:teuthology.orchestra.run.smithi060.stdout: 414796 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/json_config 2024-03-27T21:42:44.170 INFO:teuthology.orchestra.run.smithi060.stdout: 414798 4 -rw-r--r-- 1 ubuntu ubuntu 707 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/json_config/store.h 2024-03-27T21:42:44.170 INFO:teuthology.orchestra.run.smithi060.stdout: 414797 8 -rw-r--r-- 1 ubuntu ubuntu 5988 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/json_config/store.cc 2024-03-27T21:42:44.170 INFO:teuthology.orchestra.run.smithi060.stdout: 414793 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/immutable_config 2024-03-27T21:42:44.170 INFO:teuthology.orchestra.run.smithi060.stdout: 414795 12 -rw-r--r-- 1 ubuntu ubuntu 9431 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/immutable_config/store.h 2024-03-27T21:42:44.171 INFO:teuthology.orchestra.run.smithi060.stdout: 414794 16 -rw-r--r-- 1 ubuntu ubuntu 13715 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/immutable_config/store.cc 2024-03-27T21:42:44.171 INFO:teuthology.orchestra.run.smithi060.stdout: 414762 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore 2024-03-27T21:42:44.171 INFO:teuthology.orchestra.run.smithi060.stdout: 414778 4 -rw-r--r-- 1 ubuntu ubuntu 1524 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/dbstore_mgr.h 2024-03-27T21:42:44.171 INFO:teuthology.orchestra.run.smithi060.stdout: 414764 4 -rw-r--r-- 1 ubuntu ubuntu 1764 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/README.md 2024-03-27T21:42:44.171 INFO:teuthology.orchestra.run.smithi060.stdout: 414789 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/tests 2024-03-27T21:42:44.171 INFO:teuthology.orchestra.run.smithi060.stdout: 414790 4 -rw-r--r-- 1 ubuntu ubuntu 563 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/tests/CMakeLists.txt 2024-03-27T21:42:44.171 INFO:teuthology.orchestra.run.smithi060.stdout: 414791 8 -rw-r--r-- 1 ubuntu ubuntu 5319 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/tests/dbstore_mgr_tests.cc 2024-03-27T21:42:44.171 INFO:teuthology.orchestra.run.smithi060.stdout: 414792 44 -rw-r--r-- 1 ubuntu ubuntu 41384 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/tests/dbstore_tests.cc 2024-03-27T21:42:44.171 INFO:teuthology.orchestra.run.smithi060.stdout: 414763 4 -rw-r--r-- 1 ubuntu ubuntu 1967 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/CMakeLists.txt 2024-03-27T21:42:44.172 INFO:teuthology.orchestra.run.smithi060.stdout: 414779 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/sqlite 2024-03-27T21:42:44.172 INFO:teuthology.orchestra.run.smithi060.stdout: 414784 4 -rw-r--r-- 1 ubuntu ubuntu 2348 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/sqlite/error.h 2024-03-27T21:42:44.172 INFO:teuthology.orchestra.run.smithi060.stdout: 414780 4 -rw-r--r-- 1 ubuntu ubuntu 415 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/sqlite/CMakeLists.txt 2024-03-27T21:42:44.172 INFO:teuthology.orchestra.run.smithi060.stdout: 414786 24 -rw-r--r-- 1 ubuntu ubuntu 21389 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/sqlite/sqliteDB.h 2024-03-27T21:42:44.172 INFO:teuthology.orchestra.run.smithi060.stdout: 414788 4 -rw-r--r-- 1 ubuntu ubuntu 3020 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/sqlite/statement.h 2024-03-27T21:42:44.172 INFO:teuthology.orchestra.run.smithi060.stdout: 414783 4 -rw-r--r-- 1 ubuntu ubuntu 952 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/sqlite/error.cc 2024-03-27T21:42:44.172 INFO:teuthology.orchestra.run.smithi060.stdout: 414781 4 -rw-r--r-- 1 ubuntu ubuntu 907 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/sqlite/connection.cc 2024-03-27T21:42:44.172 INFO:teuthology.orchestra.run.smithi060.stdout: 414787 8 -rw-r--r-- 1 ubuntu ubuntu 7054 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/sqlite/statement.cc 2024-03-27T21:42:44.172 INFO:teuthology.orchestra.run.smithi060.stdout: 414782 4 -rw-r--r-- 1 ubuntu ubuntu 1485 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/sqlite/connection.h 2024-03-27T21:42:44.173 INFO:teuthology.orchestra.run.smithi060.stdout: 414785 96 -rw-r--r-- 1 ubuntu ubuntu 97254 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/sqlite/sqliteDB.cc 2024-03-27T21:42:44.173 INFO:teuthology.orchestra.run.smithi060.stdout: 414776 8 -rw-r--r-- 1 ubuntu ubuntu 5311 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/dbstore_main.cc 2024-03-27T21:42:44.173 INFO:teuthology.orchestra.run.smithi060.stdout: 414777 4 -rw-r--r-- 1 ubuntu ubuntu 3136 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/dbstore_mgr.cc 2024-03-27T21:42:44.173 INFO:teuthology.orchestra.run.smithi060.stdout: 414770 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/config 2024-03-27T21:42:44.173 INFO:teuthology.orchestra.run.smithi060.stdout: 414772 8 -rw-r--r-- 1 ubuntu ubuntu 8134 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/config/sqlite.h 2024-03-27T21:42:44.173 INFO:teuthology.orchestra.run.smithi060.stdout: 414771 68 -rw-r--r-- 1 ubuntu ubuntu 68901 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/config/sqlite.cc 2024-03-27T21:42:44.173 INFO:teuthology.orchestra.run.smithi060.stdout: 414775 4 -rw-r--r-- 1 ubuntu ubuntu 671 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/config/store.h 2024-03-27T21:42:44.173 INFO:teuthology.orchestra.run.smithi060.stdout: 414773 12 -rw-r--r-- 1 ubuntu ubuntu 9244 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/config/sqlite_schema.h 2024-03-27T21:42:44.174 INFO:teuthology.orchestra.run.smithi060.stdout: 414774 4 -rw-r--r-- 1 ubuntu ubuntu 892 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/config/store.cc 2024-03-27T21:42:44.174 INFO:teuthology.orchestra.run.smithi060.stdout: 414765 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/common 2024-03-27T21:42:44.174 INFO:teuthology.orchestra.run.smithi060.stdout: 414766 8 -rw-r--r-- 1 ubuntu ubuntu 4665 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/common/connection_pool.h 2024-03-27T21:42:44.174 INFO:teuthology.orchestra.run.smithi060.stdout: 414767 60 -rw-r--r-- 1 ubuntu ubuntu 60048 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/common/dbstore.cc 2024-03-27T21:42:44.174 INFO:teuthology.orchestra.run.smithi060.stdout: 414769 4 -rw-r--r-- 1 ubuntu ubuntu 316 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/common/dbstore_log.h 2024-03-27T21:42:44.174 INFO:teuthology.orchestra.run.smithi060.stdout: 414768 80 -rw-r--r-- 1 ubuntu ubuntu 80695 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/dbstore/common/dbstore.h 2024-03-27T21:42:44.174 INFO:teuthology.orchestra.run.smithi060.stdout: 414799 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/motr 2024-03-27T21:42:44.174 INFO:teuthology.orchestra.run.smithi060.stdout: 414800 120 -rw-r--r-- 1 ubuntu ubuntu 122589 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/motr/rgw_sal_motr.cc 2024-03-27T21:42:44.174 INFO:teuthology.orchestra.run.smithi060.stdout: 414801 52 -rw-r--r-- 1 ubuntu ubuntu 50671 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/driver/motr/rgw_sal_motr.h 2024-03-27T21:42:44.175 INFO:teuthology.orchestra.run.smithi060.stdout: 415201 4 -rw-r--r-- 1 ubuntu ubuntu 4088 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_token.h 2024-03-27T21:42:44.175 INFO:teuthology.orchestra.run.smithi060.stdout: 415187 12 -rw-r--r-- 1 ubuntu ubuntu 9296 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_sts.h 2024-03-27T21:42:44.175 INFO:teuthology.orchestra.run.smithi060.stdout: 415180 4 -rw-r--r-- 1 ubuntu ubuntu 1251 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_sal_fwd.h 2024-03-27T21:42:44.175 INFO:teuthology.orchestra.run.smithi060.stdout: 415126 4 -rw-r--r-- 1 ubuntu ubuntu 3845 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_realm_watcher.cc 2024-03-27T21:42:44.175 INFO:teuthology.orchestra.run.smithi060.stdout: 414974 16 -rw-r--r-- 1 ubuntu ubuntu 13842 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_client_io_filters.h 2024-03-27T21:42:44.175 INFO:teuthology.orchestra.run.smithi060.stdout: 415156 36 -rw-r--r-- 1 ubuntu ubuntu 36344 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_s3.h 2024-03-27T21:42:44.175 INFO:teuthology.orchestra.run.smithi060.stdout: 415100 4 -rw-r--r-- 1 ubuntu ubuntu 3694 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_period_history.h 2024-03-27T21:42:44.175 INFO:teuthology.orchestra.run.smithi060.stdout: 415114 4 -rw-r--r-- 1 ubuntu ubuntu 1762 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_public_access.h 2024-03-27T21:42:44.176 INFO:teuthology.orchestra.run.smithi060.stdout: 414954 32 -rw-r--r-- 1 ubuntu ubuntu 28935 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_auth_keystone.cc 2024-03-27T21:42:44.176 INFO:teuthology.orchestra.run.smithi060.stdout: 414927 12 -rwxr-xr-x 1 ubuntu ubuntu 9021 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw-orphan-list 2024-03-27T21:42:44.176 INFO:teuthology.orchestra.run.smithi060.stdout: 414995 4 -rw-r--r-- 1 ubuntu ubuntu 3216 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_data_access.h 2024-03-27T21:42:44.176 INFO:teuthology.orchestra.run.smithi060.stdout: 415084 8 -rw-r--r-- 1 ubuntu ubuntu 4908 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_object_lock.h 2024-03-27T21:42:44.176 INFO:teuthology.orchestra.run.smithi060.stdout: 415033 24 -rw-r--r-- 1 ubuntu ubuntu 21612 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_keystone.cc 2024-03-27T21:42:44.176 INFO:teuthology.orchestra.run.smithi060.stdout: 415219 4 -rw-r--r-- 1 ubuntu ubuntu 639 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_xml_enc.cc 2024-03-27T21:42:44.176 INFO:teuthology.orchestra.run.smithi060.stdout: 415073 4 -rw-r--r-- 1 ubuntu ubuntu 1412 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_multi.h 2024-03-27T21:42:44.177 INFO:teuthology.orchestra.run.smithi060.stdout: 415055 12 -rw-r--r-- 1 ubuntu ubuntu 8801 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_lua.cc 2024-03-27T21:42:44.177 INFO:teuthology.orchestra.run.smithi060.stdout: 415109 8 -rw-r--r-- 1 ubuntu ubuntu 4141 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_pool_types.h 2024-03-27T21:42:44.177 INFO:teuthology.orchestra.run.smithi060.stdout: 415035 4 -rw-r--r-- 1 ubuntu ubuntu 1493 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_kmip_client.cc 2024-03-27T21:42:44.177 INFO:teuthology.orchestra.run.smithi060.stdout: 415064 20 -rw-r--r-- 1 ubuntu ubuntu 18060 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_lua_utils.h 2024-03-27T21:42:44.177 INFO:teuthology.orchestra.run.smithi060.stdout: 415222 20 -rw-r--r-- 1 ubuntu ubuntu 17735 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_zone_types.h 2024-03-27T21:42:44.177 INFO:teuthology.orchestra.run.smithi060.stdout: 415118 4 -rw-r--r-- 1 ubuntu ubuntu 2040 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_putobj.h 2024-03-27T21:42:44.177 INFO:teuthology.orchestra.run.smithi060.stdout: 414959 4 -rw-r--r-- 1 ubuntu ubuntu 2336 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_b64.h 2024-03-27T21:42:44.177 INFO:teuthology.orchestra.run.smithi060.stdout: 415020 32 -rw-r--r-- 1 ubuntu ubuntu 30766 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_http_client.cc 2024-03-27T21:42:44.178 INFO:teuthology.orchestra.run.smithi060.stdout: 415193 20 -rw-r--r-- 1 ubuntu ubuntu 17524 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_sync_policy.cc 2024-03-27T21:42:44.178 INFO:teuthology.orchestra.run.smithi060.stdout: 415174 72 -rw-r--r-- 1 ubuntu ubuntu 71933 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_sal.h 2024-03-27T21:42:44.178 INFO:teuthology.orchestra.run.smithi060.stdout: 415120 4 -rw-r--r-- 1 ubuntu ubuntu 1751 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_quota.h 2024-03-27T21:42:44.178 INFO:teuthology.orchestra.run.smithi060.stdout: 415025 4 -rw-r--r-- 1 ubuntu ubuntu 938 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_http_errors.h 2024-03-27T21:42:44.178 INFO:teuthology.orchestra.run.smithi060.stdout: 415148 4 -rw-r--r-- 1 ubuntu ubuntu 2457 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_oidc_provider.h 2024-03-27T21:42:44.178 INFO:teuthology.orchestra.run.smithi060.stdout: 415165 4 -rw-r--r-- 1 ubuntu ubuntu 2127 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_user_policy.h 2024-03-27T21:42:44.178 INFO:teuthology.orchestra.run.smithi060.stdout: 415067 4 -rw-r--r-- 1 ubuntu ubuntu 3684 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_main.h 2024-03-27T21:42:44.178 INFO:teuthology.orchestra.run.smithi060.stdout: 415092 48 -rw-r--r-- 1 ubuntu ubuntu 46914 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_orphan.cc 2024-03-27T21:42:44.179 INFO:teuthology.orchestra.run.smithi060.stdout: 415223 8 -rwxr-xr-x 1 ubuntu ubuntu 8004 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgwam.py 2024-03-27T21:42:44.179 INFO:teuthology.orchestra.run.smithi060.stdout: 414978 4 -rw-r--r-- 1 ubuntu ubuntu 2091 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_compression.h 2024-03-27T21:42:44.179 INFO:teuthology.orchestra.run.smithi060.stdout: 414963 4 -rw-r--r-- 1 ubuntu ubuntu 1295 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_bucket.h 2024-03-27T21:42:44.179 INFO:teuthology.orchestra.run.smithi060.stdout: 414970 12 -rw-r--r-- 1 ubuntu ubuntu 11630 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_cache.cc 2024-03-27T21:42:44.179 INFO:teuthology.orchestra.run.smithi060.stdout: 415046 4 -rw-r--r-- 1 ubuntu ubuntu 3195 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_ldap.h 2024-03-27T21:42:44.179 INFO:teuthology.orchestra.run.smithi060.stdout: 415202 4 -rw-r--r-- 1 ubuntu ubuntu 2678 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_tools.cc 2024-03-27T21:42:44.179 INFO:teuthology.orchestra.run.smithi060.stdout: 415199 4 -rw-r--r-- 1 ubuntu ubuntu 4059 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_tar.h 2024-03-27T21:42:44.179 INFO:teuthology.orchestra.run.smithi060.stdout: 415213 4 -rw-r--r-- 1 ubuntu ubuntu 599 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_web_idp.h 2024-03-27T21:42:44.180 INFO:teuthology.orchestra.run.smithi060.stdout: 415190 4 -rw-r--r-- 1 ubuntu ubuntu 448 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_sync.cc 2024-03-27T21:42:44.180 INFO:teuthology.orchestra.run.smithi060.stdout: 415211 4 -rw-r--r-- 1 ubuntu ubuntu 3170 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_user.cc 2024-03-27T21:42:44.180 INFO:teuthology.orchestra.run.smithi060.stdout: 414939 8 -rw-r--r-- 1 ubuntu ubuntu 5041 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_aio_throttle.cc 2024-03-27T21:42:44.180 INFO:teuthology.orchestra.run.smithi060.stdout: 415099 12 -rw-r--r-- 1 ubuntu ubuntu 11000 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_period_history.cc 2024-03-27T21:42:44.180 INFO:teuthology.orchestra.run.smithi060.stdout: 415041 92 -rw-r--r-- 1 ubuntu ubuntu 90665 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_lc.cc 2024-03-27T21:42:44.180 INFO:teuthology.orchestra.run.smithi060.stdout: 415091 4 -rw-r--r-- 1 ubuntu ubuntu 270 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_opa.h 2024-03-27T21:42:44.180 INFO:teuthology.orchestra.run.smithi060.stdout: 415152 4 -rw-r--r-- 1 ubuntu ubuntu 924 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_ratelimit.h 2024-03-27T21:42:44.180 INFO:teuthology.orchestra.run.smithi060.stdout: 415200 4 -rw-r--r-- 1 ubuntu ubuntu 3697 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_token.cc 2024-03-27T21:42:44.180 INFO:teuthology.orchestra.run.smithi060.stdout: 414942 4 -rw-r--r-- 1 ubuntu ubuntu 2234 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_amqp.h 2024-03-27T21:42:44.181 INFO:teuthology.orchestra.run.smithi060.stdout: 414989 56 -rw-r--r-- 1 ubuntu ubuntu 57275 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_crypt.cc 2024-03-27T21:42:44.181 INFO:teuthology.orchestra.run.smithi060.stdout: 414972 4 -rw-r--r-- 1 ubuntu ubuntu 801 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_client_io.cc 2024-03-27T21:42:44.181 INFO:teuthology.orchestra.run.smithi060.stdout: 415142 4 -rw-r--r-- 1 ubuntu ubuntu 1445 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_iam.h 2024-03-27T21:42:44.181 INFO:teuthology.orchestra.run.smithi060.stdout: 415011 24 -rw-r--r-- 1 ubuntu ubuntu 20525 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_flight.cc 2024-03-27T21:42:44.181 INFO:teuthology.orchestra.run.smithi060.stdout: 415076 4 -rw-r--r-- 1 ubuntu ubuntu 942 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_multiparser.cc 2024-03-27T21:42:44.181 INFO:teuthology.orchestra.run.smithi060.stdout: 415122 12 -rw-r--r-- 1 ubuntu ubuntu 10868 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_ratelimit.h 2024-03-27T21:42:44.181 INFO:teuthology.orchestra.run.smithi060.stdout: 415169 8 -rw-r--r-- 1 ubuntu ubuntu 7005 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_role.h 2024-03-27T21:42:44.181 INFO:teuthology.orchestra.run.smithi060.stdout: 415186 20 -rw-r--r-- 1 ubuntu ubuntu 16628 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_sts.cc 2024-03-27T21:42:44.182 INFO:teuthology.orchestra.run.smithi060.stdout: 415143 4 -rw-r--r-- 1 ubuntu ubuntu 1431 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_info.cc 2024-03-27T21:42:44.182 INFO:teuthology.orchestra.run.smithi060.stdout: 415184 4 -rw-r--r-- 1 ubuntu ubuntu 509 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_string.cc 2024-03-27T21:42:44.182 INFO:teuthology.orchestra.run.smithi060.stdout: 415195 4 -rw-r--r-- 1 ubuntu ubuntu 1719 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_tag.cc 2024-03-27T21:42:44.182 INFO:teuthology.orchestra.run.smithi060.stdout: 414975 88 -rw-r--r-- 1 ubuntu ubuntu 88674 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_common.cc 2024-03-27T21:42:44.182 INFO:teuthology.orchestra.run.smithi060.stdout: 415079 4 -rw-r--r-- 1 ubuntu ubuntu 2255 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_notify_event_type.h 2024-03-27T21:42:44.182 INFO:teuthology.orchestra.run.smithi060.stdout: 415093 12 -rw-r--r-- 1 ubuntu ubuntu 8950 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_orphan.h 2024-03-27T21:42:44.182 INFO:teuthology.orchestra.run.smithi060.stdout: 415061 28 -rw-r--r-- 1 ubuntu ubuntu 27869 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_lua_request.cc 2024-03-27T21:42:44.182 INFO:teuthology.orchestra.run.smithi060.stdout: 415217 12 -rw-r--r-- 1 ubuntu ubuntu 10108 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_xml.cc 2024-03-27T21:42:44.182 INFO:teuthology.orchestra.run.smithi060.stdout: 415028 12 -rw-r--r-- 1 ubuntu ubuntu 9285 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_iam_policy_keywords.gperf 2024-03-27T21:42:44.183 INFO:teuthology.orchestra.run.smithi060.stdout: 415090 4 -rw-r--r-- 1 ubuntu ubuntu 2787 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_opa.cc 2024-03-27T21:42:44.183 INFO:teuthology.orchestra.run.smithi060.stdout: 415032 4 -rw-r--r-- 1 ubuntu ubuntu 1873 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_kafka.h 2024-03-27T21:42:44.183 INFO:teuthology.orchestra.run.smithi060.stdout: 415019 4 -rw-r--r-- 1 ubuntu ubuntu 1041 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_gc_log.h 2024-03-27T21:42:44.183 INFO:teuthology.orchestra.run.smithi060.stdout: 415001 8 -rw-r--r-- 1 ubuntu ubuntu 5839 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_dmclock_scheduler_ctx.cc 2024-03-27T21:42:44.183 INFO:teuthology.orchestra.run.smithi060.stdout: 414998 8 -rw-r--r-- 1 ubuntu ubuntu 5178 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_dmclock_async_scheduler.cc 2024-03-27T21:42:44.183 INFO:teuthology.orchestra.run.smithi060.stdout: 415181 20 -rw-r--r-- 1 ubuntu ubuntu 18296 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_sal_store.h 2024-03-27T21:42:44.183 INFO:teuthology.orchestra.run.smithi060.stdout: 414950 4 -rw-r--r-- 1 ubuntu ubuntu 1822 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_asio_frontend_timer.h 2024-03-27T21:42:44.184 INFO:teuthology.orchestra.run.smithi060.stdout: 414933 12 -rw-r--r-- 1 ubuntu ubuntu 11792 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_acl_swift.cc 2024-03-27T21:42:44.184 INFO:teuthology.orchestra.run.smithi060.stdout: 414932 4 -rw-r--r-- 1 ubuntu ubuntu 1666 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_acl_s3.h 2024-03-27T21:42:44.184 INFO:teuthology.orchestra.run.smithi060.stdout: 415198 4 -rw-r--r-- 1 ubuntu ubuntu 1004 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_tag_s3.h 2024-03-27T21:42:44.184 INFO:teuthology.orchestra.run.smithi060.stdout: 415197 4 -rw-r--r-- 1 ubuntu ubuntu 1505 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_tag_s3.cc 2024-03-27T21:42:44.184 INFO:teuthology.orchestra.run.smithi060.stdout: 415183 4 -rw-r--r-- 1 ubuntu ubuntu 699 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_signal.h 2024-03-27T21:42:44.184 INFO:teuthology.orchestra.run.smithi060.stdout: 414958 28 -rw-r--r-- 1 ubuntu ubuntu 26848 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_auth_s3.h 2024-03-27T21:42:44.184 INFO:teuthology.orchestra.run.smithi060.stdout: 415000 4 -rw-r--r-- 1 ubuntu ubuntu 2173 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_dmclock_scheduler.h 2024-03-27T21:42:44.184 INFO:teuthology.orchestra.run.smithi060.stdout: 415082 4 -rw-r--r-- 1 ubuntu ubuntu 3276 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_object_expirer.cc 2024-03-27T21:42:44.184 INFO:teuthology.orchestra.run.smithi060.stdout: 415095 4 -rw-r--r-- 1 ubuntu ubuntu 188 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_os_lib.h 2024-03-27T21:42:44.185 INFO:teuthology.orchestra.run.smithi060.stdout: 414971 8 -rw-r--r-- 1 ubuntu ubuntu 6247 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_cache.h 2024-03-27T21:42:44.185 INFO:teuthology.orchestra.run.smithi060.stdout: 415111 8 -rw-r--r-- 1 ubuntu ubuntu 4523 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_process.h 2024-03-27T21:42:44.185 INFO:teuthology.orchestra.run.smithi060.stdout: 415069 4 -rw-r--r-- 1 ubuntu ubuntu 689 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_mdlog_types.h 2024-03-27T21:42:44.185 INFO:teuthology.orchestra.run.smithi060.stdout: 415177 40 -rw-r--r-- 1 ubuntu ubuntu 39607 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_sal_dbstore.h 2024-03-27T21:42:44.185 INFO:teuthology.orchestra.run.smithi060.stdout: 415215 8 -rw-r--r-- 1 ubuntu ubuntu 6407 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_website.h 2024-03-27T21:42:44.185 INFO:teuthology.orchestra.run.smithi060.stdout: 415157 4 -rw-r--r-- 1 ubuntu ubuntu 3629 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_s3website.h 2024-03-27T21:42:44.185 INFO:teuthology.orchestra.run.smithi060.stdout: 414936 388 -rw-r--r-- 1 ubuntu ubuntu 393860 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_admin.cc 2024-03-27T21:42:44.185 INFO:teuthology.orchestra.run.smithi060.stdout: 414934 4 -rw-r--r-- 1 ubuntu ubuntu 1899 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_acl_swift.h 2024-03-27T21:42:44.186 INFO:teuthology.orchestra.run.smithi060.stdout: 414996 4 -rw-r--r-- 1 ubuntu ubuntu 981 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_dencoder.cc 2024-03-27T21:42:44.186 INFO:teuthology.orchestra.run.smithi060.stdout: 414983 8 -rw-r--r-- 1 ubuntu ubuntu 4314 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_cors.h 2024-03-27T21:42:44.186 INFO:teuthology.orchestra.run.smithi060.stdout: 415140 20 -rw-r--r-- 1 ubuntu ubuntu 16875 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_conn.h 2024-03-27T21:42:44.186 INFO:teuthology.orchestra.run.smithi060.stdout: 415015 12 -rw-r--r-- 1 ubuntu ubuntu 9321 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_formats.cc 2024-03-27T21:42:44.186 INFO:teuthology.orchestra.run.smithi060.stdout: 415016 4 -rw-r--r-- 1 ubuntu ubuntu 3826 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_formats.h 2024-03-27T21:42:44.186 INFO:teuthology.orchestra.run.smithi060.stdout: 414993 8 -rw-r--r-- 1 ubuntu ubuntu 5416 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_d3n_cacherequest.h 2024-03-27T21:42:44.186 INFO:teuthology.orchestra.run.smithi060.stdout: 415007 20 -rw-r--r-- 1 ubuntu ubuntu 16899 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_es_query.cc 2024-03-27T21:42:44.186 INFO:teuthology.orchestra.run.smithi060.stdout: 414997 4 -rw-r--r-- 1 ubuntu ubuntu 1430 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_dmclock.h 2024-03-27T21:42:44.187 INFO:teuthology.orchestra.run.smithi060.stdout: 415208 4 -rw-r--r-- 1 ubuntu ubuntu 440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_url.h 2024-03-27T21:42:44.187 INFO:teuthology.orchestra.run.smithi060.stdout: 414948 36 -rw-r--r-- 1 ubuntu ubuntu 35503 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_asio_frontend.cc 2024-03-27T21:42:44.187 INFO:teuthology.orchestra.run.smithi060.stdout: 414999 8 -rw-r--r-- 1 ubuntu ubuntu 7482 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_dmclock_async_scheduler.h 2024-03-27T21:42:44.187 INFO:teuthology.orchestra.run.smithi060.stdout: 415124 8 -rw-r--r-- 1 ubuntu ubuntu 6313 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_realm_reloader.cc 2024-03-27T21:42:44.187 INFO:teuthology.orchestra.run.smithi060.stdout: 415116 24 -rw-r--r-- 1 ubuntu ubuntu 21753 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_pubsub.h 2024-03-27T21:42:44.187 INFO:teuthology.orchestra.run.smithi060.stdout: 414928 16 -rwxr-xr-x 1 ubuntu ubuntu 12690 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw-restore-bucket-index 2024-03-27T21:42:44.187 INFO:teuthology.orchestra.run.smithi060.stdout: 414926 4 -rwxr-xr-x 1 ubuntu ubuntu 3196 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw-gap-list-comparator 2024-03-27T21:42:44.187 INFO:teuthology.orchestra.run.smithi060.stdout: 415138 4 -rw-r--r-- 1 ubuntu ubuntu 1739 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_config.h 2024-03-27T21:42:44.188 INFO:teuthology.orchestra.run.smithi060.stdout: 415030 4 -rw-r--r-- 1 ubuntu ubuntu 3086 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_jsonparser.cc 2024-03-27T21:42:44.188 INFO:teuthology.orchestra.run.smithi060.stdout: 415083 4 -rw-r--r-- 1 ubuntu ubuntu 4063 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_object_lock.cc 2024-03-27T21:42:44.188 INFO:teuthology.orchestra.run.smithi060.stdout: 415086 4 -rw-r--r-- 1 ubuntu ubuntu 3650 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_oidc_provider.h 2024-03-27T21:42:44.188 INFO:teuthology.orchestra.run.smithi060.stdout: 415018 8 -rw-r--r-- 1 ubuntu ubuntu 5004 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_frontend.h 2024-03-27T21:42:44.188 INFO:teuthology.orchestra.run.smithi060.stdout: 414969 8 -rw-r--r-- 1 ubuntu ubuntu 6946 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_bucket_types.h 2024-03-27T21:42:44.188 INFO:teuthology.orchestra.run.smithi060.stdout: 415136 12 -rw-r--r-- 1 ubuntu ubuntu 10404 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_client.h 2024-03-27T21:42:44.188 INFO:teuthology.orchestra.run.smithi060.stdout: 414941 40 -rw-r--r-- 1 ubuntu ubuntu 37637 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_amqp.cc 2024-03-27T21:42:44.188 INFO:teuthology.orchestra.run.smithi060.stdout: 415154 8 -rw-r--r-- 1 ubuntu ubuntu 6548 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_role.h 2024-03-27T21:42:44.189 INFO:teuthology.orchestra.run.smithi060.stdout: 415026 48 -rw-r--r-- 1 ubuntu ubuntu 45318 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_iam_policy.cc 2024-03-27T21:42:44.189 INFO:teuthology.orchestra.run.smithi060.stdout: 414951 32 -rw-r--r-- 1 ubuntu ubuntu 31205 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_auth.cc 2024-03-27T21:42:44.189 INFO:teuthology.orchestra.run.smithi060.stdout: 414956 4 -rw-r--r-- 1 ubuntu ubuntu 3080 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_auth_registry.h 2024-03-27T21:42:44.189 INFO:teuthology.orchestra.run.smithi060.stdout: 415205 4 -rw-r--r-- 1 ubuntu ubuntu 228 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_tracer.cc 2024-03-27T21:42:44.189 INFO:teuthology.orchestra.run.smithi060.stdout: 414987 12 -rw-r--r-- 1 ubuntu ubuntu 8860 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_cr_rest.cc 2024-03-27T21:42:44.189 INFO:teuthology.orchestra.run.smithi060.stdout: 415096 12 -rw-r--r-- 1 ubuntu ubuntu 11069 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_perf_counters.cc 2024-03-27T21:42:44.189 INFO:teuthology.orchestra.run.smithi060.stdout: 415117 4 -rw-r--r-- 1 ubuntu ubuntu 2219 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_putobj.cc 2024-03-27T21:42:44.189 INFO:teuthology.orchestra.run.smithi060.stdout: 415132 64 -rw-r--r-- 1 ubuntu ubuntu 62485 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest.cc 2024-03-27T21:42:44.190 INFO:teuthology.orchestra.run.smithi060.stdout: 415137 4 -rw-r--r-- 1 ubuntu ubuntu 1392 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_config.cc 2024-03-27T21:42:44.190 INFO:teuthology.orchestra.run.smithi060.stdout: 414953 12 -rw-r--r-- 1 ubuntu ubuntu 9946 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_auth_filters.h 2024-03-27T21:42:44.190 INFO:teuthology.orchestra.run.smithi060.stdout: 415051 4 -rw-r--r-- 1 ubuntu ubuntu 1697 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_loadgen.h 2024-03-27T21:42:44.190 INFO:teuthology.orchestra.run.smithi060.stdout: 415216 4 -rw-r--r-- 1 ubuntu ubuntu 2124 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_worker.h 2024-03-27T21:42:44.190 INFO:teuthology.orchestra.run.smithi060.stdout: 414919 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/jwt-cpp 2024-03-27T21:42:44.190 INFO:teuthology.orchestra.run.smithi060.stdout: 414921 60 -rw-r--r-- 1 ubuntu ubuntu 57647 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/jwt-cpp/jwt.h 2024-03-27T21:42:44.190 INFO:teuthology.orchestra.run.smithi060.stdout: 414920 8 -rw-r--r-- 1 ubuntu ubuntu 5000 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/jwt-cpp/base.h 2024-03-27T21:42:44.191 INFO:teuthology.orchestra.run.smithi060.stdout: 415103 12 -rw-r--r-- 1 ubuntu ubuntu 9736 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_period_pusher.cc 2024-03-27T21:42:44.191 INFO:teuthology.orchestra.run.smithi060.stdout: 415196 4 -rw-r--r-- 1 ubuntu ubuntu 1426 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_tag.h 2024-03-27T21:42:44.191 INFO:teuthology.orchestra.run.smithi060.stdout: 415080 8 -rw-r--r-- 1 ubuntu ubuntu 6883 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_obj_manifest.cc 2024-03-27T21:42:44.191 INFO:teuthology.orchestra.run.smithi060.stdout: 415066 8 -rw-r--r-- 1 ubuntu ubuntu 5607 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_main.cc 2024-03-27T21:42:44.191 INFO:teuthology.orchestra.run.smithi060.stdout: 414962 8 -rw-r--r-- 1 ubuntu ubuntu 5225 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_bucket.cc 2024-03-27T21:42:44.191 INFO:teuthology.orchestra.run.smithi060.stdout: 415014 4 -rw-r--r-- 1 ubuntu ubuntu 1934 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_flight_frontend.h 2024-03-27T21:42:44.191 INFO:teuthology.orchestra.run.smithi060.stdout: 415098 12 -rw-r--r-- 1 ubuntu ubuntu 9693 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_period.cc 2024-03-27T21:42:44.192 INFO:teuthology.orchestra.run.smithi060.stdout: 415151 16 -rw-r--r-- 1 ubuntu ubuntu 13566 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_ratelimit.cc 2024-03-27T21:42:44.192 INFO:teuthology.orchestra.run.smithi060.stdout: 414976 60 -rw-r--r-- 1 ubuntu ubuntu 61352 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_common.h 2024-03-27T21:42:44.192 INFO:teuthology.orchestra.run.smithi060.stdout: 414991 4 -rw-r--r-- 1 ubuntu ubuntu 2505 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_crypt_sanitize.cc 2024-03-27T21:42:44.192 INFO:teuthology.orchestra.run.smithi060.stdout: 414984 8 -rw-r--r-- 1 ubuntu ubuntu 7425 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_cors_s3.cc 2024-03-27T21:42:44.192 INFO:teuthology.orchestra.run.smithi060.stdout: 415221 4 -rw-r--r-- 1 ubuntu ubuntu 1412 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_zone_features.h 2024-03-27T21:42:44.192 INFO:teuthology.orchestra.run.smithi060.stdout: 415188 28 -rw-r--r-- 1 ubuntu ubuntu 25478 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_swift_auth.cc 2024-03-27T21:42:44.192 INFO:teuthology.orchestra.run.smithi060.stdout: 415002 4 -rw-r--r-- 1 ubuntu ubuntu 3159 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_dmclock_scheduler_ctx.h 2024-03-27T21:42:44.192 INFO:teuthology.orchestra.run.smithi060.stdout: 414990 8 -rw-r--r-- 1 ubuntu ubuntu 7095 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_crypt.h 2024-03-27T21:42:44.192 INFO:teuthology.orchestra.run.smithi060.stdout: 415131 4 -rw-r--r-- 1 ubuntu ubuntu 475 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_resolve.h 2024-03-27T21:42:44.193 INFO:teuthology.orchestra.run.smithi060.stdout: 415209 8 -rw-r--r-- 1 ubuntu ubuntu 6189 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_usage.cc 2024-03-27T21:42:44.193 INFO:teuthology.orchestra.run.smithi060.stdout: 415052 4 -rw-r--r-- 1 ubuntu ubuntu 3325 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_loadgen_process.cc 2024-03-27T21:42:44.193 INFO:teuthology.orchestra.run.smithi060.stdout: 414968 4 -rw-r--r-- 1 ubuntu ubuntu 3750 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_bucket_sync_cache.h 2024-03-27T21:42:44.193 INFO:teuthology.orchestra.run.smithi060.stdout: 414992 4 -rw-r--r-- 1 ubuntu ubuntu 1789 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_crypt_sanitize.h 2024-03-27T21:42:44.193 INFO:teuthology.orchestra.run.smithi060.stdout: 415107 4 -rw-r--r-- 1 ubuntu ubuntu 1557 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_policy_s3.h 2024-03-27T21:42:44.193 INFO:teuthology.orchestra.run.smithi060.stdout: 415023 4 -rw-r--r-- 1 ubuntu ubuntu 680 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_http_client_curl.h 2024-03-27T21:42:44.193 INFO:teuthology.orchestra.run.smithi060.stdout: 415125 4 -rw-r--r-- 1 ubuntu ubuntu 2367 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_realm_reloader.h 2024-03-27T21:42:44.193 INFO:teuthology.orchestra.run.smithi060.stdout: 415027 20 -rw-r--r-- 1 ubuntu ubuntu 19530 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_iam_policy.h 2024-03-27T21:42:44.194 INFO:teuthology.orchestra.run.smithi060.stdout: 414973 16 -rw-r--r-- 1 ubuntu ubuntu 15240 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_client_io.h 2024-03-27T21:42:44.194 INFO:teuthology.orchestra.run.smithi060.stdout: 415024 4 -rw-r--r-- 1 ubuntu ubuntu 1367 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_http_client_types.h 2024-03-27T21:42:44.194 INFO:teuthology.orchestra.run.smithi060.stdout: 415166 8 -rw-r--r-- 1 ubuntu ubuntu 5689 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_zero.cc 2024-03-27T21:42:44.194 INFO:teuthology.orchestra.run.smithi060.stdout: 415022 4 -rw-r--r-- 1 ubuntu ubuntu 2530 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_http_client_curl.cc 2024-03-27T21:42:44.194 INFO:teuthology.orchestra.run.smithi060.stdout: 414985 4 -rw-r--r-- 1 ubuntu ubuntu 1504 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_cors_s3.h 2024-03-27T21:42:44.194 INFO:teuthology.orchestra.run.smithi060.stdout: 415160 96 -rw-r--r-- 1 ubuntu ubuntu 96035 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_swift.cc 2024-03-27T21:42:44.194 INFO:teuthology.orchestra.run.smithi060.stdout: 415123 8 -rw-r--r-- 1 ubuntu ubuntu 7285 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_realm.cc 2024-03-27T21:42:44.194 INFO:teuthology.orchestra.run.smithi060.stdout: 415113 4 -rw-r--r-- 1 ubuntu ubuntu 1505 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_public_access.cc 2024-03-27T21:42:44.194 INFO:teuthology.orchestra.run.smithi060.stdout: 415045 4 -rw-r--r-- 1 ubuntu ubuntu 3669 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_ldap.cc 2024-03-27T21:42:44.195 INFO:teuthology.orchestra.run.smithi060.stdout: 415097 4 -rw-r--r-- 1 ubuntu ubuntu 2461 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_perf_counters.h 2024-03-27T21:42:44.195 INFO:teuthology.orchestra.run.smithi060.stdout: 415042 20 -rw-r--r-- 1 ubuntu ubuntu 16562 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_lc.h 2024-03-27T21:42:44.195 INFO:teuthology.orchestra.run.smithi060.stdout: 414931 20 -rw-r--r-- 1 ubuntu ubuntu 17756 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_acl_s3.cc 2024-03-27T21:42:44.195 INFO:teuthology.orchestra.run.smithi060.stdout: 414960 8 -rw-r--r-- 1 ubuntu ubuntu 4934 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_basic_types.cc 2024-03-27T21:42:44.195 INFO:teuthology.orchestra.run.smithi060.stdout: 415220 44 -rw-r--r-- 1 ubuntu ubuntu 44466 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_zone.cc 2024-03-27T21:42:44.195 INFO:teuthology.orchestra.run.smithi060.stdout: 415206 4 -rw-r--r-- 1 ubuntu ubuntu 905 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_tracer.h 2024-03-27T21:42:44.195 INFO:teuthology.orchestra.run.smithi060.stdout: 414938 4 -rw-r--r-- 1 ubuntu ubuntu 3197 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_aio.h 2024-03-27T21:42:44.195 INFO:teuthology.orchestra.run.smithi060.stdout: 414964 4 -rw-r--r-- 1 ubuntu ubuntu 1677 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_bucket_encryption.cc 2024-03-27T21:42:44.196 INFO:teuthology.orchestra.run.smithi060.stdout: 414967 12 -rw-r--r-- 1 ubuntu ubuntu 10089 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_bucket_layout.h 2024-03-27T21:42:44.196 INFO:teuthology.orchestra.run.smithi060.stdout: 415009 76 -rw-r--r-- 1 ubuntu ubuntu 75859 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_file.cc 2024-03-27T21:42:44.196 INFO:teuthology.orchestra.run.smithi060.stdout: 414949 4 -rw-r--r-- 1 ubuntu ubuntu 691 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_asio_frontend.h 2024-03-27T21:42:44.196 INFO:teuthology.orchestra.run.smithi060.stdout: 415168 16 -rw-r--r-- 1 ubuntu ubuntu 12464 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_role.cc 2024-03-27T21:42:44.196 INFO:teuthology.orchestra.run.smithi060.stdout: 415087 288 -rw-r--r-- 1 ubuntu ubuntu 290872 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_op.cc 2024-03-27T21:42:44.196 INFO:teuthology.orchestra.run.smithi060.stdout: 415224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services 2024-03-27T21:42:44.196 INFO:teuthology.orchestra.run.smithi060.stdout: 415255 8 -rw-r--r-- 1 ubuntu ubuntu 6751 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_meta_be_sobj.h 2024-03-27T21:42:44.196 INFO:teuthology.orchestra.run.smithi060.stdout: 415265 4 -rw-r--r-- 1 ubuntu ubuntu 1357 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_role_rados.h 2024-03-27T21:42:44.196 INFO:teuthology.orchestra.run.smithi060.stdout: 415259 8 -rw-r--r-- 1 ubuntu ubuntu 5105 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_otp.cc 2024-03-27T21:42:44.197 INFO:teuthology.orchestra.run.smithi060.stdout: 415274 4 -rw-r--r-- 1 ubuntu ubuntu 991 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_sys_obj_core_types.h 2024-03-27T21:42:44.197 INFO:teuthology.orchestra.run.smithi060.stdout: 415226 20 -rw-r--r-- 1 ubuntu ubuntu 16852 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_bi_rados.cc 2024-03-27T21:42:44.197 INFO:teuthology.orchestra.run.smithi060.stdout: 415269 8 -rw-r--r-- 1 ubuntu ubuntu 7183 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_sys_obj.h 2024-03-27T21:42:44.197 INFO:teuthology.orchestra.run.smithi060.stdout: 415276 4 -rw-r--r-- 1 ubuntu ubuntu 783 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_tier_rados.cc 2024-03-27T21:42:44.197 INFO:teuthology.orchestra.run.smithi060.stdout: 415279 8 -rw-r--r-- 1 ubuntu ubuntu 5407 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_user.h 2024-03-27T21:42:44.197 INFO:teuthology.orchestra.run.smithi060.stdout: 415229 4 -rw-r--r-- 1 ubuntu ubuntu 1886 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_bilog_rados.h 2024-03-27T21:42:44.197 INFO:teuthology.orchestra.run.smithi060.stdout: 415240 4 -rw-r--r-- 1 ubuntu ubuntu 697 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_config_key.h 2024-03-27T21:42:44.197 INFO:teuthology.orchestra.run.smithi060.stdout: 415251 4 -rw-r--r-- 1 ubuntu ubuntu 2284 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_meta_be_otp.cc 2024-03-27T21:42:44.197 INFO:teuthology.orchestra.run.smithi060.stdout: 415248 4 -rw-r--r-- 1 ubuntu ubuntu 1098 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_meta.h 2024-03-27T21:42:44.198 INFO:teuthology.orchestra.run.smithi060.stdout: 415263 4 -rw-r--r-- 1 ubuntu ubuntu 463 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_quota.h 2024-03-27T21:42:44.198 INFO:teuthology.orchestra.run.smithi060.stdout: 415243 4 -rw-r--r-- 1 ubuntu ubuntu 1012 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_finisher.cc 2024-03-27T21:42:44.198 INFO:teuthology.orchestra.run.smithi060.stdout: 415258 4 -rw-r--r-- 1 ubuntu ubuntu 2719 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_notify.h 2024-03-27T21:42:44.198 INFO:teuthology.orchestra.run.smithi060.stdout: 415271 8 -rw-r--r-- 1 ubuntu ubuntu 6952 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_sys_obj_cache.h 2024-03-27T21:42:44.198 INFO:teuthology.orchestra.run.smithi060.stdout: 415234 4 -rw-r--r-- 1 ubuntu ubuntu 1906 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_bucket_sync.h 2024-03-27T21:42:44.198 INFO:teuthology.orchestra.run.smithi060.stdout: 415247 4 -rw-r--r-- 1 ubuntu ubuntu 1039 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_meta.cc 2024-03-27T21:42:44.198 INFO:teuthology.orchestra.run.smithi060.stdout: 415245 20 -rw-r--r-- 1 ubuntu ubuntu 18075 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_mdlog.cc 2024-03-27T21:42:44.199 INFO:teuthology.orchestra.run.smithi060.stdout: 415281 12 -rw-r--r-- 1 ubuntu ubuntu 8805 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_user_rados.h 2024-03-27T21:42:44.199 INFO:teuthology.orchestra.run.smithi060.stdout: 415282 32 -rw-r--r-- 1 ubuntu ubuntu 30535 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_zone.cc 2024-03-27T21:42:44.199 INFO:teuthology.orchestra.run.smithi060.stdout: 415239 8 -rw-r--r-- 1 ubuntu ubuntu 5946 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_cls.h 2024-03-27T21:42:44.199 INFO:teuthology.orchestra.run.smithi060.stdout: 415264 4 -rw-r--r-- 1 ubuntu ubuntu 2264 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_role_rados.cc 2024-03-27T21:42:44.199 INFO:teuthology.orchestra.run.smithi060.stdout: 415260 4 -rw-r--r-- 1 ubuntu ubuntu 2780 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_otp.h 2024-03-27T21:42:44.199 INFO:teuthology.orchestra.run.smithi060.stdout: 415261 4 -rw-r--r-- 1 ubuntu ubuntu 740 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_otp_types.h 2024-03-27T21:42:44.199 INFO:teuthology.orchestra.run.smithi060.stdout: 415280 32 -rw-r--r-- 1 ubuntu ubuntu 30007 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_user_rados.cc 2024-03-27T21:42:44.199 INFO:teuthology.orchestra.run.smithi060.stdout: 415238 16 -rw-r--r-- 1 ubuntu ubuntu 12319 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_cls.cc 2024-03-27T21:42:44.199 INFO:teuthology.orchestra.run.smithi060.stdout: 415268 8 -rw-r--r-- 1 ubuntu ubuntu 5726 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_sys_obj.cc 2024-03-27T21:42:44.200 INFO:teuthology.orchestra.run.smithi060.stdout: 415283 8 -rw-r--r-- 1 ubuntu ubuntu 5957 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_zone.h 2024-03-27T21:42:44.200 INFO:teuthology.orchestra.run.smithi060.stdout: 415228 8 -rw-r--r-- 1 ubuntu ubuntu 7292 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_bilog_rados.cc 2024-03-27T21:42:44.200 INFO:teuthology.orchestra.run.smithi060.stdout: 415256 4 -rw-r--r-- 1 ubuntu ubuntu 528 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_meta_be_types.h 2024-03-27T21:42:44.200 INFO:teuthology.orchestra.run.smithi060.stdout: 415242 4 -rw-r--r-- 1 ubuntu ubuntu 1055 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_config_key_rados.h 2024-03-27T21:42:44.200 INFO:teuthology.orchestra.run.smithi060.stdout: 415225 4 -rw-r--r-- 1 ubuntu ubuntu 1402 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_bi.h 2024-03-27T21:42:44.200 INFO:teuthology.orchestra.run.smithi060.stdout: 415285 4 -rw-r--r-- 1 ubuntu ubuntu 798 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_zone_utils.h 2024-03-27T21:42:44.200 INFO:teuthology.orchestra.run.smithi060.stdout: 415253 4 -rw-r--r-- 1 ubuntu ubuntu 578 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_meta_be_params.h 2024-03-27T21:42:44.200 INFO:teuthology.orchestra.run.smithi060.stdout: 415275 4 -rw-r--r-- 1 ubuntu ubuntu 230 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_sys_obj_types.h 2024-03-27T21:42:44.200 INFO:teuthology.orchestra.run.smithi060.stdout: 415262 4 -rw-r--r-- 1 ubuntu ubuntu 443 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_quota.cc 2024-03-27T21:42:44.201 INFO:teuthology.orchestra.run.smithi060.stdout: 415227 8 -rw-r--r-- 1 ubuntu ubuntu 5798 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_bi_rados.h 2024-03-27T21:42:44.201 INFO:teuthology.orchestra.run.smithi060.stdout: 415257 16 -rw-r--r-- 1 ubuntu ubuntu 13323 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_notify.cc 2024-03-27T21:42:44.201 INFO:teuthology.orchestra.run.smithi060.stdout: 415235 32 -rw-r--r-- 1 ubuntu ubuntu 29479 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_bucket_sync_sobj.cc 2024-03-27T21:42:44.201 INFO:teuthology.orchestra.run.smithi060.stdout: 415231 8 -rw-r--r-- 1 ubuntu ubuntu 5177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_bucket.h 2024-03-27T21:42:44.201 INFO:teuthology.orchestra.run.smithi060.stdout: 415266 4 -rw-r--r-- 1 ubuntu ubuntu 1295 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_sync_modules.cc 2024-03-27T21:42:44.201 INFO:teuthology.orchestra.run.smithi060.stdout: 415246 8 -rw-r--r-- 1 ubuntu ubuntu 4327 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_mdlog.h 2024-03-27T21:42:44.201 INFO:teuthology.orchestra.run.smithi060.stdout: 415273 8 -rw-r--r-- 1 ubuntu ubuntu 5955 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_sys_obj_core.h 2024-03-27T21:42:44.201 INFO:teuthology.orchestra.run.smithi060.stdout: 415230 4 -rw-r--r-- 1 ubuntu ubuntu 488 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_bucket.cc 2024-03-27T21:42:44.202 INFO:teuthology.orchestra.run.smithi060.stdout: 415249 8 -rw-r--r-- 1 ubuntu ubuntu 5858 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_meta_be.cc 2024-03-27T21:42:44.202 INFO:teuthology.orchestra.run.smithi060.stdout: 415237 4 -rw-r--r-- 1 ubuntu ubuntu 1037 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_bucket_types.h 2024-03-27T21:42:44.202 INFO:teuthology.orchestra.run.smithi060.stdout: 415232 24 -rw-r--r-- 1 ubuntu ubuntu 22442 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_bucket_sobj.cc 2024-03-27T21:42:44.202 INFO:teuthology.orchestra.run.smithi060.stdout: 415277 4 -rw-r--r-- 1 ubuntu ubuntu 4083 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_tier_rados.h 2024-03-27T21:42:44.202 INFO:teuthology.orchestra.run.smithi060.stdout: 415272 20 -rw-r--r-- 1 ubuntu ubuntu 18047 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_sys_obj_core.cc 2024-03-27T21:42:44.202 INFO:teuthology.orchestra.run.smithi060.stdout: 415270 20 -rw-r--r-- 1 ubuntu ubuntu 19078 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_sys_obj_cache.cc 2024-03-27T21:42:44.202 INFO:teuthology.orchestra.run.smithi060.stdout: 415244 4 -rw-r--r-- 1 ubuntu ubuntu 898 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_finisher.h 2024-03-27T21:42:44.202 INFO:teuthology.orchestra.run.smithi060.stdout: 415254 8 -rw-r--r-- 1 ubuntu ubuntu 8042 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_meta_be_sobj.cc 2024-03-27T21:42:44.203 INFO:teuthology.orchestra.run.smithi060.stdout: 415267 4 -rw-r--r-- 1 ubuntu ubuntu 790 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_sync_modules.h 2024-03-27T21:42:44.203 INFO:teuthology.orchestra.run.smithi060.stdout: 415284 4 -rw-r--r-- 1 ubuntu ubuntu 1989 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_zone_utils.cc 2024-03-27T21:42:44.203 INFO:teuthology.orchestra.run.smithi060.stdout: 415250 12 -rw-r--r-- 1 ubuntu ubuntu 9532 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_meta_be.h 2024-03-27T21:42:44.203 INFO:teuthology.orchestra.run.smithi060.stdout: 415233 8 -rw-r--r-- 1 ubuntu ubuntu 7290 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_bucket_sobj.h 2024-03-27T21:42:44.203 INFO:teuthology.orchestra.run.smithi060.stdout: 415236 8 -rw-r--r-- 1 ubuntu ubuntu 4274 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_bucket_sync_sobj.h 2024-03-27T21:42:44.203 INFO:teuthology.orchestra.run.smithi060.stdout: 415252 4 -rw-r--r-- 1 ubuntu ubuntu 2477 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_meta_be_otp.h 2024-03-27T21:42:44.203 INFO:teuthology.orchestra.run.smithi060.stdout: 415278 4 -rw-r--r-- 1 ubuntu ubuntu 231 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_user.cc 2024-03-27T21:42:44.203 INFO:teuthology.orchestra.run.smithi060.stdout: 415241 4 -rw-r--r-- 1 ubuntu ubuntu 1224 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/services/svc_config_key_rados.cc 2024-03-27T21:42:44.203 INFO:teuthology.orchestra.run.smithi060.stdout: 415207 4 -rw-r--r-- 1 ubuntu ubuntu 1452 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_url.cc 2024-03-27T21:42:44.204 INFO:teuthology.orchestra.run.smithi060.stdout: 415029 4 -rw-r--r-- 1 ubuntu ubuntu 2468 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_iam_policy_keywords.h 2024-03-27T21:42:44.204 INFO:teuthology.orchestra.run.smithi060.stdout: 415031 32 -rw-r--r-- 1 ubuntu ubuntu 28787 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_kafka.cc 2024-03-27T21:42:44.204 INFO:teuthology.orchestra.run.smithi060.stdout: 414749 4 -rw-r--r-- 1 ubuntu ubuntu 1503 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/MAINTAINERS.md 2024-03-27T21:42:44.204 INFO:teuthology.orchestra.run.smithi060.stdout: 415185 8 -rw-r--r-- 1 ubuntu ubuntu 6361 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_string.h 2024-03-27T21:42:44.204 INFO:teuthology.orchestra.run.smithi060.stdout: 415133 28 -rw-r--r-- 1 ubuntu ubuntu 25007 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest.h 2024-03-27T21:42:44.204 INFO:teuthology.orchestra.run.smithi060.stdout: 414944 12 -rw-r--r-- 1 ubuntu ubuntu 11916 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_arn.cc 2024-03-27T21:42:44.204 INFO:teuthology.orchestra.run.smithi060.stdout: 415102 4 -rw-r--r-- 1 ubuntu ubuntu 597 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_period_puller.h 2024-03-27T21:42:44.204 INFO:teuthology.orchestra.run.smithi060.stdout: 415006 4 -rw-r--r-- 1 ubuntu ubuntu 2783 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_es_main.cc 2024-03-27T21:42:44.205 INFO:teuthology.orchestra.run.smithi060.stdout: 415040 4 -rw-r--r-- 1 ubuntu ubuntu 2823 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_kms.h 2024-03-27T21:42:44.205 INFO:teuthology.orchestra.run.smithi060.stdout: 415021 12 -rw-r--r-- 1 ubuntu ubuntu 9106 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_http_client.h 2024-03-27T21:42:44.205 INFO:teuthology.orchestra.run.smithi060.stdout: 415062 4 -rw-r--r-- 1 ubuntu ubuntu 530 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_lua_request.h 2024-03-27T21:42:44.205 INFO:teuthology.orchestra.run.smithi060.stdout: 414982 8 -rw-r--r-- 1 ubuntu ubuntu 6518 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_cors.cc 2024-03-27T21:42:44.205 INFO:teuthology.orchestra.run.smithi060.stdout: 415182 4 -rw-r--r-- 1 ubuntu ubuntu 2217 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_signal.cc 2024-03-27T21:42:44.205 INFO:teuthology.orchestra.run.smithi060.stdout: 415057 8 -rw-r--r-- 1 ubuntu ubuntu 6256 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_lua_background.cc 2024-03-27T21:42:44.205 INFO:teuthology.orchestra.run.smithi060.stdout: 414952 32 -rw-r--r-- 1 ubuntu ubuntu 28822 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_auth.h 2024-03-27T21:42:44.206 INFO:teuthology.orchestra.run.smithi060.stdout: 415204 4 -rw-r--r-- 1 ubuntu ubuntu 1639 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_torrent.h 2024-03-27T21:42:44.206 INFO:teuthology.orchestra.run.smithi060.stdout: 415128 4 -rw-r--r-- 1 ubuntu ubuntu 430 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_req_context.h 2024-03-27T21:42:44.206 INFO:teuthology.orchestra.run.smithi060.stdout: 415017 4 -rw-r--r-- 1 ubuntu ubuntu 2318 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_frontend.cc 2024-03-27T21:42:44.206 INFO:teuthology.orchestra.run.smithi060.stdout: 414946 8 -rw-r--r-- 1 ubuntu ubuntu 5296 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_asio_client.cc 2024-03-27T21:42:44.206 INFO:teuthology.orchestra.run.smithi060.stdout: 414994 8 -rw-r--r-- 1 ubuntu ubuntu 5434 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_data_access.cc 2024-03-27T21:42:44.206 INFO:teuthology.orchestra.run.smithi060.stdout: 414979 4 -rw-r--r-- 1 ubuntu ubuntu 2124 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_compression_types.h 2024-03-27T21:42:44.206 INFO:teuthology.orchestra.run.smithi060.stdout: 415048 8 -rw-r--r-- 1 ubuntu ubuntu 5464 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_lib.h 2024-03-27T21:42:44.206 INFO:teuthology.orchestra.run.smithi060.stdout: 415004 4 -rw-r--r-- 1 ubuntu ubuntu 2478 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_dmclock_sync_scheduler.h 2024-03-27T21:42:44.207 INFO:teuthology.orchestra.run.smithi060.stdout: 415214 12 -rw-r--r-- 1 ubuntu ubuntu 11080 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_website.cc 2024-03-27T21:42:44.207 INFO:teuthology.orchestra.run.smithi060.stdout: 415085 8 -rw-r--r-- 1 ubuntu ubuntu 4483 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_oidc_provider.cc 2024-03-27T21:42:44.207 INFO:teuthology.orchestra.run.smithi060.stdout: 415068 8 -rw-r--r-- 1 ubuntu ubuntu 5465 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_mdlog.h 2024-03-27T21:42:44.207 INFO:teuthology.orchestra.run.smithi060.stdout: 415134 4 -rw-r--r-- 1 ubuntu ubuntu 261 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_admin.h 2024-03-27T21:42:44.207 INFO:teuthology.orchestra.run.smithi060.stdout: 415149 56 -rw-r--r-- 1 ubuntu ubuntu 53501 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_pubsub.cc 2024-03-27T21:42:44.207 INFO:teuthology.orchestra.run.smithi060.stdout: 415172 8 -rw-r--r-- 1 ubuntu ubuntu 6862 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_s3select_private.h 2024-03-27T21:42:44.207 INFO:teuthology.orchestra.run.smithi060.stdout: 415039 40 -rw-r--r-- 1 ubuntu ubuntu 40817 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_kms.cc 2024-03-27T21:42:44.207 INFO:teuthology.orchestra.run.smithi060.stdout: 414980 32 -rw-r--r-- 1 ubuntu ubuntu 29529 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_coroutine.cc 2024-03-27T21:42:44.207 INFO:teuthology.orchestra.run.smithi060.stdout: 415005 4 -rw-r--r-- 1 ubuntu ubuntu 3768 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_env.cc 2024-03-27T21:42:44.208 INFO:teuthology.orchestra.run.smithi060.stdout: 415163 4 -rw-r--r-- 1 ubuntu ubuntu 876 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_usage.h 2024-03-27T21:42:44.208 INFO:teuthology.orchestra.run.smithi060.stdout: 414981 20 -rw-r--r-- 1 ubuntu ubuntu 19440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_coroutine.h 2024-03-27T21:42:44.208 INFO:teuthology.orchestra.run.smithi060.stdout: 415044 4 -rw-r--r-- 1 ubuntu ubuntu 2063 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_lc_s3.h 2024-03-27T21:42:44.208 INFO:teuthology.orchestra.run.smithi060.stdout: 415144 4 -rw-r--r-- 1 ubuntu ubuntu 839 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_info.h 2024-03-27T21:42:44.208 INFO:teuthology.orchestra.run.smithi060.stdout: 414925 16 -rwxr-xr-x 1 ubuntu ubuntu 12828 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw-gap-list 2024-03-27T21:42:44.208 INFO:teuthology.orchestra.run.smithi060.stdout: 415094 4 -rw-r--r-- 1 ubuntu ubuntu 1419 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_os_lib.cc 2024-03-27T21:42:44.208 INFO:teuthology.orchestra.run.smithi060.stdout: 414965 4 -rw-r--r-- 1 ubuntu ubuntu 3562 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_bucket_encryption.h 2024-03-27T21:42:44.208 INFO:teuthology.orchestra.run.smithi060.stdout: 415072 4 -rw-r--r-- 1 ubuntu ubuntu 2661 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_multi.cc 2024-03-27T21:42:44.209 INFO:teuthology.orchestra.run.smithi060.stdout: 415210 4 -rw-r--r-- 1 ubuntu ubuntu 897 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_usage.h 2024-03-27T21:42:44.209 INFO:teuthology.orchestra.run.smithi060.stdout: 415037 20 -rw-r--r-- 1 ubuntu ubuntu 20391 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_kmip_client_impl.cc 2024-03-27T21:42:44.209 INFO:teuthology.orchestra.run.smithi060.stdout: 415130 4 -rw-r--r-- 1 ubuntu ubuntu 860 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_resolve.cc 2024-03-27T21:42:44.209 INFO:teuthology.orchestra.run.smithi060.stdout: 414922 4 -rw-r--r-- 1 ubuntu ubuntu 1794 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/librgw.cc 2024-03-27T21:42:44.209 INFO:teuthology.orchestra.run.smithi060.stdout: 415192 4 -rw-r--r-- 1 ubuntu ubuntu 1239 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_sync_checkpoint.h 2024-03-27T21:42:44.209 INFO:teuthology.orchestra.run.smithi060.stdout: 414957 56 -rw-r--r-- 1 ubuntu ubuntu 56453 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_auth_s3.cc 2024-03-27T21:42:44.209 INFO:teuthology.orchestra.run.smithi060.stdout: 415170 40 -rw-r--r-- 1 ubuntu ubuntu 38663 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_s3select.cc 2024-03-27T21:42:44.209 INFO:teuthology.orchestra.run.smithi060.stdout: 415153 32 -rw-r--r-- 1 ubuntu ubuntu 29194 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_role.cc 2024-03-27T21:42:44.210 INFO:teuthology.orchestra.run.smithi060.stdout: 415110 16 -rw-r--r-- 1 ubuntu ubuntu 15020 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_process.cc 2024-03-27T21:42:44.210 INFO:teuthology.orchestra.run.smithi060.stdout: 415150 4 -rw-r--r-- 1 ubuntu ubuntu 1575 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_pubsub.h 2024-03-27T21:42:44.210 INFO:teuthology.orchestra.run.smithi060.stdout: 415074 4 -rw-r--r-- 1 ubuntu ubuntu 1815 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_multi_del.cc 2024-03-27T21:42:44.210 INFO:teuthology.orchestra.run.smithi060.stdout: 415171 4 -rw-r--r-- 1 ubuntu ubuntu 181 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_s3select.h 2024-03-27T21:42:44.210 INFO:teuthology.orchestra.run.smithi060.stdout: 415173 16 -rw-r--r-- 1 ubuntu ubuntu 12770 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_sal.cc 2024-03-27T21:42:44.210 INFO:teuthology.orchestra.run.smithi060.stdout: 415088 80 -rw-r--r-- 1 ubuntu ubuntu 79551 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_op.h 2024-03-27T21:42:44.210 INFO:teuthology.orchestra.run.smithi060.stdout: 415081 16 -rw-r--r-- 1 ubuntu ubuntu 15609 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_obj_types.h 2024-03-27T21:42:44.210 INFO:teuthology.orchestra.run.smithi060.stdout: 415008 4 -rw-r--r-- 1 ubuntu ubuntu 3705 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_es_query.h 2024-03-27T21:42:44.211 INFO:teuthology.orchestra.run.smithi060.stdout: 414947 4 -rw-r--r-- 1 ubuntu ubuntu 1640 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_asio_client.h 2024-03-27T21:42:44.211 INFO:teuthology.orchestra.run.smithi060.stdout: 415212 4 -rw-r--r-- 1 ubuntu ubuntu 2984 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_user_types.h 2024-03-27T21:42:44.211 INFO:teuthology.orchestra.run.smithi060.stdout: 415043 12 -rw-r--r-- 1 ubuntu ubuntu 11154 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_lc_s3.cc 2024-03-27T21:42:44.211 INFO:teuthology.orchestra.run.smithi060.stdout: 415139 24 -rw-r--r-- 1 ubuntu ubuntu 21958 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_conn.cc 2024-03-27T21:42:44.211 INFO:teuthology.orchestra.run.smithi060.stdout: 415056 4 -rw-r--r-- 1 ubuntu ubuntu 2458 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_lua.h 2024-03-27T21:42:44.211 INFO:teuthology.orchestra.run.smithi060.stdout: 415121 4 -rw-r--r-- 1 ubuntu ubuntu 2097 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_quota_types.h 2024-03-27T21:42:44.211 INFO:teuthology.orchestra.run.smithi060.stdout: 415071 20 -rw-r--r-- 1 ubuntu ubuntu 18662 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_metadata.cc 2024-03-27T21:42:44.211 INFO:teuthology.orchestra.run.smithi060.stdout: 414986 4 -rw-r--r-- 1 ubuntu ubuntu 2339 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_cors_swift.h 2024-03-27T21:42:44.212 INFO:teuthology.orchestra.run.smithi060.stdout: 415010 76 -rw-r--r-- 1 ubuntu ubuntu 77366 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_file_int.h 2024-03-27T21:42:44.212 INFO:teuthology.orchestra.run.smithi060.stdout: 414929 16 -rw-r--r-- 1 ubuntu ubuntu 13432 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_acl.cc 2024-03-27T21:42:44.212 INFO:teuthology.orchestra.run.smithi060.stdout: 414923 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/picojson 2024-03-27T21:42:44.212 INFO:teuthology.orchestra.run.smithi060.stdout: 414924 36 -rw-r--r-- 1 ubuntu ubuntu 33607 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/picojson/picojson.h 2024-03-27T21:42:44.212 INFO:teuthology.orchestra.run.smithi060.stdout: 415127 4 -rw-r--r-- 1 ubuntu ubuntu 1909 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_realm_watcher.h 2024-03-27T21:42:44.212 INFO:teuthology.orchestra.run.smithi060.stdout: 415141 8 -rw-r--r-- 1 ubuntu ubuntu 4169 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_iam.cc 2024-03-27T21:42:44.212 INFO:teuthology.orchestra.run.smithi060.stdout: 415218 12 -rw-r--r-- 1 ubuntu ubuntu 11069 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_xml.h 2024-03-27T21:42:44.212 INFO:teuthology.orchestra.run.smithi060.stdout: 415191 12 -rw-r--r-- 1 ubuntu ubuntu 9380 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_sync_checkpoint.cc 2024-03-27T21:42:44.213 INFO:teuthology.orchestra.run.smithi060.stdout: 415146 4 -rw-r--r-- 1 ubuntu ubuntu 3039 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_rest_metadata.h 2024-03-27T21:42:44.213 INFO:teuthology.orchestra.run.smithi060.stdout: 414961 8 -rw-r--r-- 1 ubuntu ubuntu 7531 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_basic_types.h 2024-03-27T21:42:44.213 INFO:teuthology.orchestra.run.smithi060.stdout: 415101 4 -rw-r--r-- 1 ubuntu ubuntu 3559 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_period_puller.cc 2024-03-27T21:42:44.213 INFO:teuthology.orchestra.run.smithi060.stdout: 415049 4 -rw-r--r-- 1 ubuntu ubuntu 3019 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rgw/rgw_lib_frontend.h 2024-03-27T21:42:44.213 INFO:teuthology.orchestra.run.smithi060.stdout: 410899 4 drwxr-xr-x 22 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd 2024-03-27T21:42:44.213 INFO:teuthology.orchestra.run.smithi060.stdout: 410916 36 -rw-r--r-- 1 ubuntu ubuntu 33924 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/ImageCtx.cc 2024-03-27T21:42:44.213 INFO:teuthology.orchestra.run.smithi060.stdout: 411227 260 -rw-r--r-- 1 ubuntu ubuntu 263565 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/librbd.cc 2024-03-27T21:42:44.213 INFO:teuthology.orchestra.run.smithi060.stdout: 410913 8 -rw-r--r-- 1 ubuntu ubuntu 4124 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/ExclusiveLock.h 2024-03-27T21:42:44.214 INFO:teuthology.orchestra.run.smithi060.stdout: 411390 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/plugin 2024-03-27T21:42:44.214 INFO:teuthology.orchestra.run.smithi060.stdout: 411397 4 -rw-r--r-- 1 ubuntu ubuntu 1455 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/plugin/WriteLogImageCache.h 2024-03-27T21:42:44.214 INFO:teuthology.orchestra.run.smithi060.stdout: 411391 4 -rw-r--r-- 1 ubuntu ubuntu 2862 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/plugin/Api.cc 2024-03-27T21:42:44.214 INFO:teuthology.orchestra.run.smithi060.stdout: 411394 4 -rw-r--r-- 1 ubuntu ubuntu 927 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/plugin/ParentCache.h 2024-03-27T21:42:44.214 INFO:teuthology.orchestra.run.smithi060.stdout: 411396 4 -rw-r--r-- 1 ubuntu ubuntu 3129 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/plugin/WriteLogImageCache.cc 2024-03-27T21:42:44.214 INFO:teuthology.orchestra.run.smithi060.stdout: 411395 4 -rw-r--r-- 1 ubuntu ubuntu 1152 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/plugin/Types.h 2024-03-27T21:42:44.214 INFO:teuthology.orchestra.run.smithi060.stdout: 411392 4 -rw-r--r-- 1 ubuntu ubuntu 2076 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/plugin/Api.h 2024-03-27T21:42:44.215 INFO:teuthology.orchestra.run.smithi060.stdout: 411393 4 -rw-r--r-- 1 ubuntu ubuntu 2338 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/plugin/ParentCache.cc 2024-03-27T21:42:44.215 INFO:teuthology.orchestra.run.smithi060.stdout: 410912 12 -rw-r--r-- 1 ubuntu ubuntu 11476 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/ExclusiveLock.cc 2024-03-27T21:42:44.215 INFO:teuthology.orchestra.run.smithi060.stdout: 410915 4 -rw-r--r-- 1 ubuntu ubuntu 359 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/Features.h 2024-03-27T21:42:44.215 INFO:teuthology.orchestra.run.smithi060.stdout: 411403 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/trash_watcher 2024-03-27T21:42:44.215 INFO:teuthology.orchestra.run.smithi060.stdout: 411404 4 -rw-r--r-- 1 ubuntu ubuntu 3267 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/trash_watcher/Types.cc 2024-03-27T21:42:44.215 INFO:teuthology.orchestra.run.smithi060.stdout: 411405 4 -rw-r--r-- 1 ubuntu ubuntu 2375 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/trash_watcher/Types.h 2024-03-27T21:42:44.215 INFO:teuthology.orchestra.run.smithi060.stdout: 411078 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/deep_copy 2024-03-27T21:42:44.215 INFO:teuthology.orchestra.run.smithi060.stdout: 411083 4 -rw-r--r-- 1 ubuntu ubuntu 1854 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/deep_copy/MetadataCopyRequest.h 2024-03-27T21:42:44.216 INFO:teuthology.orchestra.run.smithi060.stdout: 411084 32 -rw-r--r-- 1 ubuntu ubuntu 30571 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/deep_copy/ObjectCopyRequest.cc 2024-03-27T21:42:44.216 INFO:teuthology.orchestra.run.smithi060.stdout: 411089 8 -rw-r--r-- 1 ubuntu ubuntu 4112 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/deep_copy/SnapshotCopyRequest.h 2024-03-27T21:42:44.216 INFO:teuthology.orchestra.run.smithi060.stdout: 411090 8 -rw-r--r-- 1 ubuntu ubuntu 5980 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/deep_copy/SnapshotCreateRequest.cc 2024-03-27T21:42:44.216 INFO:teuthology.orchestra.run.smithi060.stdout: 411088 24 -rw-r--r-- 1 ubuntu ubuntu 23008 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/deep_copy/SnapshotCopyRequest.cc 2024-03-27T21:42:44.216 INFO:teuthology.orchestra.run.smithi060.stdout: 411086 8 -rw-r--r-- 1 ubuntu ubuntu 6500 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/deep_copy/SetHeadRequest.cc 2024-03-27T21:42:44.216 INFO:teuthology.orchestra.run.smithi060.stdout: 411085 8 -rw-r--r-- 1 ubuntu ubuntu 4396 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/deep_copy/ObjectCopyRequest.h 2024-03-27T21:42:44.216 INFO:teuthology.orchestra.run.smithi060.stdout: 411091 4 -rw-r--r-- 1 ubuntu ubuntu 2564 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/deep_copy/SnapshotCreateRequest.h 2024-03-27T21:42:44.216 INFO:teuthology.orchestra.run.smithi060.stdout: 411087 4 -rw-r--r-- 1 ubuntu ubuntu 2011 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/deep_copy/SetHeadRequest.h 2024-03-27T21:42:44.217 INFO:teuthology.orchestra.run.smithi060.stdout: 411082 4 -rw-r--r-- 1 ubuntu ubuntu 3296 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/deep_copy/MetadataCopyRequest.cc 2024-03-27T21:42:44.217 INFO:teuthology.orchestra.run.smithi060.stdout: 411080 12 -rw-r--r-- 1 ubuntu ubuntu 9199 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/deep_copy/ImageCopyRequest.cc 2024-03-27T21:42:44.217 INFO:teuthology.orchestra.run.smithi060.stdout: 411079 4 -rw-r--r-- 1 ubuntu ubuntu 1188 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/deep_copy/Handler.h 2024-03-27T21:42:44.217 INFO:teuthology.orchestra.run.smithi060.stdout: 411092 4 -rw-r--r-- 1 ubuntu ubuntu 720 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/deep_copy/Types.h 2024-03-27T21:42:44.217 INFO:teuthology.orchestra.run.smithi060.stdout: 411094 4 -rw-r--r-- 1 ubuntu ubuntu 804 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/deep_copy/Utils.h 2024-03-27T21:42:44.217 INFO:teuthology.orchestra.run.smithi060.stdout: 411093 4 -rw-r--r-- 1 ubuntu ubuntu 1954 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/deep_copy/Utils.cc 2024-03-27T21:42:44.217 INFO:teuthology.orchestra.run.smithi060.stdout: 411081 4 -rw-r--r-- 1 ubuntu ubuntu 3503 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/deep_copy/ImageCopyRequest.h 2024-03-27T21:42:44.217 INFO:teuthology.orchestra.run.smithi060.stdout: 411349 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation 2024-03-27T21:42:44.218 INFO:teuthology.orchestra.run.smithi060.stdout: 411367 4 -rw-r--r-- 1 ubuntu ubuntu 1884 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/RenameRequest.h 2024-03-27T21:42:44.218 INFO:teuthology.orchestra.run.smithi060.stdout: 411378 16 -rw-r--r-- 1 ubuntu ubuntu 13782 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/SnapshotRemoveRequest.cc 2024-03-27T21:42:44.218 INFO:teuthology.orchestra.run.smithi060.stdout: 411366 8 -rw-r--r-- 1 ubuntu ubuntu 7682 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/RenameRequest.cc 2024-03-27T21:42:44.218 INFO:teuthology.orchestra.run.smithi060.stdout: 411387 4 -rw-r--r-- 1 ubuntu ubuntu 1404 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/SparsifyRequest.h 2024-03-27T21:42:44.218 INFO:teuthology.orchestra.run.smithi060.stdout: 411369 4 -rw-r--r-- 1 ubuntu ubuntu 2749 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/Request.h 2024-03-27T21:42:44.218 INFO:teuthology.orchestra.run.smithi060.stdout: 411376 4 -rw-r--r-- 1 ubuntu ubuntu 3244 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/SnapshotProtectRequest.cc 2024-03-27T21:42:44.218 INFO:teuthology.orchestra.run.smithi060.stdout: 411361 4 -rw-r--r-- 1 ubuntu ubuntu 1477 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/MigrateRequest.h 2024-03-27T21:42:44.218 INFO:teuthology.orchestra.run.smithi060.stdout: 411385 4 -rw-r--r-- 1 ubuntu ubuntu 2390 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/SnapshotUnprotectRequest.h 2024-03-27T21:42:44.219 INFO:teuthology.orchestra.run.smithi060.stdout: 411381 4 -rw-r--r-- 1 ubuntu ubuntu 1288 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/SnapshotRenameRequest.h 2024-03-27T21:42:44.219 INFO:teuthology.orchestra.run.smithi060.stdout: 411379 4 -rw-r--r-- 1 ubuntu ubuntu 3047 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/SnapshotRemoveRequest.h 2024-03-27T21:42:44.219 INFO:teuthology.orchestra.run.smithi060.stdout: 411365 4 -rw-r--r-- 1 ubuntu ubuntu 2232 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/RebuildObjectMapRequest.h 2024-03-27T21:42:44.219 INFO:teuthology.orchestra.run.smithi060.stdout: 411352 16 -rw-r--r-- 1 ubuntu ubuntu 14977 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/EnableFeaturesRequest.cc 2024-03-27T21:42:44.219 INFO:teuthology.orchestra.run.smithi060.stdout: 411358 4 -rw-r--r-- 1 ubuntu ubuntu 1821 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/MetadataSetRequest.cc 2024-03-27T21:42:44.219 INFO:teuthology.orchestra.run.smithi060.stdout: 411382 16 -rw-r--r-- 1 ubuntu ubuntu 13321 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/SnapshotRollbackRequest.cc 2024-03-27T21:42:44.219 INFO:teuthology.orchestra.run.smithi060.stdout: 411350 20 -rw-r--r-- 1 ubuntu ubuntu 19969 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/DisableFeaturesRequest.cc 2024-03-27T21:42:44.219 INFO:teuthology.orchestra.run.smithi060.stdout: 411372 16 -rw-r--r-- 1 ubuntu ubuntu 14041 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/SnapshotCreateRequest.cc 2024-03-27T21:42:44.220 INFO:teuthology.orchestra.run.smithi060.stdout: 411362 12 -rw-r--r-- 1 ubuntu ubuntu 8962 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/ObjectMapIterate.cc 2024-03-27T21:42:44.220 INFO:teuthology.orchestra.run.smithi060.stdout: 411375 4 -rw-r--r-- 1 ubuntu ubuntu 1018 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/SnapshotLimitRequest.h 2024-03-27T21:42:44.220 INFO:teuthology.orchestra.run.smithi060.stdout: 411360 8 -rw-r--r-- 1 ubuntu ubuntu 6993 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/MigrateRequest.cc 2024-03-27T21:42:44.220 INFO:teuthology.orchestra.run.smithi060.stdout: 411353 4 -rw-r--r-- 1 ubuntu ubuntu 3231 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/EnableFeaturesRequest.h 2024-03-27T21:42:44.220 INFO:teuthology.orchestra.run.smithi060.stdout: 411370 16 -rw-r--r-- 1 ubuntu ubuntu 14389 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/ResizeRequest.cc 2024-03-27T21:42:44.220 INFO:teuthology.orchestra.run.smithi060.stdout: 411373 8 -rw-r--r-- 1 ubuntu ubuntu 5112 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/SnapshotCreateRequest.h 2024-03-27T21:42:44.220 INFO:teuthology.orchestra.run.smithi060.stdout: 411371 8 -rw-r--r-- 1 ubuntu ubuntu 5353 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/ResizeRequest.h 2024-03-27T21:42:44.221 INFO:teuthology.orchestra.run.smithi060.stdout: 411374 4 -rw-r--r-- 1 ubuntu ubuntu 1793 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/SnapshotLimitRequest.cc 2024-03-27T21:42:44.221 INFO:teuthology.orchestra.run.smithi060.stdout: 411377 4 -rw-r--r-- 1 ubuntu ubuntu 1469 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/SnapshotProtectRequest.h 2024-03-27T21:42:44.221 INFO:teuthology.orchestra.run.smithi060.stdout: 411389 4 -rw-r--r-- 1 ubuntu ubuntu 3361 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/TrimRequest.h 2024-03-27T21:42:44.221 INFO:teuthology.orchestra.run.smithi060.stdout: 411351 8 -rw-r--r-- 1 ubuntu ubuntu 4650 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/DisableFeaturesRequest.h 2024-03-27T21:42:44.221 INFO:teuthology.orchestra.run.smithi060.stdout: 411386 16 -rw-r--r-- 1 ubuntu ubuntu 14249 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/SparsifyRequest.cc 2024-03-27T21:42:44.221 INFO:teuthology.orchestra.run.smithi060.stdout: 411368 8 -rw-r--r-- 1 ubuntu ubuntu 5347 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/Request.cc 2024-03-27T21:42:44.221 INFO:teuthology.orchestra.run.smithi060.stdout: 411363 4 -rw-r--r-- 1 ubuntu ubuntu 1633 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/ObjectMapIterate.h 2024-03-27T21:42:44.221 INFO:teuthology.orchestra.run.smithi060.stdout: 411383 4 -rw-r--r-- 1 ubuntu ubuntu 3587 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/SnapshotRollbackRequest.h 2024-03-27T21:42:44.222 INFO:teuthology.orchestra.run.smithi060.stdout: 411388 12 -rw-r--r-- 1 ubuntu ubuntu 12254 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/TrimRequest.cc 2024-03-27T21:42:44.222 INFO:teuthology.orchestra.run.smithi060.stdout: 411384 12 -rw-r--r-- 1 ubuntu ubuntu 10926 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/SnapshotUnprotectRequest.cc 2024-03-27T21:42:44.222 INFO:teuthology.orchestra.run.smithi060.stdout: 411364 8 -rw-r--r-- 1 ubuntu ubuntu 7683 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/RebuildObjectMapRequest.cc 2024-03-27T21:42:44.222 INFO:teuthology.orchestra.run.smithi060.stdout: 411380 4 -rw-r--r-- 1 ubuntu ubuntu 2978 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/SnapshotRenameRequest.cc 2024-03-27T21:42:44.222 INFO:teuthology.orchestra.run.smithi060.stdout: 411354 8 -rw-r--r-- 1 ubuntu ubuntu 7802 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/FlattenRequest.cc 2024-03-27T21:42:44.222 INFO:teuthology.orchestra.run.smithi060.stdout: 411355 4 -rw-r--r-- 1 ubuntu ubuntu 1739 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/FlattenRequest.h 2024-03-27T21:42:44.222 INFO:teuthology.orchestra.run.smithi060.stdout: 411357 4 -rw-r--r-- 1 ubuntu ubuntu 1048 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/MetadataRemoveRequest.h 2024-03-27T21:42:44.222 INFO:teuthology.orchestra.run.smithi060.stdout: 411356 4 -rw-r--r-- 1 ubuntu ubuntu 1748 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/MetadataRemoveRequest.cc 2024-03-27T21:42:44.223 INFO:teuthology.orchestra.run.smithi060.stdout: 411359 4 -rw-r--r-- 1 ubuntu ubuntu 1145 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/operation/MetadataSetRequest.h 2024-03-27T21:42:44.223 INFO:teuthology.orchestra.run.smithi060.stdout: 410938 4 -rw-r--r-- 1 ubuntu ubuntu 2075 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/TrashWatcher.h 2024-03-27T21:42:44.223 INFO:teuthology.orchestra.run.smithi060.stdout: 410907 12 -rw-r--r-- 1 ubuntu ubuntu 9826 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/CMakeLists.txt 2024-03-27T21:42:44.223 INFO:teuthology.orchestra.run.smithi060.stdout: 410911 8 -rw-r--r-- 1 ubuntu ubuntu 4194 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/DeepCopyRequest.h 2024-03-27T21:42:44.223 INFO:teuthology.orchestra.run.smithi060.stdout: 410937 4 -rw-r--r-- 1 ubuntu ubuntu 3586 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/TrashWatcher.cc 2024-03-27T21:42:44.223 INFO:teuthology.orchestra.run.smithi060.stdout: 411398 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/trash 2024-03-27T21:42:44.223 INFO:teuthology.orchestra.run.smithi060.stdout: 411400 4 -rw-r--r-- 1 ubuntu ubuntu 1897 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/trash/MoveRequest.h 2024-03-27T21:42:44.223 INFO:teuthology.orchestra.run.smithi060.stdout: 411402 4 -rw-r--r-- 1 ubuntu ubuntu 3589 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/trash/RemoveRequest.h 2024-03-27T21:42:44.223 INFO:teuthology.orchestra.run.smithi060.stdout: 411399 4 -rw-r--r-- 1 ubuntu ubuntu 3186 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/trash/MoveRequest.cc 2024-03-27T21:42:44.224 INFO:teuthology.orchestra.run.smithi060.stdout: 411401 8 -rw-r--r-- 1 ubuntu ubuntu 4427 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/trash/RemoveRequest.cc 2024-03-27T21:42:44.224 INFO:teuthology.orchestra.run.smithi060.stdout: 411151 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io 2024-03-27T21:42:44.224 INFO:teuthology.orchestra.run.smithi060.stdout: 411160 4 -rw-r--r-- 1 ubuntu ubuntu 3639 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/FlushTracker.cc 2024-03-27T21:42:44.224 INFO:teuthology.orchestra.run.smithi060.stdout: 411185 8 -rw-r--r-- 1 ubuntu ubuntu 4696 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/QosImageDispatch.h 2024-03-27T21:42:44.224 INFO:teuthology.orchestra.run.smithi060.stdout: 411156 24 -rw-r--r-- 1 ubuntu ubuntu 23931 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/CopyupRequest.cc 2024-03-27T21:42:44.224 INFO:teuthology.orchestra.run.smithi060.stdout: 411176 4 -rw-r--r-- 1 ubuntu ubuntu 3719 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ObjectDispatchInterface.h 2024-03-27T21:42:44.224 INFO:teuthology.orchestra.run.smithi060.stdout: 411193 8 -rw-r--r-- 1 ubuntu ubuntu 7396 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/SimpleSchedulerObjectDispatch.h 2024-03-27T21:42:44.224 INFO:teuthology.orchestra.run.smithi060.stdout: 411184 12 -rw-r--r-- 1 ubuntu ubuntu 10317 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/QosImageDispatch.cc 2024-03-27T21:42:44.224 INFO:teuthology.orchestra.run.smithi060.stdout: 411189 4 -rw-r--r-- 1 ubuntu ubuntu 2875 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ReadResult.h 2024-03-27T21:42:44.225 INFO:teuthology.orchestra.run.smithi060.stdout: 411183 16 -rw-r--r-- 1 ubuntu ubuntu 15481 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ObjectRequest.h 2024-03-27T21:42:44.225 INFO:teuthology.orchestra.run.smithi060.stdout: 411159 4 -rw-r--r-- 1 ubuntu ubuntu 967 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/DispatcherInterface.h 2024-03-27T21:42:44.225 INFO:teuthology.orchestra.run.smithi060.stdout: 411186 8 -rw-r--r-- 1 ubuntu ubuntu 5004 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/QueueImageDispatch.cc 2024-03-27T21:42:44.225 INFO:teuthology.orchestra.run.smithi060.stdout: 411172 4 -rw-r--r-- 1 ubuntu ubuntu 2999 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/IoOperations.cc 2024-03-27T21:42:44.225 INFO:teuthology.orchestra.run.smithi060.stdout: 411178 12 -rw-r--r-- 1 ubuntu ubuntu 11715 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ObjectDispatchSpec.h 2024-03-27T21:42:44.225 INFO:teuthology.orchestra.run.smithi060.stdout: 411181 4 -rw-r--r-- 1 ubuntu ubuntu 984 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ObjectDispatcherInterface.h 2024-03-27T21:42:44.225 INFO:teuthology.orchestra.run.smithi060.stdout: 411190 8 -rw-r--r-- 1 ubuntu ubuntu 5028 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/RefreshImageDispatch.cc 2024-03-27T21:42:44.225 INFO:teuthology.orchestra.run.smithi060.stdout: 411191 4 -rw-r--r-- 1 ubuntu ubuntu 3518 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/RefreshImageDispatch.h 2024-03-27T21:42:44.226 INFO:teuthology.orchestra.run.smithi060.stdout: 411188 12 -rw-r--r-- 1 ubuntu ubuntu 8592 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ReadResult.cc 2024-03-27T21:42:44.226 INFO:teuthology.orchestra.run.smithi060.stdout: 411195 4 -rw-r--r-- 1 ubuntu ubuntu 1024 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/Types.cc 2024-03-27T21:42:44.226 INFO:teuthology.orchestra.run.smithi060.stdout: 411163 4 -rw-r--r-- 1 ubuntu ubuntu 3345 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ImageDispatch.h 2024-03-27T21:42:44.226 INFO:teuthology.orchestra.run.smithi060.stdout: 411155 4 -rw-r--r-- 1 ubuntu ubuntu 871 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/AsyncOperation.h 2024-03-27T21:42:44.226 INFO:teuthology.orchestra.run.smithi060.stdout: 411169 4 -rw-r--r-- 1 ubuntu ubuntu 1163 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ImageDispatcherInterface.h 2024-03-27T21:42:44.226 INFO:teuthology.orchestra.run.smithi060.stdout: 411165 4 -rw-r--r-- 1 ubuntu ubuntu 1403 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ImageDispatchSpec.cc 2024-03-27T21:42:44.226 INFO:teuthology.orchestra.run.smithi060.stdout: 411175 4 -rw-r--r-- 1 ubuntu ubuntu 3809 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ObjectDispatch.h 2024-03-27T21:42:44.226 INFO:teuthology.orchestra.run.smithi060.stdout: 411177 4 -rw-r--r-- 1 ubuntu ubuntu 995 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ObjectDispatchSpec.cc 2024-03-27T21:42:44.227 INFO:teuthology.orchestra.run.smithi060.stdout: 411196 12 -rw-r--r-- 1 ubuntu ubuntu 9684 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/Types.h 2024-03-27T21:42:44.227 INFO:teuthology.orchestra.run.smithi060.stdout: 411161 4 -rw-r--r-- 1 ubuntu ubuntu 1234 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/FlushTracker.h 2024-03-27T21:42:44.227 INFO:teuthology.orchestra.run.smithi060.stdout: 411192 20 -rw-r--r-- 1 ubuntu ubuntu 18211 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/SimpleSchedulerObjectDispatch.cc 2024-03-27T21:42:44.227 INFO:teuthology.orchestra.run.smithi060.stdout: 411164 4 -rw-r--r-- 1 ubuntu ubuntu 3262 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ImageDispatchInterface.h 2024-03-27T21:42:44.227 INFO:teuthology.orchestra.run.smithi060.stdout: 411168 4 -rw-r--r-- 1 ubuntu ubuntu 2067 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ImageDispatcher.h 2024-03-27T21:42:44.227 INFO:teuthology.orchestra.run.smithi060.stdout: 411180 4 -rw-r--r-- 1 ubuntu ubuntu 1651 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ObjectDispatcher.h 2024-03-27T21:42:44.227 INFO:teuthology.orchestra.run.smithi060.stdout: 411182 36 -rw-r--r-- 1 ubuntu ubuntu 35152 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ObjectRequest.cc 2024-03-27T21:42:44.227 INFO:teuthology.orchestra.run.smithi060.stdout: 411179 8 -rw-r--r-- 1 ubuntu ubuntu 7792 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ObjectDispatcher.cc 2024-03-27T21:42:44.228 INFO:teuthology.orchestra.run.smithi060.stdout: 411199 8 -rw-r--r-- 1 ubuntu ubuntu 8189 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/WriteBlockImageDispatch.cc 2024-03-27T21:42:44.228 INFO:teuthology.orchestra.run.smithi060.stdout: 411154 4 -rw-r--r-- 1 ubuntu ubuntu 2761 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/AsyncOperation.cc 2024-03-27T21:42:44.228 INFO:teuthology.orchestra.run.smithi060.stdout: 411162 8 -rw-r--r-- 1 ubuntu ubuntu 7057 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ImageDispatch.cc 2024-03-27T21:42:44.228 INFO:teuthology.orchestra.run.smithi060.stdout: 411167 12 -rw-r--r-- 1 ubuntu ubuntu 10264 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ImageDispatcher.cc 2024-03-27T21:42:44.228 INFO:teuthology.orchestra.run.smithi060.stdout: 411198 4 -rw-r--r-- 1 ubuntu ubuntu 2803 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/Utils.h 2024-03-27T21:42:44.228 INFO:teuthology.orchestra.run.smithi060.stdout: 411197 12 -rw-r--r-- 1 ubuntu ubuntu 8849 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/Utils.cc 2024-03-27T21:42:44.228 INFO:teuthology.orchestra.run.smithi060.stdout: 411174 8 -rw-r--r-- 1 ubuntu ubuntu 6378 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ObjectDispatch.cc 2024-03-27T21:42:44.228 INFO:teuthology.orchestra.run.smithi060.stdout: 411200 8 -rw-r--r-- 1 ubuntu ubuntu 4333 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/WriteBlockImageDispatch.h 2024-03-27T21:42:44.229 INFO:teuthology.orchestra.run.smithi060.stdout: 411157 4 -rw-r--r-- 1 ubuntu ubuntu 3844 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/CopyupRequest.h 2024-03-27T21:42:44.229 INFO:teuthology.orchestra.run.smithi060.stdout: 411170 32 -rw-r--r-- 1 ubuntu ubuntu 31202 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ImageRequest.cc 2024-03-27T21:42:44.229 INFO:teuthology.orchestra.run.smithi060.stdout: 411194 4 -rw-r--r-- 1 ubuntu ubuntu 400 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/TypeTraits.h 2024-03-27T21:42:44.229 INFO:teuthology.orchestra.run.smithi060.stdout: 411158 8 -rw-r--r-- 1 ubuntu ubuntu 7351 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/Dispatcher.h 2024-03-27T21:42:44.229 INFO:teuthology.orchestra.run.smithi060.stdout: 411166 12 -rw-r--r-- 1 ubuntu ubuntu 8864 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ImageDispatchSpec.h 2024-03-27T21:42:44.229 INFO:teuthology.orchestra.run.smithi060.stdout: 411173 4 -rw-r--r-- 1 ubuntu ubuntu 473 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/IoOperations.h 2024-03-27T21:42:44.229 INFO:teuthology.orchestra.run.smithi060.stdout: 411187 4 -rw-r--r-- 1 ubuntu ubuntu 3749 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/QueueImageDispatch.h 2024-03-27T21:42:44.229 INFO:teuthology.orchestra.run.smithi060.stdout: 411152 8 -rw-r--r-- 1 ubuntu ubuntu 7328 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/AioCompletion.cc 2024-03-27T21:42:44.230 INFO:teuthology.orchestra.run.smithi060.stdout: 411153 8 -rw-r--r-- 1 ubuntu ubuntu 5107 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/AioCompletion.h 2024-03-27T21:42:44.230 INFO:teuthology.orchestra.run.smithi060.stdout: 411171 16 -rw-r--r-- 1 ubuntu ubuntu 12482 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/io/ImageRequest.h 2024-03-27T21:42:44.230 INFO:teuthology.orchestra.run.smithi060.stdout: 411228 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/managed_lock 2024-03-27T21:42:44.230 INFO:teuthology.orchestra.run.smithi060.stdout: 411232 4 -rw-r--r-- 1 ubuntu ubuntu 2824 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/managed_lock/BreakRequest.h 2024-03-27T21:42:44.230 INFO:teuthology.orchestra.run.smithi060.stdout: 411230 4 -rw-r--r-- 1 ubuntu ubuntu 2727 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/managed_lock/AcquireRequest.h 2024-03-27T21:42:44.230 INFO:teuthology.orchestra.run.smithi060.stdout: 411236 4 -rw-r--r-- 1 ubuntu ubuntu 1618 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/managed_lock/ReacquireRequest.h 2024-03-27T21:42:44.230 INFO:teuthology.orchestra.run.smithi060.stdout: 411234 4 -rw-r--r-- 1 ubuntu ubuntu 1384 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/managed_lock/GetLockerRequest.h 2024-03-27T21:42:44.230 INFO:teuthology.orchestra.run.smithi060.stdout: 411231 8 -rw-r--r-- 1 ubuntu ubuntu 6969 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/managed_lock/BreakRequest.cc 2024-03-27T21:42:44.231 INFO:teuthology.orchestra.run.smithi060.stdout: 411237 4 -rw-r--r-- 1 ubuntu ubuntu 3016 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/managed_lock/ReleaseRequest.cc 2024-03-27T21:42:44.231 INFO:teuthology.orchestra.run.smithi060.stdout: 411235 4 -rw-r--r-- 1 ubuntu ubuntu 2561 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/managed_lock/ReacquireRequest.cc 2024-03-27T21:42:44.231 INFO:teuthology.orchestra.run.smithi060.stdout: 411233 4 -rw-r--r-- 1 ubuntu ubuntu 3942 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/managed_lock/GetLockerRequest.cc 2024-03-27T21:42:44.231 INFO:teuthology.orchestra.run.smithi060.stdout: 411239 4 -rw-r--r-- 1 ubuntu ubuntu 1018 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/managed_lock/Types.h 2024-03-27T21:42:44.231 INFO:teuthology.orchestra.run.smithi060.stdout: 411229 8 -rw-r--r-- 1 ubuntu ubuntu 5791 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/managed_lock/AcquireRequest.cc 2024-03-27T21:42:44.231 INFO:teuthology.orchestra.run.smithi060.stdout: 411241 4 -rw-r--r-- 1 ubuntu ubuntu 575 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/managed_lock/Utils.h 2024-03-27T21:42:44.231 INFO:teuthology.orchestra.run.smithi060.stdout: 411240 4 -rw-r--r-- 1 ubuntu ubuntu 974 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/managed_lock/Utils.cc 2024-03-27T21:42:44.231 INFO:teuthology.orchestra.run.smithi060.stdout: 411238 4 -rw-r--r-- 1 ubuntu ubuntu 1571 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/managed_lock/ReleaseRequest.h 2024-03-27T21:42:44.232 INFO:teuthology.orchestra.run.smithi060.stdout: 410903 4 -rw-r--r-- 1 ubuntu ubuntu 2043 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/AsyncObjectThrottle.h 2024-03-27T21:42:44.232 INFO:teuthology.orchestra.run.smithi060.stdout: 410917 16 -rw-r--r-- 1 ubuntu ubuntu 13091 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/ImageCtx.h 2024-03-27T21:42:44.232 INFO:teuthology.orchestra.run.smithi060.stdout: 410922 60 -rw-r--r-- 1 ubuntu ubuntu 59886 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/Journal.cc 2024-03-27T21:42:44.232 INFO:teuthology.orchestra.run.smithi060.stdout: 410926 24 -rw-r--r-- 1 ubuntu ubuntu 23779 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/ManagedLock.cc 2024-03-27T21:42:44.232 INFO:teuthology.orchestra.run.smithi060.stdout: 411406 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/watcher 2024-03-27T21:42:44.232 INFO:teuthology.orchestra.run.smithi060.stdout: 411409 4 -rw-r--r-- 1 ubuntu ubuntu 2987 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/watcher/RewatchRequest.cc 2024-03-27T21:42:44.232 INFO:teuthology.orchestra.run.smithi060.stdout: 411410 4 -rw-r--r-- 1 ubuntu ubuntu 1697 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/watcher/RewatchRequest.h 2024-03-27T21:42:44.232 INFO:teuthology.orchestra.run.smithi060.stdout: 411408 4 -rw-r--r-- 1 ubuntu ubuntu 1437 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/watcher/Notifier.h 2024-03-27T21:42:44.232 INFO:teuthology.orchestra.run.smithi060.stdout: 411411 4 -rw-r--r-- 1 ubuntu ubuntu 1041 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/watcher/Types.cc 2024-03-27T21:42:44.233 INFO:teuthology.orchestra.run.smithi060.stdout: 411407 4 -rw-r--r-- 1 ubuntu ubuntu 2897 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/watcher/Notifier.cc 2024-03-27T21:42:44.233 INFO:teuthology.orchestra.run.smithi060.stdout: 411412 4 -rw-r--r-- 1 ubuntu ubuntu 1597 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/watcher/Types.h 2024-03-27T21:42:44.233 INFO:teuthology.orchestra.run.smithi060.stdout: 411413 4 -rw-r--r-- 1 ubuntu ubuntu 1804 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/watcher/Utils.h 2024-03-27T21:42:44.233 INFO:teuthology.orchestra.run.smithi060.stdout: 410906 8 -rw-r--r-- 1 ubuntu ubuntu 5596 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/BlockGuard.h 2024-03-27T21:42:44.233 INFO:teuthology.orchestra.run.smithi060.stdout: 410921 16 -rw-r--r-- 1 ubuntu ubuntu 12304 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/ImageWatcher.h 2024-03-27T21:42:44.233 INFO:teuthology.orchestra.run.smithi060.stdout: 410924 4 -rw-r--r-- 1 ubuntu ubuntu 2245 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/LibrbdAdminSocketHook.cc 2024-03-27T21:42:44.233 INFO:teuthology.orchestra.run.smithi060.stdout: 410925 4 -rw-r--r-- 1 ubuntu ubuntu 801 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/LibrbdAdminSocketHook.h 2024-03-27T21:42:44.233 INFO:teuthology.orchestra.run.smithi060.stdout: 410942 16 -rw-r--r-- 1 ubuntu ubuntu 15460 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/WatchNotifyTypes.cc 2024-03-27T21:42:44.234 INFO:teuthology.orchestra.run.smithi060.stdout: 410932 64 -rw-r--r-- 1 ubuntu ubuntu 62994 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/Operations.cc 2024-03-27T21:42:44.234 INFO:teuthology.orchestra.run.smithi060.stdout: 410934 4 -rw-r--r-- 1 ubuntu ubuntu 2788 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/PluginRegistry.cc 2024-03-27T21:42:44.234 INFO:teuthology.orchestra.run.smithi060.stdout: 411043 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto 2024-03-27T21:42:44.234 INFO:teuthology.orchestra.run.smithi060.stdout: 411051 4 -rw-r--r-- 1 ubuntu ubuntu 1018 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/DataCryptor.h 2024-03-27T21:42:44.234 INFO:teuthology.orchestra.run.smithi060.stdout: 411046 4 -rw-r--r-- 1 ubuntu ubuntu 1174 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/CryptoContextPool.cc 2024-03-27T21:42:44.234 INFO:teuthology.orchestra.run.smithi060.stdout: 411044 8 -rw-r--r-- 1 ubuntu ubuntu 4344 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/BlockCrypto.cc 2024-03-27T21:42:44.234 INFO:teuthology.orchestra.run.smithi060.stdout: 411047 4 -rw-r--r-- 1 ubuntu ubuntu 2036 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/CryptoContextPool.h 2024-03-27T21:42:44.235 INFO:teuthology.orchestra.run.smithi060.stdout: 411045 4 -rw-r--r-- 1 ubuntu ubuntu 1693 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/BlockCrypto.h 2024-03-27T21:42:44.235 INFO:teuthology.orchestra.run.smithi060.stdout: 411062 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/luks 2024-03-27T21:42:44.235 INFO:teuthology.orchestra.run.smithi060.stdout: 411068 4 -rw-r--r-- 1 ubuntu ubuntu 1484 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/luks/Header.h 2024-03-27T21:42:44.235 INFO:teuthology.orchestra.run.smithi060.stdout: 411069 4 -rw-r--r-- 1 ubuntu ubuntu 3031 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/luks/LUKSEncryptionFormat.cc 2024-03-27T21:42:44.235 INFO:teuthology.orchestra.run.smithi060.stdout: 411073 4 -rw-r--r-- 1 ubuntu ubuntu 3708 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/luks/Magic.cc 2024-03-27T21:42:44.235 INFO:teuthology.orchestra.run.smithi060.stdout: 411067 8 -rw-r--r-- 1 ubuntu ubuntu 6924 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/luks/Header.cc 2024-03-27T21:42:44.235 INFO:teuthology.orchestra.run.smithi060.stdout: 411065 8 -rw-r--r-- 1 ubuntu ubuntu 5870 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/luks/FormatRequest.cc 2024-03-27T21:42:44.235 INFO:teuthology.orchestra.run.smithi060.stdout: 411072 4 -rw-r--r-- 1 ubuntu ubuntu 2167 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/luks/LoadRequest.h 2024-03-27T21:42:44.236 INFO:teuthology.orchestra.run.smithi060.stdout: 411074 4 -rw-r--r-- 1 ubuntu ubuntu 875 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/luks/Magic.h 2024-03-27T21:42:44.236 INFO:teuthology.orchestra.run.smithi060.stdout: 411070 4 -rw-r--r-- 1 ubuntu ubuntu 2993 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/luks/LUKSEncryptionFormat.h 2024-03-27T21:42:44.236 INFO:teuthology.orchestra.run.smithi060.stdout: 411063 8 -rw-r--r-- 1 ubuntu ubuntu 4572 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/luks/FlattenRequest.cc 2024-03-27T21:42:44.236 INFO:teuthology.orchestra.run.smithi060.stdout: 411071 8 -rw-r--r-- 1 ubuntu ubuntu 7928 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/luks/LoadRequest.cc 2024-03-27T21:42:44.236 INFO:teuthology.orchestra.run.smithi060.stdout: 411064 4 -rw-r--r-- 1 ubuntu ubuntu 1342 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/luks/FlattenRequest.h 2024-03-27T21:42:44.236 INFO:teuthology.orchestra.run.smithi060.stdout: 411066 4 -rw-r--r-- 1 ubuntu ubuntu 1732 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/luks/FormatRequest.h 2024-03-27T21:42:44.236 INFO:teuthology.orchestra.run.smithi060.stdout: 411053 4 -rw-r--r-- 1 ubuntu ubuntu 3873 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/FormatRequest.cc 2024-03-27T21:42:44.236 INFO:teuthology.orchestra.run.smithi060.stdout: 411058 4 -rw-r--r-- 1 ubuntu ubuntu 950 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/ShutDownCryptoRequest.h 2024-03-27T21:42:44.236 INFO:teuthology.orchestra.run.smithi060.stdout: 411056 4 -rw-r--r-- 1 ubuntu ubuntu 1413 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/LoadRequest.h 2024-03-27T21:42:44.237 INFO:teuthology.orchestra.run.smithi060.stdout: 411059 4 -rw-r--r-- 1 ubuntu ubuntu 362 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/Types.h 2024-03-27T21:42:44.237 INFO:teuthology.orchestra.run.smithi060.stdout: 411049 24 -rw-r--r-- 1 ubuntu ubuntu 23386 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/CryptoObjectDispatch.cc 2024-03-27T21:42:44.237 INFO:teuthology.orchestra.run.smithi060.stdout: 411055 8 -rw-r--r-- 1 ubuntu ubuntu 5769 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/LoadRequest.cc 2024-03-27T21:42:44.237 INFO:teuthology.orchestra.run.smithi060.stdout: 411061 4 -rw-r--r-- 1 ubuntu ubuntu 810 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/Utils.h 2024-03-27T21:42:44.237 INFO:teuthology.orchestra.run.smithi060.stdout: 411054 4 -rw-r--r-- 1 ubuntu ubuntu 1139 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/FormatRequest.h 2024-03-27T21:42:44.237 INFO:teuthology.orchestra.run.smithi060.stdout: 411050 4 -rw-r--r-- 1 ubuntu ubuntu 4094 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/CryptoObjectDispatch.h 2024-03-27T21:42:44.237 INFO:teuthology.orchestra.run.smithi060.stdout: 411060 4 -rw-r--r-- 1 ubuntu ubuntu 2307 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/Utils.cc 2024-03-27T21:42:44.237 INFO:teuthology.orchestra.run.smithi060.stdout: 411052 4 -rw-r--r-- 1 ubuntu ubuntu 851 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/EncryptionFormat.h 2024-03-27T21:42:44.238 INFO:teuthology.orchestra.run.smithi060.stdout: 411075 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/openssl 2024-03-27T21:42:44.238 INFO:teuthology.orchestra.run.smithi060.stdout: 411077 4 -rw-r--r-- 1 ubuntu ubuntu 1486 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/openssl/DataCryptor.h 2024-03-27T21:42:44.238 INFO:teuthology.orchestra.run.smithi060.stdout: 411076 4 -rw-r--r-- 1 ubuntu ubuntu 3903 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/openssl/DataCryptor.cc 2024-03-27T21:42:44.238 INFO:teuthology.orchestra.run.smithi060.stdout: 411057 4 -rw-r--r-- 1 ubuntu ubuntu 2387 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/ShutDownCryptoRequest.cc 2024-03-27T21:42:44.238 INFO:teuthology.orchestra.run.smithi060.stdout: 411048 4 -rw-r--r-- 1 ubuntu ubuntu 3953 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/crypto/CryptoInterface.h 2024-03-27T21:42:44.238 INFO:teuthology.orchestra.run.smithi060.stdout: 410920 56 -rw-r--r-- 1 ubuntu ubuntu 56554 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/ImageWatcher.cc 2024-03-27T21:42:44.238 INFO:teuthology.orchestra.run.smithi060.stdout: 411109 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image 2024-03-27T21:42:44.238 INFO:teuthology.orchestra.run.smithi060.stdout: 411138 4 -rw-r--r-- 1 ubuntu ubuntu 2169 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/SetFlagsRequest.cc 2024-03-27T21:42:44.239 INFO:teuthology.orchestra.run.smithi060.stdout: 411117 4 -rw-r--r-- 1 ubuntu ubuntu 2543 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/CloseRequest.h 2024-03-27T21:42:44.239 INFO:teuthology.orchestra.run.smithi060.stdout: 411121 4 -rw-r--r-- 1 ubuntu ubuntu 3437 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/DetachChildRequest.h 2024-03-27T21:42:44.239 INFO:teuthology.orchestra.run.smithi060.stdout: 411137 8 -rw-r--r-- 1 ubuntu ubuntu 6373 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/RemoveRequest.h 2024-03-27T21:42:44.239 INFO:teuthology.orchestra.run.smithi060.stdout: 411123 4 -rw-r--r-- 1 ubuntu ubuntu 1326 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/DetachParentRequest.h 2024-03-27T21:42:44.239 INFO:teuthology.orchestra.run.smithi060.stdout: 411129 8 -rw-r--r-- 1 ubuntu ubuntu 4830 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/OpenRequest.h 2024-03-27T21:42:44.239 INFO:teuthology.orchestra.run.smithi060.stdout: 411125 4 -rw-r--r-- 1 ubuntu ubuntu 2014 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/GetMetadataRequest.h 2024-03-27T21:42:44.239 INFO:teuthology.orchestra.run.smithi060.stdout: 411126 8 -rw-r--r-- 1 ubuntu ubuntu 5382 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/ListWatchersRequest.cc 2024-03-27T21:42:44.239 INFO:teuthology.orchestra.run.smithi060.stdout: 411114 20 -rw-r--r-- 1 ubuntu ubuntu 16558 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/CloneRequest.cc 2024-03-27T21:42:44.240 INFO:teuthology.orchestra.run.smithi060.stdout: 411131 4 -rw-r--r-- 1 ubuntu ubuntu 2292 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/PreRemoveRequest.h 2024-03-27T21:42:44.240 INFO:teuthology.orchestra.run.smithi060.stdout: 411110 8 -rw-r--r-- 1 ubuntu ubuntu 7622 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/AttachChildRequest.cc 2024-03-27T21:42:44.240 INFO:teuthology.orchestra.run.smithi060.stdout: 411115 8 -rw-r--r-- 1 ubuntu ubuntu 5123 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/CloneRequest.h 2024-03-27T21:42:44.240 INFO:teuthology.orchestra.run.smithi060.stdout: 411145 4 -rw-r--r-- 1 ubuntu ubuntu 2228 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/ValidatePoolRequest.h 2024-03-27T21:42:44.240 INFO:teuthology.orchestra.run.smithi060.stdout: 411135 12 -rw-r--r-- 1 ubuntu ubuntu 10041 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/RefreshRequest.h 2024-03-27T21:42:44.240 INFO:teuthology.orchestra.run.smithi060.stdout: 411134 52 -rw-r--r-- 1 ubuntu ubuntu 50603 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/RefreshRequest.cc 2024-03-27T21:42:44.240 INFO:teuthology.orchestra.run.smithi060.stdout: 411144 8 -rw-r--r-- 1 ubuntu ubuntu 5952 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/ValidatePoolRequest.cc 2024-03-27T21:42:44.240 INFO:teuthology.orchestra.run.smithi060.stdout: 411143 4 -rw-r--r-- 1 ubuntu ubuntu 499 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/Types.h 2024-03-27T21:42:44.240 INFO:teuthology.orchestra.run.smithi060.stdout: 411132 8 -rw-r--r-- 1 ubuntu ubuntu 8121 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/RefreshParentRequest.cc 2024-03-27T21:42:44.241 INFO:teuthology.orchestra.run.smithi060.stdout: 411122 4 -rw-r--r-- 1 ubuntu ubuntu 2017 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/DetachParentRequest.cc 2024-03-27T21:42:44.241 INFO:teuthology.orchestra.run.smithi060.stdout: 411120 12 -rw-r--r-- 1 ubuntu ubuntu 11042 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/DetachChildRequest.cc 2024-03-27T21:42:44.241 INFO:teuthology.orchestra.run.smithi060.stdout: 411130 12 -rw-r--r-- 1 ubuntu ubuntu 9251 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/PreRemoveRequest.cc 2024-03-27T21:42:44.241 INFO:teuthology.orchestra.run.smithi060.stdout: 411133 4 -rw-r--r-- 1 ubuntu ubuntu 3653 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/RefreshParentRequest.h 2024-03-27T21:42:44.241 INFO:teuthology.orchestra.run.smithi060.stdout: 411141 4 -rw-r--r-- 1 ubuntu ubuntu 3340 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/SetSnapRequest.h 2024-03-27T21:42:44.241 INFO:teuthology.orchestra.run.smithi060.stdout: 411124 4 -rw-r--r-- 1 ubuntu ubuntu 3391 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/GetMetadataRequest.cc 2024-03-27T21:42:44.241 INFO:teuthology.orchestra.run.smithi060.stdout: 411112 4 -rw-r--r-- 1 ubuntu ubuntu 2419 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/AttachParentRequest.cc 2024-03-27T21:42:44.242 INFO:teuthology.orchestra.run.smithi060.stdout: 411113 4 -rw-r--r-- 1 ubuntu ubuntu 1961 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/AttachParentRequest.h 2024-03-27T21:42:44.242 INFO:teuthology.orchestra.run.smithi060.stdout: 411128 24 -rw-r--r-- 1 ubuntu ubuntu 22061 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/OpenRequest.cc 2024-03-27T21:42:44.242 INFO:teuthology.orchestra.run.smithi060.stdout: 411118 24 -rw-r--r-- 1 ubuntu ubuntu 24258 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/CreateRequest.cc 2024-03-27T21:42:44.242 INFO:teuthology.orchestra.run.smithi060.stdout: 411140 12 -rw-r--r-- 1 ubuntu ubuntu 9795 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/SetSnapRequest.cc 2024-03-27T21:42:44.242 INFO:teuthology.orchestra.run.smithi060.stdout: 411139 4 -rw-r--r-- 1 ubuntu ubuntu 1235 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/SetFlagsRequest.h 2024-03-27T21:42:44.242 INFO:teuthology.orchestra.run.smithi060.stdout: 411136 16 -rw-r--r-- 1 ubuntu ubuntu 15781 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/RemoveRequest.cc 2024-03-27T21:42:44.242 INFO:teuthology.orchestra.run.smithi060.stdout: 411119 8 -rw-r--r-- 1 ubuntu ubuntu 6757 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/CreateRequest.h 2024-03-27T21:42:44.242 INFO:teuthology.orchestra.run.smithi060.stdout: 411116 12 -rw-r--r-- 1 ubuntu ubuntu 10339 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/CloseRequest.cc 2024-03-27T21:42:44.243 INFO:teuthology.orchestra.run.smithi060.stdout: 411142 4 -rw-r--r-- 1 ubuntu ubuntu 434 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/TypeTraits.h 2024-03-27T21:42:44.243 INFO:teuthology.orchestra.run.smithi060.stdout: 411111 4 -rw-r--r-- 1 ubuntu ubuntu 3250 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/AttachChildRequest.h 2024-03-27T21:42:44.243 INFO:teuthology.orchestra.run.smithi060.stdout: 411127 4 -rw-r--r-- 1 ubuntu ubuntu 1824 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image/ListWatchersRequest.h 2024-03-27T21:42:44.243 INFO:teuthology.orchestra.run.smithi060.stdout: 410900 4 -rw-r--r-- 1 ubuntu ubuntu 1827 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/AsioEngine.cc 2024-03-27T21:42:44.243 INFO:teuthology.orchestra.run.smithi060.stdout: 411273 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror 2024-03-27T21:42:44.243 INFO:teuthology.orchestra.run.smithi060.stdout: 411282 4 -rw-r--r-- 1 ubuntu ubuntu 3208 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/GetStatusRequest.cc 2024-03-27T21:42:44.243 INFO:teuthology.orchestra.run.smithi060.stdout: 411288 8 -rw-r--r-- 1 ubuntu ubuntu 4136 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/ImageStateUpdateRequest.cc 2024-03-27T21:42:44.243 INFO:teuthology.orchestra.run.smithi060.stdout: 411281 4 -rw-r--r-- 1 ubuntu ubuntu 3844 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/GetInfoRequest.h 2024-03-27T21:42:44.244 INFO:teuthology.orchestra.run.smithi060.stdout: 411289 4 -rw-r--r-- 1 ubuntu ubuntu 2026 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/ImageStateUpdateRequest.h 2024-03-27T21:42:44.244 INFO:teuthology.orchestra.run.smithi060.stdout: 411287 4 -rw-r--r-- 1 ubuntu ubuntu 1717 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/ImageRemoveRequest.h 2024-03-27T21:42:44.244 INFO:teuthology.orchestra.run.smithi060.stdout: 411278 12 -rw-r--r-- 1 ubuntu ubuntu 9157 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/EnableRequest.cc 2024-03-27T21:42:44.244 INFO:teuthology.orchestra.run.smithi060.stdout: 411276 16 -rw-r--r-- 1 ubuntu ubuntu 13445 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/DisableRequest.cc 2024-03-27T21:42:44.244 INFO:teuthology.orchestra.run.smithi060.stdout: 411275 4 -rw-r--r-- 1 ubuntu ubuntu 1723 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/DemoteRequest.h 2024-03-27T21:42:44.244 INFO:teuthology.orchestra.run.smithi060.stdout: 411292 4 -rw-r--r-- 1 ubuntu ubuntu 428 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/Types.h 2024-03-27T21:42:44.244 INFO:teuthology.orchestra.run.smithi060.stdout: 411277 8 -rw-r--r-- 1 ubuntu ubuntu 5050 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/DisableRequest.h 2024-03-27T21:42:44.244 INFO:teuthology.orchestra.run.smithi060.stdout: 411291 4 -rw-r--r-- 1 ubuntu ubuntu 1479 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/PromoteRequest.h 2024-03-27T21:42:44.244 INFO:teuthology.orchestra.run.smithi060.stdout: 411284 4 -rw-r--r-- 1 ubuntu ubuntu 2198 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/GetUuidRequest.cc 2024-03-27T21:42:44.245 INFO:teuthology.orchestra.run.smithi060.stdout: 411285 4 -rw-r--r-- 1 ubuntu ubuntu 1351 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/GetUuidRequest.h 2024-03-27T21:42:44.245 INFO:teuthology.orchestra.run.smithi060.stdout: 411290 4 -rw-r--r-- 1 ubuntu ubuntu 3337 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/PromoteRequest.cc 2024-03-27T21:42:44.245 INFO:teuthology.orchestra.run.smithi060.stdout: 411286 4 -rw-r--r-- 1 ubuntu ubuntu 2680 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/ImageRemoveRequest.cc 2024-03-27T21:42:44.245 INFO:teuthology.orchestra.run.smithi060.stdout: 411280 12 -rw-r--r-- 1 ubuntu ubuntu 8718 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/GetInfoRequest.cc 2024-03-27T21:42:44.245 INFO:teuthology.orchestra.run.smithi060.stdout: 411283 4 -rw-r--r-- 1 ubuntu ubuntu 2124 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/GetStatusRequest.h 2024-03-27T21:42:44.245 INFO:teuthology.orchestra.run.smithi060.stdout: 411274 8 -rw-r--r-- 1 ubuntu ubuntu 5882 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/DemoteRequest.cc 2024-03-27T21:42:44.245 INFO:teuthology.orchestra.run.smithi060.stdout: 411293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot 2024-03-27T21:42:44.245 INFO:teuthology.orchestra.run.smithi060.stdout: 411307 4 -rw-r--r-- 1 ubuntu ubuntu 1635 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/RemoveImageStateRequest.h 2024-03-27T21:42:44.246 INFO:teuthology.orchestra.run.smithi060.stdout: 411295 4 -rw-r--r-- 1 ubuntu ubuntu 3139 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/CreateNonPrimaryRequest.h 2024-03-27T21:42:44.246 INFO:teuthology.orchestra.run.smithi060.stdout: 411294 8 -rw-r--r-- 1 ubuntu ubuntu 7423 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/CreateNonPrimaryRequest.cc 2024-03-27T21:42:44.246 INFO:teuthology.orchestra.run.smithi060.stdout: 411300 4 -rw-r--r-- 1 ubuntu ubuntu 2818 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/GetImageStateRequest.cc 2024-03-27T21:42:44.246 INFO:teuthology.orchestra.run.smithi060.stdout: 411309 4 -rw-r--r-- 1 ubuntu ubuntu 1930 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/SetImageStateRequest.h 2024-03-27T21:42:44.246 INFO:teuthology.orchestra.run.smithi060.stdout: 411297 4 -rw-r--r-- 1 ubuntu ubuntu 2689 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/CreatePrimaryRequest.h 2024-03-27T21:42:44.246 INFO:teuthology.orchestra.run.smithi060.stdout: 411313 4 -rw-r--r-- 1 ubuntu ubuntu 2629 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/UnlinkPeerRequest.h 2024-03-27T21:42:44.246 INFO:teuthology.orchestra.run.smithi060.stdout: 411316 4 -rw-r--r-- 1 ubuntu ubuntu 3143 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/WriteImageStateRequest.cc 2024-03-27T21:42:44.246 INFO:teuthology.orchestra.run.smithi060.stdout: 411317 4 -rw-r--r-- 1 ubuntu ubuntu 1664 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/WriteImageStateRequest.h 2024-03-27T21:42:44.246 INFO:teuthology.orchestra.run.smithi060.stdout: 411306 4 -rw-r--r-- 1 ubuntu ubuntu 3387 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/RemoveImageStateRequest.cc 2024-03-27T21:42:44.247 INFO:teuthology.orchestra.run.smithi060.stdout: 411310 4 -rw-r--r-- 1 ubuntu ubuntu 2888 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/Types.cc 2024-03-27T21:42:44.247 INFO:teuthology.orchestra.run.smithi060.stdout: 411303 4 -rw-r--r-- 1 ubuntu ubuntu 1467 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/ImageMeta.h 2024-03-27T21:42:44.247 INFO:teuthology.orchestra.run.smithi060.stdout: 411299 4 -rw-r--r-- 1 ubuntu ubuntu 1593 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/DemoteRequest.h 2024-03-27T21:42:44.247 INFO:teuthology.orchestra.run.smithi060.stdout: 411296 12 -rw-r--r-- 1 ubuntu ubuntu 8304 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/CreatePrimaryRequest.cc 2024-03-27T21:42:44.247 INFO:teuthology.orchestra.run.smithi060.stdout: 411311 4 -rw-r--r-- 1 ubuntu ubuntu 3272 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/Types.h 2024-03-27T21:42:44.247 INFO:teuthology.orchestra.run.smithi060.stdout: 411305 4 -rw-r--r-- 1 ubuntu ubuntu 4036 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/PromoteRequest.h 2024-03-27T21:42:44.247 INFO:teuthology.orchestra.run.smithi060.stdout: 411304 12 -rw-r--r-- 1 ubuntu ubuntu 11030 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/PromoteRequest.cc 2024-03-27T21:42:44.247 INFO:teuthology.orchestra.run.smithi060.stdout: 411315 4 -rw-r--r-- 1 ubuntu ubuntu 1103 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/Utils.h 2024-03-27T21:42:44.248 INFO:teuthology.orchestra.run.smithi060.stdout: 411298 4 -rw-r--r-- 1 ubuntu ubuntu 3049 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/DemoteRequest.cc 2024-03-27T21:42:44.248 INFO:teuthology.orchestra.run.smithi060.stdout: 411301 4 -rw-r--r-- 1 ubuntu ubuntu 1724 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/GetImageStateRequest.h 2024-03-27T21:42:44.248 INFO:teuthology.orchestra.run.smithi060.stdout: 411312 8 -rw-r--r-- 1 ubuntu ubuntu 6813 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/UnlinkPeerRequest.cc 2024-03-27T21:42:44.248 INFO:teuthology.orchestra.run.smithi060.stdout: 411314 8 -rw-r--r-- 1 ubuntu ubuntu 5746 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/Utils.cc 2024-03-27T21:42:44.248 INFO:teuthology.orchestra.run.smithi060.stdout: 411308 8 -rw-r--r-- 1 ubuntu ubuntu 6489 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/SetImageStateRequest.cc 2024-03-27T21:42:44.248 INFO:teuthology.orchestra.run.smithi060.stdout: 411302 8 -rw-r--r-- 1 ubuntu ubuntu 5464 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/snapshot/ImageMeta.cc 2024-03-27T21:42:44.248 INFO:teuthology.orchestra.run.smithi060.stdout: 411279 4 -rw-r--r-- 1 ubuntu ubuntu 4018 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirror/EnableRequest.h 2024-03-27T21:42:44.248 INFO:teuthology.orchestra.run.smithi060.stdout: 410973 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/asio 2024-03-27T21:42:44.249 INFO:teuthology.orchestra.run.smithi060.stdout: 410975 4 -rw-r--r-- 1 ubuntu ubuntu 1270 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/asio/ContextWQ.h 2024-03-27T21:42:44.249 INFO:teuthology.orchestra.run.smithi060.stdout: 410976 4 -rw-r--r-- 1 ubuntu ubuntu 792 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/asio/Utils.h 2024-03-27T21:42:44.249 INFO:teuthology.orchestra.run.smithi060.stdout: 410974 4 -rw-r--r-- 1 ubuntu ubuntu 1263 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/asio/ContextWQ.cc 2024-03-27T21:42:44.249 INFO:teuthology.orchestra.run.smithi060.stdout: 410904 4 -rw-r--r-- 1 ubuntu ubuntu 1866 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/AsyncRequest.cc 2024-03-27T21:42:44.249 INFO:teuthology.orchestra.run.smithi060.stdout: 410905 4 -rw-r--r-- 1 ubuntu ubuntu 1604 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/AsyncRequest.h 2024-03-27T21:42:44.249 INFO:teuthology.orchestra.run.smithi060.stdout: 411149 52 -rw-r--r-- 1 ubuntu ubuntu 49410 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/internal.cc 2024-03-27T21:42:44.249 INFO:teuthology.orchestra.run.smithi060.stdout: 410923 16 -rw-r--r-- 1 ubuntu ubuntu 13084 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/Journal.h 2024-03-27T21:42:44.249 INFO:teuthology.orchestra.run.smithi060.stdout: 410939 4 -rw-r--r-- 1 ubuntu ubuntu 3665 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/Types.h 2024-03-27T21:42:44.250 INFO:teuthology.orchestra.run.smithi060.stdout: 411201 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal 2024-03-27T21:42:44.250 INFO:teuthology.orchestra.run.smithi060.stdout: 411217 8 -rw-r--r-- 1 ubuntu ubuntu 7233 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/Replay.h 2024-03-27T21:42:44.250 INFO:teuthology.orchestra.run.smithi060.stdout: 411215 4 -rw-r--r-- 1 ubuntu ubuntu 2045 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/RemoveRequest.h 2024-03-27T21:42:44.250 INFO:teuthology.orchestra.run.smithi060.stdout: 411210 4 -rw-r--r-- 1 ubuntu ubuntu 1883 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/OpenRequest.h 2024-03-27T21:42:44.250 INFO:teuthology.orchestra.run.smithi060.stdout: 411216 40 -rw-r--r-- 1 ubuntu ubuntu 38979 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/Replay.cc 2024-03-27T21:42:44.250 INFO:teuthology.orchestra.run.smithi060.stdout: 411206 4 -rw-r--r-- 1 ubuntu ubuntu 640 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/DisabledPolicy.h 2024-03-27T21:42:44.250 INFO:teuthology.orchestra.run.smithi060.stdout: 411223 28 -rw-r--r-- 1 ubuntu ubuntu 24589 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/Types.cc 2024-03-27T21:42:44.250 INFO:teuthology.orchestra.run.smithi060.stdout: 411205 4 -rw-r--r-- 1 ubuntu ubuntu 2305 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/DemoteRequest.h 2024-03-27T21:42:44.251 INFO:teuthology.orchestra.run.smithi060.stdout: 411211 4 -rw-r--r-- 1 ubuntu ubuntu 513 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/Policy.h 2024-03-27T21:42:44.251 INFO:teuthology.orchestra.run.smithi060.stdout: 411208 8 -rw-r--r-- 1 ubuntu ubuntu 4291 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/ObjectDispatch.h 2024-03-27T21:42:44.251 INFO:teuthology.orchestra.run.smithi060.stdout: 411224 20 -rw-r--r-- 1 ubuntu ubuntu 19656 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/Types.h 2024-03-27T21:42:44.251 INFO:teuthology.orchestra.run.smithi060.stdout: 411213 4 -rw-r--r-- 1 ubuntu ubuntu 2360 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/PromoteRequest.h 2024-03-27T21:42:44.251 INFO:teuthology.orchestra.run.smithi060.stdout: 411212 8 -rw-r--r-- 1 ubuntu ubuntu 6417 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/PromoteRequest.cc 2024-03-27T21:42:44.251 INFO:teuthology.orchestra.run.smithi060.stdout: 411220 4 -rw-r--r-- 1 ubuntu ubuntu 894 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/StandardPolicy.cc 2024-03-27T21:42:44.251 INFO:teuthology.orchestra.run.smithi060.stdout: 411209 8 -rw-r--r-- 1 ubuntu ubuntu 4117 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/OpenRequest.cc 2024-03-27T21:42:44.251 INFO:teuthology.orchestra.run.smithi060.stdout: 411226 4 -rw-r--r-- 1 ubuntu ubuntu 1729 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/Utils.h 2024-03-27T21:42:44.251 INFO:teuthology.orchestra.run.smithi060.stdout: 411202 8 -rw-r--r-- 1 ubuntu ubuntu 6623 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/CreateRequest.cc 2024-03-27T21:42:44.252 INFO:teuthology.orchestra.run.smithi060.stdout: 411204 8 -rw-r--r-- 1 ubuntu ubuntu 6801 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/DemoteRequest.cc 2024-03-27T21:42:44.252 INFO:teuthology.orchestra.run.smithi060.stdout: 411214 8 -rw-r--r-- 1 ubuntu ubuntu 4141 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/RemoveRequest.cc 2024-03-27T21:42:44.252 INFO:teuthology.orchestra.run.smithi060.stdout: 411203 4 -rw-r--r-- 1 ubuntu ubuntu 2939 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/CreateRequest.h 2024-03-27T21:42:44.252 INFO:teuthology.orchestra.run.smithi060.stdout: 411225 4 -rw-r--r-- 1 ubuntu ubuntu 2361 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/Utils.cc 2024-03-27T21:42:44.252 INFO:teuthology.orchestra.run.smithi060.stdout: 411207 12 -rw-r--r-- 1 ubuntu ubuntu 9041 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/ObjectDispatch.cc 2024-03-27T21:42:44.252 INFO:teuthology.orchestra.run.smithi060.stdout: 411221 4 -rw-r--r-- 1 ubuntu ubuntu 840 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/StandardPolicy.h 2024-03-27T21:42:44.252 INFO:teuthology.orchestra.run.smithi060.stdout: 411222 4 -rw-r--r-- 1 ubuntu ubuntu 617 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/TypeTraits.h 2024-03-27T21:42:44.252 INFO:teuthology.orchestra.run.smithi060.stdout: 411219 4 -rw-r--r-- 1 ubuntu ubuntu 2630 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/ResetRequest.h 2024-03-27T21:42:44.253 INFO:teuthology.orchestra.run.smithi060.stdout: 411218 8 -rw-r--r-- 1 ubuntu ubuntu 4348 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/journal/ResetRequest.cc 2024-03-27T21:42:44.253 INFO:teuthology.orchestra.run.smithi060.stdout: 410944 12 -rw-r--r-- 1 ubuntu ubuntu 10531 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/Watcher.cc 2024-03-27T21:42:44.253 INFO:teuthology.orchestra.run.smithi060.stdout: 410945 8 -rw-r--r-- 1 ubuntu ubuntu 4543 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/Watcher.h 2024-03-27T21:42:44.253 INFO:teuthology.orchestra.run.smithi060.stdout: 410927 12 -rw-r--r-- 1 ubuntu ubuntu 8221 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/ManagedLock.h 2024-03-27T21:42:44.253 INFO:teuthology.orchestra.run.smithi060.stdout: 410931 8 -rw-r--r-- 1 ubuntu ubuntu 6316 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/ObjectMap.h 2024-03-27T21:42:44.253 INFO:teuthology.orchestra.run.smithi060.stdout: 410933 8 -rw-r--r-- 1 ubuntu ubuntu 5716 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/Operations.h 2024-03-27T21:42:44.253 INFO:teuthology.orchestra.run.smithi060.stdout: 410929 4 -rw-r--r-- 1 ubuntu ubuntu 2638 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/MirroringWatcher.h 2024-03-27T21:42:44.253 INFO:teuthology.orchestra.run.smithi060.stdout: 410901 4 -rw-r--r-- 1 ubuntu ubuntu 2010 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/AsioEngine.h 2024-03-27T21:42:44.253 INFO:teuthology.orchestra.run.smithi060.stdout: 410936 8 -rw-r--r-- 1 ubuntu ubuntu 4727 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/TaskFinisher.h 2024-03-27T21:42:44.254 INFO:teuthology.orchestra.run.smithi060.stdout: 411095 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/exclusive_lock 2024-03-27T21:42:44.254 INFO:teuthology.orchestra.run.smithi060.stdout: 411102 4 -rw-r--r-- 1 ubuntu ubuntu 2910 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/exclusive_lock/PostAcquireRequest.h 2024-03-27T21:42:44.254 INFO:teuthology.orchestra.run.smithi060.stdout: 411106 4 -rw-r--r-- 1 ubuntu ubuntu 3252 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/exclusive_lock/PreReleaseRequest.h 2024-03-27T21:42:44.254 INFO:teuthology.orchestra.run.smithi060.stdout: 411099 8 -rw-r--r-- 1 ubuntu ubuntu 4518 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/exclusive_lock/ImageDispatch.h 2024-03-27T21:42:44.254 INFO:teuthology.orchestra.run.smithi060.stdout: 411100 4 -rw-r--r-- 1 ubuntu ubuntu 732 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/exclusive_lock/Policy.h 2024-03-27T21:42:44.254 INFO:teuthology.orchestra.run.smithi060.stdout: 411104 4 -rw-r--r-- 1 ubuntu ubuntu 1420 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/exclusive_lock/PreAcquireRequest.h 2024-03-27T21:42:44.254 INFO:teuthology.orchestra.run.smithi060.stdout: 411098 12 -rw-r--r-- 1 ubuntu ubuntu 10081 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/exclusive_lock/ImageDispatch.cc 2024-03-27T21:42:44.254 INFO:teuthology.orchestra.run.smithi060.stdout: 411101 12 -rw-r--r-- 1 ubuntu ubuntu 9778 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/exclusive_lock/PostAcquireRequest.cc 2024-03-27T21:42:44.255 INFO:teuthology.orchestra.run.smithi060.stdout: 411105 12 -rw-r--r-- 1 ubuntu ubuntu 10457 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/exclusive_lock/PreReleaseRequest.cc 2024-03-27T21:42:44.255 INFO:teuthology.orchestra.run.smithi060.stdout: 411107 4 -rw-r--r-- 1 ubuntu ubuntu 825 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/exclusive_lock/StandardPolicy.cc 2024-03-27T21:42:44.255 INFO:teuthology.orchestra.run.smithi060.stdout: 411108 4 -rw-r--r-- 1 ubuntu ubuntu 812 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/exclusive_lock/StandardPolicy.h 2024-03-27T21:42:44.255 INFO:teuthology.orchestra.run.smithi060.stdout: 411103 4 -rw-r--r-- 1 ubuntu ubuntu 2609 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/exclusive_lock/PreAcquireRequest.cc 2024-03-27T21:42:44.255 INFO:teuthology.orchestra.run.smithi060.stdout: 411097 4 -rw-r--r-- 1 ubuntu ubuntu 692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/exclusive_lock/AutomaticPolicy.h 2024-03-27T21:42:44.255 INFO:teuthology.orchestra.run.smithi060.stdout: 411096 4 -rw-r--r-- 1 ubuntu ubuntu 839 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/exclusive_lock/AutomaticPolicy.cc 2024-03-27T21:42:44.255 INFO:teuthology.orchestra.run.smithi060.stdout: 410918 28 -rw-r--r-- 1 ubuntu ubuntu 28604 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/ImageState.cc 2024-03-27T21:42:44.255 INFO:teuthology.orchestra.run.smithi060.stdout: 410930 16 -rw-r--r-- 1 ubuntu ubuntu 13119 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/ObjectMap.cc 2024-03-27T21:42:44.256 INFO:teuthology.orchestra.run.smithi060.stdout: 410941 12 -rw-r--r-- 1 ubuntu ubuntu 8251 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/Utils.h 2024-03-27T21:42:44.256 INFO:teuthology.orchestra.run.smithi060.stdout: 410914 4 -rw-r--r-- 1 ubuntu ubuntu 3181 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/Features.cc 2024-03-27T21:42:44.256 INFO:teuthology.orchestra.run.smithi060.stdout: 410943 16 -rw-r--r-- 1 ubuntu ubuntu 14766 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/WatchNotifyTypes.h 2024-03-27T21:42:44.256 INFO:teuthology.orchestra.run.smithi060.stdout: 410910 12 -rw-r--r-- 1 ubuntu ubuntu 9954 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/DeepCopyRequest.cc 2024-03-27T21:42:44.256 INFO:teuthology.orchestra.run.smithi060.stdout: 410909 4 -rw-r--r-- 1 ubuntu ubuntu 917 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/ConfigWatcher.h 2024-03-27T21:42:44.256 INFO:teuthology.orchestra.run.smithi060.stdout: 410940 8 -rw-r--r-- 1 ubuntu ubuntu 6934 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/Utils.cc 2024-03-27T21:42:44.256 INFO:teuthology.orchestra.run.smithi060.stdout: 411321 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map 2024-03-27T21:42:44.256 INFO:teuthology.orchestra.run.smithi060.stdout: 411340 8 -rw-r--r-- 1 ubuntu ubuntu 7246 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/SnapshotRemoveRequest.cc 2024-03-27T21:42:44.257 INFO:teuthology.orchestra.run.smithi060.stdout: 411335 4 -rw-r--r-- 1 ubuntu ubuntu 1401 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/Request.h 2024-03-27T21:42:44.257 INFO:teuthology.orchestra.run.smithi060.stdout: 411341 4 -rw-r--r-- 1 ubuntu ubuntu 2517 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/SnapshotRemoveRequest.h 2024-03-27T21:42:44.257 INFO:teuthology.orchestra.run.smithi060.stdout: 411324 16 -rw-r--r-- 1 ubuntu ubuntu 12372 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/DiffRequest.cc 2024-03-27T21:42:44.257 INFO:teuthology.orchestra.run.smithi060.stdout: 411347 4 -rw-r--r-- 1 ubuntu ubuntu 3980 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/UpdateRequest.cc 2024-03-27T21:42:44.257 INFO:teuthology.orchestra.run.smithi060.stdout: 411333 4 -rw-r--r-- 1 ubuntu ubuntu 1347 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/RemoveRequest.h 2024-03-27T21:42:44.257 INFO:teuthology.orchestra.run.smithi060.stdout: 411342 4 -rw-r--r-- 1 ubuntu ubuntu 3848 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/SnapshotRollbackRequest.cc 2024-03-27T21:42:44.257 INFO:teuthology.orchestra.run.smithi060.stdout: 411326 4 -rw-r--r-- 1 ubuntu ubuntu 2600 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/InvalidateRequest.cc 2024-03-27T21:42:44.257 INFO:teuthology.orchestra.run.smithi060.stdout: 411338 8 -rw-r--r-- 1 ubuntu ubuntu 4253 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/SnapshotCreateRequest.cc 2024-03-27T21:42:44.257 INFO:teuthology.orchestra.run.smithi060.stdout: 411345 4 -rw-r--r-- 1 ubuntu ubuntu 1885 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/UnlockRequest.cc 2024-03-27T21:42:44.258 INFO:teuthology.orchestra.run.smithi060.stdout: 411328 8 -rw-r--r-- 1 ubuntu ubuntu 4758 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/LockRequest.cc 2024-03-27T21:42:44.258 INFO:teuthology.orchestra.run.smithi060.stdout: 411336 4 -rw-r--r-- 1 ubuntu ubuntu 2144 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/ResizeRequest.cc 2024-03-27T21:42:44.258 INFO:teuthology.orchestra.run.smithi060.stdout: 411339 4 -rw-r--r-- 1 ubuntu ubuntu 1791 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/SnapshotCreateRequest.h 2024-03-27T21:42:44.258 INFO:teuthology.orchestra.run.smithi060.stdout: 411337 4 -rw-r--r-- 1 ubuntu ubuntu 1340 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/ResizeRequest.h 2024-03-27T21:42:44.258 INFO:teuthology.orchestra.run.smithi060.stdout: 411348 4 -rw-r--r-- 1 ubuntu ubuntu 3250 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/UpdateRequest.h 2024-03-27T21:42:44.258 INFO:teuthology.orchestra.run.smithi060.stdout: 411331 4 -rw-r--r-- 1 ubuntu ubuntu 3126 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/RefreshRequest.h 2024-03-27T21:42:44.258 INFO:teuthology.orchestra.run.smithi060.stdout: 411346 4 -rw-r--r-- 1 ubuntu ubuntu 950 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/UnlockRequest.h 2024-03-27T21:42:44.261 INFO:teuthology.orchestra.run.smithi060.stdout: 411334 4 -rw-r--r-- 1 ubuntu ubuntu 1960 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/Request.cc 2024-03-27T21:42:44.261 INFO:teuthology.orchestra.run.smithi060.stdout: 411330 12 -rw-r--r-- 1 ubuntu ubuntu 9804 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/RefreshRequest.cc 2024-03-27T21:42:44.262 INFO:teuthology.orchestra.run.smithi060.stdout: 411344 4 -rw-r--r-- 1 ubuntu ubuntu 868 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/Types.h 2024-03-27T21:42:44.262 INFO:teuthology.orchestra.run.smithi060.stdout: 411343 4 -rw-r--r-- 1 ubuntu ubuntu 1727 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/SnapshotRollbackRequest.h 2024-03-27T21:42:44.262 INFO:teuthology.orchestra.run.smithi060.stdout: 411322 4 -rw-r--r-- 1 ubuntu ubuntu 2710 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/CreateRequest.cc 2024-03-27T21:42:44.262 INFO:teuthology.orchestra.run.smithi060.stdout: 411332 4 -rw-r--r-- 1 ubuntu ubuntu 2326 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/RemoveRequest.cc 2024-03-27T21:42:44.262 INFO:teuthology.orchestra.run.smithi060.stdout: 411323 4 -rw-r--r-- 1 ubuntu ubuntu 1206 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/CreateRequest.h 2024-03-27T21:42:44.262 INFO:teuthology.orchestra.run.smithi060.stdout: 411329 4 -rw-r--r-- 1 ubuntu ubuntu 2431 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/LockRequest.h 2024-03-27T21:42:44.262 INFO:teuthology.orchestra.run.smithi060.stdout: 411325 4 -rw-r--r-- 1 ubuntu ubuntu 2246 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/DiffRequest.h 2024-03-27T21:42:44.262 INFO:teuthology.orchestra.run.smithi060.stdout: 411327 4 -rw-r--r-- 1 ubuntu ubuntu 1122 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/object_map/InvalidateRequest.h 2024-03-27T21:42:44.262 INFO:teuthology.orchestra.run.smithi060.stdout: 410928 8 -rw-r--r-- 1 ubuntu ubuntu 4595 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/MirroringWatcher.cc 2024-03-27T21:42:44.263 INFO:teuthology.orchestra.run.smithi060.stdout: 410908 4 -rw-r--r-- 1 ubuntu ubuntu 3053 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/ConfigWatcher.cc 2024-03-27T21:42:44.263 INFO:teuthology.orchestra.run.smithi060.stdout: 410977 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache 2024-03-27T21:42:44.263 INFO:teuthology.orchestra.run.smithi060.stdout: 410993 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl 2024-03-27T21:42:44.263 INFO:teuthology.orchestra.run.smithi060.stdout: 411002 4 -rw-r--r-- 1 ubuntu ubuntu 2182 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/InitRequest.h 2024-03-27T21:42:44.263 INFO:teuthology.orchestra.run.smithi060.stdout: 411011 12 -rw-r--r-- 1 ubuntu ubuntu 11409 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/Request.h 2024-03-27T21:42:44.263 INFO:teuthology.orchestra.run.smithi060.stdout: 410998 4 -rw-r--r-- 1 ubuntu ubuntu 1719 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/DiscardRequest.h 2024-03-27T21:42:44.263 INFO:teuthology.orchestra.run.smithi060.stdout: 411006 4 -rw-r--r-- 1 ubuntu ubuntu 2540 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/LogMap.h 2024-03-27T21:42:44.263 INFO:teuthology.orchestra.run.smithi060.stdout: 411013 4 -rw-r--r-- 1 ubuntu ubuntu 1913 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/ShutdownRequest.h 2024-03-27T21:42:44.264 INFO:teuthology.orchestra.run.smithi060.stdout: 411008 12 -rw-r--r-- 1 ubuntu ubuntu 8440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/LogOperation.h 2024-03-27T21:42:44.264 INFO:teuthology.orchestra.run.smithi060.stdout: 411009 4 -rw-r--r-- 1 ubuntu ubuntu 1132 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/ReadRequest.h 2024-03-27T21:42:44.264 INFO:teuthology.orchestra.run.smithi060.stdout: 411012 8 -rw-r--r-- 1 ubuntu ubuntu 4455 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/ShutdownRequest.cc 2024-03-27T21:42:44.264 INFO:teuthology.orchestra.run.smithi060.stdout: 411015 4 -rw-r--r-- 1 ubuntu ubuntu 2776 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/SyncPoint.h 2024-03-27T21:42:44.264 INFO:teuthology.orchestra.run.smithi060.stdout: 411018 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/rwl 2024-03-27T21:42:44.264 INFO:teuthology.orchestra.run.smithi060.stdout: 411024 4 -rw-r--r-- 1 ubuntu ubuntu 2326 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/rwl/ReadRequest.cc 2024-03-27T21:42:44.264 INFO:teuthology.orchestra.run.smithi060.stdout: 411027 4 -rw-r--r-- 1 ubuntu ubuntu 3097 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/rwl/Request.h 2024-03-27T21:42:44.264 INFO:teuthology.orchestra.run.smithi060.stdout: 411029 8 -rw-r--r-- 1 ubuntu ubuntu 4695 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/rwl/WriteLog.h 2024-03-27T21:42:44.265 INFO:teuthology.orchestra.run.smithi060.stdout: 411023 4 -rw-r--r-- 1 ubuntu ubuntu 1957 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/rwl/LogOperation.h 2024-03-27T21:42:44.265 INFO:teuthology.orchestra.run.smithi060.stdout: 411025 4 -rw-r--r-- 1 ubuntu ubuntu 1000 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/rwl/ReadRequest.h 2024-03-27T21:42:44.265 INFO:teuthology.orchestra.run.smithi060.stdout: 411021 4 -rw-r--r-- 1 ubuntu ubuntu 2712 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/rwl/LogEntry.h 2024-03-27T21:42:44.265 INFO:teuthology.orchestra.run.smithi060.stdout: 411019 8 -rw-r--r-- 1 ubuntu ubuntu 4540 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/rwl/Builder.h 2024-03-27T21:42:44.265 INFO:teuthology.orchestra.run.smithi060.stdout: 411026 4 -rw-r--r-- 1 ubuntu ubuntu 3092 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/rwl/Request.cc 2024-03-27T21:42:44.265 INFO:teuthology.orchestra.run.smithi060.stdout: 411020 4 -rw-r--r-- 1 ubuntu ubuntu 3299 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/rwl/LogEntry.cc 2024-03-27T21:42:44.265 INFO:teuthology.orchestra.run.smithi060.stdout: 411022 4 -rw-r--r-- 1 ubuntu ubuntu 1289 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/rwl/LogOperation.cc 2024-03-27T21:42:44.266 INFO:teuthology.orchestra.run.smithi060.stdout: 411028 40 -rw-r--r-- 1 ubuntu ubuntu 36933 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/rwl/WriteLog.cc 2024-03-27T21:42:44.266 INFO:teuthology.orchestra.run.smithi060.stdout: 411004 12 -rw-r--r-- 1 ubuntu ubuntu 10315 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/LogEntry.h 2024-03-27T21:42:44.266 INFO:teuthology.orchestra.run.smithi060.stdout: 411030 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/ssd 2024-03-27T21:42:44.266 INFO:teuthology.orchestra.run.smithi060.stdout: 411036 4 -rw-r--r-- 1 ubuntu ubuntu 3374 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/ssd/ReadRequest.cc 2024-03-27T21:42:44.266 INFO:teuthology.orchestra.run.smithi060.stdout: 411039 4 -rw-r--r-- 1 ubuntu ubuntu 3081 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/ssd/Request.h 2024-03-27T21:42:44.266 INFO:teuthology.orchestra.run.smithi060.stdout: 411042 8 -rw-r--r-- 1 ubuntu ubuntu 5926 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/ssd/WriteLog.h 2024-03-27T21:42:44.266 INFO:teuthology.orchestra.run.smithi060.stdout: 411035 4 -rw-r--r-- 1 ubuntu ubuntu 1131 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/ssd/LogOperation.h 2024-03-27T21:42:44.266 INFO:teuthology.orchestra.run.smithi060.stdout: 411037 4 -rw-r--r-- 1 ubuntu ubuntu 1000 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/ssd/ReadRequest.h 2024-03-27T21:42:44.267 INFO:teuthology.orchestra.run.smithi060.stdout: 411033 4 -rw-r--r-- 1 ubuntu ubuntu 2804 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/ssd/LogEntry.h 2024-03-27T21:42:44.267 INFO:teuthology.orchestra.run.smithi060.stdout: 411031 8 -rw-r--r-- 1 ubuntu ubuntu 4536 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/ssd/Builder.h 2024-03-27T21:42:44.267 INFO:teuthology.orchestra.run.smithi060.stdout: 411038 4 -rw-r--r-- 1 ubuntu ubuntu 2219 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/ssd/Request.cc 2024-03-27T21:42:44.267 INFO:teuthology.orchestra.run.smithi060.stdout: 411040 4 -rw-r--r-- 1 ubuntu ubuntu 1221 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/ssd/Types.h 2024-03-27T21:42:44.267 INFO:teuthology.orchestra.run.smithi060.stdout: 411032 4 -rw-r--r-- 1 ubuntu ubuntu 1847 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/ssd/LogEntry.cc 2024-03-27T21:42:44.267 INFO:teuthology.orchestra.run.smithi060.stdout: 411034 4 -rw-r--r-- 1 ubuntu ubuntu 1036 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/ssd/LogOperation.cc 2024-03-27T21:42:44.267 INFO:teuthology.orchestra.run.smithi060.stdout: 411041 44 -rw-r--r-- 1 ubuntu ubuntu 41635 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/ssd/WriteLog.cc 2024-03-27T21:42:44.267 INFO:teuthology.orchestra.run.smithi060.stdout: 411001 8 -rw-r--r-- 1 ubuntu ubuntu 6206 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/InitRequest.cc 2024-03-27T21:42:44.267 INFO:teuthology.orchestra.run.smithi060.stdout: 410994 88 -rw-r--r-- 1 ubuntu ubuntu 86929 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/AbstractWriteLog.cc 2024-03-27T21:42:44.268 INFO:teuthology.orchestra.run.smithi060.stdout: 411016 8 -rw-r--r-- 1 ubuntu ubuntu 6030 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/Types.cc 2024-03-27T21:42:44.268 INFO:teuthology.orchestra.run.smithi060.stdout: 410996 4 -rw-r--r-- 1 ubuntu ubuntu 2842 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/Builder.h 2024-03-27T21:42:44.268 INFO:teuthology.orchestra.run.smithi060.stdout: 411014 4 -rw-r--r-- 1 ubuntu ubuntu 4009 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/SyncPoint.cc 2024-03-27T21:42:44.268 INFO:teuthology.orchestra.run.smithi060.stdout: 410995 16 -rw-r--r-- 1 ubuntu ubuntu 14906 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/AbstractWriteLog.h 2024-03-27T21:42:44.268 INFO:teuthology.orchestra.run.smithi060.stdout: 411010 24 -rw-r--r-- 1 ubuntu ubuntu 21501 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/Request.cc 2024-03-27T21:42:44.268 INFO:teuthology.orchestra.run.smithi060.stdout: 410997 8 -rw-r--r-- 1 ubuntu ubuntu 4243 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/DiscardRequest.cc 2024-03-27T21:42:44.268 INFO:teuthology.orchestra.run.smithi060.stdout: 411017 16 -rw-r--r-- 1 ubuntu ubuntu 16288 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/Types.h 2024-03-27T21:42:44.268 INFO:teuthology.orchestra.run.smithi060.stdout: 410999 8 -rw-r--r-- 1 ubuntu ubuntu 6512 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/ImageCacheState.cc 2024-03-27T21:42:44.269 INFO:teuthology.orchestra.run.smithi060.stdout: 411003 4 -rw-r--r-- 1 ubuntu ubuntu 4025 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/LogEntry.cc 2024-03-27T21:42:44.269 INFO:teuthology.orchestra.run.smithi060.stdout: 411005 12 -rw-r--r-- 1 ubuntu ubuntu 10025 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/LogMap.cc 2024-03-27T21:42:44.269 INFO:teuthology.orchestra.run.smithi060.stdout: 411007 12 -rw-r--r-- 1 ubuntu ubuntu 11626 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/LogOperation.cc 2024-03-27T21:42:44.269 INFO:teuthology.orchestra.run.smithi060.stdout: 411000 4 -rw-r--r-- 1 ubuntu ubuntu 2251 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/pwl/ImageCacheState.h 2024-03-27T21:42:44.269 INFO:teuthology.orchestra.run.smithi060.stdout: 410991 8 -rw-r--r-- 1 ubuntu ubuntu 7751 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/WriteLogImageDispatch.cc 2024-03-27T21:42:44.269 INFO:teuthology.orchestra.run.smithi060.stdout: 410978 8 -rw-r--r-- 1 ubuntu ubuntu 5731 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/ImageWriteback.cc 2024-03-27T21:42:44.269 INFO:teuthology.orchestra.run.smithi060.stdout: 410992 4 -rw-r--r-- 1 ubuntu ubuntu 3942 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/WriteLogImageDispatch.h 2024-03-27T21:42:44.269 INFO:teuthology.orchestra.run.smithi060.stdout: 410981 8 -rw-r--r-- 1 ubuntu ubuntu 4671 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/ObjectCacherObjectDispatch.h 2024-03-27T21:42:44.270 INFO:teuthology.orchestra.run.smithi060.stdout: 410980 20 -rw-r--r-- 1 ubuntu ubuntu 17646 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/ObjectCacherObjectDispatch.cc 2024-03-27T21:42:44.270 INFO:teuthology.orchestra.run.smithi060.stdout: 410987 4 -rw-r--r-- 1 ubuntu ubuntu 557 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/Types.h 2024-03-27T21:42:44.270 INFO:teuthology.orchestra.run.smithi060.stdout: 410983 4 -rw-r--r-- 1 ubuntu ubuntu 2594 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/ObjectCacherWriteback.h 2024-03-27T21:42:44.270 INFO:teuthology.orchestra.run.smithi060.stdout: 410982 12 -rw-r--r-- 1 ubuntu ubuntu 10164 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/ObjectCacherWriteback.cc 2024-03-27T21:42:44.270 INFO:teuthology.orchestra.run.smithi060.stdout: 410989 20 -rw-r--r-- 1 ubuntu ubuntu 18026 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/WriteAroundObjectDispatch.cc 2024-03-27T21:42:44.270 INFO:teuthology.orchestra.run.smithi060.stdout: 410988 4 -rw-r--r-- 1 ubuntu ubuntu 699 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/Utils.h 2024-03-27T21:42:44.270 INFO:teuthology.orchestra.run.smithi060.stdout: 410985 8 -rw-r--r-- 1 ubuntu ubuntu 5482 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/ParentCacheObjectDispatch.h 2024-03-27T21:42:44.270 INFO:teuthology.orchestra.run.smithi060.stdout: 410986 4 -rw-r--r-- 1 ubuntu ubuntu 515 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/TypeTraits.h 2024-03-27T21:42:44.271 INFO:teuthology.orchestra.run.smithi060.stdout: 410990 8 -rw-r--r-- 1 ubuntu ubuntu 7237 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/WriteAroundObjectDispatch.h 2024-03-27T21:42:44.271 INFO:teuthology.orchestra.run.smithi060.stdout: 410984 12 -rw-r--r-- 1 ubuntu ubuntu 8298 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/ParentCacheObjectDispatch.cc 2024-03-27T21:42:44.271 INFO:teuthology.orchestra.run.smithi060.stdout: 410979 4 -rw-r--r-- 1 ubuntu ubuntu 2899 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/cache/ImageWriteback.h 2024-03-27T21:42:44.271 INFO:teuthology.orchestra.run.smithi060.stdout: 411146 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image_watcher 2024-03-27T21:42:44.271 INFO:teuthology.orchestra.run.smithi060.stdout: 411147 4 -rw-r--r-- 1 ubuntu ubuntu 2508 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image_watcher/NotifyLockOwner.cc 2024-03-27T21:42:44.271 INFO:teuthology.orchestra.run.smithi060.stdout: 411148 4 -rw-r--r-- 1 ubuntu ubuntu 1179 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/image_watcher/NotifyLockOwner.h 2024-03-27T21:42:44.271 INFO:teuthology.orchestra.run.smithi060.stdout: 410946 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api 2024-03-27T21:42:44.271 INFO:teuthology.orchestra.run.smithi060.stdout: 410950 4 -rw-r--r-- 1 ubuntu ubuntu 1851 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/DiffIterate.h 2024-03-27T21:42:44.271 INFO:teuthology.orchestra.run.smithi060.stdout: 410958 8 -rw-r--r-- 1 ubuntu ubuntu 4328 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Migration.h 2024-03-27T21:42:44.272 INFO:teuthology.orchestra.run.smithi060.stdout: 410961 8 -rw-r--r-- 1 ubuntu ubuntu 5911 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Namespace.cc 2024-03-27T21:42:44.272 INFO:teuthology.orchestra.run.smithi060.stdout: 410951 36 -rw-r--r-- 1 ubuntu ubuntu 35799 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Group.cc 2024-03-27T21:42:44.272 INFO:teuthology.orchestra.run.smithi060.stdout: 410959 68 -rw-r--r-- 1 ubuntu ubuntu 66186 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Mirror.cc 2024-03-27T21:42:44.272 INFO:teuthology.orchestra.run.smithi060.stdout: 410964 4 -rw-r--r-- 1 ubuntu ubuntu 913 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Pool.h 2024-03-27T21:42:44.272 INFO:teuthology.orchestra.run.smithi060.stdout: 410970 4 -rw-r--r-- 1 ubuntu ubuntu 1783 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Trash.h 2024-03-27T21:42:44.272 INFO:teuthology.orchestra.run.smithi060.stdout: 410968 4 -rw-r--r-- 1 ubuntu ubuntu 2125 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Snapshot.h 2024-03-27T21:42:44.272 INFO:teuthology.orchestra.run.smithi060.stdout: 410952 4 -rw-r--r-- 1 ubuntu ubuntu 2386 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Group.h 2024-03-27T21:42:44.273 INFO:teuthology.orchestra.run.smithi060.stdout: 410965 8 -rw-r--r-- 1 ubuntu ubuntu 4276 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/PoolMetadata.cc 2024-03-27T21:42:44.273 INFO:teuthology.orchestra.run.smithi060.stdout: 410956 4 -rw-r--r-- 1 ubuntu ubuntu 3000 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Io.h 2024-03-27T21:42:44.273 INFO:teuthology.orchestra.run.smithi060.stdout: 410955 20 -rw-r--r-- 1 ubuntu ubuntu 17870 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Io.cc 2024-03-27T21:42:44.273 INFO:teuthology.orchestra.run.smithi060.stdout: 410953 32 -rw-r--r-- 1 ubuntu ubuntu 30982 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Image.cc 2024-03-27T21:42:44.273 INFO:teuthology.orchestra.run.smithi060.stdout: 410962 4 -rw-r--r-- 1 ubuntu ubuntu 863 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Namespace.h 2024-03-27T21:42:44.273 INFO:teuthology.orchestra.run.smithi060.stdout: 410947 8 -rw-r--r-- 1 ubuntu ubuntu 6087 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Config.cc 2024-03-27T21:42:44.273 INFO:teuthology.orchestra.run.smithi060.stdout: 410957 64 -rw-r--r-- 1 ubuntu ubuntu 65448 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Migration.cc 2024-03-27T21:42:44.273 INFO:teuthology.orchestra.run.smithi060.stdout: 410949 16 -rw-r--r-- 1 ubuntu ubuntu 16165 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/DiffIterate.cc 2024-03-27T21:42:44.274 INFO:teuthology.orchestra.run.smithi060.stdout: 410966 4 -rw-r--r-- 1 ubuntu ubuntu 1030 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/PoolMetadata.h 2024-03-27T21:42:44.274 INFO:teuthology.orchestra.run.smithi060.stdout: 410948 4 -rw-r--r-- 1 ubuntu ubuntu 1032 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Config.h 2024-03-27T21:42:44.274 INFO:teuthology.orchestra.run.smithi060.stdout: 410960 8 -rw-r--r-- 1 ubuntu ubuntu 5990 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Mirror.h 2024-03-27T21:42:44.274 INFO:teuthology.orchestra.run.smithi060.stdout: 410972 4 -rw-r--r-- 1 ubuntu ubuntu 701 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Utils.h 2024-03-27T21:42:44.274 INFO:teuthology.orchestra.run.smithi060.stdout: 410967 12 -rw-r--r-- 1 ubuntu ubuntu 12026 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Snapshot.cc 2024-03-27T21:42:44.274 INFO:teuthology.orchestra.run.smithi060.stdout: 410969 28 -rw-r--r-- 1 ubuntu ubuntu 24816 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Trash.cc 2024-03-27T21:42:44.274 INFO:teuthology.orchestra.run.smithi060.stdout: 410954 4 -rw-r--r-- 1 ubuntu ubuntu 3265 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Image.h 2024-03-27T21:42:44.274 INFO:teuthology.orchestra.run.smithi060.stdout: 410971 4 -rw-r--r-- 1 ubuntu ubuntu 3607 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Utils.cc 2024-03-27T21:42:44.275 INFO:teuthology.orchestra.run.smithi060.stdout: 410963 12 -rw-r--r-- 1 ubuntu ubuntu 10605 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/api/Pool.cc 2024-03-27T21:42:44.275 INFO:teuthology.orchestra.run.smithi060.stdout: 410919 4 -rw-r--r-- 1 ubuntu ubuntu 3735 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/ImageState.h 2024-03-27T21:42:44.275 INFO:teuthology.orchestra.run.smithi060.stdout: 410902 4 -rw-r--r-- 1 ubuntu ubuntu 2763 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/AsyncObjectThrottle.cc 2024-03-27T21:42:44.275 INFO:teuthology.orchestra.run.smithi060.stdout: 410935 4 -rw-r--r-- 1 ubuntu ubuntu 1120 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/PluginRegistry.h 2024-03-27T21:42:44.275 INFO:teuthology.orchestra.run.smithi060.stdout: 411242 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration 2024-03-27T21:42:44.275 INFO:teuthology.orchestra.run.smithi060.stdout: 411253 12 -rw-r--r-- 1 ubuntu ubuntu 9909 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/NativeFormat.cc 2024-03-27T21:42:44.275 INFO:teuthology.orchestra.run.smithi060.stdout: 411259 8 -rw-r--r-- 1 ubuntu ubuntu 5972 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/QCOWFormat.h 2024-03-27T21:42:44.275 INFO:teuthology.orchestra.run.smithi060.stdout: 411260 8 -rw-r--r-- 1 ubuntu ubuntu 7465 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/RawFormat.cc 2024-03-27T21:42:44.275 INFO:teuthology.orchestra.run.smithi060.stdout: 411256 4 -rw-r--r-- 1 ubuntu ubuntu 2644 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/OpenSourceImageRequest.h 2024-03-27T21:42:44.276 INFO:teuthology.orchestra.run.smithi060.stdout: 411262 8 -rw-r--r-- 1 ubuntu ubuntu 6330 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/RawSnapshot.cc 2024-03-27T21:42:44.276 INFO:teuthology.orchestra.run.smithi060.stdout: 411263 4 -rw-r--r-- 1 ubuntu ubuntu 2256 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/RawSnapshot.h 2024-03-27T21:42:44.276 INFO:teuthology.orchestra.run.smithi060.stdout: 411243 8 -rw-r--r-- 1 ubuntu ubuntu 6301 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/FileStream.cc 2024-03-27T21:42:44.276 INFO:teuthology.orchestra.run.smithi060.stdout: 411248 4 -rw-r--r-- 1 ubuntu ubuntu 702 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/HttpProcessorInterface.h 2024-03-27T21:42:44.276 INFO:teuthology.orchestra.run.smithi060.stdout: 411265 4 -rw-r--r-- 1 ubuntu ubuntu 2039 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/S3Stream.h 2024-03-27T21:42:44.276 INFO:teuthology.orchestra.run.smithi060.stdout: 411247 8 -rw-r--r-- 1 ubuntu ubuntu 6322 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/HttpClient.h 2024-03-27T21:42:44.276 INFO:teuthology.orchestra.run.smithi060.stdout: 411252 4 -rw-r--r-- 1 ubuntu ubuntu 3741 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/ImageDispatch.h 2024-03-27T21:42:44.276 INFO:teuthology.orchestra.run.smithi060.stdout: 411254 4 -rw-r--r-- 1 ubuntu ubuntu 2579 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/NativeFormat.h 2024-03-27T21:42:44.277 INFO:teuthology.orchestra.run.smithi060.stdout: 411269 4 -rw-r--r-- 1 ubuntu ubuntu 783 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/StreamInterface.h 2024-03-27T21:42:44.277 INFO:teuthology.orchestra.run.smithi060.stdout: 411268 4 -rw-r--r-- 1 ubuntu ubuntu 1437 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/SourceSpecBuilder.h 2024-03-27T21:42:44.277 INFO:teuthology.orchestra.run.smithi060.stdout: 411255 8 -rw-r--r-- 1 ubuntu ubuntu 7603 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/OpenSourceImageRequest.cc 2024-03-27T21:42:44.277 INFO:teuthology.orchestra.run.smithi060.stdout: 411244 4 -rw-r--r-- 1 ubuntu ubuntu 1840 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/FileStream.h 2024-03-27T21:42:44.277 INFO:teuthology.orchestra.run.smithi060.stdout: 411261 4 -rw-r--r-- 1 ubuntu ubuntu 2455 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/RawFormat.h 2024-03-27T21:42:44.277 INFO:teuthology.orchestra.run.smithi060.stdout: 411267 8 -rw-r--r-- 1 ubuntu ubuntu 4674 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/SourceSpecBuilder.cc 2024-03-27T21:42:44.277 INFO:teuthology.orchestra.run.smithi060.stdout: 411270 4 -rw-r--r-- 1 ubuntu ubuntu 936 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/Types.h 2024-03-27T21:42:44.277 INFO:teuthology.orchestra.run.smithi060.stdout: 411246 32 -rw-r--r-- 1 ubuntu ubuntu 28892 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/HttpClient.cc 2024-03-27T21:42:44.277 INFO:teuthology.orchestra.run.smithi060.stdout: 411258 52 -rw-r--r-- 1 ubuntu ubuntu 51953 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/QCOWFormat.cc 2024-03-27T21:42:44.278 INFO:teuthology.orchestra.run.smithi060.stdout: 411264 8 -rw-r--r-- 1 ubuntu ubuntu 6118 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/S3Stream.cc 2024-03-27T21:42:44.278 INFO:teuthology.orchestra.run.smithi060.stdout: 411251 8 -rw-r--r-- 1 ubuntu ubuntu 5213 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/ImageDispatch.cc 2024-03-27T21:42:44.278 INFO:teuthology.orchestra.run.smithi060.stdout: 411272 4 -rw-r--r-- 1 ubuntu ubuntu 943 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/Utils.h 2024-03-27T21:42:44.278 INFO:teuthology.orchestra.run.smithi060.stdout: 411249 4 -rw-r--r-- 1 ubuntu ubuntu 2171 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/HttpStream.cc 2024-03-27T21:42:44.278 INFO:teuthology.orchestra.run.smithi060.stdout: 411271 8 -rw-r--r-- 1 ubuntu ubuntu 4523 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/Utils.cc 2024-03-27T21:42:44.278 INFO:teuthology.orchestra.run.smithi060.stdout: 411266 4 -rw-r--r-- 1 ubuntu ubuntu 1348 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/SnapshotInterface.h 2024-03-27T21:42:44.278 INFO:teuthology.orchestra.run.smithi060.stdout: 411250 4 -rw-r--r-- 1 ubuntu ubuntu 1750 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/HttpStream.h 2024-03-27T21:42:44.278 INFO:teuthology.orchestra.run.smithi060.stdout: 411245 4 -rw-r--r-- 1 ubuntu ubuntu 1563 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/FormatInterface.h 2024-03-27T21:42:44.279 INFO:teuthology.orchestra.run.smithi060.stdout: 411257 16 -rw-r--r-- 1 ubuntu ubuntu 15822 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/migration/QCOW.h 2024-03-27T21:42:44.279 INFO:teuthology.orchestra.run.smithi060.stdout: 411318 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirroring_watcher 2024-03-27T21:42:44.279 INFO:teuthology.orchestra.run.smithi060.stdout: 411319 4 -rw-r--r-- 1 ubuntu ubuntu 3770 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirroring_watcher/Types.cc 2024-03-27T21:42:44.279 INFO:teuthology.orchestra.run.smithi060.stdout: 411320 4 -rw-r--r-- 1 ubuntu ubuntu 2681 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/mirroring_watcher/Types.h 2024-03-27T21:42:44.279 INFO:teuthology.orchestra.run.smithi060.stdout: 411150 8 -rw-r--r-- 1 ubuntu ubuntu 5956 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librbd/internal.h 2024-03-27T21:42:44.279 INFO:teuthology.orchestra.run.smithi060.stdout: 416682 4 drwxr-xr-x 17 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools 2024-03-27T21:42:44.280 INFO:teuthology.orchestra.run.smithi060.stdout: 417054 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_nbd 2024-03-27T21:42:44.280 INFO:teuthology.orchestra.run.smithi060.stdout: 417058 4 -rwxr-xr-x 1 ubuntu ubuntu 552 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_nbd/rbd-nbd_quiesce 2024-03-27T21:42:44.280 INFO:teuthology.orchestra.run.smithi060.stdout: 417056 4 -rw-r--r-- 1 ubuntu ubuntu 2423 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_nbd/nbd-netlink.h 2024-03-27T21:42:44.280 INFO:teuthology.orchestra.run.smithi060.stdout: 417055 4 -rw-r--r-- 1 ubuntu ubuntu 170 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_nbd/CMakeLists.txt 2024-03-27T21:42:44.280 INFO:teuthology.orchestra.run.smithi060.stdout: 417057 64 -rw-r--r-- 1 ubuntu ubuntu 64293 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_nbd/rbd-nbd.cc 2024-03-27T21:42:44.280 INFO:teuthology.orchestra.run.smithi060.stdout: 417088 12 -rw-r--r-- 1 ubuntu ubuntu 10273 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rebuild_mondb.cc 2024-03-27T21:42:44.280 INFO:teuthology.orchestra.run.smithi060.stdout: 416717 4 -rw-r--r-- 1 ubuntu ubuntu 1481 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph_objectstore_tool.h 2024-03-27T21:42:44.280 INFO:teuthology.orchestra.run.smithi060.stdout: 417092 12 -rw-r--r-- 1 ubuntu ubuntu 8798 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/scratchtool.c 2024-03-27T21:42:44.281 INFO:teuthology.orchestra.run.smithi060.stdout: 417094 4 -rwxr-xr-x 1 ubuntu ubuntu 2994 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/setup-virtualenv.sh 2024-03-27T21:42:44.281 INFO:teuthology.orchestra.run.smithi060.stdout: 416719 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs 2024-03-27T21:42:44.281 INFO:teuthology.orchestra.run.smithi060.stdout: 416743 12 -rw-r--r-- 1 ubuntu ubuntu 10711 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/TableTool.cc 2024-03-27T21:42:44.281 INFO:teuthology.orchestra.run.smithi060.stdout: 416720 4 -rw-r--r-- 1 ubuntu ubuntu 1522 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/CMakeLists.txt 2024-03-27T21:42:44.281 INFO:teuthology.orchestra.run.smithi060.stdout: 416734 4 -rw-r--r-- 1 ubuntu ubuntu 1602 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/MDSUtility.h 2024-03-27T21:42:44.281 INFO:teuthology.orchestra.run.smithi060.stdout: 416745 4 -rw-r--r-- 1 ubuntu ubuntu 1012 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/cephfs-data-scan.cc 2024-03-27T21:42:44.281 INFO:teuthology.orchestra.run.smithi060.stdout: 416737 8 -rw-r--r-- 1 ubuntu ubuntu 4937 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/PgFiles.cc 2024-03-27T21:42:44.281 INFO:teuthology.orchestra.run.smithi060.stdout: 416723 16 -rw-r--r-- 1 ubuntu ubuntu 13264 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/Dumper.cc 2024-03-27T21:42:44.281 INFO:teuthology.orchestra.run.smithi060.stdout: 416732 4 -rw-r--r-- 1 ubuntu ubuntu 2745 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/JournalTool.h 2024-03-27T21:42:44.282 INFO:teuthology.orchestra.run.smithi060.stdout: 416750 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/shell 2024-03-27T21:42:44.282 INFO:teuthology.orchestra.run.smithi060.stdout: 416751 4 -rw-r--r-- 1 ubuntu ubuntu 134 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/shell/CMakeLists.txt 2024-03-27T21:42:44.282 INFO:teuthology.orchestra.run.smithi060.stdout: 416754 4 -rw-r--r-- 1 ubuntu ubuntu 133 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/shell/tox.ini 2024-03-27T21:42:44.282 INFO:teuthology.orchestra.run.smithi060.stdout: 416752 72 -rwxr-xr-x 1 ubuntu ubuntu 71725 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/shell/cephfs-shell 2024-03-27T21:42:44.282 INFO:teuthology.orchestra.run.smithi060.stdout: 416753 4 -rw-r--r-- 1 ubuntu ubuntu 704 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/shell/setup.py 2024-03-27T21:42:44.282 INFO:teuthology.orchestra.run.smithi060.stdout: 416724 4 -rw-r--r-- 1 ubuntu ubuntu 1125 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/Dumper.h 2024-03-27T21:42:44.282 INFO:teuthology.orchestra.run.smithi060.stdout: 416736 8 -rw-r--r-- 1 ubuntu ubuntu 6916 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/MetaTool.h 2024-03-27T21:42:44.283 INFO:teuthology.orchestra.run.smithi060.stdout: 416742 4 -rw-r--r-- 1 ubuntu ubuntu 761 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/RoleSelector.h 2024-03-27T21:42:44.283 INFO:teuthology.orchestra.run.smithi060.stdout: 416739 8 -rw-r--r-- 1 ubuntu ubuntu 5621 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/Resetter.cc 2024-03-27T21:42:44.283 INFO:teuthology.orchestra.run.smithi060.stdout: 416747 4 -rw-r--r-- 1 ubuntu ubuntu 3394 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/cephfs-meta-injection.cc 2024-03-27T21:42:44.283 INFO:teuthology.orchestra.run.smithi060.stdout: 416726 4 -rw-r--r-- 1 ubuntu ubuntu 920 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/EventOutput.h 2024-03-27T21:42:44.283 INFO:teuthology.orchestra.run.smithi060.stdout: 416760 4 -rw-r--r-- 1 ubuntu ubuntu 745 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/type_helper.hpp 2024-03-27T21:42:44.283 INFO:teuthology.orchestra.run.smithi060.stdout: 416731 40 -rw-r--r-- 1 ubuntu ubuntu 39285 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/JournalTool.cc 2024-03-27T21:42:44.283 INFO:teuthology.orchestra.run.smithi060.stdout: 416740 4 -rw-r--r-- 1 ubuntu ubuntu 1161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/Resetter.h 2024-03-27T21:42:44.283 INFO:teuthology.orchestra.run.smithi060.stdout: 416755 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/top 2024-03-27T21:42:44.283 INFO:teuthology.orchestra.run.smithi060.stdout: 416756 4 -rw-r--r-- 1 ubuntu ubuntu 255 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/top/CMakeLists.txt 2024-03-27T21:42:44.284 INFO:teuthology.orchestra.run.smithi060.stdout: 416759 4 -rw-r--r-- 1 ubuntu ubuntu 131 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/top/tox.ini 2024-03-27T21:42:44.284 INFO:teuthology.orchestra.run.smithi060.stdout: 416757 56 -rwxr-xr-x 1 ubuntu ubuntu 54103 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/top/cephfs-top 2024-03-27T21:42:44.284 INFO:teuthology.orchestra.run.smithi060.stdout: 416758 4 -rw-r--r-- 1 ubuntu ubuntu 662 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/top/setup.py 2024-03-27T21:42:44.284 INFO:teuthology.orchestra.run.smithi060.stdout: 416722 12 -rw-r--r-- 1 ubuntu ubuntu 8566 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/DataScan.h 2024-03-27T21:42:44.284 INFO:teuthology.orchestra.run.smithi060.stdout: 416729 16 -rw-r--r-- 1 ubuntu ubuntu 13240 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/JournalScanner.cc 2024-03-27T21:42:44.284 INFO:teuthology.orchestra.run.smithi060.stdout: 416741 4 -rw-r--r-- 1 ubuntu ubuntu 1543 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/RoleSelector.cc 2024-03-27T21:42:44.284 INFO:teuthology.orchestra.run.smithi060.stdout: 416738 4 -rw-r--r-- 1 ubuntu ubuntu 1161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/PgFiles.h 2024-03-27T21:42:44.284 INFO:teuthology.orchestra.run.smithi060.stdout: 416733 4 -rw-r--r-- 1 ubuntu ubuntu 3492 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/MDSUtility.cc 2024-03-27T21:42:44.285 INFO:teuthology.orchestra.run.smithi060.stdout: 416744 4 -rw-r--r-- 1 ubuntu ubuntu 931 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/TableTool.h 2024-03-27T21:42:44.285 INFO:teuthology.orchestra.run.smithi060.stdout: 416725 8 -rw-r--r-- 1 ubuntu ubuntu 4601 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/EventOutput.cc 2024-03-27T21:42:44.285 INFO:teuthology.orchestra.run.smithi060.stdout: 416748 4 -rw-r--r-- 1 ubuntu ubuntu 994 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/cephfs-table-tool.cc 2024-03-27T21:42:44.285 INFO:teuthology.orchestra.run.smithi060.stdout: 416746 4 -rw-r--r-- 1 ubuntu ubuntu 1391 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/cephfs-journal-tool.cc 2024-03-27T21:42:44.285 INFO:teuthology.orchestra.run.smithi060.stdout: 416749 8 -rw-r--r-- 1 ubuntu ubuntu 5901 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/first-damage.py 2024-03-27T21:42:44.285 INFO:teuthology.orchestra.run.smithi060.stdout: 416730 4 -rw-r--r-- 1 ubuntu ubuntu 3519 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/JournalScanner.h 2024-03-27T21:42:44.285 INFO:teuthology.orchestra.run.smithi060.stdout: 416735 28 -rw-r--r-- 1 ubuntu ubuntu 28460 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/MetaTool.cc 2024-03-27T21:42:44.285 INFO:teuthology.orchestra.run.smithi060.stdout: 416728 4 -rw-r--r-- 1 ubuntu ubuntu 1719 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/JournalFilter.h 2024-03-27T21:42:44.286 INFO:teuthology.orchestra.run.smithi060.stdout: 416727 12 -rw-r--r-- 1 ubuntu ubuntu 9467 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/JournalFilter.cc 2024-03-27T21:42:44.286 INFO:teuthology.orchestra.run.smithi060.stdout: 416721 72 -rw-r--r-- 1 ubuntu ubuntu 73118 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs/DataScan.cc 2024-03-27T21:42:44.286 INFO:teuthology.orchestra.run.smithi060.stdout: 416823 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rados 2024-03-27T21:42:44.286 INFO:teuthology.orchestra.run.smithi060.stdout: 416824 8 -rw-r--r-- 1 ubuntu ubuntu 4559 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rados/PoolDump.cc 2024-03-27T21:42:44.286 INFO:teuthology.orchestra.run.smithi060.stdout: 416825 4 -rw-r--r-- 1 ubuntu ubuntu 695 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rados/PoolDump.h 2024-03-27T21:42:44.286 INFO:teuthology.orchestra.run.smithi060.stdout: 416826 12 -rw-r--r-- 1 ubuntu ubuntu 11332 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rados/RadosImport.cc 2024-03-27T21:42:44.286 INFO:teuthology.orchestra.run.smithi060.stdout: 416828 128 -rw-r--r-- 1 ubuntu ubuntu 129291 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rados/rados.cc 2024-03-27T21:42:44.286 INFO:teuthology.orchestra.run.smithi060.stdout: 416827 4 -rw-r--r-- 1 ubuntu ubuntu 1120 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rados/RadosImport.h 2024-03-27T21:42:44.287 INFO:teuthology.orchestra.run.smithi060.stdout: 416683 8 -rw-r--r-- 1 ubuntu ubuntu 4707 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/CMakeLists.txt 2024-03-27T21:42:44.287 INFO:teuthology.orchestra.run.smithi060.stdout: 416716 136 -rw-r--r-- 1 ubuntu ubuntu 136972 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph_objectstore_tool.cc 2024-03-27T21:42:44.287 INFO:teuthology.orchestra.run.smithi060.stdout: 416817 12 -rw-r--r-- 1 ubuntu ubuntu 9897 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/kvstore_tool.cc 2024-03-27T21:42:44.287 INFO:teuthology.orchestra.run.smithi060.stdout: 416761 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror 2024-03-27T21:42:44.287 INFO:teuthology.orchestra.run.smithi060.stdout: 416762 4 -rw-r--r-- 1 ubuntu ubuntu 523 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/CMakeLists.txt 2024-03-27T21:42:44.287 INFO:teuthology.orchestra.run.smithi060.stdout: 416772 4 -rw-r--r-- 1 ubuntu ubuntu 2187 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/MirrorWatcher.h 2024-03-27T21:42:44.287 INFO:teuthology.orchestra.run.smithi060.stdout: 416769 20 -rw-r--r-- 1 ubuntu ubuntu 19517 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/Mirror.cc 2024-03-27T21:42:44.287 INFO:teuthology.orchestra.run.smithi060.stdout: 416784 4 -rw-r--r-- 1 ubuntu ubuntu 3429 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/main.cc 2024-03-27T21:42:44.288 INFO:teuthology.orchestra.run.smithi060.stdout: 416768 4 -rw-r--r-- 1 ubuntu ubuntu 2321 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/InstanceWatcher.h 2024-03-27T21:42:44.288 INFO:teuthology.orchestra.run.smithi060.stdout: 416785 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/watcher 2024-03-27T21:42:44.288 INFO:teuthology.orchestra.run.smithi060.stdout: 416786 4 -rw-r--r-- 1 ubuntu ubuntu 2617 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/watcher/RewatchRequest.cc 2024-03-27T21:42:44.288 INFO:teuthology.orchestra.run.smithi060.stdout: 416787 4 -rw-r--r-- 1 ubuntu ubuntu 1581 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/watcher/RewatchRequest.h 2024-03-27T21:42:44.288 INFO:teuthology.orchestra.run.smithi060.stdout: 416773 56 -rw-r--r-- 1 ubuntu ubuntu 53279 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/PeerReplayer.cc 2024-03-27T21:42:44.288 INFO:teuthology.orchestra.run.smithi060.stdout: 416777 4 -rw-r--r-- 1 ubuntu ubuntu 555 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/Types.cc 2024-03-27T21:42:44.288 INFO:teuthology.orchestra.run.smithi060.stdout: 416783 4 -rw-r--r-- 1 ubuntu ubuntu 1137 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/aio_utils.h 2024-03-27T21:42:44.288 INFO:teuthology.orchestra.run.smithi060.stdout: 416764 4 -rw-r--r-- 1 ubuntu ubuntu 2000 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/ClusterWatcher.h 2024-03-27T21:42:44.289 INFO:teuthology.orchestra.run.smithi060.stdout: 416778 4 -rw-r--r-- 1 ubuntu ubuntu 2221 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/Types.h 2024-03-27T21:42:44.289 INFO:teuthology.orchestra.run.smithi060.stdout: 416781 8 -rw-r--r-- 1 ubuntu ubuntu 8024 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/Watcher.cc 2024-03-27T21:42:44.289 INFO:teuthology.orchestra.run.smithi060.stdout: 416782 4 -rw-r--r-- 1 ubuntu ubuntu 2590 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/Watcher.h 2024-03-27T21:42:44.289 INFO:teuthology.orchestra.run.smithi060.stdout: 416770 8 -rw-r--r-- 1 ubuntu ubuntu 4300 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/Mirror.h 2024-03-27T21:42:44.289 INFO:teuthology.orchestra.run.smithi060.stdout: 416766 8 -rw-r--r-- 1 ubuntu ubuntu 4438 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/FSMirror.h 2024-03-27T21:42:44.289 INFO:teuthology.orchestra.run.smithi060.stdout: 416765 16 -rw-r--r-- 1 ubuntu ubuntu 14015 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/FSMirror.cc 2024-03-27T21:42:44.289 INFO:teuthology.orchestra.run.smithi060.stdout: 416780 4 -rw-r--r-- 1 ubuntu ubuntu 621 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/Utils.h 2024-03-27T21:42:44.289 INFO:teuthology.orchestra.run.smithi060.stdout: 416776 4 -rw-r--r-- 1 ubuntu ubuntu 1713 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/ServiceDaemon.h 2024-03-27T21:42:44.289 INFO:teuthology.orchestra.run.smithi060.stdout: 416779 8 -rw-r--r-- 1 ubuntu ubuntu 4870 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/Utils.cc 2024-03-27T21:42:44.290 INFO:teuthology.orchestra.run.smithi060.stdout: 416763 8 -rw-r--r-- 1 ubuntu ubuntu 5203 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/ClusterWatcher.cc 2024-03-27T21:42:44.290 INFO:teuthology.orchestra.run.smithi060.stdout: 416774 12 -rw-r--r-- 1 ubuntu ubuntu 11406 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/PeerReplayer.h 2024-03-27T21:42:44.290 INFO:teuthology.orchestra.run.smithi060.stdout: 416775 8 -rw-r--r-- 1 ubuntu ubuntu 6223 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/ServiceDaemon.cc 2024-03-27T21:42:44.290 INFO:teuthology.orchestra.run.smithi060.stdout: 416767 8 -rw-r--r-- 1 ubuntu ubuntu 6861 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/InstanceWatcher.cc 2024-03-27T21:42:44.290 INFO:teuthology.orchestra.run.smithi060.stdout: 416771 8 -rw-r--r-- 1 ubuntu ubuntu 4298 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/cephfs_mirror/MirrorWatcher.cc 2024-03-27T21:42:44.290 INFO:teuthology.orchestra.run.smithi060.stdout: 416715 48 -rw-r--r-- 1 ubuntu ubuntu 46350 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph_monstore_tool.cc 2024-03-27T21:42:44.290 INFO:teuthology.orchestra.run.smithi060.stdout: 416718 8 -rw-r--r-- 1 ubuntu ubuntu 6624 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph_osdomap_tool.cc 2024-03-27T21:42:44.290 INFO:teuthology.orchestra.run.smithi060.stdout: 416685 12 -rw-r--r-- 1 ubuntu ubuntu 10275 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/RadosDump.h 2024-03-27T21:42:44.291 INFO:teuthology.orchestra.run.smithi060.stdout: 416714 12 -rw-r--r-- 1 ubuntu ubuntu 9644 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph_kvstore_tool.cc 2024-03-27T21:42:44.291 INFO:teuthology.orchestra.run.smithi060.stdout: 417093 12 -rw-r--r-- 1 ubuntu ubuntu 8612 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/scratchtoolpp.cc 2024-03-27T21:42:44.291 INFO:teuthology.orchestra.run.smithi060.stdout: 416821 36 -rw-r--r-- 1 ubuntu ubuntu 35117 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/osdmaptool.cc 2024-03-27T21:42:44.291 INFO:teuthology.orchestra.run.smithi060.stdout: 417059 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_recover_tool 2024-03-27T21:42:44.291 INFO:teuthology.orchestra.run.smithi060.stdout: 417071 4 -rwxr-xr-x 1 ubuntu ubuntu 3759 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_recover_tool/osd_job 2024-03-27T21:42:44.291 INFO:teuthology.orchestra.run.smithi060.stdout: 417069 4 -rw-r--r-- 1 ubuntu ubuntu 2837 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_recover_tool/epoch_h 2024-03-27T21:42:44.291 INFO:teuthology.orchestra.run.smithi060.stdout: 417060 4 -rw-r--r-- 1 ubuntu ubuntu 834 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_recover_tool/FAQ 2024-03-27T21:42:44.291 INFO:teuthology.orchestra.run.smithi060.stdout: 417073 16 -rwxr-xr-x 1 ubuntu ubuntu 13603 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_recover_tool/test_rbd_recover_tool.sh 2024-03-27T21:42:44.291 INFO:teuthology.orchestra.run.smithi060.stdout: 417063 12 -rw-r--r-- 1 ubuntu ubuntu 9845 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_recover_tool/common_h 2024-03-27T21:42:44.292 INFO:teuthology.orchestra.run.smithi060.stdout: 417061 4 -rw-r--r-- 1 ubuntu ubuntu 4072 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_recover_tool/README 2024-03-27T21:42:44.292 INFO:teuthology.orchestra.run.smithi060.stdout: 417062 4 -rw-r--r-- 1 ubuntu ubuntu 22 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_recover_tool/TODO 2024-03-27T21:42:44.292 INFO:teuthology.orchestra.run.smithi060.stdout: 417070 12 -rw-r--r-- 1 ubuntu ubuntu 10554 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_recover_tool/metadata_h 2024-03-27T21:42:44.292 INFO:teuthology.orchestra.run.smithi060.stdout: 417068 36 -rw-r--r-- 1 ubuntu ubuntu 35184 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_recover_tool/database_h 2024-03-27T21:42:44.292 INFO:teuthology.orchestra.run.smithi060.stdout: 417064 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_recover_tool/config 2024-03-27T21:42:44.292 INFO:teuthology.orchestra.run.smithi060.stdout: 417066 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_recover_tool/config/mon_host 2024-03-27T21:42:44.292 INFO:teuthology.orchestra.run.smithi060.stdout: 417065 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_recover_tool/config/mds_host 2024-03-27T21:42:44.292 INFO:teuthology.orchestra.run.smithi060.stdout: 417067 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_recover_tool/config/osd_host_path 2024-03-27T21:42:44.293 INFO:teuthology.orchestra.run.smithi060.stdout: 417072 8 -rwxr-xr-x 1 ubuntu ubuntu 7864 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_recover_tool/rbd-recover-tool 2024-03-27T21:42:44.293 INFO:teuthology.orchestra.run.smithi060.stdout: 417090 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rgw 2024-03-27T21:42:44.293 INFO:teuthology.orchestra.run.smithi060.stdout: 417091 8 -rwxr-xr-x 1 ubuntu ubuntu 5850 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rgw/parse-cr-dump.py 2024-03-27T21:42:44.293 INFO:teuthology.orchestra.run.smithi060.stdout: 416829 4 -rw-r--r-- 1 ubuntu ubuntu 3802 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/radosacl.cc 2024-03-27T21:42:44.293 INFO:teuthology.orchestra.run.smithi060.stdout: 416706 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-lazy 2024-03-27T21:42:44.293 INFO:teuthology.orchestra.run.smithi060.stdout: 416709 20 -rwxr-xr-x 1 ubuntu ubuntu 16653 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-lazy/ceph-lazy 2024-03-27T21:42:44.293 INFO:teuthology.orchestra.run.smithi060.stdout: 416707 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-lazy/bash_completion.d 2024-03-27T21:42:44.293 INFO:teuthology.orchestra.run.smithi060.stdout: 416708 4 -rw-r--r-- 1 ubuntu ubuntu 916 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-lazy/bash_completion.d/ceph-lazy 2024-03-27T21:42:44.294 INFO:teuthology.orchestra.run.smithi060.stdout: 416792 44 -rw-r--r-- 1 ubuntu ubuntu 44467 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/crushtool.cc 2024-03-27T21:42:44.294 INFO:teuthology.orchestra.run.smithi060.stdout: 416819 16 -rw-r--r-- 1 ubuntu ubuntu 14244 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/monmaptool.cc 2024-03-27T21:42:44.294 INFO:teuthology.orchestra.run.smithi060.stdout: 416822 4 -rw-r--r-- 1 ubuntu ubuntu 3210 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/psim.cc 2024-03-27T21:42:44.294 INFO:teuthology.orchestra.run.smithi060.stdout: 416820 12 -rw-r--r-- 1 ubuntu ubuntu 10459 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/neorados.cc 2024-03-27T21:42:44.294 INFO:teuthology.orchestra.run.smithi060.stdout: 416684 4 -rw-r--r-- 1 ubuntu ubuntu 3225 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/RadosDump.cc 2024-03-27T21:42:44.294 INFO:teuthology.orchestra.run.smithi060.stdout: 416796 8 -rwxr-xr-x 1 ubuntu ubuntu 5003 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/histogram_dump.py 2024-03-27T21:42:44.294 INFO:teuthology.orchestra.run.smithi060.stdout: 416892 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_ggate 2024-03-27T21:42:44.294 INFO:teuthology.orchestra.run.smithi060.stdout: 416894 4 -rw-r--r-- 1 ubuntu ubuntu 3501 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_ggate/Driver.cc 2024-03-27T21:42:44.295 INFO:teuthology.orchestra.run.smithi060.stdout: 416896 4 -rw-r--r-- 1 ubuntu ubuntu 936 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_ggate/Request.h 2024-03-27T21:42:44.295 INFO:teuthology.orchestra.run.smithi060.stdout: 416897 8 -rw-r--r-- 1 ubuntu ubuntu 6259 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_ggate/Server.cc 2024-03-27T21:42:44.295 INFO:teuthology.orchestra.run.smithi060.stdout: 416893 4 -rw-r--r-- 1 ubuntu ubuntu 202 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_ggate/CMakeLists.txt 2024-03-27T21:42:44.295 INFO:teuthology.orchestra.run.smithi060.stdout: 416903 12 -rw-r--r-- 1 ubuntu ubuntu 9251 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_ggate/ggate_drv.c 2024-03-27T21:42:44.295 INFO:teuthology.orchestra.run.smithi060.stdout: 416905 16 -rw-r--r-- 1 ubuntu ubuntu 13814 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_ggate/main.cc 2024-03-27T21:42:44.295 INFO:teuthology.orchestra.run.smithi060.stdout: 416904 4 -rw-r--r-- 1 ubuntu ubuntu 1618 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_ggate/ggate_drv.h 2024-03-27T21:42:44.295 INFO:teuthology.orchestra.run.smithi060.stdout: 416901 4 -rw-r--r-- 1 ubuntu ubuntu 1058 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_ggate/debug.cc 2024-03-27T21:42:44.295 INFO:teuthology.orchestra.run.smithi060.stdout: 416899 4 -rw-r--r-- 1 ubuntu ubuntu 1210 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_ggate/Watcher.cc 2024-03-27T21:42:44.296 INFO:teuthology.orchestra.run.smithi060.stdout: 416902 4 -rw-r--r-- 1 ubuntu ubuntu 405 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_ggate/debug.h 2024-03-27T21:42:44.296 INFO:teuthology.orchestra.run.smithi060.stdout: 416900 4 -rw-r--r-- 1 ubuntu ubuntu 616 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_ggate/Watcher.h 2024-03-27T21:42:44.296 INFO:teuthology.orchestra.run.smithi060.stdout: 416898 4 -rw-r--r-- 1 ubuntu ubuntu 1803 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_ggate/Server.h 2024-03-27T21:42:44.296 INFO:teuthology.orchestra.run.smithi060.stdout: 416895 4 -rw-r--r-- 1 ubuntu ubuntu 987 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_ggate/Driver.h 2024-03-27T21:42:44.296 INFO:teuthology.orchestra.run.smithi060.stdout: 416686 8 -rw-r--r-- 1 ubuntu ubuntu 4709 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-client-debug.cc 2024-03-27T21:42:44.296 INFO:teuthology.orchestra.run.smithi060.stdout: 416712 12 -rw-r--r-- 1 ubuntu ubuntu 8788 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph_conf.cc 2024-03-27T21:42:44.296 INFO:teuthology.orchestra.run.smithi060.stdout: 416687 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-dencoder 2024-03-27T21:42:44.296 INFO:teuthology.orchestra.run.smithi060.stdout: 416695 4 -rw-r--r-- 1 ubuntu ubuntu 812 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-dencoder/mds_types.cc 2024-03-27T21:42:44.296 INFO:teuthology.orchestra.run.smithi060.stdout: 416701 4 -rw-r--r-- 1 ubuntu ubuntu 812 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-dencoder/rgw_types.cc 2024-03-27T21:42:44.297 INFO:teuthology.orchestra.run.smithi060.stdout: 416699 4 -rw-r--r-- 1 ubuntu ubuntu 812 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-dencoder/rbd_types.cc 2024-03-27T21:42:44.297 INFO:teuthology.orchestra.run.smithi060.stdout: 416688 4 -rw-r--r-- 1 ubuntu ubuntu 2621 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-dencoder/CMakeLists.txt 2024-03-27T21:42:44.297 INFO:teuthology.orchestra.run.smithi060.stdout: 416702 8 -rw-r--r-- 1 ubuntu ubuntu 5163 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-dencoder/rgw_types.h 2024-03-27T21:42:44.297 INFO:teuthology.orchestra.run.smithi060.stdout: 416689 12 -rw-r--r-- 1 ubuntu ubuntu 8299 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-dencoder/ceph_dencoder.cc 2024-03-27T21:42:44.297 INFO:teuthology.orchestra.run.smithi060.stdout: 416697 4 -rw-r--r-- 1 ubuntu ubuntu 989 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-dencoder/osd_types.cc 2024-03-27T21:42:44.297 INFO:teuthology.orchestra.run.smithi060.stdout: 416698 8 -rw-r--r-- 1 ubuntu ubuntu 5173 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-dencoder/osd_types.h 2024-03-27T21:42:44.297 INFO:teuthology.orchestra.run.smithi060.stdout: 416693 4 -rw-r--r-- 1 ubuntu ubuntu 2671 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-dencoder/denc_plugin.h 2024-03-27T21:42:44.297 INFO:teuthology.orchestra.run.smithi060.stdout: 416696 4 -rw-r--r-- 1 ubuntu ubuntu 2515 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-dencoder/mds_types.h 2024-03-27T21:42:44.298 INFO:teuthology.orchestra.run.smithi060.stdout: 416691 4 -rw-r--r-- 1 ubuntu ubuntu 818 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-dencoder/common_types.cc 2024-03-27T21:42:44.298 INFO:teuthology.orchestra.run.smithi060.stdout: 416690 4 -rw-r--r-- 1 ubuntu ubuntu 1918 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-dencoder/ceph_time.h 2024-03-27T21:42:44.298 INFO:teuthology.orchestra.run.smithi060.stdout: 416703 4 -rw-r--r-- 1 ubuntu ubuntu 1240 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-dencoder/sstring.h 2024-03-27T21:42:44.298 INFO:teuthology.orchestra.run.smithi060.stdout: 416704 4 -rw-r--r-- 1 ubuntu ubuntu 821 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-dencoder/str.h 2024-03-27T21:42:44.298 INFO:teuthology.orchestra.run.smithi060.stdout: 416692 16 -rw-r--r-- 1 ubuntu ubuntu 12589 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-dencoder/common_types.h 2024-03-27T21:42:44.298 INFO:teuthology.orchestra.run.smithi060.stdout: 416694 8 -rw-r--r-- 1 ubuntu ubuntu 6212 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-dencoder/denc_registry.h 2024-03-27T21:42:44.298 INFO:teuthology.orchestra.run.smithi060.stdout: 416700 4 -rw-r--r-- 1 ubuntu ubuntu 1601 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-dencoder/rbd_types.h 2024-03-27T21:42:44.298 INFO:teuthology.orchestra.run.smithi060.stdout: 416830 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd 2024-03-27T21:42:44.299 INFO:teuthology.orchestra.run.smithi060.stdout: 416841 4 -rw-r--r-- 1 ubuntu ubuntu 1759 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/Schedule.h 2024-03-27T21:42:44.299 INFO:teuthology.orchestra.run.smithi060.stdout: 416837 4 -rw-r--r-- 1 ubuntu ubuntu 1886 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/MirrorDaemonServiceInfo.h 2024-03-27T21:42:44.299 INFO:teuthology.orchestra.run.smithi060.stdout: 416833 4 -rw-r--r-- 1 ubuntu ubuntu 2107 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/CMakeLists.txt 2024-03-27T21:42:44.299 INFO:teuthology.orchestra.run.smithi060.stdout: 416843 4 -rw-r--r-- 1 ubuntu ubuntu 2482 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/Shell.h 2024-03-27T21:42:44.299 INFO:teuthology.orchestra.run.smithi060.stdout: 416840 12 -rw-r--r-- 1 ubuntu ubuntu 9856 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/Schedule.cc 2024-03-27T21:42:44.299 INFO:teuthology.orchestra.run.smithi060.stdout: 416839 4 -rw-r--r-- 1 ubuntu ubuntu 1207 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/OptionPrinter.h 2024-03-27T21:42:44.299 INFO:teuthology.orchestra.run.smithi060.stdout: 416891 4 -rw-r--r-- 1 ubuntu ubuntu 283 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/rbd.cc 2024-03-27T21:42:44.299 INFO:teuthology.orchestra.run.smithi060.stdout: 416832 12 -rw-r--r-- 1 ubuntu ubuntu 9170 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/ArgumentTypes.h 2024-03-27T21:42:44.300 INFO:teuthology.orchestra.run.smithi060.stdout: 416835 4 -rw-r--r-- 1 ubuntu ubuntu 1429 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/IndentStream.h 2024-03-27T21:42:44.300 INFO:teuthology.orchestra.run.smithi060.stdout: 416842 20 -rw-r--r-- 1 ubuntu ubuntu 16481 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/Shell.cc 2024-03-27T21:42:44.300 INFO:teuthology.orchestra.run.smithi060.stdout: 416831 20 -rw-r--r-- 1 ubuntu ubuntu 19433 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/ArgumentTypes.cc 2024-03-27T21:42:44.300 INFO:teuthology.orchestra.run.smithi060.stdout: 416836 12 -rw-r--r-- 1 ubuntu ubuntu 8701 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/MirrorDaemonServiceInfo.cc 2024-03-27T21:42:44.300 INFO:teuthology.orchestra.run.smithi060.stdout: 416845 12 -rw-r--r-- 1 ubuntu ubuntu 9916 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/Utils.h 2024-03-27T21:42:44.300 INFO:teuthology.orchestra.run.smithi060.stdout: 416844 36 -rw-r--r-- 1 ubuntu ubuntu 35865 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/Utils.cc 2024-03-27T21:42:44.300 INFO:teuthology.orchestra.run.smithi060.stdout: 416834 4 -rw-r--r-- 1 ubuntu ubuntu 1532 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/IndentStream.cc 2024-03-27T21:42:44.300 INFO:teuthology.orchestra.run.smithi060.stdout: 416846 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action 2024-03-27T21:42:44.301 INFO:teuthology.orchestra.run.smithi060.stdout: 416849 4 -rw-r--r-- 1 ubuntu ubuntu 2991 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Clone.cc 2024-03-27T21:42:44.301 INFO:teuthology.orchestra.run.smithi060.stdout: 416874 8 -rw-r--r-- 1 ubuntu ubuntu 5038 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Namespace.cc 2024-03-27T21:42:44.301 INFO:teuthology.orchestra.run.smithi060.stdout: 416883 32 -rw-r--r-- 1 ubuntu ubuntu 29881 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Snap.cc 2024-03-27T21:42:44.301 INFO:teuthology.orchestra.run.smithi060.stdout: 416889 8 -rw-r--r-- 1 ubuntu ubuntu 4189 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Watch.cc 2024-03-27T21:42:44.301 INFO:teuthology.orchestra.run.smithi060.stdout: 416878 4 -rw-r--r-- 1 ubuntu ubuntu 3569 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/PersistentCache.cc 2024-03-27T21:42:44.301 INFO:teuthology.orchestra.run.smithi060.stdout: 416861 28 -rw-r--r-- 1 ubuntu ubuntu 27199 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Group.cc 2024-03-27T21:42:44.301 INFO:teuthology.orchestra.run.smithi060.stdout: 416860 8 -rw-r--r-- 1 ubuntu ubuntu 4794 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Ggate.cc 2024-03-27T21:42:44.301 INFO:teuthology.orchestra.run.smithi060.stdout: 416864 16 -rw-r--r-- 1 ubuntu ubuntu 13586 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Info.cc 2024-03-27T21:42:44.302 INFO:teuthology.orchestra.run.smithi060.stdout: 416858 4 -rw-r--r-- 1 ubuntu ubuntu 3523 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Feature.cc 2024-03-27T21:42:44.302 INFO:teuthology.orchestra.run.smithi060.stdout: 416866 24 -rw-r--r-- 1 ubuntu ubuntu 20752 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Kernel.cc 2024-03-27T21:42:44.302 INFO:teuthology.orchestra.run.smithi060.stdout: 416875 12 -rw-r--r-- 1 ubuntu ubuntu 11882 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Nbd.cc 2024-03-27T21:42:44.302 INFO:teuthology.orchestra.run.smithi060.stdout: 416857 20 -rw-r--r-- 1 ubuntu ubuntu 17153 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Export.cc 2024-03-27T21:42:44.302 INFO:teuthology.orchestra.run.smithi060.stdout: 416856 4 -rw-r--r-- 1 ubuntu ubuntu 3826 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Encryption.cc 2024-03-27T21:42:44.302 INFO:teuthology.orchestra.run.smithi060.stdout: 416865 36 -rw-r--r-- 1 ubuntu ubuntu 33778 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Journal.cc 2024-03-27T21:42:44.302 INFO:teuthology.orchestra.run.smithi060.stdout: 416877 20 -rw-r--r-- 1 ubuntu ubuntu 19365 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Perf.cc 2024-03-27T21:42:44.302 INFO:teuthology.orchestra.run.smithi060.stdout: 416872 56 -rw-r--r-- 1 ubuntu ubuntu 54433 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/MirrorPool.cc 2024-03-27T21:42:44.303 INFO:teuthology.orchestra.run.smithi060.stdout: 416881 4 -rw-r--r-- 1 ubuntu ubuntu 2878 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Rename.cc 2024-03-27T21:42:44.303 INFO:teuthology.orchestra.run.smithi060.stdout: 416848 8 -rw-r--r-- 1 ubuntu ubuntu 5139 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Children.cc 2024-03-27T21:42:44.303 INFO:teuthology.orchestra.run.smithi060.stdout: 416882 4 -rw-r--r-- 1 ubuntu ubuntu 3434 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Resize.cc 2024-03-27T21:42:44.303 INFO:teuthology.orchestra.run.smithi060.stdout: 416873 12 -rw-r--r-- 1 ubuntu ubuntu 8440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/MirrorSnapshotSchedule.cc 2024-03-27T21:42:44.303 INFO:teuthology.orchestra.run.smithi060.stdout: 416855 12 -rw-r--r-- 1 ubuntu ubuntu 11669 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/DiskUsage.cc 2024-03-27T21:42:44.303 INFO:teuthology.orchestra.run.smithi060.stdout: 416880 8 -rw-r--r-- 1 ubuntu ubuntu 5037 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Remove.cc 2024-03-27T21:42:44.303 INFO:teuthology.orchestra.run.smithi060.stdout: 416851 8 -rw-r--r-- 1 ubuntu ubuntu 5909 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Copy.cc 2024-03-27T21:42:44.303 INFO:teuthology.orchestra.run.smithi060.stdout: 416887 12 -rw-r--r-- 1 ubuntu ubuntu 9404 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/TrashPurgeSchedule.cc 2024-03-27T21:42:44.304 INFO:teuthology.orchestra.run.smithi060.stdout: 416884 4 -rw-r--r-- 1 ubuntu ubuntu 2271 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Sparsify.cc 2024-03-27T21:42:44.304 INFO:teuthology.orchestra.run.smithi060.stdout: 416871 20 -rw-r--r-- 1 ubuntu ubuntu 19713 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/MirrorImage.cc 2024-03-27T21:42:44.304 INFO:teuthology.orchestra.run.smithi060.stdout: 416850 24 -rw-r--r-- 1 ubuntu ubuntu 23927 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Config.cc 2024-03-27T21:42:44.304 INFO:teuthology.orchestra.run.smithi060.stdout: 416870 16 -rw-r--r-- 1 ubuntu ubuntu 13322 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Migration.cc 2024-03-27T21:42:44.304 INFO:teuthology.orchestra.run.smithi060.stdout: 416859 4 -rw-r--r-- 1 ubuntu ubuntu 2483 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Flatten.cc 2024-03-27T21:42:44.304 INFO:teuthology.orchestra.run.smithi060.stdout: 416869 12 -rw-r--r-- 1 ubuntu ubuntu 10854 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/MergeDiff.cc 2024-03-27T21:42:44.304 INFO:teuthology.orchestra.run.smithi060.stdout: 416868 8 -rw-r--r-- 1 ubuntu ubuntu 7862 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Lock.cc 2024-03-27T21:42:44.304 INFO:teuthology.orchestra.run.smithi060.stdout: 416876 4 -rw-r--r-- 1 ubuntu ubuntu 3767 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/ObjectMap.cc 2024-03-27T21:42:44.305 INFO:teuthology.orchestra.run.smithi060.stdout: 416863 28 -rw-r--r-- 1 ubuntu ubuntu 27270 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Import.cc 2024-03-27T21:42:44.305 INFO:teuthology.orchestra.run.smithi060.stdout: 416854 4 -rw-r--r-- 1 ubuntu ubuntu 3928 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Diff.cc 2024-03-27T21:42:44.305 INFO:teuthology.orchestra.run.smithi060.stdout: 416886 20 -rw-r--r-- 1 ubuntu ubuntu 16639 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Trash.cc 2024-03-27T21:42:44.305 INFO:teuthology.orchestra.run.smithi060.stdout: 416852 8 -rw-r--r-- 1 ubuntu ubuntu 6794 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Create.cc 2024-03-27T21:42:44.305 INFO:teuthology.orchestra.run.smithi060.stdout: 416879 8 -rw-r--r-- 1 ubuntu ubuntu 5047 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Pool.cc 2024-03-27T21:42:44.305 INFO:teuthology.orchestra.run.smithi060.stdout: 416867 12 -rw-r--r-- 1 ubuntu ubuntu 9866 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/List.cc 2024-03-27T21:42:44.305 INFO:teuthology.orchestra.run.smithi060.stdout: 416890 8 -rw-r--r-- 1 ubuntu ubuntu 4551 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Wnbd.cc 2024-03-27T21:42:44.305 INFO:teuthology.orchestra.run.smithi060.stdout: 416853 12 -rw-r--r-- 1 ubuntu ubuntu 10206 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Device.cc 2024-03-27T21:42:44.305 INFO:teuthology.orchestra.run.smithi060.stdout: 416888 8 -rw-r--r-- 1 ubuntu ubuntu 7271 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Ubbd.cc 2024-03-27T21:42:44.306 INFO:teuthology.orchestra.run.smithi060.stdout: 416847 20 -rw-r--r-- 1 ubuntu ubuntu 17128 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Bench.cc 2024-03-27T21:42:44.306 INFO:teuthology.orchestra.run.smithi060.stdout: 416885 16 -rw-r--r-- 1 ubuntu ubuntu 13634 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/Status.cc 2024-03-27T21:42:44.306 INFO:teuthology.orchestra.run.smithi060.stdout: 416862 12 -rw-r--r-- 1 ubuntu ubuntu 9620 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/action/ImageMeta.cc 2024-03-27T21:42:44.306 INFO:teuthology.orchestra.run.smithi060.stdout: 416838 8 -rw-r--r-- 1 ubuntu ubuntu 5415 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd/OptionPrinter.cc 2024-03-27T21:42:44.306 INFO:teuthology.orchestra.run.smithi060.stdout: 416906 4 drwxr-xr-x 11 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror 2024-03-27T21:42:44.306 INFO:teuthology.orchestra.run.smithi060.stdout: 416935 12 -rw-r--r-- 1 ubuntu ubuntu 11836 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/NamespaceReplayer.h 2024-03-27T21:42:44.306 INFO:teuthology.orchestra.run.smithi060.stdout: 416947 4 -rw-r--r-- 1 ubuntu ubuntu 882 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/Threads.cc 2024-03-27T21:42:44.307 INFO:teuthology.orchestra.run.smithi060.stdout: 416942 4 -rw-r--r-- 1 ubuntu ubuntu 459 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/ProgressContext.h 2024-03-27T21:42:44.307 INFO:teuthology.orchestra.run.smithi060.stdout: 416908 4 -rw-r--r-- 1 ubuntu ubuntu 2467 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/CMakeLists.txt 2024-03-27T21:42:44.307 INFO:teuthology.orchestra.run.smithi060.stdout: 416921 8 -rw-r--r-- 1 ubuntu ubuntu 4375 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/InstanceReplayer.h 2024-03-27T21:42:44.307 INFO:teuthology.orchestra.run.smithi060.stdout: 416930 12 -rw-r--r-- 1 ubuntu ubuntu 11172 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/MirrorStatusUpdater.cc 2024-03-27T21:42:44.307 INFO:teuthology.orchestra.run.smithi060.stdout: 416928 20 -rw-r--r-- 1 ubuntu ubuntu 19591 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/Mirror.cc 2024-03-27T21:42:44.307 INFO:teuthology.orchestra.run.smithi060.stdout: 416950 4 -rw-r--r-- 1 ubuntu ubuntu 1856 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/Throttler.h 2024-03-27T21:42:44.307 INFO:teuthology.orchestra.run.smithi060.stdout: 417046 8 -rw-r--r-- 1 ubuntu ubuntu 4454 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/main.cc 2024-03-27T21:42:44.307 INFO:teuthology.orchestra.run.smithi060.stdout: 416931 4 -rw-r--r-- 1 ubuntu ubuntu 3803 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/MirrorStatusUpdater.h 2024-03-27T21:42:44.308 INFO:teuthology.orchestra.run.smithi060.stdout: 416949 8 -rw-r--r-- 1 ubuntu ubuntu 6593 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/Throttler.cc 2024-03-27T21:42:44.308 INFO:teuthology.orchestra.run.smithi060.stdout: 416926 28 -rw-r--r-- 1 ubuntu ubuntu 27856 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/LeaderWatcher.cc 2024-03-27T21:42:44.308 INFO:teuthology.orchestra.run.smithi060.stdout: 416936 4 -rw-r--r-- 1 ubuntu ubuntu 2195 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/PoolMetaCache.cc 2024-03-27T21:42:44.308 INFO:teuthology.orchestra.run.smithi060.stdout: 416914 20 -rw-r--r-- 1 ubuntu ubuntu 18046 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/ImageMap.cc 2024-03-27T21:42:44.308 INFO:teuthology.orchestra.run.smithi060.stdout: 416976 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer 2024-03-27T21:42:44.308 INFO:teuthology.orchestra.run.smithi060.stdout: 416977 16 -rw-r--r-- 1 ubuntu ubuntu 14292 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/BootstrapRequest.cc 2024-03-27T21:42:44.308 INFO:teuthology.orchestra.run.smithi060.stdout: 416978 8 -rw-r--r-- 1 ubuntu ubuntu 6112 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/BootstrapRequest.h 2024-03-27T21:42:44.308 INFO:teuthology.orchestra.run.smithi060.stdout: 416997 4 -rw-r--r-- 1 ubuntu ubuntu 785 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/TimeRollingMean.cc 2024-03-27T21:42:44.309 INFO:teuthology.orchestra.run.smithi060.stdout: 416982 8 -rw-r--r-- 1 ubuntu ubuntu 4941 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/CreateImageRequest.h 2024-03-27T21:42:44.309 INFO:teuthology.orchestra.run.smithi060.stdout: 416986 4 -rw-r--r-- 1 ubuntu ubuntu 1692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/OpenImageRequest.h 2024-03-27T21:42:44.309 INFO:teuthology.orchestra.run.smithi060.stdout: 416993 4 -rw-r--r-- 1 ubuntu ubuntu 937 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/Replayer.h 2024-03-27T21:42:44.309 INFO:teuthology.orchestra.run.smithi060.stdout: 416984 4 -rw-r--r-- 1 ubuntu ubuntu 1873 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/GetMirrorImageIdRequest.h 2024-03-27T21:42:44.309 INFO:teuthology.orchestra.run.smithi060.stdout: 416983 4 -rw-r--r-- 1 ubuntu ubuntu 2312 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/GetMirrorImageIdRequest.cc 2024-03-27T21:42:44.309 INFO:teuthology.orchestra.run.smithi060.stdout: 416994 4 -rw-r--r-- 1 ubuntu ubuntu 505 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/ReplayerListener.h 2024-03-27T21:42:44.309 INFO:teuthology.orchestra.run.smithi060.stdout: 416990 4 -rw-r--r-- 1 ubuntu ubuntu 2893 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/PrepareLocalImageRequest.h 2024-03-27T21:42:44.309 INFO:teuthology.orchestra.run.smithi060.stdout: 416987 8 -rw-r--r-- 1 ubuntu ubuntu 8058 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/OpenLocalImageRequest.cc 2024-03-27T21:42:44.310 INFO:teuthology.orchestra.run.smithi060.stdout: 416980 4 -rw-r--r-- 1 ubuntu ubuntu 1191 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/CloseImageRequest.h 2024-03-27T21:42:44.310 INFO:teuthology.orchestra.run.smithi060.stdout: 416981 16 -rw-r--r-- 1 ubuntu ubuntu 14206 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/CreateImageRequest.cc 2024-03-27T21:42:44.310 INFO:teuthology.orchestra.run.smithi060.stdout: 416979 4 -rw-r--r-- 1 ubuntu ubuntu 1545 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/CloseImageRequest.cc 2024-03-27T21:42:44.310 INFO:teuthology.orchestra.run.smithi060.stdout: 416992 8 -rw-r--r-- 1 ubuntu ubuntu 4596 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/PrepareRemoteImageRequest.h 2024-03-27T21:42:44.310 INFO:teuthology.orchestra.run.smithi060.stdout: 416985 4 -rw-r--r-- 1 ubuntu ubuntu 2150 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/OpenImageRequest.cc 2024-03-27T21:42:44.310 INFO:teuthology.orchestra.run.smithi060.stdout: 416995 4 -rw-r--r-- 1 ubuntu ubuntu 3716 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/StateBuilder.cc 2024-03-27T21:42:44.310 INFO:teuthology.orchestra.run.smithi060.stdout: 416998 4 -rw-r--r-- 1 ubuntu ubuntu 989 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/TimeRollingMean.h 2024-03-27T21:42:44.310 INFO:teuthology.orchestra.run.smithi060.stdout: 416999 4 -rw-r--r-- 1 ubuntu ubuntu 465 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/Types.h 2024-03-27T21:42:44.311 INFO:teuthology.orchestra.run.smithi060.stdout: 417002 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/journal 2024-03-27T21:42:44.311 INFO:teuthology.orchestra.run.smithi060.stdout: 417010 4 -rw-r--r-- 1 ubuntu ubuntu 2204 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/journal/ReplayStatusFormatter.h 2024-03-27T21:42:44.311 INFO:teuthology.orchestra.run.smithi060.stdout: 417009 12 -rw-r--r-- 1 ubuntu ubuntu 8484 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/journal/ReplayStatusFormatter.cc 2024-03-27T21:42:44.311 INFO:teuthology.orchestra.run.smithi060.stdout: 417005 8 -rw-r--r-- 1 ubuntu ubuntu 5959 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/journal/EventPreprocessor.cc 2024-03-27T21:42:44.311 INFO:teuthology.orchestra.run.smithi060.stdout: 417007 12 -rw-r--r-- 1 ubuntu ubuntu 10297 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/journal/PrepareReplayRequest.cc 2024-03-27T21:42:44.311 INFO:teuthology.orchestra.run.smithi060.stdout: 417012 12 -rw-r--r-- 1 ubuntu ubuntu 10900 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/journal/Replayer.h 2024-03-27T21:42:44.311 INFO:teuthology.orchestra.run.smithi060.stdout: 417015 4 -rw-r--r-- 1 ubuntu ubuntu 3318 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/journal/SyncPointHandler.cc 2024-03-27T21:42:44.311 INFO:teuthology.orchestra.run.smithi060.stdout: 417016 4 -rw-r--r-- 1 ubuntu ubuntu 1620 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/journal/SyncPointHandler.h 2024-03-27T21:42:44.312 INFO:teuthology.orchestra.run.smithi060.stdout: 417004 4 -rw-r--r-- 1 ubuntu ubuntu 3223 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/journal/CreateLocalImageRequest.h 2024-03-27T21:42:44.312 INFO:teuthology.orchestra.run.smithi060.stdout: 417013 8 -rw-r--r-- 1 ubuntu ubuntu 4535 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/journal/StateBuilder.cc 2024-03-27T21:42:44.312 INFO:teuthology.orchestra.run.smithi060.stdout: 417006 4 -rw-r--r-- 1 ubuntu ubuntu 3562 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/journal/EventPreprocessor.h 2024-03-27T21:42:44.312 INFO:teuthology.orchestra.run.smithi060.stdout: 417011 40 -rw-r--r-- 1 ubuntu ubuntu 40251 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/journal/Replayer.cc 2024-03-27T21:42:44.312 INFO:teuthology.orchestra.run.smithi060.stdout: 417003 8 -rw-r--r-- 1 ubuntu ubuntu 5096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/journal/CreateLocalImageRequest.cc 2024-03-27T21:42:44.312 INFO:teuthology.orchestra.run.smithi060.stdout: 417008 4 -rw-r--r-- 1 ubuntu ubuntu 2768 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/journal/PrepareReplayRequest.h 2024-03-27T21:42:44.312 INFO:teuthology.orchestra.run.smithi060.stdout: 417014 4 -rw-r--r-- 1 ubuntu ubuntu 2810 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/journal/StateBuilder.h 2024-03-27T21:42:44.312 INFO:teuthology.orchestra.run.smithi060.stdout: 416989 8 -rw-r--r-- 1 ubuntu ubuntu 6027 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/PrepareLocalImageRequest.cc 2024-03-27T21:42:44.312 INFO:teuthology.orchestra.run.smithi060.stdout: 416991 12 -rw-r--r-- 1 ubuntu ubuntu 9702 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/PrepareRemoteImageRequest.cc 2024-03-27T21:42:44.313 INFO:teuthology.orchestra.run.smithi060.stdout: 416996 4 -rw-r--r-- 1 ubuntu ubuntu 3121 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/StateBuilder.h 2024-03-27T21:42:44.313 INFO:teuthology.orchestra.run.smithi060.stdout: 417001 4 -rw-r--r-- 1 ubuntu ubuntu 847 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/Utils.h 2024-03-27T21:42:44.313 INFO:teuthology.orchestra.run.smithi060.stdout: 417000 4 -rw-r--r-- 1 ubuntu ubuntu 1732 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/Utils.cc 2024-03-27T21:42:44.313 INFO:teuthology.orchestra.run.smithi060.stdout: 416988 4 -rw-r--r-- 1 ubuntu ubuntu 2718 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/OpenLocalImageRequest.h 2024-03-27T21:42:44.313 INFO:teuthology.orchestra.run.smithi060.stdout: 417017 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/snapshot 2024-03-27T21:42:44.313 INFO:teuthology.orchestra.run.smithi060.stdout: 417018 20 -rw-r--r-- 1 ubuntu ubuntu 20398 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/snapshot/ApplyImageStateRequest.cc 2024-03-27T21:42:44.313 INFO:teuthology.orchestra.run.smithi060.stdout: 417022 4 -rw-r--r-- 1 ubuntu ubuntu 2039 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/snapshot/PrepareReplayRequest.cc 2024-03-27T21:42:44.313 INFO:teuthology.orchestra.run.smithi060.stdout: 417025 12 -rw-r--r-- 1 ubuntu ubuntu 11210 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/snapshot/Replayer.h 2024-03-27T21:42:44.314 INFO:teuthology.orchestra.run.smithi060.stdout: 417019 4 -rw-r--r-- 1 ubuntu ubuntu 3754 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/snapshot/ApplyImageStateRequest.h 2024-03-27T21:42:44.314 INFO:teuthology.orchestra.run.smithi060.stdout: 417021 4 -rw-r--r-- 1 ubuntu ubuntu 3377 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/snapshot/CreateLocalImageRequest.h 2024-03-27T21:42:44.314 INFO:teuthology.orchestra.run.smithi060.stdout: 417026 4 -rw-r--r-- 1 ubuntu ubuntu 3553 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/snapshot/StateBuilder.cc 2024-03-27T21:42:44.314 INFO:teuthology.orchestra.run.smithi060.stdout: 417024 52 -rw-r--r-- 1 ubuntu ubuntu 53099 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/snapshot/Replayer.cc 2024-03-27T21:42:44.314 INFO:teuthology.orchestra.run.smithi060.stdout: 417020 8 -rw-r--r-- 1 ubuntu ubuntu 6315 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/snapshot/CreateLocalImageRequest.cc 2024-03-27T21:42:44.314 INFO:teuthology.orchestra.run.smithi060.stdout: 417023 4 -rw-r--r-- 1 ubuntu ubuntu 2212 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/snapshot/PrepareReplayRequest.h 2024-03-27T21:42:44.314 INFO:teuthology.orchestra.run.smithi060.stdout: 417027 4 -rw-r--r-- 1 ubuntu ubuntu 2490 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/snapshot/StateBuilder.h 2024-03-27T21:42:44.315 INFO:teuthology.orchestra.run.smithi060.stdout: 417029 4 -rw-r--r-- 1 ubuntu ubuntu 838 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/snapshot/Utils.h 2024-03-27T21:42:44.315 INFO:teuthology.orchestra.run.smithi060.stdout: 417028 4 -rw-r--r-- 1 ubuntu ubuntu 2269 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_replayer/snapshot/Utils.cc 2024-03-27T21:42:44.315 INFO:teuthology.orchestra.run.smithi060.stdout: 416923 12 -rw-r--r-- 1 ubuntu ubuntu 9612 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/InstanceWatcher.h 2024-03-27T21:42:44.315 INFO:teuthology.orchestra.run.smithi060.stdout: 416933 4 -rw-r--r-- 1 ubuntu ubuntu 1157 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/MirrorStatusWatcher.h 2024-03-27T21:42:44.315 INFO:teuthology.orchestra.run.smithi060.stdout: 417051 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/service_daemon 2024-03-27T21:42:44.315 INFO:teuthology.orchestra.run.smithi060.stdout: 417052 4 -rw-r--r-- 1 ubuntu ubuntu 609 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/service_daemon/Types.cc 2024-03-27T21:42:44.315 INFO:teuthology.orchestra.run.smithi060.stdout: 417053 4 -rw-r--r-- 1 ubuntu ubuntu 782 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/service_daemon/Types.h 2024-03-27T21:42:44.315 INFO:teuthology.orchestra.run.smithi060.stdout: 416963 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_map 2024-03-27T21:42:44.316 INFO:teuthology.orchestra.run.smithi060.stdout: 416974 4 -rw-r--r-- 1 ubuntu ubuntu 2686 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_map/UpdateRequest.cc 2024-03-27T21:42:44.316 INFO:teuthology.orchestra.run.smithi060.stdout: 416968 4 -rw-r--r-- 1 ubuntu ubuntu 2559 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_map/SimplePolicy.cc 2024-03-27T21:42:44.316 INFO:teuthology.orchestra.run.smithi060.stdout: 416965 4 -rw-r--r-- 1 ubuntu ubuntu 1758 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_map/LoadRequest.h 2024-03-27T21:42:44.316 INFO:teuthology.orchestra.run.smithi060.stdout: 416972 4 -rw-r--r-- 1 ubuntu ubuntu 3182 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_map/Types.cc 2024-03-27T21:42:44.316 INFO:teuthology.orchestra.run.smithi060.stdout: 416975 4 -rw-r--r-- 1 ubuntu ubuntu 1811 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_map/UpdateRequest.h 2024-03-27T21:42:44.316 INFO:teuthology.orchestra.run.smithi060.stdout: 416971 4 -rw-r--r-- 1 ubuntu ubuntu 2103 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_map/StateTransition.h 2024-03-27T21:42:44.316 INFO:teuthology.orchestra.run.smithi060.stdout: 416966 16 -rw-r--r-- 1 ubuntu ubuntu 13796 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_map/Policy.cc 2024-03-27T21:42:44.316 INFO:teuthology.orchestra.run.smithi060.stdout: 416967 4 -rw-r--r-- 1 ubuntu ubuntu 3826 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_map/Policy.h 2024-03-27T21:42:44.316 INFO:teuthology.orchestra.run.smithi060.stdout: 416973 4 -rw-r--r-- 1 ubuntu ubuntu 2955 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_map/Types.h 2024-03-27T21:42:44.317 INFO:teuthology.orchestra.run.smithi060.stdout: 416969 4 -rw-r--r-- 1 ubuntu ubuntu 1000 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_map/SimplePolicy.h 2024-03-27T21:42:44.317 INFO:teuthology.orchestra.run.smithi060.stdout: 416964 8 -rw-r--r-- 1 ubuntu ubuntu 4296 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_map/LoadRequest.cc 2024-03-27T21:42:44.317 INFO:teuthology.orchestra.run.smithi060.stdout: 416970 4 -rw-r--r-- 1 ubuntu ubuntu 3531 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_map/StateTransition.cc 2024-03-27T21:42:44.317 INFO:teuthology.orchestra.run.smithi060.stdout: 417047 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/pool_watcher 2024-03-27T21:42:44.317 INFO:teuthology.orchestra.run.smithi060.stdout: 417048 4 -rw-r--r-- 1 ubuntu ubuntu 2281 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/pool_watcher/RefreshImagesRequest.cc 2024-03-27T21:42:44.317 INFO:teuthology.orchestra.run.smithi060.stdout: 417050 4 -rw-r--r-- 1 ubuntu ubuntu 656 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/pool_watcher/Types.h 2024-03-27T21:42:44.317 INFO:teuthology.orchestra.run.smithi060.stdout: 417049 4 -rw-r--r-- 1 ubuntu ubuntu 1718 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/pool_watcher/RefreshImagesRequest.h 2024-03-27T21:42:44.317 INFO:teuthology.orchestra.run.smithi060.stdout: 416917 12 -rw-r--r-- 1 ubuntu ubuntu 8195 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/ImageReplayer.h 2024-03-27T21:42:44.318 INFO:teuthology.orchestra.run.smithi060.stdout: 416934 24 -rw-r--r-- 1 ubuntu ubuntu 24303 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/NamespaceReplayer.cc 2024-03-27T21:42:44.318 INFO:teuthology.orchestra.run.smithi060.stdout: 416948 4 -rw-r--r-- 1 ubuntu ubuntu 1059 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/Threads.h 2024-03-27T21:42:44.318 INFO:teuthology.orchestra.run.smithi060.stdout: 416916 36 -rw-r--r-- 1 ubuntu ubuntu 34092 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/ImageReplayer.cc 2024-03-27T21:42:44.318 INFO:teuthology.orchestra.run.smithi060.stdout: 416915 8 -rw-r--r-- 1 ubuntu ubuntu 5441 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/ImageMap.h 2024-03-27T21:42:44.318 INFO:teuthology.orchestra.run.smithi060.stdout: 417041 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/instances 2024-03-27T21:42:44.318 INFO:teuthology.orchestra.run.smithi060.stdout: 417042 4 -rw-r--r-- 1 ubuntu ubuntu 624 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/instances/Types.h 2024-03-27T21:42:44.318 INFO:teuthology.orchestra.run.smithi060.stdout: 416951 4 -rw-r--r-- 1 ubuntu ubuntu 946 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/Types.cc 2024-03-27T21:42:44.318 INFO:teuthology.orchestra.run.smithi060.stdout: 416927 12 -rw-r--r-- 1 ubuntu ubuntu 10829 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/LeaderWatcher.h 2024-03-27T21:42:44.319 INFO:teuthology.orchestra.run.smithi060.stdout: 416939 12 -rw-r--r-- 1 ubuntu ubuntu 8480 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/PoolReplayer.h 2024-03-27T21:42:44.319 INFO:teuthology.orchestra.run.smithi060.stdout: 416911 4 -rw-r--r-- 1 ubuntu ubuntu 1918 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/ClusterWatcher.h 2024-03-27T21:42:44.319 INFO:teuthology.orchestra.run.smithi060.stdout: 416920 16 -rw-r--r-- 1 ubuntu ubuntu 16149 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/InstanceReplayer.cc 2024-03-27T21:42:44.319 INFO:teuthology.orchestra.run.smithi060.stdout: 416944 4 -rw-r--r-- 1 ubuntu ubuntu 3308 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/RemotePoolPoller.h 2024-03-27T21:42:44.319 INFO:teuthology.orchestra.run.smithi060.stdout: 416912 16 -rw-r--r-- 1 ubuntu ubuntu 16376 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/ImageDeleter.cc 2024-03-27T21:42:44.319 INFO:teuthology.orchestra.run.smithi060.stdout: 416937 4 -rw-r--r-- 1 ubuntu ubuntu 1411 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/PoolMetaCache.h 2024-03-27T21:42:44.319 INFO:teuthology.orchestra.run.smithi060.stdout: 416918 16 -rw-r--r-- 1 ubuntu ubuntu 12538 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/ImageSync.cc 2024-03-27T21:42:44.319 INFO:teuthology.orchestra.run.smithi060.stdout: 416952 8 -rw-r--r-- 1 ubuntu ubuntu 4427 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/Types.h 2024-03-27T21:42:44.319 INFO:teuthology.orchestra.run.smithi060.stdout: 416913 8 -rw-r--r-- 1 ubuntu ubuntu 5576 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/ImageDeleter.h 2024-03-27T21:42:44.320 INFO:teuthology.orchestra.run.smithi060.stdout: 417030 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_sync 2024-03-27T21:42:44.320 INFO:teuthology.orchestra.run.smithi060.stdout: 417034 4 -rw-r--r-- 1 ubuntu ubuntu 2121 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_sync/SyncPointPruneRequest.h 2024-03-27T21:42:44.320 INFO:teuthology.orchestra.run.smithi060.stdout: 417032 4 -rw-r--r-- 1 ubuntu ubuntu 2266 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_sync/SyncPointCreateRequest.h 2024-03-27T21:42:44.320 INFO:teuthology.orchestra.run.smithi060.stdout: 417031 8 -rw-r--r-- 1 ubuntu ubuntu 4667 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_sync/SyncPointCreateRequest.cc 2024-03-27T21:42:44.320 INFO:teuthology.orchestra.run.smithi060.stdout: 417033 8 -rw-r--r-- 1 ubuntu ubuntu 5936 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_sync/SyncPointPruneRequest.cc 2024-03-27T21:42:44.320 INFO:teuthology.orchestra.run.smithi060.stdout: 417035 4 -rw-r--r-- 1 ubuntu ubuntu 2017 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_sync/Types.h 2024-03-27T21:42:44.320 INFO:teuthology.orchestra.run.smithi060.stdout: 417037 4 -rw-r--r-- 1 ubuntu ubuntu 358 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_sync/Utils.h 2024-03-27T21:42:44.321 INFO:teuthology.orchestra.run.smithi060.stdout: 417036 4 -rw-r--r-- 1 ubuntu ubuntu 519 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_sync/Utils.cc 2024-03-27T21:42:44.321 INFO:teuthology.orchestra.run.smithi060.stdout: 416943 8 -rw-r--r-- 1 ubuntu ubuntu 7198 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/RemotePoolPoller.cc 2024-03-27T21:42:44.321 INFO:teuthology.orchestra.run.smithi060.stdout: 416925 8 -rw-r--r-- 1 ubuntu ubuntu 4452 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/Instances.h 2024-03-27T21:42:44.321 INFO:teuthology.orchestra.run.smithi060.stdout: 416919 8 -rw-r--r-- 1 ubuntu ubuntu 4287 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/ImageSync.h 2024-03-27T21:42:44.321 INFO:teuthology.orchestra.run.smithi060.stdout: 416929 4 -rw-r--r-- 1 ubuntu ubuntu 2376 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/Mirror.h 2024-03-27T21:42:44.321 INFO:teuthology.orchestra.run.smithi060.stdout: 416932 4 -rw-r--r-- 1 ubuntu ubuntu 2060 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/MirrorStatusWatcher.cc 2024-03-27T21:42:44.321 INFO:teuthology.orchestra.run.smithi060.stdout: 416953 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_deleter 2024-03-27T21:42:44.321 INFO:teuthology.orchestra.run.smithi060.stdout: 416957 4 -rw-r--r-- 1 ubuntu ubuntu 3322 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_deleter/TrashMoveRequest.h 2024-03-27T21:42:44.321 INFO:teuthology.orchestra.run.smithi060.stdout: 416961 4 -rw-r--r-- 1 ubuntu ubuntu 3602 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_deleter/TrashWatcher.h 2024-03-27T21:42:44.322 INFO:teuthology.orchestra.run.smithi060.stdout: 416960 12 -rw-r--r-- 1 ubuntu ubuntu 10581 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_deleter/TrashWatcher.cc 2024-03-27T21:42:44.322 INFO:teuthology.orchestra.run.smithi060.stdout: 416954 8 -rw-r--r-- 1 ubuntu ubuntu 7618 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_deleter/SnapshotPurgeRequest.cc 2024-03-27T21:42:44.322 INFO:teuthology.orchestra.run.smithi060.stdout: 416958 8 -rw-r--r-- 1 ubuntu ubuntu 7343 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_deleter/TrashRemoveRequest.cc 2024-03-27T21:42:44.322 INFO:teuthology.orchestra.run.smithi060.stdout: 416959 4 -rw-r--r-- 1 ubuntu ubuntu 2883 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_deleter/TrashRemoveRequest.h 2024-03-27T21:42:44.322 INFO:teuthology.orchestra.run.smithi060.stdout: 416962 4 -rw-r--r-- 1 ubuntu ubuntu 1177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_deleter/Types.h 2024-03-27T21:42:44.322 INFO:teuthology.orchestra.run.smithi060.stdout: 416955 4 -rw-r--r-- 1 ubuntu ubuntu 2399 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_deleter/SnapshotPurgeRequest.h 2024-03-27T21:42:44.322 INFO:teuthology.orchestra.run.smithi060.stdout: 416956 12 -rw-r--r-- 1 ubuntu ubuntu 11458 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/image_deleter/TrashMoveRequest.cc 2024-03-27T21:42:44.322 INFO:teuthology.orchestra.run.smithi060.stdout: 416907 4 -rw-r--r-- 1 ubuntu ubuntu 609 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/BaseRequest.h 2024-03-27T21:42:44.323 INFO:teuthology.orchestra.run.smithi060.stdout: 417038 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/instance_watcher 2024-03-27T21:42:44.323 INFO:teuthology.orchestra.run.smithi060.stdout: 417039 8 -rw-r--r-- 1 ubuntu ubuntu 6332 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/instance_watcher/Types.cc 2024-03-27T21:42:44.323 INFO:teuthology.orchestra.run.smithi060.stdout: 417040 8 -rw-r--r-- 1 ubuntu ubuntu 5146 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/instance_watcher/Types.h 2024-03-27T21:42:44.323 INFO:teuthology.orchestra.run.smithi060.stdout: 416924 12 -rw-r--r-- 1 ubuntu ubuntu 9765 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/Instances.cc 2024-03-27T21:42:44.323 INFO:teuthology.orchestra.run.smithi060.stdout: 416909 4 -rw-r--r-- 1 ubuntu ubuntu 998 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/CancelableRequest.h 2024-03-27T21:42:44.323 INFO:teuthology.orchestra.run.smithi060.stdout: 416946 4 -rw-r--r-- 1 ubuntu ubuntu 2775 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/ServiceDaemon.h 2024-03-27T21:42:44.323 INFO:teuthology.orchestra.run.smithi060.stdout: 416910 8 -rw-r--r-- 1 ubuntu ubuntu 7708 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/ClusterWatcher.cc 2024-03-27T21:42:44.323 INFO:teuthology.orchestra.run.smithi060.stdout: 417043 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/leader_watcher 2024-03-27T21:42:44.323 INFO:teuthology.orchestra.run.smithi060.stdout: 417044 4 -rw-r--r-- 1 ubuntu ubuntu 3880 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/leader_watcher/Types.cc 2024-03-27T21:42:44.324 INFO:teuthology.orchestra.run.smithi060.stdout: 417045 4 -rw-r--r-- 1 ubuntu ubuntu 2870 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/leader_watcher/Types.h 2024-03-27T21:42:44.324 INFO:teuthology.orchestra.run.smithi060.stdout: 416940 16 -rw-r--r-- 1 ubuntu ubuntu 13360 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/PoolWatcher.cc 2024-03-27T21:42:44.324 INFO:teuthology.orchestra.run.smithi060.stdout: 416945 12 -rw-r--r-- 1 ubuntu ubuntu 9242 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/ServiceDaemon.cc 2024-03-27T21:42:44.324 INFO:teuthology.orchestra.run.smithi060.stdout: 416922 40 -rw-r--r-- 1 ubuntu ubuntu 38105 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/InstanceWatcher.cc 2024-03-27T21:42:44.324 INFO:teuthology.orchestra.run.smithi060.stdout: 416941 8 -rw-r--r-- 1 ubuntu ubuntu 4213 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/PoolWatcher.h 2024-03-27T21:42:44.324 INFO:teuthology.orchestra.run.smithi060.stdout: 416938 36 -rw-r--r-- 1 ubuntu ubuntu 32868 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_mirror/PoolReplayer.cc 2024-03-27T21:42:44.324 INFO:teuthology.orchestra.run.smithi060.stdout: 416791 12 -rwxr-xr-x 1 ubuntu ubuntu 10743 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/crushdiff 2024-03-27T21:42:44.324 INFO:teuthology.orchestra.run.smithi060.stdout: 416705 4 -rw-r--r-- 1 ubuntu ubuntu 3698 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-diff-sorted.cc 2024-03-27T21:42:44.325 INFO:teuthology.orchestra.run.smithi060.stdout: 417074 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_wnbd 2024-03-27T21:42:44.325 INFO:teuthology.orchestra.run.smithi060.stdout: 417076 4 -rw-r--r-- 1 ubuntu ubuntu 2458 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_wnbd/rados_client_cache.cc 2024-03-27T21:42:44.325 INFO:teuthology.orchestra.run.smithi060.stdout: 417075 4 -rw-r--r-- 1 ubuntu ubuntu 479 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_wnbd/CMakeLists.txt 2024-03-27T21:42:44.325 INFO:teuthology.orchestra.run.smithi060.stdout: 417080 4 -rw-r--r-- 1 ubuntu ubuntu 3257 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_wnbd/rbd_mapping_config.cc 2024-03-27T21:42:44.325 INFO:teuthology.orchestra.run.smithi060.stdout: 417087 4 -rw-r--r-- 1 ubuntu ubuntu 2059 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_wnbd/wnbd_wmi.h 2024-03-27T21:42:44.325 INFO:teuthology.orchestra.run.smithi060.stdout: 417079 4 -rw-r--r-- 1 ubuntu ubuntu 2714 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_wnbd/rbd_mapping.h 2024-03-27T21:42:44.325 INFO:teuthology.orchestra.run.smithi060.stdout: 417081 4 -rw-r--r-- 1 ubuntu ubuntu 2207 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_wnbd/rbd_mapping_config.h 2024-03-27T21:42:44.325 INFO:teuthology.orchestra.run.smithi060.stdout: 417086 8 -rw-r--r-- 1 ubuntu ubuntu 6511 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_wnbd/wnbd_wmi.cc 2024-03-27T21:42:44.325 INFO:teuthology.orchestra.run.smithi060.stdout: 417082 48 -rw-r--r-- 1 ubuntu ubuntu 46531 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_wnbd/rbd_wnbd.cc 2024-03-27T21:42:44.326 INFO:teuthology.orchestra.run.smithi060.stdout: 417077 4 -rw-r--r-- 1 ubuntu ubuntu 1000 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_wnbd/rados_client_cache.h 2024-03-27T21:42:44.326 INFO:teuthology.orchestra.run.smithi060.stdout: 417084 16 -rw-r--r-- 1 ubuntu ubuntu 13075 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_wnbd/wnbd_handler.cc 2024-03-27T21:42:44.326 INFO:teuthology.orchestra.run.smithi060.stdout: 417078 8 -rw-r--r-- 1 ubuntu ubuntu 7175 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_wnbd/rbd_mapping.cc 2024-03-27T21:42:44.326 INFO:teuthology.orchestra.run.smithi060.stdout: 417083 4 -rw-r--r-- 1 ubuntu ubuntu 2897 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_wnbd/rbd_wnbd.h 2024-03-27T21:42:44.326 INFO:teuthology.orchestra.run.smithi060.stdout: 417085 8 -rw-r--r-- 1 ubuntu ubuntu 4827 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rbd_wnbd/wnbd_handler.h 2024-03-27T21:42:44.326 INFO:teuthology.orchestra.run.smithi060.stdout: 416710 8 -rwxr-xr-x 1 ubuntu ubuntu 4950 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph-monstore-update-crush.sh 2024-03-27T21:42:44.326 INFO:teuthology.orchestra.run.smithi060.stdout: 416788 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/contrib 2024-03-27T21:42:44.326 INFO:teuthology.orchestra.run.smithi060.stdout: 416790 12 -rwxr-xr-x 1 ubuntu ubuntu 9550 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/contrib/ceph-migrate-bluestore.bash 2024-03-27T21:42:44.327 INFO:teuthology.orchestra.run.smithi060.stdout: 416789 4 -rw-r--r-- 1 ubuntu ubuntu 397 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/contrib/README.rst 2024-03-27T21:42:44.327 INFO:teuthology.orchestra.run.smithi060.stdout: 416713 60 -rw-r--r-- 1 ubuntu ubuntu 58089 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph_dedup_tool.cc 2024-03-27T21:42:44.327 INFO:teuthology.orchestra.run.smithi060.stdout: 416797 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/immutable_object_cache 2024-03-27T21:42:44.327 INFO:teuthology.orchestra.run.smithi060.stdout: 416799 16 -rw-r--r-- 1 ubuntu ubuntu 13620 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/immutable_object_cache/CacheClient.cc 2024-03-27T21:42:44.327 INFO:teuthology.orchestra.run.smithi060.stdout: 416800 4 -rw-r--r-- 1 ubuntu ubuntu 2704 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/immutable_object_cache/CacheClient.h 2024-03-27T21:42:44.327 INFO:teuthology.orchestra.run.smithi060.stdout: 416798 4 -rw-r--r-- 1 ubuntu ubuntu 504 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/immutable_object_cache/CMakeLists.txt 2024-03-27T21:42:44.327 INFO:teuthology.orchestra.run.smithi060.stdout: 416802 4 -rw-r--r-- 1 ubuntu ubuntu 894 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/immutable_object_cache/CacheController.h 2024-03-27T21:42:44.328 INFO:teuthology.orchestra.run.smithi060.stdout: 416816 4 -rw-r--r-- 1 ubuntu ubuntu 2413 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/immutable_object_cache/main.cc 2024-03-27T21:42:44.328 INFO:teuthology.orchestra.run.smithi060.stdout: 416806 4 -rw-r--r-- 1 ubuntu ubuntu 1605 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/immutable_object_cache/CacheSession.h 2024-03-27T21:42:44.328 INFO:teuthology.orchestra.run.smithi060.stdout: 416804 4 -rw-r--r-- 1 ubuntu ubuntu 996 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/immutable_object_cache/CacheServer.h 2024-03-27T21:42:44.328 INFO:teuthology.orchestra.run.smithi060.stdout: 416805 8 -rw-r--r-- 1 ubuntu ubuntu 4126 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/immutable_object_cache/CacheSession.cc 2024-03-27T21:42:44.328 INFO:teuthology.orchestra.run.smithi060.stdout: 416810 8 -rw-r--r-- 1 ubuntu ubuntu 5870 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/immutable_object_cache/SimplePolicy.cc 2024-03-27T21:42:44.328 INFO:teuthology.orchestra.run.smithi060.stdout: 416813 8 -rw-r--r-- 1 ubuntu ubuntu 5478 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/immutable_object_cache/Types.cc 2024-03-27T21:42:44.328 INFO:teuthology.orchestra.run.smithi060.stdout: 416809 4 -rw-r--r-- 1 ubuntu ubuntu 818 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/immutable_object_cache/Policy.h 2024-03-27T21:42:44.328 INFO:teuthology.orchestra.run.smithi060.stdout: 416801 4 -rw-r--r-- 1 ubuntu ubuntu 3854 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/immutable_object_cache/CacheController.cc 2024-03-27T21:42:44.328 INFO:teuthology.orchestra.run.smithi060.stdout: 416803 4 -rw-r--r-- 1 ubuntu ubuntu 2664 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/immutable_object_cache/CacheServer.cc 2024-03-27T21:42:44.329 INFO:teuthology.orchestra.run.smithi060.stdout: 416814 8 -rw-r--r-- 1 ubuntu ubuntu 4398 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/immutable_object_cache/Types.h 2024-03-27T21:42:44.329 INFO:teuthology.orchestra.run.smithi060.stdout: 416808 4 -rw-r--r-- 1 ubuntu ubuntu 2857 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/immutable_object_cache/ObjectCacheStore.h 2024-03-27T21:42:44.329 INFO:teuthology.orchestra.run.smithi060.stdout: 416811 4 -rw-r--r-- 1 ubuntu ubuntu 1735 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/immutable_object_cache/SimplePolicy.h 2024-03-27T21:42:44.329 INFO:teuthology.orchestra.run.smithi060.stdout: 416815 4 -rw-r--r-- 1 ubuntu ubuntu 802 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/immutable_object_cache/Utils.h 2024-03-27T21:42:44.329 INFO:teuthology.orchestra.run.smithi060.stdout: 416807 16 -rw-r--r-- 1 ubuntu ubuntu 14441 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/immutable_object_cache/ObjectCacheStore.cc 2024-03-27T21:42:44.329 INFO:teuthology.orchestra.run.smithi060.stdout: 416812 4 -rw-r--r-- 1 ubuntu ubuntu 968 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/immutable_object_cache/SocketCommon.h 2024-03-27T21:42:44.329 INFO:teuthology.orchestra.run.smithi060.stdout: 416793 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/erasure-code 2024-03-27T21:42:44.329 INFO:teuthology.orchestra.run.smithi060.stdout: 416794 4 -rw-r--r-- 1 ubuntu ubuntu 230 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/erasure-code/CMakeLists.txt 2024-03-27T21:42:44.330 INFO:teuthology.orchestra.run.smithi060.stdout: 416795 12 -rw-r--r-- 1 ubuntu ubuntu 9905 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/erasure-code/ceph-erasure-code-tool.cc 2024-03-27T21:42:44.330 INFO:teuthology.orchestra.run.smithi060.stdout: 416711 12 -rw-r--r-- 1 ubuntu ubuntu 9844 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/ceph_authtool.cc 2024-03-27T21:42:44.330 INFO:teuthology.orchestra.run.smithi060.stdout: 416818 4 -rw-r--r-- 1 ubuntu ubuntu 2363 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/kvstore_tool.h 2024-03-27T21:42:44.330 INFO:teuthology.orchestra.run.smithi060.stdout: 417089 4 -rw-r--r-- 1 ubuntu ubuntu 228 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tools/rebuild_mondb.h 2024-03-27T21:42:44.330 INFO:teuthology.orchestra.run.smithi060.stdout: 415287 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/s3select 2024-03-27T21:42:44.330 INFO:teuthology.orchestra.run.smithi060.stdout: 408976 4 -rw-r--r-- 1 ubuntu ubuntu 54 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/TODO 2024-03-27T21:42:44.330 INFO:teuthology.orchestra.run.smithi060.stdout: 410759 16 -rwxr-xr-x 1 ubuntu ubuntu 14414 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/init-ceph.in 2024-03-27T21:42:44.330 INFO:teuthology.orchestra.run.smithi060.stdout: 409386 28 -rw-r--r-- 1 ubuntu ubuntu 25438 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph_osd.cc 2024-03-27T21:42:44.331 INFO:teuthology.orchestra.run.smithi060.stdout: 411982 4 -rwxr-xr-x 1 ubuntu ubuntu 2246 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mypy.ini 2024-03-27T21:42:44.331 INFO:teuthology.orchestra.run.smithi060.stdout: 414746 8 -rwxr-xr-x 1 ubuntu ubuntu 4481 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbdmap 2024-03-27T21:42:44.331 INFO:teuthology.orchestra.run.smithi060.stdout: 415288 20 -rw-r--r-- 1 ubuntu ubuntu 19213 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/sample.ceph.conf 2024-03-27T21:42:44.331 INFO:teuthology.orchestra.run.smithi060.stdout: 409379 4 -rw-r--r-- 1 ubuntu ubuntu 2397 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph.conf.twoosds 2024-03-27T21:42:44.331 INFO:teuthology.orchestra.run.smithi060.stdout: 410479 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto 2024-03-27T21:42:44.331 INFO:teuthology.orchestra.run.smithi060.stdout: 410482 4 -rw-r--r-- 1 ubuntu ubuntu 1109 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/crypto_plugin.h 2024-03-27T21:42:44.331 INFO:teuthology.orchestra.run.smithi060.stdout: 410480 4 -rw-r--r-- 1 ubuntu ubuntu 252 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/CMakeLists.txt 2024-03-27T21:42:44.331 INFO:teuthology.orchestra.run.smithi060.stdout: 410483 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/isa-l 2024-03-27T21:42:44.331 INFO:teuthology.orchestra.run.smithi060.stdout: 410486 4 -rw-r--r-- 1 ubuntu ubuntu 1770 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/isa-l/isal_crypto_accel.cc 2024-03-27T21:42:44.332 INFO:teuthology.orchestra.run.smithi060.stdout: 410485 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/isa-l/isa-l_crypto 2024-03-27T21:42:44.332 INFO:teuthology.orchestra.run.smithi060.stdout: 410484 4 -rw-r--r-- 1 ubuntu ubuntu 1371 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/isa-l/CMakeLists.txt 2024-03-27T21:42:44.332 INFO:teuthology.orchestra.run.smithi060.stdout: 410488 4 -rw-r--r-- 1 ubuntu ubuntu 966 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/isa-l/isal_crypto_plugin.cc 2024-03-27T21:42:44.332 INFO:teuthology.orchestra.run.smithi060.stdout: 410489 4 -rw-r--r-- 1 ubuntu ubuntu 1345 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/isa-l/isal_crypto_plugin.h 2024-03-27T21:42:44.332 INFO:teuthology.orchestra.run.smithi060.stdout: 410487 4 -rw-r--r-- 1 ubuntu ubuntu 1707 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/isa-l/isal_crypto_accel.h 2024-03-27T21:42:44.332 INFO:teuthology.orchestra.run.smithi060.stdout: 410481 4 -rw-r--r-- 1 ubuntu ubuntu 1869 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/crypto_accel.h 2024-03-27T21:42:44.332 INFO:teuthology.orchestra.run.smithi060.stdout: 410490 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/openssl 2024-03-27T21:42:44.332 INFO:teuthology.orchestra.run.smithi060.stdout: 410491 4 -rw-r--r-- 1 ubuntu ubuntu 564 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/openssl/CMakeLists.txt 2024-03-27T21:42:44.333 INFO:teuthology.orchestra.run.smithi060.stdout: 410492 4 -rw-r--r-- 1 ubuntu ubuntu 3626 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/openssl/openssl_crypto_accel.cc 2024-03-27T21:42:44.333 INFO:teuthology.orchestra.run.smithi060.stdout: 410495 4 -rw-r--r-- 1 ubuntu ubuntu 1023 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/openssl/openssl_crypto_plugin.h 2024-03-27T21:42:44.333 INFO:teuthology.orchestra.run.smithi060.stdout: 410494 4 -rw-r--r-- 1 ubuntu ubuntu 813 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/openssl/openssl_crypto_plugin.cc 2024-03-27T21:42:44.333 INFO:teuthology.orchestra.run.smithi060.stdout: 410493 4 -rw-r--r-- 1 ubuntu ubuntu 1725 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/openssl/openssl_crypto_accel.h 2024-03-27T21:42:44.333 INFO:teuthology.orchestra.run.smithi060.stdout: 410496 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/qat 2024-03-27T21:42:44.333 INFO:teuthology.orchestra.run.smithi060.stdout: 410497 4 -rw-r--r-- 1 ubuntu ubuntu 562 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/qat/CMakeLists.txt 2024-03-27T21:42:44.333 INFO:teuthology.orchestra.run.smithi060.stdout: 410501 4 -rw-r--r-- 1 ubuntu ubuntu 1117 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/qat/qat_crypto_plugin.h 2024-03-27T21:42:44.333 INFO:teuthology.orchestra.run.smithi060.stdout: 410498 4 -rw-r--r-- 1 ubuntu ubuntu 1498 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/qat/qat_crypto_accel.cc 2024-03-27T21:42:44.334 INFO:teuthology.orchestra.run.smithi060.stdout: 410500 4 -rw-r--r-- 1 ubuntu ubuntu 872 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/qat/qat_crypto_plugin.cc 2024-03-27T21:42:44.334 INFO:teuthology.orchestra.run.smithi060.stdout: 410499 4 -rw-r--r-- 1 ubuntu ubuntu 1825 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/qat/qat_crypto_accel.h 2024-03-27T21:42:44.334 INFO:teuthology.orchestra.run.smithi060.stdout: 410503 8 -rw-r--r-- 1 ubuntu ubuntu 6361 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/qat/qcccrypto.h 2024-03-27T21:42:44.334 INFO:teuthology.orchestra.run.smithi060.stdout: 410502 20 -rw-r--r-- 1 ubuntu ubuntu 19881 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crypto/qat/qcccrypto.cc 2024-03-27T21:42:44.334 INFO:teuthology.orchestra.run.smithi060.stdout: 409387 4 -rw-r--r-- 1 ubuntu ubuntu 13 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph_release 2024-03-27T21:42:44.334 INFO:teuthology.orchestra.run.smithi060.stdout: 412181 8 -rw-r--r-- 1 ubuntu ubuntu 7206 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/perf_histogram.h 2024-03-27T21:42:44.334 INFO:teuthology.orchestra.run.smithi060.stdout: 410504 4 -rwxr-xr-x 1 ubuntu ubuntu 1965 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cstart.sh 2024-03-27T21:42:44.334 INFO:teuthology.orchestra.run.smithi060.stdout: 409382 12 -rw-r--r-- 1 ubuntu ubuntu 10720 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph_fuse.cc 2024-03-27T21:42:44.335 INFO:teuthology.orchestra.run.smithi060.stdout: 409089 4 -rw-r--r-- 1 ubuntu ubuntu 1251 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-osd-prestart.sh 2024-03-27T21:42:44.335 INFO:teuthology.orchestra.run.smithi060.stdout: 409093 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume 2024-03-27T21:42:44.335 INFO:teuthology.orchestra.run.smithi060.stdout: 409351 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin 2024-03-27T21:42:44.335 INFO:teuthology.orchestra.run.smithi060.stdout: 409352 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs 2024-03-27T21:42:44.335 INFO:teuthology.orchestra.run.smithi060.stdout: 409353 4 -rw-r--r-- 1 ubuntu ubuntu 90 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/CMakeLists.txt 2024-03-27T21:42:44.335 INFO:teuthology.orchestra.run.smithi060.stdout: 409354 4 -rw-r--r-- 1 ubuntu ubuntu 1513 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/LICENSE 2024-03-27T21:42:44.335 INFO:teuthology.orchestra.run.smithi060.stdout: 409356 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/ceph_volume_zfs 2024-03-27T21:42:44.335 INFO:teuthology.orchestra.run.smithi060.stdout: 409360 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/ceph_volume_zfs/devices 2024-03-27T21:42:44.336 INFO:teuthology.orchestra.run.smithi060.stdout: 409362 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/ceph_volume_zfs/devices/zfs 2024-03-27T21:42:44.336 INFO:teuthology.orchestra.run.smithi060.stdout: 409365 4 -rw-r--r-- 1 ubuntu ubuntu 890 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/ceph_volume_zfs/devices/zfs/main.py 2024-03-27T21:42:44.336 INFO:teuthology.orchestra.run.smithi060.stdout: 409367 4 -rw-r--r-- 1 ubuntu ubuntu 871 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/ceph_volume_zfs/devices/zfs/zap.py 2024-03-27T21:42:44.336 INFO:teuthology.orchestra.run.smithi060.stdout: 409366 4 -rw-r--r-- 1 ubuntu ubuntu 581 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/ceph_volume_zfs/devices/zfs/prepare.py 2024-03-27T21:42:44.336 INFO:teuthology.orchestra.run.smithi060.stdout: 409363 4 -rwxr-xr-x 1 ubuntu ubuntu 77 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/ceph_volume_zfs/devices/zfs/__init__.py 2024-03-27T21:42:44.336 INFO:teuthology.orchestra.run.smithi060.stdout: 409364 4 -rw-r--r-- 1 ubuntu ubuntu 1363 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/ceph_volume_zfs/devices/zfs/inventory.py 2024-03-27T21:42:44.336 INFO:teuthology.orchestra.run.smithi060.stdout: 409361 4 -rwxr-xr-x 1 ubuntu ubuntu 42 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/ceph_volume_zfs/devices/__init__.py 2024-03-27T21:42:44.336 INFO:teuthology.orchestra.run.smithi060.stdout: 409371 8 -rwxr-xr-x 1 ubuntu ubuntu 5044 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/ceph_volume_zfs/zfs.py 2024-03-27T21:42:44.337 INFO:teuthology.orchestra.run.smithi060.stdout: 409368 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/ceph_volume_zfs/util 2024-03-27T21:42:44.337 INFO:teuthology.orchestra.run.smithi060.stdout: 409369 4 -rw-r--r-- 1 ubuntu ubuntu 24 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/ceph_volume_zfs/util/__init__.py 2024-03-27T21:42:44.337 INFO:teuthology.orchestra.run.smithi060.stdout: 409370 4 -rw-r--r-- 1 ubuntu ubuntu 3995 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/ceph_volume_zfs/util/disk.py 2024-03-27T21:42:44.337 INFO:teuthology.orchestra.run.smithi060.stdout: 409357 4 -rwxr-xr-x 1 ubuntu ubuntu 281 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/ceph_volume_zfs/__init__.py 2024-03-27T21:42:44.337 INFO:teuthology.orchestra.run.smithi060.stdout: 409358 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/ceph_volume_zfs/api 2024-03-27T21:42:44.337 INFO:teuthology.orchestra.run.smithi060.stdout: 409359 4 -rw-r--r-- 1 ubuntu ubuntu 67 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/ceph_volume_zfs/api/__init__.py 2024-03-27T21:42:44.337 INFO:teuthology.orchestra.run.smithi060.stdout: 409374 4 -rw-r--r-- 1 ubuntu ubuntu 276 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/tox.ini 2024-03-27T21:42:44.337 INFO:teuthology.orchestra.run.smithi060.stdout: 409355 4 -rw-r--r-- 1 ubuntu ubuntu 174 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/MANIFEST.in 2024-03-27T21:42:44.337 INFO:teuthology.orchestra.run.smithi060.stdout: 409372 4 -rw-r--r-- 1 ubuntu ubuntu 65 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/requirements_dev.txt 2024-03-27T21:42:44.338 INFO:teuthology.orchestra.run.smithi060.stdout: 409373 4 -rw-r--r-- 1 ubuntu ubuntu 1330 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/plugin/zfs/setup.py 2024-03-27T21:42:44.338 INFO:teuthology.orchestra.run.smithi060.stdout: 409376 4 -rw-r--r-- 1 ubuntu ubuntu 258 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/shell_tox.ini 2024-03-27T21:42:44.338 INFO:teuthology.orchestra.run.smithi060.stdout: 409095 4 -rw-r--r-- 1 ubuntu ubuntu 1033 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/CMakeLists.txt 2024-03-27T21:42:44.338 INFO:teuthology.orchestra.run.smithi060.stdout: 409378 4 -rwxr-xr-x 1 ubuntu ubuntu 110 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/tox_install_command.sh 2024-03-27T21:42:44.338 INFO:teuthology.orchestra.run.smithi060.stdout: 409377 4 -rw-r--r-- 1 ubuntu ubuntu 499 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/tox.ini 2024-03-27T21:42:44.338 INFO:teuthology.orchestra.run.smithi060.stdout: 409096 4 -rw-r--r-- 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/MANIFEST.in 2024-03-27T21:42:44.338 INFO:teuthology.orchestra.run.smithi060.stdout: 409097 4 drwxr-xr-x 10 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume 2024-03-27T21:42:44.338 INFO:teuthology.orchestra.run.smithi060.stdout: 409150 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests 2024-03-27T21:42:44.339 INFO:teuthology.orchestra.run.smithi060.stdout: 409331 8 -rw-r--r-- 1 ubuntu ubuntu 4519 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/test_terminal.py 2024-03-27T21:42:44.339 INFO:teuthology.orchestra.run.smithi060.stdout: 409154 12 -rw-r--r-- 1 ubuntu ubuntu 10342 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/conftest.py 2024-03-27T21:42:44.339 INFO:teuthology.orchestra.run.smithi060.stdout: 409155 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices 2024-03-27T21:42:44.339 INFO:teuthology.orchestra.run.smithi060.stdout: 409157 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/lvm 2024-03-27T21:42:44.339 INFO:teuthology.orchestra.run.smithi060.stdout: 409165 116 -rw-r--r-- 1 ubuntu ubuntu 116248 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/lvm/test_migrate.py 2024-03-27T21:42:44.339 INFO:teuthology.orchestra.run.smithi060.stdout: 409162 4 -rw-r--r-- 1 ubuntu ubuntu 595 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/lvm/test_create.py 2024-03-27T21:42:44.339 INFO:teuthology.orchestra.run.smithi060.stdout: 409166 8 -rw-r--r-- 1 ubuntu ubuntu 6360 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/lvm/test_prepare.py 2024-03-27T21:42:44.339 INFO:teuthology.orchestra.run.smithi060.stdout: 409161 4 -rw-r--r-- 1 ubuntu ubuntu 192 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/lvm/test_common.py 2024-03-27T21:42:44.340 INFO:teuthology.orchestra.run.smithi060.stdout: 409160 16 -rw-r--r-- 1 ubuntu ubuntu 16322 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/lvm/test_batch.py 2024-03-27T21:42:44.340 INFO:teuthology.orchestra.run.smithi060.stdout: 409168 12 -rw-r--r-- 1 ubuntu ubuntu 10275 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/lvm/test_zap.py 2024-03-27T21:42:44.340 INFO:teuthology.orchestra.run.smithi060.stdout: 409164 16 -rw-r--r-- 1 ubuntu ubuntu 15261 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/lvm/test_listing.py 2024-03-27T21:42:44.340 INFO:teuthology.orchestra.run.smithi060.stdout: 409167 4 -rw-r--r-- 1 ubuntu ubuntu 1543 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/lvm/test_trigger.py 2024-03-27T21:42:44.340 INFO:teuthology.orchestra.run.smithi060.stdout: 409158 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/lvm/__init__.py 2024-03-27T21:42:44.340 INFO:teuthology.orchestra.run.smithi060.stdout: 409163 4 -rw-r--r-- 1 ubuntu ubuntu 2512 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/lvm/test_deactivate.py 2024-03-27T21:42:44.340 INFO:teuthology.orchestra.run.smithi060.stdout: 409159 16 -rw-r--r-- 1 ubuntu ubuntu 13393 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/lvm/test_activate.py 2024-03-27T21:42:44.340 INFO:teuthology.orchestra.run.smithi060.stdout: 409177 4 -rw-r--r-- 1 ubuntu ubuntu 1350 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/test_zap.py 2024-03-27T21:42:44.341 INFO:teuthology.orchestra.run.smithi060.stdout: 409173 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/simple 2024-03-27T21:42:44.341 INFO:teuthology.orchestra.run.smithi060.stdout: 409176 4 -rw-r--r-- 1 ubuntu ubuntu 1546 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/simple/test_trigger.py 2024-03-27T21:42:44.341 INFO:teuthology.orchestra.run.smithi060.stdout: 409174 8 -rw-r--r-- 1 ubuntu ubuntu 7905 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/simple/test_activate.py 2024-03-27T21:42:44.341 INFO:teuthology.orchestra.run.smithi060.stdout: 409175 4 -rw-r--r-- 1 ubuntu ubuntu 2691 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/simple/test_scan.py 2024-03-27T21:42:44.341 INFO:teuthology.orchestra.run.smithi060.stdout: 409156 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/__init__.py 2024-03-27T21:42:44.341 INFO:teuthology.orchestra.run.smithi060.stdout: 409169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/raw 2024-03-27T21:42:44.341 INFO:teuthology.orchestra.run.smithi060.stdout: 409172 8 -rw-r--r-- 1 ubuntu ubuntu 4558 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/raw/test_prepare.py 2024-03-27T21:42:44.341 INFO:teuthology.orchestra.run.smithi060.stdout: 409171 12 -rw-r--r-- 1 ubuntu ubuntu 10354 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/raw/test_list.py 2024-03-27T21:42:44.341 INFO:teuthology.orchestra.run.smithi060.stdout: 409170 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/devices/raw/__init__.py 2024-03-27T21:42:44.342 INFO:teuthology.orchestra.run.smithi060.stdout: 409178 4 drwxr-xr-x 9 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional 2024-03-27T21:42:44.342 INFO:teuthology.orchestra.run.smithi060.stdout: 409274 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/playbooks 2024-03-27T21:42:44.342 INFO:teuthology.orchestra.run.smithi060.stdout: 409275 8 -rw-r--r-- 1 ubuntu ubuntu 4353 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/playbooks/deploy.yml 2024-03-27T21:42:44.342 INFO:teuthology.orchestra.run.smithi060.stdout: 409180 4 -rw-r--r-- 1 ubuntu ubuntu 1061 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/README.md 2024-03-27T21:42:44.342 INFO:teuthology.orchestra.run.smithi060.stdout: 409316 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/tests 2024-03-27T21:42:44.342 INFO:teuthology.orchestra.run.smithi060.stdout: 409318 4 -rw-r--r-- 1 ubuntu ubuntu 3983 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/tests/conftest.py 2024-03-27T21:42:44.342 INFO:teuthology.orchestra.run.smithi060.stdout: 409319 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/tests/osd 2024-03-27T21:42:44.343 INFO:teuthology.orchestra.run.smithi060.stdout: 409321 4 -rw-r--r-- 1 ubuntu ubuntu 2609 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/tests/osd/test_osds.py 2024-03-27T21:42:44.343 INFO:teuthology.orchestra.run.smithi060.stdout: 409320 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/tests/osd/__init__.py 2024-03-27T21:42:44.343 INFO:teuthology.orchestra.run.smithi060.stdout: 409317 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/tests/__init__.py 2024-03-27T21:42:44.343 INFO:teuthology.orchestra.run.smithi060.stdout: 409251 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm 2024-03-27T21:42:44.343 INFO:teuthology.orchestra.run.smithi060.stdout: 409270 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/playbooks 2024-03-27T21:42:44.343 INFO:teuthology.orchestra.run.smithi060.stdout: 409271 4 -rw-r--r-- 1 ubuntu ubuntu 487 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/playbooks/setup_partitions.yml 2024-03-27T21:42:44.343 INFO:teuthology.orchestra.run.smithi060.stdout: 409272 8 -rw-r--r-- 1 ubuntu ubuntu 4367 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/playbooks/test_bluestore.yml 2024-03-27T21:42:44.343 INFO:teuthology.orchestra.run.smithi060.stdout: 409252 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/centos8 2024-03-27T21:42:44.344 INFO:teuthology.orchestra.run.smithi060.stdout: 409253 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/centos8/bluestore 2024-03-27T21:42:44.344 INFO:teuthology.orchestra.run.smithi060.stdout: 409254 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/centos8/bluestore/create 2024-03-27T21:42:44.344 INFO:teuthology.orchestra.run.smithi060.stdout: 409260 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/centos8/bluestore/create/test.yml -> ../../../playbooks/test_bluestore.yml 2024-03-27T21:42:44.344 INFO:teuthology.orchestra.run.smithi060.stdout: 409261 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/centos8/bluestore/create/vagrant_variables.yml -> ../../../../vagrant_variables.yml 2024-03-27T21:42:44.344 INFO:teuthology.orchestra.run.smithi060.stdout: 409256 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/centos8/bluestore/create/group_vars 2024-03-27T21:42:44.344 INFO:teuthology.orchestra.run.smithi060.stdout: 409257 0 lrwxrwxrwx 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/centos8/bluestore/create/group_vars/all -> ../../../../../group_vars/bluestore_lvm 2024-03-27T21:42:44.344 INFO:teuthology.orchestra.run.smithi060.stdout: 409259 0 lrwxrwxrwx 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/centos8/bluestore/create/setup.yml -> ../../../playbooks/setup_partitions.yml 2024-03-27T21:42:44.344 INFO:teuthology.orchestra.run.smithi060.stdout: 409255 0 lrwxrwxrwx 1 ubuntu ubuntu 23 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/centos8/bluestore/create/Vagrantfile -> ../../../../Vagrantfile 2024-03-27T21:42:44.345 INFO:teuthology.orchestra.run.smithi060.stdout: 409258 4 -rw-r--r-- 1 ubuntu ubuntu 38 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/centos8/bluestore/create/hosts 2024-03-27T21:42:44.345 INFO:teuthology.orchestra.run.smithi060.stdout: 409262 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/centos8/bluestore/dmcrypt 2024-03-27T21:42:44.345 INFO:teuthology.orchestra.run.smithi060.stdout: 409268 4 -rw-r--r-- 1 ubuntu ubuntu 3158 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/centos8/bluestore/dmcrypt/test.yml 2024-03-27T21:42:44.345 INFO:teuthology.orchestra.run.smithi060.stdout: 409269 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/centos8/bluestore/dmcrypt/vagrant_variables.yml -> ../../../../vagrant_variables.yml 2024-03-27T21:42:44.345 INFO:teuthology.orchestra.run.smithi060.stdout: 409264 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/centos8/bluestore/dmcrypt/group_vars 2024-03-27T21:42:44.345 INFO:teuthology.orchestra.run.smithi060.stdout: 409265 0 lrwxrwxrwx 1 ubuntu ubuntu 47 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/centos8/bluestore/dmcrypt/group_vars/all -> ../../../../../group_vars/bluestore_lvm_dmcrypt 2024-03-27T21:42:44.345 INFO:teuthology.orchestra.run.smithi060.stdout: 409267 0 lrwxrwxrwx 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/centos8/bluestore/dmcrypt/setup.yml -> ../../../playbooks/setup_partitions.yml 2024-03-27T21:42:44.345 INFO:teuthology.orchestra.run.smithi060.stdout: 409263 0 lrwxrwxrwx 1 ubuntu ubuntu 23 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/centos8/bluestore/dmcrypt/Vagrantfile -> ../../../../Vagrantfile 2024-03-27T21:42:44.345 INFO:teuthology.orchestra.run.smithi060.stdout: 409266 4 -rw-r--r-- 1 ubuntu ubuntu 38 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/centos8/bluestore/dmcrypt/hosts 2024-03-27T21:42:44.346 INFO:teuthology.orchestra.run.smithi060.stdout: 409273 4 -rw-r--r-- 1 ubuntu ubuntu 3262 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/lvm/tox.ini 2024-03-27T21:42:44.346 INFO:teuthology.orchestra.run.smithi060.stdout: 409322 4 -rw-r--r-- 1 ubuntu ubuntu 2346 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/vagrant_variables.yml 2024-03-27T21:42:44.346 INFO:teuthology.orchestra.run.smithi060.stdout: 409182 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch 2024-03-27T21:42:44.346 INFO:teuthology.orchestra.run.smithi060.stdout: 409239 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/playbooks 2024-03-27T21:42:44.346 INFO:teuthology.orchestra.run.smithi060.stdout: 409243 4 -rw-r--r-- 1 ubuntu ubuntu 2350 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/playbooks/test_explicit.yml 2024-03-27T21:42:44.346 INFO:teuthology.orchestra.run.smithi060.stdout: 409242 4 -rw-r--r-- 1 ubuntu ubuntu 2145 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/playbooks/test.yml 2024-03-27T21:42:44.347 INFO:teuthology.orchestra.run.smithi060.stdout: 409241 8 -rw-r--r-- 1 ubuntu ubuntu 5678 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/playbooks/setup_mixed_type.yml 2024-03-27T21:42:44.347 INFO:teuthology.orchestra.run.smithi060.stdout: 409244 4 -rw-r--r-- 1 ubuntu ubuntu 755 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/playbooks/test_zap.yml 2024-03-27T21:42:44.347 INFO:teuthology.orchestra.run.smithi060.stdout: 409240 4 -rw-r--r-- 1 ubuntu ubuntu 277 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/playbooks/noop.yml 2024-03-27T21:42:44.347 INFO:teuthology.orchestra.run.smithi060.stdout: 409183 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8 2024-03-27T21:42:44.347 INFO:teuthology.orchestra.run.smithi060.stdout: 409184 4 drwxr-xr-x 8 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore 2024-03-27T21:42:44.347 INFO:teuthology.orchestra.run.smithi060.stdout: 409212 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type 2024-03-27T21:42:44.347 INFO:teuthology.orchestra.run.smithi060.stdout: 409218 0 lrwxrwxrwx 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type/test.yml -> ../../../playbooks/test.yml 2024-03-27T21:42:44.347 INFO:teuthology.orchestra.run.smithi060.stdout: 409220 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type/vagrant_variables.yml -> ../../../../vagrant_variables.yml 2024-03-27T21:42:44.347 INFO:teuthology.orchestra.run.smithi060.stdout: 409214 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type/group_vars 2024-03-27T21:42:44.348 INFO:teuthology.orchestra.run.smithi060.stdout: 409215 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type/group_vars/all -> ../../../../../group_vars/bluestore 2024-03-27T21:42:44.348 INFO:teuthology.orchestra.run.smithi060.stdout: 409217 0 lrwxrwxrwx 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type/setup.yml -> ../../../playbooks/setup_mixed_type.yml 2024-03-27T21:42:44.348 INFO:teuthology.orchestra.run.smithi060.stdout: 409213 0 lrwxrwxrwx 1 ubuntu ubuntu 23 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type/Vagrantfile -> ../../../../Vagrantfile 2024-03-27T21:42:44.348 INFO:teuthology.orchestra.run.smithi060.stdout: 409219 0 lrwxrwxrwx 1 ubuntu ubuntu 31 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type/test_zap.yml -> ../../../playbooks/test_zap.yml 2024-03-27T21:42:44.348 INFO:teuthology.orchestra.run.smithi060.stdout: 409216 4 -rw-r--r-- 1 ubuntu ubuntu 38 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type/hosts 2024-03-27T21:42:44.348 INFO:teuthology.orchestra.run.smithi060.stdout: 409221 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/single-type-dmcrypt 2024-03-27T21:42:44.348 INFO:teuthology.orchestra.run.smithi060.stdout: 409227 0 lrwxrwxrwx 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/single-type-dmcrypt/test.yml -> ../../../playbooks/test.yml 2024-03-27T21:42:44.348 INFO:teuthology.orchestra.run.smithi060.stdout: 409229 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/single-type-dmcrypt/vagrant_variables.yml -> ../../../../vagrant_variables.yml 2024-03-27T21:42:44.349 INFO:teuthology.orchestra.run.smithi060.stdout: 409223 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/single-type-dmcrypt/group_vars 2024-03-27T21:42:44.349 INFO:teuthology.orchestra.run.smithi060.stdout: 409224 0 lrwxrwxrwx 1 ubuntu ubuntu 42 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/single-type-dmcrypt/group_vars/all -> ../../../../../group_vars/bluestore_single 2024-03-27T21:42:44.349 INFO:teuthology.orchestra.run.smithi060.stdout: 409226 0 lrwxrwxrwx 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/single-type-dmcrypt/setup.yml -> ../../../playbooks/noop.yml 2024-03-27T21:42:44.349 INFO:teuthology.orchestra.run.smithi060.stdout: 409222 0 lrwxrwxrwx 1 ubuntu ubuntu 23 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/single-type-dmcrypt/Vagrantfile -> ../../../../Vagrantfile 2024-03-27T21:42:44.349 INFO:teuthology.orchestra.run.smithi060.stdout: 409228 0 lrwxrwxrwx 1 ubuntu ubuntu 31 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/single-type-dmcrypt/test_zap.yml -> ../../../playbooks/test_zap.yml 2024-03-27T21:42:44.349 INFO:teuthology.orchestra.run.smithi060.stdout: 409225 4 -rw-r--r-- 1 ubuntu ubuntu 38 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/single-type-dmcrypt/hosts 2024-03-27T21:42:44.349 INFO:teuthology.orchestra.run.smithi060.stdout: 409194 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-dmcrypt 2024-03-27T21:42:44.349 INFO:teuthology.orchestra.run.smithi060.stdout: 409200 0 lrwxrwxrwx 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-dmcrypt/test.yml -> ../../../playbooks/test.yml 2024-03-27T21:42:44.350 INFO:teuthology.orchestra.run.smithi060.stdout: 409202 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-dmcrypt/vagrant_variables.yml -> ../../../../vagrant_variables.yml 2024-03-27T21:42:44.350 INFO:teuthology.orchestra.run.smithi060.stdout: 409196 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-dmcrypt/group_vars 2024-03-27T21:42:44.350 INFO:teuthology.orchestra.run.smithi060.stdout: 409197 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-dmcrypt/group_vars/all -> ../../../../../group_vars/bluestore 2024-03-27T21:42:44.350 INFO:teuthology.orchestra.run.smithi060.stdout: 409199 0 lrwxrwxrwx 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-dmcrypt/setup.yml -> ../../../playbooks/setup_mixed_type.yml 2024-03-27T21:42:44.350 INFO:teuthology.orchestra.run.smithi060.stdout: 409195 0 lrwxrwxrwx 1 ubuntu ubuntu 23 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-dmcrypt/Vagrantfile -> ../../../../Vagrantfile 2024-03-27T21:42:44.350 INFO:teuthology.orchestra.run.smithi060.stdout: 409201 0 lrwxrwxrwx 1 ubuntu ubuntu 31 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-dmcrypt/test_zap.yml -> ../../../playbooks/test_zap.yml 2024-03-27T21:42:44.350 INFO:teuthology.orchestra.run.smithi060.stdout: 409198 4 -rw-r--r-- 1 ubuntu ubuntu 38 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-dmcrypt/hosts 2024-03-27T21:42:44.350 INFO:teuthology.orchestra.run.smithi060.stdout: 409203 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-explicit 2024-03-27T21:42:44.351 INFO:teuthology.orchestra.run.smithi060.stdout: 409209 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-explicit/test.yml -> ../../../playbooks/test_explicit.yml 2024-03-27T21:42:44.351 INFO:teuthology.orchestra.run.smithi060.stdout: 409211 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-explicit/vagrant_variables.yml -> ../../../../vagrant_variables.yml 2024-03-27T21:42:44.351 INFO:teuthology.orchestra.run.smithi060.stdout: 409205 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-explicit/group_vars 2024-03-27T21:42:44.351 INFO:teuthology.orchestra.run.smithi060.stdout: 409206 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-explicit/group_vars/all -> ../../../../../group_vars/bluestore 2024-03-27T21:42:44.351 INFO:teuthology.orchestra.run.smithi060.stdout: 409208 0 lrwxrwxrwx 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-explicit/setup.yml -> ../../../playbooks/setup_mixed_type.yml 2024-03-27T21:42:44.351 INFO:teuthology.orchestra.run.smithi060.stdout: 409204 0 lrwxrwxrwx 1 ubuntu ubuntu 23 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-explicit/Vagrantfile -> ../../../../Vagrantfile 2024-03-27T21:42:44.351 INFO:teuthology.orchestra.run.smithi060.stdout: 409210 0 lrwxrwxrwx 1 ubuntu ubuntu 31 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-explicit/test_zap.yml -> ../../../playbooks/test_zap.yml 2024-03-27T21:42:44.352 INFO:teuthology.orchestra.run.smithi060.stdout: 409207 4 -rw-r--r-- 1 ubuntu ubuntu 38 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-explicit/hosts 2024-03-27T21:42:44.352 INFO:teuthology.orchestra.run.smithi060.stdout: 409230 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/single-type 2024-03-27T21:42:44.352 INFO:teuthology.orchestra.run.smithi060.stdout: 409236 0 lrwxrwxrwx 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/single-type/test.yml -> ../../../playbooks/test.yml 2024-03-27T21:42:44.352 INFO:teuthology.orchestra.run.smithi060.stdout: 409238 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/single-type/vagrant_variables.yml -> ../../../../vagrant_variables.yml 2024-03-27T21:42:44.352 INFO:teuthology.orchestra.run.smithi060.stdout: 409232 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/single-type/group_vars 2024-03-27T21:42:44.352 INFO:teuthology.orchestra.run.smithi060.stdout: 409233 0 lrwxrwxrwx 1 ubuntu ubuntu 42 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/single-type/group_vars/all -> ../../../../../group_vars/bluestore_single 2024-03-27T21:42:44.352 INFO:teuthology.orchestra.run.smithi060.stdout: 409235 0 lrwxrwxrwx 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/single-type/setup.yml -> ../../../playbooks/noop.yml 2024-03-27T21:42:44.352 INFO:teuthology.orchestra.run.smithi060.stdout: 409231 0 lrwxrwxrwx 1 ubuntu ubuntu 23 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/single-type/Vagrantfile -> ../../../../Vagrantfile 2024-03-27T21:42:44.352 INFO:teuthology.orchestra.run.smithi060.stdout: 409237 0 lrwxrwxrwx 1 ubuntu ubuntu 31 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/single-type/test_zap.yml -> ../../../playbooks/test_zap.yml 2024-03-27T21:42:44.353 INFO:teuthology.orchestra.run.smithi060.stdout: 409234 4 -rw-r--r-- 1 ubuntu ubuntu 38 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/single-type/hosts 2024-03-27T21:42:44.353 INFO:teuthology.orchestra.run.smithi060.stdout: 409185 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-dmcrypt-explicit 2024-03-27T21:42:44.353 INFO:teuthology.orchestra.run.smithi060.stdout: 409191 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-dmcrypt-explicit/test.yml -> ../../../playbooks/test_explicit.yml 2024-03-27T21:42:44.353 INFO:teuthology.orchestra.run.smithi060.stdout: 409193 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-dmcrypt-explicit/vagrant_variables.yml -> ../../../../vagrant_variables.yml 2024-03-27T21:42:44.353 INFO:teuthology.orchestra.run.smithi060.stdout: 409187 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-dmcrypt-explicit/group_vars 2024-03-27T21:42:44.353 INFO:teuthology.orchestra.run.smithi060.stdout: 409188 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-dmcrypt-explicit/group_vars/all -> ../../../../../group_vars/bluestore 2024-03-27T21:42:44.353 INFO:teuthology.orchestra.run.smithi060.stdout: 409190 0 lrwxrwxrwx 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-dmcrypt-explicit/setup.yml -> ../../../playbooks/setup_mixed_type.yml 2024-03-27T21:42:44.353 INFO:teuthology.orchestra.run.smithi060.stdout: 409186 0 lrwxrwxrwx 1 ubuntu ubuntu 23 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-dmcrypt-explicit/Vagrantfile -> ../../../../Vagrantfile 2024-03-27T21:42:44.354 INFO:teuthology.orchestra.run.smithi060.stdout: 409192 0 lrwxrwxrwx 1 ubuntu ubuntu 31 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-dmcrypt-explicit/test_zap.yml -> ../../../playbooks/test_zap.yml 2024-03-27T21:42:44.354 INFO:teuthology.orchestra.run.smithi060.stdout: 409189 4 -rw-r--r-- 1 ubuntu ubuntu 38 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/centos8/bluestore/mixed-type-dmcrypt-explicit/hosts 2024-03-27T21:42:44.354 INFO:teuthology.orchestra.run.smithi060.stdout: 409245 4 -rw-r--r-- 1 ubuntu ubuntu 3499 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/batch/tox.ini 2024-03-27T21:42:44.354 INFO:teuthology.orchestra.run.smithi060.stdout: 409246 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/group_vars 2024-03-27T21:42:44.354 INFO:teuthology.orchestra.run.smithi060.stdout: 409247 4 -rw-r--r-- 1 ubuntu ubuntu 691 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/group_vars/bluestore 2024-03-27T21:42:44.354 INFO:teuthology.orchestra.run.smithi060.stdout: 409249 4 -rw-r--r-- 1 ubuntu ubuntu 732 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/group_vars/bluestore_lvm_dmcrypt 2024-03-27T21:42:44.354 INFO:teuthology.orchestra.run.smithi060.stdout: 409250 4 -rw-r--r-- 1 ubuntu ubuntu 604 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/group_vars/bluestore_single 2024-03-27T21:42:44.354 INFO:teuthology.orchestra.run.smithi060.stdout: 409248 4 -rw-r--r-- 1 ubuntu ubuntu 681 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/group_vars/bluestore_lvm 2024-03-27T21:42:44.355 INFO:teuthology.orchestra.run.smithi060.stdout: 409276 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/scripts 2024-03-27T21:42:44.355 INFO:teuthology.orchestra.run.smithi060.stdout: 409280 4 -rw-r--r-- 1 ubuntu ubuntu 593 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/scripts/vagrant_reload.sh 2024-03-27T21:42:44.355 INFO:teuthology.orchestra.run.smithi060.stdout: 409277 4 -rw-r--r-- 1 ubuntu ubuntu 285 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/scripts/generate_ssh_config.sh 2024-03-27T21:42:44.355 INFO:teuthology.orchestra.run.smithi060.stdout: 409281 4 -rw-r--r-- 1 ubuntu ubuntu 200 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/scripts/vagrant_up.sh 2024-03-27T21:42:44.355 INFO:teuthology.orchestra.run.smithi060.stdout: 409279 4 -rw-r--r-- 1 ubuntu ubuntu 911 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/scripts/test_unicode.sh 2024-03-27T21:42:44.355 INFO:teuthology.orchestra.run.smithi060.stdout: 409278 4 -rw-r--r-- 1 ubuntu ubuntu 99 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/scripts/output.py 2024-03-27T21:42:44.355 INFO:teuthology.orchestra.run.smithi060.stdout: 409282 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple 2024-03-27T21:42:44.355 INFO:teuthology.orchestra.run.smithi060.stdout: 409283 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7 2024-03-27T21:42:44.356 INFO:teuthology.orchestra.run.smithi060.stdout: 409284 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore 2024-03-27T21:42:44.356 INFO:teuthology.orchestra.run.smithi060.stdout: 409295 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/dmcrypt-luks 2024-03-27T21:42:44.356 INFO:teuthology.orchestra.run.smithi060.stdout: 409303 4 -rw-r--r-- 1 ubuntu ubuntu 352 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/dmcrypt-luks/test.yml 2024-03-27T21:42:44.356 INFO:teuthology.orchestra.run.smithi060.stdout: 409304 4 -rw-r--r-- 1 ubuntu ubuntu 2607 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/dmcrypt-luks/vagrant_variables.yml 2024-03-27T21:42:44.356 INFO:teuthology.orchestra.run.smithi060.stdout: 409297 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/dmcrypt-luks/group_vars 2024-03-27T21:42:44.356 INFO:teuthology.orchestra.run.smithi060.stdout: 409298 4 -rw-r--r-- 1 ubuntu ubuntu 488 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/dmcrypt-luks/group_vars/all 2024-03-27T21:42:44.356 INFO:teuthology.orchestra.run.smithi060.stdout: 409296 0 lrwxrwxrwx 1 ubuntu ubuntu 23 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/dmcrypt-luks/Vagrantfile -> ../../../../Vagrantfile 2024-03-27T21:42:44.356 INFO:teuthology.orchestra.run.smithi060.stdout: 409299 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/dmcrypt-luks/host_vars 2024-03-27T21:42:44.357 INFO:teuthology.orchestra.run.smithi060.stdout: 409301 4 -rw-r--r-- 1 ubuntu ubuntu 71 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/dmcrypt-luks/host_vars/osd1.yml 2024-03-27T21:42:44.357 INFO:teuthology.orchestra.run.smithi060.stdout: 409300 4 -rw-r--r-- 1 ubuntu ubuntu 94 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/dmcrypt-luks/host_vars/osd0.yml 2024-03-27T21:42:44.357 INFO:teuthology.orchestra.run.smithi060.stdout: 409302 4 -rw-r--r-- 1 ubuntu ubuntu 67 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/dmcrypt-luks/hosts 2024-03-27T21:42:44.357 INFO:teuthology.orchestra.run.smithi060.stdout: 409305 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/dmcrypt-plain 2024-03-27T21:42:44.357 INFO:teuthology.orchestra.run.smithi060.stdout: 409313 4 -rw-r--r-- 1 ubuntu ubuntu 762 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/dmcrypt-plain/test.yml 2024-03-27T21:42:44.357 INFO:teuthology.orchestra.run.smithi060.stdout: 409314 4 -rw-r--r-- 1 ubuntu ubuntu 2607 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/dmcrypt-plain/vagrant_variables.yml 2024-03-27T21:42:44.357 INFO:teuthology.orchestra.run.smithi060.stdout: 409307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/dmcrypt-plain/group_vars 2024-03-27T21:42:44.357 INFO:teuthology.orchestra.run.smithi060.stdout: 409308 4 -rw-r--r-- 1 ubuntu ubuntu 489 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/dmcrypt-plain/group_vars/all 2024-03-27T21:42:44.357 INFO:teuthology.orchestra.run.smithi060.stdout: 409306 0 lrwxrwxrwx 1 ubuntu ubuntu 23 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/dmcrypt-plain/Vagrantfile -> ../../../../Vagrantfile 2024-03-27T21:42:44.358 INFO:teuthology.orchestra.run.smithi060.stdout: 409309 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/dmcrypt-plain/host_vars 2024-03-27T21:42:44.358 INFO:teuthology.orchestra.run.smithi060.stdout: 409311 4 -rw-r--r-- 1 ubuntu ubuntu 71 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/dmcrypt-plain/host_vars/osd1.yml 2024-03-27T21:42:44.358 INFO:teuthology.orchestra.run.smithi060.stdout: 409310 4 -rw-r--r-- 1 ubuntu ubuntu 94 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/dmcrypt-plain/host_vars/osd0.yml 2024-03-27T21:42:44.358 INFO:teuthology.orchestra.run.smithi060.stdout: 409312 4 -rw-r--r-- 1 ubuntu ubuntu 67 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/dmcrypt-plain/hosts 2024-03-27T21:42:44.358 INFO:teuthology.orchestra.run.smithi060.stdout: 409285 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/activate 2024-03-27T21:42:44.358 INFO:teuthology.orchestra.run.smithi060.stdout: 409293 4 -rw-r--r-- 1 ubuntu ubuntu 762 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/activate/test.yml 2024-03-27T21:42:44.358 INFO:teuthology.orchestra.run.smithi060.stdout: 409294 4 -rw-r--r-- 1 ubuntu ubuntu 2607 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/activate/vagrant_variables.yml 2024-03-27T21:42:44.358 INFO:teuthology.orchestra.run.smithi060.stdout: 409287 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/activate/group_vars 2024-03-27T21:42:44.359 INFO:teuthology.orchestra.run.smithi060.stdout: 409288 4 -rw-r--r-- 1 ubuntu ubuntu 440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/activate/group_vars/all 2024-03-27T21:42:44.359 INFO:teuthology.orchestra.run.smithi060.stdout: 409286 0 lrwxrwxrwx 1 ubuntu ubuntu 23 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/activate/Vagrantfile -> ../../../../Vagrantfile 2024-03-27T21:42:44.359 INFO:teuthology.orchestra.run.smithi060.stdout: 409289 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/activate/host_vars 2024-03-27T21:42:44.359 INFO:teuthology.orchestra.run.smithi060.stdout: 409291 4 -rw-r--r-- 1 ubuntu ubuntu 71 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/activate/host_vars/osd1.yml 2024-03-27T21:42:44.359 INFO:teuthology.orchestra.run.smithi060.stdout: 409290 4 -rw-r--r-- 1 ubuntu ubuntu 94 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/activate/host_vars/osd0.yml 2024-03-27T21:42:44.359 INFO:teuthology.orchestra.run.smithi060.stdout: 409292 4 -rw-r--r-- 1 ubuntu ubuntu 67 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/centos7/bluestore/activate/hosts 2024-03-27T21:42:44.359 INFO:teuthology.orchestra.run.smithi060.stdout: 409315 4 -rw-r--r-- 1 ubuntu ubuntu 2587 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/simple/tox.ini 2024-03-27T21:42:44.359 INFO:teuthology.orchestra.run.smithi060.stdout: 409181 12 -rw-r--r-- 1 ubuntu ubuntu 12215 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/Vagrantfile 2024-03-27T21:42:44.360 INFO:teuthology.orchestra.run.smithi060.stdout: 409179 4 -rw-r--r-- 1 ubuntu ubuntu 71 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/functional/.gitignore 2024-03-27T21:42:44.360 INFO:teuthology.orchestra.run.smithi060.stdout: 409327 4 -rw-r--r-- 1 ubuntu ubuntu 2487 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/test_decorators.py 2024-03-27T21:42:44.360 INFO:teuthology.orchestra.run.smithi060.stdout: 409323 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/systemd 2024-03-27T21:42:44.360 INFO:teuthology.orchestra.run.smithi060.stdout: 409325 4 -rw-r--r-- 1 ubuntu ubuntu 765 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/systemd/test_systemctl.py 2024-03-27T21:42:44.360 INFO:teuthology.orchestra.run.smithi060.stdout: 409324 4 -rw-r--r-- 1 ubuntu ubuntu 1377 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/systemd/test_main.py 2024-03-27T21:42:44.360 INFO:teuthology.orchestra.run.smithi060.stdout: 409326 4 -rw-r--r-- 1 ubuntu ubuntu 3819 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/test_configuration.py 2024-03-27T21:42:44.360 INFO:teuthology.orchestra.run.smithi060.stdout: 409329 4 -rw-r--r-- 1 ubuntu ubuntu 2823 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/test_main.py 2024-03-27T21:42:44.360 INFO:teuthology.orchestra.run.smithi060.stdout: 409332 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/util 2024-03-27T21:42:44.361 INFO:teuthology.orchestra.run.smithi060.stdout: 409334 32 -rw-r--r-- 1 ubuntu ubuntu 32026 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/util/test_device.py 2024-03-27T21:42:44.361 INFO:teuthology.orchestra.run.smithi060.stdout: 409337 12 -rw-r--r-- 1 ubuntu ubuntu 11024 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/util/test_prepare.py 2024-03-27T21:42:44.361 INFO:teuthology.orchestra.run.smithi060.stdout: 409333 16 -rw-r--r-- 1 ubuntu ubuntu 14940 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/util/test_arg_validators.py 2024-03-27T21:42:44.361 INFO:teuthology.orchestra.run.smithi060.stdout: 409339 4 -rw-r--r-- 1 ubuntu ubuntu 3642 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/util/test_util.py 2024-03-27T21:42:44.361 INFO:teuthology.orchestra.run.smithi060.stdout: 409338 16 -rw-r--r-- 1 ubuntu ubuntu 12909 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/util/test_system.py 2024-03-27T21:42:44.361 INFO:teuthology.orchestra.run.smithi060.stdout: 409336 8 -rw-r--r-- 1 ubuntu ubuntu 4386 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/util/test_encryption.py 2024-03-27T21:42:44.361 INFO:teuthology.orchestra.run.smithi060.stdout: 409335 24 -rw-r--r-- 1 ubuntu ubuntu 20506 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/util/test_disk.py 2024-03-27T21:42:44.361 INFO:teuthology.orchestra.run.smithi060.stdout: 409151 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/__init__.py 2024-03-27T21:42:44.362 INFO:teuthology.orchestra.run.smithi060.stdout: 409330 4 -rw-r--r-- 1 ubuntu ubuntu 3292 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/test_process.py 2024-03-27T21:42:44.362 INFO:teuthology.orchestra.run.smithi060.stdout: 409152 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/api 2024-03-27T21:42:44.362 INFO:teuthology.orchestra.run.smithi060.stdout: 409153 36 -rw-r--r-- 1 ubuntu ubuntu 35822 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/api/test_lvm.py 2024-03-27T21:42:44.362 INFO:teuthology.orchestra.run.smithi060.stdout: 409328 12 -rw-r--r-- 1 ubuntu ubuntu 8545 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/tests/test_inventory.py 2024-03-27T21:42:44.362 INFO:teuthology.orchestra.run.smithi060.stdout: 409135 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/drive_group 2024-03-27T21:42:44.362 INFO:teuthology.orchestra.run.smithi060.stdout: 409137 4 -rw-r--r-- 1 ubuntu ubuntu 3197 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/drive_group/main.py 2024-03-27T21:42:44.362 INFO:teuthology.orchestra.run.smithi060.stdout: 409136 4 -rw-r--r-- 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/drive_group/__init__.py 2024-03-27T21:42:44.362 INFO:teuthology.orchestra.run.smithi060.stdout: 409143 8 -rw-r--r-- 1 ubuntu ubuntu 6471 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/main.py 2024-03-27T21:42:44.363 INFO:teuthology.orchestra.run.smithi060.stdout: 409107 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices 2024-03-27T21:42:44.363 INFO:teuthology.orchestra.run.smithi060.stdout: 409109 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/lvm 2024-03-27T21:42:44.363 INFO:teuthology.orchestra.run.smithi060.stdout: 409114 4 -rw-r--r-- 1 ubuntu ubuntu 2925 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/lvm/create.py 2024-03-27T21:42:44.363 INFO:teuthology.orchestra.run.smithi060.stdout: 409117 4 -rw-r--r-- 1 ubuntu ubuntu 1409 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/lvm/main.py 2024-03-27T21:42:44.363 INFO:teuthology.orchestra.run.smithi060.stdout: 409120 4 -rw-r--r-- 1 ubuntu ubuntu 2174 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/lvm/trigger.py 2024-03-27T21:42:44.363 INFO:teuthology.orchestra.run.smithi060.stdout: 409112 24 -rw-r--r-- 1 ubuntu ubuntu 23244 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/lvm/batch.py 2024-03-27T21:42:44.363 INFO:teuthology.orchestra.run.smithi060.stdout: 409115 4 -rw-r--r-- 1 ubuntu ubuntu 2751 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/lvm/deactivate.py 2024-03-27T21:42:44.363 INFO:teuthology.orchestra.run.smithi060.stdout: 409111 12 -rw-r--r-- 1 ubuntu ubuntu 11116 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/lvm/activate.py 2024-03-27T21:42:44.363 INFO:teuthology.orchestra.run.smithi060.stdout: 409121 16 -rw-r--r-- 1 ubuntu ubuntu 14561 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/lvm/zap.py 2024-03-27T21:42:44.364 INFO:teuthology.orchestra.run.smithi060.stdout: 409113 8 -rw-r--r-- 1 ubuntu ubuntu 5310 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/lvm/common.py 2024-03-27T21:42:44.364 INFO:teuthology.orchestra.run.smithi060.stdout: 409118 28 -rw-r--r-- 1 ubuntu ubuntu 27903 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/lvm/migrate.py 2024-03-27T21:42:44.364 INFO:teuthology.orchestra.run.smithi060.stdout: 409119 16 -rw-r--r-- 1 ubuntu ubuntu 12778 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/lvm/prepare.py 2024-03-27T21:42:44.364 INFO:teuthology.orchestra.run.smithi060.stdout: 409110 4 -rw-r--r-- 1 ubuntu ubuntu 29 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/lvm/__init__.py 2024-03-27T21:42:44.364 INFO:teuthology.orchestra.run.smithi060.stdout: 409116 8 -rw-r--r-- 1 ubuntu ubuntu 7286 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/lvm/listing.py 2024-03-27T21:42:44.364 INFO:teuthology.orchestra.run.smithi060.stdout: 409129 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/simple 2024-03-27T21:42:44.364 INFO:teuthology.orchestra.run.smithi060.stdout: 409132 4 -rw-r--r-- 1 ubuntu ubuntu 1116 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/simple/main.py 2024-03-27T21:42:44.364 INFO:teuthology.orchestra.run.smithi060.stdout: 409134 4 -rw-r--r-- 1 ubuntu ubuntu 2202 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/simple/trigger.py 2024-03-27T21:42:44.365 INFO:teuthology.orchestra.run.smithi060.stdout: 409131 12 -rw-r--r-- 1 ubuntu ubuntu 11830 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/simple/activate.py 2024-03-27T21:42:44.365 INFO:teuthology.orchestra.run.smithi060.stdout: 409133 16 -rw-r--r-- 1 ubuntu ubuntu 15700 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/simple/scan.py 2024-03-27T21:42:44.365 INFO:teuthology.orchestra.run.smithi060.stdout: 409130 4 -rw-r--r-- 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/simple/__init__.py 2024-03-27T21:42:44.365 INFO:teuthology.orchestra.run.smithi060.stdout: 409108 4 -rw-r--r-- 1 ubuntu ubuntu 38 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/__init__.py 2024-03-27T21:42:44.365 INFO:teuthology.orchestra.run.smithi060.stdout: 409122 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/raw 2024-03-27T21:42:44.365 INFO:teuthology.orchestra.run.smithi060.stdout: 409127 4 -rw-r--r-- 1 ubuntu ubuntu 1069 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/raw/main.py 2024-03-27T21:42:44.365 INFO:teuthology.orchestra.run.smithi060.stdout: 409126 8 -rw-r--r-- 1 ubuntu ubuntu 7259 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/raw/list.py 2024-03-27T21:42:44.365 INFO:teuthology.orchestra.run.smithi060.stdout: 409124 8 -rw-r--r-- 1 ubuntu ubuntu 5575 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/raw/activate.py 2024-03-27T21:42:44.366 INFO:teuthology.orchestra.run.smithi060.stdout: 409125 4 -rw-r--r-- 1 ubuntu ubuntu 1694 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/raw/common.py 2024-03-27T21:42:44.366 INFO:teuthology.orchestra.run.smithi060.stdout: 409128 8 -rw-r--r-- 1 ubuntu ubuntu 5797 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/raw/prepare.py 2024-03-27T21:42:44.366 INFO:teuthology.orchestra.run.smithi060.stdout: 409123 4 -rw-r--r-- 1 ubuntu ubuntu 29 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/devices/raw/__init__.py 2024-03-27T21:42:44.366 INFO:teuthology.orchestra.run.smithi060.stdout: 409145 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/systemd 2024-03-27T21:42:44.366 INFO:teuthology.orchestra.run.smithi060.stdout: 409147 4 -rw-r--r-- 1 ubuntu ubuntu 3566 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/systemd/main.py 2024-03-27T21:42:44.366 INFO:teuthology.orchestra.run.smithi060.stdout: 409148 4 -rw-r--r-- 1 ubuntu ubuntu 2306 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/systemd/systemctl.py 2024-03-27T21:42:44.366 INFO:teuthology.orchestra.run.smithi060.stdout: 409146 4 -rw-r--r-- 1 ubuntu ubuntu 30 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/systemd/__init__.py 2024-03-27T21:42:44.366 INFO:teuthology.orchestra.run.smithi060.stdout: 409144 12 -rw-r--r-- 1 ubuntu ubuntu 8463 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/process.py 2024-03-27T21:42:44.367 INFO:teuthology.orchestra.run.smithi060.stdout: 409340 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/util 2024-03-27T21:42:44.367 INFO:teuthology.orchestra.run.smithi060.stdout: 409342 12 -rw-r--r-- 1 ubuntu ubuntu 9853 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/util/arg_validators.py 2024-03-27T21:42:44.367 INFO:teuthology.orchestra.run.smithi060.stdout: 409344 28 -rw-r--r-- 1 ubuntu ubuntu 25742 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/util/device.py 2024-03-27T21:42:44.367 INFO:teuthology.orchestra.run.smithi060.stdout: 409346 12 -rw-r--r-- 1 ubuntu ubuntu 11095 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/util/encryption.py 2024-03-27T21:42:44.367 INFO:teuthology.orchestra.run.smithi060.stdout: 409348 16 -rw-r--r-- 1 ubuntu ubuntu 15676 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/util/prepare.py 2024-03-27T21:42:44.367 INFO:teuthology.orchestra.run.smithi060.stdout: 409341 4 -rw-r--r-- 1 ubuntu ubuntu 3072 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/util/__init__.py 2024-03-27T21:42:44.367 INFO:teuthology.orchestra.run.smithi060.stdout: 409349 16 -rw-r--r-- 1 ubuntu ubuntu 13375 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/util/system.py 2024-03-27T21:42:44.368 INFO:teuthology.orchestra.run.smithi060.stdout: 409345 32 -rw-r--r-- 1 ubuntu ubuntu 31039 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/util/disk.py 2024-03-27T21:42:44.368 INFO:teuthology.orchestra.run.smithi060.stdout: 409343 4 -rw-r--r-- 1 ubuntu ubuntu 2743 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/util/constants.py 2024-03-27T21:42:44.368 INFO:teuthology.orchestra.run.smithi060.stdout: 409347 8 -rw-r--r-- 1 ubuntu ubuntu 6866 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/util/lsmdisk.py 2024-03-27T21:42:44.368 INFO:teuthology.orchestra.run.smithi060.stdout: 409350 4 -rw-r--r-- 1 ubuntu ubuntu 881 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/util/templates.py 2024-03-27T21:42:44.368 INFO:teuthology.orchestra.run.smithi060.stdout: 409098 4 -rw-r--r-- 1 ubuntu ubuntu 681 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/__init__.py 2024-03-27T21:42:44.368 INFO:teuthology.orchestra.run.smithi060.stdout: 409142 4 -rw-r--r-- 1 ubuntu ubuntu 1718 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/log.py 2024-03-27T21:42:44.368 INFO:teuthology.orchestra.run.smithi060.stdout: 409139 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/inventory 2024-03-27T21:42:44.368 INFO:teuthology.orchestra.run.smithi060.stdout: 409141 4 -rw-r--r-- 1 ubuntu ubuntu 2560 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/inventory/main.py 2024-03-27T21:42:44.368 INFO:teuthology.orchestra.run.smithi060.stdout: 409140 4 -rw-r--r-- 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/inventory/__init__.py 2024-03-27T21:42:44.369 INFO:teuthology.orchestra.run.smithi060.stdout: 409149 8 -rw-r--r-- 1 ubuntu ubuntu 5516 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/terminal.py 2024-03-27T21:42:44.369 INFO:teuthology.orchestra.run.smithi060.stdout: 409106 4 -rw-r--r-- 1 ubuntu ubuntu 2535 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/decorators.py 2024-03-27T21:42:44.369 INFO:teuthology.orchestra.run.smithi060.stdout: 409099 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/activate 2024-03-27T21:42:44.369 INFO:teuthology.orchestra.run.smithi060.stdout: 409101 4 -rw-r--r-- 1 ubuntu ubuntu 2588 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/activate/main.py 2024-03-27T21:42:44.369 INFO:teuthology.orchestra.run.smithi060.stdout: 409100 4 -rw-r--r-- 1 ubuntu ubuntu 34 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/activate/__init__.py 2024-03-27T21:42:44.369 INFO:teuthology.orchestra.run.smithi060.stdout: 409105 12 -rw-r--r-- 1 ubuntu ubuntu 8831 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/configuration.py 2024-03-27T21:42:44.369 INFO:teuthology.orchestra.run.smithi060.stdout: 409138 4 -rw-r--r-- 1 ubuntu ubuntu 1616 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/exceptions.py 2024-03-27T21:42:44.369 INFO:teuthology.orchestra.run.smithi060.stdout: 409102 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/api 2024-03-27T21:42:44.370 INFO:teuthology.orchestra.run.smithi060.stdout: 409104 40 -rw-r--r-- 1 ubuntu ubuntu 40881 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/api/lvm.py 2024-03-27T21:42:44.370 INFO:teuthology.orchestra.run.smithi060.stdout: 409103 4 -rw-r--r-- 1 ubuntu ubuntu 67 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/ceph_volume/api/__init__.py 2024-03-27T21:42:44.370 INFO:teuthology.orchestra.run.smithi060.stdout: 409094 4 -rw-r--r-- 1 ubuntu ubuntu 88 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/.gitignore 2024-03-27T21:42:44.370 INFO:teuthology.orchestra.run.smithi060.stdout: 409375 4 -rw-r--r-- 1 ubuntu ubuntu 1387 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-volume/setup.py 2024-03-27T21:42:44.370 INFO:teuthology.orchestra.run.smithi060.stdout: 412194 4 -rwxr-xr-x 1 ubuntu ubuntu 1516 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ps-ceph.pl 2024-03-27T21:42:44.370 INFO:teuthology.orchestra.run.smithi060.stdout: 415332 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/seastar 2024-03-27T21:42:44.370 INFO:teuthology.orchestra.run.smithi060.stdout: 408975 8 -rw-r--r-- 1 ubuntu ubuntu 4281 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/SimpleRADOSStriper.h 2024-03-27T21:42:44.370 INFO:teuthology.orchestra.run.smithi060.stdout: 411875 4 -rwxr-xr-x 1 ubuntu ubuntu 1543 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mrgw.sh 2024-03-27T21:42:44.371 INFO:teuthology.orchestra.run.smithi060.stdout: 410867 28 -rw-r--r-- 1 ubuntu ubuntu 27917 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/libcephsqlite.cc 2024-03-27T21:42:44.371 INFO:teuthology.orchestra.run.smithi060.stdout: 409092 4 -rwxr-xr-x 1 ubuntu ubuntu 400 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-run 2024-03-27T21:42:44.371 INFO:teuthology.orchestra.run.smithi060.stdout: 411983 4 -rwxr-xr-x 1 ubuntu ubuntu 1066 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/nasm-wrapper 2024-03-27T21:42:44.371 INFO:teuthology.orchestra.run.smithi060.stdout: 417127 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/zstd 2024-03-27T21:42:44.371 INFO:teuthology.orchestra.run.smithi060.stdout: 417124 4 -rwxr-xr-x 1 ubuntu ubuntu 437 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/vnewosd.sh 2024-03-27T21:42:44.371 INFO:teuthology.orchestra.run.smithi060.stdout: 409069 4 -rwxr-xr-x 1 ubuntu ubuntu 1478 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-clsinfo 2024-03-27T21:42:44.371 INFO:teuthology.orchestra.run.smithi060.stdout: 410073 4 drwxr-xr-x 12 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson 2024-03-27T21:42:44.371 INFO:teuthology.orchestra.run.smithi060.stdout: 410135 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/mgr 2024-03-27T21:42:44.372 INFO:teuthology.orchestra.run.smithi060.stdout: 410137 4 -rw-r--r-- 1 ubuntu ubuntu 2006 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/mgr/client.h 2024-03-27T21:42:44.372 INFO:teuthology.orchestra.run.smithi060.stdout: 410136 8 -rw-r--r-- 1 ubuntu ubuntu 5536 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/mgr/client.cc 2024-03-27T21:42:44.372 INFO:teuthology.orchestra.run.smithi060.stdout: 410132 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/crush 2024-03-27T21:42:44.372 INFO:teuthology.orchestra.run.smithi060.stdout: 410133 8 -rw-r--r-- 1 ubuntu ubuntu 5942 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/crush/CrushLocation.cc 2024-03-27T21:42:44.372 INFO:teuthology.orchestra.run.smithi060.stdout: 410134 4 -rw-r--r-- 1 ubuntu ubuntu 884 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/crush/CrushLocation.h 2024-03-27T21:42:44.372 INFO:teuthology.orchestra.run.smithi060.stdout: 410074 8 -rw-r--r-- 1 ubuntu ubuntu 7945 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/CMakeLists.txt 2024-03-27T21:42:44.372 INFO:teuthology.orchestra.run.smithi060.stdout: 410138 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/mon 2024-03-27T21:42:44.372 INFO:teuthology.orchestra.run.smithi060.stdout: 410140 8 -rw-r--r-- 1 ubuntu ubuntu 6818 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/mon/MonClient.h 2024-03-27T21:42:44.373 INFO:teuthology.orchestra.run.smithi060.stdout: 410139 40 -rw-r--r-- 1 ubuntu ubuntu 36926 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/mon/MonClient.cc 2024-03-27T21:42:44.373 INFO:teuthology.orchestra.run.smithi060.stdout: 410075 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/admin 2024-03-27T21:42:44.373 INFO:teuthology.orchestra.run.smithi060.stdout: 410076 4 -rw-r--r-- 1 ubuntu ubuntu 208 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/admin/CMakeLists.txt 2024-03-27T21:42:44.373 INFO:teuthology.orchestra.run.smithi060.stdout: 410078 8 -rw-r--r-- 1 ubuntu ubuntu 5795 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/admin/admin_socket.h 2024-03-27T21:42:44.373 INFO:teuthology.orchestra.run.smithi060.stdout: 410077 20 -rw-r--r-- 1 ubuntu ubuntu 18398 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/admin/admin_socket.cc 2024-03-27T21:42:44.373 INFO:teuthology.orchestra.run.smithi060.stdout: 410082 4 -rw-r--r-- 1 ubuntu ubuntu 271 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/admin/pg_commands.h 2024-03-27T21:42:44.373 INFO:teuthology.orchestra.run.smithi060.stdout: 410081 8 -rw-r--r-- 1 ubuntu ubuntu 6796 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/admin/pg_commands.cc 2024-03-27T21:42:44.373 INFO:teuthology.orchestra.run.smithi060.stdout: 410080 4 -rw-r--r-- 1 ubuntu ubuntu 664 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/admin/osd_admin.h 2024-03-27T21:42:44.373 INFO:teuthology.orchestra.run.smithi060.stdout: 410079 20 -rw-r--r-- 1 ubuntu ubuntu 19362 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/admin/osd_admin.cc 2024-03-27T21:42:44.374 INFO:teuthology.orchestra.run.smithi060.stdout: 410164 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os 2024-03-27T21:42:44.374 INFO:teuthology.orchestra.run.smithi060.stdout: 410186 8 -rw-r--r-- 1 ubuntu ubuntu 6141 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/futurized_store.h 2024-03-27T21:42:44.374 INFO:teuthology.orchestra.run.smithi060.stdout: 410185 4 -rw-r--r-- 1 ubuntu ubuntu 971 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/futurized_store.cc 2024-03-27T21:42:44.374 INFO:teuthology.orchestra.run.smithi060.stdout: 410165 4 -rw-r--r-- 1 ubuntu ubuntu 357 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/CMakeLists.txt 2024-03-27T21:42:44.374 INFO:teuthology.orchestra.run.smithi060.stdout: 410166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/alienstore 2024-03-27T21:42:44.374 INFO:teuthology.orchestra.run.smithi060.stdout: 410175 8 -rw-r--r-- 1 ubuntu ubuntu 5384 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/alienstore/thread_pool.h 2024-03-27T21:42:44.374 INFO:teuthology.orchestra.run.smithi060.stdout: 410167 4 -rw-r--r-- 1 ubuntu ubuntu 3256 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/alienstore/CMakeLists.txt 2024-03-27T21:42:44.375 INFO:teuthology.orchestra.run.smithi060.stdout: 410169 4 -rw-r--r-- 1 ubuntu ubuntu 831 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/alienstore/alien_log.cc 2024-03-27T21:42:44.375 INFO:teuthology.orchestra.run.smithi060.stdout: 410173 4 -rw-r--r-- 1 ubuntu ubuntu 2219 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/alienstore/semaphore.h 2024-03-27T21:42:44.375 INFO:teuthology.orchestra.run.smithi060.stdout: 410168 4 -rw-r--r-- 1 ubuntu ubuntu 1233 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/alienstore/alien_collection.h 2024-03-27T21:42:44.375 INFO:teuthology.orchestra.run.smithi060.stdout: 410171 20 -rw-r--r-- 1 ubuntu ubuntu 18995 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/alienstore/alien_store.cc 2024-03-27T21:42:44.375 INFO:teuthology.orchestra.run.smithi060.stdout: 410170 4 -rw-r--r-- 1 ubuntu ubuntu 565 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/alienstore/alien_log.h 2024-03-27T21:42:44.375 INFO:teuthology.orchestra.run.smithi060.stdout: 410174 4 -rw-r--r-- 1 ubuntu ubuntu 2520 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/alienstore/thread_pool.cc 2024-03-27T21:42:44.375 INFO:teuthology.orchestra.run.smithi060.stdout: 410172 8 -rw-r--r-- 1 ubuntu ubuntu 4695 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/alienstore/alien_store.h 2024-03-27T21:42:44.375 INFO:teuthology.orchestra.run.smithi060.stdout: 410187 4 drwxr-xr-x 11 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore 2024-03-27T21:42:44.375 INFO:teuthology.orchestra.run.smithi060.stdout: 410196 4 -rw-r--r-- 1 ubuntu ubuntu 470 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/backref_manager.cc 2024-03-27T21:42:44.376 INFO:teuthology.orchestra.run.smithi060.stdout: 410220 4 -rw-r--r-- 1 ubuntu ubuntu 628 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/journal.cc 2024-03-27T21:42:44.376 INFO:teuthology.orchestra.run.smithi060.stdout: 410219 4 -rw-r--r-- 1 ubuntu ubuntu 997 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/extentmap_manager.cc 2024-03-27T21:42:44.376 INFO:teuthology.orchestra.run.smithi060.stdout: 410322 8 -rw-r--r-- 1 ubuntu ubuntu 6414 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/segment_manager.h 2024-03-27T21:42:44.376 INFO:teuthology.orchestra.run.smithi060.stdout: 410335 28 -rw-r--r-- 1 ubuntu ubuntu 28245 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/transaction_manager.cc 2024-03-27T21:42:44.376 INFO:teuthology.orchestra.run.smithi060.stdout: 410316 4 -rw-r--r-- 1 ubuntu ubuntu 3419 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/root_block.h 2024-03-27T21:42:44.376 INFO:teuthology.orchestra.run.smithi060.stdout: 410255 4 -rw-r--r-- 1 ubuntu ubuntu 389 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode.cc 2024-03-27T21:42:44.376 INFO:teuthology.orchestra.run.smithi060.stdout: 410330 8 -rw-r--r-- 1 ubuntu ubuntu 5700 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/segment_manager_group.cc 2024-03-27T21:42:44.376 INFO:teuthology.orchestra.run.smithi060.stdout: 410314 4 -rw-r--r-- 1 ubuntu ubuntu 2932 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/record_scanner.h 2024-03-27T21:42:44.376 INFO:teuthology.orchestra.run.smithi060.stdout: 410244 4 -rw-r--r-- 1 ubuntu ubuntu 1225 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/omap_manager.cc 2024-03-27T21:42:44.377 INFO:teuthology.orchestra.run.smithi060.stdout: 410188 4 -rw-r--r-- 1 ubuntu ubuntu 2410 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/CMakeLists.txt 2024-03-27T21:42:44.377 INFO:teuthology.orchestra.run.smithi060.stdout: 410221 4 -rw-r--r-- 1 ubuntu ubuntu 3290 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/journal.h 2024-03-27T21:42:44.377 INFO:teuthology.orchestra.run.smithi060.stdout: 410205 56 -rw-r--r-- 1 ubuntu ubuntu 54312 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/cache.h 2024-03-27T21:42:44.377 INFO:teuthology.orchestra.run.smithi060.stdout: 410216 8 -rw-r--r-- 1 ubuntu ubuntu 4399 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/device.h 2024-03-27T21:42:44.377 INFO:teuthology.orchestra.run.smithi060.stdout: 410233 4 -rw-r--r-- 1 ubuntu ubuntu 878 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/lba_manager.cc 2024-03-27T21:42:44.377 INFO:teuthology.orchestra.run.smithi060.stdout: 410331 8 -rw-r--r-- 1 ubuntu ubuntu 4674 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/segment_manager_group.h 2024-03-27T21:42:44.377 INFO:teuthology.orchestra.run.smithi060.stdout: 410320 64 -rw-r--r-- 1 ubuntu ubuntu 64665 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/seastore_types.h 2024-03-27T21:42:44.377 INFO:teuthology.orchestra.run.smithi060.stdout: 410191 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/backref 2024-03-27T21:42:44.378 INFO:teuthology.orchestra.run.smithi060.stdout: 410193 4 -rw-r--r-- 1 ubuntu ubuntu 3877 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/backref/backref_tree_node.h 2024-03-27T21:42:44.378 INFO:teuthology.orchestra.run.smithi060.stdout: 410192 4 -rw-r--r-- 1 ubuntu ubuntu 416 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/backref/backref_tree_node.cc 2024-03-27T21:42:44.378 INFO:teuthology.orchestra.run.smithi060.stdout: 410195 4 -rw-r--r-- 1 ubuntu ubuntu 2888 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/backref/btree_backref_manager.h 2024-03-27T21:42:44.378 INFO:teuthology.orchestra.run.smithi060.stdout: 410194 20 -rw-r--r-- 1 ubuntu ubuntu 18977 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/backref/btree_backref_manager.cc 2024-03-27T21:42:44.378 INFO:teuthology.orchestra.run.smithi060.stdout: 410245 8 -rw-r--r-- 1 ubuntu ubuntu 6721 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/omap_manager.h 2024-03-27T21:42:44.378 INFO:teuthology.orchestra.run.smithi060.stdout: 410323 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/segment_manager 2024-03-27T21:42:44.378 INFO:teuthology.orchestra.run.smithi060.stdout: 410325 8 -rw-r--r-- 1 ubuntu ubuntu 6606 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/segment_manager/block.h 2024-03-27T21:42:44.378 INFO:teuthology.orchestra.run.smithi060.stdout: 410326 8 -rw-r--r-- 1 ubuntu ubuntu 8041 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/segment_manager/ephemeral.cc 2024-03-27T21:42:44.378 INFO:teuthology.orchestra.run.smithi060.stdout: 410327 8 -rw-r--r-- 1 ubuntu ubuntu 4354 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/segment_manager/ephemeral.h 2024-03-27T21:42:44.379 INFO:teuthology.orchestra.run.smithi060.stdout: 410324 28 -rw-r--r-- 1 ubuntu ubuntu 24720 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/segment_manager/block.cc 2024-03-27T21:42:44.379 INFO:teuthology.orchestra.run.smithi060.stdout: 410329 8 -rw-r--r-- 1 ubuntu ubuntu 6464 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/segment_manager/zbd.h 2024-03-27T21:42:44.379 INFO:teuthology.orchestra.run.smithi060.stdout: 410328 24 -rw-r--r-- 1 ubuntu ubuntu 22318 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/segment_manager/zbd.cc 2024-03-27T21:42:44.379 INFO:teuthology.orchestra.run.smithi060.stdout: 410204 68 -rw-r--r-- 1 ubuntu ubuntu 69616 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/cache.cc 2024-03-27T21:42:44.379 INFO:teuthology.orchestra.run.smithi060.stdout: 410300 4 -rw-r--r-- 1 ubuntu ubuntu 639 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/random_block_manager.cc 2024-03-27T21:42:44.379 INFO:teuthology.orchestra.run.smithi060.stdout: 410302 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/random_block_manager 2024-03-27T21:42:44.379 INFO:teuthology.orchestra.run.smithi060.stdout: 410305 8 -rw-r--r-- 1 ubuntu ubuntu 6307 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/random_block_manager/block_rb_manager.cc 2024-03-27T21:42:44.379 INFO:teuthology.orchestra.run.smithi060.stdout: 410309 12 -rw-r--r-- 1 ubuntu ubuntu 10069 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/random_block_manager/nvme_block_device.h 2024-03-27T21:42:44.380 INFO:teuthology.orchestra.run.smithi060.stdout: 410306 8 -rw-r--r-- 1 ubuntu ubuntu 4251 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/random_block_manager/block_rb_manager.h 2024-03-27T21:42:44.380 INFO:teuthology.orchestra.run.smithi060.stdout: 410310 8 -rw-r--r-- 1 ubuntu ubuntu 7657 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/random_block_manager/rbm_device.cc 2024-03-27T21:42:44.380 INFO:teuthology.orchestra.run.smithi060.stdout: 410308 12 -rw-r--r-- 1 ubuntu ubuntu 9555 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/random_block_manager/nvme_block_device.cc 2024-03-27T21:42:44.380 INFO:teuthology.orchestra.run.smithi060.stdout: 410303 8 -rw-r--r-- 1 ubuntu ubuntu 7604 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/random_block_manager/avlallocator.cc 2024-03-27T21:42:44.380 INFO:teuthology.orchestra.run.smithi060.stdout: 410304 8 -rw-r--r-- 1 ubuntu ubuntu 4543 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/random_block_manager/avlallocator.h 2024-03-27T21:42:44.380 INFO:teuthology.orchestra.run.smithi060.stdout: 410311 8 -rw-r--r-- 1 ubuntu ubuntu 7167 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/random_block_manager/rbm_device.h 2024-03-27T21:42:44.380 INFO:teuthology.orchestra.run.smithi060.stdout: 410307 4 -rw-r--r-- 1 ubuntu ubuntu 2245 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/random_block_manager/extent_allocator.h 2024-03-27T21:42:44.380 INFO:teuthology.orchestra.run.smithi060.stdout: 410313 12 -rw-r--r-- 1 ubuntu ubuntu 8542 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/record_scanner.cc 2024-03-27T21:42:44.380 INFO:teuthology.orchestra.run.smithi060.stdout: 410258 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager 2024-03-27T21:42:44.381 INFO:teuthology.orchestra.run.smithi060.stdout: 410259 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree 2024-03-27T21:42:44.381 INFO:teuthology.orchestra.run.smithi060.stdout: 410269 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/node_extent_manager 2024-03-27T21:42:44.381 INFO:teuthology.orchestra.run.smithi060.stdout: 410270 8 -rw-r--r-- 1 ubuntu ubuntu 6425 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/node_extent_manager/dummy.h 2024-03-27T21:42:44.381 INFO:teuthology.orchestra.run.smithi060.stdout: 410271 4 -rw-r--r-- 1 ubuntu ubuntu 3145 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/node_extent_manager/seastore.cc 2024-03-27T21:42:44.381 INFO:teuthology.orchestra.run.smithi060.stdout: 410272 8 -rw-r--r-- 1 ubuntu ubuntu 6878 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/node_extent_manager/seastore.h 2024-03-27T21:42:44.381 INFO:teuthology.orchestra.run.smithi060.stdout: 410273 4 -rw-r--r-- 1 ubuntu ubuntu 2300 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/node_extent_manager/test_replay.h 2024-03-27T21:42:44.381 INFO:teuthology.orchestra.run.smithi060.stdout: 410262 8 -rw-r--r-- 1 ubuntu ubuntu 6567 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/fwd.h 2024-03-27T21:42:44.381 INFO:teuthology.orchestra.run.smithi060.stdout: 410266 24 -rw-r--r-- 1 ubuntu ubuntu 21818 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/node_extent_accessor.h 2024-03-27T21:42:44.382 INFO:teuthology.orchestra.run.smithi060.stdout: 410275 4 -rw-r--r-- 1 ubuntu ubuntu 2416 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/node_impl.cc 2024-03-27T21:42:44.382 INFO:teuthology.orchestra.run.smithi060.stdout: 410293 4 -rw-r--r-- 1 ubuntu ubuntu 567 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/super.cc 2024-03-27T21:42:44.382 INFO:teuthology.orchestra.run.smithi060.stdout: 410261 12 -rw-r--r-- 1 ubuntu ubuntu 8204 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/fltree_onode_manager.h 2024-03-27T21:42:44.382 INFO:teuthology.orchestra.run.smithi060.stdout: 410265 4 -rw-r--r-- 1 ubuntu ubuntu 1368 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/node_delta_recorder.h 2024-03-27T21:42:44.382 INFO:teuthology.orchestra.run.smithi060.stdout: 410276 12 -rw-r--r-- 1 ubuntu ubuntu 9486 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/node_impl.h 2024-03-27T21:42:44.382 INFO:teuthology.orchestra.run.smithi060.stdout: 410274 4 -rw-r--r-- 1 ubuntu ubuntu 3146 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/node_extent_mutable.h 2024-03-27T21:42:44.382 INFO:teuthology.orchestra.run.smithi060.stdout: 410279 8 -rw-r--r-- 1 ubuntu ubuntu 4460 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/node_types.h 2024-03-27T21:42:44.383 INFO:teuthology.orchestra.run.smithi060.stdout: 410297 8 -rw-r--r-- 1 ubuntu ubuntu 5066 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/value.cc 2024-03-27T21:42:44.383 INFO:teuthology.orchestra.run.smithi060.stdout: 410295 12 -rw-r--r-- 1 ubuntu ubuntu 11585 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/tree.h 2024-03-27T21:42:44.383 INFO:teuthology.orchestra.run.smithi060.stdout: 410298 12 -rw-r--r-- 1 ubuntu ubuntu 9925 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/value.h 2024-03-27T21:42:44.383 INFO:teuthology.orchestra.run.smithi060.stdout: 410260 12 -rw-r--r-- 1 ubuntu ubuntu 8710 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/fltree_onode_manager.cc 2024-03-27T21:42:44.383 INFO:teuthology.orchestra.run.smithi060.stdout: 410264 28 -rw-r--r-- 1 ubuntu ubuntu 24933 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/node.h 2024-03-27T21:42:44.383 INFO:teuthology.orchestra.run.smithi060.stdout: 410267 4 -rw-r--r-- 1 ubuntu ubuntu 885 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/node_extent_manager.cc 2024-03-27T21:42:44.383 INFO:teuthology.orchestra.run.smithi060.stdout: 410268 4 -rw-r--r-- 1 ubuntu ubuntu 3120 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/node_extent_manager.h 2024-03-27T21:42:44.383 INFO:teuthology.orchestra.run.smithi060.stdout: 410277 36 -rw-r--r-- 1 ubuntu ubuntu 35617 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/node_layout.h 2024-03-27T21:42:44.384 INFO:teuthology.orchestra.run.smithi060.stdout: 410294 4 -rw-r--r-- 1 ubuntu ubuntu 3839 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/super.h 2024-03-27T21:42:44.384 INFO:teuthology.orchestra.run.smithi060.stdout: 410278 8 -rw-r--r-- 1 ubuntu ubuntu 4486 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/node_layout_replayable.h 2024-03-27T21:42:44.384 INFO:teuthology.orchestra.run.smithi060.stdout: 410263 80 -rw-r--r-- 1 ubuntu ubuntu 78153 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/node.cc 2024-03-27T21:42:44.384 INFO:teuthology.orchestra.run.smithi060.stdout: 410280 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/stages 2024-03-27T21:42:44.384 INFO:teuthology.orchestra.run.smithi060.stdout: 410281 8 -rw-r--r-- 1 ubuntu ubuntu 5914 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/stages/item_iterator_stage.cc 2024-03-27T21:42:44.384 INFO:teuthology.orchestra.run.smithi060.stdout: 410286 8 -rw-r--r-- 1 ubuntu ubuntu 8009 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/stages/node_stage.h 2024-03-27T21:42:44.384 INFO:teuthology.orchestra.run.smithi060.stdout: 410291 12 -rw-r--r-- 1 ubuntu ubuntu 11770 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/stages/sub_items_stage.cc 2024-03-27T21:42:44.384 INFO:teuthology.orchestra.run.smithi060.stdout: 410285 16 -rw-r--r-- 1 ubuntu ubuntu 15824 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/stages/node_stage.cc 2024-03-27T21:42:44.384 INFO:teuthology.orchestra.run.smithi060.stdout: 410283 4 -rw-r--r-- 1 ubuntu ubuntu 949 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/stages/key_layout.cc 2024-03-27T21:42:44.385 INFO:teuthology.orchestra.run.smithi060.stdout: 410287 8 -rw-r--r-- 1 ubuntu ubuntu 5268 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/stages/node_stage_layout.cc 2024-03-27T21:42:44.385 INFO:teuthology.orchestra.run.smithi060.stdout: 410284 28 -rw-r--r-- 1 ubuntu ubuntu 26592 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/stages/key_layout.h 2024-03-27T21:42:44.385 INFO:teuthology.orchestra.run.smithi060.stdout: 410289 80 -rw-r--r-- 1 ubuntu ubuntu 80714 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/stages/stage.h 2024-03-27T21:42:44.385 INFO:teuthology.orchestra.run.smithi060.stdout: 410282 8 -rw-r--r-- 1 ubuntu ubuntu 6505 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/stages/item_iterator_stage.h 2024-03-27T21:42:44.385 INFO:teuthology.orchestra.run.smithi060.stdout: 410288 16 -rw-r--r-- 1 ubuntu ubuntu 14889 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/stages/node_stage_layout.h 2024-03-27T21:42:44.385 INFO:teuthology.orchestra.run.smithi060.stdout: 410290 16 -rw-r--r-- 1 ubuntu ubuntu 12496 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/stages/stage_types.h 2024-03-27T21:42:44.385 INFO:teuthology.orchestra.run.smithi060.stdout: 410292 16 -rw-r--r-- 1 ubuntu ubuntu 12455 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/stages/sub_items_stage.h 2024-03-27T21:42:44.385 INFO:teuthology.orchestra.run.smithi060.stdout: 410296 20 -rw-r--r-- 1 ubuntu ubuntu 17436 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager/staged-fltree/tree_utils.h 2024-03-27T21:42:44.386 INFO:teuthology.orchestra.run.smithi060.stdout: 410234 8 -rw-r--r-- 1 ubuntu ubuntu 6488 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/lba_manager.h 2024-03-27T21:42:44.386 INFO:teuthology.orchestra.run.smithi060.stdout: 410317 68 -rw-r--r-- 1 ubuntu ubuntu 67588 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/seastore.cc 2024-03-27T21:42:44.386 INFO:teuthology.orchestra.run.smithi060.stdout: 410318 16 -rw-r--r-- 1 ubuntu ubuntu 16035 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/seastore.h 2024-03-27T21:42:44.386 INFO:teuthology.orchestra.run.smithi060.stdout: 410206 8 -rw-r--r-- 1 ubuntu ubuntu 4999 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/cached_extent.cc 2024-03-27T21:42:44.386 INFO:teuthology.orchestra.run.smithi060.stdout: 410197 8 -rw-r--r-- 1 ubuntu ubuntu 4445 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/backref_manager.h 2024-03-27T21:42:44.386 INFO:teuthology.orchestra.run.smithi060.stdout: 410312 4 -rw-r--r-- 1 ubuntu ubuntu 1787 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/randomblock_manager_group.h 2024-03-27T21:42:44.386 INFO:teuthology.orchestra.run.smithi060.stdout: 410333 4 -rw-r--r-- 1 ubuntu ubuntu 277 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/transaction.cc 2024-03-27T21:42:44.387 INFO:teuthology.orchestra.run.smithi060.stdout: 410336 32 -rw-r--r-- 1 ubuntu ubuntu 30674 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/transaction_manager.h 2024-03-27T21:42:44.387 INFO:teuthology.orchestra.run.smithi060.stdout: 410215 4 -rw-r--r-- 1 ubuntu ubuntu 1359 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/device.cc 2024-03-27T21:42:44.387 INFO:teuthology.orchestra.run.smithi060.stdout: 410315 4 -rw-r--r-- 1 ubuntu ubuntu 992 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/root_block.cc 2024-03-27T21:42:44.387 INFO:teuthology.orchestra.run.smithi060.stdout: 410334 20 -rw-r--r-- 1 ubuntu ubuntu 18849 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/transaction.h 2024-03-27T21:42:44.387 INFO:teuthology.orchestra.run.smithi060.stdout: 410217 28 -rw-r--r-- 1 ubuntu ubuntu 27805 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/extent_placement_manager.cc 2024-03-27T21:42:44.387 INFO:teuthology.orchestra.run.smithi060.stdout: 410246 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/omap_manager 2024-03-27T21:42:44.387 INFO:teuthology.orchestra.run.smithi060.stdout: 410247 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/omap_manager/btree 2024-03-27T21:42:44.387 INFO:teuthology.orchestra.run.smithi060.stdout: 410251 28 -rw-r--r-- 1 ubuntu ubuntu 26266 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/omap_manager/btree/omap_btree_node_impl.cc 2024-03-27T21:42:44.387 INFO:teuthology.orchestra.run.smithi060.stdout: 410254 44 -rw-r--r-- 1 ubuntu ubuntu 42173 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/omap_manager/btree/string_kv_node_layout.h 2024-03-27T21:42:44.388 INFO:teuthology.orchestra.run.smithi060.stdout: 410250 4 -rw-r--r-- 1 ubuntu ubuntu 3874 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/omap_manager/btree/omap_btree_node.h 2024-03-27T21:42:44.388 INFO:teuthology.orchestra.run.smithi060.stdout: 410253 8 -rw-r--r-- 1 ubuntu ubuntu 4373 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/omap_manager/btree/omap_types.h 2024-03-27T21:42:44.388 INFO:teuthology.orchestra.run.smithi060.stdout: 410249 4 -rw-r--r-- 1 ubuntu ubuntu 3082 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/omap_manager/btree/btree_omap_manager.h 2024-03-27T21:42:44.388 INFO:teuthology.orchestra.run.smithi060.stdout: 410248 12 -rw-r--r-- 1 ubuntu ubuntu 9408 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/omap_manager/btree/btree_omap_manager.cc 2024-03-27T21:42:44.388 INFO:teuthology.orchestra.run.smithi060.stdout: 410252 8 -rw-r--r-- 1 ubuntu ubuntu 7350 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/omap_manager/btree/omap_btree_node_impl.h 2024-03-27T21:42:44.388 INFO:teuthology.orchestra.run.smithi060.stdout: 410222 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/journal 2024-03-27T21:42:44.388 INFO:teuthology.orchestra.run.smithi060.stdout: 410223 16 -rw-r--r-- 1 ubuntu ubuntu 13341 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/journal/circular_bounded_journal.cc 2024-03-27T21:42:44.388 INFO:teuthology.orchestra.run.smithi060.stdout: 410232 4 -rw-r--r-- 1 ubuntu ubuntu 3038 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/journal/segmented_journal.h 2024-03-27T21:42:44.389 INFO:teuthology.orchestra.run.smithi060.stdout: 410228 12 -rw-r--r-- 1 ubuntu ubuntu 9903 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/journal/record_submitter.h 2024-03-27T21:42:44.389 INFO:teuthology.orchestra.run.smithi060.stdout: 410227 20 -rw-r--r-- 1 ubuntu ubuntu 17121 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/journal/record_submitter.cc 2024-03-27T21:42:44.389 INFO:teuthology.orchestra.run.smithi060.stdout: 410230 4 -rw-r--r-- 1 ubuntu ubuntu 3545 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/journal/segment_allocator.h 2024-03-27T21:42:44.389 INFO:teuthology.orchestra.run.smithi060.stdout: 410226 8 -rw-r--r-- 1 ubuntu ubuntu 7240 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/journal/circular_journal_space.h 2024-03-27T21:42:44.389 INFO:teuthology.orchestra.run.smithi060.stdout: 410225 8 -rw-r--r-- 1 ubuntu ubuntu 7295 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/journal/circular_journal_space.cc 2024-03-27T21:42:44.389 INFO:teuthology.orchestra.run.smithi060.stdout: 410224 8 -rw-r--r-- 1 ubuntu ubuntu 6157 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/journal/circular_bounded_journal.h 2024-03-27T21:42:44.389 INFO:teuthology.orchestra.run.smithi060.stdout: 410229 12 -rw-r--r-- 1 ubuntu ubuntu 8525 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/journal/segment_allocator.cc 2024-03-27T21:42:44.389 INFO:teuthology.orchestra.run.smithi060.stdout: 410231 16 -rw-r--r-- 1 ubuntu ubuntu 14479 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/journal/segmented_journal.cc 2024-03-27T21:42:44.389 INFO:teuthology.orchestra.run.smithi060.stdout: 410241 4 -rw-r--r-- 1 ubuntu ubuntu 1259 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/logging.h 2024-03-27T21:42:44.390 INFO:teuthology.orchestra.run.smithi060.stdout: 410257 4 -rw-r--r-- 1 ubuntu ubuntu 2486 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode_manager.h 2024-03-27T21:42:44.390 INFO:teuthology.orchestra.run.smithi060.stdout: 410198 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/btree 2024-03-27T21:42:44.390 INFO:teuthology.orchestra.run.smithi060.stdout: 410202 4 -rw-r--r-- 1 ubuntu ubuntu 436 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/btree/fixed_kv_node.cc 2024-03-27T21:42:44.390 INFO:teuthology.orchestra.run.smithi060.stdout: 410203 40 -rw-r--r-- 1 ubuntu ubuntu 38909 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/btree/fixed_kv_node.h 2024-03-27T21:42:44.390 INFO:teuthology.orchestra.run.smithi060.stdout: 410199 4 -rw-r--r-- 1 ubuntu ubuntu 1089 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/btree/btree_range_pin.cc 2024-03-27T21:42:44.390 INFO:teuthology.orchestra.run.smithi060.stdout: 410201 72 -rw-r--r-- 1 ubuntu ubuntu 69892 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/btree/fixed_kv_btree.h 2024-03-27T21:42:44.390 INFO:teuthology.orchestra.run.smithi060.stdout: 410200 8 -rw-r--r-- 1 ubuntu ubuntu 5040 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/btree/btree_range_pin.h 2024-03-27T21:42:44.390 INFO:teuthology.orchestra.run.smithi060.stdout: 410189 60 -rw-r--r-- 1 ubuntu ubuntu 58811 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/async_cleaner.cc 2024-03-27T21:42:44.391 INFO:teuthology.orchestra.run.smithi060.stdout: 410235 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/lba_manager 2024-03-27T21:42:44.391 INFO:teuthology.orchestra.run.smithi060.stdout: 410236 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/lba_manager/btree 2024-03-27T21:42:44.391 INFO:teuthology.orchestra.run.smithi060.stdout: 410237 24 -rw-r--r-- 1 ubuntu ubuntu 22091 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/lba_manager/btree/btree_lba_manager.cc 2024-03-27T21:42:44.391 INFO:teuthology.orchestra.run.smithi060.stdout: 410238 12 -rw-r--r-- 1 ubuntu ubuntu 11590 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/lba_manager/btree/btree_lba_manager.h 2024-03-27T21:42:44.391 INFO:teuthology.orchestra.run.smithi060.stdout: 410239 4 -rw-r--r-- 1 ubuntu ubuntu 1474 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/lba_manager/btree/lba_btree_node.cc 2024-03-27T21:42:44.391 INFO:teuthology.orchestra.run.smithi060.stdout: 410240 12 -rw-r--r-- 1 ubuntu ubuntu 8679 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/lba_manager/btree/lba_btree_node.h 2024-03-27T21:42:44.391 INFO:teuthology.orchestra.run.smithi060.stdout: 410210 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/collection_manager 2024-03-27T21:42:44.391 INFO:teuthology.orchestra.run.smithi060.stdout: 410212 8 -rw-r--r-- 1 ubuntu ubuntu 5152 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/collection_manager/collection_flat_node.h 2024-03-27T21:42:44.392 INFO:teuthology.orchestra.run.smithi060.stdout: 410214 4 -rw-r--r-- 1 ubuntu ubuntu 1346 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/collection_manager/flat_collection_manager.h 2024-03-27T21:42:44.392 INFO:teuthology.orchestra.run.smithi060.stdout: 410211 4 -rw-r--r-- 1 ubuntu ubuntu 3070 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/collection_manager/collection_flat_node.cc 2024-03-27T21:42:44.392 INFO:teuthology.orchestra.run.smithi060.stdout: 410213 8 -rw-r--r-- 1 ubuntu ubuntu 4444 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/collection_manager/flat_collection_manager.cc 2024-03-27T21:42:44.392 INFO:teuthology.orchestra.run.smithi060.stdout: 410319 28 -rw-r--r-- 1 ubuntu ubuntu 25106 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/seastore_types.cc 2024-03-27T21:42:44.392 INFO:teuthology.orchestra.run.smithi060.stdout: 410299 8 -rw-r--r-- 1 ubuntu ubuntu 5404 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/ordering_handle.h 2024-03-27T21:42:44.392 INFO:teuthology.orchestra.run.smithi060.stdout: 410208 4 -rw-r--r-- 1 ubuntu ubuntu 494 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/collection_manager.cc 2024-03-27T21:42:44.392 INFO:teuthology.orchestra.run.smithi060.stdout: 410243 8 -rw-r--r-- 1 ubuntu ubuntu 5507 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/object_data_handler.h 2024-03-27T21:42:44.392 INFO:teuthology.orchestra.run.smithi060.stdout: 410321 4 -rw-r--r-- 1 ubuntu ubuntu 2743 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/segment_manager.cc 2024-03-27T21:42:44.393 INFO:teuthology.orchestra.run.smithi060.stdout: 410218 32 -rw-r--r-- 1 ubuntu ubuntu 30423 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/extent_placement_manager.h 2024-03-27T21:42:44.393 INFO:teuthology.orchestra.run.smithi060.stdout: 410209 4 -rw-r--r-- 1 ubuntu ubuntu 2088 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/collection_manager.h 2024-03-27T21:42:44.393 INFO:teuthology.orchestra.run.smithi060.stdout: 410332 4 -rw-r--r-- 1 ubuntu ubuntu 1222 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/segment_seq_allocator.h 2024-03-27T21:42:44.393 INFO:teuthology.orchestra.run.smithi060.stdout: 410301 8 -rw-r--r-- 1 ubuntu ubuntu 5690 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/random_block_manager.h 2024-03-27T21:42:44.393 INFO:teuthology.orchestra.run.smithi060.stdout: 410256 4 -rw-r--r-- 1 ubuntu ubuntu 3052 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/onode.h 2024-03-27T21:42:44.393 INFO:teuthology.orchestra.run.smithi060.stdout: 410207 40 -rw-r--r-- 1 ubuntu ubuntu 38985 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/cached_extent.h 2024-03-27T21:42:44.393 INFO:teuthology.orchestra.run.smithi060.stdout: 410242 56 -rw-r--r-- 1 ubuntu ubuntu 55536 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/object_data_handler.cc 2024-03-27T21:42:44.393 INFO:teuthology.orchestra.run.smithi060.stdout: 410190 48 -rw-r--r-- 1 ubuntu ubuntu 48576 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/seastore/async_cleaner.h 2024-03-27T21:42:44.393 INFO:teuthology.orchestra.run.smithi060.stdout: 410184 4 -rw-r--r-- 1 ubuntu ubuntu 915 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/futurized_collection.h 2024-03-27T21:42:44.394 INFO:teuthology.orchestra.run.smithi060.stdout: 410176 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/cyanstore 2024-03-27T21:42:44.394 INFO:teuthology.orchestra.run.smithi060.stdout: 410178 4 -rw-r--r-- 1 ubuntu ubuntu 1537 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/cyanstore/cyan_collection.cc 2024-03-27T21:42:44.394 INFO:teuthology.orchestra.run.smithi060.stdout: 410177 4 -rw-r--r-- 1 ubuntu ubuntu 156 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/cyanstore/CMakeLists.txt 2024-03-27T21:42:44.394 INFO:teuthology.orchestra.run.smithi060.stdout: 410181 4 -rw-r--r-- 1 ubuntu ubuntu 1198 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/cyanstore/cyan_object.h 2024-03-27T21:42:44.394 INFO:teuthology.orchestra.run.smithi060.stdout: 410182 28 -rw-r--r-- 1 ubuntu ubuntu 26099 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/cyanstore/cyan_store.cc 2024-03-27T21:42:44.394 INFO:teuthology.orchestra.run.smithi060.stdout: 410180 4 -rw-r--r-- 1 ubuntu ubuntu 1755 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/cyanstore/cyan_object.cc 2024-03-27T21:42:44.394 INFO:teuthology.orchestra.run.smithi060.stdout: 410179 4 -rw-r--r-- 1 ubuntu ubuntu 1446 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/cyanstore/cyan_collection.h 2024-03-27T21:42:44.394 INFO:teuthology.orchestra.run.smithi060.stdout: 410183 8 -rw-r--r-- 1 ubuntu ubuntu 6557 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/os/cyanstore/cyan_store.h 2024-03-27T21:42:44.395 INFO:teuthology.orchestra.run.smithi060.stdout: 410441 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/tools 2024-03-27T21:42:44.395 INFO:teuthology.orchestra.run.smithi060.stdout: 410445 8 -rw-r--r-- 1 ubuntu ubuntu 6016 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/tools/perf_staged_fltree.cc 2024-03-27T21:42:44.395 INFO:teuthology.orchestra.run.smithi060.stdout: 410446 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/tools/store_nbd 2024-03-27T21:42:44.395 INFO:teuthology.orchestra.run.smithi060.stdout: 410449 8 -rw-r--r-- 1 ubuntu ubuntu 7101 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/tools/store_nbd/fs_driver.cc 2024-03-27T21:42:44.395 INFO:teuthology.orchestra.run.smithi060.stdout: 410448 4 -rw-r--r-- 1 ubuntu ubuntu 3328 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/tools/store_nbd/block_driver.h 2024-03-27T21:42:44.395 INFO:teuthology.orchestra.run.smithi060.stdout: 410453 4 -rw-r--r-- 1 ubuntu ubuntu 1513 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/tools/store_nbd/tm_driver.h 2024-03-27T21:42:44.395 INFO:teuthology.orchestra.run.smithi060.stdout: 410450 4 -rw-r--r-- 1 ubuntu ubuntu 1500 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/tools/store_nbd/fs_driver.h 2024-03-27T21:42:44.395 INFO:teuthology.orchestra.run.smithi060.stdout: 410447 4 -rw-r--r-- 1 ubuntu ubuntu 502 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/tools/store_nbd/block_driver.cc 2024-03-27T21:42:44.396 INFO:teuthology.orchestra.run.smithi060.stdout: 410452 8 -rw-r--r-- 1 ubuntu ubuntu 6162 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/tools/store_nbd/tm_driver.cc 2024-03-27T21:42:44.396 INFO:teuthology.orchestra.run.smithi060.stdout: 410451 16 -rw-r--r-- 1 ubuntu ubuntu 12641 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/tools/store_nbd/store-nbd.cc 2024-03-27T21:42:44.396 INFO:teuthology.orchestra.run.smithi060.stdout: 410444 44 -rw-r--r-- 1 ubuntu ubuntu 44980 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/tools/perf_crimson_msgr.cc 2024-03-27T21:42:44.396 INFO:teuthology.orchestra.run.smithi060.stdout: 410442 4 -rw-r--r-- 1 ubuntu ubuntu 691 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/tools/CMakeLists.txt 2024-03-27T21:42:44.396 INFO:teuthology.orchestra.run.smithi060.stdout: 410443 8 -rw-r--r-- 1 ubuntu ubuntu 4761 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/tools/perf_async_msgr.cc 2024-03-27T21:42:44.396 INFO:teuthology.orchestra.run.smithi060.stdout: 410337 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd 2024-03-27T21:42:44.396 INFO:teuthology.orchestra.run.smithi060.stdout: 410416 16 -rw-r--r-- 1 ubuntu ubuntu 14999 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/pg_shard_manager.h 2024-03-27T21:42:44.396 INFO:teuthology.orchestra.run.smithi060.stdout: 410398 4 -rw-r--r-- 1 ubuntu ubuntu 2053 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osdmap_gate.h 2024-03-27T21:42:44.397 INFO:teuthology.orchestra.run.smithi060.stdout: 410404 60 -rw-r--r-- 1 ubuntu ubuntu 60941 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/pg_backend.cc 2024-03-27T21:42:44.397 INFO:teuthology.orchestra.run.smithi060.stdout: 410362 8 -rw-r--r-- 1 ubuntu ubuntu 8096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd.h 2024-03-27T21:42:44.397 INFO:teuthology.orchestra.run.smithi060.stdout: 410438 4 -rw-r--r-- 1 ubuntu ubuntu 2412 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/stop_signal.h 2024-03-27T21:42:44.397 INFO:teuthology.orchestra.run.smithi060.stdout: 410365 4 -rw-r--r-- 1 ubuntu ubuntu 2258 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_meta.h 2024-03-27T21:42:44.397 INFO:teuthology.orchestra.run.smithi060.stdout: 410344 4 -rw-r--r-- 1 ubuntu ubuntu 1215 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/ec_backend.h 2024-03-27T21:42:44.397 INFO:teuthology.orchestra.run.smithi060.stdout: 410352 16 -rw-r--r-- 1 ubuntu ubuntu 15070 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/objclass.cc 2024-03-27T21:42:44.397 INFO:teuthology.orchestra.run.smithi060.stdout: 410419 8 -rw-r--r-- 1 ubuntu ubuntu 6345 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/replicated_backend.cc 2024-03-27T21:42:44.397 INFO:teuthology.orchestra.run.smithi060.stdout: 410411 4 -rw-r--r-- 1 ubuntu ubuntu 516 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/pg_meta.h 2024-03-27T21:42:44.397 INFO:teuthology.orchestra.run.smithi060.stdout: 410338 4 -rw-r--r-- 1 ubuntu ubuntu 2238 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/CMakeLists.txt 2024-03-27T21:42:44.398 INFO:teuthology.orchestra.run.smithi060.stdout: 410401 28 -rw-r--r-- 1 ubuntu ubuntu 25077 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/pg.h 2024-03-27T21:42:44.398 INFO:teuthology.orchestra.run.smithi060.stdout: 410405 16 -rw-r--r-- 1 ubuntu ubuntu 15522 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/pg_backend.h 2024-03-27T21:42:44.398 INFO:teuthology.orchestra.run.smithi060.stdout: 410409 4 -rw-r--r-- 1 ubuntu ubuntu 3926 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/pg_map.h 2024-03-27T21:42:44.398 INFO:teuthology.orchestra.run.smithi060.stdout: 410410 4 -rw-r--r-- 1 ubuntu ubuntu 3325 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/pg_meta.cc 2024-03-27T21:42:44.398 INFO:teuthology.orchestra.run.smithi060.stdout: 410368 12 -rw-r--r-- 1 ubuntu ubuntu 11212 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operation_external_tracking.h 2024-03-27T21:42:44.398 INFO:teuthology.orchestra.run.smithi060.stdout: 410369 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations 2024-03-27T21:42:44.398 INFO:teuthology.orchestra.run.smithi060.stdout: 410389 4 -rw-r--r-- 1 ubuntu ubuntu 1662 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/recovery_subrequest.cc 2024-03-27T21:42:44.398 INFO:teuthology.orchestra.run.smithi060.stdout: 410387 4 -rw-r--r-- 1 ubuntu ubuntu 3711 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/pg_advance_map.cc 2024-03-27T21:42:44.399 INFO:teuthology.orchestra.run.smithi060.stdout: 410394 8 -rw-r--r-- 1 ubuntu ubuntu 7637 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/scrub_events.h 2024-03-27T21:42:44.399 INFO:teuthology.orchestra.run.smithi060.stdout: 410382 4 -rw-r--r-- 1 ubuntu ubuntu 2192 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/logmissing_request_reply.cc 2024-03-27T21:42:44.399 INFO:teuthology.orchestra.run.smithi060.stdout: 410378 8 -rw-r--r-- 1 ubuntu ubuntu 4361 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/internal_client_request.cc 2024-03-27T21:42:44.399 INFO:teuthology.orchestra.run.smithi060.stdout: 410396 8 -rw-r--r-- 1 ubuntu ubuntu 6311 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/snaptrim_event.h 2024-03-27T21:42:44.399 INFO:teuthology.orchestra.run.smithi060.stdout: 410383 4 -rw-r--r-- 1 ubuntu ubuntu 2694 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/logmissing_request_reply.h 2024-03-27T21:42:44.399 INFO:teuthology.orchestra.run.smithi060.stdout: 410385 8 -rw-r--r-- 1 ubuntu ubuntu 6218 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/peering_event.cc 2024-03-27T21:42:44.399 INFO:teuthology.orchestra.run.smithi060.stdout: 410393 16 -rw-r--r-- 1 ubuntu ubuntu 12739 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/scrub_events.cc 2024-03-27T21:42:44.399 INFO:teuthology.orchestra.run.smithi060.stdout: 410370 8 -rw-r--r-- 1 ubuntu ubuntu 6043 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/background_recovery.cc 2024-03-27T21:42:44.400 INFO:teuthology.orchestra.run.smithi060.stdout: 410380 4 -rw-r--r-- 1 ubuntu ubuntu 2710 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/logmissing_request.cc 2024-03-27T21:42:44.400 INFO:teuthology.orchestra.run.smithi060.stdout: 410386 8 -rw-r--r-- 1 ubuntu ubuntu 5423 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/peering_event.h 2024-03-27T21:42:44.400 INFO:teuthology.orchestra.run.smithi060.stdout: 410372 16 -rw-r--r-- 1 ubuntu ubuntu 15374 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/client_request.cc 2024-03-27T21:42:44.400 INFO:teuthology.orchestra.run.smithi060.stdout: 410391 4 -rw-r--r-- 1 ubuntu ubuntu 2577 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/replicated_request.cc 2024-03-27T21:42:44.400 INFO:teuthology.orchestra.run.smithi060.stdout: 410388 4 -rw-r--r-- 1 ubuntu ubuntu 1325 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/pg_advance_map.h 2024-03-27T21:42:44.400 INFO:teuthology.orchestra.run.smithi060.stdout: 410395 20 -rw-r--r-- 1 ubuntu ubuntu 16715 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/snaptrim_event.cc 2024-03-27T21:42:44.400 INFO:teuthology.orchestra.run.smithi060.stdout: 410375 4 -rw-r--r-- 1 ubuntu ubuntu 703 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/client_request_common.h 2024-03-27T21:42:44.400 INFO:teuthology.orchestra.run.smithi060.stdout: 410374 4 -rw-r--r-- 1 ubuntu ubuntu 3662 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/client_request_common.cc 2024-03-27T21:42:44.400 INFO:teuthology.orchestra.run.smithi060.stdout: 410384 4 -rw-r--r-- 1 ubuntu ubuntu 714 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/osdop_params.h 2024-03-27T21:42:44.401 INFO:teuthology.orchestra.run.smithi060.stdout: 410392 4 -rw-r--r-- 1 ubuntu ubuntu 2621 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/replicated_request.h 2024-03-27T21:42:44.401 INFO:teuthology.orchestra.run.smithi060.stdout: 410379 4 -rw-r--r-- 1 ubuntu ubuntu 1855 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/internal_client_request.h 2024-03-27T21:42:44.401 INFO:teuthology.orchestra.run.smithi060.stdout: 410373 12 -rw-r--r-- 1 ubuntu ubuntu 9888 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/client_request.h 2024-03-27T21:42:44.401 INFO:teuthology.orchestra.run.smithi060.stdout: 410371 4 -rw-r--r-- 1 ubuntu ubuntu 3950 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/background_recovery.h 2024-03-27T21:42:44.401 INFO:teuthology.orchestra.run.smithi060.stdout: 410390 4 -rw-r--r-- 1 ubuntu ubuntu 2550 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/recovery_subrequest.h 2024-03-27T21:42:44.401 INFO:teuthology.orchestra.run.smithi060.stdout: 410376 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/common 2024-03-27T21:42:44.401 INFO:teuthology.orchestra.run.smithi060.stdout: 410377 4 -rw-r--r-- 1 ubuntu ubuntu 988 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/common/pg_pipeline.h 2024-03-27T21:42:44.401 INFO:teuthology.orchestra.run.smithi060.stdout: 410381 4 -rw-r--r-- 1 ubuntu ubuntu 2758 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operations/logmissing_request.h 2024-03-27T21:42:44.402 INFO:teuthology.orchestra.run.smithi060.stdout: 410349 12 -rw-r--r-- 1 ubuntu ubuntu 10334 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/main.cc 2024-03-27T21:42:44.402 INFO:teuthology.orchestra.run.smithi060.stdout: 410364 8 -rw-r--r-- 1 ubuntu ubuntu 4318 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_meta.cc 2024-03-27T21:42:44.402 INFO:teuthology.orchestra.run.smithi060.stdout: 410348 4 -rw-r--r-- 1 ubuntu ubuntu 863 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/lsan_suppressions.cc 2024-03-27T21:42:44.402 INFO:teuthology.orchestra.run.smithi060.stdout: 410417 12 -rw-r--r-- 1 ubuntu ubuntu 10960 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/recovery_backend.cc 2024-03-27T21:42:44.402 INFO:teuthology.orchestra.run.smithi060.stdout: 410420 4 -rw-r--r-- 1 ubuntu ubuntu 2173 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/replicated_backend.h 2024-03-27T21:42:44.402 INFO:teuthology.orchestra.run.smithi060.stdout: 410403 4 -rw-r--r-- 1 ubuntu ubuntu 815 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/pg_activation_blocker.h 2024-03-27T21:42:44.402 INFO:teuthology.orchestra.run.smithi060.stdout: 410363 4 -rw-r--r-- 1 ubuntu ubuntu 1047 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_connection_priv.h 2024-03-27T21:42:44.402 INFO:teuthology.orchestra.run.smithi060.stdout: 410360 24 -rw-r--r-- 1 ubuntu ubuntu 20499 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/ops_executer.h 2024-03-27T21:42:44.403 INFO:teuthology.orchestra.run.smithi060.stdout: 410347 16 -rw-r--r-- 1 ubuntu ubuntu 14078 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/heartbeat.h 2024-03-27T21:42:44.403 INFO:teuthology.orchestra.run.smithi060.stdout: 410440 8 -rw-r--r-- 1 ubuntu ubuntu 7176 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/watch.h 2024-03-27T21:42:44.403 INFO:teuthology.orchestra.run.smithi060.stdout: 410342 16 -rw-r--r-- 1 ubuntu ubuntu 12813 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/backfill_state.h 2024-03-27T21:42:44.403 INFO:teuthology.orchestra.run.smithi060.stdout: 410355 12 -rw-r--r-- 1 ubuntu ubuntu 8825 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/object_context_loader.cc 2024-03-27T21:42:44.403 INFO:teuthology.orchestra.run.smithi060.stdout: 410439 12 -rw-r--r-- 1 ubuntu ubuntu 10826 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/watch.cc 2024-03-27T21:42:44.403 INFO:teuthology.orchestra.run.smithi060.stdout: 410356 4 -rw-r--r-- 1 ubuntu ubuntu 2669 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/object_context_loader.h 2024-03-27T21:42:44.403 INFO:teuthology.orchestra.run.smithi060.stdout: 410423 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/scheduler 2024-03-27T21:42:44.403 INFO:teuthology.orchestra.run.smithi060.stdout: 410425 4 -rw-r--r-- 1 ubuntu ubuntu 3071 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/scheduler/mclock_scheduler.h 2024-03-27T21:42:44.404 INFO:teuthology.orchestra.run.smithi060.stdout: 410427 4 -rw-r--r-- 1 ubuntu ubuntu 1891 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/scheduler/scheduler.h 2024-03-27T21:42:44.404 INFO:teuthology.orchestra.run.smithi060.stdout: 410424 8 -rw-r--r-- 1 ubuntu ubuntu 4857 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/scheduler/mclock_scheduler.cc 2024-03-27T21:42:44.404 INFO:teuthology.orchestra.run.smithi060.stdout: 410426 8 -rw-r--r-- 1 ubuntu ubuntu 5274 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/scheduler/scheduler.cc 2024-03-27T21:42:44.404 INFO:teuthology.orchestra.run.smithi060.stdout: 410415 4 -rw-r--r-- 1 ubuntu ubuntu 3563 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/pg_shard_manager.cc 2024-03-27T21:42:44.404 INFO:teuthology.orchestra.run.smithi060.stdout: 410436 20 -rw-r--r-- 1 ubuntu ubuntu 18735 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/shard_services.h 2024-03-27T21:42:44.404 INFO:teuthology.orchestra.run.smithi060.stdout: 410361 48 -rw-r--r-- 1 ubuntu ubuntu 47404 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd.cc 2024-03-27T21:42:44.404 INFO:teuthology.orchestra.run.smithi060.stdout: 410408 12 -rw-r--r-- 1 ubuntu ubuntu 8977 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/pg_map.cc 2024-03-27T21:42:44.404 INFO:teuthology.orchestra.run.smithi060.stdout: 410354 12 -rw-r--r-- 1 ubuntu ubuntu 8219 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/object_context.h 2024-03-27T21:42:44.405 INFO:teuthology.orchestra.run.smithi060.stdout: 410397 4 -rw-r--r-- 1 ubuntu ubuntu 2578 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osdmap_gate.cc 2024-03-27T21:42:44.405 INFO:teuthology.orchestra.run.smithi060.stdout: 410418 8 -rw-r--r-- 1 ubuntu ubuntu 7147 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/recovery_backend.h 2024-03-27T21:42:44.405 INFO:teuthology.orchestra.run.smithi060.stdout: 410399 4 -rw-r--r-- 1 ubuntu ubuntu 743 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osdmap_service.h 2024-03-27T21:42:44.405 INFO:teuthology.orchestra.run.smithi060.stdout: 410435 28 -rw-r--r-- 1 ubuntu ubuntu 28613 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/shard_services.cc 2024-03-27T21:42:44.405 INFO:teuthology.orchestra.run.smithi060.stdout: 410414 4 -rw-r--r-- 1 ubuntu ubuntu 1371 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/pg_recovery_listener.h 2024-03-27T21:42:44.405 INFO:teuthology.orchestra.run.smithi060.stdout: 410400 56 -rw-r--r-- 1 ubuntu ubuntu 53317 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/pg.cc 2024-03-27T21:42:44.405 INFO:teuthology.orchestra.run.smithi060.stdout: 410412 24 -rw-r--r-- 1 ubuntu ubuntu 20584 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/pg_recovery.cc 2024-03-27T21:42:44.405 INFO:teuthology.orchestra.run.smithi060.stdout: 410357 8 -rw-r--r-- 1 ubuntu ubuntu 6885 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/object_metadata_helper.cc 2024-03-27T21:42:44.406 INFO:teuthology.orchestra.run.smithi060.stdout: 410437 4 -rw-r--r-- 1 ubuntu ubuntu 3658 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/state.h 2024-03-27T21:42:44.406 INFO:teuthology.orchestra.run.smithi060.stdout: 410345 4 -rw-r--r-- 1 ubuntu ubuntu 1000 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/exceptions.h 2024-03-27T21:42:44.406 INFO:teuthology.orchestra.run.smithi060.stdout: 410359 52 -rw-r--r-- 1 ubuntu ubuntu 51993 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/ops_executer.cc 2024-03-27T21:42:44.406 INFO:teuthology.orchestra.run.smithi060.stdout: 410422 8 -rw-r--r-- 1 ubuntu ubuntu 6329 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/replicated_recovery_backend.h 2024-03-27T21:42:44.406 INFO:teuthology.orchestra.run.smithi060.stdout: 410421 52 -rw-r--r-- 1 ubuntu ubuntu 50218 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/replicated_recovery_backend.cc 2024-03-27T21:42:44.406 INFO:teuthology.orchestra.run.smithi060.stdout: 410428 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/scrub 2024-03-27T21:42:44.406 INFO:teuthology.orchestra.run.smithi060.stdout: 410434 8 -rw-r--r-- 1 ubuntu ubuntu 5898 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/scrub/scrub_validator.h 2024-03-27T21:42:44.406 INFO:teuthology.orchestra.run.smithi060.stdout: 410433 16 -rw-r--r-- 1 ubuntu ubuntu 13855 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/scrub/scrub_validator.cc 2024-03-27T21:42:44.407 INFO:teuthology.orchestra.run.smithi060.stdout: 410429 12 -rw-r--r-- 1 ubuntu ubuntu 8950 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/scrub/pg_scrubber.cc 2024-03-27T21:42:44.407 INFO:teuthology.orchestra.run.smithi060.stdout: 410430 4 -rw-r--r-- 1 ubuntu ubuntu 3858 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/scrub/pg_scrubber.h 2024-03-27T21:42:44.407 INFO:teuthology.orchestra.run.smithi060.stdout: 410431 4 -rw-r--r-- 1 ubuntu ubuntu 2016 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/scrub/scrub_machine.cc 2024-03-27T21:42:44.407 INFO:teuthology.orchestra.run.smithi060.stdout: 410432 20 -rw-r--r-- 1 ubuntu ubuntu 18342 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/scrub/scrub_machine.h 2024-03-27T21:42:44.407 INFO:teuthology.orchestra.run.smithi060.stdout: 410350 12 -rw-r--r-- 1 ubuntu ubuntu 8502 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/main_config_bootstrap_helpers.cc 2024-03-27T21:42:44.407 INFO:teuthology.orchestra.run.smithi060.stdout: 410341 20 -rw-r--r-- 1 ubuntu ubuntu 20220 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/backfill_state.cc 2024-03-27T21:42:44.407 INFO:teuthology.orchestra.run.smithi060.stdout: 410402 4 -rw-r--r-- 1 ubuntu ubuntu 821 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/pg_activation_blocker.cc 2024-03-27T21:42:44.407 INFO:teuthology.orchestra.run.smithi060.stdout: 410343 4 -rw-r--r-- 1 ubuntu ubuntu 1153 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/ec_backend.cc 2024-03-27T21:42:44.408 INFO:teuthology.orchestra.run.smithi060.stdout: 410339 4 -rw-r--r-- 1 ubuntu ubuntu 298 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/acked_peers.h 2024-03-27T21:42:44.408 INFO:teuthology.orchestra.run.smithi060.stdout: 410407 4 -rw-r--r-- 1 ubuntu ubuntu 1445 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/pg_interval_interrupt_condition.h 2024-03-27T21:42:44.408 INFO:teuthology.orchestra.run.smithi060.stdout: 410358 4 -rw-r--r-- 1 ubuntu ubuntu 613 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/object_metadata_helper.h 2024-03-27T21:42:44.408 INFO:teuthology.orchestra.run.smithi060.stdout: 410340 4 -rw-r--r-- 1 ubuntu ubuntu 2311 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/backfill_facades.h 2024-03-27T21:42:44.408 INFO:teuthology.orchestra.run.smithi060.stdout: 410353 4 -rw-r--r-- 1 ubuntu ubuntu 2260 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/object_context.cc 2024-03-27T21:42:44.408 INFO:teuthology.orchestra.run.smithi060.stdout: 410413 4 -rw-r--r-- 1 ubuntu ubuntu 3796 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/pg_recovery.h 2024-03-27T21:42:44.408 INFO:teuthology.orchestra.run.smithi060.stdout: 410366 8 -rw-r--r-- 1 ubuntu ubuntu 7522 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operation.cc 2024-03-27T21:42:44.408 INFO:teuthology.orchestra.run.smithi060.stdout: 410367 8 -rw-r--r-- 1 ubuntu ubuntu 8099 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/osd_operation.h 2024-03-27T21:42:44.409 INFO:teuthology.orchestra.run.smithi060.stdout: 410406 4 -rw-r--r-- 1 ubuntu ubuntu 1253 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/pg_interval_interrupt_condition.cc 2024-03-27T21:42:44.409 INFO:teuthology.orchestra.run.smithi060.stdout: 410346 24 -rw-r--r-- 1 ubuntu ubuntu 23237 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/heartbeat.cc 2024-03-27T21:42:44.409 INFO:teuthology.orchestra.run.smithi060.stdout: 410351 4 -rw-r--r-- 1 ubuntu ubuntu 2214 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/osd/main_config_bootstrap_helpers.h 2024-03-27T21:42:44.409 INFO:teuthology.orchestra.run.smithi060.stdout: 410141 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net 2024-03-27T21:42:44.409 INFO:teuthology.orchestra.run.smithi060.stdout: 410149 8 -rw-r--r-- 1 ubuntu ubuntu 6587 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/Interceptor.h 2024-03-27T21:42:44.409 INFO:teuthology.orchestra.run.smithi060.stdout: 410146 16 -rw-r--r-- 1 ubuntu ubuntu 13861 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/FrameAssemblerV2.cc 2024-03-27T21:42:44.409 INFO:teuthology.orchestra.run.smithi060.stdout: 410159 8 -rw-r--r-- 1 ubuntu ubuntu 5289 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/SocketMessenger.h 2024-03-27T21:42:44.409 INFO:teuthology.orchestra.run.smithi060.stdout: 410154 16 -rw-r--r-- 1 ubuntu ubuntu 15911 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/Socket.cc 2024-03-27T21:42:44.410 INFO:teuthology.orchestra.run.smithi060.stdout: 410158 16 -rw-r--r-- 1 ubuntu ubuntu 15763 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/SocketMessenger.cc 2024-03-27T21:42:44.410 INFO:teuthology.orchestra.run.smithi060.stdout: 410147 8 -rw-r--r-- 1 ubuntu ubuntu 6278 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/FrameAssemblerV2.h 2024-03-27T21:42:44.410 INFO:teuthology.orchestra.run.smithi060.stdout: 410162 44 -rw-r--r-- 1 ubuntu ubuntu 44285 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/io_handler.cc 2024-03-27T21:42:44.410 INFO:teuthology.orchestra.run.smithi060.stdout: 410157 8 -rw-r--r-- 1 ubuntu ubuntu 5732 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/SocketConnection.h 2024-03-27T21:42:44.410 INFO:teuthology.orchestra.run.smithi060.stdout: 410160 4 -rw-r--r-- 1 ubuntu ubuntu 3054 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/chained_dispatchers.cc 2024-03-27T21:42:44.410 INFO:teuthology.orchestra.run.smithi060.stdout: 410144 4 -rw-r--r-- 1 ubuntu ubuntu 1363 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/Errors.cc 2024-03-27T21:42:44.410 INFO:teuthology.orchestra.run.smithi060.stdout: 410153 12 -rw-r--r-- 1 ubuntu ubuntu 9115 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/ProtocolV2.h 2024-03-27T21:42:44.410 INFO:teuthology.orchestra.run.smithi060.stdout: 410150 4 -rw-r--r-- 1 ubuntu ubuntu 503 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/Messenger.cc 2024-03-27T21:42:44.411 INFO:teuthology.orchestra.run.smithi060.stdout: 410152 88 -rw-r--r-- 1 ubuntu ubuntu 89618 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/ProtocolV2.cc 2024-03-27T21:42:44.411 INFO:teuthology.orchestra.run.smithi060.stdout: 410151 4 -rw-r--r-- 1 ubuntu ubuntu 3456 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/Messenger.h 2024-03-27T21:42:44.411 INFO:teuthology.orchestra.run.smithi060.stdout: 410145 4 -rw-r--r-- 1 ubuntu ubuntu 1133 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/Errors.h 2024-03-27T21:42:44.411 INFO:teuthology.orchestra.run.smithi060.stdout: 410156 8 -rw-r--r-- 1 ubuntu ubuntu 5825 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/SocketConnection.cc 2024-03-27T21:42:44.411 INFO:teuthology.orchestra.run.smithi060.stdout: 410163 16 -rw-r--r-- 1 ubuntu ubuntu 14879 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/io_handler.h 2024-03-27T21:42:44.411 INFO:teuthology.orchestra.run.smithi060.stdout: 410161 4 -rw-r--r-- 1 ubuntu ubuntu 961 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/chained_dispatchers.h 2024-03-27T21:42:44.411 INFO:teuthology.orchestra.run.smithi060.stdout: 410148 4 -rw-r--r-- 1 ubuntu ubuntu 1583 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/Fwd.h 2024-03-27T21:42:44.411 INFO:teuthology.orchestra.run.smithi060.stdout: 410155 8 -rw-r--r-- 1 ubuntu ubuntu 5090 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/Socket.h 2024-03-27T21:42:44.411 INFO:teuthology.orchestra.run.smithi060.stdout: 410143 4 -rw-r--r-- 1 ubuntu ubuntu 2331 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/Dispatcher.h 2024-03-27T21:42:44.412 INFO:teuthology.orchestra.run.smithi060.stdout: 410142 8 -rw-r--r-- 1 ubuntu ubuntu 4589 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/net/Connection.h 2024-03-27T21:42:44.412 INFO:teuthology.orchestra.run.smithi060.stdout: 410083 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/auth 2024-03-27T21:42:44.412 INFO:teuthology.orchestra.run.smithi060.stdout: 410084 4 -rw-r--r-- 1 ubuntu ubuntu 1959 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/auth/AuthClient.h 2024-03-27T21:42:44.412 INFO:teuthology.orchestra.run.smithi060.stdout: 410088 4 -rw-r--r-- 1 ubuntu ubuntu 420 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/auth/KeyRing.h 2024-03-27T21:42:44.412 INFO:teuthology.orchestra.run.smithi060.stdout: 410087 4 -rw-r--r-- 1 ubuntu ubuntu 2455 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/auth/KeyRing.cc 2024-03-27T21:42:44.412 INFO:teuthology.orchestra.run.smithi060.stdout: 410085 4 -rw-r--r-- 1 ubuntu ubuntu 1288 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/auth/AuthServer.h 2024-03-27T21:42:44.412 INFO:teuthology.orchestra.run.smithi060.stdout: 410086 4 -rw-r--r-- 1 ubuntu ubuntu 2053 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/auth/DummyAuth.h 2024-03-27T21:42:44.412 INFO:teuthology.orchestra.run.smithi060.stdout: 410089 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common 2024-03-27T21:42:44.412 INFO:teuthology.orchestra.run.smithi060.stdout: 410114 12 -rw-r--r-- 1 ubuntu ubuntu 10044 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/logclient.cc 2024-03-27T21:42:44.413 INFO:teuthology.orchestra.run.smithi060.stdout: 410111 8 -rw-r--r-- 1 ubuntu ubuntu 7380 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/local_shared_foreign_ptr.h 2024-03-27T21:42:44.413 INFO:teuthology.orchestra.run.smithi060.stdout: 410100 4 -rw-r--r-- 1 ubuntu ubuntu 3003 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/errorator-loop.h 2024-03-27T21:42:44.413 INFO:teuthology.orchestra.run.smithi060.stdout: 410103 8 -rw-r--r-- 1 ubuntu ubuntu 5453 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/fatal_signal.cc 2024-03-27T21:42:44.413 INFO:teuthology.orchestra.run.smithi060.stdout: 410124 4 -rw-r--r-- 1 ubuntu ubuntu 1107 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/throttle.cc 2024-03-27T21:42:44.413 INFO:teuthology.orchestra.run.smithi060.stdout: 410101 52 -rw-r--r-- 1 ubuntu ubuntu 50680 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/errorator.h 2024-03-27T21:42:44.413 INFO:teuthology.orchestra.run.smithi060.stdout: 410093 4 -rw-r--r-- 1 ubuntu ubuntu 803 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/buffer_io.h 2024-03-27T21:42:44.413 INFO:teuthology.orchestra.run.smithi060.stdout: 410117 24 -rw-r--r-- 1 ubuntu ubuntu 21198 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/operation.h 2024-03-27T21:42:44.413 INFO:teuthology.orchestra.run.smithi060.stdout: 410121 4 -rw-r--r-- 1 ubuntu ubuntu 3782 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/simple_lru.h 2024-03-27T21:42:44.414 INFO:teuthology.orchestra.run.smithi060.stdout: 410097 4 -rw-r--r-- 1 ubuntu ubuntu 3238 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/config_proxy.cc 2024-03-27T21:42:44.414 INFO:teuthology.orchestra.run.smithi060.stdout: 410122 8 -rw-r--r-- 1 ubuntu ubuntu 5813 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/smp_helpers.h 2024-03-27T21:42:44.414 INFO:teuthology.orchestra.run.smithi060.stdout: 410090 4 -rw-r--r-- 1 ubuntu ubuntu 2531 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/assert.cc 2024-03-27T21:42:44.414 INFO:teuthology.orchestra.run.smithi060.stdout: 410095 4 -rw-r--r-- 1 ubuntu ubuntu 1651 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/buffer_seastar.h 2024-03-27T21:42:44.414 INFO:teuthology.orchestra.run.smithi060.stdout: 410120 8 -rw-r--r-- 1 ubuntu ubuntu 5210 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/shared_lru.h 2024-03-27T21:42:44.414 INFO:teuthology.orchestra.run.smithi060.stdout: 410105 20 -rw-r--r-- 1 ubuntu ubuntu 17059 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/fixed_kv_node_layout.h 2024-03-27T21:42:44.414 INFO:teuthology.orchestra.run.smithi060.stdout: 410106 4 -rw-r--r-- 1 ubuntu ubuntu 1099 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/formatter.cc 2024-03-27T21:42:44.414 INFO:teuthology.orchestra.run.smithi060.stdout: 410115 8 -rw-r--r-- 1 ubuntu ubuntu 6177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/logclient.h 2024-03-27T21:42:44.415 INFO:teuthology.orchestra.run.smithi060.stdout: 410126 4 -rw-r--r-- 1 ubuntu ubuntu 2583 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/tmap_helpers.cc 2024-03-27T21:42:44.415 INFO:teuthology.orchestra.run.smithi060.stdout: 410108 4 -rw-r--r-- 1 ubuntu ubuntu 1558 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/gated.h 2024-03-27T21:42:44.415 INFO:teuthology.orchestra.run.smithi060.stdout: 410119 4 -rw-r--r-- 1 ubuntu ubuntu 1591 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/perf_counters_collection.h 2024-03-27T21:42:44.415 INFO:teuthology.orchestra.run.smithi060.stdout: 410094 4 -rw-r--r-- 1 ubuntu ubuntu 3064 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/buffer_seastar.cc 2024-03-27T21:42:44.415 INFO:teuthology.orchestra.run.smithi060.stdout: 410130 4 -rw-r--r-- 1 ubuntu ubuntu 207 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/type_helpers.h 2024-03-27T21:42:44.415 INFO:teuthology.orchestra.run.smithi060.stdout: 410096 4 -rw-r--r-- 1 ubuntu ubuntu 1321 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/condition_variable.h 2024-03-27T21:42:44.415 INFO:teuthology.orchestra.run.smithi060.stdout: 410128 8 -rw-r--r-- 1 ubuntu ubuntu 4250 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/tri_mutex.cc 2024-03-27T21:42:44.415 INFO:teuthology.orchestra.run.smithi060.stdout: 410102 4 -rw-r--r-- 1 ubuntu ubuntu 1342 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/exception.h 2024-03-27T21:42:44.416 INFO:teuthology.orchestra.run.smithi060.stdout: 410104 4 -rw-r--r-- 1 ubuntu ubuntu 461 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/fatal_signal.h 2024-03-27T21:42:44.416 INFO:teuthology.orchestra.run.smithi060.stdout: 410109 60 -rw-r--r-- 1 ubuntu ubuntu 60013 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/interruptible_future.h 2024-03-27T21:42:44.416 INFO:teuthology.orchestra.run.smithi060.stdout: 410116 4 -rw-r--r-- 1 ubuntu ubuntu 1703 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/operation.cc 2024-03-27T21:42:44.416 INFO:teuthology.orchestra.run.smithi060.stdout: 410098 8 -rw-r--r-- 1 ubuntu ubuntu 8009 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/config_proxy.h 2024-03-27T21:42:44.416 INFO:teuthology.orchestra.run.smithi060.stdout: 410099 12 -rw-r--r-- 1 ubuntu ubuntu 9094 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/coroutine.h 2024-03-27T21:42:44.416 INFO:teuthology.orchestra.run.smithi060.stdout: 410131 4 -rw-r--r-- 1 ubuntu ubuntu 1050 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/utility.h 2024-03-27T21:42:44.416 INFO:teuthology.orchestra.run.smithi060.stdout: 410118 4 -rw-r--r-- 1 ubuntu ubuntu 1129 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/perf_counters_collection.cc 2024-03-27T21:42:44.416 INFO:teuthology.orchestra.run.smithi060.stdout: 410129 4 -rw-r--r-- 1 ubuntu ubuntu 3601 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/tri_mutex.h 2024-03-27T21:42:44.416 INFO:teuthology.orchestra.run.smithi060.stdout: 410112 4 -rw-r--r-- 1 ubuntu ubuntu 532 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/log.cc 2024-03-27T21:42:44.417 INFO:teuthology.orchestra.run.smithi060.stdout: 410127 4 -rw-r--r-- 1 ubuntu ubuntu 1186 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/tmap_helpers.h 2024-03-27T21:42:44.417 INFO:teuthology.orchestra.run.smithi060.stdout: 410123 4 -rw-r--r-- 1 ubuntu ubuntu 1285 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/subop_blocker.h 2024-03-27T21:42:44.417 INFO:teuthology.orchestra.run.smithi060.stdout: 410092 4 -rw-r--r-- 1 ubuntu ubuntu 1855 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/buffer_io.cc 2024-03-27T21:42:44.417 INFO:teuthology.orchestra.run.smithi060.stdout: 410107 4 -rw-r--r-- 1 ubuntu ubuntu 297 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/formatter.h 2024-03-27T21:42:44.417 INFO:teuthology.orchestra.run.smithi060.stdout: 410125 4 -rw-r--r-- 1 ubuntu ubuntu 1097 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/throttle.h 2024-03-27T21:42:44.417 INFO:teuthology.orchestra.run.smithi060.stdout: 410091 4 -rw-r--r-- 1 ubuntu ubuntu 384 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/auth_handler.h 2024-03-27T21:42:44.417 INFO:teuthology.orchestra.run.smithi060.stdout: 410110 28 -rw-r--r-- 1 ubuntu ubuntu 26948 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/layout.h 2024-03-27T21:42:44.417 INFO:teuthology.orchestra.run.smithi060.stdout: 410113 8 -rw-r--r-- 1 ubuntu ubuntu 5863 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/crimson/common/log.h 2024-03-27T21:42:44.418 INFO:teuthology.orchestra.run.smithi060.stdout: 414675 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common 2024-03-27T21:42:44.418 INFO:teuthology.orchestra.run.smithi060.stdout: 414677 4 -rw-r--r-- 1 ubuntu ubuntu 145 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/CMakeLists.txt 2024-03-27T21:42:44.418 INFO:teuthology.orchestra.run.smithi060.stdout: 414714 4 -rw-r--r-- 1 ubuntu ubuntu 30 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/requirements-lint.txt 2024-03-27T21:42:44.418 INFO:teuthology.orchestra.run.smithi060.stdout: 414717 4 -rw-r--r-- 1 ubuntu ubuntu 637 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/tox.ini 2024-03-27T21:42:44.418 INFO:teuthology.orchestra.run.smithi060.stdout: 414678 4 -rw-r--r-- 1 ubuntu ubuntu 468 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/README.rst 2024-03-27T21:42:44.418 INFO:teuthology.orchestra.run.smithi060.stdout: 414715 4 -rw-r--r-- 1 ubuntu ubuntu 235 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/requirements.txt 2024-03-27T21:42:44.418 INFO:teuthology.orchestra.run.smithi060.stdout: 414679 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph 2024-03-27T21:42:44.418 INFO:teuthology.orchestra.run.smithi060.stdout: 414701 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/tests 2024-03-27T21:42:44.418 INFO:teuthology.orchestra.run.smithi060.stdout: 414704 4 -rw-r--r-- 1 ubuntu ubuntu 3626 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/tests/factories.py 2024-03-27T21:42:44.419 INFO:teuthology.orchestra.run.smithi060.stdout: 414708 4 -rw-r--r-- 1 ubuntu ubuntu 1239 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/tests/test_hostspec.py 2024-03-27T21:42:44.419 INFO:teuthology.orchestra.run.smithi060.stdout: 414710 40 -rw-r--r-- 1 ubuntu ubuntu 38031 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/tests/test_service_spec.py 2024-03-27T21:42:44.419 INFO:teuthology.orchestra.run.smithi060.stdout: 414703 4 -rw-r--r-- 1 ubuntu ubuntu 3327 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/tests/c-v-inventory.json 2024-03-27T21:42:44.419 INFO:teuthology.orchestra.run.smithi060.stdout: 414706 24 -rw-r--r-- 1 ubuntu ubuntu 21500 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/tests/test_disk_selector.py 2024-03-27T21:42:44.419 INFO:teuthology.orchestra.run.smithi060.stdout: 414711 4 -rw-r--r-- 1 ubuntu ubuntu 2957 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/tests/test_utils.py 2024-03-27T21:42:44.419 INFO:teuthology.orchestra.run.smithi060.stdout: 414705 4 -rw-r--r-- 1 ubuntu ubuntu 1744 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/tests/test_datetime.py 2024-03-27T21:42:44.419 INFO:teuthology.orchestra.run.smithi060.stdout: 414707 24 -rw-r--r-- 1 ubuntu ubuntu 21033 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/tests/test_drive_group.py 2024-03-27T21:42:44.419 INFO:teuthology.orchestra.run.smithi060.stdout: 414702 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/tests/__init__.py 2024-03-27T21:42:44.420 INFO:teuthology.orchestra.run.smithi060.stdout: 414712 4 -rw-r--r-- 1 ubuntu ubuntu 1359 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/tests/utils.py 2024-03-27T21:42:44.420 INFO:teuthology.orchestra.run.smithi060.stdout: 414709 4 -rw-r--r-- 1 ubuntu ubuntu 3427 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/tests/test_inventory.py 2024-03-27T21:42:44.420 INFO:teuthology.orchestra.run.smithi060.stdout: 414696 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/rgw 2024-03-27T21:42:44.420 INFO:teuthology.orchestra.run.smithi060.stdout: 414698 4 -rw-r--r-- 1 ubuntu ubuntu 1898 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/rgw/diff.py 2024-03-27T21:42:44.420 INFO:teuthology.orchestra.run.smithi060.stdout: 414700 8 -rw-r--r-- 1 ubuntu ubuntu 5371 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/rgw/types.py 2024-03-27T21:42:44.420 INFO:teuthology.orchestra.run.smithi060.stdout: 414697 4 -rw-r--r-- 1 ubuntu ubuntu 50 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/rgw/__init__.py 2024-03-27T21:42:44.420 INFO:teuthology.orchestra.run.smithi060.stdout: 414699 36 -rw-r--r-- 1 ubuntu ubuntu 34361 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/rgw/rgwam_core.py 2024-03-27T21:42:44.420 INFO:teuthology.orchestra.run.smithi060.stdout: 414680 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/__init__.py 2024-03-27T21:42:44.421 INFO:teuthology.orchestra.run.smithi060.stdout: 414695 4 -rw-r--r-- 1 ubuntu ubuntu 60 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/py.typed 2024-03-27T21:42:44.421 INFO:teuthology.orchestra.run.smithi060.stdout: 414713 8 -rw-r--r-- 1 ubuntu ubuntu 5249 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/utils.py 2024-03-27T21:42:44.421 INFO:teuthology.orchestra.run.smithi060.stdout: 414681 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/deployment 2024-03-27T21:42:44.421 INFO:teuthology.orchestra.run.smithi060.stdout: 414693 8 -rw-r--r-- 1 ubuntu ubuntu 5381 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/deployment/translate.py 2024-03-27T21:42:44.421 INFO:teuthology.orchestra.run.smithi060.stdout: 414683 16 -rw-r--r-- 1 ubuntu ubuntu 15399 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/deployment/drive_group.py 2024-03-27T21:42:44.421 INFO:teuthology.orchestra.run.smithi060.stdout: 414682 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/deployment/__init__.py 2024-03-27T21:42:44.421 INFO:teuthology.orchestra.run.smithi060.stdout: 414684 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/deployment/drive_selection 2024-03-27T21:42:44.421 INFO:teuthology.orchestra.run.smithi060.stdout: 414687 4 -rw-r--r-- 1 ubuntu ubuntu 1234 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/deployment/drive_selection/filter.py 2024-03-27T21:42:44.422 INFO:teuthology.orchestra.run.smithi060.stdout: 414689 8 -rw-r--r-- 1 ubuntu ubuntu 7446 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/deployment/drive_selection/selector.py 2024-03-27T21:42:44.422 INFO:teuthology.orchestra.run.smithi060.stdout: 414688 16 -rw-r--r-- 1 ubuntu ubuntu 13115 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/deployment/drive_selection/matchers.py 2024-03-27T21:42:44.422 INFO:teuthology.orchestra.run.smithi060.stdout: 414685 4 -rw-r--r-- 1 ubuntu ubuntu 143 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/deployment/drive_selection/__init__.py 2024-03-27T21:42:44.422 INFO:teuthology.orchestra.run.smithi060.stdout: 414686 4 -rw-r--r-- 1 ubuntu ubuntu 394 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/deployment/drive_selection/example.yaml 2024-03-27T21:42:44.422 INFO:teuthology.orchestra.run.smithi060.stdout: 414692 104 -rw-r--r-- 1 ubuntu ubuntu 102556 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/deployment/service_spec.py 2024-03-27T21:42:44.422 INFO:teuthology.orchestra.run.smithi060.stdout: 414690 8 -rw-r--r-- 1 ubuntu ubuntu 5213 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/deployment/hostspec.py 2024-03-27T21:42:44.422 INFO:teuthology.orchestra.run.smithi060.stdout: 414691 8 -rw-r--r-- 1 ubuntu ubuntu 4965 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/deployment/inventory.py 2024-03-27T21:42:44.422 INFO:teuthology.orchestra.run.smithi060.stdout: 414694 4 -rw-r--r-- 1 ubuntu ubuntu 3124 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/ceph/deployment/utils.py 2024-03-27T21:42:44.423 INFO:teuthology.orchestra.run.smithi060.stdout: 414676 4 -rw-r--r-- 1 ubuntu ubuntu 30 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/.gitignore 2024-03-27T21:42:44.423 INFO:teuthology.orchestra.run.smithi060.stdout: 414716 4 -rw-r--r-- 1 ubuntu ubuntu 872 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/python-common/setup.py 2024-03-27T21:42:44.423 INFO:teuthology.orchestra.run.smithi060.stdout: 412061 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd 2024-03-27T21:42:44.423 INFO:teuthology.orchestra.run.smithi060.stdout: 412085 8 -rw-r--r-- 1 ubuntu ubuntu 8135 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/OSDCap.h 2024-03-27T21:42:44.423 INFO:teuthology.orchestra.run.smithi060.stdout: 412117 8 -rw-r--r-- 1 ubuntu ubuntu 7546 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/Watch.h 2024-03-27T21:42:44.423 INFO:teuthology.orchestra.run.smithi060.stdout: 412073 8 -rw-r--r-- 1 ubuntu ubuntu 6138 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/ECTransaction.h 2024-03-27T21:42:44.423 INFO:teuthology.orchestra.run.smithi060.stdout: 412101 4 -rw-r--r-- 1 ubuntu ubuntu 1469 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/PGStateUtils.cc 2024-03-27T21:42:44.423 INFO:teuthology.orchestra.run.smithi060.stdout: 412098 56 -rw-r--r-- 1 ubuntu ubuntu 53934 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/PGLog.h 2024-03-27T21:42:44.424 INFO:teuthology.orchestra.run.smithi060.stdout: 412076 8 -rw-r--r-- 1 ubuntu ubuntu 6010 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/ExtentCache.cc 2024-03-27T21:42:44.424 INFO:teuthology.orchestra.run.smithi060.stdout: 412063 12 -rw-r--r-- 1 ubuntu ubuntu 10011 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/ClassHandler.cc 2024-03-27T21:42:44.424 INFO:teuthology.orchestra.run.smithi060.stdout: 412114 16 -rw-r--r-- 1 ubuntu ubuntu 12701 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/SnapMapper.h 2024-03-27T21:42:44.424 INFO:teuthology.orchestra.run.smithi060.stdout: 412110 4 -rw-r--r-- 1 ubuntu ubuntu 2776 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/Session.cc 2024-03-27T21:42:44.424 INFO:teuthology.orchestra.run.smithi060.stdout: 412103 20 -rw-r--r-- 1 ubuntu ubuntu 18601 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/PGTransaction.h 2024-03-27T21:42:44.424 INFO:teuthology.orchestra.run.smithi060.stdout: 412142 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber 2024-03-27T21:42:44.424 INFO:teuthology.orchestra.run.smithi060.stdout: 412150 16 -rw-r--r-- 1 ubuntu ubuntu 13199 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber/osd_scrub_sched.h 2024-03-27T21:42:44.424 INFO:teuthology.orchestra.run.smithi060.stdout: 412163 4 -rw-r--r-- 1 ubuntu ubuntu 2543 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber/scrub_resources.h 2024-03-27T21:42:44.424 INFO:teuthology.orchestra.run.smithi060.stdout: 412160 12 -rw-r--r-- 1 ubuntu ubuntu 9208 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber/scrub_reservations.cc 2024-03-27T21:42:44.425 INFO:teuthology.orchestra.run.smithi060.stdout: 412143 12 -rw-r--r-- 1 ubuntu ubuntu 8804 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber/PrimaryLogScrub.cc 2024-03-27T21:42:44.425 INFO:teuthology.orchestra.run.smithi060.stdout: 412149 12 -rw-r--r-- 1 ubuntu ubuntu 11690 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber/osd_scrub_sched.cc 2024-03-27T21:42:44.425 INFO:teuthology.orchestra.run.smithi060.stdout: 412155 4 -rw-r--r-- 1 ubuntu ubuntu 3741 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber/scrub_job.cc 2024-03-27T21:42:44.425 INFO:teuthology.orchestra.run.smithi060.stdout: 412159 12 -rw-r--r-- 1 ubuntu ubuntu 8307 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber/scrub_machine_lstnr.h 2024-03-27T21:42:44.425 INFO:teuthology.orchestra.run.smithi060.stdout: 412146 4 -rw-r--r-- 1 ubuntu ubuntu 1838 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber/ScrubStore.h 2024-03-27T21:42:44.425 INFO:teuthology.orchestra.run.smithi060.stdout: 412151 76 -rw-r--r-- 1 ubuntu ubuntu 75002 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber/pg_scrubber.cc 2024-03-27T21:42:44.425 INFO:teuthology.orchestra.run.smithi060.stdout: 412161 8 -rw-r--r-- 1 ubuntu ubuntu 6990 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber/scrub_reservations.h 2024-03-27T21:42:44.425 INFO:teuthology.orchestra.run.smithi060.stdout: 412152 32 -rw-r--r-- 1 ubuntu ubuntu 30336 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber/pg_scrubber.h 2024-03-27T21:42:44.426 INFO:teuthology.orchestra.run.smithi060.stdout: 412156 12 -rw-r--r-- 1 ubuntu ubuntu 8478 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber/scrub_job.h 2024-03-27T21:42:44.426 INFO:teuthology.orchestra.run.smithi060.stdout: 412144 4 -rw-r--r-- 1 ubuntu ubuntu 1360 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber/PrimaryLogScrub.h 2024-03-27T21:42:44.426 INFO:teuthology.orchestra.run.smithi060.stdout: 412148 12 -rw-r--r-- 1 ubuntu ubuntu 9982 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber/osd_scrub.h 2024-03-27T21:42:44.426 INFO:teuthology.orchestra.run.smithi060.stdout: 412153 64 -rw-r--r-- 1 ubuntu ubuntu 63722 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber/scrub_backend.cc 2024-03-27T21:42:44.426 INFO:teuthology.orchestra.run.smithi060.stdout: 412154 20 -rw-r--r-- 1 ubuntu ubuntu 18408 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber/scrub_backend.h 2024-03-27T21:42:44.426 INFO:teuthology.orchestra.run.smithi060.stdout: 412145 8 -rw-r--r-- 1 ubuntu ubuntu 4442 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber/ScrubStore.cc 2024-03-27T21:42:44.426 INFO:teuthology.orchestra.run.smithi060.stdout: 412162 4 -rw-r--r-- 1 ubuntu ubuntu 2415 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber/scrub_resources.cc 2024-03-27T21:42:44.426 INFO:teuthology.orchestra.run.smithi060.stdout: 412157 40 -rw-r--r-- 1 ubuntu ubuntu 40197 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber/scrub_machine.cc 2024-03-27T21:42:44.427 INFO:teuthology.orchestra.run.smithi060.stdout: 412147 16 -rw-r--r-- 1 ubuntu ubuntu 14703 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber/osd_scrub.cc 2024-03-27T21:42:44.427 INFO:teuthology.orchestra.run.smithi060.stdout: 412158 36 -rw-r--r-- 1 ubuntu ubuntu 36281 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber/scrub_machine.h 2024-03-27T21:42:44.427 INFO:teuthology.orchestra.run.smithi060.stdout: 412116 16 -rw-r--r-- 1 ubuntu ubuntu 12915 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/Watch.cc 2024-03-27T21:42:44.427 INFO:teuthology.orchestra.run.smithi060.stdout: 412084 20 -rw-r--r-- 1 ubuntu ubuntu 17489 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/OSDCap.cc 2024-03-27T21:42:44.427 INFO:teuthology.orchestra.run.smithi060.stdout: 412088 8 -rw-r--r-- 1 ubuntu ubuntu 5189 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/OSDMapMapping.cc 2024-03-27T21:42:44.427 INFO:teuthology.orchestra.run.smithi060.stdout: 412105 88 -rw-r--r-- 1 ubuntu ubuntu 86318 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/PeeringState.h 2024-03-27T21:42:44.427 INFO:teuthology.orchestra.run.smithi060.stdout: 412120 24 -rw-r--r-- 1 ubuntu ubuntu 21112 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/objclass.cc 2024-03-27T21:42:44.427 INFO:teuthology.orchestra.run.smithi060.stdout: 412164 20 -rw-r--r-- 1 ubuntu ubuntu 17007 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scrubber_common.h 2024-03-27T21:42:44.428 INFO:teuthology.orchestra.run.smithi060.stdout: 412062 4 -rw-r--r-- 1 ubuntu ubuntu 2210 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/CMakeLists.txt 2024-03-27T21:42:44.428 INFO:teuthology.orchestra.run.smithi060.stdout: 412123 12 -rw-r--r-- 1 ubuntu ubuntu 8217 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/osd_internal_types.h 2024-03-27T21:42:44.428 INFO:teuthology.orchestra.run.smithi060.stdout: 412112 4 -rw-r--r-- 1 ubuntu ubuntu 2119 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/SnapMapReaderI.h 2024-03-27T21:42:44.428 INFO:teuthology.orchestra.run.smithi060.stdout: 412128 4 -rw-r--r-- 1 ubuntu ubuntu 228 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/osd_tracer.cc 2024-03-27T21:42:44.428 INFO:teuthology.orchestra.run.smithi060.stdout: 412106 476 -rw-r--r-- 1 ubuntu ubuntu 485520 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/PrimaryLogPG.cc 2024-03-27T21:42:44.428 INFO:teuthology.orchestra.run.smithi060.stdout: 412080 8 -rw-r--r-- 1 ubuntu ubuntu 6135 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/MissingLoc.cc 2024-03-27T21:42:44.428 INFO:teuthology.orchestra.run.smithi060.stdout: 412130 212 -rw-r--r-- 1 ubuntu ubuntu 213110 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/osd_types.cc 2024-03-27T21:42:44.428 INFO:teuthology.orchestra.run.smithi060.stdout: 412078 8 -rw-r--r-- 1 ubuntu ubuntu 6492 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/HitSet.cc 2024-03-27T21:42:44.429 INFO:teuthology.orchestra.run.smithi060.stdout: 412125 4 -rw-r--r-- 1 ubuntu ubuntu 2115 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/osd_op_util.h 2024-03-27T21:42:44.429 INFO:teuthology.orchestra.run.smithi060.stdout: 412131 212 -rw-r--r-- 1 ubuntu ubuntu 213762 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/osd_types.h 2024-03-27T21:42:44.429 INFO:teuthology.orchestra.run.smithi060.stdout: 412082 348 -rw-r--r-- 1 ubuntu ubuntu 352714 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/OSD.cc 2024-03-27T21:42:44.429 INFO:teuthology.orchestra.run.smithi060.stdout: 412089 8 -rw-r--r-- 1 ubuntu ubuntu 8140 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/OSDMapMapping.h 2024-03-27T21:42:44.429 INFO:teuthology.orchestra.run.smithi060.stdout: 412111 8 -rw-r--r-- 1 ubuntu ubuntu 6615 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/Session.h 2024-03-27T21:42:44.429 INFO:teuthology.orchestra.run.smithi060.stdout: 412109 16 -rw-r--r-- 1 ubuntu ubuntu 12368 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/ReplicatedBackend.h 2024-03-27T21:42:44.429 INFO:teuthology.orchestra.run.smithi060.stdout: 412065 12 -rw-r--r-- 1 ubuntu ubuntu 8446 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/DynamicPerfStats.h 2024-03-27T21:42:44.429 INFO:teuthology.orchestra.run.smithi060.stdout: 412075 8 -rw-r--r-- 1 ubuntu ubuntu 5509 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/ECUtil.h 2024-03-27T21:42:44.429 INFO:teuthology.orchestra.run.smithi060.stdout: 412079 16 -rw-r--r-- 1 ubuntu ubuntu 12721 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/HitSet.h 2024-03-27T21:42:44.430 INFO:teuthology.orchestra.run.smithi060.stdout: 412135 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scheduler 2024-03-27T21:42:44.430 INFO:teuthology.orchestra.run.smithi060.stdout: 412140 20 -rw-r--r-- 1 ubuntu ubuntu 18762 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scheduler/mClockScheduler.cc 2024-03-27T21:42:44.430 INFO:teuthology.orchestra.run.smithi060.stdout: 412137 4 -rw-r--r-- 1 ubuntu ubuntu 3870 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scheduler/OpScheduler.h 2024-03-27T21:42:44.430 INFO:teuthology.orchestra.run.smithi060.stdout: 412136 4 -rw-r--r-- 1 ubuntu ubuntu 1682 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scheduler/OpScheduler.cc 2024-03-27T21:42:44.430 INFO:teuthology.orchestra.run.smithi060.stdout: 412138 8 -rw-r--r-- 1 ubuntu ubuntu 5614 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scheduler/OpSchedulerItem.cc 2024-03-27T21:42:44.430 INFO:teuthology.orchestra.run.smithi060.stdout: 412141 12 -rw-r--r-- 1 ubuntu ubuntu 8345 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scheduler/mClockScheduler.h 2024-03-27T21:42:44.430 INFO:teuthology.orchestra.run.smithi060.stdout: 412139 20 -rw-r--r-- 1 ubuntu ubuntu 18874 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/scheduler/OpSchedulerItem.h 2024-03-27T21:42:44.431 INFO:teuthology.orchestra.run.smithi060.stdout: 412092 8 -rw-r--r-- 1 ubuntu ubuntu 5807 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/OpRequest.h 2024-03-27T21:42:44.431 INFO:teuthology.orchestra.run.smithi060.stdout: 412067 16 -rw-r--r-- 1 ubuntu ubuntu 13654 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/ECBackend.h 2024-03-27T21:42:44.431 INFO:teuthology.orchestra.run.smithi060.stdout: 412133 4 -rw-r--r-- 1 ubuntu ubuntu 394 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/recovery_types.cc 2024-03-27T21:42:44.431 INFO:teuthology.orchestra.run.smithi060.stdout: 412132 12 -rw-r--r-- 1 ubuntu ubuntu 10973 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/osd_types_fmt.h 2024-03-27T21:42:44.431 INFO:teuthology.orchestra.run.smithi060.stdout: 412104 232 -rw-r--r-- 1 ubuntu ubuntu 237456 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/PeeringState.cc 2024-03-27T21:42:44.431 INFO:teuthology.orchestra.run.smithi060.stdout: 412074 8 -rw-r--r-- 1 ubuntu ubuntu 6782 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/ECUtil.cc 2024-03-27T21:42:44.431 INFO:teuthology.orchestra.run.smithi060.stdout: 412100 8 -rw-r--r-- 1 ubuntu ubuntu 6510 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/PGPeeringEvent.h 2024-03-27T21:42:44.431 INFO:teuthology.orchestra.run.smithi060.stdout: 412066 60 -rw-r--r-- 1 ubuntu ubuntu 57847 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/ECBackend.cc 2024-03-27T21:42:44.431 INFO:teuthology.orchestra.run.smithi060.stdout: 412124 12 -rw-r--r-- 1 ubuntu ubuntu 8816 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/osd_op_util.cc 2024-03-27T21:42:44.432 INFO:teuthology.orchestra.run.smithi060.stdout: 412129 4 -rw-r--r-- 1 ubuntu ubuntu 251 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/osd_tracer.h 2024-03-27T21:42:44.432 INFO:teuthology.orchestra.run.smithi060.stdout: 412108 76 -rw-r--r-- 1 ubuntu ubuntu 74336 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/ReplicatedBackend.cc 2024-03-27T21:42:44.432 INFO:teuthology.orchestra.run.smithi060.stdout: 412122 4 -rw-r--r-- 1 ubuntu ubuntu 600 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/object_state_fmt.h 2024-03-27T21:42:44.432 INFO:teuthology.orchestra.run.smithi060.stdout: 412096 20 -rw-r--r-- 1 ubuntu ubuntu 18704 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/PGBackend.h 2024-03-27T21:42:44.432 INFO:teuthology.orchestra.run.smithi060.stdout: 412127 8 -rw-r--r-- 1 ubuntu ubuntu 5373 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/osd_perf_counters.h 2024-03-27T21:42:44.432 INFO:teuthology.orchestra.run.smithi060.stdout: 412091 8 -rw-r--r-- 1 ubuntu ubuntu 4532 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/OpRequest.cc 2024-03-27T21:42:44.432 INFO:teuthology.orchestra.run.smithi060.stdout: 412121 4 -rw-r--r-- 1 ubuntu ubuntu 4084 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/object_state.h 2024-03-27T21:42:44.432 INFO:teuthology.orchestra.run.smithi060.stdout: 412064 4 -rw-r--r-- 1 ubuntu ubuntu 3455 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/ClassHandler.h 2024-03-27T21:42:44.432 INFO:teuthology.orchestra.run.smithi060.stdout: 412087 60 -rw-r--r-- 1 ubuntu ubuntu 59846 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/OSDMap.h 2024-03-27T21:42:44.433 INFO:teuthology.orchestra.run.smithi060.stdout: 412099 4 -rw-r--r-- 1 ubuntu ubuntu 458 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/PGPeeringEvent.cc 2024-03-27T21:42:44.433 INFO:teuthology.orchestra.run.smithi060.stdout: 412118 4 -rw-r--r-- 1 ubuntu ubuntu 3304 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/error_code.cc 2024-03-27T21:42:44.433 INFO:teuthology.orchestra.run.smithi060.stdout: 412068 32 -rw-r--r-- 1 ubuntu ubuntu 28906 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/ECCommon.cc 2024-03-27T21:42:44.433 INFO:teuthology.orchestra.run.smithi060.stdout: 412107 68 -rw-r--r-- 1 ubuntu ubuntu 65904 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/PrimaryLogPG.h 2024-03-27T21:42:44.433 INFO:teuthology.orchestra.run.smithi060.stdout: 412093 84 -rw-r--r-- 1 ubuntu ubuntu 83302 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/PG.cc 2024-03-27T21:42:44.433 INFO:teuthology.orchestra.run.smithi060.stdout: 412095 20 -rw-r--r-- 1 ubuntu ubuntu 17716 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/PGBackend.cc 2024-03-27T21:42:44.433 INFO:teuthology.orchestra.run.smithi060.stdout: 412070 12 -rw-r--r-- 1 ubuntu ubuntu 11275 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/ECMsgTypes.cc 2024-03-27T21:42:44.433 INFO:teuthology.orchestra.run.smithi060.stdout: 412126 24 -rw-r--r-- 1 ubuntu ubuntu 21052 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/osd_perf_counters.cc 2024-03-27T21:42:44.434 INFO:teuthology.orchestra.run.smithi060.stdout: 412081 12 -rw-r--r-- 1 ubuntu ubuntu 10587 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/MissingLoc.h 2024-03-27T21:42:44.434 INFO:teuthology.orchestra.run.smithi060.stdout: 412094 44 -rw-r--r-- 1 ubuntu ubuntu 44975 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/PG.h 2024-03-27T21:42:44.434 INFO:teuthology.orchestra.run.smithi060.stdout: 412069 28 -rw-r--r-- 1 ubuntu ubuntu 26621 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/ECCommon.h 2024-03-27T21:42:44.434 INFO:teuthology.orchestra.run.smithi060.stdout: 412077 16 -rw-r--r-- 1 ubuntu ubuntu 14556 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/ExtentCache.h 2024-03-27T21:42:44.434 INFO:teuthology.orchestra.run.smithi060.stdout: 412097 40 -rw-r--r-- 1 ubuntu ubuntu 40687 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/PGLog.cc 2024-03-27T21:42:44.434 INFO:teuthology.orchestra.run.smithi060.stdout: 412086 236 -rw-r--r-- 1 ubuntu ubuntu 239133 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/OSDMap.cc 2024-03-27T21:42:44.434 INFO:teuthology.orchestra.run.smithi060.stdout: 412083 64 -rw-r--r-- 1 ubuntu ubuntu 62758 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/OSD.h 2024-03-27T21:42:44.434 INFO:teuthology.orchestra.run.smithi060.stdout: 412113 32 -rw-r--r-- 1 ubuntu ubuntu 30545 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/SnapMapper.cc 2024-03-27T21:42:44.435 INFO:teuthology.orchestra.run.smithi060.stdout: 412071 8 -rw-r--r-- 1 ubuntu ubuntu 4930 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/ECMsgTypes.h 2024-03-27T21:42:44.435 INFO:teuthology.orchestra.run.smithi060.stdout: 412102 4 -rw-r--r-- 1 ubuntu ubuntu 2007 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/PGStateUtils.h 2024-03-27T21:42:44.435 INFO:teuthology.orchestra.run.smithi060.stdout: 412090 4 -rw-r--r-- 1 ubuntu ubuntu 914 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/ObjectVersioner.h 2024-03-27T21:42:44.435 INFO:teuthology.orchestra.run.smithi060.stdout: 412119 4 -rw-r--r-- 1 ubuntu ubuntu 1499 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/error_code.h 2024-03-27T21:42:44.435 INFO:teuthology.orchestra.run.smithi060.stdout: 412072 20 -rw-r--r-- 1 ubuntu ubuntu 18726 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/ECTransaction.cc 2024-03-27T21:42:44.435 INFO:teuthology.orchestra.run.smithi060.stdout: 412134 4 -rw-r--r-- 1 ubuntu ubuntu 2343 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/recovery_types.h 2024-03-27T21:42:44.435 INFO:teuthology.orchestra.run.smithi060.stdout: 412115 4 -rw-r--r-- 1 ubuntu ubuntu 3480 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osd/TierAgentState.h 2024-03-27T21:42:44.435 INFO:teuthology.orchestra.run.smithi060.stdout: 410892 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/libradosstriper 2024-03-27T21:42:44.436 INFO:teuthology.orchestra.run.smithi060.stdout: 410893 4 -rw-r--r-- 1 ubuntu ubuntu 484 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/libradosstriper/CMakeLists.txt 2024-03-27T21:42:44.436 INFO:teuthology.orchestra.run.smithi060.stdout: 410894 4 -rw-r--r-- 1 ubuntu ubuntu 1347 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/libradosstriper/MultiAioCompletionImpl.cc 2024-03-27T21:42:44.436 INFO:teuthology.orchestra.run.smithi060.stdout: 410895 8 -rw-r--r-- 1 ubuntu ubuntu 4333 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/libradosstriper/MultiAioCompletionImpl.h 2024-03-27T21:42:44.436 INFO:teuthology.orchestra.run.smithi060.stdout: 410897 12 -rw-r--r-- 1 ubuntu ubuntu 9688 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/libradosstriper/RadosStriperImpl.h 2024-03-27T21:42:44.436 INFO:teuthology.orchestra.run.smithi060.stdout: 410898 20 -rw-r--r-- 1 ubuntu ubuntu 20082 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/libradosstriper/libradosstriper.cc 2024-03-27T21:42:44.436 INFO:teuthology.orchestra.run.smithi060.stdout: 410896 60 -rw-r--r-- 1 ubuntu ubuntu 58551 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/libradosstriper/RadosStriperImpl.cc 2024-03-27T21:42:44.436 INFO:teuthology.orchestra.run.smithi060.stdout: 409380 48 -rwxr-xr-x 1 ubuntu ubuntu 46913 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph.in 2024-03-27T21:42:44.436 INFO:teuthology.orchestra.run.smithi060.stdout: 410762 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/jaegertracing 2024-03-27T21:42:44.437 INFO:teuthology.orchestra.run.smithi060.stdout: 410764 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/jaegertracing/opentelemetry-cpp 2024-03-27T21:42:44.437 INFO:teuthology.orchestra.run.smithi060.stdout: 410763 4 -rw-r--r-- 1 ubuntu ubuntu 79 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/jaegertracing/config.yml 2024-03-27T21:42:44.437 INFO:teuthology.orchestra.run.smithi060.stdout: 410521 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/dokan 2024-03-27T21:42:44.437 INFO:teuthology.orchestra.run.smithi060.stdout: 410529 4 -rw-r--r-- 1 ubuntu ubuntu 446 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/dokan/utils.h 2024-03-27T21:42:44.437 INFO:teuthology.orchestra.run.smithi060.stdout: 410525 8 -rw-r--r-- 1 ubuntu ubuntu 6880 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/dokan/dbg.cc 2024-03-27T21:42:44.437 INFO:teuthology.orchestra.run.smithi060.stdout: 410522 4 -rw-r--r-- 1 ubuntu ubuntu 394 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/dokan/CMakeLists.txt 2024-03-27T21:42:44.437 INFO:teuthology.orchestra.run.smithi060.stdout: 410526 4 -rw-r--r-- 1 ubuntu ubuntu 614 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/dokan/dbg.h 2024-03-27T21:42:44.437 INFO:teuthology.orchestra.run.smithi060.stdout: 410527 12 -rw-r--r-- 1 ubuntu ubuntu 10063 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/dokan/options.cc 2024-03-27T21:42:44.437 INFO:teuthology.orchestra.run.smithi060.stdout: 410524 4 -rw-r--r-- 1 ubuntu ubuntu 1642 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/dokan/ceph_dokan.h 2024-03-27T21:42:44.438 INFO:teuthology.orchestra.run.smithi060.stdout: 410523 36 -rw-r--r-- 1 ubuntu ubuntu 33700 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/dokan/ceph_dokan.cc 2024-03-27T21:42:44.438 INFO:teuthology.orchestra.run.smithi060.stdout: 410528 4 -rw-r--r-- 1 ubuntu ubuntu 758 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/dokan/utils.cc 2024-03-27T21:42:44.438 INFO:teuthology.orchestra.run.smithi060.stdout: 409070 4 -rw-r--r-- 1 ubuntu ubuntu 394 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-coverage.in 2024-03-27T21:42:44.438 INFO:teuthology.orchestra.run.smithi060.stdout: 409067 4 -rw-r--r-- 1 ubuntu ubuntu 2846 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/btrfs_ioc_test.c 2024-03-27T21:42:44.438 INFO:teuthology.orchestra.run.smithi060.stdout: 410801 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal 2024-03-27T21:42:44.438 INFO:teuthology.orchestra.run.smithi060.stdout: 410808 4 -rw-r--r-- 1 ubuntu ubuntu 3161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/FutureImpl.h 2024-03-27T21:42:44.438 INFO:teuthology.orchestra.run.smithi060.stdout: 410806 4 -rw-r--r-- 1 ubuntu ubuntu 1106 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/Future.h 2024-03-27T21:42:44.439 INFO:teuthology.orchestra.run.smithi060.stdout: 410819 8 -rw-r--r-- 1 ubuntu ubuntu 5545 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/Journaler.h 2024-03-27T21:42:44.439 INFO:teuthology.orchestra.run.smithi060.stdout: 410807 4 -rw-r--r-- 1 ubuntu ubuntu 4087 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/FutureImpl.cc 2024-03-27T21:42:44.439 INFO:teuthology.orchestra.run.smithi060.stdout: 410818 16 -rw-r--r-- 1 ubuntu ubuntu 13888 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/Journaler.cc 2024-03-27T21:42:44.439 INFO:teuthology.orchestra.run.smithi060.stdout: 410802 4 -rw-r--r-- 1 ubuntu ubuntu 296 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/CMakeLists.txt 2024-03-27T21:42:44.439 INFO:teuthology.orchestra.run.smithi060.stdout: 410811 4 -rw-r--r-- 1 ubuntu ubuntu 758 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/JournalMetadataListener.h 2024-03-27T21:42:44.439 INFO:teuthology.orchestra.run.smithi060.stdout: 410825 4 -rw-r--r-- 1 ubuntu ubuntu 408 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/ReplayHandler.h 2024-03-27T21:42:44.439 INFO:teuthology.orchestra.run.smithi060.stdout: 410814 16 -rw-r--r-- 1 ubuntu ubuntu 14655 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/JournalRecorder.cc 2024-03-27T21:42:44.439 INFO:teuthology.orchestra.run.smithi060.stdout: 410805 4 -rw-r--r-- 1 ubuntu ubuntu 1026 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/Future.cc 2024-03-27T21:42:44.440 INFO:teuthology.orchestra.run.smithi060.stdout: 410809 36 -rw-r--r-- 1 ubuntu ubuntu 35478 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/JournalMetadata.cc 2024-03-27T21:42:44.440 INFO:teuthology.orchestra.run.smithi060.stdout: 410820 12 -rw-r--r-- 1 ubuntu ubuntu 10124 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/ObjectPlayer.cc 2024-03-27T21:42:44.440 INFO:teuthology.orchestra.run.smithi060.stdout: 410803 8 -rw-r--r-- 1 ubuntu ubuntu 4346 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/Entry.cc 2024-03-27T21:42:44.440 INFO:teuthology.orchestra.run.smithi060.stdout: 410812 32 -rw-r--r-- 1 ubuntu ubuntu 28705 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/JournalPlayer.cc 2024-03-27T21:42:44.440 INFO:teuthology.orchestra.run.smithi060.stdout: 410826 4 -rw-r--r-- 1 ubuntu ubuntu 637 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/Settings.h 2024-03-27T21:42:44.440 INFO:teuthology.orchestra.run.smithi060.stdout: 410810 12 -rw-r--r-- 1 ubuntu ubuntu 11486 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/JournalMetadata.h 2024-03-27T21:42:44.440 INFO:teuthology.orchestra.run.smithi060.stdout: 410827 4 -rw-r--r-- 1 ubuntu ubuntu 720 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/Types.h 2024-03-27T21:42:44.440 INFO:teuthology.orchestra.run.smithi060.stdout: 410815 4 -rw-r--r-- 1 ubuntu ubuntu 3690 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/JournalRecorder.h 2024-03-27T21:42:44.441 INFO:teuthology.orchestra.run.smithi060.stdout: 410823 8 -rw-r--r-- 1 ubuntu ubuntu 4394 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/ObjectRecorder.h 2024-03-27T21:42:44.441 INFO:teuthology.orchestra.run.smithi060.stdout: 410822 16 -rw-r--r-- 1 ubuntu ubuntu 13578 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/ObjectRecorder.cc 2024-03-27T21:42:44.441 INFO:teuthology.orchestra.run.smithi060.stdout: 410804 4 -rw-r--r-- 1 ubuntu ubuntu 1402 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/Entry.h 2024-03-27T21:42:44.441 INFO:teuthology.orchestra.run.smithi060.stdout: 410829 4 -rw-r--r-- 1 ubuntu ubuntu 1389 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/Utils.h 2024-03-27T21:42:44.441 INFO:teuthology.orchestra.run.smithi060.stdout: 410817 4 -rw-r--r-- 1 ubuntu ubuntu 2407 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/JournalTrimmer.h 2024-03-27T21:42:44.441 INFO:teuthology.orchestra.run.smithi060.stdout: 410813 8 -rw-r--r-- 1 ubuntu ubuntu 5090 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/JournalPlayer.h 2024-03-27T21:42:44.441 INFO:teuthology.orchestra.run.smithi060.stdout: 410821 4 -rw-r--r-- 1 ubuntu ubuntu 3672 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/ObjectPlayer.h 2024-03-27T21:42:44.441 INFO:teuthology.orchestra.run.smithi060.stdout: 410828 4 -rw-r--r-- 1 ubuntu ubuntu 670 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/Utils.cc 2024-03-27T21:42:44.442 INFO:teuthology.orchestra.run.smithi060.stdout: 410824 4 -rw-r--r-- 1 ubuntu ubuntu 695 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/ReplayEntry.h 2024-03-27T21:42:44.442 INFO:teuthology.orchestra.run.smithi060.stdout: 410816 8 -rw-r--r-- 1 ubuntu ubuntu 7743 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/journal/JournalTrimmer.cc 2024-03-27T21:42:44.442 INFO:teuthology.orchestra.run.smithi060.stdout: 411992 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/objsync 2024-03-27T21:42:44.442 INFO:teuthology.orchestra.run.smithi060.stdout: 411993 4 -rwxr-xr-x 1 ubuntu ubuntu 1087 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/objsync/boto_del.py 2024-03-27T21:42:44.442 INFO:teuthology.orchestra.run.smithi060.stdout: 410037 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor 2024-03-27T21:42:44.442 INFO:teuthology.orchestra.run.smithi060.stdout: 410050 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/lz4 2024-03-27T21:42:44.442 INFO:teuthology.orchestra.run.smithi060.stdout: 410053 4 -rw-r--r-- 1 ubuntu ubuntu 1138 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/lz4/CompressionPluginLZ4.h 2024-03-27T21:42:44.443 INFO:teuthology.orchestra.run.smithi060.stdout: 410051 4 -rw-r--r-- 1 ubuntu ubuntu 464 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/lz4/CMakeLists.txt 2024-03-27T21:42:44.443 INFO:teuthology.orchestra.run.smithi060.stdout: 410054 8 -rw-r--r-- 1 ubuntu ubuntu 4428 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/lz4/LZ4Compressor.cc 2024-03-27T21:42:44.443 INFO:teuthology.orchestra.run.smithi060.stdout: 410055 4 -rw-r--r-- 1 ubuntu ubuntu 1328 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/lz4/LZ4Compressor.h 2024-03-27T21:42:44.443 INFO:teuthology.orchestra.run.smithi060.stdout: 410052 4 -rw-r--r-- 1 ubuntu ubuntu 1006 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/lz4/CompressionPluginLZ4.cc 2024-03-27T21:42:44.443 INFO:teuthology.orchestra.run.smithi060.stdout: 410042 8 -rw-r--r-- 1 ubuntu ubuntu 6806 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/QatAccel.cc 2024-03-27T21:42:44.443 INFO:teuthology.orchestra.run.smithi060.stdout: 410038 4 -rw-r--r-- 1 ubuntu ubuntu 1043 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/CMakeLists.txt 2024-03-27T21:42:44.443 INFO:teuthology.orchestra.run.smithi060.stdout: 410039 4 -rw-r--r-- 1 ubuntu ubuntu 1091 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/CompressionPlugin.h 2024-03-27T21:42:44.443 INFO:teuthology.orchestra.run.smithi060.stdout: 410043 4 -rw-r--r-- 1 ubuntu ubuntu 1455 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/QatAccel.h 2024-03-27T21:42:44.443 INFO:teuthology.orchestra.run.smithi060.stdout: 410040 4 -rw-r--r-- 1 ubuntu ubuntu 3020 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/Compressor.cc 2024-03-27T21:42:44.444 INFO:teuthology.orchestra.run.smithi060.stdout: 410041 4 -rw-r--r-- 1 ubuntu ubuntu 3008 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/Compressor.h 2024-03-27T21:42:44.444 INFO:teuthology.orchestra.run.smithi060.stdout: 410061 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/zlib 2024-03-27T21:42:44.444 INFO:teuthology.orchestra.run.smithi060.stdout: 410062 8 -rw-r--r-- 1 ubuntu ubuntu 5100 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/zlib/CMakeLists.txt 2024-03-27T21:42:44.444 INFO:teuthology.orchestra.run.smithi060.stdout: 410066 4 -rw-r--r-- 1 ubuntu ubuntu 1552 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/zlib/ZlibCompressor.h 2024-03-27T21:42:44.444 INFO:teuthology.orchestra.run.smithi060.stdout: 410064 4 -rw-r--r-- 1 ubuntu ubuntu 1726 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/zlib/CompressionPluginZlib.h 2024-03-27T21:42:44.444 INFO:teuthology.orchestra.run.smithi060.stdout: 410063 4 -rw-r--r-- 1 ubuntu ubuntu 1102 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/zlib/CompressionPluginZlib.cc 2024-03-27T21:42:44.444 INFO:teuthology.orchestra.run.smithi060.stdout: 410065 8 -rw-r--r-- 1 ubuntu ubuntu 8008 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/zlib/ZlibCompressor.cc 2024-03-27T21:42:44.444 INFO:teuthology.orchestra.run.smithi060.stdout: 410067 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/zstd 2024-03-27T21:42:44.445 INFO:teuthology.orchestra.run.smithi060.stdout: 410068 4 -rw-r--r-- 1 ubuntu ubuntu 519 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/zstd/CMakeLists.txt 2024-03-27T21:42:44.445 INFO:teuthology.orchestra.run.smithi060.stdout: 410069 4 -rw-r--r-- 1 ubuntu ubuntu 1020 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/zstd/CompressionPluginZstd.cc 2024-03-27T21:42:44.445 INFO:teuthology.orchestra.run.smithi060.stdout: 410071 4 -rw-r--r-- 1 ubuntu ubuntu 3132 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/zstd/ZstdCompressor.h 2024-03-27T21:42:44.445 INFO:teuthology.orchestra.run.smithi060.stdout: 410070 4 -rw-r--r-- 1 ubuntu ubuntu 1181 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/zstd/CompressionPluginZstd.h 2024-03-27T21:42:44.445 INFO:teuthology.orchestra.run.smithi060.stdout: 410056 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/snappy 2024-03-27T21:42:44.445 INFO:teuthology.orchestra.run.smithi060.stdout: 410057 4 -rw-r--r-- 1 ubuntu ubuntu 381 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/snappy/CMakeLists.txt 2024-03-27T21:42:44.445 INFO:teuthology.orchestra.run.smithi060.stdout: 410059 4 -rw-r--r-- 1 ubuntu ubuntu 1173 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/snappy/CompressionPluginSnappy.h 2024-03-27T21:42:44.445 INFO:teuthology.orchestra.run.smithi060.stdout: 410060 4 -rw-r--r-- 1 ubuntu ubuntu 2878 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/snappy/SnappyCompressor.h 2024-03-27T21:42:44.446 INFO:teuthology.orchestra.run.smithi060.stdout: 410058 4 -rw-r--r-- 1 ubuntu ubuntu 1106 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/snappy/CompressionPluginSnappy.cc 2024-03-27T21:42:44.446 INFO:teuthology.orchestra.run.smithi060.stdout: 410044 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/brotli 2024-03-27T21:42:44.446 INFO:teuthology.orchestra.run.smithi060.stdout: 410046 4 -rw-r--r-- 1 ubuntu ubuntu 982 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/brotli/BrotliCompressor.h 2024-03-27T21:42:44.446 INFO:teuthology.orchestra.run.smithi060.stdout: 410047 4 -rw-r--r-- 1 ubuntu ubuntu 1150 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/brotli/CMakeLists.txt 2024-03-27T21:42:44.446 INFO:teuthology.orchestra.run.smithi060.stdout: 410049 4 -rw-r--r-- 1 ubuntu ubuntu 916 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/brotli/CompressionPluginBrotli.h 2024-03-27T21:42:44.446 INFO:teuthology.orchestra.run.smithi060.stdout: 410048 4 -rw-r--r-- 1 ubuntu ubuntu 454 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/brotli/CompressionPluginBrotli.cc 2024-03-27T21:42:44.446 INFO:teuthology.orchestra.run.smithi060.stdout: 410045 4 -rw-r--r-- 1 ubuntu ubuntu 3661 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/compressor/brotli/BrotliCompressor.cc 2024-03-27T21:42:44.446 INFO:teuthology.orchestra.run.smithi060.stdout: 409384 4 -rw-r--r-- 1 ubuntu ubuntu 1832 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph_mgr.cc 2024-03-27T21:42:44.447 INFO:teuthology.orchestra.run.smithi060.stdout: 410869 4 -rw-r--r-- 1 ubuntu ubuntu 1799 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/librados-config.cc 2024-03-27T21:42:44.447 INFO:teuthology.orchestra.run.smithi060.stdout: 408988 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth 2024-03-27T21:42:44.447 INFO:teuthology.orchestra.run.smithi060.stdout: 408991 4 -rw-r--r-- 1 ubuntu ubuntu 1390 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/AuthClient.h 2024-03-27T21:42:44.447 INFO:teuthology.orchestra.run.smithi060.stdout: 409011 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/cephx 2024-03-27T21:42:44.447 INFO:teuthology.orchestra.run.smithi060.stdout: 409016 16 -rw-r--r-- 1 ubuntu ubuntu 13315 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/cephx/CephxKeyServer.cc 2024-03-27T21:42:44.447 INFO:teuthology.orchestra.run.smithi060.stdout: 409015 4 -rw-r--r-- 1 ubuntu ubuntu 2017 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/cephx/CephxClientHandler.h 2024-03-27T21:42:44.447 INFO:teuthology.orchestra.run.smithi060.stdout: 409012 4 -rw-r--r-- 1 ubuntu ubuntu 1371 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/cephx/CephxAuthorizeHandler.cc 2024-03-27T21:42:44.447 INFO:teuthology.orchestra.run.smithi060.stdout: 409018 20 -rw-r--r-- 1 ubuntu ubuntu 17861 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/cephx/CephxProtocol.cc 2024-03-27T21:42:44.447 INFO:teuthology.orchestra.run.smithi060.stdout: 409023 4 -rw-r--r-- 1 ubuntu ubuntu 1128 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/cephx/CephxSessionHandler.h 2024-03-27T21:42:44.448 INFO:teuthology.orchestra.run.smithi060.stdout: 409019 20 -rw-r--r-- 1 ubuntu ubuntu 17713 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/cephx/CephxProtocol.h 2024-03-27T21:42:44.448 INFO:teuthology.orchestra.run.smithi060.stdout: 409014 12 -rw-r--r-- 1 ubuntu ubuntu 9770 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/cephx/CephxClientHandler.cc 2024-03-27T21:42:44.448 INFO:teuthology.orchestra.run.smithi060.stdout: 409022 8 -rw-r--r-- 1 ubuntu ubuntu 5829 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/cephx/CephxSessionHandler.cc 2024-03-27T21:42:44.448 INFO:teuthology.orchestra.run.smithi060.stdout: 409020 16 -rw-r--r-- 1 ubuntu ubuntu 13167 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/cephx/CephxServiceHandler.cc 2024-03-27T21:42:44.448 INFO:teuthology.orchestra.run.smithi060.stdout: 409017 12 -rw-r--r-- 1 ubuntu ubuntu 10203 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/cephx/CephxKeyServer.h 2024-03-27T21:42:44.448 INFO:teuthology.orchestra.run.smithi060.stdout: 409021 4 -rw-r--r-- 1 ubuntu ubuntu 1544 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/cephx/CephxServiceHandler.h 2024-03-27T21:42:44.448 INFO:teuthology.orchestra.run.smithi060.stdout: 409013 4 -rw-r--r-- 1 ubuntu ubuntu 1114 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/cephx/CephxAuthorizeHandler.h 2024-03-27T21:42:44.448 INFO:teuthology.orchestra.run.smithi060.stdout: 409005 8 -rw-r--r-- 1 ubuntu ubuntu 6151 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/Crypto.h 2024-03-27T21:42:44.449 INFO:teuthology.orchestra.run.smithi060.stdout: 408990 4 -rw-r--r-- 1 ubuntu ubuntu 1331 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/AuthAuthorizeHandler.h 2024-03-27T21:42:44.449 INFO:teuthology.orchestra.run.smithi060.stdout: 408997 4 -rw-r--r-- 1 ubuntu ubuntu 2310 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/AuthRegistry.h 2024-03-27T21:42:44.449 INFO:teuthology.orchestra.run.smithi060.stdout: 409003 4 -rw-r--r-- 1 ubuntu ubuntu 649 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/CMakeLists.txt 2024-03-27T21:42:44.449 INFO:teuthology.orchestra.run.smithi060.stdout: 409008 4 -rw-r--r-- 1 ubuntu ubuntu 3520 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/KeyRing.h 2024-03-27T21:42:44.449 INFO:teuthology.orchestra.run.smithi060.stdout: 409001 4 -rw-r--r-- 1 ubuntu ubuntu 1463 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/AuthSessionHandler.cc 2024-03-27T21:42:44.449 INFO:teuthology.orchestra.run.smithi060.stdout: 408989 12 -rw-r--r-- 1 ubuntu ubuntu 9777 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/Auth.h 2024-03-27T21:42:44.450 INFO:teuthology.orchestra.run.smithi060.stdout: 409010 4 -rw-r--r-- 1 ubuntu ubuntu 1376 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/RotatingKeyRing.h 2024-03-27T21:42:44.450 INFO:teuthology.orchestra.run.smithi060.stdout: 408992 4 -rw-r--r-- 1 ubuntu ubuntu 1005 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/AuthClientHandler.cc 2024-03-27T21:42:44.450 INFO:teuthology.orchestra.run.smithi060.stdout: 409034 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/none 2024-03-27T21:42:44.450 INFO:teuthology.orchestra.run.smithi060.stdout: 409037 4 -rw-r--r-- 1 ubuntu ubuntu 1751 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/none/AuthNoneClientHandler.h 2024-03-27T21:42:44.450 INFO:teuthology.orchestra.run.smithi060.stdout: 409036 4 -rw-r--r-- 1 ubuntu ubuntu 1123 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/none/AuthNoneAuthorizeHandler.h 2024-03-27T21:42:44.450 INFO:teuthology.orchestra.run.smithi060.stdout: 409035 4 -rw-r--r-- 1 ubuntu ubuntu 1502 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/none/AuthNoneAuthorizeHandler.cc 2024-03-27T21:42:44.450 INFO:teuthology.orchestra.run.smithi060.stdout: 409040 4 -rw-r--r-- 1 ubuntu ubuntu 529 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/none/AuthNoneSessionHandler.h 2024-03-27T21:42:44.451 INFO:teuthology.orchestra.run.smithi060.stdout: 409039 4 -rw-r--r-- 1 ubuntu ubuntu 1268 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/none/AuthNoneServiceHandler.h 2024-03-27T21:42:44.451 INFO:teuthology.orchestra.run.smithi060.stdout: 409038 4 -rw-r--r-- 1 ubuntu ubuntu 1142 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/none/AuthNoneProtocol.h 2024-03-27T21:42:44.451 INFO:teuthology.orchestra.run.smithi060.stdout: 409024 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/krb 2024-03-27T21:42:44.451 INFO:teuthology.orchestra.run.smithi060.stdout: 409031 8 -rw-r--r-- 1 ubuntu ubuntu 7650 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/krb/KrbServiceHandler.cpp 2024-03-27T21:42:44.451 INFO:teuthology.orchestra.run.smithi060.stdout: 409025 4 -rw-r--r-- 1 ubuntu ubuntu 1345 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/krb/KrbAuthorizeHandler.cpp 2024-03-27T21:42:44.451 INFO:teuthology.orchestra.run.smithi060.stdout: 409032 4 -rw-r--r-- 1 ubuntu ubuntu 1758 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/krb/KrbServiceHandler.hpp 2024-03-27T21:42:44.451 INFO:teuthology.orchestra.run.smithi060.stdout: 409027 12 -rw-r--r-- 1 ubuntu ubuntu 8553 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/krb/KrbClientHandler.cpp 2024-03-27T21:42:44.451 INFO:teuthology.orchestra.run.smithi060.stdout: 409033 4 -rw-r--r-- 1 ubuntu ubuntu 867 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/krb/KrbSessionHandler.hpp 2024-03-27T21:42:44.451 INFO:teuthology.orchestra.run.smithi060.stdout: 409028 4 -rw-r--r-- 1 ubuntu ubuntu 2259 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/krb/KrbClientHandler.hpp 2024-03-27T21:42:44.452 INFO:teuthology.orchestra.run.smithi060.stdout: 409026 4 -rw-r--r-- 1 ubuntu ubuntu 1104 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/krb/KrbAuthorizeHandler.hpp 2024-03-27T21:42:44.452 INFO:teuthology.orchestra.run.smithi060.stdout: 409030 4 -rw-r--r-- 1 ubuntu ubuntu 3909 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/krb/KrbProtocol.hpp 2024-03-27T21:42:44.452 INFO:teuthology.orchestra.run.smithi060.stdout: 409029 4 -rw-r--r-- 1 ubuntu ubuntu 2782 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/krb/KrbProtocol.cpp 2024-03-27T21:42:44.452 INFO:teuthology.orchestra.run.smithi060.stdout: 408996 12 -rw-r--r-- 1 ubuntu ubuntu 9647 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/AuthRegistry.cc 2024-03-27T21:42:44.452 INFO:teuthology.orchestra.run.smithi060.stdout: 408994 4 -rw-r--r-- 1 ubuntu ubuntu 2140 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/AuthMethodList.cc 2024-03-27T21:42:44.452 INFO:teuthology.orchestra.run.smithi060.stdout: 408995 4 -rw-r--r-- 1 ubuntu ubuntu 955 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/AuthMethodList.h 2024-03-27T21:42:44.452 INFO:teuthology.orchestra.run.smithi060.stdout: 409004 20 -rw-r--r-- 1 ubuntu ubuntu 17227 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/Crypto.cc 2024-03-27T21:42:44.452 INFO:teuthology.orchestra.run.smithi060.stdout: 409000 4 -rw-r--r-- 1 ubuntu ubuntu 2573 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/AuthServiceHandler.h 2024-03-27T21:42:44.453 INFO:teuthology.orchestra.run.smithi060.stdout: 409041 4 -rw-r--r-- 1 ubuntu ubuntu 1916 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/scheme.txt 2024-03-27T21:42:44.453 INFO:teuthology.orchestra.run.smithi060.stdout: 409002 4 -rw-r--r-- 1 ubuntu ubuntu 1290 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/AuthSessionHandler.h 2024-03-27T21:42:44.453 INFO:teuthology.orchestra.run.smithi060.stdout: 409007 8 -rw-r--r-- 1 ubuntu ubuntu 6441 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/KeyRing.cc 2024-03-27T21:42:44.453 INFO:teuthology.orchestra.run.smithi060.stdout: 409009 4 -rw-r--r-- 1 ubuntu ubuntu 1678 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/RotatingKeyRing.cc 2024-03-27T21:42:44.453 INFO:teuthology.orchestra.run.smithi060.stdout: 408998 4 -rw-r--r-- 1 ubuntu ubuntu 1509 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/AuthServer.h 2024-03-27T21:42:44.453 INFO:teuthology.orchestra.run.smithi060.stdout: 408993 4 -rw-r--r-- 1 ubuntu ubuntu 1947 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/AuthClientHandler.h 2024-03-27T21:42:44.453 INFO:teuthology.orchestra.run.smithi060.stdout: 408999 4 -rw-r--r-- 1 ubuntu ubuntu 2304 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/AuthServiceHandler.cc 2024-03-27T21:42:44.453 INFO:teuthology.orchestra.run.smithi060.stdout: 409006 4 -rw-r--r-- 1 ubuntu ubuntu 1471 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/auth/DummyAuth.h 2024-03-27T21:42:44.454 INFO:teuthology.orchestra.run.smithi060.stdout: 410579 4 -rw-r--r-- 1 ubuntu ubuntu 92 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/etc-rbdmap 2024-03-27T21:42:44.454 INFO:teuthology.orchestra.run.smithi060.stdout: 409388 4 -rw-r--r-- 1 ubuntu ubuntu 2891 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph_syn.cc 2024-03-27T21:42:44.454 INFO:teuthology.orchestra.run.smithi060.stdout: 410072 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cpp_redis 2024-03-27T21:42:44.454 INFO:teuthology.orchestra.run.smithi060.stdout: 410600 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/fmt 2024-03-27T21:42:44.454 INFO:teuthology.orchestra.run.smithi060.stdout: 409068 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/c-ares 2024-03-27T21:42:44.454 INFO:teuthology.orchestra.run.smithi060.stdout: 410765 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java 2024-03-27T21:42:44.454 INFO:teuthology.orchestra.run.smithi060.stdout: 410767 4 -rw-r--r-- 1 ubuntu ubuntu 2055 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/CMakeLists.txt 2024-03-27T21:42:44.454 INFO:teuthology.orchestra.run.smithi060.stdout: 410768 4 -rw-r--r-- 1 ubuntu ubuntu 1564 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/README 2024-03-27T21:42:44.455 INFO:teuthology.orchestra.run.smithi060.stdout: 410792 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/test 2024-03-27T21:42:44.455 INFO:teuthology.orchestra.run.smithi060.stdout: 410793 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/test/com 2024-03-27T21:42:44.455 INFO:teuthology.orchestra.run.smithi060.stdout: 410794 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/test/com/ceph 2024-03-27T21:42:44.455 INFO:teuthology.orchestra.run.smithi060.stdout: 410795 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/test/com/ceph/fs 2024-03-27T21:42:44.455 INFO:teuthology.orchestra.run.smithi060.stdout: 410798 4 -rw-r--r-- 1 ubuntu ubuntu 2896 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/test/com/ceph/fs/CephMountCreateTest.java 2024-03-27T21:42:44.455 INFO:teuthology.orchestra.run.smithi060.stdout: 410797 4 -rw-r--r-- 1 ubuntu ubuntu 1670 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/test/com/ceph/fs/CephDoubleMountTest.java 2024-03-27T21:42:44.455 INFO:teuthology.orchestra.run.smithi060.stdout: 410799 28 -rw-r--r-- 1 ubuntu ubuntu 26218 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/test/com/ceph/fs/CephMountTest.java 2024-03-27T21:42:44.455 INFO:teuthology.orchestra.run.smithi060.stdout: 410800 8 -rw-r--r-- 1 ubuntu ubuntu 4869 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/test/com/ceph/fs/CephUnmountedTest.java 2024-03-27T21:42:44.456 INFO:teuthology.orchestra.run.smithi060.stdout: 410796 4 -rw-r--r-- 1 ubuntu ubuntu 1601 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/test/com/ceph/fs/CephAllTests.java 2024-03-27T21:42:44.456 INFO:teuthology.orchestra.run.smithi060.stdout: 410769 4 -rw-r--r-- 1 ubuntu ubuntu 2543 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/build.xml 2024-03-27T21:42:44.456 INFO:teuthology.orchestra.run.smithi060.stdout: 410786 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/native 2024-03-27T21:42:44.456 INFO:teuthology.orchestra.run.smithi060.stdout: 410787 4 -rw-r--r-- 1 ubuntu ubuntu 448 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/native/CMakeLists.txt 2024-03-27T21:42:44.456 INFO:teuthology.orchestra.run.smithi060.stdout: 410789 4 -rw-r--r-- 1 ubuntu ubuntu 2195 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/native/JniConstants.h 2024-03-27T21:42:44.456 INFO:teuthology.orchestra.run.smithi060.stdout: 410790 4 -rw-r--r-- 1 ubuntu ubuntu 1617 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/native/ScopedLocalRef.h 2024-03-27T21:42:44.456 INFO:teuthology.orchestra.run.smithi060.stdout: 410791 80 -rw-r--r-- 1 ubuntu ubuntu 79866 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/native/libcephfs_jni.cc 2024-03-27T21:42:44.456 INFO:teuthology.orchestra.run.smithi060.stdout: 410788 4 -rw-r--r-- 1 ubuntu ubuntu 1491 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/native/JniConstants.cpp 2024-03-27T21:42:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: 410770 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/java 2024-03-27T21:42:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: 410771 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/java/com 2024-03-27T21:42:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: 410772 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/java/com/ceph 2024-03-27T21:42:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: 410773 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/java/com/ceph/crush 2024-03-27T21:42:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: 410774 4 -rw-r--r-- 1 ubuntu ubuntu 1449 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/java/com/ceph/crush/Bucket.java 2024-03-27T21:42:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: 410775 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/java/com/ceph/fs 2024-03-27T21:42:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: 410780 4 -rw-r--r-- 1 ubuntu ubuntu 3139 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/java/com/ceph/fs/CephNativeLoader.java 2024-03-27T21:42:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: 410781 4 -rw-r--r-- 1 ubuntu ubuntu 1538 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/java/com/ceph/fs/CephNotDirectoryException.java 2024-03-27T21:42:44.458 INFO:teuthology.orchestra.run.smithi060.stdout: 410777 4 -rw-r--r-- 1 ubuntu ubuntu 1561 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/java/com/ceph/fs/CephFileAlreadyExistsException.java 2024-03-27T21:42:44.458 INFO:teuthology.orchestra.run.smithi060.stdout: 410776 4 -rw-r--r-- 1 ubuntu ubuntu 1535 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/java/com/ceph/fs/CephAlreadyMountedException.java 2024-03-27T21:42:44.458 INFO:teuthology.orchestra.run.smithi060.stdout: 410778 4 -rw-r--r-- 1 ubuntu ubuntu 1868 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/java/com/ceph/fs/CephFileExtent.java 2024-03-27T21:42:44.458 INFO:teuthology.orchestra.run.smithi060.stdout: 410782 4 -rw-r--r-- 1 ubuntu ubuntu 1511 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/java/com/ceph/fs/CephNotMountedException.java 2024-03-27T21:42:44.458 INFO:teuthology.orchestra.run.smithi060.stdout: 410779 32 -rw-r--r-- 1 ubuntu ubuntu 29477 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/java/com/ceph/fs/CephMount.java 2024-03-27T21:42:44.458 INFO:teuthology.orchestra.run.smithi060.stdout: 410783 4 -rw-r--r-- 1 ubuntu ubuntu 1492 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/java/com/ceph/fs/CephPoolException.java 2024-03-27T21:42:44.458 INFO:teuthology.orchestra.run.smithi060.stdout: 410785 4 -rw-r--r-- 1 ubuntu ubuntu 1321 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/java/com/ceph/fs/CephStatVFS.java 2024-03-27T21:42:44.461 INFO:teuthology.orchestra.run.smithi060.stdout: 410784 4 -rw-r--r-- 1 ubuntu ubuntu 1665 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/java/com/ceph/fs/CephStat.java 2024-03-27T21:42:44.462 INFO:teuthology.orchestra.run.smithi060.stdout: 410766 4 -rw-r--r-- 1 ubuntu ubuntu 87 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/java/.gitignore 2024-03-27T21:42:44.462 INFO:teuthology.orchestra.run.smithi060.stdout: 410868 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/libkmip 2024-03-27T21:42:44.462 INFO:teuthology.orchestra.run.smithi060.stdout: 412195 4 -rwxr-xr-x 1 ubuntu ubuntu 626 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/push_to_qemu.pl 2024-03-27T21:42:44.462 INFO:teuthology.orchestra.run.smithi060.stdout: 409681 4 -rw-r--r-- 1 ubuntu ubuntu 1662 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls_crypto.cc 2024-03-27T21:42:44.462 INFO:teuthology.orchestra.run.smithi060.stdout: 410761 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/isa-l 2024-03-27T21:42:44.462 INFO:teuthology.orchestra.run.smithi060.stdout: 409066 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/blkin 2024-03-27T21:42:44.462 INFO:teuthology.orchestra.run.smithi060.stdout: 412190 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/powerdns 2024-03-27T21:42:44.462 INFO:teuthology.orchestra.run.smithi060.stdout: 412191 4 -rw-r--r-- 1 ubuntu ubuntu 1671 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/powerdns/README.md 2024-03-27T21:42:44.463 INFO:teuthology.orchestra.run.smithi060.stdout: 412193 4 -rw-r--r-- 1 ubuntu ubuntu 329 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/powerdns/rgw-pdns.conf.in 2024-03-27T21:42:44.463 INFO:teuthology.orchestra.run.smithi060.stdout: 412192 12 -rwxr-xr-x 1 ubuntu ubuntu 8790 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/powerdns/pdns-backend-rgw.py 2024-03-27T21:42:44.463 INFO:teuthology.orchestra.run.smithi060.stdout: 411423 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds 2024-03-27T21:42:44.463 INFO:teuthology.orchestra.run.smithi060.stdout: 411556 100 -rw-r--r-- 1 ubuntu ubuntu 98916 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/journal.cc 2024-03-27T21:42:44.463 INFO:teuthology.orchestra.run.smithi060.stdout: 411523 16 -rw-r--r-- 1 ubuntu ubuntu 13410 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/SnapServer.cc 2024-03-27T21:42:44.463 INFO:teuthology.orchestra.run.smithi060.stdout: 411467 8 -rw-r--r-- 1 ubuntu ubuntu 5249 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSContext.h 2024-03-27T21:42:44.463 INFO:teuthology.orchestra.run.smithi060.stdout: 411506 8 -rw-r--r-- 1 ubuntu ubuntu 7103 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/RecoveryQueue.cc 2024-03-27T21:42:44.463 INFO:teuthology.orchestra.run.smithi060.stdout: 411478 24 -rw-r--r-- 1 ubuntu ubuntu 23308 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSRank.h 2024-03-27T21:42:44.463 INFO:teuthology.orchestra.run.smithi060.stdout: 411490 16 -rw-r--r-- 1 ubuntu ubuntu 14144 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MetricsHandler.cc 2024-03-27T21:42:44.464 INFO:teuthology.orchestra.run.smithi060.stdout: 411435 160 -rw-r--r-- 1 ubuntu ubuntu 160609 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/CInode.cc 2024-03-27T21:42:44.464 INFO:teuthology.orchestra.run.smithi060.stdout: 411496 36 -rw-r--r-- 1 ubuntu ubuntu 33090 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/OpenFileTable.cc 2024-03-27T21:42:44.464 INFO:teuthology.orchestra.run.smithi060.stdout: 411430 4 -rw-r--r-- 1 ubuntu ubuntu 362 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/BoostUrlImpl.cc 2024-03-27T21:42:44.464 INFO:teuthology.orchestra.run.smithi060.stdout: 411518 20 -rw-r--r-- 1 ubuntu ubuntu 18817 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/SimpleLock.h 2024-03-27T21:42:44.464 INFO:teuthology.orchestra.run.smithi060.stdout: 411464 4 -rw-r--r-- 1 ubuntu ubuntu 3323 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSCacheObject.cc 2024-03-27T21:42:44.464 INFO:teuthology.orchestra.run.smithi060.stdout: 411531 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/events 2024-03-27T21:42:44.464 INFO:teuthology.orchestra.run.smithi060.stdout: 411541 8 -rw-r--r-- 1 ubuntu ubuntu 4541 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/events/EPeerUpdate.h 2024-03-27T21:42:44.464 INFO:teuthology.orchestra.run.smithi060.stdout: 411545 4 -rw-r--r-- 1 ubuntu ubuntu 2289 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/events/ESession.h 2024-03-27T21:42:44.465 INFO:teuthology.orchestra.run.smithi060.stdout: 411538 24 -rw-r--r-- 1 ubuntu ubuntu 20710 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/events/EMetaBlob.h 2024-03-27T21:42:44.465 INFO:teuthology.orchestra.run.smithi060.stdout: 411543 4 -rw-r--r-- 1 ubuntu ubuntu 1216 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/events/EResetJournal.h 2024-03-27T21:42:44.465 INFO:teuthology.orchestra.run.smithi060.stdout: 411548 4 -rw-r--r-- 1 ubuntu ubuntu 1422 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/events/ETableClient.h 2024-03-27T21:42:44.465 INFO:teuthology.orchestra.run.smithi060.stdout: 411550 4 -rw-r--r-- 1 ubuntu ubuntu 1462 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/events/EUpdate.h 2024-03-27T21:42:44.465 INFO:teuthology.orchestra.run.smithi060.stdout: 411536 4 -rw-r--r-- 1 ubuntu ubuntu 1651 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/events/EImportStart.h 2024-03-27T21:42:44.465 INFO:teuthology.orchestra.run.smithi060.stdout: 411547 4 -rw-r--r-- 1 ubuntu ubuntu 1484 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/events/ESubtreeMap.h 2024-03-27T21:42:44.465 INFO:teuthology.orchestra.run.smithi060.stdout: 411546 4 -rw-r--r-- 1 ubuntu ubuntu 1857 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/events/ESessions.h 2024-03-27T21:42:44.465 INFO:teuthology.orchestra.run.smithi060.stdout: 411537 4 -rw-r--r-- 1 ubuntu ubuntu 1187 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/events/ELid.h 2024-03-27T21:42:44.466 INFO:teuthology.orchestra.run.smithi060.stdout: 411540 4 -rw-r--r-- 1 ubuntu ubuntu 1685 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/events/EOpen.h 2024-03-27T21:42:44.466 INFO:teuthology.orchestra.run.smithi060.stdout: 411533 4 -rw-r--r-- 1 ubuntu ubuntu 1532 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/events/EExport.h 2024-03-27T21:42:44.466 INFO:teuthology.orchestra.run.smithi060.stdout: 411535 4 -rw-r--r-- 1 ubuntu ubuntu 1404 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/events/EImportFinish.h 2024-03-27T21:42:44.466 INFO:teuthology.orchestra.run.smithi060.stdout: 411539 4 -rw-r--r-- 1 ubuntu ubuntu 946 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/events/ENoOp.h 2024-03-27T21:42:44.466 INFO:teuthology.orchestra.run.smithi060.stdout: 411542 4 -rw-r--r-- 1 ubuntu ubuntu 1330 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/events/EPurged.h 2024-03-27T21:42:44.466 INFO:teuthology.orchestra.run.smithi060.stdout: 411532 4 -rw-r--r-- 1 ubuntu ubuntu 1183 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/events/ECommitted.h 2024-03-27T21:42:44.466 INFO:teuthology.orchestra.run.smithi060.stdout: 411534 4 -rw-r--r-- 1 ubuntu ubuntu 2279 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/events/EFragment.h 2024-03-27T21:42:44.466 INFO:teuthology.orchestra.run.smithi060.stdout: 411544 4 -rw-r--r-- 1 ubuntu ubuntu 1149 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/events/ESegment.h 2024-03-27T21:42:44.466 INFO:teuthology.orchestra.run.smithi060.stdout: 411549 4 -rw-r--r-- 1 ubuntu ubuntu 1851 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/events/ETableServer.h 2024-03-27T21:42:44.467 INFO:teuthology.orchestra.run.smithi060.stdout: 411488 24 -rw-r--r-- 1 ubuntu ubuntu 22427 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MetricAggregator.cc 2024-03-27T21:42:44.467 INFO:teuthology.orchestra.run.smithi060.stdout: 411512 4 -rw-r--r-- 1 ubuntu ubuntu 884 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/SegmentBoundary.h 2024-03-27T21:42:44.467 INFO:teuthology.orchestra.run.smithi060.stdout: 411557 12 -rw-r--r-- 1 ubuntu ubuntu 10165 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/locks.c 2024-03-27T21:42:44.467 INFO:teuthology.orchestra.run.smithi060.stdout: 411431 20 -rw-r--r-- 1 ubuntu ubuntu 16782 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/CDentry.cc 2024-03-27T21:42:44.467 INFO:teuthology.orchestra.run.smithi060.stdout: 411473 4 -rw-r--r-- 1 ubuntu ubuntu 779 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSMetaRequest.h 2024-03-27T21:42:44.467 INFO:teuthology.orchestra.run.smithi060.stdout: 411513 388 -rw-r--r-- 1 ubuntu ubuntu 394942 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/Server.cc 2024-03-27T21:42:44.467 INFO:teuthology.orchestra.run.smithi060.stdout: 411437 4 -rw-r--r-- 1 ubuntu ubuntu 1108 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/CMakeLists.txt 2024-03-27T21:42:44.468 INFO:teuthology.orchestra.run.smithi060.stdout: 411554 8 -rw-r--r-- 1 ubuntu ubuntu 4234 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/inode_backtrace.cc 2024-03-27T21:42:44.468 INFO:teuthology.orchestra.run.smithi060.stdout: 411460 52 -rw-r--r-- 1 ubuntu ubuntu 51659 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDLog.cc 2024-03-27T21:42:44.468 INFO:teuthology.orchestra.run.smithi060.stdout: 411530 4 -rw-r--r-- 1 ubuntu ubuntu 3808 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/cephfs_features.h 2024-03-27T21:42:44.468 INFO:teuthology.orchestra.run.smithi060.stdout: 411450 4 -rw-r--r-- 1 ubuntu ubuntu 1514 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/LocalLockC.h 2024-03-27T21:42:44.468 INFO:teuthology.orchestra.run.smithi060.stdout: 411515 40 -rw-r--r-- 1 ubuntu ubuntu 38918 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/SessionMap.cc 2024-03-27T21:42:44.468 INFO:teuthology.orchestra.run.smithi060.stdout: 411501 8 -rw-r--r-- 1 ubuntu ubuntu 6906 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/QuiesceAgent.h 2024-03-27T21:42:44.468 INFO:teuthology.orchestra.run.smithi060.stdout: 411525 24 -rw-r--r-- 1 ubuntu ubuntu 22745 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/StrayManager.cc 2024-03-27T21:42:44.468 INFO:teuthology.orchestra.run.smithi060.stdout: 411482 8 -rw-r--r-- 1 ubuntu ubuntu 7505 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSTableClient.cc 2024-03-27T21:42:44.469 INFO:teuthology.orchestra.run.smithi060.stdout: 411462 16 -rw-r--r-- 1 ubuntu ubuntu 14912 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSAuthCaps.cc 2024-03-27T21:42:44.469 INFO:teuthology.orchestra.run.smithi060.stdout: 411438 12 -rw-r--r-- 1 ubuntu ubuntu 8285 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/Capability.cc 2024-03-27T21:42:44.469 INFO:teuthology.orchestra.run.smithi060.stdout: 411472 28 -rw-r--r-- 1 ubuntu ubuntu 26924 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSMap.h 2024-03-27T21:42:44.469 INFO:teuthology.orchestra.run.smithi060.stdout: 411447 4 -rw-r--r-- 1 ubuntu ubuntu 2878 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/InoTable.h 2024-03-27T21:42:44.469 INFO:teuthology.orchestra.run.smithi060.stdout: 411491 4 -rw-r--r-- 1 ubuntu ubuntu 3640 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MetricsHandler.h 2024-03-27T21:42:44.469 INFO:teuthology.orchestra.run.smithi060.stdout: 411426 4 -rw-r--r-- 1 ubuntu ubuntu 861 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/BatchOp.cc 2024-03-27T21:42:44.469 INFO:teuthology.orchestra.run.smithi060.stdout: 411519 12 -rw-r--r-- 1 ubuntu ubuntu 8528 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/SnapClient.cc 2024-03-27T21:42:44.469 INFO:teuthology.orchestra.run.smithi060.stdout: 411486 8 -rw-r--r-- 1 ubuntu ubuntu 4294 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/Mantle.cc 2024-03-27T21:42:44.469 INFO:teuthology.orchestra.run.smithi060.stdout: 411446 8 -rw-r--r-- 1 ubuntu ubuntu 6224 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/InoTable.cc 2024-03-27T21:42:44.470 INFO:teuthology.orchestra.run.smithi060.stdout: 411481 4 -rw-r--r-- 1 ubuntu ubuntu 2483 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSTable.h 2024-03-27T21:42:44.470 INFO:teuthology.orchestra.run.smithi060.stdout: 411521 16 -rw-r--r-- 1 ubuntu ubuntu 13967 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/SnapRealm.cc 2024-03-27T21:42:44.470 INFO:teuthology.orchestra.run.smithi060.stdout: 411429 4 -rw-r--r-- 1 ubuntu ubuntu 3476 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/Beacon.h 2024-03-27T21:42:44.470 INFO:teuthology.orchestra.run.smithi060.stdout: 411479 24 -rw-r--r-- 1 ubuntu ubuntu 21513 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSRankQuiesce.cc 2024-03-27T21:42:44.470 INFO:teuthology.orchestra.run.smithi060.stdout: 411495 20 -rw-r--r-- 1 ubuntu ubuntu 16868 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/Mutation.h 2024-03-27T21:42:44.470 INFO:teuthology.orchestra.run.smithi060.stdout: 411471 40 -rw-r--r-- 1 ubuntu ubuntu 39928 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSMap.cc 2024-03-27T21:42:44.470 INFO:teuthology.orchestra.run.smithi060.stdout: 411433 104 -rw-r--r-- 1 ubuntu ubuntu 106118 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/CDir.cc 2024-03-27T21:42:44.470 INFO:teuthology.orchestra.run.smithi060.stdout: 411484 12 -rw-r--r-- 1 ubuntu ubuntu 10955 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSTableServer.cc 2024-03-27T21:42:44.471 INFO:teuthology.orchestra.run.smithi060.stdout: 411463 8 -rw-r--r-- 1 ubuntu ubuntu 8039 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSAuthCaps.h 2024-03-27T21:42:44.471 INFO:teuthology.orchestra.run.smithi060.stdout: 411493 16 -rw-r--r-- 1 ubuntu ubuntu 14285 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/Migrator.h 2024-03-27T21:42:44.471 INFO:teuthology.orchestra.run.smithi060.stdout: 411527 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/balancers 2024-03-27T21:42:44.471 INFO:teuthology.orchestra.run.smithi060.stdout: 411528 4 -rw-r--r-- 1 ubuntu ubuntu 1304 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/balancers/greedyspill.lua 2024-03-27T21:42:44.471 INFO:teuthology.orchestra.run.smithi060.stdout: 411508 8 -rw-r--r-- 1 ubuntu ubuntu 6343 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/ScatterLock.h 2024-03-27T21:42:44.471 INFO:teuthology.orchestra.run.smithi060.stdout: 411451 188 -rw-r--r-- 1 ubuntu ubuntu 190119 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/Locker.cc 2024-03-27T21:42:44.471 INFO:teuthology.orchestra.run.smithi060.stdout: 411425 4 -rw-r--r-- 1 ubuntu ubuntu 1904 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/Anchor.h 2024-03-27T21:42:44.471 INFO:teuthology.orchestra.run.smithi060.stdout: 411522 8 -rw-r--r-- 1 ubuntu ubuntu 4281 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/SnapRealm.h 2024-03-27T21:42:44.471 INFO:teuthology.orchestra.run.smithi060.stdout: 411498 28 -rw-r--r-- 1 ubuntu ubuntu 25969 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/PurgeQueue.cc 2024-03-27T21:42:44.472 INFO:teuthology.orchestra.run.smithi060.stdout: 411558 4 -rw-r--r-- 1 ubuntu ubuntu 2801 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/locks.h 2024-03-27T21:42:44.472 INFO:teuthology.orchestra.run.smithi060.stdout: 411562 8 -rw-r--r-- 1 ubuntu ubuntu 5846 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/snap.cc 2024-03-27T21:42:44.472 INFO:teuthology.orchestra.run.smithi060.stdout: 411551 24 -rw-r--r-- 1 ubuntu ubuntu 22235 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/flock.cc 2024-03-27T21:42:44.472 INFO:teuthology.orchestra.run.smithi060.stdout: 411474 12 -rw-r--r-- 1 ubuntu ubuntu 11492 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSPerfMetricTypes.h 2024-03-27T21:42:44.472 INFO:teuthology.orchestra.run.smithi060.stdout: 411452 12 -rw-r--r-- 1 ubuntu ubuntu 11222 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/Locker.h 2024-03-27T21:42:44.472 INFO:teuthology.orchestra.run.smithi060.stdout: 411561 32 -rw-r--r-- 1 ubuntu ubuntu 29420 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/mdstypes.h 2024-03-27T21:42:44.472 INFO:teuthology.orchestra.run.smithi060.stdout: 411492 112 -rw-r--r-- 1 ubuntu ubuntu 111030 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/Migrator.cc 2024-03-27T21:42:44.472 INFO:teuthology.orchestra.run.smithi060.stdout: 411507 4 -rw-r--r-- 1 ubuntu ubuntu 1380 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/RecoveryQueue.h 2024-03-27T21:42:44.472 INFO:teuthology.orchestra.run.smithi060.stdout: 411517 8 -rw-r--r-- 1 ubuntu ubuntu 4121 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/SimpleLock.cc 2024-03-27T21:42:44.473 INFO:teuthology.orchestra.run.smithi060.stdout: 411453 8 -rw-r--r-- 1 ubuntu ubuntu 5947 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/LogEvent.cc 2024-03-27T21:42:44.473 INFO:teuthology.orchestra.run.smithi060.stdout: 411459 56 -rw-r--r-- 1 ubuntu ubuntu 54978 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDCache.h 2024-03-27T21:42:44.473 INFO:teuthology.orchestra.run.smithi060.stdout: 411480 8 -rw-r--r-- 1 ubuntu ubuntu 5060 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSTable.cc 2024-03-27T21:42:44.473 INFO:teuthology.orchestra.run.smithi060.stdout: 411457 8 -rw-r--r-- 1 ubuntu ubuntu 4646 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDBalancer.h 2024-03-27T21:42:44.473 INFO:teuthology.orchestra.run.smithi060.stdout: 411477 128 -rw-r--r-- 1 ubuntu ubuntu 128576 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSRank.cc 2024-03-27T21:42:44.473 INFO:teuthology.orchestra.run.smithi060.stdout: 411465 12 -rw-r--r-- 1 ubuntu ubuntu 10557 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSCacheObject.h 2024-03-27T21:42:44.473 INFO:teuthology.orchestra.run.smithi060.stdout: 411456 48 -rw-r--r-- 1 ubuntu ubuntu 47003 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDBalancer.cc 2024-03-27T21:42:44.473 INFO:teuthology.orchestra.run.smithi060.stdout: 411559 4 -rw-r--r-- 1 ubuntu ubuntu 2215 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/mds_table_types.h 2024-03-27T21:42:44.474 INFO:teuthology.orchestra.run.smithi060.stdout: 411505 12 -rw-r--r-- 1 ubuntu ubuntu 8920 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/QuiesceDbManager.h 2024-03-27T21:42:44.474 INFO:teuthology.orchestra.run.smithi060.stdout: 411503 8 -rw-r--r-- 1 ubuntu ubuntu 5346 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/QuiesceDbEncoding.h 2024-03-27T21:42:44.474 INFO:teuthology.orchestra.run.smithi060.stdout: 411502 28 -rw-r--r-- 1 ubuntu ubuntu 26373 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/QuiesceDb.h 2024-03-27T21:42:44.474 INFO:teuthology.orchestra.run.smithi060.stdout: 411511 8 -rw-r--r-- 1 ubuntu ubuntu 8046 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/ScrubStack.h 2024-03-27T21:42:44.474 INFO:teuthology.orchestra.run.smithi060.stdout: 411432 12 -rw-r--r-- 1 ubuntu ubuntu 11689 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/CDentry.h 2024-03-27T21:42:44.474 INFO:teuthology.orchestra.run.smithi060.stdout: 411444 4 -rw-r--r-- 1 ubuntu ubuntu 2102 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/FSMapUser.cc 2024-03-27T21:42:44.474 INFO:teuthology.orchestra.run.smithi060.stdout: 411445 4 -rw-r--r-- 1 ubuntu ubuntu 1727 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/FSMapUser.h 2024-03-27T21:42:44.474 INFO:teuthology.orchestra.run.smithi060.stdout: 411469 40 -rw-r--r-- 1 ubuntu ubuntu 39652 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSDaemon.cc 2024-03-27T21:42:44.475 INFO:teuthology.orchestra.run.smithi060.stdout: 411524 4 -rw-r--r-- 1 ubuntu ubuntu 4006 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/SnapServer.h 2024-03-27T21:42:44.475 INFO:teuthology.orchestra.run.smithi060.stdout: 411509 4 -rw-r--r-- 1 ubuntu ubuntu 2457 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/ScrubHeader.h 2024-03-27T21:42:44.475 INFO:teuthology.orchestra.run.smithi060.stdout: 411427 4 -rw-r--r-- 1 ubuntu ubuntu 895 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/BatchOp.h 2024-03-27T21:42:44.475 INFO:teuthology.orchestra.run.smithi060.stdout: 411455 4 -rw-r--r-- 1 ubuntu ubuntu 3242 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/LogSegment.h 2024-03-27T21:42:44.475 INFO:teuthology.orchestra.run.smithi060.stdout: 411468 4 -rw-r--r-- 1 ubuntu ubuntu 924 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSContinuation.h 2024-03-27T21:42:44.475 INFO:teuthology.orchestra.run.smithi060.stdout: 411441 8 -rw-r--r-- 1 ubuntu ubuntu 4946 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/DamageTable.h 2024-03-27T21:42:44.475 INFO:teuthology.orchestra.run.smithi060.stdout: 411514 24 -rw-r--r-- 1 ubuntu ubuntu 24248 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/Server.h 2024-03-27T21:42:44.475 INFO:teuthology.orchestra.run.smithi060.stdout: 411487 4 -rw-r--r-- 1 ubuntu ubuntu 926 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/Mantle.h 2024-03-27T21:42:44.476 INFO:teuthology.orchestra.run.smithi060.stdout: 411497 8 -rw-r--r-- 1 ubuntu ubuntu 4245 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/OpenFileTable.h 2024-03-27T21:42:44.476 INFO:teuthology.orchestra.run.smithi060.stdout: 411500 12 -rw-r--r-- 1 ubuntu ubuntu 8535 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/QuiesceAgent.cc 2024-03-27T21:42:44.476 INFO:teuthology.orchestra.run.smithi060.stdout: 411461 8 -rw-r--r-- 1 ubuntu ubuntu 7718 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDLog.h 2024-03-27T21:42:44.476 INFO:teuthology.orchestra.run.smithi060.stdout: 411434 28 -rw-r--r-- 1 ubuntu ubuntu 25872 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/CDir.h 2024-03-27T21:42:44.476 INFO:teuthology.orchestra.run.smithi060.stdout: 411428 20 -rw-r--r-- 1 ubuntu ubuntu 17497 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/Beacon.cc 2024-03-27T21:42:44.476 INFO:teuthology.orchestra.run.smithi060.stdout: 411489 4 -rw-r--r-- 1 ubuntu ubuntu 2612 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MetricAggregator.h 2024-03-27T21:42:44.476 INFO:teuthology.orchestra.run.smithi060.stdout: 411476 4 -rw-r--r-- 1 ubuntu ubuntu 1669 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSPinger.h 2024-03-27T21:42:44.476 INFO:teuthology.orchestra.run.smithi060.stdout: 411448 4 -rw-r--r-- 1 ubuntu ubuntu 3216 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/JournalPointer.cc 2024-03-27T21:42:44.477 INFO:teuthology.orchestra.run.smithi060.stdout: 411458 424 -rw-r--r-- 1 ubuntu ubuntu 433263 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDCache.cc 2024-03-27T21:42:44.477 INFO:teuthology.orchestra.run.smithi060.stdout: 411516 24 -rw-r--r-- 1 ubuntu ubuntu 23677 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/SessionMap.h 2024-03-27T21:42:44.477 INFO:teuthology.orchestra.run.smithi060.stdout: 411520 4 -rw-r--r-- 1 ubuntu ubuntu 3507 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/SnapClient.h 2024-03-27T21:42:44.477 INFO:teuthology.orchestra.run.smithi060.stdout: 411552 12 -rw-r--r-- 1 ubuntu ubuntu 10330 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/flock.h 2024-03-27T21:42:44.477 INFO:teuthology.orchestra.run.smithi060.stdout: 411529 4 -rw-r--r-- 1 ubuntu ubuntu 1702 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/cephfs_features.cc 2024-03-27T21:42:44.477 INFO:teuthology.orchestra.run.smithi060.stdout: 411443 20 -rw-r--r-- 1 ubuntu ubuntu 18374 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/FSMap.h 2024-03-27T21:42:44.477 INFO:teuthology.orchestra.run.smithi060.stdout: 411440 8 -rw-r--r-- 1 ubuntu ubuntu 7557 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/DamageTable.cc 2024-03-27T21:42:44.477 INFO:teuthology.orchestra.run.smithi060.stdout: 411485 4 -rw-r--r-- 1 ubuntu ubuntu 3850 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSTableServer.h 2024-03-27T21:42:44.478 INFO:teuthology.orchestra.run.smithi060.stdout: 411466 4 -rw-r--r-- 1 ubuntu ubuntu 3707 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSContext.cc 2024-03-27T21:42:44.478 INFO:teuthology.orchestra.run.smithi060.stdout: 411449 4 -rw-r--r-- 1 ubuntu ubuntu 2067 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/JournalPointer.h 2024-03-27T21:42:44.478 INFO:teuthology.orchestra.run.smithi060.stdout: 411499 8 -rw-r--r-- 1 ubuntu ubuntu 6390 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/PurgeQueue.h 2024-03-27T21:42:44.478 INFO:teuthology.orchestra.run.smithi060.stdout: 411424 4 -rw-r--r-- 1 ubuntu ubuntu 1464 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/Anchor.cc 2024-03-27T21:42:44.478 INFO:teuthology.orchestra.run.smithi060.stdout: 411510 36 -rw-r--r-- 1 ubuntu ubuntu 33537 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/ScrubStack.cc 2024-03-27T21:42:44.478 INFO:teuthology.orchestra.run.smithi060.stdout: 411563 4 -rw-r--r-- 1 ubuntu ubuntu 3321 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/snap.h 2024-03-27T21:42:44.478 INFO:teuthology.orchestra.run.smithi060.stdout: 411439 12 -rw-r--r-- 1 ubuntu ubuntu 11877 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/Capability.h 2024-03-27T21:42:44.478 INFO:teuthology.orchestra.run.smithi060.stdout: 411560 24 -rw-r--r-- 1 ubuntu ubuntu 23797 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/mdstypes.cc 2024-03-27T21:42:44.479 INFO:teuthology.orchestra.run.smithi060.stdout: 411454 4 -rw-r--r-- 1 ubuntu ubuntu 3479 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/LogEvent.h 2024-03-27T21:42:44.479 INFO:teuthology.orchestra.run.smithi060.stdout: 411504 40 -rw-r--r-- 1 ubuntu ubuntu 38833 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/QuiesceDbManager.cc 2024-03-27T21:42:44.479 INFO:teuthology.orchestra.run.smithi060.stdout: 411483 4 -rw-r--r-- 1 ubuntu ubuntu 2713 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSTableClient.h 2024-03-27T21:42:44.479 INFO:teuthology.orchestra.run.smithi060.stdout: 411442 36 -rw-r--r-- 1 ubuntu ubuntu 36123 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/FSMap.cc 2024-03-27T21:42:44.479 INFO:teuthology.orchestra.run.smithi060.stdout: 411475 4 -rw-r--r-- 1 ubuntu ubuntu 3352 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSPinger.cc 2024-03-27T21:42:44.479 INFO:teuthology.orchestra.run.smithi060.stdout: 411470 8 -rw-r--r-- 1 ubuntu ubuntu 4247 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/MDSDaemon.h 2024-03-27T21:42:44.479 INFO:teuthology.orchestra.run.smithi060.stdout: 411555 4 -rw-r--r-- 1 ubuntu ubuntu 3291 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/inode_backtrace.h 2024-03-27T21:42:44.479 INFO:teuthology.orchestra.run.smithi060.stdout: 411526 8 -rw-r--r-- 1 ubuntu ubuntu 6539 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/StrayManager.h 2024-03-27T21:42:44.479 INFO:teuthology.orchestra.run.smithi060.stdout: 411436 44 -rw-r--r-- 1 ubuntu ubuntu 44366 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/CInode.h 2024-03-27T21:42:44.480 INFO:teuthology.orchestra.run.smithi060.stdout: 411553 4 -rw-r--r-- 1 ubuntu ubuntu 995 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/fscrypt.h 2024-03-27T21:42:44.480 INFO:teuthology.orchestra.run.smithi060.stdout: 411494 16 -rw-r--r-- 1 ubuntu ubuntu 15450 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mds/Mutation.cc 2024-03-27T21:42:44.480 INFO:teuthology.orchestra.run.smithi060.stdout: 409506 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client 2024-03-27T21:42:44.480 INFO:teuthology.orchestra.run.smithi060.stdout: 409520 12 -rw-r--r-- 1 ubuntu ubuntu 9576 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/Inode.h 2024-03-27T21:42:44.480 INFO:teuthology.orchestra.run.smithi060.stdout: 409507 4 -rw-r--r-- 1 ubuntu ubuntu 240 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/CMakeLists.txt 2024-03-27T21:42:44.480 INFO:teuthology.orchestra.run.smithi060.stdout: 409512 4 -rw-r--r-- 1 ubuntu ubuntu 3545 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/Delegation.cc 2024-03-27T21:42:44.480 INFO:teuthology.orchestra.run.smithi060.stdout: 409522 4 -rw-r--r-- 1 ubuntu ubuntu 2337 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/MetaRequest.cc 2024-03-27T21:42:44.480 INFO:teuthology.orchestra.run.smithi060.stdout: 409537 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/hypertable 2024-03-27T21:42:44.481 INFO:teuthology.orchestra.run.smithi060.stdout: 409538 16 -rw-r--r-- 1 ubuntu ubuntu 14707 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/hypertable/CephBroker.cc 2024-03-27T21:42:44.481 INFO:teuthology.orchestra.run.smithi060.stdout: 409539 4 -rw-r--r-- 1 ubuntu ubuntu 3854 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/hypertable/CephBroker.h 2024-03-27T21:42:44.481 INFO:teuthology.orchestra.run.smithi060.stdout: 409524 4 -rw-r--r-- 1 ubuntu ubuntu 1613 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/MetaSession.cc 2024-03-27T21:42:44.481 INFO:teuthology.orchestra.run.smithi060.stdout: 409509 68 -rw-r--r-- 1 ubuntu ubuntu 69571 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/Client.h 2024-03-27T21:42:44.481 INFO:teuthology.orchestra.run.smithi060.stdout: 409511 4 -rw-r--r-- 1 ubuntu ubuntu 1649 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/ClientSnapRealm.h 2024-03-27T21:42:44.481 INFO:teuthology.orchestra.run.smithi060.stdout: 409523 8 -rw-r--r-- 1 ubuntu ubuntu 7195 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/MetaRequest.h 2024-03-27T21:42:44.481 INFO:teuthology.orchestra.run.smithi060.stdout: 409527 8 -rw-r--r-- 1 ubuntu ubuntu 7286 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/RWRef.h 2024-03-27T21:42:44.481 INFO:teuthology.orchestra.run.smithi060.stdout: 409536 4 -rw-r--r-- 1 ubuntu ubuntu 701 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/fuse_ll.h 2024-03-27T21:42:44.481 INFO:teuthology.orchestra.run.smithi060.stdout: 409542 4 -rw-r--r-- 1 ubuntu ubuntu 1001 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/posix_acl.h 2024-03-27T21:42:44.482 INFO:teuthology.orchestra.run.smithi060.stdout: 409530 4 -rw-r--r-- 1 ubuntu ubuntu 1702 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/Trace.cc 2024-03-27T21:42:44.482 INFO:teuthology.orchestra.run.smithi060.stdout: 409514 4 -rw-r--r-- 1 ubuntu ubuntu 879 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/Dentry.cc 2024-03-27T21:42:44.482 INFO:teuthology.orchestra.run.smithi060.stdout: 409519 24 -rw-r--r-- 1 ubuntu ubuntu 22309 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/Inode.cc 2024-03-27T21:42:44.482 INFO:teuthology.orchestra.run.smithi060.stdout: 409516 4 -rw-r--r-- 1 ubuntu ubuntu 416 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/Dir.h 2024-03-27T21:42:44.482 INFO:teuthology.orchestra.run.smithi060.stdout: 409515 4 -rw-r--r-- 1 ubuntu ubuntu 2735 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/Dentry.h 2024-03-27T21:42:44.482 INFO:teuthology.orchestra.run.smithi060.stdout: 409518 4 -rw-r--r-- 1 ubuntu ubuntu 1539 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/Fh.h 2024-03-27T21:42:44.482 INFO:teuthology.orchestra.run.smithi060.stdout: 409513 4 -rw-r--r-- 1 ubuntu ubuntu 1493 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/Delegation.h 2024-03-27T21:42:44.482 INFO:teuthology.orchestra.run.smithi060.stdout: 409526 4 -rw-r--r-- 1 ubuntu ubuntu 2406 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/ObjecterWriteback.h 2024-03-27T21:42:44.483 INFO:teuthology.orchestra.run.smithi060.stdout: 409529 8 -rw-r--r-- 1 ubuntu ubuntu 7239 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/SyntheticClient.h 2024-03-27T21:42:44.483 INFO:teuthology.orchestra.run.smithi060.stdout: 409510 4 -rw-r--r-- 1 ubuntu ubuntu 1923 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/ClientSnapRealm.cc 2024-03-27T21:42:44.483 INFO:teuthology.orchestra.run.smithi060.stdout: 409517 4 -rw-r--r-- 1 ubuntu ubuntu 678 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/Fh.cc 2024-03-27T21:42:44.483 INFO:teuthology.orchestra.run.smithi060.stdout: 409540 4 -rw-r--r-- 1 ubuntu ubuntu 1482 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/ioctl.h 2024-03-27T21:42:44.483 INFO:teuthology.orchestra.run.smithi060.stdout: 409528 96 -rw-r--r-- 1 ubuntu ubuntu 95004 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/SyntheticClient.cc 2024-03-27T21:42:44.483 INFO:teuthology.orchestra.run.smithi060.stdout: 409533 8 -rw-r--r-- 1 ubuntu ubuntu 4594 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/barrier.cc 2024-03-27T21:42:44.483 INFO:teuthology.orchestra.run.smithi060.stdout: 409543 4 -rw-r--r-- 1 ubuntu ubuntu 3691 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/test_ioctls.c 2024-03-27T21:42:44.483 INFO:teuthology.orchestra.run.smithi060.stdout: 409508 488 -rw-r--r-- 1 ubuntu ubuntu 496935 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/Client.cc 2024-03-27T21:42:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: 409521 4 -rw-r--r-- 1 ubuntu ubuntu 341 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/InodeRef.h 2024-03-27T21:42:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: 409534 4 -rw-r--r-- 1 ubuntu ubuntu 2273 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/barrier.h 2024-03-27T21:42:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: 409541 8 -rw-r--r-- 1 ubuntu ubuntu 6553 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/posix_acl.cc 2024-03-27T21:42:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: 409525 4 -rw-r--r-- 1 ubuntu ubuntu 2069 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/MetaSession.h 2024-03-27T21:42:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: 409531 4 -rw-r--r-- 1 ubuntu ubuntu 1348 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/Trace.h 2024-03-27T21:42:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: 409532 4 -rw-r--r-- 1 ubuntu ubuntu 2311 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/UserPerm.h 2024-03-27T21:42:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: 409535 52 -rw-r--r-- 1 ubuntu ubuntu 50224 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/client/fuse_ll.cc 2024-03-27T21:42:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: 411867 4 -rwxr-xr-x 1 ubuntu ubuntu 3340 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mount.fuse.ceph 2024-03-27T21:42:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: 415286 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rocksdb 2024-03-27T21:42:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: 409391 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm 2024-03-27T21:42:44.485 INFO:teuthology.orchestra.run.smithi060.stdout: 409482 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/tests 2024-03-27T21:42:44.485 INFO:teuthology.orchestra.run.smithi060.stdout: 409489 112 -rw-r--r-- 1 ubuntu ubuntu 113714 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/tests/test_cephadm.py 2024-03-27T21:42:44.485 INFO:teuthology.orchestra.run.smithi060.stdout: 409493 24 -rw-r--r-- 1 ubuntu ubuntu 23162 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/tests/test_deploy.py 2024-03-27T21:42:44.485 INFO:teuthology.orchestra.run.smithi060.stdout: 409491 12 -rw-r--r-- 1 ubuntu ubuntu 9921 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/tests/test_custom_container.py 2024-03-27T21:42:44.485 INFO:teuthology.orchestra.run.smithi060.stdout: 409500 32 -rw-r--r-- 1 ubuntu ubuntu 29904 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/tests/test_util_funcs.py 2024-03-27T21:42:44.485 INFO:teuthology.orchestra.run.smithi060.stdout: 409492 4 -rw-r--r-- 1 ubuntu ubuntu 2539 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/tests/test_daemon_form.py 2024-03-27T21:42:44.485 INFO:teuthology.orchestra.run.smithi060.stdout: 409495 12 -rw-r--r-- 1 ubuntu ubuntu 9851 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/tests/test_ingress.py 2024-03-27T21:42:44.485 INFO:teuthology.orchestra.run.smithi060.stdout: 409498 8 -rw-r--r-- 1 ubuntu ubuntu 6702 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/tests/test_nfs.py 2024-03-27T21:42:44.486 INFO:teuthology.orchestra.run.smithi060.stdout: 409490 4 -rw-r--r-- 1 ubuntu ubuntu 2000 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/tests/test_container_engine.py 2024-03-27T21:42:44.486 INFO:teuthology.orchestra.run.smithi060.stdout: 409499 8 -rw-r--r-- 1 ubuntu ubuntu 5276 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/tests/test_unit_file.py 2024-03-27T21:42:44.486 INFO:teuthology.orchestra.run.smithi060.stdout: 409497 16 -rw-r--r-- 1 ubuntu ubuntu 13684 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/tests/test_networks.py 2024-03-27T21:42:44.486 INFO:teuthology.orchestra.run.smithi060.stdout: 409483 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/tests/__init__.py 2024-03-27T21:42:44.486 INFO:teuthology.orchestra.run.smithi060.stdout: 409494 4 -rw-r--r-- 1 ubuntu ubuntu 2354 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/tests/test_enclosure.py 2024-03-27T21:42:44.486 INFO:teuthology.orchestra.run.smithi060.stdout: 409487 12 -rw-r--r-- 1 ubuntu ubuntu 9767 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/tests/fixtures.py 2024-03-27T21:42:44.486 INFO:teuthology.orchestra.run.smithi060.stdout: 409484 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/tests/build 2024-03-27T21:42:44.486 INFO:teuthology.orchestra.run.smithi060.stdout: 409486 8 -rw-r--r-- 1 ubuntu ubuntu 6146 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/tests/build/test_cephadm_build.py 2024-03-27T21:42:44.487 INFO:teuthology.orchestra.run.smithi060.stdout: 409485 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/tests/build/__init__.py 2024-03-27T21:42:44.487 INFO:teuthology.orchestra.run.smithi060.stdout: 409488 32 -rw-r--r-- 1 ubuntu ubuntu 30728 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/tests/test_agent.py 2024-03-27T21:42:44.487 INFO:teuthology.orchestra.run.smithi060.stdout: 409496 4 -rw-r--r-- 1 ubuntu ubuntu 1551 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/tests/test_logrotate_config.py 2024-03-27T21:42:44.487 INFO:teuthology.orchestra.run.smithi060.stdout: 409393 4 -rw-r--r-- 1 ubuntu ubuntu 1001 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/CMakeLists.txt 2024-03-27T21:42:44.487 INFO:teuthology.orchestra.run.smithi060.stdout: 409503 4 -rwxr-xr-x 1 ubuntu ubuntu 2331 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/vstart-smoke.sh 2024-03-27T21:42:44.487 INFO:teuthology.orchestra.run.smithi060.stdout: 409501 4 -rw-r--r-- 1 ubuntu ubuntu 2078 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/tox.ini 2024-03-27T21:42:44.487 INFO:teuthology.orchestra.run.smithi060.stdout: 409394 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/box 2024-03-27T21:42:44.487 INFO:teuthology.orchestra.run.smithi060.stdout: 409409 16 -rw-r--r-- 1 ubuntu ubuntu 13334 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/box/util.py 2024-03-27T21:42:44.487 INFO:teuthology.orchestra.run.smithi060.stdout: 409407 4 -rw-r--r-- 1 ubuntu ubuntu 4095 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/box/host.py 2024-03-27T21:42:44.488 INFO:teuthology.orchestra.run.smithi060.stdout: 409399 4 -rw-r--r-- 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/box/daemon.json 2024-03-27T21:42:44.488 INFO:teuthology.orchestra.run.smithi060.stdout: 409402 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/box/docker 2024-03-27T21:42:44.488 INFO:teuthology.orchestra.run.smithi060.stdout: 409403 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/box/docker/ceph 2024-03-27T21:42:44.488 INFO:teuthology.orchestra.run.smithi060.stdout: 409404 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/box/docker/ceph/.bashrc 2024-03-27T21:42:44.488 INFO:teuthology.orchestra.run.smithi060.stdout: 409406 4 -rw-r--r-- 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/box/docker/ceph/locale.conf 2024-03-27T21:42:44.488 INFO:teuthology.orchestra.run.smithi060.stdout: 409405 4 -rw-r--r-- 1 ubuntu ubuntu 75 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/box/docker/ceph/Dockerfile 2024-03-27T21:42:44.488 INFO:teuthology.orchestra.run.smithi060.stdout: 409397 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/box/__init__.py 2024-03-27T21:42:44.488 INFO:teuthology.orchestra.run.smithi060.stdout: 409408 8 -rw-r--r-- 1 ubuntu ubuntu 5262 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/box/osd.py 2024-03-27T21:42:44.489 INFO:teuthology.orchestra.run.smithi060.stdout: 409401 4 -rw-r--r-- 1 ubuntu ubuntu 250 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/box/docker-compose.cgroup1.yml 2024-03-27T21:42:44.489 INFO:teuthology.orchestra.run.smithi060.stdout: 409400 4 -rw-r--r-- 1 ubuntu ubuntu 792 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/box/docker-compose-docker.yml 2024-03-27T21:42:44.489 INFO:teuthology.orchestra.run.smithi060.stdout: 409398 16 -rwxr-xr-x 1 ubuntu ubuntu 14100 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/box/box.py 2024-03-27T21:42:44.489 INFO:teuthology.orchestra.run.smithi060.stdout: 409396 4 -rw-r--r-- 1 ubuntu ubuntu 3040 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/box/DockerfilePodman 2024-03-27T21:42:44.489 INFO:teuthology.orchestra.run.smithi060.stdout: 409395 4 -rw-r--r-- 1 ubuntu ubuntu 1129 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/box/DockerfileDocker 2024-03-27T21:42:44.489 INFO:teuthology.orchestra.run.smithi060.stdout: 409468 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/containers 2024-03-27T21:42:44.489 INFO:teuthology.orchestra.run.smithi060.stdout: 409469 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/containers/keepalived 2024-03-27T21:42:44.489 INFO:teuthology.orchestra.run.smithi060.stdout: 409472 12 -rw-r--r-- 1 ubuntu ubuntu 11431 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/containers/keepalived/README.md 2024-03-27T21:42:44.490 INFO:teuthology.orchestra.run.smithi060.stdout: 409471 4 -rw-r--r-- 1 ubuntu ubuntu 1079 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/containers/keepalived/LICENSE 2024-03-27T21:42:44.490 INFO:teuthology.orchestra.run.smithi060.stdout: 409470 4 -rw-r--r-- 1 ubuntu ubuntu 639 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/containers/keepalived/Dockerfile 2024-03-27T21:42:44.490 INFO:teuthology.orchestra.run.smithi060.stdout: 409473 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/containers/keepalived/skel 2024-03-27T21:42:44.490 INFO:teuthology.orchestra.run.smithi060.stdout: 409474 4 -rwxr-xr-x 1 ubuntu ubuntu 557 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/containers/keepalived/skel/init.sh 2024-03-27T21:42:44.490 INFO:teuthology.orchestra.run.smithi060.stdout: 409410 20 -rwxr-xr-x 1 ubuntu ubuntu 18744 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/build.py 2024-03-27T21:42:44.490 INFO:teuthology.orchestra.run.smithi060.stdout: 409502 4 -rwxr-xr-x 1 ubuntu ubuntu 146 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/vstart-cleanup.sh 2024-03-27T21:42:44.490 INFO:teuthology.orchestra.run.smithi060.stdout: 409413 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib 2024-03-27T21:42:44.490 INFO:teuthology.orchestra.run.smithi060.stdout: 409423 8 -rw-r--r-- 1 ubuntu ubuntu 5986 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/context_getters.py 2024-03-27T21:42:44.491 INFO:teuthology.orchestra.run.smithi060.stdout: 409453 8 -rw-r--r-- 1 ubuntu ubuntu 6773 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/ssh.py 2024-03-27T21:42:44.491 INFO:teuthology.orchestra.run.smithi060.stdout: 409439 8 -rw-r--r-- 1 ubuntu ubuntu 5744 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/data_utils.py 2024-03-27T21:42:44.491 INFO:teuthology.orchestra.run.smithi060.stdout: 409448 8 -rw-r--r-- 1 ubuntu ubuntu 7335 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/locking.py 2024-03-27T21:42:44.491 INFO:teuthology.orchestra.run.smithi060.stdout: 409418 8 -rw-r--r-- 1 ubuntu ubuntu 6770 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/container_daemon_form.py 2024-03-27T21:42:44.491 INFO:teuthology.orchestra.run.smithi060.stdout: 409446 8 -rw-r--r-- 1 ubuntu ubuntu 5912 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/firewalld.py 2024-03-27T21:42:44.491 INFO:teuthology.orchestra.run.smithi060.stdout: 409422 4 -rw-r--r-- 1 ubuntu ubuntu 2329 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/context.py 2024-03-27T21:42:44.491 INFO:teuthology.orchestra.run.smithi060.stdout: 409450 12 -rw-r--r-- 1 ubuntu ubuntu 9532 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/net_utils.py 2024-03-27T21:42:44.491 INFO:teuthology.orchestra.run.smithi060.stdout: 409445 8 -rw-r--r-- 1 ubuntu ubuntu 4914 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/file_utils.py 2024-03-27T21:42:44.492 INFO:teuthology.orchestra.run.smithi060.stdout: 409421 24 -rw-r--r-- 1 ubuntu ubuntu 20620 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/container_types.py 2024-03-27T21:42:44.492 INFO:teuthology.orchestra.run.smithi060.stdout: 409451 20 -rw-r--r-- 1 ubuntu ubuntu 20185 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/packagers.py 2024-03-27T21:42:44.492 INFO:teuthology.orchestra.run.smithi060.stdout: 409420 8 -rw-r--r-- 1 ubuntu ubuntu 6468 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/container_engines.py 2024-03-27T21:42:44.492 INFO:teuthology.orchestra.run.smithi060.stdout: 409467 8 -rw-r--r-- 1 ubuntu ubuntu 5426 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/templating.py 2024-03-27T21:42:44.492 INFO:teuthology.orchestra.run.smithi060.stdout: 409416 12 -rw-r--r-- 1 ubuntu ubuntu 11007 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/call_wrappers.py 2024-03-27T21:42:44.492 INFO:teuthology.orchestra.run.smithi060.stdout: 409444 4 -rw-r--r-- 1 ubuntu ubuntu 1607 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/exe_utils.py 2024-03-27T21:42:44.492 INFO:teuthology.orchestra.run.smithi060.stdout: 409442 4 -rw-r--r-- 1 ubuntu ubuntu 1364 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/deployment_utils.py 2024-03-27T21:42:44.492 INFO:teuthology.orchestra.run.smithi060.stdout: 409426 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/daemons 2024-03-27T21:42:44.492 INFO:teuthology.orchestra.run.smithi060.stdout: 409432 16 -rw-r--r-- 1 ubuntu ubuntu 14354 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/daemons/monitoring.py 2024-03-27T21:42:44.493 INFO:teuthology.orchestra.run.smithi060.stdout: 409430 12 -rw-r--r-- 1 ubuntu ubuntu 9443 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/daemons/ingress.py 2024-03-27T21:42:44.493 INFO:teuthology.orchestra.run.smithi060.stdout: 409431 12 -rw-r--r-- 1 ubuntu ubuntu 9206 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/daemons/iscsi.py 2024-03-27T21:42:44.493 INFO:teuthology.orchestra.run.smithi060.stdout: 409429 8 -rw-r--r-- 1 ubuntu ubuntu 7377 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/daemons/custom.py 2024-03-27T21:42:44.493 INFO:teuthology.orchestra.run.smithi060.stdout: 409434 8 -rw-r--r-- 1 ubuntu ubuntu 5065 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/daemons/node_proxy.py 2024-03-27T21:42:44.493 INFO:teuthology.orchestra.run.smithi060.stdout: 409428 16 -rw-r--r-- 1 ubuntu ubuntu 16214 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/daemons/ceph.py 2024-03-27T21:42:44.493 INFO:teuthology.orchestra.run.smithi060.stdout: 409433 8 -rw-r--r-- 1 ubuntu ubuntu 7640 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/daemons/nfs.py 2024-03-27T21:42:44.493 INFO:teuthology.orchestra.run.smithi060.stdout: 409435 8 -rw-r--r-- 1 ubuntu ubuntu 6593 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/daemons/nvmeof.py 2024-03-27T21:42:44.493 INFO:teuthology.orchestra.run.smithi060.stdout: 409438 4 -rw-r--r-- 1 ubuntu ubuntu 4015 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/daemons/tracing.py 2024-03-27T21:42:44.493 INFO:teuthology.orchestra.run.smithi060.stdout: 409427 4 -rw-r--r-- 1 ubuntu ubuntu 603 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/daemons/__init__.py 2024-03-27T21:42:44.494 INFO:teuthology.orchestra.run.smithi060.stdout: 409436 16 -rw-r--r-- 1 ubuntu ubuntu 15086 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/daemons/smb.py 2024-03-27T21:42:44.494 INFO:teuthology.orchestra.run.smithi060.stdout: 409437 8 -rw-r--r-- 1 ubuntu ubuntu 7777 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/daemons/snmp.py 2024-03-27T21:42:44.494 INFO:teuthology.orchestra.run.smithi060.stdout: 409414 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/__init__.py 2024-03-27T21:42:44.494 INFO:teuthology.orchestra.run.smithi060.stdout: 409424 8 -rw-r--r-- 1 ubuntu ubuntu 4259 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/daemon_form.py 2024-03-27T21:42:44.494 INFO:teuthology.orchestra.run.smithi060.stdout: 409449 12 -rw-r--r-- 1 ubuntu ubuntu 8220 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/logging.py 2024-03-27T21:42:44.494 INFO:teuthology.orchestra.run.smithi060.stdout: 409456 12 -rw-r--r-- 1 ubuntu ubuntu 8244 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/systemd_unit.py 2024-03-27T21:42:44.494 INFO:teuthology.orchestra.run.smithi060.stdout: 409425 8 -rw-r--r-- 1 ubuntu ubuntu 7777 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/daemon_identity.py 2024-03-27T21:42:44.494 INFO:teuthology.orchestra.run.smithi060.stdout: 409452 12 -rw-r--r-- 1 ubuntu ubuntu 8543 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/runscripts.py 2024-03-27T21:42:44.495 INFO:teuthology.orchestra.run.smithi060.stdout: 409419 4 -rw-r--r-- 1 ubuntu ubuntu 551 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/container_engine_base.py 2024-03-27T21:42:44.495 INFO:teuthology.orchestra.run.smithi060.stdout: 409417 4 -rw-r--r-- 1 ubuntu ubuntu 2289 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/constants.py 2024-03-27T21:42:44.495 INFO:teuthology.orchestra.run.smithi060.stdout: 409457 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/templates 2024-03-27T21:42:44.495 INFO:teuthology.orchestra.run.smithi060.stdout: 409464 4 -rw-r--r-- 1 ubuntu ubuntu 1267 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/templates/init_ctr.service.j2 2024-03-27T21:42:44.495 INFO:teuthology.orchestra.run.smithi060.stdout: 409459 4 -rw-r--r-- 1 ubuntu ubuntu 1201 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/templates/ceph.service.j2 2024-03-27T21:42:44.495 INFO:teuthology.orchestra.run.smithi060.stdout: 409462 4 -rw-r--r-- 1 ubuntu ubuntu 205 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/templates/dropin.service.j2 2024-03-27T21:42:44.495 INFO:teuthology.orchestra.run.smithi060.stdout: 409460 4 -rw-r--r-- 1 ubuntu ubuntu 133 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/templates/cephadm.logrotate.config.j2 2024-03-27T21:42:44.495 INFO:teuthology.orchestra.run.smithi060.stdout: 409466 4 -rw-r--r-- 1 ubuntu ubuntu 1034 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/templates/sidecar.service.j2 2024-03-27T21:42:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: 409463 4 -rw-r--r-- 1 ubuntu ubuntu 1488 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/templates/init_containers.run.j2 2024-03-27T21:42:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: 409465 4 -rw-r--r-- 1 ubuntu ubuntu 508 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/templates/sidecar.run.j2 2024-03-27T21:42:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: 409458 4 -rw-r--r-- 1 ubuntu ubuntu 307 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/templates/agent.service.j2 2024-03-27T21:42:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: 409461 4 -rw-r--r-- 1 ubuntu ubuntu 280 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/templates/cluster.logrotate.config.j2 2024-03-27T21:42:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: 409455 4 -rw-r--r-- 1 ubuntu ubuntu 2536 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/systemd.py 2024-03-27T21:42:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: 409440 4 -rw-r--r-- 1 ubuntu ubuntu 1440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/decorators.py 2024-03-27T21:42:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: 409447 36 -rw-r--r-- 1 ubuntu ubuntu 34087 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/host_facts.py 2024-03-27T21:42:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: 409443 4 -rw-r--r-- 1 ubuntu ubuntu 270 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/exceptions.py 2024-03-27T21:42:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: 409454 8 -rw-r--r-- 1 ubuntu ubuntu 4288 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/sysctl.py 2024-03-27T21:42:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: 409441 4 -rw-r--r-- 1 ubuntu ubuntu 388 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/deploy.py 2024-03-27T21:42:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: 409415 4 -rw-r--r-- 1 ubuntu ubuntu 1050 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadmlib/agent.py 2024-03-27T21:42:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: 409412 212 -rwxr-xr-x 1 ubuntu ubuntu 214311 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/cephadm.py 2024-03-27T21:42:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: 409475 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/samples 2024-03-27T21:42:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: 409479 4 -rw-r--r-- 1 ubuntu ubuntu 293 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/samples/nfs.json 2024-03-27T21:42:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: 409481 8 -rw-r--r-- 1 ubuntu ubuntu 6330 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/samples/rgw_ssl.json 2024-03-27T21:42:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: 409476 4 -rw-r--r-- 1 ubuntu ubuntu 747 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/samples/alertmanager.json 2024-03-27T21:42:44.498 INFO:teuthology.orchestra.run.smithi060.stdout: 409480 4 -rw-r--r-- 1 ubuntu ubuntu 340 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/samples/prometheus.json 2024-03-27T21:42:44.498 INFO:teuthology.orchestra.run.smithi060.stdout: 409478 8 -rw-r--r-- 1 ubuntu ubuntu 4811 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/samples/grafana.json 2024-03-27T21:42:44.498 INFO:teuthology.orchestra.run.smithi060.stdout: 409477 4 -rw-r--r-- 1 ubuntu ubuntu 1011 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/samples/custom_container.json 2024-03-27T21:42:44.498 INFO:teuthology.orchestra.run.smithi060.stdout: 409392 4 -rw-r--r-- 1 ubuntu ubuntu 63 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/.gitignore 2024-03-27T21:42:44.498 INFO:teuthology.orchestra.run.smithi060.stdout: 409504 4 -rw-r--r-- 1 ubuntu ubuntu 783 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/zipapp-reqs.txt 2024-03-27T21:42:44.498 INFO:teuthology.orchestra.run.smithi060.stdout: 409411 4 -rwxr-xr-x 1 ubuntu ubuntu 120 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cephadm/build.sh 2024-03-27T21:42:44.498 INFO:teuthology.orchestra.run.smithi060.stdout: 411981 4 -rw-r--r-- 1 ubuntu ubuntu 400 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mypy-constrains.txt 2024-03-27T21:42:44.498 INFO:teuthology.orchestra.run.smithi060.stdout: 408971 4 -rw-r--r-- 1 ubuntu ubuntu 133 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/.gitignore 2024-03-27T21:42:44.499 INFO:teuthology.orchestra.run.smithi060.stdout: 409071 8 -rwxr-xr-x 1 ubuntu ubuntu 4192 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-crash.in 2024-03-27T21:42:44.499 INFO:teuthology.orchestra.run.smithi060.stdout: 414724 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay 2024-03-27T21:42:44.499 INFO:teuthology.orchestra.run.smithi060.stdout: 414743 4 -rw-r--r-- 1 ubuntu ubuntu 2670 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay/rbd_loc.cc 2024-03-27T21:42:44.499 INFO:teuthology.orchestra.run.smithi060.stdout: 414744 4 -rw-r--r-- 1 ubuntu ubuntu 2490 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay/rbd_loc.hpp 2024-03-27T21:42:44.499 INFO:teuthology.orchestra.run.smithi060.stdout: 414730 4 -rw-r--r-- 1 ubuntu ubuntu 1059 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay/CMakeLists.txt 2024-03-27T21:42:44.499 INFO:teuthology.orchestra.run.smithi060.stdout: 414740 12 -rw-r--r-- 1 ubuntu ubuntu 8880 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay/ios.hpp 2024-03-27T21:42:44.499 INFO:teuthology.orchestra.run.smithi060.stdout: 414726 12 -rw-r--r-- 1 ubuntu ubuntu 10290 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay/ActionTypes.h 2024-03-27T21:42:44.499 INFO:teuthology.orchestra.run.smithi060.stdout: 414733 4 -rw-r--r-- 1 ubuntu ubuntu 1220 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay/PendingIO.cc 2024-03-27T21:42:44.500 INFO:teuthology.orchestra.run.smithi060.stdout: 414742 8 -rw-r--r-- 1 ubuntu ubuntu 4707 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay/rbd-replay.cc 2024-03-27T21:42:44.500 INFO:teuthology.orchestra.run.smithi060.stdout: 414734 4 -rw-r--r-- 1 ubuntu ubuntu 1474 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay/PendingIO.hpp 2024-03-27T21:42:44.500 INFO:teuthology.orchestra.run.smithi060.stdout: 414729 4 -rw-r--r-- 1 ubuntu ubuntu 773 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay/BufferReader.h 2024-03-27T21:42:44.500 INFO:teuthology.orchestra.run.smithi060.stdout: 414732 4 -rw-r--r-- 1 ubuntu ubuntu 1287 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay/ImageNameMap.hpp 2024-03-27T21:42:44.500 INFO:teuthology.orchestra.run.smithi060.stdout: 414741 20 -rw-r--r-- 1 ubuntu ubuntu 20311 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay/rbd-replay-prep.cc 2024-03-27T21:42:44.500 INFO:teuthology.orchestra.run.smithi060.stdout: 414737 12 -rw-r--r-- 1 ubuntu ubuntu 8538 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay/actions.cc 2024-03-27T21:42:44.500 INFO:teuthology.orchestra.run.smithi060.stdout: 414739 8 -rw-r--r-- 1 ubuntu ubuntu 7319 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay/ios.cc 2024-03-27T21:42:44.500 INFO:teuthology.orchestra.run.smithi060.stdout: 414735 12 -rw-r--r-- 1 ubuntu ubuntu 11257 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay/Replayer.cc 2024-03-27T21:42:44.500 INFO:teuthology.orchestra.run.smithi060.stdout: 414745 4 -rw-r--r-- 1 ubuntu ubuntu 847 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay/rbd_replay_debug.hpp 2024-03-27T21:42:44.501 INFO:teuthology.orchestra.run.smithi060.stdout: 414725 12 -rw-r--r-- 1 ubuntu ubuntu 11577 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay/ActionTypes.cc 2024-03-27T21:42:44.501 INFO:teuthology.orchestra.run.smithi060.stdout: 414727 4 -rw-r--r-- 1 ubuntu ubuntu 2248 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay/BoundedBuffer.hpp 2024-03-27T21:42:44.501 INFO:teuthology.orchestra.run.smithi060.stdout: 414736 4 -rw-r--r-- 1 ubuntu ubuntu 3876 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay/Replayer.hpp 2024-03-27T21:42:44.501 INFO:teuthology.orchestra.run.smithi060.stdout: 414728 4 -rw-r--r-- 1 ubuntu ubuntu 1026 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay/BufferReader.cc 2024-03-27T21:42:44.501 INFO:teuthology.orchestra.run.smithi060.stdout: 414738 12 -rw-r--r-- 1 ubuntu ubuntu 8631 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay/actions.hpp 2024-03-27T21:42:44.501 INFO:teuthology.orchestra.run.smithi060.stdout: 414731 4 -rw-r--r-- 1 ubuntu ubuntu 1538 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/rbd_replay/ImageNameMap.cc 2024-03-27T21:42:44.501 INFO:teuthology.orchestra.run.smithi060.stdout: 415336 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/telemetry 2024-03-27T21:42:44.501 INFO:teuthology.orchestra.run.smithi060.stdout: 415337 28 -rw-r--r-- 1 ubuntu ubuntu 27327 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/telemetry/telemetry.patches 2024-03-27T21:42:44.502 INFO:teuthology.orchestra.run.smithi060.stdout: 411415 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/log 2024-03-27T21:42:44.502 INFO:teuthology.orchestra.run.smithi060.stdout: 411417 16 -rw-r--r-- 1 ubuntu ubuntu 14755 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/log/Log.cc 2024-03-27T21:42:44.502 INFO:teuthology.orchestra.run.smithi060.stdout: 411419 8 -rw-r--r-- 1 ubuntu ubuntu 5692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/log/LogClock.h 2024-03-27T21:42:44.502 INFO:teuthology.orchestra.run.smithi060.stdout: 411416 4 -rw-r--r-- 1 ubuntu ubuntu 2598 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/log/Entry.h 2024-03-27T21:42:44.502 INFO:teuthology.orchestra.run.smithi060.stdout: 411421 12 -rw-r--r-- 1 ubuntu ubuntu 10860 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/log/test.cc 2024-03-27T21:42:44.502 INFO:teuthology.orchestra.run.smithi060.stdout: 411418 4 -rw-r--r-- 1 ubuntu ubuntu 3335 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/log/Log.h 2024-03-27T21:42:44.502 INFO:teuthology.orchestra.run.smithi060.stdout: 411420 4 -rw-r--r-- 1 ubuntu ubuntu 3065 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/log/SubsystemMap.h 2024-03-27T21:42:44.502 INFO:teuthology.orchestra.run.smithi060.stdout: 411979 4 -rwxr-xr-x 1 ubuntu ubuntu 1274 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mstop.sh 2024-03-27T21:42:44.502 INFO:teuthology.orchestra.run.smithi060.stdout: 412165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osdc 2024-03-27T21:42:44.503 INFO:teuthology.orchestra.run.smithi060.stdout: 412170 16 -rw-r--r-- 1 ubuntu ubuntu 16039 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osdc/Journaler.h 2024-03-27T21:42:44.503 INFO:teuthology.orchestra.run.smithi060.stdout: 412171 80 -rw-r--r-- 1 ubuntu ubuntu 79721 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osdc/ObjectCacher.cc 2024-03-27T21:42:44.503 INFO:teuthology.orchestra.run.smithi060.stdout: 412178 4 -rw-r--r-- 1 ubuntu ubuntu 2106 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osdc/WritebackHandler.h 2024-03-27T21:42:44.503 INFO:teuthology.orchestra.run.smithi060.stdout: 412167 16 -rw-r--r-- 1 ubuntu ubuntu 13736 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osdc/Filer.cc 2024-03-27T21:42:44.503 INFO:teuthology.orchestra.run.smithi060.stdout: 412177 4 -rw-r--r-- 1 ubuntu ubuntu 1427 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osdc/StriperTypes.h 2024-03-27T21:42:44.503 INFO:teuthology.orchestra.run.smithi060.stdout: 412169 44 -rw-r--r-- 1 ubuntu ubuntu 43617 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osdc/Journaler.cc 2024-03-27T21:42:44.503 INFO:teuthology.orchestra.run.smithi060.stdout: 412166 4 -rw-r--r-- 1 ubuntu ubuntu 234 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osdc/CMakeLists.txt 2024-03-27T21:42:44.503 INFO:teuthology.orchestra.run.smithi060.stdout: 412168 8 -rw-r--r-- 1 ubuntu ubuntu 7905 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osdc/Filer.h 2024-03-27T21:42:44.504 INFO:teuthology.orchestra.run.smithi060.stdout: 412175 20 -rw-r--r-- 1 ubuntu ubuntu 18104 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osdc/Striper.cc 2024-03-27T21:42:44.504 INFO:teuthology.orchestra.run.smithi060.stdout: 412172 24 -rw-r--r-- 1 ubuntu ubuntu 22408 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osdc/ObjectCacher.h 2024-03-27T21:42:44.504 INFO:teuthology.orchestra.run.smithi060.stdout: 412173 152 -rw-r--r-- 1 ubuntu ubuntu 152845 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osdc/Objecter.cc 2024-03-27T21:42:44.504 INFO:teuthology.orchestra.run.smithi060.stdout: 412176 8 -rw-r--r-- 1 ubuntu ubuntu 4429 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osdc/Striper.h 2024-03-27T21:42:44.504 INFO:teuthology.orchestra.run.smithi060.stdout: 412179 8 -rw-r--r-- 1 ubuntu ubuntu 4682 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osdc/error_code.cc 2024-03-27T21:42:44.504 INFO:teuthology.orchestra.run.smithi060.stdout: 412180 4 -rw-r--r-- 1 ubuntu ubuntu 1381 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osdc/error_code.h 2024-03-27T21:42:44.504 INFO:teuthology.orchestra.run.smithi060.stdout: 412174 132 -rw-r--r-- 1 ubuntu ubuntu 135161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/osdc/Objecter.h 2024-03-27T21:42:44.504 INFO:teuthology.orchestra.run.smithi060.stdout: 410852 32 -rw-r--r-- 1 ubuntu ubuntu 32172 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/krbd.cc 2024-03-27T21:42:44.505 INFO:teuthology.orchestra.run.smithi060.stdout: 411984 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/neorados 2024-03-27T21:42:44.505 INFO:teuthology.orchestra.run.smithi060.stdout: 411985 4 -rw-r--r-- 1 ubuntu ubuntu 1482 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/neorados/CMakeLists.txt 2024-03-27T21:42:44.505 INFO:teuthology.orchestra.run.smithi060.stdout: 411987 4 -rw-r--r-- 1 ubuntu ubuntu 3412 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/neorados/RADOSImpl.cc 2024-03-27T21:42:44.505 INFO:teuthology.orchestra.run.smithi060.stdout: 411986 56 -rw-r--r-- 1 ubuntu ubuntu 55927 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/neorados/RADOS.cc 2024-03-27T21:42:44.505 INFO:teuthology.orchestra.run.smithi060.stdout: 411988 4 -rw-r--r-- 1 ubuntu ubuntu 3200 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/neorados/RADOSImpl.h 2024-03-27T21:42:44.505 INFO:teuthology.orchestra.run.smithi060.stdout: 410530 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code 2024-03-27T21:42:44.505 INFO:teuthology.orchestra.run.smithi060.stdout: 410532 12 -rw-r--r-- 1 ubuntu ubuntu 10541 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/ErasureCode.cc 2024-03-27T21:42:44.505 INFO:teuthology.orchestra.run.smithi060.stdout: 410536 4 -rw-r--r-- 1 ubuntu ubuntu 2197 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/ErasureCodePlugin.h 2024-03-27T21:42:44.505 INFO:teuthology.orchestra.run.smithi060.stdout: 410531 4 -rw-r--r-- 1 ubuntu ubuntu 1212 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/CMakeLists.txt 2024-03-27T21:42:44.506 INFO:teuthology.orchestra.run.smithi060.stdout: 410534 20 -rw-r--r-- 1 ubuntu ubuntu 18572 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/ErasureCodeInterface.h 2024-03-27T21:42:44.506 INFO:teuthology.orchestra.run.smithi060.stdout: 410537 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/clay 2024-03-27T21:42:44.506 INFO:teuthology.orchestra.run.smithi060.stdout: 410538 4 -rw-r--r-- 1 ubuntu ubuntu 354 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/clay/CMakeLists.txt 2024-03-27T21:42:44.506 INFO:teuthology.orchestra.run.smithi060.stdout: 410542 4 -rw-r--r-- 1 ubuntu ubuntu 920 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/clay/ErasureCodePluginClay.h 2024-03-27T21:42:44.506 INFO:teuthology.orchestra.run.smithi060.stdout: 410540 8 -rw-r--r-- 1 ubuntu ubuntu 4308 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/clay/ErasureCodeClay.h 2024-03-27T21:42:44.506 INFO:teuthology.orchestra.run.smithi060.stdout: 410541 4 -rw-r--r-- 1 ubuntu ubuntu 1414 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/clay/ErasureCodePluginClay.cc 2024-03-27T21:42:44.506 INFO:teuthology.orchestra.run.smithi060.stdout: 410539 28 -rw-r--r-- 1 ubuntu ubuntu 26786 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/clay/ErasureCodeClay.cc 2024-03-27T21:42:44.506 INFO:teuthology.orchestra.run.smithi060.stdout: 410564 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/lrc 2024-03-27T21:42:44.507 INFO:teuthology.orchestra.run.smithi060.stdout: 410566 28 -rw-r--r-- 1 ubuntu ubuntu 25527 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/lrc/ErasureCodeLrc.cc 2024-03-27T21:42:44.507 INFO:teuthology.orchestra.run.smithi060.stdout: 410565 4 -rw-r--r-- 1 ubuntu ubuntu 388 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/lrc/CMakeLists.txt 2024-03-27T21:42:44.507 INFO:teuthology.orchestra.run.smithi060.stdout: 410569 4 -rw-r--r-- 1 ubuntu ubuntu 956 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/lrc/ErasureCodePluginLrc.h 2024-03-27T21:42:44.507 INFO:teuthology.orchestra.run.smithi060.stdout: 410568 4 -rw-r--r-- 1 ubuntu ubuntu 1487 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/lrc/ErasureCodePluginLrc.cc 2024-03-27T21:42:44.507 INFO:teuthology.orchestra.run.smithi060.stdout: 410567 8 -rw-r--r-- 1 ubuntu ubuntu 4503 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/lrc/ErasureCodeLrc.h 2024-03-27T21:42:44.507 INFO:teuthology.orchestra.run.smithi060.stdout: 410535 8 -rw-r--r-- 1 ubuntu ubuntu 5362 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/ErasureCodePlugin.cc 2024-03-27T21:42:44.507 INFO:teuthology.orchestra.run.smithi060.stdout: 410533 4 -rw-r--r-- 1 ubuntu ubuntu 3761 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/ErasureCode.h 2024-03-27T21:42:44.508 INFO:teuthology.orchestra.run.smithi060.stdout: 410554 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/jerasure 2024-03-27T21:42:44.508 INFO:teuthology.orchestra.run.smithi060.stdout: 410555 4 -rw-r--r-- 1 ubuntu ubuntu 2948 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/jerasure/CMakeLists.txt 2024-03-27T21:42:44.508 INFO:teuthology.orchestra.run.smithi060.stdout: 410559 4 -rw-r--r-- 1 ubuntu ubuntu 971 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/jerasure/ErasureCodePluginJerasure.h 2024-03-27T21:42:44.508 INFO:teuthology.orchestra.run.smithi060.stdout: 410560 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/jerasure/gf-complete 2024-03-27T21:42:44.508 INFO:teuthology.orchestra.run.smithi060.stdout: 410561 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/jerasure/jerasure 2024-03-27T21:42:44.508 INFO:teuthology.orchestra.run.smithi060.stdout: 410563 4 -rw-r--r-- 1 ubuntu ubuntu 705 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/jerasure/jerasure_init.h 2024-03-27T21:42:44.508 INFO:teuthology.orchestra.run.smithi060.stdout: 410557 8 -rw-r--r-- 1 ubuntu ubuntu 7165 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/jerasure/ErasureCodeJerasure.h 2024-03-27T21:42:44.508 INFO:teuthology.orchestra.run.smithi060.stdout: 410556 16 -rw-r--r-- 1 ubuntu ubuntu 14436 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/jerasure/ErasureCodeJerasure.cc 2024-03-27T21:42:44.509 INFO:teuthology.orchestra.run.smithi060.stdout: 410558 4 -rw-r--r-- 1 ubuntu ubuntu 2825 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/jerasure/ErasureCodePluginJerasure.cc 2024-03-27T21:42:44.509 INFO:teuthology.orchestra.run.smithi060.stdout: 410562 4 -rw-r--r-- 1 ubuntu ubuntu 984 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/jerasure/jerasure_init.cc 2024-03-27T21:42:44.509 INFO:teuthology.orchestra.run.smithi060.stdout: 410543 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/isa 2024-03-27T21:42:44.509 INFO:teuthology.orchestra.run.smithi060.stdout: 410545 16 -rw-r--r-- 1 ubuntu ubuntu 12762 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/isa/ErasureCodeIsa.cc 2024-03-27T21:42:44.509 INFO:teuthology.orchestra.run.smithi060.stdout: 410552 8 -rw-r--r-- 1 ubuntu ubuntu 8089 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/isa/xor_op.cc 2024-03-27T21:42:44.509 INFO:teuthology.orchestra.run.smithi060.stdout: 410544 8 -rw-r--r-- 1 ubuntu ubuntu 4602 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/isa/CMakeLists.txt 2024-03-27T21:42:44.509 INFO:teuthology.orchestra.run.smithi060.stdout: 410551 4 -rw-r--r-- 1 ubuntu ubuntu 2595 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/isa/README 2024-03-27T21:42:44.509 INFO:teuthology.orchestra.run.smithi060.stdout: 410547 12 -rw-r--r-- 1 ubuntu ubuntu 11175 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/isa/ErasureCodeIsaTableCache.cc 2024-03-27T21:42:44.510 INFO:teuthology.orchestra.run.smithi060.stdout: 410549 4 -rw-r--r-- 1 ubuntu ubuntu 2657 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/isa/ErasureCodePluginIsa.cc 2024-03-27T21:42:44.510 INFO:teuthology.orchestra.run.smithi060.stdout: 410548 4 -rw-r--r-- 1 ubuntu ubuntu 4080 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/isa/ErasureCodeIsaTableCache.h 2024-03-27T21:42:44.510 INFO:teuthology.orchestra.run.smithi060.stdout: 410546 4 -rw-r--r-- 1 ubuntu ubuntu 3894 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/isa/ErasureCodeIsa.h 2024-03-27T21:42:44.510 INFO:teuthology.orchestra.run.smithi060.stdout: 410553 8 -rw-r--r-- 1 ubuntu ubuntu 4674 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/isa/xor_op.h 2024-03-27T21:42:44.510 INFO:teuthology.orchestra.run.smithi060.stdout: 410550 4 -rw-r--r-- 1 ubuntu ubuntu 1030 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/isa/ErasureCodePluginIsa.h 2024-03-27T21:42:44.510 INFO:teuthology.orchestra.run.smithi060.stdout: 410570 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/shec 2024-03-27T21:42:44.510 INFO:teuthology.orchestra.run.smithi060.stdout: 410577 8 -rw-r--r-- 1 ubuntu ubuntu 4490 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/shec/ErasureCodeShecTableCache.h 2024-03-27T21:42:44.511 INFO:teuthology.orchestra.run.smithi060.stdout: 410576 12 -rw-r--r-- 1 ubuntu ubuntu 10928 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/shec/ErasureCodeShecTableCache.cc 2024-03-27T21:42:44.511 INFO:teuthology.orchestra.run.smithi060.stdout: 410571 4 -rw-r--r-- 1 ubuntu ubuntu 944 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/shec/CMakeLists.txt 2024-03-27T21:42:44.511 INFO:teuthology.orchestra.run.smithi060.stdout: 410575 8 -rw-r--r-- 1 ubuntu ubuntu 4272 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/shec/ErasureCodeShec.h 2024-03-27T21:42:44.511 INFO:teuthology.orchestra.run.smithi060.stdout: 410578 4 -rw-r--r-- 1 ubuntu ubuntu 2145 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/shec/determinant.c 2024-03-27T21:42:44.511 INFO:teuthology.orchestra.run.smithi060.stdout: 410574 24 -rw-r--r-- 1 ubuntu ubuntu 20700 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/shec/ErasureCodeShec.cc 2024-03-27T21:42:44.511 INFO:teuthology.orchestra.run.smithi060.stdout: 410572 4 -rw-r--r-- 1 ubuntu ubuntu 2560 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/shec/ErasureCodePluginShec.cc 2024-03-27T21:42:44.511 INFO:teuthology.orchestra.run.smithi060.stdout: 410573 4 -rw-r--r-- 1 ubuntu ubuntu 1035 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/erasure-code/shec/ErasureCodePluginShec.h 2024-03-27T21:42:44.511 INFO:teuthology.orchestra.run.smithi060.stdout: 409091 4 -rwxr-xr-x 1 ubuntu ubuntu 452 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-rbdnamer 2024-03-27T21:42:44.512 INFO:teuthology.orchestra.run.smithi060.stdout: 417095 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing 2024-03-27T21:42:44.512 INFO:teuthology.orchestra.run.smithi060.stdout: 417099 4 -rw-r--r-- 1 ubuntu ubuntu 119 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/bluestore.c 2024-03-27T21:42:44.512 INFO:teuthology.orchestra.run.smithi060.stdout: 417117 4 -rw-r--r-- 1 ubuntu ubuntu 451 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/pg.tp 2024-03-27T21:42:44.512 INFO:teuthology.orchestra.run.smithi060.stdout: 417098 4 -rw-r--r-- 1 ubuntu ubuntu 2257 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/README.md 2024-03-27T21:42:44.512 INFO:teuthology.orchestra.run.smithi060.stdout: 417102 4 -rw-r--r-- 1 ubuntu ubuntu 574 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/cyg_profile.tp 2024-03-27T21:42:44.512 INFO:teuthology.orchestra.run.smithi060.stdout: 417107 100 -rw-r--r-- 1 ubuntu ubuntu 100849 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/librados.tp 2024-03-27T21:42:44.512 INFO:teuthology.orchestra.run.smithi060.stdout: 417113 4 -rw-r--r-- 1 ubuntu ubuntu 1429 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/oprequest.tp 2024-03-27T21:42:44.512 INFO:teuthology.orchestra.run.smithi060.stdout: 417108 4 -rw-r--r-- 1 ubuntu ubuntu 117 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/librbd.c 2024-03-27T21:42:44.512 INFO:teuthology.orchestra.run.smithi060.stdout: 417097 4 -rw-r--r-- 1 ubuntu ubuntu 2297 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/CMakeLists.txt 2024-03-27T21:42:44.513 INFO:teuthology.orchestra.run.smithi060.stdout: 417105 4 -rw-r--r-- 1 ubuntu ubuntu 1443 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/eventtrace.tp 2024-03-27T21:42:44.513 INFO:teuthology.orchestra.run.smithi060.stdout: 417103 4 -rw-r--r-- 1 ubuntu ubuntu 754 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/cyg_profile_functions.c 2024-03-27T21:42:44.513 INFO:teuthology.orchestra.run.smithi060.stdout: 417106 4 -rw-r--r-- 1 ubuntu ubuntu 119 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/librados.c 2024-03-27T21:42:44.513 INFO:teuthology.orchestra.run.smithi060.stdout: 417119 4 -rw-r--r-- 1 ubuntu ubuntu 724 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/rgw_op.tp 2024-03-27T21:42:44.513 INFO:teuthology.orchestra.run.smithi060.stdout: 417115 24 -rw-r--r-- 1 ubuntu ubuntu 20627 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/osd.tp 2024-03-27T21:42:44.513 INFO:teuthology.orchestra.run.smithi060.stdout: 417111 16 -rw-r--r-- 1 ubuntu ubuntu 15616 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/objectstore.tp 2024-03-27T21:42:44.513 INFO:teuthology.orchestra.run.smithi060.stdout: 417122 8 -rw-r--r-- 1 ubuntu ubuntu 4280 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/tracing-common.h 2024-03-27T21:42:44.513 INFO:teuthology.orchestra.run.smithi060.stdout: 417120 4 -rw-r--r-- 1 ubuntu ubuntu 120 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/rgw_rados.c 2024-03-27T21:42:44.514 INFO:teuthology.orchestra.run.smithi060.stdout: 417112 4 -rw-r--r-- 1 ubuntu ubuntu 120 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/oprequest.c 2024-03-27T21:42:44.514 INFO:teuthology.orchestra.run.smithi060.stdout: 417121 4 -rw-r--r-- 1 ubuntu ubuntu 936 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/rgw_rados.tp 2024-03-27T21:42:44.514 INFO:teuthology.orchestra.run.smithi060.stdout: 417100 8 -rw-r--r-- 1 ubuntu ubuntu 5804 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/bluestore.tp 2024-03-27T21:42:44.514 INFO:teuthology.orchestra.run.smithi060.stdout: 417104 4 -rw-r--r-- 1 ubuntu ubuntu 120 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/eventtrace.c 2024-03-27T21:42:44.514 INFO:teuthology.orchestra.run.smithi060.stdout: 417109 72 -rw-r--r-- 1 ubuntu ubuntu 71769 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/librbd.tp 2024-03-27T21:42:44.514 INFO:teuthology.orchestra.run.smithi060.stdout: 417110 4 -rw-r--r-- 1 ubuntu ubuntu 122 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/objectstore.c 2024-03-27T21:42:44.514 INFO:teuthology.orchestra.run.smithi060.stdout: 417096 4 -rw-r--r-- 1 ubuntu ubuntu 57 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/.gitignore 2024-03-27T21:42:44.514 INFO:teuthology.orchestra.run.smithi060.stdout: 417116 4 -rw-r--r-- 1 ubuntu ubuntu 113 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/pg.c 2024-03-27T21:42:44.515 INFO:teuthology.orchestra.run.smithi060.stdout: 417114 4 -rw-r--r-- 1 ubuntu ubuntu 114 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/osd.c 2024-03-27T21:42:44.515 INFO:teuthology.orchestra.run.smithi060.stdout: 417118 4 -rw-r--r-- 1 ubuntu ubuntu 117 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/rgw_op.c 2024-03-27T21:42:44.515 INFO:teuthology.orchestra.run.smithi060.stdout: 417101 4 -rw-r--r-- 1 ubuntu ubuntu 122 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/tracing/cyg_profile.c 2024-03-27T21:42:44.515 INFO:teuthology.orchestra.run.smithi060.stdout: 409048 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/blk 2024-03-27T21:42:44.515 INFO:teuthology.orchestra.run.smithi060.stdout: 409055 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/blk/kernel 2024-03-27T21:42:44.515 INFO:teuthology.orchestra.run.smithi060.stdout: 409057 8 -rw-r--r-- 1 ubuntu ubuntu 5177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/blk/kernel/KernelDevice.h 2024-03-27T21:42:44.515 INFO:teuthology.orchestra.run.smithi060.stdout: 409058 8 -rw-r--r-- 1 ubuntu ubuntu 5354 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/blk/kernel/io_uring.cc 2024-03-27T21:42:44.515 INFO:teuthology.orchestra.run.smithi060.stdout: 409059 4 -rw-r--r-- 1 ubuntu ubuntu 861 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/blk/kernel/io_uring.h 2024-03-27T21:42:44.515 INFO:teuthology.orchestra.run.smithi060.stdout: 409056 48 -rw-r--r-- 1 ubuntu ubuntu 46516 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/blk/kernel/KernelDevice.cc 2024-03-27T21:42:44.516 INFO:teuthology.orchestra.run.smithi060.stdout: 409060 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/blk/pmem 2024-03-27T21:42:44.516 INFO:teuthology.orchestra.run.smithi060.stdout: 409061 12 -rw-r--r-- 1 ubuntu ubuntu 9091 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/blk/pmem/PMEMDevice.cc 2024-03-27T21:42:44.516 INFO:teuthology.orchestra.run.smithi060.stdout: 409062 4 -rw-r--r-- 1 ubuntu ubuntu 2206 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/blk/pmem/PMEMDevice.h 2024-03-27T21:42:44.516 INFO:teuthology.orchestra.run.smithi060.stdout: 409063 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/blk/spdk 2024-03-27T21:42:44.516 INFO:teuthology.orchestra.run.smithi060.stdout: 409065 4 -rw-r--r-- 1 ubuntu ubuntu 2323 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/blk/spdk/NVMEDevice.h 2024-03-27T21:42:44.516 INFO:teuthology.orchestra.run.smithi060.stdout: 409064 28 -rw-r--r-- 1 ubuntu ubuntu 28078 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/blk/spdk/NVMEDevice.cc 2024-03-27T21:42:44.516 INFO:teuthology.orchestra.run.smithi060.stdout: 409051 4 -rw-r--r-- 1 ubuntu ubuntu 986 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/blk/CMakeLists.txt 2024-03-27T21:42:44.516 INFO:teuthology.orchestra.run.smithi060.stdout: 409049 8 -rw-r--r-- 1 ubuntu ubuntu 4837 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/blk/BlockDevice.cc 2024-03-27T21:42:44.517 INFO:teuthology.orchestra.run.smithi060.stdout: 409050 12 -rw-r--r-- 1 ubuntu ubuntu 8809 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/blk/BlockDevice.h 2024-03-27T21:42:44.517 INFO:teuthology.orchestra.run.smithi060.stdout: 409052 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/blk/aio 2024-03-27T21:42:44.517 INFO:teuthology.orchestra.run.smithi060.stdout: 409054 4 -rw-r--r-- 1 ubuntu ubuntu 3965 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/blk/aio/aio.h 2024-03-27T21:42:44.517 INFO:teuthology.orchestra.run.smithi060.stdout: 409053 4 -rw-r--r-- 1 ubuntu ubuntu 3200 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/blk/aio/aio.cc 2024-03-27T21:42:44.517 INFO:teuthology.orchestra.run.smithi060.stdout: 415335 8 -rwxr-xr-x 1 ubuntu ubuntu 6991 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/stop.sh 2024-03-27T21:42:44.517 INFO:teuthology.orchestra.run.smithi060.stdout: 410589 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/extblkdev 2024-03-27T21:42:44.517 INFO:teuthology.orchestra.run.smithi060.stdout: 410590 4 -rw-r--r-- 1 ubuntu ubuntu 310 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/extblkdev/CMakeLists.txt 2024-03-27T21:42:44.517 INFO:teuthology.orchestra.run.smithi060.stdout: 410592 8 -rw-r--r-- 1 ubuntu ubuntu 7639 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/extblkdev/ExtBlkDevPlugin.cc 2024-03-27T21:42:44.517 INFO:teuthology.orchestra.run.smithi060.stdout: 410591 8 -rw-r--r-- 1 ubuntu ubuntu 4914 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/extblkdev/ExtBlkDevInterface.h 2024-03-27T21:42:44.518 INFO:teuthology.orchestra.run.smithi060.stdout: 410593 4 -rw-r--r-- 1 ubuntu ubuntu 1100 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/extblkdev/ExtBlkDevPlugin.h 2024-03-27T21:42:44.518 INFO:teuthology.orchestra.run.smithi060.stdout: 410594 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/extblkdev/vdo 2024-03-27T21:42:44.518 INFO:teuthology.orchestra.run.smithi060.stdout: 410595 4 -rw-r--r-- 1 ubuntu ubuntu 183 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/extblkdev/vdo/CMakeLists.txt 2024-03-27T21:42:44.518 INFO:teuthology.orchestra.run.smithi060.stdout: 410596 4 -rw-r--r-- 1 ubuntu ubuntu 1559 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/extblkdev/vdo/ExtBlkDevPluginVdo.cc 2024-03-27T21:42:44.518 INFO:teuthology.orchestra.run.smithi060.stdout: 410599 4 -rw-r--r-- 1 ubuntu ubuntu 1741 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/extblkdev/vdo/ExtBlkDevVdo.h 2024-03-27T21:42:44.518 INFO:teuthology.orchestra.run.smithi060.stdout: 410597 4 -rw-r--r-- 1 ubuntu ubuntu 1103 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/extblkdev/vdo/ExtBlkDevPluginVdo.h 2024-03-27T21:42:44.518 INFO:teuthology.orchestra.run.smithi060.stdout: 410598 8 -rw-r--r-- 1 ubuntu ubuntu 4693 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/extblkdev/vdo/ExtBlkDevVdo.cc 2024-03-27T21:42:44.519 INFO:teuthology.orchestra.run.smithi060.stdout: 411978 4 -rwxr-xr-x 1 ubuntu ubuntu 1208 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/mstart.sh 2024-03-27T21:42:44.519 INFO:teuthology.orchestra.run.smithi060.stdout: 411989 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/objclass 2024-03-27T21:42:44.519 INFO:teuthology.orchestra.run.smithi060.stdout: 411991 8 -rw-r--r-- 1 ubuntu ubuntu 7686 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/objclass/objclass.h 2024-03-27T21:42:44.519 INFO:teuthology.orchestra.run.smithi060.stdout: 411990 4 -rw-r--r-- 1 ubuntu ubuntu 4016 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/objclass/class_api.cc 2024-03-27T21:42:44.519 INFO:teuthology.orchestra.run.smithi060.stdout: 409683 12 drwxr-xr-x 6 ubuntu ubuntu 12288 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common 2024-03-27T21:42:44.519 INFO:teuthology.orchestra.run.smithi060.stdout: 409914 4 -rw-r--r-- 1 ubuntu ubuntu 2249 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/mime.c 2024-03-27T21:42:44.519 INFO:teuthology.orchestra.run.smithi060.stdout: 409990 4 -rw-r--r-- 1 ubuntu ubuntu 3477 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/simple_cache.hpp 2024-03-27T21:42:44.519 INFO:teuthology.orchestra.run.smithi060.stdout: 409754 8 -rw-r--r-- 1 ubuntu ubuntu 4269 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/TextTable.h 2024-03-27T21:42:44.519 INFO:teuthology.orchestra.run.smithi060.stdout: 409810 8 -rw-r--r-- 1 ubuntu ubuntu 7514 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ceph_crypto.cc 2024-03-27T21:42:44.520 INFO:teuthology.orchestra.run.smithi060.stdout: 409869 8 -rw-r--r-- 1 ubuntu ubuntu 7649 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/dout.h 2024-03-27T21:42:44.520 INFO:teuthology.orchestra.run.smithi060.stdout: 409874 4 -rw-r--r-- 1 ubuntu ubuntu 570 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/environment.h 2024-03-27T21:42:44.520 INFO:teuthology.orchestra.run.smithi060.stdout: 409834 4 -rw-r--r-- 1 ubuntu ubuntu 2202 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/condition_variable_debug.cc 2024-03-27T21:42:44.520 INFO:teuthology.orchestra.run.smithi060.stdout: 410011 8 -rw-r--r-- 1 ubuntu ubuntu 6560 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/utf8.c 2024-03-27T21:42:44.520 INFO:teuthology.orchestra.run.smithi060.stdout: 409871 4 -rw-r--r-- 1 ubuntu ubuntu 3464 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/entity_name.cc 2024-03-27T21:42:44.520 INFO:teuthology.orchestra.run.smithi060.stdout: 409702 4 -rw-r--r-- 1 ubuntu ubuntu 3877 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Cycles.h 2024-03-27T21:42:44.520 INFO:teuthology.orchestra.run.smithi060.stdout: 409950 4 -rw-r--r-- 1 ubuntu ubuntu 602 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/page.cc 2024-03-27T21:42:44.520 INFO:teuthology.orchestra.run.smithi060.stdout: 409912 8 -rw-r--r-- 1 ubuntu ubuntu 5167 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/map_cacher.hpp 2024-03-27T21:42:44.521 INFO:teuthology.orchestra.run.smithi060.stdout: 409709 4 -rw-r--r-- 1 ubuntu ubuntu 2910 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Finisher.cc 2024-03-27T21:42:44.521 INFO:teuthology.orchestra.run.smithi060.stdout: 409696 12 -rw-r--r-- 1 ubuntu ubuntu 10243 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ConfUtils.cc 2024-03-27T21:42:44.521 INFO:teuthology.orchestra.run.smithi060.stdout: 409895 16 -rw-r--r-- 1 ubuntu ubuntu 14363 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/hobject.cc 2024-03-27T21:42:44.521 INFO:teuthology.orchestra.run.smithi060.stdout: 409799 4 -rw-r--r-- 1 ubuntu ubuntu 2628 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/blkdev.h 2024-03-27T21:42:44.521 INFO:teuthology.orchestra.run.smithi060.stdout: 409854 4 -rw-r--r-- 1 ubuntu ubuntu 495 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/crc32c_intel_fast.h 2024-03-27T21:42:44.521 INFO:teuthology.orchestra.run.smithi060.stdout: 409892 4 -rw-r--r-- 1 ubuntu ubuntu 630 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/hex.h 2024-03-27T21:42:44.521 INFO:teuthology.orchestra.run.smithi060.stdout: 409730 4 -rw-r--r-- 1 ubuntu ubuntu 2380 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/MemoryModel.cc 2024-03-27T21:42:44.521 INFO:teuthology.orchestra.run.smithi060.stdout: 409991 4 -rw-r--r-- 1 ubuntu ubuntu 3757 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/snap_types.cc 2024-03-27T21:42:44.521 INFO:teuthology.orchestra.run.smithi060.stdout: 409821 4 -rw-r--r-- 1 ubuntu ubuntu 895 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ceph_strings.h 2024-03-27T21:42:44.522 INFO:teuthology.orchestra.run.smithi060.stdout: 410008 4 -rw-r--r-- 1 ubuntu ubuntu 825 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/types.cc 2024-03-27T21:42:44.522 INFO:teuthology.orchestra.run.smithi060.stdout: 409923 8 -rw-r--r-- 1 ubuntu ubuntu 4670 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/obj_bencher.h 2024-03-27T21:42:44.522 INFO:teuthology.orchestra.run.smithi060.stdout: 409713 24 -rw-r--r-- 1 ubuntu ubuntu 22098 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Formatter.cc 2024-03-27T21:42:44.522 INFO:teuthology.orchestra.run.smithi060.stdout: 409987 12 -rw-r--r-- 1 ubuntu ubuntu 10608 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/shunique_lock.h 2024-03-27T21:42:44.522 INFO:teuthology.orchestra.run.smithi060.stdout: 409714 16 -rw-r--r-- 1 ubuntu ubuntu 14439 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Formatter.h 2024-03-27T21:42:44.522 INFO:teuthology.orchestra.run.smithi060.stdout: 409919 8 -rw-r--r-- 1 ubuntu ubuntu 4859 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/mutex_debug.h 2024-03-27T21:42:44.522 INFO:teuthology.orchestra.run.smithi060.stdout: 409801 20 -rw-r--r-- 1 ubuntu ubuntu 17565 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/bloom_filter.hpp 2024-03-27T21:42:44.522 INFO:teuthology.orchestra.run.smithi060.stdout: 410036 4 -rw-r--r-- 1 ubuntu ubuntu 2365 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/zipkin_trace.h 2024-03-27T21:42:44.523 INFO:teuthology.orchestra.run.smithi060.stdout: 409716 4 -rw-r--r-- 1 ubuntu ubuntu 2052 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Graylog.h 2024-03-27T21:42:44.523 INFO:teuthology.orchestra.run.smithi060.stdout: 409864 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/detail 2024-03-27T21:42:44.523 INFO:teuthology.orchestra.run.smithi060.stdout: 409865 4 -rw-r--r-- 1 ubuntu ubuntu 741 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/detail/construct_suspended.h 2024-03-27T21:42:44.523 INFO:teuthology.orchestra.run.smithi060.stdout: 409887 8 -rw-r--r-- 1 ubuntu ubuntu 4799 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/fork_function.h 2024-03-27T21:42:44.523 INFO:teuthology.orchestra.run.smithi060.stdout: 409879 8 -rw-r--r-- 1 ubuntu ubuntu 6180 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/escape.cc 2024-03-27T21:42:44.523 INFO:teuthology.orchestra.run.smithi060.stdout: 409758 16 -rw-r--r-- 1 ubuntu ubuntu 12846 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Throttle.h 2024-03-27T21:42:44.523 INFO:teuthology.orchestra.run.smithi060.stdout: 409907 4 -rw-r--r-- 1 ubuntu ubuntu 418 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/linux_version.c 2024-03-27T21:42:44.523 INFO:teuthology.orchestra.run.smithi060.stdout: 409846 8 -rw-r--r-- 1 ubuntu ubuntu 6550 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/containers.h 2024-03-27T21:42:44.524 INFO:teuthology.orchestra.run.smithi060.stdout: 409690 4 -rw-r--r-- 1 ubuntu ubuntu 776 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/CDC.h 2024-03-27T21:42:44.524 INFO:teuthology.orchestra.run.smithi060.stdout: 409835 4 -rw-r--r-- 1 ubuntu ubuntu 2464 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/condition_variable_debug.h 2024-03-27T21:42:44.524 INFO:teuthology.orchestra.run.smithi060.stdout: 410003 4 -rw-r--r-- 1 ubuntu ubuntu 2094 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/subsys_types.h 2024-03-27T21:42:44.524 INFO:teuthology.orchestra.run.smithi060.stdout: 409698 4 -rw-r--r-- 1 ubuntu ubuntu 1009 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ContextCompletion.cc 2024-03-27T21:42:44.524 INFO:teuthology.orchestra.run.smithi060.stdout: 409697 4 -rw-r--r-- 1 ubuntu ubuntu 2911 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ConfUtils.h 2024-03-27T21:42:44.524 INFO:teuthology.orchestra.run.smithi060.stdout: 409872 4 -rw-r--r-- 1 ubuntu ubuntu 2624 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/entity_name.h 2024-03-27T21:42:44.524 INFO:teuthology.orchestra.run.smithi060.stdout: 409851 8 -rw-r--r-- 1 ubuntu ubuntu 5556 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/crc32c_intel_baseline.c 2024-03-27T21:42:44.524 INFO:teuthology.orchestra.run.smithi060.stdout: 409971 4 -rw-r--r-- 1 ubuntu ubuntu 1311 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/reverse.c 2024-03-27T21:42:44.525 INFO:teuthology.orchestra.run.smithi060.stdout: 409691 8 -rw-r--r-- 1 ubuntu ubuntu 5771 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/CMakeLists.txt 2024-03-27T21:42:44.525 INFO:teuthology.orchestra.run.smithi060.stdout: 409886 4 -rw-r--r-- 1 ubuntu ubuntu 2772 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/fmt_common.h 2024-03-27T21:42:44.525 INFO:teuthology.orchestra.run.smithi060.stdout: 409804 4 -rw-r--r-- 1 ubuntu ubuntu 985 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/buffer_instrumentation.h 2024-03-27T21:42:44.525 INFO:teuthology.orchestra.run.smithi060.stdout: 409830 4 -rw-r--r-- 1 ubuntu ubuntu 3850 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/common_init.cc 2024-03-27T21:42:44.525 INFO:teuthology.orchestra.run.smithi060.stdout: 409975 8 -rw-r--r-- 1 ubuntu ubuntu 5972 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/safe_io.c 2024-03-27T21:42:44.525 INFO:teuthology.orchestra.run.smithi060.stdout: 409771 8 -rw-r--r-- 1 ubuntu ubuntu 7240 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/admin_socket.h 2024-03-27T21:42:44.525 INFO:teuthology.orchestra.run.smithi060.stdout: 409813 4 -rw-r--r-- 1 ubuntu ubuntu 2099 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ceph_fs.cc 2024-03-27T21:42:44.525 INFO:teuthology.orchestra.run.smithi060.stdout: 409833 4 -rw-r--r-- 1 ubuntu ubuntu 796 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/compiler_extensions.h 2024-03-27T21:42:44.526 INFO:teuthology.orchestra.run.smithi060.stdout: 409961 20 -rw-r--r-- 1 ubuntu ubuntu 17586 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/pick_address.cc 2024-03-27T21:42:44.526 INFO:teuthology.orchestra.run.smithi060.stdout: 409751 12 -rw-r--r-- 1 ubuntu ubuntu 9643 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/SubProcess.cc 2024-03-27T21:42:44.526 INFO:teuthology.orchestra.run.smithi060.stdout: 409826 4 -rw-r--r-- 1 ubuntu ubuntu 3517 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/cmdparse.h 2024-03-27T21:42:44.526 INFO:teuthology.orchestra.run.smithi060.stdout: 409686 12 -rw-r--r-- 1 ubuntu ubuntu 10916 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/AsyncReserver.h 2024-03-27T21:42:44.526 INFO:teuthology.orchestra.run.smithi060.stdout: 409740 8 -rw-r--r-- 1 ubuntu ubuntu 4692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/PriorityCache.h 2024-03-27T21:42:44.526 INFO:teuthology.orchestra.run.smithi060.stdout: 409876 4 -rw-r--r-- 1 ubuntu ubuntu 363 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/errno.h 2024-03-27T21:42:44.526 INFO:teuthology.orchestra.run.smithi060.stdout: 409978 12 -rw-r--r-- 1 ubuntu ubuntu 11317 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/scrub_types.h 2024-03-27T21:42:44.526 INFO:teuthology.orchestra.run.smithi060.stdout: 410014 4 -rw-r--r-- 1 ubuntu ubuntu 544 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/valgrind.h 2024-03-27T21:42:44.527 INFO:teuthology.orchestra.run.smithi060.stdout: 409841 8 -rw-r--r-- 1 ubuntu ubuntu 4266 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/config_obs_mgr.h 2024-03-27T21:42:44.527 INFO:teuthology.orchestra.run.smithi060.stdout: 410012 4 -rw-r--r-- 1 ubuntu ubuntu 1754 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/utf8.h 2024-03-27T21:42:44.527 INFO:teuthology.orchestra.run.smithi060.stdout: 410007 4 -rw-r--r-- 1 ubuntu ubuntu 1591 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/tracked_int_ptr.hpp 2024-03-27T21:42:44.527 INFO:teuthology.orchestra.run.smithi060.stdout: 409796 4 -rw-r--r-- 1 ubuntu ubuntu 913 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/bit_str.h 2024-03-27T21:42:44.527 INFO:teuthology.orchestra.run.smithi060.stdout: 409779 8 -rw-r--r-- 1 ubuntu ubuntu 7321 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/assert.cc 2024-03-27T21:42:44.527 INFO:teuthology.orchestra.run.smithi060.stdout: 409712 4 -rw-r--r-- 1 ubuntu ubuntu 519 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/FixedCDC.h 2024-03-27T21:42:44.527 INFO:teuthology.orchestra.run.smithi060.stdout: 409969 4 -rw-r--r-- 1 ubuntu ubuntu 1473 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/random_string.h 2024-03-27T21:42:44.527 INFO:teuthology.orchestra.run.smithi060.stdout: 410015 4 -rw-r--r-- 1 ubuntu ubuntu 1323 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/version.cc 2024-03-27T21:42:44.527 INFO:teuthology.orchestra.run.smithi060.stdout: 409875 4 -rw-r--r-- 1 ubuntu ubuntu 359 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/errno.cc 2024-03-27T21:42:44.528 INFO:teuthology.orchestra.run.smithi060.stdout: 409859 32 -rw-r--r-- 1 ubuntu ubuntu 31832 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/crc32c_ppc_constants.h 2024-03-27T21:42:44.528 INFO:teuthology.orchestra.run.smithi060.stdout: 409705 4 -rw-r--r-- 1 ubuntu ubuntu 3808 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/EventTrace.cc 2024-03-27T21:42:44.528 INFO:teuthology.orchestra.run.smithi060.stdout: 409737 4 -rw-r--r-- 1 ubuntu ubuntu 3578 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Preforker.h 2024-03-27T21:42:44.528 INFO:teuthology.orchestra.run.smithi060.stdout: 409693 4 -rw-r--r-- 1 ubuntu ubuntu 761 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Clock.h 2024-03-27T21:42:44.528 INFO:teuthology.orchestra.run.smithi060.stdout: 409766 12 -rw-r--r-- 1 ubuntu ubuntu 10119 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/WeightedPriorityQueue.h 2024-03-27T21:42:44.528 INFO:teuthology.orchestra.run.smithi060.stdout: 409963 8 -rw-r--r-- 1 ubuntu ubuntu 7342 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ppc-asm.h 2024-03-27T21:42:44.528 INFO:teuthology.orchestra.run.smithi060.stdout: 409927 12 -rw-r--r-- 1 ubuntu ubuntu 11502 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options.h 2024-03-27T21:42:44.528 INFO:teuthology.orchestra.run.smithi060.stdout: 409881 4 -rw-r--r-- 1 ubuntu ubuntu 1702 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/event_socket.h 2024-03-27T21:42:44.529 INFO:teuthology.orchestra.run.smithi060.stdout: 409770 32 -rw-r--r-- 1 ubuntu ubuntu 30914 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/admin_socket.cc 2024-03-27T21:42:44.529 INFO:teuthology.orchestra.run.smithi060.stdout: 409915 4 -rw-r--r-- 1 ubuntu ubuntu 1732 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/mime.h 2024-03-27T21:42:44.529 INFO:teuthology.orchestra.run.smithi060.stdout: 409901 8 -rw-r--r-- 1 ubuntu ubuntu 8002 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/intrusive_lru.h 2024-03-27T21:42:44.529 INFO:teuthology.orchestra.run.smithi060.stdout: 409986 8 -rw-r--r-- 1 ubuntu ubuntu 4351 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/sharedptr_registry.hpp 2024-03-27T21:42:44.529 INFO:teuthology.orchestra.run.smithi060.stdout: 409773 4 -rw-r--r-- 1 ubuntu ubuntu 908 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/admin_socket_client.h 2024-03-27T21:42:44.529 INFO:teuthology.orchestra.run.smithi060.stdout: 409717 4 -rw-r--r-- 1 ubuntu ubuntu 246 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/HBHandle.h 2024-03-27T21:42:44.529 INFO:teuthology.orchestra.run.smithi060.stdout: 409906 4 -rw-r--r-- 1 ubuntu ubuntu 714 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/likely.h 2024-03-27T21:42:44.529 INFO:teuthology.orchestra.run.smithi060.stdout: 409746 8 -rw-r--r-- 1 ubuntu ubuntu 5014 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/RefCountedObj.h 2024-03-27T21:42:44.530 INFO:teuthology.orchestra.run.smithi060.stdout: 409966 4 -rw-r--r-- 1 ubuntu ubuntu 1561 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/pretty_binary.h 2024-03-27T21:42:44.530 INFO:teuthology.orchestra.run.smithi060.stdout: 409979 40 -rw-r--r-- 1 ubuntu ubuntu 40788 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/sctp_crc32.c 2024-03-27T21:42:44.530 INFO:teuthology.orchestra.run.smithi060.stdout: 410018 4 -rw-r--r-- 1 ubuntu ubuntu 771 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/weighted_shuffle.h 2024-03-27T21:42:44.530 INFO:teuthology.orchestra.run.smithi060.stdout: 409748 8 -rw-r--r-- 1 ubuntu ubuntu 4663 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/SloppyCRCMap.cc 2024-03-27T21:42:44.530 INFO:teuthology.orchestra.run.smithi060.stdout: 409829 12 -rw-r--r-- 1 ubuntu ubuntu 12220 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/cohort_lru.h 2024-03-27T21:42:44.530 INFO:teuthology.orchestra.run.smithi060.stdout: 409805 16 -rw-r--r-- 1 ubuntu ubuntu 14874 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ceph_argparse.cc 2024-03-27T21:42:44.530 INFO:teuthology.orchestra.run.smithi060.stdout: 409734 4 -rw-r--r-- 1 ubuntu ubuntu 1586 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/OutputDataSocket.h 2024-03-27T21:42:44.530 INFO:teuthology.orchestra.run.smithi060.stdout: 409729 8 -rw-r--r-- 1 ubuntu ubuntu 6114 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/LogEntry.h 2024-03-27T21:42:44.531 INFO:teuthology.orchestra.run.smithi060.stdout: 409839 4 -rw-r--r-- 1 ubuntu ubuntu 205 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/config_fwd.h 2024-03-27T21:42:44.531 INFO:teuthology.orchestra.run.smithi060.stdout: 409974 4 -rw-r--r-- 1 ubuntu ubuntu 865 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/run_cmd.h 2024-03-27T21:42:44.531 INFO:teuthology.orchestra.run.smithi060.stdout: 409970 4 -rw-r--r-- 1 ubuntu ubuntu 1032 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ref.h 2024-03-27T21:42:44.531 INFO:teuthology.orchestra.run.smithi060.stdout: 409764 16 -rw-r--r-- 1 ubuntu ubuntu 14216 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/TrackedOp.cc 2024-03-27T21:42:44.531 INFO:teuthology.orchestra.run.smithi060.stdout: 409775 4 -rw-r--r-- 1 ubuntu ubuntu 2217 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/allocate_unique.h 2024-03-27T21:42:44.531 INFO:teuthology.orchestra.run.smithi060.stdout: 409803 64 -rw-r--r-- 1 ubuntu ubuntu 63893 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/buffer.cc 2024-03-27T21:42:44.531 INFO:teuthology.orchestra.run.smithi060.stdout: 409836 44 -rw-r--r-- 1 ubuntu ubuntu 44610 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/config.cc 2024-03-27T21:42:44.531 INFO:teuthology.orchestra.run.smithi060.stdout: 409844 4 -rw-r--r-- 1 ubuntu ubuntu 2281 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/config_values.cc 2024-03-27T21:42:44.531 INFO:teuthology.orchestra.run.smithi060.stdout: 409700 8 -rw-r--r-- 1 ubuntu ubuntu 5470 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Continuation.h 2024-03-27T21:42:44.532 INFO:teuthology.orchestra.run.smithi060.stdout: 409992 4 -rw-r--r-- 1 ubuntu ubuntu 3395 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/snap_types.h 2024-03-27T21:42:44.532 INFO:teuthology.orchestra.run.smithi060.stdout: 409707 8 -rw-r--r-- 1 ubuntu ubuntu 4705 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/FastCDC.cc 2024-03-27T21:42:44.532 INFO:teuthology.orchestra.run.smithi060.stdout: 409761 4 -rw-r--r-- 1 ubuntu ubuntu 3185 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Timer.h 2024-03-27T21:42:44.532 INFO:teuthology.orchestra.run.smithi060.stdout: 409905 4 -rw-r--r-- 1 ubuntu ubuntu 949 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/item_history.h 2024-03-27T21:42:44.532 INFO:teuthology.orchestra.run.smithi060.stdout: 409885 4 -rw-r--r-- 1 ubuntu ubuntu 534 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/fd.h 2024-03-27T21:42:44.532 INFO:teuthology.orchestra.run.smithi060.stdout: 409928 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options 2024-03-27T21:42:44.532 INFO:teuthology.orchestra.run.smithi060.stdout: 409939 44 -rw-r--r-- 1 ubuntu ubuntu 41792 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options/mds.yaml.in 2024-03-27T21:42:44.532 INFO:teuthology.orchestra.run.smithi060.stdout: 409929 4 -rw-r--r-- 1 ubuntu ubuntu 3569 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options/CMakeLists.txt 2024-03-27T21:42:44.533 INFO:teuthology.orchestra.run.smithi060.stdout: 409945 120 -rw-r--r-- 1 ubuntu ubuntu 121049 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options/rgw.yaml.in 2024-03-27T21:42:44.533 INFO:teuthology.orchestra.run.smithi060.stdout: 409944 24 -rw-r--r-- 1 ubuntu ubuntu 23168 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options/rbd.yaml.in 2024-03-27T21:42:44.533 INFO:teuthology.orchestra.run.smithi060.stdout: 409940 12 -rw-r--r-- 1 ubuntu ubuntu 9820 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options/mgr.yaml.in 2024-03-27T21:42:44.533 INFO:teuthology.orchestra.run.smithi060.stdout: 409935 180 -rw-r--r-- 1 ubuntu ubuntu 183230 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options/global.yaml.in 2024-03-27T21:42:44.533 INFO:teuthology.orchestra.run.smithi060.stdout: 409941 40 -rw-r--r-- 1 ubuntu ubuntu 38597 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options/mon.yaml.in 2024-03-27T21:42:44.533 INFO:teuthology.orchestra.run.smithi060.stdout: 409934 4 -rw-r--r-- 1 ubuntu ubuntu 3843 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options/crimson.yaml.in 2024-03-27T21:42:44.533 INFO:teuthology.orchestra.run.smithi060.stdout: 409938 16 -rw-r--r-- 1 ubuntu ubuntu 15171 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options/mds-client.yaml.in 2024-03-27T21:42:44.533 INFO:teuthology.orchestra.run.smithi060.stdout: 409930 4 -rw-r--r-- 1 ubuntu ubuntu 1792 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options/build_options.cc 2024-03-27T21:42:44.534 INFO:teuthology.orchestra.run.smithi060.stdout: 409946 4 -rwxr-xr-x 1 ubuntu ubuntu 1235 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options/validate-options.py 2024-03-27T21:42:44.534 INFO:teuthology.orchestra.run.smithi060.stdout: 409942 52 -rw-r--r-- 1 ubuntu ubuntu 49617 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options/osd.yaml.in 2024-03-27T21:42:44.534 INFO:teuthology.orchestra.run.smithi060.stdout: 409932 4 -rw-r--r-- 1 ubuntu ubuntu 1120 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options/ceph-exporter.yaml.in 2024-03-27T21:42:44.534 INFO:teuthology.orchestra.run.smithi060.stdout: 409936 4 -rw-r--r-- 1 ubuntu ubuntu 2639 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options/immutable-object-cache.yaml.in 2024-03-27T21:42:44.534 INFO:teuthology.orchestra.run.smithi060.stdout: 409931 4 -rw-r--r-- 1 ubuntu ubuntu 196 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options/build_options.h 2024-03-27T21:42:44.534 INFO:teuthology.orchestra.run.smithi060.stdout: 409947 12 -rwxr-xr-x 1 ubuntu ubuntu 10377 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options/y2c.py 2024-03-27T21:42:44.534 INFO:teuthology.orchestra.run.smithi060.stdout: 409933 4 -rw-r--r-- 1 ubuntu ubuntu 3468 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options/cephfs-mirror.yaml.in 2024-03-27T21:42:44.534 INFO:teuthology.orchestra.run.smithi060.stdout: 409937 4 -rw-r--r-- 1 ubuntu ubuntu 398 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options/legacy_config_opts.h 2024-03-27T21:42:44.535 INFO:teuthology.orchestra.run.smithi060.stdout: 409943 8 -rw-r--r-- 1 ubuntu ubuntu 5426 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options/rbd-mirror.yaml.in 2024-03-27T21:42:44.535 INFO:teuthology.orchestra.run.smithi060.stdout: 409870 4 -rw-r--r-- 1 ubuntu ubuntu 652 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/dummy.cc 2024-03-27T21:42:44.535 INFO:teuthology.orchestra.run.smithi060.stdout: 409995 24 -rw-r--r-- 1 ubuntu ubuntu 22268 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/sstring.hh 2024-03-27T21:42:44.535 INFO:teuthology.orchestra.run.smithi060.stdout: 409922 44 -rw-r--r-- 1 ubuntu ubuntu 43800 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/obj_bencher.cc 2024-03-27T21:42:44.535 INFO:teuthology.orchestra.run.smithi060.stdout: 409900 8 -rw-r--r-- 1 ubuntu ubuntu 7720 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/interval_map.h 2024-03-27T21:42:44.535 INFO:teuthology.orchestra.run.smithi060.stdout: 410006 8 -rw-r--r-- 1 ubuntu ubuntu 5398 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/tracer.h 2024-03-27T21:42:44.535 INFO:teuthology.orchestra.run.smithi060.stdout: 409831 4 -rw-r--r-- 1 ubuntu ubuntu 2802 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/common_init.h 2024-03-27T21:42:44.535 INFO:teuthology.orchestra.run.smithi060.stdout: 409715 8 -rw-r--r-- 1 ubuntu ubuntu 4802 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Graylog.cc 2024-03-27T21:42:44.535 INFO:teuthology.orchestra.run.smithi060.stdout: 409926 12 -rw-r--r-- 1 ubuntu ubuntu 8445 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/options.cc 2024-03-27T21:42:44.536 INFO:teuthology.orchestra.run.smithi060.stdout: 409735 8 -rw-r--r-- 1 ubuntu ubuntu 6167 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/PluginRegistry.cc 2024-03-27T21:42:44.536 INFO:teuthology.orchestra.run.smithi060.stdout: 409953 4 -rw-r--r-- 1 ubuntu ubuntu 3119 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/perf_counters_cache.cc 2024-03-27T21:42:44.536 INFO:teuthology.orchestra.run.smithi060.stdout: 409688 4 -rw-r--r-- 1 ubuntu ubuntu 1492 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/BackTrace.h 2024-03-27T21:42:44.536 INFO:teuthology.orchestra.run.smithi060.stdout: 409724 4 -rw-r--r-- 1 ubuntu ubuntu 1881 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Journald.h 2024-03-27T21:42:44.536 INFO:teuthology.orchestra.run.smithi060.stdout: 409965 4 -rw-r--r-- 1 ubuntu ubuntu 1916 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/pretty_binary.cc 2024-03-27T21:42:44.536 INFO:teuthology.orchestra.run.smithi060.stdout: 409996 12 -rw-r--r-- 1 ubuntu ubuntu 10890 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/static_ptr.h 2024-03-27T21:42:44.536 INFO:teuthology.orchestra.run.smithi060.stdout: 409815 24 -rw-r--r-- 1 ubuntu ubuntu 21286 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ceph_json.cc 2024-03-27T21:42:44.536 INFO:teuthology.orchestra.run.smithi060.stdout: 409820 12 -rw-r--r-- 1 ubuntu ubuntu 10931 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ceph_strings.cc 2024-03-27T21:42:44.537 INFO:teuthology.orchestra.run.smithi060.stdout: 409909 12 -rw-r--r-- 1 ubuntu ubuntu 10866 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/lockdep.cc 2024-03-27T21:42:44.537 INFO:teuthology.orchestra.run.smithi060.stdout: 409910 4 -rw-r--r-- 1 ubuntu ubuntu 1442 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/lockdep.h 2024-03-27T21:42:44.537 INFO:teuthology.orchestra.run.smithi060.stdout: 409999 4 -rw-r--r-- 1 ubuntu ubuntu 923 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/strescape.h 2024-03-27T21:42:44.537 INFO:teuthology.orchestra.run.smithi060.stdout: 409897 4 -rw-r--r-- 1 ubuntu ubuntu 1014 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/hostname.cc 2024-03-27T21:42:44.537 INFO:teuthology.orchestra.run.smithi060.stdout: 409951 20 -rw-r--r-- 1 ubuntu ubuntu 17224 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/perf_counters.cc 2024-03-27T21:42:44.537 INFO:teuthology.orchestra.run.smithi060.stdout: 409962 4 -rw-r--r-- 1 ubuntu ubuntu 3198 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/pick_address.h 2024-03-27T21:42:44.537 INFO:teuthology.orchestra.run.smithi060.stdout: 409838 4 -rw-r--r-- 1 ubuntu ubuntu 1523 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/config_cacher.h 2024-03-27T21:42:44.537 INFO:teuthology.orchestra.run.smithi060.stdout: 409769 4 -rw-r--r-- 1 ubuntu ubuntu 3678 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/addr_parsing.c 2024-03-27T21:42:44.538 INFO:teuthology.orchestra.run.smithi060.stdout: 409816 28 -rw-r--r-- 1 ubuntu ubuntu 25446 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ceph_json.h 2024-03-27T21:42:44.538 INFO:teuthology.orchestra.run.smithi060.stdout: 409956 4 -rw-r--r-- 1 ubuntu ubuntu 1351 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/perf_counters_collection.h 2024-03-27T21:42:44.538 INFO:teuthology.orchestra.run.smithi060.stdout: 409948 4 -rw-r--r-- 1 ubuntu ubuntu 335 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ostream_temp.cc 2024-03-27T21:42:44.538 INFO:teuthology.orchestra.run.smithi060.stdout: 410017 8 -rw-r--r-- 1 ubuntu ubuntu 8120 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/versioned_variant.h 2024-03-27T21:42:44.538 INFO:teuthology.orchestra.run.smithi060.stdout: 409960 8 -rw-r--r-- 1 ubuntu ubuntu 7206 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/perf_histogram.h 2024-03-27T21:42:44.538 INFO:teuthology.orchestra.run.smithi060.stdout: 409848 16 -rw-r--r-- 1 ubuntu ubuntu 15943 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/crc32c.cc 2024-03-27T21:42:44.538 INFO:teuthology.orchestra.run.smithi060.stdout: 409756 4 -rw-r--r-- 1 ubuntu ubuntu 1941 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Thread.h 2024-03-27T21:42:44.538 INFO:teuthology.orchestra.run.smithi060.stdout: 409957 8 -rw-r--r-- 1 ubuntu ubuntu 6091 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/perf_counters_key.cc 2024-03-27T21:42:44.539 INFO:teuthology.orchestra.run.smithi060.stdout: 409732 4 -rw-r--r-- 1 ubuntu ubuntu 2183 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/OpQueue.h 2024-03-27T21:42:44.539 INFO:teuthology.orchestra.run.smithi060.stdout: 409893 4 -rw-r--r-- 1 ubuntu ubuntu 1415 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/histogram.cc 2024-03-27T21:42:44.539 INFO:teuthology.orchestra.run.smithi060.stdout: 409847 8 -rw-r--r-- 1 ubuntu ubuntu 4309 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/convenience.h 2024-03-27T21:42:44.539 INFO:teuthology.orchestra.run.smithi060.stdout: 409985 4 -rw-r--r-- 1 ubuntu ubuntu 1060 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/shared_mutex_debug.h 2024-03-27T21:42:44.539 INFO:teuthology.orchestra.run.smithi060.stdout: 409684 4 -rw-r--r-- 1 ubuntu ubuntu 1041 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/AsyncOpTracker.cc 2024-03-27T21:42:44.539 INFO:teuthology.orchestra.run.smithi060.stdout: 409883 8 -rw-r--r-- 1 ubuntu ubuntu 5559 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/fault_injector.h 2024-03-27T21:42:44.539 INFO:teuthology.orchestra.run.smithi060.stdout: 409768 20 -rw-r--r-- 1 ubuntu ubuntu 18509 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/WorkQueue.h 2024-03-27T21:42:44.539 INFO:teuthology.orchestra.run.smithi060.stdout: 409993 8 -rw-r--r-- 1 ubuntu ubuntu 5205 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/solaris_errno.cc 2024-03-27T21:42:44.540 INFO:teuthology.orchestra.run.smithi060.stdout: 409814 4 -rw-r--r-- 1 ubuntu ubuntu 2949 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ceph_hash.cc 2024-03-27T21:42:44.540 INFO:teuthology.orchestra.run.smithi060.stdout: 409972 4 -rw-r--r-- 1 ubuntu ubuntu 691 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/reverse.h 2024-03-27T21:42:44.540 INFO:teuthology.orchestra.run.smithi060.stdout: 409817 8 -rw-r--r-- 1 ubuntu ubuntu 6338 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ceph_mutex.h 2024-03-27T21:42:44.540 INFO:teuthology.orchestra.run.smithi060.stdout: 409997 4 -rw-r--r-- 1 ubuntu ubuntu 1638 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/str_list.cc 2024-03-27T21:42:44.540 INFO:teuthology.orchestra.run.smithi060.stdout: 409763 4 -rw-r--r-- 1 ubuntu ubuntu 2258 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/TracepointProvider.h 2024-03-27T21:42:44.540 INFO:teuthology.orchestra.run.smithi060.stdout: 409739 16 -rw-r--r-- 1 ubuntu ubuntu 14280 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/PriorityCache.cc 2024-03-27T21:42:44.540 INFO:teuthology.orchestra.run.smithi060.stdout: 409689 4 -rw-r--r-- 1 ubuntu ubuntu 1010 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/CDC.cc 2024-03-27T21:42:44.540 INFO:teuthology.orchestra.run.smithi060.stdout: 409723 8 -rw-r--r-- 1 ubuntu ubuntu 8058 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Journald.cc 2024-03-27T21:42:44.541 INFO:teuthology.orchestra.run.smithi060.stdout: 409884 4 -rw-r--r-- 1 ubuntu ubuntu 1457 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/fd.cc 2024-03-27T21:42:44.541 INFO:teuthology.orchestra.run.smithi060.stdout: 409694 4 -rw-r--r-- 1 ubuntu ubuntu 3192 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/CommandTable.h 2024-03-27T21:42:44.541 INFO:teuthology.orchestra.run.smithi060.stdout: 410000 8 -rw-r--r-- 1 ubuntu ubuntu 7829 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/strtol.cc 2024-03-27T21:42:44.541 INFO:teuthology.orchestra.run.smithi060.stdout: 409952 12 -rw-r--r-- 1 ubuntu ubuntu 11376 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/perf_counters.h 2024-03-27T21:42:44.541 INFO:teuthology.orchestra.run.smithi060.stdout: 409882 4 -rw-r--r-- 1 ubuntu ubuntu 1494 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/fair_mutex.h 2024-03-27T21:42:44.541 INFO:teuthology.orchestra.run.smithi060.stdout: 410016 4 -rw-r--r-- 1 ubuntu ubuntu 940 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/version.h 2024-03-27T21:42:44.541 INFO:teuthology.orchestra.run.smithi060.stdout: 410009 4 -rw-r--r-- 1 ubuntu ubuntu 1438 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/url_escape.cc 2024-03-27T21:42:44.541 INFO:teuthology.orchestra.run.smithi060.stdout: 409980 4 -rw-r--r-- 1 ubuntu ubuntu 236 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/sctp_crc32.h 2024-03-27T21:42:44.541 INFO:teuthology.orchestra.run.smithi060.stdout: 409827 4 -rw-r--r-- 1 ubuntu ubuntu 2668 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/code_environment.cc 2024-03-27T21:42:44.542 INFO:teuthology.orchestra.run.smithi060.stdout: 409853 4 -rw-r--r-- 1 ubuntu ubuntu 1143 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/crc32c_intel_fast.c 2024-03-27T21:42:44.542 INFO:teuthology.orchestra.run.smithi060.stdout: 409749 4 -rw-r--r-- 1 ubuntu ubuntu 1964 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/SloppyCRCMap.h 2024-03-27T21:42:44.542 INFO:teuthology.orchestra.run.smithi060.stdout: 409908 4 -rw-r--r-- 1 ubuntu ubuntu 351 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/linux_version.h 2024-03-27T21:42:44.542 INFO:teuthology.orchestra.run.smithi060.stdout: 409731 4 -rw-r--r-- 1 ubuntu ubuntu 1073 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/MemoryModel.h 2024-03-27T21:42:44.542 INFO:teuthology.orchestra.run.smithi060.stdout: 409850 4 -rw-r--r-- 1 ubuntu ubuntu 445 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/crc32c_aarch64.h 2024-03-27T21:42:44.542 INFO:teuthology.orchestra.run.smithi060.stdout: 409721 4 -rw-r--r-- 1 ubuntu ubuntu 2828 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/HeartbeatMap.h 2024-03-27T21:42:44.542 INFO:teuthology.orchestra.run.smithi060.stdout: 409973 4 -rw-r--r-- 1 ubuntu ubuntu 2475 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/run_cmd.cc 2024-03-27T21:42:44.542 INFO:teuthology.orchestra.run.smithi060.stdout: 409741 4 -rw-r--r-- 1 ubuntu ubuntu 1362 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/QueueRing.h 2024-03-27T21:42:44.543 INFO:teuthology.orchestra.run.smithi060.stdout: 409780 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/async 2024-03-27T21:42:44.543 INFO:teuthology.orchestra.run.smithi060.stdout: 409783 8 -rw-r--r-- 1 ubuntu ubuntu 7225 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/async/blocked_completion.h 2024-03-27T21:42:44.543 INFO:teuthology.orchestra.run.smithi060.stdout: 409786 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/async/detail 2024-03-27T21:42:44.543 INFO:teuthology.orchestra.run.smithi060.stdout: 409787 8 -rw-r--r-- 1 ubuntu ubuntu 5078 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/async/detail/shared_lock.h 2024-03-27T21:42:44.543 INFO:teuthology.orchestra.run.smithi060.stdout: 409788 12 -rw-r--r-- 1 ubuntu ubuntu 9317 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/async/detail/shared_mutex.h 2024-03-27T21:42:44.543 INFO:teuthology.orchestra.run.smithi060.stdout: 409792 8 -rw-r--r-- 1 ubuntu ubuntu 5059 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/async/waiter.h 2024-03-27T21:42:44.543 INFO:teuthology.orchestra.run.smithi060.stdout: 409782 4 -rw-r--r-- 1 ubuntu ubuntu 1272 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/async/bind_like.h 2024-03-27T21:42:44.543 INFO:teuthology.orchestra.run.smithi060.stdout: 409785 4 -rw-r--r-- 1 ubuntu ubuntu 2920 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/async/context_pool.h 2024-03-27T21:42:44.543 INFO:teuthology.orchestra.run.smithi060.stdout: 409790 4 -rw-r--r-- 1 ubuntu ubuntu 3105 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/async/librados_completion.h 2024-03-27T21:42:44.544 INFO:teuthology.orchestra.run.smithi060.stdout: 409791 8 -rw-r--r-- 1 ubuntu ubuntu 6016 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/async/shared_mutex.h 2024-03-27T21:42:44.544 INFO:teuthology.orchestra.run.smithi060.stdout: 409789 4 -rw-r--r-- 1 ubuntu ubuntu 3119 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/async/forward_handler.h 2024-03-27T21:42:44.544 INFO:teuthology.orchestra.run.smithi060.stdout: 409781 4 -rw-r--r-- 1 ubuntu ubuntu 3359 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/async/bind_handler.h 2024-03-27T21:42:44.544 INFO:teuthology.orchestra.run.smithi060.stdout: 409784 16 -rw-r--r-- 1 ubuntu ubuntu 12854 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/async/completion.h 2024-03-27T21:42:44.544 INFO:teuthology.orchestra.run.smithi060.stdout: 409793 4 -rw-r--r-- 1 ubuntu ubuntu 1767 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/async/yield_context.h 2024-03-27T21:42:44.544 INFO:teuthology.orchestra.run.smithi060.stdout: 409959 4 -rw-r--r-- 1 ubuntu ubuntu 3271 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/perf_histogram.cc 2024-03-27T21:42:44.544 INFO:teuthology.orchestra.run.smithi060.stdout: 409911 4 -rw-r--r-- 1 ubuntu ubuntu 2813 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/lru_map.h 2024-03-27T21:42:44.544 INFO:teuthology.orchestra.run.smithi060.stdout: 409711 4 -rw-r--r-- 1 ubuntu ubuntu 464 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/FixedCDC.cc 2024-03-27T21:42:44.545 INFO:teuthology.orchestra.run.smithi060.stdout: 409743 8 -rw-r--r-- 1 ubuntu ubuntu 5508 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Readahead.cc 2024-03-27T21:42:44.545 INFO:teuthology.orchestra.run.smithi060.stdout: 409868 4 -rw-r--r-- 1 ubuntu ubuntu 201 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/dout.cc 2024-03-27T21:42:44.545 INFO:teuthology.orchestra.run.smithi060.stdout: 409797 20 -rw-r--r-- 1 ubuntu ubuntu 18122 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/bit_vector.hpp 2024-03-27T21:42:44.545 INFO:teuthology.orchestra.run.smithi060.stdout: 409867 8 -rw-r--r-- 1 ubuntu ubuntu 4743 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/dns_resolve.h 2024-03-27T21:42:44.545 INFO:teuthology.orchestra.run.smithi060.stdout: 409913 8 -rw-r--r-- 1 ubuntu ubuntu 4469 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/mempool.cc 2024-03-27T21:42:44.545 INFO:teuthology.orchestra.run.smithi060.stdout: 409757 24 -rw-r--r-- 1 ubuntu ubuntu 21092 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Throttle.cc 2024-03-27T21:42:44.545 INFO:teuthology.orchestra.run.smithi060.stdout: 409862 4 -rw-r--r-- 1 ubuntu ubuntu 850 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/debug.h 2024-03-27T21:42:44.545 INFO:teuthology.orchestra.run.smithi060.stdout: 409843 4 -rw-r--r-- 1 ubuntu ubuntu 671 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/config_tracker.h 2024-03-27T21:42:44.545 INFO:teuthology.orchestra.run.smithi060.stdout: 409983 12 -rw-r--r-- 1 ubuntu ubuntu 9272 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/shared_cache.hpp 2024-03-27T21:42:44.546 INFO:teuthology.orchestra.run.smithi060.stdout: 409898 4 -rw-r--r-- 1 ubuntu ubuntu 601 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/hostname.h 2024-03-27T21:42:44.546 INFO:teuthology.orchestra.run.smithi060.stdout: 410019 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/win32 2024-03-27T21:42:44.546 INFO:teuthology.orchestra.run.smithi060.stdout: 410028 4 -rw-r--r-- 1 ubuntu ubuntu 3309 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/win32/ifaddrs.cc 2024-03-27T21:42:44.546 INFO:teuthology.orchestra.run.smithi060.stdout: 410033 4 -rw-r--r-- 1 ubuntu ubuntu 1598 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/win32/syslog.cc 2024-03-27T21:42:44.546 INFO:teuthology.orchestra.run.smithi060.stdout: 410024 4 -rw-r--r-- 1 ubuntu ubuntu 890 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/win32/dlfcn.cc 2024-03-27T21:42:44.546 INFO:teuthology.orchestra.run.smithi060.stdout: 410020 8 -rw-r--r-- 1 ubuntu ubuntu 7959 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/win32/SubProcess.cc 2024-03-27T21:42:44.546 INFO:teuthology.orchestra.run.smithi060.stdout: 410026 24 -rw-r--r-- 1 ubuntu ubuntu 22524 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/win32/errno.cc 2024-03-27T21:42:44.546 INFO:teuthology.orchestra.run.smithi060.stdout: 410035 4 -rw-r--r-- 1 ubuntu ubuntu 444 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/win32/wstring.h 2024-03-27T21:42:44.547 INFO:teuthology.orchestra.run.smithi060.stdout: 410034 4 -rw-r--r-- 1 ubuntu ubuntu 656 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/win32/wstring.cc 2024-03-27T21:42:44.547 INFO:teuthology.orchestra.run.smithi060.stdout: 410030 4 -rw-r--r-- 1 ubuntu ubuntu 890 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/win32/registry.h 2024-03-27T21:42:44.547 INFO:teuthology.orchestra.run.smithi060.stdout: 410031 8 -rw-r--r-- 1 ubuntu ubuntu 4638 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/win32/service.cc 2024-03-27T21:42:44.547 INFO:teuthology.orchestra.run.smithi060.stdout: 410027 4 -rw-r--r-- 1 ubuntu ubuntu 574 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/win32/event_logging.mc 2024-03-27T21:42:44.547 INFO:teuthology.orchestra.run.smithi060.stdout: 410032 4 -rw-r--r-- 1 ubuntu ubuntu 1240 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/win32/service.h 2024-03-27T21:42:44.547 INFO:teuthology.orchestra.run.smithi060.stdout: 410029 8 -rw-r--r-- 1 ubuntu ubuntu 4285 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/win32/registry.cc 2024-03-27T21:42:44.547 INFO:teuthology.orchestra.run.smithi060.stdout: 410022 4 -rw-r--r-- 1 ubuntu ubuntu 327 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/win32/code_page.manifest 2024-03-27T21:42:44.547 INFO:teuthology.orchestra.run.smithi060.stdout: 410023 4 -rw-r--r-- 1 ubuntu ubuntu 89 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/win32/code_page.rc 2024-03-27T21:42:44.548 INFO:teuthology.orchestra.run.smithi060.stdout: 410021 4 -rw-r--r-- 1 ubuntu ubuntu 2263 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/win32/blkdev.cc 2024-03-27T21:42:44.548 INFO:teuthology.orchestra.run.smithi060.stdout: 410025 4 -rw-r--r-- 1 ubuntu ubuntu 1588 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/win32/dns_resolve.cc 2024-03-27T21:42:44.548 INFO:teuthology.orchestra.run.smithi060.stdout: 409738 12 -rw-r--r-- 1 ubuntu ubuntu 9480 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/PrioritizedQueue.h 2024-03-27T21:42:44.548 INFO:teuthology.orchestra.run.smithi060.stdout: 409899 8 -rw-r--r-- 1 ubuntu ubuntu 7222 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/inline_variant.h 2024-03-27T21:42:44.548 INFO:teuthology.orchestra.run.smithi060.stdout: 409860 4 -rw-r--r-- 1 ubuntu ubuntu 3457 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/crc32c_ppc_fast_zero_asm.S 2024-03-27T21:42:44.548 INFO:teuthology.orchestra.run.smithi060.stdout: 409989 4 -rw-r--r-- 1 ubuntu ubuntu 1410 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/signal.h 2024-03-27T21:42:44.548 INFO:teuthology.orchestra.run.smithi060.stdout: 409873 4 -rw-r--r-- 1 ubuntu ubuntu 942 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/environment.cc 2024-03-27T21:42:44.548 INFO:teuthology.orchestra.run.smithi060.stdout: 409753 4 -rw-r--r-- 1 ubuntu ubuntu 2242 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/TextTable.cc 2024-03-27T21:42:44.549 INFO:teuthology.orchestra.run.smithi060.stdout: 409708 4 -rw-r--r-- 1 ubuntu ubuntu 1803 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/FastCDC.h 2024-03-27T21:42:44.549 INFO:teuthology.orchestra.run.smithi060.stdout: 409825 20 -rw-r--r-- 1 ubuntu ubuntu 19223 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/cmdparse.cc 2024-03-27T21:42:44.549 INFO:teuthology.orchestra.run.smithi060.stdout: 409755 8 -rw-r--r-- 1 ubuntu ubuntu 4861 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Thread.cc 2024-03-27T21:42:44.549 INFO:teuthology.orchestra.run.smithi060.stdout: 409954 4 -rw-r--r-- 1 ubuntu ubuntu 3121 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/perf_counters_cache.h 2024-03-27T21:42:44.549 INFO:teuthology.orchestra.run.smithi060.stdout: 409802 8 -rw-r--r-- 1 ubuntu ubuntu 5718 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/bounded_key_counter.h 2024-03-27T21:42:44.549 INFO:teuthology.orchestra.run.smithi060.stdout: 409695 4 -rw-r--r-- 1 ubuntu ubuntu 3204 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Cond.h 2024-03-27T21:42:44.549 INFO:teuthology.orchestra.run.smithi060.stdout: 409823 20 -rw-r--r-- 1 ubuntu ubuntu 19697 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ceph_time.h 2024-03-27T21:42:44.549 INFO:teuthology.orchestra.run.smithi060.stdout: 409752 4 -rw-r--r-- 1 ubuntu ubuntu 2902 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/SubProcess.h 2024-03-27T21:42:44.550 INFO:teuthology.orchestra.run.smithi060.stdout: 409778 4 -rw-r--r-- 1 ubuntu ubuntu 432 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/armor.h 2024-03-27T21:42:44.550 INFO:teuthology.orchestra.run.smithi060.stdout: 409998 8 -rw-r--r-- 1 ubuntu ubuntu 4811 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/str_map.cc 2024-03-27T21:42:44.550 INFO:teuthology.orchestra.run.smithi060.stdout: 409776 4 -rw-r--r-- 1 ubuntu ubuntu 202 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/arch.h 2024-03-27T21:42:44.550 INFO:teuthology.orchestra.run.smithi060.stdout: 409891 4 -rw-r--r-- 1 ubuntu ubuntu 965 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/hex.cc 2024-03-27T21:42:44.550 INFO:teuthology.orchestra.run.smithi060.stdout: 409904 4 -rw-r--r-- 1 ubuntu ubuntu 1803 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/iso_8601.h 2024-03-27T21:42:44.550 INFO:teuthology.orchestra.run.smithi060.stdout: 409762 4 -rw-r--r-- 1 ubuntu ubuntu 1148 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/TracepointProvider.cc 2024-03-27T21:42:44.550 INFO:teuthology.orchestra.run.smithi060.stdout: 409726 12 -rw-r--r-- 1 ubuntu ubuntu 9242 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/LogClient.cc 2024-03-27T21:42:44.550 INFO:teuthology.orchestra.run.smithi060.stdout: 409807 4 -rw-r--r-- 1 ubuntu ubuntu 2655 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ceph_atomic.h 2024-03-27T21:42:44.551 INFO:teuthology.orchestra.run.smithi060.stdout: 409725 4 -rw-r--r-- 1 ubuntu ubuntu 3133 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/LRUSet.h 2024-03-27T21:42:44.551 INFO:teuthology.orchestra.run.smithi060.stdout: 409982 4 -rw-r--r-- 1 ubuntu ubuntu 312 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/secret.h 2024-03-27T21:42:44.551 INFO:teuthology.orchestra.run.smithi060.stdout: 409842 12 -rw-r--r-- 1 ubuntu ubuntu 10199 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/config_proxy.h 2024-03-27T21:42:44.551 INFO:teuthology.orchestra.run.smithi060.stdout: 409819 4 -rw-r--r-- 1 ubuntu ubuntu 2117 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ceph_releases.h 2024-03-27T21:42:44.551 INFO:teuthology.orchestra.run.smithi060.stdout: 409767 12 -rw-r--r-- 1 ubuntu ubuntu 10365 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/WorkQueue.cc 2024-03-27T21:42:44.551 INFO:teuthology.orchestra.run.smithi060.stdout: 409727 8 -rw-r--r-- 1 ubuntu ubuntu 6440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/LogClient.h 2024-03-27T21:42:44.551 INFO:teuthology.orchestra.run.smithi060.stdout: 409852 4 -rw-r--r-- 1 ubuntu ubuntu 299 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/crc32c_intel_baseline.h 2024-03-27T21:42:44.551 INFO:teuthology.orchestra.run.smithi060.stdout: 409955 4 -rw-r--r-- 1 ubuntu ubuntu 1808 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/perf_counters_collection.cc 2024-03-27T21:42:44.551 INFO:teuthology.orchestra.run.smithi060.stdout: 409967 4 -rw-r--r-- 1 ubuntu ubuntu 717 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ptr_wrapper.h 2024-03-27T21:42:44.552 INFO:teuthology.orchestra.run.smithi060.stdout: 409822 12 -rw-r--r-- 1 ubuntu ubuntu 9023 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ceph_time.cc 2024-03-27T21:42:44.552 INFO:teuthology.orchestra.run.smithi060.stdout: 409811 8 -rw-r--r-- 1 ubuntu ubuntu 5791 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ceph_crypto.h 2024-03-27T21:42:44.552 INFO:teuthology.orchestra.run.smithi060.stdout: 409818 4 -rw-r--r-- 1 ubuntu ubuntu 1577 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ceph_releases.cc 2024-03-27T21:42:44.552 INFO:teuthology.orchestra.run.smithi060.stdout: 410010 4 -rw-r--r-- 1 ubuntu ubuntu 240 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/url_escape.h 2024-03-27T21:42:44.552 INFO:teuthology.orchestra.run.smithi060.stdout: 409877 8 -rw-r--r-- 1 ubuntu ubuntu 5577 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/error_code.cc 2024-03-27T21:42:44.552 INFO:teuthology.orchestra.run.smithi060.stdout: 409703 4 -rw-r--r-- 1 ubuntu ubuntu 1755 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/DecayCounter.cc 2024-03-27T21:42:44.552 INFO:teuthology.orchestra.run.smithi060.stdout: 409981 4 -rw-r--r-- 1 ubuntu ubuntu 2024 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/secret.c 2024-03-27T21:42:44.552 INFO:teuthology.orchestra.run.smithi060.stdout: 409798 28 -rw-r--r-- 1 ubuntu ubuntu 27143 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/blkdev.cc 2024-03-27T21:42:44.553 INFO:teuthology.orchestra.run.smithi060.stdout: 409917 4 -rw-r--r-- 1 ubuntu ubuntu 576 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/module.h 2024-03-27T21:42:44.553 INFO:teuthology.orchestra.run.smithi060.stdout: 409896 16 -rw-r--r-- 1 ubuntu ubuntu 15191 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/hobject.h 2024-03-27T21:42:44.553 INFO:teuthology.orchestra.run.smithi060.stdout: 409692 8 -rw-r--r-- 1 ubuntu ubuntu 6958 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Checksummer.h 2024-03-27T21:42:44.553 INFO:teuthology.orchestra.run.smithi060.stdout: 409750 8 -rw-r--r-- 1 ubuntu ubuntu 4934 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/StackStringStream.h 2024-03-27T21:42:44.553 INFO:teuthology.orchestra.run.smithi060.stdout: 409718 4 -rw-r--r-- 1 ubuntu ubuntu 3603 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/HTMLFormatter.cc 2024-03-27T21:42:44.553 INFO:teuthology.orchestra.run.smithi060.stdout: 409806 4 -rw-r--r-- 1 ubuntu ubuntu 3045 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ceph_argparse.h 2024-03-27T21:42:44.553 INFO:teuthology.orchestra.run.smithi060.stdout: 409858 16 -rw-r--r-- 1 ubuntu ubuntu 14816 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/crc32c_ppc_asm.S 2024-03-27T21:42:44.553 INFO:teuthology.orchestra.run.smithi060.stdout: 409812 4 -rw-r--r-- 1 ubuntu ubuntu 336 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ceph_frag.cc 2024-03-27T21:42:44.553 INFO:teuthology.orchestra.run.smithi060.stdout: 409888 12 -rw-r--r-- 1 ubuntu ubuntu 10162 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/freebsd_errno.cc 2024-03-27T21:42:44.554 INFO:teuthology.orchestra.run.smithi060.stdout: 409880 4 -rw-r--r-- 1 ubuntu ubuntu 2043 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/escape.h 2024-03-27T21:42:44.554 INFO:teuthology.orchestra.run.smithi060.stdout: 409976 4 -rw-r--r-- 1 ubuntu ubuntu 2482 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/safe_io.h 2024-03-27T21:42:44.554 INFO:teuthology.orchestra.run.smithi060.stdout: 409733 12 -rw-r--r-- 1 ubuntu ubuntu 10488 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/OutputDataSocket.cc 2024-03-27T21:42:44.554 INFO:teuthology.orchestra.run.smithi060.stdout: 409920 8 -rw-r--r-- 1 ubuntu ubuntu 5230 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/numa.cc 2024-03-27T21:42:44.554 INFO:teuthology.orchestra.run.smithi060.stdout: 409845 4 -rw-r--r-- 1 ubuntu ubuntu 3254 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/config_values.h 2024-03-27T21:42:44.554 INFO:teuthology.orchestra.run.smithi060.stdout: 409687 4 -rw-r--r-- 1 ubuntu ubuntu 2248 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/BackTrace.cc 2024-03-27T21:42:44.554 INFO:teuthology.orchestra.run.smithi060.stdout: 409772 8 -rw-r--r-- 1 ubuntu ubuntu 4598 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/admin_socket_client.cc 2024-03-27T21:42:44.554 INFO:teuthology.orchestra.run.smithi060.stdout: 410002 4 -rw-r--r-- 1 ubuntu ubuntu 3084 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/subsys.h 2024-03-27T21:42:44.555 INFO:teuthology.orchestra.run.smithi060.stdout: 409744 8 -rw-r--r-- 1 ubuntu ubuntu 5113 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Readahead.h 2024-03-27T21:42:44.555 INFO:teuthology.orchestra.run.smithi060.stdout: 409890 4 -rw-r--r-- 1 ubuntu ubuntu 1474 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/function_signature.h 2024-03-27T21:42:44.555 INFO:teuthology.orchestra.run.smithi060.stdout: 409861 8 -rw-r--r-- 1 ubuntu ubuntu 5457 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/darwin_errno.cc 2024-03-27T21:42:44.555 INFO:teuthology.orchestra.run.smithi060.stdout: 409794 12 -rw-r--r-- 1 ubuntu ubuntu 8854 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/autovector.h 2024-03-27T21:42:44.555 INFO:teuthology.orchestra.run.smithi060.stdout: 409706 4 -rw-r--r-- 1 ubuntu ubuntu 2594 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/EventTrace.h 2024-03-27T21:42:44.555 INFO:teuthology.orchestra.run.smithi060.stdout: 409777 4 -rw-r--r-- 1 ubuntu ubuntu 2454 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/armor.c 2024-03-27T21:42:44.555 INFO:teuthology.orchestra.run.smithi060.stdout: 409728 12 -rw-r--r-- 1 ubuntu ubuntu 8443 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/LogEntry.cc 2024-03-27T21:42:44.556 INFO:teuthology.orchestra.run.smithi060.stdout: 409925 4 -rw-r--r-- 1 ubuntu ubuntu 633 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/openssl_opts_handler.h 2024-03-27T21:42:44.556 INFO:teuthology.orchestra.run.smithi060.stdout: 409968 4 -rw-r--r-- 1 ubuntu ubuntu 3907 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/random_string.cc 2024-03-27T21:42:44.556 INFO:teuthology.orchestra.run.smithi060.stdout: 409719 4 -rw-r--r-- 1 ubuntu ubuntu 1201 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/HTMLFormatter.h 2024-03-27T21:42:44.556 INFO:teuthology.orchestra.run.smithi060.stdout: 409840 4 -rw-r--r-- 1 ubuntu ubuntu 1819 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/config_obs.h 2024-03-27T21:42:44.556 INFO:teuthology.orchestra.run.smithi060.stdout: 409747 4 -rw-r--r-- 1 ubuntu ubuntu 864 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Semaphore.h 2024-03-27T21:42:44.556 INFO:teuthology.orchestra.run.smithi060.stdout: 409745 4 -rw-r--r-- 1 ubuntu ubuntu 1025 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/RefCountedObj.cc 2024-03-27T21:42:44.556 INFO:teuthology.orchestra.run.smithi060.stdout: 410001 4 -rw-r--r-- 1 ubuntu ubuntu 3318 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/strtol.h 2024-03-27T21:42:44.556 INFO:teuthology.orchestra.run.smithi060.stdout: 409964 4 -rw-r--r-- 1 ubuntu ubuntu 2567 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ppc-opcode.h 2024-03-27T21:42:44.557 INFO:teuthology.orchestra.run.smithi060.stdout: 410004 4 -rw-r--r-- 1 ubuntu ubuntu 1314 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/sync_filesystem.h 2024-03-27T21:42:44.557 INFO:teuthology.orchestra.run.smithi060.stdout: 409857 4 -rw-r--r-- 1 ubuntu ubuntu 577 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/crc32c_ppc.h 2024-03-27T21:42:44.557 INFO:teuthology.orchestra.run.smithi060.stdout: 409856 4 -rw-r--r-- 1 ubuntu ubuntu 3676 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/crc32c_ppc.c 2024-03-27T21:42:44.557 INFO:teuthology.orchestra.run.smithi060.stdout: 409902 8 -rw-r--r-- 1 ubuntu ubuntu 4515 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ipaddr.cc 2024-03-27T21:42:44.557 INFO:teuthology.orchestra.run.smithi060.stdout: 409795 4 -rw-r--r-- 1 ubuntu ubuntu 1710 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/bit_str.cc 2024-03-27T21:42:44.557 INFO:teuthology.orchestra.run.smithi060.stdout: 409918 4 -rw-r--r-- 1 ubuntu ubuntu 1508 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/mutex_debug.cc 2024-03-27T21:42:44.557 INFO:teuthology.orchestra.run.smithi060.stdout: 409710 8 -rw-r--r-- 1 ubuntu ubuntu 6739 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Finisher.h 2024-03-27T21:42:44.557 INFO:teuthology.orchestra.run.smithi060.stdout: 409903 8 -rw-r--r-- 1 ubuntu ubuntu 4735 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/iso_8601.cc 2024-03-27T21:42:44.558 INFO:teuthology.orchestra.run.smithi060.stdout: 409949 4 -rw-r--r-- 1 ubuntu ubuntu 1379 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ostream_temp.h 2024-03-27T21:42:44.558 INFO:teuthology.orchestra.run.smithi060.stdout: 409704 4 -rw-r--r-- 1 ubuntu ubuntu 2902 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/DecayCounter.h 2024-03-27T21:42:44.558 INFO:teuthology.orchestra.run.smithi060.stdout: 409916 4 -rw-r--r-- 1 ubuntu ubuntu 2084 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/module.c 2024-03-27T21:42:44.558 INFO:teuthology.orchestra.run.smithi060.stdout: 409824 12 -rw-r--r-- 1 ubuntu ubuntu 9144 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ceph_timer.h 2024-03-27T21:42:44.558 INFO:teuthology.orchestra.run.smithi060.stdout: 409809 12 -rw-r--r-- 1 ubuntu ubuntu 10853 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ceph_context.h 2024-03-27T21:42:44.558 INFO:teuthology.orchestra.run.smithi060.stdout: 409977 8 -rw-r--r-- 1 ubuntu ubuntu 7180 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/scrub_types.cc 2024-03-27T21:42:44.558 INFO:teuthology.orchestra.run.smithi060.stdout: 409808 32 -rw-r--r-- 1 ubuntu ubuntu 30193 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ceph_context.cc 2024-03-27T21:42:44.558 INFO:teuthology.orchestra.run.smithi060.stdout: 409685 4 -rw-r--r-- 1 ubuntu ubuntu 1018 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/AsyncOpTracker.h 2024-03-27T21:42:44.558 INFO:teuthology.orchestra.run.smithi060.stdout: 409894 4 -rw-r--r-- 1 ubuntu ubuntu 2973 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/histogram.h 2024-03-27T21:42:44.559 INFO:teuthology.orchestra.run.smithi060.stdout: 409760 8 -rw-r--r-- 1 ubuntu ubuntu 6609 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Timer.cc 2024-03-27T21:42:44.559 INFO:teuthology.orchestra.run.smithi060.stdout: 409924 8 -rw-r--r-- 1 ubuntu ubuntu 4164 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/openssl_opts_handler.cc 2024-03-27T21:42:44.559 INFO:teuthology.orchestra.run.smithi060.stdout: 409832 16 -rw-r--r-- 1 ubuntu ubuntu 12911 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/compat.cc 2024-03-27T21:42:44.559 INFO:teuthology.orchestra.run.smithi060.stdout: 409837 16 -rw-r--r-- 1 ubuntu ubuntu 13127 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/config.h 2024-03-27T21:42:44.559 INFO:teuthology.orchestra.run.smithi060.stdout: 409800 4 -rw-r--r-- 1 ubuntu ubuntu 3722 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/bloom_filter.cc 2024-03-27T21:42:44.559 INFO:teuthology.orchestra.run.smithi060.stdout: 409958 8 -rw-r--r-- 1 ubuntu ubuntu 4245 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/perf_counters_key.h 2024-03-27T21:42:44.559 INFO:teuthology.orchestra.run.smithi060.stdout: 409774 8 -rw-r--r-- 1 ubuntu ubuntu 5361 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/aix_errno.cc 2024-03-27T21:42:44.559 INFO:teuthology.orchestra.run.smithi060.stdout: 409849 12 -rw-r--r-- 1 ubuntu ubuntu 8681 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/crc32c_aarch64.c 2024-03-27T21:42:44.560 INFO:teuthology.orchestra.run.smithi060.stdout: 409701 8 -rw-r--r-- 1 ubuntu ubuntu 7888 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Cycles.cc 2024-03-27T21:42:44.560 INFO:teuthology.orchestra.run.smithi060.stdout: 409699 4 -rw-r--r-- 1 ubuntu ubuntu 957 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ContextCompletion.h 2024-03-27T21:42:44.560 INFO:teuthology.orchestra.run.smithi060.stdout: 409878 8 -rw-r--r-- 1 ubuntu ubuntu 4341 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/error_code.h 2024-03-27T21:42:44.560 INFO:teuthology.orchestra.run.smithi060.stdout: 409921 4 -rw-r--r-- 1 ubuntu ubuntu 634 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/numa.h 2024-03-27T21:42:44.560 INFO:teuthology.orchestra.run.smithi060.stdout: 409984 4 -rw-r--r-- 1 ubuntu ubuntu 3780 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/shared_mutex_debug.cc 2024-03-27T21:42:44.560 INFO:teuthology.orchestra.run.smithi060.stdout: 410013 12 -rw-r--r-- 1 ubuntu ubuntu 11412 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/util.cc 2024-03-27T21:42:44.560 INFO:teuthology.orchestra.run.smithi060.stdout: 410005 4 -rw-r--r-- 1 ubuntu ubuntu 3747 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/tracer.cc 2024-03-27T21:42:44.560 INFO:teuthology.orchestra.run.smithi060.stdout: 409742 8 -rw-r--r-- 1 ubuntu ubuntu 6180 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/RWLock.h 2024-03-27T21:42:44.561 INFO:teuthology.orchestra.run.smithi060.stdout: 409994 4 -rw-r--r-- 1 ubuntu ubuntu 2899 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/split.h 2024-03-27T21:42:44.561 INFO:teuthology.orchestra.run.smithi060.stdout: 409889 8 -rw-r--r-- 1 ubuntu ubuntu 4209 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/fs_types.cc 2024-03-27T21:42:44.561 INFO:teuthology.orchestra.run.smithi060.stdout: 409988 4 -rw-r--r-- 1 ubuntu ubuntu 2257 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/signal.cc 2024-03-27T21:42:44.561 INFO:teuthology.orchestra.run.smithi060.stdout: 409720 8 -rw-r--r-- 1 ubuntu ubuntu 5379 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/HeartbeatMap.cc 2024-03-27T21:42:44.561 INFO:teuthology.orchestra.run.smithi060.stdout: 409855 28 -rw-r--r-- 1 ubuntu ubuntu 25455 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/crc32c_intel_fast_zero_asm.s 2024-03-27T21:42:44.561 INFO:teuthology.orchestra.run.smithi060.stdout: 409759 4 -rw-r--r-- 1 ubuntu ubuntu 606 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/ThrottleInterface.h 2024-03-27T21:42:44.561 INFO:teuthology.orchestra.run.smithi060.stdout: 409765 16 -rw-r--r-- 1 ubuntu ubuntu 13095 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/TrackedOp.h 2024-03-27T21:42:44.561 INFO:teuthology.orchestra.run.smithi060.stdout: 409828 4 -rw-r--r-- 1 ubuntu ubuntu 1173 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/code_environment.h 2024-03-27T21:42:44.561 INFO:teuthology.orchestra.run.smithi060.stdout: 409863 8 -rw-r--r-- 1 ubuntu ubuntu 7899 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/deleter.h 2024-03-27T21:42:44.562 INFO:teuthology.orchestra.run.smithi060.stdout: 409736 4 -rw-r--r-- 1 ubuntu ubuntu 1844 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/PluginRegistry.h 2024-03-27T21:42:44.562 INFO:teuthology.orchestra.run.smithi060.stdout: 409866 12 -rw-r--r-- 1 ubuntu ubuntu 9264 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/dns_resolve.cc 2024-03-27T21:42:44.562 INFO:teuthology.orchestra.run.smithi060.stdout: 409722 4 -rw-r--r-- 1 ubuntu ubuntu 4046 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/common/Initialize.h 2024-03-27T21:42:44.562 INFO:teuthology.orchestra.run.smithi060.stdout: 409544 4 drwxr-xr-x 24 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls 2024-03-27T21:42:44.562 INFO:teuthology.orchestra.run.smithi060.stdout: 409609 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/otp 2024-03-27T21:42:44.562 INFO:teuthology.orchestra.run.smithi060.stdout: 409612 4 -rw-r--r-- 1 ubuntu ubuntu 2404 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/otp/cls_otp_client.h 2024-03-27T21:42:44.562 INFO:teuthology.orchestra.run.smithi060.stdout: 409613 4 -rw-r--r-- 1 ubuntu ubuntu 3707 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/otp/cls_otp_ops.h 2024-03-27T21:42:44.562 INFO:teuthology.orchestra.run.smithi060.stdout: 409610 16 -rw-r--r-- 1 ubuntu ubuntu 13535 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/otp/cls_otp.cc 2024-03-27T21:42:44.563 INFO:teuthology.orchestra.run.smithi060.stdout: 409614 4 -rw-r--r-- 1 ubuntu ubuntu 1811 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/otp/cls_otp_types.cc 2024-03-27T21:42:44.563 INFO:teuthology.orchestra.run.smithi060.stdout: 409611 8 -rw-r--r-- 1 ubuntu ubuntu 5178 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/otp/cls_otp_client.cc 2024-03-27T21:42:44.563 INFO:teuthology.orchestra.run.smithi060.stdout: 409615 4 -rw-r--r-- 1 ubuntu ubuntu 3717 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/otp/cls_otp_types.h 2024-03-27T21:42:44.563 INFO:teuthology.orchestra.run.smithi060.stdout: 409656 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/test_remote_reads 2024-03-27T21:42:44.563 INFO:teuthology.orchestra.run.smithi060.stdout: 409657 4 -rw-r--r-- 1 ubuntu ubuntu 2571 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/test_remote_reads/cls_test_remote_reads.cc 2024-03-27T21:42:44.563 INFO:teuthology.orchestra.run.smithi060.stdout: 409560 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/cephfs 2024-03-27T21:42:44.563 INFO:teuthology.orchestra.run.smithi060.stdout: 409563 8 -rw-r--r-- 1 ubuntu ubuntu 5704 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/cephfs/cls_cephfs_client.cc 2024-03-27T21:42:44.563 INFO:teuthology.orchestra.run.smithi060.stdout: 409562 4 -rw-r--r-- 1 ubuntu ubuntu 3249 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/cephfs/cls_cephfs.h 2024-03-27T21:42:44.564 INFO:teuthology.orchestra.run.smithi060.stdout: 409561 8 -rw-r--r-- 1 ubuntu ubuntu 5239 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/cephfs/cls_cephfs.cc 2024-03-27T21:42:44.564 INFO:teuthology.orchestra.run.smithi060.stdout: 409564 4 -rw-r--r-- 1 ubuntu ubuntu 907 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/cephfs/cls_cephfs_client.h 2024-03-27T21:42:44.564 INFO:teuthology.orchestra.run.smithi060.stdout: 409552 12 -rw-r--r-- 1 ubuntu ubuntu 10713 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/CMakeLists.txt 2024-03-27T21:42:44.564 INFO:teuthology.orchestra.run.smithi060.stdout: 409632 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/refcount 2024-03-27T21:42:44.564 INFO:teuthology.orchestra.run.smithi060.stdout: 409635 4 -rw-r--r-- 1 ubuntu ubuntu 2298 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/refcount/cls_refcount_client.h 2024-03-27T21:42:44.564 INFO:teuthology.orchestra.run.smithi060.stdout: 409634 4 -rw-r--r-- 1 ubuntu ubuntu 1498 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/refcount/cls_refcount_client.cc 2024-03-27T21:42:44.564 INFO:teuthology.orchestra.run.smithi060.stdout: 409636 4 -rw-r--r-- 1 ubuntu ubuntu 2675 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/refcount/cls_refcount_ops.cc 2024-03-27T21:42:44.564 INFO:teuthology.orchestra.run.smithi060.stdout: 409633 8 -rw-r--r-- 1 ubuntu ubuntu 4924 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/refcount/cls_refcount.cc 2024-03-27T21:42:44.565 INFO:teuthology.orchestra.run.smithi060.stdout: 409637 4 -rw-r--r-- 1 ubuntu ubuntu 3762 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/refcount/cls_refcount_ops.h 2024-03-27T21:42:44.565 INFO:teuthology.orchestra.run.smithi060.stdout: 409584 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/lock 2024-03-27T21:42:44.565 INFO:teuthology.orchestra.run.smithi060.stdout: 409591 8 -rw-r--r-- 1 ubuntu ubuntu 5432 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/lock/cls_lock_types.h 2024-03-27T21:42:44.565 INFO:teuthology.orchestra.run.smithi060.stdout: 409589 8 -rw-r--r-- 1 ubuntu ubuntu 6144 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/lock/cls_lock_ops.h 2024-03-27T21:42:44.565 INFO:teuthology.orchestra.run.smithi060.stdout: 409588 8 -rw-r--r-- 1 ubuntu ubuntu 5901 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/lock/cls_lock_ops.cc 2024-03-27T21:42:44.565 INFO:teuthology.orchestra.run.smithi060.stdout: 409590 4 -rw-r--r-- 1 ubuntu ubuntu 2554 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/lock/cls_lock_types.cc 2024-03-27T21:42:44.565 INFO:teuthology.orchestra.run.smithi060.stdout: 409587 8 -rw-r--r-- 1 ubuntu ubuntu 5196 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/lock/cls_lock_client.h 2024-03-27T21:42:44.565 INFO:teuthology.orchestra.run.smithi060.stdout: 409586 12 -rw-r--r-- 1 ubuntu ubuntu 8471 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/lock/cls_lock_client.cc 2024-03-27T21:42:44.565 INFO:teuthology.orchestra.run.smithi060.stdout: 409585 16 -rw-r--r-- 1 ubuntu ubuntu 16306 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/lock/cls_lock.cc 2024-03-27T21:42:44.566 INFO:teuthology.orchestra.run.smithi060.stdout: 409605 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/numops 2024-03-27T21:42:44.566 INFO:teuthology.orchestra.run.smithi060.stdout: 409606 4 -rw-r--r-- 1 ubuntu ubuntu 3970 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/numops/cls_numops.cc 2024-03-27T21:42:44.566 INFO:teuthology.orchestra.run.smithi060.stdout: 409608 4 -rw-r--r-- 1 ubuntu ubuntu 1447 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/numops/cls_numops_client.h 2024-03-27T21:42:44.566 INFO:teuthology.orchestra.run.smithi060.stdout: 409607 4 -rw-r--r-- 1 ubuntu ubuntu 1970 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/numops/cls_numops_client.cc 2024-03-27T21:42:44.566 INFO:teuthology.orchestra.run.smithi060.stdout: 409654 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/sdk 2024-03-27T21:42:44.566 INFO:teuthology.orchestra.run.smithi060.stdout: 409655 4 -rw-r--r-- 1 ubuntu ubuntu 2984 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/sdk/cls_sdk.cc 2024-03-27T21:42:44.566 INFO:teuthology.orchestra.run.smithi060.stdout: 409576 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/hello 2024-03-27T21:42:44.566 INFO:teuthology.orchestra.run.smithi060.stdout: 409577 12 -rw-r--r-- 1 ubuntu ubuntu 11371 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/hello/cls_hello.cc 2024-03-27T21:42:44.567 INFO:teuthology.orchestra.run.smithi060.stdout: 409638 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rgw 2024-03-27T21:42:44.567 INFO:teuthology.orchestra.run.smithi060.stdout: 409639 144 -rw-r--r-- 1 ubuntu ubuntu 144322 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rgw/cls_rgw.cc 2024-03-27T21:42:44.567 INFO:teuthology.orchestra.run.smithi060.stdout: 409641 28 -rw-r--r-- 1 ubuntu ubuntu 27311 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rgw/cls_rgw_client.h 2024-03-27T21:42:44.567 INFO:teuthology.orchestra.run.smithi060.stdout: 409643 16 -rw-r--r-- 1 ubuntu ubuntu 15556 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rgw/cls_rgw_ops.cc 2024-03-27T21:42:44.567 INFO:teuthology.orchestra.run.smithi060.stdout: 409645 28 -rw-r--r-- 1 ubuntu ubuntu 25931 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rgw/cls_rgw_types.cc 2024-03-27T21:42:44.567 INFO:teuthology.orchestra.run.smithi060.stdout: 409642 4 -rw-r--r-- 1 ubuntu ubuntu 2787 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rgw/cls_rgw_const.h 2024-03-27T21:42:44.567 INFO:teuthology.orchestra.run.smithi060.stdout: 409644 44 -rw-r--r-- 1 ubuntu ubuntu 43504 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rgw/cls_rgw_ops.h 2024-03-27T21:42:44.567 INFO:teuthology.orchestra.run.smithi060.stdout: 409646 40 -rw-r--r-- 1 ubuntu ubuntu 37386 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rgw/cls_rgw_types.h 2024-03-27T21:42:44.568 INFO:teuthology.orchestra.run.smithi060.stdout: 409640 36 -rw-r--r-- 1 ubuntu ubuntu 35165 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rgw/cls_rgw_client.cc 2024-03-27T21:42:44.568 INFO:teuthology.orchestra.run.smithi060.stdout: 409545 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/2pc_queue 2024-03-27T21:42:44.568 INFO:teuthology.orchestra.run.smithi060.stdout: 409548 8 -rw-r--r-- 1 ubuntu ubuntu 5596 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/2pc_queue/cls_2pc_queue_client.h 2024-03-27T21:42:44.568 INFO:teuthology.orchestra.run.smithi060.stdout: 409549 4 -rw-r--r-- 1 ubuntu ubuntu 594 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/2pc_queue/cls_2pc_queue_const.h 2024-03-27T21:42:44.568 INFO:teuthology.orchestra.run.smithi060.stdout: 409547 8 -rw-r--r-- 1 ubuntu ubuntu 7367 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/2pc_queue/cls_2pc_queue_client.cc 2024-03-27T21:42:44.568 INFO:teuthology.orchestra.run.smithi060.stdout: 409550 8 -rw-r--r-- 1 ubuntu ubuntu 5648 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/2pc_queue/cls_2pc_queue_ops.h 2024-03-27T21:42:44.568 INFO:teuthology.orchestra.run.smithi060.stdout: 409551 4 -rw-r--r-- 1 ubuntu ubuntu 3368 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/2pc_queue/cls_2pc_queue_types.h 2024-03-27T21:42:44.568 INFO:teuthology.orchestra.run.smithi060.stdout: 409546 24 -rw-r--r-- 1 ubuntu ubuntu 24471 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/2pc_queue/cls_2pc_queue.cc 2024-03-27T21:42:44.568 INFO:teuthology.orchestra.run.smithi060.stdout: 409598 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/lua 2024-03-27T21:42:44.569 INFO:teuthology.orchestra.run.smithi060.stdout: 409602 4 -rw-r--r-- 1 ubuntu ubuntu 333 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/lua/cls_lua_client.h 2024-03-27T21:42:44.569 INFO:teuthology.orchestra.run.smithi060.stdout: 409601 4 -rw-r--r-- 1 ubuntu ubuntu 953 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/lua/cls_lua_client.cc 2024-03-27T21:42:44.569 INFO:teuthology.orchestra.run.smithi060.stdout: 409599 28 -rw-r--r-- 1 ubuntu ubuntu 25597 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/lua/cls_lua.cc 2024-03-27T21:42:44.569 INFO:teuthology.orchestra.run.smithi060.stdout: 409604 4 -rw-r--r-- 1 ubuntu ubuntu 3757 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/lua/lua_bufferlist.cc 2024-03-27T21:42:44.569 INFO:teuthology.orchestra.run.smithi060.stdout: 409600 4 -rw-r--r-- 1 ubuntu ubuntu 300 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/lua/cls_lua.h 2024-03-27T21:42:44.569 INFO:teuthology.orchestra.run.smithi060.stdout: 409603 4 -rw-r--r-- 1 ubuntu ubuntu 582 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/lua/cls_lua_ops.h 2024-03-27T21:42:44.569 INFO:teuthology.orchestra.run.smithi060.stdout: 409658 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/timeindex 2024-03-27T21:42:44.569 INFO:teuthology.orchestra.run.smithi060.stdout: 409663 4 -rw-r--r-- 1 ubuntu ubuntu 543 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/timeindex/cls_timeindex_types.cc 2024-03-27T21:42:44.570 INFO:teuthology.orchestra.run.smithi060.stdout: 409661 4 -rw-r--r-- 1 ubuntu ubuntu 2571 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/timeindex/cls_timeindex_client.h 2024-03-27T21:42:44.570 INFO:teuthology.orchestra.run.smithi060.stdout: 409662 4 -rw-r--r-- 1 ubuntu ubuntu 4093 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/timeindex/cls_timeindex_ops.h 2024-03-27T21:42:44.570 INFO:teuthology.orchestra.run.smithi060.stdout: 409659 8 -rw-r--r-- 1 ubuntu ubuntu 6232 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/timeindex/cls_timeindex.cc 2024-03-27T21:42:44.570 INFO:teuthology.orchestra.run.smithi060.stdout: 409660 4 -rw-r--r-- 1 ubuntu ubuntu 2730 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/timeindex/cls_timeindex_client.cc 2024-03-27T21:42:44.570 INFO:teuthology.orchestra.run.smithi060.stdout: 409664 4 -rw-r--r-- 1 ubuntu ubuntu 1176 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/timeindex/cls_timeindex_types.h 2024-03-27T21:42:44.570 INFO:teuthology.orchestra.run.smithi060.stdout: 409665 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/user 2024-03-27T21:42:44.570 INFO:teuthology.orchestra.run.smithi060.stdout: 409669 4 -rw-r--r-- 1 ubuntu ubuntu 3204 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/user/cls_user_ops.cc 2024-03-27T21:42:44.570 INFO:teuthology.orchestra.run.smithi060.stdout: 409668 4 -rw-r--r-- 1 ubuntu ubuntu 1284 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/user/cls_user_client.h 2024-03-27T21:42:44.571 INFO:teuthology.orchestra.run.smithi060.stdout: 409671 4 -rw-r--r-- 1 ubuntu ubuntu 2995 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/user/cls_user_types.cc 2024-03-27T21:42:44.571 INFO:teuthology.orchestra.run.smithi060.stdout: 409672 8 -rw-r--r-- 1 ubuntu ubuntu 6382 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/user/cls_user_types.h 2024-03-27T21:42:44.571 INFO:teuthology.orchestra.run.smithi060.stdout: 409670 8 -rw-r--r-- 1 ubuntu ubuntu 6495 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/user/cls_user_ops.h 2024-03-27T21:42:44.571 INFO:teuthology.orchestra.run.smithi060.stdout: 409667 8 -rw-r--r-- 1 ubuntu ubuntu 4490 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/user/cls_user_client.cc 2024-03-27T21:42:44.571 INFO:teuthology.orchestra.run.smithi060.stdout: 409666 16 -rw-r--r-- 1 ubuntu ubuntu 13953 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/user/cls_user.cc 2024-03-27T21:42:44.571 INFO:teuthology.orchestra.run.smithi060.stdout: 409647 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rgw_gc 2024-03-27T21:42:44.571 INFO:teuthology.orchestra.run.smithi060.stdout: 409649 4 -rw-r--r-- 1 ubuntu ubuntu 2817 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rgw_gc/cls_rgw_gc_client.cc 2024-03-27T21:42:44.571 INFO:teuthology.orchestra.run.smithi060.stdout: 409648 20 -rw-r--r-- 1 ubuntu ubuntu 19778 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rgw_gc/cls_rgw_gc.cc 2024-03-27T21:42:44.572 INFO:teuthology.orchestra.run.smithi060.stdout: 409652 4 -rw-r--r-- 1 ubuntu ubuntu 1975 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rgw_gc/cls_rgw_gc_ops.h 2024-03-27T21:42:44.572 INFO:teuthology.orchestra.run.smithi060.stdout: 409651 4 -rw-r--r-- 1 ubuntu ubuntu 432 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rgw_gc/cls_rgw_gc_const.h 2024-03-27T21:42:44.572 INFO:teuthology.orchestra.run.smithi060.stdout: 409653 4 -rw-r--r-- 1 ubuntu ubuntu 1819 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rgw_gc/cls_rgw_gc_types.h 2024-03-27T21:42:44.572 INFO:teuthology.orchestra.run.smithi060.stdout: 409650 4 -rw-r--r-- 1 ubuntu ubuntu 1093 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rgw_gc/cls_rgw_gc_client.h 2024-03-27T21:42:44.572 INFO:teuthology.orchestra.run.smithi060.stdout: 409616 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/queue 2024-03-27T21:42:44.572 INFO:teuthology.orchestra.run.smithi060.stdout: 409617 4 -rw-r--r-- 1 ubuntu ubuntu 3702 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/queue/cls_queue.cc 2024-03-27T21:42:44.572 INFO:teuthology.orchestra.run.smithi060.stdout: 409621 8 -rw-r--r-- 1 ubuntu ubuntu 6154 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/queue/cls_queue_ops.h 2024-03-27T21:42:44.572 INFO:teuthology.orchestra.run.smithi060.stdout: 409619 4 -rw-r--r-- 1 ubuntu ubuntu 1089 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/queue/cls_queue_client.h 2024-03-27T21:42:44.573 INFO:teuthology.orchestra.run.smithi060.stdout: 409624 8 -rw-r--r-- 1 ubuntu ubuntu 4366 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/queue/cls_queue_types.h 2024-03-27T21:42:44.573 INFO:teuthology.orchestra.run.smithi060.stdout: 409618 4 -rw-r--r-- 1 ubuntu ubuntu 2962 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/queue/cls_queue_client.cc 2024-03-27T21:42:44.573 INFO:teuthology.orchestra.run.smithi060.stdout: 409620 4 -rw-r--r-- 1 ubuntu ubuntu 319 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/queue/cls_queue_const.h 2024-03-27T21:42:44.573 INFO:teuthology.orchestra.run.smithi060.stdout: 409622 20 -rw-r--r-- 1 ubuntu ubuntu 19710 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/queue/cls_queue_src.cc 2024-03-27T21:42:44.573 INFO:teuthology.orchestra.run.smithi060.stdout: 409623 4 -rw-r--r-- 1 ubuntu ubuntu 832 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/queue/cls_queue_src.h 2024-03-27T21:42:44.573 INFO:teuthology.orchestra.run.smithi060.stdout: 409578 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/journal 2024-03-27T21:42:44.573 INFO:teuthology.orchestra.run.smithi060.stdout: 409582 8 -rw-r--r-- 1 ubuntu ubuntu 4976 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/journal/cls_journal_types.cc 2024-03-27T21:42:44.573 INFO:teuthology.orchestra.run.smithi060.stdout: 409581 8 -rw-r--r-- 1 ubuntu ubuntu 4807 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/journal/cls_journal_client.h 2024-03-27T21:42:44.574 INFO:teuthology.orchestra.run.smithi060.stdout: 409580 16 -rw-r--r-- 1 ubuntu ubuntu 14026 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/journal/cls_journal_client.cc 2024-03-27T21:42:44.574 INFO:teuthology.orchestra.run.smithi060.stdout: 409583 8 -rw-r--r-- 1 ubuntu ubuntu 4645 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/journal/cls_journal_types.h 2024-03-27T21:42:44.574 INFO:teuthology.orchestra.run.smithi060.stdout: 409579 36 -rw-r--r-- 1 ubuntu ubuntu 35914 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/journal/cls_journal.cc 2024-03-27T21:42:44.574 INFO:teuthology.orchestra.run.smithi060.stdout: 409572 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/fifo 2024-03-27T21:42:44.574 INFO:teuthology.orchestra.run.smithi060.stdout: 409575 20 -rw-r--r-- 1 ubuntu ubuntu 16656 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/fifo/cls_fifo_types.h 2024-03-27T21:42:44.574 INFO:teuthology.orchestra.run.smithi060.stdout: 409573 24 -rw-r--r-- 1 ubuntu ubuntu 24562 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/fifo/cls_fifo.cc 2024-03-27T21:42:44.574 INFO:teuthology.orchestra.run.smithi060.stdout: 409574 12 -rw-r--r-- 1 ubuntu ubuntu 9127 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/fifo/cls_fifo_ops.h 2024-03-27T21:42:44.574 INFO:teuthology.orchestra.run.smithi060.stdout: 409625 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rbd 2024-03-27T21:42:44.575 INFO:teuthology.orchestra.run.smithi060.stdout: 409627 8 -rw-r--r-- 1 ubuntu ubuntu 7846 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rbd/cls_rbd.h 2024-03-27T21:42:44.575 INFO:teuthology.orchestra.run.smithi060.stdout: 409628 80 -rw-r--r-- 1 ubuntu ubuntu 78107 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rbd/cls_rbd_client.cc 2024-03-27T21:42:44.575 INFO:teuthology.orchestra.run.smithi060.stdout: 409626 232 -rw-r--r-- 1 ubuntu ubuntu 234089 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rbd/cls_rbd.cc 2024-03-27T21:42:44.575 INFO:teuthology.orchestra.run.smithi060.stdout: 409629 40 -rw-r--r-- 1 ubuntu ubuntu 37002 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rbd/cls_rbd_client.h 2024-03-27T21:42:44.575 INFO:teuthology.orchestra.run.smithi060.stdout: 409630 40 -rw-r--r-- 1 ubuntu ubuntu 40161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rbd/cls_rbd_types.cc 2024-03-27T21:42:44.575 INFO:teuthology.orchestra.run.smithi060.stdout: 409631 32 -rw-r--r-- 1 ubuntu ubuntu 31366 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/rbd/cls_rbd_types.h 2024-03-27T21:42:44.575 INFO:teuthology.orchestra.run.smithi060.stdout: 409553 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/cas 2024-03-27T21:42:44.575 INFO:teuthology.orchestra.run.smithi060.stdout: 409554 8 -rw-r--r-- 1 ubuntu ubuntu 5029 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/cas/cls_cas.cc 2024-03-27T21:42:44.575 INFO:teuthology.orchestra.run.smithi060.stdout: 409556 4 -rw-r--r-- 1 ubuntu ubuntu 981 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/cas/cls_cas_client.h 2024-03-27T21:42:44.576 INFO:teuthology.orchestra.run.smithi060.stdout: 409557 4 -rw-r--r-- 1 ubuntu ubuntu 2878 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/cas/cls_cas_internal.cc 2024-03-27T21:42:44.576 INFO:teuthology.orchestra.run.smithi060.stdout: 409555 4 -rw-r--r-- 1 ubuntu ubuntu 1419 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/cas/cls_cas_client.cc 2024-03-27T21:42:44.576 INFO:teuthology.orchestra.run.smithi060.stdout: 409559 4 -rw-r--r-- 1 ubuntu ubuntu 2451 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/cas/cls_cas_ops.h 2024-03-27T21:42:44.576 INFO:teuthology.orchestra.run.smithi060.stdout: 409558 12 -rw-r--r-- 1 ubuntu ubuntu 10261 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/cas/cls_cas_internal.h 2024-03-27T21:42:44.576 INFO:teuthology.orchestra.run.smithi060.stdout: 409565 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/cmpomap 2024-03-27T21:42:44.576 INFO:teuthology.orchestra.run.smithi060.stdout: 409566 4 -rw-r--r-- 1 ubuntu ubuntu 274 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/cmpomap/CMakeLists.txt 2024-03-27T21:42:44.576 INFO:teuthology.orchestra.run.smithi060.stdout: 409571 4 -rw-r--r-- 1 ubuntu ubuntu 1265 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/cmpomap/types.h 2024-03-27T21:42:44.576 INFO:teuthology.orchestra.run.smithi060.stdout: 409568 4 -rw-r--r-- 1 ubuntu ubuntu 2780 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/cmpomap/client.h 2024-03-27T21:42:44.577 INFO:teuthology.orchestra.run.smithi060.stdout: 409567 4 -rw-r--r-- 1 ubuntu ubuntu 1889 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/cmpomap/client.cc 2024-03-27T21:42:44.577 INFO:teuthology.orchestra.run.smithi060.stdout: 409570 12 -rw-r--r-- 1 ubuntu ubuntu 8775 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/cmpomap/server.cc 2024-03-27T21:42:44.577 INFO:teuthology.orchestra.run.smithi060.stdout: 409569 4 -rw-r--r-- 1 ubuntu ubuntu 2204 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/cmpomap/ops.h 2024-03-27T21:42:44.577 INFO:teuthology.orchestra.run.smithi060.stdout: 409592 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/log 2024-03-27T21:42:44.577 INFO:teuthology.orchestra.run.smithi060.stdout: 409595 4 -rw-r--r-- 1 ubuntu ubuntu 1710 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/log/cls_log_client.h 2024-03-27T21:42:44.577 INFO:teuthology.orchestra.run.smithi060.stdout: 409597 4 -rw-r--r-- 1 ubuntu ubuntu 2980 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/log/cls_log_types.h 2024-03-27T21:42:44.577 INFO:teuthology.orchestra.run.smithi060.stdout: 409594 4 -rw-r--r-- 1 ubuntu ubuntu 4094 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/log/cls_log_client.cc 2024-03-27T21:42:44.577 INFO:teuthology.orchestra.run.smithi060.stdout: 409596 8 -rw-r--r-- 1 ubuntu ubuntu 6204 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/log/cls_log_ops.h 2024-03-27T21:42:44.578 INFO:teuthology.orchestra.run.smithi060.stdout: 409593 8 -rw-r--r-- 1 ubuntu ubuntu 7318 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/log/cls_log.cc 2024-03-27T21:42:44.578 INFO:teuthology.orchestra.run.smithi060.stdout: 409673 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/version 2024-03-27T21:42:44.578 INFO:teuthology.orchestra.run.smithi060.stdout: 409678 4 -rw-r--r-- 1 ubuntu ubuntu 342 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/version/cls_version_types.cc 2024-03-27T21:42:44.578 INFO:teuthology.orchestra.run.smithi060.stdout: 409677 4 -rw-r--r-- 1 ubuntu ubuntu 3570 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/version/cls_version_ops.h 2024-03-27T21:42:44.578 INFO:teuthology.orchestra.run.smithi060.stdout: 409674 8 -rw-r--r-- 1 ubuntu ubuntu 5503 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/version/cls_version.cc 2024-03-27T21:42:44.578 INFO:teuthology.orchestra.run.smithi060.stdout: 409676 4 -rw-r--r-- 1 ubuntu ubuntu 1075 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/version/cls_version_client.h 2024-03-27T21:42:44.578 INFO:teuthology.orchestra.run.smithi060.stdout: 409675 4 -rw-r--r-- 1 ubuntu ubuntu 2202 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/version/cls_version_client.cc 2024-03-27T21:42:44.578 INFO:teuthology.orchestra.run.smithi060.stdout: 409679 4 -rw-r--r-- 1 ubuntu ubuntu 2353 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/cls/version/cls_version_types.h 2024-03-27T21:42:44.579 INFO:teuthology.orchestra.run.smithi060.stdout: 409381 8 -rw-r--r-- 1 ubuntu ubuntu 6846 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph_common.sh 2024-03-27T21:42:44.579 INFO:teuthology.orchestra.run.smithi060.stdout: 409073 4 -rw-r--r-- 1 ubuntu ubuntu 2096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/ceph-debugpack.in 2024-03-27T21:42:44.579 INFO:teuthology.orchestra.run.smithi060.stdout: 409042 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/bash_completion 2024-03-27T21:42:44.579 INFO:teuthology.orchestra.run.smithi060.stdout: 409045 4 -rw-r--r-- 1 ubuntu ubuntu 1190 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/bash_completion/rados 2024-03-27T21:42:44.579 INFO:teuthology.orchestra.run.smithi060.stdout: 409043 4 -rw-r--r-- 1 ubuntu ubuntu 244 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/bash_completion/CMakeLists.txt 2024-03-27T21:42:44.579 INFO:teuthology.orchestra.run.smithi060.stdout: 409046 4 -rw-r--r-- 1 ubuntu ubuntu 1825 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/bash_completion/radosgw-admin 2024-03-27T21:42:44.579 INFO:teuthology.orchestra.run.smithi060.stdout: 409044 4 -rw-r--r-- 1 ubuntu ubuntu 1488 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/bash_completion/ceph 2024-03-27T21:42:44.579 INFO:teuthology.orchestra.run.smithi060.stdout: 409047 4 -rw-r--r-- 1 ubuntu ubuntu 2754 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/bash_completion/rbd 2024-03-27T21:42:44.579 INFO:teuthology.orchestra.run.smithi060.stdout: 408987 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/arrow 2024-03-27T21:42:44.580 INFO:teuthology.orchestra.run.smithi060.stdout: 411564 12 drwxr-xr-x 2 ubuntu ubuntu 12288 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages 2024-03-27T21:42:44.580 INFO:teuthology.orchestra.run.smithi060.stdout: 411702 4 -rw-r--r-- 1 ubuntu ubuntu 2197 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGPushReply.h 2024-03-27T21:42:44.580 INFO:teuthology.orchestra.run.smithi060.stdout: 411730 4 -rw-r--r-- 1 ubuntu ubuntu 1034 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MServiceMap.h 2024-03-27T21:42:44.580 INFO:teuthology.orchestra.run.smithi060.stdout: 411620 4 -rw-r--r-- 1 ubuntu ubuntu 2285 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSFragmentNotify.h 2024-03-27T21:42:44.580 INFO:teuthology.orchestra.run.smithi060.stdout: 411618 4 -rw-r--r-- 1 ubuntu ubuntu 1601 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSFindIno.h 2024-03-27T21:42:44.580 INFO:teuthology.orchestra.run.smithi060.stdout: 411671 4 -rw-r--r-- 1 ubuntu ubuntu 1362 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonUsedPendingKeys.h 2024-03-27T21:42:44.580 INFO:teuthology.orchestra.run.smithi060.stdout: 411735 4 -rw-r--r-- 1 ubuntu ubuntu 2728 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MWatchNotify.h 2024-03-27T21:42:44.580 INFO:teuthology.orchestra.run.smithi060.stdout: 411659 4 -rw-r--r-- 1 ubuntu ubuntu 1357 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonHealthChecks.h 2024-03-27T21:42:44.581 INFO:teuthology.orchestra.run.smithi060.stdout: 411581 4 -rw-r--r-- 1 ubuntu ubuntu 2106 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MClientSnap.h 2024-03-27T21:42:44.581 INFO:teuthology.orchestra.run.smithi060.stdout: 411610 4 -rw-r--r-- 1 ubuntu ubuntu 1806 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MHeartbeat.h 2024-03-27T21:42:44.581 INFO:teuthology.orchestra.run.smithi060.stdout: 411644 4 -rw-r--r-- 1 ubuntu ubuntu 1470 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMgrMap.h 2024-03-27T21:42:44.581 INFO:teuthology.orchestra.run.smithi060.stdout: 411712 4 -rw-r--r-- 1 ubuntu ubuntu 3400 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGUpdateLogMissing.h 2024-03-27T21:42:44.581 INFO:teuthology.orchestra.run.smithi060.stdout: 411700 4 -rw-r--r-- 1 ubuntu ubuntu 2426 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGPull.h 2024-03-27T21:42:44.581 INFO:teuthology.orchestra.run.smithi060.stdout: 411578 16 -rw-r--r-- 1 ubuntu ubuntu 12868 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MClientRequest.h 2024-03-27T21:42:44.581 INFO:teuthology.orchestra.run.smithi060.stdout: 411594 4 -rw-r--r-- 1 ubuntu ubuntu 1686 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MExportDirCancel.h 2024-03-27T21:42:44.581 INFO:teuthology.orchestra.run.smithi060.stdout: 411657 4 -rw-r--r-- 1 ubuntu ubuntu 1309 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonGlobalID.h 2024-03-27T21:42:44.581 INFO:teuthology.orchestra.run.smithi060.stdout: 411704 4 -rw-r--r-- 1 ubuntu ubuntu 1761 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGQuery2.h 2024-03-27T21:42:44.582 INFO:teuthology.orchestra.run.smithi060.stdout: 411680 4 -rw-r--r-- 1 ubuntu ubuntu 3391 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDFailure.h 2024-03-27T21:42:44.582 INFO:teuthology.orchestra.run.smithi060.stdout: 411565 4 -rw-r--r-- 1 ubuntu ubuntu 1750 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MAuth.h 2024-03-27T21:42:44.582 INFO:teuthology.orchestra.run.smithi060.stdout: 411615 4 -rw-r--r-- 1 ubuntu ubuntu 1489 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MLogAck.h 2024-03-27T21:42:44.582 INFO:teuthology.orchestra.run.smithi060.stdout: 411736 4 -rw-r--r-- 1 ubuntu ubuntu 1661 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/PaxosServiceMessage.h 2024-03-27T21:42:44.582 INFO:teuthology.orchestra.run.smithi060.stdout: 411639 4 -rw-r--r-- 1 ubuntu ubuntu 1227 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMgrClose.h 2024-03-27T21:42:44.582 INFO:teuthology.orchestra.run.smithi060.stdout: 411672 4 -rw-r--r-- 1 ubuntu ubuntu 1364 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDAlive.h 2024-03-27T21:42:44.582 INFO:teuthology.orchestra.run.smithi060.stdout: 411664 4 -rw-r--r-- 1 ubuntu ubuntu 3080 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonPing.h 2024-03-27T21:42:44.582 INFO:teuthology.orchestra.run.smithi060.stdout: 411641 4 -rw-r--r-- 1 ubuntu ubuntu 1102 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMgrCommandReply.h 2024-03-27T21:42:44.583 INFO:teuthology.orchestra.run.smithi060.stdout: 411656 4 -rw-r--r-- 1 ubuntu ubuntu 1633 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonGetVersionReply.h 2024-03-27T21:42:44.583 INFO:teuthology.orchestra.run.smithi060.stdout: 411682 4 -rw-r--r-- 1 ubuntu ubuntu 3127 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDForceRecovery.h 2024-03-27T21:42:44.583 INFO:teuthology.orchestra.run.smithi060.stdout: 411611 4 -rw-r--r-- 1 ubuntu ubuntu 1787 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MInodeFileCaps.h 2024-03-27T21:42:44.583 INFO:teuthology.orchestra.run.smithi060.stdout: 411663 8 -rw-r--r-- 1 ubuntu ubuntu 4178 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonPaxos.h 2024-03-27T21:42:44.583 INFO:teuthology.orchestra.run.smithi060.stdout: 411601 4 -rw-r--r-- 1 ubuntu ubuntu 1876 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MExportDirPrepAck.h 2024-03-27T21:42:44.583 INFO:teuthology.orchestra.run.smithi060.stdout: 411621 4 -rw-r--r-- 1 ubuntu ubuntu 1778 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSFragmentNotifyAck.h 2024-03-27T21:42:44.583 INFO:teuthology.orchestra.run.smithi060.stdout: 411713 4 -rw-r--r-- 1 ubuntu ubuntu 3067 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGUpdateLogMissingReply.h 2024-03-27T21:42:44.583 INFO:teuthology.orchestra.run.smithi060.stdout: 411693 4 -rw-r--r-- 1 ubuntu ubuntu 1980 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGInfo.h 2024-03-27T21:42:44.584 INFO:teuthology.orchestra.run.smithi060.stdout: 411605 4 -rw-r--r-- 1 ubuntu ubuntu 976 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MGatherCaps.h 2024-03-27T21:42:44.584 INFO:teuthology.orchestra.run.smithi060.stdout: 411631 4 -rw-r--r-- 1 ubuntu ubuntu 1455 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSQuiesceDbListing.h 2024-03-27T21:42:44.584 INFO:teuthology.orchestra.run.smithi060.stdout: 411685 4 -rw-r--r-- 1 ubuntu ubuntu 1600 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDMarkMeDead.h 2024-03-27T21:42:44.584 INFO:teuthology.orchestra.run.smithi060.stdout: 411607 4 -rw-r--r-- 1 ubuntu ubuntu 1219 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MGetConfig.h 2024-03-27T21:42:44.584 INFO:teuthology.orchestra.run.smithi060.stdout: 411619 4 -rw-r--r-- 1 ubuntu ubuntu 1633 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSFindInoReply.h 2024-03-27T21:42:44.584 INFO:teuthology.orchestra.run.smithi060.stdout: 411669 4 -rw-r--r-- 1 ubuntu ubuntu 1450 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonSubscribeAck.h 2024-03-27T21:42:44.584 INFO:teuthology.orchestra.run.smithi060.stdout: 411626 4 -rw-r--r-- 1 ubuntu ubuntu 1723 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSOpenIno.h 2024-03-27T21:42:44.584 INFO:teuthology.orchestra.run.smithi060.stdout: 411577 12 -rw-r--r-- 1 ubuntu ubuntu 11222 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MClientReply.h 2024-03-27T21:42:44.585 INFO:teuthology.orchestra.run.smithi060.stdout: 411580 4 -rw-r--r-- 1 ubuntu ubuntu 3720 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MClientSession.h 2024-03-27T21:42:44.585 INFO:teuthology.orchestra.run.smithi060.stdout: 411582 4 -rw-r--r-- 1 ubuntu ubuntu 1335 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MCommand.h 2024-03-27T21:42:44.585 INFO:teuthology.orchestra.run.smithi060.stdout: 411722 4 -rw-r--r-- 1 ubuntu ubuntu 2746 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MPGStats.h 2024-03-27T21:42:44.585 INFO:teuthology.orchestra.run.smithi060.stdout: 411683 4 -rw-r--r-- 1 ubuntu ubuntu 1419 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDFull.h 2024-03-27T21:42:44.585 INFO:teuthology.orchestra.run.smithi060.stdout: 411726 4 -rw-r--r-- 1 ubuntu ubuntu 2209 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MPoolOpReply.h 2024-03-27T21:42:44.585 INFO:teuthology.orchestra.run.smithi060.stdout: 411568 4 -rw-r--r-- 1 ubuntu ubuntu 3254 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MCacheExpire.h 2024-03-27T21:42:44.585 INFO:teuthology.orchestra.run.smithi060.stdout: 411584 4 -rw-r--r-- 1 ubuntu ubuntu 1167 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MConfig.h 2024-03-27T21:42:44.585 INFO:teuthology.orchestra.run.smithi060.stdout: 411705 4 -rw-r--r-- 1 ubuntu ubuntu 1812 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGReadyToMerge.h 2024-03-27T21:42:44.586 INFO:teuthology.orchestra.run.smithi060.stdout: 411724 4 -rw-r--r-- 1 ubuntu ubuntu 791 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MPing.h 2024-03-27T21:42:44.586 INFO:teuthology.orchestra.run.smithi060.stdout: 411576 8 -rw-r--r-- 1 ubuntu ubuntu 4489 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MClientReconnect.h 2024-03-27T21:42:44.586 INFO:teuthology.orchestra.run.smithi060.stdout: 411593 4 -rw-r--r-- 1 ubuntu ubuntu 1603 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MExportDirAck.h 2024-03-27T21:42:44.586 INFO:teuthology.orchestra.run.smithi060.stdout: 411588 4 -rw-r--r-- 1 ubuntu ubuntu 2849 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MDiscover.h 2024-03-27T21:42:44.586 INFO:teuthology.orchestra.run.smithi060.stdout: 411660 4 -rw-r--r-- 1 ubuntu ubuntu 2725 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonJoin.h 2024-03-27T21:42:44.586 INFO:teuthology.orchestra.run.smithi060.stdout: 411653 4 -rw-r--r-- 1 ubuntu ubuntu 1102 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonGetPurgedSnaps.h 2024-03-27T21:42:44.586 INFO:teuthology.orchestra.run.smithi060.stdout: 411597 4 -rw-r--r-- 1 ubuntu ubuntu 1821 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MExportDirFinish.h 2024-03-27T21:42:44.586 INFO:teuthology.orchestra.run.smithi060.stdout: 411698 4 -rw-r--r-- 1 ubuntu ubuntu 2294 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGNotify.h 2024-03-27T21:42:44.587 INFO:teuthology.orchestra.run.smithi060.stdout: 411632 8 -rw-r--r-- 1 ubuntu ubuntu 4552 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSResolve.h 2024-03-27T21:42:44.587 INFO:teuthology.orchestra.run.smithi060.stdout: 411572 4 -rw-r--r-- 1 ubuntu ubuntu 1446 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MClientMetrics.h 2024-03-27T21:42:44.587 INFO:teuthology.orchestra.run.smithi060.stdout: 411717 4 -rw-r--r-- 1 ubuntu ubuntu 4049 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDRepOpReply.h 2024-03-27T21:42:44.587 INFO:teuthology.orchestra.run.smithi060.stdout: 411640 4 -rw-r--r-- 1 ubuntu ubuntu 988 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMgrCommand.h 2024-03-27T21:42:44.587 INFO:teuthology.orchestra.run.smithi060.stdout: 411733 4 -rw-r--r-- 1 ubuntu ubuntu 2211 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MTimeCheck.h 2024-03-27T21:42:44.587 INFO:teuthology.orchestra.run.smithi060.stdout: 411715 8 -rw-r--r-- 1 ubuntu ubuntu 4653 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPing.h 2024-03-27T21:42:44.587 INFO:teuthology.orchestra.run.smithi060.stdout: 411636 4 -rw-r--r-- 1 ubuntu ubuntu 1724 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSSnapUpdate.h 2024-03-27T21:42:44.587 INFO:teuthology.orchestra.run.smithi060.stdout: 411587 4 -rw-r--r-- 1 ubuntu ubuntu 2972 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MDirUpdate.h 2024-03-27T21:42:44.587 INFO:teuthology.orchestra.run.smithi060.stdout: 411695 4 -rw-r--r-- 1 ubuntu ubuntu 1589 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGLease.h 2024-03-27T21:42:44.588 INFO:teuthology.orchestra.run.smithi060.stdout: 411623 4 -rw-r--r-- 1 ubuntu ubuntu 2557 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSMap.h 2024-03-27T21:42:44.588 INFO:teuthology.orchestra.run.smithi060.stdout: 411711 4 -rw-r--r-- 1 ubuntu ubuntu 2112 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGTrim.h 2024-03-27T21:42:44.588 INFO:teuthology.orchestra.run.smithi060.stdout: 411590 4 -rw-r--r-- 1 ubuntu ubuntu 1783 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MExportCaps.h 2024-03-27T21:42:44.588 INFO:teuthology.orchestra.run.smithi060.stdout: 411589 8 -rw-r--r-- 1 ubuntu ubuntu 6051 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MDiscoverReply.h 2024-03-27T21:42:44.588 INFO:teuthology.orchestra.run.smithi060.stdout: 411692 4 -rw-r--r-- 1 ubuntu ubuntu 981 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGCreated.h 2024-03-27T21:42:44.588 INFO:teuthology.orchestra.run.smithi060.stdout: 411721 4 -rw-r--r-- 1 ubuntu ubuntu 3181 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDScrubReserve.h 2024-03-27T21:42:44.588 INFO:teuthology.orchestra.run.smithi060.stdout: 411573 4 -rw-r--r-- 1 ubuntu ubuntu 1350 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MClientQuota.h 2024-03-27T21:42:44.588 INFO:teuthology.orchestra.run.smithi060.stdout: 411575 4 -rw-r--r-- 1 ubuntu ubuntu 2071 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MClientReclaimReply.h 2024-03-27T21:42:44.589 INFO:teuthology.orchestra.run.smithi060.stdout: 411637 4 -rw-r--r-- 1 ubuntu ubuntu 1933 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSTableRequest.h 2024-03-27T21:42:44.589 INFO:teuthology.orchestra.run.smithi060.stdout: 411731 4 -rw-r--r-- 1 ubuntu ubuntu 1937 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MStatfs.h 2024-03-27T21:42:44.589 INFO:teuthology.orchestra.run.smithi060.stdout: 411574 4 -rw-r--r-- 1 ubuntu ubuntu 1934 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MClientReclaim.h 2024-03-27T21:42:44.589 INFO:teuthology.orchestra.run.smithi060.stdout: 411684 8 -rw-r--r-- 1 ubuntu ubuntu 5746 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDMap.h 2024-03-27T21:42:44.589 INFO:teuthology.orchestra.run.smithi060.stdout: 411720 4 -rw-r--r-- 1 ubuntu ubuntu 1559 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDScrub2.h 2024-03-27T21:42:44.589 INFO:teuthology.orchestra.run.smithi060.stdout: 411613 4 -rw-r--r-- 1 ubuntu ubuntu 3334 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MLock.h 2024-03-27T21:42:44.589 INFO:teuthology.orchestra.run.smithi060.stdout: 411651 4 -rw-r--r-- 1 ubuntu ubuntu 977 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonGetMap.h 2024-03-27T21:42:44.589 INFO:teuthology.orchestra.run.smithi060.stdout: 411609 4 -rw-r--r-- 1 ubuntu ubuntu 1956 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MGetPoolStatsReply.h 2024-03-27T21:42:44.589 INFO:teuthology.orchestra.run.smithi060.stdout: 411608 4 -rw-r--r-- 1 ubuntu ubuntu 1479 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MGetPoolStats.h 2024-03-27T21:42:44.590 INFO:teuthology.orchestra.run.smithi060.stdout: 411654 4 -rw-r--r-- 1 ubuntu ubuntu 1310 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonGetPurgedSnapsReply.h 2024-03-27T21:42:44.590 INFO:teuthology.orchestra.run.smithi060.stdout: 411658 4 -rw-r--r-- 1 ubuntu ubuntu 1532 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonHealth.h 2024-03-27T21:42:44.590 INFO:teuthology.orchestra.run.smithi060.stdout: 411666 4 -rw-r--r-- 1 ubuntu ubuntu 1557 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonQuorumService.h 2024-03-27T21:42:44.590 INFO:teuthology.orchestra.run.smithi060.stdout: 411710 4 -rw-r--r-- 1 ubuntu ubuntu 1737 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGTemp.h 2024-03-27T21:42:44.590 INFO:teuthology.orchestra.run.smithi060.stdout: 411686 4 -rw-r--r-- 1 ubuntu ubuntu 2937 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDMarkMeDown.h 2024-03-27T21:42:44.590 INFO:teuthology.orchestra.run.smithi060.stdout: 411612 4 -rw-r--r-- 1 ubuntu ubuntu 1233 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MKVData.h 2024-03-27T21:42:44.590 INFO:teuthology.orchestra.run.smithi060.stdout: 411647 4 -rw-r--r-- 1 ubuntu ubuntu 2142 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMgrUpdate.h 2024-03-27T21:42:44.590 INFO:teuthology.orchestra.run.smithi060.stdout: 411642 4 -rw-r--r-- 1 ubuntu ubuntu 2512 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMgrConfigure.h 2024-03-27T21:42:44.591 INFO:teuthology.orchestra.run.smithi060.stdout: 411598 4 -rw-r--r-- 1 ubuntu ubuntu 2724 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MExportDirNotify.h 2024-03-27T21:42:44.591 INFO:teuthology.orchestra.run.smithi060.stdout: 411673 4 -rw-r--r-- 1 ubuntu ubuntu 2295 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDBackoff.h 2024-03-27T21:42:44.591 INFO:teuthology.orchestra.run.smithi060.stdout: 411585 4 -rw-r--r-- 1 ubuntu ubuntu 2173 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MDentryLink.h 2024-03-27T21:42:44.591 INFO:teuthology.orchestra.run.smithi060.stdout: 411681 4 -rw-r--r-- 1 ubuntu ubuntu 548 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDFastDispatchOp.h 2024-03-27T21:42:44.591 INFO:teuthology.orchestra.run.smithi060.stdout: 411628 8 -rw-r--r-- 1 ubuntu ubuntu 8015 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSPeerRequest.h 2024-03-27T21:42:44.591 INFO:teuthology.orchestra.run.smithi060.stdout: 411604 8 -rw-r--r-- 1 ubuntu ubuntu 5094 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MForward.h 2024-03-27T21:42:44.591 INFO:teuthology.orchestra.run.smithi060.stdout: 411649 4 -rw-r--r-- 1 ubuntu ubuntu 2425 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonCommandAck.h 2024-03-27T21:42:44.591 INFO:teuthology.orchestra.run.smithi060.stdout: 411571 4 -rw-r--r-- 1 ubuntu ubuntu 2656 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MClientLease.h 2024-03-27T21:42:44.592 INFO:teuthology.orchestra.run.smithi060.stdout: 411688 12 -rw-r--r-- 1 ubuntu ubuntu 10058 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDOpReply.h 2024-03-27T21:42:44.592 INFO:teuthology.orchestra.run.smithi060.stdout: 411602 4 -rw-r--r-- 1 ubuntu ubuntu 1440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MFSMap.h 2024-03-27T21:42:44.592 INFO:teuthology.orchestra.run.smithi060.stdout: 411625 4 -rw-r--r-- 1 ubuntu ubuntu 361 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSOp.h 2024-03-27T21:42:44.592 INFO:teuthology.orchestra.run.smithi060.stdout: 411703 4 -rw-r--r-- 1 ubuntu ubuntu 2012 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGQuery.h 2024-03-27T21:42:44.592 INFO:teuthology.orchestra.run.smithi060.stdout: 411596 4 -rw-r--r-- 1 ubuntu ubuntu 1996 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MExportDirDiscoverAck.h 2024-03-27T21:42:44.592 INFO:teuthology.orchestra.run.smithi060.stdout: 411716 8 -rw-r--r-- 1 ubuntu ubuntu 5187 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDRepOp.h 2024-03-27T21:42:44.592 INFO:teuthology.orchestra.run.smithi060.stdout: 411690 4 -rw-r--r-- 1 ubuntu ubuntu 2051 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGBackfillRemove.h 2024-03-27T21:42:44.592 INFO:teuthology.orchestra.run.smithi060.stdout: 411668 4 -rw-r--r-- 1 ubuntu ubuntu 2812 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonSubscribe.h 2024-03-27T21:42:44.593 INFO:teuthology.orchestra.run.smithi060.stdout: 411567 8 -rw-r--r-- 1 ubuntu ubuntu 5103 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MBackfillReserve.h 2024-03-27T21:42:44.593 INFO:teuthology.orchestra.run.smithi060.stdout: 411699 4 -rw-r--r-- 1 ubuntu ubuntu 1949 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGNotify2.h 2024-03-27T21:42:44.593 INFO:teuthology.orchestra.run.smithi060.stdout: 411570 12 -rw-r--r-- 1 ubuntu ubuntu 10587 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MClientCaps.h 2024-03-27T21:42:44.593 INFO:teuthology.orchestra.run.smithi060.stdout: 411606 4 -rw-r--r-- 1 ubuntu ubuntu 981 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MGenericMessage.h 2024-03-27T21:42:44.593 INFO:teuthology.orchestra.run.smithi060.stdout: 411714 4 -rw-r--r-- 1 ubuntu ubuntu 1080 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPeeringOp.h 2024-03-27T21:42:44.593 INFO:teuthology.orchestra.run.smithi060.stdout: 411679 4 -rw-r--r-- 1 ubuntu ubuntu 2042 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDECSubOpWriteReply.h 2024-03-27T21:42:44.593 INFO:teuthology.orchestra.run.smithi060.stdout: 411734 4 -rw-r--r-- 1 ubuntu ubuntu 2212 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MTimeCheck2.h 2024-03-27T21:42:44.593 INFO:teuthology.orchestra.run.smithi060.stdout: 411670 4 -rw-r--r-- 1 ubuntu ubuntu 3336 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonSync.h 2024-03-27T21:42:44.594 INFO:teuthology.orchestra.run.smithi060.stdout: 411648 4 -rw-r--r-- 1 ubuntu ubuntu 2669 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonCommand.h 2024-03-27T21:42:44.594 INFO:teuthology.orchestra.run.smithi060.stdout: 411676 4 -rw-r--r-- 1 ubuntu ubuntu 2005 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDECSubOpRead.h 2024-03-27T21:42:44.594 INFO:teuthology.orchestra.run.smithi060.stdout: 411732 4 -rw-r--r-- 1 ubuntu ubuntu 1312 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MStatfsReply.h 2024-03-27T21:42:44.594 INFO:teuthology.orchestra.run.smithi060.stdout: 411599 4 -rw-r--r-- 1 ubuntu ubuntu 1891 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MExportDirNotifyAck.h 2024-03-27T21:42:44.594 INFO:teuthology.orchestra.run.smithi060.stdout: 411701 4 -rw-r--r-- 1 ubuntu ubuntu 2503 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGPush.h 2024-03-27T21:42:44.594 INFO:teuthology.orchestra.run.smithi060.stdout: 411586 4 -rw-r--r-- 1 ubuntu ubuntu 1914 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MDentryUnlink.h 2024-03-27T21:42:44.594 INFO:teuthology.orchestra.run.smithi060.stdout: 411707 4 -rw-r--r-- 1 ubuntu ubuntu 1719 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGRecoveryDeleteReply.h 2024-03-27T21:42:44.594 INFO:teuthology.orchestra.run.smithi060.stdout: 411719 4 -rw-r--r-- 1 ubuntu ubuntu 2196 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDRepScrubMap.h 2024-03-27T21:42:44.595 INFO:teuthology.orchestra.run.smithi060.stdout: 411634 4 -rw-r--r-- 1 ubuntu ubuntu 3987 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSScrub.h 2024-03-27T21:42:44.595 INFO:teuthology.orchestra.run.smithi060.stdout: 411652 4 -rw-r--r-- 1 ubuntu ubuntu 2345 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonGetOSDMap.h 2024-03-27T21:42:44.595 INFO:teuthology.orchestra.run.smithi060.stdout: 411616 12 -rw-r--r-- 1 ubuntu ubuntu 10183 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSBeacon.h 2024-03-27T21:42:44.595 INFO:teuthology.orchestra.run.smithi060.stdout: 411622 4 -rw-r--r-- 1 ubuntu ubuntu 1800 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSLoadTargets.h 2024-03-27T21:42:44.595 INFO:teuthology.orchestra.run.smithi060.stdout: 411624 4 -rw-r--r-- 1 ubuntu ubuntu 1533 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSMetrics.h 2024-03-27T21:42:44.595 INFO:teuthology.orchestra.run.smithi060.stdout: 411687 20 -rw-r--r-- 1 ubuntu ubuntu 17329 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDOp.h 2024-03-27T21:42:44.595 INFO:teuthology.orchestra.run.smithi060.stdout: 411635 4 -rw-r--r-- 1 ubuntu ubuntu 2592 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSScrubStats.h 2024-03-27T21:42:44.595 INFO:teuthology.orchestra.run.smithi060.stdout: 411603 4 -rw-r--r-- 1 ubuntu ubuntu 1611 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MFSMapUser.h 2024-03-27T21:42:44.596 INFO:teuthology.orchestra.run.smithi060.stdout: 411566 4 -rw-r--r-- 1 ubuntu ubuntu 1869 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MAuthReply.h 2024-03-27T21:42:44.596 INFO:teuthology.orchestra.run.smithi060.stdout: 411579 4 -rw-r--r-- 1 ubuntu ubuntu 2152 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MClientRequestForward.h 2024-03-27T21:42:44.596 INFO:teuthology.orchestra.run.smithi060.stdout: 411667 4 -rw-r--r-- 1 ubuntu ubuntu 2395 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonScrub.h 2024-03-27T21:42:44.596 INFO:teuthology.orchestra.run.smithi060.stdout: 411678 4 -rw-r--r-- 1 ubuntu ubuntu 2247 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDECSubOpWrite.h 2024-03-27T21:42:44.596 INFO:teuthology.orchestra.run.smithi060.stdout: 411643 4 -rw-r--r-- 1 ubuntu ubuntu 1576 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMgrDigest.h 2024-03-27T21:42:44.596 INFO:teuthology.orchestra.run.smithi060.stdout: 411633 4 -rw-r--r-- 1 ubuntu ubuntu 1777 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSResolveAck.h 2024-03-27T21:42:44.596 INFO:teuthology.orchestra.run.smithi060.stdout: 411694 4 -rw-r--r-- 1 ubuntu ubuntu 2246 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGInfo2.h 2024-03-27T21:42:44.596 INFO:teuthology.orchestra.run.smithi060.stdout: 411662 4 -rw-r--r-- 1 ubuntu ubuntu 2827 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonMgrReport.h 2024-03-27T21:42:44.596 INFO:teuthology.orchestra.run.smithi060.stdout: 411728 4 -rw-r--r-- 1 ubuntu ubuntu 1595 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MRemoveSnaps.h 2024-03-27T21:42:44.596 INFO:teuthology.orchestra.run.smithi060.stdout: 411617 16 -rw-r--r-- 1 ubuntu ubuntu 13395 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSCacheRejoin.h 2024-03-27T21:42:44.597 INFO:teuthology.orchestra.run.smithi060.stdout: 411646 8 -rw-r--r-- 1 ubuntu ubuntu 6534 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMgrReport.h 2024-03-27T21:42:44.597 INFO:teuthology.orchestra.run.smithi060.stdout: 411569 4 -rw-r--r-- 1 ubuntu ubuntu 1913 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MClientCapRelease.h 2024-03-27T21:42:44.597 INFO:teuthology.orchestra.run.smithi060.stdout: 411723 4 -rw-r--r-- 1 ubuntu ubuntu 1277 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MPGStatsAck.h 2024-03-27T21:42:44.597 INFO:teuthology.orchestra.run.smithi060.stdout: 411665 8 -rw-r--r-- 1 ubuntu ubuntu 4154 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonProbe.h 2024-03-27T21:42:44.597 INFO:teuthology.orchestra.run.smithi060.stdout: 411583 4 -rw-r--r-- 1 ubuntu ubuntu 1448 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MCommandReply.h 2024-03-27T21:42:44.597 INFO:teuthology.orchestra.run.smithi060.stdout: 411645 4 -rw-r--r-- 1 ubuntu ubuntu 2620 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMgrOpen.h 2024-03-27T21:42:44.597 INFO:teuthology.orchestra.run.smithi060.stdout: 411696 4 -rw-r--r-- 1 ubuntu ubuntu 1659 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGLeaseAck.h 2024-03-27T21:42:44.597 INFO:teuthology.orchestra.run.smithi060.stdout: 411655 4 -rw-r--r-- 1 ubuntu ubuntu 1500 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonGetVersion.h 2024-03-27T21:42:44.598 INFO:teuthology.orchestra.run.smithi060.stdout: 411661 4 -rw-r--r-- 1 ubuntu ubuntu 1585 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonMap.h 2024-03-27T21:42:44.598 INFO:teuthology.orchestra.run.smithi060.stdout: 411697 4 -rw-r--r-- 1 ubuntu ubuntu 3482 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGLog.h 2024-03-27T21:42:44.598 INFO:teuthology.orchestra.run.smithi060.stdout: 411708 4 -rw-r--r-- 1 ubuntu ubuntu 1780 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGRemove.h 2024-03-27T21:42:44.598 INFO:teuthology.orchestra.run.smithi060.stdout: 411600 4 -rw-r--r-- 1 ubuntu ubuntu 2464 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MExportDirPrep.h 2024-03-27T21:42:44.598 INFO:teuthology.orchestra.run.smithi060.stdout: 411595 4 -rw-r--r-- 1 ubuntu ubuntu 2123 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MExportDirDiscover.h 2024-03-27T21:42:44.598 INFO:teuthology.orchestra.run.smithi060.stdout: 411725 4 -rw-r--r-- 1 ubuntu ubuntu 2484 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MPoolOp.h 2024-03-27T21:42:44.598 INFO:teuthology.orchestra.run.smithi060.stdout: 411729 4 -rw-r--r-- 1 ubuntu ubuntu 2273 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MRoute.h 2024-03-27T21:42:44.599 INFO:teuthology.orchestra.run.smithi060.stdout: 411691 4 -rw-r--r-- 1 ubuntu ubuntu 1418 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGCreate2.h 2024-03-27T21:42:44.599 INFO:teuthology.orchestra.run.smithi060.stdout: 411706 4 -rw-r--r-- 1 ubuntu ubuntu 2226 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGRecoveryDelete.h 2024-03-27T21:42:44.599 INFO:teuthology.orchestra.run.smithi060.stdout: 411650 4 -rw-r--r-- 1 ubuntu ubuntu 4026 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMonElection.h 2024-03-27T21:42:44.599 INFO:teuthology.orchestra.run.smithi060.stdout: 411709 4 -rw-r--r-- 1 ubuntu ubuntu 2953 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGScan.h 2024-03-27T21:42:44.599 INFO:teuthology.orchestra.run.smithi060.stdout: 411689 4 -rw-r--r-- 1 ubuntu ubuntu 2955 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDPGBackfill.h 2024-03-27T21:42:44.599 INFO:teuthology.orchestra.run.smithi060.stdout: 411627 4 -rw-r--r-- 1 ubuntu ubuntu 1886 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSOpenInoReply.h 2024-03-27T21:42:44.599 INFO:teuthology.orchestra.run.smithi060.stdout: 411592 4 -rw-r--r-- 1 ubuntu ubuntu 1733 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MExportDir.h 2024-03-27T21:42:44.599 INFO:teuthology.orchestra.run.smithi060.stdout: 411675 4 -rw-r--r-- 1 ubuntu ubuntu 2929 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDBoot.h 2024-03-27T21:42:44.600 INFO:teuthology.orchestra.run.smithi060.stdout: 411674 4 -rw-r--r-- 1 ubuntu ubuntu 1954 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDBeacon.h 2024-03-27T21:42:44.600 INFO:teuthology.orchestra.run.smithi060.stdout: 411638 8 -rw-r--r-- 1 ubuntu ubuntu 6034 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMgrBeacon.h 2024-03-27T21:42:44.600 INFO:teuthology.orchestra.run.smithi060.stdout: 411727 4 -rw-r--r-- 1 ubuntu ubuntu 3228 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MRecoveryReserve.h 2024-03-27T21:42:44.600 INFO:teuthology.orchestra.run.smithi060.stdout: 411677 4 -rw-r--r-- 1 ubuntu ubuntu 2036 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDECSubOpReadReply.h 2024-03-27T21:42:44.600 INFO:teuthology.orchestra.run.smithi060.stdout: 411591 4 -rw-r--r-- 1 ubuntu ubuntu 1619 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MExportCapsAck.h 2024-03-27T21:42:44.600 INFO:teuthology.orchestra.run.smithi060.stdout: 411629 4 -rw-r--r-- 1 ubuntu ubuntu 1228 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSPing.h 2024-03-27T21:42:44.600 INFO:teuthology.orchestra.run.smithi060.stdout: 411630 4 -rw-r--r-- 1 ubuntu ubuntu 1391 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MMDSQuiesceDbAck.h 2024-03-27T21:42:44.600 INFO:teuthology.orchestra.run.smithi060.stdout: 411718 4 -rw-r--r-- 1 ubuntu ubuntu 3918 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MOSDRepScrub.h 2024-03-27T21:42:44.600 INFO:teuthology.orchestra.run.smithi060.stdout: 411614 4 -rw-r--r-- 1 ubuntu ubuntu 1575 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/src/messages/MLog.h 2024-03-27T21:42:44.601 INFO:teuthology.orchestra.run.smithi060.stdout: 417130 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd 2024-03-27T21:42:44.601 INFO:teuthology.orchestra.run.smithi060.stdout: 417136 4 -rw-r--r-- 1 ubuntu ubuntu 732 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/ceph-fuse@.service.in 2024-03-27T21:42:44.601 INFO:teuthology.orchestra.run.smithi060.stdout: 417132 4 -rw-r--r-- 1 ubuntu ubuntu 1361 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/CMakeLists.txt 2024-03-27T21:42:44.601 INFO:teuthology.orchestra.run.smithi060.stdout: 417148 4 -rw-r--r-- 1 ubuntu ubuntu 890 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/ceph-radosgw@.service.in 2024-03-27T21:42:44.601 INFO:teuthology.orchestra.run.smithi060.stdout: 417150 4 -rw-r--r-- 1 ubuntu ubuntu 808 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/ceph-rbd-mirror@.service.in 2024-03-27T21:42:44.601 INFO:teuthology.orchestra.run.smithi060.stdout: 417138 4 -rw-r--r-- 1 ubuntu ubuntu 779 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/ceph-immutable-object-cache@.service.in 2024-03-27T21:42:44.601 INFO:teuthology.orchestra.run.smithi060.stdout: 417135 4 -rw-r--r-- 1 ubuntu ubuntu 182 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/ceph-fuse.target 2024-03-27T21:42:44.601 INFO:teuthology.orchestra.run.smithi060.stdout: 417151 4 -rw-r--r-- 1 ubuntu ubuntu 302 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/ceph-volume@.service 2024-03-27T21:42:44.602 INFO:teuthology.orchestra.run.smithi060.stdout: 417131 4 -rw-r--r-- 1 ubuntu ubuntu 222 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/50-ceph.preset 2024-03-27T21:42:44.602 INFO:teuthology.orchestra.run.smithi060.stdout: 417147 4 -rw-r--r-- 1 ubuntu ubuntu 242 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/ceph-radosgw.target 2024-03-27T21:42:44.602 INFO:teuthology.orchestra.run.smithi060.stdout: 417142 4 -rw-r--r-- 1 ubuntu ubuntu 1048 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/ceph-mgr@.service.in 2024-03-27T21:42:44.602 INFO:teuthology.orchestra.run.smithi060.stdout: 417141 4 -rw-r--r-- 1 ubuntu ubuntu 238 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/ceph-mgr.target 2024-03-27T21:42:44.602 INFO:teuthology.orchestra.run.smithi060.stdout: 417133 4 -rw-r--r-- 1 ubuntu ubuntu 203 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/ceph-crash.service.in 2024-03-27T21:42:44.602 INFO:teuthology.orchestra.run.smithi060.stdout: 417134 4 -rw-r--r-- 1 ubuntu ubuntu 707 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/ceph-exporter.service.in 2024-03-27T21:42:44.602 INFO:teuthology.orchestra.run.smithi060.stdout: 417155 4 -rw-r--r-- 1 ubuntu ubuntu 770 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/cephfs-mirror@.service.in 2024-03-27T21:42:44.602 INFO:teuthology.orchestra.run.smithi060.stdout: 417146 4 -rw-r--r-- 1 ubuntu ubuntu 1121 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/ceph-osd@.service.in 2024-03-27T21:42:44.603 INFO:teuthology.orchestra.run.smithi060.stdout: 417154 4 -rw-r--r-- 1 ubuntu ubuntu 169 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/cephfs-mirror.target 2024-03-27T21:42:44.603 INFO:teuthology.orchestra.run.smithi060.stdout: 417153 4 -rw-r--r-- 1 ubuntu ubuntu 29 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/ceph.tmpfiles.d 2024-03-27T21:42:44.603 INFO:teuthology.orchestra.run.smithi060.stdout: 417149 4 -rw-r--r-- 1 ubuntu ubuntu 189 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/ceph-rbd-mirror.target 2024-03-27T21:42:44.603 INFO:teuthology.orchestra.run.smithi060.stdout: 417152 4 -rw-r--r-- 1 ubuntu ubuntu 129 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/ceph.target 2024-03-27T21:42:44.603 INFO:teuthology.orchestra.run.smithi060.stdout: 417143 4 -rw-r--r-- 1 ubuntu ubuntu 200 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/ceph-mon.target 2024-03-27T21:42:44.603 INFO:teuthology.orchestra.run.smithi060.stdout: 417156 4 -rw-r--r-- 1 ubuntu ubuntu 418 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/rbdmap.service.in 2024-03-27T21:42:44.603 INFO:teuthology.orchestra.run.smithi060.stdout: 417144 4 -rw-r--r-- 1 ubuntu ubuntu 1305 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/ceph-mon@.service.in 2024-03-27T21:42:44.603 INFO:teuthology.orchestra.run.smithi060.stdout: 417139 4 -rw-r--r-- 1 ubuntu ubuntu 238 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/ceph-mds.target 2024-03-27T21:42:44.603 INFO:teuthology.orchestra.run.smithi060.stdout: 417145 4 -rw-r--r-- 1 ubuntu ubuntu 238 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/ceph-osd.target 2024-03-27T21:42:44.604 INFO:teuthology.orchestra.run.smithi060.stdout: 417140 4 -rw-r--r-- 1 ubuntu ubuntu 912 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/ceph-mds@.service.in 2024-03-27T21:42:44.604 INFO:teuthology.orchestra.run.smithi060.stdout: 417137 4 -rw-r--r-- 1 ubuntu ubuntu 201 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/systemd/ceph-immutable-object-cache.target 2024-03-27T21:42:44.604 INFO:teuthology.orchestra.run.smithi060.stdout: 402901 4 -rw-r--r-- 1 ubuntu ubuntu 4032 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/CodingStyle 2024-03-27T21:42:44.604 INFO:teuthology.orchestra.run.smithi060.stdout: 402907 8 -rw-r--r-- 1 ubuntu ubuntu 5566 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/README.windows.rst 2024-03-27T21:42:44.604 INFO:teuthology.orchestra.run.smithi060.stdout: 404005 8 -rw-r--r-- 1 ubuntu ubuntu 5082 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/mingw_conf.sh 2024-03-27T21:42:44.604 INFO:teuthology.orchestra.run.smithi060.stdout: 402906 4 -rw-r--r-- 1 ubuntu ubuntu 777 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/README.solaris 2024-03-27T21:42:44.604 INFO:teuthology.orchestra.run.smithi060.stdout: 404006 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/mirroring 2024-03-27T21:42:44.604 INFO:teuthology.orchestra.run.smithi060.stdout: 404008 4 -rw-r--r-- 1 ubuntu ubuntu 2161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/mirroring/README.md 2024-03-27T21:42:44.605 INFO:teuthology.orchestra.run.smithi060.stdout: 404010 4 -rwxr-xr-x 1 ubuntu ubuntu 2560 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/mirroring/mirror-ceph.sh 2024-03-27T21:42:44.605 INFO:teuthology.orchestra.run.smithi060.stdout: 404009 4 -rw-r--r-- 1 ubuntu ubuntu 488 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/mirroring/apache2.vhost.conf 2024-03-27T21:42:44.605 INFO:teuthology.orchestra.run.smithi060.stdout: 404011 4 -rw-r--r-- 1 ubuntu ubuntu 164 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/mirroring/rsyncd.conf 2024-03-27T21:42:44.605 INFO:teuthology.orchestra.run.smithi060.stdout: 404007 4 -rw-r--r-- 1 ubuntu ubuntu 581 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/mirroring/MIRRORS 2024-03-27T21:42:44.605 INFO:teuthology.orchestra.run.smithi060.stdout: 404012 4 -rwxr-xr-x 1 ubuntu ubuntu 1327 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/mirroring/test-mirrors.sh 2024-03-27T21:42:44.605 INFO:teuthology.orchestra.run.smithi060.stdout: 403992 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/keys 2024-03-27T21:42:44.605 INFO:teuthology.orchestra.run.smithi060.stdout: 403994 12 -rw-r--r-- 1 ubuntu ubuntu 8868 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/keys/old_release.1.asc 2024-03-27T21:42:44.605 INFO:teuthology.orchestra.run.smithi060.stdout: 403993 4 -rw-r--r-- 1 ubuntu ubuntu 2438 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/keys/autobuild.asc 2024-03-27T21:42:44.605 INFO:teuthology.orchestra.run.smithi060.stdout: 403995 4 -rw-r--r-- 1 ubuntu ubuntu 1752 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/keys/old_release.2.possibly.compromised.asc 2024-03-27T21:42:44.606 INFO:teuthology.orchestra.run.smithi060.stdout: 403996 4 -rw-r--r-- 1 ubuntu ubuntu 1645 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/keys/release.asc 2024-03-27T21:42:44.606 INFO:teuthology.orchestra.run.smithi060.stdout: 402903 8 -rw-r--r-- 1 ubuntu ubuntu 5356 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/README.FreeBSD 2024-03-27T21:42:44.606 INFO:teuthology.orchestra.run.smithi060.stdout: 404115 4 drwxr-xr-x 37 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa 2024-03-27T21:42:44.606 INFO:teuthology.orchestra.run.smithi060.stdout: 404600 4 drwxr-xr-x 30 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites 2024-03-27T21:42:44.606 INFO:teuthology.orchestra.run.smithi060.stdout: 404880 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/experimental 2024-03-27T21:42:44.606 INFO:teuthology.orchestra.run.smithi060.stdout: 404882 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/experimental/multimds 2024-03-27T21:42:44.606 INFO:teuthology.orchestra.run.smithi060.stdout: 404883 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/experimental/multimds/% 2024-03-27T21:42:44.606 INFO:teuthology.orchestra.run.smithi060.stdout: 404884 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/experimental/multimds/.qa -> ../.qa/ 2024-03-27T21:42:44.607 INFO:teuthology.orchestra.run.smithi060.stdout: 404885 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/experimental/multimds/clusters 2024-03-27T21:42:44.607 INFO:teuthology.orchestra.run.smithi060.stdout: 404886 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/experimental/multimds/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.607 INFO:teuthology.orchestra.run.smithi060.stdout: 404887 4 -rw-r--r-- 1 ubuntu ubuntu 143 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/experimental/multimds/clusters/7-multimds.yaml 2024-03-27T21:42:44.607 INFO:teuthology.orchestra.run.smithi060.stdout: 404888 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/experimental/multimds/tasks 2024-03-27T21:42:44.607 INFO:teuthology.orchestra.run.smithi060.stdout: 404889 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/experimental/multimds/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.607 INFO:teuthology.orchestra.run.smithi060.stdout: 404890 4 -rw-r--r-- 1 ubuntu ubuntu 254 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/experimental/multimds/tasks/fsstress_thrash_subtrees.yaml 2024-03-27T21:42:44.607 INFO:teuthology.orchestra.run.smithi060.stdout: 404881 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/experimental/.qa -> ../.qa/ 2024-03-27T21:42:44.607 INFO:teuthology.orchestra.run.smithi060.stdout: 406227 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/netsplit 2024-03-27T21:42:44.608 INFO:teuthology.orchestra.run.smithi060.stdout: 406230 4 -rw-r--r-- 1 ubuntu ubuntu 594 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/netsplit/ceph.yaml 2024-03-27T21:42:44.608 INFO:teuthology.orchestra.run.smithi060.stdout: 406228 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/netsplit/% 2024-03-27T21:42:44.608 INFO:teuthology.orchestra.run.smithi060.stdout: 406235 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/netsplit/tests 2024-03-27T21:42:44.608 INFO:teuthology.orchestra.run.smithi060.stdout: 406236 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/netsplit/tests/.qa -> ../.qa/ 2024-03-27T21:42:44.608 INFO:teuthology.orchestra.run.smithi060.stdout: 406237 4 -rw-r--r-- 1 ubuntu ubuntu 404 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/netsplit/tests/mon_pool_ops.yaml 2024-03-27T21:42:44.608 INFO:teuthology.orchestra.run.smithi060.stdout: 406229 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/netsplit/.qa -> ../.qa/ 2024-03-27T21:42:44.608 INFO:teuthology.orchestra.run.smithi060.stdout: 406231 4 -rw-r--r-- 1 ubuntu ubuntu 278 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/netsplit/cluster.yaml 2024-03-27T21:42:44.608 INFO:teuthology.orchestra.run.smithi060.stdout: 406234 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/netsplit/supported-random-distro$ -> ../rados/basic/supported-random-distro$/ 2024-03-27T21:42:44.609 INFO:teuthology.orchestra.run.smithi060.stdout: 406233 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/netsplit/rados.yaml -> .qa/config/rados.yaml 2024-03-27T21:42:44.609 INFO:teuthology.orchestra.run.smithi060.stdout: 406232 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/netsplit/msgr.yaml -> ../../msgr/async.yaml 2024-03-27T21:42:44.609 INFO:teuthology.orchestra.run.smithi060.stdout: 404891 4 drwxr-xr-x 26 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs 2024-03-27T21:42:44.609 INFO:teuthology.orchestra.run.smithi060.stdout: 405673 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind 2024-03-27T21:42:44.609 INFO:teuthology.orchestra.run.smithi060.stdout: 405674 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/% 2024-03-27T21:42:44.609 INFO:teuthology.orchestra.run.smithi060.stdout: 405675 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/.qa -> ../.qa/ 2024-03-27T21:42:44.609 INFO:teuthology.orchestra.run.smithi060.stdout: 405678 4 -rw-r--r-- 1 ubuntu ubuntu 54 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/debug.yaml 2024-03-27T21:42:44.609 INFO:teuthology.orchestra.run.smithi060.stdout: 405679 4 drwxr-xr-x 8 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/mirror 2024-03-27T21:42:44.609 INFO:teuthology.orchestra.run.smithi060.stdout: 405694 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/mirror/overrides 2024-03-27T21:42:44.610 INFO:teuthology.orchestra.run.smithi060.stdout: 405695 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/mirror/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.610 INFO:teuthology.orchestra.run.smithi060.stdout: 405696 0 lrwxrwxrwx 1 ubuntu ubuntu 45 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/mirror/overrides/ignorelist_health.yaml -> ./.qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.610 INFO:teuthology.orchestra.run.smithi060.stdout: 405680 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/mirror/% 2024-03-27T21:42:44.610 INFO:teuthology.orchestra.run.smithi060.stdout: 405685 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/mirror/clients 2024-03-27T21:42:44.610 INFO:teuthology.orchestra.run.smithi060.stdout: 405686 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/mirror/clients/.qa -> ../.qa/ 2024-03-27T21:42:44.610 INFO:teuthology.orchestra.run.smithi060.stdout: 405687 4 -rw-r--r-- 1 ubuntu ubuntu 669 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/mirror/clients/mirror.yaml 2024-03-27T21:42:44.610 INFO:teuthology.orchestra.run.smithi060.stdout: 405681 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/mirror/.qa -> ../.qa/ 2024-03-27T21:42:44.610 INFO:teuthology.orchestra.run.smithi060.stdout: 405691 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/mirror/mount 2024-03-27T21:42:44.611 INFO:teuthology.orchestra.run.smithi060.stdout: 405692 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/mirror/mount/.qa -> ../.qa/ 2024-03-27T21:42:44.611 INFO:teuthology.orchestra.run.smithi060.stdout: 405693 4 -rw-r--r-- 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/mirror/mount/fuse.yaml 2024-03-27T21:42:44.611 INFO:teuthology.orchestra.run.smithi060.stdout: 405682 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/mirror/cephfs-mirror 2024-03-27T21:42:44.611 INFO:teuthology.orchestra.run.smithi060.stdout: 405683 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/mirror/cephfs-mirror/.qa -> ../.qa/ 2024-03-27T21:42:44.611 INFO:teuthology.orchestra.run.smithi060.stdout: 405684 4 -rw-r--r-- 1 ubuntu ubuntu 186 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/mirror/cephfs-mirror/one-per-cluster.yaml 2024-03-27T21:42:44.611 INFO:teuthology.orchestra.run.smithi060.stdout: 405697 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/mirror/tasks 2024-03-27T21:42:44.611 INFO:teuthology.orchestra.run.smithi060.stdout: 405698 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/mirror/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.611 INFO:teuthology.orchestra.run.smithi060.stdout: 405699 4 -rw-r--r-- 1 ubuntu ubuntu 164 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/mirror/tasks/mirror.yaml 2024-03-27T21:42:44.611 INFO:teuthology.orchestra.run.smithi060.stdout: 405688 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/mirror/cluster 2024-03-27T21:42:44.612 INFO:teuthology.orchestra.run.smithi060.stdout: 405689 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/mirror/cluster/.qa -> ../.qa/ 2024-03-27T21:42:44.612 INFO:teuthology.orchestra.run.smithi060.stdout: 405690 4 -rw-r--r-- 1 ubuntu ubuntu 239 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/mirror/cluster/1-node.yaml 2024-03-27T21:42:44.612 INFO:teuthology.orchestra.run.smithi060.stdout: 405676 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/begin -> .qa/cephfs/begin/ 2024-03-27T21:42:44.612 INFO:teuthology.orchestra.run.smithi060.stdout: 405677 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/valgrind/centos_latest.yaml -> .qa/distros/supported/centos_latest.yaml 2024-03-27T21:42:44.612 INFO:teuthology.orchestra.run.smithi060.stdout: 404892 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/.qa -> ../.qa/ 2024-03-27T21:42:44.612 INFO:teuthology.orchestra.run.smithi060.stdout: 405299 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell 2024-03-27T21:42:44.612 INFO:teuthology.orchestra.run.smithi060.stdout: 405314 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/overrides 2024-03-27T21:42:44.612 INFO:teuthology.orchestra.run.smithi060.stdout: 405315 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/overrides/+ 2024-03-27T21:42:44.613 INFO:teuthology.orchestra.run.smithi060.stdout: 405316 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.613 INFO:teuthology.orchestra.run.smithi060.stdout: 405319 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/overrides/no_client_pidfile.yaml -> .qa/overrides/no_client_pidfile.yaml 2024-03-27T21:42:44.613 INFO:teuthology.orchestra.run.smithi060.stdout: 405320 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/overrides/pg_health.yaml -> .qa/cephfs/overrides/pg_health.yaml 2024-03-27T21:42:44.613 INFO:teuthology.orchestra.run.smithi060.stdout: 405317 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.613 INFO:teuthology.orchestra.run.smithi060.stdout: 405318 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.613 INFO:teuthology.orchestra.run.smithi060.stdout: 405300 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/% 2024-03-27T21:42:44.613 INFO:teuthology.orchestra.run.smithi060.stdout: 405311 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/objectstore 2024-03-27T21:42:44.613 INFO:teuthology.orchestra.run.smithi060.stdout: 405312 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/objectstore/.qa -> ../.qa/ 2024-03-27T21:42:44.614 INFO:teuthology.orchestra.run.smithi060.stdout: 405313 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/objectstore/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.614 INFO:teuthology.orchestra.run.smithi060.stdout: 405301 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/.qa -> ../.qa/ 2024-03-27T21:42:44.614 INFO:teuthology.orchestra.run.smithi060.stdout: 405306 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/conf -> .qa/cephfs/conf 2024-03-27T21:42:44.614 INFO:teuthology.orchestra.run.smithi060.stdout: 405308 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/mount 2024-03-27T21:42:44.614 INFO:teuthology.orchestra.run.smithi060.stdout: 405309 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/mount/.qa -> ../.qa/ 2024-03-27T21:42:44.614 INFO:teuthology.orchestra.run.smithi060.stdout: 405310 0 lrwxrwxrwx 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/mount/fuse.yaml -> .qa/cephfs/mount/fuse.yaml 2024-03-27T21:42:44.614 INFO:teuthology.orchestra.run.smithi060.stdout: 405303 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/clusters 2024-03-27T21:42:44.614 INFO:teuthology.orchestra.run.smithi060.stdout: 405304 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.615 INFO:teuthology.orchestra.run.smithi060.stdout: 405305 0 lrwxrwxrwx 1 ubuntu ubuntu 45 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/clusters/1-mds-1-client-coloc.yaml -> .qa/cephfs/clusters/1-mds-1-client-coloc.yaml 2024-03-27T21:42:44.615 INFO:teuthology.orchestra.run.smithi060.stdout: 405302 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/begin -> .qa/cephfs/begin/ 2024-03-27T21:42:44.615 INFO:teuthology.orchestra.run.smithi060.stdout: 405321 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/tasks 2024-03-27T21:42:44.615 INFO:teuthology.orchestra.run.smithi060.stdout: 405322 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.615 INFO:teuthology.orchestra.run.smithi060.stdout: 405323 4 -rw-r--r-- 1 ubuntu ubuntu 87 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/tasks/cephfs-shell.yaml 2024-03-27T21:42:44.615 INFO:teuthology.orchestra.run.smithi060.stdout: 405307 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/shell/distro -> .qa/distros/supported 2024-03-27T21:42:44.615 INFO:teuthology.orchestra.run.smithi060.stdout: 405024 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional 2024-03-27T21:42:44.615 INFO:teuthology.orchestra.run.smithi060.stdout: 405045 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/subvol_versions 2024-03-27T21:42:44.616 INFO:teuthology.orchestra.run.smithi060.stdout: 405047 4 lrwxrwxrwx 1 ubuntu ubuntu 66 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/subvol_versions/create_subvol_version_v1.yaml -> .qa/cephfs/overrides/subvol_versions/create_subvol_version_v1.yaml 2024-03-27T21:42:44.616 INFO:teuthology.orchestra.run.smithi060.stdout: 405046 0 lrwxrwxrwx 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/subvol_versions/.qa -> ../.qa 2024-03-27T21:42:44.616 INFO:teuthology.orchestra.run.smithi060.stdout: 405048 4 lrwxrwxrwx 1 ubuntu ubuntu 66 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/subvol_versions/create_subvol_version_v2.yaml -> .qa/cephfs/overrides/subvol_versions/create_subvol_version_v2.yaml 2024-03-27T21:42:44.616 INFO:teuthology.orchestra.run.smithi060.stdout: 405038 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/overrides 2024-03-27T21:42:44.616 INFO:teuthology.orchestra.run.smithi060.stdout: 405039 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/overrides/+ 2024-03-27T21:42:44.616 INFO:teuthology.orchestra.run.smithi060.stdout: 405040 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.616 INFO:teuthology.orchestra.run.smithi060.stdout: 405043 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/overrides/no_client_pidfile.yaml -> .qa/overrides/no_client_pidfile.yaml 2024-03-27T21:42:44.616 INFO:teuthology.orchestra.run.smithi060.stdout: 405044 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/overrides/pg_health.yaml -> .qa/cephfs/overrides/pg_health.yaml 2024-03-27T21:42:44.616 INFO:teuthology.orchestra.run.smithi060.stdout: 405041 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.617 INFO:teuthology.orchestra.run.smithi060.stdout: 405042 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.617 INFO:teuthology.orchestra.run.smithi060.stdout: 405025 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/% 2024-03-27T21:42:44.617 INFO:teuthology.orchestra.run.smithi060.stdout: 405034 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/objectstore 2024-03-27T21:42:44.617 INFO:teuthology.orchestra.run.smithi060.stdout: 405037 0 lrwxrwxrwx 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/objectstore/bluestore-ec-root.yaml -> .qa/cephfs/objectstore-ec/bluestore-ec-root.yaml 2024-03-27T21:42:44.617 INFO:teuthology.orchestra.run.smithi060.stdout: 405035 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/objectstore/.qa -> ../.qa/ 2024-03-27T21:42:44.617 INFO:teuthology.orchestra.run.smithi060.stdout: 405036 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/objectstore/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.617 INFO:teuthology.orchestra.run.smithi060.stdout: 405026 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/.qa -> ../.qa/ 2024-03-27T21:42:44.617 INFO:teuthology.orchestra.run.smithi060.stdout: 405031 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/conf -> .qa/cephfs/conf 2024-03-27T21:42:44.618 INFO:teuthology.orchestra.run.smithi060.stdout: 405033 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/mount -> .qa/cephfs/mount/ 2024-03-27T21:42:44.618 INFO:teuthology.orchestra.run.smithi060.stdout: 405028 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/clusters 2024-03-27T21:42:44.618 INFO:teuthology.orchestra.run.smithi060.stdout: 405029 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.618 INFO:teuthology.orchestra.run.smithi060.stdout: 405030 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/clusters/1a3s-mds-4c-client.yaml -> .qa/cephfs/clusters/1a3s-mds-4c-client.yaml 2024-03-27T21:42:44.618 INFO:teuthology.orchestra.run.smithi060.stdout: 405027 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/begin -> .qa/cephfs/begin/ 2024-03-27T21:42:44.618 INFO:teuthology.orchestra.run.smithi060.stdout: 405049 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks 2024-03-27T21:42:44.618 INFO:teuthology.orchestra.run.smithi060.stdout: 405070 4 -rw-r--r-- 1 ubuntu ubuntu 104 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/metrics.yaml 2024-03-27T21:42:44.618 INFO:teuthology.orchestra.run.smithi060.stdout: 405068 4 -rw-r--r-- 1 ubuntu ubuntu 945 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/mds-full.yaml 2024-03-27T21:42:44.619 INFO:teuthology.orchestra.run.smithi060.stdout: 405050 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.619 INFO:teuthology.orchestra.run.smithi060.stdout: 405065 4 -rw-r--r-- 1 ubuntu ubuntu 84 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/fragment.yaml 2024-03-27T21:42:44.619 INFO:teuthology.orchestra.run.smithi060.stdout: 405053 4 -rw-r--r-- 1 ubuntu ubuntu 442 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/alternate-pool.yaml 2024-03-27T21:42:44.619 INFO:teuthology.orchestra.run.smithi060.stdout: 405069 4 -rw-r--r-- 1 ubuntu ubuntu 93 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/mds_creation_retry.yaml 2024-03-27T21:42:44.619 INFO:teuthology.orchestra.run.smithi060.stdout: 405076 4 -rw-r--r-- 1 ubuntu ubuntu 80 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/recovery-fs.yaml 2024-03-27T21:42:44.619 INFO:teuthology.orchestra.run.smithi060.stdout: 405084 4 -rw-r--r-- 1 ubuntu ubuntu 344 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/truncate_delay.yaml 2024-03-27T21:42:44.619 INFO:teuthology.orchestra.run.smithi060.stdout: 405055 4 -rw-r--r-- 1 ubuntu ubuntu 239 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/auto-repair.yaml 2024-03-27T21:42:44.619 INFO:teuthology.orchestra.run.smithi060.stdout: 405072 4 -rw-r--r-- 1 ubuntu ubuntu 88 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/openfiletable.yaml 2024-03-27T21:42:44.620 INFO:teuthology.orchestra.run.smithi060.stdout: 405089 4 -rw-r--r-- 1 ubuntu ubuntu 119 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/xfstests-dev.yaml 2024-03-27T21:42:44.620 INFO:teuthology.orchestra.run.smithi060.stdout: 405077 4 -rw-r--r-- 1 ubuntu ubuntu 432 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/scrub.yaml 2024-03-27T21:42:44.620 INFO:teuthology.orchestra.run.smithi060.stdout: 405058 4 -rw-r--r-- 1 ubuntu ubuntu 552 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/client-limits.yaml 2024-03-27T21:42:44.620 INFO:teuthology.orchestra.run.smithi060.stdout: 405052 4 -rw-r--r-- 1 ubuntu ubuntu 267 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/admin.yaml 2024-03-27T21:42:44.620 INFO:teuthology.orchestra.run.smithi060.stdout: 405080 4 -rw-r--r-- 1 ubuntu ubuntu 647 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/snap_schedule_snapdir.yaml 2024-03-27T21:42:44.620 INFO:teuthology.orchestra.run.smithi060.stdout: 405071 4 -rw-r--r-- 1 ubuntu ubuntu 151 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/multimds_misc.yaml 2024-03-27T21:42:44.620 INFO:teuthology.orchestra.run.smithi060.stdout: 405061 4 -rw-r--r-- 1 ubuntu ubuntu 728 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/damage.yaml 2024-03-27T21:42:44.620 INFO:teuthology.orchestra.run.smithi060.stdout: 405057 4 -rw-r--r-- 1 ubuntu ubuntu 154 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/cap-flush.yaml 2024-03-27T21:42:44.620 INFO:teuthology.orchestra.run.smithi060.stdout: 405075 4 -rw-r--r-- 1 ubuntu ubuntu 80 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/quota.yaml 2024-03-27T21:42:44.621 INFO:teuthology.orchestra.run.smithi060.stdout: 405067 4 -rw-r--r-- 1 ubuntu ubuntu 80 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/mds-flush.yaml 2024-03-27T21:42:44.621 INFO:teuthology.orchestra.run.smithi060.stdout: 405060 4 -rw-r--r-- 1 ubuntu ubuntu 552 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/client-recovery.yaml 2024-03-27T21:42:44.621 INFO:teuthology.orchestra.run.smithi060.stdout: 405054 4 -rw-r--r-- 1 ubuntu ubuntu 78 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/asok_dump_tree.yaml 2024-03-27T21:42:44.621 INFO:teuthology.orchestra.run.smithi060.stdout: 405085 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/workunit 2024-03-27T21:42:44.621 INFO:teuthology.orchestra.run.smithi060.stdout: 405086 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/workunit/.qa -> ../.qa/ 2024-03-27T21:42:44.621 INFO:teuthology.orchestra.run.smithi060.stdout: 405088 4 -rw-r--r-- 1 ubuntu ubuntu 62 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/workunit/quota.yaml 2024-03-27T21:42:44.621 INFO:teuthology.orchestra.run.smithi060.stdout: 405087 4 -rw-r--r-- 1 ubuntu ubuntu 72 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/workunit/dir-max-entries.yaml 2024-03-27T21:42:44.621 INFO:teuthology.orchestra.run.smithi060.stdout: 405074 4 -rw-r--r-- 1 ubuntu ubuntu 168 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/quiesce.yaml 2024-03-27T21:42:44.622 INFO:teuthology.orchestra.run.smithi060.stdout: 405062 4 -rw-r--r-- 1 ubuntu ubuntu 468 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/data-scan.yaml 2024-03-27T21:42:44.622 INFO:teuthology.orchestra.run.smithi060.stdout: 405081 4 -rw-r--r-- 1 ubuntu ubuntu 315 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/snapshots.yaml 2024-03-27T21:42:44.622 INFO:teuthology.orchestra.run.smithi060.stdout: 405079 4 -rw-r--r-- 1 ubuntu ubuntu 492 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/snap-schedule.yaml 2024-03-27T21:42:44.622 INFO:teuthology.orchestra.run.smithi060.stdout: 405073 4 -rw-r--r-- 1 ubuntu ubuntu 84 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/pool-perm.yaml 2024-03-27T21:42:44.622 INFO:teuthology.orchestra.run.smithi060.stdout: 405056 4 -rw-r--r-- 1 ubuntu ubuntu 84 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/backtrace.yaml 2024-03-27T21:42:44.622 INFO:teuthology.orchestra.run.smithi060.stdout: 405082 4 -rw-r--r-- 1 ubuntu ubuntu 81 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/strays.yaml 2024-03-27T21:42:44.622 INFO:teuthology.orchestra.run.smithi060.stdout: 405051 4 -rw-r--r-- 1 ubuntu ubuntu 211 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/acls.yaml 2024-03-27T21:42:44.622 INFO:teuthology.orchestra.run.smithi060.stdout: 405066 4 -rw-r--r-- 1 ubuntu ubuntu 331 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/journal-repair.yaml 2024-03-27T21:42:44.623 INFO:teuthology.orchestra.run.smithi060.stdout: 405078 4 -rw-r--r-- 1 ubuntu ubuntu 197 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/sessionmap.yaml 2024-03-27T21:42:44.623 INFO:teuthology.orchestra.run.smithi060.stdout: 405063 4 -rw-r--r-- 1 ubuntu ubuntu 168 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/exports.yaml 2024-03-27T21:42:44.623 INFO:teuthology.orchestra.run.smithi060.stdout: 405083 4 -rw-r--r-- 1 ubuntu ubuntu 86 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/test_journal_migration.yaml 2024-03-27T21:42:44.623 INFO:teuthology.orchestra.run.smithi060.stdout: 405059 4 -rw-r--r-- 1 ubuntu ubuntu 110 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/client-readahead.yaml 2024-03-27T21:42:44.623 INFO:teuthology.orchestra.run.smithi060.stdout: 405064 4 -rw-r--r-- 1 ubuntu ubuntu 320 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/tasks/forward-scrub.yaml 2024-03-27T21:42:44.623 INFO:teuthology.orchestra.run.smithi060.stdout: 405032 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/functional/distro -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.623 INFO:teuthology.orchestra.run.smithi060.stdout: 405208 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient 2024-03-27T21:42:44.623 INFO:teuthology.orchestra.run.smithi060.stdout: 405225 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/overrides 2024-03-27T21:42:44.623 INFO:teuthology.orchestra.run.smithi060.stdout: 405226 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/overrides/+ 2024-03-27T21:42:44.624 INFO:teuthology.orchestra.run.smithi060.stdout: 405227 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.624 INFO:teuthology.orchestra.run.smithi060.stdout: 405230 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/overrides/pg_health.yaml -> .qa/cephfs/overrides/pg_health.yaml 2024-03-27T21:42:44.624 INFO:teuthology.orchestra.run.smithi060.stdout: 405228 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.624 INFO:teuthology.orchestra.run.smithi060.stdout: 405229 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.624 INFO:teuthology.orchestra.run.smithi060.stdout: 405209 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/% 2024-03-27T21:42:44.624 INFO:teuthology.orchestra.run.smithi060.stdout: 405210 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/.qa -> ../.qa/ 2024-03-27T21:42:44.624 INFO:teuthology.orchestra.run.smithi060.stdout: 405216 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/conf -> .qa/cephfs/conf 2024-03-27T21:42:44.625 INFO:teuthology.orchestra.run.smithi060.stdout: 405220 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/mount 2024-03-27T21:42:44.625 INFO:teuthology.orchestra.run.smithi060.stdout: 405221 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/mount/.qa -> ../.qa/ 2024-03-27T21:42:44.625 INFO:teuthology.orchestra.run.smithi060.stdout: 405223 4 -rw-r--r-- 1 ubuntu ubuntu 102 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/mount/kclient.yaml.disabled 2024-03-27T21:42:44.625 INFO:teuthology.orchestra.run.smithi060.stdout: 405222 0 lrwxrwxrwx 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/mount/fuse.yaml -> .qa/cephfs/mount/fuse.yaml 2024-03-27T21:42:44.625 INFO:teuthology.orchestra.run.smithi060.stdout: 405224 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/objectstore-ec -> .qa/cephfs/objectstore-ec 2024-03-27T21:42:44.625 INFO:teuthology.orchestra.run.smithi060.stdout: 405212 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/clusters 2024-03-27T21:42:44.625 INFO:teuthology.orchestra.run.smithi060.stdout: 405213 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.625 INFO:teuthology.orchestra.run.smithi060.stdout: 405215 0 lrwxrwxrwx 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/clusters/1-mds-3-client.yaml -> .qa/cephfs/clusters/1-mds-3-client.yaml 2024-03-27T21:42:44.625 INFO:teuthology.orchestra.run.smithi060.stdout: 405214 0 lrwxrwxrwx 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/clusters/1-mds-2-client.yaml -> .qa/cephfs/clusters/1-mds-2-client.yaml 2024-03-27T21:42:44.626 INFO:teuthology.orchestra.run.smithi060.stdout: 405217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/distros 2024-03-27T21:42:44.626 INFO:teuthology.orchestra.run.smithi060.stdout: 405218 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/distros/.qa -> ../.qa/ 2024-03-27T21:42:44.626 INFO:teuthology.orchestra.run.smithi060.stdout: 405219 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/distros/ubuntu_latest.yaml -> .qa/distros/supported/ubuntu_latest.yaml 2024-03-27T21:42:44.626 INFO:teuthology.orchestra.run.smithi060.stdout: 405211 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/begin -> .qa/cephfs/begin/ 2024-03-27T21:42:44.626 INFO:teuthology.orchestra.run.smithi060.stdout: 405231 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/tasks 2024-03-27T21:42:44.626 INFO:teuthology.orchestra.run.smithi060.stdout: 405233 4 -rw-r--r-- 1 ubuntu ubuntu 322 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/tasks/cephfs_misc_tests.yaml 2024-03-27T21:42:44.626 INFO:teuthology.orchestra.run.smithi060.stdout: 405232 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.626 INFO:teuthology.orchestra.run.smithi060.stdout: 405236 4 -rw-r--r-- 1 ubuntu ubuntu 1094 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/tasks/mdtest.yaml 2024-03-27T21:42:44.627 INFO:teuthology.orchestra.run.smithi060.stdout: 405235 4 -rw-r--r-- 1 ubuntu ubuntu 993 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/tasks/ior-shared-file.yaml 2024-03-27T21:42:44.627 INFO:teuthology.orchestra.run.smithi060.stdout: 405234 4 -rw-r--r-- 1 ubuntu ubuntu 460 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multiclient/tasks/fsx-mpi.yaml.disabled 2024-03-27T21:42:44.627 INFO:teuthology.orchestra.run.smithi060.stdout: 405468 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless 2024-03-27T21:42:44.627 INFO:teuthology.orchestra.run.smithi060.stdout: 405481 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/overrides 2024-03-27T21:42:44.627 INFO:teuthology.orchestra.run.smithi060.stdout: 405482 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/overrides/+ 2024-03-27T21:42:44.627 INFO:teuthology.orchestra.run.smithi060.stdout: 405483 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.627 INFO:teuthology.orchestra.run.smithi060.stdout: 405487 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/overrides/pg_health.yaml -> .qa/cephfs/overrides/pg_health.yaml 2024-03-27T21:42:44.627 INFO:teuthology.orchestra.run.smithi060.stdout: 405485 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.628 INFO:teuthology.orchestra.run.smithi060.stdout: 405484 0 lrwxrwxrwx 1 ubuntu ubuntu 30 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/overrides/frag.yaml -> .qa/cephfs/overrides/frag.yaml 2024-03-27T21:42:44.628 INFO:teuthology.orchestra.run.smithi060.stdout: 405486 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.628 INFO:teuthology.orchestra.run.smithi060.stdout: 405469 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/% 2024-03-27T21:42:44.628 INFO:teuthology.orchestra.run.smithi060.stdout: 405470 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/.qa -> ../.qa/ 2024-03-27T21:42:44.628 INFO:teuthology.orchestra.run.smithi060.stdout: 405475 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/conf -> .qa/cephfs/conf 2024-03-27T21:42:44.628 INFO:teuthology.orchestra.run.smithi060.stdout: 405494 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/traceless 2024-03-27T21:42:44.628 INFO:teuthology.orchestra.run.smithi060.stdout: 405495 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/traceless/.qa -> ../.qa/ 2024-03-27T21:42:44.628 INFO:teuthology.orchestra.run.smithi060.stdout: 405496 4 -rw-r--r-- 1 ubuntu ubuntu 91 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/traceless/50pc.yaml 2024-03-27T21:42:44.629 INFO:teuthology.orchestra.run.smithi060.stdout: 405477 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/mount 2024-03-27T21:42:44.629 INFO:teuthology.orchestra.run.smithi060.stdout: 405478 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/mount/.qa -> ../.qa/ 2024-03-27T21:42:44.629 INFO:teuthology.orchestra.run.smithi060.stdout: 405479 0 lrwxrwxrwx 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/mount/fuse.yaml -> .qa/cephfs/mount/fuse.yaml 2024-03-27T21:42:44.629 INFO:teuthology.orchestra.run.smithi060.stdout: 405480 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/objectstore-ec -> .qa/cephfs/objectstore-ec 2024-03-27T21:42:44.629 INFO:teuthology.orchestra.run.smithi060.stdout: 405472 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/clusters 2024-03-27T21:42:44.629 INFO:teuthology.orchestra.run.smithi060.stdout: 405473 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.629 INFO:teuthology.orchestra.run.smithi060.stdout: 405474 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/clusters/fixed-2-ucephfs.yaml -> .qa/cephfs/clusters/fixed-2-ucephfs.yaml 2024-03-27T21:42:44.629 INFO:teuthology.orchestra.run.smithi060.stdout: 405471 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/begin -> .qa/cephfs/begin/ 2024-03-27T21:42:44.629 INFO:teuthology.orchestra.run.smithi060.stdout: 405488 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/tasks 2024-03-27T21:42:44.630 INFO:teuthology.orchestra.run.smithi060.stdout: 405490 0 lrwxrwxrwx 1 ubuntu ubuntu 53 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/tasks/cfuse_workunit_suites_blogbench.yaml -> .qa/cephfs/tasks/cfuse_workunit_suites_blogbench.yaml 2024-03-27T21:42:44.630 INFO:teuthology.orchestra.run.smithi060.stdout: 405489 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.630 INFO:teuthology.orchestra.run.smithi060.stdout: 405493 0 lrwxrwxrwx 1 ubuntu ubuntu 52 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/tasks/cfuse_workunit_suites_fsstress.yaml -> .qa/cephfs/tasks/cfuse_workunit_suites_fsstress.yaml 2024-03-27T21:42:44.630 INFO:teuthology.orchestra.run.smithi060.stdout: 405491 0 lrwxrwxrwx 1 ubuntu ubuntu 50 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/tasks/cfuse_workunit_suites_dbench.yaml -> .qa/cephfs/tasks/cfuse_workunit_suites_dbench.yaml 2024-03-27T21:42:44.630 INFO:teuthology.orchestra.run.smithi060.stdout: 405492 0 lrwxrwxrwx 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/tasks/cfuse_workunit_suites_ffsb.yaml -> .qa/cephfs/tasks/cfuse_workunit_suites_ffsb.yaml 2024-03-27T21:42:44.630 INFO:teuthology.orchestra.run.smithi060.stdout: 405476 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/traceless/distro -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.630 INFO:teuthology.orchestra.run.smithi060.stdout: 405324 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps 2024-03-27T21:42:44.631 INFO:teuthology.orchestra.run.smithi060.stdout: 405335 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps/overrides 2024-03-27T21:42:44.631 INFO:teuthology.orchestra.run.smithi060.stdout: 405336 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps/overrides/+ 2024-03-27T21:42:44.631 INFO:teuthology.orchestra.run.smithi060.stdout: 405337 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.631 INFO:teuthology.orchestra.run.smithi060.stdout: 405340 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps/overrides/pg_health.yaml -> .qa/cephfs/overrides/pg_health.yaml 2024-03-27T21:42:44.631 INFO:teuthology.orchestra.run.smithi060.stdout: 405338 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.631 INFO:teuthology.orchestra.run.smithi060.stdout: 405339 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.631 INFO:teuthology.orchestra.run.smithi060.stdout: 405325 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps/% 2024-03-27T21:42:44.631 INFO:teuthology.orchestra.run.smithi060.stdout: 405326 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps/.qa -> ../.qa/ 2024-03-27T21:42:44.632 INFO:teuthology.orchestra.run.smithi060.stdout: 405331 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps/conf -> .qa/cephfs/conf 2024-03-27T21:42:44.632 INFO:teuthology.orchestra.run.smithi060.stdout: 405333 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps/mount -> .qa/cephfs/mount/ 2024-03-27T21:42:44.632 INFO:teuthology.orchestra.run.smithi060.stdout: 405334 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps/objectstore-ec -> .qa/cephfs/objectstore-ec 2024-03-27T21:42:44.632 INFO:teuthology.orchestra.run.smithi060.stdout: 405328 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps/clusters 2024-03-27T21:42:44.632 INFO:teuthology.orchestra.run.smithi060.stdout: 405329 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.632 INFO:teuthology.orchestra.run.smithi060.stdout: 405330 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps/clusters/1a3s-mds-1c-client.yaml -> .qa/cephfs/clusters/1a3s-mds-1c-client.yaml 2024-03-27T21:42:44.632 INFO:teuthology.orchestra.run.smithi060.stdout: 405327 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps/begin -> .qa/cephfs/begin/ 2024-03-27T21:42:44.632 INFO:teuthology.orchestra.run.smithi060.stdout: 405341 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps/tasks 2024-03-27T21:42:44.633 INFO:teuthology.orchestra.run.smithi060.stdout: 405342 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.633 INFO:teuthology.orchestra.run.smithi060.stdout: 405343 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps/tasks/workunit 2024-03-27T21:42:44.633 INFO:teuthology.orchestra.run.smithi060.stdout: 405344 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps/tasks/workunit/.qa -> ../.qa/ 2024-03-27T21:42:44.633 INFO:teuthology.orchestra.run.smithi060.stdout: 405345 4 -rw-r--r-- 1 ubuntu ubuntu 78 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps/tasks/workunit/snaps.yaml 2024-03-27T21:42:44.633 INFO:teuthology.orchestra.run.smithi060.stdout: 405332 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/snaps/distro -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.633 INFO:teuthology.orchestra.run.smithi060.stdout: 405700 4 drwxr-xr-x 9 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify 2024-03-27T21:42:44.633 INFO:teuthology.orchestra.run.smithi060.stdout: 405727 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/overrides 2024-03-27T21:42:44.633 INFO:teuthology.orchestra.run.smithi060.stdout: 405728 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/overrides/+ 2024-03-27T21:42:44.633 INFO:teuthology.orchestra.run.smithi060.stdout: 405729 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: 405734 0 lrwxrwxrwx 1 ubuntu ubuntu 41 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/overrides/session_timeout.yaml -> .qa/cephfs/overrides/session_timeout.yaml 2024-03-27T21:42:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: 405733 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/overrides/pg_health.yaml -> .qa/cephfs/overrides/pg_health.yaml 2024-03-27T21:42:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: 405730 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: 405732 4 -rw-r--r-- 1 ubuntu ubuntu 82 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/overrides/mon-debug.yaml 2024-03-27T21:42:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: 405731 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: 405701 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/% 2024-03-27T21:42:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: 405702 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/.qa -> ../.qa/ 2024-03-27T21:42:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: 405707 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/conf -> .qa/cephfs/conf 2024-03-27T21:42:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: 405717 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/mount 2024-03-27T21:42:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: 405718 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/mount/.qa -> ../.qa/ 2024-03-27T21:42:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: 405719 0 lrwxrwxrwx 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/mount/fuse.yaml -> .qa/cephfs/mount/fuse.yaml 2024-03-27T21:42:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: 405720 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/mount/kclient 2024-03-27T21:42:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: 405721 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/mount/kclient/+ 2024-03-27T21:42:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: 405723 4 lrwxrwxrwx 1 ubuntu ubuntu 64 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/mount/kclient/k-testing.yaml -> .qa/cephfs/mount/kclient/overrides/distro/testing/k-testing.yaml 2024-03-27T21:42:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: 405722 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/mount/kclient/.qa -> ../.qa/ 2024-03-27T21:42:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: 405725 0 lrwxrwxrwx 1 ubuntu ubuntu 57 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/mount/kclient/ms-die-on-skipped.yaml -> .qa/cephfs/mount/kclient/overrides/ms-die-on-skipped.yaml 2024-03-27T21:42:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: 405724 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/mount/kclient/mount.yaml -> .qa/cephfs/mount/kclient/mount.yaml 2024-03-27T21:42:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: 405735 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/ranks 2024-03-27T21:42:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: 405736 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/ranks/.qa -> ../.qa/ 2024-03-27T21:42:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: 405737 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/ranks/1.yaml 2024-03-27T21:42:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: 405739 4 -rw-r--r-- 1 ubuntu ubuntu 136 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/ranks/5.yaml 2024-03-27T21:42:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: 405738 4 -rw-r--r-- 1 ubuntu ubuntu 136 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/ranks/3.yaml 2024-03-27T21:42:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: 405726 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/objectstore-ec -> .qa/cephfs/objectstore-ec 2024-03-27T21:42:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: 405704 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/clusters 2024-03-27T21:42:44.637 INFO:teuthology.orchestra.run.smithi060.stdout: 405706 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/clusters/1a5s-mds-1c-client.yaml -> .qa/cephfs/clusters/1a5s-mds-1c-client.yaml 2024-03-27T21:42:44.637 INFO:teuthology.orchestra.run.smithi060.stdout: 405705 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.637 INFO:teuthology.orchestra.run.smithi060.stdout: 405703 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/begin -> .qa/cephfs/begin/ 2024-03-27T21:42:44.637 INFO:teuthology.orchestra.run.smithi060.stdout: 405744 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/validater 2024-03-27T21:42:44.637 INFO:teuthology.orchestra.run.smithi060.stdout: 405745 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/validater/.qa -> ../.qa/ 2024-03-27T21:42:44.637 INFO:teuthology.orchestra.run.smithi060.stdout: 405746 4 -rw-r--r-- 1 ubuntu ubuntu 65 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/validater/lockdep.yaml 2024-03-27T21:42:44.637 INFO:teuthology.orchestra.run.smithi060.stdout: 405747 4 -rw-r--r-- 1 ubuntu ubuntu 739 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/validater/valgrind.yaml 2024-03-27T21:42:44.638 INFO:teuthology.orchestra.run.smithi060.stdout: 405740 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/tasks 2024-03-27T21:42:44.638 INFO:teuthology.orchestra.run.smithi060.stdout: 405741 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.638 INFO:teuthology.orchestra.run.smithi060.stdout: 405743 0 lrwxrwxrwx 1 ubuntu ubuntu 52 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/tasks/fsstress.yaml -> .qa/cephfs/tasks/cfuse_workunit_suites_fsstress.yaml 2024-03-27T21:42:44.638 INFO:teuthology.orchestra.run.smithi060.stdout: 405742 0 lrwxrwxrwx 1 ubuntu ubuntu 50 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/tasks/dbench.yaml -> .qa/cephfs/tasks/cfuse_workunit_suites_dbench.yaml 2024-03-27T21:42:44.638 INFO:teuthology.orchestra.run.smithi060.stdout: 405708 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/distro 2024-03-27T21:42:44.638 INFO:teuthology.orchestra.run.smithi060.stdout: 405710 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/distro/.qa -> ../.qa/ 2024-03-27T21:42:44.638 INFO:teuthology.orchestra.run.smithi060.stdout: 405712 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/distro/ubuntu 2024-03-27T21:42:44.638 INFO:teuthology.orchestra.run.smithi060.stdout: 405713 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/distro/ubuntu/+ 2024-03-27T21:42:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: 405714 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/distro/ubuntu/.qa -> ../.qa/ 2024-03-27T21:42:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: 405716 0 lrwxrwxrwx 1 ubuntu ubuntu 34 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/distro/ubuntu/ubuntu_latest.yaml -> .qa/distros/all/ubuntu_latest.yaml 2024-03-27T21:42:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: 405715 4 -rw-r--r-- 1 ubuntu ubuntu 66 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/distro/ubuntu/overrides.yaml 2024-03-27T21:42:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: 405711 0 lrwxrwxrwx 1 ubuntu ubuntu 34 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/distro/centos_latest.yaml -> .qa/distros/all/centos_latest.yaml 2024-03-27T21:42:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: 405709 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/verify/distro/$ 2024-03-27T21:42:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: 405497 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade 2024-03-27T21:42:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: 405499 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client 2024-03-27T21:42:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: 405500 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/.qa -> ../.qa/ 2024-03-27T21:42:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: 405530 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client 2024-03-27T21:42:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: 405539 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/overrides 2024-03-27T21:42:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: 405544 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/overrides/multimds 2024-03-27T21:42:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: 405545 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/overrides/multimds/.qa -> ../.qa/ 2024-03-27T21:42:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: 405547 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/overrides/multimds/yes.yaml 2024-03-27T21:42:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: 405546 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/overrides/multimds/no.yaml 2024-03-27T21:42:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: 405540 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/overrides/% 2024-03-27T21:42:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: 405541 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: 405542 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: 405543 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: 405548 4 -rw-r--r-- 1 ubuntu ubuntu 78 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/overrides/pg-warn.yaml 2024-03-27T21:42:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: 405531 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/% 2024-03-27T21:42:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: 405532 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/.qa -> ../.qa/ 2024-03-27T21:42:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: 405538 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/conf -> .qa/cephfs/conf/ 2024-03-27T21:42:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: 405533 0 lrwxrwxrwx 1 ubuntu ubuntu 58 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/bluestore-bitmap.yaml -> ../../../../../cephfs/objectstore-ec/bluestore-bitmap.yaml 2024-03-27T21:42:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: 405535 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/clusters 2024-03-27T21:42:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: 405536 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.642 INFO:teuthology.orchestra.run.smithi060.stdout: 405537 0 lrwxrwxrwx 1 ubuntu ubuntu 45 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/clusters/1-mds-2-client-micro.yaml -> .qa/cephfs/clusters/1-mds-2-client-micro.yaml 2024-03-27T21:42:44.642 INFO:teuthology.orchestra.run.smithi060.stdout: 405534 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/centos_8.stream.yaml -> .qa/distros/all/centos_8.stream.yaml 2024-03-27T21:42:44.642 INFO:teuthology.orchestra.run.smithi060.stdout: 405549 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/tasks 2024-03-27T21:42:44.642 INFO:teuthology.orchestra.run.smithi060.stdout: 405550 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/tasks/% 2024-03-27T21:42:44.642 INFO:teuthology.orchestra.run.smithi060.stdout: 405556 4 -rw-r--r-- 1 ubuntu ubuntu 405 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/tasks/4-compat_client.yaml 2024-03-27T21:42:44.642 INFO:teuthology.orchestra.run.smithi060.stdout: 405551 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.642 INFO:teuthology.orchestra.run.smithi060.stdout: 405554 4 -rw-r--r-- 1 ubuntu ubuntu 1244 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/tasks/2-upgrade.yaml 2024-03-27T21:42:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: 405552 4 -rw-r--r-- 1 ubuntu ubuntu 841 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/tasks/0-octopus.yaml 2024-03-27T21:42:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: 405557 4 -rw-r--r-- 1 ubuntu ubuntu 105 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/tasks/5-client-sanity.yaml 2024-03-27T21:42:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: 405555 4 -rw-r--r-- 1 ubuntu ubuntu 283 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/tasks/3-client-upgrade.yaml 2024-03-27T21:42:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: 405553 4 -rw-r--r-- 1 ubuntu ubuntu 208 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/upgraded_client/tasks/1-client.yaml 2024-03-27T21:42:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: 405501 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client 2024-03-27T21:42:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: 405510 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/overrides 2024-03-27T21:42:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: 405515 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/overrides/multimds 2024-03-27T21:42:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: 405516 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/overrides/multimds/.qa -> ../.qa/ 2024-03-27T21:42:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: 405518 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/overrides/multimds/yes.yaml 2024-03-27T21:42:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: 405517 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/overrides/multimds/no.yaml 2024-03-27T21:42:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: 405511 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/overrides/% 2024-03-27T21:42:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: 405512 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: 405513 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: 405514 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: 405519 4 -rw-r--r-- 1 ubuntu ubuntu 78 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/overrides/pg-warn.yaml 2024-03-27T21:42:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: 405502 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/% 2024-03-27T21:42:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: 405503 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/.qa -> ../.qa/ 2024-03-27T21:42:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: 405509 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/conf -> .qa/cephfs/conf/ 2024-03-27T21:42:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: 405504 0 lrwxrwxrwx 1 ubuntu ubuntu 58 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/bluestore-bitmap.yaml -> ../../../../../cephfs/objectstore-ec/bluestore-bitmap.yaml 2024-03-27T21:42:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: 405506 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/clusters 2024-03-27T21:42:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: 405507 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: 405508 0 lrwxrwxrwx 1 ubuntu ubuntu 45 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/clusters/1-mds-2-client-micro.yaml -> .qa/cephfs/clusters/1-mds-2-client-micro.yaml 2024-03-27T21:42:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: 405505 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/centos_8.stream.yaml -> .qa/distros/all/centos_8.stream.yaml 2024-03-27T21:42:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: 405520 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/tasks 2024-03-27T21:42:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: 405521 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/tasks/% 2024-03-27T21:42:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: 405522 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: 405525 4 -rw-r--r-- 1 ubuntu ubuntu 1244 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/tasks/2-upgrade.yaml 2024-03-27T21:42:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: 405526 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/tasks/3-compat_client 2024-03-27T21:42:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: 405527 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/tasks/3-compat_client/.qa -> ../.qa/ 2024-03-27T21:42:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: 405529 4 -rw-r--r-- 1 ubuntu ubuntu 255 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/tasks/3-compat_client/quincy.yaml 2024-03-27T21:42:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: 405528 4 -rw-r--r-- 1 ubuntu ubuntu 100 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/tasks/3-compat_client/no.yaml 2024-03-27T21:42:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: 405523 4 -rw-r--r-- 1 ubuntu ubuntu 841 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/tasks/0-octopus.yaml 2024-03-27T21:42:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: 405524 4 -rw-r--r-- 1 ubuntu ubuntu 149 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/featureful_client/old_client/tasks/1-client.yaml 2024-03-27T21:42:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: 405498 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/.qa -> ../.qa/ 2024-03-27T21:42:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: 405632 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client 2024-03-27T21:42:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: 405636 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/branch 2024-03-27T21:42:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: 405638 4 -rw-r--r-- 1 ubuntu ubuntu 30 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/branch/pacific.yaml 2024-03-27T21:42:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: 405637 4 -rw-r--r-- 1 ubuntu ubuntu 31 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/branch/nautilus.yaml 2024-03-27T21:42:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: 405645 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/overrides 2024-03-27T21:42:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: 405646 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/overrides/% 2024-03-27T21:42:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: 405647 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: 405648 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.648 INFO:teuthology.orchestra.run.smithi060.stdout: 405649 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.648 INFO:teuthology.orchestra.run.smithi060.stdout: 405650 4 -rw-r--r-- 1 ubuntu ubuntu 78 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/overrides/pg-warn.yaml 2024-03-27T21:42:44.648 INFO:teuthology.orchestra.run.smithi060.stdout: 405633 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/% 2024-03-27T21:42:44.648 INFO:teuthology.orchestra.run.smithi060.stdout: 405634 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/.qa -> ../.qa/ 2024-03-27T21:42:44.648 INFO:teuthology.orchestra.run.smithi060.stdout: 405644 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/conf -> .qa/cephfs/conf/ 2024-03-27T21:42:44.648 INFO:teuthology.orchestra.run.smithi060.stdout: 405635 0 lrwxrwxrwx 1 ubuntu ubuntu 55 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/bluestore-bitmap.yaml -> ../../../../cephfs/objectstore-ec/bluestore-bitmap.yaml 2024-03-27T21:42:44.648 INFO:teuthology.orchestra.run.smithi060.stdout: 405640 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/clusters 2024-03-27T21:42:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: 405641 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/clusters/% 2024-03-27T21:42:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: 405643 0 lrwxrwxrwx 1 ubuntu ubuntu 45 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/clusters/1-mds-1-client-micro.yaml -> .qa/cephfs/clusters/1-mds-1-client-micro.yaml 2024-03-27T21:42:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: 405642 0 lrwxrwxrwx 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/clusters/.qa -> ../.qa 2024-03-27T21:42:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: 405639 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/centos_8.stream.yaml -> .qa/distros/all/centos_8.stream.yaml 2024-03-27T21:42:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: 405651 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks 2024-03-27T21:42:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: 405661 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks/3-workload 2024-03-27T21:42:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: 405666 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks/3-workload/stress_tests 2024-03-27T21:42:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: 405672 4 lrwxrwxrwx 1 ubuntu ubuntu 63 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks/3-workload/stress_tests/kernel_untar_build.yaml -> .qa/suites/fs/workload/tasks/5-workunit/kernel_untar_build.yaml 2024-03-27T21:42:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: 405667 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks/3-workload/stress_tests/.qa -> ../.qa/ 2024-03-27T21:42:44.650 INFO:teuthology.orchestra.run.smithi060.stdout: 405671 0 lrwxrwxrwx 1 ubuntu ubuntu 58 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks/3-workload/stress_tests/iozone.yaml -> .qa/suites/fs/workload/tasks/5-workunit/suites/iozone.yaml 2024-03-27T21:42:44.650 INFO:teuthology.orchestra.run.smithi060.stdout: 405668 4 lrwxrwxrwx 1 ubuntu ubuntu 61 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks/3-workload/stress_tests/blogbench.yaml -> .qa/suites/fs/workload/tasks/5-workunit/suites/blogbench.yaml 2024-03-27T21:42:44.650 INFO:teuthology.orchestra.run.smithi060.stdout: 405670 4 lrwxrwxrwx 1 ubuntu ubuntu 60 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks/3-workload/stress_tests/fsstress.yaml -> .qa/suites/fs/workload/tasks/5-workunit/suites/fsstress.yaml 2024-03-27T21:42:44.650 INFO:teuthology.orchestra.run.smithi060.stdout: 405669 0 lrwxrwxrwx 1 ubuntu ubuntu 58 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks/3-workload/stress_tests/dbench.yaml -> .qa/suites/fs/workload/tasks/5-workunit/suites/dbench.yaml 2024-03-27T21:42:44.650 INFO:teuthology.orchestra.run.smithi060.stdout: 405662 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks/3-workload/.qa -> ../.qa/ 2024-03-27T21:42:44.650 INFO:teuthology.orchestra.run.smithi060.stdout: 405663 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks/3-workload/new_ops 2024-03-27T21:42:44.650 INFO:teuthology.orchestra.run.smithi060.stdout: 405664 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks/3-workload/new_ops/% 2024-03-27T21:42:44.650 INFO:teuthology.orchestra.run.smithi060.stdout: 405665 4 -rw-r--r-- 1 ubuntu ubuntu 81 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks/3-workload/new_ops/newops.yaml 2024-03-27T21:42:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: 405652 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks/% 2024-03-27T21:42:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: 405653 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: 405655 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks/1-mount 2024-03-27T21:42:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: 405656 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks/1-mount/.qa -> ../.qa/ 2024-03-27T21:42:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: 405657 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks/1-mount/mount -> .qa/cephfs/mount/ 2024-03-27T21:42:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: 405654 4 -rw-r--r-- 1 ubuntu ubuntu 1489 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks/0-install.yaml 2024-03-27T21:42:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: 405658 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks/2-clients 2024-03-27T21:42:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: 405660 4 -rw-r--r-- 1 ubuntu ubuntu 359 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks/2-clients/kclient.yaml 2024-03-27T21:42:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: 405659 4 -rw-r--r-- 1 ubuntu ubuntu 769 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/upgraded_client/tasks/2-clients/fuse-upgrade.yaml 2024-03-27T21:42:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: 405613 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/nofs 2024-03-27T21:42:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: 405621 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/nofs/overrides 2024-03-27T21:42:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: 405622 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/nofs/overrides/% 2024-03-27T21:42:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: 405623 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/nofs/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: 405624 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/nofs/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: 405625 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/nofs/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: 405626 4 -rw-r--r-- 1 ubuntu ubuntu 78 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/nofs/overrides/pg-warn.yaml 2024-03-27T21:42:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: 405614 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/nofs/% 2024-03-27T21:42:44.653 INFO:teuthology.orchestra.run.smithi060.stdout: 405615 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/nofs/.qa -> ../.qa/ 2024-03-27T21:42:44.653 INFO:teuthology.orchestra.run.smithi060.stdout: 405619 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/nofs/conf -> .qa/cephfs/conf/ 2024-03-27T21:42:44.653 INFO:teuthology.orchestra.run.smithi060.stdout: 405616 4 -rw-r--r-- 1 ubuntu ubuntu 158 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/nofs/README 2024-03-27T21:42:44.653 INFO:teuthology.orchestra.run.smithi060.stdout: 405617 0 lrwxrwxrwx 1 ubuntu ubuntu 47 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/nofs/bluestore-bitmap.yaml -> .qa/cephfs/objectstore-ec/bluestore-bitmap.yaml 2024-03-27T21:42:44.653 INFO:teuthology.orchestra.run.smithi060.stdout: 405618 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/nofs/centos_8.stream.yaml -> .qa/distros/all/centos_8.stream.yaml 2024-03-27T21:42:44.653 INFO:teuthology.orchestra.run.smithi060.stdout: 405620 4 -rw-r--r-- 1 ubuntu ubuntu 154 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/nofs/no-mds-cluster.yaml 2024-03-27T21:42:44.653 INFO:teuthology.orchestra.run.smithi060.stdout: 405627 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/nofs/tasks 2024-03-27T21:42:44.653 INFO:teuthology.orchestra.run.smithi060.stdout: 405628 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/nofs/tasks/% 2024-03-27T21:42:44.654 INFO:teuthology.orchestra.run.smithi060.stdout: 405629 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/nofs/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.654 INFO:teuthology.orchestra.run.smithi060.stdout: 405631 4 -rw-r--r-- 1 ubuntu ubuntu 1068 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/nofs/tasks/1-upgrade.yaml 2024-03-27T21:42:44.654 INFO:teuthology.orchestra.run.smithi060.stdout: 405630 4 -rw-r--r-- 1 ubuntu ubuntu 841 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/nofs/tasks/0-pacific.yaml 2024-03-27T21:42:44.654 INFO:teuthology.orchestra.run.smithi060.stdout: 405558 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence 2024-03-27T21:42:44.654 INFO:teuthology.orchestra.run.smithi060.stdout: 405567 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/overrides 2024-03-27T21:42:44.654 INFO:teuthology.orchestra.run.smithi060.stdout: 405571 4 -rw-r--r-- 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/overrides/ignorelist_upgrade.yaml 2024-03-27T21:42:44.654 INFO:teuthology.orchestra.run.smithi060.stdout: 405568 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/overrides/% 2024-03-27T21:42:44.654 INFO:teuthology.orchestra.run.smithi060.stdout: 405569 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.655 INFO:teuthology.orchestra.run.smithi060.stdout: 405574 4 -rw-r--r-- 1 ubuntu ubuntu 41 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/overrides/syntax.yaml 2024-03-27T21:42:44.655 INFO:teuthology.orchestra.run.smithi060.stdout: 405570 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.655 INFO:teuthology.orchestra.run.smithi060.stdout: 405572 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.655 INFO:teuthology.orchestra.run.smithi060.stdout: 405573 4 -rw-r--r-- 1 ubuntu ubuntu 78 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/overrides/pg-warn.yaml 2024-03-27T21:42:44.655 INFO:teuthology.orchestra.run.smithi060.stdout: 405559 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/% 2024-03-27T21:42:44.655 INFO:teuthology.orchestra.run.smithi060.stdout: 405560 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/.qa -> ../.qa/ 2024-03-27T21:42:44.655 INFO:teuthology.orchestra.run.smithi060.stdout: 405563 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/conf -> .qa/cephfs/conf/ 2024-03-27T21:42:44.655 INFO:teuthology.orchestra.run.smithi060.stdout: 405561 0 lrwxrwxrwx 1 ubuntu ubuntu 47 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/bluestore-bitmap.yaml -> .qa/cephfs/objectstore-ec/bluestore-bitmap.yaml 2024-03-27T21:42:44.656 INFO:teuthology.orchestra.run.smithi060.stdout: 405562 0 lrwxrwxrwx 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/centos_9.stream.yaml -> .qa/distros/podman/centos_9.stream.yaml 2024-03-27T21:42:44.656 INFO:teuthology.orchestra.run.smithi060.stdout: 405564 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/fail_fs 2024-03-27T21:42:44.656 INFO:teuthology.orchestra.run.smithi060.stdout: 405566 4 -rw-r--r-- 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/fail_fs/yes.yaml 2024-03-27T21:42:44.656 INFO:teuthology.orchestra.run.smithi060.stdout: 405565 4 -rw-r--r-- 1 ubuntu ubuntu 44 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/fail_fs/no.yaml 2024-03-27T21:42:44.656 INFO:teuthology.orchestra.run.smithi060.stdout: 405575 4 -rw-r--r-- 1 ubuntu ubuntu 115 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/roles.yaml 2024-03-27T21:42:44.656 INFO:teuthology.orchestra.run.smithi060.stdout: 405576 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks 2024-03-27T21:42:44.656 INFO:teuthology.orchestra.run.smithi060.stdout: 405577 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/% 2024-03-27T21:42:44.656 INFO:teuthology.orchestra.run.smithi060.stdout: 405578 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.657 INFO:teuthology.orchestra.run.smithi060.stdout: 405587 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/1-volume 2024-03-27T21:42:44.657 INFO:teuthology.orchestra.run.smithi060.stdout: 405588 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/1-volume/% 2024-03-27T21:42:44.657 INFO:teuthology.orchestra.run.smithi060.stdout: 405591 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/1-volume/1-ranks 2024-03-27T21:42:44.657 INFO:teuthology.orchestra.run.smithi060.stdout: 405592 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/1-volume/1-ranks/.qa -> ../.qa/ 2024-03-27T21:42:44.657 INFO:teuthology.orchestra.run.smithi060.stdout: 405593 4 -rw-r--r-- 1 ubuntu ubuntu 73 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/1-volume/1-ranks/1.yaml 2024-03-27T21:42:44.657 INFO:teuthology.orchestra.run.smithi060.stdout: 405594 4 -rw-r--r-- 1 ubuntu ubuntu 73 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/1-volume/1-ranks/2.yaml 2024-03-27T21:42:44.657 INFO:teuthology.orchestra.run.smithi060.stdout: 405589 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/1-volume/.qa -> ../.qa/ 2024-03-27T21:42:44.657 INFO:teuthology.orchestra.run.smithi060.stdout: 405590 4 -rw-r--r-- 1 ubuntu ubuntu 108 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/1-volume/0-create.yaml 2024-03-27T21:42:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: 405595 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/1-volume/2-allow_standby_replay 2024-03-27T21:42:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: 405596 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/1-volume/2-allow_standby_replay/.qa -> ../.qa/ 2024-03-27T21:42:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: 405598 4 -rw-r--r-- 1 ubuntu ubuntu 89 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/1-volume/2-allow_standby_replay/yes.yaml 2024-03-27T21:42:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: 405597 4 -rw-r--r-- 1 ubuntu ubuntu 90 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/1-volume/2-allow_standby_replay/no.yaml 2024-03-27T21:42:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: 405599 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/1-volume/3-inline 2024-03-27T21:42:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: 405600 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/1-volume/3-inline/.qa -> ../.qa/ 2024-03-27T21:42:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: 405602 4 -rw-r--r-- 1 ubuntu ubuntu 110 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/1-volume/3-inline/yes.yaml 2024-03-27T21:42:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: 405601 4 -rw-r--r-- 1 ubuntu ubuntu 81 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/1-volume/3-inline/no.yaml 2024-03-27T21:42:44.659 INFO:teuthology.orchestra.run.smithi060.stdout: 405603 4 -rw-r--r-- 1 ubuntu ubuntu 241 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/1-volume/4-verify.yaml 2024-03-27T21:42:44.659 INFO:teuthology.orchestra.run.smithi060.stdout: 405608 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/4-config-upgrade 2024-03-27T21:42:44.659 INFO:teuthology.orchestra.run.smithi060.stdout: 405609 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/4-config-upgrade/+ 2024-03-27T21:42:44.659 INFO:teuthology.orchestra.run.smithi060.stdout: 405610 4 -rw-r--r-- 1 ubuntu ubuntu 609 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/4-config-upgrade/fail_fs.yaml 2024-03-27T21:42:44.659 INFO:teuthology.orchestra.run.smithi060.stdout: 405604 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/2-client 2024-03-27T21:42:44.659 INFO:teuthology.orchestra.run.smithi060.stdout: 405606 4 -rw-r--r-- 1 ubuntu ubuntu 46 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/2-client/kclient.yaml 2024-03-27T21:42:44.659 INFO:teuthology.orchestra.run.smithi060.stdout: 405605 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/2-client/fuse.yaml 2024-03-27T21:42:44.659 INFO:teuthology.orchestra.run.smithi060.stdout: 405612 4 -rw-r--r-- 1 ubuntu ubuntu 83 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/6-verify.yaml 2024-03-27T21:42:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: 405611 4 -rw-r--r-- 1 ubuntu ubuntu 1268 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/5-upgrade-with-workload.yaml 2024-03-27T21:42:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: 405607 4 -rw-r--r-- 1 ubuntu ubuntu 1038 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml 2024-03-27T21:42:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: 405579 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/0-from 2024-03-27T21:42:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: 405580 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/0-from/.qa -> ../.qa/ 2024-03-27T21:42:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: 405581 4 -rw-r--r-- 1 ubuntu ubuntu 837 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/0-from/quincy.yaml 2024-03-27T21:42:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: 405582 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/0-from/reef 2024-03-27T21:42:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: 405586 4 -rw-r--r-- 1 ubuntu ubuntu 787 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/0-from/reef/v18.2.1.yaml 2024-03-27T21:42:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: 405584 4 -rw-r--r-- 1 ubuntu ubuntu 786 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/0-from/reef/reef.yaml 2024-03-27T21:42:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: 405585 4 -rw-r--r-- 1 ubuntu ubuntu 787 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/0-from/reef/v18.2.0.yaml 2024-03-27T21:42:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: 405583 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/0-from/reef/$ 2024-03-27T21:42:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: 405276 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission 2024-03-27T21:42:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: 405289 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/overrides 2024-03-27T21:42:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: 405290 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/overrides/+ 2024-03-27T21:42:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: 405291 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: 405294 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/overrides/pg_health.yaml -> .qa/cephfs/overrides/pg_health.yaml 2024-03-27T21:42:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: 405292 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: 405293 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.662 INFO:teuthology.orchestra.run.smithi060.stdout: 405277 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/% 2024-03-27T21:42:44.662 INFO:teuthology.orchestra.run.smithi060.stdout: 405278 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/.qa -> ../.qa/ 2024-03-27T21:42:44.662 INFO:teuthology.orchestra.run.smithi060.stdout: 405283 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/conf -> .qa/cephfs/conf 2024-03-27T21:42:44.662 INFO:teuthology.orchestra.run.smithi060.stdout: 405285 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/mount 2024-03-27T21:42:44.662 INFO:teuthology.orchestra.run.smithi060.stdout: 405286 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/mount/.qa -> ../.qa/ 2024-03-27T21:42:44.662 INFO:teuthology.orchestra.run.smithi060.stdout: 405287 0 lrwxrwxrwx 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/mount/fuse.yaml -> .qa/cephfs/mount/fuse.yaml 2024-03-27T21:42:44.662 INFO:teuthology.orchestra.run.smithi060.stdout: 405288 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/objectstore-ec -> .qa/cephfs/objectstore-ec 2024-03-27T21:42:44.662 INFO:teuthology.orchestra.run.smithi060.stdout: 405280 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/clusters 2024-03-27T21:42:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: 405281 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: 405282 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/clusters/fixed-2-ucephfs.yaml -> .qa/cephfs/clusters/fixed-2-ucephfs.yaml 2024-03-27T21:42:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: 405279 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/begin -> .qa/cephfs/begin/ 2024-03-27T21:42:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: 405295 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/tasks 2024-03-27T21:42:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: 405296 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: 405298 4 -rw-r--r-- 1 ubuntu ubuntu 196 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/tasks/cfuse_workunit_suites_pjd.yaml 2024-03-27T21:42:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: 405297 4 -rw-r--r-- 1 ubuntu ubuntu 207 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/tasks/cfuse_workunit_misc.yaml 2024-03-27T21:42:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: 405284 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/permission/distro -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: 405148 4 drwxr-xr-x 9 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror 2024-03-27T21:42:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: 405169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/overrides 2024-03-27T21:42:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: 405170 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/overrides/+ 2024-03-27T21:42:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: 405171 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: 405173 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/overrides/pg_health.yaml -> .qa/cephfs/overrides/pg_health.yaml 2024-03-27T21:42:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: 405172 0 lrwxrwxrwx 1 ubuntu ubuntu 45 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/overrides/ignorelist_health.yaml -> ./.qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: 405149 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/% 2024-03-27T21:42:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: 405155 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/clients 2024-03-27T21:42:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: 405156 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/clients/+ 2024-03-27T21:42:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: 405157 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/clients/.qa -> ../.qa/ 2024-03-27T21:42:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: 405158 4 -rw-r--r-- 1 ubuntu ubuntu 669 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/clients/mirror.yaml 2024-03-27T21:42:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: 405166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/objectstore 2024-03-27T21:42:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: 405167 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/objectstore/.qa -> ../.qa/ 2024-03-27T21:42:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: 405168 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/objectstore/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: 405150 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/.qa -> ../.qa/ 2024-03-27T21:42:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: 405163 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/mount 2024-03-27T21:42:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: 405164 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/mount/.qa -> ../.qa/ 2024-03-27T21:42:44.666 INFO:teuthology.orchestra.run.smithi060.stdout: 405165 4 -rw-r--r-- 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/mount/fuse.yaml 2024-03-27T21:42:44.666 INFO:teuthology.orchestra.run.smithi060.stdout: 405152 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/cephfs-mirror 2024-03-27T21:42:44.666 INFO:teuthology.orchestra.run.smithi060.stdout: 405153 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/cephfs-mirror/.qa -> ../.qa/ 2024-03-27T21:42:44.666 INFO:teuthology.orchestra.run.smithi060.stdout: 405154 4 -rw-r--r-- 1 ubuntu ubuntu 112 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/cephfs-mirror/one-per-cluster.yaml 2024-03-27T21:42:44.666 INFO:teuthology.orchestra.run.smithi060.stdout: 405151 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/begin -> .qa/cephfs/begin/ 2024-03-27T21:42:44.666 INFO:teuthology.orchestra.run.smithi060.stdout: 405174 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/supported-random-distros$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.666 INFO:teuthology.orchestra.run.smithi060.stdout: 405175 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/tasks 2024-03-27T21:42:44.666 INFO:teuthology.orchestra.run.smithi060.stdout: 405176 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: 405177 4 -rw-r--r-- 1 ubuntu ubuntu 164 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/tasks/mirror.yaml 2024-03-27T21:42:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: 405159 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/cluster 2024-03-27T21:42:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: 405160 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/cluster/+ 2024-03-27T21:42:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: 405161 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/cluster/.qa -> ../.qa/ 2024-03-27T21:42:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: 405162 4 -rw-r--r-- 1 ubuntu ubuntu 238 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror/cluster/1-node.yaml 2024-03-27T21:42:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: 405445 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top 2024-03-27T21:42:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: 405459 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/overrides 2024-03-27T21:42:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: 405460 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/overrides/+ 2024-03-27T21:42:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: 405461 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: 405463 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/overrides/pg_health.yaml -> .qa/cephfs/overrides/pg_health.yaml 2024-03-27T21:42:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: 405462 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: 405446 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/% 2024-03-27T21:42:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: 405456 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/objectstore 2024-03-27T21:42:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: 405457 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/objectstore/.qa -> ../.qa/ 2024-03-27T21:42:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: 405458 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/objectstore/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: 405447 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/.qa -> ../.qa/ 2024-03-27T21:42:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: 405453 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/mount 2024-03-27T21:42:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: 405454 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/mount/.qa -> ../.qa/ 2024-03-27T21:42:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: 405455 0 lrwxrwxrwx 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/mount/fuse.yaml -> .qa/cephfs/mount/fuse.yaml 2024-03-27T21:42:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: 405448 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/begin -> .qa/cephfs/begin/ 2024-03-27T21:42:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: 405464 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/supported-random-distros$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: 405465 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/tasks 2024-03-27T21:42:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: 405466 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: 405467 4 -rw-r--r-- 1 ubuntu ubuntu 80 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/tasks/fstop.yaml 2024-03-27T21:42:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: 405449 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/cluster 2024-03-27T21:42:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: 405450 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/cluster/+ 2024-03-27T21:42:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: 405451 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/cluster/.qa -> ../.qa/ 2024-03-27T21:42:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: 405452 4 -rw-r--r-- 1 ubuntu ubuntu 176 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/top/cluster/1-node.yaml 2024-03-27T21:42:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: 404998 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full 2024-03-27T21:42:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: 405014 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/overrides 2024-03-27T21:42:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: 405015 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/overrides/+ 2024-03-27T21:42:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: 405016 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: 405019 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/overrides/no_client_pidfile.yaml -> .qa/overrides/no_client_pidfile.yaml 2024-03-27T21:42:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: 405020 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/overrides/pg_health.yaml -> .qa/cephfs/overrides/pg_health.yaml 2024-03-27T21:42:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: 405017 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: 405018 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: 404999 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/% 2024-03-27T21:42:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: 405010 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/objectstore 2024-03-27T21:42:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: 405011 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/objectstore/.qa -> ../.qa/ 2024-03-27T21:42:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: 405012 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/objectstore/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: 405000 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/.qa -> ../.qa/ 2024-03-27T21:42:44.672 INFO:teuthology.orchestra.run.smithi060.stdout: 405005 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/conf -> .qa/cephfs/conf 2024-03-27T21:42:44.672 INFO:teuthology.orchestra.run.smithi060.stdout: 405007 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/mount 2024-03-27T21:42:44.672 INFO:teuthology.orchestra.run.smithi060.stdout: 405008 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/mount/.qa -> ../.qa/ 2024-03-27T21:42:44.672 INFO:teuthology.orchestra.run.smithi060.stdout: 405009 4 -rw-r--r-- 1 ubuntu ubuntu 22 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/mount/fuse.yaml 2024-03-27T21:42:44.672 INFO:teuthology.orchestra.run.smithi060.stdout: 405002 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/clusters 2024-03-27T21:42:44.672 INFO:teuthology.orchestra.run.smithi060.stdout: 405003 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.672 INFO:teuthology.orchestra.run.smithi060.stdout: 405004 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/clusters/1-node-1-mds-1-osd.yaml -> .qa/cephfs/clusters/1-node-1-mds-1-osd.yaml 2024-03-27T21:42:44.672 INFO:teuthology.orchestra.run.smithi060.stdout: 405001 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/begin -> .qa/cephfs/begin/ 2024-03-27T21:42:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: 405021 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/tasks 2024-03-27T21:42:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: 405022 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: 405023 4 -rw-r--r-- 1 ubuntu ubuntu 666 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/tasks/mgr-osd-full.yaml 2024-03-27T21:42:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: 405013 4 -rw-r--r-- 1 ubuntu ubuntu 443 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/overrides.yaml 2024-03-27T21:42:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: 405006 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/full/distro -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: 405346 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash 2024-03-27T21:42:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: 405347 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/.qa -> ../.qa/ 2024-03-27T21:42:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: 405390 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads 2024-03-27T21:42:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: 405405 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/overrides 2024-03-27T21:42:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: 405416 0 lrwxrwxrwx 1 ubuntu ubuntu 45 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/overrides/prefetch_entire_dirfrags -> .qa/cephfs/overrides/prefetch_entire_dirfrags 2024-03-27T21:42:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: 405406 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/overrides/% 2024-03-27T21:42:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: 405407 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: 405418 0 lrwxrwxrwx 1 ubuntu ubuntu 41 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/overrides/session_timeout.yaml -> .qa/cephfs/overrides/session_timeout.yaml 2024-03-27T21:42:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: 405419 0 lrwxrwxrwx 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/overrides/thrashosds-health.yaml -> .qa/tasks/thrashosds-health.yaml 2024-03-27T21:42:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: 405408 4 -rw-r--r-- 1 ubuntu ubuntu 120 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/overrides/client-shutdown.yaml 2024-03-27T21:42:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: 405410 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: 405409 0 lrwxrwxrwx 1 ubuntu ubuntu 30 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/overrides/frag.yaml -> .qa/cephfs/overrides/frag.yaml 2024-03-27T21:42:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: 405411 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: 405417 4 -rw-r--r-- 1 ubuntu ubuntu 81 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/overrides/races.yaml 2024-03-27T21:42:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: 405412 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/overrides/prefetch_dirfrags 2024-03-27T21:42:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: 405413 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/overrides/prefetch_dirfrags/.qa -> ../.qa/ 2024-03-27T21:42:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: 405415 4 -rw-r--r-- 1 ubuntu ubuntu 80 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/overrides/prefetch_dirfrags/yes.yaml 2024-03-27T21:42:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: 405414 4 -rw-r--r-- 1 ubuntu ubuntu 81 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/overrides/prefetch_dirfrags/no.yaml 2024-03-27T21:42:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: 405391 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/% 2024-03-27T21:42:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: 405400 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/msgr-failures 2024-03-27T21:42:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: 405401 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: 405403 4 -rw-r--r-- 1 ubuntu ubuntu 502 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/msgr-failures/osd-mds-delay.yaml 2024-03-27T21:42:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: 405402 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/msgr-failures/none.yaml 2024-03-27T21:42:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: 405392 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: 405397 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/conf -> .qa/cephfs/conf 2024-03-27T21:42:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: 405399 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/mount -> .qa/cephfs/mount/ 2024-03-27T21:42:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: 405420 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/ranks 2024-03-27T21:42:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: 405421 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/ranks/.qa -> ../.qa/ 2024-03-27T21:42:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: 405422 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/ranks/1.yaml 2024-03-27T21:42:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: 405424 4 -rw-r--r-- 1 ubuntu ubuntu 136 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/ranks/5.yaml 2024-03-27T21:42:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: 405423 4 -rw-r--r-- 1 ubuntu ubuntu 136 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/ranks/3.yaml 2024-03-27T21:42:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: 405404 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/objectstore-ec -> .qa/cephfs/objectstore-ec 2024-03-27T21:42:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: 405394 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/clusters 2024-03-27T21:42:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: 405396 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/clusters/1a5s-mds-1c-client.yaml -> .qa/cephfs/clusters/1a5s-mds-1c-client.yaml 2024-03-27T21:42:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: 405395 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: 405393 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/begin -> .qa/cephfs/begin/ 2024-03-27T21:42:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: 405425 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/tasks 2024-03-27T21:42:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: 405433 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/tasks/2-workunit 2024-03-27T21:42:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: 405439 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/tasks/2-workunit/suites 2024-03-27T21:42:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: 405440 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/tasks/2-workunit/suites/.qa -> ../.qa/ 2024-03-27T21:42:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: 405443 4 -rw-r--r-- 1 ubuntu ubuntu 70 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/tasks/2-workunit/suites/iozone.yaml 2024-03-27T21:42:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: 405444 4 -rw-r--r-- 1 ubuntu ubuntu 161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/tasks/2-workunit/suites/pjd.yaml 2024-03-27T21:42:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: 405442 4 -rw-r--r-- 1 ubuntu ubuntu 88 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/tasks/2-workunit/suites/fsstress.yaml 2024-03-27T21:42:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: 405441 4 -rw-r--r-- 1 ubuntu ubuntu 193 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/tasks/2-workunit/suites/ffsb.yaml 2024-03-27T21:42:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: 405435 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/tasks/2-workunit/fs 2024-03-27T21:42:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: 405436 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/tasks/2-workunit/fs/.qa -> ../.qa/ 2024-03-27T21:42:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: 405437 4 -rw-r--r-- 1 ubuntu ubuntu 78 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/tasks/2-workunit/fs/snaps.yaml 2024-03-27T21:42:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: 405438 0 lrwxrwxrwx 1 ubuntu ubuntu 49 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/tasks/2-workunit/fs/trivial_sync.yaml -> .qa/cephfs/tasks/cfuse_workunit_trivial_sync.yaml 2024-03-27T21:42:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: 405434 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/tasks/2-workunit/.qa -> ../.qa/ 2024-03-27T21:42:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: 405426 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/tasks/% 2024-03-27T21:42:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: 405427 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: 405428 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/tasks/1-thrash 2024-03-27T21:42:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: 405429 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/tasks/1-thrash/.qa -> ../.qa/ 2024-03-27T21:42:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: 405430 4 -rw-r--r-- 1 ubuntu ubuntu 90 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/tasks/1-thrash/mds.yaml 2024-03-27T21:42:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: 405431 4 -rw-r--r-- 1 ubuntu ubuntu 176 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/tasks/1-thrash/mon.yaml 2024-03-27T21:42:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: 405432 4 -rw-r--r-- 1 ubuntu ubuntu 182 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/tasks/1-thrash/osd.yaml 2024-03-27T21:42:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: 405398 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/workloads/distro -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: 405348 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs 2024-03-27T21:42:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: 405365 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/overrides 2024-03-27T21:42:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: 405366 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/overrides/+ 2024-03-27T21:42:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: 405367 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: 405372 4 -rw-r--r-- 1 ubuntu ubuntu 241 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/overrides/multifs.yaml 2024-03-27T21:42:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: 405373 0 lrwxrwxrwx 1 ubuntu ubuntu 41 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/overrides/session_timeout.yaml -> .qa/cephfs/overrides/session_timeout.yaml 2024-03-27T21:42:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: 405374 0 lrwxrwxrwx 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/overrides/thrashosds-health.yaml -> .qa/tasks/thrashosds-health.yaml 2024-03-27T21:42:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: 405368 4 -rw-r--r-- 1 ubuntu ubuntu 120 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/overrides/client-shutdown.yaml 2024-03-27T21:42:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: 405370 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: 405369 0 lrwxrwxrwx 1 ubuntu ubuntu 30 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/overrides/frag.yaml -> .qa/cephfs/overrides/frag.yaml 2024-03-27T21:42:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: 405371 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: 405349 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/% 2024-03-27T21:42:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: 405362 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/objectstore 2024-03-27T21:42:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: 405363 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/objectstore/.qa -> ../.qa/ 2024-03-27T21:42:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: 405364 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/objectstore/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: 405358 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/msgr-failures 2024-03-27T21:42:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: 405359 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: 405361 4 -rw-r--r-- 1 ubuntu ubuntu 502 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/msgr-failures/osd-mds-delay.yaml 2024-03-27T21:42:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: 405360 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/msgr-failures/none.yaml 2024-03-27T21:42:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: 405350 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/.qa -> ../.qa/ 2024-03-27T21:42:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: 405355 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/conf -> .qa/cephfs/conf 2024-03-27T21:42:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: 405357 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/mount -> .qa/cephfs/mount/ 2024-03-27T21:42:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: 405352 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/clusters 2024-03-27T21:42:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: 405353 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: 405354 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/clusters/1a3s-mds-2c-client.yaml -> .qa/cephfs/clusters/1a3s-mds-2c-client.yaml 2024-03-27T21:42:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: 405351 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/begin -> .qa/cephfs/begin/ 2024-03-27T21:42:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: 405375 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/tasks 2024-03-27T21:42:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: 405382 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/tasks/2-workunit 2024-03-27T21:42:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: 405383 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/tasks/2-workunit/.qa -> ../.qa/ 2024-03-27T21:42:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: 405384 4 -rw-r--r-- 1 ubuntu ubuntu 78 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/tasks/2-workunit/cfuse_workunit_snaptests.yaml 2024-03-27T21:42:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: 405387 0 lrwxrwxrwx 1 ubuntu ubuntu 49 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/tasks/2-workunit/cfuse_workunit_trivial_sync.yaml -> .qa/cephfs/tasks/cfuse_workunit_trivial_sync.yaml 2024-03-27T21:42:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: 405385 0 lrwxrwxrwx 1 ubuntu ubuntu 52 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/tasks/2-workunit/cfuse_workunit_suites_fsstress.yaml -> .qa/cephfs/tasks/cfuse_workunit_suites_fsstress.yaml 2024-03-27T21:42:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: 405389 4 -rw-r--r-- 1 ubuntu ubuntu 70 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/tasks/2-workunit/iozone.yaml 2024-03-27T21:42:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: 405386 4 -rw-r--r-- 1 ubuntu ubuntu 161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/tasks/2-workunit/cfuse_workunit_suites_pjd.yaml 2024-03-27T21:42:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: 405388 4 -rw-r--r-- 1 ubuntu ubuntu 193 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/tasks/2-workunit/ffsb.yaml 2024-03-27T21:42:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: 405376 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/tasks/% 2024-03-27T21:42:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: 405377 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: 405378 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/tasks/1-thrash 2024-03-27T21:42:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: 405379 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/tasks/1-thrash/.qa -> ../.qa/ 2024-03-27T21:42:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: 405380 4 -rw-r--r-- 1 ubuntu ubuntu 90 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/tasks/1-thrash/mds.yaml 2024-03-27T21:42:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: 405381 4 -rw-r--r-- 1 ubuntu ubuntu 176 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/tasks/1-thrash/mon.yaml 2024-03-27T21:42:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: 405356 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/thrash/multifs/distro -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: 405090 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs 2024-03-27T21:42:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: 405102 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/overrides 2024-03-27T21:42:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: 405103 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/overrides/+ 2024-03-27T21:42:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: 405104 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: 405107 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/overrides/no_client_pidfile.yaml -> .qa/overrides/no_client_pidfile.yaml 2024-03-27T21:42:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: 405108 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/overrides/pg_health.yaml -> .qa/cephfs/overrides/pg_health.yaml 2024-03-27T21:42:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: 405105 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: 405106 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: 405091 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/% 2024-03-27T21:42:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: 405099 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/objectstore 2024-03-27T21:42:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: 405100 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/objectstore/.qa -> ../.qa/ 2024-03-27T21:42:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: 405101 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/objectstore/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: 405092 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/.qa -> ../.qa/ 2024-03-27T21:42:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: 405097 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/conf -> .qa/cephfs/conf 2024-03-27T21:42:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: 405094 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/clusters 2024-03-27T21:42:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: 405095 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: 405096 0 lrwxrwxrwx 1 ubuntu ubuntu 45 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/clusters/1-mds-1-client-coloc.yaml -> .qa/cephfs/clusters/1-mds-1-client-coloc.yaml 2024-03-27T21:42:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: 405093 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/begin -> .qa/cephfs/begin/ 2024-03-27T21:42:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: 405109 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/tasks 2024-03-27T21:42:44.688 INFO:teuthology.orchestra.run.smithi060.stdout: 405118 4 -rw-r--r-- 1 ubuntu ubuntu 275 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/tasks/libcephfs_python.yaml 2024-03-27T21:42:44.688 INFO:teuthology.orchestra.run.smithi060.stdout: 405112 4 -rw-r--r-- 1 ubuntu ubuntu 368 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/tasks/ino_release_cb.yaml 2024-03-27T21:42:44.688 INFO:teuthology.orchestra.run.smithi060.stdout: 405110 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.688 INFO:teuthology.orchestra.run.smithi060.stdout: 405111 4 -rw-r--r-- 1 ubuntu ubuntu 214 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/tasks/client.yaml 2024-03-27T21:42:44.688 INFO:teuthology.orchestra.run.smithi060.stdout: 405113 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/tasks/libcephfs 2024-03-27T21:42:44.688 INFO:teuthology.orchestra.run.smithi060.stdout: 405114 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/tasks/libcephfs/+ 2024-03-27T21:42:44.688 INFO:teuthology.orchestra.run.smithi060.stdout: 405115 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/tasks/libcephfs/.qa -> ../.qa/ 2024-03-27T21:42:44.688 INFO:teuthology.orchestra.run.smithi060.stdout: 405117 4 -rw-r--r-- 1 ubuntu ubuntu 285 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/tasks/libcephfs/test.yaml 2024-03-27T21:42:44.688 INFO:teuthology.orchestra.run.smithi060.stdout: 405116 0 lrwxrwxrwx 1 ubuntu ubuntu 30 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/tasks/libcephfs/frag.yaml -> .qa/cephfs/overrides/frag.yaml 2024-03-27T21:42:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: 405098 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/libcephfs/distro -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: 405119 4 drwxr-xr-x 8 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha 2024-03-27T21:42:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: 405139 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/overrides 2024-03-27T21:42:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: 405140 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/overrides/+ 2024-03-27T21:42:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: 405141 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: 405143 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/overrides/pg_health.yaml -> .qa/cephfs/overrides/pg_health.yaml 2024-03-27T21:42:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: 405142 0 lrwxrwxrwx 1 ubuntu ubuntu 45 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/overrides/ignorelist_health.yaml -> ./.qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: 405120 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/% 2024-03-27T21:42:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: 405128 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/clients 2024-03-27T21:42:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: 405129 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/clients/+ 2024-03-27T21:42:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: 405130 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/clients/.qa -> ../.qa/ 2024-03-27T21:42:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: 405131 4 -rw-r--r-- 1 ubuntu ubuntu 1558 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/clients/mirror.yaml 2024-03-27T21:42:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: 405136 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/objectstore 2024-03-27T21:42:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: 405137 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/objectstore/.qa -> ../.qa/ 2024-03-27T21:42:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: 405138 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/objectstore/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.691 INFO:teuthology.orchestra.run.smithi060.stdout: 405121 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/.qa -> ../.qa/ 2024-03-27T21:42:44.691 INFO:teuthology.orchestra.run.smithi060.stdout: 405144 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.691 INFO:teuthology.orchestra.run.smithi060.stdout: 405145 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/workloads 2024-03-27T21:42:44.691 INFO:teuthology.orchestra.run.smithi060.stdout: 405147 4 -rw-r--r-- 1 ubuntu ubuntu 710 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/workloads/cephfs-mirror-ha-workunit.yaml 2024-03-27T21:42:44.691 INFO:teuthology.orchestra.run.smithi060.stdout: 405146 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.691 INFO:teuthology.orchestra.run.smithi060.stdout: 405123 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/cephfs-mirror 2024-03-27T21:42:44.691 INFO:teuthology.orchestra.run.smithi060.stdout: 405124 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/cephfs-mirror/+ 2024-03-27T21:42:44.691 INFO:teuthology.orchestra.run.smithi060.stdout: 405125 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/cephfs-mirror/.qa -> ../.qa/ 2024-03-27T21:42:44.691 INFO:teuthology.orchestra.run.smithi060.stdout: 405127 4 -rw-r--r-- 1 ubuntu ubuntu 285 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/cephfs-mirror/2-three-per-cluster.yaml 2024-03-27T21:42:44.692 INFO:teuthology.orchestra.run.smithi060.stdout: 405126 4 -rw-r--r-- 1 ubuntu ubuntu 394 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/cephfs-mirror/1-volume-create-rm.yaml 2024-03-27T21:42:44.692 INFO:teuthology.orchestra.run.smithi060.stdout: 405122 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/begin -> .qa/cephfs/begin/ 2024-03-27T21:42:44.692 INFO:teuthology.orchestra.run.smithi060.stdout: 405132 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/cluster 2024-03-27T21:42:44.692 INFO:teuthology.orchestra.run.smithi060.stdout: 405133 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/cluster/+ 2024-03-27T21:42:44.692 INFO:teuthology.orchestra.run.smithi060.stdout: 405134 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/cluster/.qa -> ../.qa/ 2024-03-27T21:42:44.692 INFO:teuthology.orchestra.run.smithi060.stdout: 405135 4 -rw-r--r-- 1 ubuntu ubuntu 290 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mirror-ha/cluster/1-node.yaml 2024-03-27T21:42:44.692 INFO:teuthology.orchestra.run.smithi060.stdout: 405781 4 drwxr-xr-x 9 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload 2024-03-27T21:42:44.692 INFO:teuthology.orchestra.run.smithi060.stdout: 405784 0 lrwxrwxrwx 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/0-centos_9.stream.yaml -> .qa/distros/podman/centos_9.stream.yaml 2024-03-27T21:42:44.692 INFO:teuthology.orchestra.run.smithi060.stdout: 405817 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/overrides 2024-03-27T21:42:44.693 INFO:teuthology.orchestra.run.smithi060.stdout: 405818 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/overrides/+ 2024-03-27T21:42:44.693 INFO:teuthology.orchestra.run.smithi060.stdout: 405819 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.693 INFO:teuthology.orchestra.run.smithi060.stdout: 405826 0 lrwxrwxrwx 1 ubuntu ubuntu 41 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/overrides/session_timeout.yaml -> .qa/cephfs/overrides/session_timeout.yaml 2024-03-27T21:42:44.693 INFO:teuthology.orchestra.run.smithi060.stdout: 405825 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/overrides/pg_health.yaml -> .qa/cephfs/overrides/pg_health.yaml 2024-03-27T21:42:44.693 INFO:teuthology.orchestra.run.smithi060.stdout: 405824 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/overrides/osd-asserts.yaml -> .qa/cephfs/overrides/osd-asserts.yaml 2024-03-27T21:42:44.693 INFO:teuthology.orchestra.run.smithi060.stdout: 405822 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.693 INFO:teuthology.orchestra.run.smithi060.stdout: 405821 0 lrwxrwxrwx 1 ubuntu ubuntu 30 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/overrides/frag.yaml -> .qa/cephfs/overrides/frag.yaml 2024-03-27T21:42:44.693 INFO:teuthology.orchestra.run.smithi060.stdout: 405823 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.694 INFO:teuthology.orchestra.run.smithi060.stdout: 405820 4 -rw-r--r-- 1 ubuntu ubuntu 182 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/overrides/cephsqlite-timeout.yaml 2024-03-27T21:42:44.694 INFO:teuthology.orchestra.run.smithi060.stdout: 405782 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/% 2024-03-27T21:42:44.694 INFO:teuthology.orchestra.run.smithi060.stdout: 405783 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/.qa -> ../.qa/ 2024-03-27T21:42:44.694 INFO:teuthology.orchestra.run.smithi060.stdout: 405843 4 -rw-r--r-- 1 ubuntu ubuntu 58 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/standby-replay.yaml 2024-03-27T21:42:44.694 INFO:teuthology.orchestra.run.smithi060.stdout: 405813 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/omap_limit 2024-03-27T21:42:44.694 INFO:teuthology.orchestra.run.smithi060.stdout: 405814 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/omap_limit/.qa -> ../.qa/ 2024-03-27T21:42:44.694 INFO:teuthology.orchestra.run.smithi060.stdout: 405815 4 -rw-r--r-- 1 ubuntu ubuntu 85 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/omap_limit/10.yaml 2024-03-27T21:42:44.694 INFO:teuthology.orchestra.run.smithi060.stdout: 405816 4 -rw-r--r-- 1 ubuntu ubuntu 88 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/omap_limit/10000.yaml 2024-03-27T21:42:44.695 INFO:teuthology.orchestra.run.smithi060.stdout: 405795 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/conf -> .qa/cephfs/conf 2024-03-27T21:42:44.695 INFO:teuthology.orchestra.run.smithi060.stdout: 405796 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/mount 2024-03-27T21:42:44.695 INFO:teuthology.orchestra.run.smithi060.stdout: 405797 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/mount/.qa -> ../.qa/ 2024-03-27T21:42:44.695 INFO:teuthology.orchestra.run.smithi060.stdout: 405798 0 lrwxrwxrwx 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/mount/fuse.yaml -> .qa/cephfs/mount/fuse.yaml 2024-03-27T21:42:44.695 INFO:teuthology.orchestra.run.smithi060.stdout: 405799 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/mount/kclient 2024-03-27T21:42:44.695 INFO:teuthology.orchestra.run.smithi060.stdout: 405800 4 -rw-r--r-- 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/mount/kclient/% 2024-03-27T21:42:44.695 INFO:teuthology.orchestra.run.smithi060.stdout: 405808 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/mount/kclient/wsync 2024-03-27T21:42:44.695 INFO:teuthology.orchestra.run.smithi060.stdout: 405809 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/mount/kclient/wsync/.qa -> ../.qa/ 2024-03-27T21:42:44.696 INFO:teuthology.orchestra.run.smithi060.stdout: 405811 4 -rw-r--r-- 1 ubuntu ubuntu 45 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/mount/kclient/wsync/yes.yaml 2024-03-27T21:42:44.696 INFO:teuthology.orchestra.run.smithi060.stdout: 405810 4 -rw-r--r-- 1 ubuntu ubuntu 47 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/mount/kclient/wsync/no.yaml 2024-03-27T21:42:44.696 INFO:teuthology.orchestra.run.smithi060.stdout: 405801 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/mount/kclient/.qa -> ../.qa/ 2024-03-27T21:42:44.696 INFO:teuthology.orchestra.run.smithi060.stdout: 405803 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/mount/kclient/ms_mode 2024-03-27T21:42:44.696 INFO:teuthology.orchestra.run.smithi060.stdout: 405807 4 -rw-r--r-- 1 ubuntu ubuntu 54 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/mount/kclient/ms_mode/secure.yaml 2024-03-27T21:42:44.696 INFO:teuthology.orchestra.run.smithi060.stdout: 405805 4 -rw-r--r-- 1 ubuntu ubuntu 51 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/mount/kclient/ms_mode/crc.yaml 2024-03-27T21:42:44.696 INFO:teuthology.orchestra.run.smithi060.stdout: 405804 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/mount/kclient/ms_mode/.qa -> ../.qa/ 2024-03-27T21:42:44.697 INFO:teuthology.orchestra.run.smithi060.stdout: 405806 4 -rw-r--r-- 1 ubuntu ubuntu 54 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/mount/kclient/ms_mode/legacy.yaml 2024-03-27T21:42:44.697 INFO:teuthology.orchestra.run.smithi060.stdout: 405802 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/mount/kclient/base -> .qa/cephfs/mount/kclient/ 2024-03-27T21:42:44.697 INFO:teuthology.orchestra.run.smithi060.stdout: 405827 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/ranks 2024-03-27T21:42:44.697 INFO:teuthology.orchestra.run.smithi060.stdout: 405830 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/ranks/multi 2024-03-27T21:42:44.697 INFO:teuthology.orchestra.run.smithi060.stdout: 405836 4 -rw-r--r-- 1 ubuntu ubuntu 99 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/ranks/multi/export-check.yaml 2024-03-27T21:42:44.697 INFO:teuthology.orchestra.run.smithi060.stdout: 405831 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/ranks/multi/% 2024-03-27T21:42:44.697 INFO:teuthology.orchestra.run.smithi060.stdout: 405832 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/ranks/multi/balancer 2024-03-27T21:42:44.697 INFO:teuthology.orchestra.run.smithi060.stdout: 405835 4 -rw-r--r-- 1 ubuntu ubuntu 219 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/ranks/multi/balancer/random.yaml 2024-03-27T21:42:44.698 INFO:teuthology.orchestra.run.smithi060.stdout: 405833 4 -rw-r--r-- 1 ubuntu ubuntu 75 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/ranks/multi/balancer/automatic.yaml 2024-03-27T21:42:44.698 INFO:teuthology.orchestra.run.smithi060.stdout: 405834 4 -rw-r--r-- 1 ubuntu ubuntu 249 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/ranks/multi/balancer/distributed.yaml.disabled 2024-03-27T21:42:44.698 INFO:teuthology.orchestra.run.smithi060.stdout: 405837 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/ranks/multi/n 2024-03-27T21:42:44.698 INFO:teuthology.orchestra.run.smithi060.stdout: 405839 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/ranks/multi/n/5.yaml 2024-03-27T21:42:44.698 INFO:teuthology.orchestra.run.smithi060.stdout: 405838 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/ranks/multi/n/3.yaml 2024-03-27T21:42:44.698 INFO:teuthology.orchestra.run.smithi060.stdout: 405840 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/ranks/multi/replication 2024-03-27T21:42:44.698 INFO:teuthology.orchestra.run.smithi060.stdout: 405841 4 -rw-r--r-- 1 ubuntu ubuntu 494 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/ranks/multi/replication/always.yaml 2024-03-27T21:42:44.698 INFO:teuthology.orchestra.run.smithi060.stdout: 405842 4 -rw-r--r-- 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/ranks/multi/replication/default.yaml 2024-03-27T21:42:44.699 INFO:teuthology.orchestra.run.smithi060.stdout: 405828 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/ranks/.qa -> ../.qa/ 2024-03-27T21:42:44.699 INFO:teuthology.orchestra.run.smithi060.stdout: 405829 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/ranks/1.yaml 2024-03-27T21:42:44.699 INFO:teuthology.orchestra.run.smithi060.stdout: 405812 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/objectstore-ec -> .qa/cephfs/objectstore-ec 2024-03-27T21:42:44.699 INFO:teuthology.orchestra.run.smithi060.stdout: 405792 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/clusters 2024-03-27T21:42:44.699 INFO:teuthology.orchestra.run.smithi060.stdout: 405793 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.699 INFO:teuthology.orchestra.run.smithi060.stdout: 405794 0 lrwxrwxrwx 1 ubuntu ubuntu 50 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/clusters/1a11s-mds-1c-client-3node.yaml -> .qa/cephfs/clusters/1a11s-mds-1c-client-3node.yaml 2024-03-27T21:42:44.699 INFO:teuthology.orchestra.run.smithi060.stdout: 405785 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/begin 2024-03-27T21:42:44.699 INFO:teuthology.orchestra.run.smithi060.stdout: 405786 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/begin/+ 2024-03-27T21:42:44.700 INFO:teuthology.orchestra.run.smithi060.stdout: 405787 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/begin/.qa -> ../.qa/ 2024-03-27T21:42:44.700 INFO:teuthology.orchestra.run.smithi060.stdout: 405789 4 -rw-r--r-- 1 ubuntu ubuntu 367 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/begin/1-cephadm.yaml 2024-03-27T21:42:44.700 INFO:teuthology.orchestra.run.smithi060.stdout: 405790 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/begin/2-logrotate.yaml -> .qa/cephfs/begin/2-logrotate.yaml 2024-03-27T21:42:44.700 INFO:teuthology.orchestra.run.smithi060.stdout: 405791 0 lrwxrwxrwx 1 ubuntu ubuntu 31 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/begin/3-modules.yaml -> .qa/cephfs/begin/3-modules.yaml 2024-03-27T21:42:44.700 INFO:teuthology.orchestra.run.smithi060.stdout: 405788 0 lrwxrwxrwx 1 ubuntu ubuntu 31 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/begin/0-install.yaml -> .qa/cephfs/begin/0-install.yaml 2024-03-27T21:42:44.700 INFO:teuthology.orchestra.run.smithi060.stdout: 405844 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks 2024-03-27T21:42:44.700 INFO:teuthology.orchestra.run.smithi060.stdout: 405847 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/0-subvolume 2024-03-27T21:42:44.700 INFO:teuthology.orchestra.run.smithi060.stdout: 405849 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/0-subvolume/.qa -> ../.qa/ 2024-03-27T21:42:44.700 INFO:teuthology.orchestra.run.smithi060.stdout: 405850 4 -rw-r--r-- 1 ubuntu ubuntu 216 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/0-subvolume/with-namespace-isolated-and-quota.yaml 2024-03-27T21:42:44.701 INFO:teuthology.orchestra.run.smithi060.stdout: 405852 4 -rw-r--r-- 1 ubuntu ubuntu 152 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/0-subvolume/with-no-extra-options.yaml 2024-03-27T21:42:44.701 INFO:teuthology.orchestra.run.smithi060.stdout: 405848 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/0-subvolume/$ 2024-03-27T21:42:44.701 INFO:teuthology.orchestra.run.smithi060.stdout: 405853 4 -rw-r--r-- 1 ubuntu ubuntu 195 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/0-subvolume/with-quota.yaml 2024-03-27T21:42:44.701 INFO:teuthology.orchestra.run.smithi060.stdout: 405851 4 -rw-r--r-- 1 ubuntu ubuntu 197 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/0-subvolume/with-namespace-isolated.yaml 2024-03-27T21:42:44.701 INFO:teuthology.orchestra.run.smithi060.stdout: 405845 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/% 2024-03-27T21:42:44.701 INFO:teuthology.orchestra.run.smithi060.stdout: 405863 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/4-flush 2024-03-27T21:42:44.701 INFO:teuthology.orchestra.run.smithi060.stdout: 405864 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/4-flush/.qa -> ../.qa/ 2024-03-27T21:42:44.701 INFO:teuthology.orchestra.run.smithi060.stdout: 405866 4 -rw-r--r-- 1 ubuntu ubuntu 107 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/4-flush/yes.yaml 2024-03-27T21:42:44.702 INFO:teuthology.orchestra.run.smithi060.stdout: 405865 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/4-flush/no.yaml 2024-03-27T21:42:44.702 INFO:teuthology.orchestra.run.smithi060.stdout: 405846 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.702 INFO:teuthology.orchestra.run.smithi060.stdout: 405867 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/5-workunit 2024-03-27T21:42:44.702 INFO:teuthology.orchestra.run.smithi060.stdout: 405877 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/5-workunit/suites 2024-03-27T21:42:44.702 INFO:teuthology.orchestra.run.smithi060.stdout: 405878 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/5-workunit/suites/.qa -> ../.qa/ 2024-03-27T21:42:44.702 INFO:teuthology.orchestra.run.smithi060.stdout: 405885 4 -rw-r--r-- 1 ubuntu ubuntu 115 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/5-workunit/suites/iogen.yaml 2024-03-27T21:42:44.702 INFO:teuthology.orchestra.run.smithi060.stdout: 405884 4 -rw-r--r-- 1 ubuntu ubuntu 122 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/5-workunit/suites/fsync-tester.yaml 2024-03-27T21:42:44.703 INFO:teuthology.orchestra.run.smithi060.stdout: 405886 4 -rw-r--r-- 1 ubuntu ubuntu 116 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/5-workunit/suites/iozone.yaml 2024-03-27T21:42:44.703 INFO:teuthology.orchestra.run.smithi060.stdout: 405879 4 -rw-r--r-- 1 ubuntu ubuntu 152 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/5-workunit/suites/blogbench.yaml 2024-03-27T21:42:44.703 INFO:teuthology.orchestra.run.smithi060.stdout: 405887 4 -rw-r--r-- 1 ubuntu ubuntu 201 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/5-workunit/suites/pjd.yaml 2024-03-27T21:42:44.703 INFO:teuthology.orchestra.run.smithi060.stdout: 405882 4 -rw-r--r-- 1 ubuntu ubuntu 88 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/5-workunit/suites/fsstress.yaml 2024-03-27T21:42:44.703 INFO:teuthology.orchestra.run.smithi060.stdout: 405880 4 -rw-r--r-- 1 ubuntu ubuntu 70 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/5-workunit/suites/dbench.yaml 2024-03-27T21:42:44.703 INFO:teuthology.orchestra.run.smithi060.stdout: 405881 4 -rw-r--r-- 1 ubuntu ubuntu 293 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/5-workunit/suites/ffsb.yaml 2024-03-27T21:42:44.703 INFO:teuthology.orchestra.run.smithi060.stdout: 405883 4 -rw-r--r-- 1 ubuntu ubuntu 146 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/5-workunit/suites/fsx.yaml 2024-03-27T21:42:44.703 INFO:teuthology.orchestra.run.smithi060.stdout: 405875 4 -rw-r--r-- 1 ubuntu ubuntu 154 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/5-workunit/kernel_untar_build.yaml 2024-03-27T21:42:44.703 INFO:teuthology.orchestra.run.smithi060.stdout: 405876 4 -rw-r--r-- 1 ubuntu ubuntu 2055 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/5-workunit/postgres.yaml 2024-03-27T21:42:44.704 INFO:teuthology.orchestra.run.smithi060.stdout: 405870 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/5-workunit/fs 2024-03-27T21:42:44.704 INFO:teuthology.orchestra.run.smithi060.stdout: 405874 4 -rw-r--r-- 1 ubuntu ubuntu 118 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/5-workunit/fs/test_o_trunc.yaml 2024-03-27T21:42:44.704 INFO:teuthology.orchestra.run.smithi060.stdout: 405871 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/5-workunit/fs/.qa -> ../.qa/ 2024-03-27T21:42:44.704 INFO:teuthology.orchestra.run.smithi060.stdout: 405873 4 -rw-r--r-- 1 ubuntu ubuntu 253 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/5-workunit/fs/norstats.yaml 2024-03-27T21:42:44.704 INFO:teuthology.orchestra.run.smithi060.stdout: 405872 4 -rw-r--r-- 1 ubuntu ubuntu 140 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/5-workunit/fs/misc.yaml 2024-03-27T21:42:44.704 INFO:teuthology.orchestra.run.smithi060.stdout: 405868 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/5-workunit/.qa -> ../.qa/ 2024-03-27T21:42:44.704 INFO:teuthology.orchestra.run.smithi060.stdout: 405869 4 -rw-r--r-- 1 ubuntu ubuntu 110 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/5-workunit/direct_io.yaml 2024-03-27T21:42:44.704 INFO:teuthology.orchestra.run.smithi060.stdout: 405854 4 -rw-r--r-- 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/1-check-counter.yaml 2024-03-27T21:42:44.705 INFO:teuthology.orchestra.run.smithi060.stdout: 405859 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/3-snaps 2024-03-27T21:42:44.705 INFO:teuthology.orchestra.run.smithi060.stdout: 405860 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/3-snaps/.qa -> ../.qa/ 2024-03-27T21:42:44.705 INFO:teuthology.orchestra.run.smithi060.stdout: 405862 4 -rw-r--r-- 1 ubuntu ubuntu 1311 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/3-snaps/yes.yaml 2024-03-27T21:42:44.705 INFO:teuthology.orchestra.run.smithi060.stdout: 405861 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/3-snaps/no.yaml 2024-03-27T21:42:44.705 INFO:teuthology.orchestra.run.smithi060.stdout: 405855 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/2-scrub 2024-03-27T21:42:44.705 INFO:teuthology.orchestra.run.smithi060.stdout: 405856 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/2-scrub/.qa -> ../.qa/ 2024-03-27T21:42:44.705 INFO:teuthology.orchestra.run.smithi060.stdout: 405858 4 -rw-r--r-- 1 ubuntu ubuntu 177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/2-scrub/yes.yaml 2024-03-27T21:42:44.705 INFO:teuthology.orchestra.run.smithi060.stdout: 405857 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/workload/tasks/2-scrub/no.yaml 2024-03-27T21:42:44.705 INFO:teuthology.orchestra.run.smithi060.stdout: 405748 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes 2024-03-27T21:42:44.706 INFO:teuthology.orchestra.run.smithi060.stdout: 405761 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/overrides 2024-03-27T21:42:44.706 INFO:teuthology.orchestra.run.smithi060.stdout: 405762 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/overrides/+ 2024-03-27T21:42:44.706 INFO:teuthology.orchestra.run.smithi060.stdout: 405763 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.706 INFO:teuthology.orchestra.run.smithi060.stdout: 405766 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/overrides/no_client_pidfile.yaml -> .qa/overrides/no_client_pidfile.yaml 2024-03-27T21:42:44.706 INFO:teuthology.orchestra.run.smithi060.stdout: 405767 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/overrides/pg_health.yaml -> .qa/cephfs/overrides/pg_health.yaml 2024-03-27T21:42:44.706 INFO:teuthology.orchestra.run.smithi060.stdout: 405764 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.706 INFO:teuthology.orchestra.run.smithi060.stdout: 405765 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.706 INFO:teuthology.orchestra.run.smithi060.stdout: 405749 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/% 2024-03-27T21:42:44.707 INFO:teuthology.orchestra.run.smithi060.stdout: 405758 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/objectstore 2024-03-27T21:42:44.707 INFO:teuthology.orchestra.run.smithi060.stdout: 405759 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/objectstore/.qa -> ../.qa/ 2024-03-27T21:42:44.707 INFO:teuthology.orchestra.run.smithi060.stdout: 405760 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/objectstore/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.707 INFO:teuthology.orchestra.run.smithi060.stdout: 405750 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/.qa -> ../.qa/ 2024-03-27T21:42:44.707 INFO:teuthology.orchestra.run.smithi060.stdout: 405755 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/conf -> .qa/cephfs/conf 2024-03-27T21:42:44.707 INFO:teuthology.orchestra.run.smithi060.stdout: 405757 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/mount -> .qa/cephfs/mount/ 2024-03-27T21:42:44.707 INFO:teuthology.orchestra.run.smithi060.stdout: 405752 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/clusters 2024-03-27T21:42:44.707 INFO:teuthology.orchestra.run.smithi060.stdout: 405753 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.708 INFO:teuthology.orchestra.run.smithi060.stdout: 405754 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/clusters/1a3s-mds-4c-client.yaml -> .qa/cephfs/clusters/1a3s-mds-4c-client.yaml 2024-03-27T21:42:44.708 INFO:teuthology.orchestra.run.smithi060.stdout: 405751 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/begin -> .qa/cephfs/begin/ 2024-03-27T21:42:44.708 INFO:teuthology.orchestra.run.smithi060.stdout: 405768 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/tasks 2024-03-27T21:42:44.708 INFO:teuthology.orchestra.run.smithi060.stdout: 405769 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.708 INFO:teuthology.orchestra.run.smithi060.stdout: 405770 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/tasks/volumes 2024-03-27T21:42:44.708 INFO:teuthology.orchestra.run.smithi060.stdout: 405771 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/tasks/volumes/% 2024-03-27T21:42:44.708 INFO:teuthology.orchestra.run.smithi060.stdout: 405772 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/tasks/volumes/.qa -> ../.qa/ 2024-03-27T21:42:44.708 INFO:teuthology.orchestra.run.smithi060.stdout: 405774 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/tasks/volumes/test 2024-03-27T21:42:44.709 INFO:teuthology.orchestra.run.smithi060.stdout: 405775 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/tasks/volumes/test/.qa -> ../.qa/ 2024-03-27T21:42:44.709 INFO:teuthology.orchestra.run.smithi060.stdout: 405777 4 -rw-r--r-- 1 ubuntu ubuntu 136 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/tasks/volumes/test/clone.yaml 2024-03-27T21:42:44.709 INFO:teuthology.orchestra.run.smithi060.stdout: 405778 4 -rw-r--r-- 1 ubuntu ubuntu 349 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/tasks/volumes/test/finisher_per_module.yaml 2024-03-27T21:42:44.709 INFO:teuthology.orchestra.run.smithi060.stdout: 405779 4 -rw-r--r-- 1 ubuntu ubuntu 117 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/tasks/volumes/test/misc.yaml 2024-03-27T21:42:44.709 INFO:teuthology.orchestra.run.smithi060.stdout: 405776 4 -rw-r--r-- 1 ubuntu ubuntu 279 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/tasks/volumes/test/basic.yaml 2024-03-27T21:42:44.709 INFO:teuthology.orchestra.run.smithi060.stdout: 405780 4 -rw-r--r-- 1 ubuntu ubuntu 195 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/tasks/volumes/test/snapshot.yaml 2024-03-27T21:42:44.709 INFO:teuthology.orchestra.run.smithi060.stdout: 405773 4 -rw-r--r-- 1 ubuntu ubuntu 402 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/tasks/volumes/overrides.yaml 2024-03-27T21:42:44.709 INFO:teuthology.orchestra.run.smithi060.stdout: 405756 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/volumes/distro -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.710 INFO:teuthology.orchestra.run.smithi060.stdout: 405178 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients 2024-03-27T21:42:44.710 INFO:teuthology.orchestra.run.smithi060.stdout: 405197 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/overrides 2024-03-27T21:42:44.710 INFO:teuthology.orchestra.run.smithi060.stdout: 405198 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/overrides/+ 2024-03-27T21:42:44.710 INFO:teuthology.orchestra.run.smithi060.stdout: 405199 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.710 INFO:teuthology.orchestra.run.smithi060.stdout: 405203 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/overrides/pg_health.yaml -> .qa/cephfs/overrides/pg_health.yaml 2024-03-27T21:42:44.710 INFO:teuthology.orchestra.run.smithi060.stdout: 405202 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/overrides/osd-asserts.yaml -> .qa/cephfs/overrides/osd-asserts.yaml 2024-03-27T21:42:44.710 INFO:teuthology.orchestra.run.smithi060.stdout: 405200 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.710 INFO:teuthology.orchestra.run.smithi060.stdout: 405201 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.711 INFO:teuthology.orchestra.run.smithi060.stdout: 405179 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/% 2024-03-27T21:42:44.711 INFO:teuthology.orchestra.run.smithi060.stdout: 405180 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/.qa -> ../.qa/ 2024-03-27T21:42:44.711 INFO:teuthology.orchestra.run.smithi060.stdout: 405185 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/conf -> .qa/cephfs/conf 2024-03-27T21:42:44.711 INFO:teuthology.orchestra.run.smithi060.stdout: 405196 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/objectstore-ec -> .qa/cephfs/objectstore-ec 2024-03-27T21:42:44.711 INFO:teuthology.orchestra.run.smithi060.stdout: 405182 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/clusters 2024-03-27T21:42:44.711 INFO:teuthology.orchestra.run.smithi060.stdout: 405183 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.711 INFO:teuthology.orchestra.run.smithi060.stdout: 405184 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/clusters/1a3s-mds-2c-client.yaml -> .qa/cephfs/clusters/1a3s-mds-2c-client.yaml 2024-03-27T21:42:44.711 INFO:teuthology.orchestra.run.smithi060.stdout: 405181 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/begin -> .qa/cephfs/begin/ 2024-03-27T21:42:44.712 INFO:teuthology.orchestra.run.smithi060.stdout: 405195 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/kclient-overrides -> .qa/cephfs/mount/kclient/overrides/ 2024-03-27T21:42:44.712 INFO:teuthology.orchestra.run.smithi060.stdout: 405204 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/tasks 2024-03-27T21:42:44.712 INFO:teuthology.orchestra.run.smithi060.stdout: 405205 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.712 INFO:teuthology.orchestra.run.smithi060.stdout: 405207 4 -rw-r--r-- 1 ubuntu ubuntu 335 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/tasks/kernel_cfuse_workunits_untarbuild_blogbench.yaml 2024-03-27T21:42:44.712 INFO:teuthology.orchestra.run.smithi060.stdout: 405206 4 -rw-r--r-- 1 ubuntu ubuntu 327 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/tasks/kernel_cfuse_workunits_dbench_iozone.yaml 2024-03-27T21:42:44.712 INFO:teuthology.orchestra.run.smithi060.stdout: 405186 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/distro 2024-03-27T21:42:44.712 INFO:teuthology.orchestra.run.smithi060.stdout: 405188 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/distro/.qa -> ../.qa/ 2024-03-27T21:42:44.712 INFO:teuthology.orchestra.run.smithi060.stdout: 405190 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/distro/ubuntu 2024-03-27T21:42:44.712 INFO:teuthology.orchestra.run.smithi060.stdout: 405191 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/distro/ubuntu/+ 2024-03-27T21:42:44.713 INFO:teuthology.orchestra.run.smithi060.stdout: 405192 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/distro/ubuntu/.qa -> ../.qa/ 2024-03-27T21:42:44.713 INFO:teuthology.orchestra.run.smithi060.stdout: 405194 0 lrwxrwxrwx 1 ubuntu ubuntu 34 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/distro/ubuntu/ubuntu_latest.yaml -> .qa/distros/all/ubuntu_latest.yaml 2024-03-27T21:42:44.713 INFO:teuthology.orchestra.run.smithi060.stdout: 405193 4 -rw-r--r-- 1 ubuntu ubuntu 66 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/distro/ubuntu/overrides.yaml 2024-03-27T21:42:44.713 INFO:teuthology.orchestra.run.smithi060.stdout: 405189 0 lrwxrwxrwx 1 ubuntu ubuntu 34 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/distro/centos_latest.yaml -> .qa/distros/all/centos_latest.yaml 2024-03-27T21:42:44.713 INFO:teuthology.orchestra.run.smithi060.stdout: 405187 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/mixed-clients/distro/$ 2024-03-27T21:42:44.713 INFO:teuthology.orchestra.run.smithi060.stdout: 404940 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm 2024-03-27T21:42:44.713 INFO:teuthology.orchestra.run.smithi060.stdout: 404941 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/.qa -> ../.qa/ 2024-03-27T21:42:44.713 INFO:teuthology.orchestra.run.smithi060.stdout: 404942 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/multivolume 2024-03-27T21:42:44.714 INFO:teuthology.orchestra.run.smithi060.stdout: 404946 4 -rw-r--r-- 1 ubuntu ubuntu 119 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/multivolume/1-mount.yaml 2024-03-27T21:42:44.714 INFO:teuthology.orchestra.run.smithi060.stdout: 404943 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/multivolume/% 2024-03-27T21:42:44.714 INFO:teuthology.orchestra.run.smithi060.stdout: 404944 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/multivolume/.qa -> ../.qa/ 2024-03-27T21:42:44.714 INFO:teuthology.orchestra.run.smithi060.stdout: 404945 4 -rw-r--r-- 1 ubuntu ubuntu 620 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/multivolume/0-start.yaml 2024-03-27T21:42:44.714 INFO:teuthology.orchestra.run.smithi060.stdout: 404947 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/multivolume/2-workload 2024-03-27T21:42:44.714 INFO:teuthology.orchestra.run.smithi060.stdout: 404948 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/multivolume/2-workload/.qa -> ../.qa/ 2024-03-27T21:42:44.714 INFO:teuthology.orchestra.run.smithi060.stdout: 404949 0 lrwxrwxrwx 1 ubuntu ubuntu 58 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/multivolume/2-workload/dbench.yaml -> .qa/suites/fs/workload/tasks/5-workunit/suites/dbench.yaml 2024-03-27T21:42:44.715 INFO:teuthology.orchestra.run.smithi060.stdout: 404950 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/multivolume/distro 2024-03-27T21:42:44.715 INFO:teuthology.orchestra.run.smithi060.stdout: 404951 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/multivolume/distro/.qa -> ../.qa/ 2024-03-27T21:42:44.715 INFO:teuthology.orchestra.run.smithi060.stdout: 404952 0 lrwxrwxrwx 1 ubuntu ubuntu 38 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/multivolume/distro/single-container-host.yaml -> .qa/distros/single-container-host.yaml 2024-03-27T21:42:44.715 INFO:teuthology.orchestra.run.smithi060.stdout: 404953 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/renamevolume 2024-03-27T21:42:44.715 INFO:teuthology.orchestra.run.smithi060.stdout: 404961 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/renamevolume/overrides 2024-03-27T21:42:44.715 INFO:teuthology.orchestra.run.smithi060.stdout: 404962 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/renamevolume/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.715 INFO:teuthology.orchestra.run.smithi060.stdout: 404963 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/renamevolume/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.715 INFO:teuthology.orchestra.run.smithi060.stdout: 404954 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/renamevolume/% 2024-03-27T21:42:44.716 INFO:teuthology.orchestra.run.smithi060.stdout: 404955 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/renamevolume/.qa -> ../.qa/ 2024-03-27T21:42:44.716 INFO:teuthology.orchestra.run.smithi060.stdout: 404956 4 -rw-r--r-- 1 ubuntu ubuntu 586 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/renamevolume/0-start.yaml 2024-03-27T21:42:44.716 INFO:teuthology.orchestra.run.smithi060.stdout: 404957 4 -rw-r--r-- 1 ubuntu ubuntu 430 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/renamevolume/1-rename.yaml 2024-03-27T21:42:44.716 INFO:teuthology.orchestra.run.smithi060.stdout: 404958 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/renamevolume/distro 2024-03-27T21:42:44.716 INFO:teuthology.orchestra.run.smithi060.stdout: 404959 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/renamevolume/distro/.qa -> ../.qa/ 2024-03-27T21:42:44.716 INFO:teuthology.orchestra.run.smithi060.stdout: 404960 0 lrwxrwxrwx 1 ubuntu ubuntu 38 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/cephadm/renamevolume/distro/single-container-host.yaml -> .qa/distros/single-container-host.yaml 2024-03-27T21:42:44.716 INFO:teuthology.orchestra.run.smithi060.stdout: 404917 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs 2024-03-27T21:42:44.716 INFO:teuthology.orchestra.run.smithi060.stdout: 404918 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/.qa -> ../.qa/ 2024-03-27T21:42:44.717 INFO:teuthology.orchestra.run.smithi060.stdout: 404919 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/client_trim_caps 2024-03-27T21:42:44.717 INFO:teuthology.orchestra.run.smithi060.stdout: 404931 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/client_trim_caps/overrides 2024-03-27T21:42:44.717 INFO:teuthology.orchestra.run.smithi060.stdout: 404932 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/client_trim_caps/overrides/+ 2024-03-27T21:42:44.717 INFO:teuthology.orchestra.run.smithi060.stdout: 404933 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/client_trim_caps/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.717 INFO:teuthology.orchestra.run.smithi060.stdout: 404936 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/client_trim_caps/overrides/no_client_pidfile.yaml -> .qa/overrides/no_client_pidfile.yaml 2024-03-27T21:42:44.717 INFO:teuthology.orchestra.run.smithi060.stdout: 404934 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/client_trim_caps/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.717 INFO:teuthology.orchestra.run.smithi060.stdout: 404935 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/client_trim_caps/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.717 INFO:teuthology.orchestra.run.smithi060.stdout: 404920 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/client_trim_caps/% 2024-03-27T21:42:44.718 INFO:teuthology.orchestra.run.smithi060.stdout: 404928 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/client_trim_caps/objectstore 2024-03-27T21:42:44.718 INFO:teuthology.orchestra.run.smithi060.stdout: 404929 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/client_trim_caps/objectstore/.qa -> ../.qa/ 2024-03-27T21:42:44.718 INFO:teuthology.orchestra.run.smithi060.stdout: 404930 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/client_trim_caps/objectstore/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.718 INFO:teuthology.orchestra.run.smithi060.stdout: 404921 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/client_trim_caps/.qa -> ../.qa/ 2024-03-27T21:42:44.718 INFO:teuthology.orchestra.run.smithi060.stdout: 404927 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/client_trim_caps/conf -> .qa/cephfs/conf 2024-03-27T21:42:44.718 INFO:teuthology.orchestra.run.smithi060.stdout: 404924 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/client_trim_caps/clusters 2024-03-27T21:42:44.718 INFO:teuthology.orchestra.run.smithi060.stdout: 404925 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/client_trim_caps/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.718 INFO:teuthology.orchestra.run.smithi060.stdout: 404926 4 -rw-r--r-- 1 ubuntu ubuntu 224 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/client_trim_caps/clusters/small-cluster.yaml 2024-03-27T21:42:44.719 INFO:teuthology.orchestra.run.smithi060.stdout: 404922 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/client_trim_caps/begin -> .qa/cephfs/begin/ 2024-03-27T21:42:44.719 INFO:teuthology.orchestra.run.smithi060.stdout: 404923 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/client_trim_caps/centos_latest.yaml -> .qa/distros/supported/centos_latest.yaml 2024-03-27T21:42:44.719 INFO:teuthology.orchestra.run.smithi060.stdout: 404937 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/client_trim_caps/tasks 2024-03-27T21:42:44.719 INFO:teuthology.orchestra.run.smithi060.stdout: 404939 4 -rw-r--r-- 1 ubuntu ubuntu 412 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/client_trim_caps/tasks/trim-i24137.yaml 2024-03-27T21:42:44.719 INFO:teuthology.orchestra.run.smithi060.stdout: 404938 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/bugs/client_trim_caps/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.719 INFO:teuthology.orchestra.run.smithi060.stdout: 404893 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits 2024-03-27T21:42:44.719 INFO:teuthology.orchestra.run.smithi060.stdout: 404906 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/overrides 2024-03-27T21:42:44.719 INFO:teuthology.orchestra.run.smithi060.stdout: 404907 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/overrides/+ 2024-03-27T21:42:44.720 INFO:teuthology.orchestra.run.smithi060.stdout: 404908 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.720 INFO:teuthology.orchestra.run.smithi060.stdout: 404909 4 -rw-r--r-- 1 ubuntu ubuntu 79 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/overrides/faked-ino.yaml 2024-03-27T21:42:44.720 INFO:teuthology.orchestra.run.smithi060.stdout: 404912 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/overrides/pg_health.yaml -> .qa/cephfs/overrides/pg_health.yaml 2024-03-27T21:42:44.720 INFO:teuthology.orchestra.run.smithi060.stdout: 404910 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.720 INFO:teuthology.orchestra.run.smithi060.stdout: 404911 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.720 INFO:teuthology.orchestra.run.smithi060.stdout: 404894 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/% 2024-03-27T21:42:44.720 INFO:teuthology.orchestra.run.smithi060.stdout: 404895 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/.qa -> ../.qa/ 2024-03-27T21:42:44.720 INFO:teuthology.orchestra.run.smithi060.stdout: 404900 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/conf -> .qa/cephfs/conf 2024-03-27T21:42:44.721 INFO:teuthology.orchestra.run.smithi060.stdout: 404902 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/mount 2024-03-27T21:42:44.721 INFO:teuthology.orchestra.run.smithi060.stdout: 404903 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/mount/.qa -> ../.qa/ 2024-03-27T21:42:44.721 INFO:teuthology.orchestra.run.smithi060.stdout: 404904 0 lrwxrwxrwx 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/mount/fuse.yaml -> .qa/cephfs/mount/fuse.yaml 2024-03-27T21:42:44.721 INFO:teuthology.orchestra.run.smithi060.stdout: 404905 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/objectstore-ec -> .qa/cephfs/objectstore-ec 2024-03-27T21:42:44.721 INFO:teuthology.orchestra.run.smithi060.stdout: 404897 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/clusters 2024-03-27T21:42:44.721 INFO:teuthology.orchestra.run.smithi060.stdout: 404898 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.721 INFO:teuthology.orchestra.run.smithi060.stdout: 404899 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/clusters/fixed-2-ucephfs.yaml -> .qa/cephfs/clusters/fixed-2-ucephfs.yaml 2024-03-27T21:42:44.721 INFO:teuthology.orchestra.run.smithi060.stdout: 404896 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/begin -> .qa/cephfs/begin/ 2024-03-27T21:42:44.721 INFO:teuthology.orchestra.run.smithi060.stdout: 404913 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/tasks 2024-03-27T21:42:44.722 INFO:teuthology.orchestra.run.smithi060.stdout: 404914 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.722 INFO:teuthology.orchestra.run.smithi060.stdout: 404915 0 lrwxrwxrwx 1 ubuntu ubuntu 52 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/tasks/cfuse_workunit_suites_fsstress.yaml -> .qa/cephfs/tasks/cfuse_workunit_suites_fsstress.yaml 2024-03-27T21:42:44.722 INFO:teuthology.orchestra.run.smithi060.stdout: 404916 4 -rw-r--r-- 1 ubuntu ubuntu 161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/tasks/cfuse_workunit_suites_pjd.yaml 2024-03-27T21:42:44.722 INFO:teuthology.orchestra.run.smithi060.stdout: 404901 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/32bits/distro -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.722 INFO:teuthology.orchestra.run.smithi060.stdout: 404964 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt 2024-03-27T21:42:44.722 INFO:teuthology.orchestra.run.smithi060.stdout: 404979 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/overrides 2024-03-27T21:42:44.722 INFO:teuthology.orchestra.run.smithi060.stdout: 404980 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/overrides/+ 2024-03-27T21:42:44.722 INFO:teuthology.orchestra.run.smithi060.stdout: 404981 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.723 INFO:teuthology.orchestra.run.smithi060.stdout: 404987 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/overrides/pg_health.yaml -> .qa/cephfs/overrides/pg_health.yaml 2024-03-27T21:42:44.723 INFO:teuthology.orchestra.run.smithi060.stdout: 404982 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.723 INFO:teuthology.orchestra.run.smithi060.stdout: 404985 4 -rw-r--r-- 1 ubuntu ubuntu 68 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/overrides/osd.yaml 2024-03-27T21:42:44.723 INFO:teuthology.orchestra.run.smithi060.stdout: 404984 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.723 INFO:teuthology.orchestra.run.smithi060.stdout: 404986 4 -rw-r--r-- 1 ubuntu ubuntu 78 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/overrides/pg-warn.yaml 2024-03-27T21:42:44.723 INFO:teuthology.orchestra.run.smithi060.stdout: 404983 4 -rw-r--r-- 1 ubuntu ubuntu 106 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/overrides/ignorelist_health_more.yaml 2024-03-27T21:42:44.723 INFO:teuthology.orchestra.run.smithi060.stdout: 404965 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/% 2024-03-27T21:42:44.723 INFO:teuthology.orchestra.run.smithi060.stdout: 404966 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/.qa -> ../.qa/ 2024-03-27T21:42:44.724 INFO:teuthology.orchestra.run.smithi060.stdout: 404972 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/conf -> .qa/cephfs/conf/ 2024-03-27T21:42:44.724 INFO:teuthology.orchestra.run.smithi060.stdout: 404968 0 lrwxrwxrwx 1 ubuntu ubuntu 47 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/bluestore-bitmap.yaml -> .qa/cephfs/objectstore-ec/bluestore-bitmap.yaml 2024-03-27T21:42:44.724 INFO:teuthology.orchestra.run.smithi060.stdout: 404978 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/mount -> .qa/cephfs/mount 2024-03-27T21:42:44.724 INFO:teuthology.orchestra.run.smithi060.stdout: 404969 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/clusters 2024-03-27T21:42:44.724 INFO:teuthology.orchestra.run.smithi060.stdout: 404970 0 lrwxrwxrwx 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/clusters/.qa -> ../.qa 2024-03-27T21:42:44.724 INFO:teuthology.orchestra.run.smithi060.stdout: 404971 0 lrwxrwxrwx 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/clusters/1-mds-1-client.yaml -> .qa/cephfs/clusters/1-mds-1-client.yaml 2024-03-27T21:42:44.724 INFO:teuthology.orchestra.run.smithi060.stdout: 404967 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/begin -> .qa/cephfs/begin/ 2024-03-27T21:42:44.724 INFO:teuthology.orchestra.run.smithi060.stdout: 404988 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/tasks 2024-03-27T21:42:44.725 INFO:teuthology.orchestra.run.smithi060.stdout: 404989 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/tasks/% 2024-03-27T21:42:44.725 INFO:teuthology.orchestra.run.smithi060.stdout: 404990 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.725 INFO:teuthology.orchestra.run.smithi060.stdout: 404991 4 -rw-r--r-- 1 ubuntu ubuntu 254 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/tasks/0-client.yaml 2024-03-27T21:42:44.725 INFO:teuthology.orchestra.run.smithi060.stdout: 404992 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/tasks/1-tests 2024-03-27T21:42:44.725 INFO:teuthology.orchestra.run.smithi060.stdout: 404994 4 -rw-r--r-- 1 ubuntu ubuntu 140 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/tasks/1-tests/fscrypt-dbench.yaml 2024-03-27T21:42:44.725 INFO:teuthology.orchestra.run.smithi060.stdout: 404997 4 -rw-r--r-- 1 ubuntu ubuntu 134 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/tasks/1-tests/fscrypt-pjd.yaml 2024-03-27T21:42:44.725 INFO:teuthology.orchestra.run.smithi060.stdout: 404996 4 -rw-r--r-- 1 ubuntu ubuntu 140 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/tasks/1-tests/fscrypt-iozone.yaml 2024-03-27T21:42:44.725 INFO:teuthology.orchestra.run.smithi060.stdout: 404995 4 -rw-r--r-- 1 ubuntu ubuntu 136 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/tasks/1-tests/fscrypt-ffsb.yaml 2024-03-27T21:42:44.726 INFO:teuthology.orchestra.run.smithi060.stdout: 404993 4 -rw-r--r-- 1 ubuntu ubuntu 108 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/tasks/1-tests/fscrypt-common.yaml 2024-03-27T21:42:44.726 INFO:teuthology.orchestra.run.smithi060.stdout: 404973 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/distro 2024-03-27T21:42:44.726 INFO:teuthology.orchestra.run.smithi060.stdout: 404975 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/distro/.qa -> ../.qa/ 2024-03-27T21:42:44.726 INFO:teuthology.orchestra.run.smithi060.stdout: 404977 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/distro/ubuntu_latest.yaml -> .qa/distros/supported/ubuntu_latest.yaml 2024-03-27T21:42:44.726 INFO:teuthology.orchestra.run.smithi060.stdout: 404976 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/distro/centos_latest.yaml -> .qa/distros/supported/centos_latest.yaml 2024-03-27T21:42:44.726 INFO:teuthology.orchestra.run.smithi060.stdout: 404974 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/fscrypt/distro/$ 2024-03-27T21:42:44.726 INFO:teuthology.orchestra.run.smithi060.stdout: 405237 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs 2024-03-27T21:42:44.726 INFO:teuthology.orchestra.run.smithi060.stdout: 405248 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs/overrides 2024-03-27T21:42:44.727 INFO:teuthology.orchestra.run.smithi060.stdout: 405249 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs/overrides/+ 2024-03-27T21:42:44.727 INFO:teuthology.orchestra.run.smithi060.stdout: 405250 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.727 INFO:teuthology.orchestra.run.smithi060.stdout: 405254 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs/overrides/pg_health.yaml -> .qa/cephfs/overrides/pg_health.yaml 2024-03-27T21:42:44.727 INFO:teuthology.orchestra.run.smithi060.stdout: 405251 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.727 INFO:teuthology.orchestra.run.smithi060.stdout: 405253 4 -rw-r--r-- 1 ubuntu ubuntu 62 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs/overrides/mon-debug.yaml 2024-03-27T21:42:44.727 INFO:teuthology.orchestra.run.smithi060.stdout: 405252 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs/overrides/ignorelist_wrongly_marked_down.yaml -> .qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:44.727 INFO:teuthology.orchestra.run.smithi060.stdout: 405238 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs/% 2024-03-27T21:42:44.727 INFO:teuthology.orchestra.run.smithi060.stdout: 405239 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs/.qa -> ../.qa/ 2024-03-27T21:42:44.728 INFO:teuthology.orchestra.run.smithi060.stdout: 405244 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs/conf -> .qa/cephfs/conf 2024-03-27T21:42:44.728 INFO:teuthology.orchestra.run.smithi060.stdout: 405246 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs/mount -> .qa/cephfs/mount/ 2024-03-27T21:42:44.728 INFO:teuthology.orchestra.run.smithi060.stdout: 405247 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs/objectstore-ec -> .qa/cephfs/objectstore-ec 2024-03-27T21:42:44.728 INFO:teuthology.orchestra.run.smithi060.stdout: 405241 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs/clusters 2024-03-27T21:42:44.728 INFO:teuthology.orchestra.run.smithi060.stdout: 405242 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.728 INFO:teuthology.orchestra.run.smithi060.stdout: 405243 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs/clusters/1a3s-mds-2c-client.yaml -> .qa/cephfs/clusters/1a3s-mds-2c-client.yaml 2024-03-27T21:42:44.728 INFO:teuthology.orchestra.run.smithi060.stdout: 405240 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs/begin -> .qa/cephfs/begin/ 2024-03-27T21:42:44.728 INFO:teuthology.orchestra.run.smithi060.stdout: 405255 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs/tasks 2024-03-27T21:42:44.729 INFO:teuthology.orchestra.run.smithi060.stdout: 405257 4 -rw-r--r-- 1 ubuntu ubuntu 438 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs/tasks/failover.yaml 2024-03-27T21:42:44.729 INFO:teuthology.orchestra.run.smithi060.stdout: 405256 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.729 INFO:teuthology.orchestra.run.smithi060.stdout: 405258 4 -rw-r--r-- 1 ubuntu ubuntu 113 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs/tasks/multifs-auth.yaml 2024-03-27T21:42:44.729 INFO:teuthology.orchestra.run.smithi060.stdout: 405245 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/multifs/distro -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.729 INFO:teuthology.orchestra.run.smithi060.stdout: 405259 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/nfs 2024-03-27T21:42:44.729 INFO:teuthology.orchestra.run.smithi060.stdout: 405267 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/nfs/overrides 2024-03-27T21:42:44.729 INFO:teuthology.orchestra.run.smithi060.stdout: 405268 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/nfs/overrides/+ 2024-03-27T21:42:44.729 INFO:teuthology.orchestra.run.smithi060.stdout: 405269 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/nfs/overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.730 INFO:teuthology.orchestra.run.smithi060.stdout: 405271 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/nfs/overrides/pg_health.yaml -> .qa/cephfs/overrides/pg_health.yaml 2024-03-27T21:42:44.730 INFO:teuthology.orchestra.run.smithi060.stdout: 405270 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/nfs/overrides/ignorelist_health.yaml -> .qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:44.730 INFO:teuthology.orchestra.run.smithi060.stdout: 405260 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/nfs/% 2024-03-27T21:42:44.730 INFO:teuthology.orchestra.run.smithi060.stdout: 405261 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/nfs/.qa -> ../.qa/ 2024-03-27T21:42:44.730 INFO:teuthology.orchestra.run.smithi060.stdout: 405266 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/nfs/conf -> ./.qa/cephfs/conf 2024-03-27T21:42:44.730 INFO:teuthology.orchestra.run.smithi060.stdout: 405272 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/nfs/supported-random-distros$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.730 INFO:teuthology.orchestra.run.smithi060.stdout: 405273 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/nfs/tasks 2024-03-27T21:42:44.730 INFO:teuthology.orchestra.run.smithi060.stdout: 405274 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/nfs/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.731 INFO:teuthology.orchestra.run.smithi060.stdout: 405275 4 -rw-r--r-- 1 ubuntu ubuntu 78 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/nfs/tasks/nfs.yaml 2024-03-27T21:42:44.731 INFO:teuthology.orchestra.run.smithi060.stdout: 405262 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/nfs/cluster 2024-03-27T21:42:44.731 INFO:teuthology.orchestra.run.smithi060.stdout: 405263 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/nfs/cluster/+ 2024-03-27T21:42:44.731 INFO:teuthology.orchestra.run.smithi060.stdout: 405264 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/nfs/cluster/.qa -> ../.qa/ 2024-03-27T21:42:44.731 INFO:teuthology.orchestra.run.smithi060.stdout: 405265 4 -rw-r--r-- 1 ubuntu ubuntu 241 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/fs/nfs/cluster/1-node.yaml 2024-03-27T21:42:44.731 INFO:teuthology.orchestra.run.smithi060.stdout: 406505 4 drwxr-xr-x 26 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados 2024-03-27T21:42:44.731 INFO:teuthology.orchestra.run.smithi060.stdout: 406899 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec 2024-03-27T21:42:44.731 INFO:teuthology.orchestra.run.smithi060.stdout: 406902 0 lrwxrwxrwx 1 ubuntu ubuntu 19 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/ceph.yaml -> ../thrash/ceph.yaml 2024-03-27T21:42:44.731 INFO:teuthology.orchestra.run.smithi060.stdout: 406900 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/% 2024-03-27T21:42:44.732 INFO:teuthology.orchestra.run.smithi060.stdout: 406910 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/objectstore -> .qa/objectstore_debug 2024-03-27T21:42:44.732 INFO:teuthology.orchestra.run.smithi060.stdout: 406909 0 lrwxrwxrwx 1 ubuntu ubuntu 23 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/msgr-failures -> ../thrash/msgr-failures 2024-03-27T21:42:44.732 INFO:teuthology.orchestra.run.smithi060.stdout: 406901 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/.qa -> ../.qa/ 2024-03-27T21:42:44.732 INFO:teuthology.orchestra.run.smithi060.stdout: 406918 0 lrwxrwxrwx 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/thrashosds-health.yaml -> .qa/tasks/thrashosds-health.yaml 2024-03-27T21:42:44.732 INFO:teuthology.orchestra.run.smithi060.stdout: 406913 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/supported-random-distro$ -> ../basic/supported-random-distro$ 2024-03-27T21:42:44.732 INFO:teuthology.orchestra.run.smithi060.stdout: 406919 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/workloads 2024-03-27T21:42:44.732 INFO:teuthology.orchestra.run.smithi060.stdout: 406920 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.732 INFO:teuthology.orchestra.run.smithi060.stdout: 406921 0 lrwxrwxrwx 1 ubuntu ubuntu 54 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml -> .qa/erasure-code/ec-rados-plugin=shec-k=4-m=3-c=2.yaml 2024-03-27T21:42:44.733 INFO:teuthology.orchestra.run.smithi060.stdout: 406903 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/clusters 2024-03-27T21:42:44.733 INFO:teuthology.orchestra.run.smithi060.stdout: 406904 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/clusters/+ 2024-03-27T21:42:44.733 INFO:teuthology.orchestra.run.smithi060.stdout: 406905 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.733 INFO:teuthology.orchestra.run.smithi060.stdout: 406907 4 -rw-r--r-- 1 ubuntu ubuntu 87 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/clusters/openstack.yaml 2024-03-27T21:42:44.733 INFO:teuthology.orchestra.run.smithi060.stdout: 406906 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/clusters/fixed-4.yaml -> .qa/clusters/fixed-4.yaml 2024-03-27T21:42:44.733 INFO:teuthology.orchestra.run.smithi060.stdout: 406911 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/rados.yaml -> .qa/config/rados.yaml 2024-03-27T21:42:44.733 INFO:teuthology.orchestra.run.smithi060.stdout: 406912 0 lrwxrwxrwx 1 ubuntu ubuntu 30 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/recovery-overrides -> ../thrash/2-recovery-overrides 2024-03-27T21:42:44.733 INFO:teuthology.orchestra.run.smithi060.stdout: 406914 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/thrashers 2024-03-27T21:42:44.733 INFO:teuthology.orchestra.run.smithi060.stdout: 406915 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/thrashers/.qa -> ../.qa/ 2024-03-27T21:42:44.734 INFO:teuthology.orchestra.run.smithi060.stdout: 406916 4 -rw-r--r-- 1 ubuntu ubuntu 499 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/thrashers/careful.yaml 2024-03-27T21:42:44.734 INFO:teuthology.orchestra.run.smithi060.stdout: 406917 4 -rw-r--r-- 1 ubuntu ubuntu 529 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/thrashers/default.yaml 2024-03-27T21:42:44.734 INFO:teuthology.orchestra.run.smithi060.stdout: 406908 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-shec/mon_election -> .qa/mon_election 2024-03-27T21:42:44.734 INFO:teuthology.orchestra.run.smithi060.stdout: 406617 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon 2024-03-27T21:42:44.734 INFO:teuthology.orchestra.run.smithi060.stdout: 406627 0 lrwxrwxrwx 1 ubuntu ubuntu 8 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/msgr -> .qa/msgr 2024-03-27T21:42:44.734 INFO:teuthology.orchestra.run.smithi060.stdout: 406618 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/% 2024-03-27T21:42:44.734 INFO:teuthology.orchestra.run.smithi060.stdout: 406633 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/objectstore -> .qa/objectstore_debug 2024-03-27T21:42:44.734 INFO:teuthology.orchestra.run.smithi060.stdout: 406628 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/msgr-failures 2024-03-27T21:42:44.735 INFO:teuthology.orchestra.run.smithi060.stdout: 406629 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.735 INFO:teuthology.orchestra.run.smithi060.stdout: 406630 4 -rw-r--r-- 1 ubuntu ubuntu 177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/msgr-failures/few.yaml 2024-03-27T21:42:44.735 INFO:teuthology.orchestra.run.smithi060.stdout: 406631 4 -rw-r--r-- 1 ubuntu ubuntu 210 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/msgr-failures/many.yaml 2024-03-27T21:42:44.735 INFO:teuthology.orchestra.run.smithi060.stdout: 406619 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/.qa -> ../.qa/ 2024-03-27T21:42:44.735 INFO:teuthology.orchestra.run.smithi060.stdout: 406635 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/supported-random-distro$ -> ../basic/supported-random-distro$ 2024-03-27T21:42:44.735 INFO:teuthology.orchestra.run.smithi060.stdout: 406620 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/clusters 2024-03-27T21:42:44.735 INFO:teuthology.orchestra.run.smithi060.stdout: 406621 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.735 INFO:teuthology.orchestra.run.smithi060.stdout: 406623 4 -rw-r--r-- 1 ubuntu ubuntu 122 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/clusters/3.yaml 2024-03-27T21:42:44.736 INFO:teuthology.orchestra.run.smithi060.stdout: 406624 4 -rw-r--r-- 1 ubuntu ubuntu 150 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/clusters/6.yaml 2024-03-27T21:42:44.736 INFO:teuthology.orchestra.run.smithi060.stdout: 406622 4 -rw-r--r-- 1 ubuntu ubuntu 251 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/clusters/21.yaml 2024-03-27T21:42:44.736 INFO:teuthology.orchestra.run.smithi060.stdout: 406625 4 -rw-r--r-- 1 ubuntu ubuntu 167 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/clusters/9.yaml 2024-03-27T21:42:44.736 INFO:teuthology.orchestra.run.smithi060.stdout: 406634 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/rados.yaml -> .qa/config/rados.yaml 2024-03-27T21:42:44.736 INFO:teuthology.orchestra.run.smithi060.stdout: 406632 4 -rw-r--r-- 1 ubuntu ubuntu 124 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/no_pools.yaml 2024-03-27T21:42:44.736 INFO:teuthology.orchestra.run.smithi060.stdout: 406636 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/tasks 2024-03-27T21:42:44.736 INFO:teuthology.orchestra.run.smithi060.stdout: 406639 4 -rw-r--r-- 1 ubuntu ubuntu 639 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/tasks/mon_clock_with_skews.yaml 2024-03-27T21:42:44.736 INFO:teuthology.orchestra.run.smithi060.stdout: 406637 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.737 INFO:teuthology.orchestra.run.smithi060.stdout: 406640 4 -rw-r--r-- 1 ubuntu ubuntu 210 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/tasks/mon_recovery.yaml 2024-03-27T21:42:44.737 INFO:teuthology.orchestra.run.smithi060.stdout: 406638 4 -rw-r--r-- 1 ubuntu ubuntu 242 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/tasks/mon_clock_no_skews.yaml 2024-03-27T21:42:44.737 INFO:teuthology.orchestra.run.smithi060.stdout: 406626 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/multimon/mon_election -> .qa/mon_election 2024-03-27T21:42:44.737 INFO:teuthology.orchestra.run.smithi060.stdout: 406691 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/rest 2024-03-27T21:42:44.737 INFO:teuthology.orchestra.run.smithi060.stdout: 406692 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/rest/% 2024-03-27T21:42:44.737 INFO:teuthology.orchestra.run.smithi060.stdout: 406693 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/rest/.qa -> ../.qa/ 2024-03-27T21:42:44.737 INFO:teuthology.orchestra.run.smithi060.stdout: 406694 4 -rw-r--r-- 1 ubuntu ubuntu 657 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/rest/mgr-restful.yaml 2024-03-27T21:42:44.737 INFO:teuthology.orchestra.run.smithi060.stdout: 406695 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/rest/supported-random-distro$ -> ../basic/supported-random-distro$ 2024-03-27T21:42:44.737 INFO:teuthology.orchestra.run.smithi060.stdout: 406558 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr 2024-03-27T21:42:44.738 INFO:teuthology.orchestra.run.smithi060.stdout: 406573 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/random-objectstore$ -> .qa/objectstore_debug 2024-03-27T21:42:44.738 INFO:teuthology.orchestra.run.smithi060.stdout: 406559 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/% 2024-03-27T21:42:44.738 INFO:teuthology.orchestra.run.smithi060.stdout: 406560 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/.qa -> ../.qa/ 2024-03-27T21:42:44.738 INFO:teuthology.orchestra.run.smithi060.stdout: 406568 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/mgr_ttl_cache 2024-03-27T21:42:44.738 INFO:teuthology.orchestra.run.smithi060.stdout: 406569 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/mgr_ttl_cache/.qa -> ../.qa/ 2024-03-27T21:42:44.738 INFO:teuthology.orchestra.run.smithi060.stdout: 406571 0 lrwxrwxrwx 1 ubuntu ubuntu 29 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/mgr_ttl_cache/enable.yaml -> .qa/mgr_ttl_cache/enable.yaml 2024-03-27T21:42:44.738 INFO:teuthology.orchestra.run.smithi060.stdout: 406570 0 lrwxrwxrwx 1 ubuntu ubuntu 30 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/mgr_ttl_cache/disable.yaml -> .qa/mgr_ttl_cache/disable.yaml 2024-03-27T21:42:44.738 INFO:teuthology.orchestra.run.smithi060.stdout: 406565 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/debug 2024-03-27T21:42:44.739 INFO:teuthology.orchestra.run.smithi060.stdout: 406566 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/debug/.qa -> ../.qa/ 2024-03-27T21:42:44.739 INFO:teuthology.orchestra.run.smithi060.stdout: 406567 0 lrwxrwxrwx 1 ubuntu ubuntu 18 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/debug/mgr.yaml -> .qa/debug/mgr.yaml 2024-03-27T21:42:44.739 INFO:teuthology.orchestra.run.smithi060.stdout: 406574 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/supported-random-distro$ -> ../basic/supported-random-distro$ 2024-03-27T21:42:44.739 INFO:teuthology.orchestra.run.smithi060.stdout: 406561 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/clusters 2024-03-27T21:42:44.739 INFO:teuthology.orchestra.run.smithi060.stdout: 406562 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/clusters/+ 2024-03-27T21:42:44.739 INFO:teuthology.orchestra.run.smithi060.stdout: 406563 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.739 INFO:teuthology.orchestra.run.smithi060.stdout: 406564 0 lrwxrwxrwx 1 ubuntu ubuntu 28 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/clusters/2-node-mgr.yaml -> .qa/clusters/2-node-mgr.yaml 2024-03-27T21:42:44.739 INFO:teuthology.orchestra.run.smithi060.stdout: 406575 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/tasks 2024-03-27T21:42:44.740 INFO:teuthology.orchestra.run.smithi060.stdout: 406578 4 -rw-r--r-- 1 ubuntu ubuntu 446 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/tasks/failover.yaml 2024-03-27T21:42:44.740 INFO:teuthology.orchestra.run.smithi060.stdout: 406583 4 -rw-r--r-- 1 ubuntu ubuntu 448 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/tasks/prometheus.yaml 2024-03-27T21:42:44.740 INFO:teuthology.orchestra.run.smithi060.stdout: 406576 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.740 INFO:teuthology.orchestra.run.smithi060.stdout: 406582 4 -rw-r--r-- 1 ubuntu ubuntu 785 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/tasks/progress.yaml 2024-03-27T21:42:44.740 INFO:teuthology.orchestra.run.smithi060.stdout: 406577 4 -rw-r--r-- 1 ubuntu ubuntu 470 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/tasks/crash.yaml 2024-03-27T21:42:44.740 INFO:teuthology.orchestra.run.smithi060.stdout: 406581 4 -rw-r--r-- 1 ubuntu ubuntu 1596 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/tasks/per_module_finisher_stats.yaml 2024-03-27T21:42:44.740 INFO:teuthology.orchestra.run.smithi060.stdout: 406584 4 -rw-r--r-- 1 ubuntu ubuntu 453 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/tasks/workunits.yaml 2024-03-27T21:42:44.740 INFO:teuthology.orchestra.run.smithi060.stdout: 406580 4 -rw-r--r-- 1 ubuntu ubuntu 849 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/tasks/module_selftest.yaml 2024-03-27T21:42:44.740 INFO:teuthology.orchestra.run.smithi060.stdout: 406579 4 -rw-r--r-- 1 ubuntu ubuntu 542 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/tasks/insights.yaml 2024-03-27T21:42:44.741 INFO:teuthology.orchestra.run.smithi060.stdout: 406572 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/mgr/mon_election -> .qa/mon_election 2024-03-27T21:42:44.741 INFO:teuthology.orchestra.run.smithi060.stdout: 406641 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/objectstore 2024-03-27T21:42:44.741 INFO:teuthology.orchestra.run.smithi060.stdout: 406642 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/objectstore/% 2024-03-27T21:42:44.741 INFO:teuthology.orchestra.run.smithi060.stdout: 406643 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/objectstore/.qa -> ../.qa/ 2024-03-27T21:42:44.741 INFO:teuthology.orchestra.run.smithi060.stdout: 406653 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/objectstore/supported-random-distro$ -> ../basic/supported-random-distro$ 2024-03-27T21:42:44.741 INFO:teuthology.orchestra.run.smithi060.stdout: 406644 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/objectstore/backends 2024-03-27T21:42:44.741 INFO:teuthology.orchestra.run.smithi060.stdout: 406647 4 -rw-r--r-- 1 ubuntu ubuntu 138 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/objectstore/backends/fusestore.yaml 2024-03-27T21:42:44.741 INFO:teuthology.orchestra.run.smithi060.stdout: 406645 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/objectstore/backends/.qa -> ../.qa/ 2024-03-27T21:42:44.742 INFO:teuthology.orchestra.run.smithi060.stdout: 406649 4 -rw-r--r-- 1 ubuntu ubuntu 292 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/objectstore/backends/objectcacher-stress.yaml 2024-03-27T21:42:44.742 INFO:teuthology.orchestra.run.smithi060.stdout: 406652 4 -rw-r--r-- 1 ubuntu ubuntu 473 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/objectstore/backends/objectstore-memstore.yaml 2024-03-27T21:42:44.742 INFO:teuthology.orchestra.run.smithi060.stdout: 406648 4 -rw-r--r-- 1 ubuntu ubuntu 195 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/objectstore/backends/keyvaluedb.yaml 2024-03-27T21:42:44.742 INFO:teuthology.orchestra.run.smithi060.stdout: 406650 4 -rw-r--r-- 1 ubuntu ubuntu 493 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/objectstore/backends/objectstore-bluestore-a.yaml 2024-03-27T21:42:44.742 INFO:teuthology.orchestra.run.smithi060.stdout: 406646 4 -rw-r--r-- 1 ubuntu ubuntu 565 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/objectstore/backends/ceph_objectstore_tool.yaml 2024-03-27T21:42:44.742 INFO:teuthology.orchestra.run.smithi060.stdout: 406651 4 -rw-r--r-- 1 ubuntu ubuntu 490 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/objectstore/backends/objectstore-bluestore-b.yaml 2024-03-27T21:42:44.742 INFO:teuthology.orchestra.run.smithi060.stdout: 406506 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/.qa -> ../.qa/ 2024-03-27T21:42:44.742 INFO:teuthology.orchestra.run.smithi060.stdout: 406545 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/dashboard 2024-03-27T21:42:44.743 INFO:teuthology.orchestra.run.smithi060.stdout: 406553 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/dashboard/random-objectstore$ -> .qa/objectstore 2024-03-27T21:42:44.743 INFO:teuthology.orchestra.run.smithi060.stdout: 406546 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/dashboard/% 2024-03-27T21:42:44.743 INFO:teuthology.orchestra.run.smithi060.stdout: 406547 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/dashboard/.qa -> ../.qa/ 2024-03-27T21:42:44.743 INFO:teuthology.orchestra.run.smithi060.stdout: 406549 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/dashboard/debug 2024-03-27T21:42:44.743 INFO:teuthology.orchestra.run.smithi060.stdout: 406550 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/dashboard/debug/.qa -> ../.qa/ 2024-03-27T21:42:44.743 INFO:teuthology.orchestra.run.smithi060.stdout: 406551 0 lrwxrwxrwx 1 ubuntu ubuntu 18 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/dashboard/debug/mgr.yaml -> .qa/debug/mgr.yaml 2024-03-27T21:42:44.743 INFO:teuthology.orchestra.run.smithi060.stdout: 406548 0 lrwxrwxrwx 1 ubuntu ubuntu 38 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/dashboard/0-single-container-host.yaml -> .qa/distros/single-container-host.yaml 2024-03-27T21:42:44.743 INFO:teuthology.orchestra.run.smithi060.stdout: 406554 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/dashboard/tasks 2024-03-27T21:42:44.744 INFO:teuthology.orchestra.run.smithi060.stdout: 406555 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/dashboard/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.744 INFO:teuthology.orchestra.run.smithi060.stdout: 406557 4 -rw-r--r-- 1 ubuntu ubuntu 447 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/dashboard/tasks/e2e.yaml 2024-03-27T21:42:44.744 INFO:teuthology.orchestra.run.smithi060.stdout: 406556 4 -rw-r--r-- 1 ubuntu ubuntu 2406 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/dashboard/tasks/dashboard.yaml 2024-03-27T21:42:44.744 INFO:teuthology.orchestra.run.smithi060.stdout: 406552 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/dashboard/mon_election -> .qa/mon_election 2024-03-27T21:42:44.744 INFO:teuthology.orchestra.run.smithi060.stdout: 406879 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-overwrites 2024-03-27T21:42:44.744 INFO:teuthology.orchestra.run.smithi060.stdout: 406885 0 lrwxrwxrwx 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-overwrites/fast -> ../thrash-erasure-code/fast 2024-03-27T21:42:44.744 INFO:teuthology.orchestra.run.smithi060.stdout: 406883 0 lrwxrwxrwx 1 ubuntu ubuntu 19 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-overwrites/ceph.yaml -> ../thrash/ceph.yaml 2024-03-27T21:42:44.744 INFO:teuthology.orchestra.run.smithi060.stdout: 406880 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-overwrites/% 2024-03-27T21:42:44.745 INFO:teuthology.orchestra.run.smithi060.stdout: 406887 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-overwrites/msgr-failures -> ../thrash-erasure-code/msgr-failures 2024-03-27T21:42:44.745 INFO:teuthology.orchestra.run.smithi060.stdout: 406881 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-overwrites/.qa -> ../.qa/ 2024-03-27T21:42:44.745 INFO:teuthology.orchestra.run.smithi060.stdout: 406882 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-overwrites/bluestore-bitmap.yaml -> ../thrash-erasure-code/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.745 INFO:teuthology.orchestra.run.smithi060.stdout: 406892 0 lrwxrwxrwx 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-overwrites/thrashosds-health.yaml -> .qa/tasks/thrashosds-health.yaml 2024-03-27T21:42:44.745 INFO:teuthology.orchestra.run.smithi060.stdout: 406890 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-overwrites/supported-random-distro$ -> ../basic/supported-random-distro$ 2024-03-27T21:42:44.745 INFO:teuthology.orchestra.run.smithi060.stdout: 406893 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-overwrites/workloads 2024-03-27T21:42:44.745 INFO:teuthology.orchestra.run.smithi060.stdout: 406894 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-overwrites/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.745 INFO:teuthology.orchestra.run.smithi060.stdout: 406898 4 -rw-r--r-- 1 ubuntu ubuntu 445 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-overwrites/workloads/ec-snaps-few-objects-overwrites.yaml 2024-03-27T21:42:44.746 INFO:teuthology.orchestra.run.smithi060.stdout: 406897 4 -rw-r--r-- 1 ubuntu ubuntu 561 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-overwrites/workloads/ec-small-objects-overwrites.yaml 2024-03-27T21:42:44.746 INFO:teuthology.orchestra.run.smithi060.stdout: 406896 4 -rw-r--r-- 1 ubuntu ubuntu 581 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-overwrites/workloads/ec-small-objects-fast-read-overwrites.yaml 2024-03-27T21:42:44.746 INFO:teuthology.orchestra.run.smithi060.stdout: 406895 4 -rw-r--r-- 1 ubuntu ubuntu 466 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-overwrites/workloads/ec-pool-snaps-few-objects-overwrites.yaml 2024-03-27T21:42:44.746 INFO:teuthology.orchestra.run.smithi060.stdout: 406884 0 lrwxrwxrwx 1 ubuntu ubuntu 31 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-overwrites/clusters -> ../thrash-erasure-code/clusters 2024-03-27T21:42:44.746 INFO:teuthology.orchestra.run.smithi060.stdout: 406888 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-overwrites/rados.yaml -> ../thrash-erasure-code/rados.yaml 2024-03-27T21:42:44.746 INFO:teuthology.orchestra.run.smithi060.stdout: 406889 0 lrwxrwxrwx 1 ubuntu ubuntu 30 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-overwrites/recovery-overrides -> ../thrash/2-recovery-overrides 2024-03-27T21:42:44.746 INFO:teuthology.orchestra.run.smithi060.stdout: 406891 0 lrwxrwxrwx 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-overwrites/thrashers -> ../thrash-erasure-code/thrashers 2024-03-27T21:42:44.746 INFO:teuthology.orchestra.run.smithi060.stdout: 406886 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-overwrites/mon_election -> .qa/mon_election 2024-03-27T21:42:44.747 INFO:teuthology.orchestra.run.smithi060.stdout: 406585 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash 2024-03-27T21:42:44.747 INFO:teuthology.orchestra.run.smithi060.stdout: 406594 0 lrwxrwxrwx 1 ubuntu ubuntu 8 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/msgr -> .qa/msgr 2024-03-27T21:42:44.747 INFO:teuthology.orchestra.run.smithi060.stdout: 406588 4 -rw-r--r-- 1 ubuntu ubuntu 727 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/ceph.yaml 2024-03-27T21:42:44.747 INFO:teuthology.orchestra.run.smithi060.stdout: 406586 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/% 2024-03-27T21:42:44.747 INFO:teuthology.orchestra.run.smithi060.stdout: 406599 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/objectstore -> .qa/objectstore_debug 2024-03-27T21:42:44.747 INFO:teuthology.orchestra.run.smithi060.stdout: 406595 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/msgr-failures 2024-03-27T21:42:44.747 INFO:teuthology.orchestra.run.smithi060.stdout: 406598 4 -rw-r--r-- 1 ubuntu ubuntu 358 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/msgr-failures/mon-delay.yaml 2024-03-27T21:42:44.747 INFO:teuthology.orchestra.run.smithi060.stdout: 406596 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.747 INFO:teuthology.orchestra.run.smithi060.stdout: 406597 4 -rw-r--r-- 1 ubuntu ubuntu 177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/msgr-failures/few.yaml 2024-03-27T21:42:44.748 INFO:teuthology.orchestra.run.smithi060.stdout: 406587 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/.qa -> ../.qa/ 2024-03-27T21:42:44.748 INFO:teuthology.orchestra.run.smithi060.stdout: 406601 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/supported-random-distro$ -> ../basic/supported-random-distro$ 2024-03-27T21:42:44.748 INFO:teuthology.orchestra.run.smithi060.stdout: 406609 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/workloads 2024-03-27T21:42:44.748 INFO:teuthology.orchestra.run.smithi060.stdout: 406610 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.748 INFO:teuthology.orchestra.run.smithi060.stdout: 406615 4 -rw-r--r-- 1 ubuntu ubuntu 291 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/workloads/rados_mon_workunits.yaml 2024-03-27T21:42:44.748 INFO:teuthology.orchestra.run.smithi060.stdout: 406616 4 -rw-r--r-- 1 ubuntu ubuntu 219 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/workloads/snaps-few-objects.yaml 2024-03-27T21:42:44.748 INFO:teuthology.orchestra.run.smithi060.stdout: 406611 4 -rw-r--r-- 1 ubuntu ubuntu 2367 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/workloads/pool-create-delete.yaml 2024-03-27T21:42:44.748 INFO:teuthology.orchestra.run.smithi060.stdout: 406614 4 -rw-r--r-- 1 ubuntu ubuntu 624 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/workloads/rados_mon_osdmap_prune.yaml 2024-03-27T21:42:44.749 INFO:teuthology.orchestra.run.smithi060.stdout: 406613 4 -rw-r--r-- 1 ubuntu ubuntu 591 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/workloads/rados_api_tests.yaml 2024-03-27T21:42:44.749 INFO:teuthology.orchestra.run.smithi060.stdout: 406612 4 -rw-r--r-- 1 ubuntu ubuntu 206 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/workloads/rados_5925.yaml 2024-03-27T21:42:44.749 INFO:teuthology.orchestra.run.smithi060.stdout: 406589 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/clusters 2024-03-27T21:42:44.749 INFO:teuthology.orchestra.run.smithi060.stdout: 406591 4 -rw-r--r-- 1 ubuntu ubuntu 180 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/clusters/3-mons.yaml 2024-03-27T21:42:44.749 INFO:teuthology.orchestra.run.smithi060.stdout: 406590 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.749 INFO:teuthology.orchestra.run.smithi060.stdout: 406592 4 -rw-r--r-- 1 ubuntu ubuntu 222 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/clusters/9-mons.yaml 2024-03-27T21:42:44.749 INFO:teuthology.orchestra.run.smithi060.stdout: 406600 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/rados.yaml -> .qa/config/rados.yaml 2024-03-27T21:42:44.749 INFO:teuthology.orchestra.run.smithi060.stdout: 406602 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/thrashers 2024-03-27T21:42:44.749 INFO:teuthology.orchestra.run.smithi060.stdout: 406603 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/thrashers/.qa -> ../.qa/ 2024-03-27T21:42:44.750 INFO:teuthology.orchestra.run.smithi060.stdout: 406606 4 -rw-r--r-- 1 ubuntu ubuntu 179 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/thrashers/one.yaml 2024-03-27T21:42:44.750 INFO:teuthology.orchestra.run.smithi060.stdout: 406604 4 -rw-r--r-- 1 ubuntu ubuntu 248 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/thrashers/force-sync-many.yaml 2024-03-27T21:42:44.750 INFO:teuthology.orchestra.run.smithi060.stdout: 406607 4 -rw-r--r-- 1 ubuntu ubuntu 271 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/thrashers/sync-many.yaml 2024-03-27T21:42:44.750 INFO:teuthology.orchestra.run.smithi060.stdout: 406605 4 -rw-r--r-- 1 ubuntu ubuntu 353 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/thrashers/many.yaml 2024-03-27T21:42:44.750 INFO:teuthology.orchestra.run.smithi060.stdout: 406608 4 -rw-r--r-- 1 ubuntu ubuntu 249 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/thrashers/sync.yaml 2024-03-27T21:42:44.750 INFO:teuthology.orchestra.run.smithi060.stdout: 406593 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/monthrash/mon_election -> .qa/mon_election 2024-03-27T21:42:44.750 INFO:teuthology.orchestra.run.smithi060.stdout: 406654 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf 2024-03-27T21:42:44.750 INFO:teuthology.orchestra.run.smithi060.stdout: 406657 4 -rw-r--r-- 1 ubuntu ubuntu 334 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/ceph.yaml 2024-03-27T21:42:44.751 INFO:teuthology.orchestra.run.smithi060.stdout: 406655 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/% 2024-03-27T21:42:44.751 INFO:teuthology.orchestra.run.smithi060.stdout: 406659 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/objectstore 2024-03-27T21:42:44.751 INFO:teuthology.orchestra.run.smithi060.stdout: 406664 4 -rw-r--r-- 1 ubuntu ubuntu 801 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/objectstore/bluestore-low-osd-mem-target.yaml 2024-03-27T21:42:44.751 INFO:teuthology.orchestra.run.smithi060.stdout: 406661 4 -rw-r--r-- 1 ubuntu ubuntu 807 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/objectstore/bluestore-basic-min-osd-mem-target.yaml 2024-03-27T21:42:44.751 INFO:teuthology.orchestra.run.smithi060.stdout: 406660 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/objectstore/.qa -> ../.qa/ 2024-03-27T21:42:44.751 INFO:teuthology.orchestra.run.smithi060.stdout: 406662 4 -rw-r--r-- 1 ubuntu ubuntu 1327 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.751 INFO:teuthology.orchestra.run.smithi060.stdout: 406665 4 -rw-r--r-- 1 ubuntu ubuntu 1327 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/objectstore/bluestore-stupid.yaml 2024-03-27T21:42:44.751 INFO:teuthology.orchestra.run.smithi060.stdout: 406663 4 -rw-r--r-- 1 ubuntu ubuntu 724 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/objectstore/bluestore-comp.yaml 2024-03-27T21:42:44.752 INFO:teuthology.orchestra.run.smithi060.stdout: 406656 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/.qa -> ../.qa/ 2024-03-27T21:42:44.752 INFO:teuthology.orchestra.run.smithi060.stdout: 406666 4 -rw-r--r-- 1 ubuntu ubuntu 87 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/openstack.yaml 2024-03-27T21:42:44.752 INFO:teuthology.orchestra.run.smithi060.stdout: 406675 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/ubuntu_latest.yaml -> .qa/distros/supported/ubuntu_latest.yaml 2024-03-27T21:42:44.752 INFO:teuthology.orchestra.run.smithi060.stdout: 406667 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/scheduler 2024-03-27T21:42:44.752 INFO:teuthology.orchestra.run.smithi060.stdout: 406668 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/scheduler/.qa -> ../.qa/ 2024-03-27T21:42:44.752 INFO:teuthology.orchestra.run.smithi060.stdout: 406670 4 -rw-r--r-- 1 ubuntu ubuntu 79 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/scheduler/dmclock_default_shards.yaml 2024-03-27T21:42:44.752 INFO:teuthology.orchestra.run.smithi060.stdout: 406671 4 -rw-r--r-- 1 ubuntu ubuntu 66 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/scheduler/wpq_default_shards.yaml 2024-03-27T21:42:44.752 INFO:teuthology.orchestra.run.smithi060.stdout: 406669 4 -rw-r--r-- 1 ubuntu ubuntu 149 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/scheduler/dmclock_1Shard_16Threads.yaml 2024-03-27T21:42:44.753 INFO:teuthology.orchestra.run.smithi060.stdout: 406676 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/workloads 2024-03-27T21:42:44.753 INFO:teuthology.orchestra.run.smithi060.stdout: 406684 4 -rw-r--r-- 1 ubuntu ubuntu 689 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/workloads/radosbench_4K_seq_read.yaml 2024-03-27T21:42:44.753 INFO:teuthology.orchestra.run.smithi060.stdout: 406677 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.753 INFO:teuthology.orchestra.run.smithi060.stdout: 406685 4 -rw-r--r-- 1 ubuntu ubuntu 717 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/workloads/radosbench_4M_rand_read.yaml 2024-03-27T21:42:44.753 INFO:teuthology.orchestra.run.smithi060.stdout: 406682 4 -rw-r--r-- 1 ubuntu ubuntu 699 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/workloads/fio_4M_rand_write.yaml 2024-03-27T21:42:44.753 INFO:teuthology.orchestra.run.smithi060.stdout: 406680 4 -rw-r--r-- 1 ubuntu ubuntu 698 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/workloads/fio_4M_rand_read.yaml 2024-03-27T21:42:44.753 INFO:teuthology.orchestra.run.smithi060.stdout: 406689 4 -rw-r--r-- 1 ubuntu ubuntu 696 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/workloads/sample_fio.yaml 2024-03-27T21:42:44.753 INFO:teuthology.orchestra.run.smithi060.stdout: 406688 4 -rw-r--r-- 1 ubuntu ubuntu 116 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/workloads/radosbench_omap_write.yaml 2024-03-27T21:42:44.753 INFO:teuthology.orchestra.run.smithi060.stdout: 406683 4 -rw-r--r-- 1 ubuntu ubuntu 617 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/workloads/radosbench_4K_rand_read.yaml 2024-03-27T21:42:44.754 INFO:teuthology.orchestra.run.smithi060.stdout: 406678 4 -rw-r--r-- 1 ubuntu ubuntu 695 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/workloads/fio_4K_rand_read.yaml 2024-03-27T21:42:44.754 INFO:teuthology.orchestra.run.smithi060.stdout: 406681 4 -rw-r--r-- 1 ubuntu ubuntu 696 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/workloads/fio_4M_rand_rw.yaml 2024-03-27T21:42:44.754 INFO:teuthology.orchestra.run.smithi060.stdout: 406690 4 -rw-r--r-- 1 ubuntu ubuntu 688 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/workloads/sample_radosbench.yaml 2024-03-27T21:42:44.754 INFO:teuthology.orchestra.run.smithi060.stdout: 406687 4 -rw-r--r-- 1 ubuntu ubuntu 691 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/workloads/radosbench_4M_write.yaml 2024-03-27T21:42:44.754 INFO:teuthology.orchestra.run.smithi060.stdout: 406686 4 -rw-r--r-- 1 ubuntu ubuntu 692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/workloads/radosbench_4M_seq_read.yaml 2024-03-27T21:42:44.754 INFO:teuthology.orchestra.run.smithi060.stdout: 406679 4 -rw-r--r-- 1 ubuntu ubuntu 722 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/workloads/fio_4K_rand_rw.yaml 2024-03-27T21:42:44.754 INFO:teuthology.orchestra.run.smithi060.stdout: 406672 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/settings 2024-03-27T21:42:44.754 INFO:teuthology.orchestra.run.smithi060.stdout: 406673 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/settings/.qa -> ../.qa/ 2024-03-27T21:42:44.755 INFO:teuthology.orchestra.run.smithi060.stdout: 406674 4 -rw-r--r-- 1 ubuntu ubuntu 2043 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/settings/optimized.yaml 2024-03-27T21:42:44.755 INFO:teuthology.orchestra.run.smithi060.stdout: 406658 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/perf/mon_election -> .qa/mon_election 2024-03-27T21:42:44.755 INFO:teuthology.orchestra.run.smithi060.stdout: 406841 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-crush-4-nodes 2024-03-27T21:42:44.755 INFO:teuthology.orchestra.run.smithi060.stdout: 406847 0 lrwxrwxrwx 1 ubuntu ubuntu 19 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-crush-4-nodes/ceph.yaml -> ../thrash/ceph.yaml 2024-03-27T21:42:44.755 INFO:teuthology.orchestra.run.smithi060.stdout: 406842 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-crush-4-nodes/% 2024-03-27T21:42:44.755 INFO:teuthology.orchestra.run.smithi060.stdout: 406844 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-crush-4-nodes/arch 2024-03-27T21:42:44.755 INFO:teuthology.orchestra.run.smithi060.stdout: 406846 4 -rw-r--r-- 1 ubuntu ubuntu 13 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-crush-4-nodes/arch/x86_64.yaml 2024-03-27T21:42:44.755 INFO:teuthology.orchestra.run.smithi060.stdout: 406845 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-crush-4-nodes/arch/.qa -> ../.qa/ 2024-03-27T21:42:44.756 INFO:teuthology.orchestra.run.smithi060.stdout: 406850 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-crush-4-nodes/objectstore -> .qa/objectstore_debug 2024-03-27T21:42:44.756 INFO:teuthology.orchestra.run.smithi060.stdout: 406849 0 lrwxrwxrwx 1 ubuntu ubuntu 23 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-crush-4-nodes/msgr-failures -> ../thrash/msgr-failures 2024-03-27T21:42:44.756 INFO:teuthology.orchestra.run.smithi060.stdout: 406843 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-crush-4-nodes/.qa -> ../.qa/ 2024-03-27T21:42:44.756 INFO:teuthology.orchestra.run.smithi060.stdout: 406855 0 lrwxrwxrwx 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-crush-4-nodes/thrashosds-health.yaml -> .qa/tasks/thrashosds-health.yaml 2024-03-27T21:42:44.756 INFO:teuthology.orchestra.run.smithi060.stdout: 406853 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-crush-4-nodes/supported-random-distro$ -> ../basic/supported-random-distro$ 2024-03-27T21:42:44.756 INFO:teuthology.orchestra.run.smithi060.stdout: 406856 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-crush-4-nodes/workloads 2024-03-27T21:42:44.756 INFO:teuthology.orchestra.run.smithi060.stdout: 406857 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-crush-4-nodes/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.756 INFO:teuthology.orchestra.run.smithi060.stdout: 406858 4 lrwxrwxrwx 1 ubuntu ubuntu 60 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-crush-4-nodes/workloads/ec-rados-plugin=jerasure-k=2-m=2-crush.yaml -> .qa/erasure-code/ec-rados-plugin=jerasure-k=2-m=2-crush.yaml 2024-03-27T21:42:44.757 INFO:teuthology.orchestra.run.smithi060.stdout: 406859 4 lrwxrwxrwx 1 ubuntu ubuntu 60 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-crush-4-nodes/workloads/ec-rados-plugin=jerasure-k=8-m=6-crush.yaml -> .qa/erasure-code/ec-rados-plugin=jerasure-k=8-m=6-crush.yaml 2024-03-27T21:42:44.757 INFO:teuthology.orchestra.run.smithi060.stdout: 406851 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-crush-4-nodes/rados.yaml -> .qa/config/rados.yaml 2024-03-27T21:42:44.757 INFO:teuthology.orchestra.run.smithi060.stdout: 406852 0 lrwxrwxrwx 1 ubuntu ubuntu 30 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-crush-4-nodes/recovery-overrides -> ../thrash/2-recovery-overrides 2024-03-27T21:42:44.757 INFO:teuthology.orchestra.run.smithi060.stdout: 406854 0 lrwxrwxrwx 1 ubuntu ubuntu 19 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-crush-4-nodes/thrashers -> ../thrash/thrashers 2024-03-27T21:42:44.757 INFO:teuthology.orchestra.run.smithi060.stdout: 406848 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-crush-4-nodes/mon_election -> .qa/mon_election 2024-03-27T21:42:44.757 INFO:teuthology.orchestra.run.smithi060.stdout: 406739 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton 2024-03-27T21:42:44.757 INFO:teuthology.orchestra.run.smithi060.stdout: 406742 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all 2024-03-27T21:42:44.757 INFO:teuthology.orchestra.run.smithi060.stdout: 406753 4 -rw-r--r-- 1 ubuntu ubuntu 530 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/lost-unfound-delete.yaml 2024-03-27T21:42:44.758 INFO:teuthology.orchestra.run.smithi060.stdout: 406762 4 -rw-r--r-- 1 ubuntu ubuntu 3371 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/mon-memory-target-compliance.yaml.disabled 2024-03-27T21:42:44.758 INFO:teuthology.orchestra.run.smithi060.stdout: 406764 4 -rw-r--r-- 1 ubuntu ubuntu 601 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/osd-recovery-incomplete.yaml 2024-03-27T21:42:44.758 INFO:teuthology.orchestra.run.smithi060.stdout: 406778 4 -rw-r--r-- 1 ubuntu ubuntu 906 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/thrash-eio.yaml 2024-03-27T21:42:44.758 INFO:teuthology.orchestra.run.smithi060.stdout: 406768 4 -rw-r--r-- 1 ubuntu ubuntu 663 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/pg-autoscaler.yaml 2024-03-27T21:42:44.758 INFO:teuthology.orchestra.run.smithi060.stdout: 406747 4 -rw-r--r-- 1 ubuntu ubuntu 451 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/divergent_priors.yaml 2024-03-27T21:42:44.758 INFO:teuthology.orchestra.run.smithi060.stdout: 406743 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/.qa -> ../.qa/ 2024-03-27T21:42:44.758 INFO:teuthology.orchestra.run.smithi060.stdout: 406777 4 -rw-r--r-- 1 ubuntu ubuntu 1019 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/thrash-backfill-full.yaml 2024-03-27T21:42:44.758 INFO:teuthology.orchestra.run.smithi060.stdout: 406766 4 -rw-r--r-- 1 ubuntu ubuntu 515 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/peer.yaml 2024-03-27T21:42:44.758 INFO:teuthology.orchestra.run.smithi060.stdout: 406773 4 -rw-r--r-- 1 ubuntu ubuntu 1424 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/recovery-preemption.yaml 2024-03-27T21:42:44.759 INFO:teuthology.orchestra.run.smithi060.stdout: 406757 4 -rw-r--r-- 1 ubuntu ubuntu 675 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/max-pg-per-osd.from-replica.yaml 2024-03-27T21:42:44.759 INFO:teuthology.orchestra.run.smithi060.stdout: 406771 4 -rw-r--r-- 1 ubuntu ubuntu 1039 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/random-eio.yaml 2024-03-27T21:42:44.759 INFO:teuthology.orchestra.run.smithi060.stdout: 406746 4 -rw-r--r-- 1 ubuntu ubuntu 532 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/deduptool.yaml 2024-03-27T21:42:44.759 INFO:teuthology.orchestra.run.smithi060.stdout: 406755 4 -rw-r--r-- 1 ubuntu ubuntu 599 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/max-pg-per-osd.from-mon.yaml 2024-03-27T21:42:44.759 INFO:teuthology.orchestra.run.smithi060.stdout: 406767 4 -rw-r--r-- 1 ubuntu ubuntu 750 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/pg-autoscaler-progress-off.yaml 2024-03-27T21:42:44.759 INFO:teuthology.orchestra.run.smithi060.stdout: 406776 4 -rw-r--r-- 1 ubuntu ubuntu 738 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/test-noautoscale-flag.yaml 2024-03-27T21:42:44.759 INFO:teuthology.orchestra.run.smithi060.stdout: 406749 4 -rw-r--r-- 1 ubuntu ubuntu 415 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/dump-stuck.yaml 2024-03-27T21:42:44.759 INFO:teuthology.orchestra.run.smithi060.stdout: 406770 4 -rw-r--r-- 1 ubuntu ubuntu 532 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/radostool.yaml 2024-03-27T21:42:44.760 INFO:teuthology.orchestra.run.smithi060.stdout: 406784 4 -rw-r--r-- 1 ubuntu ubuntu 1965 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/thrash_cache_writeback_proxy_none.yaml 2024-03-27T21:42:44.760 INFO:teuthology.orchestra.run.smithi060.stdout: 406748 4 -rw-r--r-- 1 ubuntu ubuntu 452 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/divergent_priors2.yaml 2024-03-27T21:42:44.760 INFO:teuthology.orchestra.run.smithi060.stdout: 406750 4 -rw-r--r-- 1 ubuntu ubuntu 734 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/ec-inconsistent-hinfo.yaml 2024-03-27T21:42:44.760 INFO:teuthology.orchestra.run.smithi060.stdout: 406744 4 -rw-r--r-- 1 ubuntu ubuntu 693 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/admin-socket.yaml 2024-03-27T21:42:44.760 INFO:teuthology.orchestra.run.smithi060.stdout: 406761 4 -rw-r--r-- 1 ubuntu ubuntu 416 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/mon-config.yaml 2024-03-27T21:42:44.760 INFO:teuthology.orchestra.run.smithi060.stdout: 406758 4 -rw-r--r-- 1 ubuntu ubuntu 369 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/mon-auth-caps.yaml 2024-03-27T21:42:44.760 INFO:teuthology.orchestra.run.smithi060.stdout: 406759 4 -rw-r--r-- 1 ubuntu ubuntu 345 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/mon-config-key-caps.yaml 2024-03-27T21:42:44.760 INFO:teuthology.orchestra.run.smithi060.stdout: 406769 4 -rw-r--r-- 1 ubuntu ubuntu 859 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/pg-removal-interruption.yaml 2024-03-27T21:42:44.761 INFO:teuthology.orchestra.run.smithi060.stdout: 406772 4 -rw-r--r-- 1 ubuntu ubuntu 672 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/rebuild-mondb.yaml 2024-03-27T21:42:44.761 INFO:teuthology.orchestra.run.smithi060.stdout: 406765 4 -rw-r--r-- 1 ubuntu ubuntu 654 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/osd-recovery.yaml 2024-03-27T21:42:44.761 INFO:teuthology.orchestra.run.smithi060.stdout: 406785 4 -rw-r--r-- 1 ubuntu ubuntu 686 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/watch-notify-same-primary.yaml 2024-03-27T21:42:44.761 INFO:teuthology.orchestra.run.smithi060.stdout: 406754 4 -rw-r--r-- 1 ubuntu ubuntu 519 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/lost-unfound.yaml 2024-03-27T21:42:44.761 INFO:teuthology.orchestra.run.smithi060.stdout: 406745 4 -rw-r--r-- 1 ubuntu ubuntu 803 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/backfill-toofull.yaml 2024-03-27T21:42:44.761 INFO:teuthology.orchestra.run.smithi060.stdout: 406763 4 -rw-r--r-- 1 ubuntu ubuntu 520 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/osd-backfill.yaml 2024-03-27T21:42:44.761 INFO:teuthology.orchestra.run.smithi060.stdout: 406774 4 -rw-r--r-- 1 ubuntu ubuntu 360 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/resolve_stuck_peering.yaml 2024-03-27T21:42:44.761 INFO:teuthology.orchestra.run.smithi060.stdout: 406779 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/thrash-rados 2024-03-27T21:42:44.762 INFO:teuthology.orchestra.run.smithi060.stdout: 406780 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/thrash-rados/+ 2024-03-27T21:42:44.762 INFO:teuthology.orchestra.run.smithi060.stdout: 406781 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/thrash-rados/.qa -> ../.qa/ 2024-03-27T21:42:44.762 INFO:teuthology.orchestra.run.smithi060.stdout: 406782 4 -rw-r--r-- 1 ubuntu ubuntu 445 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/thrash-rados/thrash-rados.yaml 2024-03-27T21:42:44.762 INFO:teuthology.orchestra.run.smithi060.stdout: 406783 0 lrwxrwxrwx 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/thrash-rados/thrashosds-health.yaml -> .qa/tasks/thrashosds-health.yaml 2024-03-27T21:42:44.762 INFO:teuthology.orchestra.run.smithi060.stdout: 406751 4 -rw-r--r-- 1 ubuntu ubuntu 559 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/ec-lost-unfound.yaml 2024-03-27T21:42:44.762 INFO:teuthology.orchestra.run.smithi060.stdout: 406756 4 -rw-r--r-- 1 ubuntu ubuntu 674 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/max-pg-per-osd.from-primary.yaml 2024-03-27T21:42:44.762 INFO:teuthology.orchestra.run.smithi060.stdout: 406775 4 -rw-r--r-- 1 ubuntu ubuntu 508 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/test-crash.yaml 2024-03-27T21:42:44.762 INFO:teuthology.orchestra.run.smithi060.stdout: 406760 4 -rw-r--r-- 1 ubuntu ubuntu 405 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/mon-config-keys.yaml 2024-03-27T21:42:44.763 INFO:teuthology.orchestra.run.smithi060.stdout: 406752 4 -rw-r--r-- 1 ubuntu ubuntu 266 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/all/erasure-code-nonregression.yaml 2024-03-27T21:42:44.763 INFO:teuthology.orchestra.run.smithi060.stdout: 406787 0 lrwxrwxrwx 1 ubuntu ubuntu 8 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/msgr -> .qa/msgr 2024-03-27T21:42:44.763 INFO:teuthology.orchestra.run.smithi060.stdout: 406740 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/% 2024-03-27T21:42:44.763 INFO:teuthology.orchestra.run.smithi060.stdout: 406793 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/objectstore -> .qa/objectstore_debug 2024-03-27T21:42:44.763 INFO:teuthology.orchestra.run.smithi060.stdout: 406788 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/msgr-failures 2024-03-27T21:42:44.763 INFO:teuthology.orchestra.run.smithi060.stdout: 406789 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.763 INFO:teuthology.orchestra.run.smithi060.stdout: 406790 4 -rw-r--r-- 1 ubuntu ubuntu 198 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/msgr-failures/few.yaml 2024-03-27T21:42:44.763 INFO:teuthology.orchestra.run.smithi060.stdout: 406791 4 -rw-r--r-- 1 ubuntu ubuntu 314 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/msgr-failures/many.yaml 2024-03-27T21:42:44.764 INFO:teuthology.orchestra.run.smithi060.stdout: 406792 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/msgr-failures/none.yaml 2024-03-27T21:42:44.764 INFO:teuthology.orchestra.run.smithi060.stdout: 406741 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/.qa -> ../.qa/ 2024-03-27T21:42:44.764 INFO:teuthology.orchestra.run.smithi060.stdout: 406795 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/supported-random-distro$ -> ../basic/supported-random-distro$ 2024-03-27T21:42:44.764 INFO:teuthology.orchestra.run.smithi060.stdout: 406794 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/rados.yaml -> .qa/config/rados.yaml 2024-03-27T21:42:44.764 INFO:teuthology.orchestra.run.smithi060.stdout: 406786 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton/mon_election -> .qa/mon_election 2024-03-27T21:42:44.764 INFO:teuthology.orchestra.run.smithi060.stdout: 406712 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr 2024-03-27T21:42:44.764 INFO:teuthology.orchestra.run.smithi060.stdout: 406715 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/all 2024-03-27T21:42:44.764 INFO:teuthology.orchestra.run.smithi060.stdout: 406721 4 -rw-r--r-- 1 ubuntu ubuntu 219 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/all/ceph-post-file.yaml 2024-03-27T21:42:44.764 INFO:teuthology.orchestra.run.smithi060.stdout: 406717 4 -rw-r--r-- 1 ubuntu ubuntu 612 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/all/admin_socket_output.yaml 2024-03-27T21:42:44.765 INFO:teuthology.orchestra.run.smithi060.stdout: 406716 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/all/.qa -> ../.qa/ 2024-03-27T21:42:44.765 INFO:teuthology.orchestra.run.smithi060.stdout: 406726 4 -rw-r--r-- 1 ubuntu ubuntu 568 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/all/health-warnings.yaml 2024-03-27T21:42:44.765 INFO:teuthology.orchestra.run.smithi060.stdout: 406729 4 -rw-r--r-- 1 ubuntu ubuntu 510 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/all/librados_hello_world.yaml 2024-03-27T21:42:44.765 INFO:teuthology.orchestra.run.smithi060.stdout: 406727 4 -rw-r--r-- 1 ubuntu ubuntu 743 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/all/large-omap-object-warnings.yaml 2024-03-27T21:42:44.765 INFO:teuthology.orchestra.run.smithi060.stdout: 406724 4 -rw-r--r-- 1 ubuntu ubuntu 1200 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/all/export-after-evict.yaml 2024-03-27T21:42:44.765 INFO:teuthology.orchestra.run.smithi060.stdout: 406734 4 -rw-r--r-- 1 ubuntu ubuntu 1276 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/all/recovery-unfound-found.yaml 2024-03-27T21:42:44.765 INFO:teuthology.orchestra.run.smithi060.stdout: 406719 4 -rw-r--r-- 1 ubuntu ubuntu 1708 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/all/cache-fs-trunc.yaml 2024-03-27T21:42:44.765 INFO:teuthology.orchestra.run.smithi060.stdout: 406718 4 -rw-r--r-- 1 ubuntu ubuntu 343 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/all/balancer.yaml 2024-03-27T21:42:44.766 INFO:teuthology.orchestra.run.smithi060.stdout: 406732 4 -rw-r--r-- 1 ubuntu ubuntu 708 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/all/osd_stale_reads.yaml 2024-03-27T21:42:44.766 INFO:teuthology.orchestra.run.smithi060.stdout: 406723 4 -rw-r--r-- 1 ubuntu ubuntu 469 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/all/crushdiff.yaml 2024-03-27T21:42:44.766 INFO:teuthology.orchestra.run.smithi060.stdout: 406728 4 -rw-r--r-- 1 ubuntu ubuntu 374 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/all/lazy_omap_stats_output.yaml 2024-03-27T21:42:44.766 INFO:teuthology.orchestra.run.smithi060.stdout: 406725 4 -rw-r--r-- 1 ubuntu ubuntu 1260 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/all/full-tiering.yaml 2024-03-27T21:42:44.766 INFO:teuthology.orchestra.run.smithi060.stdout: 406731 4 -rw-r--r-- 1 ubuntu ubuntu 1077 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/all/multi-backfill-reject.yaml 2024-03-27T21:42:44.766 INFO:teuthology.orchestra.run.smithi060.stdout: 406720 4 -rw-r--r-- 1 ubuntu ubuntu 497 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/all/ceph-kvstore-tool.yaml 2024-03-27T21:42:44.766 INFO:teuthology.orchestra.run.smithi060.stdout: 406730 4 -rw-r--r-- 1 ubuntu ubuntu 447 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/all/msgr.yaml 2024-03-27T21:42:44.766 INFO:teuthology.orchestra.run.smithi060.stdout: 406735 4 -rw-r--r-- 1 ubuntu ubuntu 308 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/all/version-number-sanity.yaml 2024-03-27T21:42:44.767 INFO:teuthology.orchestra.run.smithi060.stdout: 406722 4 -rw-r--r-- 1 ubuntu ubuntu 420 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/all/ceph-snapmapper.yaml 2024-03-27T21:42:44.767 INFO:teuthology.orchestra.run.smithi060.stdout: 406733 4 -rw-r--r-- 1 ubuntu ubuntu 362 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/all/pool-access.yaml 2024-03-27T21:42:44.767 INFO:teuthology.orchestra.run.smithi060.stdout: 406713 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/% 2024-03-27T21:42:44.767 INFO:teuthology.orchestra.run.smithi060.stdout: 406714 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/.qa -> ../.qa/ 2024-03-27T21:42:44.767 INFO:teuthology.orchestra.run.smithi060.stdout: 406738 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/supported-random-distro$ -> ../basic/supported-random-distro$ 2024-03-27T21:42:44.767 INFO:teuthology.orchestra.run.smithi060.stdout: 406737 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/rados.yaml -> .qa/config/rados.yaml 2024-03-27T21:42:44.769 INFO:teuthology.orchestra.run.smithi060.stdout: 406736 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-nomsgr/mon_election -> .qa/mon_election 2024-03-27T21:42:44.770 INFO:teuthology.orchestra.run.smithi060.stdout: 406696 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-bluestore 2024-03-27T21:42:44.770 INFO:teuthology.orchestra.run.smithi060.stdout: 406699 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-bluestore/all 2024-03-27T21:42:44.770 INFO:teuthology.orchestra.run.smithi060.stdout: 406700 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-bluestore/all/.qa -> ../.qa/ 2024-03-27T21:42:44.770 INFO:teuthology.orchestra.run.smithi060.stdout: 406701 4 -rw-r--r-- 1 ubuntu ubuntu 1015 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-bluestore/all/cephtool.yaml 2024-03-27T21:42:44.770 INFO:teuthology.orchestra.run.smithi060.stdout: 406703 0 lrwxrwxrwx 1 ubuntu ubuntu 8 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-bluestore/msgr -> .qa/msgr 2024-03-27T21:42:44.770 INFO:teuthology.orchestra.run.smithi060.stdout: 406697 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-bluestore/% 2024-03-27T21:42:44.770 INFO:teuthology.orchestra.run.smithi060.stdout: 406705 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-bluestore/objectstore 2024-03-27T21:42:44.771 INFO:teuthology.orchestra.run.smithi060.stdout: 406709 0 lrwxrwxrwx 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-bluestore/objectstore/bluestore-comp-snappy.yaml -> .qa/objectstore_debug/bluestore-comp-snappy.yaml 2024-03-27T21:42:44.771 INFO:teuthology.orchestra.run.smithi060.stdout: 406706 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-bluestore/objectstore/.qa -> ../.qa/ 2024-03-27T21:42:44.771 INFO:teuthology.orchestra.run.smithi060.stdout: 406707 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-bluestore/objectstore/bluestore-bitmap.yaml -> .qa/objectstore_debug/bluestore-bitmap.yaml 2024-03-27T21:42:44.771 INFO:teuthology.orchestra.run.smithi060.stdout: 406708 0 lrwxrwxrwx 1 ubuntu ubuntu 45 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-bluestore/objectstore/bluestore-comp-lz4.yaml -> .qa/objectstore_debug/bluestore-comp-lz4.yaml 2024-03-27T21:42:44.771 INFO:teuthology.orchestra.run.smithi060.stdout: 406704 0 lrwxrwxrwx 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-bluestore/msgr-failures -> ../singleton/msgr-failures 2024-03-27T21:42:44.771 INFO:teuthology.orchestra.run.smithi060.stdout: 406698 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-bluestore/.qa -> ../.qa/ 2024-03-27T21:42:44.771 INFO:teuthology.orchestra.run.smithi060.stdout: 406711 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-bluestore/supported-random-distro$ -> ../basic/supported-random-distro$ 2024-03-27T21:42:44.771 INFO:teuthology.orchestra.run.smithi060.stdout: 406710 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-bluestore/rados.yaml -> .qa/config/rados.yaml 2024-03-27T21:42:44.772 INFO:teuthology.orchestra.run.smithi060.stdout: 406702 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/singleton-bluestore/mon_election -> .qa/mon_election 2024-03-27T21:42:44.772 INFO:teuthology.orchestra.run.smithi060.stdout: 407120 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify 2024-03-27T21:42:44.772 INFO:teuthology.orchestra.run.smithi060.stdout: 407139 0 lrwxrwxrwx 1 ubuntu ubuntu 8 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/msgr -> .qa/msgr 2024-03-27T21:42:44.772 INFO:teuthology.orchestra.run.smithi060.stdout: 407124 4 -rw-r--r-- 1 ubuntu ubuntu 328 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/ceph.yaml 2024-03-27T21:42:44.772 INFO:teuthology.orchestra.run.smithi060.stdout: 407121 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/% 2024-03-27T21:42:44.772 INFO:teuthology.orchestra.run.smithi060.stdout: 407143 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/objectstore -> .qa/objectstore_debug 2024-03-27T21:42:44.772 INFO:teuthology.orchestra.run.smithi060.stdout: 407140 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/msgr-failures 2024-03-27T21:42:44.772 INFO:teuthology.orchestra.run.smithi060.stdout: 407141 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.773 INFO:teuthology.orchestra.run.smithi060.stdout: 407142 4 -rw-r--r-- 1 ubuntu ubuntu 177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/msgr-failures/few.yaml 2024-03-27T21:42:44.773 INFO:teuthology.orchestra.run.smithi060.stdout: 407122 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/.qa -> ../.qa/ 2024-03-27T21:42:44.773 INFO:teuthology.orchestra.run.smithi060.stdout: 407125 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/clusters 2024-03-27T21:42:44.773 INFO:teuthology.orchestra.run.smithi060.stdout: 407128 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/clusters/fixed-2.yaml -> .qa/clusters/fixed-2.yaml 2024-03-27T21:42:44.773 INFO:teuthology.orchestra.run.smithi060.stdout: 407126 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/clusters/+ 2024-03-27T21:42:44.773 INFO:teuthology.orchestra.run.smithi060.stdout: 407127 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.773 INFO:teuthology.orchestra.run.smithi060.stdout: 407129 4 -rw-r--r-- 1 ubuntu ubuntu 87 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/clusters/openstack.yaml 2024-03-27T21:42:44.773 INFO:teuthology.orchestra.run.smithi060.stdout: 407144 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/rados.yaml -> .qa/config/rados.yaml 2024-03-27T21:42:44.773 INFO:teuthology.orchestra.run.smithi060.stdout: 407130 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/d-thrash 2024-03-27T21:42:44.774 INFO:teuthology.orchestra.run.smithi060.stdout: 407131 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/d-thrash/.qa -> ../.qa/ 2024-03-27T21:42:44.774 INFO:teuthology.orchestra.run.smithi060.stdout: 407132 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/d-thrash/default 2024-03-27T21:42:44.774 INFO:teuthology.orchestra.run.smithi060.stdout: 407133 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/d-thrash/default/+ 2024-03-27T21:42:44.774 INFO:teuthology.orchestra.run.smithi060.stdout: 407134 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/d-thrash/default/.qa -> ../.qa/ 2024-03-27T21:42:44.774 INFO:teuthology.orchestra.run.smithi060.stdout: 407136 0 lrwxrwxrwx 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/d-thrash/default/thrashosds-health.yaml -> .qa/tasks/thrashosds-health.yaml 2024-03-27T21:42:44.774 INFO:teuthology.orchestra.run.smithi060.stdout: 407135 4 -rw-r--r-- 1 ubuntu ubuntu 227 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/d-thrash/default/default.yaml 2024-03-27T21:42:44.774 INFO:teuthology.orchestra.run.smithi060.stdout: 407137 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/d-thrash/none.yaml 2024-03-27T21:42:44.774 INFO:teuthology.orchestra.run.smithi060.stdout: 407150 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/validater 2024-03-27T21:42:44.775 INFO:teuthology.orchestra.run.smithi060.stdout: 407151 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/validater/.qa -> ../.qa/ 2024-03-27T21:42:44.775 INFO:teuthology.orchestra.run.smithi060.stdout: 407152 4 -rw-r--r-- 1 ubuntu ubuntu 65 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/validater/lockdep.yaml 2024-03-27T21:42:44.775 INFO:teuthology.orchestra.run.smithi060.stdout: 407153 4 -rw-r--r-- 1 ubuntu ubuntu 992 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/validater/valgrind.yaml 2024-03-27T21:42:44.775 INFO:teuthology.orchestra.run.smithi060.stdout: 407123 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/centos_latest.yaml -> .qa/distros/supported/centos_latest.yaml 2024-03-27T21:42:44.775 INFO:teuthology.orchestra.run.smithi060.stdout: 407145 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/tasks 2024-03-27T21:42:44.775 INFO:teuthology.orchestra.run.smithi060.stdout: 407146 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.775 INFO:teuthology.orchestra.run.smithi060.stdout: 407149 4 -rw-r--r-- 1 ubuntu ubuntu 224 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/tasks/rados_cls_all.yaml 2024-03-27T21:42:44.775 INFO:teuthology.orchestra.run.smithi060.stdout: 407147 4 -rw-r--r-- 1 ubuntu ubuntu 235 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/tasks/mon_recovery.yaml 2024-03-27T21:42:44.776 INFO:teuthology.orchestra.run.smithi060.stdout: 407148 4 -rw-r--r-- 1 ubuntu ubuntu 743 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/tasks/rados_api_tests.yaml 2024-03-27T21:42:44.776 INFO:teuthology.orchestra.run.smithi060.stdout: 407138 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/verify/mon_election -> .qa/mon_election 2024-03-27T21:42:44.776 INFO:teuthology.orchestra.run.smithi060.stdout: 407107 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/upgrade 2024-03-27T21:42:44.776 INFO:teuthology.orchestra.run.smithi060.stdout: 407108 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/upgrade/.qa -> ../.qa/ 2024-03-27T21:42:44.776 INFO:teuthology.orchestra.run.smithi060.stdout: 407109 0 lrwxrwxrwx 1 ubuntu ubuntu 29 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/upgrade/parallel -> ../../upgrade/reef-x/parallel 2024-03-27T21:42:44.776 INFO:teuthology.orchestra.run.smithi060.stdout: 406957 4 drwxr-xr-x 11 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients 2024-03-27T21:42:44.776 INFO:teuthology.orchestra.run.smithi060.stdout: 406981 4 -rw-r--r-- 1 ubuntu ubuntu 149 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/ceph.yaml 2024-03-27T21:42:44.777 INFO:teuthology.orchestra.run.smithi060.stdout: 406958 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/% 2024-03-27T21:42:44.777 INFO:teuthology.orchestra.run.smithi060.stdout: 406987 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/d-balancer 2024-03-27T21:42:44.777 INFO:teuthology.orchestra.run.smithi060.stdout: 406988 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/d-balancer/.qa -> ../.qa/ 2024-03-27T21:42:44.777 INFO:teuthology.orchestra.run.smithi060.stdout: 406989 4 -rw-r--r-- 1 ubuntu ubuntu 148 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/d-balancer/crush-compat.yaml 2024-03-27T21:42:44.777 INFO:teuthology.orchestra.run.smithi060.stdout: 406990 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/d-balancer/on.yaml 2024-03-27T21:42:44.777 INFO:teuthology.orchestra.run.smithi060.stdout: 406992 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/msgr-failures 2024-03-27T21:42:44.777 INFO:teuthology.orchestra.run.smithi060.stdout: 406993 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.777 INFO:teuthology.orchestra.run.smithi060.stdout: 406995 4 -rw-r--r-- 1 ubuntu ubuntu 237 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/msgr-failures/few.yaml 2024-03-27T21:42:44.778 INFO:teuthology.orchestra.run.smithi060.stdout: 406994 4 -rw-r--r-- 1 ubuntu ubuntu 208 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/msgr-failures/fastclose.yaml 2024-03-27T21:42:44.778 INFO:teuthology.orchestra.run.smithi060.stdout: 406996 4 -rw-r--r-- 1 ubuntu ubuntu 324 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/msgr-failures/osd-delay.yaml 2024-03-27T21:42:44.778 INFO:teuthology.orchestra.run.smithi060.stdout: 406959 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/.qa -> ../.qa/ 2024-03-27T21:42:44.778 INFO:teuthology.orchestra.run.smithi060.stdout: 407006 0 lrwxrwxrwx 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/thrashosds-health.yaml -> .qa/tasks/thrashosds-health.yaml 2024-03-27T21:42:44.778 INFO:teuthology.orchestra.run.smithi060.stdout: 407007 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/workloads 2024-03-27T21:42:44.778 INFO:teuthology.orchestra.run.smithi060.stdout: 407008 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.778 INFO:teuthology.orchestra.run.smithi060.stdout: 407012 4 -rw-r--r-- 1 ubuntu ubuntu 219 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/workloads/snaps-few-objects.yaml 2024-03-27T21:42:44.778 INFO:teuthology.orchestra.run.smithi060.stdout: 407013 4 -rw-r--r-- 1 ubuntu ubuntu 126 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/workloads/test_rbd_api.yaml 2024-03-27T21:42:44.779 INFO:teuthology.orchestra.run.smithi060.stdout: 407010 4 -rw-r--r-- 1 ubuntu ubuntu 598 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/workloads/radosbench.yaml 2024-03-27T21:42:44.779 INFO:teuthology.orchestra.run.smithi060.stdout: 407011 4 -rw-r--r-- 1 ubuntu ubuntu 185 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/workloads/rbd_cls.yaml 2024-03-27T21:42:44.779 INFO:teuthology.orchestra.run.smithi060.stdout: 407009 4 -rw-r--r-- 1 ubuntu ubuntu 999 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/workloads/cache-snaps.yaml 2024-03-27T21:42:44.779 INFO:teuthology.orchestra.run.smithi060.stdout: 406982 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/clusters 2024-03-27T21:42:44.779 INFO:teuthology.orchestra.run.smithi060.stdout: 406983 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/clusters/+ 2024-03-27T21:42:44.779 INFO:teuthology.orchestra.run.smithi060.stdout: 406984 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.779 INFO:teuthology.orchestra.run.smithi060.stdout: 406985 4 -rw-r--r-- 1 ubuntu ubuntu 87 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/clusters/openstack.yaml 2024-03-27T21:42:44.779 INFO:teuthology.orchestra.run.smithi060.stdout: 406986 4 -rw-r--r-- 1 ubuntu ubuntu 328 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/clusters/three-plus-one.yaml 2024-03-27T21:42:44.779 INFO:teuthology.orchestra.run.smithi060.stdout: 406997 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/rados.yaml -> .qa/config/rados.yaml 2024-03-27T21:42:44.780 INFO:teuthology.orchestra.run.smithi060.stdout: 406976 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/backoff 2024-03-27T21:42:44.780 INFO:teuthology.orchestra.run.smithi060.stdout: 406980 4 -rw-r--r-- 1 ubuntu ubuntu 115 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/backoff/peering_and_degraded.yaml 2024-03-27T21:42:44.780 INFO:teuthology.orchestra.run.smithi060.stdout: 406977 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/backoff/.qa -> ../.qa/ 2024-03-27T21:42:44.780 INFO:teuthology.orchestra.run.smithi060.stdout: 406978 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/backoff/normal.yaml 2024-03-27T21:42:44.780 INFO:teuthology.orchestra.run.smithi060.stdout: 406979 4 -rw-r--r-- 1 ubuntu ubuntu 77 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/backoff/peering.yaml 2024-03-27T21:42:44.780 INFO:teuthology.orchestra.run.smithi060.stdout: 406963 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/0-size-min-size-overrides 2024-03-27T21:42:44.780 INFO:teuthology.orchestra.run.smithi060.stdout: 406965 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/0-size-min-size-overrides/2-size-2-min-size.yaml -> .qa/overrides/2-size-2-min-size.yaml 2024-03-27T21:42:44.780 INFO:teuthology.orchestra.run.smithi060.stdout: 406964 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/0-size-min-size-overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.781 INFO:teuthology.orchestra.run.smithi060.stdout: 406966 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/0-size-min-size-overrides/3-size-2-min-size.yaml -> .qa/overrides/3-size-2-min-size.yaml 2024-03-27T21:42:44.781 INFO:teuthology.orchestra.run.smithi060.stdout: 406998 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/thrashers 2024-03-27T21:42:44.781 INFO:teuthology.orchestra.run.smithi060.stdout: 406999 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/thrashers/.qa -> ../.qa/ 2024-03-27T21:42:44.781 INFO:teuthology.orchestra.run.smithi060.stdout: 407005 4 -rw-r--r-- 1 ubuntu ubuntu 660 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/thrashers/pggrow.yaml 2024-03-27T21:42:44.781 INFO:teuthology.orchestra.run.smithi060.stdout: 407002 4 -rw-r--r-- 1 ubuntu ubuntu 730 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/thrashers/mapgap.yaml 2024-03-27T21:42:44.781 INFO:teuthology.orchestra.run.smithi060.stdout: 407000 4 -rw-r--r-- 1 ubuntu ubuntu 709 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/thrashers/careful.yaml 2024-03-27T21:42:44.781 INFO:teuthology.orchestra.run.smithi060.stdout: 407004 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/thrashers/none.yaml 2024-03-27T21:42:44.781 INFO:teuthology.orchestra.run.smithi060.stdout: 407001 4 -rw-r--r-- 1 ubuntu ubuntu 672 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/thrashers/default.yaml 2024-03-27T21:42:44.781 INFO:teuthology.orchestra.run.smithi060.stdout: 407003 4 -rw-r--r-- 1 ubuntu ubuntu 571 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/thrashers/morepggrow.yaml 2024-03-27T21:42:44.782 INFO:teuthology.orchestra.run.smithi060.stdout: 406991 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/mon_election -> .qa/mon_election 2024-03-27T21:42:44.782 INFO:teuthology.orchestra.run.smithi060.stdout: 406960 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/0-distro$ 2024-03-27T21:42:44.782 INFO:teuthology.orchestra.run.smithi060.stdout: 406961 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/0-distro$/.qa -> ../.qa/ 2024-03-27T21:42:44.782 INFO:teuthology.orchestra.run.smithi060.stdout: 406962 0 lrwxrwxrwx 1 ubuntu ubuntu 55 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/0-distro$/centos_8.stream_container_tools.yaml -> .qa/distros/podman/centos_8.stream_container_tools.yaml 2024-03-27T21:42:44.782 INFO:teuthology.orchestra.run.smithi060.stdout: 406967 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/1-install 2024-03-27T21:42:44.782 INFO:teuthology.orchestra.run.smithi060.stdout: 406973 4 -rw-r--r-- 1 ubuntu ubuntu 902 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/1-install/pacific.yaml 2024-03-27T21:42:44.782 INFO:teuthology.orchestra.run.smithi060.stdout: 406968 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/1-install/.qa -> ../.qa/ 2024-03-27T21:42:44.782 INFO:teuthology.orchestra.run.smithi060.stdout: 406975 4 -rw-r--r-- 1 ubuntu ubuntu 899 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/1-install/reef.yaml 2024-03-27T21:42:44.783 INFO:teuthology.orchestra.run.smithi060.stdout: 406969 4 -rw-r--r-- 1 ubuntu ubuntu 506 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/1-install/nautilus-v1only.yaml 2024-03-27T21:42:44.783 INFO:teuthology.orchestra.run.smithi060.stdout: 406970 4 -rw-r--r-- 1 ubuntu ubuntu 508 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/1-install/nautilus-v2only.yaml 2024-03-27T21:42:44.783 INFO:teuthology.orchestra.run.smithi060.stdout: 406972 4 -rw-r--r-- 1 ubuntu ubuntu 902 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/1-install/octopus.yaml 2024-03-27T21:42:44.783 INFO:teuthology.orchestra.run.smithi060.stdout: 406974 4 -rw-r--r-- 1 ubuntu ubuntu 901 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/1-install/quincy.yaml 2024-03-27T21:42:44.783 INFO:teuthology.orchestra.run.smithi060.stdout: 406971 4 -rw-r--r-- 1 ubuntu ubuntu 404 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-old-clients/1-install/nautilus.yaml 2024-03-27T21:42:44.783 INFO:teuthology.orchestra.run.smithi060.stdout: 406796 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/standalone 2024-03-27T21:42:44.783 INFO:teuthology.orchestra.run.smithi060.stdout: 406797 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/standalone/% 2024-03-27T21:42:44.783 INFO:teuthology.orchestra.run.smithi060.stdout: 406798 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/standalone/.qa -> ../.qa/ 2024-03-27T21:42:44.784 INFO:teuthology.orchestra.run.smithi060.stdout: 406799 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/standalone/supported-random-distro$ -> ../basic/supported-random-distro$ 2024-03-27T21:42:44.784 INFO:teuthology.orchestra.run.smithi060.stdout: 406800 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/standalone/workloads 2024-03-27T21:42:44.784 INFO:teuthology.orchestra.run.smithi060.stdout: 406801 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/standalone/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.784 INFO:teuthology.orchestra.run.smithi060.stdout: 406808 4 -rw-r--r-- 1 ubuntu ubuntu 252 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/standalone/workloads/mon.yaml 2024-03-27T21:42:44.784 INFO:teuthology.orchestra.run.smithi060.stdout: 406811 4 -rw-r--r-- 1 ubuntu ubuntu 254 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/standalone/workloads/scrub.yaml 2024-03-27T21:42:44.784 INFO:teuthology.orchestra.run.smithi060.stdout: 406803 4 -rw-r--r-- 1 ubuntu ubuntu 254 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/standalone/workloads/crush.yaml 2024-03-27T21:42:44.784 INFO:teuthology.orchestra.run.smithi060.stdout: 406810 4 -rw-r--r-- 1 ubuntu ubuntu 252 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/standalone/workloads/osd.yaml 2024-03-27T21:42:44.784 INFO:teuthology.orchestra.run.smithi060.stdout: 406807 4 -rw-r--r-- 1 ubuntu ubuntu 259 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/standalone/workloads/mon-stretch.yaml 2024-03-27T21:42:44.785 INFO:teuthology.orchestra.run.smithi060.stdout: 406802 4 -rw-r--r-- 1 ubuntu ubuntu 252 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/standalone/workloads/c2c.yaml 2024-03-27T21:42:44.785 INFO:teuthology.orchestra.run.smithi060.stdout: 406806 4 -rw-r--r-- 1 ubuntu ubuntu 253 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/standalone/workloads/misc.yaml 2024-03-27T21:42:44.785 INFO:teuthology.orchestra.run.smithi060.stdout: 406809 4 -rw-r--r-- 1 ubuntu ubuntu 261 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/standalone/workloads/osd-backfill.yaml 2024-03-27T21:42:44.785 INFO:teuthology.orchestra.run.smithi060.stdout: 406805 4 -rw-r--r-- 1 ubuntu ubuntu 252 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/standalone/workloads/mgr.yaml 2024-03-27T21:42:44.785 INFO:teuthology.orchestra.run.smithi060.stdout: 406804 4 -rw-r--r-- 1 ubuntu ubuntu 261 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/standalone/workloads/erasure-code.yaml 2024-03-27T21:42:44.785 INFO:teuthology.orchestra.run.smithi060.stdout: 407014 4 drwxr-xr-x 13 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash 2024-03-27T21:42:44.785 INFO:teuthology.orchestra.run.smithi060.stdout: 407061 0 lrwxrwxrwx 1 ubuntu ubuntu 8 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/msgr -> .qa/msgr 2024-03-27T21:42:44.785 INFO:teuthology.orchestra.run.smithi060.stdout: 407044 4 -rw-r--r-- 1 ubuntu ubuntu 70 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/ceph.yaml 2024-03-27T21:42:44.786 INFO:teuthology.orchestra.run.smithi060.stdout: 407050 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/crc-failures 2024-03-27T21:42:44.786 INFO:teuthology.orchestra.run.smithi060.stdout: 407051 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/crc-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.786 INFO:teuthology.orchestra.run.smithi060.stdout: 407052 4 -rw-r--r-- 1 ubuntu ubuntu 137 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/crc-failures/bad_map_crc_failure.yaml 2024-03-27T21:42:44.786 INFO:teuthology.orchestra.run.smithi060.stdout: 407053 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/crc-failures/default.yaml 2024-03-27T21:42:44.786 INFO:teuthology.orchestra.run.smithi060.stdout: 407015 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/% 2024-03-27T21:42:44.786 INFO:teuthology.orchestra.run.smithi060.stdout: 407054 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/d-balancer 2024-03-27T21:42:44.786 INFO:teuthology.orchestra.run.smithi060.stdout: 407055 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/d-balancer/.qa -> ../.qa/ 2024-03-27T21:42:44.786 INFO:teuthology.orchestra.run.smithi060.stdout: 407059 4 -rw-r--r-- 1 ubuntu ubuntu 292 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/d-balancer/upmap-read.yaml 2024-03-27T21:42:44.786 INFO:teuthology.orchestra.run.smithi060.stdout: 407056 4 -rw-r--r-- 1 ubuntu ubuntu 148 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/d-balancer/crush-compat.yaml 2024-03-27T21:42:44.787 INFO:teuthology.orchestra.run.smithi060.stdout: 407058 4 -rw-r--r-- 1 ubuntu ubuntu 286 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/d-balancer/read.yaml 2024-03-27T21:42:44.787 INFO:teuthology.orchestra.run.smithi060.stdout: 407057 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/d-balancer/on.yaml 2024-03-27T21:42:44.787 INFO:teuthology.orchestra.run.smithi060.stdout: 407068 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/objectstore -> .qa/objectstore_debug 2024-03-27T21:42:44.787 INFO:teuthology.orchestra.run.smithi060.stdout: 407062 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/msgr-failures 2024-03-27T21:42:44.787 INFO:teuthology.orchestra.run.smithi060.stdout: 407063 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.787 INFO:teuthology.orchestra.run.smithi060.stdout: 407067 4 -rw-r--r-- 1 ubuntu ubuntu 155 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/msgr-failures/osd-dispatch-delay.yaml 2024-03-27T21:42:44.787 INFO:teuthology.orchestra.run.smithi060.stdout: 407065 4 -rw-r--r-- 1 ubuntu ubuntu 237 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/msgr-failures/few.yaml 2024-03-27T21:42:44.787 INFO:teuthology.orchestra.run.smithi060.stdout: 407064 4 -rw-r--r-- 1 ubuntu ubuntu 208 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/msgr-failures/fastclose.yaml 2024-03-27T21:42:44.788 INFO:teuthology.orchestra.run.smithi060.stdout: 407066 4 -rw-r--r-- 1 ubuntu ubuntu 324 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/msgr-failures/osd-delay.yaml 2024-03-27T21:42:44.788 INFO:teuthology.orchestra.run.smithi060.stdout: 407016 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/.qa -> ../.qa/ 2024-03-27T21:42:44.788 INFO:teuthology.orchestra.run.smithi060.stdout: 407021 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/1-pg-log-overrides 2024-03-27T21:42:44.788 INFO:teuthology.orchestra.run.smithi060.stdout: 407022 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/1-pg-log-overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.788 INFO:teuthology.orchestra.run.smithi060.stdout: 407023 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/1-pg-log-overrides/normal_pg_log.yaml 2024-03-27T21:42:44.788 INFO:teuthology.orchestra.run.smithi060.stdout: 407024 0 lrwxrwxrwx 1 ubuntu ubuntu 31 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/1-pg-log-overrides/short_pg_log.yaml -> .qa/overrides/short_pg_log.yaml 2024-03-27T21:42:44.788 INFO:teuthology.orchestra.run.smithi060.stdout: 407079 0 lrwxrwxrwx 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/thrashosds-health.yaml -> .qa/tasks/thrashosds-health.yaml 2024-03-27T21:42:44.788 INFO:teuthology.orchestra.run.smithi060.stdout: 407070 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/supported-random-distro$ -> ../basic/supported-random-distro$ 2024-03-27T21:42:44.789 INFO:teuthology.orchestra.run.smithi060.stdout: 407080 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads 2024-03-27T21:42:44.789 INFO:teuthology.orchestra.run.smithi060.stdout: 407104 4 -rw-r--r-- 1 ubuntu ubuntu 316 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/snaps-few-objects-localized.yaml 2024-03-27T21:42:44.789 INFO:teuthology.orchestra.run.smithi060.stdout: 407089 4 -rw-r--r-- 1 ubuntu ubuntu 1152 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/cache.yaml 2024-03-27T21:42:44.789 INFO:teuthology.orchestra.run.smithi060.stdout: 407103 4 -rw-r--r-- 1 ubuntu ubuntu 315 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/snaps-few-objects-balanced.yaml 2024-03-27T21:42:44.789 INFO:teuthology.orchestra.run.smithi060.stdout: 407081 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.789 INFO:teuthology.orchestra.run.smithi060.stdout: 407092 4 -rw-r--r-- 1 ubuntu ubuntu 375 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/pool-snaps-few-objects.yaml 2024-03-27T21:42:44.789 INFO:teuthology.orchestra.run.smithi060.stdout: 407105 4 -rw-r--r-- 1 ubuntu ubuntu 289 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/snaps-few-objects.yaml 2024-03-27T21:42:44.789 INFO:teuthology.orchestra.run.smithi060.stdout: 407095 4 -rw-r--r-- 1 ubuntu ubuntu 480 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/radosbench.yaml 2024-03-27T21:42:44.790 INFO:teuthology.orchestra.run.smithi060.stdout: 407096 4 -rw-r--r-- 1 ubuntu ubuntu 349 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/redirect.yaml 2024-03-27T21:42:44.790 INFO:teuthology.orchestra.run.smithi060.stdout: 407097 4 -rw-r--r-- 1 ubuntu ubuntu 341 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/redirect_promote_tests.yaml 2024-03-27T21:42:44.790 INFO:teuthology.orchestra.run.smithi060.stdout: 407087 4 -rw-r--r-- 1 ubuntu ubuntu 1230 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/cache-snaps-balanced.yaml 2024-03-27T21:42:44.790 INFO:teuthology.orchestra.run.smithi060.stdout: 407106 4 -rw-r--r-- 1 ubuntu ubuntu 207 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/write_fadvise_dontneed.yaml 2024-03-27T21:42:44.790 INFO:teuthology.orchestra.run.smithi060.stdout: 407094 4 -rw-r--r-- 1 ubuntu ubuntu 971 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/radosbench-high-concurrency.yaml 2024-03-27T21:42:44.790 INFO:teuthology.orchestra.run.smithi060.stdout: 407098 4 -rw-r--r-- 1 ubuntu ubuntu 324 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/redirect_set_object.yaml 2024-03-27T21:42:44.790 INFO:teuthology.orchestra.run.smithi060.stdout: 407086 4 -rw-r--r-- 1 ubuntu ubuntu 1342 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/cache-pool-snaps.yaml 2024-03-27T21:42:44.790 INFO:teuthology.orchestra.run.smithi060.stdout: 407085 4 -rw-r--r-- 1 ubuntu ubuntu 1159 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/cache-pool-snaps-readproxy.yaml 2024-03-27T21:42:44.791 INFO:teuthology.orchestra.run.smithi060.stdout: 407101 4 -rw-r--r-- 1 ubuntu ubuntu 440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/small-objects-localized.yaml 2024-03-27T21:42:44.791 INFO:teuthology.orchestra.run.smithi060.stdout: 407090 4 -rw-r--r-- 1 ubuntu ubuntu 477 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/dedup-io-mixed.yaml 2024-03-27T21:42:44.791 INFO:teuthology.orchestra.run.smithi060.stdout: 407093 4 -rw-r--r-- 1 ubuntu ubuntu 468 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/rados_api_tests.yaml 2024-03-27T21:42:44.791 INFO:teuthology.orchestra.run.smithi060.stdout: 407091 4 -rw-r--r-- 1 ubuntu ubuntu 540 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/dedup-io-snaps.yaml 2024-03-27T21:42:44.791 INFO:teuthology.orchestra.run.smithi060.stdout: 407088 4 -rw-r--r-- 1 ubuntu ubuntu 1206 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/cache-snaps.yaml 2024-03-27T21:42:44.791 INFO:teuthology.orchestra.run.smithi060.stdout: 407099 4 -rw-r--r-- 1 ubuntu ubuntu 321 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/set-chunks-read.yaml 2024-03-27T21:42:44.791 INFO:teuthology.orchestra.run.smithi060.stdout: 407100 4 -rw-r--r-- 1 ubuntu ubuntu 439 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/small-objects-balanced.yaml 2024-03-27T21:42:44.791 INFO:teuthology.orchestra.run.smithi060.stdout: 407084 4 -rw-r--r-- 1 ubuntu ubuntu 1140 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/cache-agent-small.yaml 2024-03-27T21:42:44.792 INFO:teuthology.orchestra.run.smithi060.stdout: 407083 4 -rw-r--r-- 1 ubuntu ubuntu 1200 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/cache-agent-big.yaml 2024-03-27T21:42:44.792 INFO:teuthology.orchestra.run.smithi060.stdout: 407102 4 -rw-r--r-- 1 ubuntu ubuntu 415 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/small-objects.yaml 2024-03-27T21:42:44.792 INFO:teuthology.orchestra.run.smithi060.stdout: 407082 4 -rw-r--r-- 1 ubuntu ubuntu 375 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/workloads/admin_socket_objecter_requests.yaml 2024-03-27T21:42:44.792 INFO:teuthology.orchestra.run.smithi060.stdout: 407045 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/clusters 2024-03-27T21:42:44.792 INFO:teuthology.orchestra.run.smithi060.stdout: 407046 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/clusters/+ 2024-03-27T21:42:44.792 INFO:teuthology.orchestra.run.smithi060.stdout: 407047 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.792 INFO:teuthology.orchestra.run.smithi060.stdout: 407049 4 -rw-r--r-- 1 ubuntu ubuntu 87 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/clusters/openstack.yaml 2024-03-27T21:42:44.792 INFO:teuthology.orchestra.run.smithi060.stdout: 407048 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/clusters/fixed-4.yaml -> .qa/clusters/fixed-4.yaml 2024-03-27T21:42:44.792 INFO:teuthology.orchestra.run.smithi060.stdout: 407069 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/rados.yaml -> .qa/config/rados.yaml 2024-03-27T21:42:44.793 INFO:teuthology.orchestra.run.smithi060.stdout: 407039 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/backoff 2024-03-27T21:42:44.793 INFO:teuthology.orchestra.run.smithi060.stdout: 407043 4 -rw-r--r-- 1 ubuntu ubuntu 115 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/backoff/peering_and_degraded.yaml 2024-03-27T21:42:44.793 INFO:teuthology.orchestra.run.smithi060.stdout: 407040 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/backoff/.qa -> ../.qa/ 2024-03-27T21:42:44.793 INFO:teuthology.orchestra.run.smithi060.stdout: 407041 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/backoff/normal.yaml 2024-03-27T21:42:44.793 INFO:teuthology.orchestra.run.smithi060.stdout: 407042 4 -rw-r--r-- 1 ubuntu ubuntu 77 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/backoff/peering.yaml 2024-03-27T21:42:44.793 INFO:teuthology.orchestra.run.smithi060.stdout: 407033 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/3-scrub-overrides 2024-03-27T21:42:44.793 INFO:teuthology.orchestra.run.smithi060.stdout: 407035 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/3-scrub-overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.793 INFO:teuthology.orchestra.run.smithi060.stdout: 407037 4 -rw-r--r-- 1 ubuntu ubuntu 66 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/3-scrub-overrides/max-simultaneous-scrubs-3.yaml 2024-03-27T21:42:44.794 INFO:teuthology.orchestra.run.smithi060.stdout: 407036 4 -rw-r--r-- 1 ubuntu ubuntu 66 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/3-scrub-overrides/max-simultaneous-scrubs-1.yaml 2024-03-27T21:42:44.794 INFO:teuthology.orchestra.run.smithi060.stdout: 407038 4 -rw-r--r-- 1 ubuntu ubuntu 66 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/3-scrub-overrides/max-simultaneous-scrubs-5.yaml 2024-03-27T21:42:44.794 INFO:teuthology.orchestra.run.smithi060.stdout: 407034 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/3-scrub-overrides/$ 2024-03-27T21:42:44.794 INFO:teuthology.orchestra.run.smithi060.stdout: 407017 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/0-size-min-size-overrides 2024-03-27T21:42:44.794 INFO:teuthology.orchestra.run.smithi060.stdout: 407019 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/0-size-min-size-overrides/2-size-2-min-size.yaml -> .qa/overrides/2-size-2-min-size.yaml 2024-03-27T21:42:44.794 INFO:teuthology.orchestra.run.smithi060.stdout: 407018 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/0-size-min-size-overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.794 INFO:teuthology.orchestra.run.smithi060.stdout: 407020 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/0-size-min-size-overrides/3-size-2-min-size.yaml -> .qa/overrides/3-size-2-min-size.yaml 2024-03-27T21:42:44.794 INFO:teuthology.orchestra.run.smithi060.stdout: 407071 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/thrashers 2024-03-27T21:42:44.794 INFO:teuthology.orchestra.run.smithi060.stdout: 407072 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/thrashers/.qa -> ../.qa/ 2024-03-27T21:42:44.795 INFO:teuthology.orchestra.run.smithi060.stdout: 407078 4 -rw-r--r-- 1 ubuntu ubuntu 666 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/thrashers/pggrow.yaml 2024-03-27T21:42:44.795 INFO:teuthology.orchestra.run.smithi060.stdout: 407075 4 -rw-r--r-- 1 ubuntu ubuntu 820 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/thrashers/mapgap.yaml 2024-03-27T21:42:44.795 INFO:teuthology.orchestra.run.smithi060.stdout: 407073 4 -rw-r--r-- 1 ubuntu ubuntu 742 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/thrashers/careful.yaml 2024-03-27T21:42:44.795 INFO:teuthology.orchestra.run.smithi060.stdout: 407077 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/thrashers/none.yaml 2024-03-27T21:42:44.795 INFO:teuthology.orchestra.run.smithi060.stdout: 407074 4 -rw-r--r-- 1 ubuntu ubuntu 800 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/thrashers/default.yaml 2024-03-27T21:42:44.795 INFO:teuthology.orchestra.run.smithi060.stdout: 407076 4 -rw-r--r-- 1 ubuntu ubuntu 554 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/thrashers/morepggrow.yaml 2024-03-27T21:42:44.795 INFO:teuthology.orchestra.run.smithi060.stdout: 407060 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/mon_election -> .qa/mon_election 2024-03-27T21:42:44.796 INFO:teuthology.orchestra.run.smithi060.stdout: 407025 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/2-recovery-overrides 2024-03-27T21:42:44.796 INFO:teuthology.orchestra.run.smithi060.stdout: 407031 4 -rw-r--r-- 1 ubuntu ubuntu 82 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/2-recovery-overrides/more-async-recovery.yaml 2024-03-27T21:42:44.796 INFO:teuthology.orchestra.run.smithi060.stdout: 407027 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/2-recovery-overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.796 INFO:teuthology.orchestra.run.smithi060.stdout: 407030 4 -rw-r--r-- 1 ubuntu ubuntu 138 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/2-recovery-overrides/more-async-partial-recovery.yaml 2024-03-27T21:42:44.796 INFO:teuthology.orchestra.run.smithi060.stdout: 407032 4 -rw-r--r-- 1 ubuntu ubuntu 99 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/2-recovery-overrides/more-partial-recovery.yaml 2024-03-27T21:42:44.796 INFO:teuthology.orchestra.run.smithi060.stdout: 407026 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/2-recovery-overrides/$ 2024-03-27T21:42:44.796 INFO:teuthology.orchestra.run.smithi060.stdout: 407029 0 lrwxrwxrwx 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/2-recovery-overrides/more-active-recovery.yaml -> .qa/overrides/more-active-recovery.yaml 2024-03-27T21:42:44.797 INFO:teuthology.orchestra.run.smithi060.stdout: 407028 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash/2-recovery-overrides/default.yaml 2024-03-27T21:42:44.797 INFO:teuthology.orchestra.run.smithi060.stdout: 406812 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big 2024-03-27T21:42:44.797 INFO:teuthology.orchestra.run.smithi060.stdout: 406815 0 lrwxrwxrwx 1 ubuntu ubuntu 19 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/ceph.yaml -> ../thrash/ceph.yaml 2024-03-27T21:42:44.797 INFO:teuthology.orchestra.run.smithi060.stdout: 406813 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/% 2024-03-27T21:42:44.797 INFO:teuthology.orchestra.run.smithi060.stdout: 406823 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/objectstore -> .qa/objectstore_debug 2024-03-27T21:42:44.797 INFO:teuthology.orchestra.run.smithi060.stdout: 406822 0 lrwxrwxrwx 1 ubuntu ubuntu 23 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/msgr-failures -> ../thrash/msgr-failures 2024-03-27T21:42:44.797 INFO:teuthology.orchestra.run.smithi060.stdout: 406814 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/.qa -> ../.qa/ 2024-03-27T21:42:44.797 INFO:teuthology.orchestra.run.smithi060.stdout: 406835 0 lrwxrwxrwx 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/thrashosds-health.yaml -> .qa/tasks/thrashosds-health.yaml 2024-03-27T21:42:44.798 INFO:teuthology.orchestra.run.smithi060.stdout: 406826 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/supported-random-distro$ -> ../basic/supported-random-distro$ 2024-03-27T21:42:44.798 INFO:teuthology.orchestra.run.smithi060.stdout: 406836 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/workloads 2024-03-27T21:42:44.798 INFO:teuthology.orchestra.run.smithi060.stdout: 406837 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.798 INFO:teuthology.orchestra.run.smithi060.stdout: 406839 0 lrwxrwxrwx 1 ubuntu ubuntu 54 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/workloads/ec-rados-plugin=jerasure-k=4-m=2.yaml -> .qa/erasure-code/ec-rados-plugin=jerasure-k=4-m=2.yaml 2024-03-27T21:42:44.798 INFO:teuthology.orchestra.run.smithi060.stdout: 406838 0 lrwxrwxrwx 1 ubuntu ubuntu 50 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/workloads/ec-rados-plugin=clay-k=4-m=2.yaml -> .qa/erasure-code/ec-rados-plugin=clay-k=4-m=2.yaml 2024-03-27T21:42:44.798 INFO:teuthology.orchestra.run.smithi060.stdout: 406840 0 lrwxrwxrwx 1 ubuntu ubuntu 53 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/workloads/ec-rados-plugin=lrc-k=4-m=2-l=3.yaml -> .qa/erasure-code/ec-rados-plugin=lrc-k=4-m=2-l=3.yaml 2024-03-27T21:42:44.798 INFO:teuthology.orchestra.run.smithi060.stdout: 406824 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/rados.yaml -> .qa/config/rados.yaml 2024-03-27T21:42:44.798 INFO:teuthology.orchestra.run.smithi060.stdout: 406825 0 lrwxrwxrwx 1 ubuntu ubuntu 30 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/recovery-overrides -> ../thrash/2-recovery-overrides 2024-03-27T21:42:44.799 INFO:teuthology.orchestra.run.smithi060.stdout: 406827 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/thrashers 2024-03-27T21:42:44.799 INFO:teuthology.orchestra.run.smithi060.stdout: 406828 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/thrashers/.qa -> ../.qa/ 2024-03-27T21:42:44.799 INFO:teuthology.orchestra.run.smithi060.stdout: 406834 4 -rw-r--r-- 1 ubuntu ubuntu 366 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/thrashers/pggrow.yaml 2024-03-27T21:42:44.799 INFO:teuthology.orchestra.run.smithi060.stdout: 406832 4 -rw-r--r-- 1 ubuntu ubuntu 531 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/thrashers/mapgap.yaml 2024-03-27T21:42:44.799 INFO:teuthology.orchestra.run.smithi060.stdout: 406831 4 -rw-r--r-- 1 ubuntu ubuntu 498 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/thrashers/fastread.yaml 2024-03-27T21:42:44.799 INFO:teuthology.orchestra.run.smithi060.stdout: 406829 4 -rw-r--r-- 1 ubuntu ubuntu 499 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/thrashers/careful.yaml 2024-03-27T21:42:44.799 INFO:teuthology.orchestra.run.smithi060.stdout: 406830 4 -rw-r--r-- 1 ubuntu ubuntu 529 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/thrashers/default.yaml 2024-03-27T21:42:44.799 INFO:teuthology.orchestra.run.smithi060.stdout: 406833 4 -rw-r--r-- 1 ubuntu ubuntu 366 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/thrashers/morepggrow.yaml 2024-03-27T21:42:44.800 INFO:teuthology.orchestra.run.smithi060.stdout: 406816 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/cluster 2024-03-27T21:42:44.800 INFO:teuthology.orchestra.run.smithi060.stdout: 406817 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/cluster/+ 2024-03-27T21:42:44.800 INFO:teuthology.orchestra.run.smithi060.stdout: 406818 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/cluster/.qa -> ../.qa/ 2024-03-27T21:42:44.800 INFO:teuthology.orchestra.run.smithi060.stdout: 406820 4 -rw-r--r-- 1 ubuntu ubuntu 87 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/cluster/openstack.yaml 2024-03-27T21:42:44.800 INFO:teuthology.orchestra.run.smithi060.stdout: 406819 4 -rw-r--r-- 1 ubuntu ubuntu 140 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/cluster/12-osds.yaml 2024-03-27T21:42:44.800 INFO:teuthology.orchestra.run.smithi060.stdout: 406821 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-big/mon_election -> .qa/mon_election 2024-03-27T21:42:44.800 INFO:teuthology.orchestra.run.smithi060.stdout: 406860 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-isa 2024-03-27T21:42:44.800 INFO:teuthology.orchestra.run.smithi060.stdout: 406866 0 lrwxrwxrwx 1 ubuntu ubuntu 19 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-isa/ceph.yaml -> ../thrash/ceph.yaml 2024-03-27T21:42:44.800 INFO:teuthology.orchestra.run.smithi060.stdout: 406861 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-isa/% 2024-03-27T21:42:44.801 INFO:teuthology.orchestra.run.smithi060.stdout: 406863 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-isa/arch 2024-03-27T21:42:44.801 INFO:teuthology.orchestra.run.smithi060.stdout: 406865 4 -rw-r--r-- 1 ubuntu ubuntu 13 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-isa/arch/x86_64.yaml 2024-03-27T21:42:44.801 INFO:teuthology.orchestra.run.smithi060.stdout: 406864 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-isa/arch/.qa -> ../.qa/ 2024-03-27T21:42:44.801 INFO:teuthology.orchestra.run.smithi060.stdout: 406870 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-isa/objectstore -> .qa/objectstore_debug 2024-03-27T21:42:44.801 INFO:teuthology.orchestra.run.smithi060.stdout: 406869 0 lrwxrwxrwx 1 ubuntu ubuntu 23 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-isa/msgr-failures -> ../thrash/msgr-failures 2024-03-27T21:42:44.801 INFO:teuthology.orchestra.run.smithi060.stdout: 406862 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-isa/.qa -> ../.qa/ 2024-03-27T21:42:44.801 INFO:teuthology.orchestra.run.smithi060.stdout: 406875 0 lrwxrwxrwx 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-isa/thrashosds-health.yaml -> .qa/tasks/thrashosds-health.yaml 2024-03-27T21:42:44.802 INFO:teuthology.orchestra.run.smithi060.stdout: 406873 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-isa/supported-random-distro$ -> ../basic/supported-random-distro$ 2024-03-27T21:42:44.802 INFO:teuthology.orchestra.run.smithi060.stdout: 406876 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-isa/workloads 2024-03-27T21:42:44.802 INFO:teuthology.orchestra.run.smithi060.stdout: 406878 0 lrwxrwxrwx 1 ubuntu ubuntu 49 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-isa/workloads/ec-rados-plugin=isa-k=2-m=1.yaml -> .qa/erasure-code/ec-rados-plugin=isa-k=2-m=1.yaml 2024-03-27T21:42:44.802 INFO:teuthology.orchestra.run.smithi060.stdout: 406877 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-isa/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.802 INFO:teuthology.orchestra.run.smithi060.stdout: 406867 0 lrwxrwxrwx 1 ubuntu ubuntu 18 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-isa/clusters -> ../thrash/clusters 2024-03-27T21:42:44.802 INFO:teuthology.orchestra.run.smithi060.stdout: 406871 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-isa/rados.yaml -> .qa/config/rados.yaml 2024-03-27T21:42:44.802 INFO:teuthology.orchestra.run.smithi060.stdout: 406872 0 lrwxrwxrwx 1 ubuntu ubuntu 30 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-isa/recovery-overrides -> ../thrash/2-recovery-overrides 2024-03-27T21:42:44.802 INFO:teuthology.orchestra.run.smithi060.stdout: 406874 0 lrwxrwxrwx 1 ubuntu ubuntu 19 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-isa/thrashers -> ../thrash/thrashers 2024-03-27T21:42:44.803 INFO:teuthology.orchestra.run.smithi060.stdout: 406868 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code-isa/mon_election -> .qa/mon_election 2024-03-27T21:42:44.803 INFO:teuthology.orchestra.run.smithi060.stdout: 406922 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code 2024-03-27T21:42:44.803 INFO:teuthology.orchestra.run.smithi060.stdout: 406927 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/fast 2024-03-27T21:42:44.803 INFO:teuthology.orchestra.run.smithi060.stdout: 406928 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/fast/.qa -> ../.qa/ 2024-03-27T21:42:44.803 INFO:teuthology.orchestra.run.smithi060.stdout: 406930 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/fast/normal.yaml 2024-03-27T21:42:44.803 INFO:teuthology.orchestra.run.smithi060.stdout: 406929 4 -rw-r--r-- 1 ubuntu ubuntu 87 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/fast/fast.yaml 2024-03-27T21:42:44.803 INFO:teuthology.orchestra.run.smithi060.stdout: 406925 4 -rw-r--r-- 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/ceph.yaml 2024-03-27T21:42:44.803 INFO:teuthology.orchestra.run.smithi060.stdout: 406923 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/% 2024-03-27T21:42:44.803 INFO:teuthology.orchestra.run.smithi060.stdout: 406933 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/objectstore -> .qa/objectstore_debug 2024-03-27T21:42:44.804 INFO:teuthology.orchestra.run.smithi060.stdout: 406932 0 lrwxrwxrwx 1 ubuntu ubuntu 23 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/msgr-failures -> ../thrash/msgr-failures 2024-03-27T21:42:44.804 INFO:teuthology.orchestra.run.smithi060.stdout: 406924 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/.qa -> ../.qa/ 2024-03-27T21:42:44.804 INFO:teuthology.orchestra.run.smithi060.stdout: 406945 0 lrwxrwxrwx 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/thrashosds-health.yaml -> .qa/tasks/thrashosds-health.yaml 2024-03-27T21:42:44.804 INFO:teuthology.orchestra.run.smithi060.stdout: 406936 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/supported-random-distro$ -> ../basic/supported-random-distro$ 2024-03-27T21:42:44.804 INFO:teuthology.orchestra.run.smithi060.stdout: 406946 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/workloads 2024-03-27T21:42:44.804 INFO:teuthology.orchestra.run.smithi060.stdout: 406947 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.804 INFO:teuthology.orchestra.run.smithi060.stdout: 406951 4 lrwxrwxrwx 1 ubuntu ubuntu 60 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/workloads/ec-rados-plugin=jerasure-k=8-m=6-crush.yaml -> .qa/erasure-code/ec-rados-plugin=jerasure-k=8-m=6-crush.yaml 2024-03-27T21:42:44.804 INFO:teuthology.orchestra.run.smithi060.stdout: 406948 0 lrwxrwxrwx 1 ubuntu ubuntu 50 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/workloads/ec-rados-plugin=clay-k=4-m=2.yaml -> .qa/erasure-code/ec-rados-plugin=clay-k=4-m=2.yaml 2024-03-27T21:42:44.805 INFO:teuthology.orchestra.run.smithi060.stdout: 406950 0 lrwxrwxrwx 1 ubuntu ubuntu 54 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/workloads/ec-rados-plugin=jerasure-k=3-m=1.yaml -> .qa/erasure-code/ec-rados-plugin=jerasure-k=3-m=1.yaml 2024-03-27T21:42:44.805 INFO:teuthology.orchestra.run.smithi060.stdout: 406952 4 -rw-r--r-- 1 ubuntu ubuntu 536 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/workloads/ec-radosbench.yaml 2024-03-27T21:42:44.805 INFO:teuthology.orchestra.run.smithi060.stdout: 406954 4 -rw-r--r-- 1 ubuntu ubuntu 371 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/workloads/ec-small-objects-fast-read.yaml 2024-03-27T21:42:44.805 INFO:teuthology.orchestra.run.smithi060.stdout: 406955 4 -rw-r--r-- 1 ubuntu ubuntu 227 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/workloads/ec-small-objects-many-deletes.yaml 2024-03-27T21:42:44.805 INFO:teuthology.orchestra.run.smithi060.stdout: 406949 0 lrwxrwxrwx 1 ubuntu ubuntu 54 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/workloads/ec-rados-plugin=jerasure-k=2-m=1.yaml -> .qa/erasure-code/ec-rados-plugin=jerasure-k=2-m=1.yaml 2024-03-27T21:42:44.805 INFO:teuthology.orchestra.run.smithi060.stdout: 406956 4 -rw-r--r-- 1 ubuntu ubuntu 351 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/workloads/ec-small-objects.yaml 2024-03-27T21:42:44.805 INFO:teuthology.orchestra.run.smithi060.stdout: 406953 4 -rw-r--r-- 1 ubuntu ubuntu 376 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/workloads/ec-small-objects-balanced.yaml 2024-03-27T21:42:44.805 INFO:teuthology.orchestra.run.smithi060.stdout: 406926 0 lrwxrwxrwx 1 ubuntu ubuntu 18 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/clusters -> ../thrash/clusters 2024-03-27T21:42:44.806 INFO:teuthology.orchestra.run.smithi060.stdout: 406934 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/rados.yaml -> .qa/config/rados.yaml 2024-03-27T21:42:44.806 INFO:teuthology.orchestra.run.smithi060.stdout: 406935 0 lrwxrwxrwx 1 ubuntu ubuntu 30 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/recovery-overrides -> ../thrash/2-recovery-overrides 2024-03-27T21:42:44.806 INFO:teuthology.orchestra.run.smithi060.stdout: 406937 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/thrashers 2024-03-27T21:42:44.806 INFO:teuthology.orchestra.run.smithi060.stdout: 406942 4 -rw-r--r-- 1 ubuntu ubuntu 535 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/thrashers/minsize_recovery.yaml 2024-03-27T21:42:44.806 INFO:teuthology.orchestra.run.smithi060.stdout: 406938 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/thrashers/.qa -> ../.qa/ 2024-03-27T21:42:44.806 INFO:teuthology.orchestra.run.smithi060.stdout: 406944 4 -rw-r--r-- 1 ubuntu ubuntu 366 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/thrashers/pggrow.yaml 2024-03-27T21:42:44.806 INFO:teuthology.orchestra.run.smithi060.stdout: 406941 4 -rw-r--r-- 1 ubuntu ubuntu 498 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/thrashers/fastread.yaml 2024-03-27T21:42:44.806 INFO:teuthology.orchestra.run.smithi060.stdout: 406939 4 -rw-r--r-- 1 ubuntu ubuntu 480 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/thrashers/careful.yaml 2024-03-27T21:42:44.806 INFO:teuthology.orchestra.run.smithi060.stdout: 406940 4 -rw-r--r-- 1 ubuntu ubuntu 510 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/thrashers/default.yaml 2024-03-27T21:42:44.807 INFO:teuthology.orchestra.run.smithi060.stdout: 406943 4 -rw-r--r-- 1 ubuntu ubuntu 366 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/thrashers/morepggrow.yaml 2024-03-27T21:42:44.807 INFO:teuthology.orchestra.run.smithi060.stdout: 406931 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/thrash-erasure-code/mon_election -> .qa/mon_election 2024-03-27T21:42:44.807 INFO:teuthology.orchestra.run.smithi060.stdout: 406539 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/cephadm 2024-03-27T21:42:44.807 INFO:teuthology.orchestra.run.smithi060.stdout: 406540 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/cephadm/.qa -> ../.qa/ 2024-03-27T21:42:44.807 INFO:teuthology.orchestra.run.smithi060.stdout: 406543 0 lrwxrwxrwx 1 ubuntu ubuntu 41 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/cephadm/smoke-singlehost -> .qa/suites/orch/cephadm/smoke-singlehost/ 2024-03-27T21:42:44.807 INFO:teuthology.orchestra.run.smithi060.stdout: 406541 0 lrwxrwxrwx 1 ubuntu ubuntu 29 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/cephadm/osds -> .qa/suites/orch/cephadm/osds/ 2024-03-27T21:42:44.807 INFO:teuthology.orchestra.run.smithi060.stdout: 406544 0 lrwxrwxrwx 1 ubuntu ubuntu 34 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/cephadm/workunits -> .qa/suites/orch/cephadm/workunits/ 2024-03-27T21:42:44.807 INFO:teuthology.orchestra.run.smithi060.stdout: 406542 0 lrwxrwxrwx 1 ubuntu ubuntu 29 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/cephadm/smoke -> .qa/suites/orch/cephadm/smoke 2024-03-27T21:42:44.808 INFO:teuthology.orchestra.run.smithi060.stdout: 407110 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/valgrind-leaks 2024-03-27T21:42:44.808 INFO:teuthology.orchestra.run.smithi060.stdout: 407114 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/valgrind-leaks/2-inject-leak 2024-03-27T21:42:44.808 INFO:teuthology.orchestra.run.smithi060.stdout: 407115 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/valgrind-leaks/2-inject-leak/.qa -> ../.qa/ 2024-03-27T21:42:44.808 INFO:teuthology.orchestra.run.smithi060.stdout: 407116 4 -rw-r--r-- 1 ubuntu ubuntu 119 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/valgrind-leaks/2-inject-leak/mon.yaml 2024-03-27T21:42:44.808 INFO:teuthology.orchestra.run.smithi060.stdout: 407118 4 -rw-r--r-- 1 ubuntu ubuntu 119 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/valgrind-leaks/2-inject-leak/osd.yaml 2024-03-27T21:42:44.808 INFO:teuthology.orchestra.run.smithi060.stdout: 407117 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/valgrind-leaks/2-inject-leak/none.yaml 2024-03-27T21:42:44.808 INFO:teuthology.orchestra.run.smithi060.stdout: 407111 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/valgrind-leaks/% 2024-03-27T21:42:44.808 INFO:teuthology.orchestra.run.smithi060.stdout: 407112 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/valgrind-leaks/.qa -> ../.qa/ 2024-03-27T21:42:44.809 INFO:teuthology.orchestra.run.smithi060.stdout: 407113 4 -rw-r--r-- 1 ubuntu ubuntu 685 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/valgrind-leaks/1-start.yaml 2024-03-27T21:42:44.809 INFO:teuthology.orchestra.run.smithi060.stdout: 407119 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/valgrind-leaks/centos_latest.yaml -> .qa/distros/supported/centos_latest.yaml 2024-03-27T21:42:44.809 INFO:teuthology.orchestra.run.smithi060.stdout: 406507 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic 2024-03-27T21:42:44.809 INFO:teuthology.orchestra.run.smithi060.stdout: 406517 0 lrwxrwxrwx 1 ubuntu ubuntu 8 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/msgr -> .qa/msgr 2024-03-27T21:42:44.809 INFO:teuthology.orchestra.run.smithi060.stdout: 406510 4 -rw-r--r-- 1 ubuntu ubuntu 380 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/ceph.yaml 2024-03-27T21:42:44.809 INFO:teuthology.orchestra.run.smithi060.stdout: 406508 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/% 2024-03-27T21:42:44.809 INFO:teuthology.orchestra.run.smithi060.stdout: 406522 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/objectstore -> .qa/objectstore_debug 2024-03-27T21:42:44.809 INFO:teuthology.orchestra.run.smithi060.stdout: 406518 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/msgr-failures 2024-03-27T21:42:44.810 INFO:teuthology.orchestra.run.smithi060.stdout: 406519 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.810 INFO:teuthology.orchestra.run.smithi060.stdout: 406520 4 -rw-r--r-- 1 ubuntu ubuntu 177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/msgr-failures/few.yaml 2024-03-27T21:42:44.810 INFO:teuthology.orchestra.run.smithi060.stdout: 406521 4 -rw-r--r-- 1 ubuntu ubuntu 177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/msgr-failures/many.yaml 2024-03-27T21:42:44.810 INFO:teuthology.orchestra.run.smithi060.stdout: 406509 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/.qa -> ../.qa/ 2024-03-27T21:42:44.810 INFO:teuthology.orchestra.run.smithi060.stdout: 406524 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.810 INFO:teuthology.orchestra.run.smithi060.stdout: 406511 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/clusters 2024-03-27T21:42:44.810 INFO:teuthology.orchestra.run.smithi060.stdout: 406514 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/clusters/fixed-2.yaml -> .qa/clusters/fixed-2.yaml 2024-03-27T21:42:44.810 INFO:teuthology.orchestra.run.smithi060.stdout: 406512 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/clusters/+ 2024-03-27T21:42:44.811 INFO:teuthology.orchestra.run.smithi060.stdout: 406513 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.811 INFO:teuthology.orchestra.run.smithi060.stdout: 406515 4 -rw-r--r-- 1 ubuntu ubuntu 87 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/clusters/openstack.yaml 2024-03-27T21:42:44.811 INFO:teuthology.orchestra.run.smithi060.stdout: 406523 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/rados.yaml -> .qa/config/rados.yaml 2024-03-27T21:42:44.811 INFO:teuthology.orchestra.run.smithi060.stdout: 406525 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/tasks 2024-03-27T21:42:44.811 INFO:teuthology.orchestra.run.smithi060.stdout: 406527 4 -rw-r--r-- 1 ubuntu ubuntu 662 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/tasks/libcephsqlite.yaml 2024-03-27T21:42:44.811 INFO:teuthology.orchestra.run.smithi060.stdout: 406535 4 -rw-r--r-- 1 ubuntu ubuntu 204 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/tasks/rados_workunit_loadgen_mostlyread.yaml 2024-03-27T21:42:44.811 INFO:teuthology.orchestra.run.smithi060.stdout: 406538 4 -rw-r--r-- 1 ubuntu ubuntu 889 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/tasks/scrub_test.yaml 2024-03-27T21:42:44.811 INFO:teuthology.orchestra.run.smithi060.stdout: 406526 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.811 INFO:teuthology.orchestra.run.smithi060.stdout: 406534 4 -rw-r--r-- 1 ubuntu ubuntu 197 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/tasks/rados_workunit_loadgen_mix.yaml 2024-03-27T21:42:44.812 INFO:teuthology.orchestra.run.smithi060.stdout: 406529 4 -rw-r--r-- 1 ubuntu ubuntu 248 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/tasks/rados_cls_all.yaml 2024-03-27T21:42:44.812 INFO:teuthology.orchestra.run.smithi060.stdout: 406531 4 -rw-r--r-- 1 ubuntu ubuntu 234 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/tasks/rados_stress_watch.yaml 2024-03-27T21:42:44.812 INFO:teuthology.orchestra.run.smithi060.stdout: 406537 4 -rw-r--r-- 1 ubuntu ubuntu 959 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/tasks/repair_test.yaml 2024-03-27T21:42:44.812 INFO:teuthology.orchestra.run.smithi060.stdout: 406528 4 -rw-r--r-- 1 ubuntu ubuntu 581 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/tasks/rados_api_tests.yaml 2024-03-27T21:42:44.812 INFO:teuthology.orchestra.run.smithi060.stdout: 406536 4 -rw-r--r-- 1 ubuntu ubuntu 371 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/tasks/readwrite.yaml 2024-03-27T21:42:44.812 INFO:teuthology.orchestra.run.smithi060.stdout: 406530 4 -rw-r--r-- 1 ubuntu ubuntu 407 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/tasks/rados_python.yaml 2024-03-27T21:42:44.812 INFO:teuthology.orchestra.run.smithi060.stdout: 406533 4 -rw-r--r-- 1 ubuntu ubuntu 197 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/tasks/rados_workunit_loadgen_big.yaml 2024-03-27T21:42:44.812 INFO:teuthology.orchestra.run.smithi060.stdout: 406532 4 -rw-r--r-- 1 ubuntu ubuntu 214 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/tasks/rados_striper.yaml 2024-03-27T21:42:44.812 INFO:teuthology.orchestra.run.smithi060.stdout: 406516 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rados/basic/mon_election -> .qa/mon_election 2024-03-27T21:42:44.813 INFO:teuthology.orchestra.run.smithi060.stdout: 404687 4 drwxr-xr-x 9 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics 2024-03-27T21:42:44.813 INFO:teuthology.orchestra.run.smithi060.stdout: 404688 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/% 2024-03-27T21:42:44.813 INFO:teuthology.orchestra.run.smithi060.stdout: 404690 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/0-clusters 2024-03-27T21:42:44.813 INFO:teuthology.orchestra.run.smithi060.stdout: 404691 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/0-clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.813 INFO:teuthology.orchestra.run.smithi060.stdout: 404692 4 -rw-r--r-- 1 ubuntu ubuntu 261 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/0-clusters/3-node.yaml 2024-03-27T21:42:44.813 INFO:teuthology.orchestra.run.smithi060.stdout: 404689 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/.qa -> ../.qa/ 2024-03-27T21:42:44.813 INFO:teuthology.orchestra.run.smithi060.stdout: 404706 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/4-epel 2024-03-27T21:42:44.813 INFO:teuthology.orchestra.run.smithi060.stdout: 404708 4 -rw-r--r-- 1 ubuntu ubuntu 108 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/4-epel/no_epel.yaml 2024-03-27T21:42:44.814 INFO:teuthology.orchestra.run.smithi060.stdout: 404707 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/4-epel/.qa -> ../.qa/ 2024-03-27T21:42:44.814 INFO:teuthology.orchestra.run.smithi060.stdout: 404709 4 -rw-r--r-- 1 ubuntu ubuntu 105 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/4-epel/use_epel.yaml 2024-03-27T21:42:44.814 INFO:teuthology.orchestra.run.smithi060.stdout: 404693 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/1-distros 2024-03-27T21:42:44.814 INFO:teuthology.orchestra.run.smithi060.stdout: 404694 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/1-distros/.qa -> ../.qa/ 2024-03-27T21:42:44.814 INFO:teuthology.orchestra.run.smithi060.stdout: 404696 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/1-distros/ubuntu_latest.yaml -> .qa/distros/supported/ubuntu_latest.yaml 2024-03-27T21:42:44.814 INFO:teuthology.orchestra.run.smithi060.stdout: 404695 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/1-distros/centos_latest.yaml -> .qa/distros/supported/centos_latest.yaml 2024-03-27T21:42:44.814 INFO:teuthology.orchestra.run.smithi060.stdout: 404697 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/2-ceph 2024-03-27T21:42:44.815 INFO:teuthology.orchestra.run.smithi060.stdout: 404698 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/2-ceph/.qa -> ../.qa/ 2024-03-27T21:42:44.815 INFO:teuthology.orchestra.run.smithi060.stdout: 404699 4 -rw-r--r-- 1 ubuntu ubuntu 822 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/2-ceph/ceph_ansible.yaml 2024-03-27T21:42:44.815 INFO:teuthology.orchestra.run.smithi060.stdout: 404700 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/3-ceph-config 2024-03-27T21:42:44.815 INFO:teuthology.orchestra.run.smithi060.stdout: 404702 4 -rw-r--r-- 1 ubuntu ubuntu 137 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/3-ceph-config/bluestore_with_dmcrypt.yaml 2024-03-27T21:42:44.815 INFO:teuthology.orchestra.run.smithi060.stdout: 404701 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/3-ceph-config/.qa -> ../.qa/ 2024-03-27T21:42:44.815 INFO:teuthology.orchestra.run.smithi060.stdout: 404705 4 -rw-r--r-- 1 ubuntu ubuntu 91 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/3-ceph-config/dmcrypt_on.yaml 2024-03-27T21:42:44.815 INFO:teuthology.orchestra.run.smithi060.stdout: 404703 4 -rw-r--r-- 1 ubuntu ubuntu 144 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/3-ceph-config/bluestore_without_dmcrypt.yaml 2024-03-27T21:42:44.815 INFO:teuthology.orchestra.run.smithi060.stdout: 404704 4 -rw-r--r-- 1 ubuntu ubuntu 95 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/3-ceph-config/dmcrypt_off.yaml 2024-03-27T21:42:44.815 INFO:teuthology.orchestra.run.smithi060.stdout: 404710 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/5-containers 2024-03-27T21:42:44.816 INFO:teuthology.orchestra.run.smithi060.stdout: 404711 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/5-containers/.qa -> ../.qa/ 2024-03-27T21:42:44.816 INFO:teuthology.orchestra.run.smithi060.stdout: 404713 4 -rw-r--r-- 1 ubuntu ubuntu 202 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/5-containers/no_containers.yaml 2024-03-27T21:42:44.816 INFO:teuthology.orchestra.run.smithi060.stdout: 404712 4 -rw-r--r-- 1 ubuntu ubuntu 205 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/5-containers/containerized.yaml 2024-03-27T21:42:44.816 INFO:teuthology.orchestra.run.smithi060.stdout: 404714 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/6-tasks 2024-03-27T21:42:44.816 INFO:teuthology.orchestra.run.smithi060.stdout: 404715 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/6-tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.816 INFO:teuthology.orchestra.run.smithi060.stdout: 404716 4 -rw-r--r-- 1 ubuntu ubuntu 83 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/cephmetrics/6-tasks/cephmetrics.yaml 2024-03-27T21:42:44.816 INFO:teuthology.orchestra.run.smithi060.stdout: 404601 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/.qa -> ../.qa/ 2024-03-27T21:42:44.816 INFO:teuthology.orchestra.run.smithi060.stdout: 406458 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/perf-basic 2024-03-27T21:42:44.817 INFO:teuthology.orchestra.run.smithi060.stdout: 406461 4 -rw-r--r-- 1 ubuntu ubuntu 494 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/perf-basic/ceph.yaml 2024-03-27T21:42:44.817 INFO:teuthology.orchestra.run.smithi060.stdout: 406459 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/perf-basic/% 2024-03-27T21:42:44.817 INFO:teuthology.orchestra.run.smithi060.stdout: 406462 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/perf-basic/objectstore 2024-03-27T21:42:44.817 INFO:teuthology.orchestra.run.smithi060.stdout: 406463 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/perf-basic/objectstore/.qa -> ../.qa/ 2024-03-27T21:42:44.817 INFO:teuthology.orchestra.run.smithi060.stdout: 406464 4 -rw-r--r-- 1 ubuntu ubuntu 327 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/perf-basic/objectstore/bluestore.yaml 2024-03-27T21:42:44.817 INFO:teuthology.orchestra.run.smithi060.stdout: 406460 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/perf-basic/.qa -> ../.qa/ 2024-03-27T21:42:44.817 INFO:teuthology.orchestra.run.smithi060.stdout: 406468 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/perf-basic/ubuntu_latest.yaml -> .qa/distros/supported/ubuntu_latest.yaml 2024-03-27T21:42:44.817 INFO:teuthology.orchestra.run.smithi060.stdout: 406469 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/perf-basic/workloads 2024-03-27T21:42:44.817 INFO:teuthology.orchestra.run.smithi060.stdout: 406472 4 -rw-r--r-- 1 ubuntu ubuntu 591 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/perf-basic/workloads/fio_4K_rand_write.yaml 2024-03-27T21:42:44.818 INFO:teuthology.orchestra.run.smithi060.stdout: 406470 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/perf-basic/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.818 INFO:teuthology.orchestra.run.smithi060.stdout: 406473 4 -rw-r--r-- 1 ubuntu ubuntu 580 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/perf-basic/workloads/radosbench_4K_write.yaml 2024-03-27T21:42:44.818 INFO:teuthology.orchestra.run.smithi060.stdout: 406471 4 -rw-r--r-- 1 ubuntu ubuntu 642 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/perf-basic/workloads/client_endpoint_rbd_4K_rand_write.yaml 2024-03-27T21:42:44.818 INFO:teuthology.orchestra.run.smithi060.stdout: 406465 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/perf-basic/settings 2024-03-27T21:42:44.818 INFO:teuthology.orchestra.run.smithi060.stdout: 406466 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/perf-basic/settings/.qa -> ../.qa/ 2024-03-27T21:42:44.818 INFO:teuthology.orchestra.run.smithi060.stdout: 406467 4 -rw-r--r-- 1 ubuntu ubuntu 2106 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/perf-basic/settings/optimized.yaml 2024-03-27T21:42:44.818 INFO:teuthology.orchestra.run.smithi060.stdout: 408025 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba 2024-03-27T21:42:44.819 INFO:teuthology.orchestra.run.smithi060.stdout: 408026 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba/% 2024-03-27T21:42:44.819 INFO:teuthology.orchestra.run.smithi060.stdout: 408040 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba/objectstore -> .qa/objectstore 2024-03-27T21:42:44.819 INFO:teuthology.orchestra.run.smithi060.stdout: 408027 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba/.qa -> ../.qa/ 2024-03-27T21:42:44.819 INFO:teuthology.orchestra.run.smithi060.stdout: 408031 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba/install 2024-03-27T21:42:44.819 INFO:teuthology.orchestra.run.smithi060.stdout: 408033 4 -rw-r--r-- 1 ubuntu ubuntu 182 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba/install/install.yaml 2024-03-27T21:42:44.819 INFO:teuthology.orchestra.run.smithi060.stdout: 408032 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba/install/.qa -> ../.qa/ 2024-03-27T21:42:44.819 INFO:teuthology.orchestra.run.smithi060.stdout: 408034 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba/mount 2024-03-27T21:42:44.819 INFO:teuthology.orchestra.run.smithi060.stdout: 408035 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba/mount/.qa -> ../.qa/ 2024-03-27T21:42:44.820 INFO:teuthology.orchestra.run.smithi060.stdout: 408037 4 -rw-r--r-- 1 ubuntu ubuntu 204 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba/mount/kclient.yaml 2024-03-27T21:42:44.820 INFO:teuthology.orchestra.run.smithi060.stdout: 408039 4 -rw-r--r-- 1 ubuntu ubuntu 82 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba/mount/noceph.yaml 2024-03-27T21:42:44.820 INFO:teuthology.orchestra.run.smithi060.stdout: 408038 4 -rw-r--r-- 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba/mount/native.yaml 2024-03-27T21:42:44.820 INFO:teuthology.orchestra.run.smithi060.stdout: 408036 4 -rw-r--r-- 1 ubuntu ubuntu 84 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba/mount/fuse.yaml 2024-03-27T21:42:44.820 INFO:teuthology.orchestra.run.smithi060.stdout: 408028 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba/clusters 2024-03-27T21:42:44.820 INFO:teuthology.orchestra.run.smithi060.stdout: 408030 4 -rw-r--r-- 1 ubuntu ubuntu 172 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba/clusters/samba-basic.yaml 2024-03-27T21:42:44.820 INFO:teuthology.orchestra.run.smithi060.stdout: 408029 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.820 INFO:teuthology.orchestra.run.smithi060.stdout: 408041 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba/workload 2024-03-27T21:42:44.821 INFO:teuthology.orchestra.run.smithi060.stdout: 408042 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba/workload/.qa -> ../.qa/ 2024-03-27T21:42:44.821 INFO:teuthology.orchestra.run.smithi060.stdout: 408043 4 -rw-r--r-- 1 ubuntu ubuntu 121 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba/workload/cifs-dbench.yaml 2024-03-27T21:42:44.821 INFO:teuthology.orchestra.run.smithi060.stdout: 408045 4 -rw-r--r-- 1 ubuntu ubuntu 127 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba/workload/cifs-kernel-build.yaml.disabled 2024-03-27T21:42:44.821 INFO:teuthology.orchestra.run.smithi060.stdout: 408046 4 -rw-r--r-- 1 ubuntu ubuntu 3221 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba/workload/smbtorture.yaml 2024-03-27T21:42:44.821 INFO:teuthology.orchestra.run.smithi060.stdout: 408044 4 -rw-r--r-- 1 ubuntu ubuntu 123 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/samba/workload/cifs-fsstress.yaml 2024-03-27T21:42:44.821 INFO:teuthology.orchestra.run.smithi060.stdout: 407682 4 drwxr-xr-x 21 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw 2024-03-27T21:42:44.821 INFO:teuthology.orchestra.run.smithi060.stdout: 407684 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/cloud-transition 2024-03-27T21:42:44.821 INFO:teuthology.orchestra.run.smithi060.stdout: 407690 0 lrwxrwxrwx 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/cloud-transition/s3tests-branch.yaml -> .qa/rgw/s3tests-branch.yaml 2024-03-27T21:42:44.821 INFO:teuthology.orchestra.run.smithi060.stdout: 407685 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/cloud-transition/+ 2024-03-27T21:42:44.822 INFO:teuthology.orchestra.run.smithi060.stdout: 407686 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/cloud-transition/.qa -> ../.qa/ 2024-03-27T21:42:44.822 INFO:teuthology.orchestra.run.smithi060.stdout: 407687 4 -rw-r--r-- 1 ubuntu ubuntu 124 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/cloud-transition/cluster.yaml 2024-03-27T21:42:44.822 INFO:teuthology.orchestra.run.smithi060.stdout: 407691 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/cloud-transition/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.822 INFO:teuthology.orchestra.run.smithi060.stdout: 407688 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/cloud-transition/ignore-pg-availability.yaml -> .qa/rgw/ignore-pg-availability.yaml 2024-03-27T21:42:44.822 INFO:teuthology.orchestra.run.smithi060.stdout: 407692 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/cloud-transition/tasks 2024-03-27T21:42:44.822 INFO:teuthology.orchestra.run.smithi060.stdout: 407694 4 -rw-r--r-- 1 ubuntu ubuntu 1907 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/cloud-transition/tasks/cloud_transition_s3tests.yaml 2024-03-27T21:42:44.822 INFO:teuthology.orchestra.run.smithi060.stdout: 407693 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/cloud-transition/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.823 INFO:teuthology.orchestra.run.smithi060.stdout: 407689 4 -rw-r--r-- 1 ubuntu ubuntu 440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/cloud-transition/overrides.yaml 2024-03-27T21:42:44.823 INFO:teuthology.orchestra.run.smithi060.stdout: 407721 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/d4n 2024-03-27T21:42:44.823 INFO:teuthology.orchestra.run.smithi060.stdout: 407722 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/d4n/+ 2024-03-27T21:42:44.823 INFO:teuthology.orchestra.run.smithi060.stdout: 407723 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/d4n/.qa -> ../.qa/ 2024-03-27T21:42:44.823 INFO:teuthology.orchestra.run.smithi060.stdout: 407724 4 -rw-r--r-- 1 ubuntu ubuntu 55 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/d4n/cluster.yaml 2024-03-27T21:42:44.823 INFO:teuthology.orchestra.run.smithi060.stdout: 407727 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/d4n/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.823 INFO:teuthology.orchestra.run.smithi060.stdout: 407725 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/d4n/ignore-pg-availability.yaml -> .qa/rgw/ignore-pg-availability.yaml 2024-03-27T21:42:44.823 INFO:teuthology.orchestra.run.smithi060.stdout: 407728 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/d4n/tasks 2024-03-27T21:42:44.823 INFO:teuthology.orchestra.run.smithi060.stdout: 407730 4 -rw-r--r-- 1 ubuntu ubuntu 398 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/d4n/tasks/rgw_d4ntests.yaml 2024-03-27T21:42:44.824 INFO:teuthology.orchestra.run.smithi060.stdout: 407729 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/d4n/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.824 INFO:teuthology.orchestra.run.smithi060.stdout: 407726 4 -rw-r--r-- 1 ubuntu ubuntu 161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/d4n/overrides.yaml 2024-03-27T21:42:44.824 INFO:teuthology.orchestra.run.smithi060.stdout: 407683 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/.qa -> ../.qa/ 2024-03-27T21:42:44.824 INFO:teuthology.orchestra.run.smithi060.stdout: 407800 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multisite 2024-03-27T21:42:44.824 INFO:teuthology.orchestra.run.smithi060.stdout: 407809 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multisite/realms 2024-03-27T21:42:44.824 INFO:teuthology.orchestra.run.smithi060.stdout: 407810 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multisite/realms/.qa -> ../.qa/ 2024-03-27T21:42:44.824 INFO:teuthology.orchestra.run.smithi060.stdout: 407812 4 -rw-r--r-- 1 ubuntu ubuntu 775 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multisite/realms/two-zonegroup.yaml 2024-03-27T21:42:44.824 INFO:teuthology.orchestra.run.smithi060.stdout: 407811 4 -rw-r--r-- 1 ubuntu ubuntu 596 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multisite/realms/three-zones.yaml 2024-03-27T21:42:44.825 INFO:teuthology.orchestra.run.smithi060.stdout: 407801 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multisite/% 2024-03-27T21:42:44.825 INFO:teuthology.orchestra.run.smithi060.stdout: 407802 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multisite/.qa -> ../.qa/ 2024-03-27T21:42:44.825 INFO:teuthology.orchestra.run.smithi060.stdout: 407807 4 -rw-r--r-- 1 ubuntu ubuntu 433 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multisite/omap_limits.yaml 2024-03-27T21:42:44.825 INFO:teuthology.orchestra.run.smithi060.stdout: 407804 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multisite/frontend -> .qa/rgw_frontend 2024-03-27T21:42:44.825 INFO:teuthology.orchestra.run.smithi060.stdout: 407813 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multisite/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.825 INFO:teuthology.orchestra.run.smithi060.stdout: 407803 4 -rw-r--r-- 1 ubuntu ubuntu 165 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multisite/clusters.yaml 2024-03-27T21:42:44.825 INFO:teuthology.orchestra.run.smithi060.stdout: 407806 4 -rw-r--r-- 1 ubuntu ubuntu 134 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multisite/notify.yaml 2024-03-27T21:42:44.825 INFO:teuthology.orchestra.run.smithi060.stdout: 407817 4 -rw-r--r-- 1 ubuntu ubuntu 521 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multisite/valgrind.yaml.disabled 2024-03-27T21:42:44.825 INFO:teuthology.orchestra.run.smithi060.stdout: 407805 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multisite/ignore-pg-availability.yaml -> .qa/rgw/ignore-pg-availability.yaml 2024-03-27T21:42:44.826 INFO:teuthology.orchestra.run.smithi060.stdout: 407814 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multisite/tasks 2024-03-27T21:42:44.826 INFO:teuthology.orchestra.run.smithi060.stdout: 407815 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multisite/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.826 INFO:teuthology.orchestra.run.smithi060.stdout: 407816 4 -rw-r--r-- 1 ubuntu ubuntu 285 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multisite/tasks/test_multi.yaml 2024-03-27T21:42:44.826 INFO:teuthology.orchestra.run.smithi060.stdout: 407808 4 -rw-r--r-- 1 ubuntu ubuntu 698 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multisite/overrides.yaml 2024-03-27T21:42:44.826 INFO:teuthology.orchestra.run.smithi060.stdout: 407843 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/service-token 2024-03-27T21:42:44.826 INFO:teuthology.orchestra.run.smithi060.stdout: 407844 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/service-token/% 2024-03-27T21:42:44.826 INFO:teuthology.orchestra.run.smithi060.stdout: 407845 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/service-token/.qa -> ../.qa/ 2024-03-27T21:42:44.826 INFO:teuthology.orchestra.run.smithi060.stdout: 407855 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/service-token/ubuntu_latest.yaml -> .qa/distros/supported/ubuntu_latest.yaml 2024-03-27T21:42:44.827 INFO:teuthology.orchestra.run.smithi060.stdout: 407849 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/service-token/frontend -> .qa/rgw_frontend 2024-03-27T21:42:44.827 INFO:teuthology.orchestra.run.smithi060.stdout: 407846 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/service-token/clusters 2024-03-27T21:42:44.827 INFO:teuthology.orchestra.run.smithi060.stdout: 407848 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/service-token/clusters/fixed-1.yaml -> .qa/clusters/fixed-1.yaml 2024-03-27T21:42:44.827 INFO:teuthology.orchestra.run.smithi060.stdout: 407847 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/service-token/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.827 INFO:teuthology.orchestra.run.smithi060.stdout: 407850 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/service-token/ignore-pg-availability.yaml -> .qa/rgw/ignore-pg-availability.yaml 2024-03-27T21:42:44.827 INFO:teuthology.orchestra.run.smithi060.stdout: 407852 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/service-token/tasks 2024-03-27T21:42:44.827 INFO:teuthology.orchestra.run.smithi060.stdout: 407853 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/service-token/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.827 INFO:teuthology.orchestra.run.smithi060.stdout: 407854 4 -rw-r--r-- 1 ubuntu ubuntu 171 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/service-token/tasks/service-token.yaml 2024-03-27T21:42:44.828 INFO:teuthology.orchestra.run.smithi060.stdout: 407851 4 -rw-r--r-- 1 ubuntu ubuntu 795 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/service-token/overrides.yaml 2024-03-27T21:42:44.828 INFO:teuthology.orchestra.run.smithi060.stdout: 407868 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/sts 2024-03-27T21:42:44.828 INFO:teuthology.orchestra.run.smithi060.stdout: 407877 0 lrwxrwxrwx 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/sts/s3tests-branch.yaml -> .qa/rgw/s3tests-branch.yaml 2024-03-27T21:42:44.828 INFO:teuthology.orchestra.run.smithi060.stdout: 407869 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/sts/% 2024-03-27T21:42:44.828 INFO:teuthology.orchestra.run.smithi060.stdout: 407876 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/sts/rgw_frontend -> .qa/rgw_frontend 2024-03-27T21:42:44.828 INFO:teuthology.orchestra.run.smithi060.stdout: 407870 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/sts/.qa -> ../.qa/ 2024-03-27T21:42:44.828 INFO:teuthology.orchestra.run.smithi060.stdout: 407873 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/sts/objectstore.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.828 INFO:teuthology.orchestra.run.smithi060.stdout: 407871 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/sts/cluster.yaml -> .qa/clusters/fixed-2.yaml 2024-03-27T21:42:44.829 INFO:teuthology.orchestra.run.smithi060.stdout: 407878 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/sts/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.829 INFO:teuthology.orchestra.run.smithi060.stdout: 407872 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/sts/ignore-pg-availability.yaml -> .qa/rgw/ignore-pg-availability.yaml 2024-03-27T21:42:44.829 INFO:teuthology.orchestra.run.smithi060.stdout: 407875 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/sts/pool-type.yaml -> .qa/rgw_pool_type/replicated.yaml 2024-03-27T21:42:44.829 INFO:teuthology.orchestra.run.smithi060.stdout: 407879 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/sts/tasks 2024-03-27T21:42:44.829 INFO:teuthology.orchestra.run.smithi060.stdout: 407880 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/sts/tasks/+ 2024-03-27T21:42:44.829 INFO:teuthology.orchestra.run.smithi060.stdout: 407881 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/sts/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.829 INFO:teuthology.orchestra.run.smithi060.stdout: 407884 4 -rw-r--r-- 1 ubuntu ubuntu 111 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/sts/tasks/2-s3tests.yaml 2024-03-27T21:42:44.829 INFO:teuthology.orchestra.run.smithi060.stdout: 407883 4 -rw-r--r-- 1 ubuntu ubuntu 241 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/sts/tasks/1-keycloak.yaml 2024-03-27T21:42:44.830 INFO:teuthology.orchestra.run.smithi060.stdout: 407882 4 -rw-r--r-- 1 ubuntu ubuntu 63 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/sts/tasks/0-install.yaml 2024-03-27T21:42:44.830 INFO:teuthology.orchestra.run.smithi060.stdout: 407874 4 -rw-r--r-- 1 ubuntu ubuntu 625 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/sts/overrides.yaml 2024-03-27T21:42:44.830 INFO:teuthology.orchestra.run.smithi060.stdout: 407767 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lua 2024-03-27T21:42:44.830 INFO:teuthology.orchestra.run.smithi060.stdout: 407772 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lua/fixed-2.yaml -> .qa/clusters/fixed-2.yaml 2024-03-27T21:42:44.830 INFO:teuthology.orchestra.run.smithi060.stdout: 407768 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lua/% 2024-03-27T21:42:44.830 INFO:teuthology.orchestra.run.smithi060.stdout: 407769 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lua/.qa -> ../.qa/ 2024-03-27T21:42:44.830 INFO:teuthology.orchestra.run.smithi060.stdout: 407775 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lua/supported-distros -> .qa/distros/supported-random-distro$/ 2024-03-27T21:42:44.830 INFO:teuthology.orchestra.run.smithi060.stdout: 407771 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lua/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.831 INFO:teuthology.orchestra.run.smithi060.stdout: 407770 0 lrwxrwxrwx 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lua/beast.yaml -> .qa/rgw_frontend/beast.yaml 2024-03-27T21:42:44.831 INFO:teuthology.orchestra.run.smithi060.stdout: 407773 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lua/ignore-pg-availability.yaml -> .qa/rgw/ignore-pg-availability.yaml 2024-03-27T21:42:44.831 INFO:teuthology.orchestra.run.smithi060.stdout: 407776 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lua/tasks 2024-03-27T21:42:44.831 INFO:teuthology.orchestra.run.smithi060.stdout: 407777 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lua/tasks/+ 2024-03-27T21:42:44.831 INFO:teuthology.orchestra.run.smithi060.stdout: 407779 4 -rw-r--r-- 1 ubuntu ubuntu 79 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lua/tasks/test_lua.yaml 2024-03-27T21:42:44.831 INFO:teuthology.orchestra.run.smithi060.stdout: 407778 4 -rw-r--r-- 1 ubuntu ubuntu 196 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lua/tasks/0-install.yaml 2024-03-27T21:42:44.831 INFO:teuthology.orchestra.run.smithi060.stdout: 407774 4 -rw-r--r-- 1 ubuntu ubuntu 155 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lua/overrides.yaml 2024-03-27T21:42:44.831 INFO:teuthology.orchestra.run.smithi060.stdout: 407923 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tools 2024-03-27T21:42:44.832 INFO:teuthology.orchestra.run.smithi060.stdout: 407924 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tools/+ 2024-03-27T21:42:44.832 INFO:teuthology.orchestra.run.smithi060.stdout: 407925 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tools/.qa -> ../.qa/ 2024-03-27T21:42:44.832 INFO:teuthology.orchestra.run.smithi060.stdout: 407927 4 -rw-r--r-- 1 ubuntu ubuntu 173 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tools/cluster.yaml 2024-03-27T21:42:44.832 INFO:teuthology.orchestra.run.smithi060.stdout: 407929 4 -rw-r--r-- 1 ubuntu ubuntu 396 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tools/tasks.yaml 2024-03-27T21:42:44.832 INFO:teuthology.orchestra.run.smithi060.stdout: 407926 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tools/centos_latest.yaml -> .qa/distros/supported/centos_latest.yaml 2024-03-27T21:42:44.832 INFO:teuthology.orchestra.run.smithi060.stdout: 407928 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tools/ignore-pg-availability.yaml -> .qa/rgw/ignore-pg-availability.yaml 2024-03-27T21:42:44.832 INFO:teuthology.orchestra.run.smithi060.stdout: 407856 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/singleton 2024-03-27T21:42:44.832 INFO:teuthology.orchestra.run.smithi060.stdout: 407859 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/singleton/all 2024-03-27T21:42:44.832 INFO:teuthology.orchestra.run.smithi060.stdout: 407860 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/singleton/all/.qa -> ../.qa/ 2024-03-27T21:42:44.833 INFO:teuthology.orchestra.run.smithi060.stdout: 407861 4 -rw-r--r-- 1 ubuntu ubuntu 367 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/singleton/all/radosgw-admin.yaml 2024-03-27T21:42:44.833 INFO:teuthology.orchestra.run.smithi060.stdout: 407857 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/singleton/% 2024-03-27T21:42:44.833 INFO:teuthology.orchestra.run.smithi060.stdout: 407864 0 lrwxrwxrwx 1 ubuntu ubuntu 22 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/singleton/objectstore -> .qa/objectstore_cephfs 2024-03-27T21:42:44.833 INFO:teuthology.orchestra.run.smithi060.stdout: 407858 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/singleton/.qa -> ../.qa/ 2024-03-27T21:42:44.833 INFO:teuthology.orchestra.run.smithi060.stdout: 407862 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/singleton/frontend -> .qa/rgw_frontend 2024-03-27T21:42:44.833 INFO:teuthology.orchestra.run.smithi060.stdout: 407867 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/singleton/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.833 INFO:teuthology.orchestra.run.smithi060.stdout: 407866 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/singleton/rgw_pool_type -> .qa/rgw_pool_type 2024-03-27T21:42:44.833 INFO:teuthology.orchestra.run.smithi060.stdout: 407863 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/singleton/ignore-pg-availability.yaml -> .qa/rgw/ignore-pg-availability.yaml 2024-03-27T21:42:44.834 INFO:teuthology.orchestra.run.smithi060.stdout: 407865 4 -rw-r--r-- 1 ubuntu ubuntu 110 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/singleton/overrides.yaml 2024-03-27T21:42:44.834 INFO:teuthology.orchestra.run.smithi060.stdout: 407885 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tempest 2024-03-27T21:42:44.834 INFO:teuthology.orchestra.run.smithi060.stdout: 407895 0 lrwxrwxrwx 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tempest/s3tests-branch.yaml -> .qa/rgw/s3tests-branch.yaml 2024-03-27T21:42:44.834 INFO:teuthology.orchestra.run.smithi060.stdout: 407886 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tempest/% 2024-03-27T21:42:44.834 INFO:teuthology.orchestra.run.smithi060.stdout: 407887 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tempest/.qa -> ../.qa/ 2024-03-27T21:42:44.834 INFO:teuthology.orchestra.run.smithi060.stdout: 407900 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tempest/ubuntu_latest.yaml -> .qa/distros/supported/ubuntu_latest.yaml 2024-03-27T21:42:44.834 INFO:teuthology.orchestra.run.smithi060.stdout: 407892 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tempest/frontend -> .qa/rgw_frontend 2024-03-27T21:42:44.834 INFO:teuthology.orchestra.run.smithi060.stdout: 407889 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tempest/clusters 2024-03-27T21:42:44.835 INFO:teuthology.orchestra.run.smithi060.stdout: 407891 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tempest/clusters/fixed-1.yaml -> .qa/clusters/fixed-1.yaml 2024-03-27T21:42:44.835 INFO:teuthology.orchestra.run.smithi060.stdout: 407890 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tempest/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.835 INFO:teuthology.orchestra.run.smithi060.stdout: 407893 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tempest/ignore-pg-availability.yaml -> .qa/rgw/ignore-pg-availability.yaml 2024-03-27T21:42:44.835 INFO:teuthology.orchestra.run.smithi060.stdout: 407896 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tempest/tasks 2024-03-27T21:42:44.835 INFO:teuthology.orchestra.run.smithi060.stdout: 407899 4 -rw-r--r-- 1 ubuntu ubuntu 1689 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tempest/tasks/tempest.yaml 2024-03-27T21:42:44.835 INFO:teuthology.orchestra.run.smithi060.stdout: 407897 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tempest/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.835 INFO:teuthology.orchestra.run.smithi060.stdout: 407898 4 -rw-r--r-- 1 ubuntu ubuntu 905 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tempest/tasks/s3tests.yaml 2024-03-27T21:42:44.836 INFO:teuthology.orchestra.run.smithi060.stdout: 407894 4 -rw-r--r-- 1 ubuntu ubuntu 669 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tempest/overrides.yaml 2024-03-27T21:42:44.836 INFO:teuthology.orchestra.run.smithi060.stdout: 407888 4 -rw-r--r-- 1 ubuntu ubuntu 265 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/tempest/0-install.yaml 2024-03-27T21:42:44.836 INFO:teuthology.orchestra.run.smithi060.stdout: 407962 4 drwxr-xr-x 10 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify 2024-03-27T21:42:44.836 INFO:teuthology.orchestra.run.smithi060.stdout: 407989 0 lrwxrwxrwx 1 ubuntu ubuntu 23 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/sharding$ -> .qa/rgw_bucket_sharding 2024-03-27T21:42:44.836 INFO:teuthology.orchestra.run.smithi060.stdout: 407988 0 lrwxrwxrwx 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/s3tests-branch.yaml -> .qa/rgw/s3tests-branch.yaml 2024-03-27T21:42:44.836 INFO:teuthology.orchestra.run.smithi060.stdout: 407963 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/% 2024-03-27T21:42:44.836 INFO:teuthology.orchestra.run.smithi060.stdout: 407981 0 lrwxrwxrwx 1 ubuntu ubuntu 22 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/objectstore -> .qa/objectstore_cephfs 2024-03-27T21:42:44.836 INFO:teuthology.orchestra.run.smithi060.stdout: 407978 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/msgr-failures 2024-03-27T21:42:44.836 INFO:teuthology.orchestra.run.smithi060.stdout: 407979 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.837 INFO:teuthology.orchestra.run.smithi060.stdout: 407980 4 -rw-r--r-- 1 ubuntu ubuntu 177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/msgr-failures/few.yaml 2024-03-27T21:42:44.837 INFO:teuthology.orchestra.run.smithi060.stdout: 407964 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/.qa -> ../.qa/ 2024-03-27T21:42:44.837 INFO:teuthology.orchestra.run.smithi060.stdout: 407975 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/inline-data$ 2024-03-27T21:42:44.837 INFO:teuthology.orchestra.run.smithi060.stdout: 407976 4 -rw-r--r-- 1 ubuntu ubuntu 41 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/inline-data$/off.yaml 2024-03-27T21:42:44.837 INFO:teuthology.orchestra.run.smithi060.stdout: 407977 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/inline-data$/on.yaml 2024-03-27T21:42:44.837 INFO:teuthology.orchestra.run.smithi060.stdout: 407969 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/datacache 2024-03-27T21:42:44.837 INFO:teuthology.orchestra.run.smithi060.stdout: 407971 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/datacache/no_datacache.yaml 2024-03-27T21:42:44.837 INFO:teuthology.orchestra.run.smithi060.stdout: 407970 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/datacache/.qa -> ../.qa/ 2024-03-27T21:42:44.838 INFO:teuthology.orchestra.run.smithi060.stdout: 407972 4 -rw-r--r-- 1 ubuntu ubuntu 435 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/datacache/rgw-datacache.yaml 2024-03-27T21:42:44.838 INFO:teuthology.orchestra.run.smithi060.stdout: 407990 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/striping$ 2024-03-27T21:42:44.838 INFO:teuthology.orchestra.run.smithi060.stdout: 407991 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/striping$/.qa -> ../.qa/ 2024-03-27T21:42:44.838 INFO:teuthology.orchestra.run.smithi060.stdout: 407993 4 -rw-r--r-- 1 ubuntu ubuntu 186 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/striping$/stripe-greater-than-chunk.yaml 2024-03-27T21:42:44.838 INFO:teuthology.orchestra.run.smithi060.stdout: 407992 4 -rw-r--r-- 1 ubuntu ubuntu 176 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/striping$/stripe-equals-chunk.yaml 2024-03-27T21:42:44.838 INFO:teuthology.orchestra.run.smithi060.stdout: 407973 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/frontend -> .qa/rgw_frontend 2024-03-27T21:42:44.838 INFO:teuthology.orchestra.run.smithi060.stdout: 407994 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.838 INFO:teuthology.orchestra.run.smithi060.stdout: 407987 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/rgw_pool_type -> .qa/rgw_pool_type 2024-03-27T21:42:44.839 INFO:teuthology.orchestra.run.smithi060.stdout: 407966 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/clusters 2024-03-27T21:42:44.839 INFO:teuthology.orchestra.run.smithi060.stdout: 407968 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/clusters/fixed-2.yaml -> .qa/clusters/fixed-2.yaml 2024-03-27T21:42:44.839 INFO:teuthology.orchestra.run.smithi060.stdout: 407967 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.839 INFO:teuthology.orchestra.run.smithi060.stdout: 408006 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/validater 2024-03-27T21:42:44.839 INFO:teuthology.orchestra.run.smithi060.stdout: 408007 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/validater/.qa -> ../.qa/ 2024-03-27T21:42:44.839 INFO:teuthology.orchestra.run.smithi060.stdout: 408008 4 -rw-r--r-- 1 ubuntu ubuntu 95 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/validater/lockdep.yaml 2024-03-27T21:42:44.839 INFO:teuthology.orchestra.run.smithi060.stdout: 408009 4 -rw-r--r-- 1 ubuntu ubuntu 546 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/validater/valgrind.yaml 2024-03-27T21:42:44.839 INFO:teuthology.orchestra.run.smithi060.stdout: 407974 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/ignore-pg-availability.yaml -> .qa/rgw/ignore-pg-availability.yaml 2024-03-27T21:42:44.839 INFO:teuthology.orchestra.run.smithi060.stdout: 407995 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/tasks 2024-03-27T21:42:44.840 INFO:teuthology.orchestra.run.smithi060.stdout: 407996 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/tasks/+ 2024-03-27T21:42:44.840 INFO:teuthology.orchestra.run.smithi060.stdout: 408003 4 -rw-r--r-- 1 ubuntu ubuntu 134 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/tasks/s3tests-java.yaml 2024-03-27T21:42:44.840 INFO:teuthology.orchestra.run.smithi060.stdout: 407998 4 -rw-r--r-- 1 ubuntu ubuntu 82 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/tasks/bucket-check.yaml 2024-03-27T21:42:44.840 INFO:teuthology.orchestra.run.smithi060.stdout: 407997 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.840 INFO:teuthology.orchestra.run.smithi060.stdout: 408005 4 -rw-r--r-- 1 ubuntu ubuntu 80 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/tasks/versioning.yaml 2024-03-27T21:42:44.840 INFO:teuthology.orchestra.run.smithi060.stdout: 408002 4 -rw-r--r-- 1 ubuntu ubuntu 77 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/tasks/reshard.yaml 2024-03-27T21:42:44.840 INFO:teuthology.orchestra.run.smithi060.stdout: 408000 4 -rw-r--r-- 1 ubuntu ubuntu 89 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/tasks/mp_reupload.yaml 2024-03-27T21:42:44.840 INFO:teuthology.orchestra.run.smithi060.stdout: 407999 4 -rw-r--r-- 1 ubuntu ubuntu 445 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/tasks/cls.yaml 2024-03-27T21:42:44.841 INFO:teuthology.orchestra.run.smithi060.stdout: 408001 4 -rw-r--r-- 1 ubuntu ubuntu 121 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/tasks/ragweed.yaml 2024-03-27T21:42:44.841 INFO:teuthology.orchestra.run.smithi060.stdout: 408004 4 -rw-r--r-- 1 ubuntu ubuntu 59 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/tasks/s3tests.yaml 2024-03-27T21:42:44.841 INFO:teuthology.orchestra.run.smithi060.stdout: 407983 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/proto 2024-03-27T21:42:44.841 INFO:teuthology.orchestra.run.smithi060.stdout: 407984 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/proto/.qa -> ../.qa/ 2024-03-27T21:42:44.841 INFO:teuthology.orchestra.run.smithi060.stdout: 407986 4 -rw-r--r-- 1 ubuntu ubuntu 267 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/proto/https.yaml 2024-03-27T21:42:44.841 INFO:teuthology.orchestra.run.smithi060.stdout: 407985 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/proto/http.yaml 2024-03-27T21:42:44.841 INFO:teuthology.orchestra.run.smithi060.stdout: 407982 4 -rw-r--r-- 1 ubuntu ubuntu 573 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/overrides.yaml 2024-03-27T21:42:44.841 INFO:teuthology.orchestra.run.smithi060.stdout: 407965 4 -rw-r--r-- 1 ubuntu ubuntu 405 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/verify/0-install.yaml 2024-03-27T21:42:44.842 INFO:teuthology.orchestra.run.smithi060.stdout: 407930 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade 2024-03-27T21:42:44.842 INFO:teuthology.orchestra.run.smithi060.stdout: 407931 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/% 2024-03-27T21:42:44.842 INFO:teuthology.orchestra.run.smithi060.stdout: 407960 0 lrwxrwxrwx 1 ubuntu ubuntu 22 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/objectstore -> .qa/objectstore_cephfs 2024-03-27T21:42:44.842 INFO:teuthology.orchestra.run.smithi060.stdout: 407953 4 -rw-r--r-- 1 ubuntu ubuntu 152 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/2-setup.yaml 2024-03-27T21:42:44.842 INFO:teuthology.orchestra.run.smithi060.stdout: 407932 0 lrwxrwxrwx 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/.qa -> ../.qa 2024-03-27T21:42:44.842 INFO:teuthology.orchestra.run.smithi060.stdout: 407954 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/3-upgrade-sequence 2024-03-27T21:42:44.842 INFO:teuthology.orchestra.run.smithi060.stdout: 407956 4 -rw-r--r-- 1 ubuntu ubuntu 601 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/3-upgrade-sequence/rgws-then-osds.yaml 2024-03-27T21:42:44.842 INFO:teuthology.orchestra.run.smithi060.stdout: 407955 4 -rw-r--r-- 1 ubuntu ubuntu 600 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/3-upgrade-sequence/osds-then-rgws.yaml 2024-03-27T21:42:44.843 INFO:teuthology.orchestra.run.smithi060.stdout: 407957 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/cluster.yaml -> .qa/clusters/fixed-2.yaml 2024-03-27T21:42:44.843 INFO:teuthology.orchestra.run.smithi060.stdout: 407958 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/frontend -> .qa/rgw_frontend 2024-03-27T21:42:44.843 INFO:teuthology.orchestra.run.smithi060.stdout: 407959 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/ignore-pg-availability.yaml -> .qa/rgw/ignore-pg-availability.yaml 2024-03-27T21:42:44.843 INFO:teuthology.orchestra.run.smithi060.stdout: 407961 4 -rw-r--r-- 1 ubuntu ubuntu 418 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/overrides.yaml 2024-03-27T21:42:44.843 INFO:teuthology.orchestra.run.smithi060.stdout: 407933 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/1-install 2024-03-27T21:42:44.843 INFO:teuthology.orchestra.run.smithi060.stdout: 407934 0 lrwxrwxrwx 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/1-install/.qa -> ../.qa 2024-03-27T21:42:44.843 INFO:teuthology.orchestra.run.smithi060.stdout: 407944 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/1-install/reef 2024-03-27T21:42:44.843 INFO:teuthology.orchestra.run.smithi060.stdout: 407951 4 -rw-r--r-- 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/1-install/reef/install.yaml 2024-03-27T21:42:44.844 INFO:teuthology.orchestra.run.smithi060.stdout: 407945 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/1-install/reef/% 2024-03-27T21:42:44.844 INFO:teuthology.orchestra.run.smithi060.stdout: 407946 0 lrwxrwxrwx 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/1-install/reef/.qa -> ../.qa 2024-03-27T21:42:44.844 INFO:teuthology.orchestra.run.smithi060.stdout: 407947 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/1-install/reef/distro$ 2024-03-27T21:42:44.844 INFO:teuthology.orchestra.run.smithi060.stdout: 407948 0 lrwxrwxrwx 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/1-install/reef/distro$/.qa -> ../.qa 2024-03-27T21:42:44.844 INFO:teuthology.orchestra.run.smithi060.stdout: 407950 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/1-install/reef/distro$/ubuntu_latest.yaml -> .qa/distros/supported/ubuntu_latest.yaml 2024-03-27T21:42:44.844 INFO:teuthology.orchestra.run.smithi060.stdout: 407949 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/1-install/reef/distro$/centos_latest.yaml -> .qa/distros/supported/centos_latest.yaml 2024-03-27T21:42:44.844 INFO:teuthology.orchestra.run.smithi060.stdout: 407952 4 -rw-r--r-- 1 ubuntu ubuntu 52 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/1-install/reef/overrides.yaml 2024-03-27T21:42:44.844 INFO:teuthology.orchestra.run.smithi060.stdout: 407935 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/1-install/quincy 2024-03-27T21:42:44.845 INFO:teuthology.orchestra.run.smithi060.stdout: 407942 4 -rw-r--r-- 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/1-install/quincy/install.yaml 2024-03-27T21:42:44.845 INFO:teuthology.orchestra.run.smithi060.stdout: 407936 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/1-install/quincy/% 2024-03-27T21:42:44.845 INFO:teuthology.orchestra.run.smithi060.stdout: 407937 0 lrwxrwxrwx 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/1-install/quincy/.qa -> ../.qa 2024-03-27T21:42:44.845 INFO:teuthology.orchestra.run.smithi060.stdout: 407938 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/1-install/quincy/distro$ 2024-03-27T21:42:44.845 INFO:teuthology.orchestra.run.smithi060.stdout: 407939 0 lrwxrwxrwx 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/1-install/quincy/distro$/.qa -> ../.qa 2024-03-27T21:42:44.845 INFO:teuthology.orchestra.run.smithi060.stdout: 407940 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/1-install/quincy/distro$/centos_latest.yaml -> .qa/distros/supported/centos_latest.yaml 2024-03-27T21:42:44.845 INFO:teuthology.orchestra.run.smithi060.stdout: 407941 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/1-install/quincy/distro$/ubuntu_20.04.yaml -> .qa/distros/all/ubuntu_20.04.yaml 2024-03-27T21:42:44.845 INFO:teuthology.orchestra.run.smithi060.stdout: 407943 4 -rw-r--r-- 1 ubuntu ubuntu 109 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/upgrade/1-install/quincy/overrides.yaml 2024-03-27T21:42:44.845 INFO:teuthology.orchestra.run.smithi060.stdout: 408010 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/website 2024-03-27T21:42:44.846 INFO:teuthology.orchestra.run.smithi060.stdout: 408020 0 lrwxrwxrwx 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/website/s3tests-branch.yaml -> .qa/rgw/s3tests-branch.yaml 2024-03-27T21:42:44.846 INFO:teuthology.orchestra.run.smithi060.stdout: 408011 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/website/% 2024-03-27T21:42:44.846 INFO:teuthology.orchestra.run.smithi060.stdout: 408012 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/website/.qa -> ../.qa/ 2024-03-27T21:42:44.846 INFO:teuthology.orchestra.run.smithi060.stdout: 408024 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/website/ubuntu_latest.yaml -> .qa/distros/supported/ubuntu_latest.yaml 2024-03-27T21:42:44.846 INFO:teuthology.orchestra.run.smithi060.stdout: 408017 4 -rw-r--r-- 1 ubuntu ubuntu 83 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/website/http.yaml 2024-03-27T21:42:44.846 INFO:teuthology.orchestra.run.smithi060.stdout: 408016 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/website/frontend -> .qa/rgw_frontend 2024-03-27T21:42:44.846 INFO:teuthology.orchestra.run.smithi060.stdout: 408013 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/website/clusters 2024-03-27T21:42:44.847 INFO:teuthology.orchestra.run.smithi060.stdout: 408015 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/website/clusters/fixed-2.yaml -> .qa/clusters/fixed-2.yaml 2024-03-27T21:42:44.847 INFO:teuthology.orchestra.run.smithi060.stdout: 408014 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/website/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.847 INFO:teuthology.orchestra.run.smithi060.stdout: 408018 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/website/ignore-pg-availability.yaml -> .qa/rgw/ignore-pg-availability.yaml 2024-03-27T21:42:44.847 INFO:teuthology.orchestra.run.smithi060.stdout: 408021 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/website/tasks 2024-03-27T21:42:44.847 INFO:teuthology.orchestra.run.smithi060.stdout: 408022 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/website/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.847 INFO:teuthology.orchestra.run.smithi060.stdout: 408023 4 -rw-r--r-- 1 ubuntu ubuntu 297 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/website/tasks/s3tests-website.yaml 2024-03-27T21:42:44.847 INFO:teuthology.orchestra.run.smithi060.stdout: 408019 4 -rw-r--r-- 1 ubuntu ubuntu 799 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/website/overrides.yaml 2024-03-27T21:42:44.847 INFO:teuthology.orchestra.run.smithi060.stdout: 407695 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt 2024-03-27T21:42:44.847 INFO:teuthology.orchestra.run.smithi060.stdout: 407719 0 lrwxrwxrwx 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/s3tests-branch.yaml -> .qa/rgw/s3tests-branch.yaml 2024-03-27T21:42:44.848 INFO:teuthology.orchestra.run.smithi060.stdout: 407698 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/0-cluster 2024-03-27T21:42:44.848 INFO:teuthology.orchestra.run.smithi060.stdout: 407700 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/0-cluster/fixed-1.yaml -> ../../../../clusters/fixed-1.yaml 2024-03-27T21:42:44.848 INFO:teuthology.orchestra.run.smithi060.stdout: 407699 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/0-cluster/.qa -> ../.qa/ 2024-03-27T21:42:44.848 INFO:teuthology.orchestra.run.smithi060.stdout: 407696 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/% 2024-03-27T21:42:44.848 INFO:teuthology.orchestra.run.smithi060.stdout: 407697 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/.qa -> ../.qa/ 2024-03-27T21:42:44.848 INFO:teuthology.orchestra.run.smithi060.stdout: 407714 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/4-tests 2024-03-27T21:42:44.848 INFO:teuthology.orchestra.run.smithi060.stdout: 407715 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/4-tests/+ 2024-03-27T21:42:44.848 INFO:teuthology.orchestra.run.smithi060.stdout: 407716 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/4-tests/.qa -> ../.qa/ 2024-03-27T21:42:44.849 INFO:teuthology.orchestra.run.smithi060.stdout: 407717 4 -rw-r--r-- 1 ubuntu ubuntu 342 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/4-tests/s3tests.yaml 2024-03-27T21:42:44.849 INFO:teuthology.orchestra.run.smithi060.stdout: 407720 0 lrwxrwxrwx 1 ubuntu ubuntu 34 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/ubuntu_latest.yaml -> .qa/distros/all/ubuntu_latest.yaml 2024-03-27T21:42:44.849 INFO:teuthology.orchestra.run.smithi060.stdout: 407711 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/3-rgw 2024-03-27T21:42:44.849 INFO:teuthology.orchestra.run.smithi060.stdout: 407713 4 -rw-r--r-- 1 ubuntu ubuntu 212 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/3-rgw/rgw.yaml 2024-03-27T21:42:44.849 INFO:teuthology.orchestra.run.smithi060.stdout: 407712 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/3-rgw/.qa -> ../.qa/ 2024-03-27T21:42:44.849 INFO:teuthology.orchestra.run.smithi060.stdout: 407701 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/1-ceph-install 2024-03-27T21:42:44.849 INFO:teuthology.orchestra.run.smithi060.stdout: 407703 4 -rw-r--r-- 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/1-ceph-install/install.yaml 2024-03-27T21:42:44.849 INFO:teuthology.orchestra.run.smithi060.stdout: 407702 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/1-ceph-install/.qa -> ../.qa/ 2024-03-27T21:42:44.849 INFO:teuthology.orchestra.run.smithi060.stdout: 407718 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/ignore-pg-availability.yaml -> .qa/rgw/ignore-pg-availability.yaml 2024-03-27T21:42:44.850 INFO:teuthology.orchestra.run.smithi060.stdout: 407704 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/2-kms 2024-03-27T21:42:44.850 INFO:teuthology.orchestra.run.smithi060.stdout: 407707 4 -rw-r--r-- 1 ubuntu ubuntu 237 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/2-kms/testing.yaml 2024-03-27T21:42:44.850 INFO:teuthology.orchestra.run.smithi060.stdout: 407705 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/2-kms/.qa -> ../.qa/ 2024-03-27T21:42:44.850 INFO:teuthology.orchestra.run.smithi060.stdout: 407706 4 -rw-r--r-- 1 ubuntu ubuntu 2672 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/2-kms/barbican.yaml 2024-03-27T21:42:44.850 INFO:teuthology.orchestra.run.smithi060.stdout: 407709 4 -rw-r--r-- 1 ubuntu ubuntu 664 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/2-kms/vault_old.yaml 2024-03-27T21:42:44.850 INFO:teuthology.orchestra.run.smithi060.stdout: 407710 4 -rw-r--r-- 1 ubuntu ubuntu 846 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/2-kms/vault_transit.yaml 2024-03-27T21:42:44.850 INFO:teuthology.orchestra.run.smithi060.stdout: 407708 4 -rw-r--r-- 1 ubuntu ubuntu 734 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/crypt/2-kms/vault_kv.yaml 2024-03-27T21:42:44.850 INFO:teuthology.orchestra.run.smithi060.stdout: 407901 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash 2024-03-27T21:42:44.851 INFO:teuthology.orchestra.run.smithi060.stdout: 407911 0 lrwxrwxrwx 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash/s3tests-branch.yaml -> .qa/rgw/s3tests-branch.yaml 2024-03-27T21:42:44.851 INFO:teuthology.orchestra.run.smithi060.stdout: 407909 4 -rw-r--r-- 1 ubuntu ubuntu 45 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash/install.yaml 2024-03-27T21:42:44.851 INFO:teuthology.orchestra.run.smithi060.stdout: 407902 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash/% 2024-03-27T21:42:44.851 INFO:teuthology.orchestra.run.smithi060.stdout: 407910 0 lrwxrwxrwx 1 ubuntu ubuntu 22 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash/objectstore -> .qa/objectstore_cephfs 2024-03-27T21:42:44.851 INFO:teuthology.orchestra.run.smithi060.stdout: 407903 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash/.qa -> ../.qa/ 2024-03-27T21:42:44.851 INFO:teuthology.orchestra.run.smithi060.stdout: 407916 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash/ubuntu_latest.yaml -> .qa/distros/supported/ubuntu_latest.yaml 2024-03-27T21:42:44.851 INFO:teuthology.orchestra.run.smithi060.stdout: 407915 0 lrwxrwxrwx 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash/thrashosds-health.yaml -> .qa/tasks/thrashosds-health.yaml 2024-03-27T21:42:44.851 INFO:teuthology.orchestra.run.smithi060.stdout: 407912 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash/thrasher 2024-03-27T21:42:44.852 INFO:teuthology.orchestra.run.smithi060.stdout: 407913 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash/thrasher/.qa -> ../.qa/ 2024-03-27T21:42:44.852 INFO:teuthology.orchestra.run.smithi060.stdout: 407914 4 -rw-r--r-- 1 ubuntu ubuntu 194 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash/thrasher/default.yaml 2024-03-27T21:42:44.852 INFO:teuthology.orchestra.run.smithi060.stdout: 407907 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash/frontend -> .qa/rgw_frontend 2024-03-27T21:42:44.852 INFO:teuthology.orchestra.run.smithi060.stdout: 407904 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash/clusters 2024-03-27T21:42:44.852 INFO:teuthology.orchestra.run.smithi060.stdout: 407906 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash/clusters/fixed-2.yaml -> .qa/clusters/fixed-2.yaml 2024-03-27T21:42:44.852 INFO:teuthology.orchestra.run.smithi060.stdout: 407905 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.852 INFO:teuthology.orchestra.run.smithi060.stdout: 407908 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash/ignore-pg-availability.yaml -> .qa/rgw/ignore-pg-availability.yaml 2024-03-27T21:42:44.852 INFO:teuthology.orchestra.run.smithi060.stdout: 407917 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash/workload 2024-03-27T21:42:44.852 INFO:teuthology.orchestra.run.smithi060.stdout: 407918 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash/workload/.qa -> ../.qa/ 2024-03-27T21:42:44.853 INFO:teuthology.orchestra.run.smithi060.stdout: 407922 4 -rw-r--r-- 1 ubuntu ubuntu 164 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash/workload/rgw_user_quota.yaml 2024-03-27T21:42:44.853 INFO:teuthology.orchestra.run.smithi060.stdout: 407919 4 -rw-r--r-- 1 ubuntu ubuntu 166 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash/workload/rgw_bucket_quota.yaml 2024-03-27T21:42:44.853 INFO:teuthology.orchestra.run.smithi060.stdout: 407921 4 -rw-r--r-- 1 ubuntu ubuntu 351 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash/workload/rgw_s3tests.yaml 2024-03-27T21:42:44.853 INFO:teuthology.orchestra.run.smithi060.stdout: 407920 4 -rw-r--r-- 1 ubuntu ubuntu 170 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/thrash/workload/rgw_multipart_upload.yaml 2024-03-27T21:42:44.853 INFO:teuthology.orchestra.run.smithi060.stdout: 407756 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lifecycle 2024-03-27T21:42:44.853 INFO:teuthology.orchestra.run.smithi060.stdout: 407762 0 lrwxrwxrwx 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lifecycle/s3tests-branch.yaml -> .qa/rgw/s3tests-branch.yaml 2024-03-27T21:42:44.853 INFO:teuthology.orchestra.run.smithi060.stdout: 407757 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lifecycle/+ 2024-03-27T21:42:44.853 INFO:teuthology.orchestra.run.smithi060.stdout: 407758 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lifecycle/.qa -> ../.qa/ 2024-03-27T21:42:44.854 INFO:teuthology.orchestra.run.smithi060.stdout: 407759 4 -rw-r--r-- 1 ubuntu ubuntu 173 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lifecycle/cluster.yaml 2024-03-27T21:42:44.854 INFO:teuthology.orchestra.run.smithi060.stdout: 407763 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lifecycle/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.854 INFO:teuthology.orchestra.run.smithi060.stdout: 407760 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lifecycle/ignore-pg-availability.yaml -> .qa/rgw/ignore-pg-availability.yaml 2024-03-27T21:42:44.854 INFO:teuthology.orchestra.run.smithi060.stdout: 407764 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lifecycle/tasks 2024-03-27T21:42:44.854 INFO:teuthology.orchestra.run.smithi060.stdout: 407765 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lifecycle/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.854 INFO:teuthology.orchestra.run.smithi060.stdout: 407766 4 -rw-r--r-- 1 ubuntu ubuntu 210 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lifecycle/tasks/rgw_s3tests.yaml 2024-03-27T21:42:44.854 INFO:teuthology.orchestra.run.smithi060.stdout: 407761 4 -rw-r--r-- 1 ubuntu ubuntu 420 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/lifecycle/overrides.yaml 2024-03-27T21:42:44.854 INFO:teuthology.orchestra.run.smithi060.stdout: 407818 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications 2024-03-27T21:42:44.854 INFO:teuthology.orchestra.run.smithi060.stdout: 407823 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/fixed-2.yaml -> .qa/clusters/fixed-2.yaml 2024-03-27T21:42:44.855 INFO:teuthology.orchestra.run.smithi060.stdout: 407819 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/% 2024-03-27T21:42:44.855 INFO:teuthology.orchestra.run.smithi060.stdout: 407820 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/.qa -> ../.qa/ 2024-03-27T21:42:44.855 INFO:teuthology.orchestra.run.smithi060.stdout: 407822 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.855 INFO:teuthology.orchestra.run.smithi060.stdout: 407821 0 lrwxrwxrwx 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/beast.yaml -> .qa/rgw_frontend/beast.yaml 2024-03-27T21:42:44.855 INFO:teuthology.orchestra.run.smithi060.stdout: 407824 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/ignore-pg-availability.yaml -> .qa/rgw/ignore-pg-availability.yaml 2024-03-27T21:42:44.855 INFO:teuthology.orchestra.run.smithi060.stdout: 407826 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/tasks 2024-03-27T21:42:44.855 INFO:teuthology.orchestra.run.smithi060.stdout: 407827 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.855 INFO:teuthology.orchestra.run.smithi060.stdout: 407828 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/tasks/amqp 2024-03-27T21:42:44.856 INFO:teuthology.orchestra.run.smithi060.stdout: 407829 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/tasks/amqp/+ 2024-03-27T21:42:44.856 INFO:teuthology.orchestra.run.smithi060.stdout: 407832 4 -rw-r--r-- 1 ubuntu ubuntu 128 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/tasks/amqp/test_amqp.yaml 2024-03-27T21:42:44.856 INFO:teuthology.orchestra.run.smithi060.stdout: 407831 0 lrwxrwxrwx 1 ubuntu ubuntu 46 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/tasks/amqp/centos_latest.yaml -> ../../.qa/distros/supported/centos_latest.yaml 2024-03-27T21:42:44.856 INFO:teuthology.orchestra.run.smithi060.stdout: 407830 4 -rw-r--r-- 1 ubuntu ubuntu 177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/tasks/amqp/0-install.yaml 2024-03-27T21:42:44.856 INFO:teuthology.orchestra.run.smithi060.stdout: 407838 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/tasks/others 2024-03-27T21:42:44.856 INFO:teuthology.orchestra.run.smithi060.stdout: 407839 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/tasks/others/+ 2024-03-27T21:42:44.856 INFO:teuthology.orchestra.run.smithi060.stdout: 407841 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/tasks/others/supported-distros -> ../../.qa/distros/supported-random-distro$/ 2024-03-27T21:42:44.856 INFO:teuthology.orchestra.run.smithi060.stdout: 407842 4 -rw-r--r-- 1 ubuntu ubuntu 70 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/tasks/others/test_others.yaml 2024-03-27T21:42:44.856 INFO:teuthology.orchestra.run.smithi060.stdout: 407840 4 -rw-r--r-- 1 ubuntu ubuntu 177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/tasks/others/0-install.yaml 2024-03-27T21:42:44.857 INFO:teuthology.orchestra.run.smithi060.stdout: 407833 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/tasks/kafka 2024-03-27T21:42:44.857 INFO:teuthology.orchestra.run.smithi060.stdout: 407834 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/tasks/kafka/+ 2024-03-27T21:42:44.857 INFO:teuthology.orchestra.run.smithi060.stdout: 407836 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/tasks/kafka/supported-distros -> ../../.qa/distros/supported-random-distro$/ 2024-03-27T21:42:44.857 INFO:teuthology.orchestra.run.smithi060.stdout: 407837 4 -rw-r--r-- 1 ubuntu ubuntu 153 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/tasks/kafka/test_kafka.yaml 2024-03-27T21:42:44.857 INFO:teuthology.orchestra.run.smithi060.stdout: 407835 4 -rw-r--r-- 1 ubuntu ubuntu 290 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/tasks/kafka/0-install.yaml 2024-03-27T21:42:44.857 INFO:teuthology.orchestra.run.smithi060.stdout: 407825 4 -rw-r--r-- 1 ubuntu ubuntu 405 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/notifications/overrides.yaml 2024-03-27T21:42:44.857 INFO:teuthology.orchestra.run.smithi060.stdout: 407731 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/dbstore 2024-03-27T21:42:44.857 INFO:teuthology.orchestra.run.smithi060.stdout: 407737 0 lrwxrwxrwx 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/dbstore/s3tests-branch.yaml -> .qa/rgw/s3tests-branch.yaml 2024-03-27T21:42:44.858 INFO:teuthology.orchestra.run.smithi060.stdout: 407732 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/dbstore/+ 2024-03-27T21:42:44.858 INFO:teuthology.orchestra.run.smithi060.stdout: 407733 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/dbstore/.qa -> ../.qa/ 2024-03-27T21:42:44.858 INFO:teuthology.orchestra.run.smithi060.stdout: 407734 4 -rw-r--r-- 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/dbstore/cluster.yaml 2024-03-27T21:42:44.858 INFO:teuthology.orchestra.run.smithi060.stdout: 407738 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/dbstore/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.858 INFO:teuthology.orchestra.run.smithi060.stdout: 407735 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/dbstore/ignore-pg-availability.yaml -> .qa/rgw/ignore-pg-availability.yaml 2024-03-27T21:42:44.858 INFO:teuthology.orchestra.run.smithi060.stdout: 407739 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/dbstore/tasks 2024-03-27T21:42:44.858 INFO:teuthology.orchestra.run.smithi060.stdout: 407740 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/dbstore/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.858 INFO:teuthology.orchestra.run.smithi060.stdout: 407741 4 -rw-r--r-- 1 ubuntu ubuntu 358 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/dbstore/tasks/rgw_s3tests.yaml 2024-03-27T21:42:44.859 INFO:teuthology.orchestra.run.smithi060.stdout: 407736 4 -rw-r--r-- 1 ubuntu ubuntu 243 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/dbstore/overrides.yaml 2024-03-27T21:42:44.859 INFO:teuthology.orchestra.run.smithi060.stdout: 407742 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/hadoop-s3a 2024-03-27T21:42:44.859 INFO:teuthology.orchestra.run.smithi060.stdout: 407743 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/hadoop-s3a/% 2024-03-27T21:42:44.859 INFO:teuthology.orchestra.run.smithi060.stdout: 407744 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/hadoop-s3a/.qa -> ../.qa/ 2024-03-27T21:42:44.859 INFO:teuthology.orchestra.run.smithi060.stdout: 407755 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/hadoop-s3a/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.859 INFO:teuthology.orchestra.run.smithi060.stdout: 407754 4 -rw-r--r-- 1 ubuntu ubuntu 143 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/hadoop-s3a/s3a-hadoop.yaml 2024-03-27T21:42:44.859 INFO:teuthology.orchestra.run.smithi060.stdout: 407745 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/hadoop-s3a/clusters 2024-03-27T21:42:44.859 INFO:teuthology.orchestra.run.smithi060.stdout: 407747 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/hadoop-s3a/clusters/fixed-2.yaml -> .qa/clusters/fixed-2.yaml 2024-03-27T21:42:44.859 INFO:teuthology.orchestra.run.smithi060.stdout: 407746 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/hadoop-s3a/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.860 INFO:teuthology.orchestra.run.smithi060.stdout: 407748 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/hadoop-s3a/hadoop 2024-03-27T21:42:44.860 INFO:teuthology.orchestra.run.smithi060.stdout: 407749 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/hadoop-s3a/hadoop/.qa -> ../.qa/ 2024-03-27T21:42:44.860 INFO:teuthology.orchestra.run.smithi060.stdout: 407751 4 -rw-r--r-- 1 ubuntu ubuntu 53 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/hadoop-s3a/hadoop/v32.yaml 2024-03-27T21:42:44.860 INFO:teuthology.orchestra.run.smithi060.stdout: 407750 4 -rw-r--r-- 1 ubuntu ubuntu 1 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/hadoop-s3a/hadoop/default.yaml 2024-03-27T21:42:44.860 INFO:teuthology.orchestra.run.smithi060.stdout: 407752 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/hadoop-s3a/ignore-pg-availability.yaml -> .qa/rgw/ignore-pg-availability.yaml 2024-03-27T21:42:44.860 INFO:teuthology.orchestra.run.smithi060.stdout: 407753 4 -rw-r--r-- 1 ubuntu ubuntu 88 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/hadoop-s3a/overrides.yaml 2024-03-27T21:42:44.860 INFO:teuthology.orchestra.run.smithi060.stdout: 407780 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multifs 2024-03-27T21:42:44.860 INFO:teuthology.orchestra.run.smithi060.stdout: 407791 0 lrwxrwxrwx 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multifs/s3tests-branch.yaml -> .qa/rgw/s3tests-branch.yaml 2024-03-27T21:42:44.861 INFO:teuthology.orchestra.run.smithi060.stdout: 407781 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multifs/% 2024-03-27T21:42:44.861 INFO:teuthology.orchestra.run.smithi060.stdout: 407788 0 lrwxrwxrwx 1 ubuntu ubuntu 22 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multifs/objectstore -> .qa/objectstore_cephfs 2024-03-27T21:42:44.861 INFO:teuthology.orchestra.run.smithi060.stdout: 407782 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multifs/.qa -> ../.qa/ 2024-03-27T21:42:44.861 INFO:teuthology.orchestra.run.smithi060.stdout: 407799 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multifs/ubuntu_latest.yaml -> .qa/distros/supported/ubuntu_latest.yaml 2024-03-27T21:42:44.861 INFO:teuthology.orchestra.run.smithi060.stdout: 407786 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multifs/frontend -> .qa/rgw_frontend 2024-03-27T21:42:44.861 INFO:teuthology.orchestra.run.smithi060.stdout: 407790 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multifs/rgw_pool_type -> .qa/rgw_pool_type 2024-03-27T21:42:44.861 INFO:teuthology.orchestra.run.smithi060.stdout: 407783 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multifs/clusters 2024-03-27T21:42:44.861 INFO:teuthology.orchestra.run.smithi060.stdout: 407785 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multifs/clusters/fixed-2.yaml -> .qa/clusters/fixed-2.yaml 2024-03-27T21:42:44.861 INFO:teuthology.orchestra.run.smithi060.stdout: 407784 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multifs/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.862 INFO:teuthology.orchestra.run.smithi060.stdout: 407787 0 lrwxrwxrwx 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multifs/ignore-pg-availability.yaml -> .qa/rgw/ignore-pg-availability.yaml 2024-03-27T21:42:44.862 INFO:teuthology.orchestra.run.smithi060.stdout: 407792 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multifs/tasks 2024-03-27T21:42:44.862 INFO:teuthology.orchestra.run.smithi060.stdout: 407793 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multifs/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.862 INFO:teuthology.orchestra.run.smithi060.stdout: 407796 4 -rw-r--r-- 1 ubuntu ubuntu 276 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multifs/tasks/rgw_ragweed.yaml 2024-03-27T21:42:44.862 INFO:teuthology.orchestra.run.smithi060.stdout: 407798 4 -rw-r--r-- 1 ubuntu ubuntu 201 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multifs/tasks/rgw_user_quota.yaml 2024-03-27T21:42:44.862 INFO:teuthology.orchestra.run.smithi060.stdout: 407794 4 -rw-r--r-- 1 ubuntu ubuntu 203 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multifs/tasks/rgw_bucket_quota.yaml 2024-03-27T21:42:44.862 INFO:teuthology.orchestra.run.smithi060.stdout: 407797 4 -rw-r--r-- 1 ubuntu ubuntu 114 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multifs/tasks/rgw_s3tests.yaml 2024-03-27T21:42:44.863 INFO:teuthology.orchestra.run.smithi060.stdout: 407795 4 -rw-r--r-- 1 ubuntu ubuntu 207 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multifs/tasks/rgw_multipart_upload.yaml 2024-03-27T21:42:44.863 INFO:teuthology.orchestra.run.smithi060.stdout: 407789 4 -rw-r--r-- 1 ubuntu ubuntu 435 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw/multifs/overrides.yaml 2024-03-27T21:42:44.863 INFO:teuthology.orchestra.run.smithi060.stdout: 404664 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy 2024-03-27T21:42:44.863 INFO:teuthology.orchestra.run.smithi060.stdout: 404665 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/% 2024-03-27T21:42:44.863 INFO:teuthology.orchestra.run.smithi060.stdout: 404666 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/.qa -> ../.qa/ 2024-03-27T21:42:44.863 INFO:teuthology.orchestra.run.smithi060.stdout: 404679 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/python_versions 2024-03-27T21:42:44.863 INFO:teuthology.orchestra.run.smithi060.stdout: 404680 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/python_versions/.qa -> ../.qa/ 2024-03-27T21:42:44.863 INFO:teuthology.orchestra.run.smithi060.stdout: 404681 4 -rw-r--r-- 1 ubuntu ubuntu 50 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/python_versions/python_2.yaml 2024-03-27T21:42:44.863 INFO:teuthology.orchestra.run.smithi060.stdout: 404682 4 -rw-r--r-- 1 ubuntu ubuntu 50 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/python_versions/python_3.yaml 2024-03-27T21:42:44.864 INFO:teuthology.orchestra.run.smithi060.stdout: 404675 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/distros 2024-03-27T21:42:44.864 INFO:teuthology.orchestra.run.smithi060.stdout: 404676 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/distros/.qa -> ../.qa/ 2024-03-27T21:42:44.864 INFO:teuthology.orchestra.run.smithi060.stdout: 404678 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/distros/ubuntu_latest.yaml -> .qa/distros/supported/ubuntu_latest.yaml 2024-03-27T21:42:44.864 INFO:teuthology.orchestra.run.smithi060.stdout: 404677 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/distros/centos_latest.yaml -> .qa/distros/supported/centos_latest.yaml 2024-03-27T21:42:44.864 INFO:teuthology.orchestra.run.smithi060.stdout: 404683 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/tasks 2024-03-27T21:42:44.864 INFO:teuthology.orchestra.run.smithi060.stdout: 404684 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.864 INFO:teuthology.orchestra.run.smithi060.stdout: 404686 4 -rw-r--r-- 1 ubuntu ubuntu 187 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/tasks/rbd_import_export.yaml 2024-03-27T21:42:44.864 INFO:teuthology.orchestra.run.smithi060.stdout: 404685 4 -rw-r--r-- 1 ubuntu ubuntu 290 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/tasks/ceph-admin-commands.yaml 2024-03-27T21:42:44.865 INFO:teuthology.orchestra.run.smithi060.stdout: 404667 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/cluster 2024-03-27T21:42:44.865 INFO:teuthology.orchestra.run.smithi060.stdout: 404668 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/cluster/.qa -> ../.qa/ 2024-03-27T21:42:44.865 INFO:teuthology.orchestra.run.smithi060.stdout: 404669 4 -rw-r--r-- 1 ubuntu ubuntu 253 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/cluster/4node.yaml 2024-03-27T21:42:44.865 INFO:teuthology.orchestra.run.smithi060.stdout: 404670 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/config 2024-03-27T21:42:44.865 INFO:teuthology.orchestra.run.smithi060.stdout: 404674 4 -rw-r--r-- 1 ubuntu ubuntu 55 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/config/ceph_volume_dmcrypt_off.yaml 2024-03-27T21:42:44.865 INFO:teuthology.orchestra.run.smithi060.stdout: 404671 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/config/.qa -> ../.qa/ 2024-03-27T21:42:44.865 INFO:teuthology.orchestra.run.smithi060.stdout: 404673 4 -rw-r--r-- 1 ubuntu ubuntu 162 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/config/ceph_volume_bluestore_dmcrypt.yaml 2024-03-27T21:42:44.865 INFO:teuthology.orchestra.run.smithi060.stdout: 404672 4 -rw-r--r-- 1 ubuntu ubuntu 142 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-deploy/config/ceph_volume_bluestore.yaml 2024-03-27T21:42:44.865 INFO:teuthology.orchestra.run.smithi060.stdout: 405904 4 drwxr-xr-x 13 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd 2024-03-27T21:42:44.866 INFO:teuthology.orchestra.run.smithi060.stdout: 405938 4 drwxr-xr-x 9 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx 2024-03-27T21:42:44.866 INFO:teuthology.orchestra.run.smithi060.stdout: 405939 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/% 2024-03-27T21:42:44.866 INFO:teuthology.orchestra.run.smithi060.stdout: 405959 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/objectstore 2024-03-27T21:42:44.866 INFO:teuthology.orchestra.run.smithi060.stdout: 405960 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/objectstore/.qa -> ../.qa/ 2024-03-27T21:42:44.866 INFO:teuthology.orchestra.run.smithi060.stdout: 405961 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/objectstore/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.866 INFO:teuthology.orchestra.run.smithi060.stdout: 405940 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/.qa -> ../.qa/ 2024-03-27T21:42:44.866 INFO:teuthology.orchestra.run.smithi060.stdout: 405952 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/ms_mode$ 2024-03-27T21:42:44.866 INFO:teuthology.orchestra.run.smithi060.stdout: 405954 4 -rw-r--r-- 1 ubuntu ubuntu 123 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/ms_mode$/crc-rxbounce.yaml 2024-03-27T21:42:44.867 INFO:teuthology.orchestra.run.smithi060.stdout: 405958 4 -rw-r--r-- 1 ubuntu ubuntu 117 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/ms_mode$/secure.yaml 2024-03-27T21:42:44.867 INFO:teuthology.orchestra.run.smithi060.stdout: 405956 4 -rw-r--r-- 1 ubuntu ubuntu 126 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/ms_mode$/legacy-rxbounce.yaml 2024-03-27T21:42:44.867 INFO:teuthology.orchestra.run.smithi060.stdout: 405955 4 -rw-r--r-- 1 ubuntu ubuntu 114 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/ms_mode$/crc.yaml 2024-03-27T21:42:44.867 INFO:teuthology.orchestra.run.smithi060.stdout: 405953 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/ms_mode$/.qa -> ../.qa/ 2024-03-27T21:42:44.867 INFO:teuthology.orchestra.run.smithi060.stdout: 405957 4 -rw-r--r-- 1 ubuntu ubuntu 117 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/ms_mode$/legacy.yaml 2024-03-27T21:42:44.867 INFO:teuthology.orchestra.run.smithi060.stdout: 405962 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/striping 2024-03-27T21:42:44.867 INFO:teuthology.orchestra.run.smithi060.stdout: 405963 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/striping/.qa -> ../.qa/ 2024-03-27T21:42:44.867 INFO:teuthology.orchestra.run.smithi060.stdout: 405972 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/striping/fancy 2024-03-27T21:42:44.868 INFO:teuthology.orchestra.run.smithi060.stdout: 405973 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/striping/fancy/% 2024-03-27T21:42:44.868 INFO:teuthology.orchestra.run.smithi060.stdout: 405975 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/striping/fancy/msgr-failures 2024-03-27T21:42:44.868 INFO:teuthology.orchestra.run.smithi060.stdout: 405976 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/striping/fancy/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.868 INFO:teuthology.orchestra.run.smithi060.stdout: 405977 4 -rw-r--r-- 1 ubuntu ubuntu 177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/striping/fancy/msgr-failures/few.yaml 2024-03-27T21:42:44.868 INFO:teuthology.orchestra.run.smithi060.stdout: 405974 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/striping/fancy/.qa -> ../.qa/ 2024-03-27T21:42:44.868 INFO:teuthology.orchestra.run.smithi060.stdout: 405978 4 -rw-r--r-- 1 ubuntu ubuntu 52 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/striping/fancy/randomized-striping-on.yaml 2024-03-27T21:42:44.868 INFO:teuthology.orchestra.run.smithi060.stdout: 405964 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/striping/default 2024-03-27T21:42:44.868 INFO:teuthology.orchestra.run.smithi060.stdout: 405965 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/striping/default/% 2024-03-27T21:42:44.868 INFO:teuthology.orchestra.run.smithi060.stdout: 405971 4 -rw-r--r-- 1 ubuntu ubuntu 53 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/striping/default/randomized-striping-off.yaml 2024-03-27T21:42:44.869 INFO:teuthology.orchestra.run.smithi060.stdout: 405967 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/striping/default/msgr-failures 2024-03-27T21:42:44.869 INFO:teuthology.orchestra.run.smithi060.stdout: 405968 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/striping/default/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.869 INFO:teuthology.orchestra.run.smithi060.stdout: 405969 4 -rw-r--r-- 1 ubuntu ubuntu 177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/striping/default/msgr-failures/few.yaml 2024-03-27T21:42:44.869 INFO:teuthology.orchestra.run.smithi060.stdout: 405970 4 -rw-r--r-- 1 ubuntu ubuntu 176 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/striping/default/msgr-failures/many.yaml 2024-03-27T21:42:44.869 INFO:teuthology.orchestra.run.smithi060.stdout: 405966 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/striping/default/.qa -> ../.qa/ 2024-03-27T21:42:44.869 INFO:teuthology.orchestra.run.smithi060.stdout: 405944 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/clusters 2024-03-27T21:42:44.869 INFO:teuthology.orchestra.run.smithi060.stdout: 405945 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.869 INFO:teuthology.orchestra.run.smithi060.stdout: 405946 4 -rw-r--r-- 1 ubuntu ubuntu 365 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/clusters/3-node.yaml 2024-03-27T21:42:44.870 INFO:teuthology.orchestra.run.smithi060.stdout: 405948 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/features 2024-03-27T21:42:44.870 INFO:teuthology.orchestra.run.smithi060.stdout: 405949 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/features/.qa -> ../.qa/ 2024-03-27T21:42:44.870 INFO:teuthology.orchestra.run.smithi060.stdout: 405950 4 -rw-r--r-- 1 ubuntu ubuntu 109 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/features/no-object-map.yaml 2024-03-27T21:42:44.870 INFO:teuthology.orchestra.run.smithi060.stdout: 405951 4 -rw-r--r-- 1 ubuntu ubuntu 148 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/features/object-map.yaml 2024-03-27T21:42:44.870 INFO:teuthology.orchestra.run.smithi060.stdout: 405941 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/ceph 2024-03-27T21:42:44.870 INFO:teuthology.orchestra.run.smithi060.stdout: 405943 4 -rw-r--r-- 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/ceph/ceph.yaml 2024-03-27T21:42:44.870 INFO:teuthology.orchestra.run.smithi060.stdout: 405942 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/ceph/.qa -> ../.qa/ 2024-03-27T21:42:44.870 INFO:teuthology.orchestra.run.smithi060.stdout: 405947 4 -rw-r--r-- 1 ubuntu ubuntu 123 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/conf.yaml 2024-03-27T21:42:44.871 INFO:teuthology.orchestra.run.smithi060.stdout: 405979 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/tasks 2024-03-27T21:42:44.871 INFO:teuthology.orchestra.run.smithi060.stdout: 405980 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.871 INFO:teuthology.orchestra.run.smithi060.stdout: 405981 4 -rw-r--r-- 1 ubuntu ubuntu 164 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/tasks/fsx-1-client.yaml 2024-03-27T21:42:44.871 INFO:teuthology.orchestra.run.smithi060.stdout: 405982 4 -rw-r--r-- 1 ubuntu ubuntu 184 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/fsx/tasks/fsx-3-client.yaml 2024-03-27T21:42:44.871 INFO:teuthology.orchestra.run.smithi060.stdout: 405905 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/.qa -> ../.qa/ 2024-03-27T21:42:44.871 INFO:teuthology.orchestra.run.smithi060.stdout: 406020 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount 2024-03-27T21:42:44.871 INFO:teuthology.orchestra.run.smithi060.stdout: 406021 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/% 2024-03-27T21:42:44.871 INFO:teuthology.orchestra.run.smithi060.stdout: 406042 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/msgr-failures 2024-03-27T21:42:44.872 INFO:teuthology.orchestra.run.smithi060.stdout: 406043 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.872 INFO:teuthology.orchestra.run.smithi060.stdout: 406044 4 -rw-r--r-- 1 ubuntu ubuntu 177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/msgr-failures/few.yaml 2024-03-27T21:42:44.872 INFO:teuthology.orchestra.run.smithi060.stdout: 406045 4 -rw-r--r-- 1 ubuntu ubuntu 176 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/msgr-failures/many.yaml 2024-03-27T21:42:44.872 INFO:teuthology.orchestra.run.smithi060.stdout: 406022 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/.qa -> ../.qa/ 2024-03-27T21:42:44.872 INFO:teuthology.orchestra.run.smithi060.stdout: 406023 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.872 INFO:teuthology.orchestra.run.smithi060.stdout: 406028 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/install 2024-03-27T21:42:44.872 INFO:teuthology.orchestra.run.smithi060.stdout: 406030 4 -rw-r--r-- 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/install/ceph.yaml 2024-03-27T21:42:44.872 INFO:teuthology.orchestra.run.smithi060.stdout: 406029 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/install/.qa -> ../.qa/ 2024-03-27T21:42:44.872 INFO:teuthology.orchestra.run.smithi060.stdout: 406031 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/ms_mode 2024-03-27T21:42:44.873 INFO:teuthology.orchestra.run.smithi060.stdout: 406041 4 -rw-r--r-- 1 ubuntu ubuntu 91 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/ms_mode/secure.yaml 2024-03-27T21:42:44.873 INFO:teuthology.orchestra.run.smithi060.stdout: 406032 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/ms_mode/.qa -> ../.qa/ 2024-03-27T21:42:44.873 INFO:teuthology.orchestra.run.smithi060.stdout: 406037 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/ms_mode/legacy$ 2024-03-27T21:42:44.873 INFO:teuthology.orchestra.run.smithi060.stdout: 406039 4 -rw-r--r-- 1 ubuntu ubuntu 100 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/ms_mode/legacy$/legacy-rxbounce.yaml 2024-03-27T21:42:44.873 INFO:teuthology.orchestra.run.smithi060.stdout: 406038 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/ms_mode/legacy$/.qa -> ../.qa/ 2024-03-27T21:42:44.873 INFO:teuthology.orchestra.run.smithi060.stdout: 406040 4 -rw-r--r-- 1 ubuntu ubuntu 91 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/ms_mode/legacy$/legacy.yaml 2024-03-27T21:42:44.873 INFO:teuthology.orchestra.run.smithi060.stdout: 406033 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/ms_mode/crc$ 2024-03-27T21:42:44.873 INFO:teuthology.orchestra.run.smithi060.stdout: 406035 4 -rw-r--r-- 1 ubuntu ubuntu 97 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/ms_mode/crc$/crc-rxbounce.yaml 2024-03-27T21:42:44.874 INFO:teuthology.orchestra.run.smithi060.stdout: 406036 4 -rw-r--r-- 1 ubuntu ubuntu 88 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/ms_mode/crc$/crc.yaml 2024-03-27T21:42:44.874 INFO:teuthology.orchestra.run.smithi060.stdout: 406034 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/ms_mode/crc$/.qa -> ../.qa/ 2024-03-27T21:42:44.874 INFO:teuthology.orchestra.run.smithi060.stdout: 406024 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/clusters 2024-03-27T21:42:44.874 INFO:teuthology.orchestra.run.smithi060.stdout: 406026 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/clusters/fixed-3.yaml -> .qa/clusters/fixed-3.yaml 2024-03-27T21:42:44.874 INFO:teuthology.orchestra.run.smithi060.stdout: 406025 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.874 INFO:teuthology.orchestra.run.smithi060.stdout: 406027 4 -rw-r--r-- 1 ubuntu ubuntu 170 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/conf.yaml 2024-03-27T21:42:44.874 INFO:teuthology.orchestra.run.smithi060.stdout: 406046 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/tasks 2024-03-27T21:42:44.874 INFO:teuthology.orchestra.run.smithi060.stdout: 406052 4 -rw-r--r-- 1 ubuntu ubuntu 86 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/tasks/krbd_latest_osdmap_on_map.yaml 2024-03-27T21:42:44.874 INFO:teuthology.orchestra.run.smithi060.stdout: 406058 4 -rw-r--r-- 1 ubuntu ubuntu 236 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/tasks/rbd_concurrent.yaml 2024-03-27T21:42:44.875 INFO:teuthology.orchestra.run.smithi060.stdout: 406047 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.875 INFO:teuthology.orchestra.run.smithi060.stdout: 406061 4 -rw-r--r-- 1 ubuntu ubuntu 67 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/tasks/rbd_kernel.yaml 2024-03-27T21:42:44.875 INFO:teuthology.orchestra.run.smithi060.stdout: 406063 4 -rw-r--r-- 1 ubuntu ubuntu 70 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/tasks/rbd_map_unmap.yaml 2024-03-27T21:42:44.875 INFO:teuthology.orchestra.run.smithi060.stdout: 406060 4 -rw-r--r-- 1 ubuntu ubuntu 443 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/tasks/rbd_image_read.yaml 2024-03-27T21:42:44.875 INFO:teuthology.orchestra.run.smithi060.stdout: 406054 4 -rw-r--r-- 1 ubuntu ubuntu 80 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/tasks/krbd_udev_enumerate.yaml 2024-03-27T21:42:44.875 INFO:teuthology.orchestra.run.smithi060.stdout: 406057 4 -rw-r--r-- 1 ubuntu ubuntu 79 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/tasks/krbd_udev_symlinks.yaml 2024-03-27T21:42:44.875 INFO:teuthology.orchestra.run.smithi060.stdout: 406051 4 -rw-r--r-- 1 ubuntu ubuntu 144 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/tasks/krbd_huge_osdmap.yaml 2024-03-27T21:42:44.875 INFO:teuthology.orchestra.run.smithi060.stdout: 406055 4 -rw-r--r-- 1 ubuntu ubuntu 164 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/tasks/krbd_udev_netlink_enobufs.yaml 2024-03-27T21:42:44.876 INFO:teuthology.orchestra.run.smithi060.stdout: 406059 4 -rw-r--r-- 1 ubuntu ubuntu 73 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/tasks/rbd_huge_tickets.yaml 2024-03-27T21:42:44.876 INFO:teuthology.orchestra.run.smithi060.stdout: 406064 4 -rw-r--r-- 1 ubuntu ubuntu 72 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/tasks/rbd_simple_big.yaml 2024-03-27T21:42:44.876 INFO:teuthology.orchestra.run.smithi060.stdout: 406048 4 -rw-r--r-- 1 ubuntu ubuntu 75 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/tasks/krbd_data_pool.yaml 2024-03-27T21:42:44.876 INFO:teuthology.orchestra.run.smithi060.stdout: 406056 4 -rw-r--r-- 1 ubuntu ubuntu 76 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/tasks/krbd_udev_netns.yaml 2024-03-27T21:42:44.876 INFO:teuthology.orchestra.run.smithi060.stdout: 406049 4 -rw-r--r-- 1 ubuntu ubuntu 82 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/tasks/krbd_exclusive_option.yaml 2024-03-27T21:42:44.876 INFO:teuthology.orchestra.run.smithi060.stdout: 406062 4 -rw-r--r-- 1 ubuntu ubuntu 76 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/tasks/rbd_map_snapshot_io.yaml 2024-03-27T21:42:44.876 INFO:teuthology.orchestra.run.smithi060.stdout: 406050 4 -rw-r--r-- 1 ubuntu ubuntu 75 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/tasks/krbd_fallocate.yaml 2024-03-27T21:42:44.876 INFO:teuthology.orchestra.run.smithi060.stdout: 406053 4 -rw-r--r-- 1 ubuntu ubuntu 76 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd-nomount/tasks/krbd_namespaces.yaml 2024-03-27T21:42:44.876 INFO:teuthology.orchestra.run.smithi060.stdout: 406118 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton 2024-03-27T21:42:44.877 INFO:teuthology.orchestra.run.smithi060.stdout: 406119 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton/% 2024-03-27T21:42:44.877 INFO:teuthology.orchestra.run.smithi060.stdout: 406120 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton/.qa -> ../.qa/ 2024-03-27T21:42:44.877 INFO:teuthology.orchestra.run.smithi060.stdout: 406123 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton/ms_mode$ 2024-03-27T21:42:44.877 INFO:teuthology.orchestra.run.smithi060.stdout: 406125 4 -rw-r--r-- 1 ubuntu ubuntu 97 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton/ms_mode$/crc-rxbounce.yaml 2024-03-27T21:42:44.877 INFO:teuthology.orchestra.run.smithi060.stdout: 406129 4 -rw-r--r-- 1 ubuntu ubuntu 91 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton/ms_mode$/secure.yaml 2024-03-27T21:42:44.877 INFO:teuthology.orchestra.run.smithi060.stdout: 406127 4 -rw-r--r-- 1 ubuntu ubuntu 100 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton/ms_mode$/legacy-rxbounce.yaml 2024-03-27T21:42:44.877 INFO:teuthology.orchestra.run.smithi060.stdout: 406126 4 -rw-r--r-- 1 ubuntu ubuntu 88 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton/ms_mode$/crc.yaml 2024-03-27T21:42:44.877 INFO:teuthology.orchestra.run.smithi060.stdout: 406124 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton/ms_mode$/.qa -> ../.qa/ 2024-03-27T21:42:44.878 INFO:teuthology.orchestra.run.smithi060.stdout: 406128 4 -rw-r--r-- 1 ubuntu ubuntu 91 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton/ms_mode$/legacy.yaml 2024-03-27T21:42:44.878 INFO:teuthology.orchestra.run.smithi060.stdout: 406121 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.878 INFO:teuthology.orchestra.run.smithi060.stdout: 406122 4 -rw-r--r-- 1 ubuntu ubuntu 170 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton/conf.yaml 2024-03-27T21:42:44.878 INFO:teuthology.orchestra.run.smithi060.stdout: 406130 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton/tasks 2024-03-27T21:42:44.878 INFO:teuthology.orchestra.run.smithi060.stdout: 406131 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.878 INFO:teuthology.orchestra.run.smithi060.stdout: 406132 4 -rw-r--r-- 1 ubuntu ubuntu 305 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton/tasks/krbd_watch_errors.yaml 2024-03-27T21:42:44.878 INFO:teuthology.orchestra.run.smithi060.stdout: 406181 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac 2024-03-27T21:42:44.878 INFO:teuthology.orchestra.run.smithi060.stdout: 406183 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/sysfs 2024-03-27T21:42:44.879 INFO:teuthology.orchestra.run.smithi060.stdout: 406184 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/sysfs/% 2024-03-27T21:42:44.879 INFO:teuthology.orchestra.run.smithi060.stdout: 406185 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/sysfs/.qa -> ../.qa/ 2024-03-27T21:42:44.879 INFO:teuthology.orchestra.run.smithi060.stdout: 406186 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/sysfs/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.879 INFO:teuthology.orchestra.run.smithi060.stdout: 406190 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/sysfs/clusters 2024-03-27T21:42:44.879 INFO:teuthology.orchestra.run.smithi060.stdout: 406192 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/sysfs/clusters/fixed-1.yaml -> .qa/clusters/fixed-1.yaml 2024-03-27T21:42:44.879 INFO:teuthology.orchestra.run.smithi060.stdout: 406191 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/sysfs/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.879 INFO:teuthology.orchestra.run.smithi060.stdout: 406187 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/sysfs/ceph 2024-03-27T21:42:44.879 INFO:teuthology.orchestra.run.smithi060.stdout: 406189 4 -rw-r--r-- 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/sysfs/ceph/ceph.yaml 2024-03-27T21:42:44.880 INFO:teuthology.orchestra.run.smithi060.stdout: 406188 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/sysfs/ceph/.qa -> ../.qa/ 2024-03-27T21:42:44.880 INFO:teuthology.orchestra.run.smithi060.stdout: 406193 4 -rw-r--r-- 1 ubuntu ubuntu 170 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/sysfs/conf.yaml 2024-03-27T21:42:44.880 INFO:teuthology.orchestra.run.smithi060.stdout: 406194 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/sysfs/tasks 2024-03-27T21:42:44.880 INFO:teuthology.orchestra.run.smithi060.stdout: 406195 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/sysfs/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.880 INFO:teuthology.orchestra.run.smithi060.stdout: 406196 4 -rw-r--r-- 1 ubuntu ubuntu 79 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/sysfs/tasks/stable_writes.yaml 2024-03-27T21:42:44.880 INFO:teuthology.orchestra.run.smithi060.stdout: 406182 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/.qa -> ../.qa/ 2024-03-27T21:42:44.880 INFO:teuthology.orchestra.run.smithi060.stdout: 406197 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/wac 2024-03-27T21:42:44.880 INFO:teuthology.orchestra.run.smithi060.stdout: 406198 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/wac/% 2024-03-27T21:42:44.880 INFO:teuthology.orchestra.run.smithi060.stdout: 406199 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/wac/.qa -> ../.qa/ 2024-03-27T21:42:44.880 INFO:teuthology.orchestra.run.smithi060.stdout: 406200 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/wac/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.881 INFO:teuthology.orchestra.run.smithi060.stdout: 406204 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/wac/clusters 2024-03-27T21:42:44.881 INFO:teuthology.orchestra.run.smithi060.stdout: 406206 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/wac/clusters/fixed-3.yaml -> .qa/clusters/fixed-3.yaml 2024-03-27T21:42:44.881 INFO:teuthology.orchestra.run.smithi060.stdout: 406205 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/wac/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.881 INFO:teuthology.orchestra.run.smithi060.stdout: 406211 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/wac/verify 2024-03-27T21:42:44.881 INFO:teuthology.orchestra.run.smithi060.stdout: 406212 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/wac/verify/.qa -> ../.qa/ 2024-03-27T21:42:44.881 INFO:teuthology.orchestra.run.smithi060.stdout: 406213 4 -rw-r--r-- 1 ubuntu ubuntu 323 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/wac/verify/many-resets.yaml 2024-03-27T21:42:44.881 INFO:teuthology.orchestra.run.smithi060.stdout: 406214 4 -rw-r--r-- 1 ubuntu ubuntu 151 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/wac/verify/no-resets.yaml 2024-03-27T21:42:44.881 INFO:teuthology.orchestra.run.smithi060.stdout: 406201 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/wac/ceph 2024-03-27T21:42:44.881 INFO:teuthology.orchestra.run.smithi060.stdout: 406203 4 -rw-r--r-- 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/wac/ceph/ceph.yaml 2024-03-27T21:42:44.882 INFO:teuthology.orchestra.run.smithi060.stdout: 406202 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/wac/ceph/.qa -> ../.qa/ 2024-03-27T21:42:44.882 INFO:teuthology.orchestra.run.smithi060.stdout: 406207 4 -rw-r--r-- 1 ubuntu ubuntu 170 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/wac/conf.yaml 2024-03-27T21:42:44.882 INFO:teuthology.orchestra.run.smithi060.stdout: 406208 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/wac/tasks 2024-03-27T21:42:44.882 INFO:teuthology.orchestra.run.smithi060.stdout: 406209 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/wac/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.882 INFO:teuthology.orchestra.run.smithi060.stdout: 406210 4 -rw-r--r-- 1 ubuntu ubuntu 108 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/wac/wac/tasks/wac.yaml 2024-03-27T21:42:44.882 INFO:teuthology.orchestra.run.smithi060.stdout: 405983 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror 2024-03-27T21:42:44.882 INFO:teuthology.orchestra.run.smithi060.stdout: 405984 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror/% 2024-03-27T21:42:44.882 INFO:teuthology.orchestra.run.smithi060.stdout: 405985 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror/.qa -> ../.qa/ 2024-03-27T21:42:44.882 INFO:teuthology.orchestra.run.smithi060.stdout: 405994 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror/ms_mode$ 2024-03-27T21:42:44.883 INFO:teuthology.orchestra.run.smithi060.stdout: 405996 4 -rw-r--r-- 1 ubuntu ubuntu 97 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror/ms_mode$/crc-rxbounce.yaml 2024-03-27T21:42:44.883 INFO:teuthology.orchestra.run.smithi060.stdout: 406000 4 -rw-r--r-- 1 ubuntu ubuntu 91 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror/ms_mode$/secure.yaml 2024-03-27T21:42:44.883 INFO:teuthology.orchestra.run.smithi060.stdout: 405998 4 -rw-r--r-- 1 ubuntu ubuntu 100 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror/ms_mode$/legacy-rxbounce.yaml 2024-03-27T21:42:44.883 INFO:teuthology.orchestra.run.smithi060.stdout: 405997 4 -rw-r--r-- 1 ubuntu ubuntu 88 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror/ms_mode$/crc.yaml 2024-03-27T21:42:44.883 INFO:teuthology.orchestra.run.smithi060.stdout: 405995 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror/ms_mode$/.qa -> ../.qa/ 2024-03-27T21:42:44.883 INFO:teuthology.orchestra.run.smithi060.stdout: 405999 4 -rw-r--r-- 1 ubuntu ubuntu 91 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror/ms_mode$/legacy.yaml 2024-03-27T21:42:44.883 INFO:teuthology.orchestra.run.smithi060.stdout: 405986 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.884 INFO:teuthology.orchestra.run.smithi060.stdout: 405991 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror/install 2024-03-27T21:42:44.884 INFO:teuthology.orchestra.run.smithi060.stdout: 405993 4 -rw-r--r-- 1 ubuntu ubuntu 253 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror/install/ceph.yaml 2024-03-27T21:42:44.884 INFO:teuthology.orchestra.run.smithi060.stdout: 405992 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror/install/.qa -> ../.qa/ 2024-03-27T21:42:44.884 INFO:teuthology.orchestra.run.smithi060.stdout: 405987 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror/clusters 2024-03-27T21:42:44.884 INFO:teuthology.orchestra.run.smithi060.stdout: 405988 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.884 INFO:teuthology.orchestra.run.smithi060.stdout: 405989 4 -rw-r--r-- 1 ubuntu ubuntu 372 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror/clusters/2-node.yaml 2024-03-27T21:42:44.884 INFO:teuthology.orchestra.run.smithi060.stdout: 405990 4 -rw-r--r-- 1 ubuntu ubuntu 123 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror/conf.yaml 2024-03-27T21:42:44.884 INFO:teuthology.orchestra.run.smithi060.stdout: 406001 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror/tasks 2024-03-27T21:42:44.884 INFO:teuthology.orchestra.run.smithi060.stdout: 406002 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.885 INFO:teuthology.orchestra.run.smithi060.stdout: 406004 4 -rw-r--r-- 1 ubuntu ubuntu 267 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror/tasks/compare-mirror-images.yaml 2024-03-27T21:42:44.885 INFO:teuthology.orchestra.run.smithi060.stdout: 406003 4 -rw-r--r-- 1 ubuntu ubuntu 284 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/mirror/tasks/compare-mirror-image-alternate-primary.yaml 2024-03-27T21:42:44.885 INFO:teuthology.orchestra.run.smithi060.stdout: 406133 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash 2024-03-27T21:42:44.885 INFO:teuthology.orchestra.run.smithi060.stdout: 406134 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/% 2024-03-27T21:42:44.885 INFO:teuthology.orchestra.run.smithi060.stdout: 406135 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/.qa -> ../.qa/ 2024-03-27T21:42:44.885 INFO:teuthology.orchestra.run.smithi060.stdout: 406144 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/ms_mode$ 2024-03-27T21:42:44.885 INFO:teuthology.orchestra.run.smithi060.stdout: 406146 4 -rw-r--r-- 1 ubuntu ubuntu 97 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/ms_mode$/crc-rxbounce.yaml 2024-03-27T21:42:44.885 INFO:teuthology.orchestra.run.smithi060.stdout: 406150 4 -rw-r--r-- 1 ubuntu ubuntu 91 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/ms_mode$/secure.yaml 2024-03-27T21:42:44.886 INFO:teuthology.orchestra.run.smithi060.stdout: 406148 4 -rw-r--r-- 1 ubuntu ubuntu 100 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/ms_mode$/legacy-rxbounce.yaml 2024-03-27T21:42:44.886 INFO:teuthology.orchestra.run.smithi060.stdout: 406147 4 -rw-r--r-- 1 ubuntu ubuntu 88 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/ms_mode$/crc.yaml 2024-03-27T21:42:44.886 INFO:teuthology.orchestra.run.smithi060.stdout: 406145 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/ms_mode$/.qa -> ../.qa/ 2024-03-27T21:42:44.886 INFO:teuthology.orchestra.run.smithi060.stdout: 406149 4 -rw-r--r-- 1 ubuntu ubuntu 91 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/ms_mode$/legacy.yaml 2024-03-27T21:42:44.886 INFO:teuthology.orchestra.run.smithi060.stdout: 406136 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.886 INFO:teuthology.orchestra.run.smithi060.stdout: 406157 0 lrwxrwxrwx 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/thrashosds-health.yaml -> .qa/tasks/thrashosds-health.yaml 2024-03-27T21:42:44.886 INFO:teuthology.orchestra.run.smithi060.stdout: 406158 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/workloads 2024-03-27T21:42:44.886 INFO:teuthology.orchestra.run.smithi060.stdout: 406160 4 -rw-r--r-- 1 ubuntu ubuntu 189 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/workloads/krbd_diff_continuous.yaml 2024-03-27T21:42:44.886 INFO:teuthology.orchestra.run.smithi060.stdout: 406159 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.887 INFO:teuthology.orchestra.run.smithi060.stdout: 406161 4 -rw-r--r-- 1 ubuntu ubuntu 223 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/workloads/rbd_fio.yaml 2024-03-27T21:42:44.887 INFO:teuthology.orchestra.run.smithi060.stdout: 406162 4 -rw-r--r-- 1 ubuntu ubuntu 108 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/workloads/rbd_workunit_suites_ffsb.yaml 2024-03-27T21:42:44.887 INFO:teuthology.orchestra.run.smithi060.stdout: 406140 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/clusters 2024-03-27T21:42:44.887 INFO:teuthology.orchestra.run.smithi060.stdout: 406142 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/clusters/fixed-3.yaml -> .qa/clusters/fixed-3.yaml 2024-03-27T21:42:44.887 INFO:teuthology.orchestra.run.smithi060.stdout: 406141 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.887 INFO:teuthology.orchestra.run.smithi060.stdout: 406137 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/ceph 2024-03-27T21:42:44.887 INFO:teuthology.orchestra.run.smithi060.stdout: 406139 4 -rw-r--r-- 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/ceph/ceph.yaml 2024-03-27T21:42:44.887 INFO:teuthology.orchestra.run.smithi060.stdout: 406138 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/ceph/.qa -> ../.qa/ 2024-03-27T21:42:44.888 INFO:teuthology.orchestra.run.smithi060.stdout: 406143 4 -rw-r--r-- 1 ubuntu ubuntu 170 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/conf.yaml 2024-03-27T21:42:44.888 INFO:teuthology.orchestra.run.smithi060.stdout: 406151 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/thrashers 2024-03-27T21:42:44.888 INFO:teuthology.orchestra.run.smithi060.stdout: 406154 4 -rw-r--r-- 1 ubuntu ubuntu 122 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/thrashers/mon-thrasher.yaml 2024-03-27T21:42:44.888 INFO:teuthology.orchestra.run.smithi060.stdout: 406152 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/thrashers/.qa -> ../.qa/ 2024-03-27T21:42:44.888 INFO:teuthology.orchestra.run.smithi060.stdout: 406155 4 -rw-r--r-- 1 ubuntu ubuntu 200 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/thrashers/pggrow.yaml 2024-03-27T21:42:44.888 INFO:teuthology.orchestra.run.smithi060.stdout: 406156 4 -rw-r--r-- 1 ubuntu ubuntu 401 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/thrashers/upmap.yaml 2024-03-27T21:42:44.888 INFO:teuthology.orchestra.run.smithi060.stdout: 406153 4 -rw-r--r-- 1 ubuntu ubuntu 322 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/thrash/thrashers/backoff.yaml 2024-03-27T21:42:44.888 INFO:teuthology.orchestra.run.smithi060.stdout: 406005 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/ms_modeless 2024-03-27T21:42:44.889 INFO:teuthology.orchestra.run.smithi060.stdout: 406006 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/ms_modeless/% 2024-03-27T21:42:44.889 INFO:teuthology.orchestra.run.smithi060.stdout: 406007 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/ms_modeless/.qa -> ../.qa/ 2024-03-27T21:42:44.889 INFO:teuthology.orchestra.run.smithi060.stdout: 406008 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/ms_modeless/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.889 INFO:teuthology.orchestra.run.smithi060.stdout: 406012 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/ms_modeless/clusters 2024-03-27T21:42:44.889 INFO:teuthology.orchestra.run.smithi060.stdout: 406014 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/ms_modeless/clusters/fixed-3.yaml -> .qa/clusters/fixed-3.yaml 2024-03-27T21:42:44.889 INFO:teuthology.orchestra.run.smithi060.stdout: 406013 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/ms_modeless/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.889 INFO:teuthology.orchestra.run.smithi060.stdout: 406009 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/ms_modeless/ceph 2024-03-27T21:42:44.889 INFO:teuthology.orchestra.run.smithi060.stdout: 406011 4 -rw-r--r-- 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/ms_modeless/ceph/ceph.yaml 2024-03-27T21:42:44.889 INFO:teuthology.orchestra.run.smithi060.stdout: 406010 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/ms_modeless/ceph/.qa -> ../.qa/ 2024-03-27T21:42:44.890 INFO:teuthology.orchestra.run.smithi060.stdout: 406015 4 -rw-r--r-- 1 ubuntu ubuntu 123 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/ms_modeless/conf.yaml 2024-03-27T21:42:44.890 INFO:teuthology.orchestra.run.smithi060.stdout: 406016 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/ms_modeless/tasks 2024-03-27T21:42:44.890 INFO:teuthology.orchestra.run.smithi060.stdout: 406019 4 -rw-r--r-- 1 ubuntu ubuntu 74 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/ms_modeless/tasks/krbd_rxbounce.yaml 2024-03-27T21:42:44.890 INFO:teuthology.orchestra.run.smithi060.stdout: 406017 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/ms_modeless/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.890 INFO:teuthology.orchestra.run.smithi060.stdout: 406018 4 -rw-r--r-- 1 ubuntu ubuntu 86 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/ms_modeless/tasks/krbd_default_map_options.yaml 2024-03-27T21:42:44.890 INFO:teuthology.orchestra.run.smithi060.stdout: 406099 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton-msgr-failures 2024-03-27T21:42:44.890 INFO:teuthology.orchestra.run.smithi060.stdout: 406100 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton-msgr-failures/% 2024-03-27T21:42:44.890 INFO:teuthology.orchestra.run.smithi060.stdout: 406111 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton-msgr-failures/msgr-failures 2024-03-27T21:42:44.891 INFO:teuthology.orchestra.run.smithi060.stdout: 406112 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton-msgr-failures/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.891 INFO:teuthology.orchestra.run.smithi060.stdout: 406113 4 -rw-r--r-- 1 ubuntu ubuntu 177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton-msgr-failures/msgr-failures/few.yaml 2024-03-27T21:42:44.891 INFO:teuthology.orchestra.run.smithi060.stdout: 406114 4 -rw-r--r-- 1 ubuntu ubuntu 176 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton-msgr-failures/msgr-failures/many.yaml 2024-03-27T21:42:44.891 INFO:teuthology.orchestra.run.smithi060.stdout: 406101 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton-msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.891 INFO:teuthology.orchestra.run.smithi060.stdout: 406104 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton-msgr-failures/ms_mode$ 2024-03-27T21:42:44.891 INFO:teuthology.orchestra.run.smithi060.stdout: 406106 4 -rw-r--r-- 1 ubuntu ubuntu 97 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton-msgr-failures/ms_mode$/crc-rxbounce.yaml 2024-03-27T21:42:44.891 INFO:teuthology.orchestra.run.smithi060.stdout: 406110 4 -rw-r--r-- 1 ubuntu ubuntu 91 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton-msgr-failures/ms_mode$/secure.yaml 2024-03-27T21:42:44.891 INFO:teuthology.orchestra.run.smithi060.stdout: 406108 4 -rw-r--r-- 1 ubuntu ubuntu 100 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton-msgr-failures/ms_mode$/legacy-rxbounce.yaml 2024-03-27T21:42:44.891 INFO:teuthology.orchestra.run.smithi060.stdout: 406107 4 -rw-r--r-- 1 ubuntu ubuntu 88 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton-msgr-failures/ms_mode$/crc.yaml 2024-03-27T21:42:44.892 INFO:teuthology.orchestra.run.smithi060.stdout: 406105 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton-msgr-failures/ms_mode$/.qa -> ../.qa/ 2024-03-27T21:42:44.892 INFO:teuthology.orchestra.run.smithi060.stdout: 406109 4 -rw-r--r-- 1 ubuntu ubuntu 91 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton-msgr-failures/ms_mode$/legacy.yaml 2024-03-27T21:42:44.892 INFO:teuthology.orchestra.run.smithi060.stdout: 406102 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton-msgr-failures/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.892 INFO:teuthology.orchestra.run.smithi060.stdout: 406103 4 -rw-r--r-- 1 ubuntu ubuntu 170 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton-msgr-failures/conf.yaml 2024-03-27T21:42:44.892 INFO:teuthology.orchestra.run.smithi060.stdout: 406115 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton-msgr-failures/tasks 2024-03-27T21:42:44.892 INFO:teuthology.orchestra.run.smithi060.stdout: 406116 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton-msgr-failures/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.892 INFO:teuthology.orchestra.run.smithi060.stdout: 406117 4 -rw-r--r-- 1 ubuntu ubuntu 917 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/singleton-msgr-failures/tasks/rbd_xfstests.yaml 2024-03-27T21:42:44.892 INFO:teuthology.orchestra.run.smithi060.stdout: 406065 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd 2024-03-27T21:42:44.892 INFO:teuthology.orchestra.run.smithi060.stdout: 406066 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/% 2024-03-27T21:42:44.893 INFO:teuthology.orchestra.run.smithi060.stdout: 406084 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/msgr-failures 2024-03-27T21:42:44.893 INFO:teuthology.orchestra.run.smithi060.stdout: 406085 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.893 INFO:teuthology.orchestra.run.smithi060.stdout: 406086 4 -rw-r--r-- 1 ubuntu ubuntu 177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/msgr-failures/few.yaml 2024-03-27T21:42:44.893 INFO:teuthology.orchestra.run.smithi060.stdout: 406087 4 -rw-r--r-- 1 ubuntu ubuntu 176 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/msgr-failures/many.yaml 2024-03-27T21:42:44.893 INFO:teuthology.orchestra.run.smithi060.stdout: 406067 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/.qa -> ../.qa/ 2024-03-27T21:42:44.893 INFO:teuthology.orchestra.run.smithi060.stdout: 406068 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.893 INFO:teuthology.orchestra.run.smithi060.stdout: 406073 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/ms_mode 2024-03-27T21:42:44.893 INFO:teuthology.orchestra.run.smithi060.stdout: 406083 4 -rw-r--r-- 1 ubuntu ubuntu 91 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/ms_mode/secure.yaml 2024-03-27T21:42:44.893 INFO:teuthology.orchestra.run.smithi060.stdout: 406074 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/ms_mode/.qa -> ../.qa/ 2024-03-27T21:42:44.894 INFO:teuthology.orchestra.run.smithi060.stdout: 406079 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/ms_mode/legacy$ 2024-03-27T21:42:44.894 INFO:teuthology.orchestra.run.smithi060.stdout: 406081 4 -rw-r--r-- 1 ubuntu ubuntu 100 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/ms_mode/legacy$/legacy-rxbounce.yaml 2024-03-27T21:42:44.894 INFO:teuthology.orchestra.run.smithi060.stdout: 406080 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/ms_mode/legacy$/.qa -> ../.qa/ 2024-03-27T21:42:44.894 INFO:teuthology.orchestra.run.smithi060.stdout: 406082 4 -rw-r--r-- 1 ubuntu ubuntu 91 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/ms_mode/legacy$/legacy.yaml 2024-03-27T21:42:44.894 INFO:teuthology.orchestra.run.smithi060.stdout: 406075 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/ms_mode/crc$ 2024-03-27T21:42:44.894 INFO:teuthology.orchestra.run.smithi060.stdout: 406077 4 -rw-r--r-- 1 ubuntu ubuntu 97 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/ms_mode/crc$/crc-rxbounce.yaml 2024-03-27T21:42:44.894 INFO:teuthology.orchestra.run.smithi060.stdout: 406078 4 -rw-r--r-- 1 ubuntu ubuntu 88 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/ms_mode/crc$/crc.yaml 2024-03-27T21:42:44.894 INFO:teuthology.orchestra.run.smithi060.stdout: 406076 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/ms_mode/crc$/.qa -> ../.qa/ 2024-03-27T21:42:44.895 INFO:teuthology.orchestra.run.smithi060.stdout: 406069 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/clusters 2024-03-27T21:42:44.895 INFO:teuthology.orchestra.run.smithi060.stdout: 406071 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/clusters/fixed-3.yaml -> .qa/clusters/fixed-3.yaml 2024-03-27T21:42:44.895 INFO:teuthology.orchestra.run.smithi060.stdout: 406070 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.895 INFO:teuthology.orchestra.run.smithi060.stdout: 406072 4 -rw-r--r-- 1 ubuntu ubuntu 170 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/conf.yaml 2024-03-27T21:42:44.895 INFO:teuthology.orchestra.run.smithi060.stdout: 406088 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/tasks 2024-03-27T21:42:44.895 INFO:teuthology.orchestra.run.smithi060.stdout: 406089 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.895 INFO:teuthology.orchestra.run.smithi060.stdout: 406097 4 -rw-r--r-- 1 ubuntu ubuntu 129 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/tasks/rbd_workunit_suites_iozone.yaml 2024-03-27T21:42:44.895 INFO:teuthology.orchestra.run.smithi060.stdout: 406090 4 -rw-r--r-- 1 ubuntu ubuntu 206 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/tasks/rbd_fio.yaml 2024-03-27T21:42:44.896 INFO:teuthology.orchestra.run.smithi060.stdout: 406094 4 -rw-r--r-- 1 ubuntu ubuntu 107 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/tasks/rbd_workunit_suites_fsstress.yaml 2024-03-27T21:42:44.896 INFO:teuthology.orchestra.run.smithi060.stdout: 406096 4 -rw-r--r-- 1 ubuntu ubuntu 309 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/tasks/rbd_workunit_suites_fsx.yaml 2024-03-27T21:42:44.896 INFO:teuthology.orchestra.run.smithi060.stdout: 406091 4 -rw-r--r-- 1 ubuntu ubuntu 265 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/tasks/rbd_workunit_kernel_untar_build.yaml 2024-03-27T21:42:44.896 INFO:teuthology.orchestra.run.smithi060.stdout: 406092 4 -rw-r--r-- 1 ubuntu ubuntu 105 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/tasks/rbd_workunit_suites_dbench.yaml 2024-03-27T21:42:44.896 INFO:teuthology.orchestra.run.smithi060.stdout: 406098 4 -rw-r--r-- 1 ubuntu ubuntu 104 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/tasks/rbd_workunit_trivial_sync.yaml 2024-03-27T21:42:44.896 INFO:teuthology.orchestra.run.smithi060.stdout: 406095 4 -rw-r--r-- 1 ubuntu ubuntu 127 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/tasks/rbd_workunit_suites_fsstress_ext4.yaml 2024-03-27T21:42:44.896 INFO:teuthology.orchestra.run.smithi060.stdout: 406093 4 -rw-r--r-- 1 ubuntu ubuntu 127 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/rbd/tasks/rbd_workunit_suites_ffsb.yaml 2024-03-27T21:42:44.896 INFO:teuthology.orchestra.run.smithi060.stdout: 405906 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic 2024-03-27T21:42:44.897 INFO:teuthology.orchestra.run.smithi060.stdout: 405907 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/% 2024-03-27T21:42:44.897 INFO:teuthology.orchestra.run.smithi060.stdout: 405908 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/.qa -> ../.qa/ 2024-03-27T21:42:44.897 INFO:teuthology.orchestra.run.smithi060.stdout: 405909 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.897 INFO:teuthology.orchestra.run.smithi060.stdout: 405917 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/ms_mode 2024-03-27T21:42:44.897 INFO:teuthology.orchestra.run.smithi060.stdout: 405927 4 -rw-r--r-- 1 ubuntu ubuntu 91 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/ms_mode/secure.yaml 2024-03-27T21:42:44.897 INFO:teuthology.orchestra.run.smithi060.stdout: 405918 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/ms_mode/.qa -> ../.qa/ 2024-03-27T21:42:44.897 INFO:teuthology.orchestra.run.smithi060.stdout: 405923 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/ms_mode/legacy$ 2024-03-27T21:42:44.897 INFO:teuthology.orchestra.run.smithi060.stdout: 405925 4 -rw-r--r-- 1 ubuntu ubuntu 100 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/ms_mode/legacy$/legacy-rxbounce.yaml 2024-03-27T21:42:44.898 INFO:teuthology.orchestra.run.smithi060.stdout: 405924 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/ms_mode/legacy$/.qa -> ../.qa/ 2024-03-27T21:42:44.898 INFO:teuthology.orchestra.run.smithi060.stdout: 405926 4 -rw-r--r-- 1 ubuntu ubuntu 91 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/ms_mode/legacy$/legacy.yaml 2024-03-27T21:42:44.898 INFO:teuthology.orchestra.run.smithi060.stdout: 405919 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/ms_mode/crc$ 2024-03-27T21:42:44.898 INFO:teuthology.orchestra.run.smithi060.stdout: 405921 4 -rw-r--r-- 1 ubuntu ubuntu 97 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/ms_mode/crc$/crc-rxbounce.yaml 2024-03-27T21:42:44.898 INFO:teuthology.orchestra.run.smithi060.stdout: 405922 4 -rw-r--r-- 1 ubuntu ubuntu 88 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/ms_mode/crc$/crc.yaml 2024-03-27T21:42:44.898 INFO:teuthology.orchestra.run.smithi060.stdout: 405920 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/ms_mode/crc$/.qa -> ../.qa/ 2024-03-27T21:42:44.898 INFO:teuthology.orchestra.run.smithi060.stdout: 405913 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/clusters 2024-03-27T21:42:44.898 INFO:teuthology.orchestra.run.smithi060.stdout: 405915 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/clusters/fixed-1.yaml -> .qa/clusters/fixed-1.yaml 2024-03-27T21:42:44.899 INFO:teuthology.orchestra.run.smithi060.stdout: 405914 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.899 INFO:teuthology.orchestra.run.smithi060.stdout: 405910 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/ceph 2024-03-27T21:42:44.899 INFO:teuthology.orchestra.run.smithi060.stdout: 405912 4 -rw-r--r-- 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/ceph/ceph.yaml 2024-03-27T21:42:44.899 INFO:teuthology.orchestra.run.smithi060.stdout: 405911 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/ceph/.qa -> ../.qa/ 2024-03-27T21:42:44.899 INFO:teuthology.orchestra.run.smithi060.stdout: 405916 4 -rw-r--r-- 1 ubuntu ubuntu 170 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/conf.yaml 2024-03-27T21:42:44.899 INFO:teuthology.orchestra.run.smithi060.stdout: 405928 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/tasks 2024-03-27T21:42:44.899 INFO:teuthology.orchestra.run.smithi060.stdout: 405935 4 -rw-r--r-- 1 ubuntu ubuntu 81 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/tasks/krbd_parent_overlap.yaml 2024-03-27T21:42:44.899 INFO:teuthology.orchestra.run.smithi060.stdout: 405936 4 -rw-r--r-- 1 ubuntu ubuntu 110 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/tasks/krbd_read_only.yaml 2024-03-27T21:42:44.899 INFO:teuthology.orchestra.run.smithi060.stdout: 405937 4 -rw-r--r-- 1 ubuntu ubuntu 87 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/tasks/krbd_whole_object_zeroout.yaml 2024-03-27T21:42:44.900 INFO:teuthology.orchestra.run.smithi060.stdout: 405929 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.900 INFO:teuthology.orchestra.run.smithi060.stdout: 405933 4 -rw-r--r-- 1 ubuntu ubuntu 75 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/tasks/krbd_modprobe.yaml 2024-03-27T21:42:44.900 INFO:teuthology.orchestra.run.smithi060.stdout: 405932 4 -rw-r--r-- 1 ubuntu ubuntu 77 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/tasks/krbd_huge_image.yaml 2024-03-27T21:42:44.900 INFO:teuthology.orchestra.run.smithi060.stdout: 405934 4 -rw-r--r-- 1 ubuntu ubuntu 80 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/tasks/krbd_msgr_segments.yaml 2024-03-27T21:42:44.900 INFO:teuthology.orchestra.run.smithi060.stdout: 405931 4 -rw-r--r-- 1 ubuntu ubuntu 214 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/tasks/krbd_discard.yaml 2024-03-27T21:42:44.900 INFO:teuthology.orchestra.run.smithi060.stdout: 405930 4 -rw-r--r-- 1 ubuntu ubuntu 79 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/basic/tasks/krbd_deep_flatten.yaml 2024-03-27T21:42:44.900 INFO:teuthology.orchestra.run.smithi060.stdout: 406163 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/unmap 2024-03-27T21:42:44.900 INFO:teuthology.orchestra.run.smithi060.stdout: 406164 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/unmap/% 2024-03-27T21:42:44.901 INFO:teuthology.orchestra.run.smithi060.stdout: 406165 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/unmap/.qa -> ../.qa/ 2024-03-27T21:42:44.901 INFO:teuthology.orchestra.run.smithi060.stdout: 406169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/unmap/clusters 2024-03-27T21:42:44.901 INFO:teuthology.orchestra.run.smithi060.stdout: 406170 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/unmap/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.901 INFO:teuthology.orchestra.run.smithi060.stdout: 406171 4 -rw-r--r-- 1 ubuntu ubuntu 391 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/unmap/clusters/separate-client.yaml 2024-03-27T21:42:44.901 INFO:teuthology.orchestra.run.smithi060.stdout: 406166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/unmap/ceph 2024-03-27T21:42:44.901 INFO:teuthology.orchestra.run.smithi060.stdout: 406168 4 -rw-r--r-- 1 ubuntu ubuntu 451 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/unmap/ceph/ceph.yaml 2024-03-27T21:42:44.901 INFO:teuthology.orchestra.run.smithi060.stdout: 406167 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/unmap/ceph/.qa -> ../.qa/ 2024-03-27T21:42:44.901 INFO:teuthology.orchestra.run.smithi060.stdout: 406172 4 -rw-r--r-- 1 ubuntu ubuntu 174 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/unmap/conf.yaml 2024-03-27T21:42:44.901 INFO:teuthology.orchestra.run.smithi060.stdout: 406178 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/unmap/tasks 2024-03-27T21:42:44.902 INFO:teuthology.orchestra.run.smithi060.stdout: 406179 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/unmap/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.902 INFO:teuthology.orchestra.run.smithi060.stdout: 406180 4 -rw-r--r-- 1 ubuntu ubuntu 89 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/unmap/tasks/unmap.yaml 2024-03-27T21:42:44.902 INFO:teuthology.orchestra.run.smithi060.stdout: 406173 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/unmap/kernels 2024-03-27T21:42:44.902 INFO:teuthology.orchestra.run.smithi060.stdout: 406174 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/unmap/kernels/.qa -> ../.qa/ 2024-03-27T21:42:44.902 INFO:teuthology.orchestra.run.smithi060.stdout: 406177 4 -rw-r--r-- 1 ubuntu ubuntu 160 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/unmap/kernels/single-major-on.yaml 2024-03-27T21:42:44.902 INFO:teuthology.orchestra.run.smithi060.stdout: 406175 4 -rw-r--r-- 1 ubuntu ubuntu 229 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/unmap/kernels/pre-single-major.yaml 2024-03-27T21:42:44.902 INFO:teuthology.orchestra.run.smithi060.stdout: 406176 4 -rw-r--r-- 1 ubuntu ubuntu 160 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/krbd/unmap/kernels/single-major-off.yaml 2024-03-27T21:42:44.902 INFO:teuthology.orchestra.run.smithi060.stdout: 408210 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade 2024-03-27T21:42:44.903 INFO:teuthology.orchestra.run.smithi060.stdout: 408212 0 lrwxrwxrwx 1 ubuntu ubuntu 22 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/cephfs -> .qa/suites/fs/upgrade/ 2024-03-27T21:42:44.903 INFO:teuthology.orchestra.run.smithi060.stdout: 408211 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/.qa -> ../.qa/ 2024-03-27T21:42:44.903 INFO:teuthology.orchestra.run.smithi060.stdout: 408323 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/telemetry-upgrade 2024-03-27T21:42:44.903 INFO:teuthology.orchestra.run.smithi060.stdout: 408324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/telemetry-upgrade/pacific-x 2024-03-27T21:42:44.903 INFO:teuthology.orchestra.run.smithi060.stdout: 408327 0 lrwxrwxrwx 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/telemetry-upgrade/pacific-x/0-random-distro$ -> .qa/distros/container-hosts 2024-03-27T21:42:44.903 INFO:teuthology.orchestra.run.smithi060.stdout: 408325 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/telemetry-upgrade/pacific-x/% 2024-03-27T21:42:44.903 INFO:teuthology.orchestra.run.smithi060.stdout: 408326 0 lrwxrwxrwx 1 ubuntu ubuntu 9 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/telemetry-upgrade/pacific-x/.qa -> ../../.qa 2024-03-27T21:42:44.903 INFO:teuthology.orchestra.run.smithi060.stdout: 408328 4 -rw-r--r-- 1 ubuntu ubuntu 460 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/telemetry-upgrade/pacific-x/0-start.yaml 2024-03-27T21:42:44.903 INFO:teuthology.orchestra.run.smithi060.stdout: 408329 4 -rw-r--r-- 1 ubuntu ubuntu 1823 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/telemetry-upgrade/pacific-x/1-tasks.yaml 2024-03-27T21:42:44.904 INFO:teuthology.orchestra.run.smithi060.stdout: 408330 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/telemetry-upgrade/quincy-x 2024-03-27T21:42:44.904 INFO:teuthology.orchestra.run.smithi060.stdout: 408333 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/telemetry-upgrade/quincy-x/0-random-distro$ -> .qa/distros/supported-container-hosts 2024-03-27T21:42:44.904 INFO:teuthology.orchestra.run.smithi060.stdout: 408331 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/telemetry-upgrade/quincy-x/% 2024-03-27T21:42:44.904 INFO:teuthology.orchestra.run.smithi060.stdout: 408332 0 lrwxrwxrwx 1 ubuntu ubuntu 9 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/telemetry-upgrade/quincy-x/.qa -> ../../.qa 2024-03-27T21:42:44.904 INFO:teuthology.orchestra.run.smithi060.stdout: 408334 4 -rw-r--r-- 1 ubuntu ubuntu 460 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/telemetry-upgrade/quincy-x/0-start.yaml 2024-03-27T21:42:44.904 INFO:teuthology.orchestra.run.smithi060.stdout: 408335 4 -rw-r--r-- 1 ubuntu ubuntu 1813 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/telemetry-upgrade/quincy-x/1-tasks.yaml 2024-03-27T21:42:44.904 INFO:teuthology.orchestra.run.smithi060.stdout: 408276 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x 2024-03-27T21:42:44.904 INFO:teuthology.orchestra.run.smithi060.stdout: 408277 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/.qa -> ../.qa/ 2024-03-27T21:42:44.905 INFO:teuthology.orchestra.run.smithi060.stdout: 408295 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split 2024-03-27T21:42:44.905 INFO:teuthology.orchestra.run.smithi060.stdout: 408301 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/2-first-half-tasks 2024-03-27T21:42:44.905 INFO:teuthology.orchestra.run.smithi060.stdout: 408302 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/2-first-half-tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.905 INFO:teuthology.orchestra.run.smithi060.stdout: 408308 4 -rw-r--r-- 1 ubuntu ubuntu 452 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/2-first-half-tasks/snaps-few-objects.yaml 2024-03-27T21:42:44.905 INFO:teuthology.orchestra.run.smithi060.stdout: 408303 4 -rw-r--r-- 1 ubuntu ubuntu 418 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/2-first-half-tasks/radosbench.yaml 2024-03-27T21:42:44.905 INFO:teuthology.orchestra.run.smithi060.stdout: 408305 4 -rw-r--r-- 1 ubuntu ubuntu 266 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/2-first-half-tasks/rbd-import-export.yaml 2024-03-27T21:42:44.905 INFO:teuthology.orchestra.run.smithi060.stdout: 408306 4 -rw-r--r-- 1 ubuntu ubuntu 245 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/2-first-half-tasks/rbd_api.yaml 2024-03-27T21:42:44.905 INFO:teuthology.orchestra.run.smithi060.stdout: 408304 4 -rw-r--r-- 1 ubuntu ubuntu 205 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/2-first-half-tasks/rbd-cls.yaml 2024-03-27T21:42:44.905 INFO:teuthology.orchestra.run.smithi060.stdout: 408307 4 -rw-r--r-- 1 ubuntu ubuntu 391 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/2-first-half-tasks/readwrite.yaml 2024-03-27T21:42:44.906 INFO:teuthology.orchestra.run.smithi060.stdout: 408296 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/% 2024-03-27T21:42:44.906 INFO:teuthology.orchestra.run.smithi060.stdout: 408297 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/.qa -> ../.qa/ 2024-03-27T21:42:44.906 INFO:teuthology.orchestra.run.smithi060.stdout: 408299 4 -rw-r--r-- 1 ubuntu ubuntu 440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/0-roles.yaml 2024-03-27T21:42:44.906 INFO:teuthology.orchestra.run.smithi060.stdout: 408309 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/3-stress-tasks 2024-03-27T21:42:44.906 INFO:teuthology.orchestra.run.smithi060.stdout: 408310 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/3-stress-tasks/+ 2024-03-27T21:42:44.906 INFO:teuthology.orchestra.run.smithi060.stdout: 408311 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/3-stress-tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.906 INFO:teuthology.orchestra.run.smithi060.stdout: 408317 4 -rw-r--r-- 1 ubuntu ubuntu 448 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/3-stress-tasks/snaps-few-objects.yaml 2024-03-27T21:42:44.906 INFO:teuthology.orchestra.run.smithi060.stdout: 408312 4 -rw-r--r-- 1 ubuntu ubuntu 528 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/3-stress-tasks/radosbench.yaml 2024-03-27T21:42:44.907 INFO:teuthology.orchestra.run.smithi060.stdout: 408314 4 -rw-r--r-- 1 ubuntu ubuntu 262 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/3-stress-tasks/rbd-import-export.yaml 2024-03-27T21:42:44.907 INFO:teuthology.orchestra.run.smithi060.stdout: 408315 4 -rw-r--r-- 1 ubuntu ubuntu 241 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/3-stress-tasks/rbd_api.yaml 2024-03-27T21:42:44.907 INFO:teuthology.orchestra.run.smithi060.stdout: 408313 4 -rw-r--r-- 1 ubuntu ubuntu 201 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/3-stress-tasks/rbd-cls.yaml 2024-03-27T21:42:44.907 INFO:teuthology.orchestra.run.smithi060.stdout: 408316 4 -rw-r--r-- 1 ubuntu ubuntu 387 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/3-stress-tasks/readwrite.yaml 2024-03-27T21:42:44.907 INFO:teuthology.orchestra.run.smithi060.stdout: 408300 4 -rw-r--r-- 1 ubuntu ubuntu 3138 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/1-start.yaml 2024-03-27T21:42:44.907 INFO:teuthology.orchestra.run.smithi060.stdout: 408322 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/mon_election -> .qa/mon_election 2024-03-27T21:42:44.907 INFO:teuthology.orchestra.run.smithi060.stdout: 408298 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/0-distro -> .qa/distros/supported-container-hosts 2024-03-27T21:42:44.907 INFO:teuthology.orchestra.run.smithi060.stdout: 408318 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/4-second-half-tasks 2024-03-27T21:42:44.907 INFO:teuthology.orchestra.run.smithi060.stdout: 408319 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/4-second-half-tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.908 INFO:teuthology.orchestra.run.smithi060.stdout: 408320 4 -rw-r--r-- 1 ubuntu ubuntu 362 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/4-second-half-tasks/radosbench.yaml 2024-03-27T21:42:44.908 INFO:teuthology.orchestra.run.smithi060.stdout: 408321 4 -rw-r--r-- 1 ubuntu ubuntu 267 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/stress-split/4-second-half-tasks/rbd-import-export.yaml 2024-03-27T21:42:44.908 INFO:teuthology.orchestra.run.smithi060.stdout: 408278 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/parallel 2024-03-27T21:42:44.908 INFO:teuthology.orchestra.run.smithi060.stdout: 408285 4 -rw-r--r-- 1 ubuntu ubuntu 650 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/parallel/upgrade-sequence.yaml 2024-03-27T21:42:44.908 INFO:teuthology.orchestra.run.smithi060.stdout: 408281 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/parallel/0-random-distro$ -> .qa/distros/supported-container-hosts 2024-03-27T21:42:44.908 INFO:teuthology.orchestra.run.smithi060.stdout: 408279 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/parallel/% 2024-03-27T21:42:44.908 INFO:teuthology.orchestra.run.smithi060.stdout: 408280 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/parallel/.qa -> ../.qa/ 2024-03-27T21:42:44.908 INFO:teuthology.orchestra.run.smithi060.stdout: 408282 4 -rw-r--r-- 1 ubuntu ubuntu 460 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/parallel/0-start.yaml 2024-03-27T21:42:44.909 INFO:teuthology.orchestra.run.smithi060.stdout: 408283 4 -rw-r--r-- 1 ubuntu ubuntu 1143 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/parallel/1-tasks.yaml 2024-03-27T21:42:44.909 INFO:teuthology.orchestra.run.smithi060.stdout: 408286 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/parallel/workload 2024-03-27T21:42:44.909 INFO:teuthology.orchestra.run.smithi060.stdout: 408287 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/parallel/workload/+ 2024-03-27T21:42:44.909 INFO:teuthology.orchestra.run.smithi060.stdout: 408294 4 -rw-r--r-- 1 ubuntu ubuntu 415 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/parallel/workload/test_rbd_python.yaml 2024-03-27T21:42:44.909 INFO:teuthology.orchestra.run.smithi060.stdout: 408288 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/parallel/workload/.qa -> ../.qa/ 2024-03-27T21:42:44.909 INFO:teuthology.orchestra.run.smithi060.stdout: 408292 4 -rw-r--r-- 1 ubuntu ubuntu 360 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/parallel/workload/rbd_import_export.yaml 2024-03-27T21:42:44.909 INFO:teuthology.orchestra.run.smithi060.stdout: 408293 4 -rw-r--r-- 1 ubuntu ubuntu 322 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/parallel/workload/test_rbd_api.yaml 2024-03-27T21:42:44.909 INFO:teuthology.orchestra.run.smithi060.stdout: 408290 4 -rw-r--r-- 1 ubuntu ubuntu 260 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/parallel/workload/rados_api.yaml 2024-03-27T21:42:44.909 INFO:teuthology.orchestra.run.smithi060.stdout: 408289 4 -rw-r--r-- 1 ubuntu ubuntu 577 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/parallel/workload/ec-rados-default.yaml 2024-03-27T21:42:44.910 INFO:teuthology.orchestra.run.smithi060.stdout: 408291 4 -rw-r--r-- 1 ubuntu ubuntu 331 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/parallel/workload/rados_loadgenbig.yaml 2024-03-27T21:42:44.910 INFO:teuthology.orchestra.run.smithi060.stdout: 408284 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/reef-x/parallel/mon_election -> .qa/mon_election 2024-03-27T21:42:44.910 INFO:teuthology.orchestra.run.smithi060.stdout: 408213 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x 2024-03-27T21:42:44.910 INFO:teuthology.orchestra.run.smithi060.stdout: 408214 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/.qa -> ../.qa/ 2024-03-27T21:42:44.910 INFO:teuthology.orchestra.run.smithi060.stdout: 408248 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split 2024-03-27T21:42:44.910 INFO:teuthology.orchestra.run.smithi060.stdout: 408254 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/2-first-half-tasks 2024-03-27T21:42:44.910 INFO:teuthology.orchestra.run.smithi060.stdout: 408255 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/2-first-half-tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.910 INFO:teuthology.orchestra.run.smithi060.stdout: 408261 4 -rw-r--r-- 1 ubuntu ubuntu 452 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/2-first-half-tasks/snaps-few-objects.yaml 2024-03-27T21:42:44.911 INFO:teuthology.orchestra.run.smithi060.stdout: 408256 4 -rw-r--r-- 1 ubuntu ubuntu 418 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/2-first-half-tasks/radosbench.yaml 2024-03-27T21:42:44.911 INFO:teuthology.orchestra.run.smithi060.stdout: 408258 4 -rw-r--r-- 1 ubuntu ubuntu 268 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/2-first-half-tasks/rbd-import-export.yaml 2024-03-27T21:42:44.911 INFO:teuthology.orchestra.run.smithi060.stdout: 408259 4 -rw-r--r-- 1 ubuntu ubuntu 247 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/2-first-half-tasks/rbd_api.yaml 2024-03-27T21:42:44.911 INFO:teuthology.orchestra.run.smithi060.stdout: 408257 4 -rw-r--r-- 1 ubuntu ubuntu 207 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/2-first-half-tasks/rbd-cls.yaml 2024-03-27T21:42:44.911 INFO:teuthology.orchestra.run.smithi060.stdout: 408260 4 -rw-r--r-- 1 ubuntu ubuntu 391 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/2-first-half-tasks/readwrite.yaml 2024-03-27T21:42:44.911 INFO:teuthology.orchestra.run.smithi060.stdout: 408249 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/% 2024-03-27T21:42:44.911 INFO:teuthology.orchestra.run.smithi060.stdout: 408250 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/.qa -> ../.qa/ 2024-03-27T21:42:44.911 INFO:teuthology.orchestra.run.smithi060.stdout: 408252 4 -rw-r--r-- 1 ubuntu ubuntu 440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/0-roles.yaml 2024-03-27T21:42:44.911 INFO:teuthology.orchestra.run.smithi060.stdout: 408262 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/3-stress-tasks 2024-03-27T21:42:44.912 INFO:teuthology.orchestra.run.smithi060.stdout: 408263 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/3-stress-tasks/+ 2024-03-27T21:42:44.912 INFO:teuthology.orchestra.run.smithi060.stdout: 408264 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/3-stress-tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.912 INFO:teuthology.orchestra.run.smithi060.stdout: 408270 4 -rw-r--r-- 1 ubuntu ubuntu 448 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/3-stress-tasks/snaps-few-objects.yaml 2024-03-27T21:42:44.912 INFO:teuthology.orchestra.run.smithi060.stdout: 408265 4 -rw-r--r-- 1 ubuntu ubuntu 528 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/3-stress-tasks/radosbench.yaml 2024-03-27T21:42:44.912 INFO:teuthology.orchestra.run.smithi060.stdout: 408267 4 -rw-r--r-- 1 ubuntu ubuntu 264 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/3-stress-tasks/rbd-import-export.yaml 2024-03-27T21:42:44.912 INFO:teuthology.orchestra.run.smithi060.stdout: 408268 4 -rw-r--r-- 1 ubuntu ubuntu 243 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/3-stress-tasks/rbd_api.yaml 2024-03-27T21:42:44.912 INFO:teuthology.orchestra.run.smithi060.stdout: 408266 4 -rw-r--r-- 1 ubuntu ubuntu 203 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/3-stress-tasks/rbd-cls.yaml 2024-03-27T21:42:44.912 INFO:teuthology.orchestra.run.smithi060.stdout: 408269 4 -rw-r--r-- 1 ubuntu ubuntu 387 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/3-stress-tasks/readwrite.yaml 2024-03-27T21:42:44.913 INFO:teuthology.orchestra.run.smithi060.stdout: 408253 4 -rw-r--r-- 1 ubuntu ubuntu 3185 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/1-start.yaml 2024-03-27T21:42:44.913 INFO:teuthology.orchestra.run.smithi060.stdout: 408275 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/mon_election -> .qa/mon_election 2024-03-27T21:42:44.913 INFO:teuthology.orchestra.run.smithi060.stdout: 408251 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/0-distro -> .qa/distros/supported-container-hosts 2024-03-27T21:42:44.913 INFO:teuthology.orchestra.run.smithi060.stdout: 408271 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/4-second-half-tasks 2024-03-27T21:42:44.913 INFO:teuthology.orchestra.run.smithi060.stdout: 408272 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/4-second-half-tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.913 INFO:teuthology.orchestra.run.smithi060.stdout: 408273 4 -rw-r--r-- 1 ubuntu ubuntu 362 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/4-second-half-tasks/radosbench.yaml 2024-03-27T21:42:44.913 INFO:teuthology.orchestra.run.smithi060.stdout: 408274 4 -rw-r--r-- 1 ubuntu ubuntu 269 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/stress-split/4-second-half-tasks/rbd-import-export.yaml 2024-03-27T21:42:44.913 INFO:teuthology.orchestra.run.smithi060.stdout: 408215 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/filestore-remove-check 2024-03-27T21:42:44.914 INFO:teuthology.orchestra.run.smithi060.stdout: 408218 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/filestore-remove-check/0-cluster 2024-03-27T21:42:44.914 INFO:teuthology.orchestra.run.smithi060.stdout: 408219 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/filestore-remove-check/0-cluster/+ 2024-03-27T21:42:44.914 INFO:teuthology.orchestra.run.smithi060.stdout: 408220 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/filestore-remove-check/0-cluster/.qa -> ../.qa/ 2024-03-27T21:42:44.914 INFO:teuthology.orchestra.run.smithi060.stdout: 408221 4 -rw-r--r-- 1 ubuntu ubuntu 121 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/filestore-remove-check/0-cluster/openstack.yaml 2024-03-27T21:42:44.914 INFO:teuthology.orchestra.run.smithi060.stdout: 408222 4 -rw-r--r-- 1 ubuntu ubuntu 770 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/filestore-remove-check/0-cluster/start.yaml 2024-03-27T21:42:44.914 INFO:teuthology.orchestra.run.smithi060.stdout: 408216 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/filestore-remove-check/% 2024-03-27T21:42:44.914 INFO:teuthology.orchestra.run.smithi060.stdout: 408227 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/filestore-remove-check/objectstore 2024-03-27T21:42:44.914 INFO:teuthology.orchestra.run.smithi060.stdout: 408228 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/filestore-remove-check/objectstore/.qa -> ../.qa/ 2024-03-27T21:42:44.915 INFO:teuthology.orchestra.run.smithi060.stdout: 408229 4 -rw-r--r-- 1 ubuntu ubuntu 247 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/filestore-remove-check/objectstore/filestore-xfs.yaml 2024-03-27T21:42:44.915 INFO:teuthology.orchestra.run.smithi060.stdout: 408217 0 lrwxrwxrwx 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/filestore-remove-check/.qa -> ../.qa 2024-03-27T21:42:44.915 INFO:teuthology.orchestra.run.smithi060.stdout: 408226 4 -rw-r--r-- 1 ubuntu ubuntu 569 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/filestore-remove-check/2\ -\ upgrade.yaml 2024-03-27T21:42:44.915 INFO:teuthology.orchestra.run.smithi060.stdout: 408223 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/filestore-remove-check/1-ceph-install 2024-03-27T21:42:44.915 INFO:teuthology.orchestra.run.smithi060.stdout: 408224 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/filestore-remove-check/1-ceph-install/.qa -> ../.qa/ 2024-03-27T21:42:44.915 INFO:teuthology.orchestra.run.smithi060.stdout: 408225 4 -rw-r--r-- 1 ubuntu ubuntu 789 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/filestore-remove-check/1-ceph-install/quincy.yaml 2024-03-27T21:42:44.915 INFO:teuthology.orchestra.run.smithi060.stdout: 408230 4 -rw-r--r-- 1 ubuntu ubuntu 343 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/filestore-remove-check/ubuntu_20.04.yaml 2024-03-27T21:42:44.915 INFO:teuthology.orchestra.run.smithi060.stdout: 408231 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/parallel 2024-03-27T21:42:44.915 INFO:teuthology.orchestra.run.smithi060.stdout: 408238 4 -rw-r--r-- 1 ubuntu ubuntu 650 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/parallel/upgrade-sequence.yaml 2024-03-27T21:42:44.916 INFO:teuthology.orchestra.run.smithi060.stdout: 408234 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/parallel/0-random-distro$ -> .qa/distros/supported-container-hosts 2024-03-27T21:42:44.916 INFO:teuthology.orchestra.run.smithi060.stdout: 408232 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/parallel/% 2024-03-27T21:42:44.916 INFO:teuthology.orchestra.run.smithi060.stdout: 408233 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/parallel/.qa -> ../.qa/ 2024-03-27T21:42:44.916 INFO:teuthology.orchestra.run.smithi060.stdout: 408235 4 -rw-r--r-- 1 ubuntu ubuntu 460 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/parallel/0-start.yaml 2024-03-27T21:42:44.916 INFO:teuthology.orchestra.run.smithi060.stdout: 408236 4 -rw-r--r-- 1 ubuntu ubuntu 1202 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/parallel/1-tasks.yaml 2024-03-27T21:42:44.916 INFO:teuthology.orchestra.run.smithi060.stdout: 408239 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/parallel/workload 2024-03-27T21:42:44.916 INFO:teuthology.orchestra.run.smithi060.stdout: 408240 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/parallel/workload/+ 2024-03-27T21:42:44.916 INFO:teuthology.orchestra.run.smithi060.stdout: 408247 4 -rw-r--r-- 1 ubuntu ubuntu 417 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/parallel/workload/test_rbd_python.yaml 2024-03-27T21:42:44.917 INFO:teuthology.orchestra.run.smithi060.stdout: 408241 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/parallel/workload/.qa -> ../.qa/ 2024-03-27T21:42:44.917 INFO:teuthology.orchestra.run.smithi060.stdout: 408245 4 -rw-r--r-- 1 ubuntu ubuntu 362 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/parallel/workload/rbd_import_export.yaml 2024-03-27T21:42:44.917 INFO:teuthology.orchestra.run.smithi060.stdout: 408246 4 -rw-r--r-- 1 ubuntu ubuntu 324 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/parallel/workload/test_rbd_api.yaml 2024-03-27T21:42:44.917 INFO:teuthology.orchestra.run.smithi060.stdout: 408243 4 -rw-r--r-- 1 ubuntu ubuntu 262 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/parallel/workload/rados_api.yaml 2024-03-27T21:42:44.917 INFO:teuthology.orchestra.run.smithi060.stdout: 408242 4 -rw-r--r-- 1 ubuntu ubuntu 577 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/parallel/workload/ec-rados-default.yaml 2024-03-27T21:42:44.917 INFO:teuthology.orchestra.run.smithi060.stdout: 408244 4 -rw-r--r-- 1 ubuntu ubuntu 333 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/parallel/workload/rados_loadgenbig.yaml 2024-03-27T21:42:44.917 INFO:teuthology.orchestra.run.smithi060.stdout: 408237 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/upgrade/quincy-x/parallel/mon_election -> .qa/mon_election 2024-03-27T21:42:44.917 INFO:teuthology.orchestra.run.smithi060.stdout: 404602 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/big 2024-03-27T21:42:44.918 INFO:teuthology.orchestra.run.smithi060.stdout: 404603 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/big/.qa -> ../.qa/ 2024-03-27T21:42:44.918 INFO:teuthology.orchestra.run.smithi060.stdout: 404604 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/big/rados-thrash 2024-03-27T21:42:44.918 INFO:teuthology.orchestra.run.smithi060.stdout: 404605 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/big/rados-thrash/% 2024-03-27T21:42:44.918 INFO:teuthology.orchestra.run.smithi060.stdout: 404606 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/big/rados-thrash/.qa -> ../.qa/ 2024-03-27T21:42:44.918 INFO:teuthology.orchestra.run.smithi060.stdout: 404615 4 -rw-r--r-- 1 ubuntu ubuntu 155 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/big/rados-thrash/openstack.yaml 2024-03-27T21:42:44.918 INFO:teuthology.orchestra.run.smithi060.stdout: 404616 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/big/rados-thrash/workloads 2024-03-27T21:42:44.918 INFO:teuthology.orchestra.run.smithi060.stdout: 404617 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/big/rados-thrash/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.918 INFO:teuthology.orchestra.run.smithi060.stdout: 404619 4 -rw-r--r-- 1 ubuntu ubuntu 217 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/big/rados-thrash/workloads/snaps-few-objects.yaml 2024-03-27T21:42:44.919 INFO:teuthology.orchestra.run.smithi060.stdout: 404618 4 -rw-r--r-- 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/big/rados-thrash/workloads/radosbench.yaml 2024-03-27T21:42:44.919 INFO:teuthology.orchestra.run.smithi060.stdout: 404610 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/big/rados-thrash/clusters 2024-03-27T21:42:44.919 INFO:teuthology.orchestra.run.smithi060.stdout: 404611 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/big/rados-thrash/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.919 INFO:teuthology.orchestra.run.smithi060.stdout: 404613 4 -rw-r--r-- 1 ubuntu ubuntu 459 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/big/rados-thrash/clusters/medium.yaml 2024-03-27T21:42:44.919 INFO:teuthology.orchestra.run.smithi060.stdout: 404614 4 -rw-r--r-- 1 ubuntu ubuntu 112 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/big/rados-thrash/clusters/small.yaml 2024-03-27T21:42:44.919 INFO:teuthology.orchestra.run.smithi060.stdout: 404612 4 -rw-r--r-- 1 ubuntu ubuntu 1512 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/big/rados-thrash/clusters/big.yaml 2024-03-27T21:42:44.919 INFO:teuthology.orchestra.run.smithi060.stdout: 404607 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/big/rados-thrash/ceph 2024-03-27T21:42:44.919 INFO:teuthology.orchestra.run.smithi060.stdout: 404608 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/big/rados-thrash/ceph/.qa -> ../.qa/ 2024-03-27T21:42:44.920 INFO:teuthology.orchestra.run.smithi060.stdout: 404609 4 -rw-r--r-- 1 ubuntu ubuntu 96 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/big/rados-thrash/ceph/cephadm.yaml 2024-03-27T21:42:44.920 INFO:teuthology.orchestra.run.smithi060.stdout: 408336 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/windows 2024-03-27T21:42:44.920 INFO:teuthology.orchestra.run.smithi060.stdout: 408337 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/windows/.qa -> ../.qa/ 2024-03-27T21:42:44.920 INFO:teuthology.orchestra.run.smithi060.stdout: 408338 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/windows/basic 2024-03-27T21:42:44.920 INFO:teuthology.orchestra.run.smithi060.stdout: 408339 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/windows/basic/% 2024-03-27T21:42:44.920 INFO:teuthology.orchestra.run.smithi060.stdout: 408340 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/windows/basic/.qa -> ../.qa/ 2024-03-27T21:42:44.920 INFO:teuthology.orchestra.run.smithi060.stdout: 408350 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/windows/basic/ubuntu_latest.yaml -> .qa/distros/supported/ubuntu_latest.yaml 2024-03-27T21:42:44.920 INFO:teuthology.orchestra.run.smithi060.stdout: 408344 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/windows/basic/install 2024-03-27T21:42:44.920 INFO:teuthology.orchestra.run.smithi060.stdout: 408346 4 -rw-r--r-- 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/windows/basic/install/install.yaml 2024-03-27T21:42:44.921 INFO:teuthology.orchestra.run.smithi060.stdout: 408345 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/windows/basic/install/.qa -> ../.qa/ 2024-03-27T21:42:44.921 INFO:teuthology.orchestra.run.smithi060.stdout: 408341 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/windows/basic/clusters 2024-03-27T21:42:44.921 INFO:teuthology.orchestra.run.smithi060.stdout: 408343 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/windows/basic/clusters/fixed-1.yaml -> .qa/clusters/fixed-1.yaml 2024-03-27T21:42:44.921 INFO:teuthology.orchestra.run.smithi060.stdout: 408342 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/windows/basic/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.921 INFO:teuthology.orchestra.run.smithi060.stdout: 408347 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/windows/basic/tasks 2024-03-27T21:42:44.921 INFO:teuthology.orchestra.run.smithi060.stdout: 408348 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/windows/basic/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.921 INFO:teuthology.orchestra.run.smithi060.stdout: 408349 4 -rw-r--r-- 1 ubuntu ubuntu 117 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/windows/basic/tasks/windows_tests.yaml 2024-03-27T21:42:44.921 INFO:teuthology.orchestra.run.smithi060.stdout: 404740 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados 2024-03-27T21:42:44.922 INFO:teuthology.orchestra.run.smithi060.stdout: 404741 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/.qa -> ../.qa/ 2024-03-27T21:42:44.922 INFO:teuthology.orchestra.run.smithi060.stdout: 404759 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf 2024-03-27T21:42:44.922 INFO:teuthology.orchestra.run.smithi060.stdout: 404760 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/% 2024-03-27T21:42:44.922 INFO:teuthology.orchestra.run.smithi060.stdout: 404767 0 lrwxrwxrwx 1 ubuntu ubuntu 41 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/crimson-supported-all-distro -> .qa/distros/crimson-supported-all-distro/ 2024-03-27T21:42:44.922 INFO:teuthology.orchestra.run.smithi060.stdout: 404773 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/objectstore 2024-03-27T21:42:44.922 INFO:teuthology.orchestra.run.smithi060.stdout: 404774 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/objectstore/.qa -> ../.qa/ 2024-03-27T21:42:44.922 INFO:teuthology.orchestra.run.smithi060.stdout: 404775 4 -rw-r--r-- 1 ubuntu ubuntu 711 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/objectstore/bluestore.yaml 2024-03-27T21:42:44.922 INFO:teuthology.orchestra.run.smithi060.stdout: 404761 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/.qa -> ../.qa/ 2024-03-27T21:42:44.923 INFO:teuthology.orchestra.run.smithi060.stdout: 404769 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/deploy 2024-03-27T21:42:44.923 INFO:teuthology.orchestra.run.smithi060.stdout: 404771 4 -rw-r--r-- 1 ubuntu ubuntu 170 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/deploy/ceph.yaml 2024-03-27T21:42:44.923 INFO:teuthology.orchestra.run.smithi060.stdout: 404770 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/deploy/.qa -> ../.qa/ 2024-03-27T21:42:44.923 INFO:teuthology.orchestra.run.smithi060.stdout: 404772 4 -rw-r--r-- 1 ubuntu ubuntu 315 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/deploy/cephadm.yaml.disabled 2024-03-27T21:42:44.923 INFO:teuthology.orchestra.run.smithi060.stdout: 404779 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/workloads 2024-03-27T21:42:44.923 INFO:teuthology.orchestra.run.smithi060.stdout: 404787 4 -rw-r--r-- 1 ubuntu ubuntu 692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/workloads/radosbench_4K_seq_read.yaml 2024-03-27T21:42:44.923 INFO:teuthology.orchestra.run.smithi060.stdout: 404780 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.923 INFO:teuthology.orchestra.run.smithi060.stdout: 404788 4 -rw-r--r-- 1 ubuntu ubuntu 720 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/workloads/radosbench_4M_rand_read.yaml 2024-03-27T21:42:44.923 INFO:teuthology.orchestra.run.smithi060.stdout: 404785 4 -rw-r--r-- 1 ubuntu ubuntu 702 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/workloads/fio_4M_rand_write.yaml 2024-03-27T21:42:44.924 INFO:teuthology.orchestra.run.smithi060.stdout: 404783 4 -rw-r--r-- 1 ubuntu ubuntu 701 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/workloads/fio_4M_rand_read.yaml 2024-03-27T21:42:44.924 INFO:teuthology.orchestra.run.smithi060.stdout: 404792 4 -rw-r--r-- 1 ubuntu ubuntu 699 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/workloads/sample_fio.yaml 2024-03-27T21:42:44.924 INFO:teuthology.orchestra.run.smithi060.stdout: 404791 4 -rw-r--r-- 1 ubuntu ubuntu 116 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/workloads/radosbench_omap_write.yaml 2024-03-27T21:42:44.924 INFO:teuthology.orchestra.run.smithi060.stdout: 404786 4 -rw-r--r-- 1 ubuntu ubuntu 717 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/workloads/radosbench_4K_rand_read.yaml 2024-03-27T21:42:44.924 INFO:teuthology.orchestra.run.smithi060.stdout: 404781 4 -rw-r--r-- 1 ubuntu ubuntu 698 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/workloads/fio_4K_rand_read.yaml 2024-03-27T21:42:44.924 INFO:teuthology.orchestra.run.smithi060.stdout: 404784 4 -rw-r--r-- 1 ubuntu ubuntu 699 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/workloads/fio_4M_rand_rw.yaml 2024-03-27T21:42:44.924 INFO:teuthology.orchestra.run.smithi060.stdout: 404793 4 -rw-r--r-- 1 ubuntu ubuntu 691 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/workloads/sample_radosbench.yaml 2024-03-27T21:42:44.924 INFO:teuthology.orchestra.run.smithi060.stdout: 404790 4 -rw-r--r-- 1 ubuntu ubuntu 694 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/workloads/radosbench_4M_write.yaml 2024-03-27T21:42:44.924 INFO:teuthology.orchestra.run.smithi060.stdout: 404789 4 -rw-r--r-- 1 ubuntu ubuntu 695 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/workloads/radosbench_4M_seq_read.yaml 2024-03-27T21:42:44.925 INFO:teuthology.orchestra.run.smithi060.stdout: 404782 4 -rw-r--r-- 1 ubuntu ubuntu 696 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/workloads/fio_4K_rand_rw.yaml 2024-03-27T21:42:44.925 INFO:teuthology.orchestra.run.smithi060.stdout: 404762 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/clusters 2024-03-27T21:42:44.925 INFO:teuthology.orchestra.run.smithi060.stdout: 404765 4 -rw-r--r-- 1 ubuntu ubuntu 675 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/clusters/fixed-2.yaml 2024-03-27T21:42:44.925 INFO:teuthology.orchestra.run.smithi060.stdout: 404766 4 -rw-r--r-- 1 ubuntu ubuntu 87 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/clusters/openstack.yaml.disabled 2024-03-27T21:42:44.925 INFO:teuthology.orchestra.run.smithi060.stdout: 404763 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/clusters/+ 2024-03-27T21:42:44.925 INFO:teuthology.orchestra.run.smithi060.stdout: 404764 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.925 INFO:teuthology.orchestra.run.smithi060.stdout: 404768 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/crimson_qa_overrides.yaml -> .qa/config/crimson_qa_overrides.yaml 2024-03-27T21:42:44.925 INFO:teuthology.orchestra.run.smithi060.stdout: 404776 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/settings 2024-03-27T21:42:44.926 INFO:teuthology.orchestra.run.smithi060.stdout: 404777 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/settings/.qa -> ../.qa/ 2024-03-27T21:42:44.926 INFO:teuthology.orchestra.run.smithi060.stdout: 404778 4 -rw-r--r-- 1 ubuntu ubuntu 2072 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/perf/settings/optimized.yaml 2024-03-27T21:42:44.926 INFO:teuthology.orchestra.run.smithi060.stdout: 404814 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/singleton 2024-03-27T21:42:44.926 INFO:teuthology.orchestra.run.smithi060.stdout: 404817 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/singleton/all 2024-03-27T21:42:44.926 INFO:teuthology.orchestra.run.smithi060.stdout: 404818 0 lrwxrwxrwx 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/singleton/all/.qa -> ../.qa 2024-03-27T21:42:44.926 INFO:teuthology.orchestra.run.smithi060.stdout: 404819 4 -rw-r--r-- 1 ubuntu ubuntu 721 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/singleton/all/osd-backfill.yaml 2024-03-27T21:42:44.926 INFO:teuthology.orchestra.run.smithi060.stdout: 404815 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/singleton/% 2024-03-27T21:42:44.926 INFO:teuthology.orchestra.run.smithi060.stdout: 404820 0 lrwxrwxrwx 1 ubuntu ubuntu 41 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/singleton/crimson-supported-all-distro -> .qa/distros/crimson-supported-all-distro/ 2024-03-27T21:42:44.927 INFO:teuthology.orchestra.run.smithi060.stdout: 404822 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/singleton/objectstore -> ../thrash/objectstore 2024-03-27T21:42:44.927 INFO:teuthology.orchestra.run.smithi060.stdout: 404816 0 lrwxrwxrwx 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/singleton/.qa -> ../.qa 2024-03-27T21:42:44.927 INFO:teuthology.orchestra.run.smithi060.stdout: 404823 0 lrwxrwxrwx 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/singleton/rados.yaml -> ./.qa/suites/rados/singleton/rados.yaml 2024-03-27T21:42:44.927 INFO:teuthology.orchestra.run.smithi060.stdout: 404821 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/singleton/crimson_qa_overrides.yaml -> .qa/config/crimson_qa_overrides.yaml 2024-03-27T21:42:44.927 INFO:teuthology.orchestra.run.smithi060.stdout: 404824 4 drwxr-xr-x 10 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash 2024-03-27T21:42:44.927 INFO:teuthology.orchestra.run.smithi060.stdout: 404825 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/% 2024-03-27T21:42:44.927 INFO:teuthology.orchestra.run.smithi060.stdout: 404848 0 lrwxrwxrwx 1 ubuntu ubuntu 41 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/crimson-supported-all-distro -> .qa/distros/crimson-supported-all-distro/ 2024-03-27T21:42:44.927 INFO:teuthology.orchestra.run.smithi060.stdout: 404854 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/objectstore 2024-03-27T21:42:44.927 INFO:teuthology.orchestra.run.smithi060.stdout: 404855 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/objectstore/.qa -> ../.qa/ 2024-03-27T21:42:44.928 INFO:teuthology.orchestra.run.smithi060.stdout: 404856 4 -rw-r--r-- 1 ubuntu ubuntu 711 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/objectstore/bluestore.yaml 2024-03-27T21:42:44.928 INFO:teuthology.orchestra.run.smithi060.stdout: 404826 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/.qa -> ../.qa/ 2024-03-27T21:42:44.928 INFO:teuthology.orchestra.run.smithi060.stdout: 404831 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/1-pg-log-overrides 2024-03-27T21:42:44.928 INFO:teuthology.orchestra.run.smithi060.stdout: 404832 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/1-pg-log-overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.928 INFO:teuthology.orchestra.run.smithi060.stdout: 404833 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/1-pg-log-overrides/normal_pg_log.yaml 2024-03-27T21:42:44.928 INFO:teuthology.orchestra.run.smithi060.stdout: 404834 0 lrwxrwxrwx 1 ubuntu ubuntu 31 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/1-pg-log-overrides/short_pg_log.yaml.disabled -> .qa/overrides/short_pg_log.yaml 2024-03-27T21:42:44.928 INFO:teuthology.orchestra.run.smithi060.stdout: 404860 0 lrwxrwxrwx 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/thrashosds-health.yaml -> .qa/tasks/thrashosds-health.yaml 2024-03-27T21:42:44.928 INFO:teuthology.orchestra.run.smithi060.stdout: 404850 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/deploy 2024-03-27T21:42:44.928 INFO:teuthology.orchestra.run.smithi060.stdout: 404852 4 -rw-r--r-- 1 ubuntu ubuntu 144 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/deploy/ceph.yaml 2024-03-27T21:42:44.929 INFO:teuthology.orchestra.run.smithi060.stdout: 404851 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/deploy/.qa -> ../.qa/ 2024-03-27T21:42:44.929 INFO:teuthology.orchestra.run.smithi060.stdout: 404853 4 -rw-r--r-- 1 ubuntu ubuntu 315 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/deploy/cephadm.yaml.disabled 2024-03-27T21:42:44.929 INFO:teuthology.orchestra.run.smithi060.stdout: 404861 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/workloads 2024-03-27T21:42:44.929 INFO:teuthology.orchestra.run.smithi060.stdout: 404871 4 -rw-r--r-- 1 ubuntu ubuntu 278 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/workloads/snaps-few-objects-localized.yaml 2024-03-27T21:42:44.929 INFO:teuthology.orchestra.run.smithi060.stdout: 404870 4 -rw-r--r-- 1 ubuntu ubuntu 277 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/workloads/snaps-few-objects-balanced.yaml 2024-03-27T21:42:44.929 INFO:teuthology.orchestra.run.smithi060.stdout: 404862 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.929 INFO:teuthology.orchestra.run.smithi060.stdout: 404864 4 -rw-r--r-- 1 ubuntu ubuntu 349 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/workloads/pool-snaps-few-objects.yaml 2024-03-27T21:42:44.929 INFO:teuthology.orchestra.run.smithi060.stdout: 404872 4 -rw-r--r-- 1 ubuntu ubuntu 253 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/workloads/snaps-few-objects.yaml 2024-03-27T21:42:44.930 INFO:teuthology.orchestra.run.smithi060.stdout: 404866 4 -rw-r--r-- 1 ubuntu ubuntu 427 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/workloads/radosbench.yaml 2024-03-27T21:42:44.930 INFO:teuthology.orchestra.run.smithi060.stdout: 404873 4 -rw-r--r-- 1 ubuntu ubuntu 137 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/workloads/write_fadvise_dontneed.yaml 2024-03-27T21:42:44.930 INFO:teuthology.orchestra.run.smithi060.stdout: 404865 4 -rw-r--r-- 1 ubuntu ubuntu 918 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/workloads/radosbench-high-concurrency.yaml 2024-03-27T21:42:44.930 INFO:teuthology.orchestra.run.smithi060.stdout: 404868 4 -rw-r--r-- 1 ubuntu ubuntu 421 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/workloads/small-objects-localized.yaml 2024-03-27T21:42:44.930 INFO:teuthology.orchestra.run.smithi060.stdout: 404867 4 -rw-r--r-- 1 ubuntu ubuntu 420 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/workloads/small-objects-balanced.yaml 2024-03-27T21:42:44.930 INFO:teuthology.orchestra.run.smithi060.stdout: 404869 4 -rw-r--r-- 1 ubuntu ubuntu 396 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/workloads/small-objects.yaml 2024-03-27T21:42:44.930 INFO:teuthology.orchestra.run.smithi060.stdout: 404863 4 -rw-r--r-- 1 ubuntu ubuntu 324 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/workloads/admin_socket_objecter_requests.yaml 2024-03-27T21:42:44.931 INFO:teuthology.orchestra.run.smithi060.stdout: 404843 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/clusters 2024-03-27T21:42:44.931 INFO:teuthology.orchestra.run.smithi060.stdout: 404846 4 -rw-r--r-- 1 ubuntu ubuntu 653 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/clusters/fixed-2.yaml 2024-03-27T21:42:44.931 INFO:teuthology.orchestra.run.smithi060.stdout: 404847 4 -rw-r--r-- 1 ubuntu ubuntu 87 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/clusters/openstack.yaml.disabled 2024-03-27T21:42:44.931 INFO:teuthology.orchestra.run.smithi060.stdout: 404844 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/clusters/+ 2024-03-27T21:42:44.931 INFO:teuthology.orchestra.run.smithi060.stdout: 404845 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.931 INFO:teuthology.orchestra.run.smithi060.stdout: 404849 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/crimson_qa_overrides.yaml -> .qa/config/crimson_qa_overrides.yaml 2024-03-27T21:42:44.931 INFO:teuthology.orchestra.run.smithi060.stdout: 404827 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/0-size-min-size-overrides 2024-03-27T21:42:44.931 INFO:teuthology.orchestra.run.smithi060.stdout: 404828 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/0-size-min-size-overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.932 INFO:teuthology.orchestra.run.smithi060.stdout: 404829 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/0-size-min-size-overrides/2-size-2-min-size.yaml.disabled -> .qa/overrides/2-size-2-min-size.yaml 2024-03-27T21:42:44.932 INFO:teuthology.orchestra.run.smithi060.stdout: 404830 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/0-size-min-size-overrides/3-size-2-min-size.yaml -> .qa/overrides/3-size-2-min-size.yaml 2024-03-27T21:42:44.932 INFO:teuthology.orchestra.run.smithi060.stdout: 404857 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/thrashers 2024-03-27T21:42:44.932 INFO:teuthology.orchestra.run.smithi060.stdout: 404858 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/thrashers/.qa -> ../.qa/ 2024-03-27T21:42:44.932 INFO:teuthology.orchestra.run.smithi060.stdout: 404859 4 -rw-r--r-- 1 ubuntu ubuntu 950 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/thrashers/default.yaml 2024-03-27T21:42:44.932 INFO:teuthology.orchestra.run.smithi060.stdout: 404835 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/2-recovery-overrides 2024-03-27T21:42:44.932 INFO:teuthology.orchestra.run.smithi060.stdout: 404842 4 -rw-r--r-- 1 ubuntu ubuntu 99 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/2-recovery-overrides/more-partial-recovery.yaml.disabled 2024-03-27T21:42:44.932 INFO:teuthology.orchestra.run.smithi060.stdout: 404837 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/2-recovery-overrides/.qa -> ../.qa/ 2024-03-27T21:42:44.932 INFO:teuthology.orchestra.run.smithi060.stdout: 404841 4 -rw-r--r-- 1 ubuntu ubuntu 82 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/2-recovery-overrides/more-async-recovery.yaml.disabled 2024-03-27T21:42:44.933 INFO:teuthology.orchestra.run.smithi060.stdout: 404836 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/2-recovery-overrides/$ 2024-03-27T21:42:44.933 INFO:teuthology.orchestra.run.smithi060.stdout: 404839 0 lrwxrwxrwx 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/2-recovery-overrides/more-active-recovery.yaml.disabled -> .qa/overrides/more-active-recovery.yaml 2024-03-27T21:42:44.933 INFO:teuthology.orchestra.run.smithi060.stdout: 404838 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/2-recovery-overrides/default.yaml 2024-03-27T21:42:44.933 INFO:teuthology.orchestra.run.smithi060.stdout: 404840 4 -rw-r--r-- 1 ubuntu ubuntu 138 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/thrash/2-recovery-overrides/more-async-partial-recovery.yaml.disabled 2024-03-27T21:42:44.933 INFO:teuthology.orchestra.run.smithi060.stdout: 404794 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/rbd 2024-03-27T21:42:44.933 INFO:teuthology.orchestra.run.smithi060.stdout: 404795 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/rbd/% 2024-03-27T21:42:44.933 INFO:teuthology.orchestra.run.smithi060.stdout: 404800 0 lrwxrwxrwx 1 ubuntu ubuntu 41 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/rbd/crimson-supported-all-distro -> .qa/distros/crimson-supported-all-distro/ 2024-03-27T21:42:44.933 INFO:teuthology.orchestra.run.smithi060.stdout: 404796 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/rbd/.qa -> ../.qa/ 2024-03-27T21:42:44.934 INFO:teuthology.orchestra.run.smithi060.stdout: 404802 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/rbd/deploy 2024-03-27T21:42:44.934 INFO:teuthology.orchestra.run.smithi060.stdout: 404804 4 -rw-r--r-- 1 ubuntu ubuntu 383 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/rbd/deploy/ceph.yaml 2024-03-27T21:42:44.934 INFO:teuthology.orchestra.run.smithi060.stdout: 404803 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/rbd/deploy/.qa -> ../.qa/ 2024-03-27T21:42:44.934 INFO:teuthology.orchestra.run.smithi060.stdout: 404805 4 -rw-r--r-- 1 ubuntu ubuntu 315 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/rbd/deploy/cephadm.yaml.disabled 2024-03-27T21:42:44.934 INFO:teuthology.orchestra.run.smithi060.stdout: 404797 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/rbd/clusters 2024-03-27T21:42:44.934 INFO:teuthology.orchestra.run.smithi060.stdout: 404799 4 -rw-r--r-- 1 ubuntu ubuntu 719 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/rbd/clusters/fixed-1.yaml 2024-03-27T21:42:44.934 INFO:teuthology.orchestra.run.smithi060.stdout: 404798 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/rbd/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.934 INFO:teuthology.orchestra.run.smithi060.stdout: 404801 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/rbd/crimson_qa_overrides.yaml -> .qa/config/crimson_qa_overrides.yaml 2024-03-27T21:42:44.935 INFO:teuthology.orchestra.run.smithi060.stdout: 404806 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/rbd/tasks 2024-03-27T21:42:44.935 INFO:teuthology.orchestra.run.smithi060.stdout: 404811 4 -rw-r--r-- 1 ubuntu ubuntu 81 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/rbd/tasks/rbd_lock_and_fence.yaml 2024-03-27T21:42:44.935 INFO:teuthology.orchestra.run.smithi060.stdout: 404807 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/rbd/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.935 INFO:teuthology.orchestra.run.smithi060.stdout: 404810 4 -rw-r--r-- 1 ubuntu ubuntu 143 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/rbd/tasks/rbd_cls_tests.yaml 2024-03-27T21:42:44.935 INFO:teuthology.orchestra.run.smithi060.stdout: 404809 4 -rw-r--r-- 1 ubuntu ubuntu 277 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/rbd/tasks/rbd_api_tests_old_format.yaml 2024-03-27T21:42:44.935 INFO:teuthology.orchestra.run.smithi060.stdout: 404812 4 -rw-r--r-- 1 ubuntu ubuntu 297 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/rbd/tasks/rbd_python_api_tests.yaml 2024-03-27T21:42:44.935 INFO:teuthology.orchestra.run.smithi060.stdout: 404813 4 -rw-r--r-- 1 ubuntu ubuntu 263 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/rbd/tasks/rbd_python_api_tests_old_format.yaml 2024-03-27T21:42:44.935 INFO:teuthology.orchestra.run.smithi060.stdout: 404808 4 -rw-r--r-- 1 ubuntu ubuntu 404 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/rbd/tasks/rbd_api_tests.yaml 2024-03-27T21:42:44.935 INFO:teuthology.orchestra.run.smithi060.stdout: 404742 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/basic 2024-03-27T21:42:44.936 INFO:teuthology.orchestra.run.smithi060.stdout: 404743 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/basic/% 2024-03-27T21:42:44.936 INFO:teuthology.orchestra.run.smithi060.stdout: 404748 0 lrwxrwxrwx 1 ubuntu ubuntu 41 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/basic/crimson-supported-all-distro -> .qa/distros/crimson-supported-all-distro/ 2024-03-27T21:42:44.936 INFO:teuthology.orchestra.run.smithi060.stdout: 404744 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/basic/.qa -> ../.qa/ 2024-03-27T21:42:44.936 INFO:teuthology.orchestra.run.smithi060.stdout: 404750 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/basic/deploy 2024-03-27T21:42:44.936 INFO:teuthology.orchestra.run.smithi060.stdout: 404752 4 -rw-r--r-- 1 ubuntu ubuntu 383 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/basic/deploy/ceph.yaml 2024-03-27T21:42:44.936 INFO:teuthology.orchestra.run.smithi060.stdout: 404751 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/basic/deploy/.qa -> ../.qa/ 2024-03-27T21:42:44.936 INFO:teuthology.orchestra.run.smithi060.stdout: 404753 4 -rw-r--r-- 1 ubuntu ubuntu 315 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/basic/deploy/cephadm.yaml.disabled 2024-03-27T21:42:44.936 INFO:teuthology.orchestra.run.smithi060.stdout: 404745 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/basic/clusters 2024-03-27T21:42:44.936 INFO:teuthology.orchestra.run.smithi060.stdout: 404747 4 -rw-r--r-- 1 ubuntu ubuntu 653 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/basic/clusters/fixed-2.yaml 2024-03-27T21:42:44.937 INFO:teuthology.orchestra.run.smithi060.stdout: 404746 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/basic/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.937 INFO:teuthology.orchestra.run.smithi060.stdout: 404749 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/basic/crimson_qa_overrides.yaml -> .qa/config/crimson_qa_overrides.yaml 2024-03-27T21:42:44.937 INFO:teuthology.orchestra.run.smithi060.stdout: 404754 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/basic/tasks 2024-03-27T21:42:44.937 INFO:teuthology.orchestra.run.smithi060.stdout: 404755 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/basic/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.937 INFO:teuthology.orchestra.run.smithi060.stdout: 404756 4 -rw-r--r-- 1 ubuntu ubuntu 635 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/basic/tasks/rados_api_tests.yaml 2024-03-27T21:42:44.937 INFO:teuthology.orchestra.run.smithi060.stdout: 404758 4 -rw-r--r-- 1 ubuntu ubuntu 318 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/basic/tasks/readwrite.yaml 2024-03-27T21:42:44.937 INFO:teuthology.orchestra.run.smithi060.stdout: 404757 4 -rw-r--r-- 1 ubuntu ubuntu 402 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados/basic/tasks/rados_python.yaml 2024-03-27T21:42:44.937 INFO:teuthology.orchestra.run.smithi060.stdout: 405888 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/hadoop 2024-03-27T21:42:44.938 INFO:teuthology.orchestra.run.smithi060.stdout: 405889 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/hadoop/.qa -> ../.qa/ 2024-03-27T21:42:44.938 INFO:teuthology.orchestra.run.smithi060.stdout: 405890 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/hadoop/basic 2024-03-27T21:42:44.938 INFO:teuthology.orchestra.run.smithi060.stdout: 405891 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/hadoop/basic/% 2024-03-27T21:42:44.938 INFO:teuthology.orchestra.run.smithi060.stdout: 405892 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/hadoop/basic/.qa -> ../.qa/ 2024-03-27T21:42:44.938 INFO:teuthology.orchestra.run.smithi060.stdout: 405893 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/hadoop/basic/clusters 2024-03-27T21:42:44.938 INFO:teuthology.orchestra.run.smithi060.stdout: 405895 4 -rw-r--r-- 1 ubuntu ubuntu 289 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/hadoop/basic/clusters/fixed-3.yaml 2024-03-27T21:42:44.938 INFO:teuthology.orchestra.run.smithi060.stdout: 405894 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/hadoop/basic/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.939 INFO:teuthology.orchestra.run.smithi060.stdout: 405896 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/hadoop/basic/distros 2024-03-27T21:42:44.939 INFO:teuthology.orchestra.run.smithi060.stdout: 405897 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/hadoop/basic/distros/.qa -> ../.qa/ 2024-03-27T21:42:44.939 INFO:teuthology.orchestra.run.smithi060.stdout: 405898 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/hadoop/basic/distros/ubuntu_latest.yaml -> .qa/distros/supported/ubuntu_latest.yaml 2024-03-27T21:42:44.939 INFO:teuthology.orchestra.run.smithi060.stdout: 405899 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/hadoop/basic/tasks 2024-03-27T21:42:44.939 INFO:teuthology.orchestra.run.smithi060.stdout: 405900 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/hadoop/basic/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.939 INFO:teuthology.orchestra.run.smithi060.stdout: 405903 4 -rw-r--r-- 1 ubuntu ubuntu 112 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/hadoop/basic/tasks/wordcount.yaml 2024-03-27T21:42:44.939 INFO:teuthology.orchestra.run.smithi060.stdout: 405902 4 -rw-r--r-- 1 ubuntu ubuntu 150 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/hadoop/basic/tasks/terasort.yaml 2024-03-27T21:42:44.939 INFO:teuthology.orchestra.run.smithi060.stdout: 405901 4 -rw-r--r-- 1 ubuntu ubuntu 106 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/hadoop/basic/tasks/repl.yaml 2024-03-27T21:42:44.939 INFO:teuthology.orchestra.run.smithi060.stdout: 404620 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/buildpackages 2024-03-27T21:42:44.940 INFO:teuthology.orchestra.run.smithi060.stdout: 404629 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/buildpackages/tests 2024-03-27T21:42:44.940 INFO:teuthology.orchestra.run.smithi060.stdout: 404630 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/buildpackages/tests/% 2024-03-27T21:42:44.940 INFO:teuthology.orchestra.run.smithi060.stdout: 404631 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/buildpackages/tests/.qa -> ../.qa/ 2024-03-27T21:42:44.940 INFO:teuthology.orchestra.run.smithi060.stdout: 404632 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/buildpackages/tests/distros -> .qa/distros/all 2024-03-27T21:42:44.940 INFO:teuthology.orchestra.run.smithi060.stdout: 404633 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/buildpackages/tests/tasks 2024-03-27T21:42:44.940 INFO:teuthology.orchestra.run.smithi060.stdout: 404634 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/buildpackages/tests/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.940 INFO:teuthology.orchestra.run.smithi060.stdout: 404635 4 -rw-r--r-- 1 ubuntu ubuntu 426 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/buildpackages/tests/tasks/release.yaml 2024-03-27T21:42:44.940 INFO:teuthology.orchestra.run.smithi060.stdout: 404621 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/buildpackages/.qa -> ../.qa/ 2024-03-27T21:42:44.941 INFO:teuthology.orchestra.run.smithi060.stdout: 404622 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/buildpackages/any 2024-03-27T21:42:44.941 INFO:teuthology.orchestra.run.smithi060.stdout: 404623 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/buildpackages/any/% 2024-03-27T21:42:44.941 INFO:teuthology.orchestra.run.smithi060.stdout: 404624 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/buildpackages/any/.qa -> ../.qa/ 2024-03-27T21:42:44.941 INFO:teuthology.orchestra.run.smithi060.stdout: 404625 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/buildpackages/any/distros -> .qa/distros/all 2024-03-27T21:42:44.941 INFO:teuthology.orchestra.run.smithi060.stdout: 404626 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/buildpackages/any/tasks 2024-03-27T21:42:44.941 INFO:teuthology.orchestra.run.smithi060.stdout: 404627 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/buildpackages/any/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.941 INFO:teuthology.orchestra.run.smithi060.stdout: 404628 4 -rw-r--r-- 1 ubuntu ubuntu 195 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/buildpackages/any/tasks/release.yaml 2024-03-27T21:42:44.941 INFO:teuthology.orchestra.run.smithi060.stdout: 408183 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt 2024-03-27T21:42:44.941 INFO:teuthology.orchestra.run.smithi060.stdout: 408184 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/.qa -> ../.qa/ 2024-03-27T21:42:44.942 INFO:teuthology.orchestra.run.smithi060.stdout: 408185 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic 2024-03-27T21:42:44.942 INFO:teuthology.orchestra.run.smithi060.stdout: 408186 4 -rw-r--r-- 1 ubuntu ubuntu 1 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/% 2024-03-27T21:42:44.942 INFO:teuthology.orchestra.run.smithi060.stdout: 408191 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/msgr-failures 2024-03-27T21:42:44.942 INFO:teuthology.orchestra.run.smithi060.stdout: 408192 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.942 INFO:teuthology.orchestra.run.smithi060.stdout: 408193 4 -rw-r--r-- 1 ubuntu ubuntu 177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/msgr-failures/few.yaml 2024-03-27T21:42:44.942 INFO:teuthology.orchestra.run.smithi060.stdout: 408194 4 -rw-r--r-- 1 ubuntu ubuntu 176 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/msgr-failures/many.yaml 2024-03-27T21:42:44.942 INFO:teuthology.orchestra.run.smithi060.stdout: 408187 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/.qa -> ../.qa/ 2024-03-27T21:42:44.942 INFO:teuthology.orchestra.run.smithi060.stdout: 408188 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/clusters 2024-03-27T21:42:44.943 INFO:teuthology.orchestra.run.smithi060.stdout: 408190 4 -rw-r--r-- 1 ubuntu ubuntu 103 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/clusters/fixed-3.yaml 2024-03-27T21:42:44.943 INFO:teuthology.orchestra.run.smithi060.stdout: 408189 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.943 INFO:teuthology.orchestra.run.smithi060.stdout: 408195 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/tasks 2024-03-27T21:42:44.943 INFO:teuthology.orchestra.run.smithi060.stdout: 408198 4 -rw-r--r-- 1 ubuntu ubuntu 111 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/tasks/bonnie.yaml 2024-03-27T21:42:44.943 INFO:teuthology.orchestra.run.smithi060.stdout: 408196 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.943 INFO:teuthology.orchestra.run.smithi060.stdout: 408202 4 -rw-r--r-- 1 ubuntu ubuntu 108 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/tasks/fio.yaml 2024-03-27T21:42:44.943 INFO:teuthology.orchestra.run.smithi060.stdout: 408199 4 -rw-r--r-- 1 ubuntu ubuntu 117 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/tasks/dbench-short.yaml 2024-03-27T21:42:44.943 INFO:teuthology.orchestra.run.smithi060.stdout: 408206 4 -rw-r--r-- 1 ubuntu ubuntu 110 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/tasks/iogen.yaml 2024-03-27T21:42:44.943 INFO:teuthology.orchestra.run.smithi060.stdout: 408205 4 -rw-r--r-- 1 ubuntu ubuntu 117 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/tasks/fsync-tester.yaml 2024-03-27T21:42:44.944 INFO:teuthology.orchestra.run.smithi060.stdout: 408208 4 -rw-r--r-- 1 ubuntu ubuntu 111 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/tasks/iozone.yaml 2024-03-27T21:42:44.944 INFO:teuthology.orchestra.run.smithi060.stdout: 408197 4 -rw-r--r-- 1 ubuntu ubuntu 114 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/tasks/blogbench.yaml 2024-03-27T21:42:44.944 INFO:teuthology.orchestra.run.smithi060.stdout: 408209 4 -rw-r--r-- 1 ubuntu ubuntu 108 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/tasks/pjd.yaml 2024-03-27T21:42:44.944 INFO:teuthology.orchestra.run.smithi060.stdout: 408207 4 -rw-r--r-- 1 ubuntu ubuntu 116 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/tasks/iozone-sync.yaml 2024-03-27T21:42:44.944 INFO:teuthology.orchestra.run.smithi060.stdout: 408203 4 -rw-r--r-- 1 ubuntu ubuntu 113 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/tasks/fsstress.yaml 2024-03-27T21:42:44.944 INFO:teuthology.orchestra.run.smithi060.stdout: 408200 4 -rw-r--r-- 1 ubuntu ubuntu 111 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/tasks/dbench.yaml 2024-03-27T21:42:44.944 INFO:teuthology.orchestra.run.smithi060.stdout: 408201 4 -rw-r--r-- 1 ubuntu ubuntu 109 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/tasks/ffsb.yaml 2024-03-27T21:42:44.944 INFO:teuthology.orchestra.run.smithi060.stdout: 408204 4 -rw-r--r-- 1 ubuntu ubuntu 315 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/tgt/basic/tasks/fsx.yaml 2024-03-27T21:42:44.945 INFO:teuthology.orchestra.run.smithi060.stdout: 404874 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/dummy 2024-03-27T21:42:44.945 INFO:teuthology.orchestra.run.smithi060.stdout: 404877 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/dummy/all 2024-03-27T21:42:44.945 INFO:teuthology.orchestra.run.smithi060.stdout: 404878 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/dummy/all/.qa -> ../.qa/ 2024-03-27T21:42:44.945 INFO:teuthology.orchestra.run.smithi060.stdout: 404879 4 -rw-r--r-- 1 ubuntu ubuntu 79 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/dummy/all/nop.yaml 2024-03-27T21:42:44.945 INFO:teuthology.orchestra.run.smithi060.stdout: 404875 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/dummy/% 2024-03-27T21:42:44.945 INFO:teuthology.orchestra.run.smithi060.stdout: 404876 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/dummy/.qa -> ../.qa/ 2024-03-27T21:42:44.945 INFO:teuthology.orchestra.run.smithi060.stdout: 408091 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress 2024-03-27T21:42:44.945 INFO:teuthology.orchestra.run.smithi060.stdout: 408093 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/bench 2024-03-27T21:42:44.945 INFO:teuthology.orchestra.run.smithi060.stdout: 408094 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/bench/% 2024-03-27T21:42:44.946 INFO:teuthology.orchestra.run.smithi060.stdout: 408095 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/bench/.qa -> ../.qa/ 2024-03-27T21:42:44.946 INFO:teuthology.orchestra.run.smithi060.stdout: 408096 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/bench/clusters 2024-03-27T21:42:44.946 INFO:teuthology.orchestra.run.smithi060.stdout: 408097 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/bench/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.946 INFO:teuthology.orchestra.run.smithi060.stdout: 408098 0 lrwxrwxrwx 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/bench/clusters/fixed-3-cephfs.yaml -> .qa/clusters/fixed-3-cephfs.yaml 2024-03-27T21:42:44.946 INFO:teuthology.orchestra.run.smithi060.stdout: 408099 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/bench/tasks 2024-03-27T21:42:44.946 INFO:teuthology.orchestra.run.smithi060.stdout: 408100 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/bench/tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.946 INFO:teuthology.orchestra.run.smithi060.stdout: 408101 4 -rw-r--r-- 1 ubuntu ubuntu 91 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/bench/tasks/cfuse_workunit_snaps.yaml 2024-03-27T21:42:44.946 INFO:teuthology.orchestra.run.smithi060.stdout: 408102 4 -rw-r--r-- 1 ubuntu ubuntu 304 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/bench/tasks/kclient_workunit_suites_fsx.yaml 2024-03-27T21:42:44.947 INFO:teuthology.orchestra.run.smithi060.stdout: 408092 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/.qa -> ../.qa/ 2024-03-27T21:42:44.947 INFO:teuthology.orchestra.run.smithi060.stdout: 408103 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/thrash 2024-03-27T21:42:44.947 INFO:teuthology.orchestra.run.smithi060.stdout: 408104 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/thrash/% 2024-03-27T21:42:44.947 INFO:teuthology.orchestra.run.smithi060.stdout: 408105 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/thrash/.qa -> ../.qa/ 2024-03-27T21:42:44.947 INFO:teuthology.orchestra.run.smithi060.stdout: 408116 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/thrash/workloads 2024-03-27T21:42:44.947 INFO:teuthology.orchestra.run.smithi060.stdout: 408117 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/thrash/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.947 INFO:teuthology.orchestra.run.smithi060.stdout: 408119 4 -rw-r--r-- 1 ubuntu ubuntu 83 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/thrash/workloads/iozone_cfuse.yaml 2024-03-27T21:42:44.947 INFO:teuthology.orchestra.run.smithi060.stdout: 408120 4 -rw-r--r-- 1 ubuntu ubuntu 60 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/thrash/workloads/radosbench.yaml 2024-03-27T21:42:44.948 INFO:teuthology.orchestra.run.smithi060.stdout: 408121 4 -rw-r--r-- 1 ubuntu ubuntu 135 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/thrash/workloads/readwrite.yaml 2024-03-27T21:42:44.948 INFO:teuthology.orchestra.run.smithi060.stdout: 408118 4 -rw-r--r-- 1 ubuntu ubuntu 83 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/thrash/workloads/bonnie_cfuse.yaml 2024-03-27T21:42:44.948 INFO:teuthology.orchestra.run.smithi060.stdout: 408106 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/thrash/clusters 2024-03-27T21:42:44.948 INFO:teuthology.orchestra.run.smithi060.stdout: 408110 4 -rw-r--r-- 1 ubuntu ubuntu 135 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/thrash/clusters/8-osd.yaml 2024-03-27T21:42:44.948 INFO:teuthology.orchestra.run.smithi060.stdout: 408107 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/thrash/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.948 INFO:teuthology.orchestra.run.smithi060.stdout: 408108 4 -rw-r--r-- 1 ubuntu ubuntu 228 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/thrash/clusters/16-osd.yaml 2024-03-27T21:42:44.948 INFO:teuthology.orchestra.run.smithi060.stdout: 408109 4 -rw-r--r-- 1 ubuntu ubuntu 79 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/thrash/clusters/3-osd-1-machine.yaml 2024-03-27T21:42:44.948 INFO:teuthology.orchestra.run.smithi060.stdout: 408111 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/thrash/thrashers 2024-03-27T21:42:44.949 INFO:teuthology.orchestra.run.smithi060.stdout: 408112 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/thrash/thrashers/.qa -> ../.qa/ 2024-03-27T21:42:44.949 INFO:teuthology.orchestra.run.smithi060.stdout: 408114 4 -rw-r--r-- 1 ubuntu ubuntu 168 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/thrash/thrashers/fast.yaml 2024-03-27T21:42:44.949 INFO:teuthology.orchestra.run.smithi060.stdout: 408113 4 -rw-r--r-- 1 ubuntu ubuntu 132 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/thrash/thrashers/default.yaml 2024-03-27T21:42:44.949 INFO:teuthology.orchestra.run.smithi060.stdout: 408115 4 -rw-r--r-- 1 ubuntu ubuntu 152 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/stress/thrash/thrashers/more-down.yaml 2024-03-27T21:42:44.949 INFO:teuthology.orchestra.run.smithi060.stdout: 404636 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible 2024-03-27T21:42:44.949 INFO:teuthology.orchestra.run.smithi060.stdout: 404637 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/.qa -> ../.qa/ 2024-03-27T21:42:44.949 INFO:teuthology.orchestra.run.smithi060.stdout: 404638 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke 2024-03-27T21:42:44.949 INFO:teuthology.orchestra.run.smithi060.stdout: 404639 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/.qa -> ../.qa/ 2024-03-27T21:42:44.949 INFO:teuthology.orchestra.run.smithi060.stdout: 404640 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic 2024-03-27T21:42:44.950 INFO:teuthology.orchestra.run.smithi060.stdout: 404654 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/3-config 2024-03-27T21:42:44.950 INFO:teuthology.orchestra.run.smithi060.stdout: 404656 4 -rw-r--r-- 1 ubuntu ubuntu 144 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/3-config/bluestore_with_dmcrypt.yaml 2024-03-27T21:42:44.950 INFO:teuthology.orchestra.run.smithi060.stdout: 404655 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/3-config/.qa -> ../.qa/ 2024-03-27T21:42:44.950 INFO:teuthology.orchestra.run.smithi060.stdout: 404658 4 -rw-r--r-- 1 ubuntu ubuntu 97 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/3-config/dmcrypt_on.yaml 2024-03-27T21:42:44.950 INFO:teuthology.orchestra.run.smithi060.stdout: 404657 4 -rw-r--r-- 1 ubuntu ubuntu 95 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/3-config/dmcrypt_off.yaml 2024-03-27T21:42:44.950 INFO:teuthology.orchestra.run.smithi060.stdout: 404641 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/% 2024-03-27T21:42:44.950 INFO:teuthology.orchestra.run.smithi060.stdout: 404643 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/0-clusters 2024-03-27T21:42:44.950 INFO:teuthology.orchestra.run.smithi060.stdout: 404644 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/0-clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.951 INFO:teuthology.orchestra.run.smithi060.stdout: 404646 4 -rw-r--r-- 1 ubuntu ubuntu 326 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/0-clusters/4-node.yaml 2024-03-27T21:42:44.951 INFO:teuthology.orchestra.run.smithi060.stdout: 404645 4 -rw-r--r-- 1 ubuntu ubuntu 311 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/0-clusters/3-node.yaml 2024-03-27T21:42:44.951 INFO:teuthology.orchestra.run.smithi060.stdout: 404659 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/4-tasks 2024-03-27T21:42:44.951 INFO:teuthology.orchestra.run.smithi060.stdout: 404660 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/4-tasks/.qa -> ../.qa/ 2024-03-27T21:42:44.951 INFO:teuthology.orchestra.run.smithi060.stdout: 404662 4 -rw-r--r-- 1 ubuntu ubuntu 127 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/4-tasks/rbd_import_export.yaml 2024-03-27T21:42:44.951 INFO:teuthology.orchestra.run.smithi060.stdout: 404663 4 -rw-r--r-- 1 ubuntu ubuntu 300 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/4-tasks/rest.yaml 2024-03-27T21:42:44.951 INFO:teuthology.orchestra.run.smithi060.stdout: 404661 4 -rw-r--r-- 1 ubuntu ubuntu 135 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/4-tasks/ceph-admin-commands.yaml 2024-03-27T21:42:44.951 INFO:teuthology.orchestra.run.smithi060.stdout: 404642 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/.qa -> ../.qa/ 2024-03-27T21:42:44.952 INFO:teuthology.orchestra.run.smithi060.stdout: 404647 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/1-distros 2024-03-27T21:42:44.952 INFO:teuthology.orchestra.run.smithi060.stdout: 404648 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/1-distros/.qa -> ../.qa/ 2024-03-27T21:42:44.952 INFO:teuthology.orchestra.run.smithi060.stdout: 404650 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/1-distros/ubuntu_latest.yaml -> .qa/distros/supported/ubuntu_latest.yaml 2024-03-27T21:42:44.952 INFO:teuthology.orchestra.run.smithi060.stdout: 404649 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/1-distros/centos_latest.yaml -> .qa/distros/supported/centos_latest.yaml 2024-03-27T21:42:44.952 INFO:teuthology.orchestra.run.smithi060.stdout: 404651 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/2-ceph 2024-03-27T21:42:44.952 INFO:teuthology.orchestra.run.smithi060.stdout: 404652 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/2-ceph/.qa -> ../.qa/ 2024-03-27T21:42:44.952 INFO:teuthology.orchestra.run.smithi060.stdout: 404653 4 -rw-r--r-- 1 ubuntu ubuntu 958 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/ceph-ansible/smoke/basic/2-ceph/ceph_ansible.yaml 2024-03-27T21:42:44.952 INFO:teuthology.orchestra.run.smithi060.stdout: 407154 4 drwxr-xr-x 21 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd 2024-03-27T21:42:44.953 INFO:teuthology.orchestra.run.smithi060.stdout: 407355 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance 2024-03-27T21:42:44.953 INFO:teuthology.orchestra.run.smithi060.stdout: 407356 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance/% 2024-03-27T21:42:44.953 INFO:teuthology.orchestra.run.smithi060.stdout: 407368 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance/qemu 2024-03-27T21:42:44.953 INFO:teuthology.orchestra.run.smithi060.stdout: 407369 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance/qemu/.qa -> ../.qa/ 2024-03-27T21:42:44.953 INFO:teuthology.orchestra.run.smithi060.stdout: 407370 4 -rw-r--r-- 1 ubuntu ubuntu 261 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance/qemu/xfstests.yaml 2024-03-27T21:42:44.953 INFO:teuthology.orchestra.run.smithi060.stdout: 407367 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance/objectstore -> .qa/objectstore 2024-03-27T21:42:44.953 INFO:teuthology.orchestra.run.smithi060.stdout: 407357 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance/.qa -> ../.qa/ 2024-03-27T21:42:44.953 INFO:teuthology.orchestra.run.smithi060.stdout: 407366 0 lrwxrwxrwx 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance/conf -> .qa/rbd/conf 2024-03-27T21:42:44.953 INFO:teuthology.orchestra.run.smithi060.stdout: 407371 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.954 INFO:teuthology.orchestra.run.smithi060.stdout: 407372 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance/workloads 2024-03-27T21:42:44.954 INFO:teuthology.orchestra.run.smithi060.stdout: 407373 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.954 INFO:teuthology.orchestra.run.smithi060.stdout: 407375 4 -rw-r--r-- 1 ubuntu ubuntu 260 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance/workloads/dynamic_features_no_cache.yaml 2024-03-27T21:42:44.954 INFO:teuthology.orchestra.run.smithi060.stdout: 407376 4 -rw-r--r-- 1 ubuntu ubuntu 175 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance/workloads/rebuild_object_map.yaml 2024-03-27T21:42:44.954 INFO:teuthology.orchestra.run.smithi060.stdout: 407374 4 -rw-r--r-- 1 ubuntu ubuntu 173 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance/workloads/dynamic_features.yaml 2024-03-27T21:42:44.954 INFO:teuthology.orchestra.run.smithi060.stdout: 407361 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance/clusters 2024-03-27T21:42:44.954 INFO:teuthology.orchestra.run.smithi060.stdout: 407364 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance/clusters/fixed-3.yaml -> .qa/clusters/fixed-3.yaml 2024-03-27T21:42:44.955 INFO:teuthology.orchestra.run.smithi060.stdout: 407362 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance/clusters/+ 2024-03-27T21:42:44.955 INFO:teuthology.orchestra.run.smithi060.stdout: 407363 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.955 INFO:teuthology.orchestra.run.smithi060.stdout: 407365 0 lrwxrwxrwx 1 ubuntu ubuntu 34 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance/clusters/openstack.yaml -> ../../qemu/clusters/openstack.yaml 2024-03-27T21:42:44.955 INFO:teuthology.orchestra.run.smithi060.stdout: 407358 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance/base 2024-03-27T21:42:44.955 INFO:teuthology.orchestra.run.smithi060.stdout: 407360 4 -rw-r--r-- 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance/base/install.yaml 2024-03-27T21:42:44.955 INFO:teuthology.orchestra.run.smithi060.stdout: 407359 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/maintenance/base/.qa -> ../.qa/ 2024-03-27T21:42:44.955 INFO:teuthology.orchestra.run.smithi060.stdout: 407226 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/device 2024-03-27T21:42:44.955 INFO:teuthology.orchestra.run.smithi060.stdout: 407227 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/device/% 2024-03-27T21:42:44.955 INFO:teuthology.orchestra.run.smithi060.stdout: 407237 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/device/objectstore -> .qa/objectstore 2024-03-27T21:42:44.956 INFO:teuthology.orchestra.run.smithi060.stdout: 407236 0 lrwxrwxrwx 1 ubuntu ubuntu 23 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/device/msgr-failures -> ../thrash/msgr-failures 2024-03-27T21:42:44.956 INFO:teuthology.orchestra.run.smithi060.stdout: 407228 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/device/.qa -> ../.qa/ 2024-03-27T21:42:44.956 INFO:teuthology.orchestra.run.smithi060.stdout: 407235 0 lrwxrwxrwx 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/device/conf -> .qa/rbd/conf 2024-03-27T21:42:44.956 INFO:teuthology.orchestra.run.smithi060.stdout: 407240 0 lrwxrwxrwx 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/device/thrashosds-health.yaml -> .qa/tasks/thrashosds-health.yaml 2024-03-27T21:42:44.956 INFO:teuthology.orchestra.run.smithi060.stdout: 407238 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/device/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.956 INFO:teuthology.orchestra.run.smithi060.stdout: 407241 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/device/workloads 2024-03-27T21:42:44.956 INFO:teuthology.orchestra.run.smithi060.stdout: 407246 4 -rw-r--r-- 1 ubuntu ubuntu 137 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/device/workloads/rbd_nbd.yaml 2024-03-27T21:42:44.957 INFO:teuthology.orchestra.run.smithi060.stdout: 407243 4 -rw-r--r-- 1 ubuntu ubuntu 189 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/device/workloads/diff-continuous-krbd.yaml 2024-03-27T21:42:44.957 INFO:teuthology.orchestra.run.smithi060.stdout: 407242 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/device/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.957 INFO:teuthology.orchestra.run.smithi060.stdout: 407244 4 -rw-r--r-- 1 ubuntu ubuntu 233 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/device/workloads/diff-continuous-nbd.yaml 2024-03-27T21:42:44.957 INFO:teuthology.orchestra.run.smithi060.stdout: 407245 4 -rw-r--r-- 1 ubuntu ubuntu 226 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/device/workloads/rbd_fsx_nbd.yaml 2024-03-27T21:42:44.957 INFO:teuthology.orchestra.run.smithi060.stdout: 407229 0 lrwxrwxrwx 1 ubuntu ubuntu 14 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/device/base -> ../thrash/base 2024-03-27T21:42:44.957 INFO:teuthology.orchestra.run.smithi060.stdout: 407239 0 lrwxrwxrwx 1 ubuntu ubuntu 19 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/device/thrashers -> ../thrash/thrashers 2024-03-27T21:42:44.957 INFO:teuthology.orchestra.run.smithi060.stdout: 407230 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/device/cluster 2024-03-27T21:42:44.958 INFO:teuthology.orchestra.run.smithi060.stdout: 407233 4 -rw-r--r-- 1 ubuntu ubuntu 96 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/device/cluster/fixed-3.yaml 2024-03-27T21:42:44.958 INFO:teuthology.orchestra.run.smithi060.stdout: 407231 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/device/cluster/+ 2024-03-27T21:42:44.958 INFO:teuthology.orchestra.run.smithi060.stdout: 407232 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/device/cluster/.qa -> ../.qa/ 2024-03-27T21:42:44.958 INFO:teuthology.orchestra.run.smithi060.stdout: 407234 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/device/cluster/openstack.yaml -> ../../thrash/clusters/openstack.yaml 2024-03-27T21:42:44.958 INFO:teuthology.orchestra.run.smithi060.stdout: 407299 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/iscsi 2024-03-27T21:42:44.958 INFO:teuthology.orchestra.run.smithi060.stdout: 407300 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/iscsi/% 2024-03-27T21:42:44.958 INFO:teuthology.orchestra.run.smithi060.stdout: 407301 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/iscsi/.qa -> ../.qa/ 2024-03-27T21:42:44.958 INFO:teuthology.orchestra.run.smithi060.stdout: 407311 0 lrwxrwxrwx 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/iscsi/conf -> .qa/rbd/conf 2024-03-27T21:42:44.959 INFO:teuthology.orchestra.run.smithi060.stdout: 407302 0 lrwxrwxrwx 1 ubuntu ubuntu 38 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/iscsi/0-single-container-host.yaml -> .qa/distros/single-container-host.yaml 2024-03-27T21:42:44.959 INFO:teuthology.orchestra.run.smithi060.stdout: 407312 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/iscsi/workloads 2024-03-27T21:42:44.959 INFO:teuthology.orchestra.run.smithi060.stdout: 407313 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/iscsi/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.959 INFO:teuthology.orchestra.run.smithi060.stdout: 407314 4 -rw-r--r-- 1 ubuntu ubuntu 548 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/iscsi/workloads/cephadm_iscsi.yaml 2024-03-27T21:42:44.959 INFO:teuthology.orchestra.run.smithi060.stdout: 407303 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/iscsi/base 2024-03-27T21:42:44.959 INFO:teuthology.orchestra.run.smithi060.stdout: 407305 4 -rw-r--r-- 1 ubuntu ubuntu 268 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/iscsi/base/install.yaml 2024-03-27T21:42:44.959 INFO:teuthology.orchestra.run.smithi060.stdout: 407304 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/iscsi/base/.qa -> ../.qa/ 2024-03-27T21:42:44.959 INFO:teuthology.orchestra.run.smithi060.stdout: 407306 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/iscsi/cluster 2024-03-27T21:42:44.960 INFO:teuthology.orchestra.run.smithi060.stdout: 407309 4 -rw-r--r-- 1 ubuntu ubuntu 223 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/iscsi/cluster/fixed-3.yaml 2024-03-27T21:42:44.960 INFO:teuthology.orchestra.run.smithi060.stdout: 407307 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/iscsi/cluster/+ 2024-03-27T21:42:44.960 INFO:teuthology.orchestra.run.smithi060.stdout: 407308 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/iscsi/cluster/.qa -> ../.qa/ 2024-03-27T21:42:44.960 INFO:teuthology.orchestra.run.smithi060.stdout: 407310 4 -rw-r--r-- 1 ubuntu ubuntu 155 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/iscsi/cluster/openstack.yaml 2024-03-27T21:42:44.960 INFO:teuthology.orchestra.run.smithi060.stdout: 407535 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu 2024-03-27T21:42:44.960 INFO:teuthology.orchestra.run.smithi060.stdout: 407536 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/% 2024-03-27T21:42:44.960 INFO:teuthology.orchestra.run.smithi060.stdout: 407559 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/objectstore -> .qa/objectstore 2024-03-27T21:42:44.960 INFO:teuthology.orchestra.run.smithi060.stdout: 407556 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/msgr-failures 2024-03-27T21:42:44.960 INFO:teuthology.orchestra.run.smithi060.stdout: 407557 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.961 INFO:teuthology.orchestra.run.smithi060.stdout: 407558 4 -rw-r--r-- 1 ubuntu ubuntu 205 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/msgr-failures/few.yaml 2024-03-27T21:42:44.961 INFO:teuthology.orchestra.run.smithi060.stdout: 407537 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/.qa -> ../.qa/ 2024-03-27T21:42:44.961 INFO:teuthology.orchestra.run.smithi060.stdout: 407549 0 lrwxrwxrwx 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/conf -> .qa/rbd/conf 2024-03-27T21:42:44.961 INFO:teuthology.orchestra.run.smithi060.stdout: 407560 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.961 INFO:teuthology.orchestra.run.smithi060.stdout: 407561 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/workloads 2024-03-27T21:42:44.961 INFO:teuthology.orchestra.run.smithi060.stdout: 407562 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.961 INFO:teuthology.orchestra.run.smithi060.stdout: 407565 4 -rw-r--r-- 1 ubuntu ubuntu 111 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/workloads/qemu_iozone.yaml.disabled 2024-03-27T21:42:44.961 INFO:teuthology.orchestra.run.smithi060.stdout: 407566 4 -rw-r--r-- 1 ubuntu ubuntu 132 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/workloads/qemu_xfstests.yaml 2024-03-27T21:42:44.962 INFO:teuthology.orchestra.run.smithi060.stdout: 407563 4 -rw-r--r-- 1 ubuntu ubuntu 105 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/workloads/qemu_bonnie.yaml 2024-03-27T21:42:44.962 INFO:teuthology.orchestra.run.smithi060.stdout: 407564 4 -rw-r--r-- 1 ubuntu ubuntu 107 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/workloads/qemu_fsstress.yaml 2024-03-27T21:42:44.962 INFO:teuthology.orchestra.run.smithi060.stdout: 407544 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/clusters 2024-03-27T21:42:44.962 INFO:teuthology.orchestra.run.smithi060.stdout: 407547 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/clusters/fixed-3.yaml -> .qa/clusters/fixed-3.yaml 2024-03-27T21:42:44.962 INFO:teuthology.orchestra.run.smithi060.stdout: 407545 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/clusters/+ 2024-03-27T21:42:44.962 INFO:teuthology.orchestra.run.smithi060.stdout: 407546 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.962 INFO:teuthology.orchestra.run.smithi060.stdout: 407548 4 -rw-r--r-- 1 ubuntu ubuntu 156 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/clusters/openstack.yaml 2024-03-27T21:42:44.962 INFO:teuthology.orchestra.run.smithi060.stdout: 407551 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/features 2024-03-27T21:42:44.963 INFO:teuthology.orchestra.run.smithi060.stdout: 407552 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/features/.qa -> ../.qa/ 2024-03-27T21:42:44.963 INFO:teuthology.orchestra.run.smithi060.stdout: 407553 4 -rw-r--r-- 1 ubuntu ubuntu 76 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/features/defaults.yaml 2024-03-27T21:42:44.963 INFO:teuthology.orchestra.run.smithi060.stdout: 407554 4 -rw-r--r-- 1 ubuntu ubuntu 77 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/features/journaling.yaml 2024-03-27T21:42:44.963 INFO:teuthology.orchestra.run.smithi060.stdout: 407555 4 -rw-r--r-- 1 ubuntu ubuntu 171 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/features/readbalance.yaml 2024-03-27T21:42:44.963 INFO:teuthology.orchestra.run.smithi060.stdout: 407550 0 lrwxrwxrwx 1 ubuntu ubuntu 18 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/data-pool -> .qa/rbd/data-pool/ 2024-03-27T21:42:44.963 INFO:teuthology.orchestra.run.smithi060.stdout: 407538 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/cache 2024-03-27T21:42:44.963 INFO:teuthology.orchestra.run.smithi060.stdout: 407539 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/cache/.qa -> ../.qa/ 2024-03-27T21:42:44.963 INFO:teuthology.orchestra.run.smithi060.stdout: 407543 4 -rw-r--r-- 1 ubuntu ubuntu 105 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/cache/writethrough.yaml 2024-03-27T21:42:44.964 INFO:teuthology.orchestra.run.smithi060.stdout: 407540 4 -rw-r--r-- 1 ubuntu ubuntu 75 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/cache/none.yaml 2024-03-27T21:42:44.964 INFO:teuthology.orchestra.run.smithi060.stdout: 407542 4 -rw-r--r-- 1 ubuntu ubuntu 110 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/cache/writeback.yaml 2024-03-27T21:42:44.964 INFO:teuthology.orchestra.run.smithi060.stdout: 407541 4 -rw-r--r-- 1 ubuntu ubuntu 112 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/qemu/cache/writearound.yaml 2024-03-27T21:42:44.964 INFO:teuthology.orchestra.run.smithi060.stdout: 407644 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind 2024-03-27T21:42:44.964 INFO:teuthology.orchestra.run.smithi060.stdout: 407654 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/validator 2024-03-27T21:42:44.964 INFO:teuthology.orchestra.run.smithi060.stdout: 407655 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/validator/.qa -> ../.qa/ 2024-03-27T21:42:44.964 INFO:teuthology.orchestra.run.smithi060.stdout: 407656 4 -rw-r--r-- 1 ubuntu ubuntu 273 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/validator/memcheck.yaml 2024-03-27T21:42:44.964 INFO:teuthology.orchestra.run.smithi060.stdout: 407645 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/% 2024-03-27T21:42:44.964 INFO:teuthology.orchestra.run.smithi060.stdout: 407653 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/objectstore -> .qa/objectstore 2024-03-27T21:42:44.965 INFO:teuthology.orchestra.run.smithi060.stdout: 407646 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/.qa -> ../.qa/ 2024-03-27T21:42:44.965 INFO:teuthology.orchestra.run.smithi060.stdout: 407652 0 lrwxrwxrwx 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/conf -> .qa/rbd/conf 2024-03-27T21:42:44.965 INFO:teuthology.orchestra.run.smithi060.stdout: 407657 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/workloads 2024-03-27T21:42:44.965 INFO:teuthology.orchestra.run.smithi060.stdout: 407658 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.965 INFO:teuthology.orchestra.run.smithi060.stdout: 407666 4 -rw-r--r-- 1 ubuntu ubuntu 178 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/workloads/rbd_mirror.yaml 2024-03-27T21:42:44.965 INFO:teuthology.orchestra.run.smithi060.stdout: 407659 4 -rw-r--r-- 1 ubuntu ubuntu 261 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/workloads/c_api_tests.yaml 2024-03-27T21:42:44.965 INFO:teuthology.orchestra.run.smithi060.stdout: 407660 4 -rw-r--r-- 1 ubuntu ubuntu 262 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/workloads/c_api_tests_with_defaults.yaml 2024-03-27T21:42:44.965 INFO:teuthology.orchestra.run.smithi060.stdout: 407665 4 -rw-r--r-- 1 ubuntu ubuntu 203 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/workloads/python_api_tests_with_journaling.yaml 2024-03-27T21:42:44.966 INFO:teuthology.orchestra.run.smithi060.stdout: 407663 4 -rw-r--r-- 1 ubuntu ubuntu 201 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/workloads/python_api_tests.yaml 2024-03-27T21:42:44.966 INFO:teuthology.orchestra.run.smithi060.stdout: 407661 4 -rw-r--r-- 1 ubuntu ubuntu 263 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/workloads/c_api_tests_with_journaling.yaml 2024-03-27T21:42:44.966 INFO:teuthology.orchestra.run.smithi060.stdout: 407664 4 -rw-r--r-- 1 ubuntu ubuntu 202 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/workloads/python_api_tests_with_defaults.yaml 2024-03-27T21:42:44.966 INFO:teuthology.orchestra.run.smithi060.stdout: 407662 4 -rw-r--r-- 1 ubuntu ubuntu 62 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/workloads/fsx.yaml 2024-03-27T21:42:44.966 INFO:teuthology.orchestra.run.smithi060.stdout: 407651 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/clusters -> ../basic/clusters 2024-03-27T21:42:44.966 INFO:teuthology.orchestra.run.smithi060.stdout: 407650 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/centos_latest.yaml -> .qa/distros/supported/centos_latest.yaml 2024-03-27T21:42:44.966 INFO:teuthology.orchestra.run.smithi060.stdout: 407647 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/base 2024-03-27T21:42:44.966 INFO:teuthology.orchestra.run.smithi060.stdout: 407649 4 -rw-r--r-- 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/base/install.yaml 2024-03-27T21:42:44.967 INFO:teuthology.orchestra.run.smithi060.stdout: 407648 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/valgrind/base/.qa -> ../.qa/ 2024-03-27T21:42:44.967 INFO:teuthology.orchestra.run.smithi060.stdout: 407281 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/immutable-object-cache 2024-03-27T21:42:44.967 INFO:teuthology.orchestra.run.smithi060.stdout: 407282 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/immutable-object-cache/% 2024-03-27T21:42:44.967 INFO:teuthology.orchestra.run.smithi060.stdout: 407283 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/immutable-object-cache/.qa -> ../.qa/ 2024-03-27T21:42:44.967 INFO:teuthology.orchestra.run.smithi060.stdout: 407289 0 lrwxrwxrwx 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/immutable-object-cache/conf -> .qa/rbd/conf 2024-03-27T21:42:44.967 INFO:teuthology.orchestra.run.smithi060.stdout: 407293 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/immutable-object-cache/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.967 INFO:teuthology.orchestra.run.smithi060.stdout: 407294 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/immutable-object-cache/workloads 2024-03-27T21:42:44.967 INFO:teuthology.orchestra.run.smithi060.stdout: 407298 4 -rw-r--r-- 1 ubuntu ubuntu 196 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/immutable-object-cache/workloads/qemu_on_immutable_object_cache_and_thrash.yaml 2024-03-27T21:42:44.968 INFO:teuthology.orchestra.run.smithi060.stdout: 407295 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/immutable-object-cache/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.968 INFO:teuthology.orchestra.run.smithi060.stdout: 407296 0 lrwxrwxrwx 1 ubuntu ubuntu 53 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/immutable-object-cache/workloads/c_api_tests_with_defaults.yaml -> ../../librbd/workloads/c_api_tests_with_defaults.yaml 2024-03-27T21:42:44.968 INFO:teuthology.orchestra.run.smithi060.stdout: 407297 4 -rw-r--r-- 1 ubuntu ubuntu 214 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/immutable-object-cache/workloads/fio_on_immutable_object_cache.yaml 2024-03-27T21:42:44.968 INFO:teuthology.orchestra.run.smithi060.stdout: 407284 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/immutable-object-cache/clusters 2024-03-27T21:42:44.968 INFO:teuthology.orchestra.run.smithi060.stdout: 407285 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/immutable-object-cache/clusters/+ 2024-03-27T21:42:44.968 INFO:teuthology.orchestra.run.smithi060.stdout: 407286 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/immutable-object-cache/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.968 INFO:teuthology.orchestra.run.smithi060.stdout: 407287 4 -rw-r--r-- 1 ubuntu ubuntu 79 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/immutable-object-cache/clusters/fix-2.yaml 2024-03-27T21:42:44.968 INFO:teuthology.orchestra.run.smithi060.stdout: 407288 4 -rw-r--r-- 1 ubuntu ubuntu 93 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/immutable-object-cache/clusters/openstack.yaml 2024-03-27T21:42:44.969 INFO:teuthology.orchestra.run.smithi060.stdout: 407290 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/immutable-object-cache/pool 2024-03-27T21:42:44.969 INFO:teuthology.orchestra.run.smithi060.stdout: 407291 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/immutable-object-cache/pool/.qa -> ../.qa/ 2024-03-27T21:42:44.969 INFO:teuthology.orchestra.run.smithi060.stdout: 407292 4 -rw-r--r-- 1 ubuntu ubuntu 330 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/immutable-object-cache/pool/ceph_and_immutable_object_cache.yaml 2024-03-27T21:42:44.969 INFO:teuthology.orchestra.run.smithi060.stdout: 407155 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/.qa -> ../.qa/ 2024-03-27T21:42:44.969 INFO:teuthology.orchestra.run.smithi060.stdout: 407247 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption 2024-03-27T21:42:44.969 INFO:teuthology.orchestra.run.smithi060.stdout: 407248 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/% 2024-03-27T21:42:44.969 INFO:teuthology.orchestra.run.smithi060.stdout: 407269 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/objectstore -> .qa/objectstore 2024-03-27T21:42:44.969 INFO:teuthology.orchestra.run.smithi060.stdout: 407266 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/msgr-failures 2024-03-27T21:42:44.969 INFO:teuthology.orchestra.run.smithi060.stdout: 407267 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.970 INFO:teuthology.orchestra.run.smithi060.stdout: 407268 4 -rw-r--r-- 1 ubuntu ubuntu 205 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/msgr-failures/few.yaml 2024-03-27T21:42:44.970 INFO:teuthology.orchestra.run.smithi060.stdout: 407249 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/.qa -> ../.qa/ 2024-03-27T21:42:44.970 INFO:teuthology.orchestra.run.smithi060.stdout: 407261 0 lrwxrwxrwx 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/conf -> .qa/rbd/conf 2024-03-27T21:42:44.970 INFO:teuthology.orchestra.run.smithi060.stdout: 407270 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.970 INFO:teuthology.orchestra.run.smithi060.stdout: 407271 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/workloads 2024-03-27T21:42:44.970 INFO:teuthology.orchestra.run.smithi060.stdout: 407277 4 -rw-r--r-- 1 ubuntu ubuntu 265 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/workloads/qemu_xfstests_luks2_luks1.yaml 2024-03-27T21:42:44.970 INFO:teuthology.orchestra.run.smithi060.stdout: 407274 4 -rw-r--r-- 1 ubuntu ubuntu 265 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/workloads/qemu_xfstests_luks1_luks1.yaml 2024-03-27T21:42:44.970 INFO:teuthology.orchestra.run.smithi060.stdout: 407272 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.971 INFO:teuthology.orchestra.run.smithi060.stdout: 407279 4 -rw-r--r-- 1 ubuntu ubuntu 227 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/workloads/qemu_xfstests_none_luks1.yaml 2024-03-27T21:42:44.971 INFO:teuthology.orchestra.run.smithi060.stdout: 407273 4 -rw-r--r-- 1 ubuntu ubuntu 234 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/workloads/qemu_xfstests_luks1.yaml 2024-03-27T21:42:44.971 INFO:teuthology.orchestra.run.smithi060.stdout: 407275 4 -rw-r--r-- 1 ubuntu ubuntu 265 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/workloads/qemu_xfstests_luks1_luks2.yaml 2024-03-27T21:42:44.971 INFO:teuthology.orchestra.run.smithi060.stdout: 407280 4 -rw-r--r-- 1 ubuntu ubuntu 227 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/workloads/qemu_xfstests_none_luks2.yaml 2024-03-27T21:42:44.971 INFO:teuthology.orchestra.run.smithi060.stdout: 407276 4 -rw-r--r-- 1 ubuntu ubuntu 234 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/workloads/qemu_xfstests_luks2.yaml 2024-03-27T21:42:44.971 INFO:teuthology.orchestra.run.smithi060.stdout: 407278 4 -rw-r--r-- 1 ubuntu ubuntu 265 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/workloads/qemu_xfstests_luks2_luks2.yaml 2024-03-27T21:42:44.971 INFO:teuthology.orchestra.run.smithi060.stdout: 407256 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/clusters 2024-03-27T21:42:44.971 INFO:teuthology.orchestra.run.smithi060.stdout: 407259 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/clusters/fixed-3.yaml -> .qa/clusters/fixed-3.yaml 2024-03-27T21:42:44.971 INFO:teuthology.orchestra.run.smithi060.stdout: 407257 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/clusters/+ 2024-03-27T21:42:44.972 INFO:teuthology.orchestra.run.smithi060.stdout: 407258 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.972 INFO:teuthology.orchestra.run.smithi060.stdout: 407260 4 -rw-r--r-- 1 ubuntu ubuntu 156 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/clusters/openstack.yaml 2024-03-27T21:42:44.972 INFO:teuthology.orchestra.run.smithi060.stdout: 407263 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/features 2024-03-27T21:42:44.972 INFO:teuthology.orchestra.run.smithi060.stdout: 407264 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/features/.qa -> ../.qa/ 2024-03-27T21:42:44.972 INFO:teuthology.orchestra.run.smithi060.stdout: 407265 4 -rw-r--r-- 1 ubuntu ubuntu 76 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/features/defaults.yaml 2024-03-27T21:42:44.972 INFO:teuthology.orchestra.run.smithi060.stdout: 407262 0 lrwxrwxrwx 1 ubuntu ubuntu 18 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/data-pool -> .qa/rbd/data-pool/ 2024-03-27T21:42:44.972 INFO:teuthology.orchestra.run.smithi060.stdout: 407250 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/cache 2024-03-27T21:42:44.972 INFO:teuthology.orchestra.run.smithi060.stdout: 407251 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/cache/.qa -> ../.qa/ 2024-03-27T21:42:44.972 INFO:teuthology.orchestra.run.smithi060.stdout: 407255 4 -rw-r--r-- 1 ubuntu ubuntu 105 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/cache/writethrough.yaml 2024-03-27T21:42:44.973 INFO:teuthology.orchestra.run.smithi060.stdout: 407252 4 -rw-r--r-- 1 ubuntu ubuntu 75 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/cache/none.yaml 2024-03-27T21:42:44.973 INFO:teuthology.orchestra.run.smithi060.stdout: 407254 4 -rw-r--r-- 1 ubuntu ubuntu 110 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/cache/writeback.yaml 2024-03-27T21:42:44.973 INFO:teuthology.orchestra.run.smithi060.stdout: 407253 4 -rw-r--r-- 1 ubuntu ubuntu 112 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/encryption/cache/writearound.yaml 2024-03-27T21:42:44.973 INFO:teuthology.orchestra.run.smithi060.stdout: 407179 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli 2024-03-27T21:42:44.973 INFO:teuthology.orchestra.run.smithi060.stdout: 407180 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/% 2024-03-27T21:42:44.973 INFO:teuthology.orchestra.run.smithi060.stdout: 407196 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/objectstore -> .qa/objectstore 2024-03-27T21:42:44.973 INFO:teuthology.orchestra.run.smithi060.stdout: 407193 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/msgr-failures 2024-03-27T21:42:44.973 INFO:teuthology.orchestra.run.smithi060.stdout: 407194 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.973 INFO:teuthology.orchestra.run.smithi060.stdout: 407195 4 -rw-r--r-- 1 ubuntu ubuntu 177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/msgr-failures/few.yaml 2024-03-27T21:42:44.974 INFO:teuthology.orchestra.run.smithi060.stdout: 407181 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/.qa -> ../.qa/ 2024-03-27T21:42:44.974 INFO:teuthology.orchestra.run.smithi060.stdout: 407186 0 lrwxrwxrwx 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/conf -> .qa/rbd/conf 2024-03-27T21:42:44.974 INFO:teuthology.orchestra.run.smithi060.stdout: 407197 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.974 INFO:teuthology.orchestra.run.smithi060.stdout: 407198 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/workloads 2024-03-27T21:42:44.974 INFO:teuthology.orchestra.run.smithi060.stdout: 407199 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.974 INFO:teuthology.orchestra.run.smithi060.stdout: 407204 4 -rw-r--r-- 1 ubuntu ubuntu 79 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/workloads/rbd_cli_migration.yaml 2024-03-27T21:42:44.974 INFO:teuthology.orchestra.run.smithi060.stdout: 407202 4 -rw-r--r-- 1 ubuntu ubuntu 79 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/workloads/rbd_cli_import_export.yaml 2024-03-27T21:42:44.974 INFO:teuthology.orchestra.run.smithi060.stdout: 407205 4 -rw-r--r-- 1 ubuntu ubuntu 205 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/workloads/rbd_support_module_recovery.yaml 2024-03-27T21:42:44.975 INFO:teuthology.orchestra.run.smithi060.stdout: 407203 4 -rw-r--r-- 1 ubuntu ubuntu 145 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/workloads/rbd_cli_luks_encryption.yaml 2024-03-27T21:42:44.975 INFO:teuthology.orchestra.run.smithi060.stdout: 407201 4 -rw-r--r-- 1 ubuntu ubuntu 76 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/workloads/rbd_cli_groups.yaml 2024-03-27T21:42:44.975 INFO:teuthology.orchestra.run.smithi060.stdout: 407200 4 -rw-r--r-- 1 ubuntu ubuntu 77 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/workloads/rbd_cli_generic.yaml 2024-03-27T21:42:44.975 INFO:teuthology.orchestra.run.smithi060.stdout: 407185 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/clusters -> ../basic/clusters 2024-03-27T21:42:44.975 INFO:teuthology.orchestra.run.smithi060.stdout: 407188 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/features 2024-03-27T21:42:44.975 INFO:teuthology.orchestra.run.smithi060.stdout: 407189 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/features/.qa -> ../.qa/ 2024-03-27T21:42:44.975 INFO:teuthology.orchestra.run.smithi060.stdout: 407192 4 -rw-r--r-- 1 ubuntu ubuntu 75 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/features/layering.yaml 2024-03-27T21:42:44.975 INFO:teuthology.orchestra.run.smithi060.stdout: 407190 4 -rw-r--r-- 1 ubuntu ubuntu 76 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/features/defaults.yaml 2024-03-27T21:42:44.975 INFO:teuthology.orchestra.run.smithi060.stdout: 407191 4 -rw-r--r-- 1 ubuntu ubuntu 77 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/features/journaling.yaml 2024-03-27T21:42:44.976 INFO:teuthology.orchestra.run.smithi060.stdout: 407187 0 lrwxrwxrwx 1 ubuntu ubuntu 18 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/data-pool -> .qa/rbd/data-pool/ 2024-03-27T21:42:44.976 INFO:teuthology.orchestra.run.smithi060.stdout: 407182 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/base 2024-03-27T21:42:44.976 INFO:teuthology.orchestra.run.smithi060.stdout: 407184 4 -rw-r--r-- 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/base/install.yaml 2024-03-27T21:42:44.976 INFO:teuthology.orchestra.run.smithi060.stdout: 407183 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli/base/.qa -> ../.qa/ 2024-03-27T21:42:44.976 INFO:teuthology.orchestra.run.smithi060.stdout: 407206 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli_v1 2024-03-27T21:42:44.976 INFO:teuthology.orchestra.run.smithi060.stdout: 407207 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli_v1/% 2024-03-27T21:42:44.976 INFO:teuthology.orchestra.run.smithi060.stdout: 407220 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli_v1/objectstore -> .qa/objectstore 2024-03-27T21:42:44.976 INFO:teuthology.orchestra.run.smithi060.stdout: 407217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli_v1/msgr-failures 2024-03-27T21:42:44.976 INFO:teuthology.orchestra.run.smithi060.stdout: 407218 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli_v1/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.977 INFO:teuthology.orchestra.run.smithi060.stdout: 407219 4 -rw-r--r-- 1 ubuntu ubuntu 177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli_v1/msgr-failures/few.yaml 2024-03-27T21:42:44.977 INFO:teuthology.orchestra.run.smithi060.stdout: 407208 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli_v1/.qa -> ../.qa/ 2024-03-27T21:42:44.977 INFO:teuthology.orchestra.run.smithi060.stdout: 407213 0 lrwxrwxrwx 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli_v1/conf -> .qa/rbd/conf 2024-03-27T21:42:44.977 INFO:teuthology.orchestra.run.smithi060.stdout: 407221 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli_v1/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.977 INFO:teuthology.orchestra.run.smithi060.stdout: 407222 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli_v1/workloads 2024-03-27T21:42:44.977 INFO:teuthology.orchestra.run.smithi060.stdout: 407223 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli_v1/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.977 INFO:teuthology.orchestra.run.smithi060.stdout: 407225 4 -rw-r--r-- 1 ubuntu ubuntu 79 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli_v1/workloads/rbd_cli_import_export.yaml 2024-03-27T21:42:44.977 INFO:teuthology.orchestra.run.smithi060.stdout: 407224 4 -rw-r--r-- 1 ubuntu ubuntu 77 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli_v1/workloads/rbd_cli_generic.yaml 2024-03-27T21:42:44.978 INFO:teuthology.orchestra.run.smithi060.stdout: 407212 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli_v1/clusters -> ../basic/clusters 2024-03-27T21:42:44.978 INFO:teuthology.orchestra.run.smithi060.stdout: 407214 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli_v1/features 2024-03-27T21:42:44.978 INFO:teuthology.orchestra.run.smithi060.stdout: 407216 4 -rw-r--r-- 1 ubuntu ubuntu 73 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli_v1/features/format-1.yaml 2024-03-27T21:42:44.978 INFO:teuthology.orchestra.run.smithi060.stdout: 407215 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli_v1/features/.qa -> ../.qa/ 2024-03-27T21:42:44.978 INFO:teuthology.orchestra.run.smithi060.stdout: 407209 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli_v1/base 2024-03-27T21:42:44.978 INFO:teuthology.orchestra.run.smithi060.stdout: 407211 4 -rw-r--r-- 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli_v1/base/install.yaml 2024-03-27T21:42:44.978 INFO:teuthology.orchestra.run.smithi060.stdout: 407210 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/cli_v1/base/.qa -> ../.qa/ 2024-03-27T21:42:44.978 INFO:teuthology.orchestra.run.smithi060.stdout: 407315 4 drwxr-xr-x 8 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd 2024-03-27T21:42:44.979 INFO:teuthology.orchestra.run.smithi060.stdout: 407316 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/% 2024-03-27T21:42:44.979 INFO:teuthology.orchestra.run.smithi060.stdout: 407331 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/extra-conf 2024-03-27T21:42:44.979 INFO:teuthology.orchestra.run.smithi060.stdout: 407332 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/extra-conf/.qa -> ../.qa/ 2024-03-27T21:42:44.979 INFO:teuthology.orchestra.run.smithi060.stdout: 407335 4 -rw-r--r-- 1 ubuntu ubuntu 83 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/extra-conf/permit-partial-discard.yaml 2024-03-27T21:42:44.979 INFO:teuthology.orchestra.run.smithi060.stdout: 407334 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/extra-conf/none.yaml 2024-03-27T21:42:44.979 INFO:teuthology.orchestra.run.smithi060.stdout: 407333 4 -rw-r--r-- 1 ubuntu ubuntu 80 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/extra-conf/copy-on-read.yaml 2024-03-27T21:42:44.979 INFO:teuthology.orchestra.run.smithi060.stdout: 407343 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/objectstore -> .qa/objectstore 2024-03-27T21:42:44.979 INFO:teuthology.orchestra.run.smithi060.stdout: 407340 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/msgr-failures 2024-03-27T21:42:44.979 INFO:teuthology.orchestra.run.smithi060.stdout: 407341 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.980 INFO:teuthology.orchestra.run.smithi060.stdout: 407342 4 -rw-r--r-- 1 ubuntu ubuntu 209 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/msgr-failures/few.yaml 2024-03-27T21:42:44.980 INFO:teuthology.orchestra.run.smithi060.stdout: 407317 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/.qa -> ../.qa/ 2024-03-27T21:42:44.980 INFO:teuthology.orchestra.run.smithi060.stdout: 407329 0 lrwxrwxrwx 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/conf -> .qa/rbd/conf 2024-03-27T21:42:44.980 INFO:teuthology.orchestra.run.smithi060.stdout: 407344 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.980 INFO:teuthology.orchestra.run.smithi060.stdout: 407345 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/workloads 2024-03-27T21:42:44.980 INFO:teuthology.orchestra.run.smithi060.stdout: 407346 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.980 INFO:teuthology.orchestra.run.smithi060.stdout: 407347 4 -rw-r--r-- 1 ubuntu ubuntu 261 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/workloads/c_api_tests.yaml 2024-03-27T21:42:44.980 INFO:teuthology.orchestra.run.smithi060.stdout: 407354 4 -rw-r--r-- 1 ubuntu ubuntu 220 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/workloads/rbd_fio.yaml 2024-03-27T21:42:44.981 INFO:teuthology.orchestra.run.smithi060.stdout: 407348 4 -rw-r--r-- 1 ubuntu ubuntu 262 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/workloads/c_api_tests_with_defaults.yaml 2024-03-27T21:42:44.981 INFO:teuthology.orchestra.run.smithi060.stdout: 407353 4 -rw-r--r-- 1 ubuntu ubuntu 203 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/workloads/python_api_tests_with_journaling.yaml 2024-03-27T21:42:44.981 INFO:teuthology.orchestra.run.smithi060.stdout: 407351 4 -rw-r--r-- 1 ubuntu ubuntu 201 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/workloads/python_api_tests.yaml 2024-03-27T21:42:44.981 INFO:teuthology.orchestra.run.smithi060.stdout: 407349 4 -rw-r--r-- 1 ubuntu ubuntu 263 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/workloads/c_api_tests_with_journaling.yaml 2024-03-27T21:42:44.981 INFO:teuthology.orchestra.run.smithi060.stdout: 407352 4 -rw-r--r-- 1 ubuntu ubuntu 202 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/workloads/python_api_tests_with_defaults.yaml 2024-03-27T21:42:44.981 INFO:teuthology.orchestra.run.smithi060.stdout: 407350 4 -rw-r--r-- 1 ubuntu ubuntu 57 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/workloads/fsx.yaml 2024-03-27T21:42:44.981 INFO:teuthology.orchestra.run.smithi060.stdout: 407324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/clusters 2024-03-27T21:42:44.981 INFO:teuthology.orchestra.run.smithi060.stdout: 407327 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/clusters/fixed-3.yaml -> .qa/clusters/fixed-3.yaml 2024-03-27T21:42:44.981 INFO:teuthology.orchestra.run.smithi060.stdout: 407325 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/clusters/+ 2024-03-27T21:42:44.982 INFO:teuthology.orchestra.run.smithi060.stdout: 407326 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.982 INFO:teuthology.orchestra.run.smithi060.stdout: 407328 4 -rw-r--r-- 1 ubuntu ubuntu 87 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/clusters/openstack.yaml 2024-03-27T21:42:44.982 INFO:teuthology.orchestra.run.smithi060.stdout: 407330 0 lrwxrwxrwx 1 ubuntu ubuntu 18 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/data-pool -> .qa/rbd/data-pool/ 2024-03-27T21:42:44.982 INFO:teuthology.orchestra.run.smithi060.stdout: 407336 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/min-compat-client 2024-03-27T21:42:44.982 INFO:teuthology.orchestra.run.smithi060.stdout: 407337 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/min-compat-client/.qa -> ../.qa/ 2024-03-27T21:42:44.982 INFO:teuthology.orchestra.run.smithi060.stdout: 407339 4 -rw-r--r-- 1 ubuntu ubuntu 89 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/min-compat-client/octopus.yaml 2024-03-27T21:42:44.982 INFO:teuthology.orchestra.run.smithi060.stdout: 407338 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/min-compat-client/default.yaml 2024-03-27T21:42:44.983 INFO:teuthology.orchestra.run.smithi060.stdout: 407318 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/cache 2024-03-27T21:42:44.983 INFO:teuthology.orchestra.run.smithi060.stdout: 407319 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/cache/.qa -> ../.qa/ 2024-03-27T21:42:44.983 INFO:teuthology.orchestra.run.smithi060.stdout: 407323 4 -rw-r--r-- 1 ubuntu ubuntu 105 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/cache/writethrough.yaml 2024-03-27T21:42:44.983 INFO:teuthology.orchestra.run.smithi060.stdout: 407320 4 -rw-r--r-- 1 ubuntu ubuntu 75 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/cache/none.yaml 2024-03-27T21:42:44.983 INFO:teuthology.orchestra.run.smithi060.stdout: 407322 4 -rw-r--r-- 1 ubuntu ubuntu 110 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/cache/writeback.yaml 2024-03-27T21:42:44.983 INFO:teuthology.orchestra.run.smithi060.stdout: 407321 4 -rw-r--r-- 1 ubuntu ubuntu 112 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/librbd/cache/writearound.yaml 2024-03-27T21:42:44.983 INFO:teuthology.orchestra.run.smithi060.stdout: 407404 4 drwxr-xr-x 8 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash 2024-03-27T21:42:44.983 INFO:teuthology.orchestra.run.smithi060.stdout: 407405 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/% 2024-03-27T21:42:44.983 INFO:teuthology.orchestra.run.smithi060.stdout: 407410 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/clients 2024-03-27T21:42:44.984 INFO:teuthology.orchestra.run.smithi060.stdout: 407411 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/clients/.qa -> ../.qa/ 2024-03-27T21:42:44.984 INFO:teuthology.orchestra.run.smithi060.stdout: 407412 4 -rw-r--r-- 1 ubuntu ubuntu 2003 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/clients/mirror.yaml 2024-03-27T21:42:44.984 INFO:teuthology.orchestra.run.smithi060.stdout: 407420 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/objectstore -> .qa/objectstore 2024-03-27T21:42:44.984 INFO:teuthology.orchestra.run.smithi060.stdout: 407419 0 lrwxrwxrwx 1 ubuntu ubuntu 22 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/msgr-failures -> ../basic/msgr-failures 2024-03-27T21:42:44.984 INFO:teuthology.orchestra.run.smithi060.stdout: 407406 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/.qa -> ../.qa/ 2024-03-27T21:42:44.984 INFO:teuthology.orchestra.run.smithi060.stdout: 407418 0 lrwxrwxrwx 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/conf -> .qa/rbd/conf 2024-03-27T21:42:44.984 INFO:teuthology.orchestra.run.smithi060.stdout: 407428 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.984 INFO:teuthology.orchestra.run.smithi060.stdout: 407429 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/workloads 2024-03-27T21:42:44.985 INFO:teuthology.orchestra.run.smithi060.stdout: 407430 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.985 INFO:teuthology.orchestra.run.smithi060.stdout: 407433 4 -rw-r--r-- 1 ubuntu ubuntu 406 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/workloads/rbd-mirror-journal-workunit.yaml 2024-03-27T21:42:44.985 INFO:teuthology.orchestra.run.smithi060.stdout: 407436 4 -rw-r--r-- 1 ubuntu ubuntu 490 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/workloads/rbd-mirror-snapshot-stress-workunit-minimum.yaml 2024-03-27T21:42:44.985 INFO:teuthology.orchestra.run.smithi060.stdout: 407434 4 -rw-r--r-- 1 ubuntu ubuntu 505 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock.yaml 2024-03-27T21:42:44.985 INFO:teuthology.orchestra.run.smithi060.stdout: 407435 4 -rw-r--r-- 1 ubuntu ubuntu 526 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/workloads/rbd-mirror-snapshot-stress-workunit-fast-diff.yaml 2024-03-27T21:42:44.985 INFO:teuthology.orchestra.run.smithi060.stdout: 407431 4 -rw-r--r-- 1 ubuntu ubuntu 967 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/workloads/rbd-mirror-fsx-workunit.yaml 2024-03-27T21:42:44.985 INFO:teuthology.orchestra.run.smithi060.stdout: 407432 4 -rw-r--r-- 1 ubuntu ubuntu 451 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/workloads/rbd-mirror-journal-stress-workunit.yaml 2024-03-27T21:42:44.985 INFO:teuthology.orchestra.run.smithi060.stdout: 407425 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/rbd-mirror 2024-03-27T21:42:44.985 INFO:teuthology.orchestra.run.smithi060.stdout: 407427 4 -rw-r--r-- 1 ubuntu ubuntu 691 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/rbd-mirror/four-per-cluster.yaml 2024-03-27T21:42:44.986 INFO:teuthology.orchestra.run.smithi060.stdout: 407426 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/rbd-mirror/.qa -> ../.qa/ 2024-03-27T21:42:44.986 INFO:teuthology.orchestra.run.smithi060.stdout: 407421 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/policy 2024-03-27T21:42:44.986 INFO:teuthology.orchestra.run.smithi060.stdout: 407422 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/policy/.qa -> ../.qa/ 2024-03-27T21:42:44.986 INFO:teuthology.orchestra.run.smithi060.stdout: 407424 4 -rw-r--r-- 1 ubuntu ubuntu 88 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/policy/simple.yaml 2024-03-27T21:42:44.986 INFO:teuthology.orchestra.run.smithi060.stdout: 407423 4 -rw-r--r-- 1 ubuntu ubuntu 86 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/policy/none.yaml 2024-03-27T21:42:44.986 INFO:teuthology.orchestra.run.smithi060.stdout: 407407 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/base 2024-03-27T21:42:44.986 INFO:teuthology.orchestra.run.smithi060.stdout: 407409 4 -rw-r--r-- 1 ubuntu ubuntu 170 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/base/install.yaml 2024-03-27T21:42:44.986 INFO:teuthology.orchestra.run.smithi060.stdout: 407408 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/base/.qa -> ../.qa/ 2024-03-27T21:42:44.987 INFO:teuthology.orchestra.run.smithi060.stdout: 407413 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/cluster 2024-03-27T21:42:44.987 INFO:teuthology.orchestra.run.smithi060.stdout: 407414 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/cluster/+ 2024-03-27T21:42:44.987 INFO:teuthology.orchestra.run.smithi060.stdout: 407415 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/cluster/.qa -> ../.qa/ 2024-03-27T21:42:44.987 INFO:teuthology.orchestra.run.smithi060.stdout: 407417 4 -rw-r--r-- 1 ubuntu ubuntu 87 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/cluster/openstack.yaml 2024-03-27T21:42:44.987 INFO:teuthology.orchestra.run.smithi060.stdout: 407416 4 -rw-r--r-- 1 ubuntu ubuntu 758 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror-thrash/cluster/2-node.yaml 2024-03-27T21:42:44.987 INFO:teuthology.orchestra.run.smithi060.stdout: 407580 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton 2024-03-27T21:42:44.987 INFO:teuthology.orchestra.run.smithi060.stdout: 407583 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/all 2024-03-27T21:42:44.987 INFO:teuthology.orchestra.run.smithi060.stdout: 407602 4 -rw-r--r-- 1 ubuntu ubuntu 172 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/all/snap-diff.yaml 2024-03-27T21:42:44.987 INFO:teuthology.orchestra.run.smithi060.stdout: 407600 4 -rw-r--r-- 1 ubuntu ubuntu 226 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/all/read-flags-writeback.yaml 2024-03-27T21:42:44.988 INFO:teuthology.orchestra.run.smithi060.stdout: 407590 4 -rw-r--r-- 1 ubuntu ubuntu 334 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/all/qemu-iotests-no-cache.yaml 2024-03-27T21:42:44.988 INFO:teuthology.orchestra.run.smithi060.stdout: 407584 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/all/.qa -> ../.qa/ 2024-03-27T21:42:44.988 INFO:teuthology.orchestra.run.smithi060.stdout: 407596 4 -rw-r--r-- 1 ubuntu ubuntu 225 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/all/rbd_mirror.yaml 2024-03-27T21:42:44.988 INFO:teuthology.orchestra.run.smithi060.stdout: 407592 4 -rw-r--r-- 1 ubuntu ubuntu 369 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/all/qemu-iotests-writeback.yaml 2024-03-27T21:42:44.988 INFO:teuthology.orchestra.run.smithi060.stdout: 407589 4 -rw-r--r-- 1 ubuntu ubuntu 143 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/all/permissions.yaml 2024-03-27T21:42:44.988 INFO:teuthology.orchestra.run.smithi060.stdout: 407585 4 -rw-r--r-- 1 ubuntu ubuntu 149 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/all/admin_socket.yaml 2024-03-27T21:42:44.988 INFO:teuthology.orchestra.run.smithi060.stdout: 407598 4 -rw-r--r-- 1 ubuntu ubuntu 106 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/all/rbdmap_RBDMAPFILE.yaml 2024-03-27T21:42:44.988 INFO:teuthology.orchestra.run.smithi060.stdout: 407593 4 -rw-r--r-- 1 ubuntu ubuntu 364 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/all/qemu-iotests-writethrough.yaml 2024-03-27T21:42:44.989 INFO:teuthology.orchestra.run.smithi060.stdout: 407601 4 -rw-r--r-- 1 ubuntu ubuntu 221 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/all/read-flags-writethrough.yaml 2024-03-27T21:42:44.989 INFO:teuthology.orchestra.run.smithi060.stdout: 407586 4 -rw-r--r-- 1 ubuntu ubuntu 179 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/all/formatted-output.yaml 2024-03-27T21:42:44.989 INFO:teuthology.orchestra.run.smithi060.stdout: 407588 4 -rw-r--r-- 1 ubuntu ubuntu 179 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/all/mon-command-help.yaml 2024-03-27T21:42:44.989 INFO:teuthology.orchestra.run.smithi060.stdout: 407594 4 -rw-r--r-- 1 ubuntu ubuntu 135 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/all/qos.yaml 2024-03-27T21:42:44.989 INFO:teuthology.orchestra.run.smithi060.stdout: 407603 4 -rw-r--r-- 1 ubuntu ubuntu 143 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/all/verify_pool.yaml 2024-03-27T21:42:44.989 INFO:teuthology.orchestra.run.smithi060.stdout: 407587 4 -rw-r--r-- 1 ubuntu ubuntu 142 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/all/merge_diff.yaml 2024-03-27T21:42:44.989 INFO:teuthology.orchestra.run.smithi060.stdout: 407597 4 -rw-r--r-- 1 ubuntu ubuntu 146 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/all/rbd_tasks.yaml 2024-03-27T21:42:44.989 INFO:teuthology.orchestra.run.smithi060.stdout: 407595 4 -rw-r--r-- 1 ubuntu ubuntu 211 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/all/rbd-vs-unmanaged-snaps.yaml 2024-03-27T21:42:44.990 INFO:teuthology.orchestra.run.smithi060.stdout: 407591 4 -rw-r--r-- 1 ubuntu ubuntu 371 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/all/qemu-iotests-writearound.yaml 2024-03-27T21:42:44.990 INFO:teuthology.orchestra.run.smithi060.stdout: 407599 4 -rw-r--r-- 1 ubuntu ubuntu 191 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/all/read-flags-no-cache.yaml 2024-03-27T21:42:44.990 INFO:teuthology.orchestra.run.smithi060.stdout: 407581 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/% 2024-03-27T21:42:44.990 INFO:teuthology.orchestra.run.smithi060.stdout: 407605 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/objectstore -> .qa/objectstore 2024-03-27T21:42:44.990 INFO:teuthology.orchestra.run.smithi060.stdout: 407582 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/.qa -> ../.qa/ 2024-03-27T21:42:44.990 INFO:teuthology.orchestra.run.smithi060.stdout: 407606 4 -rw-r--r-- 1 ubuntu ubuntu 87 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/openstack.yaml 2024-03-27T21:42:44.990 INFO:teuthology.orchestra.run.smithi060.stdout: 407604 0 lrwxrwxrwx 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/conf -> .qa/rbd/conf 2024-03-27T21:42:44.990 INFO:teuthology.orchestra.run.smithi060.stdout: 407607 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.991 INFO:teuthology.orchestra.run.smithi060.stdout: 407567 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton-bluestore 2024-03-27T21:42:44.991 INFO:teuthology.orchestra.run.smithi060.stdout: 407570 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton-bluestore/all 2024-03-27T21:42:44.991 INFO:teuthology.orchestra.run.smithi060.stdout: 407571 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton-bluestore/all/.qa -> ../.qa/ 2024-03-27T21:42:44.991 INFO:teuthology.orchestra.run.smithi060.stdout: 407572 4 -rw-r--r-- 1 ubuntu ubuntu 257 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton-bluestore/all/issue-20295.yaml 2024-03-27T21:42:44.991 INFO:teuthology.orchestra.run.smithi060.stdout: 407568 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton-bluestore/% 2024-03-27T21:42:44.991 INFO:teuthology.orchestra.run.smithi060.stdout: 407574 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton-bluestore/objectstore 2024-03-27T21:42:44.991 INFO:teuthology.orchestra.run.smithi060.stdout: 407577 0 lrwxrwxrwx 1 ubuntu ubuntu 42 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton-bluestore/objectstore/bluestore-comp-snappy.yaml -> .qa/objectstore/bluestore-comp-snappy.yaml 2024-03-27T21:42:44.991 INFO:teuthology.orchestra.run.smithi060.stdout: 407575 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton-bluestore/objectstore/.qa -> ../.qa/ 2024-03-27T21:42:44.991 INFO:teuthology.orchestra.run.smithi060.stdout: 407576 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton-bluestore/objectstore/bluestore-bitmap.yaml -> .qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:44.992 INFO:teuthology.orchestra.run.smithi060.stdout: 407569 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton-bluestore/.qa -> ../.qa/ 2024-03-27T21:42:44.992 INFO:teuthology.orchestra.run.smithi060.stdout: 407578 4 -rw-r--r-- 1 ubuntu ubuntu 87 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton-bluestore/openstack.yaml 2024-03-27T21:42:44.992 INFO:teuthology.orchestra.run.smithi060.stdout: 407573 0 lrwxrwxrwx 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton-bluestore/conf -> .qa/rbd/conf 2024-03-27T21:42:44.992 INFO:teuthology.orchestra.run.smithi060.stdout: 407579 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/singleton-bluestore/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.992 INFO:teuthology.orchestra.run.smithi060.stdout: 407437 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror 2024-03-27T21:42:44.992 INFO:teuthology.orchestra.run.smithi060.stdout: 407438 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/% 2024-03-27T21:42:44.992 INFO:teuthology.orchestra.run.smithi060.stdout: 407441 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/clients 2024-03-27T21:42:44.992 INFO:teuthology.orchestra.run.smithi060.stdout: 407442 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/clients/+ 2024-03-27T21:42:44.992 INFO:teuthology.orchestra.run.smithi060.stdout: 407443 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/clients/.qa -> ../.qa/ 2024-03-27T21:42:44.993 INFO:teuthology.orchestra.run.smithi060.stdout: 407444 4 -rw-r--r-- 1 ubuntu ubuntu 1285 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/clients/mirror-extra.yaml 2024-03-27T21:42:44.993 INFO:teuthology.orchestra.run.smithi060.stdout: 407445 0 lrwxrwxrwx 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/clients/mirror.yaml -> ../../mirror-thrash/clients/mirror.yaml 2024-03-27T21:42:44.993 INFO:teuthology.orchestra.run.smithi060.stdout: 407449 0 lrwxrwxrwx 1 ubuntu ubuntu 28 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/objectstore -> ../mirror-thrash/objectstore 2024-03-27T21:42:44.993 INFO:teuthology.orchestra.run.smithi060.stdout: 407448 0 lrwxrwxrwx 1 ubuntu ubuntu 30 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/msgr-failures -> ../mirror-thrash/msgr-failures 2024-03-27T21:42:44.993 INFO:teuthology.orchestra.run.smithi060.stdout: 407439 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/.qa -> ../.qa/ 2024-03-27T21:42:44.993 INFO:teuthology.orchestra.run.smithi060.stdout: 407447 0 lrwxrwxrwx 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/conf -> .qa/rbd/conf 2024-03-27T21:42:44.993 INFO:teuthology.orchestra.run.smithi060.stdout: 407450 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.993 INFO:teuthology.orchestra.run.smithi060.stdout: 407451 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/workloads 2024-03-27T21:42:44.994 INFO:teuthology.orchestra.run.smithi060.stdout: 407452 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.994 INFO:teuthology.orchestra.run.smithi060.stdout: 407459 4 -rw-r--r-- 1 ubuntu ubuntu 406 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/workloads/rbd-mirror-snapshot-bootstrap-workunit.yaml 2024-03-27T21:42:44.994 INFO:teuthology.orchestra.run.smithi060.stdout: 407465 4 -rw-r--r-- 1 ubuntu ubuntu 402 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/workloads/rbd-mirror-workunit-config-key.yaml 2024-03-27T21:42:44.994 INFO:teuthology.orchestra.run.smithi060.stdout: 407456 4 -rw-r--r-- 1 ubuntu ubuntu 269 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/workloads/compare-mirror-images-nbd.yaml 2024-03-27T21:42:44.994 INFO:teuthology.orchestra.run.smithi060.stdout: 407461 4 -rw-r--r-- 1 ubuntu ubuntu 477 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/workloads/rbd-mirror-snapshot-workunit-fast-diff.yaml 2024-03-27T21:42:44.994 INFO:teuthology.orchestra.run.smithi060.stdout: 407453 4 -rw-r--r-- 1 ubuntu ubuntu 247 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/workloads/compare-mirror-image-alternate-primary-krbd.yaml 2024-03-27T21:42:44.994 INFO:teuthology.orchestra.run.smithi060.stdout: 407464 4 -rw-r--r-- 1 ubuntu ubuntu 348 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/workloads/rbd-mirror-stress-workunit-min-compat-client-octopus.yaml 2024-03-27T21:42:44.994 INFO:teuthology.orchestra.run.smithi060.stdout: 407467 4 -rw-r--r-- 1 ubuntu ubuntu 455 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/workloads/rbd-mirror-workunit-policy-none.yaml 2024-03-27T21:42:44.995 INFO:teuthology.orchestra.run.smithi060.stdout: 407466 4 -rw-r--r-- 1 ubuntu ubuntu 383 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/workloads/rbd-mirror-workunit-min-compat-client-octopus.yaml 2024-03-27T21:42:44.995 INFO:teuthology.orchestra.run.smithi060.stdout: 407454 4 -rw-r--r-- 1 ubuntu ubuntu 286 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/workloads/compare-mirror-image-alternate-primary-nbd.yaml 2024-03-27T21:42:44.995 INFO:teuthology.orchestra.run.smithi060.stdout: 407458 4 -rw-r--r-- 1 ubuntu ubuntu 404 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/workloads/rbd-mirror-journal-bootstrap-workunit.yaml 2024-03-27T21:42:44.995 INFO:teuthology.orchestra.run.smithi060.stdout: 407457 4 -rw-r--r-- 1 ubuntu ubuntu 396 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/workloads/rbd-mirror-ha-workunit.yaml 2024-03-27T21:42:44.995 INFO:teuthology.orchestra.run.smithi060.stdout: 407460 4 -rw-r--r-- 1 ubuntu ubuntu 456 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/workloads/rbd-mirror-snapshot-workunit-exclusive-lock.yaml 2024-03-27T21:42:44.995 INFO:teuthology.orchestra.run.smithi060.stdout: 407468 4 -rw-r--r-- 1 ubuntu ubuntu 457 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/workloads/rbd-mirror-workunit-policy-simple.yaml 2024-03-27T21:42:44.995 INFO:teuthology.orchestra.run.smithi060.stdout: 407462 4 -rw-r--r-- 1 ubuntu ubuntu 467 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/workloads/rbd-mirror-snapshot-workunit-journaling.yaml 2024-03-27T21:42:44.995 INFO:teuthology.orchestra.run.smithi060.stdout: 407463 4 -rw-r--r-- 1 ubuntu ubuntu 441 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/workloads/rbd-mirror-snapshot-workunit-minimum.yaml 2024-03-27T21:42:44.996 INFO:teuthology.orchestra.run.smithi060.stdout: 407455 4 -rw-r--r-- 1 ubuntu ubuntu 230 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/workloads/compare-mirror-images-krbd.yaml 2024-03-27T21:42:44.996 INFO:teuthology.orchestra.run.smithi060.stdout: 407440 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/base -> ../mirror-thrash/base 2024-03-27T21:42:44.996 INFO:teuthology.orchestra.run.smithi060.stdout: 407446 0 lrwxrwxrwx 1 ubuntu ubuntu 24 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/mirror/cluster -> ../mirror-thrash/cluster 2024-03-27T21:42:44.996 INFO:teuthology.orchestra.run.smithi060.stdout: 407608 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash 2024-03-27T21:42:44.996 INFO:teuthology.orchestra.run.smithi060.stdout: 407609 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/% 2024-03-27T21:42:44.996 INFO:teuthology.orchestra.run.smithi060.stdout: 407623 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/objectstore -> .qa/objectstore 2024-03-27T21:42:44.996 INFO:teuthology.orchestra.run.smithi060.stdout: 407620 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/msgr-failures 2024-03-27T21:42:44.996 INFO:teuthology.orchestra.run.smithi060.stdout: 407621 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:44.996 INFO:teuthology.orchestra.run.smithi060.stdout: 407622 4 -rw-r--r-- 1 ubuntu ubuntu 177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/msgr-failures/few.yaml 2024-03-27T21:42:44.997 INFO:teuthology.orchestra.run.smithi060.stdout: 407610 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/.qa -> ../.qa/ 2024-03-27T21:42:44.997 INFO:teuthology.orchestra.run.smithi060.stdout: 407619 0 lrwxrwxrwx 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/conf -> .qa/rbd/conf 2024-03-27T21:42:44.997 INFO:teuthology.orchestra.run.smithi060.stdout: 407628 0 lrwxrwxrwx 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/thrashosds-health.yaml -> .qa/tasks/thrashosds-health.yaml 2024-03-27T21:42:44.997 INFO:teuthology.orchestra.run.smithi060.stdout: 407624 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:44.997 INFO:teuthology.orchestra.run.smithi060.stdout: 407629 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/workloads 2024-03-27T21:42:44.997 INFO:teuthology.orchestra.run.smithi060.stdout: 407642 4 -rw-r--r-- 1 ubuntu ubuntu 124 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/workloads/rbd_fsx_nocache.yaml 2024-03-27T21:42:44.997 INFO:teuthology.orchestra.run.smithi060.stdout: 407630 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/workloads/.qa -> ../.qa/ 2024-03-27T21:42:44.997 INFO:teuthology.orchestra.run.smithi060.stdout: 407635 4 -rw-r--r-- 1 ubuntu ubuntu 261 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/workloads/rbd_api_tests_no_locking.yaml 2024-03-27T21:42:44.998 INFO:teuthology.orchestra.run.smithi060.stdout: 407639 4 -rw-r--r-- 1 ubuntu ubuntu 160 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/workloads/rbd_fsx_copy_on_read.yaml 2024-03-27T21:42:44.998 INFO:teuthology.orchestra.run.smithi060.stdout: 407634 4 -rw-r--r-- 1 ubuntu ubuntu 263 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/workloads/rbd_api_tests_journaling.yaml 2024-03-27T21:42:44.998 INFO:teuthology.orchestra.run.smithi060.stdout: 407637 4 -rw-r--r-- 1 ubuntu ubuntu 159 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/workloads/rbd_fsx_cache_writeback.yaml 2024-03-27T21:42:44.998 INFO:teuthology.orchestra.run.smithi060.stdout: 407631 4 -rw-r--r-- 1 ubuntu ubuntu 73 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/workloads/journal.yaml 2024-03-27T21:42:44.998 INFO:teuthology.orchestra.run.smithi060.stdout: 407633 4 -rw-r--r-- 1 ubuntu ubuntu 323 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/workloads/rbd_api_tests_copy_on_read.yaml 2024-03-27T21:42:44.998 INFO:teuthology.orchestra.run.smithi060.stdout: 407638 4 -rw-r--r-- 1 ubuntu ubuntu 154 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/workloads/rbd_fsx_cache_writethrough.yaml 2024-03-27T21:42:44.998 INFO:teuthology.orchestra.run.smithi060.stdout: 407636 4 -rw-r--r-- 1 ubuntu ubuntu 161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/workloads/rbd_fsx_cache_writearound.yaml 2024-03-27T21:42:44.998 INFO:teuthology.orchestra.run.smithi060.stdout: 407632 4 -rw-r--r-- 1 ubuntu ubuntu 262 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/workloads/rbd_api_tests.yaml 2024-03-27T21:42:44.998 INFO:teuthology.orchestra.run.smithi060.stdout: 407641 4 -rw-r--r-- 1 ubuntu ubuntu 81 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/workloads/rbd_fsx_journal.yaml 2024-03-27T21:42:44.999 INFO:teuthology.orchestra.run.smithi060.stdout: 407643 4 -rw-r--r-- 1 ubuntu ubuntu 201 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/workloads/rbd_fsx_rate_limit.yaml 2024-03-27T21:42:44.999 INFO:teuthology.orchestra.run.smithi060.stdout: 407640 4 -rw-r--r-- 1 ubuntu ubuntu 76 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/workloads/rbd_fsx_deep_copy.yaml 2024-03-27T21:42:44.999 INFO:teuthology.orchestra.run.smithi060.stdout: 407614 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/clusters 2024-03-27T21:42:44.999 INFO:teuthology.orchestra.run.smithi060.stdout: 407617 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/clusters/fixed-2.yaml -> .qa/clusters/fixed-2.yaml 2024-03-27T21:42:44.999 INFO:teuthology.orchestra.run.smithi060.stdout: 407615 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/clusters/+ 2024-03-27T21:42:44.999 INFO:teuthology.orchestra.run.smithi060.stdout: 407616 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/clusters/.qa -> ../.qa/ 2024-03-27T21:42:44.999 INFO:teuthology.orchestra.run.smithi060.stdout: 407618 4 -rw-r--r-- 1 ubuntu ubuntu 155 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/clusters/openstack.yaml 2024-03-27T21:42:44.999 INFO:teuthology.orchestra.run.smithi060.stdout: 407611 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/base 2024-03-27T21:42:45.000 INFO:teuthology.orchestra.run.smithi060.stdout: 407613 4 -rw-r--r-- 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/base/install.yaml 2024-03-27T21:42:45.000 INFO:teuthology.orchestra.run.smithi060.stdout: 407612 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/base/.qa -> ../.qa/ 2024-03-27T21:42:45.000 INFO:teuthology.orchestra.run.smithi060.stdout: 407625 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/thrashers 2024-03-27T21:42:45.000 INFO:teuthology.orchestra.run.smithi060.stdout: 407626 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/thrashers/.qa -> ../.qa/ 2024-03-27T21:42:45.000 INFO:teuthology.orchestra.run.smithi060.stdout: 407627 4 -rw-r--r-- 1 ubuntu ubuntu 150 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/thrash/thrashers/default.yaml 2024-03-27T21:42:45.000 INFO:teuthology.orchestra.run.smithi060.stdout: 407485 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache 2024-03-27T21:42:45.000 INFO:teuthology.orchestra.run.smithi060.stdout: 407486 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/.qa -> ../.qa/ 2024-03-27T21:42:45.000 INFO:teuthology.orchestra.run.smithi060.stdout: 407487 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home 2024-03-27T21:42:45.001 INFO:teuthology.orchestra.run.smithi060.stdout: 407488 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/% 2024-03-27T21:42:45.001 INFO:teuthology.orchestra.run.smithi060.stdout: 407489 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/.qa -> ../.qa/ 2024-03-27T21:42:45.001 INFO:teuthology.orchestra.run.smithi060.stdout: 407511 0 lrwxrwxrwx 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/conf -> .qa/rbd/conf 2024-03-27T21:42:45.001 INFO:teuthology.orchestra.run.smithi060.stdout: 407502 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/6-cache-size 2024-03-27T21:42:45.001 INFO:teuthology.orchestra.run.smithi060.stdout: 407503 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/6-cache-size/.qa -> ../.qa/ 2024-03-27T21:42:45.001 INFO:teuthology.orchestra.run.smithi060.stdout: 407505 4 -rw-r--r-- 1 ubuntu ubuntu 89 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/6-cache-size/8G.yaml 2024-03-27T21:42:45.001 INFO:teuthology.orchestra.run.smithi060.stdout: 407504 4 -rw-r--r-- 1 ubuntu ubuntu 89 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/6-cache-size/1G.yaml 2024-03-27T21:42:45.001 INFO:teuthology.orchestra.run.smithi060.stdout: 407490 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/1-base -> ../../basic/base/ 2024-03-27T21:42:45.001 INFO:teuthology.orchestra.run.smithi060.stdout: 407496 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/3-supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:45.002 INFO:teuthology.orchestra.run.smithi060.stdout: 407498 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/5-cache-mode 2024-03-27T21:42:45.002 INFO:teuthology.orchestra.run.smithi060.stdout: 407500 4 -rw-r--r-- 1 ubuntu ubuntu 82 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/5-cache-mode/rwl.yaml 2024-03-27T21:42:45.002 INFO:teuthology.orchestra.run.smithi060.stdout: 407499 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/5-cache-mode/.qa -> ../.qa/ 2024-03-27T21:42:45.002 INFO:teuthology.orchestra.run.smithi060.stdout: 407501 4 -rw-r--r-- 1 ubuntu ubuntu 82 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/5-cache-mode/ssd.yaml 2024-03-27T21:42:45.002 INFO:teuthology.orchestra.run.smithi060.stdout: 407506 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/7-workloads 2024-03-27T21:42:45.002 INFO:teuthology.orchestra.run.smithi060.stdout: 407507 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/7-workloads/.qa -> ../.qa/ 2024-03-27T21:42:45.002 INFO:teuthology.orchestra.run.smithi060.stdout: 407509 4 -rw-r--r-- 1 ubuntu ubuntu 135 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/7-workloads/fio.yaml 2024-03-27T21:42:45.002 INFO:teuthology.orchestra.run.smithi060.stdout: 407508 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/7-workloads/c_api_tests_with_defaults.yaml -> ../../../librbd/workloads/c_api_tests_with_defaults.yaml 2024-03-27T21:42:45.003 INFO:teuthology.orchestra.run.smithi060.stdout: 407510 4 -rw-r--r-- 1 ubuntu ubuntu 183 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/7-workloads/recovery.yaml 2024-03-27T21:42:45.003 INFO:teuthology.orchestra.run.smithi060.stdout: 407497 4 -rw-r--r-- 1 ubuntu ubuntu 319 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/4-cache-path.yaml 2024-03-27T21:42:45.003 INFO:teuthology.orchestra.run.smithi060.stdout: 407491 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/2-cluster 2024-03-27T21:42:45.003 INFO:teuthology.orchestra.run.smithi060.stdout: 407492 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/2-cluster/+ 2024-03-27T21:42:45.003 INFO:teuthology.orchestra.run.smithi060.stdout: 407493 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/2-cluster/.qa -> ../.qa/ 2024-03-27T21:42:45.003 INFO:teuthology.orchestra.run.smithi060.stdout: 407494 4 -rw-r--r-- 1 ubuntu ubuntu 79 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/2-cluster/fix-2.yaml 2024-03-27T21:42:45.003 INFO:teuthology.orchestra.run.smithi060.stdout: 407495 4 -rw-r--r-- 1 ubuntu ubuntu 93 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/home/2-cluster/openstack.yaml 2024-03-27T21:42:45.003 INFO:teuthology.orchestra.run.smithi060.stdout: 407512 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs 2024-03-27T21:42:45.004 INFO:teuthology.orchestra.run.smithi060.stdout: 407513 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/% 2024-03-27T21:42:45.004 INFO:teuthology.orchestra.run.smithi060.stdout: 407514 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/.qa -> ../.qa/ 2024-03-27T21:42:45.004 INFO:teuthology.orchestra.run.smithi060.stdout: 407534 0 lrwxrwxrwx 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/conf -> .qa/rbd/conf 2024-03-27T21:42:45.004 INFO:teuthology.orchestra.run.smithi060.stdout: 407527 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/6-cache-size 2024-03-27T21:42:45.004 INFO:teuthology.orchestra.run.smithi060.stdout: 407528 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/6-cache-size/.qa -> ../.qa/ 2024-03-27T21:42:45.004 INFO:teuthology.orchestra.run.smithi060.stdout: 407529 4 -rw-r--r-- 1 ubuntu ubuntu 89 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/6-cache-size/1G.yaml 2024-03-27T21:42:45.004 INFO:teuthology.orchestra.run.smithi060.stdout: 407530 4 -rw-r--r-- 1 ubuntu ubuntu 89 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/6-cache-size/5G.yaml 2024-03-27T21:42:45.004 INFO:teuthology.orchestra.run.smithi060.stdout: 407515 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/1-base -> ../../basic/base/ 2024-03-27T21:42:45.004 INFO:teuthology.orchestra.run.smithi060.stdout: 407521 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/3-supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:45.005 INFO:teuthology.orchestra.run.smithi060.stdout: 407523 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/5-cache-mode 2024-03-27T21:42:45.005 INFO:teuthology.orchestra.run.smithi060.stdout: 407525 4 -rw-r--r-- 1 ubuntu ubuntu 82 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/5-cache-mode/rwl.yaml 2024-03-27T21:42:45.005 INFO:teuthology.orchestra.run.smithi060.stdout: 407524 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/5-cache-mode/.qa -> ../.qa/ 2024-03-27T21:42:45.005 INFO:teuthology.orchestra.run.smithi060.stdout: 407526 4 -rw-r--r-- 1 ubuntu ubuntu 82 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/5-cache-mode/ssd.yaml 2024-03-27T21:42:45.005 INFO:teuthology.orchestra.run.smithi060.stdout: 407531 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/7-workloads 2024-03-27T21:42:45.005 INFO:teuthology.orchestra.run.smithi060.stdout: 407532 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/7-workloads/.qa -> ../.qa/ 2024-03-27T21:42:45.005 INFO:teuthology.orchestra.run.smithi060.stdout: 407533 4 -rw-r--r-- 1 ubuntu ubuntu 131 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/7-workloads/qemu_xfstests.yaml 2024-03-27T21:42:45.005 INFO:teuthology.orchestra.run.smithi060.stdout: 407522 4 -rw-r--r-- 1 ubuntu ubuntu 868 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/4-cache-path.yaml 2024-03-27T21:42:45.006 INFO:teuthology.orchestra.run.smithi060.stdout: 407516 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/2-cluster 2024-03-27T21:42:45.006 INFO:teuthology.orchestra.run.smithi060.stdout: 407517 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/2-cluster/+ 2024-03-27T21:42:45.006 INFO:teuthology.orchestra.run.smithi060.stdout: 407518 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/2-cluster/.qa -> ../.qa/ 2024-03-27T21:42:45.006 INFO:teuthology.orchestra.run.smithi060.stdout: 407519 4 -rw-r--r-- 1 ubuntu ubuntu 79 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/2-cluster/fix-2.yaml 2024-03-27T21:42:45.006 INFO:teuthology.orchestra.run.smithi060.stdout: 407520 4 -rw-r--r-- 1 ubuntu ubuntu 93 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/pwl-cache/tmpfs/2-cluster/openstack.yaml 2024-03-27T21:42:45.006 INFO:teuthology.orchestra.run.smithi060.stdout: 407156 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic 2024-03-27T21:42:45.006 INFO:teuthology.orchestra.run.smithi060.stdout: 407157 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/% 2024-03-27T21:42:45.006 INFO:teuthology.orchestra.run.smithi060.stdout: 407171 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/objectstore -> .qa/objectstore 2024-03-27T21:42:45.007 INFO:teuthology.orchestra.run.smithi060.stdout: 407168 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/msgr-failures 2024-03-27T21:42:45.007 INFO:teuthology.orchestra.run.smithi060.stdout: 407169 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/msgr-failures/.qa -> ../.qa/ 2024-03-27T21:42:45.007 INFO:teuthology.orchestra.run.smithi060.stdout: 407170 4 -rw-r--r-- 1 ubuntu ubuntu 177 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/msgr-failures/few.yaml 2024-03-27T21:42:45.007 INFO:teuthology.orchestra.run.smithi060.stdout: 407158 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/.qa -> ../.qa/ 2024-03-27T21:42:45.007 INFO:teuthology.orchestra.run.smithi060.stdout: 407167 0 lrwxrwxrwx 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/conf -> .qa/rbd/conf 2024-03-27T21:42:45.007 INFO:teuthology.orchestra.run.smithi060.stdout: 407172 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:45.007 INFO:teuthology.orchestra.run.smithi060.stdout: 407162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/clusters 2024-03-27T21:42:45.007 INFO:teuthology.orchestra.run.smithi060.stdout: 407165 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/clusters/fixed-1.yaml -> .qa/clusters/fixed-1.yaml 2024-03-27T21:42:45.008 INFO:teuthology.orchestra.run.smithi060.stdout: 407163 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/clusters/+ 2024-03-27T21:42:45.008 INFO:teuthology.orchestra.run.smithi060.stdout: 407164 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/clusters/.qa -> ../.qa/ 2024-03-27T21:42:45.008 INFO:teuthology.orchestra.run.smithi060.stdout: 407166 4 -rw-r--r-- 1 ubuntu ubuntu 87 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/clusters/openstack.yaml 2024-03-27T21:42:45.008 INFO:teuthology.orchestra.run.smithi060.stdout: 407159 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/base 2024-03-27T21:42:45.008 INFO:teuthology.orchestra.run.smithi060.stdout: 407161 4 -rw-r--r-- 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/base/install.yaml 2024-03-27T21:42:45.008 INFO:teuthology.orchestra.run.smithi060.stdout: 407160 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/base/.qa -> ../.qa/ 2024-03-27T21:42:45.008 INFO:teuthology.orchestra.run.smithi060.stdout: 407173 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/tasks 2024-03-27T21:42:45.008 INFO:teuthology.orchestra.run.smithi060.stdout: 407177 4 -rw-r--r-- 1 ubuntu ubuntu 81 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/tasks/rbd_lock_and_fence.yaml 2024-03-27T21:42:45.008 INFO:teuthology.orchestra.run.smithi060.stdout: 407174 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/tasks/.qa -> ../.qa/ 2024-03-27T21:42:45.009 INFO:teuthology.orchestra.run.smithi060.stdout: 407176 4 -rw-r--r-- 1 ubuntu ubuntu 143 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/tasks/rbd_cls_tests.yaml 2024-03-27T21:42:45.009 INFO:teuthology.orchestra.run.smithi060.stdout: 407175 4 -rw-r--r-- 1 ubuntu ubuntu 228 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/tasks/rbd_api_tests_old_format.yaml 2024-03-27T21:42:45.009 INFO:teuthology.orchestra.run.smithi060.stdout: 407178 4 -rw-r--r-- 1 ubuntu ubuntu 238 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/basic/tasks/rbd_python_api_tests_old_format.yaml 2024-03-27T21:42:45.009 INFO:teuthology.orchestra.run.smithi060.stdout: 407469 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/nvmeof 2024-03-27T21:42:45.009 INFO:teuthology.orchestra.run.smithi060.stdout: 407470 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/nvmeof/% 2024-03-27T21:42:45.009 INFO:teuthology.orchestra.run.smithi060.stdout: 407471 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/nvmeof/.qa -> ../.qa/ 2024-03-27T21:42:45.009 INFO:teuthology.orchestra.run.smithi060.stdout: 407481 0 lrwxrwxrwx 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/nvmeof/conf -> .qa/rbd/conf 2024-03-27T21:42:45.009 INFO:teuthology.orchestra.run.smithi060.stdout: 407482 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/nvmeof/workloads 2024-03-27T21:42:45.010 INFO:teuthology.orchestra.run.smithi060.stdout: 407483 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/nvmeof/workloads/.qa -> ../.qa/ 2024-03-27T21:42:45.010 INFO:teuthology.orchestra.run.smithi060.stdout: 407484 4 -rw-r--r-- 1 ubuntu ubuntu 929 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/nvmeof/workloads/nvmeof_initiator.yaml 2024-03-27T21:42:45.010 INFO:teuthology.orchestra.run.smithi060.stdout: 407475 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/nvmeof/centos_latest.yaml -> .qa/distros/supported/centos_latest.yaml 2024-03-27T21:42:45.010 INFO:teuthology.orchestra.run.smithi060.stdout: 407472 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/nvmeof/base 2024-03-27T21:42:45.010 INFO:teuthology.orchestra.run.smithi060.stdout: 407474 4 -rw-r--r-- 1 ubuntu ubuntu 232 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/nvmeof/base/install.yaml 2024-03-27T21:42:45.010 INFO:teuthology.orchestra.run.smithi060.stdout: 407473 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/nvmeof/base/.qa -> ../.qa/ 2024-03-27T21:42:45.010 INFO:teuthology.orchestra.run.smithi060.stdout: 407476 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/nvmeof/cluster 2024-03-27T21:42:45.010 INFO:teuthology.orchestra.run.smithi060.stdout: 407477 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/nvmeof/cluster/+ 2024-03-27T21:42:45.011 INFO:teuthology.orchestra.run.smithi060.stdout: 407478 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/nvmeof/cluster/.qa -> ../.qa/ 2024-03-27T21:42:45.011 INFO:teuthology.orchestra.run.smithi060.stdout: 407480 4 -rw-r--r-- 1 ubuntu ubuntu 155 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/nvmeof/cluster/openstack.yaml 2024-03-27T21:42:45.011 INFO:teuthology.orchestra.run.smithi060.stdout: 407479 4 -rw-r--r-- 1 ubuntu ubuntu 211 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/nvmeof/cluster/fixed-4.yaml 2024-03-27T21:42:45.011 INFO:teuthology.orchestra.run.smithi060.stdout: 407377 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration 2024-03-27T21:42:45.011 INFO:teuthology.orchestra.run.smithi060.stdout: 407378 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/% 2024-03-27T21:42:45.011 INFO:teuthology.orchestra.run.smithi060.stdout: 407379 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/.qa -> ../.qa/ 2024-03-27T21:42:45.011 INFO:teuthology.orchestra.run.smithi060.stdout: 407387 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/4-supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:45.011 INFO:teuthology.orchestra.run.smithi060.stdout: 407400 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/9-cleanup 2024-03-27T21:42:45.011 INFO:teuthology.orchestra.run.smithi060.stdout: 407402 4 -rw-r--r-- 1 ubuntu ubuntu 82 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/9-cleanup/cleanup.yaml 2024-03-27T21:42:45.012 INFO:teuthology.orchestra.run.smithi060.stdout: 407401 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/9-cleanup/.qa -> ../.qa/ 2024-03-27T21:42:45.012 INFO:teuthology.orchestra.run.smithi060.stdout: 407403 0 lrwxrwxrwx 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/conf -> .qa/rbd/conf 2024-03-27T21:42:45.012 INFO:teuthology.orchestra.run.smithi060.stdout: 407380 0 lrwxrwxrwx 1 ubuntu ubuntu 14 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/1-base -> ../thrash/base 2024-03-27T21:42:45.012 INFO:teuthology.orchestra.run.smithi060.stdout: 407386 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/3-objectstore -> .qa/objectstore 2024-03-27T21:42:45.012 INFO:teuthology.orchestra.run.smithi060.stdout: 407394 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/7-io-workloads 2024-03-27T21:42:45.012 INFO:teuthology.orchestra.run.smithi060.stdout: 407395 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/7-io-workloads/.qa -> ../.qa/ 2024-03-27T21:42:45.012 INFO:teuthology.orchestra.run.smithi060.stdout: 407396 4 -rw-r--r-- 1 ubuntu ubuntu 371 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/7-io-workloads/qemu_xfstests.yaml 2024-03-27T21:42:45.012 INFO:teuthology.orchestra.run.smithi060.stdout: 407388 0 lrwxrwxrwx 1 ubuntu ubuntu 18 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/5-data-pool -> .qa/rbd/data-pool/ 2024-03-27T21:42:45.013 INFO:teuthology.orchestra.run.smithi060.stdout: 407389 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/6-prepare 2024-03-27T21:42:45.013 INFO:teuthology.orchestra.run.smithi060.stdout: 407391 4 -rw-r--r-- 1 ubuntu ubuntu 822 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/6-prepare/qcow2-file.yaml 2024-03-27T21:42:45.013 INFO:teuthology.orchestra.run.smithi060.stdout: 407390 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/6-prepare/.qa -> ../.qa/ 2024-03-27T21:42:45.013 INFO:teuthology.orchestra.run.smithi060.stdout: 407393 4 -rw-r--r-- 1 ubuntu ubuntu 961 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/6-prepare/raw-file.yaml 2024-03-27T21:42:45.013 INFO:teuthology.orchestra.run.smithi060.stdout: 407392 4 -rw-r--r-- 1 ubuntu ubuntu 690 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/6-prepare/qcow2-http.yaml 2024-03-27T21:42:45.013 INFO:teuthology.orchestra.run.smithi060.stdout: 407397 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/8-migrate-workloads 2024-03-27T21:42:45.013 INFO:teuthology.orchestra.run.smithi060.stdout: 407399 4 -rw-r--r-- 1 ubuntu ubuntu 325 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/8-migrate-workloads/execute.yaml 2024-03-27T21:42:45.013 INFO:teuthology.orchestra.run.smithi060.stdout: 407398 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/8-migrate-workloads/.qa -> ../.qa/ 2024-03-27T21:42:45.013 INFO:teuthology.orchestra.run.smithi060.stdout: 407381 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/2-clusters 2024-03-27T21:42:45.014 INFO:teuthology.orchestra.run.smithi060.stdout: 407384 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/2-clusters/fixed-3.yaml -> .qa/clusters/fixed-3.yaml 2024-03-27T21:42:45.014 INFO:teuthology.orchestra.run.smithi060.stdout: 407382 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/2-clusters/+ 2024-03-27T21:42:45.014 INFO:teuthology.orchestra.run.smithi060.stdout: 407383 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/2-clusters/.qa -> ../.qa/ 2024-03-27T21:42:45.014 INFO:teuthology.orchestra.run.smithi060.stdout: 407385 4 -rw-r--r-- 1 ubuntu ubuntu 156 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rbd/migration/2-clusters/openstack.yaml 2024-03-27T21:42:45.014 INFO:teuthology.orchestra.run.smithi060.stdout: 406215 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/mixed-clients 2024-03-27T21:42:45.014 INFO:teuthology.orchestra.run.smithi060.stdout: 406216 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/mixed-clients/.qa -> ../.qa/ 2024-03-27T21:42:45.014 INFO:teuthology.orchestra.run.smithi060.stdout: 406217 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/mixed-clients/basic 2024-03-27T21:42:45.014 INFO:teuthology.orchestra.run.smithi060.stdout: 406222 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/mixed-clients/basic/objectstore -> .qa/objectstore 2024-03-27T21:42:45.015 INFO:teuthology.orchestra.run.smithi060.stdout: 406218 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/mixed-clients/basic/.qa -> ../.qa/ 2024-03-27T21:42:45.015 INFO:teuthology.orchestra.run.smithi060.stdout: 406219 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/mixed-clients/basic/clusters 2024-03-27T21:42:45.015 INFO:teuthology.orchestra.run.smithi060.stdout: 406221 4 -rw-r--r-- 1 ubuntu ubuntu 99 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/mixed-clients/basic/clusters/fixed-3.yaml 2024-03-27T21:42:45.015 INFO:teuthology.orchestra.run.smithi060.stdout: 406220 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/mixed-clients/basic/clusters/.qa -> ../.qa/ 2024-03-27T21:42:45.015 INFO:teuthology.orchestra.run.smithi060.stdout: 406223 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/mixed-clients/basic/tasks 2024-03-27T21:42:45.015 INFO:teuthology.orchestra.run.smithi060.stdout: 406224 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/mixed-clients/basic/tasks/.qa -> ../.qa/ 2024-03-27T21:42:45.015 INFO:teuthology.orchestra.run.smithi060.stdout: 406226 4 -rw-r--r-- 1 ubuntu ubuntu 459 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/mixed-clients/basic/tasks/kernel_cfuse_workunits_untarbuild_blogbench.yaml 2024-03-27T21:42:45.015 INFO:teuthology.orchestra.run.smithi060.stdout: 406225 4 -rw-r--r-- 1 ubuntu ubuntu 452 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/mixed-clients/basic/tasks/kernel_cfuse_workunits_dbench_iozone.yaml 2024-03-27T21:42:45.016 INFO:teuthology.orchestra.run.smithi060.stdout: 408122 4 drwxr-xr-x 9 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology 2024-03-27T21:42:45.016 INFO:teuthology.orchestra.run.smithi060.stdout: 408123 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/.qa -> ../.qa/ 2024-03-27T21:42:45.016 INFO:teuthology.orchestra.run.smithi060.stdout: 408171 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/rgw 2024-03-27T21:42:45.016 INFO:teuthology.orchestra.run.smithi060.stdout: 408172 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/rgw/% 2024-03-27T21:42:45.016 INFO:teuthology.orchestra.run.smithi060.stdout: 408173 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/rgw/.qa -> ../.qa/ 2024-03-27T21:42:45.016 INFO:teuthology.orchestra.run.smithi060.stdout: 408174 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/rgw/distros -> .qa/distros/supported 2024-03-27T21:42:45.016 INFO:teuthology.orchestra.run.smithi060.stdout: 408175 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/rgw/tasks 2024-03-27T21:42:45.016 INFO:teuthology.orchestra.run.smithi060.stdout: 408176 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/rgw/tasks/.qa -> ../.qa/ 2024-03-27T21:42:45.016 INFO:teuthology.orchestra.run.smithi060.stdout: 408177 4 -rw-r--r-- 1 ubuntu ubuntu 465 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/rgw/tasks/s3tests-fastcgi.yaml 2024-03-27T21:42:45.017 INFO:teuthology.orchestra.run.smithi060.stdout: 408178 4 -rw-r--r-- 1 ubuntu ubuntu 561 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/rgw/tasks/s3tests-fcgi.yaml 2024-03-27T21:42:45.017 INFO:teuthology.orchestra.run.smithi060.stdout: 408179 4 -rw-r--r-- 1 ubuntu ubuntu 425 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/rgw/tasks/s3tests.yaml 2024-03-27T21:42:45.017 INFO:teuthology.orchestra.run.smithi060.stdout: 408143 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/multi-cluster 2024-03-27T21:42:45.017 INFO:teuthology.orchestra.run.smithi060.stdout: 408146 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/multi-cluster/all 2024-03-27T21:42:45.017 INFO:teuthology.orchestra.run.smithi060.stdout: 408148 4 -rw-r--r-- 1 ubuntu ubuntu 386 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/multi-cluster/all/ceph.yaml 2024-03-27T21:42:45.017 INFO:teuthology.orchestra.run.smithi060.stdout: 408147 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/multi-cluster/all/.qa -> ../.qa/ 2024-03-27T21:42:45.017 INFO:teuthology.orchestra.run.smithi060.stdout: 408150 4 -rw-r--r-- 1 ubuntu ubuntu 1189 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/multi-cluster/all/upgrade.yaml 2024-03-27T21:42:45.017 INFO:teuthology.orchestra.run.smithi060.stdout: 408149 4 -rw-r--r-- 1 ubuntu ubuntu 323 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/multi-cluster/all/thrashosds.yaml 2024-03-27T21:42:45.017 INFO:teuthology.orchestra.run.smithi060.stdout: 408151 4 -rw-r--r-- 1 ubuntu ubuntu 330 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/multi-cluster/all/workunit.yaml 2024-03-27T21:42:45.018 INFO:teuthology.orchestra.run.smithi060.stdout: 408144 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/multi-cluster/% 2024-03-27T21:42:45.018 INFO:teuthology.orchestra.run.smithi060.stdout: 408145 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/multi-cluster/.qa -> ../.qa/ 2024-03-27T21:42:45.018 INFO:teuthology.orchestra.run.smithi060.stdout: 408142 4 -rw-r--r-- 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/integration.yaml 2024-03-27T21:42:45.018 INFO:teuthology.orchestra.run.smithi060.stdout: 408124 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/buildpackages 2024-03-27T21:42:45.018 INFO:teuthology.orchestra.run.smithi060.stdout: 408125 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/buildpackages/% 2024-03-27T21:42:45.018 INFO:teuthology.orchestra.run.smithi060.stdout: 408126 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/buildpackages/.qa -> ../.qa/ 2024-03-27T21:42:45.018 INFO:teuthology.orchestra.run.smithi060.stdout: 408127 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/buildpackages/supported-distros -> .qa/distros/supported 2024-03-27T21:42:45.019 INFO:teuthology.orchestra.run.smithi060.stdout: 408128 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/buildpackages/tasks 2024-03-27T21:42:45.019 INFO:teuthology.orchestra.run.smithi060.stdout: 408129 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/buildpackages/tasks/.qa -> ../.qa/ 2024-03-27T21:42:45.019 INFO:teuthology.orchestra.run.smithi060.stdout: 408131 4 -rw-r--r-- 1 ubuntu ubuntu 302 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/buildpackages/tasks/tag.yaml 2024-03-27T21:42:45.019 INFO:teuthology.orchestra.run.smithi060.stdout: 408130 4 -rw-r--r-- 1 ubuntu ubuntu 293 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/buildpackages/tasks/default.yaml 2024-03-27T21:42:45.019 INFO:teuthology.orchestra.run.smithi060.stdout: 408161 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/nop 2024-03-27T21:42:45.019 INFO:teuthology.orchestra.run.smithi060.stdout: 408162 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/nop/% 2024-03-27T21:42:45.019 INFO:teuthology.orchestra.run.smithi060.stdout: 408163 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/nop/.qa -> ../.qa/ 2024-03-27T21:42:45.019 INFO:teuthology.orchestra.run.smithi060.stdout: 408164 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/nop/clusters 2024-03-27T21:42:45.019 INFO:teuthology.orchestra.run.smithi060.stdout: 408165 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/nop/clusters/.qa -> ../.qa/ 2024-03-27T21:42:45.020 INFO:teuthology.orchestra.run.smithi060.stdout: 408167 4 -rw-r--r-- 1 ubuntu ubuntu 38 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/nop/clusters/single.yaml 2024-03-27T21:42:45.020 INFO:teuthology.orchestra.run.smithi060.stdout: 408166 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/nop/clusters/empty.yaml 2024-03-27T21:42:45.020 INFO:teuthology.orchestra.run.smithi060.stdout: 408168 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/nop/tasks 2024-03-27T21:42:45.020 INFO:teuthology.orchestra.run.smithi060.stdout: 408169 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/nop/tasks/.qa -> ../.qa/ 2024-03-27T21:42:45.020 INFO:teuthology.orchestra.run.smithi060.stdout: 408170 4 -rw-r--r-- 1 ubuntu ubuntu 19 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/nop/tasks/nop.yaml 2024-03-27T21:42:45.020 INFO:teuthology.orchestra.run.smithi060.stdout: 408132 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/ceph 2024-03-27T21:42:45.020 INFO:teuthology.orchestra.run.smithi060.stdout: 408133 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/ceph/% 2024-03-27T21:42:45.020 INFO:teuthology.orchestra.run.smithi060.stdout: 408134 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/ceph/.qa -> ../.qa/ 2024-03-27T21:42:45.020 INFO:teuthology.orchestra.run.smithi060.stdout: 408135 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/ceph/clusters 2024-03-27T21:42:45.021 INFO:teuthology.orchestra.run.smithi060.stdout: 408136 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/ceph/clusters/.qa -> ../.qa/ 2024-03-27T21:42:45.021 INFO:teuthology.orchestra.run.smithi060.stdout: 408137 4 -rw-r--r-- 1 ubuntu ubuntu 38 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/ceph/clusters/single.yaml 2024-03-27T21:42:45.021 INFO:teuthology.orchestra.run.smithi060.stdout: 408138 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/ceph/distros -> .qa/distros/supported 2024-03-27T21:42:45.021 INFO:teuthology.orchestra.run.smithi060.stdout: 408139 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/ceph/tasks 2024-03-27T21:42:45.021 INFO:teuthology.orchestra.run.smithi060.stdout: 408140 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/ceph/tasks/.qa -> ../.qa/ 2024-03-27T21:42:45.021 INFO:teuthology.orchestra.run.smithi060.stdout: 408141 4 -rw-r--r-- 1 ubuntu ubuntu 35 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/ceph/tasks/teuthology.yaml 2024-03-27T21:42:45.021 INFO:teuthology.orchestra.run.smithi060.stdout: 408180 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/workunits 2024-03-27T21:42:45.021 INFO:teuthology.orchestra.run.smithi060.stdout: 408181 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/workunits/.qa -> ../.qa/ 2024-03-27T21:42:45.021 INFO:teuthology.orchestra.run.smithi060.stdout: 408182 4 -rw-r--r-- 1 ubuntu ubuntu 96 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/workunits/yes.yaml 2024-03-27T21:42:45.022 INFO:teuthology.orchestra.run.smithi060.stdout: 408152 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/no-ceph 2024-03-27T21:42:45.022 INFO:teuthology.orchestra.run.smithi060.stdout: 408153 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/no-ceph/% 2024-03-27T21:42:45.022 INFO:teuthology.orchestra.run.smithi060.stdout: 408154 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/no-ceph/.qa -> ../.qa/ 2024-03-27T21:42:45.022 INFO:teuthology.orchestra.run.smithi060.stdout: 408155 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/no-ceph/clusters 2024-03-27T21:42:45.022 INFO:teuthology.orchestra.run.smithi060.stdout: 408156 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/no-ceph/clusters/.qa -> ../.qa/ 2024-03-27T21:42:45.022 INFO:teuthology.orchestra.run.smithi060.stdout: 408157 4 -rw-r--r-- 1 ubuntu ubuntu 38 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/no-ceph/clusters/single.yaml 2024-03-27T21:42:45.022 INFO:teuthology.orchestra.run.smithi060.stdout: 408158 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/no-ceph/tasks 2024-03-27T21:42:45.022 INFO:teuthology.orchestra.run.smithi060.stdout: 408159 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/no-ceph/tasks/.qa -> ../.qa/ 2024-03-27T21:42:45.023 INFO:teuthology.orchestra.run.smithi060.stdout: 408160 4 -rw-r--r-- 1 ubuntu ubuntu 20 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/teuthology/no-ceph/tasks/teuthology.yaml 2024-03-27T21:42:45.023 INFO:teuthology.orchestra.run.smithi060.stdout: 404717 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental 2024-03-27T21:42:45.023 INFO:teuthology.orchestra.run.smithi060.stdout: 404718 0 lrwxrwxrwx 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/.qa -> ../.qa 2024-03-27T21:42:45.023 INFO:teuthology.orchestra.run.smithi060.stdout: 404719 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/seastore 2024-03-27T21:42:45.023 INFO:teuthology.orchestra.run.smithi060.stdout: 404720 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/seastore/.qa -> ../.qa/ 2024-03-27T21:42:45.023 INFO:teuthology.orchestra.run.smithi060.stdout: 404721 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/seastore/basic 2024-03-27T21:42:45.023 INFO:teuthology.orchestra.run.smithi060.stdout: 404722 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/seastore/basic/% 2024-03-27T21:42:45.023 INFO:teuthology.orchestra.run.smithi060.stdout: 404733 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/seastore/basic/objectstore 2024-03-27T21:42:45.023 INFO:teuthology.orchestra.run.smithi060.stdout: 404734 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/seastore/basic/objectstore/.qa -> ../.qa/ 2024-03-27T21:42:45.024 INFO:teuthology.orchestra.run.smithi060.stdout: 404735 4 -rw-r--r-- 1 ubuntu ubuntu 86 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/seastore/basic/objectstore/seastore.yaml 2024-03-27T21:42:45.024 INFO:teuthology.orchestra.run.smithi060.stdout: 404723 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/seastore/basic/.qa -> ../.qa/ 2024-03-27T21:42:45.024 INFO:teuthology.orchestra.run.smithi060.stdout: 404730 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/seastore/basic/deploy 2024-03-27T21:42:45.024 INFO:teuthology.orchestra.run.smithi060.stdout: 404732 4 -rw-r--r-- 1 ubuntu ubuntu 383 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/seastore/basic/deploy/ceph.yaml 2024-03-27T21:42:45.024 INFO:teuthology.orchestra.run.smithi060.stdout: 404731 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/seastore/basic/deploy/.qa -> ../.qa/ 2024-03-27T21:42:45.024 INFO:teuthology.orchestra.run.smithi060.stdout: 404725 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/seastore/basic/clusters 2024-03-27T21:42:45.024 INFO:teuthology.orchestra.run.smithi060.stdout: 404727 4 -rw-r--r-- 1 ubuntu ubuntu 321 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/seastore/basic/clusters/fixed-1.yaml 2024-03-27T21:42:45.024 INFO:teuthology.orchestra.run.smithi060.stdout: 404728 4 -rw-r--r-- 1 ubuntu ubuntu 401 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/seastore/basic/clusters/fixed-2.yaml 2024-03-27T21:42:45.024 INFO:teuthology.orchestra.run.smithi060.stdout: 404726 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/seastore/basic/clusters/.qa -> ../.qa/ 2024-03-27T21:42:45.025 INFO:teuthology.orchestra.run.smithi060.stdout: 404729 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/seastore/basic/crimson_qa_overrides.yaml -> .qa/config/crimson_qa_overrides.yaml 2024-03-27T21:42:45.025 INFO:teuthology.orchestra.run.smithi060.stdout: 404724 0 lrwxrwxrwx 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/seastore/basic/centos_latest.yaml -> .qa/distros/supported/centos_latest.yaml 2024-03-27T21:42:45.025 INFO:teuthology.orchestra.run.smithi060.stdout: 404736 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/seastore/basic/tasks 2024-03-27T21:42:45.025 INFO:teuthology.orchestra.run.smithi060.stdout: 404737 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/seastore/basic/tasks/.qa -> ../.qa/ 2024-03-27T21:42:45.025 INFO:teuthology.orchestra.run.smithi060.stdout: 404738 4 -rw-r--r-- 1 ubuntu ubuntu 625 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/seastore/basic/tasks/rados_api_tests.yaml 2024-03-27T21:42:45.025 INFO:teuthology.orchestra.run.smithi060.stdout: 404739 4 -rw-r--r-- 1 ubuntu ubuntu 318 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/crimson-rados-experimental/seastore/basic/tasks/readwrite.yaml 2024-03-27T21:42:45.025 INFO:teuthology.orchestra.run.smithi060.stdout: 406238 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch 2024-03-27T21:42:45.025 INFO:teuthology.orchestra.run.smithi060.stdout: 406239 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/.qa -> ../.qa/ 2024-03-27T21:42:45.026 INFO:teuthology.orchestra.run.smithi060.stdout: 406427 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook 2024-03-27T21:42:45.026 INFO:teuthology.orchestra.run.smithi060.stdout: 406428 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/.qa -> ../.qa/ 2024-03-27T21:42:45.026 INFO:teuthology.orchestra.run.smithi060.stdout: 406429 4 drwxr-xr-x 8 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke 2024-03-27T21:42:45.026 INFO:teuthology.orchestra.run.smithi060.stdout: 406436 0 lrwxrwxrwx 1 ubuntu ubuntu 28 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/0-nvme-loop.yaml -> .qa/overrides/nvme_loop.yaml 2024-03-27T21:42:45.026 INFO:teuthology.orchestra.run.smithi060.stdout: 406430 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/% 2024-03-27T21:42:45.026 INFO:teuthology.orchestra.run.smithi060.stdout: 406431 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/.qa -> ../.qa/ 2024-03-27T21:42:45.026 INFO:teuthology.orchestra.run.smithi060.stdout: 406454 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/rook 2024-03-27T21:42:45.026 INFO:teuthology.orchestra.run.smithi060.stdout: 406456 4 -rw-r--r-- 1 ubuntu ubuntu 76 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/rook/1.7.2.yaml 2024-03-27T21:42:45.027 INFO:teuthology.orchestra.run.smithi060.stdout: 406455 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/rook/.qa -> ../.qa/ 2024-03-27T21:42:45.027 INFO:teuthology.orchestra.run.smithi060.stdout: 406457 4 -rw-r--r-- 1 ubuntu ubuntu 52 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/rook/master.yaml 2024-03-27T21:42:45.027 INFO:teuthology.orchestra.run.smithi060.stdout: 406438 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/2-workload 2024-03-27T21:42:45.027 INFO:teuthology.orchestra.run.smithi060.stdout: 406439 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/2-workload/.qa -> ../.qa/ 2024-03-27T21:42:45.027 INFO:teuthology.orchestra.run.smithi060.stdout: 406441 4 -rw-r--r-- 1 ubuntu ubuntu 68 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/2-workload/radosbench.yaml 2024-03-27T21:42:45.027 INFO:teuthology.orchestra.run.smithi060.stdout: 406440 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/2-workload/none.yaml 2024-03-27T21:42:45.027 INFO:teuthology.orchestra.run.smithi060.stdout: 406449 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/net 2024-03-27T21:42:45.027 INFO:teuthology.orchestra.run.smithi060.stdout: 406450 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/net/.qa -> ../.qa/ 2024-03-27T21:42:45.027 INFO:teuthology.orchestra.run.smithi060.stdout: 406451 4 -rw-r--r-- 1 ubuntu ubuntu 46 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/net/calico.yaml 2024-03-27T21:42:45.028 INFO:teuthology.orchestra.run.smithi060.stdout: 406452 4 -rw-r--r-- 1 ubuntu ubuntu 47 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/net/flannel.yaml 2024-03-27T21:42:45.028 INFO:teuthology.orchestra.run.smithi060.stdout: 406453 4 -rw-r--r-- 1 ubuntu ubuntu 114 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/net/host.yaml 2024-03-27T21:42:45.028 INFO:teuthology.orchestra.run.smithi060.stdout: 406437 4 -rw-r--r-- 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/1-rook.yaml 2024-03-27T21:42:45.028 INFO:teuthology.orchestra.run.smithi060.stdout: 406442 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/cluster 2024-03-27T21:42:45.028 INFO:teuthology.orchestra.run.smithi060.stdout: 406443 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/cluster/.qa -> ../.qa/ 2024-03-27T21:42:45.028 INFO:teuthology.orchestra.run.smithi060.stdout: 406444 4 -rw-r--r-- 1 ubuntu ubuntu 112 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/cluster/1-node.yaml 2024-03-27T21:42:45.028 INFO:teuthology.orchestra.run.smithi060.stdout: 406445 4 -rw-r--r-- 1 ubuntu ubuntu 79 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/cluster/3-node.yaml 2024-03-27T21:42:45.028 INFO:teuthology.orchestra.run.smithi060.stdout: 406432 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/0-distro 2024-03-27T21:42:45.029 INFO:teuthology.orchestra.run.smithi060.stdout: 406433 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/0-distro/.qa -> ../.qa/ 2024-03-27T21:42:45.029 INFO:teuthology.orchestra.run.smithi060.stdout: 406434 0 lrwxrwxrwx 1 ubuntu ubuntu 45 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/0-distro/ubuntu_22.04.yaml -> .qa/distros/container-hosts/ubuntu_22.04.yaml 2024-03-27T21:42:45.029 INFO:teuthology.orchestra.run.smithi060.stdout: 406446 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/k8s 2024-03-27T21:42:45.029 INFO:teuthology.orchestra.run.smithi060.stdout: 406447 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/k8s/.qa -> ../.qa/ 2024-03-27T21:42:45.029 INFO:teuthology.orchestra.run.smithi060.stdout: 406448 4 -rw-r--r-- 1 ubuntu ubuntu 42 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/k8s/1.21.yaml 2024-03-27T21:42:45.029 INFO:teuthology.orchestra.run.smithi060.stdout: 406435 4 -rw-r--r-- 1 ubuntu ubuntu 18 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/rook/smoke/0-kubeadm.yaml 2024-03-27T21:42:45.029 INFO:teuthology.orchestra.run.smithi060.stdout: 406240 4 drwxr-xr-x 15 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm 2024-03-27T21:42:45.029 INFO:teuthology.orchestra.run.smithi060.stdout: 406241 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/.qa -> ../.qa/ 2024-03-27T21:42:45.029 INFO:teuthology.orchestra.run.smithi060.stdout: 406319 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-singlehost 2024-03-27T21:42:45.030 INFO:teuthology.orchestra.run.smithi060.stdout: 406322 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-singlehost/0-random-distro$ -> .qa/distros/supported-container-hosts 2024-03-27T21:42:45.030 INFO:teuthology.orchestra.run.smithi060.stdout: 406320 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-singlehost/% 2024-03-27T21:42:45.030 INFO:teuthology.orchestra.run.smithi060.stdout: 406321 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-singlehost/.qa -> ../.qa/ 2024-03-27T21:42:45.030 INFO:teuthology.orchestra.run.smithi060.stdout: 406324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-singlehost/2-services 2024-03-27T21:42:45.030 INFO:teuthology.orchestra.run.smithi060.stdout: 406327 4 -rw-r--r-- 1 ubuntu ubuntu 257 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-singlehost/2-services/rgw.yaml 2024-03-27T21:42:45.030 INFO:teuthology.orchestra.run.smithi060.stdout: 406325 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-singlehost/2-services/.qa -> ../.qa/ 2024-03-27T21:42:45.030 INFO:teuthology.orchestra.run.smithi060.stdout: 406326 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-singlehost/2-services/basic.yaml 2024-03-27T21:42:45.030 INFO:teuthology.orchestra.run.smithi060.stdout: 406323 4 -rw-r--r-- 1 ubuntu ubuntu 450 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-singlehost/1-start.yaml 2024-03-27T21:42:45.031 INFO:teuthology.orchestra.run.smithi060.stdout: 406328 4 -rw-r--r-- 1 ubuntu ubuntu 157 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-singlehost/3-final.yaml 2024-03-27T21:42:45.031 INFO:teuthology.orchestra.run.smithi060.stdout: 406272 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/osds 2024-03-27T21:42:45.031 INFO:teuthology.orchestra.run.smithi060.stdout: 406276 0 lrwxrwxrwx 1 ubuntu ubuntu 28 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/osds/0-nvme-loop.yaml -> .qa/overrides/nvme_loop.yaml 2024-03-27T21:42:45.031 INFO:teuthology.orchestra.run.smithi060.stdout: 406273 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/osds/% 2024-03-27T21:42:45.031 INFO:teuthology.orchestra.run.smithi060.stdout: 406274 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/osds/.qa -> ../.qa/ 2024-03-27T21:42:45.031 INFO:teuthology.orchestra.run.smithi060.stdout: 406277 4 -rw-r--r-- 1 ubuntu ubuntu 461 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/osds/1-start.yaml 2024-03-27T21:42:45.031 INFO:teuthology.orchestra.run.smithi060.stdout: 406278 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/osds/2-ops 2024-03-27T21:42:45.031 INFO:teuthology.orchestra.run.smithi060.stdout: 406284 4 -rw-r--r-- 1 ubuntu ubuntu 678 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/osds/2-ops/rmdir-reactivate.yaml 2024-03-27T21:42:45.032 INFO:teuthology.orchestra.run.smithi060.stdout: 406279 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/osds/2-ops/.qa -> ../.qa/ 2024-03-27T21:42:45.032 INFO:teuthology.orchestra.run.smithi060.stdout: 406280 4 -rw-r--r-- 1 ubuntu ubuntu 326 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/osds/2-ops/repave-all.yaml 2024-03-27T21:42:45.032 INFO:teuthology.orchestra.run.smithi060.stdout: 406281 4 -rw-r--r-- 1 ubuntu ubuntu 636 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/osds/2-ops/rm-zap-add.yaml 2024-03-27T21:42:45.032 INFO:teuthology.orchestra.run.smithi060.stdout: 406283 4 -rw-r--r-- 1 ubuntu ubuntu 592 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/osds/2-ops/rm-zap-wait.yaml 2024-03-27T21:42:45.032 INFO:teuthology.orchestra.run.smithi060.stdout: 406282 4 -rw-r--r-- 1 ubuntu ubuntu 571 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/osds/2-ops/rm-zap-flag.yaml 2024-03-27T21:42:45.032 INFO:teuthology.orchestra.run.smithi060.stdout: 406275 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/osds/0-distro -> .qa/distros/supported-container-hosts 2024-03-27T21:42:45.032 INFO:teuthology.orchestra.run.smithi060.stdout: 406286 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smb 2024-03-27T21:42:45.032 INFO:teuthology.orchestra.run.smithi060.stdout: 406287 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smb/% 2024-03-27T21:42:45.032 INFO:teuthology.orchestra.run.smithi060.stdout: 406288 0 lrwxrwxrwx 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smb/.qa -> ../.qa 2024-03-27T21:42:45.033 INFO:teuthology.orchestra.run.smithi060.stdout: 406290 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smb/tasks 2024-03-27T21:42:45.033 INFO:teuthology.orchestra.run.smithi060.stdout: 406291 0 lrwxrwxrwx 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smb/tasks/.qa -> ../.qa 2024-03-27T21:42:45.033 INFO:teuthology.orchestra.run.smithi060.stdout: 406293 4 -rw-r--r-- 1 ubuntu ubuntu 2851 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smb/tasks/deploy_smb_domain.yaml 2024-03-27T21:42:45.033 INFO:teuthology.orchestra.run.smithi060.stdout: 406292 4 -rw-r--r-- 1 ubuntu ubuntu 2662 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smb/tasks/deploy_smb_basic.yaml 2024-03-27T21:42:45.033 INFO:teuthology.orchestra.run.smithi060.stdout: 406289 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smb/0-distro -> .qa/distros/supported-container-hosts 2024-03-27T21:42:45.033 INFO:teuthology.orchestra.run.smithi060.stdout: 406243 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/mgr-nfs-upgrade 2024-03-27T21:42:45.033 INFO:teuthology.orchestra.run.smithi060.stdout: 406246 0 lrwxrwxrwx 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/mgr-nfs-upgrade/0-centos_9.stream.yaml -> .qa/distros/podman/centos_9.stream.yaml 2024-03-27T21:42:45.033 INFO:teuthology.orchestra.run.smithi060.stdout: 406253 4 -rw-r--r-- 1 ubuntu ubuntu 325 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/mgr-nfs-upgrade/4-final.yaml 2024-03-27T21:42:45.033 INFO:teuthology.orchestra.run.smithi060.stdout: 406244 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/mgr-nfs-upgrade/% 2024-03-27T21:42:45.034 INFO:teuthology.orchestra.run.smithi060.stdout: 406245 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/mgr-nfs-upgrade/.qa -> ../.qa/ 2024-03-27T21:42:45.034 INFO:teuthology.orchestra.run.smithi060.stdout: 406252 4 -rw-r--r-- 1 ubuntu ubuntu 1576 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/mgr-nfs-upgrade/3-upgrade-with-workload.yaml 2024-03-27T21:42:45.034 INFO:teuthology.orchestra.run.smithi060.stdout: 406250 4 -rw-r--r-- 1 ubuntu ubuntu 440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/mgr-nfs-upgrade/1-start.yaml 2024-03-27T21:42:45.034 INFO:teuthology.orchestra.run.smithi060.stdout: 406247 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/mgr-nfs-upgrade/1-bootstrap 2024-03-27T21:42:45.034 INFO:teuthology.orchestra.run.smithi060.stdout: 406248 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/mgr-nfs-upgrade/1-bootstrap/.qa -> ../.qa/ 2024-03-27T21:42:45.034 INFO:teuthology.orchestra.run.smithi060.stdout: 406249 4 -rw-r--r-- 1 ubuntu ubuntu 152 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/mgr-nfs-upgrade/1-bootstrap/17.2.0.yaml 2024-03-27T21:42:45.034 INFO:teuthology.orchestra.run.smithi060.stdout: 406251 4 -rw-r--r-- 1 ubuntu ubuntu 879 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/mgr-nfs-upgrade/2-nfs.yaml 2024-03-27T21:42:45.034 INFO:teuthology.orchestra.run.smithi060.stdout: 406255 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/no-agent-workunits 2024-03-27T21:42:45.035 INFO:teuthology.orchestra.run.smithi060.stdout: 406256 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/no-agent-workunits/% 2024-03-27T21:42:45.035 INFO:teuthology.orchestra.run.smithi060.stdout: 406257 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/no-agent-workunits/.qa -> ../.qa/ 2024-03-27T21:42:45.035 INFO:teuthology.orchestra.run.smithi060.stdout: 406260 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/no-agent-workunits/task 2024-03-27T21:42:45.035 INFO:teuthology.orchestra.run.smithi060.stdout: 406263 4 -rw-r--r-- 1 ubuntu ubuntu 241 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/no-agent-workunits/task/test_orch_cli.yaml 2024-03-27T21:42:45.035 INFO:teuthology.orchestra.run.smithi060.stdout: 406262 4 -rw-r--r-- 1 ubuntu ubuntu 172 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/no-agent-workunits/task/test_cephadm_timeout.yaml 2024-03-27T21:42:45.035 INFO:teuthology.orchestra.run.smithi060.stdout: 406261 4 -rw-r--r-- 1 ubuntu ubuntu 211 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/no-agent-workunits/task/test_adoption.yaml 2024-03-27T21:42:45.035 INFO:teuthology.orchestra.run.smithi060.stdout: 406264 4 -rw-r--r-- 1 ubuntu ubuntu 602 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/no-agent-workunits/task/test_orch_cli_mon.yaml 2024-03-27T21:42:45.035 INFO:teuthology.orchestra.run.smithi060.stdout: 406259 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/no-agent-workunits/mon_election -> .qa/mon_election 2024-03-27T21:42:45.035 INFO:teuthology.orchestra.run.smithi060.stdout: 406258 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/no-agent-workunits/0-distro -> .qa/distros/supported-container-hosts 2024-03-27T21:42:45.036 INFO:teuthology.orchestra.run.smithi060.stdout: 406370 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/upgrade 2024-03-27T21:42:45.036 INFO:teuthology.orchestra.run.smithi060.stdout: 406387 0 lrwxrwxrwx 1 ubuntu ubuntu 14 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/upgrade/agent -> ../smoke/agent 2024-03-27T21:42:45.036 INFO:teuthology.orchestra.run.smithi060.stdout: 406386 4 -rw-r--r-- 1 ubuntu ubuntu 230 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/upgrade/5-upgrade-ls.yaml 2024-03-27T21:42:45.036 INFO:teuthology.orchestra.run.smithi060.stdout: 406371 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/upgrade/% 2024-03-27T21:42:45.036 INFO:teuthology.orchestra.run.smithi060.stdout: 406372 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/upgrade/.qa -> ../.qa/ 2024-03-27T21:42:45.036 INFO:teuthology.orchestra.run.smithi060.stdout: 406385 4 -rw-r--r-- 1 ubuntu ubuntu 666 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/upgrade/4-wait.yaml 2024-03-27T21:42:45.036 INFO:teuthology.orchestra.run.smithi060.stdout: 406373 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/upgrade/1-start-distro 2024-03-27T21:42:45.036 INFO:teuthology.orchestra.run.smithi060.stdout: 406374 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/upgrade/1-start-distro/.qa -> ../.qa/ 2024-03-27T21:42:45.037 INFO:teuthology.orchestra.run.smithi060.stdout: 406375 4 -rw-r--r-- 1 ubuntu ubuntu 427 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/upgrade/1-start-distro/1-start-centos_9.stream.yaml 2024-03-27T21:42:45.037 INFO:teuthology.orchestra.run.smithi060.stdout: 406376 4 -rw-r--r-- 1 ubuntu ubuntu 424 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/upgrade/1-start-distro/1-start-ubuntu_20.04.yaml 2024-03-27T21:42:45.037 INFO:teuthology.orchestra.run.smithi060.stdout: 406381 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/upgrade/3-upgrade 2024-03-27T21:42:45.037 INFO:teuthology.orchestra.run.smithi060.stdout: 406382 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/upgrade/3-upgrade/.qa -> ../.qa/ 2024-03-27T21:42:45.037 INFO:teuthology.orchestra.run.smithi060.stdout: 406383 4 -rw-r--r-- 1 ubuntu ubuntu 1007 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/upgrade/3-upgrade/simple.yaml 2024-03-27T21:42:45.037 INFO:teuthology.orchestra.run.smithi060.stdout: 406384 8 -rw-r--r-- 1 ubuntu ubuntu 7926 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/upgrade/3-upgrade/staggered.yaml 2024-03-27T21:42:45.037 INFO:teuthology.orchestra.run.smithi060.stdout: 406377 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/upgrade/2-repo_digest 2024-03-27T21:42:45.037 INFO:teuthology.orchestra.run.smithi060.stdout: 406378 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/upgrade/2-repo_digest/.qa -> ../.qa/ 2024-03-27T21:42:45.038 INFO:teuthology.orchestra.run.smithi060.stdout: 406379 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/upgrade/2-repo_digest/defaut.yaml 2024-03-27T21:42:45.038 INFO:teuthology.orchestra.run.smithi060.stdout: 406380 4 -rw-r--r-- 1 ubuntu ubuntu 105 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/upgrade/2-repo_digest/repo_digest.yaml 2024-03-27T21:42:45.038 INFO:teuthology.orchestra.run.smithi060.stdout: 406388 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/upgrade/mon_election -> .qa/mon_election 2024-03-27T21:42:45.038 INFO:teuthology.orchestra.run.smithi060.stdout: 406285 0 lrwxrwxrwx 1 ubuntu ubuntu 20 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/rbd_iscsi -> .qa/suites/rbd/iscsi 2024-03-27T21:42:45.038 INFO:teuthology.orchestra.run.smithi060.stdout: 406355 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/thrash 2024-03-27T21:42:45.038 INFO:teuthology.orchestra.run.smithi060.stdout: 406367 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/thrash/fixed-2.yaml -> ../smoke/fixed-2.yaml 2024-03-27T21:42:45.038 INFO:teuthology.orchestra.run.smithi060.stdout: 406368 0 lrwxrwxrwx 1 ubuntu ubuntu 8 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/thrash/msgr -> .qa/msgr 2024-03-27T21:42:45.038 INFO:teuthology.orchestra.run.smithi060.stdout: 406356 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/thrash/% 2024-03-27T21:42:45.038 INFO:teuthology.orchestra.run.smithi060.stdout: 406369 4 -rw-r--r-- 1 ubuntu ubuntu 45 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/thrash/root.yaml 2024-03-27T21:42:45.039 INFO:teuthology.orchestra.run.smithi060.stdout: 406361 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/thrash/3-tasks 2024-03-27T21:42:45.039 INFO:teuthology.orchestra.run.smithi060.stdout: 406362 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/thrash/3-tasks/.qa -> ../.qa/ 2024-03-27T21:42:45.039 INFO:teuthology.orchestra.run.smithi060.stdout: 406366 0 lrwxrwxrwx 1 ubuntu ubuntu 56 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/thrash/3-tasks/snaps-few-objects.yaml -> .qa/suites/rados/thrash/workloads/snaps-few-objects.yaml 2024-03-27T21:42:45.039 INFO:teuthology.orchestra.run.smithi060.stdout: 406364 0 lrwxrwxrwx 1 ubuntu ubuntu 49 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/thrash/3-tasks/radosbench.yaml -> .qa/suites/rados/thrash/workloads/radosbench.yaml 2024-03-27T21:42:45.039 INFO:teuthology.orchestra.run.smithi060.stdout: 406363 0 lrwxrwxrwx 1 ubuntu ubuntu 54 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/thrash/3-tasks/rados_api_tests.yaml -> .qa/suites/rados/thrash/workloads/rados_api_tests.yaml 2024-03-27T21:42:45.039 INFO:teuthology.orchestra.run.smithi060.stdout: 406365 0 lrwxrwxrwx 1 ubuntu ubuntu 52 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/thrash/3-tasks/small-objects.yaml -> .qa/suites/rados/thrash/workloads/small-objects.yaml 2024-03-27T21:42:45.039 INFO:teuthology.orchestra.run.smithi060.stdout: 406357 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/thrash/.qa -> ../.qa/ 2024-03-27T21:42:45.039 INFO:teuthology.orchestra.run.smithi060.stdout: 406360 4 -rw-r--r-- 1 ubuntu ubuntu 696 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/thrash/2-thrash.yaml 2024-03-27T21:42:45.040 INFO:teuthology.orchestra.run.smithi060.stdout: 406359 4 -rw-r--r-- 1 ubuntu ubuntu 92 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/thrash/1-start.yaml 2024-03-27T21:42:45.040 INFO:teuthology.orchestra.run.smithi060.stdout: 406358 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/thrash/0-distro -> .qa/distros/supported-container-hosts 2024-03-27T21:42:45.040 INFO:teuthology.orchestra.run.smithi060.stdout: 406242 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/mds_upgrade_sequence -> .qa/suites/fs/upgrade/mds_upgrade_sequence/ 2024-03-27T21:42:45.040 INFO:teuthology.orchestra.run.smithi060.stdout: 406389 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/with-work 2024-03-27T21:42:45.040 INFO:teuthology.orchestra.run.smithi060.stdout: 406393 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/with-work/fixed-2.yaml -> ../smoke/fixed-2.yaml 2024-03-27T21:42:45.040 INFO:teuthology.orchestra.run.smithi060.stdout: 406399 0 lrwxrwxrwx 1 ubuntu ubuntu 8 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/with-work/msgr -> .qa/msgr 2024-03-27T21:42:45.040 INFO:teuthology.orchestra.run.smithi060.stdout: 406390 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/with-work/% 2024-03-27T21:42:45.040 INFO:teuthology.orchestra.run.smithi060.stdout: 406391 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/with-work/.qa -> ../.qa/ 2024-03-27T21:42:45.040 INFO:teuthology.orchestra.run.smithi060.stdout: 406401 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/with-work/tasks 2024-03-27T21:42:45.041 INFO:teuthology.orchestra.run.smithi060.stdout: 406402 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/with-work/tasks/.qa -> ../.qa/ 2024-03-27T21:42:45.041 INFO:teuthology.orchestra.run.smithi060.stdout: 406403 0 lrwxrwxrwx 1 ubuntu ubuntu 49 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/with-work/tasks/rados_api_tests.yaml -> .qa/suites/rados/basic/tasks/rados_api_tests.yaml 2024-03-27T21:42:45.041 INFO:teuthology.orchestra.run.smithi060.stdout: 406405 4 -rw-r--r-- 1 ubuntu ubuntu 400 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/with-work/tasks/rotate-keys.yaml 2024-03-27T21:42:45.041 INFO:teuthology.orchestra.run.smithi060.stdout: 406404 0 lrwxrwxrwx 1 ubuntu ubuntu 46 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/with-work/tasks/rados_python.yaml -> .qa/suites/rados/basic/tasks/rados_python.yaml 2024-03-27T21:42:45.041 INFO:teuthology.orchestra.run.smithi060.stdout: 406398 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/with-work/mon_election -> .qa/mon_election 2024-03-27T21:42:45.041 INFO:teuthology.orchestra.run.smithi060.stdout: 406392 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/with-work/0-distro -> .qa/distros/supported-container-hosts 2024-03-27T21:42:45.041 INFO:teuthology.orchestra.run.smithi060.stdout: 406400 4 -rw-r--r-- 1 ubuntu ubuntu 92 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/with-work/start.yaml 2024-03-27T21:42:45.041 INFO:teuthology.orchestra.run.smithi060.stdout: 406394 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/with-work/mode 2024-03-27T21:42:45.042 INFO:teuthology.orchestra.run.smithi060.stdout: 406397 4 -rw-r--r-- 1 ubuntu ubuntu 45 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/with-work/mode/root.yaml 2024-03-27T21:42:45.042 INFO:teuthology.orchestra.run.smithi060.stdout: 406395 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/with-work/mode/.qa -> ../.qa/ 2024-03-27T21:42:45.042 INFO:teuthology.orchestra.run.smithi060.stdout: 406396 4 -rw-r--r-- 1 ubuntu ubuntu 97 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/with-work/mode/packaged.yaml 2024-03-27T21:42:45.042 INFO:teuthology.orchestra.run.smithi060.stdout: 406265 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/orchestrator_cli 2024-03-27T21:42:45.042 INFO:teuthology.orchestra.run.smithi060.stdout: 406270 0 lrwxrwxrwx 1 ubuntu ubuntu 14 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/orchestrator_cli/agent -> ../smoke/agent 2024-03-27T21:42:45.042 INFO:teuthology.orchestra.run.smithi060.stdout: 406268 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/orchestrator_cli/0-random-distro$ -> .qa/distros/supported-container-hosts 2024-03-27T21:42:45.042 INFO:teuthology.orchestra.run.smithi060.stdout: 406271 4 -rw-r--r-- 1 ubuntu ubuntu 514 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/orchestrator_cli/orchestrator_cli.yaml 2024-03-27T21:42:45.042 INFO:teuthology.orchestra.run.smithi060.stdout: 406266 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/orchestrator_cli/% 2024-03-27T21:42:45.043 INFO:teuthology.orchestra.run.smithi060.stdout: 406267 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/orchestrator_cli/.qa -> ../.qa/ 2024-03-27T21:42:45.043 INFO:teuthology.orchestra.run.smithi060.stdout: 406269 0 lrwxrwxrwx 1 ubuntu ubuntu 28 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/orchestrator_cli/2-node-mgr.yaml -> .qa/clusters/2-node-mgr.yaml 2024-03-27T21:42:45.043 INFO:teuthology.orchestra.run.smithi060.stdout: 406406 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/workunits 2024-03-27T21:42:45.043 INFO:teuthology.orchestra.run.smithi060.stdout: 406410 0 lrwxrwxrwx 1 ubuntu ubuntu 14 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/workunits/agent -> ../smoke/agent 2024-03-27T21:42:45.043 INFO:teuthology.orchestra.run.smithi060.stdout: 406407 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/workunits/% 2024-03-27T21:42:45.043 INFO:teuthology.orchestra.run.smithi060.stdout: 406408 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/workunits/.qa -> ../.qa/ 2024-03-27T21:42:45.043 INFO:teuthology.orchestra.run.smithi060.stdout: 406412 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/workunits/task 2024-03-27T21:42:45.043 INFO:teuthology.orchestra.run.smithi060.stdout: 406413 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/workunits/task/.qa -> ../.qa/ 2024-03-27T21:42:45.043 INFO:teuthology.orchestra.run.smithi060.stdout: 406424 4 -rw-r--r-- 1 ubuntu ubuntu 2661 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/workunits/task/test_monitoring_stack_basic.yaml 2024-03-27T21:42:45.044 INFO:teuthology.orchestra.run.smithi060.stdout: 406415 4 -rw-r--r-- 1 ubuntu ubuntu 210 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/workunits/task/test_cephadm.yaml 2024-03-27T21:42:45.044 INFO:teuthology.orchestra.run.smithi060.stdout: 406417 4 -rw-r--r-- 1 ubuntu ubuntu 4064 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/workunits/task/test_extra_daemon_features.yaml 2024-03-27T21:42:45.044 INFO:teuthology.orchestra.run.smithi060.stdout: 406418 4 -rw-r--r-- 1 ubuntu ubuntu 2740 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/workunits/task/test_host_drain.yaml 2024-03-27T21:42:45.044 INFO:teuthology.orchestra.run.smithi060.stdout: 406426 4 -rw-r--r-- 1 ubuntu ubuntu 2084 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/workunits/task/test_set_mon_crush_locations.yaml 2024-03-27T21:42:45.044 INFO:teuthology.orchestra.run.smithi060.stdout: 406419 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/workunits/task/test_iscsi_container 2024-03-27T21:42:45.044 INFO:teuthology.orchestra.run.smithi060.stdout: 406420 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/workunits/task/test_iscsi_container/+ 2024-03-27T21:42:45.044 INFO:teuthology.orchestra.run.smithi060.stdout: 406421 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/workunits/task/test_iscsi_container/.qa -> ../.qa/ 2024-03-27T21:42:45.044 INFO:teuthology.orchestra.run.smithi060.stdout: 406423 4 -rw-r--r-- 1 ubuntu ubuntu 362 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/workunits/task/test_iscsi_container/test_iscsi_container.yaml 2024-03-27T21:42:45.045 INFO:teuthology.orchestra.run.smithi060.stdout: 406422 0 lrwxrwxrwx 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/workunits/task/test_iscsi_container/centos_9.stream.yaml -> .qa/distros/podman/centos_9.stream.yaml 2024-03-27T21:42:45.045 INFO:teuthology.orchestra.run.smithi060.stdout: 406414 4 -rw-r--r-- 1 ubuntu ubuntu 689 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/workunits/task/test_ca_signed_key.yaml 2024-03-27T21:42:45.045 INFO:teuthology.orchestra.run.smithi060.stdout: 406416 4 -rw-r--r-- 1 ubuntu ubuntu 154 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/workunits/task/test_cephadm_repos.yaml 2024-03-27T21:42:45.045 INFO:teuthology.orchestra.run.smithi060.stdout: 406425 4 -rw-r--r-- 1 ubuntu ubuntu 1313 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/workunits/task/test_rgw_multisite.yaml 2024-03-27T21:42:45.045 INFO:teuthology.orchestra.run.smithi060.stdout: 406411 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/workunits/mon_election -> .qa/mon_election 2024-03-27T21:42:45.045 INFO:teuthology.orchestra.run.smithi060.stdout: 406409 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/workunits/0-distro -> .qa/distros/supported-container-hosts 2024-03-27T21:42:45.045 INFO:teuthology.orchestra.run.smithi060.stdout: 406329 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-small 2024-03-27T21:42:45.045 INFO:teuthology.orchestra.run.smithi060.stdout: 406336 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-small/agent 2024-03-27T21:42:45.045 INFO:teuthology.orchestra.run.smithi060.stdout: 406337 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-small/agent/.qa -> ../.qa/ 2024-03-27T21:42:45.046 INFO:teuthology.orchestra.run.smithi060.stdout: 406338 4 -rw-r--r-- 1 ubuntu ubuntu 77 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-small/agent/off.yaml 2024-03-27T21:42:45.046 INFO:teuthology.orchestra.run.smithi060.stdout: 406339 4 -rw-r--r-- 1 ubuntu ubuntu 76 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-small/agent/on.yaml 2024-03-27T21:42:45.046 INFO:teuthology.orchestra.run.smithi060.stdout: 406340 4 -rw-r--r-- 1 ubuntu ubuntu 426 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-small/fixed-2.yaml 2024-03-27T21:42:45.046 INFO:teuthology.orchestra.run.smithi060.stdout: 406335 0 lrwxrwxrwx 1 ubuntu ubuntu 28 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-small/0-nvme-loop.yaml -> .qa/overrides/nvme_loop.yaml 2024-03-27T21:42:45.046 INFO:teuthology.orchestra.run.smithi060.stdout: 406330 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-small/% 2024-03-27T21:42:45.046 INFO:teuthology.orchestra.run.smithi060.stdout: 406331 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-small/.qa -> ../.qa/ 2024-03-27T21:42:45.046 INFO:teuthology.orchestra.run.smithi060.stdout: 406341 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-small/mon_election -> .qa/mon_election 2024-03-27T21:42:45.046 INFO:teuthology.orchestra.run.smithi060.stdout: 406332 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-small/0-distro 2024-03-27T21:42:45.047 INFO:teuthology.orchestra.run.smithi060.stdout: 406334 0 lrwxrwxrwx 1 ubuntu ubuntu 53 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-small/0-distro/centos_9.stream_runc.yaml -> .qa/distros/container-hosts/centos_9.stream_runc.yaml 2024-03-27T21:42:45.047 INFO:teuthology.orchestra.run.smithi060.stdout: 406333 0 lrwxrwxrwx 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-small/0-distro/.qa -> ../.qa 2024-03-27T21:42:45.047 INFO:teuthology.orchestra.run.smithi060.stdout: 406342 4 -rw-r--r-- 1 ubuntu ubuntu 356 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-small/start.yaml 2024-03-27T21:42:45.047 INFO:teuthology.orchestra.run.smithi060.stdout: 406294 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless 2024-03-27T21:42:45.047 INFO:teuthology.orchestra.run.smithi060.stdout: 406298 0 lrwxrwxrwx 1 ubuntu ubuntu 28 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/0-nvme-loop.yaml -> .qa/overrides/nvme_loop.yaml 2024-03-27T21:42:45.047 INFO:teuthology.orchestra.run.smithi060.stdout: 406295 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/% 2024-03-27T21:42:45.047 INFO:teuthology.orchestra.run.smithi060.stdout: 406296 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/.qa -> ../.qa/ 2024-03-27T21:42:45.047 INFO:teuthology.orchestra.run.smithi060.stdout: 406300 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/2-services 2024-03-27T21:42:45.047 INFO:teuthology.orchestra.run.smithi060.stdout: 406309 4 -rw-r--r-- 1 ubuntu ubuntu 2364 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/2-services/nfs-ingress-rgw-user.yaml 2024-03-27T21:42:45.048 INFO:teuthology.orchestra.run.smithi060.stdout: 406317 4 -rw-r--r-- 1 ubuntu ubuntu 257 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/2-services/rgw.yaml 2024-03-27T21:42:45.048 INFO:teuthology.orchestra.run.smithi060.stdout: 406301 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/2-services/.qa -> ../.qa/ 2024-03-27T21:42:45.048 INFO:teuthology.orchestra.run.smithi060.stdout: 406315 4 -rw-r--r-- 1 ubuntu ubuntu 182 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/2-services/nvmeof.yaml 2024-03-27T21:42:45.048 INFO:teuthology.orchestra.run.smithi060.stdout: 406303 4 -rw-r--r-- 1 ubuntu ubuntu 1405 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/2-services/client-keyring.yaml 2024-03-27T21:42:45.048 INFO:teuthology.orchestra.run.smithi060.stdout: 406311 4 -rw-r--r-- 1 ubuntu ubuntu 2167 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/2-services/nfs-ingress2.yaml 2024-03-27T21:42:45.048 INFO:teuthology.orchestra.run.smithi060.stdout: 406314 4 -rw-r--r-- 1 ubuntu ubuntu 221 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/2-services/nfs2.yaml 2024-03-27T21:42:45.048 INFO:teuthology.orchestra.run.smithi060.stdout: 406306 4 -rw-r--r-- 1 ubuntu ubuntu 248 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/2-services/mirror.yaml 2024-03-27T21:42:45.048 INFO:teuthology.orchestra.run.smithi060.stdout: 406308 4 -rw-r--r-- 1 ubuntu ubuntu 2317 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/2-services/nfs-ingress-rgw-bucket.yaml 2024-03-27T21:42:45.049 INFO:teuthology.orchestra.run.smithi060.stdout: 406307 4 -rw-r--r-- 1 ubuntu ubuntu 955 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/2-services/nfs-haproxy-proto.yaml 2024-03-27T21:42:45.049 INFO:teuthology.orchestra.run.smithi060.stdout: 406313 4 -rw-r--r-- 1 ubuntu ubuntu 234 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/2-services/nfs.yaml 2024-03-27T21:42:45.049 INFO:teuthology.orchestra.run.smithi060.stdout: 406316 4 -rw-r--r-- 1 ubuntu ubuntu 1907 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/2-services/rgw-ingress.yaml 2024-03-27T21:42:45.049 INFO:teuthology.orchestra.run.smithi060.stdout: 406304 4 -rw-r--r-- 1 ubuntu ubuntu 184 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/2-services/iscsi.yaml 2024-03-27T21:42:45.049 INFO:teuthology.orchestra.run.smithi060.stdout: 406302 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/2-services/basic.yaml 2024-03-27T21:42:45.049 INFO:teuthology.orchestra.run.smithi060.stdout: 406305 4 -rw-r--r-- 1 ubuntu ubuntu 287 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/2-services/jaeger.yaml 2024-03-27T21:42:45.049 INFO:teuthology.orchestra.run.smithi060.stdout: 406312 4 -rw-r--r-- 1 ubuntu ubuntu 1308 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/2-services/nfs-keepalive-only.yaml 2024-03-27T21:42:45.049 INFO:teuthology.orchestra.run.smithi060.stdout: 406310 4 -rw-r--r-- 1 ubuntu ubuntu 1716 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/2-services/nfs-ingress.yaml 2024-03-27T21:42:45.050 INFO:teuthology.orchestra.run.smithi060.stdout: 406299 4 -rw-r--r-- 1 ubuntu ubuntu 403 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/1-start.yaml 2024-03-27T21:42:45.050 INFO:teuthology.orchestra.run.smithi060.stdout: 406318 4 -rw-r--r-- 1 ubuntu ubuntu 270 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/3-final.yaml 2024-03-27T21:42:45.050 INFO:teuthology.orchestra.run.smithi060.stdout: 406297 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke-roleless/0-distro -> .qa/distros/supported-container-hosts 2024-03-27T21:42:45.050 INFO:teuthology.orchestra.run.smithi060.stdout: 406343 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke 2024-03-27T21:42:45.050 INFO:teuthology.orchestra.run.smithi060.stdout: 406348 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke/agent 2024-03-27T21:42:45.050 INFO:teuthology.orchestra.run.smithi060.stdout: 406349 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke/agent/.qa -> ../.qa/ 2024-03-27T21:42:45.050 INFO:teuthology.orchestra.run.smithi060.stdout: 406350 4 -rw-r--r-- 1 ubuntu ubuntu 77 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke/agent/off.yaml 2024-03-27T21:42:45.050 INFO:teuthology.orchestra.run.smithi060.stdout: 406351 4 -rw-r--r-- 1 ubuntu ubuntu 76 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke/agent/on.yaml 2024-03-27T21:42:45.051 INFO:teuthology.orchestra.run.smithi060.stdout: 406352 4 -rw-r--r-- 1 ubuntu ubuntu 456 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke/fixed-2.yaml 2024-03-27T21:42:45.051 INFO:teuthology.orchestra.run.smithi060.stdout: 406347 0 lrwxrwxrwx 1 ubuntu ubuntu 28 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke/0-nvme-loop.yaml -> .qa/overrides/nvme_loop.yaml 2024-03-27T21:42:45.051 INFO:teuthology.orchestra.run.smithi060.stdout: 406344 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke/% 2024-03-27T21:42:45.051 INFO:teuthology.orchestra.run.smithi060.stdout: 406345 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke/.qa -> ../.qa/ 2024-03-27T21:42:45.051 INFO:teuthology.orchestra.run.smithi060.stdout: 406353 0 lrwxrwxrwx 1 ubuntu ubuntu 16 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke/mon_election -> .qa/mon_election 2024-03-27T21:42:45.051 INFO:teuthology.orchestra.run.smithi060.stdout: 406346 0 lrwxrwxrwx 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke/0-distro -> .qa/distros/supported-container-hosts 2024-03-27T21:42:45.051 INFO:teuthology.orchestra.run.smithi060.stdout: 406354 4 -rw-r--r-- 1 ubuntu ubuntu 412 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/smoke/start.yaml 2024-03-27T21:42:45.051 INFO:teuthology.orchestra.run.smithi060.stdout: 406254 0 lrwxrwxrwx 1 ubuntu ubuntu 18 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/orch/cephadm/nfs -> .qa/suites/fs/nfs/ 2024-03-27T21:42:45.051 INFO:teuthology.orchestra.run.smithi060.stdout: 407667 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw-multisite-upgrade 2024-03-27T21:42:45.052 INFO:teuthology.orchestra.run.smithi060.stdout: 407668 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw-multisite-upgrade/.qa -> ../.qa/ 2024-03-27T21:42:45.052 INFO:teuthology.orchestra.run.smithi060.stdout: 407669 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw-multisite-upgrade/pacific-x 2024-03-27T21:42:45.052 INFO:teuthology.orchestra.run.smithi060.stdout: 407670 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw-multisite-upgrade/pacific-x/% 2024-03-27T21:42:45.052 INFO:teuthology.orchestra.run.smithi060.stdout: 407671 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw-multisite-upgrade/pacific-x/.qa -> ../.qa/ 2024-03-27T21:42:45.052 INFO:teuthology.orchestra.run.smithi060.stdout: 407676 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw-multisite-upgrade/pacific-x/supported-random-distro$ -> .qa/distros/supported-random-distro$ 2024-03-27T21:42:45.052 INFO:teuthology.orchestra.run.smithi060.stdout: 407678 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw-multisite-upgrade/pacific-x/upgrade 2024-03-27T21:42:45.052 INFO:teuthology.orchestra.run.smithi060.stdout: 407679 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw-multisite-upgrade/pacific-x/upgrade/.qa -> ../.qa/ 2024-03-27T21:42:45.052 INFO:teuthology.orchestra.run.smithi060.stdout: 407680 4 -rw-r--r-- 1 ubuntu ubuntu 165 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw-multisite-upgrade/pacific-x/upgrade/primary.yaml 2024-03-27T21:42:45.053 INFO:teuthology.orchestra.run.smithi060.stdout: 407681 4 -rw-r--r-- 1 ubuntu ubuntu 182 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw-multisite-upgrade/pacific-x/upgrade/secondary.yaml 2024-03-27T21:42:45.053 INFO:teuthology.orchestra.run.smithi060.stdout: 407672 4 -rw-r--r-- 1 ubuntu ubuntu 152 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw-multisite-upgrade/pacific-x/clusters.yaml 2024-03-27T21:42:45.053 INFO:teuthology.orchestra.run.smithi060.stdout: 407673 0 lrwxrwxrwx 1 ubuntu ubuntu 27 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw-multisite-upgrade/pacific-x/frontend.yaml -> .qa/rgw_frontend/beast.yaml 2024-03-27T21:42:45.053 INFO:teuthology.orchestra.run.smithi060.stdout: 407677 4 -rw-r--r-- 1 ubuntu ubuntu 300 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw-multisite-upgrade/pacific-x/tasks.yaml 2024-03-27T21:42:45.053 INFO:teuthology.orchestra.run.smithi060.stdout: 407675 4 -rw-r--r-- 1 ubuntu ubuntu 489 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw-multisite-upgrade/pacific-x/realm.yaml 2024-03-27T21:42:45.053 INFO:teuthology.orchestra.run.smithi060.stdout: 407674 4 -rw-r--r-- 1 ubuntu ubuntu 692 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/rgw-multisite-upgrade/pacific-x/overrides.yaml 2024-03-27T21:42:45.053 INFO:teuthology.orchestra.run.smithi060.stdout: 406474 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle 2024-03-27T21:42:45.053 INFO:teuthology.orchestra.run.smithi060.stdout: 406475 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/.qa -> ../.qa/ 2024-03-27T21:42:45.054 INFO:teuthology.orchestra.run.smithi060.stdout: 406476 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd 2024-03-27T21:42:45.054 INFO:teuthology.orchestra.run.smithi060.stdout: 406477 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/% 2024-03-27T21:42:45.054 INFO:teuthology.orchestra.run.smithi060.stdout: 406483 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/objectstore -> .qa/objectstore 2024-03-27T21:42:45.054 INFO:teuthology.orchestra.run.smithi060.stdout: 406478 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/.qa -> ../.qa/ 2024-03-27T21:42:45.054 INFO:teuthology.orchestra.run.smithi060.stdout: 406487 0 lrwxrwxrwx 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/supported-distros -> .qa/distros/supported 2024-03-27T21:42:45.054 INFO:teuthology.orchestra.run.smithi060.stdout: 406504 0 lrwxrwxrwx 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/thrashosds-health.yaml -> .qa/tasks/thrashosds-health.yaml 2024-03-27T21:42:45.054 INFO:teuthology.orchestra.run.smithi060.stdout: 406482 4 -rw-r--r-- 1 ubuntu ubuntu 145 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/ignorelist_health.yaml 2024-03-27T21:42:45.054 INFO:teuthology.orchestra.run.smithi060.stdout: 406479 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/clusters 2024-03-27T21:42:45.055 INFO:teuthology.orchestra.run.smithi060.stdout: 406480 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/clusters/.qa -> ../.qa/ 2024-03-27T21:42:45.055 INFO:teuthology.orchestra.run.smithi060.stdout: 406481 4 -rw-r--r-- 1 ubuntu ubuntu 92 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/clusters/3osd-1per-target.yaml 2024-03-27T21:42:45.055 INFO:teuthology.orchestra.run.smithi060.stdout: 406488 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/tasks 2024-03-27T21:42:45.055 INFO:teuthology.orchestra.run.smithi060.stdout: 406489 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/tasks/.qa -> ../.qa/ 2024-03-27T21:42:45.055 INFO:teuthology.orchestra.run.smithi060.stdout: 406503 4 -rw-r--r-- 1 ubuntu ubuntu 220 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/tasks/snaps-many-objects.yaml 2024-03-27T21:42:45.055 INFO:teuthology.orchestra.run.smithi060.stdout: 406502 4 -rw-r--r-- 1 ubuntu ubuntu 219 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/tasks/snaps-few-objects.yaml 2024-03-27T21:42:45.055 INFO:teuthology.orchestra.run.smithi060.stdout: 406494 4 -rw-r--r-- 1 ubuntu ubuntu 85 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/tasks/cfuse_workunit_suites_fsstress.yaml 2024-03-27T21:42:45.055 INFO:teuthology.orchestra.run.smithi060.stdout: 406500 4 -rw-r--r-- 1 ubuntu ubuntu 710 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/tasks/radosbench.yaml 2024-03-27T21:42:45.055 INFO:teuthology.orchestra.run.smithi060.stdout: 406498 4 -rw-r--r-- 1 ubuntu ubuntu 315 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/tasks/cfuse_workunit_suites_truncate_delay.yaml 2024-03-27T21:42:45.056 INFO:teuthology.orchestra.run.smithi060.stdout: 406497 4 -rw-r--r-- 1 ubuntu ubuntu 158 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/tasks/cfuse_workunit_suites_pjd.yaml 2024-03-27T21:42:45.056 INFO:teuthology.orchestra.run.smithi060.stdout: 406495 4 -rw-r--r-- 1 ubuntu ubuntu 96 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/tasks/cfuse_workunit_suites_fsx.yaml 2024-03-27T21:42:45.056 INFO:teuthology.orchestra.run.smithi060.stdout: 406491 4 -rw-r--r-- 1 ubuntu ubuntu 183 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/tasks/cfuse_workunit_kernel_untar_build.yaml 2024-03-27T21:42:45.056 INFO:teuthology.orchestra.run.smithi060.stdout: 406499 4 -rw-r--r-- 1 ubuntu ubuntu 347 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/tasks/rados_api_tests.yaml 2024-03-27T21:42:45.056 INFO:teuthology.orchestra.run.smithi060.stdout: 406496 4 -rw-r--r-- 1 ubuntu ubuntu 172 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/tasks/cfuse_workunit_suites_fsync.yaml 2024-03-27T21:42:45.056 INFO:teuthology.orchestra.run.smithi060.stdout: 406501 4 -rw-r--r-- 1 ubuntu ubuntu 135 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/tasks/readwrite.yaml 2024-03-27T21:42:45.056 INFO:teuthology.orchestra.run.smithi060.stdout: 406493 4 -rw-r--r-- 1 ubuntu ubuntu 205 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/tasks/cfuse_workunit_suites_ffsb.yaml 2024-03-27T21:42:45.056 INFO:teuthology.orchestra.run.smithi060.stdout: 406492 4 -rw-r--r-- 1 ubuntu ubuntu 90 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/tasks/cfuse_workunit_misc.yaml 2024-03-27T21:42:45.057 INFO:teuthology.orchestra.run.smithi060.stdout: 406490 4 -rw-r--r-- 1 ubuntu ubuntu 323 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/tasks/admin_socket_objecter_requests.yaml 2024-03-27T21:42:45.057 INFO:teuthology.orchestra.run.smithi060.stdout: 406484 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/powercycle 2024-03-27T21:42:45.057 INFO:teuthology.orchestra.run.smithi060.stdout: 406485 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/powercycle/.qa -> ../.qa/ 2024-03-27T21:42:45.057 INFO:teuthology.orchestra.run.smithi060.stdout: 406486 4 -rw-r--r-- 1 ubuntu ubuntu 450 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/powercycle/osd/powercycle/default.yaml 2024-03-27T21:42:45.057 INFO:teuthology.orchestra.run.smithi060.stdout: 408047 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke 2024-03-27T21:42:45.057 INFO:teuthology.orchestra.run.smithi060.stdout: 408048 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/.qa -> ../.qa/ 2024-03-27T21:42:45.057 INFO:teuthology.orchestra.run.smithi060.stdout: 408049 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic 2024-03-27T21:42:45.057 INFO:teuthology.orchestra.run.smithi060.stdout: 408050 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/% 2024-03-27T21:42:45.057 INFO:teuthology.orchestra.run.smithi060.stdout: 408057 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/objectstore 2024-03-27T21:42:45.058 INFO:teuthology.orchestra.run.smithi060.stdout: 408058 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/objectstore/.qa -> ../.qa/ 2024-03-27T21:42:45.058 INFO:teuthology.orchestra.run.smithi060.stdout: 408059 0 lrwxrwxrwx 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/objectstore/bluestore-bitmap.yaml -> .qa/objectstore_debug/bluestore-bitmap.yaml 2024-03-27T21:42:45.058 INFO:teuthology.orchestra.run.smithi060.stdout: 408051 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/.qa -> ../.qa/ 2024-03-27T21:42:45.058 INFO:teuthology.orchestra.run.smithi060.stdout: 408060 0 lrwxrwxrwx 1 ubuntu ubuntu 42 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/supported-random-distro$ -> ../../../distros/supported-random-distro$/ 2024-03-27T21:42:45.058 INFO:teuthology.orchestra.run.smithi060.stdout: 408052 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/clusters 2024-03-27T21:42:45.058 INFO:teuthology.orchestra.run.smithi060.stdout: 408053 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/clusters/+ 2024-03-27T21:42:45.058 INFO:teuthology.orchestra.run.smithi060.stdout: 408054 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/clusters/.qa -> ../.qa/ 2024-03-27T21:42:45.058 INFO:teuthology.orchestra.run.smithi060.stdout: 408056 4 -rw-r--r-- 1 ubuntu ubuntu 155 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/clusters/openstack.yaml 2024-03-27T21:42:45.059 INFO:teuthology.orchestra.run.smithi060.stdout: 408055 0 lrwxrwxrwx 1 ubuntu ubuntu 32 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/clusters/fixed-3-cephfs.yaml -> .qa/clusters/fixed-3-cephfs.yaml 2024-03-27T21:42:45.059 INFO:teuthology.orchestra.run.smithi060.stdout: 408061 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks 2024-03-27T21:42:45.059 INFO:teuthology.orchestra.run.smithi060.stdout: 408062 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/% 2024-03-27T21:42:45.059 INFO:teuthology.orchestra.run.smithi060.stdout: 408063 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/.qa -> ../.qa/ 2024-03-27T21:42:45.059 INFO:teuthology.orchestra.run.smithi060.stdout: 408065 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test 2024-03-27T21:42:45.059 INFO:teuthology.orchestra.run.smithi060.stdout: 408067 4 -rw-r--r-- 1 ubuntu ubuntu 159 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/cfuse_workunit_suites_blogbench.yaml 2024-03-27T21:42:45.059 INFO:teuthology.orchestra.run.smithi060.stdout: 408076 4 -rw-r--r-- 1 ubuntu ubuntu 841 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/mon_thrash.yaml 2024-03-27T21:42:45.059 INFO:teuthology.orchestra.run.smithi060.stdout: 408072 4 -rw-r--r-- 1 ubuntu ubuntu 238 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/kclient_workunit_suites_dbench.yaml 2024-03-27T21:42:45.059 INFO:teuthology.orchestra.run.smithi060.stdout: 408089 4 -rw-r--r-- 1 ubuntu ubuntu 560 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/rgw_ec_s3tests.yaml 2024-03-27T21:42:45.060 INFO:teuthology.orchestra.run.smithi060.stdout: 408069 4 -rw-r--r-- 1 ubuntu ubuntu 155 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/cfuse_workunit_suites_iozone.yaml 2024-03-27T21:42:45.060 INFO:teuthology.orchestra.run.smithi060.stdout: 408066 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/.qa -> ../.qa/ 2024-03-27T21:42:45.060 INFO:teuthology.orchestra.run.smithi060.stdout: 408083 4 -rw-r--r-- 1 ubuntu ubuntu 212 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/rados_workunit_loadgen_mix.yaml 2024-03-27T21:42:45.060 INFO:teuthology.orchestra.run.smithi060.stdout: 408080 4 -rw-r--r-- 1 ubuntu ubuntu 244 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/rados_cls_all.yaml 2024-03-27T21:42:45.060 INFO:teuthology.orchestra.run.smithi060.stdout: 408088 4 -rw-r--r-- 1 ubuntu ubuntu 301 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/rbd_workunit_suites_iozone.yaml 2024-03-27T21:42:45.060 INFO:teuthology.orchestra.run.smithi060.stdout: 408068 4 -rw-r--r-- 1 ubuntu ubuntu 146 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/cfuse_workunit_suites_fsstress.yaml 2024-03-27T21:42:45.060 INFO:teuthology.orchestra.run.smithi060.stdout: 408070 4 -rw-r--r-- 1 ubuntu ubuntu 310 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/cfuse_workunit_suites_pjd.yaml 2024-03-27T21:42:45.060 INFO:teuthology.orchestra.run.smithi060.stdout: 408074 4 -rw-r--r-- 1 ubuntu ubuntu 235 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/kclient_workunit_suites_pjd.yaml 2024-03-27T21:42:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: 408078 4 -rw-r--r-- 1 ubuntu ubuntu 999 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/rados_bench.yaml 2024-03-27T21:42:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: 408081 4 -rw-r--r-- 1 ubuntu ubuntu 732 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/rados_ec_snaps.yaml 2024-03-27T21:42:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: 408087 4 -rw-r--r-- 1 ubuntu ubuntu 275 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/rbd_python_api_tests.yaml 2024-03-27T21:42:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: 408085 4 -rw-r--r-- 1 ubuntu ubuntu 210 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/rbd_cli_import_export.yaml 2024-03-27T21:42:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: 408077 4 -rw-r--r-- 1 ubuntu ubuntu 642 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/rados_api_tests.yaml 2024-03-27T21:42:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: 408079 4 -rw-r--r-- 1 ubuntu ubuntu 1216 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/rados_cache_snaps.yaml 2024-03-27T21:42:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: 408086 4 -rw-r--r-- 1 ubuntu ubuntu 555 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/rbd_fsx.yaml 2024-03-27T21:42:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: 408082 4 -rw-r--r-- 1 ubuntu ubuntu 367 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/rados_python.yaml 2024-03-27T21:42:45.061 INFO:teuthology.orchestra.run.smithi060.stdout: 408084 4 -rw-r--r-- 1 ubuntu ubuntu 311 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/rbd_api_tests.yaml 2024-03-27T21:42:45.062 INFO:teuthology.orchestra.run.smithi060.stdout: 408075 4 -rw-r--r-- 1 ubuntu ubuntu 291 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/libcephfs_interface_tests.yaml 2024-03-27T21:42:45.062 INFO:teuthology.orchestra.run.smithi060.stdout: 408090 4 -rw-r--r-- 1 ubuntu ubuntu 535 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/rgw_s3tests.yaml 2024-03-27T21:42:45.062 INFO:teuthology.orchestra.run.smithi060.stdout: 408071 4 -rw-r--r-- 1 ubuntu ubuntu 219 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/kclient_workunit_direct_io.yaml 2024-03-27T21:42:45.062 INFO:teuthology.orchestra.run.smithi060.stdout: 408073 4 -rw-r--r-- 1 ubuntu ubuntu 240 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/test/kclient_workunit_suites_fsstress.yaml 2024-03-27T21:42:45.062 INFO:teuthology.orchestra.run.smithi060.stdout: 408064 4 -rw-r--r-- 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/suites/smoke/basic/tasks/0-install.yaml 2024-03-27T21:42:45.062 INFO:teuthology.orchestra.run.smithi060.stdout: 404413 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/overrides 2024-03-27T21:42:45.062 INFO:teuthology.orchestra.run.smithi060.stdout: 404415 4 -rw-r--r-- 1 ubuntu ubuntu 159 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/overrides/2-size-2-min-size.yaml 2024-03-27T21:42:45.062 INFO:teuthology.orchestra.run.smithi060.stdout: 404419 4 -rw-r--r-- 1 ubuntu ubuntu 64 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/overrides/no_client_pidfile.yaml 2024-03-27T21:42:45.063 INFO:teuthology.orchestra.run.smithi060.stdout: 404414 4 -rw-r--r-- 1 ubuntu ubuntu 113 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/overrides/2-size-1-min-size.yaml 2024-03-27T21:42:45.063 INFO:teuthology.orchestra.run.smithi060.stdout: 404416 4 -rw-r--r-- 1 ubuntu ubuntu 141 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/overrides/3-size-2-min-size.yaml 2024-03-27T21:42:45.063 INFO:teuthology.orchestra.run.smithi060.stdout: 404420 4 -rw-r--r-- 1 ubuntu ubuntu 20 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/overrides/nvme_loop.yaml 2024-03-27T21:42:45.063 INFO:teuthology.orchestra.run.smithi060.stdout: 404421 4 -rw-r--r-- 1 ubuntu ubuntu 142 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/overrides/short_pg_log.yaml 2024-03-27T21:42:45.063 INFO:teuthology.orchestra.run.smithi060.stdout: 404417 4 -rw-r--r-- 1 ubuntu ubuntu 170 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:45.063 INFO:teuthology.orchestra.run.smithi060.stdout: 404418 4 -rw-r--r-- 1 ubuntu ubuntu 121 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/overrides/more-active-recovery.yaml 2024-03-27T21:42:45.063 INFO:teuthology.orchestra.run.smithi060.stdout: 404515 4 -rwxr-xr-x 1 ubuntu ubuntu 665 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/runallonce.sh 2024-03-27T21:42:45.063 INFO:teuthology.orchestra.run.smithi060.stdout: 404385 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/msgr 2024-03-27T21:42:45.064 INFO:teuthology.orchestra.run.smithi060.stdout: 404387 4 -rw-r--r-- 1 ubuntu ubuntu 123 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/msgr/async-v2only.yaml 2024-03-27T21:42:45.064 INFO:teuthology.orchestra.run.smithi060.stdout: 404386 4 -rw-r--r-- 1 ubuntu ubuntu 121 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/msgr/async-v1only.yaml 2024-03-27T21:42:45.064 INFO:teuthology.orchestra.run.smithi060.stdout: 404388 4 -rw-r--r-- 1 ubuntu ubuntu 66 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/msgr/async.yaml 2024-03-27T21:42:45.064 INFO:teuthology.orchestra.run.smithi060.stdout: 404126 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/btrfs 2024-03-27T21:42:45.064 INFO:teuthology.orchestra.run.smithi060.stdout: 404130 4 -rw-r--r-- 1 ubuntu ubuntu 757 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/btrfs/create_async_snap.c 2024-03-27T21:42:45.064 INFO:teuthology.orchestra.run.smithi060.stdout: 404132 4 -rw-r--r-- 1 ubuntu ubuntu 1373 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/btrfs/test_rmdir_async_snap.c 2024-03-27T21:42:45.064 INFO:teuthology.orchestra.run.smithi060.stdout: 404131 4 -rw-r--r-- 1 ubuntu ubuntu 2148 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/btrfs/test_async_snap.c 2024-03-27T21:42:45.064 INFO:teuthology.orchestra.run.smithi060.stdout: 404129 4 -rw-r--r-- 1 ubuntu ubuntu 919 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/btrfs/clone_range.c 2024-03-27T21:42:45.064 INFO:teuthology.orchestra.run.smithi060.stdout: 404128 4 -rw-r--r-- 1 ubuntu ubuntu 167 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/btrfs/Makefile 2024-03-27T21:42:45.065 INFO:teuthology.orchestra.run.smithi060.stdout: 404127 4 -rw-r--r-- 1 ubuntu ubuntu 49 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/btrfs/.gitignore 2024-03-27T21:42:45.065 INFO:teuthology.orchestra.run.smithi060.stdout: 404517 4 -rwxr-xr-x 1 ubuntu ubuntu 129 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/runonkclient.sh 2024-03-27T21:42:45.065 INFO:teuthology.orchestra.run.smithi060.stdout: 404393 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/objectstore 2024-03-27T21:42:45.065 INFO:teuthology.orchestra.run.smithi060.stdout: 404399 4 -rw-r--r-- 1 ubuntu ubuntu 1238 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/objectstore/bluestore-hybrid.yaml 2024-03-27T21:42:45.065 INFO:teuthology.orchestra.run.smithi060.stdout: 404400 4 -rw-r--r-- 1 ubuntu ubuntu 807 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/objectstore/bluestore-low-osd-mem-target.yaml 2024-03-27T21:42:45.065 INFO:teuthology.orchestra.run.smithi060.stdout: 404396 4 -rw-r--r-- 1 ubuntu ubuntu 778 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/objectstore/bluestore-comp-snappy.yaml 2024-03-27T21:42:45.065 INFO:teuthology.orchestra.run.smithi060.stdout: 404397 4 -rw-r--r-- 1 ubuntu ubuntu 776 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/objectstore/bluestore-comp-zlib.yaml 2024-03-27T21:42:45.065 INFO:teuthology.orchestra.run.smithi060.stdout: 404394 4 -rw-r--r-- 1 ubuntu ubuntu 1339 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:45.066 INFO:teuthology.orchestra.run.smithi060.stdout: 404398 4 -rw-r--r-- 1 ubuntu ubuntu 776 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/objectstore/bluestore-comp-zstd.yaml 2024-03-27T21:42:45.066 INFO:teuthology.orchestra.run.smithi060.stdout: 404395 4 -rw-r--r-- 1 ubuntu ubuntu 775 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/objectstore/bluestore-comp-lz4.yaml 2024-03-27T21:42:45.066 INFO:teuthology.orchestra.run.smithi060.stdout: 404401 4 -rw-r--r-- 1 ubuntu ubuntu 1339 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/objectstore/bluestore-stupid.yaml 2024-03-27T21:42:45.066 INFO:teuthology.orchestra.run.smithi060.stdout: 404133 4 drwxr-xr-x 9 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs 2024-03-27T21:42:45.066 INFO:teuthology.orchestra.run.smithi060.stdout: 404202 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/overrides 2024-03-27T21:42:45.066 INFO:teuthology.orchestra.run.smithi060.stdout: 404216 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/overrides/prefetch_entire_dirfrags 2024-03-27T21:42:45.066 INFO:teuthology.orchestra.run.smithi060.stdout: 404218 4 -rw-r--r-- 1 ubuntu ubuntu 71 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/overrides/prefetch_entire_dirfrags/yes.yaml 2024-03-27T21:42:45.066 INFO:teuthology.orchestra.run.smithi060.stdout: 404217 4 -rw-r--r-- 1 ubuntu ubuntu 72 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/overrides/prefetch_entire_dirfrags/no.yaml 2024-03-27T21:42:45.067 INFO:teuthology.orchestra.run.smithi060.stdout: 404220 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/overrides/subvol_versions 2024-03-27T21:42:45.067 INFO:teuthology.orchestra.run.smithi060.stdout: 404221 4 -rw-r--r-- 1 ubuntu ubuntu 34 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/overrides/subvol_versions/create_subvol_version_v1.yaml 2024-03-27T21:42:45.067 INFO:teuthology.orchestra.run.smithi060.stdout: 404222 4 -rw-r--r-- 1 ubuntu ubuntu 34 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/overrides/subvol_versions/create_subvol_version_v2.yaml 2024-03-27T21:42:45.067 INFO:teuthology.orchestra.run.smithi060.stdout: 404203 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/overrides/.qa -> ../.qa/ 2024-03-27T21:42:45.067 INFO:teuthology.orchestra.run.smithi060.stdout: 404219 4 -rw-r--r-- 1 ubuntu ubuntu 58 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/overrides/session_timeout.yaml 2024-03-27T21:42:45.067 INFO:teuthology.orchestra.run.smithi060.stdout: 404215 4 -rw-r--r-- 1 ubuntu ubuntu 441 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/overrides/pg_health.yaml 2024-03-27T21:42:45.067 INFO:teuthology.orchestra.run.smithi060.stdout: 404214 4 -rw-r--r-- 1 ubuntu ubuntu 80 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/overrides/osd-asserts.yaml 2024-03-27T21:42:45.067 INFO:teuthology.orchestra.run.smithi060.stdout: 404212 4 -rw-r--r-- 1 ubuntu ubuntu 436 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/overrides/ignorelist_health.yaml 2024-03-27T21:42:45.068 INFO:teuthology.orchestra.run.smithi060.stdout: 404204 4 -rw-r--r-- 1 ubuntu ubuntu 174 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/overrides/frag.yaml 2024-03-27T21:42:45.068 INFO:teuthology.orchestra.run.smithi060.stdout: 404205 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/overrides/fuse 2024-03-27T21:42:45.068 INFO:teuthology.orchestra.run.smithi060.stdout: 404207 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/overrides/fuse/default-perm 2024-03-27T21:42:45.068 INFO:teuthology.orchestra.run.smithi060.stdout: 404208 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/overrides/fuse/default-perm/% 2024-03-27T21:42:45.068 INFO:teuthology.orchestra.run.smithi060.stdout: 404209 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/overrides/fuse/default-perm/.qa -> ../.qa/ 2024-03-27T21:42:45.068 INFO:teuthology.orchestra.run.smithi060.stdout: 404211 4 -rw-r--r-- 1 ubuntu ubuntu 82 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/overrides/fuse/default-perm/yes.yaml 2024-03-27T21:42:45.068 INFO:teuthology.orchestra.run.smithi060.stdout: 404210 4 -rw-r--r-- 1 ubuntu ubuntu 83 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/overrides/fuse/default-perm/no.yaml 2024-03-27T21:42:45.068 INFO:teuthology.orchestra.run.smithi060.stdout: 404206 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/overrides/fuse/.qa -> ../.qa/ 2024-03-27T21:42:45.069 INFO:teuthology.orchestra.run.smithi060.stdout: 404213 4 -rw-r--r-- 1 ubuntu ubuntu 233 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/overrides/ignorelist_wrongly_marked_down.yaml 2024-03-27T21:42:45.069 INFO:teuthology.orchestra.run.smithi060.stdout: 404134 0 lrwxrwxrwx 1 ubuntu ubuntu 3 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/.qa -> ../ 2024-03-27T21:42:45.069 INFO:teuthology.orchestra.run.smithi060.stdout: 404163 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/conf 2024-03-27T21:42:45.069 INFO:teuthology.orchestra.run.smithi060.stdout: 404164 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/conf/+ 2024-03-27T21:42:45.069 INFO:teuthology.orchestra.run.smithi060.stdout: 404165 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/conf/.qa -> ../.qa/ 2024-03-27T21:42:45.069 INFO:teuthology.orchestra.run.smithi060.stdout: 404166 4 -rw-r--r-- 1 ubuntu ubuntu 190 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/conf/client.yaml 2024-03-27T21:42:45.069 INFO:teuthology.orchestra.run.smithi060.stdout: 404167 4 -rw-r--r-- 1 ubuntu ubuntu 349 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/conf/mds.yaml 2024-03-27T21:42:45.069 INFO:teuthology.orchestra.run.smithi060.stdout: 404169 4 -rw-r--r-- 1 ubuntu ubuntu 178 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/conf/mon.yaml 2024-03-27T21:42:45.070 INFO:teuthology.orchestra.run.smithi060.stdout: 404170 4 -rw-r--r-- 1 ubuntu ubuntu 75 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/conf/osd.yaml 2024-03-27T21:42:45.070 INFO:teuthology.orchestra.run.smithi060.stdout: 404168 4 -rw-r--r-- 1 ubuntu ubuntu 107 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/conf/mgr.yaml 2024-03-27T21:42:45.070 INFO:teuthology.orchestra.run.smithi060.stdout: 404171 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount 2024-03-27T21:42:45.070 INFO:teuthology.orchestra.run.smithi060.stdout: 404172 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/.qa -> ../.qa/ 2024-03-27T21:42:45.070 INFO:teuthology.orchestra.run.smithi060.stdout: 404173 4 -rw-r--r-- 1 ubuntu ubuntu 282 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/fuse.yaml 2024-03-27T21:42:45.070 INFO:teuthology.orchestra.run.smithi060.stdout: 404174 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient 2024-03-27T21:42:45.070 INFO:teuthology.orchestra.run.smithi060.stdout: 404182 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient/overrides 2024-03-27T21:42:45.070 INFO:teuthology.orchestra.run.smithi060.stdout: 404183 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient/overrides/% 2024-03-27T21:42:45.071 INFO:teuthology.orchestra.run.smithi060.stdout: 404184 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient/overrides/.qa -> ../.qa/ 2024-03-27T21:42:45.071 INFO:teuthology.orchestra.run.smithi060.stdout: 404195 4 -rw-r--r-- 1 ubuntu ubuntu 84 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient/overrides/ms-die-on-skipped.yaml 2024-03-27T21:42:45.071 INFO:teuthology.orchestra.run.smithi060.stdout: 404185 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient/overrides/distro 2024-03-27T21:42:45.071 INFO:teuthology.orchestra.run.smithi060.stdout: 404187 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient/overrides/distro/stock 2024-03-27T21:42:45.071 INFO:teuthology.orchestra.run.smithi060.stdout: 404188 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient/overrides/distro/stock/% 2024-03-27T21:42:45.071 INFO:teuthology.orchestra.run.smithi060.stdout: 404189 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient/overrides/distro/stock/.qa -> ../.qa/ 2024-03-27T21:42:45.071 INFO:teuthology.orchestra.run.smithi060.stdout: 404191 4 -rw-r--r-- 1 ubuntu ubuntu 49 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient/overrides/distro/stock/k-stock.yaml 2024-03-27T21:42:45.071 INFO:teuthology.orchestra.run.smithi060.stdout: 404190 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient/overrides/distro/stock/centos_9.stream.yaml -> .qa/distros/all/centos_9.stream.yaml 2024-03-27T21:42:45.072 INFO:teuthology.orchestra.run.smithi060.stdout: 404186 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient/overrides/distro/.qa -> ../.qa/ 2024-03-27T21:42:45.072 INFO:teuthology.orchestra.run.smithi060.stdout: 404192 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient/overrides/distro/testing 2024-03-27T21:42:45.072 INFO:teuthology.orchestra.run.smithi060.stdout: 404194 4 -rw-r--r-- 1 ubuntu ubuntu 54 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient/overrides/distro/testing/k-testing.yaml 2024-03-27T21:42:45.072 INFO:teuthology.orchestra.run.smithi060.stdout: 404193 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient/overrides/distro/testing/.qa -> ../.qa/ 2024-03-27T21:42:45.072 INFO:teuthology.orchestra.run.smithi060.stdout: 404175 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient/% 2024-03-27T21:42:45.072 INFO:teuthology.orchestra.run.smithi060.stdout: 404176 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient/.qa -> ../.qa/ 2024-03-27T21:42:45.072 INFO:teuthology.orchestra.run.smithi060.stdout: 404177 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient/mount-syntax 2024-03-27T21:42:45.072 INFO:teuthology.orchestra.run.smithi060.stdout: 404179 4 -rw-r--r-- 1 ubuntu ubuntu 41 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient/mount-syntax/v1.yaml 2024-03-27T21:42:45.072 INFO:teuthology.orchestra.run.smithi060.stdout: 404180 4 -rw-r--r-- 1 ubuntu ubuntu 41 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient/mount-syntax/v2.yaml 2024-03-27T21:42:45.073 INFO:teuthology.orchestra.run.smithi060.stdout: 404178 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient/mount-syntax/$ 2024-03-27T21:42:45.073 INFO:teuthology.orchestra.run.smithi060.stdout: 404181 4 -rw-r--r-- 1 ubuntu ubuntu 342 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/mount/kclient/mount.yaml 2024-03-27T21:42:45.073 INFO:teuthology.orchestra.run.smithi060.stdout: 404196 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/objectstore-ec 2024-03-27T21:42:45.073 INFO:teuthology.orchestra.run.smithi060.stdout: 404201 4 -rw-r--r-- 1 ubuntu ubuntu 1250 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/objectstore-ec/bluestore-ec-root.yaml 2024-03-27T21:42:45.073 INFO:teuthology.orchestra.run.smithi060.stdout: 404199 4 -rw-r--r-- 1 ubuntu ubuntu 818 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/objectstore-ec/bluestore-comp-ec-root.yaml 2024-03-27T21:42:45.073 INFO:teuthology.orchestra.run.smithi060.stdout: 404197 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/objectstore-ec/.qa -> ../.qa/ 2024-03-27T21:42:45.073 INFO:teuthology.orchestra.run.smithi060.stdout: 404198 0 lrwxrwxrwx 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/objectstore-ec/bluestore-bitmap.yaml -> ../../objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:45.073 INFO:teuthology.orchestra.run.smithi060.stdout: 404200 4 -rw-r--r-- 1 ubuntu ubuntu 724 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/objectstore-ec/bluestore-comp.yaml 2024-03-27T21:42:45.074 INFO:teuthology.orchestra.run.smithi060.stdout: 404141 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters 2024-03-27T21:42:45.074 INFO:teuthology.orchestra.run.smithi060.stdout: 404159 4 -rw-r--r-- 1 ubuntu ubuntu 267 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters/1a5s-mds-1c-client.yaml 2024-03-27T21:42:45.074 INFO:teuthology.orchestra.run.smithi060.stdout: 404144 4 -rw-r--r-- 1 ubuntu ubuntu 167 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters/1-mds-1-client-micro.yaml 2024-03-27T21:42:45.074 INFO:teuthology.orchestra.run.smithi060.stdout: 404142 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters/.qa -> ../.qa/ 2024-03-27T21:42:45.074 INFO:teuthology.orchestra.run.smithi060.stdout: 404154 4 -rw-r--r-- 1 ubuntu ubuntu 286 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters/1a2s-mds-1c-client-3node.yaml 2024-03-27T21:42:45.074 INFO:teuthology.orchestra.run.smithi060.stdout: 404158 4 -rw-r--r-- 1 ubuntu ubuntu 307 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters/1a5s-mds-1c-client-3node.yaml 2024-03-27T21:42:45.074 INFO:teuthology.orchestra.run.smithi060.stdout: 404143 4 -rw-r--r-- 1 ubuntu ubuntu 239 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters/1-mds-1-client-coloc.yaml 2024-03-27T21:42:45.074 INFO:teuthology.orchestra.run.smithi060.stdout: 404152 4 -rw-r--r-- 1 ubuntu ubuntu 157 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters/1-node-1-mds-1-osd.yaml 2024-03-27T21:42:45.075 INFO:teuthology.orchestra.run.smithi060.stdout: 404149 4 -rw-r--r-- 1 ubuntu ubuntu 268 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters/1-mds-3-client.yaml 2024-03-27T21:42:45.075 INFO:teuthology.orchestra.run.smithi060.stdout: 404148 4 -rw-r--r-- 1 ubuntu ubuntu 262 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters/1-mds-2-client.yaml 2024-03-27T21:42:45.075 INFO:teuthology.orchestra.run.smithi060.stdout: 404147 4 -rw-r--r-- 1 ubuntu ubuntu 201 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters/1-mds-2-client-micro.yaml 2024-03-27T21:42:45.075 INFO:teuthology.orchestra.run.smithi060.stdout: 404161 4 -rw-r--r-- 1 ubuntu ubuntu 349 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters/9-mds.yaml 2024-03-27T21:42:45.075 INFO:teuthology.orchestra.run.smithi060.stdout: 404160 4 -rw-r--r-- 1 ubuntu ubuntu 307 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters/3-mds.yaml 2024-03-27T21:42:45.075 INFO:teuthology.orchestra.run.smithi060.stdout: 404151 4 -rw-r--r-- 1 ubuntu ubuntu 288 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters/1-mds-4-client.yaml 2024-03-27T21:42:45.075 INFO:teuthology.orchestra.run.smithi060.stdout: 404145 4 -rw-r--r-- 1 ubuntu ubuntu 249 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters/1-mds-1-client.yaml 2024-03-27T21:42:45.075 INFO:teuthology.orchestra.run.smithi060.stdout: 404153 4 -rw-r--r-- 1 ubuntu ubuntu 349 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters/1a11s-mds-1c-client-3node.yaml 2024-03-27T21:42:45.075 INFO:teuthology.orchestra.run.smithi060.stdout: 404150 4 -rw-r--r-- 1 ubuntu ubuntu 269 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters/1-mds-4-client-coloc.yaml 2024-03-27T21:42:45.076 INFO:teuthology.orchestra.run.smithi060.stdout: 404155 4 -rw-r--r-- 1 ubuntu ubuntu 253 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters/1a3s-mds-1c-client.yaml 2024-03-27T21:42:45.076 INFO:teuthology.orchestra.run.smithi060.stdout: 404146 4 -rw-r--r-- 1 ubuntu ubuntu 249 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters/1-mds-2-client-coloc.yaml 2024-03-27T21:42:45.076 INFO:teuthology.orchestra.run.smithi060.stdout: 404156 4 -rw-r--r-- 1 ubuntu ubuntu 263 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters/1a3s-mds-2c-client.yaml 2024-03-27T21:42:45.076 INFO:teuthology.orchestra.run.smithi060.stdout: 404162 4 -rw-r--r-- 1 ubuntu ubuntu 239 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters/fixed-2-ucephfs.yaml 2024-03-27T21:42:45.076 INFO:teuthology.orchestra.run.smithi060.stdout: 404157 4 -rw-r--r-- 1 ubuntu ubuntu 283 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/clusters/1a3s-mds-4c-client.yaml 2024-03-27T21:42:45.076 INFO:teuthology.orchestra.run.smithi060.stdout: 404135 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/begin 2024-03-27T21:42:45.076 INFO:teuthology.orchestra.run.smithi060.stdout: 404136 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/begin/+ 2024-03-27T21:42:45.076 INFO:teuthology.orchestra.run.smithi060.stdout: 404139 4 -rw-r--r-- 1 ubuntu ubuntu 44 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/begin/2-logrotate.yaml 2024-03-27T21:42:45.076 INFO:teuthology.orchestra.run.smithi060.stdout: 404138 4 -rw-r--r-- 1 ubuntu ubuntu 90 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/begin/1-ceph.yaml 2024-03-27T21:42:45.077 INFO:teuthology.orchestra.run.smithi060.stdout: 404140 4 -rw-r--r-- 1 ubuntu ubuntu 827 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/begin/3-modules.yaml 2024-03-27T21:42:45.077 INFO:teuthology.orchestra.run.smithi060.stdout: 404137 4 -rw-r--r-- 1 ubuntu ubuntu 1431 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/begin/0-install.yaml 2024-03-27T21:42:45.077 INFO:teuthology.orchestra.run.smithi060.stdout: 404231 20 -rwxr-xr-x 1 ubuntu ubuntu 17928 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/unshare_ns_mount.sh 2024-03-27T21:42:45.077 INFO:teuthology.orchestra.run.smithi060.stdout: 404223 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/tasks 2024-03-27T21:42:45.077 INFO:teuthology.orchestra.run.smithi060.stdout: 404225 4 -rw-r--r-- 1 ubuntu ubuntu 141 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/tasks/cfuse_workunit_suites_blogbench.yaml 2024-03-27T21:42:45.077 INFO:teuthology.orchestra.run.smithi060.stdout: 404224 0 lrwxrwxrwx 1 ubuntu ubuntu 7 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/tasks/.qa -> ../.qa/ 2024-03-27T21:42:45.077 INFO:teuthology.orchestra.run.smithi060.stdout: 404229 4 -rw-r--r-- 1 ubuntu ubuntu 77 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/tasks/cfuse_workunit_trivial_sync.yaml 2024-03-27T21:42:45.077 INFO:teuthology.orchestra.run.smithi060.stdout: 404228 4 -rw-r--r-- 1 ubuntu ubuntu 88 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/tasks/cfuse_workunit_suites_fsstress.yaml 2024-03-27T21:42:45.078 INFO:teuthology.orchestra.run.smithi060.stdout: 404226 4 -rw-r--r-- 1 ubuntu ubuntu 70 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/tasks/cfuse_workunit_suites_dbench.yaml 2024-03-27T21:42:45.078 INFO:teuthology.orchestra.run.smithi060.stdout: 404230 4 -rw-r--r-- 1 ubuntu ubuntu 217 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/tasks/libcephfs_interface_tests.yaml 2024-03-27T21:42:45.078 INFO:teuthology.orchestra.run.smithi060.stdout: 404227 4 -rw-r--r-- 1 ubuntu ubuntu 261 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/cephfs/tasks/cfuse_workunit_suites_ffsb.yaml 2024-03-27T21:42:45.078 INFO:teuthology.orchestra.run.smithi060.stdout: 404505 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rgw_frontend 2024-03-27T21:42:45.078 INFO:teuthology.orchestra.run.smithi060.stdout: 404506 4 -rw-r--r-- 1 ubuntu ubuntu 38 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rgw_frontend/beast.yaml 2024-03-27T21:42:45.078 INFO:teuthology.orchestra.run.smithi060.stdout: 404118 4 -rw-r--r-- 1 ubuntu ubuntu 227 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/CMakeLists.txt 2024-03-27T21:42:45.078 INFO:teuthology.orchestra.run.smithi060.stdout: 404117 0 lrwxrwxrwx 1 ubuntu ubuntu 1 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/.qa -> . 2024-03-27T21:42:45.078 INFO:teuthology.orchestra.run.smithi060.stdout: 404502 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rgw_bucket_sharding 2024-03-27T21:42:45.079 INFO:teuthology.orchestra.run.smithi060.stdout: 404504 4 -rw-r--r-- 1 ubuntu ubuntu 91 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rgw_bucket_sharding/single.yaml 2024-03-27T21:42:45.079 INFO:teuthology.orchestra.run.smithi060.stdout: 404503 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rgw_bucket_sharding/default.yaml 2024-03-27T21:42:45.079 INFO:teuthology.orchestra.run.smithi060.stdout: 404511 4 -rwxr-xr-x 1 ubuntu ubuntu 3517 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/run-standalone.sh 2024-03-27T21:42:45.079 INFO:teuthology.orchestra.run.smithi060.stdout: 404359 4 -rwxr-xr-x 1 ubuntu ubuntu 689 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/loopall.sh 2024-03-27T21:42:45.079 INFO:teuthology.orchestra.run.smithi060.stdout: 404367 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mgr_ttl_cache 2024-03-27T21:42:45.079 INFO:teuthology.orchestra.run.smithi060.stdout: 404369 4 -rw-r--r-- 1 ubuntu ubuntu 80 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mgr_ttl_cache/enable.yaml 2024-03-27T21:42:45.079 INFO:teuthology.orchestra.run.smithi060.stdout: 404368 4 -rw-r--r-- 1 ubuntu ubuntu 80 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mgr_ttl_cache/disable.yaml 2024-03-27T21:42:45.079 INFO:teuthology.orchestra.run.smithi060.stdout: 404360 4 -rw-r--r-- 1 ubuntu ubuntu 699 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/lsan.supp 2024-03-27T21:42:45.079 INFO:teuthology.orchestra.run.smithi060.stdout: 404370 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mon 2024-03-27T21:42:45.080 INFO:teuthology.orchestra.run.smithi060.stdout: 404371 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mon/bootstrap 2024-03-27T21:42:45.080 INFO:teuthology.orchestra.run.smithi060.stdout: 404374 4 -rwxr-xr-x 1 ubuntu ubuntu 1642 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mon/bootstrap/initial_members_asok.sh 2024-03-27T21:42:45.080 INFO:teuthology.orchestra.run.smithi060.stdout: 404380 4 -rwxr-xr-x 1 ubuntu ubuntu 482 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mon/bootstrap/single_host.sh 2024-03-27T21:42:45.080 INFO:teuthology.orchestra.run.smithi060.stdout: 404376 4 -rwxr-xr-x 1 ubuntu ubuntu 1495 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mon/bootstrap/simple_expand.sh 2024-03-27T21:42:45.080 INFO:teuthology.orchestra.run.smithi060.stdout: 404377 4 -rwxr-xr-x 1 ubuntu ubuntu 1084 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mon/bootstrap/simple_expand_monmap.sh 2024-03-27T21:42:45.080 INFO:teuthology.orchestra.run.smithi060.stdout: 404381 4 -rwxr-xr-x 1 ubuntu ubuntu 970 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mon/bootstrap/single_host_multi.sh 2024-03-27T21:42:45.080 INFO:teuthology.orchestra.run.smithi060.stdout: 404379 4 -rwxr-xr-x 1 ubuntu ubuntu 882 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mon/bootstrap/simple_single_expand2.sh 2024-03-27T21:42:45.080 INFO:teuthology.orchestra.run.smithi060.stdout: 404372 4 -rwxr-xr-x 1 ubuntu ubuntu 477 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mon/bootstrap/host.sh 2024-03-27T21:42:45.080 INFO:teuthology.orchestra.run.smithi060.stdout: 404373 4 -rwxr-xr-x 1 ubuntu ubuntu 959 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mon/bootstrap/initial_members.sh 2024-03-27T21:42:45.081 INFO:teuthology.orchestra.run.smithi060.stdout: 404375 4 -rwxr-xr-x 1 ubuntu ubuntu 863 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mon/bootstrap/simple.sh 2024-03-27T21:42:45.081 INFO:teuthology.orchestra.run.smithi060.stdout: 404378 4 -rwxr-xr-x 1 ubuntu ubuntu 1193 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mon/bootstrap/simple_single_expand.sh 2024-03-27T21:42:45.081 INFO:teuthology.orchestra.run.smithi060.stdout: 404120 4 -rw-r--r-- 1 ubuntu ubuntu 2377 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/README 2024-03-27T21:42:45.081 INFO:teuthology.orchestra.run.smithi060.stdout: 404518 4 -rwxr-xr-x 1 ubuntu ubuntu 1640 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/setup-chroot.sh 2024-03-27T21:42:45.081 INFO:teuthology.orchestra.run.smithi060.stdout: 404516 4 -rwxr-xr-x 1 ubuntu ubuntu 118 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/runoncfuse.sh 2024-03-27T21:42:45.081 INFO:teuthology.orchestra.run.smithi060.stdout: 408604 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/timezone 2024-03-27T21:42:45.081 INFO:teuthology.orchestra.run.smithi060.stdout: 408606 4 -rw-r--r-- 1 ubuntu ubuntu 82 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/timezone/pacific.yaml 2024-03-27T21:42:45.081 INFO:teuthology.orchestra.run.smithi060.stdout: 408605 4 -rw-r--r-- 1 ubuntu ubuntu 79 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/timezone/eastern.yaml 2024-03-27T21:42:45.082 INFO:teuthology.orchestra.run.smithi060.stdout: 408607 4 -rw-r--r-- 1 ubuntu ubuntu 162 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/timezone/random.yaml 2024-03-27T21:42:45.082 INFO:teuthology.orchestra.run.smithi060.stdout: 404485 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/releases 2024-03-27T21:42:45.082 INFO:teuthology.orchestra.run.smithi060.stdout: 404491 4 -rw-r--r-- 1 ubuntu ubuntu 138 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/releases/mimic.yaml 2024-03-27T21:42:45.082 INFO:teuthology.orchestra.run.smithi060.stdout: 404495 4 -rw-r--r-- 1 ubuntu ubuntu 233 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/releases/pacific.yaml 2024-03-27T21:42:45.082 INFO:teuthology.orchestra.run.smithi060.stdout: 404498 4 -rw-r--r-- 1 ubuntu ubuntu 138 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/releases/squid.yaml 2024-03-27T21:42:45.082 INFO:teuthology.orchestra.run.smithi060.stdout: 404497 4 -rw-r--r-- 1 ubuntu ubuntu 136 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/releases/reef.yaml 2024-03-27T21:42:45.082 INFO:teuthology.orchestra.run.smithi060.stdout: 404488 4 -rw-r--r-- 1 ubuntu ubuntu 67 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/releases/kraken.yaml 2024-03-27T21:42:45.082 INFO:teuthology.orchestra.run.smithi060.stdout: 404493 4 -rw-r--r-- 1 ubuntu ubuntu 233 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/releases/octopus.yaml 2024-03-27T21:42:45.083 INFO:teuthology.orchestra.run.smithi060.stdout: 404496 4 -rw-r--r-- 1 ubuntu ubuntu 140 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/releases/quincy.yaml 2024-03-27T21:42:45.083 INFO:teuthology.orchestra.run.smithi060.stdout: 404490 4 -rw-r--r-- 1 ubuntu ubuntu 538 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/releases/luminous.yaml 2024-03-27T21:42:45.083 INFO:teuthology.orchestra.run.smithi060.stdout: 404487 4 -rw-r--r-- 1 ubuntu ubuntu 188 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/releases/jewel.yaml 2024-03-27T21:42:45.083 INFO:teuthology.orchestra.run.smithi060.stdout: 404489 4 -rw-r--r-- 1 ubuntu ubuntu 238 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/releases/luminous-with-mgr.yaml 2024-03-27T21:42:45.083 INFO:teuthology.orchestra.run.smithi060.stdout: 404494 4 -rw-r--r-- 1 ubuntu ubuntu 142 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/releases/pacific-from-o.yaml 2024-03-27T21:42:45.083 INFO:teuthology.orchestra.run.smithi060.stdout: 404492 4 -rw-r--r-- 1 ubuntu ubuntu 233 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/releases/nautilus.yaml 2024-03-27T21:42:45.083 INFO:teuthology.orchestra.run.smithi060.stdout: 404486 4 -rw-r--r-- 1 ubuntu ubuntu 148 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/releases/infernalis.yaml 2024-03-27T21:42:45.083 INFO:teuthology.orchestra.run.smithi060.stdout: 404390 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/nightlies 2024-03-27T21:42:45.083 INFO:teuthology.orchestra.run.smithi060.stdout: 404391 4 -rwxr-xr-x 1 ubuntu ubuntu 439 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/nightlies/cron_wrapper 2024-03-27T21:42:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: 404392 4 -rwxr-xr-x 1 ubuntu ubuntu 295 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/nightlies/schedule_subset.sh 2024-03-27T21:42:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: 404249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/debug 2024-03-27T21:42:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: 404254 4 -rw-r--r-- 1 ubuntu ubuntu 46 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/debug/openstack-30G.yaml 2024-03-27T21:42:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: 404250 4 -rw-r--r-- 1 ubuntu ubuntu 114 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/debug/buildpackages.yaml 2024-03-27T21:42:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: 404251 4 -rw-r--r-- 1 ubuntu ubuntu 140 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/debug/mds_client.yaml 2024-03-27T21:42:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: 404252 4 -rw-r--r-- 1 ubuntu ubuntu 290 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/debug/mgr.yaml 2024-03-27T21:42:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: 404253 4 -rw-r--r-- 1 ubuntu ubuntu 46 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/debug/openstack-15G.yaml 2024-03-27T21:42:45.084 INFO:teuthology.orchestra.run.smithi060.stdout: 404499 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rgw 2024-03-27T21:42:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: 404501 4 -rw-r--r-- 1 ubuntu ubuntu 95 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rgw/s3tests-branch.yaml 2024-03-27T21:42:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: 404500 4 -rw-r--r-- 1 ubuntu ubuntu 316 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rgw/ignore-pg-availability.yaml 2024-03-27T21:42:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: 404389 4 -rw-r--r-- 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mypy.ini 2024-03-27T21:42:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: 408608 4 -rw-r--r-- 1 ubuntu ubuntu 939 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tox.ini 2024-03-27T21:42:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: 404507 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rgw_pool_type 2024-03-27T21:42:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: 404508 4 -rw-r--r-- 1 ubuntu ubuntu 179 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rgw_pool_type/ec-profile.yaml 2024-03-27T21:42:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: 404510 4 -rw-r--r-- 1 ubuntu ubuntu 42 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rgw_pool_type/replicated.yaml 2024-03-27T21:42:45.085 INFO:teuthology.orchestra.run.smithi060.stdout: 404509 4 -rw-r--r-- 1 ubuntu ubuntu 75 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rgw_pool_type/ec.yaml 2024-03-27T21:42:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: 404514 4 -rw-r--r-- 1 ubuntu ubuntu 1050 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/run_xfstests_qemu.sh 2024-03-27T21:42:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: 408609 16 -rw-r--r-- 1 ubuntu ubuntu 13176 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/valgrind.supp 2024-03-27T21:42:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: 404236 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/clusters 2024-03-27T21:42:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: 404242 4 -rw-r--r-- 1 ubuntu ubuntu 278 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/clusters/fixed-3.yaml 2024-03-27T21:42:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: 404239 4 -rw-r--r-- 1 ubuntu ubuntu 321 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/clusters/fixed-1.yaml 2024-03-27T21:42:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: 404240 4 -rw-r--r-- 1 ubuntu ubuntu 333 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/clusters/fixed-2.yaml 2024-03-27T21:42:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: 404238 4 -rw-r--r-- 1 ubuntu ubuntu 276 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/clusters/extra-client.yaml 2024-03-27T21:42:45.086 INFO:teuthology.orchestra.run.smithi060.stdout: 404237 4 -rw-r--r-- 1 ubuntu ubuntu 255 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/clusters/2-node-mgr.yaml 2024-03-27T21:42:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: 404241 4 -rw-r--r-- 1 ubuntu ubuntu 308 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/clusters/fixed-3-cephfs.yaml 2024-03-27T21:42:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: 404243 4 -rw-r--r-- 1 ubuntu ubuntu 345 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/clusters/fixed-4.yaml 2024-03-27T21:42:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: 404255 4 drwxr-xr-x 9 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros 2024-03-27T21:42:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: 404257 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all 2024-03-27T21:42:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: 404287 4 -rw-r--r-- 1 ubuntu ubuntu 46 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/rhel_6.3.yaml 2024-03-27T21:42:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: 404284 4 -rw-r--r-- 1 ubuntu ubuntu 51 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/opensuse_15.1.yaml 2024-03-27T21:42:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: 404269 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/centos_7.yaml -> centos_7.6.yaml 2024-03-27T21:42:45.087 INFO:teuthology.orchestra.run.smithi060.stdout: 404294 4 -rw-r--r-- 1 ubuntu ubuntu 133 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/rhel_7.7.yaml 2024-03-27T21:42:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: 404289 4 -rw-r--r-- 1 ubuntu ubuntu 46 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/rhel_6.5.yaml 2024-03-27T21:42:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: 404292 4 -rw-r--r-- 1 ubuntu ubuntu 46 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/rhel_7.5.yaml 2024-03-27T21:42:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: 404312 4 -rw-r--r-- 1 ubuntu ubuntu 50 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/ubuntu_16.04.yaml 2024-03-27T21:42:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: 404262 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/centos_7.0.yaml 2024-03-27T21:42:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: 404303 4 -rw-r--r-- 1 ubuntu ubuntu 46 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/sle_12.2.yaml 2024-03-27T21:42:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: 404300 4 -rw-r--r-- 1 ubuntu ubuntu 133 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/rhel_8.5.yaml 2024-03-27T21:42:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: 404265 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/centos_7.3.yaml 2024-03-27T21:42:45.088 INFO:teuthology.orchestra.run.smithi060.stdout: 404302 0 lrwxrwxrwx 1 ubuntu ubuntu 13 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/rhel_8.yaml -> rhel_8.6.yaml 2024-03-27T21:42:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: 404306 4 -rw-r--r-- 1 ubuntu ubuntu 46 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/sle_15.2.yaml 2024-03-27T21:42:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: 404296 4 -rw-r--r-- 1 ubuntu ubuntu 133 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/rhel_8.0.yaml 2024-03-27T21:42:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: 404295 0 lrwxrwxrwx 1 ubuntu ubuntu 13 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/rhel_7.yaml -> rhel_7.7.yaml 2024-03-27T21:42:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: 404316 0 lrwxrwxrwx 1 ubuntu ubuntu 17 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/ubuntu_latest.yaml -> ubuntu_22.04.yaml 2024-03-27T21:42:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: 404278 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/debian_6.0.yaml 2024-03-27T21:42:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: 404279 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/debian_7.0.yaml 2024-03-27T21:42:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: 404282 4 -rw-r--r-- 1 ubuntu ubuntu 47 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/fedora_18.yaml 2024-03-27T21:42:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: 404285 4 -rw-r--r-- 1 ubuntu ubuntu 51 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/opensuse_15.2.yaml 2024-03-27T21:42:45.089 INFO:teuthology.orchestra.run.smithi060.stdout: 404267 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/centos_7.5.yaml 2024-03-27T21:42:45.090 INFO:teuthology.orchestra.run.smithi060.stdout: 404310 4 -rw-r--r-- 1 ubuntu ubuntu 64 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/ubuntu_14.04_aarch64.yaml 2024-03-27T21:42:45.090 INFO:teuthology.orchestra.run.smithi060.stdout: 404288 4 -rw-r--r-- 1 ubuntu ubuntu 46 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/rhel_6.4.yaml 2024-03-27T21:42:45.090 INFO:teuthology.orchestra.run.smithi060.stdout: 404271 4 -rw-r--r-- 1 ubuntu ubuntu 135 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/centos_8.1.yaml 2024-03-27T21:42:45.090 INFO:teuthology.orchestra.run.smithi060.stdout: 404258 4 -rw-r--r-- 1 ubuntu ubuntu 30 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/centos.yaml 2024-03-27T21:42:45.090 INFO:teuthology.orchestra.run.smithi060.stdout: 404272 4 -rw-r--r-- 1 ubuntu ubuntu 135 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/centos_8.2.yaml 2024-03-27T21:42:45.090 INFO:teuthology.orchestra.run.smithi060.stdout: 404307 4 -rw-r--r-- 1 ubuntu ubuntu 50 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/ubuntu_12.04.yaml 2024-03-27T21:42:45.090 INFO:teuthology.orchestra.run.smithi060.stdout: 404313 4 -rw-r--r-- 1 ubuntu ubuntu 50 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/ubuntu_18.04.yaml 2024-03-27T21:42:45.090 INFO:teuthology.orchestra.run.smithi060.stdout: 404276 4 -rw-r--r-- 1 ubuntu ubuntu 122 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/centos_9.stream.yaml 2024-03-27T21:42:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: 404283 4 -rw-r--r-- 1 ubuntu ubuntu 47 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/fedora_19.yaml 2024-03-27T21:42:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: 404270 4 -rw-r--r-- 1 ubuntu ubuntu 135 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/centos_8.0.yaml 2024-03-27T21:42:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: 404259 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/centos_6.3.yaml 2024-03-27T21:42:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: 404263 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/centos_7.1.yaml 2024-03-27T21:42:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: 404260 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/centos_6.4.yaml 2024-03-27T21:42:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: 404311 4 -rw-r--r-- 1 ubuntu ubuntu 61 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/ubuntu_14.04_i686.yaml 2024-03-27T21:42:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: 404286 4 -rw-r--r-- 1 ubuntu ubuntu 51 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/opensuse_42.3.yaml 2024-03-27T21:42:45.091 INFO:teuthology.orchestra.run.smithi060.stdout: 404304 4 -rw-r--r-- 1 ubuntu ubuntu 46 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/sle_12.3.yaml 2024-03-27T21:42:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: 404301 4 -rw-r--r-- 1 ubuntu ubuntu 133 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/rhel_8.6.yaml 2024-03-27T21:42:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: 404277 0 lrwxrwxrwx 1 ubuntu ubuntu 20 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/centos_latest.yaml -> centos_9.stream.yaml 2024-03-27T21:42:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: 404264 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/centos_7.2.yaml 2024-03-27T21:42:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: 404290 0 lrwxrwxrwx 1 ubuntu ubuntu 13 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/rhel_6.yaml -> rhel_6.5.yaml 2024-03-27T21:42:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: 404275 0 lrwxrwxrwx 1 ubuntu ubuntu 20 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/centos_8.yaml -> centos_8.stream.yaml 2024-03-27T21:42:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: 404314 4 -rw-r--r-- 1 ubuntu ubuntu 50 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/ubuntu_20.04.yaml 2024-03-27T21:42:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: 404293 4 -rw-r--r-- 1 ubuntu ubuntu 133 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/rhel_7.6.yaml 2024-03-27T21:42:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: 404291 4 -rw-r--r-- 1 ubuntu ubuntu 46 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/rhel_7.0.yaml 2024-03-27T21:42:45.092 INFO:teuthology.orchestra.run.smithi060.stdout: 404309 4 -rw-r--r-- 1 ubuntu ubuntu 50 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/ubuntu_14.04.yaml 2024-03-27T21:42:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: 404274 4 -rw-r--r-- 1 ubuntu ubuntu 140 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/centos_8.stream.yaml 2024-03-27T21:42:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: 404268 4 -rw-r--r-- 1 ubuntu ubuntu 135 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/centos_7.6.yaml 2024-03-27T21:42:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: 404308 4 -rw-r--r-- 1 ubuntu ubuntu 50 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/ubuntu_12.10.yaml 2024-03-27T21:42:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: 404297 4 -rw-r--r-- 1 ubuntu ubuntu 133 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/rhel_8.1.yaml 2024-03-27T21:42:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: 404305 4 -rw-r--r-- 1 ubuntu ubuntu 46 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/sle_15.1.yaml 2024-03-27T21:42:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: 404261 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/centos_6.5.yaml 2024-03-27T21:42:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: 404281 4 -rw-r--r-- 1 ubuntu ubuntu 47 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/fedora_17.yaml 2024-03-27T21:42:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: 404273 4 -rw-r--r-- 1 ubuntu ubuntu 135 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/centos_8.3.yaml 2024-03-27T21:42:45.093 INFO:teuthology.orchestra.run.smithi060.stdout: 404280 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/debian_8.0.yaml 2024-03-27T21:42:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: 404315 4 -rw-r--r-- 1 ubuntu ubuntu 50 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/ubuntu_22.04.yaml 2024-03-27T21:42:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: 404298 4 -rw-r--r-- 1 ubuntu ubuntu 133 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/rhel_8.3.yaml 2024-03-27T21:42:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: 404266 4 -rw-r--r-- 1 ubuntu ubuntu 48 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/centos_7.4.yaml 2024-03-27T21:42:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: 404299 4 -rw-r--r-- 1 ubuntu ubuntu 133 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/all/rhel_8.4.yaml 2024-03-27T21:42:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: 404324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/crimson-supported-all-distro 2024-03-27T21:42:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: 404325 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/crimson-supported-all-distro/centos_latest.yaml -> ../all/centos_latest.yaml 2024-03-27T21:42:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: 404256 0 lrwxrwxrwx 1 ubuntu ubuntu 2 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/.qa -> .. 2024-03-27T21:42:45.094 INFO:teuthology.orchestra.run.smithi060.stdout: 404317 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/container-hosts 2024-03-27T21:42:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: 404322 4 -rw-r--r-- 1 ubuntu ubuntu 378 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/container-hosts/centos_9.stream_runc.yaml 2024-03-27T21:42:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: 404318 0 lrwxrwxrwx 1 ubuntu ubuntu 6 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/container-hosts/.qa -> ../.qa 2024-03-27T21:42:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: 404320 4 -rw-r--r-- 1 ubuntu ubuntu 638 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/container-hosts/centos_8.stream_container_tools_crun.yaml 2024-03-27T21:42:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: 404319 4 -rw-r--r-- 1 ubuntu ubuntu 444 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/container-hosts/centos_8.stream_container_tools.yaml 2024-03-27T21:42:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: 404321 4 -rw-r--r-- 1 ubuntu ubuntu 127 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/container-hosts/centos_9.stream.yaml 2024-03-27T21:42:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: 404323 0 lrwxrwxrwx 1 ubuntu ubuntu 33 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/container-hosts/ubuntu_22.04.yaml -> .qa/distros/all/ubuntu_22.04.yaml 2024-03-27T21:42:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: 404338 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/supported 2024-03-27T21:42:45.095 INFO:teuthology.orchestra.run.smithi060.stdout: 404340 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/supported/ubuntu_latest.yaml -> ../all/ubuntu_latest.yaml 2024-03-27T21:42:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: 404339 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/supported/centos_latest.yaml -> ../all/centos_latest.yaml 2024-03-27T21:42:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: 404334 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/supported-random-distro$ 2024-03-27T21:42:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: 404337 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/supported-random-distro$/ubuntu_latest.yaml -> ../all/ubuntu_latest.yaml 2024-03-27T21:42:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: 404336 0 lrwxrwxrwx 1 ubuntu ubuntu 25 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/supported-random-distro$/centos_latest.yaml -> ../all/centos_latest.yaml 2024-03-27T21:42:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: 404335 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/supported-random-distro$/$ 2024-03-27T21:42:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: 404326 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/podman 2024-03-27T21:42:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: 404327 4 -rw-r--r-- 1 ubuntu ubuntu 444 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/podman/centos_8.stream_container_tools.yaml 2024-03-27T21:42:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: 404328 4 -rw-r--r-- 1 ubuntu ubuntu 172 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/podman/centos_9.stream.yaml 2024-03-27T21:42:45.096 INFO:teuthology.orchestra.run.smithi060.stdout: 404329 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/single-container-host.yaml -> container-hosts/centos_9.stream.yaml 2024-03-27T21:42:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: 404330 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/supported-container-hosts 2024-03-27T21:42:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: 404332 0 lrwxrwxrwx 1 ubuntu ubuntu 44 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/supported-container-hosts/centos_9.stream_runc.yaml -> ../container-hosts/centos_9.stream_runc.yaml 2024-03-27T21:42:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: 404331 0 lrwxrwxrwx 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/supported-container-hosts/centos_9.stream.yaml -> ../container-hosts/centos_9.stream.yaml 2024-03-27T21:42:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: 404333 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/distros/supported-container-hosts/ubuntu_22.04.yaml -> ../container-hosts/ubuntu_22.04.yaml 2024-03-27T21:42:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: 404119 4 -rw-r--r-- 1 ubuntu ubuntu 83 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/Makefile 2024-03-27T21:42:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: 408603 4 -rw-r--r-- 1 ubuntu ubuntu 1001 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/test_import.py 2024-03-27T21:42:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: 404519 4 drwxr-xr-x 13 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone 2024-03-27T21:42:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: 404562 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill 2024-03-27T21:42:45.097 INFO:teuthology.orchestra.run.smithi060.stdout: 404564 4 -rwxr-xr-x 1 ubuntu ubuntu 3941 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh 2024-03-27T21:42:45.098 INFO:teuthology.orchestra.run.smithi060.stdout: 404565 32 -rwxr-xr-x 1 ubuntu ubuntu 31780 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh 2024-03-27T21:42:45.101 INFO:teuthology.orchestra.run.smithi060.stdout: 404563 16 -rwxr-xr-x 1 ubuntu ubuntu 15875 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh 2024-03-27T21:42:45.101 INFO:teuthology.orchestra.run.smithi060.stdout: 404566 28 -rwxr-xr-x 1 ubuntu ubuntu 26343 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-stats.sh 2024-03-27T21:42:45.101 INFO:teuthology.orchestra.run.smithi060.stdout: 404531 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr 2024-03-27T21:42:45.101 INFO:teuthology.orchestra.run.smithi060.stdout: 404532 8 -rwxr-xr-x 1 ubuntu ubuntu 7323 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh 2024-03-27T21:42:45.101 INFO:teuthology.orchestra.run.smithi060.stdout: 404533 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc 2024-03-27T21:42:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: 404535 8 -rwxr-xr-x 1 ubuntu ubuntu 6542 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh 2024-03-27T21:42:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: 404536 12 -rwxr-xr-x 1 ubuntu ubuntu 8982 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh 2024-03-27T21:42:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: 404537 8 -rwxr-xr-x 1 ubuntu ubuntu 4113 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh 2024-03-27T21:42:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: 404534 20 -rwxr-xr-x 1 ubuntu ubuntu 18490 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh 2024-03-27T21:42:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: 404538 4 -rwxr-xr-x 1 ubuntu ubuntu 821 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 2024-03-27T21:42:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: 404539 8 -rwxr-xr-x 1 ubuntu ubuntu 5635 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh 2024-03-27T21:42:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: 404540 12 -rwxr-xr-x 1 ubuntu ubuntu 9707 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh 2024-03-27T21:42:45.102 INFO:teuthology.orchestra.run.smithi060.stdout: 404524 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/crush 2024-03-27T21:42:45.103 INFO:teuthology.orchestra.run.smithi060.stdout: 404525 8 -rwxr-xr-x 1 ubuntu ubuntu 7592 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/crush/crush-choose-args.sh 2024-03-27T21:42:45.103 INFO:teuthology.orchestra.run.smithi060.stdout: 404526 12 -rwxr-xr-x 1 ubuntu ubuntu 9876 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/crush/crush-classes.sh 2024-03-27T21:42:45.103 INFO:teuthology.orchestra.run.smithi060.stdout: 404544 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon 2024-03-27T21:42:45.103 INFO:teuthology.orchestra.run.smithi060.stdout: 404547 8 -rwxr-xr-x 1 ubuntu ubuntu 4454 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mkfs.sh 2024-03-27T21:42:45.103 INFO:teuthology.orchestra.run.smithi060.stdout: 404550 4 -rwxr-xr-x 1 ubuntu ubuntu 2612 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-handle-forward.sh 2024-03-27T21:42:45.103 INFO:teuthology.orchestra.run.smithi060.stdout: 404558 8 -rwxr-xr-x 1 ubuntu ubuntu 7883 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/osd-erasure-code-profile.sh 2024-03-27T21:42:45.103 INFO:teuthology.orchestra.run.smithi060.stdout: 404556 8 -rwxr-xr-x 1 ubuntu ubuntu 6754 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/osd-crush.sh 2024-03-27T21:42:45.103 INFO:teuthology.orchestra.run.smithi060.stdout: 404548 8 -rwxr-xr-x 1 ubuntu ubuntu 5024 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-bind.sh 2024-03-27T21:42:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: 404553 4 -rwxr-xr-x 1 ubuntu ubuntu 1270 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-ping.sh 2024-03-27T21:42:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: 404555 4 -rwxr-xr-x 1 ubuntu ubuntu 2144 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-seesaw.sh 2024-03-27T21:42:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: 404545 4 -rwxr-xr-x 1 ubuntu ubuntu 3385 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/health-mute.sh 2024-03-27T21:42:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: 404546 16 -rwxr-xr-x 1 ubuntu ubuntu 12753 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/misc.sh 2024-03-27T21:42:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: 404560 4 -rwxr-xr-x 1 ubuntu ubuntu 2547 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/osd-pool-df.sh 2024-03-27T21:42:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: 404551 8 -rwxr-xr-x 1 ubuntu ubuntu 7527 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-last-epoch-clean.sh 2024-03-27T21:42:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: 404559 12 -rwxr-xr-x 1 ubuntu ubuntu 10501 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/osd-pool-create.sh 2024-03-27T21:42:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: 404561 4 -rwxr-xr-x 1 ubuntu ubuntu 1496 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/test_pool_quota.sh 2024-03-27T21:42:45.104 INFO:teuthology.orchestra.run.smithi060.stdout: 404554 4 -rwxr-xr-x 1 ubuntu ubuntu 1398 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-scrub.sh 2024-03-27T21:42:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: 404552 4 -rwxr-xr-x 1 ubuntu ubuntu 1440 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-osdmap-prune.sh 2024-03-27T21:42:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: 404557 8 -rwxr-xr-x 1 ubuntu ubuntu 4107 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/osd-df.sh 2024-03-27T21:42:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: 404549 4 -rwxr-xr-x 1 ubuntu ubuntu 1548 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-created-time.sh 2024-03-27T21:42:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: 404520 4 -rw-r--r-- 1 ubuntu ubuntu 840 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/README 2024-03-27T21:42:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: 404523 72 -rwxr-xr-x 1 ubuntu ubuntu 73624 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-03-27T21:42:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: 404567 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd 2024-03-27T21:42:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: 404586 4 -rwxr-xr-x 1 ubuntu ubuntu 3773 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh 2024-03-27T21:42:45.105 INFO:teuthology.orchestra.run.smithi060.stdout: 404571 4 -rwxr-xr-x 1 ubuntu ubuntu 3021 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2024-03-27T21:42:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: 404573 4 -rwxr-xr-x 1 ubuntu ubuntu 3502 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh 2024-03-27T21:42:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: 404579 4 -rwxr-xr-x 1 ubuntu ubuntu 1537 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh 2024-03-27T21:42:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: 404576 4 -rwxr-xr-x 1 ubuntu ubuntu 3114 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh 2024-03-27T21:42:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: 404582 16 -rwxr-xr-x 1 ubuntu ubuntu 16055 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh 2024-03-27T21:42:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: 404587 8 -rwxr-xr-x 1 ubuntu ubuntu 6109 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh 2024-03-27T21:42:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: 404575 4 -rwxr-xr-x 1 ubuntu ubuntu 845 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh 2024-03-27T21:42:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: 404581 8 -rwxr-xr-x 1 ubuntu ubuntu 4533 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh 2024-03-27T21:42:45.106 INFO:teuthology.orchestra.run.smithi060.stdout: 404583 12 -rwxr-xr-x 1 ubuntu ubuntu 12222 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh 2024-03-27T21:42:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: 404578 8 -rwxr-xr-x 1 ubuntu ubuntu 4487 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh 2024-03-27T21:42:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: 404580 16 -rwxr-xr-x 1 ubuntu ubuntu 16123 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh 2024-03-27T21:42:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: 404577 4 -rwxr-xr-x 1 ubuntu ubuntu 1599 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh 2024-03-27T21:42:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: 404568 4 -rwxr-xr-x 1 ubuntu ubuntu 1360 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2024-03-27T21:42:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: 404570 4 -rwxr-xr-x 1 ubuntu ubuntu 1612 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2024-03-27T21:42:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: 404585 8 -rwxr-xr-x 1 ubuntu ubuntu 6885 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh 2024-03-27T21:42:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: 404572 16 -rwxr-xr-x 1 ubuntu ubuntu 16101 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2024-03-27T21:42:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: 404569 24 -rwxr-xr-x 1 ubuntu ubuntu 23442 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2024-03-27T21:42:45.107 INFO:teuthology.orchestra.run.smithi060.stdout: 404584 4 -rwxr-xr-x 1 ubuntu ubuntu 1581 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh 2024-03-27T21:42:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: 404574 4 -rwxr-xr-x 1 ubuntu ubuntu 1940 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh 2024-03-27T21:42:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: 404521 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/c2c 2024-03-27T21:42:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: 404522 4 -rwxr-xr-x 1 ubuntu ubuntu 3139 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/c2c/c2c.sh 2024-03-27T21:42:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: 404588 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub 2024-03-27T21:42:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: 404590 12 -rwxr-xr-x 1 ubuntu ubuntu 10421 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2024-03-27T21:42:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: 404594 24 -rwxr-xr-x 1 ubuntu ubuntu 22553 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh 2024-03-27T21:42:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: 404596 12 -rw-r--r-- 1 ubuntu ubuntu 10580 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh 2024-03-27T21:42:45.108 INFO:teuthology.orchestra.run.smithi060.stdout: 404589 8 -rwxr-xr-x 1 ubuntu ubuntu 6662 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2024-03-27T21:42:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: 404593 36 -rwxr-xr-x 1 ubuntu ubuntu 33715 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh 2024-03-27T21:42:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: 404592 172 -rwxr-xr-x 1 ubuntu ubuntu 173409 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2024-03-27T21:42:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: 404595 4 -rwxr-xr-x 1 ubuntu ubuntu 2466 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh 2024-03-27T21:42:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: 404591 8 -rwxr-xr-x 1 ubuntu ubuntu 5100 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2024-03-27T21:42:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: 404527 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code 2024-03-27T21:42:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: 404528 8 -rwxr-xr-x 1 ubuntu ubuntu 4144 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh 2024-03-27T21:42:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: 404529 12 -rwxr-xr-x 1 ubuntu ubuntu 10393 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh 2024-03-27T21:42:45.109 INFO:teuthology.orchestra.run.smithi060.stdout: 404530 20 -rwxr-xr-x 1 ubuntu ubuntu 19747 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh 2024-03-27T21:42:45.110 INFO:teuthology.orchestra.run.smithi060.stdout: 404597 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/special 2024-03-27T21:42:45.110 INFO:teuthology.orchestra.run.smithi060.stdout: 404598 92 -rwxr-xr-x 1 ubuntu ubuntu 92636 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/special/ceph_objectstore_tool.py 2024-03-27T21:42:45.110 INFO:teuthology.orchestra.run.smithi060.stdout: 404599 4 -rwxr-xr-x 1 ubuntu ubuntu 880 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/special/test-failure.sh 2024-03-27T21:42:45.110 INFO:teuthology.orchestra.run.smithi060.stdout: 404541 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch 2024-03-27T21:42:45.110 INFO:teuthology.orchestra.run.smithi060.stdout: 404542 8 -rwxr-xr-x 1 ubuntu ubuntu 4639 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh 2024-03-27T21:42:45.110 INFO:teuthology.orchestra.run.smithi060.stdout: 404543 8 -rwxr-xr-x 1 ubuntu ubuntu 4966 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh 2024-03-27T21:42:45.110 INFO:teuthology.orchestra.run.smithi060.stdout: 404513 8 -rwxr-xr-x 1 ubuntu ubuntu 8000 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/run_xfstests.sh 2024-03-27T21:42:45.110 INFO:teuthology.orchestra.run.smithi060.stdout: 404422 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/packages 2024-03-27T21:42:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: 404423 4 -rw-r--r-- 1 ubuntu ubuntu 1558 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/packages/packages.yaml 2024-03-27T21:42:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: 404424 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts 2024-03-27T21:42:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: 404425 4 -rwxr-xr-x 1 ubuntu ubuntu 991 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/cephscrub.sh 2024-03-27T21:42:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: 404426 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack 2024-03-27T21:42:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: 404428 4 -rwxr-xr-x 1 ubuntu ubuntu 326 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/ceph_install.sh 2024-03-27T21:42:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: 404452 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/files 2024-03-27T21:42:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: 404454 56 -rw-r--r-- 1 ubuntu ubuntu 55353 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/files/glance-api.template.conf 2024-03-27T21:42:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: 404453 120 -rw-r--r-- 1 ubuntu ubuntu 121456 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/files/cinder.template.conf 2024-03-27T21:42:45.111 INFO:teuthology.orchestra.run.smithi060.stdout: 404455 40 -rw-r--r-- 1 ubuntu ubuntu 39814 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/files/kilo.template.conf 2024-03-27T21:42:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: 404456 120 -rw-r--r-- 1 ubuntu ubuntu 122641 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/files/nova.template.conf 2024-03-27T21:42:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: 404444 4 -rwxr-xr-x 1 ubuntu ubuntu 731 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/copy_func.sh 2024-03-27T21:42:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: 404443 4 -rwxr-xr-x 1 ubuntu ubuntu 2662 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/connectceph.sh 2024-03-27T21:42:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: 404427 4 -rw-r--r-- 1 ubuntu ubuntu 1561 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/README 2024-03-27T21:42:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: 404457 4 -rwxr-xr-x 1 ubuntu ubuntu 999 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/fix_conf_file.sh 2024-03-27T21:42:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: 404429 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/ceph_install_w_ansible 2024-03-27T21:42:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: 404431 4 -rwxr-xr-x 1 ubuntu ubuntu 1184 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/ceph_install_w_ansible/ceph_install.sh 2024-03-27T21:42:45.112 INFO:teuthology.orchestra.run.smithi060.stdout: 404440 4 -rwxr-xr-x 1 ubuntu ubuntu 760 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/ceph_install_w_ansible/repolocs.sh 2024-03-27T21:42:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: 404433 0 lrwxrwxrwx 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/ceph_install_w_ansible/copy_func.sh -> ../copy_func.sh 2024-03-27T21:42:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: 404430 4 -rw-r--r-- 1 ubuntu ubuntu 1346 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/ceph_install_w_ansible/README 2024-03-27T21:42:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: 404439 4 -rwxr-xr-x 1 ubuntu ubuntu 388 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/ceph_install_w_ansible/multi_action.sh 2024-03-27T21:42:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: 404442 4 -rwxr-xr-x 1 ubuntu ubuntu 884 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/ceph_install_w_ansible/talknice.sh 2024-03-27T21:42:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: 404441 4 -rwxr-xr-x 1 ubuntu ubuntu 277 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/ceph_install_w_ansible/staller.sh 2024-03-27T21:42:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: 404434 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/ceph_install_w_ansible/execs 2024-03-27T21:42:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: 404436 4 -rwxr-xr-x 1 ubuntu ubuntu 1464 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/ceph_install_w_ansible/execs/ceph_ansible.sh 2024-03-27T21:42:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: 404438 4 -rwxr-xr-x 1 ubuntu ubuntu 142 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/ceph_install_w_ansible/execs/edit_groupvars_osds.sh 2024-03-27T21:42:45.113 INFO:teuthology.orchestra.run.smithi060.stdout: 404437 4 -rwxr-xr-x 1 ubuntu ubuntu 101 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/ceph_install_w_ansible/execs/edit_ansible_hosts.sh 2024-03-27T21:42:45.114 INFO:teuthology.orchestra.run.smithi060.stdout: 404435 4 -rwxr-xr-x 1 ubuntu ubuntu 708 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/ceph_install_w_ansible/execs/cdn_setup.sh 2024-03-27T21:42:45.114 INFO:teuthology.orchestra.run.smithi060.stdout: 404432 4 -rw-r--r-- 1 ubuntu ubuntu 203 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/ceph_install_w_ansible/config 2024-03-27T21:42:45.114 INFO:teuthology.orchestra.run.smithi060.stdout: 404458 4 -rwxr-xr-x 1 ubuntu ubuntu 491 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/image_create.sh 2024-03-27T21:42:45.114 INFO:teuthology.orchestra.run.smithi060.stdout: 404459 4 -rwxr-xr-x 1 ubuntu ubuntu 763 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/openstack.sh 2024-03-27T21:42:45.114 INFO:teuthology.orchestra.run.smithi060.stdout: 404445 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/execs 2024-03-27T21:42:45.114 INFO:teuthology.orchestra.run.smithi060.stdout: 404447 4 -rwxr-xr-x 1 ubuntu ubuntu 1092 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/execs/ceph_cluster.sh 2024-03-27T21:42:45.114 INFO:teuthology.orchestra.run.smithi060.stdout: 404448 4 -rwxr-xr-x 1 ubuntu ubuntu 422 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/execs/libvirt-secret.sh 2024-03-27T21:42:45.114 INFO:teuthology.orchestra.run.smithi060.stdout: 404450 4 -rwxr-xr-x 1 ubuntu ubuntu 986 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/execs/run_openstack.sh 2024-03-27T21:42:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: 404449 4 -rwxr-xr-x 1 ubuntu ubuntu 554 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/execs/openstack-preinstall.sh 2024-03-27T21:42:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: 404451 4 -rwxr-xr-x 1 ubuntu ubuntu 415 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/execs/start_openstack.sh 2024-03-27T21:42:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: 404446 4 -rwxr-xr-x 1 ubuntu ubuntu 699 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/execs/ceph-pool-create.sh 2024-03-27T21:42:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: 404460 4 -rwxr-xr-x 1 ubuntu ubuntu 604 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/qa_scripts/openstack/packstack.sh 2024-03-27T21:42:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: 404356 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/libceph 2024-03-27T21:42:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: 404358 4 -rw-r--r-- 1 ubuntu ubuntu 1709 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/libceph/trivial_libceph.c 2024-03-27T21:42:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: 404357 4 -rw-r--r-- 1 ubuntu ubuntu 165 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/libceph/Makefile 2024-03-27T21:42:45.115 INFO:teuthology.orchestra.run.smithi060.stdout: 404461 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd 2024-03-27T21:42:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: 404471 12 -rw-r--r-- 1 ubuntu ubuntu 8495 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/krbd_deep_flatten.t 2024-03-27T21:42:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: 404476 4 -rw-r--r-- 1 ubuntu ubuntu 901 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/krbd_discard_granularity.t 2024-03-27T21:42:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: 404477 4 -rw-r--r-- 1 ubuntu ubuntu 878 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/krbd_get_features.t 2024-03-27T21:42:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: 404483 12 -rw-r--r-- 1 ubuntu ubuntu 10290 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/krbd_zeroout.t 2024-03-27T21:42:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: 404463 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/conf 2024-03-27T21:42:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: 404464 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/conf/+ 2024-03-27T21:42:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: 404465 4 -rw-r--r-- 1 ubuntu ubuntu 82 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/conf/disable-pool-app.yaml 2024-03-27T21:42:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: 404462 4 -rw-r--r-- 1 ubuntu ubuntu 2161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/common.sh 2024-03-27T21:42:45.116 INFO:teuthology.orchestra.run.smithi060.stdout: 404478 4 -rw-r--r-- 1 ubuntu ubuntu 1328 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/krbd_huge_image.t 2024-03-27T21:42:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: 404481 4 -rw-r--r-- 1 ubuntu ubuntu 1968 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/krbd_parent_overlap.t 2024-03-27T21:42:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: 404475 12 -rw-r--r-- 1 ubuntu ubuntu 10162 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/krbd_discard_512b.t 2024-03-27T21:42:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: 404482 4 -rw-r--r-- 1 ubuntu ubuntu 3645 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/krbd_whole_object_zeroout.t 2024-03-27T21:42:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: 404480 4 -rw-r--r-- 1 ubuntu ubuntu 2630 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/krbd_msgr_segments.t 2024-03-27T21:42:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: 404473 12 -rw-r--r-- 1 ubuntu ubuntu 11011 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/krbd_discard.t 2024-03-27T21:42:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: 404466 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/data-pool 2024-03-27T21:42:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: 404469 4 -rw-r--r-- 1 ubuntu ubuntu 189 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/data-pool/replicated.yaml 2024-03-27T21:42:45.117 INFO:teuthology.orchestra.run.smithi060.stdout: 404468 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/data-pool/none.yaml 2024-03-27T21:42:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: 404467 4 -rw-r--r-- 1 ubuntu ubuntu 873 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/data-pool/ec.yaml 2024-03-27T21:42:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: 404479 4 -rw-r--r-- 1 ubuntu ubuntu 269 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/krbd_modprobe.t 2024-03-27T21:42:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: 404484 4 -rwxr-xr-x 1 ubuntu ubuntu 676 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/rbd.sh 2024-03-27T21:42:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: 404474 12 -rw-r--r-- 1 ubuntu ubuntu 9307 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/krbd_discard_4M.t 2024-03-27T21:42:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: 404470 12 -rw-r--r-- 1 ubuntu ubuntu 9994 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/krbd_blkroset.t 2024-03-27T21:42:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: 404472 4 -rw-r--r-- 1 ubuntu ubuntu 2166 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/rbd/krbd_default_map_options.t 2024-03-27T21:42:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: 404364 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mds 2024-03-27T21:42:45.118 INFO:teuthology.orchestra.run.smithi060.stdout: 404366 4 -rwxr-xr-x 1 ubuntu ubuntu 370 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mds/test_mdstable_failures.sh 2024-03-27T21:42:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: 404365 4 -rwxr-xr-x 1 ubuntu ubuntu 506 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mds/test_anchortable.sh 2024-03-27T21:42:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: 404232 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/client 2024-03-27T21:42:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: 404234 4 -rw-r--r-- 1 ubuntu ubuntu 873 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/client/common.sh 2024-03-27T21:42:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: 404235 48 -rw-r--r-- 1 ubuntu ubuntu 45147 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/client/gen-1774.sh 2024-03-27T21:42:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: 404233 4 -rwxr-xr-x 1 ubuntu ubuntu 381 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/client/30_subdir_mount.sh 2024-03-27T21:42:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: 408351 4 drwxr-xr-x 7 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks 2024-03-27T21:42:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: 408357 20 -rw-r--r-- 1 ubuntu ubuntu 19447 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/barbican.py 2024-03-27T21:42:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: 408457 4 -rw-r--r-- 1 ubuntu ubuntu 2473 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/immutable_object_cache_thrash.py 2024-03-27T21:42:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: 408466 12 -rw-r--r-- 1 ubuntu ubuntu 8809 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/lua_tests.py 2024-03-27T21:42:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: 408562 8 -rw-r--r-- 1 ubuntu ubuntu 4238 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/rgw_multisite_tests.py 2024-03-27T21:42:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: 408537 28 -rw-r--r-- 1 ubuntu ubuntu 26167 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/qemu.py 2024-03-27T21:42:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: 408585 4 -rw-r--r-- 1 ubuntu ubuntu 1662 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/tox.py 2024-03-27T21:42:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: 408373 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephadm_cases 2024-03-27T21:42:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: 408376 4 -rw-r--r-- 1 ubuntu ubuntu 2509 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephadm_cases/test_cli_mon.py 2024-03-27T21:42:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: 408375 4 -rw-r--r-- 1 ubuntu ubuntu 2465 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephadm_cases/test_cli.py 2024-03-27T21:42:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: 408374 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephadm_cases/__init__.py 2024-03-27T21:42:45.120 INFO:teuthology.orchestra.run.smithi060.stdout: 408447 8 -rw-r--r-- 1 ubuntu ubuntu 4925 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/divergent_priors.py 2024-03-27T21:42:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: 408449 8 -rw-r--r-- 1 ubuntu ubuntu 5995 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/dnsmasq.py 2024-03-27T21:42:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: 408459 8 -rw-r--r-- 1 ubuntu ubuntu 4672 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/kclient.py 2024-03-27T21:42:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: 408553 8 -rw-r--r-- 1 ubuntu ubuntu 5837 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/rep_lost_unfound_delete.py 2024-03-27T21:42:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: 408472 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr 2024-03-27T21:42:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: 408510 8 -rw-r--r-- 1 ubuntu ubuntu 6107 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/test_failover.py 2024-03-27T21:42:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: 408511 8 -rw-r--r-- 1 ubuntu ubuntu 6853 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/test_insights.py 2024-03-27T21:42:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: 408505 4 -rw-r--r-- 1 ubuntu ubuntu 2474 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/test_cache.py 2024-03-27T21:42:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: 408515 4 -rw-r--r-- 1 ubuntu ubuntu 2507 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/test_prometheus.py 2024-03-27T21:42:45.121 INFO:teuthology.orchestra.run.smithi060.stdout: 408514 16 -rw-r--r-- 1 ubuntu ubuntu 14713 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/test_progress.py 2024-03-27T21:42:45.122 INFO:teuthology.orchestra.run.smithi060.stdout: 408474 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard 2024-03-27T21:42:45.122 INFO:teuthology.orchestra.run.smithi060.stdout: 408500 4 -rw-r--r-- 1 ubuntu ubuntu 2000 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_settings.py 2024-03-27T21:42:45.122 INFO:teuthology.orchestra.run.smithi060.stdout: 408494 20 -rw-r--r-- 1 ubuntu ubuntu 17214 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_pool.py 2024-03-27T21:42:45.122 INFO:teuthology.orchestra.run.smithi060.stdout: 408484 4 -rw-r--r-- 1 ubuntu ubuntu 1306 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_feedback.py 2024-03-27T21:42:45.122 INFO:teuthology.orchestra.run.smithi060.stdout: 408497 4 -rw-r--r-- 1 ubuntu ubuntu 1154 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_requests.py 2024-03-27T21:42:45.122 INFO:teuthology.orchestra.run.smithi060.stdout: 408492 16 -rw-r--r-- 1 ubuntu ubuntu 13727 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_osd.py 2024-03-27T21:42:45.122 INFO:teuthology.orchestra.run.smithi060.stdout: 408481 16 -rw-r--r-- 1 ubuntu ubuntu 15716 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_cluster_configuration.py 2024-03-27T21:42:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: 408498 32 -rw-r--r-- 1 ubuntu ubuntu 31731 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_rgw.py 2024-03-27T21:42:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: 408479 12 -rw-r--r-- 1 ubuntu ubuntu 12245 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_cephfs.py 2024-03-27T21:42:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: 408488 8 -rw-r--r-- 1 ubuntu ubuntu 5697 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_mgr_module.py 2024-03-27T21:42:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: 408487 4 -rw-r--r-- 1 ubuntu ubuntu 922 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_logs.py 2024-03-27T21:42:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: 408503 24 -rw-r--r-- 1 ubuntu ubuntu 22020 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_user.py 2024-03-27T21:42:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: 408499 8 -rw-r--r-- 1 ubuntu ubuntu 5072 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_role.py 2024-03-27T21:42:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: 408490 4 -rw-r--r-- 1 ubuntu ubuntu 1041 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_motd.py 2024-03-27T21:42:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: 408478 16 -rw-r--r-- 1 ubuntu ubuntu 14445 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_auth.py 2024-03-27T21:42:45.123 INFO:teuthology.orchestra.run.smithi060.stdout: 408501 4 -rw-r--r-- 1 ubuntu ubuntu 1428 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_summary.py 2024-03-27T21:42:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: 408485 12 -rw-r--r-- 1 ubuntu ubuntu 11280 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_health.py 2024-03-27T21:42:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: 408475 4 -rw-r--r-- 1 ubuntu ubuntu 28 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/__init__.py 2024-03-27T21:42:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: 408476 28 -rw-r--r-- 1 ubuntu ubuntu 25696 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/helper.py 2024-03-27T21:42:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: 408486 8 -rw-r--r-- 1 ubuntu ubuntu 6190 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_host.py 2024-03-27T21:42:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: 408491 4 -rw-r--r-- 1 ubuntu ubuntu 702 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_orchestrator.py 2024-03-27T21:42:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: 408502 4 -rw-r--r-- 1 ubuntu ubuntu 2880 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_telemetry.py 2024-03-27T21:42:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: 408482 4 -rw-r--r-- 1 ubuntu ubuntu 2801 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_crush_rule.py 2024-03-27T21:42:45.124 INFO:teuthology.orchestra.run.smithi060.stdout: 408477 4 -rw-r--r-- 1 ubuntu ubuntu 561 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_api.py 2024-03-27T21:42:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: 408489 4 -rw-r--r-- 1 ubuntu ubuntu 729 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_monitor.py 2024-03-27T21:42:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: 408483 4 -rw-r--r-- 1 ubuntu ubuntu 3390 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_erasure_code_profile.py 2024-03-27T21:42:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: 408496 8 -rw-r--r-- 1 ubuntu ubuntu 6472 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_rbd_mirroring.py 2024-03-27T21:42:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: 408495 40 -rw-r--r-- 1 ubuntu ubuntu 40160 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_rbd.py 2024-03-27T21:42:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: 408493 4 -rw-r--r-- 1 ubuntu ubuntu 2785 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_perf_counters.py 2024-03-27T21:42:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: 408480 4 -rw-r--r-- 1 ubuntu ubuntu 686 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/dashboard/test_cluster.py 2024-03-27T21:42:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: 408513 12 -rw-r--r-- 1 ubuntu ubuntu 8477 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/test_orchestrator_cli.py 2024-03-27T21:42:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: 408509 4 -rw-r--r-- 1 ubuntu ubuntu 1266 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/test_devicehealth.py 2024-03-27T21:42:45.125 INFO:teuthology.orchestra.run.smithi060.stdout: 408508 8 -rw-r--r-- 1 ubuntu ubuntu 7149 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/test_dashboard.py 2024-03-27T21:42:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: 408506 4 -rw-r--r-- 1 ubuntu ubuntu 1073 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/test_cli.py 2024-03-27T21:42:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: 408473 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/__init__.py 2024-03-27T21:42:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: 408504 12 -rw-r--r-- 1 ubuntu ubuntu 8728 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/mgr_test_case.py 2024-03-27T21:42:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: 408512 12 -rw-r--r-- 1 ubuntu ubuntu 9787 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/test_module_selftest.py 2024-03-27T21:42:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: 408507 4 -rw-r--r-- 1 ubuntu ubuntu 3380 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mgr/test_crash.py 2024-03-27T21:42:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: 408454 8 -rw-r--r-- 1 ubuntu ubuntu 6260 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/fs.py 2024-03-27T21:42:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: 408551 12 -rw-r--r-- 1 ubuntu ubuntu 8550 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/rebuild_mondb.py 2024-03-27T21:42:45.126 INFO:teuthology.orchestra.run.smithi060.stdout: 408598 8 -rw-r--r-- 1 ubuntu ubuntu 7359 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/vip.py 2024-03-27T21:42:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: 408572 12 -rw-r--r-- 1 ubuntu ubuntu 8991 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/tempest.py 2024-03-27T21:42:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: 408579 4 -rw-r--r-- 1 ubuntu ubuntu 355 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/teuthology_integration.py 2024-03-27T21:42:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: 408558 8 -rw-r--r-- 1 ubuntu ubuntu 4733 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/rgw_logsocket.py 2024-03-27T21:42:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: 408573 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/tests 2024-03-27T21:42:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: 408575 4 -rw-r--r-- 1 ubuntu ubuntu 371 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/tests/conftest.py 2024-03-27T21:42:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: 408576 4 -rw-r--r-- 1 ubuntu ubuntu 2096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/tests/test_devstack.py 2024-03-27T21:42:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: 408578 4 -rw-r--r-- 1 ubuntu ubuntu 1663 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/tests/test_radosgw_admin.py 2024-03-27T21:42:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: 408574 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/tests/__init__.py 2024-03-27T21:42:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: 408577 4 -rw-r--r-- 1 ubuntu ubuntu 81 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/tests/test_import_yaml.py 2024-03-27T21:42:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: 408545 28 -rw-r--r-- 1 ubuntu ubuntu 25715 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/rbd.py 2024-03-27T21:42:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: 408584 12 -rw-r--r-- 1 ubuntu ubuntu 8763 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/thrashosds.py 2024-03-27T21:42:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: 408546 12 -rw-r--r-- 1 ubuntu ubuntu 9543 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/rbd_fio.py 2024-03-27T21:42:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: 408356 8 -rw-r--r-- 1 ubuntu ubuntu 6634 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/backfill_toofull.py 2024-03-27T21:42:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: 408464 4 -rw-r--r-- 1 ubuntu ubuntu 223 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/logrotate.conf 2024-03-27T21:42:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: 408377 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs 2024-03-27T21:42:45.128 INFO:teuthology.orchestra.run.smithi060.stdout: 408388 84 -rw-r--r-- 1 ubuntu ubuntu 84154 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_admin.py 2024-03-27T21:42:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: 408399 36 -rw-r--r-- 1 ubuntu ubuntu 34747 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_failover.py 2024-03-27T21:42:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: 408415 4 -rw-r--r-- 1 ubuntu ubuntu 651 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_newops.py 2024-03-27T21:42:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: 408387 4 -rw-r--r-- 1 ubuntu ubuntu 1502 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_acls.py 2024-03-27T21:42:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: 408422 4 -rw-r--r-- 1 ubuntu ubuntu 1483 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_recovery_fs.py 2024-03-27T21:42:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: 408407 20 -rw-r--r-- 1 ubuntu ubuntu 16591 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_journal_repair.py 2024-03-27T21:42:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: 408405 20 -rw-r--r-- 1 ubuntu ubuntu 16812 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_full.py 2024-03-27T21:42:45.129 INFO:teuthology.orchestra.run.smithi060.stdout: 408392 48 -rw-r--r-- 1 ubuntu ubuntu 45543 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_cephfs_shell.py 2024-03-27T21:42:45.130 INFO:teuthology.orchestra.run.smithi060.stdout: 408404 4 -rw-r--r-- 1 ubuntu ubuntu 3807 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_fstop.py 2024-03-27T21:42:45.130 INFO:teuthology.orchestra.run.smithi060.stdout: 408414 12 -rw-r--r-- 1 ubuntu ubuntu 8588 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_multimds_misc.py 2024-03-27T21:42:45.130 INFO:teuthology.orchestra.run.smithi060.stdout: 408401 20 -rw-r--r-- 1 ubuntu ubuntu 20469 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_forward_scrub.py 2024-03-27T21:42:45.130 INFO:teuthology.orchestra.run.smithi060.stdout: 408412 28 -rw-r--r-- 1 ubuntu ubuntu 26790 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_misc.py 2024-03-27T21:42:45.130 INFO:teuthology.orchestra.run.smithi060.stdout: 408397 4 -rw-r--r-- 1 ubuntu ubuntu 2364 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_dump_tree.py 2024-03-27T21:42:45.130 INFO:teuthology.orchestra.run.smithi060.stdout: 408423 8 -rw-r--r-- 1 ubuntu ubuntu 7029 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_recovery_pool.py 2024-03-27T21:42:45.130 INFO:teuthology.orchestra.run.smithi060.stdout: 408379 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/admin 2024-03-27T21:42:45.130 INFO:teuthology.orchestra.run.smithi060.stdout: 408380 36 -rw-r--r-- 1 ubuntu ubuntu 32771 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/admin/test_fs_swap.py 2024-03-27T21:42:45.131 INFO:teuthology.orchestra.run.smithi060.stdout: 408418 4 -rw-r--r-- 1 ubuntu ubuntu 4005 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_pool_perm.py 2024-03-27T21:42:45.131 INFO:teuthology.orchestra.run.smithi060.stdout: 408394 32 -rw-r--r-- 1 ubuntu ubuntu 32150 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_client_recovery.py 2024-03-27T21:42:45.131 INFO:teuthology.orchestra.run.smithi060.stdout: 408426 12 -rw-r--r-- 1 ubuntu ubuntu 9335 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_sessionmap.py 2024-03-27T21:42:45.131 INFO:teuthology.orchestra.run.smithi060.stdout: 408430 8 -rw-r--r-- 1 ubuntu ubuntu 6671 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_subvolume.py 2024-03-27T21:42:45.131 INFO:teuthology.orchestra.run.smithi060.stdout: 408383 68 -rw-r--r-- 1 ubuntu ubuntu 67381 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/filesystem.py 2024-03-27T21:42:45.131 INFO:teuthology.orchestra.run.smithi060.stdout: 408420 8 -rw-r--r-- 1 ubuntu ubuntu 6308 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_quota.py 2024-03-27T21:42:45.131 INFO:teuthology.orchestra.run.smithi060.stdout: 408386 60 -rw-r--r-- 1 ubuntu ubuntu 59140 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/mount.py 2024-03-27T21:42:45.131 INFO:teuthology.orchestra.run.smithi060.stdout: 408424 8 -rw-r--r-- 1 ubuntu ubuntu 6745 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_scrub.py 2024-03-27T21:42:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: 408384 20 -rw-r--r-- 1 ubuntu ubuntu 19666 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/fuse_mount.py 2024-03-27T21:42:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: 408391 4 -rw-r--r-- 1 ubuntu ubuntu 2130 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_cap_flush.py 2024-03-27T21:42:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: 408403 4 -rw-r--r-- 1 ubuntu ubuntu 2796 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_fscrypt.py 2024-03-27T21:42:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: 408409 28 -rw-r--r-- 1 ubuntu ubuntu 24654 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_mds_metrics.py 2024-03-27T21:42:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: 408400 8 -rw-r--r-- 1 ubuntu ubuntu 5001 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_flush.py 2024-03-27T21:42:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: 408416 52 -rw-r--r-- 1 ubuntu ubuntu 52090 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_nfs.py 2024-03-27T21:42:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: 408410 4 -rw-r--r-- 1 ubuntu ubuntu 2013 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_meta_injection.py 2024-03-27T21:42:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: 408389 4 -rw-r--r-- 1 ubuntu ubuntu 2592 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_auto_repair.py 2024-03-27T21:42:45.132 INFO:teuthology.orchestra.run.smithi060.stdout: 408378 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/__init__.py 2024-03-27T21:42:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: 408381 16 -rw-r--r-- 1 ubuntu ubuntu 12712 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/caps_helper.py 2024-03-27T21:42:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: 408433 20 -rw-r--r-- 1 ubuntu ubuntu 19209 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/xfstests_dev.py 2024-03-27T21:42:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: 408413 12 -rw-r--r-- 1 ubuntu ubuntu 11333 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_multifs_auth.py 2024-03-27T21:42:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: 408429 40 -rw-r--r-- 1 ubuntu ubuntu 39778 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_strays.py 2024-03-27T21:42:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: 408427 56 -rw-r--r-- 1 ubuntu ubuntu 55628 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_snap_schedules.py 2024-03-27T21:42:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: 408432 4 -rw-r--r-- 1 ubuntu ubuntu 209 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/tests_from_xfstests_dev.py 2024-03-27T21:42:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: 408419 28 -rw-r--r-- 1 ubuntu ubuntu 27921 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_quiesce.py 2024-03-27T21:42:45.133 INFO:teuthology.orchestra.run.smithi060.stdout: 408393 20 -rw-r--r-- 1 ubuntu ubuntu 17048 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_client_limits.py 2024-03-27T21:42:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: 408417 4 -rw-r--r-- 1 ubuntu ubuntu 3151 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_openfiletable.py 2024-03-27T21:42:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: 408431 344 -rw-r--r-- 1 ubuntu ubuntu 350140 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_volumes.py 2024-03-27T21:42:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: 408398 28 -rw-r--r-- 1 ubuntu ubuntu 24907 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_exports.py 2024-03-27T21:42:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: 408382 20 -rw-r--r-- 1 ubuntu ubuntu 17940 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/cephfs_test_case.py 2024-03-27T21:42:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: 408395 28 -rw-r--r-- 1 ubuntu ubuntu 27112 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_damage.py 2024-03-27T21:42:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: 408428 28 -rw-r--r-- 1 ubuntu ubuntu 26559 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_snapshots.py 2024-03-27T21:42:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: 408411 72 -rw-r--r-- 1 ubuntu ubuntu 73722 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_mirroring.py 2024-03-27T21:42:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: 408402 16 -rw-r--r-- 1 ubuntu ubuntu 12308 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_fragment.py 2024-03-27T21:42:45.134 INFO:teuthology.orchestra.run.smithi060.stdout: 408421 4 -rw-r--r-- 1 ubuntu ubuntu 1031 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_readahead.py 2024-03-27T21:42:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: 408390 8 -rw-r--r-- 1 ubuntu ubuntu 5038 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_backtrace.py 2024-03-27T21:42:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: 408425 20 -rw-r--r-- 1 ubuntu ubuntu 17560 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_scrub_checks.py 2024-03-27T21:42:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: 408408 8 -rw-r--r-- 1 ubuntu ubuntu 4451 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_mantle.py 2024-03-27T21:42:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: 408406 4 -rw-r--r-- 1 ubuntu ubuntu 3928 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_journal_migration.py 2024-03-27T21:42:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: 408385 16 -rw-r--r-- 1 ubuntu ubuntu 13882 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/kernel_mount.py 2024-03-27T21:42:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: 408396 36 -rw-r--r-- 1 ubuntu ubuntu 33182 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs/test_data_scan.py 2024-03-27T21:42:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: 408517 4 -rw-r--r-- 1 ubuntu ubuntu 2281 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mon_recovery.py 2024-03-27T21:42:45.135 INFO:teuthology.orchestra.run.smithi060.stdout: 408435 8 -rw-r--r-- 1 ubuntu ubuntu 7264 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs_mirror_thrash.py 2024-03-27T21:42:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: 408461 20 -rw-r--r-- 1 ubuntu ubuntu 17982 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/keystone.py 2024-03-27T21:42:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: 408451 8 -rw-r--r-- 1 ubuntu ubuntu 7848 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/ec_inconsistent_hinfo.py 2024-03-27T21:42:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: 408557 8 -rw-r--r-- 1 ubuntu ubuntu 5081 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/rgw_cloudtier.py 2024-03-27T21:42:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: 408445 16 -rw-r--r-- 1 ubuntu ubuntu 13327 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/devstack.py 2024-03-27T21:42:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: 408587 4 -rw-r--r-- 1 ubuntu ubuntu 164 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/userdata_teardown.yaml 2024-03-27T21:42:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: 408570 16 -rw-r--r-- 1 ubuntu ubuntu 14841 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/scrub_test.py 2024-03-27T21:42:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: 408467 4 -rw-r--r-- 1 ubuntu ubuntu 2680 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/manypools.py 2024-03-27T21:42:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: 408534 4 -rw-r--r-- 1 ubuntu ubuntu 2753 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/populate_rbd_pool.py 2024-03-27T21:42:45.136 INFO:teuthology.orchestra.run.smithi060.stdout: 408354 8 -rw-r--r-- 1 ubuntu ubuntu 4983 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/autotest.py 2024-03-27T21:42:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: 408463 8 -rwxr-xr-x 1 ubuntu ubuntu 4936 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/locktest.py 2024-03-27T21:42:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: 408564 24 -rw-r--r-- 1 ubuntu ubuntu 21645 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/rook.py 2024-03-27T21:42:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: 408567 16 -rw-r--r-- 1 ubuntu ubuntu 15922 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/s3tests_java.py 2024-03-27T21:42:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: 408525 4 -rw-r--r-- 1 ubuntu ubuntu 2900 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/object_source_down.py 2024-03-27T21:42:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: 408581 4 -rw-r--r-- 1 ubuntu ubuntu 1617 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/thrash_pool_snaps.py 2024-03-27T21:42:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: 408520 12 -rw-r--r-- 1 ubuntu ubuntu 8338 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/netem.py 2024-03-27T21:42:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: 408523 4 -rw-r--r-- 1 ubuntu ubuntu 3908 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/nvme_loop.py 2024-03-27T21:42:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: 408601 4 -rw-r--r-- 1 ubuntu ubuntu 1634 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/watch_notify_stress.py 2024-03-27T21:42:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: 408361 8 -rw-r--r-- 1 ubuntu ubuntu 6025 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cbt_performance.py 2024-03-27T21:42:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: 408448 8 -rw-r--r-- 1 ubuntu ubuntu 6262 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/divergent_priors2.py 2024-03-27T21:42:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: 408535 16 -rw-r--r-- 1 ubuntu ubuntu 14990 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/pykmip.py 2024-03-27T21:42:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: 408530 8 -rw-r--r-- 1 ubuntu ubuntu 4571 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/osd_max_pg_per_osd.py 2024-03-27T21:42:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: 408458 8 -rw-r--r-- 1 ubuntu ubuntu 6487 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/kafka.py 2024-03-27T21:42:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: 408602 16 -rw-r--r-- 1 ubuntu ubuntu 15925 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/workunit.py 2024-03-27T21:42:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: 408469 4 -rw-r--r-- 1 ubuntu ubuntu 563 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mds_pre_upgrade.py 2024-03-27T21:42:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: 408365 36 -rw-r--r-- 1 ubuntu ubuntu 34996 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/ceph_deploy.py 2024-03-27T21:42:45.138 INFO:teuthology.orchestra.run.smithi060.stdout: 408583 4 -rw-r--r-- 1 ubuntu ubuntu 580 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/thrashosds-health.yaml 2024-03-27T21:42:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: 408462 20 -rw-r--r-- 1 ubuntu ubuntu 19665 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/kubeadm.py 2024-03-27T21:42:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: 408561 20 -rw-r--r-- 1 ubuntu ubuntu 16450 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/rgw_multisite.py 2024-03-27T21:42:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: 408367 4 -rw-r--r-- 1 ubuntu ubuntu 1796 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/ceph_iscsi_client.py 2024-03-27T21:42:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: 408446 4 -rw-r--r-- 1 ubuntu ubuntu 1944 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/die_on_err.py 2024-03-27T21:42:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: 408355 4 -rw-r--r-- 1 ubuntu ubuntu 2064 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/aver.py 2024-03-27T21:42:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: 408580 8 -rw-r--r-- 1 ubuntu ubuntu 5001 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/tgt.py 2024-03-27T21:42:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: 408468 4 -rw-r--r-- 1 ubuntu ubuntu 2606 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mds_creation_failure.py 2024-03-27T21:42:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: 408529 8 -rw-r--r-- 1 ubuntu ubuntu 7082 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/osd_failsafe_enospc.py 2024-03-27T21:42:45.139 INFO:teuthology.orchestra.run.smithi060.stdout: 408526 4 -rw-r--r-- 1 ubuntu ubuntu 2603 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/omapbench.py 2024-03-27T21:42:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: 408434 4 -rw-r--r-- 1 ubuntu ubuntu 2149 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs_mirror.py 2024-03-27T21:42:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: 408460 16 -rw-r--r-- 1 ubuntu ubuntu 16012 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/keycloak.py 2024-03-27T21:42:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: 408547 4 -rw-r--r-- 1 ubuntu ubuntu 3591 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/rbd_fsx.py 2024-03-27T21:42:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: 408360 16 -rw-r--r-- 1 ubuntu ubuntu 12957 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cbt.py 2024-03-27T21:42:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: 408543 40 -rw-r--r-- 1 ubuntu ubuntu 37554 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/radosgw_admin_rest.py 2024-03-27T21:42:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: 408559 4 -rw-r--r-- 1 ubuntu ubuntu 1375 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/rgw_module.py 2024-03-27T21:42:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: 408440 8 -rw-r--r-- 1 ubuntu ubuntu 5402 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cram.py 2024-03-27T21:42:45.140 INFO:teuthology.orchestra.run.smithi060.stdout: 408560 0 lrwxrwxrwx 1 ubuntu ubuntu 28 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/rgw_multi -> ../../src/test/rgw/rgw_multi 2024-03-27T21:42:45.141 INFO:teuthology.orchestra.run.smithi060.stdout: 408600 4 -rw-r--r-- 1 ubuntu ubuntu 3242 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/watch_notify_same_primary.py 2024-03-27T21:42:45.141 INFO:teuthology.orchestra.run.smithi060.stdout: 408516 4 -rw-r--r-- 1 ubuntu ubuntu 2165 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mon_clock_skew_check.py 2024-03-27T21:42:45.141 INFO:teuthology.orchestra.run.smithi060.stdout: 408569 4 -rw-r--r-- 1 ubuntu ubuntu 2662 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/scrub.py 2024-03-27T21:42:45.141 INFO:teuthology.orchestra.run.smithi060.stdout: 408371 4 -rw-r--r-- 1 ubuntu ubuntu 2438 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephadm.conf 2024-03-27T21:42:45.141 INFO:teuthology.orchestra.run.smithi060.stdout: 408555 4 -rw-r--r-- 1 ubuntu ubuntu 2881 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/resolve_stuck_peering.py 2024-03-27T21:42:45.141 INFO:teuthology.orchestra.run.smithi060.stdout: 408536 4 -rw-r--r-- 1 ubuntu ubuntu 1122 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/python.py 2024-03-27T21:42:45.141 INFO:teuthology.orchestra.run.smithi060.stdout: 408582 4 -rw-r--r-- 1 ubuntu ubuntu 290 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/thrasher.py 2024-03-27T21:42:45.141 INFO:teuthology.orchestra.run.smithi060.stdout: 408437 4 -rw-r--r-- 1 ubuntu ubuntu 1422 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs_upgrade_snap.py 2024-03-27T21:42:45.141 INFO:teuthology.orchestra.run.smithi060.stdout: 408519 4 -rw-r--r-- 1 ubuntu ubuntu 1511 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/multibench.py 2024-03-27T21:42:45.142 INFO:teuthology.orchestra.run.smithi060.stdout: 408368 128 -rw-r--r-- 1 ubuntu ubuntu 127109 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/ceph_manager.py 2024-03-27T21:42:45.142 INFO:teuthology.orchestra.run.smithi060.stdout: 408586 4 -rw-r--r-- 1 ubuntu ubuntu 1161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/userdata_setup.yaml 2024-03-27T21:42:45.142 INFO:teuthology.orchestra.run.smithi060.stdout: 408568 8 -rw-r--r-- 1 ubuntu ubuntu 7830 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/samba.py 2024-03-27T21:42:45.142 INFO:teuthology.orchestra.run.smithi060.stdout: 408471 4 -rw-r--r-- 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/metadata.yaml 2024-03-27T21:42:45.142 INFO:teuthology.orchestra.run.smithi060.stdout: 408518 16 -rw-r--r-- 1 ubuntu ubuntu 15102 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mon_thrash.py 2024-03-27T21:42:45.142 INFO:teuthology.orchestra.run.smithi060.stdout: 408528 4 -rw-r--r-- 1 ubuntu ubuntu 2599 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/osd_backfill.py 2024-03-27T21:42:45.142 INFO:teuthology.orchestra.run.smithi060.stdout: 408588 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/util 2024-03-27T21:42:45.142 INFO:teuthology.orchestra.run.smithi060.stdout: 408593 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/util/test 2024-03-27T21:42:45.142 INFO:teuthology.orchestra.run.smithi060.stdout: 408595 4 -rw-r--r-- 1 ubuntu ubuntu 1568 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/util/test/test_rados.py 2024-03-27T21:42:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: 408594 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/util/test/__init__.py 2024-03-27T21:42:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: 408596 4 -rw-r--r-- 1 ubuntu ubuntu 2378 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/util/workunit.py 2024-03-27T21:42:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: 408592 4 -rw-r--r-- 1 ubuntu ubuntu 2967 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/util/rgw.py 2024-03-27T21:42:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: 408589 4 -rw-r--r-- 1 ubuntu ubuntu 1049 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/util/__init__.py 2024-03-27T21:42:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: 408591 8 -rw-r--r-- 1 ubuntu ubuntu 4305 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/util/rados.py 2024-03-27T21:42:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: 408590 8 -rw-r--r-- 1 ubuntu ubuntu 4893 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/util/chacra.py 2024-03-27T21:42:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: 408556 24 -rw-r--r-- 1 ubuntu ubuntu 22411 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/rgw.py 2024-03-27T21:42:45.143 INFO:teuthology.orchestra.run.smithi060.stdout: 408353 8 -rw-r--r-- 1 ubuntu ubuntu 6476 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/admin_socket.py 2024-03-27T21:42:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: 408563 4 -rw-r--r-- 1 ubuntu ubuntu 935 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/rook-ceph.conf 2024-03-27T21:42:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: 408363 68 -rw-r--r-- 1 ubuntu ubuntu 67412 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/ceph.py 2024-03-27T21:42:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: 408524 8 -rw-r--r-- 1 ubuntu ubuntu 6172 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/nvmeof.py 2024-03-27T21:42:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: 408352 4 -rw-r--r-- 1 ubuntu ubuntu 167 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/__init__.py 2024-03-27T21:42:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: 408599 56 -rw-r--r-- 1 ubuntu ubuntu 55127 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/vstart_runner.py 2024-03-27T21:42:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: 408442 8 -rw-r--r-- 1 ubuntu ubuntu 4963 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/d4ntests.py 2024-03-27T21:42:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: 408538 4 -rw-r--r-- 1 ubuntu ubuntu 3662 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/rabbitmq.py 2024-03-27T21:42:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: 408436 8 -rw-r--r-- 1 ubuntu ubuntu 6319 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephfs_test_runner.py 2024-03-27T21:42:45.144 INFO:teuthology.orchestra.run.smithi060.stdout: 408359 4 -rw-r--r-- 1 ubuntu ubuntu 44 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/boto.cfg.template 2024-03-27T21:42:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: 408552 8 -rw-r--r-- 1 ubuntu ubuntu 7953 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/reg11184.py 2024-03-27T21:42:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: 408456 4 -rw-r--r-- 1 ubuntu ubuntu 2333 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/immutable_object_cache.py 2024-03-27T21:42:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: 408550 4 -rw-r--r-- 1 ubuntu ubuntu 3188 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/rbd_pwl_cache_recovery.py 2024-03-27T21:42:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: 408544 16 -rw-r--r-- 1 ubuntu ubuntu 12875 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/ragweed.py 2024-03-27T21:42:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: 408533 4 -rw-r--r-- 1 ubuntu ubuntu 2303 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/peering_speed_test.py 2024-03-27T21:42:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: 408540 8 -rw-r--r-- 1 ubuntu ubuntu 4728 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/radosbench.py 2024-03-27T21:42:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: 408358 4 -rw-r--r-- 1 ubuntu ubuntu 2821 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/blktrace.py 2024-03-27T21:42:45.145 INFO:teuthology.orchestra.run.smithi060.stdout: 408453 4 -rw-r--r-- 1 ubuntu ubuntu 1899 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/exec_on_cleanup.py 2024-03-27T21:42:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: 408439 4 -rw-r--r-- 1 ubuntu ubuntu 3758 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cifs_mount.py 2024-03-27T21:42:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: 408372 76 -rw-r--r-- 1 ubuntu ubuntu 75219 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/cephadm.py 2024-03-27T21:42:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: 408522 12 -rw-r--r-- 1 ubuntu ubuntu 11584 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/notification_tests.py 2024-03-27T21:42:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: 408364 4 -rw-r--r-- 1 ubuntu ubuntu 1501 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/ceph_client.py 2024-03-27T21:42:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: 408539 12 -rw-r--r-- 1 ubuntu ubuntu 10733 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/rados.py 2024-03-27T21:42:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: 408470 20 -rw-r--r-- 1 ubuntu ubuntu 17999 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/mds_thrash.py 2024-03-27T21:42:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: 408597 12 -rw-r--r-- 1 ubuntu ubuntu 9344 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/vault.py 2024-03-27T21:42:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: 408369 28 -rw-r--r-- 1 ubuntu ubuntu 26396 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/ceph_objectstore_tool.py 2024-03-27T21:42:45.146 INFO:teuthology.orchestra.run.smithi060.stdout: 408554 12 -rw-r--r-- 1 ubuntu ubuntu 9443 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/repair_test.py 2024-03-27T21:42:45.147 INFO:teuthology.orchestra.run.smithi060.stdout: 408370 16 -rw-r--r-- 1 ubuntu ubuntu 14055 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/ceph_test_case.py 2024-03-27T21:42:45.147 INFO:teuthology.orchestra.run.smithi060.stdout: 408571 8 -rw-r--r-- 1 ubuntu ubuntu 5784 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/systemd.py 2024-03-27T21:42:45.147 INFO:teuthology.orchestra.run.smithi060.stdout: 408548 4 -rw-r--r-- 1 ubuntu ubuntu 3719 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/rbd_mirror.py 2024-03-27T21:42:45.147 INFO:teuthology.orchestra.run.smithi060.stdout: 408527 12 -rw-r--r-- 1 ubuntu ubuntu 9344 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/openssl_keys.py 2024-03-27T21:42:45.147 INFO:teuthology.orchestra.run.smithi060.stdout: 408366 8 -rw-r--r-- 1 ubuntu ubuntu 5946 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/ceph_fuse.py 2024-03-27T21:42:45.147 INFO:teuthology.orchestra.run.smithi060.stdout: 408565 12 -rw-r--r-- 1 ubuntu ubuntu 8766 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/s3a_hadoop.py 2024-03-27T21:42:45.147 INFO:teuthology.orchestra.run.smithi060.stdout: 408549 8 -rw-r--r-- 1 ubuntu ubuntu 7125 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/rbd_mirror_thrash.py 2024-03-27T21:42:45.147 INFO:teuthology.orchestra.run.smithi060.stdout: 408541 8 -rw-r--r-- 1 ubuntu ubuntu 7667 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/radosbenchsweep.py 2024-03-27T21:42:45.148 INFO:teuthology.orchestra.run.smithi060.stdout: 408465 8 -rw-r--r-- 1 ubuntu ubuntu 5993 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/lost_unfound.py 2024-03-27T21:42:45.148 INFO:teuthology.orchestra.run.smithi060.stdout: 408566 32 -rw-r--r-- 1 ubuntu ubuntu 29084 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/s3tests.py 2024-03-27T21:42:45.148 INFO:teuthology.orchestra.run.smithi060.stdout: 408532 4 -rw-r--r-- 1 ubuntu ubuntu 2457 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/peer.py 2024-03-27T21:42:45.148 INFO:teuthology.orchestra.run.smithi060.stdout: 408362 4 -rw-r--r-- 1 ubuntu ubuntu 2919 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/ceph.conf.template 2024-03-27T21:42:45.148 INFO:teuthology.orchestra.run.smithi060.stdout: 408443 8 -rw-r--r-- 1 ubuntu ubuntu 5606 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/daemonwatchdog.py 2024-03-27T21:42:45.148 INFO:teuthology.orchestra.run.smithi060.stdout: 408455 8 -rw-r--r-- 1 ubuntu ubuntu 5083 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/fwd_scrub.py 2024-03-27T21:42:45.148 INFO:teuthology.orchestra.run.smithi060.stdout: 408531 8 -rw-r--r-- 1 ubuntu ubuntu 5337 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/osd_recovery.py 2024-03-27T21:42:45.148 INFO:teuthology.orchestra.run.smithi060.stdout: 408452 8 -rw-r--r-- 1 ubuntu ubuntu 5320 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/ec_lost_unfound.py 2024-03-27T21:42:45.148 INFO:teuthology.orchestra.run.smithi060.stdout: 408444 12 -rw-r--r-- 1 ubuntu ubuntu 8576 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/deduplication.py 2024-03-27T21:42:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: 408521 4 -rw-r--r-- 1 ubuntu ubuntu 2437 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/netsplit.py 2024-03-27T21:42:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: 408441 4 -rw-r--r-- 1 ubuntu ubuntu 2499 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/create_verify_lfn_objects.py 2024-03-27T21:42:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: 408438 8 -rw-r--r-- 1 ubuntu ubuntu 4748 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/check_counter.py 2024-03-27T21:42:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: 408542 44 -rw-r--r-- 1 ubuntu ubuntu 44474 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/radosgw_admin.py 2024-03-27T21:42:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: 408450 8 -rw-r--r-- 1 ubuntu ubuntu 4415 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/tasks/dump_stuck.py 2024-03-27T21:42:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: 404361 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/machine_types 2024-03-27T21:42:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: 404362 4 -rwxr-xr-x 1 ubuntu ubuntu 1217 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/machine_types/schedule_rados_ovh.sh 2024-03-27T21:42:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: 404363 4 -rw-r--r-- 1 ubuntu ubuntu 285 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/machine_types/vps.yaml 2024-03-27T21:42:45.149 INFO:teuthology.orchestra.run.smithi060.stdout: 404355 4 -rwxr-xr-x 1 ubuntu ubuntu 120 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/find-used-ports.sh 2024-03-27T21:42:45.150 INFO:teuthology.orchestra.run.smithi060.stdout: 404404 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/objectstore_debug 2024-03-27T21:42:45.150 INFO:teuthology.orchestra.run.smithi060.stdout: 404410 4 -rw-r--r-- 1 ubuntu ubuntu 1226 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/objectstore_debug/bluestore-hybrid.yaml 2024-03-27T21:42:45.150 INFO:teuthology.orchestra.run.smithi060.stdout: 404411 4 -rw-r--r-- 1 ubuntu ubuntu 801 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/objectstore_debug/bluestore-low-osd-mem-target.yaml 2024-03-27T21:42:45.150 INFO:teuthology.orchestra.run.smithi060.stdout: 404407 4 -rw-r--r-- 1 ubuntu ubuntu 772 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/objectstore_debug/bluestore-comp-snappy.yaml 2024-03-27T21:42:45.150 INFO:teuthology.orchestra.run.smithi060.stdout: 404408 4 -rw-r--r-- 1 ubuntu ubuntu 770 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/objectstore_debug/bluestore-comp-zlib.yaml 2024-03-27T21:42:45.150 INFO:teuthology.orchestra.run.smithi060.stdout: 404405 4 -rw-r--r-- 1 ubuntu ubuntu 1327 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/objectstore_debug/bluestore-bitmap.yaml 2024-03-27T21:42:45.150 INFO:teuthology.orchestra.run.smithi060.stdout: 404409 4 -rw-r--r-- 1 ubuntu ubuntu 770 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/objectstore_debug/bluestore-comp-zstd.yaml 2024-03-27T21:42:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: 404406 4 -rw-r--r-- 1 ubuntu ubuntu 769 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/objectstore_debug/bluestore-comp-lz4.yaml 2024-03-27T21:42:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: 404412 4 -rw-r--r-- 1 ubuntu ubuntu 1327 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/objectstore_debug/bluestore-stupid.yaml 2024-03-27T21:42:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: 404116 4 -rw-r--r-- 1 ubuntu ubuntu 26 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/.gitignore 2024-03-27T21:42:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: 404382 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mon_election 2024-03-27T21:42:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: 404384 4 -rw-r--r-- 1 ubuntu ubuntu 83 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mon_election/connectivity.yaml 2024-03-27T21:42:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: 404383 4 -rw-r--r-- 1 ubuntu ubuntu 83 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/mon_election/classic.yaml 2024-03-27T21:42:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: 408610 4 drwxr-xr-x 25 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits 2024-03-27T21:42:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: 408925 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-03-27T21:42:45.151 INFO:teuthology.orchestra.run.smithi060.stdout: 408935 4 -rwxr-xr-x 1 ubuntu ubuntu 207 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsync-tester.sh 2024-03-27T21:42:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: 408928 4 -rwxr-xr-x 1 ubuntu ubuntu 2474 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/cephfs_journal_tool_smoke.sh 2024-03-27T21:42:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: 408936 4 -rwxr-xr-x 1 ubuntu ubuntu 283 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iogen.sh 2024-03-27T21:42:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: 408930 4 -rwxr-xr-x 1 ubuntu ubuntu 49 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-03-27T21:42:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: 408934 4 -rwxr-xr-x 1 ubuntu ubuntu 403 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh 2024-03-27T21:42:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: 408940 4 -rw-r--r-- 1 ubuntu ubuntu 1345 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-03-27T21:42:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: 408939 4 -rwxr-xr-x 1 ubuntu ubuntu 269 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh 2024-03-27T21:42:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: 408931 4 -rwxr-xr-x 1 ubuntu ubuntu 464 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-03-27T21:42:45.152 INFO:teuthology.orchestra.run.smithi060.stdout: 408937 4 -rwxr-xr-x 1 ubuntu ubuntu 656 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone-sync.sh 2024-03-27T21:42:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: 408932 4 -rwxr-xr-x 1 ubuntu ubuntu 791 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fio.sh 2024-03-27T21:42:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: 408926 4 -rwxr-xr-x 1 ubuntu ubuntu 320 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh 2024-03-27T21:42:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: 408929 4 -rwxr-xr-x 1 ubuntu ubuntu 39 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench-short.sh 2024-03-27T21:42:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: 408927 4 -rwxr-xr-x 1 ubuntu ubuntu 193 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/bonnie.sh 2024-03-27T21:42:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: 408938 4 -rwxr-xr-x 1 ubuntu ubuntu 179 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh 2024-03-27T21:42:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: 408933 4 -rwxr-xr-x 1 ubuntu ubuntu 319 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-03-27T21:42:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: 408861 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rest 2024-03-27T21:42:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: 408862 4 -rwxr-xr-x 1 ubuntu ubuntu 233 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rest/test-restful.sh 2024-03-27T21:42:45.153 INFO:teuthology.orchestra.run.smithi060.stdout: 408863 4 -rwxr-xr-x 1 ubuntu ubuntu 2963 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rest/test_mgr_rest_api.py 2024-03-27T21:42:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: 408734 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/mgr 2024-03-27T21:42:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: 408736 4 -rwxr-xr-x 1 ubuntu ubuntu 624 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/mgr/test_per_module_finisher.sh 2024-03-27T21:42:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: 408735 4 -rwxr-xr-x 1 ubuntu ubuntu 459 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/mgr/test_localpool.sh 2024-03-27T21:42:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: 408849 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rename 2024-03-27T21:42:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: 408851 4 -rwxr-xr-x 1 ubuntu ubuntu 550 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rename/dir_pri_nul.sh 2024-03-27T21:42:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: 408850 4 -rwxr-xr-x 1 ubuntu ubuntu 861 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rename/all.sh 2024-03-27T21:42:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: 408857 4 -rwxr-xr-x 1 ubuntu ubuntu 402 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rename/pri_rem.sh 2024-03-27T21:42:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: 408853 4 -rw-r--r-- 1 ubuntu ubuntu 1911 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rename/plan.txt 2024-03-27T21:42:45.154 INFO:teuthology.orchestra.run.smithi060.stdout: 408855 4 -rwxr-xr-x 1 ubuntu ubuntu 125 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rename/pri_nul.sh 2024-03-27T21:42:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: 408852 4 -rwxr-xr-x 1 ubuntu ubuntu 161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rename/dir_pri_pri.sh 2024-03-27T21:42:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: 408858 4 -rwxr-xr-x 1 ubuntu ubuntu 368 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rename/rem_nul.sh 2024-03-27T21:42:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: 408854 4 -rwxr-xr-x 1 ubuntu ubuntu 604 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rename/prepare.sh 2024-03-27T21:42:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: 408860 4 -rwxr-xr-x 1 ubuntu ubuntu 963 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rename/rem_rem.sh 2024-03-27T21:42:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: 408856 4 -rwxr-xr-x 1 ubuntu ubuntu 172 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rename/pri_pri.sh 2024-03-27T21:42:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: 408859 4 -rwxr-xr-x 1 ubuntu ubuntu 392 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rename/rem_pri.sh 2024-03-27T21:42:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: 408753 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/objectstore 2024-03-27T21:42:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: 408754 8 -rwxr-xr-x 1 ubuntu ubuntu 4352 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/objectstore/test_fuse.sh 2024-03-27T21:42:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: 408666 4 drwxr-xr-x 9 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs 2024-03-27T21:42:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: 408682 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc 2024-03-27T21:42:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: 408694 4 -rwxr-xr-x 1 ubuntu ubuntu 1836 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/rstats.sh 2024-03-27T21:42:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: 408692 4 -rwxr-xr-x 1 ubuntu ubuntu 285 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/mkpool_layout_vxattrs.sh 2024-03-27T21:42:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: 408684 4 -rwxr-xr-x 1 ubuntu ubuntu 1039 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/chmod.sh 2024-03-27T21:42:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: 408690 4 -rwxr-xr-x 1 ubuntu ubuntu 423 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/i_complete_vs_rename.sh 2024-03-27T21:42:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: 408686 4 -rwxr-xr-x 1 ubuntu ubuntu 1025 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/direct_io.py 2024-03-27T21:42:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: 408696 4 -rwxr-xr-x 1 ubuntu ubuntu 224 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/xattrs.sh 2024-03-27T21:42:45.156 INFO:teuthology.orchestra.run.smithi060.stdout: 408693 4 -rwxr-xr-x 1 ubuntu ubuntu 644 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/multiple_rsync.sh 2024-03-27T21:42:45.157 INFO:teuthology.orchestra.run.smithi060.stdout: 408689 4 -rwxr-xr-x 1 ubuntu ubuntu 2299 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/filelock_interrupt.py 2024-03-27T21:42:45.157 INFO:teuthology.orchestra.run.smithi060.stdout: 408685 4 -rwxr-xr-x 1 ubuntu ubuntu 322 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dac_override.sh 2024-03-27T21:42:45.157 INFO:teuthology.orchestra.run.smithi060.stdout: 408695 4 -rwxr-xr-x 1 ubuntu ubuntu 59 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/trivial_sync.sh 2024-03-27T21:42:45.157 INFO:teuthology.orchestra.run.smithi060.stdout: 408691 8 -rwxr-xr-x 1 ubuntu ubuntu 4935 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/layout_vxattrs.sh 2024-03-27T21:42:45.157 INFO:teuthology.orchestra.run.smithi060.stdout: 408688 4 -rwxr-xr-x 1 ubuntu ubuntu 1475 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/filelock_deadlock.py 2024-03-27T21:42:45.157 INFO:teuthology.orchestra.run.smithi060.stdout: 408683 4 -rwxr-xr-x 1 ubuntu ubuntu 919 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/acl.sh 2024-03-27T21:42:45.157 INFO:teuthology.orchestra.run.smithi060.stdout: 408687 4 -rwxr-xr-x 1 ubuntu ubuntu 605 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dirfrag.sh 2024-03-27T21:42:45.157 INFO:teuthology.orchestra.run.smithi060.stdout: 408702 4 -rwxr-xr-x 1 ubuntu ubuntu 589 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snap-hierarchy.sh 2024-03-27T21:42:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: 408726 4 -rwxr-xr-x 1 ubuntu ubuntu 216 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh 2024-03-27T21:42:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: 408669 4 -rwxr-xr-x 1 ubuntu ubuntu 1541 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/cephfs_mirror_ha_gen.sh 2024-03-27T21:42:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: 408703 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps 2024-03-27T21:42:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: 408706 4 -rwxr-xr-x 1 ubuntu ubuntu 1522 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2024-03-27T21:42:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: 408715 4 -rwxr-xr-x 1 ubuntu ubuntu 628 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh 2024-03-27T21:42:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: 408716 4 -rwxr-xr-x 1 ubuntu ubuntu 351 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-name-limits.sh 2024-03-27T21:42:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: 408714 4 -rwxr-xr-x 1 ubuntu ubuntu 265 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-intodir.sh 2024-03-27T21:42:45.158 INFO:teuthology.orchestra.run.smithi060.stdout: 408707 4 -rwxr-xr-x 1 ubuntu ubuntu 129 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2024-03-27T21:42:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: 408712 4 -rwxr-xr-x 1 ubuntu ubuntu 915 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2024-03-27T21:42:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: 408720 4 -rwxr-xr-x 1 ubuntu ubuntu 407 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-snap-rm-cmp.sh 2024-03-27T21:42:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: 408713 4 -rwxr-xr-x 1 ubuntu ubuntu 295 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-hardlink.sh 2024-03-27T21:42:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: 408718 4 -rwxr-xr-x 1 ubuntu ubuntu 388 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-realm-split.sh 2024-03-27T21:42:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: 408717 4 -rwxr-xr-x 1 ubuntu ubuntu 1043 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-parents.sh 2024-03-27T21:42:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: 408704 4 -rwxr-xr-x 1 ubuntu ubuntu 244 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2024-03-27T21:42:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: 408711 4 -rwxr-xr-x 1 ubuntu ubuntu 178 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2024-03-27T21:42:45.159 INFO:teuthology.orchestra.run.smithi060.stdout: 408723 4 -rwxr-xr-x 1 ubuntu ubuntu 298 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh 2024-03-27T21:42:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: 408719 4 -rwxr-xr-x 1 ubuntu ubuntu 893 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-snap-rename.sh 2024-03-27T21:42:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: 408710 4 -rwxr-xr-x 1 ubuntu ubuntu 346 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2024-03-27T21:42:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: 408722 4 -rwxr-xr-x 1 ubuntu ubuntu 569 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-xattrwb.sh 2024-03-27T21:42:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: 408709 4 -rwxr-xr-x 1 ubuntu ubuntu 178 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2024-03-27T21:42:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: 408708 4 -rwxr-xr-x 1 ubuntu ubuntu 492 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2024-03-27T21:42:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: 408705 4 -rwxr-xr-x 1 ubuntu ubuntu 402 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2024-03-27T21:42:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: 408721 4 -rwxr-xr-x 1 ubuntu ubuntu 496 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-upchildrealms.sh 2024-03-27T21:42:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: 408697 4 -rwxr-xr-x 1 ubuntu ubuntu 854 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/multiclient_sync_read_eof.py 2024-03-27T21:42:45.160 INFO:teuthology.orchestra.run.smithi060.stdout: 408670 4 -rwxr-xr-x 1 ubuntu ubuntu 1223 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/cephfs_mirror_ha_verify.sh 2024-03-27T21:42:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: 408725 4 -rwxr-xr-x 1 ubuntu ubuntu 78 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_o_trunc.sh 2024-03-27T21:42:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: 408668 4 -rw-r--r-- 1 ubuntu ubuntu 134 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/Makefile 2024-03-27T21:42:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: 408676 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/full 2024-03-27T21:42:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: 408679 4 -rwxr-xr-x 1 ubuntu ubuntu 2596 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/full/subvolume_snapshot_rm.sh 2024-03-27T21:42:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: 408677 4 -rwxr-xr-x 1 ubuntu ubuntu 3667 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/full/subvolume_clone.sh 2024-03-27T21:42:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: 408678 4 -rwxr-xr-x 1 ubuntu ubuntu 1764 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/full/subvolume_rm.sh 2024-03-27T21:42:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: 408680 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/maxentries 2024-03-27T21:42:45.161 INFO:teuthology.orchestra.run.smithi060.stdout: 408681 4 -rwxr-xr-x 1 ubuntu ubuntu 2730 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/maxentries/maxentries.sh 2024-03-27T21:42:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: 408675 4 -rwxr-xr-x 1 ubuntu ubuntu 2581 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh 2024-03-27T21:42:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: 408671 4 -rw-r--r-- 1 ubuntu ubuntu 1289 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/cephfs_mirror_helpers.sh 2024-03-27T21:42:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: 408672 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/damage 2024-03-27T21:42:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: 408674 8 -rwxr-xr-x 1 ubuntu ubuntu 4385 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/damage/test-first-damage.sh 2024-03-27T21:42:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: 408673 8 -rwxr-xr-x 1 ubuntu ubuntu 4269 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/damage/test-first-damage-lost-found.sh 2024-03-27T21:42:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: 408700 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota 2024-03-27T21:42:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: 408701 4 -rwxr-xr-x 1 ubuntu ubuntu 3702 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh 2024-03-27T21:42:45.162 INFO:teuthology.orchestra.run.smithi060.stdout: 408698 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/norstats 2024-03-27T21:42:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: 408699 4 -rwxr-xr-x 1 ubuntu ubuntu 376 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/norstats/kernel_untar_tar.sh 2024-03-27T21:42:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: 408724 4 -rw-r--r-- 1 ubuntu ubuntu 851 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_o_trunc.c 2024-03-27T21:42:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: 408667 4 -rw-r--r-- 1 ubuntu ubuntu 13 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/.gitignore 2024-03-27T21:42:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: 408758 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados 2024-03-27T21:42:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: 408762 4 -rwxr-xr-x 1 ubuntu ubuntu 207 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/load-gen-mix-small.sh 2024-03-27T21:42:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: 408765 4 -rwxr-xr-x 1 ubuntu ubuntu 128 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/stress_watch.sh 2024-03-27T21:42:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: 408775 8 -rwxr-xr-x 1 ubuntu ubuntu 4467 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_large_omap_detection.py 2024-03-27T21:42:45.163 INFO:teuthology.orchestra.run.smithi060.stdout: 408761 4 -rwxr-xr-x 1 ubuntu ubuntu 208 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/load-gen-mix-small-long.sh 2024-03-27T21:42:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: 408767 8 -rwxr-xr-x 1 ubuntu ubuntu 4881 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_alloc_hint.sh 2024-03-27T21:42:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: 408781 4 -rwxr-xr-x 1 ubuntu ubuntu 1338 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_rados_timeouts.sh 2024-03-27T21:42:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: 408777 4 -rwxr-xr-x 1 ubuntu ubuntu 2134 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_librados_build.sh 2024-03-27T21:42:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: 408764 4 -rwxr-xr-x 1 ubuntu ubuntu 208 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/load-gen-mostlyread.sh 2024-03-27T21:42:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: 408768 8 -rwxr-xr-x 1 ubuntu ubuntu 5597 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_cache_pool.sh 2024-03-27T21:42:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: 408780 4 -rwxr-xr-x 1 ubuntu ubuntu 137 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh 2024-03-27T21:42:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: 408783 4 -rwxr-xr-x 1 ubuntu ubuntu 998 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/version_number_sanity.sh 2024-03-27T21:42:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: 408766 4 -rwxr-xr-x 1 ubuntu ubuntu 1979 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-03-27T21:42:45.164 INFO:teuthology.orchestra.run.smithi060.stdout: 408771 24 -rwxr-xr-x 1 ubuntu ubuntu 21829 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_dedup_tool.sh 2024-03-27T21:42:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: 408779 4 -rwxr-xr-x 1 ubuntu ubuntu 1210 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_pool_quota.sh 2024-03-27T21:42:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: 408774 4 -rwxr-xr-x 1 ubuntu ubuntu 1899 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_health_warnings.sh 2024-03-27T21:42:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: 408763 4 -rwxr-xr-x 1 ubuntu ubuntu 208 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/load-gen-mix.sh 2024-03-27T21:42:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: 408778 4 -rwxr-xr-x 1 ubuntu ubuntu 2235 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_pool_access.sh 2024-03-27T21:42:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: 408769 4 -rwxr-xr-x 1 ubuntu ubuntu 1406 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_crash.sh 2024-03-27T21:42:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: 408776 4 -rwxr-xr-x 1 ubuntu ubuntu 2769 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_libcephsqlite.sh 2024-03-27T21:42:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: 408770 4 -rwxr-xr-x 1 ubuntu ubuntu 3064 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_crushdiff.sh 2024-03-27T21:42:45.165 INFO:teuthology.orchestra.run.smithi060.stdout: 408782 32 -rwxr-xr-x 1 ubuntu ubuntu 31345 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_rados_tool.sh 2024-03-27T21:42:45.166 INFO:teuthology.orchestra.run.smithi060.stdout: 408760 4 -rwxr-xr-x 1 ubuntu ubuntu 218 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/load-gen-big.sh 2024-03-27T21:42:45.166 INFO:teuthology.orchestra.run.smithi060.stdout: 408772 4 -rwxr-xr-x 1 ubuntu ubuntu 2690 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_envlibrados_for_rocksdb.sh 2024-03-27T21:42:45.166 INFO:teuthology.orchestra.run.smithi060.stdout: 408759 4 -rwxr-xr-x 1 ubuntu ubuntu 298 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/clone.sh 2024-03-27T21:42:45.166 INFO:teuthology.orchestra.run.smithi060.stdout: 408773 4 -rwxr-xr-x 1 ubuntu ubuntu 108 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_hang.sh 2024-03-27T21:42:45.166 INFO:teuthology.orchestra.run.smithi060.stdout: 408626 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool 2024-03-27T21:42:45.166 INFO:teuthology.orchestra.run.smithi060.stdout: 408627 104 -rwxr-xr-x 1 ubuntu ubuntu 106106 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh 2024-03-27T21:42:45.166 INFO:teuthology.orchestra.run.smithi060.stdout: 408629 4 -rwxr-xr-x 1 ubuntu ubuntu 1976 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh 2024-03-27T21:42:45.166 INFO:teuthology.orchestra.run.smithi060.stdout: 408628 4 -rwxr-xr-x 1 ubuntu ubuntu 1184 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_daemon.sh 2024-03-27T21:42:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: 408737 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon 2024-03-27T21:42:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: 408738 4 -rwxr-xr-x 1 ubuntu ubuntu 2610 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/auth_caps.sh 2024-03-27T21:42:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: 408750 16 -rwxr-xr-x 1 ubuntu ubuntu 15314 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/test_mon_config_key.py 2024-03-27T21:42:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: 408743 12 -rwxr-xr-x 1 ubuntu ubuntu 8549 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh 2024-03-27T21:42:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: 408752 4 -rwxr-xr-x 1 ubuntu ubuntu 2617 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/test_noautoscale_flag.sh 2024-03-27T21:42:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: 408745 8 -rwxr-xr-x 1 ubuntu ubuntu 5383 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/pg_autoscaler.sh 2024-03-27T21:42:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: 408742 8 -rwxr-xr-x 1 ubuntu ubuntu 4685 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/config.sh 2024-03-27T21:42:45.167 INFO:teuthology.orchestra.run.smithi060.stdout: 408751 8 -rwxr-xr-x 1 ubuntu ubuntu 5230 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/test_mon_osdmap_prune.sh 2024-03-27T21:42:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: 408749 8 -rwxr-xr-x 1 ubuntu ubuntu 7172 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/test_config_key_caps.sh 2024-03-27T21:42:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: 408744 4 -rwxr-xr-x 1 ubuntu ubuntu 326 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/osd.sh 2024-03-27T21:42:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: 408740 12 -rw-r--r-- 1 ubuntu ubuntu 9820 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/caps.py 2024-03-27T21:42:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: 408746 4 -rwxr-xr-x 1 ubuntu ubuntu 3047 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/ping.py 2024-03-27T21:42:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: 408747 4 -rwxr-xr-x 1 ubuntu ubuntu 3062 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/pool_ops.sh 2024-03-27T21:42:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: 408739 4 -rwxr-xr-x 1 ubuntu ubuntu 1642 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/auth_key_rotation.sh 2024-03-27T21:42:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: 408748 4 -rwxr-xr-x 1 ubuntu ubuntu 2217 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/rbd_snaps_ops.sh 2024-03-27T21:42:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: 408741 4 -rwxr-xr-x 1 ubuntu ubuntu 2743 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/caps.sh 2024-03-27T21:42:45.168 INFO:teuthology.orchestra.run.smithi060.stdout: 408944 4 -rwxr-xr-x 1 ubuntu ubuntu 1025 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/test_telemetry_quincy_x.sh 2024-03-27T21:42:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: 408943 4 -rwxr-xr-x 1 ubuntu ubuntu 1239 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/test_telemetry_quincy.sh 2024-03-27T21:42:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: 408866 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2024-03-27T21:42:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: 408915 12 -rwxr-xr-x 1 ubuntu ubuntu 9117 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_bucket_check.py 2024-03-27T21:42:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: 408908 12 -rwxr-xr-x 1 ubuntu ubuntu 11479 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/s3_bucket_quota.pl 2024-03-27T21:42:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: 408911 8 -rw-r--r-- 1 ubuntu ubuntu 6010 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/s3_utilities.pm 2024-03-27T21:42:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: 408917 8 -rwxr-xr-x 1 ubuntu ubuntu 6865 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_datacache.py 2024-03-27T21:42:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: 408903 4 -rwxr-xr-x 1 ubuntu ubuntu 257 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-d4n.sh 2024-03-27T21:42:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: 408906 4 -rwxr-xr-x 1 ubuntu ubuntu 846 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-s3tests.sh 2024-03-27T21:42:45.170 INFO:teuthology.orchestra.run.smithi060.stdout: 408904 4 -rwxr-xr-x 1 ubuntu ubuntu 337 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-datacache.sh 2024-03-27T21:42:45.170 INFO:teuthology.orchestra.run.smithi060.stdout: 408912 8 -rwxr-xr-x 1 ubuntu ubuntu 6300 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test-keystone-service-token.py 2024-03-27T21:42:45.170 INFO:teuthology.orchestra.run.smithi060.stdout: 408900 4 -rw-r--r-- 1 ubuntu ubuntu 115 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/olh_noname_key 2024-03-27T21:42:45.170 INFO:teuthology.orchestra.run.smithi060.stdout: 408868 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum 2024-03-27T21:42:45.170 INFO:teuthology.orchestra.run.smithi060.stdout: 408874 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/README.md 2024-03-27T21:42:45.170 INFO:teuthology.orchestra.run.smithi060.stdout: 408881 12 -rwxr-xr-x 1 ubuntu ubuntu 11289 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/mvnw 2024-03-27T21:42:45.170 INFO:teuthology.orchestra.run.smithi060.stdout: 408875 0 -rw-r--r-- 1 ubuntu ubuntu 0 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/file-0b 2024-03-27T21:42:45.170 INFO:teuthology.orchestra.run.smithi060.stdout: 408885 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/src 2024-03-27T21:42:45.171 INFO:teuthology.orchestra.run.smithi060.stdout: 408892 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/src/test 2024-03-27T21:42:45.171 INFO:teuthology.orchestra.run.smithi060.stdout: 408893 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/src/test/java 2024-03-27T21:42:45.171 INFO:teuthology.orchestra.run.smithi060.stdout: 408894 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/src/test/java/io 2024-03-27T21:42:45.171 INFO:teuthology.orchestra.run.smithi060.stdout: 408895 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/src/test/java/io/ceph 2024-03-27T21:42:45.171 INFO:teuthology.orchestra.run.smithi060.stdout: 408896 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/src/test/java/io/ceph/jcksum 2024-03-27T21:42:45.171 INFO:teuthology.orchestra.run.smithi060.stdout: 408897 12 -rw-r--r-- 1 ubuntu ubuntu 9530 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/src/test/java/io/ceph/jcksum/PutObjects.java 2024-03-27T21:42:45.171 INFO:teuthology.orchestra.run.smithi060.stdout: 408886 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/src/main 2024-03-27T21:42:45.171 INFO:teuthology.orchestra.run.smithi060.stdout: 408887 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/src/main/java 2024-03-27T21:42:45.171 INFO:teuthology.orchestra.run.smithi060.stdout: 408888 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/src/main/java/io 2024-03-27T21:42:45.172 INFO:teuthology.orchestra.run.smithi060.stdout: 408889 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/src/main/java/io/ceph 2024-03-27T21:42:45.172 INFO:teuthology.orchestra.run.smithi060.stdout: 408890 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/src/main/java/io/ceph/jcksum 2024-03-27T21:42:45.172 INFO:teuthology.orchestra.run.smithi060.stdout: 408891 16 -rw-r--r-- 1 ubuntu ubuntu 16148 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/src/main/java/io/ceph/jcksum/jcksum.java 2024-03-27T21:42:45.172 INFO:teuthology.orchestra.run.smithi060.stdout: 408879 256 -rw-r--r-- 1 ubuntu ubuntu 262144 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/file-256k 2024-03-27T21:42:45.172 INFO:teuthology.orchestra.run.smithi060.stdout: 408883 4 -rw-r--r-- 1 ubuntu ubuntu 1496 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/pom-SNAPSHOT.xml 2024-03-27T21:42:45.172 INFO:teuthology.orchestra.run.smithi060.stdout: 408876 1024 -rw-r--r-- 1 ubuntu ubuntu 1048576 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/file-1m 2024-03-27T21:42:45.172 INFO:teuthology.orchestra.run.smithi060.stdout: 408877 4 -rw-r--r-- 1 ubuntu ubuntu 200 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/file-200b 2024-03-27T21:42:45.172 INFO:teuthology.orchestra.run.smithi060.stdout: 408880 4 -rw-r--r-- 1 ubuntu ubuntu 8 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/file-8b 2024-03-27T21:42:45.173 INFO:teuthology.orchestra.run.smithi060.stdout: 408869 4 -rw-r--r-- 1 ubuntu ubuntu 164 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/.gitignore 2024-03-27T21:42:45.173 INFO:teuthology.orchestra.run.smithi060.stdout: 408870 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/.mvn 2024-03-27T21:42:45.173 INFO:teuthology.orchestra.run.smithi060.stdout: 408871 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/.mvn/wrapper 2024-03-27T21:42:45.173 INFO:teuthology.orchestra.run.smithi060.stdout: 408872 64 -rw-r--r-- 1 ubuntu ubuntu 62547 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/.mvn/wrapper/maven-wrapper.jar 2024-03-27T21:42:45.173 INFO:teuthology.orchestra.run.smithi060.stdout: 408873 4 -rw-r--r-- 1 ubuntu ubuntu 1018 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/.mvn/wrapper/maven-wrapper.properties 2024-03-27T21:42:45.173 INFO:teuthology.orchestra.run.smithi060.stdout: 408878 24 -rw-r--r-- 1 ubuntu ubuntu 21983 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/file-21983b 2024-03-27T21:42:45.173 INFO:teuthology.orchestra.run.smithi060.stdout: 408884 8 -rw-r--r-- 1 ubuntu ubuntu 4116 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/pom.xml 2024-03-27T21:42:45.173 INFO:teuthology.orchestra.run.smithi060.stdout: 408882 8 -rw-r--r-- 1 ubuntu ubuntu 7591 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/jcksum/mvnw.cmd 2024-03-27T21:42:45.174 INFO:teuthology.orchestra.run.smithi060.stdout: 408901 4 -rw-r--r-- 1 ubuntu ubuntu 71 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/olh_noname_val 2024-03-27T21:42:45.174 INFO:teuthology.orchestra.run.smithi060.stdout: 408919 4 -rwxr-xr-x 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_obj.sh 2024-03-27T21:42:45.174 INFO:teuthology.orchestra.run.smithi060.stdout: 408899 4 -rwxr-xr-x 1 ubuntu ubuntu 978 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/keystone-service-token.sh 2024-03-27T21:42:45.174 INFO:teuthology.orchestra.run.smithi060.stdout: 408902 4 -rwxr-xr-x 1 ubuntu ubuntu 337 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-bucket-check.sh 2024-03-27T21:42:45.174 INFO:teuthology.orchestra.run.smithi060.stdout: 408923 4 -rwxr-xr-x 1 ubuntu ubuntu 2838 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_s3_mp_reupload.sh 2024-03-27T21:42:45.174 INFO:teuthology.orchestra.run.smithi060.stdout: 408922 4 -rwxr-xr-x 1 ubuntu ubuntu 3705 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_s3_mp_reupload.py 2024-03-27T21:42:45.174 INFO:teuthology.orchestra.run.smithi060.stdout: 408909 8 -rwxr-xr-x 1 ubuntu ubuntu 4108 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/s3_multipart_upload.pl 2024-03-27T21:42:45.174 INFO:teuthology.orchestra.run.smithi060.stdout: 408905 4 -rwxr-xr-x 1 ubuntu ubuntu 479 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-reshard.sh 2024-03-27T21:42:45.174 INFO:teuthology.orchestra.run.smithi060.stdout: 408867 4 -rwxr-xr-x 1 ubuntu ubuntu 3934 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/common.py 2024-03-27T21:42:45.175 INFO:teuthology.orchestra.run.smithi060.stdout: 408913 4 -rwxr-xr-x 1 ubuntu ubuntu 1459 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_awssdkv4_sig.sh 2024-03-27T21:42:45.175 INFO:teuthology.orchestra.run.smithi060.stdout: 408907 4 -rwxr-xr-x 1 ubuntu ubuntu 335 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-versioning.sh 2024-03-27T21:42:45.175 INFO:teuthology.orchestra.run.smithi060.stdout: 408914 4 -rwxr-xr-x 1 ubuntu ubuntu 2254 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_librgw_file.sh 2024-03-27T21:42:45.175 INFO:teuthology.orchestra.run.smithi060.stdout: 408921 16 -rwxr-xr-x 1 ubuntu ubuntu 14105 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_reshard.py 2024-03-27T21:42:45.175 INFO:teuthology.orchestra.run.smithi060.stdout: 408898 8 -rwxr-xr-x 1 ubuntu ubuntu 5898 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/keystone-fake-server.py 2024-03-27T21:42:45.175 INFO:teuthology.orchestra.run.smithi060.stdout: 408924 8 -rwxr-xr-x 1 ubuntu ubuntu 5386 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_versioning.py 2024-03-27T21:42:45.175 INFO:teuthology.orchestra.run.smithi060.stdout: 408920 16 -rwxr-xr-x 1 ubuntu ubuntu 12507 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_orphan_list.sh 2024-03-27T21:42:45.175 INFO:teuthology.orchestra.run.smithi060.stdout: 408910 8 -rwxr-xr-x 1 ubuntu ubuntu 5454 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/s3_user_quota.pl 2024-03-27T21:42:45.176 INFO:teuthology.orchestra.run.smithi060.stdout: 408916 12 -rw-r--r-- 1 ubuntu ubuntu 8640 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_d4n.py 2024-03-27T21:42:45.176 INFO:teuthology.orchestra.run.smithi060.stdout: 408918 4 -rwxr-xr-x 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_gc_log.sh 2024-03-27T21:42:45.176 INFO:teuthology.orchestra.run.smithi060.stdout: 408612 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/caps 2024-03-27T21:42:45.176 INFO:teuthology.orchestra.run.smithi060.stdout: 408613 4 -rwxr-xr-x 1 ubuntu ubuntu 1352 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/caps/mon_commands.sh 2024-03-27T21:42:45.176 INFO:teuthology.orchestra.run.smithi060.stdout: 408946 4 -rwxr-xr-x 1 ubuntu ubuntu 986 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/test_telemetry_reef_x.sh 2024-03-27T21:42:45.176 INFO:teuthology.orchestra.run.smithi060.stdout: 408942 4 -rwxr-xr-x 1 ubuntu ubuntu 1374 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/test_telemetry_pacific_x.sh 2024-03-27T21:42:45.176 INFO:teuthology.orchestra.run.smithi060.stdout: 408864 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/restart 2024-03-27T21:42:45.176 INFO:teuthology.orchestra.run.smithi060.stdout: 408865 8 -rwxr-xr-x 1 ubuntu ubuntu 6980 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/restart/test-backtraces.py 2024-03-27T21:42:45.176 INFO:teuthology.orchestra.run.smithi060.stdout: 408611 4 -rw-r--r-- 1 ubuntu ubuntu 81 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/Makefile 2024-03-27T21:42:45.177 INFO:teuthology.orchestra.run.smithi060.stdout: 408948 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/windows 2024-03-27T21:42:45.177 INFO:teuthology.orchestra.run.smithi060.stdout: 408949 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/windows/libvirt_vm 2024-03-27T21:42:45.177 INFO:teuthology.orchestra.run.smithi060.stdout: 408954 8 -rw-r--r-- 1 ubuntu ubuntu 4579 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/windows/libvirt_vm/utils.ps1 2024-03-27T21:42:45.177 INFO:teuthology.orchestra.run.smithi060.stdout: 408952 4 -rw-r--r-- 1 ubuntu ubuntu 1485 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/windows/libvirt_vm/setup.ps1 2024-03-27T21:42:45.177 INFO:teuthology.orchestra.run.smithi060.stdout: 408950 8 -rw-r--r-- 1 ubuntu ubuntu 5931 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/windows/libvirt_vm/autounattend.xml 2024-03-27T21:42:45.177 INFO:teuthology.orchestra.run.smithi060.stdout: 408951 4 -rw-r--r-- 1 ubuntu ubuntu 1763 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/windows/libvirt_vm/first-logon.ps1 2024-03-27T21:42:45.177 INFO:teuthology.orchestra.run.smithi060.stdout: 408953 8 -rwxr-xr-x 1 ubuntu ubuntu 5714 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/windows/libvirt_vm/setup.sh 2024-03-27T21:42:45.177 INFO:teuthology.orchestra.run.smithi060.stdout: 408957 32 -rw-r--r-- 1 ubuntu ubuntu 31172 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/windows/test_rbd_wnbd.py 2024-03-27T21:42:45.178 INFO:teuthology.orchestra.run.smithi060.stdout: 408955 4 -rw-r--r-- 1 ubuntu ubuntu 1163 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/windows/run-tests.ps1 2024-03-27T21:42:45.178 INFO:teuthology.orchestra.run.smithi060.stdout: 408956 4 -rw-r--r-- 1 ubuntu ubuntu 305 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/windows/run-tests.sh 2024-03-27T21:42:45.178 INFO:teuthology.orchestra.run.smithi060.stdout: 408647 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io 2024-03-27T21:42:45.178 INFO:teuthology.orchestra.run.smithi060.stdout: 408651 12 -rw-r--r-- 1 ubuntu ubuntu 9251 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io/direct_io_test.c 2024-03-27T21:42:45.178 INFO:teuthology.orchestra.run.smithi060.stdout: 408654 8 -rw-r--r-- 1 ubuntu ubuntu 5551 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io/test_sync_io.c 2024-03-27T21:42:45.178 INFO:teuthology.orchestra.run.smithi060.stdout: 408652 4 -rwxr-xr-x 1 ubuntu ubuntu 466 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io/misc.sh 2024-03-27T21:42:45.178 INFO:teuthology.orchestra.run.smithi060.stdout: 408650 4 -rwxr-xr-x 1 ubuntu ubuntu 114 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io/big.sh 2024-03-27T21:42:45.178 INFO:teuthology.orchestra.run.smithi060.stdout: 408649 4 -rw-r--r-- 1 ubuntu ubuntu 169 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io/Makefile 2024-03-27T21:42:45.179 INFO:teuthology.orchestra.run.smithi060.stdout: 408653 4 -rw-r--r-- 1 ubuntu ubuntu 1168 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io/test_short_dio_read.c 2024-03-27T21:42:45.179 INFO:teuthology.orchestra.run.smithi060.stdout: 408648 4 -rw-r--r-- 1 ubuntu ubuntu 51 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io/.gitignore 2024-03-27T21:42:45.179 INFO:teuthology.orchestra.run.smithi060.stdout: 408727 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/hadoop 2024-03-27T21:42:45.179 INFO:teuthology.orchestra.run.smithi060.stdout: 408729 4 -rwxr-xr-x 1 ubuntu ubuntu 2189 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/hadoop/terasort.sh 2024-03-27T21:42:45.179 INFO:teuthology.orchestra.run.smithi060.stdout: 408730 4 -rwxr-xr-x 1 ubuntu ubuntu 968 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/hadoop/wordcount.sh 2024-03-27T21:42:45.179 INFO:teuthology.orchestra.run.smithi060.stdout: 408728 4 -rwxr-xr-x 1 ubuntu ubuntu 1154 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/hadoop/repl.sh 2024-03-27T21:42:45.179 INFO:teuthology.orchestra.run.smithi060.stdout: 408945 4 -rwxr-xr-x 1 ubuntu ubuntu 1191 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/test_telemetry_reef.sh 2024-03-27T21:42:45.179 INFO:teuthology.orchestra.run.smithi060.stdout: 408941 4 -rwxr-xr-x 1 ubuntu ubuntu 573 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/test_telemetry_pacific.sh 2024-03-27T21:42:45.180 INFO:teuthology.orchestra.run.smithi060.stdout: 408731 4 -rwxr-xr-x 1 ubuntu ubuntu 329 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh 2024-03-27T21:42:45.180 INFO:teuthology.orchestra.run.smithi060.stdout: 408614 4 -rwxr-xr-x 1 ubuntu ubuntu 3944 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/ceph-helpers-root.sh 2024-03-27T21:42:45.180 INFO:teuthology.orchestra.run.smithi060.stdout: 408732 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs 2024-03-27T21:42:45.180 INFO:teuthology.orchestra.run.smithi060.stdout: 408733 4 -rwxr-xr-x 1 ubuntu ubuntu 209 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh 2024-03-27T21:42:45.180 INFO:teuthology.orchestra.run.smithi060.stdout: 408665 4 -rw-r--r-- 1 ubuntu ubuntu 20 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/false.sh 2024-03-27T21:42:45.180 INFO:teuthology.orchestra.run.smithi060.stdout: 408757 4 -rwxr-xr-x 1 ubuntu ubuntu 137 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/post-file.sh 2024-03-27T21:42:45.180 INFO:teuthology.orchestra.run.smithi060.stdout: 408784 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd 2024-03-27T21:42:45.180 INFO:teuthology.orchestra.run.smithi060.stdout: 408798 12 -rwxr-xr-x 1 ubuntu ubuntu 8280 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/journal.sh 2024-03-27T21:42:45.181 INFO:teuthology.orchestra.run.smithi060.stdout: 408787 4 -rwxr-xr-x 1 ubuntu ubuntu 2931 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/compare_mirror_image_alternate_primary.sh 2024-03-27T21:42:45.181 INFO:teuthology.orchestra.run.smithi060.stdout: 408822 4 -rwxr-xr-x 1 ubuntu ubuntu 1664 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/nvmeof_setup_subsystem.sh 2024-03-27T21:42:45.181 INFO:teuthology.orchestra.run.smithi060.stdout: 408796 12 -rwxr-xr-x 1 ubuntu ubuntu 8814 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/import_export.sh 2024-03-27T21:42:45.181 INFO:teuthology.orchestra.run.smithi060.stdout: 408827 4 -rwxr-xr-x 1 ubuntu ubuntu 3502 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/qos.sh 2024-03-27T21:42:45.181 INFO:teuthology.orchestra.run.smithi060.stdout: 408817 8 -rwxr-xr-x 1 ubuntu ubuntu 7100 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/merge_diff.sh 2024-03-27T21:42:45.181 INFO:teuthology.orchestra.run.smithi060.stdout: 408810 4 -rwxr-xr-x 1 ubuntu ubuntu 2397 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/krbd_udev_netns.sh 2024-03-27T21:42:45.181 INFO:teuthology.orchestra.run.smithi060.stdout: 408814 8 -rwxr-xr-x 1 ubuntu ubuntu 6933 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/luks-encryption.sh 2024-03-27T21:42:45.181 INFO:teuthology.orchestra.run.smithi060.stdout: 408789 12 -rwxr-xr-x 1 ubuntu ubuntu 9823 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/concurrent.sh 2024-03-27T21:42:45.182 INFO:teuthology.orchestra.run.smithi060.stdout: 408813 4 -rwxr-xr-x 1 ubuntu ubuntu 1208 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/krbd_watch_errors.sh 2024-03-27T21:42:45.182 INFO:teuthology.orchestra.run.smithi060.stdout: 408812 4 -rwxr-xr-x 1 ubuntu ubuntu 751 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/krbd_wac.sh 2024-03-27T21:42:45.182 INFO:teuthology.orchestra.run.smithi060.stdout: 408846 8 -rwxr-xr-x 1 ubuntu ubuntu 7928 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_rbd_tasks.sh 2024-03-27T21:42:45.182 INFO:teuthology.orchestra.run.smithi060.stdout: 408807 4 -rwxr-xr-x 1 ubuntu ubuntu 2666 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/krbd_stable_writes.sh 2024-03-27T21:42:45.182 INFO:teuthology.orchestra.run.smithi060.stdout: 408843 4 -rwxr-xr-x 1 ubuntu ubuntu 333 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2024-03-27T21:42:45.182 INFO:teuthology.orchestra.run.smithi060.stdout: 408792 4 -rwxr-xr-x 1 ubuntu ubuntu 1373 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff.sh 2024-03-27T21:42:45.182 INFO:teuthology.orchestra.run.smithi060.stdout: 408815 4 -rwxr-xr-x 1 ubuntu ubuntu 352 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/map-snapshot-io.sh 2024-03-27T21:42:45.182 INFO:teuthology.orchestra.run.smithi060.stdout: 408788 8 -rwxr-xr-x 1 ubuntu ubuntu 4514 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/compare_mirror_images.sh 2024-03-27T21:42:45.183 INFO:teuthology.orchestra.run.smithi060.stdout: 408841 4 -rwxr-xr-x 1 ubuntu ubuntu 3611 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_admin_socket.sh 2024-03-27T21:42:45.183 INFO:teuthology.orchestra.run.smithi060.stdout: 408845 4 -rwxr-xr-x 1 ubuntu ubuntu 191 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_rbd_mirror.sh 2024-03-27T21:42:45.183 INFO:teuthology.orchestra.run.smithi060.stdout: 408808 4 -rwxr-xr-x 1 ubuntu ubuntu 1248 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/krbd_udev_enumerate.sh 2024-03-27T21:42:45.183 INFO:teuthology.orchestra.run.smithi060.stdout: 408795 20 -rwxr-xr-x 1 ubuntu ubuntu 17897 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/image_read.sh 2024-03-27T21:42:45.183 INFO:teuthology.orchestra.run.smithi060.stdout: 408836 40 -rwxr-xr-x 1 ubuntu ubuntu 40061 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-03-27T21:42:45.183 INFO:teuthology.orchestra.run.smithi060.stdout: 408819 4 -rwxr-xr-x 1 ubuntu ubuntu 107 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/notify_slave.sh 2024-03-27T21:42:45.183 INFO:teuthology.orchestra.run.smithi060.stdout: 408830 8 -rwxr-xr-x 1 ubuntu ubuntu 5078 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_groups.sh 2024-03-27T21:42:45.183 INFO:teuthology.orchestra.run.smithi060.stdout: 408820 4 -rwxr-xr-x 1 ubuntu ubuntu 1901 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/nvmeof_basic_tests.sh 2024-03-27T21:42:45.183 INFO:teuthology.orchestra.run.smithi060.stdout: 408825 4 -rwxr-xr-x 1 ubuntu ubuntu 1067 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/qemu_dynamic_features.sh 2024-03-27T21:42:45.184 INFO:teuthology.orchestra.run.smithi060.stdout: 408834 4 -rwxr-xr-x 1 ubuntu ubuntu 228 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_mirror_fsx_prepare.sh 2024-03-27T21:42:45.184 INFO:teuthology.orchestra.run.smithi060.stdout: 408809 4 -rwxr-xr-x 1 ubuntu ubuntu 639 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/krbd_udev_netlink_enobufs.sh 2024-03-27T21:42:45.184 INFO:teuthology.orchestra.run.smithi060.stdout: 408824 4 -rwxr-xr-x 1 ubuntu ubuntu 1334 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/qemu-iotests.sh 2024-03-27T21:42:45.184 INFO:teuthology.orchestra.run.smithi060.stdout: 408826 4 -rwxr-xr-x 1 ubuntu ubuntu 769 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/qemu_rebuild_object_map.sh 2024-03-27T21:42:45.184 INFO:teuthology.orchestra.run.smithi060.stdout: 408847 4 -rwxr-xr-x 1 ubuntu ubuntu 1208 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_rbdmap_RBDMAPFILE.sh 2024-03-27T21:42:45.184 INFO:teuthology.orchestra.run.smithi060.stdout: 408790 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson 2024-03-27T21:42:45.184 INFO:teuthology.orchestra.run.smithi060.stdout: 408791 4 -rwxr-xr-x 1 ubuntu ubuntu 1632 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2024-03-27T21:42:45.184 INFO:teuthology.orchestra.run.smithi060.stdout: 408816 4 -rwxr-xr-x 1 ubuntu ubuntu 793 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/map-unmap.sh 2024-03-27T21:42:45.185 INFO:teuthology.orchestra.run.smithi060.stdout: 408844 4 -rwxr-xr-x 1 ubuntu ubuntu 1150 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2024-03-27T21:42:45.185 INFO:teuthology.orchestra.run.smithi060.stdout: 408833 4 -rwxr-xr-x 1 ubuntu ubuntu 1151 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_mirror_fsx_compare.sh 2024-03-27T21:42:45.185 INFO:teuthology.orchestra.run.smithi060.stdout: 408786 12 -rwxr-xr-x 1 ubuntu ubuntu 9606 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/cli_migration.sh 2024-03-27T21:42:45.185 INFO:teuthology.orchestra.run.smithi060.stdout: 408828 8 -rwxr-xr-x 1 ubuntu ubuntu 6519 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd-ggate.sh 2024-03-27T21:42:45.185 INFO:teuthology.orchestra.run.smithi060.stdout: 408821 4 -rwxr-xr-x 1 ubuntu ubuntu 1996 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/nvmeof_fio_test.sh 2024-03-27T21:42:45.185 INFO:teuthology.orchestra.run.smithi060.stdout: 408823 12 -rwxr-xr-x 1 ubuntu ubuntu 9384 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/permissions.sh 2024-03-27T21:42:45.185 INFO:teuthology.orchestra.run.smithi060.stdout: 408829 20 -rwxr-xr-x 1 ubuntu ubuntu 16545 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd-nbd.sh 2024-03-27T21:42:45.185 INFO:teuthology.orchestra.run.smithi060.stdout: 408838 4 -rwxr-xr-x 1 ubuntu ubuntu 2735 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh 2024-03-27T21:42:45.185 INFO:teuthology.orchestra.run.smithi060.stdout: 408831 32 -rwxr-xr-x 1 ubuntu ubuntu 32626 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_mirror.sh 2024-03-27T21:42:45.186 INFO:teuthology.orchestra.run.smithi060.stdout: 408801 8 -rwxr-xr-x 1 ubuntu ubuntu 6079 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/krbd_exclusive_option.sh 2024-03-27T21:42:45.186 INFO:teuthology.orchestra.run.smithi060.stdout: 408785 60 -rwxr-xr-x 1 ubuntu ubuntu 61271 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/cli_generic.sh 2024-03-27T21:42:45.186 INFO:teuthology.orchestra.run.smithi060.stdout: 408805 8 -rwxr-xr-x 1 ubuntu ubuntu 4226 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/krbd_namespaces.sh 2024-03-27T21:42:45.186 INFO:teuthology.orchestra.run.smithi060.stdout: 408794 4 -rwxr-xr-x 1 ubuntu ubuntu 1342 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/huge-tickets.sh 2024-03-27T21:42:45.186 INFO:teuthology.orchestra.run.smithi060.stdout: 408839 4 -rwxr-xr-x 1 ubuntu ubuntu 1557 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/read-flags.sh 2024-03-27T21:42:45.186 INFO:teuthology.orchestra.run.smithi060.stdout: 408806 4 -rwxr-xr-x 1 ubuntu ubuntu 2500 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/krbd_rxbounce.sh 2024-03-27T21:42:45.186 INFO:teuthology.orchestra.run.smithi060.stdout: 408797 4 -rwxr-xr-x 1 ubuntu ubuntu 396 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/issue-20295.sh 2024-03-27T21:42:45.186 INFO:teuthology.orchestra.run.smithi060.stdout: 408848 4 -rwxr-xr-x 1 ubuntu ubuntu 634 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/verify_pool.sh 2024-03-27T21:42:45.187 INFO:teuthology.orchestra.run.smithi060.stdout: 408811 4 -rwxr-xr-x 1 ubuntu ubuntu 2375 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/krbd_udev_symlinks.sh 2024-03-27T21:42:45.187 INFO:teuthology.orchestra.run.smithi060.stdout: 408842 4 -rwxr-xr-x 1 ubuntu ubuntu 183 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh 2024-03-27T21:42:45.187 INFO:teuthology.orchestra.run.smithi060.stdout: 408799 4 -rwxr-xr-x 1 ubuntu ubuntu 2952 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/kernel.sh 2024-03-27T21:42:45.187 INFO:teuthology.orchestra.run.smithi060.stdout: 408832 4 -rwxr-xr-x 1 ubuntu ubuntu 2591 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_mirror_bootstrap.sh 2024-03-27T21:42:45.187 INFO:teuthology.orchestra.run.smithi060.stdout: 408840 4 -rwxr-xr-x 1 ubuntu ubuntu 196 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/simple_big.sh 2024-03-27T21:42:45.187 INFO:teuthology.orchestra.run.smithi060.stdout: 408835 8 -rwxr-xr-x 1 ubuntu ubuntu 5136 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_mirror_ha.sh 2024-03-27T21:42:45.187 INFO:teuthology.orchestra.run.smithi060.stdout: 408804 4 -rwxr-xr-x 1 ubuntu ubuntu 471 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/krbd_latest_osdmap_on_map.sh 2024-03-27T21:42:45.187 INFO:teuthology.orchestra.run.smithi060.stdout: 408802 4 -rwxr-xr-x 1 ubuntu ubuntu 4056 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/krbd_fallocate.sh 2024-03-27T21:42:45.187 INFO:teuthology.orchestra.run.smithi060.stdout: 408800 8 -rwxr-xr-x 1 ubuntu ubuntu 6189 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/krbd_data_pool.sh 2024-03-27T21:42:45.188 INFO:teuthology.orchestra.run.smithi060.stdout: 408803 4 -rwxr-xr-x 1 ubuntu ubuntu 1487 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/krbd_huge_osdmap.sh 2024-03-27T21:42:45.188 INFO:teuthology.orchestra.run.smithi060.stdout: 408818 4 -rwxr-xr-x 1 ubuntu ubuntu 108 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/notify_master.sh 2024-03-27T21:42:45.188 INFO:teuthology.orchestra.run.smithi060.stdout: 408793 4 -rwxr-xr-x 1 ubuntu ubuntu 3089 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh 2024-03-27T21:42:45.188 INFO:teuthology.orchestra.run.smithi060.stdout: 408837 8 -rwxr-xr-x 1 ubuntu ubuntu 6284 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_mirror_stress.sh 2024-03-27T21:42:45.188 INFO:teuthology.orchestra.run.smithi060.stdout: 408630 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/client 2024-03-27T21:42:45.188 INFO:teuthology.orchestra.run.smithi060.stdout: 408631 4 -rwxr-xr-x 1 ubuntu ubuntu 37 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/client/test.sh 2024-03-27T21:42:45.188 INFO:teuthology.orchestra.run.smithi060.stdout: 408617 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm 2024-03-27T21:42:45.188 INFO:teuthology.orchestra.run.smithi060.stdout: 408620 16 -rwxr-xr-x 1 ubuntu ubuntu 15287 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh 2024-03-27T21:42:45.189 INFO:teuthology.orchestra.run.smithi060.stdout: 408623 4 -rwxr-xr-x 1 ubuntu ubuntu 819 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_iscsi_etc_hosts.sh 2024-03-27T21:42:45.189 INFO:teuthology.orchestra.run.smithi060.stdout: 408621 8 -rwxr-xr-x 1 ubuntu ubuntu 5756 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm_timeout.py 2024-03-27T21:42:45.189 INFO:teuthology.orchestra.run.smithi060.stdout: 408619 4 -rwxr-xr-x 1 ubuntu ubuntu 1838 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh 2024-03-27T21:42:45.189 INFO:teuthology.orchestra.run.smithi060.stdout: 408624 4 -rwxr-xr-x 1 ubuntu ubuntu 1060 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_iscsi_pids_limit.sh 2024-03-27T21:42:45.189 INFO:teuthology.orchestra.run.smithi060.stdout: 408622 8 -rwxr-xr-x 1 ubuntu ubuntu 4325 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh 2024-03-27T21:42:45.189 INFO:teuthology.orchestra.run.smithi060.stdout: 408618 4 -rwxr-xr-x 1 ubuntu ubuntu 1325 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/create_iscsi_disks.sh 2024-03-27T21:42:45.189 INFO:teuthology.orchestra.run.smithi060.stdout: 408625 4 -rwxr-xr-x 1 ubuntu ubuntu 1249 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh 2024-03-27T21:42:45.189 INFO:teuthology.orchestra.run.smithi060.stdout: 408755 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/osdc 2024-03-27T21:42:45.190 INFO:teuthology.orchestra.run.smithi060.stdout: 408756 4 -rwxr-xr-x 1 ubuntu ubuntu 793 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/osdc/stress_objectcacher.sh 2024-03-27T21:42:45.190 INFO:teuthology.orchestra.run.smithi060.stdout: 408655 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/erasure-code 2024-03-27T21:42:45.190 INFO:teuthology.orchestra.run.smithi060.stdout: 408662 124 -rw-r--r-- 1 ubuntu ubuntu 122971 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/erasure-code/jquery.flot.js 2024-03-27T21:42:45.190 INFO:teuthology.orchestra.run.smithi060.stdout: 408658 8 -rwxr-xr-x 1 ubuntu ubuntu 5694 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/erasure-code/bench.sh 2024-03-27T21:42:45.190 INFO:teuthology.orchestra.run.smithi060.stdout: 408657 4 -rw-r--r-- 1 ubuntu ubuntu 1168 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/erasure-code/bench.html 2024-03-27T21:42:45.190 INFO:teuthology.orchestra.run.smithi060.stdout: 408660 4 -rw-r--r-- 1 ubuntu ubuntu 1738 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/erasure-code/examples.css 2024-03-27T21:42:45.190 INFO:teuthology.orchestra.run.smithi060.stdout: 408664 4 -rw-r--r-- 1 ubuntu ubuntu 2027 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/erasure-code/plot.js 2024-03-27T21:42:45.190 INFO:teuthology.orchestra.run.smithi060.stdout: 408659 4 -rwxr-xr-x 1 ubuntu ubuntu 1196 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/erasure-code/encode-decode-non-regression.sh 2024-03-27T21:42:45.191 INFO:teuthology.orchestra.run.smithi060.stdout: 408656 4 -rw-r--r-- 1 ubuntu ubuntu 12 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/erasure-code/.gitignore 2024-03-27T21:42:45.191 INFO:teuthology.orchestra.run.smithi060.stdout: 408663 260 -rw-r--r-- 1 ubuntu ubuntu 266057 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/erasure-code/jquery.js 2024-03-27T21:42:45.191 INFO:teuthology.orchestra.run.smithi060.stdout: 408661 8 -rw-r--r-- 1 ubuntu ubuntu 6033 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/erasure-code/jquery.flot.categories.js 2024-03-27T21:42:45.191 INFO:teuthology.orchestra.run.smithi060.stdout: 408615 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/ceph-tests 2024-03-27T21:42:45.191 INFO:teuthology.orchestra.run.smithi060.stdout: 408616 4 -rwxr-xr-x 1 ubuntu ubuntu 125 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/ceph-tests/ceph-admin-commands.sh 2024-03-27T21:42:45.191 INFO:teuthology.orchestra.run.smithi060.stdout: 408947 4 -rwxr-xr-x 1 ubuntu ubuntu 20 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/true.sh 2024-03-27T21:42:45.191 INFO:teuthology.orchestra.run.smithi060.stdout: 408632 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls 2024-03-27T21:42:45.191 INFO:teuthology.orchestra.run.smithi060.stdout: 408644 4 -rwxr-xr-x 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw_gc.sh 2024-03-27T21:42:45.192 INFO:teuthology.orchestra.run.smithi060.stdout: 408643 4 -rwxr-xr-x 1 ubuntu ubuntu 114 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh 2024-03-27T21:42:45.192 INFO:teuthology.orchestra.run.smithi060.stdout: 408639 4 -rwxr-xr-x 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_log.sh 2024-03-27T21:42:45.192 INFO:teuthology.orchestra.run.smithi060.stdout: 408635 4 -rwxr-xr-x 1 ubuntu ubuntu 44 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_cmpomap.sh 2024-03-27T21:42:45.192 INFO:teuthology.orchestra.run.smithi060.stdout: 408646 4 -rwxr-xr-x 1 ubuntu ubuntu 40 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_sdk.sh 2024-03-27T21:42:45.192 INFO:teuthology.orchestra.run.smithi060.stdout: 408645 4 -rwxr-xr-x 1 ubuntu ubuntu 46 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw_stats.sh 2024-03-27T21:42:45.192 INFO:teuthology.orchestra.run.smithi060.stdout: 408642 4 -rwxr-xr-x 1 ubuntu ubuntu 45 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_refcount.sh 2024-03-27T21:42:45.192 INFO:teuthology.orchestra.run.smithi060.stdout: 408637 4 -rwxr-xr-x 1 ubuntu ubuntu 119 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_journal.sh 2024-03-27T21:42:45.192 INFO:teuthology.orchestra.run.smithi060.stdout: 408636 4 -rwxr-xr-x 1 ubuntu ubuntu 42 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_hello.sh 2024-03-27T21:42:45.192 INFO:teuthology.orchestra.run.smithi060.stdout: 408640 4 -rwxr-xr-x 1 ubuntu ubuntu 43 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_numops.sh 2024-03-27T21:42:45.193 INFO:teuthology.orchestra.run.smithi060.stdout: 408641 4 -rwxr-xr-x 1 ubuntu ubuntu 111 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2024-03-27T21:42:45.193 INFO:teuthology.orchestra.run.smithi060.stdout: 408633 4 -rwxr-xr-x 1 ubuntu ubuntu 46 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh 2024-03-27T21:42:45.193 INFO:teuthology.orchestra.run.smithi060.stdout: 408638 4 -rwxr-xr-x 1 ubuntu ubuntu 41 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_lock.sh 2024-03-27T21:42:45.193 INFO:teuthology.orchestra.run.smithi060.stdout: 408634 4 -rwxr-xr-x 1 ubuntu ubuntu 111 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_cas.sh 2024-03-27T21:42:45.193 INFO:teuthology.orchestra.run.smithi060.stdout: 404244 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/config 2024-03-27T21:42:45.193 INFO:teuthology.orchestra.run.smithi060.stdout: 404246 4 -rw-r--r-- 1 ubuntu ubuntu 396 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/config/rados.yaml 2024-03-27T21:42:45.193 INFO:teuthology.orchestra.run.smithi060.stdout: 404245 4 -rw-r--r-- 1 ubuntu ubuntu 335 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/config/crimson_qa_overrides.yaml 2024-03-27T21:42:45.193 INFO:teuthology.orchestra.run.smithi060.stdout: 404247 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/crontab 2024-03-27T21:42:45.194 INFO:teuthology.orchestra.run.smithi060.stdout: 404248 8 -rw-r--r-- 1 ubuntu ubuntu 8174 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/crontab/teuthology-cronjobs 2024-03-27T21:42:45.194 INFO:teuthology.orchestra.run.smithi060.stdout: 404121 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/archs 2024-03-27T21:42:45.194 INFO:teuthology.orchestra.run.smithi060.stdout: 404125 4 -rw-r--r-- 1 ubuntu ubuntu 13 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/archs/x86_64.yaml 2024-03-27T21:42:45.194 INFO:teuthology.orchestra.run.smithi060.stdout: 404123 4 -rw-r--r-- 1 ubuntu ubuntu 13 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/archs/armv7.yaml 2024-03-27T21:42:45.194 INFO:teuthology.orchestra.run.smithi060.stdout: 404122 4 -rw-r--r-- 1 ubuntu ubuntu 14 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/archs/aarch64.yaml 2024-03-27T21:42:45.194 INFO:teuthology.orchestra.run.smithi060.stdout: 404124 4 -rw-r--r-- 1 ubuntu ubuntu 11 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/archs/i686.yaml 2024-03-27T21:42:45.194 INFO:teuthology.orchestra.run.smithi060.stdout: 404341 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/erasure-code 2024-03-27T21:42:45.194 INFO:teuthology.orchestra.run.smithi060.stdout: 404346 4 -rw-r--r-- 1 ubuntu ubuntu 484 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/erasure-code/ec-rados-plugin=isa-k=2-m=1.yaml 2024-03-27T21:42:45.195 INFO:teuthology.orchestra.run.smithi060.stdout: 404354 4 -rw-r--r-- 1 ubuntu ubuntu 431 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/erasure-code/ec-rados-sequential.yaml 2024-03-27T21:42:45.195 INFO:teuthology.orchestra.run.smithi060.stdout: 404348 4 -rw-r--r-- 1 ubuntu ubuntu 1057 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/erasure-code/ec-rados-plugin=jerasure-k=2-m=2-crush.yaml 2024-03-27T21:42:45.195 INFO:teuthology.orchestra.run.smithi060.stdout: 404350 4 -rw-r--r-- 1 ubuntu ubuntu 555 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/erasure-code/ec-rados-plugin=jerasure-k=4-m=2.yaml 2024-03-27T21:42:45.195 INFO:teuthology.orchestra.run.smithi060.stdout: 404351 4 -rw-r--r-- 1 ubuntu ubuntu 1084 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/erasure-code/ec-rados-plugin=jerasure-k=8-m=6-crush.yaml 2024-03-27T21:42:45.195 INFO:teuthology.orchestra.run.smithi060.stdout: 404345 4 -rw-r--r-- 1 ubuntu ubuntu 472 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/erasure-code/ec-rados-plugin=clay-k=4-m=2.yaml 2024-03-27T21:42:45.195 INFO:teuthology.orchestra.run.smithi060.stdout: 404349 4 -rw-r--r-- 1 ubuntu ubuntu 706 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/erasure-code/ec-rados-plugin=jerasure-k=3-m=1.yaml 2024-03-27T21:42:45.195 INFO:teuthology.orchestra.run.smithi060.stdout: 404344 4 -rw-r--r-- 1 ubuntu ubuntu 427 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/erasure-code/ec-rados-parallel.yaml 2024-03-27T21:42:45.195 INFO:teuthology.orchestra.run.smithi060.stdout: 404347 4 -rw-r--r-- 1 ubuntu ubuntu 480 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/erasure-code/ec-rados-plugin=jerasure-k=2-m=1.yaml 2024-03-27T21:42:45.195 INFO:teuthology.orchestra.run.smithi060.stdout: 404343 4 -rw-r--r-- 1 ubuntu ubuntu 388 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/erasure-code/ec-rados-default.yaml 2024-03-27T21:42:45.196 INFO:teuthology.orchestra.run.smithi060.stdout: 404342 4 -rw-r--r-- 1 ubuntu ubuntu 2094 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/erasure-code/ec-feature-plugins-v2.yaml 2024-03-27T21:42:45.196 INFO:teuthology.orchestra.run.smithi060.stdout: 404353 4 -rw-r--r-- 1 ubuntu ubuntu 450 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/erasure-code/ec-rados-plugin=shec-k=4-m=3-c=2.yaml 2024-03-27T21:42:45.196 INFO:teuthology.orchestra.run.smithi060.stdout: 404352 4 -rw-r--r-- 1 ubuntu ubuntu 448 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/erasure-code/ec-rados-plugin=lrc-k=4-m=2-l=3.yaml 2024-03-27T21:42:45.196 INFO:teuthology.orchestra.run.smithi060.stdout: 404402 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/objectstore_cephfs 2024-03-27T21:42:45.196 INFO:teuthology.orchestra.run.smithi060.stdout: 404403 0 lrwxrwxrwx 1 ubuntu ubuntu 36 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/objectstore_cephfs/bluestore-bitmap.yaml -> ../objectstore/bluestore-bitmap.yaml 2024-03-27T21:42:45.196 INFO:teuthology.orchestra.run.smithi060.stdout: 404512 12 -rw-r--r-- 1 ubuntu ubuntu 12288 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/qa/run_xfstests-obsolete.sh 2024-03-27T21:42:45.196 INFO:teuthology.orchestra.run.smithi060.stdout: 402929 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/ceph-object-corpus 2024-03-27T21:42:45.196 INFO:teuthology.orchestra.run.smithi060.stdout: 402892 8 -rw-r--r-- 1 ubuntu ubuntu 5100 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.peoplemap 2024-03-27T21:42:45.197 INFO:teuthology.orchestra.run.smithi060.stdout: 402888 4 -rw-r--r-- 1 ubuntu ubuntu 785 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.gitmodule_mirrors 2024-03-27T21:42:45.197 INFO:teuthology.orchestra.run.smithi060.stdout: 402930 84 -rw-r--r-- 1 ubuntu ubuntu 85438 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/ceph.spec.in 2024-03-27T21:42:45.197 INFO:teuthology.orchestra.run.smithi060.stdout: 403126 4 -rwxr-xr-x 1 ubuntu ubuntu 2522 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/do_freebsd.sh 2024-03-27T21:42:45.197 INFO:teuthology.orchestra.run.smithi060.stdout: 402897 12 -rw-r--r-- 1 ubuntu ubuntu 8405 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/COPYING 2024-03-27T21:42:45.197 INFO:teuthology.orchestra.run.smithi060.stdout: 402828 4 drwxr-xr-x 8 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git 2024-03-27T21:42:45.197 INFO:teuthology.orchestra.run.smithi060.stdout: 402847 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git/refs 2024-03-27T21:42:45.197 INFO:teuthology.orchestra.run.smithi060.stdout: 402849 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:40 /home/ubuntu/cephtest/clone.client.0/.git/refs/tags 2024-03-27T21:42:45.197 INFO:teuthology.orchestra.run.smithi060.stdout: 402858 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git/refs/remotes 2024-03-27T21:42:45.197 INFO:teuthology.orchestra.run.smithi060.stdout: 402860 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git/refs/remotes/origin 2024-03-27T21:42:45.198 INFO:teuthology.orchestra.run.smithi060.stdout: 402861 4 -rw-r--r-- 1 ubuntu ubuntu 30 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git/refs/remotes/origin/HEAD 2024-03-27T21:42:45.198 INFO:teuthology.orchestra.run.smithi060.stdout: 402848 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git/refs/heads 2024-03-27T21:42:45.198 INFO:teuthology.orchestra.run.smithi060.stdout: 402868 4 -rw-r--r-- 1 ubuntu ubuntu 41 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git/refs/heads/main 2024-03-27T21:42:45.198 INFO:teuthology.orchestra.run.smithi060.stdout: 402850 1412 -rw-r--r-- 1 ubuntu ubuntu 1442590 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git/index 2024-03-27T21:42:45.198 INFO:teuthology.orchestra.run.smithi060.stdout: 402851 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:40 /home/ubuntu/cephtest/clone.client.0/.git/objects 2024-03-27T21:42:45.198 INFO:teuthology.orchestra.run.smithi060.stdout: 402853 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git/objects/pack 2024-03-27T21:42:45.198 INFO:teuthology.orchestra.run.smithi060.stdout: 402856 35588 -r--r--r-- 1 ubuntu ubuntu 36438200 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git/objects/pack/pack-bb51892f9a4dea4a5a93401a1f483759d5a197ae.idx 2024-03-27T21:42:45.198 INFO:teuthology.orchestra.run.smithi060.stdout: 402852 758312 -r--r--r-- 1 ubuntu ubuntu 776506181 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git/objects/pack/pack-bb51892f9a4dea4a5a93401a1f483759d5a197ae.pack 2024-03-27T21:42:45.199 INFO:teuthology.orchestra.run.smithi060.stdout: 402854 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:40 /home/ubuntu/cephtest/clone.client.0/.git/objects/info 2024-03-27T21:42:45.199 INFO:teuthology.orchestra.run.smithi060.stdout: 402829 4 -rw-r--r-- 1 ubuntu ubuntu 73 Mar 27 21:40 /home/ubuntu/cephtest/clone.client.0/.git/description 2024-03-27T21:42:45.199 INFO:teuthology.orchestra.run.smithi060.stdout: 402867 4 -rw-r--r-- 1 ubuntu ubuntu 21 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git/HEAD 2024-03-27T21:42:45.199 INFO:teuthology.orchestra.run.smithi060.stdout: 402830 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:40 /home/ubuntu/cephtest/clone.client.0/.git/info 2024-03-27T21:42:45.199 INFO:teuthology.orchestra.run.smithi060.stdout: 402831 4 -rw-r--r-- 1 ubuntu ubuntu 240 Mar 27 21:40 /home/ubuntu/cephtest/clone.client.0/.git/info/exclude 2024-03-27T21:42:45.199 INFO:teuthology.orchestra.run.smithi060.stdout: 402832 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:40 /home/ubuntu/cephtest/clone.client.0/.git/hooks 2024-03-27T21:42:45.199 INFO:teuthology.orchestra.run.smithi060.stdout: 402833 4 -rwxr-xr-x 1 ubuntu ubuntu 1647 Mar 27 21:40 /home/ubuntu/cephtest/clone.client.0/.git/hooks/pre-commit.sample 2024-03-27T21:42:45.199 INFO:teuthology.orchestra.run.smithi060.stdout: 402834 4 -rwxr-xr-x 1 ubuntu ubuntu 1496 Mar 27 21:40 /home/ubuntu/cephtest/clone.client.0/.git/hooks/prepare-commit-msg.sample 2024-03-27T21:42:45.200 INFO:teuthology.orchestra.run.smithi060.stdout: 402835 4 -rwxr-xr-x 1 ubuntu ubuntu 2787 Mar 27 21:40 /home/ubuntu/cephtest/clone.client.0/.git/hooks/push-to-checkout.sample 2024-03-27T21:42:45.200 INFO:teuthology.orchestra.run.smithi060.stdout: 402836 8 -rwxr-xr-x 1 ubuntu ubuntu 4726 Mar 27 21:40 /home/ubuntu/cephtest/clone.client.0/.git/hooks/fsmonitor-watchman.sample 2024-03-27T21:42:45.200 INFO:teuthology.orchestra.run.smithi060.stdout: 402837 8 -rwxr-xr-x 1 ubuntu ubuntu 4902 Mar 27 21:40 /home/ubuntu/cephtest/clone.client.0/.git/hooks/pre-rebase.sample 2024-03-27T21:42:45.200 INFO:teuthology.orchestra.run.smithi060.stdout: 402838 4 -rwxr-xr-x 1 ubuntu ubuntu 482 Mar 27 21:40 /home/ubuntu/cephtest/clone.client.0/.git/hooks/applypatch-msg.sample 2024-03-27T21:42:45.200 INFO:teuthology.orchestra.run.smithi060.stdout: 402839 4 -rwxr-xr-x 1 ubuntu ubuntu 548 Mar 27 21:40 /home/ubuntu/cephtest/clone.client.0/.git/hooks/pre-receive.sample 2024-03-27T21:42:45.200 INFO:teuthology.orchestra.run.smithi060.stdout: 402840 4 -rwxr-xr-x 1 ubuntu ubuntu 193 Mar 27 21:40 /home/ubuntu/cephtest/clone.client.0/.git/hooks/post-update.sample 2024-03-27T21:42:45.200 INFO:teuthology.orchestra.run.smithi060.stdout: 402841 4 -rwxr-xr-x 1 ubuntu ubuntu 3654 Mar 27 21:40 /home/ubuntu/cephtest/clone.client.0/.git/hooks/update.sample 2024-03-27T21:42:45.200 INFO:teuthology.orchestra.run.smithi060.stdout: 402842 4 -rwxr-xr-x 1 ubuntu ubuntu 1378 Mar 27 21:40 /home/ubuntu/cephtest/clone.client.0/.git/hooks/pre-push.sample 2024-03-27T21:42:45.200 INFO:teuthology.orchestra.run.smithi060.stdout: 402843 4 -rwxr-xr-x 1 ubuntu ubuntu 900 Mar 27 21:40 /home/ubuntu/cephtest/clone.client.0/.git/hooks/commit-msg.sample 2024-03-27T21:42:45.201 INFO:teuthology.orchestra.run.smithi060.stdout: 402844 4 -rwxr-xr-x 1 ubuntu ubuntu 428 Mar 27 21:40 /home/ubuntu/cephtest/clone.client.0/.git/hooks/pre-applypatch.sample 2024-03-27T21:42:45.201 INFO:teuthology.orchestra.run.smithi060.stdout: 402845 4 -rwxr-xr-x 1 ubuntu ubuntu 420 Mar 27 21:40 /home/ubuntu/cephtest/clone.client.0/.git/hooks/pre-merge-commit.sample 2024-03-27T21:42:45.201 INFO:teuthology.orchestra.run.smithi060.stdout: 402846 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:40 /home/ubuntu/cephtest/clone.client.0/.git/branches 2024-03-27T21:42:45.201 INFO:teuthology.orchestra.run.smithi060.stdout: 402862 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git/logs 2024-03-27T21:42:45.201 INFO:teuthology.orchestra.run.smithi060.stdout: 402863 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git/logs/refs 2024-03-27T21:42:45.201 INFO:teuthology.orchestra.run.smithi060.stdout: 402864 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git/logs/refs/remotes 2024-03-27T21:42:45.201 INFO:teuthology.orchestra.run.smithi060.stdout: 402865 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git/logs/refs/remotes/origin 2024-03-27T21:42:45.201 INFO:teuthology.orchestra.run.smithi060.stdout: 402866 4 -rw-r--r-- 1 ubuntu ubuntu 191 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git/logs/refs/remotes/origin/HEAD 2024-03-27T21:42:45.202 INFO:teuthology.orchestra.run.smithi060.stdout: 402870 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git/logs/refs/heads 2024-03-27T21:42:45.202 INFO:teuthology.orchestra.run.smithi060.stdout: 402871 4 -rw-r--r-- 1 ubuntu ubuntu 191 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git/logs/refs/heads/main 2024-03-27T21:42:45.202 INFO:teuthology.orchestra.run.smithi060.stdout: 402869 4 -rw-r--r-- 1 ubuntu ubuntu 191 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git/logs/HEAD 2024-03-27T21:42:45.202 INFO:teuthology.orchestra.run.smithi060.stdout: 402859 56 -rw-r--r-- 1 ubuntu ubuntu 55347 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git/packed-refs 2024-03-27T21:42:45.202 INFO:teuthology.orchestra.run.smithi060.stdout: 402855 4 -rw-r--r-- 1 ubuntu ubuntu 253 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.git/config 2024-03-27T21:42:45.202 INFO:teuthology.orchestra.run.smithi060.stdout: 403900 4 -rw-r--r-- 1 ubuntu ubuntu 116 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/doc_deps.deb.txt 2024-03-27T21:42:45.202 INFO:teuthology.orchestra.run.smithi060.stdout: 417160 12 -rwxr-xr-x 1 ubuntu ubuntu 11605 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/win32_deps_build.sh 2024-03-27T21:42:45.202 INFO:teuthology.orchestra.run.smithi060.stdout: 402931 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake 2024-03-27T21:42:45.203 INFO:teuthology.orchestra.run.smithi060.stdout: 402932 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules 2024-03-27T21:42:45.203 INFO:teuthology.orchestra.run.smithi060.stdout: 402935 12 -rw-r--r-- 1 ubuntu ubuntu 11325 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/BuildBoost.cmake 2024-03-27T21:42:45.203 INFO:teuthology.orchestra.run.smithi060.stdout: 403015 4 -rw-r--r-- 1 ubuntu ubuntu 655 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findzfs.cmake 2024-03-27T21:42:45.203 INFO:teuthology.orchestra.run.smithi060.stdout: 402965 4 -rw-r--r-- 1 ubuntu ubuntu 967 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindJeMalloc.cmake 2024-03-27T21:42:45.203 INFO:teuthology.orchestra.run.smithi060.stdout: 402984 4 -rw-r--r-- 1 ubuntu ubuntu 478 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindSQLite3.cmake 2024-03-27T21:42:45.203 INFO:teuthology.orchestra.run.smithi060.stdout: 403003 4 -rw-r--r-- 1 ubuntu ubuntu 2094 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findpmdk.cmake 2024-03-27T21:42:45.203 INFO:teuthology.orchestra.run.smithi060.stdout: 402991 4 -rw-r--r-- 1 ubuntu ubuntu 1155 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findblkid.cmake 2024-03-27T21:42:45.203 INFO:teuthology.orchestra.run.smithi060.stdout: 403004 4 -rw-r--r-- 1 ubuntu ubuntu 766 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findrdmacm.cmake 2024-03-27T21:42:45.203 INFO:teuthology.orchestra.run.smithi060.stdout: 402956 20 -rw-r--r-- 1 ubuntu ubuntu 17149 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindArrow.cmake 2024-03-27T21:42:45.204 INFO:teuthology.orchestra.run.smithi060.stdout: 402947 4 -rw-r--r-- 1 ubuntu ubuntu 2233 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Buildpmdk.cmake 2024-03-27T21:42:45.204 INFO:teuthology.orchestra.run.smithi060.stdout: 403018 4 -rw-r--r-- 1 ubuntu ubuntu 1829 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/LimitJobs.cmake 2024-03-27T21:42:45.204 INFO:teuthology.orchestra.run.smithi060.stdout: 402994 4 -rw-r--r-- 1 ubuntu ubuntu 1554 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Finddaxctl.cmake 2024-03-27T21:42:45.204 INFO:teuthology.orchestra.run.smithi060.stdout: 402954 4 -rw-r--r-- 1 ubuntu ubuntu 1990 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/CheckNasm.cmake 2024-03-27T21:42:45.204 INFO:teuthology.orchestra.run.smithi060.stdout: 402933 8 -rw-r--r-- 1 ubuntu ubuntu 5323 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/AddCephTest.cmake 2024-03-27T21:42:45.204 INFO:teuthology.orchestra.run.smithi060.stdout: 403009 4 -rw-r--r-- 1 ubuntu ubuntu 3795 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findutf8proc.cmake 2024-03-27T21:42:45.204 INFO:teuthology.orchestra.run.smithi060.stdout: 402993 4 -rw-r--r-- 1 ubuntu ubuntu 723 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findcap.cmake 2024-03-27T21:42:45.204 INFO:teuthology.orchestra.run.smithi060.stdout: 403000 4 -rw-r--r-- 1 ubuntu ubuntu 1476 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findmotr.cmake 2024-03-27T21:42:45.205 INFO:teuthology.orchestra.run.smithi060.stdout: 402958 116 -rw-r--r-- 1 ubuntu ubuntu 117846 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindBoost.cmake 2024-03-27T21:42:45.205 INFO:teuthology.orchestra.run.smithi060.stdout: 402964 4 -rw-r--r-- 1 ubuntu ubuntu 579 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindGSSApi.cmake 2024-03-27T21:42:45.205 INFO:teuthology.orchestra.run.smithi060.stdout: 402971 4 -rw-r--r-- 1 ubuntu ubuntu 443 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindNUMA.cmake 2024-03-27T21:42:45.205 INFO:teuthology.orchestra.run.smithi060.stdout: 402995 4 -rw-r--r-- 1 ubuntu ubuntu 1500 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Finddml.cmake 2024-03-27T21:42:45.205 INFO:teuthology.orchestra.run.smithi060.stdout: 402940 4 -rw-r--r-- 1 ubuntu ubuntu 1908 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/BuildQATzip.cmake 2024-03-27T21:42:45.205 INFO:teuthology.orchestra.run.smithi060.stdout: 402978 4 -rw-r--r-- 1 ubuntu ubuntu 1189 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindQAT.cmake 2024-03-27T21:42:45.205 INFO:teuthology.orchestra.run.smithi060.stdout: 403019 4 -rw-r--r-- 1 ubuntu ubuntu 3462 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/SIMDExt.cmake 2024-03-27T21:42:45.205 INFO:teuthology.orchestra.run.smithi060.stdout: 403010 4 -rw-r--r-- 1 ubuntu ubuntu 563 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Finduuid.cmake 2024-03-27T21:42:45.205 INFO:teuthology.orchestra.run.smithi060.stdout: 402979 4 -rw-r--r-- 1 ubuntu ubuntu 989 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindQATzip.cmake 2024-03-27T21:42:45.206 INFO:teuthology.orchestra.run.smithi060.stdout: 402946 4 -rw-r--r-- 1 ubuntu ubuntu 952 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Buildc-ares.cmake 2024-03-27T21:42:45.206 INFO:teuthology.orchestra.run.smithi060.stdout: 402961 4 -rw-r--r-- 1 ubuntu ubuntu 1989 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindDAOS.cmake 2024-03-27T21:42:45.206 INFO:teuthology.orchestra.run.smithi060.stdout: 403011 4 -rw-r--r-- 1 ubuntu ubuntu 1110 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findverbs.cmake 2024-03-27T21:42:45.206 INFO:teuthology.orchestra.run.smithi060.stdout: 402981 4 -rw-r--r-- 1 ubuntu ubuntu 998 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindRDKafka.cmake 2024-03-27T21:42:45.206 INFO:teuthology.orchestra.run.smithi060.stdout: 402966 4 -rw-r--r-- 1 ubuntu ubuntu 2545 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindLMDB.cmake 2024-03-27T21:42:45.206 INFO:teuthology.orchestra.run.smithi060.stdout: 402941 8 -rw-r--r-- 1 ubuntu ubuntu 5066 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/BuildRocksDB.cmake 2024-03-27T21:42:45.206 INFO:teuthology.orchestra.run.smithi060.stdout: 402948 4 -rw-r--r-- 1 ubuntu ubuntu 1602 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Builduring.cmake 2024-03-27T21:42:45.206 INFO:teuthology.orchestra.run.smithi060.stdout: 402963 4 -rw-r--r-- 1 ubuntu ubuntu 638 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindGMock.cmake 2024-03-27T21:42:45.207 INFO:teuthology.orchestra.run.smithi060.stdout: 402967 8 -rw-r--r-- 1 ubuntu ubuntu 4137 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindLTTngUST.cmake 2024-03-27T21:42:45.207 INFO:teuthology.orchestra.run.smithi060.stdout: 402997 4 -rw-r--r-- 1 ubuntu ubuntu 3077 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findgperftools.cmake 2024-03-27T21:42:45.207 INFO:teuthology.orchestra.run.smithi060.stdout: 403002 4 -rw-r--r-- 1 ubuntu ubuntu 1510 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findnl.cmake 2024-03-27T21:42:45.207 INFO:teuthology.orchestra.run.smithi060.stdout: 403014 4 -rw-r--r-- 1 ubuntu ubuntu 457 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findzbd.cmake 2024-03-27T21:42:45.207 INFO:teuthology.orchestra.run.smithi060.stdout: 402944 4 -rw-r--r-- 1 ubuntu ubuntu 2967 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/BuildUtf8proc.cmake 2024-03-27T21:42:45.207 INFO:teuthology.orchestra.run.smithi060.stdout: 402960 4 -rw-r--r-- 1 ubuntu ubuntu 540 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindCython.cmake 2024-03-27T21:42:45.207 INFO:teuthology.orchestra.run.smithi060.stdout: 402951 4 -rw-r--r-- 1 ubuntu ubuntu 3 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/CephCheck_link.map 2024-03-27T21:42:45.207 INFO:teuthology.orchestra.run.smithi060.stdout: 402985 4 -rw-r--r-- 1 ubuntu ubuntu 3189 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindSanitizers.cmake 2024-03-27T21:42:45.207 INFO:teuthology.orchestra.run.smithi060.stdout: 402999 4 -rw-r--r-- 1 ubuntu ubuntu 843 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findlibcryptsetup.cmake 2024-03-27T21:42:45.208 INFO:teuthology.orchestra.run.smithi060.stdout: 402945 4 -rw-r--r-- 1 ubuntu ubuntu 998 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/BuildZstd.cmake 2024-03-27T21:42:45.208 INFO:teuthology.orchestra.run.smithi060.stdout: 403022 4 -rwxr-xr-x 1 ubuntu ubuntu 1272 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/patch-dpdk-conf.sh 2024-03-27T21:42:45.208 INFO:teuthology.orchestra.run.smithi060.stdout: 402973 4 -rw-r--r-- 1 ubuntu ubuntu 1376 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindOpenLDAP.cmake 2024-03-27T21:42:45.208 INFO:teuthology.orchestra.run.smithi060.stdout: 402980 8 -rw-r--r-- 1 ubuntu ubuntu 4194 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindQatDrv.cmake 2024-03-27T21:42:45.208 INFO:teuthology.orchestra.run.smithi060.stdout: 402936 8 -rw-r--r-- 1 ubuntu ubuntu 6288 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/BuildDPDK.cmake 2024-03-27T21:42:45.208 INFO:teuthology.orchestra.run.smithi060.stdout: 402968 4 -rw-r--r-- 1 ubuntu ubuntu 1296 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindLZ4.cmake 2024-03-27T21:42:45.208 INFO:teuthology.orchestra.run.smithi060.stdout: 402949 4 -rw-r--r-- 1 ubuntu ubuntu 1748 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/CTags.cmake 2024-03-27T21:42:45.208 INFO:teuthology.orchestra.run.smithi060.stdout: 403013 4 -rw-r--r-- 1 ubuntu ubuntu 1732 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findyaml-cpp.cmake 2024-03-27T21:42:45.209 INFO:teuthology.orchestra.run.smithi060.stdout: 402988 4 -rw-r--r-- 1 ubuntu ubuntu 1544 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindZstd.cmake 2024-03-27T21:42:45.209 INFO:teuthology.orchestra.run.smithi060.stdout: 402938 4 -rw-r--r-- 1 ubuntu ubuntu 4032 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/BuildOpentelemetry.cmake 2024-03-27T21:42:45.209 INFO:teuthology.orchestra.run.smithi060.stdout: 402969 4 -rw-r--r-- 1 ubuntu ubuntu 471 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindLinuxZNS.cmake 2024-03-27T21:42:45.209 INFO:teuthology.orchestra.run.smithi060.stdout: 402972 4 -rw-r--r-- 1 ubuntu ubuntu 884 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindOATH.cmake 2024-03-27T21:42:45.209 INFO:teuthology.orchestra.run.smithi060.stdout: 402962 4 -rw-r--r-- 1 ubuntu ubuntu 2342 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindFUSE.cmake 2024-03-27T21:42:45.209 INFO:teuthology.orchestra.run.smithi060.stdout: 402952 8 -rw-r--r-- 1 ubuntu ubuntu 6357 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/CephChecks.cmake 2024-03-27T21:42:45.209 INFO:teuthology.orchestra.run.smithi060.stdout: 403005 4 -rw-r--r-- 1 ubuntu ubuntu 1025 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findsnappy.cmake 2024-03-27T21:42:45.209 INFO:teuthology.orchestra.run.smithi060.stdout: 402990 4 -rw-r--r-- 1 ubuntu ubuntu 846 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findbabeltrace.cmake 2024-03-27T21:42:45.210 INFO:teuthology.orchestra.run.smithi060.stdout: 402950 4 -rw-r--r-- 1 ubuntu ubuntu 14 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/CephCheck_link.c 2024-03-27T21:42:45.210 INFO:teuthology.orchestra.run.smithi060.stdout: 403001 4 -rw-r--r-- 1 ubuntu ubuntu 1506 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findndctl.cmake 2024-03-27T21:42:45.210 INFO:teuthology.orchestra.run.smithi060.stdout: 402983 4 -rw-r--r-- 1 ubuntu ubuntu 1667 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindRocksDB.cmake 2024-03-27T21:42:45.210 INFO:teuthology.orchestra.run.smithi060.stdout: 402942 4 -rw-r--r-- 1 ubuntu ubuntu 2940 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/BuildSPDK.cmake 2024-03-27T21:42:45.210 INFO:teuthology.orchestra.run.smithi060.stdout: 402989 4 -rw-r--r-- 1 ubuntu ubuntu 436 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findaio.cmake 2024-03-27T21:42:45.210 INFO:teuthology.orchestra.run.smithi060.stdout: 403016 4 -rw-r--r-- 1 ubuntu ubuntu 3960 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/GetGitRevisionDescription.cmake 2024-03-27T21:42:45.210 INFO:teuthology.orchestra.run.smithi060.stdout: 402959 4 -rw-r--r-- 1 ubuntu ubuntu 479 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindCUnit.cmake 2024-03-27T21:42:45.210 INFO:teuthology.orchestra.run.smithi060.stdout: 402970 4 -rw-r--r-- 1 ubuntu ubuntu 717 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindMake.cmake 2024-03-27T21:42:45.210 INFO:teuthology.orchestra.run.smithi060.stdout: 402957 4 -rw-r--r-- 1 ubuntu ubuntu 3771 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindBacktrace.cmake 2024-03-27T21:42:45.211 INFO:teuthology.orchestra.run.smithi060.stdout: 403007 4 -rw-r--r-- 1 ubuntu ubuntu 1142 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findudev.cmake 2024-03-27T21:42:45.211 INFO:teuthology.orchestra.run.smithi060.stdout: 402987 4 -rw-r--r-- 1 ubuntu ubuntu 139 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindStdFilesystem_test.cc 2024-03-27T21:42:45.211 INFO:teuthology.orchestra.run.smithi060.stdout: 402975 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindPython 2024-03-27T21:42:45.211 INFO:teuthology.orchestra.run.smithi060.stdout: 402976 200 -rw-r--r-- 1 ubuntu ubuntu 202007 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindPython/Support.cmake 2024-03-27T21:42:45.211 INFO:teuthology.orchestra.run.smithi060.stdout: 403012 4 -rw-r--r-- 1 ubuntu ubuntu 832 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findxfs.cmake 2024-03-27T21:42:45.211 INFO:teuthology.orchestra.run.smithi060.stdout: 402977 20 -rw-r--r-- 1 ubuntu ubuntu 18592 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindPython3.cmake 2024-03-27T21:42:45.211 INFO:teuthology.orchestra.run.smithi060.stdout: 402992 4 -rw-r--r-- 1 ubuntu ubuntu 947 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findc-ares.cmake 2024-03-27T21:42:45.211 INFO:teuthology.orchestra.run.smithi060.stdout: 402974 8 -rw-r--r-- 1 ubuntu ubuntu 5501 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindParquet.cmake 2024-03-27T21:42:45.212 INFO:teuthology.orchestra.run.smithi060.stdout: 402943 4 -rw-r--r-- 1 ubuntu ubuntu 628 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/BuildUBBD.cmake 2024-03-27T21:42:45.212 INFO:teuthology.orchestra.run.smithi060.stdout: 402996 4 -rw-r--r-- 1 ubuntu ubuntu 3376 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Finddpdk.cmake 2024-03-27T21:42:45.212 INFO:teuthology.orchestra.run.smithi060.stdout: 403017 4 -rw-r--r-- 1 ubuntu ubuntu 1587 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/GetGitRevisionDescription.cmake.in 2024-03-27T21:42:45.212 INFO:teuthology.orchestra.run.smithi060.stdout: 402934 8 -rw-r--r-- 1 ubuntu ubuntu 7184 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/BuildArrow.cmake 2024-03-27T21:42:45.212 INFO:teuthology.orchestra.run.smithi060.stdout: 402955 8 -rw-r--r-- 1 ubuntu ubuntu 6865 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Distutils.cmake 2024-03-27T21:42:45.212 INFO:teuthology.orchestra.run.smithi060.stdout: 402953 4 -rw-r--r-- 1 ubuntu ubuntu 2070 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/CheckCxxAtomic.cmake 2024-03-27T21:42:45.212 INFO:teuthology.orchestra.run.smithi060.stdout: 403008 4 -rw-r--r-- 1 ubuntu ubuntu 703 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Finduring.cmake 2024-03-27T21:42:45.212 INFO:teuthology.orchestra.run.smithi060.stdout: 402998 4 -rw-r--r-- 1 ubuntu ubuntu 859 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findkeyutils.cmake 2024-03-27T21:42:45.212 INFO:teuthology.orchestra.run.smithi060.stdout: 402939 4 -rw-r--r-- 1 ubuntu ubuntu 1889 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/BuildQAT.cmake 2024-03-27T21:42:45.213 INFO:teuthology.orchestra.run.smithi060.stdout: 402937 4 -rw-r--r-- 1 ubuntu ubuntu 1544 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/BuildFIO.cmake 2024-03-27T21:42:45.213 INFO:teuthology.orchestra.run.smithi060.stdout: 402986 4 -rw-r--r-- 1 ubuntu ubuntu 1750 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindStdFilesystem.cmake 2024-03-27T21:42:45.213 INFO:teuthology.orchestra.run.smithi060.stdout: 402982 4 -rw-r--r-- 1 ubuntu ubuntu 551 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/FindRabbitMQ.cmake 2024-03-27T21:42:45.213 INFO:teuthology.orchestra.run.smithi060.stdout: 403006 4 -rw-r--r-- 1 ubuntu ubuntu 1126 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/Findthrift.cmake 2024-03-27T21:42:45.213 INFO:teuthology.orchestra.run.smithi060.stdout: 403020 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/code_tests 2024-03-27T21:42:45.213 INFO:teuthology.orchestra.run.smithi060.stdout: 403021 4 -rw-r--r-- 1 ubuntu ubuntu 314 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/cmake/modules/code_tests/Sanitizers_fiber_test.cc 2024-03-27T21:42:45.213 INFO:teuthology.orchestra.run.smithi060.stdout: 402886 4 -rw-r--r-- 1 ubuntu ubuntu 1503 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.gitignore 2024-03-27T21:42:45.213 INFO:teuthology.orchestra.run.smithi060.stdout: 403125 4 -rwxr-xr-x 1 ubuntu ubuntu 2412 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/do_cmake.sh 2024-03-27T21:42:45.213 INFO:teuthology.orchestra.run.smithi060.stdout: 402910 16 -rw-r--r-- 1 ubuntu ubuntu 12859 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/SubmittingPatches-kernel.rst 2024-03-27T21:42:45.214 INFO:teuthology.orchestra.run.smithi060.stdout: 402919 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/bin 2024-03-27T21:42:45.214 INFO:teuthology.orchestra.run.smithi060.stdout: 402920 12 -rwxr-xr-x 1 ubuntu ubuntu 9140 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/bin/git-archive-all.sh 2024-03-27T21:42:45.214 INFO:teuthology.orchestra.run.smithi060.stdout: 404114 8 -rw-r--r-- 1 ubuntu ubuntu 4386 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/pom.xml 2024-03-27T21:42:45.214 INFO:teuthology.orchestra.run.smithi060.stdout: 408958 4 -rwxr-xr-x 1 ubuntu ubuntu 3014 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/run-make-check.sh 2024-03-27T21:42:45.214 INFO:teuthology.orchestra.run.smithi060.stdout: 402911 12 -rw-r--r-- 1 ubuntu ubuntu 11947 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/SubmittingPatches.rst 2024-03-27T21:42:45.214 INFO:teuthology.orchestra.run.smithi060.stdout: 417128 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/sudoers.d 2024-03-27T21:42:45.214 INFO:teuthology.orchestra.run.smithi060.stdout: 417129 4 -rw-r--r-- 1 ubuntu ubuntu 202 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/sudoers.d/ceph-smartctl 2024-03-27T21:42:45.214 INFO:teuthology.orchestra.run.smithi060.stdout: 402887 4 -rw-r--r-- 1 ubuntu ubuntu 161 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/.gitleaks.toml 2024-03-27T21:42:45.215 INFO:teuthology.orchestra.run.smithi060.stdout: 402909 20 -rw-r--r-- 1 ubuntu ubuntu 18703 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/SubmittingPatches-backports.rst 2024-03-27T21:42:45.215 INFO:teuthology.orchestra.run.smithi060.stdout: 404000 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/man 2024-03-27T21:42:45.215 INFO:teuthology.orchestra.run.smithi060.stdout: 404004 4 -rw-r--r-- 1 ubuntu ubuntu 2065 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/man/conf.py 2024-03-27T21:42:45.215 INFO:teuthology.orchestra.run.smithi060.stdout: 404002 4 -rw-r--r-- 1 ubuntu ubuntu 94 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/man/CMakeLists.txt 2024-03-27T21:42:45.215 INFO:teuthology.orchestra.run.smithi060.stdout: 404003 12 -rw-r--r-- 1 ubuntu ubuntu 8726 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/man/ceph_selinux.8 2024-03-27T21:42:45.215 INFO:teuthology.orchestra.run.smithi060.stdout: 404001 4 -rw-r--r-- 1 ubuntu ubuntu 15 Mar 27 21:41 /home/ubuntu/cephtest/clone.client.0/man/.gitignore 2024-03-27T21:42:45.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T21:42:45.216 INFO:teuthology.orchestra.run.smithi060.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-27T21:42:45.217 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-27T21:42:45.225 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-27T21:42:45.235 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-27T21:42:45.281 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi060 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-27T21:42:45.282 INFO:teuthology.nuke:Checking targets against current locks 2024-03-27T21:42:45.312 DEBUG:teuthology.nuke:shortname: smithi060 2024-03-27T21:42:45.312 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-27T21:42:45.331 DEBUG:teuthology.nuke:shortname: smithi191 2024-03-27T21:42:45.331 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-27T21:42:45.334 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-27_14:58:57-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626549', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-27 21:23:34.404231', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFseykjbMcoy37jwDf5haCtBthQp6+lf1FiaXkGRWge5FX0PR1O4cJ2V30pH6x72tmNxBTPzCx9KT7u+fUIv6h8='} 2024-03-27T21:42:45.354 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-27_14:58:57-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626549', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-27 21:23:34.405724', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFEoU/N98mvWvDGXsH9+Sn5SsdMdzH4v6vuk/Nq61Sr8dUL3puAnexOkO+5lZaf8xGqQoaGc/SGAsyBoAbeo/t0='} 2024-03-27T21:42:45.374 INFO:teuthology.orchestra.console.smithi060:Power off 2024-03-27T21:42:45.375 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-27T21:42:45.402 DEBUG:teuthology.orchestra.console.smithi060:power off output: Chassis Power Control: Down/Off 2024-03-27T21:42:45.402 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T21:42:45.421 INFO:teuthology.orchestra.console.smithi191:Power off 2024-03-27T21:42:45.421 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-27T21:42:45.432 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is on 2024-03-27T21:42:45.446 DEBUG:teuthology.orchestra.console.smithi191:power off output: Chassis Power Control: Down/Off 2024-03-27T21:42:45.446 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T21:42:45.469 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-03-27T21:42:49.434 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T21:42:49.470 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T21:42:49.545 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is on 2024-03-27T21:42:49.580 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-03-27T21:42:53.548 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T21:42:53.582 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T21:42:53.659 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is off 2024-03-27T21:42:53.692 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is off 2024-03-27T21:42:53.761 INFO:teuthology.orchestra.console.smithi060:Power off completed 2024-03-27T21:42:53.794 INFO:teuthology.orchestra.console.smithi191:Power off completed 2024-03-27T21:42:53.901 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} duration: 645.1973855495453 failure_reason: 'Command failed on smithi060 with status 128: ''rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7c37d80f7a88f4eb561c4fe02558660a650b10ac''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=fb7e5aa0366943489b588c0fdabf6654 status: fail success: false 2024-03-27T21:42:53.902 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-27T21:42:53.994 INFO:teuthology.run:FAIL